2024-09-16T15:28:21.694 INFO:root:teuthology version: 1.2.2.dev2+g3752d38 2024-09-16T15:28:21.713 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-09-16T15:28:21.797 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/avan-2024-09-16_15:21:40-fs:volumes-wip-athakkar-testing-2024-09-16-1454-distro-default-smithi/7907351 branch: wip-athakkar-testing-2024-09-16-1454 description: fs:volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mgr mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} tasks/volumes/{overrides test/misc}} email: athakkar@redhat.com first_in_suite: false job_id: '7907351' kernel: kdb: 1 sha1: distro ktype: distro last_in_suite: false log-rotate: ceph-mds: 10G ceph-osd: 10G machine_type: smithi name: avan-2024-09-16_15:21:40-fs:volumes-wip-athakkar-testing-2024-09-16-1454-distro-default-smithi no_nested_subset: false openstack: - volumes: count: 4 size: 20 - machine: disk: 200 os_type: ubuntu os_version: '22.04' overrides: admin_socket: branch: wip-athakkar-testing-2024-09-16-1454 ceph: cluster-conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 rados mon op timeout: 900 rados osd op timeout: 900 mds: debug mds: 20 debug mds balancer: 20 debug ms: 2 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mon: mon op complaint time: 120 mon warn on pool no app: false osd: osd op complaint time: 180 conf: client: pid file: '' mgr: client mount timeout: 30 debug client: 20 debug finisher: 20 debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon down mkfs grace: 300 osd: bdev async discard: true bdev enable discard: true bluestore allocator: bitmap bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug ms: 1 debug osd: 20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - FS_DEGRADED - fs.*is degraded - filesystem is degraded - FS_INLINE_DATA_DEPRECATED - FS_WITH_FAILED_MDS - MDS_ALL_DOWN - filesystem is offline - MDS_DAMAGE - MDS_DEGRADED - MDS_FAILED - MDS_INSUFFICIENT_STANDBY - insufficient standby MDS daemons available - MDS_UP_LESS_THAN_MAX - online, but wants - filesystem is online with fewer MDS than max_mds - POOL_APP_NOT_ENABLED - do not have an application enabled - overall HEALTH_ - Replacing daemon - deprecated feature inline_data - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding - PG_AVAILABILITY - PG_DEGRADED - Reduced data availability - Degraded data redundancy - OSD full dropping all updates - OSD near full - pausewr flag - failsafe engaged, dropping updates - failsafe disengaged, no longer dropping - is full \(reached quota - POOL_FULL - POOL_BACKFILLFULL sha1: 33f95d2ae1bcbe7eb04404303f179eeeaa6982df ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: flavor: default sha1: 33f95d2ae1bcbe7eb04404303f179eeeaa6982df thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-athakkar-testing-2024-09-16-1454 sha1: 33f95d2ae1bcbe7eb04404303f179eeeaa6982df owner: scheduled_avan@teuthology priority: 80 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mgr.y - mds.a - mds.c - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - client.2 - - mon.b - mon.c - mgr.x - mds.b - mds.d - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - client.3 seed: 3077 sha1: 33f95d2ae1bcbe7eb04404303f179eeeaa6982df sleep_before_teardown: 0 subset: 1/20 suite: fs:volumes suite_branch: wip-athakkar-testing-2024-09-16-1454 suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_33f95d2ae1bcbe7eb04404303f179eeeaa6982df/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: 33f95d2ae1bcbe7eb04404303f179eeeaa6982df syslog: ignorelist: - WARNING*.*check_session_state - WARNING*.*__ceph_remove_cap targets: smithi123.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJs6B1iWsHgQDJFra9/dCfSvhqd/H6C6e3mV32azcWpu9X8Fd4JgFXbmLpEoVvAqxax/eLGBI7hekTSjm3QvhPE= smithi184.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBA6D966A33EyqjY+W49aOBG+J6GnrD854kU+AJY//NJ+Yh7P7n9i7rnt27Yda6isEa0RXPpJGM+5xYH3FJ1NbfE= tasks: - install: extra_packages: deb: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror rpm: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror extra_system_packages: deb: - bison - flex - libelf-dev - libssl-dev - network-manager - iproute2 - util-linux - dump - indent - libaio-dev - libtool-bin - uuid-dev - xfslibs-dev - postgresql - postgresql-client - postgresql-common - postgresql-contrib rpm: - bison - flex - elfutils-libelf-devel - openssl-devel - NetworkManager - iproute - util-linux - libacl-devel - libaio-devel - libattr-devel - libtool - libuuid-devel - xfsdump - xfsprogs - xfsprogs-devel - libaio-devel - libtool - libuuid-devel - xfsprogs-devel - postgresql - postgresql-server - postgresql-contrib - ceph: create_rbd_pool: false - ceph-fuse: null - cephfs_test_runner: fail_on_skip: false modules: - tasks.cephfs.test_volumes.TestMisc teuthology: fragments_dropped: [] meta: {} postmerge: - local function is_kupstream() return false end - local function is_kdistro() return false end - local function is_fuse() return true end - local function syntax_version() return '' end teuthology_branch: main teuthology_sha1: 3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a timestamp: 2024-09-16_15:21:40 tube: smithi user: avan verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.3401856 2024-09-16T15:28:21.798 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_33f95d2ae1bcbe7eb04404303f179eeeaa6982df/qa; will attempt to use it 2024-09-16T15:28:21.798 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_33f95d2ae1bcbe7eb04404303f179eeeaa6982df/qa/tasks 2024-09-16T15:28:21.799 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-09-16T15:28:21.800 INFO:teuthology.task.internal:Checking packages... 2024-09-16T15:28:21.820 INFO:teuthology.task.internal:Checking packages for os_type 'ubuntu', flavor 'default' and ceph hash '33f95d2ae1bcbe7eb04404303f179eeeaa6982df' 2024-09-16T15:28:21.820 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-09-16T15:28:21.820 INFO:teuthology.packaging:ref: None 2024-09-16T15:28:21.820 INFO:teuthology.packaging:tag: None 2024-09-16T15:28:21.820 INFO:teuthology.packaging:branch: wip-athakkar-testing-2024-09-16-1454 2024-09-16T15:28:21.820 INFO:teuthology.packaging:sha1: 33f95d2ae1bcbe7eb04404303f179eeeaa6982df 2024-09-16T15:28:21.820 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&ref=wip-athakkar-testing-2024-09-16-1454 2024-09-16T15:28:21.967 INFO:teuthology.task.internal:Found packages for ceph version 19.3.0-4938-g33f95d2a-1jammy 2024-09-16T15:28:21.968 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-09-16T15:28:21.975 INFO:teuthology.task.internal:no buildpackages task found 2024-09-16T15:28:21.975 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-09-16T15:28:22.043 INFO:teuthology.task.internal:Saving configuration 2024-09-16T15:28:22.059 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-09-16T15:28:22.154 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-09-16T15:28:22.176 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi123.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/avan-2024-09-16_15:21:40-fs:volumes-wip-athakkar-testing-2024-09-16-1454-distro-default-smithi/7907351', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-09-16 15:22:48.218374', 'locked_by': 'scheduled_avan@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAILA0VpA8QR1Qb4twxHCc6FWxWUJNj6sGB639VmreywDQ'} 2024-09-16T15:28:22.196 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi184.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/avan-2024-09-16_15:21:40-fs:volumes-wip-athakkar-testing-2024-09-16-1454-distro-default-smithi/7907351', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-09-16 15:22:48.216908', 'locked_by': 'scheduled_avan@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIK1sXOucX7VDH19mm7D8WTH37KunuDzkZqp/lqUqc1oq'} 2024-09-16T15:28:22.196 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-09-16T15:28:22.203 INFO:teuthology.task.internal:roles: ubuntu@smithi123.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'client.2'] 2024-09-16T15:28:22.203 INFO:teuthology.task.internal:roles: ubuntu@smithi184.front.sepia.ceph.com - ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'client.3'] 2024-09-16T15:28:22.203 INFO:teuthology.run_tasks:Running task console_log... 2024-09-16T15:28:22.424 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f4d61c23be0>, signals=[15]) 2024-09-16T15:28:22.424 INFO:teuthology.run_tasks:Running task internal.connect... 2024-09-16T15:28:22.501 INFO:teuthology.task.internal:Opening connections... 2024-09-16T15:28:22.501 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi123.front.sepia.ceph.com 2024-09-16T15:28:22.502 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi123.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T15:28:22.580 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi184.front.sepia.ceph.com 2024-09-16T15:28:22.580 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi184.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T15:28:22.655 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-09-16T15:28:22.661 DEBUG:teuthology.orchestra.run.smithi123:> uname -m 2024-09-16T15:28:22.666 INFO:teuthology.orchestra.run.smithi123.stdout:x86_64 2024-09-16T15:28:22.666 DEBUG:teuthology.orchestra.run.smithi123:> cat /etc/os-release 2024-09-16T15:28:22.717 INFO:teuthology.orchestra.run.smithi123.stdout:PRETTY_NAME="Ubuntu 22.04.1 LTS" 2024-09-16T15:28:22.717 INFO:teuthology.orchestra.run.smithi123.stdout:NAME="Ubuntu" 2024-09-16T15:28:22.717 INFO:teuthology.orchestra.run.smithi123.stdout:VERSION_ID="22.04" 2024-09-16T15:28:22.717 INFO:teuthology.orchestra.run.smithi123.stdout:VERSION="22.04.1 LTS (Jammy Jellyfish)" 2024-09-16T15:28:22.717 INFO:teuthology.orchestra.run.smithi123.stdout:VERSION_CODENAME=jammy 2024-09-16T15:28:22.718 INFO:teuthology.orchestra.run.smithi123.stdout:ID=ubuntu 2024-09-16T15:28:22.718 INFO:teuthology.orchestra.run.smithi123.stdout:ID_LIKE=debian 2024-09-16T15:28:22.718 INFO:teuthology.orchestra.run.smithi123.stdout:HOME_URL="https://www.ubuntu.com/" 2024-09-16T15:28:22.718 INFO:teuthology.orchestra.run.smithi123.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2024-09-16T15:28:22.718 INFO:teuthology.orchestra.run.smithi123.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2024-09-16T15:28:22.718 INFO:teuthology.orchestra.run.smithi123.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2024-09-16T15:28:22.718 INFO:teuthology.orchestra.run.smithi123.stdout:UBUNTU_CODENAME=jammy 2024-09-16T15:28:22.718 INFO:teuthology.lock.ops:Updating smithi123.front.sepia.ceph.com on lock server 2024-09-16T15:28:22.745 DEBUG:teuthology.orchestra.run.smithi184:> uname -m 2024-09-16T15:28:22.750 INFO:teuthology.orchestra.run.smithi184.stdout:x86_64 2024-09-16T15:28:22.750 DEBUG:teuthology.orchestra.run.smithi184:> cat /etc/os-release 2024-09-16T15:28:22.797 INFO:teuthology.orchestra.run.smithi184.stdout:PRETTY_NAME="Ubuntu 22.04.1 LTS" 2024-09-16T15:28:22.797 INFO:teuthology.orchestra.run.smithi184.stdout:NAME="Ubuntu" 2024-09-16T15:28:22.797 INFO:teuthology.orchestra.run.smithi184.stdout:VERSION_ID="22.04" 2024-09-16T15:28:22.797 INFO:teuthology.orchestra.run.smithi184.stdout:VERSION="22.04.1 LTS (Jammy Jellyfish)" 2024-09-16T15:28:22.798 INFO:teuthology.orchestra.run.smithi184.stdout:VERSION_CODENAME=jammy 2024-09-16T15:28:22.798 INFO:teuthology.orchestra.run.smithi184.stdout:ID=ubuntu 2024-09-16T15:28:22.798 INFO:teuthology.orchestra.run.smithi184.stdout:ID_LIKE=debian 2024-09-16T15:28:22.798 INFO:teuthology.orchestra.run.smithi184.stdout:HOME_URL="https://www.ubuntu.com/" 2024-09-16T15:28:22.798 INFO:teuthology.orchestra.run.smithi184.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2024-09-16T15:28:22.798 INFO:teuthology.orchestra.run.smithi184.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2024-09-16T15:28:22.798 INFO:teuthology.orchestra.run.smithi184.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2024-09-16T15:28:22.798 INFO:teuthology.orchestra.run.smithi184.stdout:UBUNTU_CODENAME=jammy 2024-09-16T15:28:22.798 INFO:teuthology.lock.ops:Updating smithi184.front.sepia.ceph.com on lock server 2024-09-16T15:28:22.821 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-09-16T15:28:22.829 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-09-16T15:28:22.898 INFO:teuthology.task.internal:Checking for old test directory... 2024-09-16T15:28:22.898 DEBUG:teuthology.orchestra.run.smithi123:> test '!' -e /home/ubuntu/cephtest 2024-09-16T15:28:22.900 DEBUG:teuthology.orchestra.run.smithi184:> test '!' -e /home/ubuntu/cephtest 2024-09-16T15:28:22.905 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-09-16T15:28:23.009 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-09-16T15:28:23.009 DEBUG:teuthology.orchestra.run.smithi123:> test -z $(ls -A /var/lib/ceph) 2024-09-16T15:28:23.011 DEBUG:teuthology.orchestra.run.smithi184:> test -z $(ls -A /var/lib/ceph) 2024-09-16T15:28:23.022 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-09-16T15:28:23.184 INFO:teuthology.run_tasks:Running task kernel... 2024-09-16T15:28:23.198 INFO:teuthology.task.kernel:normalize config orig: {'kdb': 1, 'sha1': 'distro'} 2024-09-16T15:28:23.198 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': 1, 'sha1': 'distro'}, 'mon.b': {'kdb': 1, 'sha1': 'distro'}}, timeout 300 2024-09-16T15:28:23.198 DEBUG:teuthology.orchestra.run.smithi123:> test -f /run/.containerenv -o -f /.dockerenv 2024-09-16T15:28:23.199 DEBUG:teuthology.orchestra.run.smithi184:> test -f /run/.containerenv -o -f /.dockerenv 2024-09-16T15:28:23.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T15:28:23.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T15:28:23.204 DEBUG:teuthology.orchestra.run.smithi184:> uname -r 2024-09-16T15:28:23.204 DEBUG:teuthology.orchestra.run.smithi123:> uname -r 2024-09-16T15:28:23.250 INFO:teuthology.orchestra.run.smithi123.stdout:5.15.0-56-generic 2024-09-16T15:28:23.250 INFO:teuthology.task.kernel:Running kernel on smithi123: 5.15.0-56-generic 2024-09-16T15:28:23.250 DEBUG:teuthology.orchestra.run.smithi123:> sudo apt-get clean 2024-09-16T15:28:23.253 INFO:teuthology.orchestra.run.smithi184.stdout:5.15.0-56-generic 2024-09-16T15:28:23.253 INFO:teuthology.task.kernel:Running kernel on smithi184: 5.15.0-56-generic 2024-09-16T15:28:23.253 DEBUG:teuthology.orchestra.run.smithi184:> sudo apt-get clean 2024-09-16T15:28:23.393 DEBUG:teuthology.orchestra.run.smithi123:> sudo apt-get update 2024-09-16T15:28:23.396 DEBUG:teuthology.orchestra.run.smithi184:> sudo apt-get update 2024-09-16T15:28:23.579 INFO:teuthology.orchestra.run.smithi184.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-09-16T15:28:23.579 INFO:teuthology.orchestra.run.smithi184.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy-updates InRelease [128 kB] 2024-09-16T15:28:23.579 INFO:teuthology.orchestra.run.smithi184.stdout:Get:3 http://security.ubuntu.com/ubuntu jammy-security InRelease [129 kB] 2024-09-16T15:28:23.650 INFO:teuthology.orchestra.run.smithi184.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease [127 kB] 2024-09-16T15:28:23.684 INFO:teuthology.orchestra.run.smithi123.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-09-16T15:28:23.696 INFO:teuthology.orchestra.run.smithi123.stdout:Get:2 http://security.ubuntu.com/ubuntu jammy-security InRelease [129 kB] 2024-09-16T15:28:23.745 INFO:teuthology.orchestra.run.smithi123.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease [128 kB] 2024-09-16T15:28:23.816 INFO:teuthology.orchestra.run.smithi184.stdout:Get:5 http://security.ubuntu.com/ubuntu jammy-security/main i386 Packages [539 kB] 2024-09-16T15:28:23.883 INFO:teuthology.orchestra.run.smithi184.stdout:Get:6 http://security.ubuntu.com/ubuntu jammy-security/main amd64 Packages [1,806 kB] 2024-09-16T15:28:23.888 INFO:teuthology.orchestra.run.smithi184.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 Packages [2,025 kB] 2024-09-16T15:28:23.920 INFO:teuthology.orchestra.run.smithi184.stdout:Get:8 http://security.ubuntu.com/ubuntu jammy-security/main Translation-en [295 kB] 2024-09-16T15:28:23.923 INFO:teuthology.orchestra.run.smithi184.stdout:Get:9 http://security.ubuntu.com/ubuntu jammy-security/main amd64 c-n-f Metadata [13.3 kB] 2024-09-16T15:28:23.923 INFO:teuthology.orchestra.run.smithi184.stdout:Get:10 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 Packages [2,377 kB] 2024-09-16T15:28:23.942 INFO:teuthology.orchestra.run.smithi184.stdout:Get:11 http://security.ubuntu.com/ubuntu jammy-security/restricted i386 Packages [37.7 kB] 2024-09-16T15:28:23.942 INFO:teuthology.orchestra.run.smithi184.stdout:Get:12 http://security.ubuntu.com/ubuntu jammy-security/restricted Translation-en [409 kB] 2024-09-16T15:28:23.945 INFO:teuthology.orchestra.run.smithi184.stdout:Get:13 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 c-n-f Metadata [584 B] 2024-09-16T15:28:23.945 INFO:teuthology.orchestra.run.smithi184.stdout:Get:14 http://security.ubuntu.com/ubuntu jammy-security/universe i386 Packages [627 kB] 2024-09-16T15:28:23.950 INFO:teuthology.orchestra.run.smithi184.stdout:Get:15 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 Packages [902 kB] 2024-09-16T15:28:23.956 INFO:teuthology.orchestra.run.smithi184.stdout:Get:16 http://security.ubuntu.com/ubuntu jammy-security/universe Translation-en [176 kB] 2024-09-16T15:28:23.957 INFO:teuthology.orchestra.run.smithi184.stdout:Get:17 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 c-n-f Metadata [19.2 kB] 2024-09-16T15:28:23.978 INFO:teuthology.orchestra.run.smithi184.stdout:Get:18 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 Packages [37.2 kB] 2024-09-16T15:28:23.978 INFO:teuthology.orchestra.run.smithi184.stdout:Get:19 http://security.ubuntu.com/ubuntu jammy-security/multiverse i386 Packages [1,356 B] 2024-09-16T15:28:23.978 INFO:teuthology.orchestra.run.smithi184.stdout:Get:20 http://security.ubuntu.com/ubuntu jammy-security/multiverse Translation-en [7,588 B] 2024-09-16T15:28:23.978 INFO:teuthology.orchestra.run.smithi184.stdout:Get:21 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 c-n-f Metadata [228 B] 2024-09-16T15:28:23.996 INFO:teuthology.orchestra.run.smithi184.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy-updates/main i386 Packages [696 kB] 2024-09-16T15:28:24.004 INFO:teuthology.orchestra.run.smithi184.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy-updates/main Translation-en [353 kB] 2024-09-16T15:28:24.008 INFO:teuthology.orchestra.run.smithi184.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 c-n-f Metadata [17.8 kB] 2024-09-16T15:28:24.009 INFO:teuthology.orchestra.run.smithi184.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 Packages [2,441 kB] 2024-09-16T15:28:24.034 INFO:teuthology.orchestra.run.smithi184.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy-updates/restricted i386 Packages [39.3 kB] 2024-09-16T15:28:24.034 INFO:teuthology.orchestra.run.smithi184.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy-updates/restricted Translation-en [420 kB] 2024-09-16T15:28:24.038 INFO:teuthology.orchestra.run.smithi184.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 c-n-f Metadata [616 B] 2024-09-16T15:28:24.038 INFO:teuthology.orchestra.run.smithi184.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 Packages [1,124 kB] 2024-09-16T15:28:24.045 INFO:teuthology.orchestra.run.smithi184.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy-updates/universe i386 Packages [732 kB] 2024-09-16T15:28:24.050 INFO:teuthology.orchestra.run.smithi184.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy-updates/universe Translation-en [261 kB] 2024-09-16T15:28:24.052 INFO:teuthology.orchestra.run.smithi184.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 c-n-f Metadata [26.1 kB] 2024-09-16T15:28:24.065 INFO:teuthology.orchestra.run.smithi184.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 Packages [43.3 kB] 2024-09-16T15:28:24.065 INFO:teuthology.orchestra.run.smithi184.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse i386 Packages [4,752 B] 2024-09-16T15:28:24.065 INFO:teuthology.orchestra.run.smithi184.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse Translation-en [10.8 kB] 2024-09-16T15:28:24.066 INFO:teuthology.orchestra.run.smithi184.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 c-n-f Metadata [444 B] 2024-09-16T15:28:24.066 INFO:teuthology.orchestra.run.smithi184.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 Packages [67.8 kB] 2024-09-16T15:28:24.066 INFO:teuthology.orchestra.run.smithi184.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy-backports/main i386 Packages [59.9 kB] 2024-09-16T15:28:24.077 INFO:teuthology.orchestra.run.smithi184.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy-backports/main Translation-en [11.1 kB] 2024-09-16T15:28:24.077 INFO:teuthology.orchestra.run.smithi184.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 c-n-f Metadata [388 B] 2024-09-16T15:28:24.094 INFO:teuthology.orchestra.run.smithi184.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy-backports/universe i386 Packages [17.3 kB] 2024-09-16T15:28:24.094 INFO:teuthology.orchestra.run.smithi184.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 Packages [28.8 kB] 2024-09-16T15:28:24.094 INFO:teuthology.orchestra.run.smithi184.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy-backports/universe Translation-en [16.5 kB] 2024-09-16T15:28:24.095 INFO:teuthology.orchestra.run.smithi184.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 c-n-f Metadata [672 B] 2024-09-16T15:28:24.101 INFO:teuthology.orchestra.run.smithi123.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease [127 kB] 2024-09-16T15:28:24.141 INFO:teuthology.orchestra.run.smithi123.stdout:Get:5 http://security.ubuntu.com/ubuntu jammy-security/main i386 Packages [539 kB] 2024-09-16T15:28:24.239 INFO:teuthology.orchestra.run.smithi123.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 Packages [2,025 kB] 2024-09-16T15:28:24.466 INFO:teuthology.orchestra.run.smithi123.stdout:Get:7 http://security.ubuntu.com/ubuntu jammy-security/main amd64 Packages [1,806 kB] 2024-09-16T15:28:24.557 INFO:teuthology.orchestra.run.smithi123.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/main i386 Packages [696 kB] 2024-09-16T15:28:24.592 INFO:teuthology.orchestra.run.smithi123.stdout:Get:9 http://archive.ubuntu.com/ubuntu jammy-updates/main Translation-en [353 kB] 2024-09-16T15:28:24.607 INFO:teuthology.orchestra.run.smithi123.stdout:Get:10 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 c-n-f Metadata [17.8 kB] 2024-09-16T15:28:24.607 INFO:teuthology.orchestra.run.smithi123.stdout:Get:11 http://archive.ubuntu.com/ubuntu jammy-updates/restricted i386 Packages [39.3 kB] 2024-09-16T15:28:24.609 INFO:teuthology.orchestra.run.smithi123.stdout:Get:12 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 Packages [2,441 kB] 2024-09-16T15:28:24.640 INFO:teuthology.orchestra.run.smithi123.stdout:Get:13 http://security.ubuntu.com/ubuntu jammy-security/main Translation-en [295 kB] 2024-09-16T15:28:24.653 INFO:teuthology.orchestra.run.smithi123.stdout:Get:14 http://security.ubuntu.com/ubuntu jammy-security/main amd64 c-n-f Metadata [13.3 kB] 2024-09-16T15:28:24.653 INFO:teuthology.orchestra.run.smithi123.stdout:Get:15 http://security.ubuntu.com/ubuntu jammy-security/restricted i386 Packages [37.7 kB] 2024-09-16T15:28:24.655 INFO:teuthology.orchestra.run.smithi123.stdout:Get:16 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 Packages [2,377 kB] 2024-09-16T15:28:24.678 INFO:teuthology.orchestra.run.smithi123.stdout:Get:17 http://archive.ubuntu.com/ubuntu jammy-updates/restricted Translation-en [420 kB] 2024-09-16T15:28:24.706 INFO:teuthology.orchestra.run.smithi123.stdout:Get:18 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 c-n-f Metadata [616 B] 2024-09-16T15:28:24.706 INFO:teuthology.orchestra.run.smithi123.stdout:Get:19 http://archive.ubuntu.com/ubuntu jammy-updates/universe i386 Packages [732 kB] 2024-09-16T15:28:24.720 INFO:teuthology.orchestra.run.smithi123.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 Packages [1,124 kB] 2024-09-16T15:28:24.741 INFO:teuthology.orchestra.run.smithi123.stdout:Get:21 http://security.ubuntu.com/ubuntu jammy-security/restricted Translation-en [409 kB] 2024-09-16T15:28:24.746 INFO:teuthology.orchestra.run.smithi123.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy-updates/universe Translation-en [261 kB] 2024-09-16T15:28:24.751 INFO:teuthology.orchestra.run.smithi123.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 c-n-f Metadata [26.1 kB] 2024-09-16T15:28:24.752 INFO:teuthology.orchestra.run.smithi123.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 Packages [43.3 kB] 2024-09-16T15:28:24.753 INFO:teuthology.orchestra.run.smithi123.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse i386 Packages [4,752 B] 2024-09-16T15:28:24.754 INFO:teuthology.orchestra.run.smithi123.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse Translation-en [10.8 kB] 2024-09-16T15:28:24.758 INFO:teuthology.orchestra.run.smithi123.stdout:Get:27 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 c-n-f Metadata [584 B] 2024-09-16T15:28:24.759 INFO:teuthology.orchestra.run.smithi123.stdout:Get:28 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 Packages [902 kB] 2024-09-16T15:28:24.795 INFO:teuthology.orchestra.run.smithi123.stdout:Get:29 http://security.ubuntu.com/ubuntu jammy-security/universe i386 Packages [627 kB] 2024-09-16T15:28:24.805 INFO:teuthology.orchestra.run.smithi123.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 c-n-f Metadata [444 B] 2024-09-16T15:28:24.806 INFO:teuthology.orchestra.run.smithi123.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 Packages [67.8 kB] 2024-09-16T15:28:24.808 INFO:teuthology.orchestra.run.smithi123.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy-backports/main i386 Packages [59.9 kB] 2024-09-16T15:28:24.812 INFO:teuthology.orchestra.run.smithi123.stdout:Get:33 http://security.ubuntu.com/ubuntu jammy-security/universe Translation-en [176 kB] 2024-09-16T15:28:24.816 INFO:teuthology.orchestra.run.smithi123.stdout:Get:34 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 c-n-f Metadata [19.2 kB] 2024-09-16T15:28:24.816 INFO:teuthology.orchestra.run.smithi123.stdout:Get:35 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 Packages [37.2 kB] 2024-09-16T15:28:24.818 INFO:teuthology.orchestra.run.smithi123.stdout:Get:36 http://security.ubuntu.com/ubuntu jammy-security/multiverse i386 Packages [1,356 B] 2024-09-16T15:28:24.818 INFO:teuthology.orchestra.run.smithi123.stdout:Get:37 http://security.ubuntu.com/ubuntu jammy-security/multiverse Translation-en [7,588 B] 2024-09-16T15:28:24.836 INFO:teuthology.orchestra.run.smithi123.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy-backports/main Translation-en [11.1 kB] 2024-09-16T15:28:24.837 INFO:teuthology.orchestra.run.smithi123.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 c-n-f Metadata [388 B] 2024-09-16T15:28:24.837 INFO:teuthology.orchestra.run.smithi123.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy-backports/universe i386 Packages [17.3 kB] 2024-09-16T15:28:24.838 INFO:teuthology.orchestra.run.smithi123.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 Packages [28.8 kB] 2024-09-16T15:28:24.838 INFO:teuthology.orchestra.run.smithi123.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy-backports/universe Translation-en [16.5 kB] 2024-09-16T15:28:24.885 INFO:teuthology.orchestra.run.smithi123.stdout:Get:43 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 c-n-f Metadata [228 B] 2024-09-16T15:28:24.887 INFO:teuthology.orchestra.run.smithi123.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 c-n-f Metadata [672 B] 2024-09-16T15:28:30.248 INFO:teuthology.orchestra.run.smithi184.stdout:Fetched 16.0 MB in 2s (7,600 kB/s) 2024-09-16T15:28:30.840 INFO:teuthology.orchestra.run.smithi123.stdout:Fetched 16.0 MB in 3s (5,757 kB/s) 2024-09-16T15:28:31.768 INFO:teuthology.orchestra.run.smithi184.stdout:Reading package lists... 2024-09-16T15:28:31.785 DEBUG:teuthology.orchestra.run.smithi184:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-09-16T15:28:31.848 INFO:teuthology.orchestra.run.smithi184.stdout:Reading package lists... 2024-09-16T15:28:32.068 INFO:teuthology.orchestra.run.smithi184.stdout:Building dependency tree... 2024-09-16T15:28:32.068 INFO:teuthology.orchestra.run.smithi184.stdout:Reading state information... 2024-09-16T15:28:32.244 INFO:teuthology.orchestra.run.smithi184.stdout:The following packages were automatically installed and are no longer required: 2024-09-16T15:28:32.244 INFO:teuthology.orchestra.run.smithi184.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-09-16T15:28:32.245 INFO:teuthology.orchestra.run.smithi184.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-09-16T15:28:32.245 INFO:teuthology.orchestra.run.smithi184.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-09-16T15:28:32.246 INFO:teuthology.orchestra.run.smithi184.stdout: python2 python2-dev python2-minimal python2.7 python2.7-dev 2024-09-16T15:28:32.246 INFO:teuthology.orchestra.run.smithi184.stdout: python2.7-minimal 2024-09-16T15:28:32.246 INFO:teuthology.orchestra.run.smithi184.stdout:Use 'sudo apt autoremove' to remove them. 2024-09-16T15:28:32.246 INFO:teuthology.orchestra.run.smithi184.stdout:The following additional packages will be installed: 2024-09-16T15:28:32.246 INFO:teuthology.orchestra.run.smithi184.stdout: linux-generic linux-headers-5.15.0-121 linux-headers-5.15.0-121-generic 2024-09-16T15:28:32.247 INFO:teuthology.orchestra.run.smithi184.stdout: linux-headers-generic linux-image-5.15.0-121-generic 2024-09-16T15:28:32.247 INFO:teuthology.orchestra.run.smithi184.stdout: linux-modules-5.15.0-121-generic linux-modules-extra-5.15.0-121-generic 2024-09-16T15:28:32.248 INFO:teuthology.orchestra.run.smithi184.stdout:Suggested packages: 2024-09-16T15:28:32.249 INFO:teuthology.orchestra.run.smithi184.stdout: fdutils linux-doc | linux-source-5.15.0 linux-tools 2024-09-16T15:28:32.249 INFO:teuthology.orchestra.run.smithi184.stdout:Recommended packages: 2024-09-16T15:28:32.249 INFO:teuthology.orchestra.run.smithi184.stdout: thermald 2024-09-16T15:28:32.299 INFO:teuthology.orchestra.run.smithi184.stdout:The following NEW packages will be installed: 2024-09-16T15:28:32.299 INFO:teuthology.orchestra.run.smithi184.stdout: linux-headers-5.15.0-121 linux-headers-5.15.0-121-generic 2024-09-16T15:28:32.299 INFO:teuthology.orchestra.run.smithi184.stdout: linux-image-5.15.0-121-generic linux-modules-5.15.0-121-generic 2024-09-16T15:28:32.300 INFO:teuthology.orchestra.run.smithi184.stdout: linux-modules-extra-5.15.0-121-generic 2024-09-16T15:28:32.301 INFO:teuthology.orchestra.run.smithi184.stdout:The following packages will be upgraded: 2024-09-16T15:28:32.302 INFO:teuthology.orchestra.run.smithi184.stdout: linux-generic linux-headers-generic linux-image-generic 2024-09-16T15:28:32.352 INFO:teuthology.orchestra.run.smithi184.stdout:3 upgraded, 5 newly installed, 0 to remove and 344 not upgraded. 2024-09-16T15:28:32.352 INFO:teuthology.orchestra.run.smithi184.stdout:Need to get 113 MB of archives. 2024-09-16T15:28:32.352 INFO:teuthology.orchestra.run.smithi184.stdout:After this operation, 584 MB of additional disk space will be used. 2024-09-16T15:28:32.352 INFO:teuthology.orchestra.run.smithi184.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-5.15.0-121-generic amd64 5.15.0-121.131 [22.7 MB] 2024-09-16T15:28:32.426 INFO:teuthology.orchestra.run.smithi123.stdout:Reading package lists... 2024-09-16T15:28:32.445 DEBUG:teuthology.orchestra.run.smithi123:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-09-16T15:28:32.508 INFO:teuthology.orchestra.run.smithi123.stdout:Reading package lists... 2024-09-16T15:28:32.719 INFO:teuthology.orchestra.run.smithi123.stdout:Building dependency tree... 2024-09-16T15:28:32.719 INFO:teuthology.orchestra.run.smithi123.stdout:Reading state information... 2024-09-16T15:28:32.782 INFO:teuthology.orchestra.run.smithi184.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-5.15.0-121-generic amd64 5.15.0-121.131 [11.5 MB] 2024-09-16T15:28:32.883 INFO:teuthology.orchestra.run.smithi184.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-extra-5.15.0-121-generic amd64 5.15.0-121.131 [63.9 MB] 2024-09-16T15:28:32.893 INFO:teuthology.orchestra.run.smithi123.stdout:The following packages were automatically installed and are no longer required: 2024-09-16T15:28:32.894 INFO:teuthology.orchestra.run.smithi123.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-09-16T15:28:32.894 INFO:teuthology.orchestra.run.smithi123.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-09-16T15:28:32.894 INFO:teuthology.orchestra.run.smithi123.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-09-16T15:28:32.895 INFO:teuthology.orchestra.run.smithi123.stdout: python2 python2-dev python2-minimal python2.7 python2.7-dev 2024-09-16T15:28:32.895 INFO:teuthology.orchestra.run.smithi123.stdout: python2.7-minimal 2024-09-16T15:28:32.895 INFO:teuthology.orchestra.run.smithi123.stdout:Use 'sudo apt autoremove' to remove them. 2024-09-16T15:28:32.895 INFO:teuthology.orchestra.run.smithi123.stdout:The following additional packages will be installed: 2024-09-16T15:28:32.896 INFO:teuthology.orchestra.run.smithi123.stdout: linux-generic linux-headers-5.15.0-121 linux-headers-5.15.0-121-generic 2024-09-16T15:28:32.896 INFO:teuthology.orchestra.run.smithi123.stdout: linux-headers-generic linux-image-5.15.0-121-generic 2024-09-16T15:28:32.896 INFO:teuthology.orchestra.run.smithi123.stdout: linux-modules-5.15.0-121-generic linux-modules-extra-5.15.0-121-generic 2024-09-16T15:28:32.897 INFO:teuthology.orchestra.run.smithi123.stdout:Suggested packages: 2024-09-16T15:28:32.898 INFO:teuthology.orchestra.run.smithi123.stdout: fdutils linux-doc | linux-source-5.15.0 linux-tools 2024-09-16T15:28:32.898 INFO:teuthology.orchestra.run.smithi123.stdout:Recommended packages: 2024-09-16T15:28:32.898 INFO:teuthology.orchestra.run.smithi123.stdout: thermald 2024-09-16T15:28:32.952 INFO:teuthology.orchestra.run.smithi123.stdout:The following NEW packages will be installed: 2024-09-16T15:28:32.952 INFO:teuthology.orchestra.run.smithi123.stdout: linux-headers-5.15.0-121 linux-headers-5.15.0-121-generic 2024-09-16T15:28:32.952 INFO:teuthology.orchestra.run.smithi123.stdout: linux-image-5.15.0-121-generic linux-modules-5.15.0-121-generic 2024-09-16T15:28:32.952 INFO:teuthology.orchestra.run.smithi123.stdout: linux-modules-extra-5.15.0-121-generic 2024-09-16T15:28:32.953 INFO:teuthology.orchestra.run.smithi123.stdout:The following packages will be upgraded: 2024-09-16T15:28:32.954 INFO:teuthology.orchestra.run.smithi123.stdout: linux-generic linux-headers-generic linux-image-generic 2024-09-16T15:28:33.134 INFO:teuthology.orchestra.run.smithi123.stdout:3 upgraded, 5 newly installed, 0 to remove and 344 not upgraded. 2024-09-16T15:28:33.134 INFO:teuthology.orchestra.run.smithi123.stdout:Need to get 113 MB of archives. 2024-09-16T15:28:33.134 INFO:teuthology.orchestra.run.smithi123.stdout:After this operation, 584 MB of additional disk space will be used. 2024-09-16T15:28:33.134 INFO:teuthology.orchestra.run.smithi123.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-5.15.0-121-generic amd64 5.15.0-121.131 [22.7 MB] 2024-09-16T15:28:33.622 INFO:teuthology.orchestra.run.smithi184.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-generic amd64 5.15.0.121.121 [1,694 B] 2024-09-16T15:28:33.623 INFO:teuthology.orchestra.run.smithi184.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-generic amd64 5.15.0.121.121 [2,464 B] 2024-09-16T15:28:33.623 INFO:teuthology.orchestra.run.smithi184.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-121 all 5.15.0-121.131 [12.3 MB] 2024-09-16T15:28:33.739 INFO:teuthology.orchestra.run.smithi184.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-121-generic amd64 5.15.0-121.131 [2,827 kB] 2024-09-16T15:28:33.847 INFO:teuthology.orchestra.run.smithi184.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-generic amd64 5.15.0.121.121 [2,316 B] 2024-09-16T15:28:34.258 INFO:teuthology.orchestra.run.smithi184.stdout:Fetched 113 MB in 2s (73.7 MB/s) 2024-09-16T15:28:34.439 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package linux-modules-5.15.0-121-generic. 2024-09-16T15:28:34.573 INFO:teuthology.orchestra.run.smithi123.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-5.15.0-121-generic amd64 5.15.0-121.131 [11.5 MB] 2024-09-16T15:28:34.966 INFO:teuthology.orchestra.run.smithi123.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-extra-5.15.0-121-generic amd64 5.15.0-121.131 [63.9 MB] 2024-09-16T15:28:36.480 INFO:teuthology.orchestra.run.smithi184.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 129542 files and directories currently installed.) 2024-09-16T15:28:36.484 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../0-linux-modules-5.15.0-121-generic_5.15.0-121.131_amd64.deb ... 2024-09-16T15:28:36.679 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking linux-modules-5.15.0-121-generic (5.15.0-121.131) ... 2024-09-16T15:28:37.130 INFO:teuthology.orchestra.run.smithi123.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-generic amd64 5.15.0.121.121 [1,694 B] 2024-09-16T15:28:37.130 INFO:teuthology.orchestra.run.smithi123.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-generic amd64 5.15.0.121.121 [2,464 B] 2024-09-16T15:28:37.130 INFO:teuthology.orchestra.run.smithi123.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-121 all 5.15.0-121.131 [12.3 MB] 2024-09-16T15:28:37.474 INFO:teuthology.orchestra.run.smithi123.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-121-generic amd64 5.15.0-121.131 [2,827 kB] 2024-09-16T15:28:37.553 INFO:teuthology.orchestra.run.smithi123.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-generic amd64 5.15.0.121.121 [2,316 B] 2024-09-16T15:28:37.967 INFO:teuthology.orchestra.run.smithi123.stdout:Fetched 113 MB in 5s (24.7 MB/s) 2024-09-16T15:28:38.157 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package linux-modules-5.15.0-121-generic. 2024-09-16T15:28:38.822 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package linux-image-5.15.0-121-generic. 2024-09-16T15:28:38.843 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../1-linux-image-5.15.0-121-generic_5.15.0-121.131_amd64.deb ... 2024-09-16T15:28:38.937 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking linux-image-5.15.0-121-generic (5.15.0-121.131) ... 2024-09-16T15:28:39.409 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package linux-modules-extra-5.15.0-121-generic. 2024-09-16T15:28:39.429 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../2-linux-modules-extra-5.15.0-121-generic_5.15.0-121.131_amd64.deb ... 2024-09-16T15:28:39.578 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking linux-modules-extra-5.15.0-121-generic (5.15.0-121.131) ... 2024-09-16T15:28:40.151 INFO:teuthology.orchestra.run.smithi123.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 129542 files and directories currently installed.) 2024-09-16T15:28:40.155 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../0-linux-modules-5.15.0-121-generic_5.15.0-121.131_amd64.deb ... 2024-09-16T15:28:40.255 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking linux-modules-5.15.0-121-generic (5.15.0-121.131) ... 2024-09-16T15:28:42.741 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package linux-image-5.15.0-121-generic. 2024-09-16T15:28:42.762 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../1-linux-image-5.15.0-121-generic_5.15.0-121.131_amd64.deb ... 2024-09-16T15:28:42.840 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking linux-image-5.15.0-121-generic (5.15.0-121.131) ... 2024-09-16T15:28:43.319 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package linux-modules-extra-5.15.0-121-generic. 2024-09-16T15:28:43.339 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../2-linux-modules-extra-5.15.0-121-generic_5.15.0-121.131_amd64.deb ... 2024-09-16T15:28:43.377 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking linux-modules-extra-5.15.0-121-generic (5.15.0-121.131) ... 2024-09-16T15:28:46.443 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../3-linux-generic_5.15.0.121.121_amd64.deb ... 2024-09-16T15:28:46.587 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking linux-generic (5.15.0.121.121) over (5.15.0.56.54) ... 2024-09-16T15:28:46.903 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../4-linux-image-generic_5.15.0.121.121_amd64.deb ... 2024-09-16T15:28:47.024 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking linux-image-generic (5.15.0.121.121) over (5.15.0.56.54) ... 2024-09-16T15:28:47.318 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package linux-headers-5.15.0-121. 2024-09-16T15:28:47.329 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../5-linux-headers-5.15.0-121_5.15.0-121.131_all.deb ... 2024-09-16T15:28:47.368 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking linux-headers-5.15.0-121 (5.15.0-121.131) ... 2024-09-16T15:28:49.810 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../3-linux-generic_5.15.0.121.121_amd64.deb ... 2024-09-16T15:28:49.926 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking linux-generic (5.15.0.121.121) over (5.15.0.56.54) ... 2024-09-16T15:28:50.284 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../4-linux-image-generic_5.15.0.121.121_amd64.deb ... 2024-09-16T15:28:50.413 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking linux-image-generic (5.15.0.121.121) over (5.15.0.56.54) ... 2024-09-16T15:28:50.741 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package linux-headers-5.15.0-121. 2024-09-16T15:28:50.763 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../5-linux-headers-5.15.0-121_5.15.0-121.131_all.deb ... 2024-09-16T15:28:50.799 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking linux-headers-5.15.0-121 (5.15.0-121.131) ... 2024-09-16T15:28:52.785 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package linux-headers-5.15.0-121-generic. 2024-09-16T15:28:52.823 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../6-linux-headers-5.15.0-121-generic_5.15.0-121.131_amd64.deb ... 2024-09-16T15:28:52.876 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking linux-headers-5.15.0-121-generic (5.15.0-121.131) ... 2024-09-16T15:28:55.947 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../7-linux-headers-generic_5.15.0.121.121_amd64.deb ... 2024-09-16T15:28:56.069 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking linux-headers-generic (5.15.0.121.121) over (5.15.0.56.54) ... 2024-09-16T15:28:56.411 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up linux-headers-5.15.0-121 (5.15.0-121.131) ... 2024-09-16T15:28:56.524 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up linux-headers-5.15.0-121-generic (5.15.0-121.131) ... 2024-09-16T15:28:56.650 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up linux-headers-generic (5.15.0.121.121) ... 2024-09-16T15:28:56.718 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package linux-headers-5.15.0-121-generic. 2024-09-16T15:28:56.737 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../6-linux-headers-5.15.0-121-generic_5.15.0-121.131_amd64.deb ... 2024-09-16T15:28:56.768 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking linux-headers-5.15.0-121-generic (5.15.0-121.131) ... 2024-09-16T15:28:56.777 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up linux-image-5.15.0-121-generic (5.15.0-121.131) ... 2024-09-16T15:28:58.075 INFO:teuthology.orchestra.run.smithi184.stdout:I: /vmlinuz.old is now a symlink to boot/vmlinuz-5.15.0-56-generic 2024-09-16T15:28:58.075 INFO:teuthology.orchestra.run.smithi184.stdout:I: /initrd.img.old is now a symlink to boot/initrd.img-5.15.0-56-generic 2024-09-16T15:28:58.075 INFO:teuthology.orchestra.run.smithi184.stdout:I: /vmlinuz is now a symlink to boot/vmlinuz-5.15.0-121-generic 2024-09-16T15:28:58.075 INFO:teuthology.orchestra.run.smithi184.stdout:I: /initrd.img is now a symlink to boot/initrd.img-5.15.0-121-generic 2024-09-16T15:28:58.222 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up linux-modules-5.15.0-121-generic (5.15.0-121.131) ... 2024-09-16T15:29:00.206 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up linux-modules-extra-5.15.0-121-generic (5.15.0-121.131) ... 2024-09-16T15:29:00.504 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../7-linux-headers-generic_5.15.0.121.121_amd64.deb ... 2024-09-16T15:29:00.628 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking linux-headers-generic (5.15.0.121.121) over (5.15.0.56.54) ... 2024-09-16T15:29:00.995 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up linux-headers-5.15.0-121 (5.15.0-121.131) ... 2024-09-16T15:29:01.116 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up linux-headers-5.15.0-121-generic (5.15.0-121.131) ... 2024-09-16T15:29:01.259 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up linux-headers-generic (5.15.0.121.121) ... 2024-09-16T15:29:01.387 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up linux-image-5.15.0-121-generic (5.15.0-121.131) ... 2024-09-16T15:29:01.599 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up linux-image-generic (5.15.0.121.121) ... 2024-09-16T15:29:01.725 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up linux-generic (5.15.0.121.121) ... 2024-09-16T15:29:01.851 INFO:teuthology.orchestra.run.smithi184.stdout:Processing triggers for linux-image-5.15.0-121-generic (5.15.0-121.131) ... 2024-09-16T15:29:01.906 INFO:teuthology.orchestra.run.smithi184.stdout:/etc/kernel/postinst.d/initramfs-tools: 2024-09-16T15:29:01.906 INFO:teuthology.orchestra.run.smithi184.stdout:update-initramfs: Generating /boot/initrd.img-5.15.0-121-generic 2024-09-16T15:29:02.748 INFO:teuthology.orchestra.run.smithi123.stdout:I: /vmlinuz.old is now a symlink to boot/vmlinuz-5.15.0-56-generic 2024-09-16T15:29:02.748 INFO:teuthology.orchestra.run.smithi123.stdout:I: /initrd.img.old is now a symlink to boot/initrd.img-5.15.0-56-generic 2024-09-16T15:29:02.748 INFO:teuthology.orchestra.run.smithi123.stdout:I: /vmlinuz is now a symlink to boot/vmlinuz-5.15.0-121-generic 2024-09-16T15:29:02.748 INFO:teuthology.orchestra.run.smithi123.stdout:I: /initrd.img is now a symlink to boot/initrd.img-5.15.0-121-generic 2024-09-16T15:29:02.923 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up linux-modules-5.15.0-121-generic (5.15.0-121.131) ... 2024-09-16T15:29:04.332 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up linux-modules-extra-5.15.0-121-generic (5.15.0-121.131) ... 2024-09-16T15:29:06.125 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up linux-image-generic (5.15.0.121.121) ... 2024-09-16T15:29:06.276 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up linux-generic (5.15.0.121.121) ... 2024-09-16T15:29:06.402 INFO:teuthology.orchestra.run.smithi123.stdout:Processing triggers for linux-image-5.15.0-121-generic (5.15.0-121.131) ... 2024-09-16T15:29:06.459 INFO:teuthology.orchestra.run.smithi123.stdout:/etc/kernel/postinst.d/initramfs-tools: 2024-09-16T15:29:06.459 INFO:teuthology.orchestra.run.smithi123.stdout:update-initramfs: Generating /boot/initrd.img-5.15.0-121-generic 2024-09-16T15:29:16.260 INFO:teuthology.orchestra.run.smithi184.stdout:/etc/kernel/postinst.d/zz-update-grub: 2024-09-16T15:29:16.260 INFO:teuthology.orchestra.run.smithi184.stdout:Sourcing file `/etc/default/grub' 2024-09-16T15:29:16.280 INFO:teuthology.orchestra.run.smithi184.stdout:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-09-16T15:29:16.295 INFO:teuthology.orchestra.run.smithi184.stdout:Generating grub configuration file ... 2024-09-16T15:29:16.562 INFO:teuthology.orchestra.run.smithi184.stdout:Found linux image: /boot/vmlinuz-5.15.0-121-generic 2024-09-16T15:29:16.575 INFO:teuthology.orchestra.run.smithi184.stdout:Found initrd image: /boot/initrd.img-5.15.0-121-generic 2024-09-16T15:29:16.861 INFO:teuthology.orchestra.run.smithi184.stdout:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-09-16T15:29:16.867 INFO:teuthology.orchestra.run.smithi184.stdout:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-09-16T15:29:16.931 INFO:teuthology.orchestra.run.smithi184.stdout:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-09-16T15:29:16.937 INFO:teuthology.orchestra.run.smithi184.stdout:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-09-16T15:29:17.083 INFO:teuthology.orchestra.run.smithi184.stdout:Warning: os-prober will not be executed to detect other bootable partitions. 2024-09-16T15:29:17.083 INFO:teuthology.orchestra.run.smithi184.stdout:Systems on them will not be added to the GRUB boot configuration. 2024-09-16T15:29:17.083 INFO:teuthology.orchestra.run.smithi184.stdout:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-09-16T15:29:17.121 INFO:teuthology.orchestra.run.smithi184.stdout:done 2024-09-16T15:29:17.581 DEBUG:teuthology.orchestra.run.smithi184:> dpkg -s linux-image-generic 2024-09-16T15:29:17.608 INFO:teuthology.orchestra.run.smithi184.stdout:Package: linux-image-generic 2024-09-16T15:29:17.608 INFO:teuthology.orchestra.run.smithi184.stdout:Status: install ok installed 2024-09-16T15:29:17.608 INFO:teuthology.orchestra.run.smithi184.stdout:Priority: optional 2024-09-16T15:29:17.608 INFO:teuthology.orchestra.run.smithi184.stdout:Section: kernel 2024-09-16T15:29:17.608 INFO:teuthology.orchestra.run.smithi184.stdout:Installed-Size: 21 2024-09-16T15:29:17.608 INFO:teuthology.orchestra.run.smithi184.stdout:Maintainer: Ubuntu Kernel Team 2024-09-16T15:29:17.608 INFO:teuthology.orchestra.run.smithi184.stdout:Architecture: amd64 2024-09-16T15:29:17.608 INFO:teuthology.orchestra.run.smithi184.stdout:Source: linux-meta 2024-09-16T15:29:17.608 INFO:teuthology.orchestra.run.smithi184.stdout:Version: 5.15.0.121.121 2024-09-16T15:29:17.608 INFO:teuthology.orchestra.run.smithi184.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.4), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-121), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.4) 2024-09-16T15:29:17.609 INFO:teuthology.orchestra.run.smithi184.stdout:Depends: linux-image-5.15.0-121-generic, linux-modules-extra-5.15.0-121-generic, linux-firmware, intel-microcode, amd64-microcode 2024-09-16T15:29:17.609 INFO:teuthology.orchestra.run.smithi184.stdout:Recommends: thermald 2024-09-16T15:29:17.609 INFO:teuthology.orchestra.run.smithi184.stdout:Description: Generic Linux kernel image 2024-09-16T15:29:17.609 INFO:teuthology.orchestra.run.smithi184.stdout: This package will always depend on the latest generic kernel image 2024-09-16T15:29:17.609 INFO:teuthology.orchestra.run.smithi184.stdout: available. 2024-09-16T15:29:17.613 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.15.0-56-generic Expected: 5.15.0-121-generic 2024-09-16T15:29:17.613 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-09-16T15:29:17.613 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-09-16T15:29:17.613 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2024-09-16T15:29:17.613 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi184.front.sepia.ceph.com, path=None, version=distro) 2024-09-16T15:29:17.613 DEBUG:teuthology.orchestra.run.smithi184:> sudo apt-get clean 2024-09-16T15:29:17.802 DEBUG:teuthology.orchestra.run.smithi184:> sudo apt-get update 2024-09-16T15:29:17.934 INFO:teuthology.orchestra.run.smithi184.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-09-16T15:29:17.943 INFO:teuthology.orchestra.run.smithi184.stdout:Hit:2 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2024-09-16T15:29:17.961 INFO:teuthology.orchestra.run.smithi184.stdout:Hit:3 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2024-09-16T15:29:18.054 INFO:teuthology.orchestra.run.smithi184.stdout:Hit:4 http://security.ubuntu.com/ubuntu jammy-security InRelease 2024-09-16T15:29:19.339 INFO:teuthology.orchestra.run.smithi184.stdout:Reading package lists... 2024-09-16T15:29:19.357 DEBUG:teuthology.orchestra.run.smithi184:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-09-16T15:29:19.419 INFO:teuthology.orchestra.run.smithi184.stdout:Reading package lists... 2024-09-16T15:29:19.629 INFO:teuthology.orchestra.run.smithi184.stdout:Building dependency tree... 2024-09-16T15:29:19.630 INFO:teuthology.orchestra.run.smithi184.stdout:Reading state information... 2024-09-16T15:29:19.809 INFO:teuthology.orchestra.run.smithi184.stdout:linux-image-generic is already the newest version (5.15.0.121.121). 2024-09-16T15:29:19.809 INFO:teuthology.orchestra.run.smithi184.stdout:The following packages were automatically installed and are no longer required: 2024-09-16T15:29:19.809 INFO:teuthology.orchestra.run.smithi184.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-09-16T15:29:19.809 INFO:teuthology.orchestra.run.smithi184.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-09-16T15:29:19.810 INFO:teuthology.orchestra.run.smithi184.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-09-16T15:29:19.810 INFO:teuthology.orchestra.run.smithi184.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-09-16T15:29:19.810 INFO:teuthology.orchestra.run.smithi184.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-09-16T15:29:19.810 INFO:teuthology.orchestra.run.smithi184.stdout:Use 'sudo apt autoremove' to remove them. 2024-09-16T15:29:19.850 INFO:teuthology.orchestra.run.smithi184.stdout:0 upgraded, 0 newly installed, 0 to remove and 344 not upgraded. 2024-09-16T15:29:19.852 DEBUG:teuthology.orchestra.run.smithi184:> dpkg -s linux-image-generic 2024-09-16T15:29:19.867 INFO:teuthology.orchestra.run.smithi184.stdout:Package: linux-image-generic 2024-09-16T15:29:19.867 INFO:teuthology.orchestra.run.smithi184.stdout:Status: install ok installed 2024-09-16T15:29:19.867 INFO:teuthology.orchestra.run.smithi184.stdout:Priority: optional 2024-09-16T15:29:19.868 INFO:teuthology.orchestra.run.smithi184.stdout:Section: kernel 2024-09-16T15:29:19.868 INFO:teuthology.orchestra.run.smithi184.stdout:Installed-Size: 21 2024-09-16T15:29:19.868 INFO:teuthology.orchestra.run.smithi184.stdout:Maintainer: Ubuntu Kernel Team 2024-09-16T15:29:19.868 INFO:teuthology.orchestra.run.smithi184.stdout:Architecture: amd64 2024-09-16T15:29:19.868 INFO:teuthology.orchestra.run.smithi184.stdout:Source: linux-meta 2024-09-16T15:29:19.868 INFO:teuthology.orchestra.run.smithi184.stdout:Version: 5.15.0.121.121 2024-09-16T15:29:19.868 INFO:teuthology.orchestra.run.smithi184.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.4), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-121), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.4) 2024-09-16T15:29:19.868 INFO:teuthology.orchestra.run.smithi184.stdout:Depends: linux-image-5.15.0-121-generic, linux-modules-extra-5.15.0-121-generic, linux-firmware, intel-microcode, amd64-microcode 2024-09-16T15:29:19.868 INFO:teuthology.orchestra.run.smithi184.stdout:Recommends: thermald 2024-09-16T15:29:19.868 INFO:teuthology.orchestra.run.smithi184.stdout:Description: Generic Linux kernel image 2024-09-16T15:29:19.868 INFO:teuthology.orchestra.run.smithi184.stdout: This package will always depend on the latest generic kernel image 2024-09-16T15:29:19.868 INFO:teuthology.orchestra.run.smithi184.stdout: available. 2024-09-16T15:29:19.869 DEBUG:teuthology.orchestra.run.smithi184:> mktemp 2024-09-16T15:29:19.912 INFO:teuthology.orchestra.run.smithi184.stdout:/tmp/tmp.Fddoud8Nxr 2024-09-16T15:29:19.912 DEBUG:teuthology.orchestra.run.smithi184:> sudo cp /boot/grub/grub.cfg /tmp/tmp.Fddoud8Nxr 2024-09-16T15:29:19.983 DEBUG:teuthology.orchestra.run.smithi184:> sudo chmod 0666 /tmp/tmp.Fddoud8Nxr 2024-09-16T15:29:20.105 DEBUG:teuthology.orchestra.remote:smithi184:/tmp/tmp.Fddoud8Nxr is 10KB 2024-09-16T15:29:20.117 DEBUG:teuthology.orchestra.run.smithi184:> rm -fr /tmp/tmp.Fddoud8Nxr 2024-09-16T15:29:20.122 DEBUG:teuthology.orchestra.run.smithi184:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2024-09-16T15:29:20.173 DEBUG:teuthology.orchestra.run.smithi184:> set -ex 2024-09-16T15:29:20.173 DEBUG:teuthology.orchestra.run.smithi184:> sudo dd of=/etc/grub.d/01_ceph_kernel 2024-09-16T15:29:20.173 DEBUG:teuthology.orchestra.run.smithi184:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2024-09-16T15:29:20.249 INFO:teuthology.task.kernel:Distro Kernel Version: 5.15.0-121-generic 2024-09-16T15:29:20.249 DEBUG:teuthology.orchestra.run.smithi184:> sudo update-grub 2024-09-16T15:29:20.441 INFO:teuthology.orchestra.run.smithi123.stdout:/etc/kernel/postinst.d/zz-update-grub: 2024-09-16T15:29:20.441 INFO:teuthology.orchestra.run.smithi123.stdout:Sourcing file `/etc/default/grub' 2024-09-16T15:29:20.464 INFO:teuthology.orchestra.run.smithi123.stdout:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-09-16T15:29:20.488 INFO:teuthology.orchestra.run.smithi123.stdout:Generating grub configuration file ... 2024-09-16T15:29:20.820 INFO:teuthology.orchestra.run.smithi123.stdout:Found linux image: /boot/vmlinuz-5.15.0-121-generic 2024-09-16T15:29:20.844 INFO:teuthology.orchestra.run.smithi123.stdout:Found initrd image: /boot/initrd.img-5.15.0-121-generic 2024-09-16T15:29:21.134 INFO:teuthology.orchestra.run.smithi123.stdout:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-09-16T15:29:21.141 INFO:teuthology.orchestra.run.smithi123.stdout:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-09-16T15:29:21.216 INFO:teuthology.orchestra.run.smithi123.stdout:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-09-16T15:29:21.222 INFO:teuthology.orchestra.run.smithi123.stdout:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-09-16T15:29:21.378 INFO:teuthology.orchestra.run.smithi123.stdout:Warning: os-prober will not be executed to detect other bootable partitions. 2024-09-16T15:29:21.378 INFO:teuthology.orchestra.run.smithi123.stdout:Systems on them will not be added to the GRUB boot configuration. 2024-09-16T15:29:21.378 INFO:teuthology.orchestra.run.smithi123.stdout:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-09-16T15:29:21.416 INFO:teuthology.orchestra.run.smithi123.stdout:done 2024-09-16T15:29:21.419 INFO:teuthology.orchestra.run.smithi184.stderr:Sourcing file `/etc/default/grub' 2024-09-16T15:29:21.420 INFO:teuthology.orchestra.run.smithi184.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-09-16T15:29:21.424 INFO:teuthology.orchestra.run.smithi184.stderr:Generating grub configuration file ... 2024-09-16T15:29:21.598 INFO:teuthology.orchestra.run.smithi184.stderr:Found linux image: /boot/vmlinuz-5.15.0-121-generic 2024-09-16T15:29:21.603 INFO:teuthology.orchestra.run.smithi184.stderr:Found initrd image: /boot/initrd.img-5.15.0-121-generic 2024-09-16T15:29:21.869 INFO:teuthology.orchestra.run.smithi184.stderr:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-09-16T15:29:21.874 INFO:teuthology.orchestra.run.smithi184.stderr:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-09-16T15:29:21.887 DEBUG:teuthology.orchestra.run.smithi123:> dpkg -s linux-image-generic 2024-09-16T15:29:21.913 INFO:teuthology.orchestra.run.smithi123.stdout:Package: linux-image-generic 2024-09-16T15:29:21.913 INFO:teuthology.orchestra.run.smithi123.stdout:Status: install ok installed 2024-09-16T15:29:21.913 INFO:teuthology.orchestra.run.smithi123.stdout:Priority: optional 2024-09-16T15:29:21.913 INFO:teuthology.orchestra.run.smithi123.stdout:Section: kernel 2024-09-16T15:29:21.913 INFO:teuthology.orchestra.run.smithi123.stdout:Installed-Size: 21 2024-09-16T15:29:21.913 INFO:teuthology.orchestra.run.smithi123.stdout:Maintainer: Ubuntu Kernel Team 2024-09-16T15:29:21.913 INFO:teuthology.orchestra.run.smithi123.stdout:Architecture: amd64 2024-09-16T15:29:21.913 INFO:teuthology.orchestra.run.smithi123.stdout:Source: linux-meta 2024-09-16T15:29:21.913 INFO:teuthology.orchestra.run.smithi123.stdout:Version: 5.15.0.121.121 2024-09-16T15:29:21.913 INFO:teuthology.orchestra.run.smithi123.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.4), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-121), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.4) 2024-09-16T15:29:21.913 INFO:teuthology.orchestra.run.smithi123.stdout:Depends: linux-image-5.15.0-121-generic, linux-modules-extra-5.15.0-121-generic, linux-firmware, intel-microcode, amd64-microcode 2024-09-16T15:29:21.914 INFO:teuthology.orchestra.run.smithi123.stdout:Recommends: thermald 2024-09-16T15:29:21.914 INFO:teuthology.orchestra.run.smithi123.stdout:Description: Generic Linux kernel image 2024-09-16T15:29:21.914 INFO:teuthology.orchestra.run.smithi123.stdout: This package will always depend on the latest generic kernel image 2024-09-16T15:29:21.914 INFO:teuthology.orchestra.run.smithi123.stdout: available. 2024-09-16T15:29:21.914 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.15.0-56-generic Expected: 5.15.0-121-generic 2024-09-16T15:29:21.915 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-09-16T15:29:21.915 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-09-16T15:29:21.915 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-09-16T15:29:21.915 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi123.front.sepia.ceph.com, path=None, version=distro) 2024-09-16T15:29:21.915 DEBUG:teuthology.orchestra.run.smithi123:> sudo apt-get clean 2024-09-16T15:29:21.919 INFO:teuthology.orchestra.run.smithi184.stderr:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-09-16T15:29:21.924 INFO:teuthology.orchestra.run.smithi184.stderr:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-09-16T15:29:22.062 INFO:teuthology.orchestra.run.smithi184.stderr:Warning: os-prober will not be executed to detect other bootable partitions. 2024-09-16T15:29:22.062 INFO:teuthology.orchestra.run.smithi184.stderr:Systems on them will not be added to the GRUB boot configuration. 2024-09-16T15:29:22.063 INFO:teuthology.orchestra.run.smithi184.stderr:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-09-16T15:29:22.074 INFO:teuthology.orchestra.run.smithi184.stderr:done 2024-09-16T15:29:22.091 DEBUG:teuthology.orchestra.run.smithi184:> sudo shutdown -r now 2024-09-16T15:29:22.112 DEBUG:teuthology.orchestra.run.smithi123:> sudo apt-get update 2024-09-16T15:29:22.233 INFO:teuthology.orchestra.run.smithi123.stdout:Hit:1 http://security.ubuntu.com/ubuntu jammy-security InRelease 2024-09-16T15:29:22.354 INFO:teuthology.orchestra.run.smithi123.stdout:Hit:2 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-09-16T15:29:22.435 INFO:teuthology.orchestra.run.smithi123.stdout:Hit:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2024-09-16T15:29:22.516 INFO:teuthology.orchestra.run.smithi123.stdout:Hit:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2024-09-16T15:29:23.631 INFO:teuthology.orchestra.run.smithi123.stdout:Reading package lists... 2024-09-16T15:29:23.648 DEBUG:teuthology.orchestra.run.smithi123:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-09-16T15:29:23.701 INFO:teuthology.orchestra.run.smithi123.stdout:Reading package lists... 2024-09-16T15:29:23.888 INFO:teuthology.orchestra.run.smithi123.stdout:Building dependency tree... 2024-09-16T15:29:23.889 INFO:teuthology.orchestra.run.smithi123.stdout:Reading state information... 2024-09-16T15:29:24.055 INFO:teuthology.orchestra.run.smithi123.stdout:linux-image-generic is already the newest version (5.15.0.121.121). 2024-09-16T15:29:24.055 INFO:teuthology.orchestra.run.smithi123.stdout:The following packages were automatically installed and are no longer required: 2024-09-16T15:29:24.055 INFO:teuthology.orchestra.run.smithi123.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-09-16T15:29:24.055 INFO:teuthology.orchestra.run.smithi123.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-09-16T15:29:24.055 INFO:teuthology.orchestra.run.smithi123.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-09-16T15:29:24.056 INFO:teuthology.orchestra.run.smithi123.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-09-16T15:29:24.056 INFO:teuthology.orchestra.run.smithi123.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-09-16T15:29:24.056 INFO:teuthology.orchestra.run.smithi123.stdout:Use 'sudo apt autoremove' to remove them. 2024-09-16T15:29:24.098 INFO:teuthology.orchestra.run.smithi123.stdout:0 upgraded, 0 newly installed, 0 to remove and 344 not upgraded. 2024-09-16T15:29:24.100 DEBUG:teuthology.orchestra.run.smithi123:> dpkg -s linux-image-generic 2024-09-16T15:29:24.152 INFO:teuthology.orchestra.run.smithi123.stdout:Package: linux-image-generic 2024-09-16T15:29:24.152 INFO:teuthology.orchestra.run.smithi123.stdout:Status: install ok installed 2024-09-16T15:29:24.152 INFO:teuthology.orchestra.run.smithi123.stdout:Priority: optional 2024-09-16T15:29:24.152 INFO:teuthology.orchestra.run.smithi123.stdout:Section: kernel 2024-09-16T15:29:24.152 INFO:teuthology.orchestra.run.smithi123.stdout:Installed-Size: 21 2024-09-16T15:29:24.153 INFO:teuthology.orchestra.run.smithi123.stdout:Maintainer: Ubuntu Kernel Team 2024-09-16T15:29:24.153 INFO:teuthology.orchestra.run.smithi123.stdout:Architecture: amd64 2024-09-16T15:29:24.153 INFO:teuthology.orchestra.run.smithi123.stdout:Source: linux-meta 2024-09-16T15:29:24.153 INFO:teuthology.orchestra.run.smithi123.stdout:Version: 5.15.0.121.121 2024-09-16T15:29:24.153 INFO:teuthology.orchestra.run.smithi123.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.4), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-121), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.4) 2024-09-16T15:29:24.153 INFO:teuthology.orchestra.run.smithi123.stdout:Depends: linux-image-5.15.0-121-generic, linux-modules-extra-5.15.0-121-generic, linux-firmware, intel-microcode, amd64-microcode 2024-09-16T15:29:24.153 INFO:teuthology.orchestra.run.smithi123.stdout:Recommends: thermald 2024-09-16T15:29:24.154 INFO:teuthology.orchestra.run.smithi123.stdout:Description: Generic Linux kernel image 2024-09-16T15:29:24.154 INFO:teuthology.orchestra.run.smithi123.stdout: This package will always depend on the latest generic kernel image 2024-09-16T15:29:24.154 INFO:teuthology.orchestra.run.smithi123.stdout: available. 2024-09-16T15:29:24.154 DEBUG:teuthology.orchestra.run.smithi123:> mktemp 2024-09-16T15:29:24.196 INFO:teuthology.orchestra.run.smithi123.stdout:/tmp/tmp.hBDX82fFAl 2024-09-16T15:29:24.196 DEBUG:teuthology.orchestra.run.smithi123:> sudo cp /boot/grub/grub.cfg /tmp/tmp.hBDX82fFAl 2024-09-16T15:29:24.255 DEBUG:teuthology.orchestra.run.smithi123:> sudo chmod 0666 /tmp/tmp.hBDX82fFAl 2024-09-16T15:29:24.384 DEBUG:teuthology.orchestra.remote:smithi123:/tmp/tmp.hBDX82fFAl is 10KB 2024-09-16T15:29:24.397 DEBUG:teuthology.orchestra.run.smithi123:> rm -fr /tmp/tmp.hBDX82fFAl 2024-09-16T15:29:24.402 DEBUG:teuthology.orchestra.run.smithi123:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2024-09-16T15:29:24.456 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2024-09-16T15:29:24.456 DEBUG:teuthology.orchestra.run.smithi123:> sudo dd of=/etc/grub.d/01_ceph_kernel 2024-09-16T15:29:24.456 DEBUG:teuthology.orchestra.run.smithi123:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2024-09-16T15:29:24.519 INFO:teuthology.task.kernel:Distro Kernel Version: 5.15.0-121-generic 2024-09-16T15:29:24.519 DEBUG:teuthology.orchestra.run.smithi123:> sudo update-grub 2024-09-16T15:29:25.792 INFO:teuthology.orchestra.run.smithi123.stderr:Sourcing file `/etc/default/grub' 2024-09-16T15:29:25.794 INFO:teuthology.orchestra.run.smithi123.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-09-16T15:29:25.795 INFO:teuthology.orchestra.run.smithi123.stderr:Generating grub configuration file ... 2024-09-16T15:29:26.006 INFO:teuthology.orchestra.run.smithi123.stderr:Found linux image: /boot/vmlinuz-5.15.0-121-generic 2024-09-16T15:29:26.012 INFO:teuthology.orchestra.run.smithi123.stderr:Found initrd image: /boot/initrd.img-5.15.0-121-generic 2024-09-16T15:29:26.250 INFO:teuthology.orchestra.run.smithi123.stderr:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-09-16T15:29:26.255 INFO:teuthology.orchestra.run.smithi123.stderr:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-09-16T15:29:26.293 INFO:teuthology.orchestra.run.smithi123.stderr:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-09-16T15:29:26.298 INFO:teuthology.orchestra.run.smithi123.stderr:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-09-16T15:29:26.436 INFO:teuthology.orchestra.run.smithi123.stderr:Warning: os-prober will not be executed to detect other bootable partitions. 2024-09-16T15:29:26.436 INFO:teuthology.orchestra.run.smithi123.stderr:Systems on them will not be added to the GRUB boot configuration. 2024-09-16T15:29:26.436 INFO:teuthology.orchestra.run.smithi123.stderr:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-09-16T15:29:26.465 INFO:teuthology.orchestra.run.smithi123.stderr:done 2024-09-16T15:29:26.474 DEBUG:teuthology.orchestra.run.smithi123:> sudo shutdown -r now 2024-09-16T15:29:52.119 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2024-09-16T15:29:52.119 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi184.front.sepia.ceph.com' 2024-09-16T15:29:52.120 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi184.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T15:29:56.481 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-09-16T15:29:56.481 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi123.front.sepia.ceph.com' 2024-09-16T15:29:56.481 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi123.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T15:30:31.101 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.123 2024-09-16T15:30:40.103 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi123.front.sepia.ceph.com' 2024-09-16T15:30:40.104 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi123.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T15:30:52.124 DEBUG:teuthology.orchestra.remote:timed out 2024-09-16T15:30:58.493 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.123 2024-09-16T15:31:01.128 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi184.front.sepia.ceph.com' 2024-09-16T15:31:01.129 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi184.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T15:31:01.131 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.184 2024-09-16T15:31:10.501 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi123.front.sepia.ceph.com' 2024-09-16T15:31:10.502 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi123.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T15:31:11.145 DEBUG:teuthology.orchestra.run.smithi123:> true 2024-09-16T15:31:11.960 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi123.front.sepia.ceph.com' 2024-09-16T15:31:11.961 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.15.0-121-generic"... 2024-09-16T15:31:11.961 DEBUG:teuthology.orchestra.run.smithi123:> uname -r 2024-09-16T15:31:12.011 INFO:teuthology.orchestra.run.smithi123.stdout:5.15.0-121-generic 2024-09-16T15:31:12.011 DEBUG:teuthology.task.kernel:current kernel version is 5.15.0-121-generic vs 5.15.0-121-generic 2024-09-16T15:31:12.011 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-09-16T15:31:12.011 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2024-09-16T15:31:13.013 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-09-16T15:31:13.013 DEBUG:teuthology.orchestra.run.smithi123:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-09-16T15:31:13.093 INFO:teuthology.orchestra.run.smithi123.stdout:ttyS1 2024-09-16T15:31:13.120 DEBUG:teuthology.parallel:result is None 2024-09-16T15:31:13.132 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi184.front.sepia.ceph.com' 2024-09-16T15:31:13.133 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi184.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T15:31:13.332 DEBUG:teuthology.orchestra.run.smithi184:> true 2024-09-16T15:31:13.934 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi184.front.sepia.ceph.com' 2024-09-16T15:31:13.934 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "5.15.0-121-generic"... 2024-09-16T15:31:13.935 DEBUG:teuthology.orchestra.run.smithi184:> uname -r 2024-09-16T15:31:13.986 INFO:teuthology.orchestra.run.smithi184.stdout:5.15.0-121-generic 2024-09-16T15:31:13.986 DEBUG:teuthology.task.kernel:current kernel version is 5.15.0-121-generic vs 5.15.0-121-generic 2024-09-16T15:31:13.986 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-09-16T15:31:13.986 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2024-09-16T15:31:14.987 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-09-16T15:31:14.987 DEBUG:teuthology.orchestra.run.smithi184:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-09-16T15:31:15.315 INFO:teuthology.orchestra.run.smithi184.stdout:ttyS1 2024-09-16T15:31:15.352 DEBUG:teuthology.parallel:result is None 2024-09-16T15:31:15.352 INFO:teuthology.run_tasks:Running task internal.base... 2024-09-16T15:31:15.360 INFO:teuthology.task.internal:Creating test directory... 2024-09-16T15:31:15.360 DEBUG:teuthology.orchestra.run.smithi123:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-09-16T15:31:15.363 DEBUG:teuthology.orchestra.run.smithi184:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-09-16T15:31:15.369 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-09-16T15:31:15.444 INFO:teuthology.run_tasks:Running task internal.archive... 2024-09-16T15:31:15.540 INFO:teuthology.task.internal:Creating archive directory... 2024-09-16T15:31:15.540 DEBUG:teuthology.orchestra.run.smithi123:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-09-16T15:31:15.543 DEBUG:teuthology.orchestra.run.smithi184:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-09-16T15:31:15.561 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-09-16T15:31:15.573 INFO:teuthology.task.internal:Enabling coredump saving... 2024-09-16T15:31:15.574 DEBUG:teuthology.orchestra.run.smithi123:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-09-16T15:31:15.600 DEBUG:teuthology.orchestra.run.smithi184:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-09-16T15:31:15.618 INFO:teuthology.orchestra.run.smithi123.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T15:31:15.627 INFO:teuthology.orchestra.run.smithi184.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T15:31:15.627 INFO:teuthology.orchestra.run.smithi123.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T15:31:15.635 INFO:teuthology.orchestra.run.smithi184.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T15:31:15.636 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-09-16T15:31:15.656 INFO:teuthology.task.internal:Configuring sudo... 2024-09-16T15:31:15.657 DEBUG:teuthology.orchestra.run.smithi123:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-09-16T15:31:15.672 DEBUG:teuthology.orchestra.run.smithi184:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-09-16T15:31:15.689 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-09-16T15:31:15.750 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-09-16T15:31:15.751 DEBUG:teuthology.orchestra.run.smithi123:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-09-16T15:31:15.754 DEBUG:teuthology.orchestra.run.smithi184:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-09-16T15:31:15.759 DEBUG:teuthology.orchestra.run.smithi123:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-16T15:31:15.802 DEBUG:teuthology.orchestra.run.smithi123:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-16T15:31:15.850 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2024-09-16T15:31:15.851 DEBUG:teuthology.orchestra.run.smithi123:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-09-16T15:31:15.913 DEBUG:teuthology.orchestra.run.smithi184:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-16T15:31:15.919 DEBUG:teuthology.orchestra.run.smithi184:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-16T15:31:15.969 DEBUG:teuthology.orchestra.run.smithi184:> set -ex 2024-09-16T15:31:15.969 DEBUG:teuthology.orchestra.run.smithi184:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-09-16T15:31:16.035 DEBUG:teuthology.orchestra.run.smithi123:> sudo service rsyslog restart 2024-09-16T15:31:16.038 DEBUG:teuthology.orchestra.run.smithi184:> sudo service rsyslog restart 2024-09-16T15:31:16.107 INFO:teuthology.run_tasks:Running task internal.timer... 2024-09-16T15:31:16.115 INFO:teuthology.task.internal:Starting timer... 2024-09-16T15:31:16.115 INFO:teuthology.run_tasks:Running task pcp... 2024-09-16T15:31:16.133 INFO:teuthology.run_tasks:Running task selinux... 2024-09-16T15:31:16.215 DEBUG:teuthology.task.selinux:Excluding smithi123: OS 'ubuntu' does not support SELinux 2024-09-16T15:31:16.215 DEBUG:teuthology.task.selinux:Excluding smithi184: OS 'ubuntu' does not support SELinux 2024-09-16T15:31:16.215 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-09-16T15:31:16.215 DEBUG:teuthology.task.selinux:Existing SELinux modes: {} 2024-09-16T15:31:16.215 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-09-16T15:31:16.215 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-09-16T15:31:16.224 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-09-16T15:31:16.245 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-09-16T15:31:16.246 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi123.front.sepia.ceph.com,smithi184.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-09-16T15:38:47.575 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi123.front.sepia.ceph.com'), Remote(name='ubuntu@smithi184.front.sepia.ceph.com')] 2024-09-16T15:38:47.577 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi123.front.sepia.ceph.com' 2024-09-16T15:38:47.578 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi123.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T15:38:47.657 DEBUG:teuthology.orchestra.run.smithi123:> true 2024-09-16T15:38:47.737 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi123.front.sepia.ceph.com' 2024-09-16T15:38:47.737 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi184.front.sepia.ceph.com' 2024-09-16T15:38:47.738 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi184.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T15:38:47.814 DEBUG:teuthology.orchestra.run.smithi184:> true 2024-09-16T15:38:47.889 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi184.front.sepia.ceph.com' 2024-09-16T15:38:47.889 INFO:teuthology.run_tasks:Running task clock... 2024-09-16T15:38:47.898 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-09-16T15:38:47.898 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-09-16T15:38:47.899 DEBUG:teuthology.orchestra.run.smithi123:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-16T15:38:47.901 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-09-16T15:38:47.901 DEBUG:teuthology.orchestra.run.smithi184:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-16T15:38:47.928 INFO:teuthology.orchestra.run.smithi123.stdout:16 Sep 15:38:47 ntpd[18457]: ntpd 4.2.8p15@1.3728-o Wed Feb 16 17:13:02 UTC 2022 (1): Starting 2024-09-16T15:38:47.928 INFO:teuthology.orchestra.run.smithi123.stdout:16 Sep 15:38:47 ntpd[18457]: Command line: ntpd -gq 2024-09-16T15:38:47.928 INFO:teuthology.orchestra.run.smithi123.stdout:16 Sep 15:38:47 ntpd[18457]: ---------------------------------------------------- 2024-09-16T15:38:47.928 INFO:teuthology.orchestra.run.smithi123.stdout:16 Sep 15:38:47 ntpd[18457]: ntp-4 is maintained by Network Time Foundation, 2024-09-16T15:38:47.928 INFO:teuthology.orchestra.run.smithi123.stdout:16 Sep 15:38:47 ntpd[18457]: Inc. (NTF), a non-profit 501(c)(3) public-benefit 2024-09-16T15:38:47.928 INFO:teuthology.orchestra.run.smithi123.stdout:16 Sep 15:38:47 ntpd[18457]: corporation. Support and training for ntp-4 are 2024-09-16T15:38:47.928 INFO:teuthology.orchestra.run.smithi123.stdout:16 Sep 15:38:47 ntpd[18457]: available at https://www.nwtime.org/support 2024-09-16T15:38:47.928 INFO:teuthology.orchestra.run.smithi123.stdout:16 Sep 15:38:47 ntpd[18457]: ---------------------------------------------------- 2024-09-16T15:38:47.929 INFO:teuthology.orchestra.run.smithi123.stdout:16 Sep 15:38:47 ntpd[18457]: proto: precision = 0.043 usec (-24) 2024-09-16T15:38:47.929 INFO:teuthology.orchestra.run.smithi123.stdout:16 Sep 15:38:47 ntpd[18457]: basedate set to 2022-02-04 2024-09-16T15:38:47.929 INFO:teuthology.orchestra.run.smithi123.stdout:16 Sep 15:38:47 ntpd[18457]: gps base set to 2022-02-06 (week 2196) 2024-09-16T15:38:47.929 INFO:teuthology.orchestra.run.smithi123.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-09-16T15:38:47.930 INFO:teuthology.orchestra.run.smithi123.stdout:16 Sep 15:38:47 ntpd[18457]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-09-16T15:38:47.930 INFO:teuthology.orchestra.run.smithi123.stdout:16 Sep 15:38:47 ntpd[18457]: restrict ::: KOD does nothing without LIMITED. 2024-09-16T15:38:47.930 INFO:teuthology.orchestra.run.smithi123.stderr:restrict ::: KOD does nothing without LIMITED. 2024-09-16T15:38:47.931 INFO:teuthology.orchestra.run.smithi123.stdout:16 Sep 15:38:47 ntpd[18457]: Listen and drop on 0 v6wildcard [::]:123 2024-09-16T15:38:47.931 INFO:teuthology.orchestra.run.smithi123.stdout:16 Sep 15:38:47 ntpd[18457]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2024-09-16T15:38:47.931 INFO:teuthology.orchestra.run.smithi123.stdout:16 Sep 15:38:47 ntpd[18457]: Listen normally on 2 lo 127.0.0.1:123 2024-09-16T15:38:47.931 INFO:teuthology.orchestra.run.smithi123.stdout:16 Sep 15:38:47 ntpd[18457]: Listen normally on 3 enp3s0f1 172.21.15.123:123 2024-09-16T15:38:47.931 INFO:teuthology.orchestra.run.smithi123.stdout:16 Sep 15:38:47 ntpd[18457]: Listen normally on 4 lo [::1]:123 2024-09-16T15:38:47.931 INFO:teuthology.orchestra.run.smithi123.stdout:16 Sep 15:38:47 ntpd[18457]: Listen normally on 5 enp3s0f1 [fe80::ec4:7aff:fe8f:d3f1%5]:123 2024-09-16T15:38:47.931 INFO:teuthology.orchestra.run.smithi123.stdout:16 Sep 15:38:47 ntpd[18457]: Listening on routing socket on fd #22 for interface updates 2024-09-16T15:38:47.955 INFO:teuthology.orchestra.run.smithi184.stdout:16 Sep 15:38:47 ntpd[18494]: ntpd 4.2.8p15@1.3728-o Wed Feb 16 17:13:02 UTC 2022 (1): Starting 2024-09-16T15:38:47.955 INFO:teuthology.orchestra.run.smithi184.stdout:16 Sep 15:38:47 ntpd[18494]: Command line: ntpd -gq 2024-09-16T15:38:47.955 INFO:teuthology.orchestra.run.smithi184.stdout:16 Sep 15:38:47 ntpd[18494]: ---------------------------------------------------- 2024-09-16T15:38:47.955 INFO:teuthology.orchestra.run.smithi184.stdout:16 Sep 15:38:47 ntpd[18494]: ntp-4 is maintained by Network Time Foundation, 2024-09-16T15:38:47.955 INFO:teuthology.orchestra.run.smithi184.stdout:16 Sep 15:38:47 ntpd[18494]: Inc. (NTF), a non-profit 501(c)(3) public-benefit 2024-09-16T15:38:47.955 INFO:teuthology.orchestra.run.smithi184.stdout:16 Sep 15:38:47 ntpd[18494]: corporation. Support and training for ntp-4 are 2024-09-16T15:38:47.955 INFO:teuthology.orchestra.run.smithi184.stdout:16 Sep 15:38:47 ntpd[18494]: available at https://www.nwtime.org/support 2024-09-16T15:38:47.956 INFO:teuthology.orchestra.run.smithi184.stdout:16 Sep 15:38:47 ntpd[18494]: ---------------------------------------------------- 2024-09-16T15:38:47.956 INFO:teuthology.orchestra.run.smithi184.stdout:16 Sep 15:38:47 ntpd[18494]: proto: precision = 0.044 usec (-24) 2024-09-16T15:38:47.956 INFO:teuthology.orchestra.run.smithi184.stdout:16 Sep 15:38:47 ntpd[18494]: basedate set to 2022-02-04 2024-09-16T15:38:47.956 INFO:teuthology.orchestra.run.smithi184.stdout:16 Sep 15:38:47 ntpd[18494]: gps base set to 2022-02-06 (week 2196) 2024-09-16T15:38:47.956 INFO:teuthology.orchestra.run.smithi184.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-09-16T15:38:47.957 INFO:teuthology.orchestra.run.smithi184.stdout:16 Sep 15:38:47 ntpd[18494]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-09-16T15:38:47.957 INFO:teuthology.orchestra.run.smithi184.stdout:16 Sep 15:38:47 ntpd[18494]: restrict ::: KOD does nothing without LIMITED. 2024-09-16T15:38:47.957 INFO:teuthology.orchestra.run.smithi184.stderr:restrict ::: KOD does nothing without LIMITED. 2024-09-16T15:38:47.958 INFO:teuthology.orchestra.run.smithi184.stdout:16 Sep 15:38:47 ntpd[18494]: Listen and drop on 0 v6wildcard [::]:123 2024-09-16T15:38:47.958 INFO:teuthology.orchestra.run.smithi184.stdout:16 Sep 15:38:47 ntpd[18494]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2024-09-16T15:38:47.958 INFO:teuthology.orchestra.run.smithi184.stdout:16 Sep 15:38:47 ntpd[18494]: Listen normally on 2 lo 127.0.0.1:123 2024-09-16T15:38:47.958 INFO:teuthology.orchestra.run.smithi184.stdout:16 Sep 15:38:47 ntpd[18494]: Listen normally on 3 enp3s0f1 172.21.15.184:123 2024-09-16T15:38:47.958 INFO:teuthology.orchestra.run.smithi184.stdout:16 Sep 15:38:47 ntpd[18494]: Listen normally on 4 lo [::1]:123 2024-09-16T15:38:47.958 INFO:teuthology.orchestra.run.smithi184.stdout:16 Sep 15:38:47 ntpd[18494]: Listen normally on 5 enp3s0f1 [fe80::ec4:7aff:fe88:7b8f%5]:123 2024-09-16T15:38:47.958 INFO:teuthology.orchestra.run.smithi184.stdout:16 Sep 15:38:47 ntpd[18494]: Listening on routing socket on fd #22 for interface updates 2024-09-16T15:38:48.928 INFO:teuthology.orchestra.run.smithi123.stderr:16 Sep 15:38:48 ntpd[18457]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T15:38:48.928 INFO:teuthology.orchestra.run.smithi123.stderr:16 Sep 15:38:48 ntpd[18457]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T15:38:48.956 INFO:teuthology.orchestra.run.smithi184.stderr:16 Sep 15:38:48 ntpd[18494]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T15:38:48.956 INFO:teuthology.orchestra.run.smithi184.stderr:16 Sep 15:38:48 ntpd[18494]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T15:38:48.956 INFO:teuthology.orchestra.run.smithi184.stderr:16 Sep 15:38:48 ntpd[18494]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T15:38:48.956 INFO:teuthology.orchestra.run.smithi184.stderr:16 Sep 15:38:48 ntpd[18494]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T15:38:48.956 INFO:teuthology.orchestra.run.smithi184.stderr:16 Sep 15:38:48 ntpd[18494]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T15:38:48.956 INFO:teuthology.orchestra.run.smithi184.stderr:16 Sep 15:38:48 ntpd[18494]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T15:38:48.957 INFO:teuthology.orchestra.run.smithi184.stderr:16 Sep 15:38:48 ntpd[18494]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-16T15:38:48.957 INFO:teuthology.orchestra.run.smithi184.stderr:16 Sep 15:38:48 ntpd[18494]: can't open /var/log/ntpstats/peerstats.20240916: Permission denied 2024-09-16T15:38:49.929 INFO:teuthology.orchestra.run.smithi123.stderr:16 Sep 15:38:49 ntpd[18457]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T15:38:49.929 INFO:teuthology.orchestra.run.smithi123.stderr:16 Sep 15:38:49 ntpd[18457]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T15:38:49.929 INFO:teuthology.orchestra.run.smithi123.stderr:16 Sep 15:38:49 ntpd[18457]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T15:38:49.929 INFO:teuthology.orchestra.run.smithi123.stderr:16 Sep 15:38:49 ntpd[18457]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T15:38:49.929 INFO:teuthology.orchestra.run.smithi123.stderr:16 Sep 15:38:49 ntpd[18457]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-16T15:38:49.929 INFO:teuthology.orchestra.run.smithi123.stderr:16 Sep 15:38:49 ntpd[18457]: can't open /var/log/ntpstats/peerstats.20240916: Permission denied 2024-09-16T15:38:49.956 INFO:teuthology.orchestra.run.smithi184.stderr:16 Sep 15:38:49 ntpd[18494]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T15:38:49.956 INFO:teuthology.orchestra.run.smithi184.stderr:16 Sep 15:38:49 ntpd[18494]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T15:38:49.956 INFO:teuthology.orchestra.run.smithi184.stderr:16 Sep 15:38:49 ntpd[18494]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T15:38:49.956 INFO:teuthology.orchestra.run.smithi184.stderr:16 Sep 15:38:49 ntpd[18494]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T15:38:49.956 INFO:teuthology.orchestra.run.smithi184.stderr:16 Sep 15:38:49 ntpd[18494]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-16T15:38:49.956 INFO:teuthology.orchestra.run.smithi184.stderr:16 Sep 15:38:49 ntpd[18494]: can't open /var/log/ntpstats/peerstats.20240916: Permission denied 2024-09-16T15:38:50.956 INFO:teuthology.orchestra.run.smithi184.stderr:16 Sep 15:38:50 ntpd[18494]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T15:38:50.956 INFO:teuthology.orchestra.run.smithi184.stderr:16 Sep 15:38:50 ntpd[18494]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T15:38:50.956 INFO:teuthology.orchestra.run.smithi184.stderr:16 Sep 15:38:50 ntpd[18494]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T15:38:50.956 INFO:teuthology.orchestra.run.smithi184.stderr:16 Sep 15:38:50 ntpd[18494]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T15:38:50.956 INFO:teuthology.orchestra.run.smithi184.stderr:16 Sep 15:38:50 ntpd[18494]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-16T15:38:50.956 INFO:teuthology.orchestra.run.smithi184.stderr:16 Sep 15:38:50 ntpd[18494]: can't open /var/log/ntpstats/peerstats.20240916: Permission denied 2024-09-16T15:38:51.929 INFO:teuthology.orchestra.run.smithi123.stderr:16 Sep 15:38:51 ntpd[18457]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T15:38:51.929 INFO:teuthology.orchestra.run.smithi123.stderr:16 Sep 15:38:51 ntpd[18457]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T15:38:51.930 INFO:teuthology.orchestra.run.smithi123.stderr:16 Sep 15:38:51 ntpd[18457]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T15:38:51.930 INFO:teuthology.orchestra.run.smithi123.stderr:16 Sep 15:38:51 ntpd[18457]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T15:38:51.930 INFO:teuthology.orchestra.run.smithi123.stderr:16 Sep 15:38:51 ntpd[18457]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T15:38:51.930 INFO:teuthology.orchestra.run.smithi123.stderr:16 Sep 15:38:51 ntpd[18457]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T15:38:51.930 INFO:teuthology.orchestra.run.smithi123.stderr:16 Sep 15:38:51 ntpd[18457]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-16T15:38:51.930 INFO:teuthology.orchestra.run.smithi123.stderr:16 Sep 15:38:51 ntpd[18457]: can't open /var/log/ntpstats/peerstats.20240916: Permission denied 2024-09-16T15:38:51.930 INFO:teuthology.orchestra.run.smithi123.stderr:16 Sep 15:38:51 ntpd[18457]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T15:38:51.930 INFO:teuthology.orchestra.run.smithi123.stderr:16 Sep 15:38:51 ntpd[18457]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T15:38:51.930 INFO:teuthology.orchestra.run.smithi123.stderr:16 Sep 15:38:51 ntpd[18457]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-16T15:38:51.930 INFO:teuthology.orchestra.run.smithi123.stderr:16 Sep 15:38:51 ntpd[18457]: can't open /var/log/ntpstats/peerstats.20240916: Permission denied 2024-09-16T15:38:51.956 INFO:teuthology.orchestra.run.smithi184.stderr:16 Sep 15:38:51 ntpd[18494]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T15:38:51.956 INFO:teuthology.orchestra.run.smithi184.stderr:16 Sep 15:38:51 ntpd[18494]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T15:38:51.956 INFO:teuthology.orchestra.run.smithi184.stderr:16 Sep 15:38:51 ntpd[18494]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T15:38:51.956 INFO:teuthology.orchestra.run.smithi184.stderr:16 Sep 15:38:51 ntpd[18494]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T15:38:51.956 INFO:teuthology.orchestra.run.smithi184.stderr:16 Sep 15:38:51 ntpd[18494]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T15:38:51.956 INFO:teuthology.orchestra.run.smithi184.stderr:16 Sep 15:38:51 ntpd[18494]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T15:38:51.956 INFO:teuthology.orchestra.run.smithi184.stderr:16 Sep 15:38:51 ntpd[18494]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-16T15:38:51.956 INFO:teuthology.orchestra.run.smithi184.stderr:16 Sep 15:38:51 ntpd[18494]: can't open /var/log/ntpstats/peerstats.20240916: Permission denied 2024-09-16T15:38:51.957 INFO:teuthology.orchestra.run.smithi184.stderr:16 Sep 15:38:51 ntpd[18494]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T15:38:51.957 INFO:teuthology.orchestra.run.smithi184.stderr:16 Sep 15:38:51 ntpd[18494]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T15:38:51.957 INFO:teuthology.orchestra.run.smithi184.stderr:16 Sep 15:38:51 ntpd[18494]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-16T15:38:51.957 INFO:teuthology.orchestra.run.smithi184.stderr:16 Sep 15:38:51 ntpd[18494]: can't open /var/log/ntpstats/peerstats.20240916: Permission denied 2024-09-16T15:38:52.929 INFO:teuthology.orchestra.run.smithi123.stderr:16 Sep 15:38:52 ntpd[18457]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T15:38:52.929 INFO:teuthology.orchestra.run.smithi123.stderr:16 Sep 15:38:52 ntpd[18457]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T15:38:52.929 INFO:teuthology.orchestra.run.smithi123.stderr:16 Sep 15:38:52 ntpd[18457]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T15:38:52.929 INFO:teuthology.orchestra.run.smithi123.stderr:16 Sep 15:38:52 ntpd[18457]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T15:38:52.929 INFO:teuthology.orchestra.run.smithi123.stderr:16 Sep 15:38:52 ntpd[18457]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-16T15:38:52.929 INFO:teuthology.orchestra.run.smithi123.stderr:16 Sep 15:38:52 ntpd[18457]: can't open /var/log/ntpstats/peerstats.20240916: Permission denied 2024-09-16T15:38:52.955 INFO:teuthology.orchestra.run.smithi184.stderr:16 Sep 15:38:52 ntpd[18494]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T15:38:52.956 INFO:teuthology.orchestra.run.smithi184.stderr:16 Sep 15:38:52 ntpd[18494]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T15:38:52.956 INFO:teuthology.orchestra.run.smithi184.stderr:16 Sep 15:38:52 ntpd[18494]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T15:38:52.956 INFO:teuthology.orchestra.run.smithi184.stderr:16 Sep 15:38:52 ntpd[18494]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T15:38:52.956 INFO:teuthology.orchestra.run.smithi184.stderr:16 Sep 15:38:52 ntpd[18494]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-16T15:38:52.956 INFO:teuthology.orchestra.run.smithi184.stderr:16 Sep 15:38:52 ntpd[18494]: can't open /var/log/ntpstats/peerstats.20240916: Permission denied 2024-09-16T15:38:53.929 INFO:teuthology.orchestra.run.smithi123.stderr:16 Sep 15:38:53 ntpd[18457]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T15:38:53.930 INFO:teuthology.orchestra.run.smithi123.stderr:16 Sep 15:38:53 ntpd[18457]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T15:38:53.930 INFO:teuthology.orchestra.run.smithi123.stderr:16 Sep 15:38:53 ntpd[18457]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T15:38:53.930 INFO:teuthology.orchestra.run.smithi123.stderr:16 Sep 15:38:53 ntpd[18457]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T15:38:53.930 INFO:teuthology.orchestra.run.smithi123.stderr:16 Sep 15:38:53 ntpd[18457]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T15:38:53.930 INFO:teuthology.orchestra.run.smithi123.stderr:16 Sep 15:38:53 ntpd[18457]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T15:38:53.930 INFO:teuthology.orchestra.run.smithi123.stderr:16 Sep 15:38:53 ntpd[18457]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-16T15:38:53.930 INFO:teuthology.orchestra.run.smithi123.stderr:16 Sep 15:38:53 ntpd[18457]: can't open /var/log/ntpstats/peerstats.20240916: Permission denied 2024-09-16T15:38:53.930 INFO:teuthology.orchestra.run.smithi123.stderr:16 Sep 15:38:53 ntpd[18457]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T15:38:53.930 INFO:teuthology.orchestra.run.smithi123.stderr:16 Sep 15:38:53 ntpd[18457]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T15:38:53.930 INFO:teuthology.orchestra.run.smithi123.stderr:16 Sep 15:38:53 ntpd[18457]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-16T15:38:53.931 INFO:teuthology.orchestra.run.smithi123.stderr:16 Sep 15:38:53 ntpd[18457]: can't open /var/log/ntpstats/peerstats.20240916: Permission denied 2024-09-16T15:38:53.956 INFO:teuthology.orchestra.run.smithi184.stderr:16 Sep 15:38:53 ntpd[18494]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T15:38:53.956 INFO:teuthology.orchestra.run.smithi184.stderr:16 Sep 15:38:53 ntpd[18494]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T15:38:53.956 INFO:teuthology.orchestra.run.smithi184.stderr:16 Sep 15:38:53 ntpd[18494]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T15:38:53.956 INFO:teuthology.orchestra.run.smithi184.stderr:16 Sep 15:38:53 ntpd[18494]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T15:38:53.956 INFO:teuthology.orchestra.run.smithi184.stderr:16 Sep 15:38:53 ntpd[18494]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T15:38:53.956 INFO:teuthology.orchestra.run.smithi184.stderr:16 Sep 15:38:53 ntpd[18494]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T15:38:53.956 INFO:teuthology.orchestra.run.smithi184.stderr:16 Sep 15:38:53 ntpd[18494]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-16T15:38:53.956 INFO:teuthology.orchestra.run.smithi184.stderr:16 Sep 15:38:53 ntpd[18494]: can't open /var/log/ntpstats/peerstats.20240916: Permission denied 2024-09-16T15:38:53.957 INFO:teuthology.orchestra.run.smithi184.stderr:16 Sep 15:38:53 ntpd[18494]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T15:38:53.957 INFO:teuthology.orchestra.run.smithi184.stderr:16 Sep 15:38:53 ntpd[18494]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T15:38:53.957 INFO:teuthology.orchestra.run.smithi184.stderr:16 Sep 15:38:53 ntpd[18494]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-16T15:38:53.957 INFO:teuthology.orchestra.run.smithi184.stderr:16 Sep 15:38:53 ntpd[18494]: can't open /var/log/ntpstats/peerstats.20240916: Permission denied 2024-09-16T15:38:54.929 INFO:teuthology.orchestra.run.smithi123.stderr:16 Sep 15:38:54 ntpd[18457]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T15:38:54.929 INFO:teuthology.orchestra.run.smithi123.stderr:16 Sep 15:38:54 ntpd[18457]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T15:38:54.929 INFO:teuthology.orchestra.run.smithi123.stderr:16 Sep 15:38:54 ntpd[18457]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T15:38:54.929 INFO:teuthology.orchestra.run.smithi123.stderr:16 Sep 15:38:54 ntpd[18457]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T15:38:54.929 INFO:teuthology.orchestra.run.smithi123.stderr:16 Sep 15:38:54 ntpd[18457]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-16T15:38:54.929 INFO:teuthology.orchestra.run.smithi123.stderr:16 Sep 15:38:54 ntpd[18457]: can't open /var/log/ntpstats/peerstats.20240916: Permission denied 2024-09-16T15:38:54.956 INFO:teuthology.orchestra.run.smithi184.stderr:16 Sep 15:38:54 ntpd[18494]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T15:38:54.956 INFO:teuthology.orchestra.run.smithi184.stderr:16 Sep 15:38:54 ntpd[18494]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T15:38:54.956 INFO:teuthology.orchestra.run.smithi184.stderr:16 Sep 15:38:54 ntpd[18494]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T15:38:54.956 INFO:teuthology.orchestra.run.smithi184.stderr:16 Sep 15:38:54 ntpd[18494]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T15:38:54.956 INFO:teuthology.orchestra.run.smithi184.stdout:16 Sep 15:38:54 ntpd[18494]: ntpd: time slew -0.000846 s 2024-09-16T15:38:54.956 INFO:teuthology.orchestra.run.smithi184.stdout:ntpd: time slew -0.000846s 2024-09-16T15:38:54.957 INFO:teuthology.orchestra.run.smithi184.stderr:16 Sep 15:38:54 ntpd[18494]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-16T15:38:54.957 INFO:teuthology.orchestra.run.smithi184.stderr:16 Sep 15:38:54 ntpd[18494]: can't open /var/log/ntpstats/peerstats.20240916: Permission denied 2024-09-16T15:38:54.957 INFO:teuthology.orchestra.run.smithi184.stderr:16 Sep 15:38:54 ntpd[18494]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2024-09-16T15:38:54.957 INFO:teuthology.orchestra.run.smithi184.stderr:16 Sep 15:38:54 ntpd[18494]: can't open /var/log/ntpstats/loopstats.20240916: Permission denied 2024-09-16T15:38:55.013 INFO:teuthology.orchestra.run.smithi184.stdout: remote refid st t when poll reach delay offset jitter 2024-09-16T15:38:55.013 INFO:teuthology.orchestra.run.smithi184.stdout:============================================================================== 2024-09-16T15:38:55.013 INFO:teuthology.orchestra.run.smithi184.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-09-16T15:38:55.013 INFO:teuthology.orchestra.run.smithi184.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-09-16T15:38:55.013 INFO:teuthology.orchestra.run.smithi184.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-09-16T15:38:55.014 INFO:teuthology.orchestra.run.smithi184.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-09-16T15:38:55.929 INFO:teuthology.orchestra.run.smithi123.stderr:16 Sep 15:38:55 ntpd[18457]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T15:38:55.930 INFO:teuthology.orchestra.run.smithi123.stderr:16 Sep 15:38:55 ntpd[18457]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T15:38:55.930 INFO:teuthology.orchestra.run.smithi123.stderr:16 Sep 15:38:55 ntpd[18457]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T15:38:55.930 INFO:teuthology.orchestra.run.smithi123.stderr:16 Sep 15:38:55 ntpd[18457]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T15:38:55.930 INFO:teuthology.orchestra.run.smithi123.stderr:16 Sep 15:38:55 ntpd[18457]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T15:38:55.930 INFO:teuthology.orchestra.run.smithi123.stderr:16 Sep 15:38:55 ntpd[18457]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T15:38:55.930 INFO:teuthology.orchestra.run.smithi123.stderr:16 Sep 15:38:55 ntpd[18457]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-16T15:38:55.930 INFO:teuthology.orchestra.run.smithi123.stderr:16 Sep 15:38:55 ntpd[18457]: can't open /var/log/ntpstats/peerstats.20240916: Permission denied 2024-09-16T15:38:55.930 INFO:teuthology.orchestra.run.smithi123.stderr:16 Sep 15:38:55 ntpd[18457]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T15:38:55.930 INFO:teuthology.orchestra.run.smithi123.stderr:16 Sep 15:38:55 ntpd[18457]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T15:38:55.930 INFO:teuthology.orchestra.run.smithi123.stderr:16 Sep 15:38:55 ntpd[18457]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-16T15:38:55.930 INFO:teuthology.orchestra.run.smithi123.stderr:16 Sep 15:38:55 ntpd[18457]: can't open /var/log/ntpstats/peerstats.20240916: Permission denied 2024-09-16T15:38:55.931 INFO:teuthology.orchestra.run.smithi123.stdout:16 Sep 15:38:55 ntpd[18457]: ntpd: time slew -0.001249 s 2024-09-16T15:38:55.931 INFO:teuthology.orchestra.run.smithi123.stdout:ntpd: time slew -0.001249s 2024-09-16T15:38:55.931 INFO:teuthology.orchestra.run.smithi123.stderr:16 Sep 15:38:55 ntpd[18457]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2024-09-16T15:38:55.931 INFO:teuthology.orchestra.run.smithi123.stderr:16 Sep 15:38:55 ntpd[18457]: can't open /var/log/ntpstats/loopstats.20240916: Permission denied 2024-09-16T15:38:56.012 INFO:teuthology.orchestra.run.smithi123.stdout: remote refid st t when poll reach delay offset jitter 2024-09-16T15:38:56.012 INFO:teuthology.orchestra.run.smithi123.stdout:============================================================================== 2024-09-16T15:38:56.012 INFO:teuthology.orchestra.run.smithi123.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-09-16T15:38:56.013 INFO:teuthology.orchestra.run.smithi123.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-09-16T15:38:56.013 INFO:teuthology.orchestra.run.smithi123.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-09-16T15:38:56.013 INFO:teuthology.orchestra.run.smithi123.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-09-16T15:38:56.014 INFO:teuthology.run_tasks:Running task install... 2024-09-16T15:38:56.022 DEBUG:teuthology.task.install:project ceph 2024-09-16T15:38:56.022 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '33f95d2ae1bcbe7eb04404303f179eeeaa6982df'}} 2024-09-16T15:38:56.022 DEBUG:teuthology.task.install:config {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': '33f95d2ae1bcbe7eb04404303f179eeeaa6982df'} 2024-09-16T15:38:56.023 INFO:teuthology.task.install:Using flavor: default 2024-09-16T15:38:56.028 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-09-16T15:38:56.028 INFO:teuthology.task.install:extra packages: {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']} 2024-09-16T15:38:56.029 DEBUG:teuthology.orchestra.run.smithi123:> sudo apt-key list | grep Ceph 2024-09-16T15:38:56.029 DEBUG:teuthology.orchestra.run.smithi184:> sudo apt-key list | grep Ceph 2024-09-16T15:38:56.088 INFO:teuthology.orchestra.run.smithi184.stderr:Warning: apt-key is deprecated. Manage keyring files in trusted.gpg.d instead (see apt-key(8)). 2024-09-16T15:38:56.112 INFO:teuthology.orchestra.run.smithi123.stderr:Warning: apt-key is deprecated. Manage keyring files in trusted.gpg.d instead (see apt-key(8)). 2024-09-16T15:38:56.117 INFO:teuthology.orchestra.run.smithi184.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2024-09-16T15:38:56.117 INFO:teuthology.orchestra.run.smithi184.stdout:uid [ unknown] Ceph.com (release key) 2024-09-16T15:38:56.117 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, ceph-volume, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror on remote deb x86_64 2024-09-16T15:38:56.117 INFO:teuthology.task.install.deb:Installing system (non-project) packages: bison, flex, libelf-dev, libssl-dev, network-manager, iproute2, util-linux, dump, indent, libaio-dev, libtool-bin, uuid-dev, xfslibs-dev, postgresql, postgresql-client, postgresql-common, postgresql-contrib on remote deb x86_64 2024-09-16T15:38:56.118 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=33f95d2ae1bcbe7eb04404303f179eeeaa6982df 2024-09-16T15:38:56.139 INFO:teuthology.orchestra.run.smithi123.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2024-09-16T15:38:56.139 INFO:teuthology.orchestra.run.smithi123.stdout:uid [ unknown] Ceph.com (release key) 2024-09-16T15:38:56.140 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, ceph-volume, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror on remote deb x86_64 2024-09-16T15:38:56.140 INFO:teuthology.task.install.deb:Installing system (non-project) packages: bison, flex, libelf-dev, libssl-dev, network-manager, iproute2, util-linux, dump, indent, libaio-dev, libtool-bin, uuid-dev, xfslibs-dev, postgresql, postgresql-client, postgresql-common, postgresql-contrib on remote deb x86_64 2024-09-16T15:38:56.140 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=33f95d2ae1bcbe7eb04404303f179eeeaa6982df 2024-09-16T15:38:56.263 INFO:teuthology.task.install.deb:Pulling from https://4.chacra.ceph.com/r/ceph/wip-athakkar-testing-2024-09-16-1454/33f95d2ae1bcbe7eb04404303f179eeeaa6982df/ubuntu/jammy/flavors/default/ 2024-09-16T15:38:56.263 INFO:teuthology.task.install.deb:Package version is 19.3.0-4938-g33f95d2a-1jammy 2024-09-16T15:38:56.302 INFO:teuthology.task.install.deb:Pulling from https://4.chacra.ceph.com/r/ceph/wip-athakkar-testing-2024-09-16-1454/33f95d2ae1bcbe7eb04404303f179eeeaa6982df/ubuntu/jammy/flavors/default/ 2024-09-16T15:38:56.302 INFO:teuthology.task.install.deb:Package version is 19.3.0-4938-g33f95d2a-1jammy 2024-09-16T15:38:56.370 DEBUG:teuthology.orchestra.run.smithi184:> set -ex 2024-09-16T15:38:56.371 DEBUG:teuthology.orchestra.run.smithi184:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2024-09-16T15:38:56.385 DEBUG:teuthology.orchestra.run.smithi184:> sudo apt-get update 2024-09-16T15:38:56.410 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2024-09-16T15:38:56.410 DEBUG:teuthology.orchestra.run.smithi123:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2024-09-16T15:38:56.424 DEBUG:teuthology.orchestra.run.smithi123:> sudo apt-get update 2024-09-16T15:38:56.569 INFO:teuthology.orchestra.run.smithi184.stdout:Hit:1 http://security.ubuntu.com/ubuntu jammy-security InRelease 2024-09-16T15:38:56.593 INFO:teuthology.orchestra.run.smithi123.stdout:Hit:1 http://security.ubuntu.com/ubuntu jammy-security InRelease 2024-09-16T15:38:56.611 INFO:teuthology.orchestra.run.smithi184.stdout:Ign:2 https://4.chacra.ceph.com/r/ceph/wip-athakkar-testing-2024-09-16-1454/33f95d2ae1bcbe7eb04404303f179eeeaa6982df/ubuntu/jammy/flavors/default jammy InRelease 2024-09-16T15:38:56.633 INFO:teuthology.orchestra.run.smithi184.stdout:Get:3 https://4.chacra.ceph.com/r/ceph/wip-athakkar-testing-2024-09-16-1454/33f95d2ae1bcbe7eb04404303f179eeeaa6982df/ubuntu/jammy/flavors/default jammy Release [7,005 B] 2024-09-16T15:38:56.645 INFO:teuthology.orchestra.run.smithi123.stdout:Ign:2 https://4.chacra.ceph.com/r/ceph/wip-athakkar-testing-2024-09-16-1454/33f95d2ae1bcbe7eb04404303f179eeeaa6982df/ubuntu/jammy/flavors/default jammy InRelease 2024-09-16T15:38:56.655 INFO:teuthology.orchestra.run.smithi184.stdout:Ign:4 https://4.chacra.ceph.com/r/ceph/wip-athakkar-testing-2024-09-16-1454/33f95d2ae1bcbe7eb04404303f179eeeaa6982df/ubuntu/jammy/flavors/default jammy Release.gpg 2024-09-16T15:38:56.667 INFO:teuthology.orchestra.run.smithi123.stdout:Get:3 https://4.chacra.ceph.com/r/ceph/wip-athakkar-testing-2024-09-16-1454/33f95d2ae1bcbe7eb04404303f179eeeaa6982df/ubuntu/jammy/flavors/default jammy Release [7,005 B] 2024-09-16T15:38:56.670 INFO:teuthology.orchestra.run.smithi184.stdout:Hit:5 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-09-16T15:38:56.677 INFO:teuthology.orchestra.run.smithi184.stdout:Get:6 https://4.chacra.ceph.com/r/ceph/wip-athakkar-testing-2024-09-16-1454/33f95d2ae1bcbe7eb04404303f179eeeaa6982df/ubuntu/jammy/flavors/default jammy/main i386 Packages [3,431 B] 2024-09-16T15:38:56.689 INFO:teuthology.orchestra.run.smithi123.stdout:Ign:4 https://4.chacra.ceph.com/r/ceph/wip-athakkar-testing-2024-09-16-1454/33f95d2ae1bcbe7eb04404303f179eeeaa6982df/ubuntu/jammy/flavors/default jammy Release.gpg 2024-09-16T15:38:56.713 INFO:teuthology.orchestra.run.smithi123.stdout:Hit:5 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-09-16T15:38:56.715 INFO:teuthology.orchestra.run.smithi184.stdout:Get:7 https://4.chacra.ceph.com/r/ceph/wip-athakkar-testing-2024-09-16-1454/33f95d2ae1bcbe7eb04404303f179eeeaa6982df/ubuntu/jammy/flavors/default jammy/main amd64 Packages [18.2 kB] 2024-09-16T15:38:56.716 INFO:teuthology.orchestra.run.smithi123.stdout:Get:6 https://4.chacra.ceph.com/r/ceph/wip-athakkar-testing-2024-09-16-1454/33f95d2ae1bcbe7eb04404303f179eeeaa6982df/ubuntu/jammy/flavors/default jammy/main i386 Packages [3,431 B] 2024-09-16T15:38:56.744 INFO:teuthology.orchestra.run.smithi123.stdout:Get:7 https://4.chacra.ceph.com/r/ceph/wip-athakkar-testing-2024-09-16-1454/33f95d2ae1bcbe7eb04404303f179eeeaa6982df/ubuntu/jammy/flavors/default jammy/main amd64 Packages [18.2 kB] 2024-09-16T15:38:56.751 INFO:teuthology.orchestra.run.smithi184.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates InRelease [128 kB] 2024-09-16T15:38:56.794 INFO:teuthology.orchestra.run.smithi123.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates InRelease [128 kB] 2024-09-16T15:38:57.109 INFO:teuthology.orchestra.run.smithi184.stdout:Hit:9 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2024-09-16T15:38:57.151 INFO:teuthology.orchestra.run.smithi123.stdout:Hit:9 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2024-09-16T15:38:57.202 INFO:teuthology.orchestra.run.smithi184.stdout:Get:10 http://archive.ubuntu.com/ubuntu jammy-updates/main i386 Packages [696 kB] 2024-09-16T15:38:57.241 INFO:teuthology.orchestra.run.smithi123.stdout:Get:10 http://archive.ubuntu.com/ubuntu jammy-updates/main i386 Packages [696 kB] 2024-09-16T15:38:57.516 INFO:teuthology.orchestra.run.smithi184.stdout:Get:11 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 Packages [2,024 kB] 2024-09-16T15:38:57.555 INFO:teuthology.orchestra.run.smithi123.stdout:Get:11 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 Packages [2,024 kB] 2024-09-16T15:38:57.684 INFO:teuthology.orchestra.run.smithi184.stdout:Get:12 http://archive.ubuntu.com/ubuntu jammy-updates/universe i386 Packages [732 kB] 2024-09-16T15:38:57.707 INFO:teuthology.orchestra.run.smithi184.stdout:Get:13 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 Packages [1,124 kB] 2024-09-16T15:38:57.719 INFO:teuthology.orchestra.run.smithi123.stdout:Get:12 http://archive.ubuntu.com/ubuntu jammy-updates/universe i386 Packages [732 kB] 2024-09-16T15:38:57.745 INFO:teuthology.orchestra.run.smithi123.stdout:Get:13 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 Packages [1,124 kB] 2024-09-16T15:38:58.185 INFO:teuthology.orchestra.run.smithi184.stdout:Fetched 4,733 kB in 2s (2,934 kB/s) 2024-09-16T15:38:58.240 INFO:teuthology.orchestra.run.smithi123.stdout:Fetched 4,733 kB in 2s (2,945 kB/s) 2024-09-16T15:38:59.214 INFO:teuthology.orchestra.run.smithi184.stdout:Reading package lists... 2024-09-16T15:38:59.230 DEBUG:teuthology.orchestra.run.smithi184:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=19.3.0-4938-g33f95d2a-1jammy cephadm=19.3.0-4938-g33f95d2a-1jammy ceph-mds=19.3.0-4938-g33f95d2a-1jammy ceph-mgr=19.3.0-4938-g33f95d2a-1jammy ceph-common=19.3.0-4938-g33f95d2a-1jammy ceph-fuse=19.3.0-4938-g33f95d2a-1jammy ceph-test=19.3.0-4938-g33f95d2a-1jammy ceph-volume=19.3.0-4938-g33f95d2a-1jammy radosgw=19.3.0-4938-g33f95d2a-1jammy python3-rados=19.3.0-4938-g33f95d2a-1jammy python3-rgw=19.3.0-4938-g33f95d2a-1jammy python3-cephfs=19.3.0-4938-g33f95d2a-1jammy python3-rbd=19.3.0-4938-g33f95d2a-1jammy libcephfs2=19.3.0-4938-g33f95d2a-1jammy libcephfs-dev=19.3.0-4938-g33f95d2a-1jammy librados2=19.3.0-4938-g33f95d2a-1jammy librbd1=19.3.0-4938-g33f95d2a-1jammy rbd-fuse=19.3.0-4938-g33f95d2a-1jammy python3-cephfs=19.3.0-4938-g33f95d2a-1jammy cephfs-shell=19.3.0-4938-g33f95d2a-1jammy cephfs-top=19.3.0-4938-g33f95d2a-1jammy cephfs-mirror=19.3.0-4938-g33f95d2a-1jammy 2024-09-16T15:38:59.274 INFO:teuthology.orchestra.run.smithi123.stdout:Reading package lists... 2024-09-16T15:38:59.289 DEBUG:teuthology.orchestra.run.smithi123:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=19.3.0-4938-g33f95d2a-1jammy cephadm=19.3.0-4938-g33f95d2a-1jammy ceph-mds=19.3.0-4938-g33f95d2a-1jammy ceph-mgr=19.3.0-4938-g33f95d2a-1jammy ceph-common=19.3.0-4938-g33f95d2a-1jammy ceph-fuse=19.3.0-4938-g33f95d2a-1jammy ceph-test=19.3.0-4938-g33f95d2a-1jammy ceph-volume=19.3.0-4938-g33f95d2a-1jammy radosgw=19.3.0-4938-g33f95d2a-1jammy python3-rados=19.3.0-4938-g33f95d2a-1jammy python3-rgw=19.3.0-4938-g33f95d2a-1jammy python3-cephfs=19.3.0-4938-g33f95d2a-1jammy python3-rbd=19.3.0-4938-g33f95d2a-1jammy libcephfs2=19.3.0-4938-g33f95d2a-1jammy libcephfs-dev=19.3.0-4938-g33f95d2a-1jammy librados2=19.3.0-4938-g33f95d2a-1jammy librbd1=19.3.0-4938-g33f95d2a-1jammy rbd-fuse=19.3.0-4938-g33f95d2a-1jammy python3-cephfs=19.3.0-4938-g33f95d2a-1jammy cephfs-shell=19.3.0-4938-g33f95d2a-1jammy cephfs-top=19.3.0-4938-g33f95d2a-1jammy cephfs-mirror=19.3.0-4938-g33f95d2a-1jammy 2024-09-16T15:38:59.292 INFO:teuthology.orchestra.run.smithi184.stdout:Reading package lists... 2024-09-16T15:38:59.351 INFO:teuthology.orchestra.run.smithi123.stdout:Reading package lists... 2024-09-16T15:38:59.500 INFO:teuthology.orchestra.run.smithi184.stdout:Building dependency tree... 2024-09-16T15:38:59.501 INFO:teuthology.orchestra.run.smithi184.stdout:Reading state information... 2024-09-16T15:38:59.561 INFO:teuthology.orchestra.run.smithi123.stdout:Building dependency tree... 2024-09-16T15:38:59.562 INFO:teuthology.orchestra.run.smithi123.stdout:Reading state information... 2024-09-16T15:38:59.693 INFO:teuthology.orchestra.run.smithi184.stdout:The following packages were automatically installed and are no longer required: 2024-09-16T15:38:59.693 INFO:teuthology.orchestra.run.smithi184.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-09-16T15:38:59.694 INFO:teuthology.orchestra.run.smithi184.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-09-16T15:38:59.694 INFO:teuthology.orchestra.run.smithi184.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-09-16T15:38:59.694 INFO:teuthology.orchestra.run.smithi184.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-09-16T15:38:59.694 INFO:teuthology.orchestra.run.smithi184.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-09-16T15:38:59.694 INFO:teuthology.orchestra.run.smithi184.stdout:Use 'sudo apt autoremove' to remove them. 2024-09-16T15:38:59.695 INFO:teuthology.orchestra.run.smithi184.stdout:The following additional packages will be installed: 2024-09-16T15:38:59.695 INFO:teuthology.orchestra.run.smithi184.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-local 2024-09-16T15:38:59.695 INFO:teuthology.orchestra.run.smithi184.stdout: ceph-mgr-k8sevents ceph-mgr-modules-core ceph-mon ceph-osd jq 2024-09-16T15:38:59.696 INFO:teuthology.orchestra.run.smithi184.stdout: libdouble-conversion3 libjq1 liblttng-ust1 liblua5.3-dev libnbd0 liboath0 2024-09-16T15:38:59.696 INFO:teuthology.orchestra.run.smithi184.stdout: libonig5 libpcre2-16-0 libqt5core5a libqt5dbus5 libqt5network5 2024-09-16T15:38:59.696 INFO:teuthology.orchestra.run.smithi184.stdout: libradosstriper1 librdkafka1 libreadline-dev librgw2 libsqlite3-mod-ceph 2024-09-16T15:38:59.696 INFO:teuthology.orchestra.run.smithi184.stdout: libthrift-0.16.0 lua-any lua-sec lua-socket lua5.1 luarocks nvme-cli 2024-09-16T15:38:59.697 INFO:teuthology.orchestra.run.smithi184.stdout: pkg-config python-asyncssh-doc python-babel-localedata 2024-09-16T15:38:59.697 INFO:teuthology.orchestra.run.smithi184.stdout: python-pastedeploy-tpl python3-asyncssh python3-babel python3-bcrypt 2024-09-16T15:38:59.697 INFO:teuthology.orchestra.run.smithi184.stdout: python3-cachetools python3-ceph-argparse python3-ceph-common python3-certifi 2024-09-16T15:38:59.697 INFO:teuthology.orchestra.run.smithi184.stdout: python3-cheroot python3-cherrypy3 python3-cmd2 python3-colorama 2024-09-16T15:38:59.697 INFO:teuthology.orchestra.run.smithi184.stdout: python3-google-auth python3-idna python3-iniconfig python3-jaraco.classes 2024-09-16T15:38:59.697 INFO:teuthology.orchestra.run.smithi184.stdout: python3-jaraco.collections python3-jaraco.functools python3-jaraco.text 2024-09-16T15:38:59.697 INFO:teuthology.orchestra.run.smithi184.stdout: python3-jinja2 python3-joblib python3-kubernetes python3-logutils 2024-09-16T15:38:59.697 INFO:teuthology.orchestra.run.smithi184.stdout: python3-mako python3-markupsafe python3-natsort python3-openssl 2024-09-16T15:38:59.697 INFO:teuthology.orchestra.run.smithi184.stdout: python3-paste python3-pastedeploy python3-pastescript python3-pecan 2024-09-16T15:38:59.697 INFO:teuthology.orchestra.run.smithi184.stdout: python3-pluggy python3-portend python3-prettytable python3-psutil python3-py 2024-09-16T15:38:59.697 INFO:teuthology.orchestra.run.smithi184.stdout: python3-pyasn1 python3-pyasn1-modules python3-pygments python3-pyinotify 2024-09-16T15:38:59.697 INFO:teuthology.orchestra.run.smithi184.stdout: python3-pyperclip python3-pytest python3-repoze.lru python3-requests 2024-09-16T15:38:59.698 INFO:teuthology.orchestra.run.smithi184.stdout: python3-requests-oauthlib python3-routes python3-rsa python3-simplegeneric 2024-09-16T15:38:59.698 INFO:teuthology.orchestra.run.smithi184.stdout: python3-simplejson python3-singledispatch python3-sklearn 2024-09-16T15:38:59.698 INFO:teuthology.orchestra.run.smithi184.stdout: python3-sklearn-lib python3-tempita python3-tempora python3-threadpoolctl 2024-09-16T15:38:59.698 INFO:teuthology.orchestra.run.smithi184.stdout: python3-toml python3-urllib3 python3-waitress python3-wcwidth python3-webob 2024-09-16T15:38:59.698 INFO:teuthology.orchestra.run.smithi184.stdout: python3-websocket python3-webtest python3-werkzeug python3-zc.lockfile 2024-09-16T15:38:59.698 INFO:teuthology.orchestra.run.smithi184.stdout: qttranslations5-l10n socat unzip xmlstarlet zip 2024-09-16T15:38:59.698 INFO:teuthology.orchestra.run.smithi184.stdout:Suggested packages: 2024-09-16T15:38:59.698 INFO:teuthology.orchestra.run.smithi184.stdout: python3-influxdb readline-doc python-jinja2-doc python3-beaker 2024-09-16T15:38:59.698 INFO:teuthology.orchestra.run.smithi184.stdout: python-mako-doc python-natsort-doc python-openssl-doc python3-openssl-dbg 2024-09-16T15:38:59.699 INFO:teuthology.orchestra.run.smithi184.stdout: httpd-wsgi libapache2-mod-python libapache2-mod-scgi libjs-mochikit 2024-09-16T15:38:59.699 INFO:teuthology.orchestra.run.smithi184.stdout: python-pecan-doc python-psutil-doc subversion python-pygments-doc 2024-09-16T15:38:59.699 INFO:teuthology.orchestra.run.smithi184.stdout: python-pyinotify-doc python3-socks python-requests-doc python3-dap 2024-09-16T15:38:59.699 INFO:teuthology.orchestra.run.smithi184.stdout: python-sklearn-doc ipython3 python-waitress-doc python-webob-doc 2024-09-16T15:38:59.699 INFO:teuthology.orchestra.run.smithi184.stdout: python-webtest-doc python-werkzeug-doc python3-watchdog 2024-09-16T15:38:59.699 INFO:teuthology.orchestra.run.smithi184.stdout:Recommended packages: 2024-09-16T15:38:59.699 INFO:teuthology.orchestra.run.smithi184.stdout: btrfs-tools 2024-09-16T15:38:59.757 INFO:teuthology.orchestra.run.smithi123.stdout:The following packages were automatically installed and are no longer required: 2024-09-16T15:38:59.757 INFO:teuthology.orchestra.run.smithi123.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-09-16T15:38:59.757 INFO:teuthology.orchestra.run.smithi123.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-09-16T15:38:59.757 INFO:teuthology.orchestra.run.smithi123.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-09-16T15:38:59.758 INFO:teuthology.orchestra.run.smithi123.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-09-16T15:38:59.758 INFO:teuthology.orchestra.run.smithi123.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-09-16T15:38:59.758 INFO:teuthology.orchestra.run.smithi123.stdout:Use 'sudo apt autoremove' to remove them. 2024-09-16T15:38:59.759 INFO:teuthology.orchestra.run.smithi123.stdout:The following additional packages will be installed: 2024-09-16T15:38:59.759 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-local 2024-09-16T15:38:59.759 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-mgr-k8sevents ceph-mgr-modules-core ceph-mon ceph-osd jq 2024-09-16T15:38:59.759 INFO:teuthology.orchestra.run.smithi123.stdout: libdouble-conversion3 libjq1 liblttng-ust1 liblua5.3-dev libnbd0 liboath0 2024-09-16T15:38:59.759 INFO:teuthology.orchestra.run.smithi123.stdout: libonig5 libpcre2-16-0 libqt5core5a libqt5dbus5 libqt5network5 2024-09-16T15:38:59.759 INFO:teuthology.orchestra.run.smithi123.stdout: libradosstriper1 librdkafka1 libreadline-dev librgw2 libsqlite3-mod-ceph 2024-09-16T15:38:59.760 INFO:teuthology.orchestra.run.smithi123.stdout: libthrift-0.16.0 lua-any lua-sec lua-socket lua5.1 luarocks nvme-cli 2024-09-16T15:38:59.760 INFO:teuthology.orchestra.run.smithi123.stdout: pkg-config python-asyncssh-doc python-babel-localedata 2024-09-16T15:38:59.760 INFO:teuthology.orchestra.run.smithi123.stdout: python-pastedeploy-tpl python3-asyncssh python3-babel python3-bcrypt 2024-09-16T15:38:59.761 INFO:teuthology.orchestra.run.smithi123.stdout: python3-cachetools python3-ceph-argparse python3-ceph-common python3-certifi 2024-09-16T15:38:59.761 INFO:teuthology.orchestra.run.smithi123.stdout: python3-cheroot python3-cherrypy3 python3-cmd2 python3-colorama 2024-09-16T15:38:59.761 INFO:teuthology.orchestra.run.smithi123.stdout: python3-google-auth python3-idna python3-iniconfig python3-jaraco.classes 2024-09-16T15:38:59.761 INFO:teuthology.orchestra.run.smithi123.stdout: python3-jaraco.collections python3-jaraco.functools python3-jaraco.text 2024-09-16T15:38:59.761 INFO:teuthology.orchestra.run.smithi123.stdout: python3-jinja2 python3-joblib python3-kubernetes python3-logutils 2024-09-16T15:38:59.761 INFO:teuthology.orchestra.run.smithi123.stdout: python3-mako python3-markupsafe python3-natsort python3-openssl 2024-09-16T15:38:59.761 INFO:teuthology.orchestra.run.smithi123.stdout: python3-paste python3-pastedeploy python3-pastescript python3-pecan 2024-09-16T15:38:59.761 INFO:teuthology.orchestra.run.smithi123.stdout: python3-pluggy python3-portend python3-prettytable python3-psutil python3-py 2024-09-16T15:38:59.761 INFO:teuthology.orchestra.run.smithi123.stdout: python3-pyasn1 python3-pyasn1-modules python3-pygments python3-pyinotify 2024-09-16T15:38:59.761 INFO:teuthology.orchestra.run.smithi123.stdout: python3-pyperclip python3-pytest python3-repoze.lru python3-requests 2024-09-16T15:38:59.761 INFO:teuthology.orchestra.run.smithi123.stdout: python3-requests-oauthlib python3-routes python3-rsa python3-simplegeneric 2024-09-16T15:38:59.761 INFO:teuthology.orchestra.run.smithi123.stdout: python3-simplejson python3-singledispatch python3-sklearn 2024-09-16T15:38:59.761 INFO:teuthology.orchestra.run.smithi123.stdout: python3-sklearn-lib python3-tempita python3-tempora python3-threadpoolctl 2024-09-16T15:38:59.761 INFO:teuthology.orchestra.run.smithi123.stdout: python3-toml python3-urllib3 python3-waitress python3-wcwidth python3-webob 2024-09-16T15:38:59.762 INFO:teuthology.orchestra.run.smithi123.stdout: python3-websocket python3-webtest python3-werkzeug python3-zc.lockfile 2024-09-16T15:38:59.762 INFO:teuthology.orchestra.run.smithi123.stdout: qttranslations5-l10n socat unzip xmlstarlet zip 2024-09-16T15:38:59.762 INFO:teuthology.orchestra.run.smithi123.stdout:Suggested packages: 2024-09-16T15:38:59.762 INFO:teuthology.orchestra.run.smithi123.stdout: python3-influxdb readline-doc python-jinja2-doc python3-beaker 2024-09-16T15:38:59.762 INFO:teuthology.orchestra.run.smithi123.stdout: python-mako-doc python-natsort-doc python-openssl-doc python3-openssl-dbg 2024-09-16T15:38:59.762 INFO:teuthology.orchestra.run.smithi123.stdout: httpd-wsgi libapache2-mod-python libapache2-mod-scgi libjs-mochikit 2024-09-16T15:38:59.762 INFO:teuthology.orchestra.run.smithi123.stdout: python-pecan-doc python-psutil-doc subversion python-pygments-doc 2024-09-16T15:38:59.762 INFO:teuthology.orchestra.run.smithi123.stdout: python-pyinotify-doc python3-socks python-requests-doc python3-dap 2024-09-16T15:38:59.763 INFO:teuthology.orchestra.run.smithi123.stdout: python-sklearn-doc ipython3 python-waitress-doc python-webob-doc 2024-09-16T15:38:59.763 INFO:teuthology.orchestra.run.smithi123.stdout: python-webtest-doc python-werkzeug-doc python3-watchdog 2024-09-16T15:38:59.763 INFO:teuthology.orchestra.run.smithi123.stdout:Recommended packages: 2024-09-16T15:38:59.763 INFO:teuthology.orchestra.run.smithi123.stdout: btrfs-tools 2024-09-16T15:38:59.807 INFO:teuthology.orchestra.run.smithi184.stdout:The following NEW packages will be installed: 2024-09-16T15:38:59.807 INFO:teuthology.orchestra.run.smithi184.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2024-09-16T15:38:59.808 INFO:teuthology.orchestra.run.smithi184.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-k8sevents 2024-09-16T15:38:59.808 INFO:teuthology.orchestra.run.smithi184.stdout: ceph-mgr-modules-core ceph-mon ceph-osd ceph-test ceph-volume cephadm 2024-09-16T15:38:59.808 INFO:teuthology.orchestra.run.smithi184.stdout: cephfs-mirror cephfs-shell cephfs-top jq libcephfs-dev libcephfs2 2024-09-16T15:38:59.809 INFO:teuthology.orchestra.run.smithi184.stdout: libdouble-conversion3 libjq1 liblttng-ust1 liblua5.3-dev libnbd0 liboath0 2024-09-16T15:38:59.809 INFO:teuthology.orchestra.run.smithi184.stdout: libonig5 libpcre2-16-0 libqt5core5a libqt5dbus5 libqt5network5 librados2 2024-09-16T15:38:59.810 INFO:teuthology.orchestra.run.smithi184.stdout: libradosstriper1 librbd1 librdkafka1 libreadline-dev librgw2 2024-09-16T15:38:59.810 INFO:teuthology.orchestra.run.smithi184.stdout: libsqlite3-mod-ceph libthrift-0.16.0 lua-any lua-sec lua-socket lua5.1 2024-09-16T15:38:59.810 INFO:teuthology.orchestra.run.smithi184.stdout: luarocks nvme-cli pkg-config python-asyncssh-doc python-babel-localedata 2024-09-16T15:38:59.810 INFO:teuthology.orchestra.run.smithi184.stdout: python-pastedeploy-tpl python3-asyncssh python3-babel python3-bcrypt 2024-09-16T15:38:59.810 INFO:teuthology.orchestra.run.smithi184.stdout: python3-cachetools python3-ceph-argparse python3-ceph-common python3-cephfs 2024-09-16T15:38:59.810 INFO:teuthology.orchestra.run.smithi184.stdout: python3-certifi python3-cheroot python3-cherrypy3 python3-cmd2 2024-09-16T15:38:59.810 INFO:teuthology.orchestra.run.smithi184.stdout: python3-colorama python3-google-auth python3-idna python3-iniconfig 2024-09-16T15:38:59.810 INFO:teuthology.orchestra.run.smithi184.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2024-09-16T15:38:59.810 INFO:teuthology.orchestra.run.smithi184.stdout: python3-jaraco.text python3-jinja2 python3-joblib python3-kubernetes 2024-09-16T15:38:59.810 INFO:teuthology.orchestra.run.smithi184.stdout: python3-logutils python3-mako python3-markupsafe python3-natsort 2024-09-16T15:38:59.810 INFO:teuthology.orchestra.run.smithi184.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2024-09-16T15:38:59.810 INFO:teuthology.orchestra.run.smithi184.stdout: python3-pecan python3-pluggy python3-portend python3-prettytable 2024-09-16T15:38:59.811 INFO:teuthology.orchestra.run.smithi184.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2024-09-16T15:38:59.811 INFO:teuthology.orchestra.run.smithi184.stdout: python3-pygments python3-pyinotify python3-pyperclip python3-pytest 2024-09-16T15:38:59.811 INFO:teuthology.orchestra.run.smithi184.stdout: python3-rados python3-rbd python3-repoze.lru python3-requests 2024-09-16T15:38:59.811 INFO:teuthology.orchestra.run.smithi184.stdout: python3-requests-oauthlib python3-rgw python3-routes python3-rsa 2024-09-16T15:38:59.811 INFO:teuthology.orchestra.run.smithi184.stdout: python3-simplegeneric python3-simplejson python3-singledispatch 2024-09-16T15:38:59.811 INFO:teuthology.orchestra.run.smithi184.stdout: python3-sklearn python3-sklearn-lib python3-tempita python3-tempora 2024-09-16T15:38:59.811 INFO:teuthology.orchestra.run.smithi184.stdout: python3-threadpoolctl python3-toml python3-urllib3 python3-waitress 2024-09-16T15:38:59.811 INFO:teuthology.orchestra.run.smithi184.stdout: python3-wcwidth python3-webob python3-websocket python3-webtest 2024-09-16T15:38:59.811 INFO:teuthology.orchestra.run.smithi184.stdout: python3-werkzeug python3-zc.lockfile qttranslations5-l10n radosgw rbd-fuse 2024-09-16T15:38:59.811 INFO:teuthology.orchestra.run.smithi184.stdout: socat unzip xmlstarlet zip 2024-09-16T15:38:59.869 INFO:teuthology.orchestra.run.smithi123.stdout:The following NEW packages will be installed: 2024-09-16T15:38:59.870 INFO:teuthology.orchestra.run.smithi123.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2024-09-16T15:38:59.870 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-k8sevents 2024-09-16T15:38:59.870 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-mgr-modules-core ceph-mon ceph-osd ceph-test ceph-volume cephadm 2024-09-16T15:38:59.871 INFO:teuthology.orchestra.run.smithi123.stdout: cephfs-mirror cephfs-shell cephfs-top jq libcephfs-dev libcephfs2 2024-09-16T15:38:59.872 INFO:teuthology.orchestra.run.smithi123.stdout: libdouble-conversion3 libjq1 liblttng-ust1 liblua5.3-dev libnbd0 liboath0 2024-09-16T15:38:59.872 INFO:teuthology.orchestra.run.smithi123.stdout: libonig5 libpcre2-16-0 libqt5core5a libqt5dbus5 libqt5network5 librados2 2024-09-16T15:38:59.872 INFO:teuthology.orchestra.run.smithi123.stdout: libradosstriper1 librbd1 librdkafka1 libreadline-dev librgw2 2024-09-16T15:38:59.872 INFO:teuthology.orchestra.run.smithi123.stdout: libsqlite3-mod-ceph libthrift-0.16.0 lua-any lua-sec lua-socket lua5.1 2024-09-16T15:38:59.872 INFO:teuthology.orchestra.run.smithi123.stdout: luarocks nvme-cli pkg-config python-asyncssh-doc python-babel-localedata 2024-09-16T15:38:59.872 INFO:teuthology.orchestra.run.smithi123.stdout: python-pastedeploy-tpl python3-asyncssh python3-babel python3-bcrypt 2024-09-16T15:38:59.872 INFO:teuthology.orchestra.run.smithi123.stdout: python3-cachetools python3-ceph-argparse python3-ceph-common python3-cephfs 2024-09-16T15:38:59.872 INFO:teuthology.orchestra.run.smithi123.stdout: python3-certifi python3-cheroot python3-cherrypy3 python3-cmd2 2024-09-16T15:38:59.872 INFO:teuthology.orchestra.run.smithi123.stdout: python3-colorama python3-google-auth python3-idna python3-iniconfig 2024-09-16T15:38:59.872 INFO:teuthology.orchestra.run.smithi123.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2024-09-16T15:38:59.872 INFO:teuthology.orchestra.run.smithi123.stdout: python3-jaraco.text python3-jinja2 python3-joblib python3-kubernetes 2024-09-16T15:38:59.872 INFO:teuthology.orchestra.run.smithi123.stdout: python3-logutils python3-mako python3-markupsafe python3-natsort 2024-09-16T15:38:59.872 INFO:teuthology.orchestra.run.smithi123.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2024-09-16T15:38:59.873 INFO:teuthology.orchestra.run.smithi123.stdout: python3-pecan python3-pluggy python3-portend python3-prettytable 2024-09-16T15:38:59.873 INFO:teuthology.orchestra.run.smithi123.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2024-09-16T15:38:59.873 INFO:teuthology.orchestra.run.smithi123.stdout: python3-pygments python3-pyinotify python3-pyperclip python3-pytest 2024-09-16T15:38:59.873 INFO:teuthology.orchestra.run.smithi123.stdout: python3-rados python3-rbd python3-repoze.lru python3-requests 2024-09-16T15:38:59.873 INFO:teuthology.orchestra.run.smithi123.stdout: python3-requests-oauthlib python3-rgw python3-routes python3-rsa 2024-09-16T15:38:59.873 INFO:teuthology.orchestra.run.smithi123.stdout: python3-simplegeneric python3-simplejson python3-singledispatch 2024-09-16T15:38:59.873 INFO:teuthology.orchestra.run.smithi123.stdout: python3-sklearn python3-sklearn-lib python3-tempita python3-tempora 2024-09-16T15:38:59.873 INFO:teuthology.orchestra.run.smithi123.stdout: python3-threadpoolctl python3-toml python3-urllib3 python3-waitress 2024-09-16T15:38:59.873 INFO:teuthology.orchestra.run.smithi123.stdout: python3-wcwidth python3-webob python3-websocket python3-webtest 2024-09-16T15:38:59.873 INFO:teuthology.orchestra.run.smithi123.stdout: python3-werkzeug python3-zc.lockfile qttranslations5-l10n radosgw rbd-fuse 2024-09-16T15:38:59.873 INFO:teuthology.orchestra.run.smithi123.stdout: socat unzip xmlstarlet zip 2024-09-16T15:38:59.971 INFO:teuthology.orchestra.run.smithi184.stdout:0 upgraded, 125 newly installed, 0 to remove and 331 not upgraded. 2024-09-16T15:38:59.971 INFO:teuthology.orchestra.run.smithi184.stdout:Need to get 244 MB of archives. 2024-09-16T15:38:59.971 INFO:teuthology.orchestra.run.smithi184.stdout:After this operation, 910 MB of additional disk space will be used. 2024-09-16T15:38:59.971 INFO:teuthology.orchestra.run.smithi184.stdout:Get:1 https://4.chacra.ceph.com/r/ceph/wip-athakkar-testing-2024-09-16-1454/33f95d2ae1bcbe7eb04404303f179eeeaa6982df/ubuntu/jammy/flavors/default jammy/main amd64 librados2 amd64 19.3.0-4938-g33f95d2a-1jammy [3,634 kB] 2024-09-16T15:38:59.997 INFO:teuthology.orchestra.run.smithi184.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyperclip all 1.8.2-2 [11.5 kB] 2024-09-16T15:39:00.012 INFO:teuthology.orchestra.run.smithi123.stdout:0 upgraded, 125 newly installed, 0 to remove and 331 not upgraded. 2024-09-16T15:39:00.012 INFO:teuthology.orchestra.run.smithi123.stdout:Need to get 244 MB of archives. 2024-09-16T15:39:00.012 INFO:teuthology.orchestra.run.smithi123.stdout:After this operation, 910 MB of additional disk space will be used. 2024-09-16T15:39:00.013 INFO:teuthology.orchestra.run.smithi123.stdout:Get:1 https://4.chacra.ceph.com/r/ceph/wip-athakkar-testing-2024-09-16-1454/33f95d2ae1bcbe7eb04404303f179eeeaa6982df/ubuntu/jammy/flavors/default jammy/main amd64 librados2 amd64 19.3.0-4938-g33f95d2a-1jammy [3,634 kB] 2024-09-16T15:39:00.058 INFO:teuthology.orchestra.run.smithi123.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyperclip all 1.8.2-2 [11.5 kB] 2024-09-16T15:39:00.079 INFO:teuthology.orchestra.run.smithi184.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-wcwidth all 0.2.5+dfsg1-1 [21.9 kB] 2024-09-16T15:39:00.125 INFO:teuthology.orchestra.run.smithi184.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cmd2 all 2.3.3-1 [124 kB] 2024-09-16T15:39:00.141 INFO:teuthology.orchestra.run.smithi123.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-wcwidth all 0.2.5+dfsg1-1 [21.9 kB] 2024-09-16T15:39:00.187 INFO:teuthology.orchestra.run.smithi123.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cmd2 all 2.3.3-1 [124 kB] 2024-09-16T15:39:00.201 INFO:teuthology.orchestra.run.smithi184.stdout:Get:5 https://4.chacra.ceph.com/r/ceph/wip-athakkar-testing-2024-09-16-1454/33f95d2ae1bcbe7eb04404303f179eeeaa6982df/ubuntu/jammy/flavors/default jammy/main amd64 librbd1 amd64 19.3.0-4938-g33f95d2a-1jammy [3,334 kB] 2024-09-16T15:39:00.211 INFO:teuthology.orchestra.run.smithi123.stdout:Get:5 https://4.chacra.ceph.com/r/ceph/wip-athakkar-testing-2024-09-16-1454/33f95d2ae1bcbe7eb04404303f179eeeaa6982df/ubuntu/jammy/flavors/default jammy/main amd64 librbd1 amd64 19.3.0-4938-g33f95d2a-1jammy [3,334 kB] 2024-09-16T15:39:00.310 INFO:teuthology.orchestra.run.smithi184.stdout:Get:6 https://4.chacra.ceph.com/r/ceph/wip-athakkar-testing-2024-09-16-1454/33f95d2ae1bcbe7eb04404303f179eeeaa6982df/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs2 amd64 19.3.0-4938-g33f95d2a-1jammy [722 kB] 2024-09-16T15:39:00.311 INFO:teuthology.orchestra.run.smithi123.stdout:Get:6 https://4.chacra.ceph.com/r/ceph/wip-athakkar-testing-2024-09-16-1454/33f95d2ae1bcbe7eb04404303f179eeeaa6982df/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs2 amd64 19.3.0-4938-g33f95d2a-1jammy [722 kB] 2024-09-16T15:39:00.321 INFO:teuthology.orchestra.run.smithi184.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblttng-ust1 amd64 2.13.1-1ubuntu1 [190 kB] 2024-09-16T15:39:00.325 INFO:teuthology.orchestra.run.smithi184.stdout:Get:8 https://4.chacra.ceph.com/r/ceph/wip-athakkar-testing-2024-09-16-1454/33f95d2ae1bcbe7eb04404303f179eeeaa6982df/ubuntu/jammy/flavors/default jammy/main amd64 python3-rados amd64 19.3.0-4938-g33f95d2a-1jammy [356 kB] 2024-09-16T15:39:00.339 INFO:teuthology.orchestra.run.smithi123.stdout:Get:7 https://4.chacra.ceph.com/r/ceph/wip-athakkar-testing-2024-09-16-1454/33f95d2ae1bcbe7eb04404303f179eeeaa6982df/ubuntu/jammy/flavors/default jammy/main amd64 python3-rados amd64 19.3.0-4938-g33f95d2a-1jammy [356 kB] 2024-09-16T15:39:00.345 INFO:teuthology.orchestra.run.smithi184.stdout:Get:9 https://4.chacra.ceph.com/r/ceph/wip-athakkar-testing-2024-09-16-1454/33f95d2ae1bcbe7eb04404303f179eeeaa6982df/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-argparse all 19.3.0-4938-g33f95d2a-1jammy [32.8 kB] 2024-09-16T15:39:00.346 INFO:teuthology.orchestra.run.smithi123.stdout:Get:8 https://4.chacra.ceph.com/r/ceph/wip-athakkar-testing-2024-09-16-1454/33f95d2ae1bcbe7eb04404303f179eeeaa6982df/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-argparse all 19.3.0-4938-g33f95d2a-1jammy [32.8 kB] 2024-09-16T15:39:00.363 INFO:teuthology.orchestra.run.smithi123.stdout:Get:9 https://4.chacra.ceph.com/r/ceph/wip-athakkar-testing-2024-09-16-1454/33f95d2ae1bcbe7eb04404303f179eeeaa6982df/ubuntu/jammy/flavors/default jammy/main amd64 python3-cephfs amd64 19.3.0-4938-g33f95d2a-1jammy [180 kB] 2024-09-16T15:39:00.364 INFO:teuthology.orchestra.run.smithi184.stdout:Get:10 https://4.chacra.ceph.com/r/ceph/wip-athakkar-testing-2024-09-16-1454/33f95d2ae1bcbe7eb04404303f179eeeaa6982df/ubuntu/jammy/flavors/default jammy/main amd64 python3-cephfs amd64 19.3.0-4938-g33f95d2a-1jammy [180 kB] 2024-09-16T15:39:00.382 INFO:teuthology.orchestra.run.smithi123.stdout:Get:10 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblttng-ust1 amd64 2.13.1-1ubuntu1 [190 kB] 2024-09-16T15:39:00.403 INFO:teuthology.orchestra.run.smithi123.stdout:Get:11 https://4.chacra.ceph.com/r/ceph/wip-athakkar-testing-2024-09-16-1454/33f95d2ae1bcbe7eb04404303f179eeeaa6982df/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-common all 19.3.0-4938-g33f95d2a-1jammy [72.8 kB] 2024-09-16T15:39:00.404 INFO:teuthology.orchestra.run.smithi184.stdout:Get:11 https://4.chacra.ceph.com/r/ceph/wip-athakkar-testing-2024-09-16-1454/33f95d2ae1bcbe7eb04404303f179eeeaa6982df/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-common all 19.3.0-4938-g33f95d2a-1jammy [72.8 kB] 2024-09-16T15:39:00.411 INFO:teuthology.orchestra.run.smithi123.stdout:Get:12 https://4.chacra.ceph.com/r/ceph/wip-athakkar-testing-2024-09-16-1454/33f95d2ae1bcbe7eb04404303f179eeeaa6982df/ubuntu/jammy/flavors/default jammy/main amd64 python3-rbd amd64 19.3.0-4938-g33f95d2a-1jammy [339 kB] 2024-09-16T15:39:00.411 INFO:teuthology.orchestra.run.smithi184.stdout:Get:12 https://4.chacra.ceph.com/r/ceph/wip-athakkar-testing-2024-09-16-1454/33f95d2ae1bcbe7eb04404303f179eeeaa6982df/ubuntu/jammy/flavors/default jammy/main amd64 python3-rbd amd64 19.3.0-4938-g33f95d2a-1jammy [339 kB] 2024-09-16T15:39:00.413 INFO:teuthology.orchestra.run.smithi184.stdout:Get:13 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libdouble-conversion3 amd64 3.1.7-4 [39.0 kB] 2024-09-16T15:39:00.425 INFO:teuthology.orchestra.run.smithi184.stdout:Get:14 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libpcre2-16-0 amd64 10.39-3ubuntu0.1 [203 kB] 2024-09-16T15:39:00.444 INFO:teuthology.orchestra.run.smithi123.stdout:Get:13 https://4.chacra.ceph.com/r/ceph/wip-athakkar-testing-2024-09-16-1454/33f95d2ae1bcbe7eb04404303f179eeeaa6982df/ubuntu/jammy/flavors/default jammy/main amd64 librgw2 amd64 19.3.0-4938-g33f95d2a-1jammy [7,966 kB] 2024-09-16T15:39:00.444 INFO:teuthology.orchestra.run.smithi184.stdout:Get:15 https://4.chacra.ceph.com/r/ceph/wip-athakkar-testing-2024-09-16-1454/33f95d2ae1bcbe7eb04404303f179eeeaa6982df/ubuntu/jammy/flavors/default jammy/main amd64 librgw2 amd64 19.3.0-4938-g33f95d2a-1jammy [7,966 kB] 2024-09-16T15:39:00.474 INFO:teuthology.orchestra.run.smithi123.stdout:Get:14 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libdouble-conversion3 amd64 3.1.7-4 [39.0 kB] 2024-09-16T15:39:00.475 INFO:teuthology.orchestra.run.smithi184.stdout:Get:16 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5core5a amd64 5.15.3+dfsg-2ubuntu0.2 [2,006 kB] 2024-09-16T15:39:00.486 INFO:teuthology.orchestra.run.smithi123.stdout:Get:15 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libpcre2-16-0 amd64 10.39-3ubuntu0.1 [203 kB] 2024-09-16T15:39:00.536 INFO:teuthology.orchestra.run.smithi123.stdout:Get:16 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5core5a amd64 5.15.3+dfsg-2ubuntu0.2 [2,006 kB] 2024-09-16T15:39:00.649 INFO:teuthology.orchestra.run.smithi184.stdout:Get:17 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5dbus5 amd64 5.15.3+dfsg-2ubuntu0.2 [222 kB] 2024-09-16T15:39:00.657 INFO:teuthology.orchestra.run.smithi184.stdout:Get:18 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5network5 amd64 5.15.3+dfsg-2ubuntu0.2 [731 kB] 2024-09-16T15:39:00.685 INFO:teuthology.orchestra.run.smithi184.stdout:Get:19 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libthrift-0.16.0 amd64 0.16.0-2 [267 kB] 2024-09-16T15:39:00.695 INFO:teuthology.orchestra.run.smithi184.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libnbd0 amd64 1.10.5-1 [71.3 kB] 2024-09-16T15:39:00.697 INFO:teuthology.orchestra.run.smithi184.stdout:Get:21 https://4.chacra.ceph.com/r/ceph/wip-athakkar-testing-2024-09-16-1454/33f95d2ae1bcbe7eb04404303f179eeeaa6982df/ubuntu/jammy/flavors/default jammy/main amd64 python3-rgw amd64 19.3.0-4938-g33f95d2a-1jammy [112 kB] 2024-09-16T15:39:00.698 INFO:teuthology.orchestra.run.smithi184.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-prettytable all 2.5.0-2 [31.3 kB] 2024-09-16T15:39:00.700 INFO:teuthology.orchestra.run.smithi184.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy/universe amd64 librdkafka1 amd64 1.8.0-1build1 [633 kB] 2024-09-16T15:39:00.704 INFO:teuthology.orchestra.run.smithi184.stdout:Get:24 https://4.chacra.ceph.com/r/ceph/wip-athakkar-testing-2024-09-16-1454/33f95d2ae1bcbe7eb04404303f179eeeaa6982df/ubuntu/jammy/flavors/default jammy/main amd64 libradosstriper1 amd64 19.3.0-4938-g33f95d2a-1jammy [473 kB] 2024-09-16T15:39:00.711 INFO:teuthology.orchestra.run.smithi123.stdout:Get:17 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5dbus5 amd64 5.15.3+dfsg-2ubuntu0.2 [222 kB] 2024-09-16T15:39:00.716 INFO:teuthology.orchestra.run.smithi184.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy/main amd64 libreadline-dev amd64 8.1.2-1 [166 kB] 2024-09-16T15:39:00.720 INFO:teuthology.orchestra.run.smithi184.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblua5.3-dev amd64 5.3.6-1build1 [167 kB] 2024-09-16T15:39:00.720 INFO:teuthology.orchestra.run.smithi123.stdout:Get:18 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5network5 amd64 5.15.3+dfsg-2ubuntu0.2 [731 kB] 2024-09-16T15:39:00.723 INFO:teuthology.orchestra.run.smithi123.stdout:Get:19 https://4.chacra.ceph.com/r/ceph/wip-athakkar-testing-2024-09-16-1454/33f95d2ae1bcbe7eb04404303f179eeeaa6982df/ubuntu/jammy/flavors/default jammy/main amd64 python3-rgw amd64 19.3.0-4938-g33f95d2a-1jammy [112 kB] 2024-09-16T15:39:00.724 INFO:teuthology.orchestra.run.smithi184.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua5.1 amd64 5.1.5-8.1build4 [94.6 kB] 2024-09-16T15:39:00.725 INFO:teuthology.orchestra.run.smithi123.stdout:Get:20 https://4.chacra.ceph.com/r/ceph/wip-athakkar-testing-2024-09-16-1454/33f95d2ae1bcbe7eb04404303f179eeeaa6982df/ubuntu/jammy/flavors/default jammy/main amd64 libradosstriper1 amd64 19.3.0-4938-g33f95d2a-1jammy [473 kB] 2024-09-16T15:39:00.729 INFO:teuthology.orchestra.run.smithi184.stdout:Get:28 https://4.chacra.ceph.com/r/ceph/wip-athakkar-testing-2024-09-16-1454/33f95d2ae1bcbe7eb04404303f179eeeaa6982df/ubuntu/jammy/flavors/default jammy/main amd64 ceph-common amd64 19.3.0-4938-g33f95d2a-1jammy [28.2 MB] 2024-09-16T15:39:00.731 INFO:teuthology.orchestra.run.smithi184.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-any all 27ubuntu1 [5,034 B] 2024-09-16T15:39:00.735 INFO:teuthology.orchestra.run.smithi123.stdout:Get:21 https://4.chacra.ceph.com/r/ceph/wip-athakkar-testing-2024-09-16-1454/33f95d2ae1bcbe7eb04404303f179eeeaa6982df/ubuntu/jammy/flavors/default jammy/main amd64 ceph-common amd64 19.3.0-4938-g33f95d2a-1jammy [28.2 MB] 2024-09-16T15:39:00.747 INFO:teuthology.orchestra.run.smithi123.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libthrift-0.16.0 amd64 0.16.0-2 [267 kB] 2024-09-16T15:39:00.755 INFO:teuthology.orchestra.run.smithi123.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libnbd0 amd64 1.10.5-1 [71.3 kB] 2024-09-16T15:39:00.758 INFO:teuthology.orchestra.run.smithi123.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-prettytable all 2.5.0-2 [31.3 kB] 2024-09-16T15:39:00.759 INFO:teuthology.orchestra.run.smithi123.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy/universe amd64 librdkafka1 amd64 1.8.0-1build1 [633 kB] 2024-09-16T15:39:00.777 INFO:teuthology.orchestra.run.smithi123.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy/main amd64 libreadline-dev amd64 8.1.2-1 [166 kB] 2024-09-16T15:39:00.782 INFO:teuthology.orchestra.run.smithi123.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblua5.3-dev amd64 5.3.6-1build1 [167 kB] 2024-09-16T15:39:00.785 INFO:teuthology.orchestra.run.smithi123.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua5.1 amd64 5.1.5-8.1build4 [94.6 kB] 2024-09-16T15:39:00.794 INFO:teuthology.orchestra.run.smithi123.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-any all 27ubuntu1 [5,034 B] 2024-09-16T15:39:00.808 INFO:teuthology.orchestra.run.smithi184.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy/main amd64 zip amd64 3.0-12build2 [176 kB] 2024-09-16T15:39:00.813 INFO:teuthology.orchestra.run.smithi184.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 unzip amd64 6.0-26ubuntu3.2 [175 kB] 2024-09-16T15:39:00.817 INFO:teuthology.orchestra.run.smithi184.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy/universe amd64 luarocks all 3.8.0+dfsg1-1 [140 kB] 2024-09-16T15:39:00.820 INFO:teuthology.orchestra.run.smithi184.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy/main amd64 liboath0 amd64 2.6.7-3build1 [41.1 kB] 2024-09-16T15:39:00.822 INFO:teuthology.orchestra.run.smithi184.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.functools all 3.4.0-2 [9,030 B] 2024-09-16T15:39:00.823 INFO:teuthology.orchestra.run.smithi184.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-cheroot all 8.5.2+ds1-1ubuntu3.1 [71.1 kB] 2024-09-16T15:39:00.825 INFO:teuthology.orchestra.run.smithi184.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.classes all 3.2.1-3 [6,452 B] 2024-09-16T15:39:00.826 INFO:teuthology.orchestra.run.smithi184.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.text all 3.6.0-2 [8,716 B] 2024-09-16T15:39:00.826 INFO:teuthology.orchestra.run.smithi184.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.collections all 3.4.0-2 [11.4 kB] 2024-09-16T15:39:00.870 INFO:teuthology.orchestra.run.smithi123.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy/main amd64 zip amd64 3.0-12build2 [176 kB] 2024-09-16T15:39:00.873 INFO:teuthology.orchestra.run.smithi123.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 unzip amd64 6.0-26ubuntu3.2 [175 kB] 2024-09-16T15:39:00.878 INFO:teuthology.orchestra.run.smithi123.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy/universe amd64 luarocks all 3.8.0+dfsg1-1 [140 kB] 2024-09-16T15:39:00.881 INFO:teuthology.orchestra.run.smithi123.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy/main amd64 liboath0 amd64 2.6.7-3build1 [41.1 kB] 2024-09-16T15:39:00.882 INFO:teuthology.orchestra.run.smithi123.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.functools all 3.4.0-2 [9,030 B] 2024-09-16T15:39:00.883 INFO:teuthology.orchestra.run.smithi123.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-cheroot all 8.5.2+ds1-1ubuntu3.1 [71.1 kB] 2024-09-16T15:39:00.885 INFO:teuthology.orchestra.run.smithi123.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.classes all 3.2.1-3 [6,452 B] 2024-09-16T15:39:00.885 INFO:teuthology.orchestra.run.smithi123.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.text all 3.6.0-2 [8,716 B] 2024-09-16T15:39:00.886 INFO:teuthology.orchestra.run.smithi123.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.collections all 3.4.0-2 [11.4 kB] 2024-09-16T15:39:00.894 INFO:teuthology.orchestra.run.smithi184.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempora all 4.1.2-1 [14.8 kB] 2024-09-16T15:39:00.909 INFO:teuthology.orchestra.run.smithi184.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-portend all 3.0.0-1 [7,240 B] 2024-09-16T15:39:00.910 INFO:teuthology.orchestra.run.smithi184.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-zc.lockfile all 2.0-1 [8,980 B] 2024-09-16T15:39:00.911 INFO:teuthology.orchestra.run.smithi184.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cherrypy3 all 18.6.1-4 [208 kB] 2024-09-16T15:39:00.916 INFO:teuthology.orchestra.run.smithi184.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-natsort all 8.0.2-1 [35.3 kB] 2024-09-16T15:39:00.918 INFO:teuthology.orchestra.run.smithi184.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-logutils all 0.3.3-8 [17.6 kB] 2024-09-16T15:39:00.957 INFO:teuthology.orchestra.run.smithi123.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempora all 4.1.2-1 [14.8 kB] 2024-09-16T15:39:00.968 INFO:teuthology.orchestra.run.smithi123.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-portend all 3.0.0-1 [7,240 B] 2024-09-16T15:39:00.969 INFO:teuthology.orchestra.run.smithi123.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-zc.lockfile all 2.0-1 [8,980 B] 2024-09-16T15:39:00.969 INFO:teuthology.orchestra.run.smithi123.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cherrypy3 all 18.6.1-4 [208 kB] 2024-09-16T15:39:00.975 INFO:teuthology.orchestra.run.smithi123.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-natsort all 8.0.2-1 [35.3 kB] 2024-09-16T15:39:00.976 INFO:teuthology.orchestra.run.smithi123.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-logutils all 0.3.3-8 [17.6 kB] 2024-09-16T15:39:00.976 INFO:teuthology.orchestra.run.smithi184.stdout:Get:45 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-markupsafe amd64 2.0.1-2build1 [12.7 kB] 2024-09-16T15:39:00.977 INFO:teuthology.orchestra.run.smithi184.stdout:Get:46 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-mako all 1.1.3+ds1-2ubuntu0.1 [60.5 kB] 2024-09-16T15:39:00.979 INFO:teuthology.orchestra.run.smithi184.stdout:Get:47 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplegeneric all 0.8.1-3 [11.3 kB] 2024-09-16T15:39:00.980 INFO:teuthology.orchestra.run.smithi184.stdout:Get:48 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-singledispatch all 3.4.0.3-3 [7,320 B] 2024-09-16T15:39:01.038 INFO:teuthology.orchestra.run.smithi123.stdout:Get:45 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-markupsafe amd64 2.0.1-2build1 [12.7 kB] 2024-09-16T15:39:01.039 INFO:teuthology.orchestra.run.smithi123.stdout:Get:46 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-mako all 1.1.3+ds1-2ubuntu0.1 [60.5 kB] 2024-09-16T15:39:01.041 INFO:teuthology.orchestra.run.smithi123.stdout:Get:47 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplegeneric all 0.8.1-3 [11.3 kB] 2024-09-16T15:39:01.042 INFO:teuthology.orchestra.run.smithi123.stdout:Get:48 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-singledispatch all 3.4.0.3-3 [7,320 B] 2024-09-16T15:39:01.057 INFO:teuthology.orchestra.run.smithi184.stdout:Get:49 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-webob all 1:1.8.6-1.1ubuntu0.1 [86.7 kB] 2024-09-16T15:39:01.060 INFO:teuthology.orchestra.run.smithi184.stdout:Get:50 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-waitress all 1.4.4-1.1ubuntu1 [47.3 kB] 2024-09-16T15:39:01.061 INFO:teuthology.orchestra.run.smithi184.stdout:Get:51 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempita all 0.5.2-6ubuntu1 [15.1 kB] 2024-09-16T15:39:01.062 INFO:teuthology.orchestra.run.smithi184.stdout:Get:52 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-paste all 3.5.0+dfsg1-1 [456 kB] 2024-09-16T15:39:01.073 INFO:teuthology.orchestra.run.smithi184.stdout:Get:53 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-pastedeploy-tpl all 2.1.1-1 [4,892 B] 2024-09-16T15:39:01.119 INFO:teuthology.orchestra.run.smithi123.stdout:Get:49 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-webob all 1:1.8.6-1.1ubuntu0.1 [86.7 kB] 2024-09-16T15:39:01.122 INFO:teuthology.orchestra.run.smithi123.stdout:Get:50 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-waitress all 1.4.4-1.1ubuntu1 [47.3 kB] 2024-09-16T15:39:01.123 INFO:teuthology.orchestra.run.smithi123.stdout:Get:51 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempita all 0.5.2-6ubuntu1 [15.1 kB] 2024-09-16T15:39:01.124 INFO:teuthology.orchestra.run.smithi123.stdout:Get:52 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-paste all 3.5.0+dfsg1-1 [456 kB] 2024-09-16T15:39:01.135 INFO:teuthology.orchestra.run.smithi123.stdout:Get:53 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-pastedeploy-tpl all 2.1.1-1 [4,892 B] 2024-09-16T15:39:01.138 INFO:teuthology.orchestra.run.smithi184.stdout:Get:54 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastedeploy all 2.1.1-1 [26.6 kB] 2024-09-16T15:39:01.139 INFO:teuthology.orchestra.run.smithi184.stdout:Get:55 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webtest all 2.0.35-1 [28.5 kB] 2024-09-16T15:39:01.141 INFO:teuthology.orchestra.run.smithi184.stdout:Get:56 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pecan all 1.3.3-4ubuntu2 [87.3 kB] 2024-09-16T15:39:01.143 INFO:teuthology.orchestra.run.smithi184.stdout:Get:57 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-certifi all 2020.6.20-1 [150 kB] 2024-09-16T15:39:01.148 INFO:teuthology.orchestra.run.smithi184.stdout:Get:58 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-idna all 3.3-1ubuntu0.1 [52.1 kB] 2024-09-16T15:39:01.201 INFO:teuthology.orchestra.run.smithi123.stdout:Get:54 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastedeploy all 2.1.1-1 [26.6 kB] 2024-09-16T15:39:01.202 INFO:teuthology.orchestra.run.smithi123.stdout:Get:55 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webtest all 2.0.35-1 [28.5 kB] 2024-09-16T15:39:01.202 INFO:teuthology.orchestra.run.smithi123.stdout:Get:56 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pecan all 1.3.3-4ubuntu2 [87.3 kB] 2024-09-16T15:39:01.205 INFO:teuthology.orchestra.run.smithi123.stdout:Get:57 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-certifi all 2020.6.20-1 [150 kB] 2024-09-16T15:39:01.209 INFO:teuthology.orchestra.run.smithi123.stdout:Get:58 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-idna all 3.3-1ubuntu0.1 [52.1 kB] 2024-09-16T15:39:01.220 INFO:teuthology.orchestra.run.smithi184.stdout:Get:59 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-urllib3 all 1.26.5-1~exp1ubuntu0.1 [98.2 kB] 2024-09-16T15:39:01.222 INFO:teuthology.orchestra.run.smithi184.stdout:Get:60 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-requests all 2.25.1+dfsg-2ubuntu0.1 [48.8 kB] 2024-09-16T15:39:01.224 INFO:teuthology.orchestra.run.smithi184.stdout:Get:61 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-werkzeug all 2.0.2+dfsg1-1ubuntu0.22.04.2 [181 kB] 2024-09-16T15:39:01.229 INFO:teuthology.orchestra.run.smithi184.stdout:Get:62 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-bcrypt amd64 3.2.0-1build1 [32.7 kB] 2024-09-16T15:39:01.230 INFO:teuthology.orchestra.run.smithi184.stdout:Get:63 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-openssl all 21.0.0-1 [45.2 kB] 2024-09-16T15:39:01.282 INFO:teuthology.orchestra.run.smithi123.stdout:Get:59 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-urllib3 all 1.26.5-1~exp1ubuntu0.1 [98.2 kB] 2024-09-16T15:39:01.284 INFO:teuthology.orchestra.run.smithi123.stdout:Get:60 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-requests all 2.25.1+dfsg-2ubuntu0.1 [48.8 kB] 2024-09-16T15:39:01.286 INFO:teuthology.orchestra.run.smithi123.stdout:Get:61 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-werkzeug all 2.0.2+dfsg1-1ubuntu0.22.04.2 [181 kB] 2024-09-16T15:39:01.290 INFO:teuthology.orchestra.run.smithi123.stdout:Get:62 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-bcrypt amd64 3.2.0-1build1 [32.7 kB] 2024-09-16T15:39:01.291 INFO:teuthology.orchestra.run.smithi123.stdout:Get:63 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-openssl all 21.0.0-1 [45.2 kB] 2024-09-16T15:39:01.301 INFO:teuthology.orchestra.run.smithi184.stdout:Get:64 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-asyncssh all 2.5.0-1 [185 kB] 2024-09-16T15:39:01.306 INFO:teuthology.orchestra.run.smithi184.stdout:Get:65 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-babel-localedata all 2.8.0+dfsg.1-7 [4,982 kB] 2024-09-16T15:39:01.363 INFO:teuthology.orchestra.run.smithi123.stdout:Get:64 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-asyncssh all 2.5.0-1 [185 kB] 2024-09-16T15:39:01.368 INFO:teuthology.orchestra.run.smithi123.stdout:Get:65 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-babel-localedata all 2.8.0+dfsg.1-7 [4,982 kB] 2024-09-16T15:39:01.470 INFO:teuthology.orchestra.run.smithi184.stdout:Get:66 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-babel all 2.8.0+dfsg.1-7 [85.1 kB] 2024-09-16T15:39:01.472 INFO:teuthology.orchestra.run.smithi184.stdout:Get:67 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-jinja2 all 3.0.3-1ubuntu0.2 [108 kB] 2024-09-16T15:39:01.475 INFO:teuthology.orchestra.run.smithi184.stdout:Get:68 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-repoze.lru all 0.7-2 [12.1 kB] 2024-09-16T15:39:01.476 INFO:teuthology.orchestra.run.smithi184.stdout:Get:69 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-routes all 2.5.1-1ubuntu1 [89.0 kB] 2024-09-16T15:39:01.478 INFO:teuthology.orchestra.run.smithi184.stdout:Get:70 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn-lib amd64 0.23.2-5ubuntu6 [2,058 kB] 2024-09-16T15:39:01.514 INFO:teuthology.orchestra.run.smithi184.stdout:Get:71 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-joblib all 0.17.0-4ubuntu1 [204 kB] 2024-09-16T15:39:01.517 INFO:teuthology.orchestra.run.smithi184.stdout:Get:72 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-threadpoolctl all 3.1.0-1 [21.3 kB] 2024-09-16T15:39:01.518 INFO:teuthology.orchestra.run.smithi184.stdout:Get:73 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn all 0.23.2-5ubuntu6 [1,829 kB] 2024-09-16T15:39:01.532 INFO:teuthology.orchestra.run.smithi123.stdout:Get:66 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-babel all 2.8.0+dfsg.1-7 [85.1 kB] 2024-09-16T15:39:01.534 INFO:teuthology.orchestra.run.smithi123.stdout:Get:67 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-jinja2 all 3.0.3-1ubuntu0.2 [108 kB] 2024-09-16T15:39:01.536 INFO:teuthology.orchestra.run.smithi123.stdout:Get:68 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-repoze.lru all 0.7-2 [12.1 kB] 2024-09-16T15:39:01.536 INFO:teuthology.orchestra.run.smithi123.stdout:Get:69 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-routes all 2.5.1-1ubuntu1 [89.0 kB] 2024-09-16T15:39:01.538 INFO:teuthology.orchestra.run.smithi123.stdout:Get:70 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn-lib amd64 0.23.2-5ubuntu6 [2,058 kB] 2024-09-16T15:39:01.574 INFO:teuthology.orchestra.run.smithi123.stdout:Get:71 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-joblib all 0.17.0-4ubuntu1 [204 kB] 2024-09-16T15:39:01.577 INFO:teuthology.orchestra.run.smithi123.stdout:Get:72 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-threadpoolctl all 3.1.0-1 [21.3 kB] 2024-09-16T15:39:01.577 INFO:teuthology.orchestra.run.smithi123.stdout:Get:73 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn all 0.23.2-5ubuntu6 [1,829 kB] 2024-09-16T15:39:01.581 INFO:teuthology.orchestra.run.smithi184.stdout:Get:74 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cachetools all 5.0.0-1 [9,722 B] 2024-09-16T15:39:01.581 INFO:teuthology.orchestra.run.smithi184.stdout:Get:75 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1 all 0.4.8-1 [50.9 kB] 2024-09-16T15:39:01.640 INFO:teuthology.orchestra.run.smithi123.stdout:Get:74 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cachetools all 5.0.0-1 [9,722 B] 2024-09-16T15:39:01.640 INFO:teuthology.orchestra.run.smithi123.stdout:Get:75 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1 all 0.4.8-1 [50.9 kB] 2024-09-16T15:39:01.641 INFO:teuthology.orchestra.run.smithi184.stdout:Get:76 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1-modules all 0.2.1-1 [33.0 kB] 2024-09-16T15:39:01.642 INFO:teuthology.orchestra.run.smithi184.stdout:Get:77 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-rsa all 4.8-1 [28.4 kB] 2024-09-16T15:39:01.643 INFO:teuthology.orchestra.run.smithi184.stdout:Get:78 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-google-auth all 1.5.1-3 [35.7 kB] 2024-09-16T15:39:01.644 INFO:teuthology.orchestra.run.smithi184.stdout:Get:79 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-requests-oauthlib all 1.3.0+ds-0.1 [18.7 kB] 2024-09-16T15:39:01.645 INFO:teuthology.orchestra.run.smithi184.stdout:Get:80 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-websocket all 1.2.3-1 [34.7 kB] 2024-09-16T15:39:01.646 INFO:teuthology.orchestra.run.smithi184.stdout:Get:81 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-kubernetes all 12.0.1-1ubuntu1 [353 kB] 2024-09-16T15:39:01.653 INFO:teuthology.orchestra.run.smithi184.stdout:Get:82 http://archive.ubuntu.com/ubuntu jammy/main amd64 libonig5 amd64 6.9.7.1-2build1 [172 kB] 2024-09-16T15:39:01.700 INFO:teuthology.orchestra.run.smithi123.stdout:Get:76 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1-modules all 0.2.1-1 [33.0 kB] 2024-09-16T15:39:01.721 INFO:teuthology.orchestra.run.smithi123.stdout:Get:77 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-rsa all 4.8-1 [28.4 kB] 2024-09-16T15:39:01.722 INFO:teuthology.orchestra.run.smithi123.stdout:Get:78 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-google-auth all 1.5.1-3 [35.7 kB] 2024-09-16T15:39:01.722 INFO:teuthology.orchestra.run.smithi184.stdout:Get:83 http://archive.ubuntu.com/ubuntu jammy/main amd64 libjq1 amd64 1.6-2.1ubuntu3 [133 kB] 2024-09-16T15:39:01.736 INFO:teuthology.orchestra.run.smithi184.stdout:Get:84 http://archive.ubuntu.com/ubuntu jammy/main amd64 jq amd64 1.6-2.1ubuntu3 [52.5 kB] 2024-09-16T15:39:01.737 INFO:teuthology.orchestra.run.smithi184.stdout:Get:85 http://archive.ubuntu.com/ubuntu jammy/main amd64 socat amd64 1.7.4.1-3ubuntu4 [349 kB] 2024-09-16T15:39:01.780 INFO:teuthology.orchestra.run.smithi123.stdout:Get:79 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-requests-oauthlib all 1.3.0+ds-0.1 [18.7 kB] 2024-09-16T15:39:01.781 INFO:teuthology.orchestra.run.smithi123.stdout:Get:80 https://4.chacra.ceph.com/r/ceph/wip-athakkar-testing-2024-09-16-1454/33f95d2ae1bcbe7eb04404303f179eeeaa6982df/ubuntu/jammy/flavors/default jammy/main amd64 ceph-base amd64 19.3.0-4938-g33f95d2a-1jammy [5,206 kB] 2024-09-16T15:39:01.781 INFO:teuthology.orchestra.run.smithi123.stdout:Get:81 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-websocket all 1.2.3-1 [34.7 kB] 2024-09-16T15:39:01.782 INFO:teuthology.orchestra.run.smithi123.stdout:Get:82 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-kubernetes all 12.0.1-1ubuntu1 [353 kB] 2024-09-16T15:39:01.791 INFO:teuthology.orchestra.run.smithi123.stdout:Get:83 http://archive.ubuntu.com/ubuntu jammy/main amd64 libonig5 amd64 6.9.7.1-2build1 [172 kB] 2024-09-16T15:39:01.795 INFO:teuthology.orchestra.run.smithi123.stdout:Get:84 http://archive.ubuntu.com/ubuntu jammy/main amd64 libjq1 amd64 1.6-2.1ubuntu3 [133 kB] 2024-09-16T15:39:01.798 INFO:teuthology.orchestra.run.smithi123.stdout:Get:85 http://archive.ubuntu.com/ubuntu jammy/main amd64 jq amd64 1.6-2.1ubuntu3 [52.5 kB] 2024-09-16T15:39:01.799 INFO:teuthology.orchestra.run.smithi123.stdout:Get:86 http://archive.ubuntu.com/ubuntu jammy/main amd64 socat amd64 1.7.4.1-3ubuntu4 [349 kB] 2024-09-16T15:39:01.819 INFO:teuthology.orchestra.run.smithi184.stdout:Get:86 http://archive.ubuntu.com/ubuntu jammy/universe amd64 xmlstarlet amd64 1.6.1-2.1 [265 kB] 2024-09-16T15:39:01.819 INFO:teuthology.orchestra.run.smithi184.stdout:Get:87 https://4.chacra.ceph.com/r/ceph/wip-athakkar-testing-2024-09-16-1454/33f95d2ae1bcbe7eb04404303f179eeeaa6982df/ubuntu/jammy/flavors/default jammy/main amd64 ceph-base amd64 19.3.0-4938-g33f95d2a-1jammy [5,206 kB] 2024-09-16T15:39:01.825 INFO:teuthology.orchestra.run.smithi184.stdout:Get:88 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-colorama all 0.4.4-1 [24.5 kB] 2024-09-16T15:39:01.827 INFO:teuthology.orchestra.run.smithi184.stdout:Get:89 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-socket amd64 3.0~rc1+git+ac3201d-6 [78.9 kB] 2024-09-16T15:39:01.829 INFO:teuthology.orchestra.run.smithi184.stdout:Get:90 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-sec amd64 1.0.2-1 [37.6 kB] 2024-09-16T15:39:01.831 INFO:teuthology.orchestra.run.smithi184.stdout:Get:91 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 nvme-cli amd64 1.16-3ubuntu0.3 [474 kB] 2024-09-16T15:39:01.842 INFO:teuthology.orchestra.run.smithi184.stdout:Get:92 http://archive.ubuntu.com/ubuntu jammy/main amd64 pkg-config amd64 0.29.2-1ubuntu3 [48.2 kB] 2024-09-16T15:39:01.843 INFO:teuthology.orchestra.run.smithi184.stdout:Get:93 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python-asyncssh-doc all 2.5.0-1 [318 kB] 2024-09-16T15:39:01.862 INFO:teuthology.orchestra.run.smithi123.stdout:Get:87 https://4.chacra.ceph.com/r/ceph/wip-athakkar-testing-2024-09-16-1454/33f95d2ae1bcbe7eb04404303f179eeeaa6982df/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-modules-core all 19.3.0-4938-g33f95d2a-1jammy [252 kB] 2024-09-16T15:39:01.863 INFO:teuthology.orchestra.run.smithi123.stdout:Get:88 http://archive.ubuntu.com/ubuntu jammy/universe amd64 xmlstarlet amd64 1.6.1-2.1 [265 kB] 2024-09-16T15:39:01.865 INFO:teuthology.orchestra.run.smithi123.stdout:Get:89 https://4.chacra.ceph.com/r/ceph/wip-athakkar-testing-2024-09-16-1454/33f95d2ae1bcbe7eb04404303f179eeeaa6982df/ubuntu/jammy/flavors/default jammy/main amd64 libsqlite3-mod-ceph amd64 19.3.0-4938-g33f95d2a-1jammy [124 kB] 2024-09-16T15:39:01.866 INFO:teuthology.orchestra.run.smithi123.stdout:Get:90 https://4.chacra.ceph.com/r/ceph/wip-athakkar-testing-2024-09-16-1454/33f95d2ae1bcbe7eb04404303f179eeeaa6982df/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr amd64 19.3.0-4938-g33f95d2a-1jammy [1,562 kB] 2024-09-16T15:39:01.869 INFO:teuthology.orchestra.run.smithi123.stdout:Get:91 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-colorama all 0.4.4-1 [24.5 kB] 2024-09-16T15:39:01.885 INFO:teuthology.orchestra.run.smithi184.stdout:Get:94 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-iniconfig all 1.1.1-2 [6,024 B] 2024-09-16T15:39:01.886 INFO:teuthology.orchestra.run.smithi184.stdout:Get:95 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastescript all 2.0.2-4 [54.6 kB] 2024-09-16T15:39:01.901 INFO:teuthology.orchestra.run.smithi184.stdout:Get:96 https://4.chacra.ceph.com/r/ceph/wip-athakkar-testing-2024-09-16-1454/33f95d2ae1bcbe7eb04404303f179eeeaa6982df/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-modules-core all 19.3.0-4938-g33f95d2a-1jammy [252 kB] 2024-09-16T15:39:01.904 INFO:teuthology.orchestra.run.smithi184.stdout:Get:97 https://4.chacra.ceph.com/r/ceph/wip-athakkar-testing-2024-09-16-1454/33f95d2ae1bcbe7eb04404303f179eeeaa6982df/ubuntu/jammy/flavors/default jammy/main amd64 libsqlite3-mod-ceph amd64 19.3.0-4938-g33f95d2a-1jammy [124 kB] 2024-09-16T15:39:01.905 INFO:teuthology.orchestra.run.smithi184.stdout:Get:98 https://4.chacra.ceph.com/r/ceph/wip-athakkar-testing-2024-09-16-1454/33f95d2ae1bcbe7eb04404303f179eeeaa6982df/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr amd64 19.3.0-4938-g33f95d2a-1jammy [1,562 kB] 2024-09-16T15:39:01.937 INFO:teuthology.orchestra.run.smithi184.stdout:Get:99 https://4.chacra.ceph.com/r/ceph/wip-athakkar-testing-2024-09-16-1454/33f95d2ae1bcbe7eb04404303f179eeeaa6982df/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mon amd64 19.3.0-4938-g33f95d2a-1jammy [6,310 kB] 2024-09-16T15:39:01.937 INFO:teuthology.orchestra.run.smithi123.stdout:Get:92 https://4.chacra.ceph.com/r/ceph/wip-athakkar-testing-2024-09-16-1454/33f95d2ae1bcbe7eb04404303f179eeeaa6982df/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mon amd64 19.3.0-4938-g33f95d2a-1jammy [6,310 kB] 2024-09-16T15:39:01.944 INFO:teuthology.orchestra.run.smithi123.stdout:Get:93 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-socket amd64 3.0~rc1+git+ac3201d-6 [78.9 kB] 2024-09-16T15:39:01.946 INFO:teuthology.orchestra.run.smithi123.stdout:Get:94 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-sec amd64 1.0.2-1 [37.6 kB] 2024-09-16T15:39:01.948 INFO:teuthology.orchestra.run.smithi123.stdout:Get:95 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 nvme-cli amd64 1.16-3ubuntu0.3 [474 kB] 2024-09-16T15:39:01.959 INFO:teuthology.orchestra.run.smithi123.stdout:Get:96 http://archive.ubuntu.com/ubuntu jammy/main amd64 pkg-config amd64 0.29.2-1ubuntu3 [48.2 kB] 2024-09-16T15:39:01.960 INFO:teuthology.orchestra.run.smithi123.stdout:Get:97 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python-asyncssh-doc all 2.5.0-1 [318 kB] 2024-09-16T15:39:01.966 INFO:teuthology.orchestra.run.smithi184.stdout:Get:100 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pluggy all 0.13.0-7.1 [19.0 kB] 2024-09-16T15:39:01.968 INFO:teuthology.orchestra.run.smithi184.stdout:Get:101 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-psutil amd64 5.9.0-1build1 [158 kB] 2024-09-16T15:39:01.968 INFO:teuthology.orchestra.run.smithi123.stdout:Get:98 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-iniconfig all 1.1.1-2 [6,024 B] 2024-09-16T15:39:01.968 INFO:teuthology.orchestra.run.smithi123.stdout:Get:99 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastescript all 2.0.2-4 [54.6 kB] 2024-09-16T15:39:01.970 INFO:teuthology.orchestra.run.smithi123.stdout:Get:100 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pluggy all 0.13.0-7.1 [19.0 kB] 2024-09-16T15:39:01.971 INFO:teuthology.orchestra.run.smithi184.stdout:Get:102 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-py all 1.10.0-1 [71.9 kB] 2024-09-16T15:39:01.973 INFO:teuthology.orchestra.run.smithi184.stdout:Get:103 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pygments all 2.11.2+dfsg-2 [750 kB] 2024-09-16T15:39:01.990 INFO:teuthology.orchestra.run.smithi184.stdout:Get:104 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyinotify all 0.9.6-1.3 [24.8 kB] 2024-09-16T15:39:01.991 INFO:teuthology.orchestra.run.smithi184.stdout:Get:105 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-toml all 0.10.2-1 [16.5 kB] 2024-09-16T15:39:01.992 INFO:teuthology.orchestra.run.smithi184.stdout:Get:106 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pytest all 6.2.5-1ubuntu2 [214 kB] 2024-09-16T15:39:01.997 INFO:teuthology.orchestra.run.smithi184.stdout:Get:107 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplejson amd64 3.17.6-1build1 [54.7 kB] 2024-09-16T15:39:02.025 INFO:teuthology.orchestra.run.smithi123.stdout:Get:101 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-psutil amd64 5.9.0-1build1 [158 kB] 2024-09-16T15:39:02.029 INFO:teuthology.orchestra.run.smithi123.stdout:Get:102 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-py all 1.10.0-1 [71.9 kB] 2024-09-16T15:39:02.048 INFO:teuthology.orchestra.run.smithi184.stdout:Get:108 http://archive.ubuntu.com/ubuntu jammy/universe amd64 qttranslations5-l10n all 5.15.3-1 [1,983 kB] 2024-09-16T15:39:02.107 INFO:teuthology.orchestra.run.smithi123.stdout:Get:103 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pygments all 2.11.2+dfsg-2 [750 kB] 2024-09-16T15:39:02.124 INFO:teuthology.orchestra.run.smithi123.stdout:Get:104 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyinotify all 0.9.6-1.3 [24.8 kB] 2024-09-16T15:39:02.125 INFO:teuthology.orchestra.run.smithi123.stdout:Get:105 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-toml all 0.10.2-1 [16.5 kB] 2024-09-16T15:39:02.125 INFO:teuthology.orchestra.run.smithi123.stdout:Get:106 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pytest all 6.2.5-1ubuntu2 [214 kB] 2024-09-16T15:39:02.130 INFO:teuthology.orchestra.run.smithi123.stdout:Get:107 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplejson amd64 3.17.6-1build1 [54.7 kB] 2024-09-16T15:39:02.132 INFO:teuthology.orchestra.run.smithi123.stdout:Get:108 http://archive.ubuntu.com/ubuntu jammy/universe amd64 qttranslations5-l10n all 5.15.3-1 [1,983 kB] 2024-09-16T15:39:02.141 INFO:teuthology.orchestra.run.smithi123.stdout:Get:109 https://4.chacra.ceph.com/r/ceph/wip-athakkar-testing-2024-09-16-1454/33f95d2ae1bcbe7eb04404303f179eeeaa6982df/ubuntu/jammy/flavors/default jammy/main amd64 ceph-osd amd64 19.3.0-4938-g33f95d2a-1jammy [23.1 MB] 2024-09-16T15:39:02.141 INFO:teuthology.orchestra.run.smithi184.stdout:Get:109 https://4.chacra.ceph.com/r/ceph/wip-athakkar-testing-2024-09-16-1454/33f95d2ae1bcbe7eb04404303f179eeeaa6982df/ubuntu/jammy/flavors/default jammy/main amd64 ceph-osd amd64 19.3.0-4938-g33f95d2a-1jammy [23.1 MB] 2024-09-16T15:39:02.999 INFO:teuthology.orchestra.run.smithi123.stdout:Get:110 https://4.chacra.ceph.com/r/ceph/wip-athakkar-testing-2024-09-16-1454/33f95d2ae1bcbe7eb04404303f179eeeaa6982df/ubuntu/jammy/flavors/default jammy/main amd64 ceph amd64 19.3.0-4938-g33f95d2a-1jammy [14.0 kB] 2024-09-16T15:39:03.000 INFO:teuthology.orchestra.run.smithi123.stdout:Get:111 https://4.chacra.ceph.com/r/ceph/wip-athakkar-testing-2024-09-16-1454/33f95d2ae1bcbe7eb04404303f179eeeaa6982df/ubuntu/jammy/flavors/default jammy/main amd64 ceph-fuse amd64 19.3.0-4938-g33f95d2a-1jammy [880 kB] 2024-09-16T15:39:03.017 INFO:teuthology.orchestra.run.smithi123.stdout:Get:112 https://4.chacra.ceph.com/r/ceph/wip-athakkar-testing-2024-09-16-1454/33f95d2ae1bcbe7eb04404303f179eeeaa6982df/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mds amd64 19.3.0-4938-g33f95d2a-1jammy [2,454 kB] 2024-09-16T15:39:03.028 INFO:teuthology.orchestra.run.smithi184.stdout:Get:110 https://4.chacra.ceph.com/r/ceph/wip-athakkar-testing-2024-09-16-1454/33f95d2ae1bcbe7eb04404303f179eeeaa6982df/ubuntu/jammy/flavors/default jammy/main amd64 ceph amd64 19.3.0-4938-g33f95d2a-1jammy [14.0 kB] 2024-09-16T15:39:03.029 INFO:teuthology.orchestra.run.smithi184.stdout:Get:111 https://4.chacra.ceph.com/r/ceph/wip-athakkar-testing-2024-09-16-1454/33f95d2ae1bcbe7eb04404303f179eeeaa6982df/ubuntu/jammy/flavors/default jammy/main amd64 ceph-fuse amd64 19.3.0-4938-g33f95d2a-1jammy [880 kB] 2024-09-16T15:39:03.046 INFO:teuthology.orchestra.run.smithi184.stdout:Get:112 https://4.chacra.ceph.com/r/ceph/wip-athakkar-testing-2024-09-16-1454/33f95d2ae1bcbe7eb04404303f179eeeaa6982df/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mds amd64 19.3.0-4938-g33f95d2a-1jammy [2,454 kB] 2024-09-16T15:39:03.060 INFO:teuthology.orchestra.run.smithi123.stdout:Get:113 https://4.chacra.ceph.com/r/ceph/wip-athakkar-testing-2024-09-16-1454/33f95d2ae1bcbe7eb04404303f179eeeaa6982df/ubuntu/jammy/flavors/default jammy/main amd64 cephadm amd64 19.3.0-4938-g33f95d2a-1jammy [813 kB] 2024-09-16T15:39:03.070 INFO:teuthology.orchestra.run.smithi123.stdout:Get:114 https://4.chacra.ceph.com/r/ceph/wip-athakkar-testing-2024-09-16-1454/33f95d2ae1bcbe7eb04404303f179eeeaa6982df/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-cephadm all 19.3.0-4938-g33f95d2a-1jammy [160 kB] 2024-09-16T15:39:03.072 INFO:teuthology.orchestra.run.smithi123.stdout:Get:115 https://4.chacra.ceph.com/r/ceph/wip-athakkar-testing-2024-09-16-1454/33f95d2ae1bcbe7eb04404303f179eeeaa6982df/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-dashboard all 19.3.0-4938-g33f95d2a-1jammy [61.4 MB] 2024-09-16T15:39:03.089 INFO:teuthology.orchestra.run.smithi184.stdout:Get:113 https://4.chacra.ceph.com/r/ceph/wip-athakkar-testing-2024-09-16-1454/33f95d2ae1bcbe7eb04404303f179eeeaa6982df/ubuntu/jammy/flavors/default jammy/main amd64 cephadm amd64 19.3.0-4938-g33f95d2a-1jammy [813 kB] 2024-09-16T15:39:03.099 INFO:teuthology.orchestra.run.smithi184.stdout:Get:114 https://4.chacra.ceph.com/r/ceph/wip-athakkar-testing-2024-09-16-1454/33f95d2ae1bcbe7eb04404303f179eeeaa6982df/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-cephadm all 19.3.0-4938-g33f95d2a-1jammy [160 kB] 2024-09-16T15:39:03.101 INFO:teuthology.orchestra.run.smithi184.stdout:Get:115 https://4.chacra.ceph.com/r/ceph/wip-athakkar-testing-2024-09-16-1454/33f95d2ae1bcbe7eb04404303f179eeeaa6982df/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-dashboard all 19.3.0-4938-g33f95d2a-1jammy [61.4 MB] 2024-09-16T15:39:05.371 INFO:teuthology.orchestra.run.smithi184.stdout:Get:116 https://4.chacra.ceph.com/r/ceph/wip-athakkar-testing-2024-09-16-1454/33f95d2ae1bcbe7eb04404303f179eeeaa6982df/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-diskprediction-local all 19.3.0-4938-g33f95d2a-1jammy [8,625 kB] 2024-09-16T15:39:05.402 INFO:teuthology.orchestra.run.smithi123.stdout:Get:116 https://4.chacra.ceph.com/r/ceph/wip-athakkar-testing-2024-09-16-1454/33f95d2ae1bcbe7eb04404303f179eeeaa6982df/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-diskprediction-local all 19.3.0-4938-g33f95d2a-1jammy [8,625 kB] 2024-09-16T15:39:05.510 INFO:teuthology.orchestra.run.smithi184.stdout:Get:117 https://4.chacra.ceph.com/r/ceph/wip-athakkar-testing-2024-09-16-1454/33f95d2ae1bcbe7eb04404303f179eeeaa6982df/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-k8sevents all 19.3.0-4938-g33f95d2a-1jammy [14.2 kB] 2024-09-16T15:39:05.511 INFO:teuthology.orchestra.run.smithi184.stdout:Get:118 https://4.chacra.ceph.com/r/ceph/wip-athakkar-testing-2024-09-16-1454/33f95d2ae1bcbe7eb04404303f179eeeaa6982df/ubuntu/jammy/flavors/default jammy/main amd64 ceph-test amd64 19.3.0-4938-g33f95d2a-1jammy [48.5 MB] 2024-09-16T15:39:05.586 INFO:teuthology.orchestra.run.smithi123.stdout:Get:117 https://4.chacra.ceph.com/r/ceph/wip-athakkar-testing-2024-09-16-1454/33f95d2ae1bcbe7eb04404303f179eeeaa6982df/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-k8sevents all 19.3.0-4938-g33f95d2a-1jammy [14.2 kB] 2024-09-16T15:39:05.587 INFO:teuthology.orchestra.run.smithi123.stdout:Get:118 https://4.chacra.ceph.com/r/ceph/wip-athakkar-testing-2024-09-16-1454/33f95d2ae1bcbe7eb04404303f179eeeaa6982df/ubuntu/jammy/flavors/default jammy/main amd64 ceph-test amd64 19.3.0-4938-g33f95d2a-1jammy [48.5 MB] 2024-09-16T15:39:07.354 INFO:teuthology.orchestra.run.smithi184.stdout:Get:119 https://4.chacra.ceph.com/r/ceph/wip-athakkar-testing-2024-09-16-1454/33f95d2ae1bcbe7eb04404303f179eeeaa6982df/ubuntu/jammy/flavors/default jammy/main amd64 ceph-volume all 19.3.0-4938-g33f95d2a-1jammy [130 kB] 2024-09-16T15:39:07.357 INFO:teuthology.orchestra.run.smithi184.stdout:Get:120 https://4.chacra.ceph.com/r/ceph/wip-athakkar-testing-2024-09-16-1454/33f95d2ae1bcbe7eb04404303f179eeeaa6982df/ubuntu/jammy/flavors/default jammy/main amd64 cephfs-mirror amd64 19.3.0-4938-g33f95d2a-1jammy [233 kB] 2024-09-16T15:39:07.362 INFO:teuthology.orchestra.run.smithi184.stdout:Get:121 https://4.chacra.ceph.com/r/ceph/wip-athakkar-testing-2024-09-16-1454/33f95d2ae1bcbe7eb04404303f179eeeaa6982df/ubuntu/jammy/flavors/default jammy/main amd64 cephfs-shell all 19.3.0-4938-g33f95d2a-1jammy [33.9 kB] 2024-09-16T15:39:07.363 INFO:teuthology.orchestra.run.smithi184.stdout:Get:122 https://4.chacra.ceph.com/r/ceph/wip-athakkar-testing-2024-09-16-1454/33f95d2ae1bcbe7eb04404303f179eeeaa6982df/ubuntu/jammy/flavors/default jammy/main amd64 cephfs-top all 19.3.0-4938-g33f95d2a-1jammy [23.4 kB] 2024-09-16T15:39:07.363 INFO:teuthology.orchestra.run.smithi184.stdout:Get:123 https://4.chacra.ceph.com/r/ceph/wip-athakkar-testing-2024-09-16-1454/33f95d2ae1bcbe7eb04404303f179eeeaa6982df/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs-dev amd64 19.3.0-4938-g33f95d2a-1jammy [39.0 kB] 2024-09-16T15:39:07.365 INFO:teuthology.orchestra.run.smithi184.stdout:Get:124 https://4.chacra.ceph.com/r/ceph/wip-athakkar-testing-2024-09-16-1454/33f95d2ae1bcbe7eb04404303f179eeeaa6982df/ubuntu/jammy/flavors/default jammy/main amd64 radosgw amd64 19.3.0-4938-g33f95d2a-1jammy [15.8 MB] 2024-09-16T15:39:07.443 INFO:teuthology.orchestra.run.smithi123.stdout:Get:119 https://4.chacra.ceph.com/r/ceph/wip-athakkar-testing-2024-09-16-1454/33f95d2ae1bcbe7eb04404303f179eeeaa6982df/ubuntu/jammy/flavors/default jammy/main amd64 ceph-volume all 19.3.0-4938-g33f95d2a-1jammy [130 kB] 2024-09-16T15:39:07.446 INFO:teuthology.orchestra.run.smithi123.stdout:Get:120 https://4.chacra.ceph.com/r/ceph/wip-athakkar-testing-2024-09-16-1454/33f95d2ae1bcbe7eb04404303f179eeeaa6982df/ubuntu/jammy/flavors/default jammy/main amd64 cephfs-mirror amd64 19.3.0-4938-g33f95d2a-1jammy [233 kB] 2024-09-16T15:39:07.450 INFO:teuthology.orchestra.run.smithi123.stdout:Get:121 https://4.chacra.ceph.com/r/ceph/wip-athakkar-testing-2024-09-16-1454/33f95d2ae1bcbe7eb04404303f179eeeaa6982df/ubuntu/jammy/flavors/default jammy/main amd64 cephfs-shell all 19.3.0-4938-g33f95d2a-1jammy [33.9 kB] 2024-09-16T15:39:07.451 INFO:teuthology.orchestra.run.smithi123.stdout:Get:122 https://4.chacra.ceph.com/r/ceph/wip-athakkar-testing-2024-09-16-1454/33f95d2ae1bcbe7eb04404303f179eeeaa6982df/ubuntu/jammy/flavors/default jammy/main amd64 cephfs-top all 19.3.0-4938-g33f95d2a-1jammy [23.4 kB] 2024-09-16T15:39:07.452 INFO:teuthology.orchestra.run.smithi123.stdout:Get:123 https://4.chacra.ceph.com/r/ceph/wip-athakkar-testing-2024-09-16-1454/33f95d2ae1bcbe7eb04404303f179eeeaa6982df/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs-dev amd64 19.3.0-4938-g33f95d2a-1jammy [39.0 kB] 2024-09-16T15:39:07.453 INFO:teuthology.orchestra.run.smithi123.stdout:Get:124 https://4.chacra.ceph.com/r/ceph/wip-athakkar-testing-2024-09-16-1454/33f95d2ae1bcbe7eb04404303f179eeeaa6982df/ubuntu/jammy/flavors/default jammy/main amd64 radosgw amd64 19.3.0-4938-g33f95d2a-1jammy [15.8 MB] 2024-09-16T15:39:07.748 INFO:teuthology.orchestra.run.smithi184.stdout:Get:125 https://4.chacra.ceph.com/r/ceph/wip-athakkar-testing-2024-09-16-1454/33f95d2ae1bcbe7eb04404303f179eeeaa6982df/ubuntu/jammy/flavors/default jammy/main amd64 rbd-fuse amd64 19.3.0-4938-g33f95d2a-1jammy [91.4 kB] 2024-09-16T15:39:07.861 INFO:teuthology.orchestra.run.smithi123.stdout:Get:125 https://4.chacra.ceph.com/r/ceph/wip-athakkar-testing-2024-09-16-1454/33f95d2ae1bcbe7eb04404303f179eeeaa6982df/ubuntu/jammy/flavors/default jammy/main amd64 rbd-fuse amd64 19.3.0-4938-g33f95d2a-1jammy [91.4 kB] 2024-09-16T15:39:08.366 INFO:teuthology.orchestra.run.smithi184.stdout:Fetched 244 MB in 8s (30.8 MB/s) 2024-09-16T15:39:08.507 INFO:teuthology.orchestra.run.smithi123.stdout:Fetched 244 MB in 8s (30.6 MB/s) 2024-09-16T15:39:08.528 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package python3-pyperclip. 2024-09-16T15:39:08.618 INFO:teuthology.orchestra.run.smithi184.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 165990 files and directories currently installed.) 2024-09-16T15:39:08.623 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../000-python3-pyperclip_1.8.2-2_all.deb ... 2024-09-16T15:39:08.641 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package python3-pyperclip. 2024-09-16T15:39:08.669 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking python3-pyperclip (1.8.2-2) ... 2024-09-16T15:39:08.739 INFO:teuthology.orchestra.run.smithi123.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 165990 files and directories currently installed.) 2024-09-16T15:39:08.745 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../000-python3-pyperclip_1.8.2-2_all.deb ... 2024-09-16T15:39:08.781 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking python3-pyperclip (1.8.2-2) ... 2024-09-16T15:39:08.913 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package python3-wcwidth. 2024-09-16T15:39:08.940 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../001-python3-wcwidth_0.2.5+dfsg1-1_all.deb ... 2024-09-16T15:39:08.971 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking python3-wcwidth (0.2.5+dfsg1-1) ... 2024-09-16T15:39:09.076 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package python3-wcwidth. 2024-09-16T15:39:09.102 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../001-python3-wcwidth_0.2.5+dfsg1-1_all.deb ... 2024-09-16T15:39:09.142 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking python3-wcwidth (0.2.5+dfsg1-1) ... 2024-09-16T15:39:09.316 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package python3-cmd2. 2024-09-16T15:39:09.342 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../002-python3-cmd2_2.3.3-1_all.deb ... 2024-09-16T15:39:09.383 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking python3-cmd2 (2.3.3-1) ... 2024-09-16T15:39:09.454 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package python3-cmd2. 2024-09-16T15:39:09.480 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../002-python3-cmd2_2.3.3-1_all.deb ... 2024-09-16T15:39:09.520 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking python3-cmd2 (2.3.3-1) ... 2024-09-16T15:39:09.736 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package liblttng-ust1:amd64. 2024-09-16T15:39:09.762 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../003-liblttng-ust1_2.13.1-1ubuntu1_amd64.deb ... 2024-09-16T15:39:09.840 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package liblttng-ust1:amd64. 2024-09-16T15:39:09.844 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-09-16T15:39:09.866 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../003-liblttng-ust1_2.13.1-1ubuntu1_amd64.deb ... 2024-09-16T15:39:09.948 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-09-16T15:39:10.190 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package libdouble-conversion3:amd64. 2024-09-16T15:39:10.216 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../004-libdouble-conversion3_3.1.7-4_amd64.deb ... 2024-09-16T15:39:10.256 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking libdouble-conversion3:amd64 (3.1.7-4) ... 2024-09-16T15:39:10.285 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package libdouble-conversion3:amd64. 2024-09-16T15:39:10.311 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../004-libdouble-conversion3_3.1.7-4_amd64.deb ... 2024-09-16T15:39:10.351 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking libdouble-conversion3:amd64 (3.1.7-4) ... 2024-09-16T15:39:10.534 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package libpcre2-16-0:amd64. 2024-09-16T15:39:10.560 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../005-libpcre2-16-0_10.39-3ubuntu0.1_amd64.deb ... 2024-09-16T15:39:10.592 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-09-16T15:39:10.679 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package libpcre2-16-0:amd64. 2024-09-16T15:39:10.705 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../005-libpcre2-16-0_10.39-3ubuntu0.1_amd64.deb ... 2024-09-16T15:39:10.746 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-09-16T15:39:11.116 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package libqt5core5a:amd64. 2024-09-16T15:39:11.142 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../006-libqt5core5a_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-09-16T15:39:11.182 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-16T15:39:11.321 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package libqt5core5a:amd64. 2024-09-16T15:39:11.347 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../006-libqt5core5a_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-09-16T15:39:11.396 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-16T15:39:11.611 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package libqt5dbus5:amd64. 2024-09-16T15:39:11.637 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../007-libqt5dbus5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-09-16T15:39:11.677 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-16T15:39:11.824 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package libqt5dbus5:amd64. 2024-09-16T15:39:11.850 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../007-libqt5dbus5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-09-16T15:39:11.891 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-16T15:39:12.022 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package libqt5network5:amd64. 2024-09-16T15:39:12.048 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../008-libqt5network5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-09-16T15:39:12.088 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-16T15:39:12.261 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package libqt5network5:amd64. 2024-09-16T15:39:12.287 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../008-libqt5network5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-09-16T15:39:12.327 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-16T15:39:12.400 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package libthrift-0.16.0:amd64. 2024-09-16T15:39:12.415 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../009-libthrift-0.16.0_0.16.0-2_amd64.deb ... 2024-09-16T15:39:12.449 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-09-16T15:39:12.664 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package libthrift-0.16.0:amd64. 2024-09-16T15:39:12.690 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../009-libthrift-0.16.0_0.16.0-2_amd64.deb ... 2024-09-16T15:39:12.730 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-09-16T15:39:12.777 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package librados2. 2024-09-16T15:39:12.803 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../010-librados2_19.3.0-4938-g33f95d2a-1jammy_amd64.deb ... 2024-09-16T15:39:12.844 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking librados2 (19.3.0-4938-g33f95d2a-1jammy) ... 2024-09-16T15:39:13.067 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package librados2. 2024-09-16T15:39:13.082 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../010-librados2_19.3.0-4938-g33f95d2a-1jammy_amd64.deb ... 2024-09-16T15:39:13.125 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking librados2 (19.3.0-4938-g33f95d2a-1jammy) ... 2024-09-16T15:39:13.322 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package libnbd0. 2024-09-16T15:39:13.348 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../011-libnbd0_1.10.5-1_amd64.deb ... 2024-09-16T15:39:13.388 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking libnbd0 (1.10.5-1) ... 2024-09-16T15:39:13.611 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package libnbd0. 2024-09-16T15:39:13.638 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../011-libnbd0_1.10.5-1_amd64.deb ... 2024-09-16T15:39:13.678 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking libnbd0 (1.10.5-1) ... 2024-09-16T15:39:13.699 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package librbd1. 2024-09-16T15:39:13.725 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../012-librbd1_19.3.0-4938-g33f95d2a-1jammy_amd64.deb ... 2024-09-16T15:39:13.766 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking librbd1 (19.3.0-4938-g33f95d2a-1jammy) ... 2024-09-16T15:39:13.939 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package librbd1. 2024-09-16T15:39:13.954 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../012-librbd1_19.3.0-4938-g33f95d2a-1jammy_amd64.deb ... 2024-09-16T15:39:13.989 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking librbd1 (19.3.0-4938-g33f95d2a-1jammy) ... 2024-09-16T15:39:14.194 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package libcephfs2. 2024-09-16T15:39:14.220 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../013-libcephfs2_19.3.0-4938-g33f95d2a-1jammy_amd64.deb ... 2024-09-16T15:39:14.260 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking libcephfs2 (19.3.0-4938-g33f95d2a-1jammy) ... 2024-09-16T15:39:14.417 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package libcephfs2. 2024-09-16T15:39:14.443 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../013-libcephfs2_19.3.0-4938-g33f95d2a-1jammy_amd64.deb ... 2024-09-16T15:39:14.484 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking libcephfs2 (19.3.0-4938-g33f95d2a-1jammy) ... 2024-09-16T15:39:14.538 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package python3-rados. 2024-09-16T15:39:14.553 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../014-python3-rados_19.3.0-4938-g33f95d2a-1jammy_amd64.deb ... 2024-09-16T15:39:14.596 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking python3-rados (19.3.0-4938-g33f95d2a-1jammy) ... 2024-09-16T15:39:14.770 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package python3-rados. 2024-09-16T15:39:14.785 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../014-python3-rados_19.3.0-4938-g33f95d2a-1jammy_amd64.deb ... 2024-09-16T15:39:14.828 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking python3-rados (19.3.0-4938-g33f95d2a-1jammy) ... 2024-09-16T15:39:14.915 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package python3-ceph-argparse. 2024-09-16T15:39:14.941 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../015-python3-ceph-argparse_19.3.0-4938-g33f95d2a-1jammy_all.deb ... 2024-09-16T15:39:14.982 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking python3-ceph-argparse (19.3.0-4938-g33f95d2a-1jammy) ... 2024-09-16T15:39:15.139 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package python3-ceph-argparse. 2024-09-16T15:39:15.166 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../015-python3-ceph-argparse_19.3.0-4938-g33f95d2a-1jammy_all.deb ... 2024-09-16T15:39:15.198 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking python3-ceph-argparse (19.3.0-4938-g33f95d2a-1jammy) ... 2024-09-16T15:39:15.251 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package python3-cephfs. 2024-09-16T15:39:15.277 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../016-python3-cephfs_19.3.0-4938-g33f95d2a-1jammy_amd64.deb ... 2024-09-16T15:39:15.318 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking python3-cephfs (19.3.0-4938-g33f95d2a-1jammy) ... 2024-09-16T15:39:15.417 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package python3-cephfs. 2024-09-16T15:39:15.432 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../016-python3-cephfs_19.3.0-4938-g33f95d2a-1jammy_amd64.deb ... 2024-09-16T15:39:15.475 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking python3-cephfs (19.3.0-4938-g33f95d2a-1jammy) ... 2024-09-16T15:39:15.587 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package python3-ceph-common. 2024-09-16T15:39:15.613 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../017-python3-ceph-common_19.3.0-4938-g33f95d2a-1jammy_all.deb ... 2024-09-16T15:39:15.653 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking python3-ceph-common (19.3.0-4938-g33f95d2a-1jammy) ... 2024-09-16T15:39:15.803 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package python3-ceph-common. 2024-09-16T15:39:15.829 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../017-python3-ceph-common_19.3.0-4938-g33f95d2a-1jammy_all.deb ... 2024-09-16T15:39:15.869 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking python3-ceph-common (19.3.0-4938-g33f95d2a-1jammy) ... 2024-09-16T15:39:15.940 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package python3-prettytable. 2024-09-16T15:39:15.955 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../018-python3-prettytable_2.5.0-2_all.deb ... 2024-09-16T15:39:15.990 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking python3-prettytable (2.5.0-2) ... 2024-09-16T15:39:16.197 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package python3-prettytable. 2024-09-16T15:39:16.224 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../018-python3-prettytable_2.5.0-2_all.deb ... 2024-09-16T15:39:16.264 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking python3-prettytable (2.5.0-2) ... 2024-09-16T15:39:16.267 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package python3-rbd. 2024-09-16T15:39:16.293 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../019-python3-rbd_19.3.0-4938-g33f95d2a-1jammy_amd64.deb ... 2024-09-16T15:39:16.333 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking python3-rbd (19.3.0-4938-g33f95d2a-1jammy) ... 2024-09-16T15:39:16.659 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package python3-rbd. 2024-09-16T15:39:16.685 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../019-python3-rbd_19.3.0-4938-g33f95d2a-1jammy_amd64.deb ... 2024-09-16T15:39:16.687 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package librdkafka1:amd64. 2024-09-16T15:39:16.713 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../020-librdkafka1_1.8.0-1build1_amd64.deb ... 2024-09-16T15:39:16.725 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking python3-rbd (19.3.0-4938-g33f95d2a-1jammy) ... 2024-09-16T15:39:16.753 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking librdkafka1:amd64 (1.8.0-1build1) ... 2024-09-16T15:39:17.030 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package libreadline-dev:amd64. 2024-09-16T15:39:17.036 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package librdkafka1:amd64. 2024-09-16T15:39:17.046 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../021-libreadline-dev_8.1.2-1_amd64.deb ... 2024-09-16T15:39:17.063 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../020-librdkafka1_1.8.0-1build1_amd64.deb ... 2024-09-16T15:39:17.089 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking libreadline-dev:amd64 (8.1.2-1) ... 2024-09-16T15:39:17.095 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking librdkafka1:amd64 (1.8.0-1build1) ... 2024-09-16T15:39:17.339 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package libreadline-dev:amd64. 2024-09-16T15:39:17.365 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../021-libreadline-dev_8.1.2-1_amd64.deb ... 2024-09-16T15:39:17.366 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package liblua5.3-dev:amd64. 2024-09-16T15:39:17.381 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../022-liblua5.3-dev_5.3.6-1build1_amd64.deb ... 2024-09-16T15:39:17.405 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking libreadline-dev:amd64 (8.1.2-1) ... 2024-09-16T15:39:17.416 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking liblua5.3-dev:amd64 (5.3.6-1build1) ... 2024-09-16T15:39:17.717 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package liblua5.3-dev:amd64. 2024-09-16T15:39:17.744 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../022-liblua5.3-dev_5.3.6-1build1_amd64.deb ... 2024-09-16T15:39:17.753 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package lua5.1. 2024-09-16T15:39:17.779 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../023-lua5.1_5.1.5-8.1build4_amd64.deb ... 2024-09-16T15:39:17.783 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking liblua5.3-dev:amd64 (5.3.6-1build1) ... 2024-09-16T15:39:17.819 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking lua5.1 (5.1.5-8.1build4) ... 2024-09-16T15:39:18.114 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package lua-any. 2024-09-16T15:39:18.129 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package lua5.1. 2024-09-16T15:39:18.129 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../024-lua-any_27ubuntu1_all.deb ... 2024-09-16T15:39:18.149 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../023-lua5.1_5.1.5-8.1build4_amd64.deb ... 2024-09-16T15:39:18.172 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking lua-any (27ubuntu1) ... 2024-09-16T15:39:18.187 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking lua5.1 (5.1.5-8.1build4) ... 2024-09-16T15:39:18.433 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package zip. 2024-09-16T15:39:18.449 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../025-zip_3.0-12build2_amd64.deb ... 2024-09-16T15:39:18.465 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package lua-any. 2024-09-16T15:39:18.480 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../024-lua-any_27ubuntu1_all.deb ... 2024-09-16T15:39:18.491 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking zip (3.0-12build2) ... 2024-09-16T15:39:18.523 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking lua-any (27ubuntu1) ... 2024-09-16T15:39:18.809 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package zip. 2024-09-16T15:39:18.825 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../025-zip_3.0-12build2_amd64.deb ... 2024-09-16T15:39:18.828 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package unzip. 2024-09-16T15:39:18.854 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../026-unzip_6.0-26ubuntu3.2_amd64.deb ... 2024-09-16T15:39:18.868 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking zip (3.0-12build2) ... 2024-09-16T15:39:18.895 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking unzip (6.0-26ubuntu3.2) ... 2024-09-16T15:39:19.238 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package unzip. 2024-09-16T15:39:19.264 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../026-unzip_6.0-26ubuntu3.2_amd64.deb ... 2024-09-16T15:39:19.273 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package luarocks. 2024-09-16T15:39:19.299 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../027-luarocks_3.8.0+dfsg1-1_all.deb ... 2024-09-16T15:39:19.304 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking unzip (6.0-26ubuntu3.2) ... 2024-09-16T15:39:19.340 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking luarocks (3.8.0+dfsg1-1) ... 2024-09-16T15:39:19.658 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package luarocks. 2024-09-16T15:39:19.685 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../027-luarocks_3.8.0+dfsg1-1_all.deb ... 2024-09-16T15:39:19.706 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package librgw2. 2024-09-16T15:39:19.724 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking luarocks (3.8.0+dfsg1-1) ... 2024-09-16T15:39:19.732 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../028-librgw2_19.3.0-4938-g33f95d2a-1jammy_amd64.deb ... 2024-09-16T15:39:19.772 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking librgw2 (19.3.0-4938-g33f95d2a-1jammy) ... 2024-09-16T15:39:20.099 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package librgw2. 2024-09-16T15:39:20.126 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../028-librgw2_19.3.0-4938-g33f95d2a-1jammy_amd64.deb ... 2024-09-16T15:39:20.165 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking librgw2 (19.3.0-4938-g33f95d2a-1jammy) ... 2024-09-16T15:39:20.333 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package python3-rgw. 2024-09-16T15:39:20.381 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../029-python3-rgw_19.3.0-4938-g33f95d2a-1jammy_amd64.deb ... 2024-09-16T15:39:20.399 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking python3-rgw (19.3.0-4938-g33f95d2a-1jammy) ... 2024-09-16T15:39:20.727 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package liboath0:amd64. 2024-09-16T15:39:20.753 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../030-liboath0_2.6.7-3build1_amd64.deb ... 2024-09-16T15:39:20.777 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package python3-rgw. 2024-09-16T15:39:20.794 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking liboath0:amd64 (2.6.7-3build1) ... 2024-09-16T15:39:20.803 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../029-python3-rgw_19.3.0-4938-g33f95d2a-1jammy_amd64.deb ... 2024-09-16T15:39:20.843 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking python3-rgw (19.3.0-4938-g33f95d2a-1jammy) ... 2024-09-16T15:39:21.063 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package libradosstriper1. 2024-09-16T15:39:21.078 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../031-libradosstriper1_19.3.0-4938-g33f95d2a-1jammy_amd64.deb ... 2024-09-16T15:39:21.113 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking libradosstriper1 (19.3.0-4938-g33f95d2a-1jammy) ... 2024-09-16T15:39:21.154 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package liboath0:amd64. 2024-09-16T15:39:21.181 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../030-liboath0_2.6.7-3build1_amd64.deb ... 2024-09-16T15:39:21.221 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking liboath0:amd64 (2.6.7-3build1) ... 2024-09-16T15:39:21.499 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package libradosstriper1. 2024-09-16T15:39:21.528 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../031-libradosstriper1_19.3.0-4938-g33f95d2a-1jammy_amd64.deb ... 2024-09-16T15:39:21.533 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package ceph-common. 2024-09-16T15:39:21.559 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../032-ceph-common_19.3.0-4938-g33f95d2a-1jammy_amd64.deb ... 2024-09-16T15:39:21.599 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking ceph-common (19.3.0-4938-g33f95d2a-1jammy) ... 2024-09-16T15:39:21.690 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking libradosstriper1 (19.3.0-4938-g33f95d2a-1jammy) ... 2024-09-16T15:39:22.119 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package ceph-common. 2024-09-16T15:39:22.146 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../032-ceph-common_19.3.0-4938-g33f95d2a-1jammy_amd64.deb ... 2024-09-16T15:39:22.185 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking ceph-common (19.3.0-4938-g33f95d2a-1jammy) ... 2024-09-16T15:39:23.071 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package ceph-base. 2024-09-16T15:39:23.097 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../033-ceph-base_19.3.0-4938-g33f95d2a-1jammy_amd64.deb ... 2024-09-16T15:39:23.147 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking ceph-base (19.3.0-4938-g33f95d2a-1jammy) ... 2024-09-16T15:39:23.616 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package ceph-base. 2024-09-16T15:39:23.642 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../033-ceph-base_19.3.0-4938-g33f95d2a-1jammy_amd64.deb ... 2024-09-16T15:39:23.683 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking ceph-base (19.3.0-4938-g33f95d2a-1jammy) ... 2024-09-16T15:39:23.717 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package python3-jaraco.functools. 2024-09-16T15:39:23.743 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../034-python3-jaraco.functools_3.4.0-2_all.deb ... 2024-09-16T15:39:23.783 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking python3-jaraco.functools (3.4.0-2) ... 2024-09-16T15:39:24.069 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package python3-cheroot. 2024-09-16T15:39:24.096 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../035-python3-cheroot_8.5.2+ds1-1ubuntu3.1_all.deb ... 2024-09-16T15:39:24.136 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-09-16T15:39:24.229 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package python3-jaraco.functools. 2024-09-16T15:39:24.255 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../034-python3-jaraco.functools_3.4.0-2_all.deb ... 2024-09-16T15:39:24.295 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking python3-jaraco.functools (3.4.0-2) ... 2024-09-16T15:39:24.447 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package python3-jaraco.classes. 2024-09-16T15:39:24.473 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../036-python3-jaraco.classes_3.2.1-3_all.deb ... 2024-09-16T15:39:24.513 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking python3-jaraco.classes (3.2.1-3) ... 2024-09-16T15:39:24.598 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package python3-cheroot. 2024-09-16T15:39:24.622 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../035-python3-cheroot_8.5.2+ds1-1ubuntu3.1_all.deb ... 2024-09-16T15:39:24.664 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-09-16T15:39:24.816 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package python3-jaraco.text. 2024-09-16T15:39:24.843 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../037-python3-jaraco.text_3.6.0-2_all.deb ... 2024-09-16T15:39:24.883 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking python3-jaraco.text (3.6.0-2) ... 2024-09-16T15:39:24.985 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package python3-jaraco.classes. 2024-09-16T15:39:25.011 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../036-python3-jaraco.classes_3.2.1-3_all.deb ... 2024-09-16T15:39:25.051 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking python3-jaraco.classes (3.2.1-3) ... 2024-09-16T15:39:25.211 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package python3-jaraco.collections. 2024-09-16T15:39:25.237 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../038-python3-jaraco.collections_3.4.0-2_all.deb ... 2024-09-16T15:39:25.278 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking python3-jaraco.collections (3.4.0-2) ... 2024-09-16T15:39:25.320 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package python3-jaraco.text. 2024-09-16T15:39:25.336 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../037-python3-jaraco.text_3.6.0-2_all.deb ... 2024-09-16T15:39:25.370 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking python3-jaraco.text (3.6.0-2) ... 2024-09-16T15:39:25.547 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package python3-tempora. 2024-09-16T15:39:25.573 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../039-python3-tempora_4.1.2-1_all.deb ... 2024-09-16T15:39:25.613 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking python3-tempora (4.1.2-1) ... 2024-09-16T15:39:25.640 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package python3-jaraco.collections. 2024-09-16T15:39:25.666 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../038-python3-jaraco.collections_3.4.0-2_all.deb ... 2024-09-16T15:39:25.706 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking python3-jaraco.collections (3.4.0-2) ... 2024-09-16T15:39:25.891 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package python3-portend. 2024-09-16T15:39:25.906 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../040-python3-portend_3.0.0-1_all.deb ... 2024-09-16T15:39:25.941 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking python3-portend (3.0.0-1) ... 2024-09-16T15:39:25.993 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package python3-tempora. 2024-09-16T15:39:26.020 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../039-python3-tempora_4.1.2-1_all.deb ... 2024-09-16T15:39:26.059 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking python3-tempora (4.1.2-1) ... 2024-09-16T15:39:26.177 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package python3-zc.lockfile. 2024-09-16T15:39:26.203 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../041-python3-zc.lockfile_2.0-1_all.deb ... 2024-09-16T15:39:26.244 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking python3-zc.lockfile (2.0-1) ... 2024-09-16T15:39:26.337 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package python3-portend. 2024-09-16T15:39:26.364 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../040-python3-portend_3.0.0-1_all.deb ... 2024-09-16T15:39:26.429 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking python3-portend (3.0.0-1) ... 2024-09-16T15:39:26.547 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package python3-cherrypy3. 2024-09-16T15:39:26.573 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../042-python3-cherrypy3_18.6.1-4_all.deb ... 2024-09-16T15:39:26.613 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking python3-cherrypy3 (18.6.1-4) ... 2024-09-16T15:39:26.673 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package python3-zc.lockfile. 2024-09-16T15:39:26.691 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../041-python3-zc.lockfile_2.0-1_all.deb ... 2024-09-16T15:39:26.731 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking python3-zc.lockfile (2.0-1) ... 2024-09-16T15:39:27.025 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package python3-natsort. 2024-09-16T15:39:27.051 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../043-python3-natsort_8.0.2-1_all.deb ... 2024-09-16T15:39:27.078 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package python3-cherrypy3. 2024-09-16T15:39:27.091 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking python3-natsort (8.0.2-1) ... 2024-09-16T15:39:27.105 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../042-python3-cherrypy3_18.6.1-4_all.deb ... 2024-09-16T15:39:27.153 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking python3-cherrypy3 (18.6.1-4) ... 2024-09-16T15:39:27.370 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package python3-logutils. 2024-09-16T15:39:27.396 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../044-python3-logutils_0.3.3-8_all.deb ... 2024-09-16T15:39:27.436 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking python3-logutils (0.3.3-8) ... 2024-09-16T15:39:27.498 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package python3-natsort. 2024-09-16T15:39:27.525 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../043-python3-natsort_8.0.2-1_all.deb ... 2024-09-16T15:39:27.565 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking python3-natsort (8.0.2-1) ... 2024-09-16T15:39:27.739 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package python3-markupsafe. 2024-09-16T15:39:27.765 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../045-python3-markupsafe_2.0.1-2build1_amd64.deb ... 2024-09-16T15:39:27.805 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking python3-markupsafe (2.0.1-2build1) ... 2024-09-16T15:39:27.935 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package python3-logutils. 2024-09-16T15:39:27.962 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../044-python3-logutils_0.3.3-8_all.deb ... 2024-09-16T15:39:28.009 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking python3-logutils (0.3.3-8) ... 2024-09-16T15:39:28.108 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package python3-mako. 2024-09-16T15:39:28.134 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../046-python3-mako_1.1.3+ds1-2ubuntu0.1_all.deb ... 2024-09-16T15:39:28.175 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-09-16T15:39:28.371 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package python3-markupsafe. 2024-09-16T15:39:28.397 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../045-python3-markupsafe_2.0.1-2build1_amd64.deb ... 2024-09-16T15:39:28.446 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking python3-markupsafe (2.0.1-2build1) ... 2024-09-16T15:39:28.503 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package python3-simplegeneric. 2024-09-16T15:39:28.529 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../047-python3-simplegeneric_0.8.1-3_all.deb ... 2024-09-16T15:39:28.569 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking python3-simplegeneric (0.8.1-3) ... 2024-09-16T15:39:28.799 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package python3-mako. 2024-09-16T15:39:28.826 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../046-python3-mako_1.1.3+ds1-2ubuntu0.1_all.deb ... 2024-09-16T15:39:28.839 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package python3-singledispatch. 2024-09-16T15:39:28.865 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-09-16T15:39:28.865 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../048-python3-singledispatch_3.4.0.3-3_all.deb ... 2024-09-16T15:39:28.897 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking python3-singledispatch (3.4.0.3-3) ... 2024-09-16T15:39:29.185 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package python3-simplegeneric. 2024-09-16T15:39:29.192 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package python3-webob. 2024-09-16T15:39:29.212 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../047-python3-simplegeneric_0.8.1-3_all.deb ... 2024-09-16T15:39:29.218 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../049-python3-webob_1%3a1.8.6-1.1ubuntu0.1_all.deb ... 2024-09-16T15:39:29.252 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking python3-simplegeneric (0.8.1-3) ... 2024-09-16T15:39:29.258 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking python3-webob (1:1.8.6-1.1ubuntu0.1) ... 2024-09-16T15:39:29.530 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package python3-singledispatch. 2024-09-16T15:39:29.546 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../048-python3-singledispatch_3.4.0.3-3_all.deb ... 2024-09-16T15:39:29.580 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking python3-singledispatch (3.4.0.3-3) ... 2024-09-16T15:39:29.603 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package python3-waitress. 2024-09-16T15:39:29.629 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../050-python3-waitress_1.4.4-1.1ubuntu1_all.deb ... 2024-09-16T15:39:29.672 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking python3-waitress (1.4.4-1.1ubuntu1) ... 2024-09-16T15:39:29.858 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package python3-webob. 2024-09-16T15:39:29.885 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../049-python3-webob_1%3a1.8.6-1.1ubuntu0.1_all.deb ... 2024-09-16T15:39:29.924 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking python3-webob (1:1.8.6-1.1ubuntu0.1) ... 2024-09-16T15:39:29.964 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package python3-tempita. 2024-09-16T15:39:29.991 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../051-python3-tempita_0.5.2-6ubuntu1_all.deb ... 2024-09-16T15:39:30.030 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking python3-tempita (0.5.2-6ubuntu1) ... 2024-09-16T15:39:30.303 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package python3-waitress. 2024-09-16T15:39:30.329 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../050-python3-waitress_1.4.4-1.1ubuntu1_all.deb ... 2024-09-16T15:39:30.333 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package python3-paste. 2024-09-16T15:39:30.360 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../052-python3-paste_3.5.0+dfsg1-1_all.deb ... 2024-09-16T15:39:30.372 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking python3-waitress (1.4.4-1.1ubuntu1) ... 2024-09-16T15:39:30.399 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking python3-paste (3.5.0+dfsg1-1) ... 2024-09-16T15:39:30.680 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package python3-tempita. 2024-09-16T15:39:30.707 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../051-python3-tempita_0.5.2-6ubuntu1_all.deb ... 2024-09-16T15:39:30.747 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking python3-tempita (0.5.2-6ubuntu1) ... 2024-09-16T15:39:30.761 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2024-09-16T15:39:30.788 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../053-python-pastedeploy-tpl_2.1.1-1_all.deb ... 2024-09-16T15:39:30.827 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking python-pastedeploy-tpl (2.1.1-1) ... 2024-09-16T15:39:31.130 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package python3-pastedeploy. 2024-09-16T15:39:31.157 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../054-python3-pastedeploy_2.1.1-1_all.deb ... 2024-09-16T15:39:31.197 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking python3-pastedeploy (2.1.1-1) ... 2024-09-16T15:39:31.500 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package python3-webtest. 2024-09-16T15:39:31.526 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../055-python3-webtest_2.0.35-1_all.deb ... 2024-09-16T15:39:31.566 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking python3-webtest (2.0.35-1) ... 2024-09-16T15:39:31.709 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package python3-paste. 2024-09-16T15:39:31.736 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../052-python3-paste_3.5.0+dfsg1-1_all.deb ... 2024-09-16T15:39:32.184 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking python3-paste (3.5.0+dfsg1-1) ... 2024-09-16T15:39:32.245 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package python3-pecan. 2024-09-16T15:39:32.272 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../056-python3-pecan_1.3.3-4ubuntu2_all.deb ... 2024-09-16T15:39:32.586 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking python3-pecan (1.3.3-4ubuntu2) ... 2024-09-16T15:39:32.629 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2024-09-16T15:39:32.655 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../053-python-pastedeploy-tpl_2.1.1-1_all.deb ... 2024-09-16T15:39:32.696 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking python-pastedeploy-tpl (2.1.1-1) ... 2024-09-16T15:39:32.998 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package python3-pastedeploy. 2024-09-16T15:39:33.025 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../054-python3-pastedeploy_2.1.1-1_all.deb ... 2024-09-16T15:39:33.065 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking python3-pastedeploy (2.1.1-1) ... 2024-09-16T15:39:33.216 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package python3-certifi. 2024-09-16T15:39:33.242 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../057-python3-certifi_2020.6.20-1_all.deb ... 2024-09-16T15:39:33.282 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking python3-certifi (2020.6.20-1) ... 2024-09-16T15:39:33.368 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package python3-webtest. 2024-09-16T15:39:33.394 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../055-python3-webtest_2.0.35-1_all.deb ... 2024-09-16T15:39:33.434 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking python3-webtest (2.0.35-1) ... 2024-09-16T15:39:33.602 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package python3-idna. 2024-09-16T15:39:33.628 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../058-python3-idna_3.3-1ubuntu0.1_all.deb ... 2024-09-16T15:39:33.668 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking python3-idna (3.3-1ubuntu0.1) ... 2024-09-16T15:39:33.737 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package python3-pecan. 2024-09-16T15:39:33.753 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../056-python3-pecan_1.3.3-4ubuntu2_all.deb ... 2024-09-16T15:39:33.787 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking python3-pecan (1.3.3-4ubuntu2) ... 2024-09-16T15:39:33.971 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package python3-urllib3. 2024-09-16T15:39:33.998 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../059-python3-urllib3_1.26.5-1~exp1ubuntu0.1_all.deb ... 2024-09-16T15:39:34.037 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-09-16T15:39:34.151 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package python3-certifi. 2024-09-16T15:39:34.177 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../057-python3-certifi_2020.6.20-1_all.deb ... 2024-09-16T15:39:34.209 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking python3-certifi (2020.6.20-1) ... 2024-09-16T15:39:34.349 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package python3-requests. 2024-09-16T15:39:34.376 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../060-python3-requests_2.25.1+dfsg-2ubuntu0.1_all.deb ... 2024-09-16T15:39:34.424 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-09-16T15:39:34.529 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package python3-idna. 2024-09-16T15:39:34.555 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../058-python3-idna_3.3-1ubuntu0.1_all.deb ... 2024-09-16T15:39:34.595 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking python3-idna (3.3-1ubuntu0.1) ... 2024-09-16T15:39:34.735 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package python3-werkzeug. 2024-09-16T15:39:34.762 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../061-python3-werkzeug_2.0.2+dfsg1-1ubuntu0.22.04.2_all.deb ... 2024-09-16T15:39:34.801 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.2) ... 2024-09-16T15:39:34.907 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package python3-urllib3. 2024-09-16T15:39:34.933 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../059-python3-urllib3_1.26.5-1~exp1ubuntu0.1_all.deb ... 2024-09-16T15:39:34.973 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-09-16T15:39:35.079 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2024-09-16T15:39:35.095 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../062-ceph-mgr-modules-core_19.3.0-4938-g33f95d2a-1jammy_all.deb ... 2024-09-16T15:39:35.129 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking ceph-mgr-modules-core (19.3.0-4938-g33f95d2a-1jammy) ... 2024-09-16T15:39:35.293 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package python3-requests. 2024-09-16T15:39:35.319 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../060-python3-requests_2.25.1+dfsg-2ubuntu0.1_all.deb ... 2024-09-16T15:39:35.359 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-09-16T15:39:35.501 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package libsqlite3-mod-ceph. 2024-09-16T15:39:35.527 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../063-libsqlite3-mod-ceph_19.3.0-4938-g33f95d2a-1jammy_amd64.deb ... 2024-09-16T15:39:35.567 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking libsqlite3-mod-ceph (19.3.0-4938-g33f95d2a-1jammy) ... 2024-09-16T15:39:35.679 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package python3-werkzeug. 2024-09-16T15:39:35.705 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../061-python3-werkzeug_2.0.2+dfsg1-1ubuntu0.22.04.2_all.deb ... 2024-09-16T15:39:35.746 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.2) ... 2024-09-16T15:39:35.879 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package python3-bcrypt. 2024-09-16T15:39:35.905 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../064-python3-bcrypt_3.2.0-1build1_amd64.deb ... 2024-09-16T15:39:35.945 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking python3-bcrypt (3.2.0-1build1) ... 2024-09-16T15:39:36.049 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2024-09-16T15:39:36.075 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../062-ceph-mgr-modules-core_19.3.0-4938-g33f95d2a-1jammy_all.deb ... 2024-09-16T15:39:36.115 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking ceph-mgr-modules-core (19.3.0-4938-g33f95d2a-1jammy) ... 2024-09-16T15:39:36.214 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package python3-openssl. 2024-09-16T15:39:36.230 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../065-python3-openssl_21.0.0-1_all.deb ... 2024-09-16T15:39:36.264 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking python3-openssl (21.0.0-1) ... 2024-09-16T15:39:36.521 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package libsqlite3-mod-ceph. 2024-09-16T15:39:36.547 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../063-libsqlite3-mod-ceph_19.3.0-4938-g33f95d2a-1jammy_amd64.deb ... 2024-09-16T15:39:36.587 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking libsqlite3-mod-ceph (19.3.0-4938-g33f95d2a-1jammy) ... 2024-09-16T15:39:36.592 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package ceph-mgr. 2024-09-16T15:39:36.619 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../066-ceph-mgr_19.3.0-4938-g33f95d2a-1jammy_amd64.deb ... 2024-09-16T15:39:36.658 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking ceph-mgr (19.3.0-4938-g33f95d2a-1jammy) ... 2024-09-16T15:39:36.907 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package python3-bcrypt. 2024-09-16T15:39:36.933 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../064-python3-bcrypt_3.2.0-1build1_amd64.deb ... 2024-09-16T15:39:36.973 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking python3-bcrypt (3.2.0-1build1) ... 2024-09-16T15:39:37.054 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package ceph-mon. 2024-09-16T15:39:37.080 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../067-ceph-mon_19.3.0-4938-g33f95d2a-1jammy_amd64.deb ... 2024-09-16T15:39:37.120 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking ceph-mon (19.3.0-4938-g33f95d2a-1jammy) ... 2024-09-16T15:39:37.276 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package python3-openssl. 2024-09-16T15:39:37.302 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../065-python3-openssl_21.0.0-1_all.deb ... 2024-09-16T15:39:37.342 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking python3-openssl (21.0.0-1) ... 2024-09-16T15:39:37.665 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package ceph-osd. 2024-09-16T15:39:37.679 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package ceph-mgr. 2024-09-16T15:39:37.692 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../068-ceph-osd_19.3.0-4938-g33f95d2a-1jammy_amd64.deb ... 2024-09-16T15:39:37.706 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../066-ceph-mgr_19.3.0-4938-g33f95d2a-1jammy_amd64.deb ... 2024-09-16T15:39:37.731 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking ceph-osd (19.3.0-4938-g33f95d2a-1jammy) ... 2024-09-16T15:39:37.745 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking ceph-mgr (19.3.0-4938-g33f95d2a-1jammy) ... 2024-09-16T15:39:38.141 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package ceph-mon. 2024-09-16T15:39:38.167 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../067-ceph-mon_19.3.0-4938-g33f95d2a-1jammy_amd64.deb ... 2024-09-16T15:39:38.207 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking ceph-mon (19.3.0-4938-g33f95d2a-1jammy) ... 2024-09-16T15:39:38.768 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package ceph. 2024-09-16T15:39:38.783 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../069-ceph_19.3.0-4938-g33f95d2a-1jammy_amd64.deb ... 2024-09-16T15:39:38.817 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking ceph (19.3.0-4938-g33f95d2a-1jammy) ... 2024-09-16T15:39:38.844 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package ceph-osd. 2024-09-16T15:39:38.870 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../068-ceph-osd_19.3.0-4938-g33f95d2a-1jammy_amd64.deb ... 2024-09-16T15:39:38.910 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking ceph-osd (19.3.0-4938-g33f95d2a-1jammy) ... 2024-09-16T15:39:39.122 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package ceph-fuse. 2024-09-16T15:39:39.149 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../070-ceph-fuse_19.3.0-4938-g33f95d2a-1jammy_amd64.deb ... 2024-09-16T15:39:39.189 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking ceph-fuse (19.3.0-4938-g33f95d2a-1jammy) ... 2024-09-16T15:39:39.550 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package ceph-mds. 2024-09-16T15:39:39.577 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../071-ceph-mds_19.3.0-4938-g33f95d2a-1jammy_amd64.deb ... 2024-09-16T15:39:39.616 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking ceph-mds (19.3.0-4938-g33f95d2a-1jammy) ... 2024-09-16T15:39:39.964 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package ceph. 2024-09-16T15:39:39.986 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package cephadm. 2024-09-16T15:39:39.990 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../069-ceph_19.3.0-4938-g33f95d2a-1jammy_amd64.deb ... 2024-09-16T15:39:40.001 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../072-cephadm_19.3.0-4938-g33f95d2a-1jammy_amd64.deb ... 2024-09-16T15:39:40.030 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking ceph (19.3.0-4938-g33f95d2a-1jammy) ... 2024-09-16T15:39:40.036 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking cephadm (19.3.0-4938-g33f95d2a-1jammy) ... 2024-09-16T15:39:40.356 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package python3-asyncssh. 2024-09-16T15:39:40.358 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package ceph-fuse. 2024-09-16T15:39:40.382 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../073-python3-asyncssh_2.5.0-1_all.deb ... 2024-09-16T15:39:40.384 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../070-ceph-fuse_19.3.0-4938-g33f95d2a-1jammy_amd64.deb ... 2024-09-16T15:39:40.422 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking python3-asyncssh (2.5.0-1) ... 2024-09-16T15:39:40.424 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking ceph-fuse (19.3.0-4938-g33f95d2a-1jammy) ... 2024-09-16T15:39:40.725 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package python-babel-localedata. 2024-09-16T15:39:40.751 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../074-python-babel-localedata_2.8.0+dfsg.1-7_all.deb ... 2024-09-16T15:39:40.791 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-09-16T15:39:40.827 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package ceph-mds. 2024-09-16T15:39:40.854 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../071-ceph-mds_19.3.0-4938-g33f95d2a-1jammy_amd64.deb ... 2024-09-16T15:39:40.894 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking ceph-mds (19.3.0-4938-g33f95d2a-1jammy) ... 2024-09-16T15:39:41.255 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package cephadm. 2024-09-16T15:39:41.271 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../072-cephadm_19.3.0-4938-g33f95d2a-1jammy_amd64.deb ... 2024-09-16T15:39:41.305 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking cephadm (19.3.0-4938-g33f95d2a-1jammy) ... 2024-09-16T15:39:41.625 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package python3-asyncssh. 2024-09-16T15:39:41.651 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../073-python3-asyncssh_2.5.0-1_all.deb ... 2024-09-16T15:39:41.691 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking python3-asyncssh (2.5.0-1) ... 2024-09-16T15:39:41.779 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package python3-babel. 2024-09-16T15:39:41.806 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../075-python3-babel_2.8.0+dfsg.1-7_all.deb ... 2024-09-16T15:39:41.862 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking python3-babel (2.8.0+dfsg.1-7) ... 2024-09-16T15:39:41.977 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package python-babel-localedata. 2024-09-16T15:39:42.004 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../074-python-babel-localedata_2.8.0+dfsg.1-7_all.deb ... 2024-09-16T15:39:42.043 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-09-16T15:39:42.174 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package python3-jinja2. 2024-09-16T15:39:42.200 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../076-python3-jinja2_3.0.3-1ubuntu0.2_all.deb ... 2024-09-16T15:39:42.232 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking python3-jinja2 (3.0.3-1ubuntu0.2) ... 2024-09-16T15:39:42.544 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2024-09-16T15:39:42.570 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../077-ceph-mgr-cephadm_19.3.0-4938-g33f95d2a-1jammy_all.deb ... 2024-09-16T15:39:42.602 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking ceph-mgr-cephadm (19.3.0-4938-g33f95d2a-1jammy) ... 2024-09-16T15:39:42.930 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package python3-repoze.lru. 2024-09-16T15:39:42.957 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../078-python3-repoze.lru_0.7-2_all.deb ... 2024-09-16T15:39:42.988 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking python3-repoze.lru (0.7-2) ... 2024-09-16T15:39:43.232 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package python3-babel. 2024-09-16T15:39:43.260 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../075-python3-babel_2.8.0+dfsg.1-7_all.deb ... 2024-09-16T15:39:43.299 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking python3-babel (2.8.0+dfsg.1-7) ... 2024-09-16T15:39:43.299 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package python3-routes. 2024-09-16T15:39:43.327 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../079-python3-routes_2.5.1-1ubuntu1_all.deb ... 2024-09-16T15:39:43.357 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking python3-routes (2.5.1-1ubuntu1) ... 2024-09-16T15:39:43.627 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package python3-jinja2. 2024-09-16T15:39:43.654 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../076-python3-jinja2_3.0.3-1ubuntu0.2_all.deb ... 2024-09-16T15:39:43.693 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking python3-jinja2 (3.0.3-1ubuntu0.2) ... 2024-09-16T15:39:44.039 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2024-09-16T15:39:44.065 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../077-ceph-mgr-cephadm_19.3.0-4938-g33f95d2a-1jammy_all.deb ... 2024-09-16T15:39:44.069 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2024-09-16T15:39:44.096 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../080-ceph-mgr-dashboard_19.3.0-4938-g33f95d2a-1jammy_all.deb ... 2024-09-16T15:39:44.105 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking ceph-mgr-cephadm (19.3.0-4938-g33f95d2a-1jammy) ... 2024-09-16T15:39:44.135 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking ceph-mgr-dashboard (19.3.0-4938-g33f95d2a-1jammy) ... 2024-09-16T15:39:44.467 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package python3-repoze.lru. 2024-09-16T15:39:44.494 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../078-python3-repoze.lru_0.7-2_all.deb ... 2024-09-16T15:39:44.533 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking python3-repoze.lru (0.7-2) ... 2024-09-16T15:39:44.836 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package python3-routes. 2024-09-16T15:39:44.863 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../079-python3-routes_2.5.1-1ubuntu1_all.deb ... 2024-09-16T15:39:44.903 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking python3-routes (2.5.1-1ubuntu1) ... 2024-09-16T15:39:45.223 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2024-09-16T15:39:45.250 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../080-ceph-mgr-dashboard_19.3.0-4938-g33f95d2a-1jammy_all.deb ... 2024-09-16T15:39:45.289 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking ceph-mgr-dashboard (19.3.0-4938-g33f95d2a-1jammy) ... 2024-09-16T15:39:46.815 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package python3-sklearn-lib:amd64. 2024-09-16T15:39:46.832 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../081-python3-sklearn-lib_0.23.2-5ubuntu6_amd64.deb ... 2024-09-16T15:39:46.873 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-09-16T15:39:47.327 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package python3-joblib. 2024-09-16T15:39:47.343 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../082-python3-joblib_0.17.0-4ubuntu1_all.deb ... 2024-09-16T15:39:47.462 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking python3-joblib (0.17.0-4ubuntu1) ... 2024-09-16T15:39:47.833 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package python3-threadpoolctl. 2024-09-16T15:39:47.860 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../083-python3-threadpoolctl_3.1.0-1_all.deb ... 2024-09-16T15:39:47.899 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking python3-threadpoolctl (3.1.0-1) ... 2024-09-16T15:39:48.128 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package python3-sklearn-lib:amd64. 2024-09-16T15:39:48.158 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../081-python3-sklearn-lib_0.23.2-5ubuntu6_amd64.deb ... 2024-09-16T15:39:48.185 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package python3-sklearn. 2024-09-16T15:39:48.202 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-09-16T15:39:48.213 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../084-python3-sklearn_0.23.2-5ubuntu6_all.deb ... 2024-09-16T15:39:48.252 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking python3-sklearn (0.23.2-5ubuntu6) ... 2024-09-16T15:39:48.706 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package python3-joblib. 2024-09-16T15:39:48.734 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../082-python3-joblib_0.17.0-4ubuntu1_all.deb ... 2024-09-16T15:39:48.891 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking python3-joblib (0.17.0-4ubuntu1) ... 2024-09-16T15:39:48.936 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2024-09-16T15:39:48.964 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../085-ceph-mgr-diskprediction-local_19.3.0-4938-g33f95d2a-1jammy_all.deb ... 2024-09-16T15:39:49.002 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking ceph-mgr-diskprediction-local (19.3.0-4938-g33f95d2a-1jammy) ... 2024-09-16T15:39:49.254 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package python3-threadpoolctl. 2024-09-16T15:39:49.282 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../083-python3-threadpoolctl_3.1.0-1_all.deb ... 2024-09-16T15:39:49.320 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking python3-threadpoolctl (3.1.0-1) ... 2024-09-16T15:39:49.657 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package python3-sklearn. 2024-09-16T15:39:49.685 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../084-python3-sklearn_0.23.2-5ubuntu6_all.deb ... 2024-09-16T15:39:49.723 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking python3-sklearn (0.23.2-5ubuntu6) ... 2024-09-16T15:39:49.956 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package python3-cachetools. 2024-09-16T15:39:49.982 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../086-python3-cachetools_5.0.0-1_all.deb ... 2024-09-16T15:39:50.021 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking python3-cachetools (5.0.0-1) ... 2024-09-16T15:39:50.324 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package python3-pyasn1. 2024-09-16T15:39:50.352 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../087-python3-pyasn1_0.4.8-1_all.deb ... 2024-09-16T15:39:50.391 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking python3-pyasn1 (0.4.8-1) ... 2024-09-16T15:39:50.399 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2024-09-16T15:39:50.428 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../085-ceph-mgr-diskprediction-local_19.3.0-4938-g33f95d2a-1jammy_all.deb ... 2024-09-16T15:39:50.457 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking ceph-mgr-diskprediction-local (19.3.0-4938-g33f95d2a-1jammy) ... 2024-09-16T15:39:50.686 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package python3-pyasn1-modules. 2024-09-16T15:39:50.713 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../088-python3-pyasn1-modules_0.2.1-1_all.deb ... 2024-09-16T15:39:50.744 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking python3-pyasn1-modules (0.2.1-1) ... 2024-09-16T15:39:51.030 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package python3-rsa. 2024-09-16T15:39:51.058 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../089-python3-rsa_4.8-1_all.deb ... 2024-09-16T15:39:51.097 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking python3-rsa (4.8-1) ... 2024-09-16T15:39:51.385 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package python3-cachetools. 2024-09-16T15:39:51.413 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../086-python3-cachetools_5.0.0-1_all.deb ... 2024-09-16T15:39:51.434 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package python3-google-auth. 2024-09-16T15:39:51.451 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking python3-cachetools (5.0.0-1) ... 2024-09-16T15:39:51.461 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../090-python3-google-auth_1.5.1-3_all.deb ... 2024-09-16T15:39:51.500 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking python3-google-auth (1.5.1-3) ... 2024-09-16T15:39:51.771 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package python3-pyasn1. 2024-09-16T15:39:51.799 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../087-python3-pyasn1_0.4.8-1_all.deb ... 2024-09-16T15:39:51.820 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package python3-requests-oauthlib. 2024-09-16T15:39:51.838 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking python3-pyasn1 (0.4.8-1) ... 2024-09-16T15:39:51.848 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../091-python3-requests-oauthlib_1.3.0+ds-0.1_all.deb ... 2024-09-16T15:39:51.886 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-09-16T15:39:52.181 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package python3-websocket. 2024-09-16T15:39:52.183 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package python3-pyasn1-modules. 2024-09-16T15:39:52.209 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../092-python3-websocket_1.2.3-1_all.deb ... 2024-09-16T15:39:52.211 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../088-python3-pyasn1-modules_0.2.1-1_all.deb ... 2024-09-16T15:39:52.248 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking python3-websocket (1.2.3-1) ... 2024-09-16T15:39:52.249 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking python3-pyasn1-modules (0.2.1-1) ... 2024-09-16T15:39:52.728 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package python3-rsa. 2024-09-16T15:39:52.751 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package python3-kubernetes. 2024-09-16T15:39:52.755 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../089-python3-rsa_4.8-1_all.deb ... 2024-09-16T15:39:52.778 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../093-python3-kubernetes_12.0.1-1ubuntu1_all.deb ... 2024-09-16T15:39:52.802 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking python3-rsa (4.8-1) ... 2024-09-16T15:39:52.817 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking python3-kubernetes (12.0.1-1ubuntu1) ... 2024-09-16T15:39:53.315 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package python3-google-auth. 2024-09-16T15:39:53.343 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../090-python3-google-auth_1.5.1-3_all.deb ... 2024-09-16T15:39:53.381 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking python3-google-auth (1.5.1-3) ... 2024-09-16T15:39:53.448 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2024-09-16T15:39:53.477 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../094-ceph-mgr-k8sevents_19.3.0-4938-g33f95d2a-1jammy_all.deb ... 2024-09-16T15:39:53.506 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking ceph-mgr-k8sevents (19.3.0-4938-g33f95d2a-1jammy) ... 2024-09-16T15:39:53.676 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package python3-requests-oauthlib. 2024-09-16T15:39:53.703 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../091-python3-requests-oauthlib_1.3.0+ds-0.1_all.deb ... 2024-09-16T15:39:53.742 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-09-16T15:39:53.851 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package libonig5:amd64. 2024-09-16T15:39:53.879 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../095-libonig5_6.9.7.1-2build1_amd64.deb ... 2024-09-16T15:39:53.917 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking libonig5:amd64 (6.9.7.1-2build1) ... 2024-09-16T15:39:54.021 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package python3-websocket. 2024-09-16T15:39:54.049 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../092-python3-websocket_1.2.3-1_all.deb ... 2024-09-16T15:39:54.087 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking python3-websocket (1.2.3-1) ... 2024-09-16T15:39:54.220 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package libjq1:amd64. 2024-09-16T15:39:54.248 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../096-libjq1_1.6-2.1ubuntu3_amd64.deb ... 2024-09-16T15:39:54.278 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-09-16T15:39:54.424 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package python3-kubernetes. 2024-09-16T15:39:54.452 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../093-python3-kubernetes_12.0.1-1ubuntu1_all.deb ... 2024-09-16T15:39:54.490 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking python3-kubernetes (12.0.1-1ubuntu1) ... 2024-09-16T15:39:54.531 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package jq. 2024-09-16T15:39:54.559 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../097-jq_1.6-2.1ubuntu3_amd64.deb ... 2024-09-16T15:39:54.597 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking jq (1.6-2.1ubuntu3) ... 2024-09-16T15:39:54.850 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package socat. 2024-09-16T15:39:54.878 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../098-socat_1.7.4.1-3ubuntu4_amd64.deb ... 2024-09-16T15:39:54.916 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking socat (1.7.4.1-3ubuntu4) ... 2024-09-16T15:39:55.138 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2024-09-16T15:39:55.155 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../094-ceph-mgr-k8sevents_19.3.0-4938-g33f95d2a-1jammy_all.deb ... 2024-09-16T15:39:55.194 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package xmlstarlet. 2024-09-16T15:39:55.196 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking ceph-mgr-k8sevents (19.3.0-4938-g33f95d2a-1jammy) ... 2024-09-16T15:39:55.211 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../099-xmlstarlet_1.6.1-2.1_amd64.deb ... 2024-09-16T15:39:55.253 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking xmlstarlet (1.6.1-2.1) ... 2024-09-16T15:39:55.516 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package libonig5:amd64. 2024-09-16T15:39:55.544 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../095-libonig5_6.9.7.1-2build1_amd64.deb ... 2024-09-16T15:39:55.582 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking libonig5:amd64 (6.9.7.1-2build1) ... 2024-09-16T15:39:55.598 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package ceph-test. 2024-09-16T15:39:55.627 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../100-ceph-test_19.3.0-4938-g33f95d2a-1jammy_amd64.deb ... 2024-09-16T15:39:55.665 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking ceph-test (19.3.0-4938-g33f95d2a-1jammy) ... 2024-09-16T15:39:55.944 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package libjq1:amd64. 2024-09-16T15:39:55.972 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../096-libjq1_1.6-2.1ubuntu3_amd64.deb ... 2024-09-16T15:39:56.010 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-09-16T15:39:56.238 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package jq. 2024-09-16T15:39:56.254 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../097-jq_1.6-2.1ubuntu3_amd64.deb ... 2024-09-16T15:39:56.296 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking jq (1.6-2.1ubuntu3) ... 2024-09-16T15:39:56.549 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package socat. 2024-09-16T15:39:56.577 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../098-socat_1.7.4.1-3ubuntu4_amd64.deb ... 2024-09-16T15:39:56.615 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking socat (1.7.4.1-3ubuntu4) ... 2024-09-16T15:39:56.927 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package xmlstarlet. 2024-09-16T15:39:56.943 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../099-xmlstarlet_1.6.1-2.1_amd64.deb ... 2024-09-16T15:39:56.985 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking xmlstarlet (1.6.1-2.1) ... 2024-09-16T15:39:57.306 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package ceph-test. 2024-09-16T15:39:57.322 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../100-ceph-test_19.3.0-4938-g33f95d2a-1jammy_amd64.deb ... 2024-09-16T15:39:57.364 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking ceph-test (19.3.0-4938-g33f95d2a-1jammy) ... 2024-09-16T15:39:57.861 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package ceph-volume. 2024-09-16T15:39:57.889 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../101-ceph-volume_19.3.0-4938-g33f95d2a-1jammy_all.deb ... 2024-09-16T15:39:57.927 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking ceph-volume (19.3.0-4938-g33f95d2a-1jammy) ... 2024-09-16T15:39:58.298 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package cephfs-mirror. 2024-09-16T15:39:58.328 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../102-cephfs-mirror_19.3.0-4938-g33f95d2a-1jammy_amd64.deb ... 2024-09-16T15:39:58.364 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking cephfs-mirror (19.3.0-4938-g33f95d2a-1jammy) ... 2024-09-16T15:39:58.642 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package python3-colorama. 2024-09-16T15:39:58.659 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../103-python3-colorama_0.4.4-1_all.deb ... 2024-09-16T15:39:58.692 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking python3-colorama (0.4.4-1) ... 2024-09-16T15:39:58.936 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package cephfs-shell. 2024-09-16T15:39:58.953 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../104-cephfs-shell_19.3.0-4938-g33f95d2a-1jammy_all.deb ... 2024-09-16T15:39:58.994 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking cephfs-shell (19.3.0-4938-g33f95d2a-1jammy) ... 2024-09-16T15:39:59.247 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package cephfs-top. 2024-09-16T15:39:59.263 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../105-cephfs-top_19.3.0-4938-g33f95d2a-1jammy_all.deb ... 2024-09-16T15:39:59.314 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking cephfs-top (19.3.0-4938-g33f95d2a-1jammy) ... 2024-09-16T15:39:59.627 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package ceph-volume. 2024-09-16T15:39:59.655 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../101-ceph-volume_19.3.0-4938-g33f95d2a-1jammy_all.deb ... 2024-09-16T15:39:59.693 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking ceph-volume (19.3.0-4938-g33f95d2a-1jammy) ... 2024-09-16T15:39:59.783 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package libcephfs-dev. 2024-09-16T15:39:59.811 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../106-libcephfs-dev_19.3.0-4938-g33f95d2a-1jammy_amd64.deb ... 2024-09-16T15:39:59.849 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking libcephfs-dev (19.3.0-4938-g33f95d2a-1jammy) ... 2024-09-16T15:40:00.081 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package cephfs-mirror. 2024-09-16T15:40:00.109 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../102-cephfs-mirror_19.3.0-4938-g33f95d2a-1jammy_amd64.deb ... 2024-09-16T15:40:00.147 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking cephfs-mirror (19.3.0-4938-g33f95d2a-1jammy) ... 2024-09-16T15:40:00.185 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package lua-socket:amd64. 2024-09-16T15:40:00.214 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../107-lua-socket_3.0~rc1+git+ac3201d-6_amd64.deb ... 2024-09-16T15:40:00.251 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2024-09-16T15:40:00.442 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package python3-colorama. 2024-09-16T15:40:00.471 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../103-python3-colorama_0.4.4-1_all.deb ... 2024-09-16T15:40:00.509 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking python3-colorama (0.4.4-1) ... 2024-09-16T15:40:00.548 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package lua-sec:amd64. 2024-09-16T15:40:00.564 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../108-lua-sec_1.0.2-1_amd64.deb ... 2024-09-16T15:40:00.606 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking lua-sec:amd64 (1.0.2-1) ... 2024-09-16T15:40:00.769 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package cephfs-shell. 2024-09-16T15:40:00.785 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../104-cephfs-shell_19.3.0-4938-g33f95d2a-1jammy_all.deb ... 2024-09-16T15:40:00.836 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking cephfs-shell (19.3.0-4938-g33f95d2a-1jammy) ... 2024-09-16T15:40:00.968 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package nvme-cli. 2024-09-16T15:40:00.997 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../109-nvme-cli_1.16-3ubuntu0.3_amd64.deb ... 2024-09-16T15:40:01.034 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking nvme-cli (1.16-3ubuntu0.3) ... 2024-09-16T15:40:01.072 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package cephfs-top. 2024-09-16T15:40:01.088 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../105-cephfs-top_19.3.0-4938-g33f95d2a-1jammy_all.deb ... 2024-09-16T15:40:01.122 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking cephfs-top (19.3.0-4938-g33f95d2a-1jammy) ... 2024-09-16T15:40:01.349 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package libcephfs-dev. 2024-09-16T15:40:01.366 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../106-libcephfs-dev_19.3.0-4938-g33f95d2a-1jammy_amd64.deb ... 2024-09-16T15:40:01.408 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking libcephfs-dev (19.3.0-4938-g33f95d2a-1jammy) ... 2024-09-16T15:40:01.473 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package pkg-config. 2024-09-16T15:40:01.501 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../110-pkg-config_0.29.2-1ubuntu3_amd64.deb ... 2024-09-16T15:40:01.539 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking pkg-config (0.29.2-1ubuntu3) ... 2024-09-16T15:40:01.686 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package lua-socket:amd64. 2024-09-16T15:40:01.714 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../107-lua-socket_3.0~rc1+git+ac3201d-6_amd64.deb ... 2024-09-16T15:40:01.744 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2024-09-16T15:40:01.801 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package python-asyncssh-doc. 2024-09-16T15:40:01.829 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../111-python-asyncssh-doc_2.5.0-1_all.deb ... 2024-09-16T15:40:01.867 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking python-asyncssh-doc (2.5.0-1) ... 2024-09-16T15:40:02.065 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package lua-sec:amd64. 2024-09-16T15:40:02.093 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../108-lua-sec_1.0.2-1_amd64.deb ... 2024-09-16T15:40:02.132 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking lua-sec:amd64 (1.0.2-1) ... 2024-09-16T15:40:02.287 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package python3-iniconfig. 2024-09-16T15:40:02.315 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../112-python3-iniconfig_1.1.1-2_all.deb ... 2024-09-16T15:40:02.353 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking python3-iniconfig (1.1.1-2) ... 2024-09-16T15:40:02.519 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package nvme-cli. 2024-09-16T15:40:02.547 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../109-nvme-cli_1.16-3ubuntu0.3_amd64.deb ... 2024-09-16T15:40:02.585 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking nvme-cli (1.16-3ubuntu0.3) ... 2024-09-16T15:40:02.623 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package python3-pastescript. 2024-09-16T15:40:02.639 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../113-python3-pastescript_2.0.2-4_all.deb ... 2024-09-16T15:40:02.673 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking python3-pastescript (2.0.2-4) ... 2024-09-16T15:40:02.949 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package pkg-config. 2024-09-16T15:40:02.968 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package python3-pluggy. 2024-09-16T15:40:02.978 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../110-pkg-config_0.29.2-1ubuntu3_amd64.deb ... 2024-09-16T15:40:02.996 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../114-python3-pluggy_0.13.0-7.1_all.deb ... 2024-09-16T15:40:03.016 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking pkg-config (0.29.2-1ubuntu3) ... 2024-09-16T15:40:03.034 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking python3-pluggy (0.13.0-7.1) ... 2024-09-16T15:40:03.243 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package python-asyncssh-doc. 2024-09-16T15:40:03.260 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../111-python-asyncssh-doc_2.5.0-1_all.deb ... 2024-09-16T15:40:03.293 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking python-asyncssh-doc (2.5.0-1) ... 2024-09-16T15:40:03.345 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package python3-psutil. 2024-09-16T15:40:03.374 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../115-python3-psutil_5.9.0-1build1_amd64.deb ... 2024-09-16T15:40:03.412 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking python3-psutil (5.9.0-1build1) ... 2024-09-16T15:40:03.748 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package python3-py. 2024-09-16T15:40:03.777 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../116-python3-py_1.10.0-1_all.deb ... 2024-09-16T15:40:03.815 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking python3-py (1.10.0-1) ... 2024-09-16T15:40:03.955 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package python3-iniconfig. 2024-09-16T15:40:03.984 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../112-python3-iniconfig_1.1.1-2_all.deb ... 2024-09-16T15:40:04.013 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking python3-iniconfig (1.1.1-2) ... 2024-09-16T15:40:04.135 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package python3-pygments. 2024-09-16T15:40:04.163 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../117-python3-pygments_2.11.2+dfsg-2_all.deb ... 2024-09-16T15:40:04.201 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking python3-pygments (2.11.2+dfsg-2) ... 2024-09-16T15:40:04.283 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package python3-pastescript. 2024-09-16T15:40:04.310 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../113-python3-pastescript_2.0.2-4_all.deb ... 2024-09-16T15:40:04.349 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking python3-pastescript (2.0.2-4) ... 2024-09-16T15:40:04.647 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package python3-pyinotify. 2024-09-16T15:40:04.669 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package python3-pluggy. 2024-09-16T15:40:04.675 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../118-python3-pyinotify_0.9.6-1.3_all.deb ... 2024-09-16T15:40:04.697 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../114-python3-pluggy_0.13.0-7.1_all.deb ... 2024-09-16T15:40:04.713 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking python3-pyinotify (0.9.6-1.3) ... 2024-09-16T15:40:04.735 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking python3-pluggy (0.13.0-7.1) ... 2024-09-16T15:40:05.008 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package python3-psutil. 2024-09-16T15:40:05.025 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package python3-toml. 2024-09-16T15:40:05.035 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../115-python3-psutil_5.9.0-1build1_amd64.deb ... 2024-09-16T15:40:05.053 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../119-python3-toml_0.10.2-1_all.deb ... 2024-09-16T15:40:05.074 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking python3-psutil (5.9.0-1build1) ... 2024-09-16T15:40:05.082 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking python3-toml (0.10.2-1) ... 2024-09-16T15:40:05.394 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package python3-pytest. 2024-09-16T15:40:05.422 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../120-python3-pytest_6.2.5-1ubuntu2_all.deb ... 2024-09-16T15:40:05.427 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package python3-py. 2024-09-16T15:40:05.455 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../116-python3-py_1.10.0-1_all.deb ... 2024-09-16T15:40:05.460 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking python3-pytest (6.2.5-1ubuntu2) ... 2024-09-16T15:40:05.501 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking python3-py (1.10.0-1) ... 2024-09-16T15:40:05.763 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package python3-simplejson. 2024-09-16T15:40:05.789 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../121-python3-simplejson_3.17.6-1build1_amd64.deb ... 2024-09-16T15:40:05.830 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking python3-simplejson (3.17.6-1build1) ... 2024-09-16T15:40:05.847 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package python3-pygments. 2024-09-16T15:40:05.875 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../117-python3-pygments_2.11.2+dfsg-2_all.deb ... 2024-09-16T15:40:05.922 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking python3-pygments (2.11.2+dfsg-2) ... 2024-09-16T15:40:06.091 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package qttranslations5-l10n. 2024-09-16T15:40:06.108 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../122-qttranslations5-l10n_5.15.3-1_all.deb ... 2024-09-16T15:40:06.141 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking qttranslations5-l10n (5.15.3-1) ... 2024-09-16T15:40:06.351 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package python3-pyinotify. 2024-09-16T15:40:06.380 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../118-python3-pyinotify_0.9.6-1.3_all.deb ... 2024-09-16T15:40:06.409 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking python3-pyinotify (0.9.6-1.3) ... 2024-09-16T15:40:06.712 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package python3-toml. 2024-09-16T15:40:06.740 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../119-python3-toml_0.10.2-1_all.deb ... 2024-09-16T15:40:06.778 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking python3-toml (0.10.2-1) ... 2024-09-16T15:40:06.845 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package radosgw. 2024-09-16T15:40:06.874 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../123-radosgw_19.3.0-4938-g33f95d2a-1jammy_amd64.deb ... 2024-09-16T15:40:06.911 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking radosgw (19.3.0-4938-g33f95d2a-1jammy) ... 2024-09-16T15:40:07.090 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package python3-pytest. 2024-09-16T15:40:07.118 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../120-python3-pytest_6.2.5-1ubuntu2_all.deb ... 2024-09-16T15:40:07.156 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking python3-pytest (6.2.5-1ubuntu2) ... 2024-09-16T15:40:07.477 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package python3-simplejson. 2024-09-16T15:40:07.505 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../121-python3-simplejson_3.17.6-1build1_amd64.deb ... 2024-09-16T15:40:07.543 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking python3-simplejson (3.17.6-1build1) ... 2024-09-16T15:40:07.806 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package rbd-fuse. 2024-09-16T15:40:07.812 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package qttranslations5-l10n. 2024-09-16T15:40:07.829 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../122-qttranslations5-l10n_5.15.3-1_all.deb ... 2024-09-16T15:40:07.835 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../124-rbd-fuse_19.3.0-4938-g33f95d2a-1jammy_amd64.deb ... 2024-09-16T15:40:07.870 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking qttranslations5-l10n (5.15.3-1) ... 2024-09-16T15:40:07.872 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking rbd-fuse (19.3.0-4938-g33f95d2a-1jammy) ... 2024-09-16T15:40:08.169 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up python3-iniconfig (1.1.1-2) ... 2024-09-16T15:40:08.402 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up libdouble-conversion3:amd64 (3.1.7-4) ... 2024-09-16T15:40:08.528 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up nvme-cli (1.16-3ubuntu0.3) ... 2024-09-16T15:40:08.625 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package radosgw. 2024-09-16T15:40:08.653 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../123-radosgw_19.3.0-4938-g33f95d2a-1jammy_amd64.deb ... 2024-09-16T15:40:08.691 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking radosgw (19.3.0-4938-g33f95d2a-1jammy) ... 2024-09-16T15:40:08.780 INFO:teuthology.orchestra.run.smithi123.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmefc-boot-connections.service -> /lib/systemd/system/nvmefc-boot-connections.service. 2024-09-16T15:40:09.128 INFO:teuthology.orchestra.run.smithi123.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmf-autoconnect.service -> /lib/systemd/system/nvmf-autoconnect.service. 2024-09-16T15:40:09.559 INFO:teuthology.orchestra.run.smithi123.stdout:nvmf-connect.target is a disabled or a static unit, not starting it. 2024-09-16T15:40:09.646 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up cephadm (19.3.0-4938-g33f95d2a-1jammy) ... 2024-09-16T15:40:09.652 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package rbd-fuse. 2024-09-16T15:40:09.669 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../124-rbd-fuse_19.3.0-4938-g33f95d2a-1jammy_amd64.deb ... 2024-09-16T15:40:09.710 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking rbd-fuse (19.3.0-4938-g33f95d2a-1jammy) ... 2024-09-16T15:40:10.008 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up python3-iniconfig (1.1.1-2) ... 2024-09-16T15:40:10.258 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up libdouble-conversion3:amd64 (3.1.7-4) ... 2024-09-16T15:40:10.281 INFO:teuthology.orchestra.run.smithi123.stdout:Adding system user cephadm....done 2024-09-16T15:40:10.375 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up nvme-cli (1.16-3ubuntu0.3) ... 2024-09-16T15:40:10.479 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up python3-waitress (1.4.4-1.1ubuntu1) ... 2024-09-16T15:40:10.620 INFO:teuthology.orchestra.run.smithi184.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmefc-boot-connections.service -> /lib/systemd/system/nvmefc-boot-connections.service. 2024-09-16T15:40:10.932 INFO:teuthology.orchestra.run.smithi184.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmf-autoconnect.service -> /lib/systemd/system/nvmf-autoconnect.service. 2024-09-16T15:40:11.048 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up python3-jaraco.classes (3.2.1-3) ... 2024-09-16T15:40:11.299 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up python-asyncssh-doc (2.5.0-1) ... 2024-09-16T15:40:11.355 INFO:teuthology.orchestra.run.smithi184.stdout:nvmf-connect.target is a disabled or a static unit, not starting it. 2024-09-16T15:40:11.419 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up cephadm (19.3.0-4938-g33f95d2a-1jammy) ... 2024-09-16T15:40:11.425 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up python3-jaraco.functools (3.4.0-2) ... 2024-09-16T15:40:11.701 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up python3-repoze.lru (0.7-2) ... 2024-09-16T15:40:11.936 INFO:teuthology.orchestra.run.smithi184.stdout:Adding system user cephadm....done 2024-09-16T15:40:11.977 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up liboath0:amd64 (2.6.7-3build1) ... 2024-09-16T15:40:12.050 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up python3-waitress (1.4.4-1.1ubuntu1) ... 2024-09-16T15:40:12.103 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up python3-py (1.10.0-1) ... 2024-09-16T15:40:12.310 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up python3-jaraco.classes (3.2.1-3) ... 2024-09-16T15:40:12.413 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up python3-joblib (0.17.0-4ubuntu1) ... 2024-09-16T15:40:12.569 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up python-asyncssh-doc (2.5.0-1) ... 2024-09-16T15:40:12.695 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up python3-jaraco.functools (3.4.0-2) ... 2024-09-16T15:40:12.807 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up python3-cachetools (5.0.0-1) ... 2024-09-16T15:40:12.955 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up python3-repoze.lru (0.7-2) ... 2024-09-16T15:40:13.067 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up python3-colorama (0.4.4-1) ... 2024-09-16T15:40:13.231 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up liboath0:amd64 (2.6.7-3build1) ... 2024-09-16T15:40:13.326 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up python3-openssl (21.0.0-1) ... 2024-09-16T15:40:13.349 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up python3-py (1.10.0-1) ... 2024-09-16T15:40:13.602 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up unzip (6.0-26ubuntu3.2) ... 2024-09-16T15:40:13.634 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up python3-joblib (0.17.0-4ubuntu1) ... 2024-09-16T15:40:13.912 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up python3-bcrypt (3.2.0-1build1) ... 2024-09-16T15:40:13.977 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up python3-cachetools (5.0.0-1) ... 2024-09-16T15:40:14.189 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up python3-pyinotify (0.9.6-1.3) ... 2024-09-16T15:40:14.448 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up python3-threadpoolctl (3.1.0-1) ... 2024-09-16T15:40:14.487 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up python3-colorama (0.4.4-1) ... 2024-09-16T15:40:14.699 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up python3-ceph-argparse (19.3.0-4938-g33f95d2a-1jammy) ... 2024-09-16T15:40:14.755 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up python3-openssl (21.0.0-1) ... 2024-09-16T15:40:14.958 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-09-16T15:40:15.023 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up unzip (6.0-26ubuntu3.2) ... 2024-09-16T15:40:15.167 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up libnbd0 (1.10.5-1) ... 2024-09-16T15:40:15.290 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up python3-bcrypt (3.2.0-1build1) ... 2024-09-16T15:40:15.551 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up python3-pyinotify (0.9.6-1.3) ... 2024-09-16T15:40:15.552 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2024-09-16T15:40:15.678 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up python3-pyperclip (1.8.2-2) ... 2024-09-16T15:40:15.869 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up python3-threadpoolctl (3.1.0-1) ... 2024-09-16T15:40:15.945 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up libreadline-dev:amd64 (8.1.2-1) ... 2024-09-16T15:40:16.071 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up python3-markupsafe (2.0.1-2build1) ... 2024-09-16T15:40:16.220 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up python3-ceph-argparse (19.3.0-4938-g33f95d2a-1jammy) ... 2024-09-16T15:40:16.339 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up lua5.1 (5.1.5-8.1build4) ... 2024-09-16T15:40:16.479 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-09-16T15:40:16.631 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up libnbd0 (1.10.5-1) ... 2024-09-16T15:40:16.756 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2024-09-16T15:40:16.842 INFO:teuthology.orchestra.run.smithi123.stdout:update-alternatives: using /usr/bin/lua5.1 to provide /usr/bin/lua (lua-interpreter) in auto mode 2024-09-16T15:40:16.866 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up python3-pyperclip (1.8.2-2) ... 2024-09-16T15:40:16.887 INFO:teuthology.orchestra.run.smithi123.stdout:update-alternatives: using /usr/bin/luac5.1 to provide /usr/bin/luac (lua-compiler) in auto mode 2024-09-16T15:40:16.959 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-09-16T15:40:17.085 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up python3-psutil (5.9.0-1build1) ... 2024-09-16T15:40:17.108 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up libreadline-dev:amd64 (8.1.2-1) ... 2024-09-16T15:40:17.218 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up python3-markupsafe (2.0.1-2build1) ... 2024-09-16T15:40:17.445 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-09-16T15:40:17.477 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up lua5.1 (5.1.5-8.1build4) ... 2024-09-16T15:40:17.570 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up python3-natsort (8.0.2-1) ... 2024-09-16T15:40:17.830 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up python3-routes (2.5.1-1ubuntu1) ... 2024-09-16T15:40:17.841 INFO:teuthology.orchestra.run.smithi184.stdout:update-alternatives: using /usr/bin/lua5.1 to provide /usr/bin/lua (lua-interpreter) in auto mode 2024-09-16T15:40:17.875 INFO:teuthology.orchestra.run.smithi184.stdout:update-alternatives: using /usr/bin/luac5.1 to provide /usr/bin/luac (lua-compiler) in auto mode 2024-09-16T15:40:17.939 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-09-16T15:40:18.065 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up python3-psutil (5.9.0-1build1) ... 2024-09-16T15:40:18.089 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up python3-simplejson (3.17.6-1build1) ... 2024-09-16T15:40:18.399 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up zip (3.0-12build2) ... 2024-09-16T15:40:18.408 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-09-16T15:40:18.509 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up python3-natsort (8.0.2-1) ... 2024-09-16T15:40:18.525 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up python3-pygments (2.11.2+dfsg-2) ... 2024-09-16T15:40:18.768 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up python3-routes (2.5.1-1ubuntu1) ... 2024-09-16T15:40:19.036 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up python3-simplejson (3.17.6-1build1) ... 2024-09-16T15:40:19.168 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up python3-tempita (0.5.2-6ubuntu1) ... 2024-09-16T15:40:19.363 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up zip (3.0-12build2) ... 2024-09-16T15:40:19.428 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up python-pastedeploy-tpl (2.1.1-1) ... 2024-09-16T15:40:19.497 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up python3-pygments (2.11.2+dfsg-2) ... 2024-09-16T15:40:19.554 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up qttranslations5-l10n (5.15.3-1) ... 2024-09-16T15:40:19.663 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up python3-wcwidth (0.2.5+dfsg1-1) ... 2024-09-16T15:40:19.956 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up python3-asyncssh (2.5.0-1) ... 2024-09-16T15:40:20.182 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up python3-tempita (0.5.2-6ubuntu1) ... 2024-09-16T15:40:20.341 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up python3-certifi (2020.6.20-1) ... 2024-09-16T15:40:20.442 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up python-pastedeploy-tpl (2.1.1-1) ... 2024-09-16T15:40:20.551 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up qttranslations5-l10n (5.15.3-1) ... 2024-09-16T15:40:20.591 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up python3-paste (3.5.0+dfsg1-1) ... 2024-09-16T15:40:20.677 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up python3-wcwidth (0.2.5+dfsg1-1) ... 2024-09-16T15:40:20.960 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-09-16T15:40:21.003 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up python3-asyncssh (2.5.0-1) ... 2024-09-16T15:40:21.244 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up python3-cmd2 (2.3.3-1) ... 2024-09-16T15:40:21.396 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up python3-certifi (2020.6.20-1) ... 2024-09-16T15:40:21.553 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.2) ... 2024-09-16T15:40:21.655 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up python3-paste (3.5.0+dfsg1-1) ... 2024-09-16T15:40:21.890 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up python3-jaraco.text (3.6.0-2) ... 2024-09-16T15:40:22.049 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-09-16T15:40:22.141 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up socat (1.7.4.1-3ubuntu4) ... 2024-09-16T15:40:22.251 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up python3-ceph-common (19.3.0-4938-g33f95d2a-1jammy) ... 2024-09-16T15:40:22.333 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up python3-cmd2 (2.3.3-1) ... 2024-09-16T15:40:22.544 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up python3-idna (3.3-1ubuntu0.1) ... 2024-09-16T15:40:22.626 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.2) ... 2024-09-16T15:40:22.861 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up python3-sklearn (0.23.2-5ubuntu6) ... 2024-09-16T15:40:22.961 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up python3-jaraco.text (3.6.0-2) ... 2024-09-16T15:40:23.220 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up socat (1.7.4.1-3ubuntu4) ... 2024-09-16T15:40:23.347 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up python3-ceph-common (19.3.0-4938-g33f95d2a-1jammy) ... 2024-09-16T15:40:23.656 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up python3-idna (3.3-1ubuntu0.1) ... 2024-09-16T15:40:23.974 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up python3-sklearn (0.23.2-5ubuntu6) ... 2024-09-16T15:40:24.050 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up pkg-config (0.29.2-1ubuntu3) ... 2024-09-16T15:40:24.327 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-16T15:40:24.461 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up python3-toml (0.10.2-1) ... 2024-09-16T15:40:24.754 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up librdkafka1:amd64 (1.8.0-1build1) ... 2024-09-16T15:40:24.863 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up xmlstarlet (1.6.1-2.1) ... 2024-09-16T15:40:24.964 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-09-16T15:40:25.164 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up pkg-config (0.29.2-1ubuntu3) ... 2024-09-16T15:40:25.274 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up python3-pluggy (0.13.0-7.1) ... 2024-09-16T15:40:25.407 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-16T15:40:25.525 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up python3-zc.lockfile (2.0-1) ... 2024-09-16T15:40:25.533 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up python3-toml (0.10.2-1) ... 2024-09-16T15:40:25.767 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-16T15:40:25.784 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up librdkafka1:amd64 (1.8.0-1build1) ... 2024-09-16T15:40:25.893 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up python3-pyasn1 (0.4.8-1) ... 2024-09-16T15:40:25.910 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up xmlstarlet (1.6.1-2.1) ... 2024-09-16T15:40:26.036 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-09-16T15:40:26.187 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up python3-singledispatch (3.4.0.3-3) ... 2024-09-16T15:40:26.329 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up python3-pluggy (0.13.0-7.1) ... 2024-09-16T15:40:26.421 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up python3-logutils (0.3.3-8) ... 2024-09-16T15:40:26.589 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up python3-zc.lockfile (2.0-1) ... 2024-09-16T15:40:26.672 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up python3-tempora (4.1.2-1) ... 2024-09-16T15:40:26.840 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-16T15:40:26.915 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up python3-simplegeneric (0.8.1-3) ... 2024-09-16T15:40:26.966 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up python3-pyasn1 (0.4.8-1) ... 2024-09-16T15:40:27.191 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up python3-prettytable (2.5.0-2) ... 2024-09-16T15:40:27.242 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up python3-singledispatch (3.4.0.3-3) ... 2024-09-16T15:40:27.450 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-09-16T15:40:27.527 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up python3-logutils (0.3.3-8) ... 2024-09-16T15:40:27.576 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up python3-websocket (1.2.3-1) ... 2024-09-16T15:40:27.786 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up python3-tempora (4.1.2-1) ... 2024-09-16T15:40:27.860 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up libonig5:amd64 (6.9.7.1-2build1) ... 2024-09-16T15:40:27.987 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-09-16T15:40:28.037 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up python3-simplegeneric (0.8.1-3) ... 2024-09-16T15:40:28.271 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up python3-webob (1:1.8.6-1.1ubuntu0.1) ... 2024-09-16T15:40:28.288 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up python3-prettytable (2.5.0-2) ... 2024-09-16T15:40:28.547 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up python3-jaraco.collections (3.4.0-2) ... 2024-09-16T15:40:28.548 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-09-16T15:40:28.649 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up python3-websocket (1.2.3-1) ... 2024-09-16T15:40:28.814 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up liblua5.3-dev:amd64 (5.3.6-1build1) ... 2024-09-16T15:40:28.917 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up libonig5:amd64 (6.9.7.1-2build1) ... 2024-09-16T15:40:28.932 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up lua-sec:amd64 (1.0.2-1) ... 2024-09-16T15:40:29.042 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up python3-babel (2.8.0+dfsg.1-7) ... 2024-09-16T15:40:29.093 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-09-16T15:40:29.115 INFO:teuthology.orchestra.run.smithi123.stdout:update-alternatives: using /usr/bin/pybabel-python3 to provide /usr/bin/pybabel (pybabel) in auto mode 2024-09-16T15:40:29.360 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-09-16T15:40:29.386 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up python3-webob (1:1.8.6-1.1ubuntu0.1) ... 2024-09-16T15:40:29.486 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up python3-pytest (6.2.5-1ubuntu2) ... 2024-09-16T15:40:29.679 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up python3-jaraco.collections (3.4.0-2) ... 2024-09-16T15:40:29.871 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up python3-pastedeploy (2.1.1-1) ... 2024-09-16T15:40:29.930 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up liblua5.3-dev:amd64 (5.3.6-1build1) ... 2024-09-16T15:40:30.056 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up lua-sec:amd64 (1.0.2-1) ... 2024-09-16T15:40:30.130 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up lua-any (27ubuntu1) ... 2024-09-16T15:40:30.165 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up python3-babel (2.8.0+dfsg.1-7) ... 2024-09-16T15:40:30.237 INFO:teuthology.orchestra.run.smithi184.stdout:update-alternatives: using /usr/bin/pybabel-python3 to provide /usr/bin/pybabel (pybabel) in auto mode 2024-09-16T15:40:30.256 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up python3-portend (3.0.0-1) ... 2024-09-16T15:40:30.459 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-09-16T15:40:30.524 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-16T15:40:30.585 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up python3-pytest (6.2.5-1ubuntu2) ... 2024-09-16T15:40:30.650 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up python3-jinja2 (3.0.3-1ubuntu0.2) ... 2024-09-16T15:40:30.976 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up python3-pyasn1-modules (0.2.1-1) ... 2024-09-16T15:40:30.978 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up python3-pastedeploy (2.1.1-1) ... 2024-09-16T15:40:31.237 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up lua-any (27ubuntu1) ... 2024-09-16T15:40:31.277 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-09-16T15:40:31.372 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up python3-portend (3.0.0-1) ... 2024-09-16T15:40:31.630 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up jq (1.6-2.1ubuntu3) ... 2024-09-16T15:40:31.631 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-16T15:40:31.749 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up python3-jinja2 (3.0.3-1ubuntu0.2) ... 2024-09-16T15:40:31.756 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up python3-rsa (4.8-1) ... 2024-09-16T15:40:32.033 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up python3-webtest (2.0.35-1) ... 2024-09-16T15:40:32.033 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up python3-pyasn1-modules (0.2.1-1) ... 2024-09-16T15:40:32.292 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-09-16T15:40:32.326 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-09-16T15:40:32.552 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up python3-cherrypy3 (18.6.1-4) ... 2024-09-16T15:40:32.688 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up jq (1.6-2.1ubuntu3) ... 2024-09-16T15:40:32.822 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up python3-rsa (4.8-1) ... 2024-09-16T15:40:32.937 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up python3-pastescript (2.0.2-4) ... 2024-09-16T15:40:33.074 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up python3-webtest (2.0.35-1) ... 2024-09-16T15:40:33.213 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up python3-pecan (1.3.3-4ubuntu2) ... 2024-09-16T15:40:33.317 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-09-16T15:40:33.549 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-09-16T15:40:33.576 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up python3-cherrypy3 (18.6.1-4) ... 2024-09-16T15:40:33.675 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up librados2 (19.3.0-4938-g33f95d2a-1jammy) ... 2024-09-16T15:40:33.801 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up libsqlite3-mod-ceph (19.3.0-4938-g33f95d2a-1jammy) ... 2024-09-16T15:40:33.927 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up luarocks (3.8.0+dfsg1-1) ... 2024-09-16T15:40:33.953 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up python3-pastescript (2.0.2-4) ... 2024-09-16T15:40:34.246 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up libcephfs2 (19.3.0-4938-g33f95d2a-1jammy) ... 2024-09-16T15:40:34.255 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up python3-pecan (1.3.3-4ubuntu2) ... 2024-09-16T15:40:34.372 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up libradosstriper1 (19.3.0-4938-g33f95d2a-1jammy) ... 2024-09-16T15:40:34.498 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up python3-google-auth (1.5.1-3) ... 2024-09-16T15:40:34.590 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-09-16T15:40:34.717 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up librados2 (19.3.0-4938-g33f95d2a-1jammy) ... 2024-09-16T15:40:34.775 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up librbd1 (19.3.0-4938-g33f95d2a-1jammy) ... 2024-09-16T15:40:34.842 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up libsqlite3-mod-ceph (19.3.0-4938-g33f95d2a-1jammy) ... 2024-09-16T15:40:34.901 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up ceph-mgr-modules-core (19.3.0-4938-g33f95d2a-1jammy) ... 2024-09-16T15:40:34.968 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up luarocks (3.8.0+dfsg1-1) ... 2024-09-16T15:40:35.027 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up ceph-fuse (19.3.0-4938-g33f95d2a-1jammy) ... 2024-09-16T15:40:35.237 INFO:teuthology.orchestra.run.smithi123.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-09-16T15:40:35.238 INFO:teuthology.orchestra.run.smithi123.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-09-16T15:40:35.255 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up libcephfs2 (19.3.0-4938-g33f95d2a-1jammy) ... 2024-09-16T15:40:35.381 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up libradosstriper1 (19.3.0-4938-g33f95d2a-1jammy) ... 2024-09-16T15:40:35.532 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up python3-google-auth (1.5.1-3) ... 2024-09-16T15:40:35.703 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up libcephfs-dev (19.3.0-4938-g33f95d2a-1jammy) ... 2024-09-16T15:40:35.809 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up librbd1 (19.3.0-4938-g33f95d2a-1jammy) ... 2024-09-16T15:40:35.854 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up python3-rados (19.3.0-4938-g33f95d2a-1jammy) ... 2024-09-16T15:40:35.935 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up ceph-mgr-modules-core (19.3.0-4938-g33f95d2a-1jammy) ... 2024-09-16T15:40:35.955 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up librgw2 (19.3.0-4938-g33f95d2a-1jammy) ... 2024-09-16T15:40:36.061 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up ceph-fuse (19.3.0-4938-g33f95d2a-1jammy) ... 2024-09-16T15:40:36.064 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up cephfs-top (19.3.0-4938-g33f95d2a-1jammy) ... 2024-09-16T15:40:36.199 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up python3-kubernetes (12.0.1-1ubuntu1) ... 2024-09-16T15:40:36.275 INFO:teuthology.orchestra.run.smithi184.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-09-16T15:40:36.275 INFO:teuthology.orchestra.run.smithi184.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-09-16T15:40:36.771 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up libcephfs-dev (19.3.0-4938-g33f95d2a-1jammy) ... 2024-09-16T15:40:36.880 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up python3-rados (19.3.0-4938-g33f95d2a-1jammy) ... 2024-09-16T15:40:37.006 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up librgw2 (19.3.0-4938-g33f95d2a-1jammy) ... 2024-09-16T15:40:37.132 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up cephfs-top (19.3.0-4938-g33f95d2a-1jammy) ... 2024-09-16T15:40:37.258 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up python3-kubernetes (12.0.1-1ubuntu1) ... 2024-09-16T15:40:37.402 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up python3-rbd (19.3.0-4938-g33f95d2a-1jammy) ... 2024-09-16T15:40:37.545 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up rbd-fuse (19.3.0-4938-g33f95d2a-1jammy) ... 2024-09-16T15:40:37.671 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up python3-rgw (19.3.0-4938-g33f95d2a-1jammy) ... 2024-09-16T15:40:37.797 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up python3-cephfs (19.3.0-4938-g33f95d2a-1jammy) ... 2024-09-16T15:40:37.923 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up cephfs-shell (19.3.0-4938-g33f95d2a-1jammy) ... 2024-09-16T15:40:38.049 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up ceph-common (19.3.0-4938-g33f95d2a-1jammy) ... 2024-09-16T15:40:38.486 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up python3-rbd (19.3.0-4938-g33f95d2a-1jammy) ... 2024-09-16T15:40:38.567 INFO:teuthology.orchestra.run.smithi123.stdout:Adding group ceph....done 2024-09-16T15:40:38.587 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up rbd-fuse (19.3.0-4938-g33f95d2a-1jammy) ... 2024-09-16T15:40:38.714 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up python3-rgw (19.3.0-4938-g33f95d2a-1jammy) ... 2024-09-16T15:40:38.831 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up python3-cephfs (19.3.0-4938-g33f95d2a-1jammy) ... 2024-09-16T15:40:38.932 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up cephfs-shell (19.3.0-4938-g33f95d2a-1jammy) ... 2024-09-16T15:40:38.993 INFO:teuthology.orchestra.run.smithi123.stdout:Adding system user ceph....done 2024-09-16T15:40:39.058 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up ceph-common (19.3.0-4938-g33f95d2a-1jammy) ... 2024-09-16T15:40:39.162 INFO:teuthology.orchestra.run.smithi123.stdout:Setting system user ceph properties....done 2024-09-16T15:40:39.174 INFO:teuthology.orchestra.run.smithi123.stdout:chown: cannot access '/var/log/ceph/*.log*': No such file or directory 2024-09-16T15:40:39.312 INFO:teuthology.orchestra.run.smithi123.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2024-09-16T15:40:39.577 INFO:teuthology.orchestra.run.smithi184.stdout:Adding group ceph....done 2024-09-16T15:40:39.638 INFO:teuthology.orchestra.run.smithi123.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2024-09-16T15:40:39.970 INFO:teuthology.orchestra.run.smithi184.stdout:Adding system user ceph....done 2024-09-16T15:40:40.109 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up cephfs-mirror (19.3.0-4938-g33f95d2a-1jammy) ... 2024-09-16T15:40:40.114 INFO:teuthology.orchestra.run.smithi184.stdout:Setting system user ceph properties....done 2024-09-16T15:40:40.126 INFO:teuthology.orchestra.run.smithi184.stdout:chown: cannot access '/var/log/ceph/*.log*': No such file or directory 2024-09-16T15:40:40.263 INFO:teuthology.orchestra.run.smithi184.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2024-09-16T15:40:40.319 INFO:teuthology.orchestra.run.smithi123.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target -> /lib/systemd/system/cephfs-mirror.target. 2024-09-16T15:40:40.319 INFO:teuthology.orchestra.run.smithi123.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target -> /lib/systemd/system/cephfs-mirror.target. 2024-09-16T15:40:40.636 INFO:teuthology.orchestra.run.smithi184.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2024-09-16T15:40:40.785 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up ceph-test (19.3.0-4938-g33f95d2a-1jammy) ... 2024-09-16T15:40:40.903 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up radosgw (19.3.0-4938-g33f95d2a-1jammy) ... 2024-09-16T15:40:41.102 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up cephfs-mirror (19.3.0-4938-g33f95d2a-1jammy) ... 2024-09-16T15:40:41.301 INFO:teuthology.orchestra.run.smithi184.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target -> /lib/systemd/system/cephfs-mirror.target. 2024-09-16T15:40:41.301 INFO:teuthology.orchestra.run.smithi184.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target -> /lib/systemd/system/cephfs-mirror.target. 2024-09-16T15:40:41.753 INFO:teuthology.orchestra.run.smithi123.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-09-16T15:40:41.753 INFO:teuthology.orchestra.run.smithi123.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-09-16T15:40:41.803 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up ceph-test (19.3.0-4938-g33f95d2a-1jammy) ... 2024-09-16T15:40:41.921 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up radosgw (19.3.0-4938-g33f95d2a-1jammy) ... 2024-09-16T15:40:42.244 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up ceph-base (19.3.0-4938-g33f95d2a-1jammy) ... 2024-09-16T15:40:42.380 INFO:teuthology.orchestra.run.smithi184.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-09-16T15:40:42.380 INFO:teuthology.orchestra.run.smithi184.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-09-16T15:40:42.736 INFO:teuthology.orchestra.run.smithi123.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2024-09-16T15:40:42.856 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up ceph-base (19.3.0-4938-g33f95d2a-1jammy) ... 2024-09-16T15:40:43.202 INFO:teuthology.orchestra.run.smithi184.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2024-09-16T15:40:43.218 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up ceph-mds (19.3.0-4938-g33f95d2a-1jammy) ... 2024-09-16T15:40:43.448 INFO:teuthology.orchestra.run.smithi123.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-09-16T15:40:43.448 INFO:teuthology.orchestra.run.smithi123.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-09-16T15:40:43.658 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up ceph-mds (19.3.0-4938-g33f95d2a-1jammy) ... 2024-09-16T15:40:43.874 INFO:teuthology.orchestra.run.smithi184.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-09-16T15:40:43.874 INFO:teuthology.orchestra.run.smithi184.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-09-16T15:40:43.936 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up ceph-mgr (19.3.0-4938-g33f95d2a-1jammy) ... 2024-09-16T15:40:44.144 INFO:teuthology.orchestra.run.smithi123.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-09-16T15:40:44.144 INFO:teuthology.orchestra.run.smithi123.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-09-16T15:40:44.335 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up ceph-mgr (19.3.0-4938-g33f95d2a-1jammy) ... 2024-09-16T15:40:44.561 INFO:teuthology.orchestra.run.smithi184.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-09-16T15:40:44.561 INFO:teuthology.orchestra.run.smithi184.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-09-16T15:40:44.604 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up ceph-osd (19.3.0-4938-g33f95d2a-1jammy) ... 2024-09-16T15:40:44.878 INFO:teuthology.orchestra.run.smithi123.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-09-16T15:40:44.878 INFO:teuthology.orchestra.run.smithi123.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-09-16T15:40:45.036 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up ceph-osd (19.3.0-4938-g33f95d2a-1jammy) ... 2024-09-16T15:40:45.310 INFO:teuthology.orchestra.run.smithi184.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-09-16T15:40:45.310 INFO:teuthology.orchestra.run.smithi184.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-09-16T15:40:45.364 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up ceph-mgr-k8sevents (19.3.0-4938-g33f95d2a-1jammy) ... 2024-09-16T15:40:45.490 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up ceph-mgr-diskprediction-local (19.3.0-4938-g33f95d2a-1jammy) ... 2024-09-16T15:40:45.641 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up ceph-mon (19.3.0-4938-g33f95d2a-1jammy) ... 2024-09-16T15:40:45.852 INFO:teuthology.orchestra.run.smithi123.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-09-16T15:40:45.852 INFO:teuthology.orchestra.run.smithi123.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-09-16T15:40:46.089 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up ceph-mgr-k8sevents (19.3.0-4938-g33f95d2a-1jammy) ... 2024-09-16T15:40:46.198 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up ceph-mgr-diskprediction-local (19.3.0-4938-g33f95d2a-1jammy) ... 2024-09-16T15:40:46.342 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up ceph-mgr-cephadm (19.3.0-4938-g33f95d2a-1jammy) ... 2024-09-16T15:40:46.349 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up ceph-mon (19.3.0-4938-g33f95d2a-1jammy) ... 2024-09-16T15:40:46.493 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up ceph (19.3.0-4938-g33f95d2a-1jammy) ... 2024-09-16T15:40:46.567 INFO:teuthology.orchestra.run.smithi184.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-09-16T15:40:46.567 INFO:teuthology.orchestra.run.smithi184.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-09-16T15:40:47.024 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up ceph-mgr-dashboard (19.3.0-4938-g33f95d2a-1jammy) ... 2024-09-16T15:40:47.076 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up ceph-mgr-cephadm (19.3.0-4938-g33f95d2a-1jammy) ... 2024-09-16T15:40:47.175 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up ceph-volume (19.3.0-4938-g33f95d2a-1jammy) ... 2024-09-16T15:40:47.210 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up ceph (19.3.0-4938-g33f95d2a-1jammy) ... 2024-09-16T15:40:47.311 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up ceph-mgr-dashboard (19.3.0-4938-g33f95d2a-1jammy) ... 2024-09-16T15:40:47.437 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up ceph-volume (19.3.0-4938-g33f95d2a-1jammy) ... 2024-09-16T15:40:47.520 INFO:teuthology.orchestra.run.smithi123.stdout:Processing triggers for mailcap (3.70+nmu1ubuntu1) ... 2024-09-16T15:40:47.620 INFO:teuthology.orchestra.run.smithi123.stdout:Processing triggers for libc-bin (2.35-0ubuntu3.1) ... 2024-09-16T15:40:47.739 INFO:teuthology.orchestra.run.smithi123.stdout:Processing triggers for man-db (2.10.2-1) ... 2024-09-16T15:40:47.798 INFO:teuthology.orchestra.run.smithi184.stdout:Processing triggers for mailcap (3.70+nmu1ubuntu1) ... 2024-09-16T15:40:47.900 INFO:teuthology.orchestra.run.smithi184.stdout:Processing triggers for libc-bin (2.35-0ubuntu3.1) ... 2024-09-16T15:40:48.052 INFO:teuthology.orchestra.run.smithi184.stdout:Processing triggers for man-db (2.10.2-1) ... 2024-09-16T15:40:48.481 INFO:teuthology.orchestra.run.smithi123.stdout:Processing triggers for install-info (6.8-4build1) ... 2024-09-16T15:40:49.121 INFO:teuthology.orchestra.run.smithi123.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2024-09-16T15:40:49.124 DEBUG:teuthology.orchestra.run.smithi123:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install bison flex libelf-dev libssl-dev network-manager iproute2 util-linux dump indent libaio-dev libtool-bin uuid-dev xfslibs-dev postgresql postgresql-client postgresql-common postgresql-contrib 2024-09-16T15:40:49.181 INFO:teuthology.orchestra.run.smithi123.stdout:Reading package lists... 2024-09-16T15:40:49.195 INFO:teuthology.orchestra.run.smithi184.stdout:Processing triggers for install-info (6.8-4build1) ... 2024-09-16T15:40:49.381 INFO:teuthology.orchestra.run.smithi123.stdout:Building dependency tree... 2024-09-16T15:40:49.382 INFO:teuthology.orchestra.run.smithi123.stdout:Reading state information... 2024-09-16T15:40:49.563 INFO:teuthology.orchestra.run.smithi123.stdout:iproute2 is already the newest version (5.15.0-1ubuntu2). 2024-09-16T15:40:49.563 INFO:teuthology.orchestra.run.smithi123.stdout:libtool-bin is already the newest version (2.4.6-15build2). 2024-09-16T15:40:49.563 INFO:teuthology.orchestra.run.smithi123.stdout:xfslibs-dev is already the newest version (5.13.0-1ubuntu2). 2024-09-16T15:40:49.563 INFO:teuthology.orchestra.run.smithi123.stdout:The following packages were automatically installed and are no longer required: 2024-09-16T15:40:49.564 INFO:teuthology.orchestra.run.smithi123.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-09-16T15:40:49.564 INFO:teuthology.orchestra.run.smithi123.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-09-16T15:40:49.564 INFO:teuthology.orchestra.run.smithi123.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-09-16T15:40:49.564 INFO:teuthology.orchestra.run.smithi123.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-09-16T15:40:49.564 INFO:teuthology.orchestra.run.smithi123.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-09-16T15:40:49.564 INFO:teuthology.orchestra.run.smithi123.stdout:Use 'sudo apt autoremove' to remove them. 2024-09-16T15:40:49.565 INFO:teuthology.orchestra.run.smithi123.stdout:The following additional packages will be installed: 2024-09-16T15:40:49.565 INFO:teuthology.orchestra.run.smithi123.stdout: libbluetooth3 libfl-dev libfl2 libjson-perl libllvm14 liblzo2-2 2024-09-16T15:40:49.566 INFO:teuthology.orchestra.run.smithi123.stdout: libmbim-glib4 libmbim-proxy libmm-glib0 libndp0 libnm0 libpq5 libqmi-glib5 2024-09-16T15:40:49.566 INFO:teuthology.orchestra.run.smithi123.stdout: libqmi-proxy libssl3 libteamdctl0 libuuid1 modemmanager network-manager-pptp 2024-09-16T15:40:49.566 INFO:teuthology.orchestra.run.smithi123.stdout: postgresql-14 postgresql-client-14 postgresql-client-common ppp pptp-linux 2024-09-16T15:40:49.566 INFO:teuthology.orchestra.run.smithi123.stdout: tcl tcl8.6 usb-modeswitch usb-modeswitch-data wpasupplicant 2024-09-16T15:40:49.567 INFO:teuthology.orchestra.run.smithi123.stdout:Suggested packages: 2024-09-16T15:40:49.567 INFO:teuthology.orchestra.run.smithi123.stdout: bison-doc flex-doc indent-doc libssl-doc avahi-autoipd libteam-utils 2024-09-16T15:40:49.567 INFO:teuthology.orchestra.run.smithi123.stdout: postgresql-doc postgresql-doc-14 tcl-tclreadline comgt wvdial 2024-09-16T15:40:49.567 INFO:teuthology.orchestra.run.smithi123.stdout: util-linux-locales wpagui libengine-pkcs11-openssl 2024-09-16T15:40:49.607 INFO:teuthology.orchestra.run.smithi123.stdout:The following NEW packages will be installed: 2024-09-16T15:40:49.608 INFO:teuthology.orchestra.run.smithi123.stdout: bison dump flex indent libaio-dev libbluetooth3 libelf-dev libfl-dev libfl2 2024-09-16T15:40:49.608 INFO:teuthology.orchestra.run.smithi123.stdout: libjson-perl libllvm14 liblzo2-2 libmbim-glib4 libmbim-proxy libmm-glib0 2024-09-16T15:40:49.608 INFO:teuthology.orchestra.run.smithi123.stdout: libndp0 libnm0 libpq5 libqmi-glib5 libqmi-proxy libssl-dev libteamdctl0 2024-09-16T15:40:49.608 INFO:teuthology.orchestra.run.smithi123.stdout: modemmanager network-manager network-manager-pptp postgresql postgresql-14 2024-09-16T15:40:49.608 INFO:teuthology.orchestra.run.smithi123.stdout: postgresql-client postgresql-client-14 postgresql-client-common 2024-09-16T15:40:49.609 INFO:teuthology.orchestra.run.smithi123.stdout: postgresql-common postgresql-contrib ppp pptp-linux tcl tcl8.6 2024-09-16T15:40:49.609 INFO:teuthology.orchestra.run.smithi123.stdout: usb-modeswitch usb-modeswitch-data wpasupplicant 2024-09-16T15:40:49.609 INFO:teuthology.orchestra.run.smithi123.stdout:The following packages will be upgraded: 2024-09-16T15:40:49.610 INFO:teuthology.orchestra.run.smithi123.stdout: libssl3 libuuid1 util-linux uuid-dev 2024-09-16T15:40:49.680 INFO:teuthology.orchestra.run.smithi123.stdout:4 upgraded, 39 newly installed, 0 to remove and 327 not upgraded. 2024-09-16T15:40:49.681 INFO:teuthology.orchestra.run.smithi123.stdout:Need to get 55.8 MB of archives. 2024-09-16T15:40:49.681 INFO:teuthology.orchestra.run.smithi123.stdout:After this operation, 203 MB of additional disk space will be used. 2024-09-16T15:40:49.681 INFO:teuthology.orchestra.run.smithi123.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 util-linux amd64 2.37.2-4ubuntu3.4 [1,063 kB] 2024-09-16T15:40:49.815 INFO:teuthology.orchestra.run.smithi123.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy/main amd64 flex amd64 2.6.4-8build2 [307 kB] 2024-09-16T15:40:49.823 INFO:teuthology.orchestra.run.smithi123.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libssl3 amd64 3.0.2-0ubuntu1.18 [1,905 kB] 2024-09-16T15:40:49.858 INFO:teuthology.orchestra.run.smithi184.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2024-09-16T15:40:49.861 DEBUG:teuthology.orchestra.run.smithi184:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install bison flex libelf-dev libssl-dev network-manager iproute2 util-linux dump indent libaio-dev libtool-bin uuid-dev xfslibs-dev postgresql postgresql-client postgresql-common postgresql-contrib 2024-09-16T15:40:49.864 INFO:teuthology.orchestra.run.smithi123.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 uuid-dev amd64 2.37.2-4ubuntu3.4 [33.1 kB] 2024-09-16T15:40:49.865 INFO:teuthology.orchestra.run.smithi123.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libuuid1 amd64 2.37.2-4ubuntu3.4 [23.8 kB] 2024-09-16T15:40:49.865 INFO:teuthology.orchestra.run.smithi123.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy/main amd64 bison amd64 2:3.8.2+dfsg-1build1 [748 kB] 2024-09-16T15:40:49.876 INFO:teuthology.orchestra.run.smithi123.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblzo2-2 amd64 2.10-2build3 [53.7 kB] 2024-09-16T15:40:49.877 INFO:teuthology.orchestra.run.smithi123.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy/universe amd64 dump amd64 0.4b47-3 [133 kB] 2024-09-16T15:40:49.879 INFO:teuthology.orchestra.run.smithi123.stdout:Get:9 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 indent amd64 2.2.12-1ubuntu0.22.04.1 [90.8 kB] 2024-09-16T15:40:49.880 INFO:teuthology.orchestra.run.smithi123.stdout:Get:10 http://archive.ubuntu.com/ubuntu jammy/main amd64 libaio-dev amd64 0.3.112-13build1 [21.2 kB] 2024-09-16T15:40:49.880 INFO:teuthology.orchestra.run.smithi123.stdout:Get:11 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libbluetooth3 amd64 5.64-0ubuntu1.3 [86.9 kB] 2024-09-16T15:40:49.882 INFO:teuthology.orchestra.run.smithi123.stdout:Get:12 http://archive.ubuntu.com/ubuntu jammy/main amd64 libelf-dev amd64 0.186-1build1 [64.4 kB] 2024-09-16T15:40:49.883 INFO:teuthology.orchestra.run.smithi123.stdout:Get:13 http://archive.ubuntu.com/ubuntu jammy/main amd64 libfl2 amd64 2.6.4-8build2 [10.7 kB] 2024-09-16T15:40:49.900 INFO:teuthology.orchestra.run.smithi123.stdout:Get:14 http://archive.ubuntu.com/ubuntu jammy/main amd64 libfl-dev amd64 2.6.4-8build2 [6,236 B] 2024-09-16T15:40:49.900 INFO:teuthology.orchestra.run.smithi123.stdout:Get:15 http://archive.ubuntu.com/ubuntu jammy/main amd64 libjson-perl all 4.04000-1 [81.8 kB] 2024-09-16T15:40:49.902 INFO:teuthology.orchestra.run.smithi123.stdout:Get:16 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libllvm14 amd64 1:14.0.0-1ubuntu1.1 [24.0 MB] 2024-09-16T15:40:49.923 INFO:teuthology.orchestra.run.smithi184.stdout:Reading package lists... 2024-09-16T15:40:50.133 INFO:teuthology.orchestra.run.smithi184.stdout:Building dependency tree... 2024-09-16T15:40:50.134 INFO:teuthology.orchestra.run.smithi184.stdout:Reading state information... 2024-09-16T15:40:50.227 INFO:teuthology.orchestra.run.smithi123.stdout:Get:17 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libmbim-glib4 amd64 1.28.0-1~ubuntu20.04.1 [191 kB] 2024-09-16T15:40:50.228 INFO:teuthology.orchestra.run.smithi123.stdout:Get:18 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libmbim-proxy amd64 1.28.0-1~ubuntu20.04.1 [6,130 B] 2024-09-16T15:40:50.229 INFO:teuthology.orchestra.run.smithi123.stdout:Get:19 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libmm-glib0 amd64 1.20.0-1~ubuntu22.04.3 [263 kB] 2024-09-16T15:40:50.231 INFO:teuthology.orchestra.run.smithi123.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libndp0 amd64 1.8-0ubuntu3.1 [12.4 kB] 2024-09-16T15:40:50.232 INFO:teuthology.orchestra.run.smithi123.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libnm0 amd64 1.36.6-0ubuntu2 [456 kB] 2024-09-16T15:40:50.236 INFO:teuthology.orchestra.run.smithi123.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libpq5 amd64 14.13-0ubuntu0.22.04.1 [149 kB] 2024-09-16T15:40:50.237 INFO:teuthology.orchestra.run.smithi123.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libqmi-glib5 amd64 1.32.0-1ubuntu0.22.04.1 [772 kB] 2024-09-16T15:40:50.244 INFO:teuthology.orchestra.run.smithi123.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libqmi-proxy amd64 1.32.0-1ubuntu0.22.04.1 [6,072 B] 2024-09-16T15:40:50.244 INFO:teuthology.orchestra.run.smithi123.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libssl-dev amd64 3.0.2-0ubuntu1.18 [2,375 kB] 2024-09-16T15:40:50.265 INFO:teuthology.orchestra.run.smithi123.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy/main amd64 libteamdctl0 amd64 1.31-1build2 [12.6 kB] 2024-09-16T15:40:50.265 INFO:teuthology.orchestra.run.smithi123.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 modemmanager amd64 1.20.0-1~ubuntu22.04.3 [1,094 kB] 2024-09-16T15:40:50.275 INFO:teuthology.orchestra.run.smithi123.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 network-manager amd64 1.36.6-0ubuntu2 [2,203 kB] 2024-09-16T15:40:50.296 INFO:teuthology.orchestra.run.smithi123.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy/main amd64 ppp amd64 2.4.9-1+1ubuntu3 [363 kB] 2024-09-16T15:40:50.299 INFO:teuthology.orchestra.run.smithi123.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy/main amd64 pptp-linux amd64 1.10.0-1build3 [42.5 kB] 2024-09-16T15:40:50.300 INFO:teuthology.orchestra.run.smithi123.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy/main amd64 network-manager-pptp amd64 1.2.10-1 [30.9 kB] 2024-09-16T15:40:50.300 INFO:teuthology.orchestra.run.smithi123.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy/main amd64 postgresql-client-common all 238 [29.6 kB] 2024-09-16T15:40:50.301 INFO:teuthology.orchestra.run.smithi123.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 postgresql-client-14 amd64 14.13-0ubuntu0.22.04.1 [1,225 kB] 2024-09-16T15:40:50.312 INFO:teuthology.orchestra.run.smithi123.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy/main amd64 postgresql-common all 238 [169 kB] 2024-09-16T15:40:50.314 INFO:teuthology.orchestra.run.smithi123.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 postgresql-14 amd64 14.13-0ubuntu0.22.04.1 [16.2 MB] 2024-09-16T15:40:50.334 INFO:teuthology.orchestra.run.smithi184.stdout:iproute2 is already the newest version (5.15.0-1ubuntu2). 2024-09-16T15:40:50.334 INFO:teuthology.orchestra.run.smithi184.stdout:libtool-bin is already the newest version (2.4.6-15build2). 2024-09-16T15:40:50.334 INFO:teuthology.orchestra.run.smithi184.stdout:xfslibs-dev is already the newest version (5.13.0-1ubuntu2). 2024-09-16T15:40:50.334 INFO:teuthology.orchestra.run.smithi184.stdout:The following packages were automatically installed and are no longer required: 2024-09-16T15:40:50.334 INFO:teuthology.orchestra.run.smithi184.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-09-16T15:40:50.334 INFO:teuthology.orchestra.run.smithi184.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-09-16T15:40:50.334 INFO:teuthology.orchestra.run.smithi184.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-09-16T15:40:50.335 INFO:teuthology.orchestra.run.smithi184.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-09-16T15:40:50.335 INFO:teuthology.orchestra.run.smithi184.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-09-16T15:40:50.335 INFO:teuthology.orchestra.run.smithi184.stdout:Use 'sudo apt autoremove' to remove them. 2024-09-16T15:40:50.336 INFO:teuthology.orchestra.run.smithi184.stdout:The following additional packages will be installed: 2024-09-16T15:40:50.336 INFO:teuthology.orchestra.run.smithi184.stdout: libbluetooth3 libfl-dev libfl2 libjson-perl libllvm14 liblzo2-2 2024-09-16T15:40:50.336 INFO:teuthology.orchestra.run.smithi184.stdout: libmbim-glib4 libmbim-proxy libmm-glib0 libndp0 libnm0 libpq5 libqmi-glib5 2024-09-16T15:40:50.337 INFO:teuthology.orchestra.run.smithi184.stdout: libqmi-proxy libssl3 libteamdctl0 libuuid1 modemmanager network-manager-pptp 2024-09-16T15:40:50.337 INFO:teuthology.orchestra.run.smithi184.stdout: postgresql-14 postgresql-client-14 postgresql-client-common ppp pptp-linux 2024-09-16T15:40:50.337 INFO:teuthology.orchestra.run.smithi184.stdout: tcl tcl8.6 usb-modeswitch usb-modeswitch-data wpasupplicant 2024-09-16T15:40:50.338 INFO:teuthology.orchestra.run.smithi184.stdout:Suggested packages: 2024-09-16T15:40:50.338 INFO:teuthology.orchestra.run.smithi184.stdout: bison-doc flex-doc indent-doc libssl-doc avahi-autoipd libteam-utils 2024-09-16T15:40:50.338 INFO:teuthology.orchestra.run.smithi184.stdout: postgresql-doc postgresql-doc-14 tcl-tclreadline comgt wvdial 2024-09-16T15:40:50.338 INFO:teuthology.orchestra.run.smithi184.stdout: util-linux-locales wpagui libengine-pkcs11-openssl 2024-09-16T15:40:50.380 INFO:teuthology.orchestra.run.smithi184.stdout:The following NEW packages will be installed: 2024-09-16T15:40:50.380 INFO:teuthology.orchestra.run.smithi184.stdout: bison dump flex indent libaio-dev libbluetooth3 libelf-dev libfl-dev libfl2 2024-09-16T15:40:50.380 INFO:teuthology.orchestra.run.smithi184.stdout: libjson-perl libllvm14 liblzo2-2 libmbim-glib4 libmbim-proxy libmm-glib0 2024-09-16T15:40:50.381 INFO:teuthology.orchestra.run.smithi184.stdout: libndp0 libnm0 libpq5 libqmi-glib5 libqmi-proxy libssl-dev libteamdctl0 2024-09-16T15:40:50.381 INFO:teuthology.orchestra.run.smithi184.stdout: modemmanager network-manager network-manager-pptp postgresql postgresql-14 2024-09-16T15:40:50.381 INFO:teuthology.orchestra.run.smithi184.stdout: postgresql-client postgresql-client-14 postgresql-client-common 2024-09-16T15:40:50.381 INFO:teuthology.orchestra.run.smithi184.stdout: postgresql-common postgresql-contrib ppp pptp-linux tcl tcl8.6 2024-09-16T15:40:50.381 INFO:teuthology.orchestra.run.smithi184.stdout: usb-modeswitch usb-modeswitch-data wpasupplicant 2024-09-16T15:40:50.382 INFO:teuthology.orchestra.run.smithi184.stdout:The following packages will be upgraded: 2024-09-16T15:40:50.382 INFO:teuthology.orchestra.run.smithi184.stdout: libssl3 libuuid1 util-linux uuid-dev 2024-09-16T15:40:50.542 INFO:teuthology.orchestra.run.smithi123.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy/main amd64 postgresql all 14+238 [3,288 B] 2024-09-16T15:40:50.542 INFO:teuthology.orchestra.run.smithi123.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy/main amd64 postgresql-client all 14+238 [3,292 B] 2024-09-16T15:40:50.542 INFO:teuthology.orchestra.run.smithi123.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy/main amd64 postgresql-contrib all 14+238 [3,292 B] 2024-09-16T15:40:50.542 INFO:teuthology.orchestra.run.smithi123.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy/main amd64 tcl8.6 amd64 8.6.12+dfsg-1build1 [15.0 kB] 2024-09-16T15:40:50.542 INFO:teuthology.orchestra.run.smithi123.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy/main amd64 tcl amd64 8.6.11+1build2 [4,678 B] 2024-09-16T15:40:50.542 INFO:teuthology.orchestra.run.smithi123.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 wpasupplicant amd64 2:2.10-6ubuntu2.1 [1,482 kB] 2024-09-16T15:40:50.554 INFO:teuthology.orchestra.run.smithi123.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy/main amd64 usb-modeswitch-data all 20191128-4 [33.2 kB] 2024-09-16T15:40:50.554 INFO:teuthology.orchestra.run.smithi123.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy/main amd64 usb-modeswitch amd64 2.6.1-3ubuntu2 [46.0 kB] 2024-09-16T15:40:50.563 INFO:teuthology.orchestra.run.smithi184.stdout:4 upgraded, 39 newly installed, 0 to remove and 327 not upgraded. 2024-09-16T15:40:50.563 INFO:teuthology.orchestra.run.smithi184.stdout:Need to get 55.8 MB of archives. 2024-09-16T15:40:50.563 INFO:teuthology.orchestra.run.smithi184.stdout:After this operation, 203 MB of additional disk space will be used. 2024-09-16T15:40:50.563 INFO:teuthology.orchestra.run.smithi184.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 util-linux amd64 2.37.2-4ubuntu3.4 [1,063 kB] 2024-09-16T15:40:50.967 INFO:teuthology.orchestra.run.smithi123.stdout:Preconfiguring packages ... 2024-09-16T15:40:51.132 INFO:teuthology.orchestra.run.smithi123.stdout:Fetched 55.8 MB in 1s (59.5 MB/s) 2024-09-16T15:40:51.186 INFO:teuthology.orchestra.run.smithi184.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy/main amd64 flex amd64 2.6.4-8build2 [307 kB] 2024-09-16T15:40:51.203 INFO:teuthology.orchestra.run.smithi184.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libssl3 amd64 3.0.2-0ubuntu1.18 [1,905 kB] 2024-09-16T15:40:51.327 INFO:teuthology.orchestra.run.smithi184.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 uuid-dev amd64 2.37.2-4ubuntu3.4 [33.1 kB] 2024-09-16T15:40:51.328 INFO:teuthology.orchestra.run.smithi184.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libuuid1 amd64 2.37.2-4ubuntu3.4 [23.8 kB] 2024-09-16T15:40:51.329 INFO:teuthology.orchestra.run.smithi184.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy/main amd64 bison amd64 2:3.8.2+dfsg-1build1 [748 kB] 2024-09-16T15:40:51.357 INFO:teuthology.orchestra.run.smithi184.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblzo2-2 amd64 2.10-2build3 [53.7 kB] 2024-09-16T15:40:51.359 INFO:teuthology.orchestra.run.smithi184.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy/universe amd64 dump amd64 0.4b47-3 [133 kB] 2024-09-16T15:40:51.362 INFO:teuthology.orchestra.run.smithi184.stdout:Get:9 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 indent amd64 2.2.12-1ubuntu0.22.04.1 [90.8 kB] 2024-09-16T15:40:51.365 INFO:teuthology.orchestra.run.smithi184.stdout:Get:10 http://archive.ubuntu.com/ubuntu jammy/main amd64 libaio-dev amd64 0.3.112-13build1 [21.2 kB] 2024-09-16T15:40:51.365 INFO:teuthology.orchestra.run.smithi184.stdout:Get:11 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libbluetooth3 amd64 5.64-0ubuntu1.3 [86.9 kB] 2024-09-16T15:40:51.368 INFO:teuthology.orchestra.run.smithi184.stdout:Get:12 http://archive.ubuntu.com/ubuntu jammy/main amd64 libelf-dev amd64 0.186-1build1 [64.4 kB] 2024-09-16T15:40:51.430 INFO:teuthology.orchestra.run.smithi184.stdout:Get:13 http://archive.ubuntu.com/ubuntu jammy/main amd64 libfl2 amd64 2.6.4-8build2 [10.7 kB] 2024-09-16T15:40:51.431 INFO:teuthology.orchestra.run.smithi184.stdout:Get:14 http://archive.ubuntu.com/ubuntu jammy/main amd64 libfl-dev amd64 2.6.4-8build2 [6,236 B] 2024-09-16T15:40:51.431 INFO:teuthology.orchestra.run.smithi184.stdout:Get:15 http://archive.ubuntu.com/ubuntu jammy/main amd64 libjson-perl all 4.04000-1 [81.8 kB] 2024-09-16T15:40:51.487 INFO:teuthology.orchestra.run.smithi123.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 176352 files and directories currently installed.) 2024-09-16T15:40:51.492 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../util-linux_2.37.2-4ubuntu3.4_amd64.deb ... 2024-09-16T15:40:51.512 INFO:teuthology.orchestra.run.smithi184.stdout:Get:16 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libllvm14 amd64 1:14.0.0-1ubuntu1.1 [24.0 MB] 2024-09-16T15:40:51.637 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking util-linux (2.37.2-4ubuntu3.4) over (2.37.2-4ubuntu3) ... 2024-09-16T15:40:52.266 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up util-linux (2.37.2-4ubuntu3.4) ... 2024-09-16T15:40:52.288 INFO:teuthology.orchestra.run.smithi184.stdout:Get:17 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libmbim-glib4 amd64 1.28.0-1~ubuntu20.04.1 [191 kB] 2024-09-16T15:40:52.292 INFO:teuthology.orchestra.run.smithi184.stdout:Get:18 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libmbim-proxy amd64 1.28.0-1~ubuntu20.04.1 [6,130 B] 2024-09-16T15:40:52.293 INFO:teuthology.orchestra.run.smithi184.stdout:Get:19 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libmm-glib0 amd64 1.20.0-1~ubuntu22.04.3 [263 kB] 2024-09-16T15:40:52.299 INFO:teuthology.orchestra.run.smithi184.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libndp0 amd64 1.8-0ubuntu3.1 [12.4 kB] 2024-09-16T15:40:52.300 INFO:teuthology.orchestra.run.smithi184.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libnm0 amd64 1.36.6-0ubuntu2 [456 kB] 2024-09-16T15:40:52.310 INFO:teuthology.orchestra.run.smithi184.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libpq5 amd64 14.13-0ubuntu0.22.04.1 [149 kB] 2024-09-16T15:40:52.314 INFO:teuthology.orchestra.run.smithi184.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libqmi-glib5 amd64 1.32.0-1ubuntu0.22.04.1 [772 kB] 2024-09-16T15:40:52.331 INFO:teuthology.orchestra.run.smithi184.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libqmi-proxy amd64 1.32.0-1ubuntu0.22.04.1 [6,072 B] 2024-09-16T15:40:52.331 INFO:teuthology.orchestra.run.smithi184.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libssl-dev amd64 3.0.2-0ubuntu1.18 [2,375 kB] 2024-09-16T15:40:52.407 INFO:teuthology.orchestra.run.smithi184.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy/main amd64 libteamdctl0 amd64 1.31-1build2 [12.6 kB] 2024-09-16T15:40:52.417 INFO:teuthology.orchestra.run.smithi184.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 modemmanager amd64 1.20.0-1~ubuntu22.04.3 [1,094 kB] 2024-09-16T15:40:52.438 INFO:teuthology.orchestra.run.smithi184.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 network-manager amd64 1.36.6-0ubuntu2 [2,203 kB] 2024-09-16T15:40:52.508 INFO:teuthology.orchestra.run.smithi184.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy/main amd64 ppp amd64 2.4.9-1+1ubuntu3 [363 kB] 2024-09-16T15:40:52.515 INFO:teuthology.orchestra.run.smithi184.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy/main amd64 pptp-linux amd64 1.10.0-1build3 [42.5 kB] 2024-09-16T15:40:52.516 INFO:teuthology.orchestra.run.smithi184.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy/main amd64 network-manager-pptp amd64 1.2.10-1 [30.9 kB] 2024-09-16T15:40:52.517 INFO:teuthology.orchestra.run.smithi184.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy/main amd64 postgresql-client-common all 238 [29.6 kB] 2024-09-16T15:40:52.518 INFO:teuthology.orchestra.run.smithi184.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 postgresql-client-14 amd64 14.13-0ubuntu0.22.04.1 [1,225 kB] 2024-09-16T15:40:52.548 INFO:teuthology.orchestra.run.smithi184.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy/main amd64 postgresql-common all 238 [169 kB] 2024-09-16T15:40:52.551 INFO:teuthology.orchestra.run.smithi184.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 postgresql-14 amd64 14.13-0ubuntu0.22.04.1 [16.2 MB] 2024-09-16T15:40:53.115 INFO:teuthology.orchestra.run.smithi184.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy/main amd64 postgresql all 14+238 [3,288 B] 2024-09-16T15:40:53.115 INFO:teuthology.orchestra.run.smithi184.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy/main amd64 postgresql-client all 14+238 [3,292 B] 2024-09-16T15:40:53.115 INFO:teuthology.orchestra.run.smithi184.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy/main amd64 postgresql-contrib all 14+238 [3,292 B] 2024-09-16T15:40:53.116 INFO:teuthology.orchestra.run.smithi184.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy/main amd64 tcl8.6 amd64 8.6.12+dfsg-1build1 [15.0 kB] 2024-09-16T15:40:53.116 INFO:teuthology.orchestra.run.smithi184.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy/main amd64 tcl amd64 8.6.11+1build2 [4,678 B] 2024-09-16T15:40:53.116 INFO:teuthology.orchestra.run.smithi184.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 wpasupplicant amd64 2:2.10-6ubuntu2.1 [1,482 kB] 2024-09-16T15:40:53.145 INFO:teuthology.orchestra.run.smithi184.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy/main amd64 usb-modeswitch-data all 20191128-4 [33.2 kB] 2024-09-16T15:40:53.146 INFO:teuthology.orchestra.run.smithi184.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy/main amd64 usb-modeswitch amd64 2.6.1-3ubuntu2 [46.0 kB] 2024-09-16T15:40:53.194 INFO:teuthology.orchestra.run.smithi123.stdout:fstrim.service is a disabled or a static unit not running, not starting it. 2024-09-16T15:40:53.475 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package flex. 2024-09-16T15:40:53.561 INFO:teuthology.orchestra.run.smithi184.stdout:Preconfiguring packages ... 2024-09-16T15:40:53.579 INFO:teuthology.orchestra.run.smithi123.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 176352 files and directories currently installed.) 2024-09-16T15:40:53.585 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../flex_2.6.4-8build2_amd64.deb ... 2024-09-16T15:40:53.639 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking flex (2.6.4-8build2) ... 2024-09-16T15:40:53.722 INFO:teuthology.orchestra.run.smithi184.stdout:Fetched 55.8 MB in 3s (20.2 MB/s) 2024-09-16T15:40:54.036 INFO:teuthology.orchestra.run.smithi184.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 176352 files and directories currently installed.) 2024-09-16T15:40:54.042 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../util-linux_2.37.2-4ubuntu3.4_amd64.deb ... 2024-09-16T15:40:54.173 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../libssl3_3.0.2-0ubuntu1.18_amd64.deb ... 2024-09-16T15:40:54.193 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking util-linux (2.37.2-4ubuntu3.4) over (2.37.2-4ubuntu3) ... 2024-09-16T15:40:54.353 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking libssl3:amd64 (3.0.2-0ubuntu1.18) over (3.0.2-0ubuntu1.7) ... 2024-09-16T15:40:54.773 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up libssl3:amd64 (3.0.2-0ubuntu1.18) ... 2024-09-16T15:40:54.807 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up util-linux (2.37.2-4ubuntu3.4) ... 2024-09-16T15:40:55.344 INFO:teuthology.orchestra.run.smithi123.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 176425 files and directories currently installed.) 2024-09-16T15:40:55.350 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../uuid-dev_2.37.2-4ubuntu3.4_amd64.deb ... 2024-09-16T15:40:55.511 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking uuid-dev:amd64 (2.37.2-4ubuntu3.4) over (2.37.2-4ubuntu3) ... 2024-09-16T15:40:55.725 INFO:teuthology.orchestra.run.smithi184.stdout:fstrim.service is a disabled or a static unit not running, not starting it. 2024-09-16T15:40:55.996 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../libuuid1_2.37.2-4ubuntu3.4_amd64.deb ... 2024-09-16T15:40:56.026 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package flex. 2024-09-16T15:40:56.124 INFO:teuthology.orchestra.run.smithi184.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 176352 files and directories currently installed.) 2024-09-16T15:40:56.130 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../flex_2.6.4-8build2_amd64.deb ... 2024-09-16T15:40:56.183 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking flex (2.6.4-8build2) ... 2024-09-16T15:40:56.251 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking libuuid1:amd64 (2.37.2-4ubuntu3.4) over (2.37.2-4ubuntu3) ... 2024-09-16T15:40:56.628 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up libuuid1:amd64 (2.37.2-4ubuntu3.4) ... 2024-09-16T15:40:56.868 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../libssl3_3.0.2-0ubuntu1.18_amd64.deb ... 2024-09-16T15:40:56.994 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package bison. 2024-09-16T15:40:57.041 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking libssl3:amd64 (3.0.2-0ubuntu1.18) over (3.0.2-0ubuntu1.7) ... 2024-09-16T15:40:57.098 INFO:teuthology.orchestra.run.smithi123.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 176425 files and directories currently installed.) 2024-09-16T15:40:57.104 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../00-bison_2%3a3.8.2+dfsg-1build1_amd64.deb ... 2024-09-16T15:40:57.146 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking bison (2:3.8.2+dfsg-1build1) ... 2024-09-16T15:40:57.478 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up libssl3:amd64 (3.0.2-0ubuntu1.18) ... 2024-09-16T15:40:57.531 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package liblzo2-2:amd64. 2024-09-16T15:40:57.558 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../01-liblzo2-2_2.10-2build3_amd64.deb ... 2024-09-16T15:40:57.588 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking liblzo2-2:amd64 (2.10-2build3) ... 2024-09-16T15:40:57.916 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package dump. 2024-09-16T15:40:57.944 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../02-dump_0.4b47-3_amd64.deb ... 2024-09-16T15:40:57.991 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking dump (0.4b47-3) ... 2024-09-16T15:40:58.034 INFO:teuthology.orchestra.run.smithi184.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 176425 files and directories currently installed.) 2024-09-16T15:40:58.039 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../uuid-dev_2.37.2-4ubuntu3.4_amd64.deb ... 2024-09-16T15:40:58.166 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking uuid-dev:amd64 (2.37.2-4ubuntu3.4) over (2.37.2-4ubuntu3) ... 2024-09-16T15:40:58.286 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package indent. 2024-09-16T15:40:58.302 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../03-indent_2.2.12-1ubuntu0.22.04.1_amd64.deb ... 2024-09-16T15:40:58.344 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking indent (2.2.12-1ubuntu0.22.04.1) ... 2024-09-16T15:40:58.589 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package libaio-dev:amd64. 2024-09-16T15:40:58.592 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../libuuid1_2.37.2-4ubuntu3.4_amd64.deb ... 2024-09-16T15:40:58.605 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../04-libaio-dev_0.3.112-13build1_amd64.deb ... 2024-09-16T15:40:58.647 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking libaio-dev:amd64 (0.3.112-13build1) ... 2024-09-16T15:40:58.705 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking libuuid1:amd64 (2.37.2-4ubuntu3.4) over (2.37.2-4ubuntu3) ... 2024-09-16T15:40:58.993 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package libbluetooth3:amd64. 2024-09-16T15:40:59.021 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../05-libbluetooth3_5.64-0ubuntu1.3_amd64.deb ... 2024-09-16T15:40:59.042 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up libuuid1:amd64 (2.37.2-4ubuntu3.4) ... 2024-09-16T15:40:59.059 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking libbluetooth3:amd64 (5.64-0ubuntu1.3) ... 2024-09-16T15:40:59.320 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package libelf-dev:amd64. 2024-09-16T15:40:59.336 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../06-libelf-dev_0.186-1build1_amd64.deb ... 2024-09-16T15:40:59.355 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package bison. 2024-09-16T15:40:59.378 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking libelf-dev:amd64 (0.186-1build1) ... 2024-09-16T15:40:59.448 INFO:teuthology.orchestra.run.smithi184.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 176425 files and directories currently installed.) 2024-09-16T15:40:59.454 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../00-bison_2%3a3.8.2+dfsg-1build1_amd64.deb ... 2024-09-16T15:40:59.508 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking bison (2:3.8.2+dfsg-1build1) ... 2024-09-16T15:40:59.682 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package libfl2:amd64. 2024-09-16T15:40:59.698 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../07-libfl2_2.6.4-8build2_amd64.deb ... 2024-09-16T15:40:59.740 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking libfl2:amd64 (2.6.4-8build2) ... 2024-09-16T15:40:59.943 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package liblzo2-2:amd64. 2024-09-16T15:40:59.968 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package libfl-dev:amd64. 2024-09-16T15:40:59.971 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../01-liblzo2-2_2.10-2build3_amd64.deb ... 2024-09-16T15:40:59.984 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../08-libfl-dev_2.6.4-8build2_amd64.deb ... 2024-09-16T15:41:00.009 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking liblzo2-2:amd64 (2.10-2build3) ... 2024-09-16T15:41:00.026 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking libfl-dev:amd64 (2.6.4-8build2) ... 2024-09-16T15:41:00.296 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package dump. 2024-09-16T15:41:00.312 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package libjson-perl. 2024-09-16T15:41:00.312 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../02-dump_0.4b47-3_amd64.deb ... 2024-09-16T15:41:00.340 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../09-libjson-perl_4.04000-1_all.deb ... 2024-09-16T15:41:00.354 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking dump (0.4b47-3) ... 2024-09-16T15:41:00.387 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking libjson-perl (4.04000-1) ... 2024-09-16T15:41:00.650 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package indent. 2024-09-16T15:41:00.678 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../03-indent_2.2.12-1ubuntu0.22.04.1_amd64.deb ... 2024-09-16T15:41:00.716 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking indent (2.2.12-1ubuntu0.22.04.1) ... 2024-09-16T15:41:00.765 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package libllvm14:amd64. 2024-09-16T15:41:00.793 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../10-libllvm14_1%3a14.0.0-1ubuntu1.1_amd64.deb ... 2024-09-16T15:41:00.840 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking libllvm14:amd64 (1:14.0.0-1ubuntu1.1) ... 2024-09-16T15:41:00.986 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package libaio-dev:amd64. 2024-09-16T15:41:01.002 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../04-libaio-dev_0.3.112-13build1_amd64.deb ... 2024-09-16T15:41:01.044 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking libaio-dev:amd64 (0.3.112-13build1) ... 2024-09-16T15:41:01.340 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package libbluetooth3:amd64. 2024-09-16T15:41:01.368 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../05-libbluetooth3_5.64-0ubuntu1.3_amd64.deb ... 2024-09-16T15:41:01.415 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking libbluetooth3:amd64 (5.64-0ubuntu1.3) ... 2024-09-16T15:41:01.685 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package libelf-dev:amd64. 2024-09-16T15:41:01.701 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../06-libelf-dev_0.186-1build1_amd64.deb ... 2024-09-16T15:41:01.743 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking libelf-dev:amd64 (0.186-1build1) ... 2024-09-16T15:41:02.080 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package libfl2:amd64. 2024-09-16T15:41:02.108 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../07-libfl2_2.6.4-8build2_amd64.deb ... 2024-09-16T15:41:02.146 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking libfl2:amd64 (2.6.4-8build2) ... 2024-09-16T15:41:02.366 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package libfl-dev:amd64. 2024-09-16T15:41:02.382 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../08-libfl-dev_2.6.4-8build2_amd64.deb ... 2024-09-16T15:41:02.424 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking libfl-dev:amd64 (2.6.4-8build2) ... 2024-09-16T15:41:02.543 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package libmbim-glib4:amd64. 2024-09-16T15:41:02.572 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../11-libmbim-glib4_1.28.0-1~ubuntu20.04.1_amd64.deb ... 2024-09-16T15:41:02.626 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking libmbim-glib4:amd64 (1.28.0-1~ubuntu20.04.1) ... 2024-09-16T15:41:02.677 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package libjson-perl. 2024-09-16T15:41:02.693 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../09-libjson-perl_4.04000-1_all.deb ... 2024-09-16T15:41:02.735 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking libjson-perl (4.04000-1) ... 2024-09-16T15:41:02.904 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package libmbim-proxy. 2024-09-16T15:41:02.922 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../12-libmbim-proxy_1.28.0-1~ubuntu20.04.1_amd64.deb ... 2024-09-16T15:41:02.962 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking libmbim-proxy (1.28.0-1~ubuntu20.04.1) ... 2024-09-16T15:41:03.055 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package libllvm14:amd64. 2024-09-16T15:41:03.083 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../10-libllvm14_1%3a14.0.0-1ubuntu1.1_amd64.deb ... 2024-09-16T15:41:03.121 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking libllvm14:amd64 (1:14.0.0-1ubuntu1.1) ... 2024-09-16T15:41:03.298 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package libmm-glib0:amd64. 2024-09-16T15:41:03.327 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../13-libmm-glib0_1.20.0-1~ubuntu22.04.3_amd64.deb ... 2024-09-16T15:41:03.365 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking libmm-glib0:amd64 (1.20.0-1~ubuntu22.04.3) ... 2024-09-16T15:41:03.702 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package libndp0:amd64. 2024-09-16T15:41:03.730 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../14-libndp0_1.8-0ubuntu3.1_amd64.deb ... 2024-09-16T15:41:03.768 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking libndp0:amd64 (1.8-0ubuntu3.1) ... 2024-09-16T15:41:04.104 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package libnm0:amd64. 2024-09-16T15:41:04.133 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../15-libnm0_1.36.6-0ubuntu2_amd64.deb ... 2024-09-16T15:41:04.170 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking libnm0:amd64 (1.36.6-0ubuntu2) ... 2024-09-16T15:41:04.524 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package libpq5:amd64. 2024-09-16T15:41:04.552 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../16-libpq5_14.13-0ubuntu0.22.04.1_amd64.deb ... 2024-09-16T15:41:04.590 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking libpq5:amd64 (14.13-0ubuntu0.22.04.1) ... 2024-09-16T15:41:04.766 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package libmbim-glib4:amd64. 2024-09-16T15:41:04.794 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../11-libmbim-glib4_1.28.0-1~ubuntu20.04.1_amd64.deb ... 2024-09-16T15:41:04.824 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking libmbim-glib4:amd64 (1.28.0-1~ubuntu20.04.1) ... 2024-09-16T15:41:04.926 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package libqmi-glib5:amd64. 2024-09-16T15:41:04.954 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../17-libqmi-glib5_1.32.0-1ubuntu0.22.04.1_amd64.deb ... 2024-09-16T15:41:04.993 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking libqmi-glib5:amd64 (1.32.0-1ubuntu0.22.04.1) ... 2024-09-16T15:41:05.086 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package libmbim-proxy. 2024-09-16T15:41:05.114 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../12-libmbim-proxy_1.28.0-1~ubuntu20.04.1_amd64.deb ... 2024-09-16T15:41:05.152 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking libmbim-proxy (1.28.0-1~ubuntu20.04.1) ... 2024-09-16T15:41:05.304 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package libqmi-proxy. 2024-09-16T15:41:05.321 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../18-libqmi-proxy_1.32.0-1ubuntu0.22.04.1_amd64.deb ... 2024-09-16T15:41:05.362 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking libqmi-proxy (1.32.0-1ubuntu0.22.04.1) ... 2024-09-16T15:41:05.480 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package libmm-glib0:amd64. 2024-09-16T15:41:05.508 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../13-libmm-glib0_1.20.0-1~ubuntu22.04.3_amd64.deb ... 2024-09-16T15:41:05.546 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking libmm-glib0:amd64 (1.20.0-1~ubuntu22.04.3) ... 2024-09-16T15:41:05.598 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package libssl-dev:amd64. 2024-09-16T15:41:05.626 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../19-libssl-dev_3.0.2-0ubuntu1.18_amd64.deb ... 2024-09-16T15:41:05.664 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking libssl-dev:amd64 (3.0.2-0ubuntu1.18) ... 2024-09-16T15:41:05.875 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package libndp0:amd64. 2024-09-16T15:41:05.904 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../14-libndp0_1.8-0ubuntu3.1_amd64.deb ... 2024-09-16T15:41:05.933 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking libndp0:amd64 (1.8-0ubuntu3.1) ... 2024-09-16T15:41:06.202 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package libteamdctl0:amd64. 2024-09-16T15:41:06.230 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../20-libteamdctl0_1.31-1build2_amd64.deb ... 2024-09-16T15:41:06.253 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package libnm0:amd64. 2024-09-16T15:41:06.268 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking libteamdctl0:amd64 (1.31-1build2) ... 2024-09-16T15:41:06.281 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../15-libnm0_1.36.6-0ubuntu2_amd64.deb ... 2024-09-16T15:41:06.319 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking libnm0:amd64 (1.36.6-0ubuntu2) ... 2024-09-16T15:41:06.605 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package modemmanager. 2024-09-16T15:41:06.633 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../21-modemmanager_1.20.0-1~ubuntu22.04.3_amd64.deb ... 2024-09-16T15:41:06.671 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking modemmanager (1.20.0-1~ubuntu22.04.3) ... 2024-09-16T15:41:06.691 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package libpq5:amd64. 2024-09-16T15:41:06.720 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../16-libpq5_14.13-0ubuntu0.22.04.1_amd64.deb ... 2024-09-16T15:41:06.774 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking libpq5:amd64 (14.13-0ubuntu0.22.04.1) ... 2024-09-16T15:41:07.094 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package libqmi-glib5:amd64. 2024-09-16T15:41:07.123 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../17-libqmi-glib5_1.32.0-1ubuntu0.22.04.1_amd64.deb ... 2024-09-16T15:41:07.160 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking libqmi-glib5:amd64 (1.32.0-1ubuntu0.22.04.1) ... 2024-09-16T15:41:07.238 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package network-manager. 2024-09-16T15:41:07.266 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../22-network-manager_1.36.6-0ubuntu2_amd64.deb ... 2024-09-16T15:41:07.313 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking network-manager (1.36.6-0ubuntu2) ... 2024-09-16T15:41:07.463 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package libqmi-proxy. 2024-09-16T15:41:07.480 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../18-libqmi-proxy_1.32.0-1ubuntu0.22.04.1_amd64.deb ... 2024-09-16T15:41:07.521 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking libqmi-proxy (1.32.0-1ubuntu0.22.04.1) ... 2024-09-16T15:41:07.766 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package libssl-dev:amd64. 2024-09-16T15:41:07.783 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../19-libssl-dev_3.0.2-0ubuntu1.18_amd64.deb ... 2024-09-16T15:41:07.824 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking libssl-dev:amd64 (3.0.2-0ubuntu1.18) ... 2024-09-16T15:41:07.970 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package ppp. 2024-09-16T15:41:07.999 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../23-ppp_2.4.9-1+1ubuntu3_amd64.deb ... 2024-09-16T15:41:08.270 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking ppp (2.4.9-1+1ubuntu3) ... 2024-09-16T15:41:08.336 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package libteamdctl0:amd64. 2024-09-16T15:41:08.365 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../20-libteamdctl0_1.31-1build2_amd64.deb ... 2024-09-16T15:41:08.403 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking libteamdctl0:amd64 (1.31-1build2) ... 2024-09-16T15:41:08.533 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package pptp-linux. 2024-09-16T15:41:08.549 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../24-pptp-linux_1.10.0-1build3_amd64.deb ... 2024-09-16T15:41:08.583 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking pptp-linux (1.10.0-1build3) ... 2024-09-16T15:41:08.739 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package modemmanager. 2024-09-16T15:41:08.768 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../21-modemmanager_1.20.0-1~ubuntu22.04.3_amd64.deb ... 2024-09-16T15:41:08.806 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking modemmanager (1.20.0-1~ubuntu22.04.3) ... 2024-09-16T15:41:08.844 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package network-manager-pptp. 2024-09-16T15:41:08.872 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../25-network-manager-pptp_1.2.10-1_amd64.deb ... 2024-09-16T15:41:08.911 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking network-manager-pptp (1.2.10-1) ... 2024-09-16T15:41:09.214 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package postgresql-client-common. 2024-09-16T15:41:09.242 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../26-postgresql-client-common_238_all.deb ... 2024-09-16T15:41:09.280 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking postgresql-client-common (238) ... 2024-09-16T15:41:09.471 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package network-manager. 2024-09-16T15:41:09.499 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../22-network-manager_1.36.6-0ubuntu2_amd64.deb ... 2024-09-16T15:41:09.542 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package postgresql-client-14. 2024-09-16T15:41:09.546 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking network-manager (1.36.6-0ubuntu2) ... 2024-09-16T15:41:09.558 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../27-postgresql-client-14_14.13-0ubuntu0.22.04.1_amd64.deb ... 2024-09-16T15:41:09.600 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking postgresql-client-14 (14.13-0ubuntu0.22.04.1) ... 2024-09-16T15:41:10.129 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package ppp. 2024-09-16T15:41:10.147 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package postgresql-common. 2024-09-16T15:41:10.158 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../23-ppp_2.4.9-1+1ubuntu3_amd64.deb ... 2024-09-16T15:41:10.176 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../28-postgresql-common_238_all.deb ... 2024-09-16T15:41:10.232 INFO:teuthology.orchestra.run.smithi123.stdout:Adding 'diversion of /usr/bin/pg_config to /usr/bin/pg_config.libpq-dev by postgresql-common' 2024-09-16T15:41:10.261 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking postgresql-common (238) ... 2024-09-16T15:41:10.421 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking ppp (2.4.9-1+1ubuntu3) ... 2024-09-16T15:41:10.729 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package postgresql-14. 2024-09-16T15:41:10.734 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package pptp-linux. 2024-09-16T15:41:10.757 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../29-postgresql-14_14.13-0ubuntu0.22.04.1_amd64.deb ... 2024-09-16T15:41:10.762 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../24-pptp-linux_1.10.0-1build3_amd64.deb ... 2024-09-16T15:41:10.800 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking pptp-linux (1.10.0-1build3) ... 2024-09-16T15:41:10.815 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking postgresql-14 (14.13-0ubuntu0.22.04.1) ... 2024-09-16T15:41:11.053 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package network-manager-pptp. 2024-09-16T15:41:11.070 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../25-network-manager-pptp_1.2.10-1_amd64.deb ... 2024-09-16T15:41:11.111 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking network-manager-pptp (1.2.10-1) ... 2024-09-16T15:41:11.473 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package postgresql-client-common. 2024-09-16T15:41:11.501 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../26-postgresql-client-common_238_all.deb ... 2024-09-16T15:41:11.548 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking postgresql-client-common (238) ... 2024-09-16T15:41:12.045 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package postgresql-client-14. 2024-09-16T15:41:12.073 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../27-postgresql-client-14_14.13-0ubuntu0.22.04.1_amd64.deb ... 2024-09-16T15:41:12.111 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking postgresql-client-14 (14.13-0ubuntu0.22.04.1) ... 2024-09-16T15:41:12.248 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package postgresql. 2024-09-16T15:41:12.278 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../30-postgresql_14+238_all.deb ... 2024-09-16T15:41:12.315 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking postgresql (14+238) ... 2024-09-16T15:41:12.558 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package postgresql-client. 2024-09-16T15:41:12.575 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../31-postgresql-client_14+238_all.deb ... 2024-09-16T15:41:12.617 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking postgresql-client (14+238) ... 2024-09-16T15:41:12.624 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package postgresql-common. 2024-09-16T15:41:12.653 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../28-postgresql-common_238_all.deb ... 2024-09-16T15:41:12.718 INFO:teuthology.orchestra.run.smithi184.stdout:Adding 'diversion of /usr/bin/pg_config to /usr/bin/pg_config.libpq-dev by postgresql-common' 2024-09-16T15:41:12.748 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking postgresql-common (238) ... 2024-09-16T15:41:12.836 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package postgresql-contrib. 2024-09-16T15:41:12.852 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../32-postgresql-contrib_14+238_all.deb ... 2024-09-16T15:41:12.894 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking postgresql-contrib (14+238) ... 2024-09-16T15:41:13.157 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package postgresql-14. 2024-09-16T15:41:13.185 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../29-postgresql-14_14.13-0ubuntu0.22.04.1_amd64.deb ... 2024-09-16T15:41:13.196 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package tcl8.6. 2024-09-16T15:41:13.218 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking postgresql-14 (14.13-0ubuntu0.22.04.1) ... 2024-09-16T15:41:13.225 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../33-tcl8.6_8.6.12+dfsg-1build1_amd64.deb ... 2024-09-16T15:41:13.263 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking tcl8.6 (8.6.12+dfsg-1build1) ... 2024-09-16T15:41:13.549 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package tcl. 2024-09-16T15:41:13.578 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../34-tcl_8.6.11+1build2_amd64.deb ... 2024-09-16T15:41:13.620 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking tcl (8.6.11+1build2) ... 2024-09-16T15:41:13.968 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package wpasupplicant. 2024-09-16T15:41:13.997 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../35-wpasupplicant_2%3a2.10-6ubuntu2.1_amd64.deb ... 2024-09-16T15:41:14.035 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking wpasupplicant (2:2.10-6ubuntu2.1) ... 2024-09-16T15:41:14.423 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package usb-modeswitch-data. 2024-09-16T15:41:14.439 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../36-usb-modeswitch-data_20191128-4_all.deb ... 2024-09-16T15:41:14.481 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking usb-modeswitch-data (20191128-4) ... 2024-09-16T15:41:14.485 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package postgresql. 2024-09-16T15:41:14.502 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../30-postgresql_14+238_all.deb ... 2024-09-16T15:41:14.535 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking postgresql (14+238) ... 2024-09-16T15:41:14.746 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package postgresql-client. 2024-09-16T15:41:14.762 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../31-postgresql-client_14+238_all.deb ... 2024-09-16T15:41:14.800 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package usb-modeswitch. 2024-09-16T15:41:14.804 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking postgresql-client (14+238) ... 2024-09-16T15:41:14.829 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../37-usb-modeswitch_2.6.1-3ubuntu2_amd64.deb ... 2024-09-16T15:41:14.875 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking usb-modeswitch (2.6.1-3ubuntu2) ... 2024-09-16T15:41:15.048 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package postgresql-contrib. 2024-09-16T15:41:15.064 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../32-postgresql-contrib_14+238_all.deb ... 2024-09-16T15:41:15.106 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking postgresql-contrib (14+238) ... 2024-09-16T15:41:15.163 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up postgresql-client-common (238) ... 2024-09-16T15:41:15.373 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up flex (2.6.4-8build2) ... 2024-09-16T15:41:15.442 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package tcl8.6. 2024-09-16T15:41:15.471 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../33-tcl8.6_8.6.12+dfsg-1build1_amd64.deb ... 2024-09-16T15:41:15.499 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up tcl8.6 (8.6.12+dfsg-1build1) ... 2024-09-16T15:41:15.509 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking tcl8.6 (8.6.12+dfsg-1build1) ... 2024-09-16T15:41:15.625 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up libpq5:amd64 (14.13-0ubuntu0.22.04.1) ... 2024-09-16T15:41:15.742 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up liblzo2-2:amd64 (2.10-2build3) ... 2024-09-16T15:41:15.803 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package tcl. 2024-09-16T15:41:15.832 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../34-tcl_8.6.11+1build2_amd64.deb ... 2024-09-16T15:41:15.860 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up libaio-dev:amd64 (0.3.112-13build1) ... 2024-09-16T15:41:15.875 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking tcl (8.6.11+1build2) ... 2024-09-16T15:41:15.978 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up postgresql-client-14 (14.13-0ubuntu0.22.04.1) ... 2024-09-16T15:41:16.206 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package wpasupplicant. 2024-09-16T15:41:16.235 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../35-wpasupplicant_2%3a2.10-6ubuntu2.1_amd64.deb ... 2024-09-16T15:41:16.272 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking wpasupplicant (2:2.10-6ubuntu2.1) ... 2024-09-16T15:41:16.511 INFO:teuthology.orchestra.run.smithi123.stdout:update-alternatives: using /usr/share/postgresql/14/man/man1/psql.1.gz to provide /usr/share/man/man1/psql.1.gz (psql.1.gz) in auto mode 2024-09-16T15:41:16.600 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up wpasupplicant (2:2.10-6ubuntu2.1) ... 2024-09-16T15:41:16.677 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package usb-modeswitch-data. 2024-09-16T15:41:16.694 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../36-usb-modeswitch-data_20191128-4_all.deb ... 2024-09-16T15:41:16.835 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking usb-modeswitch-data (20191128-4) ... 2024-09-16T15:41:16.963 INFO:teuthology.orchestra.run.smithi123.stdout:Created symlink /etc/systemd/system/dbus-fi.w1.wpa_supplicant1.service -> /lib/systemd/system/wpa_supplicant.service. 2024-09-16T15:41:16.963 INFO:teuthology.orchestra.run.smithi123.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/wpa_supplicant.service -> /lib/systemd/system/wpa_supplicant.service. 2024-09-16T15:41:17.478 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up uuid-dev:amd64 (2.37.2-4ubuntu3.4) ... 2024-09-16T15:41:17.604 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up libteamdctl0:amd64 (1.31-1build2) ... 2024-09-16T15:41:17.672 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package usb-modeswitch. 2024-09-16T15:41:17.700 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../37-usb-modeswitch_2.6.1-3ubuntu2_amd64.deb ... 2024-09-16T15:41:17.730 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up libssl-dev:amd64 (3.0.2-0ubuntu1.18) ... 2024-09-16T15:41:17.747 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking usb-modeswitch (2.6.1-3ubuntu2) ... 2024-09-16T15:41:18.044 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up postgresql-client-common (238) ... 2024-09-16T15:41:18.147 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up usb-modeswitch-data (20191128-4) ... 2024-09-16T15:41:18.253 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up flex (2.6.4-8build2) ... 2024-09-16T15:41:18.379 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up tcl8.6 (8.6.12+dfsg-1build1) ... 2024-09-16T15:41:18.465 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up libelf-dev:amd64 (0.186-1build1) ... 2024-09-16T15:41:18.505 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up libpq5:amd64 (14.13-0ubuntu0.22.04.1) ... 2024-09-16T15:41:18.591 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up libfl2:amd64 (2.6.4-8build2) ... 2024-09-16T15:41:18.631 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up liblzo2-2:amd64 (2.10-2build3) ... 2024-09-16T15:41:18.717 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up libnm0:amd64 (1.36.6-0ubuntu2) ... 2024-09-16T15:41:18.741 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up libaio-dev:amd64 (0.3.112-13build1) ... 2024-09-16T15:41:18.826 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up ppp (2.4.9-1+1ubuntu3) ... 2024-09-16T15:41:18.850 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up postgresql-client-14 (14.13-0ubuntu0.22.04.1) ... 2024-09-16T15:41:19.343 INFO:teuthology.orchestra.run.smithi184.stdout:update-alternatives: using /usr/share/postgresql/14/man/man1/psql.1.gz to provide /usr/share/man/man1/psql.1.gz (psql.1.gz) in auto mode 2024-09-16T15:41:19.423 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up wpasupplicant (2:2.10-6ubuntu2.1) ... 2024-09-16T15:41:19.573 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up libllvm14:amd64 (1:14.0.0-1ubuntu1.1) ... 2024-09-16T15:41:19.699 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up dump (0.4b47-3) ... 2024-09-16T15:41:19.783 INFO:teuthology.orchestra.run.smithi184.stdout:Created symlink /etc/systemd/system/dbus-fi.w1.wpa_supplicant1.service -> /lib/systemd/system/wpa_supplicant.service. 2024-09-16T15:41:19.784 INFO:teuthology.orchestra.run.smithi184.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/wpa_supplicant.service -> /lib/systemd/system/wpa_supplicant.service. 2024-09-16T15:41:19.793 INFO:teuthology.orchestra.run.smithi123.stdout:update-alternatives: using /usr/sbin/rmt-dump to provide /usr/sbin/rmt (rmt) in auto mode 2024-09-16T15:41:19.859 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up libmm-glib0:amd64 (1.20.0-1~ubuntu22.04.3) ... 2024-09-16T15:41:19.976 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up libbluetooth3:amd64 (5.64-0ubuntu1.3) ... 2024-09-16T15:41:20.094 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up tcl (8.6.11+1build2) ... 2024-09-16T15:41:20.228 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up libjson-perl (4.04000-1) ... 2024-09-16T15:41:20.259 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up uuid-dev:amd64 (2.37.2-4ubuntu3.4) ... 2024-09-16T15:41:20.354 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up bison (2:3.8.2+dfsg-1build1) ... 2024-09-16T15:41:20.360 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up libteamdctl0:amd64 (1.31-1build2) ... 2024-09-16T15:41:20.444 INFO:teuthology.orchestra.run.smithi123.stdout:update-alternatives: using /usr/bin/bison.yacc to provide /usr/bin/yacc (yacc) in auto mode 2024-09-16T15:41:20.486 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up libssl-dev:amd64 (3.0.2-0ubuntu1.18) ... 2024-09-16T15:41:20.514 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up indent (2.2.12-1ubuntu0.22.04.1) ... 2024-09-16T15:41:20.612 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up usb-modeswitch-data (20191128-4) ... 2024-09-16T15:41:20.640 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up usb-modeswitch (2.6.1-3ubuntu2) ... 2024-09-16T15:41:20.755 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up libelf-dev:amd64 (0.186-1build1) ... 2024-09-16T15:41:20.816 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up libndp0:amd64 (1.8-0ubuntu3.1) ... 2024-09-16T15:41:20.880 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up libfl2:amd64 (2.6.4-8build2) ... 2024-09-16T15:41:20.942 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up libmbim-glib4:amd64 (1.28.0-1~ubuntu20.04.1) ... 2024-09-16T15:41:21.006 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up libnm0:amd64 (1.36.6-0ubuntu2) ... 2024-09-16T15:41:21.060 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up pptp-linux (1.10.0-1build3) ... 2024-09-16T15:41:21.149 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up ppp (2.4.9-1+1ubuntu3) ... 2024-09-16T15:41:21.220 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up postgresql-client (14+238) ... 2024-09-16T15:41:21.337 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up network-manager (1.36.6-0ubuntu2) ... 2024-09-16T15:41:21.738 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up libllvm14:amd64 (1:14.0.0-1ubuntu1.1) ... 2024-09-16T15:41:21.803 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-09-16T15:41:21.803 INFO:teuthology.orchestra.run.smithi123.stdout:The following network interfaces were found in /etc/network/interfaces 2024-09-16T15:41:21.803 INFO:teuthology.orchestra.run.smithi123.stdout:which means they are currently configured by ifupdown: 2024-09-16T15:41:21.804 INFO:teuthology.orchestra.run.smithi123.stdout:- enp3s0f1 2024-09-16T15:41:21.804 INFO:teuthology.orchestra.run.smithi123.stdout:If you want to manage those interfaces with NetworkManager instead 2024-09-16T15:41:21.804 INFO:teuthology.orchestra.run.smithi123.stdout:remove their configuration from /etc/network/interfaces. 2024-09-16T15:41:21.804 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-09-16T15:41:21.864 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up dump (0.4b47-3) ... 2024-09-16T15:41:21.920 INFO:teuthology.orchestra.run.smithi123.stdout:Created symlink /etc/systemd/system/dbus-org.freedesktop.nm-dispatcher.service -> /lib/systemd/system/NetworkManager-dispatcher.service. 2024-09-16T15:41:21.958 INFO:teuthology.orchestra.run.smithi184.stdout:update-alternatives: using /usr/sbin/rmt-dump to provide /usr/sbin/rmt (rmt) in auto mode 2024-09-16T15:41:22.015 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up libmm-glib0:amd64 (1.20.0-1~ubuntu22.04.3) ... 2024-09-16T15:41:22.264 INFO:teuthology.orchestra.run.smithi123.stdout:Created symlink /etc/systemd/system/network-online.target.wants/NetworkManager-wait-online.service -> /lib/systemd/system/NetworkManager-wait-online.service. 2024-09-16T15:41:22.484 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up libbluetooth3:amd64 (5.64-0ubuntu1.3) ... 2024-09-16T15:41:22.587 INFO:teuthology.orchestra.run.smithi123.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/NetworkManager.service -> /lib/systemd/system/NetworkManager.service. 2024-09-16T15:41:22.660 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up tcl (8.6.11+1build2) ... 2024-09-16T15:41:22.769 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up libjson-perl (4.04000-1) ... 2024-09-16T15:41:22.895 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up bison (2:3.8.2+dfsg-1build1) ... 2024-09-16T15:41:22.985 INFO:teuthology.orchestra.run.smithi184.stdout:update-alternatives: using /usr/bin/bison.yacc to provide /usr/bin/yacc (yacc) in auto mode 2024-09-16T15:41:23.055 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up indent (2.2.12-1ubuntu0.22.04.1) ... 2024-09-16T15:41:23.073 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up libmbim-proxy (1.28.0-1~ubuntu20.04.1) ... 2024-09-16T15:41:23.181 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up usb-modeswitch (2.6.1-3ubuntu2) ... 2024-09-16T15:41:23.224 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up libfl-dev:amd64 (2.6.4-8build2) ... 2024-09-16T15:41:23.350 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up network-manager-pptp (1.2.10-1) ... 2024-09-16T15:41:23.357 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up libndp0:amd64 (1.8-0ubuntu3.1) ... 2024-09-16T15:41:23.484 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up libmbim-glib4:amd64 (1.28.0-1~ubuntu20.04.1) ... 2024-09-16T15:41:23.585 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up pptp-linux (1.10.0-1build3) ... 2024-09-16T15:41:23.878 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up postgresql-client (14+238) ... 2024-09-16T15:41:23.893 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up postgresql-common (238) ... 2024-09-16T15:41:24.004 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up network-manager (1.36.6-0ubuntu2) ... 2024-09-16T15:41:24.468 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-09-16T15:41:24.468 INFO:teuthology.orchestra.run.smithi184.stdout:The following network interfaces were found in /etc/network/interfaces 2024-09-16T15:41:24.468 INFO:teuthology.orchestra.run.smithi184.stdout:which means they are currently configured by ifupdown: 2024-09-16T15:41:24.468 INFO:teuthology.orchestra.run.smithi184.stdout:- enp3s0f1 2024-09-16T15:41:24.469 INFO:teuthology.orchestra.run.smithi184.stdout:If you want to manage those interfaces with NetworkManager instead 2024-09-16T15:41:24.469 INFO:teuthology.orchestra.run.smithi184.stdout:remove their configuration from /etc/network/interfaces. 2024-09-16T15:41:24.469 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-09-16T15:41:24.586 INFO:teuthology.orchestra.run.smithi184.stdout:Created symlink /etc/systemd/system/dbus-org.freedesktop.nm-dispatcher.service -> /lib/systemd/system/NetworkManager-dispatcher.service. 2024-09-16T15:41:24.755 INFO:teuthology.orchestra.run.smithi123.stdout:Adding user postgres to group ssl-cert 2024-09-16T15:41:24.897 INFO:teuthology.orchestra.run.smithi184.stdout:Created symlink /etc/systemd/system/network-online.target.wants/NetworkManager-wait-online.service -> /lib/systemd/system/NetworkManager-wait-online.service. 2024-09-16T15:41:24.954 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-09-16T15:41:24.954 INFO:teuthology.orchestra.run.smithi123.stdout:Creating config file /etc/postgresql-common/createcluster.conf with new version 2024-09-16T15:41:25.143 INFO:teuthology.orchestra.run.smithi123.stdout:Building PostgreSQL dictionaries from installed myspell/hunspell packages... 2024-09-16T15:41:25.143 INFO:teuthology.orchestra.run.smithi123.stdout:Removing obsolete dictionary files: 2024-09-16T15:41:25.230 INFO:teuthology.orchestra.run.smithi184.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/NetworkManager.service -> /lib/systemd/system/NetworkManager.service. 2024-09-16T15:41:25.681 INFO:teuthology.orchestra.run.smithi123.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/postgresql.service -> /lib/systemd/system/postgresql.service. 2024-09-16T15:41:25.723 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up libmbim-proxy (1.28.0-1~ubuntu20.04.1) ... 2024-09-16T15:41:25.858 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up libfl-dev:amd64 (2.6.4-8build2) ... 2024-09-16T15:41:25.992 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up network-manager-pptp (1.2.10-1) ... 2024-09-16T15:41:26.095 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up postgresql-14 (14.13-0ubuntu0.22.04.1) ... 2024-09-16T15:41:26.110 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up postgresql-common (238) ... 2024-09-16T15:41:26.424 INFO:teuthology.orchestra.run.smithi123.stdout:Creating new PostgreSQL cluster 14/main ... 2024-09-16T15:41:26.442 INFO:teuthology.orchestra.run.smithi123.stdout:/usr/lib/postgresql/14/bin/initdb -D /var/lib/postgresql/14/main --auth-local peer --auth-host scram-sha-256 --no-instructions 2024-09-16T15:41:26.489 INFO:teuthology.orchestra.run.smithi123.stdout:The files belonging to this database system will be owned by user "postgres". 2024-09-16T15:41:26.489 INFO:teuthology.orchestra.run.smithi123.stdout:This user must also own the server process. 2024-09-16T15:41:26.489 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-09-16T15:41:26.489 INFO:teuthology.orchestra.run.smithi123.stdout:The database cluster will be initialized with locale "en_US". 2024-09-16T15:41:26.489 INFO:teuthology.orchestra.run.smithi123.stdout:The default database encoding has accordingly been set to "LATIN1". 2024-09-16T15:41:26.489 INFO:teuthology.orchestra.run.smithi123.stdout:The default text search configuration will be set to "english". 2024-09-16T15:41:26.489 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-09-16T15:41:26.489 INFO:teuthology.orchestra.run.smithi123.stdout:Data page checksums are disabled. 2024-09-16T15:41:26.490 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-09-16T15:41:26.490 INFO:teuthology.orchestra.run.smithi123.stdout:fixing permissions on existing directory /var/lib/postgresql/14/main ... ok 2024-09-16T15:41:26.490 INFO:teuthology.orchestra.run.smithi123.stdout:creating subdirectories ... ok 2024-09-16T15:41:26.490 INFO:teuthology.orchestra.run.smithi123.stdout:selecting dynamic shared memory implementation ... posix 2024-09-16T15:41:26.505 INFO:teuthology.orchestra.run.smithi123.stdout:selecting default max_connections ... 100 2024-09-16T15:41:26.524 INFO:teuthology.orchestra.run.smithi123.stdout:selecting default shared_buffers ... 128MB 2024-09-16T15:41:26.534 INFO:teuthology.orchestra.run.smithi123.stdout:selecting default time zone ... Etc/UTC 2024-09-16T15:41:26.535 INFO:teuthology.orchestra.run.smithi123.stdout:creating configuration files ... ok 2024-09-16T15:41:26.649 INFO:teuthology.orchestra.run.smithi123.stdout:running bootstrap script ... ok 2024-09-16T15:41:26.964 INFO:teuthology.orchestra.run.smithi184.stdout:Adding user postgres to group ssl-cert 2024-09-16T15:41:27.193 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-09-16T15:41:27.193 INFO:teuthology.orchestra.run.smithi184.stdout:Creating config file /etc/postgresql-common/createcluster.conf with new version 2024-09-16T15:41:27.292 INFO:teuthology.orchestra.run.smithi123.stdout:performing post-bootstrap initialization ... ok 2024-09-16T15:41:27.376 INFO:teuthology.orchestra.run.smithi184.stdout:Building PostgreSQL dictionaries from installed myspell/hunspell packages... 2024-09-16T15:41:27.376 INFO:teuthology.orchestra.run.smithi184.stdout:Removing obsolete dictionary files: 2024-09-16T15:41:27.906 INFO:teuthology.orchestra.run.smithi123.stdout:syncing data to disk ... ok 2024-09-16T15:41:27.917 INFO:teuthology.orchestra.run.smithi184.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/postgresql.service -> /lib/systemd/system/postgresql.service. 2024-09-16T15:41:28.312 INFO:teuthology.orchestra.run.smithi123.stdout:update-alternatives: using /usr/share/postgresql/14/man/man1/postmaster.1.gz to provide /usr/share/man/man1/postmaster.1.gz (postmaster.1.gz) in auto mode 2024-09-16T15:41:28.337 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up postgresql-14 (14.13-0ubuntu0.22.04.1) ... 2024-09-16T15:41:28.669 INFO:teuthology.orchestra.run.smithi184.stdout:Creating new PostgreSQL cluster 14/main ... 2024-09-16T15:41:28.689 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/lib/postgresql/14/bin/initdb -D /var/lib/postgresql/14/main --auth-local peer --auth-host scram-sha-256 --no-instructions 2024-09-16T15:41:28.734 INFO:teuthology.orchestra.run.smithi184.stdout:The files belonging to this database system will be owned by user "postgres". 2024-09-16T15:41:28.734 INFO:teuthology.orchestra.run.smithi184.stdout:This user must also own the server process. 2024-09-16T15:41:28.734 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-09-16T15:41:28.734 INFO:teuthology.orchestra.run.smithi184.stdout:The database cluster will be initialized with locale "en_US". 2024-09-16T15:41:28.734 INFO:teuthology.orchestra.run.smithi184.stdout:The default database encoding has accordingly been set to "LATIN1". 2024-09-16T15:41:28.734 INFO:teuthology.orchestra.run.smithi184.stdout:The default text search configuration will be set to "english". 2024-09-16T15:41:28.734 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-09-16T15:41:28.734 INFO:teuthology.orchestra.run.smithi184.stdout:Data page checksums are disabled. 2024-09-16T15:41:28.734 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-09-16T15:41:28.735 INFO:teuthology.orchestra.run.smithi184.stdout:fixing permissions on existing directory /var/lib/postgresql/14/main ... ok 2024-09-16T15:41:28.735 INFO:teuthology.orchestra.run.smithi184.stdout:creating subdirectories ... ok 2024-09-16T15:41:28.735 INFO:teuthology.orchestra.run.smithi184.stdout:selecting dynamic shared memory implementation ... posix 2024-09-16T15:41:28.750 INFO:teuthology.orchestra.run.smithi184.stdout:selecting default max_connections ... 100 2024-09-16T15:41:28.771 INFO:teuthology.orchestra.run.smithi184.stdout:selecting default shared_buffers ... 128MB 2024-09-16T15:41:28.778 INFO:teuthology.orchestra.run.smithi184.stdout:selecting default time zone ... Etc/UTC 2024-09-16T15:41:28.779 INFO:teuthology.orchestra.run.smithi184.stdout:creating configuration files ... ok 2024-09-16T15:41:28.891 INFO:teuthology.orchestra.run.smithi184.stdout:running bootstrap script ... ok 2024-09-16T15:41:29.553 INFO:teuthology.orchestra.run.smithi184.stdout:performing post-bootstrap initialization ... ok 2024-09-16T15:41:30.207 INFO:teuthology.orchestra.run.smithi184.stdout:syncing data to disk ... ok 2024-09-16T15:41:30.583 INFO:teuthology.orchestra.run.smithi184.stdout:update-alternatives: using /usr/share/postgresql/14/man/man1/postmaster.1.gz to provide /usr/share/man/man1/postmaster.1.gz (postmaster.1.gz) in auto mode 2024-09-16T15:41:31.092 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up libqmi-glib5:amd64 (1.32.0-1ubuntu0.22.04.1) ... 2024-09-16T15:41:31.210 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up postgresql-contrib (14+238) ... 2024-09-16T15:41:31.327 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up postgresql (14+238) ... 2024-09-16T15:41:31.453 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up libqmi-proxy (1.32.0-1ubuntu0.22.04.1) ... 2024-09-16T15:41:31.579 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up modemmanager (1.20.0-1~ubuntu22.04.3) ... 2024-09-16T15:41:31.834 INFO:teuthology.orchestra.run.smithi123.stdout:Created symlink /etc/systemd/system/dbus-org.freedesktop.ModemManager1.service -> /lib/systemd/system/ModemManager.service. 2024-09-16T15:41:31.834 INFO:teuthology.orchestra.run.smithi123.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ModemManager.service -> /lib/systemd/system/ModemManager.service. 2024-09-16T15:41:32.398 INFO:teuthology.orchestra.run.smithi123.stdout:Processing triggers for mailcap (3.70+nmu1ubuntu1) ... 2024-09-16T15:41:32.491 INFO:teuthology.orchestra.run.smithi123.stdout:Processing triggers for hicolor-icon-theme (0.17-2) ... 2024-09-16T15:41:32.793 INFO:teuthology.orchestra.run.smithi123.stdout:Processing triggers for libc-bin (2.35-0ubuntu3.1) ... 2024-09-16T15:41:32.945 INFO:teuthology.orchestra.run.smithi123.stdout:Processing triggers for man-db (2.10.2-1) ... 2024-09-16T15:41:33.336 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up libqmi-glib5:amd64 (1.32.0-1ubuntu0.22.04.1) ... 2024-09-16T15:41:33.462 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up postgresql-contrib (14+238) ... 2024-09-16T15:41:33.588 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up postgresql (14+238) ... 2024-09-16T15:41:33.697 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up libqmi-proxy (1.32.0-1ubuntu0.22.04.1) ... 2024-09-16T15:41:33.798 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up modemmanager (1.20.0-1~ubuntu22.04.3) ... 2024-09-16T15:41:34.036 INFO:teuthology.orchestra.run.smithi184.stdout:Created symlink /etc/systemd/system/dbus-org.freedesktop.ModemManager1.service -> /lib/systemd/system/ModemManager.service. 2024-09-16T15:41:34.036 INFO:teuthology.orchestra.run.smithi184.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ModemManager.service -> /lib/systemd/system/ModemManager.service. 2024-09-16T15:41:34.256 INFO:teuthology.orchestra.run.smithi123.stdout:Processing triggers for dbus (1.12.20-2ubuntu4.1) ... 2024-09-16T15:41:34.356 INFO:teuthology.orchestra.run.smithi123.stdout:Processing triggers for install-info (6.8-4build1) ... 2024-09-16T15:41:34.584 INFO:teuthology.orchestra.run.smithi184.stdout:Processing triggers for mailcap (3.70+nmu1ubuntu1) ... 2024-09-16T15:41:34.685 INFO:teuthology.orchestra.run.smithi184.stdout:Processing triggers for hicolor-icon-theme (0.17-2) ... 2024-09-16T15:41:34.864 INFO:teuthology.orchestra.run.smithi123.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2024-09-16T15:41:34.870 DEBUG:teuthology.parallel:result is None 2024-09-16T15:41:34.987 INFO:teuthology.orchestra.run.smithi184.stdout:Processing triggers for libc-bin (2.35-0ubuntu3.1) ... 2024-09-16T15:41:35.139 INFO:teuthology.orchestra.run.smithi184.stdout:Processing triggers for man-db (2.10.2-1) ... 2024-09-16T15:41:36.492 INFO:teuthology.orchestra.run.smithi184.stdout:Processing triggers for dbus (1.12.20-2ubuntu4.1) ... 2024-09-16T15:41:36.601 INFO:teuthology.orchestra.run.smithi184.stdout:Processing triggers for install-info (6.8-4build1) ... 2024-09-16T15:41:37.129 INFO:teuthology.orchestra.run.smithi184.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2024-09-16T15:41:37.134 DEBUG:teuthology.parallel:result is None 2024-09-16T15:41:37.135 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=33f95d2ae1bcbe7eb04404303f179eeeaa6982df 2024-09-16T15:41:37.284 DEBUG:teuthology.orchestra.run.smithi123:> dpkg-query -W -f '${Version}' ceph 2024-09-16T15:41:37.303 INFO:teuthology.orchestra.run.smithi123.stdout:19.3.0-4938-g33f95d2a-1jammy 2024-09-16T15:41:37.304 INFO:teuthology.packaging:The installed version of ceph is 19.3.0-4938-g33f95d2a-1jammy 2024-09-16T15:41:37.304 INFO:teuthology.task.install:The correct ceph version 19.3.0-4938-g33f95d2a-1jammy is installed. 2024-09-16T15:41:37.306 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=33f95d2ae1bcbe7eb04404303f179eeeaa6982df 2024-09-16T15:41:37.440 DEBUG:teuthology.orchestra.run.smithi184:> dpkg-query -W -f '${Version}' ceph 2024-09-16T15:41:37.459 INFO:teuthology.orchestra.run.smithi184.stdout:19.3.0-4938-g33f95d2a-1jammy 2024-09-16T15:41:37.460 INFO:teuthology.packaging:The installed version of ceph is 19.3.0-4938-g33f95d2a-1jammy 2024-09-16T15:41:37.460 INFO:teuthology.task.install:The correct ceph version 19.3.0-4938-g33f95d2a-1jammy is installed. 2024-09-16T15:41:37.461 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-09-16T15:41:37.462 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2024-09-16T15:41:37.462 DEBUG:teuthology.orchestra.run.smithi123:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-09-16T15:41:37.477 DEBUG:teuthology.orchestra.run.smithi184:> set -ex 2024-09-16T15:41:37.477 DEBUG:teuthology.orchestra.run.smithi184:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-09-16T15:41:37.513 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-09-16T15:41:37.514 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2024-09-16T15:41:37.514 DEBUG:teuthology.orchestra.run.smithi123:> sudo dd of=/usr/bin/daemon-helper 2024-09-16T15:41:37.529 DEBUG:teuthology.orchestra.run.smithi123:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-09-16T15:41:37.585 DEBUG:teuthology.orchestra.run.smithi184:> set -ex 2024-09-16T15:41:37.585 DEBUG:teuthology.orchestra.run.smithi184:> sudo dd of=/usr/bin/daemon-helper 2024-09-16T15:41:37.599 DEBUG:teuthology.orchestra.run.smithi184:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-09-16T15:41:37.653 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-09-16T15:41:37.653 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2024-09-16T15:41:37.653 DEBUG:teuthology.orchestra.run.smithi123:> sudo dd of=/usr/bin/adjust-ulimits 2024-09-16T15:41:37.667 DEBUG:teuthology.orchestra.run.smithi123:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-09-16T15:41:37.721 DEBUG:teuthology.orchestra.run.smithi184:> set -ex 2024-09-16T15:41:37.721 DEBUG:teuthology.orchestra.run.smithi184:> sudo dd of=/usr/bin/adjust-ulimits 2024-09-16T15:41:37.733 DEBUG:teuthology.orchestra.run.smithi184:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-09-16T15:41:37.789 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-09-16T15:41:37.789 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2024-09-16T15:41:37.789 DEBUG:teuthology.orchestra.run.smithi123:> sudo dd of=/usr/bin/stdin-killer 2024-09-16T15:41:37.802 DEBUG:teuthology.orchestra.run.smithi123:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-09-16T15:41:37.857 DEBUG:teuthology.orchestra.run.smithi184:> set -ex 2024-09-16T15:41:37.857 DEBUG:teuthology.orchestra.run.smithi184:> sudo dd of=/usr/bin/stdin-killer 2024-09-16T15:41:37.870 DEBUG:teuthology.orchestra.run.smithi184:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-09-16T15:41:37.925 INFO:teuthology.run_tasks:Running task ceph... 2024-09-16T15:41:38.001 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-09-16T15:41:38.002 DEBUG:teuthology.orchestra.run.smithi123:> sudo chmod 777 /var/log/ceph 2024-09-16T15:41:38.004 DEBUG:teuthology.orchestra.run.smithi184:> sudo chmod 777 /var/log/ceph 2024-09-16T15:41:38.016 INFO:tasks.ceph:Disabling ceph logrotate... 2024-09-16T15:41:38.017 DEBUG:teuthology.orchestra.run.smithi123:> sudo rm -f -- /etc/logrotate.d/ceph 2024-09-16T15:41:38.059 DEBUG:teuthology.orchestra.run.smithi184:> sudo rm -f -- /etc/logrotate.d/ceph 2024-09-16T15:41:38.071 INFO:tasks.ceph:Creating extra log directories... 2024-09-16T15:41:38.071 DEBUG:teuthology.orchestra.run.smithi123:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-09-16T15:41:38.115 DEBUG:teuthology.orchestra.run.smithi184:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-09-16T15:41:38.133 INFO:tasks.ceph:Setting up log rotation with {'ceph-mds': '10G', 'ceph-osd': '10G'} 2024-09-16T15:41:38.133 INFO:tasks.ceph:writing logrotate stanza for ceph-mds 2024-09-16T15:41:38.133 INFO:tasks.ceph:writing logrotate stanza for ceph-osd 2024-09-16T15:41:38.133 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2024-09-16T15:41:38.133 DEBUG:teuthology.orchestra.run.smithi123:> dd of=/home/ubuntu/cephtest/logrotate.ceph-test.conf 2024-09-16T15:41:38.174 DEBUG:teuthology.orchestra.run.smithi123:> sudo mv /home/ubuntu/cephtest/logrotate.ceph-test.conf /etc/logrotate.d/ceph-test.conf && sudo chmod 0644 /etc/logrotate.d/ceph-test.conf && sudo chown root.root /etc/logrotate.d/ceph-test.conf 2024-09-16T15:41:38.248 DEBUG:teuthology.orchestra.run.smithi184:> set -ex 2024-09-16T15:41:38.248 DEBUG:teuthology.orchestra.run.smithi184:> dd of=/home/ubuntu/cephtest/logrotate.ceph-test.conf 2024-09-16T15:41:38.254 DEBUG:teuthology.orchestra.run.smithi184:> sudo mv /home/ubuntu/cephtest/logrotate.ceph-test.conf /etc/logrotate.d/ceph-test.conf && sudo chmod 0644 /etc/logrotate.d/ceph-test.conf && sudo chown root.root /etc/logrotate.d/ceph-test.conf 2024-09-16T15:41:38.335 INFO:tasks.ceph:Creating ceph cluster ceph... 2024-09-16T15:41:38.335 INFO:tasks.ceph:config {'conf': {'client': {'pid file': ''}, 'mgr': {'client mount timeout': 30, 'debug client': 20, 'debug finisher': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon down mkfs grace': 300}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'fs.*is degraded', 'filesystem is degraded', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'filesystem is offline', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'insufficient standby MDS daemons available', 'MDS_UP_LESS_THAN_MAX', 'online, but wants', 'filesystem is online with fewer MDS than max_mds', 'POOL_APP_NOT_ENABLED', 'do not have an application enabled', 'overall HEALTH_', 'Replacing daemon', 'deprecated feature inline_data', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'PG_AVAILABILITY', 'PG_DEGRADED', 'Reduced data availability', 'Degraded data redundancy', 'OSD full dropping all updates', 'OSD near full', 'pausewr flag', 'failsafe engaged, dropping updates', 'failsafe disengaged, no longer dropping', 'is full \\(reached quota', 'POOL_FULL', 'POOL_BACKFILLFULL'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2024-09-16T15:41:38.335 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/avan-2024-09-16_15:21:40-fs:volumes-wip-athakkar-testing-2024-09-16-1454-distro-default-smithi/7907351', 'branch': 'wip-athakkar-testing-2024-09-16-1454', 'description': 'fs:volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mgr mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} tasks/volumes/{overrides test/misc}}', 'email': 'athakkar@redhat.com', 'first_in_suite': False, 'job_id': '7907351', 'kernel': {'kdb': 1, 'sha1': 'distro'}, 'ktype': 'distro', 'last_in_suite': False, 'log-rotate': {'ceph-mds': '10G', 'ceph-osd': '10G'}, 'machine_type': 'smithi', 'name': 'avan-2024-09-16_15:21:40-fs:volumes-wip-athakkar-testing-2024-09-16-1454-distro-default-smithi', 'no_nested_subset': False, 'openstack': [{'volumes': {'count': 4, 'size': 20}}, {'machine': {'disk': 200}}], 'os_type': 'ubuntu', 'os_version': '22.04', 'overrides': {'admin_socket': {'branch': 'wip-athakkar-testing-2024-09-16-1454'}, 'ceph': {'cluster-conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 2, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mon': {'mon op complaint time': 120, 'mon warn on pool no app': False}, 'osd': {'osd op complaint time': 180}}, 'conf': {'client': {'pid file': ''}, 'mgr': {'client mount timeout': 30, 'debug client': 20, 'debug finisher': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon down mkfs grace': 300}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'fs.*is degraded', 'filesystem is degraded', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'filesystem is offline', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'insufficient standby MDS daemons available', 'MDS_UP_LESS_THAN_MAX', 'online, but wants', 'filesystem is online with fewer MDS than max_mds', 'POOL_APP_NOT_ENABLED', 'do not have an application enabled', 'overall HEALTH_', 'Replacing daemon', 'deprecated feature inline_data', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'PG_AVAILABILITY', 'PG_DEGRADED', 'Reduced data availability', 'Degraded data redundancy', 'OSD full dropping all updates', 'OSD near full', 'pausewr flag', 'failsafe engaged, dropping updates', 'failsafe disengaged, no longer dropping', 'is full \\(reached quota', 'POOL_FULL', 'POOL_BACKFILLFULL'], 'sha1': '33f95d2ae1bcbe7eb04404303f179eeeaa6982df'}, 'ceph-deploy': {'bluestore': True, 'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs'}, 'install': {'ceph': {'flavor': 'default', 'sha1': '33f95d2ae1bcbe7eb04404303f179eeeaa6982df'}}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'wip-athakkar-testing-2024-09-16-1454', 'sha1': '33f95d2ae1bcbe7eb04404303f179eeeaa6982df'}}, 'owner': 'scheduled_avan@teuthology', 'priority': 80, 'repo': 'https://git.ceph.com/ceph-ci.git', 'roles': [['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'client.2'], ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'client.3']], 'seed': 3077, 'sha1': '33f95d2ae1bcbe7eb04404303f179eeeaa6982df', 'sleep_before_teardown': 0, 'subset': '1/20', 'suite': 'fs:volumes', 'suite_branch': 'wip-athakkar-testing-2024-09-16-1454', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph-c_33f95d2ae1bcbe7eb04404303f179eeeaa6982df/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://git.ceph.com/ceph-ci.git', 'suite_sha1': '33f95d2ae1bcbe7eb04404303f179eeeaa6982df', 'syslog': {'ignorelist': ['WARNING*.*check_session_state', 'WARNING*.*__ceph_remove_cap']}, 'targets': {'smithi123.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJs6B1iWsHgQDJFra9/dCfSvhqd/H6C6e3mV32azcWpu9X8Fd4JgFXbmLpEoVvAqxax/eLGBI7hekTSjm3QvhPE=', 'smithi184.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBA6D966A33EyqjY+W49aOBG+J6GnrD854kU+AJY//NJ+Yh7P7n9i7rnt27Yda6isEa0RXPpJGM+5xYH3FJ1NbfE='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': 1, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': '33f95d2ae1bcbe7eb04404303f179eeeaa6982df'}}, {'ceph': {'create_rbd_pool': False, 'cluster-conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 2, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mon': {'mon op complaint time': 120, 'mon warn on pool no app': False}, 'osd': {'osd op complaint time': 180}}, 'conf': {'client': {'pid file': ''}, 'mgr': {'client mount timeout': 30, 'debug client': 20, 'debug finisher': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon down mkfs grace': 300}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'fs.*is degraded', 'filesystem is degraded', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'filesystem is offline', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'insufficient standby MDS daemons available', 'MDS_UP_LESS_THAN_MAX', 'online, but wants', 'filesystem is online with fewer MDS than max_mds', 'POOL_APP_NOT_ENABLED', 'do not have an application enabled', 'overall HEALTH_', 'Replacing daemon', 'deprecated feature inline_data', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'PG_AVAILABILITY', 'PG_DEGRADED', 'Reduced data availability', 'Degraded data redundancy', 'OSD full dropping all updates', 'OSD near full', 'pausewr flag', 'failsafe engaged, dropping updates', 'failsafe disengaged, no longer dropping', 'is full \\(reached quota', 'POOL_FULL', 'POOL_BACKFILLFULL'], 'sha1': '33f95d2ae1bcbe7eb04404303f179eeeaa6982df', 'cluster': 'ceph'}}, {'ceph-fuse': None}, {'cephfs_test_runner': {'fail_on_skip': False, 'modules': ['tasks.cephfs.test_volumes.TestMisc']}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': ['local function is_kupstream() return false end', 'local function is_kdistro() return false end', 'local function is_fuse() return true end', "local function syntax_version() return '' end"]}, 'teuthology_branch': 'main', 'teuthology_sha1': '3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a', 'timestamp': '2024-09-16_15:21:40', 'tube': 'smithi', 'user': 'avan', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.3401856'} 2024-09-16T15:41:38.336 DEBUG:teuthology.orchestra.run.smithi123:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-09-16T15:41:38.338 DEBUG:teuthology.orchestra.run.smithi184:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-09-16T15:41:38.378 DEBUG:teuthology.orchestra.run.smithi123:> sudo install -d -m0777 -- /var/run/ceph 2024-09-16T15:41:38.387 DEBUG:teuthology.orchestra.run.smithi184:> sudo install -d -m0777 -- /var/run/ceph 2024-09-16T15:41:38.434 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2024-09-16T15:41:38.434 DEBUG:teuthology.orchestra.run.smithi123:> dd if=/scratch_devs of=/dev/stdout 2024-09-16T15:41:38.446 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-09-16T15:41:38.446 DEBUG:teuthology.orchestra.run.smithi123:> stat /dev/vg_nvme/lv_1 2024-09-16T15:41:38.494 INFO:teuthology.orchestra.run.smithi123.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-09-16T15:41:38.494 INFO:teuthology.orchestra.run.smithi123.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-16T15:41:38.494 INFO:teuthology.orchestra.run.smithi123.stdout:Device: 5h/5d Inode: 871 Links: 1 2024-09-16T15:41:38.494 INFO:teuthology.orchestra.run.smithi123.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-16T15:41:38.494 INFO:teuthology.orchestra.run.smithi123.stdout:Access: 2024-09-16 15:35:12.673327170 +0000 2024-09-16T15:41:38.494 INFO:teuthology.orchestra.run.smithi123.stdout:Modify: 2024-09-16 15:35:12.389329702 +0000 2024-09-16T15:41:38.494 INFO:teuthology.orchestra.run.smithi123.stdout:Change: 2024-09-16 15:35:12.389329702 +0000 2024-09-16T15:41:38.494 INFO:teuthology.orchestra.run.smithi123.stdout: Birth: - 2024-09-16T15:41:38.495 DEBUG:teuthology.orchestra.run.smithi123:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-09-16T15:41:38.548 INFO:teuthology.orchestra.run.smithi123.stderr:1+0 records in 2024-09-16T15:41:38.548 INFO:teuthology.orchestra.run.smithi123.stderr:1+0 records out 2024-09-16T15:41:38.548 INFO:teuthology.orchestra.run.smithi123.stderr:512 bytes copied, 0.000330228 s, 1.6 MB/s 2024-09-16T15:41:38.549 DEBUG:teuthology.orchestra.run.smithi123:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-09-16T15:41:38.599 DEBUG:teuthology.orchestra.run.smithi123:> stat /dev/vg_nvme/lv_2 2024-09-16T15:41:38.649 INFO:teuthology.orchestra.run.smithi123.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-09-16T15:41:38.649 INFO:teuthology.orchestra.run.smithi123.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-16T15:41:38.650 INFO:teuthology.orchestra.run.smithi123.stdout:Device: 5h/5d Inode: 891 Links: 1 2024-09-16T15:41:38.650 INFO:teuthology.orchestra.run.smithi123.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-16T15:41:38.650 INFO:teuthology.orchestra.run.smithi123.stdout:Access: 2024-09-16 15:35:13.153322888 +0000 2024-09-16T15:41:38.650 INFO:teuthology.orchestra.run.smithi123.stdout:Modify: 2024-09-16 15:35:12.913325028 +0000 2024-09-16T15:41:38.650 INFO:teuthology.orchestra.run.smithi123.stdout:Change: 2024-09-16 15:35:12.913325028 +0000 2024-09-16T15:41:38.650 INFO:teuthology.orchestra.run.smithi123.stdout: Birth: - 2024-09-16T15:41:38.650 DEBUG:teuthology.orchestra.run.smithi123:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-09-16T15:41:38.703 INFO:teuthology.orchestra.run.smithi123.stderr:1+0 records in 2024-09-16T15:41:38.703 INFO:teuthology.orchestra.run.smithi123.stderr:1+0 records out 2024-09-16T15:41:38.703 INFO:teuthology.orchestra.run.smithi123.stderr:512 bytes copied, 0.000317819 s, 1.6 MB/s 2024-09-16T15:41:38.704 DEBUG:teuthology.orchestra.run.smithi123:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-09-16T15:41:38.751 DEBUG:teuthology.orchestra.run.smithi123:> stat /dev/vg_nvme/lv_3 2024-09-16T15:41:38.798 INFO:teuthology.orchestra.run.smithi123.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-09-16T15:41:38.798 INFO:teuthology.orchestra.run.smithi123.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-16T15:41:38.798 INFO:teuthology.orchestra.run.smithi123.stdout:Device: 5h/5d Inode: 904 Links: 1 2024-09-16T15:41:38.798 INFO:teuthology.orchestra.run.smithi123.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-16T15:41:38.798 INFO:teuthology.orchestra.run.smithi123.stdout:Access: 2024-09-16 15:35:13.629318641 +0000 2024-09-16T15:41:38.798 INFO:teuthology.orchestra.run.smithi123.stdout:Modify: 2024-09-16 15:35:13.381320853 +0000 2024-09-16T15:41:38.798 INFO:teuthology.orchestra.run.smithi123.stdout:Change: 2024-09-16 15:35:13.381320853 +0000 2024-09-16T15:41:38.798 INFO:teuthology.orchestra.run.smithi123.stdout: Birth: - 2024-09-16T15:41:38.798 DEBUG:teuthology.orchestra.run.smithi123:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-09-16T15:41:38.851 INFO:teuthology.orchestra.run.smithi123.stderr:1+0 records in 2024-09-16T15:41:38.851 INFO:teuthology.orchestra.run.smithi123.stderr:1+0 records out 2024-09-16T15:41:38.851 INFO:teuthology.orchestra.run.smithi123.stderr:512 bytes copied, 0.000311379 s, 1.6 MB/s 2024-09-16T15:41:38.853 DEBUG:teuthology.orchestra.run.smithi123:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-09-16T15:41:38.899 DEBUG:teuthology.orchestra.run.smithi123:> stat /dev/vg_nvme/lv_4 2024-09-16T15:41:38.946 INFO:teuthology.orchestra.run.smithi123.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-09-16T15:41:38.946 INFO:teuthology.orchestra.run.smithi123.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-16T15:41:38.946 INFO:teuthology.orchestra.run.smithi123.stdout:Device: 5h/5d Inode: 926 Links: 1 2024-09-16T15:41:38.946 INFO:teuthology.orchestra.run.smithi123.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-16T15:41:38.946 INFO:teuthology.orchestra.run.smithi123.stdout:Access: 2024-09-16 15:35:14.173313789 +0000 2024-09-16T15:41:38.946 INFO:teuthology.orchestra.run.smithi123.stdout:Modify: 2024-09-16 15:35:13.873316465 +0000 2024-09-16T15:41:38.946 INFO:teuthology.orchestra.run.smithi123.stdout:Change: 2024-09-16 15:35:13.873316465 +0000 2024-09-16T15:41:38.946 INFO:teuthology.orchestra.run.smithi123.stdout: Birth: - 2024-09-16T15:41:38.946 DEBUG:teuthology.orchestra.run.smithi123:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-09-16T15:41:39.000 INFO:teuthology.orchestra.run.smithi123.stderr:1+0 records in 2024-09-16T15:41:39.000 INFO:teuthology.orchestra.run.smithi123.stderr:1+0 records out 2024-09-16T15:41:39.000 INFO:teuthology.orchestra.run.smithi123.stderr:512 bytes copied, 0.000352864 s, 1.5 MB/s 2024-09-16T15:41:39.001 DEBUG:teuthology.orchestra.run.smithi123:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-09-16T15:41:39.051 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-09-16T15:41:39.051 DEBUG:teuthology.orchestra.run.smithi184:> set -ex 2024-09-16T15:41:39.051 DEBUG:teuthology.orchestra.run.smithi184:> dd if=/scratch_devs of=/dev/stdout 2024-09-16T15:41:39.057 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-09-16T15:41:39.057 DEBUG:teuthology.orchestra.run.smithi184:> stat /dev/vg_nvme/lv_1 2024-09-16T15:41:39.106 INFO:teuthology.orchestra.run.smithi184.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-09-16T15:41:39.106 INFO:teuthology.orchestra.run.smithi184.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-16T15:41:39.106 INFO:teuthology.orchestra.run.smithi184.stdout:Device: 5h/5d Inode: 880 Links: 1 2024-09-16T15:41:39.106 INFO:teuthology.orchestra.run.smithi184.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-16T15:41:39.106 INFO:teuthology.orchestra.run.smithi184.stdout:Access: 2024-09-16 15:35:12.632531327 +0000 2024-09-16T15:41:39.107 INFO:teuthology.orchestra.run.smithi184.stdout:Modify: 2024-09-16 15:35:12.368533686 +0000 2024-09-16T15:41:39.107 INFO:teuthology.orchestra.run.smithi184.stdout:Change: 2024-09-16 15:35:12.368533686 +0000 2024-09-16T15:41:39.107 INFO:teuthology.orchestra.run.smithi184.stdout: Birth: - 2024-09-16T15:41:39.107 DEBUG:teuthology.orchestra.run.smithi184:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-09-16T15:41:39.160 INFO:teuthology.orchestra.run.smithi184.stderr:1+0 records in 2024-09-16T15:41:39.161 INFO:teuthology.orchestra.run.smithi184.stderr:1+0 records out 2024-09-16T15:41:39.161 INFO:teuthology.orchestra.run.smithi184.stderr:512 bytes copied, 0.000422881 s, 1.2 MB/s 2024-09-16T15:41:39.162 DEBUG:teuthology.orchestra.run.smithi184:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-09-16T15:41:39.211 DEBUG:teuthology.orchestra.run.smithi184:> stat /dev/vg_nvme/lv_2 2024-09-16T15:41:39.258 INFO:teuthology.orchestra.run.smithi184.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-09-16T15:41:39.258 INFO:teuthology.orchestra.run.smithi184.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-16T15:41:39.258 INFO:teuthology.orchestra.run.smithi184.stdout:Device: 5h/5d Inode: 893 Links: 1 2024-09-16T15:41:39.258 INFO:teuthology.orchestra.run.smithi184.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-16T15:41:39.258 INFO:teuthology.orchestra.run.smithi184.stdout:Access: 2024-09-16 15:35:13.116527004 +0000 2024-09-16T15:41:39.258 INFO:teuthology.orchestra.run.smithi184.stdout:Modify: 2024-09-16 15:35:12.860529291 +0000 2024-09-16T15:41:39.258 INFO:teuthology.orchestra.run.smithi184.stdout:Change: 2024-09-16 15:35:12.860529291 +0000 2024-09-16T15:41:39.258 INFO:teuthology.orchestra.run.smithi184.stdout: Birth: - 2024-09-16T15:41:39.258 DEBUG:teuthology.orchestra.run.smithi184:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-09-16T15:41:39.312 INFO:teuthology.orchestra.run.smithi184.stderr:1+0 records in 2024-09-16T15:41:39.313 INFO:teuthology.orchestra.run.smithi184.stderr:1+0 records out 2024-09-16T15:41:39.313 INFO:teuthology.orchestra.run.smithi184.stderr:512 bytes copied, 0.000443054 s, 1.2 MB/s 2024-09-16T15:41:39.314 DEBUG:teuthology.orchestra.run.smithi184:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-09-16T15:41:39.364 DEBUG:teuthology.orchestra.run.smithi184:> stat /dev/vg_nvme/lv_3 2024-09-16T15:41:39.414 INFO:teuthology.orchestra.run.smithi184.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-09-16T15:41:39.414 INFO:teuthology.orchestra.run.smithi184.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-16T15:41:39.414 INFO:teuthology.orchestra.run.smithi184.stdout:Device: 5h/5d Inode: 912 Links: 1 2024-09-16T15:41:39.414 INFO:teuthology.orchestra.run.smithi184.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-16T15:41:39.414 INFO:teuthology.orchestra.run.smithi184.stdout:Access: 2024-09-16 15:35:13.596522716 +0000 2024-09-16T15:41:39.414 INFO:teuthology.orchestra.run.smithi184.stdout:Modify: 2024-09-16 15:35:13.344524968 +0000 2024-09-16T15:41:39.415 INFO:teuthology.orchestra.run.smithi184.stdout:Change: 2024-09-16 15:35:13.344524968 +0000 2024-09-16T15:41:39.415 INFO:teuthology.orchestra.run.smithi184.stdout: Birth: - 2024-09-16T15:41:39.415 DEBUG:teuthology.orchestra.run.smithi184:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-09-16T15:41:39.468 INFO:teuthology.orchestra.run.smithi184.stderr:1+0 records in 2024-09-16T15:41:39.468 INFO:teuthology.orchestra.run.smithi184.stderr:1+0 records out 2024-09-16T15:41:39.468 INFO:teuthology.orchestra.run.smithi184.stderr:512 bytes copied, 0.00040646 s, 1.3 MB/s 2024-09-16T15:41:39.470 DEBUG:teuthology.orchestra.run.smithi184:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-09-16T15:41:39.519 DEBUG:teuthology.orchestra.run.smithi184:> stat /dev/vg_nvme/lv_4 2024-09-16T15:41:39.566 INFO:teuthology.orchestra.run.smithi184.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-09-16T15:41:39.566 INFO:teuthology.orchestra.run.smithi184.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-16T15:41:39.566 INFO:teuthology.orchestra.run.smithi184.stdout:Device: 5h/5d Inode: 929 Links: 1 2024-09-16T15:41:39.566 INFO:teuthology.orchestra.run.smithi184.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-16T15:41:39.567 INFO:teuthology.orchestra.run.smithi184.stdout:Access: 2024-09-16 15:35:14.072518466 +0000 2024-09-16T15:41:39.567 INFO:teuthology.orchestra.run.smithi184.stdout:Modify: 2024-09-16 15:35:13.820520715 +0000 2024-09-16T15:41:39.567 INFO:teuthology.orchestra.run.smithi184.stdout:Change: 2024-09-16 15:35:13.820520715 +0000 2024-09-16T15:41:39.567 INFO:teuthology.orchestra.run.smithi184.stdout: Birth: - 2024-09-16T15:41:39.567 DEBUG:teuthology.orchestra.run.smithi184:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-09-16T15:41:39.621 INFO:teuthology.orchestra.run.smithi184.stderr:1+0 records in 2024-09-16T15:41:39.621 INFO:teuthology.orchestra.run.smithi184.stderr:1+0 records out 2024-09-16T15:41:39.621 INFO:teuthology.orchestra.run.smithi184.stderr:512 bytes copied, 0.00040736 s, 1.3 MB/s 2024-09-16T15:41:39.622 DEBUG:teuthology.orchestra.run.smithi184:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-09-16T15:41:39.671 INFO:tasks.ceph:osd dev map: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-09-16T15:41:39.671 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi123.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi184.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-09-16T15:41:39.671 INFO:tasks.ceph:Generating config... 2024-09-16T15:41:39.673 INFO:tasks.ceph:[client] pid file = 2024-09-16T15:41:39.673 INFO:tasks.ceph:[mgr] client mount timeout = 30 2024-09-16T15:41:39.673 INFO:tasks.ceph:[mgr] debug client = 20 2024-09-16T15:41:39.673 INFO:tasks.ceph:[mgr] debug finisher = 20 2024-09-16T15:41:39.673 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-09-16T15:41:39.673 INFO:tasks.ceph:[mgr] debug ms = 1 2024-09-16T15:41:39.673 INFO:tasks.ceph:[mon] debug mon = 20 2024-09-16T15:41:39.673 INFO:tasks.ceph:[mon] debug ms = 1 2024-09-16T15:41:39.673 INFO:tasks.ceph:[mon] debug paxos = 20 2024-09-16T15:41:39.673 INFO:tasks.ceph:[mon] mon down mkfs grace = 300 2024-09-16T15:41:39.673 INFO:tasks.ceph:[osd] bdev async discard = True 2024-09-16T15:41:39.674 INFO:tasks.ceph:[osd] bdev enable discard = True 2024-09-16T15:41:39.674 INFO:tasks.ceph:[osd] bluestore allocator = bitmap 2024-09-16T15:41:39.674 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2024-09-16T15:41:39.674 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2024-09-16T15:41:39.674 INFO:tasks.ceph:[osd] debug bluefs = 1/20 2024-09-16T15:41:39.674 INFO:tasks.ceph:[osd] debug bluestore = 1/20 2024-09-16T15:41:39.674 INFO:tasks.ceph:[osd] debug ms = 1 2024-09-16T15:41:39.674 INFO:tasks.ceph:[osd] debug osd = 20 2024-09-16T15:41:39.674 INFO:tasks.ceph:[osd] debug rocksdb = 4/10 2024-09-16T15:41:39.674 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2024-09-16T15:41:39.674 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2024-09-16T15:41:39.674 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2024-09-16T15:41:39.674 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2024-09-16T15:41:39.674 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2024-09-16T15:41:39.675 INFO:tasks.ceph:Setting up mon.a... 2024-09-16T15:41:39.675 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2024-09-16T15:41:39.710 INFO:teuthology.orchestra.run.smithi123.stdout:creating /etc/ceph/ceph.keyring 2024-09-16T15:41:39.719 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2024-09-16T15:41:39.796 DEBUG:teuthology.orchestra.run.smithi123:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-09-16T15:41:39.848 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.123'), ('mon.b', '172.21.15.184'), ('mon.c', '[v2:172.21.15.184:3301,v1:172.21.15.184:6790]')] 2024-09-16T15:41:39.849 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.123,172.21.15.184,[v2:172.21.15.184:3301,v1:172.21.15.184:6790]'}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false', 'client mount timeout': 30, 'debug client': 20, 'debug finisher': 20}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon down mkfs grace': 300}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'pid file': ''}, 'mon.a': {}, 'mon.b': {}, 'mon.c': {}, 'mds.a': {}, 'mds.c': {}, 'mds.b': {}, 'mds.d': {}} 2024-09-16T15:41:39.850 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2024-09-16T15:41:39.850 DEBUG:teuthology.orchestra.run.smithi123:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-09-16T15:41:39.894 DEBUG:teuthology.orchestra.run.smithi123:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.123 --add b 172.21.15.184 --addv c '[v2:172.21.15.184:3301,v1:172.21.15.184:6790]' --print /home/ubuntu/cephtest/ceph.monmap 2024-09-16T15:41:39.978 INFO:teuthology.orchestra.run.smithi123.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2024-09-16T15:41:39.979 INFO:teuthology.orchestra.run.smithi123.stdout:monmaptool: generated fsid 56077f89-18c8-486f-ae7e-eecc0bb7822c 2024-09-16T15:41:39.979 INFO:teuthology.orchestra.run.smithi123.stdout:setting min_mon_release = quincy 2024-09-16T15:41:39.979 INFO:teuthology.orchestra.run.smithi123.stdout:epoch 0 2024-09-16T15:41:39.979 INFO:teuthology.orchestra.run.smithi123.stdout:fsid 56077f89-18c8-486f-ae7e-eecc0bb7822c 2024-09-16T15:41:39.979 INFO:teuthology.orchestra.run.smithi123.stdout:last_changed 2024-09-16T15:41:39.980980+0000 2024-09-16T15:41:39.979 INFO:teuthology.orchestra.run.smithi123.stdout:created 2024-09-16T15:41:39.980980+0000 2024-09-16T15:41:39.979 INFO:teuthology.orchestra.run.smithi123.stdout:min_mon_release 17 (quincy) 2024-09-16T15:41:39.979 INFO:teuthology.orchestra.run.smithi123.stdout:election_strategy: 1 2024-09-16T15:41:39.979 INFO:teuthology.orchestra.run.smithi123.stdout:0: [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] mon.a 2024-09-16T15:41:39.979 INFO:teuthology.orchestra.run.smithi123.stdout:1: [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] mon.b 2024-09-16T15:41:39.979 INFO:teuthology.orchestra.run.smithi123.stdout:2: [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] mon.c 2024-09-16T15:41:39.979 INFO:teuthology.orchestra.run.smithi123.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2024-09-16T15:41:39.981 DEBUG:teuthology.orchestra.run.smithi123:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-09-16T15:41:40.026 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID 56077f89-18c8-486f-ae7e-eecc0bb7822c... 2024-09-16T15:41:40.027 DEBUG:teuthology.orchestra.run.smithi123:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-09-16T15:41:40.071 DEBUG:teuthology.orchestra.run.smithi184:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-09-16T15:41:40.099 INFO:teuthology.orchestra.run.smithi123.stdout:[global] 2024-09-16T15:41:40.100 INFO:teuthology.orchestra.run.smithi123.stdout: chdir = "" 2024-09-16T15:41:40.100 INFO:teuthology.orchestra.run.smithi123.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-09-16T15:41:40.100 INFO:teuthology.orchestra.run.smithi123.stdout: auth supported = cephx 2024-09-16T15:41:40.100 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-09-16T15:41:40.100 INFO:teuthology.orchestra.run.smithi123.stdout: filestore xattr use omap = true 2024-09-16T15:41:40.100 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-09-16T15:41:40.100 INFO:teuthology.orchestra.run.smithi123.stdout: mon clock drift allowed = 1.000 2024-09-16T15:41:40.100 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-09-16T15:41:40.100 INFO:teuthology.orchestra.run.smithi123.stdout: osd crush chooseleaf type = 0 2024-09-16T15:41:40.100 INFO:teuthology.orchestra.run.smithi123.stdout: auth debug = true 2024-09-16T15:41:40.100 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-09-16T15:41:40.100 INFO:teuthology.orchestra.run.smithi123.stdout: ms die on old message = true 2024-09-16T15:41:40.100 INFO:teuthology.orchestra.run.smithi123.stdout: ms die on bug = true 2024-09-16T15:41:40.101 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-09-16T15:41:40.101 INFO:teuthology.orchestra.run.smithi123.stdout: mon max pg per osd = 10000 # >= luminous 2024-09-16T15:41:40.101 INFO:teuthology.orchestra.run.smithi123.stdout: mon pg warn max object skew = 0 2024-09-16T15:41:40.101 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-09-16T15:41:40.101 INFO:teuthology.orchestra.run.smithi123.stdout: # disable pg_autoscaler by default for new pools 2024-09-16T15:41:40.101 INFO:teuthology.orchestra.run.smithi123.stdout: osd_pool_default_pg_autoscale_mode = off 2024-09-16T15:41:40.101 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-09-16T15:41:40.101 INFO:teuthology.orchestra.run.smithi123.stdout: osd pool default size = 2 2024-09-16T15:41:40.101 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-09-16T15:41:40.101 INFO:teuthology.orchestra.run.smithi123.stdout: mon osd allow primary affinity = true 2024-09-16T15:41:40.101 INFO:teuthology.orchestra.run.smithi123.stdout: mon osd allow pg remap = true 2024-09-16T15:41:40.101 INFO:teuthology.orchestra.run.smithi123.stdout: mon warn on legacy crush tunables = false 2024-09-16T15:41:40.101 INFO:teuthology.orchestra.run.smithi123.stdout: mon warn on crush straw calc version zero = false 2024-09-16T15:41:40.101 INFO:teuthology.orchestra.run.smithi123.stdout: mon warn on no sortbitwise = false 2024-09-16T15:41:40.102 INFO:teuthology.orchestra.run.smithi123.stdout: mon warn on osd down out interval zero = false 2024-09-16T15:41:40.102 INFO:teuthology.orchestra.run.smithi123.stdout: mon warn on too few osds = false 2024-09-16T15:41:40.102 INFO:teuthology.orchestra.run.smithi123.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-09-16T15:41:40.102 INFO:teuthology.orchestra.run.smithi123.stdout: mon_warn_on_pool_no_redundancy = false 2024-09-16T15:41:40.102 INFO:teuthology.orchestra.run.smithi123.stdout: mon_allow_pool_size_one = true 2024-09-16T15:41:40.102 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-09-16T15:41:40.102 INFO:teuthology.orchestra.run.smithi123.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-09-16T15:41:40.102 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-09-16T15:41:40.102 INFO:teuthology.orchestra.run.smithi123.stdout: osd default data pool replay window = 5 2024-09-16T15:41:40.102 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-09-16T15:41:40.102 INFO:teuthology.orchestra.run.smithi123.stdout: mon allow pool delete = true 2024-09-16T15:41:40.102 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-09-16T15:41:40.102 INFO:teuthology.orchestra.run.smithi123.stdout: mon cluster log file level = debug 2024-09-16T15:41:40.103 INFO:teuthology.orchestra.run.smithi123.stdout: debug asserts on shutdown = true 2024-09-16T15:41:40.103 INFO:teuthology.orchestra.run.smithi123.stdout: mon health detail to clog = false 2024-09-16T15:41:40.103 INFO:teuthology.orchestra.run.smithi123.stdout: mon host = "172.21.15.123,172.21.15.184,[v2:172.21.15.184:3301,v1:172.21.15.184:6790]" 2024-09-16T15:41:40.103 INFO:teuthology.orchestra.run.smithi123.stdout: fsid = 56077f89-18c8-486f-ae7e-eecc0bb7822c 2024-09-16T15:41:40.103 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-09-16T15:41:40.103 INFO:teuthology.orchestra.run.smithi123.stdout:[osd] 2024-09-16T15:41:40.103 INFO:teuthology.orchestra.run.smithi123.stdout: osd journal size = 100 2024-09-16T15:41:40.103 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-09-16T15:41:40.103 INFO:teuthology.orchestra.run.smithi123.stdout: osd scrub load threshold = 5.0 2024-09-16T15:41:40.103 INFO:teuthology.orchestra.run.smithi123.stdout: osd scrub max interval = 600 2024-09-16T15:41:40.103 INFO:teuthology.orchestra.run.smithi123.stdout: osd mclock profile = high_recovery_ops 2024-09-16T15:41:40.103 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-09-16T15:41:40.103 INFO:teuthology.orchestra.run.smithi123.stdout: osd recover clone overlap = true 2024-09-16T15:41:40.103 INFO:teuthology.orchestra.run.smithi123.stdout: osd recovery max chunk = 1048576 2024-09-16T15:41:40.104 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-09-16T15:41:40.104 INFO:teuthology.orchestra.run.smithi123.stdout: osd debug shutdown = true 2024-09-16T15:41:40.104 INFO:teuthology.orchestra.run.smithi123.stdout: osd debug op order = true 2024-09-16T15:41:40.104 INFO:teuthology.orchestra.run.smithi123.stdout: osd debug verify stray on activate = true 2024-09-16T15:41:40.104 INFO:teuthology.orchestra.run.smithi123.stdout: osd debug trim objects = true 2024-09-16T15:41:40.104 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-09-16T15:41:40.104 INFO:teuthology.orchestra.run.smithi123.stdout: osd open classes on start = true 2024-09-16T15:41:40.104 INFO:teuthology.orchestra.run.smithi123.stdout: osd debug pg log writeout = true 2024-09-16T15:41:40.104 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-09-16T15:41:40.104 INFO:teuthology.orchestra.run.smithi123.stdout: osd deep scrub update digest min age = 30 2024-09-16T15:41:40.104 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-09-16T15:41:40.104 INFO:teuthology.orchestra.run.smithi123.stdout: osd map max advance = 10 2024-09-16T15:41:40.104 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-09-16T15:41:40.104 INFO:teuthology.orchestra.run.smithi123.stdout: journal zero on create = true 2024-09-16T15:41:40.105 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-09-16T15:41:40.105 INFO:teuthology.orchestra.run.smithi123.stdout: filestore ondisk finisher threads = 3 2024-09-16T15:41:40.105 INFO:teuthology.orchestra.run.smithi123.stdout: filestore apply finisher threads = 3 2024-09-16T15:41:40.105 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-09-16T15:41:40.105 INFO:teuthology.orchestra.run.smithi123.stdout: bdev debug aio = true 2024-09-16T15:41:40.105 INFO:teuthology.orchestra.run.smithi123.stdout: osd debug misdirected ops = true 2024-09-16T15:41:40.105 INFO:teuthology.orchestra.run.smithi123.stdout: bdev async discard = True 2024-09-16T15:41:40.105 INFO:teuthology.orchestra.run.smithi123.stdout: bdev enable discard = True 2024-09-16T15:41:40.105 INFO:teuthology.orchestra.run.smithi123.stdout: bluestore allocator = bitmap 2024-09-16T15:41:40.105 INFO:teuthology.orchestra.run.smithi123.stdout: bluestore block size = 96636764160 2024-09-16T15:41:40.105 INFO:teuthology.orchestra.run.smithi123.stdout: bluestore fsck on mount = True 2024-09-16T15:41:40.105 INFO:teuthology.orchestra.run.smithi123.stdout: debug bluefs = 1/20 2024-09-16T15:41:40.106 INFO:teuthology.orchestra.run.smithi123.stdout: debug bluestore = 1/20 2024-09-16T15:41:40.106 INFO:teuthology.orchestra.run.smithi123.stdout: debug ms = 1 2024-09-16T15:41:40.106 INFO:teuthology.orchestra.run.smithi123.stdout: debug osd = 20 2024-09-16T15:41:40.106 INFO:teuthology.orchestra.run.smithi123.stdout: debug rocksdb = 4/10 2024-09-16T15:41:40.106 INFO:teuthology.orchestra.run.smithi123.stdout: mon osd backfillfull_ratio = 0.85 2024-09-16T15:41:40.106 INFO:teuthology.orchestra.run.smithi123.stdout: mon osd full ratio = 0.9 2024-09-16T15:41:40.106 INFO:teuthology.orchestra.run.smithi123.stdout: mon osd nearfull ratio = 0.8 2024-09-16T15:41:40.106 INFO:teuthology.orchestra.run.smithi123.stdout: osd failsafe full ratio = 0.95 2024-09-16T15:41:40.106 INFO:teuthology.orchestra.run.smithi123.stdout: osd objectstore = bluestore 2024-09-16T15:41:40.106 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-09-16T15:41:40.106 INFO:teuthology.orchestra.run.smithi123.stdout:[mgr] 2024-09-16T15:41:40.106 INFO:teuthology.orchestra.run.smithi123.stdout: debug ms = 1 2024-09-16T15:41:40.106 INFO:teuthology.orchestra.run.smithi123.stdout: debug mgr = 20 2024-09-16T15:41:40.106 INFO:teuthology.orchestra.run.smithi123.stdout: debug mon = 20 2024-09-16T15:41:40.107 INFO:teuthology.orchestra.run.smithi123.stdout: debug auth = 20 2024-09-16T15:41:40.107 INFO:teuthology.orchestra.run.smithi123.stdout: mon reweight min pgs per osd = 4 2024-09-16T15:41:40.107 INFO:teuthology.orchestra.run.smithi123.stdout: mon reweight min bytes per osd = 10 2024-09-16T15:41:40.107 INFO:teuthology.orchestra.run.smithi123.stdout: mgr/telemetry/nag = false 2024-09-16T15:41:40.107 INFO:teuthology.orchestra.run.smithi123.stdout: client mount timeout = 30 2024-09-16T15:41:40.107 INFO:teuthology.orchestra.run.smithi123.stdout: debug client = 20 2024-09-16T15:41:40.107 INFO:teuthology.orchestra.run.smithi123.stdout: debug finisher = 20 2024-09-16T15:41:40.107 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-09-16T15:41:40.107 INFO:teuthology.orchestra.run.smithi123.stdout:[mon] 2024-09-16T15:41:40.107 INFO:teuthology.orchestra.run.smithi123.stdout: debug ms = 1 2024-09-16T15:41:40.107 INFO:teuthology.orchestra.run.smithi123.stdout: debug mon = 20 2024-09-16T15:41:40.107 INFO:teuthology.orchestra.run.smithi123.stdout: debug paxos = 20 2024-09-16T15:41:40.107 INFO:teuthology.orchestra.run.smithi123.stdout: debug auth = 20 2024-09-16T15:41:40.108 INFO:teuthology.orchestra.run.smithi123.stdout: mon data avail warn = 5 2024-09-16T15:41:40.108 INFO:teuthology.orchestra.run.smithi123.stdout: mon mgr mkfs grace = 240 2024-09-16T15:41:40.108 INFO:teuthology.orchestra.run.smithi123.stdout: mon reweight min pgs per osd = 4 2024-09-16T15:41:40.108 INFO:teuthology.orchestra.run.smithi123.stdout: mon osd reporter subtree level = osd 2024-09-16T15:41:40.108 INFO:teuthology.orchestra.run.smithi123.stdout: mon osd prime pg temp = true 2024-09-16T15:41:40.108 INFO:teuthology.orchestra.run.smithi123.stdout: mon reweight min bytes per osd = 10 2024-09-16T15:41:40.108 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-09-16T15:41:40.108 INFO:teuthology.orchestra.run.smithi123.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-09-16T15:41:40.108 INFO:teuthology.orchestra.run.smithi123.stdout: auth mon ticket ttl = 660 # 11m 2024-09-16T15:41:40.108 INFO:teuthology.orchestra.run.smithi123.stdout: auth service ticket ttl = 240 # 4m 2024-09-16T15:41:40.108 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-09-16T15:41:40.108 INFO:teuthology.orchestra.run.smithi123.stdout: # don't complain about insecure global_id in the test suite 2024-09-16T15:41:40.108 INFO:teuthology.orchestra.run.smithi123.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-09-16T15:41:40.109 INFO:teuthology.orchestra.run.smithi123.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-09-16T15:41:40.109 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-09-16T15:41:40.109 INFO:teuthology.orchestra.run.smithi123.stdout: # 1m isn't quite enough 2024-09-16T15:41:40.109 INFO:teuthology.orchestra.run.smithi123.stdout: mon_down_mkfs_grace = 2m 2024-09-16T15:41:40.109 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-09-16T15:41:40.109 INFO:teuthology.orchestra.run.smithi123.stdout: mon_warn_on_filestore_osds = false 2024-09-16T15:41:40.109 INFO:teuthology.orchestra.run.smithi123.stdout: mon down mkfs grace = 300 2024-09-16T15:41:40.109 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-09-16T15:41:40.109 INFO:teuthology.orchestra.run.smithi123.stdout:[client] 2024-09-16T15:41:40.109 INFO:teuthology.orchestra.run.smithi123.stdout: rgw cache enabled = true 2024-09-16T15:41:40.109 INFO:teuthology.orchestra.run.smithi123.stdout: rgw enable ops log = true 2024-09-16T15:41:40.109 INFO:teuthology.orchestra.run.smithi123.stdout: rgw enable usage log = true 2024-09-16T15:41:40.109 INFO:teuthology.orchestra.run.smithi123.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-09-16T15:41:40.109 INFO:teuthology.orchestra.run.smithi123.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-09-16T15:41:40.109 INFO:teuthology.orchestra.run.smithi123.stdout: pid file = "" 2024-09-16T15:41:40.110 INFO:teuthology.orchestra.run.smithi123.stdout:[mon.a] 2024-09-16T15:41:40.110 INFO:teuthology.orchestra.run.smithi123.stdout:[mon.b] 2024-09-16T15:41:40.110 INFO:teuthology.orchestra.run.smithi123.stdout:[mon.c] 2024-09-16T15:41:40.110 INFO:teuthology.orchestra.run.smithi123.stdout:[mds.a] 2024-09-16T15:41:40.110 INFO:teuthology.orchestra.run.smithi123.stdout:[mds.c] 2024-09-16T15:41:40.110 INFO:teuthology.orchestra.run.smithi123.stdout:[mds.b] 2024-09-16T15:41:40.110 INFO:teuthology.orchestra.run.smithi123.stdout:[mds.d] 2024-09-16T15:41:40.110 INFO:teuthology.orchestra.run.smithi184.stdout:[global] 2024-09-16T15:41:40.111 INFO:teuthology.orchestra.run.smithi184.stdout: chdir = "" 2024-09-16T15:41:40.111 INFO:teuthology.orchestra.run.smithi184.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-09-16T15:41:40.111 INFO:teuthology.orchestra.run.smithi184.stdout: auth supported = cephx 2024-09-16T15:41:40.111 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-09-16T15:41:40.111 INFO:teuthology.orchestra.run.smithi184.stdout: filestore xattr use omap = true 2024-09-16T15:41:40.111 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-09-16T15:41:40.111 INFO:teuthology.orchestra.run.smithi184.stdout: mon clock drift allowed = 1.000 2024-09-16T15:41:40.111 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-09-16T15:41:40.111 INFO:teuthology.orchestra.run.smithi184.stdout: osd crush chooseleaf type = 0 2024-09-16T15:41:40.111 INFO:teuthology.orchestra.run.smithi184.stdout: auth debug = true 2024-09-16T15:41:40.111 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-09-16T15:41:40.111 INFO:teuthology.orchestra.run.smithi184.stdout: ms die on old message = true 2024-09-16T15:41:40.111 INFO:teuthology.orchestra.run.smithi184.stdout: ms die on bug = true 2024-09-16T15:41:40.111 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-09-16T15:41:40.111 INFO:teuthology.orchestra.run.smithi184.stdout: mon max pg per osd = 10000 # >= luminous 2024-09-16T15:41:40.112 INFO:teuthology.orchestra.run.smithi184.stdout: mon pg warn max object skew = 0 2024-09-16T15:41:40.112 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-09-16T15:41:40.112 INFO:teuthology.orchestra.run.smithi184.stdout: # disable pg_autoscaler by default for new pools 2024-09-16T15:41:40.112 INFO:teuthology.orchestra.run.smithi184.stdout: osd_pool_default_pg_autoscale_mode = off 2024-09-16T15:41:40.112 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-09-16T15:41:40.112 INFO:teuthology.orchestra.run.smithi184.stdout: osd pool default size = 2 2024-09-16T15:41:40.112 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-09-16T15:41:40.112 INFO:teuthology.orchestra.run.smithi184.stdout: mon osd allow primary affinity = true 2024-09-16T15:41:40.112 INFO:teuthology.orchestra.run.smithi184.stdout: mon osd allow pg remap = true 2024-09-16T15:41:40.112 INFO:teuthology.orchestra.run.smithi184.stdout: mon warn on legacy crush tunables = false 2024-09-16T15:41:40.112 INFO:teuthology.orchestra.run.smithi184.stdout: mon warn on crush straw calc version zero = false 2024-09-16T15:41:40.112 INFO:teuthology.orchestra.run.smithi184.stdout: mon warn on no sortbitwise = false 2024-09-16T15:41:40.112 INFO:teuthology.orchestra.run.smithi184.stdout: mon warn on osd down out interval zero = false 2024-09-16T15:41:40.113 INFO:teuthology.orchestra.run.smithi184.stdout: mon warn on too few osds = false 2024-09-16T15:41:40.113 INFO:teuthology.orchestra.run.smithi184.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-09-16T15:41:40.113 INFO:teuthology.orchestra.run.smithi184.stdout: mon_warn_on_pool_no_redundancy = false 2024-09-16T15:41:40.113 INFO:teuthology.orchestra.run.smithi184.stdout: mon_allow_pool_size_one = true 2024-09-16T15:41:40.113 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-09-16T15:41:40.113 INFO:teuthology.orchestra.run.smithi184.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-09-16T15:41:40.113 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-09-16T15:41:40.113 INFO:teuthology.orchestra.run.smithi184.stdout: osd default data pool replay window = 5 2024-09-16T15:41:40.113 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-09-16T15:41:40.113 INFO:teuthology.orchestra.run.smithi184.stdout: mon allow pool delete = true 2024-09-16T15:41:40.113 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-09-16T15:41:40.113 INFO:teuthology.orchestra.run.smithi184.stdout: mon cluster log file level = debug 2024-09-16T15:41:40.113 INFO:teuthology.orchestra.run.smithi184.stdout: debug asserts on shutdown = true 2024-09-16T15:41:40.113 INFO:teuthology.orchestra.run.smithi184.stdout: mon health detail to clog = false 2024-09-16T15:41:40.114 INFO:teuthology.orchestra.run.smithi184.stdout: mon host = "172.21.15.123,172.21.15.184,[v2:172.21.15.184:3301,v1:172.21.15.184:6790]" 2024-09-16T15:41:40.114 INFO:teuthology.orchestra.run.smithi184.stdout: fsid = 56077f89-18c8-486f-ae7e-eecc0bb7822c 2024-09-16T15:41:40.114 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-09-16T15:41:40.114 INFO:teuthology.orchestra.run.smithi184.stdout:[osd] 2024-09-16T15:41:40.114 INFO:teuthology.orchestra.run.smithi184.stdout: osd journal size = 100 2024-09-16T15:41:40.114 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-09-16T15:41:40.114 INFO:teuthology.orchestra.run.smithi184.stdout: osd scrub load threshold = 5.0 2024-09-16T15:41:40.114 INFO:teuthology.orchestra.run.smithi184.stdout: osd scrub max interval = 600 2024-09-16T15:41:40.114 INFO:teuthology.orchestra.run.smithi184.stdout: osd mclock profile = high_recovery_ops 2024-09-16T15:41:40.114 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-09-16T15:41:40.114 INFO:teuthology.orchestra.run.smithi184.stdout: osd recover clone overlap = true 2024-09-16T15:41:40.114 INFO:teuthology.orchestra.run.smithi184.stdout: osd recovery max chunk = 1048576 2024-09-16T15:41:40.114 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-09-16T15:41:40.114 INFO:teuthology.orchestra.run.smithi184.stdout: osd debug shutdown = true 2024-09-16T15:41:40.115 INFO:teuthology.orchestra.run.smithi184.stdout: osd debug op order = true 2024-09-16T15:41:40.115 INFO:teuthology.orchestra.run.smithi184.stdout: osd debug verify stray on activate = true 2024-09-16T15:41:40.115 INFO:teuthology.orchestra.run.smithi184.stdout: osd debug trim objects = true 2024-09-16T15:41:40.115 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-09-16T15:41:40.115 INFO:teuthology.orchestra.run.smithi184.stdout: osd open classes on start = true 2024-09-16T15:41:40.115 INFO:teuthology.orchestra.run.smithi184.stdout: osd debug pg log writeout = true 2024-09-16T15:41:40.115 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-09-16T15:41:40.115 INFO:teuthology.orchestra.run.smithi184.stdout: osd deep scrub update digest min age = 30 2024-09-16T15:41:40.115 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-09-16T15:41:40.115 INFO:teuthology.orchestra.run.smithi184.stdout: osd map max advance = 10 2024-09-16T15:41:40.115 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-09-16T15:41:40.115 INFO:teuthology.orchestra.run.smithi184.stdout: journal zero on create = true 2024-09-16T15:41:40.115 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-09-16T15:41:40.115 INFO:teuthology.orchestra.run.smithi184.stdout: filestore ondisk finisher threads = 3 2024-09-16T15:41:40.116 INFO:teuthology.orchestra.run.smithi184.stdout: filestore apply finisher threads = 3 2024-09-16T15:41:40.116 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-09-16T15:41:40.116 INFO:teuthology.orchestra.run.smithi184.stdout: bdev debug aio = true 2024-09-16T15:41:40.116 INFO:teuthology.orchestra.run.smithi184.stdout: osd debug misdirected ops = true 2024-09-16T15:41:40.116 INFO:teuthology.orchestra.run.smithi184.stdout: bdev async discard = True 2024-09-16T15:41:40.116 INFO:teuthology.orchestra.run.smithi184.stdout: bdev enable discard = True 2024-09-16T15:41:40.116 INFO:teuthology.orchestra.run.smithi184.stdout: bluestore allocator = bitmap 2024-09-16T15:41:40.116 INFO:teuthology.orchestra.run.smithi184.stdout: bluestore block size = 96636764160 2024-09-16T15:41:40.116 INFO:teuthology.orchestra.run.smithi184.stdout: bluestore fsck on mount = True 2024-09-16T15:41:40.116 INFO:teuthology.orchestra.run.smithi184.stdout: debug bluefs = 1/20 2024-09-16T15:41:40.116 INFO:teuthology.orchestra.run.smithi184.stdout: debug bluestore = 1/20 2024-09-16T15:41:40.116 INFO:teuthology.orchestra.run.smithi184.stdout: debug ms = 1 2024-09-16T15:41:40.116 INFO:teuthology.orchestra.run.smithi184.stdout: debug osd = 20 2024-09-16T15:41:40.116 INFO:teuthology.orchestra.run.smithi184.stdout: debug rocksdb = 4/10 2024-09-16T15:41:40.117 INFO:teuthology.orchestra.run.smithi184.stdout: mon osd backfillfull_ratio = 0.85 2024-09-16T15:41:40.117 INFO:teuthology.orchestra.run.smithi184.stdout: mon osd full ratio = 0.9 2024-09-16T15:41:40.117 INFO:teuthology.orchestra.run.smithi184.stdout: mon osd nearfull ratio = 0.8 2024-09-16T15:41:40.117 INFO:teuthology.orchestra.run.smithi184.stdout: osd failsafe full ratio = 0.95 2024-09-16T15:41:40.117 INFO:teuthology.orchestra.run.smithi184.stdout: osd objectstore = bluestore 2024-09-16T15:41:40.117 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-09-16T15:41:40.117 INFO:teuthology.orchestra.run.smithi184.stdout:[mgr] 2024-09-16T15:41:40.117 INFO:teuthology.orchestra.run.smithi184.stdout: debug ms = 1 2024-09-16T15:41:40.117 INFO:teuthology.orchestra.run.smithi184.stdout: debug mgr = 20 2024-09-16T15:41:40.117 INFO:teuthology.orchestra.run.smithi184.stdout: debug mon = 20 2024-09-16T15:41:40.117 INFO:teuthology.orchestra.run.smithi184.stdout: debug auth = 20 2024-09-16T15:41:40.117 INFO:teuthology.orchestra.run.smithi184.stdout: mon reweight min pgs per osd = 4 2024-09-16T15:41:40.117 INFO:teuthology.orchestra.run.smithi184.stdout: mon reweight min bytes per osd = 10 2024-09-16T15:41:40.117 INFO:teuthology.orchestra.run.smithi184.stdout: mgr/telemetry/nag = false 2024-09-16T15:41:40.117 INFO:teuthology.orchestra.run.smithi184.stdout: client mount timeout = 30 2024-09-16T15:41:40.118 INFO:teuthology.orchestra.run.smithi184.stdout: debug client = 20 2024-09-16T15:41:40.118 INFO:teuthology.orchestra.run.smithi184.stdout: debug finisher = 20 2024-09-16T15:41:40.118 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-09-16T15:41:40.118 INFO:teuthology.orchestra.run.smithi184.stdout:[mon] 2024-09-16T15:41:40.118 INFO:teuthology.orchestra.run.smithi184.stdout: debug ms = 1 2024-09-16T15:41:40.118 INFO:teuthology.orchestra.run.smithi184.stdout: debug mon = 20 2024-09-16T15:41:40.118 INFO:teuthology.orchestra.run.smithi184.stdout: debug paxos = 20 2024-09-16T15:41:40.118 INFO:teuthology.orchestra.run.smithi184.stdout: debug auth = 20 2024-09-16T15:41:40.118 INFO:teuthology.orchestra.run.smithi184.stdout: mon data avail warn = 5 2024-09-16T15:41:40.118 INFO:teuthology.orchestra.run.smithi184.stdout: mon mgr mkfs grace = 240 2024-09-16T15:41:40.118 INFO:teuthology.orchestra.run.smithi184.stdout: mon reweight min pgs per osd = 4 2024-09-16T15:41:40.118 INFO:teuthology.orchestra.run.smithi184.stdout: mon osd reporter subtree level = osd 2024-09-16T15:41:40.118 INFO:teuthology.orchestra.run.smithi184.stdout: mon osd prime pg temp = true 2024-09-16T15:41:40.118 INFO:teuthology.orchestra.run.smithi184.stdout: mon reweight min bytes per osd = 10 2024-09-16T15:41:40.119 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-09-16T15:41:40.119 INFO:teuthology.orchestra.run.smithi184.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-09-16T15:41:40.119 INFO:teuthology.orchestra.run.smithi184.stdout: auth mon ticket ttl = 660 # 11m 2024-09-16T15:41:40.119 INFO:teuthology.orchestra.run.smithi184.stdout: auth service ticket ttl = 240 # 4m 2024-09-16T15:41:40.119 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-09-16T15:41:40.119 INFO:teuthology.orchestra.run.smithi184.stdout: # don't complain about insecure global_id in the test suite 2024-09-16T15:41:40.119 INFO:teuthology.orchestra.run.smithi184.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-09-16T15:41:40.119 INFO:teuthology.orchestra.run.smithi184.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-09-16T15:41:40.119 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-09-16T15:41:40.119 INFO:teuthology.orchestra.run.smithi184.stdout: # 1m isn't quite enough 2024-09-16T15:41:40.119 INFO:teuthology.orchestra.run.smithi184.stdout: mon_down_mkfs_grace = 2m 2024-09-16T15:41:40.119 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-09-16T15:41:40.119 INFO:teuthology.orchestra.run.smithi184.stdout: mon_warn_on_filestore_osds = false 2024-09-16T15:41:40.119 INFO:teuthology.orchestra.run.smithi184.stdout: mon down mkfs grace = 300 2024-09-16T15:41:40.120 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-09-16T15:41:40.120 INFO:teuthology.orchestra.run.smithi184.stdout:[client] 2024-09-16T15:41:40.120 INFO:teuthology.orchestra.run.smithi184.stdout: rgw cache enabled = true 2024-09-16T15:41:40.120 INFO:teuthology.orchestra.run.smithi184.stdout: rgw enable ops log = true 2024-09-16T15:41:40.120 INFO:teuthology.orchestra.run.smithi184.stdout: rgw enable usage log = true 2024-09-16T15:41:40.120 INFO:teuthology.orchestra.run.smithi184.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-09-16T15:41:40.120 INFO:teuthology.orchestra.run.smithi184.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-09-16T15:41:40.120 INFO:teuthology.orchestra.run.smithi184.stdout: pid file = "" 2024-09-16T15:41:40.120 INFO:teuthology.orchestra.run.smithi184.stdout:[mon.a] 2024-09-16T15:41:40.120 INFO:teuthology.orchestra.run.smithi184.stdout:[mon.b] 2024-09-16T15:41:40.120 INFO:teuthology.orchestra.run.smithi184.stdout:[mon.c] 2024-09-16T15:41:40.120 INFO:teuthology.orchestra.run.smithi184.stdout:[mds.a] 2024-09-16T15:41:40.121 INFO:teuthology.orchestra.run.smithi184.stdout:[mds.c] 2024-09-16T15:41:40.121 INFO:teuthology.orchestra.run.smithi184.stdout:[mds.b] 2024-09-16T15:41:40.121 INFO:teuthology.orchestra.run.smithi184.stdout:[mds.d] 2024-09-16T15:41:40.122 INFO:tasks.ceph:Creating admin key on mon.a... 2024-09-16T15:41:40.122 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2024-09-16T15:41:40.215 INFO:tasks.ceph:Copying monmap to all nodes... 2024-09-16T15:41:40.215 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2024-09-16T15:41:40.215 DEBUG:teuthology.orchestra.run.smithi123:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2024-09-16T15:41:40.220 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2024-09-16T15:41:40.220 DEBUG:teuthology.orchestra.run.smithi123:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2024-09-16T15:41:40.269 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi123.front.sepia.ceph.com 2024-09-16T15:41:40.269 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2024-09-16T15:41:40.270 DEBUG:teuthology.orchestra.run.smithi123:> sudo dd of=/etc/ceph/ceph.keyring 2024-09-16T15:41:40.270 DEBUG:teuthology.orchestra.run.smithi123:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-09-16T15:41:40.333 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2024-09-16T15:41:40.333 DEBUG:teuthology.orchestra.run.smithi123:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-09-16T15:41:40.382 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi184.front.sepia.ceph.com 2024-09-16T15:41:40.382 DEBUG:teuthology.orchestra.run.smithi184:> set -ex 2024-09-16T15:41:40.382 DEBUG:teuthology.orchestra.run.smithi184:> sudo dd of=/etc/ceph/ceph.keyring 2024-09-16T15:41:40.382 DEBUG:teuthology.orchestra.run.smithi184:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-09-16T15:41:40.405 DEBUG:teuthology.orchestra.run.smithi184:> set -ex 2024-09-16T15:41:40.406 DEBUG:teuthology.orchestra.run.smithi184:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-09-16T15:41:40.454 INFO:tasks.ceph:Setting up mon nodes... 2024-09-16T15:41:40.454 INFO:tasks.ceph:Setting up mgr nodes... 2024-09-16T15:41:40.454 DEBUG:teuthology.orchestra.run.smithi123:> sudo mkdir -p /var/lib/ceph/mgr/ceph-y && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.y /var/lib/ceph/mgr/ceph-y/keyring 2024-09-16T15:41:40.499 INFO:teuthology.orchestra.run.smithi123.stdout:creating /var/lib/ceph/mgr/ceph-y/keyring 2024-09-16T15:41:40.502 DEBUG:teuthology.orchestra.run.smithi184:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2024-09-16T15:41:40.546 INFO:teuthology.orchestra.run.smithi184.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2024-09-16T15:41:40.563 INFO:tasks.ceph:Setting up mds nodes... 2024-09-16T15:41:40.563 DEBUG:teuthology.orchestra.run.smithi123:> sudo mkdir -p /var/lib/ceph/mds/ceph-a && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.a /var/lib/ceph/mds/ceph-a/keyring 2024-09-16T15:41:40.606 INFO:teuthology.orchestra.run.smithi123.stdout:creating /var/lib/ceph/mds/ceph-a/keyring 2024-09-16T15:41:40.609 DEBUG:teuthology.orchestra.run.smithi123:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-a 2024-09-16T15:41:40.666 DEBUG:teuthology.orchestra.run.smithi123:> sudo mkdir -p /var/lib/ceph/mds/ceph-c && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.c /var/lib/ceph/mds/ceph-c/keyring 2024-09-16T15:41:40.748 INFO:teuthology.orchestra.run.smithi123.stdout:creating /var/lib/ceph/mds/ceph-c/keyring 2024-09-16T15:41:40.751 DEBUG:teuthology.orchestra.run.smithi123:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-c 2024-09-16T15:41:40.805 DEBUG:teuthology.orchestra.run.smithi184:> sudo mkdir -p /var/lib/ceph/mds/ceph-b && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.b /var/lib/ceph/mds/ceph-b/keyring 2024-09-16T15:41:40.850 INFO:teuthology.orchestra.run.smithi184.stdout:creating /var/lib/ceph/mds/ceph-b/keyring 2024-09-16T15:41:40.853 DEBUG:teuthology.orchestra.run.smithi184:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-b 2024-09-16T15:41:40.865 DEBUG:teuthology.orchestra.run.smithi184:> sudo mkdir -p /var/lib/ceph/mds/ceph-d && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.d /var/lib/ceph/mds/ceph-d/keyring 2024-09-16T15:41:40.951 INFO:teuthology.orchestra.run.smithi184.stdout:creating /var/lib/ceph/mds/ceph-d/keyring 2024-09-16T15:41:40.954 DEBUG:teuthology.orchestra.run.smithi184:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-d 2024-09-16T15:41:41.008 INFO:tasks.ceph_client:Setting up client nodes... 2024-09-16T15:41:41.008 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-09-16T15:41:41.045 INFO:teuthology.orchestra.run.smithi123.stdout:creating /etc/ceph/ceph.client.0.keyring 2024-09-16T15:41:41.056 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.2 /etc/ceph/ceph.client.2.keyring && sudo chmod 0644 /etc/ceph/ceph.client.2.keyring 2024-09-16T15:41:41.130 INFO:teuthology.orchestra.run.smithi123.stdout:creating /etc/ceph/ceph.client.2.keyring 2024-09-16T15:41:41.141 DEBUG:teuthology.orchestra.run.smithi184:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/ceph.client.1.keyring && sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-09-16T15:41:41.175 INFO:teuthology.orchestra.run.smithi184.stdout:creating /etc/ceph/ceph.client.1.keyring 2024-09-16T15:41:41.187 DEBUG:teuthology.orchestra.run.smithi184:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.3 /etc/ceph/ceph.client.3.keyring && sudo chmod 0644 /etc/ceph/ceph.client.3.keyring 2024-09-16T15:41:41.266 INFO:teuthology.orchestra.run.smithi184.stdout:creating /etc/ceph/ceph.client.3.keyring 2024-09-16T15:41:41.277 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-09-16T15:41:41.277 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi123.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi184.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-09-16T15:41:41.278 DEBUG:teuthology.orchestra.run.smithi123:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2024-09-16T15:41:41.291 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-09-16T15:41:41.291 INFO:tasks.ceph:role: osd.0 2024-09-16T15:41:41.291 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi123.front.sepia.ceph.com 2024-09-16T15:41:41.292 DEBUG:teuthology.orchestra.run.smithi123:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-09-16T15:41:41.353 INFO:teuthology.orchestra.run.smithi123.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-09-16T15:41:41.353 INFO:teuthology.orchestra.run.smithi123.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-16T15:41:41.353 INFO:teuthology.orchestra.run.smithi123.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-16T15:41:41.353 INFO:teuthology.orchestra.run.smithi123.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-09-16T15:41:41.353 INFO:teuthology.orchestra.run.smithi123.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-16T15:41:41.353 INFO:teuthology.orchestra.run.smithi123.stdout: = sunit=0 swidth=0 blks 2024-09-16T15:41:41.353 INFO:teuthology.orchestra.run.smithi123.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-16T15:41:41.354 INFO:teuthology.orchestra.run.smithi123.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-09-16T15:41:41.354 INFO:teuthology.orchestra.run.smithi123.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-16T15:41:41.354 INFO:teuthology.orchestra.run.smithi123.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-16T15:41:41.803 INFO:teuthology.orchestra.run.smithi123.stdout:Discarding blocks...Done. 2024-09-16T15:41:41.805 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi123.front.sepia.ceph.com -o noatime 2024-09-16T15:41:41.805 DEBUG:teuthology.orchestra.run.smithi123:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2024-09-16T15:41:41.940 DEBUG:teuthology.orchestra.run.smithi123:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2024-09-16T15:41:41.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T15:41:41.952 INFO:teuthology.orchestra.run.smithi123.stderr:sudo: /sbin/restorecon: command not found 2024-09-16T15:41:41.952 DEBUG:teuthology.orchestra.run.smithi123:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2024-09-16T15:41:42.006 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-09-16T15:41:42.006 INFO:tasks.ceph:role: osd.1 2024-09-16T15:41:42.006 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi123.front.sepia.ceph.com 2024-09-16T15:41:42.006 DEBUG:teuthology.orchestra.run.smithi123:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-09-16T15:41:42.073 INFO:teuthology.orchestra.run.smithi123.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-09-16T15:41:42.073 INFO:teuthology.orchestra.run.smithi123.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-16T15:41:42.073 INFO:teuthology.orchestra.run.smithi123.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-16T15:41:42.073 INFO:teuthology.orchestra.run.smithi123.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-09-16T15:41:42.073 INFO:teuthology.orchestra.run.smithi123.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-16T15:41:42.073 INFO:teuthology.orchestra.run.smithi123.stdout: = sunit=0 swidth=0 blks 2024-09-16T15:41:42.073 INFO:teuthology.orchestra.run.smithi123.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-16T15:41:42.073 INFO:teuthology.orchestra.run.smithi123.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-09-16T15:41:42.074 INFO:teuthology.orchestra.run.smithi123.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-16T15:41:42.074 INFO:teuthology.orchestra.run.smithi123.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-16T15:41:42.506 INFO:teuthology.orchestra.run.smithi123.stdout:Discarding blocks...Done. 2024-09-16T15:41:42.508 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi123.front.sepia.ceph.com -o noatime 2024-09-16T15:41:42.508 DEBUG:teuthology.orchestra.run.smithi123:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2024-09-16T15:41:42.569 DEBUG:teuthology.orchestra.run.smithi123:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2024-09-16T15:41:42.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T15:41:42.626 INFO:teuthology.orchestra.run.smithi123.stderr:sudo: /sbin/restorecon: command not found 2024-09-16T15:41:42.627 DEBUG:teuthology.orchestra.run.smithi123:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2024-09-16T15:41:42.680 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-09-16T15:41:42.680 INFO:tasks.ceph:role: osd.2 2024-09-16T15:41:42.680 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi123.front.sepia.ceph.com 2024-09-16T15:41:42.680 DEBUG:teuthology.orchestra.run.smithi123:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-09-16T15:41:42.741 INFO:teuthology.orchestra.run.smithi123.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-09-16T15:41:42.741 INFO:teuthology.orchestra.run.smithi123.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-16T15:41:42.741 INFO:teuthology.orchestra.run.smithi123.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-16T15:41:42.742 INFO:teuthology.orchestra.run.smithi123.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-09-16T15:41:42.742 INFO:teuthology.orchestra.run.smithi123.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-16T15:41:42.742 INFO:teuthology.orchestra.run.smithi123.stdout: = sunit=0 swidth=0 blks 2024-09-16T15:41:42.742 INFO:teuthology.orchestra.run.smithi123.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-16T15:41:42.742 INFO:teuthology.orchestra.run.smithi123.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-09-16T15:41:42.742 INFO:teuthology.orchestra.run.smithi123.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-16T15:41:42.742 INFO:teuthology.orchestra.run.smithi123.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-16T15:41:43.217 INFO:teuthology.orchestra.run.smithi123.stdout:Discarding blocks...Done. 2024-09-16T15:41:43.219 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi123.front.sepia.ceph.com -o noatime 2024-09-16T15:41:43.219 DEBUG:teuthology.orchestra.run.smithi123:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2024-09-16T15:41:43.314 DEBUG:teuthology.orchestra.run.smithi123:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2024-09-16T15:41:43.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T15:41:43.325 INFO:teuthology.orchestra.run.smithi123.stderr:sudo: /sbin/restorecon: command not found 2024-09-16T15:41:43.326 DEBUG:teuthology.orchestra.run.smithi123:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2024-09-16T15:41:43.381 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-09-16T15:41:43.381 INFO:tasks.ceph:role: osd.3 2024-09-16T15:41:43.381 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi123.front.sepia.ceph.com 2024-09-16T15:41:43.381 DEBUG:teuthology.orchestra.run.smithi123:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-09-16T15:41:43.446 INFO:teuthology.orchestra.run.smithi123.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-09-16T15:41:43.446 INFO:teuthology.orchestra.run.smithi123.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-16T15:41:43.446 INFO:teuthology.orchestra.run.smithi123.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-16T15:41:43.446 INFO:teuthology.orchestra.run.smithi123.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-09-16T15:41:43.447 INFO:teuthology.orchestra.run.smithi123.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-16T15:41:43.447 INFO:teuthology.orchestra.run.smithi123.stdout: = sunit=0 swidth=0 blks 2024-09-16T15:41:43.447 INFO:teuthology.orchestra.run.smithi123.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-16T15:41:43.447 INFO:teuthology.orchestra.run.smithi123.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-09-16T15:41:43.447 INFO:teuthology.orchestra.run.smithi123.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-16T15:41:43.447 INFO:teuthology.orchestra.run.smithi123.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-16T15:41:43.903 INFO:teuthology.orchestra.run.smithi123.stdout:Discarding blocks...Done. 2024-09-16T15:41:43.905 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi123.front.sepia.ceph.com -o noatime 2024-09-16T15:41:43.905 DEBUG:teuthology.orchestra.run.smithi123:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-3 2024-09-16T15:41:44.048 DEBUG:teuthology.orchestra.run.smithi123:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2024-09-16T15:41:44.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T15:41:44.060 INFO:teuthology.orchestra.run.smithi123.stderr:sudo: /sbin/restorecon: command not found 2024-09-16T15:41:44.061 DEBUG:teuthology.orchestra.run.smithi123:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-16T15:41:44.165 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:41:44.161+0000 7f803423b7c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2024-09-16T15:41:44.173 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:41:44.169+0000 7f803423b7c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2024-09-16T15:41:44.173 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:41:44.169+0000 7f803423b7c0 -1 bdev(0x5586c53c7800 /var/lib/ceph/osd/ceph-0/block) open stat got: (1) Operation not permitted 2024-09-16T15:41:44.173 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:41:44.173+0000 7f803423b7c0 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2024-09-16T15:41:47.312 DEBUG:teuthology.orchestra.run.smithi123:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-09-16T15:41:47.327 DEBUG:teuthology.orchestra.run.smithi123:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-16T15:41:47.408 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:41:47.405+0000 7f5da0e1e7c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2024-09-16T15:41:47.409 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:41:47.405+0000 7f5da0e1e7c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2024-09-16T15:41:47.409 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:41:47.405+0000 7f5da0e1e7c0 -1 bdev(0x563c7f863800 /var/lib/ceph/osd/ceph-1/block) open stat got: (1) Operation not permitted 2024-09-16T15:41:47.409 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:41:47.405+0000 7f5da0e1e7c0 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2024-09-16T15:41:50.582 DEBUG:teuthology.orchestra.run.smithi123:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-09-16T15:41:50.638 DEBUG:teuthology.orchestra.run.smithi123:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-16T15:41:50.720 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:41:50.717+0000 7f8948f7c7c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2024-09-16T15:41:50.721 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:41:50.717+0000 7f8948f7c7c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2024-09-16T15:41:50.721 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:41:50.721+0000 7f8948f7c7c0 -1 bdev(0x5626f1125800 /var/lib/ceph/osd/ceph-2/block) open stat got: (1) Operation not permitted 2024-09-16T15:41:50.721 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:41:50.721+0000 7f8948f7c7c0 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2024-09-16T15:41:53.901 DEBUG:teuthology.orchestra.run.smithi123:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-09-16T15:41:53.955 DEBUG:teuthology.orchestra.run.smithi123:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-16T15:41:54.037 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:41:54.037+0000 7f2c93aa57c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2024-09-16T15:41:54.038 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:41:54.037+0000 7f2c93aa57c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2024-09-16T15:41:54.038 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:41:54.037+0000 7f2c93aa57c0 -1 bdev(0x563bddd89800 /var/lib/ceph/osd/ceph-3/block) open stat got: (1) Operation not permitted 2024-09-16T15:41:54.038 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:41:54.037+0000 7f2c93aa57c0 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2024-09-16T15:41:57.161 DEBUG:teuthology.orchestra.run.smithi123:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-09-16T15:41:57.217 DEBUG:teuthology.orchestra.run.smithi184:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2024-09-16T15:41:57.232 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-09-16T15:41:57.232 INFO:tasks.ceph:role: osd.4 2024-09-16T15:41:57.232 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi184.front.sepia.ceph.com 2024-09-16T15:41:57.232 DEBUG:teuthology.orchestra.run.smithi184:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-09-16T15:41:57.294 INFO:teuthology.orchestra.run.smithi184.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-09-16T15:41:57.294 INFO:teuthology.orchestra.run.smithi184.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-16T15:41:57.294 INFO:teuthology.orchestra.run.smithi184.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-16T15:41:57.295 INFO:teuthology.orchestra.run.smithi184.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-09-16T15:41:57.295 INFO:teuthology.orchestra.run.smithi184.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-16T15:41:57.295 INFO:teuthology.orchestra.run.smithi184.stdout: = sunit=0 swidth=0 blks 2024-09-16T15:41:57.295 INFO:teuthology.orchestra.run.smithi184.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-16T15:41:57.295 INFO:teuthology.orchestra.run.smithi184.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-09-16T15:41:57.295 INFO:teuthology.orchestra.run.smithi184.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-16T15:41:57.295 INFO:teuthology.orchestra.run.smithi184.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-16T15:41:57.776 INFO:teuthology.orchestra.run.smithi184.stdout:Discarding blocks...Done. 2024-09-16T15:41:57.778 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi184.front.sepia.ceph.com -o noatime 2024-09-16T15:41:57.778 DEBUG:teuthology.orchestra.run.smithi184:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-4 2024-09-16T15:41:57.939 DEBUG:teuthology.orchestra.run.smithi184:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2024-09-16T15:41:57.950 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: /sbin/restorecon: command not found 2024-09-16T15:41:57.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T15:41:57.951 DEBUG:teuthology.orchestra.run.smithi184:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2024-09-16T15:41:58.005 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-09-16T15:41:58.005 INFO:tasks.ceph:role: osd.5 2024-09-16T15:41:58.006 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi184.front.sepia.ceph.com 2024-09-16T15:41:58.006 DEBUG:teuthology.orchestra.run.smithi184:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-09-16T15:41:58.070 INFO:teuthology.orchestra.run.smithi184.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-09-16T15:41:58.070 INFO:teuthology.orchestra.run.smithi184.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-16T15:41:58.070 INFO:teuthology.orchestra.run.smithi184.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-16T15:41:58.070 INFO:teuthology.orchestra.run.smithi184.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-09-16T15:41:58.070 INFO:teuthology.orchestra.run.smithi184.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-16T15:41:58.070 INFO:teuthology.orchestra.run.smithi184.stdout: = sunit=0 swidth=0 blks 2024-09-16T15:41:58.070 INFO:teuthology.orchestra.run.smithi184.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-16T15:41:58.070 INFO:teuthology.orchestra.run.smithi184.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-09-16T15:41:58.070 INFO:teuthology.orchestra.run.smithi184.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-16T15:41:58.070 INFO:teuthology.orchestra.run.smithi184.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-16T15:41:58.509 INFO:teuthology.orchestra.run.smithi184.stdout:Discarding blocks...Done. 2024-09-16T15:41:58.511 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi184.front.sepia.ceph.com -o noatime 2024-09-16T15:41:58.511 DEBUG:teuthology.orchestra.run.smithi184:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-5 2024-09-16T15:41:58.595 DEBUG:teuthology.orchestra.run.smithi184:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2024-09-16T15:41:58.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T15:41:58.646 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: /sbin/restorecon: command not found 2024-09-16T15:41:58.647 DEBUG:teuthology.orchestra.run.smithi184:> sudo mkdir -p /var/lib/ceph/osd/ceph-6 2024-09-16T15:41:58.699 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-09-16T15:41:58.700 INFO:tasks.ceph:role: osd.6 2024-09-16T15:41:58.700 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi184.front.sepia.ceph.com 2024-09-16T15:41:58.700 DEBUG:teuthology.orchestra.run.smithi184:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-09-16T15:41:58.762 INFO:teuthology.orchestra.run.smithi184.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-09-16T15:41:58.762 INFO:teuthology.orchestra.run.smithi184.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-16T15:41:58.762 INFO:teuthology.orchestra.run.smithi184.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-16T15:41:58.762 INFO:teuthology.orchestra.run.smithi184.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-09-16T15:41:58.762 INFO:teuthology.orchestra.run.smithi184.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-16T15:41:58.762 INFO:teuthology.orchestra.run.smithi184.stdout: = sunit=0 swidth=0 blks 2024-09-16T15:41:58.762 INFO:teuthology.orchestra.run.smithi184.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-16T15:41:58.762 INFO:teuthology.orchestra.run.smithi184.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-09-16T15:41:58.762 INFO:teuthology.orchestra.run.smithi184.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-16T15:41:58.763 INFO:teuthology.orchestra.run.smithi184.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-16T15:41:59.221 INFO:teuthology.orchestra.run.smithi184.stdout:Discarding blocks...Done. 2024-09-16T15:41:59.223 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi184.front.sepia.ceph.com -o noatime 2024-09-16T15:41:59.223 DEBUG:teuthology.orchestra.run.smithi184:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-6 2024-09-16T15:41:59.366 DEBUG:teuthology.orchestra.run.smithi184:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-6 2024-09-16T15:41:59.377 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: /sbin/restorecon: command not found 2024-09-16T15:41:59.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T15:41:59.378 DEBUG:teuthology.orchestra.run.smithi184:> sudo mkdir -p /var/lib/ceph/osd/ceph-7 2024-09-16T15:41:59.434 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-09-16T15:41:59.434 INFO:tasks.ceph:role: osd.7 2024-09-16T15:41:59.434 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi184.front.sepia.ceph.com 2024-09-16T15:41:59.434 DEBUG:teuthology.orchestra.run.smithi184:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-09-16T15:41:59.497 INFO:teuthology.orchestra.run.smithi184.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-09-16T15:41:59.497 INFO:teuthology.orchestra.run.smithi184.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-16T15:41:59.498 INFO:teuthology.orchestra.run.smithi184.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-16T15:41:59.498 INFO:teuthology.orchestra.run.smithi184.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-09-16T15:41:59.498 INFO:teuthology.orchestra.run.smithi184.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-16T15:41:59.498 INFO:teuthology.orchestra.run.smithi184.stdout: = sunit=0 swidth=0 blks 2024-09-16T15:41:59.498 INFO:teuthology.orchestra.run.smithi184.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-16T15:41:59.498 INFO:teuthology.orchestra.run.smithi184.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-09-16T15:41:59.498 INFO:teuthology.orchestra.run.smithi184.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-16T15:41:59.498 INFO:teuthology.orchestra.run.smithi184.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-16T15:41:59.996 INFO:teuthology.orchestra.run.smithi184.stdout:Discarding blocks...Done. 2024-09-16T15:41:59.998 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi184.front.sepia.ceph.com -o noatime 2024-09-16T15:41:59.999 DEBUG:teuthology.orchestra.run.smithi184:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-7 2024-09-16T15:42:00.156 DEBUG:teuthology.orchestra.run.smithi184:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-7 2024-09-16T15:42:00.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T15:42:00.166 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: /sbin/restorecon: command not found 2024-09-16T15:42:00.167 DEBUG:teuthology.orchestra.run.smithi184:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-16T15:42:00.274 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:42:00.273+0000 7f1f26fe57c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2024-09-16T15:42:00.277 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:42:00.273+0000 7f1f26fe57c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2024-09-16T15:42:00.277 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:42:00.273+0000 7f1f26fe57c0 -1 bdev(0x55a61748d800 /var/lib/ceph/osd/ceph-4/block) open stat got: (1) Operation not permitted 2024-09-16T15:42:00.277 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:42:00.277+0000 7f1f26fe57c0 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2024-09-16T15:42:03.423 DEBUG:teuthology.orchestra.run.smithi184:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-09-16T15:42:03.438 DEBUG:teuthology.orchestra.run.smithi184:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-16T15:42:03.520 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:42:03.517+0000 7f3b002d87c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2024-09-16T15:42:03.520 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:42:03.517+0000 7f3b002d87c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2024-09-16T15:42:03.521 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:42:03.517+0000 7f3b002d87c0 -1 bdev(0x55cd7a689800 /var/lib/ceph/osd/ceph-5/block) open stat got: (1) Operation not permitted 2024-09-16T15:42:03.521 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:42:03.517+0000 7f3b002d87c0 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2024-09-16T15:42:06.640 DEBUG:teuthology.orchestra.run.smithi184:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-09-16T15:42:06.693 DEBUG:teuthology.orchestra.run.smithi184:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 6 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-16T15:42:06.771 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:42:06.769+0000 7fc1d791f7c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-6/keyring: can't open /var/lib/ceph/osd/ceph-6/keyring: (2) No such file or directory 2024-09-16T15:42:06.772 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:42:06.769+0000 7fc1d791f7c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-6/keyring 2024-09-16T15:42:06.772 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:42:06.769+0000 7fc1d791f7c0 -1 bdev(0x55d3b194b800 /var/lib/ceph/osd/ceph-6/block) open stat got: (1) Operation not permitted 2024-09-16T15:42:06.772 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:42:06.769+0000 7fc1d791f7c0 -1 bluestore(/var/lib/ceph/osd/ceph-6) _read_fsid unparsable uuid 2024-09-16T15:42:08.337 DEBUG:teuthology.orchestra.run.smithi123:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-16T15:42:08.340 DEBUG:teuthology.orchestra.run.smithi184:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-16T15:42:09.905 DEBUG:teuthology.orchestra.run.smithi184:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-09-16T15:42:09.919 DEBUG:teuthology.orchestra.run.smithi184:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 7 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-16T15:42:09.999 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:42:09.997+0000 7fbb2f75c7c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-7/keyring: can't open /var/lib/ceph/osd/ceph-7/keyring: (2) No such file or directory 2024-09-16T15:42:10.000 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:42:09.997+0000 7fbb2f75c7c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-7/keyring 2024-09-16T15:42:10.000 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:42:09.997+0000 7fbb2f75c7c0 -1 bdev(0x55d4d9c73800 /var/lib/ceph/osd/ceph-7/block) open stat got: (1) Operation not permitted 2024-09-16T15:42:10.000 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:42:09.997+0000 7fbb2f75c7c0 -1 bluestore(/var/lib/ceph/osd/ceph-7) _read_fsid unparsable uuid 2024-09-16T15:42:13.141 DEBUG:teuthology.orchestra.run.smithi184:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-09-16T15:42:13.198 INFO:tasks.ceph:Reading keys from all nodes... 2024-09-16T15:42:13.198 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2024-09-16T15:42:13.198 DEBUG:teuthology.orchestra.run.smithi123:> sudo dd if=/var/lib/ceph/mgr/ceph-y/keyring of=/dev/stdout 2024-09-16T15:42:13.213 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2024-09-16T15:42:13.213 DEBUG:teuthology.orchestra.run.smithi123:> sudo dd if=/var/lib/ceph/mds/ceph-a/keyring of=/dev/stdout 2024-09-16T15:42:13.269 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2024-09-16T15:42:13.269 DEBUG:teuthology.orchestra.run.smithi123:> sudo dd if=/var/lib/ceph/mds/ceph-c/keyring of=/dev/stdout 2024-09-16T15:42:13.325 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2024-09-16T15:42:13.325 DEBUG:teuthology.orchestra.run.smithi123:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2024-09-16T15:42:13.381 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2024-09-16T15:42:13.381 DEBUG:teuthology.orchestra.run.smithi123:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2024-09-16T15:42:13.437 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2024-09-16T15:42:13.437 DEBUG:teuthology.orchestra.run.smithi123:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2024-09-16T15:42:13.493 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2024-09-16T15:42:13.493 DEBUG:teuthology.orchestra.run.smithi123:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2024-09-16T15:42:13.549 DEBUG:teuthology.orchestra.run.smithi184:> set -ex 2024-09-16T15:42:13.549 DEBUG:teuthology.orchestra.run.smithi184:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2024-09-16T15:42:13.563 DEBUG:teuthology.orchestra.run.smithi184:> set -ex 2024-09-16T15:42:13.563 DEBUG:teuthology.orchestra.run.smithi184:> sudo dd if=/var/lib/ceph/mds/ceph-b/keyring of=/dev/stdout 2024-09-16T15:42:13.617 DEBUG:teuthology.orchestra.run.smithi184:> set -ex 2024-09-16T15:42:13.617 DEBUG:teuthology.orchestra.run.smithi184:> sudo dd if=/var/lib/ceph/mds/ceph-d/keyring of=/dev/stdout 2024-09-16T15:42:13.673 DEBUG:teuthology.orchestra.run.smithi184:> set -ex 2024-09-16T15:42:13.673 DEBUG:teuthology.orchestra.run.smithi184:> sudo dd if=/var/lib/ceph/osd/ceph-4/keyring of=/dev/stdout 2024-09-16T15:42:13.726 DEBUG:teuthology.orchestra.run.smithi184:> set -ex 2024-09-16T15:42:13.726 DEBUG:teuthology.orchestra.run.smithi184:> sudo dd if=/var/lib/ceph/osd/ceph-5/keyring of=/dev/stdout 2024-09-16T15:42:13.781 DEBUG:teuthology.orchestra.run.smithi184:> set -ex 2024-09-16T15:42:13.782 DEBUG:teuthology.orchestra.run.smithi184:> sudo dd if=/var/lib/ceph/osd/ceph-6/keyring of=/dev/stdout 2024-09-16T15:42:13.836 DEBUG:teuthology.orchestra.run.smithi184:> set -ex 2024-09-16T15:42:13.836 DEBUG:teuthology.orchestra.run.smithi184:> sudo dd if=/var/lib/ceph/osd/ceph-7/keyring of=/dev/stdout 2024-09-16T15:42:13.889 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2024-09-16T15:42:13.889 DEBUG:teuthology.orchestra.run.smithi123:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2024-09-16T15:42:13.895 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2024-09-16T15:42:13.895 DEBUG:teuthology.orchestra.run.smithi123:> dd if=/etc/ceph/ceph.client.2.keyring of=/dev/stdout 2024-09-16T15:42:13.942 DEBUG:teuthology.orchestra.run.smithi184:> set -ex 2024-09-16T15:42:13.942 DEBUG:teuthology.orchestra.run.smithi184:> dd if=/etc/ceph/ceph.client.1.keyring of=/dev/stdout 2024-09-16T15:42:13.947 DEBUG:teuthology.orchestra.run.smithi184:> set -ex 2024-09-16T15:42:13.948 DEBUG:teuthology.orchestra.run.smithi184:> dd if=/etc/ceph/ceph.client.3.keyring of=/dev/stdout 2024-09-16T15:42:13.994 INFO:tasks.ceph:Adding keys to all mons... 2024-09-16T15:42:13.994 DEBUG:teuthology.orchestra.run.smithi123:> sudo tee -a /etc/ceph/ceph.keyring 2024-09-16T15:42:13.997 DEBUG:teuthology.orchestra.run.smithi184:> sudo tee -a /etc/ceph/ceph.keyring 2024-09-16T15:42:14.040 INFO:teuthology.orchestra.run.smithi123.stdout:[mgr.y] 2024-09-16T15:42:14.040 INFO:teuthology.orchestra.run.smithi123.stdout: key = AQC0UehmkjrlHRAA+3xbQC+SPWGWW1gND7xcAg== 2024-09-16T15:42:14.040 INFO:teuthology.orchestra.run.smithi123.stdout:[mds.a] 2024-09-16T15:42:14.040 INFO:teuthology.orchestra.run.smithi123.stdout: key = AQC0UehmbZpOJBAAaYGyDcWrt4k0FS5P/bb+4A== 2024-09-16T15:42:14.040 INFO:teuthology.orchestra.run.smithi123.stdout:[mds.c] 2024-09-16T15:42:14.040 INFO:teuthology.orchestra.run.smithi123.stdout: key = AQC0UehmCBjCLBAAXd85J25r7cgrXeDCG7pahg== 2024-09-16T15:42:14.041 INFO:teuthology.orchestra.run.smithi123.stdout:[osd.0] 2024-09-16T15:42:14.041 INFO:teuthology.orchestra.run.smithi123.stdout: key = AQC4UehmT633CRAAIhpkeIBOzwJO0kiXLu0x9Q== 2024-09-16T15:42:14.041 INFO:teuthology.orchestra.run.smithi123.stdout:[osd.1] 2024-09-16T15:42:14.041 INFO:teuthology.orchestra.run.smithi123.stdout: key = AQC7UehmFwF7GBAAvOUG+rWjANKT8wm/ai4xGg== 2024-09-16T15:42:14.041 INFO:teuthology.orchestra.run.smithi123.stdout:[osd.2] 2024-09-16T15:42:14.041 INFO:teuthology.orchestra.run.smithi123.stdout: key = AQC+UehmlCwWKxAAl3iO7ZALWyaorQH2urHKCw== 2024-09-16T15:42:14.041 INFO:teuthology.orchestra.run.smithi123.stdout:[osd.3] 2024-09-16T15:42:14.041 INFO:teuthology.orchestra.run.smithi123.stdout: key = AQDCUehmD8VgAhAAneoIeGr3MflSmi+nyavj2A== 2024-09-16T15:42:14.041 INFO:teuthology.orchestra.run.smithi123.stdout:[mgr.x] 2024-09-16T15:42:14.041 INFO:teuthology.orchestra.run.smithi123.stdout: key = AQC0UehmfPilIBAAERlBbduOgAU9ba0n/1tl3g== 2024-09-16T15:42:14.041 INFO:teuthology.orchestra.run.smithi123.stdout:[mds.b] 2024-09-16T15:42:14.041 INFO:teuthology.orchestra.run.smithi123.stdout: key = AQC0UehmF1HFMhAADcvqXNKdHZMPAbLc+se4Aw== 2024-09-16T15:42:14.042 INFO:teuthology.orchestra.run.smithi123.stdout:[mds.d] 2024-09-16T15:42:14.042 INFO:teuthology.orchestra.run.smithi123.stdout: key = AQC0UehmMh3ZOBAAx0RCa3R9RMT93/icZhPGyg== 2024-09-16T15:42:14.042 INFO:teuthology.orchestra.run.smithi123.stdout:[osd.4] 2024-09-16T15:42:14.042 INFO:teuthology.orchestra.run.smithi123.stdout: key = AQDIUehmA9B9EBAAsu0X5LPx1TmatZeHzjnVig== 2024-09-16T15:42:14.042 INFO:teuthology.orchestra.run.smithi123.stdout:[osd.5] 2024-09-16T15:42:14.042 INFO:teuthology.orchestra.run.smithi123.stdout: key = AQDLUehmcugdHxAAQMSS1ZH71FUzCx5A+yNnZg== 2024-09-16T15:42:14.042 INFO:teuthology.orchestra.run.smithi123.stdout:[osd.6] 2024-09-16T15:42:14.042 INFO:teuthology.orchestra.run.smithi123.stdout: key = AQDOUehmWLwYLhAAhOlQLepcWElsVPMnHlZm4w== 2024-09-16T15:42:14.042 INFO:teuthology.orchestra.run.smithi123.stdout:[osd.7] 2024-09-16T15:42:14.042 INFO:teuthology.orchestra.run.smithi123.stdout: key = AQDSUehmipkbABAAQuL/2O7w0tsM1OKeOil97w== 2024-09-16T15:42:14.042 INFO:teuthology.orchestra.run.smithi123.stdout:[client.0] 2024-09-16T15:42:14.042 INFO:teuthology.orchestra.run.smithi123.stdout: key = AQC1Uehm7GDTAhAAHWWVIyjVVnTeeozBR8DfFA== 2024-09-16T15:42:14.042 INFO:teuthology.orchestra.run.smithi123.stdout:[client.2] 2024-09-16T15:42:14.043 INFO:teuthology.orchestra.run.smithi123.stdout: key = AQC1Uehm1ADrBxAAW/8ayNHI6Wq/YlI4ivFZiw== 2024-09-16T15:42:14.043 INFO:teuthology.orchestra.run.smithi123.stdout:[client.1] 2024-09-16T15:42:14.043 INFO:teuthology.orchestra.run.smithi123.stdout: key = AQC1UehmFbaWChAAFtRmKucm+pbKtCNvRCTNqA== 2024-09-16T15:42:14.043 INFO:teuthology.orchestra.run.smithi123.stdout:[client.3] 2024-09-16T15:42:14.043 INFO:teuthology.orchestra.run.smithi123.stdout: key = AQC1UehmLv/3DxAAuGloBxSuza95uajpfiZ9MQ== 2024-09-16T15:42:14.048 INFO:teuthology.orchestra.run.smithi184.stdout:[mgr.y] 2024-09-16T15:42:14.048 INFO:teuthology.orchestra.run.smithi184.stdout: key = AQC0UehmkjrlHRAA+3xbQC+SPWGWW1gND7xcAg== 2024-09-16T15:42:14.048 INFO:teuthology.orchestra.run.smithi184.stdout:[mds.a] 2024-09-16T15:42:14.048 INFO:teuthology.orchestra.run.smithi184.stdout: key = AQC0UehmbZpOJBAAaYGyDcWrt4k0FS5P/bb+4A== 2024-09-16T15:42:14.048 INFO:teuthology.orchestra.run.smithi184.stdout:[mds.c] 2024-09-16T15:42:14.049 INFO:teuthology.orchestra.run.smithi184.stdout: key = AQC0UehmCBjCLBAAXd85J25r7cgrXeDCG7pahg== 2024-09-16T15:42:14.049 INFO:teuthology.orchestra.run.smithi184.stdout:[osd.0] 2024-09-16T15:42:14.049 INFO:teuthology.orchestra.run.smithi184.stdout: key = AQC4UehmT633CRAAIhpkeIBOzwJO0kiXLu0x9Q== 2024-09-16T15:42:14.049 INFO:teuthology.orchestra.run.smithi184.stdout:[osd.1] 2024-09-16T15:42:14.049 INFO:teuthology.orchestra.run.smithi184.stdout: key = AQC7UehmFwF7GBAAvOUG+rWjANKT8wm/ai4xGg== 2024-09-16T15:42:14.049 INFO:teuthology.orchestra.run.smithi184.stdout:[osd.2] 2024-09-16T15:42:14.049 INFO:teuthology.orchestra.run.smithi184.stdout: key = AQC+UehmlCwWKxAAl3iO7ZALWyaorQH2urHKCw== 2024-09-16T15:42:14.049 INFO:teuthology.orchestra.run.smithi184.stdout:[osd.3] 2024-09-16T15:42:14.049 INFO:teuthology.orchestra.run.smithi184.stdout: key = AQDCUehmD8VgAhAAneoIeGr3MflSmi+nyavj2A== 2024-09-16T15:42:14.049 INFO:teuthology.orchestra.run.smithi184.stdout:[mgr.x] 2024-09-16T15:42:14.049 INFO:teuthology.orchestra.run.smithi184.stdout: key = AQC0UehmfPilIBAAERlBbduOgAU9ba0n/1tl3g== 2024-09-16T15:42:14.049 INFO:teuthology.orchestra.run.smithi184.stdout:[mds.b] 2024-09-16T15:42:14.049 INFO:teuthology.orchestra.run.smithi184.stdout: key = AQC0UehmF1HFMhAADcvqXNKdHZMPAbLc+se4Aw== 2024-09-16T15:42:14.050 INFO:teuthology.orchestra.run.smithi184.stdout:[mds.d] 2024-09-16T15:42:14.050 INFO:teuthology.orchestra.run.smithi184.stdout: key = AQC0UehmMh3ZOBAAx0RCa3R9RMT93/icZhPGyg== 2024-09-16T15:42:14.050 INFO:teuthology.orchestra.run.smithi184.stdout:[osd.4] 2024-09-16T15:42:14.050 INFO:teuthology.orchestra.run.smithi184.stdout: key = AQDIUehmA9B9EBAAsu0X5LPx1TmatZeHzjnVig== 2024-09-16T15:42:14.050 INFO:teuthology.orchestra.run.smithi184.stdout:[osd.5] 2024-09-16T15:42:14.050 INFO:teuthology.orchestra.run.smithi184.stdout: key = AQDLUehmcugdHxAAQMSS1ZH71FUzCx5A+yNnZg== 2024-09-16T15:42:14.050 INFO:teuthology.orchestra.run.smithi184.stdout:[osd.6] 2024-09-16T15:42:14.050 INFO:teuthology.orchestra.run.smithi184.stdout: key = AQDOUehmWLwYLhAAhOlQLepcWElsVPMnHlZm4w== 2024-09-16T15:42:14.050 INFO:teuthology.orchestra.run.smithi184.stdout:[osd.7] 2024-09-16T15:42:14.050 INFO:teuthology.orchestra.run.smithi184.stdout: key = AQDSUehmipkbABAAQuL/2O7w0tsM1OKeOil97w== 2024-09-16T15:42:14.050 INFO:teuthology.orchestra.run.smithi184.stdout:[client.0] 2024-09-16T15:42:14.050 INFO:teuthology.orchestra.run.smithi184.stdout: key = AQC1Uehm7GDTAhAAHWWVIyjVVnTeeozBR8DfFA== 2024-09-16T15:42:14.050 INFO:teuthology.orchestra.run.smithi184.stdout:[client.2] 2024-09-16T15:42:14.050 INFO:teuthology.orchestra.run.smithi184.stdout: key = AQC1Uehm1ADrBxAAW/8ayNHI6Wq/YlI4ivFZiw== 2024-09-16T15:42:14.051 INFO:teuthology.orchestra.run.smithi184.stdout:[client.1] 2024-09-16T15:42:14.051 INFO:teuthology.orchestra.run.smithi184.stdout: key = AQC1UehmFbaWChAAFtRmKucm+pbKtCNvRCTNqA== 2024-09-16T15:42:14.051 INFO:teuthology.orchestra.run.smithi184.stdout:[client.3] 2024-09-16T15:42:14.051 INFO:teuthology.orchestra.run.smithi184.stdout: key = AQC1UehmLv/3DxAAuGloBxSuza95uajpfiZ9MQ== 2024-09-16T15:42:14.052 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-09-16T15:42:14.087 DEBUG:teuthology.orchestra.run.smithi184:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-09-16T15:42:14.138 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.a --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-09-16T15:42:14.175 DEBUG:teuthology.orchestra.run.smithi184:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.a --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-09-16T15:42:14.222 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.c --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-09-16T15:42:14.224 DEBUG:teuthology.orchestra.run.smithi184:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.c --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-09-16T15:42:14.308 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T15:42:14.310 DEBUG:teuthology.orchestra.run.smithi184:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T15:42:14.348 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T15:42:14.396 DEBUG:teuthology.orchestra.run.smithi184:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T15:42:14.439 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T15:42:14.483 DEBUG:teuthology.orchestra.run.smithi184:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T15:42:14.526 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T15:42:14.528 DEBUG:teuthology.orchestra.run.smithi184:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T15:42:14.603 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-09-16T15:42:14.611 DEBUG:teuthology.orchestra.run.smithi184:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-09-16T15:42:14.688 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.b --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-09-16T15:42:14.690 DEBUG:teuthology.orchestra.run.smithi184:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.b --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-09-16T15:42:14.776 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.d --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-09-16T15:42:14.779 DEBUG:teuthology.orchestra.run.smithi184:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.d --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-09-16T15:42:14.858 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T15:42:14.863 DEBUG:teuthology.orchestra.run.smithi184:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T15:42:14.939 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T15:42:14.947 DEBUG:teuthology.orchestra.run.smithi184:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T15:42:15.026 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T15:42:15.031 DEBUG:teuthology.orchestra.run.smithi184:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T15:42:15.073 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T15:42:15.119 DEBUG:teuthology.orchestra.run.smithi184:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T15:42:15.157 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-16T15:42:15.199 DEBUG:teuthology.orchestra.run.smithi184:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-16T15:42:15.247 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.2 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-16T15:42:15.249 DEBUG:teuthology.orchestra.run.smithi184:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.2 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-16T15:42:15.292 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-16T15:42:15.335 DEBUG:teuthology.orchestra.run.smithi184:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-16T15:42:15.383 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.3 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-16T15:42:15.385 DEBUG:teuthology.orchestra.run.smithi184:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.3 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-16T15:42:15.429 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-09-16T15:42:15.429 DEBUG:teuthology.orchestra.run.smithi123:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2024-09-16T15:42:15.477 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-09-16T15:42:15.573 DEBUG:teuthology.orchestra.run.smithi123:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2024-09-16T15:42:15.584 DEBUG:teuthology.orchestra.run.smithi184:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2024-09-16T15:42:15.598 DEBUG:teuthology.orchestra.run.smithi184:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-09-16T15:42:15.694 DEBUG:teuthology.orchestra.run.smithi184:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2024-09-16T15:42:15.705 DEBUG:teuthology.orchestra.run.smithi184:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2024-09-16T15:42:15.759 DEBUG:teuthology.orchestra.run.smithi184:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-09-16T15:42:15.846 DEBUG:teuthology.orchestra.run.smithi184:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2024-09-16T15:42:15.857 DEBUG:teuthology.orchestra.run.smithi123:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-09-16T15:42:15.859 DEBUG:teuthology.orchestra.run.smithi184:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-09-16T15:42:15.910 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2024-09-16T15:42:15.910 INFO:tasks.ceph.mon.a:Restarting daemon 2024-09-16T15:42:15.910 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-09-16T15:42:15.913 INFO:tasks.ceph.mon.a:Started 2024-09-16T15:42:15.913 INFO:tasks.ceph.mon.b:Restarting daemon 2024-09-16T15:42:15.913 DEBUG:teuthology.orchestra.run.smithi184:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2024-09-16T15:42:15.954 INFO:tasks.ceph.mon.b:Started 2024-09-16T15:42:15.955 INFO:tasks.ceph.mon.c:Restarting daemon 2024-09-16T15:42:15.955 DEBUG:teuthology.orchestra.run.smithi184:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2024-09-16T15:42:15.957 INFO:tasks.ceph.mon.c:Started 2024-09-16T15:42:15.957 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2024-09-16T15:42:15.957 INFO:tasks.ceph.mgr.y:Restarting daemon 2024-09-16T15:42:15.957 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i y 2024-09-16T15:42:15.960 INFO:tasks.ceph.mgr.y:Started 2024-09-16T15:42:15.960 INFO:tasks.ceph.mgr.x:Restarting daemon 2024-09-16T15:42:15.960 DEBUG:teuthology.orchestra.run.smithi184:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2024-09-16T15:42:15.962 INFO:tasks.ceph.mgr.x:Started 2024-09-16T15:42:15.962 INFO:tasks.ceph:setting config [client] client_mount_timeout = 600 2024-09-16T15:42:15.963 DEBUG:teuthology.orchestra.run.smithi123:> sudo ceph --cluster ceph config set client client_mount_timeout 600 2024-09-16T15:42:15.965 INFO:tasks.ceph:setting config [client] debug_client = 20 2024-09-16T15:42:15.965 DEBUG:teuthology.orchestra.run.smithi123:> sudo ceph --cluster ceph config set client debug_client 20 2024-09-16T15:42:16.010 INFO:tasks.ceph:setting config [client] debug_ms = 1 2024-09-16T15:42:16.010 DEBUG:teuthology.orchestra.run.smithi123:> sudo ceph --cluster ceph config set client debug_ms 1 2024-09-16T15:42:16.054 INFO:tasks.ceph:setting config [client] rados_mon_op_timeout = 900 2024-09-16T15:42:16.054 DEBUG:teuthology.orchestra.run.smithi123:> sudo ceph --cluster ceph config set client rados_mon_op_timeout 900 2024-09-16T15:42:16.098 INFO:tasks.ceph:setting config [client] rados_osd_op_timeout = 900 2024-09-16T15:42:16.098 DEBUG:teuthology.orchestra.run.smithi123:> sudo ceph --cluster ceph config set client rados_osd_op_timeout 900 2024-09-16T15:42:16.142 INFO:tasks.ceph:setting config [mds] debug_mds = 20 2024-09-16T15:42:16.142 DEBUG:teuthology.orchestra.run.smithi123:> sudo ceph --cluster ceph config set mds debug_mds 20 2024-09-16T15:42:16.190 INFO:tasks.ceph:setting config [mds] debug_mds_balancer = 20 2024-09-16T15:42:16.190 DEBUG:teuthology.orchestra.run.smithi123:> sudo ceph --cluster ceph config set mds debug_mds_balancer 20 2024-09-16T15:42:16.234 INFO:tasks.ceph:setting config [mds] debug_ms = 2 2024-09-16T15:42:16.234 DEBUG:teuthology.orchestra.run.smithi123:> sudo ceph --cluster ceph config set mds debug_ms 2 2024-09-16T15:42:16.278 INFO:tasks.ceph:setting config [mds] mds_debug_frag = True 2024-09-16T15:42:16.278 DEBUG:teuthology.orchestra.run.smithi123:> sudo ceph --cluster ceph config set mds mds_debug_frag True 2024-09-16T15:42:16.322 INFO:tasks.ceph:setting config [mds] mds_debug_scatterstat = True 2024-09-16T15:42:16.322 DEBUG:teuthology.orchestra.run.smithi123:> sudo ceph --cluster ceph config set mds mds_debug_scatterstat True 2024-09-16T15:42:16.366 INFO:tasks.ceph:setting config [mds] mds_op_complaint_time = 180 2024-09-16T15:42:16.366 DEBUG:teuthology.orchestra.run.smithi123:> sudo ceph --cluster ceph config set mds mds_op_complaint_time 180 2024-09-16T15:42:16.410 INFO:tasks.ceph:setting config [mds] mds_verify_scatter = True 2024-09-16T15:42:16.410 DEBUG:teuthology.orchestra.run.smithi123:> sudo ceph --cluster ceph config set mds mds_verify_scatter True 2024-09-16T15:42:16.454 INFO:tasks.ceph:setting config [mds] osd_op_complaint_time = 180 2024-09-16T15:42:16.454 DEBUG:teuthology.orchestra.run.smithi123:> sudo ceph --cluster ceph config set mds osd_op_complaint_time 180 2024-09-16T15:42:16.498 INFO:tasks.ceph:setting config [mds] rados_mon_op_timeout = 900 2024-09-16T15:42:16.498 DEBUG:teuthology.orchestra.run.smithi123:> sudo ceph --cluster ceph config set mds rados_mon_op_timeout 900 2024-09-16T15:42:16.542 INFO:tasks.ceph:setting config [mds] rados_osd_op_timeout = 900 2024-09-16T15:42:16.542 DEBUG:teuthology.orchestra.run.smithi123:> sudo ceph --cluster ceph config set mds rados_osd_op_timeout 900 2024-09-16T15:42:16.586 INFO:tasks.ceph:setting config [mon] mon_op_complaint_time = 120 2024-09-16T15:42:16.586 DEBUG:teuthology.orchestra.run.smithi123:> sudo ceph --cluster ceph config set mon mon_op_complaint_time 120 2024-09-16T15:42:16.634 INFO:tasks.ceph:setting config [mon] mon_warn_on_pool_no_app = False 2024-09-16T15:42:16.634 DEBUG:teuthology.orchestra.run.smithi123:> sudo ceph --cluster ceph config set mon mon_warn_on_pool_no_app False 2024-09-16T15:42:16.678 INFO:tasks.ceph:setting config [osd] osd_op_complaint_time = 180 2024-09-16T15:42:16.678 DEBUG:teuthology.orchestra.run.smithi123:> sudo ceph --cluster ceph config set osd osd_op_complaint_time 180 2024-09-16T15:42:16.722 DEBUG:tasks.ceph:set 18 configs 2024-09-16T15:42:16.722 DEBUG:tasks.ceph:waiting for RemoteProcess(client=, args=['sudo', 'ceph', '--cluster', 'ceph', 'config', 'set', 'client', 'client_mount_timeout', '600'], check_status=True, hostname='smithi123') 2024-09-16T15:42:26.657 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:26.652+0000 7f2f3d1c4640 1 -- 172.21.15.123:0/1746528084 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f2f200039f0 msgr2=0x7f2f380c6910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:42:26.657 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:26.652+0000 7f2f3d1c4640 1 --2- 172.21.15.123:0/1746528084 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f2f200039f0 0x7f2f380c6910 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f2f240072a0 tx=0x7f2f280016c0 comp rx=0 tx=0).stop 2024-09-16T15:42:26.658 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:26.652+0000 7f2756ffd640 1 -- 172.21.15.123:0/1053230492 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f276c154750 msgr2=0x7f276c14c400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:42:26.658 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:26.652+0000 7f2756ffd640 1 --2- 172.21.15.123:0/1053230492 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f276c154750 0x7f276c14c400 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f27580b0040 tx=0x7f2758019ef0 comp rx=0 tx=0).stop 2024-09-16T15:42:26.658 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:26.652+0000 7f2f3d1c4640 1 -- 172.21.15.123:0/1746528084 shutdown_connections 2024-09-16T15:42:26.659 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:26.652+0000 7f2f3d1c4640 1 --2- 172.21.15.123:0/1746528084 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2f380b53c0 0x7f2f380b5820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:42:26.659 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:26.652+0000 7f2f3d1c4640 1 --2- 172.21.15.123:0/1746528084 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2f24006260 0x7f2f380c6e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:42:26.659 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:26.652+0000 7f2f3d1c4640 1 --2- 172.21.15.123:0/1746528084 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f2f200039f0 0x7f2f380c6910 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:42:26.659 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:26.652+0000 7f2f3d1c4640 1 -- 172.21.15.123:0/1746528084 >> 172.21.15.123:0/1746528084 conn(0x7f2f380d0bc0 msgr2=0x7f2f380ade60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:42:26.659 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:26.656+0000 7f2f3d1c4640 1 -- 172.21.15.123:0/1746528084 shutdown_connections 2024-09-16T15:42:26.660 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:26.656+0000 7f2f3d1c4640 1 -- 172.21.15.123:0/1746528084 wait complete. 2024-09-16T15:42:26.661 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:26.656+0000 7f2756ffd640 1 -- 172.21.15.123:0/1053230492 shutdown_connections 2024-09-16T15:42:26.662 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:26.656+0000 7f2756ffd640 1 --2- 172.21.15.123:0/1053230492 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f276c14b090 0x7f276c14b510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:42:26.662 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:26.656+0000 7f2756ffd640 1 --2- 172.21.15.123:0/1053230492 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f276c154750 0x7f276c14c400 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:42:26.662 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:26.656+0000 7f2756ffd640 1 --2- 172.21.15.123:0/1053230492 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f275c004350 0x7f276c152da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:42:26.662 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:26.656+0000 7f2756ffd640 1 -- 172.21.15.123:0/1053230492 >> 172.21.15.123:0/1053230492 conn(0x7f276c0d0bf0 msgr2=0x7f276c0c7350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:42:26.666 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:26.664+0000 7f2756ffd640 1 -- 172.21.15.123:0/1053230492 shutdown_connections 2024-09-16T15:42:26.666 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:26.664+0000 7f2756ffd640 1 -- 172.21.15.123:0/1053230492 wait complete. 2024-09-16T15:42:26.682 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:26.676+0000 7f35a1b34640 1 -- 172.21.15.123:0/2754686381 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command([{prefix=config set, name=debug_ms}] v 0) -- 0x7f35740050f0 con 0x7f359c1f3d50 2024-09-16T15:42:26.688 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:26.684+0000 7f3583fff640 1 -- 172.21.15.123:0/2754686381 <== mon.0 v2:172.21.15.123:3300/0 9 ==== mon_command_ack([{prefix=config set, name=debug_ms}]=0 v7)=0 v7) ==== 106+0+0 (secure 0 0 0) 0x7f3594020070 con 0x7f359c1f3d50 2024-09-16T15:42:26.698 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:26.696+0000 7efcdcece640 1 -- 172.21.15.123:0/3570918530 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command([{prefix=config set, name=client_mount_timeout}] v 0) -- 0x7efcb0005110 con 0x7efcc40052f0 2024-09-16T15:42:26.704 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:26.700+0000 7fb3327fc640 1 -- 172.21.15.123:0/2661158985 <== mon.0 v2:172.21.15.123:3300/0 9 ==== config(3 keys) ==== 85+0+0 (secure 0 0 0) 0x7fb3380c5c30 con 0x7fb320006df0 2024-09-16T15:42:26.704 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:26.700+0000 7f2602ffd640 1 -- 172.21.15.123:0/3217953989 <== mon.0 v2:172.21.15.123:3300/0 9 ==== config(3 keys) ==== 85+0+0 (secure 0 0 0) 0x7f2604003c60 con 0x7f25f40072d0 2024-09-16T15:42:26.705 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:26.700+0000 7f805dffb640 1 -- 172.21.15.123:0/3378124232 <== mon.0 v2:172.21.15.123:3300/0 9 ==== config(3 keys) ==== 85+0+0 (secure 0 0 0) 0x7f8064028730 con 0x7f804c00ed80 2024-09-16T15:42:26.705 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:26.700+0000 7f3583fff640 1 -- 172.21.15.123:0/2754686381 <== mon.0 v2:172.21.15.123:3300/0 10 ==== config(3 keys) ==== 85+0+0 (secure 0 0 0) 0x7f35940249d0 con 0x7f359c1f3d50 2024-09-16T15:42:26.705 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:26.700+0000 7ff0d9ffb640 1 -- 172.21.15.123:0/3877245395 <== mon.0 v2:172.21.15.123:3300/0 9 ==== config(3 keys) ==== 85+0+0 (secure 0 0 0) 0x7ff0e00ae7e0 con 0x7ff0d00039c0 2024-09-16T15:42:26.706 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:26.700+0000 7f3220ff9640 1 -- 172.21.15.123:0/4092934445 <== mon.2 v2:172.21.15.184:3301/0 9 ==== config(3 keys) ==== 85+0+0 (secure 0 0 0) 0x7f321c0040e0 con 0x7f322415b360 2024-09-16T15:42:26.706 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:26.700+0000 7efccf7fe640 1 -- 172.21.15.123:0/3570918530 <== mon.0 v2:172.21.15.123:3300/0 9 ==== mon_command_ack([{prefix=config set, name=client_mount_timeout}]=0 v8)=0 v8) ==== 123+0+0 (secure 0 0 0) 0x7efcc0031770 con 0x7efcc40052f0 2024-09-16T15:42:26.706 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:26.700+0000 7efccf7fe640 1 -- 172.21.15.123:0/3570918530 <== mon.0 v2:172.21.15.123:3300/0 10 ==== config(3 keys) ==== 85+0+0 (secure 0 0 0) 0x7efcc0028e50 con 0x7efcc40052f0 2024-09-16T15:42:26.706 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:26.700+0000 7f0cd27fc640 1 -- 172.21.15.123:0/3539876130 <== mon.2 v2:172.21.15.184:3301/0 9 ==== config(3 keys) ==== 85+0+0 (secure 0 0 0) 0x7f0cd80313f0 con 0x7f0cc80039c0 2024-09-16T15:42:26.707 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:26.700+0000 7f8d4ffff640 1 -- 172.21.15.123:0/657446615 <== mon.2 v2:172.21.15.184:3301/0 9 ==== config(3 keys) ==== 85+0+0 (secure 0 0 0) 0x7f8d5c003cc0 con 0x7f8d68155930 2024-09-16T15:42:26.707 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:26.704+0000 7f35a1b34640 1 -- 172.21.15.123:0/2754686381 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f359c1f3d50 msgr2=0x7f359c1f9fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:42:26.708 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:26.704+0000 7f35a1b34640 1 --2- 172.21.15.123:0/2754686381 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f359c1f3d50 0x7f359c1f9fd0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f359c0cc6a0 tx=0x7f3594011aa0 comp rx=0 tx=0).stop 2024-09-16T15:42:26.708 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:26.704+0000 7f35a1b34640 1 -- 172.21.15.123:0/2754686381 shutdown_connections 2024-09-16T15:42:26.708 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:26.704+0000 7f35a1b34640 1 --2- 172.21.15.123:0/2754686381 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f359c1f3d50 0x7f359c1f9fd0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:42:26.708 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:26.704+0000 7f35a1b34640 1 --2- 172.21.15.123:0/2754686381 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f359c0b5df0 0x7f359c14d870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:42:26.708 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:26.704+0000 7f35a1b34640 1 --2- 172.21.15.123:0/2754686381 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f358800a570 0x7f359c0b4770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:42:26.708 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:26.704+0000 7f35a1b34640 1 -- 172.21.15.123:0/2754686381 >> 172.21.15.123:0/2754686381 conn(0x7f359c0741a0 msgr2=0x7f359c06ee50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:42:26.708 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:26.704+0000 7f35a1b34640 1 -- 172.21.15.123:0/2754686381 shutdown_connections 2024-09-16T15:42:26.708 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:26.704+0000 7f35a1b34640 1 -- 172.21.15.123:0/2754686381 wait complete. 2024-09-16T15:42:26.708 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:26.704+0000 7fa01cff9640 1 -- 172.21.15.123:0/2989593181 <== mon.1 v2:172.21.15.184:3300/0 9 ==== config(3 keys) ==== 85+0+0 (secure 0 0 0) 0x7fa0100286d0 con 0x7fa0080039c0 2024-09-16T15:42:26.709 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:26.704+0000 7f737bfff640 1 -- 172.21.15.123:0/1962648437 <== mon.1 v2:172.21.15.184:3300/0 9 ==== config(3 keys) ==== 85+0+0 (secure 0 0 0) 0x7f738c003bf0 con 0x7f738000a570 2024-09-16T15:42:26.716 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:26.712+0000 7f55c8f57640 1 -- 172.21.15.123:0/3974913995 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f55a400f380 msgr2=0x7f55a40117d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:42:26.716 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:26.712+0000 7f55c8f57640 1 --2- 172.21.15.123:0/3974913995 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f55a400f380 0x7f55a40117d0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f55b4005fd0 tx=0x7f55b40162c0 comp rx=0 tx=0).stop 2024-09-16T15:42:26.716 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:26.712+0000 7f55c2d9d640 1 -- 172.21.15.123:0/3974913995 reap_dead start 2024-09-16T15:42:26.716 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:26.712+0000 7f55c8f57640 1 -- 172.21.15.123:0/3974913995 shutdown_connections 2024-09-16T15:42:26.716 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:26.712+0000 7f55c8f57640 1 -- 172.21.15.123:0/3974913995 >> 172.21.15.123:0/3974913995 conn(0x7f55c40d0bf0 msgr2=0x7f55c40d1000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:42:26.716 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:26.712+0000 7f55c8f57640 1 -- 172.21.15.123:0/3974913995 shutdown_connections 2024-09-16T15:42:26.717 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:26.712+0000 7f7399ef4640 1 -- 172.21.15.123:0/1962648437 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_command([{prefix=config set, name=rados_osd_op_timeout}] v 0) -- 0x7f736c0050f0 con 0x7f738000a570 2024-09-16T15:42:26.718 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:26.712+0000 7f55c8f57640 1 -- 172.21.15.123:0/3974913995 wait complete. 2024-09-16T15:42:26.718 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:26.716+0000 7f55c8f57640 1 Processor -- start 2024-09-16T15:42:26.719 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:26.716+0000 7f55c8f57640 1 -- start start 2024-09-16T15:42:26.719 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:26.716+0000 7f55c8f57640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f55c4160bc0 0x7f55c40c7cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:42:26.719 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:26.716+0000 7f55c8f57640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f55c40cb1e0 0x7f55c40cb620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:42:26.721 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:26.716+0000 7f55c1d9b640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f55c40cb1e0 0x7f55c40cb620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:42:26.721 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:26.716+0000 7f55c1d9b640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f55c40cb1e0 0x7f55c40cb620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:49698/0 (socket says 172.21.15.123:49698) 2024-09-16T15:42:26.722 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:26.716+0000 7f55c1d9b640 1 -- 172.21.15.123:0/3913268890 learned_addr learned my addr 172.21.15.123:0/3913268890 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:42:26.722 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:26.716+0000 7f55c259c640 1 --2- 172.21.15.123:0/3913268890 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f55c4160bc0 0x7f55c40c7cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:42:26.722 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:26.716+0000 7f55c8f57640 1 --2- 172.21.15.123:0/3913268890 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f55c4207e20 0x7f55c42082a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:42:26.722 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:26.716+0000 7f55c8f57640 1 -- 172.21.15.123:0/3913268890 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f55c40c6710 con 0x7f55c40cb1e0 2024-09-16T15:42:26.722 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:26.716+0000 7f55c8f57640 1 -- 172.21.15.123:0/3913268890 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f55c40c6590 con 0x7f55c4160bc0 2024-09-16T15:42:26.722 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:26.716+0000 7f55c8f57640 1 -- 172.21.15.123:0/3913268890 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f55c4073600 con 0x7f55c4207e20 2024-09-16T15:42:26.723 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:26.716+0000 7f55c2d9d640 1 --2- 172.21.15.123:0/3913268890 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f55c4207e20 0x7f55c42082a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:42:26.723 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:26.716+0000 7fa026924640 1 -- 172.21.15.123:0/2989593181 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_command([{prefix=config set, name=rados_mon_op_timeout}] v 0) -- 0x7f9ff8005110 con 0x7fa0080039c0 2024-09-16T15:42:26.728 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:26.716+0000 7f55c1d9b640 1 -- 172.21.15.123:0/3913268890 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f55c4207e20 msgr2=0x7f55c42082a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:42:26.729 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:26.716+0000 7f55c1d9b640 1 --2- 172.21.15.123:0/3913268890 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f55c4207e20 0x7f55c42082a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:42:26.729 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:26.716+0000 7f55c1d9b640 1 -- 172.21.15.123:0/3913268890 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f55c4160bc0 msgr2=0x7f55c40c7cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:42:26.729 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:26.716+0000 7f55c1d9b640 1 --2- 172.21.15.123:0/3913268890 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f55c4160bc0 0x7f55c40c7cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:42:26.729 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:26.716+0000 7f55c1d9b640 1 -- 172.21.15.123:0/3913268890 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f55c4154700 con 0x7f55c40cb1e0 2024-09-16T15:42:26.729 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:26.716+0000 7f55c1d9b640 1 --2- 172.21.15.123:0/3913268890 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f55c40cb1e0 0x7f55c40cb620 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f55b8003d20 tx=0x7f55b8013820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:42:26.729 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:26.720+0000 7f55b2ffd640 1 -- 172.21.15.123:0/3913268890 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55b8001740 con 0x7f55c40cb1e0 2024-09-16T15:42:26.729 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:26.720+0000 7f55b2ffd640 1 -- 172.21.15.123:0/3913268890 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(3 keys) ==== 85+0+0 (secure 0 0 0) 0x7f55b80018e0 con 0x7f55c40cb1e0 2024-09-16T15:42:26.729 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:26.720+0000 7f55c8f57640 1 -- 172.21.15.123:0/3913268890 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f55c4204b60 con 0x7f55c40cb1e0 2024-09-16T15:42:26.730 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:26.720+0000 7f55c8f57640 1 -- 172.21.15.123:0/3913268890 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f55c4204830 con 0x7f55c40cb1e0 2024-09-16T15:42:26.730 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:26.720+0000 7f55b2ffd640 1 -- 172.21.15.123:0/3913268890 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55b8001cd0 con 0x7f55c40cb1e0 2024-09-16T15:42:26.730 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:26.720+0000 7f55b2ffd640 1 -- 172.21.15.123:0/3913268890 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f55b801d440 con 0x7f55c40cb1e0 2024-09-16T15:42:26.730 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:26.720+0000 7f55b2ffd640 1 -- 172.21.15.123:0/3913268890 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f55b8010d00 con 0x7f55c40cb1e0 2024-09-16T15:42:26.730 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:26.720+0000 7f737bfff640 1 -- 172.21.15.123:0/1962648437 <== mon.1 v2:172.21.15.184:3300/0 10 ==== mon_command_ack([{prefix=config set, name=rados_osd_op_timeout}]=0 v9)=0 v9) ==== 120+0+0 (secure 0 0 0) 0x7f73941f3980 con 0x7f738000a570 2024-09-16T15:42:26.730 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:26.720+0000 7f8d4dffb640 1 -- 172.21.15.123:0/657446615 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_command([{prefix=config set, name=osd_op_complaint_time}] v 0) -- 0x7f8d400050f0 con 0x7f8d68155930 2024-09-16T15:42:26.731 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:26.724+0000 7f55c8f57640 1 -- 172.21.15.123:0/3913268890 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f55a8005090 con 0x7f55c40cb1e0 2024-09-16T15:42:26.731 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:26.728+0000 7efcdcece640 1 -- 172.21.15.123:0/3570918530 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7efcc40052f0 msgr2=0x7efcd80af720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:42:26.731 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:26.728+0000 7efcdcece640 1 --2- 172.21.15.123:0/3570918530 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7efcc40052f0 0x7efcd80af720 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7efcc00119f0 tx=0x7efcc0012c00 comp rx=0 tx=0).stop 2024-09-16T15:42:26.731 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:26.728+0000 7efcdcece640 1 -- 172.21.15.123:0/3570918530 shutdown_connections 2024-09-16T15:42:26.731 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:26.728+0000 7efcdcece640 1 --2- 172.21.15.123:0/3570918530 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7efcd80af120 0x7efcd80c8ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:42:26.731 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:26.728+0000 7efcdcece640 1 --2- 172.21.15.123:0/3570918530 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7efcc40052f0 0x7efcd80af720 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:42:26.732 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:26.728+0000 7efcdcece640 1 --2- 172.21.15.123:0/3570918530 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7efcc4004810 0x7efcd8155ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:42:26.732 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:26.728+0000 7efcdcece640 1 -- 172.21.15.123:0/3570918530 >> 172.21.15.123:0/3570918530 conn(0x7efcd80d0bf0 msgr2=0x7efcd80cd500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:42:26.732 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:26.728+0000 7f55b2ffd640 1 -- 172.21.15.123:0/3913268890 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+75856 (secure 0 0 0) 0x7f55b80103f0 con 0x7f55c40cb1e0 2024-09-16T15:42:26.732 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:26.728+0000 7ff0d9ffb640 1 -- 172.21.15.123:0/3877245395 <== mon.0 v2:172.21.15.123:3300/0 10 ==== config(4 keys) ==== 116+0+0 (secure 0 0 0) 0x7ff0e00ae9b0 con 0x7ff0d00039c0 2024-09-16T15:42:26.733 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:26.728+0000 7f2602ffd640 1 -- 172.21.15.123:0/3217953989 <== mon.0 v2:172.21.15.123:3300/0 10 ==== config(4 keys) ==== 116+0+0 (secure 0 0 0) 0x7f2604004a00 con 0x7f25f40072d0 2024-09-16T15:42:26.734 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:26.728+0000 7f55b2ffd640 1 -- 172.21.15.123:0/3913268890 <== mon.0 v2:172.21.15.123:3300/0 7 ==== config(4 keys) ==== 116+0+0 (secure 0 0 0) 0x7f55b801d960 con 0x7f55c40cb1e0 2024-09-16T15:42:26.735 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:26.728+0000 7efcdcece640 1 -- 172.21.15.123:0/3570918530 shutdown_connections 2024-09-16T15:42:26.735 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:26.728+0000 7efcdcece640 1 -- 172.21.15.123:0/3570918530 wait complete. 2024-09-16T15:42:26.736 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:26.732+0000 7fb3327fc640 1 -- 172.21.15.123:0/2661158985 <== mon.0 v2:172.21.15.123:3300/0 10 ==== config(4 keys) ==== 116+0+0 (secure 0 0 0) 0x7fb3380ab560 con 0x7fb320006df0 2024-09-16T15:42:26.736 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:26.732+0000 7f0cd27fc640 1 -- 172.21.15.123:0/3539876130 <== mon.2 v2:172.21.15.184:3301/0 10 ==== config(4 keys) ==== 116+0+0 (secure 0 0 0) 0x7f0cd80315c0 con 0x7f0cc80039c0 2024-09-16T15:42:26.737 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:26.732+0000 7f3220ff9640 1 -- 172.21.15.123:0/4092934445 <== mon.2 v2:172.21.15.184:3301/0 10 ==== config(4 keys) ==== 116+0+0 (secure 0 0 0) 0x7f321c01fa90 con 0x7f322415b360 2024-09-16T15:42:26.737 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:26.732+0000 7f8d4ffff640 1 -- 172.21.15.123:0/657446615 <== mon.2 v2:172.21.15.184:3301/0 10 ==== config(4 keys) ==== 116+0+0 (secure 0 0 0) 0x7f8d5c028b30 con 0x7f8d68155930 2024-09-16T15:42:26.738 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:26.732+0000 7fa01cff9640 1 -- 172.21.15.123:0/2989593181 <== mon.1 v2:172.21.15.184:3300/0 10 ==== config(4 keys) ==== 116+0+0 (secure 0 0 0) 0x7fa010028880 con 0x7fa0080039c0 2024-09-16T15:42:26.738 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:26.732+0000 7f737bfff640 1 -- 172.21.15.123:0/1962648437 <== mon.1 v2:172.21.15.184:3300/0 11 ==== config(4 keys) ==== 116+0+0 (secure 0 0 0) 0x7f738c003e20 con 0x7f738000a570 2024-09-16T15:42:26.739 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:26.732+0000 7f805dffb640 1 -- 172.21.15.123:0/3378124232 <== mon.0 v2:172.21.15.123:3300/0 10 ==== config(4 keys) ==== 116+0+0 (secure 0 0 0) 0x7f8064028900 con 0x7f804c00ed80 2024-09-16T15:42:26.739 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:26.732+0000 7fa01cff9640 1 -- 172.21.15.123:0/2989593181 <== mon.1 v2:172.21.15.184:3300/0 11 ==== mon_command_ack([{prefix=config set, name=rados_mon_op_timeout}]=0 v10)=0 v10) ==== 123+0+0 (secure 0 0 0) 0x7fa01001e990 con 0x7fa0080039c0 2024-09-16T15:42:26.744 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:26.740+0000 7f8d4ffff640 1 -- 172.21.15.123:0/657446615 <== mon.2 v2:172.21.15.184:3301/0 11 ==== mon_command_ack([{prefix=config set, name=osd_op_complaint_time}]=0 v11)=0 v11) ==== 121+0+0 (secure 0 0 0) 0x7f8d5c01e450 con 0x7f8d68155930 2024-09-16T15:42:26.747 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:26.744+0000 7f8d4dffb640 1 -- 172.21.15.123:0/657446615 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f8d68155930 msgr2=0x7f8d680c9de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:42:26.747 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:26.744+0000 7f8d4dffb640 1 --2- 172.21.15.123:0/657446615 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f8d68155930 0x7f8d680c9de0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f8d5c005e60 tx=0x7f8d5c011ef0 comp rx=0 tx=0).stop 2024-09-16T15:42:26.747 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:26.744+0000 7f8d4dffb640 1 -- 172.21.15.123:0/657446615 shutdown_connections 2024-09-16T15:42:26.747 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:26.744+0000 7f8d4dffb640 1 --2- 172.21.15.123:0/657446615 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f8d68155e60 0x7f8d6815bcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:42:26.747 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:26.744+0000 7f8d4dffb640 1 --2- 172.21.15.123:0/657446615 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f8d68155930 0x7f8d680c9de0 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:42:26.747 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:26.744+0000 7f8d4dffb640 1 --2- 172.21.15.123:0/657446615 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8d68155170 0x7f8d680c98a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:42:26.748 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:26.744+0000 7f8d4dffb640 1 -- 172.21.15.123:0/657446615 >> 172.21.15.123:0/657446615 conn(0x7f8d680d0bc0 msgr2=0x7f8d680b5a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:42:26.748 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:26.744+0000 7f8d4dffb640 1 -- 172.21.15.123:0/657446615 shutdown_connections 2024-09-16T15:42:26.748 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:26.744+0000 7f8d4dffb640 1 -- 172.21.15.123:0/657446615 wait complete. 2024-09-16T15:42:26.756 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:26.752+0000 7f2618942640 1 -- 172.21.15.123:0/3217953989 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command([{prefix=config set, name=debug_mds_balancer}] v 0) -- 0x7f25ec005300 con 0x7f25f40072d0 2024-09-16T15:42:26.760 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:26.756+0000 7f2602ffd640 1 -- 172.21.15.123:0/3217953989 <== mon.0 v2:172.21.15.123:3300/0 11 ==== mon_command_ack([{prefix=config set, name=debug_mds_balancer}]=0 v12)=0 v12) ==== 117+0+0 (secure 0 0 0) 0x7f2604004e40 con 0x7f25f40072d0 2024-09-16T15:42:26.770 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:26.767+0000 7fa026924640 1 -- 172.21.15.123:0/2989593181 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa0080039c0 msgr2=0x7fa0200d00c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:42:26.770 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:26.767+0000 7fa026924640 1 --2- 172.21.15.123:0/2989593181 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa0080039c0 0x7fa0200d00c0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fa010004ca0 tx=0x7fa010013820 comp rx=0 tx=0).stop 2024-09-16T15:42:26.770 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:26.767+0000 7fa026924640 1 -- 172.21.15.123:0/2989593181 shutdown_connections 2024-09-16T15:42:26.770 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:26.767+0000 7f7399ef4640 1 -- 172.21.15.123:0/1962648437 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f738000a570 msgr2=0x7f7394159ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:42:26.770 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:26.767+0000 7f7399ef4640 1 --2- 172.21.15.123:0/1962648437 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f738000a570 0x7f7394159ab0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f738c01cac0 tx=0x7f737c005840 comp rx=0 tx=0).stop 2024-09-16T15:42:26.771 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:26.767+0000 7fa026924640 1 --2- 172.21.15.123:0/2989593181 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa0080052f0 0x7fa0200c7c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:42:26.771 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:26.767+0000 7fa026924640 1 --2- 172.21.15.123:0/2989593181 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fa008004810 0x7fa0200c6a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:42:26.771 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:26.767+0000 7fa026924640 1 --2- 172.21.15.123:0/2989593181 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa0080039c0 0x7fa0200d00c0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:42:26.771 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:26.767+0000 7fa026924640 1 -- 172.21.15.123:0/2989593181 >> 172.21.15.123:0/2989593181 conn(0x7fa0200d0bc0 msgr2=0x7fa0200b3b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:42:26.771 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:26.767+0000 7fa026924640 1 -- 172.21.15.123:0/2989593181 shutdown_connections 2024-09-16T15:42:26.772 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:26.767+0000 7f7399ef4640 1 -- 172.21.15.123:0/1962648437 shutdown_connections 2024-09-16T15:42:26.772 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:26.767+0000 7f7399ef4640 1 --2- 172.21.15.123:0/1962648437 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f738000a570 0x7f7394159ab0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:42:26.772 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:26.767+0000 7f7399ef4640 1 --2- 172.21.15.123:0/1962648437 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f737c004d50 0x7f7394158680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:42:26.772 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:26.767+0000 7f7399ef4640 1 --2- 172.21.15.123:0/1962648437 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f737c003fd0 0x7f73941f29b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:42:26.772 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:26.767+0000 7f7399ef4640 1 -- 172.21.15.123:0/1962648437 >> 172.21.15.123:0/1962648437 conn(0x7f73940d0bc0 msgr2=0x7f73940cb210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:42:26.773 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:26.767+0000 7fa026924640 1 -- 172.21.15.123:0/2989593181 wait complete. 2024-09-16T15:42:26.773 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:26.767+0000 7f7399ef4640 1 -- 172.21.15.123:0/1962648437 shutdown_connections 2024-09-16T15:42:26.773 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:26.767+0000 7f7399ef4640 1 -- 172.21.15.123:0/1962648437 wait complete. 2024-09-16T15:42:26.774 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:26.767+0000 7f0ce3ab6640 1 -- 172.21.15.123:0/3539876130 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_command([{prefix=config set, name=rados_mon_op_timeout}] v 0) -- 0x7f0cb8005300 con 0x7f0cc80039c0 2024-09-16T15:42:26.775 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:26.771+0000 7f806f29e640 1 -- 172.21.15.123:0/3378124232 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command([{prefix=config set, name=debug_client}] v 0) -- 0x7f8040005300 con 0x7f804c00ed80 2024-09-16T15:42:26.777 DEBUG:tasks.ceph:waiting for RemoteProcess(client=, args=['sudo', 'ceph', '--cluster', 'ceph', 'config', 'set', 'client', 'debug_client', '20'], check_status=True, hostname='smithi123') 2024-09-16T15:42:26.782 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:26.779+0000 7f2618942640 1 -- 172.21.15.123:0/3217953989 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f25f40072d0 msgr2=0x7f2614157720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:42:26.782 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:26.779+0000 7f2618942640 1 --2- 172.21.15.123:0/3217953989 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f25f40072d0 0x7f2614157720 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f2604012c40 tx=0x7f2604011810 comp rx=0 tx=0).stop 2024-09-16T15:42:26.782 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:26.779+0000 7f0cd27fc640 1 -- 172.21.15.123:0/3539876130 <== mon.2 v2:172.21.15.184:3301/0 11 ==== mon_command_ack([{prefix=config set, name=rados_mon_op_timeout}]=0 v13)=0 v13) ==== 120+0+0 (secure 0 0 0) 0x7f0cd8019040 con 0x7f0cc80039c0 2024-09-16T15:42:26.784 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:26.779+0000 7f2618942640 1 -- 172.21.15.123:0/3217953989 shutdown_connections 2024-09-16T15:42:26.784 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:26.779+0000 7f2618942640 1 --2- 172.21.15.123:0/3217953989 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f25fc004810 0x7f261415d400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:42:26.784 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:26.779+0000 7f2618942640 1 --2- 172.21.15.123:0/3217953989 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f25fc003a10 0x7f26140b3e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:42:26.784 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:26.779+0000 7f2618942640 1 --2- 172.21.15.123:0/3217953989 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f25f40072d0 0x7f2614157720 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:42:26.784 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:26.779+0000 7f2618942640 1 -- 172.21.15.123:0/3217953989 >> 172.21.15.123:0/3217953989 conn(0x7f26140741a0 msgr2=0x7f26140ad500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:42:26.784 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:26.779+0000 7f2618942640 1 -- 172.21.15.123:0/3217953989 shutdown_connections 2024-09-16T15:42:26.784 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:26.779+0000 7f2618942640 1 -- 172.21.15.123:0/3217953989 wait complete. 2024-09-16T15:42:26.785 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:26.779+0000 7f805dffb640 1 -- 172.21.15.123:0/3378124232 <== mon.0 v2:172.21.15.123:3300/0 11 ==== mon_command_ack([{prefix=config set, name=debug_client}]=0 v14)=0 v14) ==== 114+0+0 (secure 0 0 0) 0x7f8064019040 con 0x7f804c00ed80 2024-09-16T15:42:26.785 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:26.779+0000 7ff0d9ffb640 1 -- 172.21.15.123:0/3877245395 <== mon.0 v2:172.21.15.123:3300/0 11 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7ff0e00aed50 con 0x7ff0d00039c0 2024-09-16T15:42:26.785 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:26.779+0000 7fb3327fc640 1 -- 172.21.15.123:0/2661158985 <== mon.0 v2:172.21.15.123:3300/0 11 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fb3380ab900 con 0x7fb320006df0 2024-09-16T15:42:26.787 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:26.783+0000 7fb343b29640 1 -- 172.21.15.123:0/2661158985 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command([{prefix=config set, name=mds_verify_scatter}] v 0) -- 0x7fb318005300 con 0x7fb320006df0 2024-09-16T15:42:26.787 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:26.783+0000 7ff0eb4da640 1 -- 172.21.15.123:0/3877245395 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command([{prefix=config set, name=osd_op_complaint_time}] v 0) -- 0x7ff0c0005300 con 0x7ff0d00039c0 2024-09-16T15:42:26.788 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:26.783+0000 7f322a599640 1 -- 172.21.15.123:0/4092934445 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_command([{prefix=config set, name=mon_op_complaint_time}] v 0) -- 0x7f31fc005300 con 0x7f322415b360 2024-09-16T15:42:26.788 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:26.783+0000 7f805dffb640 1 -- 172.21.15.123:0/3378124232 <== mon.0 v2:172.21.15.123:3300/0 12 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f8064028c70 con 0x7f804c00ed80 2024-09-16T15:42:26.788 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:26.783+0000 7f0ce3ab6640 1 -- 172.21.15.123:0/3539876130 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f0cc80039c0 msgr2=0x7f0cdc0c3b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:42:26.788 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:26.783+0000 7f0ce3ab6640 1 --2- 172.21.15.123:0/3539876130 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f0cc80039c0 0x7f0cdc0c3b50 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f0cd8005fd0 tx=0x7f0cd801ff60 comp rx=0 tx=0).stop 2024-09-16T15:42:26.789 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:26.783+0000 7f3220ff9640 1 -- 172.21.15.123:0/4092934445 <== mon.2 v2:172.21.15.184:3301/0 11 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f321c0118f0 con 0x7f322415b360 2024-09-16T15:42:26.790 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:26.783+0000 7f806f29e640 1 -- 172.21.15.123:0/3378124232 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f804c00ed80 msgr2=0x7f806815eb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:42:26.790 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:26.783+0000 7f806f29e640 1 --2- 172.21.15.123:0/3378124232 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f804c00ed80 0x7f806815eb70 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f806401fa80 tx=0x7f806401ff40 comp rx=0 tx=0).stop 2024-09-16T15:42:26.790 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:26.783+0000 7f0ce3ab6640 1 -- 172.21.15.123:0/3539876130 shutdown_connections 2024-09-16T15:42:26.792 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:26.783+0000 7f806f29e640 1 -- 172.21.15.123:0/3378124232 shutdown_connections 2024-09-16T15:42:26.792 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:26.783+0000 7f806f29e640 1 --2- 172.21.15.123:0/3378124232 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f806814ee10 0x7f80680b4130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:42:26.792 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:26.783+0000 7f806f29e640 1 --2- 172.21.15.123:0/3378124232 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f806815a670 0x7f80680b6500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:42:26.792 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:26.783+0000 7f806f29e640 1 --2- 172.21.15.123:0/3378124232 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f804c00ed80 0x7f806815eb70 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:42:26.792 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:26.783+0000 7f806f29e640 1 -- 172.21.15.123:0/3378124232 >> 172.21.15.123:0/3378124232 conn(0x7f80680d0bf0 msgr2=0x7f806806d950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:42:26.792 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:26.783+0000 7f0ce3ab6640 1 --2- 172.21.15.123:0/3539876130 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f0cdc06d3d0 0x7f0cdc0c46f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:42:26.793 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:26.783+0000 7f55b2ffd640 1 -- 172.21.15.123:0/3913268890 <== mon.0 v2:172.21.15.123:3300/0 8 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f55b801dbf0 con 0x7f55c40cb1e0 2024-09-16T15:42:26.793 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:26.783+0000 7f0ce3ab6640 1 --2- 172.21.15.123:0/3539876130 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0cc8004810 0x7f0cdc0c8280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:42:26.794 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:26.783+0000 7f806f29e640 1 -- 172.21.15.123:0/3378124232 shutdown_connections 2024-09-16T15:42:26.795 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:26.783+0000 7f0ce3ab6640 1 --2- 172.21.15.123:0/3539876130 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f0cc80039c0 0x7f0cdc0c3b50 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:42:26.795 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:26.783+0000 7f0ce3ab6640 1 -- 172.21.15.123:0/3539876130 >> 172.21.15.123:0/3539876130 conn(0x7f0cdc0d0bc0 msgr2=0x7f0cdc0d0fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:42:26.795 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:26.783+0000 7f0ce3ab6640 1 -- 172.21.15.123:0/3539876130 shutdown_connections 2024-09-16T15:42:26.796 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:26.783+0000 7f0ce3ab6640 1 -- 172.21.15.123:0/3539876130 wait complete. 2024-09-16T15:42:26.796 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:26.787+0000 7fb3327fc640 1 -- 172.21.15.123:0/2661158985 <== mon.0 v2:172.21.15.123:3300/0 12 ==== mon_command_ack([{prefix=config set, name=mds_verify_scatter}]=0 v15)=0 v15) ==== 119+0+0 (secure 0 0 0) 0x7fb3380abce0 con 0x7fb320006df0 2024-09-16T15:42:26.796 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:26.787+0000 7fb343b29640 1 -- 172.21.15.123:0/2661158985 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb320006df0 msgr2=0x7fb33c09d250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:42:26.796 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:26.787+0000 7fb343b29640 1 --2- 172.21.15.123:0/2661158985 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb320006df0 0x7fb33c09d250 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fb338027580 tx=0x7fb328005840 comp rx=0 tx=0).stop 2024-09-16T15:42:26.796 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:26.791+0000 7f806f29e640 1 -- 172.21.15.123:0/3378124232 wait complete. 2024-09-16T15:42:26.797 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:26.791+0000 7ff0d9ffb640 1 -- 172.21.15.123:0/3877245395 <== mon.0 v2:172.21.15.123:3300/0 12 ==== mon_command_ack([{prefix=config set, name=osd_op_complaint_time}]=0 v16)=0 v16) ==== 121+0+0 (secure 0 0 0) 0x7ff0e00cc070 con 0x7ff0d00039c0 2024-09-16T15:42:26.799 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:26.795+0000 7ff0eb4da640 1 -- 172.21.15.123:0/3877245395 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff0d00039c0 msgr2=0x7ff0e41e52f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:42:26.799 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:26.795+0000 7ff0eb4da640 1 --2- 172.21.15.123:0/3877245395 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff0d00039c0 0x7ff0e41e52f0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7ff0e00b2000 tx=0x7ff0d0005840 comp rx=0 tx=0).stop 2024-09-16T15:42:26.799 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:26.795+0000 7ff0eb4da640 1 -- 172.21.15.123:0/3877245395 shutdown_connections 2024-09-16T15:42:26.799 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:26.795+0000 7ff0eb4da640 1 --2- 172.21.15.123:0/3877245395 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff0d00052a0 0x7ff0e41e3280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:42:26.799 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:26.795+0000 7ff0eb4da640 1 --2- 172.21.15.123:0/3877245395 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7ff0d0004810 0x7ff0e4146d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:42:26.799 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:26.795+0000 7ff0eb4da640 1 --2- 172.21.15.123:0/3877245395 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff0d00039c0 0x7ff0e41e52f0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:42:26.799 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:26.795+0000 7ff0eb4da640 1 -- 172.21.15.123:0/3877245395 >> 172.21.15.123:0/3877245395 conn(0x7ff0e40d0bf0 msgr2=0x7ff0e4150830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:42:26.799 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:26.795+0000 7fb343b29640 1 -- 172.21.15.123:0/2661158985 shutdown_connections 2024-09-16T15:42:26.800 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:26.795+0000 7fb343b29640 1 --2- 172.21.15.123:0/2661158985 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb328004d50 0x7fb33c0c5160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:42:26.800 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:26.795+0000 7fb343b29640 1 --2- 172.21.15.123:0/2661158985 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fb3280039c0 0x7fb33c1567a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:42:26.800 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:26.795+0000 7fb343b29640 1 --2- 172.21.15.123:0/2661158985 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb320006df0 0x7fb33c09d250 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:42:26.800 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:26.795+0000 7fb343b29640 1 -- 172.21.15.123:0/2661158985 >> 172.21.15.123:0/2661158985 conn(0x7fb33c0b1dc0 msgr2=0x7fb33c074910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:42:26.800 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:26.795+0000 7fb343b29640 1 -- 172.21.15.123:0/2661158985 shutdown_connections 2024-09-16T15:42:26.800 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:26.795+0000 7fb343b29640 1 -- 172.21.15.123:0/2661158985 wait complete. 2024-09-16T15:42:26.800 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:26.795+0000 7ff0eb4da640 1 -- 172.21.15.123:0/3877245395 shutdown_connections 2024-09-16T15:42:26.800 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:26.795+0000 7ff0eb4da640 1 -- 172.21.15.123:0/3877245395 wait complete. 2024-09-16T15:42:26.804 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:26.799+0000 7f3220ff9640 1 -- 172.21.15.123:0/4092934445 <== mon.2 v2:172.21.15.184:3301/0 12 ==== mon_command_ack([{prefix=config set, name=mon_op_complaint_time}]=0 v17)=0 v17) ==== 121+0+0 (secure 0 0 0) 0x7f321c011c70 con 0x7f322415b360 2024-09-16T15:42:26.806 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:26.803+0000 7f322a599640 1 -- 172.21.15.123:0/4092934445 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f322415b360 msgr2=0x7f322415b7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:42:26.807 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:26.803+0000 7f322a599640 1 --2- 172.21.15.123:0/4092934445 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f322415b360 0x7f322415b7e0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f32240b6de0 tx=0x7f321c012a50 comp rx=0 tx=0).stop 2024-09-16T15:42:26.807 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:26.803+0000 7f322a599640 1 -- 172.21.15.123:0/4092934445 shutdown_connections 2024-09-16T15:42:26.807 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:26.803+0000 7f322a599640 1 --2- 172.21.15.123:0/4092934445 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f322415b360 0x7f322415b7e0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:42:26.807 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:26.803+0000 7f322a599640 1 --2- 172.21.15.123:0/4092934445 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f32241598a0 0x7f32240b5190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:42:26.807 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:26.803+0000 7f322a599640 1 --2- 172.21.15.123:0/4092934445 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f320c0040a0 0x7f32241f8f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:42:26.807 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:26.803+0000 7f322a599640 1 -- 172.21.15.123:0/4092934445 >> 172.21.15.123:0/4092934445 conn(0x7f32240d0bf0 msgr2=0x7f32240c6bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:42:26.807 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:26.803+0000 7f322a599640 1 -- 172.21.15.123:0/4092934445 shutdown_connections 2024-09-16T15:42:26.807 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:26.803+0000 7f322a599640 1 -- 172.21.15.123:0/4092934445 wait complete. 2024-09-16T15:42:26.833 DEBUG:tasks.ceph:waiting for RemoteProcess(client=, args=['sudo', 'ceph', '--cluster', 'ceph', 'config', 'set', 'client', 'debug_ms', '1'], check_status=True, hostname='smithi123') 2024-09-16T15:42:26.833 DEBUG:tasks.ceph:waiting for RemoteProcess(client=, args=['sudo', 'ceph', '--cluster', 'ceph', 'config', 'set', 'client', 'rados_mon_op_timeout', '900'], check_status=True, hostname='smithi123') 2024-09-16T15:42:26.833 DEBUG:tasks.ceph:waiting for RemoteProcess(client=, args=['sudo', 'ceph', '--cluster', 'ceph', 'config', 'set', 'client', 'rados_osd_op_timeout', '900'], check_status=True, hostname='smithi123') 2024-09-16T15:42:26.833 DEBUG:tasks.ceph:waiting for RemoteProcess(client=, args=['sudo', 'ceph', '--cluster', 'ceph', 'config', 'set', 'mds', 'debug_mds', '20'], check_status=True, hostname='smithi123') 2024-09-16T15:42:26.850 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:26.847+0000 7f55c8f57640 1 -- 172.21.15.123:0/3913268890 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command([{prefix=config set, name=debug_mds}] v 0) -- 0x7f559c005300 con 0x7f55c40cb1e0 2024-09-16T15:42:26.852 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:26.847+0000 7f55b2ffd640 1 -- 172.21.15.123:0/3913268890 <== mon.0 v2:172.21.15.123:3300/0 9 ==== mon_command_ack([{prefix=config set, name=debug_mds}]=0 v18)=0 v18) ==== 108+0+0 (secure 0 0 0) 0x7f55b8027480 con 0x7f55c40cb1e0 2024-09-16T15:42:26.853 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:26.847+0000 7f55c8f57640 1 -- 172.21.15.123:0/3913268890 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f55c40cb1e0 msgr2=0x7f55c40cb620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:42:26.854 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:26.847+0000 7f55c8f57640 1 --2- 172.21.15.123:0/3913268890 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f55c40cb1e0 0x7f55c40cb620 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f55b8003d20 tx=0x7f55b8013820 comp rx=0 tx=0).stop 2024-09-16T15:42:26.854 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:26.847+0000 7f55c8f57640 1 -- 172.21.15.123:0/3913268890 shutdown_connections 2024-09-16T15:42:26.854 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:26.847+0000 7f55c8f57640 1 --2- 172.21.15.123:0/3913268890 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f55c4207e20 0x7f55c42082a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:42:26.854 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:26.847+0000 7f55c8f57640 1 --2- 172.21.15.123:0/3913268890 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f55c40cb1e0 0x7f55c40cb620 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:42:26.854 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:26.847+0000 7f55c8f57640 1 --2- 172.21.15.123:0/3913268890 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f55c4160bc0 0x7f55c40c7cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:42:26.854 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:26.847+0000 7f55c8f57640 1 -- 172.21.15.123:0/3913268890 >> 172.21.15.123:0/3913268890 conn(0x7f55c40d0bf0 msgr2=0x7f55c40b18d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:42:26.854 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:26.847+0000 7f55c8f57640 1 -- 172.21.15.123:0/3913268890 shutdown_connections 2024-09-16T15:42:26.854 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:26.847+0000 7f55c8f57640 1 -- 172.21.15.123:0/3913268890 wait complete. 2024-09-16T15:42:26.868 DEBUG:tasks.ceph:waiting for RemoteProcess(client=, args=['sudo', 'ceph', '--cluster', 'ceph', 'config', 'set', 'mds', 'debug_mds_balancer', '20'], check_status=True, hostname='smithi123') 2024-09-16T15:42:26.868 DEBUG:tasks.ceph:waiting for RemoteProcess(client=, args=['sudo', 'ceph', '--cluster', 'ceph', 'config', 'set', 'mds', 'debug_ms', '2'], check_status=True, hostname='smithi123') 2024-09-16T15:42:26.868 DEBUG:tasks.ceph:waiting for RemoteProcess(client=, args=['sudo', 'ceph', '--cluster', 'ceph', 'config', 'set', 'mds', 'mds_debug_frag', 'True'], check_status=True, hostname='smithi123') 2024-09-16T15:42:26.869 DEBUG:tasks.ceph:waiting for RemoteProcess(client=, args=['sudo', 'ceph', '--cluster', 'ceph', 'config', 'set', 'mds', 'mds_debug_scatterstat', 'True'], check_status=True, hostname='smithi123') 2024-09-16T15:42:26.869 DEBUG:tasks.ceph:waiting for RemoteProcess(client=, args=['sudo', 'ceph', '--cluster', 'ceph', 'config', 'set', 'mds', 'mds_op_complaint_time', '180'], check_status=True, hostname='smithi123') 2024-09-16T15:42:26.935 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:26.931+0000 7f5d3f247640 1 -- 172.21.15.123:0/2453823941 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5d1c00fa40 msgr2=0x7f5d1c00fe50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:42:26.935 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:26.931+0000 7f5d3f247640 1 --2- 172.21.15.123:0/2453823941 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5d1c00fa40 0x7f5d1c00fe50 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f5d1c0104f0 tx=0x7f5d2c0162c0 comp rx=0 tx=0).stop 2024-09-16T15:42:26.935 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:26.931+0000 7f5d3d7e4640 1 -- 172.21.15.123:0/2453823941 reap_dead start 2024-09-16T15:42:26.935 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:26.931+0000 7f5d3f247640 1 -- 172.21.15.123:0/2453823941 shutdown_connections 2024-09-16T15:42:26.935 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:26.931+0000 7f5d3f247640 1 --2- 172.21.15.123:0/2453823941 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5d1c00fa40 0x7f5d1c00fe50 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:42:26.935 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:26.931+0000 7f5d3f247640 1 -- 172.21.15.123:0/2453823941 >> 172.21.15.123:0/2453823941 conn(0x7f5d3809aa40 msgr2=0x7f5d3809ae50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:42:26.936 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:26.931+0000 7f5d3f247640 1 -- 172.21.15.123:0/2453823941 shutdown_connections 2024-09-16T15:42:26.936 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:26.931+0000 7f5d3f247640 1 -- 172.21.15.123:0/2453823941 wait complete. 2024-09-16T15:42:26.936 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:26.931+0000 7f5d3f247640 1 Processor -- start 2024-09-16T15:42:26.936 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:26.931+0000 7f5d3f247640 1 -- start start 2024-09-16T15:42:26.938 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:26.931+0000 7f5d3f247640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5d1c00fa40 0x7f5d38154320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:42:26.939 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:26.931+0000 7f5d3f247640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f5d380ca870 0x7f5d38152a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:42:26.939 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:26.931+0000 7f5d3f247640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5d380cacf0 0x7f5d38152f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:42:26.939 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:26.931+0000 7f5d3f247640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f5d380b21f0 con 0x7f5d380cacf0 2024-09-16T15:42:26.939 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:26.931+0000 7f5d3f247640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f5d380cf400 con 0x7f5d1c00fa40 2024-09-16T15:42:26.939 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:26.931+0000 7f5d3f247640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f5d380b2370 con 0x7f5d380ca870 2024-09-16T15:42:26.939 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:26.931+0000 7f5d3d7e4640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5d380cacf0 0x7f5d38152f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:42:26.939 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:26.931+0000 7f5d3d7e4640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5d380cacf0 0x7f5d38152f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:49720/0 (socket says 172.21.15.123:49720) 2024-09-16T15:42:26.939 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:26.931+0000 7f5d3d7e4640 1 -- 172.21.15.123:0/461948162 learned_addr learned my addr 172.21.15.123:0/461948162 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:42:26.939 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:26.931+0000 7f5d37fff640 1 --2- 172.21.15.123:0/461948162 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f5d380ca870 0x7f5d38152a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:42:26.939 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:26.931+0000 7f5d3cfe3640 1 --2- 172.21.15.123:0/461948162 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5d1c00fa40 0x7f5d38154320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:42:26.939 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:26.931+0000 7f5d3d7e4640 1 -- 172.21.15.123:0/461948162 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f5d380ca870 msgr2=0x7f5d38152a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:42:26.939 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:26.931+0000 7f5d3d7e4640 1 --2- 172.21.15.123:0/461948162 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f5d380ca870 0x7f5d38152a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:42:26.940 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:26.931+0000 7f5d3d7e4640 1 -- 172.21.15.123:0/461948162 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5d1c00fa40 msgr2=0x7f5d38154320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:42:26.940 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:26.931+0000 7f5d3d7e4640 1 --2- 172.21.15.123:0/461948162 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5d1c00fa40 0x7f5d38154320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:42:26.940 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:26.931+0000 7f5d3d7e4640 1 -- 172.21.15.123:0/461948162 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5d380b5f40 con 0x7f5d380cacf0 2024-09-16T15:42:26.940 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:26.931+0000 7f5d3d7e4640 1 --2- 172.21.15.123:0/461948162 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5d380cacf0 0x7f5d38152f50 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f5d1c006140 tx=0x7f5d3000c900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:42:26.940 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:26.931+0000 7f5d357fa640 1 -- 172.21.15.123:0/461948162 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d30003980 con 0x7f5d380cacf0 2024-09-16T15:42:26.940 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:26.935+0000 7f5d357fa640 1 -- 172.21.15.123:0/461948162 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f5d30003b20 con 0x7f5d380cacf0 2024-09-16T15:42:26.940 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:26.935+0000 7f5d3f247640 1 -- 172.21.15.123:0/461948162 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5d381fb680 con 0x7f5d380cacf0 2024-09-16T15:42:26.940 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:26.935+0000 7f5d357fa640 1 -- 172.21.15.123:0/461948162 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d30004020 con 0x7f5d380cacf0 2024-09-16T15:42:26.940 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:26.935+0000 7f5d3f247640 1 -- 172.21.15.123:0/461948162 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5d381fbc00 con 0x7f5d380cacf0 2024-09-16T15:42:26.940 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:26.935+0000 7f5d357fa640 1 -- 172.21.15.123:0/461948162 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f5d300041c0 con 0x7f5d380cacf0 2024-09-16T15:42:26.940 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:26.935+0000 7f5d357fa640 1 -- 172.21.15.123:0/461948162 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f5d30007880 con 0x7f5d380cacf0 2024-09-16T15:42:26.940 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:26.935+0000 7f5d3f247640 1 -- 172.21.15.123:0/461948162 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5d380b28d0 con 0x7f5d380cacf0 2024-09-16T15:42:26.941 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:26.935+0000 7f5d357fa640 1 -- 172.21.15.123:0/461948162 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+75856 (secure 0 0 0) 0x7f5d380b5f40 con 0x7f5d380cacf0 2024-09-16T15:42:26.996 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:26.991+0000 7f5d3f247640 1 -- 172.21.15.123:0/461948162 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command([{prefix=config set, name=mds_op_complaint_time}] v 0) -- 0x7f5d3806ed70 con 0x7f5d380cacf0 2024-09-16T15:42:27.000 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:26.995+0000 7f5d357fa640 1 -- 172.21.15.123:0/461948162 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mds_op_complaint_time}]=0 v19)=0 v19) ==== 121+0+0 (secure 0 0 0) 0x7f5d30004640 con 0x7f5d380cacf0 2024-09-16T15:42:27.001 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:26.995+0000 7f5d3f247640 1 -- 172.21.15.123:0/461948162 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5d380cacf0 msgr2=0x7f5d38152f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:42:27.001 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:26.995+0000 7f5d3f247640 1 --2- 172.21.15.123:0/461948162 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5d380cacf0 0x7f5d38152f50 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f5d1c006140 tx=0x7f5d3000c900 comp rx=0 tx=0).stop 2024-09-16T15:42:27.001 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:26.995+0000 7f5d3f247640 1 -- 172.21.15.123:0/461948162 shutdown_connections 2024-09-16T15:42:27.001 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:26.995+0000 7f5d3f247640 1 --2- 172.21.15.123:0/461948162 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5d380cacf0 0x7f5d38152f50 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:42:27.001 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:26.995+0000 7f5d3f247640 1 --2- 172.21.15.123:0/461948162 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f5d380ca870 0x7f5d38152a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:42:27.001 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:26.995+0000 7f5d3f247640 1 --2- 172.21.15.123:0/461948162 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5d1c00fa40 0x7f5d38154320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:42:27.001 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:26.995+0000 7f5d3f247640 1 -- 172.21.15.123:0/461948162 >> 172.21.15.123:0/461948162 conn(0x7f5d3809aa40 msgr2=0x7f5d3806dd00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:42:27.002 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:26.995+0000 7f5d3f247640 1 -- 172.21.15.123:0/461948162 shutdown_connections 2024-09-16T15:42:27.002 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:26.995+0000 7f5d3f247640 1 -- 172.21.15.123:0/461948162 wait complete. 2024-09-16T15:42:27.015 DEBUG:tasks.ceph:waiting for RemoteProcess(client=, args=['sudo', 'ceph', '--cluster', 'ceph', 'config', 'set', 'mds', 'mds_verify_scatter', 'True'], check_status=True, hostname='smithi123') 2024-09-16T15:42:27.015 DEBUG:tasks.ceph:waiting for RemoteProcess(client=, args=['sudo', 'ceph', '--cluster', 'ceph', 'config', 'set', 'mds', 'osd_op_complaint_time', '180'], check_status=True, hostname='smithi123') 2024-09-16T15:42:27.016 DEBUG:tasks.ceph:waiting for RemoteProcess(client=, args=['sudo', 'ceph', '--cluster', 'ceph', 'config', 'set', 'mds', 'rados_mon_op_timeout', '900'], check_status=True, hostname='smithi123') 2024-09-16T15:42:27.016 DEBUG:tasks.ceph:waiting for RemoteProcess(client=, args=['sudo', 'ceph', '--cluster', 'ceph', 'config', 'set', 'mds', 'rados_osd_op_timeout', '900'], check_status=True, hostname='smithi123') 2024-09-16T15:42:27.016 DEBUG:tasks.ceph:waiting for RemoteProcess(client=, args=['sudo', 'ceph', '--cluster', 'ceph', 'config', 'set', 'mon', 'mon_op_complaint_time', '120'], check_status=True, hostname='smithi123') 2024-09-16T15:42:27.016 DEBUG:tasks.ceph:waiting for RemoteProcess(client=, args=['sudo', 'ceph', '--cluster', 'ceph', 'config', 'set', 'mon', 'mon_warn_on_pool_no_app', 'False'], check_status=True, hostname='smithi123') 2024-09-16T15:42:27.016 DEBUG:tasks.ceph:waiting for RemoteProcess(client=, args=['sudo', 'ceph', '--cluster', 'ceph', 'config', 'set', 'osd', 'osd_op_complaint_time', '180'], check_status=True, hostname='smithi123') 2024-09-16T15:42:27.016 INFO:tasks.ceph:Setting crush tunables to default 2024-09-16T15:42:27.017 DEBUG:teuthology.orchestra.run.smithi123:> sudo ceph --cluster ceph osd crush tunables default 2024-09-16T15:42:27.138 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:27.131+0000 7f3269a72640 1 -- 172.21.15.123:0/515459513 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f32500052a0 msgr2=0x7f325000b9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:42:27.138 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:27.131+0000 7f3269a72640 1 --2- 172.21.15.123:0/515459513 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f32500052a0 0x7f325000b9d0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f325c0043e0 tx=0x7f325c01ca60 comp rx=0 tx=0).stop 2024-09-16T15:42:27.138 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:27.131+0000 7f3269a72640 1 -- 172.21.15.123:0/515459513 shutdown_connections 2024-09-16T15:42:27.138 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:27.131+0000 7f3269a72640 1 --2- 172.21.15.123:0/515459513 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f32500052a0 0x7f325000b9d0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:42:27.138 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:27.131+0000 7f3269a72640 1 -- 172.21.15.123:0/515459513 >> 172.21.15.123:0/515459513 conn(0x7f326409aa40 msgr2=0x7f326409ae50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:42:27.138 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:27.131+0000 7f3269a72640 1 -- 172.21.15.123:0/515459513 shutdown_connections 2024-09-16T15:42:27.138 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:27.131+0000 7f3269a72640 1 -- 172.21.15.123:0/515459513 wait complete. 2024-09-16T15:42:27.139 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:27.135+0000 7f3269a72640 1 Processor -- start 2024-09-16T15:42:27.139 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:27.135+0000 7f3269a72640 1 -- start start 2024-09-16T15:42:27.141 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:27.135+0000 7f3269a72640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f32500052a0 0x7f3264159190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:42:27.141 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:27.135+0000 7f3269a72640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f326406dd00 0x7f326415c2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:42:27.141 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:27.135+0000 7f3269a72640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f32640b6a60 0x7f32640b6ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:42:27.141 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:27.135+0000 7f3269a72640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f32640ceb20 con 0x7f32500052a0 2024-09-16T15:42:27.141 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:27.135+0000 7f3269a72640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f32640ce9a0 con 0x7f32640b6a60 2024-09-16T15:42:27.141 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:27.135+0000 7f3269a72640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f32640d2040 con 0x7f326406dd00 2024-09-16T15:42:27.141 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:27.135+0000 7f32637fe640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f32500052a0 0x7f3264159190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:42:27.141 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:27.135+0000 7f32637fe640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f32500052a0 0x7f3264159190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:49748/0 (socket says 172.21.15.123:49748) 2024-09-16T15:42:27.142 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:27.135+0000 7f32637fe640 1 -- 172.21.15.123:0/442520994 learned_addr learned my addr 172.21.15.123:0/442520994 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:42:27.142 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:27.135+0000 7f3262ffd640 1 --2- 172.21.15.123:0/442520994 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f326406dd00 0x7f326415c2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:42:27.142 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:27.135+0000 7f3263fff640 1 --2- 172.21.15.123:0/442520994 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f32640b6a60 0x7f32640b6ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:42:27.142 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:27.135+0000 7f32637fe640 1 -- 172.21.15.123:0/442520994 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f326406dd00 msgr2=0x7f326415c2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:42:27.142 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:27.135+0000 7f32637fe640 1 --2- 172.21.15.123:0/442520994 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f326406dd00 0x7f326415c2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:42:27.142 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:27.135+0000 7f32637fe640 1 -- 172.21.15.123:0/442520994 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f32640b6a60 msgr2=0x7f32640b6ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:42:27.142 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:27.135+0000 7f32637fe640 1 --2- 172.21.15.123:0/442520994 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f32640b6a60 0x7f32640b6ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:42:27.142 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:27.135+0000 7f32637fe640 1 -- 172.21.15.123:0/442520994 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f32640c5500 con 0x7f32500052a0 2024-09-16T15:42:27.142 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:27.135+0000 7f32637fe640 1 --2- 172.21.15.123:0/442520994 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f32500052a0 0x7f3264159190 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f32540016d0 tx=0x7f325400e6b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:42:27.142 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:27.135+0000 7f325bfff640 1 -- 172.21.15.123:0/442520994 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3254012850 con 0x7f32500052a0 2024-09-16T15:42:27.142 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:27.135+0000 7f325bfff640 1 -- 172.21.15.123:0/442520994 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f3254012ae0 con 0x7f32500052a0 2024-09-16T15:42:27.142 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:27.135+0000 7f3269a72640 1 -- 172.21.15.123:0/442520994 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f32641f4af0 con 0x7f32500052a0 2024-09-16T15:42:27.142 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:27.135+0000 7f325bfff640 1 -- 172.21.15.123:0/442520994 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f325401a400 con 0x7f32500052a0 2024-09-16T15:42:27.143 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:27.135+0000 7f3269a72640 1 -- 172.21.15.123:0/442520994 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f32641f50b0 con 0x7f32500052a0 2024-09-16T15:42:27.143 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:27.135+0000 7f325bfff640 1 -- 172.21.15.123:0/442520994 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f325400d030 con 0x7f32500052a0 2024-09-16T15:42:27.143 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:27.135+0000 7f325bfff640 1 -- 172.21.15.123:0/442520994 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f32540238a0 con 0x7f32500052a0 2024-09-16T15:42:27.143 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:27.135+0000 7f3269a72640 1 -- 172.21.15.123:0/442520994 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f32640b7420 con 0x7f32500052a0 2024-09-16T15:42:27.143 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:27.139+0000 7f325bfff640 1 -- 172.21.15.123:0/442520994 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+75856 (secure 0 0 0) 0x7f3254028080 con 0x7f32500052a0 2024-09-16T15:42:27.203 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:27.199+0000 7f3269a72640 1 -- 172.21.15.123:0/442520994 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) -- 0x7f32640ceca0 con 0x7f32500052a0 2024-09-16T15:42:27.206 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:27.203+0000 7f325bfff640 1 -- 172.21.15.123:0/442520994 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v2) ==== 124+0+0 (secure 0 0 0) 0x7f3254016030 con 0x7f32500052a0 2024-09-16T15:42:27.206 INFO:teuthology.orchestra.run.smithi123.stderr:adjusted tunables profile to default 2024-09-16T15:42:27.207 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:27.203+0000 7f3269a72640 1 -- 172.21.15.123:0/442520994 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f32500052a0 msgr2=0x7f3264159190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:42:27.207 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:27.203+0000 7f3269a72640 1 --2- 172.21.15.123:0/442520994 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f32500052a0 0x7f3264159190 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f32540016d0 tx=0x7f325400e6b0 comp rx=0 tx=0).stop 2024-09-16T15:42:27.207 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:27.203+0000 7f3269a72640 1 -- 172.21.15.123:0/442520994 shutdown_connections 2024-09-16T15:42:27.207 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:27.203+0000 7f3269a72640 1 --2- 172.21.15.123:0/442520994 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f32640b6a60 0x7f32640b6ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:42:27.208 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:27.203+0000 7f3269a72640 1 --2- 172.21.15.123:0/442520994 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f326406dd00 0x7f326415c2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:42:27.208 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:27.203+0000 7f3269a72640 1 --2- 172.21.15.123:0/442520994 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f32500052a0 0x7f3264159190 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:42:27.208 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:27.203+0000 7f3269a72640 1 -- 172.21.15.123:0/442520994 >> 172.21.15.123:0/442520994 conn(0x7f326409aa40 msgr2=0x7f32640af300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:42:27.208 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:27.203+0000 7f3269a72640 1 -- 172.21.15.123:0/442520994 shutdown_connections 2024-09-16T15:42:27.208 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:27.203+0000 7f3269a72640 1 -- 172.21.15.123:0/442520994 wait complete. 2024-09-16T15:42:27.223 INFO:tasks.ceph:check_enable_crimson: False 2024-09-16T15:42:27.223 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2024-09-16T15:42:27.223 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2024-09-16T15:42:27.223 DEBUG:teuthology.orchestra.run.smithi123:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2024-09-16T15:42:27.233 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2024-09-16T15:42:27.233 DEBUG:teuthology.orchestra.run.smithi123:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2024-09-16T15:42:27.294 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2024-09-16T15:42:27.294 DEBUG:teuthology.orchestra.run.smithi123:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2024-09-16T15:42:27.348 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2024-09-16T15:42:27.348 DEBUG:teuthology.orchestra.run.smithi123:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2024-09-16T15:42:27.401 DEBUG:teuthology.orchestra.run.smithi184:> set -ex 2024-09-16T15:42:27.401 DEBUG:teuthology.orchestra.run.smithi184:> sudo dd if=/var/lib/ceph/osd/ceph-4/fsid of=/dev/stdout 2024-09-16T15:42:27.413 DEBUG:teuthology.orchestra.run.smithi184:> set -ex 2024-09-16T15:42:27.413 DEBUG:teuthology.orchestra.run.smithi184:> sudo dd if=/var/lib/ceph/osd/ceph-5/fsid of=/dev/stdout 2024-09-16T15:42:27.465 DEBUG:teuthology.orchestra.run.smithi184:> set -ex 2024-09-16T15:42:27.465 DEBUG:teuthology.orchestra.run.smithi184:> sudo dd if=/var/lib/ceph/osd/ceph-6/fsid of=/dev/stdout 2024-09-16T15:42:27.516 DEBUG:teuthology.orchestra.run.smithi184:> set -ex 2024-09-16T15:42:27.516 DEBUG:teuthology.orchestra.run.smithi184:> sudo dd if=/var/lib/ceph/osd/ceph-7/fsid of=/dev/stdout 2024-09-16T15:42:27.569 DEBUG:teuthology.orchestra.run.smithi184:> sudo ceph --cluster ceph osd new 9e7c1955-a607-4671-9be7-f2ef760ceeaa 0 2024-09-16T15:42:27.734 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:42:27.728+0000 7f824c7c0640 1 -- 172.21.15.184:0/2816087162 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f82440b31a0 msgr2=0x7f82440d1b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:42:27.734 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:42:27.728+0000 7f824c7c0640 1 --2- 172.21.15.184:0/2816087162 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f82440b31a0 0x7f82440d1b20 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f8234009900 tx=0x7f823401f090 comp rx=0 tx=0).stop 2024-09-16T15:42:27.734 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:42:27.728+0000 7f824c7c0640 1 -- 172.21.15.184:0/2816087162 shutdown_connections 2024-09-16T15:42:27.734 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:42:27.728+0000 7f824c7c0640 1 --2- 172.21.15.184:0/2816087162 >> v2:172.21.15.123:3300/0 conn(0x7f82440d2060 0x7f824415f770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:42:27.734 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:42:27.728+0000 7f824c7c0640 1 --2- 172.21.15.184:0/2816087162 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f82440b31a0 0x7f82440d1b20 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:42:27.734 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:42:27.728+0000 7f824c7c0640 1 --2- 172.21.15.184:0/2816087162 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f824415bd60 0x7f82440cf850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:42:27.734 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:42:27.728+0000 7f824c7c0640 1 -- 172.21.15.184:0/2816087162 >> 172.21.15.184:0/2816087162 conn(0x7f824409aa40 msgr2=0x7f824409ae50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:42:27.734 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:42:27.728+0000 7f824c7c0640 1 -- 172.21.15.184:0/2816087162 shutdown_connections 2024-09-16T15:42:27.734 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:42:27.728+0000 7f824c7c0640 1 -- 172.21.15.184:0/2816087162 wait complete. 2024-09-16T15:42:27.735 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:42:27.732+0000 7f824c7c0640 1 Processor -- start 2024-09-16T15:42:27.736 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:42:27.732+0000 7f824c7c0640 1 -- start start 2024-09-16T15:42:27.738 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:42:27.732+0000 7f824c7c0640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f82440b31a0 0x7f82441f2570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:42:27.739 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:42:27.732+0000 7f824c7c0640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f82440d2060 0x7f82440b5790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:42:27.739 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:42:27.732+0000 7f824c7c0640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f824415bd60 0x7f82440b0e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:42:27.739 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:42:27.732+0000 7f824c7c0640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f8244160040 con 0x7f824415bd60 2024-09-16T15:42:27.739 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:42:27.732+0000 7f824c7c0640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f824415fec0 con 0x7f82440d2060 2024-09-16T15:42:27.739 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:42:27.732+0000 7f824c7c0640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f82441601c0 con 0x7f82440b31a0 2024-09-16T15:42:27.739 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:42:27.732+0000 7f824a55c640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f82440b31a0 0x7f82441f2570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:42:27.739 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:42:27.732+0000 7f8249d5b640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f82440d2060 0x7f82440b5790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:42:27.739 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:42:27.732+0000 7f824a55c640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f82440b31a0 0x7f82441f2570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3301/0 says I am v2:172.21.15.184:40636/0 (socket says 172.21.15.184:40636) 2024-09-16T15:42:27.739 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:42:27.732+0000 7f8249d5b640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f82440d2060 0x7f82440b5790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.184:53932/0 (socket says 172.21.15.184:53932) 2024-09-16T15:42:27.739 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:42:27.732+0000 7f824a55c640 1 -- 172.21.15.184:0/3996673423 learned_addr learned my addr 172.21.15.184:0/3996673423 (peer_addr_for_me v2:172.21.15.184:0/0) 2024-09-16T15:42:27.739 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:42:27.732+0000 7f824ad5d640 1 --2- 172.21.15.184:0/3996673423 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f824415bd60 0x7f82440b0e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:42:27.739 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:42:27.732+0000 7f824a55c640 1 -- 172.21.15.184:0/3996673423 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f82440d2060 msgr2=0x7f82440b5790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:42:27.740 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:42:27.732+0000 7f824a55c640 1 --2- 172.21.15.184:0/3996673423 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f82440d2060 0x7f82440b5790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:42:27.740 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:42:27.732+0000 7f824a55c640 1 -- 172.21.15.184:0/3996673423 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f824415bd60 msgr2=0x7f82440b0e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:42:27.740 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:42:27.732+0000 7f824a55c640 1 --2- 172.21.15.184:0/3996673423 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f824415bd60 0x7f82440b0e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:42:27.740 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:42:27.732+0000 7f824a55c640 1 -- 172.21.15.184:0/3996673423 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f824415c270 con 0x7f82440b31a0 2024-09-16T15:42:27.740 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:42:27.732+0000 7f824ad5d640 1 --2- 172.21.15.184:0/3996673423 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f824415bd60 0x7f82440b0e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T15:42:27.740 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:42:27.732+0000 7f8249d5b640 1 --2- 172.21.15.184:0/3996673423 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f82440d2060 0x7f82440b5790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T15:42:27.740 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:42:27.732+0000 7f824a55c640 1 --2- 172.21.15.184:0/3996673423 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f82440b31a0 0x7f82441f2570 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f824000ed00 tx=0x7f824000c6a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:42:27.740 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:42:27.732+0000 7f823affd640 1 -- 172.21.15.184:0/3996673423 <== mon.2 v2:172.21.15.184:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8240003b80 con 0x7f82440b31a0 2024-09-16T15:42:27.740 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:42:27.732+0000 7f823affd640 1 -- 172.21.15.184:0/3996673423 <== mon.2 v2:172.21.15.184:3301/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f8240003d20 con 0x7f82440b31a0 2024-09-16T15:42:27.740 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:42:27.732+0000 7f824c7c0640 1 -- 172.21.15.184:0/3996673423 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f82441f2ba0 con 0x7f82440b31a0 2024-09-16T15:42:27.740 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:42:27.732+0000 7f823affd640 1 -- 172.21.15.184:0/3996673423 <== mon.2 v2:172.21.15.184:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8240004280 con 0x7f82440b31a0 2024-09-16T15:42:27.740 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:42:27.732+0000 7f824c7c0640 1 -- 172.21.15.184:0/3996673423 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f82441f3160 con 0x7f82440b31a0 2024-09-16T15:42:27.741 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:42:27.736+0000 7f823affd640 1 -- 172.21.15.184:0/3996673423 <== mon.2 v2:172.21.15.184:3301/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f8240004420 con 0x7f82440b31a0 2024-09-16T15:42:27.741 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:42:27.736+0000 7f823affd640 1 -- 172.21.15.184:0/3996673423 <== mon.2 v2:172.21.15.184:3301/0 5 ==== osd_map(2..2 src has 1..2) ==== 725+0+0 (secure 0 0 0) 0x7f8240018430 con 0x7f82440b31a0 2024-09-16T15:42:27.741 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:42:27.736+0000 7f824c7c0640 1 -- 172.21.15.184:0/3996673423 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f824415f770 con 0x7f82440b31a0 2024-09-16T15:42:27.741 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:42:27.736+0000 7f823affd640 1 -- 172.21.15.184:0/3996673423 <== mon.2 v2:172.21.15.184:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+75856 (secure 0 0 0) 0x7f8240009820 con 0x7f82440b31a0 2024-09-16T15:42:27.812 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:42:27.808+0000 7f824c7c0640 1 -- 172.21.15.184:0/3996673423 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_command({"prefix": "osd new", "uuid": "9e7c1955-a607-4671-9be7-f2ef760ceeaa", "id": 0} v 0) -- 0x7f82440b2d30 con 0x7f82440b31a0 2024-09-16T15:42:27.816 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:42:27.812+0000 7f823affd640 1 -- 172.21.15.184:0/3996673423 <== mon.2 v2:172.21.15.184:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "9e7c1955-a607-4671-9be7-f2ef760ceeaa", "id": 0}]=0 v3) ==== 112+0+2 (secure 0 0 0) 0x7f8240004ee0 con 0x7f82440b31a0 2024-09-16T15:42:27.817 INFO:teuthology.orchestra.run.smithi184.stdout:0 2024-09-16T15:42:27.818 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:42:27.812+0000 7f824c7c0640 1 -- 172.21.15.184:0/3996673423 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f82440b31a0 msgr2=0x7f82441f2570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:42:27.818 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:42:27.812+0000 7f824c7c0640 1 --2- 172.21.15.184:0/3996673423 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f82440b31a0 0x7f82441f2570 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f824000ed00 tx=0x7f824000c6a0 comp rx=0 tx=0).stop 2024-09-16T15:42:27.818 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:42:27.812+0000 7f824c7c0640 1 -- 172.21.15.184:0/3996673423 shutdown_connections 2024-09-16T15:42:27.818 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:42:27.812+0000 7f824c7c0640 1 --2- 172.21.15.184:0/3996673423 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f824415bd60 0x7f82440b0e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:42:27.818 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:42:27.812+0000 7f824c7c0640 1 --2- 172.21.15.184:0/3996673423 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f82440d2060 0x7f82440b5790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:42:27.818 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:42:27.812+0000 7f824c7c0640 1 --2- 172.21.15.184:0/3996673423 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f82440b31a0 0x7f82441f2570 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:42:27.818 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:42:27.812+0000 7f824c7c0640 1 -- 172.21.15.184:0/3996673423 >> 172.21.15.184:0/3996673423 conn(0x7f824409aa40 msgr2=0x7f82440cec60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:42:27.818 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:42:27.812+0000 7f824c7c0640 1 -- 172.21.15.184:0/3996673423 shutdown_connections 2024-09-16T15:42:27.818 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:42:27.812+0000 7f824c7c0640 1 -- 172.21.15.184:0/3996673423 wait complete. 2024-09-16T15:42:27.833 DEBUG:teuthology.orchestra.run.smithi184:> sudo ceph --cluster ceph osd new c540cfde-0977-4742-9b17-fd634668e010 1 2024-09-16T15:42:27.952 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:42:27.948+0000 7f6a1acd8640 1 -- 172.21.15.184:0/3100782798 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f6a1415e620 msgr2=0x7f6a1406dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:42:27.952 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:42:27.948+0000 7f6a1acd8640 1 --2- 172.21.15.184:0/3100782798 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f6a1415e620 0x7f6a1406dd00 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f6a00009840 tx=0x7f6a0001c760 comp rx=0 tx=0).stop 2024-09-16T15:42:27.953 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:42:27.948+0000 7f6a1acd8640 1 -- 172.21.15.184:0/3100782798 shutdown_connections 2024-09-16T15:42:27.953 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:42:27.948+0000 7f6a1acd8640 1 --2- 172.21.15.184:0/3100782798 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f6a1415e620 0x7f6a1406dd00 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:42:27.953 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:42:27.948+0000 7f6a1acd8640 1 --2- 172.21.15.184:0/3100782798 >> v2:172.21.15.123:3300/0 conn(0x7f6a1415de80 0x7f6a1406f530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:42:27.953 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:42:27.948+0000 7f6a1acd8640 1 -- 172.21.15.184:0/3100782798 >> 172.21.15.184:0/3100782798 conn(0x7f6a140b28f0 msgr2=0x7f6a140b2d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:42:27.953 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:42:27.948+0000 7f6a1acd8640 1 -- 172.21.15.184:0/3100782798 shutdown_connections 2024-09-16T15:42:27.953 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:42:27.948+0000 7f6a1acd8640 1 -- 172.21.15.184:0/3100782798 wait complete. 2024-09-16T15:42:27.953 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:42:27.948+0000 7f6a1acd8640 1 Processor -- start 2024-09-16T15:42:27.954 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:42:27.948+0000 7f6a1acd8640 1 -- start start 2024-09-16T15:42:27.956 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:42:27.952+0000 7f6a1acd8640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f6a1406e240 0x7f6a1414a1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:42:27.956 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:42:27.952+0000 7f6a1acd8640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6a1415de80 0x7f6a1414be40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:42:27.956 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:42:27.952+0000 7f6a1acd8640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6a1415e620 0x7f6a14164aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:42:27.956 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:42:27.952+0000 7f6a1acd8640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f6a14167db0 con 0x7f6a1415e620 2024-09-16T15:42:27.956 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:42:27.952+0000 7f6a1acd8640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f6a14167c30 con 0x7f6a1415de80 2024-09-16T15:42:27.956 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:42:27.952+0000 7f6a1acd8640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f6a14167f30 con 0x7f6a1406e240 2024-09-16T15:42:27.956 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:42:27.952+0000 7f6a18a74640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f6a1406e240 0x7f6a1414a1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:42:27.956 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:42:27.952+0000 7f6a18a74640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f6a1406e240 0x7f6a1414a1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3301/0 says I am v2:172.21.15.184:40660/0 (socket says 172.21.15.184:40660) 2024-09-16T15:42:27.956 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:42:27.952+0000 7f6a18a74640 1 -- 172.21.15.184:0/1253171415 learned_addr learned my addr 172.21.15.184:0/1253171415 (peer_addr_for_me v2:172.21.15.184:0/0) 2024-09-16T15:42:27.956 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:42:27.952+0000 7f6a13fff640 1 --2- 172.21.15.184:0/1253171415 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6a1415de80 0x7f6a1414be40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:42:27.956 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:42:27.952+0000 7f6a18a74640 1 -- 172.21.15.184:0/1253171415 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6a1415de80 msgr2=0x7f6a1414be40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:42:27.957 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:42:27.952+0000 7f6a19275640 1 --2- 172.21.15.184:0/1253171415 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6a1415e620 0x7f6a14164aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:42:27.957 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:42:27.952+0000 7f6a18a74640 1 --2- 172.21.15.184:0/1253171415 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6a1415de80 0x7f6a1414be40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:42:27.957 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:42:27.952+0000 7f6a18a74640 1 -- 172.21.15.184:0/1253171415 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6a1415e620 msgr2=0x7f6a14164aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:42:27.957 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:42:27.952+0000 7f6a18a74640 1 --2- 172.21.15.184:0/1253171415 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6a1415e620 0x7f6a14164aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:42:27.957 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:42:27.952+0000 7f6a18a74640 1 -- 172.21.15.184:0/1253171415 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6a1414b080 con 0x7f6a1406e240 2024-09-16T15:42:27.957 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:42:27.952+0000 7f6a19275640 1 --2- 172.21.15.184:0/1253171415 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6a1415e620 0x7f6a14164aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T15:42:27.957 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:42:27.952+0000 7f6a18a74640 1 --2- 172.21.15.184:0/1253171415 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f6a1406e240 0x7f6a1414a1d0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f6a0800d980 tx=0x7f6a0800de40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:42:27.957 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:42:27.952+0000 7f6a117fa640 1 -- 172.21.15.184:0/1253171415 <== mon.2 v2:172.21.15.184:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a08017070 con 0x7f6a1406e240 2024-09-16T15:42:27.957 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:42:27.952+0000 7f6a117fa640 1 -- 172.21.15.184:0/1253171415 <== mon.2 v2:172.21.15.184:3301/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f6a08009d20 con 0x7f6a1406e240 2024-09-16T15:42:27.957 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:42:27.952+0000 7f6a1acd8640 1 -- 172.21.15.184:0/1253171415 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6a1414b360 con 0x7f6a1406e240 2024-09-16T15:42:27.957 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:42:27.952+0000 7f6a117fa640 1 -- 172.21.15.184:0/1253171415 <== mon.2 v2:172.21.15.184:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a080106c0 con 0x7f6a1406e240 2024-09-16T15:42:27.957 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:42:27.952+0000 7f6a1acd8640 1 -- 172.21.15.184:0/1253171415 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f6a14164460 con 0x7f6a1406e240 2024-09-16T15:42:27.958 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:42:27.952+0000 7f6a117fa640 1 -- 172.21.15.184:0/1253171415 <== mon.2 v2:172.21.15.184:3301/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f6a08010860 con 0x7f6a1406e240 2024-09-16T15:42:27.958 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:42:27.952+0000 7f6a117fa640 1 -- 172.21.15.184:0/1253171415 <== mon.2 v2:172.21.15.184:3301/0 5 ==== osd_map(3..3 src has 1..3) ==== 840+0+0 (secure 0 0 0) 0x7f6a0800f3c0 con 0x7f6a1406e240 2024-09-16T15:42:27.958 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:42:27.952+0000 7f6a1acd8640 1 -- 172.21.15.184:0/1253171415 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6a1406dd00 con 0x7f6a1406e240 2024-09-16T15:42:27.958 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:42:27.952+0000 7f6a117fa640 1 -- 172.21.15.184:0/1253171415 <== mon.2 v2:172.21.15.184:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+75856 (secure 0 0 0) 0x7f6a08010d30 con 0x7f6a1406e240 2024-09-16T15:42:28.029 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:42:28.024+0000 7f6a1acd8640 1 -- 172.21.15.184:0/1253171415 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_command({"prefix": "osd new", "uuid": "c540cfde-0977-4742-9b17-fd634668e010", "id": 1} v 0) -- 0x7f6a140b0910 con 0x7f6a1406e240 2024-09-16T15:42:28.035 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:42:28.032+0000 7f6a117fa640 1 -- 172.21.15.184:0/1253171415 <== mon.2 v2:172.21.15.184:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "c540cfde-0977-4742-9b17-fd634668e010", "id": 1}]=0 v4) ==== 112+0+2 (secure 0 0 0) 0x7f6a0801b400 con 0x7f6a1406e240 2024-09-16T15:42:28.035 INFO:teuthology.orchestra.run.smithi184.stdout:1 2024-09-16T15:42:28.037 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:42:28.032+0000 7f6a1acd8640 1 -- 172.21.15.184:0/1253171415 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f6a1406e240 msgr2=0x7f6a1414a1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:42:28.037 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:42:28.032+0000 7f6a1acd8640 1 --2- 172.21.15.184:0/1253171415 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f6a1406e240 0x7f6a1414a1d0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f6a0800d980 tx=0x7f6a0800de40 comp rx=0 tx=0).stop 2024-09-16T15:42:28.037 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:42:28.032+0000 7f6a1acd8640 1 -- 172.21.15.184:0/1253171415 shutdown_connections 2024-09-16T15:42:28.037 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:42:28.032+0000 7f6a1acd8640 1 --2- 172.21.15.184:0/1253171415 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6a1415e620 0x7f6a14164aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:42:28.037 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:42:28.032+0000 7f6a1acd8640 1 --2- 172.21.15.184:0/1253171415 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6a1415de80 0x7f6a1414be40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:42:28.037 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:42:28.032+0000 7f6a1acd8640 1 --2- 172.21.15.184:0/1253171415 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f6a1406e240 0x7f6a1414a1d0 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:42:28.038 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:42:28.032+0000 7f6a1acd8640 1 -- 172.21.15.184:0/1253171415 >> 172.21.15.184:0/1253171415 conn(0x7f6a140b28f0 msgr2=0x7f6a140b0e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:42:28.038 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:42:28.032+0000 7f6a1acd8640 1 -- 172.21.15.184:0/1253171415 shutdown_connections 2024-09-16T15:42:28.038 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:42:28.032+0000 7f6a1acd8640 1 -- 172.21.15.184:0/1253171415 wait complete. 2024-09-16T15:42:28.052 DEBUG:teuthology.orchestra.run.smithi184:> sudo ceph --cluster ceph osd new 9864a496-f169-486b-95c4-1d59162793a7 2 2024-09-16T15:42:28.169 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:42:28.164+0000 7f6d2da32640 1 -- 172.21.15.184:0/3049697975 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6d280cff50 msgr2=0x7f6d280d0380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:42:28.170 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:42:28.164+0000 7f6d2da32640 1 --2- 172.21.15.184:0/3049697975 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6d280cff50 0x7f6d280d0380 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f6d28074af0 tx=0x7f6d1c01c890 comp rx=0 tx=0).stop 2024-09-16T15:42:28.170 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:42:28.164+0000 7f6d2da32640 1 -- 172.21.15.184:0/3049697975 shutdown_connections 2024-09-16T15:42:28.170 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:42:28.164+0000 7f6d2da32640 1 --2- 172.21.15.184:0/3049697975 >> v2:172.21.15.123:3300/0 conn(0x7f6d28070500 0x7f6d28070960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:42:28.170 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:42:28.164+0000 7f6d2da32640 1 --2- 172.21.15.184:0/3049697975 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6d280cff50 0x7f6d280d0380 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:42:28.170 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:42:28.164+0000 7f6d2da32640 1 -- 172.21.15.184:0/3049697975 >> 172.21.15.184:0/3049697975 conn(0x7f6d280d0bc0 msgr2=0x7f6d280d0fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:42:28.170 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:42:28.164+0000 7f6d2da32640 1 -- 172.21.15.184:0/3049697975 shutdown_connections 2024-09-16T15:42:28.170 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:42:28.164+0000 7f6d2da32640 1 -- 172.21.15.184:0/3049697975 wait complete. 2024-09-16T15:42:28.170 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:42:28.168+0000 7f6d2da32640 1 Processor -- start 2024-09-16T15:42:28.171 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:42:28.168+0000 7f6d2da32640 1 -- start start 2024-09-16T15:42:28.173 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:42:28.168+0000 7f6d2da32640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6d28070500 0x7f6d281f8ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:42:28.173 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:42:28.168+0000 7f6d2da32640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f6d280cff50 0x7f6d28157ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:42:28.173 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:42:28.168+0000 7f6d2da32640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6d28155000 0x7f6d280cd450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:42:28.174 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:42:28.168+0000 7f6d2da32640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f6d28167db0 con 0x7f6d28070500 2024-09-16T15:42:28.174 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:42:28.168+0000 7f6d2da32640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f6d28167c30 con 0x7f6d28155000 2024-09-16T15:42:28.174 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:42:28.168+0000 7f6d2da32640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f6d28167f30 con 0x7f6d280cff50 2024-09-16T15:42:28.174 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:42:28.168+0000 7f6d267fc640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f6d280cff50 0x7f6d28157ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:42:28.174 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:42:28.168+0000 7f6d277fe640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6d28155000 0x7f6d280cd450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:42:28.174 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:42:28.168+0000 7f6d267fc640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f6d280cff50 0x7f6d28157ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3301/0 says I am v2:172.21.15.184:40674/0 (socket says 172.21.15.184:40674) 2024-09-16T15:42:28.174 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:42:28.168+0000 7f6d267fc640 1 -- 172.21.15.184:0/1568950313 learned_addr learned my addr 172.21.15.184:0/1568950313 (peer_addr_for_me v2:172.21.15.184:0/0) 2024-09-16T15:42:28.174 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:42:28.168+0000 7f6d26ffd640 1 --2- 172.21.15.184:0/1568950313 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6d28070500 0x7f6d281f8ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:42:28.174 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:42:28.168+0000 7f6d267fc640 1 -- 172.21.15.184:0/1568950313 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6d28155000 msgr2=0x7f6d280cd450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:42:28.174 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:42:28.168+0000 7f6d267fc640 1 --2- 172.21.15.184:0/1568950313 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6d28155000 0x7f6d280cd450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:42:28.174 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:42:28.168+0000 7f6d267fc640 1 -- 172.21.15.184:0/1568950313 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6d28070500 msgr2=0x7f6d281f8ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:42:28.174 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:42:28.168+0000 7f6d267fc640 1 --2- 172.21.15.184:0/1568950313 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6d28070500 0x7f6d281f8ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:42:28.174 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:42:28.168+0000 7f6d267fc640 1 -- 172.21.15.184:0/1568950313 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6d281f9000 con 0x7f6d280cff50 2024-09-16T15:42:28.174 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:42:28.168+0000 7f6d277fe640 1 --2- 172.21.15.184:0/1568950313 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6d28155000 0x7f6d280cd450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T15:42:28.175 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:42:28.168+0000 7f6d267fc640 1 --2- 172.21.15.184:0/1568950313 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f6d280cff50 0x7f6d28157ca0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f6d1c0b0040 tx=0x7f6d1c005760 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:42:28.175 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:42:28.168+0000 7f6d0ffff640 1 -- 172.21.15.184:0/1568950313 <== mon.2 v2:172.21.15.184:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d1c0b2040 con 0x7f6d280cff50 2024-09-16T15:42:28.175 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:42:28.168+0000 7f6d0ffff640 1 -- 172.21.15.184:0/1568950313 <== mon.2 v2:172.21.15.184:3301/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f6d1c005330 con 0x7f6d280cff50 2024-09-16T15:42:28.175 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:42:28.168+0000 7f6d2da32640 1 -- 172.21.15.184:0/1568950313 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6d281f92d0 con 0x7f6d280cff50 2024-09-16T15:42:28.175 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:42:28.168+0000 7f6d0ffff640 1 -- 172.21.15.184:0/1568950313 <== mon.2 v2:172.21.15.184:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d1c003700 con 0x7f6d280cff50 2024-09-16T15:42:28.175 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:42:28.168+0000 7f6d2da32640 1 -- 172.21.15.184:0/1568950313 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f6d281f9890 con 0x7f6d280cff50 2024-09-16T15:42:28.175 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:42:28.168+0000 7f6d0ffff640 1 -- 172.21.15.184:0/1568950313 <== mon.2 v2:172.21.15.184:3301/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f6d1c0a5950 con 0x7f6d280cff50 2024-09-16T15:42:28.175 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:42:28.168+0000 7f6d0ffff640 1 -- 172.21.15.184:0/1568950313 <== mon.2 v2:172.21.15.184:3301/0 5 ==== osd_map(4..4 src has 1..4) ==== 955+0+0 (secure 0 0 0) 0x7f6d1c004770 con 0x7f6d280cff50 2024-09-16T15:42:28.176 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:42:28.168+0000 7f6d2da32640 1 -- 172.21.15.184:0/1568950313 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6d28070960 con 0x7f6d280cff50 2024-09-16T15:42:28.176 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:42:28.172+0000 7f6d0ffff640 1 -- 172.21.15.184:0/1568950313 <== mon.2 v2:172.21.15.184:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+75856 (secure 0 0 0) 0x7f6d1c0164b0 con 0x7f6d280cff50 2024-09-16T15:42:28.246 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:42:28.244+0000 7f6d2da32640 1 -- 172.21.15.184:0/1568950313 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_command({"prefix": "osd new", "uuid": "9864a496-f169-486b-95c4-1d59162793a7", "id": 2} v 0) -- 0x7f6d28074740 con 0x7f6d280cff50 2024-09-16T15:42:28.253 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:42:28.248+0000 7f6d0ffff640 1 -- 172.21.15.184:0/1568950313 <== mon.2 v2:172.21.15.184:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "9864a496-f169-486b-95c4-1d59162793a7", "id": 2}]=0 v5) ==== 112+0+2 (secure 0 0 0) 0x7f6d1c0038a0 con 0x7f6d280cff50 2024-09-16T15:42:28.253 INFO:teuthology.orchestra.run.smithi184.stdout:2 2024-09-16T15:42:28.254 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:42:28.252+0000 7f6d2da32640 1 -- 172.21.15.184:0/1568950313 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f6d280cff50 msgr2=0x7f6d28157ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:42:28.255 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:42:28.252+0000 7f6d2da32640 1 --2- 172.21.15.184:0/1568950313 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f6d280cff50 0x7f6d28157ca0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f6d1c0b0040 tx=0x7f6d1c005760 comp rx=0 tx=0).stop 2024-09-16T15:42:28.255 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:42:28.252+0000 7f6d2da32640 1 -- 172.21.15.184:0/1568950313 shutdown_connections 2024-09-16T15:42:28.255 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:42:28.252+0000 7f6d2da32640 1 --2- 172.21.15.184:0/1568950313 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6d28155000 0x7f6d280cd450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:42:28.255 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:42:28.252+0000 7f6d2da32640 1 --2- 172.21.15.184:0/1568950313 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f6d280cff50 0x7f6d28157ca0 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:42:28.255 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:42:28.252+0000 7f6d2da32640 1 --2- 172.21.15.184:0/1568950313 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6d28070500 0x7f6d281f8ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:42:28.255 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:42:28.252+0000 7f6d2da32640 1 -- 172.21.15.184:0/1568950313 >> 172.21.15.184:0/1568950313 conn(0x7f6d280d0bc0 msgr2=0x7f6d28154330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:42:28.255 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:42:28.252+0000 7f6d2da32640 1 -- 172.21.15.184:0/1568950313 shutdown_connections 2024-09-16T15:42:28.255 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:42:28.252+0000 7f6d2da32640 1 -- 172.21.15.184:0/1568950313 wait complete. 2024-09-16T15:42:28.270 DEBUG:teuthology.orchestra.run.smithi184:> sudo ceph --cluster ceph osd new c6d1f817-f074-4325-95e1-578eba13edda 3 2024-09-16T15:42:28.391 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:42:28.388+0000 7f73591df640 1 -- 172.21.15.184:0/1536927673 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f73540d2230 msgr2=0x7f73540c69f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:42:28.391 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:42:28.388+0000 7f73591df640 1 --2- 172.21.15.184:0/1536927673 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f73540d2230 0x7f73540c69f0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f7354074720 tx=0x7f734c01f580 comp rx=0 tx=0).stop 2024-09-16T15:42:28.391 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:42:28.388+0000 7f73591df640 1 -- 172.21.15.184:0/1536927673 shutdown_connections 2024-09-16T15:42:28.391 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:42:28.388+0000 7f73591df640 1 --2- 172.21.15.184:0/1536927673 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f73540d2230 0x7f73540c69f0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:42:28.391 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:42:28.388+0000 7f73591df640 1 --2- 172.21.15.184:0/1536927673 >> v2:172.21.15.123:3300/0 conn(0x7f735415bd60 0x7f73540cf850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:42:28.391 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:42:28.388+0000 7f73591df640 1 -- 172.21.15.184:0/1536927673 >> 172.21.15.184:0/1536927673 conn(0x7f735409aa40 msgr2=0x7f735409ae50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:42:28.391 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:42:28.388+0000 7f73591df640 1 -- 172.21.15.184:0/1536927673 shutdown_connections 2024-09-16T15:42:28.391 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:42:28.388+0000 7f73591df640 1 -- 172.21.15.184:0/1536927673 wait complete. 2024-09-16T15:42:28.392 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:42:28.388+0000 7f73591df640 1 Processor -- start 2024-09-16T15:42:28.392 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:42:28.388+0000 7f73591df640 1 -- start start 2024-09-16T15:42:28.394 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:42:28.388+0000 7f73591df640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f73540b31a0 0x7f7354148400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:42:28.394 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:42:28.388+0000 7f73591df640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f73540d2230 0x7f735415eba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:42:28.394 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:42:28.388+0000 7f73591df640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f735415bd60 0x7f735414aca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:42:28.394 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:42:28.388+0000 7f73591df640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f73540c73d0 con 0x7f73540b31a0 2024-09-16T15:42:28.394 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:42:28.388+0000 7f73591df640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f73540c7250 con 0x7f735415bd60 2024-09-16T15:42:28.395 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:42:28.388+0000 7f73591df640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f73540c7550 con 0x7f73540d2230 2024-09-16T15:42:28.395 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:42:28.388+0000 7f735259c640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f73540d2230 0x7f735415eba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:42:28.395 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:42:28.388+0000 7f735359e640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f735415bd60 0x7f735414aca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:42:28.395 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:42:28.388+0000 7f735259c640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f73540d2230 0x7f735415eba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3301/0 says I am v2:172.21.15.184:40690/0 (socket says 172.21.15.184:40690) 2024-09-16T15:42:28.395 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:42:28.388+0000 7f735359e640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f735415bd60 0x7f735414aca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.184:53984/0 (socket says 172.21.15.184:53984) 2024-09-16T15:42:28.395 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:42:28.388+0000 7f735259c640 1 -- 172.21.15.184:0/2750522199 learned_addr learned my addr 172.21.15.184:0/2750522199 (peer_addr_for_me v2:172.21.15.184:0/0) 2024-09-16T15:42:28.395 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:42:28.388+0000 7f7352d9d640 1 --2- 172.21.15.184:0/2750522199 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f73540b31a0 0x7f7354148400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:42:28.395 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:42:28.388+0000 7f735359e640 1 -- 172.21.15.184:0/2750522199 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f73540d2230 msgr2=0x7f735415eba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:42:28.395 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:42:28.388+0000 7f735359e640 1 --2- 172.21.15.184:0/2750522199 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f73540d2230 0x7f735415eba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:42:28.395 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:42:28.388+0000 7f735359e640 1 -- 172.21.15.184:0/2750522199 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f73540b31a0 msgr2=0x7f7354148400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:42:28.395 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:42:28.388+0000 7f735359e640 1 --2- 172.21.15.184:0/2750522199 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f73540b31a0 0x7f7354148400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:42:28.395 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:42:28.388+0000 7f735359e640 1 -- 172.21.15.184:0/2750522199 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f73540b1a10 con 0x7f735415bd60 2024-09-16T15:42:28.395 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:42:28.388+0000 7f7352d9d640 1 --2- 172.21.15.184:0/2750522199 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f73540b31a0 0x7f7354148400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T15:42:28.395 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:42:28.388+0000 7f735259c640 1 --2- 172.21.15.184:0/2750522199 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f73540d2230 0x7f735415eba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T15:42:28.396 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:42:28.388+0000 7f735359e640 1 --2- 172.21.15.184:0/2750522199 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f735415bd60 0x7f735414aca0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f73540c6090 tx=0x7f734c0b06b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:42:28.396 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:42:28.388+0000 7f733b7fe640 1 -- 172.21.15.184:0/2750522199 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f734c0bf070 con 0x7f735415bd60 2024-09-16T15:42:28.396 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:42:28.388+0000 7f733b7fe640 1 -- 172.21.15.184:0/2750522199 <== mon.1 v2:172.21.15.184:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f734c004b40 con 0x7f735415bd60 2024-09-16T15:42:28.396 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:42:28.388+0000 7f73591df640 1 -- 172.21.15.184:0/2750522199 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f73540b1170 con 0x7f735415bd60 2024-09-16T15:42:28.396 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:42:28.388+0000 7f733b7fe640 1 -- 172.21.15.184:0/2750522199 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f734c0b5040 con 0x7f735415bd60 2024-09-16T15:42:28.396 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:42:28.388+0000 7f73591df640 1 -- 172.21.15.184:0/2750522199 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f73540b1730 con 0x7f735415bd60 2024-09-16T15:42:28.396 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:42:28.392+0000 7f733b7fe640 1 -- 172.21.15.184:0/2750522199 <== mon.1 v2:172.21.15.184:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f734c0c2070 con 0x7f735415bd60 2024-09-16T15:42:28.396 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:42:28.392+0000 7f733b7fe640 1 -- 172.21.15.184:0/2750522199 <== mon.1 v2:172.21.15.184:3300/0 5 ==== osd_map(5..5 src has 1..5) ==== 1070+0+0 (secure 0 0 0) 0x7f734c0044f0 con 0x7f735415bd60 2024-09-16T15:42:28.397 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:42:28.392+0000 7f73591df640 1 -- 172.21.15.184:0/2750522199 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f73540c7ae0 con 0x7f735415bd60 2024-09-16T15:42:28.397 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:42:28.392+0000 7f733b7fe640 1 -- 172.21.15.184:0/2750522199 <== mon.1 v2:172.21.15.184:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+75856 (secure 0 0 0) 0x7f734c0190e0 con 0x7f735415bd60 2024-09-16T15:42:28.467 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:42:28.464+0000 7f73591df640 1 -- 172.21.15.184:0/2750522199 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_command({"prefix": "osd new", "uuid": "c6d1f817-f074-4325-95e1-578eba13edda", "id": 3} v 0) -- 0x7f7354148940 con 0x7f735415bd60 2024-09-16T15:42:28.475 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:42:28.472+0000 7f733b7fe640 1 -- 172.21.15.184:0/2750522199 <== mon.1 v2:172.21.15.184:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "c6d1f817-f074-4325-95e1-578eba13edda", "id": 3}]=0 v6) ==== 112+0+2 (secure 0 0 0) 0x7f734c004740 con 0x7f735415bd60 2024-09-16T15:42:28.475 INFO:teuthology.orchestra.run.smithi184.stdout:3 2024-09-16T15:42:28.477 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:42:28.472+0000 7f73591df640 1 -- 172.21.15.184:0/2750522199 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f735415bd60 msgr2=0x7f735414aca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:42:28.477 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:42:28.472+0000 7f73591df640 1 --2- 172.21.15.184:0/2750522199 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f735415bd60 0x7f735414aca0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f73540c6090 tx=0x7f734c0b06b0 comp rx=0 tx=0).stop 2024-09-16T15:42:28.477 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:42:28.472+0000 7f73591df640 1 -- 172.21.15.184:0/2750522199 shutdown_connections 2024-09-16T15:42:28.477 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:42:28.472+0000 7f73591df640 1 --2- 172.21.15.184:0/2750522199 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f735415bd60 0x7f735414aca0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:42:28.477 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:42:28.472+0000 7f73591df640 1 --2- 172.21.15.184:0/2750522199 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f73540d2230 0x7f735415eba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:42:28.477 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:42:28.472+0000 7f73591df640 1 --2- 172.21.15.184:0/2750522199 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f73540b31a0 0x7f7354148400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:42:28.477 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:42:28.472+0000 7f73591df640 1 -- 172.21.15.184:0/2750522199 >> 172.21.15.184:0/2750522199 conn(0x7f735409aa40 msgr2=0x7f73540cb4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:42:28.477 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:42:28.472+0000 7f73591df640 1 -- 172.21.15.184:0/2750522199 shutdown_connections 2024-09-16T15:42:28.477 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:42:28.472+0000 7f73591df640 1 -- 172.21.15.184:0/2750522199 wait complete. 2024-09-16T15:42:28.492 DEBUG:teuthology.orchestra.run.smithi184:> sudo ceph --cluster ceph osd new 200a8df4-50ef-40a5-b48a-58981269c086 4 2024-09-16T15:42:28.610 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:42:28.604+0000 7fdebff27640 1 -- 172.21.15.184:0/2709436077 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fdea4004810 msgr2=0x7fdea4004c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:42:28.610 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:42:28.604+0000 7fdebff27640 1 --2- 172.21.15.184:0/2709436077 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fdea4004810 0x7fdea4004c90 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fdeb400c260 tx=0x7fdeb401f6c0 comp rx=0 tx=0).stop 2024-09-16T15:42:28.610 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:42:28.608+0000 7fdebff27640 1 -- 172.21.15.184:0/2709436077 shutdown_connections 2024-09-16T15:42:28.610 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:42:28.608+0000 7fdebff27640 1 --2- 172.21.15.184:0/2709436077 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fdea4004810 0x7fdea4004c90 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:42:28.610 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:42:28.608+0000 7fdebff27640 1 -- 172.21.15.184:0/2709436077 >> 172.21.15.184:0/2709436077 conn(0x7fdeb80d0bc0 msgr2=0x7fdeb80d0fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:42:28.610 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:42:28.608+0000 7fdebff27640 1 -- 172.21.15.184:0/2709436077 shutdown_connections 2024-09-16T15:42:28.611 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:42:28.608+0000 7fdebff27640 1 -- 172.21.15.184:0/2709436077 wait complete. 2024-09-16T15:42:28.611 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:42:28.608+0000 7fdebff27640 1 Processor -- start 2024-09-16T15:42:28.611 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:42:28.608+0000 7fdebff27640 1 -- start start 2024-09-16T15:42:28.613 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:42:28.608+0000 7fdebff27640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fdea4004810 0x7fdeb81f3200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:42:28.613 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:42:28.608+0000 7fdebff27640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdeb815db80 0x7fdeb815a490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:42:28.613 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:42:28.608+0000 7fdebff27640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fdeb81525f0 0x7fdeb8152a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:42:28.613 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:42:28.608+0000 7fdebff27640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fdeb8073bd0 con 0x7fdeb815db80 2024-09-16T15:42:28.613 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:42:28.608+0000 7fdebff27640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fdeb8073a50 con 0x7fdeb81525f0 2024-09-16T15:42:28.613 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:42:28.608+0000 7fdebff27640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fdeb80cff50 con 0x7fdea4004810 2024-09-16T15:42:28.613 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:42:28.608+0000 7fdebdcc3640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fdea4004810 0x7fdeb81f3200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:42:28.614 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:42:28.608+0000 7fdebe4c4640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fdeb81525f0 0x7fdeb8152a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:42:28.614 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:42:28.608+0000 7fdebdcc3640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fdea4004810 0x7fdeb81f3200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3301/0 says I am v2:172.21.15.184:40700/0 (socket says 172.21.15.184:40700) 2024-09-16T15:42:28.614 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:42:28.608+0000 7fdebdcc3640 1 -- 172.21.15.184:0/3423844901 learned_addr learned my addr 172.21.15.184:0/3423844901 (peer_addr_for_me v2:172.21.15.184:0/0) 2024-09-16T15:42:28.614 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:42:28.608+0000 7fdebe4c4640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fdeb81525f0 0x7fdeb8152a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.184:54008/0 (socket says 172.21.15.184:54008) 2024-09-16T15:42:28.614 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:42:28.608+0000 7fdebd4c2640 1 --2- 172.21.15.184:0/3423844901 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdeb815db80 0x7fdeb815a490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:42:28.614 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:42:28.608+0000 7fdebdcc3640 1 -- 172.21.15.184:0/3423844901 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fdeb81525f0 msgr2=0x7fdeb8152a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:42:28.614 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:42:28.608+0000 7fdebdcc3640 1 --2- 172.21.15.184:0/3423844901 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fdeb81525f0 0x7fdeb8152a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:42:28.614 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:42:28.608+0000 7fdebdcc3640 1 -- 172.21.15.184:0/3423844901 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdeb815db80 msgr2=0x7fdeb815a490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:42:28.614 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:42:28.608+0000 7fdebdcc3640 1 --2- 172.21.15.184:0/3423844901 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdeb815db80 0x7fdeb815a490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:42:28.614 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:42:28.608+0000 7fdebdcc3640 1 -- 172.21.15.184:0/3423844901 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdeb8072010 con 0x7fdea4004810 2024-09-16T15:42:28.614 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:42:28.608+0000 7fdebe4c4640 1 --2- 172.21.15.184:0/3423844901 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fdeb81525f0 0x7fdeb8152a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T15:42:28.614 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:42:28.608+0000 7fdebdcc3640 1 --2- 172.21.15.184:0/3423844901 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fdea4004810 0x7fdeb81f3200 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fdea8003ee0 tx=0x7fdea800b9c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:42:28.614 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:42:28.608+0000 7fdeae7fc640 1 -- 172.21.15.184:0/3423844901 <== mon.2 v2:172.21.15.184:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdea80107d0 con 0x7fdea4004810 2024-09-16T15:42:28.615 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:42:28.608+0000 7fdeae7fc640 1 -- 172.21.15.184:0/3423844901 <== mon.2 v2:172.21.15.184:3301/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fdea8010970 con 0x7fdea4004810 2024-09-16T15:42:28.615 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:42:28.608+0000 7fdeae7fc640 1 -- 172.21.15.184:0/3423844901 <== mon.2 v2:172.21.15.184:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdea8010d40 con 0x7fdea4004810 2024-09-16T15:42:28.615 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:42:28.608+0000 7fdebff27640 1 -- 172.21.15.184:0/3423844901 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdeb81f3740 con 0x7fdea4004810 2024-09-16T15:42:28.615 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:42:28.608+0000 7fdebff27640 1 -- 172.21.15.184:0/3423844901 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fdeb8150fc0 con 0x7fdea4004810 2024-09-16T15:42:28.615 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:42:28.608+0000 7fdeae7fc640 1 -- 172.21.15.184:0/3423844901 <== mon.2 v2:172.21.15.184:3301/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fdea80237d0 con 0x7fdea4004810 2024-09-16T15:42:28.615 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:42:28.608+0000 7fdeae7fc640 1 -- 172.21.15.184:0/3423844901 <== mon.2 v2:172.21.15.184:3301/0 5 ==== osd_map(6..6 src has 1..6) ==== 1185+0+0 (secure 0 0 0) 0x7fdea801a430 con 0x7fdea4004810 2024-09-16T15:42:28.616 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:42:28.612+0000 7fdebff27640 1 -- 172.21.15.184:0/3423844901 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdeb815e600 con 0x7fdea4004810 2024-09-16T15:42:28.616 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:42:28.612+0000 7fdeae7fc640 1 -- 172.21.15.184:0/3423844901 <== mon.2 v2:172.21.15.184:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+75856 (secure 0 0 0) 0x7fdea8011440 con 0x7fdea4004810 2024-09-16T15:42:28.686 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:42:28.684+0000 7fdebff27640 1 -- 172.21.15.184:0/3423844901 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_command({"prefix": "osd new", "uuid": "200a8df4-50ef-40a5-b48a-58981269c086", "id": 4} v 0) -- 0x7fdeb80cb300 con 0x7fdea4004810 2024-09-16T15:42:28.693 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:42:28.688+0000 7fdeae7fc640 1 -- 172.21.15.184:0/3423844901 <== mon.2 v2:172.21.15.184:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "200a8df4-50ef-40a5-b48a-58981269c086", "id": 4}]=0 v7) ==== 112+0+2 (secure 0 0 0) 0x7fdea8011620 con 0x7fdea4004810 2024-09-16T15:42:28.694 INFO:teuthology.orchestra.run.smithi184.stdout:4 2024-09-16T15:42:28.695 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:42:28.692+0000 7fdebff27640 1 -- 172.21.15.184:0/3423844901 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fdea4004810 msgr2=0x7fdeb81f3200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:42:28.695 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:42:28.692+0000 7fdebff27640 1 --2- 172.21.15.184:0/3423844901 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fdea4004810 0x7fdeb81f3200 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fdea8003ee0 tx=0x7fdea800b9c0 comp rx=0 tx=0).stop 2024-09-16T15:42:28.696 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:42:28.692+0000 7fdebff27640 1 -- 172.21.15.184:0/3423844901 shutdown_connections 2024-09-16T15:42:28.696 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:42:28.692+0000 7fdebff27640 1 --2- 172.21.15.184:0/3423844901 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fdeb81525f0 0x7fdeb8152a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:42:28.696 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:42:28.692+0000 7fdebff27640 1 --2- 172.21.15.184:0/3423844901 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdeb815db80 0x7fdeb815a490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:42:28.696 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:42:28.692+0000 7fdebff27640 1 --2- 172.21.15.184:0/3423844901 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fdea4004810 0x7fdeb81f3200 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:42:28.696 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:42:28.692+0000 7fdebff27640 1 -- 172.21.15.184:0/3423844901 >> 172.21.15.184:0/3423844901 conn(0x7fdeb80d0bc0 msgr2=0x7fdeb80cb5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:42:28.696 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:42:28.692+0000 7fdebff27640 1 -- 172.21.15.184:0/3423844901 shutdown_connections 2024-09-16T15:42:28.697 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:42:28.692+0000 7fdebff27640 1 -- 172.21.15.184:0/3423844901 wait complete. 2024-09-16T15:42:28.710 DEBUG:teuthology.orchestra.run.smithi184:> sudo ceph --cluster ceph osd new 32d5a900-da3d-412c-bc19-13861a2368d8 5 2024-09-16T15:42:28.824 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:42:28.820+0000 7f752c794640 1 -- 172.21.15.184:0/1399009151 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f752415bf50 msgr2=0x7f75240b1610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:42:28.825 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:42:28.820+0000 7f752c794640 1 --2- 172.21.15.184:0/1399009151 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f752415bf50 0x7f75240b1610 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f7524072830 tx=0x7f752001cd70 comp rx=0 tx=0).stop 2024-09-16T15:42:28.825 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:42:28.820+0000 7f752c794640 1 -- 172.21.15.184:0/1399009151 shutdown_connections 2024-09-16T15:42:28.825 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:42:28.820+0000 7f752c794640 1 --2- 172.21.15.184:0/1399009151 >> v2:172.21.15.123:3300/0 conn(0x7f75240b0360 0x7f75240d00c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:42:28.825 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:42:28.820+0000 7f752c794640 1 --2- 172.21.15.184:0/1399009151 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f752415bf50 0x7f75240b1610 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:42:28.825 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:42:28.820+0000 7f752c794640 1 -- 172.21.15.184:0/1399009151 >> 172.21.15.184:0/1399009151 conn(0x7f7524074210 msgr2=0x7f7524074620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:42:28.825 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:42:28.820+0000 7f752c794640 1 -- 172.21.15.184:0/1399009151 shutdown_connections 2024-09-16T15:42:28.825 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:42:28.820+0000 7f752c794640 1 -- 172.21.15.184:0/1399009151 wait complete. 2024-09-16T15:42:28.825 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:42:28.820+0000 7f752c794640 1 Processor -- start 2024-09-16T15:42:28.826 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:42:28.820+0000 7f752c794640 1 -- start start 2024-09-16T15:42:28.828 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:42:28.824+0000 7f752c794640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7524074dc0 0x7f75240ccf30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:42:28.828 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:42:28.824+0000 7f752c794640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f75240b0360 0x7f75241f7ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:42:28.828 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:42:28.824+0000 7f752c794640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f752415bf50 0x7f75241f8010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:42:28.828 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:42:28.824+0000 7f752c794640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f7524168c20 con 0x7f7524074dc0 2024-09-16T15:42:28.828 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:42:28.824+0000 7f752c794640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f7524168aa0 con 0x7f752415bf50 2024-09-16T15:42:28.828 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:42:28.824+0000 7f752c794640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f7524168da0 con 0x7f75240b0360 2024-09-16T15:42:28.828 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:42:28.824+0000 7f752ad31640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f752415bf50 0x7f75241f8010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:42:28.828 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:42:28.824+0000 7f7529d2f640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f75240b0360 0x7f75241f7ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:42:28.829 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:42:28.824+0000 7f752ad31640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f752415bf50 0x7f75241f8010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.184:54030/0 (socket says 172.21.15.184:54030) 2024-09-16T15:42:28.829 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:42:28.824+0000 7f7529d2f640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f75240b0360 0x7f75241f7ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3301/0 says I am v2:172.21.15.184:40708/0 (socket says 172.21.15.184:40708) 2024-09-16T15:42:28.829 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:42:28.824+0000 7f752ad31640 1 -- 172.21.15.184:0/3306157527 learned_addr learned my addr 172.21.15.184:0/3306157527 (peer_addr_for_me v2:172.21.15.184:0/0) 2024-09-16T15:42:28.829 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:42:28.824+0000 7f752a530640 1 --2- 172.21.15.184:0/3306157527 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7524074dc0 0x7f75240ccf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:42:28.829 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:42:28.824+0000 7f752ad31640 1 -- 172.21.15.184:0/3306157527 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f75240b0360 msgr2=0x7f75241f7ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:42:28.829 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:42:28.824+0000 7f752ad31640 1 --2- 172.21.15.184:0/3306157527 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f75240b0360 0x7f75241f7ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:42:28.829 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:42:28.824+0000 7f752ad31640 1 -- 172.21.15.184:0/3306157527 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7524074dc0 msgr2=0x7f75240ccf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:42:28.829 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:42:28.824+0000 7f752ad31640 1 --2- 172.21.15.184:0/3306157527 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7524074dc0 0x7f75240ccf30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:42:28.829 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:42:28.824+0000 7f752ad31640 1 -- 172.21.15.184:0/3306157527 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f75240cc900 con 0x7f752415bf50 2024-09-16T15:42:28.829 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:42:28.824+0000 7f7529d2f640 1 --2- 172.21.15.184:0/3306157527 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f75240b0360 0x7f75241f7ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T15:42:28.829 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:42:28.824+0000 7f752ad31640 1 --2- 172.21.15.184:0/3306157527 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f752415bf50 0x7f75241f8010 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f751c00b410 tx=0x7f751c00b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:42:28.829 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:42:28.824+0000 7f751affd640 1 -- 172.21.15.184:0/3306157527 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f751c015020 con 0x7f752415bf50 2024-09-16T15:42:28.830 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:42:28.824+0000 7f751affd640 1 -- 172.21.15.184:0/3306157527 <== mon.1 v2:172.21.15.184:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f751c011920 con 0x7f752415bf50 2024-09-16T15:42:28.830 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:42:28.824+0000 7f752c794640 1 -- 172.21.15.184:0/3306157527 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f75240cac70 con 0x7f752415bf50 2024-09-16T15:42:28.830 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:42:28.824+0000 7f751affd640 1 -- 172.21.15.184:0/3306157527 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f751c011d70 con 0x7f752415bf50 2024-09-16T15:42:28.830 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:42:28.824+0000 7f752c794640 1 -- 172.21.15.184:0/3306157527 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f75240cb230 con 0x7f752415bf50 2024-09-16T15:42:28.830 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:42:28.824+0000 7f751affd640 1 -- 172.21.15.184:0/3306157527 <== mon.1 v2:172.21.15.184:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f751c018020 con 0x7f752415bf50 2024-09-16T15:42:28.830 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:42:28.824+0000 7f751affd640 1 -- 172.21.15.184:0/3306157527 <== mon.1 v2:172.21.15.184:3300/0 5 ==== osd_map(7..7 src has 1..7) ==== 1300+0+0 (secure 0 0 0) 0x7f751c010380 con 0x7f752415bf50 2024-09-16T15:42:28.830 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:42:28.824+0000 7f752c794640 1 -- 172.21.15.184:0/3306157527 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f75240d00c0 con 0x7f752415bf50 2024-09-16T15:42:28.830 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:42:28.824+0000 7f751affd640 1 -- 172.21.15.184:0/3306157527 <== mon.1 v2:172.21.15.184:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+75856 (secure 0 0 0) 0x7f751c01f020 con 0x7f752415bf50 2024-09-16T15:42:28.938 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:42:28.936+0000 7f752c794640 1 -- 172.21.15.184:0/3306157527 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_command({"prefix": "osd new", "uuid": "32d5a900-da3d-412c-bc19-13861a2368d8", "id": 5} v 0) -- 0x7f7524097010 con 0x7f752415bf50 2024-09-16T15:42:28.948 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:42:28.944+0000 7f751affd640 1 -- 172.21.15.184:0/3306157527 <== mon.1 v2:172.21.15.184:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "32d5a900-da3d-412c-bc19-13861a2368d8", "id": 5}]=0 v8) ==== 112+0+2 (secure 0 0 0) 0x7f751c019480 con 0x7f752415bf50 2024-09-16T15:42:28.948 INFO:teuthology.orchestra.run.smithi184.stdout:5 2024-09-16T15:42:28.954 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:42:28.952+0000 7f752c794640 1 -- 172.21.15.184:0/3306157527 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f752415bf50 msgr2=0x7f75241f8010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:42:28.955 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:42:28.952+0000 7f752c794640 1 --2- 172.21.15.184:0/3306157527 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f752415bf50 0x7f75241f8010 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f751c00b410 tx=0x7f751c00b8d0 comp rx=0 tx=0).stop 2024-09-16T15:42:28.955 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:42:28.952+0000 7f752c794640 1 -- 172.21.15.184:0/3306157527 shutdown_connections 2024-09-16T15:42:28.955 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:42:28.952+0000 7f752c794640 1 --2- 172.21.15.184:0/3306157527 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f752415bf50 0x7f75241f8010 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:42:28.955 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:42:28.952+0000 7f752c794640 1 --2- 172.21.15.184:0/3306157527 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f75240b0360 0x7f75241f7ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:42:28.955 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:42:28.952+0000 7f752c794640 1 --2- 172.21.15.184:0/3306157527 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7524074dc0 0x7f75240ccf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:42:28.955 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:42:28.952+0000 7f752c794640 1 -- 172.21.15.184:0/3306157527 >> 172.21.15.184:0/3306157527 conn(0x7f7524074210 msgr2=0x7f75240940a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:42:28.955 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:42:28.952+0000 7f752c794640 1 -- 172.21.15.184:0/3306157527 shutdown_connections 2024-09-16T15:42:28.955 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:42:28.952+0000 7f752c794640 1 -- 172.21.15.184:0/3306157527 wait complete. 2024-09-16T15:42:28.974 DEBUG:teuthology.orchestra.run.smithi184:> sudo ceph --cluster ceph osd new cde6f39e-aa1b-4260-ac19-774ee6aaca72 6 2024-09-16T15:42:29.089 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:42:29.084+0000 7fa5f2e6d640 1 -- 172.21.15.184:0/3301817328 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fa5d8004810 msgr2=0x7fa5d8004c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:42:29.089 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:42:29.084+0000 7fa5f2e6d640 1 --2- 172.21.15.184:0/3301817328 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fa5d8004810 0x7fa5d8004c90 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fa5d4001ec0 tx=0x7fa5d401cfa0 comp rx=0 tx=0).stop 2024-09-16T15:42:29.090 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:42:29.084+0000 7fa5f2e6d640 1 -- 172.21.15.184:0/3301817328 shutdown_connections 2024-09-16T15:42:29.090 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:42:29.084+0000 7fa5f2e6d640 1 --2- 172.21.15.184:0/3301817328 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fa5d8004810 0x7fa5d8004c90 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:42:29.090 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:42:29.084+0000 7fa5f2e6d640 1 -- 172.21.15.184:0/3301817328 >> 172.21.15.184:0/3301817328 conn(0x7fa5ec09aa40 msgr2=0x7fa5ec09ae50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:42:29.090 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:42:29.084+0000 7fa5f2e6d640 1 -- 172.21.15.184:0/3301817328 shutdown_connections 2024-09-16T15:42:29.090 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:42:29.084+0000 7fa5f2e6d640 1 -- 172.21.15.184:0/3301817328 wait complete. 2024-09-16T15:42:29.090 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:42:29.088+0000 7fa5f2e6d640 1 Processor -- start 2024-09-16T15:42:29.091 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:42:29.088+0000 7fa5f2e6d640 1 -- start start 2024-09-16T15:42:29.092 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:42:29.088+0000 7fa5f2e6d640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa5d8004810 0x7fa5ec0cef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:42:29.092 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:42:29.088+0000 7fa5f2e6d640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa5ec0b7650 0x7fa5ec14ca40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:42:29.092 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:42:29.088+0000 7fa5f2e6d640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fa5ec0cd6e0 0x7fa5ec0b4ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:42:29.092 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:42:29.088+0000 7fa5f2e6d640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fa5ec0cfd40 con 0x7fa5ec0b7650 2024-09-16T15:42:29.093 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:42:29.088+0000 7fa5f2e6d640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fa5ec0cfbc0 con 0x7fa5d8004810 2024-09-16T15:42:29.093 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:42:29.088+0000 7fa5f2e6d640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fa5ec0d1b20 con 0x7fa5ec0cd6e0 2024-09-16T15:42:29.093 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:42:29.088+0000 7fa5f0c09640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa5d8004810 0x7fa5ec0cef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:42:29.093 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:42:29.088+0000 7fa5f140a640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fa5ec0cd6e0 0x7fa5ec0b4ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:42:29.093 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:42:29.088+0000 7fa5f0c09640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa5d8004810 0x7fa5ec0cef60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.184:54056/0 (socket says 172.21.15.184:54056) 2024-09-16T15:42:29.093 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:42:29.088+0000 7fa5f0c09640 1 -- 172.21.15.184:0/4048693527 learned_addr learned my addr 172.21.15.184:0/4048693527 (peer_addr_for_me v2:172.21.15.184:0/0) 2024-09-16T15:42:29.093 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:42:29.088+0000 7fa5ebfff640 1 --2- 172.21.15.184:0/4048693527 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa5ec0b7650 0x7fa5ec14ca40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:42:29.093 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:42:29.088+0000 7fa5f0c09640 1 -- 172.21.15.184:0/4048693527 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fa5ec0cd6e0 msgr2=0x7fa5ec0b4ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:42:29.093 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:42:29.088+0000 7fa5f0c09640 1 --2- 172.21.15.184:0/4048693527 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fa5ec0cd6e0 0x7fa5ec0b4ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:42:29.093 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:42:29.088+0000 7fa5f0c09640 1 -- 172.21.15.184:0/4048693527 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa5ec0b7650 msgr2=0x7fa5ec14ca40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:42:29.093 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:42:29.088+0000 7fa5f0c09640 1 --2- 172.21.15.184:0/4048693527 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa5ec0b7650 0x7fa5ec14ca40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:42:29.093 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:42:29.088+0000 7fa5f0c09640 1 -- 172.21.15.184:0/4048693527 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa5ec0cdec0 con 0x7fa5d8004810 2024-09-16T15:42:29.094 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:42:29.088+0000 7fa5f140a640 1 --2- 172.21.15.184:0/4048693527 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fa5ec0cd6e0 0x7fa5ec0b4ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T15:42:29.094 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:42:29.088+0000 7fa5ebfff640 1 --2- 172.21.15.184:0/4048693527 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa5ec0b7650 0x7fa5ec14ca40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T15:42:29.094 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:42:29.088+0000 7fa5f0c09640 1 --2- 172.21.15.184:0/4048693527 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa5d8004810 0x7fa5ec0cef60 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fa5dc006090 tx=0x7fa5dc006490 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:42:29.094 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:42:29.088+0000 7fa5e97fa640 1 -- 172.21.15.184:0/4048693527 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5dc001a10 con 0x7fa5d8004810 2024-09-16T15:42:29.094 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:42:29.088+0000 7fa5e97fa640 1 -- 172.21.15.184:0/4048693527 <== mon.1 v2:172.21.15.184:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fa5dc016020 con 0x7fa5d8004810 2024-09-16T15:42:29.094 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:42:29.088+0000 7fa5f2e6d640 1 -- 172.21.15.184:0/4048693527 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa5ec0ce1a0 con 0x7fa5d8004810 2024-09-16T15:42:29.094 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:42:29.088+0000 7fa5e97fa640 1 -- 172.21.15.184:0/4048693527 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5dc00fb60 con 0x7fa5d8004810 2024-09-16T15:42:29.094 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:42:29.088+0000 7fa5f2e6d640 1 -- 172.21.15.184:0/4048693527 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa5ec0ce760 con 0x7fa5d8004810 2024-09-16T15:42:29.094 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:42:29.088+0000 7fa5e97fa640 1 -- 172.21.15.184:0/4048693527 <== mon.1 v2:172.21.15.184:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fa5dc0117c0 con 0x7fa5d8004810 2024-09-16T15:42:29.094 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:42:29.088+0000 7fa5e97fa640 1 -- 172.21.15.184:0/4048693527 <== mon.1 v2:172.21.15.184:3300/0 5 ==== osd_map(8..8 src has 1..8) ==== 1415+0+0 (secure 0 0 0) 0x7fa5dc0103c0 con 0x7fa5d8004810 2024-09-16T15:42:29.094 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:42:29.088+0000 7fa5f2e6d640 1 -- 172.21.15.184:0/4048693527 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa5ec15f770 con 0x7fa5d8004810 2024-09-16T15:42:29.095 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:42:29.092+0000 7fa5e97fa640 1 -- 172.21.15.184:0/4048693527 <== mon.1 v2:172.21.15.184:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+75856 (secure 0 0 0) 0x7fa5dc020020 con 0x7fa5d8004810 2024-09-16T15:42:29.179 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:42:29.176+0000 7fa5f2e6d640 1 -- 172.21.15.184:0/4048693527 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_command({"prefix": "osd new", "uuid": "cde6f39e-aa1b-4260-ac19-774ee6aaca72", "id": 6} v 0) -- 0x7fa5ec0d2b30 con 0x7fa5d8004810 2024-09-16T15:42:29.186 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:42:29.184+0000 7fa5e97fa640 1 -- 172.21.15.184:0/4048693527 <== mon.1 v2:172.21.15.184:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "cde6f39e-aa1b-4260-ac19-774ee6aaca72", "id": 6}]=0 v9) ==== 112+0+2 (secure 0 0 0) 0x7fa5dc011ab0 con 0x7fa5d8004810 2024-09-16T15:42:29.187 INFO:teuthology.orchestra.run.smithi184.stdout:6 2024-09-16T15:42:29.188 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:42:29.184+0000 7fa5f2e6d640 1 -- 172.21.15.184:0/4048693527 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa5d8004810 msgr2=0x7fa5ec0cef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:42:29.188 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:42:29.184+0000 7fa5f2e6d640 1 --2- 172.21.15.184:0/4048693527 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa5d8004810 0x7fa5ec0cef60 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fa5dc006090 tx=0x7fa5dc006490 comp rx=0 tx=0).stop 2024-09-16T15:42:29.188 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:42:29.184+0000 7fa5f2e6d640 1 -- 172.21.15.184:0/4048693527 shutdown_connections 2024-09-16T15:42:29.188 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:42:29.184+0000 7fa5f2e6d640 1 --2- 172.21.15.184:0/4048693527 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fa5ec0cd6e0 0x7fa5ec0b4ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:42:29.188 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:42:29.184+0000 7fa5f2e6d640 1 --2- 172.21.15.184:0/4048693527 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa5ec0b7650 0x7fa5ec14ca40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:42:29.188 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:42:29.184+0000 7fa5f2e6d640 1 --2- 172.21.15.184:0/4048693527 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa5d8004810 0x7fa5ec0cef60 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:42:29.188 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:42:29.184+0000 7fa5f2e6d640 1 -- 172.21.15.184:0/4048693527 >> 172.21.15.184:0/4048693527 conn(0x7fa5ec09aa40 msgr2=0x7fa5ec1600d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:42:29.188 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:42:29.184+0000 7fa5f2e6d640 1 -- 172.21.15.184:0/4048693527 shutdown_connections 2024-09-16T15:42:29.189 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:42:29.184+0000 7fa5f2e6d640 1 -- 172.21.15.184:0/4048693527 wait complete. 2024-09-16T15:42:29.204 DEBUG:teuthology.orchestra.run.smithi184:> sudo ceph --cluster ceph osd new 4f381778-eee1-47c6-804f-df0e6b33e313 7 2024-09-16T15:42:29.317 INFO:tasks.ceph.mgr.x.smithi184.stderr:/usr/lib/python3/dist-packages/scipy/__init__.py:67: UserWarning: NumPy was imported from a Python sub-interpreter but NumPy does not properly support sub-interpreters. This will likely work for most users but might cause hard to track down issues or subtle bugs. A common user of the rare sub-interpreter feature is wsgi which also allows single-interpreter mode. 2024-09-16T15:42:29.318 INFO:tasks.ceph.mgr.x.smithi184.stderr:Improvements in the case of bugs are welcome, but is not on the NumPy roadmap, and full support may require significant effort to achieve. 2024-09-16T15:42:29.318 INFO:tasks.ceph.mgr.x.smithi184.stderr: from numpy import show_config as show_numpy_config 2024-09-16T15:42:29.327 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:42:29.324+0000 7fd58b455640 1 -- 172.21.15.184:0/1687767232 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd58415d780 msgr2=0x7fd584072f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:42:29.327 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:42:29.324+0000 7fd58b455640 1 --2- 172.21.15.184:0/1687767232 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd58415d780 0x7fd584072f20 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fd5840b2700 tx=0x7fd58001c830 comp rx=0 tx=0).stop 2024-09-16T15:42:29.327 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:42:29.324+0000 7fd58b455640 1 -- 172.21.15.184:0/1687767232 shutdown_connections 2024-09-16T15:42:29.327 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:42:29.324+0000 7fd58b455640 1 --2- 172.21.15.184:0/1687767232 >> v2:172.21.15.123:3300/0 conn(0x7fd58406eac0 0x7fd5840cbf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:42:29.327 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:42:29.324+0000 7fd58b455640 1 --2- 172.21.15.184:0/1687767232 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd58415d780 0x7fd584072f20 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:42:29.327 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:42:29.324+0000 7fd58b455640 1 -- 172.21.15.184:0/1687767232 >> 172.21.15.184:0/1687767232 conn(0x7fd5840b1710 msgr2=0x7fd5840b1b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:42:29.327 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:42:29.324+0000 7fd58b455640 1 -- 172.21.15.184:0/1687767232 shutdown_connections 2024-09-16T15:42:29.327 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:42:29.324+0000 7fd58b455640 1 -- 172.21.15.184:0/1687767232 wait complete. 2024-09-16T15:42:29.328 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:42:29.324+0000 7fd58b455640 1 Processor -- start 2024-09-16T15:42:29.328 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:42:29.324+0000 7fd58b455640 1 -- start start 2024-09-16T15:42:29.330 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:42:29.324+0000 7fd58b455640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd5840cbaf0 0x7fd58413ca10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:42:29.330 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:42:29.324+0000 7fd58b455640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd58415d780 0x7fd5840c31f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:42:29.330 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:42:29.324+0000 7fd58b455640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fd58414fb30 0x7fd5840c3730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:42:29.330 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:42:29.324+0000 7fd58b455640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fd584167490 con 0x7fd58415d780 2024-09-16T15:42:29.330 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:42:29.324+0000 7fd58b455640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fd584167310 con 0x7fd5840cbaf0 2024-09-16T15:42:29.330 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:42:29.324+0000 7fd58b455640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fd584167610 con 0x7fd58414fb30 2024-09-16T15:42:29.330 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:42:29.324+0000 7fd5891f1640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd5840cbaf0 0x7fd58413ca10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:42:29.331 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:42:29.324+0000 7fd5899f2640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fd58414fb30 0x7fd5840c3730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:42:29.331 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:42:29.324+0000 7fd5891f1640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd5840cbaf0 0x7fd58413ca10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.184:54076/0 (socket says 172.21.15.184:54076) 2024-09-16T15:42:29.331 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:42:29.324+0000 7fd5891f1640 1 -- 172.21.15.184:0/3466830498 learned_addr learned my addr 172.21.15.184:0/3466830498 (peer_addr_for_me v2:172.21.15.184:0/0) 2024-09-16T15:42:29.331 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:42:29.324+0000 7fd5889f0640 1 --2- 172.21.15.184:0/3466830498 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd58415d780 0x7fd5840c31f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:42:29.331 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:42:29.324+0000 7fd5891f1640 1 -- 172.21.15.184:0/3466830498 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fd58414fb30 msgr2=0x7fd5840c3730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:42:29.331 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:42:29.324+0000 7fd5891f1640 1 --2- 172.21.15.184:0/3466830498 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fd58414fb30 0x7fd5840c3730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:42:29.331 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:42:29.324+0000 7fd5891f1640 1 -- 172.21.15.184:0/3466830498 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd58415d780 msgr2=0x7fd5840c31f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:42:29.331 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:42:29.324+0000 7fd5891f1640 1 --2- 172.21.15.184:0/3466830498 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd58415d780 0x7fd5840c31f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:42:29.331 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:42:29.324+0000 7fd5891f1640 1 -- 172.21.15.184:0/3466830498 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd584153f60 con 0x7fd5840cbaf0 2024-09-16T15:42:29.331 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:42:29.324+0000 7fd5899f2640 1 --2- 172.21.15.184:0/3466830498 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fd58414fb30 0x7fd5840c3730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T15:42:29.331 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:42:29.324+0000 7fd5891f1640 1 --2- 172.21.15.184:0/3466830498 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd5840cbaf0 0x7fd58413ca10 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fd5800b0040 tx=0x7fd5800040e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:42:29.331 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:42:29.324+0000 7fd579ffb640 1 -- 172.21.15.184:0/3466830498 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5800042d0 con 0x7fd5840cbaf0 2024-09-16T15:42:29.331 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:42:29.324+0000 7fd579ffb640 1 -- 172.21.15.184:0/3466830498 <== mon.1 v2:172.21.15.184:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fd580005030 con 0x7fd5840cbaf0 2024-09-16T15:42:29.331 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:42:29.324+0000 7fd58b455640 1 -- 172.21.15.184:0/3466830498 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd584154240 con 0x7fd5840cbaf0 2024-09-16T15:42:29.332 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:42:29.324+0000 7fd579ffb640 1 -- 172.21.15.184:0/3466830498 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd580005420 con 0x7fd5840cbaf0 2024-09-16T15:42:29.332 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:42:29.324+0000 7fd58b455640 1 -- 172.21.15.184:0/3466830498 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd58413cf50 con 0x7fd5840cbaf0 2024-09-16T15:42:29.332 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:42:29.324+0000 7fd579ffb640 1 -- 172.21.15.184:0/3466830498 <== mon.1 v2:172.21.15.184:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fd5800bf030 con 0x7fd5840cbaf0 2024-09-16T15:42:29.332 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:42:29.324+0000 7fd579ffb640 1 -- 172.21.15.184:0/3466830498 <== mon.1 v2:172.21.15.184:3300/0 5 ==== osd_map(9..9 src has 1..9) ==== 1530+0+0 (secure 0 0 0) 0x7fd5800b7480 con 0x7fd5840cbaf0 2024-09-16T15:42:29.332 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:42:29.328+0000 7fd58b455640 1 -- 172.21.15.184:0/3466830498 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd5840cbf30 con 0x7fd5840cbaf0 2024-09-16T15:42:29.332 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:42:29.328+0000 7fd579ffb640 1 -- 172.21.15.184:0/3466830498 <== mon.1 v2:172.21.15.184:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+75856 (secure 0 0 0) 0x7fd5800c6080 con 0x7fd5840cbaf0 2024-09-16T15:42:29.420 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:42:29.416+0000 7fd58b455640 1 -- 172.21.15.184:0/3466830498 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_command({"prefix": "osd new", "uuid": "4f381778-eee1-47c6-804f-df0e6b33e313", "id": 7} v 0) -- 0x7fd5840cddc0 con 0x7fd5840cbaf0 2024-09-16T15:42:29.426 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:42:29.424+0000 7fd579ffb640 1 -- 172.21.15.184:0/3466830498 <== mon.1 v2:172.21.15.184:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "4f381778-eee1-47c6-804f-df0e6b33e313", "id": 7}]=0 v10) ==== 112+0+2 (secure 0 0 0) 0x7fd5800164b0 con 0x7fd5840cbaf0 2024-09-16T15:42:29.426 INFO:teuthology.orchestra.run.smithi184.stdout:7 2024-09-16T15:42:29.427 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:42:29.424+0000 7fd58b455640 1 -- 172.21.15.184:0/3466830498 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd5840cbaf0 msgr2=0x7fd58413ca10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:42:29.427 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:42:29.424+0000 7fd58b455640 1 --2- 172.21.15.184:0/3466830498 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd5840cbaf0 0x7fd58413ca10 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fd5800b0040 tx=0x7fd5800040e0 comp rx=0 tx=0).stop 2024-09-16T15:42:29.427 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:42:29.424+0000 7fd58b455640 1 -- 172.21.15.184:0/3466830498 shutdown_connections 2024-09-16T15:42:29.428 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:42:29.424+0000 7fd58b455640 1 --2- 172.21.15.184:0/3466830498 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fd58414fb30 0x7fd5840c3730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:42:29.428 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:42:29.424+0000 7fd58b455640 1 --2- 172.21.15.184:0/3466830498 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd58415d780 0x7fd5840c31f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:42:29.428 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:42:29.424+0000 7fd58b455640 1 --2- 172.21.15.184:0/3466830498 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd5840cbaf0 0x7fd58413ca10 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:42:29.428 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:42:29.424+0000 7fd58b455640 1 -- 172.21.15.184:0/3466830498 >> 172.21.15.184:0/3466830498 conn(0x7fd5840b1710 msgr2=0x7fd5840ce2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:42:29.428 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:42:29.424+0000 7fd58b455640 1 -- 172.21.15.184:0/3466830498 shutdown_connections 2024-09-16T15:42:29.428 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:42:29.424+0000 7fd58b455640 1 -- 172.21.15.184:0/3466830498 wait complete. 2024-09-16T15:42:29.428 INFO:tasks.ceph.mgr.y.smithi123.stderr:/usr/lib/python3/dist-packages/scipy/__init__.py:67: UserWarning: NumPy was imported from a Python sub-interpreter but NumPy does not properly support sub-interpreters. This will likely work for most users but might cause hard to track down issues or subtle bugs. A common user of the rare sub-interpreter feature is wsgi which also allows single-interpreter mode. 2024-09-16T15:42:29.428 INFO:tasks.ceph.mgr.y.smithi123.stderr:Improvements in the case of bugs are welcome, but is not on the NumPy roadmap, and full support may require significant effort to achieve. 2024-09-16T15:42:29.429 INFO:tasks.ceph.mgr.y.smithi123.stderr: from numpy import show_config as show_numpy_config 2024-09-16T15:42:29.442 INFO:tasks.ceph.osd.0:Restarting daemon 2024-09-16T15:42:29.442 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2024-09-16T15:42:29.445 INFO:tasks.ceph.osd.0:Started 2024-09-16T15:42:29.445 INFO:tasks.ceph.osd.1:Restarting daemon 2024-09-16T15:42:29.445 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2024-09-16T15:42:29.447 INFO:tasks.ceph.osd.1:Started 2024-09-16T15:42:29.447 INFO:tasks.ceph.osd.2:Restarting daemon 2024-09-16T15:42:29.448 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2024-09-16T15:42:29.450 INFO:tasks.ceph.osd.2:Started 2024-09-16T15:42:29.450 INFO:tasks.ceph.osd.3:Restarting daemon 2024-09-16T15:42:29.450 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2024-09-16T15:42:29.452 INFO:tasks.ceph.osd.3:Started 2024-09-16T15:42:29.452 INFO:tasks.ceph.osd.4:Restarting daemon 2024-09-16T15:42:29.452 DEBUG:teuthology.orchestra.run.smithi184:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 4 2024-09-16T15:42:29.454 INFO:tasks.ceph.osd.4:Started 2024-09-16T15:42:29.454 INFO:tasks.ceph.osd.5:Restarting daemon 2024-09-16T15:42:29.454 DEBUG:teuthology.orchestra.run.smithi184:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 5 2024-09-16T15:42:29.456 INFO:tasks.ceph.osd.5:Started 2024-09-16T15:42:29.457 INFO:tasks.ceph.osd.6:Restarting daemon 2024-09-16T15:42:29.457 DEBUG:teuthology.orchestra.run.smithi184:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 6 2024-09-16T15:42:29.459 INFO:tasks.ceph.osd.6:Started 2024-09-16T15:42:29.459 INFO:tasks.ceph.osd.7:Restarting daemon 2024-09-16T15:42:29.459 DEBUG:teuthology.orchestra.run.smithi184:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 7 2024-09-16T15:42:29.461 INFO:tasks.ceph.osd.7:Started 2024-09-16T15:42:29.461 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-16T15:42:29.610 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:29.603+0000 7f1ee263e640 1 -- 172.21.15.123:0/2749543025 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1ec4004d50 msgr2=0x7f1ec40051d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:42:29.610 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:29.603+0000 7f1ee263e640 1 --2- 172.21.15.123:0/2749543025 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1ec4004d50 0x7f1ec40051d0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f1ecc009eb0 tx=0x7f1ecc01d5b0 comp rx=0 tx=0).stop 2024-09-16T15:42:29.610 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:29.603+0000 7f1ee0bdb640 1 -- 172.21.15.123:0/2749543025 reap_dead start 2024-09-16T15:42:29.610 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:29.603+0000 7f1ee263e640 1 -- 172.21.15.123:0/2749543025 shutdown_connections 2024-09-16T15:42:29.610 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:29.603+0000 7f1ee263e640 1 -- 172.21.15.123:0/2749543025 >> 172.21.15.123:0/2749543025 conn(0x7f1edc09aa40 msgr2=0x7f1edc09ae50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:42:29.610 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:29.603+0000 7f1ee263e640 1 -- 172.21.15.123:0/2749543025 shutdown_connections 2024-09-16T15:42:29.610 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:29.603+0000 7f1ee263e640 1 -- 172.21.15.123:0/2749543025 wait complete. 2024-09-16T15:42:29.611 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:29.607+0000 7f1ee263e640 1 Processor -- start 2024-09-16T15:42:29.611 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:29.607+0000 7f1ee263e640 1 -- start start 2024-09-16T15:42:29.613 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:29.607+0000 7f1ee263e640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f1ec4004d50 0x7f1edc1f2550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:42:29.613 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:29.607+0000 7f1ee263e640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1edc1f5440 0x7f1edc0cded0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:42:29.613 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:29.607+0000 7f1ee263e640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1edc1f3710 0x7f1edc0aeb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:42:29.613 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:29.607+0000 7f1ee263e640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f1edc1649b0 con 0x7f1edc1f5440 2024-09-16T15:42:29.613 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:29.607+0000 7f1ee263e640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f1edc164830 con 0x7f1edc1f3710 2024-09-16T15:42:29.613 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:29.607+0000 7f1ee263e640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f1edc164b30 con 0x7f1ec4004d50 2024-09-16T15:42:29.613 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:29.607+0000 7f1edb7fe640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1edc1f5440 0x7f1edc0cded0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:42:29.614 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:29.607+0000 7f1edb7fe640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1edc1f5440 0x7f1edc0cded0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:49786/0 (socket says 172.21.15.123:49786) 2024-09-16T15:42:29.614 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:29.607+0000 7f1edb7fe640 1 -- 172.21.15.123:0/2397467090 learned_addr learned my addr 172.21.15.123:0/2397467090 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:42:29.614 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:29.607+0000 7f1edbfff640 1 --2- 172.21.15.123:0/2397467090 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f1ec4004d50 0x7f1edc1f2550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:42:29.614 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:29.607+0000 7f1ee0bdb640 1 --2- 172.21.15.123:0/2397467090 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1edc1f3710 0x7f1edc0aeb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:42:29.614 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:29.607+0000 7f1edb7fe640 1 -- 172.21.15.123:0/2397467090 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f1ec4004d50 msgr2=0x7f1edc1f2550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:42:29.614 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:29.607+0000 7f1edb7fe640 1 --2- 172.21.15.123:0/2397467090 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f1ec4004d50 0x7f1edc1f2550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:42:29.614 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:29.607+0000 7f1edb7fe640 1 -- 172.21.15.123:0/2397467090 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1edc1f3710 msgr2=0x7f1edc0aeb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:42:29.614 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:29.607+0000 7f1edb7fe640 1 --2- 172.21.15.123:0/2397467090 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1edc1f3710 0x7f1edc0aeb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:42:29.614 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:29.607+0000 7f1edb7fe640 1 -- 172.21.15.123:0/2397467090 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1edc1f58c0 con 0x7f1edc1f5440 2024-09-16T15:42:29.614 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:29.607+0000 7f1ee0bdb640 1 --2- 172.21.15.123:0/2397467090 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1edc1f3710 0x7f1edc0aeb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T15:42:29.614 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:29.607+0000 7f1edb7fe640 1 --2- 172.21.15.123:0/2397467090 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1edc1f5440 0x7f1edc0cded0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f1ed000e360 tx=0x7f1ed000e820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:42:29.614 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:29.607+0000 7f1ed8ff9640 1 -- 172.21.15.123:0/2397467090 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ed0012850 con 0x7f1edc1f5440 2024-09-16T15:42:29.615 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:29.607+0000 7f1ed8ff9640 1 -- 172.21.15.123:0/2397467090 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f1ed0012ae0 con 0x7f1edc1f5440 2024-09-16T15:42:29.615 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:29.607+0000 7f1ee263e640 1 -- 172.21.15.123:0/2397467090 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1edc1f2b60 con 0x7f1edc1f5440 2024-09-16T15:42:29.615 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:29.607+0000 7f1ed8ff9640 1 -- 172.21.15.123:0/2397467090 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ed001a400 con 0x7f1edc1f5440 2024-09-16T15:42:29.615 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:29.607+0000 7f1ee263e640 1 -- 172.21.15.123:0/2397467090 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1edc1f3120 con 0x7f1edc1f5440 2024-09-16T15:42:29.615 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:29.607+0000 7f1ed8ff9640 1 -- 172.21.15.123:0/2397467090 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f1ed000d030 con 0x7f1edc1f5440 2024-09-16T15:42:29.615 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:29.607+0000 7f1ed8ff9640 1 -- 172.21.15.123:0/2397467090 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(10..10 src has 1..10) ==== 1645+0+0 (secure 0 0 0) 0x7f1ed0011aa0 con 0x7f1edc1f5440 2024-09-16T15:42:29.615 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:29.607+0000 7f1ee263e640 1 -- 172.21.15.123:0/2397467090 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1edc0d1b50 con 0x7f1edc1f5440 2024-09-16T15:42:29.617 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:29.611+0000 7f1ed8ff9640 1 -- 172.21.15.123:0/2397467090 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+75856 (secure 0 0 0) 0x7f1ed0016030 con 0x7f1edc1f5440 2024-09-16T15:42:29.678 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:29.675+0000 7f1ee263e640 1 -- 172.21.15.123:0/2397467090 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f1edc0b0da0 con 0x7f1edc1f5440 2024-09-16T15:42:29.679 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:29.675+0000 7f1ed8ff9640 1 -- 172.21.15.123:0/2397467090 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) ==== 74+0+7058 (secure 0 0 0) 0x7f1ed00113d0 con 0x7f1edc1f5440 2024-09-16T15:42:29.679 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-09-16T15:42:29.679 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":10,"fsid":"56077f89-18c8-486f-ae7e-eecc0bb7822c","created":"2024-09-16T15:42:26.364892+0000","modified":"2024-09-16T15:42:29.423662+0000","last_up_change":"0.000000","last_in_change":"2024-09-16T15:42:29.423662+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"9e7c1955-a607-4671-9be7-f2ef760ceeaa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"c540cfde-0977-4742-9b17-fd634668e010","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"9864a496-f169-486b-95c4-1d59162793a7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"c6d1f817-f074-4325-95e1-578eba13edda","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"200a8df4-50ef-40a5-b48a-58981269c086","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"32d5a900-da3d-412c-bc19-13861a2368d8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"cde6f39e-aa1b-4260-ac19-774ee6aaca72","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4f381778-eee1-47c6-804f-df0e6b33e313","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T15:42:29.680 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:29.675+0000 7f1ee263e640 1 -- 172.21.15.123:0/2397467090 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1edc1f5440 msgr2=0x7f1edc0cded0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:42:29.680 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:29.675+0000 7f1ee263e640 1 --2- 172.21.15.123:0/2397467090 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1edc1f5440 0x7f1edc0cded0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f1ed000e360 tx=0x7f1ed000e820 comp rx=0 tx=0).stop 2024-09-16T15:42:29.680 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:29.675+0000 7f1ee263e640 1 -- 172.21.15.123:0/2397467090 shutdown_connections 2024-09-16T15:42:29.680 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:29.675+0000 7f1ee263e640 1 --2- 172.21.15.123:0/2397467090 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1edc1f3710 0x7f1edc0aeb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:42:29.680 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:29.675+0000 7f1ee263e640 1 --2- 172.21.15.123:0/2397467090 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1edc1f5440 0x7f1edc0cded0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:42:29.681 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:29.675+0000 7f1ee263e640 1 --2- 172.21.15.123:0/2397467090 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f1ec4004d50 0x7f1edc1f2550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:42:29.681 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:29.675+0000 7f1ee263e640 1 -- 172.21.15.123:0/2397467090 >> 172.21.15.123:0/2397467090 conn(0x7f1edc09aa40 msgr2=0x7f1edc156670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:42:29.681 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:29.675+0000 7f1ee263e640 1 -- 172.21.15.123:0/2397467090 shutdown_connections 2024-09-16T15:42:29.681 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:29.675+0000 7f1ee263e640 1 -- 172.21.15.123:0/2397467090 wait complete. 2024-09-16T15:42:29.697 INFO:tasks.ceph.ceph_manager.ceph:[] 2024-09-16T15:42:29.697 INFO:tasks.ceph:Waiting for OSDs to come up 2024-09-16T15:42:30.504 DEBUG:teuthology.orchestra.run.smithi123:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-09-16T15:42:30.623 INFO:teuthology.misc.health.smithi123.stderr:2024-09-16T15:42:30.619+0000 7f62c5b24640 1 -- 172.21.15.123:0/562262211 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f62a8004d50 msgr2=0x7f62a80051d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:42:30.623 INFO:teuthology.misc.health.smithi123.stderr:2024-09-16T15:42:30.619+0000 7f62c5b24640 1 --2- 172.21.15.123:0/562262211 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f62a8004d50 0x7f62a80051d0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f62b000c660 tx=0x7f62b001d2e0 comp rx=0 tx=0).stop 2024-09-16T15:42:30.623 INFO:teuthology.misc.health.smithi123.stderr:2024-09-16T15:42:30.619+0000 7f62bffff640 1 -- 172.21.15.123:0/562262211 reap_dead start 2024-09-16T15:42:30.623 INFO:teuthology.misc.health.smithi123.stderr:2024-09-16T15:42:30.619+0000 7f62c5b24640 1 -- 172.21.15.123:0/562262211 shutdown_connections 2024-09-16T15:42:30.623 INFO:teuthology.misc.health.smithi123.stderr:2024-09-16T15:42:30.619+0000 7f62c5b24640 1 -- 172.21.15.123:0/562262211 >> 172.21.15.123:0/562262211 conn(0x7f62c009a9f0 msgr2=0x7f62c009ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:42:30.623 INFO:teuthology.misc.health.smithi123.stderr:2024-09-16T15:42:30.619+0000 7f62c5b24640 1 -- 172.21.15.123:0/562262211 shutdown_connections 2024-09-16T15:42:30.623 INFO:teuthology.misc.health.smithi123.stderr:2024-09-16T15:42:30.619+0000 7f62c5b24640 1 -- 172.21.15.123:0/562262211 wait complete. 2024-09-16T15:42:30.626 INFO:teuthology.misc.health.smithi123.stderr:2024-09-16T15:42:30.619+0000 7f62c5b24640 1 Processor -- start 2024-09-16T15:42:30.626 INFO:teuthology.misc.health.smithi123.stderr:2024-09-16T15:42:30.619+0000 7f62c5b24640 1 -- start start 2024-09-16T15:42:30.626 INFO:teuthology.misc.health.smithi123.stderr:2024-09-16T15:42:30.619+0000 7f62c5b24640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f62c015dc80 0x7f62c00c7ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:42:30.626 INFO:teuthology.misc.health.smithi123.stderr:2024-09-16T15:42:30.619+0000 7f62c5b24640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f62c01f59f0 0x7f62c01f2ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:42:30.626 INFO:teuthology.misc.health.smithi123.stderr:2024-09-16T15:42:30.619+0000 7f62c5b24640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f62c01f30e0 0x7f62c00ccb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:42:30.626 INFO:teuthology.misc.health.smithi123.stderr:2024-09-16T15:42:30.619+0000 7f62c5b24640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f62c00b0110 con 0x7f62c01f30e0 2024-09-16T15:42:30.626 INFO:teuthology.misc.health.smithi123.stderr:2024-09-16T15:42:30.619+0000 7f62c5b24640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f62c00d2bd0 con 0x7f62c015dc80 2024-09-16T15:42:30.626 INFO:teuthology.misc.health.smithi123.stderr:2024-09-16T15:42:30.619+0000 7f62c5b24640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f62c00b0290 con 0x7f62c01f59f0 2024-09-16T15:42:30.626 INFO:teuthology.misc.health.smithi123.stderr:2024-09-16T15:42:30.619+0000 7f62bffff640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f62c01f30e0 0x7f62c00ccb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:42:30.626 INFO:teuthology.misc.health.smithi123.stderr:2024-09-16T15:42:30.619+0000 7f62bffff640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f62c01f30e0 0x7f62c00ccb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:49802/0 (socket says 172.21.15.123:49802) 2024-09-16T15:42:30.626 INFO:teuthology.misc.health.smithi123.stderr:2024-09-16T15:42:30.619+0000 7f62bffff640 1 -- 172.21.15.123:0/1169720470 learned_addr learned my addr 172.21.15.123:0/1169720470 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:42:30.627 INFO:teuthology.misc.health.smithi123.stderr:2024-09-16T15:42:30.619+0000 7f62beffd640 1 --2- 172.21.15.123:0/1169720470 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f62c01f59f0 0x7f62c01f2ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:42:30.627 INFO:teuthology.misc.health.smithi123.stderr:2024-09-16T15:42:30.619+0000 7f62bf7fe640 1 --2- 172.21.15.123:0/1169720470 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f62c015dc80 0x7f62c00c7ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:42:30.627 INFO:teuthology.misc.health.smithi123.stderr:2024-09-16T15:42:30.619+0000 7f62bffff640 1 -- 172.21.15.123:0/1169720470 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f62c01f59f0 msgr2=0x7f62c01f2ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:42:30.627 INFO:teuthology.misc.health.smithi123.stderr:2024-09-16T15:42:30.619+0000 7f62bffff640 1 --2- 172.21.15.123:0/1169720470 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f62c01f59f0 0x7f62c01f2ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:42:30.627 INFO:teuthology.misc.health.smithi123.stderr:2024-09-16T15:42:30.619+0000 7f62bffff640 1 -- 172.21.15.123:0/1169720470 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f62c015dc80 msgr2=0x7f62c00c7ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:42:30.627 INFO:teuthology.misc.health.smithi123.stderr:2024-09-16T15:42:30.619+0000 7f62bffff640 1 --2- 172.21.15.123:0/1169720470 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f62c015dc80 0x7f62c00c7ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:42:30.627 INFO:teuthology.misc.health.smithi123.stderr:2024-09-16T15:42:30.619+0000 7f62bffff640 1 -- 172.21.15.123:0/1169720470 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f62c01f2400 con 0x7f62c01f30e0 2024-09-16T15:42:30.627 INFO:teuthology.misc.health.smithi123.stderr:2024-09-16T15:42:30.619+0000 7f62bf7fe640 1 --2- 172.21.15.123:0/1169720470 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f62c015dc80 0x7f62c00c7ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T15:42:30.627 INFO:teuthology.misc.health.smithi123.stderr:2024-09-16T15:42:30.619+0000 7f62bffff640 1 --2- 172.21.15.123:0/1169720470 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f62c01f30e0 0x7f62c00ccb20 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f62b001cda0 tx=0x7f62b00012c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:42:30.627 INFO:teuthology.misc.health.smithi123.stderr:2024-09-16T15:42:30.619+0000 7f62a7fff640 1 -- 172.21.15.123:0/1169720470 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62b000c280 con 0x7f62c01f30e0 2024-09-16T15:42:30.627 INFO:teuthology.misc.health.smithi123.stderr:2024-09-16T15:42:30.619+0000 7f62a7fff640 1 -- 172.21.15.123:0/1169720470 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f62b00063b0 con 0x7f62c01f30e0 2024-09-16T15:42:30.628 INFO:teuthology.misc.health.smithi123.stderr:2024-09-16T15:42:30.619+0000 7f62a7fff640 1 -- 172.21.15.123:0/1169720470 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62b0002090 con 0x7f62c01f30e0 2024-09-16T15:42:30.628 INFO:teuthology.misc.health.smithi123.stderr:2024-09-16T15:42:30.619+0000 7f62c5b24640 1 -- 172.21.15.123:0/1169720470 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f62c01f5e70 con 0x7f62c01f30e0 2024-09-16T15:42:30.628 INFO:teuthology.misc.health.smithi123.stderr:2024-09-16T15:42:30.619+0000 7f62c5b24640 1 -- 172.21.15.123:0/1169720470 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f62c01f6400 con 0x7f62c01f30e0 2024-09-16T15:42:30.628 INFO:teuthology.misc.health.smithi123.stderr:2024-09-16T15:42:30.619+0000 7f62a7fff640 1 -- 172.21.15.123:0/1169720470 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f62b00ba810 con 0x7f62c01f30e0 2024-09-16T15:42:30.628 INFO:teuthology.misc.health.smithi123.stderr:2024-09-16T15:42:30.619+0000 7f62a7fff640 1 -- 172.21.15.123:0/1169720470 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(10..10 src has 1..10) ==== 1645+0+0 (secure 0 0 0) 0x7f62b00aeb40 con 0x7f62c01f30e0 2024-09-16T15:42:30.628 INFO:teuthology.misc.health.smithi123.stderr:2024-09-16T15:42:30.623+0000 7f62c5b24640 1 -- 172.21.15.123:0/1169720470 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f62c00b07f0 con 0x7f62c01f30e0 2024-09-16T15:42:30.628 INFO:teuthology.misc.health.smithi123.stderr:2024-09-16T15:42:30.623+0000 7f62a7fff640 1 -- 172.21.15.123:0/1169720470 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+75856 (secure 0 0 0) 0x7f62c01f2400 con 0x7f62c01f30e0 2024-09-16T15:42:30.687 INFO:teuthology.misc.health.smithi123.stderr:2024-09-16T15:42:30.683+0000 7f62c5b24640 1 -- 172.21.15.123:0/1169720470 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f62c006d750 con 0x7f62c01f30e0 2024-09-16T15:42:30.688 INFO:teuthology.misc.health.smithi123.stderr:2024-09-16T15:42:30.683+0000 7f62a7fff640 1 -- 172.21.15.123:0/1169720470 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) ==== 74+0+7058 (secure 0 0 0) 0x7f62b0017090 con 0x7f62c01f30e0 2024-09-16T15:42:30.688 INFO:teuthology.misc.health.smithi123.stdout: 2024-09-16T15:42:30.689 INFO:teuthology.misc.health.smithi123.stdout:{"epoch":10,"fsid":"56077f89-18c8-486f-ae7e-eecc0bb7822c","created":"2024-09-16T15:42:26.364892+0000","modified":"2024-09-16T15:42:29.423662+0000","last_up_change":"0.000000","last_in_change":"2024-09-16T15:42:29.423662+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"9e7c1955-a607-4671-9be7-f2ef760ceeaa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"c540cfde-0977-4742-9b17-fd634668e010","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"9864a496-f169-486b-95c4-1d59162793a7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"c6d1f817-f074-4325-95e1-578eba13edda","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"200a8df4-50ef-40a5-b48a-58981269c086","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"32d5a900-da3d-412c-bc19-13861a2368d8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"cde6f39e-aa1b-4260-ac19-774ee6aaca72","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4f381778-eee1-47c6-804f-df0e6b33e313","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T15:42:30.690 INFO:teuthology.misc.health.smithi123.stderr:2024-09-16T15:42:30.683+0000 7f62c5b24640 1 -- 172.21.15.123:0/1169720470 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f62c01f30e0 msgr2=0x7f62c00ccb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:42:30.690 INFO:teuthology.misc.health.smithi123.stderr:2024-09-16T15:42:30.683+0000 7f62c5b24640 1 --2- 172.21.15.123:0/1169720470 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f62c01f30e0 0x7f62c00ccb20 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f62b001cda0 tx=0x7f62b00012c0 comp rx=0 tx=0).stop 2024-09-16T15:42:30.690 INFO:teuthology.misc.health.smithi123.stderr:2024-09-16T15:42:30.683+0000 7f62c5b24640 1 -- 172.21.15.123:0/1169720470 shutdown_connections 2024-09-16T15:42:30.690 INFO:teuthology.misc.health.smithi123.stderr:2024-09-16T15:42:30.683+0000 7f62c5b24640 1 --2- 172.21.15.123:0/1169720470 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f62c01f30e0 0x7f62c00ccb20 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:42:30.690 INFO:teuthology.misc.health.smithi123.stderr:2024-09-16T15:42:30.683+0000 7f62c5b24640 1 --2- 172.21.15.123:0/1169720470 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f62c01f59f0 0x7f62c01f2ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:42:30.690 INFO:teuthology.misc.health.smithi123.stderr:2024-09-16T15:42:30.683+0000 7f62c5b24640 1 --2- 172.21.15.123:0/1169720470 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f62c015dc80 0x7f62c00c7ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:42:30.690 INFO:teuthology.misc.health.smithi123.stderr:2024-09-16T15:42:30.683+0000 7f62c5b24640 1 -- 172.21.15.123:0/1169720470 >> 172.21.15.123:0/1169720470 conn(0x7f62c009a9f0 msgr2=0x7f62c0071c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:42:30.690 INFO:teuthology.misc.health.smithi123.stderr:2024-09-16T15:42:30.687+0000 7f62c5b24640 1 -- 172.21.15.123:0/1169720470 shutdown_connections 2024-09-16T15:42:30.691 INFO:teuthology.misc.health.smithi123.stderr:2024-09-16T15:42:30.687+0000 7f62c5b24640 1 -- 172.21.15.123:0/1169720470 wait complete. 2024-09-16T15:42:30.705 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-09-16T15:42:30.784 INFO:tasks.ceph.osd.7.smithi184.stderr:2024-09-16T15:42:30.780+0000 7fe53ea4d7c0 -1 Falling back to public interface 2024-09-16T15:42:30.856 INFO:tasks.ceph.osd.5.smithi184.stderr:2024-09-16T15:42:30.852+0000 7f4f156b27c0 -1 Falling back to public interface 2024-09-16T15:42:30.878 INFO:tasks.ceph.osd.0.smithi123.stderr:2024-09-16T15:42:30.875+0000 7f3e04cd37c0 -1 Falling back to public interface 2024-09-16T15:42:31.071 INFO:tasks.ceph.osd.3.smithi123.stderr:2024-09-16T15:42:31.067+0000 7f8ab69ff7c0 -1 Falling back to public interface 2024-09-16T15:42:31.127 INFO:tasks.ceph.osd.4.smithi184.stderr:2024-09-16T15:42:31.124+0000 7f948d41b7c0 -1 Falling back to public interface 2024-09-16T15:42:31.143 INFO:tasks.ceph.osd.6.smithi184.stderr:2024-09-16T15:42:31.140+0000 7f99211a47c0 -1 Falling back to public interface 2024-09-16T15:42:31.158 INFO:tasks.ceph.osd.1.smithi123.stderr:2024-09-16T15:42:31.155+0000 7f811d4d47c0 -1 Falling back to public interface 2024-09-16T15:42:31.174 INFO:tasks.ceph.osd.2.smithi123.stderr:2024-09-16T15:42:31.171+0000 7fdb1f04e7c0 -1 Falling back to public interface 2024-09-16T15:42:32.255 INFO:tasks.ceph.mgr.x.smithi184.stderr:2024-09-16T15:42:32.252+0000 7fda8ca18640 -1 mgr.server handle_report got status from non-daemon mon.c 2024-09-16T15:42:32.256 INFO:tasks.ceph.mgr.x.smithi184.stderr:2024-09-16T15:42:32.252+0000 7fda8ca18640 -1 mgr.server handle_report got status from non-daemon mon.b 2024-09-16T15:42:34.714 INFO:tasks.ceph.osd.7.smithi184.stderr:2024-09-16T15:42:34.707+0000 7fe53ea4d7c0 -1 osd.7 0 log_to_monitors true 2024-09-16T15:42:35.000 INFO:tasks.ceph.osd.0.smithi123.stderr:2024-09-16T15:42:34.999+0000 7f3e04cd37c0 -1 osd.0 0 log_to_monitors true 2024-09-16T15:42:35.128 INFO:tasks.ceph.osd.5.smithi184.stderr:2024-09-16T15:42:35.123+0000 7f4f156b27c0 -1 osd.5 0 log_to_monitors true 2024-09-16T15:42:35.198 INFO:tasks.ceph.osd.1.smithi123.stderr:2024-09-16T15:42:35.199+0000 7f811d4d47c0 -1 osd.1 0 log_to_monitors true 2024-09-16T15:42:35.228 INFO:tasks.ceph.osd.2.smithi123.stderr:2024-09-16T15:42:35.227+0000 7fdb1f04e7c0 -1 osd.2 0 log_to_monitors true 2024-09-16T15:42:35.335 INFO:tasks.ceph.osd.3.smithi123.stderr:2024-09-16T15:42:35.335+0000 7f8ab69ff7c0 -1 osd.3 0 log_to_monitors true 2024-09-16T15:42:35.364 INFO:tasks.ceph.osd.4.smithi184.stderr:2024-09-16T15:42:35.359+0000 7f948d41b7c0 -1 osd.4 0 log_to_monitors true 2024-09-16T15:42:35.727 INFO:tasks.ceph.osd.6.smithi184.stderr:2024-09-16T15:42:35.723+0000 7f99211a47c0 -1 osd.6 0 log_to_monitors true 2024-09-16T15:42:37.510 DEBUG:teuthology.orchestra.run.smithi123:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-09-16T15:42:37.685 INFO:teuthology.misc.health.smithi123.stderr:2024-09-16T15:42:37.683+0000 7f6dcd2bd640 1 -- 172.21.15.123:0/3417905277 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6dc81518d0 msgr2=0x7f6dc815ecb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:42:37.685 INFO:teuthology.misc.health.smithi123.stderr:2024-09-16T15:42:37.683+0000 7f6dcd2bd640 1 --2- 172.21.15.123:0/3417905277 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6dc81518d0 0x7f6dc815ecb0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f6db8009900 tx=0x7f6db801c820 comp rx=0 tx=0).stop 2024-09-16T15:42:37.685 INFO:teuthology.misc.health.smithi123.stderr:2024-09-16T15:42:37.683+0000 7f6dcd2bd640 1 -- 172.21.15.123:0/3417905277 shutdown_connections 2024-09-16T15:42:37.685 INFO:teuthology.misc.health.smithi123.stderr:2024-09-16T15:42:37.683+0000 7f6dcd2bd640 1 --2- 172.21.15.123:0/3417905277 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f6dc8073f10 0x7f6dc806fc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:42:37.685 INFO:teuthology.misc.health.smithi123.stderr:2024-09-16T15:42:37.683+0000 7f6dcd2bd640 1 --2- 172.21.15.123:0/3417905277 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6dc81518d0 0x7f6dc815ecb0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:42:37.685 INFO:teuthology.misc.health.smithi123.stderr:2024-09-16T15:42:37.683+0000 7f6dcd2bd640 1 -- 172.21.15.123:0/3417905277 >> 172.21.15.123:0/3417905277 conn(0x7f6dc806dd30 msgr2=0x7f6dc806e140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:42:37.686 INFO:teuthology.misc.health.smithi123.stderr:2024-09-16T15:42:37.683+0000 7f6dcd2bd640 1 -- 172.21.15.123:0/3417905277 shutdown_connections 2024-09-16T15:42:37.686 INFO:teuthology.misc.health.smithi123.stderr:2024-09-16T15:42:37.683+0000 7f6dcd2bd640 1 -- 172.21.15.123:0/3417905277 wait complete. 2024-09-16T15:42:37.686 INFO:tasks.ceph.osd.7.smithi184.stderr:2024-09-16T15:42:37.679+0000 7fe53a9f6640 -1 osd.7 0 waiting for initial osdmap 2024-09-16T15:42:37.687 INFO:teuthology.misc.health.smithi123.stderr:2024-09-16T15:42:37.687+0000 7f6dcd2bd640 1 Processor -- start 2024-09-16T15:42:37.687 INFO:teuthology.misc.health.smithi123.stderr:2024-09-16T15:42:37.687+0000 7f6dcd2bd640 1 -- start start 2024-09-16T15:42:37.689 INFO:teuthology.misc.health.smithi123.stderr:2024-09-16T15:42:37.687+0000 7f6dcd2bd640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f6dc8073f10 0x7f6dc814bf70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:42:37.689 INFO:teuthology.misc.health.smithi123.stderr:2024-09-16T15:42:37.687+0000 7f6dcd2bd640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6dc81518d0 0x7f6dc814c730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:42:37.689 INFO:teuthology.misc.health.smithi123.stderr:2024-09-16T15:42:37.687+0000 7f6dcd2bd640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6dc815f1f0 0x7f6dc814f170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:42:37.689 INFO:teuthology.misc.health.smithi123.stderr:2024-09-16T15:42:37.687+0000 7f6dcd2bd640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f6dc8167910 con 0x7f6dc815f1f0 2024-09-16T15:42:37.689 INFO:teuthology.misc.health.smithi123.stderr:2024-09-16T15:42:37.687+0000 7f6dcd2bd640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f6dc8070fd0 con 0x7f6dc81518d0 2024-09-16T15:42:37.689 INFO:teuthology.misc.health.smithi123.stderr:2024-09-16T15:42:37.687+0000 7f6dcd2bd640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f6dc8167a90 con 0x7f6dc8073f10 2024-09-16T15:42:37.690 INFO:teuthology.misc.health.smithi123.stderr:2024-09-16T15:42:37.687+0000 7f6dc77fe640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6dc815f1f0 0x7f6dc814f170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:42:37.690 INFO:teuthology.misc.health.smithi123.stderr:2024-09-16T15:42:37.687+0000 7f6dc77fe640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6dc815f1f0 0x7f6dc814f170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:49896/0 (socket says 172.21.15.123:49896) 2024-09-16T15:42:37.690 INFO:teuthology.misc.health.smithi123.stderr:2024-09-16T15:42:37.687+0000 7f6dc77fe640 1 -- 172.21.15.123:0/3753867449 learned_addr learned my addr 172.21.15.123:0/3753867449 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:42:37.690 INFO:teuthology.misc.health.smithi123.stderr:2024-09-16T15:42:37.687+0000 7f6dc6ffd640 1 --2- 172.21.15.123:0/3753867449 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f6dc8073f10 0x7f6dc814bf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:42:37.690 INFO:teuthology.misc.health.smithi123.stderr:2024-09-16T15:42:37.687+0000 7f6dc67fc640 1 --2- 172.21.15.123:0/3753867449 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6dc81518d0 0x7f6dc814c730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:42:37.690 INFO:teuthology.misc.health.smithi123.stderr:2024-09-16T15:42:37.687+0000 7f6dc77fe640 1 -- 172.21.15.123:0/3753867449 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f6dc8073f10 msgr2=0x7f6dc814bf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:42:37.690 INFO:teuthology.misc.health.smithi123.stderr:2024-09-16T15:42:37.687+0000 7f6dc77fe640 1 --2- 172.21.15.123:0/3753867449 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f6dc8073f10 0x7f6dc814bf70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:42:37.690 INFO:teuthology.misc.health.smithi123.stderr:2024-09-16T15:42:37.687+0000 7f6dc77fe640 1 -- 172.21.15.123:0/3753867449 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6dc81518d0 msgr2=0x7f6dc814c730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:42:37.690 INFO:teuthology.misc.health.smithi123.stderr:2024-09-16T15:42:37.687+0000 7f6dc77fe640 1 --2- 172.21.15.123:0/3753867449 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6dc81518d0 0x7f6dc814c730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:42:37.691 INFO:teuthology.misc.health.smithi123.stderr:2024-09-16T15:42:37.687+0000 7f6dc77fe640 1 -- 172.21.15.123:0/3753867449 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6dc81fef00 con 0x7f6dc815f1f0 2024-09-16T15:42:37.691 INFO:teuthology.misc.health.smithi123.stderr:2024-09-16T15:42:37.687+0000 7f6dc77fe640 1 --2- 172.21.15.123:0/3753867449 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6dc815f1f0 0x7f6dc814f170 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f6dc000eaf0 tx=0x7f6dc000c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:42:37.692 INFO:teuthology.misc.health.smithi123.stderr:2024-09-16T15:42:37.691+0000 7f6daf7fe640 1 -- 172.21.15.123:0/3753867449 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6dc0004b30 con 0x7f6dc815f1f0 2024-09-16T15:42:37.692 INFO:teuthology.misc.health.smithi123.stderr:2024-09-16T15:42:37.691+0000 7f6daf7fe640 1 -- 172.21.15.123:0/3753867449 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f6dc0004cf0 con 0x7f6dc815f1f0 2024-09-16T15:42:37.692 INFO:teuthology.misc.health.smithi123.stderr:2024-09-16T15:42:37.691+0000 7f6daf7fe640 1 -- 172.21.15.123:0/3753867449 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6dc0005280 con 0x7f6dc815f1f0 2024-09-16T15:42:37.692 INFO:teuthology.misc.health.smithi123.stderr:2024-09-16T15:42:37.691+0000 7f6dcd2bd640 1 -- 172.21.15.123:0/3753867449 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6dc81ff1e0 con 0x7f6dc815f1f0 2024-09-16T15:42:37.692 INFO:teuthology.misc.health.smithi123.stderr:2024-09-16T15:42:37.691+0000 7f6dcd2bd640 1 -- 172.21.15.123:0/3753867449 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6dc81ff7a0 con 0x7f6dc815f1f0 2024-09-16T15:42:37.694 INFO:teuthology.misc.health.smithi123.stderr:2024-09-16T15:42:37.695+0000 7f6daf7fe640 1 -- 172.21.15.123:0/3753867449 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 4) ==== 97024+0+0 (secure 0 0 0) 0x7f6dc0005420 con 0x7f6dc815f1f0 2024-09-16T15:42:37.694 INFO:teuthology.misc.health.smithi123.stderr:2024-09-16T15:42:37.695+0000 7f6daf7fe640 1 --2- 172.21.15.123:0/3753867449 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f6db4072f00 0x7f6db40753c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:42:37.695 INFO:teuthology.misc.health.smithi123.stderr:2024-09-16T15:42:37.695+0000 7f6daf7fe640 1 -- 172.21.15.123:0/3753867449 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(13..13 src has 1..13) ==== 2231+0+0 (secure 0 0 0) 0x7f6dc0093b10 con 0x7f6dc815f1f0 2024-09-16T15:42:37.695 INFO:teuthology.misc.health.smithi123.stderr:2024-09-16T15:42:37.695+0000 7f6dc6ffd640 1 --2- 172.21.15.123:0/3753867449 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f6db4072f00 0x7f6db40753c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:42:37.695 INFO:teuthology.misc.health.smithi123.stderr:2024-09-16T15:42:37.695+0000 7f6dcd2bd640 1 -- 172.21.15.123:0/3753867449 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6dc806fc80 con 0x7f6dc815f1f0 2024-09-16T15:42:37.695 INFO:teuthology.misc.health.smithi123.stderr:2024-09-16T15:42:37.695+0000 7f6dc6ffd640 1 --2- 172.21.15.123:0/3753867449 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f6db4072f00 0x7f6db40753c0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f6db80af000 tx=0x7f6db801b060 comp rx=0 tx=0).ready entity=mgr.4135 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:42:37.704 INFO:teuthology.misc.health.smithi123.stderr:2024-09-16T15:42:37.703+0000 7f6daf7fe640 1 -- 172.21.15.123:0/3753867449 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f6dc005f560 con 0x7f6dc815f1f0 2024-09-16T15:42:37.732 INFO:tasks.ceph.osd.7.smithi184.stderr:2024-09-16T15:42:37.727+0000 7fe532828640 -1 osd.7 13 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-16T15:42:37.733 INFO:tasks.ceph.osd.5.smithi184.stderr:2024-09-16T15:42:37.727+0000 7f4f1165b640 -1 osd.5 0 waiting for initial osdmap 2024-09-16T15:42:37.774 INFO:tasks.ceph.osd.5.smithi184.stderr:2024-09-16T15:42:37.771+0000 7f4f0948d640 -1 osd.5 13 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-16T15:42:37.990 INFO:tasks.ceph.osd.1.smithi123.stderr:2024-09-16T15:42:37.991+0000 7f811947d640 -1 osd.1 0 waiting for initial osdmap 2024-09-16T15:42:38.026 INFO:teuthology.misc.health.smithi123.stderr:2024-09-16T15:42:38.023+0000 7f6dcd2bd640 1 -- 172.21.15.123:0/3753867449 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f6dc8150870 con 0x7f6dc815f1f0 2024-09-16T15:42:38.027 INFO:tasks.ceph.osd.1.smithi123.stderr:2024-09-16T15:42:38.027+0000 7f81112af640 -1 osd.1 13 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-16T15:42:38.028 INFO:teuthology.misc.health.smithi123.stderr:2024-09-16T15:42:38.027+0000 7f6daf7fe640 1 -- 172.21.15.123:0/3753867449 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v13) ==== 74+0+7058 (secure 0 0 0) 0x7f6dc0063f90 con 0x7f6dc815f1f0 2024-09-16T15:42:38.028 INFO:teuthology.misc.health.smithi123.stdout: 2024-09-16T15:42:38.028 INFO:teuthology.misc.health.smithi123.stdout:{"epoch":13,"fsid":"56077f89-18c8-486f-ae7e-eecc0bb7822c","created":"2024-09-16T15:42:26.364892+0000","modified":"2024-09-16T15:42:37.250209+0000","last_up_change":"0.000000","last_in_change":"2024-09-16T15:42:29.423662+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"9e7c1955-a607-4671-9be7-f2ef760ceeaa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"c540cfde-0977-4742-9b17-fd634668e010","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"9864a496-f169-486b-95c4-1d59162793a7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"c6d1f817-f074-4325-95e1-578eba13edda","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"200a8df4-50ef-40a5-b48a-58981269c086","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"32d5a900-da3d-412c-bc19-13861a2368d8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"cde6f39e-aa1b-4260-ac19-774ee6aaca72","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4f381778-eee1-47c6-804f-df0e6b33e313","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T15:42:38.031 INFO:teuthology.misc.health.smithi123.stderr:2024-09-16T15:42:38.031+0000 7f6dcd2bd640 1 -- 172.21.15.123:0/3753867449 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f6db4072f00 msgr2=0x7f6db40753c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:42:38.031 INFO:teuthology.misc.health.smithi123.stderr:2024-09-16T15:42:38.031+0000 7f6dcd2bd640 1 --2- 172.21.15.123:0/3753867449 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f6db4072f00 0x7f6db40753c0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f6db80af000 tx=0x7f6db801b060 comp rx=0 tx=0).stop 2024-09-16T15:42:38.031 INFO:teuthology.misc.health.smithi123.stderr:2024-09-16T15:42:38.031+0000 7f6dcd2bd640 1 -- 172.21.15.123:0/3753867449 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6dc815f1f0 msgr2=0x7f6dc814f170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:42:38.032 INFO:teuthology.misc.health.smithi123.stderr:2024-09-16T15:42:38.031+0000 7f6dcd2bd640 1 --2- 172.21.15.123:0/3753867449 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6dc815f1f0 0x7f6dc814f170 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f6dc000eaf0 tx=0x7f6dc000c2d0 comp rx=0 tx=0).stop 2024-09-16T15:42:38.032 INFO:teuthology.misc.health.smithi123.stderr:2024-09-16T15:42:38.031+0000 7f6dcd2bd640 1 -- 172.21.15.123:0/3753867449 shutdown_connections 2024-09-16T15:42:38.032 INFO:teuthology.misc.health.smithi123.stderr:2024-09-16T15:42:38.031+0000 7f6dcd2bd640 1 --2- 172.21.15.123:0/3753867449 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f6db4072f00 0x7f6db40753c0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:42:38.033 INFO:teuthology.misc.health.smithi123.stderr:2024-09-16T15:42:38.031+0000 7f6dcd2bd640 1 --2- 172.21.15.123:0/3753867449 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6dc815f1f0 0x7f6dc814f170 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:42:38.033 INFO:teuthology.misc.health.smithi123.stderr:2024-09-16T15:42:38.031+0000 7f6dcd2bd640 1 --2- 172.21.15.123:0/3753867449 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6dc81518d0 0x7f6dc814c730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:42:38.033 INFO:teuthology.misc.health.smithi123.stderr:2024-09-16T15:42:38.031+0000 7f6dcd2bd640 1 --2- 172.21.15.123:0/3753867449 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f6dc8073f10 0x7f6dc814bf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:42:38.033 INFO:teuthology.misc.health.smithi123.stderr:2024-09-16T15:42:38.031+0000 7f6dcd2bd640 1 -- 172.21.15.123:0/3753867449 >> 172.21.15.123:0/3753867449 conn(0x7f6dc806dd30 msgr2=0x7f6dc8073800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:42:38.033 INFO:teuthology.misc.health.smithi123.stderr:2024-09-16T15:42:38.031+0000 7f6dcd2bd640 1 -- 172.21.15.123:0/3753867449 shutdown_connections 2024-09-16T15:42:38.033 INFO:teuthology.misc.health.smithi123.stderr:2024-09-16T15:42:38.031+0000 7f6dcd2bd640 1 -- 172.21.15.123:0/3753867449 wait complete. 2024-09-16T15:42:38.053 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-09-16T15:42:38.177 INFO:tasks.ceph.osd.2.smithi123.stderr:2024-09-16T15:42:38.175+0000 7fdb1b809640 -1 osd.2 0 waiting for initial osdmap 2024-09-16T15:42:38.181 INFO:tasks.ceph.osd.0.smithi123.stderr:2024-09-16T15:42:38.179+0000 7f3e00c7c640 -1 osd.0 0 waiting for initial osdmap 2024-09-16T15:42:38.206 INFO:tasks.ceph.osd.2.smithi123.stderr:2024-09-16T15:42:38.207+0000 7fdb12e29640 -1 osd.2 13 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-16T15:42:38.219 INFO:tasks.ceph.osd.0.smithi123.stderr:2024-09-16T15:42:38.219+0000 7f3df8aae640 -1 osd.0 13 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-16T15:42:38.402 DEBUG:teuthology.orchestra.run.smithi123:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-16T15:42:38.405 DEBUG:teuthology.orchestra.run.smithi184:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-16T15:42:38.503 INFO:tasks.ceph.osd.3.smithi123.stderr:2024-09-16T15:42:38.503+0000 7f8ab31ba640 -1 osd.3 0 waiting for initial osdmap 2024-09-16T15:42:38.523 INFO:tasks.ceph.osd.3.smithi123.stderr:2024-09-16T15:42:38.523+0000 7f8aaa7da640 -1 osd.3 13 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-16T15:42:38.607 INFO:tasks.ceph.osd.6.smithi184.stderr:2024-09-16T15:42:38.603+0000 7f991d14d640 -1 osd.6 0 waiting for initial osdmap 2024-09-16T15:42:38.623 INFO:tasks.ceph.osd.6.smithi184.stderr:2024-09-16T15:42:38.619+0000 7f9914f7f640 -1 osd.6 13 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-16T15:42:38.645 INFO:tasks.ceph.osd.4.smithi184.stderr:2024-09-16T15:42:38.639+0000 7f9489bd6640 -1 osd.4 0 waiting for initial osdmap 2024-09-16T15:42:38.663 INFO:tasks.ceph.osd.4.smithi184.stderr:2024-09-16T15:42:38.659+0000 7f94811f6640 -1 osd.4 13 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-16T15:42:44.859 DEBUG:teuthology.orchestra.run.smithi123:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-09-16T15:42:44.989 INFO:teuthology.misc.health.smithi123.stderr:2024-09-16T15:42:44.987+0000 7fa68d742640 1 -- 172.21.15.123:0/2673167913 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa68815f670 msgr2=0x7fa6880cc640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:42:44.989 INFO:teuthology.misc.health.smithi123.stderr:2024-09-16T15:42:44.987+0000 7fa68d742640 1 --2- 172.21.15.123:0/2673167913 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa68815f670 0x7fa6880cc640 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fa688076390 tx=0x7fa67001c8c0 comp rx=0 tx=0).stop 2024-09-16T15:42:44.989 INFO:teuthology.misc.health.smithi123.stderr:2024-09-16T15:42:44.987+0000 7fa68d742640 1 -- 172.21.15.123:0/2673167913 shutdown_connections 2024-09-16T15:42:44.989 INFO:teuthology.misc.health.smithi123.stderr:2024-09-16T15:42:44.987+0000 7fa68d742640 1 --2- 172.21.15.123:0/2673167913 >> v2:172.21.15.184:3300/0 conn(0x7fa68815fa50 0x7fa688073090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:42:44.989 INFO:teuthology.misc.health.smithi123.stderr:2024-09-16T15:42:44.987+0000 7fa68d742640 1 --2- 172.21.15.123:0/2673167913 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa68815f670 0x7fa6880cc640 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:42:44.989 INFO:teuthology.misc.health.smithi123.stderr:2024-09-16T15:42:44.987+0000 7fa68d742640 1 -- 172.21.15.123:0/2673167913 >> 172.21.15.123:0/2673167913 conn(0x7fa6880b1650 msgr2=0x7fa6880b1a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:42:44.989 INFO:teuthology.misc.health.smithi123.stderr:2024-09-16T15:42:44.987+0000 7fa68d742640 1 -- 172.21.15.123:0/2673167913 shutdown_connections 2024-09-16T15:42:44.989 INFO:teuthology.misc.health.smithi123.stderr:2024-09-16T15:42:44.987+0000 7fa68d742640 1 -- 172.21.15.123:0/2673167913 wait complete. 2024-09-16T15:42:44.990 INFO:teuthology.misc.health.smithi123.stderr:2024-09-16T15:42:44.987+0000 7fa68d742640 1 Processor -- start 2024-09-16T15:42:44.990 INFO:teuthology.misc.health.smithi123.stderr:2024-09-16T15:42:44.987+0000 7fa68d742640 1 -- start start 2024-09-16T15:42:44.992 INFO:teuthology.misc.health.smithi123.stderr:2024-09-16T15:42:44.991+0000 7fa68d742640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa68815c080 0x7fa6881560a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:42:44.993 INFO:teuthology.misc.health.smithi123.stderr:2024-09-16T15:42:44.991+0000 7fa68d742640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fa68815f670 0x7fa6881f9b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:42:44.993 INFO:teuthology.misc.health.smithi123.stderr:2024-09-16T15:42:44.991+0000 7fa68d742640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa68815fa50 0x7fa688145be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:42:44.993 INFO:teuthology.misc.health.smithi123.stderr:2024-09-16T15:42:44.991+0000 7fa68d742640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fa68815ffd0 con 0x7fa68815c080 2024-09-16T15:42:44.993 INFO:teuthology.misc.health.smithi123.stderr:2024-09-16T15:42:44.991+0000 7fa68d742640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fa688169010 con 0x7fa68815fa50 2024-09-16T15:42:44.993 INFO:teuthology.misc.health.smithi123.stderr:2024-09-16T15:42:44.991+0000 7fa68d742640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fa688169190 con 0x7fa68815f670 2024-09-16T15:42:44.993 INFO:teuthology.misc.health.smithi123.stderr:2024-09-16T15:42:44.991+0000 7fa686ffd640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa68815c080 0x7fa6881560a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:42:44.993 INFO:teuthology.misc.health.smithi123.stderr:2024-09-16T15:42:44.991+0000 7fa686ffd640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa68815c080 0x7fa6881560a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:44776/0 (socket says 172.21.15.123:44776) 2024-09-16T15:42:44.993 INFO:teuthology.misc.health.smithi123.stderr:2024-09-16T15:42:44.991+0000 7fa686ffd640 1 -- 172.21.15.123:0/1348053503 learned_addr learned my addr 172.21.15.123:0/1348053503 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:42:44.993 INFO:teuthology.misc.health.smithi123.stderr:2024-09-16T15:42:44.991+0000 7fa6867fc640 1 --2- 172.21.15.123:0/1348053503 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fa68815f670 0x7fa6881f9b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:42:44.994 INFO:teuthology.misc.health.smithi123.stderr:2024-09-16T15:42:44.991+0000 7fa686ffd640 1 -- 172.21.15.123:0/1348053503 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fa68815f670 msgr2=0x7fa6881f9b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:42:44.994 INFO:teuthology.misc.health.smithi123.stderr:2024-09-16T15:42:44.991+0000 7fa686ffd640 1 --2- 172.21.15.123:0/1348053503 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fa68815f670 0x7fa6881f9b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:42:44.994 INFO:teuthology.misc.health.smithi123.stderr:2024-09-16T15:42:44.991+0000 7fa6877fe640 1 --2- 172.21.15.123:0/1348053503 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa68815fa50 0x7fa688145be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:42:44.994 INFO:teuthology.misc.health.smithi123.stderr:2024-09-16T15:42:44.991+0000 7fa686ffd640 1 -- 172.21.15.123:0/1348053503 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa68815fa50 msgr2=0x7fa688145be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:42:44.994 INFO:teuthology.misc.health.smithi123.stderr:2024-09-16T15:42:44.991+0000 7fa686ffd640 1 --2- 172.21.15.123:0/1348053503 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa68815fa50 0x7fa688145be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:42:44.994 INFO:teuthology.misc.health.smithi123.stderr:2024-09-16T15:42:44.991+0000 7fa686ffd640 1 -- 172.21.15.123:0/1348053503 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa6881653c0 con 0x7fa68815c080 2024-09-16T15:42:44.994 INFO:teuthology.misc.health.smithi123.stderr:2024-09-16T15:42:44.991+0000 7fa6877fe640 1 --2- 172.21.15.123:0/1348053503 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa68815fa50 0x7fa688145be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T15:42:44.994 INFO:teuthology.misc.health.smithi123.stderr:2024-09-16T15:42:44.991+0000 7fa686ffd640 1 --2- 172.21.15.123:0/1348053503 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa68815c080 0x7fa6881560a0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fa6700b0040 tx=0x7fa6700a6f60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:42:44.994 INFO:teuthology.misc.health.smithi123.stderr:2024-09-16T15:42:44.991+0000 7fa67f7fe640 1 -- 172.21.15.123:0/1348053503 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6700b2040 con 0x7fa68815c080 2024-09-16T15:42:44.994 INFO:teuthology.misc.health.smithi123.stderr:2024-09-16T15:42:44.991+0000 7fa67f7fe640 1 -- 172.21.15.123:0/1348053503 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fa670005070 con 0x7fa68815c080 2024-09-16T15:42:44.994 INFO:teuthology.misc.health.smithi123.stderr:2024-09-16T15:42:44.991+0000 7fa67f7fe640 1 -- 172.21.15.123:0/1348053503 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6700054c0 con 0x7fa68815c080 2024-09-16T15:42:44.994 INFO:teuthology.misc.health.smithi123.stderr:2024-09-16T15:42:44.991+0000 7fa68d742640 1 -- 172.21.15.123:0/1348053503 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa6881656a0 con 0x7fa68815c080 2024-09-16T15:42:44.995 INFO:teuthology.misc.health.smithi123.stderr:2024-09-16T15:42:44.991+0000 7fa68d742640 1 -- 172.21.15.123:0/1348053503 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa688146120 con 0x7fa68815c080 2024-09-16T15:42:44.995 INFO:teuthology.misc.health.smithi123.stderr:2024-09-16T15:42:44.991+0000 7fa68d742640 1 -- 172.21.15.123:0/1348053503 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa6880cc640 con 0x7fa68815c080 2024-09-16T15:42:44.995 INFO:teuthology.misc.health.smithi123.stderr:2024-09-16T15:42:44.991+0000 7fa67f7fe640 1 -- 172.21.15.123:0/1348053503 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 5) ==== 97080+0+0 (secure 0 0 0) 0x7fa6700046d0 con 0x7fa68815c080 2024-09-16T15:42:44.996 INFO:teuthology.misc.health.smithi123.stderr:2024-09-16T15:42:44.991+0000 7fa67f7fe640 1 --2- 172.21.15.123:0/1348053503 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fa668072ec0 0x7fa668075380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:42:44.996 INFO:teuthology.misc.health.smithi123.stderr:2024-09-16T15:42:44.991+0000 7fa67f7fe640 1 -- 172.21.15.123:0/1348053503 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(17..17 src has 1..17) ==== 4857+0+0 (secure 0 0 0) 0x7fa67012d160 con 0x7fa68815c080 2024-09-16T15:42:44.996 INFO:teuthology.misc.health.smithi123.stderr:2024-09-16T15:42:44.995+0000 7fa6867fc640 1 --2- 172.21.15.123:0/1348053503 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fa668072ec0 0x7fa668075380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:42:44.996 INFO:teuthology.misc.health.smithi123.stderr:2024-09-16T15:42:44.995+0000 7fa6867fc640 1 --2- 172.21.15.123:0/1348053503 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fa668072ec0 0x7fa668075380 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fa6780058d0 tx=0x7fa678005860 comp rx=0 tx=0).ready entity=mgr.4135 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:42:45.020 INFO:teuthology.misc.health.smithi123.stderr:2024-09-16T15:42:45.019+0000 7fa67f7fe640 1 -- 172.21.15.123:0/1348053503 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7fa6700bc060 con 0x7fa68815c080 2024-09-16T15:42:45.196 INFO:teuthology.misc.health.smithi123.stderr:2024-09-16T15:42:45.195+0000 7fa68d742640 1 -- 172.21.15.123:0/1348053503 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fa6880af940 con 0x7fa68815c080 2024-09-16T15:42:45.198 INFO:teuthology.misc.health.smithi123.stderr:2024-09-16T15:42:45.195+0000 7fa67f7fe640 1 -- 172.21.15.123:0/1348053503 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v17) ==== 74+0+12956 (secure 0 0 0) 0x7fa6700164b0 con 0x7fa68815c080 2024-09-16T15:42:45.198 INFO:teuthology.misc.health.smithi123.stdout: 2024-09-16T15:42:45.198 INFO:teuthology.misc.health.smithi123.stdout:{"epoch":17,"fsid":"56077f89-18c8-486f-ae7e-eecc0bb7822c","created":"2024-09-16T15:42:26.364892+0000","modified":"2024-09-16T15:42:41.751559+0000","last_up_change":"2024-09-16T15:42:38.707234+0000","last_in_change":"2024-09-16T15:42:29.423662+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-16T15:42:39.287764+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9e7c1955-a607-4671-9be7-f2ef760ceeaa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6800","nonce":3882044969},{"type":"v1","addr":"172.21.15.123:6801","nonce":3882044969}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6802","nonce":3882044969},{"type":"v1","addr":"172.21.15.123:6803","nonce":3882044969}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6806","nonce":3882044969},{"type":"v1","addr":"172.21.15.123:6807","nonce":3882044969}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6804","nonce":3882044969},{"type":"v1","addr":"172.21.15.123:6805","nonce":3882044969}]},"public_addr":"172.21.15.123:6801/3882044969","cluster_addr":"172.21.15.123:6803/3882044969","heartbeat_back_addr":"172.21.15.123:6807/3882044969","heartbeat_front_addr":"172.21.15.123:6805/3882044969","state":["exists","up"]},{"osd":1,"uuid":"c540cfde-0977-4742-9b17-fd634668e010","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6816","nonce":4037001850},{"type":"v1","addr":"172.21.15.123:6817","nonce":4037001850}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6818","nonce":4037001850},{"type":"v1","addr":"172.21.15.123:6819","nonce":4037001850}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6822","nonce":4037001850},{"type":"v1","addr":"172.21.15.123:6823","nonce":4037001850}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6820","nonce":4037001850},{"type":"v1","addr":"172.21.15.123:6821","nonce":4037001850}]},"public_addr":"172.21.15.123:6817/4037001850","cluster_addr":"172.21.15.123:6819/4037001850","heartbeat_back_addr":"172.21.15.123:6823/4037001850","heartbeat_front_addr":"172.21.15.123:6821/4037001850","state":["exists","up"]},{"osd":2,"uuid":"9864a496-f169-486b-95c4-1d59162793a7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6824","nonce":1332143088},{"type":"v1","addr":"172.21.15.123:6825","nonce":1332143088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6826","nonce":1332143088},{"type":"v1","addr":"172.21.15.123:6827","nonce":1332143088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6830","nonce":1332143088},{"type":"v1","addr":"172.21.15.123:6831","nonce":1332143088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6828","nonce":1332143088},{"type":"v1","addr":"172.21.15.123:6829","nonce":1332143088}]},"public_addr":"172.21.15.123:6825/1332143088","cluster_addr":"172.21.15.123:6827/1332143088","heartbeat_back_addr":"172.21.15.123:6831/1332143088","heartbeat_front_addr":"172.21.15.123:6829/1332143088","state":["exists","up"]},{"osd":3,"uuid":"c6d1f817-f074-4325-95e1-578eba13edda","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6808","nonce":988586998},{"type":"v1","addr":"172.21.15.123:6809","nonce":988586998}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6810","nonce":988586998},{"type":"v1","addr":"172.21.15.123:6811","nonce":988586998}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6814","nonce":988586998},{"type":"v1","addr":"172.21.15.123:6815","nonce":988586998}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6812","nonce":988586998},{"type":"v1","addr":"172.21.15.123:6813","nonce":988586998}]},"public_addr":"172.21.15.123:6809/988586998","cluster_addr":"172.21.15.123:6811/988586998","heartbeat_back_addr":"172.21.15.123:6815/988586998","heartbeat_front_addr":"172.21.15.123:6813/988586998","state":["exists","up"]},{"osd":4,"uuid":"200a8df4-50ef-40a5-b48a-58981269c086","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6816","nonce":3387813890},{"type":"v1","addr":"172.21.15.184:6817","nonce":3387813890}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6818","nonce":3387813890},{"type":"v1","addr":"172.21.15.184:6819","nonce":3387813890}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6822","nonce":3387813890},{"type":"v1","addr":"172.21.15.184:6823","nonce":3387813890}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6820","nonce":3387813890},{"type":"v1","addr":"172.21.15.184:6821","nonce":3387813890}]},"public_addr":"172.21.15.184:6817/3387813890","cluster_addr":"172.21.15.184:6819/3387813890","heartbeat_back_addr":"172.21.15.184:6823/3387813890","heartbeat_front_addr":"172.21.15.184:6821/3387813890","state":["exists","up"]},{"osd":5,"uuid":"32d5a900-da3d-412c-bc19-13861a2368d8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6808","nonce":2680916017},{"type":"v1","addr":"172.21.15.184:6809","nonce":2680916017}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6810","nonce":2680916017},{"type":"v1","addr":"172.21.15.184:6811","nonce":2680916017}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6814","nonce":2680916017},{"type":"v1","addr":"172.21.15.184:6815","nonce":2680916017}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6812","nonce":2680916017},{"type":"v1","addr":"172.21.15.184:6813","nonce":2680916017}]},"public_addr":"172.21.15.184:6809/2680916017","cluster_addr":"172.21.15.184:6811/2680916017","heartbeat_back_addr":"172.21.15.184:6815/2680916017","heartbeat_front_addr":"172.21.15.184:6813/2680916017","state":["exists","up"]},{"osd":6,"uuid":"cde6f39e-aa1b-4260-ac19-774ee6aaca72","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6824","nonce":2183120457},{"type":"v1","addr":"172.21.15.184:6825","nonce":2183120457}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6826","nonce":2183120457},{"type":"v1","addr":"172.21.15.184:6827","nonce":2183120457}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6830","nonce":2183120457},{"type":"v1","addr":"172.21.15.184:6831","nonce":2183120457}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6828","nonce":2183120457},{"type":"v1","addr":"172.21.15.184:6829","nonce":2183120457}]},"public_addr":"172.21.15.184:6825/2183120457","cluster_addr":"172.21.15.184:6827/2183120457","heartbeat_back_addr":"172.21.15.184:6831/2183120457","heartbeat_front_addr":"172.21.15.184:6829/2183120457","state":["exists","up"]},{"osd":7,"uuid":"4f381778-eee1-47c6-804f-df0e6b33e313","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6800","nonce":4102957285},{"type":"v1","addr":"172.21.15.184:6801","nonce":4102957285}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6802","nonce":4102957285},{"type":"v1","addr":"172.21.15.184:6803","nonce":4102957285}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6806","nonce":4102957285},{"type":"v1","addr":"172.21.15.184:6807","nonce":4102957285}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6804","nonce":4102957285},{"type":"v1","addr":"172.21.15.184:6805","nonce":4102957285}]},"public_addr":"172.21.15.184:6801/4102957285","cluster_addr":"172.21.15.184:6803/4102957285","heartbeat_back_addr":"172.21.15.184:6807/4102957285","heartbeat_front_addr":"172.21.15.184:6805/4102957285","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:35.972210+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.222007+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.198617+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.385663+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.358935+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.156066+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.752973+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:35.682846+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T15:42:45.201 INFO:teuthology.misc.health.smithi123.stderr:2024-09-16T15:42:45.199+0000 7fa68d742640 1 -- 172.21.15.123:0/1348053503 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fa668072ec0 msgr2=0x7fa668075380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:42:45.201 INFO:teuthology.misc.health.smithi123.stderr:2024-09-16T15:42:45.199+0000 7fa68d742640 1 --2- 172.21.15.123:0/1348053503 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fa668072ec0 0x7fa668075380 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fa6780058d0 tx=0x7fa678005860 comp rx=0 tx=0).stop 2024-09-16T15:42:45.201 INFO:teuthology.misc.health.smithi123.stderr:2024-09-16T15:42:45.199+0000 7fa68d742640 1 -- 172.21.15.123:0/1348053503 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa68815c080 msgr2=0x7fa6881560a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:42:45.201 INFO:teuthology.misc.health.smithi123.stderr:2024-09-16T15:42:45.199+0000 7fa68d742640 1 --2- 172.21.15.123:0/1348053503 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa68815c080 0x7fa6881560a0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fa6700b0040 tx=0x7fa6700a6f60 comp rx=0 tx=0).stop 2024-09-16T15:42:45.201 INFO:teuthology.misc.health.smithi123.stderr:2024-09-16T15:42:45.199+0000 7fa68d742640 1 -- 172.21.15.123:0/1348053503 shutdown_connections 2024-09-16T15:42:45.201 INFO:teuthology.misc.health.smithi123.stderr:2024-09-16T15:42:45.199+0000 7fa68d742640 1 --2- 172.21.15.123:0/1348053503 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fa668072ec0 0x7fa668075380 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:42:45.201 INFO:teuthology.misc.health.smithi123.stderr:2024-09-16T15:42:45.199+0000 7fa68d742640 1 --2- 172.21.15.123:0/1348053503 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa68815fa50 0x7fa688145be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:42:45.201 INFO:teuthology.misc.health.smithi123.stderr:2024-09-16T15:42:45.199+0000 7fa68d742640 1 --2- 172.21.15.123:0/1348053503 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fa68815f670 0x7fa6881f9b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:42:45.202 INFO:teuthology.misc.health.smithi123.stderr:2024-09-16T15:42:45.199+0000 7fa68d742640 1 --2- 172.21.15.123:0/1348053503 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa68815c080 0x7fa6881560a0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:42:45.202 INFO:teuthology.misc.health.smithi123.stderr:2024-09-16T15:42:45.199+0000 7fa68d742640 1 -- 172.21.15.123:0/1348053503 >> 172.21.15.123:0/1348053503 conn(0x7fa6880b1650 msgr2=0x7fa6880afda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:42:45.202 INFO:teuthology.misc.health.smithi123.stderr:2024-09-16T15:42:45.199+0000 7fa68d742640 1 -- 172.21.15.123:0/1348053503 shutdown_connections 2024-09-16T15:42:45.202 INFO:teuthology.misc.health.smithi123.stderr:2024-09-16T15:42:45.199+0000 7fa68d742640 1 -- 172.21.15.123:0/1348053503 wait complete. 2024-09-16T15:42:45.217 DEBUG:teuthology.misc:8 of 8 OSDs are up 2024-09-16T15:42:45.217 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2024-09-16T15:42:45.217 INFO:tasks.ceph.mds.a:Restarting daemon 2024-09-16T15:42:45.217 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-09-16T15:42:45.220 INFO:tasks.ceph.mds.a:Started 2024-09-16T15:42:45.220 INFO:tasks.ceph.mds.c:Restarting daemon 2024-09-16T15:42:45.220 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-09-16T15:42:45.222 INFO:tasks.ceph.mds.c:Started 2024-09-16T15:42:45.222 INFO:tasks.ceph.mds.b:Restarting daemon 2024-09-16T15:42:45.222 DEBUG:teuthology.orchestra.run.smithi184:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-09-16T15:42:45.225 INFO:tasks.ceph.mds.b:Started 2024-09-16T15:42:45.225 INFO:tasks.ceph.mds.d:Restarting daemon 2024-09-16T15:42:45.225 DEBUG:teuthology.orchestra.run.smithi184:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-09-16T15:42:45.228 INFO:tasks.ceph.mds.d:Started 2024-09-16T15:42:45.228 INFO:tasks.ceph:Setting up CephFS filesystem(s)... 2024-09-16T15:42:45.228 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-16T15:42:45.298 INFO:tasks.ceph.mds.c.smithi123.stdout:starting mds.c at 2024-09-16T15:42:45.300 INFO:tasks.ceph.mds.a.smithi123.stdout:starting mds.a at 2024-09-16T15:42:45.302 INFO:tasks.ceph.mds.b.smithi184.stdout:starting mds.b at 2024-09-16T15:42:45.305 INFO:tasks.ceph.mds.d.smithi184.stdout:starting mds.d at 2024-09-16T15:42:45.346 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:45.343+0000 7ff0639fc640 1 -- 172.21.15.123:0/201496971 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff048004810 msgr2=0x7ff048004c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:42:45.346 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:45.343+0000 7ff0639fc640 1 --2- 172.21.15.123:0/201496971 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff048004810 0x7ff048004c90 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7ff05c0d4fa0 tx=0x7ff04c01d200 comp rx=0 tx=0).stop 2024-09-16T15:42:45.346 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:45.343+0000 7ff061f99640 1 -- 172.21.15.123:0/201496971 reap_dead start 2024-09-16T15:42:45.346 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:45.343+0000 7ff0639fc640 1 -- 172.21.15.123:0/201496971 shutdown_connections 2024-09-16T15:42:45.346 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:45.343+0000 7ff0639fc640 1 -- 172.21.15.123:0/201496971 >> 172.21.15.123:0/201496971 conn(0x7ff05c0d3ea0 msgr2=0x7ff05c0d42b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:42:45.346 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:45.343+0000 7ff0639fc640 1 -- 172.21.15.123:0/201496971 shutdown_connections 2024-09-16T15:42:45.346 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:45.343+0000 7ff0639fc640 1 -- 172.21.15.123:0/201496971 wait complete. 2024-09-16T15:42:45.347 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:45.347+0000 7ff0639fc640 1 Processor -- start 2024-09-16T15:42:45.347 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:45.347+0000 7ff0639fc640 1 -- start start 2024-09-16T15:42:45.349 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:45.347+0000 7ff0639fc640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7ff048004810 0x7ff05c1fb3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:42:45.350 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:45.347+0000 7ff0639fc640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff05c15c680 0x7ff05c14c180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:42:45.350 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:45.347+0000 7ff0639fc640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff05c0cccc0 0x7ff05c0cd140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:42:45.350 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:45.347+0000 7ff0639fc640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7ff05c0d1f20 con 0x7ff05c15c680 2024-09-16T15:42:45.350 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:45.347+0000 7ff0639fc640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7ff05c0d1da0 con 0x7ff05c0cccc0 2024-09-16T15:42:45.350 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:45.347+0000 7ff0639fc640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7ff05c0d20a0 con 0x7ff048004810 2024-09-16T15:42:45.350 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:45.347+0000 7ff060f97640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff05c15c680 0x7ff05c14c180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:42:45.350 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:45.347+0000 7ff060f97640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff05c15c680 0x7ff05c14c180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:44812/0 (socket says 172.21.15.123:44812) 2024-09-16T15:42:45.350 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:45.347+0000 7ff060f97640 1 -- 172.21.15.123:0/247612931 learned_addr learned my addr 172.21.15.123:0/247612931 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:42:45.351 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:45.347+0000 7ff061f99640 1 --2- 172.21.15.123:0/247612931 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff05c0cccc0 0x7ff05c0cd140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:42:45.351 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:45.347+0000 7ff061798640 1 --2- 172.21.15.123:0/247612931 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7ff048004810 0x7ff05c1fb3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:42:45.351 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:45.347+0000 7ff060f97640 1 -- 172.21.15.123:0/247612931 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7ff048004810 msgr2=0x7ff05c1fb3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:42:45.351 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:45.347+0000 7ff060f97640 1 --2- 172.21.15.123:0/247612931 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7ff048004810 0x7ff05c1fb3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:42:45.351 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:45.347+0000 7ff060f97640 1 -- 172.21.15.123:0/247612931 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff05c0cccc0 msgr2=0x7ff05c0cd140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:42:45.351 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:45.347+0000 7ff060f97640 1 --2- 172.21.15.123:0/247612931 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff05c0cccc0 0x7ff05c0cd140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:42:45.351 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:45.347+0000 7ff060f97640 1 -- 172.21.15.123:0/247612931 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff05c1fb910 con 0x7ff05c15c680 2024-09-16T15:42:45.351 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:45.347+0000 7ff060f97640 1 --2- 172.21.15.123:0/247612931 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff05c15c680 0x7ff05c14c180 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7ff04c0af040 tx=0x7ff04c0ace30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:42:45.351 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:45.347+0000 7ff051ffb640 1 -- 172.21.15.123:0/247612931 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff04c0bc020 con 0x7ff05c15c680 2024-09-16T15:42:45.351 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:45.347+0000 7ff051ffb640 1 -- 172.21.15.123:0/247612931 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7ff04c005020 con 0x7ff05c15c680 2024-09-16T15:42:45.351 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:45.347+0000 7ff051ffb640 1 -- 172.21.15.123:0/247612931 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff04c0b1040 con 0x7ff05c15c680 2024-09-16T15:42:45.351 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:45.347+0000 7ff0639fc640 1 -- 172.21.15.123:0/247612931 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff05c1f7550 con 0x7ff05c15c680 2024-09-16T15:42:45.352 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:45.347+0000 7ff0639fc640 1 -- 172.21.15.123:0/247612931 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff05c1f7b10 con 0x7ff05c15c680 2024-09-16T15:42:45.352 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:45.347+0000 7ff0639fc640 1 -- 172.21.15.123:0/247612931 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff05c0d22b0 con 0x7ff05c15c680 2024-09-16T15:42:45.352 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:45.351+0000 7ff051ffb640 1 -- 172.21.15.123:0/247612931 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 5) ==== 97080+0+0 (secure 0 0 0) 0x7ff04c0bf070 con 0x7ff05c15c680 2024-09-16T15:42:45.353 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:45.351+0000 7ff051ffb640 1 --2- 172.21.15.123:0/247612931 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7ff040072ef0 0x7ff0400753b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:42:45.353 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:45.351+0000 7ff051ffb640 1 -- 172.21.15.123:0/247612931 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(17..17 src has 1..17) ==== 4857+0+0 (secure 0 0 0) 0x7ff04c12e110 con 0x7ff05c15c680 2024-09-16T15:42:45.353 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:45.351+0000 7ff061798640 1 --2- 172.21.15.123:0/247612931 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7ff040072ef0 0x7ff0400753b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:42:45.353 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:45.351+0000 7ff061798640 1 --2- 172.21.15.123:0/247612931 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7ff040072ef0 0x7ff0400753b0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7ff058005fd0 tx=0x7ff058005f40 comp rx=0 tx=0).ready entity=mgr.4135 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:42:45.355 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:45.355+0000 7ff051ffb640 1 -- 172.21.15.123:0/247612931 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7ff05c1fb910 con 0x7ff05c15c680 2024-09-16T15:42:45.564 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:45.563+0000 7ff0639fc640 1 -- 172.21.15.123:0/247612931 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff05c0d49b0 con 0x7ff05c15c680 2024-09-16T15:42:45.565 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:45.563+0000 7ff051ffb640 1 -- 172.21.15.123:0/247612931 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 2 v2) ==== 93+0+1437 (secure 0 0 0) 0x7ff04c016f40 con 0x7ff05c15c680 2024-09-16T15:42:45.565 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-09-16T15:42:45.565 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":2,"btime":"2024-09-16T15:42:45:328223+0000","default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4305,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6833/2227496057","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6832","nonce":2227496057},{"type":"v1","addr":"172.21.15.123:6833","nonce":2227496057}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[]} 2024-09-16T15:42:45.566 INFO:teuthology.orchestra.run.smithi123.stderr:dumped fsmap epoch 2 2024-09-16T15:42:45.568 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:45.567+0000 7ff0639fc640 1 -- 172.21.15.123:0/247612931 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7ff040072ef0 msgr2=0x7ff0400753b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:42:45.568 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:45.567+0000 7ff0639fc640 1 --2- 172.21.15.123:0/247612931 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7ff040072ef0 0x7ff0400753b0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7ff058005fd0 tx=0x7ff058005f40 comp rx=0 tx=0).stop 2024-09-16T15:42:45.568 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:45.567+0000 7ff0639fc640 1 -- 172.21.15.123:0/247612931 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff05c15c680 msgr2=0x7ff05c14c180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:42:45.568 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:45.567+0000 7ff0639fc640 1 --2- 172.21.15.123:0/247612931 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff05c15c680 0x7ff05c14c180 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7ff04c0af040 tx=0x7ff04c0ace30 comp rx=0 tx=0).stop 2024-09-16T15:42:45.568 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:45.567+0000 7ff0639fc640 1 -- 172.21.15.123:0/247612931 shutdown_connections 2024-09-16T15:42:45.568 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:45.567+0000 7ff0639fc640 1 --2- 172.21.15.123:0/247612931 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7ff040072ef0 0x7ff0400753b0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:42:45.568 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:45.567+0000 7ff0639fc640 1 --2- 172.21.15.123:0/247612931 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff05c0cccc0 0x7ff05c0cd140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:42:45.568 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:45.567+0000 7ff0639fc640 1 --2- 172.21.15.123:0/247612931 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff05c15c680 0x7ff05c14c180 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:42:45.568 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:45.567+0000 7ff0639fc640 1 --2- 172.21.15.123:0/247612931 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7ff048004810 0x7ff05c1fb3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:42:45.568 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:45.567+0000 7ff0639fc640 1 -- 172.21.15.123:0/247612931 >> 172.21.15.123:0/247612931 conn(0x7ff05c0d3ea0 msgr2=0x7ff05c06ebb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:42:45.568 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:45.567+0000 7ff0639fc640 1 -- 172.21.15.123:0/247612931 shutdown_connections 2024-09-16T15:42:45.569 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:45.567+0000 7ff0639fc640 1 -- 172.21.15.123:0/247612931 wait complete. 2024-09-16T15:42:47.585 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-16T15:42:47.719 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:47.719+0000 7f2e6971a640 1 -- 172.21.15.123:0/1470966504 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2e640cf5a0 msgr2=0x7f2e6406eee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:42:47.719 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:47.719+0000 7f2e6971a640 1 --2- 172.21.15.123:0/1470966504 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2e640cf5a0 0x7f2e6406eee0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f2e64070360 tx=0x7f2e5801c7a0 comp rx=0 tx=0).stop 2024-09-16T15:42:47.720 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:47.719+0000 7f2e6971a640 1 -- 172.21.15.123:0/1470966504 shutdown_connections 2024-09-16T15:42:47.720 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:47.719+0000 7f2e6971a640 1 --2- 172.21.15.123:0/1470966504 >> v2:172.21.15.184:3300/0 conn(0x7f2e6415ec40 0x7f2e6406f420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:42:47.720 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:47.719+0000 7f2e6971a640 1 --2- 172.21.15.123:0/1470966504 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2e640cf5a0 0x7f2e6406eee0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:42:47.720 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:47.719+0000 7f2e6971a640 1 -- 172.21.15.123:0/1470966504 >> 172.21.15.123:0/1470966504 conn(0x7f2e640cfb10 msgr2=0x7f2e640cff20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:42:47.720 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:47.719+0000 7f2e6971a640 1 -- 172.21.15.123:0/1470966504 shutdown_connections 2024-09-16T15:42:47.720 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:47.719+0000 7f2e6971a640 1 -- 172.21.15.123:0/1470966504 wait complete. 2024-09-16T15:42:47.720 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:47.719+0000 7f2e6971a640 1 Processor -- start 2024-09-16T15:42:47.721 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:47.719+0000 7f2e6971a640 1 -- start start 2024-09-16T15:42:47.723 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:47.719+0000 7f2e6971a640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2e640cf5a0 0x7f2e641f94d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:42:47.723 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:47.719+0000 7f2e6971a640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f2e64158120 0x7f2e64147d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:42:47.723 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:47.719+0000 7f2e6971a640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2e6415ec40 0x7f2e64148240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:42:47.723 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:47.719+0000 7f2e6971a640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f2e6415b1e0 con 0x7f2e640cf5a0 2024-09-16T15:42:47.723 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:47.719+0000 7f2e6971a640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f2e6415b060 con 0x7f2e6415ec40 2024-09-16T15:42:47.723 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:47.719+0000 7f2e6971a640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f2e6415b360 con 0x7f2e64158120 2024-09-16T15:42:47.723 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:47.719+0000 7f2e62ffd640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2e640cf5a0 0x7f2e641f94d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:42:47.723 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:47.719+0000 7f2e62ffd640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2e640cf5a0 0x7f2e641f94d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:44832/0 (socket says 172.21.15.123:44832) 2024-09-16T15:42:47.723 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:47.719+0000 7f2e62ffd640 1 -- 172.21.15.123:0/843045677 learned_addr learned my addr 172.21.15.123:0/843045677 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:42:47.723 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:47.719+0000 7f2e637fe640 1 --2- 172.21.15.123:0/843045677 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2e6415ec40 0x7f2e64148240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:42:47.724 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:47.719+0000 7f2e627fc640 1 --2- 172.21.15.123:0/843045677 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f2e64158120 0x7f2e64147d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:42:47.724 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:47.719+0000 7f2e62ffd640 1 -- 172.21.15.123:0/843045677 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f2e64158120 msgr2=0x7f2e64147d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:42:47.724 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:47.719+0000 7f2e62ffd640 1 --2- 172.21.15.123:0/843045677 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f2e64158120 0x7f2e64147d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:42:47.724 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:47.719+0000 7f2e62ffd640 1 -- 172.21.15.123:0/843045677 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2e6415ec40 msgr2=0x7f2e64148240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:42:47.724 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:47.719+0000 7f2e62ffd640 1 --2- 172.21.15.123:0/843045677 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2e6415ec40 0x7f2e64148240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:42:47.724 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:47.719+0000 7f2e62ffd640 1 -- 172.21.15.123:0/843045677 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2e6414fea0 con 0x7f2e640cf5a0 2024-09-16T15:42:47.724 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:47.719+0000 7f2e627fc640 1 --2- 172.21.15.123:0/843045677 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f2e64158120 0x7f2e64147d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T15:42:47.724 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:47.719+0000 7f2e62ffd640 1 --2- 172.21.15.123:0/843045677 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2e640cf5a0 0x7f2e641f94d0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f2e5400c530 tx=0x7f2e5400c9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:42:47.724 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:47.719+0000 7f2e4b7fe640 1 -- 172.21.15.123:0/843045677 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e54017070 con 0x7f2e640cf5a0 2024-09-16T15:42:47.724 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:47.719+0000 7f2e4b7fe640 1 -- 172.21.15.123:0/843045677 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f2e540053c0 con 0x7f2e640cf5a0 2024-09-16T15:42:47.724 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:47.719+0000 7f2e4b7fe640 1 -- 172.21.15.123:0/843045677 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e54012890 con 0x7f2e640cf5a0 2024-09-16T15:42:47.724 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:47.723+0000 7f2e6971a640 1 -- 172.21.15.123:0/843045677 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2e64152c60 con 0x7f2e640cf5a0 2024-09-16T15:42:47.725 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:47.723+0000 7f2e6971a640 1 -- 172.21.15.123:0/843045677 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2e64153220 con 0x7f2e640cf5a0 2024-09-16T15:42:47.725 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:47.723+0000 7f2e6971a640 1 -- 172.21.15.123:0/843045677 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2e640ca320 con 0x7f2e640cf5a0 2024-09-16T15:42:47.725 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:47.723+0000 7f2e4b7fe640 1 -- 172.21.15.123:0/843045677 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 5) ==== 97080+0+0 (secure 0 0 0) 0x7f2e5401a070 con 0x7f2e640cf5a0 2024-09-16T15:42:47.725 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:47.723+0000 7f2e4b7fe640 1 --2- 172.21.15.123:0/843045677 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f2e50072ec0 0x7f2e50075380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:42:47.726 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:47.723+0000 7f2e4b7fe640 1 -- 172.21.15.123:0/843045677 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(17..17 src has 1..17) ==== 4857+0+0 (secure 0 0 0) 0x7f2e54092dc0 con 0x7f2e640cf5a0 2024-09-16T15:42:47.726 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:47.723+0000 7f2e627fc640 1 --2- 172.21.15.123:0/843045677 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f2e50072ec0 0x7f2e50075380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:42:47.726 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:47.723+0000 7f2e627fc640 1 --2- 172.21.15.123:0/843045677 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f2e50072ec0 0x7f2e50075380 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f2e580af000 tx=0x7f2e5801afe0 comp rx=0 tx=0).ready entity=mgr.4135 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:42:47.729 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:47.727+0000 7f2e4b7fe640 1 -- 172.21.15.123:0/843045677 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f2e5405e360 con 0x7f2e640cf5a0 2024-09-16T15:42:47.938 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:47.939+0000 7f2e6971a640 1 -- 172.21.15.123:0/843045677 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f2e64150260 con 0x7f2e640cf5a0 2024-09-16T15:42:47.939 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:47.939+0000 7f2e4b7fe640 1 -- 172.21.15.123:0/843045677 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 3 v3) ==== 93+0+3915 (secure 0 0 0) 0x7f2e5405dc60 con 0x7f2e640cf5a0 2024-09-16T15:42:47.939 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-09-16T15:42:47.939 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":3,"btime":"2024-09-16T15:42:46:331440+0000","default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4280,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6835/1497621896","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":1497621896},{"type":"v1","addr":"172.21.15.184:6835","nonce":1497621896}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4305,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6833/2227496057","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6832","nonce":2227496057},{"type":"v1","addr":"172.21.15.123:6833","nonce":2227496057}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4311,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6835/294455998","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6834","nonce":294455998},{"type":"v1","addr":"172.21.15.123:6835","nonce":294455998}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4348,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6837/1469324095","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6836","nonce":1469324095},{"type":"v1","addr":"172.21.15.184:6837","nonce":1469324095}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[]} 2024-09-16T15:42:47.939 INFO:teuthology.orchestra.run.smithi123.stderr:dumped fsmap epoch 3 2024-09-16T15:42:47.942 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:47.939+0000 7f2e6971a640 1 -- 172.21.15.123:0/843045677 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f2e50072ec0 msgr2=0x7f2e50075380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:42:47.942 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:47.939+0000 7f2e6971a640 1 --2- 172.21.15.123:0/843045677 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f2e50072ec0 0x7f2e50075380 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f2e580af000 tx=0x7f2e5801afe0 comp rx=0 tx=0).stop 2024-09-16T15:42:47.942 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:47.939+0000 7f2e6971a640 1 -- 172.21.15.123:0/843045677 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2e640cf5a0 msgr2=0x7f2e641f94d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:42:47.942 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:47.939+0000 7f2e6971a640 1 --2- 172.21.15.123:0/843045677 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2e640cf5a0 0x7f2e641f94d0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f2e5400c530 tx=0x7f2e5400c9f0 comp rx=0 tx=0).stop 2024-09-16T15:42:47.942 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:47.939+0000 7f2e6971a640 1 -- 172.21.15.123:0/843045677 shutdown_connections 2024-09-16T15:42:47.942 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:47.939+0000 7f2e6971a640 1 --2- 172.21.15.123:0/843045677 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f2e50072ec0 0x7f2e50075380 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:42:47.942 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:47.939+0000 7f2e6971a640 1 --2- 172.21.15.123:0/843045677 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2e6415ec40 0x7f2e64148240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:42:47.942 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:47.939+0000 7f2e6971a640 1 --2- 172.21.15.123:0/843045677 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f2e64158120 0x7f2e64147d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:42:47.943 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:47.939+0000 7f2e6971a640 1 --2- 172.21.15.123:0/843045677 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2e640cf5a0 0x7f2e641f94d0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:42:47.943 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:47.939+0000 7f2e6971a640 1 -- 172.21.15.123:0/843045677 >> 172.21.15.123:0/843045677 conn(0x7f2e640cfb10 msgr2=0x7f2e64153f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:42:47.943 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:47.939+0000 7f2e6971a640 1 -- 172.21.15.123:0/843045677 shutdown_connections 2024-09-16T15:42:47.943 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:47.939+0000 7f2e6971a640 1 -- 172.21.15.123:0/843045677 wait complete. 2024-09-16T15:42:47.959 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-09-16T15:42:48.082 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:48.079+0000 7f103fa21640 1 -- 172.21.15.123:0/2143740002 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1024004810 msgr2=0x7f1024004c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:42:48.083 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:48.079+0000 7f103fa21640 1 --2- 172.21.15.123:0/2143740002 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1024004810 0x7f1024004c90 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f1028009b60 tx=0x7f102801d240 comp rx=0 tx=0).stop 2024-09-16T15:42:48.083 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:48.083+0000 7f103fa21640 1 -- 172.21.15.123:0/2143740002 shutdown_connections 2024-09-16T15:42:48.083 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:48.083+0000 7f103fa21640 1 --2- 172.21.15.123:0/2143740002 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1024004810 0x7f1024004c90 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:42:48.083 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:48.083+0000 7f103fa21640 1 -- 172.21.15.123:0/2143740002 >> 172.21.15.123:0/2143740002 conn(0x7f10380d3ea0 msgr2=0x7f10380d42b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:42:48.083 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:48.083+0000 7f103fa21640 1 -- 172.21.15.123:0/2143740002 shutdown_connections 2024-09-16T15:42:48.083 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:48.083+0000 7f103fa21640 1 -- 172.21.15.123:0/2143740002 wait complete. 2024-09-16T15:42:48.083 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:48.083+0000 7f103fa21640 1 Processor -- start 2024-09-16T15:42:48.084 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:48.083+0000 7f103fa21640 1 -- start start 2024-09-16T15:42:48.085 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:48.083+0000 7f103fa21640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f1024004810 0x7f10380b9770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:42:48.085 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:48.083+0000 7f103fa21640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f10381fb1c0 0x7f103815a520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:42:48.086 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:48.083+0000 7f103fa21640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f103815e420 0x7f103815e8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:42:48.086 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:48.083+0000 7f103fa21640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f10380cf4d0 con 0x7f10381fb1c0 2024-09-16T15:42:48.086 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:48.083+0000 7f103fa21640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f10380cf350 con 0x7f103815e420 2024-09-16T15:42:48.086 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:48.083+0000 7f103fa21640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f10380cf650 con 0x7f1024004810 2024-09-16T15:42:48.086 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:48.083+0000 7f103cfbc640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f10381fb1c0 0x7f103815a520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:42:48.086 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:48.083+0000 7f103cfbc640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f10381fb1c0 0x7f103815a520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:44862/0 (socket says 172.21.15.123:44862) 2024-09-16T15:42:48.086 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:48.083+0000 7f103cfbc640 1 -- 172.21.15.123:0/3440124143 learned_addr learned my addr 172.21.15.123:0/3440124143 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:42:48.086 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:48.083+0000 7f103dfbe640 1 --2- 172.21.15.123:0/3440124143 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f103815e420 0x7f103815e8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:42:48.086 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:48.083+0000 7f103d7bd640 1 --2- 172.21.15.123:0/3440124143 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f1024004810 0x7f10380b9770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:42:48.086 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:48.083+0000 7f103cfbc640 1 -- 172.21.15.123:0/3440124143 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f1024004810 msgr2=0x7f10380b9770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:42:48.086 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:48.083+0000 7f103cfbc640 1 --2- 172.21.15.123:0/3440124143 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f1024004810 0x7f10380b9770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:42:48.086 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:48.083+0000 7f103cfbc640 1 -- 172.21.15.123:0/3440124143 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f103815e420 msgr2=0x7f103815e8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:42:48.086 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:48.083+0000 7f103cfbc640 1 --2- 172.21.15.123:0/3440124143 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f103815e420 0x7f103815e8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:42:48.087 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:48.083+0000 7f103cfbc640 1 -- 172.21.15.123:0/3440124143 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f10381fb640 con 0x7f10381fb1c0 2024-09-16T15:42:48.087 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:48.083+0000 7f103dfbe640 1 --2- 172.21.15.123:0/3440124143 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f103815e420 0x7f103815e8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T15:42:48.087 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:48.083+0000 7f103cfbc640 1 --2- 172.21.15.123:0/3440124143 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f10381fb1c0 0x7f103815a520 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f10280af040 tx=0x7f1028005520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:42:48.087 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:48.083+0000 7f102dffb640 1 -- 172.21.15.123:0/3440124143 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10280b1040 con 0x7f10381fb1c0 2024-09-16T15:42:48.087 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:48.083+0000 7f102dffb640 1 -- 172.21.15.123:0/3440124143 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f1028004120 con 0x7f10381fb1c0 2024-09-16T15:42:48.087 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:48.083+0000 7f102dffb640 1 -- 172.21.15.123:0/3440124143 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10280ac7b0 con 0x7f10381fb1c0 2024-09-16T15:42:48.087 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:48.083+0000 7f103fa21640 1 -- 172.21.15.123:0/3440124143 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f10381f8380 con 0x7f10381fb1c0 2024-09-16T15:42:48.087 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:48.083+0000 7f103fa21640 1 -- 172.21.15.123:0/3440124143 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f10381f8920 con 0x7f10381fb1c0 2024-09-16T15:42:48.088 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:48.083+0000 7f103fa21640 1 -- 172.21.15.123:0/3440124143 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f10380d1e80 con 0x7f10381fb1c0 2024-09-16T15:42:48.088 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:48.087+0000 7f102dffb640 1 -- 172.21.15.123:0/3440124143 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 5) ==== 97080+0+0 (secure 0 0 0) 0x7f10280ac950 con 0x7f10381fb1c0 2024-09-16T15:42:48.088 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:48.087+0000 7f102dffb640 1 --2- 172.21.15.123:0/3440124143 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f101c072ec0 0x7f101c075380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:42:48.088 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:48.087+0000 7f102dffb640 1 -- 172.21.15.123:0/3440124143 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(17..17 src has 1..17) ==== 4857+0+0 (secure 0 0 0) 0x7f102812ea90 con 0x7f10381fb1c0 2024-09-16T15:42:48.088 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:48.087+0000 7f103d7bd640 1 --2- 172.21.15.123:0/3440124143 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f101c072ec0 0x7f101c075380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:42:48.088 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:48.087+0000 7f103d7bd640 1 --2- 172.21.15.123:0/3440124143 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f101c072ec0 0x7f101c075380 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f103806b850 tx=0x7f1034006ac0 comp rx=0 tx=0).ready entity=mgr.4135 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:42:48.091 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:48.090+0000 7f102dffb640 1 -- 172.21.15.123:0/3440124143 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f10381fb640 con 0x7f10381fb1c0 2024-09-16T15:42:48.257 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:48.254+0000 7f103fa21640 1 -- 172.21.15.123:0/3440124143 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) -- 0x7f10380b5e50 con 0x7f10381fb1c0 2024-09-16T15:42:48.258 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:48.254+0000 7f102dffb640 1 -- 172.21.15.123:0/3440124143 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v17) ==== 84+0+65 (secure 0 0 0) 0x7f1028016f80 con 0x7f10381fb1c0 2024-09-16T15:42:48.258 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-09-16T15:42:48.258 INFO:teuthology.orchestra.run.smithi123.stdout:[ 2024-09-16T15:42:48.258 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-09-16T15:42:48.259 INFO:teuthology.orchestra.run.smithi123.stdout: "poolnum": 1, 2024-09-16T15:42:48.259 INFO:teuthology.orchestra.run.smithi123.stdout: "poolname": ".mgr" 2024-09-16T15:42:48.259 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-16T15:42:48.259 INFO:teuthology.orchestra.run.smithi123.stdout:] 2024-09-16T15:42:48.260 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:48.258+0000 7f103fa21640 1 -- 172.21.15.123:0/3440124143 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f101c072ec0 msgr2=0x7f101c075380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:42:48.260 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:48.258+0000 7f103fa21640 1 --2- 172.21.15.123:0/3440124143 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f101c072ec0 0x7f101c075380 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f103806b850 tx=0x7f1034006ac0 comp rx=0 tx=0).stop 2024-09-16T15:42:48.260 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:48.258+0000 7f103fa21640 1 -- 172.21.15.123:0/3440124143 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f10381fb1c0 msgr2=0x7f103815a520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:42:48.261 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:48.258+0000 7f103fa21640 1 --2- 172.21.15.123:0/3440124143 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f10381fb1c0 0x7f103815a520 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f10280af040 tx=0x7f1028005520 comp rx=0 tx=0).stop 2024-09-16T15:42:48.261 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:48.258+0000 7f103fa21640 1 -- 172.21.15.123:0/3440124143 shutdown_connections 2024-09-16T15:42:48.261 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:48.258+0000 7f103fa21640 1 --2- 172.21.15.123:0/3440124143 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f101c072ec0 0x7f101c075380 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:42:48.261 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:48.258+0000 7f103fa21640 1 --2- 172.21.15.123:0/3440124143 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f103815e420 0x7f103815e8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:42:48.261 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:48.258+0000 7f103fa21640 1 --2- 172.21.15.123:0/3440124143 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f10381fb1c0 0x7f103815a520 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:42:48.261 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:48.258+0000 7f103fa21640 1 --2- 172.21.15.123:0/3440124143 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f1024004810 0x7f10380b9770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:42:48.261 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:48.258+0000 7f103fa21640 1 -- 172.21.15.123:0/3440124143 >> 172.21.15.123:0/3440124143 conn(0x7f10380d3ea0 msgr2=0x7f103806e780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:42:48.261 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:48.258+0000 7f103fa21640 1 -- 172.21.15.123:0/3440124143 shutdown_connections 2024-09-16T15:42:48.261 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:48.258+0000 7f103fa21640 1 -- 172.21.15.123:0/3440124143 wait complete. 2024-09-16T15:42:48.275 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-09-16T15:42:48.275 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-09-16T15:42:48.405 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:48.402+0000 7fd84bcc2640 1 -- 172.21.15.123:0/1136084657 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd844160310 msgr2=0x7fd844070470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:42:48.405 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:48.402+0000 7fd84bcc2640 1 --2- 172.21.15.123:0/1136084657 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd844160310 0x7fd844070470 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fd8440cf9f0 tx=0x7fd83401c830 comp rx=0 tx=0).stop 2024-09-16T15:42:48.405 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:48.402+0000 7fd84bcc2640 1 -- 172.21.15.123:0/1136084657 shutdown_connections 2024-09-16T15:42:48.405 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:48.402+0000 7fd84bcc2640 1 --2- 172.21.15.123:0/1136084657 >> v2:172.21.15.184:3300/0 conn(0x7fd84415f690 0x7fd84415fb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:42:48.405 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:48.402+0000 7fd84bcc2640 1 --2- 172.21.15.123:0/1136084657 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd844160310 0x7fd844070470 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:42:48.405 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:48.402+0000 7fd84bcc2640 1 -- 172.21.15.123:0/1136084657 >> 172.21.15.123:0/1136084657 conn(0x7fd8440b1630 msgr2=0x7fd8440b1a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:42:48.405 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:48.402+0000 7fd84bcc2640 1 -- 172.21.15.123:0/1136084657 shutdown_connections 2024-09-16T15:42:48.405 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:48.402+0000 7fd84bcc2640 1 -- 172.21.15.123:0/1136084657 wait complete. 2024-09-16T15:42:48.406 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:48.402+0000 7fd84bcc2640 1 Processor -- start 2024-09-16T15:42:48.406 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:48.402+0000 7fd84bcc2640 1 -- start start 2024-09-16T15:42:48.408 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:48.406+0000 7fd84bcc2640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fd84415bc40 0x7fd8440c52b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:42:48.408 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:48.406+0000 7fd84bcc2640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd84415f690 0x7fd8441464c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:42:48.408 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:48.406+0000 7fd84bcc2640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd844160310 0x7fd8441f7560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:42:48.409 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:48.406+0000 7fd84bcc2640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fd844168bb0 con 0x7fd84415f690 2024-09-16T15:42:48.409 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:48.406+0000 7fd84bcc2640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fd844168a30 con 0x7fd844160310 2024-09-16T15:42:48.409 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:48.406+0000 7fd84bcc2640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fd844168d30 con 0x7fd84415bc40 2024-09-16T15:42:48.409 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:48.406+0000 7fd84925d640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd84415f690 0x7fd8441464c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:42:48.409 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:48.406+0000 7fd84925d640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd84415f690 0x7fd8441464c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:44880/0 (socket says 172.21.15.123:44880) 2024-09-16T15:42:48.409 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:48.406+0000 7fd84925d640 1 -- 172.21.15.123:0/3730749457 learned_addr learned my addr 172.21.15.123:0/3730749457 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:42:48.409 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:48.406+0000 7fd849a5e640 1 --2- 172.21.15.123:0/3730749457 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fd84415bc40 0x7fd8440c52b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:42:48.409 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:48.406+0000 7fd84a25f640 1 --2- 172.21.15.123:0/3730749457 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd844160310 0x7fd8441f7560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:42:48.409 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:48.406+0000 7fd84925d640 1 -- 172.21.15.123:0/3730749457 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fd84415bc40 msgr2=0x7fd8440c52b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:42:48.409 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:48.406+0000 7fd84925d640 1 --2- 172.21.15.123:0/3730749457 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fd84415bc40 0x7fd8440c52b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:42:48.409 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:48.406+0000 7fd84925d640 1 -- 172.21.15.123:0/3730749457 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd844160310 msgr2=0x7fd8441f7560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:42:48.410 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:48.406+0000 7fd84925d640 1 --2- 172.21.15.123:0/3730749457 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd844160310 0x7fd8441f7560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:42:48.410 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:48.406+0000 7fd84925d640 1 -- 172.21.15.123:0/3730749457 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd844165810 con 0x7fd84415f690 2024-09-16T15:42:48.410 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:48.406+0000 7fd84a25f640 1 --2- 172.21.15.123:0/3730749457 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd844160310 0x7fd8441f7560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T15:42:48.410 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:48.406+0000 7fd84925d640 1 --2- 172.21.15.123:0/3730749457 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd84415f690 0x7fd8441464c0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fd834000c10 tx=0x7fd8340a56b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:42:48.410 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:48.406+0000 7fd83a7fc640 1 -- 172.21.15.123:0/3730749457 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8340ba020 con 0x7fd84415f690 2024-09-16T15:42:48.410 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:48.406+0000 7fd83a7fc640 1 -- 172.21.15.123:0/3730749457 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fd834005070 con 0x7fd84415f690 2024-09-16T15:42:48.410 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:48.406+0000 7fd84bcc2640 1 -- 172.21.15.123:0/3730749457 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd844165af0 con 0x7fd84415f690 2024-09-16T15:42:48.410 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:48.406+0000 7fd83a7fc640 1 -- 172.21.15.123:0/3730749457 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8340b2040 con 0x7fd84415f690 2024-09-16T15:42:48.410 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:48.406+0000 7fd84bcc2640 1 -- 172.21.15.123:0/3730749457 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd8441660b0 con 0x7fd84415f690 2024-09-16T15:42:48.410 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:48.406+0000 7fd84bcc2640 1 -- 172.21.15.123:0/3730749457 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd8440766e0 con 0x7fd84415f690 2024-09-16T15:42:48.411 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:48.406+0000 7fd83a7fc640 1 -- 172.21.15.123:0/3730749457 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 5) ==== 97080+0+0 (secure 0 0 0) 0x7fd8340054c0 con 0x7fd84415f690 2024-09-16T15:42:48.411 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:48.406+0000 7fd83a7fc640 1 --2- 172.21.15.123:0/3730749457 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fd828072ec0 0x7fd828075380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:42:48.411 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:48.406+0000 7fd83a7fc640 1 -- 172.21.15.123:0/3730749457 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(17..17 src has 1..17) ==== 4857+0+0 (secure 0 0 0) 0x7fd83412db40 con 0x7fd84415f690 2024-09-16T15:42:48.411 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:48.406+0000 7fd849a5e640 1 --2- 172.21.15.123:0/3730749457 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fd828072ec0 0x7fd828075380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:42:48.411 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:48.410+0000 7fd849a5e640 1 --2- 172.21.15.123:0/3730749457 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fd828072ec0 0x7fd828075380 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fd840004e30 tx=0x7fd840009040 comp rx=0 tx=0).ready entity=mgr.4135 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:42:48.413 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:48.410+0000 7fd83a7fc640 1 -- 172.21.15.123:0/3730749457 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7fd8340c4060 con 0x7fd84415f690 2024-09-16T15:42:48.576 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:48.574+0000 7fd84bcc2640 1 -- 172.21.15.123:0/3730749457 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) -- 0x7fd84415fb20 con 0x7fd84415f690 2024-09-16T15:42:49.360 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:49.358+0000 7fd83a7fc640 1 -- 172.21.15.123:0/3730749457 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v18) ==== 138+0+0 (secure 0 0 0) 0x7fd8340164b0 con 0x7fd84415f690 2024-09-16T15:42:49.361 INFO:teuthology.orchestra.run.smithi123.stderr:pool 'cephfs_metadata' created 2024-09-16T15:42:49.364 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:49.362+0000 7fd84bcc2640 1 -- 172.21.15.123:0/3730749457 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fd828072ec0 msgr2=0x7fd828075380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:42:49.364 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:49.362+0000 7fd84bcc2640 1 --2- 172.21.15.123:0/3730749457 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fd828072ec0 0x7fd828075380 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fd840004e30 tx=0x7fd840009040 comp rx=0 tx=0).stop 2024-09-16T15:42:49.364 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:49.362+0000 7fd84bcc2640 1 -- 172.21.15.123:0/3730749457 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd84415f690 msgr2=0x7fd8441464c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:42:49.364 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:49.362+0000 7fd84bcc2640 1 --2- 172.21.15.123:0/3730749457 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd84415f690 0x7fd8441464c0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fd834000c10 tx=0x7fd8340a56b0 comp rx=0 tx=0).stop 2024-09-16T15:42:49.364 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:49.362+0000 7fd84bcc2640 1 -- 172.21.15.123:0/3730749457 shutdown_connections 2024-09-16T15:42:49.364 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:49.362+0000 7fd84bcc2640 1 --2- 172.21.15.123:0/3730749457 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fd828072ec0 0x7fd828075380 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:42:49.364 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:49.362+0000 7fd84bcc2640 1 --2- 172.21.15.123:0/3730749457 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd844160310 0x7fd8441f7560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:42:49.364 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:49.362+0000 7fd84bcc2640 1 --2- 172.21.15.123:0/3730749457 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd84415f690 0x7fd8441464c0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:42:49.365 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:49.362+0000 7fd84bcc2640 1 --2- 172.21.15.123:0/3730749457 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fd84415bc40 0x7fd8440c52b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:42:49.365 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:49.362+0000 7fd84bcc2640 1 -- 172.21.15.123:0/3730749457 >> 172.21.15.123:0/3730749457 conn(0x7fd8440b1630 msgr2=0x7fd8440b1a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:42:49.365 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:49.362+0000 7fd84bcc2640 1 -- 172.21.15.123:0/3730749457 shutdown_connections 2024-09-16T15:42:49.365 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:49.362+0000 7fd84bcc2640 1 -- 172.21.15.123:0/3730749457 wait complete. 2024-09-16T15:42:49.388 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-09-16T15:42:49.514 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:49.510+0000 7f6d6d313640 1 -- 172.21.15.123:0/3494348492 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6d6815a9d0 msgr2=0x7f6d6806ebe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:42:49.514 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:49.510+0000 7f6d6d313640 1 --2- 172.21.15.123:0/3494348492 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6d6815a9d0 0x7f6d6806ebe0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f6d680724a0 tx=0x7f6d5801c800 comp rx=0 tx=0).stop 2024-09-16T15:42:49.514 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:49.510+0000 7f6d6d313640 1 -- 172.21.15.123:0/3494348492 shutdown_connections 2024-09-16T15:42:49.514 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:49.510+0000 7f6d6d313640 1 --2- 172.21.15.123:0/3494348492 >> v2:172.21.15.184:3300/0 conn(0x7f6d68072c70 0x7f6d680cc3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:42:49.515 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:49.510+0000 7f6d6d313640 1 --2- 172.21.15.123:0/3494348492 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6d6815a9d0 0x7f6d6806ebe0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:42:49.515 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:49.510+0000 7f6d6d313640 1 -- 172.21.15.123:0/3494348492 >> 172.21.15.123:0/3494348492 conn(0x7f6d680b1710 msgr2=0x7f6d680b1b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:42:49.515 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:49.510+0000 7f6d6d313640 1 -- 172.21.15.123:0/3494348492 shutdown_connections 2024-09-16T15:42:49.515 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:49.510+0000 7f6d6d313640 1 -- 172.21.15.123:0/3494348492 wait complete. 2024-09-16T15:42:49.515 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:49.514+0000 7f6d6d313640 1 Processor -- start 2024-09-16T15:42:49.515 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:49.514+0000 7f6d6d313640 1 -- start start 2024-09-16T15:42:49.517 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:49.514+0000 7f6d6d313640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6d68072720 0x7f6d68156370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:42:49.517 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:49.514+0000 7f6d6d313640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f6d68072c70 0x7f6d680cfad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:42:49.517 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:49.514+0000 7f6d6d313640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6d6815a9d0 0x7f6d680d0010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:42:49.517 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:49.514+0000 7f6d6d313640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f6d68168380 con 0x7f6d6815a9d0 2024-09-16T15:42:49.517 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:49.514+0000 7f6d6d313640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f6d68168200 con 0x7f6d68072720 2024-09-16T15:42:49.517 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:49.514+0000 7f6d6d313640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f6d68168500 con 0x7f6d68072c70 2024-09-16T15:42:49.517 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:49.514+0000 7f6d677fe640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6d6815a9d0 0x7f6d680d0010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:42:49.517 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:49.514+0000 7f6d677fe640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6d6815a9d0 0x7f6d680d0010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:44896/0 (socket says 172.21.15.123:44896) 2024-09-16T15:42:49.518 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:49.514+0000 7f6d677fe640 1 -- 172.21.15.123:0/2393469126 learned_addr learned my addr 172.21.15.123:0/2393469126 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:42:49.518 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:49.514+0000 7f6d66ffd640 1 --2- 172.21.15.123:0/2393469126 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6d68072720 0x7f6d68156370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:42:49.518 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:49.514+0000 7f6d667fc640 1 --2- 172.21.15.123:0/2393469126 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f6d68072c70 0x7f6d680cfad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:42:49.518 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:49.514+0000 7f6d677fe640 1 -- 172.21.15.123:0/2393469126 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f6d68072c70 msgr2=0x7f6d680cfad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:42:49.518 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:49.514+0000 7f6d677fe640 1 --2- 172.21.15.123:0/2393469126 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f6d68072c70 0x7f6d680cfad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:42:49.518 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:49.514+0000 7f6d677fe640 1 -- 172.21.15.123:0/2393469126 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6d68072720 msgr2=0x7f6d68156370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:42:49.518 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:49.514+0000 7f6d677fe640 1 --2- 172.21.15.123:0/2393469126 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6d68072720 0x7f6d68156370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:42:49.518 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:49.514+0000 7f6d677fe640 1 -- 172.21.15.123:0/2393469126 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6d68156e30 con 0x7f6d6815a9d0 2024-09-16T15:42:49.518 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:49.514+0000 7f6d677fe640 1 --2- 172.21.15.123:0/2393469126 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6d6815a9d0 0x7f6d680d0010 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f6d6000b970 tx=0x7f6d6000be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:42:49.518 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:49.514+0000 7f6d4f7fe640 1 -- 172.21.15.123:0/2393469126 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d60019070 con 0x7f6d6815a9d0 2024-09-16T15:42:49.518 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:49.514+0000 7f6d4f7fe640 1 -- 172.21.15.123:0/2393469126 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f6d6000cc30 con 0x7f6d6815a9d0 2024-09-16T15:42:49.518 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:49.514+0000 7f6d6d313640 1 -- 172.21.15.123:0/2393469126 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6d68157110 con 0x7f6d6815a9d0 2024-09-16T15:42:49.518 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:49.514+0000 7f6d4f7fe640 1 -- 172.21.15.123:0/2393469126 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d6000ce60 con 0x7f6d6815a9d0 2024-09-16T15:42:49.518 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:49.514+0000 7f6d6d313640 1 -- 172.21.15.123:0/2393469126 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6d68145930 con 0x7f6d6815a9d0 2024-09-16T15:42:49.519 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:49.514+0000 7f6d6d313640 1 -- 172.21.15.123:0/2393469126 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6d68145750 con 0x7f6d6815a9d0 2024-09-16T15:42:49.519 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:49.518+0000 7f6d4f7fe640 1 -- 172.21.15.123:0/2393469126 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 5) ==== 97080+0+0 (secure 0 0 0) 0x7f6d6001c070 con 0x7f6d6815a9d0 2024-09-16T15:42:49.519 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:49.518+0000 7f6d4f7fe640 1 --2- 172.21.15.123:0/2393469126 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f6d54072ec0 0x7f6d54075380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:42:49.519 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:49.518+0000 7f6d4f7fe640 1 -- 172.21.15.123:0/2393469126 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(18..18 src has 1..18) ==== 5227+0+0 (secure 0 0 0) 0x7f6d60093bd0 con 0x7f6d6815a9d0 2024-09-16T15:42:49.519 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:49.518+0000 7f6d66ffd640 1 --2- 172.21.15.123:0/2393469126 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f6d54072ec0 0x7f6d54075380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:42:49.520 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:49.518+0000 7f6d66ffd640 1 --2- 172.21.15.123:0/2393469126 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f6d54072ec0 0x7f6d54075380 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f6d580af000 tx=0x7f6d5801b040 comp rx=0 tx=0).ready entity=mgr.4135 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:42:49.523 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:49.522+0000 7f6d4f7fe640 1 -- 172.21.15.123:0/2393469126 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f6d68145750 con 0x7f6d6815a9d0 2024-09-16T15:42:49.696 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:49.694+0000 7f6d6d313640 1 -- 172.21.15.123:0/2393469126 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) -- 0x7f6d680aa150 con 0x7f6d6815a9d0 2024-09-16T15:42:50.372 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:50.370+0000 7f6d4f7fe640 1 -- 172.21.15.123:0/2393469126 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v19) ==== 170+0+0 (secure 0 0 0) 0x7f6d680aa150 con 0x7f6d6815a9d0 2024-09-16T15:42:50.372 INFO:teuthology.orchestra.run.smithi123.stderr:pool 'cephfs_data' created 2024-09-16T15:42:50.376 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:50.374+0000 7f6d6d313640 1 -- 172.21.15.123:0/2393469126 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f6d54072ec0 msgr2=0x7f6d54075380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:42:50.376 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:50.374+0000 7f6d6d313640 1 --2- 172.21.15.123:0/2393469126 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f6d54072ec0 0x7f6d54075380 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f6d580af000 tx=0x7f6d5801b040 comp rx=0 tx=0).stop 2024-09-16T15:42:50.376 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:50.374+0000 7f6d6d313640 1 -- 172.21.15.123:0/2393469126 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6d6815a9d0 msgr2=0x7f6d680d0010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:42:50.376 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:50.374+0000 7f6d6d313640 1 --2- 172.21.15.123:0/2393469126 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6d6815a9d0 0x7f6d680d0010 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f6d6000b970 tx=0x7f6d6000be30 comp rx=0 tx=0).stop 2024-09-16T15:42:50.376 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:50.374+0000 7f6d6d313640 1 -- 172.21.15.123:0/2393469126 shutdown_connections 2024-09-16T15:42:50.376 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:50.374+0000 7f6d6d313640 1 --2- 172.21.15.123:0/2393469126 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f6d54072ec0 0x7f6d54075380 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:42:50.376 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:50.374+0000 7f6d6d313640 1 --2- 172.21.15.123:0/2393469126 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6d6815a9d0 0x7f6d680d0010 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:42:50.376 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:50.374+0000 7f6d6d313640 1 --2- 172.21.15.123:0/2393469126 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f6d68072c70 0x7f6d680cfad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:42:50.376 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:50.374+0000 7f6d6d313640 1 --2- 172.21.15.123:0/2393469126 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6d68072720 0x7f6d68156370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:42:50.376 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:50.374+0000 7f6d6d313640 1 -- 172.21.15.123:0/2393469126 >> 172.21.15.123:0/2393469126 conn(0x7f6d680b1710 msgr2=0x7f6d6815d740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:42:50.377 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:50.374+0000 7f6d6d313640 1 -- 172.21.15.123:0/2393469126 shutdown_connections 2024-09-16T15:42:50.377 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:50.374+0000 7f6d6d313640 1 -- 172.21.15.123:0/2393469126 wait complete. 2024-09-16T15:42:50.413 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-09-16T15:42:50.540 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:50.538+0000 7fdb953a0640 1 -- 172.21.15.123:0/1402088736 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdb900cbb40 msgr2=0x7fdb900cbf80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:42:50.540 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:50.538+0000 7fdb953a0640 1 --2- 172.21.15.123:0/1402088736 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdb900cbb40 0x7fdb900cbf80 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fdb84009930 tx=0x7fdb8401c8b0 comp rx=0 tx=0).stop 2024-09-16T15:42:50.540 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:50.538+0000 7fdb953a0640 1 -- 172.21.15.123:0/1402088736 shutdown_connections 2024-09-16T15:42:50.540 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:50.538+0000 7fdb953a0640 1 --2- 172.21.15.123:0/1402088736 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdb900cbb40 0x7fdb900cbf80 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:42:50.540 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:50.538+0000 7fdb953a0640 1 --2- 172.21.15.123:0/1402088736 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fdb9015a100 0x7fdb90072f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:42:50.540 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:50.538+0000 7fdb953a0640 1 -- 172.21.15.123:0/1402088736 >> 172.21.15.123:0/1402088736 conn(0x7fdb900b1710 msgr2=0x7fdb900b1b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:42:50.540 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:50.538+0000 7fdb953a0640 1 -- 172.21.15.123:0/1402088736 shutdown_connections 2024-09-16T15:42:50.540 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:50.538+0000 7fdb953a0640 1 -- 172.21.15.123:0/1402088736 wait complete. 2024-09-16T15:42:50.541 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:50.538+0000 7fdb953a0640 1 Processor -- start 2024-09-16T15:42:50.541 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:50.538+0000 7fdb953a0640 1 -- start start 2024-09-16T15:42:50.543 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:50.538+0000 7fdb953a0640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fdb900cbb40 0x7fdb900c3280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:42:50.543 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:50.538+0000 7fdb953a0640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdb9015a100 0x7fdb900ce420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:42:50.543 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:50.538+0000 7fdb953a0640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fdb9015ddb0 0x7fdb900ce960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:42:50.543 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:50.538+0000 7fdb953a0640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fdb901681d0 con 0x7fdb9015a100 2024-09-16T15:42:50.543 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:50.538+0000 7fdb953a0640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fdb90168050 con 0x7fdb9015ddb0 2024-09-16T15:42:50.543 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:50.538+0000 7fdb953a0640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fdb90168350 con 0x7fdb900cbb40 2024-09-16T15:42:50.543 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:50.538+0000 7fdb8e7fc640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdb9015a100 0x7fdb900ce420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:42:50.544 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:50.538+0000 7fdb8e7fc640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdb9015a100 0x7fdb900ce420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:44920/0 (socket says 172.21.15.123:44920) 2024-09-16T15:42:50.544 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:50.538+0000 7fdb8e7fc640 1 -- 172.21.15.123:0/1144248813 learned_addr learned my addr 172.21.15.123:0/1144248813 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:42:50.544 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:50.538+0000 7fdb8effd640 1 --2- 172.21.15.123:0/1144248813 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fdb900cbb40 0x7fdb900c3280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:42:50.544 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:50.538+0000 7fdb8f7fe640 1 --2- 172.21.15.123:0/1144248813 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fdb9015ddb0 0x7fdb900ce960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:42:50.544 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:50.538+0000 7fdb8e7fc640 1 -- 172.21.15.123:0/1144248813 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fdb900cbb40 msgr2=0x7fdb900c3280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:42:50.544 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:50.538+0000 7fdb8e7fc640 1 --2- 172.21.15.123:0/1144248813 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fdb900cbb40 0x7fdb900c3280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:42:50.544 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:50.542+0000 7fdb8e7fc640 1 -- 172.21.15.123:0/1144248813 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fdb9015ddb0 msgr2=0x7fdb900ce960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:42:50.544 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:50.542+0000 7fdb8e7fc640 1 --2- 172.21.15.123:0/1144248813 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fdb9015ddb0 0x7fdb900ce960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:42:50.544 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:50.542+0000 7fdb8e7fc640 1 -- 172.21.15.123:0/1144248813 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdb900c3ab0 con 0x7fdb9015a100 2024-09-16T15:42:50.544 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:50.542+0000 7fdb8effd640 1 --2- 172.21.15.123:0/1144248813 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fdb900cbb40 0x7fdb900c3280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T15:42:50.544 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:50.542+0000 7fdb8f7fe640 1 --2- 172.21.15.123:0/1144248813 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fdb9015ddb0 0x7fdb900ce960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T15:42:50.544 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:50.542+0000 7fdb8e7fc640 1 --2- 172.21.15.123:0/1144248813 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdb9015a100 0x7fdb900ce420 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fdb8401c6e0 tx=0x7fdb840040c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:42:50.545 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:50.542+0000 7fdb777fe640 1 -- 172.21.15.123:0/1144248813 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb840b2040 con 0x7fdb9015a100 2024-09-16T15:42:50.545 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:50.542+0000 7fdb777fe640 1 -- 172.21.15.123:0/1144248813 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fdb840043a0 con 0x7fdb9015a100 2024-09-16T15:42:50.545 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:50.542+0000 7fdb953a0640 1 -- 172.21.15.123:0/1144248813 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdb900c3d90 con 0x7fdb9015a100 2024-09-16T15:42:50.545 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:50.542+0000 7fdb777fe640 1 -- 172.21.15.123:0/1144248813 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb84003860 con 0x7fdb9015a100 2024-09-16T15:42:50.545 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:50.542+0000 7fdb953a0640 1 -- 172.21.15.123:0/1144248813 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdb900c37c0 con 0x7fdb9015a100 2024-09-16T15:42:50.546 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:50.542+0000 7fdb953a0640 1 -- 172.21.15.123:0/1144248813 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdb900cbf80 con 0x7fdb9015a100 2024-09-16T15:42:50.546 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:50.542+0000 7fdb777fe640 1 -- 172.21.15.123:0/1144248813 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 5) ==== 97080+0+0 (secure 0 0 0) 0x7fdb840bf020 con 0x7fdb9015a100 2024-09-16T15:42:50.546 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:50.542+0000 7fdb777fe640 1 --2- 172.21.15.123:0/1144248813 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fdb7c072ec0 0x7fdb7c075380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:42:50.546 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:50.542+0000 7fdb777fe640 1 -- 172.21.15.123:0/1144248813 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(19..19 src has 1..19) ==== 5609+0+0 (secure 0 0 0) 0x7fdb8412e790 con 0x7fdb9015a100 2024-09-16T15:42:50.546 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:50.542+0000 7fdb8effd640 1 --2- 172.21.15.123:0/1144248813 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fdb7c072ec0 0x7fdb7c075380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:42:50.546 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:50.546+0000 7fdb8effd640 1 --2- 172.21.15.123:0/1144248813 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fdb7c072ec0 0x7fdb7c075380 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fdb80005fd0 tx=0x7fdb80005d40 comp rx=0 tx=0).ready entity=mgr.4135 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:42:50.549 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:50.546+0000 7fdb777fe640 1 -- 172.21.15.123:0/1144248813 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7fdb840c6020 con 0x7fdb9015a100 2024-09-16T15:42:50.760 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:50.758+0000 7fdb953a0640 1 -- 172.21.15.123:0/1144248813 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) -- 0x7fdb900730c0 con 0x7fdb9015a100 2024-09-16T15:42:51.400 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:51.398+0000 7fdb777fe640 1 -- 172.21.15.123:0/1144248813 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 2 and data pool 3 v4) ==== 172+0+0 (secure 0 0 0) 0x7fdb840164d0 con 0x7fdb9015a100 2024-09-16T15:42:51.405 INFO:teuthology.orchestra.run.smithi123.stderr:new fs with metadata pool 2 and data pool 3 2024-09-16T15:42:51.415 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:51.414+0000 7fdb953a0640 1 -- 172.21.15.123:0/1144248813 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fdb7c072ec0 msgr2=0x7fdb7c075380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:42:51.415 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:51.414+0000 7fdb953a0640 1 --2- 172.21.15.123:0/1144248813 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fdb7c072ec0 0x7fdb7c075380 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fdb80005fd0 tx=0x7fdb80005d40 comp rx=0 tx=0).stop 2024-09-16T15:42:51.415 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:51.414+0000 7fdb953a0640 1 -- 172.21.15.123:0/1144248813 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdb9015a100 msgr2=0x7fdb900ce420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:42:51.416 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:51.414+0000 7fdb953a0640 1 --2- 172.21.15.123:0/1144248813 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdb9015a100 0x7fdb900ce420 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fdb8401c6e0 tx=0x7fdb840040c0 comp rx=0 tx=0).stop 2024-09-16T15:42:51.416 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:51.414+0000 7fdb953a0640 1 -- 172.21.15.123:0/1144248813 shutdown_connections 2024-09-16T15:42:51.416 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:51.414+0000 7fdb953a0640 1 --2- 172.21.15.123:0/1144248813 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fdb7c072ec0 0x7fdb7c075380 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:42:51.416 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:51.414+0000 7fdb953a0640 1 --2- 172.21.15.123:0/1144248813 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fdb9015ddb0 0x7fdb900ce960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:42:51.416 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:51.414+0000 7fdb953a0640 1 --2- 172.21.15.123:0/1144248813 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdb9015a100 0x7fdb900ce420 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:42:51.416 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:51.414+0000 7fdb953a0640 1 --2- 172.21.15.123:0/1144248813 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fdb900cbb40 0x7fdb900c3280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:42:51.416 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:51.414+0000 7fdb953a0640 1 -- 172.21.15.123:0/1144248813 >> 172.21.15.123:0/1144248813 conn(0x7fdb900b1710 msgr2=0x7fdb9015c6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:42:51.417 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:51.414+0000 7fdb953a0640 1 -- 172.21.15.123:0/1144248813 shutdown_connections 2024-09-16T15:42:51.417 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:51.414+0000 7fdb953a0640 1 -- 172.21.15.123:0/1144248813 wait complete. 2024-09-16T15:42:51.447 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-16T15:42:51.575 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:51.574+0000 7f9d0f4bb640 1 -- 172.21.15.123:0/2290814991 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9d080b2910 msgr2=0x7f9d080ad030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:42:51.575 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:51.574+0000 7f9d0f4bb640 1 --2- 172.21.15.123:0/2290814991 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9d080b2910 0x7f9d080ad030 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f9d080ccd20 tx=0x7f9cf801db30 comp rx=0 tx=0).stop 2024-09-16T15:42:51.576 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:51.574+0000 7f9d0f4bb640 1 -- 172.21.15.123:0/2290814991 shutdown_connections 2024-09-16T15:42:51.576 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:51.574+0000 7f9d0f4bb640 1 --2- 172.21.15.123:0/2290814991 >> v2:172.21.15.184:3300/0 conn(0x7f9d08074800 0x7f9d080ad570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:42:51.576 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:51.574+0000 7f9d0f4bb640 1 --2- 172.21.15.123:0/2290814991 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9d080b2910 0x7f9d080ad030 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:42:51.576 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:51.574+0000 7f9d0f4bb640 1 --2- 172.21.15.123:0/2290814991 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f9d08074420 0x7f9d080ce190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:42:51.576 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:51.574+0000 7f9d0f4bb640 1 -- 172.21.15.123:0/2290814991 >> 172.21.15.123:0/2290814991 conn(0x7f9d080b1df0 msgr2=0x7f9d080b2200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:42:51.576 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:51.574+0000 7f9d0f4bb640 1 -- 172.21.15.123:0/2290814991 shutdown_connections 2024-09-16T15:42:51.576 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:51.574+0000 7f9d0f4bb640 1 -- 172.21.15.123:0/2290814991 wait complete. 2024-09-16T15:42:51.579 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:51.574+0000 7f9d0f4bb640 1 Processor -- start 2024-09-16T15:42:51.579 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:51.574+0000 7f9d0f4bb640 1 -- start start 2024-09-16T15:42:51.579 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:51.574+0000 7f9d0f4bb640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f9d08074420 0x7f9d08153840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:42:51.579 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:51.574+0000 7f9d0f4bb640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f9d08074800 0x7f9d08099530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:42:51.579 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:51.574+0000 7f9d0f4bb640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9d080b2910 0x7f9d080b3250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:42:51.579 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:51.574+0000 7f9d0f4bb640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f9d080aff20 con 0x7f9d080b2910 2024-09-16T15:42:51.579 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:51.574+0000 7f9d0f4bb640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f9d080afda0 con 0x7f9d08074420 2024-09-16T15:42:51.579 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:51.574+0000 7f9d0f4bb640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f9d080b00a0 con 0x7f9d08074800 2024-09-16T15:42:51.580 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:51.574+0000 7f9d0da58640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9d080b2910 0x7f9d080b3250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:42:51.580 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:51.574+0000 7f9d0da58640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9d080b2910 0x7f9d080b3250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:44940/0 (socket says 172.21.15.123:44940) 2024-09-16T15:42:51.580 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:51.574+0000 7f9d0da58640 1 -- 172.21.15.123:0/2307535376 learned_addr learned my addr 172.21.15.123:0/2307535376 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:42:51.580 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:51.574+0000 7f9d0d257640 1 --2- 172.21.15.123:0/2307535376 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f9d08074420 0x7f9d08153840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:42:51.580 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:51.574+0000 7f9d0ca56640 1 --2- 172.21.15.123:0/2307535376 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f9d08074800 0x7f9d08099530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:42:51.580 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:51.574+0000 7f9d0da58640 1 -- 172.21.15.123:0/2307535376 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f9d08074800 msgr2=0x7f9d08099530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:42:51.580 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:51.574+0000 7f9d0da58640 1 --2- 172.21.15.123:0/2307535376 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f9d08074800 0x7f9d08099530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:42:51.580 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:51.574+0000 7f9d0da58640 1 -- 172.21.15.123:0/2307535376 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f9d08074420 msgr2=0x7f9d08153840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:42:51.580 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:51.574+0000 7f9d0da58640 1 --2- 172.21.15.123:0/2307535376 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f9d08074420 0x7f9d08153840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:42:51.580 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:51.574+0000 7f9d0da58640 1 -- 172.21.15.123:0/2307535376 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9d081fb9a0 con 0x7f9d080b2910 2024-09-16T15:42:51.580 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:51.574+0000 7f9d0ca56640 1 --2- 172.21.15.123:0/2307535376 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f9d08074800 0x7f9d08099530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T15:42:51.581 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:51.574+0000 7f9d0da58640 1 --2- 172.21.15.123:0/2307535376 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9d080b2910 0x7f9d080b3250 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f9d0000cce0 tx=0x7f9d0000a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:42:51.581 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:51.574+0000 7f9cfdffb640 1 -- 172.21.15.123:0/2307535376 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d0000f450 con 0x7f9d080b2910 2024-09-16T15:42:51.581 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:51.574+0000 7f9cfdffb640 1 -- 172.21.15.123:0/2307535376 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f9d0000f700 con 0x7f9d080b2910 2024-09-16T15:42:51.581 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:51.574+0000 7f9d0f4bb640 1 -- 172.21.15.123:0/2307535376 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9d081fbc80 con 0x7f9d080b2910 2024-09-16T15:42:51.581 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:51.574+0000 7f9cfdffb640 1 -- 172.21.15.123:0/2307535376 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d0000fb50 con 0x7f9d080b2910 2024-09-16T15:42:51.581 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:51.578+0000 7f9d0f4bb640 1 -- 172.21.15.123:0/2307535376 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9d081fc240 con 0x7f9d080b2910 2024-09-16T15:42:51.581 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:51.578+0000 7f9d0f4bb640 1 -- 172.21.15.123:0/2307535376 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9d080b0630 con 0x7f9d080b2910 2024-09-16T15:42:51.582 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:51.578+0000 7f9cfdffb640 1 -- 172.21.15.123:0/2307535376 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 5) ==== 97080+0+0 (secure 0 0 0) 0x7f9d0000fcf0 con 0x7f9d080b2910 2024-09-16T15:42:51.582 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:51.578+0000 7f9cfdffb640 1 --2- 172.21.15.123:0/2307535376 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f9cec072ec0 0x7f9cec075380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:42:51.582 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:51.578+0000 7f9cfdffb640 1 -- 172.21.15.123:0/2307535376 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f9d000949d0 con 0x7f9d080b2910 2024-09-16T15:42:51.582 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:51.578+0000 7f9d0d257640 1 --2- 172.21.15.123:0/2307535376 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f9cec072ec0 0x7f9cec075380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:42:51.582 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:51.578+0000 7f9d0d257640 1 --2- 172.21.15.123:0/2307535376 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f9cec072ec0 0x7f9cec075380 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f9d04005950 tx=0x7f9d0400a4e0 comp rx=0 tx=0).ready entity=mgr.4135 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:42:51.584 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:51.582+0000 7f9cfdffb640 1 -- 172.21.15.123:0/2307535376 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f9d00014030 con 0x7f9d080b2910 2024-09-16T15:42:51.757 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:51.754+0000 7f9d0f4bb640 1 -- 172.21.15.123:0/2307535376 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f9d080aef70 con 0x7f9d080b2910 2024-09-16T15:42:51.759 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:51.758+0000 7f9cfdffb640 1 -- 172.21.15.123:0/2307535376 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) ==== 74+0+16738 (secure 0 0 0) 0x7f9d0005fc20 con 0x7f9d080b2910 2024-09-16T15:42:51.760 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-09-16T15:42:51.760 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":20,"fsid":"56077f89-18c8-486f-ae7e-eecc0bb7822c","created":"2024-09-16T15:42:26.364892+0000","modified":"2024-09-16T15:42:50.762353+0000","last_up_change":"2024-09-16T15:42:38.707234+0000","last_in_change":"2024-09-16T15:42:29.423662+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-16T15:42:39.287764+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-09-16T15:42:48.583563+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-09-16T15:42:49.704665+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9e7c1955-a607-4671-9be7-f2ef760ceeaa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6800","nonce":3882044969},{"type":"v1","addr":"172.21.15.123:6801","nonce":3882044969}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6802","nonce":3882044969},{"type":"v1","addr":"172.21.15.123:6803","nonce":3882044969}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6806","nonce":3882044969},{"type":"v1","addr":"172.21.15.123:6807","nonce":3882044969}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6804","nonce":3882044969},{"type":"v1","addr":"172.21.15.123:6805","nonce":3882044969}]},"public_addr":"172.21.15.123:6801/3882044969","cluster_addr":"172.21.15.123:6803/3882044969","heartbeat_back_addr":"172.21.15.123:6807/3882044969","heartbeat_front_addr":"172.21.15.123:6805/3882044969","state":["exists","up"]},{"osd":1,"uuid":"c540cfde-0977-4742-9b17-fd634668e010","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6816","nonce":4037001850},{"type":"v1","addr":"172.21.15.123:6817","nonce":4037001850}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6818","nonce":4037001850},{"type":"v1","addr":"172.21.15.123:6819","nonce":4037001850}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6822","nonce":4037001850},{"type":"v1","addr":"172.21.15.123:6823","nonce":4037001850}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6820","nonce":4037001850},{"type":"v1","addr":"172.21.15.123:6821","nonce":4037001850}]},"public_addr":"172.21.15.123:6817/4037001850","cluster_addr":"172.21.15.123:6819/4037001850","heartbeat_back_addr":"172.21.15.123:6823/4037001850","heartbeat_front_addr":"172.21.15.123:6821/4037001850","state":["exists","up"]},{"osd":2,"uuid":"9864a496-f169-486b-95c4-1d59162793a7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6824","nonce":1332143088},{"type":"v1","addr":"172.21.15.123:6825","nonce":1332143088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6826","nonce":1332143088},{"type":"v1","addr":"172.21.15.123:6827","nonce":1332143088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6830","nonce":1332143088},{"type":"v1","addr":"172.21.15.123:6831","nonce":1332143088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6828","nonce":1332143088},{"type":"v1","addr":"172.21.15.123:6829","nonce":1332143088}]},"public_addr":"172.21.15.123:6825/1332143088","cluster_addr":"172.21.15.123:6827/1332143088","heartbeat_back_addr":"172.21.15.123:6831/1332143088","heartbeat_front_addr":"172.21.15.123:6829/1332143088","state":["exists","up"]},{"osd":3,"uuid":"c6d1f817-f074-4325-95e1-578eba13edda","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6808","nonce":988586998},{"type":"v1","addr":"172.21.15.123:6809","nonce":988586998}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6810","nonce":988586998},{"type":"v1","addr":"172.21.15.123:6811","nonce":988586998}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6814","nonce":988586998},{"type":"v1","addr":"172.21.15.123:6815","nonce":988586998}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6812","nonce":988586998},{"type":"v1","addr":"172.21.15.123:6813","nonce":988586998}]},"public_addr":"172.21.15.123:6809/988586998","cluster_addr":"172.21.15.123:6811/988586998","heartbeat_back_addr":"172.21.15.123:6815/988586998","heartbeat_front_addr":"172.21.15.123:6813/988586998","state":["exists","up"]},{"osd":4,"uuid":"200a8df4-50ef-40a5-b48a-58981269c086","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6816","nonce":3387813890},{"type":"v1","addr":"172.21.15.184:6817","nonce":3387813890}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6818","nonce":3387813890},{"type":"v1","addr":"172.21.15.184:6819","nonce":3387813890}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6822","nonce":3387813890},{"type":"v1","addr":"172.21.15.184:6823","nonce":3387813890}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6820","nonce":3387813890},{"type":"v1","addr":"172.21.15.184:6821","nonce":3387813890}]},"public_addr":"172.21.15.184:6817/3387813890","cluster_addr":"172.21.15.184:6819/3387813890","heartbeat_back_addr":"172.21.15.184:6823/3387813890","heartbeat_front_addr":"172.21.15.184:6821/3387813890","state":["exists","up"]},{"osd":5,"uuid":"32d5a900-da3d-412c-bc19-13861a2368d8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6808","nonce":2680916017},{"type":"v1","addr":"172.21.15.184:6809","nonce":2680916017}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6810","nonce":2680916017},{"type":"v1","addr":"172.21.15.184:6811","nonce":2680916017}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6814","nonce":2680916017},{"type":"v1","addr":"172.21.15.184:6815","nonce":2680916017}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6812","nonce":2680916017},{"type":"v1","addr":"172.21.15.184:6813","nonce":2680916017}]},"public_addr":"172.21.15.184:6809/2680916017","cluster_addr":"172.21.15.184:6811/2680916017","heartbeat_back_addr":"172.21.15.184:6815/2680916017","heartbeat_front_addr":"172.21.15.184:6813/2680916017","state":["exists","up"]},{"osd":6,"uuid":"cde6f39e-aa1b-4260-ac19-774ee6aaca72","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6824","nonce":2183120457},{"type":"v1","addr":"172.21.15.184:6825","nonce":2183120457}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6826","nonce":2183120457},{"type":"v1","addr":"172.21.15.184:6827","nonce":2183120457}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6830","nonce":2183120457},{"type":"v1","addr":"172.21.15.184:6831","nonce":2183120457}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6828","nonce":2183120457},{"type":"v1","addr":"172.21.15.184:6829","nonce":2183120457}]},"public_addr":"172.21.15.184:6825/2183120457","cluster_addr":"172.21.15.184:6827/2183120457","heartbeat_back_addr":"172.21.15.184:6831/2183120457","heartbeat_front_addr":"172.21.15.184:6829/2183120457","state":["exists","up"]},{"osd":7,"uuid":"4f381778-eee1-47c6-804f-df0e6b33e313","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6800","nonce":4102957285},{"type":"v1","addr":"172.21.15.184:6801","nonce":4102957285}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6802","nonce":4102957285},{"type":"v1","addr":"172.21.15.184:6803","nonce":4102957285}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6806","nonce":4102957285},{"type":"v1","addr":"172.21.15.184:6807","nonce":4102957285}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6804","nonce":4102957285},{"type":"v1","addr":"172.21.15.184:6805","nonce":4102957285}]},"public_addr":"172.21.15.184:6801/4102957285","cluster_addr":"172.21.15.184:6803/4102957285","heartbeat_back_addr":"172.21.15.184:6807/4102957285","heartbeat_front_addr":"172.21.15.184:6805/4102957285","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:35.972210+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.222007+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.198617+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.385663+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.358935+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.156066+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.752973+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:35.682846+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T15:42:51.762 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:51.758+0000 7f9d0f4bb640 1 -- 172.21.15.123:0/2307535376 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f9cec072ec0 msgr2=0x7f9cec075380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:42:51.762 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:51.758+0000 7f9d0f4bb640 1 --2- 172.21.15.123:0/2307535376 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f9cec072ec0 0x7f9cec075380 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f9d04005950 tx=0x7f9d0400a4e0 comp rx=0 tx=0).stop 2024-09-16T15:42:51.762 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:51.758+0000 7f9d0f4bb640 1 -- 172.21.15.123:0/2307535376 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9d080b2910 msgr2=0x7f9d080b3250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:42:51.762 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:51.758+0000 7f9d0f4bb640 1 --2- 172.21.15.123:0/2307535376 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9d080b2910 0x7f9d080b3250 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f9d0000cce0 tx=0x7f9d0000a620 comp rx=0 tx=0).stop 2024-09-16T15:42:51.762 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:51.758+0000 7f9d0f4bb640 1 -- 172.21.15.123:0/2307535376 shutdown_connections 2024-09-16T15:42:51.762 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:51.758+0000 7f9d0f4bb640 1 --2- 172.21.15.123:0/2307535376 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f9cec072ec0 0x7f9cec075380 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:42:51.762 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:51.758+0000 7f9d0f4bb640 1 --2- 172.21.15.123:0/2307535376 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9d080b2910 0x7f9d080b3250 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:42:51.762 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:51.758+0000 7f9d0f4bb640 1 --2- 172.21.15.123:0/2307535376 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f9d08074800 0x7f9d08099530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:42:51.762 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:51.758+0000 7f9d0f4bb640 1 --2- 172.21.15.123:0/2307535376 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f9d08074420 0x7f9d08153840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:42:51.762 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:51.758+0000 7f9d0f4bb640 1 -- 172.21.15.123:0/2307535376 >> 172.21.15.123:0/2307535376 conn(0x7f9d080b1df0 msgr2=0x7f9d08156ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:42:51.763 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:51.758+0000 7f9d0f4bb640 1 -- 172.21.15.123:0/2307535376 shutdown_connections 2024-09-16T15:42:51.763 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:51.758+0000 7f9d0f4bb640 1 -- 172.21.15.123:0/2307535376 wait complete. 2024-09-16T15:42:51.779 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-16T15:42:51.906 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:51.902+0000 7f95c1d7c640 1 -- 172.21.15.123:0/1479961345 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f95bc0b1490 msgr2=0x7f95bc0cbab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:42:51.907 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:51.902+0000 7f95c1d7c640 1 --2- 172.21.15.123:0/1479961345 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f95bc0b1490 0x7f95bc0cbab0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f95a4009840 tx=0x7f95a401c7f0 comp rx=0 tx=0).stop 2024-09-16T15:42:51.907 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:51.902+0000 7f95c1d7c640 1 -- 172.21.15.123:0/1479961345 shutdown_connections 2024-09-16T15:42:51.907 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:51.902+0000 7f95c1d7c640 1 --2- 172.21.15.123:0/1479961345 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f95bc0b1490 0x7f95bc0cbab0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:42:51.907 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:51.902+0000 7f95c1d7c640 1 -- 172.21.15.123:0/1479961345 >> 172.21.15.123:0/1479961345 conn(0x7f95bc074230 msgr2=0x7f95bc074610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:42:51.907 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:51.906+0000 7f95c1d7c640 1 -- 172.21.15.123:0/1479961345 shutdown_connections 2024-09-16T15:42:51.907 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:51.906+0000 7f95c1d7c640 1 -- 172.21.15.123:0/1479961345 wait complete. 2024-09-16T15:42:51.908 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:51.906+0000 7f95c1d7c640 1 Processor -- start 2024-09-16T15:42:51.908 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:51.906+0000 7f95c1d7c640 1 -- start start 2024-09-16T15:42:51.910 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:51.906+0000 7f95c1d7c640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f95bc068960 0x7f95bc0cf720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:42:51.910 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:51.906+0000 7f95c1d7c640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f95bc0b1490 0x7f95bc099430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:42:51.910 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:51.906+0000 7f95c1d7c640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f95bc15bd80 0x7f95bc099970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:42:51.910 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:51.906+0000 7f95c1d7c640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f95bc16a520 con 0x7f95bc0b1490 2024-09-16T15:42:51.910 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:51.906+0000 7f95c1d7c640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f95bc16a3a0 con 0x7f95bc068960 2024-09-16T15:42:51.911 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:51.906+0000 7f95c1d7c640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f95bc16a6a0 con 0x7f95bc15bd80 2024-09-16T15:42:51.911 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:51.906+0000 7f95baffd640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f95bc0b1490 0x7f95bc099430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:42:51.911 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:51.906+0000 7f95baffd640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f95bc0b1490 0x7f95bc099430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:44970/0 (socket says 172.21.15.123:44970) 2024-09-16T15:42:51.911 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:51.906+0000 7f95baffd640 1 -- 172.21.15.123:0/3416302660 learned_addr learned my addr 172.21.15.123:0/3416302660 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:42:51.911 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:51.906+0000 7f95baffd640 1 -- 172.21.15.123:0/3416302660 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f95bc15bd80 msgr2=0x7f95bc099970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:42:51.911 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:51.906+0000 7f95baffd640 1 --2- 172.21.15.123:0/3416302660 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f95bc15bd80 0x7f95bc099970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:42:51.911 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:51.906+0000 7f95baffd640 1 -- 172.21.15.123:0/3416302660 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f95bc068960 msgr2=0x7f95bc0cf720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:42:51.911 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:51.906+0000 7f95bb7fe640 1 --2- 172.21.15.123:0/3416302660 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f95bc068960 0x7f95bc0cf720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:42:51.911 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:51.906+0000 7f95baffd640 1 --2- 172.21.15.123:0/3416302660 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f95bc068960 0x7f95bc0cf720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:42:51.911 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:51.906+0000 7f95baffd640 1 -- 172.21.15.123:0/3416302660 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f95bc156e10 con 0x7f95bc0b1490 2024-09-16T15:42:51.911 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:51.906+0000 7f95baffd640 1 --2- 172.21.15.123:0/3416302660 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f95bc0b1490 0x7f95bc099430 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f95a401aea0 tx=0x7f95a40040c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:42:51.912 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:51.906+0000 7f95b3fff640 1 -- 172.21.15.123:0/3416302660 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95a40bb070 con 0x7f95bc0b1490 2024-09-16T15:42:51.912 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:51.906+0000 7f95b3fff640 1 -- 172.21.15.123:0/3416302660 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f95a4005030 con 0x7f95bc0b1490 2024-09-16T15:42:51.912 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:51.906+0000 7f95c1d7c640 1 -- 172.21.15.123:0/3416302660 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f95bc1570b0 con 0x7f95bc0b1490 2024-09-16T15:42:51.912 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:51.906+0000 7f95b3fff640 1 -- 172.21.15.123:0/3416302660 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95a40b2040 con 0x7f95bc0b1490 2024-09-16T15:42:51.912 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:51.906+0000 7f95c1d7c640 1 -- 172.21.15.123:0/3416302660 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f95bc157670 con 0x7f95bc0b1490 2024-09-16T15:42:51.912 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:51.910+0000 7f95c1d7c640 1 -- 172.21.15.123:0/3416302660 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f95bc0b0d80 con 0x7f95bc0b1490 2024-09-16T15:42:51.913 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:51.910+0000 7f95b3fff640 1 -- 172.21.15.123:0/3416302660 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 5) ==== 97080+0+0 (secure 0 0 0) 0x7f95a40be070 con 0x7f95bc0b1490 2024-09-16T15:42:51.913 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:51.910+0000 7f95b3fff640 1 --2- 172.21.15.123:0/3416302660 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f959c072ec0 0x7f959c075380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:42:51.913 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:51.910+0000 7f95b3fff640 1 -- 172.21.15.123:0/3416302660 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f95a412d890 con 0x7f95bc0b1490 2024-09-16T15:42:51.913 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:51.910+0000 7f95bb7fe640 1 --2- 172.21.15.123:0/3416302660 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f959c072ec0 0x7f959c075380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:42:51.913 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:51.910+0000 7f95bb7fe640 1 --2- 172.21.15.123:0/3416302660 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f959c072ec0 0x7f959c075380 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f95ac00a750 tx=0x7f95ac008040 comp rx=0 tx=0).ready entity=mgr.4135 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:42:51.915 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:51.914+0000 7f95b3fff640 1 -- 172.21.15.123:0/3416302660 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f95a4016410 con 0x7f95bc0b1490 2024-09-16T15:42:52.089 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:52.086+0000 7f95c1d7c640 1 -- 172.21.15.123:0/3416302660 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f95bc152e50 con 0x7f95bc0b1490 2024-09-16T15:42:52.090 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:52.090+0000 7f95b3fff640 1 -- 172.21.15.123:0/3416302660 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) ==== 74+0+16738 (secure 0 0 0) 0x7f95a40f8b70 con 0x7f95bc0b1490 2024-09-16T15:42:52.091 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-09-16T15:42:52.091 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":20,"fsid":"56077f89-18c8-486f-ae7e-eecc0bb7822c","created":"2024-09-16T15:42:26.364892+0000","modified":"2024-09-16T15:42:50.762353+0000","last_up_change":"2024-09-16T15:42:38.707234+0000","last_in_change":"2024-09-16T15:42:29.423662+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-16T15:42:39.287764+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-09-16T15:42:48.583563+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-09-16T15:42:49.704665+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9e7c1955-a607-4671-9be7-f2ef760ceeaa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6800","nonce":3882044969},{"type":"v1","addr":"172.21.15.123:6801","nonce":3882044969}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6802","nonce":3882044969},{"type":"v1","addr":"172.21.15.123:6803","nonce":3882044969}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6806","nonce":3882044969},{"type":"v1","addr":"172.21.15.123:6807","nonce":3882044969}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6804","nonce":3882044969},{"type":"v1","addr":"172.21.15.123:6805","nonce":3882044969}]},"public_addr":"172.21.15.123:6801/3882044969","cluster_addr":"172.21.15.123:6803/3882044969","heartbeat_back_addr":"172.21.15.123:6807/3882044969","heartbeat_front_addr":"172.21.15.123:6805/3882044969","state":["exists","up"]},{"osd":1,"uuid":"c540cfde-0977-4742-9b17-fd634668e010","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6816","nonce":4037001850},{"type":"v1","addr":"172.21.15.123:6817","nonce":4037001850}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6818","nonce":4037001850},{"type":"v1","addr":"172.21.15.123:6819","nonce":4037001850}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6822","nonce":4037001850},{"type":"v1","addr":"172.21.15.123:6823","nonce":4037001850}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6820","nonce":4037001850},{"type":"v1","addr":"172.21.15.123:6821","nonce":4037001850}]},"public_addr":"172.21.15.123:6817/4037001850","cluster_addr":"172.21.15.123:6819/4037001850","heartbeat_back_addr":"172.21.15.123:6823/4037001850","heartbeat_front_addr":"172.21.15.123:6821/4037001850","state":["exists","up"]},{"osd":2,"uuid":"9864a496-f169-486b-95c4-1d59162793a7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6824","nonce":1332143088},{"type":"v1","addr":"172.21.15.123:6825","nonce":1332143088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6826","nonce":1332143088},{"type":"v1","addr":"172.21.15.123:6827","nonce":1332143088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6830","nonce":1332143088},{"type":"v1","addr":"172.21.15.123:6831","nonce":1332143088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6828","nonce":1332143088},{"type":"v1","addr":"172.21.15.123:6829","nonce":1332143088}]},"public_addr":"172.21.15.123:6825/1332143088","cluster_addr":"172.21.15.123:6827/1332143088","heartbeat_back_addr":"172.21.15.123:6831/1332143088","heartbeat_front_addr":"172.21.15.123:6829/1332143088","state":["exists","up"]},{"osd":3,"uuid":"c6d1f817-f074-4325-95e1-578eba13edda","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6808","nonce":988586998},{"type":"v1","addr":"172.21.15.123:6809","nonce":988586998}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6810","nonce":988586998},{"type":"v1","addr":"172.21.15.123:6811","nonce":988586998}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6814","nonce":988586998},{"type":"v1","addr":"172.21.15.123:6815","nonce":988586998}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6812","nonce":988586998},{"type":"v1","addr":"172.21.15.123:6813","nonce":988586998}]},"public_addr":"172.21.15.123:6809/988586998","cluster_addr":"172.21.15.123:6811/988586998","heartbeat_back_addr":"172.21.15.123:6815/988586998","heartbeat_front_addr":"172.21.15.123:6813/988586998","state":["exists","up"]},{"osd":4,"uuid":"200a8df4-50ef-40a5-b48a-58981269c086","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6816","nonce":3387813890},{"type":"v1","addr":"172.21.15.184:6817","nonce":3387813890}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6818","nonce":3387813890},{"type":"v1","addr":"172.21.15.184:6819","nonce":3387813890}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6822","nonce":3387813890},{"type":"v1","addr":"172.21.15.184:6823","nonce":3387813890}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6820","nonce":3387813890},{"type":"v1","addr":"172.21.15.184:6821","nonce":3387813890}]},"public_addr":"172.21.15.184:6817/3387813890","cluster_addr":"172.21.15.184:6819/3387813890","heartbeat_back_addr":"172.21.15.184:6823/3387813890","heartbeat_front_addr":"172.21.15.184:6821/3387813890","state":["exists","up"]},{"osd":5,"uuid":"32d5a900-da3d-412c-bc19-13861a2368d8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6808","nonce":2680916017},{"type":"v1","addr":"172.21.15.184:6809","nonce":2680916017}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6810","nonce":2680916017},{"type":"v1","addr":"172.21.15.184:6811","nonce":2680916017}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6814","nonce":2680916017},{"type":"v1","addr":"172.21.15.184:6815","nonce":2680916017}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6812","nonce":2680916017},{"type":"v1","addr":"172.21.15.184:6813","nonce":2680916017}]},"public_addr":"172.21.15.184:6809/2680916017","cluster_addr":"172.21.15.184:6811/2680916017","heartbeat_back_addr":"172.21.15.184:6815/2680916017","heartbeat_front_addr":"172.21.15.184:6813/2680916017","state":["exists","up"]},{"osd":6,"uuid":"cde6f39e-aa1b-4260-ac19-774ee6aaca72","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6824","nonce":2183120457},{"type":"v1","addr":"172.21.15.184:6825","nonce":2183120457}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6826","nonce":2183120457},{"type":"v1","addr":"172.21.15.184:6827","nonce":2183120457}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6830","nonce":2183120457},{"type":"v1","addr":"172.21.15.184:6831","nonce":2183120457}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6828","nonce":2183120457},{"type":"v1","addr":"172.21.15.184:6829","nonce":2183120457}]},"public_addr":"172.21.15.184:6825/2183120457","cluster_addr":"172.21.15.184:6827/2183120457","heartbeat_back_addr":"172.21.15.184:6831/2183120457","heartbeat_front_addr":"172.21.15.184:6829/2183120457","state":["exists","up"]},{"osd":7,"uuid":"4f381778-eee1-47c6-804f-df0e6b33e313","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6800","nonce":4102957285},{"type":"v1","addr":"172.21.15.184:6801","nonce":4102957285}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6802","nonce":4102957285},{"type":"v1","addr":"172.21.15.184:6803","nonce":4102957285}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6806","nonce":4102957285},{"type":"v1","addr":"172.21.15.184:6807","nonce":4102957285}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6804","nonce":4102957285},{"type":"v1","addr":"172.21.15.184:6805","nonce":4102957285}]},"public_addr":"172.21.15.184:6801/4102957285","cluster_addr":"172.21.15.184:6803/4102957285","heartbeat_back_addr":"172.21.15.184:6807/4102957285","heartbeat_front_addr":"172.21.15.184:6805/4102957285","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:35.972210+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.222007+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.198617+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.385663+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.358935+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.156066+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.752973+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:35.682846+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T15:42:52.093 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:52.090+0000 7f95c1d7c640 1 -- 172.21.15.123:0/3416302660 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f959c072ec0 msgr2=0x7f959c075380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:42:52.093 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:52.090+0000 7f95c1d7c640 1 --2- 172.21.15.123:0/3416302660 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f959c072ec0 0x7f959c075380 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f95ac00a750 tx=0x7f95ac008040 comp rx=0 tx=0).stop 2024-09-16T15:42:52.093 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:52.090+0000 7f95c1d7c640 1 -- 172.21.15.123:0/3416302660 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f95bc0b1490 msgr2=0x7f95bc099430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:42:52.094 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:52.090+0000 7f95c1d7c640 1 --2- 172.21.15.123:0/3416302660 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f95bc0b1490 0x7f95bc099430 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f95a401aea0 tx=0x7f95a40040c0 comp rx=0 tx=0).stop 2024-09-16T15:42:52.094 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:52.090+0000 7f95c1d7c640 1 -- 172.21.15.123:0/3416302660 shutdown_connections 2024-09-16T15:42:52.094 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:52.090+0000 7f95c1d7c640 1 --2- 172.21.15.123:0/3416302660 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f959c072ec0 0x7f959c075380 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:42:52.094 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:52.090+0000 7f95c1d7c640 1 --2- 172.21.15.123:0/3416302660 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f95bc15bd80 0x7f95bc099970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:42:52.094 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:52.090+0000 7f95c1d7c640 1 --2- 172.21.15.123:0/3416302660 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f95bc0b1490 0x7f95bc099430 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:42:52.094 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:52.090+0000 7f95c1d7c640 1 --2- 172.21.15.123:0/3416302660 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f95bc068960 0x7f95bc0cf720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:42:52.094 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:52.090+0000 7f95c1d7c640 1 -- 172.21.15.123:0/3416302660 >> 172.21.15.123:0/3416302660 conn(0x7f95bc074230 msgr2=0x7f95bc070300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:42:52.094 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:52.090+0000 7f95c1d7c640 1 -- 172.21.15.123:0/3416302660 shutdown_connections 2024-09-16T15:42:52.094 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:52.090+0000 7f95c1d7c640 1 -- 172.21.15.123:0/3416302660 wait complete. 2024-09-16T15:42:52.110 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-09-16T15:42:52.241 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:52.238+0000 7f02fe1fa640 1 -- 172.21.15.123:0/821792594 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f02f80cdb00 msgr2=0x7f02f80d00f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:42:52.241 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:52.238+0000 7f02fe1fa640 1 --2- 172.21.15.123:0/821792594 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f02f80cdb00 0x7f02f80d00f0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f02f80b5140 tx=0x7f02ec01c770 comp rx=0 tx=0).stop 2024-09-16T15:42:52.241 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:52.238+0000 7f02fe1fa640 1 -- 172.21.15.123:0/821792594 shutdown_connections 2024-09-16T15:42:52.241 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:52.238+0000 7f02fe1fa640 1 --2- 172.21.15.123:0/821792594 >> v2:172.21.15.184:3300/0 conn(0x7f02f80cfca0 0x7f02f815f0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:42:52.241 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:52.238+0000 7f02fe1fa640 1 --2- 172.21.15.123:0/821792594 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f02f80cdb00 0x7f02f80d00f0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:42:52.241 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:52.238+0000 7f02fe1fa640 1 -- 172.21.15.123:0/821792594 >> 172.21.15.123:0/821792594 conn(0x7f02f80d0b70 msgr2=0x7f02f80d0f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:42:52.241 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:52.238+0000 7f02fe1fa640 1 -- 172.21.15.123:0/821792594 shutdown_connections 2024-09-16T15:42:52.241 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:52.238+0000 7f02fe1fa640 1 -- 172.21.15.123:0/821792594 wait complete. 2024-09-16T15:42:52.242 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:52.238+0000 7f02fe1fa640 1 Processor -- start 2024-09-16T15:42:52.242 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:52.238+0000 7f02fe1fa640 1 -- start start 2024-09-16T15:42:52.244 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:52.242+0000 7f02fe1fa640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f02f80cdb00 0x7f02f81468d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:42:52.244 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:52.242+0000 7f02fe1fa640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f02f80cfca0 0x7f02f81559e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:42:52.244 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:52.242+0000 7f02fe1fa640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f02f815aa90 0x7f02f8155f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:42:52.244 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:52.242+0000 7f02fe1fa640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f02f8167a60 con 0x7f02f80cdb00 2024-09-16T15:42:52.244 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:52.242+0000 7f02fe1fa640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f02f80b4880 con 0x7f02f80cfca0 2024-09-16T15:42:52.244 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:52.242+0000 7f02fe1fa640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f02f8167be0 con 0x7f02f815aa90 2024-09-16T15:42:52.244 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:52.242+0000 7f02f77fe640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f02f80cdb00 0x7f02f81468d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:42:52.244 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:52.242+0000 7f02f77fe640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f02f80cdb00 0x7f02f81468d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:44986/0 (socket says 172.21.15.123:44986) 2024-09-16T15:42:52.245 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:52.242+0000 7f02f77fe640 1 -- 172.21.15.123:0/4221149518 learned_addr learned my addr 172.21.15.123:0/4221149518 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:42:52.245 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:52.242+0000 7f02f6ffd640 1 --2- 172.21.15.123:0/4221149518 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f02f80cfca0 0x7f02f81559e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:42:52.245 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:52.242+0000 7f02f7fff640 1 --2- 172.21.15.123:0/4221149518 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f02f815aa90 0x7f02f8155f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:42:52.245 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:52.242+0000 7f02f77fe640 1 -- 172.21.15.123:0/4221149518 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f02f815aa90 msgr2=0x7f02f8155f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:42:52.245 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:52.242+0000 7f02f77fe640 1 --2- 172.21.15.123:0/4221149518 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f02f815aa90 0x7f02f8155f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:42:52.245 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:52.242+0000 7f02f77fe640 1 -- 172.21.15.123:0/4221149518 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f02f80cfca0 msgr2=0x7f02f81559e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:42:52.245 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:52.242+0000 7f02f77fe640 1 --2- 172.21.15.123:0/4221149518 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f02f80cfca0 0x7f02f81559e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:42:52.245 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:52.242+0000 7f02f77fe640 1 -- 172.21.15.123:0/4221149518 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f02f8156460 con 0x7f02f80cdb00 2024-09-16T15:42:52.245 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:52.242+0000 7f02f77fe640 1 --2- 172.21.15.123:0/4221149518 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f02f80cdb00 0x7f02f81468d0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f02f000c3f0 tx=0x7f02f000c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:42:52.245 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:52.242+0000 7f02dffff640 1 -- 172.21.15.123:0/4221149518 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02f0017070 con 0x7f02f80cdb00 2024-09-16T15:42:52.245 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:52.242+0000 7f02dffff640 1 -- 172.21.15.123:0/4221149518 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f02f00053c0 con 0x7f02f80cdb00 2024-09-16T15:42:52.246 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:52.242+0000 7f02fe1fa640 1 -- 172.21.15.123:0/4221149518 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f02f8154f20 con 0x7f02f80cdb00 2024-09-16T15:42:52.246 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:52.242+0000 7f02dffff640 1 -- 172.21.15.123:0/4221149518 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02f0003860 con 0x7f02f80cdb00 2024-09-16T15:42:52.246 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:52.242+0000 7f02fe1fa640 1 -- 172.21.15.123:0/4221149518 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f02f81554e0 con 0x7f02f80cdb00 2024-09-16T15:42:52.247 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:52.242+0000 7f02fe1fa640 1 -- 172.21.15.123:0/4221149518 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f02f8154c40 con 0x7f02f80cdb00 2024-09-16T15:42:52.247 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:52.242+0000 7f02dffff640 1 -- 172.21.15.123:0/4221149518 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 5) ==== 97080+0+0 (secure 0 0 0) 0x7f02f001a070 con 0x7f02f80cdb00 2024-09-16T15:42:52.247 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:52.242+0000 7f02dffff640 1 --2- 172.21.15.123:0/4221149518 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f02e4072ec0 0x7f02e4075380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:42:52.247 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:52.242+0000 7f02dffff640 1 -- 172.21.15.123:0/4221149518 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f02f00930a0 con 0x7f02f80cdb00 2024-09-16T15:42:52.247 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:52.242+0000 7f02f6ffd640 1 --2- 172.21.15.123:0/4221149518 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f02e4072ec0 0x7f02e4075380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:42:52.247 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:52.246+0000 7f02f6ffd640 1 --2- 172.21.15.123:0/4221149518 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f02e4072ec0 0x7f02e4075380 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f02ec0af000 tx=0x7f02ec01afb0 comp rx=0 tx=0).ready entity=mgr.4135 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:42:52.250 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:52.246+0000 7f02dffff640 1 -- 172.21.15.123:0/4221149518 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f02f005e380 con 0x7f02f80cdb00 2024-09-16T15:42:52.461 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:52.458+0000 7f02fe1fa640 1 -- 172.21.15.123:0/4221149518 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) -- 0x7f02f814c7f0 con 0x7f02f80cdb00 2024-09-16T15:42:53.413 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:53.410+0000 7f02dffff640 1 -- 172.21.15.123:0/4221149518 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v7) ==== 118+0+0 (secure 0 0 0) 0x7f02f005dc80 con 0x7f02f80cdb00 2024-09-16T15:42:53.417 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:53.414+0000 7f02fe1fa640 1 -- 172.21.15.123:0/4221149518 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f02e4072ec0 msgr2=0x7f02e4075380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:42:53.417 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:53.414+0000 7f02fe1fa640 1 --2- 172.21.15.123:0/4221149518 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f02e4072ec0 0x7f02e4075380 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f02ec0af000 tx=0x7f02ec01afb0 comp rx=0 tx=0).stop 2024-09-16T15:42:53.417 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:53.414+0000 7f02fe1fa640 1 -- 172.21.15.123:0/4221149518 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f02f80cdb00 msgr2=0x7f02f81468d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:42:53.418 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:53.414+0000 7f02fe1fa640 1 --2- 172.21.15.123:0/4221149518 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f02f80cdb00 0x7f02f81468d0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f02f000c3f0 tx=0x7f02f000c8b0 comp rx=0 tx=0).stop 2024-09-16T15:42:53.418 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:53.414+0000 7f02fe1fa640 1 -- 172.21.15.123:0/4221149518 shutdown_connections 2024-09-16T15:42:53.418 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:53.414+0000 7f02fe1fa640 1 --2- 172.21.15.123:0/4221149518 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f02e4072ec0 0x7f02e4075380 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:42:53.418 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:53.414+0000 7f02fe1fa640 1 --2- 172.21.15.123:0/4221149518 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f02f815aa90 0x7f02f8155f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:42:53.418 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:53.414+0000 7f02fe1fa640 1 --2- 172.21.15.123:0/4221149518 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f02f80cfca0 0x7f02f81559e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:42:53.418 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:53.414+0000 7f02fe1fa640 1 --2- 172.21.15.123:0/4221149518 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f02f80cdb00 0x7f02f81468d0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:42:53.418 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:53.414+0000 7f02fe1fa640 1 -- 172.21.15.123:0/4221149518 >> 172.21.15.123:0/4221149518 conn(0x7f02f80d0b70 msgr2=0x7f02f80b08c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:42:53.418 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:53.414+0000 7f02fe1fa640 1 -- 172.21.15.123:0/4221149518 shutdown_connections 2024-09-16T15:42:53.418 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:53.414+0000 7f02fe1fa640 1 -- 172.21.15.123:0/4221149518 wait complete. 2024-09-16T15:42:53.438 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-09-16T15:42:53.438 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-09-16T15:42:53.574 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:53.570+0000 7f41e8d59640 1 -- 172.21.15.123:0/3668866561 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f41e4152980 msgr2=0x7f41e40cf3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:42:53.574 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:53.570+0000 7f41e8d59640 1 --2- 172.21.15.123:0/3668866561 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f41e4152980 0x7f41e40cf3b0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f41d4009900 tx=0x7f41d401c8b0 comp rx=0 tx=0).stop 2024-09-16T15:42:53.574 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:53.570+0000 7f41e8d59640 1 -- 172.21.15.123:0/3668866561 shutdown_connections 2024-09-16T15:42:53.574 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:53.570+0000 7f41e8d59640 1 --2- 172.21.15.123:0/3668866561 >> v2:172.21.15.184:3300/0 conn(0x7f41e40cd710 0x7f41e40cc550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:42:53.574 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:53.570+0000 7f41e8d59640 1 --2- 172.21.15.123:0/3668866561 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f41e4152980 0x7f41e40cf3b0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:42:53.574 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:53.570+0000 7f41e8d59640 1 -- 172.21.15.123:0/3668866561 >> 172.21.15.123:0/3668866561 conn(0x7f41e40b1710 msgr2=0x7f41e40b1b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:42:53.574 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:53.570+0000 7f41e8d59640 1 -- 172.21.15.123:0/3668866561 shutdown_connections 2024-09-16T15:42:53.575 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:53.570+0000 7f41e8d59640 1 -- 172.21.15.123:0/3668866561 wait complete. 2024-09-16T15:42:53.575 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:53.574+0000 7f41e8d59640 1 Processor -- start 2024-09-16T15:42:53.575 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:53.574+0000 7f41e8d59640 1 -- start start 2024-09-16T15:42:53.578 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:53.574+0000 7f41e8d59640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f41e40cd710 0x7f41e4150910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:42:53.578 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:53.574+0000 7f41e8d59640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f41e4152980 0x7f41e40c8d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:42:53.578 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:53.574+0000 7f41e8d59640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f41e4152d60 0x7f41e4153210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:42:53.578 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:53.574+0000 7f41e8d59640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f41e4070030 con 0x7f41e4152d60 2024-09-16T15:42:53.578 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:53.574+0000 7f41e8d59640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f41e406feb0 con 0x7f41e40cd710 2024-09-16T15:42:53.578 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:53.574+0000 7f41e8d59640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f41e40701b0 con 0x7f41e4152980 2024-09-16T15:42:53.578 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:53.574+0000 7f41e2d9d640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f41e4152d60 0x7f41e4153210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:42:53.578 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:53.574+0000 7f41e259c640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f41e40cd710 0x7f41e4150910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:42:53.579 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:53.574+0000 7f41e2d9d640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f41e4152d60 0x7f41e4153210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:40136/0 (socket says 172.21.15.123:40136) 2024-09-16T15:42:53.579 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:53.574+0000 7f41e2d9d640 1 -- 172.21.15.123:0/3491274277 learned_addr learned my addr 172.21.15.123:0/3491274277 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:42:53.579 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:53.574+0000 7f41e1d9b640 1 --2- 172.21.15.123:0/3491274277 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f41e4152980 0x7f41e40c8d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:42:53.579 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:53.574+0000 7f41e2d9d640 1 -- 172.21.15.123:0/3491274277 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f41e4152980 msgr2=0x7f41e40c8d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:42:53.579 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:53.574+0000 7f41e2d9d640 1 --2- 172.21.15.123:0/3491274277 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f41e4152980 0x7f41e40c8d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:42:53.579 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:53.574+0000 7f41e2d9d640 1 -- 172.21.15.123:0/3491274277 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f41e40cd710 msgr2=0x7f41e4150910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:42:53.579 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:53.574+0000 7f41e2d9d640 1 --2- 172.21.15.123:0/3491274277 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f41e40cd710 0x7f41e4150910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:42:53.579 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:53.574+0000 7f41e2d9d640 1 -- 172.21.15.123:0/3491274277 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f41e40c92d0 con 0x7f41e4152d60 2024-09-16T15:42:53.579 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:53.574+0000 7f41e2d9d640 1 --2- 172.21.15.123:0/3491274277 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f41e4152d60 0x7f41e4153210 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f41dc009d30 tx=0x7f41dc00c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:42:53.579 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:53.574+0000 7f41d2ffd640 1 -- 172.21.15.123:0/3491274277 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41dc003b80 con 0x7f41e4152d60 2024-09-16T15:42:53.579 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:53.574+0000 7f41d2ffd640 1 -- 172.21.15.123:0/3491274277 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f41dc003d20 con 0x7f41e4152d60 2024-09-16T15:42:53.579 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:53.574+0000 7f41e8d59640 1 -- 172.21.15.123:0/3491274277 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f41e40c77f0 con 0x7f41e4152d60 2024-09-16T15:42:53.580 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:53.574+0000 7f41d2ffd640 1 -- 172.21.15.123:0/3491274277 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41dc004280 con 0x7f41e4152d60 2024-09-16T15:42:53.580 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:53.574+0000 7f41e8d59640 1 -- 172.21.15.123:0/3491274277 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f41e40c9870 con 0x7f41e4152d60 2024-09-16T15:42:53.581 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:53.578+0000 7f41e8d59640 1 -- 172.21.15.123:0/3491274277 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f41e4070730 con 0x7f41e4152d60 2024-09-16T15:42:53.581 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:53.578+0000 7f41d2ffd640 1 -- 172.21.15.123:0/3491274277 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 5) ==== 97080+0+0 (secure 0 0 0) 0x7f41dc004420 con 0x7f41e4152d60 2024-09-16T15:42:53.581 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:53.578+0000 7f41d2ffd640 1 --2- 172.21.15.123:0/3491274277 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f41c4072ec0 0x7f41c4075380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:42:53.581 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:53.578+0000 7f41d2ffd640 1 -- 172.21.15.123:0/3491274277 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f41dc094b90 con 0x7f41e4152d60 2024-09-16T15:42:53.581 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:53.578+0000 7f41e259c640 1 --2- 172.21.15.123:0/3491274277 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f41c4072ec0 0x7f41c4075380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:42:53.581 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:53.578+0000 7f41e259c640 1 --2- 172.21.15.123:0/3491274277 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f41c4072ec0 0x7f41c4075380 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f41d40af000 tx=0x7f41d401b0f0 comp rx=0 tx=0).ready entity=mgr.4135 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:42:53.584 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:53.582+0000 7f41d2ffd640 1 -- 172.21.15.123:0/3491274277 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f41dc05fde0 con 0x7f41e4152d60 2024-09-16T15:42:53.788 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:53.786+0000 7f41e8d59640 1 -- 172.21.15.123:0/3491274277 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) -- 0x7f41e40b0680 con 0x7f41e4152d60 2024-09-16T15:42:54.429 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:54.426+0000 7f41d2ffd640 1 -- 172.21.15.123:0/3491274277 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v8) ==== 122+0+0 (secure 0 0 0) 0x7f41dc05f6e0 con 0x7f41e4152d60 2024-09-16T15:42:54.434 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:54.430+0000 7f41e8d59640 1 -- 172.21.15.123:0/3491274277 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f41c4072ec0 msgr2=0x7f41c4075380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:42:54.434 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:54.430+0000 7f41e8d59640 1 --2- 172.21.15.123:0/3491274277 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f41c4072ec0 0x7f41c4075380 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f41d40af000 tx=0x7f41d401b0f0 comp rx=0 tx=0).stop 2024-09-16T15:42:54.434 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:54.430+0000 7f41e8d59640 1 -- 172.21.15.123:0/3491274277 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f41e4152d60 msgr2=0x7f41e4153210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:42:54.434 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:54.430+0000 7f41e8d59640 1 --2- 172.21.15.123:0/3491274277 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f41e4152d60 0x7f41e4153210 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f41dc009d30 tx=0x7f41dc00c6a0 comp rx=0 tx=0).stop 2024-09-16T15:42:54.434 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:54.430+0000 7f41e8d59640 1 -- 172.21.15.123:0/3491274277 shutdown_connections 2024-09-16T15:42:54.434 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:54.430+0000 7f41e8d59640 1 --2- 172.21.15.123:0/3491274277 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f41c4072ec0 0x7f41c4075380 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:42:54.434 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:54.430+0000 7f41e8d59640 1 --2- 172.21.15.123:0/3491274277 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f41e4152d60 0x7f41e4153210 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:42:54.434 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:54.430+0000 7f41e8d59640 1 --2- 172.21.15.123:0/3491274277 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f41e4152980 0x7f41e40c8d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:42:54.434 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:54.430+0000 7f41e8d59640 1 --2- 172.21.15.123:0/3491274277 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f41e40cd710 0x7f41e4150910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:42:54.434 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:54.430+0000 7f41e8d59640 1 -- 172.21.15.123:0/3491274277 >> 172.21.15.123:0/3491274277 conn(0x7f41e40b1710 msgr2=0x7f41e415d290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:42:54.434 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:54.430+0000 7f41e8d59640 1 -- 172.21.15.123:0/3491274277 shutdown_connections 2024-09-16T15:42:54.435 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:54.430+0000 7f41e8d59640 1 -- 172.21.15.123:0/3491274277 wait complete. 2024-09-16T15:42:54.452 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-09-16T15:42:54.452 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-16T15:42:54.569 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:54.566+0000 7f37fc4f5640 1 -- 172.21.15.123:0/1498856061 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f37f415e160 msgr2=0x7f37f4073f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:42:54.569 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:54.566+0000 7f37fc4f5640 1 --2- 172.21.15.123:0/1498856061 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f37f415e160 0x7f37f4073f30 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f37f0009840 tx=0x7f37f001c790 comp rx=0 tx=0).stop 2024-09-16T15:42:54.569 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:54.566+0000 7f37fc4f5640 1 -- 172.21.15.123:0/1498856061 shutdown_connections 2024-09-16T15:42:54.569 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:54.566+0000 7f37fc4f5640 1 --2- 172.21.15.123:0/1498856061 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f37f4074470 0x7f37f40cd6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:42:54.569 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:54.566+0000 7f37fc4f5640 1 --2- 172.21.15.123:0/1498856061 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f37f415e160 0x7f37f4073f30 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:42:54.570 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:54.566+0000 7f37fc4f5640 1 -- 172.21.15.123:0/1498856061 >> 172.21.15.123:0/1498856061 conn(0x7f37f40cfb10 msgr2=0x7f37f40cff20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:42:54.570 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:54.566+0000 7f37fc4f5640 1 -- 172.21.15.123:0/1498856061 shutdown_connections 2024-09-16T15:42:54.570 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:54.566+0000 7f37fc4f5640 1 -- 172.21.15.123:0/1498856061 wait complete. 2024-09-16T15:42:54.570 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:54.566+0000 7f37fc4f5640 1 Processor -- start 2024-09-16T15:42:54.570 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:54.566+0000 7f37fc4f5640 1 -- start start 2024-09-16T15:42:54.572 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:54.570+0000 7f37fc4f5640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f37f4074470 0x7f37f414fa30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:42:54.572 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:54.570+0000 7f37fc4f5640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f37f40cdc20 0x7f37f40c75f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:42:54.573 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:54.570+0000 7f37fc4f5640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f37f415e160 0x7f37f40c83e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:42:54.573 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:54.570+0000 7f37fc4f5640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f37f4167f50 con 0x7f37f4074470 2024-09-16T15:42:54.573 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:54.570+0000 7f37fc4f5640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f37f4167dd0 con 0x7f37f415e160 2024-09-16T15:42:54.573 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:54.570+0000 7f37fc4f5640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f37f41680d0 con 0x7f37f40cdc20 2024-09-16T15:42:54.573 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:54.570+0000 7f37fa291640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f37f4074470 0x7f37f414fa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:42:54.573 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:54.570+0000 7f37fa291640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f37f4074470 0x7f37f414fa30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:40146/0 (socket says 172.21.15.123:40146) 2024-09-16T15:42:54.573 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:54.570+0000 7f37fa291640 1 -- 172.21.15.123:0/301244833 learned_addr learned my addr 172.21.15.123:0/301244833 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:42:54.573 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:54.570+0000 7f37f9a90640 1 --2- 172.21.15.123:0/301244833 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f37f40cdc20 0x7f37f40c75f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:42:54.573 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:54.570+0000 7f37fa291640 1 -- 172.21.15.123:0/301244833 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f37f40cdc20 msgr2=0x7f37f40c75f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:42:54.573 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:54.570+0000 7f37fa291640 1 --2- 172.21.15.123:0/301244833 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f37f40cdc20 0x7f37f40c75f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:42:54.574 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:54.570+0000 7f37fa291640 1 -- 172.21.15.123:0/301244833 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f37f415e160 msgr2=0x7f37f40c83e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:42:54.574 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:54.570+0000 7f37faa92640 1 --2- 172.21.15.123:0/301244833 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f37f415e160 0x7f37f40c83e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:42:54.574 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:54.570+0000 7f37fa291640 1 --2- 172.21.15.123:0/301244833 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f37f415e160 0x7f37f40c83e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:42:54.574 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:54.570+0000 7f37fa291640 1 -- 172.21.15.123:0/301244833 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f37f40c7ed0 con 0x7f37f4074470 2024-09-16T15:42:54.574 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:54.570+0000 7f37fa291640 1 --2- 172.21.15.123:0/301244833 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f37f4074470 0x7f37f414fa30 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f37f0009790 tx=0x7f37f00a56a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:42:54.574 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:54.570+0000 7f37eaffd640 1 -- 172.21.15.123:0/301244833 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37f00bb070 con 0x7f37f4074470 2024-09-16T15:42:54.574 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:54.570+0000 7f37eaffd640 1 -- 172.21.15.123:0/301244833 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f37f0004430 con 0x7f37f4074470 2024-09-16T15:42:54.574 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:54.570+0000 7f37fc4f5640 1 -- 172.21.15.123:0/301244833 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f37f4152e30 con 0x7f37f4074470 2024-09-16T15:42:54.574 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:54.570+0000 7f37eaffd640 1 -- 172.21.15.123:0/301244833 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37f00b2040 con 0x7f37f4074470 2024-09-16T15:42:54.574 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:54.570+0000 7f37fc4f5640 1 -- 172.21.15.123:0/301244833 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f37f41533f0 con 0x7f37f4074470 2024-09-16T15:42:54.574 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:54.570+0000 7f37fc4f5640 1 -- 172.21.15.123:0/301244833 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f37f40cd6e0 con 0x7f37f4074470 2024-09-16T15:42:54.575 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:54.570+0000 7f37eaffd640 1 -- 172.21.15.123:0/301244833 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 5) ==== 97080+0+0 (secure 0 0 0) 0x7f37f00be070 con 0x7f37f4074470 2024-09-16T15:42:54.575 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:54.570+0000 7f37eaffd640 1 --2- 172.21.15.123:0/301244833 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f37d8072ec0 0x7f37d8075380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:42:54.575 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:54.570+0000 7f37eaffd640 1 -- 172.21.15.123:0/301244833 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f37f012d5c0 con 0x7f37f4074470 2024-09-16T15:42:54.575 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:54.570+0000 7f37f9a90640 1 --2- 172.21.15.123:0/301244833 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f37d8072ec0 0x7f37d8075380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:42:54.575 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:54.574+0000 7f37f9a90640 1 --2- 172.21.15.123:0/301244833 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f37d8072ec0 0x7f37d8075380 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f37f40cdb30 tx=0x7f37e4005bf0 comp rx=0 tx=0).ready entity=mgr.4135 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:42:54.577 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:54.574+0000 7f37eaffd640 1 -- 172.21.15.123:0/301244833 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f37f0016410 con 0x7f37f4074470 2024-09-16T15:42:54.779 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:54.778+0000 7f37fc4f5640 1 -- 172.21.15.123:0/301244833 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f37f40c9ff0 con 0x7f37f4074470 2024-09-16T15:42:54.779 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:54.778+0000 7f37eaffd640 1 -- 172.21.15.123:0/301244833 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) ==== 93+0+5226 (secure 0 0 0) 0x7f37f00f8930 con 0x7f37f4074470 2024-09-16T15:42:54.780 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-09-16T15:42:54.780 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":8,"btime":"2024-09-16T15:42:54:421548+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4280,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6835/1497621896","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":1497621896},{"type":"v1","addr":"172.21.15.184:6835","nonce":1497621896}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4305,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6833/2227496057","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6832","nonce":2227496057},{"type":"v1","addr":"172.21.15.123:6833","nonce":2227496057}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4311,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6835/294455998","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6834","nonce":294455998},{"type":"v1","addr":"172.21.15.123:6835","nonce":294455998}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-16T15:42:50.762280+0000","modified":"2024-09-16T15:42:53.789989+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4348},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4348":{"gid":4348,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.184:6837/1469324095","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6836","nonce":1469324095},{"type":"v1","addr":"172.21.15.184:6837","nonce":1469324095}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4348,"qdb_cluster":[4348]},"id":1}]} 2024-09-16T15:42:54.780 INFO:teuthology.orchestra.run.smithi123.stderr:dumped fsmap epoch 8 2024-09-16T15:42:54.782 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:54.778+0000 7f37fc4f5640 1 -- 172.21.15.123:0/301244833 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f37d8072ec0 msgr2=0x7f37d8075380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:42:54.782 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:54.778+0000 7f37fc4f5640 1 --2- 172.21.15.123:0/301244833 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f37d8072ec0 0x7f37d8075380 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f37f40cdb30 tx=0x7f37e4005bf0 comp rx=0 tx=0).stop 2024-09-16T15:42:54.782 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:54.778+0000 7f37fc4f5640 1 -- 172.21.15.123:0/301244833 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f37f4074470 msgr2=0x7f37f414fa30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:42:54.782 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:54.778+0000 7f37fc4f5640 1 --2- 172.21.15.123:0/301244833 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f37f4074470 0x7f37f414fa30 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f37f0009790 tx=0x7f37f00a56a0 comp rx=0 tx=0).stop 2024-09-16T15:42:54.782 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:54.778+0000 7f37fc4f5640 1 -- 172.21.15.123:0/301244833 shutdown_connections 2024-09-16T15:42:54.782 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:54.778+0000 7f37fc4f5640 1 --2- 172.21.15.123:0/301244833 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f37d8072ec0 0x7f37d8075380 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:42:54.782 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:54.778+0000 7f37fc4f5640 1 --2- 172.21.15.123:0/301244833 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f37f415e160 0x7f37f40c83e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:42:54.783 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:54.778+0000 7f37fc4f5640 1 --2- 172.21.15.123:0/301244833 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f37f40cdc20 0x7f37f40c75f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:42:54.783 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:54.778+0000 7f37fc4f5640 1 --2- 172.21.15.123:0/301244833 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f37f4074470 0x7f37f414fa30 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:42:54.783 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:54.778+0000 7f37fc4f5640 1 -- 172.21.15.123:0/301244833 >> 172.21.15.123:0/301244833 conn(0x7f37f40cfb10 msgr2=0x7f37f40cc950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:42:54.783 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:54.778+0000 7f37fc4f5640 1 -- 172.21.15.123:0/301244833 shutdown_connections 2024-09-16T15:42:54.783 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:54.778+0000 7f37fc4f5640 1 -- 172.21.15.123:0/301244833 wait complete. 2024-09-16T15:42:54.797 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-16T15:42:54.915 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:54.914+0000 7efc78b27640 1 -- 172.21.15.123:0/2386508718 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7efc7415df70 msgr2=0x7efc740d0620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:42:54.915 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:54.914+0000 7efc78b27640 1 --2- 172.21.15.123:0/2386508718 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7efc7415df70 0x7efc740d0620 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7efc5c009900 tx=0x7efc5c01c8b0 comp rx=0 tx=0).stop 2024-09-16T15:42:54.915 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:54.914+0000 7efc78b27640 1 -- 172.21.15.123:0/2386508718 shutdown_connections 2024-09-16T15:42:54.916 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:54.914+0000 7efc78b27640 1 --2- 172.21.15.123:0/2386508718 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7efc740b5360 0x7efc740d0b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:42:54.916 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:54.914+0000 7efc78b27640 1 --2- 172.21.15.123:0/2386508718 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7efc7415df70 0x7efc740d0620 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:42:54.916 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:54.914+0000 7efc78b27640 1 -- 172.21.15.123:0/2386508718 >> 172.21.15.123:0/2386508718 conn(0x7efc740d3e90 msgr2=0x7efc740d4270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:42:54.916 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:54.914+0000 7efc78b27640 1 -- 172.21.15.123:0/2386508718 shutdown_connections 2024-09-16T15:42:54.916 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:54.914+0000 7efc78b27640 1 -- 172.21.15.123:0/2386508718 wait complete. 2024-09-16T15:42:54.918 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:54.914+0000 7efc78b27640 1 Processor -- start 2024-09-16T15:42:54.918 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:54.914+0000 7efc78b27640 1 -- start start 2024-09-16T15:42:54.918 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:54.914+0000 7efc78b27640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7efc740b5360 0x7efc740ca450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:42:54.918 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:54.914+0000 7efc78b27640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7efc740d4980 0x7efc740b4550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:42:54.918 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:54.914+0000 7efc78b27640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7efc7415df70 0x7efc7414abe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:42:54.918 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:54.914+0000 7efc78b27640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7efc7416b430 con 0x7efc7415df70 2024-09-16T15:42:54.918 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:54.914+0000 7efc78b27640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7efc7416b2b0 con 0x7efc740b5360 2024-09-16T15:42:54.918 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:54.914+0000 7efc78b27640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7efc7416b5b0 con 0x7efc740d4980 2024-09-16T15:42:54.919 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:54.914+0000 7efc72d9d640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7efc7415df70 0x7efc7414abe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:42:54.919 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:54.914+0000 7efc72d9d640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7efc7415df70 0x7efc7414abe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:40166/0 (socket says 172.21.15.123:40166) 2024-09-16T15:42:54.919 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:54.914+0000 7efc72d9d640 1 -- 172.21.15.123:0/2640529728 learned_addr learned my addr 172.21.15.123:0/2640529728 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:42:54.919 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:54.914+0000 7efc6bfff640 1 --2- 172.21.15.123:0/2640529728 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7efc740d4980 0x7efc740b4550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:42:54.919 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:54.914+0000 7efc7259c640 1 --2- 172.21.15.123:0/2640529728 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7efc740b5360 0x7efc740ca450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:42:54.919 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:54.914+0000 7efc72d9d640 1 -- 172.21.15.123:0/2640529728 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7efc740d4980 msgr2=0x7efc740b4550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:42:54.919 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:54.914+0000 7efc72d9d640 1 --2- 172.21.15.123:0/2640529728 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7efc740d4980 0x7efc740b4550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:42:54.919 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:54.914+0000 7efc72d9d640 1 -- 172.21.15.123:0/2640529728 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7efc740b5360 msgr2=0x7efc740ca450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:42:54.919 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:54.914+0000 7efc72d9d640 1 --2- 172.21.15.123:0/2640529728 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7efc740b5360 0x7efc740ca450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:42:54.919 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:54.914+0000 7efc72d9d640 1 -- 172.21.15.123:0/2640529728 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efc740cb000 con 0x7efc7415df70 2024-09-16T15:42:54.919 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:54.914+0000 7efc72d9d640 1 --2- 172.21.15.123:0/2640529728 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7efc7415df70 0x7efc7414abe0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7efc6c00cce0 tx=0x7efc6c00a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:42:54.919 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:54.914+0000 7efc6affd640 1 -- 172.21.15.123:0/2640529728 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc6c004b30 con 0x7efc7415df70 2024-09-16T15:42:54.920 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:54.914+0000 7efc6affd640 1 -- 172.21.15.123:0/2640529728 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7efc6c004dc0 con 0x7efc7415df70 2024-09-16T15:42:54.920 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:54.914+0000 7efc6affd640 1 -- 172.21.15.123:0/2640529728 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc6c005280 con 0x7efc7415df70 2024-09-16T15:42:54.920 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:54.914+0000 7efc78b27640 1 -- 172.21.15.123:0/2640529728 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efc740cb2e0 con 0x7efc7415df70 2024-09-16T15:42:54.920 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:54.914+0000 7efc78b27640 1 -- 172.21.15.123:0/2640529728 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efc740ca990 con 0x7efc7415df70 2024-09-16T15:42:54.921 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:54.918+0000 7efc78b27640 1 -- 172.21.15.123:0/2640529728 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efc740d1c20 con 0x7efc7415df70 2024-09-16T15:42:54.921 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:54.918+0000 7efc6affd640 1 -- 172.21.15.123:0/2640529728 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 5) ==== 97080+0+0 (secure 0 0 0) 0x7efc6c005420 con 0x7efc7415df70 2024-09-16T15:42:54.921 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:54.918+0000 7efc6affd640 1 --2- 172.21.15.123:0/2640529728 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7efc54072ec0 0x7efc54075380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:42:54.921 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:54.918+0000 7efc6affd640 1 -- 172.21.15.123:0/2640529728 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7efc6c094640 con 0x7efc7415df70 2024-09-16T15:42:54.921 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:54.918+0000 7efc7259c640 1 --2- 172.21.15.123:0/2640529728 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7efc54072ec0 0x7efc54075380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:42:54.921 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:54.918+0000 7efc7259c640 1 --2- 172.21.15.123:0/2640529728 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7efc54072ec0 0x7efc54075380 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7efc5c0af000 tx=0x7efc5c01b0f0 comp rx=0 tx=0).ready entity=mgr.4135 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:42:54.923 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:54.922+0000 7efc6affd640 1 -- 172.21.15.123:0/2640529728 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7efc6c014030 con 0x7efc7415df70 2024-09-16T15:42:55.081 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:55.078+0000 7efc78b27640 1 -- 172.21.15.123:0/2640529728 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7efc740760f0 con 0x7efc7415df70 2024-09-16T15:42:55.082 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:55.078+0000 7efc6affd640 1 -- 172.21.15.123:0/2640529728 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) ==== 74+0+16725 (secure 0 0 0) 0x7efc6c05f890 con 0x7efc7415df70 2024-09-16T15:42:55.082 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-09-16T15:42:55.083 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":21,"fsid":"56077f89-18c8-486f-ae7e-eecc0bb7822c","created":"2024-09-16T15:42:26.364892+0000","modified":"2024-09-16T15:42:52.401147+0000","last_up_change":"2024-09-16T15:42:38.707234+0000","last_in_change":"2024-09-16T15:42:29.423662+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-16T15:42:39.287764+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-09-16T15:42:48.583563+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-09-16T15:42:49.704665+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9e7c1955-a607-4671-9be7-f2ef760ceeaa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6800","nonce":3882044969},{"type":"v1","addr":"172.21.15.123:6801","nonce":3882044969}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6802","nonce":3882044969},{"type":"v1","addr":"172.21.15.123:6803","nonce":3882044969}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6806","nonce":3882044969},{"type":"v1","addr":"172.21.15.123:6807","nonce":3882044969}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6804","nonce":3882044969},{"type":"v1","addr":"172.21.15.123:6805","nonce":3882044969}]},"public_addr":"172.21.15.123:6801/3882044969","cluster_addr":"172.21.15.123:6803/3882044969","heartbeat_back_addr":"172.21.15.123:6807/3882044969","heartbeat_front_addr":"172.21.15.123:6805/3882044969","state":["exists","up"]},{"osd":1,"uuid":"c540cfde-0977-4742-9b17-fd634668e010","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6816","nonce":4037001850},{"type":"v1","addr":"172.21.15.123:6817","nonce":4037001850}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6818","nonce":4037001850},{"type":"v1","addr":"172.21.15.123:6819","nonce":4037001850}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6822","nonce":4037001850},{"type":"v1","addr":"172.21.15.123:6823","nonce":4037001850}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6820","nonce":4037001850},{"type":"v1","addr":"172.21.15.123:6821","nonce":4037001850}]},"public_addr":"172.21.15.123:6817/4037001850","cluster_addr":"172.21.15.123:6819/4037001850","heartbeat_back_addr":"172.21.15.123:6823/4037001850","heartbeat_front_addr":"172.21.15.123:6821/4037001850","state":["exists","up"]},{"osd":2,"uuid":"9864a496-f169-486b-95c4-1d59162793a7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6824","nonce":1332143088},{"type":"v1","addr":"172.21.15.123:6825","nonce":1332143088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6826","nonce":1332143088},{"type":"v1","addr":"172.21.15.123:6827","nonce":1332143088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6830","nonce":1332143088},{"type":"v1","addr":"172.21.15.123:6831","nonce":1332143088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6828","nonce":1332143088},{"type":"v1","addr":"172.21.15.123:6829","nonce":1332143088}]},"public_addr":"172.21.15.123:6825/1332143088","cluster_addr":"172.21.15.123:6827/1332143088","heartbeat_back_addr":"172.21.15.123:6831/1332143088","heartbeat_front_addr":"172.21.15.123:6829/1332143088","state":["exists","up"]},{"osd":3,"uuid":"c6d1f817-f074-4325-95e1-578eba13edda","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6808","nonce":988586998},{"type":"v1","addr":"172.21.15.123:6809","nonce":988586998}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6810","nonce":988586998},{"type":"v1","addr":"172.21.15.123:6811","nonce":988586998}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6814","nonce":988586998},{"type":"v1","addr":"172.21.15.123:6815","nonce":988586998}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6812","nonce":988586998},{"type":"v1","addr":"172.21.15.123:6813","nonce":988586998}]},"public_addr":"172.21.15.123:6809/988586998","cluster_addr":"172.21.15.123:6811/988586998","heartbeat_back_addr":"172.21.15.123:6815/988586998","heartbeat_front_addr":"172.21.15.123:6813/988586998","state":["exists","up"]},{"osd":4,"uuid":"200a8df4-50ef-40a5-b48a-58981269c086","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6816","nonce":3387813890},{"type":"v1","addr":"172.21.15.184:6817","nonce":3387813890}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6818","nonce":3387813890},{"type":"v1","addr":"172.21.15.184:6819","nonce":3387813890}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6822","nonce":3387813890},{"type":"v1","addr":"172.21.15.184:6823","nonce":3387813890}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6820","nonce":3387813890},{"type":"v1","addr":"172.21.15.184:6821","nonce":3387813890}]},"public_addr":"172.21.15.184:6817/3387813890","cluster_addr":"172.21.15.184:6819/3387813890","heartbeat_back_addr":"172.21.15.184:6823/3387813890","heartbeat_front_addr":"172.21.15.184:6821/3387813890","state":["exists","up"]},{"osd":5,"uuid":"32d5a900-da3d-412c-bc19-13861a2368d8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6808","nonce":2680916017},{"type":"v1","addr":"172.21.15.184:6809","nonce":2680916017}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6810","nonce":2680916017},{"type":"v1","addr":"172.21.15.184:6811","nonce":2680916017}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6814","nonce":2680916017},{"type":"v1","addr":"172.21.15.184:6815","nonce":2680916017}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6812","nonce":2680916017},{"type":"v1","addr":"172.21.15.184:6813","nonce":2680916017}]},"public_addr":"172.21.15.184:6809/2680916017","cluster_addr":"172.21.15.184:6811/2680916017","heartbeat_back_addr":"172.21.15.184:6815/2680916017","heartbeat_front_addr":"172.21.15.184:6813/2680916017","state":["exists","up"]},{"osd":6,"uuid":"cde6f39e-aa1b-4260-ac19-774ee6aaca72","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6824","nonce":2183120457},{"type":"v1","addr":"172.21.15.184:6825","nonce":2183120457}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6826","nonce":2183120457},{"type":"v1","addr":"172.21.15.184:6827","nonce":2183120457}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6830","nonce":2183120457},{"type":"v1","addr":"172.21.15.184:6831","nonce":2183120457}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6828","nonce":2183120457},{"type":"v1","addr":"172.21.15.184:6829","nonce":2183120457}]},"public_addr":"172.21.15.184:6825/2183120457","cluster_addr":"172.21.15.184:6827/2183120457","heartbeat_back_addr":"172.21.15.184:6831/2183120457","heartbeat_front_addr":"172.21.15.184:6829/2183120457","state":["exists","up"]},{"osd":7,"uuid":"4f381778-eee1-47c6-804f-df0e6b33e313","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6800","nonce":4102957285},{"type":"v1","addr":"172.21.15.184:6801","nonce":4102957285}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6802","nonce":4102957285},{"type":"v1","addr":"172.21.15.184:6803","nonce":4102957285}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6806","nonce":4102957285},{"type":"v1","addr":"172.21.15.184:6807","nonce":4102957285}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6804","nonce":4102957285},{"type":"v1","addr":"172.21.15.184:6805","nonce":4102957285}]},"public_addr":"172.21.15.184:6801/4102957285","cluster_addr":"172.21.15.184:6803/4102957285","heartbeat_back_addr":"172.21.15.184:6807/4102957285","heartbeat_front_addr":"172.21.15.184:6805/4102957285","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:35.972210+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.222007+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.198617+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.385663+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.358935+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.156066+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.752973+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:35.682846+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T15:42:55.085 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:55.082+0000 7efc78b27640 1 -- 172.21.15.123:0/2640529728 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7efc54072ec0 msgr2=0x7efc54075380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:42:55.085 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:55.082+0000 7efc78b27640 1 --2- 172.21.15.123:0/2640529728 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7efc54072ec0 0x7efc54075380 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7efc5c0af000 tx=0x7efc5c01b0f0 comp rx=0 tx=0).stop 2024-09-16T15:42:55.085 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:55.082+0000 7efc78b27640 1 -- 172.21.15.123:0/2640529728 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7efc7415df70 msgr2=0x7efc7414abe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:42:55.085 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:55.082+0000 7efc78b27640 1 --2- 172.21.15.123:0/2640529728 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7efc7415df70 0x7efc7414abe0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7efc6c00cce0 tx=0x7efc6c00a620 comp rx=0 tx=0).stop 2024-09-16T15:42:55.085 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:55.082+0000 7efc78b27640 1 -- 172.21.15.123:0/2640529728 shutdown_connections 2024-09-16T15:42:55.086 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:55.082+0000 7efc78b27640 1 --2- 172.21.15.123:0/2640529728 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7efc54072ec0 0x7efc54075380 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:42:55.086 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:55.082+0000 7efc78b27640 1 --2- 172.21.15.123:0/2640529728 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7efc7415df70 0x7efc7414abe0 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:42:55.086 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:55.082+0000 7efc78b27640 1 --2- 172.21.15.123:0/2640529728 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7efc740d4980 0x7efc740b4550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:42:55.086 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:55.082+0000 7efc78b27640 1 --2- 172.21.15.123:0/2640529728 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7efc740b5360 0x7efc740ca450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:42:55.086 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:55.082+0000 7efc78b27640 1 -- 172.21.15.123:0/2640529728 >> 172.21.15.123:0/2640529728 conn(0x7efc740d3e90 msgr2=0x7efc740d4270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:42:55.086 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:55.082+0000 7efc78b27640 1 -- 172.21.15.123:0/2640529728 shutdown_connections 2024-09-16T15:42:55.086 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:55.082+0000 7efc78b27640 1 -- 172.21.15.123:0/2640529728 wait complete. 2024-09-16T15:42:55.100 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-09-16T15:42:55.100 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T15:42:55.219 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:55.218+0000 7ff3df873640 1 -- 172.21.15.123:0/2310601468 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff3d815e160 msgr2=0x7ff3d8073f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:42:55.220 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:55.218+0000 7ff3df873640 1 --2- 172.21.15.123:0/2310601468 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff3d815e160 0x7ff3d8073f30 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7ff3d4009900 tx=0x7ff3d401c7c0 comp rx=0 tx=0).stop 2024-09-16T15:42:55.220 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:55.218+0000 7ff3df873640 1 -- 172.21.15.123:0/2310601468 shutdown_connections 2024-09-16T15:42:55.220 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:55.218+0000 7ff3df873640 1 --2- 172.21.15.123:0/2310601468 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7ff3d815e820 0x7ff3d815ecb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:42:55.220 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:55.218+0000 7ff3df873640 1 --2- 172.21.15.123:0/2310601468 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff3d815e160 0x7ff3d8073f30 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:42:55.220 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:55.218+0000 7ff3df873640 1 -- 172.21.15.123:0/2310601468 >> 172.21.15.123:0/2310601468 conn(0x7ff3d80cfb10 msgr2=0x7ff3d80cff20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:42:55.220 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:55.218+0000 7ff3df873640 1 -- 172.21.15.123:0/2310601468 shutdown_connections 2024-09-16T15:42:55.220 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:55.218+0000 7ff3df873640 1 -- 172.21.15.123:0/2310601468 wait complete. 2024-09-16T15:42:55.220 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:55.218+0000 7ff3df873640 1 Processor -- start 2024-09-16T15:42:55.221 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:55.218+0000 7ff3df873640 1 -- start start 2024-09-16T15:42:55.222 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:55.218+0000 7ff3df873640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff3d8074470 0x7ff3d81522a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:42:55.222 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:55.218+0000 7ff3df873640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7ff3d815e160 0x7ff3d80c6be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:42:55.222 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:55.218+0000 7ff3df873640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff3d815e820 0x7ff3d80c7120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:42:55.223 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:55.218+0000 7ff3df873640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7ff3d80cf5b0 con 0x7ff3d8074470 2024-09-16T15:42:55.223 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:55.218+0000 7ff3df873640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7ff3d80cf430 con 0x7ff3d815e820 2024-09-16T15:42:55.223 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:55.218+0000 7ff3df873640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7ff3d81663c0 con 0x7ff3d815e160 2024-09-16T15:42:55.223 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:55.218+0000 7ff3dd60f640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff3d8074470 0x7ff3d81522a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:42:55.223 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:55.218+0000 7ff3dd60f640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff3d8074470 0x7ff3d81522a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:40186/0 (socket says 172.21.15.123:40186) 2024-09-16T15:42:55.223 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:55.218+0000 7ff3dd60f640 1 -- 172.21.15.123:0/3061540298 learned_addr learned my addr 172.21.15.123:0/3061540298 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:42:55.223 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:55.218+0000 7ff3dde10640 1 --2- 172.21.15.123:0/3061540298 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff3d815e820 0x7ff3d80c7120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:42:55.223 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:55.218+0000 7ff3dce0e640 1 --2- 172.21.15.123:0/3061540298 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7ff3d815e160 0x7ff3d80c6be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:42:55.223 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:55.218+0000 7ff3dd60f640 1 -- 172.21.15.123:0/3061540298 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7ff3d815e160 msgr2=0x7ff3d80c6be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:42:55.223 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:55.218+0000 7ff3dd60f640 1 --2- 172.21.15.123:0/3061540298 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7ff3d815e160 0x7ff3d80c6be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:42:55.223 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:55.218+0000 7ff3dd60f640 1 -- 172.21.15.123:0/3061540298 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff3d815e820 msgr2=0x7ff3d80c7120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:42:55.223 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:55.218+0000 7ff3dd60f640 1 --2- 172.21.15.123:0/3061540298 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff3d815e820 0x7ff3d80c7120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:42:55.223 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:55.218+0000 7ff3dd60f640 1 -- 172.21.15.123:0/3061540298 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff3d8151aa0 con 0x7ff3d8074470 2024-09-16T15:42:55.223 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:55.218+0000 7ff3dce0e640 1 --2- 172.21.15.123:0/3061540298 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7ff3d815e160 0x7ff3d80c6be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T15:42:55.224 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:55.218+0000 7ff3dd60f640 1 --2- 172.21.15.123:0/3061540298 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff3d8074470 0x7ff3d81522a0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7ff3d4009850 tx=0x7ff3d40a5710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:42:55.224 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:55.218+0000 7ff3cdffb640 1 -- 172.21.15.123:0/3061540298 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3d40b2040 con 0x7ff3d8074470 2024-09-16T15:42:55.224 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:55.218+0000 7ff3cdffb640 1 -- 172.21.15.123:0/3061540298 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7ff3d4004400 con 0x7ff3d8074470 2024-09-16T15:42:55.224 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:55.218+0000 7ff3df873640 1 -- 172.21.15.123:0/3061540298 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff3d80b4530 con 0x7ff3d8074470 2024-09-16T15:42:55.224 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:55.218+0000 7ff3cdffb640 1 -- 172.21.15.123:0/3061540298 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3d4005290 con 0x7ff3d8074470 2024-09-16T15:42:55.224 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:55.218+0000 7ff3df873640 1 -- 172.21.15.123:0/3061540298 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff3d80b4af0 con 0x7ff3d8074470 2024-09-16T15:42:55.225 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:55.222+0000 7ff3df873640 1 -- 172.21.15.123:0/3061540298 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff3d8166920 con 0x7ff3d8074470 2024-09-16T15:42:55.225 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:55.222+0000 7ff3cdffb640 1 -- 172.21.15.123:0/3061540298 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 5) ==== 97080+0+0 (secure 0 0 0) 0x7ff3d4005430 con 0x7ff3d8074470 2024-09-16T15:42:55.225 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:55.222+0000 7ff3cdffb640 1 --2- 172.21.15.123:0/3061540298 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7ff3bc072ef0 0x7ff3bc0753b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:42:55.225 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:55.222+0000 7ff3cdffb640 1 -- 172.21.15.123:0/3061540298 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7ff3d412d900 con 0x7ff3d8074470 2024-09-16T15:42:55.225 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:55.222+0000 7ff3dce0e640 1 --2- 172.21.15.123:0/3061540298 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7ff3bc072ef0 0x7ff3bc0753b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:42:55.225 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:55.222+0000 7ff3dce0e640 1 --2- 172.21.15.123:0/3061540298 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7ff3bc072ef0 0x7ff3bc0753b0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7ff3d8153c30 tx=0x7ff3c8009040 comp rx=0 tx=0).ready entity=mgr.4135 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:42:55.228 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:55.226+0000 7ff3cdffb640 1 -- 172.21.15.123:0/3061540298 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7ff3d40164d0 con 0x7ff3d8074470 2024-09-16T15:42:55.402 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:55.398+0000 7ff3df873640 1 -- 172.21.15.123:0/3061540298 --> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7ff3d80d06c0 con 0x7ff3bc072ef0 2024-09-16T15:42:55.414 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:55.414+0000 7ff3cdffb640 1 -- 172.21.15.123:0/3061540298 <== mgr.4135 v2:172.21.15.184:6832/3114346423 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+251941 (secure 0 0 0) 0x7ff3d80d06c0 con 0x7ff3bc072ef0 2024-09-16T15:42:55.417 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-09-16T15:42:55.419 INFO:teuthology.orchestra.run.smithi123.stderr:dumped all 2024-09-16T15:42:55.420 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:55.414+0000 7ff3df873640 1 -- 172.21.15.123:0/3061540298 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7ff3bc072ef0 msgr2=0x7ff3bc0753b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:42:55.420 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:55.414+0000 7ff3df873640 1 --2- 172.21.15.123:0/3061540298 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7ff3bc072ef0 0x7ff3bc0753b0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7ff3d8153c30 tx=0x7ff3c8009040 comp rx=0 tx=0).stop 2024-09-16T15:42:55.420 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:55.414+0000 7ff3df873640 1 -- 172.21.15.123:0/3061540298 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff3d8074470 msgr2=0x7ff3d81522a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:42:55.420 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:55.414+0000 7ff3df873640 1 --2- 172.21.15.123:0/3061540298 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff3d8074470 0x7ff3d81522a0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7ff3d4009850 tx=0x7ff3d40a5710 comp rx=0 tx=0).stop 2024-09-16T15:42:55.420 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:55.414+0000 7ff3df873640 1 -- 172.21.15.123:0/3061540298 shutdown_connections 2024-09-16T15:42:55.420 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:55.414+0000 7ff3df873640 1 --2- 172.21.15.123:0/3061540298 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7ff3bc072ef0 0x7ff3bc0753b0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:42:55.420 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:55.414+0000 7ff3df873640 1 --2- 172.21.15.123:0/3061540298 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff3d815e820 0x7ff3d80c7120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:42:55.420 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:55.414+0000 7ff3df873640 1 --2- 172.21.15.123:0/3061540298 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7ff3d815e160 0x7ff3d80c6be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:42:55.420 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:55.414+0000 7ff3df873640 1 --2- 172.21.15.123:0/3061540298 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff3d8074470 0x7ff3d81522a0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:42:55.420 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:55.414+0000 7ff3df873640 1 -- 172.21.15.123:0/3061540298 >> 172.21.15.123:0/3061540298 conn(0x7ff3d80cfb10 msgr2=0x7ff3d80ce860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:42:55.420 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:55.418+0000 7ff3df873640 1 -- 172.21.15.123:0/3061540298 shutdown_connections 2024-09-16T15:42:55.420 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:55.418+0000 7ff3df873640 1 -- 172.21.15.123:0/3061540298 wait complete. 2024-09-16T15:42:55.435 INFO:teuthology.orchestra.run.smithi123.stdout:{"pg_ready":true,"pg_map":{"version":22,"stamp":"2024-09-16T15:42:55.246068+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":146,"num_write_kb":1642,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":111,"ondisk_log_size":111,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":185,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219140,"kb_used_data":2888,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755580,"statfs":{"total":773094113280,"available":772869713920,"internally_reserved":0,"allocated":2957312,"data_stored":1778268,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":38,"apply_latency_ms":38,"commit_latency_ns":38000000,"apply_latency_ns":38000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.883486"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T15:42:52.725091+0000","last_change":"2024-09-16T15:42:51.434234+0000","last_active":"2024-09-16T15:42:52.725091+0000","last_peered":"2024-09-16T15:42:52.725091+0000","last_clean":"2024-09-16T15:42:52.725091+0000","last_became_active":"2024-09-16T15:42:51.433660+0000","last_became_peered":"2024-09-16T15:42:51.433660+0000","last_unstale":"2024-09-16T15:42:52.725091+0000","last_undegraded":"2024-09-16T15:42:52.725091+0000","last_fullsized":"2024-09-16T15:42:52.725091+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_clean_scrub_stamp":"2024-09-16T15:42:50.361965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T02:07:35.605190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T15:42:52.418662+0000","last_change":"2024-09-16T15:42:51.412677+0000","last_active":"2024-09-16T15:42:52.418662+0000","last_peered":"2024-09-16T15:42:52.418662+0000","last_clean":"2024-09-16T15:42:52.418662+0000","last_became_active":"2024-09-16T15:42:51.412366+0000","last_became_peered":"2024-09-16T15:42:51.412366+0000","last_unstale":"2024-09-16T15:42:52.418662+0000","last_undegraded":"2024-09-16T15:42:52.418662+0000","last_fullsized":"2024-09-16T15:42:52.418662+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_clean_scrub_stamp":"2024-09-16T15:42:50.361965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:21.758409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T15:42:52.876476+0000","last_change":"2024-09-16T15:42:51.406454+0000","last_active":"2024-09-16T15:42:52.876476+0000","last_peered":"2024-09-16T15:42:52.876476+0000","last_clean":"2024-09-16T15:42:52.876476+0000","last_became_active":"2024-09-16T15:42:51.405679+0000","last_became_peered":"2024-09-16T15:42:51.405679+0000","last_unstale":"2024-09-16T15:42:52.876476+0000","last_undegraded":"2024-09-16T15:42:52.876476+0000","last_fullsized":"2024-09-16T15:42:52.876476+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_clean_scrub_stamp":"2024-09-16T15:42:50.361965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T02:33:32.417205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T15:42:52.638879+0000","last_change":"2024-09-16T15:42:51.423931+0000","last_active":"2024-09-16T15:42:52.638879+0000","last_peered":"2024-09-16T15:42:52.638879+0000","last_clean":"2024-09-16T15:42:52.638879+0000","last_became_active":"2024-09-16T15:42:51.423561+0000","last_became_peered":"2024-09-16T15:42:51.423561+0000","last_unstale":"2024-09-16T15:42:52.638879+0000","last_undegraded":"2024-09-16T15:42:52.638879+0000","last_fullsized":"2024-09-16T15:42:52.638879+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_clean_scrub_stamp":"2024-09-16T15:42:50.361965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T03:35:03.081638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T15:42:52.724573+0000","last_change":"2024-09-16T15:42:51.425601+0000","last_active":"2024-09-16T15:42:52.724573+0000","last_peered":"2024-09-16T15:42:52.724573+0000","last_clean":"2024-09-16T15:42:52.724573+0000","last_became_active":"2024-09-16T15:42:51.425040+0000","last_became_peered":"2024-09-16T15:42:51.425040+0000","last_unstale":"2024-09-16T15:42:52.724573+0000","last_undegraded":"2024-09-16T15:42:52.724573+0000","last_fullsized":"2024-09-16T15:42:52.724573+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_clean_scrub_stamp":"2024-09-16T15:42:50.361965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T00:05:49.936465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T15:42:52.638668+0000","last_change":"2024-09-16T15:42:51.429218+0000","last_active":"2024-09-16T15:42:52.638668+0000","last_peered":"2024-09-16T15:42:52.638668+0000","last_clean":"2024-09-16T15:42:52.638668+0000","last_became_active":"2024-09-16T15:42:51.428838+0000","last_became_peered":"2024-09-16T15:42:51.428838+0000","last_unstale":"2024-09-16T15:42:52.638668+0000","last_undegraded":"2024-09-16T15:42:52.638668+0000","last_fullsized":"2024-09-16T15:42:52.638668+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_clean_scrub_stamp":"2024-09-16T15:42:50.361965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:21.758409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T15:42:52.723991+0000","last_change":"2024-09-16T15:42:51.424389+0000","last_active":"2024-09-16T15:42:52.723991+0000","last_peered":"2024-09-16T15:42:52.723991+0000","last_clean":"2024-09-16T15:42:52.723991+0000","last_became_active":"2024-09-16T15:42:51.423840+0000","last_became_peered":"2024-09-16T15:42:51.423840+0000","last_unstale":"2024-09-16T15:42:52.723991+0000","last_undegraded":"2024-09-16T15:42:52.723991+0000","last_fullsized":"2024-09-16T15:42:52.723991+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_clean_scrub_stamp":"2024-09-16T15:42:50.361965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T18:05:24.521404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T15:42:52.874446+0000","last_change":"2024-09-16T15:42:51.428815+0000","last_active":"2024-09-16T15:42:52.874446+0000","last_peered":"2024-09-16T15:42:52.874446+0000","last_clean":"2024-09-16T15:42:52.874446+0000","last_became_active":"2024-09-16T15:42:51.427658+0000","last_became_peered":"2024-09-16T15:42:51.427658+0000","last_unstale":"2024-09-16T15:42:52.874446+0000","last_undegraded":"2024-09-16T15:42:52.874446+0000","last_fullsized":"2024-09-16T15:42:52.874446+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_clean_scrub_stamp":"2024-09-16T15:42:50.361965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T03:23:33.232936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T15:42:52.725119+0000","last_change":"2024-09-16T15:42:51.435164+0000","last_active":"2024-09-16T15:42:52.725119+0000","last_peered":"2024-09-16T15:42:52.725119+0000","last_clean":"2024-09-16T15:42:52.725119+0000","last_became_active":"2024-09-16T15:42:51.434702+0000","last_became_peered":"2024-09-16T15:42:51.434702+0000","last_unstale":"2024-09-16T15:42:52.725119+0000","last_undegraded":"2024-09-16T15:42:52.725119+0000","last_fullsized":"2024-09-16T15:42:52.725119+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_clean_scrub_stamp":"2024-09-16T15:42:50.361965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:40:28.779257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T15:42:52.417480+0000","last_change":"2024-09-16T15:42:51.425817+0000","last_active":"2024-09-16T15:42:52.417480+0000","last_peered":"2024-09-16T15:42:52.417480+0000","last_clean":"2024-09-16T15:42:52.417480+0000","last_became_active":"2024-09-16T15:42:51.425279+0000","last_became_peered":"2024-09-16T15:42:51.425279+0000","last_unstale":"2024-09-16T15:42:52.417480+0000","last_undegraded":"2024-09-16T15:42:52.417480+0000","last_fullsized":"2024-09-16T15:42:52.417480+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_clean_scrub_stamp":"2024-09-16T15:42:50.361965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:10:07.925485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T15:42:52.416844+0000","last_change":"2024-09-16T15:42:51.426883+0000","last_active":"2024-09-16T15:42:52.416844+0000","last_peered":"2024-09-16T15:42:52.416844+0000","last_clean":"2024-09-16T15:42:52.416844+0000","last_became_active":"2024-09-16T15:42:51.426556+0000","last_became_peered":"2024-09-16T15:42:51.426556+0000","last_unstale":"2024-09-16T15:42:52.416844+0000","last_undegraded":"2024-09-16T15:42:52.416844+0000","last_fullsized":"2024-09-16T15:42:52.416844+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_clean_scrub_stamp":"2024-09-16T15:42:50.361965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:55:46.525397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T15:42:52.418805+0000","last_change":"2024-09-16T15:42:51.394761+0000","last_active":"2024-09-16T15:42:52.418805+0000","last_peered":"2024-09-16T15:42:52.418805+0000","last_clean":"2024-09-16T15:42:52.418805+0000","last_became_active":"2024-09-16T15:42:51.394005+0000","last_became_peered":"2024-09-16T15:42:51.394005+0000","last_unstale":"2024-09-16T15:42:52.418805+0000","last_undegraded":"2024-09-16T15:42:52.418805+0000","last_fullsized":"2024-09-16T15:42:52.418805+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_clean_scrub_stamp":"2024-09-16T15:42:50.361965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:17:36.442767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T15:42:52.875964+0000","last_change":"2024-09-16T15:42:51.411614+0000","last_active":"2024-09-16T15:42:52.875964+0000","last_peered":"2024-09-16T15:42:52.875964+0000","last_clean":"2024-09-16T15:42:52.875964+0000","last_became_active":"2024-09-16T15:42:51.409702+0000","last_became_peered":"2024-09-16T15:42:51.409702+0000","last_unstale":"2024-09-16T15:42:52.875964+0000","last_undegraded":"2024-09-16T15:42:52.875964+0000","last_fullsized":"2024-09-16T15:42:52.875964+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_clean_scrub_stamp":"2024-09-16T15:42:50.361965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T00:31:04.101900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T15:42:52.874545+0000","last_change":"2024-09-16T15:42:51.403206+0000","last_active":"2024-09-16T15:42:52.874545+0000","last_peered":"2024-09-16T15:42:52.874545+0000","last_clean":"2024-09-16T15:42:52.874545+0000","last_became_active":"2024-09-16T15:42:51.402528+0000","last_became_peered":"2024-09-16T15:42:51.402528+0000","last_unstale":"2024-09-16T15:42:52.874545+0000","last_undegraded":"2024-09-16T15:42:52.874545+0000","last_fullsized":"2024-09-16T15:42:52.874545+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_clean_scrub_stamp":"2024-09-16T15:42:50.361965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:39:05.860718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T15:42:52.418038+0000","last_change":"2024-09-16T15:42:51.411905+0000","last_active":"2024-09-16T15:42:52.418038+0000","last_peered":"2024-09-16T15:42:52.418038+0000","last_clean":"2024-09-16T15:42:52.418038+0000","last_became_active":"2024-09-16T15:42:51.411540+0000","last_became_peered":"2024-09-16T15:42:51.411540+0000","last_unstale":"2024-09-16T15:42:52.418038+0000","last_undegraded":"2024-09-16T15:42:52.418038+0000","last_fullsized":"2024-09-16T15:42:52.418038+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_clean_scrub_stamp":"2024-09-16T15:42:50.361965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:07:17.519138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T15:42:52.638310+0000","last_change":"2024-09-16T15:42:51.419623+0000","last_active":"2024-09-16T15:42:52.638310+0000","last_peered":"2024-09-16T15:42:52.638310+0000","last_clean":"2024-09-16T15:42:52.638310+0000","last_became_active":"2024-09-16T15:42:51.419319+0000","last_became_peered":"2024-09-16T15:42:51.419319+0000","last_unstale":"2024-09-16T15:42:52.638310+0000","last_undegraded":"2024-09-16T15:42:52.638310+0000","last_fullsized":"2024-09-16T15:42:52.638310+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_clean_scrub_stamp":"2024-09-16T15:42:50.361965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T02:33:57.643582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T15:42:52.640012+0000","last_change":"2024-09-16T15:42:51.436411+0000","last_active":"2024-09-16T15:42:52.640012+0000","last_peered":"2024-09-16T15:42:52.640012+0000","last_clean":"2024-09-16T15:42:52.640012+0000","last_became_active":"2024-09-16T15:42:51.435952+0000","last_became_peered":"2024-09-16T15:42:51.435952+0000","last_unstale":"2024-09-16T15:42:52.640012+0000","last_undegraded":"2024-09-16T15:42:52.640012+0000","last_fullsized":"2024-09-16T15:42:52.640012+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_clean_scrub_stamp":"2024-09-16T15:42:50.361965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:08:57.795460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T15:42:52.738414+0000","last_change":"2024-09-16T15:42:51.404567+0000","last_active":"2024-09-16T15:42:52.738414+0000","last_peered":"2024-09-16T15:42:52.738414+0000","last_clean":"2024-09-16T15:42:52.738414+0000","last_became_active":"2024-09-16T15:42:51.403756+0000","last_became_peered":"2024-09-16T15:42:51.403756+0000","last_unstale":"2024-09-16T15:42:52.738414+0000","last_undegraded":"2024-09-16T15:42:52.738414+0000","last_fullsized":"2024-09-16T15:42:52.738414+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_clean_scrub_stamp":"2024-09-16T15:42:50.361965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T01:04:06.975937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T15:42:52.725148+0000","last_change":"2024-09-16T15:42:51.434196+0000","last_active":"2024-09-16T15:42:52.725148+0000","last_peered":"2024-09-16T15:42:52.725148+0000","last_clean":"2024-09-16T15:42:52.725148+0000","last_became_active":"2024-09-16T15:42:51.433748+0000","last_became_peered":"2024-09-16T15:42:51.433748+0000","last_unstale":"2024-09-16T15:42:52.725148+0000","last_undegraded":"2024-09-16T15:42:52.725148+0000","last_fullsized":"2024-09-16T15:42:52.725148+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_clean_scrub_stamp":"2024-09-16T15:42:50.361965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:08:57.795460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T15:42:52.416804+0000","last_change":"2024-09-16T15:42:51.398201+0000","last_active":"2024-09-16T15:42:52.416804+0000","last_peered":"2024-09-16T15:42:52.416804+0000","last_clean":"2024-09-16T15:42:52.416804+0000","last_became_active":"2024-09-16T15:42:51.396092+0000","last_became_peered":"2024-09-16T15:42:51.396092+0000","last_unstale":"2024-09-16T15:42:52.416804+0000","last_undegraded":"2024-09-16T15:42:52.416804+0000","last_fullsized":"2024-09-16T15:42:52.416804+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_clean_scrub_stamp":"2024-09-16T15:42:50.361965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:16:26.836026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T15:42:52.874601+0000","last_change":"2024-09-16T15:42:51.404937+0000","last_active":"2024-09-16T15:42:52.874601+0000","last_peered":"2024-09-16T15:42:52.874601+0000","last_clean":"2024-09-16T15:42:52.874601+0000","last_became_active":"2024-09-16T15:42:51.404498+0000","last_became_peered":"2024-09-16T15:42:51.404498+0000","last_unstale":"2024-09-16T15:42:52.874601+0000","last_undegraded":"2024-09-16T15:42:52.874601+0000","last_fullsized":"2024-09-16T15:42:52.874601+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_clean_scrub_stamp":"2024-09-16T15:42:50.361965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:16:06.557859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T15:42:52.638770+0000","last_change":"2024-09-16T15:42:51.436767+0000","last_active":"2024-09-16T15:42:52.638770+0000","last_peered":"2024-09-16T15:42:52.638770+0000","last_clean":"2024-09-16T15:42:52.638770+0000","last_became_active":"2024-09-16T15:42:51.436076+0000","last_became_peered":"2024-09-16T15:42:51.436076+0000","last_unstale":"2024-09-16T15:42:52.638770+0000","last_undegraded":"2024-09-16T15:42:52.638770+0000","last_fullsized":"2024-09-16T15:42:52.638770+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_clean_scrub_stamp":"2024-09-16T15:42:50.361965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T02:42:34.108192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T15:42:52.876650+0000","last_change":"2024-09-16T15:42:51.407561+0000","last_active":"2024-09-16T15:42:52.876650+0000","last_peered":"2024-09-16T15:42:52.876650+0000","last_clean":"2024-09-16T15:42:52.876650+0000","last_became_active":"2024-09-16T15:42:51.407242+0000","last_became_peered":"2024-09-16T15:42:51.407242+0000","last_unstale":"2024-09-16T15:42:52.876650+0000","last_undegraded":"2024-09-16T15:42:52.876650+0000","last_fullsized":"2024-09-16T15:42:52.876650+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_clean_scrub_stamp":"2024-09-16T15:42:50.361965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:37:26.430063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T15:42:52.638795+0000","last_change":"2024-09-16T15:42:51.420155+0000","last_active":"2024-09-16T15:42:52.638795+0000","last_peered":"2024-09-16T15:42:52.638795+0000","last_clean":"2024-09-16T15:42:52.638795+0000","last_became_active":"2024-09-16T15:42:51.419779+0000","last_became_peered":"2024-09-16T15:42:51.419779+0000","last_unstale":"2024-09-16T15:42:52.638795+0000","last_undegraded":"2024-09-16T15:42:52.638795+0000","last_fullsized":"2024-09-16T15:42:52.638795+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_clean_scrub_stamp":"2024-09-16T15:42:50.361965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:28:43.699424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T15:42:52.875086+0000","last_change":"2024-09-16T15:42:51.412701+0000","last_active":"2024-09-16T15:42:52.875086+0000","last_peered":"2024-09-16T15:42:52.875086+0000","last_clean":"2024-09-16T15:42:52.875086+0000","last_became_active":"2024-09-16T15:42:51.411770+0000","last_became_peered":"2024-09-16T15:42:51.411770+0000","last_unstale":"2024-09-16T15:42:52.875086+0000","last_undegraded":"2024-09-16T15:42:52.875086+0000","last_fullsized":"2024-09-16T15:42:52.875086+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_clean_scrub_stamp":"2024-09-16T15:42:50.361965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T02:37:19.881471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T15:42:52.417630+0000","last_change":"2024-09-16T15:42:51.423741+0000","last_active":"2024-09-16T15:42:52.417630+0000","last_peered":"2024-09-16T15:42:52.417630+0000","last_clean":"2024-09-16T15:42:52.417630+0000","last_became_active":"2024-09-16T15:42:51.423444+0000","last_became_peered":"2024-09-16T15:42:51.423444+0000","last_unstale":"2024-09-16T15:42:52.417630+0000","last_undegraded":"2024-09-16T15:42:52.417630+0000","last_fullsized":"2024-09-16T15:42:52.417630+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_clean_scrub_stamp":"2024-09-16T15:42:50.361965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T18:56:18.021098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T15:42:52.417698+0000","last_change":"2024-09-16T15:42:51.410257+0000","last_active":"2024-09-16T15:42:52.417698+0000","last_peered":"2024-09-16T15:42:52.417698+0000","last_clean":"2024-09-16T15:42:52.417698+0000","last_became_active":"2024-09-16T15:42:51.409838+0000","last_became_peered":"2024-09-16T15:42:51.409838+0000","last_unstale":"2024-09-16T15:42:52.417698+0000","last_undegraded":"2024-09-16T15:42:52.417698+0000","last_fullsized":"2024-09-16T15:42:52.417698+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_clean_scrub_stamp":"2024-09-16T15:42:50.361965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T01:10:38.539716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T15:42:52.875794+0000","last_change":"2024-09-16T15:42:51.414267+0000","last_active":"2024-09-16T15:42:52.875794+0000","last_peered":"2024-09-16T15:42:52.875794+0000","last_clean":"2024-09-16T15:42:52.875794+0000","last_became_active":"2024-09-16T15:42:51.412201+0000","last_became_peered":"2024-09-16T15:42:51.412201+0000","last_unstale":"2024-09-16T15:42:52.875794+0000","last_undegraded":"2024-09-16T15:42:52.875794+0000","last_fullsized":"2024-09-16T15:42:52.875794+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_clean_scrub_stamp":"2024-09-16T15:42:50.361965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:59:06.080508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T15:42:52.875146+0000","last_change":"2024-09-16T15:42:51.412344+0000","last_active":"2024-09-16T15:42:52.875146+0000","last_peered":"2024-09-16T15:42:52.875146+0000","last_clean":"2024-09-16T15:42:52.875146+0000","last_became_active":"2024-09-16T15:42:51.410933+0000","last_became_peered":"2024-09-16T15:42:51.410933+0000","last_unstale":"2024-09-16T15:42:52.875146+0000","last_undegraded":"2024-09-16T15:42:52.875146+0000","last_fullsized":"2024-09-16T15:42:52.875146+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_clean_scrub_stamp":"2024-09-16T15:42:50.361965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T01:38:29.275555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T15:42:52.639286+0000","last_change":"2024-09-16T15:42:51.419645+0000","last_active":"2024-09-16T15:42:52.639286+0000","last_peered":"2024-09-16T15:42:52.639286+0000","last_clean":"2024-09-16T15:42:52.639286+0000","last_became_active":"2024-09-16T15:42:51.419335+0000","last_became_peered":"2024-09-16T15:42:51.419335+0000","last_unstale":"2024-09-16T15:42:52.639286+0000","last_undegraded":"2024-09-16T15:42:52.639286+0000","last_fullsized":"2024-09-16T15:42:52.639286+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_clean_scrub_stamp":"2024-09-16T15:42:50.361965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T02:58:03.717735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T15:42:52.738151+0000","last_change":"2024-09-16T15:42:51.425478+0000","last_active":"2024-09-16T15:42:52.738151+0000","last_peered":"2024-09-16T15:42:52.738151+0000","last_clean":"2024-09-16T15:42:52.738151+0000","last_became_active":"2024-09-16T15:42:51.425150+0000","last_became_peered":"2024-09-16T15:42:51.425150+0000","last_unstale":"2024-09-16T15:42:52.738151+0000","last_undegraded":"2024-09-16T15:42:52.738151+0000","last_fullsized":"2024-09-16T15:42:52.738151+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_clean_scrub_stamp":"2024-09-16T15:42:50.361965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T00:53:53.123181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T15:42:52.416202+0000","last_change":"2024-09-16T15:42:51.395997+0000","last_active":"2024-09-16T15:42:52.416202+0000","last_peered":"2024-09-16T15:42:52.416202+0000","last_clean":"2024-09-16T15:42:52.416202+0000","last_became_active":"2024-09-16T15:42:51.395182+0000","last_became_peered":"2024-09-16T15:42:51.395182+0000","last_unstale":"2024-09-16T15:42:52.416202+0000","last_undegraded":"2024-09-16T15:42:52.416202+0000","last_fullsized":"2024-09-16T15:42:52.416202+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_clean_scrub_stamp":"2024-09-16T15:42:50.361965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:22:21.461659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T15:42:52.418393+0000","last_change":"2024-09-16T15:42:51.398579+0000","last_active":"2024-09-16T15:42:52.418393+0000","last_peered":"2024-09-16T15:42:52.418393+0000","last_clean":"2024-09-16T15:42:52.418393+0000","last_became_active":"2024-09-16T15:42:51.397865+0000","last_became_peered":"2024-09-16T15:42:51.397865+0000","last_unstale":"2024-09-16T15:42:52.418393+0000","last_undegraded":"2024-09-16T15:42:52.418393+0000","last_fullsized":"2024-09-16T15:42:52.418393+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_clean_scrub_stamp":"2024-09-16T15:42:50.361965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:44:41.707611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"20'1","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T15:42:52.876772+0000","last_change":"2024-09-16T15:42:50.407972+0000","last_active":"2024-09-16T15:42:52.876772+0000","last_peered":"2024-09-16T15:42:52.876772+0000","last_clean":"2024-09-16T15:42:52.876772+0000","last_became_active":"2024-09-16T15:42:50.407571+0000","last_became_peered":"2024-09-16T15:42:50.407571+0000","last_unstale":"2024-09-16T15:42:52.876772+0000","last_undegraded":"2024-09-16T15:42:52.876772+0000","last_fullsized":"2024-09-16T15:42:52.876772+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:42:49.352371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:42:49.352371+0000","last_clean_scrub_stamp":"2024-09-16T15:42:49.352371+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T02:32:51.461439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T15:42:52.876489+0000","last_change":"2024-09-16T15:42:51.403656+0000","last_active":"2024-09-16T15:42:52.876489+0000","last_peered":"2024-09-16T15:42:52.876489+0000","last_clean":"2024-09-16T15:42:52.876489+0000","last_became_active":"2024-09-16T15:42:51.403183+0000","last_became_peered":"2024-09-16T15:42:51.403183+0000","last_unstale":"2024-09-16T15:42:52.876489+0000","last_undegraded":"2024-09-16T15:42:52.876489+0000","last_fullsized":"2024-09-16T15:42:52.876489+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_clean_scrub_stamp":"2024-09-16T15:42:50.361965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T01:01:27.144919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"20'1","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T15:42:52.415759+0000","last_change":"2024-09-16T15:42:50.405255+0000","last_active":"2024-09-16T15:42:52.415759+0000","last_peered":"2024-09-16T15:42:52.415759+0000","last_clean":"2024-09-16T15:42:52.415759+0000","last_became_active":"2024-09-16T15:42:50.404469+0000","last_became_peered":"2024-09-16T15:42:50.404469+0000","last_unstale":"2024-09-16T15:42:52.415759+0000","last_undegraded":"2024-09-16T15:42:52.415759+0000","last_fullsized":"2024-09-16T15:42:52.415759+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:42:49.352371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:42:49.352371+0000","last_clean_scrub_stamp":"2024-09-16T15:42:49.352371+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:41:46.870937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T15:42:52.877073+0000","last_change":"2024-09-16T15:42:50.399198+0000","last_active":"2024-09-16T15:42:52.877073+0000","last_peered":"2024-09-16T15:42:52.877073+0000","last_clean":"2024-09-16T15:42:52.877073+0000","last_became_active":"2024-09-16T15:42:50.398892+0000","last_became_peered":"2024-09-16T15:42:50.398892+0000","last_unstale":"2024-09-16T15:42:52.877073+0000","last_undegraded":"2024-09-16T15:42:52.877073+0000","last_fullsized":"2024-09-16T15:42:52.877073+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:42:49.352371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:42:49.352371+0000","last_clean_scrub_stamp":"2024-09-16T15:42:49.352371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T03:32:21.032967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T15:42:52.875003+0000","last_change":"2024-09-16T15:42:51.412892+0000","last_active":"2024-09-16T15:42:52.875003+0000","last_peered":"2024-09-16T15:42:52.875003+0000","last_clean":"2024-09-16T15:42:52.875003+0000","last_became_active":"2024-09-16T15:42:51.407080+0000","last_became_peered":"2024-09-16T15:42:51.407080+0000","last_unstale":"2024-09-16T15:42:52.875003+0000","last_undegraded":"2024-09-16T15:42:52.875003+0000","last_fullsized":"2024-09-16T15:42:52.875003+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_clean_scrub_stamp":"2024-09-16T15:42:50.361965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T02:17:29.317460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T15:42:52.638690+0000","last_change":"2024-09-16T15:42:50.427196+0000","last_active":"2024-09-16T15:42:52.638690+0000","last_peered":"2024-09-16T15:42:52.638690+0000","last_clean":"2024-09-16T15:42:52.638690+0000","last_became_active":"2024-09-16T15:42:50.426397+0000","last_became_peered":"2024-09-16T15:42:50.426397+0000","last_unstale":"2024-09-16T15:42:52.638690+0000","last_undegraded":"2024-09-16T15:42:52.638690+0000","last_fullsized":"2024-09-16T15:42:52.638690+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:42:49.352371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:42:49.352371+0000","last_clean_scrub_stamp":"2024-09-16T15:42:49.352371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T22:48:39.726611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T15:42:52.417585+0000","last_change":"2024-09-16T15:42:51.410683+0000","last_active":"2024-09-16T15:42:52.417585+0000","last_peered":"2024-09-16T15:42:52.417585+0000","last_clean":"2024-09-16T15:42:52.417585+0000","last_became_active":"2024-09-16T15:42:51.409695+0000","last_became_peered":"2024-09-16T15:42:51.409695+0000","last_unstale":"2024-09-16T15:42:52.417585+0000","last_undegraded":"2024-09-16T15:42:52.417585+0000","last_fullsized":"2024-09-16T15:42:52.417585+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_clean_scrub_stamp":"2024-09-16T15:42:50.361965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T01:38:29.275555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T15:42:52.417220+0000","last_change":"2024-09-16T15:42:50.420642+0000","last_active":"2024-09-16T15:42:52.417220+0000","last_peered":"2024-09-16T15:42:52.417220+0000","last_clean":"2024-09-16T15:42:52.417220+0000","last_became_active":"2024-09-16T15:42:50.419895+0000","last_became_peered":"2024-09-16T15:42:50.419895+0000","last_unstale":"2024-09-16T15:42:52.417220+0000","last_undegraded":"2024-09-16T15:42:52.417220+0000","last_fullsized":"2024-09-16T15:42:52.417220+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:42:49.352371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:42:49.352371+0000","last_clean_scrub_stamp":"2024-09-16T15:42:49.352371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:40:01.292941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T15:42:52.639747+0000","last_change":"2024-09-16T15:42:51.434841+0000","last_active":"2024-09-16T15:42:52.639747+0000","last_peered":"2024-09-16T15:42:52.639747+0000","last_clean":"2024-09-16T15:42:52.639747+0000","last_became_active":"2024-09-16T15:42:51.434532+0000","last_became_peered":"2024-09-16T15:42:51.434532+0000","last_unstale":"2024-09-16T15:42:52.639747+0000","last_undegraded":"2024-09-16T15:42:52.639747+0000","last_fullsized":"2024-09-16T15:42:52.639747+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_clean_scrub_stamp":"2024-09-16T15:42:50.361965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T18:44:49.865687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T15:42:52.639491+0000","last_change":"2024-09-16T15:42:50.419196+0000","last_active":"2024-09-16T15:42:52.639491+0000","last_peered":"2024-09-16T15:42:52.639491+0000","last_clean":"2024-09-16T15:42:52.639491+0000","last_became_active":"2024-09-16T15:42:50.415873+0000","last_became_peered":"2024-09-16T15:42:50.415873+0000","last_unstale":"2024-09-16T15:42:52.639491+0000","last_undegraded":"2024-09-16T15:42:52.639491+0000","last_fullsized":"2024-09-16T15:42:52.639491+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:42:49.352371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:42:49.352371+0000","last_clean_scrub_stamp":"2024-09-16T15:42:49.352371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:26:07.798125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T15:42:52.877644+0000","last_change":"2024-09-16T15:42:51.425682+0000","last_active":"2024-09-16T15:42:52.877644+0000","last_peered":"2024-09-16T15:42:52.877644+0000","last_clean":"2024-09-16T15:42:52.877644+0000","last_became_active":"2024-09-16T15:42:51.425336+0000","last_became_peered":"2024-09-16T15:42:51.425336+0000","last_unstale":"2024-09-16T15:42:52.877644+0000","last_undegraded":"2024-09-16T15:42:52.877644+0000","last_fullsized":"2024-09-16T15:42:52.877644+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_clean_scrub_stamp":"2024-09-16T15:42:50.361965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T02:33:57.643582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"20'3","reported_seq":18,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T15:42:52.418373+0000","last_change":"2024-09-16T15:42:50.415090+0000","last_active":"2024-09-16T15:42:52.418373+0000","last_peered":"2024-09-16T15:42:52.418373+0000","last_clean":"2024-09-16T15:42:52.418373+0000","last_became_active":"2024-09-16T15:42:50.414818+0000","last_became_peered":"2024-09-16T15:42:50.414818+0000","last_unstale":"2024-09-16T15:42:52.418373+0000","last_undegraded":"2024-09-16T15:42:52.418373+0000","last_fullsized":"2024-09-16T15:42:52.418373+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:42:49.352371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:42:49.352371+0000","last_clean_scrub_stamp":"2024-09-16T15:42:49.352371+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:36:36.506348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T15:42:52.419154+0000","last_change":"2024-09-16T15:42:51.425357+0000","last_active":"2024-09-16T15:42:52.419154+0000","last_peered":"2024-09-16T15:42:52.419154+0000","last_clean":"2024-09-16T15:42:52.419154+0000","last_became_active":"2024-09-16T15:42:51.425074+0000","last_became_peered":"2024-09-16T15:42:51.425074+0000","last_unstale":"2024-09-16T15:42:52.419154+0000","last_undegraded":"2024-09-16T15:42:52.419154+0000","last_fullsized":"2024-09-16T15:42:52.419154+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_clean_scrub_stamp":"2024-09-16T15:42:50.361965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:59:03.380666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"20'6","reported_seq":21,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T15:42:52.416548+0000","last_change":"2024-09-16T15:42:50.396749+0000","last_active":"2024-09-16T15:42:52.416548+0000","last_peered":"2024-09-16T15:42:52.416548+0000","last_clean":"2024-09-16T15:42:52.416548+0000","last_became_active":"2024-09-16T15:42:50.395963+0000","last_became_peered":"2024-09-16T15:42:50.395963+0000","last_unstale":"2024-09-16T15:42:52.416548+0000","last_undegraded":"2024-09-16T15:42:52.416548+0000","last_fullsized":"2024-09-16T15:42:52.416548+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:42:49.352371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:42:49.352371+0000","last_clean_scrub_stamp":"2024-09-16T15:42:49.352371+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T23:01:15.996297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T15:42:52.724385+0000","last_change":"2024-09-16T15:42:51.427006+0000","last_active":"2024-09-16T15:42:52.724385+0000","last_peered":"2024-09-16T15:42:52.724385+0000","last_clean":"2024-09-16T15:42:52.724385+0000","last_became_active":"2024-09-16T15:42:51.426730+0000","last_became_peered":"2024-09-16T15:42:51.426730+0000","last_unstale":"2024-09-16T15:42:52.724385+0000","last_undegraded":"2024-09-16T15:42:52.724385+0000","last_fullsized":"2024-09-16T15:42:52.724385+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_clean_scrub_stamp":"2024-09-16T15:42:50.361965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:49:13.477881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"20'2","reported_seq":17,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T15:42:52.876798+0000","last_change":"2024-09-16T15:42:50.401803+0000","last_active":"2024-09-16T15:42:52.876798+0000","last_peered":"2024-09-16T15:42:52.876798+0000","last_clean":"2024-09-16T15:42:52.876798+0000","last_became_active":"2024-09-16T15:42:50.401495+0000","last_became_peered":"2024-09-16T15:42:50.401495+0000","last_unstale":"2024-09-16T15:42:52.876798+0000","last_undegraded":"2024-09-16T15:42:52.876798+0000","last_fullsized":"2024-09-16T15:42:52.876798+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:42:49.352371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:42:49.352371+0000","last_clean_scrub_stamp":"2024-09-16T15:42:49.352371+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T18:06:23.054055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T15:42:52.875115+0000","last_change":"2024-09-16T15:42:51.428923+0000","last_active":"2024-09-16T15:42:52.875115+0000","last_peered":"2024-09-16T15:42:52.875115+0000","last_clean":"2024-09-16T15:42:52.875115+0000","last_became_active":"2024-09-16T15:42:51.428550+0000","last_became_peered":"2024-09-16T15:42:51.428550+0000","last_unstale":"2024-09-16T15:42:52.875115+0000","last_undegraded":"2024-09-16T15:42:52.875115+0000","last_fullsized":"2024-09-16T15:42:52.875115+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_clean_scrub_stamp":"2024-09-16T15:42:50.361965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:57:03.171257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T15:42:52.639097+0000","last_change":"2024-09-16T15:42:50.426919+0000","last_active":"2024-09-16T15:42:52.639097+0000","last_peered":"2024-09-16T15:42:52.639097+0000","last_clean":"2024-09-16T15:42:52.639097+0000","last_became_active":"2024-09-16T15:42:50.426141+0000","last_became_peered":"2024-09-16T15:42:50.426141+0000","last_unstale":"2024-09-16T15:42:52.639097+0000","last_undegraded":"2024-09-16T15:42:52.639097+0000","last_fullsized":"2024-09-16T15:42:52.639097+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:42:49.352371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:42:49.352371+0000","last_clean_scrub_stamp":"2024-09-16T15:42:49.352371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:23:09.046920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T15:42:52.724923+0000","last_change":"2024-09-16T15:42:51.425867+0000","last_active":"2024-09-16T15:42:52.724923+0000","last_peered":"2024-09-16T15:42:52.724923+0000","last_clean":"2024-09-16T15:42:52.724923+0000","last_became_active":"2024-09-16T15:42:51.425313+0000","last_became_peered":"2024-09-16T15:42:51.425313+0000","last_unstale":"2024-09-16T15:42:52.724923+0000","last_undegraded":"2024-09-16T15:42:52.724923+0000","last_fullsized":"2024-09-16T15:42:52.724923+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_clean_scrub_stamp":"2024-09-16T15:42:50.361965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T18:02:05.435198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T15:42:52.415548+0000","last_change":"2024-09-16T15:42:50.405263+0000","last_active":"2024-09-16T15:42:52.415548+0000","last_peered":"2024-09-16T15:42:52.415548+0000","last_clean":"2024-09-16T15:42:52.415548+0000","last_became_active":"2024-09-16T15:42:50.404419+0000","last_became_peered":"2024-09-16T15:42:50.404419+0000","last_unstale":"2024-09-16T15:42:52.415548+0000","last_undegraded":"2024-09-16T15:42:52.415548+0000","last_fullsized":"2024-09-16T15:42:52.415548+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:42:49.352371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:42:49.352371+0000","last_clean_scrub_stamp":"2024-09-16T15:42:49.352371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:40:33.707149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T15:42:52.415669+0000","last_change":"2024-09-16T15:42:51.397798+0000","last_active":"2024-09-16T15:42:52.415669+0000","last_peered":"2024-09-16T15:42:52.415669+0000","last_clean":"2024-09-16T15:42:52.415669+0000","last_became_active":"2024-09-16T15:42:51.397070+0000","last_became_peered":"2024-09-16T15:42:51.397070+0000","last_unstale":"2024-09-16T15:42:52.415669+0000","last_undegraded":"2024-09-16T15:42:52.415669+0000","last_fullsized":"2024-09-16T15:42:52.415669+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_clean_scrub_stamp":"2024-09-16T15:42:50.361965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:24:48.779934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"20'8","reported_seq":23,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T15:42:52.737837+0000","last_change":"2024-09-16T15:42:50.386440+0000","last_active":"2024-09-16T15:42:52.737837+0000","last_peered":"2024-09-16T15:42:52.737837+0000","last_clean":"2024-09-16T15:42:52.737837+0000","last_became_active":"2024-09-16T15:42:50.386100+0000","last_became_peered":"2024-09-16T15:42:50.386100+0000","last_unstale":"2024-09-16T15:42:52.737837+0000","last_undegraded":"2024-09-16T15:42:52.737837+0000","last_fullsized":"2024-09-16T15:42:52.737837+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:42:49.352371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:42:49.352371+0000","last_clean_scrub_stamp":"2024-09-16T15:42:49.352371+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:32:51.019126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T15:42:52.723952+0000","last_change":"2024-09-16T15:42:51.424679+0000","last_active":"2024-09-16T15:42:52.723952+0000","last_peered":"2024-09-16T15:42:52.723952+0000","last_clean":"2024-09-16T15:42:52.723952+0000","last_became_active":"2024-09-16T15:42:51.424119+0000","last_became_peered":"2024-09-16T15:42:51.424119+0000","last_unstale":"2024-09-16T15:42:52.723952+0000","last_undegraded":"2024-09-16T15:42:52.723952+0000","last_fullsized":"2024-09-16T15:42:52.723952+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_clean_scrub_stamp":"2024-09-16T15:42:50.361965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T23:53:36.869388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"20'1","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T15:42:52.419009+0000","last_change":"2024-09-16T15:42:50.395317+0000","last_active":"2024-09-16T15:42:52.419009+0000","last_peered":"2024-09-16T15:42:52.419009+0000","last_clean":"2024-09-16T15:42:52.419009+0000","last_became_active":"2024-09-16T15:42:50.394495+0000","last_became_peered":"2024-09-16T15:42:50.394495+0000","last_unstale":"2024-09-16T15:42:52.419009+0000","last_undegraded":"2024-09-16T15:42:52.419009+0000","last_fullsized":"2024-09-16T15:42:52.419009+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:42:49.352371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:42:49.352371+0000","last_clean_scrub_stamp":"2024-09-16T15:42:49.352371+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:45:54.681341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T15:42:52.724959+0000","last_change":"2024-09-16T15:42:51.423467+0000","last_active":"2024-09-16T15:42:52.724959+0000","last_peered":"2024-09-16T15:42:52.724959+0000","last_clean":"2024-09-16T15:42:52.724959+0000","last_became_active":"2024-09-16T15:42:51.423161+0000","last_became_peered":"2024-09-16T15:42:51.423161+0000","last_unstale":"2024-09-16T15:42:52.724959+0000","last_undegraded":"2024-09-16T15:42:52.724959+0000","last_fullsized":"2024-09-16T15:42:52.724959+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_clean_scrub_stamp":"2024-09-16T15:42:50.361965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T23:53:22.795898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"20'2","reported_seq":17,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T15:42:52.738233+0000","last_change":"2024-09-16T15:42:50.384867+0000","last_active":"2024-09-16T15:42:52.738233+0000","last_peered":"2024-09-16T15:42:52.738233+0000","last_clean":"2024-09-16T15:42:52.738233+0000","last_became_active":"2024-09-16T15:42:50.384581+0000","last_became_peered":"2024-09-16T15:42:50.384581+0000","last_unstale":"2024-09-16T15:42:52.738233+0000","last_undegraded":"2024-09-16T15:42:52.738233+0000","last_fullsized":"2024-09-16T15:42:52.738233+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:42:49.352371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:42:49.352371+0000","last_clean_scrub_stamp":"2024-09-16T15:42:49.352371+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T22:48:39.726611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T15:42:52.725417+0000","last_change":"2024-09-16T15:42:51.434874+0000","last_active":"2024-09-16T15:42:52.725417+0000","last_peered":"2024-09-16T15:42:52.725417+0000","last_clean":"2024-09-16T15:42:52.725417+0000","last_became_active":"2024-09-16T15:42:51.434561+0000","last_became_peered":"2024-09-16T15:42:51.434561+0000","last_unstale":"2024-09-16T15:42:52.725417+0000","last_undegraded":"2024-09-16T15:42:52.725417+0000","last_fullsized":"2024-09-16T15:42:52.725417+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_clean_scrub_stamp":"2024-09-16T15:42:50.361965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:13:10.293897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"20'1","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T15:42:52.418305+0000","last_change":"2024-09-16T15:42:50.399974+0000","last_active":"2024-09-16T15:42:52.418305+0000","last_peered":"2024-09-16T15:42:52.418305+0000","last_clean":"2024-09-16T15:42:52.418305+0000","last_became_active":"2024-09-16T15:42:50.399714+0000","last_became_peered":"2024-09-16T15:42:50.399714+0000","last_unstale":"2024-09-16T15:42:52.418305+0000","last_undegraded":"2024-09-16T15:42:52.418305+0000","last_fullsized":"2024-09-16T15:42:52.418305+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:42:49.352371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:42:49.352371+0000","last_clean_scrub_stamp":"2024-09-16T15:42:49.352371+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T02:37:24.494137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T15:42:52.639516+0000","last_change":"2024-09-16T15:42:51.420665+0000","last_active":"2024-09-16T15:42:52.639516+0000","last_peered":"2024-09-16T15:42:52.639516+0000","last_clean":"2024-09-16T15:42:52.639516+0000","last_became_active":"2024-09-16T15:42:51.420340+0000","last_became_peered":"2024-09-16T15:42:51.420340+0000","last_unstale":"2024-09-16T15:42:52.639516+0000","last_undegraded":"2024-09-16T15:42:52.639516+0000","last_fullsized":"2024-09-16T15:42:52.639516+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_clean_scrub_stamp":"2024-09-16T15:42:50.361965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T00:05:49.936465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T15:42:52.738027+0000","last_change":"2024-09-16T15:42:50.412850+0000","last_active":"2024-09-16T15:42:52.738027+0000","last_peered":"2024-09-16T15:42:52.738027+0000","last_clean":"2024-09-16T15:42:52.738027+0000","last_became_active":"2024-09-16T15:42:50.412492+0000","last_became_peered":"2024-09-16T15:42:50.412492+0000","last_unstale":"2024-09-16T15:42:52.738027+0000","last_undegraded":"2024-09-16T15:42:52.738027+0000","last_fullsized":"2024-09-16T15:42:52.738027+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:42:49.352371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:42:49.352371+0000","last_clean_scrub_stamp":"2024-09-16T15:42:49.352371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T22:41:47.420539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T15:42:52.724515+0000","last_change":"2024-09-16T15:42:51.428052+0000","last_active":"2024-09-16T15:42:52.724515+0000","last_peered":"2024-09-16T15:42:52.724515+0000","last_clean":"2024-09-16T15:42:52.724515+0000","last_became_active":"2024-09-16T15:42:51.427776+0000","last_became_peered":"2024-09-16T15:42:51.427776+0000","last_unstale":"2024-09-16T15:42:52.724515+0000","last_undegraded":"2024-09-16T15:42:52.724515+0000","last_fullsized":"2024-09-16T15:42:52.724515+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_clean_scrub_stamp":"2024-09-16T15:42:50.361965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T22:34:25.305861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"16'72","reported_seq":111,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T15:42:52.723702+0000","last_change":"2024-09-16T15:42:40.881820+0000","last_active":"2024-09-16T15:42:52.723702+0000","last_peered":"2024-09-16T15:42:52.723702+0000","last_clean":"2024-09-16T15:42:52.723702+0000","last_became_active":"2024-09-16T15:42:40.879965+0000","last_became_peered":"2024-09-16T15:42:40.879965+0000","last_unstale":"2024-09-16T15:42:52.723702+0000","last_undegraded":"2024-09-16T15:42:52.723702+0000","last_fullsized":"2024-09-16T15:42:52.723702+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:42:39.728762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:42:39.728762+0000","last_clean_scrub_stamp":"2024-09-16T15:42:39.728762+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:22:18.768932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T15:42:52.638416+0000","last_change":"2024-09-16T15:42:50.413082+0000","last_active":"2024-09-16T15:42:52.638416+0000","last_peered":"2024-09-16T15:42:52.638416+0000","last_clean":"2024-09-16T15:42:52.638416+0000","last_became_active":"2024-09-16T15:42:50.412050+0000","last_became_peered":"2024-09-16T15:42:50.412050+0000","last_unstale":"2024-09-16T15:42:52.638416+0000","last_undegraded":"2024-09-16T15:42:52.638416+0000","last_fullsized":"2024-09-16T15:42:52.638416+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:42:49.352371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:42:49.352371+0000","last_clean_scrub_stamp":"2024-09-16T15:42:49.352371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:32:51.019126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T15:42:52.875839+0000","last_change":"2024-09-16T15:42:51.430477+0000","last_active":"2024-09-16T15:42:52.875839+0000","last_peered":"2024-09-16T15:42:52.875839+0000","last_clean":"2024-09-16T15:42:52.875839+0000","last_became_active":"2024-09-16T15:42:51.430071+0000","last_became_peered":"2024-09-16T15:42:51.430071+0000","last_unstale":"2024-09-16T15:42:52.875839+0000","last_undegraded":"2024-09-16T15:42:52.875839+0000","last_fullsized":"2024-09-16T15:42:52.875839+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_clean_scrub_stamp":"2024-09-16T15:42:50.361965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T18:56:18.021098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T15:42:52.723623+0000","last_change":"2024-09-16T15:42:50.419981+0000","last_active":"2024-09-16T15:42:52.723623+0000","last_peered":"2024-09-16T15:42:52.723623+0000","last_clean":"2024-09-16T15:42:52.723623+0000","last_became_active":"2024-09-16T15:42:50.418405+0000","last_became_peered":"2024-09-16T15:42:50.418405+0000","last_unstale":"2024-09-16T15:42:52.723623+0000","last_undegraded":"2024-09-16T15:42:52.723623+0000","last_fullsized":"2024-09-16T15:42:52.723623+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:42:49.352371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:42:49.352371+0000","last_clean_scrub_stamp":"2024-09-16T15:42:49.352371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T00:53:52.113587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T15:42:52.874487+0000","last_change":"2024-09-16T15:42:51.403404+0000","last_active":"2024-09-16T15:42:52.874487+0000","last_peered":"2024-09-16T15:42:52.874487+0000","last_clean":"2024-09-16T15:42:52.874487+0000","last_became_active":"2024-09-16T15:42:51.402978+0000","last_became_peered":"2024-09-16T15:42:51.402978+0000","last_unstale":"2024-09-16T15:42:52.874487+0000","last_undegraded":"2024-09-16T15:42:52.874487+0000","last_fullsized":"2024-09-16T15:42:52.874487+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_clean_scrub_stamp":"2024-09-16T15:42:50.361965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T00:04:54.212949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T15:42:52.738033+0000","last_change":"2024-09-16T15:42:50.391465+0000","last_active":"2024-09-16T15:42:52.738033+0000","last_peered":"2024-09-16T15:42:52.738033+0000","last_clean":"2024-09-16T15:42:52.738033+0000","last_became_active":"2024-09-16T15:42:50.391091+0000","last_became_peered":"2024-09-16T15:42:50.391091+0000","last_unstale":"2024-09-16T15:42:52.738033+0000","last_undegraded":"2024-09-16T15:42:52.738033+0000","last_fullsized":"2024-09-16T15:42:52.738033+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:42:49.352371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:42:49.352371+0000","last_clean_scrub_stamp":"2024-09-16T15:42:49.352371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:26:07.798125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T15:42:52.417576+0000","last_change":"2024-09-16T15:42:51.404294+0000","last_active":"2024-09-16T15:42:52.417576+0000","last_peered":"2024-09-16T15:42:52.417576+0000","last_clean":"2024-09-16T15:42:52.417576+0000","last_became_active":"2024-09-16T15:42:51.403997+0000","last_became_peered":"2024-09-16T15:42:51.403997+0000","last_unstale":"2024-09-16T15:42:52.417576+0000","last_undegraded":"2024-09-16T15:42:52.417576+0000","last_fullsized":"2024-09-16T15:42:52.417576+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_clean_scrub_stamp":"2024-09-16T15:42:50.361965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:20:36.872940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T15:42:52.639350+0000","last_change":"2024-09-16T15:42:50.422498+0000","last_active":"2024-09-16T15:42:52.639350+0000","last_peered":"2024-09-16T15:42:52.639350+0000","last_clean":"2024-09-16T15:42:52.639350+0000","last_became_active":"2024-09-16T15:42:50.421727+0000","last_became_peered":"2024-09-16T15:42:50.421727+0000","last_unstale":"2024-09-16T15:42:52.639350+0000","last_undegraded":"2024-09-16T15:42:52.639350+0000","last_fullsized":"2024-09-16T15:42:52.639350+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:42:49.352371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:42:49.352371+0000","last_clean_scrub_stamp":"2024-09-16T15:42:49.352371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:36:36.506348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T15:42:52.419302+0000","last_change":"2024-09-16T15:42:51.413177+0000","last_active":"2024-09-16T15:42:52.419302+0000","last_peered":"2024-09-16T15:42:52.419302+0000","last_clean":"2024-09-16T15:42:52.419302+0000","last_became_active":"2024-09-16T15:42:51.412881+0000","last_became_peered":"2024-09-16T15:42:51.412881+0000","last_unstale":"2024-09-16T15:42:52.419302+0000","last_undegraded":"2024-09-16T15:42:52.419302+0000","last_fullsized":"2024-09-16T15:42:52.419302+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_clean_scrub_stamp":"2024-09-16T15:42:50.361965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:10:16.623518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T15:42:52.418085+0000","last_change":"2024-09-16T15:42:50.394813+0000","last_active":"2024-09-16T15:42:52.418085+0000","last_peered":"2024-09-16T15:42:52.418085+0000","last_clean":"2024-09-16T15:42:52.418085+0000","last_became_active":"2024-09-16T15:42:50.389679+0000","last_became_peered":"2024-09-16T15:42:50.389679+0000","last_unstale":"2024-09-16T15:42:52.418085+0000","last_undegraded":"2024-09-16T15:42:52.418085+0000","last_fullsized":"2024-09-16T15:42:52.418085+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:42:49.352371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:42:49.352371+0000","last_clean_scrub_stamp":"2024-09-16T15:42:49.352371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:23:09.046920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T15:42:52.639557+0000","last_change":"2024-09-16T15:42:51.424119+0000","last_active":"2024-09-16T15:42:52.639557+0000","last_peered":"2024-09-16T15:42:52.639557+0000","last_clean":"2024-09-16T15:42:52.639557+0000","last_became_active":"2024-09-16T15:42:51.423771+0000","last_became_peered":"2024-09-16T15:42:51.423771+0000","last_unstale":"2024-09-16T15:42:52.639557+0000","last_undegraded":"2024-09-16T15:42:52.639557+0000","last_fullsized":"2024-09-16T15:42:52.639557+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_clean_scrub_stamp":"2024-09-16T15:42:50.361965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T23:53:22.795898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T15:42:52.724991+0000","last_change":"2024-09-16T15:42:50.419065+0000","last_active":"2024-09-16T15:42:52.724991+0000","last_peered":"2024-09-16T15:42:52.724991+0000","last_clean":"2024-09-16T15:42:52.724991+0000","last_became_active":"2024-09-16T15:42:50.418438+0000","last_became_peered":"2024-09-16T15:42:50.418438+0000","last_unstale":"2024-09-16T15:42:52.724991+0000","last_undegraded":"2024-09-16T15:42:52.724991+0000","last_fullsized":"2024-09-16T15:42:52.724991+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:42:49.352371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:42:49.352371+0000","last_clean_scrub_stamp":"2024-09-16T15:42:49.352371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T01:42:29.977143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T15:42:52.418120+0000","last_change":"2024-09-16T15:42:51.407174+0000","last_active":"2024-09-16T15:42:52.418120+0000","last_peered":"2024-09-16T15:42:52.418120+0000","last_clean":"2024-09-16T15:42:52.418120+0000","last_became_active":"2024-09-16T15:42:51.406880+0000","last_became_peered":"2024-09-16T15:42:51.406880+0000","last_unstale":"2024-09-16T15:42:52.418120+0000","last_undegraded":"2024-09-16T15:42:52.418120+0000","last_fullsized":"2024-09-16T15:42:52.418120+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_clean_scrub_stamp":"2024-09-16T15:42:50.361965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T02:33:57.643582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"20'1","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T15:42:52.418471+0000","last_change":"2024-09-16T15:42:50.397599+0000","last_active":"2024-09-16T15:42:52.418471+0000","last_peered":"2024-09-16T15:42:52.418471+0000","last_clean":"2024-09-16T15:42:52.418471+0000","last_became_active":"2024-09-16T15:42:50.397301+0000","last_became_peered":"2024-09-16T15:42:50.397301+0000","last_unstale":"2024-09-16T15:42:52.418471+0000","last_undegraded":"2024-09-16T15:42:52.418471+0000","last_fullsized":"2024-09-16T15:42:52.418471+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:42:49.352371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:42:49.352371+0000","last_clean_scrub_stamp":"2024-09-16T15:42:49.352371+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T01:49:08.734586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T15:42:52.875834+0000","last_change":"2024-09-16T15:42:51.424110+0000","last_active":"2024-09-16T15:42:52.875834+0000","last_peered":"2024-09-16T15:42:52.875834+0000","last_clean":"2024-09-16T15:42:52.875834+0000","last_became_active":"2024-09-16T15:42:51.423798+0000","last_became_peered":"2024-09-16T15:42:51.423798+0000","last_unstale":"2024-09-16T15:42:52.875834+0000","last_undegraded":"2024-09-16T15:42:52.875834+0000","last_fullsized":"2024-09-16T15:42:52.875834+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_clean_scrub_stamp":"2024-09-16T15:42:50.361965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T22:53:48.581346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"20'2","reported_seq":17,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T15:42:52.417268+0000","last_change":"2024-09-16T15:42:50.416032+0000","last_active":"2024-09-16T15:42:52.417268+0000","last_peered":"2024-09-16T15:42:52.417268+0000","last_clean":"2024-09-16T15:42:52.417268+0000","last_became_active":"2024-09-16T15:42:50.415325+0000","last_became_peered":"2024-09-16T15:42:50.415325+0000","last_unstale":"2024-09-16T15:42:52.417268+0000","last_undegraded":"2024-09-16T15:42:52.417268+0000","last_fullsized":"2024-09-16T15:42:52.417268+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:42:49.352371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:42:49.352371+0000","last_clean_scrub_stamp":"2024-09-16T15:42:49.352371+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T02:11:26.332763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T15:42:52.877501+0000","last_change":"2024-09-16T15:42:51.407137+0000","last_active":"2024-09-16T15:42:52.877501+0000","last_peered":"2024-09-16T15:42:52.877501+0000","last_clean":"2024-09-16T15:42:52.877501+0000","last_became_active":"2024-09-16T15:42:51.406766+0000","last_became_peered":"2024-09-16T15:42:51.406766+0000","last_unstale":"2024-09-16T15:42:52.877501+0000","last_undegraded":"2024-09-16T15:42:52.877501+0000","last_fullsized":"2024-09-16T15:42:52.877501+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_clean_scrub_stamp":"2024-09-16T15:42:50.361965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:07:17.519138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T15:42:52.723539+0000","last_change":"2024-09-16T15:42:50.419106+0000","last_active":"2024-09-16T15:42:52.723539+0000","last_peered":"2024-09-16T15:42:52.723539+0000","last_clean":"2024-09-16T15:42:52.723539+0000","last_became_active":"2024-09-16T15:42:50.417462+0000","last_became_peered":"2024-09-16T15:42:50.417462+0000","last_unstale":"2024-09-16T15:42:52.723539+0000","last_undegraded":"2024-09-16T15:42:52.723539+0000","last_fullsized":"2024-09-16T15:42:52.723539+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:42:49.352371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:42:49.352371+0000","last_clean_scrub_stamp":"2024-09-16T15:42:49.352371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T22:58:24.474495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T15:42:52.418379+0000","last_change":"2024-09-16T15:42:51.412043+0000","last_active":"2024-09-16T15:42:52.418379+0000","last_peered":"2024-09-16T15:42:52.418379+0000","last_clean":"2024-09-16T15:42:52.418379+0000","last_became_active":"2024-09-16T15:42:51.411434+0000","last_became_peered":"2024-09-16T15:42:51.411434+0000","last_unstale":"2024-09-16T15:42:52.418379+0000","last_undegraded":"2024-09-16T15:42:52.418379+0000","last_fullsized":"2024-09-16T15:42:52.418379+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_clean_scrub_stamp":"2024-09-16T15:42:50.361965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:40:52.417913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T15:42:52.417895+0000","last_change":"2024-09-16T15:42:50.408565+0000","last_active":"2024-09-16T15:42:52.417895+0000","last_peered":"2024-09-16T15:42:52.417895+0000","last_clean":"2024-09-16T15:42:52.417895+0000","last_became_active":"2024-09-16T15:42:50.408254+0000","last_became_peered":"2024-09-16T15:42:50.408254+0000","last_unstale":"2024-09-16T15:42:52.417895+0000","last_undegraded":"2024-09-16T15:42:52.417895+0000","last_fullsized":"2024-09-16T15:42:52.417895+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:42:49.352371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:42:49.352371+0000","last_clean_scrub_stamp":"2024-09-16T15:42:49.352371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:26:07.798125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T15:42:52.876598+0000","last_change":"2024-09-16T15:42:51.413873+0000","last_active":"2024-09-16T15:42:52.876598+0000","last_peered":"2024-09-16T15:42:52.876598+0000","last_clean":"2024-09-16T15:42:52.876598+0000","last_became_active":"2024-09-16T15:42:51.406117+0000","last_became_peered":"2024-09-16T15:42:51.406117+0000","last_unstale":"2024-09-16T15:42:52.876598+0000","last_undegraded":"2024-09-16T15:42:52.876598+0000","last_fullsized":"2024-09-16T15:42:52.876598+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_clean_scrub_stamp":"2024-09-16T15:42:50.361965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:12:23.570707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T15:42:52.417696+0000","last_change":"2024-09-16T15:42:50.388233+0000","last_active":"2024-09-16T15:42:52.417696+0000","last_peered":"2024-09-16T15:42:52.417696+0000","last_clean":"2024-09-16T15:42:52.417696+0000","last_became_active":"2024-09-16T15:42:50.387920+0000","last_became_peered":"2024-09-16T15:42:50.387920+0000","last_unstale":"2024-09-16T15:42:52.417696+0000","last_undegraded":"2024-09-16T15:42:52.417696+0000","last_fullsized":"2024-09-16T15:42:52.417696+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:42:49.352371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:42:49.352371+0000","last_clean_scrub_stamp":"2024-09-16T15:42:49.352371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T22:48:39.726611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T15:42:52.876523+0000","last_change":"2024-09-16T15:42:51.414425+0000","last_active":"2024-09-16T15:42:52.876523+0000","last_peered":"2024-09-16T15:42:52.876523+0000","last_clean":"2024-09-16T15:42:52.876523+0000","last_became_active":"2024-09-16T15:42:51.410797+0000","last_became_peered":"2024-09-16T15:42:51.410797+0000","last_unstale":"2024-09-16T15:42:52.876523+0000","last_undegraded":"2024-09-16T15:42:52.876523+0000","last_fullsized":"2024-09-16T15:42:52.876523+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_clean_scrub_stamp":"2024-09-16T15:42:50.361965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T18:40:25.285207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"20'2","reported_seq":17,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T15:42:52.725055+0000","last_change":"2024-09-16T15:42:50.423780+0000","last_active":"2024-09-16T15:42:52.725055+0000","last_peered":"2024-09-16T15:42:52.725055+0000","last_clean":"2024-09-16T15:42:52.725055+0000","last_became_active":"2024-09-16T15:42:50.423037+0000","last_became_peered":"2024-09-16T15:42:50.423037+0000","last_unstale":"2024-09-16T15:42:52.725055+0000","last_undegraded":"2024-09-16T15:42:52.725055+0000","last_fullsized":"2024-09-16T15:42:52.725055+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:42:49.352371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:42:49.352371+0000","last_clean_scrub_stamp":"2024-09-16T15:42:49.352371+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T01:04:05.966343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T15:42:52.415974+0000","last_change":"2024-09-16T15:42:51.397509+0000","last_active":"2024-09-16T15:42:52.415974+0000","last_peered":"2024-09-16T15:42:52.415974+0000","last_clean":"2024-09-16T15:42:52.415974+0000","last_became_active":"2024-09-16T15:42:51.396735+0000","last_became_peered":"2024-09-16T15:42:51.396735+0000","last_unstale":"2024-09-16T15:42:52.415974+0000","last_undegraded":"2024-09-16T15:42:52.415974+0000","last_fullsized":"2024-09-16T15:42:52.415974+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_clean_scrub_stamp":"2024-09-16T15:42:50.361965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T02:47:48.200896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T15:42:52.639530+0000","last_change":"2024-09-16T15:42:51.423874+0000","last_active":"2024-09-16T15:42:52.639530+0000","last_peered":"2024-09-16T15:42:52.639530+0000","last_clean":"2024-09-16T15:42:52.639530+0000","last_became_active":"2024-09-16T15:42:51.423455+0000","last_became_peered":"2024-09-16T15:42:51.423455+0000","last_unstale":"2024-09-16T15:42:52.639530+0000","last_undegraded":"2024-09-16T15:42:52.639530+0000","last_fullsized":"2024-09-16T15:42:52.639530+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_clean_scrub_stamp":"2024-09-16T15:42:50.361965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T22:43:57.612493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"20'2","reported_seq":17,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T15:42:52.724179+0000","last_change":"2024-09-16T15:42:50.428990+0000","last_active":"2024-09-16T15:42:52.724179+0000","last_peered":"2024-09-16T15:42:52.724179+0000","last_clean":"2024-09-16T15:42:52.724179+0000","last_became_active":"2024-09-16T15:42:50.428159+0000","last_became_peered":"2024-09-16T15:42:50.428159+0000","last_unstale":"2024-09-16T15:42:52.724179+0000","last_undegraded":"2024-09-16T15:42:52.724179+0000","last_fullsized":"2024-09-16T15:42:52.724179+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:42:49.352371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:42:49.352371+0000","last_clean_scrub_stamp":"2024-09-16T15:42:49.352371+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T01:17:11.293386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T15:42:52.638975+0000","last_change":"2024-09-16T15:42:51.421429+0000","last_active":"2024-09-16T15:42:52.638975+0000","last_peered":"2024-09-16T15:42:52.638975+0000","last_clean":"2024-09-16T15:42:52.638975+0000","last_became_active":"2024-09-16T15:42:51.420671+0000","last_became_peered":"2024-09-16T15:42:51.420671+0000","last_unstale":"2024-09-16T15:42:52.638975+0000","last_undegraded":"2024-09-16T15:42:52.638975+0000","last_fullsized":"2024-09-16T15:42:52.638975+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_clean_scrub_stamp":"2024-09-16T15:42:50.361965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T00:34:44.681110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"20'2","reported_seq":17,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T15:42:52.724740+0000","last_change":"2024-09-16T15:42:50.425037+0000","last_active":"2024-09-16T15:42:52.724740+0000","last_peered":"2024-09-16T15:42:52.724740+0000","last_clean":"2024-09-16T15:42:52.724740+0000","last_became_active":"2024-09-16T15:42:50.423907+0000","last_became_peered":"2024-09-16T15:42:50.423907+0000","last_unstale":"2024-09-16T15:42:52.724740+0000","last_undegraded":"2024-09-16T15:42:52.724740+0000","last_fullsized":"2024-09-16T15:42:52.724740+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:42:49.352371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:42:49.352371+0000","last_clean_scrub_stamp":"2024-09-16T15:42:49.352371+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:56:59.748634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T15:42:52.877982+0000","last_change":"2024-09-16T15:42:51.425681+0000","last_active":"2024-09-16T15:42:52.877982+0000","last_peered":"2024-09-16T15:42:52.877982+0000","last_clean":"2024-09-16T15:42:52.877982+0000","last_became_active":"2024-09-16T15:42:51.425367+0000","last_became_peered":"2024-09-16T15:42:51.425367+0000","last_unstale":"2024-09-16T15:42:52.877982+0000","last_undegraded":"2024-09-16T15:42:52.877982+0000","last_fullsized":"2024-09-16T15:42:52.877982+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_clean_scrub_stamp":"2024-09-16T15:42:50.361965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:21.758409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"20'1","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T15:42:52.875580+0000","last_change":"2024-09-16T15:42:50.392203+0000","last_active":"2024-09-16T15:42:52.875580+0000","last_peered":"2024-09-16T15:42:52.875580+0000","last_clean":"2024-09-16T15:42:52.875580+0000","last_became_active":"2024-09-16T15:42:50.391647+0000","last_became_peered":"2024-09-16T15:42:50.391647+0000","last_unstale":"2024-09-16T15:42:52.875580+0000","last_undegraded":"2024-09-16T15:42:52.875580+0000","last_fullsized":"2024-09-16T15:42:52.875580+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:42:49.352371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:42:49.352371+0000","last_clean_scrub_stamp":"2024-09-16T15:42:49.352371+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T22:35:29.889897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T15:42:52.876061+0000","last_change":"2024-09-16T15:42:51.408021+0000","last_active":"2024-09-16T15:42:52.876061+0000","last_peered":"2024-09-16T15:42:52.876061+0000","last_clean":"2024-09-16T15:42:52.876061+0000","last_became_active":"2024-09-16T15:42:51.407085+0000","last_became_peered":"2024-09-16T15:42:51.407085+0000","last_unstale":"2024-09-16T15:42:52.876061+0000","last_undegraded":"2024-09-16T15:42:52.876061+0000","last_fullsized":"2024-09-16T15:42:52.876061+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_clean_scrub_stamp":"2024-09-16T15:42:50.361965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:38:38.580439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"20'4","reported_seq":19,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T15:42:52.878108+0000","last_change":"2024-09-16T15:42:50.415954+0000","last_active":"2024-09-16T15:42:52.878108+0000","last_peered":"2024-09-16T15:42:52.878108+0000","last_clean":"2024-09-16T15:42:52.878108+0000","last_became_active":"2024-09-16T15:42:50.415619+0000","last_became_peered":"2024-09-16T15:42:50.415619+0000","last_unstale":"2024-09-16T15:42:52.878108+0000","last_undegraded":"2024-09-16T15:42:52.878108+0000","last_fullsized":"2024-09-16T15:42:52.878108+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:42:49.352371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:42:49.352371+0000","last_clean_scrub_stamp":"2024-09-16T15:42:49.352371+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T02:01:17.143570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542150,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27824,"kb_used_data":792,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344016,"statfs":{"total":96636764160,"available":96608272384,"internally_reserved":0,"allocated":811008,"data_stored":665039,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542149,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27256,"kb_used_data":220,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344584,"statfs":{"total":96636764160,"available":96608854016,"internally_reserved":0,"allocated":225280,"data_stored":75701,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":5,"up_from":14,"seq":60129542150,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27240,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344600,"statfs":{"total":96636764160,"available":96608870400,"internally_reserved":0,"allocated":217088,"data_stored":74179,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542148,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27252,"kb_used_data":220,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344588,"statfs":{"total":96636764160,"available":96608858112,"internally_reserved":0,"allocated":225280,"data_stored":74215,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":3,"up_from":14,"seq":60129542149,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27256,"kb_used_data":220,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344584,"statfs":{"total":96636764160,"available":96608854016,"internally_reserved":0,"allocated":225280,"data_stored":75701,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":2,"up_from":14,"seq":60129542149,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27236,"kb_used_data":208,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344604,"statfs":{"total":96636764160,"available":96608874496,"internally_reserved":0,"allocated":212992,"data_stored":74089,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":14,"seq":60129542151,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27244,"kb_used_data":216,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344596,"statfs":{"total":96636764160,"available":96608866304,"internally_reserved":0,"allocated":221184,"data_stored":74249,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":14,"seq":60129542151,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27832,"kb_used_data":800,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344008,"statfs":{"total":96636764160,"available":96608264192,"internally_reserved":0,"allocated":819200,"data_stored":665095,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T15:42:55.439 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T15:42:55.599 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:55.598+0000 7f2018677640 1 -- 172.21.15.123:0/3489476351 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f20100726c0 msgr2=0x7f2010073ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:42:55.599 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:55.598+0000 7f2018677640 1 --2- 172.21.15.123:0/3489476351 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f20100726c0 0x7f2010073ef0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f20100cd650 tx=0x7f200001c740 comp rx=0 tx=0).stop 2024-09-16T15:42:55.599 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:55.598+0000 7f2018677640 1 -- 172.21.15.123:0/3489476351 shutdown_connections 2024-09-16T15:42:55.599 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:55.598+0000 7f2018677640 1 --2- 172.21.15.123:0/3489476351 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f20100726c0 0x7f2010073ef0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:42:55.599 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:55.598+0000 7f2018677640 1 --2- 172.21.15.123:0/3489476351 >> v2:172.21.15.184:3300/0 conn(0x7f201015bca0 0x7f20100d0630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:42:55.599 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:55.598+0000 7f2018677640 1 -- 172.21.15.123:0/3489476351 >> 172.21.15.123:0/3489476351 conn(0x7f20100cfb10 msgr2=0x7f20100cff20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:42:55.599 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:55.598+0000 7f2018677640 1 -- 172.21.15.123:0/3489476351 shutdown_connections 2024-09-16T15:42:55.599 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:55.598+0000 7f2018677640 1 -- 172.21.15.123:0/3489476351 wait complete. 2024-09-16T15:42:55.600 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:55.598+0000 7f2018677640 1 Processor -- start 2024-09-16T15:42:55.600 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:55.598+0000 7f2018677640 1 -- start start 2024-09-16T15:42:55.602 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:55.598+0000 7f2018677640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f20100726c0 0x7f20100b3460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:42:55.602 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:55.598+0000 7f2018677640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2010074430 0x7f2010164e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:42:55.602 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:55.598+0000 7f2018677640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f201015bca0 0x7f2010165340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:42:55.602 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:55.598+0000 7f2018677640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f2010168e30 con 0x7f201015bca0 2024-09-16T15:42:55.602 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:55.598+0000 7f2018677640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f2010168cb0 con 0x7f2010074430 2024-09-16T15:42:55.602 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:55.598+0000 7f2018677640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f2010168fb0 con 0x7f20100726c0 2024-09-16T15:42:55.602 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:55.598+0000 7f2016c14640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f201015bca0 0x7f2010165340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:42:55.602 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:55.598+0000 7f2016c14640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f201015bca0 0x7f2010165340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:40192/0 (socket says 172.21.15.123:40192) 2024-09-16T15:42:55.602 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:55.598+0000 7f2016c14640 1 -- 172.21.15.123:0/906323991 learned_addr learned my addr 172.21.15.123:0/906323991 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:42:55.602 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:55.598+0000 7f2015c12640 1 --2- 172.21.15.123:0/906323991 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2010074430 0x7f2010164e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:42:55.603 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:55.598+0000 7f2016413640 1 --2- 172.21.15.123:0/906323991 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f20100726c0 0x7f20100b3460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:42:55.603 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:55.598+0000 7f2016c14640 1 -- 172.21.15.123:0/906323991 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f20100726c0 msgr2=0x7f20100b3460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:42:55.603 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:55.598+0000 7f2016c14640 1 --2- 172.21.15.123:0/906323991 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f20100726c0 0x7f20100b3460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:42:55.603 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:55.598+0000 7f2016c14640 1 -- 172.21.15.123:0/906323991 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2010074430 msgr2=0x7f2010164e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:42:55.603 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:55.598+0000 7f2016c14640 1 --2- 172.21.15.123:0/906323991 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2010074430 0x7f2010164e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:42:55.603 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:55.598+0000 7f2016c14640 1 -- 172.21.15.123:0/906323991 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f20100b29a0 con 0x7f201015bca0 2024-09-16T15:42:55.603 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:55.598+0000 7f2016413640 1 --2- 172.21.15.123:0/906323991 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f20100726c0 0x7f20100b3460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T15:42:55.603 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:55.598+0000 7f2016c14640 1 --2- 172.21.15.123:0/906323991 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f201015bca0 0x7f2010165340 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f200800c770 tx=0x7f200800cc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:42:55.603 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:55.598+0000 7f2006ffd640 1 -- 172.21.15.123:0/906323991 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2008017070 con 0x7f201015bca0 2024-09-16T15:42:55.603 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:55.598+0000 7f2006ffd640 1 -- 172.21.15.123:0/906323991 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f2008005070 con 0x7f201015bca0 2024-09-16T15:42:55.603 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:55.598+0000 7f2006ffd640 1 -- 172.21.15.123:0/906323991 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20080054c0 con 0x7f201015bca0 2024-09-16T15:42:55.603 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:55.598+0000 7f2018677640 1 -- 172.21.15.123:0/906323991 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f20101f8970 con 0x7f201015bca0 2024-09-16T15:42:55.604 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:55.598+0000 7f2018677640 1 -- 172.21.15.123:0/906323991 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f20100b2c60 con 0x7f201015bca0 2024-09-16T15:42:55.604 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:55.602+0000 7f2018677640 1 -- 172.21.15.123:0/906323991 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2010073ef0 con 0x7f201015bca0 2024-09-16T15:42:55.604 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:55.602+0000 7f2006ffd640 1 -- 172.21.15.123:0/906323991 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 5) ==== 97080+0+0 (secure 0 0 0) 0x7f200801a070 con 0x7f201015bca0 2024-09-16T15:42:55.604 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:55.602+0000 7f2006ffd640 1 --2- 172.21.15.123:0/906323991 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f1ff4072ec0 0x7f1ff4075380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:42:55.605 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:55.602+0000 7f2006ffd640 1 -- 172.21.15.123:0/906323991 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f2008093460 con 0x7f201015bca0 2024-09-16T15:42:55.605 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:55.602+0000 7f2016413640 1 --2- 172.21.15.123:0/906323991 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f1ff4072ec0 0x7f1ff4075380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:42:55.605 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:55.602+0000 7f2016413640 1 --2- 172.21.15.123:0/906323991 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f1ff4072ec0 0x7f1ff4075380 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f200c0058d0 tx=0x7f200c005cc0 comp rx=0 tx=0).ready entity=mgr.4135 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:42:55.608 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:55.606+0000 7f2006ffd640 1 -- 172.21.15.123:0/906323991 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f200805e6b0 con 0x7f201015bca0 2024-09-16T15:42:55.763 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:55.762+0000 7f2018677640 1 -- 172.21.15.123:0/906323991 --> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f201015f630 con 0x7f1ff4072ec0 2024-09-16T15:42:55.774 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:55.774+0000 7f2006ffd640 1 -- 172.21.15.123:0/906323991 <== mgr.4135 v2:172.21.15.184:6832/3114346423 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+251941 (secure 0 0 0) 0x7f201015f630 con 0x7f1ff4072ec0 2024-09-16T15:42:55.776 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-09-16T15:42:55.778 INFO:teuthology.orchestra.run.smithi123.stderr:dumped all 2024-09-16T15:42:55.779 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:55.778+0000 7f2018677640 1 -- 172.21.15.123:0/906323991 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f1ff4072ec0 msgr2=0x7f1ff4075380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:42:55.779 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:55.778+0000 7f2018677640 1 --2- 172.21.15.123:0/906323991 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f1ff4072ec0 0x7f1ff4075380 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f200c0058d0 tx=0x7f200c005cc0 comp rx=0 tx=0).stop 2024-09-16T15:42:55.779 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:55.778+0000 7f2018677640 1 -- 172.21.15.123:0/906323991 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f201015bca0 msgr2=0x7f2010165340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:42:55.779 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:55.778+0000 7f2018677640 1 --2- 172.21.15.123:0/906323991 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f201015bca0 0x7f2010165340 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f200800c770 tx=0x7f200800cc30 comp rx=0 tx=0).stop 2024-09-16T15:42:55.779 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:55.778+0000 7f2018677640 1 -- 172.21.15.123:0/906323991 shutdown_connections 2024-09-16T15:42:55.779 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:55.778+0000 7f2018677640 1 --2- 172.21.15.123:0/906323991 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f1ff4072ec0 0x7f1ff4075380 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:42:55.780 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:55.778+0000 7f2018677640 1 --2- 172.21.15.123:0/906323991 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f201015bca0 0x7f2010165340 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:42:55.780 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:55.778+0000 7f2018677640 1 --2- 172.21.15.123:0/906323991 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2010074430 0x7f2010164e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:42:55.780 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:55.778+0000 7f2018677640 1 --2- 172.21.15.123:0/906323991 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f20100726c0 0x7f20100b3460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:42:55.780 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:55.778+0000 7f2018677640 1 -- 172.21.15.123:0/906323991 >> 172.21.15.123:0/906323991 conn(0x7f20100cfb10 msgr2=0x7f201006eee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:42:55.780 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:55.778+0000 7f2018677640 1 -- 172.21.15.123:0/906323991 shutdown_connections 2024-09-16T15:42:55.780 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:55.778+0000 7f2018677640 1 -- 172.21.15.123:0/906323991 wait complete. 2024-09-16T15:42:55.794 INFO:teuthology.orchestra.run.smithi123.stdout:{"pg_ready":true,"pg_map":{"version":22,"stamp":"2024-09-16T15:42:55.246068+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":146,"num_write_kb":1642,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":111,"ondisk_log_size":111,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":185,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219140,"kb_used_data":2888,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755580,"statfs":{"total":773094113280,"available":772869713920,"internally_reserved":0,"allocated":2957312,"data_stored":1778268,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":38,"apply_latency_ms":38,"commit_latency_ns":38000000,"apply_latency_ns":38000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.883486"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T15:42:52.725091+0000","last_change":"2024-09-16T15:42:51.434234+0000","last_active":"2024-09-16T15:42:52.725091+0000","last_peered":"2024-09-16T15:42:52.725091+0000","last_clean":"2024-09-16T15:42:52.725091+0000","last_became_active":"2024-09-16T15:42:51.433660+0000","last_became_peered":"2024-09-16T15:42:51.433660+0000","last_unstale":"2024-09-16T15:42:52.725091+0000","last_undegraded":"2024-09-16T15:42:52.725091+0000","last_fullsized":"2024-09-16T15:42:52.725091+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_clean_scrub_stamp":"2024-09-16T15:42:50.361965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T02:07:35.605190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T15:42:52.418662+0000","last_change":"2024-09-16T15:42:51.412677+0000","last_active":"2024-09-16T15:42:52.418662+0000","last_peered":"2024-09-16T15:42:52.418662+0000","last_clean":"2024-09-16T15:42:52.418662+0000","last_became_active":"2024-09-16T15:42:51.412366+0000","last_became_peered":"2024-09-16T15:42:51.412366+0000","last_unstale":"2024-09-16T15:42:52.418662+0000","last_undegraded":"2024-09-16T15:42:52.418662+0000","last_fullsized":"2024-09-16T15:42:52.418662+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_clean_scrub_stamp":"2024-09-16T15:42:50.361965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:21.758409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T15:42:52.876476+0000","last_change":"2024-09-16T15:42:51.406454+0000","last_active":"2024-09-16T15:42:52.876476+0000","last_peered":"2024-09-16T15:42:52.876476+0000","last_clean":"2024-09-16T15:42:52.876476+0000","last_became_active":"2024-09-16T15:42:51.405679+0000","last_became_peered":"2024-09-16T15:42:51.405679+0000","last_unstale":"2024-09-16T15:42:52.876476+0000","last_undegraded":"2024-09-16T15:42:52.876476+0000","last_fullsized":"2024-09-16T15:42:52.876476+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_clean_scrub_stamp":"2024-09-16T15:42:50.361965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T02:33:32.417205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T15:42:52.638879+0000","last_change":"2024-09-16T15:42:51.423931+0000","last_active":"2024-09-16T15:42:52.638879+0000","last_peered":"2024-09-16T15:42:52.638879+0000","last_clean":"2024-09-16T15:42:52.638879+0000","last_became_active":"2024-09-16T15:42:51.423561+0000","last_became_peered":"2024-09-16T15:42:51.423561+0000","last_unstale":"2024-09-16T15:42:52.638879+0000","last_undegraded":"2024-09-16T15:42:52.638879+0000","last_fullsized":"2024-09-16T15:42:52.638879+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_clean_scrub_stamp":"2024-09-16T15:42:50.361965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T03:35:03.081638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T15:42:52.724573+0000","last_change":"2024-09-16T15:42:51.425601+0000","last_active":"2024-09-16T15:42:52.724573+0000","last_peered":"2024-09-16T15:42:52.724573+0000","last_clean":"2024-09-16T15:42:52.724573+0000","last_became_active":"2024-09-16T15:42:51.425040+0000","last_became_peered":"2024-09-16T15:42:51.425040+0000","last_unstale":"2024-09-16T15:42:52.724573+0000","last_undegraded":"2024-09-16T15:42:52.724573+0000","last_fullsized":"2024-09-16T15:42:52.724573+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_clean_scrub_stamp":"2024-09-16T15:42:50.361965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T00:05:49.936465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T15:42:52.638668+0000","last_change":"2024-09-16T15:42:51.429218+0000","last_active":"2024-09-16T15:42:52.638668+0000","last_peered":"2024-09-16T15:42:52.638668+0000","last_clean":"2024-09-16T15:42:52.638668+0000","last_became_active":"2024-09-16T15:42:51.428838+0000","last_became_peered":"2024-09-16T15:42:51.428838+0000","last_unstale":"2024-09-16T15:42:52.638668+0000","last_undegraded":"2024-09-16T15:42:52.638668+0000","last_fullsized":"2024-09-16T15:42:52.638668+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_clean_scrub_stamp":"2024-09-16T15:42:50.361965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:21.758409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T15:42:52.723991+0000","last_change":"2024-09-16T15:42:51.424389+0000","last_active":"2024-09-16T15:42:52.723991+0000","last_peered":"2024-09-16T15:42:52.723991+0000","last_clean":"2024-09-16T15:42:52.723991+0000","last_became_active":"2024-09-16T15:42:51.423840+0000","last_became_peered":"2024-09-16T15:42:51.423840+0000","last_unstale":"2024-09-16T15:42:52.723991+0000","last_undegraded":"2024-09-16T15:42:52.723991+0000","last_fullsized":"2024-09-16T15:42:52.723991+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_clean_scrub_stamp":"2024-09-16T15:42:50.361965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T18:05:24.521404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T15:42:52.874446+0000","last_change":"2024-09-16T15:42:51.428815+0000","last_active":"2024-09-16T15:42:52.874446+0000","last_peered":"2024-09-16T15:42:52.874446+0000","last_clean":"2024-09-16T15:42:52.874446+0000","last_became_active":"2024-09-16T15:42:51.427658+0000","last_became_peered":"2024-09-16T15:42:51.427658+0000","last_unstale":"2024-09-16T15:42:52.874446+0000","last_undegraded":"2024-09-16T15:42:52.874446+0000","last_fullsized":"2024-09-16T15:42:52.874446+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_clean_scrub_stamp":"2024-09-16T15:42:50.361965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T03:23:33.232936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T15:42:52.725119+0000","last_change":"2024-09-16T15:42:51.435164+0000","last_active":"2024-09-16T15:42:52.725119+0000","last_peered":"2024-09-16T15:42:52.725119+0000","last_clean":"2024-09-16T15:42:52.725119+0000","last_became_active":"2024-09-16T15:42:51.434702+0000","last_became_peered":"2024-09-16T15:42:51.434702+0000","last_unstale":"2024-09-16T15:42:52.725119+0000","last_undegraded":"2024-09-16T15:42:52.725119+0000","last_fullsized":"2024-09-16T15:42:52.725119+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_clean_scrub_stamp":"2024-09-16T15:42:50.361965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:40:28.779257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T15:42:52.417480+0000","last_change":"2024-09-16T15:42:51.425817+0000","last_active":"2024-09-16T15:42:52.417480+0000","last_peered":"2024-09-16T15:42:52.417480+0000","last_clean":"2024-09-16T15:42:52.417480+0000","last_became_active":"2024-09-16T15:42:51.425279+0000","last_became_peered":"2024-09-16T15:42:51.425279+0000","last_unstale":"2024-09-16T15:42:52.417480+0000","last_undegraded":"2024-09-16T15:42:52.417480+0000","last_fullsized":"2024-09-16T15:42:52.417480+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_clean_scrub_stamp":"2024-09-16T15:42:50.361965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:10:07.925485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T15:42:52.416844+0000","last_change":"2024-09-16T15:42:51.426883+0000","last_active":"2024-09-16T15:42:52.416844+0000","last_peered":"2024-09-16T15:42:52.416844+0000","last_clean":"2024-09-16T15:42:52.416844+0000","last_became_active":"2024-09-16T15:42:51.426556+0000","last_became_peered":"2024-09-16T15:42:51.426556+0000","last_unstale":"2024-09-16T15:42:52.416844+0000","last_undegraded":"2024-09-16T15:42:52.416844+0000","last_fullsized":"2024-09-16T15:42:52.416844+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_clean_scrub_stamp":"2024-09-16T15:42:50.361965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:55:46.525397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T15:42:52.418805+0000","last_change":"2024-09-16T15:42:51.394761+0000","last_active":"2024-09-16T15:42:52.418805+0000","last_peered":"2024-09-16T15:42:52.418805+0000","last_clean":"2024-09-16T15:42:52.418805+0000","last_became_active":"2024-09-16T15:42:51.394005+0000","last_became_peered":"2024-09-16T15:42:51.394005+0000","last_unstale":"2024-09-16T15:42:52.418805+0000","last_undegraded":"2024-09-16T15:42:52.418805+0000","last_fullsized":"2024-09-16T15:42:52.418805+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_clean_scrub_stamp":"2024-09-16T15:42:50.361965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:17:36.442767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T15:42:52.875964+0000","last_change":"2024-09-16T15:42:51.411614+0000","last_active":"2024-09-16T15:42:52.875964+0000","last_peered":"2024-09-16T15:42:52.875964+0000","last_clean":"2024-09-16T15:42:52.875964+0000","last_became_active":"2024-09-16T15:42:51.409702+0000","last_became_peered":"2024-09-16T15:42:51.409702+0000","last_unstale":"2024-09-16T15:42:52.875964+0000","last_undegraded":"2024-09-16T15:42:52.875964+0000","last_fullsized":"2024-09-16T15:42:52.875964+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_clean_scrub_stamp":"2024-09-16T15:42:50.361965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T00:31:04.101900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T15:42:52.874545+0000","last_change":"2024-09-16T15:42:51.403206+0000","last_active":"2024-09-16T15:42:52.874545+0000","last_peered":"2024-09-16T15:42:52.874545+0000","last_clean":"2024-09-16T15:42:52.874545+0000","last_became_active":"2024-09-16T15:42:51.402528+0000","last_became_peered":"2024-09-16T15:42:51.402528+0000","last_unstale":"2024-09-16T15:42:52.874545+0000","last_undegraded":"2024-09-16T15:42:52.874545+0000","last_fullsized":"2024-09-16T15:42:52.874545+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_clean_scrub_stamp":"2024-09-16T15:42:50.361965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:39:05.860718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T15:42:52.418038+0000","last_change":"2024-09-16T15:42:51.411905+0000","last_active":"2024-09-16T15:42:52.418038+0000","last_peered":"2024-09-16T15:42:52.418038+0000","last_clean":"2024-09-16T15:42:52.418038+0000","last_became_active":"2024-09-16T15:42:51.411540+0000","last_became_peered":"2024-09-16T15:42:51.411540+0000","last_unstale":"2024-09-16T15:42:52.418038+0000","last_undegraded":"2024-09-16T15:42:52.418038+0000","last_fullsized":"2024-09-16T15:42:52.418038+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_clean_scrub_stamp":"2024-09-16T15:42:50.361965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:07:17.519138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T15:42:52.638310+0000","last_change":"2024-09-16T15:42:51.419623+0000","last_active":"2024-09-16T15:42:52.638310+0000","last_peered":"2024-09-16T15:42:52.638310+0000","last_clean":"2024-09-16T15:42:52.638310+0000","last_became_active":"2024-09-16T15:42:51.419319+0000","last_became_peered":"2024-09-16T15:42:51.419319+0000","last_unstale":"2024-09-16T15:42:52.638310+0000","last_undegraded":"2024-09-16T15:42:52.638310+0000","last_fullsized":"2024-09-16T15:42:52.638310+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_clean_scrub_stamp":"2024-09-16T15:42:50.361965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T02:33:57.643582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T15:42:52.640012+0000","last_change":"2024-09-16T15:42:51.436411+0000","last_active":"2024-09-16T15:42:52.640012+0000","last_peered":"2024-09-16T15:42:52.640012+0000","last_clean":"2024-09-16T15:42:52.640012+0000","last_became_active":"2024-09-16T15:42:51.435952+0000","last_became_peered":"2024-09-16T15:42:51.435952+0000","last_unstale":"2024-09-16T15:42:52.640012+0000","last_undegraded":"2024-09-16T15:42:52.640012+0000","last_fullsized":"2024-09-16T15:42:52.640012+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_clean_scrub_stamp":"2024-09-16T15:42:50.361965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:08:57.795460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T15:42:52.738414+0000","last_change":"2024-09-16T15:42:51.404567+0000","last_active":"2024-09-16T15:42:52.738414+0000","last_peered":"2024-09-16T15:42:52.738414+0000","last_clean":"2024-09-16T15:42:52.738414+0000","last_became_active":"2024-09-16T15:42:51.403756+0000","last_became_peered":"2024-09-16T15:42:51.403756+0000","last_unstale":"2024-09-16T15:42:52.738414+0000","last_undegraded":"2024-09-16T15:42:52.738414+0000","last_fullsized":"2024-09-16T15:42:52.738414+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_clean_scrub_stamp":"2024-09-16T15:42:50.361965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T01:04:06.975937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T15:42:52.725148+0000","last_change":"2024-09-16T15:42:51.434196+0000","last_active":"2024-09-16T15:42:52.725148+0000","last_peered":"2024-09-16T15:42:52.725148+0000","last_clean":"2024-09-16T15:42:52.725148+0000","last_became_active":"2024-09-16T15:42:51.433748+0000","last_became_peered":"2024-09-16T15:42:51.433748+0000","last_unstale":"2024-09-16T15:42:52.725148+0000","last_undegraded":"2024-09-16T15:42:52.725148+0000","last_fullsized":"2024-09-16T15:42:52.725148+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_clean_scrub_stamp":"2024-09-16T15:42:50.361965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:08:57.795460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T15:42:52.416804+0000","last_change":"2024-09-16T15:42:51.398201+0000","last_active":"2024-09-16T15:42:52.416804+0000","last_peered":"2024-09-16T15:42:52.416804+0000","last_clean":"2024-09-16T15:42:52.416804+0000","last_became_active":"2024-09-16T15:42:51.396092+0000","last_became_peered":"2024-09-16T15:42:51.396092+0000","last_unstale":"2024-09-16T15:42:52.416804+0000","last_undegraded":"2024-09-16T15:42:52.416804+0000","last_fullsized":"2024-09-16T15:42:52.416804+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_clean_scrub_stamp":"2024-09-16T15:42:50.361965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:16:26.836026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T15:42:52.874601+0000","last_change":"2024-09-16T15:42:51.404937+0000","last_active":"2024-09-16T15:42:52.874601+0000","last_peered":"2024-09-16T15:42:52.874601+0000","last_clean":"2024-09-16T15:42:52.874601+0000","last_became_active":"2024-09-16T15:42:51.404498+0000","last_became_peered":"2024-09-16T15:42:51.404498+0000","last_unstale":"2024-09-16T15:42:52.874601+0000","last_undegraded":"2024-09-16T15:42:52.874601+0000","last_fullsized":"2024-09-16T15:42:52.874601+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_clean_scrub_stamp":"2024-09-16T15:42:50.361965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:16:06.557859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T15:42:52.638770+0000","last_change":"2024-09-16T15:42:51.436767+0000","last_active":"2024-09-16T15:42:52.638770+0000","last_peered":"2024-09-16T15:42:52.638770+0000","last_clean":"2024-09-16T15:42:52.638770+0000","last_became_active":"2024-09-16T15:42:51.436076+0000","last_became_peered":"2024-09-16T15:42:51.436076+0000","last_unstale":"2024-09-16T15:42:52.638770+0000","last_undegraded":"2024-09-16T15:42:52.638770+0000","last_fullsized":"2024-09-16T15:42:52.638770+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_clean_scrub_stamp":"2024-09-16T15:42:50.361965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T02:42:34.108192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T15:42:52.876650+0000","last_change":"2024-09-16T15:42:51.407561+0000","last_active":"2024-09-16T15:42:52.876650+0000","last_peered":"2024-09-16T15:42:52.876650+0000","last_clean":"2024-09-16T15:42:52.876650+0000","last_became_active":"2024-09-16T15:42:51.407242+0000","last_became_peered":"2024-09-16T15:42:51.407242+0000","last_unstale":"2024-09-16T15:42:52.876650+0000","last_undegraded":"2024-09-16T15:42:52.876650+0000","last_fullsized":"2024-09-16T15:42:52.876650+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_clean_scrub_stamp":"2024-09-16T15:42:50.361965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:37:26.430063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T15:42:52.638795+0000","last_change":"2024-09-16T15:42:51.420155+0000","last_active":"2024-09-16T15:42:52.638795+0000","last_peered":"2024-09-16T15:42:52.638795+0000","last_clean":"2024-09-16T15:42:52.638795+0000","last_became_active":"2024-09-16T15:42:51.419779+0000","last_became_peered":"2024-09-16T15:42:51.419779+0000","last_unstale":"2024-09-16T15:42:52.638795+0000","last_undegraded":"2024-09-16T15:42:52.638795+0000","last_fullsized":"2024-09-16T15:42:52.638795+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_clean_scrub_stamp":"2024-09-16T15:42:50.361965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:28:43.699424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T15:42:52.875086+0000","last_change":"2024-09-16T15:42:51.412701+0000","last_active":"2024-09-16T15:42:52.875086+0000","last_peered":"2024-09-16T15:42:52.875086+0000","last_clean":"2024-09-16T15:42:52.875086+0000","last_became_active":"2024-09-16T15:42:51.411770+0000","last_became_peered":"2024-09-16T15:42:51.411770+0000","last_unstale":"2024-09-16T15:42:52.875086+0000","last_undegraded":"2024-09-16T15:42:52.875086+0000","last_fullsized":"2024-09-16T15:42:52.875086+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_clean_scrub_stamp":"2024-09-16T15:42:50.361965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T02:37:19.881471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T15:42:52.417630+0000","last_change":"2024-09-16T15:42:51.423741+0000","last_active":"2024-09-16T15:42:52.417630+0000","last_peered":"2024-09-16T15:42:52.417630+0000","last_clean":"2024-09-16T15:42:52.417630+0000","last_became_active":"2024-09-16T15:42:51.423444+0000","last_became_peered":"2024-09-16T15:42:51.423444+0000","last_unstale":"2024-09-16T15:42:52.417630+0000","last_undegraded":"2024-09-16T15:42:52.417630+0000","last_fullsized":"2024-09-16T15:42:52.417630+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_clean_scrub_stamp":"2024-09-16T15:42:50.361965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T18:56:18.021098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T15:42:52.417698+0000","last_change":"2024-09-16T15:42:51.410257+0000","last_active":"2024-09-16T15:42:52.417698+0000","last_peered":"2024-09-16T15:42:52.417698+0000","last_clean":"2024-09-16T15:42:52.417698+0000","last_became_active":"2024-09-16T15:42:51.409838+0000","last_became_peered":"2024-09-16T15:42:51.409838+0000","last_unstale":"2024-09-16T15:42:52.417698+0000","last_undegraded":"2024-09-16T15:42:52.417698+0000","last_fullsized":"2024-09-16T15:42:52.417698+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_clean_scrub_stamp":"2024-09-16T15:42:50.361965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T01:10:38.539716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T15:42:52.875794+0000","last_change":"2024-09-16T15:42:51.414267+0000","last_active":"2024-09-16T15:42:52.875794+0000","last_peered":"2024-09-16T15:42:52.875794+0000","last_clean":"2024-09-16T15:42:52.875794+0000","last_became_active":"2024-09-16T15:42:51.412201+0000","last_became_peered":"2024-09-16T15:42:51.412201+0000","last_unstale":"2024-09-16T15:42:52.875794+0000","last_undegraded":"2024-09-16T15:42:52.875794+0000","last_fullsized":"2024-09-16T15:42:52.875794+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_clean_scrub_stamp":"2024-09-16T15:42:50.361965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:59:06.080508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T15:42:52.875146+0000","last_change":"2024-09-16T15:42:51.412344+0000","last_active":"2024-09-16T15:42:52.875146+0000","last_peered":"2024-09-16T15:42:52.875146+0000","last_clean":"2024-09-16T15:42:52.875146+0000","last_became_active":"2024-09-16T15:42:51.410933+0000","last_became_peered":"2024-09-16T15:42:51.410933+0000","last_unstale":"2024-09-16T15:42:52.875146+0000","last_undegraded":"2024-09-16T15:42:52.875146+0000","last_fullsized":"2024-09-16T15:42:52.875146+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_clean_scrub_stamp":"2024-09-16T15:42:50.361965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T01:38:29.275555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T15:42:52.639286+0000","last_change":"2024-09-16T15:42:51.419645+0000","last_active":"2024-09-16T15:42:52.639286+0000","last_peered":"2024-09-16T15:42:52.639286+0000","last_clean":"2024-09-16T15:42:52.639286+0000","last_became_active":"2024-09-16T15:42:51.419335+0000","last_became_peered":"2024-09-16T15:42:51.419335+0000","last_unstale":"2024-09-16T15:42:52.639286+0000","last_undegraded":"2024-09-16T15:42:52.639286+0000","last_fullsized":"2024-09-16T15:42:52.639286+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_clean_scrub_stamp":"2024-09-16T15:42:50.361965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T02:58:03.717735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T15:42:52.738151+0000","last_change":"2024-09-16T15:42:51.425478+0000","last_active":"2024-09-16T15:42:52.738151+0000","last_peered":"2024-09-16T15:42:52.738151+0000","last_clean":"2024-09-16T15:42:52.738151+0000","last_became_active":"2024-09-16T15:42:51.425150+0000","last_became_peered":"2024-09-16T15:42:51.425150+0000","last_unstale":"2024-09-16T15:42:52.738151+0000","last_undegraded":"2024-09-16T15:42:52.738151+0000","last_fullsized":"2024-09-16T15:42:52.738151+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_clean_scrub_stamp":"2024-09-16T15:42:50.361965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T00:53:53.123181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T15:42:52.416202+0000","last_change":"2024-09-16T15:42:51.395997+0000","last_active":"2024-09-16T15:42:52.416202+0000","last_peered":"2024-09-16T15:42:52.416202+0000","last_clean":"2024-09-16T15:42:52.416202+0000","last_became_active":"2024-09-16T15:42:51.395182+0000","last_became_peered":"2024-09-16T15:42:51.395182+0000","last_unstale":"2024-09-16T15:42:52.416202+0000","last_undegraded":"2024-09-16T15:42:52.416202+0000","last_fullsized":"2024-09-16T15:42:52.416202+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_clean_scrub_stamp":"2024-09-16T15:42:50.361965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:22:21.461659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T15:42:52.418393+0000","last_change":"2024-09-16T15:42:51.398579+0000","last_active":"2024-09-16T15:42:52.418393+0000","last_peered":"2024-09-16T15:42:52.418393+0000","last_clean":"2024-09-16T15:42:52.418393+0000","last_became_active":"2024-09-16T15:42:51.397865+0000","last_became_peered":"2024-09-16T15:42:51.397865+0000","last_unstale":"2024-09-16T15:42:52.418393+0000","last_undegraded":"2024-09-16T15:42:52.418393+0000","last_fullsized":"2024-09-16T15:42:52.418393+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_clean_scrub_stamp":"2024-09-16T15:42:50.361965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:44:41.707611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"20'1","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T15:42:52.876772+0000","last_change":"2024-09-16T15:42:50.407972+0000","last_active":"2024-09-16T15:42:52.876772+0000","last_peered":"2024-09-16T15:42:52.876772+0000","last_clean":"2024-09-16T15:42:52.876772+0000","last_became_active":"2024-09-16T15:42:50.407571+0000","last_became_peered":"2024-09-16T15:42:50.407571+0000","last_unstale":"2024-09-16T15:42:52.876772+0000","last_undegraded":"2024-09-16T15:42:52.876772+0000","last_fullsized":"2024-09-16T15:42:52.876772+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:42:49.352371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:42:49.352371+0000","last_clean_scrub_stamp":"2024-09-16T15:42:49.352371+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T02:32:51.461439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T15:42:52.876489+0000","last_change":"2024-09-16T15:42:51.403656+0000","last_active":"2024-09-16T15:42:52.876489+0000","last_peered":"2024-09-16T15:42:52.876489+0000","last_clean":"2024-09-16T15:42:52.876489+0000","last_became_active":"2024-09-16T15:42:51.403183+0000","last_became_peered":"2024-09-16T15:42:51.403183+0000","last_unstale":"2024-09-16T15:42:52.876489+0000","last_undegraded":"2024-09-16T15:42:52.876489+0000","last_fullsized":"2024-09-16T15:42:52.876489+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_clean_scrub_stamp":"2024-09-16T15:42:50.361965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T01:01:27.144919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"20'1","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T15:42:52.415759+0000","last_change":"2024-09-16T15:42:50.405255+0000","last_active":"2024-09-16T15:42:52.415759+0000","last_peered":"2024-09-16T15:42:52.415759+0000","last_clean":"2024-09-16T15:42:52.415759+0000","last_became_active":"2024-09-16T15:42:50.404469+0000","last_became_peered":"2024-09-16T15:42:50.404469+0000","last_unstale":"2024-09-16T15:42:52.415759+0000","last_undegraded":"2024-09-16T15:42:52.415759+0000","last_fullsized":"2024-09-16T15:42:52.415759+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:42:49.352371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:42:49.352371+0000","last_clean_scrub_stamp":"2024-09-16T15:42:49.352371+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:41:46.870937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T15:42:52.877073+0000","last_change":"2024-09-16T15:42:50.399198+0000","last_active":"2024-09-16T15:42:52.877073+0000","last_peered":"2024-09-16T15:42:52.877073+0000","last_clean":"2024-09-16T15:42:52.877073+0000","last_became_active":"2024-09-16T15:42:50.398892+0000","last_became_peered":"2024-09-16T15:42:50.398892+0000","last_unstale":"2024-09-16T15:42:52.877073+0000","last_undegraded":"2024-09-16T15:42:52.877073+0000","last_fullsized":"2024-09-16T15:42:52.877073+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:42:49.352371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:42:49.352371+0000","last_clean_scrub_stamp":"2024-09-16T15:42:49.352371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T03:32:21.032967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T15:42:52.875003+0000","last_change":"2024-09-16T15:42:51.412892+0000","last_active":"2024-09-16T15:42:52.875003+0000","last_peered":"2024-09-16T15:42:52.875003+0000","last_clean":"2024-09-16T15:42:52.875003+0000","last_became_active":"2024-09-16T15:42:51.407080+0000","last_became_peered":"2024-09-16T15:42:51.407080+0000","last_unstale":"2024-09-16T15:42:52.875003+0000","last_undegraded":"2024-09-16T15:42:52.875003+0000","last_fullsized":"2024-09-16T15:42:52.875003+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_clean_scrub_stamp":"2024-09-16T15:42:50.361965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T02:17:29.317460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T15:42:52.638690+0000","last_change":"2024-09-16T15:42:50.427196+0000","last_active":"2024-09-16T15:42:52.638690+0000","last_peered":"2024-09-16T15:42:52.638690+0000","last_clean":"2024-09-16T15:42:52.638690+0000","last_became_active":"2024-09-16T15:42:50.426397+0000","last_became_peered":"2024-09-16T15:42:50.426397+0000","last_unstale":"2024-09-16T15:42:52.638690+0000","last_undegraded":"2024-09-16T15:42:52.638690+0000","last_fullsized":"2024-09-16T15:42:52.638690+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:42:49.352371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:42:49.352371+0000","last_clean_scrub_stamp":"2024-09-16T15:42:49.352371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T22:48:39.726611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T15:42:52.417585+0000","last_change":"2024-09-16T15:42:51.410683+0000","last_active":"2024-09-16T15:42:52.417585+0000","last_peered":"2024-09-16T15:42:52.417585+0000","last_clean":"2024-09-16T15:42:52.417585+0000","last_became_active":"2024-09-16T15:42:51.409695+0000","last_became_peered":"2024-09-16T15:42:51.409695+0000","last_unstale":"2024-09-16T15:42:52.417585+0000","last_undegraded":"2024-09-16T15:42:52.417585+0000","last_fullsized":"2024-09-16T15:42:52.417585+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_clean_scrub_stamp":"2024-09-16T15:42:50.361965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T01:38:29.275555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T15:42:52.417220+0000","last_change":"2024-09-16T15:42:50.420642+0000","last_active":"2024-09-16T15:42:52.417220+0000","last_peered":"2024-09-16T15:42:52.417220+0000","last_clean":"2024-09-16T15:42:52.417220+0000","last_became_active":"2024-09-16T15:42:50.419895+0000","last_became_peered":"2024-09-16T15:42:50.419895+0000","last_unstale":"2024-09-16T15:42:52.417220+0000","last_undegraded":"2024-09-16T15:42:52.417220+0000","last_fullsized":"2024-09-16T15:42:52.417220+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:42:49.352371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:42:49.352371+0000","last_clean_scrub_stamp":"2024-09-16T15:42:49.352371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:40:01.292941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T15:42:52.639747+0000","last_change":"2024-09-16T15:42:51.434841+0000","last_active":"2024-09-16T15:42:52.639747+0000","last_peered":"2024-09-16T15:42:52.639747+0000","last_clean":"2024-09-16T15:42:52.639747+0000","last_became_active":"2024-09-16T15:42:51.434532+0000","last_became_peered":"2024-09-16T15:42:51.434532+0000","last_unstale":"2024-09-16T15:42:52.639747+0000","last_undegraded":"2024-09-16T15:42:52.639747+0000","last_fullsized":"2024-09-16T15:42:52.639747+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_clean_scrub_stamp":"2024-09-16T15:42:50.361965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T18:44:49.865687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T15:42:52.639491+0000","last_change":"2024-09-16T15:42:50.419196+0000","last_active":"2024-09-16T15:42:52.639491+0000","last_peered":"2024-09-16T15:42:52.639491+0000","last_clean":"2024-09-16T15:42:52.639491+0000","last_became_active":"2024-09-16T15:42:50.415873+0000","last_became_peered":"2024-09-16T15:42:50.415873+0000","last_unstale":"2024-09-16T15:42:52.639491+0000","last_undegraded":"2024-09-16T15:42:52.639491+0000","last_fullsized":"2024-09-16T15:42:52.639491+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:42:49.352371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:42:49.352371+0000","last_clean_scrub_stamp":"2024-09-16T15:42:49.352371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:26:07.798125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T15:42:52.877644+0000","last_change":"2024-09-16T15:42:51.425682+0000","last_active":"2024-09-16T15:42:52.877644+0000","last_peered":"2024-09-16T15:42:52.877644+0000","last_clean":"2024-09-16T15:42:52.877644+0000","last_became_active":"2024-09-16T15:42:51.425336+0000","last_became_peered":"2024-09-16T15:42:51.425336+0000","last_unstale":"2024-09-16T15:42:52.877644+0000","last_undegraded":"2024-09-16T15:42:52.877644+0000","last_fullsized":"2024-09-16T15:42:52.877644+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_clean_scrub_stamp":"2024-09-16T15:42:50.361965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T02:33:57.643582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"20'3","reported_seq":18,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T15:42:52.418373+0000","last_change":"2024-09-16T15:42:50.415090+0000","last_active":"2024-09-16T15:42:52.418373+0000","last_peered":"2024-09-16T15:42:52.418373+0000","last_clean":"2024-09-16T15:42:52.418373+0000","last_became_active":"2024-09-16T15:42:50.414818+0000","last_became_peered":"2024-09-16T15:42:50.414818+0000","last_unstale":"2024-09-16T15:42:52.418373+0000","last_undegraded":"2024-09-16T15:42:52.418373+0000","last_fullsized":"2024-09-16T15:42:52.418373+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:42:49.352371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:42:49.352371+0000","last_clean_scrub_stamp":"2024-09-16T15:42:49.352371+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:36:36.506348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T15:42:52.419154+0000","last_change":"2024-09-16T15:42:51.425357+0000","last_active":"2024-09-16T15:42:52.419154+0000","last_peered":"2024-09-16T15:42:52.419154+0000","last_clean":"2024-09-16T15:42:52.419154+0000","last_became_active":"2024-09-16T15:42:51.425074+0000","last_became_peered":"2024-09-16T15:42:51.425074+0000","last_unstale":"2024-09-16T15:42:52.419154+0000","last_undegraded":"2024-09-16T15:42:52.419154+0000","last_fullsized":"2024-09-16T15:42:52.419154+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_clean_scrub_stamp":"2024-09-16T15:42:50.361965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:59:03.380666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"20'6","reported_seq":21,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T15:42:52.416548+0000","last_change":"2024-09-16T15:42:50.396749+0000","last_active":"2024-09-16T15:42:52.416548+0000","last_peered":"2024-09-16T15:42:52.416548+0000","last_clean":"2024-09-16T15:42:52.416548+0000","last_became_active":"2024-09-16T15:42:50.395963+0000","last_became_peered":"2024-09-16T15:42:50.395963+0000","last_unstale":"2024-09-16T15:42:52.416548+0000","last_undegraded":"2024-09-16T15:42:52.416548+0000","last_fullsized":"2024-09-16T15:42:52.416548+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:42:49.352371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:42:49.352371+0000","last_clean_scrub_stamp":"2024-09-16T15:42:49.352371+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T23:01:15.996297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T15:42:52.724385+0000","last_change":"2024-09-16T15:42:51.427006+0000","last_active":"2024-09-16T15:42:52.724385+0000","last_peered":"2024-09-16T15:42:52.724385+0000","last_clean":"2024-09-16T15:42:52.724385+0000","last_became_active":"2024-09-16T15:42:51.426730+0000","last_became_peered":"2024-09-16T15:42:51.426730+0000","last_unstale":"2024-09-16T15:42:52.724385+0000","last_undegraded":"2024-09-16T15:42:52.724385+0000","last_fullsized":"2024-09-16T15:42:52.724385+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_clean_scrub_stamp":"2024-09-16T15:42:50.361965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:49:13.477881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"20'2","reported_seq":17,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T15:42:52.876798+0000","last_change":"2024-09-16T15:42:50.401803+0000","last_active":"2024-09-16T15:42:52.876798+0000","last_peered":"2024-09-16T15:42:52.876798+0000","last_clean":"2024-09-16T15:42:52.876798+0000","last_became_active":"2024-09-16T15:42:50.401495+0000","last_became_peered":"2024-09-16T15:42:50.401495+0000","last_unstale":"2024-09-16T15:42:52.876798+0000","last_undegraded":"2024-09-16T15:42:52.876798+0000","last_fullsized":"2024-09-16T15:42:52.876798+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:42:49.352371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:42:49.352371+0000","last_clean_scrub_stamp":"2024-09-16T15:42:49.352371+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T18:06:23.054055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T15:42:52.875115+0000","last_change":"2024-09-16T15:42:51.428923+0000","last_active":"2024-09-16T15:42:52.875115+0000","last_peered":"2024-09-16T15:42:52.875115+0000","last_clean":"2024-09-16T15:42:52.875115+0000","last_became_active":"2024-09-16T15:42:51.428550+0000","last_became_peered":"2024-09-16T15:42:51.428550+0000","last_unstale":"2024-09-16T15:42:52.875115+0000","last_undegraded":"2024-09-16T15:42:52.875115+0000","last_fullsized":"2024-09-16T15:42:52.875115+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_clean_scrub_stamp":"2024-09-16T15:42:50.361965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:57:03.171257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T15:42:52.639097+0000","last_change":"2024-09-16T15:42:50.426919+0000","last_active":"2024-09-16T15:42:52.639097+0000","last_peered":"2024-09-16T15:42:52.639097+0000","last_clean":"2024-09-16T15:42:52.639097+0000","last_became_active":"2024-09-16T15:42:50.426141+0000","last_became_peered":"2024-09-16T15:42:50.426141+0000","last_unstale":"2024-09-16T15:42:52.639097+0000","last_undegraded":"2024-09-16T15:42:52.639097+0000","last_fullsized":"2024-09-16T15:42:52.639097+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:42:49.352371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:42:49.352371+0000","last_clean_scrub_stamp":"2024-09-16T15:42:49.352371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:23:09.046920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T15:42:52.724923+0000","last_change":"2024-09-16T15:42:51.425867+0000","last_active":"2024-09-16T15:42:52.724923+0000","last_peered":"2024-09-16T15:42:52.724923+0000","last_clean":"2024-09-16T15:42:52.724923+0000","last_became_active":"2024-09-16T15:42:51.425313+0000","last_became_peered":"2024-09-16T15:42:51.425313+0000","last_unstale":"2024-09-16T15:42:52.724923+0000","last_undegraded":"2024-09-16T15:42:52.724923+0000","last_fullsized":"2024-09-16T15:42:52.724923+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_clean_scrub_stamp":"2024-09-16T15:42:50.361965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T18:02:05.435198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T15:42:52.415548+0000","last_change":"2024-09-16T15:42:50.405263+0000","last_active":"2024-09-16T15:42:52.415548+0000","last_peered":"2024-09-16T15:42:52.415548+0000","last_clean":"2024-09-16T15:42:52.415548+0000","last_became_active":"2024-09-16T15:42:50.404419+0000","last_became_peered":"2024-09-16T15:42:50.404419+0000","last_unstale":"2024-09-16T15:42:52.415548+0000","last_undegraded":"2024-09-16T15:42:52.415548+0000","last_fullsized":"2024-09-16T15:42:52.415548+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:42:49.352371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:42:49.352371+0000","last_clean_scrub_stamp":"2024-09-16T15:42:49.352371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:40:33.707149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T15:42:52.415669+0000","last_change":"2024-09-16T15:42:51.397798+0000","last_active":"2024-09-16T15:42:52.415669+0000","last_peered":"2024-09-16T15:42:52.415669+0000","last_clean":"2024-09-16T15:42:52.415669+0000","last_became_active":"2024-09-16T15:42:51.397070+0000","last_became_peered":"2024-09-16T15:42:51.397070+0000","last_unstale":"2024-09-16T15:42:52.415669+0000","last_undegraded":"2024-09-16T15:42:52.415669+0000","last_fullsized":"2024-09-16T15:42:52.415669+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_clean_scrub_stamp":"2024-09-16T15:42:50.361965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:24:48.779934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"20'8","reported_seq":23,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T15:42:52.737837+0000","last_change":"2024-09-16T15:42:50.386440+0000","last_active":"2024-09-16T15:42:52.737837+0000","last_peered":"2024-09-16T15:42:52.737837+0000","last_clean":"2024-09-16T15:42:52.737837+0000","last_became_active":"2024-09-16T15:42:50.386100+0000","last_became_peered":"2024-09-16T15:42:50.386100+0000","last_unstale":"2024-09-16T15:42:52.737837+0000","last_undegraded":"2024-09-16T15:42:52.737837+0000","last_fullsized":"2024-09-16T15:42:52.737837+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:42:49.352371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:42:49.352371+0000","last_clean_scrub_stamp":"2024-09-16T15:42:49.352371+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:32:51.019126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T15:42:52.723952+0000","last_change":"2024-09-16T15:42:51.424679+0000","last_active":"2024-09-16T15:42:52.723952+0000","last_peered":"2024-09-16T15:42:52.723952+0000","last_clean":"2024-09-16T15:42:52.723952+0000","last_became_active":"2024-09-16T15:42:51.424119+0000","last_became_peered":"2024-09-16T15:42:51.424119+0000","last_unstale":"2024-09-16T15:42:52.723952+0000","last_undegraded":"2024-09-16T15:42:52.723952+0000","last_fullsized":"2024-09-16T15:42:52.723952+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_clean_scrub_stamp":"2024-09-16T15:42:50.361965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T23:53:36.869388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"20'1","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T15:42:52.419009+0000","last_change":"2024-09-16T15:42:50.395317+0000","last_active":"2024-09-16T15:42:52.419009+0000","last_peered":"2024-09-16T15:42:52.419009+0000","last_clean":"2024-09-16T15:42:52.419009+0000","last_became_active":"2024-09-16T15:42:50.394495+0000","last_became_peered":"2024-09-16T15:42:50.394495+0000","last_unstale":"2024-09-16T15:42:52.419009+0000","last_undegraded":"2024-09-16T15:42:52.419009+0000","last_fullsized":"2024-09-16T15:42:52.419009+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:42:49.352371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:42:49.352371+0000","last_clean_scrub_stamp":"2024-09-16T15:42:49.352371+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:45:54.681341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T15:42:52.724959+0000","last_change":"2024-09-16T15:42:51.423467+0000","last_active":"2024-09-16T15:42:52.724959+0000","last_peered":"2024-09-16T15:42:52.724959+0000","last_clean":"2024-09-16T15:42:52.724959+0000","last_became_active":"2024-09-16T15:42:51.423161+0000","last_became_peered":"2024-09-16T15:42:51.423161+0000","last_unstale":"2024-09-16T15:42:52.724959+0000","last_undegraded":"2024-09-16T15:42:52.724959+0000","last_fullsized":"2024-09-16T15:42:52.724959+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_clean_scrub_stamp":"2024-09-16T15:42:50.361965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T23:53:22.795898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"20'2","reported_seq":17,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T15:42:52.738233+0000","last_change":"2024-09-16T15:42:50.384867+0000","last_active":"2024-09-16T15:42:52.738233+0000","last_peered":"2024-09-16T15:42:52.738233+0000","last_clean":"2024-09-16T15:42:52.738233+0000","last_became_active":"2024-09-16T15:42:50.384581+0000","last_became_peered":"2024-09-16T15:42:50.384581+0000","last_unstale":"2024-09-16T15:42:52.738233+0000","last_undegraded":"2024-09-16T15:42:52.738233+0000","last_fullsized":"2024-09-16T15:42:52.738233+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:42:49.352371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:42:49.352371+0000","last_clean_scrub_stamp":"2024-09-16T15:42:49.352371+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T22:48:39.726611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T15:42:52.725417+0000","last_change":"2024-09-16T15:42:51.434874+0000","last_active":"2024-09-16T15:42:52.725417+0000","last_peered":"2024-09-16T15:42:52.725417+0000","last_clean":"2024-09-16T15:42:52.725417+0000","last_became_active":"2024-09-16T15:42:51.434561+0000","last_became_peered":"2024-09-16T15:42:51.434561+0000","last_unstale":"2024-09-16T15:42:52.725417+0000","last_undegraded":"2024-09-16T15:42:52.725417+0000","last_fullsized":"2024-09-16T15:42:52.725417+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_clean_scrub_stamp":"2024-09-16T15:42:50.361965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:13:10.293897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"20'1","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T15:42:52.418305+0000","last_change":"2024-09-16T15:42:50.399974+0000","last_active":"2024-09-16T15:42:52.418305+0000","last_peered":"2024-09-16T15:42:52.418305+0000","last_clean":"2024-09-16T15:42:52.418305+0000","last_became_active":"2024-09-16T15:42:50.399714+0000","last_became_peered":"2024-09-16T15:42:50.399714+0000","last_unstale":"2024-09-16T15:42:52.418305+0000","last_undegraded":"2024-09-16T15:42:52.418305+0000","last_fullsized":"2024-09-16T15:42:52.418305+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:42:49.352371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:42:49.352371+0000","last_clean_scrub_stamp":"2024-09-16T15:42:49.352371+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T02:37:24.494137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T15:42:52.639516+0000","last_change":"2024-09-16T15:42:51.420665+0000","last_active":"2024-09-16T15:42:52.639516+0000","last_peered":"2024-09-16T15:42:52.639516+0000","last_clean":"2024-09-16T15:42:52.639516+0000","last_became_active":"2024-09-16T15:42:51.420340+0000","last_became_peered":"2024-09-16T15:42:51.420340+0000","last_unstale":"2024-09-16T15:42:52.639516+0000","last_undegraded":"2024-09-16T15:42:52.639516+0000","last_fullsized":"2024-09-16T15:42:52.639516+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_clean_scrub_stamp":"2024-09-16T15:42:50.361965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T00:05:49.936465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T15:42:52.738027+0000","last_change":"2024-09-16T15:42:50.412850+0000","last_active":"2024-09-16T15:42:52.738027+0000","last_peered":"2024-09-16T15:42:52.738027+0000","last_clean":"2024-09-16T15:42:52.738027+0000","last_became_active":"2024-09-16T15:42:50.412492+0000","last_became_peered":"2024-09-16T15:42:50.412492+0000","last_unstale":"2024-09-16T15:42:52.738027+0000","last_undegraded":"2024-09-16T15:42:52.738027+0000","last_fullsized":"2024-09-16T15:42:52.738027+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:42:49.352371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:42:49.352371+0000","last_clean_scrub_stamp":"2024-09-16T15:42:49.352371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T22:41:47.420539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T15:42:52.724515+0000","last_change":"2024-09-16T15:42:51.428052+0000","last_active":"2024-09-16T15:42:52.724515+0000","last_peered":"2024-09-16T15:42:52.724515+0000","last_clean":"2024-09-16T15:42:52.724515+0000","last_became_active":"2024-09-16T15:42:51.427776+0000","last_became_peered":"2024-09-16T15:42:51.427776+0000","last_unstale":"2024-09-16T15:42:52.724515+0000","last_undegraded":"2024-09-16T15:42:52.724515+0000","last_fullsized":"2024-09-16T15:42:52.724515+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_clean_scrub_stamp":"2024-09-16T15:42:50.361965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T22:34:25.305861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"16'72","reported_seq":111,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T15:42:52.723702+0000","last_change":"2024-09-16T15:42:40.881820+0000","last_active":"2024-09-16T15:42:52.723702+0000","last_peered":"2024-09-16T15:42:52.723702+0000","last_clean":"2024-09-16T15:42:52.723702+0000","last_became_active":"2024-09-16T15:42:40.879965+0000","last_became_peered":"2024-09-16T15:42:40.879965+0000","last_unstale":"2024-09-16T15:42:52.723702+0000","last_undegraded":"2024-09-16T15:42:52.723702+0000","last_fullsized":"2024-09-16T15:42:52.723702+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:42:39.728762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:42:39.728762+0000","last_clean_scrub_stamp":"2024-09-16T15:42:39.728762+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:22:18.768932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T15:42:52.638416+0000","last_change":"2024-09-16T15:42:50.413082+0000","last_active":"2024-09-16T15:42:52.638416+0000","last_peered":"2024-09-16T15:42:52.638416+0000","last_clean":"2024-09-16T15:42:52.638416+0000","last_became_active":"2024-09-16T15:42:50.412050+0000","last_became_peered":"2024-09-16T15:42:50.412050+0000","last_unstale":"2024-09-16T15:42:52.638416+0000","last_undegraded":"2024-09-16T15:42:52.638416+0000","last_fullsized":"2024-09-16T15:42:52.638416+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:42:49.352371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:42:49.352371+0000","last_clean_scrub_stamp":"2024-09-16T15:42:49.352371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:32:51.019126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T15:42:52.875839+0000","last_change":"2024-09-16T15:42:51.430477+0000","last_active":"2024-09-16T15:42:52.875839+0000","last_peered":"2024-09-16T15:42:52.875839+0000","last_clean":"2024-09-16T15:42:52.875839+0000","last_became_active":"2024-09-16T15:42:51.430071+0000","last_became_peered":"2024-09-16T15:42:51.430071+0000","last_unstale":"2024-09-16T15:42:52.875839+0000","last_undegraded":"2024-09-16T15:42:52.875839+0000","last_fullsized":"2024-09-16T15:42:52.875839+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_clean_scrub_stamp":"2024-09-16T15:42:50.361965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T18:56:18.021098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T15:42:52.723623+0000","last_change":"2024-09-16T15:42:50.419981+0000","last_active":"2024-09-16T15:42:52.723623+0000","last_peered":"2024-09-16T15:42:52.723623+0000","last_clean":"2024-09-16T15:42:52.723623+0000","last_became_active":"2024-09-16T15:42:50.418405+0000","last_became_peered":"2024-09-16T15:42:50.418405+0000","last_unstale":"2024-09-16T15:42:52.723623+0000","last_undegraded":"2024-09-16T15:42:52.723623+0000","last_fullsized":"2024-09-16T15:42:52.723623+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:42:49.352371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:42:49.352371+0000","last_clean_scrub_stamp":"2024-09-16T15:42:49.352371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T00:53:52.113587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T15:42:52.874487+0000","last_change":"2024-09-16T15:42:51.403404+0000","last_active":"2024-09-16T15:42:52.874487+0000","last_peered":"2024-09-16T15:42:52.874487+0000","last_clean":"2024-09-16T15:42:52.874487+0000","last_became_active":"2024-09-16T15:42:51.402978+0000","last_became_peered":"2024-09-16T15:42:51.402978+0000","last_unstale":"2024-09-16T15:42:52.874487+0000","last_undegraded":"2024-09-16T15:42:52.874487+0000","last_fullsized":"2024-09-16T15:42:52.874487+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_clean_scrub_stamp":"2024-09-16T15:42:50.361965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T00:04:54.212949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T15:42:52.738033+0000","last_change":"2024-09-16T15:42:50.391465+0000","last_active":"2024-09-16T15:42:52.738033+0000","last_peered":"2024-09-16T15:42:52.738033+0000","last_clean":"2024-09-16T15:42:52.738033+0000","last_became_active":"2024-09-16T15:42:50.391091+0000","last_became_peered":"2024-09-16T15:42:50.391091+0000","last_unstale":"2024-09-16T15:42:52.738033+0000","last_undegraded":"2024-09-16T15:42:52.738033+0000","last_fullsized":"2024-09-16T15:42:52.738033+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:42:49.352371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:42:49.352371+0000","last_clean_scrub_stamp":"2024-09-16T15:42:49.352371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:26:07.798125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T15:42:52.417576+0000","last_change":"2024-09-16T15:42:51.404294+0000","last_active":"2024-09-16T15:42:52.417576+0000","last_peered":"2024-09-16T15:42:52.417576+0000","last_clean":"2024-09-16T15:42:52.417576+0000","last_became_active":"2024-09-16T15:42:51.403997+0000","last_became_peered":"2024-09-16T15:42:51.403997+0000","last_unstale":"2024-09-16T15:42:52.417576+0000","last_undegraded":"2024-09-16T15:42:52.417576+0000","last_fullsized":"2024-09-16T15:42:52.417576+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_clean_scrub_stamp":"2024-09-16T15:42:50.361965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:20:36.872940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T15:42:52.639350+0000","last_change":"2024-09-16T15:42:50.422498+0000","last_active":"2024-09-16T15:42:52.639350+0000","last_peered":"2024-09-16T15:42:52.639350+0000","last_clean":"2024-09-16T15:42:52.639350+0000","last_became_active":"2024-09-16T15:42:50.421727+0000","last_became_peered":"2024-09-16T15:42:50.421727+0000","last_unstale":"2024-09-16T15:42:52.639350+0000","last_undegraded":"2024-09-16T15:42:52.639350+0000","last_fullsized":"2024-09-16T15:42:52.639350+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:42:49.352371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:42:49.352371+0000","last_clean_scrub_stamp":"2024-09-16T15:42:49.352371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:36:36.506348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T15:42:52.419302+0000","last_change":"2024-09-16T15:42:51.413177+0000","last_active":"2024-09-16T15:42:52.419302+0000","last_peered":"2024-09-16T15:42:52.419302+0000","last_clean":"2024-09-16T15:42:52.419302+0000","last_became_active":"2024-09-16T15:42:51.412881+0000","last_became_peered":"2024-09-16T15:42:51.412881+0000","last_unstale":"2024-09-16T15:42:52.419302+0000","last_undegraded":"2024-09-16T15:42:52.419302+0000","last_fullsized":"2024-09-16T15:42:52.419302+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_clean_scrub_stamp":"2024-09-16T15:42:50.361965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:10:16.623518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T15:42:52.418085+0000","last_change":"2024-09-16T15:42:50.394813+0000","last_active":"2024-09-16T15:42:52.418085+0000","last_peered":"2024-09-16T15:42:52.418085+0000","last_clean":"2024-09-16T15:42:52.418085+0000","last_became_active":"2024-09-16T15:42:50.389679+0000","last_became_peered":"2024-09-16T15:42:50.389679+0000","last_unstale":"2024-09-16T15:42:52.418085+0000","last_undegraded":"2024-09-16T15:42:52.418085+0000","last_fullsized":"2024-09-16T15:42:52.418085+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:42:49.352371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:42:49.352371+0000","last_clean_scrub_stamp":"2024-09-16T15:42:49.352371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:23:09.046920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T15:42:52.639557+0000","last_change":"2024-09-16T15:42:51.424119+0000","last_active":"2024-09-16T15:42:52.639557+0000","last_peered":"2024-09-16T15:42:52.639557+0000","last_clean":"2024-09-16T15:42:52.639557+0000","last_became_active":"2024-09-16T15:42:51.423771+0000","last_became_peered":"2024-09-16T15:42:51.423771+0000","last_unstale":"2024-09-16T15:42:52.639557+0000","last_undegraded":"2024-09-16T15:42:52.639557+0000","last_fullsized":"2024-09-16T15:42:52.639557+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_clean_scrub_stamp":"2024-09-16T15:42:50.361965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T23:53:22.795898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T15:42:52.724991+0000","last_change":"2024-09-16T15:42:50.419065+0000","last_active":"2024-09-16T15:42:52.724991+0000","last_peered":"2024-09-16T15:42:52.724991+0000","last_clean":"2024-09-16T15:42:52.724991+0000","last_became_active":"2024-09-16T15:42:50.418438+0000","last_became_peered":"2024-09-16T15:42:50.418438+0000","last_unstale":"2024-09-16T15:42:52.724991+0000","last_undegraded":"2024-09-16T15:42:52.724991+0000","last_fullsized":"2024-09-16T15:42:52.724991+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:42:49.352371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:42:49.352371+0000","last_clean_scrub_stamp":"2024-09-16T15:42:49.352371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T01:42:29.977143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T15:42:52.418120+0000","last_change":"2024-09-16T15:42:51.407174+0000","last_active":"2024-09-16T15:42:52.418120+0000","last_peered":"2024-09-16T15:42:52.418120+0000","last_clean":"2024-09-16T15:42:52.418120+0000","last_became_active":"2024-09-16T15:42:51.406880+0000","last_became_peered":"2024-09-16T15:42:51.406880+0000","last_unstale":"2024-09-16T15:42:52.418120+0000","last_undegraded":"2024-09-16T15:42:52.418120+0000","last_fullsized":"2024-09-16T15:42:52.418120+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_clean_scrub_stamp":"2024-09-16T15:42:50.361965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T02:33:57.643582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"20'1","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T15:42:52.418471+0000","last_change":"2024-09-16T15:42:50.397599+0000","last_active":"2024-09-16T15:42:52.418471+0000","last_peered":"2024-09-16T15:42:52.418471+0000","last_clean":"2024-09-16T15:42:52.418471+0000","last_became_active":"2024-09-16T15:42:50.397301+0000","last_became_peered":"2024-09-16T15:42:50.397301+0000","last_unstale":"2024-09-16T15:42:52.418471+0000","last_undegraded":"2024-09-16T15:42:52.418471+0000","last_fullsized":"2024-09-16T15:42:52.418471+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:42:49.352371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:42:49.352371+0000","last_clean_scrub_stamp":"2024-09-16T15:42:49.352371+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T01:49:08.734586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T15:42:52.875834+0000","last_change":"2024-09-16T15:42:51.424110+0000","last_active":"2024-09-16T15:42:52.875834+0000","last_peered":"2024-09-16T15:42:52.875834+0000","last_clean":"2024-09-16T15:42:52.875834+0000","last_became_active":"2024-09-16T15:42:51.423798+0000","last_became_peered":"2024-09-16T15:42:51.423798+0000","last_unstale":"2024-09-16T15:42:52.875834+0000","last_undegraded":"2024-09-16T15:42:52.875834+0000","last_fullsized":"2024-09-16T15:42:52.875834+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_clean_scrub_stamp":"2024-09-16T15:42:50.361965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T22:53:48.581346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"20'2","reported_seq":17,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T15:42:52.417268+0000","last_change":"2024-09-16T15:42:50.416032+0000","last_active":"2024-09-16T15:42:52.417268+0000","last_peered":"2024-09-16T15:42:52.417268+0000","last_clean":"2024-09-16T15:42:52.417268+0000","last_became_active":"2024-09-16T15:42:50.415325+0000","last_became_peered":"2024-09-16T15:42:50.415325+0000","last_unstale":"2024-09-16T15:42:52.417268+0000","last_undegraded":"2024-09-16T15:42:52.417268+0000","last_fullsized":"2024-09-16T15:42:52.417268+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:42:49.352371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:42:49.352371+0000","last_clean_scrub_stamp":"2024-09-16T15:42:49.352371+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T02:11:26.332763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T15:42:52.877501+0000","last_change":"2024-09-16T15:42:51.407137+0000","last_active":"2024-09-16T15:42:52.877501+0000","last_peered":"2024-09-16T15:42:52.877501+0000","last_clean":"2024-09-16T15:42:52.877501+0000","last_became_active":"2024-09-16T15:42:51.406766+0000","last_became_peered":"2024-09-16T15:42:51.406766+0000","last_unstale":"2024-09-16T15:42:52.877501+0000","last_undegraded":"2024-09-16T15:42:52.877501+0000","last_fullsized":"2024-09-16T15:42:52.877501+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_clean_scrub_stamp":"2024-09-16T15:42:50.361965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:07:17.519138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T15:42:52.723539+0000","last_change":"2024-09-16T15:42:50.419106+0000","last_active":"2024-09-16T15:42:52.723539+0000","last_peered":"2024-09-16T15:42:52.723539+0000","last_clean":"2024-09-16T15:42:52.723539+0000","last_became_active":"2024-09-16T15:42:50.417462+0000","last_became_peered":"2024-09-16T15:42:50.417462+0000","last_unstale":"2024-09-16T15:42:52.723539+0000","last_undegraded":"2024-09-16T15:42:52.723539+0000","last_fullsized":"2024-09-16T15:42:52.723539+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:42:49.352371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:42:49.352371+0000","last_clean_scrub_stamp":"2024-09-16T15:42:49.352371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T22:58:24.474495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T15:42:52.418379+0000","last_change":"2024-09-16T15:42:51.412043+0000","last_active":"2024-09-16T15:42:52.418379+0000","last_peered":"2024-09-16T15:42:52.418379+0000","last_clean":"2024-09-16T15:42:52.418379+0000","last_became_active":"2024-09-16T15:42:51.411434+0000","last_became_peered":"2024-09-16T15:42:51.411434+0000","last_unstale":"2024-09-16T15:42:52.418379+0000","last_undegraded":"2024-09-16T15:42:52.418379+0000","last_fullsized":"2024-09-16T15:42:52.418379+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_clean_scrub_stamp":"2024-09-16T15:42:50.361965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:40:52.417913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T15:42:52.417895+0000","last_change":"2024-09-16T15:42:50.408565+0000","last_active":"2024-09-16T15:42:52.417895+0000","last_peered":"2024-09-16T15:42:52.417895+0000","last_clean":"2024-09-16T15:42:52.417895+0000","last_became_active":"2024-09-16T15:42:50.408254+0000","last_became_peered":"2024-09-16T15:42:50.408254+0000","last_unstale":"2024-09-16T15:42:52.417895+0000","last_undegraded":"2024-09-16T15:42:52.417895+0000","last_fullsized":"2024-09-16T15:42:52.417895+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:42:49.352371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:42:49.352371+0000","last_clean_scrub_stamp":"2024-09-16T15:42:49.352371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:26:07.798125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T15:42:52.876598+0000","last_change":"2024-09-16T15:42:51.413873+0000","last_active":"2024-09-16T15:42:52.876598+0000","last_peered":"2024-09-16T15:42:52.876598+0000","last_clean":"2024-09-16T15:42:52.876598+0000","last_became_active":"2024-09-16T15:42:51.406117+0000","last_became_peered":"2024-09-16T15:42:51.406117+0000","last_unstale":"2024-09-16T15:42:52.876598+0000","last_undegraded":"2024-09-16T15:42:52.876598+0000","last_fullsized":"2024-09-16T15:42:52.876598+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_clean_scrub_stamp":"2024-09-16T15:42:50.361965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:12:23.570707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T15:42:52.417696+0000","last_change":"2024-09-16T15:42:50.388233+0000","last_active":"2024-09-16T15:42:52.417696+0000","last_peered":"2024-09-16T15:42:52.417696+0000","last_clean":"2024-09-16T15:42:52.417696+0000","last_became_active":"2024-09-16T15:42:50.387920+0000","last_became_peered":"2024-09-16T15:42:50.387920+0000","last_unstale":"2024-09-16T15:42:52.417696+0000","last_undegraded":"2024-09-16T15:42:52.417696+0000","last_fullsized":"2024-09-16T15:42:52.417696+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:42:49.352371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:42:49.352371+0000","last_clean_scrub_stamp":"2024-09-16T15:42:49.352371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T22:48:39.726611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T15:42:52.876523+0000","last_change":"2024-09-16T15:42:51.414425+0000","last_active":"2024-09-16T15:42:52.876523+0000","last_peered":"2024-09-16T15:42:52.876523+0000","last_clean":"2024-09-16T15:42:52.876523+0000","last_became_active":"2024-09-16T15:42:51.410797+0000","last_became_peered":"2024-09-16T15:42:51.410797+0000","last_unstale":"2024-09-16T15:42:52.876523+0000","last_undegraded":"2024-09-16T15:42:52.876523+0000","last_fullsized":"2024-09-16T15:42:52.876523+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_clean_scrub_stamp":"2024-09-16T15:42:50.361965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T18:40:25.285207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"20'2","reported_seq":17,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T15:42:52.725055+0000","last_change":"2024-09-16T15:42:50.423780+0000","last_active":"2024-09-16T15:42:52.725055+0000","last_peered":"2024-09-16T15:42:52.725055+0000","last_clean":"2024-09-16T15:42:52.725055+0000","last_became_active":"2024-09-16T15:42:50.423037+0000","last_became_peered":"2024-09-16T15:42:50.423037+0000","last_unstale":"2024-09-16T15:42:52.725055+0000","last_undegraded":"2024-09-16T15:42:52.725055+0000","last_fullsized":"2024-09-16T15:42:52.725055+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:42:49.352371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:42:49.352371+0000","last_clean_scrub_stamp":"2024-09-16T15:42:49.352371+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T01:04:05.966343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T15:42:52.415974+0000","last_change":"2024-09-16T15:42:51.397509+0000","last_active":"2024-09-16T15:42:52.415974+0000","last_peered":"2024-09-16T15:42:52.415974+0000","last_clean":"2024-09-16T15:42:52.415974+0000","last_became_active":"2024-09-16T15:42:51.396735+0000","last_became_peered":"2024-09-16T15:42:51.396735+0000","last_unstale":"2024-09-16T15:42:52.415974+0000","last_undegraded":"2024-09-16T15:42:52.415974+0000","last_fullsized":"2024-09-16T15:42:52.415974+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_clean_scrub_stamp":"2024-09-16T15:42:50.361965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T02:47:48.200896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T15:42:52.639530+0000","last_change":"2024-09-16T15:42:51.423874+0000","last_active":"2024-09-16T15:42:52.639530+0000","last_peered":"2024-09-16T15:42:52.639530+0000","last_clean":"2024-09-16T15:42:52.639530+0000","last_became_active":"2024-09-16T15:42:51.423455+0000","last_became_peered":"2024-09-16T15:42:51.423455+0000","last_unstale":"2024-09-16T15:42:52.639530+0000","last_undegraded":"2024-09-16T15:42:52.639530+0000","last_fullsized":"2024-09-16T15:42:52.639530+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_clean_scrub_stamp":"2024-09-16T15:42:50.361965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T22:43:57.612493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"20'2","reported_seq":17,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T15:42:52.724179+0000","last_change":"2024-09-16T15:42:50.428990+0000","last_active":"2024-09-16T15:42:52.724179+0000","last_peered":"2024-09-16T15:42:52.724179+0000","last_clean":"2024-09-16T15:42:52.724179+0000","last_became_active":"2024-09-16T15:42:50.428159+0000","last_became_peered":"2024-09-16T15:42:50.428159+0000","last_unstale":"2024-09-16T15:42:52.724179+0000","last_undegraded":"2024-09-16T15:42:52.724179+0000","last_fullsized":"2024-09-16T15:42:52.724179+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:42:49.352371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:42:49.352371+0000","last_clean_scrub_stamp":"2024-09-16T15:42:49.352371+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T01:17:11.293386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T15:42:52.638975+0000","last_change":"2024-09-16T15:42:51.421429+0000","last_active":"2024-09-16T15:42:52.638975+0000","last_peered":"2024-09-16T15:42:52.638975+0000","last_clean":"2024-09-16T15:42:52.638975+0000","last_became_active":"2024-09-16T15:42:51.420671+0000","last_became_peered":"2024-09-16T15:42:51.420671+0000","last_unstale":"2024-09-16T15:42:52.638975+0000","last_undegraded":"2024-09-16T15:42:52.638975+0000","last_fullsized":"2024-09-16T15:42:52.638975+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_clean_scrub_stamp":"2024-09-16T15:42:50.361965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T00:34:44.681110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"20'2","reported_seq":17,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T15:42:52.724740+0000","last_change":"2024-09-16T15:42:50.425037+0000","last_active":"2024-09-16T15:42:52.724740+0000","last_peered":"2024-09-16T15:42:52.724740+0000","last_clean":"2024-09-16T15:42:52.724740+0000","last_became_active":"2024-09-16T15:42:50.423907+0000","last_became_peered":"2024-09-16T15:42:50.423907+0000","last_unstale":"2024-09-16T15:42:52.724740+0000","last_undegraded":"2024-09-16T15:42:52.724740+0000","last_fullsized":"2024-09-16T15:42:52.724740+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:42:49.352371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:42:49.352371+0000","last_clean_scrub_stamp":"2024-09-16T15:42:49.352371+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:56:59.748634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T15:42:52.877982+0000","last_change":"2024-09-16T15:42:51.425681+0000","last_active":"2024-09-16T15:42:52.877982+0000","last_peered":"2024-09-16T15:42:52.877982+0000","last_clean":"2024-09-16T15:42:52.877982+0000","last_became_active":"2024-09-16T15:42:51.425367+0000","last_became_peered":"2024-09-16T15:42:51.425367+0000","last_unstale":"2024-09-16T15:42:52.877982+0000","last_undegraded":"2024-09-16T15:42:52.877982+0000","last_fullsized":"2024-09-16T15:42:52.877982+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_clean_scrub_stamp":"2024-09-16T15:42:50.361965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:21.758409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"20'1","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T15:42:52.875580+0000","last_change":"2024-09-16T15:42:50.392203+0000","last_active":"2024-09-16T15:42:52.875580+0000","last_peered":"2024-09-16T15:42:52.875580+0000","last_clean":"2024-09-16T15:42:52.875580+0000","last_became_active":"2024-09-16T15:42:50.391647+0000","last_became_peered":"2024-09-16T15:42:50.391647+0000","last_unstale":"2024-09-16T15:42:52.875580+0000","last_undegraded":"2024-09-16T15:42:52.875580+0000","last_fullsized":"2024-09-16T15:42:52.875580+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:42:49.352371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:42:49.352371+0000","last_clean_scrub_stamp":"2024-09-16T15:42:49.352371+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T22:35:29.889897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T15:42:52.876061+0000","last_change":"2024-09-16T15:42:51.408021+0000","last_active":"2024-09-16T15:42:52.876061+0000","last_peered":"2024-09-16T15:42:52.876061+0000","last_clean":"2024-09-16T15:42:52.876061+0000","last_became_active":"2024-09-16T15:42:51.407085+0000","last_became_peered":"2024-09-16T15:42:51.407085+0000","last_unstale":"2024-09-16T15:42:52.876061+0000","last_undegraded":"2024-09-16T15:42:52.876061+0000","last_fullsized":"2024-09-16T15:42:52.876061+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_clean_scrub_stamp":"2024-09-16T15:42:50.361965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:38:38.580439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"20'4","reported_seq":19,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T15:42:52.878108+0000","last_change":"2024-09-16T15:42:50.415954+0000","last_active":"2024-09-16T15:42:52.878108+0000","last_peered":"2024-09-16T15:42:52.878108+0000","last_clean":"2024-09-16T15:42:52.878108+0000","last_became_active":"2024-09-16T15:42:50.415619+0000","last_became_peered":"2024-09-16T15:42:50.415619+0000","last_unstale":"2024-09-16T15:42:52.878108+0000","last_undegraded":"2024-09-16T15:42:52.878108+0000","last_fullsized":"2024-09-16T15:42:52.878108+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:42:49.352371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:42:49.352371+0000","last_clean_scrub_stamp":"2024-09-16T15:42:49.352371+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T02:01:17.143570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542150,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27824,"kb_used_data":792,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344016,"statfs":{"total":96636764160,"available":96608272384,"internally_reserved":0,"allocated":811008,"data_stored":665039,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542149,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27256,"kb_used_data":220,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344584,"statfs":{"total":96636764160,"available":96608854016,"internally_reserved":0,"allocated":225280,"data_stored":75701,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":5,"up_from":14,"seq":60129542150,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27240,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344600,"statfs":{"total":96636764160,"available":96608870400,"internally_reserved":0,"allocated":217088,"data_stored":74179,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542148,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27252,"kb_used_data":220,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344588,"statfs":{"total":96636764160,"available":96608858112,"internally_reserved":0,"allocated":225280,"data_stored":74215,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":3,"up_from":14,"seq":60129542149,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27256,"kb_used_data":220,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344584,"statfs":{"total":96636764160,"available":96608854016,"internally_reserved":0,"allocated":225280,"data_stored":75701,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":2,"up_from":14,"seq":60129542149,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27236,"kb_used_data":208,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344604,"statfs":{"total":96636764160,"available":96608874496,"internally_reserved":0,"allocated":212992,"data_stored":74089,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":14,"seq":60129542151,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27244,"kb_used_data":216,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344596,"statfs":{"total":96636764160,"available":96608866304,"internally_reserved":0,"allocated":221184,"data_stored":74249,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":14,"seq":60129542151,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27832,"kb_used_data":800,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344008,"statfs":{"total":96636764160,"available":96608264192,"internally_reserved":0,"allocated":819200,"data_stored":665095,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T15:42:55.798 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-09-16T15:42:55.799 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config log 1 --format=json 2024-09-16T15:42:55.799 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2024-09-16T15:42:55.958 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:55.954+0000 7f4d25e2e640 1 -- 172.21.15.123:0/3799648533 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4d2015ec40 msgr2=0x7f4d2006f420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:42:55.958 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:55.954+0000 7f4d25e2e640 1 --2- 172.21.15.123:0/3799648533 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4d2015ec40 0x7f4d2006f420 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f4d20076280 tx=0x7f4d1801cc60 comp rx=0 tx=0).stop 2024-09-16T15:42:55.958 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:55.954+0000 7f4d25e2e640 1 -- 172.21.15.123:0/3799648533 shutdown_connections 2024-09-16T15:42:55.958 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:55.954+0000 7f4d25e2e640 1 --2- 172.21.15.123:0/3799648533 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4d2015ec40 0x7f4d2006f420 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:42:55.958 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:55.954+0000 7f4d25e2e640 1 --2- 172.21.15.123:0/3799648533 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f4d200cf5a0 0x7f4d2006eee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:42:55.958 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:55.954+0000 7f4d25e2e640 1 --2- 172.21.15.123:0/3799648533 >> v2:172.21.15.184:3300/0 conn(0x7f4d20158120 0x7f4d200ca320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:42:55.959 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:55.954+0000 7f4d25e2e640 1 -- 172.21.15.123:0/3799648533 >> 172.21.15.123:0/3799648533 conn(0x7f4d200cfb10 msgr2=0x7f4d200cff20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:42:55.959 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:55.954+0000 7f4d25e2e640 1 -- 172.21.15.123:0/3799648533 shutdown_connections 2024-09-16T15:42:55.959 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:55.954+0000 7f4d25e2e640 1 -- 172.21.15.123:0/3799648533 wait complete. 2024-09-16T15:42:55.959 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:55.958+0000 7f4d25e2e640 1 Processor -- start 2024-09-16T15:42:55.959 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:55.958+0000 7f4d25e2e640 1 -- start start 2024-09-16T15:42:55.961 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:55.958+0000 7f4d25e2e640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f4d200cf5a0 0x7f4d2014e230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:42:55.961 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:55.958+0000 7f4d25e2e640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4d20158120 0x7f4d200b4e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:42:55.961 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:55.958+0000 7f4d25e2e640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4d2015ec40 0x7f4d200b5360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:42:55.961 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:55.958+0000 7f4d25e2e640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f4d2015b170 con 0x7f4d20158120 2024-09-16T15:42:55.961 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:55.958+0000 7f4d25e2e640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f4d2015aff0 con 0x7f4d2015ec40 2024-09-16T15:42:55.961 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:55.958+0000 7f4d25e2e640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f4d2015b2f0 con 0x7f4d200cf5a0 2024-09-16T15:42:55.961 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:55.958+0000 7f4d1effd640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4d20158120 0x7f4d200b4e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:42:55.962 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:55.958+0000 7f4d1effd640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4d20158120 0x7f4d200b4e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:40204/0 (socket says 172.21.15.123:40204) 2024-09-16T15:42:55.962 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:55.958+0000 7f4d1effd640 1 -- 172.21.15.123:0/4258307725 learned_addr learned my addr 172.21.15.123:0/4258307725 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:42:55.962 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:55.958+0000 7f4d1ffff640 1 --2- 172.21.15.123:0/4258307725 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4d2015ec40 0x7f4d200b5360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:42:55.962 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:55.958+0000 7f4d1f7fe640 1 --2- 172.21.15.123:0/4258307725 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f4d200cf5a0 0x7f4d2014e230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:42:55.962 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:55.958+0000 7f4d1effd640 1 -- 172.21.15.123:0/4258307725 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f4d200cf5a0 msgr2=0x7f4d2014e230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:42:55.962 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:55.958+0000 7f4d1effd640 1 --2- 172.21.15.123:0/4258307725 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f4d200cf5a0 0x7f4d2014e230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:42:55.962 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:55.958+0000 7f4d1effd640 1 -- 172.21.15.123:0/4258307725 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4d2015ec40 msgr2=0x7f4d200b5360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:42:55.962 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:55.958+0000 7f4d1effd640 1 --2- 172.21.15.123:0/4258307725 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4d2015ec40 0x7f4d200b5360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:42:55.962 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:55.958+0000 7f4d1effd640 1 -- 172.21.15.123:0/4258307725 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4d201f8a40 con 0x7f4d20158120 2024-09-16T15:42:55.962 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:55.958+0000 7f4d1ffff640 1 --2- 172.21.15.123:0/4258307725 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4d2015ec40 0x7f4d200b5360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T15:42:55.962 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:55.958+0000 7f4d1effd640 1 --2- 172.21.15.123:0/4258307725 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4d20158120 0x7f4d200b4e20 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f4d1400d750 tx=0x7f4d1400dc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:42:55.962 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:55.958+0000 7f4d07fff640 1 -- 172.21.15.123:0/4258307725 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d14017070 con 0x7f4d20158120 2024-09-16T15:42:55.962 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:55.958+0000 7f4d07fff640 1 -- 172.21.15.123:0/4258307725 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f4d14009d20 con 0x7f4d20158120 2024-09-16T15:42:55.963 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:55.958+0000 7f4d25e2e640 1 -- 172.21.15.123:0/4258307725 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4d201f8d20 con 0x7f4d20158120 2024-09-16T15:42:55.963 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:55.958+0000 7f4d07fff640 1 -- 172.21.15.123:0/4258307725 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d1400de60 con 0x7f4d20158120 2024-09-16T15:42:55.963 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:55.958+0000 7f4d25e2e640 1 -- 172.21.15.123:0/4258307725 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4d201f92e0 con 0x7f4d20158120 2024-09-16T15:42:55.964 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:55.958+0000 7f4d25e2e640 1 -- 172.21.15.123:0/4258307725 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4d200ca320 con 0x7f4d20158120 2024-09-16T15:42:55.964 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:55.962+0000 7f4d07fff640 1 -- 172.21.15.123:0/4258307725 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 5) ==== 97080+0+0 (secure 0 0 0) 0x7f4d14005280 con 0x7f4d20158120 2024-09-16T15:42:55.964 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:55.962+0000 7f4d07fff640 1 --2- 172.21.15.123:0/4258307725 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f4d0c072ec0 0x7f4d0c075380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:42:55.964 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:55.962+0000 7f4d07fff640 1 -- 172.21.15.123:0/4258307725 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f4d14093f70 con 0x7f4d20158120 2024-09-16T15:42:55.964 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:55.962+0000 7f4d1f7fe640 1 --2- 172.21.15.123:0/4258307725 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f4d0c072ec0 0x7f4d0c075380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:42:55.964 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:55.962+0000 7f4d1f7fe640 1 --2- 172.21.15.123:0/4258307725 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f4d0c072ec0 0x7f4d0c075380 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f4d100058d0 tx=0x7f4d10005860 comp rx=0 tx=0).ready entity=mgr.4135 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:42:55.966 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:55.966+0000 7f4d07fff640 1 -- 172.21.15.123:0/4258307725 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f4d1405f1c0 con 0x7f4d20158120 2024-09-16T15:42:56.131 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:56.130+0000 7f4d25e2e640 1 -- 172.21.15.123:0/4258307725 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "config log", "num": 1, "format": "json"} v 0) -- 0x7f4d200d06c0 con 0x7f4d20158120 2024-09-16T15:42:56.132 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:56.130+0000 7f4d07fff640 1 -- 172.21.15.123:0/4258307725 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "config log", "num": 1, "format": "json"}]=0 v27) ==== 86+0+160 (secure 0 0 0) 0x7f4d1405eac0 con 0x7f4d20158120 2024-09-16T15:42:56.132 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-09-16T15:42:56.135 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:56.130+0000 7f4d25e2e640 1 -- 172.21.15.123:0/4258307725 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f4d0c072ec0 msgr2=0x7f4d0c075380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:42:56.135 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:56.130+0000 7f4d25e2e640 1 --2- 172.21.15.123:0/4258307725 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f4d0c072ec0 0x7f4d0c075380 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f4d100058d0 tx=0x7f4d10005860 comp rx=0 tx=0).stop 2024-09-16T15:42:56.135 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:56.130+0000 7f4d25e2e640 1 -- 172.21.15.123:0/4258307725 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4d20158120 msgr2=0x7f4d200b4e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:42:56.135 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:56.130+0000 7f4d25e2e640 1 --2- 172.21.15.123:0/4258307725 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4d20158120 0x7f4d200b4e20 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f4d1400d750 tx=0x7f4d1400dc10 comp rx=0 tx=0).stop 2024-09-16T15:42:56.135 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:56.134+0000 7f4d25e2e640 1 -- 172.21.15.123:0/4258307725 shutdown_connections 2024-09-16T15:42:56.135 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:56.134+0000 7f4d25e2e640 1 --2- 172.21.15.123:0/4258307725 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f4d0c072ec0 0x7f4d0c075380 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:42:56.135 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:56.134+0000 7f4d25e2e640 1 --2- 172.21.15.123:0/4258307725 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4d2015ec40 0x7f4d200b5360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:42:56.135 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:56.134+0000 7f4d25e2e640 1 --2- 172.21.15.123:0/4258307725 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4d20158120 0x7f4d200b4e20 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:42:56.135 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:56.134+0000 7f4d25e2e640 1 --2- 172.21.15.123:0/4258307725 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f4d200cf5a0 0x7f4d2014e230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:42:56.135 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:56.134+0000 7f4d25e2e640 1 -- 172.21.15.123:0/4258307725 >> 172.21.15.123:0/4258307725 conn(0x7f4d200cfb10 msgr2=0x7f4d200ce900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:42:56.135 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:56.134+0000 7f4d25e2e640 1 -- 172.21.15.123:0/4258307725 shutdown_connections 2024-09-16T15:42:56.135 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:56.134+0000 7f4d25e2e640 1 -- 172.21.15.123:0/4258307725 wait complete. 2024-09-16T15:42:56.149 INFO:teuthology.orchestra.run.smithi123.stdout:[{"version":27,"timestamp":"2024-09-16T15:42:38.648130+0000","name":"","changes":[{"name":"osd.4/osd_mclock_max_capacity_iops_ssd","new_value":"6108.444971"}]}] 2024-09-16T15:42:56.150 INFO:tasks.ceph_manager:config epoch is 27 2024-09-16T15:42:56.150 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-09-16T15:42:56.150 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2024-09-16T15:42:56.150 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2024-09-16T15:42:56.263 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:56.262+0000 7f0ea32ef640 1 -- 172.21.15.123:0/2197850169 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0e84004810 msgr2=0x7f0e84004c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:42:56.263 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:56.262+0000 7f0ea32ef640 1 --2- 172.21.15.123:0/2197850169 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0e84004810 0x7f0e84004c90 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f0e90001660 tx=0x7f0e9001c730 comp rx=0 tx=0).stop 2024-09-16T15:42:56.263 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:56.262+0000 7f0ea188c640 1 -- 172.21.15.123:0/2197850169 reap_dead start 2024-09-16T15:42:56.263 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:56.262+0000 7f0ea32ef640 1 -- 172.21.15.123:0/2197850169 shutdown_connections 2024-09-16T15:42:56.264 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:56.262+0000 7f0ea32ef640 1 -- 172.21.15.123:0/2197850169 >> 172.21.15.123:0/2197850169 conn(0x7f0e9c0cfb10 msgr2=0x7f0e9c0cff20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:42:56.264 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:56.262+0000 7f0ea32ef640 1 -- 172.21.15.123:0/2197850169 shutdown_connections 2024-09-16T15:42:56.264 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:56.262+0000 7f0ea32ef640 1 -- 172.21.15.123:0/2197850169 wait complete. 2024-09-16T15:42:56.264 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:56.262+0000 7f0ea32ef640 1 Processor -- start 2024-09-16T15:42:56.264 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:56.262+0000 7f0ea32ef640 1 -- start start 2024-09-16T15:42:56.266 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:56.262+0000 7f0ea32ef640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f0e84004810 0x7f0e9c0c7fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:42:56.266 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:56.262+0000 7f0ea32ef640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f0e9c155e40 0x7f0e9c0c6be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:42:56.266 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:56.262+0000 7f0ea32ef640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0e9c1f9cb0 0x7f0e9c1fa130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:42:56.266 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:56.262+0000 7f0ea32ef640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f0e9c0cf5b0 con 0x7f0e9c1f9cb0 2024-09-16T15:42:56.267 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:56.262+0000 7f0ea32ef640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f0e9c0cf430 con 0x7f0e84004810 2024-09-16T15:42:56.267 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:56.262+0000 7f0ea32ef640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f0e9c1673d0 con 0x7f0e9c155e40 2024-09-16T15:42:56.267 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:56.262+0000 7f0ea188c640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0e9c1f9cb0 0x7f0e9c1fa130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:42:56.267 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:56.262+0000 7f0ea188c640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0e9c1f9cb0 0x7f0e9c1fa130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:40222/0 (socket says 172.21.15.123:40222) 2024-09-16T15:42:56.267 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:56.262+0000 7f0ea188c640 1 -- 172.21.15.123:0/398782418 learned_addr learned my addr 172.21.15.123:0/398782418 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:42:56.267 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:56.262+0000 7f0ea088a640 1 --2- 172.21.15.123:0/398782418 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f0e9c155e40 0x7f0e9c0c6be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:42:56.267 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:56.262+0000 7f0ea108b640 1 --2- 172.21.15.123:0/398782418 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f0e84004810 0x7f0e9c0c7fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:42:56.267 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:56.262+0000 7f0ea188c640 1 -- 172.21.15.123:0/398782418 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f0e9c155e40 msgr2=0x7f0e9c0c6be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:42:56.267 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:56.262+0000 7f0ea188c640 1 --2- 172.21.15.123:0/398782418 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f0e9c155e40 0x7f0e9c0c6be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:42:56.267 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:56.262+0000 7f0ea188c640 1 -- 172.21.15.123:0/398782418 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f0e84004810 msgr2=0x7f0e9c0c7fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:42:56.267 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:56.262+0000 7f0ea188c640 1 --2- 172.21.15.123:0/398782418 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f0e84004810 0x7f0e9c0c7fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:42:56.267 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:56.262+0000 7f0ea188c640 1 -- 172.21.15.123:0/398782418 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0e9c1562c0 con 0x7f0e9c1f9cb0 2024-09-16T15:42:56.267 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:56.262+0000 7f0ea108b640 1 --2- 172.21.15.123:0/398782418 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f0e84004810 0x7f0e9c0c7fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T15:42:56.267 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:56.262+0000 7f0ea188c640 1 --2- 172.21.15.123:0/398782418 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0e9c1f9cb0 0x7f0e9c1fa130 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f0e8c004a60 tx=0x7f0e8c00dd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:42:56.268 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:56.262+0000 7f0e99ffb640 1 -- 172.21.15.123:0/398782418 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e8c016020 con 0x7f0e9c1f9cb0 2024-09-16T15:42:56.268 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:56.262+0000 7f0e99ffb640 1 -- 172.21.15.123:0/398782418 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f0e8c00b840 con 0x7f0e9c1f9cb0 2024-09-16T15:42:56.268 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:56.262+0000 7f0ea32ef640 1 -- 172.21.15.123:0/398782418 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0e9c0c7120 con 0x7f0e9c1f9cb0 2024-09-16T15:42:56.268 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:56.262+0000 7f0e99ffb640 1 -- 172.21.15.123:0/398782418 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e8c00bc90 con 0x7f0e9c1f9cb0 2024-09-16T15:42:56.268 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:56.262+0000 7f0ea32ef640 1 -- 172.21.15.123:0/398782418 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0e9c1658d0 con 0x7f0e9c1f9cb0 2024-09-16T15:42:56.269 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:56.266+0000 7f0ea32ef640 1 -- 172.21.15.123:0/398782418 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0e9c1f9610 con 0x7f0e9c1f9cb0 2024-09-16T15:42:56.269 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:56.266+0000 7f0e99ffb640 1 -- 172.21.15.123:0/398782418 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 5) ==== 97080+0+0 (secure 0 0 0) 0x7f0e8c019020 con 0x7f0e9c1f9cb0 2024-09-16T15:42:56.269 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:56.266+0000 7f0e99ffb640 1 --2- 172.21.15.123:0/398782418 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f0e88072ef0 0x7f0e880753b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:42:56.269 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:56.266+0000 7f0e99ffb640 1 -- 172.21.15.123:0/398782418 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f0e8c093a30 con 0x7f0e9c1f9cb0 2024-09-16T15:42:56.269 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:56.266+0000 7f0ea108b640 1 --2- 172.21.15.123:0/398782418 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f0e88072ef0 0x7f0e880753b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:42:56.269 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:56.266+0000 7f0ea108b640 1 --2- 172.21.15.123:0/398782418 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f0e88072ef0 0x7f0e880753b0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f0e94005d10 tx=0x7f0e94005f00 comp rx=0 tx=0).ready entity=mgr.4135 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:42:56.272 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:56.270+0000 7f0e99ffb640 1 -- 172.21.15.123:0/398782418 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f0e8c020030 con 0x7f0e9c1f9cb0 2024-09-16T15:42:56.347 INFO:tasks.ceph.mds.d.smithi184.stderr:2024-09-16T15:42:56.347+0000 7fb2b2e57640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-09-16T15:42:56.471 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:56.470+0000 7f0ea32ef640 1 -- 172.21.15.123:0/398782418 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) -- 0x7f0e9c0cd6e0 con 0x7f0e9c1f9cb0 2024-09-16T15:42:56.474 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:56.470+0000 7f0e99ffb640 1 -- 172.21.15.123:0/398782418 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v5) ==== 74+0+186090 (secure 0 0 0) 0x7f0e9c1562c0 con 0x7f0e9c1f9cb0 2024-09-16T15:42:56.476 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-09-16T15:42:56.477 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:56.474+0000 7f0ea32ef640 1 -- 172.21.15.123:0/398782418 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f0e88072ef0 msgr2=0x7f0e880753b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:42:56.477 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:56.474+0000 7f0ea32ef640 1 --2- 172.21.15.123:0/398782418 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f0e88072ef0 0x7f0e880753b0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f0e94005d10 tx=0x7f0e94005f00 comp rx=0 tx=0).stop 2024-09-16T15:42:56.477 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:56.474+0000 7f0ea32ef640 1 -- 172.21.15.123:0/398782418 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0e9c1f9cb0 msgr2=0x7f0e9c1fa130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:42:56.478 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:56.474+0000 7f0ea32ef640 1 --2- 172.21.15.123:0/398782418 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0e9c1f9cb0 0x7f0e9c1fa130 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f0e8c004a60 tx=0x7f0e8c00dd50 comp rx=0 tx=0).stop 2024-09-16T15:42:56.478 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:56.474+0000 7f0ea32ef640 1 -- 172.21.15.123:0/398782418 shutdown_connections 2024-09-16T15:42:56.478 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:56.474+0000 7f0ea32ef640 1 --2- 172.21.15.123:0/398782418 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f0e88072ef0 0x7f0e880753b0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:42:56.478 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:56.474+0000 7f0ea32ef640 1 --2- 172.21.15.123:0/398782418 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0e9c1f9cb0 0x7f0e9c1fa130 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:42:56.478 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:56.474+0000 7f0ea32ef640 1 --2- 172.21.15.123:0/398782418 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f0e9c155e40 0x7f0e9c0c6be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:42:56.478 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:56.474+0000 7f0ea32ef640 1 --2- 172.21.15.123:0/398782418 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f0e84004810 0x7f0e9c0c7fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:42:56.478 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:56.474+0000 7f0ea32ef640 1 -- 172.21.15.123:0/398782418 >> 172.21.15.123:0/398782418 conn(0x7f0e9c0cfb10 msgr2=0x7f0e9c158dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:42:56.478 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:56.474+0000 7f0ea32ef640 1 -- 172.21.15.123:0/398782418 shutdown_connections 2024-09-16T15:42:56.478 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:56.474+0000 7f0ea32ef640 1 -- 172.21.15.123:0/398782418 wait complete. 2024-09-16T15:42:56.492 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":5,"flags":0,"active_gid":4135,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":3114346423},{"type":"v1","addr":"172.21.15.184:6833","nonce":3114346423}]},"active_addr":"172.21.15.184:6833/3114346423","active_change":"2024-09-16T15:42:31.230175+0000","active_mgr_features":4540719139924082687,"available":true,"standbys":[{"gid":4150,"name":"y","mgr_features":4540719139924082687,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to, use commas to separate multiple","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.27.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:10.4.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_nginx":{"name":"container_image_nginx","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nginx:sclorg-nginx-126","min":"","max":"","enum_allowed":[],"desc":"Nginx container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.7.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.17","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_oauth2_proxy":{"name":"container_image_oauth2_proxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/oauth2-proxy/oauth2-proxy:v7.6.0","min":"","max":"","enum_allowed":[],"desc":"oauth2-proxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.51.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba_metrics":{"name":"container_image_samba_metrics","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-metrics:latest","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB metrics exporter container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"ssh_keepalive_count_max":{"name":"ssh_keepalive_count_max","type":"int","level":"advanced","flags":0,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"How many times ssh connections can fail liveness checks before the host is marked offline","long_desc":"","tags":[],"see_also":[]},"ssh_keepalive_interval":{"name":"ssh_keepalive_interval","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"How often ssh connections are checked for liveness","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_requests":{"name":"max_requests","type":"int","level":"advanced","flags":0,"default_value":"500","min":"","max":"","enum_allowed":[],"desc":"Maximum number of requests to keep in memory. When new request comes in, the oldest request will be removed if the number of requests exceeds the max request number.if un-finished request is removed, error message will be logged in the ceph-mgr log.","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"secondary_zone_period_retry_limit":{"name":"secondary_zone_period_retry_limit","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"RGW module period update retry limit for secondary site","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to, use commas to separate multiple","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.27.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:10.4.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_nginx":{"name":"container_image_nginx","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nginx:sclorg-nginx-126","min":"","max":"","enum_allowed":[],"desc":"Nginx container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.7.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.17","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_oauth2_proxy":{"name":"container_image_oauth2_proxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/oauth2-proxy/oauth2-proxy:v7.6.0","min":"","max":"","enum_allowed":[],"desc":"oauth2-proxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.51.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba_metrics":{"name":"container_image_samba_metrics","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-metrics:latest","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB metrics exporter container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"ssh_keepalive_count_max":{"name":"ssh_keepalive_count_max","type":"int","level":"advanced","flags":0,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"How many times ssh connections can fail liveness checks before the host is marked offline","long_desc":"","tags":[],"see_also":[]},"ssh_keepalive_interval":{"name":"ssh_keepalive_interval","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"How often ssh connections are checked for liveness","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_requests":{"name":"max_requests","type":"int","level":"advanced","flags":0,"default_value":"500","min":"","max":"","enum_allowed":[],"desc":"Maximum number of requests to keep in memory. When new request comes in, the oldest request will be removed if the number of requests exceeds the max request number.if un-finished request is removed, error message will be logged in the ceph-mgr log.","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"secondary_zone_period_retry_limit":{"name":"secondary_zone_period_retry_limit","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"RGW module period update retry limit for secondary site","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.184:0","nonce":1715342118}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.184:0","nonce":2291367472}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.184:0","nonce":1402401038}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.184:0","nonce":2406729396}]}]} 2024-09-16T15:42:56.496 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2024-09-16T15:42:56.496 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2024-09-16T15:42:56.496 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-16T15:42:56.616 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:56.614+0000 7f8c8593d640 1 -- 172.21.15.123:0/3973870487 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8c800cf5a0 msgr2=0x7f8c8006eee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:42:56.616 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:56.614+0000 7f8c8593d640 1 --2- 172.21.15.123:0/3973870487 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8c800cf5a0 0x7f8c8006eee0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f8c68009840 tx=0x7f8c6801c700 comp rx=0 tx=0).stop 2024-09-16T15:42:56.616 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:56.614+0000 7f8c8593d640 1 -- 172.21.15.123:0/3973870487 shutdown_connections 2024-09-16T15:42:56.616 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:56.614+0000 7f8c8593d640 1 --2- 172.21.15.123:0/3973870487 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8c800cf5a0 0x7f8c8006eee0 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:42:56.616 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:56.614+0000 7f8c8593d640 1 -- 172.21.15.123:0/3973870487 >> 172.21.15.123:0/3973870487 conn(0x7f8c800cfb10 msgr2=0x7f8c800cff20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:42:56.616 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:56.614+0000 7f8c8593d640 1 -- 172.21.15.123:0/3973870487 shutdown_connections 2024-09-16T15:42:56.616 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:56.614+0000 7f8c8593d640 1 -- 172.21.15.123:0/3973870487 wait complete. 2024-09-16T15:42:56.617 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:56.614+0000 7f8c8593d640 1 Processor -- start 2024-09-16T15:42:56.617 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:56.614+0000 7f8c8593d640 1 -- start start 2024-09-16T15:42:56.619 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:56.614+0000 7f8c8593d640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f8c800cf5a0 0x7f8c8014e500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:42:56.620 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:56.618+0000 7f8c8593d640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8c80158120 0x7f8c8014dce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:42:56.620 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:56.618+0000 7f8c8593d640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f8c8015ec40 0x7f8c80153b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:42:56.620 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:56.618+0000 7f8c8593d640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f8c8006ec90 con 0x7f8c80158120 2024-09-16T15:42:56.620 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:56.618+0000 7f8c8593d640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f8c8006eb10 con 0x7f8c8015ec40 2024-09-16T15:42:56.620 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:56.618+0000 7f8c8593d640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f8c801694f0 con 0x7f8c800cf5a0 2024-09-16T15:42:56.620 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:56.618+0000 7f8c767fc640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8c80158120 0x7f8c8014dce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:42:56.620 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:56.618+0000 7f8c767fc640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8c80158120 0x7f8c8014dce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:40240/0 (socket says 172.21.15.123:40240) 2024-09-16T15:42:56.620 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:56.618+0000 7f8c767fc640 1 -- 172.21.15.123:0/3535028721 learned_addr learned my addr 172.21.15.123:0/3535028721 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:42:56.620 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:56.618+0000 7f8c7f7fe640 1 --2- 172.21.15.123:0/3535028721 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f8c8015ec40 0x7f8c80153b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:42:56.620 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:56.618+0000 7f8c7effd640 1 --2- 172.21.15.123:0/3535028721 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f8c800cf5a0 0x7f8c8014e500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:42:56.620 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:56.618+0000 7f8c767fc640 1 -- 172.21.15.123:0/3535028721 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f8c800cf5a0 msgr2=0x7f8c8014e500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:42:56.620 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:56.618+0000 7f8c767fc640 1 --2- 172.21.15.123:0/3535028721 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f8c800cf5a0 0x7f8c8014e500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:42:56.620 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:56.618+0000 7f8c767fc640 1 -- 172.21.15.123:0/3535028721 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f8c8015ec40 msgr2=0x7f8c80153b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:42:56.621 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:56.618+0000 7f8c767fc640 1 --2- 172.21.15.123:0/3535028721 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f8c8015ec40 0x7f8c80153b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:42:56.621 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:56.618+0000 7f8c767fc640 1 -- 172.21.15.123:0/3535028721 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8c80074120 con 0x7f8c80158120 2024-09-16T15:42:56.621 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:56.618+0000 7f8c7effd640 1 --2- 172.21.15.123:0/3535028721 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f8c800cf5a0 0x7f8c8014e500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T15:42:56.621 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:56.618+0000 7f8c767fc640 1 --2- 172.21.15.123:0/3535028721 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8c80158120 0x7f8c8014dce0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f8c68009790 tx=0x7f8c680a86e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:42:56.621 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:56.618+0000 7f8c7cff9640 1 -- 172.21.15.123:0/3535028721 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c68005350 con 0x7f8c80158120 2024-09-16T15:42:56.621 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:56.618+0000 7f8c7cff9640 1 -- 172.21.15.123:0/3535028721 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f8c680054f0 con 0x7f8c80158120 2024-09-16T15:42:56.621 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:56.618+0000 7f8c7cff9640 1 -- 172.21.15.123:0/3535028721 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c68004910 con 0x7f8c80158120 2024-09-16T15:42:56.621 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:56.618+0000 7f8c8593d640 1 -- 172.21.15.123:0/3535028721 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8c80074390 con 0x7f8c80158120 2024-09-16T15:42:56.621 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:56.618+0000 7f8c8593d640 1 -- 172.21.15.123:0/3535028721 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8c8015a9c0 con 0x7f8c80158120 2024-09-16T15:42:56.622 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:56.618+0000 7f8c8593d640 1 -- 172.21.15.123:0/3535028721 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8c8006eee0 con 0x7f8c80158120 2024-09-16T15:42:56.622 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:56.618+0000 7f8c7cff9640 1 -- 172.21.15.123:0/3535028721 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 5) ==== 97080+0+0 (secure 0 0 0) 0x7f8c68004ab0 con 0x7f8c80158120 2024-09-16T15:42:56.622 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:56.618+0000 7f8c7cff9640 1 --2- 172.21.15.123:0/3535028721 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f8c60072ec0 0x7f8c60075380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:42:56.622 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:56.618+0000 7f8c7cff9640 1 -- 172.21.15.123:0/3535028721 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f8c6812e510 con 0x7f8c80158120 2024-09-16T15:42:56.622 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:56.618+0000 7f8c7effd640 1 --2- 172.21.15.123:0/3535028721 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f8c60072ec0 0x7f8c60075380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:42:56.623 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:56.622+0000 7f8c7effd640 1 --2- 172.21.15.123:0/3535028721 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f8c60072ec0 0x7f8c60075380 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f8c70000f70 tx=0x7f8c70009040 comp rx=0 tx=0).ready entity=mgr.4135 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:42:56.626 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:56.622+0000 7f8c7cff9640 1 -- 172.21.15.123:0/3535028721 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f8c680ac030 con 0x7f8c80158120 2024-09-16T15:42:56.787 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:56.786+0000 7f8c8593d640 1 -- 172.21.15.123:0/3535028721 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f8c800729a0 con 0x7f8c80158120 2024-09-16T15:42:56.789 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:56.786+0000 7f8c7cff9640 1 -- 172.21.15.123:0/3535028721 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) ==== 74+0+16725 (secure 0 0 0) 0x7f8c68016410 con 0x7f8c80158120 2024-09-16T15:42:56.790 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-09-16T15:42:56.790 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":21,"fsid":"56077f89-18c8-486f-ae7e-eecc0bb7822c","created":"2024-09-16T15:42:26.364892+0000","modified":"2024-09-16T15:42:52.401147+0000","last_up_change":"2024-09-16T15:42:38.707234+0000","last_in_change":"2024-09-16T15:42:29.423662+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-16T15:42:39.287764+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-09-16T15:42:48.583563+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-09-16T15:42:49.704665+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9e7c1955-a607-4671-9be7-f2ef760ceeaa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6800","nonce":3882044969},{"type":"v1","addr":"172.21.15.123:6801","nonce":3882044969}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6802","nonce":3882044969},{"type":"v1","addr":"172.21.15.123:6803","nonce":3882044969}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6806","nonce":3882044969},{"type":"v1","addr":"172.21.15.123:6807","nonce":3882044969}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6804","nonce":3882044969},{"type":"v1","addr":"172.21.15.123:6805","nonce":3882044969}]},"public_addr":"172.21.15.123:6801/3882044969","cluster_addr":"172.21.15.123:6803/3882044969","heartbeat_back_addr":"172.21.15.123:6807/3882044969","heartbeat_front_addr":"172.21.15.123:6805/3882044969","state":["exists","up"]},{"osd":1,"uuid":"c540cfde-0977-4742-9b17-fd634668e010","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6816","nonce":4037001850},{"type":"v1","addr":"172.21.15.123:6817","nonce":4037001850}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6818","nonce":4037001850},{"type":"v1","addr":"172.21.15.123:6819","nonce":4037001850}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6822","nonce":4037001850},{"type":"v1","addr":"172.21.15.123:6823","nonce":4037001850}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6820","nonce":4037001850},{"type":"v1","addr":"172.21.15.123:6821","nonce":4037001850}]},"public_addr":"172.21.15.123:6817/4037001850","cluster_addr":"172.21.15.123:6819/4037001850","heartbeat_back_addr":"172.21.15.123:6823/4037001850","heartbeat_front_addr":"172.21.15.123:6821/4037001850","state":["exists","up"]},{"osd":2,"uuid":"9864a496-f169-486b-95c4-1d59162793a7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6824","nonce":1332143088},{"type":"v1","addr":"172.21.15.123:6825","nonce":1332143088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6826","nonce":1332143088},{"type":"v1","addr":"172.21.15.123:6827","nonce":1332143088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6830","nonce":1332143088},{"type":"v1","addr":"172.21.15.123:6831","nonce":1332143088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6828","nonce":1332143088},{"type":"v1","addr":"172.21.15.123:6829","nonce":1332143088}]},"public_addr":"172.21.15.123:6825/1332143088","cluster_addr":"172.21.15.123:6827/1332143088","heartbeat_back_addr":"172.21.15.123:6831/1332143088","heartbeat_front_addr":"172.21.15.123:6829/1332143088","state":["exists","up"]},{"osd":3,"uuid":"c6d1f817-f074-4325-95e1-578eba13edda","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6808","nonce":988586998},{"type":"v1","addr":"172.21.15.123:6809","nonce":988586998}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6810","nonce":988586998},{"type":"v1","addr":"172.21.15.123:6811","nonce":988586998}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6814","nonce":988586998},{"type":"v1","addr":"172.21.15.123:6815","nonce":988586998}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6812","nonce":988586998},{"type":"v1","addr":"172.21.15.123:6813","nonce":988586998}]},"public_addr":"172.21.15.123:6809/988586998","cluster_addr":"172.21.15.123:6811/988586998","heartbeat_back_addr":"172.21.15.123:6815/988586998","heartbeat_front_addr":"172.21.15.123:6813/988586998","state":["exists","up"]},{"osd":4,"uuid":"200a8df4-50ef-40a5-b48a-58981269c086","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6816","nonce":3387813890},{"type":"v1","addr":"172.21.15.184:6817","nonce":3387813890}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6818","nonce":3387813890},{"type":"v1","addr":"172.21.15.184:6819","nonce":3387813890}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6822","nonce":3387813890},{"type":"v1","addr":"172.21.15.184:6823","nonce":3387813890}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6820","nonce":3387813890},{"type":"v1","addr":"172.21.15.184:6821","nonce":3387813890}]},"public_addr":"172.21.15.184:6817/3387813890","cluster_addr":"172.21.15.184:6819/3387813890","heartbeat_back_addr":"172.21.15.184:6823/3387813890","heartbeat_front_addr":"172.21.15.184:6821/3387813890","state":["exists","up"]},{"osd":5,"uuid":"32d5a900-da3d-412c-bc19-13861a2368d8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6808","nonce":2680916017},{"type":"v1","addr":"172.21.15.184:6809","nonce":2680916017}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6810","nonce":2680916017},{"type":"v1","addr":"172.21.15.184:6811","nonce":2680916017}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6814","nonce":2680916017},{"type":"v1","addr":"172.21.15.184:6815","nonce":2680916017}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6812","nonce":2680916017},{"type":"v1","addr":"172.21.15.184:6813","nonce":2680916017}]},"public_addr":"172.21.15.184:6809/2680916017","cluster_addr":"172.21.15.184:6811/2680916017","heartbeat_back_addr":"172.21.15.184:6815/2680916017","heartbeat_front_addr":"172.21.15.184:6813/2680916017","state":["exists","up"]},{"osd":6,"uuid":"cde6f39e-aa1b-4260-ac19-774ee6aaca72","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6824","nonce":2183120457},{"type":"v1","addr":"172.21.15.184:6825","nonce":2183120457}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6826","nonce":2183120457},{"type":"v1","addr":"172.21.15.184:6827","nonce":2183120457}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6830","nonce":2183120457},{"type":"v1","addr":"172.21.15.184:6831","nonce":2183120457}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6828","nonce":2183120457},{"type":"v1","addr":"172.21.15.184:6829","nonce":2183120457}]},"public_addr":"172.21.15.184:6825/2183120457","cluster_addr":"172.21.15.184:6827/2183120457","heartbeat_back_addr":"172.21.15.184:6831/2183120457","heartbeat_front_addr":"172.21.15.184:6829/2183120457","state":["exists","up"]},{"osd":7,"uuid":"4f381778-eee1-47c6-804f-df0e6b33e313","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6800","nonce":4102957285},{"type":"v1","addr":"172.21.15.184:6801","nonce":4102957285}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6802","nonce":4102957285},{"type":"v1","addr":"172.21.15.184:6803","nonce":4102957285}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6806","nonce":4102957285},{"type":"v1","addr":"172.21.15.184:6807","nonce":4102957285}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6804","nonce":4102957285},{"type":"v1","addr":"172.21.15.184:6805","nonce":4102957285}]},"public_addr":"172.21.15.184:6801/4102957285","cluster_addr":"172.21.15.184:6803/4102957285","heartbeat_back_addr":"172.21.15.184:6807/4102957285","heartbeat_front_addr":"172.21.15.184:6805/4102957285","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:35.972210+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.222007+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.198617+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.385663+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.358935+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.156066+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.752973+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:35.682846+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T15:42:56.792 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:56.790+0000 7f8c8593d640 1 -- 172.21.15.123:0/3535028721 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f8c60072ec0 msgr2=0x7f8c60075380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:42:56.792 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:56.790+0000 7f8c8593d640 1 --2- 172.21.15.123:0/3535028721 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f8c60072ec0 0x7f8c60075380 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f8c70000f70 tx=0x7f8c70009040 comp rx=0 tx=0).stop 2024-09-16T15:42:56.792 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:56.790+0000 7f8c8593d640 1 -- 172.21.15.123:0/3535028721 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8c80158120 msgr2=0x7f8c8014dce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:42:56.792 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:56.790+0000 7f8c8593d640 1 --2- 172.21.15.123:0/3535028721 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8c80158120 0x7f8c8014dce0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f8c68009790 tx=0x7f8c680a86e0 comp rx=0 tx=0).stop 2024-09-16T15:42:56.792 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:56.790+0000 7f8c8593d640 1 -- 172.21.15.123:0/3535028721 shutdown_connections 2024-09-16T15:42:56.792 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:56.790+0000 7f8c8593d640 1 --2- 172.21.15.123:0/3535028721 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f8c60072ec0 0x7f8c60075380 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:42:56.792 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:56.790+0000 7f8c8593d640 1 --2- 172.21.15.123:0/3535028721 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f8c8015ec40 0x7f8c80153b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:42:56.792 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:56.790+0000 7f8c8593d640 1 --2- 172.21.15.123:0/3535028721 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8c80158120 0x7f8c8014dce0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:42:56.792 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:56.790+0000 7f8c8593d640 1 --2- 172.21.15.123:0/3535028721 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f8c800cf5a0 0x7f8c8014e500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:42:56.792 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:56.790+0000 7f8c8593d640 1 -- 172.21.15.123:0/3535028721 >> 172.21.15.123:0/3535028721 conn(0x7f8c800cfb10 msgr2=0x7f8c800b0e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:42:56.792 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:56.790+0000 7f8c8593d640 1 -- 172.21.15.123:0/3535028721 shutdown_connections 2024-09-16T15:42:56.792 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:56.790+0000 7f8c8593d640 1 -- 172.21.15.123:0/3535028721 wait complete. 2024-09-16T15:42:56.807 INFO:tasks.ceph.ceph_manager.ceph:all up! 2024-09-16T15:42:56.808 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-16T15:42:56.919 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:56.918+0000 7fc75c318640 1 -- 172.21.15.123:0/3100511305 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc75415e160 msgr2=0x7fc754073f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:42:56.919 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:56.918+0000 7fc75c318640 1 --2- 172.21.15.123:0/3100511305 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc75415e160 0x7fc754073f30 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fc7540cf1f0 tx=0x7fc74401c6e0 comp rx=0 tx=0).stop 2024-09-16T15:42:56.919 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:56.918+0000 7fc75c318640 1 -- 172.21.15.123:0/3100511305 shutdown_connections 2024-09-16T15:42:56.919 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:56.918+0000 7fc75c318640 1 --2- 172.21.15.123:0/3100511305 >> v2:172.21.15.184:3300/0 conn(0x7fc754074470 0x7fc7540cd6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:42:56.919 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:56.918+0000 7fc75c318640 1 --2- 172.21.15.123:0/3100511305 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc75415e160 0x7fc754073f30 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:42:56.919 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:56.918+0000 7fc75c318640 1 -- 172.21.15.123:0/3100511305 >> 172.21.15.123:0/3100511305 conn(0x7fc7540cfb10 msgr2=0x7fc7540cff20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:42:56.919 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:56.918+0000 7fc75c318640 1 -- 172.21.15.123:0/3100511305 shutdown_connections 2024-09-16T15:42:56.919 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:56.918+0000 7fc75c318640 1 -- 172.21.15.123:0/3100511305 wait complete. 2024-09-16T15:42:56.920 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:56.918+0000 7fc75c318640 1 Processor -- start 2024-09-16T15:42:56.920 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:56.918+0000 7fc75c318640 1 -- start start 2024-09-16T15:42:56.922 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:56.918+0000 7fc75c318640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc754074470 0x7fc754155920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:42:56.922 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:56.918+0000 7fc75c318640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc7540cdc20 0x7fc7541552f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:42:56.922 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:56.918+0000 7fc75c318640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fc75415e160 0x7fc7540c45a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:42:56.922 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:56.918+0000 7fc75c318640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fc754167dd0 con 0x7fc7540cdc20 2024-09-16T15:42:56.922 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:56.918+0000 7fc75c318640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fc7540cf620 con 0x7fc754074470 2024-09-16T15:42:56.922 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:56.918+0000 7fc75c318640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fc754167f50 con 0x7fc75415e160 2024-09-16T15:42:56.922 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:56.918+0000 7fc7598b3640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc7540cdc20 0x7fc7541552f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:42:56.922 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:56.918+0000 7fc7598b3640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc7540cdc20 0x7fc7541552f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:40250/0 (socket says 172.21.15.123:40250) 2024-09-16T15:42:56.922 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:56.918+0000 7fc7598b3640 1 -- 172.21.15.123:0/11582298 learned_addr learned my addr 172.21.15.123:0/11582298 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:42:56.922 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:56.918+0000 7fc7598b3640 1 -- 172.21.15.123:0/11582298 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fc75415e160 msgr2=0x7fc7540c45a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:42:56.922 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:56.918+0000 7fc7598b3640 1 --2- 172.21.15.123:0/11582298 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fc75415e160 0x7fc7540c45a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:42:56.923 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:56.918+0000 7fc7598b3640 1 -- 172.21.15.123:0/11582298 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc754074470 msgr2=0x7fc754155920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:42:56.923 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:56.918+0000 7fc7598b3640 1 --2- 172.21.15.123:0/11582298 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc754074470 0x7fc754155920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:42:56.923 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:56.918+0000 7fc7598b3640 1 -- 172.21.15.123:0/11582298 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc75415cab0 con 0x7fc7540cdc20 2024-09-16T15:42:56.923 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:56.918+0000 7fc7598b3640 1 --2- 172.21.15.123:0/11582298 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc7540cdc20 0x7fc7541552f0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fc748009fd0 tx=0x7fc74800c270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:42:56.923 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:56.918+0000 7fc752ffd640 1 -- 172.21.15.123:0/11582298 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc748011470 con 0x7fc7540cdc20 2024-09-16T15:42:56.923 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:56.918+0000 7fc752ffd640 1 -- 172.21.15.123:0/11582298 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fc748011700 con 0x7fc7540cdc20 2024-09-16T15:42:56.923 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:56.918+0000 7fc75c318640 1 -- 172.21.15.123:0/11582298 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc75415cd90 con 0x7fc7540cdc20 2024-09-16T15:42:56.923 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:56.918+0000 7fc752ffd640 1 -- 172.21.15.123:0/11582298 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc748011b30 con 0x7fc7540cdc20 2024-09-16T15:42:56.923 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:56.918+0000 7fc75c318640 1 -- 172.21.15.123:0/11582298 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc75415d350 con 0x7fc7540cdc20 2024-09-16T15:42:56.924 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:56.922+0000 7fc75c318640 1 -- 172.21.15.123:0/11582298 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc7540cd6e0 con 0x7fc7540cdc20 2024-09-16T15:42:56.924 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:56.922+0000 7fc752ffd640 1 -- 172.21.15.123:0/11582298 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 5) ==== 97080+0+0 (secure 0 0 0) 0x7fc748011cd0 con 0x7fc7540cdc20 2024-09-16T15:42:56.924 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:56.922+0000 7fc752ffd640 1 --2- 172.21.15.123:0/11582298 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fc740072ec0 0x7fc740075380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:42:56.924 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:56.922+0000 7fc752ffd640 1 -- 172.21.15.123:0/11582298 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7fc748094b30 con 0x7fc7540cdc20 2024-09-16T15:42:56.924 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:56.922+0000 7fc75a0b4640 1 --2- 172.21.15.123:0/11582298 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fc740072ec0 0x7fc740075380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:42:56.925 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:56.922+0000 7fc75a0b4640 1 --2- 172.21.15.123:0/11582298 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fc740072ec0 0x7fc740075380 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fc7440af000 tx=0x7fc744019480 comp rx=0 tx=0).ready entity=mgr.4135 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:42:56.927 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:56.926+0000 7fc752ffd640 1 -- 172.21.15.123:0/11582298 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7fc748016030 con 0x7fc7540cdc20 2024-09-16T15:42:57.095 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.094+0000 7fc75c318640 1 -- 172.21.15.123:0/11582298 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fc7540d06c0 con 0x7fc7540cdc20 2024-09-16T15:42:57.097 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.094+0000 7fc752ffd640 1 -- 172.21.15.123:0/11582298 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) ==== 74+0+16725 (secure 0 0 0) 0x7fc74805fd80 con 0x7fc7540cdc20 2024-09-16T15:42:57.098 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-09-16T15:42:57.098 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":21,"fsid":"56077f89-18c8-486f-ae7e-eecc0bb7822c","created":"2024-09-16T15:42:26.364892+0000","modified":"2024-09-16T15:42:52.401147+0000","last_up_change":"2024-09-16T15:42:38.707234+0000","last_in_change":"2024-09-16T15:42:29.423662+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-16T15:42:39.287764+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-09-16T15:42:48.583563+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-09-16T15:42:49.704665+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9e7c1955-a607-4671-9be7-f2ef760ceeaa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6800","nonce":3882044969},{"type":"v1","addr":"172.21.15.123:6801","nonce":3882044969}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6802","nonce":3882044969},{"type":"v1","addr":"172.21.15.123:6803","nonce":3882044969}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6806","nonce":3882044969},{"type":"v1","addr":"172.21.15.123:6807","nonce":3882044969}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6804","nonce":3882044969},{"type":"v1","addr":"172.21.15.123:6805","nonce":3882044969}]},"public_addr":"172.21.15.123:6801/3882044969","cluster_addr":"172.21.15.123:6803/3882044969","heartbeat_back_addr":"172.21.15.123:6807/3882044969","heartbeat_front_addr":"172.21.15.123:6805/3882044969","state":["exists","up"]},{"osd":1,"uuid":"c540cfde-0977-4742-9b17-fd634668e010","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6816","nonce":4037001850},{"type":"v1","addr":"172.21.15.123:6817","nonce":4037001850}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6818","nonce":4037001850},{"type":"v1","addr":"172.21.15.123:6819","nonce":4037001850}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6822","nonce":4037001850},{"type":"v1","addr":"172.21.15.123:6823","nonce":4037001850}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6820","nonce":4037001850},{"type":"v1","addr":"172.21.15.123:6821","nonce":4037001850}]},"public_addr":"172.21.15.123:6817/4037001850","cluster_addr":"172.21.15.123:6819/4037001850","heartbeat_back_addr":"172.21.15.123:6823/4037001850","heartbeat_front_addr":"172.21.15.123:6821/4037001850","state":["exists","up"]},{"osd":2,"uuid":"9864a496-f169-486b-95c4-1d59162793a7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6824","nonce":1332143088},{"type":"v1","addr":"172.21.15.123:6825","nonce":1332143088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6826","nonce":1332143088},{"type":"v1","addr":"172.21.15.123:6827","nonce":1332143088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6830","nonce":1332143088},{"type":"v1","addr":"172.21.15.123:6831","nonce":1332143088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6828","nonce":1332143088},{"type":"v1","addr":"172.21.15.123:6829","nonce":1332143088}]},"public_addr":"172.21.15.123:6825/1332143088","cluster_addr":"172.21.15.123:6827/1332143088","heartbeat_back_addr":"172.21.15.123:6831/1332143088","heartbeat_front_addr":"172.21.15.123:6829/1332143088","state":["exists","up"]},{"osd":3,"uuid":"c6d1f817-f074-4325-95e1-578eba13edda","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6808","nonce":988586998},{"type":"v1","addr":"172.21.15.123:6809","nonce":988586998}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6810","nonce":988586998},{"type":"v1","addr":"172.21.15.123:6811","nonce":988586998}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6814","nonce":988586998},{"type":"v1","addr":"172.21.15.123:6815","nonce":988586998}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6812","nonce":988586998},{"type":"v1","addr":"172.21.15.123:6813","nonce":988586998}]},"public_addr":"172.21.15.123:6809/988586998","cluster_addr":"172.21.15.123:6811/988586998","heartbeat_back_addr":"172.21.15.123:6815/988586998","heartbeat_front_addr":"172.21.15.123:6813/988586998","state":["exists","up"]},{"osd":4,"uuid":"200a8df4-50ef-40a5-b48a-58981269c086","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6816","nonce":3387813890},{"type":"v1","addr":"172.21.15.184:6817","nonce":3387813890}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6818","nonce":3387813890},{"type":"v1","addr":"172.21.15.184:6819","nonce":3387813890}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6822","nonce":3387813890},{"type":"v1","addr":"172.21.15.184:6823","nonce":3387813890}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6820","nonce":3387813890},{"type":"v1","addr":"172.21.15.184:6821","nonce":3387813890}]},"public_addr":"172.21.15.184:6817/3387813890","cluster_addr":"172.21.15.184:6819/3387813890","heartbeat_back_addr":"172.21.15.184:6823/3387813890","heartbeat_front_addr":"172.21.15.184:6821/3387813890","state":["exists","up"]},{"osd":5,"uuid":"32d5a900-da3d-412c-bc19-13861a2368d8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6808","nonce":2680916017},{"type":"v1","addr":"172.21.15.184:6809","nonce":2680916017}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6810","nonce":2680916017},{"type":"v1","addr":"172.21.15.184:6811","nonce":2680916017}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6814","nonce":2680916017},{"type":"v1","addr":"172.21.15.184:6815","nonce":2680916017}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6812","nonce":2680916017},{"type":"v1","addr":"172.21.15.184:6813","nonce":2680916017}]},"public_addr":"172.21.15.184:6809/2680916017","cluster_addr":"172.21.15.184:6811/2680916017","heartbeat_back_addr":"172.21.15.184:6815/2680916017","heartbeat_front_addr":"172.21.15.184:6813/2680916017","state":["exists","up"]},{"osd":6,"uuid":"cde6f39e-aa1b-4260-ac19-774ee6aaca72","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6824","nonce":2183120457},{"type":"v1","addr":"172.21.15.184:6825","nonce":2183120457}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6826","nonce":2183120457},{"type":"v1","addr":"172.21.15.184:6827","nonce":2183120457}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6830","nonce":2183120457},{"type":"v1","addr":"172.21.15.184:6831","nonce":2183120457}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6828","nonce":2183120457},{"type":"v1","addr":"172.21.15.184:6829","nonce":2183120457}]},"public_addr":"172.21.15.184:6825/2183120457","cluster_addr":"172.21.15.184:6827/2183120457","heartbeat_back_addr":"172.21.15.184:6831/2183120457","heartbeat_front_addr":"172.21.15.184:6829/2183120457","state":["exists","up"]},{"osd":7,"uuid":"4f381778-eee1-47c6-804f-df0e6b33e313","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6800","nonce":4102957285},{"type":"v1","addr":"172.21.15.184:6801","nonce":4102957285}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6802","nonce":4102957285},{"type":"v1","addr":"172.21.15.184:6803","nonce":4102957285}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6806","nonce":4102957285},{"type":"v1","addr":"172.21.15.184:6807","nonce":4102957285}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6804","nonce":4102957285},{"type":"v1","addr":"172.21.15.184:6805","nonce":4102957285}]},"public_addr":"172.21.15.184:6801/4102957285","cluster_addr":"172.21.15.184:6803/4102957285","heartbeat_back_addr":"172.21.15.184:6807/4102957285","heartbeat_front_addr":"172.21.15.184:6805/4102957285","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:35.972210+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.222007+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.198617+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.385663+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.358935+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.156066+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.752973+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:35.682846+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T15:42:57.100 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.098+0000 7fc75c318640 1 -- 172.21.15.123:0/11582298 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fc740072ec0 msgr2=0x7fc740075380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:42:57.100 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.098+0000 7fc75c318640 1 --2- 172.21.15.123:0/11582298 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fc740072ec0 0x7fc740075380 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fc7440af000 tx=0x7fc744019480 comp rx=0 tx=0).stop 2024-09-16T15:42:57.100 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.098+0000 7fc75c318640 1 -- 172.21.15.123:0/11582298 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc7540cdc20 msgr2=0x7fc7541552f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:42:57.100 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.098+0000 7fc75c318640 1 --2- 172.21.15.123:0/11582298 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc7540cdc20 0x7fc7541552f0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fc748009fd0 tx=0x7fc74800c270 comp rx=0 tx=0).stop 2024-09-16T15:42:57.100 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.098+0000 7fc75c318640 1 -- 172.21.15.123:0/11582298 shutdown_connections 2024-09-16T15:42:57.100 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.098+0000 7fc75c318640 1 --2- 172.21.15.123:0/11582298 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fc740072ec0 0x7fc740075380 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:42:57.100 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.098+0000 7fc75c318640 1 --2- 172.21.15.123:0/11582298 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fc75415e160 0x7fc7540c45a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:42:57.100 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.098+0000 7fc75c318640 1 --2- 172.21.15.123:0/11582298 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc7540cdc20 0x7fc7541552f0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:42:57.100 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.098+0000 7fc75c318640 1 --2- 172.21.15.123:0/11582298 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc754074470 0x7fc754155920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:42:57.100 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.098+0000 7fc75c318640 1 -- 172.21.15.123:0/11582298 >> 172.21.15.123:0/11582298 conn(0x7fc7540cfb10 msgr2=0x7fc7540c5130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:42:57.101 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.098+0000 7fc75c318640 1 -- 172.21.15.123:0/11582298 shutdown_connections 2024-09-16T15:42:57.101 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.098+0000 7fc75c318640 1 -- 172.21.15.123:0/11582298 wait complete. 2024-09-16T15:42:57.116 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2024-09-16T15:42:57.116 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2024-09-16T15:42:57.117 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2024-09-16T15:42:57.117 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2024-09-16T15:42:57.117 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 flush_pg_stats 2024-09-16T15:42:57.118 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 flush_pg_stats 2024-09-16T15:42:57.118 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 flush_pg_stats 2024-09-16T15:42:57.119 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 flush_pg_stats 2024-09-16T15:42:57.301 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.298+0000 7f254359e640 1 -- 172.21.15.123:0/1176860194 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f253c0377e0 msgr2=0x7f253c008300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:42:57.301 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.298+0000 7f254359e640 1 --2- 172.21.15.123:0/1176860194 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f253c0377e0 0x7f253c008300 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f254406bc60 tx=0x7f2544098450 comp rx=0 tx=0).stop 2024-09-16T15:42:57.302 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.298+0000 7f40acafc640 1 -- 172.21.15.123:0/349853835 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f40a00e6e60 msgr2=0x7f40a00530c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:42:57.302 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.298+0000 7f40acafc640 1 --2- 172.21.15.123:0/349853835 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f40a00e6e60 0x7f40a00530c0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f409c009900 tx=0x7f409c01c8b0 comp rx=0 tx=0).stop 2024-09-16T15:42:57.302 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.298+0000 7f40acafc640 1 -- 172.21.15.123:0/349853835 shutdown_connections 2024-09-16T15:42:57.302 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.298+0000 7f40acafc640 1 --2- 172.21.15.123:0/349853835 >> v2:172.21.15.184:3300/0 conn(0x7f40a0006d30 0x7f40a00071a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:42:57.302 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.298+0000 7f40acafc640 1 --2- 172.21.15.123:0/349853835 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f40a00e6e60 0x7f40a00530c0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:42:57.302 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.298+0000 7f40acafc640 1 -- 172.21.15.123:0/349853835 >> 172.21.15.123:0/349853835 conn(0x7f40a003a3e0 msgr2=0x7f40a003a7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:42:57.302 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.298+0000 7f40acafc640 1 -- 172.21.15.123:0/349853835 shutdown_connections 2024-09-16T15:42:57.303 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.298+0000 7f254359e640 1 -- 172.21.15.123:0/1176860194 shutdown_connections 2024-09-16T15:42:57.303 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.298+0000 7f254359e640 1 --2- 172.21.15.123:0/1176860194 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f253c0377e0 0x7f253c008300 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:42:57.303 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.298+0000 7f254359e640 1 --2- 172.21.15.123:0/1176860194 >> v2:172.21.15.184:3300/0 conn(0x7f253c0e2c90 0x7f253c036a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:42:57.303 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.298+0000 7f254359e640 1 -- 172.21.15.123:0/1176860194 >> 172.21.15.123:0/1176860194 conn(0x7f253c01e8a0 msgr2=0x7f253c055650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:42:57.303 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.298+0000 7f2bcb083640 1 -- 172.21.15.123:0/3416343872 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2bc40cc690 msgr2=0x7f2bc406e4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:42:57.303 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.298+0000 7f2bcb083640 1 --2- 172.21.15.123:0/3416343872 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2bc40cc690 0x7f2bc406e4a0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f2bc40b22a0 tx=0x7f2bb401c790 comp rx=0 tx=0).stop 2024-09-16T15:42:57.304 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.298+0000 7f2bcb083640 1 -- 172.21.15.123:0/3416343872 shutdown_connections 2024-09-16T15:42:57.304 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.298+0000 7f2bcb083640 1 --2- 172.21.15.123:0/3416343872 >> v2:172.21.15.184:3300/0 conn(0x7f2bc40ca150 0x7f2bc406e9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:42:57.304 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.298+0000 7f2bcb083640 1 --2- 172.21.15.123:0/3416343872 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2bc40cc690 0x7f2bc406e4a0 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:42:57.304 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.298+0000 7f2bcb083640 1 -- 172.21.15.123:0/3416343872 >> 172.21.15.123:0/3416343872 conn(0x7f2bc40b1550 msgr2=0x7f2bc40b1960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:42:57.304 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.298+0000 7f254359e640 1 -- 172.21.15.123:0/1176860194 shutdown_connections 2024-09-16T15:42:57.304 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.302+0000 7f254359e640 1 -- 172.21.15.123:0/1176860194 wait complete. 2024-09-16T15:42:57.305 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.298+0000 7f40acafc640 1 -- 172.21.15.123:0/349853835 wait complete. 2024-09-16T15:42:57.305 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.302+0000 7f40acafc640 1 Processor -- start 2024-09-16T15:42:57.305 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.302+0000 7f254359e640 1 Processor -- start 2024-09-16T15:42:57.305 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.302+0000 7f254359e640 1 -- start start 2024-09-16T15:42:57.306 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.302+0000 7f254359e640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f253c034910 0x7f253c0330e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:42:57.306 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.302+0000 7f254359e640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f253c0377e0 0x7f253c1836f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:42:57.306 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.302+0000 7f254359e640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f253c0e2c90 0x7f253c18d5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:42:57.306 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.302+0000 7f254359e640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f253c0ecb00 con 0x7f253c0377e0 2024-09-16T15:42:57.306 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.302+0000 7f254359e640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f253c0ec980 con 0x7f253c034910 2024-09-16T15:42:57.306 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.302+0000 7f254359e640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f253c0ecc80 con 0x7f253c0e2c90 2024-09-16T15:42:57.306 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.302+0000 7f40acafc640 1 -- start start 2024-09-16T15:42:57.306 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.302+0000 7f40acafc640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f40a0006d30 0x7f40a00486f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:42:57.307 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.302+0000 7f40acafc640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f40a0037e20 0x7f40a00da210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:42:57.307 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.302+0000 7f40acafc640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f40a00e6e60 0x7f40a000a690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:42:57.307 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.302+0000 7f40acafc640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f40a00f3a40 con 0x7f40a0037e20 2024-09-16T15:42:57.307 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.302+0000 7f40acafc640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f40a00f38c0 con 0x7f40a00e6e60 2024-09-16T15:42:57.307 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.302+0000 7f40acafc640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f40a00f3bc0 con 0x7f40a0006d30 2024-09-16T15:42:57.307 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.302+0000 7f40a5d9b640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f40a0037e20 0x7f40a00da210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:42:57.307 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.302+0000 7f40a5d9b640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f40a0037e20 0x7f40a00da210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:40298/0 (socket says 172.21.15.123:40298) 2024-09-16T15:42:57.307 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.302+0000 7f40a5d9b640 1 -- 172.21.15.123:0/392369015 learned_addr learned my addr 172.21.15.123:0/392369015 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:42:57.307 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.302+0000 7f40a6d9d640 1 --2- 172.21.15.123:0/392369015 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f40a00e6e60 0x7f40a000a690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:42:57.307 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.302+0000 7f40a659c640 1 --2- 172.21.15.123:0/392369015 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f40a0006d30 0x7f40a00486f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:42:57.308 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.302+0000 7f2bcb083640 1 -- 172.21.15.123:0/3416343872 shutdown_connections 2024-09-16T15:42:57.308 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.302+0000 7f2bcb083640 1 -- 172.21.15.123:0/3416343872 wait complete. 2024-09-16T15:42:57.308 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.302+0000 7f2bcb083640 1 Processor -- start 2024-09-16T15:42:57.308 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.302+0000 7f9f33403640 1 -- 172.21.15.123:0/828047649 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f9f2c159b90 msgr2=0x7f9f2c0cd5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:42:57.309 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.302+0000 7f9f33403640 1 --2- 172.21.15.123:0/828047649 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f9f2c159b90 0x7f9f2c0cd5f0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f9f1c00afd0 tx=0x7f9f1c01d250 comp rx=0 tx=0).stop 2024-09-16T15:42:57.309 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.302+0000 7fb7418ad640 1 -- 172.21.15.123:0/1372372867 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb73c1629a0 msgr2=0x7fb73c15db50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:42:57.309 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.302+0000 7fb7418ad640 1 --2- 172.21.15.123:0/1372372867 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb73c1629a0 0x7fb73c15db50 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fb73c06fe00 tx=0x7fb72c01c890 comp rx=0 tx=0).stop 2024-09-16T15:42:57.309 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.302+0000 7fb7418ad640 1 -- 172.21.15.123:0/1372372867 shutdown_connections 2024-09-16T15:42:57.309 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.302+0000 7fb7418ad640 1 --2- 172.21.15.123:0/1372372867 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fb73c161a80 0x7fb73c0cf850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:42:57.309 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.302+0000 7fb7418ad640 1 --2- 172.21.15.123:0/1372372867 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb73c1629a0 0x7fb73c15db50 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:42:57.309 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.302+0000 7fb7418ad640 1 --2- 172.21.15.123:0/1372372867 >> v2:172.21.15.123:3300/0 conn(0x7fb73c0b5480 0x7fb73c0b6d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:42:57.309 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.302+0000 7fb7418ad640 1 -- 172.21.15.123:0/1372372867 >> 172.21.15.123:0/1372372867 conn(0x7fb73c09d200 msgr2=0x7fb73c0d3f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:42:57.310 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.302+0000 7fb7418ad640 1 -- 172.21.15.123:0/1372372867 shutdown_connections 2024-09-16T15:42:57.310 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.302+0000 7fb7418ad640 1 -- 172.21.15.123:0/1372372867 wait complete. 2024-09-16T15:42:57.310 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.302+0000 7f40a659c640 1 -- 172.21.15.123:0/392369015 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f40a00e6e60 msgr2=0x7f40a000a690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:42:57.310 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.302+0000 7f40a659c640 1 --2- 172.21.15.123:0/392369015 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f40a00e6e60 0x7f40a000a690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:42:57.310 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.302+0000 7f40a659c640 1 -- 172.21.15.123:0/392369015 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f40a0037e20 msgr2=0x7f40a00da210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:42:57.310 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.302+0000 7f40a659c640 1 --2- 172.21.15.123:0/392369015 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f40a0037e20 0x7f40a00da210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:42:57.310 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.302+0000 7f40a659c640 1 -- 172.21.15.123:0/392369015 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f40a000b150 con 0x7f40a0006d30 2024-09-16T15:42:57.311 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.302+0000 7f2541d9b640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f253c0377e0 0x7f253c1836f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:42:57.311 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.302+0000 7f2541d9b640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f253c0377e0 0x7f253c1836f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:40306/0 (socket says 172.21.15.123:40306) 2024-09-16T15:42:57.311 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.302+0000 7f2541d9b640 1 -- 172.21.15.123:0/521860267 learned_addr learned my addr 172.21.15.123:0/521860267 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:42:57.311 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.302+0000 7f2542d9d640 1 --2- 172.21.15.123:0/521860267 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f253c0e2c90 0x7f253c18d5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:42:57.311 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.302+0000 7f254259c640 1 --2- 172.21.15.123:0/521860267 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f253c034910 0x7f253c0330e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:42:57.311 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.302+0000 7f2541d9b640 1 -- 172.21.15.123:0/521860267 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f253c0e2c90 msgr2=0x7f253c18d5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:42:57.311 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.302+0000 7f2541d9b640 1 --2- 172.21.15.123:0/521860267 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f253c0e2c90 0x7f253c18d5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:42:57.311 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.302+0000 7f2541d9b640 1 -- 172.21.15.123:0/521860267 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f253c034910 msgr2=0x7f253c0330e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:42:57.311 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.302+0000 7f2541d9b640 1 --2- 172.21.15.123:0/521860267 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f253c034910 0x7f253c0330e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:42:57.312 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.302+0000 7f2541d9b640 1 -- 172.21.15.123:0/521860267 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f253c18db20 con 0x7f253c0377e0 2024-09-16T15:42:57.312 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.302+0000 7f2541d9b640 1 --2- 172.21.15.123:0/521860267 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f253c0377e0 0x7f253c1836f0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f253800c480 tx=0x7f253800c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:42:57.312 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.302+0000 7f2532ffd640 1 -- 172.21.15.123:0/521860267 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25380052d0 con 0x7f253c0377e0 2024-09-16T15:42:57.312 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.302+0000 7f2532ffd640 1 -- 172.21.15.123:0/521860267 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f2538005470 con 0x7f253c0377e0 2024-09-16T15:42:57.312 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.302+0000 7f254359e640 1 -- 172.21.15.123:0/521860267 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f253c0336c0 con 0x7f253c0377e0 2024-09-16T15:42:57.312 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.302+0000 7f2532ffd640 1 -- 172.21.15.123:0/521860267 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2538012890 con 0x7f253c0377e0 2024-09-16T15:42:57.312 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.302+0000 7f254359e640 1 -- 172.21.15.123:0/521860267 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f253c033c80 con 0x7f253c0377e0 2024-09-16T15:42:57.313 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.302+0000 7fb7418ad640 1 Processor -- start 2024-09-16T15:42:57.313 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.306+0000 7fb7418ad640 1 -- start start 2024-09-16T15:42:57.313 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.302+0000 7f2bcb083640 1 -- start start 2024-09-16T15:42:57.313 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.306+0000 7f2bcb083640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2bc40ca150 0x7f2bc40c4d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:42:57.313 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.306+0000 7f2bcb083640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2bc40cc690 0x7f2bc416ac50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:42:57.313 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.306+0000 7f2bcb083640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f2bc40cca70 0x7f2bc40b0240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:42:57.313 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.306+0000 7f2bcb083640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f2bc41806d0 con 0x7f2bc40ca150 2024-09-16T15:42:57.313 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.306+0000 7f2bcb083640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f2bc4180550 con 0x7f2bc40cc690 2024-09-16T15:42:57.314 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.306+0000 7f2bcb083640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f2bc4180850 con 0x7f2bc40cca70 2024-09-16T15:42:57.314 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.302+0000 7f9f33403640 1 -- 172.21.15.123:0/828047649 shutdown_connections 2024-09-16T15:42:57.314 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.302+0000 7f9f33403640 1 --2- 172.21.15.123:0/828047649 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f9f2c159b90 0x7f9f2c0cd5f0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:42:57.314 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.302+0000 7f9f33403640 1 -- 172.21.15.123:0/828047649 >> 172.21.15.123:0/828047649 conn(0x7f9f2c0b1550 msgr2=0x7f9f2c0b1960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:42:57.314 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.306+0000 7f2bca882640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f2bc40cca70 0x7f2bc40b0240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:42:57.314 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.306+0000 7f2bca882640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f2bc40cca70 0x7f2bc40b0240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3301/0 says I am v2:172.21.15.123:49222/0 (socket says 172.21.15.123:49222) 2024-09-16T15:42:57.315 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.306+0000 7f2bca882640 1 -- 172.21.15.123:0/1712634364 learned_addr learned my addr 172.21.15.123:0/1712634364 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:42:57.315 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.306+0000 7f2bc9880640 1 --2- 172.21.15.123:0/1712634364 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2bc40cc690 0x7f2bc416ac50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:42:57.315 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.306+0000 7f2bca081640 1 --2- 172.21.15.123:0/1712634364 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2bc40ca150 0x7f2bc40c4d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:42:57.315 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.306+0000 7f9f33403640 1 -- 172.21.15.123:0/828047649 shutdown_connections 2024-09-16T15:42:57.315 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.306+0000 7f9f33403640 1 -- 172.21.15.123:0/828047649 wait complete. 2024-09-16T15:42:57.315 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.306+0000 7f2bca081640 1 -- 172.21.15.123:0/1712634364 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f2bc40cca70 msgr2=0x7f2bc40b0240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:42:57.315 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.306+0000 7f2bca081640 1 --2- 172.21.15.123:0/1712634364 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f2bc40cca70 0x7f2bc40b0240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:42:57.316 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.306+0000 7f2bca081640 1 -- 172.21.15.123:0/1712634364 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2bc40cc690 msgr2=0x7f2bc416ac50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:42:57.316 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.306+0000 7f2bca081640 1 --2- 172.21.15.123:0/1712634364 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2bc40cc690 0x7f2bc416ac50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:42:57.316 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.306+0000 7f2bca081640 1 -- 172.21.15.123:0/1712634364 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2bc40c84f0 con 0x7f2bc40ca150 2024-09-16T15:42:57.316 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.306+0000 7f254359e640 1 -- 172.21.15.123:0/521860267 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f253c008300 con 0x7f253c0377e0 2024-09-16T15:42:57.316 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.306+0000 7f2bca081640 1 --2- 172.21.15.123:0/1712634364 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2bc40ca150 0x7f2bc40c4d10 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f2bb40ac000 tx=0x7f2bb40a6a80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:42:57.317 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.306+0000 7f40a659c640 1 --2- 172.21.15.123:0/392369015 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f40a0006d30 0x7f40a00486f0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f409c0ac000 tx=0x7f409c005d60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:42:57.317 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.306+0000 7f2bbaffd640 1 -- 172.21.15.123:0/1712634364 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2bb4004230 con 0x7f2bc40ca150 2024-09-16T15:42:57.317 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.306+0000 7f2bbaffd640 1 -- 172.21.15.123:0/1712634364 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f2bb40043d0 con 0x7f2bc40ca150 2024-09-16T15:42:57.317 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.306+0000 7f2bbaffd640 1 -- 172.21.15.123:0/1712634364 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2bb40052d0 con 0x7f2bc40ca150 2024-09-16T15:42:57.318 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.306+0000 7f4096ffd640 1 -- 172.21.15.123:0/392369015 <== mon.2 v2:172.21.15.184:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f409c0b2040 con 0x7f40a0006d30 2024-09-16T15:42:57.318 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.306+0000 7f4096ffd640 1 -- 172.21.15.123:0/392369015 <== mon.2 v2:172.21.15.184:3301/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f409c003910 con 0x7f40a0006d30 2024-09-16T15:42:57.318 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.306+0000 7f40acafc640 1 -- 172.21.15.123:0/392369015 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f40a000b430 con 0x7f40a0006d30 2024-09-16T15:42:57.318 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.306+0000 7f40acafc640 1 -- 172.21.15.123:0/392369015 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f40a000b9f0 con 0x7f40a0006d30 2024-09-16T15:42:57.318 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.306+0000 7f4096ffd640 1 -- 172.21.15.123:0/392369015 <== mon.2 v2:172.21.15.184:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f409c003d60 con 0x7f40a0006d30 2024-09-16T15:42:57.318 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.306+0000 7f2bcb083640 1 -- 172.21.15.123:0/1712634364 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2bc40c87d0 con 0x7f2bc40ca150 2024-09-16T15:42:57.318 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.306+0000 7f2bcb083640 1 -- 172.21.15.123:0/1712634364 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2bc40c8d90 con 0x7f2bc40ca150 2024-09-16T15:42:57.319 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.306+0000 7f9f33403640 1 Processor -- start 2024-09-16T15:42:57.319 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.306+0000 7fb7418ad640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fb73c0b5480 0x7fb73c09e640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:42:57.319 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.306+0000 7fb7418ad640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb73c161a80 0x7fb73c0a1640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:42:57.319 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.306+0000 7fb7418ad640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb73c1629a0 0x7fb73c204d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:42:57.319 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.306+0000 7fb7408ab640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fb73c0b5480 0x7fb73c09e640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:42:57.319 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.306+0000 7fb7408ab640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fb73c0b5480 0x7fb73c09e640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3301/0 says I am v2:172.21.15.123:49226/0 (socket says 172.21.15.123:49226) 2024-09-16T15:42:57.320 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.306+0000 7fb73bfff640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb73c161a80 0x7fb73c0a1640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:42:57.320 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.306+0000 7fb73bfff640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb73c161a80 0x7fb73c0a1640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.123:53356/0 (socket says 172.21.15.123:53356) 2024-09-16T15:42:57.320 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.306+0000 7fb73bfff640 1 -- 172.21.15.123:0/2116289007 learned_addr learned my addr 172.21.15.123:0/2116289007 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:42:57.320 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.306+0000 7fb7418ad640 1 -- 172.21.15.123:0/2116289007 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fb73c0d0930 con 0x7fb73c1629a0 2024-09-16T15:42:57.320 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.306+0000 7fb7418ad640 1 -- 172.21.15.123:0/2116289007 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fb73c0d2990 con 0x7fb73c161a80 2024-09-16T15:42:57.320 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.306+0000 7fb7418ad640 1 -- 172.21.15.123:0/2116289007 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fb73c0d0ab0 con 0x7fb73c0b5480 2024-09-16T15:42:57.320 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.306+0000 7fb73bfff640 1 -- 172.21.15.123:0/2116289007 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fb73c0b5480 msgr2=0x7fb73c09e640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:42:57.320 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.306+0000 7fb73bfff640 1 --2- 172.21.15.123:0/2116289007 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fb73c0b5480 0x7fb73c09e640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:42:57.320 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.306+0000 7fb73bfff640 1 -- 172.21.15.123:0/2116289007 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb73c1629a0 msgr2=0x7fb73c204d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:42:57.320 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.306+0000 7fb73bfff640 1 --2- 172.21.15.123:0/2116289007 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb73c1629a0 0x7fb73c204d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:42:57.320 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.306+0000 7fb73bfff640 1 -- 172.21.15.123:0/2116289007 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb73c09df30 con 0x7fb73c161a80 2024-09-16T15:42:57.320 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.306+0000 7fb73bfff640 1 --2- 172.21.15.123:0/2116289007 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb73c161a80 0x7fb73c0a1640 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fb72c01af40 tx=0x7fb72c005e90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:42:57.321 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.306+0000 7f2532ffd640 1 -- 172.21.15.123:0/521860267 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 5) ==== 97080+0+0 (secure 0 0 0) 0x7f2538003600 con 0x7f253c0377e0 2024-09-16T15:42:57.321 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.306+0000 7f2bcb083640 1 -- 172.21.15.123:0/1712634364 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f2bc416e800 con 0x7f2bc40ca150 2024-09-16T15:42:57.321 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.310+0000 7f40acafc640 1 -- 172.21.15.123:0/392369015 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f40a00071a0 con 0x7f40a0006d30 2024-09-16T15:42:57.322 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.306+0000 7fd363417640 1 -- 172.21.15.123:0/3317373809 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fd35c0b2350 msgr2=0x7fd35c0d1a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:42:57.322 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.306+0000 7fd363417640 1 --2- 172.21.15.123:0/3317373809 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fd35c0b2350 0x7fd35c0d1a40 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fd358009840 tx=0x7fd35801c820 comp rx=0 tx=0).stop 2024-09-16T15:42:57.322 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.306+0000 7fd363417640 1 -- 172.21.15.123:0/3317373809 shutdown_connections 2024-09-16T15:42:57.322 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.306+0000 7fd363417640 1 --2- 172.21.15.123:0/3317373809 >> v2:172.21.15.123:3300/0 conn(0x7fd35c0d1f80 0x7fd35c0aedb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:42:57.322 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.306+0000 7fd363417640 1 --2- 172.21.15.123:0/3317373809 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fd35c0b2350 0x7fd35c0d1a40 secure :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fd358009840 tx=0x7fd35801c820 comp rx=0 tx=0).stop 2024-09-16T15:42:57.322 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.306+0000 7fd363417640 1 -- 172.21.15.123:0/3317373809 >> 172.21.15.123:0/3317373809 conn(0x7fd35c09aa40 msgr2=0x7fd35c09ae50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:42:57.322 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.306+0000 7fd363417640 1 -- 172.21.15.123:0/3317373809 shutdown_connections 2024-09-16T15:42:57.323 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.310+0000 7f2532ffd640 1 --2- 172.21.15.123:0/521860267 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f2524072ec0 0x7f2524075380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:42:57.323 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.310+0000 7f2532ffd640 1 -- 172.21.15.123:0/521860267 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f25380938c0 con 0x7f253c0377e0 2024-09-16T15:42:57.323 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.310+0000 7f2532ffd640 1 --2- 172.21.15.123:0/521860267 >> [v2:172.21.15.123:6800/3882044969,v1:172.21.15.123:6801/3882044969] conn(0x7f252407b060 0x7f252407d480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:42:57.323 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.310+0000 7f2532ffd640 1 -- 172.21.15.123:0/521860267 --> [v2:172.21.15.123:6800/3882044969,v1:172.21.15.123:6801/3882044969] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f252407db30 con 0x7f252407b060 2024-09-16T15:42:57.323 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.310+0000 7f254259c640 1 --2- 172.21.15.123:0/521860267 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f2524072ec0 0x7f2524075380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:42:57.323 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.310+0000 7f2542d9d640 1 --2- 172.21.15.123:0/521860267 >> [v2:172.21.15.123:6800/3882044969,v1:172.21.15.123:6801/3882044969] conn(0x7f252407b060 0x7f252407d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:42:57.323 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.310+0000 7f2542d9d640 1 --2- 172.21.15.123:0/521860267 >> [v2:172.21.15.123:6800/3882044969,v1:172.21.15.123:6801/3882044969] conn(0x7f252407b060 0x7f252407d480 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:42:57.324 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.310+0000 7f2bbaffd640 1 -- 172.21.15.123:0/1712634364 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 5) ==== 97080+0+0 (secure 0 0 0) 0x7f2bb40a6780 con 0x7f2bc40ca150 2024-09-16T15:42:57.324 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.310+0000 7f4096ffd640 1 -- 172.21.15.123:0/392369015 <== mon.2 v2:172.21.15.184:3301/0 4 ==== mgrmap(e 5) ==== 97080+0+0 (secure 0 0 0) 0x7f409c0b8470 con 0x7f40a0006d30 2024-09-16T15:42:57.324 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.310+0000 7f254259c640 1 --2- 172.21.15.123:0/521860267 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f2524072ec0 0x7f2524075380 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f2534009b40 tx=0x7f2534009260 comp rx=0 tx=0).ready entity=mgr.4135 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:42:57.325 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.310+0000 7fb7397fa640 1 -- 172.21.15.123:0/2116289007 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb72c0b2040 con 0x7fb73c161a80 2024-09-16T15:42:57.325 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.310+0000 7fb7397fa640 1 -- 172.21.15.123:0/2116289007 <== mon.1 v2:172.21.15.184:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fb72c004320 con 0x7fb73c161a80 2024-09-16T15:42:57.325 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.310+0000 7fb7397fa640 1 -- 172.21.15.123:0/2116289007 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb72c0044f0 con 0x7fb73c161a80 2024-09-16T15:42:57.325 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.310+0000 7fb7418ad640 1 -- 172.21.15.123:0/2116289007 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb73c204590 con 0x7fb73c161a80 2024-09-16T15:42:57.325 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.310+0000 7fb7418ad640 1 -- 172.21.15.123:0/2116289007 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb73c2052d0 con 0x7fb73c161a80 2024-09-16T15:42:57.325 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.310+0000 7f2532ffd640 1 -- 172.21.15.123:0/521860267 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f253805ab00 con 0x7f253c0377e0 2024-09-16T15:42:57.325 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.310+0000 7f2532ffd640 1 -- 172.21.15.123:0/521860267 <== osd.0 v2:172.21.15.123:6800/3882044969 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f252407db30 con 0x7f252407b060 2024-09-16T15:42:57.326 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.310+0000 7f4096ffd640 1 --2- 172.21.15.123:0/392369015 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f4088072ec0 0x7f4088075380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:42:57.326 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.310+0000 7f4096ffd640 1 -- 172.21.15.123:0/392369015 <== mon.2 v2:172.21.15.184:3301/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f409c12e660 con 0x7f40a0006d30 2024-09-16T15:42:57.326 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.310+0000 7f4096ffd640 1 --2- 172.21.15.123:0/392369015 >> [v2:172.21.15.123:6816/4037001850,v1:172.21.15.123:6817/4037001850] conn(0x7f408807b060 0x7f408807d480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:42:57.326 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.310+0000 7f4096ffd640 1 -- 172.21.15.123:0/392369015 --> [v2:172.21.15.123:6816/4037001850,v1:172.21.15.123:6817/4037001850] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f408807db30 con 0x7f408807b060 2024-09-16T15:42:57.326 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.310+0000 7f4096ffd640 1 -- 172.21.15.123:0/392369015 <== mon.2 v2:172.21.15.184:3301/0 6 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f409c12ea50 con 0x7f40a0006d30 2024-09-16T15:42:57.326 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.310+0000 7f40a6d9d640 1 --2- 172.21.15.123:0/392369015 >> [v2:172.21.15.123:6816/4037001850,v1:172.21.15.123:6817/4037001850] conn(0x7f408807b060 0x7f408807d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:42:57.326 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.310+0000 7f40a5d9b640 1 --2- 172.21.15.123:0/392369015 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f4088072ec0 0x7f4088075380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:42:57.326 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.314+0000 7f40a5d9b640 1 --2- 172.21.15.123:0/392369015 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f4088072ec0 0x7f4088075380 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f40a0048c30 tx=0x7f4098009040 comp rx=0 tx=0).ready entity=mgr.4135 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:42:57.327 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.314+0000 7f40a6d9d640 1 --2- 172.21.15.123:0/392369015 >> [v2:172.21.15.123:6816/4037001850,v1:172.21.15.123:6817/4037001850] conn(0x7f408807b060 0x7f408807d480 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:42:57.327 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.314+0000 7f4096ffd640 1 -- 172.21.15.123:0/392369015 <== osd.1 v2:172.21.15.123:6816/4037001850 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f408807db30 con 0x7f408807b060 2024-09-16T15:42:57.327 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.310+0000 7f2bbaffd640 1 --2- 172.21.15.123:0/1712634364 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f2ba8072ec0 0x7f2ba8075380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:42:57.327 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.310+0000 7f2bbaffd640 1 -- 172.21.15.123:0/1712634364 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f2bb412d9f0 con 0x7f2bc40ca150 2024-09-16T15:42:57.327 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.310+0000 7f2bbaffd640 1 --2- 172.21.15.123:0/1712634364 >> [v2:172.21.15.123:6824/1332143088,v1:172.21.15.123:6825/1332143088] conn(0x7f2ba807b060 0x7f2ba807d480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:42:57.327 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.310+0000 7f2bbaffd640 1 -- 172.21.15.123:0/1712634364 --> [v2:172.21.15.123:6824/1332143088,v1:172.21.15.123:6825/1332143088] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f2bb40beeb0 con 0x7f2ba807b060 2024-09-16T15:42:57.327 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.310+0000 7f2bc9880640 1 --2- 172.21.15.123:0/1712634364 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f2ba8072ec0 0x7f2ba8075380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:42:57.328 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.310+0000 7f2bca882640 1 --2- 172.21.15.123:0/1712634364 >> [v2:172.21.15.123:6824/1332143088,v1:172.21.15.123:6825/1332143088] conn(0x7f2ba807b060 0x7f2ba807d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:42:57.328 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.310+0000 7f2bbaffd640 1 -- 172.21.15.123:0/1712634364 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f2bb40f4cd0 con 0x7f2bc40ca150 2024-09-16T15:42:57.328 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.310+0000 7f2bca882640 1 --2- 172.21.15.123:0/1712634364 >> [v2:172.21.15.123:6824/1332143088,v1:172.21.15.123:6825/1332143088] conn(0x7f2ba807b060 0x7f2ba807d480 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:42:57.328 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.310+0000 7f2bc9880640 1 --2- 172.21.15.123:0/1712634364 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f2ba8072ec0 0x7f2ba8075380 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f2bc40c7d40 tx=0x7f2bc0011040 comp rx=0 tx=0).ready entity=mgr.4135 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:42:57.328 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.314+0000 7f2bbaffd640 1 -- 172.21.15.123:0/1712634364 <== osd.2 v2:172.21.15.123:6824/1332143088 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f2bb40beeb0 con 0x7f2ba807b060 2024-09-16T15:42:57.328 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.310+0000 7f9f33403640 1 -- start start 2024-09-16T15:42:57.328 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.310+0000 7f9f33403640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f9f2c159b90 0x7f9f2c0c9950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:42:57.328 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.310+0000 7f9f33403640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9f2c159f70 0x7f9f2c158fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:42:57.329 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.310+0000 7f9f33403640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f9f2c15e080 0x7f9f2c1f9a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:42:57.329 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.310+0000 7f9f33403640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f9f2c0cce00 con 0x7f9f2c159f70 2024-09-16T15:42:57.329 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.310+0000 7f9f33403640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f9f2c0ccc80 con 0x7f9f2c159b90 2024-09-16T15:42:57.329 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.310+0000 7f9f33403640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f9f2c16c8e0 con 0x7f9f2c15e080 2024-09-16T15:42:57.329 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.310+0000 7f9f32c02640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f9f2c15e080 0x7f9f2c1f9a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:42:57.329 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.310+0000 7f9f32c02640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f9f2c15e080 0x7f9f2c1f9a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3301/0 says I am v2:172.21.15.123:49250/0 (socket says 172.21.15.123:49250) 2024-09-16T15:42:57.329 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.310+0000 7f9f32c02640 1 -- 172.21.15.123:0/2226962612 learned_addr learned my addr 172.21.15.123:0/2226962612 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:42:57.329 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.310+0000 7f9f31c00640 1 --2- 172.21.15.123:0/2226962612 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9f2c159f70 0x7f9f2c158fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:42:57.329 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.310+0000 7f9f32401640 1 --2- 172.21.15.123:0/2226962612 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f9f2c159b90 0x7f9f2c0c9950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:42:57.329 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.310+0000 7f9f32401640 1 -- 172.21.15.123:0/2226962612 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f9f2c15e080 msgr2=0x7f9f2c1f9a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:42:57.329 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.310+0000 7f9f32401640 1 --2- 172.21.15.123:0/2226962612 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f9f2c15e080 0x7f9f2c1f9a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:42:57.330 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.310+0000 7f9f32401640 1 -- 172.21.15.123:0/2226962612 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9f2c159f70 msgr2=0x7f9f2c158fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:42:57.330 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.310+0000 7f9f32401640 1 --2- 172.21.15.123:0/2226962612 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9f2c159f70 0x7f9f2c158fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:42:57.330 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.310+0000 7f9f32401640 1 -- 172.21.15.123:0/2226962612 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9f2c159510 con 0x7f9f2c159b90 2024-09-16T15:42:57.330 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.314+0000 7f9f32401640 1 --2- 172.21.15.123:0/2226962612 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f9f2c159b90 0x7f9f2c0c9950 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f9f2c0cf980 tx=0x7f9f1c005fa0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:42:57.330 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.314+0000 7f9f22ffd640 1 -- 172.21.15.123:0/2226962612 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f1c0ba070 con 0x7f9f2c159b90 2024-09-16T15:42:57.330 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.314+0000 7f9f33403640 1 -- 172.21.15.123:0/2226962612 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9f2c15c8d0 con 0x7f9f2c159b90 2024-09-16T15:42:57.330 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.314+0000 7f9f33403640 1 -- 172.21.15.123:0/2226962612 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9f2c2087f0 con 0x7f9f2c159b90 2024-09-16T15:42:57.330 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.314+0000 7f9f22ffd640 1 -- 172.21.15.123:0/2226962612 <== mon.1 v2:172.21.15.184:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f9f1c004240 con 0x7f9f2c159b90 2024-09-16T15:42:57.330 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.314+0000 7f9f22ffd640 1 -- 172.21.15.123:0/2226962612 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f24001a00 con 0x7f9f2c159b90 2024-09-16T15:42:57.330 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.314+0000 7f9f33403640 1 -- 172.21.15.123:0/2226962612 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f9f14001040 con 0x7f9f2c159b90 2024-09-16T15:42:57.330 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.314+0000 7f9f22ffd640 1 -- 172.21.15.123:0/2226962612 <== mon.1 v2:172.21.15.184:3300/0 4 ==== mgrmap(e 5) ==== 97080+0+0 (secure 0 0 0) 0x7f9f1c003600 con 0x7f9f2c159b90 2024-09-16T15:42:57.330 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.318+0000 7f9f22ffd640 1 --2- 172.21.15.123:0/2226962612 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f9f10072ec0 0x7f9f10075380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:42:57.331 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.318+0000 7f9f22ffd640 1 -- 172.21.15.123:0/2226962612 <== mon.1 v2:172.21.15.184:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f9f1c12e070 con 0x7f9f2c159b90 2024-09-16T15:42:57.331 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.318+0000 7f9f22ffd640 1 --2- 172.21.15.123:0/2226962612 >> [v2:172.21.15.123:6808/988586998,v1:172.21.15.123:6809/988586998] conn(0x7f9f1007b060 0x7f9f1007d480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:42:57.331 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.318+0000 7f9f22ffd640 1 -- 172.21.15.123:0/2226962612 --> [v2:172.21.15.123:6808/988586998,v1:172.21.15.123:6809/988586998] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f9f1c0c8e50 con 0x7f9f1007b060 2024-09-16T15:42:57.331 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.318+0000 7f9f22ffd640 1 -- 172.21.15.123:0/2226962612 <== mon.1 v2:172.21.15.184:3300/0 6 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f9f1c12e420 con 0x7f9f2c159b90 2024-09-16T15:42:57.331 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.318+0000 7f9f31c00640 1 --2- 172.21.15.123:0/2226962612 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f9f10072ec0 0x7f9f10075380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:42:57.331 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.318+0000 7f9f31c00640 1 --2- 172.21.15.123:0/2226962612 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f9f10072ec0 0x7f9f10075380 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f9f2c16c8b0 tx=0x7f9f28009040 comp rx=0 tx=0).ready entity=mgr.4135 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:42:57.331 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.310+0000 7fb7418ad640 1 -- 172.21.15.123:0/2116289007 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fb73c09dce0 con 0x7fb73c161a80 2024-09-16T15:42:57.331 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.314+0000 7fb7397fa640 1 -- 172.21.15.123:0/2116289007 <== mon.1 v2:172.21.15.184:3300/0 4 ==== mgrmap(e 5) ==== 97080+0+0 (secure 0 0 0) 0x7fb72c0052d0 con 0x7fb73c161a80 2024-09-16T15:42:57.331 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.314+0000 7fb7397fa640 1 --2- 172.21.15.123:0/2116289007 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fb728072f10 0x7fb7280753d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:42:57.332 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.314+0000 7fb7397fa640 1 -- 172.21.15.123:0/2116289007 <== mon.1 v2:172.21.15.184:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7fb72c12d700 con 0x7fb73c161a80 2024-09-16T15:42:57.332 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.314+0000 7fb7397fa640 1 --2- 172.21.15.123:0/2116289007 >> [v2:172.21.15.184:6816/3387813890,v1:172.21.15.184:6817/3387813890] conn(0x7fb72807b130 0x7fb72807d550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:42:57.332 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.318+0000 7fb7397fa640 1 -- 172.21.15.123:0/2116289007 --> [v2:172.21.15.184:6816/3387813890,v1:172.21.15.184:6817/3387813890] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fb72c007e80 con 0x7fb72807b130 2024-09-16T15:42:57.332 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.318+0000 7fb7397fa640 1 -- 172.21.15.123:0/2116289007 <== mon.1 v2:172.21.15.184:3300/0 6 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7fb72c12dbe0 con 0x7fb73c161a80 2024-09-16T15:42:57.332 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.318+0000 7fb7408ab640 1 --2- 172.21.15.123:0/2116289007 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fb728072f10 0x7fb7280753d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:42:57.332 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.318+0000 7fb7410ac640 1 --2- 172.21.15.123:0/2116289007 >> [v2:172.21.15.184:6816/3387813890,v1:172.21.15.184:6817/3387813890] conn(0x7fb72807b130 0x7fb72807d550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:42:57.332 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.318+0000 7fb7410ac640 1 --2- 172.21.15.123:0/2116289007 >> [v2:172.21.15.184:6816/3387813890,v1:172.21.15.184:6817/3387813890] conn(0x7fb72807b130 0x7fb72807d550 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:42:57.332 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.318+0000 7fb7408ab640 1 --2- 172.21.15.123:0/2116289007 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fb728072f10 0x7fb7280753d0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fb734009ef0 tx=0x7fb734009500 comp rx=0 tx=0).ready entity=mgr.4135 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:42:57.332 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.318+0000 7fb7397fa640 1 -- 172.21.15.123:0/2116289007 <== osd.4 v2:172.21.15.184:6816/3387813890 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fb72c007e80 con 0x7fb72807b130 2024-09-16T15:42:57.332 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.310+0000 7fd363417640 1 -- 172.21.15.123:0/3317373809 wait complete. 2024-09-16T15:42:57.333 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.310+0000 7fd363417640 1 Processor -- start 2024-09-16T15:42:57.333 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.310+0000 7fd363417640 1 -- start start 2024-09-16T15:42:57.333 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.310+0000 7fd363417640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd35c0d1f80 0x7fd35c151160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:42:57.333 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.310+0000 7fd363417640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd35c15f3a0 0x7fd35c0cb620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:42:57.333 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.310+0000 7fd363417640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fd35c15a250 0x7fd35c219680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:42:57.333 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.310+0000 7fd363417640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fd35c168da0 con 0x7fd35c0d1f80 2024-09-16T15:42:57.333 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.310+0000 7fd363417640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fd35c0b0090 con 0x7fd35c15f3a0 2024-09-16T15:42:57.333 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.310+0000 7fd363417640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fd35c168f20 con 0x7fd35c15a250 2024-09-16T15:42:57.333 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.310+0000 7fd3611b3640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd35c0d1f80 0x7fd35c151160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:42:57.333 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.310+0000 7fd3611b3640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd35c0d1f80 0x7fd35c151160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:40364/0 (socket says 172.21.15.123:40364) 2024-09-16T15:42:57.333 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.310+0000 7fd3611b3640 1 -- 172.21.15.123:0/1077118448 learned_addr learned my addr 172.21.15.123:0/1077118448 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:42:57.333 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.310+0000 7fd3619b4640 1 --2- 172.21.15.123:0/1077118448 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fd35c15a250 0x7fd35c219680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:42:57.334 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.310+0000 7fd3611b3640 1 -- 172.21.15.123:0/1077118448 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fd35c15a250 msgr2=0x7fd35c219680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:42:57.334 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.310+0000 7fd3611b3640 1 --2- 172.21.15.123:0/1077118448 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fd35c15a250 0x7fd35c219680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:42:57.334 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.310+0000 7fd3611b3640 1 -- 172.21.15.123:0/1077118448 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd35c15f3a0 msgr2=0x7fd35c0cb620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:42:57.334 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.310+0000 7fd3611b3640 1 --2- 172.21.15.123:0/1077118448 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd35c15f3a0 0x7fd35c0cb620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:42:57.334 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.310+0000 7fd3611b3640 1 -- 172.21.15.123:0/1077118448 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd35c1590a0 con 0x7fd35c0d1f80 2024-09-16T15:42:57.334 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.310+0000 7fd3619b4640 1 --2- 172.21.15.123:0/1077118448 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fd35c15a250 0x7fd35c219680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T15:42:57.334 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.314+0000 7fd3611b3640 1 --2- 172.21.15.123:0/1077118448 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd35c0d1f80 0x7fd35c151160 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fd35400c7c0 tx=0x7fd35400cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:42:57.334 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.314+0000 7fd351ffb640 1 -- 172.21.15.123:0/1077118448 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd354003b80 con 0x7fd35c0d1f80 2024-09-16T15:42:57.334 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.314+0000 7fd351ffb640 1 -- 172.21.15.123:0/1077118448 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fd354003d20 con 0x7fd35c0d1f80 2024-09-16T15:42:57.334 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.314+0000 7fd351ffb640 1 -- 172.21.15.123:0/1077118448 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd35400aa90 con 0x7fd35c0d1f80 2024-09-16T15:42:57.334 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.314+0000 7fd363417640 1 -- 172.21.15.123:0/1077118448 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd35c159340 con 0x7fd35c0d1f80 2024-09-16T15:42:57.334 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.314+0000 7fd363417640 1 -- 172.21.15.123:0/1077118448 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd35c159900 con 0x7fd35c0d1f80 2024-09-16T15:42:57.335 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.314+0000 7fd363417640 1 -- 172.21.15.123:0/1077118448 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fd35c0d1a40 con 0x7fd35c0d1f80 2024-09-16T15:42:57.335 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.318+0000 7fed11995640 1 -- 172.21.15.123:0/1292300799 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fecf8003c90 msgr2=0x7fecf8004090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:42:57.335 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.318+0000 7fed11995640 1 --2- 172.21.15.123:0/1292300799 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fecf8003c90 0x7fecf8004090 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fecf400a9b0 tx=0x7fecf401c8f0 comp rx=0 tx=0).stop 2024-09-16T15:42:57.335 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.318+0000 7fed11995640 1 -- 172.21.15.123:0/1292300799 shutdown_connections 2024-09-16T15:42:57.335 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.318+0000 7fed11995640 1 --2- 172.21.15.123:0/1292300799 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fecf8003c90 0x7fecf8004090 secure :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fecf400a9b0 tx=0x7fecf401c8f0 comp rx=0 tx=0).stop 2024-09-16T15:42:57.335 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.318+0000 7fed11995640 1 -- 172.21.15.123:0/1292300799 >> 172.21.15.123:0/1292300799 conn(0x7fed0c0736a0 msgr2=0x7fed0c073ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:42:57.335 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.318+0000 7fed11995640 1 -- 172.21.15.123:0/1292300799 shutdown_connections 2024-09-16T15:42:57.335 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.318+0000 7fed11995640 1 -- 172.21.15.123:0/1292300799 wait complete. 2024-09-16T15:42:57.335 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.318+0000 7fed11995640 1 Processor -- start 2024-09-16T15:42:57.336 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.318+0000 7fed11995640 1 -- start start 2024-09-16T15:42:57.336 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.318+0000 7fed11995640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fed0c1424b0 0x7fed0c157e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:42:57.336 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.318+0000 7fed11995640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fed0c142890 0x7fed0c158370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:42:57.336 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.318+0000 7fed11995640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fed0c15d670 0x7fed0c15dac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:42:57.336 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.318+0000 7fed11995640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fed0c0b3be0 con 0x7fed0c15d670 2024-09-16T15:42:57.336 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.318+0000 7fed11995640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fed0c15e5f0 con 0x7fed0c142890 2024-09-16T15:42:57.336 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.318+0000 7fed11995640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fed0c0b3d60 con 0x7fed0c1424b0 2024-09-16T15:42:57.336 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.318+0000 7fed03fff640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fed0c142890 0x7fed0c158370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:42:57.336 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.318+0000 7fed03fff640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fed0c142890 0x7fed0c158370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.123:53394/0 (socket says 172.21.15.123:53394) 2024-09-16T15:42:57.336 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.318+0000 7fed03fff640 1 -- 172.21.15.123:0/621987134 learned_addr learned my addr 172.21.15.123:0/621987134 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:42:57.336 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.318+0000 7fed0affd640 1 --2- 172.21.15.123:0/621987134 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fed0c1424b0 0x7fed0c157e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:42:57.336 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.318+0000 7fed03fff640 1 -- 172.21.15.123:0/621987134 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fed0c1424b0 msgr2=0x7fed0c157e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:42:57.337 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.318+0000 7fed03fff640 1 --2- 172.21.15.123:0/621987134 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fed0c1424b0 0x7fed0c157e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:42:57.337 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.318+0000 7fed03fff640 1 -- 172.21.15.123:0/621987134 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fed0c15d670 msgr2=0x7fed0c15dac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:42:57.337 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.318+0000 7fed03fff640 1 --2- 172.21.15.123:0/621987134 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fed0c15d670 0x7fed0c15dac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:42:57.337 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.318+0000 7fed03fff640 1 -- 172.21.15.123:0/621987134 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fed0c068ae0 con 0x7fed0c142890 2024-09-16T15:42:57.337 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.318+0000 7fed03fff640 1 --2- 172.21.15.123:0/621987134 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fed0c142890 0x7fed0c158370 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fecfc006f40 tx=0x7fecfc0039d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:42:57.337 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.318+0000 7fbf13fff640 1 -- 172.21.15.123:0/306030741 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbf1415e4b0 msgr2=0x7fbf1415e8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:42:57.337 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.318+0000 7fbf13fff640 1 --2- 172.21.15.123:0/306030741 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbf1415e4b0 0x7fbf1415e8d0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fbf08009840 tx=0x7fbf0801c7f0 comp rx=0 tx=0).stop 2024-09-16T15:42:57.338 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.318+0000 7fbf13fff640 1 -- 172.21.15.123:0/306030741 shutdown_connections 2024-09-16T15:42:57.338 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.318+0000 7fbf13fff640 1 --2- 172.21.15.123:0/306030741 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbf1415e4b0 0x7fbf1415e8d0 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:42:57.338 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.318+0000 7fbf13fff640 1 --2- 172.21.15.123:0/306030741 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fbf140afde0 0x7fbf1406fbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:42:57.338 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.318+0000 7fbf13fff640 1 -- 172.21.15.123:0/306030741 >> 172.21.15.123:0/306030741 conn(0x7fbf1406dd00 msgr2=0x7fbf1406e110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:42:57.338 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.318+0000 7fbf13fff640 1 -- 172.21.15.123:0/306030741 shutdown_connections 2024-09-16T15:42:57.338 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.318+0000 7fbf13fff640 1 -- 172.21.15.123:0/306030741 wait complete. 2024-09-16T15:42:57.338 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.322+0000 7fbf13fff640 1 Processor -- start 2024-09-16T15:42:57.339 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.322+0000 7fed08ff9640 1 -- 172.21.15.123:0/621987134 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fecfc015070 con 0x7fed0c142890 2024-09-16T15:42:57.339 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.322+0000 7fed11995640 1 -- 172.21.15.123:0/621987134 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fed0c15e000 con 0x7fed0c142890 2024-09-16T15:42:57.339 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.322+0000 7fed11995640 1 -- 172.21.15.123:0/621987134 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fed0c069080 con 0x7fed0c142890 2024-09-16T15:42:57.339 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.322+0000 7fed11995640 1 -- 172.21.15.123:0/621987134 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fed0c161a00 con 0x7fed0c142890 2024-09-16T15:42:57.339 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.322+0000 7fed08ff9640 1 -- 172.21.15.123:0/621987134 <== mon.1 v2:172.21.15.184:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fecfc011900 con 0x7fed0c142890 2024-09-16T15:42:57.339 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.322+0000 7fed08ff9640 1 -- 172.21.15.123:0/621987134 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fecfc011d30 con 0x7fed0c142890 2024-09-16T15:42:57.340 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.322+0000 7fd351ffb640 1 -- 172.21.15.123:0/1077118448 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 5) ==== 97080+0+0 (secure 0 0 0) 0x7fd35400ac30 con 0x7fd35c0d1f80 2024-09-16T15:42:57.340 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.322+0000 7fd351ffb640 1 --2- 172.21.15.123:0/1077118448 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fd340072f10 0x7fd3400753d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:42:57.340 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.322+0000 7fd351ffb640 1 -- 172.21.15.123:0/1077118448 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7fd3540945b0 con 0x7fd35c0d1f80 2024-09-16T15:42:57.340 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.322+0000 7fd351ffb640 1 --2- 172.21.15.123:0/1077118448 >> [v2:172.21.15.184:6808/2680916017,v1:172.21.15.184:6809/2680916017] conn(0x7fd34007b190 0x7fd34007d5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:42:57.340 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.322+0000 7fd351ffb640 1 -- 172.21.15.123:0/1077118448 --> [v2:172.21.15.184:6808/2680916017,v1:172.21.15.184:6809/2680916017] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fd354019e50 con 0x7fd34007b190 2024-09-16T15:42:57.340 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.322+0000 7fd351ffb640 1 -- 172.21.15.123:0/1077118448 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7fd354094960 con 0x7fd35c0d1f80 2024-09-16T15:42:57.340 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.322+0000 7fd3609b2640 1 --2- 172.21.15.123:0/1077118448 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fd340072f10 0x7fd3400753d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:42:57.341 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.322+0000 7fed08ff9640 1 -- 172.21.15.123:0/621987134 <== mon.1 v2:172.21.15.184:3300/0 4 ==== mgrmap(e 5) ==== 97080+0+0 (secure 0 0 0) 0x7fecfc018070 con 0x7fed0c142890 2024-09-16T15:42:57.341 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.322+0000 7fd3619b4640 1 --2- 172.21.15.123:0/1077118448 >> [v2:172.21.15.184:6808/2680916017,v1:172.21.15.184:6809/2680916017] conn(0x7fd34007b190 0x7fd34007d5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:42:57.341 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.322+0000 7fed08ff9640 1 --2- 172.21.15.123:0/621987134 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fecec072ec0 0x7fecec075380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:42:57.341 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.326+0000 7fed08ff9640 1 -- 172.21.15.123:0/621987134 <== mon.1 v2:172.21.15.184:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7fecfc094ce0 con 0x7fed0c142890 2024-09-16T15:42:57.342 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.326+0000 7fbf13fff640 1 -- start start 2024-09-16T15:42:57.342 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.326+0000 7fed08ff9640 1 --2- 172.21.15.123:0/621987134 >> [v2:172.21.15.184:6824/2183120457,v1:172.21.15.184:6825/2183120457] conn(0x7fecec07b060 0x7fecec07d480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:42:57.342 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.326+0000 7fed08ff9640 1 -- 172.21.15.123:0/621987134 --> [v2:172.21.15.184:6824/2183120457,v1:172.21.15.184:6825/2183120457] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fecec07db30 con 0x7fecec07b060 2024-09-16T15:42:57.342 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.326+0000 7fed08ff9640 1 -- 172.21.15.123:0/621987134 <== mon.1 v2:172.21.15.184:3300/0 6 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7fecfc095170 con 0x7fed0c142890 2024-09-16T15:42:57.342 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.326+0000 7fed0affd640 1 --2- 172.21.15.123:0/621987134 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fecec072ec0 0x7fecec075380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:42:57.343 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.326+0000 7fd3619b4640 1 --2- 172.21.15.123:0/1077118448 >> [v2:172.21.15.184:6808/2680916017,v1:172.21.15.184:6809/2680916017] conn(0x7fd34007b190 0x7fd34007d5b0 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:42:57.343 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.326+0000 7fed0b7fe640 1 --2- 172.21.15.123:0/621987134 >> [v2:172.21.15.184:6824/2183120457,v1:172.21.15.184:6825/2183120457] conn(0x7fecec07b060 0x7fecec07d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:42:57.343 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.326+0000 7fd3609b2640 1 --2- 172.21.15.123:0/1077118448 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fd340072f10 0x7fd3400753d0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fd3580af000 tx=0x7fd3580195c0 comp rx=0 tx=0).ready entity=mgr.4135 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:42:57.344 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.326+0000 7fed0affd640 1 --2- 172.21.15.123:0/621987134 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fecec072ec0 0x7fecec075380 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fecf40ae000 tx=0x7fecf4019690 comp rx=0 tx=0).ready entity=mgr.4135 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:42:57.344 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.326+0000 7fed0b7fe640 1 --2- 172.21.15.123:0/621987134 >> [v2:172.21.15.184:6824/2183120457,v1:172.21.15.184:6825/2183120457] conn(0x7fecec07b060 0x7fecec07d480 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:42:57.344 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.326+0000 7fd351ffb640 1 -- 172.21.15.123:0/1077118448 <== osd.5 v2:172.21.15.184:6808/2680916017 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fd354019e50 con 0x7fd34007b190 2024-09-16T15:42:57.344 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.326+0000 7fbf13fff640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fbf140afde0 0x7fbf14147890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:42:57.345 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.326+0000 7fbf13fff640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fbf1415e4b0 0x7fbf14148db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:42:57.345 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.326+0000 7fbf13fff640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbf1415ee10 0x7fbf1415c550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:42:57.345 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.326+0000 7fbf13fff640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fbf1406cdb0 con 0x7fbf1415ee10 2024-09-16T15:42:57.345 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.326+0000 7fbf13fff640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fbf1406cc30 con 0x7fbf140afde0 2024-09-16T15:42:57.345 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.326+0000 7fbf13fff640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fbf1406cf30 con 0x7fbf1415e4b0 2024-09-16T15:42:57.345 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.326+0000 7fbf137fe640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbf1415ee10 0x7fbf1415c550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:42:57.345 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.326+0000 7fbf137fe640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbf1415ee10 0x7fbf1415c550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:40386/0 (socket says 172.21.15.123:40386) 2024-09-16T15:42:57.345 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.326+0000 7fbf137fe640 1 -- 172.21.15.123:0/2401094378 learned_addr learned my addr 172.21.15.123:0/2401094378 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:42:57.345 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.326+0000 7fbf12ffd640 1 --2- 172.21.15.123:0/2401094378 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fbf140afde0 0x7fbf14147890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:42:57.345 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.326+0000 7fbf127fc640 1 --2- 172.21.15.123:0/2401094378 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fbf1415e4b0 0x7fbf14148db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:42:57.345 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.326+0000 7fbf137fe640 1 -- 172.21.15.123:0/2401094378 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fbf1415e4b0 msgr2=0x7fbf14148db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:42:57.345 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.326+0000 7fbf137fe640 1 --2- 172.21.15.123:0/2401094378 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fbf1415e4b0 0x7fbf14148db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:42:57.345 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.326+0000 7fbf137fe640 1 -- 172.21.15.123:0/2401094378 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fbf140afde0 msgr2=0x7fbf14147890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:42:57.346 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.326+0000 7fbf137fe640 1 --2- 172.21.15.123:0/2401094378 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fbf140afde0 0x7fbf14147890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:42:57.346 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.326+0000 7fbf137fe640 1 -- 172.21.15.123:0/2401094378 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbf14147ff0 con 0x7fbf1415ee10 2024-09-16T15:42:57.346 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.326+0000 7fbf137fe640 1 --2- 172.21.15.123:0/2401094378 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbf1415ee10 0x7fbf1415c550 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fbf0c00b9c0 tx=0x7fbf0c00be80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:42:57.346 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.326+0000 7f9f32c02640 1 --2- 172.21.15.123:0/2226962612 >> [v2:172.21.15.123:6808/988586998,v1:172.21.15.123:6809/988586998] conn(0x7f9f1007b060 0x7f9f1007d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:42:57.346 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.326+0000 7f9f32c02640 1 --2- 172.21.15.123:0/2226962612 >> [v2:172.21.15.123:6808/988586998,v1:172.21.15.123:6809/988586998] conn(0x7f9f1007b060 0x7f9f1007d480 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:42:57.346 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.326+0000 7f9f22ffd640 1 -- 172.21.15.123:0/2226962612 <== osd.3 v2:172.21.15.123:6808/988586998 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f9f1c0c8e50 con 0x7f9f1007b060 2024-09-16T15:42:57.346 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.330+0000 7fed08ff9640 1 -- 172.21.15.123:0/621987134 <== osd.6 v2:172.21.15.184:6824/2183120457 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fecec07db30 con 0x7fecec07b060 2024-09-16T15:42:57.347 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.326+0000 7fbefb7fe640 1 -- 172.21.15.123:0/2401094378 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf0c019070 con 0x7fbf1415ee10 2024-09-16T15:42:57.347 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.326+0000 7fbefb7fe640 1 -- 172.21.15.123:0/2401094378 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fbf0c00cc90 con 0x7fbf1415ee10 2024-09-16T15:42:57.347 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.326+0000 7fbefb7fe640 1 -- 172.21.15.123:0/2401094378 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf0c0146a0 con 0x7fbf1415ee10 2024-09-16T15:42:57.347 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.326+0000 7fbf13fff640 1 -- 172.21.15.123:0/2401094378 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbf141482d0 con 0x7fbf1415ee10 2024-09-16T15:42:57.348 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.326+0000 7fbf13fff640 1 -- 172.21.15.123:0/2401094378 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbf14148890 con 0x7fbf1415ee10 2024-09-16T15:42:57.348 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.326+0000 7fbf13fff640 1 -- 172.21.15.123:0/2401094378 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fbf140cf580 con 0x7fbf1415ee10 2024-09-16T15:42:57.348 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.330+0000 7fbefb7fe640 1 -- 172.21.15.123:0/2401094378 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 5) ==== 97080+0+0 (secure 0 0 0) 0x7fbf0c007370 con 0x7fbf1415ee10 2024-09-16T15:42:57.348 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.330+0000 7fbefb7fe640 1 --2- 172.21.15.123:0/2401094378 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fbf00072ef0 0x7fbf000753b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:42:57.348 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.330+0000 7fbf12ffd640 1 --2- 172.21.15.123:0/2401094378 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fbf00072ef0 0x7fbf000753b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:42:57.348 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.330+0000 7fbefb7fe640 1 -- 172.21.15.123:0/2401094378 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7fbf0c091df0 con 0x7fbf1415ee10 2024-09-16T15:42:57.348 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.330+0000 7fbefb7fe640 1 --2- 172.21.15.123:0/2401094378 >> [v2:172.21.15.184:6800/4102957285,v1:172.21.15.184:6801/4102957285] conn(0x7fbf0007b0f0 0x7fbf0007d510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:42:57.348 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.330+0000 7fbefb7fe640 1 -- 172.21.15.123:0/2401094378 --> [v2:172.21.15.184:6800/4102957285,v1:172.21.15.184:6801/4102957285] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fbf0c013ec0 con 0x7fbf0007b0f0 2024-09-16T15:42:57.348 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.330+0000 7fbefb7fe640 1 -- 172.21.15.123:0/2401094378 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7fbf0c05ab10 con 0x7fbf1415ee10 2024-09-16T15:42:57.348 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.334+0000 7fbf127fc640 1 --2- 172.21.15.123:0/2401094378 >> [v2:172.21.15.184:6800/4102957285,v1:172.21.15.184:6801/4102957285] conn(0x7fbf0007b0f0 0x7fbf0007d510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:42:57.348 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.334+0000 7fbf12ffd640 1 --2- 172.21.15.123:0/2401094378 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fbf00072ef0 0x7fbf000753b0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fbf04004510 tx=0x7fbf0400b440 comp rx=0 tx=0).ready entity=mgr.4135 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:42:57.349 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.334+0000 7fbf127fc640 1 --2- 172.21.15.123:0/2401094378 >> [v2:172.21.15.184:6800/4102957285,v1:172.21.15.184:6801/4102957285] conn(0x7fbf0007b0f0 0x7fbf0007d510 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:42:57.349 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.334+0000 7fbefb7fe640 1 -- 172.21.15.123:0/2401094378 <== osd.7 v2:172.21.15.184:6800/4102957285 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fbf0c013ec0 con 0x7fbf0007b0f0 2024-09-16T15:42:57.349 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.342+0000 7f2bcb083640 1 -- 172.21.15.123:0/1712634364 --> [v2:172.21.15.123:6824/1332143088,v1:172.21.15.123:6825/1332143088] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f2bc40d08d0 con 0x7f2ba807b060 2024-09-16T15:42:57.349 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.342+0000 7f2bbaffd640 1 -- 172.21.15.123:0/1712634364 <== osd.2 v2:172.21.15.123:6824/1332143088 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f2bc40d08d0 con 0x7f2ba807b060 2024-09-16T15:42:57.350 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.342+0000 7f2bcb083640 1 -- 172.21.15.123:0/1712634364 >> [v2:172.21.15.123:6824/1332143088,v1:172.21.15.123:6825/1332143088] conn(0x7f2ba807b060 msgr2=0x7f2ba807d480 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:42:57.350 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.342+0000 7f2bcb083640 1 --2- 172.21.15.123:0/1712634364 >> [v2:172.21.15.123:6824/1332143088,v1:172.21.15.123:6825/1332143088] conn(0x7f2ba807b060 0x7f2ba807d480 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:42:57.350 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.342+0000 7f2bcb083640 1 -- 172.21.15.123:0/1712634364 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f2ba8072ec0 msgr2=0x7f2ba8075380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:42:57.350 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.342+0000 7f2bcb083640 1 --2- 172.21.15.123:0/1712634364 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f2ba8072ec0 0x7f2ba8075380 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f2bc40c7d40 tx=0x7f2bc0011040 comp rx=0 tx=0).stop 2024-09-16T15:42:57.350 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.342+0000 7f2bcb083640 1 -- 172.21.15.123:0/1712634364 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2bc40ca150 msgr2=0x7f2bc40c4d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:42:57.351 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.342+0000 7f2bcb083640 1 --2- 172.21.15.123:0/1712634364 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2bc40ca150 0x7f2bc40c4d10 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f2bb40ac000 tx=0x7f2bb40a6a80 comp rx=0 tx=0).stop 2024-09-16T15:42:57.351 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.342+0000 7f2bca882640 1 -- 172.21.15.123:0/1712634364 reap_dead start 2024-09-16T15:42:57.351 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.342+0000 7f2bcb083640 1 -- 172.21.15.123:0/1712634364 shutdown_connections 2024-09-16T15:42:57.351 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.342+0000 7f2bcb083640 1 -- 172.21.15.123:0/1712634364 >> 172.21.15.123:0/1712634364 conn(0x7f2bc40b1550 msgr2=0x7f2bc40af4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:42:57.351 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.342+0000 7f2bcb083640 1 -- 172.21.15.123:0/1712634364 shutdown_connections 2024-09-16T15:42:57.351 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.342+0000 7f2bcb083640 1 -- 172.21.15.123:0/1712634364 wait complete. 2024-09-16T15:42:57.351 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.346+0000 7f254359e640 1 -- 172.21.15.123:0/521860267 --> [v2:172.21.15.123:6800/3882044969,v1:172.21.15.123:6801/3882044969] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f253c032ec0 con 0x7f252407b060 2024-09-16T15:42:57.352 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.346+0000 7f2532ffd640 1 -- 172.21.15.123:0/521860267 <== osd.0 v2:172.21.15.123:6800/3882044969 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f253c032ec0 con 0x7f252407b060 2024-09-16T15:42:57.352 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.346+0000 7fd363417640 1 -- 172.21.15.123:0/1077118448 --> [v2:172.21.15.184:6808/2680916017,v1:172.21.15.184:6809/2680916017] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fd35c0aefb0 con 0x7fd34007b190 2024-09-16T15:42:57.352 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.346+0000 7fb7418ad640 1 -- 172.21.15.123:0/2116289007 --> [v2:172.21.15.184:6816/3387813890,v1:172.21.15.184:6817/3387813890] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fb73c09dce0 con 0x7fb72807b130 2024-09-16T15:42:57.353 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.346+0000 7f254359e640 1 -- 172.21.15.123:0/521860267 >> [v2:172.21.15.123:6800/3882044969,v1:172.21.15.123:6801/3882044969] conn(0x7f252407b060 msgr2=0x7f252407d480 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:42:57.353 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.346+0000 7f254359e640 1 --2- 172.21.15.123:0/521860267 >> [v2:172.21.15.123:6800/3882044969,v1:172.21.15.123:6801/3882044969] conn(0x7f252407b060 0x7f252407d480 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:42:57.353 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.346+0000 7f254359e640 1 -- 172.21.15.123:0/521860267 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f2524072ec0 msgr2=0x7f2524075380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:42:57.353 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.346+0000 7f254359e640 1 --2- 172.21.15.123:0/521860267 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f2524072ec0 0x7f2524075380 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f2534009b40 tx=0x7f2534009260 comp rx=0 tx=0).stop 2024-09-16T15:42:57.353 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.346+0000 7f254359e640 1 -- 172.21.15.123:0/521860267 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f253c0377e0 msgr2=0x7f253c1836f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:42:57.353 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.346+0000 7f254359e640 1 --2- 172.21.15.123:0/521860267 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f253c0377e0 0x7f253c1836f0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f253800c480 tx=0x7f253800c940 comp rx=0 tx=0).stop 2024-09-16T15:42:57.353 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.346+0000 7f2542d9d640 1 -- 172.21.15.123:0/521860267 reap_dead start 2024-09-16T15:42:57.354 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.346+0000 7f254359e640 1 -- 172.21.15.123:0/521860267 shutdown_connections 2024-09-16T15:42:57.354 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.346+0000 7f254359e640 1 -- 172.21.15.123:0/521860267 >> 172.21.15.123:0/521860267 conn(0x7f253c01e8a0 msgr2=0x7f253c0e0fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:42:57.354 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.346+0000 7f254359e640 1 -- 172.21.15.123:0/521860267 shutdown_connections 2024-09-16T15:42:57.354 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.346+0000 7f254359e640 1 -- 172.21.15.123:0/521860267 wait complete. 2024-09-16T15:42:57.354 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.346+0000 7fb7397fa640 1 -- 172.21.15.123:0/2116289007 <== osd.4 v2:172.21.15.184:6816/3387813890 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7fb73c09dce0 con 0x7fb72807b130 2024-09-16T15:42:57.355 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.346+0000 7f40acafc640 1 -- 172.21.15.123:0/392369015 --> [v2:172.21.15.123:6816/4037001850,v1:172.21.15.123:6817/4037001850] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f40a00071a0 con 0x7f408807b060 2024-09-16T15:42:57.355 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.346+0000 7fd351ffb640 1 -- 172.21.15.123:0/1077118448 <== osd.5 v2:172.21.15.184:6808/2680916017 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7fd35c0aefb0 con 0x7fd34007b190 2024-09-16T15:42:57.355 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.346+0000 7fd363417640 1 -- 172.21.15.123:0/1077118448 >> [v2:172.21.15.184:6808/2680916017,v1:172.21.15.184:6809/2680916017] conn(0x7fd34007b190 msgr2=0x7fd34007d5b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:42:57.355 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.346+0000 7fd363417640 1 --2- 172.21.15.123:0/1077118448 >> [v2:172.21.15.184:6808/2680916017,v1:172.21.15.184:6809/2680916017] conn(0x7fd34007b190 0x7fd34007d5b0 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:42:57.355 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.346+0000 7fd363417640 1 -- 172.21.15.123:0/1077118448 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fd340072f10 msgr2=0x7fd3400753d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:42:57.355 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.346+0000 7fd363417640 1 --2- 172.21.15.123:0/1077118448 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fd340072f10 0x7fd3400753d0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fd3580af000 tx=0x7fd3580195c0 comp rx=0 tx=0).stop 2024-09-16T15:42:57.355 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.346+0000 7fd363417640 1 -- 172.21.15.123:0/1077118448 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd35c0d1f80 msgr2=0x7fd35c151160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:42:57.356 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.346+0000 7fd363417640 1 --2- 172.21.15.123:0/1077118448 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd35c0d1f80 0x7fd35c151160 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fd35400c7c0 tx=0x7fd35400cc80 comp rx=0 tx=0).stop 2024-09-16T15:42:57.356 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.346+0000 7fd3619b4640 1 -- 172.21.15.123:0/1077118448 reap_dead start 2024-09-16T15:42:57.356 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.346+0000 7fb7418ad640 1 -- 172.21.15.123:0/2116289007 >> [v2:172.21.15.184:6816/3387813890,v1:172.21.15.184:6817/3387813890] conn(0x7fb72807b130 msgr2=0x7fb72807d550 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:42:57.356 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.346+0000 7fb7418ad640 1 --2- 172.21.15.123:0/2116289007 >> [v2:172.21.15.184:6816/3387813890,v1:172.21.15.184:6817/3387813890] conn(0x7fb72807b130 0x7fb72807d550 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:42:57.356 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.346+0000 7fb7418ad640 1 -- 172.21.15.123:0/2116289007 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fb728072f10 msgr2=0x7fb7280753d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:42:57.356 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.346+0000 7fb7418ad640 1 --2- 172.21.15.123:0/2116289007 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fb728072f10 0x7fb7280753d0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fb734009ef0 tx=0x7fb734009500 comp rx=0 tx=0).stop 2024-09-16T15:42:57.357 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.346+0000 7f4096ffd640 1 -- 172.21.15.123:0/392369015 <== osd.1 v2:172.21.15.123:6816/4037001850 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f40a00071a0 con 0x7f408807b060 2024-09-16T15:42:57.357 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.346+0000 7fb7418ad640 1 -- 172.21.15.123:0/2116289007 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb73c161a80 msgr2=0x7fb73c0a1640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:42:57.357 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.346+0000 7fb7418ad640 1 --2- 172.21.15.123:0/2116289007 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb73c161a80 0x7fb73c0a1640 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fb72c01af40 tx=0x7fb72c005e90 comp rx=0 tx=0).stop 2024-09-16T15:42:57.357 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.346+0000 7fb7410ac640 1 -- 172.21.15.123:0/2116289007 reap_dead start 2024-09-16T15:42:57.357 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.346+0000 7fb7418ad640 1 -- 172.21.15.123:0/2116289007 shutdown_connections 2024-09-16T15:42:57.357 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.346+0000 7fb7418ad640 1 -- 172.21.15.123:0/2116289007 >> 172.21.15.123:0/2116289007 conn(0x7fb73c09d200 msgr2=0x7fb73c06d430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:42:57.358 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.346+0000 7fd363417640 1 -- 172.21.15.123:0/1077118448 shutdown_connections 2024-09-16T15:42:57.358 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.346+0000 7fd363417640 1 -- 172.21.15.123:0/1077118448 >> 172.21.15.123:0/1077118448 conn(0x7fd35c09aa40 msgr2=0x7fd35c15e270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:42:57.358 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.346+0000 7fed11995640 1 -- 172.21.15.123:0/621987134 --> [v2:172.21.15.184:6824/2183120457,v1:172.21.15.184:6825/2183120457] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fed0c15cc10 con 0x7fecec07b060 2024-09-16T15:42:57.359 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.346+0000 7fb7418ad640 1 -- 172.21.15.123:0/2116289007 shutdown_connections 2024-09-16T15:42:57.359 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.346+0000 7fb7418ad640 1 -- 172.21.15.123:0/2116289007 wait complete. 2024-09-16T15:42:57.359 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.346+0000 7fd363417640 1 -- 172.21.15.123:0/1077118448 shutdown_connections 2024-09-16T15:42:57.359 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.346+0000 7fd363417640 1 -- 172.21.15.123:0/1077118448 wait complete. 2024-09-16T15:42:57.360 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.350+0000 7fed08ff9640 1 -- 172.21.15.123:0/621987134 <== osd.6 v2:172.21.15.184:6824/2183120457 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7fed0c15cc10 con 0x7fecec07b060 2024-09-16T15:42:57.361 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.350+0000 7f40acafc640 1 -- 172.21.15.123:0/392369015 >> [v2:172.21.15.123:6816/4037001850,v1:172.21.15.123:6817/4037001850] conn(0x7f408807b060 msgr2=0x7f408807d480 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:42:57.361 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.350+0000 7f40acafc640 1 --2- 172.21.15.123:0/392369015 >> [v2:172.21.15.123:6816/4037001850,v1:172.21.15.123:6817/4037001850] conn(0x7f408807b060 0x7f408807d480 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:42:57.361 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.350+0000 7f40acafc640 1 -- 172.21.15.123:0/392369015 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f4088072ec0 msgr2=0x7f4088075380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:42:57.361 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.350+0000 7f40acafc640 1 --2- 172.21.15.123:0/392369015 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f4088072ec0 0x7f4088075380 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f40a0048c30 tx=0x7f4098009040 comp rx=0 tx=0).stop 2024-09-16T15:42:57.361 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.350+0000 7f40acafc640 1 -- 172.21.15.123:0/392369015 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f40a0006d30 msgr2=0x7f40a00486f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:42:57.361 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.350+0000 7f40acafc640 1 --2- 172.21.15.123:0/392369015 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f40a0006d30 0x7f40a00486f0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f409c0ac000 tx=0x7f409c005d60 comp rx=0 tx=0).stop 2024-09-16T15:42:57.361 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.350+0000 7f40a6d9d640 1 -- 172.21.15.123:0/392369015 reap_dead start 2024-09-16T15:42:57.361 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.350+0000 7f40acafc640 1 -- 172.21.15.123:0/392369015 shutdown_connections 2024-09-16T15:42:57.361 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.350+0000 7f40acafc640 1 -- 172.21.15.123:0/392369015 >> 172.21.15.123:0/392369015 conn(0x7f40a003a3e0 msgr2=0x7f40a003caa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:42:57.361 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.350+0000 7f40acafc640 1 -- 172.21.15.123:0/392369015 shutdown_connections 2024-09-16T15:42:57.361 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.350+0000 7f40acafc640 1 -- 172.21.15.123:0/392369015 wait complete. 2024-09-16T15:42:57.362 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.350+0000 7fed11995640 1 -- 172.21.15.123:0/621987134 >> [v2:172.21.15.184:6824/2183120457,v1:172.21.15.184:6825/2183120457] conn(0x7fecec07b060 msgr2=0x7fecec07d480 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:42:57.362 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.350+0000 7fed11995640 1 --2- 172.21.15.123:0/621987134 >> [v2:172.21.15.184:6824/2183120457,v1:172.21.15.184:6825/2183120457] conn(0x7fecec07b060 0x7fecec07d480 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:42:57.362 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.350+0000 7fed11995640 1 -- 172.21.15.123:0/621987134 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fecec072ec0 msgr2=0x7fecec075380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:42:57.363 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.350+0000 7fed11995640 1 --2- 172.21.15.123:0/621987134 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fecec072ec0 0x7fecec075380 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fecf40ae000 tx=0x7fecf4019690 comp rx=0 tx=0).stop 2024-09-16T15:42:57.363 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.350+0000 7fed11995640 1 -- 172.21.15.123:0/621987134 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fed0c142890 msgr2=0x7fed0c158370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:42:57.363 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.350+0000 7fed11995640 1 --2- 172.21.15.123:0/621987134 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fed0c142890 0x7fed0c158370 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fecfc006f40 tx=0x7fecfc0039d0 comp rx=0 tx=0).stop 2024-09-16T15:42:57.363 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.350+0000 7fed0b7fe640 1 -- 172.21.15.123:0/621987134 reap_dead start 2024-09-16T15:42:57.363 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.350+0000 7fed11995640 1 -- 172.21.15.123:0/621987134 shutdown_connections 2024-09-16T15:42:57.363 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.350+0000 7fed11995640 1 -- 172.21.15.123:0/621987134 >> 172.21.15.123:0/621987134 conn(0x7fed0c0736a0 msgr2=0x7fed0c14f3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:42:57.363 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.350+0000 7fed11995640 1 -- 172.21.15.123:0/621987134 shutdown_connections 2024-09-16T15:42:57.363 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.350+0000 7fed11995640 1 -- 172.21.15.123:0/621987134 wait complete. 2024-09-16T15:42:57.363 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.354+0000 7f9f33403640 1 -- 172.21.15.123:0/2226962612 --> [v2:172.21.15.123:6808/988586998,v1:172.21.15.123:6809/988586998] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f9f14002d60 con 0x7f9f1007b060 2024-09-16T15:42:57.364 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.358+0000 7f9f22ffd640 1 -- 172.21.15.123:0/2226962612 <== osd.3 v2:172.21.15.123:6808/988586998 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f9f14002d60 con 0x7f9f1007b060 2024-09-16T15:42:57.365 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.358+0000 7f9f33403640 1 -- 172.21.15.123:0/2226962612 >> [v2:172.21.15.123:6808/988586998,v1:172.21.15.123:6809/988586998] conn(0x7f9f1007b060 msgr2=0x7f9f1007d480 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:42:57.365 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.358+0000 7f9f33403640 1 --2- 172.21.15.123:0/2226962612 >> [v2:172.21.15.123:6808/988586998,v1:172.21.15.123:6809/988586998] conn(0x7f9f1007b060 0x7f9f1007d480 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:42:57.365 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.358+0000 7f9f33403640 1 -- 172.21.15.123:0/2226962612 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f9f10072ec0 msgr2=0x7f9f10075380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:42:57.365 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.358+0000 7f9f33403640 1 --2- 172.21.15.123:0/2226962612 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f9f10072ec0 0x7f9f10075380 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f9f2c16c8b0 tx=0x7f9f28009040 comp rx=0 tx=0).stop 2024-09-16T15:42:57.365 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.358+0000 7f9f33403640 1 -- 172.21.15.123:0/2226962612 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f9f2c159b90 msgr2=0x7f9f2c0c9950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:42:57.365 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.358+0000 7f9f33403640 1 --2- 172.21.15.123:0/2226962612 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f9f2c159b90 0x7f9f2c0c9950 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f9f2c0cf980 tx=0x7f9f1c005fa0 comp rx=0 tx=0).stop 2024-09-16T15:42:57.365 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.358+0000 7f9f32c02640 1 -- 172.21.15.123:0/2226962612 reap_dead start 2024-09-16T15:42:57.365 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.358+0000 7f9f33403640 1 -- 172.21.15.123:0/2226962612 shutdown_connections 2024-09-16T15:42:57.365 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.358+0000 7f9f33403640 1 -- 172.21.15.123:0/2226962612 >> 172.21.15.123:0/2226962612 conn(0x7f9f2c0b1550 msgr2=0x7f9f2c0d01d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:42:57.365 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.358+0000 7f9f33403640 1 -- 172.21.15.123:0/2226962612 shutdown_connections 2024-09-16T15:42:57.366 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.358+0000 7f9f33403640 1 -- 172.21.15.123:0/2226962612 wait complete. 2024-09-16T15:42:57.372 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.370+0000 7fbf13fff640 1 -- 172.21.15.123:0/2401094378 --> [v2:172.21.15.184:6800/4102957285,v1:172.21.15.184:6801/4102957285] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fbf1406fbe0 con 0x7fbf0007b0f0 2024-09-16T15:42:57.373 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.370+0000 7fbefb7fe640 1 -- 172.21.15.123:0/2401094378 <== osd.7 v2:172.21.15.184:6800/4102957285 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7fbf1406fbe0 con 0x7fbf0007b0f0 2024-09-16T15:42:57.374 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.370+0000 7fbef97fa640 1 -- 172.21.15.123:0/2401094378 >> [v2:172.21.15.184:6800/4102957285,v1:172.21.15.184:6801/4102957285] conn(0x7fbf0007b0f0 msgr2=0x7fbf0007d510 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:42:57.374 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.370+0000 7fbef97fa640 1 --2- 172.21.15.123:0/2401094378 >> [v2:172.21.15.184:6800/4102957285,v1:172.21.15.184:6801/4102957285] conn(0x7fbf0007b0f0 0x7fbf0007d510 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:42:57.375 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.374+0000 7fbef97fa640 1 -- 172.21.15.123:0/2401094378 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fbf00072ef0 msgr2=0x7fbf000753b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:42:57.375 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.374+0000 7fbef97fa640 1 --2- 172.21.15.123:0/2401094378 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fbf00072ef0 0x7fbf000753b0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fbf04004510 tx=0x7fbf0400b440 comp rx=0 tx=0).stop 2024-09-16T15:42:57.375 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.374+0000 7fbef97fa640 1 -- 172.21.15.123:0/2401094378 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbf1415ee10 msgr2=0x7fbf1415c550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:42:57.375 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.374+0000 7fbef97fa640 1 --2- 172.21.15.123:0/2401094378 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbf1415ee10 0x7fbf1415c550 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fbf0c00b9c0 tx=0x7fbf0c00be80 comp rx=0 tx=0).stop 2024-09-16T15:42:57.376 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.374+0000 7fbf137fe640 1 -- 172.21.15.123:0/2401094378 reap_dead start 2024-09-16T15:42:57.376 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.374+0000 7fbef97fa640 1 -- 172.21.15.123:0/2401094378 shutdown_connections 2024-09-16T15:42:57.376 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.374+0000 7fbef97fa640 1 -- 172.21.15.123:0/2401094378 >> 172.21.15.123:0/2401094378 conn(0x7fbf1406dd00 msgr2=0x7fbf140b5610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:42:57.376 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.374+0000 7fbef97fa640 1 -- 172.21.15.123:0/2401094378 shutdown_connections 2024-09-16T15:42:57.376 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.374+0000 7fbef97fa640 1 -- 172.21.15.123:0/2401094378 wait complete. 2024-09-16T15:42:57.381 INFO:teuthology.orchestra.run.smithi123.stdout:60129542150 2024-09-16T15:42:57.381 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-09-16T15:42:57.385 INFO:teuthology.orchestra.run.smithi123.stdout:60129542149 2024-09-16T15:42:57.385 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-09-16T15:42:57.386 INFO:teuthology.orchestra.run.smithi123.stdout:60129542150 2024-09-16T15:42:57.386 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2024-09-16T15:42:57.389 INFO:teuthology.orchestra.run.smithi123.stdout:60129542151 2024-09-16T15:42:57.389 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-09-16T15:42:57.390 INFO:teuthology.orchestra.run.smithi123.stdout:60129542150 2024-09-16T15:42:57.390 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-09-16T15:42:57.395 INFO:teuthology.orchestra.run.smithi123.stdout:60129542152 2024-09-16T15:42:57.395 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-09-16T15:42:57.397 INFO:teuthology.orchestra.run.smithi123.stdout:60129542152 2024-09-16T15:42:57.397 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-09-16T15:42:57.407 INFO:teuthology.orchestra.run.smithi123.stdout:60129542151 2024-09-16T15:42:57.408 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2024-09-16T15:42:57.531 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.526+0000 7f34e1ad7640 1 -- 172.21.15.123:0/314483757 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f34dc15b5f0 msgr2=0x7f34dc06cbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:42:57.531 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.526+0000 7f34e1ad7640 1 --2- 172.21.15.123:0/314483757 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f34dc15b5f0 0x7f34dc06cbf0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f34dc0af470 tx=0x7f34cc01c8c0 comp rx=0 tx=0).stop 2024-09-16T15:42:57.531 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.530+0000 7f34e1ad7640 1 -- 172.21.15.123:0/314483757 shutdown_connections 2024-09-16T15:42:57.531 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.530+0000 7f34e1ad7640 1 --2- 172.21.15.123:0/314483757 >> v2:172.21.15.184:3300/0 conn(0x7f34dc06d130 0x7f34dc0b2b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:42:57.531 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.530+0000 7f34e1ad7640 1 --2- 172.21.15.123:0/314483757 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f34dc15b5f0 0x7f34dc06cbf0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:42:57.531 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.530+0000 7f34e1ad7640 1 -- 172.21.15.123:0/314483757 >> 172.21.15.123:0/314483757 conn(0x7f34dc0b1550 msgr2=0x7f34dc0b1960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:42:57.531 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.530+0000 7f34e1ad7640 1 -- 172.21.15.123:0/314483757 shutdown_connections 2024-09-16T15:42:57.533 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.530+0000 7f34e1ad7640 1 -- 172.21.15.123:0/314483757 wait complete. 2024-09-16T15:42:57.533 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.530+0000 7f34e1ad7640 1 Processor -- start 2024-09-16T15:42:57.533 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.530+0000 7f34e1ad7640 1 -- start start 2024-09-16T15:42:57.534 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.530+0000 7f34e1ad7640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f34dc06d130 0x7f34dc204870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:42:57.534 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.530+0000 7f34e1ad7640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f34dc0b30a0 0x7f34dc06f1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:42:57.534 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.530+0000 7f34e1ad7640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f34dc15b5f0 0x7f34dc207430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:42:57.535 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.530+0000 7f34e1ad7640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f34dc16ab30 con 0x7f34dc0b30a0 2024-09-16T15:42:57.535 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.530+0000 7f34e1ad7640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f34dc15bb60 con 0x7f34dc06d130 2024-09-16T15:42:57.535 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.530+0000 7f34e1ad7640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f34dc16acb0 con 0x7f34dc15b5f0 2024-09-16T15:42:57.535 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.530+0000 7f34daffd640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f34dc0b30a0 0x7f34dc06f1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:42:57.535 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.530+0000 7f34daffd640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f34dc0b30a0 0x7f34dc06f1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:40396/0 (socket says 172.21.15.123:40396) 2024-09-16T15:42:57.535 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.530+0000 7f34daffd640 1 -- 172.21.15.123:0/3788276589 learned_addr learned my addr 172.21.15.123:0/3788276589 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:42:57.535 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.534+0000 7f34dbfff640 1 --2- 172.21.15.123:0/3788276589 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f34dc15b5f0 0x7f34dc207430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:42:57.535 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.534+0000 7f34db7fe640 1 --2- 172.21.15.123:0/3788276589 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f34dc06d130 0x7f34dc204870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:42:57.536 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.534+0000 7f34daffd640 1 -- 172.21.15.123:0/3788276589 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f34dc15b5f0 msgr2=0x7f34dc207430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:42:57.536 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.534+0000 7f34daffd640 1 --2- 172.21.15.123:0/3788276589 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f34dc15b5f0 0x7f34dc207430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:42:57.536 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.534+0000 7f34daffd640 1 -- 172.21.15.123:0/3788276589 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f34dc06d130 msgr2=0x7f34dc204870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:42:57.536 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.534+0000 7f34daffd640 1 --2- 172.21.15.123:0/3788276589 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f34dc06d130 0x7f34dc204870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:42:57.536 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.534+0000 7f34daffd640 1 -- 172.21.15.123:0/3788276589 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f34dc204db0 con 0x7f34dc0b30a0 2024-09-16T15:42:57.536 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.534+0000 7f34daffd640 1 --2- 172.21.15.123:0/3788276589 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f34dc0b30a0 0x7f34dc06f1d0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f34d4009fd0 tx=0x7f34d400c270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:42:57.538 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.534+0000 7f34c3fff640 1 -- 172.21.15.123:0/3788276589 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34d4011470 con 0x7f34dc0b30a0 2024-09-16T15:42:57.538 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.534+0000 7f34e1ad7640 1 -- 172.21.15.123:0/3788276589 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f34dc207e80 con 0x7f34dc0b30a0 2024-09-16T15:42:57.538 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.534+0000 7f34e1ad7640 1 -- 172.21.15.123:0/3788276589 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f34dc204070 con 0x7f34dc0b30a0 2024-09-16T15:42:57.541 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.538+0000 7f34c3fff640 1 -- 172.21.15.123:0/3788276589 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f34d4011700 con 0x7f34dc0b30a0 2024-09-16T15:42:57.541 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.538+0000 7f34c3fff640 1 -- 172.21.15.123:0/3788276589 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34d4011b30 con 0x7f34dc0b30a0 2024-09-16T15:42:57.542 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.542+0000 7f34e1ad7640 1 -- 172.21.15.123:0/3788276589 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f34bc0052a0 con 0x7f34dc0b30a0 2024-09-16T15:42:57.543 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.542+0000 7f299c478640 1 -- 172.21.15.123:0/1523749952 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f299406ce50 msgr2=0x7f29940b2b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:42:57.543 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.542+0000 7f2999212640 1 -- 172.21.15.123:0/1523749952 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f298c0b2070 con 0x7f299406ce50 2024-09-16T15:42:57.544 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.542+0000 7f34c3fff640 1 -- 172.21.15.123:0/3788276589 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 5) ==== 97080+0+0 (secure 0 0 0) 0x7f34d401a400 con 0x7f34dc0b30a0 2024-09-16T15:42:57.544 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.542+0000 7f34c3fff640 1 --2- 172.21.15.123:0/3788276589 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f34c8072f90 0x7f34c8075450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:42:57.544 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.542+0000 7f299c478640 1 --2- 172.21.15.123:0/1523749952 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f299406ce50 0x7f29940b2b60 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f299406fc70 tx=0x7f298c01cdc0 comp rx=0 tx=0).stop 2024-09-16T15:42:57.545 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.542+0000 7f34c3fff640 1 -- 172.21.15.123:0/3788276589 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f34d4094950 con 0x7f34dc0b30a0 2024-09-16T15:42:57.545 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.542+0000 7f34db7fe640 1 --2- 172.21.15.123:0/3788276589 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f34c8072f90 0x7f34c8075450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:42:57.545 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.542+0000 7f34db7fe640 1 --2- 172.21.15.123:0/3788276589 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f34c8072f90 0x7f34c8075450 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f34cc0af000 tx=0x7f34cc01b100 comp rx=0 tx=0).ready entity=mgr.4135 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:42:57.552 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.550+0000 7f299c478640 1 -- 172.21.15.123:0/1523749952 shutdown_connections 2024-09-16T15:42:57.553 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.550+0000 7f299c478640 1 --2- 172.21.15.123:0/1523749952 >> v2:172.21.15.184:3300/0 conn(0x7f299415ee20 0x7f2994161cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:42:57.553 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.550+0000 7f299c478640 1 --2- 172.21.15.123:0/1523749952 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f29940b30a0 0x7f2994161790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:42:57.553 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.550+0000 7f299c478640 1 --2- 172.21.15.123:0/1523749952 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f299406ce50 0x7f29940b2b60 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:42:57.553 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.550+0000 7f299c478640 1 -- 172.21.15.123:0/1523749952 >> 172.21.15.123:0/1523749952 conn(0x7f29940b1550 msgr2=0x7f29940b1960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:42:57.553 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.550+0000 7f299c478640 1 -- 172.21.15.123:0/1523749952 shutdown_connections 2024-09-16T15:42:57.554 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.550+0000 7f34c3fff640 1 -- 172.21.15.123:0/3788276589 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f34d4016030 con 0x7f34dc0b30a0 2024-09-16T15:42:57.554 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.550+0000 7f299c478640 1 -- 172.21.15.123:0/1523749952 wait complete. 2024-09-16T15:42:57.555 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.550+0000 7f1160dfa640 1 -- 172.21.15.123:0/3466605315 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f115c06d230 msgr2=0x7f115c0cfd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:42:57.555 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.550+0000 7f1160dfa640 1 --2- 172.21.15.123:0/3466605315 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f115c06d230 0x7f115c0cfd90 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f115c15bbb0 tx=0x7f115401ccf0 comp rx=0 tx=0).stop 2024-09-16T15:42:57.555 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.550+0000 7f1160dfa640 1 -- 172.21.15.123:0/3466605315 shutdown_connections 2024-09-16T15:42:57.555 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.550+0000 7f1160dfa640 1 --2- 172.21.15.123:0/3466605315 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f115c06d230 0x7f115c0cfd90 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:42:57.555 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.550+0000 7f1160dfa640 1 --2- 172.21.15.123:0/3466605315 >> v2:172.21.15.123:3300/0 conn(0x7f115c0cdfc0 0x7f115c0cf850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:42:57.555 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.550+0000 7f1160dfa640 1 -- 172.21.15.123:0/3466605315 >> 172.21.15.123:0/3466605315 conn(0x7f115c0cd430 msgr2=0x7f115c0cd840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:42:57.555 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.550+0000 7f1160dfa640 1 -- 172.21.15.123:0/3466605315 shutdown_connections 2024-09-16T15:42:57.556 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.554+0000 7f299c478640 1 Processor -- start 2024-09-16T15:42:57.556 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.554+0000 7f1160dfa640 1 -- 172.21.15.123:0/3466605315 wait complete. 2024-09-16T15:42:57.556 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.554+0000 7f299c478640 1 -- start start 2024-09-16T15:42:57.557 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.554+0000 7f299c478640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f299406ce50 0x7f2994146960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:42:57.557 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.554+0000 7f1160dfa640 1 Processor -- start 2024-09-16T15:42:57.558 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.554+0000 7f299c478640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f29940b30a0 0x7f2994152c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:42:57.558 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.554+0000 7f299c478640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f299415ee20 0x7f299414b990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:42:57.558 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.554+0000 7f299c478640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f299406eba0 con 0x7f29940b30a0 2024-09-16T15:42:57.558 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.554+0000 7f299c478640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f299406ea20 con 0x7f299406ce50 2024-09-16T15:42:57.558 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.554+0000 7f299c478640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f299406ed20 con 0x7f299415ee20 2024-09-16T15:42:57.558 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.554+0000 7f2999a13640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f29940b30a0 0x7f2994152c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:42:57.558 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.554+0000 7f299a214640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f299406ce50 0x7f2994146960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:42:57.558 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.554+0000 7f2999a13640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f29940b30a0 0x7f2994152c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:40430/0 (socket says 172.21.15.123:40430) 2024-09-16T15:42:57.558 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.554+0000 7f2999a13640 1 -- 172.21.15.123:0/3966728518 learned_addr learned my addr 172.21.15.123:0/3966728518 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:42:57.558 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.554+0000 7f299aa15640 1 --2- 172.21.15.123:0/3966728518 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f299415ee20 0x7f299414b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:42:57.559 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.554+0000 7f299a214640 1 -- 172.21.15.123:0/3966728518 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f299415ee20 msgr2=0x7f299414b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:42:57.559 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.554+0000 7f299a214640 1 --2- 172.21.15.123:0/3966728518 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f299415ee20 0x7f299414b990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:42:57.559 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.554+0000 7f299a214640 1 -- 172.21.15.123:0/3966728518 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f29940b30a0 msgr2=0x7f2994152c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:42:57.559 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.554+0000 7f299a214640 1 --2- 172.21.15.123:0/3966728518 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f29940b30a0 0x7f2994152c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:42:57.559 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.554+0000 7f299a214640 1 -- 172.21.15.123:0/3966728518 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2994151900 con 0x7f299406ce50 2024-09-16T15:42:57.559 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.554+0000 7f299aa15640 1 --2- 172.21.15.123:0/3966728518 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f299415ee20 0x7f299414b990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T15:42:57.559 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.554+0000 7f1160dfa640 1 -- start start 2024-09-16T15:42:57.560 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.554+0000 7f1160dfa640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f115c06ce50 0x7f115c162060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:42:57.560 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.554+0000 7f1160dfa640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f115c06d230 0x7f115c1602c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:42:57.560 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.554+0000 7f1160dfa640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f115c0cdfc0 0x7f115c1454b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:42:57.560 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.554+0000 7f299a214640 1 --2- 172.21.15.123:0/3966728518 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f299406ce50 0x7f2994146960 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f298c0ac000 tx=0x7f298c0093d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:42:57.561 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.554+0000 7f1160dfa640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f115c074200 con 0x7f115c06ce50 2024-09-16T15:42:57.561 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.554+0000 7f1160dfa640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f115c074080 con 0x7f115c0cdfc0 2024-09-16T15:42:57.561 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.554+0000 7f1160dfa640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f115c16f210 con 0x7f115c06d230 2024-09-16T15:42:57.561 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.554+0000 7f1159d9b640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f115c06d230 0x7f115c1602c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:42:57.561 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.554+0000 7f1159d9b640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f115c06d230 0x7f115c1602c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3301/0 says I am v2:172.21.15.123:49318/0 (socket says 172.21.15.123:49318) 2024-09-16T15:42:57.562 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.554+0000 7f1159d9b640 1 -- 172.21.15.123:0/3499345705 learned_addr learned my addr 172.21.15.123:0/3499345705 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:42:57.562 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.554+0000 7f115ad9d640 1 --2- 172.21.15.123:0/3499345705 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f115c0cdfc0 0x7f115c1454b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:42:57.562 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.554+0000 7f115a59c640 1 --2- 172.21.15.123:0/3499345705 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f115c06ce50 0x7f115c162060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:42:57.562 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.558+0000 7f115ad9d640 1 -- 172.21.15.123:0/3499345705 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f115c06d230 msgr2=0x7f115c1602c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:42:57.562 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.558+0000 7f115ad9d640 1 --2- 172.21.15.123:0/3499345705 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f115c06d230 0x7f115c1602c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:42:57.562 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.558+0000 7f115ad9d640 1 -- 172.21.15.123:0/3499345705 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f115c06ce50 msgr2=0x7f115c162060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:42:57.562 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.558+0000 7f115ad9d640 1 --2- 172.21.15.123:0/3499345705 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f115c06ce50 0x7f115c162060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:42:57.562 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.558+0000 7f115ad9d640 1 -- 172.21.15.123:0/3499345705 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f115c16cfd0 con 0x7f115c0cdfc0 2024-09-16T15:42:57.562 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.558+0000 7f115a59c640 1 --2- 172.21.15.123:0/3499345705 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f115c06ce50 0x7f115c162060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T15:42:57.562 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.558+0000 7f115ad9d640 1 --2- 172.21.15.123:0/3499345705 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f115c0cdfc0 0x7f115c1454b0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f115c15e4b0 tx=0x7f11540a5b20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:42:57.563 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.558+0000 7f298affd640 1 -- 172.21.15.123:0/3966728518 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f298c0bc070 con 0x7f299406ce50 2024-09-16T15:42:57.563 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.558+0000 7f298affd640 1 -- 172.21.15.123:0/3966728518 <== mon.1 v2:172.21.15.184:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f298c004370 con 0x7f299406ce50 2024-09-16T15:42:57.563 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.558+0000 7f299c478640 1 -- 172.21.15.123:0/3966728518 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2994151be0 con 0x7f299406ce50 2024-09-16T15:42:57.563 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.558+0000 7f298affd640 1 -- 172.21.15.123:0/3966728518 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f298c0b2040 con 0x7f299406ce50 2024-09-16T15:42:57.563 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.558+0000 7f299c478640 1 -- 172.21.15.123:0/3966728518 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f29941521a0 con 0x7f299406ce50 2024-09-16T15:42:57.564 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.558+0000 7f114affd640 1 -- 172.21.15.123:0/3499345705 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11540bb070 con 0x7f115c0cdfc0 2024-09-16T15:42:57.564 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.558+0000 7f114affd640 1 -- 172.21.15.123:0/3499345705 <== mon.1 v2:172.21.15.184:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f1154004e20 con 0x7f115c0cdfc0 2024-09-16T15:42:57.564 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.558+0000 7f114affd640 1 -- 172.21.15.123:0/3499345705 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11540b2040 con 0x7f115c0cdfc0 2024-09-16T15:42:57.564 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.558+0000 7f1160dfa640 1 -- 172.21.15.123:0/3499345705 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f115c1609c0 con 0x7f115c0cdfc0 2024-09-16T15:42:57.564 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.558+0000 7f1160dfa640 1 -- 172.21.15.123:0/3499345705 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f115c1516b0 con 0x7f115c0cdfc0 2024-09-16T15:42:57.564 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.558+0000 7f299c478640 1 -- 172.21.15.123:0/3966728518 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f29940b2b60 con 0x7f299406ce50 2024-09-16T15:42:57.565 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.558+0000 7f1160dfa640 1 -- 172.21.15.123:0/3499345705 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f11400052a0 con 0x7f115c0cdfc0 2024-09-16T15:42:57.565 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.562+0000 7f114affd640 1 -- 172.21.15.123:0/3499345705 <== mon.1 v2:172.21.15.184:3300/0 4 ==== mgrmap(e 5) ==== 97080+0+0 (secure 0 0 0) 0x7f11540be070 con 0x7f115c0cdfc0 2024-09-16T15:42:57.565 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.562+0000 7f114affd640 1 --2- 172.21.15.123:0/3499345705 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f113c072f90 0x7f113c075450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:42:57.565 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.562+0000 7f114affd640 1 -- 172.21.15.123:0/3499345705 <== mon.1 v2:172.21.15.184:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f115412dee0 con 0x7f115c0cdfc0 2024-09-16T15:42:57.565 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.562+0000 7f298affd640 1 -- 172.21.15.123:0/3966728518 <== mon.1 v2:172.21.15.184:3300/0 4 ==== mgrmap(e 5) ==== 97080+0+0 (secure 0 0 0) 0x7f298c0bf070 con 0x7f299406ce50 2024-09-16T15:42:57.566 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.562+0000 7f115a59c640 1 --2- 172.21.15.123:0/3499345705 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f113c072f90 0x7f113c075450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:42:57.566 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.562+0000 7f298affd640 1 --2- 172.21.15.123:0/3966728518 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f2978072fc0 0x7f2978075480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:42:57.566 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.562+0000 7f115a59c640 1 --2- 172.21.15.123:0/3499345705 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f113c072f90 0x7f113c075450 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f115c1626a0 tx=0x7f1150009040 comp rx=0 tx=0).ready entity=mgr.4135 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:42:57.567 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.562+0000 7f298affd640 1 -- 172.21.15.123:0/3966728518 <== mon.1 v2:172.21.15.184:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f298c12dcf0 con 0x7f299406ce50 2024-09-16T15:42:57.567 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.562+0000 7f2999a13640 1 --2- 172.21.15.123:0/3966728518 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f2978072fc0 0x7f2978075480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:42:57.567 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.562+0000 7f2999a13640 1 --2- 172.21.15.123:0/3966728518 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f2978072fc0 0x7f2978075480 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f299414cf30 tx=0x7f299000b410 comp rx=0 tx=0).ready entity=mgr.4135 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:42:57.571 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.570+0000 7f429cb54640 1 -- 172.21.15.123:0/3541958959 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f429806e640 msgr2=0x7f42980cf7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:42:57.571 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.570+0000 7f429cb54640 1 --2- 172.21.15.123:0/3541958959 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f429806e640 0x7f42980cf7a0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f4290009bf0 tx=0x7f429001cb40 comp rx=0 tx=0).stop 2024-09-16T15:42:57.572 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.570+0000 7f429cb54640 1 -- 172.21.15.123:0/3541958959 shutdown_connections 2024-09-16T15:42:57.572 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.570+0000 7f429cb54640 1 --2- 172.21.15.123:0/3541958959 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f429806e640 0x7f42980cf7a0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:42:57.572 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.570+0000 7f429cb54640 1 -- 172.21.15.123:0/3541958959 >> 172.21.15.123:0/3541958959 conn(0x7f42980b2c20 msgr2=0x7f42980b3030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:42:57.572 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.570+0000 7f429cb54640 1 -- 172.21.15.123:0/3541958959 shutdown_connections 2024-09-16T15:42:57.572 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.570+0000 7f429cb54640 1 -- 172.21.15.123:0/3541958959 wait complete. 2024-09-16T15:42:57.573 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.570+0000 7f429cb54640 1 Processor -- start 2024-09-16T15:42:57.573 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.570+0000 7f429cb54640 1 -- start start 2024-09-16T15:42:57.574 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.570+0000 7f429cb54640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f429806e640 0x7f4298143b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:42:57.574 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.570+0000 7f429cb54640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f429815bbf0 0x7f4298153700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:42:57.574 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.570+0000 7f429cb54640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f429815bfd0 0x7f429814d690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:42:57.574 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.570+0000 7f429cb54640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f4298169de0 con 0x7f429815bfd0 2024-09-16T15:42:57.574 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.570+0000 7f429cb54640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f4298169c60 con 0x7f429806e640 2024-09-16T15:42:57.574 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.570+0000 7f429cb54640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f4298169f60 con 0x7f429815bbf0 2024-09-16T15:42:57.575 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.570+0000 7f4296d9d640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f429815bfd0 0x7f429814d690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:42:57.575 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.570+0000 7f4296d9d640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f429815bfd0 0x7f429814d690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:40442/0 (socket says 172.21.15.123:40442) 2024-09-16T15:42:57.575 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.570+0000 7f4296d9d640 1 -- 172.21.15.123:0/273894857 learned_addr learned my addr 172.21.15.123:0/273894857 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:42:57.575 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.570+0000 7f429659c640 1 --2- 172.21.15.123:0/273894857 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f429806e640 0x7f4298143b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:42:57.575 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.570+0000 7f4295d9b640 1 --2- 172.21.15.123:0/273894857 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f429815bbf0 0x7f4298153700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:42:57.576 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.574+0000 7f4296d9d640 1 -- 172.21.15.123:0/273894857 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f429815bbf0 msgr2=0x7f4298153700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:42:57.576 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.574+0000 7f4296d9d640 1 --2- 172.21.15.123:0/273894857 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f429815bbf0 0x7f4298153700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:42:57.576 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.574+0000 7f4296d9d640 1 -- 172.21.15.123:0/273894857 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f429806e640 msgr2=0x7f4298143b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:42:57.576 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.574+0000 7f4296d9d640 1 --2- 172.21.15.123:0/273894857 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f429806e640 0x7f4298143b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:42:57.576 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.574+0000 7f4296d9d640 1 -- 172.21.15.123:0/273894857 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4298153c40 con 0x7f429815bfd0 2024-09-16T15:42:57.576 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.574+0000 7f4296d9d640 1 --2- 172.21.15.123:0/273894857 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f429815bfd0 0x7f429814d690 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f429001c5b0 tx=0x7f4290005ae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:42:57.576 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.574+0000 7f4286ffd640 1 -- 172.21.15.123:0/273894857 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42900b2040 con 0x7f429815bfd0 2024-09-16T15:42:57.576 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.574+0000 7f4286ffd640 1 -- 172.21.15.123:0/273894857 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f42900bb070 con 0x7f429815bfd0 2024-09-16T15:42:57.576 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.574+0000 7f429cb54640 1 -- 172.21.15.123:0/273894857 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f42980711f0 con 0x7f429815bfd0 2024-09-16T15:42:57.576 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.574+0000 7f4286ffd640 1 -- 172.21.15.123:0/273894857 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42900045b0 con 0x7f429815bfd0 2024-09-16T15:42:57.583 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.574+0000 7f298affd640 1 -- 172.21.15.123:0/3966728518 <== mon.1 v2:172.21.15.184:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f298c0169b0 con 0x7f299406ce50 2024-09-16T15:42:57.584 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.574+0000 7f4286ffd640 1 -- 172.21.15.123:0/273894857 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 5) ==== 97080+0+0 (secure 0 0 0) 0x7f42900c0460 con 0x7f429815bfd0 2024-09-16T15:42:57.584 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.574+0000 7f429cb54640 1 -- 172.21.15.123:0/273894857 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f42980b02e0 con 0x7f429815bfd0 2024-09-16T15:42:57.584 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.574+0000 7f4286ffd640 1 --2- 172.21.15.123:0/273894857 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f4278072ec0 0x7f4278075380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:42:57.584 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.574+0000 7f4286ffd640 1 -- 172.21.15.123:0/273894857 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f429012c9f0 con 0x7f429815bfd0 2024-09-16T15:42:57.584 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.578+0000 7f429cb54640 1 -- 172.21.15.123:0/273894857 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f42980cf7a0 con 0x7f429815bfd0 2024-09-16T15:42:57.584 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.582+0000 7f4286ffd640 1 -- 172.21.15.123:0/273894857 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f42900167f0 con 0x7f429815bfd0 2024-09-16T15:42:57.584 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.582+0000 7f429659c640 1 --2- 172.21.15.123:0/273894857 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f4278072ec0 0x7f4278075380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:42:57.585 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.582+0000 7f429659c640 1 --2- 172.21.15.123:0/273894857 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f4278072ec0 0x7f4278075380 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f428c0010a0 tx=0x7f428c009040 comp rx=0 tx=0).ready entity=mgr.4135 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:42:57.588 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.586+0000 7f114affd640 1 -- 172.21.15.123:0/3499345705 <== mon.1 v2:172.21.15.184:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f11540c5020 con 0x7f115c0cdfc0 2024-09-16T15:42:57.592 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.590+0000 7f280c9c7640 1 -- 172.21.15.123:0/3847914558 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f280815bc10 msgr2=0x7f28080cbf00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:42:57.592 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.590+0000 7f280c9c7640 1 --2- 172.21.15.123:0/3847914558 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f280815bc10 0x7f28080cbf00 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f27f8009900 tx=0x7f27f801f0d0 comp rx=0 tx=0).stop 2024-09-16T15:42:57.592 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.590+0000 7f280c9c7640 1 -- 172.21.15.123:0/3847914558 shutdown_connections 2024-09-16T15:42:57.592 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.590+0000 7f280c9c7640 1 --2- 172.21.15.123:0/3847914558 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f28080726c0 0x7f280806e650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:42:57.592 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.590+0000 7f280c9c7640 1 --2- 172.21.15.123:0/3847914558 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f280815bc10 0x7f28080cbf00 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:42:57.592 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.590+0000 7f280c9c7640 1 -- 172.21.15.123:0/3847914558 >> 172.21.15.123:0/3847914558 conn(0x7f28080cfb10 msgr2=0x7f28080cff20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:42:57.592 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.590+0000 7f280c9c7640 1 -- 172.21.15.123:0/3847914558 shutdown_connections 2024-09-16T15:42:57.592 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.590+0000 7f280c9c7640 1 -- 172.21.15.123:0/3847914558 wait complete. 2024-09-16T15:42:57.593 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.590+0000 7f280c9c7640 1 Processor -- start 2024-09-16T15:42:57.594 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.594+0000 7f280c9c7640 1 -- start start 2024-09-16T15:42:57.596 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.594+0000 7f280c9c7640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f280806eb90 0x7f2808145ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:42:57.597 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.594+0000 7f280c9c7640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f28080726c0 0x7f280814a750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:42:57.597 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.594+0000 7f280c9c7640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f280815bc10 0x7f280815d170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:42:57.597 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.594+0000 7f280c9c7640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f28080ca870 con 0x7f28080726c0 2024-09-16T15:42:57.597 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.594+0000 7f280c9c7640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f280806f230 con 0x7f280806eb90 2024-09-16T15:42:57.597 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.594+0000 7f280c9c7640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f28080ca9f0 con 0x7f280815bc10 2024-09-16T15:42:57.597 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.594+0000 7f2805d9b640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f28080726c0 0x7f280814a750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:42:57.597 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.594+0000 7f2805d9b640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f28080726c0 0x7f280814a750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:40454/0 (socket says 172.21.15.123:40454) 2024-09-16T15:42:57.597 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.594+0000 7f2805d9b640 1 -- 172.21.15.123:0/91201568 learned_addr learned my addr 172.21.15.123:0/91201568 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:42:57.597 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.594+0000 7f2806d9d640 1 --2- 172.21.15.123:0/91201568 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f280815bc10 0x7f280815d170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:42:57.598 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.594+0000 7f2806d9d640 1 -- 172.21.15.123:0/91201568 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f280806eb90 msgr2=0x7f2808145ab0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-09-16T15:42:57.599 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.594+0000 7f2806d9d640 1 --2- 172.21.15.123:0/91201568 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f280806eb90 0x7f2808145ab0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:42:57.599 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.594+0000 7f2806d9d640 1 -- 172.21.15.123:0/91201568 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f28080726c0 msgr2=0x7f280814a750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:42:57.599 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.594+0000 7f2806d9d640 1 --2- 172.21.15.123:0/91201568 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f28080726c0 0x7f280814a750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:42:57.599 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.594+0000 7f2806d9d640 1 -- 172.21.15.123:0/91201568 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f280815e960 con 0x7f280815bc10 2024-09-16T15:42:57.599 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.594+0000 7f2805d9b640 1 --2- 172.21.15.123:0/91201568 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f28080726c0 0x7f280814a750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T15:42:57.599 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.594+0000 7f2806d9d640 1 --2- 172.21.15.123:0/91201568 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f280815bc10 0x7f280815d170 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f280000c730 tx=0x7f280000cbf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:42:57.599 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.598+0000 7f27f6ffd640 1 -- 172.21.15.123:0/91201568 <== mon.2 v2:172.21.15.184:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2800003b80 con 0x7f280815bc10 2024-09-16T15:42:57.599 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.598+0000 7f27f6ffd640 1 -- 172.21.15.123:0/91201568 <== mon.2 v2:172.21.15.184:3301/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f2800003d20 con 0x7f280815bc10 2024-09-16T15:42:57.599 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.598+0000 7f280c9c7640 1 -- 172.21.15.123:0/91201568 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f28080c58e0 con 0x7f280815bc10 2024-09-16T15:42:57.599 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.598+0000 7f280c9c7640 1 -- 172.21.15.123:0/91201568 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f28080c5bc0 con 0x7f280815bc10 2024-09-16T15:42:57.600 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.598+0000 7f27f6ffd640 1 -- 172.21.15.123:0/91201568 <== mon.2 v2:172.21.15.184:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2800005330 con 0x7f280815bc10 2024-09-16T15:42:57.602 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.598+0000 7f27f6ffd640 1 -- 172.21.15.123:0/91201568 <== mon.2 v2:172.21.15.184:3301/0 4 ==== mgrmap(e 5) ==== 97080+0+0 (secure 0 0 0) 0x7f280000a640 con 0x7f280815bc10 2024-09-16T15:42:57.602 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.602+0000 7f27f6ffd640 1 --2- 172.21.15.123:0/91201568 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f27e8072f90 0x7f27e8075450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:42:57.603 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.602+0000 7f27f6ffd640 1 -- 172.21.15.123:0/91201568 <== mon.2 v2:172.21.15.184:3301/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f2800093d00 con 0x7f280815bc10 2024-09-16T15:42:57.603 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.602+0000 7f280659c640 1 --2- 172.21.15.123:0/91201568 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f27e8072f90 0x7f27e8075450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:42:57.604 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.602+0000 7f280c9c7640 1 -- 172.21.15.123:0/91201568 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f28080cae70 con 0x7f280815bc10 2024-09-16T15:42:57.604 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.602+0000 7f280659c640 1 --2- 172.21.15.123:0/91201568 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f27e8072f90 0x7f27e8075450 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f27f80b2000 tx=0x7f27f801d910 comp rx=0 tx=0).ready entity=mgr.4135 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:42:57.618 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.614+0000 7f27f6ffd640 1 -- 172.21.15.123:0/91201568 <== mon.2 v2:172.21.15.184:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f280005efc0 con 0x7f280815bc10 2024-09-16T15:42:57.630 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.626+0000 7f58a67d3640 1 -- 172.21.15.123:0/1038429765 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f58a0161db0 msgr2=0x7f58a0162220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:42:57.631 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.626+0000 7f58a67d3640 1 --2- 172.21.15.123:0/1038429765 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f58a0161db0 0x7f58a0162220 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f58a0073bd0 tx=0x7f589801cda0 comp rx=0 tx=0).stop 2024-09-16T15:42:57.631 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.626+0000 7f58a67d3640 1 -- 172.21.15.123:0/1038429765 shutdown_connections 2024-09-16T15:42:57.631 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.626+0000 7f58a67d3640 1 --2- 172.21.15.123:0/1038429765 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f58a0161db0 0x7f58a0162220 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:42:57.631 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.626+0000 7f58a67d3640 1 --2- 172.21.15.123:0/1038429765 >> v2:172.21.15.184:3300/0 conn(0x7f58a0074800 0x7f58a015d010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:42:57.631 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.626+0000 7f58a67d3640 1 -- 172.21.15.123:0/1038429765 >> 172.21.15.123:0/1038429765 conn(0x7f58a00b1df0 msgr2=0x7f58a00b2200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:42:57.631 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.630+0000 7f58a67d3640 1 -- 172.21.15.123:0/1038429765 shutdown_connections 2024-09-16T15:42:57.631 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.630+0000 7f58a67d3640 1 -- 172.21.15.123:0/1038429765 wait complete. 2024-09-16T15:42:57.633 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.630+0000 7f58a67d3640 1 Processor -- start 2024-09-16T15:42:57.633 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.630+0000 7f58a67d3640 1 -- start start 2024-09-16T15:42:57.634 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.634+0000 7f58a67d3640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f58a0074420 0x7f58a01594b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:42:57.634 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.634+0000 7f58a67d3640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f58a0074800 0x7f58a016cb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:42:57.634 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.634+0000 7f58a67d3640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f58a0161db0 0x7f58a00c4040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:42:57.635 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.634+0000 7f58a4d70640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f58a0161db0 0x7f58a00c4040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:42:57.636 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.634+0000 7f58a4d70640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f58a0161db0 0x7f58a00c4040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:40482/0 (socket says 172.21.15.123:40482) 2024-09-16T15:42:57.636 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.634+0000 7f58a4d70640 1 -- 172.21.15.123:0/1917534694 learned_addr learned my addr 172.21.15.123:0/1917534694 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:42:57.636 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.634+0000 7f589ffff640 1 --2- 172.21.15.123:0/1917534694 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f58a0074420 0x7f58a01594b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:42:57.636 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.634+0000 7f589f7fe640 1 --2- 172.21.15.123:0/1917534694 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f58a0074800 0x7f58a016cb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:42:57.636 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.634+0000 7f58a67d3640 1 -- 172.21.15.123:0/1917534694 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f58a015d550 con 0x7f58a0161db0 2024-09-16T15:42:57.636 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.634+0000 7f58a67d3640 1 -- 172.21.15.123:0/1917534694 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f58a00cca40 con 0x7f58a0074800 2024-09-16T15:42:57.636 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.634+0000 7f58a67d3640 1 -- 172.21.15.123:0/1917534694 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f58a015d6d0 con 0x7f58a0074420 2024-09-16T15:42:57.637 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.634+0000 7f58a4d70640 1 -- 172.21.15.123:0/1917534694 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f58a0074420 msgr2=0x7f58a01594b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:42:57.637 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.634+0000 7f58a4d70640 1 --2- 172.21.15.123:0/1917534694 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f58a0074420 0x7f58a01594b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:42:57.637 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.634+0000 7f58a4d70640 1 -- 172.21.15.123:0/1917534694 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f58a0074800 msgr2=0x7f58a016cb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:42:57.637 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.634+0000 7f58a4d70640 1 --2- 172.21.15.123:0/1917534694 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f58a0074800 0x7f58a016cb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:42:57.637 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.634+0000 7f58a4d70640 1 -- 172.21.15.123:0/1917534694 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f58a01599f0 con 0x7f58a0161db0 2024-09-16T15:42:57.637 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.634+0000 7fb6ec932640 1 -- 172.21.15.123:0/1549086764 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb6d0004810 msgr2=0x7fb6d0004c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:42:57.637 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.634+0000 7fb6ec932640 1 --2- 172.21.15.123:0/1549086764 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb6d0004810 0x7fb6d0004c90 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fb6d8006f80 tx=0x7fb6d801cee0 comp rx=0 tx=0).stop 2024-09-16T15:42:57.638 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.634+0000 7f58a4d70640 1 --2- 172.21.15.123:0/1917534694 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f58a0161db0 0x7f58a00c4040 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f58980b0040 tx=0x7f58980adb80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:42:57.638 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.634+0000 7fb6ec932640 1 -- 172.21.15.123:0/1549086764 shutdown_connections 2024-09-16T15:42:57.638 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.634+0000 7fb6ec932640 1 --2- 172.21.15.123:0/1549086764 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb6d0004810 0x7fb6d0004c90 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:42:57.639 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.634+0000 7fb6ec932640 1 -- 172.21.15.123:0/1549086764 >> 172.21.15.123:0/1549086764 conn(0x7fb6e80b2c20 msgr2=0x7fb6e80b3030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:42:57.639 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.634+0000 7fb6ec932640 1 -- 172.21.15.123:0/1549086764 shutdown_connections 2024-09-16T15:42:57.639 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.634+0000 7fb6ec932640 1 -- 172.21.15.123:0/1549086764 wait complete. 2024-09-16T15:42:57.639 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.634+0000 7f589cff9640 1 -- 172.21.15.123:0/1917534694 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58980ba030 con 0x7f58a0161db0 2024-09-16T15:42:57.639 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.634+0000 7f589cff9640 1 -- 172.21.15.123:0/1917534694 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f58980add70 con 0x7f58a0161db0 2024-09-16T15:42:57.639 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.634+0000 7f589cff9640 1 -- 172.21.15.123:0/1917534694 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58980b2040 con 0x7f58a0161db0 2024-09-16T15:42:57.639 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.634+0000 7f58a67d3640 1 -- 172.21.15.123:0/1917534694 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f58a0159ca0 con 0x7f58a0161db0 2024-09-16T15:42:57.640 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.638+0000 7fb6ec932640 1 Processor -- start 2024-09-16T15:42:57.640 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.638+0000 7f589cff9640 1 -- 172.21.15.123:0/1917534694 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 5) ==== 97080+0+0 (secure 0 0 0) 0x7f58980bd070 con 0x7f58a0161db0 2024-09-16T15:42:57.640 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.638+0000 7f589cff9640 1 --2- 172.21.15.123:0/1917534694 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f588c072f10 0x7f588c0753d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:42:57.641 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.638+0000 7fb6ec932640 1 -- start start 2024-09-16T15:42:57.641 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.638+0000 7fb6ec932640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fb6d0004810 0x7fb6e81fc3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:42:57.641 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.638+0000 7fb6ec932640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb6e814c750 0x7fb6e8151fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:42:57.641 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.638+0000 7fb6ec932640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb6e81517a0 0x7fb6e80cef10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:42:57.641 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.638+0000 7fb6ec932640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fb6e80b4530 con 0x7fb6e81517a0 2024-09-16T15:42:57.641 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.638+0000 7fb6ec932640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fb6e80743a0 con 0x7fb6e814c750 2024-09-16T15:42:57.641 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.638+0000 7fb6ec932640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fb6e80b46b0 con 0x7fb6d0004810 2024-09-16T15:42:57.642 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.638+0000 7f58a67d3640 1 -- 172.21.15.123:0/1917534694 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f58a00c6610 con 0x7f58a0161db0 2024-09-16T15:42:57.642 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.638+0000 7fb6e6d9d640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb6e81517a0 0x7fb6e80cef10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:42:57.642 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.638+0000 7fb6e6d9d640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb6e81517a0 0x7fb6e80cef10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:40484/0 (socket says 172.21.15.123:40484) 2024-09-16T15:42:57.642 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.638+0000 7fb6e659c640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fb6d0004810 0x7fb6e81fc3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:42:57.642 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.638+0000 7fb6e6d9d640 1 -- 172.21.15.123:0/2511987309 learned_addr learned my addr 172.21.15.123:0/2511987309 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:42:57.643 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.638+0000 7f589cff9640 1 -- 172.21.15.123:0/1917534694 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f58980a5c30 con 0x7f58a0161db0 2024-09-16T15:42:57.643 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.638+0000 7fb6e6d9d640 1 -- 172.21.15.123:0/2511987309 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fb6d0004810 msgr2=0x7fb6e81fc3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:42:57.643 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.638+0000 7fb6e6d9d640 1 --2- 172.21.15.123:0/2511987309 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fb6d0004810 0x7fb6e81fc3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:42:57.643 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.638+0000 7fb6e6d9d640 1 -- 172.21.15.123:0/2511987309 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb6e814c750 msgr2=0x7fb6e8151fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:42:57.643 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.638+0000 7fb6e6d9d640 1 --2- 172.21.15.123:0/2511987309 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb6e814c750 0x7fb6e8151fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:42:57.643 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.638+0000 7fb6e6d9d640 1 -- 172.21.15.123:0/2511987309 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb6e81524e0 con 0x7fb6e81517a0 2024-09-16T15:42:57.643 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.638+0000 7f589ffff640 1 --2- 172.21.15.123:0/1917534694 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f588c072f10 0x7f588c0753d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:42:57.644 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.638+0000 7fb6e6d9d640 1 --2- 172.21.15.123:0/2511987309 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb6e81517a0 0x7fb6e80cef10 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fb6e0006870 tx=0x7fb6e0001aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:42:57.644 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.638+0000 7f589ffff640 1 --2- 172.21.15.123:0/1917534694 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f588c072f10 0x7f588c0753d0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f5890007fd0 tx=0x7f5890007e40 comp rx=0 tx=0).ready entity=mgr.4135 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:42:57.645 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.642+0000 7fb6d6ffd640 1 -- 172.21.15.123:0/2511987309 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6e00107d0 con 0x7fb6e81517a0 2024-09-16T15:42:57.645 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.642+0000 7fb6ec932640 1 -- 172.21.15.123:0/2511987309 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb6e8152780 con 0x7fb6e81517a0 2024-09-16T15:42:57.645 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.642+0000 7fb6ec932640 1 -- 172.21.15.123:0/2511987309 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb6e8152d40 con 0x7fb6e81517a0 2024-09-16T15:42:57.645 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.642+0000 7f58a67d3640 1 -- 172.21.15.123:0/1917534694 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f58a015ad50 con 0x7f58a0161db0 2024-09-16T15:42:57.645 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.642+0000 7fb6d6ffd640 1 -- 172.21.15.123:0/2511987309 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fb6e0016070 con 0x7fb6e81517a0 2024-09-16T15:42:57.646 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.642+0000 7fb6d4ff9640 1 -- 172.21.15.123:0/2511987309 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb6cc0052a0 con 0x7fb6e81517a0 2024-09-16T15:42:57.650 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.646+0000 7fb6d6ffd640 1 -- 172.21.15.123:0/2511987309 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6e0010d20 con 0x7fb6e81517a0 2024-09-16T15:42:57.650 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.646+0000 7fb6d6ffd640 1 -- 172.21.15.123:0/2511987309 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 5) ==== 97080+0+0 (secure 0 0 0) 0x7fb6e001b460 con 0x7fb6e81517a0 2024-09-16T15:42:57.650 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.650+0000 7fb6d6ffd640 1 --2- 172.21.15.123:0/2511987309 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fb6c8072ff0 0x7fb6c80754b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:42:57.651 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.650+0000 7fb6d6ffd640 1 -- 172.21.15.123:0/2511987309 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7fb6e0095200 con 0x7fb6e81517a0 2024-09-16T15:42:57.651 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.650+0000 7fb6e659c640 1 --2- 172.21.15.123:0/2511987309 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fb6c8072ff0 0x7fb6c80754b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:42:57.652 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.650+0000 7fa22310f640 1 -- 172.21.15.123:0/2563449531 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fa21c06e860 msgr2=0x7fa21c070760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:42:57.652 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.650+0000 7fa22310f640 1 --2- 172.21.15.123:0/2563449531 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fa21c06e860 0x7fa21c070760 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fa214009c80 tx=0x7fa21401cc50 comp rx=0 tx=0).stop 2024-09-16T15:42:57.652 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.650+0000 7fa22310f640 1 -- 172.21.15.123:0/2563449531 shutdown_connections 2024-09-16T15:42:57.652 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.650+0000 7fa22310f640 1 --2- 172.21.15.123:0/2563449531 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fa21c06e860 0x7fa21c070760 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:42:57.652 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.650+0000 7fa22310f640 1 --2- 172.21.15.123:0/2563449531 >> v2:172.21.15.184:3300/0 conn(0x7fa21c0d05e0 0x7fa21c074050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:42:57.652 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.650+0000 7fa22310f640 1 -- 172.21.15.123:0/2563449531 >> 172.21.15.123:0/2563449531 conn(0x7fa21c0b2c20 msgr2=0x7fa21c0b3030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:42:57.652 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.650+0000 7fa22310f640 1 -- 172.21.15.123:0/2563449531 shutdown_connections 2024-09-16T15:42:57.653 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.650+0000 7fb6d6ffd640 1 -- 172.21.15.123:0/2511987309 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7fb6e0060430 con 0x7fb6e81517a0 2024-09-16T15:42:57.654 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.650+0000 7fa22310f640 1 -- 172.21.15.123:0/2563449531 wait complete. 2024-09-16T15:42:57.654 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.650+0000 7fb6e659c640 1 --2- 172.21.15.123:0/2511987309 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fb6c8072ff0 0x7fb6c80754b0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fb6e81554c0 tx=0x7fb6dc005f40 comp rx=0 tx=0).ready entity=mgr.4135 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:42:57.655 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.654+0000 7fa22310f640 1 Processor -- start 2024-09-16T15:42:57.655 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.654+0000 7fa22310f640 1 -- start start 2024-09-16T15:42:57.656 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.654+0000 7fa22310f640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa21c06e860 0x7fa21c143620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:42:57.657 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.654+0000 7fa22310f640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fa21c070320 0x7fa21c0cf7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:42:57.657 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.654+0000 7fa22310f640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa21c0cfce0 0x7fa21c153ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:42:57.657 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.654+0000 7fa22310f640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fa21c0b48c0 con 0x7fa21c06e860 2024-09-16T15:42:57.657 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.654+0000 7fa22310f640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fa21c0b4740 con 0x7fa21c0cfce0 2024-09-16T15:42:57.657 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.654+0000 7fa22310f640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fa21c0b4a40 con 0x7fa21c070320 2024-09-16T15:42:57.657 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.654+0000 7fa21bfff640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fa21c070320 0x7fa21c0cf7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:42:57.657 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.654+0000 7fa21bfff640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fa21c070320 0x7fa21c0cf7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3301/0 says I am v2:172.21.15.123:49380/0 (socket says 172.21.15.123:49380) 2024-09-16T15:42:57.657 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.654+0000 7fa21bfff640 1 -- 172.21.15.123:0/2181738665 learned_addr learned my addr 172.21.15.123:0/2181738665 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:42:57.657 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.654+0000 7fa2216ac640 1 --2- 172.21.15.123:0/2181738665 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa21c0cfce0 0x7fa21c153ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:42:57.658 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.654+0000 7fa220eab640 1 --2- 172.21.15.123:0/2181738665 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa21c06e860 0x7fa21c143620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:42:57.658 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.654+0000 7fa21bfff640 1 -- 172.21.15.123:0/2181738665 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa21c0cfce0 msgr2=0x7fa21c153ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:42:57.659 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.654+0000 7fa21bfff640 1 --2- 172.21.15.123:0/2181738665 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa21c0cfce0 0x7fa21c153ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:42:57.659 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.654+0000 7fa21bfff640 1 -- 172.21.15.123:0/2181738665 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa21c06e860 msgr2=0x7fa21c143620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:42:57.659 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.654+0000 7fa21bfff640 1 --2- 172.21.15.123:0/2181738665 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa21c06e860 0x7fa21c143620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:42:57.659 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.654+0000 7fa21bfff640 1 -- 172.21.15.123:0/2181738665 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa21c0ce110 con 0x7fa21c070320 2024-09-16T15:42:57.659 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.654+0000 7fa220eab640 1 --2- 172.21.15.123:0/2181738665 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa21c06e860 0x7fa21c143620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T15:42:57.659 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.654+0000 7fa21bfff640 1 --2- 172.21.15.123:0/2181738665 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fa21c070320 0x7fa21c0cf7a0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fa21000faa0 tx=0x7fa21000ff60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:42:57.659 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.658+0000 7fa2197fa640 1 -- 172.21.15.123:0/2181738665 <== mon.2 v2:172.21.15.184:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa21001a070 con 0x7fa21c070320 2024-09-16T15:42:57.659 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.658+0000 7fa22310f640 1 -- 172.21.15.123:0/2181738665 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa21c0ce3f0 con 0x7fa21c070320 2024-09-16T15:42:57.659 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.658+0000 7fa22310f640 1 -- 172.21.15.123:0/2181738665 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fa21c0ce9b0 con 0x7fa21c070320 2024-09-16T15:42:57.659 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.658+0000 7fa2197fa640 1 -- 172.21.15.123:0/2181738665 <== mon.2 v2:172.21.15.184:3301/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fa21000d730 con 0x7fa21c070320 2024-09-16T15:42:57.660 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.658+0000 7fa2197fa640 1 -- 172.21.15.123:0/2181738665 <== mon.2 v2:172.21.15.184:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa21000dbc0 con 0x7fa21c070320 2024-09-16T15:42:57.660 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.658+0000 7fa22310f640 1 -- 172.21.15.123:0/2181738665 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa1fc0052a0 con 0x7fa21c070320 2024-09-16T15:42:57.661 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.658+0000 7fa2197fa640 1 -- 172.21.15.123:0/2181738665 <== mon.2 v2:172.21.15.184:3301/0 4 ==== mgrmap(e 5) ==== 97080+0+0 (secure 0 0 0) 0x7fa21001d070 con 0x7fa21c070320 2024-09-16T15:42:57.662 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.658+0000 7f589cff9640 1 -- 172.21.15.123:0/1917534694 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f5898016990 con 0x7f58a0161db0 2024-09-16T15:42:57.662 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.658+0000 7fa2197fa640 1 --2- 172.21.15.123:0/2181738665 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fa200072ec0 0x7fa200075380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:42:57.662 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.658+0000 7fa2197fa640 1 -- 172.21.15.123:0/2181738665 <== mon.2 v2:172.21.15.184:3301/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7fa210096a80 con 0x7fa21c070320 2024-09-16T15:42:57.662 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.662+0000 7fa220eab640 1 --2- 172.21.15.123:0/2181738665 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fa200072ec0 0x7fa200075380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:42:57.663 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.662+0000 7fa220eab640 1 --2- 172.21.15.123:0/2181738665 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fa200072ec0 0x7fa200075380 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fa21c06ed20 tx=0x7fa2080090e0 comp rx=0 tx=0).ready entity=mgr.4135 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:42:57.673 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.670+0000 7fa2197fa640 1 -- 172.21.15.123:0/2181738665 <== mon.2 v2:172.21.15.184:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7fa210024020 con 0x7fa21c070320 2024-09-16T15:42:57.889 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.886+0000 7f34e1ad7640 1 -- 172.21.15.123:0/3788276589 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) -- 0x7f34bc005590 con 0x7f34dc0b30a0 2024-09-16T15:42:57.889 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.886+0000 7f34c3fff640 1 -- 172.21.15.123:0/3788276589 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f34d405f530 con 0x7f34dc0b30a0 2024-09-16T15:42:57.890 INFO:teuthology.orchestra.run.smithi123.stdout:60129542149 2024-09-16T15:42:57.893 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.890+0000 7f34e1ad7640 1 -- 172.21.15.123:0/3788276589 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f34c8072f90 msgr2=0x7f34c8075450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:42:57.893 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.890+0000 7f34e1ad7640 1 --2- 172.21.15.123:0/3788276589 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f34c8072f90 0x7f34c8075450 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f34cc0af000 tx=0x7f34cc01b100 comp rx=0 tx=0).stop 2024-09-16T15:42:57.893 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.890+0000 7f34e1ad7640 1 -- 172.21.15.123:0/3788276589 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f34dc0b30a0 msgr2=0x7f34dc06f1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:42:57.893 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.890+0000 7f34e1ad7640 1 --2- 172.21.15.123:0/3788276589 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f34dc0b30a0 0x7f34dc06f1d0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f34d4009fd0 tx=0x7f34d400c270 comp rx=0 tx=0).stop 2024-09-16T15:42:57.893 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.890+0000 7f34e1ad7640 1 -- 172.21.15.123:0/3788276589 shutdown_connections 2024-09-16T15:42:57.893 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.890+0000 7f34e1ad7640 1 --2- 172.21.15.123:0/3788276589 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f34c8072f90 0x7f34c8075450 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:42:57.893 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.890+0000 7f34e1ad7640 1 --2- 172.21.15.123:0/3788276589 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f34dc15b5f0 0x7f34dc207430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:42:57.893 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.890+0000 7f34e1ad7640 1 --2- 172.21.15.123:0/3788276589 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f34dc0b30a0 0x7f34dc06f1d0 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:42:57.894 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.890+0000 7f34e1ad7640 1 --2- 172.21.15.123:0/3788276589 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f34dc06d130 0x7f34dc204870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:42:57.894 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.890+0000 7f34e1ad7640 1 -- 172.21.15.123:0/3788276589 >> 172.21.15.123:0/3788276589 conn(0x7f34dc0b1550 msgr2=0x7f34dc1603c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:42:57.894 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.890+0000 7f34e1ad7640 1 -- 172.21.15.123:0/3788276589 shutdown_connections 2024-09-16T15:42:57.894 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.890+0000 7f34e1ad7640 1 -- 172.21.15.123:0/3788276589 wait complete. 2024-09-16T15:42:57.911 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.910+0000 7f1160dfa640 1 -- 172.21.15.123:0/3499345705 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) -- 0x7f1140005860 con 0x7f115c0cdfc0 2024-09-16T15:42:57.911 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.910+0000 7f114affd640 1 -- 172.21.15.123:0/3499345705 <== mon.1 v2:172.21.15.184:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f1154016970 con 0x7f115c0cdfc0 2024-09-16T15:42:57.914 INFO:teuthology.orchestra.run.smithi123.stdout:60129542150 2024-09-16T15:42:57.916 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542150 got 60129542149 for osd.2 2024-09-16T15:42:57.917 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.914+0000 7f1160dfa640 1 -- 172.21.15.123:0/3499345705 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f113c072f90 msgr2=0x7f113c075450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:42:57.917 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.914+0000 7f1160dfa640 1 --2- 172.21.15.123:0/3499345705 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f113c072f90 0x7f113c075450 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f115c1626a0 tx=0x7f1150009040 comp rx=0 tx=0).stop 2024-09-16T15:42:57.917 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.914+0000 7f1160dfa640 1 -- 172.21.15.123:0/3499345705 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f115c0cdfc0 msgr2=0x7f115c1454b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:42:57.917 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.914+0000 7f1160dfa640 1 --2- 172.21.15.123:0/3499345705 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f115c0cdfc0 0x7f115c1454b0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f115c15e4b0 tx=0x7f11540a5b20 comp rx=0 tx=0).stop 2024-09-16T15:42:57.917 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.914+0000 7f1160dfa640 1 -- 172.21.15.123:0/3499345705 shutdown_connections 2024-09-16T15:42:57.918 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.914+0000 7f1160dfa640 1 --2- 172.21.15.123:0/3499345705 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f113c072f90 0x7f113c075450 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:42:57.918 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.914+0000 7f1160dfa640 1 --2- 172.21.15.123:0/3499345705 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f115c0cdfc0 0x7f115c1454b0 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:42:57.918 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.914+0000 7f1160dfa640 1 --2- 172.21.15.123:0/3499345705 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f115c06d230 0x7f115c1602c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:42:57.918 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.914+0000 7f1160dfa640 1 --2- 172.21.15.123:0/3499345705 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f115c06ce50 0x7f115c162060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:42:57.918 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.914+0000 7f1160dfa640 1 -- 172.21.15.123:0/3499345705 >> 172.21.15.123:0/3499345705 conn(0x7f115c0cd430 msgr2=0x7f115c16c8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:42:57.918 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.914+0000 7f1160dfa640 1 -- 172.21.15.123:0/3499345705 shutdown_connections 2024-09-16T15:42:57.918 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.914+0000 7f1160dfa640 1 -- 172.21.15.123:0/3499345705 wait complete. 2024-09-16T15:42:57.918 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.914+0000 7f299c478640 1 -- 172.21.15.123:0/3966728518 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) -- 0x7f2994070270 con 0x7f299406ce50 2024-09-16T15:42:57.918 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.914+0000 7f298affd640 1 -- 172.21.15.123:0/3966728518 <== mon.1 v2:172.21.15.184:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f298c0f8960 con 0x7f299406ce50 2024-09-16T15:42:57.919 INFO:teuthology.orchestra.run.smithi123.stdout:60129542148 2024-09-16T15:42:57.921 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.918+0000 7f299c478640 1 -- 172.21.15.123:0/3966728518 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f2978072fc0 msgr2=0x7f2978075480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:42:57.921 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.918+0000 7f299c478640 1 --2- 172.21.15.123:0/3966728518 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f2978072fc0 0x7f2978075480 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f299414cf30 tx=0x7f299000b410 comp rx=0 tx=0).stop 2024-09-16T15:42:57.921 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.918+0000 7f299c478640 1 -- 172.21.15.123:0/3966728518 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f299406ce50 msgr2=0x7f2994146960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:42:57.922 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.918+0000 7f299c478640 1 --2- 172.21.15.123:0/3966728518 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f299406ce50 0x7f2994146960 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f298c0ac000 tx=0x7f298c0093d0 comp rx=0 tx=0).stop 2024-09-16T15:42:57.922 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.918+0000 7f299c478640 1 -- 172.21.15.123:0/3966728518 shutdown_connections 2024-09-16T15:42:57.922 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.918+0000 7f299c478640 1 --2- 172.21.15.123:0/3966728518 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f2978072fc0 0x7f2978075480 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:42:57.922 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.918+0000 7f299c478640 1 --2- 172.21.15.123:0/3966728518 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f299415ee20 0x7f299414b990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:42:57.922 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.918+0000 7f299c478640 1 --2- 172.21.15.123:0/3966728518 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f29940b30a0 0x7f2994152c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:42:57.922 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.918+0000 7f299c478640 1 --2- 172.21.15.123:0/3966728518 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f299406ce50 0x7f2994146960 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:42:57.922 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.918+0000 7f299c478640 1 -- 172.21.15.123:0/3966728518 >> 172.21.15.123:0/3966728518 conn(0x7f29940b1550 msgr2=0x7f299406f260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:42:57.922 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.918+0000 7f299c478640 1 -- 172.21.15.123:0/3966728518 shutdown_connections 2024-09-16T15:42:57.922 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.918+0000 7f299c478640 1 -- 172.21.15.123:0/3966728518 wait complete. 2024-09-16T15:42:57.941 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.938+0000 7f429cb54640 1 -- 172.21.15.123:0/273894857 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) -- 0x7f4298075b70 con 0x7f429815bfd0 2024-09-16T15:42:57.941 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.938+0000 7f4286ffd640 1 -- 172.21.15.123:0/273894857 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f42900f9040 con 0x7f429815bfd0 2024-09-16T15:42:57.942 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542151 got 60129542150 for osd.5 2024-09-16T15:42:57.942 INFO:teuthology.orchestra.run.smithi123.stdout:60129542149 2024-09-16T15:42:57.946 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.942+0000 7f429cb54640 1 -- 172.21.15.123:0/273894857 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f4278072ec0 msgr2=0x7f4278075380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:42:57.946 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.942+0000 7f429cb54640 1 --2- 172.21.15.123:0/273894857 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f4278072ec0 0x7f4278075380 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f428c0010a0 tx=0x7f428c009040 comp rx=0 tx=0).stop 2024-09-16T15:42:57.946 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.942+0000 7f429cb54640 1 -- 172.21.15.123:0/273894857 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f429815bfd0 msgr2=0x7f429814d690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:42:57.946 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.942+0000 7f429cb54640 1 --2- 172.21.15.123:0/273894857 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f429815bfd0 0x7f429814d690 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f429001c5b0 tx=0x7f4290005ae0 comp rx=0 tx=0).stop 2024-09-16T15:42:57.946 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.942+0000 7f429cb54640 1 -- 172.21.15.123:0/273894857 shutdown_connections 2024-09-16T15:42:57.946 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.942+0000 7f429cb54640 1 --2- 172.21.15.123:0/273894857 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f4278072ec0 0x7f4278075380 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:42:57.946 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.942+0000 7f429cb54640 1 --2- 172.21.15.123:0/273894857 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f429815bfd0 0x7f429814d690 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:42:57.946 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.942+0000 7f429cb54640 1 --2- 172.21.15.123:0/273894857 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f429815bbf0 0x7f4298153700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:42:57.946 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.942+0000 7f429cb54640 1 --2- 172.21.15.123:0/273894857 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f429806e640 0x7f4298143b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:42:57.946 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.942+0000 7f429cb54640 1 -- 172.21.15.123:0/273894857 >> 172.21.15.123:0/273894857 conn(0x7f42980b2c20 msgr2=0x7f4298158340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:42:57.946 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.942+0000 7f429cb54640 1 -- 172.21.15.123:0/273894857 shutdown_connections 2024-09-16T15:42:57.947 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.942+0000 7f429cb54640 1 -- 172.21.15.123:0/273894857 wait complete. 2024-09-16T15:42:57.947 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542149 got 60129542148 for osd.4 2024-09-16T15:42:57.950 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.946+0000 7f280c9c7640 1 -- 172.21.15.123:0/91201568 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) -- 0x7f28080cd910 con 0x7f280815bc10 2024-09-16T15:42:57.950 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.950+0000 7f27f6ffd640 1 -- 172.21.15.123:0/91201568 <== mon.2 v2:172.21.15.184:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f280005e8c0 con 0x7f280815bc10 2024-09-16T15:42:57.950 INFO:teuthology.orchestra.run.smithi123.stdout:60129542149 2024-09-16T15:42:57.954 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.950+0000 7f280c9c7640 1 -- 172.21.15.123:0/91201568 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f27e8072f90 msgr2=0x7f27e8075450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:42:57.954 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.950+0000 7f280c9c7640 1 --2- 172.21.15.123:0/91201568 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f27e8072f90 0x7f27e8075450 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f27f80b2000 tx=0x7f27f801d910 comp rx=0 tx=0).stop 2024-09-16T15:42:57.954 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.950+0000 7f280c9c7640 1 -- 172.21.15.123:0/91201568 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f280815bc10 msgr2=0x7f280815d170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:42:57.954 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.950+0000 7f280c9c7640 1 --2- 172.21.15.123:0/91201568 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f280815bc10 0x7f280815d170 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f280000c730 tx=0x7f280000cbf0 comp rx=0 tx=0).stop 2024-09-16T15:42:57.954 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.950+0000 7f280c9c7640 1 -- 172.21.15.123:0/91201568 shutdown_connections 2024-09-16T15:42:57.954 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.950+0000 7f280c9c7640 1 --2- 172.21.15.123:0/91201568 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f27e8072f90 0x7f27e8075450 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:42:57.954 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.950+0000 7f280c9c7640 1 --2- 172.21.15.123:0/91201568 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f280815bc10 0x7f280815d170 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:42:57.955 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.950+0000 7f280c9c7640 1 --2- 172.21.15.123:0/91201568 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f28080726c0 0x7f280814a750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:42:57.955 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.950+0000 7f280c9c7640 1 --2- 172.21.15.123:0/91201568 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f280806eb90 0x7f2808145ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:42:57.955 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.950+0000 7f280c9c7640 1 -- 172.21.15.123:0/91201568 >> 172.21.15.123:0/91201568 conn(0x7f28080cfb10 msgr2=0x7f2808073f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:42:57.955 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.950+0000 7f280c9c7640 1 -- 172.21.15.123:0/91201568 shutdown_connections 2024-09-16T15:42:57.955 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.950+0000 7f280c9c7640 1 -- 172.21.15.123:0/91201568 wait complete. 2024-09-16T15:42:57.961 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.958+0000 7fb6d4ff9640 1 -- 172.21.15.123:0/2511987309 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) -- 0x7fb6cc005860 con 0x7fb6e81517a0 2024-09-16T15:42:57.961 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.958+0000 7fb6d6ffd640 1 -- 172.21.15.123:0/2511987309 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fb6e005fd30 con 0x7fb6e81517a0 2024-09-16T15:42:57.961 INFO:teuthology.orchestra.run.smithi123.stdout:60129542151 2024-09-16T15:42:57.963 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.962+0000 7fb6d4ff9640 1 -- 172.21.15.123:0/2511987309 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fb6c8072ff0 msgr2=0x7fb6c80754b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:42:57.963 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.962+0000 7fb6d4ff9640 1 --2- 172.21.15.123:0/2511987309 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fb6c8072ff0 0x7fb6c80754b0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fb6e81554c0 tx=0x7fb6dc005f40 comp rx=0 tx=0).stop 2024-09-16T15:42:57.964 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.962+0000 7fb6d4ff9640 1 -- 172.21.15.123:0/2511987309 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb6e81517a0 msgr2=0x7fb6e80cef10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:42:57.964 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.962+0000 7fb6d4ff9640 1 --2- 172.21.15.123:0/2511987309 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb6e81517a0 0x7fb6e80cef10 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fb6e0006870 tx=0x7fb6e0001aa0 comp rx=0 tx=0).stop 2024-09-16T15:42:57.964 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.962+0000 7fb6d4ff9640 1 -- 172.21.15.123:0/2511987309 shutdown_connections 2024-09-16T15:42:57.964 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.962+0000 7fb6d4ff9640 1 --2- 172.21.15.123:0/2511987309 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fb6c8072ff0 0x7fb6c80754b0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:42:57.964 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.962+0000 7fb6d4ff9640 1 --2- 172.21.15.123:0/2511987309 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb6e81517a0 0x7fb6e80cef10 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:42:57.964 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.962+0000 7fb6d4ff9640 1 --2- 172.21.15.123:0/2511987309 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb6e814c750 0x7fb6e8151fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:42:57.964 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.962+0000 7fb6d4ff9640 1 --2- 172.21.15.123:0/2511987309 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fb6d0004810 0x7fb6e81fc3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:42:57.964 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.962+0000 7fb6d4ff9640 1 -- 172.21.15.123:0/2511987309 >> 172.21.15.123:0/2511987309 conn(0x7fb6e80b2c20 msgr2=0x7fb6e80b0a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:42:57.964 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.962+0000 7fb6d4ff9640 1 -- 172.21.15.123:0/2511987309 shutdown_connections 2024-09-16T15:42:57.964 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.962+0000 7fb6d4ff9640 1 -- 172.21.15.123:0/2511987309 wait complete. 2024-09-16T15:42:57.968 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.966+0000 7f58a67d3640 1 -- 172.21.15.123:0/1917534694 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) -- 0x7f58a0162220 con 0x7f58a0161db0 2024-09-16T15:42:57.968 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.966+0000 7f589cff9640 1 -- 172.21.15.123:0/1917534694 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f58980f8b40 con 0x7f58a0161db0 2024-09-16T15:42:57.968 INFO:teuthology.orchestra.run.smithi123.stdout:60129542150 2024-09-16T15:42:57.970 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.970+0000 7f58a67d3640 1 -- 172.21.15.123:0/1917534694 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f588c072f10 msgr2=0x7f588c0753d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:42:57.971 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.970+0000 7f58a67d3640 1 --2- 172.21.15.123:0/1917534694 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f588c072f10 0x7f588c0753d0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f5890007fd0 tx=0x7f5890007e40 comp rx=0 tx=0).stop 2024-09-16T15:42:57.971 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.970+0000 7f58a67d3640 1 -- 172.21.15.123:0/1917534694 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f58a0161db0 msgr2=0x7f58a00c4040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:42:57.971 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.970+0000 7f58a67d3640 1 --2- 172.21.15.123:0/1917534694 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f58a0161db0 0x7f58a00c4040 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f58980b0040 tx=0x7f58980adb80 comp rx=0 tx=0).stop 2024-09-16T15:42:57.971 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.970+0000 7f58a67d3640 1 -- 172.21.15.123:0/1917534694 shutdown_connections 2024-09-16T15:42:57.971 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.970+0000 7f58a67d3640 1 --2- 172.21.15.123:0/1917534694 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f588c072f10 0x7f588c0753d0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:42:57.971 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.970+0000 7f58a67d3640 1 --2- 172.21.15.123:0/1917534694 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f58a0161db0 0x7f58a00c4040 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:42:57.971 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.970+0000 7f58a67d3640 1 --2- 172.21.15.123:0/1917534694 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f58a0074800 0x7f58a016cb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:42:57.971 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.970+0000 7f58a67d3640 1 --2- 172.21.15.123:0/1917534694 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f58a0074420 0x7f58a01594b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:42:57.971 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.970+0000 7f58a67d3640 1 -- 172.21.15.123:0/1917534694 >> 172.21.15.123:0/1917534694 conn(0x7f58a00b1df0 msgr2=0x7f58a015c9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:42:57.971 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.970+0000 7f58a67d3640 1 -- 172.21.15.123:0/1917534694 shutdown_connections 2024-09-16T15:42:57.972 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.970+0000 7f58a67d3640 1 -- 172.21.15.123:0/1917534694 wait complete. 2024-09-16T15:42:57.973 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542150 got 60129542149 for osd.6 2024-09-16T15:42:57.973 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542150 got 60129542149 for osd.3 2024-09-16T15:42:57.983 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542152 got 60129542151 for osd.1 2024-09-16T15:42:57.988 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542151 got 60129542150 for osd.7 2024-09-16T15:42:57.994 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.990+0000 7fa22310f640 1 -- 172.21.15.123:0/2181738665 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) -- 0x7fa1fc005860 con 0x7fa21c070320 2024-09-16T15:42:57.995 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.994+0000 7fa2197fa640 1 -- 172.21.15.123:0/2181738665 <== mon.2 v2:172.21.15.184:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fa2100616f0 con 0x7fa21c070320 2024-09-16T15:42:57.995 INFO:teuthology.orchestra.run.smithi123.stdout:60129542151 2024-09-16T15:42:57.997 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.994+0000 7fa22310f640 1 -- 172.21.15.123:0/2181738665 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fa200072ec0 msgr2=0x7fa200075380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:42:57.997 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.994+0000 7fa22310f640 1 --2- 172.21.15.123:0/2181738665 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fa200072ec0 0x7fa200075380 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fa21c06ed20 tx=0x7fa2080090e0 comp rx=0 tx=0).stop 2024-09-16T15:42:57.997 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.994+0000 7fa22310f640 1 -- 172.21.15.123:0/2181738665 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fa21c070320 msgr2=0x7fa21c0cf7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:42:57.997 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.994+0000 7fa22310f640 1 --2- 172.21.15.123:0/2181738665 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fa21c070320 0x7fa21c0cf7a0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fa21000faa0 tx=0x7fa21000ff60 comp rx=0 tx=0).stop 2024-09-16T15:42:57.998 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.994+0000 7fa22310f640 1 -- 172.21.15.123:0/2181738665 shutdown_connections 2024-09-16T15:42:57.998 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.994+0000 7fa22310f640 1 --2- 172.21.15.123:0/2181738665 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fa200072ec0 0x7fa200075380 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:42:57.998 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.994+0000 7fa22310f640 1 --2- 172.21.15.123:0/2181738665 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa21c0cfce0 0x7fa21c153ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:42:57.998 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.994+0000 7fa22310f640 1 --2- 172.21.15.123:0/2181738665 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fa21c070320 0x7fa21c0cf7a0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:42:57.998 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.994+0000 7fa22310f640 1 --2- 172.21.15.123:0/2181738665 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa21c06e860 0x7fa21c143620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:42:57.998 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.994+0000 7fa22310f640 1 -- 172.21.15.123:0/2181738665 >> 172.21.15.123:0/2181738665 conn(0x7fa21c0b2c20 msgr2=0x7fa21c15c720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:42:57.998 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.994+0000 7fa22310f640 1 -- 172.21.15.123:0/2181738665 shutdown_connections 2024-09-16T15:42:57.998 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:57.994+0000 7fa22310f640 1 -- 172.21.15.123:0/2181738665 wait complete. 2024-09-16T15:42:58.011 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542152 got 60129542151 for osd.0 2024-09-16T15:42:58.916 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-09-16T15:42:58.943 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-09-16T15:42:58.948 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-09-16T15:42:58.973 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2024-09-16T15:42:58.974 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-09-16T15:42:58.983 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-09-16T15:42:58.989 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2024-09-16T15:42:59.012 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-09-16T15:42:59.086 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:59.082+0000 7f9f658ec640 1 -- 172.21.15.123:0/2708434561 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9f600cdd20 msgr2=0x7f9f600ca160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:42:59.087 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:59.082+0000 7f9f658ec640 1 --2- 172.21.15.123:0/2708434561 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9f600cdd20 0x7f9f600ca160 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f9f58009c80 tx=0x7f9f5801cda0 comp rx=0 tx=0).stop 2024-09-16T15:42:59.087 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:59.086+0000 7f9f658ec640 1 -- 172.21.15.123:0/2708434561 shutdown_connections 2024-09-16T15:42:59.087 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:59.086+0000 7f9f658ec640 1 --2- 172.21.15.123:0/2708434561 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9f600cdd20 0x7f9f600ca160 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:42:59.087 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:59.086+0000 7f9f658ec640 1 -- 172.21.15.123:0/2708434561 >> 172.21.15.123:0/2708434561 conn(0x7f9f600cfb10 msgr2=0x7f9f600cff20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:42:59.087 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:59.086+0000 7f9f658ec640 1 -- 172.21.15.123:0/2708434561 shutdown_connections 2024-09-16T15:42:59.088 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:59.086+0000 7f9f658ec640 1 -- 172.21.15.123:0/2708434561 wait complete. 2024-09-16T15:42:59.089 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:59.086+0000 7f9f658ec640 1 Processor -- start 2024-09-16T15:42:59.090 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:59.086+0000 7f9f658ec640 1 -- start start 2024-09-16T15:42:59.090 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:59.086+0000 7f9f658ec640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9f600700a0 0x7f9f60156b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:42:59.090 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:59.086+0000 7f9f658ec640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f9f600cd940 0x7f9f60151720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:42:59.090 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:59.086+0000 7f9f658ec640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f9f600cdd20 0x7f9f60154c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:42:59.091 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:59.086+0000 7f9f658ec640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f9f6015dbe0 con 0x7f9f600700a0 2024-09-16T15:42:59.091 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:59.086+0000 7f9f658ec640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f9f6015da60 con 0x7f9f600cd940 2024-09-16T15:42:59.091 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:59.086+0000 7f9f658ec640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f9f6015dd60 con 0x7f9f600cdd20 2024-09-16T15:42:59.091 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:59.086+0000 7f9f5effd640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9f600700a0 0x7f9f60156b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:42:59.091 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:59.086+0000 7f9f5effd640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9f600700a0 0x7f9f60156b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:40518/0 (socket says 172.21.15.123:40518) 2024-09-16T15:42:59.091 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:59.086+0000 7f9f5effd640 1 -- 172.21.15.123:0/3285851566 learned_addr learned my addr 172.21.15.123:0/3285851566 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:42:59.091 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:59.090+0000 7f9f5f7fe640 1 --2- 172.21.15.123:0/3285851566 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f9f600cdd20 0x7f9f60154c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:42:59.091 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:59.090+0000 7f9f5e7fc640 1 --2- 172.21.15.123:0/3285851566 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f9f600cd940 0x7f9f60151720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:42:59.091 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:59.090+0000 7f9f5effd640 1 -- 172.21.15.123:0/3285851566 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f9f600cdd20 msgr2=0x7f9f60154c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:42:59.091 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:59.090+0000 7f9f5effd640 1 --2- 172.21.15.123:0/3285851566 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f9f600cdd20 0x7f9f60154c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:42:59.091 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:59.090+0000 7f9f5effd640 1 -- 172.21.15.123:0/3285851566 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f9f600cd940 msgr2=0x7f9f60151720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:42:59.091 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:59.090+0000 7f9f5effd640 1 --2- 172.21.15.123:0/3285851566 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f9f600cd940 0x7f9f60151720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:42:59.091 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:59.090+0000 7f9f5effd640 1 -- 172.21.15.123:0/3285851566 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9f60074530 con 0x7f9f600700a0 2024-09-16T15:42:59.092 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:59.090+0000 7f9f5effd640 1 --2- 172.21.15.123:0/3285851566 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9f600700a0 0x7f9f60156b30 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f9f5000c4b0 tx=0x7f9f5000c970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:42:59.092 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:59.090+0000 7f9f4ffff640 1 -- 172.21.15.123:0/3285851566 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f500052d0 con 0x7f9f600700a0 2024-09-16T15:42:59.093 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:59.090+0000 7f9f658ec640 1 -- 172.21.15.123:0/3285851566 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9f60158700 con 0x7f9f600700a0 2024-09-16T15:42:59.093 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:59.090+0000 7f9f658ec640 1 -- 172.21.15.123:0/3285851566 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9f60158cc0 con 0x7f9f600700a0 2024-09-16T15:42:59.093 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:59.090+0000 7f9f4ffff640 1 -- 172.21.15.123:0/3285851566 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f9f50005470 con 0x7f9f600700a0 2024-09-16T15:42:59.093 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:59.090+0000 7f9f4ffff640 1 -- 172.21.15.123:0/3285851566 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f50012870 con 0x7f9f600700a0 2024-09-16T15:42:59.093 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:59.090+0000 7f9f658ec640 1 -- 172.21.15.123:0/3285851566 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9f3c0052a0 con 0x7f9f600700a0 2024-09-16T15:42:59.093 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:59.090+0000 7f9f4ffff640 1 -- 172.21.15.123:0/3285851566 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 5) ==== 97080+0+0 (secure 0 0 0) 0x7f9f50012ad0 con 0x7f9f600700a0 2024-09-16T15:42:59.094 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:59.090+0000 7f9f4ffff640 1 --2- 172.21.15.123:0/3285851566 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f9f44072ec0 0x7f9f44075380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:42:59.094 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:59.094+0000 7f9f4ffff640 1 -- 172.21.15.123:0/3285851566 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f9f50093930 con 0x7f9f600700a0 2024-09-16T15:42:59.094 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:59.094+0000 7f9f5e7fc640 1 --2- 172.21.15.123:0/3285851566 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f9f44072ec0 0x7f9f44075380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:42:59.095 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:59.094+0000 7f9f5e7fc640 1 --2- 172.21.15.123:0/3285851566 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f9f44072ec0 0x7f9f44075380 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f9f600b10f0 tx=0x7f9f54009040 comp rx=0 tx=0).ready entity=mgr.4135 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:42:59.102 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:59.102+0000 7f9f4ffff640 1 -- 172.21.15.123:0/3285851566 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f9f5005eb80 con 0x7f9f600700a0 2024-09-16T15:42:59.117 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:59.114+0000 7fa6011a1640 1 -- 172.21.15.123:0/1719879785 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa5fc0d29e0 msgr2=0x7fa5fc0d1a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:42:59.117 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:59.114+0000 7fa6011a1640 1 --2- 172.21.15.123:0/1719879785 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa5fc0d29e0 0x7fa5fc0d1a60 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fa5f4009c80 tx=0x7fa5f401cd10 comp rx=0 tx=0).stop 2024-09-16T15:42:59.117 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:59.114+0000 7fa6011a1640 1 -- 172.21.15.123:0/1719879785 shutdown_connections 2024-09-16T15:42:59.117 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:59.114+0000 7fa6011a1640 1 --2- 172.21.15.123:0/1719879785 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa5fc0d29e0 0x7fa5fc0d1a60 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:42:59.117 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:59.114+0000 7fa6011a1640 1 -- 172.21.15.123:0/1719879785 >> 172.21.15.123:0/1719879785 conn(0x7fa5fc0d3c80 msgr2=0x7fa5fc0d4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:42:59.118 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:59.114+0000 7fa6011a1640 1 -- 172.21.15.123:0/1719879785 shutdown_connections 2024-09-16T15:42:59.118 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:59.114+0000 7fa6011a1640 1 -- 172.21.15.123:0/1719879785 wait complete. 2024-09-16T15:42:59.119 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:59.118+0000 7fa6011a1640 1 Processor -- start 2024-09-16T15:42:59.121 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:59.118+0000 7fa6011a1640 1 -- start start 2024-09-16T15:42:59.121 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:59.118+0000 7fa6011a1640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fa5fc0b6de0 0x7fa5fc0b4420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:42:59.121 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:59.118+0000 7fa6011a1640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa5fc0d29e0 0x7fa5fc0a11d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:42:59.121 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:59.118+0000 7fa6011a1640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa5fc165790 0x7fa5fc15ec00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:42:59.121 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:59.118+0000 7fa6011a1640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fa5fc170cc0 con 0x7fa5fc165790 2024-09-16T15:42:59.121 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:59.118+0000 7fa6011a1640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fa5fc170b40 con 0x7fa5fc0d29e0 2024-09-16T15:42:59.121 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:59.118+0000 7fa6011a1640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fa5fc170e40 con 0x7fa5fc0b6de0 2024-09-16T15:42:59.121 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:59.118+0000 7fa5fa59c640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa5fc0d29e0 0x7fa5fc0a11d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:42:59.121 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:59.118+0000 7fa5fa59c640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa5fc0d29e0 0x7fa5fc0a11d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.123:53550/0 (socket says 172.21.15.123:53550) 2024-09-16T15:42:59.121 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:59.118+0000 7fa5fa59c640 1 -- 172.21.15.123:0/2160615292 learned_addr learned my addr 172.21.15.123:0/2160615292 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:42:59.122 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:59.118+0000 7fa5fad9d640 1 --2- 172.21.15.123:0/2160615292 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fa5fc0b6de0 0x7fa5fc0b4420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:42:59.122 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:59.118+0000 7fa5fb59e640 1 --2- 172.21.15.123:0/2160615292 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa5fc165790 0x7fa5fc15ec00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:42:59.122 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:59.118+0000 7fa5fa59c640 1 -- 172.21.15.123:0/2160615292 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fa5fc0b6de0 msgr2=0x7fa5fc0b4420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:42:59.122 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:59.118+0000 7fa5fa59c640 1 --2- 172.21.15.123:0/2160615292 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fa5fc0b6de0 0x7fa5fc0b4420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:42:59.123 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:59.118+0000 7fa5fa59c640 1 -- 172.21.15.123:0/2160615292 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa5fc165790 msgr2=0x7fa5fc15ec00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:42:59.123 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:59.118+0000 7fa5fa59c640 1 --2- 172.21.15.123:0/2160615292 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa5fc165790 0x7fa5fc15ec00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:42:59.123 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:59.118+0000 7fa5fa59c640 1 -- 172.21.15.123:0/2160615292 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa5fc15f5e0 con 0x7fa5fc0d29e0 2024-09-16T15:42:59.123 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:59.118+0000 7fa5fb59e640 1 --2- 172.21.15.123:0/2160615292 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa5fc165790 0x7fa5fc15ec00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T15:42:59.123 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:59.118+0000 7fa5fa59c640 1 --2- 172.21.15.123:0/2160615292 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa5fc0d29e0 0x7fa5fc0a11d0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fa5f4000f90 tx=0x7fa5f40a5b70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:42:59.123 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:59.118+0000 7fa5e37fe640 1 -- 172.21.15.123:0/2160615292 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5f40bc070 con 0x7fa5fc0d29e0 2024-09-16T15:42:59.123 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:59.118+0000 7fa6011a1640 1 -- 172.21.15.123:0/2160615292 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa5fc15f8c0 con 0x7fa5fc0d29e0 2024-09-16T15:42:59.123 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:59.118+0000 7fa6011a1640 1 -- 172.21.15.123:0/2160615292 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa5fc15fe80 con 0x7fa5fc0d29e0 2024-09-16T15:42:59.123 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:59.122+0000 7fa5e37fe640 1 -- 172.21.15.123:0/2160615292 <== mon.1 v2:172.21.15.184:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fa5f40a5cf0 con 0x7fa5fc0d29e0 2024-09-16T15:42:59.123 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:59.122+0000 7fa5e37fe640 1 -- 172.21.15.123:0/2160615292 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5f40b2040 con 0x7fa5fc0d29e0 2024-09-16T15:42:59.123 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:59.122+0000 7fa6011a1640 1 -- 172.21.15.123:0/2160615292 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa5dc0052a0 con 0x7fa5fc0d29e0 2024-09-16T15:42:59.126 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:59.122+0000 7fa5e37fe640 1 -- 172.21.15.123:0/2160615292 <== mon.1 v2:172.21.15.184:3300/0 4 ==== mgrmap(e 5) ==== 97080+0+0 (secure 0 0 0) 0x7fa5f40bf070 con 0x7fa5fc0d29e0 2024-09-16T15:42:59.126 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:59.122+0000 7fa5e37fe640 1 --2- 172.21.15.123:0/2160615292 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fa5e8072ec0 0x7fa5e8075380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:42:59.126 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:59.122+0000 7fa5e37fe640 1 -- 172.21.15.123:0/2160615292 <== mon.1 v2:172.21.15.184:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7fa5f412dc50 con 0x7fa5fc0d29e0 2024-09-16T15:42:59.128 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:59.126+0000 7fa5fad9d640 1 --2- 172.21.15.123:0/2160615292 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fa5e8072ec0 0x7fa5e8075380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:42:59.128 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:59.126+0000 7fa5fad9d640 1 --2- 172.21.15.123:0/2160615292 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fa5e8072ec0 0x7fa5e8075380 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fa5fc0b43b0 tx=0x7fa5f00013d0 comp rx=0 tx=0).ready entity=mgr.4135 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:42:59.135 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:59.134+0000 7fcda2b06640 1 -- 172.21.15.123:0/2784951798 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcd88003a60 msgr2=0x7fcd88003e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:42:59.136 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:59.134+0000 7fcda2b06640 1 --2- 172.21.15.123:0/2784951798 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcd88003a60 0x7fcd88003e40 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fcd8c0098a0 tx=0x7fcd8c01c6a0 comp rx=0 tx=0).stop 2024-09-16T15:42:59.137 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:59.134+0000 7fcda2b06640 1 -- 172.21.15.123:0/2784951798 shutdown_connections 2024-09-16T15:42:59.138 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:59.138+0000 7fa5e37fe640 1 -- 172.21.15.123:0/2160615292 <== mon.1 v2:172.21.15.184:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7fa5f4016990 con 0x7fa5fc0d29e0 2024-09-16T15:42:59.142 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:59.134+0000 7fcda2b06640 1 --2- 172.21.15.123:0/2784951798 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcd88003a60 0x7fcd88003e40 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:42:59.142 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:59.134+0000 7fcda2b06640 1 -- 172.21.15.123:0/2784951798 >> 172.21.15.123:0/2784951798 conn(0x7fcd9c06cbf0 msgr2=0x7fcd9c06d000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:42:59.142 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:59.134+0000 7fcda2b06640 1 -- 172.21.15.123:0/2784951798 shutdown_connections 2024-09-16T15:42:59.142 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:59.134+0000 7fcda2b06640 1 -- 172.21.15.123:0/2784951798 wait complete. 2024-09-16T15:42:59.142 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:59.134+0000 7fcda2b06640 1 Processor -- start 2024-09-16T15:42:59.142 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:59.134+0000 7fcda2b06640 1 -- start start 2024-09-16T15:42:59.142 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:59.138+0000 7fcda2b06640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcd88003a60 0x7fcd9c161eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:42:59.142 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:59.138+0000 7fcda2b06640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fcd9c207f50 0x7fcd9c164da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:42:59.142 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:59.138+0000 7fcda2b06640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fcd9c2083d0 0x7fcd9c1652e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:42:59.142 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:59.138+0000 7fcda2b06640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fcd9c0cce20 con 0x7fcd88003a60 2024-09-16T15:42:59.142 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:59.138+0000 7fcda2b06640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fcd9c0ccca0 con 0x7fcd9c207f50 2024-09-16T15:42:59.142 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:59.138+0000 7fcda2b06640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fcd9c0ccfa0 con 0x7fcd9c2083d0 2024-09-16T15:42:59.143 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:59.138+0000 7fcda1b04640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcd88003a60 0x7fcd9c161eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:42:59.143 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:59.138+0000 7fcda1b04640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcd88003a60 0x7fcd9c161eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:40560/0 (socket says 172.21.15.123:40560) 2024-09-16T15:42:59.143 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:59.138+0000 7fcda1b04640 1 -- 172.21.15.123:0/3442428753 learned_addr learned my addr 172.21.15.123:0/3442428753 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:42:59.143 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:59.138+0000 7fcda1b04640 1 -- 172.21.15.123:0/3442428753 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fcd9c2083d0 msgr2=0x7fcd9c1652e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T15:42:59.143 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:59.138+0000 7fcda1b04640 1 --2- 172.21.15.123:0/3442428753 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fcd9c2083d0 0x7fcd9c1652e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:42:59.143 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:59.138+0000 7fcda1b04640 1 -- 172.21.15.123:0/3442428753 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fcd9c207f50 msgr2=0x7fcd9c164da0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T15:42:59.143 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:59.138+0000 7fcda1b04640 1 --2- 172.21.15.123:0/3442428753 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fcd9c207f50 0x7fcd9c164da0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:42:59.143 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:59.138+0000 7fcda1b04640 1 -- 172.21.15.123:0/3442428753 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcd9c1623f0 con 0x7fcd88003a60 2024-09-16T15:42:59.143 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:59.138+0000 7fcda1b04640 1 --2- 172.21.15.123:0/3442428753 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcd88003a60 0x7fcd9c161eb0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fcd8c0ac040 tx=0x7fcd8c0056c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:42:59.146 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:59.146+0000 7fcd927fc640 1 -- 172.21.15.123:0/3442428753 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd8c004910 con 0x7fcd88003a60 2024-09-16T15:42:59.151 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:59.146+0000 7fcda2b06640 1 -- 172.21.15.123:0/3442428753 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcd9c208910 con 0x7fcd88003a60 2024-09-16T15:42:59.151 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:59.146+0000 7fcda2b06640 1 -- 172.21.15.123:0/3442428753 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcd9c162780 con 0x7fcd88003a60 2024-09-16T15:42:59.151 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:59.146+0000 7fcda2b06640 1 -- 172.21.15.123:0/3442428753 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcd9c0b3790 con 0x7fcd88003a60 2024-09-16T15:42:59.151 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:59.150+0000 7fcd927fc640 1 -- 172.21.15.123:0/3442428753 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fcd8c004ba0 con 0x7fcd88003a60 2024-09-16T15:42:59.151 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:59.150+0000 7fcd927fc640 1 -- 172.21.15.123:0/3442428753 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd8c0aa4f0 con 0x7fcd88003a60 2024-09-16T15:42:59.151 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:59.150+0000 7fcd927fc640 1 -- 172.21.15.123:0/3442428753 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 5) ==== 97080+0+0 (secure 0 0 0) 0x7fcd8c0aa6b0 con 0x7fcd88003a60 2024-09-16T15:42:59.156 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:59.154+0000 7fcd927fc640 1 --2- 172.21.15.123:0/3442428753 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fcd80072fc0 0x7fcd80075480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:42:59.156 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:59.154+0000 7fcd927fc640 1 -- 172.21.15.123:0/3442428753 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7fcd8c12ea10 con 0x7fcd88003a60 2024-09-16T15:42:59.156 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:59.154+0000 7fcda1303640 1 --2- 172.21.15.123:0/3442428753 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fcd80072fc0 0x7fcd80075480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:42:59.157 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:59.154+0000 7fcda1303640 1 --2- 172.21.15.123:0/3442428753 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fcd80072fc0 0x7fcd80075480 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fcd9c0d0480 tx=0x7fcd98014040 comp rx=0 tx=0).ready entity=mgr.4135 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:42:59.158 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:59.154+0000 7fcd927fc640 1 -- 172.21.15.123:0/3442428753 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7fcd8c0163b0 con 0x7fcd88003a60 2024-09-16T15:42:59.187 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:59.186+0000 7fed776ad640 1 -- 172.21.15.123:0/122514002 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fed7016b430 msgr2=0x7fed700aecb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:42:59.187 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:59.186+0000 7fed776ad640 1 --2- 172.21.15.123:0/122514002 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fed7016b430 0x7fed700aecb0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fed68009c80 tx=0x7fed6801cd10 comp rx=0 tx=0).stop 2024-09-16T15:42:59.188 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:59.186+0000 7fed776ad640 1 -- 172.21.15.123:0/122514002 shutdown_connections 2024-09-16T15:42:59.188 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:59.186+0000 7fed776ad640 1 --2- 172.21.15.123:0/122514002 >> v2:172.21.15.184:3300/0 conn(0x7fed700723f0 0x7fed700b2e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:42:59.188 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:59.186+0000 7fed776ad640 1 --2- 172.21.15.123:0/122514002 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fed7016b430 0x7fed700aecb0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:42:59.188 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:59.186+0000 7fed776ad640 1 -- 172.21.15.123:0/122514002 >> 172.21.15.123:0/122514002 conn(0x7fed700b1550 msgr2=0x7fed700b1960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:42:59.188 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:59.186+0000 7fed776ad640 1 -- 172.21.15.123:0/122514002 shutdown_connections 2024-09-16T15:42:59.188 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:59.186+0000 7fed776ad640 1 -- 172.21.15.123:0/122514002 wait complete. 2024-09-16T15:42:59.192 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:59.186+0000 7f1b3d0be640 1 -- 172.21.15.123:0/3146367495 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f1b200053d0 msgr2=0x7f1b2000b9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:42:59.192 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:59.186+0000 7f1b3d0be640 1 --2- 172.21.15.123:0/3146367495 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f1b200053d0 0x7f1b2000b9d0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f1b3000c610 tx=0x7f1b3001d2a0 comp rx=0 tx=0).stop 2024-09-16T15:42:59.192 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:59.186+0000 7f1b3759e640 1 -- 172.21.15.123:0/3146367495 reap_dead start 2024-09-16T15:42:59.192 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:59.186+0000 7f1b3d0be640 1 -- 172.21.15.123:0/3146367495 shutdown_connections 2024-09-16T15:42:59.192 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:59.186+0000 7f1b3d0be640 1 -- 172.21.15.123:0/3146367495 >> 172.21.15.123:0/3146367495 conn(0x7f1b380ccbf0 msgr2=0x7f1b380cd000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:42:59.193 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:59.186+0000 7fed776ad640 1 Processor -- start 2024-09-16T15:42:59.193 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:59.186+0000 7f1b3d0be640 1 -- 172.21.15.123:0/3146367495 shutdown_connections 2024-09-16T15:42:59.193 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:59.186+0000 7f1b3d0be640 1 -- 172.21.15.123:0/3146367495 wait complete. 2024-09-16T15:42:59.193 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:59.190+0000 7fed776ad640 1 -- start start 2024-09-16T15:42:59.194 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:59.190+0000 7fed776ad640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fed7006fe90 0x7fed70164380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:42:59.194 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:59.190+0000 7fed776ad640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fed700723f0 0x7fed700ace60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:42:59.194 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:59.190+0000 7fed776ad640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fed7016b430 0x7fed70163a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:42:59.194 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:59.190+0000 7fed776ad640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fed70178ff0 con 0x7fed7006fe90 2024-09-16T15:42:59.194 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:59.190+0000 7fed776ad640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fed70178e70 con 0x7fed700723f0 2024-09-16T15:42:59.194 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:59.190+0000 7fed776ad640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fed70179170 con 0x7fed7016b430 2024-09-16T15:42:59.194 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:59.190+0000 7fed75449640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fed7006fe90 0x7fed70164380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:42:59.194 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:59.190+0000 7fed75449640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fed7006fe90 0x7fed70164380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:40576/0 (socket says 172.21.15.123:40576) 2024-09-16T15:42:59.194 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:59.190+0000 7fed75449640 1 -- 172.21.15.123:0/331066929 learned_addr learned my addr 172.21.15.123:0/331066929 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:42:59.194 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:59.190+0000 7f1b3d0be640 1 Processor -- start 2024-09-16T15:42:59.195 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:59.190+0000 7f1b3d0be640 1 -- start start 2024-09-16T15:42:59.195 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:59.190+0000 7fed75c4a640 1 --2- 172.21.15.123:0/331066929 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fed7016b430 0x7fed70163a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:42:59.195 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:59.190+0000 7fed74c48640 1 --2- 172.21.15.123:0/331066929 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fed700723f0 0x7fed700ace60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:42:59.195 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:59.190+0000 7f1b3d0be640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1b200053d0 0x7f1b38156d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:42:59.195 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:59.190+0000 7f1b3d0be640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f1b38145770 0x7f1b381589e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:42:59.196 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:59.190+0000 7f1b3d0be640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1b3815a9e0 0x7f1b3815ae40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:42:59.196 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:59.190+0000 7f1b3d0be640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f1b380cb5b0 con 0x7f1b200053d0 2024-09-16T15:42:59.196 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:59.190+0000 7f1b3d0be640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f1b380cb430 con 0x7f1b3815a9e0 2024-09-16T15:42:59.196 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:59.190+0000 7f1b3d0be640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f1b380cb730 con 0x7f1b38145770 2024-09-16T15:42:59.196 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:59.190+0000 7fed75c4a640 1 -- 172.21.15.123:0/331066929 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fed700723f0 msgr2=0x7fed700ace60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:42:59.196 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:59.190+0000 7fed75c4a640 1 --2- 172.21.15.123:0/331066929 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fed700723f0 0x7fed700ace60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:42:59.196 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:59.190+0000 7fed75c4a640 1 -- 172.21.15.123:0/331066929 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fed7006fe90 msgr2=0x7fed70164380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:42:59.196 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:59.190+0000 7fed75c4a640 1 --2- 172.21.15.123:0/331066929 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fed7006fe90 0x7fed70164380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:42:59.197 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:59.190+0000 7fed75c4a640 1 -- 172.21.15.123:0/331066929 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fed70165160 con 0x7fed7016b430 2024-09-16T15:42:59.197 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:59.190+0000 7f1b3659c640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f1b38145770 0x7f1b381589e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:42:59.197 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:59.190+0000 7f1b3659c640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f1b38145770 0x7f1b381589e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3301/0 says I am v2:172.21.15.123:49482/0 (socket says 172.21.15.123:49482) 2024-09-16T15:42:59.197 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:59.190+0000 7f1b3659c640 1 -- 172.21.15.123:0/1056123993 learned_addr learned my addr 172.21.15.123:0/1056123993 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:42:59.197 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:59.190+0000 7f1b36d9d640 1 --2- 172.21.15.123:0/1056123993 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1b200053d0 0x7f1b38156d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:42:59.198 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:59.190+0000 7fed75c4a640 1 --2- 172.21.15.123:0/331066929 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fed7016b430 0x7fed70163a20 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fed6000e9a0 tx=0x7fed6000ee60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:42:59.198 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:59.190+0000 7f1b3659c640 1 -- 172.21.15.123:0/1056123993 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1b3815a9e0 msgr2=0x7f1b3815ae40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:42:59.198 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:59.190+0000 7f1b3659c640 1 --2- 172.21.15.123:0/1056123993 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1b3815a9e0 0x7f1b3815ae40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:42:59.198 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:59.190+0000 7f1b3659c640 1 -- 172.21.15.123:0/1056123993 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1b200053d0 msgr2=0x7f1b38156d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:42:59.198 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:59.190+0000 7f1b3659c640 1 --2- 172.21.15.123:0/1056123993 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1b200053d0 0x7f1b38156d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:42:59.198 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:59.190+0000 7f1b3659c640 1 -- 172.21.15.123:0/1056123993 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1b381f7cd0 con 0x7f1b38145770 2024-09-16T15:42:59.199 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:59.190+0000 7fed65ffb640 1 -- 172.21.15.123:0/331066929 <== mon.2 v2:172.21.15.184:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed6000ccc0 con 0x7fed7016b430 2024-09-16T15:42:59.199 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:59.190+0000 7f1b3659c640 1 --2- 172.21.15.123:0/1056123993 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f1b38145770 0x7f1b381589e0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f1b2800b3f0 tx=0x7f1b2800b8b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:42:59.199 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:59.190+0000 7fed776ad640 1 -- 172.21.15.123:0/331066929 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fed70163f60 con 0x7fed7016b430 2024-09-16T15:42:59.199 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:59.190+0000 7fed776ad640 1 -- 172.21.15.123:0/331066929 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fed70169020 con 0x7fed7016b430 2024-09-16T15:42:59.199 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:59.190+0000 7fed65ffb640 1 -- 172.21.15.123:0/331066929 <== mon.2 v2:172.21.15.184:3301/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fed60004020 con 0x7fed7016b430 2024-09-16T15:42:59.199 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:59.190+0000 7fed65ffb640 1 -- 172.21.15.123:0/331066929 <== mon.2 v2:172.21.15.184:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed60004450 con 0x7fed7016b430 2024-09-16T15:42:59.200 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:59.190+0000 7f1b1f7fe640 1 -- 172.21.15.123:0/1056123993 <== mon.2 v2:172.21.15.184:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b28015020 con 0x7f1b38145770 2024-09-16T15:42:59.200 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:59.194+0000 7f1b1f7fe640 1 -- 172.21.15.123:0/1056123993 <== mon.2 v2:172.21.15.184:3301/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f1b28011920 con 0x7f1b38145770 2024-09-16T15:42:59.200 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:59.194+0000 7f1b1f7fe640 1 -- 172.21.15.123:0/1056123993 <== mon.2 v2:172.21.15.184:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b28011d70 con 0x7f1b38145770 2024-09-16T15:42:59.200 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:59.194+0000 7f1b3d0be640 1 -- 172.21.15.123:0/1056123993 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1b381573c0 con 0x7f1b38145770 2024-09-16T15:42:59.200 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:59.194+0000 7f1b3d0be640 1 -- 172.21.15.123:0/1056123993 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f1b381f8270 con 0x7f1b38145770 2024-09-16T15:42:59.200 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:59.194+0000 7fed776ad640 1 -- 172.21.15.123:0/331066929 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fed500052a0 con 0x7fed7016b430 2024-09-16T15:42:59.201 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:59.194+0000 7f1b1d7fa640 1 -- 172.21.15.123:0/1056123993 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1b100052a0 con 0x7f1b38145770 2024-09-16T15:42:59.202 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:59.194+0000 7fcf041ea640 1 -- 172.21.15.123:0/976350627 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcee8003a60 msgr2=0x7fcee8003e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:42:59.202 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:59.194+0000 7fcf041ea640 1 --2- 172.21.15.123:0/976350627 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcee8003a60 0x7fcee8003e80 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fcef40013e0 tx=0x7fcef401d2f0 comp rx=0 tx=0).stop 2024-09-16T15:42:59.202 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:59.194+0000 7fcf041ea640 1 -- 172.21.15.123:0/976350627 shutdown_connections 2024-09-16T15:42:59.202 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:59.194+0000 7fcf041ea640 1 --2- 172.21.15.123:0/976350627 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcee8003a60 0x7fcee8003e80 secure :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fcef40013e0 tx=0x7fcef401d2f0 comp rx=0 tx=0).stop 2024-09-16T15:42:59.202 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:59.194+0000 7fcf041ea640 1 -- 172.21.15.123:0/976350627 >> 172.21.15.123:0/976350627 conn(0x7fcefc0d3c80 msgr2=0x7fcefc0d4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:42:59.202 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:59.194+0000 7fcf041ea640 1 -- 172.21.15.123:0/976350627 shutdown_connections 2024-09-16T15:42:59.202 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:59.194+0000 7fcf041ea640 1 -- 172.21.15.123:0/976350627 wait complete. 2024-09-16T15:42:59.202 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:59.198+0000 7fcf041ea640 1 Processor -- start 2024-09-16T15:42:59.202 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:59.194+0000 7fed65ffb640 1 -- 172.21.15.123:0/331066929 <== mon.2 v2:172.21.15.184:3301/0 4 ==== mgrmap(e 5) ==== 97080+0+0 (secure 0 0 0) 0x7fed60038bb0 con 0x7fed7016b430 2024-09-16T15:42:59.203 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:59.198+0000 7fed65ffb640 1 --2- 172.21.15.123:0/331066929 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fed54072ec0 0x7fed54075380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:42:59.203 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:59.198+0000 7fed65ffb640 1 -- 172.21.15.123:0/331066929 <== mon.2 v2:172.21.15.184:3301/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7fed60094440 con 0x7fed7016b430 2024-09-16T15:42:59.203 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:59.198+0000 7fed75449640 1 --2- 172.21.15.123:0/331066929 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fed54072ec0 0x7fed54075380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:42:59.203 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:59.198+0000 7fcf041ea640 1 -- start start 2024-09-16T15:42:59.203 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:59.198+0000 7fcf041ea640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fcefc15c8c0 0x7fcefc15cca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:42:59.204 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:59.198+0000 7fcf041ea640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcefc1ff940 0x7fcefc0b4100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:42:59.204 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:59.198+0000 7fcf041ea640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fcefc0b8ee0 0x7fcefc0b9340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:42:59.204 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:59.198+0000 7fcf041ea640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fcefc0b0d70 con 0x7fcefc1ff940 2024-09-16T15:42:59.204 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:59.198+0000 7fcf041ea640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fcefc0b0bf0 con 0x7fcefc15c8c0 2024-09-16T15:42:59.204 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:59.198+0000 7fcf041ea640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fcefc0b0ef0 con 0x7fcefc0b8ee0 2024-09-16T15:42:59.204 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:59.198+0000 7fcf02787640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fcefc0b8ee0 0x7fcefc0b9340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:42:59.204 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:59.198+0000 7fcf01f86640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fcefc15c8c0 0x7fcefc15cca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:42:59.204 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:59.198+0000 7fcf02787640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fcefc0b8ee0 0x7fcefc0b9340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3301/0 says I am v2:172.21.15.123:49504/0 (socket says 172.21.15.123:49504) 2024-09-16T15:42:59.204 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:59.198+0000 7fcf02787640 1 -- 172.21.15.123:0/3293390570 learned_addr learned my addr 172.21.15.123:0/3293390570 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:42:59.204 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:59.198+0000 7f1b1f7fe640 1 -- 172.21.15.123:0/1056123993 <== mon.2 v2:172.21.15.184:3301/0 4 ==== mgrmap(e 5) ==== 97080+0+0 (secure 0 0 0) 0x7f1b28018020 con 0x7f1b38145770 2024-09-16T15:42:59.205 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:59.198+0000 7f1b1f7fe640 1 --2- 172.21.15.123:0/1056123993 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f1b24072ef0 0x7f1b240753b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:42:59.205 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:59.198+0000 7f1b1f7fe640 1 -- 172.21.15.123:0/1056123993 <== mon.2 v2:172.21.15.184:3301/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f1b28094390 con 0x7f1b38145770 2024-09-16T15:42:59.205 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:59.198+0000 7f1b36d9d640 1 --2- 172.21.15.123:0/1056123993 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f1b24072ef0 0x7f1b240753b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:42:59.205 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:59.198+0000 7fcf01785640 1 --2- 172.21.15.123:0/3293390570 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcefc1ff940 0x7fcefc0b4100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:42:59.205 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:59.198+0000 7fcf02787640 1 -- 172.21.15.123:0/3293390570 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fcefc15c8c0 msgr2=0x7fcefc15cca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:42:59.205 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:59.198+0000 7fcf02787640 1 --2- 172.21.15.123:0/3293390570 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fcefc15c8c0 0x7fcefc15cca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:42:59.205 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:59.198+0000 7fcf02787640 1 -- 172.21.15.123:0/3293390570 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcefc1ff940 msgr2=0x7fcefc0b4100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:42:59.205 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:59.198+0000 7fcf02787640 1 --2- 172.21.15.123:0/3293390570 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcefc1ff940 0x7fcefc0b4100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:42:59.206 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:59.198+0000 7fcf02787640 1 -- 172.21.15.123:0/3293390570 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcefc0b8120 con 0x7fcefc0b8ee0 2024-09-16T15:42:59.206 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:59.198+0000 7fcf02787640 1 --2- 172.21.15.123:0/3293390570 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fcefc0b8ee0 0x7fcefc0b9340 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fcefc0d3ab0 tx=0x7fcef8011b10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:42:59.206 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:59.198+0000 7f1b36d9d640 1 --2- 172.21.15.123:0/1056123993 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f1b24072ef0 0x7f1b240753b0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f1b2c00b850 tx=0x7f1b2c008040 comp rx=0 tx=0).ready entity=mgr.4135 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:42:59.206 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:59.202+0000 7fed75449640 1 --2- 172.21.15.123:0/331066929 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fed54072ec0 0x7fed54075380 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fed6c001910 tx=0x7fed6c009040 comp rx=0 tx=0).ready entity=mgr.4135 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:42:59.209 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:59.202+0000 7fd9c9113640 1 -- 172.21.15.123:0/3335576856 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd9b40052f0 msgr2=0x7fd9b400b9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:42:59.209 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:59.202+0000 7fd9c9113640 1 --2- 172.21.15.123:0/3335576856 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd9b40052f0 0x7fd9b400b9d0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fd9c406cb30 tx=0x7fd9c4098290 comp rx=0 tx=0).stop 2024-09-16T15:42:59.209 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:59.202+0000 7fd9c9113640 1 -- 172.21.15.123:0/3335576856 shutdown_connections 2024-09-16T15:42:59.209 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:59.202+0000 7fd9c9113640 1 --2- 172.21.15.123:0/3335576856 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd9b40052f0 0x7fd9b400b9d0 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:42:59.209 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:59.202+0000 7fd9c9113640 1 -- 172.21.15.123:0/3335576856 >> 172.21.15.123:0/3335576856 conn(0x7fd9bc01d780 msgr2=0x7fd9bc054570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:42:59.209 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:59.202+0000 7fd9c9113640 1 -- 172.21.15.123:0/3335576856 shutdown_connections 2024-09-16T15:42:59.209 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:59.202+0000 7fd9c9113640 1 -- 172.21.15.123:0/3335576856 wait complete. 2024-09-16T15:42:59.209 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:59.202+0000 7fd9c9113640 1 Processor -- start 2024-09-16T15:42:59.209 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:59.202+0000 7fd9c9113640 1 -- start start 2024-09-16T15:42:59.209 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:59.206+0000 7fd9c9113640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd9b40052f0 0x7fd9bc0d77d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:42:59.209 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:59.206+0000 7fd9c9113640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fd9bc0e3ea0 0x7fd9bc0da290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:42:59.210 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:59.206+0000 7fd9c9113640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd9bc039590 0x7fd9bc0d6950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:42:59.210 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:59.206+0000 7fd9c3fff640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd9b40052f0 0x7fd9bc0d77d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:42:59.210 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:59.206+0000 7fd9c9113640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fd9bc0e7270 con 0x7fd9b40052f0 2024-09-16T15:42:59.210 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:59.206+0000 7fd9c9113640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fd9bc0e70f0 con 0x7fd9bc039590 2024-09-16T15:42:59.210 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:59.206+0000 7fd9c9113640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fd9bc0e73f0 con 0x7fd9bc0e3ea0 2024-09-16T15:42:59.210 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:59.206+0000 7fd9c3fff640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd9b40052f0 0x7fd9bc0d77d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:40646/0 (socket says 172.21.15.123:40646) 2024-09-16T15:42:59.210 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:59.206+0000 7fd9c3fff640 1 -- 172.21.15.123:0/2017203069 learned_addr learned my addr 172.21.15.123:0/2017203069 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:42:59.210 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:59.206+0000 7fd9c8912640 1 --2- 172.21.15.123:0/2017203069 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd9bc039590 0x7fd9bc0d6950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:42:59.210 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:59.206+0000 7fd9c37fe640 1 --2- 172.21.15.123:0/2017203069 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fd9bc0e3ea0 0x7fd9bc0da290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:42:59.210 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:59.206+0000 7fd9c3fff640 1 -- 172.21.15.123:0/2017203069 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fd9bc0e3ea0 msgr2=0x7fd9bc0da290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:42:59.210 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:59.206+0000 7fd9c3fff640 1 --2- 172.21.15.123:0/2017203069 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fd9bc0e3ea0 0x7fd9bc0da290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:42:59.210 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:59.206+0000 7fd9c3fff640 1 -- 172.21.15.123:0/2017203069 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd9bc039590 msgr2=0x7fd9bc0d6950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:42:59.210 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:59.206+0000 7fd9c3fff640 1 --2- 172.21.15.123:0/2017203069 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd9bc039590 0x7fd9bc0d6950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:42:59.211 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:59.206+0000 7fd9c3fff640 1 -- 172.21.15.123:0/2017203069 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd9bc0d7d10 con 0x7fd9b40052f0 2024-09-16T15:42:59.211 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:59.206+0000 7fd9c3fff640 1 --2- 172.21.15.123:0/2017203069 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd9b40052f0 0x7fd9bc0d77d0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fd9b001fcc0 tx=0x7fd9b001d6b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:42:59.211 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:59.206+0000 7fd9c0ff9640 1 -- 172.21.15.123:0/2017203069 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9b001fdb0 con 0x7fd9b40052f0 2024-09-16T15:42:59.211 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:59.206+0000 7fd9c0ff9640 1 -- 172.21.15.123:0/2017203069 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fd9b0005030 con 0x7fd9b40052f0 2024-09-16T15:42:59.211 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:59.206+0000 7fd9c0ff9640 1 -- 172.21.15.123:0/2017203069 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9b0005460 con 0x7fd9b40052f0 2024-09-16T15:42:59.211 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:59.206+0000 7fd9c9113640 1 -- 172.21.15.123:0/2017203069 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd9bc0d94d0 con 0x7fd9b40052f0 2024-09-16T15:42:59.211 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:59.206+0000 7fd9c9113640 1 -- 172.21.15.123:0/2017203069 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd9bc0d9a90 con 0x7fd9b40052f0 2024-09-16T15:42:59.211 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:59.206+0000 7fd9ae7fc640 1 -- 172.21.15.123:0/2017203069 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd99c0052a0 con 0x7fd9b40052f0 2024-09-16T15:42:59.212 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:59.210+0000 7fd9c0ff9640 1 -- 172.21.15.123:0/2017203069 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 5) ==== 97080+0+0 (secure 0 0 0) 0x7fd9b00047b0 con 0x7fd9b40052f0 2024-09-16T15:42:59.212 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:59.210+0000 7fed65ffb640 1 -- 172.21.15.123:0/331066929 <== mon.2 v2:172.21.15.184:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7fed6005f020 con 0x7fed7016b430 2024-09-16T15:42:59.214 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:59.210+0000 7fd9c0ff9640 1 --2- 172.21.15.123:0/2017203069 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fd9a8072ec0 0x7fd9a8075380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:42:59.214 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:59.210+0000 7fd9c0ff9640 1 -- 172.21.15.123:0/2017203069 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7fd9b00aa060 con 0x7fd9b40052f0 2024-09-16T15:42:59.214 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:59.210+0000 7fd9c37fe640 1 --2- 172.21.15.123:0/2017203069 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fd9a8072ec0 0x7fd9a8075380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:42:59.214 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:59.210+0000 7fd9c37fe640 1 --2- 172.21.15.123:0/2017203069 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fd9a8072ec0 0x7fd9a8075380 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fd9b8001b20 tx=0x7fd9b8001ab0 comp rx=0 tx=0).ready entity=mgr.4135 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:42:59.219 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:59.218+0000 7fd9c0ff9640 1 -- 172.21.15.123:0/2017203069 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7fd9bc0d7d10 con 0x7fd9b40052f0 2024-09-16T15:42:59.226 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:59.222+0000 7fcef27fc640 1 -- 172.21.15.123:0/3293390570 <== mon.2 v2:172.21.15.184:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcef8004230 con 0x7fcefc0b8ee0 2024-09-16T15:42:59.226 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:59.222+0000 7fcef27fc640 1 -- 172.21.15.123:0/3293390570 <== mon.2 v2:172.21.15.184:3301/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fcef80043d0 con 0x7fcefc0b8ee0 2024-09-16T15:42:59.226 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:59.222+0000 7fcef27fc640 1 -- 172.21.15.123:0/3293390570 <== mon.2 v2:172.21.15.184:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcef8005280 con 0x7fcefc0b8ee0 2024-09-16T15:42:59.227 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:59.222+0000 7fcf041ea640 1 -- 172.21.15.123:0/3293390570 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcefc0b8400 con 0x7fcefc0b8ee0 2024-09-16T15:42:59.227 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:59.222+0000 7fcf041ea640 1 -- 172.21.15.123:0/3293390570 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fcefc0b89c0 con 0x7fcefc0b8ee0 2024-09-16T15:42:59.227 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:59.222+0000 7fcf041ea640 1 -- 172.21.15.123:0/3293390570 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcefc0b8bc0 con 0x7fcefc0b8ee0 2024-09-16T15:42:59.227 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:59.222+0000 7f1b1f7fe640 1 -- 172.21.15.123:0/1056123993 <== mon.2 v2:172.21.15.184:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f1b2805f5c0 con 0x7f1b38145770 2024-09-16T15:42:59.228 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:59.226+0000 7fcef27fc640 1 -- 172.21.15.123:0/3293390570 <== mon.2 v2:172.21.15.184:3301/0 4 ==== mgrmap(e 5) ==== 97080+0+0 (secure 0 0 0) 0x7fcef800f830 con 0x7fcefc0b8ee0 2024-09-16T15:42:59.228 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:59.226+0000 7fcef27fc640 1 --2- 172.21.15.123:0/3293390570 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fcee0072f90 0x7fcee0075450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:42:59.228 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:59.226+0000 7fcef27fc640 1 -- 172.21.15.123:0/3293390570 <== mon.2 v2:172.21.15.184:3301/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7fcef8094b70 con 0x7fcefc0b8ee0 2024-09-16T15:42:59.229 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:59.226+0000 7fcf01f86640 1 --2- 172.21.15.123:0/3293390570 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fcee0072f90 0x7fcee0075450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:42:59.230 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:59.226+0000 7fcf01f86640 1 --2- 172.21.15.123:0/3293390570 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fcee0072f90 0x7fcee0075450 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fcee8004790 tx=0x7fcef401a0e0 comp rx=0 tx=0).ready entity=mgr.4135 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:42:59.240 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:59.238+0000 7fcef27fc640 1 -- 172.21.15.123:0/3293390570 <== mon.2 v2:172.21.15.184:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7fcefc0b8bc0 con 0x7fcefc0b8ee0 2024-09-16T15:42:59.247 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:59.246+0000 7f972303f640 1 -- 172.21.15.123:0/1170432761 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f970c004da0 msgr2=0x7f970c005220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:42:59.247 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:59.246+0000 7f972303f640 1 --2- 172.21.15.123:0/1170432761 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f970c004da0 0x7f970c005220 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f971c0d2670 tx=0x7f971401d110 comp rx=0 tx=0).stop 2024-09-16T15:42:59.247 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:59.246+0000 7f97215dc640 1 -- 172.21.15.123:0/1170432761 reap_dead start 2024-09-16T15:42:59.247 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:59.246+0000 7f972303f640 1 -- 172.21.15.123:0/1170432761 shutdown_connections 2024-09-16T15:42:59.247 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:59.246+0000 7f972303f640 1 -- 172.21.15.123:0/1170432761 >> 172.21.15.123:0/1170432761 conn(0x7f971c09aa40 msgr2=0x7f971c09ae50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:42:59.247 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:59.246+0000 7f972303f640 1 -- 172.21.15.123:0/1170432761 shutdown_connections 2024-09-16T15:42:59.249 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:59.246+0000 7f972303f640 1 -- 172.21.15.123:0/1170432761 wait complete. 2024-09-16T15:42:59.249 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:59.246+0000 7f972303f640 1 Processor -- start 2024-09-16T15:42:59.249 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:59.246+0000 7f972303f640 1 -- start start 2024-09-16T15:42:59.251 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:59.250+0000 7f972303f640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f971c0c7da0 0x7f971c200880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:42:59.251 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:59.250+0000 7f972303f640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f971c0c5d30 0x7f971c200dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:42:59.251 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:59.250+0000 7f972303f640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f971c0cba70 0x7f971c204a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:42:59.251 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:59.250+0000 7f972303f640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f971c1649b0 con 0x7f971c0cba70 2024-09-16T15:42:59.251 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:59.250+0000 7f972303f640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f971c164830 con 0x7f971c0c5d30 2024-09-16T15:42:59.252 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:59.250+0000 7f972303f640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f971c164b30 con 0x7f971c0c7da0 2024-09-16T15:42:59.252 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:59.250+0000 7f97215dc640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f971c0cba70 0x7f971c204a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:42:59.252 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:59.250+0000 7f97215dc640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f971c0cba70 0x7f971c204a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:40664/0 (socket says 172.21.15.123:40664) 2024-09-16T15:42:59.252 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:59.250+0000 7f97215dc640 1 -- 172.21.15.123:0/2127794507 learned_addr learned my addr 172.21.15.123:0/2127794507 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:42:59.252 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:59.250+0000 7f9720ddb640 1 --2- 172.21.15.123:0/2127794507 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f971c0c7da0 0x7f971c200880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:42:59.252 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:59.250+0000 7f971bfff640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f971c0c5d30 0x7f971c200dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:42:59.253 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:59.250+0000 7f97215dc640 1 -- 172.21.15.123:0/2127794507 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f971c0c7da0 msgr2=0x7f971c200880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:42:59.253 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:59.250+0000 7f97215dc640 1 --2- 172.21.15.123:0/2127794507 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f971c0c7da0 0x7f971c200880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:42:59.253 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:59.250+0000 7f97215dc640 1 -- 172.21.15.123:0/2127794507 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f971c0c5d30 msgr2=0x7f971c200dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:42:59.253 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:59.250+0000 7f97215dc640 1 --2- 172.21.15.123:0/2127794507 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f971c0c5d30 0x7f971c200dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:42:59.253 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:59.250+0000 7f97215dc640 1 -- 172.21.15.123:0/2127794507 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f971c0c8db0 con 0x7f971c0cba70 2024-09-16T15:42:59.253 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:59.250+0000 7f97215dc640 1 --2- 172.21.15.123:0/2127794507 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f971c0cba70 0x7f971c204a10 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f97140af040 tx=0x7f97140adb80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:42:59.254 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:59.254+0000 7f97197fa640 1 -- 172.21.15.123:0/2127794507 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97140b1040 con 0x7f971c0cba70 2024-09-16T15:42:59.254 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:59.254+0000 7f97197fa640 1 -- 172.21.15.123:0/2127794507 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f97140a5e00 con 0x7f971c0cba70 2024-09-16T15:42:59.256 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:59.254+0000 7f97197fa640 1 -- 172.21.15.123:0/2127794507 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9714009a00 con 0x7f971c0cba70 2024-09-16T15:42:59.256 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:59.254+0000 7f972303f640 1 -- 172.21.15.123:0/2127794507 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f971c0c61c0 con 0x7f971c0cba70 2024-09-16T15:42:59.256 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:59.254+0000 7f972303f640 1 -- 172.21.15.123:0/2127794507 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f971c0c64a0 con 0x7f971c0cba70 2024-09-16T15:42:59.257 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:59.254+0000 7f972303f640 1 -- 172.21.15.123:0/2127794507 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f971c06d6a0 con 0x7f971c0cba70 2024-09-16T15:42:59.258 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:59.254+0000 7f97197fa640 1 -- 172.21.15.123:0/2127794507 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 5) ==== 97080+0+0 (secure 0 0 0) 0x7f97140bf060 con 0x7f971c0cba70 2024-09-16T15:42:59.258 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:59.258+0000 7f97197fa640 1 --2- 172.21.15.123:0/2127794507 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f970c074690 0x7f970c076b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:42:59.259 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:59.258+0000 7f97197fa640 1 -- 172.21.15.123:0/2127794507 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f971412db10 con 0x7f971c0cba70 2024-09-16T15:42:59.259 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:59.258+0000 7f9720ddb640 1 --2- 172.21.15.123:0/2127794507 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f970c074690 0x7f970c076b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:42:59.260 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:59.258+0000 7f9720ddb640 1 --2- 172.21.15.123:0/2127794507 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f970c074690 0x7f970c076b50 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f9708001870 tx=0x7f97080017c0 comp rx=0 tx=0).ready entity=mgr.4135 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:42:59.267 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:59.266+0000 7f97197fa640 1 -- 172.21.15.123:0/2127794507 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f9714016e20 con 0x7f971c0cba70 2024-09-16T15:42:59.446 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:59.442+0000 7f9f658ec640 1 -- 172.21.15.123:0/3285851566 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) -- 0x7f9f3c005860 con 0x7f9f600700a0 2024-09-16T15:42:59.446 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:59.446+0000 7f9f4ffff640 1 -- 172.21.15.123:0/3285851566 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f9f5005e480 con 0x7f9f600700a0 2024-09-16T15:42:59.450 INFO:teuthology.orchestra.run.smithi123.stdout:60129542149 2024-09-16T15:42:59.455 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:59.454+0000 7f9f4dffb640 1 -- 172.21.15.123:0/3285851566 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f9f44072ec0 msgr2=0x7f9f44075380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:42:59.455 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:59.454+0000 7f9f4dffb640 1 --2- 172.21.15.123:0/3285851566 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f9f44072ec0 0x7f9f44075380 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f9f600b10f0 tx=0x7f9f54009040 comp rx=0 tx=0).stop 2024-09-16T15:42:59.455 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:59.454+0000 7f9f4dffb640 1 -- 172.21.15.123:0/3285851566 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9f600700a0 msgr2=0x7f9f60156b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:42:59.455 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:59.454+0000 7f9f4dffb640 1 --2- 172.21.15.123:0/3285851566 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9f600700a0 0x7f9f60156b30 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f9f5000c4b0 tx=0x7f9f5000c970 comp rx=0 tx=0).stop 2024-09-16T15:42:59.455 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:59.454+0000 7f9f4dffb640 1 -- 172.21.15.123:0/3285851566 shutdown_connections 2024-09-16T15:42:59.455 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:59.454+0000 7f9f4dffb640 1 --2- 172.21.15.123:0/3285851566 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f9f44072ec0 0x7f9f44075380 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:42:59.456 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:59.454+0000 7f9f4dffb640 1 --2- 172.21.15.123:0/3285851566 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f9f600cdd20 0x7f9f60154c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:42:59.456 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:59.454+0000 7f9f4dffb640 1 --2- 172.21.15.123:0/3285851566 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f9f600cd940 0x7f9f60151720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:42:59.456 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:59.454+0000 7f9f4dffb640 1 --2- 172.21.15.123:0/3285851566 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9f600700a0 0x7f9f60156b30 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:42:59.456 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:59.454+0000 7f9f4dffb640 1 -- 172.21.15.123:0/3285851566 >> 172.21.15.123:0/3285851566 conn(0x7f9f600cfb10 msgr2=0x7f9f6006eb20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:42:59.456 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:59.454+0000 7f9f4dffb640 1 -- 172.21.15.123:0/3285851566 shutdown_connections 2024-09-16T15:42:59.456 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:59.454+0000 7f9f4dffb640 1 -- 172.21.15.123:0/3285851566 wait complete. 2024-09-16T15:42:59.478 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542150 got 60129542149 for osd.2 2024-09-16T15:42:59.502 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:59.498+0000 7fcda2b06640 1 -- 172.21.15.123:0/3442428753 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) -- 0x7fcd9c16a1e0 con 0x7fcd88003a60 2024-09-16T15:42:59.502 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:59.502+0000 7fcd927fc640 1 -- 172.21.15.123:0/3442428753 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fcd8c100030 con 0x7fcd88003a60 2024-09-16T15:42:59.502 INFO:teuthology.orchestra.run.smithi123.stdout:60129542151 2024-09-16T15:42:59.506 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:59.502+0000 7fcda2b06640 1 -- 172.21.15.123:0/3442428753 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fcd80072fc0 msgr2=0x7fcd80075480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:42:59.507 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:59.502+0000 7fcda2b06640 1 --2- 172.21.15.123:0/3442428753 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fcd80072fc0 0x7fcd80075480 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fcd9c0d0480 tx=0x7fcd98014040 comp rx=0 tx=0).stop 2024-09-16T15:42:59.507 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:59.502+0000 7fcda2b06640 1 -- 172.21.15.123:0/3442428753 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcd88003a60 msgr2=0x7fcd9c161eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:42:59.507 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:59.502+0000 7fcda2b06640 1 --2- 172.21.15.123:0/3442428753 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcd88003a60 0x7fcd9c161eb0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fcd8c0ac040 tx=0x7fcd8c0056c0 comp rx=0 tx=0).stop 2024-09-16T15:42:59.507 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:59.502+0000 7fcda2b06640 1 -- 172.21.15.123:0/3442428753 shutdown_connections 2024-09-16T15:42:59.507 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:59.502+0000 7fcda2b06640 1 --2- 172.21.15.123:0/3442428753 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fcd80072fc0 0x7fcd80075480 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:42:59.507 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:59.502+0000 7fcda2b06640 1 --2- 172.21.15.123:0/3442428753 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fcd9c2083d0 0x7fcd9c1652e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:42:59.507 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:59.506+0000 7fcda2b06640 1 --2- 172.21.15.123:0/3442428753 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fcd9c207f50 0x7fcd9c164da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:42:59.507 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:59.506+0000 7fcda2b06640 1 --2- 172.21.15.123:0/3442428753 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcd88003a60 0x7fcd9c161eb0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:42:59.507 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:59.506+0000 7fcda2b06640 1 -- 172.21.15.123:0/3442428753 >> 172.21.15.123:0/3442428753 conn(0x7fcd9c06cbf0 msgr2=0x7fcd9c0cd8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:42:59.507 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:59.506+0000 7fcda2b06640 1 -- 172.21.15.123:0/3442428753 shutdown_connections 2024-09-16T15:42:59.507 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:59.506+0000 7fcda2b06640 1 -- 172.21.15.123:0/3442428753 wait complete. 2024-09-16T15:42:59.509 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:59.506+0000 7fa6011a1640 1 -- 172.21.15.123:0/2160615292 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) -- 0x7fa5dc005860 con 0x7fa5fc0d29e0 2024-09-16T15:42:59.509 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:59.506+0000 7fa5e37fe640 1 -- 172.21.15.123:0/2160615292 <== mon.1 v2:172.21.15.184:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fa5f40f8840 con 0x7fa5fc0d29e0 2024-09-16T15:42:59.510 INFO:teuthology.orchestra.run.smithi123.stdout:60129542149 2024-09-16T15:42:59.514 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:59.510+0000 7fa6011a1640 1 -- 172.21.15.123:0/2160615292 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fa5e8072ec0 msgr2=0x7fa5e8075380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:42:59.514 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:59.510+0000 7fa6011a1640 1 --2- 172.21.15.123:0/2160615292 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fa5e8072ec0 0x7fa5e8075380 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fa5fc0b43b0 tx=0x7fa5f00013d0 comp rx=0 tx=0).stop 2024-09-16T15:42:59.515 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:59.510+0000 7fa6011a1640 1 -- 172.21.15.123:0/2160615292 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa5fc0d29e0 msgr2=0x7fa5fc0a11d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:42:59.515 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:59.510+0000 7fa6011a1640 1 --2- 172.21.15.123:0/2160615292 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa5fc0d29e0 0x7fa5fc0a11d0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fa5f4000f90 tx=0x7fa5f40a5b70 comp rx=0 tx=0).stop 2024-09-16T15:42:59.515 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:59.510+0000 7fa6011a1640 1 -- 172.21.15.123:0/2160615292 shutdown_connections 2024-09-16T15:42:59.515 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:59.510+0000 7fa6011a1640 1 --2- 172.21.15.123:0/2160615292 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fa5e8072ec0 0x7fa5e8075380 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:42:59.515 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:59.510+0000 7fa6011a1640 1 --2- 172.21.15.123:0/2160615292 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa5fc165790 0x7fa5fc15ec00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:42:59.515 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:59.510+0000 7fa6011a1640 1 --2- 172.21.15.123:0/2160615292 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa5fc0d29e0 0x7fa5fc0a11d0 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:42:59.515 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:59.510+0000 7fa6011a1640 1 --2- 172.21.15.123:0/2160615292 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fa5fc0b6de0 0x7fa5fc0b4420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:42:59.515 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:59.510+0000 7fa6011a1640 1 -- 172.21.15.123:0/2160615292 >> 172.21.15.123:0/2160615292 conn(0x7fa5fc0d3c80 msgr2=0x7fa5fc0b07d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:42:59.515 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:59.514+0000 7fa6011a1640 1 -- 172.21.15.123:0/2160615292 shutdown_connections 2024-09-16T15:42:59.515 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:59.514+0000 7fa6011a1640 1 -- 172.21.15.123:0/2160615292 wait complete. 2024-09-16T15:42:59.532 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542151 got 60129542151 for osd.5 2024-09-16T15:42:59.533 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:59.530+0000 7fcf041ea640 1 -- 172.21.15.123:0/3293390570 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) -- 0x7fcefc15fb20 con 0x7fcefc0b8ee0 2024-09-16T15:42:59.533 DEBUG:teuthology.parallel:result is None 2024-09-16T15:42:59.533 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:59.530+0000 7fcef27fc640 1 -- 172.21.15.123:0/3293390570 <== mon.2 v2:172.21.15.184:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fcef805f7e0 con 0x7fcefc0b8ee0 2024-09-16T15:42:59.533 INFO:teuthology.orchestra.run.smithi123.stdout:60129542152 2024-09-16T15:42:59.535 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:59.534+0000 7fed776ad640 1 -- 172.21.15.123:0/331066929 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) -- 0x7fed50005860 con 0x7fed7016b430 2024-09-16T15:42:59.536 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:59.534+0000 7fcf041ea640 1 -- 172.21.15.123:0/3293390570 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fcee0072f90 msgr2=0x7fcee0075450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:42:59.536 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:59.534+0000 7fcf041ea640 1 --2- 172.21.15.123:0/3293390570 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fcee0072f90 0x7fcee0075450 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fcee8004790 tx=0x7fcef401a0e0 comp rx=0 tx=0).stop 2024-09-16T15:42:59.537 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:59.534+0000 7fcf041ea640 1 -- 172.21.15.123:0/3293390570 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fcefc0b8ee0 msgr2=0x7fcefc0b9340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:42:59.537 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:59.534+0000 7fcf041ea640 1 --2- 172.21.15.123:0/3293390570 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fcefc0b8ee0 0x7fcefc0b9340 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fcefc0d3ab0 tx=0x7fcef8011b10 comp rx=0 tx=0).stop 2024-09-16T15:42:59.537 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:59.534+0000 7fcf041ea640 1 -- 172.21.15.123:0/3293390570 shutdown_connections 2024-09-16T15:42:59.537 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:59.534+0000 7fcf041ea640 1 --2- 172.21.15.123:0/3293390570 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fcee0072f90 0x7fcee0075450 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:42:59.537 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:59.534+0000 7fcf041ea640 1 --2- 172.21.15.123:0/3293390570 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fcefc0b8ee0 0x7fcefc0b9340 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:42:59.537 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:59.534+0000 7fcf041ea640 1 --2- 172.21.15.123:0/3293390570 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcefc1ff940 0x7fcefc0b4100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:42:59.537 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:59.534+0000 7fcf041ea640 1 --2- 172.21.15.123:0/3293390570 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fcefc15c8c0 0x7fcefc15cca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:42:59.537 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:59.534+0000 7fcf041ea640 1 -- 172.21.15.123:0/3293390570 >> 172.21.15.123:0/3293390570 conn(0x7fcefc0d3c80 msgr2=0x7fcefc0d22f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:42:59.537 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:59.534+0000 7fcf041ea640 1 -- 172.21.15.123:0/3293390570 shutdown_connections 2024-09-16T15:42:59.537 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:59.534+0000 7fcf041ea640 1 -- 172.21.15.123:0/3293390570 wait complete. 2024-09-16T15:42:59.537 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:59.534+0000 7fed65ffb640 1 -- 172.21.15.123:0/331066929 <== mon.2 v2:172.21.15.184:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fed60063a50 con 0x7fed7016b430 2024-09-16T15:42:59.538 INFO:teuthology.orchestra.run.smithi123.stdout:60129542150 2024-09-16T15:42:59.540 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:59.538+0000 7fed776ad640 1 -- 172.21.15.123:0/331066929 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fed54072ec0 msgr2=0x7fed54075380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:42:59.540 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:59.538+0000 7fed776ad640 1 --2- 172.21.15.123:0/331066929 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fed54072ec0 0x7fed54075380 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fed6c001910 tx=0x7fed6c009040 comp rx=0 tx=0).stop 2024-09-16T15:42:59.540 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:59.538+0000 7fed776ad640 1 -- 172.21.15.123:0/331066929 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fed7016b430 msgr2=0x7fed70163a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:42:59.540 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:59.538+0000 7fed776ad640 1 --2- 172.21.15.123:0/331066929 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fed7016b430 0x7fed70163a20 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fed6000e9a0 tx=0x7fed6000ee60 comp rx=0 tx=0).stop 2024-09-16T15:42:59.540 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:59.538+0000 7fed776ad640 1 -- 172.21.15.123:0/331066929 shutdown_connections 2024-09-16T15:42:59.541 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:59.538+0000 7fed776ad640 1 --2- 172.21.15.123:0/331066929 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fed54072ec0 0x7fed54075380 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:42:59.541 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:59.538+0000 7fed776ad640 1 --2- 172.21.15.123:0/331066929 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fed7016b430 0x7fed70163a20 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:42:59.541 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:59.538+0000 7fed776ad640 1 --2- 172.21.15.123:0/331066929 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fed700723f0 0x7fed700ace60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:42:59.541 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:59.538+0000 7fed776ad640 1 --2- 172.21.15.123:0/331066929 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fed7006fe90 0x7fed70164380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:42:59.541 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:59.538+0000 7fed776ad640 1 -- 172.21.15.123:0/331066929 >> 172.21.15.123:0/331066929 conn(0x7fed700b1550 msgr2=0x7fed700af760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:42:59.541 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:59.538+0000 7fed776ad640 1 -- 172.21.15.123:0/331066929 shutdown_connections 2024-09-16T15:42:59.541 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:59.538+0000 7fed776ad640 1 -- 172.21.15.123:0/331066929 wait complete. 2024-09-16T15:42:59.542 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542149 got 60129542149 for osd.4 2024-09-16T15:42:59.542 DEBUG:teuthology.parallel:result is None 2024-09-16T15:42:59.554 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542152 got 60129542152 for osd.1 2024-09-16T15:42:59.554 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:59.550+0000 7fd9ae7fc640 1 -- 172.21.15.123:0/2017203069 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) -- 0x7fd99c005860 con 0x7fd9b40052f0 2024-09-16T15:42:59.555 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:59.554+0000 7fd9c0ff9640 1 -- 172.21.15.123:0/2017203069 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fd9b00704d0 con 0x7fd9b40052f0 2024-09-16T15:42:59.555 DEBUG:teuthology.parallel:result is None 2024-09-16T15:42:59.555 INFO:teuthology.orchestra.run.smithi123.stdout:60129542150 2024-09-16T15:42:59.557 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:59.554+0000 7fd9ae7fc640 1 -- 172.21.15.123:0/2017203069 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fd9a8072ec0 msgr2=0x7fd9a8075380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:42:59.557 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:59.554+0000 7fd9ae7fc640 1 --2- 172.21.15.123:0/2017203069 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fd9a8072ec0 0x7fd9a8075380 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fd9b8001b20 tx=0x7fd9b8001ab0 comp rx=0 tx=0).stop 2024-09-16T15:42:59.557 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:59.554+0000 7fd9ae7fc640 1 -- 172.21.15.123:0/2017203069 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd9b40052f0 msgr2=0x7fd9bc0d77d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:42:59.557 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:59.554+0000 7fd9ae7fc640 1 --2- 172.21.15.123:0/2017203069 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd9b40052f0 0x7fd9bc0d77d0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fd9b001fcc0 tx=0x7fd9b001d6b0 comp rx=0 tx=0).stop 2024-09-16T15:42:59.557 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:59.554+0000 7fd9ae7fc640 1 -- 172.21.15.123:0/2017203069 shutdown_connections 2024-09-16T15:42:59.557 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:59.554+0000 7fd9ae7fc640 1 --2- 172.21.15.123:0/2017203069 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fd9a8072ec0 0x7fd9a8075380 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:42:59.558 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:59.554+0000 7fd9ae7fc640 1 --2- 172.21.15.123:0/2017203069 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd9bc039590 0x7fd9bc0d6950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:42:59.558 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:59.554+0000 7fd9ae7fc640 1 --2- 172.21.15.123:0/2017203069 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fd9bc0e3ea0 0x7fd9bc0da290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:42:59.558 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:59.554+0000 7fd9ae7fc640 1 --2- 172.21.15.123:0/2017203069 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd9b40052f0 0x7fd9bc0d77d0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:42:59.558 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:59.554+0000 7fd9ae7fc640 1 -- 172.21.15.123:0/2017203069 >> 172.21.15.123:0/2017203069 conn(0x7fd9bc01d780 msgr2=0x7fd9bc0062d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:42:59.558 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:59.554+0000 7fd9ae7fc640 1 -- 172.21.15.123:0/2017203069 shutdown_connections 2024-09-16T15:42:59.558 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:59.554+0000 7fd9ae7fc640 1 -- 172.21.15.123:0/2017203069 wait complete. 2024-09-16T15:42:59.566 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542150 got 60129542150 for osd.3 2024-09-16T15:42:59.566 DEBUG:teuthology.parallel:result is None 2024-09-16T15:42:59.572 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:59.570+0000 7f972303f640 1 -- 172.21.15.123:0/2127794507 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) -- 0x7f971c0cfbc0 con 0x7f971c0cba70 2024-09-16T15:42:59.572 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:59.570+0000 7f97197fa640 1 -- 172.21.15.123:0/2127794507 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f97140f8660 con 0x7f971c0cba70 2024-09-16T15:42:59.572 INFO:teuthology.orchestra.run.smithi123.stdout:60129542152 2024-09-16T15:42:59.572 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542150 got 60129542150 for osd.6 2024-09-16T15:42:59.573 DEBUG:teuthology.parallel:result is None 2024-09-16T15:42:59.574 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:59.570+0000 7f972303f640 1 -- 172.21.15.123:0/2127794507 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f970c074690 msgr2=0x7f970c076b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:42:59.574 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:59.570+0000 7f972303f640 1 --2- 172.21.15.123:0/2127794507 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f970c074690 0x7f970c076b50 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f9708001870 tx=0x7f97080017c0 comp rx=0 tx=0).stop 2024-09-16T15:42:59.574 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:59.570+0000 7f972303f640 1 -- 172.21.15.123:0/2127794507 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f971c0cba70 msgr2=0x7f971c204a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:42:59.574 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:59.570+0000 7f972303f640 1 --2- 172.21.15.123:0/2127794507 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f971c0cba70 0x7f971c204a10 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f97140af040 tx=0x7f97140adb80 comp rx=0 tx=0).stop 2024-09-16T15:42:59.574 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:59.570+0000 7f972303f640 1 -- 172.21.15.123:0/2127794507 shutdown_connections 2024-09-16T15:42:59.574 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:59.570+0000 7f972303f640 1 --2- 172.21.15.123:0/2127794507 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f970c074690 0x7f970c076b50 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:42:59.575 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:59.570+0000 7f972303f640 1 --2- 172.21.15.123:0/2127794507 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f971c0cba70 0x7f971c204a10 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:42:59.575 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:59.570+0000 7f972303f640 1 --2- 172.21.15.123:0/2127794507 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f971c0c5d30 0x7f971c200dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:42:59.575 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:59.570+0000 7f972303f640 1 --2- 172.21.15.123:0/2127794507 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f971c0c7da0 0x7f971c200880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:42:59.575 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:59.570+0000 7f972303f640 1 -- 172.21.15.123:0/2127794507 >> 172.21.15.123:0/2127794507 conn(0x7f971c09aa40 msgr2=0x7f971c09ae50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:42:59.575 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:59.570+0000 7f972303f640 1 -- 172.21.15.123:0/2127794507 shutdown_connections 2024-09-16T15:42:59.575 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:59.570+0000 7f972303f640 1 -- 172.21.15.123:0/2127794507 wait complete. 2024-09-16T15:42:59.575 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:59.574+0000 7f1b1d7fa640 1 -- 172.21.15.123:0/1056123993 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) -- 0x7f1b10005860 con 0x7f1b38145770 2024-09-16T15:42:59.575 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:59.574+0000 7f1b1f7fe640 1 -- 172.21.15.123:0/1056123993 <== mon.2 v2:172.21.15.184:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f1b2805eec0 con 0x7f1b38145770 2024-09-16T15:42:59.576 INFO:teuthology.orchestra.run.smithi123.stdout:60129542151 2024-09-16T15:42:59.578 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:59.574+0000 7f1b1d7fa640 1 -- 172.21.15.123:0/1056123993 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f1b24072ef0 msgr2=0x7f1b240753b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:42:59.578 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:59.574+0000 7f1b1d7fa640 1 --2- 172.21.15.123:0/1056123993 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f1b24072ef0 0x7f1b240753b0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f1b2c00b850 tx=0x7f1b2c008040 comp rx=0 tx=0).stop 2024-09-16T15:42:59.578 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:59.574+0000 7f1b1d7fa640 1 -- 172.21.15.123:0/1056123993 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f1b38145770 msgr2=0x7f1b381589e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:42:59.578 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:59.574+0000 7f1b1d7fa640 1 --2- 172.21.15.123:0/1056123993 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f1b38145770 0x7f1b381589e0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f1b2800b3f0 tx=0x7f1b2800b8b0 comp rx=0 tx=0).stop 2024-09-16T15:42:59.578 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:59.574+0000 7f1b1d7fa640 1 -- 172.21.15.123:0/1056123993 shutdown_connections 2024-09-16T15:42:59.578 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:59.574+0000 7f1b1d7fa640 1 --2- 172.21.15.123:0/1056123993 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f1b24072ef0 0x7f1b240753b0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:42:59.578 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:59.574+0000 7f1b1d7fa640 1 --2- 172.21.15.123:0/1056123993 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1b3815a9e0 0x7f1b3815ae40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:42:59.578 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:59.574+0000 7f1b1d7fa640 1 --2- 172.21.15.123:0/1056123993 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f1b38145770 0x7f1b381589e0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:42:59.578 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:59.574+0000 7f1b1d7fa640 1 --2- 172.21.15.123:0/1056123993 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1b200053d0 0x7f1b38156d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:42:59.579 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:59.574+0000 7f1b1d7fa640 1 -- 172.21.15.123:0/1056123993 >> 172.21.15.123:0/1056123993 conn(0x7f1b380ccbf0 msgr2=0x7f1b380aded0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:42:59.579 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:59.574+0000 7f1b1d7fa640 1 -- 172.21.15.123:0/1056123993 shutdown_connections 2024-09-16T15:42:59.579 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:42:59.574+0000 7f1b1d7fa640 1 -- 172.21.15.123:0/1056123993 wait complete. 2024-09-16T15:42:59.591 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542152 got 60129542152 for osd.0 2024-09-16T15:42:59.591 DEBUG:teuthology.parallel:result is None 2024-09-16T15:42:59.594 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542151 got 60129542151 for osd.7 2024-09-16T15:42:59.594 DEBUG:teuthology.parallel:result is None 2024-09-16T15:43:00.479 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-09-16T15:43:00.601 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:00.598+0000 7f71f6869640 1 -- 172.21.15.123:0/2197934899 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f71d8004d50 msgr2=0x7f71d80051d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:43:00.601 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:00.598+0000 7f71f6869640 1 --2- 172.21.15.123:0/2197934899 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f71d8004d50 0x7f71d80051d0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f71e8009e30 tx=0x7f71e801d1b0 comp rx=0 tx=0).stop 2024-09-16T15:43:00.601 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:00.598+0000 7f71f4e06640 1 -- 172.21.15.123:0/2197934899 reap_dead start 2024-09-16T15:43:00.601 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:00.598+0000 7f71f6869640 1 -- 172.21.15.123:0/2197934899 shutdown_connections 2024-09-16T15:43:00.601 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:00.598+0000 7f71f6869640 1 -- 172.21.15.123:0/2197934899 >> 172.21.15.123:0/2197934899 conn(0x7f71f00cfb10 msgr2=0x7f71f00cff20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:43:00.601 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:00.598+0000 7f71f6869640 1 -- 172.21.15.123:0/2197934899 shutdown_connections 2024-09-16T15:43:00.601 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:00.598+0000 7f71f6869640 1 -- 172.21.15.123:0/2197934899 wait complete. 2024-09-16T15:43:00.602 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:00.598+0000 7f71f6869640 1 Processor -- start 2024-09-16T15:43:00.602 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:00.598+0000 7f71f6869640 1 -- start start 2024-09-16T15:43:00.604 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:00.602+0000 7f71f6869640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f71d8004d50 0x7f71f01f7d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:43:00.604 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:00.602+0000 7f71f6869640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f71f01e60c0 0x7f71f01f8280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:43:00.605 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:00.602+0000 7f71f6869640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f71f00b0cd0 0x7f71f0151cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:43:00.605 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:00.602+0000 7f71f6869640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f71f00cf430 con 0x7f71f01e60c0 2024-09-16T15:43:00.605 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:00.602+0000 7f71f6869640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f71f00ca630 con 0x7f71d8004d50 2024-09-16T15:43:00.605 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:00.602+0000 7f71f6869640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f71f00cf5b0 con 0x7f71f00b0cd0 2024-09-16T15:43:00.605 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:00.602+0000 7f71ef7fe640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f71f01e60c0 0x7f71f01f8280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:43:00.605 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:00.602+0000 7f71ef7fe640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f71f01e60c0 0x7f71f01f8280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:40682/0 (socket says 172.21.15.123:40682) 2024-09-16T15:43:00.605 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:00.602+0000 7f71ef7fe640 1 -- 172.21.15.123:0/2075649458 learned_addr learned my addr 172.21.15.123:0/2075649458 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:43:00.605 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:00.602+0000 7f71effff640 1 --2- 172.21.15.123:0/2075649458 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f71d8004d50 0x7f71f01f7d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:43:00.605 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:00.602+0000 7f71ef7fe640 1 -- 172.21.15.123:0/2075649458 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f71f00b0cd0 msgr2=0x7f71f0151cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:43:00.605 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:00.602+0000 7f71f4e06640 1 --2- 172.21.15.123:0/2075649458 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f71f00b0cd0 0x7f71f0151cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:43:00.605 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:00.602+0000 7f71ef7fe640 1 --2- 172.21.15.123:0/2075649458 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f71f00b0cd0 0x7f71f0151cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:00.606 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:00.602+0000 7f71ef7fe640 1 -- 172.21.15.123:0/2075649458 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f71d8004d50 msgr2=0x7f71f01f7d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:43:00.606 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:00.602+0000 7f71ef7fe640 1 --2- 172.21.15.123:0/2075649458 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f71d8004d50 0x7f71f01f7d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:00.606 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:00.602+0000 7f71ef7fe640 1 -- 172.21.15.123:0/2075649458 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f71f00c90c0 con 0x7f71f01e60c0 2024-09-16T15:43:00.606 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:00.602+0000 7f71ef7fe640 1 --2- 172.21.15.123:0/2075649458 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f71f01e60c0 0x7f71f01f8280 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f71e400f490 tx=0x7f71e400f950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:43:00.606 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:00.602+0000 7f71ecff9640 1 -- 172.21.15.123:0/2075649458 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71e401a070 con 0x7f71f01e60c0 2024-09-16T15:43:00.606 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:00.602+0000 7f71ecff9640 1 -- 172.21.15.123:0/2075649458 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f71e4008590 con 0x7f71f01e60c0 2024-09-16T15:43:00.606 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:00.602+0000 7f71f6869640 1 -- 172.21.15.123:0/2075649458 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f71f00c9330 con 0x7f71f01e60c0 2024-09-16T15:43:00.606 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:00.602+0000 7f71ecff9640 1 -- 172.21.15.123:0/2075649458 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71e4006980 con 0x7f71f01e60c0 2024-09-16T15:43:00.606 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:00.602+0000 7f71f6869640 1 -- 172.21.15.123:0/2075649458 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f71f01f8960 con 0x7f71f01e60c0 2024-09-16T15:43:00.606 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:00.602+0000 7f71f6869640 1 -- 172.21.15.123:0/2075649458 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f71f00740a0 con 0x7f71f01e60c0 2024-09-16T15:43:00.607 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:00.602+0000 7f71ecff9640 1 -- 172.21.15.123:0/2075649458 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 5) ==== 97080+0+0 (secure 0 0 0) 0x7f71e401d070 con 0x7f71f01e60c0 2024-09-16T15:43:00.607 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:00.602+0000 7f71ecff9640 1 --2- 172.21.15.123:0/2075649458 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f71dc072ec0 0x7f71dc075380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:43:00.607 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:00.602+0000 7f71ecff9640 1 -- 172.21.15.123:0/2075649458 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f71e40960f0 con 0x7f71f01e60c0 2024-09-16T15:43:00.607 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:00.602+0000 7f71effff640 1 --2- 172.21.15.123:0/2075649458 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f71dc072ec0 0x7f71dc075380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:43:00.607 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:00.606+0000 7f71effff640 1 --2- 172.21.15.123:0/2075649458 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f71dc072ec0 0x7f71dc075380 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f71e0005fd0 tx=0x7f71e0005e60 comp rx=0 tx=0).ready entity=mgr.4135 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:43:00.610 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:00.606+0000 7f71ecff9640 1 -- 172.21.15.123:0/2075649458 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f71e40612b0 con 0x7f71f01e60c0 2024-09-16T15:43:00.769 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:00.766+0000 7f71f6869640 1 -- 172.21.15.123:0/2075649458 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) -- 0x7f71f00d06c0 con 0x7f71f01e60c0 2024-09-16T15:43:00.770 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:00.766+0000 7f71ecff9640 1 -- 172.21.15.123:0/2075649458 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f71e4060bb0 con 0x7f71f01e60c0 2024-09-16T15:43:00.770 INFO:teuthology.orchestra.run.smithi123.stdout:60129542150 2024-09-16T15:43:00.773 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:00.770+0000 7f71f6869640 1 -- 172.21.15.123:0/2075649458 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f71dc072ec0 msgr2=0x7f71dc075380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:43:00.773 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:00.770+0000 7f71f6869640 1 --2- 172.21.15.123:0/2075649458 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f71dc072ec0 0x7f71dc075380 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f71e0005fd0 tx=0x7f71e0005e60 comp rx=0 tx=0).stop 2024-09-16T15:43:00.773 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:00.770+0000 7f71f6869640 1 -- 172.21.15.123:0/2075649458 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f71f01e60c0 msgr2=0x7f71f01f8280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:43:00.773 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:00.770+0000 7f71f6869640 1 --2- 172.21.15.123:0/2075649458 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f71f01e60c0 0x7f71f01f8280 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f71e400f490 tx=0x7f71e400f950 comp rx=0 tx=0).stop 2024-09-16T15:43:00.773 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:00.770+0000 7f71f6869640 1 -- 172.21.15.123:0/2075649458 shutdown_connections 2024-09-16T15:43:00.773 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:00.770+0000 7f71f6869640 1 --2- 172.21.15.123:0/2075649458 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f71dc072ec0 0x7f71dc075380 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:00.773 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:00.770+0000 7f71f6869640 1 --2- 172.21.15.123:0/2075649458 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f71f00b0cd0 0x7f71f0151cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:00.773 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:00.770+0000 7f71f6869640 1 --2- 172.21.15.123:0/2075649458 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f71f01e60c0 0x7f71f01f8280 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:00.774 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:00.770+0000 7f71f6869640 1 --2- 172.21.15.123:0/2075649458 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f71d8004d50 0x7f71f01f7d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:00.774 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:00.770+0000 7f71f6869640 1 -- 172.21.15.123:0/2075649458 >> 172.21.15.123:0/2075649458 conn(0x7f71f00cfb10 msgr2=0x7f71f00cb150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:43:00.774 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:00.770+0000 7f71f6869640 1 -- 172.21.15.123:0/2075649458 shutdown_connections 2024-09-16T15:43:00.774 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:00.770+0000 7f71f6869640 1 -- 172.21.15.123:0/2075649458 wait complete. 2024-09-16T15:43:00.787 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542150 got 60129542150 for osd.2 2024-09-16T15:43:00.787 DEBUG:teuthology.parallel:result is None 2024-09-16T15:43:00.788 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-09-16T15:43:00.788 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T15:43:00.900 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:00.898+0000 7f01ecb85640 1 -- 172.21.15.123:0/421905692 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f01e80b1a60 msgr2=0x7f01e8168d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:43:00.900 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:00.898+0000 7f01ecb85640 1 --2- 172.21.15.123:0/421905692 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f01e80b1a60 0x7f01e8168d50 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f01e0009c80 tx=0x7f01e001cc50 comp rx=0 tx=0).stop 2024-09-16T15:43:00.900 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:00.898+0000 7f01ecb85640 1 -- 172.21.15.123:0/421905692 shutdown_connections 2024-09-16T15:43:00.900 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:00.898+0000 7f01ecb85640 1 --2- 172.21.15.123:0/421905692 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f01e80b1a60 0x7f01e8168d50 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:00.900 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:00.898+0000 7f01ecb85640 1 -- 172.21.15.123:0/421905692 >> 172.21.15.123:0/421905692 conn(0x7f01e80d3ea0 msgr2=0x7f01e80d42b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:43:00.901 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:00.898+0000 7f01ecb85640 1 -- 172.21.15.123:0/421905692 shutdown_connections 2024-09-16T15:43:00.901 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:00.898+0000 7f01ecb85640 1 -- 172.21.15.123:0/421905692 wait complete. 2024-09-16T15:43:00.901 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:00.898+0000 7f01ecb85640 1 Processor -- start 2024-09-16T15:43:00.901 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:00.898+0000 7f01ecb85640 1 -- start start 2024-09-16T15:43:00.904 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:00.902+0000 7f01ecb85640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f01e80b1a60 0x7f01e80b92a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:43:00.904 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:00.902+0000 7f01ecb85640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f01e80b2b00 0x7f01e8153cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:43:00.904 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:00.902+0000 7f01ecb85640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f01e8161620 0x7f01e80b8860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:43:00.904 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:00.902+0000 7f01ecb85640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f01e816be90 con 0x7f01e8161620 2024-09-16T15:43:00.904 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:00.902+0000 7f01ecb85640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f01e816bd10 con 0x7f01e80b1a60 2024-09-16T15:43:00.904 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:00.902+0000 7f01ecb85640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f01e816c010 con 0x7f01e80b2b00 2024-09-16T15:43:00.904 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:00.902+0000 7f01e6d9d640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f01e8161620 0x7f01e80b8860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:43:00.904 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:00.902+0000 7f01e6d9d640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f01e8161620 0x7f01e80b8860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:40704/0 (socket says 172.21.15.123:40704) 2024-09-16T15:43:00.904 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:00.902+0000 7f01e6d9d640 1 -- 172.21.15.123:0/1205923406 learned_addr learned my addr 172.21.15.123:0/1205923406 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:43:00.904 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:00.902+0000 7f01dffff640 1 --2- 172.21.15.123:0/1205923406 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f01e80b2b00 0x7f01e8153cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:43:00.905 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:00.902+0000 7f01e659c640 1 --2- 172.21.15.123:0/1205923406 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f01e80b1a60 0x7f01e80b92a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:43:00.905 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:00.902+0000 7f01e6d9d640 1 -- 172.21.15.123:0/1205923406 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f01e80b2b00 msgr2=0x7f01e8153cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:43:00.905 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:00.902+0000 7f01e6d9d640 1 --2- 172.21.15.123:0/1205923406 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f01e80b2b00 0x7f01e8153cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:00.905 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:00.902+0000 7f01e6d9d640 1 -- 172.21.15.123:0/1205923406 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f01e80b1a60 msgr2=0x7f01e80b92a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:43:00.905 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:00.902+0000 7f01e6d9d640 1 --2- 172.21.15.123:0/1205923406 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f01e80b1a60 0x7f01e80b92a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:00.905 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:00.902+0000 7f01e6d9d640 1 -- 172.21.15.123:0/1205923406 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f01e8069970 con 0x7f01e8161620 2024-09-16T15:43:00.905 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:00.902+0000 7f01e6d9d640 1 --2- 172.21.15.123:0/1205923406 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f01e8161620 0x7f01e80b8860 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f01e0009bd0 tx=0x7f01e00a5b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:43:00.905 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:00.902+0000 7f01deffd640 1 -- 172.21.15.123:0/1205923406 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01e00bb070 con 0x7f01e8161620 2024-09-16T15:43:00.905 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:00.902+0000 7f01deffd640 1 -- 172.21.15.123:0/1205923406 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f01e0004d60 con 0x7f01e8161620 2024-09-16T15:43:00.905 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:00.902+0000 7f01deffd640 1 -- 172.21.15.123:0/1205923406 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01e00b2040 con 0x7f01e8161620 2024-09-16T15:43:00.905 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:00.902+0000 7f01ecb85640 1 -- 172.21.15.123:0/1205923406 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f01e80b9880 con 0x7f01e8161620 2024-09-16T15:43:00.905 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:00.902+0000 7f01ecb85640 1 -- 172.21.15.123:0/1205923406 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f01e80b3500 con 0x7f01e8161620 2024-09-16T15:43:00.906 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:00.902+0000 7f01ecb85640 1 -- 172.21.15.123:0/1205923406 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f01e80d2560 con 0x7f01e8161620 2024-09-16T15:43:00.906 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:00.902+0000 7f01deffd640 1 -- 172.21.15.123:0/1205923406 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 5) ==== 97080+0+0 (secure 0 0 0) 0x7f01e0004320 con 0x7f01e8161620 2024-09-16T15:43:00.906 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:00.902+0000 7f01deffd640 1 --2- 172.21.15.123:0/1205923406 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f01c8072ec0 0x7f01c8075380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:43:00.906 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:00.902+0000 7f01deffd640 1 -- 172.21.15.123:0/1205923406 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f01e012dce0 con 0x7f01e8161620 2024-09-16T15:43:00.906 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:00.902+0000 7f01e659c640 1 --2- 172.21.15.123:0/1205923406 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f01c8072ec0 0x7f01c8075380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:43:00.907 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:00.906+0000 7f01e659c640 1 --2- 172.21.15.123:0/1205923406 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f01c8072ec0 0x7f01c8075380 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f01d800a750 tx=0x7f01d8008040 comp rx=0 tx=0).ready entity=mgr.4135 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:43:00.909 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:00.906+0000 7f01deffd640 1 -- 172.21.15.123:0/1205923406 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f01e0016990 con 0x7f01e8161620 2024-09-16T15:43:01.066 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:01.062+0000 7f01ecb85640 1 -- 172.21.15.123:0/1205923406 --> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f01e806e500 con 0x7f01c8072ec0 2024-09-16T15:43:01.078 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:01.074+0000 7f01deffd640 1 -- 172.21.15.123:0/1205923406 <== mgr.4135 v2:172.21.15.184:6832/3114346423 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+251825 (secure 0 0 0) 0x7f01e806e500 con 0x7f01c8072ec0 2024-09-16T15:43:01.080 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-09-16T15:43:01.082 INFO:teuthology.orchestra.run.smithi123.stderr:dumped all 2024-09-16T15:43:01.083 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:01.078+0000 7f01ecb85640 1 -- 172.21.15.123:0/1205923406 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f01c8072ec0 msgr2=0x7f01c8075380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:43:01.084 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:01.078+0000 7f01ecb85640 1 --2- 172.21.15.123:0/1205923406 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f01c8072ec0 0x7f01c8075380 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f01d800a750 tx=0x7f01d8008040 comp rx=0 tx=0).stop 2024-09-16T15:43:01.084 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:01.078+0000 7f01ecb85640 1 -- 172.21.15.123:0/1205923406 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f01e8161620 msgr2=0x7f01e80b8860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:43:01.084 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:01.078+0000 7f01ecb85640 1 --2- 172.21.15.123:0/1205923406 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f01e8161620 0x7f01e80b8860 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f01e0009bd0 tx=0x7f01e00a5b20 comp rx=0 tx=0).stop 2024-09-16T15:43:01.084 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:01.078+0000 7f01ecb85640 1 -- 172.21.15.123:0/1205923406 shutdown_connections 2024-09-16T15:43:01.084 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:01.078+0000 7f01ecb85640 1 --2- 172.21.15.123:0/1205923406 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f01c8072ec0 0x7f01c8075380 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:01.084 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:01.078+0000 7f01ecb85640 1 --2- 172.21.15.123:0/1205923406 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f01e8161620 0x7f01e80b8860 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:01.084 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:01.078+0000 7f01ecb85640 1 --2- 172.21.15.123:0/1205923406 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f01e80b2b00 0x7f01e8153cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:01.084 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:01.078+0000 7f01ecb85640 1 --2- 172.21.15.123:0/1205923406 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f01e80b1a60 0x7f01e80b92a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:01.084 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:01.078+0000 7f01ecb85640 1 -- 172.21.15.123:0/1205923406 >> 172.21.15.123:0/1205923406 conn(0x7f01e80d3ea0 msgr2=0x7f01e806d2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:43:01.084 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:01.078+0000 7f01ecb85640 1 -- 172.21.15.123:0/1205923406 shutdown_connections 2024-09-16T15:43:01.085 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:01.078+0000 7f01ecb85640 1 -- 172.21.15.123:0/1205923406 wait complete. 2024-09-16T15:43:01.097 INFO:teuthology.orchestra.run.smithi123.stdout:{"pg_ready":true,"pg_map":{"version":24,"stamp":"2024-09-16T15:42:59.247569+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":146,"num_write_kb":1642,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":111,"ondisk_log_size":111,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219140,"kb_used_data":2888,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755580,"statfs":{"total":773094113280,"available":772869713920,"internally_reserved":0,"allocated":2957312,"data_stored":1778268,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002848"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T15:42:52.725091+0000","last_change":"2024-09-16T15:42:51.434234+0000","last_active":"2024-09-16T15:42:52.725091+0000","last_peered":"2024-09-16T15:42:52.725091+0000","last_clean":"2024-09-16T15:42:52.725091+0000","last_became_active":"2024-09-16T15:42:51.433660+0000","last_became_peered":"2024-09-16T15:42:51.433660+0000","last_unstale":"2024-09-16T15:42:52.725091+0000","last_undegraded":"2024-09-16T15:42:52.725091+0000","last_fullsized":"2024-09-16T15:42:52.725091+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_clean_scrub_stamp":"2024-09-16T15:42:50.361965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T02:07:35.605190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T15:42:52.418662+0000","last_change":"2024-09-16T15:42:51.412677+0000","last_active":"2024-09-16T15:42:52.418662+0000","last_peered":"2024-09-16T15:42:52.418662+0000","last_clean":"2024-09-16T15:42:52.418662+0000","last_became_active":"2024-09-16T15:42:51.412366+0000","last_became_peered":"2024-09-16T15:42:51.412366+0000","last_unstale":"2024-09-16T15:42:52.418662+0000","last_undegraded":"2024-09-16T15:42:52.418662+0000","last_fullsized":"2024-09-16T15:42:52.418662+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_clean_scrub_stamp":"2024-09-16T15:42:50.361965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:21.758409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T15:42:52.876476+0000","last_change":"2024-09-16T15:42:51.406454+0000","last_active":"2024-09-16T15:42:52.876476+0000","last_peered":"2024-09-16T15:42:52.876476+0000","last_clean":"2024-09-16T15:42:52.876476+0000","last_became_active":"2024-09-16T15:42:51.405679+0000","last_became_peered":"2024-09-16T15:42:51.405679+0000","last_unstale":"2024-09-16T15:42:52.876476+0000","last_undegraded":"2024-09-16T15:42:52.876476+0000","last_fullsized":"2024-09-16T15:42:52.876476+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_clean_scrub_stamp":"2024-09-16T15:42:50.361965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T02:33:32.417205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T15:42:52.638879+0000","last_change":"2024-09-16T15:42:51.423931+0000","last_active":"2024-09-16T15:42:52.638879+0000","last_peered":"2024-09-16T15:42:52.638879+0000","last_clean":"2024-09-16T15:42:52.638879+0000","last_became_active":"2024-09-16T15:42:51.423561+0000","last_became_peered":"2024-09-16T15:42:51.423561+0000","last_unstale":"2024-09-16T15:42:52.638879+0000","last_undegraded":"2024-09-16T15:42:52.638879+0000","last_fullsized":"2024-09-16T15:42:52.638879+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_clean_scrub_stamp":"2024-09-16T15:42:50.361965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T03:35:03.081638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T15:42:52.724573+0000","last_change":"2024-09-16T15:42:51.425601+0000","last_active":"2024-09-16T15:42:52.724573+0000","last_peered":"2024-09-16T15:42:52.724573+0000","last_clean":"2024-09-16T15:42:52.724573+0000","last_became_active":"2024-09-16T15:42:51.425040+0000","last_became_peered":"2024-09-16T15:42:51.425040+0000","last_unstale":"2024-09-16T15:42:52.724573+0000","last_undegraded":"2024-09-16T15:42:52.724573+0000","last_fullsized":"2024-09-16T15:42:52.724573+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_clean_scrub_stamp":"2024-09-16T15:42:50.361965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T00:05:49.936465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T15:42:52.638668+0000","last_change":"2024-09-16T15:42:51.429218+0000","last_active":"2024-09-16T15:42:52.638668+0000","last_peered":"2024-09-16T15:42:52.638668+0000","last_clean":"2024-09-16T15:42:52.638668+0000","last_became_active":"2024-09-16T15:42:51.428838+0000","last_became_peered":"2024-09-16T15:42:51.428838+0000","last_unstale":"2024-09-16T15:42:52.638668+0000","last_undegraded":"2024-09-16T15:42:52.638668+0000","last_fullsized":"2024-09-16T15:42:52.638668+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_clean_scrub_stamp":"2024-09-16T15:42:50.361965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:21.758409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T15:42:52.723991+0000","last_change":"2024-09-16T15:42:51.424389+0000","last_active":"2024-09-16T15:42:52.723991+0000","last_peered":"2024-09-16T15:42:52.723991+0000","last_clean":"2024-09-16T15:42:52.723991+0000","last_became_active":"2024-09-16T15:42:51.423840+0000","last_became_peered":"2024-09-16T15:42:51.423840+0000","last_unstale":"2024-09-16T15:42:52.723991+0000","last_undegraded":"2024-09-16T15:42:52.723991+0000","last_fullsized":"2024-09-16T15:42:52.723991+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_clean_scrub_stamp":"2024-09-16T15:42:50.361965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T18:05:24.521404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T15:42:52.874446+0000","last_change":"2024-09-16T15:42:51.428815+0000","last_active":"2024-09-16T15:42:52.874446+0000","last_peered":"2024-09-16T15:42:52.874446+0000","last_clean":"2024-09-16T15:42:52.874446+0000","last_became_active":"2024-09-16T15:42:51.427658+0000","last_became_peered":"2024-09-16T15:42:51.427658+0000","last_unstale":"2024-09-16T15:42:52.874446+0000","last_undegraded":"2024-09-16T15:42:52.874446+0000","last_fullsized":"2024-09-16T15:42:52.874446+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_clean_scrub_stamp":"2024-09-16T15:42:50.361965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T03:23:33.232936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T15:42:52.725119+0000","last_change":"2024-09-16T15:42:51.435164+0000","last_active":"2024-09-16T15:42:52.725119+0000","last_peered":"2024-09-16T15:42:52.725119+0000","last_clean":"2024-09-16T15:42:52.725119+0000","last_became_active":"2024-09-16T15:42:51.434702+0000","last_became_peered":"2024-09-16T15:42:51.434702+0000","last_unstale":"2024-09-16T15:42:52.725119+0000","last_undegraded":"2024-09-16T15:42:52.725119+0000","last_fullsized":"2024-09-16T15:42:52.725119+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_clean_scrub_stamp":"2024-09-16T15:42:50.361965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:40:28.779257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T15:42:52.417480+0000","last_change":"2024-09-16T15:42:51.425817+0000","last_active":"2024-09-16T15:42:52.417480+0000","last_peered":"2024-09-16T15:42:52.417480+0000","last_clean":"2024-09-16T15:42:52.417480+0000","last_became_active":"2024-09-16T15:42:51.425279+0000","last_became_peered":"2024-09-16T15:42:51.425279+0000","last_unstale":"2024-09-16T15:42:52.417480+0000","last_undegraded":"2024-09-16T15:42:52.417480+0000","last_fullsized":"2024-09-16T15:42:52.417480+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_clean_scrub_stamp":"2024-09-16T15:42:50.361965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:10:07.925485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T15:42:52.416844+0000","last_change":"2024-09-16T15:42:51.426883+0000","last_active":"2024-09-16T15:42:52.416844+0000","last_peered":"2024-09-16T15:42:52.416844+0000","last_clean":"2024-09-16T15:42:52.416844+0000","last_became_active":"2024-09-16T15:42:51.426556+0000","last_became_peered":"2024-09-16T15:42:51.426556+0000","last_unstale":"2024-09-16T15:42:52.416844+0000","last_undegraded":"2024-09-16T15:42:52.416844+0000","last_fullsized":"2024-09-16T15:42:52.416844+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_clean_scrub_stamp":"2024-09-16T15:42:50.361965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:55:46.525397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T15:42:52.418805+0000","last_change":"2024-09-16T15:42:51.394761+0000","last_active":"2024-09-16T15:42:52.418805+0000","last_peered":"2024-09-16T15:42:52.418805+0000","last_clean":"2024-09-16T15:42:52.418805+0000","last_became_active":"2024-09-16T15:42:51.394005+0000","last_became_peered":"2024-09-16T15:42:51.394005+0000","last_unstale":"2024-09-16T15:42:52.418805+0000","last_undegraded":"2024-09-16T15:42:52.418805+0000","last_fullsized":"2024-09-16T15:42:52.418805+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_clean_scrub_stamp":"2024-09-16T15:42:50.361965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:17:36.442767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T15:42:52.875964+0000","last_change":"2024-09-16T15:42:51.411614+0000","last_active":"2024-09-16T15:42:52.875964+0000","last_peered":"2024-09-16T15:42:52.875964+0000","last_clean":"2024-09-16T15:42:52.875964+0000","last_became_active":"2024-09-16T15:42:51.409702+0000","last_became_peered":"2024-09-16T15:42:51.409702+0000","last_unstale":"2024-09-16T15:42:52.875964+0000","last_undegraded":"2024-09-16T15:42:52.875964+0000","last_fullsized":"2024-09-16T15:42:52.875964+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_clean_scrub_stamp":"2024-09-16T15:42:50.361965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T00:31:04.101900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T15:42:52.874545+0000","last_change":"2024-09-16T15:42:51.403206+0000","last_active":"2024-09-16T15:42:52.874545+0000","last_peered":"2024-09-16T15:42:52.874545+0000","last_clean":"2024-09-16T15:42:52.874545+0000","last_became_active":"2024-09-16T15:42:51.402528+0000","last_became_peered":"2024-09-16T15:42:51.402528+0000","last_unstale":"2024-09-16T15:42:52.874545+0000","last_undegraded":"2024-09-16T15:42:52.874545+0000","last_fullsized":"2024-09-16T15:42:52.874545+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_clean_scrub_stamp":"2024-09-16T15:42:50.361965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:39:05.860718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T15:42:52.418038+0000","last_change":"2024-09-16T15:42:51.411905+0000","last_active":"2024-09-16T15:42:52.418038+0000","last_peered":"2024-09-16T15:42:52.418038+0000","last_clean":"2024-09-16T15:42:52.418038+0000","last_became_active":"2024-09-16T15:42:51.411540+0000","last_became_peered":"2024-09-16T15:42:51.411540+0000","last_unstale":"2024-09-16T15:42:52.418038+0000","last_undegraded":"2024-09-16T15:42:52.418038+0000","last_fullsized":"2024-09-16T15:42:52.418038+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_clean_scrub_stamp":"2024-09-16T15:42:50.361965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:07:17.519138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T15:42:52.638310+0000","last_change":"2024-09-16T15:42:51.419623+0000","last_active":"2024-09-16T15:42:52.638310+0000","last_peered":"2024-09-16T15:42:52.638310+0000","last_clean":"2024-09-16T15:42:52.638310+0000","last_became_active":"2024-09-16T15:42:51.419319+0000","last_became_peered":"2024-09-16T15:42:51.419319+0000","last_unstale":"2024-09-16T15:42:52.638310+0000","last_undegraded":"2024-09-16T15:42:52.638310+0000","last_fullsized":"2024-09-16T15:42:52.638310+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_clean_scrub_stamp":"2024-09-16T15:42:50.361965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T02:33:57.643582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T15:42:52.640012+0000","last_change":"2024-09-16T15:42:51.436411+0000","last_active":"2024-09-16T15:42:52.640012+0000","last_peered":"2024-09-16T15:42:52.640012+0000","last_clean":"2024-09-16T15:42:52.640012+0000","last_became_active":"2024-09-16T15:42:51.435952+0000","last_became_peered":"2024-09-16T15:42:51.435952+0000","last_unstale":"2024-09-16T15:42:52.640012+0000","last_undegraded":"2024-09-16T15:42:52.640012+0000","last_fullsized":"2024-09-16T15:42:52.640012+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_clean_scrub_stamp":"2024-09-16T15:42:50.361965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:08:57.795460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T15:42:52.738414+0000","last_change":"2024-09-16T15:42:51.404567+0000","last_active":"2024-09-16T15:42:52.738414+0000","last_peered":"2024-09-16T15:42:52.738414+0000","last_clean":"2024-09-16T15:42:52.738414+0000","last_became_active":"2024-09-16T15:42:51.403756+0000","last_became_peered":"2024-09-16T15:42:51.403756+0000","last_unstale":"2024-09-16T15:42:52.738414+0000","last_undegraded":"2024-09-16T15:42:52.738414+0000","last_fullsized":"2024-09-16T15:42:52.738414+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_clean_scrub_stamp":"2024-09-16T15:42:50.361965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T01:04:06.975937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T15:42:52.725148+0000","last_change":"2024-09-16T15:42:51.434196+0000","last_active":"2024-09-16T15:42:52.725148+0000","last_peered":"2024-09-16T15:42:52.725148+0000","last_clean":"2024-09-16T15:42:52.725148+0000","last_became_active":"2024-09-16T15:42:51.433748+0000","last_became_peered":"2024-09-16T15:42:51.433748+0000","last_unstale":"2024-09-16T15:42:52.725148+0000","last_undegraded":"2024-09-16T15:42:52.725148+0000","last_fullsized":"2024-09-16T15:42:52.725148+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_clean_scrub_stamp":"2024-09-16T15:42:50.361965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:08:57.795460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T15:42:52.416804+0000","last_change":"2024-09-16T15:42:51.398201+0000","last_active":"2024-09-16T15:42:52.416804+0000","last_peered":"2024-09-16T15:42:52.416804+0000","last_clean":"2024-09-16T15:42:52.416804+0000","last_became_active":"2024-09-16T15:42:51.396092+0000","last_became_peered":"2024-09-16T15:42:51.396092+0000","last_unstale":"2024-09-16T15:42:52.416804+0000","last_undegraded":"2024-09-16T15:42:52.416804+0000","last_fullsized":"2024-09-16T15:42:52.416804+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_clean_scrub_stamp":"2024-09-16T15:42:50.361965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:16:26.836026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T15:42:52.874601+0000","last_change":"2024-09-16T15:42:51.404937+0000","last_active":"2024-09-16T15:42:52.874601+0000","last_peered":"2024-09-16T15:42:52.874601+0000","last_clean":"2024-09-16T15:42:52.874601+0000","last_became_active":"2024-09-16T15:42:51.404498+0000","last_became_peered":"2024-09-16T15:42:51.404498+0000","last_unstale":"2024-09-16T15:42:52.874601+0000","last_undegraded":"2024-09-16T15:42:52.874601+0000","last_fullsized":"2024-09-16T15:42:52.874601+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_clean_scrub_stamp":"2024-09-16T15:42:50.361965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:16:06.557859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T15:42:52.638770+0000","last_change":"2024-09-16T15:42:51.436767+0000","last_active":"2024-09-16T15:42:52.638770+0000","last_peered":"2024-09-16T15:42:52.638770+0000","last_clean":"2024-09-16T15:42:52.638770+0000","last_became_active":"2024-09-16T15:42:51.436076+0000","last_became_peered":"2024-09-16T15:42:51.436076+0000","last_unstale":"2024-09-16T15:42:52.638770+0000","last_undegraded":"2024-09-16T15:42:52.638770+0000","last_fullsized":"2024-09-16T15:42:52.638770+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_clean_scrub_stamp":"2024-09-16T15:42:50.361965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T02:42:34.108192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T15:42:52.876650+0000","last_change":"2024-09-16T15:42:51.407561+0000","last_active":"2024-09-16T15:42:52.876650+0000","last_peered":"2024-09-16T15:42:52.876650+0000","last_clean":"2024-09-16T15:42:52.876650+0000","last_became_active":"2024-09-16T15:42:51.407242+0000","last_became_peered":"2024-09-16T15:42:51.407242+0000","last_unstale":"2024-09-16T15:42:52.876650+0000","last_undegraded":"2024-09-16T15:42:52.876650+0000","last_fullsized":"2024-09-16T15:42:52.876650+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_clean_scrub_stamp":"2024-09-16T15:42:50.361965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:37:26.430063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T15:42:52.638795+0000","last_change":"2024-09-16T15:42:51.420155+0000","last_active":"2024-09-16T15:42:52.638795+0000","last_peered":"2024-09-16T15:42:52.638795+0000","last_clean":"2024-09-16T15:42:52.638795+0000","last_became_active":"2024-09-16T15:42:51.419779+0000","last_became_peered":"2024-09-16T15:42:51.419779+0000","last_unstale":"2024-09-16T15:42:52.638795+0000","last_undegraded":"2024-09-16T15:42:52.638795+0000","last_fullsized":"2024-09-16T15:42:52.638795+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_clean_scrub_stamp":"2024-09-16T15:42:50.361965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:28:43.699424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T15:42:52.875086+0000","last_change":"2024-09-16T15:42:51.412701+0000","last_active":"2024-09-16T15:42:52.875086+0000","last_peered":"2024-09-16T15:42:52.875086+0000","last_clean":"2024-09-16T15:42:52.875086+0000","last_became_active":"2024-09-16T15:42:51.411770+0000","last_became_peered":"2024-09-16T15:42:51.411770+0000","last_unstale":"2024-09-16T15:42:52.875086+0000","last_undegraded":"2024-09-16T15:42:52.875086+0000","last_fullsized":"2024-09-16T15:42:52.875086+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_clean_scrub_stamp":"2024-09-16T15:42:50.361965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T02:37:19.881471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T15:42:52.417630+0000","last_change":"2024-09-16T15:42:51.423741+0000","last_active":"2024-09-16T15:42:52.417630+0000","last_peered":"2024-09-16T15:42:52.417630+0000","last_clean":"2024-09-16T15:42:52.417630+0000","last_became_active":"2024-09-16T15:42:51.423444+0000","last_became_peered":"2024-09-16T15:42:51.423444+0000","last_unstale":"2024-09-16T15:42:52.417630+0000","last_undegraded":"2024-09-16T15:42:52.417630+0000","last_fullsized":"2024-09-16T15:42:52.417630+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_clean_scrub_stamp":"2024-09-16T15:42:50.361965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T18:56:18.021098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T15:42:52.417698+0000","last_change":"2024-09-16T15:42:51.410257+0000","last_active":"2024-09-16T15:42:52.417698+0000","last_peered":"2024-09-16T15:42:52.417698+0000","last_clean":"2024-09-16T15:42:52.417698+0000","last_became_active":"2024-09-16T15:42:51.409838+0000","last_became_peered":"2024-09-16T15:42:51.409838+0000","last_unstale":"2024-09-16T15:42:52.417698+0000","last_undegraded":"2024-09-16T15:42:52.417698+0000","last_fullsized":"2024-09-16T15:42:52.417698+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_clean_scrub_stamp":"2024-09-16T15:42:50.361965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T01:10:38.539716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T15:42:52.875794+0000","last_change":"2024-09-16T15:42:51.414267+0000","last_active":"2024-09-16T15:42:52.875794+0000","last_peered":"2024-09-16T15:42:52.875794+0000","last_clean":"2024-09-16T15:42:52.875794+0000","last_became_active":"2024-09-16T15:42:51.412201+0000","last_became_peered":"2024-09-16T15:42:51.412201+0000","last_unstale":"2024-09-16T15:42:52.875794+0000","last_undegraded":"2024-09-16T15:42:52.875794+0000","last_fullsized":"2024-09-16T15:42:52.875794+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_clean_scrub_stamp":"2024-09-16T15:42:50.361965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:59:06.080508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T15:42:52.875146+0000","last_change":"2024-09-16T15:42:51.412344+0000","last_active":"2024-09-16T15:42:52.875146+0000","last_peered":"2024-09-16T15:42:52.875146+0000","last_clean":"2024-09-16T15:42:52.875146+0000","last_became_active":"2024-09-16T15:42:51.410933+0000","last_became_peered":"2024-09-16T15:42:51.410933+0000","last_unstale":"2024-09-16T15:42:52.875146+0000","last_undegraded":"2024-09-16T15:42:52.875146+0000","last_fullsized":"2024-09-16T15:42:52.875146+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_clean_scrub_stamp":"2024-09-16T15:42:50.361965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T01:38:29.275555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T15:42:52.639286+0000","last_change":"2024-09-16T15:42:51.419645+0000","last_active":"2024-09-16T15:42:52.639286+0000","last_peered":"2024-09-16T15:42:52.639286+0000","last_clean":"2024-09-16T15:42:52.639286+0000","last_became_active":"2024-09-16T15:42:51.419335+0000","last_became_peered":"2024-09-16T15:42:51.419335+0000","last_unstale":"2024-09-16T15:42:52.639286+0000","last_undegraded":"2024-09-16T15:42:52.639286+0000","last_fullsized":"2024-09-16T15:42:52.639286+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_clean_scrub_stamp":"2024-09-16T15:42:50.361965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T02:58:03.717735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T15:42:52.738151+0000","last_change":"2024-09-16T15:42:51.425478+0000","last_active":"2024-09-16T15:42:52.738151+0000","last_peered":"2024-09-16T15:42:52.738151+0000","last_clean":"2024-09-16T15:42:52.738151+0000","last_became_active":"2024-09-16T15:42:51.425150+0000","last_became_peered":"2024-09-16T15:42:51.425150+0000","last_unstale":"2024-09-16T15:42:52.738151+0000","last_undegraded":"2024-09-16T15:42:52.738151+0000","last_fullsized":"2024-09-16T15:42:52.738151+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_clean_scrub_stamp":"2024-09-16T15:42:50.361965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T00:53:53.123181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T15:42:52.416202+0000","last_change":"2024-09-16T15:42:51.395997+0000","last_active":"2024-09-16T15:42:52.416202+0000","last_peered":"2024-09-16T15:42:52.416202+0000","last_clean":"2024-09-16T15:42:52.416202+0000","last_became_active":"2024-09-16T15:42:51.395182+0000","last_became_peered":"2024-09-16T15:42:51.395182+0000","last_unstale":"2024-09-16T15:42:52.416202+0000","last_undegraded":"2024-09-16T15:42:52.416202+0000","last_fullsized":"2024-09-16T15:42:52.416202+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_clean_scrub_stamp":"2024-09-16T15:42:50.361965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:22:21.461659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T15:42:52.418393+0000","last_change":"2024-09-16T15:42:51.398579+0000","last_active":"2024-09-16T15:42:52.418393+0000","last_peered":"2024-09-16T15:42:52.418393+0000","last_clean":"2024-09-16T15:42:52.418393+0000","last_became_active":"2024-09-16T15:42:51.397865+0000","last_became_peered":"2024-09-16T15:42:51.397865+0000","last_unstale":"2024-09-16T15:42:52.418393+0000","last_undegraded":"2024-09-16T15:42:52.418393+0000","last_fullsized":"2024-09-16T15:42:52.418393+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_clean_scrub_stamp":"2024-09-16T15:42:50.361965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:44:41.707611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"20'1","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T15:42:52.876772+0000","last_change":"2024-09-16T15:42:50.407972+0000","last_active":"2024-09-16T15:42:52.876772+0000","last_peered":"2024-09-16T15:42:52.876772+0000","last_clean":"2024-09-16T15:42:52.876772+0000","last_became_active":"2024-09-16T15:42:50.407571+0000","last_became_peered":"2024-09-16T15:42:50.407571+0000","last_unstale":"2024-09-16T15:42:52.876772+0000","last_undegraded":"2024-09-16T15:42:52.876772+0000","last_fullsized":"2024-09-16T15:42:52.876772+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:42:49.352371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:42:49.352371+0000","last_clean_scrub_stamp":"2024-09-16T15:42:49.352371+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T02:32:51.461439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T15:42:52.876489+0000","last_change":"2024-09-16T15:42:51.403656+0000","last_active":"2024-09-16T15:42:52.876489+0000","last_peered":"2024-09-16T15:42:52.876489+0000","last_clean":"2024-09-16T15:42:52.876489+0000","last_became_active":"2024-09-16T15:42:51.403183+0000","last_became_peered":"2024-09-16T15:42:51.403183+0000","last_unstale":"2024-09-16T15:42:52.876489+0000","last_undegraded":"2024-09-16T15:42:52.876489+0000","last_fullsized":"2024-09-16T15:42:52.876489+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_clean_scrub_stamp":"2024-09-16T15:42:50.361965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T01:01:27.144919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"20'1","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T15:42:52.415759+0000","last_change":"2024-09-16T15:42:50.405255+0000","last_active":"2024-09-16T15:42:52.415759+0000","last_peered":"2024-09-16T15:42:52.415759+0000","last_clean":"2024-09-16T15:42:52.415759+0000","last_became_active":"2024-09-16T15:42:50.404469+0000","last_became_peered":"2024-09-16T15:42:50.404469+0000","last_unstale":"2024-09-16T15:42:52.415759+0000","last_undegraded":"2024-09-16T15:42:52.415759+0000","last_fullsized":"2024-09-16T15:42:52.415759+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:42:49.352371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:42:49.352371+0000","last_clean_scrub_stamp":"2024-09-16T15:42:49.352371+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:41:46.870937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T15:42:52.877073+0000","last_change":"2024-09-16T15:42:50.399198+0000","last_active":"2024-09-16T15:42:52.877073+0000","last_peered":"2024-09-16T15:42:52.877073+0000","last_clean":"2024-09-16T15:42:52.877073+0000","last_became_active":"2024-09-16T15:42:50.398892+0000","last_became_peered":"2024-09-16T15:42:50.398892+0000","last_unstale":"2024-09-16T15:42:52.877073+0000","last_undegraded":"2024-09-16T15:42:52.877073+0000","last_fullsized":"2024-09-16T15:42:52.877073+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:42:49.352371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:42:49.352371+0000","last_clean_scrub_stamp":"2024-09-16T15:42:49.352371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T03:32:21.032967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T15:42:52.875003+0000","last_change":"2024-09-16T15:42:51.412892+0000","last_active":"2024-09-16T15:42:52.875003+0000","last_peered":"2024-09-16T15:42:52.875003+0000","last_clean":"2024-09-16T15:42:52.875003+0000","last_became_active":"2024-09-16T15:42:51.407080+0000","last_became_peered":"2024-09-16T15:42:51.407080+0000","last_unstale":"2024-09-16T15:42:52.875003+0000","last_undegraded":"2024-09-16T15:42:52.875003+0000","last_fullsized":"2024-09-16T15:42:52.875003+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_clean_scrub_stamp":"2024-09-16T15:42:50.361965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T02:17:29.317460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T15:42:52.638690+0000","last_change":"2024-09-16T15:42:50.427196+0000","last_active":"2024-09-16T15:42:52.638690+0000","last_peered":"2024-09-16T15:42:52.638690+0000","last_clean":"2024-09-16T15:42:52.638690+0000","last_became_active":"2024-09-16T15:42:50.426397+0000","last_became_peered":"2024-09-16T15:42:50.426397+0000","last_unstale":"2024-09-16T15:42:52.638690+0000","last_undegraded":"2024-09-16T15:42:52.638690+0000","last_fullsized":"2024-09-16T15:42:52.638690+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:42:49.352371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:42:49.352371+0000","last_clean_scrub_stamp":"2024-09-16T15:42:49.352371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T22:48:39.726611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T15:42:52.417585+0000","last_change":"2024-09-16T15:42:51.410683+0000","last_active":"2024-09-16T15:42:52.417585+0000","last_peered":"2024-09-16T15:42:52.417585+0000","last_clean":"2024-09-16T15:42:52.417585+0000","last_became_active":"2024-09-16T15:42:51.409695+0000","last_became_peered":"2024-09-16T15:42:51.409695+0000","last_unstale":"2024-09-16T15:42:52.417585+0000","last_undegraded":"2024-09-16T15:42:52.417585+0000","last_fullsized":"2024-09-16T15:42:52.417585+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_clean_scrub_stamp":"2024-09-16T15:42:50.361965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T01:38:29.275555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T15:42:52.417220+0000","last_change":"2024-09-16T15:42:50.420642+0000","last_active":"2024-09-16T15:42:52.417220+0000","last_peered":"2024-09-16T15:42:52.417220+0000","last_clean":"2024-09-16T15:42:52.417220+0000","last_became_active":"2024-09-16T15:42:50.419895+0000","last_became_peered":"2024-09-16T15:42:50.419895+0000","last_unstale":"2024-09-16T15:42:52.417220+0000","last_undegraded":"2024-09-16T15:42:52.417220+0000","last_fullsized":"2024-09-16T15:42:52.417220+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:42:49.352371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:42:49.352371+0000","last_clean_scrub_stamp":"2024-09-16T15:42:49.352371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:40:01.292941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T15:42:52.639747+0000","last_change":"2024-09-16T15:42:51.434841+0000","last_active":"2024-09-16T15:42:52.639747+0000","last_peered":"2024-09-16T15:42:52.639747+0000","last_clean":"2024-09-16T15:42:52.639747+0000","last_became_active":"2024-09-16T15:42:51.434532+0000","last_became_peered":"2024-09-16T15:42:51.434532+0000","last_unstale":"2024-09-16T15:42:52.639747+0000","last_undegraded":"2024-09-16T15:42:52.639747+0000","last_fullsized":"2024-09-16T15:42:52.639747+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_clean_scrub_stamp":"2024-09-16T15:42:50.361965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T18:44:49.865687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T15:42:52.639491+0000","last_change":"2024-09-16T15:42:50.419196+0000","last_active":"2024-09-16T15:42:52.639491+0000","last_peered":"2024-09-16T15:42:52.639491+0000","last_clean":"2024-09-16T15:42:52.639491+0000","last_became_active":"2024-09-16T15:42:50.415873+0000","last_became_peered":"2024-09-16T15:42:50.415873+0000","last_unstale":"2024-09-16T15:42:52.639491+0000","last_undegraded":"2024-09-16T15:42:52.639491+0000","last_fullsized":"2024-09-16T15:42:52.639491+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:42:49.352371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:42:49.352371+0000","last_clean_scrub_stamp":"2024-09-16T15:42:49.352371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:26:07.798125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T15:42:52.877644+0000","last_change":"2024-09-16T15:42:51.425682+0000","last_active":"2024-09-16T15:42:52.877644+0000","last_peered":"2024-09-16T15:42:52.877644+0000","last_clean":"2024-09-16T15:42:52.877644+0000","last_became_active":"2024-09-16T15:42:51.425336+0000","last_became_peered":"2024-09-16T15:42:51.425336+0000","last_unstale":"2024-09-16T15:42:52.877644+0000","last_undegraded":"2024-09-16T15:42:52.877644+0000","last_fullsized":"2024-09-16T15:42:52.877644+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_clean_scrub_stamp":"2024-09-16T15:42:50.361965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T02:33:57.643582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"20'3","reported_seq":18,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T15:42:52.418373+0000","last_change":"2024-09-16T15:42:50.415090+0000","last_active":"2024-09-16T15:42:52.418373+0000","last_peered":"2024-09-16T15:42:52.418373+0000","last_clean":"2024-09-16T15:42:52.418373+0000","last_became_active":"2024-09-16T15:42:50.414818+0000","last_became_peered":"2024-09-16T15:42:50.414818+0000","last_unstale":"2024-09-16T15:42:52.418373+0000","last_undegraded":"2024-09-16T15:42:52.418373+0000","last_fullsized":"2024-09-16T15:42:52.418373+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:42:49.352371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:42:49.352371+0000","last_clean_scrub_stamp":"2024-09-16T15:42:49.352371+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:36:36.506348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T15:42:52.419154+0000","last_change":"2024-09-16T15:42:51.425357+0000","last_active":"2024-09-16T15:42:52.419154+0000","last_peered":"2024-09-16T15:42:52.419154+0000","last_clean":"2024-09-16T15:42:52.419154+0000","last_became_active":"2024-09-16T15:42:51.425074+0000","last_became_peered":"2024-09-16T15:42:51.425074+0000","last_unstale":"2024-09-16T15:42:52.419154+0000","last_undegraded":"2024-09-16T15:42:52.419154+0000","last_fullsized":"2024-09-16T15:42:52.419154+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_clean_scrub_stamp":"2024-09-16T15:42:50.361965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:59:03.380666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"20'6","reported_seq":21,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T15:42:52.416548+0000","last_change":"2024-09-16T15:42:50.396749+0000","last_active":"2024-09-16T15:42:52.416548+0000","last_peered":"2024-09-16T15:42:52.416548+0000","last_clean":"2024-09-16T15:42:52.416548+0000","last_became_active":"2024-09-16T15:42:50.395963+0000","last_became_peered":"2024-09-16T15:42:50.395963+0000","last_unstale":"2024-09-16T15:42:52.416548+0000","last_undegraded":"2024-09-16T15:42:52.416548+0000","last_fullsized":"2024-09-16T15:42:52.416548+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:42:49.352371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:42:49.352371+0000","last_clean_scrub_stamp":"2024-09-16T15:42:49.352371+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T23:01:15.996297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T15:42:52.724385+0000","last_change":"2024-09-16T15:42:51.427006+0000","last_active":"2024-09-16T15:42:52.724385+0000","last_peered":"2024-09-16T15:42:52.724385+0000","last_clean":"2024-09-16T15:42:52.724385+0000","last_became_active":"2024-09-16T15:42:51.426730+0000","last_became_peered":"2024-09-16T15:42:51.426730+0000","last_unstale":"2024-09-16T15:42:52.724385+0000","last_undegraded":"2024-09-16T15:42:52.724385+0000","last_fullsized":"2024-09-16T15:42:52.724385+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_clean_scrub_stamp":"2024-09-16T15:42:50.361965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:49:13.477881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"20'2","reported_seq":17,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T15:42:52.876798+0000","last_change":"2024-09-16T15:42:50.401803+0000","last_active":"2024-09-16T15:42:52.876798+0000","last_peered":"2024-09-16T15:42:52.876798+0000","last_clean":"2024-09-16T15:42:52.876798+0000","last_became_active":"2024-09-16T15:42:50.401495+0000","last_became_peered":"2024-09-16T15:42:50.401495+0000","last_unstale":"2024-09-16T15:42:52.876798+0000","last_undegraded":"2024-09-16T15:42:52.876798+0000","last_fullsized":"2024-09-16T15:42:52.876798+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:42:49.352371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:42:49.352371+0000","last_clean_scrub_stamp":"2024-09-16T15:42:49.352371+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T18:06:23.054055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T15:42:52.875115+0000","last_change":"2024-09-16T15:42:51.428923+0000","last_active":"2024-09-16T15:42:52.875115+0000","last_peered":"2024-09-16T15:42:52.875115+0000","last_clean":"2024-09-16T15:42:52.875115+0000","last_became_active":"2024-09-16T15:42:51.428550+0000","last_became_peered":"2024-09-16T15:42:51.428550+0000","last_unstale":"2024-09-16T15:42:52.875115+0000","last_undegraded":"2024-09-16T15:42:52.875115+0000","last_fullsized":"2024-09-16T15:42:52.875115+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_clean_scrub_stamp":"2024-09-16T15:42:50.361965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:57:03.171257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T15:42:52.639097+0000","last_change":"2024-09-16T15:42:50.426919+0000","last_active":"2024-09-16T15:42:52.639097+0000","last_peered":"2024-09-16T15:42:52.639097+0000","last_clean":"2024-09-16T15:42:52.639097+0000","last_became_active":"2024-09-16T15:42:50.426141+0000","last_became_peered":"2024-09-16T15:42:50.426141+0000","last_unstale":"2024-09-16T15:42:52.639097+0000","last_undegraded":"2024-09-16T15:42:52.639097+0000","last_fullsized":"2024-09-16T15:42:52.639097+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:42:49.352371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:42:49.352371+0000","last_clean_scrub_stamp":"2024-09-16T15:42:49.352371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:23:09.046920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T15:42:52.724923+0000","last_change":"2024-09-16T15:42:51.425867+0000","last_active":"2024-09-16T15:42:52.724923+0000","last_peered":"2024-09-16T15:42:52.724923+0000","last_clean":"2024-09-16T15:42:52.724923+0000","last_became_active":"2024-09-16T15:42:51.425313+0000","last_became_peered":"2024-09-16T15:42:51.425313+0000","last_unstale":"2024-09-16T15:42:52.724923+0000","last_undegraded":"2024-09-16T15:42:52.724923+0000","last_fullsized":"2024-09-16T15:42:52.724923+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_clean_scrub_stamp":"2024-09-16T15:42:50.361965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T18:02:05.435198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T15:42:52.415548+0000","last_change":"2024-09-16T15:42:50.405263+0000","last_active":"2024-09-16T15:42:52.415548+0000","last_peered":"2024-09-16T15:42:52.415548+0000","last_clean":"2024-09-16T15:42:52.415548+0000","last_became_active":"2024-09-16T15:42:50.404419+0000","last_became_peered":"2024-09-16T15:42:50.404419+0000","last_unstale":"2024-09-16T15:42:52.415548+0000","last_undegraded":"2024-09-16T15:42:52.415548+0000","last_fullsized":"2024-09-16T15:42:52.415548+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:42:49.352371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:42:49.352371+0000","last_clean_scrub_stamp":"2024-09-16T15:42:49.352371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:40:33.707149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T15:42:52.415669+0000","last_change":"2024-09-16T15:42:51.397798+0000","last_active":"2024-09-16T15:42:52.415669+0000","last_peered":"2024-09-16T15:42:52.415669+0000","last_clean":"2024-09-16T15:42:52.415669+0000","last_became_active":"2024-09-16T15:42:51.397070+0000","last_became_peered":"2024-09-16T15:42:51.397070+0000","last_unstale":"2024-09-16T15:42:52.415669+0000","last_undegraded":"2024-09-16T15:42:52.415669+0000","last_fullsized":"2024-09-16T15:42:52.415669+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_clean_scrub_stamp":"2024-09-16T15:42:50.361965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:24:48.779934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"20'8","reported_seq":23,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T15:42:52.737837+0000","last_change":"2024-09-16T15:42:50.386440+0000","last_active":"2024-09-16T15:42:52.737837+0000","last_peered":"2024-09-16T15:42:52.737837+0000","last_clean":"2024-09-16T15:42:52.737837+0000","last_became_active":"2024-09-16T15:42:50.386100+0000","last_became_peered":"2024-09-16T15:42:50.386100+0000","last_unstale":"2024-09-16T15:42:52.737837+0000","last_undegraded":"2024-09-16T15:42:52.737837+0000","last_fullsized":"2024-09-16T15:42:52.737837+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:42:49.352371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:42:49.352371+0000","last_clean_scrub_stamp":"2024-09-16T15:42:49.352371+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:32:51.019126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T15:42:52.723952+0000","last_change":"2024-09-16T15:42:51.424679+0000","last_active":"2024-09-16T15:42:52.723952+0000","last_peered":"2024-09-16T15:42:52.723952+0000","last_clean":"2024-09-16T15:42:52.723952+0000","last_became_active":"2024-09-16T15:42:51.424119+0000","last_became_peered":"2024-09-16T15:42:51.424119+0000","last_unstale":"2024-09-16T15:42:52.723952+0000","last_undegraded":"2024-09-16T15:42:52.723952+0000","last_fullsized":"2024-09-16T15:42:52.723952+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_clean_scrub_stamp":"2024-09-16T15:42:50.361965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T23:53:36.869388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"20'1","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T15:42:52.419009+0000","last_change":"2024-09-16T15:42:50.395317+0000","last_active":"2024-09-16T15:42:52.419009+0000","last_peered":"2024-09-16T15:42:52.419009+0000","last_clean":"2024-09-16T15:42:52.419009+0000","last_became_active":"2024-09-16T15:42:50.394495+0000","last_became_peered":"2024-09-16T15:42:50.394495+0000","last_unstale":"2024-09-16T15:42:52.419009+0000","last_undegraded":"2024-09-16T15:42:52.419009+0000","last_fullsized":"2024-09-16T15:42:52.419009+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:42:49.352371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:42:49.352371+0000","last_clean_scrub_stamp":"2024-09-16T15:42:49.352371+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:45:54.681341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T15:42:52.724959+0000","last_change":"2024-09-16T15:42:51.423467+0000","last_active":"2024-09-16T15:42:52.724959+0000","last_peered":"2024-09-16T15:42:52.724959+0000","last_clean":"2024-09-16T15:42:52.724959+0000","last_became_active":"2024-09-16T15:42:51.423161+0000","last_became_peered":"2024-09-16T15:42:51.423161+0000","last_unstale":"2024-09-16T15:42:52.724959+0000","last_undegraded":"2024-09-16T15:42:52.724959+0000","last_fullsized":"2024-09-16T15:42:52.724959+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_clean_scrub_stamp":"2024-09-16T15:42:50.361965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T23:53:22.795898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"20'2","reported_seq":17,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T15:42:52.738233+0000","last_change":"2024-09-16T15:42:50.384867+0000","last_active":"2024-09-16T15:42:52.738233+0000","last_peered":"2024-09-16T15:42:52.738233+0000","last_clean":"2024-09-16T15:42:52.738233+0000","last_became_active":"2024-09-16T15:42:50.384581+0000","last_became_peered":"2024-09-16T15:42:50.384581+0000","last_unstale":"2024-09-16T15:42:52.738233+0000","last_undegraded":"2024-09-16T15:42:52.738233+0000","last_fullsized":"2024-09-16T15:42:52.738233+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:42:49.352371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:42:49.352371+0000","last_clean_scrub_stamp":"2024-09-16T15:42:49.352371+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T22:48:39.726611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T15:42:52.725417+0000","last_change":"2024-09-16T15:42:51.434874+0000","last_active":"2024-09-16T15:42:52.725417+0000","last_peered":"2024-09-16T15:42:52.725417+0000","last_clean":"2024-09-16T15:42:52.725417+0000","last_became_active":"2024-09-16T15:42:51.434561+0000","last_became_peered":"2024-09-16T15:42:51.434561+0000","last_unstale":"2024-09-16T15:42:52.725417+0000","last_undegraded":"2024-09-16T15:42:52.725417+0000","last_fullsized":"2024-09-16T15:42:52.725417+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_clean_scrub_stamp":"2024-09-16T15:42:50.361965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:13:10.293897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"20'1","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T15:42:52.418305+0000","last_change":"2024-09-16T15:42:50.399974+0000","last_active":"2024-09-16T15:42:52.418305+0000","last_peered":"2024-09-16T15:42:52.418305+0000","last_clean":"2024-09-16T15:42:52.418305+0000","last_became_active":"2024-09-16T15:42:50.399714+0000","last_became_peered":"2024-09-16T15:42:50.399714+0000","last_unstale":"2024-09-16T15:42:52.418305+0000","last_undegraded":"2024-09-16T15:42:52.418305+0000","last_fullsized":"2024-09-16T15:42:52.418305+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:42:49.352371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:42:49.352371+0000","last_clean_scrub_stamp":"2024-09-16T15:42:49.352371+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T02:37:24.494137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T15:42:52.639516+0000","last_change":"2024-09-16T15:42:51.420665+0000","last_active":"2024-09-16T15:42:52.639516+0000","last_peered":"2024-09-16T15:42:52.639516+0000","last_clean":"2024-09-16T15:42:52.639516+0000","last_became_active":"2024-09-16T15:42:51.420340+0000","last_became_peered":"2024-09-16T15:42:51.420340+0000","last_unstale":"2024-09-16T15:42:52.639516+0000","last_undegraded":"2024-09-16T15:42:52.639516+0000","last_fullsized":"2024-09-16T15:42:52.639516+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_clean_scrub_stamp":"2024-09-16T15:42:50.361965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T00:05:49.936465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T15:42:52.738027+0000","last_change":"2024-09-16T15:42:50.412850+0000","last_active":"2024-09-16T15:42:52.738027+0000","last_peered":"2024-09-16T15:42:52.738027+0000","last_clean":"2024-09-16T15:42:52.738027+0000","last_became_active":"2024-09-16T15:42:50.412492+0000","last_became_peered":"2024-09-16T15:42:50.412492+0000","last_unstale":"2024-09-16T15:42:52.738027+0000","last_undegraded":"2024-09-16T15:42:52.738027+0000","last_fullsized":"2024-09-16T15:42:52.738027+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:42:49.352371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:42:49.352371+0000","last_clean_scrub_stamp":"2024-09-16T15:42:49.352371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T22:41:47.420539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T15:42:52.724515+0000","last_change":"2024-09-16T15:42:51.428052+0000","last_active":"2024-09-16T15:42:52.724515+0000","last_peered":"2024-09-16T15:42:52.724515+0000","last_clean":"2024-09-16T15:42:52.724515+0000","last_became_active":"2024-09-16T15:42:51.427776+0000","last_became_peered":"2024-09-16T15:42:51.427776+0000","last_unstale":"2024-09-16T15:42:52.724515+0000","last_undegraded":"2024-09-16T15:42:52.724515+0000","last_fullsized":"2024-09-16T15:42:52.724515+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_clean_scrub_stamp":"2024-09-16T15:42:50.361965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T22:34:25.305861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"16'72","reported_seq":111,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T15:42:52.723702+0000","last_change":"2024-09-16T15:42:40.881820+0000","last_active":"2024-09-16T15:42:52.723702+0000","last_peered":"2024-09-16T15:42:52.723702+0000","last_clean":"2024-09-16T15:42:52.723702+0000","last_became_active":"2024-09-16T15:42:40.879965+0000","last_became_peered":"2024-09-16T15:42:40.879965+0000","last_unstale":"2024-09-16T15:42:52.723702+0000","last_undegraded":"2024-09-16T15:42:52.723702+0000","last_fullsized":"2024-09-16T15:42:52.723702+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:42:39.728762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:42:39.728762+0000","last_clean_scrub_stamp":"2024-09-16T15:42:39.728762+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:22:18.768932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T15:42:52.638416+0000","last_change":"2024-09-16T15:42:50.413082+0000","last_active":"2024-09-16T15:42:52.638416+0000","last_peered":"2024-09-16T15:42:52.638416+0000","last_clean":"2024-09-16T15:42:52.638416+0000","last_became_active":"2024-09-16T15:42:50.412050+0000","last_became_peered":"2024-09-16T15:42:50.412050+0000","last_unstale":"2024-09-16T15:42:52.638416+0000","last_undegraded":"2024-09-16T15:42:52.638416+0000","last_fullsized":"2024-09-16T15:42:52.638416+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:42:49.352371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:42:49.352371+0000","last_clean_scrub_stamp":"2024-09-16T15:42:49.352371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:32:51.019126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T15:42:52.875839+0000","last_change":"2024-09-16T15:42:51.430477+0000","last_active":"2024-09-16T15:42:52.875839+0000","last_peered":"2024-09-16T15:42:52.875839+0000","last_clean":"2024-09-16T15:42:52.875839+0000","last_became_active":"2024-09-16T15:42:51.430071+0000","last_became_peered":"2024-09-16T15:42:51.430071+0000","last_unstale":"2024-09-16T15:42:52.875839+0000","last_undegraded":"2024-09-16T15:42:52.875839+0000","last_fullsized":"2024-09-16T15:42:52.875839+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_clean_scrub_stamp":"2024-09-16T15:42:50.361965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T18:56:18.021098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T15:42:52.723623+0000","last_change":"2024-09-16T15:42:50.419981+0000","last_active":"2024-09-16T15:42:52.723623+0000","last_peered":"2024-09-16T15:42:52.723623+0000","last_clean":"2024-09-16T15:42:52.723623+0000","last_became_active":"2024-09-16T15:42:50.418405+0000","last_became_peered":"2024-09-16T15:42:50.418405+0000","last_unstale":"2024-09-16T15:42:52.723623+0000","last_undegraded":"2024-09-16T15:42:52.723623+0000","last_fullsized":"2024-09-16T15:42:52.723623+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:42:49.352371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:42:49.352371+0000","last_clean_scrub_stamp":"2024-09-16T15:42:49.352371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T00:53:52.113587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T15:42:52.874487+0000","last_change":"2024-09-16T15:42:51.403404+0000","last_active":"2024-09-16T15:42:52.874487+0000","last_peered":"2024-09-16T15:42:52.874487+0000","last_clean":"2024-09-16T15:42:52.874487+0000","last_became_active":"2024-09-16T15:42:51.402978+0000","last_became_peered":"2024-09-16T15:42:51.402978+0000","last_unstale":"2024-09-16T15:42:52.874487+0000","last_undegraded":"2024-09-16T15:42:52.874487+0000","last_fullsized":"2024-09-16T15:42:52.874487+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_clean_scrub_stamp":"2024-09-16T15:42:50.361965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T00:04:54.212949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T15:42:52.738033+0000","last_change":"2024-09-16T15:42:50.391465+0000","last_active":"2024-09-16T15:42:52.738033+0000","last_peered":"2024-09-16T15:42:52.738033+0000","last_clean":"2024-09-16T15:42:52.738033+0000","last_became_active":"2024-09-16T15:42:50.391091+0000","last_became_peered":"2024-09-16T15:42:50.391091+0000","last_unstale":"2024-09-16T15:42:52.738033+0000","last_undegraded":"2024-09-16T15:42:52.738033+0000","last_fullsized":"2024-09-16T15:42:52.738033+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:42:49.352371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:42:49.352371+0000","last_clean_scrub_stamp":"2024-09-16T15:42:49.352371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:26:07.798125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T15:42:52.417576+0000","last_change":"2024-09-16T15:42:51.404294+0000","last_active":"2024-09-16T15:42:52.417576+0000","last_peered":"2024-09-16T15:42:52.417576+0000","last_clean":"2024-09-16T15:42:52.417576+0000","last_became_active":"2024-09-16T15:42:51.403997+0000","last_became_peered":"2024-09-16T15:42:51.403997+0000","last_unstale":"2024-09-16T15:42:52.417576+0000","last_undegraded":"2024-09-16T15:42:52.417576+0000","last_fullsized":"2024-09-16T15:42:52.417576+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_clean_scrub_stamp":"2024-09-16T15:42:50.361965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:20:36.872940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T15:42:52.639350+0000","last_change":"2024-09-16T15:42:50.422498+0000","last_active":"2024-09-16T15:42:52.639350+0000","last_peered":"2024-09-16T15:42:52.639350+0000","last_clean":"2024-09-16T15:42:52.639350+0000","last_became_active":"2024-09-16T15:42:50.421727+0000","last_became_peered":"2024-09-16T15:42:50.421727+0000","last_unstale":"2024-09-16T15:42:52.639350+0000","last_undegraded":"2024-09-16T15:42:52.639350+0000","last_fullsized":"2024-09-16T15:42:52.639350+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:42:49.352371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:42:49.352371+0000","last_clean_scrub_stamp":"2024-09-16T15:42:49.352371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:36:36.506348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T15:42:52.419302+0000","last_change":"2024-09-16T15:42:51.413177+0000","last_active":"2024-09-16T15:42:52.419302+0000","last_peered":"2024-09-16T15:42:52.419302+0000","last_clean":"2024-09-16T15:42:52.419302+0000","last_became_active":"2024-09-16T15:42:51.412881+0000","last_became_peered":"2024-09-16T15:42:51.412881+0000","last_unstale":"2024-09-16T15:42:52.419302+0000","last_undegraded":"2024-09-16T15:42:52.419302+0000","last_fullsized":"2024-09-16T15:42:52.419302+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_clean_scrub_stamp":"2024-09-16T15:42:50.361965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:10:16.623518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T15:42:52.418085+0000","last_change":"2024-09-16T15:42:50.394813+0000","last_active":"2024-09-16T15:42:52.418085+0000","last_peered":"2024-09-16T15:42:52.418085+0000","last_clean":"2024-09-16T15:42:52.418085+0000","last_became_active":"2024-09-16T15:42:50.389679+0000","last_became_peered":"2024-09-16T15:42:50.389679+0000","last_unstale":"2024-09-16T15:42:52.418085+0000","last_undegraded":"2024-09-16T15:42:52.418085+0000","last_fullsized":"2024-09-16T15:42:52.418085+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:42:49.352371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:42:49.352371+0000","last_clean_scrub_stamp":"2024-09-16T15:42:49.352371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:23:09.046920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T15:42:52.639557+0000","last_change":"2024-09-16T15:42:51.424119+0000","last_active":"2024-09-16T15:42:52.639557+0000","last_peered":"2024-09-16T15:42:52.639557+0000","last_clean":"2024-09-16T15:42:52.639557+0000","last_became_active":"2024-09-16T15:42:51.423771+0000","last_became_peered":"2024-09-16T15:42:51.423771+0000","last_unstale":"2024-09-16T15:42:52.639557+0000","last_undegraded":"2024-09-16T15:42:52.639557+0000","last_fullsized":"2024-09-16T15:42:52.639557+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_clean_scrub_stamp":"2024-09-16T15:42:50.361965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T23:53:22.795898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T15:42:52.724991+0000","last_change":"2024-09-16T15:42:50.419065+0000","last_active":"2024-09-16T15:42:52.724991+0000","last_peered":"2024-09-16T15:42:52.724991+0000","last_clean":"2024-09-16T15:42:52.724991+0000","last_became_active":"2024-09-16T15:42:50.418438+0000","last_became_peered":"2024-09-16T15:42:50.418438+0000","last_unstale":"2024-09-16T15:42:52.724991+0000","last_undegraded":"2024-09-16T15:42:52.724991+0000","last_fullsized":"2024-09-16T15:42:52.724991+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:42:49.352371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:42:49.352371+0000","last_clean_scrub_stamp":"2024-09-16T15:42:49.352371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T01:42:29.977143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T15:42:52.418120+0000","last_change":"2024-09-16T15:42:51.407174+0000","last_active":"2024-09-16T15:42:52.418120+0000","last_peered":"2024-09-16T15:42:52.418120+0000","last_clean":"2024-09-16T15:42:52.418120+0000","last_became_active":"2024-09-16T15:42:51.406880+0000","last_became_peered":"2024-09-16T15:42:51.406880+0000","last_unstale":"2024-09-16T15:42:52.418120+0000","last_undegraded":"2024-09-16T15:42:52.418120+0000","last_fullsized":"2024-09-16T15:42:52.418120+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_clean_scrub_stamp":"2024-09-16T15:42:50.361965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T02:33:57.643582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"20'1","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T15:42:52.418471+0000","last_change":"2024-09-16T15:42:50.397599+0000","last_active":"2024-09-16T15:42:52.418471+0000","last_peered":"2024-09-16T15:42:52.418471+0000","last_clean":"2024-09-16T15:42:52.418471+0000","last_became_active":"2024-09-16T15:42:50.397301+0000","last_became_peered":"2024-09-16T15:42:50.397301+0000","last_unstale":"2024-09-16T15:42:52.418471+0000","last_undegraded":"2024-09-16T15:42:52.418471+0000","last_fullsized":"2024-09-16T15:42:52.418471+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:42:49.352371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:42:49.352371+0000","last_clean_scrub_stamp":"2024-09-16T15:42:49.352371+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T01:49:08.734586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T15:42:52.875834+0000","last_change":"2024-09-16T15:42:51.424110+0000","last_active":"2024-09-16T15:42:52.875834+0000","last_peered":"2024-09-16T15:42:52.875834+0000","last_clean":"2024-09-16T15:42:52.875834+0000","last_became_active":"2024-09-16T15:42:51.423798+0000","last_became_peered":"2024-09-16T15:42:51.423798+0000","last_unstale":"2024-09-16T15:42:52.875834+0000","last_undegraded":"2024-09-16T15:42:52.875834+0000","last_fullsized":"2024-09-16T15:42:52.875834+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_clean_scrub_stamp":"2024-09-16T15:42:50.361965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T22:53:48.581346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"20'2","reported_seq":17,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T15:42:52.417268+0000","last_change":"2024-09-16T15:42:50.416032+0000","last_active":"2024-09-16T15:42:52.417268+0000","last_peered":"2024-09-16T15:42:52.417268+0000","last_clean":"2024-09-16T15:42:52.417268+0000","last_became_active":"2024-09-16T15:42:50.415325+0000","last_became_peered":"2024-09-16T15:42:50.415325+0000","last_unstale":"2024-09-16T15:42:52.417268+0000","last_undegraded":"2024-09-16T15:42:52.417268+0000","last_fullsized":"2024-09-16T15:42:52.417268+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:42:49.352371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:42:49.352371+0000","last_clean_scrub_stamp":"2024-09-16T15:42:49.352371+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T02:11:26.332763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T15:42:52.877501+0000","last_change":"2024-09-16T15:42:51.407137+0000","last_active":"2024-09-16T15:42:52.877501+0000","last_peered":"2024-09-16T15:42:52.877501+0000","last_clean":"2024-09-16T15:42:52.877501+0000","last_became_active":"2024-09-16T15:42:51.406766+0000","last_became_peered":"2024-09-16T15:42:51.406766+0000","last_unstale":"2024-09-16T15:42:52.877501+0000","last_undegraded":"2024-09-16T15:42:52.877501+0000","last_fullsized":"2024-09-16T15:42:52.877501+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_clean_scrub_stamp":"2024-09-16T15:42:50.361965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:07:17.519138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T15:42:52.723539+0000","last_change":"2024-09-16T15:42:50.419106+0000","last_active":"2024-09-16T15:42:52.723539+0000","last_peered":"2024-09-16T15:42:52.723539+0000","last_clean":"2024-09-16T15:42:52.723539+0000","last_became_active":"2024-09-16T15:42:50.417462+0000","last_became_peered":"2024-09-16T15:42:50.417462+0000","last_unstale":"2024-09-16T15:42:52.723539+0000","last_undegraded":"2024-09-16T15:42:52.723539+0000","last_fullsized":"2024-09-16T15:42:52.723539+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:42:49.352371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:42:49.352371+0000","last_clean_scrub_stamp":"2024-09-16T15:42:49.352371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T22:58:24.474495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T15:42:52.418379+0000","last_change":"2024-09-16T15:42:51.412043+0000","last_active":"2024-09-16T15:42:52.418379+0000","last_peered":"2024-09-16T15:42:52.418379+0000","last_clean":"2024-09-16T15:42:52.418379+0000","last_became_active":"2024-09-16T15:42:51.411434+0000","last_became_peered":"2024-09-16T15:42:51.411434+0000","last_unstale":"2024-09-16T15:42:52.418379+0000","last_undegraded":"2024-09-16T15:42:52.418379+0000","last_fullsized":"2024-09-16T15:42:52.418379+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_clean_scrub_stamp":"2024-09-16T15:42:50.361965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:40:52.417913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T15:42:52.417895+0000","last_change":"2024-09-16T15:42:50.408565+0000","last_active":"2024-09-16T15:42:52.417895+0000","last_peered":"2024-09-16T15:42:52.417895+0000","last_clean":"2024-09-16T15:42:52.417895+0000","last_became_active":"2024-09-16T15:42:50.408254+0000","last_became_peered":"2024-09-16T15:42:50.408254+0000","last_unstale":"2024-09-16T15:42:52.417895+0000","last_undegraded":"2024-09-16T15:42:52.417895+0000","last_fullsized":"2024-09-16T15:42:52.417895+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:42:49.352371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:42:49.352371+0000","last_clean_scrub_stamp":"2024-09-16T15:42:49.352371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:26:07.798125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T15:42:52.876598+0000","last_change":"2024-09-16T15:42:51.413873+0000","last_active":"2024-09-16T15:42:52.876598+0000","last_peered":"2024-09-16T15:42:52.876598+0000","last_clean":"2024-09-16T15:42:52.876598+0000","last_became_active":"2024-09-16T15:42:51.406117+0000","last_became_peered":"2024-09-16T15:42:51.406117+0000","last_unstale":"2024-09-16T15:42:52.876598+0000","last_undegraded":"2024-09-16T15:42:52.876598+0000","last_fullsized":"2024-09-16T15:42:52.876598+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_clean_scrub_stamp":"2024-09-16T15:42:50.361965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:12:23.570707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T15:42:52.417696+0000","last_change":"2024-09-16T15:42:50.388233+0000","last_active":"2024-09-16T15:42:52.417696+0000","last_peered":"2024-09-16T15:42:52.417696+0000","last_clean":"2024-09-16T15:42:52.417696+0000","last_became_active":"2024-09-16T15:42:50.387920+0000","last_became_peered":"2024-09-16T15:42:50.387920+0000","last_unstale":"2024-09-16T15:42:52.417696+0000","last_undegraded":"2024-09-16T15:42:52.417696+0000","last_fullsized":"2024-09-16T15:42:52.417696+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:42:49.352371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:42:49.352371+0000","last_clean_scrub_stamp":"2024-09-16T15:42:49.352371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T22:48:39.726611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T15:42:52.876523+0000","last_change":"2024-09-16T15:42:51.414425+0000","last_active":"2024-09-16T15:42:52.876523+0000","last_peered":"2024-09-16T15:42:52.876523+0000","last_clean":"2024-09-16T15:42:52.876523+0000","last_became_active":"2024-09-16T15:42:51.410797+0000","last_became_peered":"2024-09-16T15:42:51.410797+0000","last_unstale":"2024-09-16T15:42:52.876523+0000","last_undegraded":"2024-09-16T15:42:52.876523+0000","last_fullsized":"2024-09-16T15:42:52.876523+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_clean_scrub_stamp":"2024-09-16T15:42:50.361965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T18:40:25.285207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"20'2","reported_seq":17,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T15:42:52.725055+0000","last_change":"2024-09-16T15:42:50.423780+0000","last_active":"2024-09-16T15:42:52.725055+0000","last_peered":"2024-09-16T15:42:52.725055+0000","last_clean":"2024-09-16T15:42:52.725055+0000","last_became_active":"2024-09-16T15:42:50.423037+0000","last_became_peered":"2024-09-16T15:42:50.423037+0000","last_unstale":"2024-09-16T15:42:52.725055+0000","last_undegraded":"2024-09-16T15:42:52.725055+0000","last_fullsized":"2024-09-16T15:42:52.725055+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:42:49.352371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:42:49.352371+0000","last_clean_scrub_stamp":"2024-09-16T15:42:49.352371+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T01:04:05.966343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T15:42:52.415974+0000","last_change":"2024-09-16T15:42:51.397509+0000","last_active":"2024-09-16T15:42:52.415974+0000","last_peered":"2024-09-16T15:42:52.415974+0000","last_clean":"2024-09-16T15:42:52.415974+0000","last_became_active":"2024-09-16T15:42:51.396735+0000","last_became_peered":"2024-09-16T15:42:51.396735+0000","last_unstale":"2024-09-16T15:42:52.415974+0000","last_undegraded":"2024-09-16T15:42:52.415974+0000","last_fullsized":"2024-09-16T15:42:52.415974+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_clean_scrub_stamp":"2024-09-16T15:42:50.361965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T02:47:48.200896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T15:42:52.639530+0000","last_change":"2024-09-16T15:42:51.423874+0000","last_active":"2024-09-16T15:42:52.639530+0000","last_peered":"2024-09-16T15:42:52.639530+0000","last_clean":"2024-09-16T15:42:52.639530+0000","last_became_active":"2024-09-16T15:42:51.423455+0000","last_became_peered":"2024-09-16T15:42:51.423455+0000","last_unstale":"2024-09-16T15:42:52.639530+0000","last_undegraded":"2024-09-16T15:42:52.639530+0000","last_fullsized":"2024-09-16T15:42:52.639530+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_clean_scrub_stamp":"2024-09-16T15:42:50.361965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T22:43:57.612493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"20'2","reported_seq":17,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T15:42:52.724179+0000","last_change":"2024-09-16T15:42:50.428990+0000","last_active":"2024-09-16T15:42:52.724179+0000","last_peered":"2024-09-16T15:42:52.724179+0000","last_clean":"2024-09-16T15:42:52.724179+0000","last_became_active":"2024-09-16T15:42:50.428159+0000","last_became_peered":"2024-09-16T15:42:50.428159+0000","last_unstale":"2024-09-16T15:42:52.724179+0000","last_undegraded":"2024-09-16T15:42:52.724179+0000","last_fullsized":"2024-09-16T15:42:52.724179+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:42:49.352371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:42:49.352371+0000","last_clean_scrub_stamp":"2024-09-16T15:42:49.352371+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T01:17:11.293386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T15:42:52.638975+0000","last_change":"2024-09-16T15:42:51.421429+0000","last_active":"2024-09-16T15:42:52.638975+0000","last_peered":"2024-09-16T15:42:52.638975+0000","last_clean":"2024-09-16T15:42:52.638975+0000","last_became_active":"2024-09-16T15:42:51.420671+0000","last_became_peered":"2024-09-16T15:42:51.420671+0000","last_unstale":"2024-09-16T15:42:52.638975+0000","last_undegraded":"2024-09-16T15:42:52.638975+0000","last_fullsized":"2024-09-16T15:42:52.638975+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_clean_scrub_stamp":"2024-09-16T15:42:50.361965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T00:34:44.681110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"20'2","reported_seq":17,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T15:42:52.724740+0000","last_change":"2024-09-16T15:42:50.425037+0000","last_active":"2024-09-16T15:42:52.724740+0000","last_peered":"2024-09-16T15:42:52.724740+0000","last_clean":"2024-09-16T15:42:52.724740+0000","last_became_active":"2024-09-16T15:42:50.423907+0000","last_became_peered":"2024-09-16T15:42:50.423907+0000","last_unstale":"2024-09-16T15:42:52.724740+0000","last_undegraded":"2024-09-16T15:42:52.724740+0000","last_fullsized":"2024-09-16T15:42:52.724740+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:42:49.352371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:42:49.352371+0000","last_clean_scrub_stamp":"2024-09-16T15:42:49.352371+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:56:59.748634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T15:42:52.877982+0000","last_change":"2024-09-16T15:42:51.425681+0000","last_active":"2024-09-16T15:42:52.877982+0000","last_peered":"2024-09-16T15:42:52.877982+0000","last_clean":"2024-09-16T15:42:52.877982+0000","last_became_active":"2024-09-16T15:42:51.425367+0000","last_became_peered":"2024-09-16T15:42:51.425367+0000","last_unstale":"2024-09-16T15:42:52.877982+0000","last_undegraded":"2024-09-16T15:42:52.877982+0000","last_fullsized":"2024-09-16T15:42:52.877982+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_clean_scrub_stamp":"2024-09-16T15:42:50.361965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:21.758409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"20'1","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T15:42:52.875580+0000","last_change":"2024-09-16T15:42:50.392203+0000","last_active":"2024-09-16T15:42:52.875580+0000","last_peered":"2024-09-16T15:42:52.875580+0000","last_clean":"2024-09-16T15:42:52.875580+0000","last_became_active":"2024-09-16T15:42:50.391647+0000","last_became_peered":"2024-09-16T15:42:50.391647+0000","last_unstale":"2024-09-16T15:42:52.875580+0000","last_undegraded":"2024-09-16T15:42:52.875580+0000","last_fullsized":"2024-09-16T15:42:52.875580+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:42:49.352371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:42:49.352371+0000","last_clean_scrub_stamp":"2024-09-16T15:42:49.352371+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T22:35:29.889897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T15:42:52.876061+0000","last_change":"2024-09-16T15:42:51.408021+0000","last_active":"2024-09-16T15:42:52.876061+0000","last_peered":"2024-09-16T15:42:52.876061+0000","last_clean":"2024-09-16T15:42:52.876061+0000","last_became_active":"2024-09-16T15:42:51.407085+0000","last_became_peered":"2024-09-16T15:42:51.407085+0000","last_unstale":"2024-09-16T15:42:52.876061+0000","last_undegraded":"2024-09-16T15:42:52.876061+0000","last_fullsized":"2024-09-16T15:42:52.876061+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_clean_scrub_stamp":"2024-09-16T15:42:50.361965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:38:38.580439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"20'4","reported_seq":19,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T15:42:52.878108+0000","last_change":"2024-09-16T15:42:50.415954+0000","last_active":"2024-09-16T15:42:52.878108+0000","last_peered":"2024-09-16T15:42:52.878108+0000","last_clean":"2024-09-16T15:42:52.878108+0000","last_became_active":"2024-09-16T15:42:50.415619+0000","last_became_peered":"2024-09-16T15:42:50.415619+0000","last_unstale":"2024-09-16T15:42:52.878108+0000","last_undegraded":"2024-09-16T15:42:52.878108+0000","last_fullsized":"2024-09-16T15:42:52.878108+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:42:49.352371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:42:49.352371+0000","last_clean_scrub_stamp":"2024-09-16T15:42:49.352371+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T02:01:17.143570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542151,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27824,"kb_used_data":792,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344016,"statfs":{"total":96636764160,"available":96608272384,"internally_reserved":0,"allocated":811008,"data_stored":665039,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542150,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27256,"kb_used_data":220,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344584,"statfs":{"total":96636764160,"available":96608854016,"internally_reserved":0,"allocated":225280,"data_stored":75701,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":14,"seq":60129542151,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27240,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344600,"statfs":{"total":96636764160,"available":96608870400,"internally_reserved":0,"allocated":217088,"data_stored":74179,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542149,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27252,"kb_used_data":220,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344588,"statfs":{"total":96636764160,"available":96608858112,"internally_reserved":0,"allocated":225280,"data_stored":74215,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":14,"seq":60129542150,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27256,"kb_used_data":220,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344584,"statfs":{"total":96636764160,"available":96608854016,"internally_reserved":0,"allocated":225280,"data_stored":75701,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":14,"seq":60129542150,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27236,"kb_used_data":208,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344604,"statfs":{"total":96636764160,"available":96608874496,"internally_reserved":0,"allocated":212992,"data_stored":74089,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":14,"seq":60129542152,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27244,"kb_used_data":216,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344596,"statfs":{"total":96636764160,"available":96608866304,"internally_reserved":0,"allocated":221184,"data_stored":74249,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":14,"seq":60129542152,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27832,"kb_used_data":800,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344008,"statfs":{"total":96636764160,"available":96608264192,"internally_reserved":0,"allocated":819200,"data_stored":665095,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T15:43:01.101 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T15:43:01.259 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:01.258+0000 7f28b5024640 1 -- 172.21.15.123:0/2364919321 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f289c0040c0 msgr2=0x7f289c004810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:43:01.259 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:01.258+0000 7f28b5024640 1 --2- 172.21.15.123:0/2364919321 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f289c0040c0 0x7f289c004810 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f28a00016c0 tx=0x7f28a001c790 comp rx=0 tx=0).stop 2024-09-16T15:43:01.260 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:01.258+0000 7f28af59e640 1 -- 172.21.15.123:0/2364919321 reap_dead start 2024-09-16T15:43:01.260 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:01.258+0000 7f28b5024640 1 -- 172.21.15.123:0/2364919321 shutdown_connections 2024-09-16T15:43:01.260 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:01.258+0000 7f28b5024640 1 -- 172.21.15.123:0/2364919321 >> 172.21.15.123:0/2364919321 conn(0x7f28b0074bd0 msgr2=0x7f28b009aa40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:43:01.260 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:01.258+0000 7f28b5024640 1 -- 172.21.15.123:0/2364919321 shutdown_connections 2024-09-16T15:43:01.260 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:01.258+0000 7f28b5024640 1 -- 172.21.15.123:0/2364919321 wait complete. 2024-09-16T15:43:01.260 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:01.258+0000 7f28b5024640 1 Processor -- start 2024-09-16T15:43:01.261 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:01.258+0000 7f28b5024640 1 -- start start 2024-09-16T15:43:01.262 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:01.258+0000 7f28b5024640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f289c0040c0 0x7f28b01532f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:43:01.262 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:01.258+0000 7f28b5024640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f28b00c6b10 0x7f28b00c6f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:43:01.262 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:01.258+0000 7f28b5024640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f28b00b04c0 0x7f28b00b0940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:43:01.262 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:01.258+0000 7f28b5024640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f28b0167a60 con 0x7f28b00b04c0 2024-09-16T15:43:01.262 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:01.258+0000 7f28b5024640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f28b01678e0 con 0x7f28b00c6b10 2024-09-16T15:43:01.263 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:01.258+0000 7f28b5024640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f28b0167be0 con 0x7f289c0040c0 2024-09-16T15:43:01.263 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:01.258+0000 7f28af59e640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f28b00b04c0 0x7f28b00b0940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:43:01.263 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:01.258+0000 7f28af59e640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f28b00b04c0 0x7f28b00b0940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:40724/0 (socket says 172.21.15.123:40724) 2024-09-16T15:43:01.263 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:01.258+0000 7f28af59e640 1 -- 172.21.15.123:0/1618196190 learned_addr learned my addr 172.21.15.123:0/1618196190 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:43:01.263 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:01.258+0000 7f28aed9d640 1 --2- 172.21.15.123:0/1618196190 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f289c0040c0 0x7f28b01532f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:43:01.263 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:01.258+0000 7f28ae59c640 1 --2- 172.21.15.123:0/1618196190 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f28b00c6b10 0x7f28b00c6f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:43:01.263 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:01.258+0000 7f28af59e640 1 -- 172.21.15.123:0/1618196190 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f289c0040c0 msgr2=0x7f28b01532f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:43:01.263 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:01.258+0000 7f28af59e640 1 --2- 172.21.15.123:0/1618196190 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f289c0040c0 0x7f28b01532f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:01.263 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:01.258+0000 7f28af59e640 1 -- 172.21.15.123:0/1618196190 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f28b00c6b10 msgr2=0x7f28b00c6f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:43:01.263 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:01.258+0000 7f28af59e640 1 --2- 172.21.15.123:0/1618196190 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f28b00c6b10 0x7f28b00c6f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:01.263 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:01.258+0000 7f28af59e640 1 -- 172.21.15.123:0/1618196190 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f28b0153830 con 0x7f28b00b04c0 2024-09-16T15:43:01.263 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:01.258+0000 7f28af59e640 1 --2- 172.21.15.123:0/1618196190 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f28b00b04c0 0x7f28b00b0940 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f28b00d2730 tx=0x7f28a800dd90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:43:01.263 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:01.258+0000 7f28a77fe640 1 -- 172.21.15.123:0/1618196190 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28a8004d30 con 0x7f28b00b04c0 2024-09-16T15:43:01.263 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:01.258+0000 7f28a77fe640 1 -- 172.21.15.123:0/1618196190 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f28a800ba20 con 0x7f28b00b04c0 2024-09-16T15:43:01.264 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:01.258+0000 7f28b5024640 1 -- 172.21.15.123:0/1618196190 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f28b00c7660 con 0x7f28b00b04c0 2024-09-16T15:43:01.264 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:01.258+0000 7f28a77fe640 1 -- 172.21.15.123:0/1618196190 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28a800be50 con 0x7f28b00b04c0 2024-09-16T15:43:01.264 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:01.258+0000 7f28b5024640 1 -- 172.21.15.123:0/1618196190 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f28b00c7c20 con 0x7f28b00b04c0 2024-09-16T15:43:01.264 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:01.262+0000 7f28b5024640 1 -- 172.21.15.123:0/1618196190 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f28b00d1c10 con 0x7f28b00b04c0 2024-09-16T15:43:01.264 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:01.262+0000 7f28a77fe640 1 -- 172.21.15.123:0/1618196190 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 5) ==== 97080+0+0 (secure 0 0 0) 0x7f28a8010460 con 0x7f28b00b04c0 2024-09-16T15:43:01.264 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:01.262+0000 7f28a77fe640 1 --2- 172.21.15.123:0/1618196190 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f2890072ef0 0x7f28900753b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:43:01.265 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:01.262+0000 7f28a77fe640 1 -- 172.21.15.123:0/1618196190 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f28a80937f0 con 0x7f28b00b04c0 2024-09-16T15:43:01.265 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:01.262+0000 7f28aed9d640 1 --2- 172.21.15.123:0/1618196190 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f2890072ef0 0x7f28900753b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:43:01.265 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:01.262+0000 7f28aed9d640 1 --2- 172.21.15.123:0/1618196190 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f2890072ef0 0x7f28900753b0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f2898001870 tx=0x7f2898001e40 comp rx=0 tx=0).ready entity=mgr.4135 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:43:01.268 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:01.266+0000 7f28a77fe640 1 -- 172.21.15.123:0/1618196190 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f28a805ea40 con 0x7f28b00b04c0 2024-09-16T15:43:01.424 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:01.422+0000 7f28b5024640 1 -- 172.21.15.123:0/1618196190 --> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f28b00ce400 con 0x7f2890072ef0 2024-09-16T15:43:01.436 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:01.434+0000 7f28a77fe640 1 -- 172.21.15.123:0/1618196190 <== mgr.4135 v2:172.21.15.184:6832/3114346423 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+251825 (secure 0 0 0) 0x7f28b00ce400 con 0x7f2890072ef0 2024-09-16T15:43:01.438 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-09-16T15:43:01.440 INFO:teuthology.orchestra.run.smithi123.stderr:dumped all 2024-09-16T15:43:01.440 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:01.438+0000 7f28b5024640 1 -- 172.21.15.123:0/1618196190 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f2890072ef0 msgr2=0x7f28900753b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:43:01.441 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:01.438+0000 7f28b5024640 1 --2- 172.21.15.123:0/1618196190 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f2890072ef0 0x7f28900753b0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f2898001870 tx=0x7f2898001e40 comp rx=0 tx=0).stop 2024-09-16T15:43:01.441 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:01.438+0000 7f28b5024640 1 -- 172.21.15.123:0/1618196190 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f28b00b04c0 msgr2=0x7f28b00b0940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:43:01.441 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:01.438+0000 7f28b5024640 1 --2- 172.21.15.123:0/1618196190 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f28b00b04c0 0x7f28b00b0940 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f28b00d2730 tx=0x7f28a800dd90 comp rx=0 tx=0).stop 2024-09-16T15:43:01.441 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:01.438+0000 7f28b5024640 1 -- 172.21.15.123:0/1618196190 shutdown_connections 2024-09-16T15:43:01.441 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:01.438+0000 7f28b5024640 1 --2- 172.21.15.123:0/1618196190 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f2890072ef0 0x7f28900753b0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:01.441 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:01.438+0000 7f28b5024640 1 --2- 172.21.15.123:0/1618196190 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f28b00b04c0 0x7f28b00b0940 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:01.441 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:01.438+0000 7f28b5024640 1 --2- 172.21.15.123:0/1618196190 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f28b00c6b10 0x7f28b00c6f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:01.441 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:01.438+0000 7f28b5024640 1 --2- 172.21.15.123:0/1618196190 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f289c0040c0 0x7f28b01532f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:01.441 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:01.438+0000 7f28b5024640 1 -- 172.21.15.123:0/1618196190 >> 172.21.15.123:0/1618196190 conn(0x7f28b0074bd0 msgr2=0x7f28b0070730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:43:01.441 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:01.438+0000 7f28b5024640 1 -- 172.21.15.123:0/1618196190 shutdown_connections 2024-09-16T15:43:01.441 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:01.438+0000 7f28b5024640 1 -- 172.21.15.123:0/1618196190 wait complete. 2024-09-16T15:43:01.455 INFO:teuthology.orchestra.run.smithi123.stdout:{"pg_ready":true,"pg_map":{"version":25,"stamp":"2024-09-16T15:43:01.248526+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":146,"num_write_kb":1642,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":111,"ondisk_log_size":111,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219140,"kb_used_data":2888,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755580,"statfs":{"total":773094113280,"available":772869713920,"internally_reserved":0,"allocated":2957312,"data_stored":1778268,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.870117"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T15:42:52.725091+0000","last_change":"2024-09-16T15:42:51.434234+0000","last_active":"2024-09-16T15:42:52.725091+0000","last_peered":"2024-09-16T15:42:52.725091+0000","last_clean":"2024-09-16T15:42:52.725091+0000","last_became_active":"2024-09-16T15:42:51.433660+0000","last_became_peered":"2024-09-16T15:42:51.433660+0000","last_unstale":"2024-09-16T15:42:52.725091+0000","last_undegraded":"2024-09-16T15:42:52.725091+0000","last_fullsized":"2024-09-16T15:42:52.725091+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_clean_scrub_stamp":"2024-09-16T15:42:50.361965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T02:07:35.605190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T15:42:52.418662+0000","last_change":"2024-09-16T15:42:51.412677+0000","last_active":"2024-09-16T15:42:52.418662+0000","last_peered":"2024-09-16T15:42:52.418662+0000","last_clean":"2024-09-16T15:42:52.418662+0000","last_became_active":"2024-09-16T15:42:51.412366+0000","last_became_peered":"2024-09-16T15:42:51.412366+0000","last_unstale":"2024-09-16T15:42:52.418662+0000","last_undegraded":"2024-09-16T15:42:52.418662+0000","last_fullsized":"2024-09-16T15:42:52.418662+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_clean_scrub_stamp":"2024-09-16T15:42:50.361965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:21.758409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T15:42:52.876476+0000","last_change":"2024-09-16T15:42:51.406454+0000","last_active":"2024-09-16T15:42:52.876476+0000","last_peered":"2024-09-16T15:42:52.876476+0000","last_clean":"2024-09-16T15:42:52.876476+0000","last_became_active":"2024-09-16T15:42:51.405679+0000","last_became_peered":"2024-09-16T15:42:51.405679+0000","last_unstale":"2024-09-16T15:42:52.876476+0000","last_undegraded":"2024-09-16T15:42:52.876476+0000","last_fullsized":"2024-09-16T15:42:52.876476+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_clean_scrub_stamp":"2024-09-16T15:42:50.361965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T02:33:32.417205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T15:42:52.638879+0000","last_change":"2024-09-16T15:42:51.423931+0000","last_active":"2024-09-16T15:42:52.638879+0000","last_peered":"2024-09-16T15:42:52.638879+0000","last_clean":"2024-09-16T15:42:52.638879+0000","last_became_active":"2024-09-16T15:42:51.423561+0000","last_became_peered":"2024-09-16T15:42:51.423561+0000","last_unstale":"2024-09-16T15:42:52.638879+0000","last_undegraded":"2024-09-16T15:42:52.638879+0000","last_fullsized":"2024-09-16T15:42:52.638879+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_clean_scrub_stamp":"2024-09-16T15:42:50.361965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T03:35:03.081638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T15:42:52.724573+0000","last_change":"2024-09-16T15:42:51.425601+0000","last_active":"2024-09-16T15:42:52.724573+0000","last_peered":"2024-09-16T15:42:52.724573+0000","last_clean":"2024-09-16T15:42:52.724573+0000","last_became_active":"2024-09-16T15:42:51.425040+0000","last_became_peered":"2024-09-16T15:42:51.425040+0000","last_unstale":"2024-09-16T15:42:52.724573+0000","last_undegraded":"2024-09-16T15:42:52.724573+0000","last_fullsized":"2024-09-16T15:42:52.724573+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_clean_scrub_stamp":"2024-09-16T15:42:50.361965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T00:05:49.936465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T15:42:52.638668+0000","last_change":"2024-09-16T15:42:51.429218+0000","last_active":"2024-09-16T15:42:52.638668+0000","last_peered":"2024-09-16T15:42:52.638668+0000","last_clean":"2024-09-16T15:42:52.638668+0000","last_became_active":"2024-09-16T15:42:51.428838+0000","last_became_peered":"2024-09-16T15:42:51.428838+0000","last_unstale":"2024-09-16T15:42:52.638668+0000","last_undegraded":"2024-09-16T15:42:52.638668+0000","last_fullsized":"2024-09-16T15:42:52.638668+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_clean_scrub_stamp":"2024-09-16T15:42:50.361965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:21.758409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T15:42:52.723991+0000","last_change":"2024-09-16T15:42:51.424389+0000","last_active":"2024-09-16T15:42:52.723991+0000","last_peered":"2024-09-16T15:42:52.723991+0000","last_clean":"2024-09-16T15:42:52.723991+0000","last_became_active":"2024-09-16T15:42:51.423840+0000","last_became_peered":"2024-09-16T15:42:51.423840+0000","last_unstale":"2024-09-16T15:42:52.723991+0000","last_undegraded":"2024-09-16T15:42:52.723991+0000","last_fullsized":"2024-09-16T15:42:52.723991+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_clean_scrub_stamp":"2024-09-16T15:42:50.361965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T18:05:24.521404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T15:42:52.874446+0000","last_change":"2024-09-16T15:42:51.428815+0000","last_active":"2024-09-16T15:42:52.874446+0000","last_peered":"2024-09-16T15:42:52.874446+0000","last_clean":"2024-09-16T15:42:52.874446+0000","last_became_active":"2024-09-16T15:42:51.427658+0000","last_became_peered":"2024-09-16T15:42:51.427658+0000","last_unstale":"2024-09-16T15:42:52.874446+0000","last_undegraded":"2024-09-16T15:42:52.874446+0000","last_fullsized":"2024-09-16T15:42:52.874446+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_clean_scrub_stamp":"2024-09-16T15:42:50.361965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T03:23:33.232936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T15:42:52.725119+0000","last_change":"2024-09-16T15:42:51.435164+0000","last_active":"2024-09-16T15:42:52.725119+0000","last_peered":"2024-09-16T15:42:52.725119+0000","last_clean":"2024-09-16T15:42:52.725119+0000","last_became_active":"2024-09-16T15:42:51.434702+0000","last_became_peered":"2024-09-16T15:42:51.434702+0000","last_unstale":"2024-09-16T15:42:52.725119+0000","last_undegraded":"2024-09-16T15:42:52.725119+0000","last_fullsized":"2024-09-16T15:42:52.725119+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_clean_scrub_stamp":"2024-09-16T15:42:50.361965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:40:28.779257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T15:42:52.417480+0000","last_change":"2024-09-16T15:42:51.425817+0000","last_active":"2024-09-16T15:42:52.417480+0000","last_peered":"2024-09-16T15:42:52.417480+0000","last_clean":"2024-09-16T15:42:52.417480+0000","last_became_active":"2024-09-16T15:42:51.425279+0000","last_became_peered":"2024-09-16T15:42:51.425279+0000","last_unstale":"2024-09-16T15:42:52.417480+0000","last_undegraded":"2024-09-16T15:42:52.417480+0000","last_fullsized":"2024-09-16T15:42:52.417480+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_clean_scrub_stamp":"2024-09-16T15:42:50.361965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:10:07.925485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T15:42:52.416844+0000","last_change":"2024-09-16T15:42:51.426883+0000","last_active":"2024-09-16T15:42:52.416844+0000","last_peered":"2024-09-16T15:42:52.416844+0000","last_clean":"2024-09-16T15:42:52.416844+0000","last_became_active":"2024-09-16T15:42:51.426556+0000","last_became_peered":"2024-09-16T15:42:51.426556+0000","last_unstale":"2024-09-16T15:42:52.416844+0000","last_undegraded":"2024-09-16T15:42:52.416844+0000","last_fullsized":"2024-09-16T15:42:52.416844+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_clean_scrub_stamp":"2024-09-16T15:42:50.361965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:55:46.525397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T15:42:52.418805+0000","last_change":"2024-09-16T15:42:51.394761+0000","last_active":"2024-09-16T15:42:52.418805+0000","last_peered":"2024-09-16T15:42:52.418805+0000","last_clean":"2024-09-16T15:42:52.418805+0000","last_became_active":"2024-09-16T15:42:51.394005+0000","last_became_peered":"2024-09-16T15:42:51.394005+0000","last_unstale":"2024-09-16T15:42:52.418805+0000","last_undegraded":"2024-09-16T15:42:52.418805+0000","last_fullsized":"2024-09-16T15:42:52.418805+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_clean_scrub_stamp":"2024-09-16T15:42:50.361965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:17:36.442767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T15:42:52.875964+0000","last_change":"2024-09-16T15:42:51.411614+0000","last_active":"2024-09-16T15:42:52.875964+0000","last_peered":"2024-09-16T15:42:52.875964+0000","last_clean":"2024-09-16T15:42:52.875964+0000","last_became_active":"2024-09-16T15:42:51.409702+0000","last_became_peered":"2024-09-16T15:42:51.409702+0000","last_unstale":"2024-09-16T15:42:52.875964+0000","last_undegraded":"2024-09-16T15:42:52.875964+0000","last_fullsized":"2024-09-16T15:42:52.875964+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_clean_scrub_stamp":"2024-09-16T15:42:50.361965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T00:31:04.101900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T15:42:52.874545+0000","last_change":"2024-09-16T15:42:51.403206+0000","last_active":"2024-09-16T15:42:52.874545+0000","last_peered":"2024-09-16T15:42:52.874545+0000","last_clean":"2024-09-16T15:42:52.874545+0000","last_became_active":"2024-09-16T15:42:51.402528+0000","last_became_peered":"2024-09-16T15:42:51.402528+0000","last_unstale":"2024-09-16T15:42:52.874545+0000","last_undegraded":"2024-09-16T15:42:52.874545+0000","last_fullsized":"2024-09-16T15:42:52.874545+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_clean_scrub_stamp":"2024-09-16T15:42:50.361965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:39:05.860718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T15:42:52.418038+0000","last_change":"2024-09-16T15:42:51.411905+0000","last_active":"2024-09-16T15:42:52.418038+0000","last_peered":"2024-09-16T15:42:52.418038+0000","last_clean":"2024-09-16T15:42:52.418038+0000","last_became_active":"2024-09-16T15:42:51.411540+0000","last_became_peered":"2024-09-16T15:42:51.411540+0000","last_unstale":"2024-09-16T15:42:52.418038+0000","last_undegraded":"2024-09-16T15:42:52.418038+0000","last_fullsized":"2024-09-16T15:42:52.418038+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_clean_scrub_stamp":"2024-09-16T15:42:50.361965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:07:17.519138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T15:42:52.638310+0000","last_change":"2024-09-16T15:42:51.419623+0000","last_active":"2024-09-16T15:42:52.638310+0000","last_peered":"2024-09-16T15:42:52.638310+0000","last_clean":"2024-09-16T15:42:52.638310+0000","last_became_active":"2024-09-16T15:42:51.419319+0000","last_became_peered":"2024-09-16T15:42:51.419319+0000","last_unstale":"2024-09-16T15:42:52.638310+0000","last_undegraded":"2024-09-16T15:42:52.638310+0000","last_fullsized":"2024-09-16T15:42:52.638310+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_clean_scrub_stamp":"2024-09-16T15:42:50.361965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T02:33:57.643582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T15:42:52.640012+0000","last_change":"2024-09-16T15:42:51.436411+0000","last_active":"2024-09-16T15:42:52.640012+0000","last_peered":"2024-09-16T15:42:52.640012+0000","last_clean":"2024-09-16T15:42:52.640012+0000","last_became_active":"2024-09-16T15:42:51.435952+0000","last_became_peered":"2024-09-16T15:42:51.435952+0000","last_unstale":"2024-09-16T15:42:52.640012+0000","last_undegraded":"2024-09-16T15:42:52.640012+0000","last_fullsized":"2024-09-16T15:42:52.640012+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_clean_scrub_stamp":"2024-09-16T15:42:50.361965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:08:57.795460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T15:42:52.738414+0000","last_change":"2024-09-16T15:42:51.404567+0000","last_active":"2024-09-16T15:42:52.738414+0000","last_peered":"2024-09-16T15:42:52.738414+0000","last_clean":"2024-09-16T15:42:52.738414+0000","last_became_active":"2024-09-16T15:42:51.403756+0000","last_became_peered":"2024-09-16T15:42:51.403756+0000","last_unstale":"2024-09-16T15:42:52.738414+0000","last_undegraded":"2024-09-16T15:42:52.738414+0000","last_fullsized":"2024-09-16T15:42:52.738414+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_clean_scrub_stamp":"2024-09-16T15:42:50.361965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T01:04:06.975937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T15:42:52.725148+0000","last_change":"2024-09-16T15:42:51.434196+0000","last_active":"2024-09-16T15:42:52.725148+0000","last_peered":"2024-09-16T15:42:52.725148+0000","last_clean":"2024-09-16T15:42:52.725148+0000","last_became_active":"2024-09-16T15:42:51.433748+0000","last_became_peered":"2024-09-16T15:42:51.433748+0000","last_unstale":"2024-09-16T15:42:52.725148+0000","last_undegraded":"2024-09-16T15:42:52.725148+0000","last_fullsized":"2024-09-16T15:42:52.725148+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_clean_scrub_stamp":"2024-09-16T15:42:50.361965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:08:57.795460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T15:42:52.416804+0000","last_change":"2024-09-16T15:42:51.398201+0000","last_active":"2024-09-16T15:42:52.416804+0000","last_peered":"2024-09-16T15:42:52.416804+0000","last_clean":"2024-09-16T15:42:52.416804+0000","last_became_active":"2024-09-16T15:42:51.396092+0000","last_became_peered":"2024-09-16T15:42:51.396092+0000","last_unstale":"2024-09-16T15:42:52.416804+0000","last_undegraded":"2024-09-16T15:42:52.416804+0000","last_fullsized":"2024-09-16T15:42:52.416804+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_clean_scrub_stamp":"2024-09-16T15:42:50.361965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:16:26.836026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T15:42:52.874601+0000","last_change":"2024-09-16T15:42:51.404937+0000","last_active":"2024-09-16T15:42:52.874601+0000","last_peered":"2024-09-16T15:42:52.874601+0000","last_clean":"2024-09-16T15:42:52.874601+0000","last_became_active":"2024-09-16T15:42:51.404498+0000","last_became_peered":"2024-09-16T15:42:51.404498+0000","last_unstale":"2024-09-16T15:42:52.874601+0000","last_undegraded":"2024-09-16T15:42:52.874601+0000","last_fullsized":"2024-09-16T15:42:52.874601+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_clean_scrub_stamp":"2024-09-16T15:42:50.361965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:16:06.557859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T15:42:52.638770+0000","last_change":"2024-09-16T15:42:51.436767+0000","last_active":"2024-09-16T15:42:52.638770+0000","last_peered":"2024-09-16T15:42:52.638770+0000","last_clean":"2024-09-16T15:42:52.638770+0000","last_became_active":"2024-09-16T15:42:51.436076+0000","last_became_peered":"2024-09-16T15:42:51.436076+0000","last_unstale":"2024-09-16T15:42:52.638770+0000","last_undegraded":"2024-09-16T15:42:52.638770+0000","last_fullsized":"2024-09-16T15:42:52.638770+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_clean_scrub_stamp":"2024-09-16T15:42:50.361965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T02:42:34.108192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T15:42:52.876650+0000","last_change":"2024-09-16T15:42:51.407561+0000","last_active":"2024-09-16T15:42:52.876650+0000","last_peered":"2024-09-16T15:42:52.876650+0000","last_clean":"2024-09-16T15:42:52.876650+0000","last_became_active":"2024-09-16T15:42:51.407242+0000","last_became_peered":"2024-09-16T15:42:51.407242+0000","last_unstale":"2024-09-16T15:42:52.876650+0000","last_undegraded":"2024-09-16T15:42:52.876650+0000","last_fullsized":"2024-09-16T15:42:52.876650+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_clean_scrub_stamp":"2024-09-16T15:42:50.361965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:37:26.430063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T15:42:52.638795+0000","last_change":"2024-09-16T15:42:51.420155+0000","last_active":"2024-09-16T15:42:52.638795+0000","last_peered":"2024-09-16T15:42:52.638795+0000","last_clean":"2024-09-16T15:42:52.638795+0000","last_became_active":"2024-09-16T15:42:51.419779+0000","last_became_peered":"2024-09-16T15:42:51.419779+0000","last_unstale":"2024-09-16T15:42:52.638795+0000","last_undegraded":"2024-09-16T15:42:52.638795+0000","last_fullsized":"2024-09-16T15:42:52.638795+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_clean_scrub_stamp":"2024-09-16T15:42:50.361965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:28:43.699424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T15:42:52.875086+0000","last_change":"2024-09-16T15:42:51.412701+0000","last_active":"2024-09-16T15:42:52.875086+0000","last_peered":"2024-09-16T15:42:52.875086+0000","last_clean":"2024-09-16T15:42:52.875086+0000","last_became_active":"2024-09-16T15:42:51.411770+0000","last_became_peered":"2024-09-16T15:42:51.411770+0000","last_unstale":"2024-09-16T15:42:52.875086+0000","last_undegraded":"2024-09-16T15:42:52.875086+0000","last_fullsized":"2024-09-16T15:42:52.875086+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_clean_scrub_stamp":"2024-09-16T15:42:50.361965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T02:37:19.881471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T15:42:52.417630+0000","last_change":"2024-09-16T15:42:51.423741+0000","last_active":"2024-09-16T15:42:52.417630+0000","last_peered":"2024-09-16T15:42:52.417630+0000","last_clean":"2024-09-16T15:42:52.417630+0000","last_became_active":"2024-09-16T15:42:51.423444+0000","last_became_peered":"2024-09-16T15:42:51.423444+0000","last_unstale":"2024-09-16T15:42:52.417630+0000","last_undegraded":"2024-09-16T15:42:52.417630+0000","last_fullsized":"2024-09-16T15:42:52.417630+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_clean_scrub_stamp":"2024-09-16T15:42:50.361965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T18:56:18.021098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T15:42:52.417698+0000","last_change":"2024-09-16T15:42:51.410257+0000","last_active":"2024-09-16T15:42:52.417698+0000","last_peered":"2024-09-16T15:42:52.417698+0000","last_clean":"2024-09-16T15:42:52.417698+0000","last_became_active":"2024-09-16T15:42:51.409838+0000","last_became_peered":"2024-09-16T15:42:51.409838+0000","last_unstale":"2024-09-16T15:42:52.417698+0000","last_undegraded":"2024-09-16T15:42:52.417698+0000","last_fullsized":"2024-09-16T15:42:52.417698+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_clean_scrub_stamp":"2024-09-16T15:42:50.361965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T01:10:38.539716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T15:42:52.875794+0000","last_change":"2024-09-16T15:42:51.414267+0000","last_active":"2024-09-16T15:42:52.875794+0000","last_peered":"2024-09-16T15:42:52.875794+0000","last_clean":"2024-09-16T15:42:52.875794+0000","last_became_active":"2024-09-16T15:42:51.412201+0000","last_became_peered":"2024-09-16T15:42:51.412201+0000","last_unstale":"2024-09-16T15:42:52.875794+0000","last_undegraded":"2024-09-16T15:42:52.875794+0000","last_fullsized":"2024-09-16T15:42:52.875794+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_clean_scrub_stamp":"2024-09-16T15:42:50.361965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:59:06.080508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T15:42:52.875146+0000","last_change":"2024-09-16T15:42:51.412344+0000","last_active":"2024-09-16T15:42:52.875146+0000","last_peered":"2024-09-16T15:42:52.875146+0000","last_clean":"2024-09-16T15:42:52.875146+0000","last_became_active":"2024-09-16T15:42:51.410933+0000","last_became_peered":"2024-09-16T15:42:51.410933+0000","last_unstale":"2024-09-16T15:42:52.875146+0000","last_undegraded":"2024-09-16T15:42:52.875146+0000","last_fullsized":"2024-09-16T15:42:52.875146+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_clean_scrub_stamp":"2024-09-16T15:42:50.361965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T01:38:29.275555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T15:42:52.639286+0000","last_change":"2024-09-16T15:42:51.419645+0000","last_active":"2024-09-16T15:42:52.639286+0000","last_peered":"2024-09-16T15:42:52.639286+0000","last_clean":"2024-09-16T15:42:52.639286+0000","last_became_active":"2024-09-16T15:42:51.419335+0000","last_became_peered":"2024-09-16T15:42:51.419335+0000","last_unstale":"2024-09-16T15:42:52.639286+0000","last_undegraded":"2024-09-16T15:42:52.639286+0000","last_fullsized":"2024-09-16T15:42:52.639286+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_clean_scrub_stamp":"2024-09-16T15:42:50.361965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T02:58:03.717735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T15:42:52.738151+0000","last_change":"2024-09-16T15:42:51.425478+0000","last_active":"2024-09-16T15:42:52.738151+0000","last_peered":"2024-09-16T15:42:52.738151+0000","last_clean":"2024-09-16T15:42:52.738151+0000","last_became_active":"2024-09-16T15:42:51.425150+0000","last_became_peered":"2024-09-16T15:42:51.425150+0000","last_unstale":"2024-09-16T15:42:52.738151+0000","last_undegraded":"2024-09-16T15:42:52.738151+0000","last_fullsized":"2024-09-16T15:42:52.738151+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_clean_scrub_stamp":"2024-09-16T15:42:50.361965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T00:53:53.123181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T15:42:52.416202+0000","last_change":"2024-09-16T15:42:51.395997+0000","last_active":"2024-09-16T15:42:52.416202+0000","last_peered":"2024-09-16T15:42:52.416202+0000","last_clean":"2024-09-16T15:42:52.416202+0000","last_became_active":"2024-09-16T15:42:51.395182+0000","last_became_peered":"2024-09-16T15:42:51.395182+0000","last_unstale":"2024-09-16T15:42:52.416202+0000","last_undegraded":"2024-09-16T15:42:52.416202+0000","last_fullsized":"2024-09-16T15:42:52.416202+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_clean_scrub_stamp":"2024-09-16T15:42:50.361965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:22:21.461659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T15:42:52.418393+0000","last_change":"2024-09-16T15:42:51.398579+0000","last_active":"2024-09-16T15:42:52.418393+0000","last_peered":"2024-09-16T15:42:52.418393+0000","last_clean":"2024-09-16T15:42:52.418393+0000","last_became_active":"2024-09-16T15:42:51.397865+0000","last_became_peered":"2024-09-16T15:42:51.397865+0000","last_unstale":"2024-09-16T15:42:52.418393+0000","last_undegraded":"2024-09-16T15:42:52.418393+0000","last_fullsized":"2024-09-16T15:42:52.418393+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_clean_scrub_stamp":"2024-09-16T15:42:50.361965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:44:41.707611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"20'1","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T15:42:52.876772+0000","last_change":"2024-09-16T15:42:50.407972+0000","last_active":"2024-09-16T15:42:52.876772+0000","last_peered":"2024-09-16T15:42:52.876772+0000","last_clean":"2024-09-16T15:42:52.876772+0000","last_became_active":"2024-09-16T15:42:50.407571+0000","last_became_peered":"2024-09-16T15:42:50.407571+0000","last_unstale":"2024-09-16T15:42:52.876772+0000","last_undegraded":"2024-09-16T15:42:52.876772+0000","last_fullsized":"2024-09-16T15:42:52.876772+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:42:49.352371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:42:49.352371+0000","last_clean_scrub_stamp":"2024-09-16T15:42:49.352371+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T02:32:51.461439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T15:42:52.876489+0000","last_change":"2024-09-16T15:42:51.403656+0000","last_active":"2024-09-16T15:42:52.876489+0000","last_peered":"2024-09-16T15:42:52.876489+0000","last_clean":"2024-09-16T15:42:52.876489+0000","last_became_active":"2024-09-16T15:42:51.403183+0000","last_became_peered":"2024-09-16T15:42:51.403183+0000","last_unstale":"2024-09-16T15:42:52.876489+0000","last_undegraded":"2024-09-16T15:42:52.876489+0000","last_fullsized":"2024-09-16T15:42:52.876489+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_clean_scrub_stamp":"2024-09-16T15:42:50.361965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T01:01:27.144919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"20'1","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T15:42:52.415759+0000","last_change":"2024-09-16T15:42:50.405255+0000","last_active":"2024-09-16T15:42:52.415759+0000","last_peered":"2024-09-16T15:42:52.415759+0000","last_clean":"2024-09-16T15:42:52.415759+0000","last_became_active":"2024-09-16T15:42:50.404469+0000","last_became_peered":"2024-09-16T15:42:50.404469+0000","last_unstale":"2024-09-16T15:42:52.415759+0000","last_undegraded":"2024-09-16T15:42:52.415759+0000","last_fullsized":"2024-09-16T15:42:52.415759+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:42:49.352371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:42:49.352371+0000","last_clean_scrub_stamp":"2024-09-16T15:42:49.352371+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:41:46.870937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T15:42:52.877073+0000","last_change":"2024-09-16T15:42:50.399198+0000","last_active":"2024-09-16T15:42:52.877073+0000","last_peered":"2024-09-16T15:42:52.877073+0000","last_clean":"2024-09-16T15:42:52.877073+0000","last_became_active":"2024-09-16T15:42:50.398892+0000","last_became_peered":"2024-09-16T15:42:50.398892+0000","last_unstale":"2024-09-16T15:42:52.877073+0000","last_undegraded":"2024-09-16T15:42:52.877073+0000","last_fullsized":"2024-09-16T15:42:52.877073+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:42:49.352371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:42:49.352371+0000","last_clean_scrub_stamp":"2024-09-16T15:42:49.352371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T03:32:21.032967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T15:42:52.875003+0000","last_change":"2024-09-16T15:42:51.412892+0000","last_active":"2024-09-16T15:42:52.875003+0000","last_peered":"2024-09-16T15:42:52.875003+0000","last_clean":"2024-09-16T15:42:52.875003+0000","last_became_active":"2024-09-16T15:42:51.407080+0000","last_became_peered":"2024-09-16T15:42:51.407080+0000","last_unstale":"2024-09-16T15:42:52.875003+0000","last_undegraded":"2024-09-16T15:42:52.875003+0000","last_fullsized":"2024-09-16T15:42:52.875003+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_clean_scrub_stamp":"2024-09-16T15:42:50.361965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T02:17:29.317460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T15:42:52.638690+0000","last_change":"2024-09-16T15:42:50.427196+0000","last_active":"2024-09-16T15:42:52.638690+0000","last_peered":"2024-09-16T15:42:52.638690+0000","last_clean":"2024-09-16T15:42:52.638690+0000","last_became_active":"2024-09-16T15:42:50.426397+0000","last_became_peered":"2024-09-16T15:42:50.426397+0000","last_unstale":"2024-09-16T15:42:52.638690+0000","last_undegraded":"2024-09-16T15:42:52.638690+0000","last_fullsized":"2024-09-16T15:42:52.638690+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:42:49.352371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:42:49.352371+0000","last_clean_scrub_stamp":"2024-09-16T15:42:49.352371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T22:48:39.726611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T15:42:52.417585+0000","last_change":"2024-09-16T15:42:51.410683+0000","last_active":"2024-09-16T15:42:52.417585+0000","last_peered":"2024-09-16T15:42:52.417585+0000","last_clean":"2024-09-16T15:42:52.417585+0000","last_became_active":"2024-09-16T15:42:51.409695+0000","last_became_peered":"2024-09-16T15:42:51.409695+0000","last_unstale":"2024-09-16T15:42:52.417585+0000","last_undegraded":"2024-09-16T15:42:52.417585+0000","last_fullsized":"2024-09-16T15:42:52.417585+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_clean_scrub_stamp":"2024-09-16T15:42:50.361965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T01:38:29.275555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T15:42:52.417220+0000","last_change":"2024-09-16T15:42:50.420642+0000","last_active":"2024-09-16T15:42:52.417220+0000","last_peered":"2024-09-16T15:42:52.417220+0000","last_clean":"2024-09-16T15:42:52.417220+0000","last_became_active":"2024-09-16T15:42:50.419895+0000","last_became_peered":"2024-09-16T15:42:50.419895+0000","last_unstale":"2024-09-16T15:42:52.417220+0000","last_undegraded":"2024-09-16T15:42:52.417220+0000","last_fullsized":"2024-09-16T15:42:52.417220+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:42:49.352371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:42:49.352371+0000","last_clean_scrub_stamp":"2024-09-16T15:42:49.352371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:40:01.292941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T15:42:52.639747+0000","last_change":"2024-09-16T15:42:51.434841+0000","last_active":"2024-09-16T15:42:52.639747+0000","last_peered":"2024-09-16T15:42:52.639747+0000","last_clean":"2024-09-16T15:42:52.639747+0000","last_became_active":"2024-09-16T15:42:51.434532+0000","last_became_peered":"2024-09-16T15:42:51.434532+0000","last_unstale":"2024-09-16T15:42:52.639747+0000","last_undegraded":"2024-09-16T15:42:52.639747+0000","last_fullsized":"2024-09-16T15:42:52.639747+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_clean_scrub_stamp":"2024-09-16T15:42:50.361965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T18:44:49.865687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T15:42:52.639491+0000","last_change":"2024-09-16T15:42:50.419196+0000","last_active":"2024-09-16T15:42:52.639491+0000","last_peered":"2024-09-16T15:42:52.639491+0000","last_clean":"2024-09-16T15:42:52.639491+0000","last_became_active":"2024-09-16T15:42:50.415873+0000","last_became_peered":"2024-09-16T15:42:50.415873+0000","last_unstale":"2024-09-16T15:42:52.639491+0000","last_undegraded":"2024-09-16T15:42:52.639491+0000","last_fullsized":"2024-09-16T15:42:52.639491+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:42:49.352371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:42:49.352371+0000","last_clean_scrub_stamp":"2024-09-16T15:42:49.352371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:26:07.798125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T15:42:52.877644+0000","last_change":"2024-09-16T15:42:51.425682+0000","last_active":"2024-09-16T15:42:52.877644+0000","last_peered":"2024-09-16T15:42:52.877644+0000","last_clean":"2024-09-16T15:42:52.877644+0000","last_became_active":"2024-09-16T15:42:51.425336+0000","last_became_peered":"2024-09-16T15:42:51.425336+0000","last_unstale":"2024-09-16T15:42:52.877644+0000","last_undegraded":"2024-09-16T15:42:52.877644+0000","last_fullsized":"2024-09-16T15:42:52.877644+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_clean_scrub_stamp":"2024-09-16T15:42:50.361965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T02:33:57.643582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"20'3","reported_seq":18,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T15:42:52.418373+0000","last_change":"2024-09-16T15:42:50.415090+0000","last_active":"2024-09-16T15:42:52.418373+0000","last_peered":"2024-09-16T15:42:52.418373+0000","last_clean":"2024-09-16T15:42:52.418373+0000","last_became_active":"2024-09-16T15:42:50.414818+0000","last_became_peered":"2024-09-16T15:42:50.414818+0000","last_unstale":"2024-09-16T15:42:52.418373+0000","last_undegraded":"2024-09-16T15:42:52.418373+0000","last_fullsized":"2024-09-16T15:42:52.418373+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:42:49.352371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:42:49.352371+0000","last_clean_scrub_stamp":"2024-09-16T15:42:49.352371+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:36:36.506348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T15:42:52.419154+0000","last_change":"2024-09-16T15:42:51.425357+0000","last_active":"2024-09-16T15:42:52.419154+0000","last_peered":"2024-09-16T15:42:52.419154+0000","last_clean":"2024-09-16T15:42:52.419154+0000","last_became_active":"2024-09-16T15:42:51.425074+0000","last_became_peered":"2024-09-16T15:42:51.425074+0000","last_unstale":"2024-09-16T15:42:52.419154+0000","last_undegraded":"2024-09-16T15:42:52.419154+0000","last_fullsized":"2024-09-16T15:42:52.419154+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_clean_scrub_stamp":"2024-09-16T15:42:50.361965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:59:03.380666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"20'6","reported_seq":21,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T15:42:52.416548+0000","last_change":"2024-09-16T15:42:50.396749+0000","last_active":"2024-09-16T15:42:52.416548+0000","last_peered":"2024-09-16T15:42:52.416548+0000","last_clean":"2024-09-16T15:42:52.416548+0000","last_became_active":"2024-09-16T15:42:50.395963+0000","last_became_peered":"2024-09-16T15:42:50.395963+0000","last_unstale":"2024-09-16T15:42:52.416548+0000","last_undegraded":"2024-09-16T15:42:52.416548+0000","last_fullsized":"2024-09-16T15:42:52.416548+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:42:49.352371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:42:49.352371+0000","last_clean_scrub_stamp":"2024-09-16T15:42:49.352371+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T23:01:15.996297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T15:42:52.724385+0000","last_change":"2024-09-16T15:42:51.427006+0000","last_active":"2024-09-16T15:42:52.724385+0000","last_peered":"2024-09-16T15:42:52.724385+0000","last_clean":"2024-09-16T15:42:52.724385+0000","last_became_active":"2024-09-16T15:42:51.426730+0000","last_became_peered":"2024-09-16T15:42:51.426730+0000","last_unstale":"2024-09-16T15:42:52.724385+0000","last_undegraded":"2024-09-16T15:42:52.724385+0000","last_fullsized":"2024-09-16T15:42:52.724385+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_clean_scrub_stamp":"2024-09-16T15:42:50.361965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:49:13.477881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"20'2","reported_seq":17,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T15:42:52.876798+0000","last_change":"2024-09-16T15:42:50.401803+0000","last_active":"2024-09-16T15:42:52.876798+0000","last_peered":"2024-09-16T15:42:52.876798+0000","last_clean":"2024-09-16T15:42:52.876798+0000","last_became_active":"2024-09-16T15:42:50.401495+0000","last_became_peered":"2024-09-16T15:42:50.401495+0000","last_unstale":"2024-09-16T15:42:52.876798+0000","last_undegraded":"2024-09-16T15:42:52.876798+0000","last_fullsized":"2024-09-16T15:42:52.876798+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:42:49.352371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:42:49.352371+0000","last_clean_scrub_stamp":"2024-09-16T15:42:49.352371+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T18:06:23.054055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T15:42:52.875115+0000","last_change":"2024-09-16T15:42:51.428923+0000","last_active":"2024-09-16T15:42:52.875115+0000","last_peered":"2024-09-16T15:42:52.875115+0000","last_clean":"2024-09-16T15:42:52.875115+0000","last_became_active":"2024-09-16T15:42:51.428550+0000","last_became_peered":"2024-09-16T15:42:51.428550+0000","last_unstale":"2024-09-16T15:42:52.875115+0000","last_undegraded":"2024-09-16T15:42:52.875115+0000","last_fullsized":"2024-09-16T15:42:52.875115+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_clean_scrub_stamp":"2024-09-16T15:42:50.361965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:57:03.171257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T15:42:52.639097+0000","last_change":"2024-09-16T15:42:50.426919+0000","last_active":"2024-09-16T15:42:52.639097+0000","last_peered":"2024-09-16T15:42:52.639097+0000","last_clean":"2024-09-16T15:42:52.639097+0000","last_became_active":"2024-09-16T15:42:50.426141+0000","last_became_peered":"2024-09-16T15:42:50.426141+0000","last_unstale":"2024-09-16T15:42:52.639097+0000","last_undegraded":"2024-09-16T15:42:52.639097+0000","last_fullsized":"2024-09-16T15:42:52.639097+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:42:49.352371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:42:49.352371+0000","last_clean_scrub_stamp":"2024-09-16T15:42:49.352371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:23:09.046920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T15:42:52.724923+0000","last_change":"2024-09-16T15:42:51.425867+0000","last_active":"2024-09-16T15:42:52.724923+0000","last_peered":"2024-09-16T15:42:52.724923+0000","last_clean":"2024-09-16T15:42:52.724923+0000","last_became_active":"2024-09-16T15:42:51.425313+0000","last_became_peered":"2024-09-16T15:42:51.425313+0000","last_unstale":"2024-09-16T15:42:52.724923+0000","last_undegraded":"2024-09-16T15:42:52.724923+0000","last_fullsized":"2024-09-16T15:42:52.724923+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_clean_scrub_stamp":"2024-09-16T15:42:50.361965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T18:02:05.435198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T15:42:52.415548+0000","last_change":"2024-09-16T15:42:50.405263+0000","last_active":"2024-09-16T15:42:52.415548+0000","last_peered":"2024-09-16T15:42:52.415548+0000","last_clean":"2024-09-16T15:42:52.415548+0000","last_became_active":"2024-09-16T15:42:50.404419+0000","last_became_peered":"2024-09-16T15:42:50.404419+0000","last_unstale":"2024-09-16T15:42:52.415548+0000","last_undegraded":"2024-09-16T15:42:52.415548+0000","last_fullsized":"2024-09-16T15:42:52.415548+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:42:49.352371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:42:49.352371+0000","last_clean_scrub_stamp":"2024-09-16T15:42:49.352371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:40:33.707149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T15:42:52.415669+0000","last_change":"2024-09-16T15:42:51.397798+0000","last_active":"2024-09-16T15:42:52.415669+0000","last_peered":"2024-09-16T15:42:52.415669+0000","last_clean":"2024-09-16T15:42:52.415669+0000","last_became_active":"2024-09-16T15:42:51.397070+0000","last_became_peered":"2024-09-16T15:42:51.397070+0000","last_unstale":"2024-09-16T15:42:52.415669+0000","last_undegraded":"2024-09-16T15:42:52.415669+0000","last_fullsized":"2024-09-16T15:42:52.415669+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_clean_scrub_stamp":"2024-09-16T15:42:50.361965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:24:48.779934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"20'8","reported_seq":23,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T15:42:52.737837+0000","last_change":"2024-09-16T15:42:50.386440+0000","last_active":"2024-09-16T15:42:52.737837+0000","last_peered":"2024-09-16T15:42:52.737837+0000","last_clean":"2024-09-16T15:42:52.737837+0000","last_became_active":"2024-09-16T15:42:50.386100+0000","last_became_peered":"2024-09-16T15:42:50.386100+0000","last_unstale":"2024-09-16T15:42:52.737837+0000","last_undegraded":"2024-09-16T15:42:52.737837+0000","last_fullsized":"2024-09-16T15:42:52.737837+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:42:49.352371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:42:49.352371+0000","last_clean_scrub_stamp":"2024-09-16T15:42:49.352371+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:32:51.019126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T15:42:52.723952+0000","last_change":"2024-09-16T15:42:51.424679+0000","last_active":"2024-09-16T15:42:52.723952+0000","last_peered":"2024-09-16T15:42:52.723952+0000","last_clean":"2024-09-16T15:42:52.723952+0000","last_became_active":"2024-09-16T15:42:51.424119+0000","last_became_peered":"2024-09-16T15:42:51.424119+0000","last_unstale":"2024-09-16T15:42:52.723952+0000","last_undegraded":"2024-09-16T15:42:52.723952+0000","last_fullsized":"2024-09-16T15:42:52.723952+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_clean_scrub_stamp":"2024-09-16T15:42:50.361965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T23:53:36.869388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"20'1","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T15:42:52.419009+0000","last_change":"2024-09-16T15:42:50.395317+0000","last_active":"2024-09-16T15:42:52.419009+0000","last_peered":"2024-09-16T15:42:52.419009+0000","last_clean":"2024-09-16T15:42:52.419009+0000","last_became_active":"2024-09-16T15:42:50.394495+0000","last_became_peered":"2024-09-16T15:42:50.394495+0000","last_unstale":"2024-09-16T15:42:52.419009+0000","last_undegraded":"2024-09-16T15:42:52.419009+0000","last_fullsized":"2024-09-16T15:42:52.419009+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:42:49.352371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:42:49.352371+0000","last_clean_scrub_stamp":"2024-09-16T15:42:49.352371+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:45:54.681341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T15:42:52.724959+0000","last_change":"2024-09-16T15:42:51.423467+0000","last_active":"2024-09-16T15:42:52.724959+0000","last_peered":"2024-09-16T15:42:52.724959+0000","last_clean":"2024-09-16T15:42:52.724959+0000","last_became_active":"2024-09-16T15:42:51.423161+0000","last_became_peered":"2024-09-16T15:42:51.423161+0000","last_unstale":"2024-09-16T15:42:52.724959+0000","last_undegraded":"2024-09-16T15:42:52.724959+0000","last_fullsized":"2024-09-16T15:42:52.724959+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_clean_scrub_stamp":"2024-09-16T15:42:50.361965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T23:53:22.795898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"20'2","reported_seq":17,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T15:42:52.738233+0000","last_change":"2024-09-16T15:42:50.384867+0000","last_active":"2024-09-16T15:42:52.738233+0000","last_peered":"2024-09-16T15:42:52.738233+0000","last_clean":"2024-09-16T15:42:52.738233+0000","last_became_active":"2024-09-16T15:42:50.384581+0000","last_became_peered":"2024-09-16T15:42:50.384581+0000","last_unstale":"2024-09-16T15:42:52.738233+0000","last_undegraded":"2024-09-16T15:42:52.738233+0000","last_fullsized":"2024-09-16T15:42:52.738233+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:42:49.352371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:42:49.352371+0000","last_clean_scrub_stamp":"2024-09-16T15:42:49.352371+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T22:48:39.726611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T15:42:52.725417+0000","last_change":"2024-09-16T15:42:51.434874+0000","last_active":"2024-09-16T15:42:52.725417+0000","last_peered":"2024-09-16T15:42:52.725417+0000","last_clean":"2024-09-16T15:42:52.725417+0000","last_became_active":"2024-09-16T15:42:51.434561+0000","last_became_peered":"2024-09-16T15:42:51.434561+0000","last_unstale":"2024-09-16T15:42:52.725417+0000","last_undegraded":"2024-09-16T15:42:52.725417+0000","last_fullsized":"2024-09-16T15:42:52.725417+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_clean_scrub_stamp":"2024-09-16T15:42:50.361965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:13:10.293897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"20'1","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T15:42:52.418305+0000","last_change":"2024-09-16T15:42:50.399974+0000","last_active":"2024-09-16T15:42:52.418305+0000","last_peered":"2024-09-16T15:42:52.418305+0000","last_clean":"2024-09-16T15:42:52.418305+0000","last_became_active":"2024-09-16T15:42:50.399714+0000","last_became_peered":"2024-09-16T15:42:50.399714+0000","last_unstale":"2024-09-16T15:42:52.418305+0000","last_undegraded":"2024-09-16T15:42:52.418305+0000","last_fullsized":"2024-09-16T15:42:52.418305+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:42:49.352371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:42:49.352371+0000","last_clean_scrub_stamp":"2024-09-16T15:42:49.352371+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T02:37:24.494137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T15:42:52.639516+0000","last_change":"2024-09-16T15:42:51.420665+0000","last_active":"2024-09-16T15:42:52.639516+0000","last_peered":"2024-09-16T15:42:52.639516+0000","last_clean":"2024-09-16T15:42:52.639516+0000","last_became_active":"2024-09-16T15:42:51.420340+0000","last_became_peered":"2024-09-16T15:42:51.420340+0000","last_unstale":"2024-09-16T15:42:52.639516+0000","last_undegraded":"2024-09-16T15:42:52.639516+0000","last_fullsized":"2024-09-16T15:42:52.639516+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_clean_scrub_stamp":"2024-09-16T15:42:50.361965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T00:05:49.936465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T15:42:52.738027+0000","last_change":"2024-09-16T15:42:50.412850+0000","last_active":"2024-09-16T15:42:52.738027+0000","last_peered":"2024-09-16T15:42:52.738027+0000","last_clean":"2024-09-16T15:42:52.738027+0000","last_became_active":"2024-09-16T15:42:50.412492+0000","last_became_peered":"2024-09-16T15:42:50.412492+0000","last_unstale":"2024-09-16T15:42:52.738027+0000","last_undegraded":"2024-09-16T15:42:52.738027+0000","last_fullsized":"2024-09-16T15:42:52.738027+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:42:49.352371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:42:49.352371+0000","last_clean_scrub_stamp":"2024-09-16T15:42:49.352371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T22:41:47.420539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T15:42:52.724515+0000","last_change":"2024-09-16T15:42:51.428052+0000","last_active":"2024-09-16T15:42:52.724515+0000","last_peered":"2024-09-16T15:42:52.724515+0000","last_clean":"2024-09-16T15:42:52.724515+0000","last_became_active":"2024-09-16T15:42:51.427776+0000","last_became_peered":"2024-09-16T15:42:51.427776+0000","last_unstale":"2024-09-16T15:42:52.724515+0000","last_undegraded":"2024-09-16T15:42:52.724515+0000","last_fullsized":"2024-09-16T15:42:52.724515+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_clean_scrub_stamp":"2024-09-16T15:42:50.361965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T22:34:25.305861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"16'72","reported_seq":111,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T15:42:52.723702+0000","last_change":"2024-09-16T15:42:40.881820+0000","last_active":"2024-09-16T15:42:52.723702+0000","last_peered":"2024-09-16T15:42:52.723702+0000","last_clean":"2024-09-16T15:42:52.723702+0000","last_became_active":"2024-09-16T15:42:40.879965+0000","last_became_peered":"2024-09-16T15:42:40.879965+0000","last_unstale":"2024-09-16T15:42:52.723702+0000","last_undegraded":"2024-09-16T15:42:52.723702+0000","last_fullsized":"2024-09-16T15:42:52.723702+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:42:39.728762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:42:39.728762+0000","last_clean_scrub_stamp":"2024-09-16T15:42:39.728762+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:22:18.768932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T15:42:52.638416+0000","last_change":"2024-09-16T15:42:50.413082+0000","last_active":"2024-09-16T15:42:52.638416+0000","last_peered":"2024-09-16T15:42:52.638416+0000","last_clean":"2024-09-16T15:42:52.638416+0000","last_became_active":"2024-09-16T15:42:50.412050+0000","last_became_peered":"2024-09-16T15:42:50.412050+0000","last_unstale":"2024-09-16T15:42:52.638416+0000","last_undegraded":"2024-09-16T15:42:52.638416+0000","last_fullsized":"2024-09-16T15:42:52.638416+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:42:49.352371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:42:49.352371+0000","last_clean_scrub_stamp":"2024-09-16T15:42:49.352371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:32:51.019126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T15:42:52.875839+0000","last_change":"2024-09-16T15:42:51.430477+0000","last_active":"2024-09-16T15:42:52.875839+0000","last_peered":"2024-09-16T15:42:52.875839+0000","last_clean":"2024-09-16T15:42:52.875839+0000","last_became_active":"2024-09-16T15:42:51.430071+0000","last_became_peered":"2024-09-16T15:42:51.430071+0000","last_unstale":"2024-09-16T15:42:52.875839+0000","last_undegraded":"2024-09-16T15:42:52.875839+0000","last_fullsized":"2024-09-16T15:42:52.875839+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_clean_scrub_stamp":"2024-09-16T15:42:50.361965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T18:56:18.021098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T15:42:52.723623+0000","last_change":"2024-09-16T15:42:50.419981+0000","last_active":"2024-09-16T15:42:52.723623+0000","last_peered":"2024-09-16T15:42:52.723623+0000","last_clean":"2024-09-16T15:42:52.723623+0000","last_became_active":"2024-09-16T15:42:50.418405+0000","last_became_peered":"2024-09-16T15:42:50.418405+0000","last_unstale":"2024-09-16T15:42:52.723623+0000","last_undegraded":"2024-09-16T15:42:52.723623+0000","last_fullsized":"2024-09-16T15:42:52.723623+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:42:49.352371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:42:49.352371+0000","last_clean_scrub_stamp":"2024-09-16T15:42:49.352371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T00:53:52.113587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T15:42:52.874487+0000","last_change":"2024-09-16T15:42:51.403404+0000","last_active":"2024-09-16T15:42:52.874487+0000","last_peered":"2024-09-16T15:42:52.874487+0000","last_clean":"2024-09-16T15:42:52.874487+0000","last_became_active":"2024-09-16T15:42:51.402978+0000","last_became_peered":"2024-09-16T15:42:51.402978+0000","last_unstale":"2024-09-16T15:42:52.874487+0000","last_undegraded":"2024-09-16T15:42:52.874487+0000","last_fullsized":"2024-09-16T15:42:52.874487+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_clean_scrub_stamp":"2024-09-16T15:42:50.361965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T00:04:54.212949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T15:42:52.738033+0000","last_change":"2024-09-16T15:42:50.391465+0000","last_active":"2024-09-16T15:42:52.738033+0000","last_peered":"2024-09-16T15:42:52.738033+0000","last_clean":"2024-09-16T15:42:52.738033+0000","last_became_active":"2024-09-16T15:42:50.391091+0000","last_became_peered":"2024-09-16T15:42:50.391091+0000","last_unstale":"2024-09-16T15:42:52.738033+0000","last_undegraded":"2024-09-16T15:42:52.738033+0000","last_fullsized":"2024-09-16T15:42:52.738033+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:42:49.352371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:42:49.352371+0000","last_clean_scrub_stamp":"2024-09-16T15:42:49.352371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:26:07.798125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T15:42:52.417576+0000","last_change":"2024-09-16T15:42:51.404294+0000","last_active":"2024-09-16T15:42:52.417576+0000","last_peered":"2024-09-16T15:42:52.417576+0000","last_clean":"2024-09-16T15:42:52.417576+0000","last_became_active":"2024-09-16T15:42:51.403997+0000","last_became_peered":"2024-09-16T15:42:51.403997+0000","last_unstale":"2024-09-16T15:42:52.417576+0000","last_undegraded":"2024-09-16T15:42:52.417576+0000","last_fullsized":"2024-09-16T15:42:52.417576+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_clean_scrub_stamp":"2024-09-16T15:42:50.361965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:20:36.872940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T15:42:52.639350+0000","last_change":"2024-09-16T15:42:50.422498+0000","last_active":"2024-09-16T15:42:52.639350+0000","last_peered":"2024-09-16T15:42:52.639350+0000","last_clean":"2024-09-16T15:42:52.639350+0000","last_became_active":"2024-09-16T15:42:50.421727+0000","last_became_peered":"2024-09-16T15:42:50.421727+0000","last_unstale":"2024-09-16T15:42:52.639350+0000","last_undegraded":"2024-09-16T15:42:52.639350+0000","last_fullsized":"2024-09-16T15:42:52.639350+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:42:49.352371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:42:49.352371+0000","last_clean_scrub_stamp":"2024-09-16T15:42:49.352371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:36:36.506348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T15:42:52.419302+0000","last_change":"2024-09-16T15:42:51.413177+0000","last_active":"2024-09-16T15:42:52.419302+0000","last_peered":"2024-09-16T15:42:52.419302+0000","last_clean":"2024-09-16T15:42:52.419302+0000","last_became_active":"2024-09-16T15:42:51.412881+0000","last_became_peered":"2024-09-16T15:42:51.412881+0000","last_unstale":"2024-09-16T15:42:52.419302+0000","last_undegraded":"2024-09-16T15:42:52.419302+0000","last_fullsized":"2024-09-16T15:42:52.419302+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_clean_scrub_stamp":"2024-09-16T15:42:50.361965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:10:16.623518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T15:42:52.418085+0000","last_change":"2024-09-16T15:42:50.394813+0000","last_active":"2024-09-16T15:42:52.418085+0000","last_peered":"2024-09-16T15:42:52.418085+0000","last_clean":"2024-09-16T15:42:52.418085+0000","last_became_active":"2024-09-16T15:42:50.389679+0000","last_became_peered":"2024-09-16T15:42:50.389679+0000","last_unstale":"2024-09-16T15:42:52.418085+0000","last_undegraded":"2024-09-16T15:42:52.418085+0000","last_fullsized":"2024-09-16T15:42:52.418085+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:42:49.352371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:42:49.352371+0000","last_clean_scrub_stamp":"2024-09-16T15:42:49.352371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:23:09.046920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T15:42:52.639557+0000","last_change":"2024-09-16T15:42:51.424119+0000","last_active":"2024-09-16T15:42:52.639557+0000","last_peered":"2024-09-16T15:42:52.639557+0000","last_clean":"2024-09-16T15:42:52.639557+0000","last_became_active":"2024-09-16T15:42:51.423771+0000","last_became_peered":"2024-09-16T15:42:51.423771+0000","last_unstale":"2024-09-16T15:42:52.639557+0000","last_undegraded":"2024-09-16T15:42:52.639557+0000","last_fullsized":"2024-09-16T15:42:52.639557+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_clean_scrub_stamp":"2024-09-16T15:42:50.361965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T23:53:22.795898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T15:42:52.724991+0000","last_change":"2024-09-16T15:42:50.419065+0000","last_active":"2024-09-16T15:42:52.724991+0000","last_peered":"2024-09-16T15:42:52.724991+0000","last_clean":"2024-09-16T15:42:52.724991+0000","last_became_active":"2024-09-16T15:42:50.418438+0000","last_became_peered":"2024-09-16T15:42:50.418438+0000","last_unstale":"2024-09-16T15:42:52.724991+0000","last_undegraded":"2024-09-16T15:42:52.724991+0000","last_fullsized":"2024-09-16T15:42:52.724991+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:42:49.352371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:42:49.352371+0000","last_clean_scrub_stamp":"2024-09-16T15:42:49.352371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T01:42:29.977143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T15:42:52.418120+0000","last_change":"2024-09-16T15:42:51.407174+0000","last_active":"2024-09-16T15:42:52.418120+0000","last_peered":"2024-09-16T15:42:52.418120+0000","last_clean":"2024-09-16T15:42:52.418120+0000","last_became_active":"2024-09-16T15:42:51.406880+0000","last_became_peered":"2024-09-16T15:42:51.406880+0000","last_unstale":"2024-09-16T15:42:52.418120+0000","last_undegraded":"2024-09-16T15:42:52.418120+0000","last_fullsized":"2024-09-16T15:42:52.418120+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_clean_scrub_stamp":"2024-09-16T15:42:50.361965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T02:33:57.643582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"20'1","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T15:42:52.418471+0000","last_change":"2024-09-16T15:42:50.397599+0000","last_active":"2024-09-16T15:42:52.418471+0000","last_peered":"2024-09-16T15:42:52.418471+0000","last_clean":"2024-09-16T15:42:52.418471+0000","last_became_active":"2024-09-16T15:42:50.397301+0000","last_became_peered":"2024-09-16T15:42:50.397301+0000","last_unstale":"2024-09-16T15:42:52.418471+0000","last_undegraded":"2024-09-16T15:42:52.418471+0000","last_fullsized":"2024-09-16T15:42:52.418471+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:42:49.352371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:42:49.352371+0000","last_clean_scrub_stamp":"2024-09-16T15:42:49.352371+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T01:49:08.734586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T15:42:52.875834+0000","last_change":"2024-09-16T15:42:51.424110+0000","last_active":"2024-09-16T15:42:52.875834+0000","last_peered":"2024-09-16T15:42:52.875834+0000","last_clean":"2024-09-16T15:42:52.875834+0000","last_became_active":"2024-09-16T15:42:51.423798+0000","last_became_peered":"2024-09-16T15:42:51.423798+0000","last_unstale":"2024-09-16T15:42:52.875834+0000","last_undegraded":"2024-09-16T15:42:52.875834+0000","last_fullsized":"2024-09-16T15:42:52.875834+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_clean_scrub_stamp":"2024-09-16T15:42:50.361965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T22:53:48.581346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"20'2","reported_seq":17,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T15:42:52.417268+0000","last_change":"2024-09-16T15:42:50.416032+0000","last_active":"2024-09-16T15:42:52.417268+0000","last_peered":"2024-09-16T15:42:52.417268+0000","last_clean":"2024-09-16T15:42:52.417268+0000","last_became_active":"2024-09-16T15:42:50.415325+0000","last_became_peered":"2024-09-16T15:42:50.415325+0000","last_unstale":"2024-09-16T15:42:52.417268+0000","last_undegraded":"2024-09-16T15:42:52.417268+0000","last_fullsized":"2024-09-16T15:42:52.417268+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:42:49.352371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:42:49.352371+0000","last_clean_scrub_stamp":"2024-09-16T15:42:49.352371+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T02:11:26.332763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T15:42:52.877501+0000","last_change":"2024-09-16T15:42:51.407137+0000","last_active":"2024-09-16T15:42:52.877501+0000","last_peered":"2024-09-16T15:42:52.877501+0000","last_clean":"2024-09-16T15:42:52.877501+0000","last_became_active":"2024-09-16T15:42:51.406766+0000","last_became_peered":"2024-09-16T15:42:51.406766+0000","last_unstale":"2024-09-16T15:42:52.877501+0000","last_undegraded":"2024-09-16T15:42:52.877501+0000","last_fullsized":"2024-09-16T15:42:52.877501+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_clean_scrub_stamp":"2024-09-16T15:42:50.361965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:07:17.519138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T15:42:52.723539+0000","last_change":"2024-09-16T15:42:50.419106+0000","last_active":"2024-09-16T15:42:52.723539+0000","last_peered":"2024-09-16T15:42:52.723539+0000","last_clean":"2024-09-16T15:42:52.723539+0000","last_became_active":"2024-09-16T15:42:50.417462+0000","last_became_peered":"2024-09-16T15:42:50.417462+0000","last_unstale":"2024-09-16T15:42:52.723539+0000","last_undegraded":"2024-09-16T15:42:52.723539+0000","last_fullsized":"2024-09-16T15:42:52.723539+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:42:49.352371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:42:49.352371+0000","last_clean_scrub_stamp":"2024-09-16T15:42:49.352371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T22:58:24.474495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T15:42:52.418379+0000","last_change":"2024-09-16T15:42:51.412043+0000","last_active":"2024-09-16T15:42:52.418379+0000","last_peered":"2024-09-16T15:42:52.418379+0000","last_clean":"2024-09-16T15:42:52.418379+0000","last_became_active":"2024-09-16T15:42:51.411434+0000","last_became_peered":"2024-09-16T15:42:51.411434+0000","last_unstale":"2024-09-16T15:42:52.418379+0000","last_undegraded":"2024-09-16T15:42:52.418379+0000","last_fullsized":"2024-09-16T15:42:52.418379+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_clean_scrub_stamp":"2024-09-16T15:42:50.361965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:40:52.417913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T15:42:52.417895+0000","last_change":"2024-09-16T15:42:50.408565+0000","last_active":"2024-09-16T15:42:52.417895+0000","last_peered":"2024-09-16T15:42:52.417895+0000","last_clean":"2024-09-16T15:42:52.417895+0000","last_became_active":"2024-09-16T15:42:50.408254+0000","last_became_peered":"2024-09-16T15:42:50.408254+0000","last_unstale":"2024-09-16T15:42:52.417895+0000","last_undegraded":"2024-09-16T15:42:52.417895+0000","last_fullsized":"2024-09-16T15:42:52.417895+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:42:49.352371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:42:49.352371+0000","last_clean_scrub_stamp":"2024-09-16T15:42:49.352371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:26:07.798125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T15:42:52.876598+0000","last_change":"2024-09-16T15:42:51.413873+0000","last_active":"2024-09-16T15:42:52.876598+0000","last_peered":"2024-09-16T15:42:52.876598+0000","last_clean":"2024-09-16T15:42:52.876598+0000","last_became_active":"2024-09-16T15:42:51.406117+0000","last_became_peered":"2024-09-16T15:42:51.406117+0000","last_unstale":"2024-09-16T15:42:52.876598+0000","last_undegraded":"2024-09-16T15:42:52.876598+0000","last_fullsized":"2024-09-16T15:42:52.876598+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_clean_scrub_stamp":"2024-09-16T15:42:50.361965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:12:23.570707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T15:42:52.417696+0000","last_change":"2024-09-16T15:42:50.388233+0000","last_active":"2024-09-16T15:42:52.417696+0000","last_peered":"2024-09-16T15:42:52.417696+0000","last_clean":"2024-09-16T15:42:52.417696+0000","last_became_active":"2024-09-16T15:42:50.387920+0000","last_became_peered":"2024-09-16T15:42:50.387920+0000","last_unstale":"2024-09-16T15:42:52.417696+0000","last_undegraded":"2024-09-16T15:42:52.417696+0000","last_fullsized":"2024-09-16T15:42:52.417696+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:42:49.352371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:42:49.352371+0000","last_clean_scrub_stamp":"2024-09-16T15:42:49.352371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T22:48:39.726611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T15:42:52.876523+0000","last_change":"2024-09-16T15:42:51.414425+0000","last_active":"2024-09-16T15:42:52.876523+0000","last_peered":"2024-09-16T15:42:52.876523+0000","last_clean":"2024-09-16T15:42:52.876523+0000","last_became_active":"2024-09-16T15:42:51.410797+0000","last_became_peered":"2024-09-16T15:42:51.410797+0000","last_unstale":"2024-09-16T15:42:52.876523+0000","last_undegraded":"2024-09-16T15:42:52.876523+0000","last_fullsized":"2024-09-16T15:42:52.876523+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_clean_scrub_stamp":"2024-09-16T15:42:50.361965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T18:40:25.285207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"20'2","reported_seq":17,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T15:42:52.725055+0000","last_change":"2024-09-16T15:42:50.423780+0000","last_active":"2024-09-16T15:42:52.725055+0000","last_peered":"2024-09-16T15:42:52.725055+0000","last_clean":"2024-09-16T15:42:52.725055+0000","last_became_active":"2024-09-16T15:42:50.423037+0000","last_became_peered":"2024-09-16T15:42:50.423037+0000","last_unstale":"2024-09-16T15:42:52.725055+0000","last_undegraded":"2024-09-16T15:42:52.725055+0000","last_fullsized":"2024-09-16T15:42:52.725055+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:42:49.352371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:42:49.352371+0000","last_clean_scrub_stamp":"2024-09-16T15:42:49.352371+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T01:04:05.966343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T15:42:52.415974+0000","last_change":"2024-09-16T15:42:51.397509+0000","last_active":"2024-09-16T15:42:52.415974+0000","last_peered":"2024-09-16T15:42:52.415974+0000","last_clean":"2024-09-16T15:42:52.415974+0000","last_became_active":"2024-09-16T15:42:51.396735+0000","last_became_peered":"2024-09-16T15:42:51.396735+0000","last_unstale":"2024-09-16T15:42:52.415974+0000","last_undegraded":"2024-09-16T15:42:52.415974+0000","last_fullsized":"2024-09-16T15:42:52.415974+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_clean_scrub_stamp":"2024-09-16T15:42:50.361965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T02:47:48.200896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T15:42:52.639530+0000","last_change":"2024-09-16T15:42:51.423874+0000","last_active":"2024-09-16T15:42:52.639530+0000","last_peered":"2024-09-16T15:42:52.639530+0000","last_clean":"2024-09-16T15:42:52.639530+0000","last_became_active":"2024-09-16T15:42:51.423455+0000","last_became_peered":"2024-09-16T15:42:51.423455+0000","last_unstale":"2024-09-16T15:42:52.639530+0000","last_undegraded":"2024-09-16T15:42:52.639530+0000","last_fullsized":"2024-09-16T15:42:52.639530+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_clean_scrub_stamp":"2024-09-16T15:42:50.361965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T22:43:57.612493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"20'2","reported_seq":17,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T15:42:52.724179+0000","last_change":"2024-09-16T15:42:50.428990+0000","last_active":"2024-09-16T15:42:52.724179+0000","last_peered":"2024-09-16T15:42:52.724179+0000","last_clean":"2024-09-16T15:42:52.724179+0000","last_became_active":"2024-09-16T15:42:50.428159+0000","last_became_peered":"2024-09-16T15:42:50.428159+0000","last_unstale":"2024-09-16T15:42:52.724179+0000","last_undegraded":"2024-09-16T15:42:52.724179+0000","last_fullsized":"2024-09-16T15:42:52.724179+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:42:49.352371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:42:49.352371+0000","last_clean_scrub_stamp":"2024-09-16T15:42:49.352371+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T01:17:11.293386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T15:42:52.638975+0000","last_change":"2024-09-16T15:42:51.421429+0000","last_active":"2024-09-16T15:42:52.638975+0000","last_peered":"2024-09-16T15:42:52.638975+0000","last_clean":"2024-09-16T15:42:52.638975+0000","last_became_active":"2024-09-16T15:42:51.420671+0000","last_became_peered":"2024-09-16T15:42:51.420671+0000","last_unstale":"2024-09-16T15:42:52.638975+0000","last_undegraded":"2024-09-16T15:42:52.638975+0000","last_fullsized":"2024-09-16T15:42:52.638975+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_clean_scrub_stamp":"2024-09-16T15:42:50.361965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T00:34:44.681110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"20'2","reported_seq":17,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T15:42:52.724740+0000","last_change":"2024-09-16T15:42:50.425037+0000","last_active":"2024-09-16T15:42:52.724740+0000","last_peered":"2024-09-16T15:42:52.724740+0000","last_clean":"2024-09-16T15:42:52.724740+0000","last_became_active":"2024-09-16T15:42:50.423907+0000","last_became_peered":"2024-09-16T15:42:50.423907+0000","last_unstale":"2024-09-16T15:42:52.724740+0000","last_undegraded":"2024-09-16T15:42:52.724740+0000","last_fullsized":"2024-09-16T15:42:52.724740+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:42:49.352371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:42:49.352371+0000","last_clean_scrub_stamp":"2024-09-16T15:42:49.352371+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:56:59.748634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T15:42:52.877982+0000","last_change":"2024-09-16T15:42:51.425681+0000","last_active":"2024-09-16T15:42:52.877982+0000","last_peered":"2024-09-16T15:42:52.877982+0000","last_clean":"2024-09-16T15:42:52.877982+0000","last_became_active":"2024-09-16T15:42:51.425367+0000","last_became_peered":"2024-09-16T15:42:51.425367+0000","last_unstale":"2024-09-16T15:42:52.877982+0000","last_undegraded":"2024-09-16T15:42:52.877982+0000","last_fullsized":"2024-09-16T15:42:52.877982+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_clean_scrub_stamp":"2024-09-16T15:42:50.361965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:21.758409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"20'1","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T15:42:52.875580+0000","last_change":"2024-09-16T15:42:50.392203+0000","last_active":"2024-09-16T15:42:52.875580+0000","last_peered":"2024-09-16T15:42:52.875580+0000","last_clean":"2024-09-16T15:42:52.875580+0000","last_became_active":"2024-09-16T15:42:50.391647+0000","last_became_peered":"2024-09-16T15:42:50.391647+0000","last_unstale":"2024-09-16T15:42:52.875580+0000","last_undegraded":"2024-09-16T15:42:52.875580+0000","last_fullsized":"2024-09-16T15:42:52.875580+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:42:49.352371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:42:49.352371+0000","last_clean_scrub_stamp":"2024-09-16T15:42:49.352371+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T22:35:29.889897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T15:42:52.876061+0000","last_change":"2024-09-16T15:42:51.408021+0000","last_active":"2024-09-16T15:42:52.876061+0000","last_peered":"2024-09-16T15:42:52.876061+0000","last_clean":"2024-09-16T15:42:52.876061+0000","last_became_active":"2024-09-16T15:42:51.407085+0000","last_became_peered":"2024-09-16T15:42:51.407085+0000","last_unstale":"2024-09-16T15:42:52.876061+0000","last_undegraded":"2024-09-16T15:42:52.876061+0000","last_fullsized":"2024-09-16T15:42:52.876061+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:42:50.361965+0000","last_clean_scrub_stamp":"2024-09-16T15:42:50.361965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:38:38.580439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"20'4","reported_seq":19,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T15:42:52.878108+0000","last_change":"2024-09-16T15:42:50.415954+0000","last_active":"2024-09-16T15:42:52.878108+0000","last_peered":"2024-09-16T15:42:52.878108+0000","last_clean":"2024-09-16T15:42:52.878108+0000","last_became_active":"2024-09-16T15:42:50.415619+0000","last_became_peered":"2024-09-16T15:42:50.415619+0000","last_unstale":"2024-09-16T15:42:52.878108+0000","last_undegraded":"2024-09-16T15:42:52.878108+0000","last_fullsized":"2024-09-16T15:42:52.878108+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:42:49.352371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:42:49.352371+0000","last_clean_scrub_stamp":"2024-09-16T15:42:49.352371+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T02:01:17.143570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542152,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27824,"kb_used_data":792,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344016,"statfs":{"total":96636764160,"available":96608272384,"internally_reserved":0,"allocated":811008,"data_stored":665039,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542151,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27256,"kb_used_data":220,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344584,"statfs":{"total":96636764160,"available":96608854016,"internally_reserved":0,"allocated":225280,"data_stored":75701,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":14,"seq":60129542152,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27240,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344600,"statfs":{"total":96636764160,"available":96608870400,"internally_reserved":0,"allocated":217088,"data_stored":74179,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542150,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27252,"kb_used_data":220,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344588,"statfs":{"total":96636764160,"available":96608858112,"internally_reserved":0,"allocated":225280,"data_stored":74215,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":14,"seq":60129542151,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27256,"kb_used_data":220,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344584,"statfs":{"total":96636764160,"available":96608854016,"internally_reserved":0,"allocated":225280,"data_stored":75701,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":14,"seq":60129542151,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27236,"kb_used_data":208,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344604,"statfs":{"total":96636764160,"available":96608874496,"internally_reserved":0,"allocated":212992,"data_stored":74089,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":14,"seq":60129542153,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27244,"kb_used_data":216,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344596,"statfs":{"total":96636764160,"available":96608866304,"internally_reserved":0,"allocated":221184,"data_stored":74249,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":14,"seq":60129542153,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27832,"kb_used_data":800,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344008,"statfs":{"total":96636764160,"available":96608264192,"internally_reserved":0,"allocated":819200,"data_stored":665095,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T15:43:01.458 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-09-16T15:43:01.458 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-09-16T15:43:01.458 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2024-09-16T15:43:01.458 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2024-09-16T15:43:01.613 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:01.610+0000 7f38b9be9640 1 -- 172.21.15.123:0/543814053 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f38b40cc900 msgr2=0x7f38b40ce5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:43:01.614 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:01.610+0000 7f38b9be9640 1 --2- 172.21.15.123:0/543814053 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f38b40cc900 0x7f38b40ce5e0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f38b406eb10 tx=0x7f38ac01f520 comp rx=0 tx=0).stop 2024-09-16T15:43:01.614 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:01.610+0000 7f38b9be9640 1 -- 172.21.15.123:0/543814053 shutdown_connections 2024-09-16T15:43:01.614 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:01.610+0000 7f38b9be9640 1 --2- 172.21.15.123:0/543814053 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f38b40cc900 0x7f38b40ce5e0 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:01.614 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:01.610+0000 7f38b9be9640 1 --2- 172.21.15.123:0/543814053 >> v2:172.21.15.184:3300/0 conn(0x7f38b415e380 0x7f38b40b0500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:01.614 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:01.610+0000 7f38b9be9640 1 -- 172.21.15.123:0/543814053 >> 172.21.15.123:0/543814053 conn(0x7f38b40b1df0 msgr2=0x7f38b40b2200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:43:01.614 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:01.610+0000 7f38b9be9640 1 -- 172.21.15.123:0/543814053 shutdown_connections 2024-09-16T15:43:01.614 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:01.610+0000 7f38b9be9640 1 -- 172.21.15.123:0/543814053 wait complete. 2024-09-16T15:43:01.614 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:01.614+0000 7f38b9be9640 1 Processor -- start 2024-09-16T15:43:01.615 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:01.614+0000 7f38b9be9640 1 -- start start 2024-09-16T15:43:01.616 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:01.614+0000 7f38b9be9640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f38b40cc3b0 0x7f38b415c370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:43:01.616 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:01.614+0000 7f38b9be9640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f38b40cc900 0x7f38b41f3ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:43:01.617 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:01.614+0000 7f38b9be9640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f38b415e380 0x7f38b41f4020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:43:01.617 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:01.614+0000 7f38b9be9640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f38b40b0dd0 con 0x7f38b40cc3b0 2024-09-16T15:43:01.617 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:01.614+0000 7f38b9be9640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f38b40b0c50 con 0x7f38b415e380 2024-09-16T15:43:01.617 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:01.614+0000 7f38b9be9640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f38b40b0f50 con 0x7f38b40cc900 2024-09-16T15:43:01.617 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:01.614+0000 7f38b37fe640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f38b40cc3b0 0x7f38b415c370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:43:01.617 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:01.614+0000 7f38b37fe640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f38b40cc3b0 0x7f38b415c370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:40750/0 (socket says 172.21.15.123:40750) 2024-09-16T15:43:01.617 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:01.614+0000 7f38b37fe640 1 -- 172.21.15.123:0/472534952 learned_addr learned my addr 172.21.15.123:0/472534952 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:43:01.617 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:01.614+0000 7f38b3fff640 1 --2- 172.21.15.123:0/472534952 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f38b415e380 0x7f38b41f4020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:43:01.617 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:01.614+0000 7f38b37fe640 1 -- 172.21.15.123:0/472534952 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f38b40cc900 msgr2=0x7f38b41f3ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:43:01.617 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:01.614+0000 7f38aaffd640 1 --2- 172.21.15.123:0/472534952 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f38b40cc900 0x7f38b41f3ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:43:01.617 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:01.614+0000 7f38b37fe640 1 --2- 172.21.15.123:0/472534952 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f38b40cc900 0x7f38b41f3ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:01.617 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:01.614+0000 7f38b37fe640 1 -- 172.21.15.123:0/472534952 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f38b415e380 msgr2=0x7f38b41f4020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:43:01.618 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:01.614+0000 7f38b37fe640 1 --2- 172.21.15.123:0/472534952 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f38b415e380 0x7f38b41f4020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:01.618 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:01.614+0000 7f38b37fe640 1 -- 172.21.15.123:0/472534952 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f38b40c6fd0 con 0x7f38b40cc3b0 2024-09-16T15:43:01.618 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:01.614+0000 7f38b3fff640 1 --2- 172.21.15.123:0/472534952 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f38b415e380 0x7f38b41f4020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T15:43:01.618 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:01.614+0000 7f38b37fe640 1 --2- 172.21.15.123:0/472534952 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f38b40cc3b0 0x7f38b415c370 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f38a400b410 tx=0x7f38a400b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:43:01.618 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:01.614+0000 7f38b0ff9640 1 -- 172.21.15.123:0/472534952 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38a4015020 con 0x7f38b40cc3b0 2024-09-16T15:43:01.618 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:01.614+0000 7f38b0ff9640 1 -- 172.21.15.123:0/472534952 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f38a4011920 con 0x7f38b40cc3b0 2024-09-16T15:43:01.618 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:01.614+0000 7f38b9be9640 1 -- 172.21.15.123:0/472534952 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f38b40c6cf0 con 0x7f38b40cc3b0 2024-09-16T15:43:01.618 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:01.614+0000 7f38b0ff9640 1 -- 172.21.15.123:0/472534952 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38a4011d70 con 0x7f38b40cc3b0 2024-09-16T15:43:01.618 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:01.614+0000 7f38b9be9640 1 -- 172.21.15.123:0/472534952 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f38b4159d50 con 0x7f38b40cc3b0 2024-09-16T15:43:01.619 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:01.614+0000 7f38b9be9640 1 -- 172.21.15.123:0/472534952 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f38b415a130 con 0x7f38b40cc3b0 2024-09-16T15:43:01.619 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:01.614+0000 7f38b0ff9640 1 -- 172.21.15.123:0/472534952 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 5) ==== 97080+0+0 (secure 0 0 0) 0x7f38a4018020 con 0x7f38b40cc3b0 2024-09-16T15:43:01.619 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:01.618+0000 7f38b0ff9640 1 --2- 172.21.15.123:0/472534952 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f3894072ec0 0x7f3894075380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:43:01.619 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:01.618+0000 7f38b0ff9640 1 -- 172.21.15.123:0/472534952 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f38a4093ca0 con 0x7f38b40cc3b0 2024-09-16T15:43:01.619 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:01.618+0000 7f38aaffd640 1 --2- 172.21.15.123:0/472534952 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f3894072ec0 0x7f3894075380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:43:01.620 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:01.618+0000 7f38aaffd640 1 --2- 172.21.15.123:0/472534952 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f3894072ec0 0x7f3894075380 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f38b4155eb0 tx=0x7f389c009040 comp rx=0 tx=0).ready entity=mgr.4135 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:43:01.622 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:01.622+0000 7f38b0ff9640 1 -- 172.21.15.123:0/472534952 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f38b415a130 con 0x7f38b40cc3b0 2024-09-16T15:43:01.816 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:01.814+0000 7f38b9be9640 1 -- 172.21.15.123:0/472534952 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) -- 0x7f38b40b0500 con 0x7f38b40cc3b0 2024-09-16T15:43:01.816 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:01.814+0000 7f38b0ff9640 1 -- 172.21.15.123:0/472534952 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) ==== 72+0+46 (secure 0 0 0) 0x7f38a401f020 con 0x7f38b40cc3b0 2024-09-16T15:43:01.816 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-09-16T15:43:01.817 INFO:teuthology.orchestra.run.smithi123.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-09-16T15:43:01.819 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:01.818+0000 7f38b9be9640 1 -- 172.21.15.123:0/472534952 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f3894072ec0 msgr2=0x7f3894075380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:43:01.819 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:01.818+0000 7f38b9be9640 1 --2- 172.21.15.123:0/472534952 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f3894072ec0 0x7f3894075380 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f38b4155eb0 tx=0x7f389c009040 comp rx=0 tx=0).stop 2024-09-16T15:43:01.819 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:01.818+0000 7f38b9be9640 1 -- 172.21.15.123:0/472534952 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f38b40cc3b0 msgr2=0x7f38b415c370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:43:01.819 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:01.818+0000 7f38b9be9640 1 --2- 172.21.15.123:0/472534952 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f38b40cc3b0 0x7f38b415c370 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f38a400b410 tx=0x7f38a400b8d0 comp rx=0 tx=0).stop 2024-09-16T15:43:01.819 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:01.818+0000 7f38b9be9640 1 -- 172.21.15.123:0/472534952 shutdown_connections 2024-09-16T15:43:01.819 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:01.818+0000 7f38b9be9640 1 --2- 172.21.15.123:0/472534952 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f3894072ec0 0x7f3894075380 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:01.820 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:01.818+0000 7f38b9be9640 1 --2- 172.21.15.123:0/472534952 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f38b415e380 0x7f38b41f4020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:01.820 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:01.818+0000 7f38b9be9640 1 --2- 172.21.15.123:0/472534952 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f38b40cc900 0x7f38b41f3ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:01.820 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:01.818+0000 7f38b9be9640 1 --2- 172.21.15.123:0/472534952 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f38b40cc3b0 0x7f38b415c370 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:01.820 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:01.818+0000 7f38b9be9640 1 -- 172.21.15.123:0/472534952 >> 172.21.15.123:0/472534952 conn(0x7f38b40b1df0 msgr2=0x7f38b40cd160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:43:01.820 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:01.818+0000 7f38b9be9640 1 -- 172.21.15.123:0/472534952 shutdown_connections 2024-09-16T15:43:01.820 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:01.818+0000 7f38b9be9640 1 -- 172.21.15.123:0/472534952 wait complete. 2024-09-16T15:43:01.834 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2024-09-16T15:43:01.834 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-16T15:43:01.947 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:01.946+0000 7f36d5758640 1 -- 172.21.15.123:0/2068231662 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f36d00cd6e0 msgr2=0x7f36d00cdb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:43:01.947 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:01.946+0000 7f36d5758640 1 --2- 172.21.15.123:0/2068231662 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f36d00cd6e0 0x7f36d00cdb20 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f36c4009900 tx=0x7f36c401c850 comp rx=0 tx=0).stop 2024-09-16T15:43:01.947 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:01.946+0000 7f36d5758640 1 -- 172.21.15.123:0/2068231662 shutdown_connections 2024-09-16T15:43:01.947 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:01.946+0000 7f36d5758640 1 --2- 172.21.15.123:0/2068231662 >> v2:172.21.15.184:3300/0 conn(0x7f36d015e820 0x7f36d015ecd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:01.947 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:01.946+0000 7f36d5758640 1 --2- 172.21.15.123:0/2068231662 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f36d00cd6e0 0x7f36d00cdb20 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:01.947 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:01.946+0000 7f36d5758640 1 -- 172.21.15.123:0/2068231662 >> 172.21.15.123:0/2068231662 conn(0x7f36d00cfb10 msgr2=0x7f36d00cff20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:43:01.947 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:01.946+0000 7f36d5758640 1 -- 172.21.15.123:0/2068231662 shutdown_connections 2024-09-16T15:43:01.947 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:01.946+0000 7f36d5758640 1 -- 172.21.15.123:0/2068231662 wait complete. 2024-09-16T15:43:01.948 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:01.946+0000 7f36d5758640 1 Processor -- start 2024-09-16T15:43:01.948 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:01.946+0000 7f36d5758640 1 -- start start 2024-09-16T15:43:01.950 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:01.946+0000 7f36d5758640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f36d00cd6e0 0x7f36d00c4050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:43:01.950 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:01.946+0000 7f36d5758640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f36d015e160 0x7f36d00af800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:43:01.950 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:01.946+0000 7f36d5758640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f36d015e820 0x7f36d01506d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:43:01.950 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:01.946+0000 7f36d5758640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f36d00cf5b0 con 0x7f36d015e160 2024-09-16T15:43:01.950 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:01.946+0000 7f36d5758640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f36d00cf430 con 0x7f36d015e820 2024-09-16T15:43:01.951 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:01.946+0000 7f36d5758640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f36d01683d0 con 0x7f36d00cd6e0 2024-09-16T15:43:01.951 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:01.946+0000 7f36ce7fc640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f36d015e160 0x7f36d00af800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:43:01.951 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:01.946+0000 7f36ce7fc640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f36d015e160 0x7f36d00af800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:40752/0 (socket says 172.21.15.123:40752) 2024-09-16T15:43:01.951 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:01.946+0000 7f36ce7fc640 1 -- 172.21.15.123:0/1730543536 learned_addr learned my addr 172.21.15.123:0/1730543536 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:43:01.951 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:01.946+0000 7f36ceffd640 1 --2- 172.21.15.123:0/1730543536 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f36d00cd6e0 0x7f36d00c4050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:43:01.951 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:01.946+0000 7f36cf7fe640 1 --2- 172.21.15.123:0/1730543536 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f36d015e820 0x7f36d01506d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:43:01.951 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:01.946+0000 7f36ce7fc640 1 -- 172.21.15.123:0/1730543536 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f36d00cd6e0 msgr2=0x7f36d00c4050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:43:01.951 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:01.946+0000 7f36ce7fc640 1 --2- 172.21.15.123:0/1730543536 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f36d00cd6e0 0x7f36d00c4050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:01.951 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:01.946+0000 7f36ce7fc640 1 -- 172.21.15.123:0/1730543536 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f36d015e820 msgr2=0x7f36d01506d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:43:01.951 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:01.946+0000 7f36ce7fc640 1 --2- 172.21.15.123:0/1730543536 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f36d015e820 0x7f36d01506d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:01.951 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:01.946+0000 7f36ce7fc640 1 -- 172.21.15.123:0/1730543536 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f36d00b2b20 con 0x7f36d015e160 2024-09-16T15:43:01.951 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:01.946+0000 7f36ce7fc640 1 --2- 172.21.15.123:0/1730543536 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f36d015e160 0x7f36d00af800 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f36c4009850 tx=0x7f36c40a5640 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:43:01.951 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:01.946+0000 7f36b77fe640 1 -- 172.21.15.123:0/1730543536 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36c40b2040 con 0x7f36d015e160 2024-09-16T15:43:01.952 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:01.946+0000 7f36b77fe640 1 -- 172.21.15.123:0/1730543536 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f36c4004270 con 0x7f36d015e160 2024-09-16T15:43:01.952 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:01.946+0000 7f36d5758640 1 -- 172.21.15.123:0/1730543536 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f36d00b2e00 con 0x7f36d015e160 2024-09-16T15:43:01.952 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:01.950+0000 7f36b77fe640 1 -- 172.21.15.123:0/1730543536 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36c4005070 con 0x7f36d015e160 2024-09-16T15:43:01.952 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:01.950+0000 7f36d5758640 1 -- 172.21.15.123:0/1730543536 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f36d00b33c0 con 0x7f36d015e160 2024-09-16T15:43:01.952 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:01.950+0000 7f36d5758640 1 -- 172.21.15.123:0/1730543536 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f36d00cdb20 con 0x7f36d015e160 2024-09-16T15:43:01.953 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:01.950+0000 7f36b77fe640 1 -- 172.21.15.123:0/1730543536 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 5) ==== 97080+0+0 (secure 0 0 0) 0x7f36c40bd020 con 0x7f36d015e160 2024-09-16T15:43:01.953 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:01.950+0000 7f36b77fe640 1 --2- 172.21.15.123:0/1730543536 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f36bc072ec0 0x7f36bc075380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:43:01.953 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:01.950+0000 7f36b77fe640 1 -- 172.21.15.123:0/1730543536 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f36c412daf0 con 0x7f36d015e160 2024-09-16T15:43:01.953 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:01.950+0000 7f36ceffd640 1 --2- 172.21.15.123:0/1730543536 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f36bc072ec0 0x7f36bc075380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:43:01.953 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:01.950+0000 7f36ceffd640 1 --2- 172.21.15.123:0/1730543536 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f36bc072ec0 0x7f36bc075380 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f36c00016f0 tx=0x7f36c0009040 comp rx=0 tx=0).ready entity=mgr.4135 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:43:01.956 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:01.954+0000 7f36b77fe640 1 -- 172.21.15.123:0/1730543536 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f36c40164d0 con 0x7f36d015e160 2024-09-16T15:43:02.149 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:02.146+0000 7f36d5758640 1 -- 172.21.15.123:0/1730543536 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f36d00ce3c0 con 0x7f36d015e160 2024-09-16T15:43:02.150 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:02.146+0000 7f36b77fe640 1 -- 172.21.15.123:0/1730543536 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) ==== 93+0+5226 (secure 0 0 0) 0x7f36c40f8d50 con 0x7f36d015e160 2024-09-16T15:43:02.151 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-09-16T15:43:02.151 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":8,"btime":"2024-09-16T15:42:54:421548+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4280,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6835/1497621896","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":1497621896},{"type":"v1","addr":"172.21.15.184:6835","nonce":1497621896}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4305,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6833/2227496057","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6832","nonce":2227496057},{"type":"v1","addr":"172.21.15.123:6833","nonce":2227496057}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4311,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6835/294455998","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6834","nonce":294455998},{"type":"v1","addr":"172.21.15.123:6835","nonce":294455998}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-16T15:42:50.762280+0000","modified":"2024-09-16T15:42:53.789989+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4348},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4348":{"gid":4348,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.184:6837/1469324095","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6836","nonce":1469324095},{"type":"v1","addr":"172.21.15.184:6837","nonce":1469324095}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4348,"qdb_cluster":[4348]},"id":1}]} 2024-09-16T15:43:02.151 INFO:teuthology.orchestra.run.smithi123.stderr:dumped fsmap epoch 8 2024-09-16T15:43:02.154 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:02.150+0000 7f36d5758640 1 -- 172.21.15.123:0/1730543536 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f36bc072ec0 msgr2=0x7f36bc075380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:43:02.154 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:02.150+0000 7f36d5758640 1 --2- 172.21.15.123:0/1730543536 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f36bc072ec0 0x7f36bc075380 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f36c00016f0 tx=0x7f36c0009040 comp rx=0 tx=0).stop 2024-09-16T15:43:02.154 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:02.150+0000 7f36d5758640 1 -- 172.21.15.123:0/1730543536 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f36d015e160 msgr2=0x7f36d00af800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:43:02.154 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:02.150+0000 7f36d5758640 1 --2- 172.21.15.123:0/1730543536 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f36d015e160 0x7f36d00af800 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f36c4009850 tx=0x7f36c40a5640 comp rx=0 tx=0).stop 2024-09-16T15:43:02.154 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:02.150+0000 7f36d5758640 1 -- 172.21.15.123:0/1730543536 shutdown_connections 2024-09-16T15:43:02.155 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:02.150+0000 7f36d5758640 1 --2- 172.21.15.123:0/1730543536 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f36bc072ec0 0x7f36bc075380 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:02.155 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:02.150+0000 7f36d5758640 1 --2- 172.21.15.123:0/1730543536 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f36d015e820 0x7f36d01506d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:02.155 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:02.150+0000 7f36d5758640 1 --2- 172.21.15.123:0/1730543536 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f36d015e160 0x7f36d00af800 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:02.155 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:02.150+0000 7f36d5758640 1 --2- 172.21.15.123:0/1730543536 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f36d00cd6e0 0x7f36d00c4050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:02.155 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:02.150+0000 7f36d5758640 1 -- 172.21.15.123:0/1730543536 >> 172.21.15.123:0/1730543536 conn(0x7f36d00cfb10 msgr2=0x7f36d00ce8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:43:02.155 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:02.150+0000 7f36d5758640 1 -- 172.21.15.123:0/1730543536 shutdown_connections 2024-09-16T15:43:02.155 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:02.150+0000 7f36d5758640 1 -- 172.21.15.123:0/1730543536 wait complete. 2024-09-16T15:43:02.170 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-16T15:43:02.295 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:02.294+0000 7f6400cae640 1 -- 172.21.15.123:0/3869565916 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f63e8003a40 msgr2=0x7f63e8003e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:43:02.295 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:02.294+0000 7f6400cae640 1 --2- 172.21.15.123:0/3869565916 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f63e8003a40 0x7f63e8003e60 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f63f400c610 tx=0x7f63f401d290 comp rx=0 tx=0).stop 2024-09-16T15:43:02.295 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:02.294+0000 7f63fad9d640 1 -- 172.21.15.123:0/3869565916 reap_dead start 2024-09-16T15:43:02.295 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:02.294+0000 7f6400cae640 1 -- 172.21.15.123:0/3869565916 shutdown_connections 2024-09-16T15:43:02.295 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:02.294+0000 7f6400cae640 1 -- 172.21.15.123:0/3869565916 >> 172.21.15.123:0/3869565916 conn(0x7f63fc0cfb10 msgr2=0x7f63fc0cff20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:43:02.296 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:02.294+0000 7f6400cae640 1 -- 172.21.15.123:0/3869565916 shutdown_connections 2024-09-16T15:43:02.296 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:02.294+0000 7f6400cae640 1 -- 172.21.15.123:0/3869565916 wait complete. 2024-09-16T15:43:02.296 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:02.294+0000 7f6400cae640 1 Processor -- start 2024-09-16T15:43:02.296 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:02.294+0000 7f6400cae640 1 -- start start 2024-09-16T15:43:02.298 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:02.294+0000 7f6400cae640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f63e8003a40 0x7f63fc1f5ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:43:02.299 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:02.294+0000 7f6400cae640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f63fc15e4c0 0x7f63fc0c8d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:43:02.299 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:02.294+0000 7f6400cae640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f63fc14b680 0x7f63fc0c6be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:43:02.299 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:02.294+0000 7f6400cae640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f63fc167c60 con 0x7f63fc14b680 2024-09-16T15:43:02.299 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:02.294+0000 7f6400cae640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f63fc06fa20 con 0x7f63e8003a40 2024-09-16T15:43:02.299 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:02.294+0000 7f6400cae640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f63fc167de0 con 0x7f63fc15e4c0 2024-09-16T15:43:02.299 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:02.294+0000 7f63fad9d640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f63fc14b680 0x7f63fc0c6be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:43:02.299 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:02.294+0000 7f63fad9d640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f63fc14b680 0x7f63fc0c6be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:40770/0 (socket says 172.21.15.123:40770) 2024-09-16T15:43:02.299 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:02.294+0000 7f63fad9d640 1 -- 172.21.15.123:0/291251126 learned_addr learned my addr 172.21.15.123:0/291251126 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:43:02.299 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:02.294+0000 7f63f9d9b640 1 --2- 172.21.15.123:0/291251126 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f63fc15e4c0 0x7f63fc0c8d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:43:02.299 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:02.294+0000 7f63fa59c640 1 --2- 172.21.15.123:0/291251126 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f63e8003a40 0x7f63fc1f5ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:43:02.299 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:02.294+0000 7f63fad9d640 1 -- 172.21.15.123:0/291251126 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f63fc15e4c0 msgr2=0x7f63fc0c8d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:43:02.300 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:02.294+0000 7f63fad9d640 1 --2- 172.21.15.123:0/291251126 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f63fc15e4c0 0x7f63fc0c8d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:02.300 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:02.294+0000 7f63fad9d640 1 -- 172.21.15.123:0/291251126 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f63e8003a40 msgr2=0x7f63fc1f5ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:43:02.300 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:02.294+0000 7f63fad9d640 1 --2- 172.21.15.123:0/291251126 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f63e8003a40 0x7f63fc1f5ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:02.300 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:02.294+0000 7f63fad9d640 1 -- 172.21.15.123:0/291251126 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f63fc15e920 con 0x7f63fc14b680 2024-09-16T15:43:02.300 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:02.294+0000 7f63fad9d640 1 --2- 172.21.15.123:0/291251126 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f63fc14b680 0x7f63fc0c6be0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f63fc076230 tx=0x7f63ec011b10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:43:02.300 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:02.294+0000 7f63f2ffd640 1 -- 172.21.15.123:0/291251126 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63ec004230 con 0x7f63fc14b680 2024-09-16T15:43:02.300 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:02.294+0000 7f63f2ffd640 1 -- 172.21.15.123:0/291251126 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f63ec016070 con 0x7f63fc14b680 2024-09-16T15:43:02.300 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:02.294+0000 7f63f2ffd640 1 -- 172.21.15.123:0/291251126 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63ec00fb20 con 0x7f63fc14b680 2024-09-16T15:43:02.300 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:02.298+0000 7f6400cae640 1 -- 172.21.15.123:0/291251126 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f63fc15eba0 con 0x7f63fc14b680 2024-09-16T15:43:02.300 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:02.298+0000 7f6400cae640 1 -- 172.21.15.123:0/291251126 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f63fc15c800 con 0x7f63fc14b680 2024-09-16T15:43:02.301 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:02.298+0000 7f6400cae640 1 -- 172.21.15.123:0/291251126 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f63fc15c340 con 0x7f63fc14b680 2024-09-16T15:43:02.301 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:02.298+0000 7f63f2ffd640 1 -- 172.21.15.123:0/291251126 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 5) ==== 97080+0+0 (secure 0 0 0) 0x7f63ec003600 con 0x7f63fc14b680 2024-09-16T15:43:02.301 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:02.298+0000 7f63f2ffd640 1 --2- 172.21.15.123:0/291251126 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f63dc072ef0 0x7f63dc0753b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:43:02.301 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:02.298+0000 7f63f2ffd640 1 -- 172.21.15.123:0/291251126 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f63ec094810 con 0x7f63fc14b680 2024-09-16T15:43:02.302 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:02.298+0000 7f63fa59c640 1 --2- 172.21.15.123:0/291251126 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f63dc072ef0 0x7f63dc0753b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:43:02.302 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:02.298+0000 7f63fa59c640 1 --2- 172.21.15.123:0/291251126 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f63dc072ef0 0x7f63dc0753b0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f63f40ac000 tx=0x7f63f401a0d0 comp rx=0 tx=0).ready entity=mgr.4135 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:43:02.304 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:02.302+0000 7f63f2ffd640 1 -- 172.21.15.123:0/291251126 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f63fc15c340 con 0x7f63fc14b680 2024-09-16T15:43:02.498 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:02.498+0000 7f6400cae640 1 -- 172.21.15.123:0/291251126 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f63fc06e750 con 0x7f63fc14b680 2024-09-16T15:43:02.499 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:02.498+0000 7f63f2ffd640 1 -- 172.21.15.123:0/291251126 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) ==== 93+0+5226 (secure 0 0 0) 0x7f63ec022020 con 0x7f63fc14b680 2024-09-16T15:43:02.499 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-09-16T15:43:02.499 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":8,"btime":"2024-09-16T15:42:54:421548+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4280,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6835/1497621896","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":1497621896},{"type":"v1","addr":"172.21.15.184:6835","nonce":1497621896}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4305,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6833/2227496057","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6832","nonce":2227496057},{"type":"v1","addr":"172.21.15.123:6833","nonce":2227496057}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4311,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6835/294455998","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6834","nonce":294455998},{"type":"v1","addr":"172.21.15.123:6835","nonce":294455998}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-16T15:42:50.762280+0000","modified":"2024-09-16T15:42:53.789989+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4348},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4348":{"gid":4348,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.184:6837/1469324095","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6836","nonce":1469324095},{"type":"v1","addr":"172.21.15.184:6837","nonce":1469324095}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4348,"qdb_cluster":[4348]},"id":1}]} 2024-09-16T15:43:02.500 INFO:teuthology.orchestra.run.smithi123.stderr:dumped fsmap epoch 8 2024-09-16T15:43:02.502 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:02.498+0000 7f6400cae640 1 -- 172.21.15.123:0/291251126 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f63dc072ef0 msgr2=0x7f63dc0753b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:43:02.502 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:02.498+0000 7f6400cae640 1 --2- 172.21.15.123:0/291251126 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f63dc072ef0 0x7f63dc0753b0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f63f40ac000 tx=0x7f63f401a0d0 comp rx=0 tx=0).stop 2024-09-16T15:43:02.502 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:02.498+0000 7f6400cae640 1 -- 172.21.15.123:0/291251126 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f63fc14b680 msgr2=0x7f63fc0c6be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:43:02.502 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:02.498+0000 7f6400cae640 1 --2- 172.21.15.123:0/291251126 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f63fc14b680 0x7f63fc0c6be0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f63fc076230 tx=0x7f63ec011b10 comp rx=0 tx=0).stop 2024-09-16T15:43:02.502 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:02.498+0000 7f6400cae640 1 -- 172.21.15.123:0/291251126 shutdown_connections 2024-09-16T15:43:02.502 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:02.498+0000 7f6400cae640 1 --2- 172.21.15.123:0/291251126 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f63dc072ef0 0x7f63dc0753b0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:02.502 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:02.498+0000 7f6400cae640 1 --2- 172.21.15.123:0/291251126 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f63fc14b680 0x7f63fc0c6be0 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:02.502 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:02.498+0000 7f6400cae640 1 --2- 172.21.15.123:0/291251126 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f63fc15e4c0 0x7f63fc0c8d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:02.503 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:02.498+0000 7f6400cae640 1 --2- 172.21.15.123:0/291251126 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f63e8003a40 0x7f63fc1f5ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:02.503 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:02.498+0000 7f6400cae640 1 -- 172.21.15.123:0/291251126 >> 172.21.15.123:0/291251126 conn(0x7f63fc0cfb10 msgr2=0x7f63fc0b29a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:43:02.503 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:02.498+0000 7f6400cae640 1 -- 172.21.15.123:0/291251126 shutdown_connections 2024-09-16T15:43:02.503 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:02.498+0000 7f6400cae640 1 -- 172.21.15.123:0/291251126 wait complete. 2024-09-16T15:43:02.517 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-16T15:43:02.640 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:02.638+0000 7fa910ff8640 1 -- 172.21.15.123:0/3690133292 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa90c074600 msgr2=0x7fa90c06e750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:43:02.640 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:02.638+0000 7fa910ff8640 1 --2- 172.21.15.123:0/3690133292 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa90c074600 0x7fa90c06e750 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fa90c0cd650 tx=0x7fa90401cac0 comp rx=0 tx=0).stop 2024-09-16T15:43:02.640 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:02.638+0000 7fa910ff8640 1 -- 172.21.15.123:0/3690133292 shutdown_connections 2024-09-16T15:43:02.640 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:02.638+0000 7fa910ff8640 1 --2- 172.21.15.123:0/3690133292 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa90c074600 0x7fa90c06e750 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:02.641 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:02.638+0000 7fa910ff8640 1 --2- 172.21.15.123:0/3690133292 >> v2:172.21.15.184:3300/0 conn(0x7fa90c15bca0 0x7fa90c0d0630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:02.641 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:02.638+0000 7fa910ff8640 1 -- 172.21.15.123:0/3690133292 >> 172.21.15.123:0/3690133292 conn(0x7fa90c0cfb10 msgr2=0x7fa90c0cff20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:43:02.641 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:02.638+0000 7fa910ff8640 1 -- 172.21.15.123:0/3690133292 shutdown_connections 2024-09-16T15:43:02.641 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:02.638+0000 7fa910ff8640 1 -- 172.21.15.123:0/3690133292 wait complete. 2024-09-16T15:43:02.641 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:02.638+0000 7fa910ff8640 1 Processor -- start 2024-09-16T15:43:02.641 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:02.638+0000 7fa910ff8640 1 -- start start 2024-09-16T15:43:02.643 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:02.638+0000 7fa910ff8640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fa90c0726c0 0x7fa90c0c5ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:43:02.643 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:02.638+0000 7fa910ff8640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa90c074600 0x7fa90c152b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:43:02.643 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:02.638+0000 7fa910ff8640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa90c15bca0 0x7fa90c0cef70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:43:02.643 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:02.638+0000 7fa910ff8640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fa90c168a50 con 0x7fa90c074600 2024-09-16T15:43:02.643 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:02.638+0000 7fa910ff8640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fa90c06f980 con 0x7fa90c15bca0 2024-09-16T15:43:02.643 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:02.638+0000 7fa910ff8640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fa90c168bd0 con 0x7fa90c0726c0 2024-09-16T15:43:02.643 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:02.638+0000 7fa909d9b640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa90c074600 0x7fa90c152b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:43:02.644 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:02.638+0000 7fa909d9b640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa90c074600 0x7fa90c152b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:41686/0 (socket says 172.21.15.123:41686) 2024-09-16T15:43:02.644 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:02.642+0000 7fa909d9b640 1 -- 172.21.15.123:0/2098123231 learned_addr learned my addr 172.21.15.123:0/2098123231 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:43:02.644 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:02.642+0000 7fa90a59c640 1 --2- 172.21.15.123:0/2098123231 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fa90c0726c0 0x7fa90c0c5ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:43:02.644 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:02.642+0000 7fa90ad9d640 1 --2- 172.21.15.123:0/2098123231 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa90c15bca0 0x7fa90c0cef70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:43:02.644 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:02.642+0000 7fa909d9b640 1 -- 172.21.15.123:0/2098123231 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fa90c0726c0 msgr2=0x7fa90c0c5ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:43:02.644 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:02.642+0000 7fa909d9b640 1 --2- 172.21.15.123:0/2098123231 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fa90c0726c0 0x7fa90c0c5ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:02.644 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:02.642+0000 7fa909d9b640 1 -- 172.21.15.123:0/2098123231 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa90c15bca0 msgr2=0x7fa90c0cef70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:43:02.644 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:02.642+0000 7fa909d9b640 1 --2- 172.21.15.123:0/2098123231 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa90c15bca0 0x7fa90c0cef70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:02.644 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:02.642+0000 7fa909d9b640 1 -- 172.21.15.123:0/2098123231 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa90c0c63e0 con 0x7fa90c074600 2024-09-16T15:43:02.644 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:02.642+0000 7fa909d9b640 1 --2- 172.21.15.123:0/2098123231 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa90c074600 0x7fa90c152b80 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fa90000c490 tx=0x7fa90000c950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:43:02.644 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:02.642+0000 7fa8faffd640 1 -- 172.21.15.123:0/2098123231 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa900016020 con 0x7fa90c074600 2024-09-16T15:43:02.644 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:02.642+0000 7fa8faffd640 1 -- 172.21.15.123:0/2098123231 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fa900012940 con 0x7fa90c074600 2024-09-16T15:43:02.644 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:02.642+0000 7fa910ff8640 1 -- 172.21.15.123:0/2098123231 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa90c15ed20 con 0x7fa90c074600 2024-09-16T15:43:02.644 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:02.642+0000 7fa8faffd640 1 -- 172.21.15.123:0/2098123231 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa900012d90 con 0x7fa90c074600 2024-09-16T15:43:02.645 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:02.642+0000 7fa910ff8640 1 -- 172.21.15.123:0/2098123231 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa90c0c5980 con 0x7fa90c074600 2024-09-16T15:43:02.645 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:02.642+0000 7fa910ff8640 1 -- 172.21.15.123:0/2098123231 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa90c06e750 con 0x7fa90c074600 2024-09-16T15:43:02.645 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:02.642+0000 7fa8faffd640 1 -- 172.21.15.123:0/2098123231 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 5) ==== 97080+0+0 (secure 0 0 0) 0x7fa900019020 con 0x7fa90c074600 2024-09-16T15:43:02.645 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:02.642+0000 7fa8faffd640 1 --2- 172.21.15.123:0/2098123231 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fa8ec072f10 0x7fa8ec0753d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:43:02.646 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:02.642+0000 7fa8faffd640 1 -- 172.21.15.123:0/2098123231 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7fa900093a00 con 0x7fa90c074600 2024-09-16T15:43:02.646 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:02.642+0000 7fa90a59c640 1 --2- 172.21.15.123:0/2098123231 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fa8ec072f10 0x7fa8ec0753d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:43:02.646 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:02.642+0000 7fa90a59c640 1 --2- 172.21.15.123:0/2098123231 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fa8ec072f10 0x7fa8ec0753d0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fa8fc005fd0 tx=0x7fa8fc005e40 comp rx=0 tx=0).ready entity=mgr.4135 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:43:02.649 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:02.646+0000 7fa8faffd640 1 -- 172.21.15.123:0/2098123231 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7fa900020030 con 0x7fa90c074600 2024-09-16T15:43:02.816 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:02.814+0000 7fa910ff8640 1 -- 172.21.15.123:0/2098123231 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fa90c15f630 con 0x7fa90c074600 2024-09-16T15:43:02.818 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:02.814+0000 7fa8faffd640 1 -- 172.21.15.123:0/2098123231 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) ==== 74+0+16725 (secure 0 0 0) 0x7fa90005ee80 con 0x7fa90c074600 2024-09-16T15:43:02.818 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-09-16T15:43:02.818 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":21,"fsid":"56077f89-18c8-486f-ae7e-eecc0bb7822c","created":"2024-09-16T15:42:26.364892+0000","modified":"2024-09-16T15:42:52.401147+0000","last_up_change":"2024-09-16T15:42:38.707234+0000","last_in_change":"2024-09-16T15:42:29.423662+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-16T15:42:39.287764+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-09-16T15:42:48.583563+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-09-16T15:42:49.704665+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9e7c1955-a607-4671-9be7-f2ef760ceeaa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6800","nonce":3882044969},{"type":"v1","addr":"172.21.15.123:6801","nonce":3882044969}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6802","nonce":3882044969},{"type":"v1","addr":"172.21.15.123:6803","nonce":3882044969}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6806","nonce":3882044969},{"type":"v1","addr":"172.21.15.123:6807","nonce":3882044969}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6804","nonce":3882044969},{"type":"v1","addr":"172.21.15.123:6805","nonce":3882044969}]},"public_addr":"172.21.15.123:6801/3882044969","cluster_addr":"172.21.15.123:6803/3882044969","heartbeat_back_addr":"172.21.15.123:6807/3882044969","heartbeat_front_addr":"172.21.15.123:6805/3882044969","state":["exists","up"]},{"osd":1,"uuid":"c540cfde-0977-4742-9b17-fd634668e010","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6816","nonce":4037001850},{"type":"v1","addr":"172.21.15.123:6817","nonce":4037001850}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6818","nonce":4037001850},{"type":"v1","addr":"172.21.15.123:6819","nonce":4037001850}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6822","nonce":4037001850},{"type":"v1","addr":"172.21.15.123:6823","nonce":4037001850}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6820","nonce":4037001850},{"type":"v1","addr":"172.21.15.123:6821","nonce":4037001850}]},"public_addr":"172.21.15.123:6817/4037001850","cluster_addr":"172.21.15.123:6819/4037001850","heartbeat_back_addr":"172.21.15.123:6823/4037001850","heartbeat_front_addr":"172.21.15.123:6821/4037001850","state":["exists","up"]},{"osd":2,"uuid":"9864a496-f169-486b-95c4-1d59162793a7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6824","nonce":1332143088},{"type":"v1","addr":"172.21.15.123:6825","nonce":1332143088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6826","nonce":1332143088},{"type":"v1","addr":"172.21.15.123:6827","nonce":1332143088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6830","nonce":1332143088},{"type":"v1","addr":"172.21.15.123:6831","nonce":1332143088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6828","nonce":1332143088},{"type":"v1","addr":"172.21.15.123:6829","nonce":1332143088}]},"public_addr":"172.21.15.123:6825/1332143088","cluster_addr":"172.21.15.123:6827/1332143088","heartbeat_back_addr":"172.21.15.123:6831/1332143088","heartbeat_front_addr":"172.21.15.123:6829/1332143088","state":["exists","up"]},{"osd":3,"uuid":"c6d1f817-f074-4325-95e1-578eba13edda","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6808","nonce":988586998},{"type":"v1","addr":"172.21.15.123:6809","nonce":988586998}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6810","nonce":988586998},{"type":"v1","addr":"172.21.15.123:6811","nonce":988586998}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6814","nonce":988586998},{"type":"v1","addr":"172.21.15.123:6815","nonce":988586998}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6812","nonce":988586998},{"type":"v1","addr":"172.21.15.123:6813","nonce":988586998}]},"public_addr":"172.21.15.123:6809/988586998","cluster_addr":"172.21.15.123:6811/988586998","heartbeat_back_addr":"172.21.15.123:6815/988586998","heartbeat_front_addr":"172.21.15.123:6813/988586998","state":["exists","up"]},{"osd":4,"uuid":"200a8df4-50ef-40a5-b48a-58981269c086","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6816","nonce":3387813890},{"type":"v1","addr":"172.21.15.184:6817","nonce":3387813890}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6818","nonce":3387813890},{"type":"v1","addr":"172.21.15.184:6819","nonce":3387813890}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6822","nonce":3387813890},{"type":"v1","addr":"172.21.15.184:6823","nonce":3387813890}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6820","nonce":3387813890},{"type":"v1","addr":"172.21.15.184:6821","nonce":3387813890}]},"public_addr":"172.21.15.184:6817/3387813890","cluster_addr":"172.21.15.184:6819/3387813890","heartbeat_back_addr":"172.21.15.184:6823/3387813890","heartbeat_front_addr":"172.21.15.184:6821/3387813890","state":["exists","up"]},{"osd":5,"uuid":"32d5a900-da3d-412c-bc19-13861a2368d8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6808","nonce":2680916017},{"type":"v1","addr":"172.21.15.184:6809","nonce":2680916017}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6810","nonce":2680916017},{"type":"v1","addr":"172.21.15.184:6811","nonce":2680916017}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6814","nonce":2680916017},{"type":"v1","addr":"172.21.15.184:6815","nonce":2680916017}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6812","nonce":2680916017},{"type":"v1","addr":"172.21.15.184:6813","nonce":2680916017}]},"public_addr":"172.21.15.184:6809/2680916017","cluster_addr":"172.21.15.184:6811/2680916017","heartbeat_back_addr":"172.21.15.184:6815/2680916017","heartbeat_front_addr":"172.21.15.184:6813/2680916017","state":["exists","up"]},{"osd":6,"uuid":"cde6f39e-aa1b-4260-ac19-774ee6aaca72","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6824","nonce":2183120457},{"type":"v1","addr":"172.21.15.184:6825","nonce":2183120457}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6826","nonce":2183120457},{"type":"v1","addr":"172.21.15.184:6827","nonce":2183120457}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6830","nonce":2183120457},{"type":"v1","addr":"172.21.15.184:6831","nonce":2183120457}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6828","nonce":2183120457},{"type":"v1","addr":"172.21.15.184:6829","nonce":2183120457}]},"public_addr":"172.21.15.184:6825/2183120457","cluster_addr":"172.21.15.184:6827/2183120457","heartbeat_back_addr":"172.21.15.184:6831/2183120457","heartbeat_front_addr":"172.21.15.184:6829/2183120457","state":["exists","up"]},{"osd":7,"uuid":"4f381778-eee1-47c6-804f-df0e6b33e313","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6800","nonce":4102957285},{"type":"v1","addr":"172.21.15.184:6801","nonce":4102957285}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6802","nonce":4102957285},{"type":"v1","addr":"172.21.15.184:6803","nonce":4102957285}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6806","nonce":4102957285},{"type":"v1","addr":"172.21.15.184:6807","nonce":4102957285}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6804","nonce":4102957285},{"type":"v1","addr":"172.21.15.184:6805","nonce":4102957285}]},"public_addr":"172.21.15.184:6801/4102957285","cluster_addr":"172.21.15.184:6803/4102957285","heartbeat_back_addr":"172.21.15.184:6807/4102957285","heartbeat_front_addr":"172.21.15.184:6805/4102957285","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:35.972210+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.222007+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.198617+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.385663+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.358935+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.156066+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.752973+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:35.682846+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T15:43:02.820 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:02.818+0000 7fa910ff8640 1 -- 172.21.15.123:0/2098123231 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fa8ec072f10 msgr2=0x7fa8ec0753d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:43:02.820 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:02.818+0000 7fa910ff8640 1 --2- 172.21.15.123:0/2098123231 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fa8ec072f10 0x7fa8ec0753d0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fa8fc005fd0 tx=0x7fa8fc005e40 comp rx=0 tx=0).stop 2024-09-16T15:43:02.820 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:02.818+0000 7fa910ff8640 1 -- 172.21.15.123:0/2098123231 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa90c074600 msgr2=0x7fa90c152b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:43:02.820 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:02.818+0000 7fa910ff8640 1 --2- 172.21.15.123:0/2098123231 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa90c074600 0x7fa90c152b80 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fa90000c490 tx=0x7fa90000c950 comp rx=0 tx=0).stop 2024-09-16T15:43:02.821 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:02.818+0000 7fa910ff8640 1 -- 172.21.15.123:0/2098123231 shutdown_connections 2024-09-16T15:43:02.821 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:02.818+0000 7fa910ff8640 1 --2- 172.21.15.123:0/2098123231 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fa8ec072f10 0x7fa8ec0753d0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:02.821 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:02.818+0000 7fa910ff8640 1 --2- 172.21.15.123:0/2098123231 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa90c15bca0 0x7fa90c0cef70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:02.821 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:02.818+0000 7fa910ff8640 1 --2- 172.21.15.123:0/2098123231 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa90c074600 0x7fa90c152b80 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:02.821 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:02.818+0000 7fa910ff8640 1 --2- 172.21.15.123:0/2098123231 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fa90c0726c0 0x7fa90c0c5ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:02.821 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:02.818+0000 7fa910ff8640 1 -- 172.21.15.123:0/2098123231 >> 172.21.15.123:0/2098123231 conn(0x7fa90c0cfb10 msgr2=0x7fa90c0b4650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:43:02.821 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:02.818+0000 7fa910ff8640 1 -- 172.21.15.123:0/2098123231 shutdown_connections 2024-09-16T15:43:02.821 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:02.818+0000 7fa910ff8640 1 -- 172.21.15.123:0/2098123231 wait complete. 2024-09-16T15:43:02.836 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-16T15:43:02.960 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:02.958+0000 7f88aa032640 1 -- 172.21.15.123:0/3572711360 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f88a40cbc60 msgr2=0x7f88a40cd300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:43:02.960 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:02.958+0000 7f88aa032640 1 --2- 172.21.15.123:0/3572711360 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f88a40cbc60 0x7f88a40cd300 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f88a4073b30 tx=0x7f889c01c880 comp rx=0 tx=0).stop 2024-09-16T15:43:02.960 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:02.958+0000 7f88aa032640 1 -- 172.21.15.123:0/3572711360 shutdown_connections 2024-09-16T15:43:02.960 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:02.958+0000 7f88aa032640 1 --2- 172.21.15.123:0/3572711360 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f88a40cbc60 0x7f88a40cd300 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:02.960 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:02.958+0000 7f88aa032640 1 --2- 172.21.15.123:0/3572711360 >> v2:172.21.15.184:3300/0 conn(0x7f88a415f420 0x7f88a40ccdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:02.960 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:02.958+0000 7f88aa032640 1 -- 172.21.15.123:0/3572711360 >> 172.21.15.123:0/3572711360 conn(0x7f88a40732a0 msgr2=0x7f88a4074150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:43:02.960 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:02.958+0000 7f88aa032640 1 -- 172.21.15.123:0/3572711360 shutdown_connections 2024-09-16T15:43:02.960 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:02.958+0000 7f88aa032640 1 -- 172.21.15.123:0/3572711360 wait complete. 2024-09-16T15:43:02.961 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:02.958+0000 7f88aa032640 1 Processor -- start 2024-09-16T15:43:02.961 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:02.958+0000 7f88aa032640 1 -- start start 2024-09-16T15:43:02.962 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:02.958+0000 7f88aa032640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f88a40af3e0 0x7f88a415c250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:43:02.962 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:02.958+0000 7f88aa032640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f88a40cbc60 0x7f88a40c8b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:43:02.962 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:02.958+0000 7f88aa032640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f88a415f420 0x7f88a41554f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:43:02.962 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:02.958+0000 7f88aa032640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f88a40d09e0 con 0x7f88a415f420 2024-09-16T15:43:02.963 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:02.958+0000 7f88aa032640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f88a40d0860 con 0x7f88a40cbc60 2024-09-16T15:43:02.963 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:02.958+0000 7f88aa032640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f88a40d0b60 con 0x7f88a40af3e0 2024-09-16T15:43:02.963 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:02.958+0000 7f88a3fff640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f88a415f420 0x7f88a41554f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:43:02.963 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:02.958+0000 7f88a3fff640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f88a415f420 0x7f88a41554f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:41698/0 (socket says 172.21.15.123:41698) 2024-09-16T15:43:02.963 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:02.958+0000 7f88a3fff640 1 -- 172.21.15.123:0/34014814 learned_addr learned my addr 172.21.15.123:0/34014814 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:43:02.963 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:02.958+0000 7f88a37fe640 1 --2- 172.21.15.123:0/34014814 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f88a40af3e0 0x7f88a415c250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:43:02.963 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:02.958+0000 7f88a2ffd640 1 --2- 172.21.15.123:0/34014814 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f88a40cbc60 0x7f88a40c8b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:43:02.963 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:02.958+0000 7f88a3fff640 1 -- 172.21.15.123:0/34014814 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f88a40af3e0 msgr2=0x7f88a415c250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:43:02.963 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:02.958+0000 7f88a3fff640 1 --2- 172.21.15.123:0/34014814 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f88a40af3e0 0x7f88a415c250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:02.963 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:02.958+0000 7f88a3fff640 1 -- 172.21.15.123:0/34014814 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f88a40cbc60 msgr2=0x7f88a40c8b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:43:02.963 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:02.958+0000 7f88a3fff640 1 --2- 172.21.15.123:0/34014814 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f88a40cbc60 0x7f88a40c8b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:02.963 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:02.958+0000 7f88a3fff640 1 -- 172.21.15.123:0/34014814 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f88a40c9a90 con 0x7f88a415f420 2024-09-16T15:43:02.963 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:02.958+0000 7f88a3fff640 1 --2- 172.21.15.123:0/34014814 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f88a415f420 0x7f88a41554f0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f889c009850 tx=0x7f889c0a56e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:43:02.964 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:02.962+0000 7f888bfff640 1 -- 172.21.15.123:0/34014814 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f889c0b2040 con 0x7f88a415f420 2024-09-16T15:43:02.964 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:02.962+0000 7f888bfff640 1 -- 172.21.15.123:0/34014814 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f889c005030 con 0x7f88a415f420 2024-09-16T15:43:02.964 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:02.962+0000 7f88aa032640 1 -- 172.21.15.123:0/34014814 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f88a40c9d70 con 0x7f88a415f420 2024-09-16T15:43:02.964 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:02.962+0000 7f888bfff640 1 -- 172.21.15.123:0/34014814 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f889c0044f0 con 0x7f88a415f420 2024-09-16T15:43:02.964 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:02.962+0000 7f88aa032640 1 -- 172.21.15.123:0/34014814 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f88a415c790 con 0x7f88a415f420 2024-09-16T15:43:02.965 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:02.962+0000 7f88aa032640 1 -- 172.21.15.123:0/34014814 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f88a40d1110 con 0x7f88a415f420 2024-09-16T15:43:02.965 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:02.962+0000 7f888bfff640 1 -- 172.21.15.123:0/34014814 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 5) ==== 97080+0+0 (secure 0 0 0) 0x7f889c0be060 con 0x7f88a415f420 2024-09-16T15:43:02.965 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:02.962+0000 7f888bfff640 1 --2- 172.21.15.123:0/34014814 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f8890072ec0 0x7f8890075380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:43:02.965 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:02.962+0000 7f888bfff640 1 -- 172.21.15.123:0/34014814 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f889c12d760 con 0x7f88a415f420 2024-09-16T15:43:02.965 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:02.962+0000 7f88a37fe640 1 --2- 172.21.15.123:0/34014814 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f8890072ec0 0x7f8890075380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:43:02.965 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:02.962+0000 7f88a37fe640 1 --2- 172.21.15.123:0/34014814 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f8890072ec0 0x7f8890075380 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f8898005c50 tx=0x7f8898005f50 comp rx=0 tx=0).ready entity=mgr.4135 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:43:02.968 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:02.966+0000 7f888bfff640 1 -- 172.21.15.123:0/34014814 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f889c016470 con 0x7f88a415f420 2024-09-16T15:43:03.177 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:03.174+0000 7f88aa032640 1 -- 172.21.15.123:0/34014814 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f88a40ccba0 con 0x7f88a415f420 2024-09-16T15:43:03.177 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:03.174+0000 7f888bfff640 1 -- 172.21.15.123:0/34014814 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) ==== 93+0+5226 (secure 0 0 0) 0x7f889c0f89b0 con 0x7f88a415f420 2024-09-16T15:43:03.178 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-09-16T15:43:03.178 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":8,"btime":"2024-09-16T15:42:54:421548+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4280,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6835/1497621896","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":1497621896},{"type":"v1","addr":"172.21.15.184:6835","nonce":1497621896}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4305,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6833/2227496057","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6832","nonce":2227496057},{"type":"v1","addr":"172.21.15.123:6833","nonce":2227496057}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4311,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6835/294455998","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6834","nonce":294455998},{"type":"v1","addr":"172.21.15.123:6835","nonce":294455998}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-16T15:42:50.762280+0000","modified":"2024-09-16T15:42:53.789989+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4348},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4348":{"gid":4348,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.184:6837/1469324095","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6836","nonce":1469324095},{"type":"v1","addr":"172.21.15.184:6837","nonce":1469324095}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4348,"qdb_cluster":[4348]},"id":1}]} 2024-09-16T15:43:03.178 INFO:teuthology.orchestra.run.smithi123.stderr:dumped fsmap epoch 8 2024-09-16T15:43:03.180 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:03.178+0000 7f88aa032640 1 -- 172.21.15.123:0/34014814 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f8890072ec0 msgr2=0x7f8890075380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:43:03.180 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:03.178+0000 7f88aa032640 1 --2- 172.21.15.123:0/34014814 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f8890072ec0 0x7f8890075380 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f8898005c50 tx=0x7f8898005f50 comp rx=0 tx=0).stop 2024-09-16T15:43:03.180 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:03.178+0000 7f88aa032640 1 -- 172.21.15.123:0/34014814 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f88a415f420 msgr2=0x7f88a41554f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:43:03.181 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:03.178+0000 7f88aa032640 1 --2- 172.21.15.123:0/34014814 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f88a415f420 0x7f88a41554f0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f889c009850 tx=0x7f889c0a56e0 comp rx=0 tx=0).stop 2024-09-16T15:43:03.181 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:03.178+0000 7f88aa032640 1 -- 172.21.15.123:0/34014814 shutdown_connections 2024-09-16T15:43:03.181 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:03.178+0000 7f88aa032640 1 --2- 172.21.15.123:0/34014814 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f8890072ec0 0x7f8890075380 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:03.181 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:03.178+0000 7f88aa032640 1 --2- 172.21.15.123:0/34014814 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f88a415f420 0x7f88a41554f0 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:03.181 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:03.178+0000 7f88aa032640 1 --2- 172.21.15.123:0/34014814 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f88a40cbc60 0x7f88a40c8b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:03.181 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:03.178+0000 7f88aa032640 1 --2- 172.21.15.123:0/34014814 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f88a40af3e0 0x7f88a415c250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:03.181 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:03.178+0000 7f88aa032640 1 -- 172.21.15.123:0/34014814 >> 172.21.15.123:0/34014814 conn(0x7f88a40732a0 msgr2=0x7f88a4074150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:43:03.181 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:03.178+0000 7f88aa032640 1 -- 172.21.15.123:0/34014814 shutdown_connections 2024-09-16T15:43:03.181 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:03.178+0000 7f88aa032640 1 -- 172.21.15.123:0/34014814 wait complete. 2024-09-16T15:43:03.600 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 8, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-09-16T15:42:50.762280+0000', 'modified': '2024-09-16T15:42:53.789989+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 4348}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4348': {'gid': 4348, 'name': 'd', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.184:6837/1469324095', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.184:6836', 'nonce': 1469324095}, {'type': 'v1', 'addr': '172.21.15.184:6837', 'nonce': 1469324095}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540719139924082687, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4348, 'qdb_cluster': [4348]} 2024-09-16T15:43:03.600 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-09-16T15:43:03.600 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.d status 2024-09-16T15:43:03.740 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:03.738+0000 7f41c94ca640 1 -- 172.21.15.123:0/3959318485 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f41c415bca0 msgr2=0x7f41c40d0630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:43:03.740 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:03.738+0000 7f41c94ca640 1 --2- 172.21.15.123:0/3959318485 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f41c415bca0 0x7f41c40d0630 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f41c406c970 tx=0x7f41bc01c740 comp rx=0 tx=0).stop 2024-09-16T15:43:03.740 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:03.738+0000 7f41c94ca640 1 -- 172.21.15.123:0/3959318485 shutdown_connections 2024-09-16T15:43:03.740 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:03.738+0000 7f41c94ca640 1 --2- 172.21.15.123:0/3959318485 >> v2:172.21.15.184:3300/0 conn(0x7f41c4074600 0x7f41c406e750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:03.740 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:03.738+0000 7f41c94ca640 1 --2- 172.21.15.123:0/3959318485 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f41c415bca0 0x7f41c40d0630 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:03.740 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:03.738+0000 7f41c94ca640 1 -- 172.21.15.123:0/3959318485 >> 172.21.15.123:0/3959318485 conn(0x7f41c40cfb10 msgr2=0x7f41c40cff20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:43:03.740 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:03.738+0000 7f41c94ca640 1 -- 172.21.15.123:0/3959318485 shutdown_connections 2024-09-16T15:43:03.740 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:03.738+0000 7f41c94ca640 1 -- 172.21.15.123:0/3959318485 wait complete. 2024-09-16T15:43:03.741 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:03.738+0000 7f41c94ca640 1 Processor -- start 2024-09-16T15:43:03.741 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:03.738+0000 7f41c94ca640 1 -- start start 2024-09-16T15:43:03.743 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:03.738+0000 7f41c94ca640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f41c40726c0 0x7f41c41f62a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:43:03.743 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:03.738+0000 7f41c94ca640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f41c4074600 0x7f41c40c8f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:43:03.743 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:03.738+0000 7f41c94ca640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f41c415bca0 0x7f41c40c7e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:43:03.743 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:03.738+0000 7f41c94ca640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f41c4168bd0 con 0x7f41c415bca0 2024-09-16T15:43:03.743 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:03.738+0000 7f41c94ca640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f41c4168a50 con 0x7f41c40726c0 2024-09-16T15:43:03.743 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:03.738+0000 7f41c94ca640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f41c4168d50 con 0x7f41c4074600 2024-09-16T15:43:03.743 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:03.738+0000 7f41c37fe640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f41c415bca0 0x7f41c40c7e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:43:03.743 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:03.738+0000 7f41c37fe640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f41c415bca0 0x7f41c40c7e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:41718/0 (socket says 172.21.15.123:41718) 2024-09-16T15:43:03.743 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:03.738+0000 7f41c37fe640 1 -- 172.21.15.123:0/2570744096 learned_addr learned my addr 172.21.15.123:0/2570744096 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:43:03.744 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:03.738+0000 7f41c27fc640 1 --2- 172.21.15.123:0/2570744096 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f41c4074600 0x7f41c40c8f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:43:03.744 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:03.738+0000 7f41c37fe640 1 -- 172.21.15.123:0/2570744096 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f41c4074600 msgr2=0x7f41c40c8f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:43:03.744 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:03.738+0000 7f41c37fe640 1 --2- 172.21.15.123:0/2570744096 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f41c4074600 0x7f41c40c8f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:03.744 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:03.738+0000 7f41c37fe640 1 -- 172.21.15.123:0/2570744096 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f41c40726c0 msgr2=0x7f41c41f62a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:43:03.744 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:03.738+0000 7f41c37fe640 1 --2- 172.21.15.123:0/2570744096 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f41c40726c0 0x7f41c41f62a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:03.744 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:03.738+0000 7f41c37fe640 1 -- 172.21.15.123:0/2570744096 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f41c415efb0 con 0x7f41c415bca0 2024-09-16T15:43:03.744 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:03.742+0000 7f41c37fe640 1 --2- 172.21.15.123:0/2570744096 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f41c415bca0 0x7f41c40c7e20 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f41b400cce0 tx=0x7f41b400a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:43:03.744 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:03.742+0000 7f41bb7fe640 1 -- 172.21.15.123:0/2570744096 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41b400f450 con 0x7f41c415bca0 2024-09-16T15:43:03.744 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:03.742+0000 7f41bb7fe640 1 -- 172.21.15.123:0/2570744096 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f41b400f700 con 0x7f41c415bca0 2024-09-16T15:43:03.744 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:03.742+0000 7f41c94ca640 1 -- 172.21.15.123:0/2570744096 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f41c4157bb0 con 0x7f41c415bca0 2024-09-16T15:43:03.744 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:03.742+0000 7f41bb7fe640 1 -- 172.21.15.123:0/2570744096 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41b400fb50 con 0x7f41c415bca0 2024-09-16T15:43:03.744 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:03.742+0000 7f41c94ca640 1 -- 172.21.15.123:0/2570744096 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f41c4158170 con 0x7f41c415bca0 2024-09-16T15:43:03.745 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:03.742+0000 7f41bb7fe640 1 -- 172.21.15.123:0/2570744096 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 5) ==== 97080+0+0 (secure 0 0 0) 0x7f41b400fcf0 con 0x7f41c415bca0 2024-09-16T15:43:03.745 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:03.742+0000 7f41bb7fe640 1 --2- 172.21.15.123:0/2570744096 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f41a4072ec0 0x7f41a4075380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:43:03.745 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:03.742+0000 7f41bb7fe640 1 -- 172.21.15.123:0/2570744096 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f41b40949d0 con 0x7f41c415bca0 2024-09-16T15:43:03.746 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:03.742+0000 7f41c2ffd640 1 --2- 172.21.15.123:0/2570744096 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f41a4072ec0 0x7f41a4075380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:43:03.746 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:03.742+0000 7f41cc5b4000 1 Processor -- start 2024-09-16T15:43:03.746 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:03.742+0000 7f41cc5b4000 1 -- start start 2024-09-16T15:43:03.748 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:03.742+0000 7f41c2ffd640 1 --2- 172.21.15.123:0/2570744096 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f41a4072ec0 0x7f41a4075380 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f41bc0af000 tx=0x7f41bc0194e0 comp rx=0 tx=0).ready entity=mgr.4135 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:43:03.748 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:03.746+0000 7f41cc5b4000 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558221b099d0 0x558221b09db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:43:03.748 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:03.746+0000 7f41cc5b4000 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x558221b0a2f0 0x558221abd8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:43:03.748 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:03.746+0000 7f41cc5b4000 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x558221abde10 0x558221abe1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:43:03.748 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:03.746+0000 7f41cc5b4000 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x5582219de3f0 con 0x558221abde10 2024-09-16T15:43:03.749 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:03.746+0000 7f41cc5b4000 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x55822191bb20 con 0x558221b099d0 2024-09-16T15:43:03.749 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:03.746+0000 7f41cc5b4000 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x55822190af50 con 0x558221b0a2f0 2024-09-16T15:43:03.749 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:03.746+0000 7f41c37fe640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x558221abde10 0x558221abe1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:43:03.749 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:03.746+0000 7f41c37fe640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x558221abde10 0x558221abe1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:41734/0 (socket says 172.21.15.123:41734) 2024-09-16T15:43:03.749 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:03.746+0000 7f41c37fe640 1 -- 172.21.15.123:0/4003375094 learned_addr learned my addr 172.21.15.123:0/4003375094 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:43:03.749 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:03.746+0000 7f41c2ffd640 1 --2- 172.21.15.123:0/4003375094 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558221b099d0 0x558221b09db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:43:03.749 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:03.746+0000 7f41c27fc640 1 --2- 172.21.15.123:0/4003375094 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x558221b0a2f0 0x558221abd8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:43:03.749 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:03.746+0000 7f41c37fe640 1 -- 172.21.15.123:0/4003375094 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x558221b0a2f0 msgr2=0x558221abd8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T15:43:03.749 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:03.746+0000 7f41c37fe640 1 --2- 172.21.15.123:0/4003375094 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x558221b0a2f0 0x558221abd8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:03.749 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:03.746+0000 7f41c37fe640 1 -- 172.21.15.123:0/4003375094 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558221b099d0 msgr2=0x558221b09db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T15:43:03.749 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:03.746+0000 7f41c37fe640 1 --2- 172.21.15.123:0/4003375094 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558221b099d0 0x558221b09db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:03.749 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:03.746+0000 7f41c37fe640 1 -- 172.21.15.123:0/4003375094 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5582218d2590 con 0x558221abde10 2024-09-16T15:43:03.749 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:03.746+0000 7f41c27fc640 1 --2- 172.21.15.123:0/4003375094 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x558221b0a2f0 0x558221abd8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T15:43:03.750 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:03.746+0000 7f41c37fe640 1 --2- 172.21.15.123:0/4003375094 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x558221abde10 0x558221abe1f0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f41b4092f20 tx=0x7f41b405f030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f81e423bb4a7a9fd server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:43:03.750 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:03.746+0000 7f41b8ff9640 1 -- 172.21.15.123:0/4003375094 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41b4084070 con 0x558221abde10 2024-09-16T15:43:03.750 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:03.746+0000 7f41b8ff9640 1 -- 172.21.15.123:0/4003375094 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f41b4064000 con 0x558221abde10 2024-09-16T15:43:03.750 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:03.746+0000 7f41b8ff9640 1 -- 172.21.15.123:0/4003375094 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41b4068c40 con 0x558221abde10 2024-09-16T15:43:03.750 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:03.746+0000 7f41cc5b4000 1 -- 172.21.15.123:0/4003375094 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x558221abde10 msgr2=0x558221abe1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:43:03.750 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:03.746+0000 7f41cc5b4000 1 --2- 172.21.15.123:0/4003375094 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x558221abde10 0x558221abe1f0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f41b4092f20 tx=0x7f41b405f030 comp rx=0 tx=0).stop 2024-09-16T15:43:03.750 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:03.746+0000 7f41cc5b4000 1 -- 172.21.15.123:0/4003375094 shutdown_connections 2024-09-16T15:43:03.750 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:03.746+0000 7f41cc5b4000 1 --2- 172.21.15.123:0/4003375094 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x558221abde10 0x558221abe1f0 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:03.750 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:03.746+0000 7f41cc5b4000 1 --2- 172.21.15.123:0/4003375094 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x558221b0a2f0 0x558221abd8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:03.750 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:03.746+0000 7f41cc5b4000 1 --2- 172.21.15.123:0/4003375094 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558221b099d0 0x558221b09db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:03.750 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:03.746+0000 7f41cc5b4000 1 -- 172.21.15.123:0/4003375094 >> 172.21.15.123:0/4003375094 conn(0x5582219a7130 msgr2=0x558221aece80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:43:03.750 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:03.746+0000 7f41cc5b4000 1 -- 172.21.15.123:0/4003375094 shutdown_connections 2024-09-16T15:43:03.750 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:03.746+0000 7f41cc5b4000 1 -- 172.21.15.123:0/4003375094 wait complete. 2024-09-16T15:43:03.753 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:03.746+0000 7f41cc5b4000 1 -- start start 2024-09-16T15:43:03.753 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:03.746+0000 7f41cc5b4000 1 Processor -- start 2024-09-16T15:43:03.753 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:03.746+0000 7f41cc5b4000 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x558221abde10 0x558221aea6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:43:03.753 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:03.746+0000 7f41cc5b4000 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558221b099d0 0x558221aeae70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:43:03.753 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:03.746+0000 7f41cc5b4000 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x558221b0a2f0 0x558221aeb3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:43:03.753 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:03.746+0000 7f41cc5b4000 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x5582218a55d0 con 0x558221b0a2f0 2024-09-16T15:43:03.754 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:03.746+0000 7f41cc5b4000 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x55822190ed60 con 0x558221b099d0 2024-09-16T15:43:03.754 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:03.746+0000 7f41cc5b4000 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x5582218b1500 con 0x558221abde10 2024-09-16T15:43:03.754 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:03.746+0000 7f41cc5b4000 10 client.0 _ll_register_callbacks cb 0x5582218b5810 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T15:43:03.754 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:03.746+0000 7f41c37fe640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x558221b0a2f0 0x558221aeb3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:43:03.754 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:03.746+0000 7f41c37fe640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x558221b0a2f0 0x558221aeb3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:41736/0 (socket says 172.21.15.123:41736) 2024-09-16T15:43:03.754 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:03.746+0000 7f41c37fe640 1 -- 172.21.15.123:0/2780183151 learned_addr learned my addr 172.21.15.123:0/2780183151 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:43:03.754 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:03.746+0000 7f41c2ffd640 1 --2- 172.21.15.123:0/2780183151 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x558221abde10 0x558221aea6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:43:03.754 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:03.746+0000 7f41c37fe640 1 -- 172.21.15.123:0/2780183151 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x558221abde10 msgr2=0x558221aea6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T15:43:03.754 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:03.746+0000 7f41c37fe640 1 --2- 172.21.15.123:0/2780183151 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x558221abde10 0x558221aea6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:03.754 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:03.746+0000 7f41c37fe640 1 -- 172.21.15.123:0/2780183151 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558221b099d0 msgr2=0x558221aeae70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T15:43:03.754 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:03.746+0000 7f41c37fe640 1 --2- 172.21.15.123:0/2780183151 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558221b099d0 0x558221aeae70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:03.754 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:03.746+0000 7f41c37fe640 1 -- 172.21.15.123:0/2780183151 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5582218a9170 con 0x558221b0a2f0 2024-09-16T15:43:03.754 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:03.746+0000 7f41c37fe640 1 --2- 172.21.15.123:0/2780183151 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x558221b0a2f0 0x558221aeb3b0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f41b405bce0 tx=0x7f41b4068cd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2c61d881e613cc67 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:43:03.754 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:03.746+0000 7f41a1ffb640 10 client.0 ms_handle_connect on v2:172.21.15.123:3300/0 2024-09-16T15:43:03.755 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:03.746+0000 7f41a1ffb640 1 -- 172.21.15.123:0/2780183151 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41b4079020 con 0x558221b0a2f0 2024-09-16T15:43:03.755 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:03.746+0000 7f41a1ffb640 1 -- 172.21.15.123:0/2780183151 <== mon.0 v2:172.21.15.123:3300/0 2 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f41b4066200 con 0x558221b0a2f0 2024-09-16T15:43:03.755 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:03.746+0000 7f41cc5b4000 1 -- 172.21.15.123:0/2780183151 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x5582218a30f0 con 0x558221b0a2f0 2024-09-16T15:43:03.755 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:03.750+0000 7f41a1ffb640 1 -- 172.21.15.123:0/2780183151 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_get_version_reply(handle=1 version=8) ==== 24+0+0 (secure 0 0 0) 0x7f41b4084040 con 0x558221b0a2f0 2024-09-16T15:43:03.755 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:03.750+0000 7f41cc5b4000 10 client.4641 fetch_fsmap learned FSMap version 8 2024-09-16T15:43:03.755 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:03.750+0000 7f41cc5b4000 1 -- 172.21.15.123:0/2780183151 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,fsmap=8,monmap=2+}) -- 0x55822198fef0 con 0x558221b0a2f0 2024-09-16T15:43:03.755 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:03.750+0000 7f41a1ffb640 1 -- 172.21.15.123:0/2780183151 <== mon.0 v2:172.21.15.123:3300/0 4 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x5582218a30f0 con 0x558221b0a2f0 2024-09-16T15:43:03.755 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:03.750+0000 7f41a1ffb640 1 -- 172.21.15.123:0/2780183151 <== mon.0 v2:172.21.15.123:3300/0 5 ==== fsmap(e 8) ==== 3502+0+0 (secure 0 0 0) 0x7f41b4068040 con 0x558221b0a2f0 2024-09-16T15:43:03.755 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:03.750+0000 7f41cc5b4000 10 client.4641 fetch_fsmap finished waiting for FSMap version 8 2024-09-16T15:43:03.755 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:03.750+0000 7f41cc5b4000 10 client.4641 resolve_mds: resolved name 'd' to daemon mds.d 2024-09-16T15:43:03.755 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:03.750+0000 7f41cc5b4000 20 client.4641 populate_metadata read hostname 'smithi123' 2024-09-16T15:43:03.755 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:03.750+0000 7f41cc5b4000 1 --2- 172.21.15.123:0/2780183151 >> [v2:172.21.15.184:6836/1469324095,v1:172.21.15.184:6837/1469324095] conn(0x558221aec0f0 0x558221b1bc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:43:03.755 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:03.750+0000 7f41cc5b4000 4 client.4641 mds_command: new command op to 4348 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-16T15:43:03.755 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:03.750+0000 7f41cc5b4000 1 -- 172.21.15.123:0/2780183151 --> [v2:172.21.15.184:6836/1469324095,v1:172.21.15.184:6837/1469324095] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55822191da70 con 0x558221aec0f0 2024-09-16T15:43:03.755 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:03.750+0000 7f41c2ffd640 1 --2- 172.21.15.123:0/2780183151 >> [v2:172.21.15.184:6836/1469324095,v1:172.21.15.184:6837/1469324095] conn(0x558221aec0f0 0x558221b1bc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:43:03.756 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:03.750+0000 7f41a1ffb640 10 client.4641 ms_handle_connect on v2:172.21.15.184:6836/1469324095 2024-09-16T15:43:03.756 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:03.750+0000 7f41c2ffd640 1 --2- 172.21.15.123:0/2780183151 >> [v2:172.21.15.184:6836/1469324095,v1:172.21.15.184:6837/1469324095] conn(0x558221aec0f0 0x558221b1bc90 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ff2b03694531d0df server_cookie=5f53e446fc11b9e7 in_seq=0 out_seq=0 2024-09-16T15:43:03.757 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:03.750+0000 7f41a1ffb640 1 -- 172.21.15.123:0/2780183151 <== mds.0 v2:172.21.15.184:6836/1469324095 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x55822191da70 con 0x558221aec0f0 2024-09-16T15:43:03.757 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:03.750+0000 7f41a1ffb640 10 client.4641 handle_command_reply: tid=0 2024-09-16T15:43:03.757 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:03.750+0000 7f41cc5b4000 1 client.4641 shutdown 2024-09-16T15:43:03.757 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:03.750+0000 7f41cc5b4000 1 -- 172.21.15.123:0/2780183151 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x558221b0a2f0 msgr2=0x558221aeb3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:43:03.757 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:03.750+0000 7f41cc5b4000 1 --2- 172.21.15.123:0/2780183151 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x558221b0a2f0 0x558221aeb3b0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f41b405bce0 tx=0x7f41b4068cd0 comp rx=0 tx=0).stop 2024-09-16T15:43:03.757 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:03.750+0000 7f41cc5b4000 1 -- 172.21.15.123:0/2780183151 shutdown_connections 2024-09-16T15:43:03.757 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:03.750+0000 7f41cc5b4000 1 --2- 172.21.15.123:0/2780183151 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x558221b0a2f0 0x558221aeb3b0 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:03.757 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:03.750+0000 7f41cc5b4000 1 --2- 172.21.15.123:0/2780183151 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558221b099d0 0x558221aeae70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:03.757 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:03.750+0000 7f41cc5b4000 1 --2- 172.21.15.123:0/2780183151 >> [v2:172.21.15.184:6836/1469324095,v1:172.21.15.184:6837/1469324095] conn(0x558221aec0f0 0x558221b1bc90 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:03.757 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:03.750+0000 7f41cc5b4000 1 --2- 172.21.15.123:0/2780183151 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x558221abde10 0x558221aea6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:03.757 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:03.750+0000 7f41a1ffb640 0 client.4641 ms_handle_reset on v2:172.21.15.184:6836/1469324095 2024-09-16T15:43:03.757 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:03.750+0000 7f41cc5b4000 1 -- 172.21.15.123:0/2780183151 >> 172.21.15.123:0/2780183151 conn(0x5582219a7130 msgr2=0x558221abcfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:43:03.758 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:03.750+0000 7f41cc5b4000 1 -- 172.21.15.123:0/2780183151 shutdown_connections 2024-09-16T15:43:03.758 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:03.750+0000 7f41cc5b4000 1 -- 172.21.15.123:0/2780183151 wait complete. 2024-09-16T15:43:03.758 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:03.754+0000 7f41cc5b4000 20 client.4641 trim_cache size 0 max 16384 2024-09-16T15:43:03.773 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:03.770+0000 7f41cc5b4000 1 Processor -- start 2024-09-16T15:43:03.773 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:03.770+0000 7f41cc5b4000 1 -- start start 2024-09-16T15:43:03.774 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:03.770+0000 7f41cc5b4000 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x558221abde10 0x558221b847f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:43:03.775 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:03.770+0000 7f41cc5b4000 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x558221b099d0 0x558221b84d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:43:03.775 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:03.770+0000 7f41cc5b4000 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558221b0a2f0 0x558221b85270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:43:03.775 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:03.770+0000 7f41cc5b4000 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f41c4201ae0 con 0x558221abde10 2024-09-16T15:43:03.775 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:03.770+0000 7f41cc5b4000 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f41c4200b40 con 0x558221b0a2f0 2024-09-16T15:43:03.775 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:03.770+0000 7f41cc5b4000 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f41c41f9d20 con 0x558221b099d0 2024-09-16T15:43:03.775 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:03.770+0000 7f41c2ffd640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x558221abde10 0x558221b847f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:43:03.775 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:03.770+0000 7f41c2ffd640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x558221abde10 0x558221b847f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:41742/0 (socket says 172.21.15.123:41742) 2024-09-16T15:43:03.775 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:03.770+0000 7f41c2ffd640 1 -- 172.21.15.123:0/4008538402 learned_addr learned my addr 172.21.15.123:0/4008538402 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:43:03.775 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:03.770+0000 7f41c2ffd640 1 -- 172.21.15.123:0/4008538402 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x558221b099d0 msgr2=0x558221b84d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T15:43:03.775 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:03.770+0000 7f41c2ffd640 1 --2- 172.21.15.123:0/4008538402 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x558221b099d0 0x558221b84d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:03.775 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:03.770+0000 7f41c2ffd640 1 -- 172.21.15.123:0/4008538402 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558221b0a2f0 msgr2=0x558221b85270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T15:43:03.775 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:03.770+0000 7f41c2ffd640 1 --2- 172.21.15.123:0/4008538402 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558221b0a2f0 0x558221b85270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:03.776 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:03.770+0000 7f41c2ffd640 1 -- 172.21.15.123:0/4008538402 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558221966d90 con 0x558221abde10 2024-09-16T15:43:03.776 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:03.774+0000 7f41c2ffd640 1 --2- 172.21.15.123:0/4008538402 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x558221abde10 0x558221b847f0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f41bc01a370 tx=0x7f41bc0b7000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b950cc4be9e8bd3f server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:43:03.776 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:03.774+0000 7f41a3fff640 1 -- 172.21.15.123:0/4008538402 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41bc0c1040 con 0x558221abde10 2024-09-16T15:43:03.776 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:03.774+0000 7f41a3fff640 1 -- 172.21.15.123:0/4008538402 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f41bc0d2070 con 0x558221abde10 2024-09-16T15:43:03.776 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:03.774+0000 7f41a3fff640 1 -- 172.21.15.123:0/4008538402 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41bc0b2040 con 0x558221abde10 2024-09-16T15:43:03.780 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:03.774+0000 7f41cc5b4000 1 -- 172.21.15.123:0/4008538402 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x558221abde10 msgr2=0x558221b847f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:43:03.780 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:03.774+0000 7f41cc5b4000 1 --2- 172.21.15.123:0/4008538402 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x558221abde10 0x558221b847f0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f41bc01a370 tx=0x7f41bc0b7000 comp rx=0 tx=0).stop 2024-09-16T15:43:03.780 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:03.774+0000 7f41cc5b4000 1 -- 172.21.15.123:0/4008538402 shutdown_connections 2024-09-16T15:43:03.780 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:03.774+0000 7f41cc5b4000 1 --2- 172.21.15.123:0/4008538402 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558221b0a2f0 0x558221b85270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:03.780 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:03.774+0000 7f41cc5b4000 1 --2- 172.21.15.123:0/4008538402 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x558221b099d0 0x558221b84d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:03.780 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:03.774+0000 7f41cc5b4000 1 --2- 172.21.15.123:0/4008538402 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x558221abde10 0x558221b847f0 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:03.780 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:03.774+0000 7f41cc5b4000 1 -- 172.21.15.123:0/4008538402 >> 172.21.15.123:0/4008538402 conn(0x5582219a7130 msgr2=0x558221ac5a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:43:03.781 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:03.774+0000 7f41cc5b4000 1 -- 172.21.15.123:0/4008538402 shutdown_connections 2024-09-16T15:43:03.781 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:03.774+0000 7f41cc5b4000 1 -- 172.21.15.123:0/4008538402 wait complete. 2024-09-16T15:43:03.781 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:03.774+0000 7f41cc5b4000 1 -- start start 2024-09-16T15:43:03.781 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:03.774+0000 7f41cc5b4000 1 Processor -- start 2024-09-16T15:43:03.781 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:03.774+0000 7f41cc5b4000 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x558221abde10 0x558221b88830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:43:03.781 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:03.774+0000 7f41cc5b4000 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558221b099d0 0x558221b88d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:43:03.781 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:03.774+0000 7f41cc5b4000 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x558221b0a2f0 0x558221b892b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:43:03.781 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:03.774+0000 7f41cc5b4000 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x5582218d6ec0 con 0x558221b0a2f0 2024-09-16T15:43:03.781 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:03.774+0000 7f41cc5b4000 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f41c42004f0 con 0x558221b099d0 2024-09-16T15:43:03.781 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:03.774+0000 7f41cc5b4000 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x5582218a55d0 con 0x558221abde10 2024-09-16T15:43:03.781 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:03.774+0000 7f41cc5b4000 10 client.0 _ll_register_callbacks cb 0x7f41b404b070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T15:43:03.781 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:03.774+0000 7f41c37fe640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x558221b0a2f0 0x558221b892b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:43:03.781 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:03.774+0000 7f41c37fe640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x558221b0a2f0 0x558221b892b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:41756/0 (socket says 172.21.15.123:41756) 2024-09-16T15:43:03.782 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:03.774+0000 7f41c37fe640 1 -- 172.21.15.123:0/1360842805 learned_addr learned my addr 172.21.15.123:0/1360842805 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:43:03.782 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:03.774+0000 7f41c37fe640 1 -- 172.21.15.123:0/1360842805 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x558221abde10 msgr2=0x558221b88830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T15:43:03.782 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:03.774+0000 7f41c37fe640 1 --2- 172.21.15.123:0/1360842805 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x558221abde10 0x558221b88830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:03.782 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:03.774+0000 7f41c37fe640 1 -- 172.21.15.123:0/1360842805 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558221b099d0 msgr2=0x558221b88d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T15:43:03.782 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:03.774+0000 7f41c37fe640 1 --2- 172.21.15.123:0/1360842805 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558221b099d0 0x558221b88d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:03.782 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:03.774+0000 7f41c37fe640 1 -- 172.21.15.123:0/1360842805 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5582218d1a90 con 0x558221b0a2f0 2024-09-16T15:43:03.782 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:03.774+0000 7f41c37fe640 1 --2- 172.21.15.123:0/1360842805 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x558221b0a2f0 0x558221b892b0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f41b405f4b0 tx=0x7f41b4066430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6f9a7c23502fc77c server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:43:03.782 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:03.774+0000 7f41a1ffb640 10 client.0 ms_handle_connect on v2:172.21.15.123:3300/0 2024-09-16T15:43:03.782 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:03.774+0000 7f41a1ffb640 1 -- 172.21.15.123:0/1360842805 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41b405b3c0 con 0x558221b0a2f0 2024-09-16T15:43:03.782 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:03.774+0000 7f41a1ffb640 1 -- 172.21.15.123:0/1360842805 <== mon.0 v2:172.21.15.123:3300/0 2 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f41b405e400 con 0x558221b0a2f0 2024-09-16T15:43:03.782 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:03.774+0000 7f41cc5b4000 1 -- 172.21.15.123:0/1360842805 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f41bc0d2070 con 0x558221b0a2f0 2024-09-16T15:43:03.782 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:03.774+0000 7f41a1ffb640 1 -- 172.21.15.123:0/1360842805 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_get_version_reply(handle=1 version=8) ==== 24+0+0 (secure 0 0 0) 0x7f41b4075050 con 0x558221b0a2f0 2024-09-16T15:43:03.782 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:03.774+0000 7f41cc5b4000 10 client.4647 fetch_fsmap learned FSMap version 8 2024-09-16T15:43:03.783 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:03.774+0000 7f41cc5b4000 1 -- 172.21.15.123:0/1360842805 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,fsmap=8,monmap=2+}) -- 0x5582218b0050 con 0x558221b0a2f0 2024-09-16T15:43:03.783 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:03.774+0000 7f41a1ffb640 1 -- 172.21.15.123:0/1360842805 <== mon.0 v2:172.21.15.123:3300/0 4 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f41bc0d2070 con 0x558221b0a2f0 2024-09-16T15:43:03.783 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:03.774+0000 7f41a1ffb640 1 -- 172.21.15.123:0/1360842805 <== mon.0 v2:172.21.15.123:3300/0 5 ==== fsmap(e 8) ==== 3502+0+0 (secure 0 0 0) 0x7f41b4068040 con 0x558221b0a2f0 2024-09-16T15:43:03.783 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:03.774+0000 7f41cc5b4000 10 client.4647 fetch_fsmap finished waiting for FSMap version 8 2024-09-16T15:43:03.783 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:03.774+0000 7f41cc5b4000 10 client.4647 resolve_mds: resolved name 'd' to daemon mds.d 2024-09-16T15:43:03.783 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:03.774+0000 7f41cc5b4000 20 client.4647 populate_metadata read hostname 'smithi123' 2024-09-16T15:43:03.783 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:03.774+0000 7f41cc5b4000 1 --2- 172.21.15.123:0/1360842805 >> [v2:172.21.15.184:6836/1469324095,v1:172.21.15.184:6837/1469324095] conn(0x558221b1e450 0x558221b1e830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:43:03.783 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:03.774+0000 7f41cc5b4000 4 client.4647 mds_command: new command op to 4348 tid=0 multi_id=0 [{"prefix": "status"}] 2024-09-16T15:43:03.783 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:03.774+0000 7f41cc5b4000 1 -- 172.21.15.123:0/1360842805 --> [v2:172.21.15.184:6836/1469324095,v1:172.21.15.184:6837/1469324095] -- command(tid 0: {"prefix": "status"}) -- 0x558221919520 con 0x558221b1e450 2024-09-16T15:43:03.783 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:03.774+0000 7f41c2ffd640 1 --2- 172.21.15.123:0/1360842805 >> [v2:172.21.15.184:6836/1469324095,v1:172.21.15.184:6837/1469324095] conn(0x558221b1e450 0x558221b1e830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:43:03.783 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:03.778+0000 7f41a1ffb640 10 client.4647 ms_handle_connect on v2:172.21.15.184:6836/1469324095 2024-09-16T15:43:03.784 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:03.778+0000 7f41c2ffd640 1 --2- 172.21.15.123:0/1360842805 >> [v2:172.21.15.184:6836/1469324095,v1:172.21.15.184:6837/1469324095] conn(0x558221b1e450 0x558221b1e830 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f7f974c854e22bf4 server_cookie=c3b443bcd0c86338 in_seq=0 out_seq=0 2024-09-16T15:43:03.784 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:03.778+0000 7f41a1ffb640 1 -- 172.21.15.123:0/1360842805 <== mds.0 v2:172.21.15.184:6836/1469324095 1 ==== command_reply(tid 0: 0 ) ==== 8+0+324 (crc 0 0 0) 0x558221919520 con 0x558221b1e450 2024-09-16T15:43:03.784 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:03.778+0000 7f41a1ffb640 10 client.4647 handle_command_reply: tid=0 2024-09-16T15:43:03.784 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:03.778+0000 7f41cc5b4000 1 client.4647 shutdown 2024-09-16T15:43:03.784 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:03.778+0000 7f41cc5b4000 1 -- 172.21.15.123:0/1360842805 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x558221b0a2f0 msgr2=0x558221b892b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:43:03.784 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:03.778+0000 7f41cc5b4000 1 --2- 172.21.15.123:0/1360842805 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x558221b0a2f0 0x558221b892b0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f41b405f4b0 tx=0x7f41b4066430 comp rx=0 tx=0).stop 2024-09-16T15:43:03.784 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:03.778+0000 7f41cc5b4000 1 -- 172.21.15.123:0/1360842805 shutdown_connections 2024-09-16T15:43:03.784 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:03.778+0000 7f41cc5b4000 1 --2- 172.21.15.123:0/1360842805 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x558221b0a2f0 0x558221b892b0 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:03.784 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:03.778+0000 7f41cc5b4000 1 --2- 172.21.15.123:0/1360842805 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x558221b099d0 0x558221b88d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:03.784 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:03.778+0000 7f41cc5b4000 1 --2- 172.21.15.123:0/1360842805 >> [v2:172.21.15.184:6836/1469324095,v1:172.21.15.184:6837/1469324095] conn(0x558221b1e450 0x558221b1e830 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:03.784 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:03.778+0000 7f41cc5b4000 1 --2- 172.21.15.123:0/1360842805 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x558221abde10 0x558221b88830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:03.784 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:03.778+0000 7f41a1ffb640 0 client.4647 ms_handle_reset on v2:172.21.15.184:6836/1469324095 2024-09-16T15:43:03.785 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:03.778+0000 7f41cc5b4000 1 -- 172.21.15.123:0/1360842805 >> 172.21.15.123:0/1360842805 conn(0x5582219a7130 msgr2=0x558221b085b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:43:03.785 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:03.778+0000 7f41cc5b4000 1 -- 172.21.15.123:0/1360842805 shutdown_connections 2024-09-16T15:43:03.785 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:03.778+0000 7f41cc5b4000 1 -- 172.21.15.123:0/1360842805 wait complete. 2024-09-16T15:43:03.785 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:03.778+0000 7f41cc5b4000 20 client.4647 trim_cache size 0 max 16384 2024-09-16T15:43:03.785 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2024-09-16T15:43:03.785 INFO:teuthology.orchestra.run.smithi123.stdout: "cluster_fsid": "56077f89-18c8-486f-ae7e-eecc0bb7822c", 2024-09-16T15:43:03.785 INFO:teuthology.orchestra.run.smithi123.stdout: "whoami": 0, 2024-09-16T15:43:03.785 INFO:teuthology.orchestra.run.smithi123.stdout: "id": 4348, 2024-09-16T15:43:03.785 INFO:teuthology.orchestra.run.smithi123.stdout: "want_state": "up:active", 2024-09-16T15:43:03.786 INFO:teuthology.orchestra.run.smithi123.stdout: "state": "up:active", 2024-09-16T15:43:03.786 INFO:teuthology.orchestra.run.smithi123.stdout: "fs_name": "cephfs", 2024-09-16T15:43:03.786 INFO:teuthology.orchestra.run.smithi123.stdout: "rank_uptime": 12.374769132000001, 2024-09-16T15:43:03.786 INFO:teuthology.orchestra.run.smithi123.stdout: "mdsmap_epoch": 8, 2024-09-16T15:43:03.786 INFO:teuthology.orchestra.run.smithi123.stdout: "osdmap_epoch": 21, 2024-09-16T15:43:03.786 INFO:teuthology.orchestra.run.smithi123.stdout: "osdmap_epoch_barrier": 21, 2024-09-16T15:43:03.786 INFO:teuthology.orchestra.run.smithi123.stdout: "uptime": 18.472074482 2024-09-16T15:43:03.786 INFO:teuthology.orchestra.run.smithi123.stdout:} 2024-09-16T15:43:03.787 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:03.778+0000 7f41c94ca640 1 -- 172.21.15.123:0/2570744096 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f41a4072ec0 msgr2=0x7f41a4075380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:43:03.787 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:03.778+0000 7f41c94ca640 1 --2- 172.21.15.123:0/2570744096 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f41a4072ec0 0x7f41a4075380 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f41bc0af000 tx=0x7f41bc0194e0 comp rx=0 tx=0).stop 2024-09-16T15:43:03.787 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:03.778+0000 7f41c94ca640 1 -- 172.21.15.123:0/2570744096 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f41c415bca0 msgr2=0x7f41c40c7e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:43:03.787 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:03.778+0000 7f41c94ca640 1 --2- 172.21.15.123:0/2570744096 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f41c415bca0 0x7f41c40c7e20 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f41b400cce0 tx=0x7f41b400a620 comp rx=0 tx=0).stop 2024-09-16T15:43:03.787 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:03.778+0000 7f41c94ca640 1 -- 172.21.15.123:0/2570744096 shutdown_connections 2024-09-16T15:43:03.787 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:03.778+0000 7f41c94ca640 1 --2- 172.21.15.123:0/2570744096 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f41a4072ec0 0x7f41a4075380 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:03.787 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:03.778+0000 7f41c94ca640 1 --2- 172.21.15.123:0/2570744096 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f41c415bca0 0x7f41c40c7e20 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:03.787 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:03.778+0000 7f41c94ca640 1 --2- 172.21.15.123:0/2570744096 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f41c4074600 0x7f41c40c8f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:03.787 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:03.778+0000 7f41c94ca640 1 --2- 172.21.15.123:0/2570744096 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f41c40726c0 0x7f41c41f62a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:03.787 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:03.778+0000 7f41c94ca640 1 -- 172.21.15.123:0/2570744096 >> 172.21.15.123:0/2570744096 conn(0x7f41c40cfb10 msgr2=0x7f41c4075960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:43:03.788 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:03.778+0000 7f41c94ca640 1 -- 172.21.15.123:0/2570744096 shutdown_connections 2024-09-16T15:43:03.788 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:03.778+0000 7f41c94ca640 1 -- 172.21.15.123:0/2570744096 wait complete. 2024-09-16T15:43:03.796 INFO:teuthology.run_tasks:Running task ceph-fuse... 2024-09-16T15:43:03.820 INFO:tasks.ceph_fuse:Running ceph_fuse task... 2024-09-16T15:43:03.820 INFO:tasks.ceph_fuse:config is {'client.0': {}, 'client.2': {}, 'client.1': {}, 'client.3': {}} 2024-09-16T15:43:03.820 INFO:tasks.ceph_fuse:client.0 config is {} 2024-09-16T15:43:03.820 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-09-16T15:43:03.820 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-09-16T15:43:03.821 INFO:tasks.ceph_fuse:client.2 config is {} 2024-09-16T15:43:03.821 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-09-16T15:43:03.821 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.2 2024-09-16T15:43:03.821 INFO:tasks.ceph_fuse:client.1 config is {} 2024-09-16T15:43:03.821 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-09-16T15:43:03.821 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2024-09-16T15:43:03.821 INFO:tasks.ceph_fuse:client.3 config is {} 2024-09-16T15:43:03.821 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-09-16T15:43:03.821 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.3 2024-09-16T15:43:03.821 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:43:03.821 DEBUG:teuthology.orchestra.run.smithi123:> ip netns list 2024-09-16T15:43:03.827 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:43:03.827 DEBUG:teuthology.orchestra.run.smithi123:> sudo ip link delete ceph-brx 2024-09-16T15:43:03.887 INFO:teuthology.orchestra.run.smithi123.stderr:Cannot find device "ceph-brx" 2024-09-16T15:43:03.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T15:43:03.888 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:43:03.888 DEBUG:teuthology.orchestra.run.smithi184:> ip netns list 2024-09-16T15:43:03.895 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:43:03.895 DEBUG:teuthology.orchestra.run.smithi184:> sudo ip link delete ceph-brx 2024-09-16T15:43:03.956 INFO:teuthology.orchestra.run.smithi184.stderr:Cannot find device "ceph-brx" 2024-09-16T15:43:03.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T15:43:03.957 INFO:tasks.ceph_fuse:Mounting ceph-fuse clients... 2024-09-16T15:43:03.957 DEBUG:tasks.ceph_fuse:passing mntargs=[] 2024-09-16T15:43:03.957 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-09-16T15:43:04.086 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:04.082+0000 7f05cc5fc640 1 -- 172.21.15.123:0/120069958 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f05c40ca870 msgr2=0x7f05c41506d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:43:04.087 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:04.082+0000 7f05cc5fc640 1 --2- 172.21.15.123:0/120069958 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f05c40ca870 0x7f05c41506d0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f05c000acd0 tx=0x7f05c001def0 comp rx=0 tx=0).stop 2024-09-16T15:43:04.087 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:04.082+0000 7f05cc5fc640 1 -- 172.21.15.123:0/120069958 shutdown_connections 2024-09-16T15:43:04.087 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:04.082+0000 7f05cc5fc640 1 --2- 172.21.15.123:0/120069958 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f05c40ca870 0x7f05c41506d0 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:04.087 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:04.086+0000 7f05cc5fc640 1 -- 172.21.15.123:0/120069958 >> 172.21.15.123:0/120069958 conn(0x7f05c40cfb10 msgr2=0x7f05c40cff20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:43:04.087 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:04.086+0000 7f05cc5fc640 1 -- 172.21.15.123:0/120069958 shutdown_connections 2024-09-16T15:43:04.087 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:04.086+0000 7f05cc5fc640 1 -- 172.21.15.123:0/120069958 wait complete. 2024-09-16T15:43:04.087 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:04.086+0000 7f05cc5fc640 1 Processor -- start 2024-09-16T15:43:04.088 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:04.086+0000 7f05cc5fc640 1 -- start start 2024-09-16T15:43:04.089 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:04.086+0000 7f05cc5fc640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f05c406fc00 0x7f05c4157950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:43:04.090 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:04.086+0000 7f05cc5fc640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f05c40ca870 0x7f05c414b730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:43:04.090 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:04.086+0000 7f05cc5fc640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f05c40cf3f0 0x7f05c414f8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:43:04.090 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:04.086+0000 7f05cc5fc640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f05c4168540 con 0x7f05c40ca870 2024-09-16T15:43:04.090 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:04.086+0000 7f05cc5fc640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f05c41683c0 con 0x7f05c406fc00 2024-09-16T15:43:04.090 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:04.086+0000 7f05cc5fc640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f05c41686c0 con 0x7f05c40cf3f0 2024-09-16T15:43:04.090 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:04.086+0000 7f05c9b97640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f05c40ca870 0x7f05c414b730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:43:04.090 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:04.086+0000 7f05c9b97640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f05c40ca870 0x7f05c414b730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:41764/0 (socket says 172.21.15.123:41764) 2024-09-16T15:43:04.090 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:04.086+0000 7f05c9b97640 1 -- 172.21.15.123:0/3317709318 learned_addr learned my addr 172.21.15.123:0/3317709318 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:43:04.090 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:04.086+0000 7f05ca398640 1 --2- 172.21.15.123:0/3317709318 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f05c406fc00 0x7f05c4157950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:43:04.090 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:04.086+0000 7f05cab99640 1 --2- 172.21.15.123:0/3317709318 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f05c40cf3f0 0x7f05c414f8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:43:04.090 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:04.086+0000 7f05c9b97640 1 -- 172.21.15.123:0/3317709318 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f05c40cf3f0 msgr2=0x7f05c414f8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:43:04.090 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:04.086+0000 7f05c9b97640 1 --2- 172.21.15.123:0/3317709318 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f05c40cf3f0 0x7f05c414f8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:04.090 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:04.086+0000 7f05c9b97640 1 -- 172.21.15.123:0/3317709318 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f05c406fc00 msgr2=0x7f05c4157950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:43:04.091 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:04.086+0000 7f05c9b97640 1 --2- 172.21.15.123:0/3317709318 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f05c406fc00 0x7f05c4157950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:04.091 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:04.086+0000 7f05c9b97640 1 -- 172.21.15.123:0/3317709318 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f05c414bc70 con 0x7f05c40ca870 2024-09-16T15:43:04.091 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:04.086+0000 7f05c9b97640 1 --2- 172.21.15.123:0/3317709318 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f05c40ca870 0x7f05c414b730 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f05b400c400 tx=0x7f05b400c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:43:04.091 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:04.086+0000 7f05baffd640 1 -- 172.21.15.123:0/3317709318 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05b4017070 con 0x7f05c40ca870 2024-09-16T15:43:04.091 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:04.086+0000 7f05baffd640 1 -- 172.21.15.123:0/3317709318 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f05b4003600 con 0x7f05c40ca870 2024-09-16T15:43:04.091 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:04.086+0000 7f05cc5fc640 1 -- 172.21.15.123:0/3317709318 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f05c414c500 con 0x7f05c40ca870 2024-09-16T15:43:04.091 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:04.086+0000 7f05baffd640 1 -- 172.21.15.123:0/3317709318 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05b4003a50 con 0x7f05c40ca870 2024-09-16T15:43:04.091 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:04.086+0000 7f05cc5fc640 1 -- 172.21.15.123:0/3317709318 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f05c414f450 con 0x7f05c40ca870 2024-09-16T15:43:04.092 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:04.086+0000 7f05cc5fc640 1 -- 172.21.15.123:0/3317709318 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f05c41506d0 con 0x7f05c40ca870 2024-09-16T15:43:04.092 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:04.090+0000 7f05baffd640 1 -- 172.21.15.123:0/3317709318 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 5) ==== 97080+0+0 (secure 0 0 0) 0x7f05b401a070 con 0x7f05c40ca870 2024-09-16T15:43:04.092 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:04.090+0000 7f05baffd640 1 --2- 172.21.15.123:0/3317709318 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f05a8072ef0 0x7f05a80753b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:43:04.092 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:04.090+0000 7f05baffd640 1 -- 172.21.15.123:0/3317709318 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f05b40931a0 con 0x7f05c40ca870 2024-09-16T15:43:04.092 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:04.090+0000 7f05ca398640 1 --2- 172.21.15.123:0/3317709318 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f05a8072ef0 0x7f05a80753b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:43:04.092 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:04.090+0000 7f05ca398640 1 --2- 172.21.15.123:0/3317709318 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f05a8072ef0 0x7f05a80753b0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f05bc004e30 tx=0x7f05bc009040 comp rx=0 tx=0).ready entity=mgr.4135 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:43:04.095 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:04.094+0000 7f05baffd640 1 -- 172.21.15.123:0/3317709318 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f05b405e350 con 0x7f05c40ca870 2024-09-16T15:43:04.305 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:04.302+0000 7f05cc5fc640 1 -- 172.21.15.123:0/3317709318 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f05c40cd6e0 con 0x7f05c40ca870 2024-09-16T15:43:04.306 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:04.302+0000 7f05baffd640 1 -- 172.21.15.123:0/3317709318 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v8) ==== 53+0+73 (secure 0 0 0) 0x7f05b405dc50 con 0x7f05c40ca870 2024-09-16T15:43:04.306 INFO:teuthology.orchestra.run.smithi123.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-09-16T15:43:04.309 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:04.306+0000 7f05cc5fc640 1 -- 172.21.15.123:0/3317709318 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f05a8072ef0 msgr2=0x7f05a80753b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:43:04.309 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:04.306+0000 7f05cc5fc640 1 --2- 172.21.15.123:0/3317709318 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f05a8072ef0 0x7f05a80753b0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f05bc004e30 tx=0x7f05bc009040 comp rx=0 tx=0).stop 2024-09-16T15:43:04.309 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:04.306+0000 7f05cc5fc640 1 -- 172.21.15.123:0/3317709318 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f05c40ca870 msgr2=0x7f05c414b730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:43:04.309 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:04.306+0000 7f05cc5fc640 1 --2- 172.21.15.123:0/3317709318 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f05c40ca870 0x7f05c414b730 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f05b400c400 tx=0x7f05b400c8c0 comp rx=0 tx=0).stop 2024-09-16T15:43:04.309 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:04.306+0000 7f05cc5fc640 1 -- 172.21.15.123:0/3317709318 shutdown_connections 2024-09-16T15:43:04.309 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:04.306+0000 7f05cc5fc640 1 --2- 172.21.15.123:0/3317709318 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f05a8072ef0 0x7f05a80753b0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:04.309 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:04.306+0000 7f05cc5fc640 1 --2- 172.21.15.123:0/3317709318 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f05c40cf3f0 0x7f05c414f8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:04.309 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:04.306+0000 7f05cc5fc640 1 --2- 172.21.15.123:0/3317709318 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f05c40ca870 0x7f05c414b730 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:04.309 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:04.306+0000 7f05cc5fc640 1 --2- 172.21.15.123:0/3317709318 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f05c406fc00 0x7f05c4157950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:04.309 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:04.306+0000 7f05cc5fc640 1 -- 172.21.15.123:0/3317709318 >> 172.21.15.123:0/3317709318 conn(0x7f05c40cfb10 msgr2=0x7f05c415ae10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:43:04.309 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:04.306+0000 7f05cc5fc640 1 -- 172.21.15.123:0/3317709318 shutdown_connections 2024-09-16T15:43:04.309 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:04.306+0000 7f05cc5fc640 1 -- 172.21.15.123:0/3317709318 wait complete. 2024-09-16T15:43:04.325 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-09-16T15:43:04.325 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-09-16T15:43:04.325 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi123.front.sepia.ceph.com 2024-09-16T15:43:04.325 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-09-16T15:43:04.326 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-09-16T15:43:04.326 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-09-16T15:43:04.326 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-09-16T15:43:04.326 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-09-16T15:43:04.326 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.0' 2024-09-16T15:43:04.326 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:43:04.326 DEBUG:teuthology.orchestra.run.smithi123:> ip addr 2024-09-16T15:43:04.332 INFO:teuthology.orchestra.run.smithi123.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-09-16T15:43:04.332 INFO:teuthology.orchestra.run.smithi123.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-09-16T15:43:04.332 INFO:teuthology.orchestra.run.smithi123.stdout: inet 127.0.0.1/8 scope host lo 2024-09-16T15:43:04.332 INFO:teuthology.orchestra.run.smithi123.stdout: valid_lft forever preferred_lft forever 2024-09-16T15:43:04.332 INFO:teuthology.orchestra.run.smithi123.stdout: inet6 ::1/128 scope host 2024-09-16T15:43:04.332 INFO:teuthology.orchestra.run.smithi123.stdout: valid_lft forever preferred_lft forever 2024-09-16T15:43:04.332 INFO:teuthology.orchestra.run.smithi123.stdout:2: eno1: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-09-16T15:43:04.332 INFO:teuthology.orchestra.run.smithi123.stdout: link/ether 0c:c4:7a:d8:da:6c brd ff:ff:ff:ff:ff:ff 2024-09-16T15:43:04.332 INFO:teuthology.orchestra.run.smithi123.stdout: altname enp7s0f0 2024-09-16T15:43:04.333 INFO:teuthology.orchestra.run.smithi123.stdout:3: eno2: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-09-16T15:43:04.333 INFO:teuthology.orchestra.run.smithi123.stdout: link/ether 0c:c4:7a:d8:da:6d brd ff:ff:ff:ff:ff:ff 2024-09-16T15:43:04.333 INFO:teuthology.orchestra.run.smithi123.stdout: altname enp7s0f1 2024-09-16T15:43:04.333 INFO:teuthology.orchestra.run.smithi123.stdout:4: enp3s0f0: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-09-16T15:43:04.333 INFO:teuthology.orchestra.run.smithi123.stdout: link/ether 0c:c4:7a:8f:d3:f0 brd ff:ff:ff:ff:ff:ff 2024-09-16T15:43:04.333 INFO:teuthology.orchestra.run.smithi123.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-09-16T15:43:04.333 INFO:teuthology.orchestra.run.smithi123.stdout: link/ether 0c:c4:7a:8f:d3:f1 brd ff:ff:ff:ff:ff:ff 2024-09-16T15:43:04.333 INFO:teuthology.orchestra.run.smithi123.stdout: inet 172.21.15.123/20 metric 100 brd 172.21.15.255 scope global dynamic enp3s0f1 2024-09-16T15:43:04.333 INFO:teuthology.orchestra.run.smithi123.stdout: valid_lft 42484sec preferred_lft 42484sec 2024-09-16T15:43:04.333 INFO:teuthology.orchestra.run.smithi123.stdout: inet6 fe80::ec4:7aff:fe8f:d3f1/64 scope link 2024-09-16T15:43:04.333 INFO:teuthology.orchestra.run.smithi123.stdout: valid_lft forever preferred_lft forever 2024-09-16T15:43:04.333 INFO:teuthology.orchestra.run.smithi123.stdout:6: docker0: mtu 1500 qdisc noqueue state DOWN group default 2024-09-16T15:43:04.333 INFO:teuthology.orchestra.run.smithi123.stdout: link/ether 02:42:dd:82:a2:6e brd ff:ff:ff:ff:ff:ff 2024-09-16T15:43:04.333 INFO:teuthology.orchestra.run.smithi123.stdout: inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0 2024-09-16T15:43:04.333 INFO:teuthology.orchestra.run.smithi123.stdout: valid_lft forever preferred_lft forever 2024-09-16T15:43:04.334 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-09-16T15:43:04.334 DEBUG:teuthology.orchestra.run.smithi123:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-09-16T15:43:04.334 DEBUG:teuthology.orchestra.run.smithi123:> set -e 2024-09-16T15:43:04.334 DEBUG:teuthology.orchestra.run.smithi123:> sudo ip link add name ceph-brx type bridge 2024-09-16T15:43:04.334 DEBUG:teuthology.orchestra.run.smithi123:> sudo ip addr flush dev ceph-brx 2024-09-16T15:43:04.334 DEBUG:teuthology.orchestra.run.smithi123:> sudo ip link set ceph-brx up 2024-09-16T15:43:04.334 DEBUG:teuthology.orchestra.run.smithi123:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-09-16T15:43:04.334 DEBUG:teuthology.orchestra.run.smithi123:> ') 2024-09-16T15:43:04.419 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:04 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-09-16T15:43:04.451 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:04 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-16T15:43:04.458 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:43:04.458 DEBUG:teuthology.orchestra.run.smithi123:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-09-16T15:43:04.467 INFO:teuthology.orchestra.run.smithi123.stdout:1 2024-09-16T15:43:04.469 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:43:04.469 DEBUG:teuthology.orchestra.run.smithi123:> route 2024-09-16T15:43:04.544 INFO:teuthology.orchestra.run.smithi123.stdout:Kernel IP routing table 2024-09-16T15:43:04.544 INFO:teuthology.orchestra.run.smithi123.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-09-16T15:43:04.544 INFO:teuthology.orchestra.run.smithi123.stdout:default _gateway 0.0.0.0 UG 0 0 0 enp3s0f1 2024-09-16T15:43:04.544 INFO:teuthology.orchestra.run.smithi123.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-09-16T15:43:04.544 INFO:teuthology.orchestra.run.smithi123.stdout:158.69.67.47 _gateway 255.255.255.255 UGH 100 0 0 enp3s0f1 2024-09-16T15:43:04.544 INFO:teuthology.orchestra.run.smithi123.stdout:172.17.0.0 0.0.0.0 255.255.0.0 U 0 0 0 docker0 2024-09-16T15:43:04.544 INFO:teuthology.orchestra.run.smithi123.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-09-16T15:43:04.545 INFO:teuthology.orchestra.run.smithi123.stdout:ns1.front.sepia 0.0.0.0 255.255.255.255 UH 100 0 0 enp3s0f1 2024-09-16T15:43:04.545 INFO:teuthology.orchestra.run.smithi123.stdout:172.21.0.2 0.0.0.0 255.255.255.255 UH 100 0 0 enp3s0f1 2024-09-16T15:43:04.545 INFO:teuthology.orchestra.run.smithi123.stdout:_gateway 0.0.0.0 255.255.255.255 UH 100 0 0 enp3s0f1 2024-09-16T15:43:04.545 INFO:teuthology.orchestra.run.smithi123.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-09-16T15:43:04.545 DEBUG:teuthology.orchestra.run.smithi123:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-09-16T15:43:04.545 DEBUG:teuthology.orchestra.run.smithi123:> set -e 2024-09-16T15:43:04.545 DEBUG:teuthology.orchestra.run.smithi123:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-09-16T15:43:04.545 DEBUG:teuthology.orchestra.run.smithi123:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-09-16T15:43:04.546 DEBUG:teuthology.orchestra.run.smithi123:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-09-16T15:43:04.546 DEBUG:teuthology.orchestra.run.smithi123:> ') 2024-09-16T15:43:04.632 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:04 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-09-16T15:43:04.656 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:04 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-16T15:43:04.663 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:43:04.663 DEBUG:teuthology.orchestra.run.smithi123:> ip netns list 2024-09-16T15:43:04.669 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:43:04.669 DEBUG:teuthology.orchestra.run.smithi123:> ip netns list-id 2024-09-16T15:43:04.714 DEBUG:teuthology.orchestra.run.smithi123:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-09-16T15:43:04.714 DEBUG:teuthology.orchestra.run.smithi123:> set -e 2024-09-16T15:43:04.714 DEBUG:teuthology.orchestra.run.smithi123:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.0 2024-09-16T15:43:04.714 DEBUG:teuthology.orchestra.run.smithi123:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.0 0 2024-09-16T15:43:04.714 DEBUG:teuthology.orchestra.run.smithi123:> ') 2024-09-16T15:43:04.802 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:04 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-09-16T15:43:04.823 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:04 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-16T15:43:04.830 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.0' with 192.168.0.1/16 2024-09-16T15:43:04.831 DEBUG:teuthology.orchestra.run.smithi123:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-09-16T15:43:04.831 DEBUG:teuthology.orchestra.run.smithi123:> set -e 2024-09-16T15:43:04.831 DEBUG:teuthology.orchestra.run.smithi123:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.0 type veth peer name brx.0 2024-09-16T15:43:04.831 DEBUG:teuthology.orchestra.run.smithi123:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-09-16T15:43:04.831 DEBUG:teuthology.orchestra.run.smithi123:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set veth0 up 2024-09-16T15:43:04.831 DEBUG:teuthology.orchestra.run.smithi123:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set lo up 2024-09-16T15:43:04.831 DEBUG:teuthology.orchestra.run.smithi123:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip route add default via 192.168.255.254 2024-09-16T15:43:04.831 DEBUG:teuthology.orchestra.run.smithi123:> ') 2024-09-16T15:43:04.917 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:04 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-09-16T15:43:04.975 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:04 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-16T15:43:04.981 DEBUG:teuthology.orchestra.run.smithi123:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-09-16T15:43:04.981 DEBUG:teuthology.orchestra.run.smithi123:> set -e 2024-09-16T15:43:04.982 DEBUG:teuthology.orchestra.run.smithi123:> sudo ip link set brx.0 up 2024-09-16T15:43:04.982 DEBUG:teuthology.orchestra.run.smithi123:> sudo ip link set dev brx.0 master ceph-brx 2024-09-16T15:43:04.982 DEBUG:teuthology.orchestra.run.smithi123:> ') 2024-09-16T15:43:05.059 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:05 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-09-16T15:43:05.075 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:05 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-16T15:43:05.081 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2024-09-16T15:43:05.082 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-09-16T15:43:05.082 DEBUG:teuthology.orchestra.run.smithi123:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-09-16T15:43:05.127 INFO:teuthology.orchestra.run.smithi123.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-09-16T15:43:05.127 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-09-16T15:43:05.127 DEBUG:teuthology.orchestra.run.smithi123:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-09-16T15:43:05.174 DEBUG:teuthology.orchestra.run.smithi123:> sudo modprobe fuse 2024-09-16T15:43:05.232 DEBUG:teuthology.orchestra.run.smithi123:> cat /proc/self/mounts | awk '{print $2}' 2024-09-16T15:43:05.284 INFO:teuthology.orchestra.run.smithi123.stdout:/sys 2024-09-16T15:43:05.284 INFO:teuthology.orchestra.run.smithi123.stdout:/proc 2024-09-16T15:43:05.284 INFO:teuthology.orchestra.run.smithi123.stdout:/dev 2024-09-16T15:43:05.284 INFO:teuthology.orchestra.run.smithi123.stdout:/dev/pts 2024-09-16T15:43:05.284 INFO:teuthology.orchestra.run.smithi123.stdout:/run 2024-09-16T15:43:05.284 INFO:teuthology.orchestra.run.smithi123.stdout:/ 2024-09-16T15:43:05.284 INFO:teuthology.orchestra.run.smithi123.stdout:/sys/kernel/security 2024-09-16T15:43:05.285 INFO:teuthology.orchestra.run.smithi123.stdout:/dev/shm 2024-09-16T15:43:05.285 INFO:teuthology.orchestra.run.smithi123.stdout:/run/lock 2024-09-16T15:43:05.285 INFO:teuthology.orchestra.run.smithi123.stdout:/sys/fs/cgroup 2024-09-16T15:43:05.285 INFO:teuthology.orchestra.run.smithi123.stdout:/sys/fs/pstore 2024-09-16T15:43:05.285 INFO:teuthology.orchestra.run.smithi123.stdout:/sys/fs/bpf 2024-09-16T15:43:05.285 INFO:teuthology.orchestra.run.smithi123.stdout:/proc/sys/fs/binfmt_misc 2024-09-16T15:43:05.285 INFO:teuthology.orchestra.run.smithi123.stdout:/dev/hugepages 2024-09-16T15:43:05.285 INFO:teuthology.orchestra.run.smithi123.stdout:/dev/mqueue 2024-09-16T15:43:05.285 INFO:teuthology.orchestra.run.smithi123.stdout:/sys/kernel/debug 2024-09-16T15:43:05.285 INFO:teuthology.orchestra.run.smithi123.stdout:/sys/kernel/tracing 2024-09-16T15:43:05.285 INFO:teuthology.orchestra.run.smithi123.stdout:/proc/fs/nfsd 2024-09-16T15:43:05.286 INFO:teuthology.orchestra.run.smithi123.stdout:/sys/fs/fuse/connections 2024-09-16T15:43:05.286 INFO:teuthology.orchestra.run.smithi123.stdout:/sys/kernel/config 2024-09-16T15:43:05.286 INFO:teuthology.orchestra.run.smithi123.stdout:/run/credentials/systemd-sysusers.service 2024-09-16T15:43:05.286 INFO:teuthology.orchestra.run.smithi123.stdout:/run/rpc_pipefs 2024-09-16T15:43:05.286 INFO:teuthology.orchestra.run.smithi123.stdout:/run/user/1000 2024-09-16T15:43:05.286 INFO:teuthology.orchestra.run.smithi123.stdout:/var/lib/ceph 2024-09-16T15:43:05.286 INFO:teuthology.orchestra.run.smithi123.stdout:/var/lib/ceph/osd/ceph-0 2024-09-16T15:43:05.286 INFO:teuthology.orchestra.run.smithi123.stdout:/var/lib/ceph/osd/ceph-1 2024-09-16T15:43:05.286 INFO:teuthology.orchestra.run.smithi123.stdout:/var/lib/ceph/osd/ceph-2 2024-09-16T15:43:05.286 INFO:teuthology.orchestra.run.smithi123.stdout:/var/lib/ceph/osd/ceph-3 2024-09-16T15:43:05.286 INFO:teuthology.orchestra.run.smithi123.stdout:/run/netns 2024-09-16T15:43:05.286 INFO:teuthology.orchestra.run.smithi123.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-09-16T15:43:05.287 INFO:teuthology.orchestra.run.smithi123.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-09-16T15:43:05.287 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:43:05.288 DEBUG:teuthology.orchestra.run.smithi123:> ls /sys/fs/fuse/connections 2024-09-16T15:43:05.335 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-09-16T15:43:05.335 DEBUG:teuthology.orchestra.run.smithi123:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2024-09-16T15:43:05.380 DEBUG:teuthology.orchestra.run.smithi123:> sudo modprobe fuse 2024-09-16T15:43:05.393 DEBUG:teuthology.orchestra.run.smithi123:> cat /proc/self/mounts | awk '{print $2}' 2024-09-16T15:43:05.439 INFO:teuthology.orchestra.run.smithi123.stdout:/sys 2024-09-16T15:43:05.439 INFO:teuthology.orchestra.run.smithi123.stdout:/proc 2024-09-16T15:43:05.439 INFO:teuthology.orchestra.run.smithi123.stdout:/dev 2024-09-16T15:43:05.439 INFO:teuthology.orchestra.run.smithi123.stdout:/dev/pts 2024-09-16T15:43:05.439 INFO:teuthology.orchestra.run.smithi123.stdout:/run 2024-09-16T15:43:05.439 INFO:teuthology.orchestra.run.smithi123.stdout:/ 2024-09-16T15:43:05.439 INFO:teuthology.orchestra.run.smithi123.stdout:/sys/kernel/security 2024-09-16T15:43:05.440 INFO:teuthology.orchestra.run.smithi123.stdout:/dev/shm 2024-09-16T15:43:05.440 INFO:teuthology.orchestra.run.smithi123.stdout:/run/lock 2024-09-16T15:43:05.440 INFO:teuthology.orchestra.run.smithi123.stdout:/sys/fs/cgroup 2024-09-16T15:43:05.440 INFO:teuthology.orchestra.run.smithi123.stdout:/sys/fs/pstore 2024-09-16T15:43:05.440 INFO:teuthology.orchestra.run.smithi123.stdout:/sys/fs/bpf 2024-09-16T15:43:05.440 INFO:teuthology.orchestra.run.smithi123.stdout:/proc/sys/fs/binfmt_misc 2024-09-16T15:43:05.440 INFO:teuthology.orchestra.run.smithi123.stdout:/dev/hugepages 2024-09-16T15:43:05.440 INFO:teuthology.orchestra.run.smithi123.stdout:/dev/mqueue 2024-09-16T15:43:05.440 INFO:teuthology.orchestra.run.smithi123.stdout:/sys/kernel/debug 2024-09-16T15:43:05.440 INFO:teuthology.orchestra.run.smithi123.stdout:/sys/kernel/tracing 2024-09-16T15:43:05.440 INFO:teuthology.orchestra.run.smithi123.stdout:/proc/fs/nfsd 2024-09-16T15:43:05.440 INFO:teuthology.orchestra.run.smithi123.stdout:/sys/fs/fuse/connections 2024-09-16T15:43:05.441 INFO:teuthology.orchestra.run.smithi123.stdout:/sys/kernel/config 2024-09-16T15:43:05.441 INFO:teuthology.orchestra.run.smithi123.stdout:/run/credentials/systemd-sysusers.service 2024-09-16T15:43:05.441 INFO:teuthology.orchestra.run.smithi123.stdout:/run/rpc_pipefs 2024-09-16T15:43:05.441 INFO:teuthology.orchestra.run.smithi123.stdout:/run/user/1000 2024-09-16T15:43:05.441 INFO:teuthology.orchestra.run.smithi123.stdout:/var/lib/ceph 2024-09-16T15:43:05.441 INFO:teuthology.orchestra.run.smithi123.stdout:/var/lib/ceph/osd/ceph-0 2024-09-16T15:43:05.441 INFO:teuthology.orchestra.run.smithi123.stdout:/var/lib/ceph/osd/ceph-1 2024-09-16T15:43:05.441 INFO:teuthology.orchestra.run.smithi123.stdout:/var/lib/ceph/osd/ceph-2 2024-09-16T15:43:05.441 INFO:teuthology.orchestra.run.smithi123.stdout:/var/lib/ceph/osd/ceph-3 2024-09-16T15:43:05.441 INFO:teuthology.orchestra.run.smithi123.stdout:/run/netns 2024-09-16T15:43:05.441 INFO:teuthology.orchestra.run.smithi123.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-09-16T15:43:05.441 INFO:teuthology.orchestra.run.smithi123.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-09-16T15:43:05.442 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:43:05.442 DEBUG:teuthology.orchestra.run.smithi123:> ls /sys/fs/fuse/connections 2024-09-16T15:43:05.457 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi123.stderr:2024-09-16T15:43:05.454+0000 7f2f8c25f580 -1 init, newargv = 0x55efdf59ec70 newargc=15 2024-09-16T15:43:05.458 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi123.stderr:ceph-fuse[36236]: starting ceph client 2024-09-16T15:43:05.464 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi123.stderr:ceph-fuse[36236]: starting fuse 2024-09-16T15:43:06.563 DEBUG:teuthology.orchestra.run.smithi123:> sudo modprobe fuse 2024-09-16T15:43:06.580 DEBUG:teuthology.orchestra.run.smithi123:> cat /proc/self/mounts | awk '{print $2}' 2024-09-16T15:43:06.628 INFO:teuthology.orchestra.run.smithi123.stdout:/sys 2024-09-16T15:43:06.628 INFO:teuthology.orchestra.run.smithi123.stdout:/proc 2024-09-16T15:43:06.628 INFO:teuthology.orchestra.run.smithi123.stdout:/dev 2024-09-16T15:43:06.628 INFO:teuthology.orchestra.run.smithi123.stdout:/dev/pts 2024-09-16T15:43:06.628 INFO:teuthology.orchestra.run.smithi123.stdout:/run 2024-09-16T15:43:06.628 INFO:teuthology.orchestra.run.smithi123.stdout:/ 2024-09-16T15:43:06.628 INFO:teuthology.orchestra.run.smithi123.stdout:/sys/kernel/security 2024-09-16T15:43:06.628 INFO:teuthology.orchestra.run.smithi123.stdout:/dev/shm 2024-09-16T15:43:06.628 INFO:teuthology.orchestra.run.smithi123.stdout:/run/lock 2024-09-16T15:43:06.628 INFO:teuthology.orchestra.run.smithi123.stdout:/sys/fs/cgroup 2024-09-16T15:43:06.629 INFO:teuthology.orchestra.run.smithi123.stdout:/sys/fs/pstore 2024-09-16T15:43:06.629 INFO:teuthology.orchestra.run.smithi123.stdout:/sys/fs/bpf 2024-09-16T15:43:06.629 INFO:teuthology.orchestra.run.smithi123.stdout:/proc/sys/fs/binfmt_misc 2024-09-16T15:43:06.629 INFO:teuthology.orchestra.run.smithi123.stdout:/dev/hugepages 2024-09-16T15:43:06.629 INFO:teuthology.orchestra.run.smithi123.stdout:/dev/mqueue 2024-09-16T15:43:06.629 INFO:teuthology.orchestra.run.smithi123.stdout:/sys/kernel/debug 2024-09-16T15:43:06.629 INFO:teuthology.orchestra.run.smithi123.stdout:/sys/kernel/tracing 2024-09-16T15:43:06.629 INFO:teuthology.orchestra.run.smithi123.stdout:/proc/fs/nfsd 2024-09-16T15:43:06.629 INFO:teuthology.orchestra.run.smithi123.stdout:/sys/fs/fuse/connections 2024-09-16T15:43:06.629 INFO:teuthology.orchestra.run.smithi123.stdout:/sys/kernel/config 2024-09-16T15:43:06.629 INFO:teuthology.orchestra.run.smithi123.stdout:/run/credentials/systemd-sysusers.service 2024-09-16T15:43:06.629 INFO:teuthology.orchestra.run.smithi123.stdout:/run/rpc_pipefs 2024-09-16T15:43:06.629 INFO:teuthology.orchestra.run.smithi123.stdout:/run/user/1000 2024-09-16T15:43:06.629 INFO:teuthology.orchestra.run.smithi123.stdout:/var/lib/ceph 2024-09-16T15:43:06.630 INFO:teuthology.orchestra.run.smithi123.stdout:/var/lib/ceph/osd/ceph-0 2024-09-16T15:43:06.630 INFO:teuthology.orchestra.run.smithi123.stdout:/var/lib/ceph/osd/ceph-1 2024-09-16T15:43:06.630 INFO:teuthology.orchestra.run.smithi123.stdout:/var/lib/ceph/osd/ceph-2 2024-09-16T15:43:06.630 INFO:teuthology.orchestra.run.smithi123.stdout:/var/lib/ceph/osd/ceph-3 2024-09-16T15:43:06.630 INFO:teuthology.orchestra.run.smithi123.stdout:/run/netns 2024-09-16T15:43:06.630 INFO:teuthology.orchestra.run.smithi123.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-09-16T15:43:06.630 INFO:teuthology.orchestra.run.smithi123.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-09-16T15:43:06.630 INFO:teuthology.orchestra.run.smithi123.stdout:/home/ubuntu/cephtest/mnt.0 2024-09-16T15:43:06.631 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:43:06.631 DEBUG:teuthology.orchestra.run.smithi123:> ls /sys/fs/fuse/connections 2024-09-16T15:43:06.679 INFO:teuthology.orchestra.run.smithi123.stdout:45 2024-09-16T15:43:06.679 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [45] 2024-09-16T15:43:06.679 DEBUG:teuthology.orchestra.run.smithi123:> sudo stdin-killer -- python3 -c ' 2024-09-16T15:43:06.679 DEBUG:teuthology.orchestra.run.smithi123:> import glob 2024-09-16T15:43:06.679 DEBUG:teuthology.orchestra.run.smithi123:> import re 2024-09-16T15:43:06.679 DEBUG:teuthology.orchestra.run.smithi123:> import os 2024-09-16T15:43:06.679 DEBUG:teuthology.orchestra.run.smithi123:> import subprocess 2024-09-16T15:43:06.679 DEBUG:teuthology.orchestra.run.smithi123:> 2024-09-16T15:43:06.680 DEBUG:teuthology.orchestra.run.smithi123:> def _find_admin_socket(client_name): 2024-09-16T15:43:06.680 DEBUG:teuthology.orchestra.run.smithi123:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2024-09-16T15:43:06.680 DEBUG:teuthology.orchestra.run.smithi123:> files = glob.glob(asok_path) 2024-09-16T15:43:06.680 DEBUG:teuthology.orchestra.run.smithi123:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-09-16T15:43:06.680 DEBUG:teuthology.orchestra.run.smithi123:> 2024-09-16T15:43:06.680 DEBUG:teuthology.orchestra.run.smithi123:> # Given a non-glob path, it better be there 2024-09-16T15:43:06.680 DEBUG:teuthology.orchestra.run.smithi123:> if "*" not in asok_path: 2024-09-16T15:43:06.680 DEBUG:teuthology.orchestra.run.smithi123:> assert(len(files) == 1) 2024-09-16T15:43:06.680 DEBUG:teuthology.orchestra.run.smithi123:> return files[0] 2024-09-16T15:43:06.680 DEBUG:teuthology.orchestra.run.smithi123:> 2024-09-16T15:43:06.680 DEBUG:teuthology.orchestra.run.smithi123:> for f in files: 2024-09-16T15:43:06.680 DEBUG:teuthology.orchestra.run.smithi123:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-09-16T15:43:06.680 DEBUG:teuthology.orchestra.run.smithi123:> if os.path.exists("/proc/{0}".format(pid)): 2024-09-16T15:43:06.680 DEBUG:teuthology.orchestra.run.smithi123:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-09-16T15:43:06.680 DEBUG:teuthology.orchestra.run.smithi123:> contents = proc_f.read() 2024-09-16T15:43:06.680 DEBUG:teuthology.orchestra.run.smithi123:> if mountpoint in contents: 2024-09-16T15:43:06.681 DEBUG:teuthology.orchestra.run.smithi123:> return f 2024-09-16T15:43:06.681 DEBUG:teuthology.orchestra.run.smithi123:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-09-16T15:43:06.681 DEBUG:teuthology.orchestra.run.smithi123:> 2024-09-16T15:43:06.681 DEBUG:teuthology.orchestra.run.smithi123:> print(_find_admin_socket("client.0")) 2024-09-16T15:43:06.681 DEBUG:teuthology.orchestra.run.smithi123:> ' 2024-09-16T15:43:06.804 INFO:teuthology.orchestra.run.smithi123.stdout:/var/run/ceph/ceph-client.0.36236.asok 2024-09-16T15:43:06.808 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:06 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-16T15:43:06.815 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.36236.asok 2024-09-16T15:43:06.815 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:43:06.815 DEBUG:teuthology.orchestra.run.smithi123:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.36236.asok status 2024-09-16T15:43:06.967 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2024-09-16T15:43:06.967 INFO:teuthology.orchestra.run.smithi123.stdout: "metadata": { 2024-09-16T15:43:06.967 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph_sha1": "33f95d2ae1bcbe7eb04404303f179eeeaa6982df", 2024-09-16T15:43:06.967 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph_version": "ceph version 19.3.0-4938-g33f95d2a (33f95d2ae1bcbe7eb04404303f179eeeaa6982df) squid (dev)", 2024-09-16T15:43:06.967 INFO:teuthology.orchestra.run.smithi123.stdout: "entity_id": "0", 2024-09-16T15:43:06.968 INFO:teuthology.orchestra.run.smithi123.stdout: "hostname": "smithi123", 2024-09-16T15:43:06.968 INFO:teuthology.orchestra.run.smithi123.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2024-09-16T15:43:06.968 INFO:teuthology.orchestra.run.smithi123.stdout: "pid": "36236", 2024-09-16T15:43:06.968 INFO:teuthology.orchestra.run.smithi123.stdout: "root": "/" 2024-09-16T15:43:06.968 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:43:06.968 INFO:teuthology.orchestra.run.smithi123.stdout: "dentry_count": 0, 2024-09-16T15:43:06.968 INFO:teuthology.orchestra.run.smithi123.stdout: "dentry_pinned_count": 0, 2024-09-16T15:43:06.968 INFO:teuthology.orchestra.run.smithi123.stdout: "id": 4662, 2024-09-16T15:43:06.968 INFO:teuthology.orchestra.run.smithi123.stdout: "inst": { 2024-09-16T15:43:06.968 INFO:teuthology.orchestra.run.smithi123.stdout: "name": { 2024-09-16T15:43:06.968 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "client", 2024-09-16T15:43:06.968 INFO:teuthology.orchestra.run.smithi123.stdout: "num": 4662 2024-09-16T15:43:06.968 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:43:06.969 INFO:teuthology.orchestra.run.smithi123.stdout: "addr": { 2024-09-16T15:43:06.969 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "v1", 2024-09-16T15:43:06.969 INFO:teuthology.orchestra.run.smithi123.stdout: "addr": "192.168.0.1:0", 2024-09-16T15:43:06.969 INFO:teuthology.orchestra.run.smithi123.stdout: "nonce": 3062267695 2024-09-16T15:43:06.969 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-16T15:43:06.969 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:43:06.969 INFO:teuthology.orchestra.run.smithi123.stdout: "addr": { 2024-09-16T15:43:06.969 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "v1", 2024-09-16T15:43:06.969 INFO:teuthology.orchestra.run.smithi123.stdout: "addr": "192.168.0.1:0", 2024-09-16T15:43:06.969 INFO:teuthology.orchestra.run.smithi123.stdout: "nonce": 3062267695 2024-09-16T15:43:06.969 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:43:06.969 INFO:teuthology.orchestra.run.smithi123.stdout: "inst_str": "client.4662 192.168.0.1:0/3062267695", 2024-09-16T15:43:06.969 INFO:teuthology.orchestra.run.smithi123.stdout: "addr_str": "192.168.0.1:0/3062267695", 2024-09-16T15:43:06.969 INFO:teuthology.orchestra.run.smithi123.stdout: "inode_count": 1, 2024-09-16T15:43:06.969 INFO:teuthology.orchestra.run.smithi123.stdout: "mds_epoch": 8, 2024-09-16T15:43:06.970 INFO:teuthology.orchestra.run.smithi123.stdout: "osd_epoch": 21, 2024-09-16T15:43:06.970 INFO:teuthology.orchestra.run.smithi123.stdout: "osd_epoch_barrier": 0, 2024-09-16T15:43:06.970 INFO:teuthology.orchestra.run.smithi123.stdout: "blocklisted": false, 2024-09-16T15:43:06.970 INFO:teuthology.orchestra.run.smithi123.stdout: "fs_name": "cephfs" 2024-09-16T15:43:06.970 INFO:teuthology.orchestra.run.smithi123.stdout:} 2024-09-16T15:43:06.981 DEBUG:tasks.ceph_fuse:passing mntargs=[] 2024-09-16T15:43:06.982 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-09-16T15:43:07.113 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:07.110+0000 7f04a6be6640 1 -- 172.21.15.123:0/2036486716 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f0488003a60 msgr2=0x7f0488003e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:43:07.113 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:07.110+0000 7f04a6be6640 1 --2- 172.21.15.123:0/2036486716 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f0488003a60 0x7f0488003e80 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f04900016c0 tx=0x7f049001c710 comp rx=0 tx=0).stop 2024-09-16T15:43:07.113 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:07.110+0000 7f04a6be6640 1 -- 172.21.15.123:0/2036486716 shutdown_connections 2024-09-16T15:43:07.113 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:07.110+0000 7f04a6be6640 1 --2- 172.21.15.123:0/2036486716 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f0488003a60 0x7f0488003e80 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:07.113 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:07.110+0000 7f04a6be6640 1 -- 172.21.15.123:0/2036486716 >> 172.21.15.123:0/2036486716 conn(0x7f04a009aa40 msgr2=0x7f04a009ae50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:43:07.113 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:07.110+0000 7f04a6be6640 1 -- 172.21.15.123:0/2036486716 shutdown_connections 2024-09-16T15:43:07.113 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:07.110+0000 7f04a6be6640 1 -- 172.21.15.123:0/2036486716 wait complete. 2024-09-16T15:43:07.114 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:07.110+0000 7f04a6be6640 1 Processor -- start 2024-09-16T15:43:07.114 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:07.110+0000 7f04a6be6640 1 -- start start 2024-09-16T15:43:07.116 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:07.110+0000 7f04a6be6640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0488003a60 0x7f04a00b7670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:43:07.116 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:07.114+0000 7f04a6be6640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f04a014cc70 0x7f04a00c5ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:43:07.116 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:07.114+0000 7f04a6be6640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f04a014ec80 0x7f04a014f100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:43:07.116 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:07.114+0000 7f04a6be6640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f04a00d01f0 con 0x7f0488003a60 2024-09-16T15:43:07.116 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:07.114+0000 7f04a6be6640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f04a00cc800 con 0x7f04a014cc70 2024-09-16T15:43:07.116 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:07.114+0000 7f04a6be6640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f04a00d0370 con 0x7f04a014ec80 2024-09-16T15:43:07.116 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:07.114+0000 7f04a4982640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0488003a60 0x7f04a00b7670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:43:07.116 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:07.114+0000 7f04a4982640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0488003a60 0x7f04a00b7670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:41778/0 (socket says 172.21.15.123:41778) 2024-09-16T15:43:07.116 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:07.114+0000 7f04a4982640 1 -- 172.21.15.123:0/1359425681 learned_addr learned my addr 172.21.15.123:0/1359425681 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:43:07.116 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:07.114+0000 7f049ffff640 1 --2- 172.21.15.123:0/1359425681 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f04a014cc70 0x7f04a00c5ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:43:07.116 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:07.114+0000 7f04a5183640 1 --2- 172.21.15.123:0/1359425681 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f04a014ec80 0x7f04a014f100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:43:07.116 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:07.114+0000 7f04a4982640 1 -- 172.21.15.123:0/1359425681 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f04a014ec80 msgr2=0x7f04a014f100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:43:07.117 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:07.114+0000 7f04a4982640 1 --2- 172.21.15.123:0/1359425681 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f04a014ec80 0x7f04a014f100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:07.117 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:07.114+0000 7f04a4982640 1 -- 172.21.15.123:0/1359425681 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f04a014cc70 msgr2=0x7f04a00c5ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:43:07.117 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:07.114+0000 7f04a4982640 1 --2- 172.21.15.123:0/1359425681 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f04a014cc70 0x7f04a00c5ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:07.117 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:07.114+0000 7f04a4982640 1 -- 172.21.15.123:0/1359425681 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f04a015edb0 con 0x7f0488003a60 2024-09-16T15:43:07.117 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:07.114+0000 7f04a5183640 1 --2- 172.21.15.123:0/1359425681 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f04a014ec80 0x7f04a014f100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T15:43:07.117 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:07.114+0000 7f04a4982640 1 --2- 172.21.15.123:0/1359425681 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0488003a60 0x7f04a00b7670 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f0490006040 tx=0x7f0490003be0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:43:07.117 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:07.114+0000 7f049d7fa640 1 -- 172.21.15.123:0/1359425681 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04900ad070 con 0x7f0488003a60 2024-09-16T15:43:07.117 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:07.114+0000 7f049d7fa640 1 -- 172.21.15.123:0/1359425681 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f0490006500 con 0x7f0488003a60 2024-09-16T15:43:07.117 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:07.114+0000 7f04a6be6640 1 -- 172.21.15.123:0/1359425681 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f04a014f640 con 0x7f0488003a60 2024-09-16T15:43:07.117 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:07.114+0000 7f049d7fa640 1 -- 172.21.15.123:0/1359425681 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0490001310 con 0x7f0488003a60 2024-09-16T15:43:07.117 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:07.114+0000 7f04a6be6640 1 -- 172.21.15.123:0/1359425681 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f04a015f300 con 0x7f0488003a60 2024-09-16T15:43:07.118 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:07.114+0000 7f04a6be6640 1 -- 172.21.15.123:0/1359425681 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f04a00d1d20 con 0x7f0488003a60 2024-09-16T15:43:07.118 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:07.114+0000 7f049d7fa640 1 -- 172.21.15.123:0/1359425681 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 5) ==== 97080+0+0 (secure 0 0 0) 0x7f04900a7450 con 0x7f0488003a60 2024-09-16T15:43:07.118 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:07.114+0000 7f049d7fa640 1 --2- 172.21.15.123:0/1359425681 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f048c072ec0 0x7f048c075380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:43:07.118 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:07.114+0000 7f049d7fa640 1 -- 172.21.15.123:0/1359425681 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f049012dcf0 con 0x7f0488003a60 2024-09-16T15:43:07.118 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:07.114+0000 7f049ffff640 1 --2- 172.21.15.123:0/1359425681 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f048c072ec0 0x7f048c075380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:43:07.119 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:07.118+0000 7f049ffff640 1 --2- 172.21.15.123:0/1359425681 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f048c072ec0 0x7f048c075380 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f0494000fc0 tx=0x7f04940067b0 comp rx=0 tx=0).ready entity=mgr.4135 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:43:07.122 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:07.118+0000 7f049d7fa640 1 -- 172.21.15.123:0/1359425681 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f0490132070 con 0x7f0488003a60 2024-09-16T15:43:07.335 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:07.334+0000 7f04a6be6640 1 -- 172.21.15.123:0/1359425681 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f04a00c8920 con 0x7f0488003a60 2024-09-16T15:43:07.336 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:07.334+0000 7f049d7fa640 1 -- 172.21.15.123:0/1359425681 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v8) ==== 53+0+73 (secure 0 0 0) 0x7f0490016470 con 0x7f0488003a60 2024-09-16T15:43:07.336 INFO:teuthology.orchestra.run.smithi123.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-09-16T15:43:07.339 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:07.338+0000 7f04a6be6640 1 -- 172.21.15.123:0/1359425681 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f048c072ec0 msgr2=0x7f048c075380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:43:07.339 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:07.338+0000 7f04a6be6640 1 --2- 172.21.15.123:0/1359425681 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f048c072ec0 0x7f048c075380 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f0494000fc0 tx=0x7f04940067b0 comp rx=0 tx=0).stop 2024-09-16T15:43:07.339 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:07.338+0000 7f04a6be6640 1 -- 172.21.15.123:0/1359425681 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0488003a60 msgr2=0x7f04a00b7670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:43:07.339 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:07.338+0000 7f04a6be6640 1 --2- 172.21.15.123:0/1359425681 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0488003a60 0x7f04a00b7670 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f0490006040 tx=0x7f0490003be0 comp rx=0 tx=0).stop 2024-09-16T15:43:07.339 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:07.338+0000 7f04a6be6640 1 -- 172.21.15.123:0/1359425681 shutdown_connections 2024-09-16T15:43:07.339 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:07.338+0000 7f04a6be6640 1 --2- 172.21.15.123:0/1359425681 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f048c072ec0 0x7f048c075380 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:07.339 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:07.338+0000 7f04a6be6640 1 --2- 172.21.15.123:0/1359425681 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f04a014ec80 0x7f04a014f100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:07.339 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:07.338+0000 7f04a6be6640 1 --2- 172.21.15.123:0/1359425681 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f04a014cc70 0x7f04a00c5ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:07.339 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:07.338+0000 7f04a6be6640 1 --2- 172.21.15.123:0/1359425681 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0488003a60 0x7f04a00b7670 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:07.339 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:07.338+0000 7f04a6be6640 1 -- 172.21.15.123:0/1359425681 >> 172.21.15.123:0/1359425681 conn(0x7f04a009aa40 msgr2=0x7f04a006e000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:43:07.339 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:07.338+0000 7f04a6be6640 1 -- 172.21.15.123:0/1359425681 shutdown_connections 2024-09-16T15:43:07.339 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:07.338+0000 7f04a6be6640 1 -- 172.21.15.123:0/1359425681 wait complete. 2024-09-16T15:43:07.355 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-09-16T15:43:07.355 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-09-16T15:43:07.356 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi123.front.sepia.ceph.com 2024-09-16T15:43:07.356 INFO:tasks.cephfs.mount:self.client.name = client.2 2024-09-16T15:43:07.356 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.2 2024-09-16T15:43:07.356 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-09-16T15:43:07.356 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-09-16T15:43:07.356 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-09-16T15:43:07.356 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.2' 2024-09-16T15:43:07.356 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:43:07.356 DEBUG:teuthology.orchestra.run.smithi123:> ip addr 2024-09-16T15:43:07.362 INFO:teuthology.orchestra.run.smithi123.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-09-16T15:43:07.362 INFO:teuthology.orchestra.run.smithi123.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-09-16T15:43:07.362 INFO:teuthology.orchestra.run.smithi123.stdout: inet 127.0.0.1/8 scope host lo 2024-09-16T15:43:07.363 INFO:teuthology.orchestra.run.smithi123.stdout: valid_lft forever preferred_lft forever 2024-09-16T15:43:07.363 INFO:teuthology.orchestra.run.smithi123.stdout: inet6 ::1/128 scope host 2024-09-16T15:43:07.363 INFO:teuthology.orchestra.run.smithi123.stdout: valid_lft forever preferred_lft forever 2024-09-16T15:43:07.363 INFO:teuthology.orchestra.run.smithi123.stdout:2: eno1: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-09-16T15:43:07.363 INFO:teuthology.orchestra.run.smithi123.stdout: link/ether 0c:c4:7a:d8:da:6c brd ff:ff:ff:ff:ff:ff 2024-09-16T15:43:07.363 INFO:teuthology.orchestra.run.smithi123.stdout: altname enp7s0f0 2024-09-16T15:43:07.363 INFO:teuthology.orchestra.run.smithi123.stdout:3: eno2: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-09-16T15:43:07.363 INFO:teuthology.orchestra.run.smithi123.stdout: link/ether 0c:c4:7a:d8:da:6d brd ff:ff:ff:ff:ff:ff 2024-09-16T15:43:07.363 INFO:teuthology.orchestra.run.smithi123.stdout: altname enp7s0f1 2024-09-16T15:43:07.363 INFO:teuthology.orchestra.run.smithi123.stdout:4: enp3s0f0: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-09-16T15:43:07.363 INFO:teuthology.orchestra.run.smithi123.stdout: link/ether 0c:c4:7a:8f:d3:f0 brd ff:ff:ff:ff:ff:ff 2024-09-16T15:43:07.363 INFO:teuthology.orchestra.run.smithi123.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-09-16T15:43:07.364 INFO:teuthology.orchestra.run.smithi123.stdout: link/ether 0c:c4:7a:8f:d3:f1 brd ff:ff:ff:ff:ff:ff 2024-09-16T15:43:07.364 INFO:teuthology.orchestra.run.smithi123.stdout: inet 172.21.15.123/20 metric 100 brd 172.21.15.255 scope global dynamic enp3s0f1 2024-09-16T15:43:07.364 INFO:teuthology.orchestra.run.smithi123.stdout: valid_lft 42481sec preferred_lft 42481sec 2024-09-16T15:43:07.364 INFO:teuthology.orchestra.run.smithi123.stdout: inet6 fe80::ec4:7aff:fe8f:d3f1/64 scope link 2024-09-16T15:43:07.364 INFO:teuthology.orchestra.run.smithi123.stdout: valid_lft forever preferred_lft forever 2024-09-16T15:43:07.364 INFO:teuthology.orchestra.run.smithi123.stdout:6: docker0: mtu 1500 qdisc noqueue state DOWN group default 2024-09-16T15:43:07.364 INFO:teuthology.orchestra.run.smithi123.stdout: link/ether 02:42:dd:82:a2:6e brd ff:ff:ff:ff:ff:ff 2024-09-16T15:43:07.364 INFO:teuthology.orchestra.run.smithi123.stdout: inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0 2024-09-16T15:43:07.364 INFO:teuthology.orchestra.run.smithi123.stdout: valid_lft forever preferred_lft forever 2024-09-16T15:43:07.364 INFO:teuthology.orchestra.run.smithi123.stdout:7: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-09-16T15:43:07.364 INFO:teuthology.orchestra.run.smithi123.stdout: link/ether 5e:1f:84:6d:79:ae brd ff:ff:ff:ff:ff:ff 2024-09-16T15:43:07.364 INFO:teuthology.orchestra.run.smithi123.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-09-16T15:43:07.364 INFO:teuthology.orchestra.run.smithi123.stdout: valid_lft forever preferred_lft forever 2024-09-16T15:43:07.364 INFO:teuthology.orchestra.run.smithi123.stdout: inet6 fe80::5c1f:84ff:fe6d:79ae/64 scope link 2024-09-16T15:43:07.365 INFO:teuthology.orchestra.run.smithi123.stdout: valid_lft forever preferred_lft forever 2024-09-16T15:43:07.365 INFO:teuthology.orchestra.run.smithi123.stdout:8: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-09-16T15:43:07.365 INFO:teuthology.orchestra.run.smithi123.stdout: link/ether 0e:5c:f6:1f:82:2f brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-09-16T15:43:07.365 INFO:teuthology.orchestra.run.smithi123.stdout: inet6 fe80::c5c:f6ff:fe1f:822f/64 scope link 2024-09-16T15:43:07.365 INFO:teuthology.orchestra.run.smithi123.stdout: valid_lft forever preferred_lft forever 2024-09-16T15:43:07.365 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:43:07.365 DEBUG:teuthology.orchestra.run.smithi123:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-09-16T15:43:07.416 INFO:teuthology.orchestra.run.smithi123.stdout:1 2024-09-16T15:43:07.418 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:43:07.418 DEBUG:teuthology.orchestra.run.smithi123:> route 2024-09-16T15:43:07.471 INFO:teuthology.orchestra.run.smithi123.stdout:Kernel IP routing table 2024-09-16T15:43:07.472 INFO:teuthology.orchestra.run.smithi123.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-09-16T15:43:07.472 INFO:teuthology.orchestra.run.smithi123.stdout:default _gateway 0.0.0.0 UG 0 0 0 enp3s0f1 2024-09-16T15:43:07.472 INFO:teuthology.orchestra.run.smithi123.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-09-16T15:43:07.472 INFO:teuthology.orchestra.run.smithi123.stdout:158.69.67.47 _gateway 255.255.255.255 UGH 100 0 0 enp3s0f1 2024-09-16T15:43:07.472 INFO:teuthology.orchestra.run.smithi123.stdout:172.17.0.0 0.0.0.0 255.255.0.0 U 0 0 0 docker0 2024-09-16T15:43:07.472 INFO:teuthology.orchestra.run.smithi123.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-09-16T15:43:07.472 INFO:teuthology.orchestra.run.smithi123.stdout:ns1.front.sepia 0.0.0.0 255.255.255.255 UH 100 0 0 enp3s0f1 2024-09-16T15:43:07.472 INFO:teuthology.orchestra.run.smithi123.stdout:172.21.0.2 0.0.0.0 255.255.255.255 UH 100 0 0 enp3s0f1 2024-09-16T15:43:07.472 INFO:teuthology.orchestra.run.smithi123.stdout:_gateway 0.0.0.0 255.255.255.255 UH 100 0 0 enp3s0f1 2024-09-16T15:43:07.472 INFO:teuthology.orchestra.run.smithi123.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-09-16T15:43:07.473 DEBUG:teuthology.orchestra.run.smithi123:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-09-16T15:43:07.473 DEBUG:teuthology.orchestra.run.smithi123:> set -e 2024-09-16T15:43:07.473 DEBUG:teuthology.orchestra.run.smithi123:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-09-16T15:43:07.473 DEBUG:teuthology.orchestra.run.smithi123:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-09-16T15:43:07.473 DEBUG:teuthology.orchestra.run.smithi123:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-09-16T15:43:07.473 DEBUG:teuthology.orchestra.run.smithi123:> ') 2024-09-16T15:43:07.561 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:07 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-09-16T15:43:07.590 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:07 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-16T15:43:07.598 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:43:07.598 DEBUG:teuthology.orchestra.run.smithi123:> ip netns list 2024-09-16T15:43:07.604 INFO:teuthology.orchestra.run.smithi123.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-09-16T15:43:07.604 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:43:07.604 DEBUG:teuthology.orchestra.run.smithi123:> ip netns list-id 2024-09-16T15:43:07.651 INFO:teuthology.orchestra.run.smithi123.stdout:nsid 0 (iproute2 netns name: ceph-ns--home-ubuntu-cephtest-mnt.0) 2024-09-16T15:43:07.651 DEBUG:teuthology.orchestra.run.smithi123:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-09-16T15:43:07.651 DEBUG:teuthology.orchestra.run.smithi123:> set -e 2024-09-16T15:43:07.651 DEBUG:teuthology.orchestra.run.smithi123:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.2 2024-09-16T15:43:07.651 DEBUG:teuthology.orchestra.run.smithi123:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.2 1 2024-09-16T15:43:07.651 DEBUG:teuthology.orchestra.run.smithi123:> ') 2024-09-16T15:43:07.740 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:07 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-09-16T15:43:07.761 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:07 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-16T15:43:07.768 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:43:07.769 DEBUG:teuthology.orchestra.run.smithi123:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr 2024-09-16T15:43:07.780 INFO:teuthology.orchestra.run.smithi123.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-09-16T15:43:07.780 INFO:teuthology.orchestra.run.smithi123.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-09-16T15:43:07.780 INFO:teuthology.orchestra.run.smithi123.stdout: inet 127.0.0.1/8 scope host lo 2024-09-16T15:43:07.780 INFO:teuthology.orchestra.run.smithi123.stdout: valid_lft forever preferred_lft forever 2024-09-16T15:43:07.780 INFO:teuthology.orchestra.run.smithi123.stdout: inet6 ::1/128 scope host 2024-09-16T15:43:07.780 INFO:teuthology.orchestra.run.smithi123.stdout: valid_lft forever preferred_lft forever 2024-09-16T15:43:07.780 INFO:teuthology.orchestra.run.smithi123.stdout:2: veth0@if8: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-09-16T15:43:07.780 INFO:teuthology.orchestra.run.smithi123.stdout: link/ether 6a:06:60:99:c0:9a brd ff:ff:ff:ff:ff:ff link-netnsid 0 2024-09-16T15:43:07.780 INFO:teuthology.orchestra.run.smithi123.stdout: inet 192.168.0.1/16 brd 192.168.255.255 scope global veth0 2024-09-16T15:43:07.781 INFO:teuthology.orchestra.run.smithi123.stdout: valid_lft forever preferred_lft forever 2024-09-16T15:43:07.781 INFO:teuthology.orchestra.run.smithi123.stdout: inet6 fe80::6806:60ff:fe99:c09a/64 scope link 2024-09-16T15:43:07.781 INFO:teuthology.orchestra.run.smithi123.stdout: valid_lft forever preferred_lft forever 2024-09-16T15:43:07.782 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:43:07.782 DEBUG:teuthology.orchestra.run.smithi123:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr 2024-09-16T15:43:07.833 INFO:teuthology.orchestra.run.smithi123.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-09-16T15:43:07.833 INFO:teuthology.orchestra.run.smithi123.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-09-16T15:43:07.833 INFO:teuthology.orchestra.run.smithi123.stdout: inet 127.0.0.1/8 scope host lo 2024-09-16T15:43:07.833 INFO:teuthology.orchestra.run.smithi123.stdout: valid_lft forever preferred_lft forever 2024-09-16T15:43:07.833 INFO:teuthology.orchestra.run.smithi123.stdout: inet6 ::1/128 scope host 2024-09-16T15:43:07.833 INFO:teuthology.orchestra.run.smithi123.stdout: valid_lft forever preferred_lft forever 2024-09-16T15:43:07.833 INFO:teuthology.orchestra.run.smithi123.stdout:2: veth0@if8: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-09-16T15:43:07.833 INFO:teuthology.orchestra.run.smithi123.stdout: link/ether 6a:06:60:99:c0:9a brd ff:ff:ff:ff:ff:ff link-netnsid 0 2024-09-16T15:43:07.833 INFO:teuthology.orchestra.run.smithi123.stdout: inet 192.168.0.1/16 brd 192.168.255.255 scope global veth0 2024-09-16T15:43:07.833 INFO:teuthology.orchestra.run.smithi123.stdout: valid_lft forever preferred_lft forever 2024-09-16T15:43:07.834 INFO:teuthology.orchestra.run.smithi123.stdout: inet6 fe80::6806:60ff:fe99:c09a/64 scope link 2024-09-16T15:43:07.834 INFO:teuthology.orchestra.run.smithi123.stdout: valid_lft forever preferred_lft forever 2024-09-16T15:43:07.835 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.2' with 192.168.0.2/16 2024-09-16T15:43:07.835 DEBUG:teuthology.orchestra.run.smithi123:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-09-16T15:43:07.835 DEBUG:teuthology.orchestra.run.smithi123:> set -e 2024-09-16T15:43:07.835 DEBUG:teuthology.orchestra.run.smithi123:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.2 type veth peer name brx.1 2024-09-16T15:43:07.835 DEBUG:teuthology.orchestra.run.smithi123:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.2 ip addr add 192.168.0.2/16 brd 192.168.255.255 dev veth0 2024-09-16T15:43:07.835 DEBUG:teuthology.orchestra.run.smithi123:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.2 ip link set veth0 up 2024-09-16T15:43:07.835 DEBUG:teuthology.orchestra.run.smithi123:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.2 ip link set lo up 2024-09-16T15:43:07.835 DEBUG:teuthology.orchestra.run.smithi123:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.2 ip route add default via 192.168.255.254 2024-09-16T15:43:07.835 DEBUG:teuthology.orchestra.run.smithi123:> ') 2024-09-16T15:43:07.923 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:07 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-09-16T15:43:07.968 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:07 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-16T15:43:07.975 DEBUG:teuthology.orchestra.run.smithi123:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-09-16T15:43:07.975 DEBUG:teuthology.orchestra.run.smithi123:> set -e 2024-09-16T15:43:07.975 DEBUG:teuthology.orchestra.run.smithi123:> sudo ip link set brx.1 up 2024-09-16T15:43:07.975 DEBUG:teuthology.orchestra.run.smithi123:> sudo ip link set dev brx.1 master ceph-brx 2024-09-16T15:43:07.975 DEBUG:teuthology.orchestra.run.smithi123:> ') 2024-09-16T15:43:08.014 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:08 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-09-16T15:43:08.029 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:08 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-16T15:43:08.035 INFO:tasks.cephfs.fuse_mount:Client client.2 config is {} 2024-09-16T15:43:08.036 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-09-16T15:43:08.036 DEBUG:teuthology.orchestra.run.smithi123:> mkdir -p -v /home/ubuntu/cephtest/mnt.2 2024-09-16T15:43:08.082 INFO:teuthology.orchestra.run.smithi123.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.2' 2024-09-16T15:43:08.083 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-09-16T15:43:08.083 DEBUG:teuthology.orchestra.run.smithi123:> chmod 0000 /home/ubuntu/cephtest/mnt.2 2024-09-16T15:43:08.130 DEBUG:teuthology.orchestra.run.smithi123:> sudo modprobe fuse 2024-09-16T15:43:08.192 DEBUG:teuthology.orchestra.run.smithi123:> cat /proc/self/mounts | awk '{print $2}' 2024-09-16T15:43:08.241 INFO:teuthology.orchestra.run.smithi123.stdout:/sys 2024-09-16T15:43:08.241 INFO:teuthology.orchestra.run.smithi123.stdout:/proc 2024-09-16T15:43:08.241 INFO:teuthology.orchestra.run.smithi123.stdout:/dev 2024-09-16T15:43:08.241 INFO:teuthology.orchestra.run.smithi123.stdout:/dev/pts 2024-09-16T15:43:08.241 INFO:teuthology.orchestra.run.smithi123.stdout:/run 2024-09-16T15:43:08.241 INFO:teuthology.orchestra.run.smithi123.stdout:/ 2024-09-16T15:43:08.242 INFO:teuthology.orchestra.run.smithi123.stdout:/sys/kernel/security 2024-09-16T15:43:08.242 INFO:teuthology.orchestra.run.smithi123.stdout:/dev/shm 2024-09-16T15:43:08.242 INFO:teuthology.orchestra.run.smithi123.stdout:/run/lock 2024-09-16T15:43:08.242 INFO:teuthology.orchestra.run.smithi123.stdout:/sys/fs/cgroup 2024-09-16T15:43:08.242 INFO:teuthology.orchestra.run.smithi123.stdout:/sys/fs/pstore 2024-09-16T15:43:08.242 INFO:teuthology.orchestra.run.smithi123.stdout:/sys/fs/bpf 2024-09-16T15:43:08.242 INFO:teuthology.orchestra.run.smithi123.stdout:/proc/sys/fs/binfmt_misc 2024-09-16T15:43:08.242 INFO:teuthology.orchestra.run.smithi123.stdout:/dev/hugepages 2024-09-16T15:43:08.242 INFO:teuthology.orchestra.run.smithi123.stdout:/dev/mqueue 2024-09-16T15:43:08.242 INFO:teuthology.orchestra.run.smithi123.stdout:/sys/kernel/debug 2024-09-16T15:43:08.242 INFO:teuthology.orchestra.run.smithi123.stdout:/sys/kernel/tracing 2024-09-16T15:43:08.242 INFO:teuthology.orchestra.run.smithi123.stdout:/proc/fs/nfsd 2024-09-16T15:43:08.242 INFO:teuthology.orchestra.run.smithi123.stdout:/sys/fs/fuse/connections 2024-09-16T15:43:08.242 INFO:teuthology.orchestra.run.smithi123.stdout:/sys/kernel/config 2024-09-16T15:43:08.243 INFO:teuthology.orchestra.run.smithi123.stdout:/run/credentials/systemd-sysusers.service 2024-09-16T15:43:08.243 INFO:teuthology.orchestra.run.smithi123.stdout:/run/rpc_pipefs 2024-09-16T15:43:08.243 INFO:teuthology.orchestra.run.smithi123.stdout:/run/user/1000 2024-09-16T15:43:08.243 INFO:teuthology.orchestra.run.smithi123.stdout:/var/lib/ceph 2024-09-16T15:43:08.243 INFO:teuthology.orchestra.run.smithi123.stdout:/var/lib/ceph/osd/ceph-0 2024-09-16T15:43:08.243 INFO:teuthology.orchestra.run.smithi123.stdout:/var/lib/ceph/osd/ceph-1 2024-09-16T15:43:08.243 INFO:teuthology.orchestra.run.smithi123.stdout:/var/lib/ceph/osd/ceph-2 2024-09-16T15:43:08.243 INFO:teuthology.orchestra.run.smithi123.stdout:/var/lib/ceph/osd/ceph-3 2024-09-16T15:43:08.243 INFO:teuthology.orchestra.run.smithi123.stdout:/run/netns 2024-09-16T15:43:08.243 INFO:teuthology.orchestra.run.smithi123.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-09-16T15:43:08.243 INFO:teuthology.orchestra.run.smithi123.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-09-16T15:43:08.243 INFO:teuthology.orchestra.run.smithi123.stdout:/home/ubuntu/cephtest/mnt.0 2024-09-16T15:43:08.243 INFO:teuthology.orchestra.run.smithi123.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-09-16T15:43:08.244 INFO:teuthology.orchestra.run.smithi123.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-09-16T15:43:08.244 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:43:08.244 DEBUG:teuthology.orchestra.run.smithi123:> ls /sys/fs/fuse/connections 2024-09-16T15:43:08.295 INFO:teuthology.orchestra.run.smithi123.stdout:45 2024-09-16T15:43:08.295 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [45] 2024-09-16T15:43:08.296 DEBUG:teuthology.orchestra.run.smithi123:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.2 --id 2) 2024-09-16T15:43:08.340 DEBUG:teuthology.orchestra.run.smithi123:> sudo modprobe fuse 2024-09-16T15:43:08.354 DEBUG:teuthology.orchestra.run.smithi123:> cat /proc/self/mounts | awk '{print $2}' 2024-09-16T15:43:08.404 INFO:teuthology.orchestra.run.smithi123.stdout:/sys 2024-09-16T15:43:08.404 INFO:teuthology.orchestra.run.smithi123.stdout:/proc 2024-09-16T15:43:08.404 INFO:teuthology.orchestra.run.smithi123.stdout:/dev 2024-09-16T15:43:08.404 INFO:teuthology.orchestra.run.smithi123.stdout:/dev/pts 2024-09-16T15:43:08.404 INFO:teuthology.orchestra.run.smithi123.stdout:/run 2024-09-16T15:43:08.404 INFO:teuthology.orchestra.run.smithi123.stdout:/ 2024-09-16T15:43:08.404 INFO:teuthology.orchestra.run.smithi123.stdout:/sys/kernel/security 2024-09-16T15:43:08.404 INFO:teuthology.orchestra.run.smithi123.stdout:/dev/shm 2024-09-16T15:43:08.404 INFO:teuthology.orchestra.run.smithi123.stdout:/run/lock 2024-09-16T15:43:08.405 INFO:teuthology.orchestra.run.smithi123.stdout:/sys/fs/cgroup 2024-09-16T15:43:08.405 INFO:teuthology.orchestra.run.smithi123.stdout:/sys/fs/pstore 2024-09-16T15:43:08.405 INFO:teuthology.orchestra.run.smithi123.stdout:/sys/fs/bpf 2024-09-16T15:43:08.405 INFO:teuthology.orchestra.run.smithi123.stdout:/proc/sys/fs/binfmt_misc 2024-09-16T15:43:08.405 INFO:teuthology.orchestra.run.smithi123.stdout:/dev/hugepages 2024-09-16T15:43:08.405 INFO:teuthology.orchestra.run.smithi123.stdout:/dev/mqueue 2024-09-16T15:43:08.405 INFO:teuthology.orchestra.run.smithi123.stdout:/sys/kernel/debug 2024-09-16T15:43:08.405 INFO:teuthology.orchestra.run.smithi123.stdout:/sys/kernel/tracing 2024-09-16T15:43:08.405 INFO:teuthology.orchestra.run.smithi123.stdout:/proc/fs/nfsd 2024-09-16T15:43:08.405 INFO:teuthology.orchestra.run.smithi123.stdout:/sys/fs/fuse/connections 2024-09-16T15:43:08.405 INFO:teuthology.orchestra.run.smithi123.stdout:/sys/kernel/config 2024-09-16T15:43:08.405 INFO:teuthology.orchestra.run.smithi123.stdout:/run/credentials/systemd-sysusers.service 2024-09-16T15:43:08.406 INFO:teuthology.orchestra.run.smithi123.stdout:/run/rpc_pipefs 2024-09-16T15:43:08.406 INFO:teuthology.orchestra.run.smithi123.stdout:/run/user/1000 2024-09-16T15:43:08.406 INFO:teuthology.orchestra.run.smithi123.stdout:/var/lib/ceph 2024-09-16T15:43:08.406 INFO:teuthology.orchestra.run.smithi123.stdout:/var/lib/ceph/osd/ceph-0 2024-09-16T15:43:08.406 INFO:teuthology.orchestra.run.smithi123.stdout:/var/lib/ceph/osd/ceph-1 2024-09-16T15:43:08.406 INFO:teuthology.orchestra.run.smithi123.stdout:/var/lib/ceph/osd/ceph-2 2024-09-16T15:43:08.406 INFO:teuthology.orchestra.run.smithi123.stdout:/var/lib/ceph/osd/ceph-3 2024-09-16T15:43:08.406 INFO:teuthology.orchestra.run.smithi123.stdout:/run/netns 2024-09-16T15:43:08.406 INFO:teuthology.orchestra.run.smithi123.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-09-16T15:43:08.406 INFO:teuthology.orchestra.run.smithi123.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-09-16T15:43:08.406 INFO:teuthology.orchestra.run.smithi123.stdout:/home/ubuntu/cephtest/mnt.0 2024-09-16T15:43:08.406 INFO:teuthology.orchestra.run.smithi123.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-09-16T15:43:08.406 INFO:teuthology.orchestra.run.smithi123.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-09-16T15:43:08.407 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:43:08.407 DEBUG:teuthology.orchestra.run.smithi123:> ls /sys/fs/fuse/connections 2024-09-16T15:43:08.417 INFO:tasks.cephfs.fuse_mount.ceph-fuse.2.smithi123.stderr:2024-09-16T15:43:08.414+0000 7f3ffe10e580 -1 init, newargv = 0x562f1e7022e0 newargc=15 2024-09-16T15:43:08.418 INFO:tasks.cephfs.fuse_mount.ceph-fuse.2.smithi123.stderr:ceph-fuse[36383]: starting ceph client 2024-09-16T15:43:08.421 INFO:teuthology.orchestra.run.smithi123.stdout:45 2024-09-16T15:43:08.425 INFO:tasks.cephfs.fuse_mount.ceph-fuse.2.smithi123.stderr:ceph-fuse[36383]: starting fuse 2024-09-16T15:43:08.455 DEBUG:teuthology.orchestra.run.smithi123:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-16T15:43:08.457 DEBUG:teuthology.orchestra.run.smithi184:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-16T15:43:09.523 DEBUG:teuthology.orchestra.run.smithi123:> sudo modprobe fuse 2024-09-16T15:43:09.540 DEBUG:teuthology.orchestra.run.smithi123:> cat /proc/self/mounts | awk '{print $2}' 2024-09-16T15:43:09.592 INFO:teuthology.orchestra.run.smithi123.stdout:/sys 2024-09-16T15:43:09.592 INFO:teuthology.orchestra.run.smithi123.stdout:/proc 2024-09-16T15:43:09.592 INFO:teuthology.orchestra.run.smithi123.stdout:/dev 2024-09-16T15:43:09.592 INFO:teuthology.orchestra.run.smithi123.stdout:/dev/pts 2024-09-16T15:43:09.593 INFO:teuthology.orchestra.run.smithi123.stdout:/run 2024-09-16T15:43:09.593 INFO:teuthology.orchestra.run.smithi123.stdout:/ 2024-09-16T15:43:09.593 INFO:teuthology.orchestra.run.smithi123.stdout:/sys/kernel/security 2024-09-16T15:43:09.593 INFO:teuthology.orchestra.run.smithi123.stdout:/dev/shm 2024-09-16T15:43:09.593 INFO:teuthology.orchestra.run.smithi123.stdout:/run/lock 2024-09-16T15:43:09.593 INFO:teuthology.orchestra.run.smithi123.stdout:/sys/fs/cgroup 2024-09-16T15:43:09.593 INFO:teuthology.orchestra.run.smithi123.stdout:/sys/fs/pstore 2024-09-16T15:43:09.593 INFO:teuthology.orchestra.run.smithi123.stdout:/sys/fs/bpf 2024-09-16T15:43:09.593 INFO:teuthology.orchestra.run.smithi123.stdout:/proc/sys/fs/binfmt_misc 2024-09-16T15:43:09.593 INFO:teuthology.orchestra.run.smithi123.stdout:/dev/hugepages 2024-09-16T15:43:09.594 INFO:teuthology.orchestra.run.smithi123.stdout:/dev/mqueue 2024-09-16T15:43:09.594 INFO:teuthology.orchestra.run.smithi123.stdout:/sys/kernel/debug 2024-09-16T15:43:09.594 INFO:teuthology.orchestra.run.smithi123.stdout:/sys/kernel/tracing 2024-09-16T15:43:09.594 INFO:teuthology.orchestra.run.smithi123.stdout:/proc/fs/nfsd 2024-09-16T15:43:09.594 INFO:teuthology.orchestra.run.smithi123.stdout:/sys/fs/fuse/connections 2024-09-16T15:43:09.594 INFO:teuthology.orchestra.run.smithi123.stdout:/sys/kernel/config 2024-09-16T15:43:09.594 INFO:teuthology.orchestra.run.smithi123.stdout:/run/credentials/systemd-sysusers.service 2024-09-16T15:43:09.594 INFO:teuthology.orchestra.run.smithi123.stdout:/run/rpc_pipefs 2024-09-16T15:43:09.594 INFO:teuthology.orchestra.run.smithi123.stdout:/run/user/1000 2024-09-16T15:43:09.594 INFO:teuthology.orchestra.run.smithi123.stdout:/var/lib/ceph 2024-09-16T15:43:09.594 INFO:teuthology.orchestra.run.smithi123.stdout:/var/lib/ceph/osd/ceph-0 2024-09-16T15:43:09.594 INFO:teuthology.orchestra.run.smithi123.stdout:/var/lib/ceph/osd/ceph-1 2024-09-16T15:43:09.595 INFO:teuthology.orchestra.run.smithi123.stdout:/var/lib/ceph/osd/ceph-2 2024-09-16T15:43:09.595 INFO:teuthology.orchestra.run.smithi123.stdout:/var/lib/ceph/osd/ceph-3 2024-09-16T15:43:09.595 INFO:teuthology.orchestra.run.smithi123.stdout:/run/netns 2024-09-16T15:43:09.595 INFO:teuthology.orchestra.run.smithi123.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-09-16T15:43:09.595 INFO:teuthology.orchestra.run.smithi123.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-09-16T15:43:09.595 INFO:teuthology.orchestra.run.smithi123.stdout:/home/ubuntu/cephtest/mnt.0 2024-09-16T15:43:09.595 INFO:teuthology.orchestra.run.smithi123.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-09-16T15:43:09.595 INFO:teuthology.orchestra.run.smithi123.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-09-16T15:43:09.595 INFO:teuthology.orchestra.run.smithi123.stdout:/home/ubuntu/cephtest/mnt.2 2024-09-16T15:43:09.596 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:43:09.596 DEBUG:teuthology.orchestra.run.smithi123:> ls /sys/fs/fuse/connections 2024-09-16T15:43:09.643 INFO:teuthology.orchestra.run.smithi123.stdout:45 2024-09-16T15:43:09.643 INFO:teuthology.orchestra.run.smithi123.stdout:46 2024-09-16T15:43:09.643 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [45, 46] 2024-09-16T15:43:09.643 DEBUG:teuthology.orchestra.run.smithi123:> sudo stdin-killer -- python3 -c ' 2024-09-16T15:43:09.643 DEBUG:teuthology.orchestra.run.smithi123:> import glob 2024-09-16T15:43:09.643 DEBUG:teuthology.orchestra.run.smithi123:> import re 2024-09-16T15:43:09.644 DEBUG:teuthology.orchestra.run.smithi123:> import os 2024-09-16T15:43:09.644 DEBUG:teuthology.orchestra.run.smithi123:> import subprocess 2024-09-16T15:43:09.644 DEBUG:teuthology.orchestra.run.smithi123:> 2024-09-16T15:43:09.644 DEBUG:teuthology.orchestra.run.smithi123:> def _find_admin_socket(client_name): 2024-09-16T15:43:09.644 DEBUG:teuthology.orchestra.run.smithi123:> asok_path = "/var/run/ceph/ceph-client.2.*.asok" 2024-09-16T15:43:09.644 DEBUG:teuthology.orchestra.run.smithi123:> files = glob.glob(asok_path) 2024-09-16T15:43:09.644 DEBUG:teuthology.orchestra.run.smithi123:> mountpoint = "/home/ubuntu/cephtest/mnt.2" 2024-09-16T15:43:09.644 DEBUG:teuthology.orchestra.run.smithi123:> 2024-09-16T15:43:09.644 DEBUG:teuthology.orchestra.run.smithi123:> # Given a non-glob path, it better be there 2024-09-16T15:43:09.644 DEBUG:teuthology.orchestra.run.smithi123:> if "*" not in asok_path: 2024-09-16T15:43:09.644 DEBUG:teuthology.orchestra.run.smithi123:> assert(len(files) == 1) 2024-09-16T15:43:09.644 DEBUG:teuthology.orchestra.run.smithi123:> return files[0] 2024-09-16T15:43:09.645 DEBUG:teuthology.orchestra.run.smithi123:> 2024-09-16T15:43:09.645 DEBUG:teuthology.orchestra.run.smithi123:> for f in files: 2024-09-16T15:43:09.645 DEBUG:teuthology.orchestra.run.smithi123:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-09-16T15:43:09.645 DEBUG:teuthology.orchestra.run.smithi123:> if os.path.exists("/proc/{0}".format(pid)): 2024-09-16T15:43:09.645 DEBUG:teuthology.orchestra.run.smithi123:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-09-16T15:43:09.645 DEBUG:teuthology.orchestra.run.smithi123:> contents = proc_f.read() 2024-09-16T15:43:09.645 DEBUG:teuthology.orchestra.run.smithi123:> if mountpoint in contents: 2024-09-16T15:43:09.645 DEBUG:teuthology.orchestra.run.smithi123:> return f 2024-09-16T15:43:09.645 DEBUG:teuthology.orchestra.run.smithi123:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-09-16T15:43:09.645 DEBUG:teuthology.orchestra.run.smithi123:> 2024-09-16T15:43:09.645 DEBUG:teuthology.orchestra.run.smithi123:> print(_find_admin_socket("client.2")) 2024-09-16T15:43:09.645 DEBUG:teuthology.orchestra.run.smithi123:> ' 2024-09-16T15:43:09.766 INFO:teuthology.orchestra.run.smithi123.stdout:/var/run/ceph/ceph-client.2.36383.asok 2024-09-16T15:43:09.770 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:09 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-16T15:43:09.778 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.2.36383.asok 2024-09-16T15:43:09.778 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:43:09.778 DEBUG:teuthology.orchestra.run.smithi123:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.2.36383.asok status 2024-09-16T15:43:09.930 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2024-09-16T15:43:09.930 INFO:teuthology.orchestra.run.smithi123.stdout: "metadata": { 2024-09-16T15:43:09.930 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph_sha1": "33f95d2ae1bcbe7eb04404303f179eeeaa6982df", 2024-09-16T15:43:09.930 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph_version": "ceph version 19.3.0-4938-g33f95d2a (33f95d2ae1bcbe7eb04404303f179eeeaa6982df) squid (dev)", 2024-09-16T15:43:09.930 INFO:teuthology.orchestra.run.smithi123.stdout: "entity_id": "2", 2024-09-16T15:43:09.931 INFO:teuthology.orchestra.run.smithi123.stdout: "hostname": "smithi123", 2024-09-16T15:43:09.931 INFO:teuthology.orchestra.run.smithi123.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.2", 2024-09-16T15:43:09.931 INFO:teuthology.orchestra.run.smithi123.stdout: "pid": "36383", 2024-09-16T15:43:09.931 INFO:teuthology.orchestra.run.smithi123.stdout: "root": "/" 2024-09-16T15:43:09.931 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:43:09.931 INFO:teuthology.orchestra.run.smithi123.stdout: "dentry_count": 0, 2024-09-16T15:43:09.931 INFO:teuthology.orchestra.run.smithi123.stdout: "dentry_pinned_count": 0, 2024-09-16T15:43:09.931 INFO:teuthology.orchestra.run.smithi123.stdout: "id": 4674, 2024-09-16T15:43:09.931 INFO:teuthology.orchestra.run.smithi123.stdout: "inst": { 2024-09-16T15:43:09.931 INFO:teuthology.orchestra.run.smithi123.stdout: "name": { 2024-09-16T15:43:09.931 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "client", 2024-09-16T15:43:09.931 INFO:teuthology.orchestra.run.smithi123.stdout: "num": 4674 2024-09-16T15:43:09.931 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:43:09.931 INFO:teuthology.orchestra.run.smithi123.stdout: "addr": { 2024-09-16T15:43:09.931 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "v1", 2024-09-16T15:43:09.932 INFO:teuthology.orchestra.run.smithi123.stdout: "addr": "192.168.0.2:0", 2024-09-16T15:43:09.932 INFO:teuthology.orchestra.run.smithi123.stdout: "nonce": 1210794849 2024-09-16T15:43:09.932 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-16T15:43:09.932 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:43:09.932 INFO:teuthology.orchestra.run.smithi123.stdout: "addr": { 2024-09-16T15:43:09.932 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "v1", 2024-09-16T15:43:09.932 INFO:teuthology.orchestra.run.smithi123.stdout: "addr": "192.168.0.2:0", 2024-09-16T15:43:09.932 INFO:teuthology.orchestra.run.smithi123.stdout: "nonce": 1210794849 2024-09-16T15:43:09.932 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:43:09.932 INFO:teuthology.orchestra.run.smithi123.stdout: "inst_str": "client.4674 192.168.0.2:0/1210794849", 2024-09-16T15:43:09.932 INFO:teuthology.orchestra.run.smithi123.stdout: "addr_str": "192.168.0.2:0/1210794849", 2024-09-16T15:43:09.932 INFO:teuthology.orchestra.run.smithi123.stdout: "inode_count": 1, 2024-09-16T15:43:09.932 INFO:teuthology.orchestra.run.smithi123.stdout: "mds_epoch": 8, 2024-09-16T15:43:09.932 INFO:teuthology.orchestra.run.smithi123.stdout: "osd_epoch": 21, 2024-09-16T15:43:09.933 INFO:teuthology.orchestra.run.smithi123.stdout: "osd_epoch_barrier": 0, 2024-09-16T15:43:09.933 INFO:teuthology.orchestra.run.smithi123.stdout: "blocklisted": false, 2024-09-16T15:43:09.933 INFO:teuthology.orchestra.run.smithi123.stdout: "fs_name": "cephfs" 2024-09-16T15:43:09.933 INFO:teuthology.orchestra.run.smithi123.stdout:} 2024-09-16T15:43:09.944 DEBUG:tasks.ceph_fuse:passing mntargs=[] 2024-09-16T15:43:09.944 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-09-16T15:43:10.071 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:10.070+0000 7f78c52df640 1 -- 172.21.15.123:0/1508168024 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f78c015ac00 msgr2=0x7f78c00702f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:43:10.071 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:10.070+0000 7f78c52df640 1 --2- 172.21.15.123:0/1508168024 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f78c015ac00 0x7f78c00702f0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f78b0009900 tx=0x7f78b001c8e0 comp rx=0 tx=0).stop 2024-09-16T15:43:10.071 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:10.070+0000 7f78c52df640 1 -- 172.21.15.123:0/1508168024 shutdown_connections 2024-09-16T15:43:10.072 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:10.070+0000 7f78c52df640 1 --2- 172.21.15.123:0/1508168024 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f78c00b3100 0x7f78c00cc1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:10.072 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:10.070+0000 7f78c52df640 1 --2- 172.21.15.123:0/1508168024 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f78c015ac00 0x7f78c00702f0 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:10.072 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:10.070+0000 7f78c52df640 1 -- 172.21.15.123:0/1508168024 >> 172.21.15.123:0/1508168024 conn(0x7f78c00b2610 msgr2=0x7f78c00b29f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:43:10.072 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:10.070+0000 7f78c52df640 1 -- 172.21.15.123:0/1508168024 shutdown_connections 2024-09-16T15:43:10.072 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:10.070+0000 7f78c52df640 1 -- 172.21.15.123:0/1508168024 wait complete. 2024-09-16T15:43:10.072 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:10.070+0000 7f78c52df640 1 Processor -- start 2024-09-16T15:43:10.072 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:10.070+0000 7f78c52df640 1 -- start start 2024-09-16T15:43:10.074 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:10.070+0000 7f78c52df640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f78c0070830 0x7f78c01fc1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:43:10.074 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:10.070+0000 7f78c52df640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f78c00b3100 0x7f78c00ca450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:43:10.074 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:10.070+0000 7f78c52df640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f78c015ac00 0x7f78c00c6530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:43:10.074 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:10.070+0000 7f78c52df640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f78c0168660 con 0x7f78c015ac00 2024-09-16T15:43:10.074 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:10.070+0000 7f78c52df640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f78c01684e0 con 0x7f78c0070830 2024-09-16T15:43:10.074 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:10.070+0000 7f78c52df640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f78c01687e0 con 0x7f78c00b3100 2024-09-16T15:43:10.074 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:10.070+0000 7f78bf7fe640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f78c015ac00 0x7f78c00c6530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:43:10.074 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:10.070+0000 7f78bf7fe640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f78c015ac00 0x7f78c00c6530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:41790/0 (socket says 172.21.15.123:41790) 2024-09-16T15:43:10.074 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:10.070+0000 7f78bf7fe640 1 -- 172.21.15.123:0/3026941243 learned_addr learned my addr 172.21.15.123:0/3026941243 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:43:10.075 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:10.070+0000 7f78beffd640 1 --2- 172.21.15.123:0/3026941243 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f78c0070830 0x7f78c01fc1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:43:10.075 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:10.070+0000 7f78be7fc640 1 --2- 172.21.15.123:0/3026941243 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f78c00b3100 0x7f78c00ca450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:43:10.075 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:10.070+0000 7f78bf7fe640 1 -- 172.21.15.123:0/3026941243 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f78c00b3100 msgr2=0x7f78c00ca450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:43:10.075 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:10.070+0000 7f78bf7fe640 1 --2- 172.21.15.123:0/3026941243 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f78c00b3100 0x7f78c00ca450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:10.075 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:10.070+0000 7f78bf7fe640 1 -- 172.21.15.123:0/3026941243 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f78c0070830 msgr2=0x7f78c01fc1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:43:10.075 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:10.070+0000 7f78bf7fe640 1 --2- 172.21.15.123:0/3026941243 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f78c0070830 0x7f78c01fc1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:10.075 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:10.070+0000 7f78bf7fe640 1 -- 172.21.15.123:0/3026941243 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f78c00c7010 con 0x7f78c015ac00 2024-09-16T15:43:10.075 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:10.070+0000 7f78bf7fe640 1 --2- 172.21.15.123:0/3026941243 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f78c015ac00 0x7f78c00c6530 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f78b800c780 tx=0x7f78b800cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:43:10.076 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:10.074+0000 7f78a77fe640 1 -- 172.21.15.123:0/3026941243 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78b8016020 con 0x7f78c015ac00 2024-09-16T15:43:10.076 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:10.074+0000 7f78a77fe640 1 -- 172.21.15.123:0/3026941243 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f78b800aae0 con 0x7f78c015ac00 2024-09-16T15:43:10.076 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:10.074+0000 7f78c52df640 1 -- 172.21.15.123:0/3026941243 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f78c00c72f0 con 0x7f78c015ac00 2024-09-16T15:43:10.076 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:10.074+0000 7f78a77fe640 1 -- 172.21.15.123:0/3026941243 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78b8012420 con 0x7f78c015ac00 2024-09-16T15:43:10.077 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:10.074+0000 7f78c52df640 1 -- 172.21.15.123:0/3026941243 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f78c01fbcc0 con 0x7f78c015ac00 2024-09-16T15:43:10.077 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:10.074+0000 7f78c52df640 1 -- 172.21.15.123:0/3026941243 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f78c01fbec0 con 0x7f78c015ac00 2024-09-16T15:43:10.077 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:10.074+0000 7f78a77fe640 1 -- 172.21.15.123:0/3026941243 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 5) ==== 97080+0+0 (secure 0 0 0) 0x7f78b8019020 con 0x7f78c015ac00 2024-09-16T15:43:10.077 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:10.074+0000 7f78a77fe640 1 --2- 172.21.15.123:0/3026941243 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f78ac072ec0 0x7f78ac075380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:43:10.077 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:10.074+0000 7f78a77fe640 1 -- 172.21.15.123:0/3026941243 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f78b8093fa0 con 0x7f78c015ac00 2024-09-16T15:43:10.077 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:10.074+0000 7f78beffd640 1 --2- 172.21.15.123:0/3026941243 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f78ac072ec0 0x7f78ac075380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:43:10.078 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:10.074+0000 7f78beffd640 1 --2- 172.21.15.123:0/3026941243 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f78ac072ec0 0x7f78ac075380 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f78b00af000 tx=0x7f78b001b120 comp rx=0 tx=0).ready entity=mgr.4135 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:43:10.081 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:10.078+0000 7f78a77fe640 1 -- 172.21.15.123:0/3026941243 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f78c01fbec0 con 0x7f78c015ac00 2024-09-16T15:43:10.284 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:10.282+0000 7f78c52df640 1 -- 172.21.15.123:0/3026941243 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f78c00cc1b0 con 0x7f78c015ac00 2024-09-16T15:43:10.285 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:10.282+0000 7f78a77fe640 1 -- 172.21.15.123:0/3026941243 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v8) ==== 53+0+73 (secure 0 0 0) 0x7f78b8020020 con 0x7f78c015ac00 2024-09-16T15:43:10.285 INFO:teuthology.orchestra.run.smithi123.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-09-16T15:43:10.287 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:10.286+0000 7f78c52df640 1 -- 172.21.15.123:0/3026941243 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f78ac072ec0 msgr2=0x7f78ac075380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:43:10.288 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:10.286+0000 7f78c52df640 1 --2- 172.21.15.123:0/3026941243 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f78ac072ec0 0x7f78ac075380 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f78b00af000 tx=0x7f78b001b120 comp rx=0 tx=0).stop 2024-09-16T15:43:10.288 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:10.286+0000 7f78c52df640 1 -- 172.21.15.123:0/3026941243 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f78c015ac00 msgr2=0x7f78c00c6530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:43:10.288 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:10.286+0000 7f78c52df640 1 --2- 172.21.15.123:0/3026941243 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f78c015ac00 0x7f78c00c6530 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f78b800c780 tx=0x7f78b800cc40 comp rx=0 tx=0).stop 2024-09-16T15:43:10.288 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:10.286+0000 7f78c52df640 1 -- 172.21.15.123:0/3026941243 shutdown_connections 2024-09-16T15:43:10.288 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:10.286+0000 7f78c52df640 1 --2- 172.21.15.123:0/3026941243 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f78ac072ec0 0x7f78ac075380 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:10.288 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:10.286+0000 7f78c52df640 1 --2- 172.21.15.123:0/3026941243 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f78c015ac00 0x7f78c00c6530 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:10.288 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:10.286+0000 7f78c52df640 1 --2- 172.21.15.123:0/3026941243 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f78c00b3100 0x7f78c00ca450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:10.288 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:10.286+0000 7f78c52df640 1 --2- 172.21.15.123:0/3026941243 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f78c0070830 0x7f78c01fc1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:10.288 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:10.286+0000 7f78c52df640 1 -- 172.21.15.123:0/3026941243 >> 172.21.15.123:0/3026941243 conn(0x7f78c00b2610 msgr2=0x7f78c00b29f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:43:10.288 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:10.286+0000 7f78c52df640 1 -- 172.21.15.123:0/3026941243 shutdown_connections 2024-09-16T15:43:10.288 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:10.286+0000 7f78c52df640 1 -- 172.21.15.123:0/3026941243 wait complete. 2024-09-16T15:43:10.304 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-09-16T15:43:10.304 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-09-16T15:43:10.304 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi184.front.sepia.ceph.com 2024-09-16T15:43:10.304 INFO:tasks.cephfs.mount:self.client.name = client.1 2024-09-16T15:43:10.304 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2024-09-16T15:43:10.304 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-09-16T15:43:10.304 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-09-16T15:43:10.304 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-09-16T15:43:10.304 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.1' 2024-09-16T15:43:10.304 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:43:10.305 DEBUG:teuthology.orchestra.run.smithi184:> ip addr 2024-09-16T15:43:10.312 INFO:teuthology.orchestra.run.smithi184.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-09-16T15:43:10.312 INFO:teuthology.orchestra.run.smithi184.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-09-16T15:43:10.312 INFO:teuthology.orchestra.run.smithi184.stdout: inet 127.0.0.1/8 scope host lo 2024-09-16T15:43:10.312 INFO:teuthology.orchestra.run.smithi184.stdout: valid_lft forever preferred_lft forever 2024-09-16T15:43:10.312 INFO:teuthology.orchestra.run.smithi184.stdout: inet6 ::1/128 scope host 2024-09-16T15:43:10.312 INFO:teuthology.orchestra.run.smithi184.stdout: valid_lft forever preferred_lft forever 2024-09-16T15:43:10.312 INFO:teuthology.orchestra.run.smithi184.stdout:2: eno1: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-09-16T15:43:10.312 INFO:teuthology.orchestra.run.smithi184.stdout: link/ether 0c:c4:7a:d9:37:16 brd ff:ff:ff:ff:ff:ff 2024-09-16T15:43:10.312 INFO:teuthology.orchestra.run.smithi184.stdout: altname enp7s0f0 2024-09-16T15:43:10.312 INFO:teuthology.orchestra.run.smithi184.stdout:3: eno2: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-09-16T15:43:10.312 INFO:teuthology.orchestra.run.smithi184.stdout: link/ether 0c:c4:7a:d9:37:17 brd ff:ff:ff:ff:ff:ff 2024-09-16T15:43:10.313 INFO:teuthology.orchestra.run.smithi184.stdout: altname enp7s0f1 2024-09-16T15:43:10.313 INFO:teuthology.orchestra.run.smithi184.stdout:4: enp3s0f0: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-09-16T15:43:10.313 INFO:teuthology.orchestra.run.smithi184.stdout: link/ether 0c:c4:7a:88:7b:8e brd ff:ff:ff:ff:ff:ff 2024-09-16T15:43:10.313 INFO:teuthology.orchestra.run.smithi184.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-09-16T15:43:10.313 INFO:teuthology.orchestra.run.smithi184.stdout: link/ether 0c:c4:7a:88:7b:8f brd ff:ff:ff:ff:ff:ff 2024-09-16T15:43:10.313 INFO:teuthology.orchestra.run.smithi184.stdout: inet 172.21.15.184/20 metric 100 brd 172.21.15.255 scope global dynamic enp3s0f1 2024-09-16T15:43:10.313 INFO:teuthology.orchestra.run.smithi184.stdout: valid_lft 42467sec preferred_lft 42467sec 2024-09-16T15:43:10.313 INFO:teuthology.orchestra.run.smithi184.stdout: inet6 fe80::ec4:7aff:fe88:7b8f/64 scope link 2024-09-16T15:43:10.313 INFO:teuthology.orchestra.run.smithi184.stdout: valid_lft forever preferred_lft forever 2024-09-16T15:43:10.313 INFO:teuthology.orchestra.run.smithi184.stdout:6: docker0: mtu 1500 qdisc noqueue state DOWN group default 2024-09-16T15:43:10.313 INFO:teuthology.orchestra.run.smithi184.stdout: link/ether 02:42:5d:0a:44:b9 brd ff:ff:ff:ff:ff:ff 2024-09-16T15:43:10.313 INFO:teuthology.orchestra.run.smithi184.stdout: inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0 2024-09-16T15:43:10.313 INFO:teuthology.orchestra.run.smithi184.stdout: valid_lft forever preferred_lft forever 2024-09-16T15:43:10.314 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-09-16T15:43:10.314 DEBUG:teuthology.orchestra.run.smithi184:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-09-16T15:43:10.314 DEBUG:teuthology.orchestra.run.smithi184:> set -e 2024-09-16T15:43:10.314 DEBUG:teuthology.orchestra.run.smithi184:> sudo ip link add name ceph-brx type bridge 2024-09-16T15:43:10.314 DEBUG:teuthology.orchestra.run.smithi184:> sudo ip addr flush dev ceph-brx 2024-09-16T15:43:10.314 DEBUG:teuthology.orchestra.run.smithi184:> sudo ip link set ceph-brx up 2024-09-16T15:43:10.314 DEBUG:teuthology.orchestra.run.smithi184:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-09-16T15:43:10.314 DEBUG:teuthology.orchestra.run.smithi184:> ') 2024-09-16T15:43:10.400 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:43:10 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-09-16T15:43:10.435 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:43:10 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-16T15:43:10.442 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:43:10.442 DEBUG:teuthology.orchestra.run.smithi184:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-09-16T15:43:10.452 INFO:teuthology.orchestra.run.smithi184.stdout:1 2024-09-16T15:43:10.453 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:43:10.453 DEBUG:teuthology.orchestra.run.smithi184:> route 2024-09-16T15:43:10.518 INFO:teuthology.orchestra.run.smithi184.stdout:Kernel IP routing table 2024-09-16T15:43:10.518 INFO:teuthology.orchestra.run.smithi184.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-09-16T15:43:10.518 INFO:teuthology.orchestra.run.smithi184.stdout:default _gateway 0.0.0.0 UG 0 0 0 enp3s0f1 2024-09-16T15:43:10.518 INFO:teuthology.orchestra.run.smithi184.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-09-16T15:43:10.518 INFO:teuthology.orchestra.run.smithi184.stdout:158.69.67.47 _gateway 255.255.255.255 UGH 100 0 0 enp3s0f1 2024-09-16T15:43:10.518 INFO:teuthology.orchestra.run.smithi184.stdout:172.17.0.0 0.0.0.0 255.255.0.0 U 0 0 0 docker0 2024-09-16T15:43:10.519 INFO:teuthology.orchestra.run.smithi184.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-09-16T15:43:10.519 INFO:teuthology.orchestra.run.smithi184.stdout:ns1.front.sepia 0.0.0.0 255.255.255.255 UH 100 0 0 enp3s0f1 2024-09-16T15:43:10.519 INFO:teuthology.orchestra.run.smithi184.stdout:172.21.0.2 0.0.0.0 255.255.255.255 UH 100 0 0 enp3s0f1 2024-09-16T15:43:10.519 INFO:teuthology.orchestra.run.smithi184.stdout:_gateway 0.0.0.0 255.255.255.255 UH 100 0 0 enp3s0f1 2024-09-16T15:43:10.519 INFO:teuthology.orchestra.run.smithi184.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-09-16T15:43:10.519 DEBUG:teuthology.orchestra.run.smithi184:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-09-16T15:43:10.519 DEBUG:teuthology.orchestra.run.smithi184:> set -e 2024-09-16T15:43:10.519 DEBUG:teuthology.orchestra.run.smithi184:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-09-16T15:43:10.519 DEBUG:teuthology.orchestra.run.smithi184:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-09-16T15:43:10.519 DEBUG:teuthology.orchestra.run.smithi184:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-09-16T15:43:10.520 DEBUG:teuthology.orchestra.run.smithi184:> ') 2024-09-16T15:43:10.604 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:43:10 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-09-16T15:43:10.634 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:43:10 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-16T15:43:10.641 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:43:10.641 DEBUG:teuthology.orchestra.run.smithi184:> ip netns list 2024-09-16T15:43:10.687 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:43:10.687 DEBUG:teuthology.orchestra.run.smithi184:> ip netns list-id 2024-09-16T15:43:10.735 DEBUG:teuthology.orchestra.run.smithi184:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-09-16T15:43:10.735 DEBUG:teuthology.orchestra.run.smithi184:> set -e 2024-09-16T15:43:10.735 DEBUG:teuthology.orchestra.run.smithi184:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.1 2024-09-16T15:43:10.735 DEBUG:teuthology.orchestra.run.smithi184:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.1 0 2024-09-16T15:43:10.735 DEBUG:teuthology.orchestra.run.smithi184:> ') 2024-09-16T15:43:10.824 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:43:10 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-09-16T15:43:10.844 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:43:10 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-16T15:43:10.852 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.1' with 192.168.0.1/16 2024-09-16T15:43:10.852 DEBUG:teuthology.orchestra.run.smithi184:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-09-16T15:43:10.852 DEBUG:teuthology.orchestra.run.smithi184:> set -e 2024-09-16T15:43:10.852 DEBUG:teuthology.orchestra.run.smithi184:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.1 type veth peer name brx.0 2024-09-16T15:43:10.852 DEBUG:teuthology.orchestra.run.smithi184:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-09-16T15:43:10.853 DEBUG:teuthology.orchestra.run.smithi184:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip link set veth0 up 2024-09-16T15:43:10.853 DEBUG:teuthology.orchestra.run.smithi184:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip link set lo up 2024-09-16T15:43:10.853 DEBUG:teuthology.orchestra.run.smithi184:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip route add default via 192.168.255.254 2024-09-16T15:43:10.853 DEBUG:teuthology.orchestra.run.smithi184:> ') 2024-09-16T15:43:10.892 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:43:10 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-09-16T15:43:10.947 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:43:10 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-16T15:43:10.954 DEBUG:teuthology.orchestra.run.smithi184:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-09-16T15:43:10.954 DEBUG:teuthology.orchestra.run.smithi184:> set -e 2024-09-16T15:43:10.954 DEBUG:teuthology.orchestra.run.smithi184:> sudo ip link set brx.0 up 2024-09-16T15:43:10.954 DEBUG:teuthology.orchestra.run.smithi184:> sudo ip link set dev brx.0 master ceph-brx 2024-09-16T15:43:10.954 DEBUG:teuthology.orchestra.run.smithi184:> ') 2024-09-16T15:43:11.036 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:43:11 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-09-16T15:43:11.050 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:43:11 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-16T15:43:11.057 INFO:tasks.cephfs.fuse_mount:Client client.1 config is {} 2024-09-16T15:43:11.057 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-09-16T15:43:11.057 DEBUG:teuthology.orchestra.run.smithi184:> mkdir -p -v /home/ubuntu/cephtest/mnt.1 2024-09-16T15:43:11.107 INFO:teuthology.orchestra.run.smithi184.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.1' 2024-09-16T15:43:11.107 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-09-16T15:43:11.107 DEBUG:teuthology.orchestra.run.smithi184:> chmod 0000 /home/ubuntu/cephtest/mnt.1 2024-09-16T15:43:11.154 DEBUG:teuthology.orchestra.run.smithi184:> sudo modprobe fuse 2024-09-16T15:43:11.214 DEBUG:teuthology.orchestra.run.smithi184:> cat /proc/self/mounts | awk '{print $2}' 2024-09-16T15:43:11.265 INFO:teuthology.orchestra.run.smithi184.stdout:/sys 2024-09-16T15:43:11.265 INFO:teuthology.orchestra.run.smithi184.stdout:/proc 2024-09-16T15:43:11.265 INFO:teuthology.orchestra.run.smithi184.stdout:/dev 2024-09-16T15:43:11.265 INFO:teuthology.orchestra.run.smithi184.stdout:/dev/pts 2024-09-16T15:43:11.265 INFO:teuthology.orchestra.run.smithi184.stdout:/run 2024-09-16T15:43:11.265 INFO:teuthology.orchestra.run.smithi184.stdout:/ 2024-09-16T15:43:11.265 INFO:teuthology.orchestra.run.smithi184.stdout:/sys/kernel/security 2024-09-16T15:43:11.265 INFO:teuthology.orchestra.run.smithi184.stdout:/dev/shm 2024-09-16T15:43:11.265 INFO:teuthology.orchestra.run.smithi184.stdout:/run/lock 2024-09-16T15:43:11.266 INFO:teuthology.orchestra.run.smithi184.stdout:/sys/fs/cgroup 2024-09-16T15:43:11.266 INFO:teuthology.orchestra.run.smithi184.stdout:/sys/fs/pstore 2024-09-16T15:43:11.266 INFO:teuthology.orchestra.run.smithi184.stdout:/sys/fs/bpf 2024-09-16T15:43:11.266 INFO:teuthology.orchestra.run.smithi184.stdout:/proc/sys/fs/binfmt_misc 2024-09-16T15:43:11.266 INFO:teuthology.orchestra.run.smithi184.stdout:/dev/hugepages 2024-09-16T15:43:11.266 INFO:teuthology.orchestra.run.smithi184.stdout:/dev/mqueue 2024-09-16T15:43:11.266 INFO:teuthology.orchestra.run.smithi184.stdout:/sys/kernel/debug 2024-09-16T15:43:11.266 INFO:teuthology.orchestra.run.smithi184.stdout:/sys/kernel/tracing 2024-09-16T15:43:11.266 INFO:teuthology.orchestra.run.smithi184.stdout:/proc/fs/nfsd 2024-09-16T15:43:11.266 INFO:teuthology.orchestra.run.smithi184.stdout:/sys/fs/fuse/connections 2024-09-16T15:43:11.266 INFO:teuthology.orchestra.run.smithi184.stdout:/sys/kernel/config 2024-09-16T15:43:11.266 INFO:teuthology.orchestra.run.smithi184.stdout:/run/credentials/systemd-sysusers.service 2024-09-16T15:43:11.266 INFO:teuthology.orchestra.run.smithi184.stdout:/run/rpc_pipefs 2024-09-16T15:43:11.267 INFO:teuthology.orchestra.run.smithi184.stdout:/run/user/1000 2024-09-16T15:43:11.267 INFO:teuthology.orchestra.run.smithi184.stdout:/var/lib/ceph 2024-09-16T15:43:11.267 INFO:teuthology.orchestra.run.smithi184.stdout:/var/lib/ceph/osd/ceph-4 2024-09-16T15:43:11.267 INFO:teuthology.orchestra.run.smithi184.stdout:/var/lib/ceph/osd/ceph-5 2024-09-16T15:43:11.267 INFO:teuthology.orchestra.run.smithi184.stdout:/var/lib/ceph/osd/ceph-6 2024-09-16T15:43:11.267 INFO:teuthology.orchestra.run.smithi184.stdout:/var/lib/ceph/osd/ceph-7 2024-09-16T15:43:11.267 INFO:teuthology.orchestra.run.smithi184.stdout:/run/netns 2024-09-16T15:43:11.267 INFO:teuthology.orchestra.run.smithi184.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-09-16T15:43:11.267 INFO:teuthology.orchestra.run.smithi184.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-09-16T15:43:11.267 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:43:11.267 DEBUG:teuthology.orchestra.run.smithi184:> ls /sys/fs/fuse/connections 2024-09-16T15:43:11.315 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-09-16T15:43:11.315 DEBUG:teuthology.orchestra.run.smithi184:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.1 --id 1) 2024-09-16T15:43:11.360 DEBUG:teuthology.orchestra.run.smithi184:> sudo modprobe fuse 2024-09-16T15:43:11.373 DEBUG:teuthology.orchestra.run.smithi184:> cat /proc/self/mounts | awk '{print $2}' 2024-09-16T15:43:11.420 INFO:teuthology.orchestra.run.smithi184.stdout:/sys 2024-09-16T15:43:11.420 INFO:teuthology.orchestra.run.smithi184.stdout:/proc 2024-09-16T15:43:11.420 INFO:teuthology.orchestra.run.smithi184.stdout:/dev 2024-09-16T15:43:11.421 INFO:teuthology.orchestra.run.smithi184.stdout:/dev/pts 2024-09-16T15:43:11.421 INFO:teuthology.orchestra.run.smithi184.stdout:/run 2024-09-16T15:43:11.421 INFO:teuthology.orchestra.run.smithi184.stdout:/ 2024-09-16T15:43:11.421 INFO:teuthology.orchestra.run.smithi184.stdout:/sys/kernel/security 2024-09-16T15:43:11.421 INFO:teuthology.orchestra.run.smithi184.stdout:/dev/shm 2024-09-16T15:43:11.421 INFO:teuthology.orchestra.run.smithi184.stdout:/run/lock 2024-09-16T15:43:11.421 INFO:teuthology.orchestra.run.smithi184.stdout:/sys/fs/cgroup 2024-09-16T15:43:11.421 INFO:teuthology.orchestra.run.smithi184.stdout:/sys/fs/pstore 2024-09-16T15:43:11.421 INFO:teuthology.orchestra.run.smithi184.stdout:/sys/fs/bpf 2024-09-16T15:43:11.421 INFO:teuthology.orchestra.run.smithi184.stdout:/proc/sys/fs/binfmt_misc 2024-09-16T15:43:11.421 INFO:teuthology.orchestra.run.smithi184.stdout:/dev/hugepages 2024-09-16T15:43:11.422 INFO:teuthology.orchestra.run.smithi184.stdout:/dev/mqueue 2024-09-16T15:43:11.422 INFO:teuthology.orchestra.run.smithi184.stdout:/sys/kernel/debug 2024-09-16T15:43:11.422 INFO:teuthology.orchestra.run.smithi184.stdout:/sys/kernel/tracing 2024-09-16T15:43:11.422 INFO:teuthology.orchestra.run.smithi184.stdout:/proc/fs/nfsd 2024-09-16T15:43:11.422 INFO:teuthology.orchestra.run.smithi184.stdout:/sys/fs/fuse/connections 2024-09-16T15:43:11.422 INFO:teuthology.orchestra.run.smithi184.stdout:/sys/kernel/config 2024-09-16T15:43:11.422 INFO:teuthology.orchestra.run.smithi184.stdout:/run/credentials/systemd-sysusers.service 2024-09-16T15:43:11.422 INFO:teuthology.orchestra.run.smithi184.stdout:/run/rpc_pipefs 2024-09-16T15:43:11.422 INFO:teuthology.orchestra.run.smithi184.stdout:/run/user/1000 2024-09-16T15:43:11.422 INFO:teuthology.orchestra.run.smithi184.stdout:/var/lib/ceph 2024-09-16T15:43:11.422 INFO:teuthology.orchestra.run.smithi184.stdout:/var/lib/ceph/osd/ceph-4 2024-09-16T15:43:11.422 INFO:teuthology.orchestra.run.smithi184.stdout:/var/lib/ceph/osd/ceph-5 2024-09-16T15:43:11.422 INFO:teuthology.orchestra.run.smithi184.stdout:/var/lib/ceph/osd/ceph-6 2024-09-16T15:43:11.423 INFO:teuthology.orchestra.run.smithi184.stdout:/var/lib/ceph/osd/ceph-7 2024-09-16T15:43:11.423 INFO:teuthology.orchestra.run.smithi184.stdout:/run/netns 2024-09-16T15:43:11.423 INFO:teuthology.orchestra.run.smithi184.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-09-16T15:43:11.423 INFO:teuthology.orchestra.run.smithi184.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-09-16T15:43:11.423 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:43:11.423 DEBUG:teuthology.orchestra.run.smithi184:> ls /sys/fs/fuse/connections 2024-09-16T15:43:11.436 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi184.stderr:2024-09-16T15:43:11.434+0000 7fe8fe8f7580 -1 init, newargv = 0x56545ae8e8e0 newargc=15 2024-09-16T15:43:11.436 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi184.stderr:ceph-fuse[34345]: starting ceph client 2024-09-16T15:43:11.442 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi184.stderr:ceph-fuse[34345]: starting fuse 2024-09-16T15:43:12.541 DEBUG:teuthology.orchestra.run.smithi184:> sudo modprobe fuse 2024-09-16T15:43:12.557 DEBUG:teuthology.orchestra.run.smithi184:> cat /proc/self/mounts | awk '{print $2}' 2024-09-16T15:43:12.604 INFO:teuthology.orchestra.run.smithi184.stdout:/sys 2024-09-16T15:43:12.604 INFO:teuthology.orchestra.run.smithi184.stdout:/proc 2024-09-16T15:43:12.604 INFO:teuthology.orchestra.run.smithi184.stdout:/dev 2024-09-16T15:43:12.604 INFO:teuthology.orchestra.run.smithi184.stdout:/dev/pts 2024-09-16T15:43:12.604 INFO:teuthology.orchestra.run.smithi184.stdout:/run 2024-09-16T15:43:12.605 INFO:teuthology.orchestra.run.smithi184.stdout:/ 2024-09-16T15:43:12.605 INFO:teuthology.orchestra.run.smithi184.stdout:/sys/kernel/security 2024-09-16T15:43:12.605 INFO:teuthology.orchestra.run.smithi184.stdout:/dev/shm 2024-09-16T15:43:12.605 INFO:teuthology.orchestra.run.smithi184.stdout:/run/lock 2024-09-16T15:43:12.605 INFO:teuthology.orchestra.run.smithi184.stdout:/sys/fs/cgroup 2024-09-16T15:43:12.605 INFO:teuthology.orchestra.run.smithi184.stdout:/sys/fs/pstore 2024-09-16T15:43:12.605 INFO:teuthology.orchestra.run.smithi184.stdout:/sys/fs/bpf 2024-09-16T15:43:12.605 INFO:teuthology.orchestra.run.smithi184.stdout:/proc/sys/fs/binfmt_misc 2024-09-16T15:43:12.605 INFO:teuthology.orchestra.run.smithi184.stdout:/dev/hugepages 2024-09-16T15:43:12.605 INFO:teuthology.orchestra.run.smithi184.stdout:/dev/mqueue 2024-09-16T15:43:12.605 INFO:teuthology.orchestra.run.smithi184.stdout:/sys/kernel/debug 2024-09-16T15:43:12.605 INFO:teuthology.orchestra.run.smithi184.stdout:/sys/kernel/tracing 2024-09-16T15:43:12.605 INFO:teuthology.orchestra.run.smithi184.stdout:/proc/fs/nfsd 2024-09-16T15:43:12.606 INFO:teuthology.orchestra.run.smithi184.stdout:/sys/fs/fuse/connections 2024-09-16T15:43:12.606 INFO:teuthology.orchestra.run.smithi184.stdout:/sys/kernel/config 2024-09-16T15:43:12.606 INFO:teuthology.orchestra.run.smithi184.stdout:/run/credentials/systemd-sysusers.service 2024-09-16T15:43:12.606 INFO:teuthology.orchestra.run.smithi184.stdout:/run/rpc_pipefs 2024-09-16T15:43:12.606 INFO:teuthology.orchestra.run.smithi184.stdout:/run/user/1000 2024-09-16T15:43:12.606 INFO:teuthology.orchestra.run.smithi184.stdout:/var/lib/ceph 2024-09-16T15:43:12.606 INFO:teuthology.orchestra.run.smithi184.stdout:/var/lib/ceph/osd/ceph-4 2024-09-16T15:43:12.606 INFO:teuthology.orchestra.run.smithi184.stdout:/var/lib/ceph/osd/ceph-5 2024-09-16T15:43:12.606 INFO:teuthology.orchestra.run.smithi184.stdout:/var/lib/ceph/osd/ceph-6 2024-09-16T15:43:12.606 INFO:teuthology.orchestra.run.smithi184.stdout:/var/lib/ceph/osd/ceph-7 2024-09-16T15:43:12.606 INFO:teuthology.orchestra.run.smithi184.stdout:/run/netns 2024-09-16T15:43:12.606 INFO:teuthology.orchestra.run.smithi184.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-09-16T15:43:12.606 INFO:teuthology.orchestra.run.smithi184.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-09-16T15:43:12.606 INFO:teuthology.orchestra.run.smithi184.stdout:/home/ubuntu/cephtest/mnt.1 2024-09-16T15:43:12.607 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:43:12.607 DEBUG:teuthology.orchestra.run.smithi184:> ls /sys/fs/fuse/connections 2024-09-16T15:43:12.655 INFO:teuthology.orchestra.run.smithi184.stdout:45 2024-09-16T15:43:12.655 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [45] 2024-09-16T15:43:12.655 DEBUG:teuthology.orchestra.run.smithi184:> sudo stdin-killer -- python3 -c ' 2024-09-16T15:43:12.655 DEBUG:teuthology.orchestra.run.smithi184:> import glob 2024-09-16T15:43:12.655 DEBUG:teuthology.orchestra.run.smithi184:> import re 2024-09-16T15:43:12.655 DEBUG:teuthology.orchestra.run.smithi184:> import os 2024-09-16T15:43:12.656 DEBUG:teuthology.orchestra.run.smithi184:> import subprocess 2024-09-16T15:43:12.656 DEBUG:teuthology.orchestra.run.smithi184:> 2024-09-16T15:43:12.656 DEBUG:teuthology.orchestra.run.smithi184:> def _find_admin_socket(client_name): 2024-09-16T15:43:12.656 DEBUG:teuthology.orchestra.run.smithi184:> asok_path = "/var/run/ceph/ceph-client.1.*.asok" 2024-09-16T15:43:12.656 DEBUG:teuthology.orchestra.run.smithi184:> files = glob.glob(asok_path) 2024-09-16T15:43:12.656 DEBUG:teuthology.orchestra.run.smithi184:> mountpoint = "/home/ubuntu/cephtest/mnt.1" 2024-09-16T15:43:12.656 DEBUG:teuthology.orchestra.run.smithi184:> 2024-09-16T15:43:12.656 DEBUG:teuthology.orchestra.run.smithi184:> # Given a non-glob path, it better be there 2024-09-16T15:43:12.656 DEBUG:teuthology.orchestra.run.smithi184:> if "*" not in asok_path: 2024-09-16T15:43:12.656 DEBUG:teuthology.orchestra.run.smithi184:> assert(len(files) == 1) 2024-09-16T15:43:12.656 DEBUG:teuthology.orchestra.run.smithi184:> return files[0] 2024-09-16T15:43:12.656 DEBUG:teuthology.orchestra.run.smithi184:> 2024-09-16T15:43:12.656 DEBUG:teuthology.orchestra.run.smithi184:> for f in files: 2024-09-16T15:43:12.656 DEBUG:teuthology.orchestra.run.smithi184:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-09-16T15:43:12.657 DEBUG:teuthology.orchestra.run.smithi184:> if os.path.exists("/proc/{0}".format(pid)): 2024-09-16T15:43:12.657 DEBUG:teuthology.orchestra.run.smithi184:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-09-16T15:43:12.657 DEBUG:teuthology.orchestra.run.smithi184:> contents = proc_f.read() 2024-09-16T15:43:12.657 DEBUG:teuthology.orchestra.run.smithi184:> if mountpoint in contents: 2024-09-16T15:43:12.657 DEBUG:teuthology.orchestra.run.smithi184:> return f 2024-09-16T15:43:12.657 DEBUG:teuthology.orchestra.run.smithi184:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-09-16T15:43:12.657 DEBUG:teuthology.orchestra.run.smithi184:> 2024-09-16T15:43:12.657 DEBUG:teuthology.orchestra.run.smithi184:> print(_find_admin_socket("client.1")) 2024-09-16T15:43:12.657 DEBUG:teuthology.orchestra.run.smithi184:> ' 2024-09-16T15:43:12.780 INFO:teuthology.orchestra.run.smithi184.stdout:/var/run/ceph/ceph-client.1.34345.asok 2024-09-16T15:43:12.784 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:43:12 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-16T15:43:12.791 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.1.34345.asok 2024-09-16T15:43:12.791 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:43:12.791 DEBUG:teuthology.orchestra.run.smithi184:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.1.34345.asok status 2024-09-16T15:43:12.945 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-16T15:43:12.945 INFO:teuthology.orchestra.run.smithi184.stdout: "metadata": { 2024-09-16T15:43:12.945 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph_sha1": "33f95d2ae1bcbe7eb04404303f179eeeaa6982df", 2024-09-16T15:43:12.945 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph_version": "ceph version 19.3.0-4938-g33f95d2a (33f95d2ae1bcbe7eb04404303f179eeeaa6982df) squid (dev)", 2024-09-16T15:43:12.945 INFO:teuthology.orchestra.run.smithi184.stdout: "entity_id": "1", 2024-09-16T15:43:12.945 INFO:teuthology.orchestra.run.smithi184.stdout: "hostname": "smithi184", 2024-09-16T15:43:12.946 INFO:teuthology.orchestra.run.smithi184.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.1", 2024-09-16T15:43:12.946 INFO:teuthology.orchestra.run.smithi184.stdout: "pid": "34345", 2024-09-16T15:43:12.946 INFO:teuthology.orchestra.run.smithi184.stdout: "root": "/" 2024-09-16T15:43:12.946 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-16T15:43:12.946 INFO:teuthology.orchestra.run.smithi184.stdout: "dentry_count": 0, 2024-09-16T15:43:12.946 INFO:teuthology.orchestra.run.smithi184.stdout: "dentry_pinned_count": 0, 2024-09-16T15:43:12.946 INFO:teuthology.orchestra.run.smithi184.stdout: "id": 4591, 2024-09-16T15:43:12.946 INFO:teuthology.orchestra.run.smithi184.stdout: "inst": { 2024-09-16T15:43:12.946 INFO:teuthology.orchestra.run.smithi184.stdout: "name": { 2024-09-16T15:43:12.946 INFO:teuthology.orchestra.run.smithi184.stdout: "type": "client", 2024-09-16T15:43:12.946 INFO:teuthology.orchestra.run.smithi184.stdout: "num": 4591 2024-09-16T15:43:12.946 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-16T15:43:12.946 INFO:teuthology.orchestra.run.smithi184.stdout: "addr": { 2024-09-16T15:43:12.946 INFO:teuthology.orchestra.run.smithi184.stdout: "type": "v1", 2024-09-16T15:43:12.947 INFO:teuthology.orchestra.run.smithi184.stdout: "addr": "192.168.0.1:0", 2024-09-16T15:43:12.947 INFO:teuthology.orchestra.run.smithi184.stdout: "nonce": 2856724591 2024-09-16T15:43:12.947 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-16T15:43:12.947 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-16T15:43:12.947 INFO:teuthology.orchestra.run.smithi184.stdout: "addr": { 2024-09-16T15:43:12.947 INFO:teuthology.orchestra.run.smithi184.stdout: "type": "v1", 2024-09-16T15:43:12.947 INFO:teuthology.orchestra.run.smithi184.stdout: "addr": "192.168.0.1:0", 2024-09-16T15:43:12.947 INFO:teuthology.orchestra.run.smithi184.stdout: "nonce": 2856724591 2024-09-16T15:43:12.947 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-16T15:43:12.947 INFO:teuthology.orchestra.run.smithi184.stdout: "inst_str": "client.4591 192.168.0.1:0/2856724591", 2024-09-16T15:43:12.947 INFO:teuthology.orchestra.run.smithi184.stdout: "addr_str": "192.168.0.1:0/2856724591", 2024-09-16T15:43:12.947 INFO:teuthology.orchestra.run.smithi184.stdout: "inode_count": 1, 2024-09-16T15:43:12.947 INFO:teuthology.orchestra.run.smithi184.stdout: "mds_epoch": 8, 2024-09-16T15:43:12.948 INFO:teuthology.orchestra.run.smithi184.stdout: "osd_epoch": 21, 2024-09-16T15:43:12.948 INFO:teuthology.orchestra.run.smithi184.stdout: "osd_epoch_barrier": 0, 2024-09-16T15:43:12.948 INFO:teuthology.orchestra.run.smithi184.stdout: "blocklisted": false, 2024-09-16T15:43:12.948 INFO:teuthology.orchestra.run.smithi184.stdout: "fs_name": "cephfs" 2024-09-16T15:43:12.948 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-16T15:43:12.959 DEBUG:tasks.ceph_fuse:passing mntargs=[] 2024-09-16T15:43:12.959 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-09-16T15:43:13.094 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:13.090+0000 7fae6346f640 1 -- 172.21.15.123:0/2599006608 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fae5c15b120 msgr2=0x7fae5c073600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:43:13.095 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:13.090+0000 7fae6346f640 1 --2- 172.21.15.123:0/2599006608 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fae5c15b120 0x7fae5c073600 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fae5c069a70 tx=0x7fae5801cd80 comp rx=0 tx=0).stop 2024-09-16T15:43:13.095 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:13.090+0000 7fae6346f640 1 -- 172.21.15.123:0/2599006608 shutdown_connections 2024-09-16T15:43:13.095 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:13.090+0000 7fae6346f640 1 --2- 172.21.15.123:0/2599006608 >> v2:172.21.15.184:3300/0 conn(0x7fae5c073b40 0x7fae5c0cfcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:13.095 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:13.090+0000 7fae6346f640 1 --2- 172.21.15.123:0/2599006608 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fae5c15b120 0x7fae5c073600 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:13.095 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:13.094+0000 7fae6346f640 1 -- 172.21.15.123:0/2599006608 >> 172.21.15.123:0/2599006608 conn(0x7fae5c0cd5d0 msgr2=0x7fae5c0cd9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:43:13.095 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:13.094+0000 7fae6346f640 1 -- 172.21.15.123:0/2599006608 shutdown_connections 2024-09-16T15:43:13.095 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:13.094+0000 7fae6346f640 1 -- 172.21.15.123:0/2599006608 wait complete. 2024-09-16T15:43:13.095 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:13.094+0000 7fae6346f640 1 Processor -- start 2024-09-16T15:43:13.096 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:13.094+0000 7fae6346f640 1 -- start start 2024-09-16T15:43:13.098 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:13.094+0000 7fae6346f640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fae5c073b40 0x7fae5c1471d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:43:13.098 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:13.094+0000 7fae6346f640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fae5c151b10 0x7fae5c14e980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:43:13.098 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:13.094+0000 7fae6346f640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fae5c15b120 0x7fae5c14eec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:43:13.098 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:13.094+0000 7fae6346f640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fae5c06d350 con 0x7fae5c073b40 2024-09-16T15:43:13.098 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:13.094+0000 7fae6346f640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fae5c06d1d0 con 0x7fae5c151b10 2024-09-16T15:43:13.098 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:13.094+0000 7fae6346f640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fae5c06d4d0 con 0x7fae5c15b120 2024-09-16T15:43:13.098 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:13.094+0000 7fae6120b640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fae5c073b40 0x7fae5c1471d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:43:13.098 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:13.094+0000 7fae6120b640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fae5c073b40 0x7fae5c1471d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:51516/0 (socket says 172.21.15.123:51516) 2024-09-16T15:43:13.099 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:13.094+0000 7fae6120b640 1 -- 172.21.15.123:0/243099191 learned_addr learned my addr 172.21.15.123:0/243099191 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:43:13.099 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:13.094+0000 7fae61a0c640 1 --2- 172.21.15.123:0/243099191 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fae5c15b120 0x7fae5c14eec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:43:13.099 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:13.094+0000 7fae60a0a640 1 --2- 172.21.15.123:0/243099191 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fae5c151b10 0x7fae5c14e980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:43:13.099 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:13.094+0000 7fae6120b640 1 -- 172.21.15.123:0/243099191 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fae5c15b120 msgr2=0x7fae5c14eec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:43:13.099 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:13.094+0000 7fae6120b640 1 --2- 172.21.15.123:0/243099191 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fae5c15b120 0x7fae5c14eec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:13.099 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:13.094+0000 7fae6120b640 1 -- 172.21.15.123:0/243099191 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fae5c151b10 msgr2=0x7fae5c14e980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:43:13.099 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:13.094+0000 7fae6120b640 1 --2- 172.21.15.123:0/243099191 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fae5c151b10 0x7fae5c14e980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:13.099 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:13.094+0000 7fae6120b640 1 -- 172.21.15.123:0/243099191 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fae5c1530a0 con 0x7fae5c073b40 2024-09-16T15:43:13.099 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:13.094+0000 7fae6120b640 1 --2- 172.21.15.123:0/243099191 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fae5c073b40 0x7fae5c1471d0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fae5400c440 tx=0x7fae5400c900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:43:13.099 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:13.094+0000 7fae51ffb640 1 -- 172.21.15.123:0/243099191 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae54017070 con 0x7fae5c073b40 2024-09-16T15:43:13.099 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:13.094+0000 7fae51ffb640 1 -- 172.21.15.123:0/243099191 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fae54003650 con 0x7fae5c073b40 2024-09-16T15:43:13.099 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:13.094+0000 7fae6346f640 1 -- 172.21.15.123:0/243099191 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fae5c153380 con 0x7fae5c073b40 2024-09-16T15:43:13.099 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:13.094+0000 7fae51ffb640 1 -- 172.21.15.123:0/243099191 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae54003ae0 con 0x7fae5c073b40 2024-09-16T15:43:13.099 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:13.094+0000 7fae6346f640 1 -- 172.21.15.123:0/243099191 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fae5c153940 con 0x7fae5c073b40 2024-09-16T15:43:13.100 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:13.094+0000 7fae6346f640 1 -- 172.21.15.123:0/243099191 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fae5c06da70 con 0x7fae5c073b40 2024-09-16T15:43:13.100 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:13.098+0000 7fae51ffb640 1 -- 172.21.15.123:0/243099191 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 5) ==== 97080+0+0 (secure 0 0 0) 0x7fae5401a070 con 0x7fae5c073b40 2024-09-16T15:43:13.100 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:13.098+0000 7fae51ffb640 1 --2- 172.21.15.123:0/243099191 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fae48072ec0 0x7fae48075380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:43:13.100 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:13.098+0000 7fae51ffb640 1 -- 172.21.15.123:0/243099191 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7fae54093100 con 0x7fae5c073b40 2024-09-16T15:43:13.100 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:13.098+0000 7fae60a0a640 1 --2- 172.21.15.123:0/243099191 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fae48072ec0 0x7fae48075380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:43:13.101 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:13.098+0000 7fae60a0a640 1 --2- 172.21.15.123:0/243099191 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fae48072ec0 0x7fae48075380 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fae580af000 tx=0x7fae5801b5c0 comp rx=0 tx=0).ready entity=mgr.4135 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:43:13.104 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:13.102+0000 7fae51ffb640 1 -- 172.21.15.123:0/243099191 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7fae5405e3e0 con 0x7fae5c073b40 2024-09-16T15:43:13.316 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:13.314+0000 7fae6346f640 1 -- 172.21.15.123:0/243099191 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7fae5c0cae80 con 0x7fae5c073b40 2024-09-16T15:43:13.320 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:13.318+0000 7fae51ffb640 1 -- 172.21.15.123:0/243099191 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v8) ==== 53+0+73 (secure 0 0 0) 0x7fae5405dce0 con 0x7fae5c073b40 2024-09-16T15:43:13.320 INFO:teuthology.orchestra.run.smithi123.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-09-16T15:43:13.322 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:13.318+0000 7fae6346f640 1 -- 172.21.15.123:0/243099191 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fae48072ec0 msgr2=0x7fae48075380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:43:13.322 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:13.318+0000 7fae6346f640 1 --2- 172.21.15.123:0/243099191 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fae48072ec0 0x7fae48075380 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fae580af000 tx=0x7fae5801b5c0 comp rx=0 tx=0).stop 2024-09-16T15:43:13.322 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:13.318+0000 7fae6346f640 1 -- 172.21.15.123:0/243099191 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fae5c073b40 msgr2=0x7fae5c1471d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:43:13.323 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:13.318+0000 7fae6346f640 1 --2- 172.21.15.123:0/243099191 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fae5c073b40 0x7fae5c1471d0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fae5400c440 tx=0x7fae5400c900 comp rx=0 tx=0).stop 2024-09-16T15:43:13.323 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:13.318+0000 7fae6346f640 1 -- 172.21.15.123:0/243099191 shutdown_connections 2024-09-16T15:43:13.323 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:13.318+0000 7fae6346f640 1 --2- 172.21.15.123:0/243099191 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fae48072ec0 0x7fae48075380 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:13.323 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:13.318+0000 7fae6346f640 1 --2- 172.21.15.123:0/243099191 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fae5c15b120 0x7fae5c14eec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:13.323 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:13.318+0000 7fae6346f640 1 --2- 172.21.15.123:0/243099191 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fae5c151b10 0x7fae5c14e980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:13.323 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:13.318+0000 7fae6346f640 1 --2- 172.21.15.123:0/243099191 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fae5c073b40 0x7fae5c1471d0 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:13.323 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:13.318+0000 7fae6346f640 1 -- 172.21.15.123:0/243099191 >> 172.21.15.123:0/243099191 conn(0x7fae5c0cd5d0 msgr2=0x7fae5c0ceca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:43:13.323 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:13.318+0000 7fae6346f640 1 -- 172.21.15.123:0/243099191 shutdown_connections 2024-09-16T15:43:13.323 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:13.318+0000 7fae6346f640 1 -- 172.21.15.123:0/243099191 wait complete. 2024-09-16T15:43:13.339 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-09-16T15:43:13.339 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-09-16T15:43:13.339 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi184.front.sepia.ceph.com 2024-09-16T15:43:13.339 INFO:tasks.cephfs.mount:self.client.name = client.3 2024-09-16T15:43:13.339 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.3 2024-09-16T15:43:13.339 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-09-16T15:43:13.340 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-09-16T15:43:13.340 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-09-16T15:43:13.340 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.3' 2024-09-16T15:43:13.340 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:43:13.340 DEBUG:teuthology.orchestra.run.smithi184:> ip addr 2024-09-16T15:43:13.347 INFO:teuthology.orchestra.run.smithi184.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-09-16T15:43:13.347 INFO:teuthology.orchestra.run.smithi184.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-09-16T15:43:13.347 INFO:teuthology.orchestra.run.smithi184.stdout: inet 127.0.0.1/8 scope host lo 2024-09-16T15:43:13.347 INFO:teuthology.orchestra.run.smithi184.stdout: valid_lft forever preferred_lft forever 2024-09-16T15:43:13.347 INFO:teuthology.orchestra.run.smithi184.stdout: inet6 ::1/128 scope host 2024-09-16T15:43:13.348 INFO:teuthology.orchestra.run.smithi184.stdout: valid_lft forever preferred_lft forever 2024-09-16T15:43:13.348 INFO:teuthology.orchestra.run.smithi184.stdout:2: eno1: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-09-16T15:43:13.348 INFO:teuthology.orchestra.run.smithi184.stdout: link/ether 0c:c4:7a:d9:37:16 brd ff:ff:ff:ff:ff:ff 2024-09-16T15:43:13.348 INFO:teuthology.orchestra.run.smithi184.stdout: altname enp7s0f0 2024-09-16T15:43:13.348 INFO:teuthology.orchestra.run.smithi184.stdout:3: eno2: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-09-16T15:43:13.348 INFO:teuthology.orchestra.run.smithi184.stdout: link/ether 0c:c4:7a:d9:37:17 brd ff:ff:ff:ff:ff:ff 2024-09-16T15:43:13.348 INFO:teuthology.orchestra.run.smithi184.stdout: altname enp7s0f1 2024-09-16T15:43:13.348 INFO:teuthology.orchestra.run.smithi184.stdout:4: enp3s0f0: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-09-16T15:43:13.348 INFO:teuthology.orchestra.run.smithi184.stdout: link/ether 0c:c4:7a:88:7b:8e brd ff:ff:ff:ff:ff:ff 2024-09-16T15:43:13.348 INFO:teuthology.orchestra.run.smithi184.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-09-16T15:43:13.349 INFO:teuthology.orchestra.run.smithi184.stdout: link/ether 0c:c4:7a:88:7b:8f brd ff:ff:ff:ff:ff:ff 2024-09-16T15:43:13.349 INFO:teuthology.orchestra.run.smithi184.stdout: inet 172.21.15.184/20 metric 100 brd 172.21.15.255 scope global dynamic enp3s0f1 2024-09-16T15:43:13.349 INFO:teuthology.orchestra.run.smithi184.stdout: valid_lft 42464sec preferred_lft 42464sec 2024-09-16T15:43:13.349 INFO:teuthology.orchestra.run.smithi184.stdout: inet6 fe80::ec4:7aff:fe88:7b8f/64 scope link 2024-09-16T15:43:13.349 INFO:teuthology.orchestra.run.smithi184.stdout: valid_lft forever preferred_lft forever 2024-09-16T15:43:13.349 INFO:teuthology.orchestra.run.smithi184.stdout:6: docker0: mtu 1500 qdisc noqueue state DOWN group default 2024-09-16T15:43:13.349 INFO:teuthology.orchestra.run.smithi184.stdout: link/ether 02:42:5d:0a:44:b9 brd ff:ff:ff:ff:ff:ff 2024-09-16T15:43:13.349 INFO:teuthology.orchestra.run.smithi184.stdout: inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0 2024-09-16T15:43:13.349 INFO:teuthology.orchestra.run.smithi184.stdout: valid_lft forever preferred_lft forever 2024-09-16T15:43:13.349 INFO:teuthology.orchestra.run.smithi184.stdout:7: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-09-16T15:43:13.350 INFO:teuthology.orchestra.run.smithi184.stdout: link/ether 5e:1f:84:6d:79:ae brd ff:ff:ff:ff:ff:ff 2024-09-16T15:43:13.350 INFO:teuthology.orchestra.run.smithi184.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-09-16T15:43:13.350 INFO:teuthology.orchestra.run.smithi184.stdout: valid_lft forever preferred_lft forever 2024-09-16T15:43:13.350 INFO:teuthology.orchestra.run.smithi184.stdout: inet6 fe80::5c1f:84ff:fe6d:79ae/64 scope link 2024-09-16T15:43:13.350 INFO:teuthology.orchestra.run.smithi184.stdout: valid_lft forever preferred_lft forever 2024-09-16T15:43:13.350 INFO:teuthology.orchestra.run.smithi184.stdout:8: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-09-16T15:43:13.350 INFO:teuthology.orchestra.run.smithi184.stdout: link/ether 0e:5c:f6:1f:82:2f brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.1 2024-09-16T15:43:13.350 INFO:teuthology.orchestra.run.smithi184.stdout: inet6 fe80::c5c:f6ff:fe1f:822f/64 scope link 2024-09-16T15:43:13.350 INFO:teuthology.orchestra.run.smithi184.stdout: valid_lft forever preferred_lft forever 2024-09-16T15:43:13.350 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:43:13.351 DEBUG:teuthology.orchestra.run.smithi184:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-09-16T15:43:13.402 INFO:teuthology.orchestra.run.smithi184.stdout:1 2024-09-16T15:43:13.404 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:43:13.404 DEBUG:teuthology.orchestra.run.smithi184:> route 2024-09-16T15:43:13.455 INFO:teuthology.orchestra.run.smithi184.stdout:Kernel IP routing table 2024-09-16T15:43:13.455 INFO:teuthology.orchestra.run.smithi184.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-09-16T15:43:13.455 INFO:teuthology.orchestra.run.smithi184.stdout:default _gateway 0.0.0.0 UG 0 0 0 enp3s0f1 2024-09-16T15:43:13.456 INFO:teuthology.orchestra.run.smithi184.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-09-16T15:43:13.456 INFO:teuthology.orchestra.run.smithi184.stdout:158.69.67.47 _gateway 255.255.255.255 UGH 100 0 0 enp3s0f1 2024-09-16T15:43:13.456 INFO:teuthology.orchestra.run.smithi184.stdout:172.17.0.0 0.0.0.0 255.255.0.0 U 0 0 0 docker0 2024-09-16T15:43:13.456 INFO:teuthology.orchestra.run.smithi184.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-09-16T15:43:13.456 INFO:teuthology.orchestra.run.smithi184.stdout:ns1.front.sepia 0.0.0.0 255.255.255.255 UH 100 0 0 enp3s0f1 2024-09-16T15:43:13.456 INFO:teuthology.orchestra.run.smithi184.stdout:172.21.0.2 0.0.0.0 255.255.255.255 UH 100 0 0 enp3s0f1 2024-09-16T15:43:13.456 INFO:teuthology.orchestra.run.smithi184.stdout:_gateway 0.0.0.0 255.255.255.255 UH 100 0 0 enp3s0f1 2024-09-16T15:43:13.456 INFO:teuthology.orchestra.run.smithi184.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-09-16T15:43:13.456 DEBUG:teuthology.orchestra.run.smithi184:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-09-16T15:43:13.456 DEBUG:teuthology.orchestra.run.smithi184:> set -e 2024-09-16T15:43:13.457 DEBUG:teuthology.orchestra.run.smithi184:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-09-16T15:43:13.457 DEBUG:teuthology.orchestra.run.smithi184:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-09-16T15:43:13.457 DEBUG:teuthology.orchestra.run.smithi184:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-09-16T15:43:13.457 DEBUG:teuthology.orchestra.run.smithi184:> ') 2024-09-16T15:43:13.544 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:43:13 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-09-16T15:43:13.572 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:43:13 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-16T15:43:13.579 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:43:13.579 DEBUG:teuthology.orchestra.run.smithi184:> ip netns list 2024-09-16T15:43:13.628 INFO:teuthology.orchestra.run.smithi184.stdout:ceph-ns--home-ubuntu-cephtest-mnt.1 (id: 0) 2024-09-16T15:43:13.628 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:43:13.628 DEBUG:teuthology.orchestra.run.smithi184:> ip netns list-id 2024-09-16T15:43:13.679 INFO:teuthology.orchestra.run.smithi184.stdout:nsid 0 (iproute2 netns name: ceph-ns--home-ubuntu-cephtest-mnt.1) 2024-09-16T15:43:13.679 DEBUG:teuthology.orchestra.run.smithi184:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-09-16T15:43:13.679 DEBUG:teuthology.orchestra.run.smithi184:> set -e 2024-09-16T15:43:13.679 DEBUG:teuthology.orchestra.run.smithi184:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.3 2024-09-16T15:43:13.680 DEBUG:teuthology.orchestra.run.smithi184:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.3 1 2024-09-16T15:43:13.680 DEBUG:teuthology.orchestra.run.smithi184:> ') 2024-09-16T15:43:13.771 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:43:13 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-09-16T15:43:13.792 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:43:13 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-16T15:43:13.800 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:43:13.800 DEBUG:teuthology.orchestra.run.smithi184:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip addr 2024-09-16T15:43:13.813 INFO:teuthology.orchestra.run.smithi184.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-09-16T15:43:13.813 INFO:teuthology.orchestra.run.smithi184.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-09-16T15:43:13.813 INFO:teuthology.orchestra.run.smithi184.stdout: inet 127.0.0.1/8 scope host lo 2024-09-16T15:43:13.813 INFO:teuthology.orchestra.run.smithi184.stdout: valid_lft forever preferred_lft forever 2024-09-16T15:43:13.813 INFO:teuthology.orchestra.run.smithi184.stdout: inet6 ::1/128 scope host 2024-09-16T15:43:13.814 INFO:teuthology.orchestra.run.smithi184.stdout: valid_lft forever preferred_lft forever 2024-09-16T15:43:13.814 INFO:teuthology.orchestra.run.smithi184.stdout:2: veth0@if8: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-09-16T15:43:13.814 INFO:teuthology.orchestra.run.smithi184.stdout: link/ether ae:11:81:82:ba:08 brd ff:ff:ff:ff:ff:ff link-netnsid 0 2024-09-16T15:43:13.814 INFO:teuthology.orchestra.run.smithi184.stdout: inet 192.168.0.1/16 brd 192.168.255.255 scope global veth0 2024-09-16T15:43:13.814 INFO:teuthology.orchestra.run.smithi184.stdout: valid_lft forever preferred_lft forever 2024-09-16T15:43:13.814 INFO:teuthology.orchestra.run.smithi184.stdout: inet6 fe80::ac11:81ff:fe82:ba08/64 scope link 2024-09-16T15:43:13.814 INFO:teuthology.orchestra.run.smithi184.stdout: valid_lft forever preferred_lft forever 2024-09-16T15:43:13.815 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:43:13.815 DEBUG:teuthology.orchestra.run.smithi184:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip addr 2024-09-16T15:43:13.871 INFO:teuthology.orchestra.run.smithi184.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-09-16T15:43:13.871 INFO:teuthology.orchestra.run.smithi184.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-09-16T15:43:13.871 INFO:teuthology.orchestra.run.smithi184.stdout: inet 127.0.0.1/8 scope host lo 2024-09-16T15:43:13.871 INFO:teuthology.orchestra.run.smithi184.stdout: valid_lft forever preferred_lft forever 2024-09-16T15:43:13.871 INFO:teuthology.orchestra.run.smithi184.stdout: inet6 ::1/128 scope host 2024-09-16T15:43:13.872 INFO:teuthology.orchestra.run.smithi184.stdout: valid_lft forever preferred_lft forever 2024-09-16T15:43:13.872 INFO:teuthology.orchestra.run.smithi184.stdout:2: veth0@if8: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-09-16T15:43:13.872 INFO:teuthology.orchestra.run.smithi184.stdout: link/ether ae:11:81:82:ba:08 brd ff:ff:ff:ff:ff:ff link-netnsid 0 2024-09-16T15:43:13.872 INFO:teuthology.orchestra.run.smithi184.stdout: inet 192.168.0.1/16 brd 192.168.255.255 scope global veth0 2024-09-16T15:43:13.872 INFO:teuthology.orchestra.run.smithi184.stdout: valid_lft forever preferred_lft forever 2024-09-16T15:43:13.872 INFO:teuthology.orchestra.run.smithi184.stdout: inet6 fe80::ac11:81ff:fe82:ba08/64 scope link 2024-09-16T15:43:13.872 INFO:teuthology.orchestra.run.smithi184.stdout: valid_lft forever preferred_lft forever 2024-09-16T15:43:13.873 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.3' with 192.168.0.2/16 2024-09-16T15:43:13.873 DEBUG:teuthology.orchestra.run.smithi184:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-09-16T15:43:13.873 DEBUG:teuthology.orchestra.run.smithi184:> set -e 2024-09-16T15:43:13.873 DEBUG:teuthology.orchestra.run.smithi184:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.3 type veth peer name brx.1 2024-09-16T15:43:13.873 DEBUG:teuthology.orchestra.run.smithi184:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.3 ip addr add 192.168.0.2/16 brd 192.168.255.255 dev veth0 2024-09-16T15:43:13.873 DEBUG:teuthology.orchestra.run.smithi184:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.3 ip link set veth0 up 2024-09-16T15:43:13.873 DEBUG:teuthology.orchestra.run.smithi184:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.3 ip link set lo up 2024-09-16T15:43:13.874 DEBUG:teuthology.orchestra.run.smithi184:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.3 ip route add default via 192.168.255.254 2024-09-16T15:43:13.874 DEBUG:teuthology.orchestra.run.smithi184:> ') 2024-09-16T15:43:13.960 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:43:13 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-09-16T15:43:14.007 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:43:14 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-16T15:43:14.014 DEBUG:teuthology.orchestra.run.smithi184:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-09-16T15:43:14.014 DEBUG:teuthology.orchestra.run.smithi184:> set -e 2024-09-16T15:43:14.014 DEBUG:teuthology.orchestra.run.smithi184:> sudo ip link set brx.1 up 2024-09-16T15:43:14.014 DEBUG:teuthology.orchestra.run.smithi184:> sudo ip link set dev brx.1 master ceph-brx 2024-09-16T15:43:14.014 DEBUG:teuthology.orchestra.run.smithi184:> ') 2024-09-16T15:43:14.053 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:43:14 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-09-16T15:43:14.068 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:43:14 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-16T15:43:14.075 INFO:tasks.cephfs.fuse_mount:Client client.3 config is {} 2024-09-16T15:43:14.075 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-09-16T15:43:14.075 DEBUG:teuthology.orchestra.run.smithi184:> mkdir -p -v /home/ubuntu/cephtest/mnt.3 2024-09-16T15:43:14.127 INFO:teuthology.orchestra.run.smithi184.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.3' 2024-09-16T15:43:14.127 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-09-16T15:43:14.127 DEBUG:teuthology.orchestra.run.smithi184:> chmod 0000 /home/ubuntu/cephtest/mnt.3 2024-09-16T15:43:14.174 DEBUG:teuthology.orchestra.run.smithi184:> sudo modprobe fuse 2024-09-16T15:43:14.225 DEBUG:teuthology.orchestra.run.smithi184:> cat /proc/self/mounts | awk '{print $2}' 2024-09-16T15:43:14.274 INFO:teuthology.orchestra.run.smithi184.stdout:/sys 2024-09-16T15:43:14.274 INFO:teuthology.orchestra.run.smithi184.stdout:/proc 2024-09-16T15:43:14.274 INFO:teuthology.orchestra.run.smithi184.stdout:/dev 2024-09-16T15:43:14.275 INFO:teuthology.orchestra.run.smithi184.stdout:/dev/pts 2024-09-16T15:43:14.275 INFO:teuthology.orchestra.run.smithi184.stdout:/run 2024-09-16T15:43:14.275 INFO:teuthology.orchestra.run.smithi184.stdout:/ 2024-09-16T15:43:14.275 INFO:teuthology.orchestra.run.smithi184.stdout:/sys/kernel/security 2024-09-16T15:43:14.275 INFO:teuthology.orchestra.run.smithi184.stdout:/dev/shm 2024-09-16T15:43:14.275 INFO:teuthology.orchestra.run.smithi184.stdout:/run/lock 2024-09-16T15:43:14.275 INFO:teuthology.orchestra.run.smithi184.stdout:/sys/fs/cgroup 2024-09-16T15:43:14.275 INFO:teuthology.orchestra.run.smithi184.stdout:/sys/fs/pstore 2024-09-16T15:43:14.275 INFO:teuthology.orchestra.run.smithi184.stdout:/sys/fs/bpf 2024-09-16T15:43:14.275 INFO:teuthology.orchestra.run.smithi184.stdout:/proc/sys/fs/binfmt_misc 2024-09-16T15:43:14.275 INFO:teuthology.orchestra.run.smithi184.stdout:/dev/hugepages 2024-09-16T15:43:14.275 INFO:teuthology.orchestra.run.smithi184.stdout:/dev/mqueue 2024-09-16T15:43:14.276 INFO:teuthology.orchestra.run.smithi184.stdout:/sys/kernel/debug 2024-09-16T15:43:14.276 INFO:teuthology.orchestra.run.smithi184.stdout:/sys/kernel/tracing 2024-09-16T15:43:14.276 INFO:teuthology.orchestra.run.smithi184.stdout:/proc/fs/nfsd 2024-09-16T15:43:14.276 INFO:teuthology.orchestra.run.smithi184.stdout:/sys/fs/fuse/connections 2024-09-16T15:43:14.276 INFO:teuthology.orchestra.run.smithi184.stdout:/sys/kernel/config 2024-09-16T15:43:14.276 INFO:teuthology.orchestra.run.smithi184.stdout:/run/credentials/systemd-sysusers.service 2024-09-16T15:43:14.276 INFO:teuthology.orchestra.run.smithi184.stdout:/run/rpc_pipefs 2024-09-16T15:43:14.276 INFO:teuthology.orchestra.run.smithi184.stdout:/run/user/1000 2024-09-16T15:43:14.276 INFO:teuthology.orchestra.run.smithi184.stdout:/var/lib/ceph 2024-09-16T15:43:14.276 INFO:teuthology.orchestra.run.smithi184.stdout:/var/lib/ceph/osd/ceph-4 2024-09-16T15:43:14.276 INFO:teuthology.orchestra.run.smithi184.stdout:/var/lib/ceph/osd/ceph-5 2024-09-16T15:43:14.276 INFO:teuthology.orchestra.run.smithi184.stdout:/var/lib/ceph/osd/ceph-6 2024-09-16T15:43:14.277 INFO:teuthology.orchestra.run.smithi184.stdout:/var/lib/ceph/osd/ceph-7 2024-09-16T15:43:14.277 INFO:teuthology.orchestra.run.smithi184.stdout:/run/netns 2024-09-16T15:43:14.277 INFO:teuthology.orchestra.run.smithi184.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-09-16T15:43:14.277 INFO:teuthology.orchestra.run.smithi184.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-09-16T15:43:14.277 INFO:teuthology.orchestra.run.smithi184.stdout:/home/ubuntu/cephtest/mnt.1 2024-09-16T15:43:14.277 INFO:teuthology.orchestra.run.smithi184.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-09-16T15:43:14.277 INFO:teuthology.orchestra.run.smithi184.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-09-16T15:43:14.278 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:43:14.278 DEBUG:teuthology.orchestra.run.smithi184:> ls /sys/fs/fuse/connections 2024-09-16T15:43:14.327 INFO:teuthology.orchestra.run.smithi184.stdout:45 2024-09-16T15:43:14.327 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [45] 2024-09-16T15:43:14.327 DEBUG:teuthology.orchestra.run.smithi184:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.3 --id 3) 2024-09-16T15:43:14.372 DEBUG:teuthology.orchestra.run.smithi184:> sudo modprobe fuse 2024-09-16T15:43:14.387 DEBUG:teuthology.orchestra.run.smithi184:> cat /proc/self/mounts | awk '{print $2}' 2024-09-16T15:43:14.435 INFO:teuthology.orchestra.run.smithi184.stdout:/sys 2024-09-16T15:43:14.435 INFO:teuthology.orchestra.run.smithi184.stdout:/proc 2024-09-16T15:43:14.435 INFO:teuthology.orchestra.run.smithi184.stdout:/dev 2024-09-16T15:43:14.435 INFO:teuthology.orchestra.run.smithi184.stdout:/dev/pts 2024-09-16T15:43:14.435 INFO:teuthology.orchestra.run.smithi184.stdout:/run 2024-09-16T15:43:14.435 INFO:teuthology.orchestra.run.smithi184.stdout:/ 2024-09-16T15:43:14.435 INFO:teuthology.orchestra.run.smithi184.stdout:/sys/kernel/security 2024-09-16T15:43:14.436 INFO:teuthology.orchestra.run.smithi184.stdout:/dev/shm 2024-09-16T15:43:14.436 INFO:teuthology.orchestra.run.smithi184.stdout:/run/lock 2024-09-16T15:43:14.436 INFO:teuthology.orchestra.run.smithi184.stdout:/sys/fs/cgroup 2024-09-16T15:43:14.436 INFO:teuthology.orchestra.run.smithi184.stdout:/sys/fs/pstore 2024-09-16T15:43:14.436 INFO:teuthology.orchestra.run.smithi184.stdout:/sys/fs/bpf 2024-09-16T15:43:14.436 INFO:teuthology.orchestra.run.smithi184.stdout:/proc/sys/fs/binfmt_misc 2024-09-16T15:43:14.436 INFO:teuthology.orchestra.run.smithi184.stdout:/dev/hugepages 2024-09-16T15:43:14.436 INFO:teuthology.orchestra.run.smithi184.stdout:/dev/mqueue 2024-09-16T15:43:14.436 INFO:teuthology.orchestra.run.smithi184.stdout:/sys/kernel/debug 2024-09-16T15:43:14.436 INFO:teuthology.orchestra.run.smithi184.stdout:/sys/kernel/tracing 2024-09-16T15:43:14.436 INFO:teuthology.orchestra.run.smithi184.stdout:/proc/fs/nfsd 2024-09-16T15:43:14.436 INFO:teuthology.orchestra.run.smithi184.stdout:/sys/fs/fuse/connections 2024-09-16T15:43:14.436 INFO:teuthology.orchestra.run.smithi184.stdout:/sys/kernel/config 2024-09-16T15:43:14.437 INFO:teuthology.orchestra.run.smithi184.stdout:/run/credentials/systemd-sysusers.service 2024-09-16T15:43:14.437 INFO:teuthology.orchestra.run.smithi184.stdout:/run/rpc_pipefs 2024-09-16T15:43:14.437 INFO:teuthology.orchestra.run.smithi184.stdout:/run/user/1000 2024-09-16T15:43:14.437 INFO:teuthology.orchestra.run.smithi184.stdout:/var/lib/ceph 2024-09-16T15:43:14.437 INFO:teuthology.orchestra.run.smithi184.stdout:/var/lib/ceph/osd/ceph-4 2024-09-16T15:43:14.437 INFO:teuthology.orchestra.run.smithi184.stdout:/var/lib/ceph/osd/ceph-5 2024-09-16T15:43:14.437 INFO:teuthology.orchestra.run.smithi184.stdout:/var/lib/ceph/osd/ceph-6 2024-09-16T15:43:14.437 INFO:teuthology.orchestra.run.smithi184.stdout:/var/lib/ceph/osd/ceph-7 2024-09-16T15:43:14.437 INFO:teuthology.orchestra.run.smithi184.stdout:/run/netns 2024-09-16T15:43:14.437 INFO:teuthology.orchestra.run.smithi184.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-09-16T15:43:14.437 INFO:teuthology.orchestra.run.smithi184.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-09-16T15:43:14.437 INFO:teuthology.orchestra.run.smithi184.stdout:/home/ubuntu/cephtest/mnt.1 2024-09-16T15:43:14.437 INFO:teuthology.orchestra.run.smithi184.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-09-16T15:43:14.438 INFO:teuthology.orchestra.run.smithi184.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-09-16T15:43:14.438 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:43:14.439 DEBUG:teuthology.orchestra.run.smithi184:> ls /sys/fs/fuse/connections 2024-09-16T15:43:14.450 INFO:tasks.cephfs.fuse_mount.ceph-fuse.3.smithi184.stderr:2024-09-16T15:43:14.446+0000 7f3a7cc04580 -1 init, newargv = 0x55e680a12100 newargc=15 2024-09-16T15:43:14.450 INFO:tasks.cephfs.fuse_mount.ceph-fuse.3.smithi184.stderr:ceph-fuse[34464]: starting ceph client 2024-09-16T15:43:14.454 INFO:teuthology.orchestra.run.smithi184.stdout:45 2024-09-16T15:43:14.456 INFO:tasks.cephfs.fuse_mount.ceph-fuse.3.smithi184.stderr:ceph-fuse[34464]: starting fuse 2024-09-16T15:43:15.555 DEBUG:teuthology.orchestra.run.smithi184:> sudo modprobe fuse 2024-09-16T15:43:15.572 DEBUG:teuthology.orchestra.run.smithi184:> cat /proc/self/mounts | awk '{print $2}' 2024-09-16T15:43:15.621 INFO:teuthology.orchestra.run.smithi184.stdout:/sys 2024-09-16T15:43:15.622 INFO:teuthology.orchestra.run.smithi184.stdout:/proc 2024-09-16T15:43:15.622 INFO:teuthology.orchestra.run.smithi184.stdout:/dev 2024-09-16T15:43:15.622 INFO:teuthology.orchestra.run.smithi184.stdout:/dev/pts 2024-09-16T15:43:15.622 INFO:teuthology.orchestra.run.smithi184.stdout:/run 2024-09-16T15:43:15.622 INFO:teuthology.orchestra.run.smithi184.stdout:/ 2024-09-16T15:43:15.622 INFO:teuthology.orchestra.run.smithi184.stdout:/sys/kernel/security 2024-09-16T15:43:15.622 INFO:teuthology.orchestra.run.smithi184.stdout:/dev/shm 2024-09-16T15:43:15.622 INFO:teuthology.orchestra.run.smithi184.stdout:/run/lock 2024-09-16T15:43:15.622 INFO:teuthology.orchestra.run.smithi184.stdout:/sys/fs/cgroup 2024-09-16T15:43:15.622 INFO:teuthology.orchestra.run.smithi184.stdout:/sys/fs/pstore 2024-09-16T15:43:15.622 INFO:teuthology.orchestra.run.smithi184.stdout:/sys/fs/bpf 2024-09-16T15:43:15.622 INFO:teuthology.orchestra.run.smithi184.stdout:/proc/sys/fs/binfmt_misc 2024-09-16T15:43:15.622 INFO:teuthology.orchestra.run.smithi184.stdout:/dev/hugepages 2024-09-16T15:43:15.623 INFO:teuthology.orchestra.run.smithi184.stdout:/dev/mqueue 2024-09-16T15:43:15.623 INFO:teuthology.orchestra.run.smithi184.stdout:/sys/kernel/debug 2024-09-16T15:43:15.623 INFO:teuthology.orchestra.run.smithi184.stdout:/sys/kernel/tracing 2024-09-16T15:43:15.623 INFO:teuthology.orchestra.run.smithi184.stdout:/proc/fs/nfsd 2024-09-16T15:43:15.623 INFO:teuthology.orchestra.run.smithi184.stdout:/sys/fs/fuse/connections 2024-09-16T15:43:15.623 INFO:teuthology.orchestra.run.smithi184.stdout:/sys/kernel/config 2024-09-16T15:43:15.623 INFO:teuthology.orchestra.run.smithi184.stdout:/run/credentials/systemd-sysusers.service 2024-09-16T15:43:15.623 INFO:teuthology.orchestra.run.smithi184.stdout:/run/rpc_pipefs 2024-09-16T15:43:15.623 INFO:teuthology.orchestra.run.smithi184.stdout:/run/user/1000 2024-09-16T15:43:15.623 INFO:teuthology.orchestra.run.smithi184.stdout:/var/lib/ceph 2024-09-16T15:43:15.623 INFO:teuthology.orchestra.run.smithi184.stdout:/var/lib/ceph/osd/ceph-4 2024-09-16T15:43:15.623 INFO:teuthology.orchestra.run.smithi184.stdout:/var/lib/ceph/osd/ceph-5 2024-09-16T15:43:15.623 INFO:teuthology.orchestra.run.smithi184.stdout:/var/lib/ceph/osd/ceph-6 2024-09-16T15:43:15.624 INFO:teuthology.orchestra.run.smithi184.stdout:/var/lib/ceph/osd/ceph-7 2024-09-16T15:43:15.624 INFO:teuthology.orchestra.run.smithi184.stdout:/run/netns 2024-09-16T15:43:15.624 INFO:teuthology.orchestra.run.smithi184.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-09-16T15:43:15.624 INFO:teuthology.orchestra.run.smithi184.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-09-16T15:43:15.624 INFO:teuthology.orchestra.run.smithi184.stdout:/home/ubuntu/cephtest/mnt.1 2024-09-16T15:43:15.624 INFO:teuthology.orchestra.run.smithi184.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-09-16T15:43:15.624 INFO:teuthology.orchestra.run.smithi184.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-09-16T15:43:15.624 INFO:teuthology.orchestra.run.smithi184.stdout:/home/ubuntu/cephtest/mnt.3 2024-09-16T15:43:15.625 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:43:15.625 DEBUG:teuthology.orchestra.run.smithi184:> ls /sys/fs/fuse/connections 2024-09-16T15:43:15.671 INFO:teuthology.orchestra.run.smithi184.stdout:45 2024-09-16T15:43:15.671 INFO:teuthology.orchestra.run.smithi184.stdout:46 2024-09-16T15:43:15.672 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [45, 46] 2024-09-16T15:43:15.672 DEBUG:teuthology.orchestra.run.smithi184:> sudo stdin-killer -- python3 -c ' 2024-09-16T15:43:15.672 DEBUG:teuthology.orchestra.run.smithi184:> import glob 2024-09-16T15:43:15.672 DEBUG:teuthology.orchestra.run.smithi184:> import re 2024-09-16T15:43:15.672 DEBUG:teuthology.orchestra.run.smithi184:> import os 2024-09-16T15:43:15.672 DEBUG:teuthology.orchestra.run.smithi184:> import subprocess 2024-09-16T15:43:15.672 DEBUG:teuthology.orchestra.run.smithi184:> 2024-09-16T15:43:15.672 DEBUG:teuthology.orchestra.run.smithi184:> def _find_admin_socket(client_name): 2024-09-16T15:43:15.672 DEBUG:teuthology.orchestra.run.smithi184:> asok_path = "/var/run/ceph/ceph-client.3.*.asok" 2024-09-16T15:43:15.672 DEBUG:teuthology.orchestra.run.smithi184:> files = glob.glob(asok_path) 2024-09-16T15:43:15.673 DEBUG:teuthology.orchestra.run.smithi184:> mountpoint = "/home/ubuntu/cephtest/mnt.3" 2024-09-16T15:43:15.673 DEBUG:teuthology.orchestra.run.smithi184:> 2024-09-16T15:43:15.673 DEBUG:teuthology.orchestra.run.smithi184:> # Given a non-glob path, it better be there 2024-09-16T15:43:15.673 DEBUG:teuthology.orchestra.run.smithi184:> if "*" not in asok_path: 2024-09-16T15:43:15.673 DEBUG:teuthology.orchestra.run.smithi184:> assert(len(files) == 1) 2024-09-16T15:43:15.673 DEBUG:teuthology.orchestra.run.smithi184:> return files[0] 2024-09-16T15:43:15.673 DEBUG:teuthology.orchestra.run.smithi184:> 2024-09-16T15:43:15.673 DEBUG:teuthology.orchestra.run.smithi184:> for f in files: 2024-09-16T15:43:15.673 DEBUG:teuthology.orchestra.run.smithi184:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-09-16T15:43:15.673 DEBUG:teuthology.orchestra.run.smithi184:> if os.path.exists("/proc/{0}".format(pid)): 2024-09-16T15:43:15.673 DEBUG:teuthology.orchestra.run.smithi184:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-09-16T15:43:15.673 DEBUG:teuthology.orchestra.run.smithi184:> contents = proc_f.read() 2024-09-16T15:43:15.673 DEBUG:teuthology.orchestra.run.smithi184:> if mountpoint in contents: 2024-09-16T15:43:15.674 DEBUG:teuthology.orchestra.run.smithi184:> return f 2024-09-16T15:43:15.674 DEBUG:teuthology.orchestra.run.smithi184:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-09-16T15:43:15.674 DEBUG:teuthology.orchestra.run.smithi184:> 2024-09-16T15:43:15.674 DEBUG:teuthology.orchestra.run.smithi184:> print(_find_admin_socket("client.3")) 2024-09-16T15:43:15.674 DEBUG:teuthology.orchestra.run.smithi184:> ' 2024-09-16T15:43:15.800 INFO:teuthology.orchestra.run.smithi184.stdout:/var/run/ceph/ceph-client.3.34464.asok 2024-09-16T15:43:15.804 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:43:15 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-16T15:43:15.812 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.3.34464.asok 2024-09-16T15:43:15.812 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:43:15.812 DEBUG:teuthology.orchestra.run.smithi184:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.3.34464.asok status 2024-09-16T15:43:15.962 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-16T15:43:15.962 INFO:teuthology.orchestra.run.smithi184.stdout: "metadata": { 2024-09-16T15:43:15.962 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph_sha1": "33f95d2ae1bcbe7eb04404303f179eeeaa6982df", 2024-09-16T15:43:15.962 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph_version": "ceph version 19.3.0-4938-g33f95d2a (33f95d2ae1bcbe7eb04404303f179eeeaa6982df) squid (dev)", 2024-09-16T15:43:15.962 INFO:teuthology.orchestra.run.smithi184.stdout: "entity_id": "3", 2024-09-16T15:43:15.962 INFO:teuthology.orchestra.run.smithi184.stdout: "hostname": "smithi184", 2024-09-16T15:43:15.962 INFO:teuthology.orchestra.run.smithi184.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.3", 2024-09-16T15:43:15.962 INFO:teuthology.orchestra.run.smithi184.stdout: "pid": "34464", 2024-09-16T15:43:15.962 INFO:teuthology.orchestra.run.smithi184.stdout: "root": "/" 2024-09-16T15:43:15.963 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-16T15:43:15.963 INFO:teuthology.orchestra.run.smithi184.stdout: "dentry_count": 0, 2024-09-16T15:43:15.963 INFO:teuthology.orchestra.run.smithi184.stdout: "dentry_pinned_count": 0, 2024-09-16T15:43:15.963 INFO:teuthology.orchestra.run.smithi184.stdout: "id": 4454, 2024-09-16T15:43:15.963 INFO:teuthology.orchestra.run.smithi184.stdout: "inst": { 2024-09-16T15:43:15.963 INFO:teuthology.orchestra.run.smithi184.stdout: "name": { 2024-09-16T15:43:15.963 INFO:teuthology.orchestra.run.smithi184.stdout: "type": "client", 2024-09-16T15:43:15.963 INFO:teuthology.orchestra.run.smithi184.stdout: "num": 4454 2024-09-16T15:43:15.963 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-16T15:43:15.963 INFO:teuthology.orchestra.run.smithi184.stdout: "addr": { 2024-09-16T15:43:15.963 INFO:teuthology.orchestra.run.smithi184.stdout: "type": "v1", 2024-09-16T15:43:15.963 INFO:teuthology.orchestra.run.smithi184.stdout: "addr": "192.168.0.2:0", 2024-09-16T15:43:15.963 INFO:teuthology.orchestra.run.smithi184.stdout: "nonce": 1554737055 2024-09-16T15:43:15.963 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-16T15:43:15.964 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-16T15:43:15.964 INFO:teuthology.orchestra.run.smithi184.stdout: "addr": { 2024-09-16T15:43:15.964 INFO:teuthology.orchestra.run.smithi184.stdout: "type": "v1", 2024-09-16T15:43:15.964 INFO:teuthology.orchestra.run.smithi184.stdout: "addr": "192.168.0.2:0", 2024-09-16T15:43:15.964 INFO:teuthology.orchestra.run.smithi184.stdout: "nonce": 1554737055 2024-09-16T15:43:15.964 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-16T15:43:15.964 INFO:teuthology.orchestra.run.smithi184.stdout: "inst_str": "client.4454 192.168.0.2:0/1554737055", 2024-09-16T15:43:15.964 INFO:teuthology.orchestra.run.smithi184.stdout: "addr_str": "192.168.0.2:0/1554737055", 2024-09-16T15:43:15.964 INFO:teuthology.orchestra.run.smithi184.stdout: "inode_count": 1, 2024-09-16T15:43:15.964 INFO:teuthology.orchestra.run.smithi184.stdout: "mds_epoch": 8, 2024-09-16T15:43:15.964 INFO:teuthology.orchestra.run.smithi184.stdout: "osd_epoch": 21, 2024-09-16T15:43:15.964 INFO:teuthology.orchestra.run.smithi184.stdout: "osd_epoch_barrier": 0, 2024-09-16T15:43:15.964 INFO:teuthology.orchestra.run.smithi184.stdout: "blocklisted": false, 2024-09-16T15:43:15.965 INFO:teuthology.orchestra.run.smithi184.stdout: "fs_name": "cephfs" 2024-09-16T15:43:15.965 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-16T15:43:15.976 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:43:15.976 DEBUG:teuthology.orchestra.run.smithi123:> stat --file-system '--printf=%T 2024-09-16T15:43:15.976 DEBUG:teuthology.orchestra.run.smithi123:> ' -- /home/ubuntu/cephtest/mnt.0 2024-09-16T15:43:15.983 INFO:teuthology.orchestra.run.smithi123.stdout:fuseblk 2024-09-16T15:43:15.984 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2024-09-16T15:43:15.984 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:43:15.984 DEBUG:teuthology.orchestra.run.smithi123:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-09-16T15:43:16.043 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:43:16.043 DEBUG:teuthology.orchestra.run.smithi123:> stat --file-system '--printf=%T 2024-09-16T15:43:16.043 DEBUG:teuthology.orchestra.run.smithi123:> ' -- /home/ubuntu/cephtest/mnt.2 2024-09-16T15:43:16.092 INFO:teuthology.orchestra.run.smithi123.stdout:fuseblk 2024-09-16T15:43:16.092 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.2 2024-09-16T15:43:16.092 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:43:16.093 DEBUG:teuthology.orchestra.run.smithi123:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.2 2024-09-16T15:43:16.153 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:43:16.153 DEBUG:teuthology.orchestra.run.smithi184:> stat --file-system '--printf=%T 2024-09-16T15:43:16.153 DEBUG:teuthology.orchestra.run.smithi184:> ' -- /home/ubuntu/cephtest/mnt.1 2024-09-16T15:43:16.160 INFO:teuthology.orchestra.run.smithi184.stdout:fuseblk 2024-09-16T15:43:16.160 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2024-09-16T15:43:16.161 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:43:16.161 DEBUG:teuthology.orchestra.run.smithi184:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.1 2024-09-16T15:43:16.216 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:43:16.216 DEBUG:teuthology.orchestra.run.smithi184:> stat --file-system '--printf=%T 2024-09-16T15:43:16.216 DEBUG:teuthology.orchestra.run.smithi184:> ' -- /home/ubuntu/cephtest/mnt.3 2024-09-16T15:43:16.264 INFO:teuthology.orchestra.run.smithi184.stdout:fuseblk 2024-09-16T15:43:16.264 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.3 2024-09-16T15:43:16.264 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:43:16.264 DEBUG:teuthology.orchestra.run.smithi184:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.3 2024-09-16T15:43:16.319 INFO:teuthology.run_tasks:Running task cephfs_test_runner... 2024-09-16T15:43:16.402 INFO:tasks.cephfs_test_runner:Starting test: test_binary_metafile_on_legacy_to_v1_upgrade (tasks.cephfs.test_volumes.TestMisc) Validate binary .meta file on legacy subvol root doesn't break the system 2024-09-16T15:43:16.403 INFO:tasks.cephfs_test_runner:test_binary_metafile_on_legacy_to_v1_upgrade (tasks.cephfs.test_volumes.TestMisc) 2024-09-16T15:43:16.403 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-16T15:43:16.539 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:16.538+0000 7f645c8f7640 1 -- 172.21.15.123:0/3110416075 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6440004090 msgr2=0x7f6440004810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:43:16.540 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:16.538+0000 7f645c8f7640 1 --2- 172.21.15.123:0/3110416075 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6440004090 0x7f6440004810 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f644800c290 tx=0x7f644801cf70 comp rx=0 tx=0).stop 2024-09-16T15:43:16.540 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:16.538+0000 7f6456d9d640 1 -- 172.21.15.123:0/3110416075 reap_dead start 2024-09-16T15:43:16.540 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:16.538+0000 7f645c8f7640 1 -- 172.21.15.123:0/3110416075 shutdown_connections 2024-09-16T15:43:16.540 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:16.538+0000 7f645c8f7640 1 -- 172.21.15.123:0/3110416075 >> 172.21.15.123:0/3110416075 conn(0x7f64580cfb10 msgr2=0x7f64580cff20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:43:16.540 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:16.538+0000 7f645c8f7640 1 -- 172.21.15.123:0/3110416075 shutdown_connections 2024-09-16T15:43:16.540 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:16.538+0000 7f645c8f7640 1 -- 172.21.15.123:0/3110416075 wait complete. 2024-09-16T15:43:16.540 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:16.538+0000 7f645c8f7640 1 Processor -- start 2024-09-16T15:43:16.541 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:16.538+0000 7f645c8f7640 1 -- start start 2024-09-16T15:43:16.542 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:16.538+0000 7f645c8f7640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f6440004090 0x7f64581f7960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:43:16.542 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:16.538+0000 7f645c8f7640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f64581fb9b0 0x7f6458154710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:43:16.542 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:16.538+0000 7f645c8f7640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6458154c50 0x7f64580c9580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:43:16.543 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:16.538+0000 7f645c8f7640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f645806f7a0 con 0x7f64581fb9b0 2024-09-16T15:43:16.543 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:16.538+0000 7f645c8f7640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f645806f620 con 0x7f6458154c50 2024-09-16T15:43:16.543 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:16.538+0000 7f645c8f7640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f645806f920 con 0x7f6440004090 2024-09-16T15:43:16.543 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:16.538+0000 7f6455d9b640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f64581fb9b0 0x7f6458154710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:43:16.543 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:16.538+0000 7f6455d9b640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f64581fb9b0 0x7f6458154710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:51532/0 (socket says 172.21.15.123:51532) 2024-09-16T15:43:16.543 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:16.538+0000 7f6455d9b640 1 -- 172.21.15.123:0/2671834019 learned_addr learned my addr 172.21.15.123:0/2671834019 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:43:16.543 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:16.538+0000 7f645659c640 1 --2- 172.21.15.123:0/2671834019 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f6440004090 0x7f64581f7960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:43:16.543 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:16.538+0000 7f6456d9d640 1 --2- 172.21.15.123:0/2671834019 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6458154c50 0x7f64580c9580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:43:16.543 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:16.538+0000 7f6455d9b640 1 -- 172.21.15.123:0/2671834019 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f6440004090 msgr2=0x7f64581f7960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:43:16.543 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:16.538+0000 7f6455d9b640 1 --2- 172.21.15.123:0/2671834019 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f6440004090 0x7f64581f7960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:16.543 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:16.538+0000 7f6455d9b640 1 -- 172.21.15.123:0/2671834019 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6458154c50 msgr2=0x7f64580c9580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:43:16.543 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:16.538+0000 7f6455d9b640 1 --2- 172.21.15.123:0/2671834019 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6458154c50 0x7f64580c9580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:16.544 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:16.538+0000 7f6455d9b640 1 -- 172.21.15.123:0/2671834019 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f64580ce1c0 con 0x7f64581fb9b0 2024-09-16T15:43:16.544 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:16.538+0000 7f6455d9b640 1 --2- 172.21.15.123:0/2671834019 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f64581fb9b0 0x7f6458154710 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f644801d450 tx=0x7f6448006d70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:43:16.544 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:16.538+0000 7f6446ffd640 1 -- 172.21.15.123:0/2671834019 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64480060d0 con 0x7f64581fb9b0 2024-09-16T15:43:16.544 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:16.542+0000 7f6446ffd640 1 -- 172.21.15.123:0/2671834019 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f6448003600 con 0x7f64581fb9b0 2024-09-16T15:43:16.544 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:16.542+0000 7f645c8f7640 1 -- 172.21.15.123:0/2671834019 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f64581f8000 con 0x7f64581fb9b0 2024-09-16T15:43:16.544 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:16.542+0000 7f6446ffd640 1 -- 172.21.15.123:0/2671834019 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6448003a30 con 0x7f64581fb9b0 2024-09-16T15:43:16.544 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:16.542+0000 7f645c8f7640 1 -- 172.21.15.123:0/2671834019 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f64581f85c0 con 0x7f64581fb9b0 2024-09-16T15:43:16.545 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:16.542+0000 7f645c8f7640 1 -- 172.21.15.123:0/2671834019 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f64580d0630 con 0x7f64581fb9b0 2024-09-16T15:43:16.545 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:16.542+0000 7f6446ffd640 1 -- 172.21.15.123:0/2671834019 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 5) ==== 97080+0+0 (secure 0 0 0) 0x7f6448003bd0 con 0x7f64581fb9b0 2024-09-16T15:43:16.545 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:16.542+0000 7f6446ffd640 1 --2- 172.21.15.123:0/2671834019 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f6438072ef0 0x7f64380753b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:43:16.545 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:16.542+0000 7f6446ffd640 1 -- 172.21.15.123:0/2671834019 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f644812dff0 con 0x7f64581fb9b0 2024-09-16T15:43:16.545 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:16.542+0000 7f645659c640 1 --2- 172.21.15.123:0/2671834019 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f6438072ef0 0x7f64380753b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:43:16.546 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:16.542+0000 7f645659c640 1 --2- 172.21.15.123:0/2671834019 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f6438072ef0 0x7f64380753b0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f644c0058d0 tx=0x7f644c005860 comp rx=0 tx=0).ready entity=mgr.4135 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:43:16.549 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:16.546+0000 7f6446ffd640 1 -- 172.21.15.123:0/2671834019 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f64480c0030 con 0x7f64581fb9b0 2024-09-16T15:43:16.723 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:16.722+0000 7f645c8f7640 1 -- 172.21.15.123:0/2671834019 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f645815f630 con 0x7f64581fb9b0 2024-09-16T15:43:16.726 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:16.722+0000 7f6446ffd640 1 -- 172.21.15.123:0/2671834019 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) ==== 74+0+16725 (secure 0 0 0) 0x7f6448016d50 con 0x7f64581fb9b0 2024-09-16T15:43:16.726 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-09-16T15:43:16.726 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":21,"fsid":"56077f89-18c8-486f-ae7e-eecc0bb7822c","created":"2024-09-16T15:42:26.364892+0000","modified":"2024-09-16T15:42:52.401147+0000","last_up_change":"2024-09-16T15:42:38.707234+0000","last_in_change":"2024-09-16T15:42:29.423662+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-16T15:42:39.287764+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-09-16T15:42:48.583563+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-09-16T15:42:49.704665+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9e7c1955-a607-4671-9be7-f2ef760ceeaa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6800","nonce":3882044969},{"type":"v1","addr":"172.21.15.123:6801","nonce":3882044969}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6802","nonce":3882044969},{"type":"v1","addr":"172.21.15.123:6803","nonce":3882044969}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6806","nonce":3882044969},{"type":"v1","addr":"172.21.15.123:6807","nonce":3882044969}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6804","nonce":3882044969},{"type":"v1","addr":"172.21.15.123:6805","nonce":3882044969}]},"public_addr":"172.21.15.123:6801/3882044969","cluster_addr":"172.21.15.123:6803/3882044969","heartbeat_back_addr":"172.21.15.123:6807/3882044969","heartbeat_front_addr":"172.21.15.123:6805/3882044969","state":["exists","up"]},{"osd":1,"uuid":"c540cfde-0977-4742-9b17-fd634668e010","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6816","nonce":4037001850},{"type":"v1","addr":"172.21.15.123:6817","nonce":4037001850}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6818","nonce":4037001850},{"type":"v1","addr":"172.21.15.123:6819","nonce":4037001850}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6822","nonce":4037001850},{"type":"v1","addr":"172.21.15.123:6823","nonce":4037001850}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6820","nonce":4037001850},{"type":"v1","addr":"172.21.15.123:6821","nonce":4037001850}]},"public_addr":"172.21.15.123:6817/4037001850","cluster_addr":"172.21.15.123:6819/4037001850","heartbeat_back_addr":"172.21.15.123:6823/4037001850","heartbeat_front_addr":"172.21.15.123:6821/4037001850","state":["exists","up"]},{"osd":2,"uuid":"9864a496-f169-486b-95c4-1d59162793a7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6824","nonce":1332143088},{"type":"v1","addr":"172.21.15.123:6825","nonce":1332143088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6826","nonce":1332143088},{"type":"v1","addr":"172.21.15.123:6827","nonce":1332143088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6830","nonce":1332143088},{"type":"v1","addr":"172.21.15.123:6831","nonce":1332143088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6828","nonce":1332143088},{"type":"v1","addr":"172.21.15.123:6829","nonce":1332143088}]},"public_addr":"172.21.15.123:6825/1332143088","cluster_addr":"172.21.15.123:6827/1332143088","heartbeat_back_addr":"172.21.15.123:6831/1332143088","heartbeat_front_addr":"172.21.15.123:6829/1332143088","state":["exists","up"]},{"osd":3,"uuid":"c6d1f817-f074-4325-95e1-578eba13edda","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6808","nonce":988586998},{"type":"v1","addr":"172.21.15.123:6809","nonce":988586998}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6810","nonce":988586998},{"type":"v1","addr":"172.21.15.123:6811","nonce":988586998}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6814","nonce":988586998},{"type":"v1","addr":"172.21.15.123:6815","nonce":988586998}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6812","nonce":988586998},{"type":"v1","addr":"172.21.15.123:6813","nonce":988586998}]},"public_addr":"172.21.15.123:6809/988586998","cluster_addr":"172.21.15.123:6811/988586998","heartbeat_back_addr":"172.21.15.123:6815/988586998","heartbeat_front_addr":"172.21.15.123:6813/988586998","state":["exists","up"]},{"osd":4,"uuid":"200a8df4-50ef-40a5-b48a-58981269c086","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6816","nonce":3387813890},{"type":"v1","addr":"172.21.15.184:6817","nonce":3387813890}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6818","nonce":3387813890},{"type":"v1","addr":"172.21.15.184:6819","nonce":3387813890}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6822","nonce":3387813890},{"type":"v1","addr":"172.21.15.184:6823","nonce":3387813890}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6820","nonce":3387813890},{"type":"v1","addr":"172.21.15.184:6821","nonce":3387813890}]},"public_addr":"172.21.15.184:6817/3387813890","cluster_addr":"172.21.15.184:6819/3387813890","heartbeat_back_addr":"172.21.15.184:6823/3387813890","heartbeat_front_addr":"172.21.15.184:6821/3387813890","state":["exists","up"]},{"osd":5,"uuid":"32d5a900-da3d-412c-bc19-13861a2368d8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6808","nonce":2680916017},{"type":"v1","addr":"172.21.15.184:6809","nonce":2680916017}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6810","nonce":2680916017},{"type":"v1","addr":"172.21.15.184:6811","nonce":2680916017}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6814","nonce":2680916017},{"type":"v1","addr":"172.21.15.184:6815","nonce":2680916017}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6812","nonce":2680916017},{"type":"v1","addr":"172.21.15.184:6813","nonce":2680916017}]},"public_addr":"172.21.15.184:6809/2680916017","cluster_addr":"172.21.15.184:6811/2680916017","heartbeat_back_addr":"172.21.15.184:6815/2680916017","heartbeat_front_addr":"172.21.15.184:6813/2680916017","state":["exists","up"]},{"osd":6,"uuid":"cde6f39e-aa1b-4260-ac19-774ee6aaca72","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6824","nonce":2183120457},{"type":"v1","addr":"172.21.15.184:6825","nonce":2183120457}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6826","nonce":2183120457},{"type":"v1","addr":"172.21.15.184:6827","nonce":2183120457}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6830","nonce":2183120457},{"type":"v1","addr":"172.21.15.184:6831","nonce":2183120457}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6828","nonce":2183120457},{"type":"v1","addr":"172.21.15.184:6829","nonce":2183120457}]},"public_addr":"172.21.15.184:6825/2183120457","cluster_addr":"172.21.15.184:6827/2183120457","heartbeat_back_addr":"172.21.15.184:6831/2183120457","heartbeat_front_addr":"172.21.15.184:6829/2183120457","state":["exists","up"]},{"osd":7,"uuid":"4f381778-eee1-47c6-804f-df0e6b33e313","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6800","nonce":4102957285},{"type":"v1","addr":"172.21.15.184:6801","nonce":4102957285}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6802","nonce":4102957285},{"type":"v1","addr":"172.21.15.184:6803","nonce":4102957285}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6806","nonce":4102957285},{"type":"v1","addr":"172.21.15.184:6807","nonce":4102957285}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6804","nonce":4102957285},{"type":"v1","addr":"172.21.15.184:6805","nonce":4102957285}]},"public_addr":"172.21.15.184:6801/4102957285","cluster_addr":"172.21.15.184:6803/4102957285","heartbeat_back_addr":"172.21.15.184:6807/4102957285","heartbeat_front_addr":"172.21.15.184:6805/4102957285","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:35.972210+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.222007+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.198617+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.385663+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.358935+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.156066+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.752973+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:35.682846+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T15:43:16.728 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:16.726+0000 7f645c8f7640 1 -- 172.21.15.123:0/2671834019 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f6438072ef0 msgr2=0x7f64380753b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:43:16.729 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:16.726+0000 7f645c8f7640 1 --2- 172.21.15.123:0/2671834019 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f6438072ef0 0x7f64380753b0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f644c0058d0 tx=0x7f644c005860 comp rx=0 tx=0).stop 2024-09-16T15:43:16.729 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:16.726+0000 7f645c8f7640 1 -- 172.21.15.123:0/2671834019 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f64581fb9b0 msgr2=0x7f6458154710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:43:16.729 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:16.726+0000 7f645c8f7640 1 --2- 172.21.15.123:0/2671834019 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f64581fb9b0 0x7f6458154710 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f644801d450 tx=0x7f6448006d70 comp rx=0 tx=0).stop 2024-09-16T15:43:16.729 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:16.726+0000 7f645c8f7640 1 -- 172.21.15.123:0/2671834019 shutdown_connections 2024-09-16T15:43:16.729 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:16.726+0000 7f645c8f7640 1 --2- 172.21.15.123:0/2671834019 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f6438072ef0 0x7f64380753b0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:16.729 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:16.726+0000 7f645c8f7640 1 --2- 172.21.15.123:0/2671834019 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6458154c50 0x7f64580c9580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:16.729 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:16.726+0000 7f645c8f7640 1 --2- 172.21.15.123:0/2671834019 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f64581fb9b0 0x7f6458154710 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:16.729 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:16.726+0000 7f645c8f7640 1 --2- 172.21.15.123:0/2671834019 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f6440004090 0x7f64581f7960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:16.729 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:16.726+0000 7f645c8f7640 1 -- 172.21.15.123:0/2671834019 >> 172.21.15.123:0/2671834019 conn(0x7f64580cfb10 msgr2=0x7f6458157a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:43:16.729 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:16.726+0000 7f645c8f7640 1 -- 172.21.15.123:0/2671834019 shutdown_connections 2024-09-16T15:43:16.729 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:16.726+0000 7f645c8f7640 1 -- 172.21.15.123:0/2671834019 wait complete. 2024-09-16T15:43:16.746 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-16T15:42:39.287764+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-09-16T15:42:48.583563+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-09-16T15:42:49.704665+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-09-16T15:43:16.746 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-09-16T15:43:16.874 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:16.870+0000 7f7208853640 1 -- 172.21.15.123:0/165207482 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f72040701e0 msgr2=0x7f720415b570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:43:16.874 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:16.870+0000 7f7208853640 1 --2- 172.21.15.123:0/165207482 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f72040701e0 0x7f720415b570 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f7204073730 tx=0x7f71f801c830 comp rx=0 tx=0).stop 2024-09-16T15:43:16.874 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:16.870+0000 7f7208853640 1 -- 172.21.15.123:0/165207482 shutdown_connections 2024-09-16T15:43:16.874 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:16.870+0000 7f7208853640 1 --2- 172.21.15.123:0/165207482 >> v2:172.21.15.184:3300/0 conn(0x7f72040c31c0 0x7f720415bab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:16.875 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:16.870+0000 7f7208853640 1 --2- 172.21.15.123:0/165207482 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f72040701e0 0x7f720415b570 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:16.875 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:16.870+0000 7f7208853640 1 --2- 172.21.15.123:0/165207482 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f72040cf030 0x7f720406fca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:16.875 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:16.870+0000 7f7208853640 1 -- 172.21.15.123:0/165207482 >> 172.21.15.123:0/165207482 conn(0x7f72040b1710 msgr2=0x7f72040b1b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:43:16.875 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:16.870+0000 7f7208853640 1 -- 172.21.15.123:0/165207482 shutdown_connections 2024-09-16T15:43:16.875 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:16.870+0000 7f7208853640 1 -- 172.21.15.123:0/165207482 wait complete. 2024-09-16T15:43:16.875 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:16.874+0000 7f7208853640 1 Processor -- start 2024-09-16T15:43:16.875 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:16.874+0000 7f7208853640 1 -- start start 2024-09-16T15:43:16.877 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:16.874+0000 7f7208853640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f72040701e0 0x7f72041546f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:43:16.877 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:16.874+0000 7f7208853640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f72040c31c0 0x7f72041f6830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:43:16.877 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:16.874+0000 7f7208853640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f72040cf030 0x7f72040c8550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:43:16.878 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:16.874+0000 7f7208853640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f720415fa40 con 0x7f72040cf030 2024-09-16T15:43:16.878 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:16.874+0000 7f7208853640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f720415f8c0 con 0x7f72040701e0 2024-09-16T15:43:16.878 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:16.874+0000 7f7208853640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f720415fbc0 con 0x7f72040c31c0 2024-09-16T15:43:16.878 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:16.874+0000 7f7202d9d640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f72040cf030 0x7f72040c8550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:43:16.878 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:16.874+0000 7f7202d9d640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f72040cf030 0x7f72040c8550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:51548/0 (socket says 172.21.15.123:51548) 2024-09-16T15:43:16.878 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:16.874+0000 7f7202d9d640 1 -- 172.21.15.123:0/541581290 learned_addr learned my addr 172.21.15.123:0/541581290 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:43:16.878 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:16.874+0000 7f7201d9b640 1 --2- 172.21.15.123:0/541581290 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f72040c31c0 0x7f72041f6830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:43:16.878 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:16.874+0000 7f720259c640 1 --2- 172.21.15.123:0/541581290 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f72040701e0 0x7f72041546f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:43:16.878 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:16.874+0000 7f7202d9d640 1 -- 172.21.15.123:0/541581290 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f72040c31c0 msgr2=0x7f72041f6830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:43:16.878 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:16.874+0000 7f7202d9d640 1 --2- 172.21.15.123:0/541581290 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f72040c31c0 0x7f72041f6830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:16.878 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:16.874+0000 7f7202d9d640 1 -- 172.21.15.123:0/541581290 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f72040701e0 msgr2=0x7f72041546f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:43:16.878 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:16.874+0000 7f7202d9d640 1 --2- 172.21.15.123:0/541581290 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f72040701e0 0x7f72041546f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:16.878 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:16.874+0000 7f7202d9d640 1 -- 172.21.15.123:0/541581290 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7204154c30 con 0x7f72040cf030 2024-09-16T15:43:16.878 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:16.874+0000 7f7201d9b640 1 --2- 172.21.15.123:0/541581290 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f72040c31c0 0x7f72041f6830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T15:43:16.879 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:16.874+0000 7f720259c640 1 --2- 172.21.15.123:0/541581290 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f72040701e0 0x7f72041546f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T15:43:16.879 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:16.874+0000 7f7202d9d640 1 --2- 172.21.15.123:0/541581290 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f72040cf030 0x7f72040c8550 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f71fc00b970 tx=0x7f71fc00be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:43:16.879 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:16.874+0000 7f71f2ffd640 1 -- 172.21.15.123:0/541581290 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71fc019070 con 0x7f72040cf030 2024-09-16T15:43:16.879 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:16.874+0000 7f71f2ffd640 1 -- 172.21.15.123:0/541581290 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f71fc00cc50 con 0x7f72040cf030 2024-09-16T15:43:16.879 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:16.874+0000 7f7208853640 1 -- 172.21.15.123:0/541581290 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f72040c9010 con 0x7f72040cf030 2024-09-16T15:43:16.879 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:16.874+0000 7f71f2ffd640 1 -- 172.21.15.123:0/541581290 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71fc0146a0 con 0x7f72040cf030 2024-09-16T15:43:16.879 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:16.874+0000 7f7208853640 1 -- 172.21.15.123:0/541581290 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f72041f6030 con 0x7f72040cf030 2024-09-16T15:43:16.880 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:16.874+0000 7f7208853640 1 -- 172.21.15.123:0/541581290 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f720406fca0 con 0x7f72040cf030 2024-09-16T15:43:16.880 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:16.878+0000 7f71f2ffd640 1 -- 172.21.15.123:0/541581290 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 5) ==== 97080+0+0 (secure 0 0 0) 0x7f71fc01c070 con 0x7f72040cf030 2024-09-16T15:43:16.880 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:16.878+0000 7f71f2ffd640 1 --2- 172.21.15.123:0/541581290 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f71e4072ec0 0x7f71e4075380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:43:16.880 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:16.878+0000 7f71f2ffd640 1 -- 172.21.15.123:0/541581290 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f71fc093c20 con 0x7f72040cf030 2024-09-16T15:43:16.880 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:16.878+0000 7f720259c640 1 --2- 172.21.15.123:0/541581290 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f71e4072ec0 0x7f71e4075380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:43:16.880 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:16.878+0000 7f720259c640 1 --2- 172.21.15.123:0/541581290 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f71e4072ec0 0x7f71e4075380 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f7204151e40 tx=0x7f71f4008040 comp rx=0 tx=0).ready entity=mgr.4135 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:43:16.884 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:16.882+0000 7f71f2ffd640 1 -- 172.21.15.123:0/541581290 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f71fc05edd0 con 0x7f72040cf030 2024-09-16T15:43:17.049 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:17.046+0000 7f7208853640 1 -- 172.21.15.123:0/541581290 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f7204074630 con 0x7f72040cf030 2024-09-16T15:43:17.050 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:17.046+0000 7f71f2ffd640 1 -- 172.21.15.123:0/541581290 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v21) ==== 93+0+10 (secure 0 0 0) 0x7f71fc05e6d0 con 0x7f72040cf030 2024-09-16T15:43:17.050 INFO:teuthology.orchestra.run.smithi123.stdout:pg_num: 1 2024-09-16T15:43:17.052 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:17.050+0000 7f7208853640 1 -- 172.21.15.123:0/541581290 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f71e4072ec0 msgr2=0x7f71e4075380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:43:17.052 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:17.050+0000 7f7208853640 1 --2- 172.21.15.123:0/541581290 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f71e4072ec0 0x7f71e4075380 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f7204151e40 tx=0x7f71f4008040 comp rx=0 tx=0).stop 2024-09-16T15:43:17.053 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:17.050+0000 7f7208853640 1 -- 172.21.15.123:0/541581290 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f72040cf030 msgr2=0x7f72040c8550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:43:17.053 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:17.050+0000 7f7208853640 1 --2- 172.21.15.123:0/541581290 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f72040cf030 0x7f72040c8550 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f71fc00b970 tx=0x7f71fc00be30 comp rx=0 tx=0).stop 2024-09-16T15:43:17.053 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:17.050+0000 7f7208853640 1 -- 172.21.15.123:0/541581290 shutdown_connections 2024-09-16T15:43:17.053 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:17.050+0000 7f7208853640 1 --2- 172.21.15.123:0/541581290 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f71e4072ec0 0x7f71e4075380 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:17.053 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:17.050+0000 7f7208853640 1 --2- 172.21.15.123:0/541581290 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f72040cf030 0x7f72040c8550 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:17.053 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:17.050+0000 7f7208853640 1 --2- 172.21.15.123:0/541581290 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f72040c31c0 0x7f72041f6830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:17.053 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:17.050+0000 7f7208853640 1 --2- 172.21.15.123:0/541581290 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f72040701e0 0x7f72041546f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:17.053 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:17.050+0000 7f7208853640 1 -- 172.21.15.123:0/541581290 >> 172.21.15.123:0/541581290 conn(0x7f72040b1710 msgr2=0x7f72040737a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:43:17.053 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:17.050+0000 7f7208853640 1 -- 172.21.15.123:0/541581290 shutdown_connections 2024-09-16T15:43:17.053 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:17.050+0000 7f7208853640 1 -- 172.21.15.123:0/541581290 wait complete. 2024-09-16T15:43:17.067 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-09-16T15:43:17.183 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:17.182+0000 7f89cb286640 1 -- 172.21.15.123:0/284008644 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f89c415a170 msgr2=0x7f89c4072f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:43:17.183 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:17.182+0000 7f89cb286640 1 --2- 172.21.15.123:0/284008644 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f89c415a170 0x7f89c4072f30 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f89c0009c80 tx=0x7f89c001cce0 comp rx=0 tx=0).stop 2024-09-16T15:43:17.183 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:17.182+0000 7f89cb286640 1 -- 172.21.15.123:0/284008644 shutdown_connections 2024-09-16T15:43:17.183 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:17.182+0000 7f89cb286640 1 --2- 172.21.15.123:0/284008644 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f89c415a170 0x7f89c4072f30 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:17.183 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:17.182+0000 7f89cb286640 1 -- 172.21.15.123:0/284008644 >> 172.21.15.123:0/284008644 conn(0x7f89c40b1710 msgr2=0x7f89c40b1b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:43:17.183 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:17.182+0000 7f89cb286640 1 -- 172.21.15.123:0/284008644 shutdown_connections 2024-09-16T15:43:17.183 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:17.182+0000 7f89cb286640 1 -- 172.21.15.123:0/284008644 wait complete. 2024-09-16T15:43:17.184 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:17.182+0000 7f89cb286640 1 Processor -- start 2024-09-16T15:43:17.184 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:17.182+0000 7f89cb286640 1 -- start start 2024-09-16T15:43:17.186 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:17.182+0000 7f89cb286640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f89c40ca650 0x7f89c4149850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:43:17.186 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:17.182+0000 7f89cb286640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f89c415a170 0x7f89c40cfad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:43:17.186 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:17.182+0000 7f89cb286640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f89c415ddc0 0x7f89c40d0010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:43:17.186 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:17.182+0000 7f89cb286640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f89c4168b00 con 0x7f89c415ddc0 2024-09-16T15:43:17.186 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:17.182+0000 7f89cb286640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f89c4168980 con 0x7f89c40ca650 2024-09-16T15:43:17.186 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:17.182+0000 7f89cb286640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f89c4168c80 con 0x7f89c415a170 2024-09-16T15:43:17.186 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:17.182+0000 7f89c9823640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f89c415ddc0 0x7f89c40d0010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:43:17.187 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:17.182+0000 7f89c9823640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f89c415ddc0 0x7f89c40d0010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:51566/0 (socket says 172.21.15.123:51566) 2024-09-16T15:43:17.187 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:17.182+0000 7f89c9823640 1 -- 172.21.15.123:0/810288487 learned_addr learned my addr 172.21.15.123:0/810288487 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:43:17.187 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:17.182+0000 7f89c9022640 1 --2- 172.21.15.123:0/810288487 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f89c40ca650 0x7f89c4149850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:43:17.187 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:17.182+0000 7f89c9823640 1 -- 172.21.15.123:0/810288487 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f89c415a170 msgr2=0x7f89c40cfad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:43:17.187 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:17.182+0000 7f89c9823640 1 --2- 172.21.15.123:0/810288487 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f89c415a170 0x7f89c40cfad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:17.187 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:17.182+0000 7f89c9823640 1 -- 172.21.15.123:0/810288487 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f89c40ca650 msgr2=0x7f89c4149850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:43:17.187 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:17.182+0000 7f89c9823640 1 --2- 172.21.15.123:0/810288487 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f89c40ca650 0x7f89c4149850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:17.187 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:17.182+0000 7f89c9823640 1 -- 172.21.15.123:0/810288487 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f89c4149d90 con 0x7f89c415ddc0 2024-09-16T15:43:17.187 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:17.182+0000 7f89c9823640 1 --2- 172.21.15.123:0/810288487 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f89c415ddc0 0x7f89c40d0010 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f89bc00c3c0 tx=0x7f89bc00c7c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:43:17.187 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:17.182+0000 7f89b9ffb640 1 -- 172.21.15.123:0/810288487 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89bc017070 con 0x7f89c415ddc0 2024-09-16T15:43:17.187 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:17.182+0000 7f89b9ffb640 1 -- 172.21.15.123:0/810288487 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f89bc003600 con 0x7f89c415ddc0 2024-09-16T15:43:17.187 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:17.182+0000 7f89cb286640 1 -- 172.21.15.123:0/810288487 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f89c415dac0 con 0x7f89c415ddc0 2024-09-16T15:43:17.188 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:17.182+0000 7f89b9ffb640 1 -- 172.21.15.123:0/810288487 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89bc003a50 con 0x7f89c415ddc0 2024-09-16T15:43:17.188 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:17.182+0000 7f89cb286640 1 -- 172.21.15.123:0/810288487 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f89c40b4e00 con 0x7f89c415ddc0 2024-09-16T15:43:17.188 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:17.186+0000 7f89cb286640 1 -- 172.21.15.123:0/810288487 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f89c4072f30 con 0x7f89c415ddc0 2024-09-16T15:43:17.189 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:17.186+0000 7f89b9ffb640 1 -- 172.21.15.123:0/810288487 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 5) ==== 97080+0+0 (secure 0 0 0) 0x7f89bc01a070 con 0x7f89c415ddc0 2024-09-16T15:43:17.189 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:17.186+0000 7f89b9ffb640 1 --2- 172.21.15.123:0/810288487 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f89a8072ec0 0x7f89a8075380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:43:17.189 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:17.186+0000 7f89b9ffb640 1 -- 172.21.15.123:0/810288487 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f89bc093100 con 0x7f89c415ddc0 2024-09-16T15:43:17.189 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:17.186+0000 7f89c9022640 1 --2- 172.21.15.123:0/810288487 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f89a8072ec0 0x7f89a8075380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:43:17.189 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:17.186+0000 7f89c9022640 1 --2- 172.21.15.123:0/810288487 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f89a8072ec0 0x7f89a8075380 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f89c00af000 tx=0x7f89c001b520 comp rx=0 tx=0).ready entity=mgr.4135 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:43:17.192 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:17.190+0000 7f89b9ffb640 1 -- 172.21.15.123:0/810288487 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f89c4149d90 con 0x7f89c415ddc0 2024-09-16T15:43:17.350 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:17.350+0000 7f89cb286640 1 -- 172.21.15.123:0/810288487 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f89c40b1ec0 con 0x7f89c415ddc0 2024-09-16T15:43:17.351 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:17.350+0000 7f89b9ffb640 1 -- 172.21.15.123:0/810288487 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v21) ==== 104+0+11 (secure 0 0 0) 0x7f89bc05e350 con 0x7f89c415ddc0 2024-09-16T15:43:17.351 INFO:teuthology.orchestra.run.smithi123.stdout:pg_num: 32 2024-09-16T15:43:17.353 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:17.350+0000 7f89cb286640 1 -- 172.21.15.123:0/810288487 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f89a8072ec0 msgr2=0x7f89a8075380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:43:17.354 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:17.350+0000 7f89cb286640 1 --2- 172.21.15.123:0/810288487 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f89a8072ec0 0x7f89a8075380 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f89c00af000 tx=0x7f89c001b520 comp rx=0 tx=0).stop 2024-09-16T15:43:17.354 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:17.350+0000 7f89cb286640 1 -- 172.21.15.123:0/810288487 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f89c415ddc0 msgr2=0x7f89c40d0010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:43:17.354 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:17.350+0000 7f89cb286640 1 --2- 172.21.15.123:0/810288487 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f89c415ddc0 0x7f89c40d0010 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f89bc00c3c0 tx=0x7f89bc00c7c0 comp rx=0 tx=0).stop 2024-09-16T15:43:17.354 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:17.350+0000 7f89cb286640 1 -- 172.21.15.123:0/810288487 shutdown_connections 2024-09-16T15:43:17.354 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:17.350+0000 7f89cb286640 1 --2- 172.21.15.123:0/810288487 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f89a8072ec0 0x7f89a8075380 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:17.354 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:17.350+0000 7f89cb286640 1 --2- 172.21.15.123:0/810288487 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f89c415ddc0 0x7f89c40d0010 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:17.354 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:17.350+0000 7f89cb286640 1 --2- 172.21.15.123:0/810288487 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f89c415a170 0x7f89c40cfad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:17.354 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:17.350+0000 7f89cb286640 1 --2- 172.21.15.123:0/810288487 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f89c40ca650 0x7f89c4149850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:17.354 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:17.350+0000 7f89cb286640 1 -- 172.21.15.123:0/810288487 >> 172.21.15.123:0/810288487 conn(0x7f89c40b1710 msgr2=0x7f89c40cbbd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:43:17.354 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:17.350+0000 7f89cb286640 1 -- 172.21.15.123:0/810288487 shutdown_connections 2024-09-16T15:43:17.354 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:17.350+0000 7f89cb286640 1 -- 172.21.15.123:0/810288487 wait complete. 2024-09-16T15:43:17.369 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-09-16T15:43:17.492 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:17.490+0000 7fdbf0b1c640 1 -- 172.21.15.123:0/1577249734 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fdbec0c3230 msgr2=0x7fdbec0cedd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:43:17.492 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:17.490+0000 7fdbf0b1c640 1 --2- 172.21.15.123:0/1577249734 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fdbec0c3230 0x7fdbec0cedd0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fdbd4009840 tx=0x7fdbd401c820 comp rx=0 tx=0).stop 2024-09-16T15:43:17.493 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:17.490+0000 7fdbf0b1c640 1 -- 172.21.15.123:0/1577249734 shutdown_connections 2024-09-16T15:43:17.493 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:17.490+0000 7fdbf0b1c640 1 --2- 172.21.15.123:0/1577249734 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fdbec0c3230 0x7fdbec0cedd0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:17.493 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:17.490+0000 7fdbf0b1c640 1 --2- 172.21.15.123:0/1577249734 >> v2:172.21.15.184:3300/0 conn(0x7fdbec15b860 0x7fdbec06fca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:17.493 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:17.490+0000 7fdbf0b1c640 1 -- 172.21.15.123:0/1577249734 >> 172.21.15.123:0/1577249734 conn(0x7fdbec0b1710 msgr2=0x7fdbec0b1b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:43:17.493 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:17.490+0000 7fdbf0b1c640 1 -- 172.21.15.123:0/1577249734 shutdown_connections 2024-09-16T15:43:17.493 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:17.490+0000 7fdbf0b1c640 1 -- 172.21.15.123:0/1577249734 wait complete. 2024-09-16T15:43:17.493 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:17.490+0000 7fdbf0b1c640 1 Processor -- start 2024-09-16T15:43:17.494 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:17.490+0000 7fdbf0b1c640 1 -- start start 2024-09-16T15:43:17.495 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:17.490+0000 7fdbf0b1c640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fdbec0701e0 0x7fdbec0c61c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:43:17.495 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:17.494+0000 7fdbf0b1c640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fdbec0c3230 0x7fdbec0cfd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:43:17.495 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:17.494+0000 7fdbf0b1c640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdbec15b860 0x7fdbec148d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:43:17.495 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:17.494+0000 7fdbf0b1c640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fdbec169190 con 0x7fdbec15b860 2024-09-16T15:43:17.496 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:17.494+0000 7fdbf0b1c640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fdbec169010 con 0x7fdbec0c3230 2024-09-16T15:43:17.496 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:17.494+0000 7fdbf0b1c640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fdbec169310 con 0x7fdbec0701e0 2024-09-16T15:43:17.496 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:17.494+0000 7fdbead9d640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdbec15b860 0x7fdbec148d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:43:17.496 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:17.494+0000 7fdbead9d640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdbec15b860 0x7fdbec148d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:51580/0 (socket says 172.21.15.123:51580) 2024-09-16T15:43:17.496 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:17.494+0000 7fdbead9d640 1 -- 172.21.15.123:0/2425458041 learned_addr learned my addr 172.21.15.123:0/2425458041 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:43:17.496 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:17.494+0000 7fdbe1d9b640 1 --2- 172.21.15.123:0/2425458041 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fdbec0c3230 0x7fdbec0cfd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:43:17.496 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:17.494+0000 7fdbea59c640 1 --2- 172.21.15.123:0/2425458041 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fdbec0701e0 0x7fdbec0c61c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:43:17.496 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:17.494+0000 7fdbead9d640 1 -- 172.21.15.123:0/2425458041 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fdbec0701e0 msgr2=0x7fdbec0c61c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:43:17.496 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:17.494+0000 7fdbead9d640 1 --2- 172.21.15.123:0/2425458041 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fdbec0701e0 0x7fdbec0c61c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:17.496 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:17.494+0000 7fdbead9d640 1 -- 172.21.15.123:0/2425458041 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fdbec0c3230 msgr2=0x7fdbec0cfd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:43:17.496 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:17.494+0000 7fdbead9d640 1 --2- 172.21.15.123:0/2425458041 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fdbec0c3230 0x7fdbec0cfd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:17.496 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:17.494+0000 7fdbead9d640 1 -- 172.21.15.123:0/2425458041 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdbec153300 con 0x7fdbec15b860 2024-09-16T15:43:17.496 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:17.494+0000 7fdbea59c640 1 --2- 172.21.15.123:0/2425458041 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fdbec0701e0 0x7fdbec0c61c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T15:43:17.497 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:17.494+0000 7fdbead9d640 1 --2- 172.21.15.123:0/2425458041 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdbec15b860 0x7fdbec148d10 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fdbe400cbc0 tx=0x7fdbe400ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:43:17.497 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:17.494+0000 7fdbe37fe640 1 -- 172.21.15.123:0/2425458041 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdbe4016020 con 0x7fdbec15b860 2024-09-16T15:43:17.497 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:17.494+0000 7fdbf0b1c640 1 -- 172.21.15.123:0/2425458041 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdbec0c5180 con 0x7fdbec15b860 2024-09-16T15:43:17.497 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:17.494+0000 7fdbe37fe640 1 -- 172.21.15.123:0/2425458041 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fdbe400ab00 con 0x7fdbec15b860 2024-09-16T15:43:17.497 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:17.494+0000 7fdbf0b1c640 1 -- 172.21.15.123:0/2425458041 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdbec0c5740 con 0x7fdbec15b860 2024-09-16T15:43:17.497 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:17.494+0000 7fdbe37fe640 1 -- 172.21.15.123:0/2425458041 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdbe4012420 con 0x7fdbec15b860 2024-09-16T15:43:17.498 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:17.494+0000 7fdbf0b1c640 1 -- 172.21.15.123:0/2425458041 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdbec0cedd0 con 0x7fdbec15b860 2024-09-16T15:43:17.498 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:17.494+0000 7fdbe37fe640 1 -- 172.21.15.123:0/2425458041 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 5) ==== 97080+0+0 (secure 0 0 0) 0x7fdbe4019020 con 0x7fdbec15b860 2024-09-16T15:43:17.498 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:17.494+0000 7fdbe37fe640 1 --2- 172.21.15.123:0/2425458041 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fdbcc072ec0 0x7fdbcc075380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:43:17.498 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:17.494+0000 7fdbe37fe640 1 -- 172.21.15.123:0/2425458041 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7fdbe40940d0 con 0x7fdbec15b860 2024-09-16T15:43:17.498 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:17.494+0000 7fdbea59c640 1 --2- 172.21.15.123:0/2425458041 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fdbcc072ec0 0x7fdbcc075380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:43:17.498 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:17.498+0000 7fdbea59c640 1 --2- 172.21.15.123:0/2425458041 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fdbcc072ec0 0x7fdbcc075380 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fdbec06e360 tx=0x7fdbdc0091e0 comp rx=0 tx=0).ready entity=mgr.4135 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:43:17.501 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:17.498+0000 7fdbe37fe640 1 -- 172.21.15.123:0/2425458041 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7fdbe4020020 con 0x7fdbec15b860 2024-09-16T15:43:17.660 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:17.658+0000 7fdbf0b1c640 1 -- 172.21.15.123:0/2425458041 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fdbec073ec0 con 0x7fdbec15b860 2024-09-16T15:43:17.660 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:17.658+0000 7fdbe37fe640 1 -- 172.21.15.123:0/2425458041 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v21) ==== 100+0+11 (secure 0 0 0) 0x7fdbe405f2a0 con 0x7fdbec15b860 2024-09-16T15:43:17.661 INFO:teuthology.orchestra.run.smithi123.stdout:pg_num: 64 2024-09-16T15:43:17.663 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:17.662+0000 7fdbf0b1c640 1 -- 172.21.15.123:0/2425458041 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fdbcc072ec0 msgr2=0x7fdbcc075380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:43:17.663 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:17.662+0000 7fdbf0b1c640 1 --2- 172.21.15.123:0/2425458041 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fdbcc072ec0 0x7fdbcc075380 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fdbec06e360 tx=0x7fdbdc0091e0 comp rx=0 tx=0).stop 2024-09-16T15:43:17.663 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:17.662+0000 7fdbf0b1c640 1 -- 172.21.15.123:0/2425458041 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdbec15b860 msgr2=0x7fdbec148d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:43:17.663 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:17.662+0000 7fdbf0b1c640 1 --2- 172.21.15.123:0/2425458041 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdbec15b860 0x7fdbec148d10 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fdbe400cbc0 tx=0x7fdbe400ccb0 comp rx=0 tx=0).stop 2024-09-16T15:43:17.663 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:17.662+0000 7fdbf0b1c640 1 -- 172.21.15.123:0/2425458041 shutdown_connections 2024-09-16T15:43:17.663 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:17.662+0000 7fdbf0b1c640 1 --2- 172.21.15.123:0/2425458041 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fdbcc072ec0 0x7fdbcc075380 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:17.663 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:17.662+0000 7fdbf0b1c640 1 --2- 172.21.15.123:0/2425458041 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdbec15b860 0x7fdbec148d10 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:17.663 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:17.662+0000 7fdbf0b1c640 1 --2- 172.21.15.123:0/2425458041 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fdbec0c3230 0x7fdbec0cfd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:17.664 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:17.662+0000 7fdbf0b1c640 1 --2- 172.21.15.123:0/2425458041 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fdbec0701e0 0x7fdbec0c61c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:17.664 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:17.662+0000 7fdbf0b1c640 1 -- 172.21.15.123:0/2425458041 >> 172.21.15.123:0/2425458041 conn(0x7fdbec0b1710 msgr2=0x7fdbec0741e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:43:17.664 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:17.662+0000 7fdbf0b1c640 1 -- 172.21.15.123:0/2425458041 shutdown_connections 2024-09-16T15:43:17.664 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:17.662+0000 7fdbf0b1c640 1 -- 172.21.15.123:0/2425458041 wait complete. 2024-09-16T15:43:17.678 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_volumes.TestMisc.test_binary_metafile_on_legacy_to_v1_upgrade' 2024-09-16T15:43:17.807 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:17.806+0000 7fbb46620640 1 -- 172.21.15.123:0/4211369293 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbb4015be80 msgr2=0x7fbb400708c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:43:17.807 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:17.806+0000 7fbb46620640 1 --2- 172.21.15.123:0/4211369293 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbb4015be80 0x7fbb400708c0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fbb30005fd0 tx=0x7fbb3001c820 comp rx=0 tx=0).stop 2024-09-16T15:43:17.808 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:17.806+0000 7fbb46620640 1 -- 172.21.15.123:0/4211369293 shutdown_connections 2024-09-16T15:43:17.808 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:17.806+0000 7fbb46620640 1 --2- 172.21.15.123:0/4211369293 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbb4015be80 0x7fbb400708c0 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:17.808 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:17.806+0000 7fbb46620640 1 -- 172.21.15.123:0/4211369293 >> 172.21.15.123:0/4211369293 conn(0x7fbb4008bfd0 msgr2=0x7fbb4009b150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:43:17.808 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:17.806+0000 7fbb46620640 1 -- 172.21.15.123:0/4211369293 shutdown_connections 2024-09-16T15:43:17.808 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:17.806+0000 7fbb46620640 1 -- 172.21.15.123:0/4211369293 wait complete. 2024-09-16T15:43:17.808 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:17.806+0000 7fbb46620640 1 Processor -- start 2024-09-16T15:43:17.808 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:17.806+0000 7fbb46620640 1 -- start start 2024-09-16T15:43:17.810 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:17.806+0000 7fbb46620640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fbb400d25d0 0x7fbb4015ae60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:43:17.810 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:17.806+0000 7fbb46620640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fbb4015be80 0x7fbb400b1970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:43:17.810 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:17.806+0000 7fbb46620640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbb4006fa00 0x7fbb40154350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:43:17.810 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:17.806+0000 7fbb46620640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fbb40169e40 con 0x7fbb4006fa00 2024-09-16T15:43:17.810 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:17.806+0000 7fbb46620640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fbb40169cc0 con 0x7fbb400d25d0 2024-09-16T15:43:17.811 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:17.806+0000 7fbb46620640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fbb40169fc0 con 0x7fbb4015be80 2024-09-16T15:43:17.811 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:17.806+0000 7fbb44bbd640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbb4006fa00 0x7fbb40154350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:43:17.811 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:17.806+0000 7fbb44bbd640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbb4006fa00 0x7fbb40154350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:51590/0 (socket says 172.21.15.123:51590) 2024-09-16T15:43:17.811 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:17.806+0000 7fbb44bbd640 1 -- 172.21.15.123:0/494559121 learned_addr learned my addr 172.21.15.123:0/494559121 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:43:17.811 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:17.806+0000 7fbb44bbd640 1 -- 172.21.15.123:0/494559121 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fbb4015be80 msgr2=0x7fbb400b1970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:43:17.811 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:17.806+0000 7fbb44bbd640 1 --2- 172.21.15.123:0/494559121 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fbb4015be80 0x7fbb400b1970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:17.811 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:17.806+0000 7fbb44bbd640 1 -- 172.21.15.123:0/494559121 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fbb400d25d0 msgr2=0x7fbb4015ae60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:43:17.811 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:17.806+0000 7fbb3ffff640 1 --2- 172.21.15.123:0/494559121 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fbb400d25d0 0x7fbb4015ae60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:43:17.811 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:17.806+0000 7fbb44bbd640 1 --2- 172.21.15.123:0/494559121 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fbb400d25d0 0x7fbb4015ae60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:17.811 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:17.806+0000 7fbb44bbd640 1 -- 172.21.15.123:0/494559121 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbb40070ad0 con 0x7fbb4006fa00 2024-09-16T15:43:17.811 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:17.806+0000 7fbb44bbd640 1 --2- 172.21.15.123:0/494559121 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbb4006fa00 0x7fbb40154350 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fbb3800caf0 tx=0x7fbb3800cbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:43:17.811 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:17.806+0000 7fbb3d7fa640 1 -- 172.21.15.123:0/494559121 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb38016020 con 0x7fbb4006fa00 2024-09-16T15:43:17.811 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:17.806+0000 7fbb3d7fa640 1 -- 172.21.15.123:0/494559121 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fbb3800aae0 con 0x7fbb4006fa00 2024-09-16T15:43:17.812 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:17.806+0000 7fbb46620640 1 -- 172.21.15.123:0/494559121 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbb40154d50 con 0x7fbb4006fa00 2024-09-16T15:43:17.812 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:17.806+0000 7fbb3d7fa640 1 -- 172.21.15.123:0/494559121 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb3800ce30 con 0x7fbb4006fa00 2024-09-16T15:43:17.812 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:17.806+0000 7fbb46620640 1 -- 172.21.15.123:0/494559121 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbb400b1eb0 con 0x7fbb4006fa00 2024-09-16T15:43:17.812 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:17.810+0000 7fbb46620640 1 -- 172.21.15.123:0/494559121 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbb40154f90 con 0x7fbb4006fa00 2024-09-16T15:43:17.812 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:17.810+0000 7fbb3d7fa640 1 -- 172.21.15.123:0/494559121 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 5) ==== 97080+0+0 (secure 0 0 0) 0x7fbb38019020 con 0x7fbb4006fa00 2024-09-16T15:43:17.813 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:17.810+0000 7fbb3d7fa640 1 --2- 172.21.15.123:0/494559121 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fbb20072ec0 0x7fbb20075380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:43:17.813 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:17.810+0000 7fbb3d7fa640 1 -- 172.21.15.123:0/494559121 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7fbb380940a0 con 0x7fbb4006fa00 2024-09-16T15:43:17.813 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:17.810+0000 7fbb3ffff640 1 --2- 172.21.15.123:0/494559121 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fbb20072ec0 0x7fbb20075380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:43:17.813 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:17.810+0000 7fbb3ffff640 1 --2- 172.21.15.123:0/494559121 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fbb20072ec0 0x7fbb20075380 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fbb300ae000 tx=0x7fbb3001b060 comp rx=0 tx=0).ready entity=mgr.4135 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:43:17.816 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:17.814+0000 7fbb3d7fa640 1 -- 172.21.15.123:0/494559121 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7fbb38020020 con 0x7fbb4006fa00 2024-09-16T15:43:18.053 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:18.050+0000 7fbb46620640 1 -- 172.21.15.123:0/494559121 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_volumes.TestMisc.test_binary_metafile_on_legacy_to_v1_upgrade"]} v 0) -- 0x7fbb40153cd0 con 0x7fbb4006fa00 2024-09-16T15:43:18.336 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:18.334+0000 7fbb3d7fa640 1 -- 172.21.15.123:0/494559121 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_volumes.TestMisc.test_binary_metafile_on_legacy_to_v1_upgrade"]}]=0 v45) ==== 161+0+0 (secure 0 0 0) 0x7fbb3805f2f0 con 0x7fbb4006fa00 2024-09-16T15:43:18.340 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:18.338+0000 7fbb46620640 1 -- 172.21.15.123:0/494559121 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fbb20072ec0 msgr2=0x7fbb20075380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:43:18.340 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:18.338+0000 7fbb46620640 1 --2- 172.21.15.123:0/494559121 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fbb20072ec0 0x7fbb20075380 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fbb300ae000 tx=0x7fbb3001b060 comp rx=0 tx=0).stop 2024-09-16T15:43:18.341 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:18.338+0000 7fbb46620640 1 -- 172.21.15.123:0/494559121 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbb4006fa00 msgr2=0x7fbb40154350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:43:18.341 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:18.338+0000 7fbb46620640 1 --2- 172.21.15.123:0/494559121 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbb4006fa00 0x7fbb40154350 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fbb3800caf0 tx=0x7fbb3800cbe0 comp rx=0 tx=0).stop 2024-09-16T15:43:18.341 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:18.338+0000 7fbb46620640 1 -- 172.21.15.123:0/494559121 shutdown_connections 2024-09-16T15:43:18.341 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:18.338+0000 7fbb46620640 1 --2- 172.21.15.123:0/494559121 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fbb20072ec0 0x7fbb20075380 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:18.341 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:18.338+0000 7fbb46620640 1 --2- 172.21.15.123:0/494559121 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbb4006fa00 0x7fbb40154350 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:18.341 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:18.338+0000 7fbb46620640 1 --2- 172.21.15.123:0/494559121 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fbb4015be80 0x7fbb400b1970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:18.341 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:18.338+0000 7fbb46620640 1 --2- 172.21.15.123:0/494559121 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fbb400d25d0 0x7fbb4015ae60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:18.341 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:18.338+0000 7fbb46620640 1 -- 172.21.15.123:0/494559121 >> 172.21.15.123:0/494559121 conn(0x7fbb4008bfd0 msgr2=0x7fbb401588e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:43:18.341 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:18.338+0000 7fbb46620640 1 -- 172.21.15.123:0/494559121 shutdown_connections 2024-09-16T15:43:18.341 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:18.338+0000 7fbb46620640 1 -- 172.21.15.123:0/494559121 wait complete. 2024-09-16T15:43:18.362 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config dump 2024-09-16T15:43:18.487 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:18.482+0000 7f8f1576e640 1 -- 172.21.15.123:0/2716533680 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8f100cf5a0 msgr2=0x7f8f1006eee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:43:18.487 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:18.482+0000 7f8f1576e640 1 --2- 172.21.15.123:0/2716533680 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8f100cf5a0 0x7f8f1006eee0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f8f10076230 tx=0x7f8ef801c830 comp rx=0 tx=0).stop 2024-09-16T15:43:18.487 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:18.486+0000 7f8f1576e640 1 -- 172.21.15.123:0/2716533680 shutdown_connections 2024-09-16T15:43:18.487 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:18.486+0000 7f8f1576e640 1 --2- 172.21.15.123:0/2716533680 >> v2:172.21.15.184:3300/0 conn(0x7f8f1015ec40 0x7f8f1006f420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:18.487 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:18.486+0000 7f8f1576e640 1 --2- 172.21.15.123:0/2716533680 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8f100cf5a0 0x7f8f1006eee0 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:18.487 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:18.486+0000 7f8f1576e640 1 --2- 172.21.15.123:0/2716533680 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f8f10158120 0x7f8f100ca320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:18.487 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:18.486+0000 7f8f1576e640 1 -- 172.21.15.123:0/2716533680 >> 172.21.15.123:0/2716533680 conn(0x7f8f100cfb10 msgr2=0x7f8f100cff20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:43:18.487 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:18.486+0000 7f8f1576e640 1 -- 172.21.15.123:0/2716533680 shutdown_connections 2024-09-16T15:43:18.487 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:18.486+0000 7f8f1576e640 1 -- 172.21.15.123:0/2716533680 wait complete. 2024-09-16T15:43:18.488 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:18.486+0000 7f8f1576e640 1 Processor -- start 2024-09-16T15:43:18.488 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:18.486+0000 7f8f1576e640 1 -- start start 2024-09-16T15:43:18.489 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:18.486+0000 7f8f1576e640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8f100cf5a0 0x7f8f10155050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:43:18.489 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:18.486+0000 7f8f1576e640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f8f10158120 0x7f8f100b5290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:43:18.489 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:18.486+0000 7f8f1576e640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f8f1015ec40 0x7f8f101f6220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:43:18.489 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:18.486+0000 7f8f1576e640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f8f1006e650 con 0x7f8f100cf5a0 2024-09-16T15:43:18.490 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:18.486+0000 7f8f1576e640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f8f1015b7b0 con 0x7f8f10158120 2024-09-16T15:43:18.490 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:18.486+0000 7f8f1576e640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f8f1006e7d0 con 0x7f8f1015ec40 2024-09-16T15:43:18.490 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:18.486+0000 7f8f0effd640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8f100cf5a0 0x7f8f10155050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:43:18.490 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:18.486+0000 7f8f0effd640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8f100cf5a0 0x7f8f10155050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:51606/0 (socket says 172.21.15.123:51606) 2024-09-16T15:43:18.490 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:18.486+0000 7f8f0effd640 1 -- 172.21.15.123:0/420862695 learned_addr learned my addr 172.21.15.123:0/420862695 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:43:18.490 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:18.486+0000 7f8f0effd640 1 -- 172.21.15.123:0/420862695 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f8f1015ec40 msgr2=0x7f8f101f6220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:43:18.490 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:18.486+0000 7f8f067fc640 1 --2- 172.21.15.123:0/420862695 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f8f10158120 0x7f8f100b5290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:43:18.490 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:18.486+0000 7f8f0f7fe640 1 --2- 172.21.15.123:0/420862695 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f8f1015ec40 0x7f8f101f6220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:43:18.490 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:18.486+0000 7f8f0effd640 1 --2- 172.21.15.123:0/420862695 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f8f1015ec40 0x7f8f101f6220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:18.490 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:18.486+0000 7f8f0effd640 1 -- 172.21.15.123:0/420862695 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f8f10158120 msgr2=0x7f8f100b5290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:43:18.490 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:18.486+0000 7f8f0effd640 1 --2- 172.21.15.123:0/420862695 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f8f10158120 0x7f8f100b5290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:18.490 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:18.486+0000 7f8f0effd640 1 -- 172.21.15.123:0/420862695 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8f101534c0 con 0x7f8f100cf5a0 2024-09-16T15:43:18.490 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:18.486+0000 7f8f0effd640 1 --2- 172.21.15.123:0/420862695 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8f100cf5a0 0x7f8f10155050 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f8f0000c940 tx=0x7f8f0000ce00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:43:18.490 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:18.486+0000 7f8f07fff640 1 -- 172.21.15.123:0/420862695 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f00004220 con 0x7f8f100cf5a0 2024-09-16T15:43:18.491 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:18.486+0000 7f8f07fff640 1 -- 172.21.15.123:0/420862695 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f8f000079b0 con 0x7f8f100cf5a0 2024-09-16T15:43:18.491 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:18.486+0000 7f8f1576e640 1 -- 172.21.15.123:0/420862695 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8f101537a0 con 0x7f8f100cf5a0 2024-09-16T15:43:18.491 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:18.486+0000 7f8f07fff640 1 -- 172.21.15.123:0/420862695 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f00007e00 con 0x7f8f100cf5a0 2024-09-16T15:43:18.491 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:18.486+0000 7f8f1576e640 1 -- 172.21.15.123:0/420862695 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8f10155630 con 0x7f8f100cf5a0 2024-09-16T15:43:18.491 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:18.486+0000 7f8f1576e640 1 -- 172.21.15.123:0/420862695 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8f100ca320 con 0x7f8f100cf5a0 2024-09-16T15:43:18.491 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:18.490+0000 7f8f07fff640 1 -- 172.21.15.123:0/420862695 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 5) ==== 97080+0+0 (secure 0 0 0) 0x7f8f00003600 con 0x7f8f100cf5a0 2024-09-16T15:43:18.492 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:18.490+0000 7f8f07fff640 1 --2- 172.21.15.123:0/420862695 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f8ef0072ef0 0x7f8ef00753b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:43:18.492 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:18.490+0000 7f8f07fff640 1 -- 172.21.15.123:0/420862695 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f8f000942a0 con 0x7f8f100cf5a0 2024-09-16T15:43:18.492 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:18.490+0000 7f8f067fc640 1 --2- 172.21.15.123:0/420862695 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f8ef0072ef0 0x7f8ef00753b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:43:18.492 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:18.490+0000 7f8f067fc640 1 --2- 172.21.15.123:0/420862695 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f8ef0072ef0 0x7f8ef00753b0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f8ef80af000 tx=0x7f8ef801b070 comp rx=0 tx=0).ready entity=mgr.4135 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:43:18.495 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:18.494+0000 7f8f07fff640 1 -- 172.21.15.123:0/420862695 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f8f00014030 con 0x7f8f100cf5a0 2024-09-16T15:43:18.667 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:18.666+0000 7f8f1576e640 1 -- 172.21.15.123:0/420862695 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "config dump"} v 0) -- 0x7f8f100b0610 con 0x7f8f100cf5a0 2024-09-16T15:43:18.667 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:18.666+0000 7f8f07fff640 1 -- 172.21.15.123:0/420862695 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "config dump"}]=0 v27) ==== 59+0+1998 (secure 0 0 0) 0x7f8f0005f4f0 con 0x7f8f100cf5a0 2024-09-16T15:43:18.668 INFO:teuthology.orchestra.run.smithi123.stdout:WHO MASK LEVEL OPTION VALUE RO 2024-09-16T15:43:18.668 INFO:teuthology.orchestra.run.smithi123.stdout:mon advanced mon_op_complaint_time 120 2024-09-16T15:43:18.668 INFO:teuthology.orchestra.run.smithi123.stdout:mon dev mon_warn_on_pool_no_app false 2024-09-16T15:43:18.668 INFO:teuthology.orchestra.run.smithi123.stdout:osd advanced osd_op_complaint_time 180.000000 2024-09-16T15:43:18.668 INFO:teuthology.orchestra.run.smithi123.stdout:osd.0 basic osd_mclock_max_capacity_iops_ssd 4805.300677 2024-09-16T15:43:18.668 INFO:teuthology.orchestra.run.smithi123.stdout:osd.1 basic osd_mclock_max_capacity_iops_ssd 4806.240607 2024-09-16T15:43:18.668 INFO:teuthology.orchestra.run.smithi123.stdout:osd.2 basic osd_mclock_max_capacity_iops_ssd 4046.441686 2024-09-16T15:43:18.668 INFO:teuthology.orchestra.run.smithi123.stdout:osd.3 basic osd_mclock_max_capacity_iops_ssd 6882.576123 2024-09-16T15:43:18.668 INFO:teuthology.orchestra.run.smithi123.stdout:osd.4 basic osd_mclock_max_capacity_iops_ssd 6108.444971 2024-09-16T15:43:18.668 INFO:teuthology.orchestra.run.smithi123.stdout:osd.5 basic osd_mclock_max_capacity_iops_ssd 4995.096306 2024-09-16T15:43:18.669 INFO:teuthology.orchestra.run.smithi123.stdout:osd.6 basic osd_mclock_max_capacity_iops_ssd 6237.822095 2024-09-16T15:43:18.669 INFO:teuthology.orchestra.run.smithi123.stdout:osd.7 basic osd_mclock_max_capacity_iops_ssd 5202.385138 2024-09-16T15:43:18.669 INFO:teuthology.orchestra.run.smithi123.stdout:mds advanced debug_mds 20/20 2024-09-16T15:43:18.669 INFO:teuthology.orchestra.run.smithi123.stdout:mds advanced debug_mds_balancer 20/20 2024-09-16T15:43:18.669 INFO:teuthology.orchestra.run.smithi123.stdout:mds advanced debug_ms 2/2 2024-09-16T15:43:18.669 INFO:teuthology.orchestra.run.smithi123.stdout:mds dev mds_debug_frag true 2024-09-16T15:43:18.669 INFO:teuthology.orchestra.run.smithi123.stdout:mds dev mds_debug_scatterstat true 2024-09-16T15:43:18.669 INFO:teuthology.orchestra.run.smithi123.stdout:mds advanced mds_op_complaint_time 180.000000 2024-09-16T15:43:18.669 INFO:teuthology.orchestra.run.smithi123.stdout:mds dev mds_verify_scatter true 2024-09-16T15:43:18.669 INFO:teuthology.orchestra.run.smithi123.stdout:mds advanced osd_op_complaint_time 180.000000 2024-09-16T15:43:18.669 INFO:teuthology.orchestra.run.smithi123.stdout:mds advanced rados_mon_op_timeout 900 2024-09-16T15:43:18.669 INFO:teuthology.orchestra.run.smithi123.stdout:mds advanced rados_osd_op_timeout 900 2024-09-16T15:43:18.669 INFO:teuthology.orchestra.run.smithi123.stdout:client advanced client_mount_timeout 600 2024-09-16T15:43:18.669 INFO:teuthology.orchestra.run.smithi123.stdout:client advanced debug_client 20/20 2024-09-16T15:43:18.670 INFO:teuthology.orchestra.run.smithi123.stdout:client advanced debug_ms 1/1 2024-09-16T15:43:18.670 INFO:teuthology.orchestra.run.smithi123.stdout:client advanced rados_mon_op_timeout 900 2024-09-16T15:43:18.670 INFO:teuthology.orchestra.run.smithi123.stdout:client advanced rados_osd_op_timeout 900 2024-09-16T15:43:18.670 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:18.666+0000 7f8f1576e640 1 -- 172.21.15.123:0/420862695 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f8ef0072ef0 msgr2=0x7f8ef00753b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:43:18.671 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:18.666+0000 7f8f1576e640 1 --2- 172.21.15.123:0/420862695 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f8ef0072ef0 0x7f8ef00753b0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f8ef80af000 tx=0x7f8ef801b070 comp rx=0 tx=0).stop 2024-09-16T15:43:18.671 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:18.666+0000 7f8f1576e640 1 -- 172.21.15.123:0/420862695 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8f100cf5a0 msgr2=0x7f8f10155050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:43:18.671 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:18.666+0000 7f8f1576e640 1 --2- 172.21.15.123:0/420862695 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8f100cf5a0 0x7f8f10155050 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f8f0000c940 tx=0x7f8f0000ce00 comp rx=0 tx=0).stop 2024-09-16T15:43:18.671 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:18.666+0000 7f8f1576e640 1 -- 172.21.15.123:0/420862695 shutdown_connections 2024-09-16T15:43:18.671 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:18.666+0000 7f8f1576e640 1 --2- 172.21.15.123:0/420862695 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f8ef0072ef0 0x7f8ef00753b0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:18.671 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:18.666+0000 7f8f1576e640 1 --2- 172.21.15.123:0/420862695 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f8f1015ec40 0x7f8f101f6220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:18.671 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:18.666+0000 7f8f1576e640 1 --2- 172.21.15.123:0/420862695 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f8f10158120 0x7f8f100b5290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:18.671 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:18.666+0000 7f8f1576e640 1 --2- 172.21.15.123:0/420862695 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8f100cf5a0 0x7f8f10155050 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:18.671 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:18.666+0000 7f8f1576e640 1 -- 172.21.15.123:0/420862695 >> 172.21.15.123:0/420862695 conn(0x7f8f100cfb10 msgr2=0x7f8f101561a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:43:18.671 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:18.666+0000 7f8f1576e640 1 -- 172.21.15.123:0/420862695 shutdown_connections 2024-09-16T15:43:18.671 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:18.666+0000 7f8f1576e640 1 -- 172.21.15.123:0/420862695 wait complete. 2024-09-16T15:43:18.686 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-09-16T15:43:18.815 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:18.814+0000 7ff5ea0ee640 1 -- 172.21.15.123:0/3640012536 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff5cc003a10 msgr2=0x7ff5cc003e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:43:18.816 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:18.814+0000 7ff5ea0ee640 1 --2- 172.21.15.123:0/3640012536 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff5cc003a10 0x7ff5cc003e30 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7ff5d400c2c0 tx=0x7ff5d401cf50 comp rx=0 tx=0).stop 2024-09-16T15:43:18.816 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:18.814+0000 7ff5e3fff640 1 -- 172.21.15.123:0/3640012536 reap_dead start 2024-09-16T15:43:18.816 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:18.814+0000 7ff5ea0ee640 1 -- 172.21.15.123:0/3640012536 shutdown_connections 2024-09-16T15:43:18.816 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:18.814+0000 7ff5ea0ee640 1 -- 172.21.15.123:0/3640012536 >> 172.21.15.123:0/3640012536 conn(0x7ff5e40d0bc0 msgr2=0x7ff5e40d0fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:43:18.816 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:18.814+0000 7ff5ea0ee640 1 -- 172.21.15.123:0/3640012536 shutdown_connections 2024-09-16T15:43:18.816 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:18.814+0000 7ff5ea0ee640 1 -- 172.21.15.123:0/3640012536 wait complete. 2024-09-16T15:43:18.817 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:18.814+0000 7ff5ea0ee640 1 Processor -- start 2024-09-16T15:43:18.817 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:18.814+0000 7ff5ea0ee640 1 -- start start 2024-09-16T15:43:18.819 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:18.814+0000 7ff5ea0ee640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff5e414e0b0 0x7ff5e40cb940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:43:18.819 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:18.814+0000 7ff5ea0ee640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7ff5e40cbe80 0x7ff5e41f5d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:43:18.819 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:18.814+0000 7ff5ea0ee640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff5e41f6280 0x7ff5e40c4a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:43:18.819 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:18.814+0000 7ff5ea0ee640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7ff5e4165d20 con 0x7ff5e414e0b0 2024-09-16T15:43:18.819 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:18.814+0000 7ff5ea0ee640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7ff5e4165ba0 con 0x7ff5e41f6280 2024-09-16T15:43:18.819 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:18.814+0000 7ff5ea0ee640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7ff5e4165ea0 con 0x7ff5e40cbe80 2024-09-16T15:43:18.819 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:18.814+0000 7ff5e37fe640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff5e414e0b0 0x7ff5e40cb940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:43:18.819 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:18.814+0000 7ff5e37fe640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff5e414e0b0 0x7ff5e40cb940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:51624/0 (socket says 172.21.15.123:51624) 2024-09-16T15:43:18.820 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:18.814+0000 7ff5e37fe640 1 -- 172.21.15.123:0/2079955662 learned_addr learned my addr 172.21.15.123:0/2079955662 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:43:18.820 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:18.814+0000 7ff5e3fff640 1 --2- 172.21.15.123:0/2079955662 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff5e41f6280 0x7ff5e40c4a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:43:18.820 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:18.814+0000 7ff5e2ffd640 1 --2- 172.21.15.123:0/2079955662 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7ff5e40cbe80 0x7ff5e41f5d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:43:18.820 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:18.814+0000 7ff5e37fe640 1 -- 172.21.15.123:0/2079955662 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7ff5e40cbe80 msgr2=0x7ff5e41f5d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:43:18.820 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:18.814+0000 7ff5e37fe640 1 --2- 172.21.15.123:0/2079955662 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7ff5e40cbe80 0x7ff5e41f5d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:18.820 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:18.814+0000 7ff5e37fe640 1 -- 172.21.15.123:0/2079955662 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff5e41f6280 msgr2=0x7ff5e40c4a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:43:18.820 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:18.814+0000 7ff5e37fe640 1 --2- 172.21.15.123:0/2079955662 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff5e41f6280 0x7ff5e40c4a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:18.820 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:18.814+0000 7ff5e37fe640 1 -- 172.21.15.123:0/2079955662 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff5e40c8e40 con 0x7ff5e414e0b0 2024-09-16T15:43:18.820 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:18.818+0000 7ff5e37fe640 1 --2- 172.21.15.123:0/2079955662 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff5e414e0b0 0x7ff5e40cb940 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7ff5d401ca10 tx=0x7ff5d4001cc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:43:18.820 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:18.818+0000 7ff5cbfff640 1 -- 172.21.15.123:0/2079955662 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5d40016c0 con 0x7ff5e414e0b0 2024-09-16T15:43:18.820 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:18.818+0000 7ff5cbfff640 1 -- 172.21.15.123:0/2079955662 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7ff5d40a7e50 con 0x7ff5e414e0b0 2024-09-16T15:43:18.820 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:18.818+0000 7ff5cbfff640 1 -- 172.21.15.123:0/2079955662 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5d40064c0 con 0x7ff5e414e0b0 2024-09-16T15:43:18.821 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:18.818+0000 7ff5ea0ee640 1 -- 172.21.15.123:0/2079955662 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff5e40c9120 con 0x7ff5e414e0b0 2024-09-16T15:43:18.821 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:18.818+0000 7ff5ea0ee640 1 -- 172.21.15.123:0/2079955662 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff5e40c96e0 con 0x7ff5e414e0b0 2024-09-16T15:43:18.822 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:18.818+0000 7ff5ea0ee640 1 -- 172.21.15.123:0/2079955662 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff5e4070990 con 0x7ff5e414e0b0 2024-09-16T15:43:18.822 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:18.818+0000 7ff5cbfff640 1 -- 172.21.15.123:0/2079955662 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 5) ==== 97080+0+0 (secure 0 0 0) 0x7ff5d40036e0 con 0x7ff5e414e0b0 2024-09-16T15:43:18.822 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:18.818+0000 7ff5cbfff640 1 --2- 172.21.15.123:0/2079955662 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7ff5d0072ef0 0x7ff5d00753b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:43:18.822 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:18.818+0000 7ff5cbfff640 1 -- 172.21.15.123:0/2079955662 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7ff5d40b4020 con 0x7ff5e414e0b0 2024-09-16T15:43:18.822 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:18.818+0000 7ff5e2ffd640 1 --2- 172.21.15.123:0/2079955662 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7ff5d0072ef0 0x7ff5d00753b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:43:18.822 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:18.818+0000 7ff5e2ffd640 1 --2- 172.21.15.123:0/2079955662 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7ff5d0072ef0 0x7ff5d00753b0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7ff5e4154740 tx=0x7ff5d800b2c0 comp rx=0 tx=0).ready entity=mgr.4135 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:43:18.825 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:18.822+0000 7ff5cbfff640 1 -- 172.21.15.123:0/2079955662 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7ff5d4016d30 con 0x7ff5e414e0b0 2024-09-16T15:43:18.997 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:18.994+0000 7ff5ea0ee640 1 -- 172.21.15.123:0/2079955662 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) -- 0x7ff5e40746e0 con 0x7ff5e414e0b0 2024-09-16T15:43:19.003 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:19.002+0000 7ff5cbfff640 1 -- 172.21.15.123:0/2079955662 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v28)=0 v28) ==== 122+0+0 (secure 0 0 0) 0x7ff5d40f8d40 con 0x7ff5e414e0b0 2024-09-16T15:43:19.006 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:19.002+0000 7ff5ea0ee640 1 -- 172.21.15.123:0/2079955662 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7ff5d0072ef0 msgr2=0x7ff5d00753b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:43:19.006 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:19.002+0000 7ff5ea0ee640 1 --2- 172.21.15.123:0/2079955662 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7ff5d0072ef0 0x7ff5d00753b0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7ff5e4154740 tx=0x7ff5d800b2c0 comp rx=0 tx=0).stop 2024-09-16T15:43:19.006 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:19.002+0000 7ff5ea0ee640 1 -- 172.21.15.123:0/2079955662 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff5e414e0b0 msgr2=0x7ff5e40cb940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:43:19.007 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:19.002+0000 7ff5ea0ee640 1 --2- 172.21.15.123:0/2079955662 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff5e414e0b0 0x7ff5e40cb940 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7ff5d401ca10 tx=0x7ff5d4001cc0 comp rx=0 tx=0).stop 2024-09-16T15:43:19.007 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:19.002+0000 7ff5ea0ee640 1 -- 172.21.15.123:0/2079955662 shutdown_connections 2024-09-16T15:43:19.007 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:19.002+0000 7ff5ea0ee640 1 --2- 172.21.15.123:0/2079955662 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7ff5d0072ef0 0x7ff5d00753b0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:19.007 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:19.002+0000 7ff5ea0ee640 1 --2- 172.21.15.123:0/2079955662 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff5e41f6280 0x7ff5e40c4a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:19.007 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:19.002+0000 7ff5ea0ee640 1 --2- 172.21.15.123:0/2079955662 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7ff5e40cbe80 0x7ff5e41f5d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:19.007 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:19.002+0000 7ff5ea0ee640 1 --2- 172.21.15.123:0/2079955662 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff5e414e0b0 0x7ff5e40cb940 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:19.007 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:19.002+0000 7ff5ea0ee640 1 -- 172.21.15.123:0/2079955662 >> 172.21.15.123:0/2079955662 conn(0x7ff5e40d0bc0 msgr2=0x7ff5e4164a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:43:19.007 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:19.002+0000 7ff5ea0ee640 1 -- 172.21.15.123:0/2079955662 shutdown_connections 2024-09-16T15:43:19.007 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:19.002+0000 7ff5ea0ee640 1 -- 172.21.15.123:0/2079955662 wait complete. 2024-09-16T15:43:19.023 INFO:teuthology.nuke:Clearing teuthology firewall rules... 2024-09-16T15:43:19.023 DEBUG:teuthology.orchestra.run.smithi123:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-09-16T15:43:19.079 DEBUG:teuthology.orchestra.run.smithi184:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-09-16T15:43:19.152 INFO:teuthology.nuke:Cleared teuthology firewall rules. 2024-09-16T15:43:19.153 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2024-09-16T15:43:19.153 DEBUG:teuthology.orchestra.run.smithi123:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-16T15:43:19.157 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2024-09-16T15:43:19.157 DEBUG:teuthology.orchestra.run.smithi123:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-16T15:43:19.203 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-09-16T15:43:19.372 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:19.370+0000 7fd29a33c640 1 -- 172.21.15.123:0/1882579249 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd2940cdc20 msgr2=0x7fd2940ca860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:43:19.372 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:19.370+0000 7fd29a33c640 1 --2- 172.21.15.123:0/1882579249 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd2940cdc20 0x7fd2940ca860 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fd294070730 tx=0x7fd28c01ca30 comp rx=0 tx=0).stop 2024-09-16T15:43:19.372 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:19.370+0000 7fd29a33c640 1 -- 172.21.15.123:0/1882579249 shutdown_connections 2024-09-16T15:43:19.372 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:19.370+0000 7fd29a33c640 1 --2- 172.21.15.123:0/1882579249 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd2940cdc20 0x7fd2940ca860 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:19.372 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:19.370+0000 7fd29a33c640 1 --2- 172.21.15.123:0/1882579249 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fd294074470 0x7fd2940cd6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:19.372 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:19.370+0000 7fd29a33c640 1 --2- 172.21.15.123:0/1882579249 >> v2:172.21.15.184:3300/0 conn(0x7fd29415e160 0x7fd294073f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:19.372 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:19.370+0000 7fd29a33c640 1 -- 172.21.15.123:0/1882579249 >> 172.21.15.123:0/1882579249 conn(0x7fd2940cfb10 msgr2=0x7fd2940cff20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:43:19.372 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:19.370+0000 7fd29a33c640 1 -- 172.21.15.123:0/1882579249 shutdown_connections 2024-09-16T15:43:19.372 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:19.370+0000 7fd29a33c640 1 -- 172.21.15.123:0/1882579249 wait complete. 2024-09-16T15:43:19.373 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:19.370+0000 7fd29a33c640 1 Processor -- start 2024-09-16T15:43:19.373 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:19.370+0000 7fd29a33c640 1 -- start start 2024-09-16T15:43:19.374 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:19.370+0000 7fd29a33c640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd294074470 0x7fd294153360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:43:19.374 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:19.370+0000 7fd29a33c640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd2940cdc20 0x7fd29415aff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:43:19.374 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:19.370+0000 7fd29a33c640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fd29415e160 0x7fd2941585e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:43:19.375 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:19.370+0000 7fd29a33c640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fd29415ebb0 con 0x7fd294074470 2024-09-16T15:43:19.375 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:19.370+0000 7fd29a33c640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fd29415ea30 con 0x7fd2940cdc20 2024-09-16T15:43:19.375 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:19.370+0000 7fd29a33c640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fd29415ed30 con 0x7fd29415e160 2024-09-16T15:43:19.375 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:19.370+0000 7fd293fff640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd294074470 0x7fd294153360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:43:19.375 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:19.370+0000 7fd293fff640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd294074470 0x7fd294153360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:51634/0 (socket says 172.21.15.123:51634) 2024-09-16T15:43:19.375 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:19.370+0000 7fd293fff640 1 -- 172.21.15.123:0/1522762251 learned_addr learned my addr 172.21.15.123:0/1522762251 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:43:19.375 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:19.370+0000 7fd2937fe640 1 --2- 172.21.15.123:0/1522762251 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd2940cdc20 0x7fd29415aff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:43:19.375 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:19.370+0000 7fd2988d9640 1 --2- 172.21.15.123:0/1522762251 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fd29415e160 0x7fd2941585e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:43:19.375 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:19.370+0000 7fd293fff640 1 -- 172.21.15.123:0/1522762251 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fd29415e160 msgr2=0x7fd2941585e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:43:19.375 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:19.370+0000 7fd293fff640 1 --2- 172.21.15.123:0/1522762251 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fd29415e160 0x7fd2941585e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:19.375 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:19.370+0000 7fd293fff640 1 -- 172.21.15.123:0/1522762251 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd2940cdc20 msgr2=0x7fd29415aff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:43:19.375 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:19.370+0000 7fd293fff640 1 --2- 172.21.15.123:0/1522762251 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd2940cdc20 0x7fd29415aff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:19.375 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:19.370+0000 7fd293fff640 1 -- 172.21.15.123:0/1522762251 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd29415ba90 con 0x7fd294074470 2024-09-16T15:43:19.376 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:19.370+0000 7fd2937fe640 1 --2- 172.21.15.123:0/1522762251 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd2940cdc20 0x7fd29415aff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T15:43:19.376 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:19.370+0000 7fd293fff640 1 --2- 172.21.15.123:0/1522762251 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd294074470 0x7fd294153360 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fd28400b410 tx=0x7fd28400b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:43:19.376 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:19.370+0000 7fd290ff9640 1 -- 172.21.15.123:0/1522762251 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd284015020 con 0x7fd294074470 2024-09-16T15:43:19.376 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:19.370+0000 7fd290ff9640 1 -- 172.21.15.123:0/1522762251 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fd284011920 con 0x7fd294074470 2024-09-16T15:43:19.376 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:19.370+0000 7fd290ff9640 1 -- 172.21.15.123:0/1522762251 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd284011d70 con 0x7fd294074470 2024-09-16T15:43:19.376 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:19.374+0000 7fd29a33c640 1 -- 172.21.15.123:0/1522762251 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd29415bd70 con 0x7fd294074470 2024-09-16T15:43:19.377 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:19.374+0000 7fd29a33c640 1 -- 172.21.15.123:0/1522762251 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd29415c330 con 0x7fd294074470 2024-09-16T15:43:19.377 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:19.374+0000 7fd29a33c640 1 -- 172.21.15.123:0/1522762251 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd2940ca860 con 0x7fd294074470 2024-09-16T15:43:19.377 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:19.374+0000 7fd290ff9640 1 -- 172.21.15.123:0/1522762251 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 5) ==== 97080+0+0 (secure 0 0 0) 0x7fd284018020 con 0x7fd294074470 2024-09-16T15:43:19.377 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:19.374+0000 7fd290ff9640 1 --2- 172.21.15.123:0/1522762251 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fd280072ec0 0x7fd280075380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:43:19.377 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:19.374+0000 7fd290ff9640 1 -- 172.21.15.123:0/1522762251 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7fd284093c20 con 0x7fd294074470 2024-09-16T15:43:19.377 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:19.374+0000 7fd2937fe640 1 --2- 172.21.15.123:0/1522762251 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fd280072ec0 0x7fd280075380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:43:19.378 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:19.374+0000 7fd2937fe640 1 --2- 172.21.15.123:0/1522762251 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fd280072ec0 0x7fd280075380 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fd288005fd0 tx=0x7fd288005d60 comp rx=0 tx=0).ready entity=mgr.4135 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:43:19.381 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:19.378+0000 7fd290ff9640 1 -- 172.21.15.123:0/1522762251 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7fd28405ee70 con 0x7fd294074470 2024-09-16T15:43:19.564 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:19.562+0000 7fd29a33c640 1 -- 172.21.15.123:0/1522762251 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7fd2940d06c0 con 0x7fd294074470 2024-09-16T15:43:19.565 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:19.562+0000 7fd290ff9640 1 -- 172.21.15.123:0/1522762251 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v21) ==== 80+0+0 (secure 0 0 0) 0x7fd28405e770 con 0x7fd294074470 2024-09-16T15:43:19.565 INFO:teuthology.orchestra.run.smithi123.stderr:listed 0 entries 2024-09-16T15:43:19.567 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:19.566+0000 7fd29a33c640 1 -- 172.21.15.123:0/1522762251 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fd280072ec0 msgr2=0x7fd280075380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:43:19.567 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:19.566+0000 7fd29a33c640 1 --2- 172.21.15.123:0/1522762251 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fd280072ec0 0x7fd280075380 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fd288005fd0 tx=0x7fd288005d60 comp rx=0 tx=0).stop 2024-09-16T15:43:19.567 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:19.566+0000 7fd29a33c640 1 -- 172.21.15.123:0/1522762251 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd294074470 msgr2=0x7fd294153360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:43:19.568 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:19.566+0000 7fd29a33c640 1 --2- 172.21.15.123:0/1522762251 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd294074470 0x7fd294153360 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fd28400b410 tx=0x7fd28400b8d0 comp rx=0 tx=0).stop 2024-09-16T15:43:19.568 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:19.566+0000 7fd29a33c640 1 -- 172.21.15.123:0/1522762251 shutdown_connections 2024-09-16T15:43:19.568 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:19.566+0000 7fd29a33c640 1 --2- 172.21.15.123:0/1522762251 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fd280072ec0 0x7fd280075380 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:19.568 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:19.566+0000 7fd29a33c640 1 --2- 172.21.15.123:0/1522762251 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fd29415e160 0x7fd2941585e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:19.568 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:19.566+0000 7fd29a33c640 1 --2- 172.21.15.123:0/1522762251 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd2940cdc20 0x7fd29415aff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:19.568 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:19.566+0000 7fd29a33c640 1 --2- 172.21.15.123:0/1522762251 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd294074470 0x7fd294153360 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:19.568 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:19.566+0000 7fd29a33c640 1 -- 172.21.15.123:0/1522762251 >> 172.21.15.123:0/1522762251 conn(0x7fd2940cfb10 msgr2=0x7fd29406f930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:43:19.568 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:19.566+0000 7fd29a33c640 1 -- 172.21.15.123:0/1522762251 shutdown_connections 2024-09-16T15:43:19.568 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:19.566+0000 7fd29a33c640 1 -- 172.21.15.123:0/1522762251 wait complete. 2024-09-16T15:43:19.584 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2024-09-16T15:43:19.584 DEBUG:teuthology.orchestra.run.smithi123:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-16T15:43:19.590 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-09-16T15:43:19.758 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:19.754+0000 7f46a2631640 1 -- 172.21.15.123:0/2919594007 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f469c15ac00 msgr2=0x7f469c0702f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:43:19.758 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:19.754+0000 7f46a2631640 1 --2- 172.21.15.123:0/2919594007 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f469c15ac00 0x7f469c0702f0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f468c009840 tx=0x7f468c01c820 comp rx=0 tx=0).stop 2024-09-16T15:43:19.758 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:19.754+0000 7f46a2631640 1 -- 172.21.15.123:0/2919594007 shutdown_connections 2024-09-16T15:43:19.758 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:19.754+0000 7f46a2631640 1 --2- 172.21.15.123:0/2919594007 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f469c15ac00 0x7f469c0702f0 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:19.758 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:19.754+0000 7f46a2631640 1 -- 172.21.15.123:0/2919594007 >> 172.21.15.123:0/2919594007 conn(0x7f469c0b2610 msgr2=0x7f469c0b29f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:43:19.759 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:19.754+0000 7f46a2631640 1 -- 172.21.15.123:0/2919594007 shutdown_connections 2024-09-16T15:43:19.759 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:19.754+0000 7f46a2631640 1 -- 172.21.15.123:0/2919594007 wait complete. 2024-09-16T15:43:19.759 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:19.758+0000 7f46a2631640 1 Processor -- start 2024-09-16T15:43:19.759 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:19.758+0000 7f46a2631640 1 -- start start 2024-09-16T15:43:19.761 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:19.758+0000 7f46a2631640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f469c0b3100 0x7f469c1480b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:43:19.761 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:19.758+0000 7f46a2631640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f469c0cc8c0 0x7f469c1577d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:43:19.762 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:19.758+0000 7f46a2631640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f469c15ac00 0x7f469c157d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:43:19.762 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:19.758+0000 7f46a2631640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f469c169720 con 0x7f469c15ac00 2024-09-16T15:43:19.762 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:19.758+0000 7f46a2631640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f469c1695a0 con 0x7f469c0b3100 2024-09-16T15:43:19.762 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:19.758+0000 7f46a2631640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f469c1698a0 con 0x7f469c0cc8c0 2024-09-16T15:43:19.762 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:19.758+0000 7f46a0bce640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f469c15ac00 0x7f469c157d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:43:19.762 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:19.758+0000 7f46a0bce640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f469c15ac00 0x7f469c157d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:51646/0 (socket says 172.21.15.123:51646) 2024-09-16T15:43:19.762 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:19.758+0000 7f46a0bce640 1 -- 172.21.15.123:0/2885029391 learned_addr learned my addr 172.21.15.123:0/2885029391 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:43:19.762 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:19.758+0000 7f469bfff640 1 --2- 172.21.15.123:0/2885029391 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f469c0b3100 0x7f469c1480b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:43:19.762 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:19.758+0000 7f469b7fe640 1 --2- 172.21.15.123:0/2885029391 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f469c0cc8c0 0x7f469c1577d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:43:19.762 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:19.758+0000 7f46a0bce640 1 -- 172.21.15.123:0/2885029391 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f469c0cc8c0 msgr2=0x7f469c1577d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:43:19.762 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:19.758+0000 7f46a0bce640 1 --2- 172.21.15.123:0/2885029391 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f469c0cc8c0 0x7f469c1577d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:19.762 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:19.758+0000 7f46a0bce640 1 -- 172.21.15.123:0/2885029391 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f469c0b3100 msgr2=0x7f469c1480b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:43:19.762 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:19.758+0000 7f46a0bce640 1 --2- 172.21.15.123:0/2885029391 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f469c0b3100 0x7f469c1480b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:19.763 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:19.758+0000 7f46a0bce640 1 -- 172.21.15.123:0/2885029391 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f469c156380 con 0x7f469c15ac00 2024-09-16T15:43:19.763 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:19.758+0000 7f469bfff640 1 --2- 172.21.15.123:0/2885029391 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f469c0b3100 0x7f469c1480b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T15:43:19.763 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:19.758+0000 7f46a0bce640 1 --2- 172.21.15.123:0/2885029391 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f469c15ac00 0x7f469c157d10 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f469400cbf0 tx=0x7f469400cce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:43:19.763 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:19.758+0000 7f4698ff9640 1 -- 172.21.15.123:0/2885029391 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4694016020 con 0x7f469c15ac00 2024-09-16T15:43:19.763 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:19.758+0000 7f4698ff9640 1 -- 172.21.15.123:0/2885029391 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f469400ab40 con 0x7f469c15ac00 2024-09-16T15:43:19.763 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:19.758+0000 7f46a2631640 1 -- 172.21.15.123:0/2885029391 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f469c156660 con 0x7f469c15ac00 2024-09-16T15:43:19.763 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:19.758+0000 7f4698ff9640 1 -- 172.21.15.123:0/2885029391 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4694012420 con 0x7f469c15ac00 2024-09-16T15:43:19.763 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:19.758+0000 7f46a2631640 1 -- 172.21.15.123:0/2885029391 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f469c156c20 con 0x7f469c15ac00 2024-09-16T15:43:19.763 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:19.758+0000 7f46a2631640 1 -- 172.21.15.123:0/2885029391 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f469c0702f0 con 0x7f469c15ac00 2024-09-16T15:43:19.764 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:19.762+0000 7f4698ff9640 1 -- 172.21.15.123:0/2885029391 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 5) ==== 97080+0+0 (secure 0 0 0) 0x7f4694019020 con 0x7f469c15ac00 2024-09-16T15:43:19.764 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:19.762+0000 7f4698ff9640 1 --2- 172.21.15.123:0/2885029391 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f4688072ec0 0x7f4688075380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:43:19.764 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:19.762+0000 7f4698ff9640 1 -- 172.21.15.123:0/2885029391 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f4694094140 con 0x7f469c15ac00 2024-09-16T15:43:19.764 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:19.762+0000 7f469bfff640 1 --2- 172.21.15.123:0/2885029391 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f4688072ec0 0x7f4688075380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:43:19.764 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:19.762+0000 7f469bfff640 1 --2- 172.21.15.123:0/2885029391 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f4688072ec0 0x7f4688075380 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f468c0af000 tx=0x7f468c01b060 comp rx=0 tx=0).ready entity=mgr.4135 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:43:19.767 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:19.766+0000 7f4698ff9640 1 -- 172.21.15.123:0/2885029391 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f4694020020 con 0x7f469c15ac00 2024-09-16T15:43:19.950 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:19.945+0000 7f46a2631640 1 -- 172.21.15.123:0/2885029391 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f469c0b6aa0 con 0x7f469c15ac00 2024-09-16T15:43:19.951 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:19.949+0000 7f4698ff9640 1 -- 172.21.15.123:0/2885029391 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v21) ==== 80+0+0 (secure 0 0 0) 0x7f469405f330 con 0x7f469c15ac00 2024-09-16T15:43:19.951 INFO:teuthology.orchestra.run.smithi123.stderr:listed 0 entries 2024-09-16T15:43:19.953 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:19.949+0000 7f46a2631640 1 -- 172.21.15.123:0/2885029391 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f4688072ec0 msgr2=0x7f4688075380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:43:19.954 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:19.949+0000 7f46a2631640 1 --2- 172.21.15.123:0/2885029391 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f4688072ec0 0x7f4688075380 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f468c0af000 tx=0x7f468c01b060 comp rx=0 tx=0).stop 2024-09-16T15:43:19.954 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:19.949+0000 7f46a2631640 1 -- 172.21.15.123:0/2885029391 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f469c15ac00 msgr2=0x7f469c157d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:43:19.954 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:19.949+0000 7f46a2631640 1 --2- 172.21.15.123:0/2885029391 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f469c15ac00 0x7f469c157d10 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f469400cbf0 tx=0x7f469400cce0 comp rx=0 tx=0).stop 2024-09-16T15:43:19.954 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:19.949+0000 7f46a2631640 1 -- 172.21.15.123:0/2885029391 shutdown_connections 2024-09-16T15:43:19.954 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:19.949+0000 7f46a2631640 1 --2- 172.21.15.123:0/2885029391 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f4688072ec0 0x7f4688075380 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:19.954 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:19.949+0000 7f46a2631640 1 --2- 172.21.15.123:0/2885029391 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f469c15ac00 0x7f469c157d10 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:19.954 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:19.949+0000 7f46a2631640 1 --2- 172.21.15.123:0/2885029391 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f469c0cc8c0 0x7f469c1577d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:19.954 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:19.949+0000 7f46a2631640 1 --2- 172.21.15.123:0/2885029391 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f469c0b3100 0x7f469c1480b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:19.954 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:19.949+0000 7f46a2631640 1 -- 172.21.15.123:0/2885029391 >> 172.21.15.123:0/2885029391 conn(0x7f469c0b2610 msgr2=0x7f469c0758d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:43:19.954 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:19.949+0000 7f46a2631640 1 -- 172.21.15.123:0/2885029391 shutdown_connections 2024-09-16T15:43:19.954 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:19.949+0000 7f46a2631640 1 -- 172.21.15.123:0/2885029391 wait complete. 2024-09-16T15:43:19.970 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi123.front.sepia.ceph.com... 2024-09-16T15:43:19.970 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:43:19.971 DEBUG:teuthology.orchestra.run.smithi123:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2024-09-16T15:43:19.999 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T15:43:19.999 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi123.front.sepia.ceph.com 2024-09-16T15:43:20.000 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:43:20.000 DEBUG:teuthology.orchestra.run.smithi123:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-09-16T15:43:20.054 DEBUG:teuthology.orchestra.run.smithi184:> set -ex 2024-09-16T15:43:20.054 DEBUG:teuthology.orchestra.run.smithi184:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-16T15:43:20.061 DEBUG:teuthology.orchestra.run.smithi184:> set -ex 2024-09-16T15:43:20.061 DEBUG:teuthology.orchestra.run.smithi184:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-16T15:43:20.108 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-09-16T15:43:20.241 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:20.237+0000 7f0241d05640 1 -- 172.21.15.123:0/3212424838 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f023c0ca6a0 msgr2=0x7f023c1506d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:43:20.241 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:20.237+0000 7f0241d05640 1 --2- 172.21.15.123:0/3212424838 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f023c0ca6a0 0x7f023c1506d0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f022c00a910 tx=0x7f022c01c760 comp rx=0 tx=0).stop 2024-09-16T15:43:20.241 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:20.237+0000 7f0241d05640 1 -- 172.21.15.123:0/3212424838 shutdown_connections 2024-09-16T15:43:20.241 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:20.237+0000 7f0241d05640 1 --2- 172.21.15.123:0/3212424838 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f023c0ca6a0 0x7f023c1506d0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:20.241 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:20.237+0000 7f0241d05640 1 --2- 172.21.15.123:0/3212424838 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f023c0cf3f0 0x7f023c0ca160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:20.241 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:20.237+0000 7f0241d05640 1 -- 172.21.15.123:0/3212424838 >> 172.21.15.123:0/3212424838 conn(0x7f023c0cfb10 msgr2=0x7f023c0cff20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:43:20.241 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:20.237+0000 7f0241d05640 1 -- 172.21.15.123:0/3212424838 shutdown_connections 2024-09-16T15:43:20.242 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:20.237+0000 7f0241d05640 1 -- 172.21.15.123:0/3212424838 wait complete. 2024-09-16T15:43:20.242 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:20.237+0000 7f0241d05640 1 Processor -- start 2024-09-16T15:43:20.242 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:20.241+0000 7f0241d05640 1 -- start start 2024-09-16T15:43:20.245 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:20.241+0000 7f0241d05640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f023c06fc00 0x7f023c14a570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:43:20.245 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:20.241+0000 7f0241d05640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f023c0ca6a0 0x7f023c156240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:43:20.245 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:20.241+0000 7f0241d05640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f023c0cf3f0 0x7f023c156780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:43:20.245 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:20.241+0000 7f0241d05640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f023c06ea60 con 0x7f023c0cf3f0 2024-09-16T15:43:20.245 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:20.241+0000 7f0241d05640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f023c06e8e0 con 0x7f023c06fc00 2024-09-16T15:43:20.245 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:20.241+0000 7f0241d05640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f023c06ebe0 con 0x7f023c0ca6a0 2024-09-16T15:43:20.245 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:20.241+0000 7f023bfff640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f023c0cf3f0 0x7f023c156780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:43:20.245 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:20.241+0000 7f023bfff640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f023c0cf3f0 0x7f023c156780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:51650/0 (socket says 172.21.15.123:51650) 2024-09-16T15:43:20.245 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:20.241+0000 7f023bfff640 1 -- 172.21.15.123:0/1550260149 learned_addr learned my addr 172.21.15.123:0/1550260149 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:43:20.245 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:20.241+0000 7f023b7fe640 1 --2- 172.21.15.123:0/1550260149 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f023c06fc00 0x7f023c14a570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:43:20.245 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:20.241+0000 7f023affd640 1 --2- 172.21.15.123:0/1550260149 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f023c0ca6a0 0x7f023c156240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:43:20.245 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:20.241+0000 7f023bfff640 1 -- 172.21.15.123:0/1550260149 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f023c0ca6a0 msgr2=0x7f023c156240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:43:20.246 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:20.241+0000 7f023bfff640 1 --2- 172.21.15.123:0/1550260149 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f023c0ca6a0 0x7f023c156240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:20.246 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:20.241+0000 7f023bfff640 1 -- 172.21.15.123:0/1550260149 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f023c06fc00 msgr2=0x7f023c14a570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:43:20.246 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:20.241+0000 7f023bfff640 1 --2- 172.21.15.123:0/1550260149 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f023c06fc00 0x7f023c14a570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:20.246 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:20.241+0000 7f023bfff640 1 -- 172.21.15.123:0/1550260149 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f023c165a60 con 0x7f023c0cf3f0 2024-09-16T15:43:20.246 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:20.241+0000 7f023bfff640 1 --2- 172.21.15.123:0/1550260149 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f023c0cf3f0 0x7f023c156780 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f022c01a390 tx=0x7f022c005780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:43:20.246 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:20.241+0000 7f0233fff640 1 -- 172.21.15.123:0/1550260149 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f022c0b2040 con 0x7f023c0cf3f0 2024-09-16T15:43:20.246 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:20.241+0000 7f0233fff640 1 -- 172.21.15.123:0/1550260149 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f022c0bb070 con 0x7f023c0cf3f0 2024-09-16T15:43:20.246 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:20.241+0000 7f0241d05640 1 -- 172.21.15.123:0/1550260149 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f023c0b4c60 con 0x7f023c0cf3f0 2024-09-16T15:43:20.246 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:20.241+0000 7f0233fff640 1 -- 172.21.15.123:0/1550260149 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f022c0042a0 con 0x7f023c0cf3f0 2024-09-16T15:43:20.246 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:20.241+0000 7f0241d05640 1 -- 172.21.15.123:0/1550260149 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f023c0b4f40 con 0x7f023c0cf3f0 2024-09-16T15:43:20.247 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:20.241+0000 7f0241d05640 1 -- 172.21.15.123:0/1550260149 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f023c1506d0 con 0x7f023c0cf3f0 2024-09-16T15:43:20.247 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:20.241+0000 7f0233fff640 1 -- 172.21.15.123:0/1550260149 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 5) ==== 97080+0+0 (secure 0 0 0) 0x7f022c0a5640 con 0x7f023c0cf3f0 2024-09-16T15:43:20.247 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:20.245+0000 7f0233fff640 1 --2- 172.21.15.123:0/1550260149 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f021c072ec0 0x7f021c075380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:43:20.247 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:20.245+0000 7f0233fff640 1 -- 172.21.15.123:0/1550260149 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f022c12e110 con 0x7f023c0cf3f0 2024-09-16T15:43:20.247 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:20.245+0000 7f023b7fe640 1 --2- 172.21.15.123:0/1550260149 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f021c072ec0 0x7f021c075380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:43:20.248 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:20.245+0000 7f023b7fe640 1 --2- 172.21.15.123:0/1550260149 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f021c072ec0 0x7f021c075380 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f0234001ee0 tx=0x7f0234009040 comp rx=0 tx=0).ready entity=mgr.4135 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:43:20.251 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:20.249+0000 7f0233fff640 1 -- 172.21.15.123:0/1550260149 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f022c016350 con 0x7f023c0cf3f0 2024-09-16T15:43:20.439 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:20.437+0000 7f0241d05640 1 -- 172.21.15.123:0/1550260149 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f023c0cd6e0 con 0x7f023c0cf3f0 2024-09-16T15:43:20.440 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:20.437+0000 7f0233fff640 1 -- 172.21.15.123:0/1550260149 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v21) ==== 80+0+0 (secure 0 0 0) 0x7f022c0f9360 con 0x7f023c0cf3f0 2024-09-16T15:43:20.440 INFO:teuthology.orchestra.run.smithi123.stderr:listed 0 entries 2024-09-16T15:43:20.442 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:20.437+0000 7f0241d05640 1 -- 172.21.15.123:0/1550260149 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f021c072ec0 msgr2=0x7f021c075380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:43:20.442 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:20.437+0000 7f0241d05640 1 --2- 172.21.15.123:0/1550260149 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f021c072ec0 0x7f021c075380 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f0234001ee0 tx=0x7f0234009040 comp rx=0 tx=0).stop 2024-09-16T15:43:20.443 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:20.437+0000 7f0241d05640 1 -- 172.21.15.123:0/1550260149 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f023c0cf3f0 msgr2=0x7f023c156780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:43:20.443 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:20.437+0000 7f0241d05640 1 --2- 172.21.15.123:0/1550260149 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f023c0cf3f0 0x7f023c156780 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f022c01a390 tx=0x7f022c005780 comp rx=0 tx=0).stop 2024-09-16T15:43:20.443 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:20.437+0000 7f0241d05640 1 -- 172.21.15.123:0/1550260149 shutdown_connections 2024-09-16T15:43:20.443 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:20.437+0000 7f0241d05640 1 --2- 172.21.15.123:0/1550260149 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f021c072ec0 0x7f021c075380 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:20.443 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:20.437+0000 7f0241d05640 1 --2- 172.21.15.123:0/1550260149 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f023c0cf3f0 0x7f023c156780 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:20.443 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:20.437+0000 7f0241d05640 1 --2- 172.21.15.123:0/1550260149 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f023c0ca6a0 0x7f023c156240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:20.443 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:20.437+0000 7f0241d05640 1 --2- 172.21.15.123:0/1550260149 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f023c06fc00 0x7f023c14a570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:20.443 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:20.437+0000 7f0241d05640 1 -- 172.21.15.123:0/1550260149 >> 172.21.15.123:0/1550260149 conn(0x7f023c0cfb10 msgr2=0x7f023c0b2be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:43:20.443 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:20.437+0000 7f0241d05640 1 -- 172.21.15.123:0/1550260149 shutdown_connections 2024-09-16T15:43:20.443 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:20.437+0000 7f0241d05640 1 -- 172.21.15.123:0/1550260149 wait complete. 2024-09-16T15:43:20.459 DEBUG:teuthology.orchestra.run.smithi184:> set -ex 2024-09-16T15:43:20.459 DEBUG:teuthology.orchestra.run.smithi184:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-16T15:43:20.466 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-09-16T15:43:20.589 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:20.585+0000 7faaec490640 1 -- 172.21.15.123:0/1538997475 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7faae40726c0 msgr2=0x7faae4073ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:43:20.589 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:20.585+0000 7faaec490640 1 --2- 172.21.15.123:0/1538997475 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7faae40726c0 0x7faae4073ef0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7faae40cd650 tx=0x7faad401c800 comp rx=0 tx=0).stop 2024-09-16T15:43:20.589 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:20.585+0000 7faaec490640 1 -- 172.21.15.123:0/1538997475 shutdown_connections 2024-09-16T15:43:20.589 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:20.585+0000 7faaec490640 1 --2- 172.21.15.123:0/1538997475 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7faae40726c0 0x7faae4073ef0 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:20.589 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:20.585+0000 7faaec490640 1 --2- 172.21.15.123:0/1538997475 >> v2:172.21.15.184:3300/0 conn(0x7faae415bca0 0x7faae40d0630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:20.589 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:20.585+0000 7faaec490640 1 -- 172.21.15.123:0/1538997475 >> 172.21.15.123:0/1538997475 conn(0x7faae40cfb10 msgr2=0x7faae40cff20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:43:20.589 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:20.585+0000 7faaec490640 1 -- 172.21.15.123:0/1538997475 shutdown_connections 2024-09-16T15:43:20.590 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:20.585+0000 7faaec490640 1 -- 172.21.15.123:0/1538997475 wait complete. 2024-09-16T15:43:20.590 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:20.585+0000 7faaec490640 1 Processor -- start 2024-09-16T15:43:20.590 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:20.585+0000 7faaec490640 1 -- start start 2024-09-16T15:43:20.593 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:20.589+0000 7faaec490640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7faae40726c0 0x7faae4151d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:43:20.593 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:20.589+0000 7faaec490640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7faae4074430 0x7faae40b38a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:43:20.593 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:20.589+0000 7faaec490640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7faae415bca0 0x7faae40b3de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:43:20.593 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:20.589+0000 7faaec490640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7faae4168ef0 con 0x7faae415bca0 2024-09-16T15:43:20.593 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:20.589+0000 7faaec490640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7faae4168d70 con 0x7faae40726c0 2024-09-16T15:43:20.593 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:20.589+0000 7faaec490640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7faae4169070 con 0x7faae4074430 2024-09-16T15:43:20.593 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:20.589+0000 7faaeaa2d640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7faae415bca0 0x7faae40b3de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:43:20.593 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:20.589+0000 7faaeaa2d640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7faae415bca0 0x7faae40b3de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:51672/0 (socket says 172.21.15.123:51672) 2024-09-16T15:43:20.593 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:20.589+0000 7faaeaa2d640 1 -- 172.21.15.123:0/1429108156 learned_addr learned my addr 172.21.15.123:0/1429108156 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:43:20.594 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:20.589+0000 7faaea22c640 1 --2- 172.21.15.123:0/1429108156 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7faae40726c0 0x7faae4151d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:43:20.594 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:20.589+0000 7faae9a2b640 1 --2- 172.21.15.123:0/1429108156 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7faae4074430 0x7faae40b38a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:43:20.594 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:20.589+0000 7faaeaa2d640 1 -- 172.21.15.123:0/1429108156 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7faae4074430 msgr2=0x7faae40b38a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:43:20.594 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:20.589+0000 7faaeaa2d640 1 --2- 172.21.15.123:0/1429108156 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7faae4074430 0x7faae40b38a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:20.594 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:20.589+0000 7faaeaa2d640 1 -- 172.21.15.123:0/1429108156 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7faae40726c0 msgr2=0x7faae4151d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:43:20.594 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:20.589+0000 7faaeaa2d640 1 --2- 172.21.15.123:0/1429108156 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7faae40726c0 0x7faae4151d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:20.594 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:20.589+0000 7faaeaa2d640 1 -- 172.21.15.123:0/1429108156 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faae41536c0 con 0x7faae415bca0 2024-09-16T15:43:20.594 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:20.589+0000 7faaea22c640 1 --2- 172.21.15.123:0/1429108156 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7faae40726c0 0x7faae4151d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T15:43:20.594 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:20.589+0000 7faaeaa2d640 1 --2- 172.21.15.123:0/1429108156 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7faae415bca0 0x7faae40b3de0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7faadc00c7a0 tx=0x7faadc00cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:43:20.594 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:20.589+0000 7faadaffd640 1 -- 172.21.15.123:0/1429108156 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faadc003b80 con 0x7faae415bca0 2024-09-16T15:43:20.594 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:20.589+0000 7faadaffd640 1 -- 172.21.15.123:0/1429108156 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7faadc003d40 con 0x7faae415bca0 2024-09-16T15:43:20.594 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:20.589+0000 7faadaffd640 1 -- 172.21.15.123:0/1429108156 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faadc00aab0 con 0x7faae415bca0 2024-09-16T15:43:20.594 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:20.589+0000 7faaec490640 1 -- 172.21.15.123:0/1429108156 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faae40c9470 con 0x7faae415bca0 2024-09-16T15:43:20.595 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:20.589+0000 7faaec490640 1 -- 172.21.15.123:0/1429108156 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faae40c9a30 con 0x7faae415bca0 2024-09-16T15:43:20.595 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:20.589+0000 7faaec490640 1 -- 172.21.15.123:0/1429108156 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faae4073ef0 con 0x7faae415bca0 2024-09-16T15:43:20.595 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:20.593+0000 7faadaffd640 1 -- 172.21.15.123:0/1429108156 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 5) ==== 97080+0+0 (secure 0 0 0) 0x7faadc00ac50 con 0x7faae415bca0 2024-09-16T15:43:20.596 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:20.593+0000 7faadaffd640 1 --2- 172.21.15.123:0/1429108156 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7faac8072ec0 0x7faac8075380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:43:20.596 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:20.593+0000 7faadaffd640 1 -- 172.21.15.123:0/1429108156 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7faadc0938a0 con 0x7faae415bca0 2024-09-16T15:43:20.596 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:20.593+0000 7faaea22c640 1 --2- 172.21.15.123:0/1429108156 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7faac8072ec0 0x7faac8075380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:43:20.596 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:20.593+0000 7faaea22c640 1 --2- 172.21.15.123:0/1429108156 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7faac8072ec0 0x7faac8075380 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7faae00058d0 tx=0x7faae0005ca0 comp rx=0 tx=0).ready entity=mgr.4135 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:43:20.599 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:20.597+0000 7faadaffd640 1 -- 172.21.15.123:0/1429108156 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7faadc020060 con 0x7faae415bca0 2024-09-16T15:43:20.768 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:20.765+0000 7faaec490640 1 -- 172.21.15.123:0/1429108156 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7faae40700e0 con 0x7faae415bca0 2024-09-16T15:43:20.768 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:20.765+0000 7faadaffd640 1 -- 172.21.15.123:0/1429108156 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v21) ==== 80+0+0 (secure 0 0 0) 0x7faae41536c0 con 0x7faae415bca0 2024-09-16T15:43:20.768 INFO:teuthology.orchestra.run.smithi123.stderr:listed 0 entries 2024-09-16T15:43:20.771 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:20.765+0000 7faaec490640 1 -- 172.21.15.123:0/1429108156 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7faac8072ec0 msgr2=0x7faac8075380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:43:20.771 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:20.765+0000 7faaec490640 1 --2- 172.21.15.123:0/1429108156 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7faac8072ec0 0x7faac8075380 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7faae00058d0 tx=0x7faae0005ca0 comp rx=0 tx=0).stop 2024-09-16T15:43:20.771 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:20.765+0000 7faaec490640 1 -- 172.21.15.123:0/1429108156 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7faae415bca0 msgr2=0x7faae40b3de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:43:20.771 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:20.765+0000 7faaec490640 1 --2- 172.21.15.123:0/1429108156 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7faae415bca0 0x7faae40b3de0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7faadc00c7a0 tx=0x7faadc00cc60 comp rx=0 tx=0).stop 2024-09-16T15:43:20.771 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:20.769+0000 7faaec490640 1 -- 172.21.15.123:0/1429108156 shutdown_connections 2024-09-16T15:43:20.771 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:20.769+0000 7faaec490640 1 --2- 172.21.15.123:0/1429108156 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7faac8072ec0 0x7faac8075380 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:20.771 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:20.769+0000 7faaec490640 1 --2- 172.21.15.123:0/1429108156 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7faae415bca0 0x7faae40b3de0 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:20.771 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:20.769+0000 7faaec490640 1 --2- 172.21.15.123:0/1429108156 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7faae4074430 0x7faae40b38a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:20.771 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:20.769+0000 7faaec490640 1 --2- 172.21.15.123:0/1429108156 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7faae40726c0 0x7faae4151d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:20.772 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:20.769+0000 7faaec490640 1 -- 172.21.15.123:0/1429108156 >> 172.21.15.123:0/1429108156 conn(0x7faae40cfb10 msgr2=0x7faae40b4640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:43:20.772 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:20.769+0000 7faaec490640 1 -- 172.21.15.123:0/1429108156 shutdown_connections 2024-09-16T15:43:20.772 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:20.769+0000 7faaec490640 1 -- 172.21.15.123:0/1429108156 wait complete. 2024-09-16T15:43:20.786 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi184.front.sepia.ceph.com... 2024-09-16T15:43:20.786 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:43:20.786 DEBUG:teuthology.orchestra.run.smithi184:> sudo fusermount -u /home/ubuntu/cephtest/mnt.1 2024-09-16T15:43:20.809 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T15:43:20.810 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi184.front.sepia.ceph.com 2024-09-16T15:43:20.810 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:43:20.810 DEBUG:teuthology.orchestra.run.smithi184:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2024-09-16T15:43:20.859 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2024-09-16T15:43:20.859 DEBUG:teuthology.orchestra.run.smithi123:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-16T15:43:20.865 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2024-09-16T15:43:20.865 DEBUG:teuthology.orchestra.run.smithi123:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-16T15:43:20.911 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-09-16T15:43:21.286 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:21.281+0000 7fe2702fc640 1 -- 172.21.15.123:0/2540719669 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe2680d2820 msgr2=0x7fe2680cea00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:43:21.286 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:21.281+0000 7fe2702fc640 1 --2- 172.21.15.123:0/2540719669 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe2680d2820 0x7fe2680cea00 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fe26808ce10 tx=0x7fe25801c770 comp rx=0 tx=0).stop 2024-09-16T15:43:21.286 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:21.281+0000 7fe2702fc640 1 -- 172.21.15.123:0/2540719669 shutdown_connections 2024-09-16T15:43:21.286 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:21.281+0000 7fe2702fc640 1 --2- 172.21.15.123:0/2540719669 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe2680d2820 0x7fe2680cea00 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:21.286 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:21.281+0000 7fe2702fc640 1 --2- 172.21.15.123:0/2540719669 >> v2:172.21.15.184:3300/0 conn(0x7fe2680b2b00 0x7fe2680d22e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:21.286 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:21.281+0000 7fe2702fc640 1 --2- 172.21.15.123:0/2540719669 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fe268161620 0x7fe2680d1da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:21.286 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:21.281+0000 7fe2702fc640 1 -- 172.21.15.123:0/2540719669 >> 172.21.15.123:0/2540719669 conn(0x7fe2680d3ea0 msgr2=0x7fe2680d42b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:43:21.286 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:21.281+0000 7fe2702fc640 1 -- 172.21.15.123:0/2540719669 shutdown_connections 2024-09-16T15:43:21.286 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:21.281+0000 7fe2702fc640 1 -- 172.21.15.123:0/2540719669 wait complete. 2024-09-16T15:43:21.287 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:21.285+0000 7fe2702fc640 1 Processor -- start 2024-09-16T15:43:21.287 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:21.285+0000 7fe2702fc640 1 -- start start 2024-09-16T15:43:21.288 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:21.285+0000 7fe2702fc640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fe2680b2b00 0x7fe2681f7fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:43:21.288 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:21.285+0000 7fe2702fc640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe2680d2820 0x7fe2681560b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:43:21.288 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:21.285+0000 7fe2702fc640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe268161620 0x7fe2681faad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:43:21.289 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:21.285+0000 7fe2702fc640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fe2680b1140 con 0x7fe268161620 2024-09-16T15:43:21.289 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:21.285+0000 7fe2702fc640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fe2680b0fc0 con 0x7fe2680d2820 2024-09-16T15:43:21.289 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:21.285+0000 7fe2702fc640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fe2680b12c0 con 0x7fe2680b2b00 2024-09-16T15:43:21.289 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:21.285+0000 7fe26e899640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe268161620 0x7fe2681faad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:43:21.289 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:21.285+0000 7fe26e899640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe268161620 0x7fe2681faad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:51678/0 (socket says 172.21.15.123:51678) 2024-09-16T15:43:21.289 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:21.285+0000 7fe26e899640 1 -- 172.21.15.123:0/1220567949 learned_addr learned my addr 172.21.15.123:0/1220567949 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:43:21.289 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:21.285+0000 7fe26d897640 1 --2- 172.21.15.123:0/1220567949 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe2680d2820 0x7fe2681560b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:43:21.289 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:21.285+0000 7fe26e098640 1 --2- 172.21.15.123:0/1220567949 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fe2680b2b00 0x7fe2681f7fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:43:21.289 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:21.285+0000 7fe26e899640 1 -- 172.21.15.123:0/1220567949 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fe2680b2b00 msgr2=0x7fe2681f7fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:43:21.289 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:21.285+0000 7fe26e899640 1 --2- 172.21.15.123:0/1220567949 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fe2680b2b00 0x7fe2681f7fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:21.289 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:21.285+0000 7fe26e899640 1 -- 172.21.15.123:0/1220567949 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe2680d2820 msgr2=0x7fe2681560b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:43:21.289 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:21.285+0000 7fe26e899640 1 --2- 172.21.15.123:0/1220567949 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe2680d2820 0x7fe2681560b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:21.289 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:21.285+0000 7fe26e899640 1 -- 172.21.15.123:0/1220567949 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe2681f74e0 con 0x7fe268161620 2024-09-16T15:43:21.290 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:21.285+0000 7fe26e899640 1 --2- 172.21.15.123:0/1220567949 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe268161620 0x7fe2681faad0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fe258009850 tx=0x7fe2580a5610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:43:21.290 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:21.285+0000 7fe266ffd640 1 -- 172.21.15.123:0/1220567949 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2580b2040 con 0x7fe268161620 2024-09-16T15:43:21.290 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:21.285+0000 7fe266ffd640 1 -- 172.21.15.123:0/1220567949 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fe2580043b0 con 0x7fe268161620 2024-09-16T15:43:21.290 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:21.285+0000 7fe2702fc640 1 -- 172.21.15.123:0/1220567949 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe2681fb4c0 con 0x7fe268161620 2024-09-16T15:43:21.290 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:21.285+0000 7fe266ffd640 1 -- 172.21.15.123:0/1220567949 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe258003860 con 0x7fe268161620 2024-09-16T15:43:21.290 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:21.285+0000 7fe2702fc640 1 -- 172.21.15.123:0/1220567949 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe2681f77a0 con 0x7fe268161620 2024-09-16T15:43:21.291 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:21.285+0000 7fe2702fc640 1 -- 172.21.15.123:0/1220567949 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe2680cea00 con 0x7fe268161620 2024-09-16T15:43:21.291 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:21.285+0000 7fe266ffd640 1 -- 172.21.15.123:0/1220567949 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 5) ==== 97080+0+0 (secure 0 0 0) 0x7fe2580bf060 con 0x7fe268161620 2024-09-16T15:43:21.291 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:21.285+0000 7fe266ffd640 1 --2- 172.21.15.123:0/1220567949 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fe254072ec0 0x7fe254075380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:43:21.291 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:21.285+0000 7fe266ffd640 1 -- 172.21.15.123:0/1220567949 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7fe25812d7f0 con 0x7fe268161620 2024-09-16T15:43:21.291 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:21.289+0000 7fe26e098640 1 --2- 172.21.15.123:0/1220567949 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fe254072ec0 0x7fe254075380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:43:21.291 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:21.289+0000 7fe26e098640 1 --2- 172.21.15.123:0/1220567949 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fe254072ec0 0x7fe254075380 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fe260009e00 tx=0x7fe260009390 comp rx=0 tx=0).ready entity=mgr.4135 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:43:21.295 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:21.293+0000 7fe266ffd640 1 -- 172.21.15.123:0/1220567949 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7fe2580164b0 con 0x7fe268161620 2024-09-16T15:43:21.479 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:21.477+0000 7fe2702fc640 1 -- 172.21.15.123:0/1220567949 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7fe2680760f0 con 0x7fe268161620 2024-09-16T15:43:21.480 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:21.477+0000 7fe266ffd640 1 -- 172.21.15.123:0/1220567949 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v21) ==== 80+0+0 (secure 0 0 0) 0x7fe2580f8ad0 con 0x7fe268161620 2024-09-16T15:43:21.480 INFO:teuthology.orchestra.run.smithi123.stderr:listed 0 entries 2024-09-16T15:43:21.482 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:21.477+0000 7fe2702fc640 1 -- 172.21.15.123:0/1220567949 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fe254072ec0 msgr2=0x7fe254075380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:43:21.482 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:21.477+0000 7fe2702fc640 1 --2- 172.21.15.123:0/1220567949 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fe254072ec0 0x7fe254075380 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fe260009e00 tx=0x7fe260009390 comp rx=0 tx=0).stop 2024-09-16T15:43:21.482 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:21.477+0000 7fe2702fc640 1 -- 172.21.15.123:0/1220567949 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe268161620 msgr2=0x7fe2681faad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:43:21.482 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:21.477+0000 7fe2702fc640 1 --2- 172.21.15.123:0/1220567949 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe268161620 0x7fe2681faad0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fe258009850 tx=0x7fe2580a5610 comp rx=0 tx=0).stop 2024-09-16T15:43:21.482 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:21.477+0000 7fe2702fc640 1 -- 172.21.15.123:0/1220567949 shutdown_connections 2024-09-16T15:43:21.482 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:21.477+0000 7fe2702fc640 1 --2- 172.21.15.123:0/1220567949 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fe254072ec0 0x7fe254075380 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:21.483 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:21.477+0000 7fe2702fc640 1 --2- 172.21.15.123:0/1220567949 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe268161620 0x7fe2681faad0 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:21.483 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:21.477+0000 7fe2702fc640 1 --2- 172.21.15.123:0/1220567949 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe2680d2820 0x7fe2681560b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:21.483 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:21.477+0000 7fe2702fc640 1 --2- 172.21.15.123:0/1220567949 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fe2680b2b00 0x7fe2681f7fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:21.483 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:21.477+0000 7fe2702fc640 1 -- 172.21.15.123:0/1220567949 >> 172.21.15.123:0/1220567949 conn(0x7fe2680d3ea0 msgr2=0x7fe26806e950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:43:21.483 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:21.477+0000 7fe2702fc640 1 -- 172.21.15.123:0/1220567949 shutdown_connections 2024-09-16T15:43:21.483 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:21.477+0000 7fe2702fc640 1 -- 172.21.15.123:0/1220567949 wait complete. 2024-09-16T15:43:21.499 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2024-09-16T15:43:21.499 DEBUG:teuthology.orchestra.run.smithi123:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-16T15:43:21.504 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-09-16T15:43:21.668 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:21.665+0000 7f74d577d640 1 -- 172.21.15.123:0/3708977129 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f74b80052f0 msgr2=0x7f74b800b9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:43:21.668 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:21.665+0000 7f74d577d640 1 --2- 172.21.15.123:0/3708977129 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f74b80052f0 0x7f74b800b9d0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f74c8009ee0 tx=0x7f74c801d630 comp rx=0 tx=0).stop 2024-09-16T15:43:21.668 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:21.665+0000 7f74d577d640 1 -- 172.21.15.123:0/3708977129 shutdown_connections 2024-09-16T15:43:21.669 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:21.665+0000 7f74d577d640 1 --2- 172.21.15.123:0/3708977129 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f74b80052f0 0x7f74b800b9d0 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:21.669 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:21.665+0000 7f74d577d640 1 -- 172.21.15.123:0/3708977129 >> 172.21.15.123:0/3708977129 conn(0x7f74d00cfb10 msgr2=0x7f74d00cff20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:43:21.669 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:21.665+0000 7f74d577d640 1 -- 172.21.15.123:0/3708977129 shutdown_connections 2024-09-16T15:43:21.669 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:21.665+0000 7f74d577d640 1 -- 172.21.15.123:0/3708977129 wait complete. 2024-09-16T15:43:21.669 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:21.665+0000 7f74d577d640 1 Processor -- start 2024-09-16T15:43:21.669 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:21.665+0000 7f74d577d640 1 -- start start 2024-09-16T15:43:21.671 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:21.665+0000 7f74d577d640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f74b80052f0 0x7f74d0152fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:43:21.671 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:21.665+0000 7f74d577d640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f74d00ce880 0x7f74d00c4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:43:21.672 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:21.665+0000 7f74d577d640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f74d00c5290 0x7f74d01f7df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:43:21.672 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:21.665+0000 7f74d577d640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f74d00d09c0 con 0x7f74d00ce880 2024-09-16T15:43:21.672 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:21.665+0000 7f74d577d640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f74d00d0840 con 0x7f74d00c5290 2024-09-16T15:43:21.672 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:21.665+0000 7f74d577d640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f74d015c190 con 0x7f74b80052f0 2024-09-16T15:43:21.672 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:21.665+0000 7f74ce7fc640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f74d00ce880 0x7f74d00c4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:43:21.672 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:21.665+0000 7f74ce7fc640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f74d00ce880 0x7f74d00c4d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:51692/0 (socket says 172.21.15.123:51692) 2024-09-16T15:43:21.672 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:21.665+0000 7f74ce7fc640 1 -- 172.21.15.123:0/2706679326 learned_addr learned my addr 172.21.15.123:0/2706679326 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:43:21.672 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:21.665+0000 7f74ceffd640 1 --2- 172.21.15.123:0/2706679326 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f74b80052f0 0x7f74d0152fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:43:21.672 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:21.665+0000 7f74cf7fe640 1 --2- 172.21.15.123:0/2706679326 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f74d00c5290 0x7f74d01f7df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:43:21.672 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:21.665+0000 7f74ce7fc640 1 -- 172.21.15.123:0/2706679326 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f74b80052f0 msgr2=0x7f74d0152fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:43:21.672 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:21.665+0000 7f74ce7fc640 1 --2- 172.21.15.123:0/2706679326 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f74b80052f0 0x7f74d0152fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:21.672 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:21.665+0000 7f74ce7fc640 1 -- 172.21.15.123:0/2706679326 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f74d00c5290 msgr2=0x7f74d01f7df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:43:21.672 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:21.665+0000 7f74ce7fc640 1 --2- 172.21.15.123:0/2706679326 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f74d00c5290 0x7f74d01f7df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:21.673 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:21.665+0000 7f74ce7fc640 1 -- 172.21.15.123:0/2706679326 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f74d00ce540 con 0x7f74d00ce880 2024-09-16T15:43:21.673 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:21.669+0000 7f74ce7fc640 1 --2- 172.21.15.123:0/2706679326 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f74d00ce880 0x7f74d00c4d50 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f74c400d5e0 tx=0x7f74c400daa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:43:21.673 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:21.669+0000 7f74b77fe640 1 -- 172.21.15.123:0/2706679326 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74c4016020 con 0x7f74d00ce880 2024-09-16T15:43:21.673 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:21.669+0000 7f74b77fe640 1 -- 172.21.15.123:0/2706679326 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f74c4012630 con 0x7f74d00ce880 2024-09-16T15:43:21.673 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:21.669+0000 7f74d577d640 1 -- 172.21.15.123:0/2706679326 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f74d015c310 con 0x7f74d00ce880 2024-09-16T15:43:21.673 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:21.669+0000 7f74b77fe640 1 -- 172.21.15.123:0/2706679326 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74c4012a40 con 0x7f74d00ce880 2024-09-16T15:43:21.673 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:21.669+0000 7f74d577d640 1 -- 172.21.15.123:0/2706679326 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f74d015c8d0 con 0x7f74d00ce880 2024-09-16T15:43:21.673 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:21.669+0000 7f74d577d640 1 -- 172.21.15.123:0/2706679326 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f74d0074270 con 0x7f74d00ce880 2024-09-16T15:43:21.674 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:21.669+0000 7f74b77fe640 1 -- 172.21.15.123:0/2706679326 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 5) ==== 97080+0+0 (secure 0 0 0) 0x7f74c4019020 con 0x7f74d00ce880 2024-09-16T15:43:21.674 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:21.669+0000 7f74b77fe640 1 --2- 172.21.15.123:0/2706679326 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f74bc072ef0 0x7f74bc0753b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:43:21.674 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:21.669+0000 7f74b77fe640 1 -- 172.21.15.123:0/2706679326 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f74c4093a40 con 0x7f74d00ce880 2024-09-16T15:43:21.674 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:21.669+0000 7f74ceffd640 1 --2- 172.21.15.123:0/2706679326 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f74bc072ef0 0x7f74bc0753b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:43:21.674 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:21.669+0000 7f74ceffd640 1 --2- 172.21.15.123:0/2706679326 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f74bc072ef0 0x7f74bc0753b0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f74c00046e0 tx=0x7f74c0001570 comp rx=0 tx=0).ready entity=mgr.4135 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:43:21.677 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:21.673+0000 7f74b77fe640 1 -- 172.21.15.123:0/2706679326 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f74c4020020 con 0x7f74d00ce880 2024-09-16T15:43:21.848 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:21.845+0000 7f74d577d640 1 -- 172.21.15.123:0/2706679326 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f74d015f630 con 0x7f74d00ce880 2024-09-16T15:43:21.849 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:21.845+0000 7f74b77fe640 1 -- 172.21.15.123:0/2706679326 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v21) ==== 80+0+0 (secure 0 0 0) 0x7f74c4093e20 con 0x7f74d00ce880 2024-09-16T15:43:21.849 INFO:teuthology.orchestra.run.smithi123.stderr:listed 0 entries 2024-09-16T15:43:21.851 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:21.849+0000 7f74d577d640 1 -- 172.21.15.123:0/2706679326 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f74bc072ef0 msgr2=0x7f74bc0753b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:43:21.851 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:21.849+0000 7f74d577d640 1 --2- 172.21.15.123:0/2706679326 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f74bc072ef0 0x7f74bc0753b0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f74c00046e0 tx=0x7f74c0001570 comp rx=0 tx=0).stop 2024-09-16T15:43:21.851 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:21.849+0000 7f74d577d640 1 -- 172.21.15.123:0/2706679326 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f74d00ce880 msgr2=0x7f74d00c4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:43:21.851 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:21.849+0000 7f74d577d640 1 --2- 172.21.15.123:0/2706679326 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f74d00ce880 0x7f74d00c4d50 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f74c400d5e0 tx=0x7f74c400daa0 comp rx=0 tx=0).stop 2024-09-16T15:43:21.851 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:21.849+0000 7f74d577d640 1 -- 172.21.15.123:0/2706679326 shutdown_connections 2024-09-16T15:43:21.852 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:21.849+0000 7f74d577d640 1 --2- 172.21.15.123:0/2706679326 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f74bc072ef0 0x7f74bc0753b0 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:21.852 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:21.849+0000 7f74d577d640 1 --2- 172.21.15.123:0/2706679326 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f74d00c5290 0x7f74d01f7df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:21.852 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:21.849+0000 7f74d577d640 1 --2- 172.21.15.123:0/2706679326 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f74d00ce880 0x7f74d00c4d50 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:21.852 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:21.849+0000 7f74d577d640 1 --2- 172.21.15.123:0/2706679326 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f74b80052f0 0x7f74d0152fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:21.852 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:21.849+0000 7f74d577d640 1 -- 172.21.15.123:0/2706679326 >> 172.21.15.123:0/2706679326 conn(0x7f74d00cfb10 msgr2=0x7f74d00cafd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:43:21.852 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:21.849+0000 7f74d577d640 1 -- 172.21.15.123:0/2706679326 shutdown_connections 2024-09-16T15:43:21.852 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:21.849+0000 7f74d577d640 1 -- 172.21.15.123:0/2706679326 wait complete. 2024-09-16T15:43:21.867 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi123.front.sepia.ceph.com... 2024-09-16T15:43:21.867 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:43:21.867 DEBUG:teuthology.orchestra.run.smithi123:> sudo fusermount -u /home/ubuntu/cephtest/mnt.2 2024-09-16T15:43:21.877 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T15:43:21.878 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi123.front.sepia.ceph.com 2024-09-16T15:43:21.878 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:43:21.878 DEBUG:teuthology.orchestra.run.smithi123:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.2) 2024-09-16T15:43:21.923 DEBUG:teuthology.orchestra.run.smithi184:> set -ex 2024-09-16T15:43:21.923 DEBUG:teuthology.orchestra.run.smithi184:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-16T15:43:21.930 DEBUG:teuthology.orchestra.run.smithi184:> set -ex 2024-09-16T15:43:21.930 DEBUG:teuthology.orchestra.run.smithi184:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-16T15:43:21.975 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-09-16T15:43:22.105 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:22.101+0000 7f4c05037640 1 -- 172.21.15.123:0/2664213060 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f4c000cf5a0 msgr2=0x7f4c0006eee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:43:22.105 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:22.101+0000 7f4c05037640 1 --2- 172.21.15.123:0/2664213060 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f4c000cf5a0 0x7f4c0006eee0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f4bf400aa10 tx=0x7f4bf401db10 comp rx=0 tx=0).stop 2024-09-16T15:43:22.105 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:22.101+0000 7f4c05037640 1 -- 172.21.15.123:0/2664213060 shutdown_connections 2024-09-16T15:43:22.105 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:22.101+0000 7f4c05037640 1 --2- 172.21.15.123:0/2664213060 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f4c000cf5a0 0x7f4c0006eee0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:22.105 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:22.101+0000 7f4c05037640 1 --2- 172.21.15.123:0/2664213060 >> v2:172.21.15.184:3300/0 conn(0x7f4c00158120 0x7f4c000ca320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:22.105 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:22.101+0000 7f4c05037640 1 -- 172.21.15.123:0/2664213060 >> 172.21.15.123:0/2664213060 conn(0x7f4c000cfb10 msgr2=0x7f4c000cff20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:43:22.106 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:22.101+0000 7f4c05037640 1 -- 172.21.15.123:0/2664213060 shutdown_connections 2024-09-16T15:43:22.106 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:22.101+0000 7f4c05037640 1 -- 172.21.15.123:0/2664213060 wait complete. 2024-09-16T15:43:22.106 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:22.101+0000 7f4c05037640 1 Processor -- start 2024-09-16T15:43:22.106 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:22.105+0000 7f4c05037640 1 -- start start 2024-09-16T15:43:22.109 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:22.105+0000 7f4c05037640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4c000cf5a0 0x7f4c000b5480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:43:22.109 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:22.105+0000 7f4c05037640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4c00158120 0x7f4c0014ca50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:43:22.109 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:22.105+0000 7f4c05037640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f4c0015ec40 0x7f4c000b29a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:43:22.109 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:22.105+0000 7f4c05037640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f4c0015b770 con 0x7f4c000cf5a0 2024-09-16T15:43:22.109 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:22.105+0000 7f4c05037640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f4c0015b5f0 con 0x7f4c00158120 2024-09-16T15:43:22.109 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:22.105+0000 7f4c05037640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f4c001673d0 con 0x7f4c0015ec40 2024-09-16T15:43:22.109 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:22.105+0000 7f4bfed9d640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4c000cf5a0 0x7f4c000b5480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:43:22.109 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:22.105+0000 7f4bfed9d640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4c000cf5a0 0x7f4c000b5480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:51706/0 (socket says 172.21.15.123:51706) 2024-09-16T15:43:22.109 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:22.105+0000 7f4bfed9d640 1 -- 172.21.15.123:0/565723559 learned_addr learned my addr 172.21.15.123:0/565723559 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:43:22.110 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:22.105+0000 7f4bfe59c640 1 --2- 172.21.15.123:0/565723559 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4c00158120 0x7f4c0014ca50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:43:22.110 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:22.105+0000 7f4bff59e640 1 --2- 172.21.15.123:0/565723559 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f4c0015ec40 0x7f4c000b29a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:43:22.110 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:22.105+0000 7f4bfed9d640 1 -- 172.21.15.123:0/565723559 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f4c0015ec40 msgr2=0x7f4c000b29a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:43:22.110 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:22.105+0000 7f4bfed9d640 1 --2- 172.21.15.123:0/565723559 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f4c0015ec40 0x7f4c000b29a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:22.110 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:22.105+0000 7f4bfed9d640 1 -- 172.21.15.123:0/565723559 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4c00158120 msgr2=0x7f4c0014ca50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:43:22.110 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:22.105+0000 7f4bfed9d640 1 --2- 172.21.15.123:0/565723559 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4c00158120 0x7f4c0014ca50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:22.110 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:22.105+0000 7f4bfed9d640 1 -- 172.21.15.123:0/565723559 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4c0014d600 con 0x7f4c000cf5a0 2024-09-16T15:43:22.110 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:22.105+0000 7f4bfed9d640 1 --2- 172.21.15.123:0/565723559 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4c000cf5a0 0x7f4c000b5480 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f4bf000ea40 tx=0x7f4bf000ef00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:43:22.110 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:22.105+0000 7f4be77fe640 1 -- 172.21.15.123:0/565723559 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4bf000ce40 con 0x7f4c000cf5a0 2024-09-16T15:43:22.110 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:22.105+0000 7f4be77fe640 1 -- 172.21.15.123:0/565723559 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f4bf0004020 con 0x7f4c000cf5a0 2024-09-16T15:43:22.110 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:22.105+0000 7f4c05037640 1 -- 172.21.15.123:0/565723559 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4c0014d8e0 con 0x7f4c000cf5a0 2024-09-16T15:43:22.110 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:22.105+0000 7f4be77fe640 1 -- 172.21.15.123:0/565723559 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4bf0004470 con 0x7f4c000cf5a0 2024-09-16T15:43:22.110 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:22.105+0000 7f4c05037640 1 -- 172.21.15.123:0/565723559 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4c0014dea0 con 0x7f4c000cf5a0 2024-09-16T15:43:22.111 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:22.105+0000 7f4c05037640 1 -- 172.21.15.123:0/565723559 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4c0006eee0 con 0x7f4c000cf5a0 2024-09-16T15:43:22.111 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:22.105+0000 7f4be77fe640 1 -- 172.21.15.123:0/565723559 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 5) ==== 97080+0+0 (secure 0 0 0) 0x7f4bf0003600 con 0x7f4c000cf5a0 2024-09-16T15:43:22.111 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:22.109+0000 7f4be77fe640 1 --2- 172.21.15.123:0/565723559 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f4bec072ec0 0x7f4bec075380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:43:22.111 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:22.109+0000 7f4be77fe640 1 -- 172.21.15.123:0/565723559 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f4bf0094810 con 0x7f4c000cf5a0 2024-09-16T15:43:22.112 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:22.109+0000 7f4bfe59c640 1 --2- 172.21.15.123:0/565723559 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f4bec072ec0 0x7f4bec075380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:43:22.112 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:22.109+0000 7f4bfe59c640 1 --2- 172.21.15.123:0/565723559 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f4bec072ec0 0x7f4bec075380 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f4bf40b0000 tx=0x7f4bf401c350 comp rx=0 tx=0).ready entity=mgr.4135 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:43:22.115 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:22.113+0000 7f4be77fe640 1 -- 172.21.15.123:0/565723559 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f4bf005f9e0 con 0x7f4c000cf5a0 2024-09-16T15:43:22.300 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:22.297+0000 7f4c05037640 1 -- 172.21.15.123:0/565723559 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f4c000d06c0 con 0x7f4c000cf5a0 2024-09-16T15:43:22.300 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:22.297+0000 7f4be77fe640 1 -- 172.21.15.123:0/565723559 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v21) ==== 80+0+0 (secure 0 0 0) 0x7f4bf005f2e0 con 0x7f4c000cf5a0 2024-09-16T15:43:22.301 INFO:teuthology.orchestra.run.smithi123.stderr:listed 0 entries 2024-09-16T15:43:22.303 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:22.297+0000 7f4c05037640 1 -- 172.21.15.123:0/565723559 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f4bec072ec0 msgr2=0x7f4bec075380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:43:22.303 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:22.297+0000 7f4c05037640 1 --2- 172.21.15.123:0/565723559 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f4bec072ec0 0x7f4bec075380 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f4bf40b0000 tx=0x7f4bf401c350 comp rx=0 tx=0).stop 2024-09-16T15:43:22.303 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:22.297+0000 7f4c05037640 1 -- 172.21.15.123:0/565723559 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4c000cf5a0 msgr2=0x7f4c000b5480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:43:22.303 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:22.297+0000 7f4c05037640 1 --2- 172.21.15.123:0/565723559 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4c000cf5a0 0x7f4c000b5480 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f4bf000ea40 tx=0x7f4bf000ef00 comp rx=0 tx=0).stop 2024-09-16T15:43:22.304 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:22.301+0000 7f4c05037640 1 -- 172.21.15.123:0/565723559 shutdown_connections 2024-09-16T15:43:22.304 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:22.301+0000 7f4c05037640 1 --2- 172.21.15.123:0/565723559 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f4bec072ec0 0x7f4bec075380 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:22.304 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:22.301+0000 7f4c05037640 1 --2- 172.21.15.123:0/565723559 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f4c0015ec40 0x7f4c000b29a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:22.304 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:22.301+0000 7f4c05037640 1 --2- 172.21.15.123:0/565723559 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4c00158120 0x7f4c0014ca50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:22.304 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:22.301+0000 7f4c05037640 1 --2- 172.21.15.123:0/565723559 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4c000cf5a0 0x7f4c000b5480 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:22.304 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:22.301+0000 7f4c05037640 1 -- 172.21.15.123:0/565723559 >> 172.21.15.123:0/565723559 conn(0x7f4c000cfb10 msgr2=0x7f4c000ca140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:43:22.304 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:22.301+0000 7f4c05037640 1 -- 172.21.15.123:0/565723559 shutdown_connections 2024-09-16T15:43:22.304 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:22.301+0000 7f4c05037640 1 -- 172.21.15.123:0/565723559 wait complete. 2024-09-16T15:43:22.320 DEBUG:teuthology.orchestra.run.smithi184:> set -ex 2024-09-16T15:43:22.320 DEBUG:teuthology.orchestra.run.smithi184:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-16T15:43:22.327 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-09-16T15:43:22.451 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:22.449+0000 7f0dc8613640 1 -- 172.21.15.123:0/2234318741 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0dc00b0a50 msgr2=0x7f0dc00b0ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:43:22.451 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:22.449+0000 7f0dc8613640 1 --2- 172.21.15.123:0/2234318741 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0dc00b0a50 0x7f0dc00b0ee0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f0db8009c80 tx=0x7f0db801cc50 comp rx=0 tx=0).stop 2024-09-16T15:43:22.452 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:22.449+0000 7f0dc8613640 1 -- 172.21.15.123:0/2234318741 shutdown_connections 2024-09-16T15:43:22.452 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:22.449+0000 7f0dc8613640 1 --2- 172.21.15.123:0/2234318741 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0dc00b0a50 0x7f0dc00b0ee0 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:22.452 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:22.449+0000 7f0dc8613640 1 --2- 172.21.15.123:0/2234318741 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f0dc00b2b00 0x7f0dc00d24b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:22.452 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:22.449+0000 7f0dc8613640 1 -- 172.21.15.123:0/2234318741 >> 172.21.15.123:0/2234318741 conn(0x7f0dc00d3ea0 msgr2=0x7f0dc00d42b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:43:22.452 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:22.449+0000 7f0dc8613640 1 -- 172.21.15.123:0/2234318741 shutdown_connections 2024-09-16T15:43:22.452 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:22.449+0000 7f0dc8613640 1 -- 172.21.15.123:0/2234318741 wait complete. 2024-09-16T15:43:22.453 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:22.449+0000 7f0dc8613640 1 Processor -- start 2024-09-16T15:43:22.453 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:22.449+0000 7f0dc8613640 1 -- start start 2024-09-16T15:43:22.455 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:22.449+0000 7f0dc8613640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f0dc00b0a50 0x7f0dc01503d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:43:22.455 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:22.449+0000 7f0dc8613640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0dc00b2b00 0x7f0dc01587e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:43:22.455 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:22.449+0000 7f0dc8613640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f0dc0161620 0x7f0dc0158d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:43:22.455 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:22.449+0000 7f0dc8613640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f0dc016b0f0 con 0x7f0dc00b2b00 2024-09-16T15:43:22.455 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:22.449+0000 7f0dc8613640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f0dc016af70 con 0x7f0dc0161620 2024-09-16T15:43:22.455 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:22.449+0000 7f0dc8613640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f0dc016b270 con 0x7f0dc00b0a50 2024-09-16T15:43:22.455 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:22.449+0000 7f0dc5bae640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0dc00b2b00 0x7f0dc01587e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:43:22.455 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:22.449+0000 7f0dc5bae640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0dc00b2b00 0x7f0dc01587e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:36374/0 (socket says 172.21.15.123:36374) 2024-09-16T15:43:22.455 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:22.449+0000 7f0dc5bae640 1 -- 172.21.15.123:0/4059750715 learned_addr learned my addr 172.21.15.123:0/4059750715 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:43:22.455 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:22.449+0000 7f0dc6bb0640 1 --2- 172.21.15.123:0/4059750715 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f0dc0161620 0x7f0dc0158d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:43:22.456 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:22.449+0000 7f0dc63af640 1 --2- 172.21.15.123:0/4059750715 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f0dc00b0a50 0x7f0dc01503d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:43:22.456 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:22.449+0000 7f0dc5bae640 1 -- 172.21.15.123:0/4059750715 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f0dc00b0a50 msgr2=0x7f0dc01503d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:43:22.456 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:22.449+0000 7f0dc5bae640 1 --2- 172.21.15.123:0/4059750715 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f0dc00b0a50 0x7f0dc01503d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:22.456 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:22.449+0000 7f0dc5bae640 1 -- 172.21.15.123:0/4059750715 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f0dc0161620 msgr2=0x7f0dc0158d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:43:22.456 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:22.449+0000 7f0dc5bae640 1 --2- 172.21.15.123:0/4059750715 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f0dc0161620 0x7f0dc0158d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:22.456 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:22.449+0000 7f0dc5bae640 1 -- 172.21.15.123:0/4059750715 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0dc0152b40 con 0x7f0dc00b2b00 2024-09-16T15:43:22.456 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:22.449+0000 7f0dc5bae640 1 --2- 172.21.15.123:0/4059750715 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0dc00b2b00 0x7f0dc01587e0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f0db000b580 tx=0x7f0db000ba40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:43:22.456 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:22.449+0000 7f0db6ffd640 1 -- 172.21.15.123:0/4059750715 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0db0014070 con 0x7f0dc00b2b00 2024-09-16T15:43:22.456 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:22.449+0000 7f0db6ffd640 1 -- 172.21.15.123:0/4059750715 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f0db0010980 con 0x7f0dc00b2b00 2024-09-16T15:43:22.456 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:22.449+0000 7f0dc8613640 1 -- 172.21.15.123:0/4059750715 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0dc00b9bc0 con 0x7f0dc00b2b00 2024-09-16T15:43:22.456 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:22.449+0000 7f0db6ffd640 1 -- 172.21.15.123:0/4059750715 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0db0010dd0 con 0x7f0dc00b2b00 2024-09-16T15:43:22.457 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:22.449+0000 7f0dc8613640 1 -- 172.21.15.123:0/4059750715 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0dc0152060 con 0x7f0dc00b2b00 2024-09-16T15:43:22.457 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:22.453+0000 7f0dc8613640 1 -- 172.21.15.123:0/4059750715 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0dc00b0ee0 con 0x7f0dc00b2b00 2024-09-16T15:43:22.457 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:22.453+0000 7f0db6ffd640 1 -- 172.21.15.123:0/4059750715 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 5) ==== 97080+0+0 (secure 0 0 0) 0x7f0db0017070 con 0x7f0dc00b2b00 2024-09-16T15:43:22.458 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:22.453+0000 7f0db6ffd640 1 --2- 172.21.15.123:0/4059750715 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f0da4072ec0 0x7f0da4075380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:43:22.458 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:22.453+0000 7f0db6ffd640 1 -- 172.21.15.123:0/4059750715 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f0db0093cc0 con 0x7f0dc00b2b00 2024-09-16T15:43:22.458 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:22.453+0000 7f0dc63af640 1 --2- 172.21.15.123:0/4059750715 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f0da4072ec0 0x7f0da4075380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:43:22.458 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:22.453+0000 7f0dc63af640 1 --2- 172.21.15.123:0/4059750715 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f0da4072ec0 0x7f0da4075380 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f0dbc006fd0 tx=0x7f0dbc008040 comp rx=0 tx=0).ready entity=mgr.4135 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:43:22.460 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:22.457+0000 7f0db6ffd640 1 -- 172.21.15.123:0/4059750715 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f0db005ef20 con 0x7f0dc00b2b00 2024-09-16T15:43:22.639 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:22.637+0000 7f0dc8613640 1 -- 172.21.15.123:0/4059750715 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f0dc00b27a0 con 0x7f0dc00b2b00 2024-09-16T15:43:22.640 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:22.637+0000 7f0db6ffd640 1 -- 172.21.15.123:0/4059750715 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v21) ==== 80+0+0 (secure 0 0 0) 0x7f0db005e820 con 0x7f0dc00b2b00 2024-09-16T15:43:22.640 INFO:teuthology.orchestra.run.smithi123.stderr:listed 0 entries 2024-09-16T15:43:22.642 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:22.637+0000 7f0dc8613640 1 -- 172.21.15.123:0/4059750715 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f0da4072ec0 msgr2=0x7f0da4075380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:43:22.642 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:22.637+0000 7f0dc8613640 1 --2- 172.21.15.123:0/4059750715 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f0da4072ec0 0x7f0da4075380 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f0dbc006fd0 tx=0x7f0dbc008040 comp rx=0 tx=0).stop 2024-09-16T15:43:22.642 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:22.637+0000 7f0dc8613640 1 -- 172.21.15.123:0/4059750715 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0dc00b2b00 msgr2=0x7f0dc01587e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:43:22.642 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:22.637+0000 7f0dc8613640 1 --2- 172.21.15.123:0/4059750715 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0dc00b2b00 0x7f0dc01587e0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f0db000b580 tx=0x7f0db000ba40 comp rx=0 tx=0).stop 2024-09-16T15:43:22.642 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:22.637+0000 7f0dc8613640 1 -- 172.21.15.123:0/4059750715 shutdown_connections 2024-09-16T15:43:22.642 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:22.637+0000 7f0dc8613640 1 --2- 172.21.15.123:0/4059750715 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f0da4072ec0 0x7f0da4075380 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:22.642 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:22.637+0000 7f0dc8613640 1 --2- 172.21.15.123:0/4059750715 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f0dc0161620 0x7f0dc0158d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:22.642 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:22.637+0000 7f0dc8613640 1 --2- 172.21.15.123:0/4059750715 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0dc00b2b00 0x7f0dc01587e0 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:22.643 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:22.637+0000 7f0dc8613640 1 --2- 172.21.15.123:0/4059750715 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f0dc00b0a50 0x7f0dc01503d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:22.643 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:22.637+0000 7f0dc8613640 1 -- 172.21.15.123:0/4059750715 >> 172.21.15.123:0/4059750715 conn(0x7f0dc00d3ea0 msgr2=0x7f0dc00b62c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:43:22.643 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:22.637+0000 7f0dc8613640 1 -- 172.21.15.123:0/4059750715 shutdown_connections 2024-09-16T15:43:22.643 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:22.637+0000 7f0dc8613640 1 -- 172.21.15.123:0/4059750715 wait complete. 2024-09-16T15:43:22.657 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi184.front.sepia.ceph.com... 2024-09-16T15:43:22.658 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:43:22.658 DEBUG:teuthology.orchestra.run.smithi184:> sudo fusermount -u /home/ubuntu/cephtest/mnt.3 2024-09-16T15:43:22.672 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T15:43:22.673 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi184.front.sepia.ceph.com 2024-09-16T15:43:22.673 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:43:22.673 DEBUG:teuthology.orchestra.run.smithi184:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.3) 2024-09-16T15:43:22.720 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-09-16T15:43:22.720 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-16T15:43:22.844 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:22.841+0000 7ff3d2012640 1 -- 172.21.15.123:0/617023749 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff3cc074600 msgr2=0x7ff3cc06e750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:43:22.844 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:22.841+0000 7ff3d2012640 1 --2- 172.21.15.123:0/617023749 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff3cc074600 0x7ff3cc06e750 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7ff3cc0cd650 tx=0x7ff3c401c9e0 comp rx=0 tx=0).stop 2024-09-16T15:43:22.844 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:22.841+0000 7ff3d2012640 1 -- 172.21.15.123:0/617023749 shutdown_connections 2024-09-16T15:43:22.844 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:22.841+0000 7ff3d2012640 1 --2- 172.21.15.123:0/617023749 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff3cc074600 0x7ff3cc06e750 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:22.844 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:22.841+0000 7ff3d2012640 1 --2- 172.21.15.123:0/617023749 >> v2:172.21.15.184:3300/0 conn(0x7ff3cc15bca0 0x7ff3cc0d0630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:22.844 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:22.841+0000 7ff3d2012640 1 -- 172.21.15.123:0/617023749 >> 172.21.15.123:0/617023749 conn(0x7ff3cc0cfb10 msgr2=0x7ff3cc0cff20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:43:22.844 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:22.841+0000 7ff3d2012640 1 -- 172.21.15.123:0/617023749 shutdown_connections 2024-09-16T15:43:22.844 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:22.841+0000 7ff3d2012640 1 -- 172.21.15.123:0/617023749 wait complete. 2024-09-16T15:43:22.845 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:22.841+0000 7ff3d2012640 1 Processor -- start 2024-09-16T15:43:22.845 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:22.841+0000 7ff3d2012640 1 -- start start 2024-09-16T15:43:22.847 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:22.841+0000 7ff3d2012640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff3cc0726c0 0x7ff3cc14ebd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:43:22.847 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:22.841+0000 7ff3d2012640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7ff3cc074600 0x7ff3cc0cef30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:43:22.847 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:22.841+0000 7ff3d2012640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff3cc15bca0 0x7ff3cc0c4ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:43:22.847 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:22.841+0000 7ff3d2012640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7ff3cc168a50 con 0x7ff3cc15bca0 2024-09-16T15:43:22.847 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:22.841+0000 7ff3d2012640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7ff3cc06fa60 con 0x7ff3cc0726c0 2024-09-16T15:43:22.848 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:22.841+0000 7ff3d2012640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7ff3cc168bd0 con 0x7ff3cc074600 2024-09-16T15:43:22.848 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:22.841+0000 7ff3cbfff640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff3cc15bca0 0x7ff3cc0c4ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:43:22.848 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:22.841+0000 7ff3cbfff640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff3cc15bca0 0x7ff3cc0c4ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:36388/0 (socket says 172.21.15.123:36388) 2024-09-16T15:43:22.848 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:22.841+0000 7ff3cbfff640 1 -- 172.21.15.123:0/2478027921 learned_addr learned my addr 172.21.15.123:0/2478027921 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:43:22.848 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:22.841+0000 7ff3cb7fe640 1 --2- 172.21.15.123:0/2478027921 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff3cc0726c0 0x7ff3cc14ebd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:43:22.848 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:22.841+0000 7ff3cbfff640 1 -- 172.21.15.123:0/2478027921 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7ff3cc074600 msgr2=0x7ff3cc0cef30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:43:22.848 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:22.841+0000 7ff3cbfff640 1 --2- 172.21.15.123:0/2478027921 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7ff3cc074600 0x7ff3cc0cef30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:22.848 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:22.841+0000 7ff3cbfff640 1 -- 172.21.15.123:0/2478027921 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff3cc0726c0 msgr2=0x7ff3cc14ebd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:43:22.848 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:22.841+0000 7ff3cbfff640 1 --2- 172.21.15.123:0/2478027921 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff3cc0726c0 0x7ff3cc14ebd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:22.848 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:22.841+0000 7ff3cbfff640 1 -- 172.21.15.123:0/2478027921 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff3cc06a0f0 con 0x7ff3cc15bca0 2024-09-16T15:43:22.848 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:22.841+0000 7ff3cbfff640 1 --2- 172.21.15.123:0/2478027921 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff3cc15bca0 0x7ff3cc0c4ef0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7ff3c401cec0 tx=0x7ff3c4005b00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:43:22.849 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:22.841+0000 7ff3b3fff640 1 -- 172.21.15.123:0/2478027921 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3c40bb070 con 0x7ff3cc15bca0 2024-09-16T15:43:22.849 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:22.841+0000 7ff3b3fff640 1 -- 172.21.15.123:0/2478027921 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7ff3c4004a50 con 0x7ff3cc15bca0 2024-09-16T15:43:22.849 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:22.841+0000 7ff3d2012640 1 -- 172.21.15.123:0/2478027921 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff3cc1f7980 con 0x7ff3cc15bca0 2024-09-16T15:43:22.849 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:22.841+0000 7ff3b3fff640 1 -- 172.21.15.123:0/2478027921 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3c40b2040 con 0x7ff3cc15bca0 2024-09-16T15:43:22.849 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:22.845+0000 7ff3d2012640 1 -- 172.21.15.123:0/2478027921 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff3cc1f7f40 con 0x7ff3cc15bca0 2024-09-16T15:43:22.849 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:22.845+0000 7ff3d2012640 1 -- 172.21.15.123:0/2478027921 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff3cc06e750 con 0x7ff3cc15bca0 2024-09-16T15:43:22.850 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:22.845+0000 7ff3b3fff640 1 -- 172.21.15.123:0/2478027921 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 5) ==== 97080+0+0 (secure 0 0 0) 0x7ff3c40be070 con 0x7ff3cc15bca0 2024-09-16T15:43:22.850 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:22.845+0000 7ff3b3fff640 1 --2- 172.21.15.123:0/2478027921 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7ff3b8072ef0 0x7ff3b80753b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:43:22.850 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:22.845+0000 7ff3b3fff640 1 -- 172.21.15.123:0/2478027921 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7ff3c412d950 con 0x7ff3cc15bca0 2024-09-16T15:43:22.850 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:22.845+0000 7ff3cb7fe640 1 --2- 172.21.15.123:0/2478027921 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7ff3b8072ef0 0x7ff3b80753b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:43:22.850 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:22.845+0000 7ff3cb7fe640 1 --2- 172.21.15.123:0/2478027921 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7ff3b8072ef0 0x7ff3b80753b0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7ff3c00058d0 tx=0x7ff3c0005860 comp rx=0 tx=0).ready entity=mgr.4135 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:43:22.852 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:22.849+0000 7ff3b3fff640 1 -- 172.21.15.123:0/2478027921 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7ff3c4016690 con 0x7ff3cc15bca0 2024-09-16T15:43:23.044 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:23.041+0000 7ff3d2012640 1 -- 172.21.15.123:0/2478027921 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff3cc0b3d10 con 0x7ff3cc15bca0 2024-09-16T15:43:23.044 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:23.041+0000 7ff3b3fff640 1 -- 172.21.15.123:0/2478027921 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) ==== 93+0+5226 (secure 0 0 0) 0x7ff3c40f8ba0 con 0x7ff3cc15bca0 2024-09-16T15:43:23.045 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-09-16T15:43:23.045 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":8,"btime":"2024-09-16T15:42:54:421548+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4280,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6835/1497621896","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":1497621896},{"type":"v1","addr":"172.21.15.184:6835","nonce":1497621896}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4305,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6833/2227496057","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6832","nonce":2227496057},{"type":"v1","addr":"172.21.15.123:6833","nonce":2227496057}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4311,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6835/294455998","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6834","nonce":294455998},{"type":"v1","addr":"172.21.15.123:6835","nonce":294455998}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-16T15:42:50.762280+0000","modified":"2024-09-16T15:42:53.789989+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4348},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4348":{"gid":4348,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.184:6837/1469324095","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6836","nonce":1469324095},{"type":"v1","addr":"172.21.15.184:6837","nonce":1469324095}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4348,"qdb_cluster":[4348]},"id":1}]} 2024-09-16T15:43:23.045 INFO:teuthology.orchestra.run.smithi123.stderr:dumped fsmap epoch 8 2024-09-16T15:43:23.047 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:23.041+0000 7ff3d2012640 1 -- 172.21.15.123:0/2478027921 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7ff3b8072ef0 msgr2=0x7ff3b80753b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:43:23.047 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:23.041+0000 7ff3d2012640 1 --2- 172.21.15.123:0/2478027921 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7ff3b8072ef0 0x7ff3b80753b0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7ff3c00058d0 tx=0x7ff3c0005860 comp rx=0 tx=0).stop 2024-09-16T15:43:23.047 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:23.045+0000 7ff3d2012640 1 -- 172.21.15.123:0/2478027921 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff3cc15bca0 msgr2=0x7ff3cc0c4ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:43:23.047 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:23.045+0000 7ff3d2012640 1 --2- 172.21.15.123:0/2478027921 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff3cc15bca0 0x7ff3cc0c4ef0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7ff3c401cec0 tx=0x7ff3c4005b00 comp rx=0 tx=0).stop 2024-09-16T15:43:23.047 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:23.045+0000 7ff3d2012640 1 -- 172.21.15.123:0/2478027921 shutdown_connections 2024-09-16T15:43:23.047 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:23.045+0000 7ff3d2012640 1 --2- 172.21.15.123:0/2478027921 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7ff3b8072ef0 0x7ff3b80753b0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:23.047 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:23.045+0000 7ff3d2012640 1 --2- 172.21.15.123:0/2478027921 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff3cc15bca0 0x7ff3cc0c4ef0 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:23.047 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:23.045+0000 7ff3d2012640 1 --2- 172.21.15.123:0/2478027921 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7ff3cc074600 0x7ff3cc0cef30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:23.047 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:23.045+0000 7ff3d2012640 1 --2- 172.21.15.123:0/2478027921 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff3cc0726c0 0x7ff3cc14ebd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:23.047 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:23.045+0000 7ff3d2012640 1 -- 172.21.15.123:0/2478027921 >> 172.21.15.123:0/2478027921 conn(0x7ff3cc0cfb10 msgr2=0x7ff3cc0b4670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:43:23.047 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:23.045+0000 7ff3d2012640 1 -- 172.21.15.123:0/2478027921 shutdown_connections 2024-09-16T15:43:23.048 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:23.045+0000 7ff3d2012640 1 -- 172.21.15.123:0/2478027921 wait complete. 2024-09-16T15:43:23.062 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-16T15:43:23.184 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:23.181+0000 7fec0f3dd640 1 -- 172.21.15.123:0/3103411960 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7febf4004b20 msgr2=0x7febf4004f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:43:23.184 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:23.181+0000 7fec0f3dd640 1 --2- 172.21.15.123:0/3103411960 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7febf4004b20 0x7febf4004f40 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7fec0400bb10 tx=0x7fec0401cde0 comp rx=0 tx=0).stop 2024-09-16T15:43:23.184 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:23.181+0000 7fec0d97a640 1 -- 172.21.15.123:0/3103411960 reap_dead start 2024-09-16T15:43:23.184 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:23.181+0000 7fec0f3dd640 1 -- 172.21.15.123:0/3103411960 shutdown_connections 2024-09-16T15:43:23.184 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:23.181+0000 7fec0f3dd640 1 -- 172.21.15.123:0/3103411960 >> 172.21.15.123:0/3103411960 conn(0x7fec080cfb10 msgr2=0x7fec080cff20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:43:23.184 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:23.181+0000 7fec0f3dd640 1 -- 172.21.15.123:0/3103411960 shutdown_connections 2024-09-16T15:43:23.185 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:23.181+0000 7fec0f3dd640 1 -- 172.21.15.123:0/3103411960 wait complete. 2024-09-16T15:43:23.185 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:23.181+0000 7fec0f3dd640 1 Processor -- start 2024-09-16T15:43:23.185 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:23.181+0000 7fec0f3dd640 1 -- start start 2024-09-16T15:43:23.187 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:23.181+0000 7fec0f3dd640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7febf4004b20 0x7fec081510b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:43:23.188 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:23.181+0000 7fec0f3dd640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fec080c94e0 0x7fec080c4a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:43:23.188 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:23.181+0000 7fec0f3dd640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fec080b36d0 0x7fec080b3b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:43:23.188 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:23.181+0000 7fec0f3dd640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fec0815b120 con 0x7fec080c94e0 2024-09-16T15:43:23.188 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:23.181+0000 7fec0f3dd640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fec0815afa0 con 0x7fec080b36d0 2024-09-16T15:43:23.188 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:23.181+0000 7fec0f3dd640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fec0815b2a0 con 0x7febf4004b20 2024-09-16T15:43:23.188 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:23.181+0000 7fec0c978640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fec080c94e0 0x7fec080c4a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:43:23.188 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:23.181+0000 7fec0c978640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fec080c94e0 0x7fec080c4a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:36392/0 (socket says 172.21.15.123:36392) 2024-09-16T15:43:23.188 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:23.181+0000 7fec0c978640 1 -- 172.21.15.123:0/4117366716 learned_addr learned my addr 172.21.15.123:0/4117366716 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:43:23.188 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:23.181+0000 7fec0d179640 1 --2- 172.21.15.123:0/4117366716 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7febf4004b20 0x7fec081510b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:43:23.188 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:23.181+0000 7fec0d97a640 1 --2- 172.21.15.123:0/4117366716 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fec080b36d0 0x7fec080b3b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:43:23.188 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:23.181+0000 7fec0c978640 1 -- 172.21.15.123:0/4117366716 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7febf4004b20 msgr2=0x7fec081510b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:43:23.189 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:23.181+0000 7fec0c978640 1 --2- 172.21.15.123:0/4117366716 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7febf4004b20 0x7fec081510b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:23.189 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:23.181+0000 7fec0c978640 1 -- 172.21.15.123:0/4117366716 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fec080b36d0 msgr2=0x7fec080b3b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:43:23.189 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:23.181+0000 7fec0c978640 1 --2- 172.21.15.123:0/4117366716 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fec080b36d0 0x7fec080b3b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:23.189 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:23.181+0000 7fec0c978640 1 -- 172.21.15.123:0/4117366716 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fec081515f0 con 0x7fec080c94e0 2024-09-16T15:43:23.189 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:23.181+0000 7fec0d97a640 1 --2- 172.21.15.123:0/4117366716 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fec080b36d0 0x7fec080b3b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T15:43:23.189 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:23.181+0000 7fec0c978640 1 --2- 172.21.15.123:0/4117366716 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fec080c94e0 0x7fec080c4a70 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7febf800c700 tx=0x7febf800cbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:43:23.189 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:23.185+0000 7febfdffb640 1 -- 172.21.15.123:0/4117366716 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7febf8005480 con 0x7fec080c94e0 2024-09-16T15:43:23.189 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:23.185+0000 7febfdffb640 1 -- 172.21.15.123:0/4117366716 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7febf800ce10 con 0x7fec080c94e0 2024-09-16T15:43:23.189 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:23.185+0000 7fec0f3dd640 1 -- 172.21.15.123:0/4117366716 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fec080c4fb0 con 0x7fec080c94e0 2024-09-16T15:43:23.189 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:23.185+0000 7febfdffb640 1 -- 172.21.15.123:0/4117366716 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7febf8003860 con 0x7fec080c94e0 2024-09-16T15:43:23.189 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:23.185+0000 7fec0f3dd640 1 -- 172.21.15.123:0/4117366716 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fec080c9aa0 con 0x7fec080c94e0 2024-09-16T15:43:23.189 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:23.185+0000 7fec0f3dd640 1 -- 172.21.15.123:0/4117366716 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fec0815aa60 con 0x7fec080c94e0 2024-09-16T15:43:23.190 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:23.185+0000 7febfdffb640 1 -- 172.21.15.123:0/4117366716 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 5) ==== 97080+0+0 (secure 0 0 0) 0x7febf8003a00 con 0x7fec080c94e0 2024-09-16T15:43:23.190 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:23.185+0000 7febfdffb640 1 --2- 172.21.15.123:0/4117366716 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7febec072ec0 0x7febec075380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:43:23.190 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:23.185+0000 7febfdffb640 1 -- 172.21.15.123:0/4117366716 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7febf8093c30 con 0x7fec080c94e0 2024-09-16T15:43:23.190 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:23.185+0000 7fec0d179640 1 --2- 172.21.15.123:0/4117366716 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7febec072ec0 0x7febec075380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:43:23.191 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:23.185+0000 7fec0d179640 1 --2- 172.21.15.123:0/4117366716 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7febec072ec0 0x7febec075380 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fec040b0000 tx=0x7fec04019b80 comp rx=0 tx=0).ready entity=mgr.4135 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:43:23.193 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:23.189+0000 7febfdffb640 1 -- 172.21.15.123:0/4117366716 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7febf805ee80 con 0x7fec080c94e0 2024-09-16T15:43:23.388 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:23.385+0000 7fec0f3dd640 1 -- 172.21.15.123:0/4117366716 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fec080d06c0 con 0x7fec080c94e0 2024-09-16T15:43:23.389 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:23.385+0000 7febfdffb640 1 -- 172.21.15.123:0/4117366716 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) ==== 93+0+5226 (secure 0 0 0) 0x7febf805e780 con 0x7fec080c94e0 2024-09-16T15:43:23.389 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-09-16T15:43:23.390 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":8,"btime":"2024-09-16T15:42:54:421548+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4280,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6835/1497621896","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":1497621896},{"type":"v1","addr":"172.21.15.184:6835","nonce":1497621896}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4305,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6833/2227496057","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6832","nonce":2227496057},{"type":"v1","addr":"172.21.15.123:6833","nonce":2227496057}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4311,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6835/294455998","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6834","nonce":294455998},{"type":"v1","addr":"172.21.15.123:6835","nonce":294455998}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-16T15:42:50.762280+0000","modified":"2024-09-16T15:42:53.789989+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4348},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4348":{"gid":4348,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.184:6837/1469324095","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6836","nonce":1469324095},{"type":"v1","addr":"172.21.15.184:6837","nonce":1469324095}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4348,"qdb_cluster":[4348]},"id":1}]} 2024-09-16T15:43:23.390 INFO:teuthology.orchestra.run.smithi123.stderr:dumped fsmap epoch 8 2024-09-16T15:43:23.392 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:23.389+0000 7fec0f3dd640 1 -- 172.21.15.123:0/4117366716 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7febec072ec0 msgr2=0x7febec075380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:43:23.392 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:23.389+0000 7fec0f3dd640 1 --2- 172.21.15.123:0/4117366716 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7febec072ec0 0x7febec075380 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fec040b0000 tx=0x7fec04019b80 comp rx=0 tx=0).stop 2024-09-16T15:43:23.392 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:23.389+0000 7fec0f3dd640 1 -- 172.21.15.123:0/4117366716 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fec080c94e0 msgr2=0x7fec080c4a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:43:23.392 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:23.389+0000 7fec0f3dd640 1 --2- 172.21.15.123:0/4117366716 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fec080c94e0 0x7fec080c4a70 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7febf800c700 tx=0x7febf800cbc0 comp rx=0 tx=0).stop 2024-09-16T15:43:23.392 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:23.389+0000 7fec0f3dd640 1 -- 172.21.15.123:0/4117366716 shutdown_connections 2024-09-16T15:43:23.393 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:23.389+0000 7fec0f3dd640 1 --2- 172.21.15.123:0/4117366716 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7febec072ec0 0x7febec075380 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:23.393 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:23.389+0000 7fec0f3dd640 1 --2- 172.21.15.123:0/4117366716 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fec080b36d0 0x7fec080b3b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:23.393 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:23.389+0000 7fec0f3dd640 1 --2- 172.21.15.123:0/4117366716 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fec080c94e0 0x7fec080c4a70 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:23.393 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:23.389+0000 7fec0f3dd640 1 --2- 172.21.15.123:0/4117366716 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7febf4004b20 0x7fec081510b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:23.393 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:23.389+0000 7fec0f3dd640 1 -- 172.21.15.123:0/4117366716 >> 172.21.15.123:0/4117366716 conn(0x7fec080cfb10 msgr2=0x7fec08073bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:43:23.393 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:23.389+0000 7fec0f3dd640 1 -- 172.21.15.123:0/4117366716 shutdown_connections 2024-09-16T15:43:23.393 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:23.389+0000 7fec0f3dd640 1 -- 172.21.15.123:0/4117366716 wait complete. 2024-09-16T15:43:23.408 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-16T15:43:23.532 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:23.529+0000 7f3960253640 1 -- 172.21.15.123:0/3854722963 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3944004810 msgr2=0x7f3944004c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:43:23.532 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:23.529+0000 7f3960253640 1 --2- 172.21.15.123:0/3854722963 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3944004810 0x7f3944004c90 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f3948009b60 tx=0x7f394801d240 comp rx=0 tx=0).stop 2024-09-16T15:43:23.533 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:23.529+0000 7f3960253640 1 -- 172.21.15.123:0/3854722963 shutdown_connections 2024-09-16T15:43:23.533 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:23.529+0000 7f3960253640 1 --2- 172.21.15.123:0/3854722963 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3944004810 0x7f3944004c90 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:23.533 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:23.529+0000 7f3960253640 1 -- 172.21.15.123:0/3854722963 >> 172.21.15.123:0/3854722963 conn(0x7f39580cfb10 msgr2=0x7f39580cff20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:43:23.533 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:23.529+0000 7f3960253640 1 -- 172.21.15.123:0/3854722963 shutdown_connections 2024-09-16T15:43:23.533 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:23.529+0000 7f3960253640 1 -- 172.21.15.123:0/3854722963 wait complete. 2024-09-16T15:43:23.533 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:23.529+0000 7f3960253640 1 Processor -- start 2024-09-16T15:43:23.533 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:23.529+0000 7f3960253640 1 -- start start 2024-09-16T15:43:23.535 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:23.529+0000 7f3960253640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3944004810 0x7f39581f7140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:43:23.535 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:23.529+0000 7f3960253640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f39581fa830 0x7f3958153ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:43:23.536 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:23.529+0000 7f3960253640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f3958159290 0x7f39580b29a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:43:23.536 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:23.529+0000 7f3960253640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f39580cab80 con 0x7f3944004810 2024-09-16T15:43:23.536 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:23.529+0000 7f3960253640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f3958074480 con 0x7f39581fa830 2024-09-16T15:43:23.536 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:23.529+0000 7f3960253640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f39580cad00 con 0x7f3958159290 2024-09-16T15:43:23.536 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:23.533+0000 7f395dfef640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3944004810 0x7f39581f7140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:43:23.536 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:23.533+0000 7f395e7f0640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f3958159290 0x7f39580b29a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:43:23.536 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:23.533+0000 7f395dfef640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3944004810 0x7f39581f7140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:36406/0 (socket says 172.21.15.123:36406) 2024-09-16T15:43:23.536 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:23.533+0000 7f395dfef640 1 -- 172.21.15.123:0/731267567 learned_addr learned my addr 172.21.15.123:0/731267567 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:43:23.536 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:23.533+0000 7f395d7ee640 1 --2- 172.21.15.123:0/731267567 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f39581fa830 0x7f3958153ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:43:23.536 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:23.533+0000 7f395dfef640 1 -- 172.21.15.123:0/731267567 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f3958159290 msgr2=0x7f39580b29a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:43:23.536 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:23.533+0000 7f395dfef640 1 --2- 172.21.15.123:0/731267567 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f3958159290 0x7f39580b29a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:23.537 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:23.533+0000 7f395dfef640 1 -- 172.21.15.123:0/731267567 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f39581fa830 msgr2=0x7f3958153ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:43:23.537 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:23.533+0000 7f395dfef640 1 --2- 172.21.15.123:0/731267567 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f39581fa830 0x7f3958153ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:23.537 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:23.533+0000 7f395dfef640 1 -- 172.21.15.123:0/731267567 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3958075030 con 0x7f3944004810 2024-09-16T15:43:23.537 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:23.533+0000 7f395dfef640 1 --2- 172.21.15.123:0/731267567 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3944004810 0x7f39581f7140 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f395400d810 tx=0x7f395400dcd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:43:23.537 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:23.533+0000 7f394e7fc640 1 -- 172.21.15.123:0/731267567 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3954016020 con 0x7f3944004810 2024-09-16T15:43:23.537 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:23.533+0000 7f394e7fc640 1 -- 172.21.15.123:0/731267567 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f395400b6c0 con 0x7f3944004810 2024-09-16T15:43:23.537 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:23.533+0000 7f3960253640 1 -- 172.21.15.123:0/731267567 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f39580b5990 con 0x7f3944004810 2024-09-16T15:43:23.537 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:23.533+0000 7f394e7fc640 1 -- 172.21.15.123:0/731267567 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f395400baf0 con 0x7f3944004810 2024-09-16T15:43:23.537 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:23.533+0000 7f3960253640 1 -- 172.21.15.123:0/731267567 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3958159c00 con 0x7f3944004810 2024-09-16T15:43:23.538 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:23.533+0000 7f3960253640 1 -- 172.21.15.123:0/731267567 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f395806e8c0 con 0x7f3944004810 2024-09-16T15:43:23.538 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:23.533+0000 7f394e7fc640 1 -- 172.21.15.123:0/731267567 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 5) ==== 97080+0+0 (secure 0 0 0) 0x7f3954019020 con 0x7f3944004810 2024-09-16T15:43:23.538 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:23.533+0000 7f394e7fc640 1 --2- 172.21.15.123:0/731267567 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f393c072ec0 0x7f393c075380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:43:23.538 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:23.533+0000 7f394e7fc640 1 -- 172.21.15.123:0/731267567 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f3954093f50 con 0x7f3944004810 2024-09-16T15:43:23.538 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:23.533+0000 7f395d7ee640 1 --2- 172.21.15.123:0/731267567 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f393c072ec0 0x7f393c075380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:43:23.539 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:23.537+0000 7f395d7ee640 1 --2- 172.21.15.123:0/731267567 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f393c072ec0 0x7f393c075380 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f39480ae000 tx=0x7f3948019fe0 comp rx=0 tx=0).ready entity=mgr.4135 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:43:23.541 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:23.537+0000 7f394e7fc640 1 -- 172.21.15.123:0/731267567 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f3954020020 con 0x7f3944004810 2024-09-16T15:43:23.703 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:23.701+0000 7f3960253640 1 -- 172.21.15.123:0/731267567 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f39580700e0 con 0x7f3944004810 2024-09-16T15:43:23.705 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:23.701+0000 7f394e7fc640 1 -- 172.21.15.123:0/731267567 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) ==== 74+0+16725 (secure 0 0 0) 0x7f395405ec30 con 0x7f3944004810 2024-09-16T15:43:23.706 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-09-16T15:43:23.706 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":21,"fsid":"56077f89-18c8-486f-ae7e-eecc0bb7822c","created":"2024-09-16T15:42:26.364892+0000","modified":"2024-09-16T15:42:52.401147+0000","last_up_change":"2024-09-16T15:42:38.707234+0000","last_in_change":"2024-09-16T15:42:29.423662+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-16T15:42:39.287764+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-09-16T15:42:48.583563+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-09-16T15:42:49.704665+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9e7c1955-a607-4671-9be7-f2ef760ceeaa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6800","nonce":3882044969},{"type":"v1","addr":"172.21.15.123:6801","nonce":3882044969}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6802","nonce":3882044969},{"type":"v1","addr":"172.21.15.123:6803","nonce":3882044969}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6806","nonce":3882044969},{"type":"v1","addr":"172.21.15.123:6807","nonce":3882044969}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6804","nonce":3882044969},{"type":"v1","addr":"172.21.15.123:6805","nonce":3882044969}]},"public_addr":"172.21.15.123:6801/3882044969","cluster_addr":"172.21.15.123:6803/3882044969","heartbeat_back_addr":"172.21.15.123:6807/3882044969","heartbeat_front_addr":"172.21.15.123:6805/3882044969","state":["exists","up"]},{"osd":1,"uuid":"c540cfde-0977-4742-9b17-fd634668e010","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6816","nonce":4037001850},{"type":"v1","addr":"172.21.15.123:6817","nonce":4037001850}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6818","nonce":4037001850},{"type":"v1","addr":"172.21.15.123:6819","nonce":4037001850}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6822","nonce":4037001850},{"type":"v1","addr":"172.21.15.123:6823","nonce":4037001850}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6820","nonce":4037001850},{"type":"v1","addr":"172.21.15.123:6821","nonce":4037001850}]},"public_addr":"172.21.15.123:6817/4037001850","cluster_addr":"172.21.15.123:6819/4037001850","heartbeat_back_addr":"172.21.15.123:6823/4037001850","heartbeat_front_addr":"172.21.15.123:6821/4037001850","state":["exists","up"]},{"osd":2,"uuid":"9864a496-f169-486b-95c4-1d59162793a7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6824","nonce":1332143088},{"type":"v1","addr":"172.21.15.123:6825","nonce":1332143088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6826","nonce":1332143088},{"type":"v1","addr":"172.21.15.123:6827","nonce":1332143088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6830","nonce":1332143088},{"type":"v1","addr":"172.21.15.123:6831","nonce":1332143088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6828","nonce":1332143088},{"type":"v1","addr":"172.21.15.123:6829","nonce":1332143088}]},"public_addr":"172.21.15.123:6825/1332143088","cluster_addr":"172.21.15.123:6827/1332143088","heartbeat_back_addr":"172.21.15.123:6831/1332143088","heartbeat_front_addr":"172.21.15.123:6829/1332143088","state":["exists","up"]},{"osd":3,"uuid":"c6d1f817-f074-4325-95e1-578eba13edda","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6808","nonce":988586998},{"type":"v1","addr":"172.21.15.123:6809","nonce":988586998}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6810","nonce":988586998},{"type":"v1","addr":"172.21.15.123:6811","nonce":988586998}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6814","nonce":988586998},{"type":"v1","addr":"172.21.15.123:6815","nonce":988586998}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6812","nonce":988586998},{"type":"v1","addr":"172.21.15.123:6813","nonce":988586998}]},"public_addr":"172.21.15.123:6809/988586998","cluster_addr":"172.21.15.123:6811/988586998","heartbeat_back_addr":"172.21.15.123:6815/988586998","heartbeat_front_addr":"172.21.15.123:6813/988586998","state":["exists","up"]},{"osd":4,"uuid":"200a8df4-50ef-40a5-b48a-58981269c086","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6816","nonce":3387813890},{"type":"v1","addr":"172.21.15.184:6817","nonce":3387813890}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6818","nonce":3387813890},{"type":"v1","addr":"172.21.15.184:6819","nonce":3387813890}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6822","nonce":3387813890},{"type":"v1","addr":"172.21.15.184:6823","nonce":3387813890}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6820","nonce":3387813890},{"type":"v1","addr":"172.21.15.184:6821","nonce":3387813890}]},"public_addr":"172.21.15.184:6817/3387813890","cluster_addr":"172.21.15.184:6819/3387813890","heartbeat_back_addr":"172.21.15.184:6823/3387813890","heartbeat_front_addr":"172.21.15.184:6821/3387813890","state":["exists","up"]},{"osd":5,"uuid":"32d5a900-da3d-412c-bc19-13861a2368d8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6808","nonce":2680916017},{"type":"v1","addr":"172.21.15.184:6809","nonce":2680916017}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6810","nonce":2680916017},{"type":"v1","addr":"172.21.15.184:6811","nonce":2680916017}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6814","nonce":2680916017},{"type":"v1","addr":"172.21.15.184:6815","nonce":2680916017}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6812","nonce":2680916017},{"type":"v1","addr":"172.21.15.184:6813","nonce":2680916017}]},"public_addr":"172.21.15.184:6809/2680916017","cluster_addr":"172.21.15.184:6811/2680916017","heartbeat_back_addr":"172.21.15.184:6815/2680916017","heartbeat_front_addr":"172.21.15.184:6813/2680916017","state":["exists","up"]},{"osd":6,"uuid":"cde6f39e-aa1b-4260-ac19-774ee6aaca72","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6824","nonce":2183120457},{"type":"v1","addr":"172.21.15.184:6825","nonce":2183120457}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6826","nonce":2183120457},{"type":"v1","addr":"172.21.15.184:6827","nonce":2183120457}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6830","nonce":2183120457},{"type":"v1","addr":"172.21.15.184:6831","nonce":2183120457}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6828","nonce":2183120457},{"type":"v1","addr":"172.21.15.184:6829","nonce":2183120457}]},"public_addr":"172.21.15.184:6825/2183120457","cluster_addr":"172.21.15.184:6827/2183120457","heartbeat_back_addr":"172.21.15.184:6831/2183120457","heartbeat_front_addr":"172.21.15.184:6829/2183120457","state":["exists","up"]},{"osd":7,"uuid":"4f381778-eee1-47c6-804f-df0e6b33e313","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6800","nonce":4102957285},{"type":"v1","addr":"172.21.15.184:6801","nonce":4102957285}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6802","nonce":4102957285},{"type":"v1","addr":"172.21.15.184:6803","nonce":4102957285}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6806","nonce":4102957285},{"type":"v1","addr":"172.21.15.184:6807","nonce":4102957285}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6804","nonce":4102957285},{"type":"v1","addr":"172.21.15.184:6805","nonce":4102957285}]},"public_addr":"172.21.15.184:6801/4102957285","cluster_addr":"172.21.15.184:6803/4102957285","heartbeat_back_addr":"172.21.15.184:6807/4102957285","heartbeat_front_addr":"172.21.15.184:6805/4102957285","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:35.972210+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.222007+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.198617+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.385663+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.358935+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.156066+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.752973+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:35.682846+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T15:43:23.708 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:23.705+0000 7f3960253640 1 -- 172.21.15.123:0/731267567 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f393c072ec0 msgr2=0x7f393c075380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:43:23.708 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:23.705+0000 7f3960253640 1 --2- 172.21.15.123:0/731267567 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f393c072ec0 0x7f393c075380 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f39480ae000 tx=0x7f3948019fe0 comp rx=0 tx=0).stop 2024-09-16T15:43:23.708 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:23.705+0000 7f3960253640 1 -- 172.21.15.123:0/731267567 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3944004810 msgr2=0x7f39581f7140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:43:23.709 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:23.705+0000 7f3960253640 1 --2- 172.21.15.123:0/731267567 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3944004810 0x7f39581f7140 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f395400d810 tx=0x7f395400dcd0 comp rx=0 tx=0).stop 2024-09-16T15:43:23.709 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:23.705+0000 7f3960253640 1 -- 172.21.15.123:0/731267567 shutdown_connections 2024-09-16T15:43:23.709 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:23.705+0000 7f3960253640 1 --2- 172.21.15.123:0/731267567 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f393c072ec0 0x7f393c075380 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:23.709 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:23.705+0000 7f3960253640 1 --2- 172.21.15.123:0/731267567 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f3958159290 0x7f39580b29a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:23.709 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:23.705+0000 7f3960253640 1 --2- 172.21.15.123:0/731267567 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f39581fa830 0x7f3958153ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:23.709 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:23.705+0000 7f3960253640 1 --2- 172.21.15.123:0/731267567 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3944004810 0x7f39581f7140 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:23.709 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:23.705+0000 7f3960253640 1 -- 172.21.15.123:0/731267567 >> 172.21.15.123:0/731267567 conn(0x7f39580cfb10 msgr2=0x7f39580c3fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:43:23.709 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:23.705+0000 7f3960253640 1 -- 172.21.15.123:0/731267567 shutdown_connections 2024-09-16T15:43:23.709 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:23.705+0000 7f3960253640 1 -- 172.21.15.123:0/731267567 wait complete. 2024-09-16T15:43:23.724 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-09-16T15:43:23.724 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-16T15:43:23.836 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:23.833+0000 7fb262bb6640 1 -- 172.21.15.123:0/718702774 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb248004810 msgr2=0x7fb248004c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:43:23.836 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:23.833+0000 7fb262bb6640 1 --2- 172.21.15.123:0/718702774 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb248004810 0x7fb248004c90 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7fb24400bbc0 tx=0x7fb24401ce30 comp rx=0 tx=0).stop 2024-09-16T15:43:23.836 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:23.833+0000 7fb261153640 1 -- 172.21.15.123:0/718702774 reap_dead start 2024-09-16T15:43:23.836 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:23.833+0000 7fb262bb6640 1 -- 172.21.15.123:0/718702774 shutdown_connections 2024-09-16T15:43:23.836 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:23.833+0000 7fb262bb6640 1 -- 172.21.15.123:0/718702774 >> 172.21.15.123:0/718702774 conn(0x7fb25c0cfb10 msgr2=0x7fb25c0cff20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:43:23.836 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:23.833+0000 7fb262bb6640 1 -- 172.21.15.123:0/718702774 shutdown_connections 2024-09-16T15:43:23.836 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:23.833+0000 7fb262bb6640 1 -- 172.21.15.123:0/718702774 wait complete. 2024-09-16T15:43:23.839 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:23.833+0000 7fb262bb6640 1 Processor -- start 2024-09-16T15:43:23.839 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:23.833+0000 7fb262bb6640 1 -- start start 2024-09-16T15:43:23.839 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:23.833+0000 7fb262bb6640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb248004810 0x7fb25c1fbe00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:43:23.839 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:23.833+0000 7fb262bb6640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fb25c14d4c0 0x7fb25c14a040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:43:23.839 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:23.833+0000 7fb262bb6640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb25c14a9a0 0x7fb25c14ae20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:43:23.839 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:23.833+0000 7fb262bb6640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fb25c06f1f0 con 0x7fb25c14a9a0 2024-09-16T15:43:23.839 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:23.833+0000 7fb262bb6640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fb25c06f070 con 0x7fb248004810 2024-09-16T15:43:23.839 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:23.833+0000 7fb262bb6640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fb25c06f370 con 0x7fb25c14d4c0 2024-09-16T15:43:23.839 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:23.833+0000 7fb261153640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb25c14a9a0 0x7fb25c14ae20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:43:23.839 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:23.833+0000 7fb261153640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb25c14a9a0 0x7fb25c14ae20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:36416/0 (socket says 172.21.15.123:36416) 2024-09-16T15:43:23.839 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:23.833+0000 7fb261153640 1 -- 172.21.15.123:0/3863485446 learned_addr learned my addr 172.21.15.123:0/3863485446 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:43:23.839 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:23.833+0000 7fb260952640 1 --2- 172.21.15.123:0/3863485446 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb248004810 0x7fb25c1fbe00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:43:23.840 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:23.833+0000 7fb25bfff640 1 --2- 172.21.15.123:0/3863485446 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fb25c14d4c0 0x7fb25c14a040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:43:23.840 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:23.833+0000 7fb261153640 1 -- 172.21.15.123:0/3863485446 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fb25c14d4c0 msgr2=0x7fb25c14a040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:43:23.840 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:23.833+0000 7fb261153640 1 --2- 172.21.15.123:0/3863485446 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fb25c14d4c0 0x7fb25c14a040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:23.840 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:23.833+0000 7fb261153640 1 -- 172.21.15.123:0/3863485446 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb248004810 msgr2=0x7fb25c1fbe00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:43:23.840 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:23.833+0000 7fb261153640 1 --2- 172.21.15.123:0/3863485446 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb248004810 0x7fb25c1fbe00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:23.840 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:23.833+0000 7fb261153640 1 -- 172.21.15.123:0/3863485446 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb25c163de0 con 0x7fb25c14a9a0 2024-09-16T15:43:23.840 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:23.833+0000 7fb261153640 1 --2- 172.21.15.123:0/3863485446 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb25c14a9a0 0x7fb25c14ae20 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7fb254001bf0 tx=0x7fb25400dc50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:43:23.840 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:23.833+0000 7fb2597fa640 1 -- 172.21.15.123:0/3863485446 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb254018070 con 0x7fb25c14a9a0 2024-09-16T15:43:23.840 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:23.833+0000 7fb2597fa640 1 -- 172.21.15.123:0/3863485446 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fb25400bac0 con 0x7fb25c14a9a0 2024-09-16T15:43:23.840 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:23.833+0000 7fb262bb6640 1 -- 172.21.15.123:0/3863485446 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb25c1640c0 con 0x7fb25c14a9a0 2024-09-16T15:43:23.840 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:23.833+0000 7fb2597fa640 1 -- 172.21.15.123:0/3863485446 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb25400dea0 con 0x7fb25c14a9a0 2024-09-16T15:43:23.841 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:23.833+0000 7fb262bb6640 1 -- 172.21.15.123:0/3863485446 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb25c164680 con 0x7fb25c14a9a0 2024-09-16T15:43:23.841 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:23.837+0000 7fb262bb6640 1 -- 172.21.15.123:0/3863485446 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb25c164850 con 0x7fb25c14a9a0 2024-09-16T15:43:23.842 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:23.837+0000 7fb2597fa640 1 -- 172.21.15.123:0/3863485446 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 5) ==== 97080+0+0 (secure 0 0 0) 0x7fb25401b070 con 0x7fb25c14a9a0 2024-09-16T15:43:23.842 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:23.837+0000 7fb2597fa640 1 --2- 172.21.15.123:0/3863485446 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fb23c072ef0 0x7fb23c0753b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:43:23.842 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:23.837+0000 7fb2597fa640 1 -- 172.21.15.123:0/3863485446 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7fb254093550 con 0x7fb25c14a9a0 2024-09-16T15:43:23.842 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:23.837+0000 7fb260952640 1 --2- 172.21.15.123:0/3863485446 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fb23c072ef0 0x7fb23c0753b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:43:23.842 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:23.837+0000 7fb260952640 1 --2- 172.21.15.123:0/3863485446 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fb23c072ef0 0x7fb23c0753b0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fb24c009910 tx=0x7fb24c0091e0 comp rx=0 tx=0).ready entity=mgr.4135 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:43:23.845 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:23.841+0000 7fb2597fa640 1 -- 172.21.15.123:0/3863485446 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7fb25405e7b0 con 0x7fb25c14a9a0 2024-09-16T15:43:24.037 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:24.033+0000 7fb262bb6640 1 -- 172.21.15.123:0/3863485446 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb25c0c3350 con 0x7fb25c14a9a0 2024-09-16T15:43:24.038 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:24.033+0000 7fb2597fa640 1 -- 172.21.15.123:0/3863485446 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) ==== 93+0+5226 (secure 0 0 0) 0x7fb25405e0b0 con 0x7fb25c14a9a0 2024-09-16T15:43:24.038 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-09-16T15:43:24.038 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":8,"btime":"2024-09-16T15:42:54:421548+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4280,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6835/1497621896","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":1497621896},{"type":"v1","addr":"172.21.15.184:6835","nonce":1497621896}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4305,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6833/2227496057","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6832","nonce":2227496057},{"type":"v1","addr":"172.21.15.123:6833","nonce":2227496057}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4311,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6835/294455998","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6834","nonce":294455998},{"type":"v1","addr":"172.21.15.123:6835","nonce":294455998}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-16T15:42:50.762280+0000","modified":"2024-09-16T15:42:53.789989+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4348},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4348":{"gid":4348,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.184:6837/1469324095","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6836","nonce":1469324095},{"type":"v1","addr":"172.21.15.184:6837","nonce":1469324095}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4348,"qdb_cluster":[4348]},"id":1}]} 2024-09-16T15:43:24.039 INFO:teuthology.orchestra.run.smithi123.stderr:dumped fsmap epoch 8 2024-09-16T15:43:24.041 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:24.037+0000 7fb262bb6640 1 -- 172.21.15.123:0/3863485446 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fb23c072ef0 msgr2=0x7fb23c0753b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:43:24.041 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:24.037+0000 7fb262bb6640 1 --2- 172.21.15.123:0/3863485446 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fb23c072ef0 0x7fb23c0753b0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fb24c009910 tx=0x7fb24c0091e0 comp rx=0 tx=0).stop 2024-09-16T15:43:24.041 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:24.037+0000 7fb262bb6640 1 -- 172.21.15.123:0/3863485446 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb25c14a9a0 msgr2=0x7fb25c14ae20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:43:24.041 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:24.037+0000 7fb262bb6640 1 --2- 172.21.15.123:0/3863485446 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb25c14a9a0 0x7fb25c14ae20 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7fb254001bf0 tx=0x7fb25400dc50 comp rx=0 tx=0).stop 2024-09-16T15:43:24.041 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:24.037+0000 7fb262bb6640 1 -- 172.21.15.123:0/3863485446 shutdown_connections 2024-09-16T15:43:24.041 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:24.037+0000 7fb262bb6640 1 --2- 172.21.15.123:0/3863485446 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fb23c072ef0 0x7fb23c0753b0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:24.042 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:24.037+0000 7fb262bb6640 1 --2- 172.21.15.123:0/3863485446 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb25c14a9a0 0x7fb25c14ae20 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:24.042 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:24.037+0000 7fb262bb6640 1 --2- 172.21.15.123:0/3863485446 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fb25c14d4c0 0x7fb25c14a040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:24.042 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:24.037+0000 7fb262bb6640 1 --2- 172.21.15.123:0/3863485446 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb248004810 0x7fb25c1fbe00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:24.042 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:24.037+0000 7fb262bb6640 1 -- 172.21.15.123:0/3863485446 >> 172.21.15.123:0/3863485446 conn(0x7fb25c0cfb10 msgr2=0x7fb25c0cc450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:43:24.042 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:24.037+0000 7fb262bb6640 1 -- 172.21.15.123:0/3863485446 shutdown_connections 2024-09-16T15:43:24.042 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:24.037+0000 7fb262bb6640 1 -- 172.21.15.123:0/3863485446 wait complete. 2024-09-16T15:43:24.056 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-16T15:43:24.170 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:24.165+0000 7fdc99008640 1 -- 172.21.15.123:0/4024535011 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdc940726c0 msgr2=0x7fdc940d0630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:43:24.170 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:24.165+0000 7fdc99008640 1 --2- 172.21.15.123:0/4024535011 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdc940726c0 0x7fdc940d0630 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7fdc9406c970 tx=0x7fdc8801c800 comp rx=0 tx=0).stop 2024-09-16T15:43:24.171 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:24.165+0000 7fdc99008640 1 -- 172.21.15.123:0/4024535011 shutdown_connections 2024-09-16T15:43:24.171 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:24.165+0000 7fdc99008640 1 --2- 172.21.15.123:0/4024535011 >> v2:172.21.15.184:3300/0 conn(0x7fdc94074600 0x7fdc9406e750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:24.171 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:24.165+0000 7fdc99008640 1 --2- 172.21.15.123:0/4024535011 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdc940726c0 0x7fdc940d0630 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:24.171 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:24.165+0000 7fdc99008640 1 -- 172.21.15.123:0/4024535011 >> 172.21.15.123:0/4024535011 conn(0x7fdc940cfb10 msgr2=0x7fdc940cff20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:43:24.171 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:24.165+0000 7fdc99008640 1 -- 172.21.15.123:0/4024535011 shutdown_connections 2024-09-16T15:43:24.171 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:24.165+0000 7fdc99008640 1 -- 172.21.15.123:0/4024535011 wait complete. 2024-09-16T15:43:24.173 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:24.169+0000 7fdc99008640 1 Processor -- start 2024-09-16T15:43:24.173 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:24.169+0000 7fdc99008640 1 -- start start 2024-09-16T15:43:24.173 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:24.169+0000 7fdc99008640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdc940726c0 0x7fdc940b3810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:43:24.174 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:24.169+0000 7fdc99008640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fdc94074600 0x7fdc940c76f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:43:24.174 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:24.169+0000 7fdc99008640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fdc9415bca0 0x7fdc94154300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:43:24.174 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:24.169+0000 7fdc99008640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fdc94168bd0 con 0x7fdc940726c0 2024-09-16T15:43:24.174 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:24.169+0000 7fdc99008640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fdc94168a50 con 0x7fdc94074600 2024-09-16T15:43:24.174 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:24.169+0000 7fdc99008640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fdc94168d50 con 0x7fdc9415bca0 2024-09-16T15:43:24.174 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:24.169+0000 7fdc92d9d640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdc940726c0 0x7fdc940b3810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:43:24.174 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:24.169+0000 7fdc92d9d640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdc940726c0 0x7fdc940b3810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:36426/0 (socket says 172.21.15.123:36426) 2024-09-16T15:43:24.174 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:24.169+0000 7fdc92d9d640 1 -- 172.21.15.123:0/1346480883 learned_addr learned my addr 172.21.15.123:0/1346480883 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:43:24.174 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:24.169+0000 7fdc9259c640 1 --2- 172.21.15.123:0/1346480883 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fdc94074600 0x7fdc940c76f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:43:24.174 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:24.169+0000 7fdc9359e640 1 --2- 172.21.15.123:0/1346480883 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fdc9415bca0 0x7fdc94154300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:43:24.174 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:24.169+0000 7fdc92d9d640 1 -- 172.21.15.123:0/1346480883 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fdc9415bca0 msgr2=0x7fdc94154300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:43:24.174 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:24.169+0000 7fdc92d9d640 1 --2- 172.21.15.123:0/1346480883 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fdc9415bca0 0x7fdc94154300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:24.175 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:24.169+0000 7fdc92d9d640 1 -- 172.21.15.123:0/1346480883 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fdc94074600 msgr2=0x7fdc940c76f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:43:24.175 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:24.169+0000 7fdc92d9d640 1 --2- 172.21.15.123:0/1346480883 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fdc94074600 0x7fdc940c76f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:24.175 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:24.169+0000 7fdc92d9d640 1 -- 172.21.15.123:0/1346480883 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdc940b3d50 con 0x7fdc940726c0 2024-09-16T15:43:24.175 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:24.169+0000 7fdc9359e640 1 --2- 172.21.15.123:0/1346480883 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fdc9415bca0 0x7fdc94154300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T15:43:24.175 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:24.169+0000 7fdc92d9d640 1 --2- 172.21.15.123:0/1346480883 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdc940726c0 0x7fdc940b3810 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7fdc8c00ba80 tx=0x7fdc8c00bf40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:43:24.175 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:24.169+0000 7fdc7b7fe640 1 -- 172.21.15.123:0/1346480883 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc8c019070 con 0x7fdc940726c0 2024-09-16T15:43:24.175 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:24.169+0000 7fdc7b7fe640 1 -- 172.21.15.123:0/1346480883 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fdc8c00cd80 con 0x7fdc940726c0 2024-09-16T15:43:24.175 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:24.169+0000 7fdc99008640 1 -- 172.21.15.123:0/1346480883 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdc940c7c30 con 0x7fdc940726c0 2024-09-16T15:43:24.175 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:24.169+0000 7fdc7b7fe640 1 -- 172.21.15.123:0/1346480883 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc8c0146a0 con 0x7fdc940726c0 2024-09-16T15:43:24.175 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:24.169+0000 7fdc99008640 1 -- 172.21.15.123:0/1346480883 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdc9415cba0 con 0x7fdc940726c0 2024-09-16T15:43:24.175 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:24.169+0000 7fdc99008640 1 -- 172.21.15.123:0/1346480883 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdc940d0630 con 0x7fdc940726c0 2024-09-16T15:43:24.176 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:24.169+0000 7fdc7b7fe640 1 -- 172.21.15.123:0/1346480883 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 5) ==== 97080+0+0 (secure 0 0 0) 0x7fdc8c01d3f0 con 0x7fdc940726c0 2024-09-16T15:43:24.176 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:24.173+0000 7fdc7b7fe640 1 --2- 172.21.15.123:0/1346480883 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fdc80072ef0 0x7fdc800753b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:43:24.176 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:24.173+0000 7fdc7b7fe640 1 -- 172.21.15.123:0/1346480883 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7fdc8c093b10 con 0x7fdc940726c0 2024-09-16T15:43:24.176 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:24.173+0000 7fdc9259c640 1 --2- 172.21.15.123:0/1346480883 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fdc80072ef0 0x7fdc800753b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:43:24.177 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:24.173+0000 7fdc9259c640 1 --2- 172.21.15.123:0/1346480883 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fdc80072ef0 0x7fdc800753b0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fdc880af000 tx=0x7fdc8801b040 comp rx=0 tx=0).ready entity=mgr.4135 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:43:24.179 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:24.177+0000 7fdc7b7fe640 1 -- 172.21.15.123:0/1346480883 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7fdc940b3d50 con 0x7fdc940726c0 2024-09-16T15:43:24.383 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:24.381+0000 7fdc99008640 1 -- 172.21.15.123:0/1346480883 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fdc9415f630 con 0x7fdc940726c0 2024-09-16T15:43:24.383 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:24.381+0000 7fdc7b7fe640 1 -- 172.21.15.123:0/1346480883 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) ==== 93+0+5226 (secure 0 0 0) 0x7fdc8c05ed60 con 0x7fdc940726c0 2024-09-16T15:43:24.384 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-09-16T15:43:24.384 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":8,"btime":"2024-09-16T15:42:54:421548+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4280,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6835/1497621896","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":1497621896},{"type":"v1","addr":"172.21.15.184:6835","nonce":1497621896}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4305,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6833/2227496057","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6832","nonce":2227496057},{"type":"v1","addr":"172.21.15.123:6833","nonce":2227496057}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4311,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6835/294455998","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6834","nonce":294455998},{"type":"v1","addr":"172.21.15.123:6835","nonce":294455998}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-16T15:42:50.762280+0000","modified":"2024-09-16T15:42:53.789989+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4348},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4348":{"gid":4348,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.184:6837/1469324095","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6836","nonce":1469324095},{"type":"v1","addr":"172.21.15.184:6837","nonce":1469324095}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4348,"qdb_cluster":[4348]},"id":1}]} 2024-09-16T15:43:24.384 INFO:teuthology.orchestra.run.smithi123.stderr:dumped fsmap epoch 8 2024-09-16T15:43:24.386 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:24.381+0000 7fdc99008640 1 -- 172.21.15.123:0/1346480883 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fdc80072ef0 msgr2=0x7fdc800753b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:43:24.386 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:24.381+0000 7fdc99008640 1 --2- 172.21.15.123:0/1346480883 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fdc80072ef0 0x7fdc800753b0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fdc880af000 tx=0x7fdc8801b040 comp rx=0 tx=0).stop 2024-09-16T15:43:24.386 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:24.381+0000 7fdc99008640 1 -- 172.21.15.123:0/1346480883 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdc940726c0 msgr2=0x7fdc940b3810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:43:24.387 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:24.381+0000 7fdc99008640 1 --2- 172.21.15.123:0/1346480883 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdc940726c0 0x7fdc940b3810 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7fdc8c00ba80 tx=0x7fdc8c00bf40 comp rx=0 tx=0).stop 2024-09-16T15:43:24.387 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:24.381+0000 7fdc99008640 1 -- 172.21.15.123:0/1346480883 shutdown_connections 2024-09-16T15:43:24.387 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:24.381+0000 7fdc99008640 1 --2- 172.21.15.123:0/1346480883 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fdc80072ef0 0x7fdc800753b0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:24.387 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:24.381+0000 7fdc99008640 1 --2- 172.21.15.123:0/1346480883 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fdc9415bca0 0x7fdc94154300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:24.387 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:24.381+0000 7fdc99008640 1 --2- 172.21.15.123:0/1346480883 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fdc94074600 0x7fdc940c76f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:24.387 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:24.381+0000 7fdc99008640 1 --2- 172.21.15.123:0/1346480883 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdc940726c0 0x7fdc940b3810 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:24.387 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:24.381+0000 7fdc99008640 1 -- 172.21.15.123:0/1346480883 >> 172.21.15.123:0/1346480883 conn(0x7fdc940cfb10 msgr2=0x7fdc940cfef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:43:24.387 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:24.381+0000 7fdc99008640 1 -- 172.21.15.123:0/1346480883 shutdown_connections 2024-09-16T15:43:24.387 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:24.381+0000 7fdc99008640 1 -- 172.21.15.123:0/1346480883 wait complete. 2024-09-16T15:43:24.401 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-16T15:43:24.524 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:24.521+0000 7f9ca3d85640 1 -- 172.21.15.123:0/2174218033 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9c9c0b25a0 msgr2=0x7f9c9c0d1c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:43:24.524 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:24.521+0000 7f9ca3d85640 1 --2- 172.21.15.123:0/2174218033 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9c9c0b25a0 0x7f9c9c0d1c10 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f9c8c009900 tx=0x7f9c8c01c7f0 comp rx=0 tx=0).stop 2024-09-16T15:43:24.524 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:24.521+0000 7f9ca3d85640 1 -- 172.21.15.123:0/2174218033 shutdown_connections 2024-09-16T15:43:24.524 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:24.521+0000 7f9ca3d85640 1 --2- 172.21.15.123:0/2174218033 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9c9c0b25a0 0x7f9c9c0d1c10 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:24.524 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:24.521+0000 7f9ca3d85640 1 --2- 172.21.15.123:0/2174218033 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f9c9c15bda0 0x7f9c9c0d0310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:24.524 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:24.521+0000 7f9ca3d85640 1 -- 172.21.15.123:0/2174218033 >> 172.21.15.123:0/2174218033 conn(0x7f9c9c074bd0 msgr2=0x7f9c9c09aa40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:43:24.524 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:24.521+0000 7f9ca3d85640 1 -- 172.21.15.123:0/2174218033 shutdown_connections 2024-09-16T15:43:24.524 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:24.521+0000 7f9ca3d85640 1 -- 172.21.15.123:0/2174218033 wait complete. 2024-09-16T15:43:24.526 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:24.521+0000 7f9ca3d85640 1 Processor -- start 2024-09-16T15:43:24.527 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:24.521+0000 7f9ca3d85640 1 -- start start 2024-09-16T15:43:24.527 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:24.521+0000 7f9ca3d85640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f9c9c0b25a0 0x7f9c9c158170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:43:24.527 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:24.521+0000 7f9ca3d85640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f9c9c0d0850 0x7f9c9c14de10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:43:24.527 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:24.521+0000 7f9ca3d85640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9c9c15bda0 0x7f9c9c15ace0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:43:24.527 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:24.521+0000 7f9ca3d85640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f9c9c1692a0 con 0x7f9c9c15bda0 2024-09-16T15:43:24.527 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:24.521+0000 7f9ca3d85640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f9c9c169120 con 0x7f9c9c0b25a0 2024-09-16T15:43:24.527 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:24.521+0000 7f9ca3d85640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f9c9c169420 con 0x7f9c9c0d0850 2024-09-16T15:43:24.527 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:24.521+0000 7f9ca2322640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9c9c15bda0 0x7f9c9c15ace0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:43:24.527 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:24.521+0000 7f9ca2322640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9c9c15bda0 0x7f9c9c15ace0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:36434/0 (socket says 172.21.15.123:36434) 2024-09-16T15:43:24.527 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:24.521+0000 7f9ca2322640 1 -- 172.21.15.123:0/242002012 learned_addr learned my addr 172.21.15.123:0/242002012 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:43:24.528 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:24.521+0000 7f9ca1b21640 1 --2- 172.21.15.123:0/242002012 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f9c9c0b25a0 0x7f9c9c158170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:43:24.528 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:24.521+0000 7f9ca1320640 1 --2- 172.21.15.123:0/242002012 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f9c9c0d0850 0x7f9c9c14de10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:43:24.528 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:24.521+0000 7f9ca2322640 1 -- 172.21.15.123:0/242002012 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f9c9c0d0850 msgr2=0x7f9c9c14de10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:43:24.528 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:24.521+0000 7f9ca2322640 1 --2- 172.21.15.123:0/242002012 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f9c9c0d0850 0x7f9c9c14de10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:24.528 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:24.521+0000 7f9ca2322640 1 -- 172.21.15.123:0/242002012 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f9c9c0b25a0 msgr2=0x7f9c9c158170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:43:24.528 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:24.521+0000 7f9ca2322640 1 --2- 172.21.15.123:0/242002012 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f9c9c0b25a0 0x7f9c9c158170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:24.528 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:24.521+0000 7f9ca2322640 1 -- 172.21.15.123:0/242002012 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9c9c14e780 con 0x7f9c9c15bda0 2024-09-16T15:43:24.528 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:24.521+0000 7f9ca2322640 1 --2- 172.21.15.123:0/242002012 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9c9c15bda0 0x7f9c9c15ace0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f9c9400c780 tx=0x7f9c9400cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:43:24.528 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:24.521+0000 7f9c927fc640 1 -- 172.21.15.123:0/242002012 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c94004b70 con 0x7f9c9c15bda0 2024-09-16T15:43:24.529 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:24.521+0000 7f9c927fc640 1 -- 172.21.15.123:0/242002012 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f9c94004e00 con 0x7f9c9c15bda0 2024-09-16T15:43:24.529 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:24.521+0000 7f9c927fc640 1 -- 172.21.15.123:0/242002012 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c9400aa30 con 0x7f9c9c15bda0 2024-09-16T15:43:24.529 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:24.521+0000 7f9ca3d85640 1 -- 172.21.15.123:0/242002012 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9c9c14ea60 con 0x7f9c9c15bda0 2024-09-16T15:43:24.529 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:24.521+0000 7f9ca3d85640 1 -- 172.21.15.123:0/242002012 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9c9c14f020 con 0x7f9c9c15bda0 2024-09-16T15:43:24.530 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:24.525+0000 7f9ca3d85640 1 -- 172.21.15.123:0/242002012 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9c9c0aec10 con 0x7f9c9c15bda0 2024-09-16T15:43:24.530 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:24.525+0000 7f9c927fc640 1 -- 172.21.15.123:0/242002012 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 5) ==== 97080+0+0 (secure 0 0 0) 0x7f9c9400abd0 con 0x7f9c9c15bda0 2024-09-16T15:43:24.530 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:24.525+0000 7f9c927fc640 1 --2- 172.21.15.123:0/242002012 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f9c80072ef0 0x7f9c800753b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:43:24.530 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:24.525+0000 7f9c927fc640 1 -- 172.21.15.123:0/242002012 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f9c94094040 con 0x7f9c9c15bda0 2024-09-16T15:43:24.530 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:24.525+0000 7f9ca1b21640 1 --2- 172.21.15.123:0/242002012 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f9c80072ef0 0x7f9c800753b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:43:24.530 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:24.525+0000 7f9ca1b21640 1 --2- 172.21.15.123:0/242002012 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f9c80072ef0 0x7f9c800753b0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f9c980058d0 tx=0x7f9c98005860 comp rx=0 tx=0).ready entity=mgr.4135 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:43:24.532 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:24.529+0000 7f9c927fc640 1 -- 172.21.15.123:0/242002012 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f9c94020070 con 0x7f9c9c15bda0 2024-09-16T15:43:24.693 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:24.689+0000 7f9ca3d85640 1 -- 172.21.15.123:0/242002012 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f9c9c069d70 con 0x7f9c9c15bda0 2024-09-16T15:43:24.696 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:24.693+0000 7f9c927fc640 1 -- 172.21.15.123:0/242002012 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) ==== 74+0+16725 (secure 0 0 0) 0x7f9c9405f320 con 0x7f9c9c15bda0 2024-09-16T15:43:24.696 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-09-16T15:43:24.696 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":21,"fsid":"56077f89-18c8-486f-ae7e-eecc0bb7822c","created":"2024-09-16T15:42:26.364892+0000","modified":"2024-09-16T15:42:52.401147+0000","last_up_change":"2024-09-16T15:42:38.707234+0000","last_in_change":"2024-09-16T15:42:29.423662+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-16T15:42:39.287764+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-09-16T15:42:48.583563+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-09-16T15:42:49.704665+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9e7c1955-a607-4671-9be7-f2ef760ceeaa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6800","nonce":3882044969},{"type":"v1","addr":"172.21.15.123:6801","nonce":3882044969}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6802","nonce":3882044969},{"type":"v1","addr":"172.21.15.123:6803","nonce":3882044969}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6806","nonce":3882044969},{"type":"v1","addr":"172.21.15.123:6807","nonce":3882044969}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6804","nonce":3882044969},{"type":"v1","addr":"172.21.15.123:6805","nonce":3882044969}]},"public_addr":"172.21.15.123:6801/3882044969","cluster_addr":"172.21.15.123:6803/3882044969","heartbeat_back_addr":"172.21.15.123:6807/3882044969","heartbeat_front_addr":"172.21.15.123:6805/3882044969","state":["exists","up"]},{"osd":1,"uuid":"c540cfde-0977-4742-9b17-fd634668e010","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6816","nonce":4037001850},{"type":"v1","addr":"172.21.15.123:6817","nonce":4037001850}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6818","nonce":4037001850},{"type":"v1","addr":"172.21.15.123:6819","nonce":4037001850}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6822","nonce":4037001850},{"type":"v1","addr":"172.21.15.123:6823","nonce":4037001850}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6820","nonce":4037001850},{"type":"v1","addr":"172.21.15.123:6821","nonce":4037001850}]},"public_addr":"172.21.15.123:6817/4037001850","cluster_addr":"172.21.15.123:6819/4037001850","heartbeat_back_addr":"172.21.15.123:6823/4037001850","heartbeat_front_addr":"172.21.15.123:6821/4037001850","state":["exists","up"]},{"osd":2,"uuid":"9864a496-f169-486b-95c4-1d59162793a7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6824","nonce":1332143088},{"type":"v1","addr":"172.21.15.123:6825","nonce":1332143088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6826","nonce":1332143088},{"type":"v1","addr":"172.21.15.123:6827","nonce":1332143088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6830","nonce":1332143088},{"type":"v1","addr":"172.21.15.123:6831","nonce":1332143088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6828","nonce":1332143088},{"type":"v1","addr":"172.21.15.123:6829","nonce":1332143088}]},"public_addr":"172.21.15.123:6825/1332143088","cluster_addr":"172.21.15.123:6827/1332143088","heartbeat_back_addr":"172.21.15.123:6831/1332143088","heartbeat_front_addr":"172.21.15.123:6829/1332143088","state":["exists","up"]},{"osd":3,"uuid":"c6d1f817-f074-4325-95e1-578eba13edda","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6808","nonce":988586998},{"type":"v1","addr":"172.21.15.123:6809","nonce":988586998}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6810","nonce":988586998},{"type":"v1","addr":"172.21.15.123:6811","nonce":988586998}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6814","nonce":988586998},{"type":"v1","addr":"172.21.15.123:6815","nonce":988586998}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6812","nonce":988586998},{"type":"v1","addr":"172.21.15.123:6813","nonce":988586998}]},"public_addr":"172.21.15.123:6809/988586998","cluster_addr":"172.21.15.123:6811/988586998","heartbeat_back_addr":"172.21.15.123:6815/988586998","heartbeat_front_addr":"172.21.15.123:6813/988586998","state":["exists","up"]},{"osd":4,"uuid":"200a8df4-50ef-40a5-b48a-58981269c086","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6816","nonce":3387813890},{"type":"v1","addr":"172.21.15.184:6817","nonce":3387813890}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6818","nonce":3387813890},{"type":"v1","addr":"172.21.15.184:6819","nonce":3387813890}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6822","nonce":3387813890},{"type":"v1","addr":"172.21.15.184:6823","nonce":3387813890}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6820","nonce":3387813890},{"type":"v1","addr":"172.21.15.184:6821","nonce":3387813890}]},"public_addr":"172.21.15.184:6817/3387813890","cluster_addr":"172.21.15.184:6819/3387813890","heartbeat_back_addr":"172.21.15.184:6823/3387813890","heartbeat_front_addr":"172.21.15.184:6821/3387813890","state":["exists","up"]},{"osd":5,"uuid":"32d5a900-da3d-412c-bc19-13861a2368d8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6808","nonce":2680916017},{"type":"v1","addr":"172.21.15.184:6809","nonce":2680916017}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6810","nonce":2680916017},{"type":"v1","addr":"172.21.15.184:6811","nonce":2680916017}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6814","nonce":2680916017},{"type":"v1","addr":"172.21.15.184:6815","nonce":2680916017}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6812","nonce":2680916017},{"type":"v1","addr":"172.21.15.184:6813","nonce":2680916017}]},"public_addr":"172.21.15.184:6809/2680916017","cluster_addr":"172.21.15.184:6811/2680916017","heartbeat_back_addr":"172.21.15.184:6815/2680916017","heartbeat_front_addr":"172.21.15.184:6813/2680916017","state":["exists","up"]},{"osd":6,"uuid":"cde6f39e-aa1b-4260-ac19-774ee6aaca72","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6824","nonce":2183120457},{"type":"v1","addr":"172.21.15.184:6825","nonce":2183120457}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6826","nonce":2183120457},{"type":"v1","addr":"172.21.15.184:6827","nonce":2183120457}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6830","nonce":2183120457},{"type":"v1","addr":"172.21.15.184:6831","nonce":2183120457}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6828","nonce":2183120457},{"type":"v1","addr":"172.21.15.184:6829","nonce":2183120457}]},"public_addr":"172.21.15.184:6825/2183120457","cluster_addr":"172.21.15.184:6827/2183120457","heartbeat_back_addr":"172.21.15.184:6831/2183120457","heartbeat_front_addr":"172.21.15.184:6829/2183120457","state":["exists","up"]},{"osd":7,"uuid":"4f381778-eee1-47c6-804f-df0e6b33e313","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6800","nonce":4102957285},{"type":"v1","addr":"172.21.15.184:6801","nonce":4102957285}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6802","nonce":4102957285},{"type":"v1","addr":"172.21.15.184:6803","nonce":4102957285}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6806","nonce":4102957285},{"type":"v1","addr":"172.21.15.184:6807","nonce":4102957285}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6804","nonce":4102957285},{"type":"v1","addr":"172.21.15.184:6805","nonce":4102957285}]},"public_addr":"172.21.15.184:6801/4102957285","cluster_addr":"172.21.15.184:6803/4102957285","heartbeat_back_addr":"172.21.15.184:6807/4102957285","heartbeat_front_addr":"172.21.15.184:6805/4102957285","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:35.972210+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.222007+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.198617+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.385663+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.358935+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.156066+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.752973+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:35.682846+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T15:43:24.698 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:24.693+0000 7f9ca3d85640 1 -- 172.21.15.123:0/242002012 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f9c80072ef0 msgr2=0x7f9c800753b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:43:24.698 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:24.693+0000 7f9ca3d85640 1 --2- 172.21.15.123:0/242002012 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f9c80072ef0 0x7f9c800753b0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f9c980058d0 tx=0x7f9c98005860 comp rx=0 tx=0).stop 2024-09-16T15:43:24.698 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:24.693+0000 7f9ca3d85640 1 -- 172.21.15.123:0/242002012 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9c9c15bda0 msgr2=0x7f9c9c15ace0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:43:24.698 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:24.693+0000 7f9ca3d85640 1 --2- 172.21.15.123:0/242002012 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9c9c15bda0 0x7f9c9c15ace0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f9c9400c780 tx=0x7f9c9400cc40 comp rx=0 tx=0).stop 2024-09-16T15:43:24.699 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:24.693+0000 7f9ca3d85640 1 -- 172.21.15.123:0/242002012 shutdown_connections 2024-09-16T15:43:24.699 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:24.693+0000 7f9ca3d85640 1 --2- 172.21.15.123:0/242002012 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f9c80072ef0 0x7f9c800753b0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:24.699 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:24.693+0000 7f9ca3d85640 1 --2- 172.21.15.123:0/242002012 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9c9c15bda0 0x7f9c9c15ace0 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:24.699 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:24.693+0000 7f9ca3d85640 1 --2- 172.21.15.123:0/242002012 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f9c9c0d0850 0x7f9c9c14de10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:24.699 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:24.693+0000 7f9ca3d85640 1 --2- 172.21.15.123:0/242002012 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f9c9c0b25a0 0x7f9c9c158170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:24.699 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:24.693+0000 7f9ca3d85640 1 -- 172.21.15.123:0/242002012 >> 172.21.15.123:0/242002012 conn(0x7f9c9c074bd0 msgr2=0x7f9c9c0b37b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:43:24.699 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:24.693+0000 7f9ca3d85640 1 -- 172.21.15.123:0/242002012 shutdown_connections 2024-09-16T15:43:24.699 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:24.693+0000 7f9ca3d85640 1 -- 172.21.15.123:0/242002012 wait complete. 2024-09-16T15:43:24.713 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-09-16T15:43:24.825 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:24.821+0000 7f5b2caed640 1 -- 172.21.15.123:0/1435816978 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5b2815e820 msgr2=0x7f5b2815ec90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:43:24.825 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:24.821+0000 7f5b2caed640 1 --2- 172.21.15.123:0/1435816978 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5b2815e820 0x7f5b2815ec90 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f5b280cf1f0 tx=0x7f5b2001ccf0 comp rx=0 tx=0).stop 2024-09-16T15:43:24.825 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:24.821+0000 7f5b2caed640 1 -- 172.21.15.123:0/1435816978 shutdown_connections 2024-09-16T15:43:24.825 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:24.821+0000 7f5b2caed640 1 --2- 172.21.15.123:0/1435816978 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5b2815e820 0x7f5b2815ec90 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:24.826 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:24.821+0000 7f5b2caed640 1 --2- 172.21.15.123:0/1435816978 >> v2:172.21.15.184:3300/0 conn(0x7f5b280cd6e0 0x7f5b280cdb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:24.826 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:24.821+0000 7f5b2caed640 1 -- 172.21.15.123:0/1435816978 >> 172.21.15.123:0/1435816978 conn(0x7f5b280cfb10 msgr2=0x7f5b280cff20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:43:24.826 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:24.821+0000 7f5b2caed640 1 -- 172.21.15.123:0/1435816978 shutdown_connections 2024-09-16T15:43:24.826 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:24.821+0000 7f5b2caed640 1 -- 172.21.15.123:0/1435816978 wait complete. 2024-09-16T15:43:24.826 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:24.821+0000 7f5b2caed640 1 Processor -- start 2024-09-16T15:43:24.826 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:24.821+0000 7f5b2caed640 1 -- start start 2024-09-16T15:43:24.828 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:24.825+0000 7f5b2caed640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5b280cd6e0 0x7f5b280c9820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:43:24.828 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:24.825+0000 7f5b2caed640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5b2815e160 0x7f5b2814eac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:43:24.828 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:24.825+0000 7f5b2caed640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f5b2815e820 0x7f5b281f5bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:43:24.828 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:24.825+0000 7f5b2caed640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f5b280cf5b0 con 0x7f5b2815e160 2024-09-16T15:43:24.828 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:24.825+0000 7f5b2caed640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f5b280cf430 con 0x7f5b280cd6e0 2024-09-16T15:43:24.829 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:24.825+0000 7f5b2caed640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f5b281683d0 con 0x7f5b2815e820 2024-09-16T15:43:24.829 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:24.825+0000 7f5b25d9b640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5b2815e160 0x7f5b2814eac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:43:24.829 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:24.825+0000 7f5b25d9b640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5b2815e160 0x7f5b2814eac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:36452/0 (socket says 172.21.15.123:36452) 2024-09-16T15:43:24.829 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:24.825+0000 7f5b25d9b640 1 -- 172.21.15.123:0/430028530 learned_addr learned my addr 172.21.15.123:0/430028530 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:43:24.829 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:24.825+0000 7f5b26d9d640 1 --2- 172.21.15.123:0/430028530 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f5b2815e820 0x7f5b281f5bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:43:24.829 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:24.825+0000 7f5b25d9b640 1 -- 172.21.15.123:0/430028530 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f5b2815e820 msgr2=0x7f5b281f5bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:43:24.829 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:24.825+0000 7f5b25d9b640 1 --2- 172.21.15.123:0/430028530 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f5b2815e820 0x7f5b281f5bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:24.829 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:24.825+0000 7f5b25d9b640 1 -- 172.21.15.123:0/430028530 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5b280cd6e0 msgr2=0x7f5b280c9820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:43:24.829 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:24.825+0000 7f5b2659c640 1 --2- 172.21.15.123:0/430028530 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5b280cd6e0 0x7f5b280c9820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:43:24.829 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:24.825+0000 7f5b25d9b640 1 --2- 172.21.15.123:0/430028530 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5b280cd6e0 0x7f5b280c9820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:24.829 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:24.825+0000 7f5b25d9b640 1 -- 172.21.15.123:0/430028530 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5b28158880 con 0x7f5b2815e160 2024-09-16T15:43:24.829 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:24.825+0000 7f5b2659c640 1 --2- 172.21.15.123:0/430028530 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5b280cd6e0 0x7f5b280c9820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T15:43:24.829 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:24.825+0000 7f5b25d9b640 1 --2- 172.21.15.123:0/430028530 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5b2815e160 0x7f5b2814eac0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f5b1c00c970 tx=0x7f5b1c00ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:43:24.829 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:24.825+0000 7f5b16ffd640 1 -- 172.21.15.123:0/430028530 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b1c007b50 con 0x7f5b2815e160 2024-09-16T15:43:24.830 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:24.825+0000 7f5b16ffd640 1 -- 172.21.15.123:0/430028530 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f5b1c007de0 con 0x7f5b2815e160 2024-09-16T15:43:24.830 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:24.825+0000 7f5b2caed640 1 -- 172.21.15.123:0/430028530 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5b28158b60 con 0x7f5b2815e160 2024-09-16T15:43:24.830 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:24.825+0000 7f5b16ffd640 1 -- 172.21.15.123:0/430028530 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b1c005280 con 0x7f5b2815e160 2024-09-16T15:43:24.830 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:24.825+0000 7f5b2caed640 1 -- 172.21.15.123:0/430028530 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5b281f6110 con 0x7f5b2815e160 2024-09-16T15:43:24.830 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:24.825+0000 7f5b2caed640 1 -- 172.21.15.123:0/430028530 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5b280cdb50 con 0x7f5b2815e160 2024-09-16T15:43:24.831 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:24.825+0000 7f5b16ffd640 1 -- 172.21.15.123:0/430028530 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 5) ==== 97080+0+0 (secure 0 0 0) 0x7f5b1c005420 con 0x7f5b2815e160 2024-09-16T15:43:24.831 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:24.825+0000 7f5b16ffd640 1 --2- 172.21.15.123:0/430028530 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f5b08072ec0 0x7f5b08075380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:43:24.831 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:24.825+0000 7f5b16ffd640 1 -- 172.21.15.123:0/430028530 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f5b1c094430 con 0x7f5b2815e160 2024-09-16T15:43:24.831 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:24.825+0000 7f5b2659c640 1 --2- 172.21.15.123:0/430028530 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f5b08072ec0 0x7f5b08075380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:43:24.831 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:24.829+0000 7f5b2659c640 1 --2- 172.21.15.123:0/430028530 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f5b08072ec0 0x7f5b08075380 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f5b180016e0 tx=0x7f5b18009040 comp rx=0 tx=0).ready entity=mgr.4135 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:43:24.834 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:24.829+0000 7f5b16ffd640 1 -- 172.21.15.123:0/430028530 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f5b1c014030 con 0x7f5b2815e160 2024-09-16T15:43:25.027 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:25.025+0000 7f5b2caed640 1 -- 172.21.15.123:0/430028530 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7f5b280c3350 con 0x7f5b2815e160 2024-09-16T15:43:25.028 INFO:tasks.ceph.mon.a.smithi123.stderr:2024-09-16T15:43:25.025+0000 7f268d076640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-09-16T15:43:25.030 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:25.025+0000 7f5b16ffd640 1 -- 172.21.15.123:0/430028530 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v9) ==== 161+0+0 (secure 0 0 0) 0x7f5b1c05f5e0 con 0x7f5b2815e160 2024-09-16T15:43:25.030 INFO:teuthology.orchestra.run.smithi123.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-09-16T15:43:25.033 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:25.029+0000 7f5b2caed640 1 -- 172.21.15.123:0/430028530 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f5b08072ec0 msgr2=0x7f5b08075380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:43:25.033 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:25.029+0000 7f5b2caed640 1 --2- 172.21.15.123:0/430028530 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f5b08072ec0 0x7f5b08075380 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f5b180016e0 tx=0x7f5b18009040 comp rx=0 tx=0).stop 2024-09-16T15:43:25.033 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:25.029+0000 7f5b2caed640 1 -- 172.21.15.123:0/430028530 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5b2815e160 msgr2=0x7f5b2814eac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:43:25.033 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:25.029+0000 7f5b2caed640 1 --2- 172.21.15.123:0/430028530 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5b2815e160 0x7f5b2814eac0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f5b1c00c970 tx=0x7f5b1c00ce30 comp rx=0 tx=0).stop 2024-09-16T15:43:25.033 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:25.029+0000 7f5b2caed640 1 -- 172.21.15.123:0/430028530 shutdown_connections 2024-09-16T15:43:25.033 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:25.029+0000 7f5b2caed640 1 --2- 172.21.15.123:0/430028530 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f5b08072ec0 0x7f5b08075380 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:25.033 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:25.029+0000 7f5b2caed640 1 --2- 172.21.15.123:0/430028530 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f5b2815e820 0x7f5b281f5bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:25.033 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:25.029+0000 7f5b2caed640 1 --2- 172.21.15.123:0/430028530 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5b2815e160 0x7f5b2814eac0 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:25.033 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:25.029+0000 7f5b2caed640 1 --2- 172.21.15.123:0/430028530 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5b280cd6e0 0x7f5b280c9820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:25.034 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:25.029+0000 7f5b2caed640 1 -- 172.21.15.123:0/430028530 >> 172.21.15.123:0/430028530 conn(0x7f5b280cfb10 msgr2=0x7f5b281558a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:43:25.034 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:25.029+0000 7f5b2caed640 1 -- 172.21.15.123:0/430028530 shutdown_connections 2024-09-16T15:43:25.034 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:25.029+0000 7f5b2caed640 1 -- 172.21.15.123:0/430028530 wait complete. 2024-09-16T15:43:25.044 INFO:tasks.ceph.mds.d.smithi184.stderr: -2302> 2024-09-16T15:42:56.347+0000 7fb2b2e57640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-09-16T15:43:25.049 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-09-16T15:43:25.088 INFO:tasks.ceph.mds.d.smithi184.stdout:starting mds.d at 2024-09-16T15:43:25.165 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:25.161+0000 7f8d961b0640 1 -- 172.21.15.123:0/2370194965 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8d900cf4e0 msgr2=0x7f8d9015f6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:43:25.165 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:25.161+0000 7f8d961b0640 1 --2- 172.21.15.123:0/2370194965 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8d900cf4e0 0x7f8d9015f6b0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f8d88009930 tx=0x7f8d8801c8b0 comp rx=0 tx=0).stop 2024-09-16T15:43:25.165 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:25.161+0000 7f8d961b0640 1 -- 172.21.15.123:0/2370194965 shutdown_connections 2024-09-16T15:43:25.165 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:25.161+0000 7f8d961b0640 1 --2- 172.21.15.123:0/2370194965 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8d900cf4e0 0x7f8d9015f6b0 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:25.165 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:25.161+0000 7f8d961b0640 1 --2- 172.21.15.123:0/2370194965 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f8d9015b9f0 0x7f8d9006fca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:25.166 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:25.161+0000 7f8d961b0640 1 -- 172.21.15.123:0/2370194965 >> 172.21.15.123:0/2370194965 conn(0x7f8d900b1710 msgr2=0x7f8d900b1b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:43:25.166 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:25.161+0000 7f8d961b0640 1 -- 172.21.15.123:0/2370194965 shutdown_connections 2024-09-16T15:43:25.166 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:25.161+0000 7f8d961b0640 1 -- 172.21.15.123:0/2370194965 wait complete. 2024-09-16T15:43:25.166 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:25.161+0000 7f8d961b0640 1 Processor -- start 2024-09-16T15:43:25.166 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:25.161+0000 7f8d961b0640 1 -- start start 2024-09-16T15:43:25.168 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:25.165+0000 7f8d961b0640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8d900c3230 0x7f8d901faa80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:43:25.168 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:25.165+0000 7f8d961b0640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f8d900cf4e0 0x7f8d9014d3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:43:25.168 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:25.165+0000 7f8d961b0640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f8d9015b9f0 0x7f8d90153dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:43:25.168 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:25.165+0000 7f8d961b0640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f8d90169140 con 0x7f8d900c3230 2024-09-16T15:43:25.168 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:25.165+0000 7f8d961b0640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f8d90168fc0 con 0x7f8d900cf4e0 2024-09-16T15:43:25.168 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:25.165+0000 7f8d961b0640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f8d901692c0 con 0x7f8d9015b9f0 2024-09-16T15:43:25.168 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:25.165+0000 7f8d8f7fe640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8d900c3230 0x7f8d901faa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:43:25.169 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:25.165+0000 7f8d8f7fe640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8d900c3230 0x7f8d901faa80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:36466/0 (socket says 172.21.15.123:36466) 2024-09-16T15:43:25.169 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:25.165+0000 7f8d8f7fe640 1 -- 172.21.15.123:0/2576260285 learned_addr learned my addr 172.21.15.123:0/2576260285 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:43:25.169 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:25.165+0000 7f8d8ffff640 1 --2- 172.21.15.123:0/2576260285 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f8d9015b9f0 0x7f8d90153dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:43:25.169 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:25.165+0000 7f8d8effd640 1 --2- 172.21.15.123:0/2576260285 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f8d900cf4e0 0x7f8d9014d3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:43:25.169 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:25.165+0000 7f8d8f7fe640 1 -- 172.21.15.123:0/2576260285 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f8d9015b9f0 msgr2=0x7f8d90153dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:43:25.169 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:25.165+0000 7f8d8f7fe640 1 --2- 172.21.15.123:0/2576260285 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f8d9015b9f0 0x7f8d90153dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:25.169 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:25.165+0000 7f8d8f7fe640 1 -- 172.21.15.123:0/2576260285 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f8d900cf4e0 msgr2=0x7f8d9014d3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:43:25.169 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:25.165+0000 7f8d8f7fe640 1 --2- 172.21.15.123:0/2576260285 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f8d900cf4e0 0x7f8d9014d3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:25.169 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:25.165+0000 7f8d8f7fe640 1 -- 172.21.15.123:0/2576260285 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8d901fb840 con 0x7f8d900c3230 2024-09-16T15:43:25.169 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:25.165+0000 7f8d8f7fe640 1 --2- 172.21.15.123:0/2576260285 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8d900c3230 0x7f8d901faa80 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f8d8000ccb0 tx=0x7f8d80007610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:43:25.169 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:25.165+0000 7f8d77fff640 1 -- 172.21.15.123:0/2576260285 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d80004b30 con 0x7f8d900c3230 2024-09-16T15:43:25.169 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:25.165+0000 7f8d77fff640 1 -- 172.21.15.123:0/2576260285 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f8d80004de0 con 0x7f8d900c3230 2024-09-16T15:43:25.170 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:25.165+0000 7f8d961b0640 1 -- 172.21.15.123:0/2576260285 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8d901fbb20 con 0x7f8d900c3230 2024-09-16T15:43:25.170 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:25.165+0000 7f8d77fff640 1 -- 172.21.15.123:0/2576260285 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d80005280 con 0x7f8d900c3230 2024-09-16T15:43:25.170 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:25.165+0000 7f8d961b0640 1 -- 172.21.15.123:0/2576260285 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8d901fc0e0 con 0x7f8d900c3230 2024-09-16T15:43:25.170 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:25.165+0000 7f8d961b0640 1 -- 172.21.15.123:0/2576260285 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8d9015f6b0 con 0x7f8d900c3230 2024-09-16T15:43:25.170 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:25.165+0000 7f8d77fff640 1 -- 172.21.15.123:0/2576260285 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 5) ==== 97080+0+0 (secure 0 0 0) 0x7f8d80005420 con 0x7f8d900c3230 2024-09-16T15:43:25.171 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:25.165+0000 7f8d77fff640 1 --2- 172.21.15.123:0/2576260285 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f8d7c072ec0 0x7f8d7c075380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:43:25.171 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:25.165+0000 7f8d77fff640 1 -- 172.21.15.123:0/2576260285 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 5795+0+0 (secure 0 0 0) 0x7f8d80094a20 con 0x7f8d900c3230 2024-09-16T15:43:25.171 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:25.165+0000 7f8d8effd640 1 --2- 172.21.15.123:0/2576260285 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f8d7c072ec0 0x7f8d7c075380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:43:25.171 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:25.165+0000 7f8d8effd640 1 --2- 172.21.15.123:0/2576260285 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f8d7c072ec0 0x7f8d7c075380 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f8d90154ff0 tx=0x7f8d84009040 comp rx=0 tx=0).ready entity=mgr.4135 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:43:25.173 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:25.169+0000 7f8d77fff640 1 -- 172.21.15.123:0/2576260285 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f8d80014030 con 0x7f8d900c3230 2024-09-16T15:43:25.367 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:25.365+0000 7f8d961b0640 1 -- 172.21.15.123:0/2576260285 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) -- 0x7f8d90153150 con 0x7f8d900c3230 2024-09-16T15:43:26.039 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:26.037+0000 7f8d77fff640 1 -- 172.21.15.123:0/2576260285 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v10) ==== 104+0+0 (secure 0 0 0) 0x7f8d8005fc10 con 0x7f8d900c3230 2024-09-16T15:43:26.042 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:26.037+0000 7f8d961b0640 1 -- 172.21.15.123:0/2576260285 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f8d7c072ec0 msgr2=0x7f8d7c075380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:43:26.043 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:26.037+0000 7f8d961b0640 1 --2- 172.21.15.123:0/2576260285 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f8d7c072ec0 0x7f8d7c075380 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f8d90154ff0 tx=0x7f8d84009040 comp rx=0 tx=0).stop 2024-09-16T15:43:26.043 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:26.037+0000 7f8d961b0640 1 -- 172.21.15.123:0/2576260285 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8d900c3230 msgr2=0x7f8d901faa80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:43:26.043 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:26.037+0000 7f8d961b0640 1 --2- 172.21.15.123:0/2576260285 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8d900c3230 0x7f8d901faa80 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f8d8000ccb0 tx=0x7f8d80007610 comp rx=0 tx=0).stop 2024-09-16T15:43:26.043 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:26.037+0000 7f8d961b0640 1 -- 172.21.15.123:0/2576260285 shutdown_connections 2024-09-16T15:43:26.043 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:26.037+0000 7f8d961b0640 1 --2- 172.21.15.123:0/2576260285 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f8d7c072ec0 0x7f8d7c075380 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:26.043 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:26.037+0000 7f8d961b0640 1 --2- 172.21.15.123:0/2576260285 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f8d9015b9f0 0x7f8d90153dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:26.043 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:26.037+0000 7f8d961b0640 1 --2- 172.21.15.123:0/2576260285 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f8d900cf4e0 0x7f8d9014d3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:26.043 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:26.037+0000 7f8d961b0640 1 --2- 172.21.15.123:0/2576260285 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8d900c3230 0x7f8d901faa80 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:26.043 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:26.037+0000 7f8d961b0640 1 -- 172.21.15.123:0/2576260285 >> 172.21.15.123:0/2576260285 conn(0x7f8d900b1710 msgr2=0x7f8d900b1af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:43:26.043 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:26.037+0000 7f8d961b0640 1 -- 172.21.15.123:0/2576260285 shutdown_connections 2024-09-16T15:43:26.043 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:26.037+0000 7f8d961b0640 1 -- 172.21.15.123:0/2576260285 wait complete. 2024-09-16T15:43:26.064 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-09-16T15:43:26.201 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:26.197+0000 7f7f7db07640 1 -- 172.21.15.123:0/2155021399 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7f78073390 msgr2=0x7f7f7806ff30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:43:26.202 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:26.197+0000 7f7f7db07640 1 --2- 172.21.15.123:0/2155021399 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7f78073390 0x7f7f7806ff30 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f7f7806a600 tx=0x7f7f6001c8c0 comp rx=0 tx=0).stop 2024-09-16T15:43:26.202 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:26.197+0000 7f7f7db07640 1 -- 172.21.15.123:0/2155021399 shutdown_connections 2024-09-16T15:43:26.202 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:26.197+0000 7f7f7db07640 1 --2- 172.21.15.123:0/2155021399 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7f78073390 0x7f7f7806ff30 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:26.202 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:26.197+0000 7f7f7db07640 1 --2- 172.21.15.123:0/2155021399 >> v2:172.21.15.184:3300/0 conn(0x7f7f780aee40 0x7f7f7815b980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:26.202 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:26.197+0000 7f7f7db07640 1 -- 172.21.15.123:0/2155021399 >> 172.21.15.123:0/2155021399 conn(0x7f7f780b1630 msgr2=0x7f7f780b1a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:43:26.202 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:26.197+0000 7f7f7db07640 1 -- 172.21.15.123:0/2155021399 shutdown_connections 2024-09-16T15:43:26.202 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:26.197+0000 7f7f7db07640 1 -- 172.21.15.123:0/2155021399 wait complete. 2024-09-16T15:43:26.204 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:26.197+0000 7f7f7db07640 1 Processor -- start 2024-09-16T15:43:26.204 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:26.197+0000 7f7f7db07640 1 -- start start 2024-09-16T15:43:26.204 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:26.201+0000 7f7f7db07640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f7f78070470 0x7f7f781f8370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:43:26.204 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:26.201+0000 7f7f7db07640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7f78073390 0x7f7f780cea70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:43:26.204 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:26.201+0000 7f7f7db07640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7f780aee40 0x7f7f780adb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:43:26.204 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:26.201+0000 7f7f7db07640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f7f780b3e10 con 0x7f7f780aee40 2024-09-16T15:43:26.204 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:26.201+0000 7f7f7db07640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f7f780b3c90 con 0x7f7f78073390 2024-09-16T15:43:26.205 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:26.201+0000 7f7f7db07640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f7f780b3f90 con 0x7f7f78070470 2024-09-16T15:43:26.205 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:26.201+0000 7f7f77fff640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7f780aee40 0x7f7f780adb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:43:26.205 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:26.201+0000 7f7f77fff640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7f780aee40 0x7f7f780adb10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:36488/0 (socket says 172.21.15.123:36488) 2024-09-16T15:43:26.205 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:26.201+0000 7f7f77fff640 1 -- 172.21.15.123:0/2136648149 learned_addr learned my addr 172.21.15.123:0/2136648149 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:43:26.205 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:26.201+0000 7f7f777fe640 1 --2- 172.21.15.123:0/2136648149 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f7f78070470 0x7f7f781f8370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:43:26.205 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:26.201+0000 7f7f76ffd640 1 --2- 172.21.15.123:0/2136648149 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7f78073390 0x7f7f780cea70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:43:26.205 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:26.201+0000 7f7f77fff640 1 -- 172.21.15.123:0/2136648149 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f7f78070470 msgr2=0x7f7f781f8370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:43:26.205 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:26.201+0000 7f7f77fff640 1 --2- 172.21.15.123:0/2136648149 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f7f78070470 0x7f7f781f8370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:26.205 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:26.201+0000 7f7f77fff640 1 -- 172.21.15.123:0/2136648149 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7f78073390 msgr2=0x7f7f780cea70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:43:26.205 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:26.201+0000 7f7f77fff640 1 --2- 172.21.15.123:0/2136648149 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7f78073390 0x7f7f780cea70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:26.205 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:26.201+0000 7f7f77fff640 1 -- 172.21.15.123:0/2136648149 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7f781f8a70 con 0x7f7f780aee40 2024-09-16T15:43:26.206 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:26.201+0000 7f7f76ffd640 1 --2- 172.21.15.123:0/2136648149 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7f78073390 0x7f7f780cea70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T15:43:26.206 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:26.201+0000 7f7f77fff640 1 --2- 172.21.15.123:0/2136648149 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7f780aee40 0x7f7f780adb10 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f7f7000c3c0 tx=0x7f7f7000c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:43:26.206 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:26.201+0000 7f7f6ffff640 1 -- 172.21.15.123:0/2136648149 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f70011790 con 0x7f7f780aee40 2024-09-16T15:43:26.206 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:26.201+0000 7f7f6ffff640 1 -- 172.21.15.123:0/2136648149 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f7f70017070 con 0x7f7f780aee40 2024-09-16T15:43:26.206 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:26.201+0000 7f7f7db07640 1 -- 172.21.15.123:0/2136648149 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7f781f8d50 con 0x7f7f780aee40 2024-09-16T15:43:26.206 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:26.201+0000 7f7f6ffff640 1 -- 172.21.15.123:0/2136648149 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f70011cc0 con 0x7f7f780aee40 2024-09-16T15:43:26.206 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:26.201+0000 7f7f7db07640 1 -- 172.21.15.123:0/2136648149 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7f780cd100 con 0x7f7f780aee40 2024-09-16T15:43:26.207 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:26.201+0000 7f7f7db07640 1 -- 172.21.15.123:0/2136648149 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7f781f9270 con 0x7f7f780aee40 2024-09-16T15:43:26.207 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:26.201+0000 7f7f6ffff640 1 -- 172.21.15.123:0/2136648149 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 5) ==== 97080+0+0 (secure 0 0 0) 0x7f7f7001c460 con 0x7f7f780aee40 2024-09-16T15:43:26.207 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:26.201+0000 7f7f6ffff640 1 --2- 172.21.15.123:0/2136648149 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f7f58072ec0 0x7f7f58075380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:43:26.207 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:26.201+0000 7f7f6ffff640 1 -- 172.21.15.123:0/2136648149 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 5795+0+0 (secure 0 0 0) 0x7f7f700946e0 con 0x7f7f780aee40 2024-09-16T15:43:26.207 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:26.205+0000 7f7f777fe640 1 --2- 172.21.15.123:0/2136648149 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f7f58072ec0 0x7f7f58075380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:43:26.208 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:26.205+0000 7f7f777fe640 1 --2- 172.21.15.123:0/2136648149 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f7f58072ec0 0x7f7f58075380 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f7f68005fd0 tx=0x7f7f6800a780 comp rx=0 tx=0).ready entity=mgr.4135 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:43:26.210 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:26.205+0000 7f7f6ffff640 1 -- 172.21.15.123:0/2136648149 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f7f781f9270 con 0x7f7f780aee40 2024-09-16T15:43:26.385 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:26.381+0000 7f7f7db07640 1 -- 172.21.15.123:0/2136648149 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) -- 0x7f7f7806ff30 con 0x7f7f780aee40 2024-09-16T15:43:27.376 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:27.373+0000 7f7f6ffff640 1 -- 172.21.15.123:0/2136648149 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v23) ==== 181+0+0 (secure 0 0 0) 0x7f7f7005f9b0 con 0x7f7f780aee40 2024-09-16T15:43:27.376 INFO:teuthology.orchestra.run.smithi123.stderr:pool 'cephfs_metadata' removed 2024-09-16T15:43:27.380 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:27.377+0000 7f7f7db07640 1 -- 172.21.15.123:0/2136648149 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f7f58072ec0 msgr2=0x7f7f58075380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:43:27.380 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:27.377+0000 7f7f7db07640 1 --2- 172.21.15.123:0/2136648149 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f7f58072ec0 0x7f7f58075380 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f7f68005fd0 tx=0x7f7f6800a780 comp rx=0 tx=0).stop 2024-09-16T15:43:27.380 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:27.377+0000 7f7f7db07640 1 -- 172.21.15.123:0/2136648149 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7f780aee40 msgr2=0x7f7f780adb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:43:27.380 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:27.377+0000 7f7f7db07640 1 --2- 172.21.15.123:0/2136648149 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7f780aee40 0x7f7f780adb10 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f7f7000c3c0 tx=0x7f7f7000c880 comp rx=0 tx=0).stop 2024-09-16T15:43:27.380 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:27.377+0000 7f7f7db07640 1 -- 172.21.15.123:0/2136648149 shutdown_connections 2024-09-16T15:43:27.380 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:27.377+0000 7f7f7db07640 1 --2- 172.21.15.123:0/2136648149 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f7f58072ec0 0x7f7f58075380 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:27.380 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:27.377+0000 7f7f7db07640 1 --2- 172.21.15.123:0/2136648149 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7f780aee40 0x7f7f780adb10 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:27.380 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:27.377+0000 7f7f7db07640 1 --2- 172.21.15.123:0/2136648149 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7f78073390 0x7f7f780cea70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:27.380 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:27.377+0000 7f7f7db07640 1 --2- 172.21.15.123:0/2136648149 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f7f78070470 0x7f7f781f8370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:27.380 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:27.377+0000 7f7f7db07640 1 -- 172.21.15.123:0/2136648149 >> 172.21.15.123:0/2136648149 conn(0x7f7f780b1630 msgr2=0x7f7f7815ca00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:43:27.381 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:27.377+0000 7f7f7db07640 1 -- 172.21.15.123:0/2136648149 shutdown_connections 2024-09-16T15:43:27.381 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:27.377+0000 7f7f7db07640 1 -- 172.21.15.123:0/2136648149 wait complete. 2024-09-16T15:43:27.401 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-09-16T15:43:27.534 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:27.529+0000 7f239bc63640 1 -- 172.21.15.123:0/851005866 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f239415fd80 msgr2=0x7f23940704e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:43:27.534 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:27.529+0000 7f239bc63640 1 --2- 172.21.15.123:0/851005866 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f239415fd80 0x7f23940704e0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f2384009900 tx=0x7f238401c8e0 comp rx=0 tx=0).stop 2024-09-16T15:43:27.534 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:27.529+0000 7f239bc63640 1 -- 172.21.15.123:0/851005866 shutdown_connections 2024-09-16T15:43:27.534 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:27.529+0000 7f239bc63640 1 --2- 172.21.15.123:0/851005866 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f2394074440 0x7f23940cdf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:27.534 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:27.529+0000 7f239bc63640 1 --2- 172.21.15.123:0/851005866 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f239415fd80 0x7f23940704e0 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:27.534 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:27.529+0000 7f239bc63640 1 -- 172.21.15.123:0/851005866 >> 172.21.15.123:0/851005866 conn(0x7f23940b1630 msgr2=0x7f23940b1a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:43:27.534 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:27.529+0000 7f239bc63640 1 -- 172.21.15.123:0/851005866 shutdown_connections 2024-09-16T15:43:27.534 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:27.529+0000 7f239bc63640 1 -- 172.21.15.123:0/851005866 wait complete. 2024-09-16T15:43:27.535 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:27.533+0000 7f239bc63640 1 Processor -- start 2024-09-16T15:43:27.535 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:27.533+0000 7f239bc63640 1 -- start start 2024-09-16T15:43:27.537 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:27.533+0000 7f239bc63640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2394074440 0x7f23941f75f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:43:27.537 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:27.533+0000 7f239bc63640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f2394159680 0x7f23940b3a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:43:27.537 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:27.533+0000 7f239bc63640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f239415fd80 0x7f2394164bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:43:27.537 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:27.533+0000 7f239bc63640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f2394168da0 con 0x7f2394074440 2024-09-16T15:43:27.537 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:27.533+0000 7f239bc63640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f2394168c20 con 0x7f239415fd80 2024-09-16T15:43:27.537 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:27.533+0000 7f239bc63640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f2394168f20 con 0x7f2394159680 2024-09-16T15:43:27.537 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:27.533+0000 7f23999ff640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2394074440 0x7f23941f75f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:43:27.538 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:27.533+0000 7f23999ff640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2394074440 0x7f23941f75f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:36512/0 (socket says 172.21.15.123:36512) 2024-09-16T15:43:27.538 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:27.533+0000 7f23999ff640 1 -- 172.21.15.123:0/2544416348 learned_addr learned my addr 172.21.15.123:0/2544416348 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:43:27.538 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:27.533+0000 7f23991fe640 1 --2- 172.21.15.123:0/2544416348 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f2394159680 0x7f23940b3a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:43:27.538 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:27.533+0000 7f239a200640 1 --2- 172.21.15.123:0/2544416348 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f239415fd80 0x7f2394164bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:43:27.538 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:27.533+0000 7f23999ff640 1 -- 172.21.15.123:0/2544416348 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f2394159680 msgr2=0x7f23940b3a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:43:27.538 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:27.533+0000 7f23999ff640 1 --2- 172.21.15.123:0/2544416348 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f2394159680 0x7f23940b3a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:27.538 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:27.533+0000 7f23999ff640 1 -- 172.21.15.123:0/2544416348 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f239415fd80 msgr2=0x7f2394164bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:43:27.538 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:27.533+0000 7f23999ff640 1 --2- 172.21.15.123:0/2544416348 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f239415fd80 0x7f2394164bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:27.538 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:27.533+0000 7f23999ff640 1 -- 172.21.15.123:0/2544416348 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f23940b4380 con 0x7f2394074440 2024-09-16T15:43:27.538 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:27.533+0000 7f23991fe640 1 --2- 172.21.15.123:0/2544416348 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f2394159680 0x7f23940b3a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T15:43:27.538 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:27.533+0000 7f23999ff640 1 --2- 172.21.15.123:0/2544416348 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2394074440 0x7f23941f75f0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f239000c4b0 tx=0x7f239000c970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:43:27.539 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:27.533+0000 7f238a7fc640 1 -- 172.21.15.123:0/2544416348 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f23900052d0 con 0x7f2394074440 2024-09-16T15:43:27.539 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:27.533+0000 7f238a7fc640 1 -- 172.21.15.123:0/2544416348 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f2390005470 con 0x7f2394074440 2024-09-16T15:43:27.539 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:27.533+0000 7f239bc63640 1 -- 172.21.15.123:0/2544416348 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2394154940 con 0x7f2394074440 2024-09-16T15:43:27.539 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:27.533+0000 7f238a7fc640 1 -- 172.21.15.123:0/2544416348 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2390012870 con 0x7f2394074440 2024-09-16T15:43:27.539 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:27.533+0000 7f239bc63640 1 -- 172.21.15.123:0/2544416348 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2394154f00 con 0x7f2394074440 2024-09-16T15:43:27.539 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:27.533+0000 7f239bc63640 1 -- 172.21.15.123:0/2544416348 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f23940cdf40 con 0x7f2394074440 2024-09-16T15:43:27.540 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:27.533+0000 7f238a7fc640 1 -- 172.21.15.123:0/2544416348 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 5) ==== 97080+0+0 (secure 0 0 0) 0x7f2390003600 con 0x7f2394074440 2024-09-16T15:43:27.540 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:27.533+0000 7f238a7fc640 1 --2- 172.21.15.123:0/2544416348 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f2378072ec0 0x7f2378075380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:43:27.540 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:27.537+0000 7f238a7fc640 1 -- 172.21.15.123:0/2544416348 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5357+0+0 (secure 0 0 0) 0x7f2390093830 con 0x7f2394074440 2024-09-16T15:43:27.540 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:27.537+0000 7f23991fe640 1 --2- 172.21.15.123:0/2544416348 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f2378072ec0 0x7f2378075380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:43:27.540 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:27.537+0000 7f23991fe640 1 --2- 172.21.15.123:0/2544416348 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f2378072ec0 0x7f2378075380 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f23840af000 tx=0x7f238401b120 comp rx=0 tx=0).ready entity=mgr.4135 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:43:27.542 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:27.541+0000 7f238a7fc640 1 -- 172.21.15.123:0/2544416348 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f239005ebe0 con 0x7f2394074440 2024-09-16T15:43:27.716 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:27.713+0000 7f239bc63640 1 -- 172.21.15.123:0/2544416348 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) -- 0x7f2394070040 con 0x7f2394074440 2024-09-16T15:43:28.385 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:28.381+0000 7f238a7fc640 1 -- 172.21.15.123:0/2544416348 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v24) ==== 169+0+0 (secure 0 0 0) 0x7f239005e4e0 con 0x7f2394074440 2024-09-16T15:43:28.386 INFO:teuthology.orchestra.run.smithi123.stderr:pool 'cephfs_data' removed 2024-09-16T15:43:28.392 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:28.389+0000 7f239bc63640 1 -- 172.21.15.123:0/2544416348 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f2378072ec0 msgr2=0x7f2378075380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:43:28.392 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:28.389+0000 7f239bc63640 1 --2- 172.21.15.123:0/2544416348 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f2378072ec0 0x7f2378075380 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f23840af000 tx=0x7f238401b120 comp rx=0 tx=0).stop 2024-09-16T15:43:28.392 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:28.389+0000 7f239bc63640 1 -- 172.21.15.123:0/2544416348 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2394074440 msgr2=0x7f23941f75f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:43:28.392 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:28.389+0000 7f239bc63640 1 --2- 172.21.15.123:0/2544416348 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2394074440 0x7f23941f75f0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f239000c4b0 tx=0x7f239000c970 comp rx=0 tx=0).stop 2024-09-16T15:43:28.392 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:28.389+0000 7f239bc63640 1 -- 172.21.15.123:0/2544416348 shutdown_connections 2024-09-16T15:43:28.392 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:28.389+0000 7f239bc63640 1 --2- 172.21.15.123:0/2544416348 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f2378072ec0 0x7f2378075380 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:28.392 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:28.389+0000 7f239bc63640 1 --2- 172.21.15.123:0/2544416348 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f239415fd80 0x7f2394164bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:28.392 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:28.389+0000 7f239bc63640 1 --2- 172.21.15.123:0/2544416348 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f2394159680 0x7f23940b3a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:28.392 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:28.389+0000 7f239bc63640 1 --2- 172.21.15.123:0/2544416348 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2394074440 0x7f23941f75f0 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:28.393 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:28.389+0000 7f239bc63640 1 -- 172.21.15.123:0/2544416348 >> 172.21.15.123:0/2544416348 conn(0x7f23940b1630 msgr2=0x7f23941588a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:43:28.393 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:28.389+0000 7f239bc63640 1 -- 172.21.15.123:0/2544416348 shutdown_connections 2024-09-16T15:43:28.393 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:28.389+0000 7f239bc63640 1 -- 172.21.15.123:0/2544416348 wait complete. 2024-09-16T15:43:28.414 INFO:tasks.ceph.mds.a:Restarting daemon 2024-09-16T15:43:28.414 INFO:tasks.ceph.mds.a:Stopping old one... 2024-09-16T15:43:28.414 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-09-16T15:43:28.414 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T15:43:28.415 INFO:tasks.ceph.mds.c:Restarting daemon 2024-09-16T15:43:28.415 INFO:tasks.ceph.mds.c:Stopping old one... 2024-09-16T15:43:28.415 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-09-16T15:43:28.415 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T15:43:28.415 INFO:tasks.ceph.mds.b:Restarting daemon 2024-09-16T15:43:28.415 INFO:tasks.ceph.mds.b:Stopping old one... 2024-09-16T15:43:28.415 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-09-16T15:43:28.416 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T15:43:28.416 INFO:tasks.ceph.mds.d:Restarting daemon 2024-09-16T15:43:28.416 INFO:tasks.ceph.mds.d:Stopping old one... 2024-09-16T15:43:28.416 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-09-16T15:43:28.416 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T15:43:28.425 INFO:tasks.ceph.mds.a:Stopped 2024-09-16T15:43:28.425 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-09-16T15:43:28.427 INFO:tasks.ceph.mds.c:Stopped 2024-09-16T15:43:28.427 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-09-16T15:43:28.432 INFO:tasks.ceph.mds.d:Stopped 2024-09-16T15:43:28.433 DEBUG:teuthology.orchestra.run.smithi184:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-09-16T15:43:28.472 INFO:tasks.ceph.mds.a:Started 2024-09-16T15:43:28.472 DEBUG:teuthology.parallel:result is None 2024-09-16T15:43:28.473 INFO:tasks.ceph.mds.c:Started 2024-09-16T15:43:28.473 DEBUG:teuthology.parallel:result is None 2024-09-16T15:43:28.476 INFO:tasks.ceph.mds.d:Started 2024-09-16T15:43:28.476 DEBUG:teuthology.parallel:result is None 2024-09-16T15:43:28.535 INFO:tasks.ceph.mds.c.smithi123.stdout:starting mds.c at 2024-09-16T15:43:28.540 INFO:tasks.ceph.mds.a.smithi123.stdout:starting mds.a at 2024-09-16T15:43:28.548 INFO:tasks.ceph.mds.d.smithi184.stdout:starting mds.d at 2024-09-16T15:43:31.035 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~0s 2024-09-16T15:43:34.433 INFO:tasks.ceph.mds.b:Stopped 2024-09-16T15:43:34.433 DEBUG:teuthology.orchestra.run.smithi184:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-09-16T15:43:34.436 INFO:tasks.ceph.mds.b:Started 2024-09-16T15:43:34.436 DEBUG:teuthology.parallel:result is None 2024-09-16T15:43:34.436 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-09-16T15:43:34.511 INFO:tasks.ceph.mds.b.smithi184.stdout:starting mds.b at 2024-09-16T15:43:34.574 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:34.569+0000 7fd9cbeff640 1 -- 172.21.15.123:0/1749195028 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fd9c40cc260 msgr2=0x7fd9c40cffa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:43:34.574 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:34.569+0000 7fd9cbeff640 1 --2- 172.21.15.123:0/1749195028 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fd9c40cc260 0x7fd9c40cffa0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fd9b4009900 tx=0x7fd9b401c8b0 comp rx=0 tx=0).stop 2024-09-16T15:43:34.574 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:34.569+0000 7fd9cbeff640 1 -- 172.21.15.123:0/1749195028 shutdown_connections 2024-09-16T15:43:34.574 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:34.569+0000 7fd9cbeff640 1 --2- 172.21.15.123:0/1749195028 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fd9c40cc260 0x7fd9c40cffa0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:34.574 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:34.569+0000 7fd9cbeff640 1 --2- 172.21.15.123:0/1749195028 >> v2:172.21.15.184:3300/0 conn(0x7fd9c4074860 0x7fd9c40cbd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:34.574 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:34.569+0000 7fd9cbeff640 1 -- 172.21.15.123:0/1749195028 >> 172.21.15.123:0/1749195028 conn(0x7fd9c40732a0 msgr2=0x7fd9c4074150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:43:34.574 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:34.569+0000 7fd9cbeff640 1 -- 172.21.15.123:0/1749195028 shutdown_connections 2024-09-16T15:43:34.574 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:34.569+0000 7fd9cbeff640 1 -- 172.21.15.123:0/1749195028 wait complete. 2024-09-16T15:43:34.575 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:34.569+0000 7fd9cbeff640 1 Processor -- start 2024-09-16T15:43:34.575 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:34.573+0000 7fd9cbeff640 1 -- start start 2024-09-16T15:43:34.577 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:34.573+0000 7fd9cbeff640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd9c4074860 0x7fd9c40c9f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:43:34.577 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:34.573+0000 7fd9cbeff640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd9c40cc260 0x7fd9c40cd7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:43:34.577 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:34.573+0000 7fd9cbeff640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fd9c415f9a0 0x7fd9c4097940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:43:34.577 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:34.573+0000 7fd9cbeff640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fd9c415d430 con 0x7fd9c40cc260 2024-09-16T15:43:34.577 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:34.573+0000 7fd9cbeff640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fd9c415d2b0 con 0x7fd9c4074860 2024-09-16T15:43:34.577 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:34.573+0000 7fd9cbeff640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fd9c415d5b0 con 0x7fd9c415f9a0 2024-09-16T15:43:34.577 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:34.573+0000 7fd9c949a640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd9c40cc260 0x7fd9c40cd7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:43:34.577 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:34.573+0000 7fd9c949a640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd9c40cc260 0x7fd9c40cd7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:44724/0 (socket says 172.21.15.123:44724) 2024-09-16T15:43:34.577 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:34.573+0000 7fd9c949a640 1 -- 172.21.15.123:0/3059418834 learned_addr learned my addr 172.21.15.123:0/3059418834 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:43:34.577 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:34.573+0000 7fd9c9c9b640 1 --2- 172.21.15.123:0/3059418834 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd9c4074860 0x7fd9c40c9f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:43:34.577 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:34.573+0000 7fd9ca49c640 1 --2- 172.21.15.123:0/3059418834 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fd9c415f9a0 0x7fd9c4097940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:43:34.578 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:34.573+0000 7fd9c949a640 1 -- 172.21.15.123:0/3059418834 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fd9c415f9a0 msgr2=0x7fd9c4097940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:43:34.578 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:34.573+0000 7fd9c949a640 1 --2- 172.21.15.123:0/3059418834 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fd9c415f9a0 0x7fd9c4097940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:34.578 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:34.573+0000 7fd9c949a640 1 -- 172.21.15.123:0/3059418834 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd9c4074860 msgr2=0x7fd9c40c9f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:43:34.578 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:34.573+0000 7fd9c949a640 1 --2- 172.21.15.123:0/3059418834 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd9c4074860 0x7fd9c40c9f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:34.578 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:34.573+0000 7fd9c949a640 1 -- 172.21.15.123:0/3059418834 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd9c4156190 con 0x7fd9c40cc260 2024-09-16T15:43:34.578 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:34.573+0000 7fd9c949a640 1 --2- 172.21.15.123:0/3059418834 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd9c40cc260 0x7fd9c40cd7b0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7fd9c000ec00 tx=0x7fd9c000c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:43:34.578 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:34.573+0000 7fd9ba7fc640 1 -- 172.21.15.123:0/3059418834 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9c0010630 con 0x7fd9c40cc260 2024-09-16T15:43:34.578 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:34.573+0000 7fd9ba7fc640 1 -- 172.21.15.123:0/3059418834 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fd9c00108c0 con 0x7fd9c40cc260 2024-09-16T15:43:34.578 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:34.573+0000 7fd9cbeff640 1 -- 172.21.15.123:0/3059418834 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd9c41f8210 con 0x7fd9c40cc260 2024-09-16T15:43:34.578 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:34.573+0000 7fd9ba7fc640 1 -- 172.21.15.123:0/3059418834 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9c0010cf0 con 0x7fd9c40cc260 2024-09-16T15:43:34.578 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:34.573+0000 7fd9cbeff640 1 -- 172.21.15.123:0/3059418834 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd9c41f87d0 con 0x7fd9c40cc260 2024-09-16T15:43:34.579 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:34.573+0000 7fd9cbeff640 1 -- 172.21.15.123:0/3059418834 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd9c40cbd20 con 0x7fd9c40cc260 2024-09-16T15:43:34.579 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:34.573+0000 7fd9ba7fc640 1 -- 172.21.15.123:0/3059418834 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 5) ==== 97080+0+0 (secure 0 0 0) 0x7fd9c0018420 con 0x7fd9c40cc260 2024-09-16T15:43:34.579 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:34.577+0000 7fd9ba7fc640 1 --2- 172.21.15.123:0/3059418834 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fd9b0072ef0 0x7fd9b00753b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:43:34.579 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:34.577+0000 7fd9ba7fc640 1 -- 172.21.15.123:0/3059418834 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 4943+0+0 (secure 0 0 0) 0x7fd9c0094800 con 0x7fd9c40cc260 2024-09-16T15:43:34.579 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:34.577+0000 7fd9c9c9b640 1 --2- 172.21.15.123:0/3059418834 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fd9b0072ef0 0x7fd9b00753b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:43:34.580 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:34.577+0000 7fd9c9c9b640 1 --2- 172.21.15.123:0/3059418834 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fd9b0072ef0 0x7fd9b00753b0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fd9bc007fd0 tx=0x7fd9bc009040 comp rx=0 tx=0).ready entity=mgr.4135 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:43:34.583 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:34.581+0000 7fd9ba7fc640 1 -- 172.21.15.123:0/3059418834 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7fd9c005f710 con 0x7fd9c40cc260 2024-09-16T15:43:34.756 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:34.753+0000 7fd9cbeff640 1 -- 172.21.15.123:0/3059418834 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) -- 0x7fd9c40995d0 con 0x7fd9c40cc260 2024-09-16T15:43:36.430 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:36.425+0000 7fd9ba7fc640 1 -- 172.21.15.123:0/3059418834 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v29) ==== 97+0+0 (secure 0 0 0) 0x7fd9c0064140 con 0x7fd9c40cc260 2024-09-16T15:43:36.431 INFO:teuthology.orchestra.run.smithi123.stderr: removed all blocklist entries 2024-09-16T15:43:36.433 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:36.429+0000 7fd9cbeff640 1 -- 172.21.15.123:0/3059418834 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fd9b0072ef0 msgr2=0x7fd9b00753b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:43:36.433 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:36.429+0000 7fd9cbeff640 1 --2- 172.21.15.123:0/3059418834 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fd9b0072ef0 0x7fd9b00753b0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fd9bc007fd0 tx=0x7fd9bc009040 comp rx=0 tx=0).stop 2024-09-16T15:43:36.433 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:36.429+0000 7fd9cbeff640 1 -- 172.21.15.123:0/3059418834 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd9c40cc260 msgr2=0x7fd9c40cd7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:43:36.434 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:36.429+0000 7fd9cbeff640 1 --2- 172.21.15.123:0/3059418834 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd9c40cc260 0x7fd9c40cd7b0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7fd9c000ec00 tx=0x7fd9c000c6a0 comp rx=0 tx=0).stop 2024-09-16T15:43:36.434 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:36.429+0000 7fd9cbeff640 1 -- 172.21.15.123:0/3059418834 shutdown_connections 2024-09-16T15:43:36.434 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:36.429+0000 7fd9cbeff640 1 --2- 172.21.15.123:0/3059418834 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fd9b0072ef0 0x7fd9b00753b0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:36.434 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:36.429+0000 7fd9cbeff640 1 --2- 172.21.15.123:0/3059418834 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fd9c415f9a0 0x7fd9c4097940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:36.434 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:36.429+0000 7fd9cbeff640 1 --2- 172.21.15.123:0/3059418834 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd9c40cc260 0x7fd9c40cd7b0 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:36.434 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:36.429+0000 7fd9cbeff640 1 --2- 172.21.15.123:0/3059418834 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd9c4074860 0x7fd9c40c9f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:36.434 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:36.429+0000 7fd9cbeff640 1 -- 172.21.15.123:0/3059418834 >> 172.21.15.123:0/3059418834 conn(0x7fd9c40732a0 msgr2=0x7fd9c4074150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:43:36.434 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:36.429+0000 7fd9cbeff640 1 -- 172.21.15.123:0/3059418834 shutdown_connections 2024-09-16T15:43:36.434 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:36.429+0000 7fd9cbeff640 1 -- 172.21.15.123:0/3059418834 wait complete. 2024-09-16T15:43:36.456 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-09-16T15:43:36.584 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:36.581+0000 7f50b7514640 1 -- 172.21.15.123:0/3237512915 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f50b015bca0 msgr2=0x7f50b00d0630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:43:36.585 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:36.581+0000 7f50b7514640 1 --2- 172.21.15.123:0/3237512915 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f50b015bca0 0x7f50b00d0630 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f50b006c970 tx=0x7f50ac01c740 comp rx=0 tx=0).stop 2024-09-16T15:43:36.585 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:36.581+0000 7f50b7514640 1 -- 172.21.15.123:0/3237512915 shutdown_connections 2024-09-16T15:43:36.585 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:36.581+0000 7f50b7514640 1 --2- 172.21.15.123:0/3237512915 >> v2:172.21.15.184:3300/0 conn(0x7f50b0074600 0x7f50b006e750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:36.585 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:36.581+0000 7f50b7514640 1 --2- 172.21.15.123:0/3237512915 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f50b015bca0 0x7f50b00d0630 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:36.585 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:36.581+0000 7f50b7514640 1 -- 172.21.15.123:0/3237512915 >> 172.21.15.123:0/3237512915 conn(0x7f50b00cfb10 msgr2=0x7f50b00cff20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:43:36.585 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:36.581+0000 7f50b7514640 1 -- 172.21.15.123:0/3237512915 shutdown_connections 2024-09-16T15:43:36.585 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:36.581+0000 7f50b7514640 1 -- 172.21.15.123:0/3237512915 wait complete. 2024-09-16T15:43:36.586 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:36.581+0000 7f50b7514640 1 Processor -- start 2024-09-16T15:43:36.586 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:36.581+0000 7f50b7514640 1 -- start start 2024-09-16T15:43:36.587 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:36.581+0000 7f50b7514640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f50b00726c0 0x7f50b015c7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:43:36.587 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:36.581+0000 7f50b7514640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f50b0074600 0x7f50b014e0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:43:36.587 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:36.581+0000 7f50b7514640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f50b015bca0 0x7f50b015cd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:43:36.588 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:36.581+0000 7f50b7514640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f50b0168bd0 con 0x7f50b0074600 2024-09-16T15:43:36.588 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:36.581+0000 7f50b7514640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f50b0168a50 con 0x7f50b015bca0 2024-09-16T15:43:36.588 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:36.581+0000 7f50b7514640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f50b0168d50 con 0x7f50b00726c0 2024-09-16T15:43:36.588 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:36.581+0000 7f50b4aaf640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f50b0074600 0x7f50b014e0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:43:36.588 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:36.581+0000 7f50b4aaf640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f50b0074600 0x7f50b014e0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:44734/0 (socket says 172.21.15.123:44734) 2024-09-16T15:43:36.588 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:36.581+0000 7f50b4aaf640 1 -- 172.21.15.123:0/1459644810 learned_addr learned my addr 172.21.15.123:0/1459644810 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:43:36.588 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:36.581+0000 7f50b4aaf640 1 -- 172.21.15.123:0/1459644810 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f50b00726c0 msgr2=0x7f50b015c7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:43:36.588 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:36.581+0000 7f50b4aaf640 1 --2- 172.21.15.123:0/1459644810 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f50b00726c0 0x7f50b015c7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:36.588 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:36.581+0000 7f50b4aaf640 1 -- 172.21.15.123:0/1459644810 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f50b015bca0 msgr2=0x7f50b015cd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:43:36.588 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:36.581+0000 7f50b4aaf640 1 --2- 172.21.15.123:0/1459644810 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f50b015bca0 0x7f50b015cd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:36.588 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:36.581+0000 7f50b4aaf640 1 -- 172.21.15.123:0/1459644810 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f50b015ee00 con 0x7f50b0074600 2024-09-16T15:43:36.588 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:36.585+0000 7f50b4aaf640 1 --2- 172.21.15.123:0/1459644810 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f50b0074600 0x7f50b014e0f0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f50a000c450 tx=0x7f50a000c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:43:36.588 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:36.585+0000 7f50a5ffb640 1 -- 172.21.15.123:0/1459644810 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50a0016020 con 0x7f50b0074600 2024-09-16T15:43:36.588 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:36.585+0000 7f50a5ffb640 1 -- 172.21.15.123:0/1459644810 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f50a0012920 con 0x7f50b0074600 2024-09-16T15:43:36.589 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:36.585+0000 7f50b7514640 1 -- 172.21.15.123:0/1459644810 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f50b015f0e0 con 0x7f50b0074600 2024-09-16T15:43:36.589 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:36.585+0000 7f50a5ffb640 1 -- 172.21.15.123:0/1459644810 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50a0012d70 con 0x7f50b0074600 2024-09-16T15:43:36.589 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:36.585+0000 7f50b7514640 1 -- 172.21.15.123:0/1459644810 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f50b00c8000 con 0x7f50b0074600 2024-09-16T15:43:36.589 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:36.585+0000 7f50b7514640 1 -- 172.21.15.123:0/1459644810 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f50b006e750 con 0x7f50b0074600 2024-09-16T15:43:36.590 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:36.585+0000 7f50a5ffb640 1 -- 172.21.15.123:0/1459644810 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 5) ==== 97080+0+0 (secure 0 0 0) 0x7f50a0019020 con 0x7f50b0074600 2024-09-16T15:43:36.590 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:36.585+0000 7f50a5ffb640 1 --2- 172.21.15.123:0/1459644810 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f5094072ec0 0x7f5094075380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:43:36.590 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:36.585+0000 7f50a5ffb640 1 -- 172.21.15.123:0/1459644810 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(29..29 src has 1..29) ==== 4857+0+0 (secure 0 0 0) 0x7f50a00939c0 con 0x7f50b0074600 2024-09-16T15:43:36.590 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:36.585+0000 7f50b52b0640 1 --2- 172.21.15.123:0/1459644810 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f5094072ec0 0x7f5094075380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:43:36.590 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:36.585+0000 7f50b52b0640 1 --2- 172.21.15.123:0/1459644810 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f5094072ec0 0x7f5094075380 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f50ac0af000 tx=0x7f50ac0194e0 comp rx=0 tx=0).ready entity=mgr.4135 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:43:36.593 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:36.589+0000 7f50a5ffb640 1 -- 172.21.15.123:0/1459644810 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f50a0095e20 con 0x7f50b0074600 2024-09-16T15:43:36.830 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:36.825+0000 7f50b7514640 1 -- 172.21.15.123:0/1459644810 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) -- 0x7f50b0070110 con 0x7f50b0074600 2024-09-16T15:43:36.831 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:36.829+0000 7f50a5ffb640 1 -- 172.21.15.123:0/1459644810 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v10) ==== 80+0+6645 (secure 0 0 0) 0x7f50a0020020 con 0x7f50b0074600 2024-09-16T15:43:36.831 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-09-16T15:43:36.831 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2024-09-16T15:43:36.831 INFO:teuthology.orchestra.run.smithi123.stdout: "auth_dump": [ 2024-09-16T15:43:36.831 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-09-16T15:43:36.831 INFO:teuthology.orchestra.run.smithi123.stdout: "entity": "mds.a", 2024-09-16T15:43:36.831 INFO:teuthology.orchestra.run.smithi123.stdout: "key": "AQC0UehmbZpOJBAAaYGyDcWrt4k0FS5P/bb+4A==", 2024-09-16T15:43:36.831 INFO:teuthology.orchestra.run.smithi123.stdout: "caps": { 2024-09-16T15:43:36.832 INFO:teuthology.orchestra.run.smithi123.stdout: "mds": "allow", 2024-09-16T15:43:36.832 INFO:teuthology.orchestra.run.smithi123.stdout: "mgr": "allow *", 2024-09-16T15:43:36.832 INFO:teuthology.orchestra.run.smithi123.stdout: "mon": "allow *", 2024-09-16T15:43:36.832 INFO:teuthology.orchestra.run.smithi123.stdout: "osd": "allow *" 2024-09-16T15:43:36.832 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-16T15:43:36.832 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:43:36.832 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-09-16T15:43:36.832 INFO:teuthology.orchestra.run.smithi123.stdout: "entity": "mds.b", 2024-09-16T15:43:36.832 INFO:teuthology.orchestra.run.smithi123.stdout: "key": "AQC0UehmF1HFMhAADcvqXNKdHZMPAbLc+se4Aw==", 2024-09-16T15:43:36.832 INFO:teuthology.orchestra.run.smithi123.stdout: "caps": { 2024-09-16T15:43:36.832 INFO:teuthology.orchestra.run.smithi123.stdout: "mds": "allow", 2024-09-16T15:43:36.832 INFO:teuthology.orchestra.run.smithi123.stdout: "mgr": "allow *", 2024-09-16T15:43:36.832 INFO:teuthology.orchestra.run.smithi123.stdout: "mon": "allow *", 2024-09-16T15:43:36.832 INFO:teuthology.orchestra.run.smithi123.stdout: "osd": "allow *" 2024-09-16T15:43:36.833 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-16T15:43:36.833 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:43:36.833 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-09-16T15:43:36.833 INFO:teuthology.orchestra.run.smithi123.stdout: "entity": "mds.c", 2024-09-16T15:43:36.833 INFO:teuthology.orchestra.run.smithi123.stdout: "key": "AQC0UehmCBjCLBAAXd85J25r7cgrXeDCG7pahg==", 2024-09-16T15:43:36.833 INFO:teuthology.orchestra.run.smithi123.stdout: "caps": { 2024-09-16T15:43:36.833 INFO:teuthology.orchestra.run.smithi123.stdout: "mds": "allow", 2024-09-16T15:43:36.833 INFO:teuthology.orchestra.run.smithi123.stdout: "mgr": "allow *", 2024-09-16T15:43:36.833 INFO:teuthology.orchestra.run.smithi123.stdout: "mon": "allow *", 2024-09-16T15:43:36.833 INFO:teuthology.orchestra.run.smithi123.stdout: "osd": "allow *" 2024-09-16T15:43:36.833 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-16T15:43:36.833 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:43:36.833 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-09-16T15:43:36.834 INFO:teuthology.orchestra.run.smithi123.stdout: "entity": "mds.d", 2024-09-16T15:43:36.834 INFO:teuthology.orchestra.run.smithi123.stdout: "key": "AQC0UehmMh3ZOBAAx0RCa3R9RMT93/icZhPGyg==", 2024-09-16T15:43:36.834 INFO:teuthology.orchestra.run.smithi123.stdout: "caps": { 2024-09-16T15:43:36.834 INFO:teuthology.orchestra.run.smithi123.stdout: "mds": "allow", 2024-09-16T15:43:36.834 INFO:teuthology.orchestra.run.smithi123.stdout: "mgr": "allow *", 2024-09-16T15:43:36.834 INFO:teuthology.orchestra.run.smithi123.stdout: "mon": "allow *", 2024-09-16T15:43:36.834 INFO:teuthology.orchestra.run.smithi123.stdout: "osd": "allow *" 2024-09-16T15:43:36.834 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-16T15:43:36.834 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:43:36.834 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-09-16T15:43:36.834 INFO:teuthology.orchestra.run.smithi123.stdout: "entity": "osd.0", 2024-09-16T15:43:36.834 INFO:teuthology.orchestra.run.smithi123.stdout: "key": "AQC4UehmT633CRAAIhpkeIBOzwJO0kiXLu0x9Q==", 2024-09-16T15:43:36.834 INFO:teuthology.orchestra.run.smithi123.stdout: "caps": { 2024-09-16T15:43:36.835 INFO:teuthology.orchestra.run.smithi123.stdout: "mgr": "allow profile osd", 2024-09-16T15:43:36.835 INFO:teuthology.orchestra.run.smithi123.stdout: "mon": "allow profile osd", 2024-09-16T15:43:36.835 INFO:teuthology.orchestra.run.smithi123.stdout: "osd": "allow *" 2024-09-16T15:43:36.835 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-16T15:43:36.835 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:43:36.835 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-09-16T15:43:36.835 INFO:teuthology.orchestra.run.smithi123.stdout: "entity": "osd.1", 2024-09-16T15:43:36.835 INFO:teuthology.orchestra.run.smithi123.stdout: "key": "AQC7UehmFwF7GBAAvOUG+rWjANKT8wm/ai4xGg==", 2024-09-16T15:43:36.835 INFO:teuthology.orchestra.run.smithi123.stdout: "caps": { 2024-09-16T15:43:36.835 INFO:teuthology.orchestra.run.smithi123.stdout: "mgr": "allow profile osd", 2024-09-16T15:43:36.835 INFO:teuthology.orchestra.run.smithi123.stdout: "mon": "allow profile osd", 2024-09-16T15:43:36.835 INFO:teuthology.orchestra.run.smithi123.stdout: "osd": "allow *" 2024-09-16T15:43:36.835 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-16T15:43:36.835 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:43:36.835 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-09-16T15:43:36.836 INFO:teuthology.orchestra.run.smithi123.stdout: "entity": "osd.2", 2024-09-16T15:43:36.836 INFO:teuthology.orchestra.run.smithi123.stdout: "key": "AQC+UehmlCwWKxAAl3iO7ZALWyaorQH2urHKCw==", 2024-09-16T15:43:36.836 INFO:teuthology.orchestra.run.smithi123.stdout: "caps": { 2024-09-16T15:43:36.836 INFO:teuthology.orchestra.run.smithi123.stdout: "mgr": "allow profile osd", 2024-09-16T15:43:36.836 INFO:teuthology.orchestra.run.smithi123.stdout: "mon": "allow profile osd", 2024-09-16T15:43:36.836 INFO:teuthology.orchestra.run.smithi123.stdout: "osd": "allow *" 2024-09-16T15:43:36.836 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-16T15:43:36.836 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:43:36.836 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-09-16T15:43:36.836 INFO:teuthology.orchestra.run.smithi123.stdout: "entity": "osd.3", 2024-09-16T15:43:36.836 INFO:teuthology.orchestra.run.smithi123.stdout: "key": "AQDCUehmD8VgAhAAneoIeGr3MflSmi+nyavj2A==", 2024-09-16T15:43:36.836 INFO:teuthology.orchestra.run.smithi123.stdout: "caps": { 2024-09-16T15:43:36.836 INFO:teuthology.orchestra.run.smithi123.stdout: "mgr": "allow profile osd", 2024-09-16T15:43:36.836 INFO:teuthology.orchestra.run.smithi123.stdout: "mon": "allow profile osd", 2024-09-16T15:43:36.837 INFO:teuthology.orchestra.run.smithi123.stdout: "osd": "allow *" 2024-09-16T15:43:36.837 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-16T15:43:36.837 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:43:36.837 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-09-16T15:43:36.837 INFO:teuthology.orchestra.run.smithi123.stdout: "entity": "osd.4", 2024-09-16T15:43:36.837 INFO:teuthology.orchestra.run.smithi123.stdout: "key": "AQDIUehmA9B9EBAAsu0X5LPx1TmatZeHzjnVig==", 2024-09-16T15:43:36.837 INFO:teuthology.orchestra.run.smithi123.stdout: "caps": { 2024-09-16T15:43:36.837 INFO:teuthology.orchestra.run.smithi123.stdout: "mgr": "allow profile osd", 2024-09-16T15:43:36.837 INFO:teuthology.orchestra.run.smithi123.stdout: "mon": "allow profile osd", 2024-09-16T15:43:36.837 INFO:teuthology.orchestra.run.smithi123.stdout: "osd": "allow *" 2024-09-16T15:43:36.837 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-16T15:43:36.837 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:43:36.837 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-09-16T15:43:36.837 INFO:teuthology.orchestra.run.smithi123.stdout: "entity": "osd.5", 2024-09-16T15:43:36.838 INFO:teuthology.orchestra.run.smithi123.stdout: "key": "AQDLUehmcugdHxAAQMSS1ZH71FUzCx5A+yNnZg==", 2024-09-16T15:43:36.838 INFO:teuthology.orchestra.run.smithi123.stdout: "caps": { 2024-09-16T15:43:36.838 INFO:teuthology.orchestra.run.smithi123.stdout: "mgr": "allow profile osd", 2024-09-16T15:43:36.838 INFO:teuthology.orchestra.run.smithi123.stdout: "mon": "allow profile osd", 2024-09-16T15:43:36.838 INFO:teuthology.orchestra.run.smithi123.stdout: "osd": "allow *" 2024-09-16T15:43:36.838 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-16T15:43:36.838 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:43:36.838 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-09-16T15:43:36.838 INFO:teuthology.orchestra.run.smithi123.stdout: "entity": "osd.6", 2024-09-16T15:43:36.838 INFO:teuthology.orchestra.run.smithi123.stdout: "key": "AQDOUehmWLwYLhAAhOlQLepcWElsVPMnHlZm4w==", 2024-09-16T15:43:36.838 INFO:teuthology.orchestra.run.smithi123.stdout: "caps": { 2024-09-16T15:43:36.838 INFO:teuthology.orchestra.run.smithi123.stdout: "mgr": "allow profile osd", 2024-09-16T15:43:36.838 INFO:teuthology.orchestra.run.smithi123.stdout: "mon": "allow profile osd", 2024-09-16T15:43:36.838 INFO:teuthology.orchestra.run.smithi123.stdout: "osd": "allow *" 2024-09-16T15:43:36.839 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-16T15:43:36.839 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:43:36.839 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-09-16T15:43:36.839 INFO:teuthology.orchestra.run.smithi123.stdout: "entity": "osd.7", 2024-09-16T15:43:36.839 INFO:teuthology.orchestra.run.smithi123.stdout: "key": "AQDSUehmipkbABAAQuL/2O7w0tsM1OKeOil97w==", 2024-09-16T15:43:36.839 INFO:teuthology.orchestra.run.smithi123.stdout: "caps": { 2024-09-16T15:43:36.839 INFO:teuthology.orchestra.run.smithi123.stdout: "mgr": "allow profile osd", 2024-09-16T15:43:36.839 INFO:teuthology.orchestra.run.smithi123.stdout: "mon": "allow profile osd", 2024-09-16T15:43:36.839 INFO:teuthology.orchestra.run.smithi123.stdout: "osd": "allow *" 2024-09-16T15:43:36.839 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-16T15:43:36.839 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:43:36.839 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-09-16T15:43:36.839 INFO:teuthology.orchestra.run.smithi123.stdout: "entity": "client.0", 2024-09-16T15:43:36.839 INFO:teuthology.orchestra.run.smithi123.stdout: "key": "AQC1Uehm7GDTAhAAHWWVIyjVVnTeeozBR8DfFA==", 2024-09-16T15:43:36.839 INFO:teuthology.orchestra.run.smithi123.stdout: "caps": { 2024-09-16T15:43:36.840 INFO:teuthology.orchestra.run.smithi123.stdout: "mds": "allow", 2024-09-16T15:43:36.840 INFO:teuthology.orchestra.run.smithi123.stdout: "mgr": "allow r", 2024-09-16T15:43:36.840 INFO:teuthology.orchestra.run.smithi123.stdout: "mon": "allow rw", 2024-09-16T15:43:36.840 INFO:teuthology.orchestra.run.smithi123.stdout: "osd": "allow rwx" 2024-09-16T15:43:36.840 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-16T15:43:36.840 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:43:36.840 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-09-16T15:43:36.840 INFO:teuthology.orchestra.run.smithi123.stdout: "entity": "client.1", 2024-09-16T15:43:36.840 INFO:teuthology.orchestra.run.smithi123.stdout: "key": "AQC1UehmFbaWChAAFtRmKucm+pbKtCNvRCTNqA==", 2024-09-16T15:43:36.840 INFO:teuthology.orchestra.run.smithi123.stdout: "caps": { 2024-09-16T15:43:36.840 INFO:teuthology.orchestra.run.smithi123.stdout: "mds": "allow", 2024-09-16T15:43:36.840 INFO:teuthology.orchestra.run.smithi123.stdout: "mgr": "allow r", 2024-09-16T15:43:36.840 INFO:teuthology.orchestra.run.smithi123.stdout: "mon": "allow rw", 2024-09-16T15:43:36.840 INFO:teuthology.orchestra.run.smithi123.stdout: "osd": "allow rwx" 2024-09-16T15:43:36.841 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-16T15:43:36.841 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:43:36.841 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-09-16T15:43:36.841 INFO:teuthology.orchestra.run.smithi123.stdout: "entity": "client.2", 2024-09-16T15:43:36.841 INFO:teuthology.orchestra.run.smithi123.stdout: "key": "AQC1Uehm1ADrBxAAW/8ayNHI6Wq/YlI4ivFZiw==", 2024-09-16T15:43:36.841 INFO:teuthology.orchestra.run.smithi123.stdout: "caps": { 2024-09-16T15:43:36.841 INFO:teuthology.orchestra.run.smithi123.stdout: "mds": "allow", 2024-09-16T15:43:36.841 INFO:teuthology.orchestra.run.smithi123.stdout: "mgr": "allow r", 2024-09-16T15:43:36.841 INFO:teuthology.orchestra.run.smithi123.stdout: "mon": "allow rw", 2024-09-16T15:43:36.841 INFO:teuthology.orchestra.run.smithi123.stdout: "osd": "allow rwx" 2024-09-16T15:43:36.841 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-16T15:43:36.841 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:43:36.841 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-09-16T15:43:36.841 INFO:teuthology.orchestra.run.smithi123.stdout: "entity": "client.3", 2024-09-16T15:43:36.842 INFO:teuthology.orchestra.run.smithi123.stdout: "key": "AQC1UehmLv/3DxAAuGloBxSuza95uajpfiZ9MQ==", 2024-09-16T15:43:36.842 INFO:teuthology.orchestra.run.smithi123.stdout: "caps": { 2024-09-16T15:43:36.842 INFO:teuthology.orchestra.run.smithi123.stdout: "mds": "allow", 2024-09-16T15:43:36.842 INFO:teuthology.orchestra.run.smithi123.stdout: "mgr": "allow r", 2024-09-16T15:43:36.842 INFO:teuthology.orchestra.run.smithi123.stdout: "mon": "allow rw", 2024-09-16T15:43:36.842 INFO:teuthology.orchestra.run.smithi123.stdout: "osd": "allow rwx" 2024-09-16T15:43:36.842 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-16T15:43:36.842 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:43:36.842 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-09-16T15:43:36.842 INFO:teuthology.orchestra.run.smithi123.stdout: "entity": "client.admin", 2024-09-16T15:43:36.842 INFO:teuthology.orchestra.run.smithi123.stdout: "key": "AQC0Uehm1ZDRDBAAhyQ1BmJg1Mw6wQ04I2CbFA==", 2024-09-16T15:43:36.842 INFO:teuthology.orchestra.run.smithi123.stdout: "caps": { 2024-09-16T15:43:36.842 INFO:teuthology.orchestra.run.smithi123.stdout: "mds": "allow *", 2024-09-16T15:43:36.842 INFO:teuthology.orchestra.run.smithi123.stdout: "mgr": "allow *", 2024-09-16T15:43:36.843 INFO:teuthology.orchestra.run.smithi123.stdout: "mon": "allow *", 2024-09-16T15:43:36.843 INFO:teuthology.orchestra.run.smithi123.stdout: "osd": "allow *" 2024-09-16T15:43:36.843 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-16T15:43:36.843 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:43:36.843 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-09-16T15:43:36.843 INFO:teuthology.orchestra.run.smithi123.stdout: "entity": "client.bootstrap-mds", 2024-09-16T15:43:36.843 INFO:teuthology.orchestra.run.smithi123.stdout: "key": "AQDiUehmTHXcFRAADQLvtDrR5JrOuCfC395Mxg==", 2024-09-16T15:43:36.843 INFO:teuthology.orchestra.run.smithi123.stdout: "caps": { 2024-09-16T15:43:36.843 INFO:teuthology.orchestra.run.smithi123.stdout: "mon": "allow profile bootstrap-mds" 2024-09-16T15:43:36.843 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-16T15:43:36.843 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:43:36.843 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-09-16T15:43:36.843 INFO:teuthology.orchestra.run.smithi123.stdout: "entity": "client.bootstrap-mgr", 2024-09-16T15:43:36.843 INFO:teuthology.orchestra.run.smithi123.stdout: "key": "AQDiUehmb5LcFRAAYsUX8T7QC/zhJV11AWoCzQ==", 2024-09-16T15:43:36.843 INFO:teuthology.orchestra.run.smithi123.stdout: "caps": { 2024-09-16T15:43:36.844 INFO:teuthology.orchestra.run.smithi123.stdout: "mon": "allow profile bootstrap-mgr" 2024-09-16T15:43:36.844 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-16T15:43:36.844 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:43:36.844 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-09-16T15:43:36.844 INFO:teuthology.orchestra.run.smithi123.stdout: "entity": "client.bootstrap-osd", 2024-09-16T15:43:36.844 INFO:teuthology.orchestra.run.smithi123.stdout: "key": "AQDiUehmjKncFRAA/R2q4xn3zN2qNblDkHoT9g==", 2024-09-16T15:43:36.844 INFO:teuthology.orchestra.run.smithi123.stdout: "caps": { 2024-09-16T15:43:36.844 INFO:teuthology.orchestra.run.smithi123.stdout: "mon": "allow profile bootstrap-osd" 2024-09-16T15:43:36.844 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-16T15:43:36.844 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:43:36.844 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-09-16T15:43:36.844 INFO:teuthology.orchestra.run.smithi123.stdout: "entity": "client.bootstrap-rbd", 2024-09-16T15:43:36.844 INFO:teuthology.orchestra.run.smithi123.stdout: "key": "AQDiUehmt8PcFRAAmFAX8TWjEstMwXwcSAPVMA==", 2024-09-16T15:43:36.844 INFO:teuthology.orchestra.run.smithi123.stdout: "caps": { 2024-09-16T15:43:36.845 INFO:teuthology.orchestra.run.smithi123.stdout: "mon": "allow profile bootstrap-rbd" 2024-09-16T15:43:36.845 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-16T15:43:36.845 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:43:36.845 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-09-16T15:43:36.845 INFO:teuthology.orchestra.run.smithi123.stdout: "entity": "client.bootstrap-rbd-mirror", 2024-09-16T15:43:36.845 INFO:teuthology.orchestra.run.smithi123.stdout: "key": "AQDiUehmBtvcFRAANHUJMZPmoegv0jNxf4WQDQ==", 2024-09-16T15:43:36.845 INFO:teuthology.orchestra.run.smithi123.stdout: "caps": { 2024-09-16T15:43:36.845 INFO:teuthology.orchestra.run.smithi123.stdout: "mon": "allow profile bootstrap-rbd-mirror" 2024-09-16T15:43:36.845 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-16T15:43:36.845 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:43:36.845 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-09-16T15:43:36.845 INFO:teuthology.orchestra.run.smithi123.stdout: "entity": "client.bootstrap-rgw", 2024-09-16T15:43:36.845 INFO:teuthology.orchestra.run.smithi123.stdout: "key": "AQDiUehm0PLcFRAAeo/O+HXOSfv54X4z2bH4cg==", 2024-09-16T15:43:36.845 INFO:teuthology.orchestra.run.smithi123.stdout: "caps": { 2024-09-16T15:43:36.846 INFO:teuthology.orchestra.run.smithi123.stdout: "mon": "allow profile bootstrap-rgw" 2024-09-16T15:43:36.846 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-16T15:43:36.846 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:43:36.846 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-09-16T15:43:36.846 INFO:teuthology.orchestra.run.smithi123.stdout: "entity": "mgr.x", 2024-09-16T15:43:36.846 INFO:teuthology.orchestra.run.smithi123.stdout: "key": "AQC0UehmfPilIBAAERlBbduOgAU9ba0n/1tl3g==", 2024-09-16T15:43:36.846 INFO:teuthology.orchestra.run.smithi123.stdout: "caps": { 2024-09-16T15:43:36.846 INFO:teuthology.orchestra.run.smithi123.stdout: "mds": "allow *", 2024-09-16T15:43:36.846 INFO:teuthology.orchestra.run.smithi123.stdout: "mon": "allow profile mgr", 2024-09-16T15:43:36.846 INFO:teuthology.orchestra.run.smithi123.stdout: "osd": "allow *" 2024-09-16T15:43:36.846 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-16T15:43:36.846 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:43:36.846 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-09-16T15:43:36.846 INFO:teuthology.orchestra.run.smithi123.stdout: "entity": "mgr.y", 2024-09-16T15:43:36.846 INFO:teuthology.orchestra.run.smithi123.stdout: "key": "AQC0UehmkjrlHRAA+3xbQC+SPWGWW1gND7xcAg==", 2024-09-16T15:43:36.847 INFO:teuthology.orchestra.run.smithi123.stdout: "caps": { 2024-09-16T15:43:36.847 INFO:teuthology.orchestra.run.smithi123.stdout: "mds": "allow *", 2024-09-16T15:43:36.847 INFO:teuthology.orchestra.run.smithi123.stdout: "mon": "allow profile mgr", 2024-09-16T15:43:36.847 INFO:teuthology.orchestra.run.smithi123.stdout: "osd": "allow *" 2024-09-16T15:43:36.847 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-16T15:43:36.847 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-16T15:43:36.847 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-09-16T15:43:36.847 INFO:teuthology.orchestra.run.smithi123.stdout:} 2024-09-16T15:43:36.848 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:36.829+0000 7f50b7514640 1 -- 172.21.15.123:0/1459644810 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f5094072ec0 msgr2=0x7f5094075380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:43:36.848 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:36.829+0000 7f50b7514640 1 --2- 172.21.15.123:0/1459644810 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f5094072ec0 0x7f5094075380 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f50ac0af000 tx=0x7f50ac0194e0 comp rx=0 tx=0).stop 2024-09-16T15:43:36.848 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:36.829+0000 7f50b7514640 1 -- 172.21.15.123:0/1459644810 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f50b0074600 msgr2=0x7f50b014e0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:43:36.848 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:36.829+0000 7f50b7514640 1 --2- 172.21.15.123:0/1459644810 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f50b0074600 0x7f50b014e0f0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f50a000c450 tx=0x7f50a000c910 comp rx=0 tx=0).stop 2024-09-16T15:43:36.848 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:36.829+0000 7f50b7514640 1 -- 172.21.15.123:0/1459644810 shutdown_connections 2024-09-16T15:43:36.848 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:36.829+0000 7f50b7514640 1 --2- 172.21.15.123:0/1459644810 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f5094072ec0 0x7f5094075380 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:36.849 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:36.829+0000 7f50b7514640 1 --2- 172.21.15.123:0/1459644810 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f50b015bca0 0x7f50b015cd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:36.849 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:36.829+0000 7f50b7514640 1 --2- 172.21.15.123:0/1459644810 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f50b0074600 0x7f50b014e0f0 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:36.849 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:36.829+0000 7f50b7514640 1 --2- 172.21.15.123:0/1459644810 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f50b00726c0 0x7f50b015c7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:36.849 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:36.829+0000 7f50b7514640 1 -- 172.21.15.123:0/1459644810 >> 172.21.15.123:0/1459644810 conn(0x7f50b00cfb10 msgr2=0x7f50b00b4720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:43:36.849 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:36.829+0000 7f50b7514640 1 -- 172.21.15.123:0/1459644810 shutdown_connections 2024-09-16T15:43:36.849 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:36.829+0000 7f50b7514640 1 -- 172.21.15.123:0/1459644810 wait complete. 2024-09-16T15:43:36.850 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-mds 2024-09-16T15:43:36.966 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:36.961+0000 7f08d01b8640 1 -- 172.21.15.123:0/4185939843 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f08c80cf5a0 msgr2=0x7f08c806eee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:43:36.966 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:36.961+0000 7f08d01b8640 1 --2- 172.21.15.123:0/4185939843 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f08c80cf5a0 0x7f08c806eee0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f08b8009840 tx=0x7f08b801c700 comp rx=0 tx=0).stop 2024-09-16T15:43:36.966 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:36.961+0000 7f08d01b8640 1 -- 172.21.15.123:0/4185939843 shutdown_connections 2024-09-16T15:43:36.966 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:36.961+0000 7f08d01b8640 1 --2- 172.21.15.123:0/4185939843 >> v2:172.21.15.184:3300/0 conn(0x7f08c815ec40 0x7f08c806f420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:36.966 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:36.961+0000 7f08d01b8640 1 --2- 172.21.15.123:0/4185939843 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f08c80cf5a0 0x7f08c806eee0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:36.966 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:36.961+0000 7f08d01b8640 1 -- 172.21.15.123:0/4185939843 >> 172.21.15.123:0/4185939843 conn(0x7f08c80cfb10 msgr2=0x7f08c80cff20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:43:36.966 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:36.961+0000 7f08d01b8640 1 -- 172.21.15.123:0/4185939843 shutdown_connections 2024-09-16T15:43:36.966 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:36.961+0000 7f08d01b8640 1 -- 172.21.15.123:0/4185939843 wait complete. 2024-09-16T15:43:36.967 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:36.961+0000 7f08d01b8640 1 Processor -- start 2024-09-16T15:43:36.967 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:36.961+0000 7f08d01b8640 1 -- start start 2024-09-16T15:43:36.969 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:36.965+0000 7f08d01b8640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f08c80cf5a0 0x7f08c8148ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:43:36.969 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:36.965+0000 7f08d01b8640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f08c8158120 0x7f08c8149910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:43:36.969 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:36.965+0000 7f08d01b8640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f08c815ec40 0x7f08c814b180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:43:36.969 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:36.965+0000 7f08d01b8640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f08c815b280 con 0x7f08c80cf5a0 2024-09-16T15:43:36.969 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:36.965+0000 7f08d01b8640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f08c815b100 con 0x7f08c8158120 2024-09-16T15:43:36.969 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:36.965+0000 7f08d01b8640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f08c815b400 con 0x7f08c815ec40 2024-09-16T15:43:36.970 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:36.965+0000 7f08cdf54640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f08c80cf5a0 0x7f08c8148ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:43:36.970 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:36.965+0000 7f08cdf54640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f08c80cf5a0 0x7f08c8148ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:44740/0 (socket says 172.21.15.123:44740) 2024-09-16T15:43:36.970 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:36.965+0000 7f08cdf54640 1 -- 172.21.15.123:0/1008897589 learned_addr learned my addr 172.21.15.123:0/1008897589 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:43:36.970 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:36.965+0000 7f08ce755640 1 --2- 172.21.15.123:0/1008897589 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f08c815ec40 0x7f08c814b180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:43:36.970 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:36.965+0000 7f08cd753640 1 --2- 172.21.15.123:0/1008897589 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f08c8158120 0x7f08c8149910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:43:36.970 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:36.965+0000 7f08cdf54640 1 -- 172.21.15.123:0/1008897589 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f08c815ec40 msgr2=0x7f08c814b180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:43:36.970 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:36.965+0000 7f08cdf54640 1 --2- 172.21.15.123:0/1008897589 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f08c815ec40 0x7f08c814b180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:36.970 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:36.965+0000 7f08cdf54640 1 -- 172.21.15.123:0/1008897589 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f08c8158120 msgr2=0x7f08c8149910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:43:36.970 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:36.965+0000 7f08cdf54640 1 --2- 172.21.15.123:0/1008897589 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f08c8158120 0x7f08c8149910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:36.970 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:36.965+0000 7f08cdf54640 1 -- 172.21.15.123:0/1008897589 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f08c814a3d0 con 0x7f08c80cf5a0 2024-09-16T15:43:36.970 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:36.965+0000 7f08cdf54640 1 --2- 172.21.15.123:0/1008897589 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f08c80cf5a0 0x7f08c8148ae0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f08c400c490 tx=0x7f08c400c950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:43:36.970 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:36.965+0000 7f08be7fc640 1 -- 172.21.15.123:0/1008897589 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08c4016020 con 0x7f08c80cf5a0 2024-09-16T15:43:36.970 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:36.965+0000 7f08be7fc640 1 -- 172.21.15.123:0/1008897589 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f08c4012940 con 0x7f08c80cf5a0 2024-09-16T15:43:36.971 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:36.965+0000 7f08d01b8640 1 -- 172.21.15.123:0/1008897589 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f08c814a6b0 con 0x7f08c80cf5a0 2024-09-16T15:43:36.971 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:36.965+0000 7f08be7fc640 1 -- 172.21.15.123:0/1008897589 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08c4012d90 con 0x7f08c80cf5a0 2024-09-16T15:43:36.971 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:36.965+0000 7f08d01b8640 1 -- 172.21.15.123:0/1008897589 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f08c80728b0 con 0x7f08c80cf5a0 2024-09-16T15:43:36.972 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:36.965+0000 7f08d01b8640 1 -- 172.21.15.123:0/1008897589 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f08c814abd0 con 0x7f08c80cf5a0 2024-09-16T15:43:36.972 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:36.965+0000 7f08be7fc640 1 -- 172.21.15.123:0/1008897589 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 5) ==== 97080+0+0 (secure 0 0 0) 0x7f08c4019020 con 0x7f08c80cf5a0 2024-09-16T15:43:36.972 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:36.965+0000 7f08be7fc640 1 --2- 172.21.15.123:0/1008897589 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f08ac072ef0 0x7f08ac0753b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:43:36.972 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:36.969+0000 7f08be7fc640 1 -- 172.21.15.123:0/1008897589 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(29..29 src has 1..29) ==== 4857+0+0 (secure 0 0 0) 0x7f08c40939a0 con 0x7f08c80cf5a0 2024-09-16T15:43:36.972 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:36.969+0000 7f08cd753640 1 --2- 172.21.15.123:0/1008897589 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f08ac072ef0 0x7f08ac0753b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:43:36.972 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:36.969+0000 7f08cd753640 1 --2- 172.21.15.123:0/1008897589 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f08ac072ef0 0x7f08ac0753b0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f08b80ae000 tx=0x7f08b801af40 comp rx=0 tx=0).ready entity=mgr.4135 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:43:36.975 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:36.973+0000 7f08be7fc640 1 -- 172.21.15.123:0/1008897589 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f08c814abd0 con 0x7f08c80cf5a0 2024-09-16T15:43:37.202 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:37.197+0000 7f08d01b8640 1 -- 172.21.15.123:0/1008897589 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-mds"} v 0) -- 0x7f08c80ca320 con 0x7f08c80cf5a0 2024-09-16T15:43:37.206 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:37.201+0000 7f08be7fc640 1 -- 172.21.15.123:0/1008897589 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-mds"}]=0 v11) ==== 90+0+0 (secure 0 0 0) 0x7f08c4020020 con 0x7f08c80cf5a0 2024-09-16T15:43:37.208 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:37.205+0000 7f08d01b8640 1 -- 172.21.15.123:0/1008897589 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f08ac072ef0 msgr2=0x7f08ac0753b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:43:37.209 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:37.205+0000 7f08d01b8640 1 --2- 172.21.15.123:0/1008897589 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f08ac072ef0 0x7f08ac0753b0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f08b80ae000 tx=0x7f08b801af40 comp rx=0 tx=0).stop 2024-09-16T15:43:37.209 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:37.205+0000 7f08d01b8640 1 -- 172.21.15.123:0/1008897589 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f08c80cf5a0 msgr2=0x7f08c8148ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:43:37.209 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:37.205+0000 7f08d01b8640 1 --2- 172.21.15.123:0/1008897589 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f08c80cf5a0 0x7f08c8148ae0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f08c400c490 tx=0x7f08c400c950 comp rx=0 tx=0).stop 2024-09-16T15:43:37.209 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:37.205+0000 7f08d01b8640 1 -- 172.21.15.123:0/1008897589 shutdown_connections 2024-09-16T15:43:37.209 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:37.205+0000 7f08d01b8640 1 --2- 172.21.15.123:0/1008897589 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f08ac072ef0 0x7f08ac0753b0 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:37.209 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:37.205+0000 7f08d01b8640 1 --2- 172.21.15.123:0/1008897589 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f08c815ec40 0x7f08c814b180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:37.209 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:37.205+0000 7f08d01b8640 1 --2- 172.21.15.123:0/1008897589 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f08c8158120 0x7f08c8149910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:37.209 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:37.205+0000 7f08d01b8640 1 --2- 172.21.15.123:0/1008897589 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f08c80cf5a0 0x7f08c8148ae0 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:37.209 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:37.205+0000 7f08d01b8640 1 -- 172.21.15.123:0/1008897589 >> 172.21.15.123:0/1008897589 conn(0x7f08c80cfb10 msgr2=0x7f08c815d690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:43:37.209 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:37.205+0000 7f08d01b8640 1 -- 172.21.15.123:0/1008897589 shutdown_connections 2024-09-16T15:43:37.209 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:37.205+0000 7f08d01b8640 1 -- 172.21.15.123:0/1008897589 wait complete. 2024-09-16T15:43:37.223 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-mgr 2024-09-16T15:43:37.337 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:37.333+0000 7f5887195640 1 -- 172.21.15.123:0/2521463629 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f58680052d0 msgr2=0x7f586800b9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:43:37.337 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:37.333+0000 7f5887195640 1 --2- 172.21.15.123:0/2521463629 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f58680052d0 0x7f586800b9d0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f587800c610 tx=0x7f587801f9c0 comp rx=0 tx=0).stop 2024-09-16T15:43:37.338 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:37.333+0000 7f5887195640 1 -- 172.21.15.123:0/2521463629 shutdown_connections 2024-09-16T15:43:37.338 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:37.333+0000 7f5887195640 1 --2- 172.21.15.123:0/2521463629 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f58680052d0 0x7f586800b9d0 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:37.338 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:37.333+0000 7f5887195640 1 -- 172.21.15.123:0/2521463629 >> 172.21.15.123:0/2521463629 conn(0x7f58800cfb10 msgr2=0x7f58800cff20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:43:37.338 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:37.333+0000 7f5887195640 1 -- 172.21.15.123:0/2521463629 shutdown_connections 2024-09-16T15:43:37.338 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:37.333+0000 7f5887195640 1 -- 172.21.15.123:0/2521463629 wait complete. 2024-09-16T15:43:37.338 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:37.333+0000 7f5887195640 1 Processor -- start 2024-09-16T15:43:37.338 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:37.333+0000 7f5887195640 1 -- start start 2024-09-16T15:43:37.340 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:37.337+0000 7f5887195640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f58680052d0 0x7f588014d710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:43:37.340 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:37.337+0000 7f5887195640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f58800ad8c0 0x7f58800c7c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:43:37.340 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:37.337+0000 7f5887195640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f58800c8170 0x7f58800b02f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:43:37.340 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:37.337+0000 7f5887195640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f58800d09c0 con 0x7f58800ad8c0 2024-09-16T15:43:37.340 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:37.337+0000 7f5887195640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f58800d0840 con 0x7f58680052d0 2024-09-16T15:43:37.340 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:37.337+0000 7f5887195640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f5880073ef0 con 0x7f58800c8170 2024-09-16T15:43:37.340 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:37.337+0000 7f587ffff640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f58800ad8c0 0x7f58800c7c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:43:37.341 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:37.337+0000 7f587ffff640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f58800ad8c0 0x7f58800c7c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:44758/0 (socket says 172.21.15.123:44758) 2024-09-16T15:43:37.341 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:37.337+0000 7f587ffff640 1 -- 172.21.15.123:0/2609467985 learned_addr learned my addr 172.21.15.123:0/2609467985 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:43:37.341 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:37.337+0000 7f587ffff640 1 -- 172.21.15.123:0/2609467985 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f58800c8170 msgr2=0x7f58800b02f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:43:37.341 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:37.337+0000 7f5884f31640 1 --2- 172.21.15.123:0/2609467985 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f58680052d0 0x7f588014d710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:43:37.341 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:37.337+0000 7f587ffff640 1 --2- 172.21.15.123:0/2609467985 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f58800c8170 0x7f58800b02f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:37.341 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:37.337+0000 7f587ffff640 1 -- 172.21.15.123:0/2609467985 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f58680052d0 msgr2=0x7f588014d710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:43:37.341 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:37.337+0000 7f587ffff640 1 --2- 172.21.15.123:0/2609467985 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f58680052d0 0x7f588014d710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:37.341 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:37.337+0000 7f587ffff640 1 -- 172.21.15.123:0/2609467985 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f588014dc50 con 0x7f58800ad8c0 2024-09-16T15:43:37.341 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:37.337+0000 7f587ffff640 1 --2- 172.21.15.123:0/2609467985 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f58800ad8c0 0x7f58800c7c30 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f5874004da0 tx=0x7f5874001640 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:43:37.341 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:37.337+0000 7f587d7fa640 1 -- 172.21.15.123:0/2609467985 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5874001d50 con 0x7f58800ad8c0 2024-09-16T15:43:37.341 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:37.337+0000 7f587d7fa640 1 -- 172.21.15.123:0/2609467985 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f5874016070 con 0x7f58800ad8c0 2024-09-16T15:43:37.341 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:37.337+0000 7f5887195640 1 -- 172.21.15.123:0/2609467985 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f58800af530 con 0x7f58800ad8c0 2024-09-16T15:43:37.341 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:37.337+0000 7f587d7fa640 1 -- 172.21.15.123:0/2609467985 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5874010b80 con 0x7f58800ad8c0 2024-09-16T15:43:37.342 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:37.337+0000 7f5887195640 1 -- 172.21.15.123:0/2609467985 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f58800afaf0 con 0x7f58800ad8c0 2024-09-16T15:43:37.342 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:37.337+0000 7f5887195640 1 -- 172.21.15.123:0/2609467985 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f58800ca160 con 0x7f58800ad8c0 2024-09-16T15:43:37.342 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:37.337+0000 7f587d7fa640 1 -- 172.21.15.123:0/2609467985 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 5) ==== 97080+0+0 (secure 0 0 0) 0x7f587401b460 con 0x7f58800ad8c0 2024-09-16T15:43:37.342 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:37.337+0000 7f587d7fa640 1 --2- 172.21.15.123:0/2609467985 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f586c072ef0 0x7f586c0753b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:43:37.342 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:37.337+0000 7f587d7fa640 1 -- 172.21.15.123:0/2609467985 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(29..29 src has 1..29) ==== 4857+0+0 (secure 0 0 0) 0x7f58740942d0 con 0x7f58800ad8c0 2024-09-16T15:43:37.343 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:37.337+0000 7f5884f31640 1 --2- 172.21.15.123:0/2609467985 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f586c072ef0 0x7f586c0753b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:43:37.343 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:37.341+0000 7f5884f31640 1 --2- 172.21.15.123:0/2609467985 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f586c072ef0 0x7f586c0753b0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f58700057d0 tx=0x7f5870005860 comp rx=0 tx=0).ready entity=mgr.4135 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:43:37.346 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:37.341+0000 7f587d7fa640 1 -- 172.21.15.123:0/2609467985 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f588014dc50 con 0x7f58800ad8c0 2024-09-16T15:43:37.574 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:37.569+0000 7f5887195640 1 -- 172.21.15.123:0/2609467985 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-mgr"} v 0) -- 0x7f588015f630 con 0x7f58800ad8c0 2024-09-16T15:43:37.578 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:37.573+0000 7f587d7fa640 1 -- 172.21.15.123:0/2609467985 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-mgr"}]=0 v12) ==== 90+0+0 (secure 0 0 0) 0x7f587405f190 con 0x7f58800ad8c0 2024-09-16T15:43:37.580 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:37.577+0000 7f5887195640 1 -- 172.21.15.123:0/2609467985 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f586c072ef0 msgr2=0x7f586c0753b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:43:37.581 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:37.577+0000 7f5887195640 1 --2- 172.21.15.123:0/2609467985 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f586c072ef0 0x7f586c0753b0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f58700057d0 tx=0x7f5870005860 comp rx=0 tx=0).stop 2024-09-16T15:43:37.581 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:37.577+0000 7f5887195640 1 -- 172.21.15.123:0/2609467985 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f58800ad8c0 msgr2=0x7f58800c7c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:43:37.581 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:37.577+0000 7f5887195640 1 --2- 172.21.15.123:0/2609467985 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f58800ad8c0 0x7f58800c7c30 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f5874004da0 tx=0x7f5874001640 comp rx=0 tx=0).stop 2024-09-16T15:43:37.581 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:37.577+0000 7f5887195640 1 -- 172.21.15.123:0/2609467985 shutdown_connections 2024-09-16T15:43:37.581 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:37.577+0000 7f5887195640 1 --2- 172.21.15.123:0/2609467985 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f586c072ef0 0x7f586c0753b0 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:37.581 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:37.577+0000 7f5887195640 1 --2- 172.21.15.123:0/2609467985 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f58800c8170 0x7f58800b02f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:37.581 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:37.577+0000 7f5887195640 1 --2- 172.21.15.123:0/2609467985 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f58800ad8c0 0x7f58800c7c30 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:37.581 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:37.577+0000 7f5887195640 1 --2- 172.21.15.123:0/2609467985 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f58680052d0 0x7f588014d710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:37.581 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:37.577+0000 7f5887195640 1 -- 172.21.15.123:0/2609467985 >> 172.21.15.123:0/2609467985 conn(0x7f58800cfb10 msgr2=0x7f58800cf160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:43:37.581 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:37.577+0000 7f5887195640 1 -- 172.21.15.123:0/2609467985 shutdown_connections 2024-09-16T15:43:37.581 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:37.577+0000 7f5887195640 1 -- 172.21.15.123:0/2609467985 wait complete. 2024-09-16T15:43:37.596 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-osd 2024-09-16T15:43:37.723 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:37.717+0000 7f5d06c0f640 1 -- 172.21.15.123:0/3552600776 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5d0006ce50 msgr2=0x7f5d000d00e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:43:37.723 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:37.717+0000 7f5d06c0f640 1 --2- 172.21.15.123:0/3552600776 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5d0006ce50 0x7f5d000d00e0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f5cf00098f0 tx=0x7f5cf001c8b0 comp rx=0 tx=0).stop 2024-09-16T15:43:37.723 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:37.717+0000 7f5d06c0f640 1 -- 172.21.15.123:0/3552600776 shutdown_connections 2024-09-16T15:43:37.723 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:37.717+0000 7f5d06c0f640 1 --2- 172.21.15.123:0/3552600776 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f5d00073600 0x7f5d00152e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:37.723 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:37.717+0000 7f5d06c0f640 1 --2- 172.21.15.123:0/3552600776 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5d0006ce50 0x7f5d000d00e0 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:37.723 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:37.717+0000 7f5d06c0f640 1 -- 172.21.15.123:0/3552600776 >> 172.21.15.123:0/3552600776 conn(0x7f5d000cd5d0 msgr2=0x7f5d000cd9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:43:37.723 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:37.717+0000 7f5d06c0f640 1 -- 172.21.15.123:0/3552600776 shutdown_connections 2024-09-16T15:43:37.723 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:37.717+0000 7f5d06c0f640 1 -- 172.21.15.123:0/3552600776 wait complete. 2024-09-16T15:43:37.724 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:37.721+0000 7f5d06c0f640 1 Processor -- start 2024-09-16T15:43:37.724 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:37.721+0000 7f5d06c0f640 1 -- start start 2024-09-16T15:43:37.726 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:37.721+0000 7f5d06c0f640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f5d0006ce50 0x7f5d000c39e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:43:37.726 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:37.721+0000 7f5d06c0f640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5d00073600 0x7f5d001533a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:43:37.726 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:37.721+0000 7f5d06c0f640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5d000d0620 0x7f5d001538e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:43:37.726 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:37.721+0000 7f5d06c0f640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f5d0015ca10 con 0x7f5d00073600 2024-09-16T15:43:37.726 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:37.721+0000 7f5d06c0f640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f5d0015c890 con 0x7f5d000d0620 2024-09-16T15:43:37.726 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:37.721+0000 7f5d06c0f640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f5d0015cb90 con 0x7f5d0006ce50 2024-09-16T15:43:37.726 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:37.721+0000 7f5cfffff640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5d00073600 0x7f5d001533a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:43:37.726 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:37.721+0000 7f5cfffff640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5d00073600 0x7f5d001533a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:44774/0 (socket says 172.21.15.123:44774) 2024-09-16T15:43:37.726 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:37.721+0000 7f5cfffff640 1 -- 172.21.15.123:0/3582317389 learned_addr learned my addr 172.21.15.123:0/3582317389 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:43:37.726 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:37.721+0000 7f5d049ab640 1 --2- 172.21.15.123:0/3582317389 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f5d0006ce50 0x7f5d000c39e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:43:37.727 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:37.721+0000 7f5d051ac640 1 --2- 172.21.15.123:0/3582317389 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5d000d0620 0x7f5d001538e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:43:37.727 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:37.721+0000 7f5cfffff640 1 -- 172.21.15.123:0/3582317389 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f5d0006ce50 msgr2=0x7f5d000c39e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:43:37.727 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:37.721+0000 7f5cfffff640 1 --2- 172.21.15.123:0/3582317389 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f5d0006ce50 0x7f5d000c39e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:37.727 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:37.721+0000 7f5cfffff640 1 -- 172.21.15.123:0/3582317389 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5d000d0620 msgr2=0x7f5d001538e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:43:37.727 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:37.721+0000 7f5cfffff640 1 --2- 172.21.15.123:0/3582317389 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5d000d0620 0x7f5d001538e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:37.727 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:37.721+0000 7f5cfffff640 1 -- 172.21.15.123:0/3582317389 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5d000c3f20 con 0x7f5d00073600 2024-09-16T15:43:37.727 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:37.721+0000 7f5cfffff640 1 --2- 172.21.15.123:0/3582317389 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5d00073600 0x7f5d001533a0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f5cf400b3e0 tx=0x7f5cf400b8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:43:37.727 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:37.721+0000 7f5cfd7fa640 1 -- 172.21.15.123:0/3582317389 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5cf4015020 con 0x7f5d00073600 2024-09-16T15:43:37.727 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:37.721+0000 7f5cfd7fa640 1 -- 172.21.15.123:0/3582317389 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f5cf4011920 con 0x7f5d00073600 2024-09-16T15:43:37.727 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:37.721+0000 7f5d06c0f640 1 -- 172.21.15.123:0/3582317389 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5d000b6850 con 0x7f5d00073600 2024-09-16T15:43:37.727 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:37.721+0000 7f5cfd7fa640 1 -- 172.21.15.123:0/3582317389 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5cf4011db0 con 0x7f5d00073600 2024-09-16T15:43:37.727 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:37.721+0000 7f5d06c0f640 1 -- 172.21.15.123:0/3582317389 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5d0014e840 con 0x7f5d00073600 2024-09-16T15:43:37.728 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:37.721+0000 7f5d06c0f640 1 -- 172.21.15.123:0/3582317389 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5d0015d120 con 0x7f5d00073600 2024-09-16T15:43:37.728 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:37.725+0000 7f5cfd7fa640 1 -- 172.21.15.123:0/3582317389 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 5) ==== 97080+0+0 (secure 0 0 0) 0x7f5cf4018020 con 0x7f5d00073600 2024-09-16T15:43:37.728 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:37.725+0000 7f5cfd7fa640 1 --2- 172.21.15.123:0/3582317389 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f5cec072ec0 0x7f5cec075380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:43:37.729 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:37.725+0000 7f5cfd7fa640 1 -- 172.21.15.123:0/3582317389 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(29..29 src has 1..29) ==== 4857+0+0 (secure 0 0 0) 0x7f5cf4093680 con 0x7f5d00073600 2024-09-16T15:43:37.729 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:37.725+0000 7f5d049ab640 1 --2- 172.21.15.123:0/3582317389 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f5cec072ec0 0x7f5cec075380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:43:37.729 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:37.725+0000 7f5d049ab640 1 --2- 172.21.15.123:0/3582317389 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f5cec072ec0 0x7f5cec075380 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f5cf00af000 tx=0x7f5cf001b0f0 comp rx=0 tx=0).ready entity=mgr.4135 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:43:37.732 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:37.729+0000 7f5cfd7fa640 1 -- 172.21.15.123:0/3582317389 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f5cf401f030 con 0x7f5d00073600 2024-09-16T15:43:37.753 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b has been restored 2024-09-16T15:43:37.953 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:37.949+0000 7f5d06c0f640 1 -- 172.21.15.123:0/3582317389 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-osd"} v 0) -- 0x7f5d000b0810 con 0x7f5d00073600 2024-09-16T15:43:37.956 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:37.953+0000 7f5cfd7fa640 1 -- 172.21.15.123:0/3582317389 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-osd"}]=0 v13) ==== 90+0+0 (secure 0 0 0) 0x7f5cf405ec20 con 0x7f5d00073600 2024-09-16T15:43:37.959 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:37.953+0000 7f5d06c0f640 1 -- 172.21.15.123:0/3582317389 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f5cec072ec0 msgr2=0x7f5cec075380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:43:37.959 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:37.953+0000 7f5d06c0f640 1 --2- 172.21.15.123:0/3582317389 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f5cec072ec0 0x7f5cec075380 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f5cf00af000 tx=0x7f5cf001b0f0 comp rx=0 tx=0).stop 2024-09-16T15:43:37.959 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:37.953+0000 7f5d06c0f640 1 -- 172.21.15.123:0/3582317389 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5d00073600 msgr2=0x7f5d001533a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:43:37.959 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:37.953+0000 7f5d06c0f640 1 --2- 172.21.15.123:0/3582317389 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5d00073600 0x7f5d001533a0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f5cf400b3e0 tx=0x7f5cf400b8a0 comp rx=0 tx=0).stop 2024-09-16T15:43:37.959 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:37.953+0000 7f5d06c0f640 1 -- 172.21.15.123:0/3582317389 shutdown_connections 2024-09-16T15:43:37.959 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:37.953+0000 7f5d06c0f640 1 --2- 172.21.15.123:0/3582317389 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f5cec072ec0 0x7f5cec075380 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:37.959 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:37.953+0000 7f5d06c0f640 1 --2- 172.21.15.123:0/3582317389 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5d000d0620 0x7f5d001538e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:37.959 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:37.953+0000 7f5d06c0f640 1 --2- 172.21.15.123:0/3582317389 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5d00073600 0x7f5d001533a0 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:37.959 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:37.953+0000 7f5d06c0f640 1 --2- 172.21.15.123:0/3582317389 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f5d0006ce50 0x7f5d000c39e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:37.959 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:37.953+0000 7f5d06c0f640 1 -- 172.21.15.123:0/3582317389 >> 172.21.15.123:0/3582317389 conn(0x7f5d000cd5d0 msgr2=0x7f5d00145770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:43:37.960 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:37.953+0000 7f5d06c0f640 1 -- 172.21.15.123:0/3582317389 shutdown_connections 2024-09-16T15:43:37.960 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:37.953+0000 7f5d06c0f640 1 -- 172.21.15.123:0/3582317389 wait complete. 2024-09-16T15:43:37.974 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-rbd 2024-09-16T15:43:38.101 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:38.097+0000 7f1e6e24a640 1 -- 172.21.15.123:0/1418570257 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1e50004810 msgr2=0x7f1e50004c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:43:38.102 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:38.097+0000 7f1e6e24a640 1 --2- 172.21.15.123:0/1418570257 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1e50004810 0x7f1e50004c90 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f1e5c0016f0 tx=0x7f1e5c01c760 comp rx=0 tx=0).stop 2024-09-16T15:43:38.102 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:38.097+0000 7f1e6e24a640 1 -- 172.21.15.123:0/1418570257 shutdown_connections 2024-09-16T15:43:38.102 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:38.097+0000 7f1e6e24a640 1 --2- 172.21.15.123:0/1418570257 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1e50004810 0x7f1e50004c90 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:38.102 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:38.097+0000 7f1e6e24a640 1 -- 172.21.15.123:0/1418570257 >> 172.21.15.123:0/1418570257 conn(0x7f1e680d3ea0 msgr2=0x7f1e680d42b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:43:38.102 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:38.097+0000 7f1e6e24a640 1 -- 172.21.15.123:0/1418570257 shutdown_connections 2024-09-16T15:43:38.102 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:38.097+0000 7f1e6e24a640 1 -- 172.21.15.123:0/1418570257 wait complete. 2024-09-16T15:43:38.103 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:38.097+0000 7f1e6e24a640 1 Processor -- start 2024-09-16T15:43:38.103 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:38.097+0000 7f1e6e24a640 1 -- start start 2024-09-16T15:43:38.104 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:38.101+0000 7f1e6e24a640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f1e50004810 0x7f1e680c7cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:43:38.105 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:38.101+0000 7f1e6e24a640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1e681fb9b0 0x7f1e681f6f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:43:38.105 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:38.101+0000 7f1e6e24a640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1e681f7440 0x7f1e680b4800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:43:38.105 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:38.101+0000 7f1e6e24a640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f1e680d2be0 con 0x7f1e681f7440 2024-09-16T15:43:38.105 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:38.101+0000 7f1e6e24a640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f1e680d2a60 con 0x7f1e681fb9b0 2024-09-16T15:43:38.105 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:38.101+0000 7f1e6e24a640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f1e680b0e30 con 0x7f1e50004810 2024-09-16T15:43:38.105 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:38.101+0000 7f1e67fff640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1e681f7440 0x7f1e680b4800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:43:38.105 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:38.101+0000 7f1e67fff640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1e681f7440 0x7f1e680b4800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:44804/0 (socket says 172.21.15.123:44804) 2024-09-16T15:43:38.105 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:38.101+0000 7f1e67fff640 1 -- 172.21.15.123:0/1222537827 learned_addr learned my addr 172.21.15.123:0/1222537827 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:43:38.105 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:38.101+0000 7f1e677fe640 1 --2- 172.21.15.123:0/1222537827 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f1e50004810 0x7f1e680c7cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:43:38.105 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:38.101+0000 7f1e66ffd640 1 --2- 172.21.15.123:0/1222537827 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1e681fb9b0 0x7f1e681f6f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:43:38.105 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:38.101+0000 7f1e67fff640 1 -- 172.21.15.123:0/1222537827 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f1e50004810 msgr2=0x7f1e680c7cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:43:38.105 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:38.101+0000 7f1e67fff640 1 --2- 172.21.15.123:0/1222537827 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f1e50004810 0x7f1e680c7cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:38.105 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:38.101+0000 7f1e67fff640 1 -- 172.21.15.123:0/1222537827 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1e681fb9b0 msgr2=0x7f1e681f6f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:43:38.106 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:38.101+0000 7f1e67fff640 1 --2- 172.21.15.123:0/1222537827 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1e681fb9b0 0x7f1e681f6f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:38.106 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:38.101+0000 7f1e67fff640 1 -- 172.21.15.123:0/1222537827 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1e680c8510 con 0x7f1e681f7440 2024-09-16T15:43:38.106 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:38.101+0000 7f1e66ffd640 1 --2- 172.21.15.123:0/1222537827 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1e681fb9b0 0x7f1e681f6f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T15:43:38.106 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:38.101+0000 7f1e67fff640 1 --2- 172.21.15.123:0/1222537827 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1e681f7440 0x7f1e680b4800 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f1e6806f900 tx=0x7f1e6000cae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:43:38.106 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:38.101+0000 7f1e4ffff640 1 -- 172.21.15.123:0/1222537827 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e60017070 con 0x7f1e681f7440 2024-09-16T15:43:38.106 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:38.101+0000 7f1e4ffff640 1 -- 172.21.15.123:0/1222537827 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f1e60005070 con 0x7f1e681f7440 2024-09-16T15:43:38.106 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:38.101+0000 7f1e6e24a640 1 -- 172.21.15.123:0/1222537827 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1e680c87f0 con 0x7f1e681f7440 2024-09-16T15:43:38.106 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:38.101+0000 7f1e4ffff640 1 -- 172.21.15.123:0/1222537827 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e60005480 con 0x7f1e681f7440 2024-09-16T15:43:38.106 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:38.101+0000 7f1e6e24a640 1 -- 172.21.15.123:0/1222537827 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1e681fb1b0 con 0x7f1e681f7440 2024-09-16T15:43:38.107 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:38.101+0000 7f1e6e24a640 1 -- 172.21.15.123:0/1222537827 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1e680b1340 con 0x7f1e681f7440 2024-09-16T15:43:38.107 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:38.101+0000 7f1e4ffff640 1 -- 172.21.15.123:0/1222537827 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 5) ==== 97080+0+0 (secure 0 0 0) 0x7f1e6001a070 con 0x7f1e681f7440 2024-09-16T15:43:38.107 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:38.101+0000 7f1e4ffff640 1 --2- 172.21.15.123:0/1222537827 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f1e54072ec0 0x7f1e54075380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:43:38.107 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:38.101+0000 7f1e4ffff640 1 -- 172.21.15.123:0/1222537827 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(29..29 src has 1..29) ==== 4857+0+0 (secure 0 0 0) 0x7f1e60093280 con 0x7f1e681f7440 2024-09-16T15:43:38.107 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:38.105+0000 7f1e677fe640 1 --2- 172.21.15.123:0/1222537827 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f1e54072ec0 0x7f1e54075380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:43:38.107 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:38.105+0000 7f1e677fe640 1 --2- 172.21.15.123:0/1222537827 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f1e54072ec0 0x7f1e54075380 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f1e680cd5d0 tx=0x7f1e58005f40 comp rx=0 tx=0).ready entity=mgr.4135 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:43:38.110 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:38.105+0000 7f1e4ffff640 1 -- 172.21.15.123:0/1222537827 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f1e60021020 con 0x7f1e681f7440 2024-09-16T15:43:38.338 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:38.333+0000 7f1e6e24a640 1 -- 172.21.15.123:0/1222537827 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-rbd"} v 0) -- 0x7f1e680d4530 con 0x7f1e681f7440 2024-09-16T15:43:38.342 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:38.337+0000 7f1e4ffff640 1 -- 172.21.15.123:0/1222537827 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-rbd"}]=0 v14) ==== 90+0+0 (secure 0 0 0) 0x7f1e6005e910 con 0x7f1e681f7440 2024-09-16T15:43:38.344 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:38.341+0000 7f1e6e24a640 1 -- 172.21.15.123:0/1222537827 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f1e54072ec0 msgr2=0x7f1e54075380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:43:38.344 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:38.341+0000 7f1e6e24a640 1 --2- 172.21.15.123:0/1222537827 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f1e54072ec0 0x7f1e54075380 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f1e680cd5d0 tx=0x7f1e58005f40 comp rx=0 tx=0).stop 2024-09-16T15:43:38.344 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:38.341+0000 7f1e6e24a640 1 -- 172.21.15.123:0/1222537827 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1e681f7440 msgr2=0x7f1e680b4800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:43:38.344 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:38.341+0000 7f1e6e24a640 1 --2- 172.21.15.123:0/1222537827 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1e681f7440 0x7f1e680b4800 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f1e6806f900 tx=0x7f1e6000cae0 comp rx=0 tx=0).stop 2024-09-16T15:43:38.344 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:38.341+0000 7f1e6e24a640 1 -- 172.21.15.123:0/1222537827 shutdown_connections 2024-09-16T15:43:38.344 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:38.341+0000 7f1e6e24a640 1 --2- 172.21.15.123:0/1222537827 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f1e54072ec0 0x7f1e54075380 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:38.344 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:38.341+0000 7f1e6e24a640 1 --2- 172.21.15.123:0/1222537827 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1e681f7440 0x7f1e680b4800 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:38.345 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:38.341+0000 7f1e6e24a640 1 --2- 172.21.15.123:0/1222537827 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1e681fb9b0 0x7f1e681f6f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:38.345 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:38.341+0000 7f1e6e24a640 1 --2- 172.21.15.123:0/1222537827 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f1e50004810 0x7f1e680c7cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:38.345 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:38.341+0000 7f1e6e24a640 1 -- 172.21.15.123:0/1222537827 >> 172.21.15.123:0/1222537827 conn(0x7f1e680d3ea0 msgr2=0x7f1e6806e020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:43:38.345 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:38.341+0000 7f1e6e24a640 1 -- 172.21.15.123:0/1222537827 shutdown_connections 2024-09-16T15:43:38.345 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:38.341+0000 7f1e6e24a640 1 -- 172.21.15.123:0/1222537827 wait complete. 2024-09-16T15:43:38.359 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-rbd-mirror 2024-09-16T15:43:38.480 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:38.477+0000 7f18ec73b640 1 -- 172.21.15.123:0/2875455182 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f18e415bca0 msgr2=0x7f18e40d0630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:43:38.480 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:38.477+0000 7f18ec73b640 1 --2- 172.21.15.123:0/2875455182 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f18e415bca0 0x7f18e40d0630 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f18e4076230 tx=0x7f18dc01c540 comp rx=0 tx=0).stop 2024-09-16T15:43:38.480 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:38.477+0000 7f18ec73b640 1 -- 172.21.15.123:0/2875455182 shutdown_connections 2024-09-16T15:43:38.480 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:38.477+0000 7f18ec73b640 1 --2- 172.21.15.123:0/2875455182 >> v2:172.21.15.184:3300/0 conn(0x7f18e4074430 0x7f18e40ca160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:38.480 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:38.477+0000 7f18ec73b640 1 --2- 172.21.15.123:0/2875455182 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f18e40726c0 0x7f18e4073ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:38.480 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:38.477+0000 7f18ec73b640 1 --2- 172.21.15.123:0/2875455182 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f18e415bca0 0x7f18e40d0630 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:38.481 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:38.477+0000 7f18ec73b640 1 -- 172.21.15.123:0/2875455182 >> 172.21.15.123:0/2875455182 conn(0x7f18e40cfb10 msgr2=0x7f18e40cff20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:43:38.481 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:38.477+0000 7f18ec73b640 1 -- 172.21.15.123:0/2875455182 shutdown_connections 2024-09-16T15:43:38.481 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:38.477+0000 7f18ec73b640 1 -- 172.21.15.123:0/2875455182 wait complete. 2024-09-16T15:43:38.481 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:38.477+0000 7f18ec73b640 1 Processor -- start 2024-09-16T15:43:38.481 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:38.477+0000 7f18ec73b640 1 -- start start 2024-09-16T15:43:38.483 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:38.477+0000 7f18ec73b640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f18e40726c0 0x7f18e4165050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:43:38.483 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:38.477+0000 7f18ec73b640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f18e4074430 0x7f18e40c5950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:43:38.483 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:38.477+0000 7f18ec73b640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f18e415bca0 0x7f18e40c5e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:43:38.483 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:38.477+0000 7f18ec73b640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f18e406ec20 con 0x7f18e415bca0 2024-09-16T15:43:38.483 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:38.477+0000 7f18ec73b640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f18e406eaa0 con 0x7f18e4074430 2024-09-16T15:43:38.483 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:38.477+0000 7f18ec73b640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f18e406eda0 con 0x7f18e40726c0 2024-09-16T15:43:38.483 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:38.477+0000 7f18eacd8640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f18e415bca0 0x7f18e40c5e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:43:38.483 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:38.477+0000 7f18eacd8640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f18e415bca0 0x7f18e40c5e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:44834/0 (socket says 172.21.15.123:44834) 2024-09-16T15:43:38.483 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:38.477+0000 7f18eacd8640 1 -- 172.21.15.123:0/2116714385 learned_addr learned my addr 172.21.15.123:0/2116714385 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:43:38.483 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:38.477+0000 7f18ea4d7640 1 --2- 172.21.15.123:0/2116714385 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f18e40726c0 0x7f18e4165050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:43:38.483 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:38.477+0000 7f18eacd8640 1 -- 172.21.15.123:0/2116714385 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f18e40726c0 msgr2=0x7f18e4165050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:43:38.484 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:38.477+0000 7f18eacd8640 1 --2- 172.21.15.123:0/2116714385 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f18e40726c0 0x7f18e4165050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:38.484 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:38.477+0000 7f18eacd8640 1 -- 172.21.15.123:0/2116714385 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f18e4074430 msgr2=0x7f18e40c5950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:43:38.484 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:38.477+0000 7f18eacd8640 1 --2- 172.21.15.123:0/2116714385 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f18e4074430 0x7f18e40c5950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:38.484 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:38.477+0000 7f18eacd8640 1 -- 172.21.15.123:0/2116714385 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f18e41f7950 con 0x7f18e415bca0 2024-09-16T15:43:38.484 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:38.477+0000 7f18eacd8640 1 --2- 172.21.15.123:0/2116714385 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f18e415bca0 0x7f18e40c5e90 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f18e000ea20 tx=0x7f18e000eee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:43:38.484 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:38.477+0000 7f18daffd640 1 -- 172.21.15.123:0/2116714385 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18e000cd80 con 0x7f18e415bca0 2024-09-16T15:43:38.484 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:38.477+0000 7f18daffd640 1 -- 172.21.15.123:0/2116714385 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f18e0004c40 con 0x7f18e415bca0 2024-09-16T15:43:38.484 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:38.477+0000 7f18ec73b640 1 -- 172.21.15.123:0/2116714385 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f18e41f7c30 con 0x7f18e415bca0 2024-09-16T15:43:38.484 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:38.477+0000 7f18daffd640 1 -- 172.21.15.123:0/2116714385 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18e0005280 con 0x7f18e415bca0 2024-09-16T15:43:38.484 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:38.477+0000 7f18ec73b640 1 -- 172.21.15.123:0/2116714385 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f18e41f81f0 con 0x7f18e415bca0 2024-09-16T15:43:38.485 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:38.481+0000 7f18ec73b640 1 -- 172.21.15.123:0/2116714385 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f18e40ca160 con 0x7f18e415bca0 2024-09-16T15:43:38.485 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:38.481+0000 7f18daffd640 1 -- 172.21.15.123:0/2116714385 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 5) ==== 97080+0+0 (secure 0 0 0) 0x7f18e0005420 con 0x7f18e415bca0 2024-09-16T15:43:38.485 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:38.481+0000 7f18daffd640 1 --2- 172.21.15.123:0/2116714385 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f18c8072ec0 0x7f18c8075380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:43:38.485 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:38.481+0000 7f18daffd640 1 -- 172.21.15.123:0/2116714385 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(29..29 src has 1..29) ==== 4857+0+0 (secure 0 0 0) 0x7f18e0094760 con 0x7f18e415bca0 2024-09-16T15:43:38.485 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:38.481+0000 7f18ea4d7640 1 --2- 172.21.15.123:0/2116714385 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f18c8072ec0 0x7f18c8075380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:43:38.486 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:38.481+0000 7f18ea4d7640 1 --2- 172.21.15.123:0/2116714385 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f18c8072ec0 0x7f18c8075380 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f18dc0af000 tx=0x7f18dc01ad80 comp rx=0 tx=0).ready entity=mgr.4135 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:43:38.488 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:38.485+0000 7f18daffd640 1 -- 172.21.15.123:0/2116714385 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f18e0016030 con 0x7f18e415bca0 2024-09-16T15:43:38.645 DEBUG:teuthology.orchestra.run.smithi123:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-16T15:43:38.648 DEBUG:teuthology.orchestra.run.smithi184:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-16T15:43:38.708 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:38.705+0000 7f18ec73b640 1 -- 172.21.15.123:0/2116714385 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-rbd-mirror"} v 0) -- 0x7f18e415f630 con 0x7f18e415bca0 2024-09-16T15:43:38.711 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:38.705+0000 7f18daffd640 1 -- 172.21.15.123:0/2116714385 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-rbd-mirror"}]=0 v15) ==== 97+0+0 (secure 0 0 0) 0x7f18e005fe70 con 0x7f18e415bca0 2024-09-16T15:43:38.714 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:38.709+0000 7f18ec73b640 1 -- 172.21.15.123:0/2116714385 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f18c8072ec0 msgr2=0x7f18c8075380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:43:38.714 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:38.709+0000 7f18ec73b640 1 --2- 172.21.15.123:0/2116714385 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f18c8072ec0 0x7f18c8075380 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f18dc0af000 tx=0x7f18dc01ad80 comp rx=0 tx=0).stop 2024-09-16T15:43:38.714 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:38.709+0000 7f18ec73b640 1 -- 172.21.15.123:0/2116714385 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f18e415bca0 msgr2=0x7f18e40c5e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:43:38.714 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:38.709+0000 7f18ec73b640 1 --2- 172.21.15.123:0/2116714385 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f18e415bca0 0x7f18e40c5e90 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f18e000ea20 tx=0x7f18e000eee0 comp rx=0 tx=0).stop 2024-09-16T15:43:38.714 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:38.709+0000 7f18ec73b640 1 -- 172.21.15.123:0/2116714385 shutdown_connections 2024-09-16T15:43:38.714 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:38.709+0000 7f18ec73b640 1 --2- 172.21.15.123:0/2116714385 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f18c8072ec0 0x7f18c8075380 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:38.714 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:38.709+0000 7f18ec73b640 1 --2- 172.21.15.123:0/2116714385 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f18e415bca0 0x7f18e40c5e90 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:38.715 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:38.709+0000 7f18ec73b640 1 --2- 172.21.15.123:0/2116714385 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f18e4074430 0x7f18e40c5950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:38.715 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:38.709+0000 7f18ec73b640 1 --2- 172.21.15.123:0/2116714385 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f18e40726c0 0x7f18e4165050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:38.715 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:38.709+0000 7f18ec73b640 1 -- 172.21.15.123:0/2116714385 >> 172.21.15.123:0/2116714385 conn(0x7f18e40cfb10 msgr2=0x7f18e40b41b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:43:38.715 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:38.709+0000 7f18ec73b640 1 -- 172.21.15.123:0/2116714385 shutdown_connections 2024-09-16T15:43:38.715 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:38.709+0000 7f18ec73b640 1 -- 172.21.15.123:0/2116714385 wait complete. 2024-09-16T15:43:38.729 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-rgw 2024-09-16T15:43:38.896 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:38.893+0000 7f707f5fc640 1 -- 172.21.15.123:0/2031748387 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7064004810 msgr2=0x7f7064004c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:43:38.896 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:38.893+0000 7f707f5fc640 1 --2- 172.21.15.123:0/2031748387 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7064004810 0x7f7064004c90 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f7068001860 tx=0x7f706801c8e0 comp rx=0 tx=0).stop 2024-09-16T15:43:38.896 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:38.893+0000 7f707db99640 1 -- 172.21.15.123:0/2031748387 reap_dead start 2024-09-16T15:43:38.896 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:38.893+0000 7f707f5fc640 1 -- 172.21.15.123:0/2031748387 shutdown_connections 2024-09-16T15:43:38.896 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:38.893+0000 7f707f5fc640 1 -- 172.21.15.123:0/2031748387 >> 172.21.15.123:0/2031748387 conn(0x7f7078074bd0 msgr2=0x7f707809aa40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:43:38.896 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:38.893+0000 7f707f5fc640 1 -- 172.21.15.123:0/2031748387 shutdown_connections 2024-09-16T15:43:38.897 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:38.893+0000 7f707f5fc640 1 -- 172.21.15.123:0/2031748387 wait complete. 2024-09-16T15:43:38.899 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:38.893+0000 7f707f5fc640 1 Processor -- start 2024-09-16T15:43:38.899 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:38.893+0000 7f707f5fc640 1 -- start start 2024-09-16T15:43:38.899 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:38.893+0000 7f707f5fc640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f7064004810 0x7f70780c94d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:43:38.899 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:38.893+0000 7f707f5fc640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f70780afa10 0x7f70780afe90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:43:38.899 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:38.893+0000 7f707f5fc640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f70781f7b30 0x7f70781f7fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:43:38.899 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:38.893+0000 7f707f5fc640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f7078167ab0 con 0x7f70780afa10 2024-09-16T15:43:38.899 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:38.893+0000 7f707f5fc640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f7078167930 con 0x7f70781f7b30 2024-09-16T15:43:38.900 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:38.893+0000 7f707f5fc640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f7078167c30 con 0x7f7064004810 2024-09-16T15:43:38.900 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:38.893+0000 7f707cb97640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f70780afa10 0x7f70780afe90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:43:38.900 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:38.893+0000 7f707cb97640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f70780afa10 0x7f70780afe90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:44850/0 (socket says 172.21.15.123:44850) 2024-09-16T15:43:38.900 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:38.893+0000 7f707cb97640 1 -- 172.21.15.123:0/2243566644 learned_addr learned my addr 172.21.15.123:0/2243566644 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:43:38.900 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:38.893+0000 7f707db99640 1 --2- 172.21.15.123:0/2243566644 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f70781f7b30 0x7f70781f7fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:43:38.900 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:38.893+0000 7f707d398640 1 --2- 172.21.15.123:0/2243566644 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f7064004810 0x7f70780c94d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:43:38.900 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:38.893+0000 7f707cb97640 1 -- 172.21.15.123:0/2243566644 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f7064004810 msgr2=0x7f70780c94d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:43:38.900 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:38.893+0000 7f707cb97640 1 --2- 172.21.15.123:0/2243566644 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f7064004810 0x7f70780c94d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:38.900 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:38.893+0000 7f707cb97640 1 -- 172.21.15.123:0/2243566644 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f70781f7b30 msgr2=0x7f70781f7fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:43:38.900 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:38.893+0000 7f707cb97640 1 --2- 172.21.15.123:0/2243566644 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f70781f7b30 0x7f70781f7fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:38.901 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:38.893+0000 7f707cb97640 1 -- 172.21.15.123:0/2243566644 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f70780b69a0 con 0x7f70780afa10 2024-09-16T15:43:38.901 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:38.893+0000 7f707cb97640 1 --2- 172.21.15.123:0/2243566644 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f70780afa10 0x7f70780afe90 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f7068005040 tx=0x7f7068003c80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:43:38.901 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:38.893+0000 7f706dffb640 1 -- 172.21.15.123:0/2243566644 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7068001480 con 0x7f70780afa10 2024-09-16T15:43:38.901 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:38.893+0000 7f706dffb640 1 -- 172.21.15.123:0/2243566644 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f70680a5850 con 0x7f70780afa10 2024-09-16T15:43:38.901 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:38.893+0000 7f707f5fc640 1 -- 172.21.15.123:0/2243566644 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f70780b5be0 con 0x7f70780afa10 2024-09-16T15:43:38.901 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:38.893+0000 7f706dffb640 1 -- 172.21.15.123:0/2243566644 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70680068d0 con 0x7f70780afa10 2024-09-16T15:43:38.901 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:38.893+0000 7f707f5fc640 1 -- 172.21.15.123:0/2243566644 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f70780b61a0 con 0x7f70780afa10 2024-09-16T15:43:38.902 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:38.897+0000 7f707f5fc640 1 -- 172.21.15.123:0/2243566644 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f70780d1c10 con 0x7f70780afa10 2024-09-16T15:43:38.902 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:38.897+0000 7f706dffb640 1 -- 172.21.15.123:0/2243566644 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 5) ==== 97080+0+0 (secure 0 0 0) 0x7f70680a5ca0 con 0x7f70780afa10 2024-09-16T15:43:38.902 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:38.897+0000 7f706dffb640 1 --2- 172.21.15.123:0/2243566644 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f705c072ef0 0x7f705c0753b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:43:38.902 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:38.897+0000 7f706dffb640 1 -- 172.21.15.123:0/2243566644 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(29..29 src has 1..29) ==== 4857+0+0 (secure 0 0 0) 0x7f706812db20 con 0x7f70780afa10 2024-09-16T15:43:38.902 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:38.897+0000 7f707d398640 1 --2- 172.21.15.123:0/2243566644 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f705c072ef0 0x7f705c0753b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:43:38.902 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:38.897+0000 7f707d398640 1 --2- 172.21.15.123:0/2243566644 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f705c072ef0 0x7f705c0753b0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f7074000c10 tx=0x7f7074000ee0 comp rx=0 tx=0).ready entity=mgr.4135 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:43:38.904 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:38.901+0000 7f706dffb640 1 -- 172.21.15.123:0/2243566644 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f7068016690 con 0x7f70780afa10 2024-09-16T15:43:39.120 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:39.117+0000 7f707f5fc640 1 -- 172.21.15.123:0/2243566644 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-rgw"} v 0) -- 0x7f70780cc2a0 con 0x7f70780afa10 2024-09-16T15:43:39.123 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:39.117+0000 7f706dffb640 1 -- 172.21.15.123:0/2243566644 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-rgw"}]=0 v16) ==== 90+0+0 (secure 0 0 0) 0x7f70680f91d0 con 0x7f70780afa10 2024-09-16T15:43:39.126 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:39.121+0000 7f707f5fc640 1 -- 172.21.15.123:0/2243566644 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f705c072ef0 msgr2=0x7f705c0753b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:43:39.126 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:39.121+0000 7f707f5fc640 1 --2- 172.21.15.123:0/2243566644 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f705c072ef0 0x7f705c0753b0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f7074000c10 tx=0x7f7074000ee0 comp rx=0 tx=0).stop 2024-09-16T15:43:39.126 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:39.121+0000 7f707f5fc640 1 -- 172.21.15.123:0/2243566644 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f70780afa10 msgr2=0x7f70780afe90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:43:39.126 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:39.121+0000 7f707f5fc640 1 --2- 172.21.15.123:0/2243566644 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f70780afa10 0x7f70780afe90 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f7068005040 tx=0x7f7068003c80 comp rx=0 tx=0).stop 2024-09-16T15:43:39.126 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:39.121+0000 7f707f5fc640 1 -- 172.21.15.123:0/2243566644 shutdown_connections 2024-09-16T15:43:39.126 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:39.121+0000 7f707f5fc640 1 --2- 172.21.15.123:0/2243566644 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f705c072ef0 0x7f705c0753b0 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:39.126 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:39.121+0000 7f707f5fc640 1 --2- 172.21.15.123:0/2243566644 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f70781f7b30 0x7f70781f7fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:39.127 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:39.121+0000 7f707f5fc640 1 --2- 172.21.15.123:0/2243566644 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f70780afa10 0x7f70780afe90 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:39.127 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:39.121+0000 7f707f5fc640 1 --2- 172.21.15.123:0/2243566644 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f7064004810 0x7f70780c94d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:39.127 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:39.121+0000 7f707f5fc640 1 -- 172.21.15.123:0/2243566644 >> 172.21.15.123:0/2243566644 conn(0x7f7078074bd0 msgr2=0x7f70780aede0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:43:39.127 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:39.121+0000 7f707f5fc640 1 -- 172.21.15.123:0/2243566644 shutdown_connections 2024-09-16T15:43:39.127 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:39.121+0000 7f707f5fc640 1 -- 172.21.15.123:0/2243566644 wait complete. 2024-09-16T15:43:39.142 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-09-16T15:43:39.257 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:39.253+0000 7f597e088640 1 -- 172.21.15.123:0/4017693285 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5960003a60 msgr2=0x7f5960003ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:43:39.257 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:39.253+0000 7f597e088640 1 --2- 172.21.15.123:0/4017693285 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5960003a60 0x7f5960003ea0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f5968009b60 tx=0x7f596801d220 comp rx=0 tx=0).stop 2024-09-16T15:43:39.257 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:39.253+0000 7f597e088640 1 -- 172.21.15.123:0/4017693285 shutdown_connections 2024-09-16T15:43:39.257 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:39.253+0000 7f597e088640 1 --2- 172.21.15.123:0/4017693285 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5960003a60 0x7f5960003ea0 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:39.257 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:39.253+0000 7f597e088640 1 -- 172.21.15.123:0/4017693285 >> 172.21.15.123:0/4017693285 conn(0x7f59780cfb10 msgr2=0x7f59780cff20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:43:39.258 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:39.253+0000 7f597e088640 1 -- 172.21.15.123:0/4017693285 shutdown_connections 2024-09-16T15:43:39.258 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:39.253+0000 7f597e088640 1 -- 172.21.15.123:0/4017693285 wait complete. 2024-09-16T15:43:39.260 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:39.253+0000 7f597e088640 1 Processor -- start 2024-09-16T15:43:39.260 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:39.253+0000 7f597e088640 1 -- start start 2024-09-16T15:43:39.260 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:39.253+0000 7f597e088640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f5960003a60 0x7f5978154050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:43:39.260 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:39.253+0000 7f597e088640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f59780b3380 0x7f59780b4600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:43:39.260 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:39.253+0000 7f597e088640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f59781580f0 0x7f59780b4b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:43:39.260 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:39.253+0000 7f597e088640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f597815edb0 con 0x7f59781580f0 2024-09-16T15:43:39.260 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:39.253+0000 7f597e088640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f597815ec30 con 0x7f59780b3380 2024-09-16T15:43:39.261 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:39.253+0000 7f597e088640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f597815ef30 con 0x7f5960003a60 2024-09-16T15:43:39.261 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:39.257+0000 7f5977fff640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f59781580f0 0x7f59780b4b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:43:39.261 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:39.257+0000 7f5977fff640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f59781580f0 0x7f59780b4b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:44870/0 (socket says 172.21.15.123:44870) 2024-09-16T15:43:39.261 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:39.257+0000 7f5977fff640 1 -- 172.21.15.123:0/103946546 learned_addr learned my addr 172.21.15.123:0/103946546 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:43:39.261 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:39.257+0000 7f59777fe640 1 --2- 172.21.15.123:0/103946546 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f5960003a60 0x7f5978154050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:43:39.261 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:39.257+0000 7f5976ffd640 1 --2- 172.21.15.123:0/103946546 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f59780b3380 0x7f59780b4600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:43:39.261 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:39.257+0000 7f5977fff640 1 -- 172.21.15.123:0/103946546 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f5960003a60 msgr2=0x7f5978154050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:43:39.261 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:39.257+0000 7f5977fff640 1 --2- 172.21.15.123:0/103946546 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f5960003a60 0x7f5978154050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:39.261 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:39.257+0000 7f5977fff640 1 -- 172.21.15.123:0/103946546 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f59780b3380 msgr2=0x7f59780b4600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:43:39.261 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:39.257+0000 7f5977fff640 1 --2- 172.21.15.123:0/103946546 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f59780b3380 0x7f59780b4600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:39.262 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:39.257+0000 7f5977fff640 1 -- 172.21.15.123:0/103946546 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5978154590 con 0x7f59781580f0 2024-09-16T15:43:39.262 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:39.257+0000 7f59777fe640 1 --2- 172.21.15.123:0/103946546 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f5960003a60 0x7f5978154050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T15:43:39.262 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:39.257+0000 7f5977fff640 1 --2- 172.21.15.123:0/103946546 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f59781580f0 0x7f59780b4b40 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f59780c3570 tx=0x7f597000cc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:43:39.262 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:39.257+0000 7f595ffff640 1 -- 172.21.15.123:0/103946546 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5970003980 con 0x7f59781580f0 2024-09-16T15:43:39.262 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:39.257+0000 7f595ffff640 1 -- 172.21.15.123:0/103946546 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f5970003c10 con 0x7f59781580f0 2024-09-16T15:43:39.262 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:39.257+0000 7f597e088640 1 -- 172.21.15.123:0/103946546 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f59780b3800 con 0x7f59781580f0 2024-09-16T15:43:39.262 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:39.257+0000 7f595ffff640 1 -- 172.21.15.123:0/103946546 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5970005110 con 0x7f59781580f0 2024-09-16T15:43:39.262 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:39.257+0000 7f597e088640 1 -- 172.21.15.123:0/103946546 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f59780b3dc0 con 0x7f59781580f0 2024-09-16T15:43:39.263 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:39.257+0000 7f597e088640 1 -- 172.21.15.123:0/103946546 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f597815f490 con 0x7f59781580f0 2024-09-16T15:43:39.263 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:39.257+0000 7f595ffff640 1 -- 172.21.15.123:0/103946546 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 5) ==== 97080+0+0 (secure 0 0 0) 0x7f5970011070 con 0x7f59781580f0 2024-09-16T15:43:39.264 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:39.257+0000 7f595ffff640 1 --2- 172.21.15.123:0/103946546 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f5964072ec0 0x7f5964075380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:43:39.264 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:39.257+0000 7f595ffff640 1 -- 172.21.15.123:0/103946546 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(29..29 src has 1..29) ==== 4857+0+0 (secure 0 0 0) 0x7f5970093060 con 0x7f59781580f0 2024-09-16T15:43:39.264 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:39.257+0000 7f59777fe640 1 --2- 172.21.15.123:0/103946546 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f5964072ec0 0x7f5964075380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:43:39.264 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:39.257+0000 7f59777fe640 1 --2- 172.21.15.123:0/103946546 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f5964072ec0 0x7f5964075380 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f59680ae000 tx=0x7f596801a010 comp rx=0 tx=0).ready entity=mgr.4135 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:43:39.266 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:39.261+0000 7f595ffff640 1 -- 172.21.15.123:0/103946546 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f5970019060 con 0x7f59781580f0 2024-09-16T15:43:39.438 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:39.433+0000 7f597e088640 1 -- 172.21.15.123:0/103946546 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) -- 0x7f59780d06c0 con 0x7f59781580f0 2024-09-16T15:43:39.439 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:39.433+0000 7f595ffff640 1 -- 172.21.15.123:0/103946546 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v29) ==== 84+0+65 (secure 0 0 0) 0x7f597005e690 con 0x7f59781580f0 2024-09-16T15:43:39.439 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-09-16T15:43:39.439 INFO:teuthology.orchestra.run.smithi123.stdout:[ 2024-09-16T15:43:39.439 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-09-16T15:43:39.439 INFO:teuthology.orchestra.run.smithi123.stdout: "poolnum": 1, 2024-09-16T15:43:39.439 INFO:teuthology.orchestra.run.smithi123.stdout: "poolname": ".mgr" 2024-09-16T15:43:39.439 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-16T15:43:39.439 INFO:teuthology.orchestra.run.smithi123.stdout:] 2024-09-16T15:43:39.441 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:39.437+0000 7f597e088640 1 -- 172.21.15.123:0/103946546 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f5964072ec0 msgr2=0x7f5964075380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:43:39.441 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:39.437+0000 7f597e088640 1 --2- 172.21.15.123:0/103946546 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f5964072ec0 0x7f5964075380 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f59680ae000 tx=0x7f596801a010 comp rx=0 tx=0).stop 2024-09-16T15:43:39.442 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:39.437+0000 7f597e088640 1 -- 172.21.15.123:0/103946546 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f59781580f0 msgr2=0x7f59780b4b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:43:39.442 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:39.437+0000 7f597e088640 1 --2- 172.21.15.123:0/103946546 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f59781580f0 0x7f59780b4b40 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f59780c3570 tx=0x7f597000cc00 comp rx=0 tx=0).stop 2024-09-16T15:43:39.442 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:39.437+0000 7f597e088640 1 -- 172.21.15.123:0/103946546 shutdown_connections 2024-09-16T15:43:39.442 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:39.437+0000 7f597e088640 1 --2- 172.21.15.123:0/103946546 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f5964072ec0 0x7f5964075380 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:39.442 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:39.437+0000 7f597e088640 1 --2- 172.21.15.123:0/103946546 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f59781580f0 0x7f59780b4b40 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:39.442 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:39.437+0000 7f597e088640 1 --2- 172.21.15.123:0/103946546 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f59780b3380 0x7f59780b4600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:39.442 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:39.437+0000 7f597e088640 1 --2- 172.21.15.123:0/103946546 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f5960003a60 0x7f5978154050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:39.442 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:39.437+0000 7f597e088640 1 -- 172.21.15.123:0/103946546 >> 172.21.15.123:0/103946546 conn(0x7f59780cfb10 msgr2=0x7f59780ca820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:43:39.442 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:39.437+0000 7f597e088640 1 -- 172.21.15.123:0/103946546 shutdown_connections 2024-09-16T15:43:39.442 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:39.437+0000 7f597e088640 1 -- 172.21.15.123:0/103946546 wait complete. 2024-09-16T15:43:39.457 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-09-16T15:43:39.457 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-09-16T15:43:39.587 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:39.581+0000 7f98c97c7640 1 -- 172.21.15.123:0/1008961500 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f98c4159680 msgr2=0x7f98c40704e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:43:39.587 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:39.581+0000 7f98c97c7640 1 --2- 172.21.15.123:0/1008961500 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f98c4159680 0x7f98c40704e0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f98c4051540 tx=0x7f98b401c880 comp rx=0 tx=0).stop 2024-09-16T15:43:39.587 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:39.581+0000 7f98c97c7640 1 -- 172.21.15.123:0/1008961500 shutdown_connections 2024-09-16T15:43:39.587 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:39.581+0000 7f98c97c7640 1 --2- 172.21.15.123:0/1008961500 >> v2:172.21.15.184:3300/0 conn(0x7f98c4074440 0x7f98c415f670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:39.587 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:39.581+0000 7f98c97c7640 1 --2- 172.21.15.123:0/1008961500 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f98c4159680 0x7f98c40704e0 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:39.587 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:39.581+0000 7f98c97c7640 1 -- 172.21.15.123:0/1008961500 >> 172.21.15.123:0/1008961500 conn(0x7f98c40b1630 msgr2=0x7f98c40b1a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:43:39.587 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:39.581+0000 7f98c97c7640 1 -- 172.21.15.123:0/1008961500 shutdown_connections 2024-09-16T15:43:39.587 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:39.581+0000 7f98c97c7640 1 -- 172.21.15.123:0/1008961500 wait complete. 2024-09-16T15:43:39.588 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:39.585+0000 7f98c97c7640 1 Processor -- start 2024-09-16T15:43:39.588 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:39.585+0000 7f98c97c7640 1 -- start start 2024-09-16T15:43:39.590 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:39.585+0000 7f98c97c7640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f98c4074440 0x7f98c4153690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:43:39.590 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:39.585+0000 7f98c97c7640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f98c4159680 0x7f98c40af5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:43:39.590 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:39.585+0000 7f98c97c7640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f98c415fbb0 0x7f98c40afae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:43:39.590 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:39.585+0000 7f98c97c7640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f98c4169180 con 0x7f98c415fbb0 2024-09-16T15:43:39.591 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:39.585+0000 7f98c97c7640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f98c4169000 con 0x7f98c4159680 2024-09-16T15:43:39.591 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:39.585+0000 7f98c97c7640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f98c4169300 con 0x7f98c4074440 2024-09-16T15:43:39.591 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:39.585+0000 7f98c37fe640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f98c415fbb0 0x7f98c40afae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:43:39.591 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:39.585+0000 7f98c37fe640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f98c415fbb0 0x7f98c40afae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:44882/0 (socket says 172.21.15.123:44882) 2024-09-16T15:43:39.591 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:39.585+0000 7f98c37fe640 1 -- 172.21.15.123:0/3771522647 learned_addr learned my addr 172.21.15.123:0/3771522647 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:43:39.591 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:39.585+0000 7f98c27fc640 1 --2- 172.21.15.123:0/3771522647 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f98c4159680 0x7f98c40af5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:43:39.591 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:39.585+0000 7f98c2ffd640 1 --2- 172.21.15.123:0/3771522647 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f98c4074440 0x7f98c4153690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:43:39.591 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:39.585+0000 7f98c37fe640 1 -- 172.21.15.123:0/3771522647 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f98c4074440 msgr2=0x7f98c4153690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:43:39.591 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:39.585+0000 7f98c37fe640 1 --2- 172.21.15.123:0/3771522647 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f98c4074440 0x7f98c4153690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:39.591 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:39.585+0000 7f98c37fe640 1 -- 172.21.15.123:0/3771522647 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f98c4159680 msgr2=0x7f98c40af5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:43:39.591 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:39.585+0000 7f98c37fe640 1 --2- 172.21.15.123:0/3771522647 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f98c4159680 0x7f98c40af5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:39.592 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:39.585+0000 7f98c37fe640 1 -- 172.21.15.123:0/3771522647 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f98c41542b0 con 0x7f98c415fbb0 2024-09-16T15:43:39.592 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:39.585+0000 7f98c37fe640 1 --2- 172.21.15.123:0/3771522647 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f98c415fbb0 0x7f98c40afae0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f98bc00b970 tx=0x7f98bc00be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:43:39.592 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:39.585+0000 7f98ab7fe640 1 -- 172.21.15.123:0/3771522647 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98bc019070 con 0x7f98c415fbb0 2024-09-16T15:43:39.592 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:39.585+0000 7f98ab7fe640 1 -- 172.21.15.123:0/3771522647 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f98bc00cc30 con 0x7f98c415fbb0 2024-09-16T15:43:39.592 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:39.585+0000 7f98c97c7640 1 -- 172.21.15.123:0/3771522647 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f98c4154590 con 0x7f98c415fbb0 2024-09-16T15:43:39.592 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:39.585+0000 7f98ab7fe640 1 -- 172.21.15.123:0/3771522647 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98bc0146a0 con 0x7f98c415fbb0 2024-09-16T15:43:39.592 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:39.585+0000 7f98c97c7640 1 -- 172.21.15.123:0/3771522647 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f98c4154b50 con 0x7f98c415fbb0 2024-09-16T15:43:39.593 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:39.585+0000 7f98c97c7640 1 -- 172.21.15.123:0/3771522647 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f98c415f670 con 0x7f98c415fbb0 2024-09-16T15:43:39.593 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:39.589+0000 7f98ab7fe640 1 -- 172.21.15.123:0/3771522647 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 5) ==== 97080+0+0 (secure 0 0 0) 0x7f98bc01c070 con 0x7f98c415fbb0 2024-09-16T15:43:39.593 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:39.589+0000 7f98ab7fe640 1 --2- 172.21.15.123:0/3771522647 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f98b0072ec0 0x7f98b0075380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:43:39.593 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:39.589+0000 7f98ab7fe640 1 -- 172.21.15.123:0/3771522647 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(29..29 src has 1..29) ==== 4857+0+0 (secure 0 0 0) 0x7f98bc093740 con 0x7f98c415fbb0 2024-09-16T15:43:39.593 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:39.589+0000 7f98c2ffd640 1 --2- 172.21.15.123:0/3771522647 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f98b0072ec0 0x7f98b0075380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:43:39.594 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:39.589+0000 7f98c2ffd640 1 --2- 172.21.15.123:0/3771522647 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f98b0072ec0 0x7f98b0075380 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f98b40af000 tx=0x7f98b401b0c0 comp rx=0 tx=0).ready entity=mgr.4135 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:43:39.596 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:39.593+0000 7f98ab7fe640 1 -- 172.21.15.123:0/3771522647 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f98bc05ece0 con 0x7f98c415fbb0 2024-09-16T15:43:39.769 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:39.765+0000 7f98c97c7640 1 -- 172.21.15.123:0/3771522647 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) -- 0x7f98c4070040 con 0x7f98c415fbb0 2024-09-16T15:43:40.457 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:40.453+0000 7f98ab7fe640 1 -- 172.21.15.123:0/3771522647 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v30) ==== 138+0+0 (secure 0 0 0) 0x7f98bc05e5e0 con 0x7f98c415fbb0 2024-09-16T15:43:40.457 INFO:teuthology.orchestra.run.smithi123.stderr:pool 'cephfs_metadata' created 2024-09-16T15:43:40.460 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:40.457+0000 7f98c97c7640 1 -- 172.21.15.123:0/3771522647 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f98b0072ec0 msgr2=0x7f98b0075380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:43:40.460 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:40.457+0000 7f98c97c7640 1 --2- 172.21.15.123:0/3771522647 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f98b0072ec0 0x7f98b0075380 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f98b40af000 tx=0x7f98b401b0c0 comp rx=0 tx=0).stop 2024-09-16T15:43:40.460 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:40.457+0000 7f98c97c7640 1 -- 172.21.15.123:0/3771522647 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f98c415fbb0 msgr2=0x7f98c40afae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:43:40.460 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:40.457+0000 7f98c97c7640 1 --2- 172.21.15.123:0/3771522647 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f98c415fbb0 0x7f98c40afae0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f98bc00b970 tx=0x7f98bc00be30 comp rx=0 tx=0).stop 2024-09-16T15:43:40.460 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:40.457+0000 7f98c97c7640 1 -- 172.21.15.123:0/3771522647 shutdown_connections 2024-09-16T15:43:40.460 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:40.457+0000 7f98c97c7640 1 --2- 172.21.15.123:0/3771522647 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f98b0072ec0 0x7f98b0075380 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:40.460 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:40.457+0000 7f98c97c7640 1 --2- 172.21.15.123:0/3771522647 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f98c415fbb0 0x7f98c40afae0 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:40.460 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:40.457+0000 7f98c97c7640 1 --2- 172.21.15.123:0/3771522647 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f98c4159680 0x7f98c40af5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:40.461 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:40.457+0000 7f98c97c7640 1 --2- 172.21.15.123:0/3771522647 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f98c4074440 0x7f98c4153690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:40.461 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:40.457+0000 7f98c97c7640 1 -- 172.21.15.123:0/3771522647 >> 172.21.15.123:0/3771522647 conn(0x7f98c40b1630 msgr2=0x7f98c4156ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:43:40.461 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:40.457+0000 7f98c97c7640 1 -- 172.21.15.123:0/3771522647 shutdown_connections 2024-09-16T15:43:40.461 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:40.457+0000 7f98c97c7640 1 -- 172.21.15.123:0/3771522647 wait complete. 2024-09-16T15:43:40.483 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-09-16T15:43:40.618 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:40.612+0000 7f5a9df93640 1 -- 172.21.15.123:0/3094420760 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5a80004810 msgr2=0x7f5a80004c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:43:40.619 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:40.612+0000 7f5a9df93640 1 --2- 172.21.15.123:0/3094420760 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5a80004810 0x7f5a80004c90 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f5a980c3640 tx=0x7f5a8c01c6f0 comp rx=0 tx=0).stop 2024-09-16T15:43:40.619 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:40.612+0000 7f5a9df93640 1 -- 172.21.15.123:0/3094420760 shutdown_connections 2024-09-16T15:43:40.619 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:40.612+0000 7f5a9df93640 1 --2- 172.21.15.123:0/3094420760 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5a80004810 0x7f5a80004c90 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:40.619 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:40.612+0000 7f5a9df93640 1 -- 172.21.15.123:0/3094420760 >> 172.21.15.123:0/3094420760 conn(0x7f5a980b1710 msgr2=0x7f5a980b1b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:43:40.619 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:40.612+0000 7f5a9df93640 1 -- 172.21.15.123:0/3094420760 shutdown_connections 2024-09-16T15:43:40.619 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:40.612+0000 7f5a9df93640 1 -- 172.21.15.123:0/3094420760 wait complete. 2024-09-16T15:43:40.619 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:40.616+0000 7f5a9df93640 1 Processor -- start 2024-09-16T15:43:40.620 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:40.616+0000 7f5a9df93640 1 -- start start 2024-09-16T15:43:40.622 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:40.616+0000 7f5a9df93640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f5a80004810 0x7f5a9815de90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:43:40.622 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:40.616+0000 7f5a9df93640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5a9814fab0 0x7f5a9814ff10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:43:40.622 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:40.616+0000 7f5a9df93640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5a98151cc0 0x7f5a98152140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:43:40.622 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:40.616+0000 7f5a9df93640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f5a980cf160 con 0x7f5a9814fab0 2024-09-16T15:43:40.622 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:40.616+0000 7f5a9df93640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f5a980cefe0 con 0x7f5a98151cc0 2024-09-16T15:43:40.622 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:40.616+0000 7f5a9df93640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f5a980cf2e0 con 0x7f5a80004810 2024-09-16T15:43:40.622 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:40.616+0000 7f5a96ffd640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5a9814fab0 0x7f5a9814ff10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:43:40.622 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:40.616+0000 7f5a96ffd640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5a9814fab0 0x7f5a9814ff10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:44902/0 (socket says 172.21.15.123:44902) 2024-09-16T15:43:40.622 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:40.616+0000 7f5a96ffd640 1 -- 172.21.15.123:0/118414494 learned_addr learned my addr 172.21.15.123:0/118414494 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:43:40.622 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:40.616+0000 7f5a97fff640 1 --2- 172.21.15.123:0/118414494 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5a98151cc0 0x7f5a98152140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:43:40.623 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:40.616+0000 7f5a977fe640 1 --2- 172.21.15.123:0/118414494 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f5a80004810 0x7f5a9815de90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:43:40.623 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:40.616+0000 7f5a96ffd640 1 -- 172.21.15.123:0/118414494 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f5a80004810 msgr2=0x7f5a9815de90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:43:40.623 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:40.616+0000 7f5a96ffd640 1 --2- 172.21.15.123:0/118414494 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f5a80004810 0x7f5a9815de90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:40.623 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:40.616+0000 7f5a96ffd640 1 -- 172.21.15.123:0/118414494 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5a98151cc0 msgr2=0x7f5a98152140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:43:40.623 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:40.616+0000 7f5a96ffd640 1 --2- 172.21.15.123:0/118414494 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5a98151cc0 0x7f5a98152140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:40.623 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:40.616+0000 7f5a96ffd640 1 -- 172.21.15.123:0/118414494 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5a981f2940 con 0x7f5a9814fab0 2024-09-16T15:43:40.623 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:40.616+0000 7f5a96ffd640 1 --2- 172.21.15.123:0/118414494 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5a9814fab0 0x7f5a9814ff10 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f5a8c01adf0 tx=0x7f5a8c001420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:43:40.623 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:40.616+0000 7f5a7ffff640 1 -- 172.21.15.123:0/118414494 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a8c0036c0 con 0x7f5a9814fab0 2024-09-16T15:43:40.623 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:40.616+0000 7f5a7ffff640 1 -- 172.21.15.123:0/118414494 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f5a8c003950 con 0x7f5a9814fab0 2024-09-16T15:43:40.623 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:40.616+0000 7f5a9df93640 1 -- 172.21.15.123:0/118414494 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5a98150620 con 0x7f5a9814fab0 2024-09-16T15:43:40.623 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:40.616+0000 7f5a7ffff640 1 -- 172.21.15.123:0/118414494 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a8c003d00 con 0x7f5a9814fab0 2024-09-16T15:43:40.623 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:40.616+0000 7f5a9df93640 1 -- 172.21.15.123:0/118414494 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5a981f2ee0 con 0x7f5a9814fab0 2024-09-16T15:43:40.625 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:40.616+0000 7f5a9df93640 1 -- 172.21.15.123:0/118414494 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5a9806fe60 con 0x7f5a9814fab0 2024-09-16T15:43:40.625 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:40.620+0000 7f5a7ffff640 1 -- 172.21.15.123:0/118414494 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 5) ==== 97080+0+0 (secure 0 0 0) 0x7f5a8c0ae430 con 0x7f5a9814fab0 2024-09-16T15:43:40.625 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:40.620+0000 7f5a7ffff640 1 --2- 172.21.15.123:0/118414494 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f5a84072ef0 0x7f5a840753b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:43:40.625 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:40.620+0000 7f5a7ffff640 1 -- 172.21.15.123:0/118414494 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 5227+0+0 (secure 0 0 0) 0x7f5a8c12d290 con 0x7f5a9814fab0 2024-09-16T15:43:40.625 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:40.620+0000 7f5a977fe640 1 --2- 172.21.15.123:0/118414494 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f5a84072ef0 0x7f5a840753b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:43:40.625 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:40.620+0000 7f5a977fe640 1 --2- 172.21.15.123:0/118414494 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f5a84072ef0 0x7f5a840753b0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f5a88005fd0 tx=0x7f5a88005f40 comp rx=0 tx=0).ready entity=mgr.4135 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:43:40.629 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:40.624+0000 7f5a7ffff640 1 -- 172.21.15.123:0/118414494 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f5a8c016450 con 0x7f5a9814fab0 2024-09-16T15:43:40.802 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:40.796+0000 7f5a9df93640 1 -- 172.21.15.123:0/118414494 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) -- 0x7f5a980742d0 con 0x7f5a9814fab0 2024-09-16T15:43:41.468 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:41.464+0000 7f5a7ffff640 1 -- 172.21.15.123:0/118414494 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v31) ==== 170+0+0 (secure 0 0 0) 0x7f5a8c0f86c0 con 0x7f5a9814fab0 2024-09-16T15:43:41.468 INFO:teuthology.orchestra.run.smithi123.stderr:pool 'cephfs_data' created 2024-09-16T15:43:41.471 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:41.464+0000 7f5a9df93640 1 -- 172.21.15.123:0/118414494 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f5a84072ef0 msgr2=0x7f5a840753b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:43:41.471 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:41.464+0000 7f5a9df93640 1 --2- 172.21.15.123:0/118414494 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f5a84072ef0 0x7f5a840753b0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f5a88005fd0 tx=0x7f5a88005f40 comp rx=0 tx=0).stop 2024-09-16T15:43:41.471 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:41.464+0000 7f5a9df93640 1 -- 172.21.15.123:0/118414494 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5a9814fab0 msgr2=0x7f5a9814ff10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:43:41.472 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:41.464+0000 7f5a9df93640 1 --2- 172.21.15.123:0/118414494 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5a9814fab0 0x7f5a9814ff10 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f5a8c01adf0 tx=0x7f5a8c001420 comp rx=0 tx=0).stop 2024-09-16T15:43:41.472 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:41.468+0000 7f5a9df93640 1 -- 172.21.15.123:0/118414494 shutdown_connections 2024-09-16T15:43:41.472 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:41.468+0000 7f5a9df93640 1 --2- 172.21.15.123:0/118414494 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f5a84072ef0 0x7f5a840753b0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:41.472 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:41.468+0000 7f5a9df93640 1 --2- 172.21.15.123:0/118414494 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5a98151cc0 0x7f5a98152140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:41.472 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:41.468+0000 7f5a9df93640 1 --2- 172.21.15.123:0/118414494 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5a9814fab0 0x7f5a9814ff10 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:41.472 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:41.468+0000 7f5a9df93640 1 --2- 172.21.15.123:0/118414494 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f5a80004810 0x7f5a9815de90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:41.472 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:41.468+0000 7f5a9df93640 1 -- 172.21.15.123:0/118414494 >> 172.21.15.123:0/118414494 conn(0x7f5a980b1710 msgr2=0x7f5a980b2a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:43:41.472 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:41.468+0000 7f5a9df93640 1 -- 172.21.15.123:0/118414494 shutdown_connections 2024-09-16T15:43:41.473 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:41.468+0000 7f5a9df93640 1 -- 172.21.15.123:0/118414494 wait complete. 2024-09-16T15:43:41.503 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-09-16T15:43:41.633 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:41.628+0000 7f10c72ab640 1 -- 172.21.15.123:0/1436295309 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f10ac004da0 msgr2=0x7f10ac005220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:43:41.633 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:41.628+0000 7f10c72ab640 1 --2- 172.21.15.123:0/1436295309 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f10ac004da0 0x7f10ac005220 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f10b8006fd0 tx=0x7f10b801cad0 comp rx=0 tx=0).stop 2024-09-16T15:43:41.633 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:41.628+0000 7f10c5848640 1 -- 172.21.15.123:0/1436295309 reap_dead start 2024-09-16T15:43:41.633 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:41.628+0000 7f10c72ab640 1 -- 172.21.15.123:0/1436295309 shutdown_connections 2024-09-16T15:43:41.633 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:41.628+0000 7f10c72ab640 1 -- 172.21.15.123:0/1436295309 >> 172.21.15.123:0/1436295309 conn(0x7f10c009aa40 msgr2=0x7f10c009ae50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:43:41.633 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:41.628+0000 7f10c72ab640 1 -- 172.21.15.123:0/1436295309 shutdown_connections 2024-09-16T15:43:41.633 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:41.628+0000 7f10c72ab640 1 -- 172.21.15.123:0/1436295309 wait complete. 2024-09-16T15:43:41.634 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:41.628+0000 7f10c72ab640 1 Processor -- start 2024-09-16T15:43:41.634 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:41.628+0000 7f10c72ab640 1 -- start start 2024-09-16T15:43:41.636 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:41.628+0000 7f10c72ab640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f10ac004da0 0x7f10c00b0130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:43:41.636 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:41.628+0000 7f10c72ab640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f10c00b7910 0x7f10c015cce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:43:41.636 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:41.628+0000 7f10c72ab640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f10c00c85a0 0x7f10c015d220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:43:41.636 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:41.628+0000 7f10c72ab640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f10c015e420 con 0x7f10c00c85a0 2024-09-16T15:43:41.636 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:41.628+0000 7f10c72ab640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f10c00d1b20 con 0x7f10ac004da0 2024-09-16T15:43:41.636 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:41.628+0000 7f10c72ab640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f10c015e5a0 con 0x7f10c00b7910 2024-09-16T15:43:41.636 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:41.628+0000 7f10c5848640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f10c00c85a0 0x7f10c015d220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:43:41.636 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:41.628+0000 7f10c5848640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f10c00c85a0 0x7f10c015d220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:44930/0 (socket says 172.21.15.123:44930) 2024-09-16T15:43:41.636 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:41.628+0000 7f10c5848640 1 -- 172.21.15.123:0/3623630685 learned_addr learned my addr 172.21.15.123:0/3623630685 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:43:41.636 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:41.628+0000 7f10c5047640 1 --2- 172.21.15.123:0/3623630685 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f10ac004da0 0x7f10c00b0130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:43:41.637 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:41.632+0000 7f10c4846640 1 --2- 172.21.15.123:0/3623630685 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f10c00b7910 0x7f10c015cce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:43:41.637 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:41.632+0000 7f10c5848640 1 -- 172.21.15.123:0/3623630685 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f10c00b7910 msgr2=0x7f10c015cce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:43:41.637 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:41.632+0000 7f10c5848640 1 --2- 172.21.15.123:0/3623630685 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f10c00b7910 0x7f10c015cce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:41.637 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:41.632+0000 7f10c5848640 1 -- 172.21.15.123:0/3623630685 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f10ac004da0 msgr2=0x7f10c00b0130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:43:41.637 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:41.632+0000 7f10c5848640 1 --2- 172.21.15.123:0/3623630685 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f10ac004da0 0x7f10c00b0130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:41.637 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:41.632+0000 7f10c5848640 1 -- 172.21.15.123:0/3623630685 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f10c01f46e0 con 0x7f10c00c85a0 2024-09-16T15:43:41.637 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:41.632+0000 7f10c5848640 1 --2- 172.21.15.123:0/3623630685 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f10c00c85a0 0x7f10c015d220 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f10b8006040 tx=0x7f10b8003e80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:43:41.637 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:41.632+0000 7f10b5ffb640 1 -- 172.21.15.123:0/3623630685 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10b8006cb0 con 0x7f10c00c85a0 2024-09-16T15:43:41.637 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:41.632+0000 7f10b5ffb640 1 -- 172.21.15.123:0/3623630685 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f10b80019f0 con 0x7f10c00c85a0 2024-09-16T15:43:41.637 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:41.632+0000 7f10c72ab640 1 -- 172.21.15.123:0/3623630685 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f10c01f49a0 con 0x7f10c00c85a0 2024-09-16T15:43:41.637 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:41.632+0000 7f10b5ffb640 1 -- 172.21.15.123:0/3623630685 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10b8006330 con 0x7f10c00c85a0 2024-09-16T15:43:41.637 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:41.632+0000 7f10c72ab640 1 -- 172.21.15.123:0/3623630685 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f10c01f4f60 con 0x7f10c00c85a0 2024-09-16T15:43:41.638 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:41.632+0000 7f10c72ab640 1 -- 172.21.15.123:0/3623630685 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f10c015dd10 con 0x7f10c00c85a0 2024-09-16T15:43:41.638 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:41.632+0000 7f10b5ffb640 1 -- 172.21.15.123:0/3623630685 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 5) ==== 97080+0+0 (secure 0 0 0) 0x7f10b80064d0 con 0x7f10c00c85a0 2024-09-16T15:43:41.638 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:41.632+0000 7f10b5ffb640 1 --2- 172.21.15.123:0/3623630685 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f10a4072ec0 0x7f10a4075380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:43:41.638 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:41.632+0000 7f10b5ffb640 1 -- 172.21.15.123:0/3623630685 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(31..31 src has 1..31) ==== 5609+0+0 (secure 0 0 0) 0x7f10b812daa0 con 0x7f10c00c85a0 2024-09-16T15:43:41.638 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:41.632+0000 7f10c5047640 1 --2- 172.21.15.123:0/3623630685 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f10a4072ec0 0x7f10a4075380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:43:41.639 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:41.636+0000 7f10c5047640 1 --2- 172.21.15.123:0/3623630685 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f10a4072ec0 0x7f10a4075380 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f10b0001ec0 tx=0x7f10b0001580 comp rx=0 tx=0).ready entity=mgr.4135 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:43:41.642 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:41.636+0000 7f10b5ffb640 1 -- 172.21.15.123:0/3623630685 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f10b8016830 con 0x7f10c00c85a0 2024-09-16T15:43:41.850 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:41.844+0000 7f10c72ab640 1 -- 172.21.15.123:0/3623630685 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) -- 0x7f10c0162400 con 0x7f10c00c85a0 2024-09-16T15:43:42.517 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:42.512+0000 7f10b5ffb640 1 -- 172.21.15.123:0/3623630685 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 4 and data pool 5 v14) ==== 172+0+0 (secure 0 0 0) 0x7f10b80f8700 con 0x7f10c00c85a0 2024-09-16T15:43:42.518 INFO:teuthology.orchestra.run.smithi123.stderr:new fs with metadata pool 4 and data pool 5 2024-09-16T15:43:42.524 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:42.520+0000 7f10c72ab640 1 -- 172.21.15.123:0/3623630685 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f10a4072ec0 msgr2=0x7f10a4075380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:43:42.524 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:42.520+0000 7f10c72ab640 1 --2- 172.21.15.123:0/3623630685 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f10a4072ec0 0x7f10a4075380 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f10b0001ec0 tx=0x7f10b0001580 comp rx=0 tx=0).stop 2024-09-16T15:43:42.525 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:42.520+0000 7f10c72ab640 1 -- 172.21.15.123:0/3623630685 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f10c00c85a0 msgr2=0x7f10c015d220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:43:42.525 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:42.520+0000 7f10c72ab640 1 --2- 172.21.15.123:0/3623630685 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f10c00c85a0 0x7f10c015d220 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f10b8006040 tx=0x7f10b8003e80 comp rx=0 tx=0).stop 2024-09-16T15:43:42.525 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:42.520+0000 7f10c72ab640 1 -- 172.21.15.123:0/3623630685 shutdown_connections 2024-09-16T15:43:42.525 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:42.520+0000 7f10c72ab640 1 --2- 172.21.15.123:0/3623630685 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f10a4072ec0 0x7f10a4075380 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:42.525 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:42.520+0000 7f10c72ab640 1 --2- 172.21.15.123:0/3623630685 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f10c00c85a0 0x7f10c015d220 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:42.526 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:42.520+0000 7f10c72ab640 1 --2- 172.21.15.123:0/3623630685 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f10c00b7910 0x7f10c015cce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:42.526 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:42.520+0000 7f10c72ab640 1 --2- 172.21.15.123:0/3623630685 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f10ac004da0 0x7f10c00b0130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:42.526 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:42.520+0000 7f10c72ab640 1 -- 172.21.15.123:0/3623630685 >> 172.21.15.123:0/3623630685 conn(0x7f10c009aa40 msgr2=0x7f10c0070a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:43:42.526 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:42.520+0000 7f10c72ab640 1 -- 172.21.15.123:0/3623630685 shutdown_connections 2024-09-16T15:43:42.526 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:42.520+0000 7f10c72ab640 1 -- 172.21.15.123:0/3623630685 wait complete. 2024-09-16T15:43:42.552 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-16T15:43:42.672 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:42.668+0000 7f7983b7c640 1 -- 172.21.15.123:0/178345431 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f797c15bd30 msgr2=0x7f797c06cbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:43:42.672 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:42.668+0000 7f7983b7c640 1 --2- 172.21.15.123:0/178345431 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f797c15bd30 0x7f797c06cbf0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f797c0b6c90 tx=0x7f796c01c790 comp rx=0 tx=0).stop 2024-09-16T15:43:42.672 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:42.668+0000 7f7983b7c640 1 -- 172.21.15.123:0/178345431 shutdown_connections 2024-09-16T15:43:42.672 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:42.668+0000 7f7983b7c640 1 --2- 172.21.15.123:0/178345431 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f797c15bd30 0x7f797c06cbf0 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:42.673 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:42.668+0000 7f7983b7c640 1 -- 172.21.15.123:0/178345431 >> 172.21.15.123:0/178345431 conn(0x7f797c0b2610 msgr2=0x7f797c0b29f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:43:42.673 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:42.668+0000 7f7983b7c640 1 -- 172.21.15.123:0/178345431 shutdown_connections 2024-09-16T15:43:42.673 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:42.668+0000 7f7983b7c640 1 -- 172.21.15.123:0/178345431 wait complete. 2024-09-16T15:43:42.673 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:42.668+0000 7f7983b7c640 1 Processor -- start 2024-09-16T15:43:42.673 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:42.668+0000 7f7983b7c640 1 -- start start 2024-09-16T15:43:42.675 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:42.668+0000 7f7983b7c640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f797c071d00 0x7f797c157b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:43:42.675 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:42.668+0000 7f7983b7c640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f797c0afd60 0x7f797c0cfe00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:43:42.675 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:42.668+0000 7f7983b7c640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f797c15bd30 0x7f797c14a8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:43:42.675 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:42.668+0000 7f7983b7c640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f797c16a2c0 con 0x7f797c071d00 2024-09-16T15:43:42.675 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:42.668+0000 7f7983b7c640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f797c16a140 con 0x7f797c0afd60 2024-09-16T15:43:42.676 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:42.668+0000 7f7983b7c640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f797c16a440 con 0x7f797c15bd30 2024-09-16T15:43:42.676 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:42.668+0000 7f7981918640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f797c071d00 0x7f797c157b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:43:42.676 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:42.668+0000 7f7981918640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f797c071d00 0x7f797c157b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:41548/0 (socket says 172.21.15.123:41548) 2024-09-16T15:43:42.676 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:42.668+0000 7f7981918640 1 -- 172.21.15.123:0/2307987561 learned_addr learned my addr 172.21.15.123:0/2307987561 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:43:42.676 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:42.668+0000 7f7981117640 1 --2- 172.21.15.123:0/2307987561 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f797c0afd60 0x7f797c0cfe00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:43:42.676 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:42.668+0000 7f7981918640 1 -- 172.21.15.123:0/2307987561 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f797c15bd30 msgr2=0x7f797c14a8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:43:42.676 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:42.668+0000 7f7981918640 1 --2- 172.21.15.123:0/2307987561 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f797c15bd30 0x7f797c14a8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:42.676 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:42.668+0000 7f7981918640 1 -- 172.21.15.123:0/2307987561 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f797c0afd60 msgr2=0x7f797c0cfe00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:43:42.676 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:42.668+0000 7f7981918640 1 --2- 172.21.15.123:0/2307987561 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f797c0afd60 0x7f797c0cfe00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:42.676 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:42.668+0000 7f7981918640 1 -- 172.21.15.123:0/2307987561 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f797c156dd0 con 0x7f797c071d00 2024-09-16T15:43:42.676 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:42.668+0000 7f7981918640 1 --2- 172.21.15.123:0/2307987561 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f797c071d00 0x7f797c157b90 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f796c01a3c0 tx=0x7f796c005040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:43:42.676 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:42.668+0000 7f79727fc640 1 -- 172.21.15.123:0/2307987561 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f796c0ac070 con 0x7f797c071d00 2024-09-16T15:43:42.677 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:42.668+0000 7f79727fc640 1 -- 172.21.15.123:0/2307987561 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f796c005520 con 0x7f797c071d00 2024-09-16T15:43:42.677 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:42.668+0000 7f7983b7c640 1 -- 172.21.15.123:0/2307987561 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f797c1570b0 con 0x7f797c071d00 2024-09-16T15:43:42.677 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:42.668+0000 7f79727fc640 1 -- 172.21.15.123:0/2307987561 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f796c004290 con 0x7f797c071d00 2024-09-16T15:43:42.677 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:42.672+0000 7f7983b7c640 1 -- 172.21.15.123:0/2307987561 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f797c157670 con 0x7f797c071d00 2024-09-16T15:43:42.678 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:42.672+0000 7f7983b7c640 1 -- 172.21.15.123:0/2307987561 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f797c06cbf0 con 0x7f797c071d00 2024-09-16T15:43:42.678 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:42.672+0000 7f79727fc640 1 -- 172.21.15.123:0/2307987561 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 5) ==== 97080+0+0 (secure 0 0 0) 0x7f796c0af070 con 0x7f797c071d00 2024-09-16T15:43:42.678 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:42.672+0000 7f79727fc640 1 --2- 172.21.15.123:0/2307987561 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f7960072ec0 0x7f7960075380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:43:42.678 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:42.672+0000 7f79727fc640 1 -- 172.21.15.123:0/2307987561 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(32..32 src has 1..32) ==== 5709+0+0 (secure 0 0 0) 0x7f796c12e050 con 0x7f797c071d00 2024-09-16T15:43:42.678 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:42.672+0000 7f7981117640 1 --2- 172.21.15.123:0/2307987561 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f7960072ec0 0x7f7960075380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:43:42.678 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:42.672+0000 7f7981117640 1 --2- 172.21.15.123:0/2307987561 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f7960072ec0 0x7f7960075380 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f797c151260 tx=0x7f7978009040 comp rx=0 tx=0).ready entity=mgr.4135 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:43:42.681 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:42.676+0000 7f79727fc640 1 -- 172.21.15.123:0/2307987561 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f796c016410 con 0x7f797c071d00 2024-09-16T15:43:42.854 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:42.848+0000 7f7983b7c640 1 -- 172.21.15.123:0/2307987561 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f797c0b09d0 con 0x7f797c071d00 2024-09-16T15:43:42.855 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:42.848+0000 7f79727fc640 1 -- 172.21.15.123:0/2307987561 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v32) ==== 74+0+16802 (secure 0 0 0) 0x7f796c0f9180 con 0x7f797c071d00 2024-09-16T15:43:42.855 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-09-16T15:43:42.855 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":32,"fsid":"56077f89-18c8-486f-ae7e-eecc0bb7822c","created":"2024-09-16T15:42:26.364892+0000","modified":"2024-09-16T15:43:41.851697+0000","last_up_change":"2024-09-16T15:42:38.707234+0000","last_in_change":"2024-09-16T15:42:29.423662+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-16T15:42:39.287764+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2024-09-16T15:43:39.775631+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"32","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data","create_time":"2024-09-16T15:43:40.809234+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"32","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9e7c1955-a607-4671-9be7-f2ef760ceeaa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6800","nonce":3882044969},{"type":"v1","addr":"172.21.15.123:6801","nonce":3882044969}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6802","nonce":3882044969},{"type":"v1","addr":"172.21.15.123:6803","nonce":3882044969}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6806","nonce":3882044969},{"type":"v1","addr":"172.21.15.123:6807","nonce":3882044969}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6804","nonce":3882044969},{"type":"v1","addr":"172.21.15.123:6805","nonce":3882044969}]},"public_addr":"172.21.15.123:6801/3882044969","cluster_addr":"172.21.15.123:6803/3882044969","heartbeat_back_addr":"172.21.15.123:6807/3882044969","heartbeat_front_addr":"172.21.15.123:6805/3882044969","state":["exists","up"]},{"osd":1,"uuid":"c540cfde-0977-4742-9b17-fd634668e010","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6816","nonce":4037001850},{"type":"v1","addr":"172.21.15.123:6817","nonce":4037001850}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6818","nonce":4037001850},{"type":"v1","addr":"172.21.15.123:6819","nonce":4037001850}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6822","nonce":4037001850},{"type":"v1","addr":"172.21.15.123:6823","nonce":4037001850}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6820","nonce":4037001850},{"type":"v1","addr":"172.21.15.123:6821","nonce":4037001850}]},"public_addr":"172.21.15.123:6817/4037001850","cluster_addr":"172.21.15.123:6819/4037001850","heartbeat_back_addr":"172.21.15.123:6823/4037001850","heartbeat_front_addr":"172.21.15.123:6821/4037001850","state":["exists","up"]},{"osd":2,"uuid":"9864a496-f169-486b-95c4-1d59162793a7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6824","nonce":1332143088},{"type":"v1","addr":"172.21.15.123:6825","nonce":1332143088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6826","nonce":1332143088},{"type":"v1","addr":"172.21.15.123:6827","nonce":1332143088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6830","nonce":1332143088},{"type":"v1","addr":"172.21.15.123:6831","nonce":1332143088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6828","nonce":1332143088},{"type":"v1","addr":"172.21.15.123:6829","nonce":1332143088}]},"public_addr":"172.21.15.123:6825/1332143088","cluster_addr":"172.21.15.123:6827/1332143088","heartbeat_back_addr":"172.21.15.123:6831/1332143088","heartbeat_front_addr":"172.21.15.123:6829/1332143088","state":["exists","up"]},{"osd":3,"uuid":"c6d1f817-f074-4325-95e1-578eba13edda","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6808","nonce":988586998},{"type":"v1","addr":"172.21.15.123:6809","nonce":988586998}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6810","nonce":988586998},{"type":"v1","addr":"172.21.15.123:6811","nonce":988586998}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6814","nonce":988586998},{"type":"v1","addr":"172.21.15.123:6815","nonce":988586998}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6812","nonce":988586998},{"type":"v1","addr":"172.21.15.123:6813","nonce":988586998}]},"public_addr":"172.21.15.123:6809/988586998","cluster_addr":"172.21.15.123:6811/988586998","heartbeat_back_addr":"172.21.15.123:6815/988586998","heartbeat_front_addr":"172.21.15.123:6813/988586998","state":["exists","up"]},{"osd":4,"uuid":"200a8df4-50ef-40a5-b48a-58981269c086","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6816","nonce":3387813890},{"type":"v1","addr":"172.21.15.184:6817","nonce":3387813890}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6818","nonce":3387813890},{"type":"v1","addr":"172.21.15.184:6819","nonce":3387813890}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6822","nonce":3387813890},{"type":"v1","addr":"172.21.15.184:6823","nonce":3387813890}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6820","nonce":3387813890},{"type":"v1","addr":"172.21.15.184:6821","nonce":3387813890}]},"public_addr":"172.21.15.184:6817/3387813890","cluster_addr":"172.21.15.184:6819/3387813890","heartbeat_back_addr":"172.21.15.184:6823/3387813890","heartbeat_front_addr":"172.21.15.184:6821/3387813890","state":["exists","up"]},{"osd":5,"uuid":"32d5a900-da3d-412c-bc19-13861a2368d8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6808","nonce":2680916017},{"type":"v1","addr":"172.21.15.184:6809","nonce":2680916017}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6810","nonce":2680916017},{"type":"v1","addr":"172.21.15.184:6811","nonce":2680916017}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6814","nonce":2680916017},{"type":"v1","addr":"172.21.15.184:6815","nonce":2680916017}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6812","nonce":2680916017},{"type":"v1","addr":"172.21.15.184:6813","nonce":2680916017}]},"public_addr":"172.21.15.184:6809/2680916017","cluster_addr":"172.21.15.184:6811/2680916017","heartbeat_back_addr":"172.21.15.184:6815/2680916017","heartbeat_front_addr":"172.21.15.184:6813/2680916017","state":["exists","up"]},{"osd":6,"uuid":"cde6f39e-aa1b-4260-ac19-774ee6aaca72","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6824","nonce":2183120457},{"type":"v1","addr":"172.21.15.184:6825","nonce":2183120457}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6826","nonce":2183120457},{"type":"v1","addr":"172.21.15.184:6827","nonce":2183120457}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6830","nonce":2183120457},{"type":"v1","addr":"172.21.15.184:6831","nonce":2183120457}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6828","nonce":2183120457},{"type":"v1","addr":"172.21.15.184:6829","nonce":2183120457}]},"public_addr":"172.21.15.184:6825/2183120457","cluster_addr":"172.21.15.184:6827/2183120457","heartbeat_back_addr":"172.21.15.184:6831/2183120457","heartbeat_front_addr":"172.21.15.184:6829/2183120457","state":["exists","up"]},{"osd":7,"uuid":"4f381778-eee1-47c6-804f-df0e6b33e313","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6800","nonce":4102957285},{"type":"v1","addr":"172.21.15.184:6801","nonce":4102957285}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6802","nonce":4102957285},{"type":"v1","addr":"172.21.15.184:6803","nonce":4102957285}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6806","nonce":4102957285},{"type":"v1","addr":"172.21.15.184:6807","nonce":4102957285}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6804","nonce":4102957285},{"type":"v1","addr":"172.21.15.184:6805","nonce":4102957285}]},"public_addr":"172.21.15.184:6801/4102957285","cluster_addr":"172.21.15.184:6803/4102957285","heartbeat_back_addr":"172.21.15.184:6807/4102957285","heartbeat_front_addr":"172.21.15.184:6805/4102957285","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:35.972210+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.222007+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.198617+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.385663+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.358935+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.156066+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.752973+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:35.682846+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T15:43:42.857 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:42.852+0000 7f7983b7c640 1 -- 172.21.15.123:0/2307987561 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f7960072ec0 msgr2=0x7f7960075380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:43:42.857 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:42.852+0000 7f7983b7c640 1 --2- 172.21.15.123:0/2307987561 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f7960072ec0 0x7f7960075380 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f797c151260 tx=0x7f7978009040 comp rx=0 tx=0).stop 2024-09-16T15:43:42.858 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:42.852+0000 7f7983b7c640 1 -- 172.21.15.123:0/2307987561 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f797c071d00 msgr2=0x7f797c157b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:43:42.858 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:42.852+0000 7f7983b7c640 1 --2- 172.21.15.123:0/2307987561 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f797c071d00 0x7f797c157b90 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f796c01a3c0 tx=0x7f796c005040 comp rx=0 tx=0).stop 2024-09-16T15:43:42.858 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:42.852+0000 7f7983b7c640 1 -- 172.21.15.123:0/2307987561 shutdown_connections 2024-09-16T15:43:42.858 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:42.852+0000 7f7983b7c640 1 --2- 172.21.15.123:0/2307987561 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f7960072ec0 0x7f7960075380 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:42.858 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:42.852+0000 7f7983b7c640 1 --2- 172.21.15.123:0/2307987561 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f797c15bd30 0x7f797c14a8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:42.858 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:42.852+0000 7f7983b7c640 1 --2- 172.21.15.123:0/2307987561 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f797c0afd60 0x7f797c0cfe00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:42.858 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:42.852+0000 7f7983b7c640 1 --2- 172.21.15.123:0/2307987561 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f797c071d00 0x7f797c157b90 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:42.858 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:42.852+0000 7f7983b7c640 1 -- 172.21.15.123:0/2307987561 >> 172.21.15.123:0/2307987561 conn(0x7f797c0b2610 msgr2=0x7f797c0b42d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:43:42.858 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:42.852+0000 7f7983b7c640 1 -- 172.21.15.123:0/2307987561 shutdown_connections 2024-09-16T15:43:42.858 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:42.852+0000 7f7983b7c640 1 -- 172.21.15.123:0/2307987561 wait complete. 2024-09-16T15:43:42.876 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-16T15:43:43.000 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:42.996+0000 7f3ef8ac1640 1 -- 172.21.15.123:0/3150361095 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3ef40726c0 msgr2=0x7f3ef4073ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:43:43.000 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:42.996+0000 7f3ef8ac1640 1 --2- 172.21.15.123:0/3150361095 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3ef40726c0 0x7f3ef4073ef0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f3ef40cd650 tx=0x7f3ee801c540 comp rx=0 tx=0).stop 2024-09-16T15:43:43.001 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:42.996+0000 7f3ef8ac1640 1 -- 172.21.15.123:0/3150361095 shutdown_connections 2024-09-16T15:43:43.001 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:42.996+0000 7f3ef8ac1640 1 --2- 172.21.15.123:0/3150361095 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f3ef4074430 0x7f3ef40ca160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:43.001 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:42.996+0000 7f3ef8ac1640 1 --2- 172.21.15.123:0/3150361095 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3ef40726c0 0x7f3ef4073ef0 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:43.001 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:42.996+0000 7f3ef8ac1640 1 --2- 172.21.15.123:0/3150361095 >> v2:172.21.15.184:3300/0 conn(0x7f3ef415bca0 0x7f3ef40d0630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:43.001 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:42.996+0000 7f3ef8ac1640 1 -- 172.21.15.123:0/3150361095 >> 172.21.15.123:0/3150361095 conn(0x7f3ef40cfb10 msgr2=0x7f3ef40cff20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:43:43.001 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:42.996+0000 7f3ef8ac1640 1 -- 172.21.15.123:0/3150361095 shutdown_connections 2024-09-16T15:43:43.001 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:42.996+0000 7f3ef8ac1640 1 -- 172.21.15.123:0/3150361095 wait complete. 2024-09-16T15:43:43.002 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:42.996+0000 7f3ef8ac1640 1 Processor -- start 2024-09-16T15:43:43.002 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:42.996+0000 7f3ef8ac1640 1 -- start start 2024-09-16T15:43:43.003 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:42.996+0000 7f3ef8ac1640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f3ef40726c0 0x7f3ef414b8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:43:43.003 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:42.996+0000 7f3ef8ac1640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3ef4074430 0x7f3ef414cc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:43:43.004 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:42.996+0000 7f3ef8ac1640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3ef415bca0 0x7f3ef4165c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:43:43.004 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:42.996+0000 7f3ef8ac1640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f3ef406ec50 con 0x7f3ef4074430 2024-09-16T15:43:43.004 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:42.996+0000 7f3ef8ac1640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f3ef406ead0 con 0x7f3ef415bca0 2024-09-16T15:43:43.004 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:42.996+0000 7f3ef8ac1640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f3ef406edd0 con 0x7f3ef40726c0 2024-09-16T15:43:43.004 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:42.996+0000 7f3ef1d9b640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3ef4074430 0x7f3ef414cc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:43:43.004 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:42.996+0000 7f3ef1d9b640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3ef4074430 0x7f3ef414cc00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:41560/0 (socket says 172.21.15.123:41560) 2024-09-16T15:43:43.004 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:42.996+0000 7f3ef1d9b640 1 -- 172.21.15.123:0/3617523720 learned_addr learned my addr 172.21.15.123:0/3617523720 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:43:43.004 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:42.996+0000 7f3ef259c640 1 --2- 172.21.15.123:0/3617523720 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f3ef40726c0 0x7f3ef414b8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:43:43.004 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:42.996+0000 7f3ef2d9d640 1 --2- 172.21.15.123:0/3617523720 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3ef415bca0 0x7f3ef4165c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:43:43.004 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:42.996+0000 7f3ef1d9b640 1 -- 172.21.15.123:0/3617523720 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f3ef40726c0 msgr2=0x7f3ef414b8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:43:43.004 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:42.996+0000 7f3ef1d9b640 1 --2- 172.21.15.123:0/3617523720 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f3ef40726c0 0x7f3ef414b8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:43.004 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:42.996+0000 7f3ef1d9b640 1 -- 172.21.15.123:0/3617523720 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3ef415bca0 msgr2=0x7f3ef4165c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:43:43.004 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:42.996+0000 7f3ef1d9b640 1 --2- 172.21.15.123:0/3617523720 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3ef415bca0 0x7f3ef4165c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:43.005 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:42.996+0000 7f3ef1d9b640 1 -- 172.21.15.123:0/3617523720 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3ef40cee80 con 0x7f3ef4074430 2024-09-16T15:43:43.005 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:43.000+0000 7f3ef1d9b640 1 --2- 172.21.15.123:0/3617523720 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3ef4074430 0x7f3ef414cc00 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f3ee801abf0 tx=0x7f3ee80a84c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:43:43.005 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:43.000+0000 7f3ee2ffd640 1 -- 172.21.15.123:0/3617523720 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ee80bf070 con 0x7f3ef4074430 2024-09-16T15:43:43.005 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:43.000+0000 7f3ee2ffd640 1 -- 172.21.15.123:0/3617523720 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f3ee8004040 con 0x7f3ef4074430 2024-09-16T15:43:43.005 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:43.000+0000 7f3ef8ac1640 1 -- 172.21.15.123:0/3617523720 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3ef4164e80 con 0x7f3ef4074430 2024-09-16T15:43:43.005 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:43.000+0000 7f3ee2ffd640 1 -- 172.21.15.123:0/3617523720 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ee8004490 con 0x7f3ef4074430 2024-09-16T15:43:43.005 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:43.000+0000 7f3ef8ac1640 1 -- 172.21.15.123:0/3617523720 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3ef4165440 con 0x7f3ef4074430 2024-09-16T15:43:43.006 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:43.000+0000 7f3ef8ac1640 1 -- 172.21.15.123:0/3617523720 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3ef415f630 con 0x7f3ef4074430 2024-09-16T15:43:43.006 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:43.000+0000 7f3ee2ffd640 1 -- 172.21.15.123:0/3617523720 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 5) ==== 97080+0+0 (secure 0 0 0) 0x7f3ee80c2070 con 0x7f3ef4074430 2024-09-16T15:43:43.006 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:43.000+0000 7f3ee2ffd640 1 --2- 172.21.15.123:0/3617523720 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f3ed4072ec0 0x7f3ed4075380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:43:43.006 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:43.000+0000 7f3ee2ffd640 1 -- 172.21.15.123:0/3617523720 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(32..32 src has 1..32) ==== 5709+0+0 (secure 0 0 0) 0x7f3ee812e190 con 0x7f3ef4074430 2024-09-16T15:43:43.006 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:43.000+0000 7f3ef259c640 1 --2- 172.21.15.123:0/3617523720 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f3ed4072ec0 0x7f3ed4075380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:43:43.006 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:43.000+0000 7f3ef259c640 1 --2- 172.21.15.123:0/3617523720 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f3ed4072ec0 0x7f3ed4075380 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f3ee40058d0 tx=0x7f3ee4005860 comp rx=0 tx=0).ready entity=mgr.4135 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:43:43.010 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:43.004+0000 7f3ee2ffd640 1 -- 172.21.15.123:0/3617523720 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f3ee80ac030 con 0x7f3ef4074430 2024-09-16T15:43:43.181 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:43.176+0000 7f3ef8ac1640 1 -- 172.21.15.123:0/3617523720 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f3ef40700e0 con 0x7f3ef4074430 2024-09-16T15:43:43.183 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:43.180+0000 7f3ee2ffd640 1 -- 172.21.15.123:0/3617523720 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v32) ==== 74+0+16802 (secure 0 0 0) 0x7f3ee80164b0 con 0x7f3ef4074430 2024-09-16T15:43:43.184 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-09-16T15:43:43.184 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":32,"fsid":"56077f89-18c8-486f-ae7e-eecc0bb7822c","created":"2024-09-16T15:42:26.364892+0000","modified":"2024-09-16T15:43:41.851697+0000","last_up_change":"2024-09-16T15:42:38.707234+0000","last_in_change":"2024-09-16T15:42:29.423662+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-16T15:42:39.287764+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2024-09-16T15:43:39.775631+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"32","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data","create_time":"2024-09-16T15:43:40.809234+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"32","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9e7c1955-a607-4671-9be7-f2ef760ceeaa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6800","nonce":3882044969},{"type":"v1","addr":"172.21.15.123:6801","nonce":3882044969}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6802","nonce":3882044969},{"type":"v1","addr":"172.21.15.123:6803","nonce":3882044969}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6806","nonce":3882044969},{"type":"v1","addr":"172.21.15.123:6807","nonce":3882044969}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6804","nonce":3882044969},{"type":"v1","addr":"172.21.15.123:6805","nonce":3882044969}]},"public_addr":"172.21.15.123:6801/3882044969","cluster_addr":"172.21.15.123:6803/3882044969","heartbeat_back_addr":"172.21.15.123:6807/3882044969","heartbeat_front_addr":"172.21.15.123:6805/3882044969","state":["exists","up"]},{"osd":1,"uuid":"c540cfde-0977-4742-9b17-fd634668e010","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6816","nonce":4037001850},{"type":"v1","addr":"172.21.15.123:6817","nonce":4037001850}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6818","nonce":4037001850},{"type":"v1","addr":"172.21.15.123:6819","nonce":4037001850}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6822","nonce":4037001850},{"type":"v1","addr":"172.21.15.123:6823","nonce":4037001850}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6820","nonce":4037001850},{"type":"v1","addr":"172.21.15.123:6821","nonce":4037001850}]},"public_addr":"172.21.15.123:6817/4037001850","cluster_addr":"172.21.15.123:6819/4037001850","heartbeat_back_addr":"172.21.15.123:6823/4037001850","heartbeat_front_addr":"172.21.15.123:6821/4037001850","state":["exists","up"]},{"osd":2,"uuid":"9864a496-f169-486b-95c4-1d59162793a7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6824","nonce":1332143088},{"type":"v1","addr":"172.21.15.123:6825","nonce":1332143088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6826","nonce":1332143088},{"type":"v1","addr":"172.21.15.123:6827","nonce":1332143088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6830","nonce":1332143088},{"type":"v1","addr":"172.21.15.123:6831","nonce":1332143088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6828","nonce":1332143088},{"type":"v1","addr":"172.21.15.123:6829","nonce":1332143088}]},"public_addr":"172.21.15.123:6825/1332143088","cluster_addr":"172.21.15.123:6827/1332143088","heartbeat_back_addr":"172.21.15.123:6831/1332143088","heartbeat_front_addr":"172.21.15.123:6829/1332143088","state":["exists","up"]},{"osd":3,"uuid":"c6d1f817-f074-4325-95e1-578eba13edda","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6808","nonce":988586998},{"type":"v1","addr":"172.21.15.123:6809","nonce":988586998}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6810","nonce":988586998},{"type":"v1","addr":"172.21.15.123:6811","nonce":988586998}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6814","nonce":988586998},{"type":"v1","addr":"172.21.15.123:6815","nonce":988586998}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6812","nonce":988586998},{"type":"v1","addr":"172.21.15.123:6813","nonce":988586998}]},"public_addr":"172.21.15.123:6809/988586998","cluster_addr":"172.21.15.123:6811/988586998","heartbeat_back_addr":"172.21.15.123:6815/988586998","heartbeat_front_addr":"172.21.15.123:6813/988586998","state":["exists","up"]},{"osd":4,"uuid":"200a8df4-50ef-40a5-b48a-58981269c086","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6816","nonce":3387813890},{"type":"v1","addr":"172.21.15.184:6817","nonce":3387813890}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6818","nonce":3387813890},{"type":"v1","addr":"172.21.15.184:6819","nonce":3387813890}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6822","nonce":3387813890},{"type":"v1","addr":"172.21.15.184:6823","nonce":3387813890}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6820","nonce":3387813890},{"type":"v1","addr":"172.21.15.184:6821","nonce":3387813890}]},"public_addr":"172.21.15.184:6817/3387813890","cluster_addr":"172.21.15.184:6819/3387813890","heartbeat_back_addr":"172.21.15.184:6823/3387813890","heartbeat_front_addr":"172.21.15.184:6821/3387813890","state":["exists","up"]},{"osd":5,"uuid":"32d5a900-da3d-412c-bc19-13861a2368d8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6808","nonce":2680916017},{"type":"v1","addr":"172.21.15.184:6809","nonce":2680916017}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6810","nonce":2680916017},{"type":"v1","addr":"172.21.15.184:6811","nonce":2680916017}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6814","nonce":2680916017},{"type":"v1","addr":"172.21.15.184:6815","nonce":2680916017}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6812","nonce":2680916017},{"type":"v1","addr":"172.21.15.184:6813","nonce":2680916017}]},"public_addr":"172.21.15.184:6809/2680916017","cluster_addr":"172.21.15.184:6811/2680916017","heartbeat_back_addr":"172.21.15.184:6815/2680916017","heartbeat_front_addr":"172.21.15.184:6813/2680916017","state":["exists","up"]},{"osd":6,"uuid":"cde6f39e-aa1b-4260-ac19-774ee6aaca72","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6824","nonce":2183120457},{"type":"v1","addr":"172.21.15.184:6825","nonce":2183120457}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6826","nonce":2183120457},{"type":"v1","addr":"172.21.15.184:6827","nonce":2183120457}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6830","nonce":2183120457},{"type":"v1","addr":"172.21.15.184:6831","nonce":2183120457}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6828","nonce":2183120457},{"type":"v1","addr":"172.21.15.184:6829","nonce":2183120457}]},"public_addr":"172.21.15.184:6825/2183120457","cluster_addr":"172.21.15.184:6827/2183120457","heartbeat_back_addr":"172.21.15.184:6831/2183120457","heartbeat_front_addr":"172.21.15.184:6829/2183120457","state":["exists","up"]},{"osd":7,"uuid":"4f381778-eee1-47c6-804f-df0e6b33e313","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6800","nonce":4102957285},{"type":"v1","addr":"172.21.15.184:6801","nonce":4102957285}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6802","nonce":4102957285},{"type":"v1","addr":"172.21.15.184:6803","nonce":4102957285}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6806","nonce":4102957285},{"type":"v1","addr":"172.21.15.184:6807","nonce":4102957285}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6804","nonce":4102957285},{"type":"v1","addr":"172.21.15.184:6805","nonce":4102957285}]},"public_addr":"172.21.15.184:6801/4102957285","cluster_addr":"172.21.15.184:6803/4102957285","heartbeat_back_addr":"172.21.15.184:6807/4102957285","heartbeat_front_addr":"172.21.15.184:6805/4102957285","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:35.972210+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.222007+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.198617+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.385663+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.358935+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.156066+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.752973+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:35.682846+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T15:43:43.186 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:43.180+0000 7f3ef8ac1640 1 -- 172.21.15.123:0/3617523720 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f3ed4072ec0 msgr2=0x7f3ed4075380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:43:43.186 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:43.180+0000 7f3ef8ac1640 1 --2- 172.21.15.123:0/3617523720 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f3ed4072ec0 0x7f3ed4075380 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f3ee40058d0 tx=0x7f3ee4005860 comp rx=0 tx=0).stop 2024-09-16T15:43:43.186 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:43.180+0000 7f3ef8ac1640 1 -- 172.21.15.123:0/3617523720 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3ef4074430 msgr2=0x7f3ef414cc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:43:43.186 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:43.180+0000 7f3ef8ac1640 1 --2- 172.21.15.123:0/3617523720 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3ef4074430 0x7f3ef414cc00 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f3ee801abf0 tx=0x7f3ee80a84c0 comp rx=0 tx=0).stop 2024-09-16T15:43:43.186 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:43.180+0000 7f3ef8ac1640 1 -- 172.21.15.123:0/3617523720 shutdown_connections 2024-09-16T15:43:43.186 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:43.180+0000 7f3ef8ac1640 1 --2- 172.21.15.123:0/3617523720 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f3ed4072ec0 0x7f3ed4075380 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:43.186 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:43.180+0000 7f3ef8ac1640 1 --2- 172.21.15.123:0/3617523720 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3ef415bca0 0x7f3ef4165c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:43.186 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:43.180+0000 7f3ef8ac1640 1 --2- 172.21.15.123:0/3617523720 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3ef4074430 0x7f3ef414cc00 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:43.186 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:43.180+0000 7f3ef8ac1640 1 --2- 172.21.15.123:0/3617523720 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f3ef40726c0 0x7f3ef414b8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:43.187 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:43.180+0000 7f3ef8ac1640 1 -- 172.21.15.123:0/3617523720 >> 172.21.15.123:0/3617523720 conn(0x7f3ef40cfb10 msgr2=0x7f3ef40c3b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:43:43.187 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:43.180+0000 7f3ef8ac1640 1 -- 172.21.15.123:0/3617523720 shutdown_connections 2024-09-16T15:43:43.187 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:43.180+0000 7f3ef8ac1640 1 -- 172.21.15.123:0/3617523720 wait complete. 2024-09-16T15:43:43.202 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-09-16T15:43:43.334 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:43.328+0000 7f5e6bfe0640 1 -- 172.21.15.123:0/4176399905 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5e6415a9d0 msgr2=0x7f5e6406ebe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:43:43.334 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:43.328+0000 7f5e6bfe0640 1 --2- 172.21.15.123:0/4176399905 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5e6415a9d0 0x7f5e6406ebe0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f5e58009900 tx=0x7f5e5801c820 comp rx=0 tx=0).stop 2024-09-16T15:43:43.334 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:43.328+0000 7f5e6bfe0640 1 -- 172.21.15.123:0/4176399905 shutdown_connections 2024-09-16T15:43:43.334 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:43.328+0000 7f5e6bfe0640 1 --2- 172.21.15.123:0/4176399905 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5e6415a9d0 0x7f5e6406ebe0 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:43.334 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:43.328+0000 7f5e6bfe0640 1 -- 172.21.15.123:0/4176399905 >> 172.21.15.123:0/4176399905 conn(0x7f5e640b1710 msgr2=0x7f5e640b1b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:43:43.335 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:43.328+0000 7f5e6bfe0640 1 -- 172.21.15.123:0/4176399905 shutdown_connections 2024-09-16T15:43:43.335 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:43.328+0000 7f5e6bfe0640 1 -- 172.21.15.123:0/4176399905 wait complete. 2024-09-16T15:43:43.335 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:43.332+0000 7f5e6bfe0640 1 Processor -- start 2024-09-16T15:43:43.335 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:43.332+0000 7f5e6bfe0640 1 -- start start 2024-09-16T15:43:43.337 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:43.332+0000 7f5e6bfe0640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5e64072c70 0x7f5e6414bf00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:43:43.337 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:43.332+0000 7f5e6bfe0640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5e640ccad0 0x7f5e64154fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:43:43.337 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:43.332+0000 7f5e6bfe0640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f5e6415a9d0 0x7f5e641533e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:43:43.337 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:43.332+0000 7f5e6bfe0640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f5e64169420 con 0x7f5e640ccad0 2024-09-16T15:43:43.337 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:43.332+0000 7f5e6bfe0640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f5e6415af50 con 0x7f5e64072c70 2024-09-16T15:43:43.337 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:43.332+0000 7f5e6bfe0640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f5e641695a0 con 0x7f5e6415a9d0 2024-09-16T15:43:43.338 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:43.332+0000 7f5e6957b640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5e640ccad0 0x7f5e64154fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:43:43.338 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:43.332+0000 7f5e6957b640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5e640ccad0 0x7f5e64154fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:41578/0 (socket says 172.21.15.123:41578) 2024-09-16T15:43:43.338 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:43.332+0000 7f5e6957b640 1 -- 172.21.15.123:0/3791526900 learned_addr learned my addr 172.21.15.123:0/3791526900 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:43:43.338 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:43.332+0000 7f5e69d7c640 1 --2- 172.21.15.123:0/3791526900 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5e64072c70 0x7f5e6414bf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:43:43.338 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:43.332+0000 7f5e6a57d640 1 --2- 172.21.15.123:0/3791526900 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f5e6415a9d0 0x7f5e641533e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:43:43.338 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:43.332+0000 7f5e6957b640 1 -- 172.21.15.123:0/3791526900 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f5e6415a9d0 msgr2=0x7f5e641533e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:43:43.338 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:43.332+0000 7f5e6957b640 1 --2- 172.21.15.123:0/3791526900 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f5e6415a9d0 0x7f5e641533e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:43.338 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:43.332+0000 7f5e6957b640 1 -- 172.21.15.123:0/3791526900 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5e64072c70 msgr2=0x7f5e6414bf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:43:43.338 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:43.332+0000 7f5e6957b640 1 --2- 172.21.15.123:0/3791526900 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5e64072c70 0x7f5e6414bf00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:43.338 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:43.332+0000 7f5e6957b640 1 -- 172.21.15.123:0/3791526900 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5e64155500 con 0x7f5e640ccad0 2024-09-16T15:43:43.338 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:43.332+0000 7f5e69d7c640 1 --2- 172.21.15.123:0/3791526900 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5e64072c70 0x7f5e6414bf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T15:43:43.338 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:43.332+0000 7f5e6957b640 1 --2- 172.21.15.123:0/3791526900 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5e640ccad0 0x7f5e64154fc0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f5e6000c3c0 tx=0x7f5e6000c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:43:43.338 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:43.332+0000 7f5e567fc640 1 -- 172.21.15.123:0/3791526900 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e60017070 con 0x7f5e640ccad0 2024-09-16T15:43:43.339 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:43.332+0000 7f5e567fc640 1 -- 172.21.15.123:0/3791526900 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f5e60003600 con 0x7f5e640ccad0 2024-09-16T15:43:43.339 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:43.332+0000 7f5e6bfe0640 1 -- 172.21.15.123:0/3791526900 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5e6414db50 con 0x7f5e640ccad0 2024-09-16T15:43:43.339 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:43.332+0000 7f5e567fc640 1 -- 172.21.15.123:0/3791526900 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e60003a50 con 0x7f5e640ccad0 2024-09-16T15:43:43.339 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:43.332+0000 7f5e6bfe0640 1 -- 172.21.15.123:0/3791526900 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5e64156160 con 0x7f5e640ccad0 2024-09-16T15:43:43.340 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:43.332+0000 7f5e6bfe0640 1 -- 172.21.15.123:0/3791526900 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5e6406ebe0 con 0x7f5e640ccad0 2024-09-16T15:43:43.340 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:43.332+0000 7f5e567fc640 1 -- 172.21.15.123:0/3791526900 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 5) ==== 97080+0+0 (secure 0 0 0) 0x7f5e6001a070 con 0x7f5e640ccad0 2024-09-16T15:43:43.340 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:43.336+0000 7f5e567fc640 1 --2- 172.21.15.123:0/3791526900 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f5e48072ec0 0x7f5e48075380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:43:43.340 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:43.336+0000 7f5e567fc640 1 -- 172.21.15.123:0/3791526900 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(32..32 src has 1..32) ==== 5709+0+0 (secure 0 0 0) 0x7f5e600933b0 con 0x7f5e640ccad0 2024-09-16T15:43:43.340 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:43.336+0000 7f5e69d7c640 1 --2- 172.21.15.123:0/3791526900 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f5e48072ec0 0x7f5e48075380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:43:43.340 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:43.336+0000 7f5e69d7c640 1 --2- 172.21.15.123:0/3791526900 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f5e48072ec0 0x7f5e48075380 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f5e580af000 tx=0x7f5e5801b060 comp rx=0 tx=0).ready entity=mgr.4135 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:43:43.342 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:43.336+0000 7f5e567fc640 1 -- 172.21.15.123:0/3791526900 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f5e60021020 con 0x7f5e640ccad0 2024-09-16T15:43:43.541 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:43.536+0000 7f5e6bfe0640 1 -- 172.21.15.123:0/3791526900 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) -- 0x7f5e640ca9a0 con 0x7f5e640ccad0 2024-09-16T15:43:44.540 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:44.536+0000 7f5e567fc640 1 -- 172.21.15.123:0/3791526900 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v17) ==== 118+0+0 (secure 0 0 0) 0x7f5e64155500 con 0x7f5e640ccad0 2024-09-16T15:43:44.544 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:44.540+0000 7f5e6bfe0640 1 -- 172.21.15.123:0/3791526900 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f5e48072ec0 msgr2=0x7f5e48075380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:43:44.545 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:44.540+0000 7f5e6bfe0640 1 --2- 172.21.15.123:0/3791526900 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f5e48072ec0 0x7f5e48075380 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f5e580af000 tx=0x7f5e5801b060 comp rx=0 tx=0).stop 2024-09-16T15:43:44.545 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:44.540+0000 7f5e6bfe0640 1 -- 172.21.15.123:0/3791526900 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5e640ccad0 msgr2=0x7f5e64154fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:43:44.545 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:44.540+0000 7f5e6bfe0640 1 --2- 172.21.15.123:0/3791526900 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5e640ccad0 0x7f5e64154fc0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f5e6000c3c0 tx=0x7f5e6000c880 comp rx=0 tx=0).stop 2024-09-16T15:43:44.545 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:44.540+0000 7f5e6bfe0640 1 -- 172.21.15.123:0/3791526900 shutdown_connections 2024-09-16T15:43:44.545 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:44.540+0000 7f5e6bfe0640 1 --2- 172.21.15.123:0/3791526900 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f5e48072ec0 0x7f5e48075380 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:44.545 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:44.540+0000 7f5e6bfe0640 1 --2- 172.21.15.123:0/3791526900 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f5e6415a9d0 0x7f5e641533e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:44.545 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:44.540+0000 7f5e6bfe0640 1 --2- 172.21.15.123:0/3791526900 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5e640ccad0 0x7f5e64154fc0 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:44.545 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:44.540+0000 7f5e6bfe0640 1 --2- 172.21.15.123:0/3791526900 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5e64072c70 0x7f5e6414bf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:44.545 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:44.540+0000 7f5e6bfe0640 1 -- 172.21.15.123:0/3791526900 >> 172.21.15.123:0/3791526900 conn(0x7f5e640b1710 msgr2=0x7f5e640cc3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:43:44.545 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:44.540+0000 7f5e6bfe0640 1 -- 172.21.15.123:0/3791526900 shutdown_connections 2024-09-16T15:43:44.545 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:44.540+0000 7f5e6bfe0640 1 -- 172.21.15.123:0/3791526900 wait complete. 2024-09-16T15:43:44.567 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-09-16T15:43:44.568 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-09-16T15:43:44.700 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:44.696+0000 7fabc8edf640 1 -- 172.21.15.123:0/3102702604 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fabac004d50 msgr2=0x7fabac0051d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:43:44.700 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:44.696+0000 7fabc8edf640 1 --2- 172.21.15.123:0/3102702604 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fabac004d50 0x7fabac0051d0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7fabb4005fd0 tx=0x7fabb401cd80 comp rx=0 tx=0).stop 2024-09-16T15:43:44.701 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:44.696+0000 7fabc2d9d640 1 -- 172.21.15.123:0/3102702604 reap_dead start 2024-09-16T15:43:44.701 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:44.696+0000 7fabc8edf640 1 -- 172.21.15.123:0/3102702604 shutdown_connections 2024-09-16T15:43:44.701 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:44.696+0000 7fabc8edf640 1 -- 172.21.15.123:0/3102702604 >> 172.21.15.123:0/3102702604 conn(0x7fabc40b1710 msgr2=0x7fabc40b1b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:43:44.701 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:44.696+0000 7fabc8edf640 1 -- 172.21.15.123:0/3102702604 shutdown_connections 2024-09-16T15:43:44.701 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:44.696+0000 7fabc8edf640 1 -- 172.21.15.123:0/3102702604 wait complete. 2024-09-16T15:43:44.701 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:44.696+0000 7fabc8edf640 1 Processor -- start 2024-09-16T15:43:44.701 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:44.696+0000 7fabc8edf640 1 -- start start 2024-09-16T15:43:44.703 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:44.696+0000 7fabc8edf640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fabac004d50 0x7fabc41f8c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:43:44.704 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:44.696+0000 7fabc8edf640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fabc41f5920 0x7fabc40b0e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:43:44.704 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:44.696+0000 7fabc8edf640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fabc40c5620 0x7fabc4156dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:43:44.704 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:44.696+0000 7fabc8edf640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fabc415f6b0 con 0x7fabac004d50 2024-09-16T15:43:44.704 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:44.696+0000 7fabc8edf640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fabc415be50 con 0x7fabc40c5620 2024-09-16T15:43:44.704 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:44.696+0000 7fabc8edf640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fabc415f830 con 0x7fabc41f5920 2024-09-16T15:43:44.704 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:44.696+0000 7fabc259c640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fabac004d50 0x7fabc41f8c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:43:44.704 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:44.696+0000 7fabc259c640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fabac004d50 0x7fabc41f8c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:41604/0 (socket says 172.21.15.123:41604) 2024-09-16T15:43:44.704 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:44.696+0000 7fabc259c640 1 -- 172.21.15.123:0/2976159666 learned_addr learned my addr 172.21.15.123:0/2976159666 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:43:44.704 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:44.696+0000 7fabc2d9d640 1 --2- 172.21.15.123:0/2976159666 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fabc40c5620 0x7fabc4156dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:43:44.704 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:44.696+0000 7fabc259c640 1 -- 172.21.15.123:0/2976159666 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fabc41f5920 msgr2=0x7fabc40b0e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:43:44.704 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:44.696+0000 7fabc259c640 1 --2- 172.21.15.123:0/2976159666 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fabc41f5920 0x7fabc40b0e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:44.704 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:44.696+0000 7fabc259c640 1 -- 172.21.15.123:0/2976159666 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fabc40c5620 msgr2=0x7fabc4156dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:43:44.705 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:44.696+0000 7fabc259c640 1 --2- 172.21.15.123:0/2976159666 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fabc40c5620 0x7fabc4156dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:44.705 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:44.696+0000 7fabc259c640 1 -- 172.21.15.123:0/2976159666 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fabc41f91d0 con 0x7fabac004d50 2024-09-16T15:43:44.705 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:44.696+0000 7fabc2d9d640 1 --2- 172.21.15.123:0/2976159666 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fabc40c5620 0x7fabc4156dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T15:43:44.705 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:44.696+0000 7fabc259c640 1 --2- 172.21.15.123:0/2976159666 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fabac004d50 0x7fabc41f8c90 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7fabbc00ec10 tx=0x7fabbc00c640 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:43:44.705 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:44.696+0000 7fabb2ffd640 1 -- 172.21.15.123:0/2976159666 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fabbc011470 con 0x7fabac004d50 2024-09-16T15:43:44.705 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:44.696+0000 7fabb2ffd640 1 -- 172.21.15.123:0/2976159666 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fabbc011700 con 0x7fabac004d50 2024-09-16T15:43:44.705 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:44.696+0000 7fabc8edf640 1 -- 172.21.15.123:0/2976159666 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fabc41f4ea0 con 0x7fabac004d50 2024-09-16T15:43:44.705 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:44.696+0000 7fabb2ffd640 1 -- 172.21.15.123:0/2976159666 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fabbc011b10 con 0x7fabac004d50 2024-09-16T15:43:44.705 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:44.696+0000 7fabc8edf640 1 -- 172.21.15.123:0/2976159666 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fabc41f5140 con 0x7fabac004d50 2024-09-16T15:43:44.705 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:44.700+0000 7fabc8edf640 1 -- 172.21.15.123:0/2976159666 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fabc406fca0 con 0x7fabac004d50 2024-09-16T15:43:44.706 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:44.700+0000 7fabb2ffd640 1 -- 172.21.15.123:0/2976159666 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 5) ==== 97080+0+0 (secure 0 0 0) 0x7fabbc01a400 con 0x7fabac004d50 2024-09-16T15:43:44.706 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:44.700+0000 7fabb2ffd640 1 --2- 172.21.15.123:0/2976159666 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7faba4072ec0 0x7faba4075380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:43:44.706 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:44.700+0000 7fabb2ffd640 1 -- 172.21.15.123:0/2976159666 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(33..33 src has 1..33) ==== 5709+0+0 (secure 0 0 0) 0x7fabbc094fa0 con 0x7fabac004d50 2024-09-16T15:43:44.706 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:44.700+0000 7fabc1d9b640 1 --2- 172.21.15.123:0/2976159666 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7faba4072ec0 0x7faba4075380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:43:44.706 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:44.700+0000 7fabc1d9b640 1 --2- 172.21.15.123:0/2976159666 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7faba4072ec0 0x7faba4075380 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fabc40cc370 tx=0x7fabb8001480 comp rx=0 tx=0).ready entity=mgr.4135 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:43:44.709 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:44.704+0000 7fabb2ffd640 1 -- 172.21.15.123:0/2976159666 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7fabc41f91d0 con 0x7fabac004d50 2024-09-16T15:43:44.918 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:44.912+0000 7fabc8edf640 1 -- 172.21.15.123:0/2976159666 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) -- 0x7fabc406e3b0 con 0x7fabac004d50 2024-09-16T15:43:45.419 INFO:tasks.ceph.mds.a.smithi123.stderr:2024-09-16T15:43:45.412+0000 7f85fdbc5640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-09-16T15:43:45.554 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:45.548+0000 7fabb2ffd640 1 -- 172.21.15.123:0/2976159666 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v18) ==== 122+0+0 (secure 0 0 0) 0x7fabbc016030 con 0x7fabac004d50 2024-09-16T15:43:45.558 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:45.552+0000 7fabc8edf640 1 -- 172.21.15.123:0/2976159666 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7faba4072ec0 msgr2=0x7faba4075380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:43:45.558 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:45.552+0000 7fabc8edf640 1 --2- 172.21.15.123:0/2976159666 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7faba4072ec0 0x7faba4075380 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fabc40cc370 tx=0x7fabb8001480 comp rx=0 tx=0).stop 2024-09-16T15:43:45.558 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:45.552+0000 7fabc8edf640 1 -- 172.21.15.123:0/2976159666 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fabac004d50 msgr2=0x7fabc41f8c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:43:45.558 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:45.552+0000 7fabc8edf640 1 --2- 172.21.15.123:0/2976159666 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fabac004d50 0x7fabc41f8c90 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7fabbc00ec10 tx=0x7fabbc00c640 comp rx=0 tx=0).stop 2024-09-16T15:43:45.558 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:45.552+0000 7fabc8edf640 1 -- 172.21.15.123:0/2976159666 shutdown_connections 2024-09-16T15:43:45.558 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:45.552+0000 7fabc8edf640 1 --2- 172.21.15.123:0/2976159666 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7faba4072ec0 0x7faba4075380 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:45.558 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:45.552+0000 7fabc8edf640 1 --2- 172.21.15.123:0/2976159666 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fabc40c5620 0x7fabc4156dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:45.558 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:45.552+0000 7fabc8edf640 1 --2- 172.21.15.123:0/2976159666 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fabc41f5920 0x7fabc40b0e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:45.558 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:45.552+0000 7fabc8edf640 1 --2- 172.21.15.123:0/2976159666 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fabac004d50 0x7fabc41f8c90 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:45.559 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:45.552+0000 7fabc8edf640 1 -- 172.21.15.123:0/2976159666 >> 172.21.15.123:0/2976159666 conn(0x7fabc40b1710 msgr2=0x7fabc40b4420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:43:45.559 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:45.552+0000 7fabc8edf640 1 -- 172.21.15.123:0/2976159666 shutdown_connections 2024-09-16T15:43:45.559 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:45.552+0000 7fabc8edf640 1 -- 172.21.15.123:0/2976159666 wait complete. 2024-09-16T15:43:45.577 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-09-16T15:43:45.577 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-16T15:43:45.704 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:45.700+0000 7f1e04fa4640 1 -- 172.21.15.123:0/3473187437 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1e000b2b00 msgr2=0x7f1e000d24b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:43:45.704 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:45.700+0000 7f1e04fa4640 1 --2- 172.21.15.123:0/3473187437 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1e000b2b00 0x7f1e000d24b0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f1e0008d220 tx=0x7f1df401c770 comp rx=0 tx=0).stop 2024-09-16T15:43:45.704 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:45.700+0000 7f1e04fa4640 1 -- 172.21.15.123:0/3473187437 shutdown_connections 2024-09-16T15:43:45.705 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:45.700+0000 7f1e04fa4640 1 --2- 172.21.15.123:0/3473187437 >> v2:172.21.15.184:3300/0 conn(0x7f1e000b0a50 0x7f1e000b0ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:45.705 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:45.700+0000 7f1e04fa4640 1 --2- 172.21.15.123:0/3473187437 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1e000b2b00 0x7f1e000d24b0 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:45.705 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:45.700+0000 7f1e04fa4640 1 -- 172.21.15.123:0/3473187437 >> 172.21.15.123:0/3473187437 conn(0x7f1e000d3ea0 msgr2=0x7f1e000d42b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:43:45.705 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:45.700+0000 7f1e04fa4640 1 -- 172.21.15.123:0/3473187437 shutdown_connections 2024-09-16T15:43:45.705 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:45.700+0000 7f1e04fa4640 1 -- 172.21.15.123:0/3473187437 wait complete. 2024-09-16T15:43:45.705 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:45.700+0000 7f1e04fa4640 1 Processor -- start 2024-09-16T15:43:45.705 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:45.700+0000 7f1e04fa4640 1 -- start start 2024-09-16T15:43:45.707 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:45.700+0000 7f1e04fa4640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f1e000b0a50 0x7f1e00151ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:43:45.707 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:45.700+0000 7f1e04fa4640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1e000b2b00 0x7f1e000b6cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:43:45.708 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:45.700+0000 7f1e04fa4640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1e00161620 0x7f1e001f7c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:43:45.708 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:45.700+0000 7f1e04fa4640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f1e0016b0f0 con 0x7f1e00161620 2024-09-16T15:43:45.708 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:45.700+0000 7f1e04fa4640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f1e0016af70 con 0x7f1e000b2b00 2024-09-16T15:43:45.708 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:45.700+0000 7f1e04fa4640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f1e0016b270 con 0x7f1e000b0a50 2024-09-16T15:43:45.708 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:45.700+0000 7f1dfed9d640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1e00161620 0x7f1e001f7c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:43:45.708 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:45.700+0000 7f1dfed9d640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1e00161620 0x7f1e001f7c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:41636/0 (socket says 172.21.15.123:41636) 2024-09-16T15:43:45.708 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:45.700+0000 7f1dfed9d640 1 -- 172.21.15.123:0/1209043235 learned_addr learned my addr 172.21.15.123:0/1209043235 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:43:45.708 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:45.700+0000 7f1dfdd9b640 1 --2- 172.21.15.123:0/1209043235 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1e000b2b00 0x7f1e000b6cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:43:45.708 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:45.700+0000 7f1dfe59c640 1 --2- 172.21.15.123:0/1209043235 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f1e000b0a50 0x7f1e00151ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:43:45.708 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:45.700+0000 7f1dfed9d640 1 -- 172.21.15.123:0/1209043235 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f1e000b0a50 msgr2=0x7f1e00151ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:43:45.708 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:45.700+0000 7f1dfed9d640 1 --2- 172.21.15.123:0/1209043235 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f1e000b0a50 0x7f1e00151ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:45.708 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:45.700+0000 7f1dfed9d640 1 -- 172.21.15.123:0/1209043235 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1e000b2b00 msgr2=0x7f1e000b6cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:43:45.709 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:45.700+0000 7f1dfed9d640 1 --2- 172.21.15.123:0/1209043235 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1e000b2b00 0x7f1e000b6cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:45.709 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:45.700+0000 7f1dfed9d640 1 -- 172.21.15.123:0/1209043235 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1e001f8670 con 0x7f1e00161620 2024-09-16T15:43:45.709 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:45.700+0000 7f1dfed9d640 1 --2- 172.21.15.123:0/1209043235 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1e00161620 0x7f1e001f7c20 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f1df800cce0 tx=0x7f1df800a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:43:45.709 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:45.704+0000 7f1deeffd640 1 -- 172.21.15.123:0/1209043235 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1df800f450 con 0x7f1e00161620 2024-09-16T15:43:45.709 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:45.704+0000 7f1deeffd640 1 -- 172.21.15.123:0/1209043235 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f1df800f700 con 0x7f1e00161620 2024-09-16T15:43:45.709 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:45.704+0000 7f1e04fa4640 1 -- 172.21.15.123:0/1209043235 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1e001f8950 con 0x7f1e00161620 2024-09-16T15:43:45.709 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:45.704+0000 7f1deeffd640 1 -- 172.21.15.123:0/1209043235 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1df800fb50 con 0x7f1e00161620 2024-09-16T15:43:45.709 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:45.704+0000 7f1e04fa4640 1 -- 172.21.15.123:0/1209043235 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1e00152930 con 0x7f1e00161620 2024-09-16T15:43:45.710 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:45.704+0000 7f1e04fa4640 1 -- 172.21.15.123:0/1209043235 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1e000b0ea0 con 0x7f1e00161620 2024-09-16T15:43:45.710 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:45.704+0000 7f1deeffd640 1 -- 172.21.15.123:0/1209043235 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 5) ==== 97080+0+0 (secure 0 0 0) 0x7f1df800fcf0 con 0x7f1e00161620 2024-09-16T15:43:45.710 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:45.704+0000 7f1deeffd640 1 --2- 172.21.15.123:0/1209043235 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f1de0072ef0 0x7f1de00753b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:43:45.710 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:45.704+0000 7f1deeffd640 1 -- 172.21.15.123:0/1209043235 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(33..33 src has 1..33) ==== 5709+0+0 (secure 0 0 0) 0x7f1df80949f0 con 0x7f1e00161620 2024-09-16T15:43:45.710 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:45.704+0000 7f1dfe59c640 1 --2- 172.21.15.123:0/1209043235 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f1de0072ef0 0x7f1de00753b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:43:45.710 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:45.704+0000 7f1dfe59c640 1 --2- 172.21.15.123:0/1209043235 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f1de0072ef0 0x7f1de00753b0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f1df000a750 tx=0x7f1df0008040 comp rx=0 tx=0).ready entity=mgr.4135 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:43:45.713 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:45.708+0000 7f1deeffd640 1 -- 172.21.15.123:0/1209043235 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f1df8014030 con 0x7f1e00161620 2024-09-16T15:43:45.923 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:45.920+0000 7f1e04fa4640 1 -- 172.21.15.123:0/1209043235 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f1e001f8b20 con 0x7f1e00161620 2024-09-16T15:43:45.924 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:45.920+0000 7f1deeffd640 1 -- 172.21.15.123:0/1209043235 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 18 v18) ==== 94+0+5229 (secure 0 0 0) 0x7f1df8066070 con 0x7f1e00161620 2024-09-16T15:43:45.924 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-09-16T15:43:45.924 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":18,"btime":"2024-09-16T15:43:45:540259+0000","default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4541,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6837/249053959","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6836","nonce":249053959},{"type":"v1","addr":"172.21.15.184:6837","nonce":249053959}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":13},{"gid":4705,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6835/1285950902","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":1285950902},{"type":"v1","addr":"172.21.15.184:6835","nonce":1285950902}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":4866,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6833/1030632114","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6832","nonce":1030632114},{"type":"v1","addr":"172.21.15.123:6833","nonce":1030632114}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11}],"filesystems":[{"mdsmap":{"epoch":18,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-16T15:43:41.851612+0000","modified":"2024-09-16T15:43:44.919014+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4869},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4869":{"gid":4869,"name":"a","rank":0,"incarnation":14,"state":"up:active","state_seq":5,"addr":"172.21.15.123:6835/822918970","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6834","nonce":822918970},{"type":"v1","addr":"172.21.15.123:6835","nonce":822918970}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4869,"qdb_cluster":[4869]},"id":2}]} 2024-09-16T15:43:45.925 INFO:teuthology.orchestra.run.smithi123.stderr:dumped fsmap epoch 18 2024-09-16T15:43:45.927 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:45.920+0000 7f1e04fa4640 1 -- 172.21.15.123:0/1209043235 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f1de0072ef0 msgr2=0x7f1de00753b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:43:45.927 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:45.920+0000 7f1e04fa4640 1 --2- 172.21.15.123:0/1209043235 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f1de0072ef0 0x7f1de00753b0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f1df000a750 tx=0x7f1df0008040 comp rx=0 tx=0).stop 2024-09-16T15:43:45.927 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:45.920+0000 7f1e04fa4640 1 -- 172.21.15.123:0/1209043235 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1e00161620 msgr2=0x7f1e001f7c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:43:45.927 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:45.920+0000 7f1e04fa4640 1 --2- 172.21.15.123:0/1209043235 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1e00161620 0x7f1e001f7c20 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f1df800cce0 tx=0x7f1df800a620 comp rx=0 tx=0).stop 2024-09-16T15:43:45.927 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:45.920+0000 7f1e04fa4640 1 -- 172.21.15.123:0/1209043235 shutdown_connections 2024-09-16T15:43:45.927 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:45.920+0000 7f1e04fa4640 1 --2- 172.21.15.123:0/1209043235 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f1de0072ef0 0x7f1de00753b0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:45.927 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:45.920+0000 7f1e04fa4640 1 --2- 172.21.15.123:0/1209043235 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1e00161620 0x7f1e001f7c20 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:45.927 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:45.920+0000 7f1e04fa4640 1 --2- 172.21.15.123:0/1209043235 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1e000b2b00 0x7f1e000b6cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:45.927 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:45.920+0000 7f1e04fa4640 1 --2- 172.21.15.123:0/1209043235 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f1e000b0a50 0x7f1e00151ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:45.928 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:45.920+0000 7f1e04fa4640 1 -- 172.21.15.123:0/1209043235 >> 172.21.15.123:0/1209043235 conn(0x7f1e000d3ea0 msgr2=0x7f1e00159d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:43:45.928 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:45.920+0000 7f1e04fa4640 1 -- 172.21.15.123:0/1209043235 shutdown_connections 2024-09-16T15:43:45.928 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:45.920+0000 7f1e04fa4640 1 -- 172.21.15.123:0/1209043235 wait complete. 2024-09-16T15:43:45.944 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-16T15:43:46.072 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:46.068+0000 7fe67a1a9640 1 -- 172.21.15.123:0/1071305383 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe674074600 msgr2=0x7fe67406e750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:43:46.072 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:46.068+0000 7fe67a1a9640 1 --2- 172.21.15.123:0/1071305383 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe674074600 0x7fe67406e750 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7fe664009840 tx=0x7fe66401c760 comp rx=0 tx=0).stop 2024-09-16T15:43:46.072 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:46.068+0000 7fe67a1a9640 1 -- 172.21.15.123:0/1071305383 shutdown_connections 2024-09-16T15:43:46.072 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:46.068+0000 7fe67a1a9640 1 --2- 172.21.15.123:0/1071305383 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe674074600 0x7fe67406e750 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:46.072 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:46.068+0000 7fe67a1a9640 1 --2- 172.21.15.123:0/1071305383 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fe6740726c0 0x7fe6740d0630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:46.072 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:46.068+0000 7fe67a1a9640 1 -- 172.21.15.123:0/1071305383 >> 172.21.15.123:0/1071305383 conn(0x7fe6740cfb10 msgr2=0x7fe6740cff20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:43:46.072 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:46.068+0000 7fe67a1a9640 1 -- 172.21.15.123:0/1071305383 shutdown_connections 2024-09-16T15:43:46.072 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:46.068+0000 7fe67a1a9640 1 -- 172.21.15.123:0/1071305383 wait complete. 2024-09-16T15:43:46.073 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:46.068+0000 7fe67a1a9640 1 Processor -- start 2024-09-16T15:43:46.073 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:46.068+0000 7fe67a1a9640 1 -- start start 2024-09-16T15:43:46.075 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:46.068+0000 7fe67a1a9640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe6740726c0 0x7fe6740c90a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:43:46.075 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:46.068+0000 7fe67a1a9640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fe674074600 0x7fe6740c7950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:43:46.075 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:46.068+0000 7fe67a1a9640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe67415bca0 0x7fe6741f89b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:43:46.075 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:46.068+0000 7fe67a1a9640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fe674168a50 con 0x7fe67415bca0 2024-09-16T15:43:46.075 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:46.068+0000 7fe67a1a9640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fe67406fa60 con 0x7fe6740726c0 2024-09-16T15:43:46.075 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:46.068+0000 7fe67a1a9640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fe674168bd0 con 0x7fe674074600 2024-09-16T15:43:46.075 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:46.068+0000 7fe673fff640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe67415bca0 0x7fe6741f89b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:43:46.075 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:46.068+0000 7fe673fff640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe67415bca0 0x7fe6741f89b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:41648/0 (socket says 172.21.15.123:41648) 2024-09-16T15:43:46.075 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:46.068+0000 7fe673fff640 1 -- 172.21.15.123:0/1245063411 learned_addr learned my addr 172.21.15.123:0/1245063411 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:43:46.075 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:46.068+0000 7fe6737fe640 1 --2- 172.21.15.123:0/1245063411 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe6740726c0 0x7fe6740c90a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:43:46.076 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:46.068+0000 7fe672ffd640 1 --2- 172.21.15.123:0/1245063411 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fe674074600 0x7fe6740c7950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:43:46.076 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:46.068+0000 7fe673fff640 1 -- 172.21.15.123:0/1245063411 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fe674074600 msgr2=0x7fe6740c7950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:43:46.076 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:46.068+0000 7fe673fff640 1 --2- 172.21.15.123:0/1245063411 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fe674074600 0x7fe6740c7950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:46.076 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:46.068+0000 7fe673fff640 1 -- 172.21.15.123:0/1245063411 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe6740726c0 msgr2=0x7fe6740c90a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:43:46.076 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:46.068+0000 7fe673fff640 1 --2- 172.21.15.123:0/1245063411 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe6740726c0 0x7fe6740c90a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:46.076 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:46.068+0000 7fe673fff640 1 -- 172.21.15.123:0/1245063411 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe6740c8660 con 0x7fe67415bca0 2024-09-16T15:43:46.076 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:46.068+0000 7fe6737fe640 1 --2- 172.21.15.123:0/1245063411 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe6740726c0 0x7fe6740c90a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T15:43:46.076 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:46.068+0000 7fe673fff640 1 --2- 172.21.15.123:0/1245063411 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe67415bca0 0x7fe6741f89b0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7fe66401a390 tx=0x7fe6640040d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:43:46.076 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:46.068+0000 7fe66bfff640 1 -- 172.21.15.123:0/1245063411 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6640bb070 con 0x7fe67415bca0 2024-09-16T15:43:46.076 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:46.068+0000 7fe66bfff640 1 -- 172.21.15.123:0/1245063411 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fe664005030 con 0x7fe67415bca0 2024-09-16T15:43:46.076 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:46.068+0000 7fe67a1a9640 1 -- 172.21.15.123:0/1245063411 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe6740c8380 con 0x7fe67415bca0 2024-09-16T15:43:46.076 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:46.068+0000 7fe66bfff640 1 -- 172.21.15.123:0/1245063411 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6640b2040 con 0x7fe67415bca0 2024-09-16T15:43:46.076 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:46.068+0000 7fe67a1a9640 1 -- 172.21.15.123:0/1245063411 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe67415f030 con 0x7fe67415bca0 2024-09-16T15:43:46.077 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:46.072+0000 7fe67a1a9640 1 -- 172.21.15.123:0/1245063411 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe6740d0630 con 0x7fe67415bca0 2024-09-16T15:43:46.077 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:46.072+0000 7fe66bfff640 1 -- 172.21.15.123:0/1245063411 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 5) ==== 97080+0+0 (secure 0 0 0) 0x7fe6640be070 con 0x7fe67415bca0 2024-09-16T15:43:46.077 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:46.072+0000 7fe66bfff640 1 --2- 172.21.15.123:0/1245063411 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fe654072ec0 0x7fe654075380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:43:46.078 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:46.072+0000 7fe66bfff640 1 -- 172.21.15.123:0/1245063411 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(33..33 src has 1..33) ==== 5709+0+0 (secure 0 0 0) 0x7fe66412d820 con 0x7fe67415bca0 2024-09-16T15:43:46.078 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:46.072+0000 7fe6737fe640 1 --2- 172.21.15.123:0/1245063411 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fe654072ec0 0x7fe654075380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:43:46.078 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:46.072+0000 7fe6737fe640 1 --2- 172.21.15.123:0/1245063411 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fe654072ec0 0x7fe654075380 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fe66c007fd0 tx=0x7fe66c009040 comp rx=0 tx=0).ready entity=mgr.4135 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:43:46.081 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:46.076+0000 7fe66bfff640 1 -- 172.21.15.123:0/1245063411 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7fe664016410 con 0x7fe67415bca0 2024-09-16T15:43:46.246 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:46.240+0000 7fe67a1a9640 1 -- 172.21.15.123:0/1245063411 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fe67415f660 con 0x7fe67415bca0 2024-09-16T15:43:46.248 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:46.244+0000 7fe66bfff640 1 -- 172.21.15.123:0/1245063411 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v33) ==== 74+0+16789 (secure 0 0 0) 0x7fe6640f8b00 con 0x7fe67415bca0 2024-09-16T15:43:46.248 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-09-16T15:43:46.248 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":33,"fsid":"56077f89-18c8-486f-ae7e-eecc0bb7822c","created":"2024-09-16T15:42:26.364892+0000","modified":"2024-09-16T15:43:43.471260+0000","last_up_change":"2024-09-16T15:42:38.707234+0000","last_in_change":"2024-09-16T15:42:29.423662+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-16T15:42:39.287764+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2024-09-16T15:43:39.775631+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"32","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data","create_time":"2024-09-16T15:43:40.809234+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"33","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9e7c1955-a607-4671-9be7-f2ef760ceeaa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6800","nonce":3882044969},{"type":"v1","addr":"172.21.15.123:6801","nonce":3882044969}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6802","nonce":3882044969},{"type":"v1","addr":"172.21.15.123:6803","nonce":3882044969}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6806","nonce":3882044969},{"type":"v1","addr":"172.21.15.123:6807","nonce":3882044969}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6804","nonce":3882044969},{"type":"v1","addr":"172.21.15.123:6805","nonce":3882044969}]},"public_addr":"172.21.15.123:6801/3882044969","cluster_addr":"172.21.15.123:6803/3882044969","heartbeat_back_addr":"172.21.15.123:6807/3882044969","heartbeat_front_addr":"172.21.15.123:6805/3882044969","state":["exists","up"]},{"osd":1,"uuid":"c540cfde-0977-4742-9b17-fd634668e010","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6816","nonce":4037001850},{"type":"v1","addr":"172.21.15.123:6817","nonce":4037001850}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6818","nonce":4037001850},{"type":"v1","addr":"172.21.15.123:6819","nonce":4037001850}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6822","nonce":4037001850},{"type":"v1","addr":"172.21.15.123:6823","nonce":4037001850}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6820","nonce":4037001850},{"type":"v1","addr":"172.21.15.123:6821","nonce":4037001850}]},"public_addr":"172.21.15.123:6817/4037001850","cluster_addr":"172.21.15.123:6819/4037001850","heartbeat_back_addr":"172.21.15.123:6823/4037001850","heartbeat_front_addr":"172.21.15.123:6821/4037001850","state":["exists","up"]},{"osd":2,"uuid":"9864a496-f169-486b-95c4-1d59162793a7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6824","nonce":1332143088},{"type":"v1","addr":"172.21.15.123:6825","nonce":1332143088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6826","nonce":1332143088},{"type":"v1","addr":"172.21.15.123:6827","nonce":1332143088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6830","nonce":1332143088},{"type":"v1","addr":"172.21.15.123:6831","nonce":1332143088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6828","nonce":1332143088},{"type":"v1","addr":"172.21.15.123:6829","nonce":1332143088}]},"public_addr":"172.21.15.123:6825/1332143088","cluster_addr":"172.21.15.123:6827/1332143088","heartbeat_back_addr":"172.21.15.123:6831/1332143088","heartbeat_front_addr":"172.21.15.123:6829/1332143088","state":["exists","up"]},{"osd":3,"uuid":"c6d1f817-f074-4325-95e1-578eba13edda","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6808","nonce":988586998},{"type":"v1","addr":"172.21.15.123:6809","nonce":988586998}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6810","nonce":988586998},{"type":"v1","addr":"172.21.15.123:6811","nonce":988586998}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6814","nonce":988586998},{"type":"v1","addr":"172.21.15.123:6815","nonce":988586998}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6812","nonce":988586998},{"type":"v1","addr":"172.21.15.123:6813","nonce":988586998}]},"public_addr":"172.21.15.123:6809/988586998","cluster_addr":"172.21.15.123:6811/988586998","heartbeat_back_addr":"172.21.15.123:6815/988586998","heartbeat_front_addr":"172.21.15.123:6813/988586998","state":["exists","up"]},{"osd":4,"uuid":"200a8df4-50ef-40a5-b48a-58981269c086","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6816","nonce":3387813890},{"type":"v1","addr":"172.21.15.184:6817","nonce":3387813890}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6818","nonce":3387813890},{"type":"v1","addr":"172.21.15.184:6819","nonce":3387813890}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6822","nonce":3387813890},{"type":"v1","addr":"172.21.15.184:6823","nonce":3387813890}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6820","nonce":3387813890},{"type":"v1","addr":"172.21.15.184:6821","nonce":3387813890}]},"public_addr":"172.21.15.184:6817/3387813890","cluster_addr":"172.21.15.184:6819/3387813890","heartbeat_back_addr":"172.21.15.184:6823/3387813890","heartbeat_front_addr":"172.21.15.184:6821/3387813890","state":["exists","up"]},{"osd":5,"uuid":"32d5a900-da3d-412c-bc19-13861a2368d8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6808","nonce":2680916017},{"type":"v1","addr":"172.21.15.184:6809","nonce":2680916017}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6810","nonce":2680916017},{"type":"v1","addr":"172.21.15.184:6811","nonce":2680916017}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6814","nonce":2680916017},{"type":"v1","addr":"172.21.15.184:6815","nonce":2680916017}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6812","nonce":2680916017},{"type":"v1","addr":"172.21.15.184:6813","nonce":2680916017}]},"public_addr":"172.21.15.184:6809/2680916017","cluster_addr":"172.21.15.184:6811/2680916017","heartbeat_back_addr":"172.21.15.184:6815/2680916017","heartbeat_front_addr":"172.21.15.184:6813/2680916017","state":["exists","up"]},{"osd":6,"uuid":"cde6f39e-aa1b-4260-ac19-774ee6aaca72","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6824","nonce":2183120457},{"type":"v1","addr":"172.21.15.184:6825","nonce":2183120457}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6826","nonce":2183120457},{"type":"v1","addr":"172.21.15.184:6827","nonce":2183120457}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6830","nonce":2183120457},{"type":"v1","addr":"172.21.15.184:6831","nonce":2183120457}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6828","nonce":2183120457},{"type":"v1","addr":"172.21.15.184:6829","nonce":2183120457}]},"public_addr":"172.21.15.184:6825/2183120457","cluster_addr":"172.21.15.184:6827/2183120457","heartbeat_back_addr":"172.21.15.184:6831/2183120457","heartbeat_front_addr":"172.21.15.184:6829/2183120457","state":["exists","up"]},{"osd":7,"uuid":"4f381778-eee1-47c6-804f-df0e6b33e313","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6800","nonce":4102957285},{"type":"v1","addr":"172.21.15.184:6801","nonce":4102957285}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6802","nonce":4102957285},{"type":"v1","addr":"172.21.15.184:6803","nonce":4102957285}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6806","nonce":4102957285},{"type":"v1","addr":"172.21.15.184:6807","nonce":4102957285}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6804","nonce":4102957285},{"type":"v1","addr":"172.21.15.184:6805","nonce":4102957285}]},"public_addr":"172.21.15.184:6801/4102957285","cluster_addr":"172.21.15.184:6803/4102957285","heartbeat_back_addr":"172.21.15.184:6807/4102957285","heartbeat_front_addr":"172.21.15.184:6805/4102957285","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:35.972210+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.222007+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.198617+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.385663+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.358935+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.156066+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.752973+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:35.682846+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T15:43:46.251 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:46.244+0000 7fe67a1a9640 1 -- 172.21.15.123:0/1245063411 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fe654072ec0 msgr2=0x7fe654075380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:43:46.251 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:46.244+0000 7fe67a1a9640 1 --2- 172.21.15.123:0/1245063411 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fe654072ec0 0x7fe654075380 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fe66c007fd0 tx=0x7fe66c009040 comp rx=0 tx=0).stop 2024-09-16T15:43:46.251 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:46.244+0000 7fe67a1a9640 1 -- 172.21.15.123:0/1245063411 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe67415bca0 msgr2=0x7fe6741f89b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:43:46.251 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:46.244+0000 7fe67a1a9640 1 --2- 172.21.15.123:0/1245063411 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe67415bca0 0x7fe6741f89b0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7fe66401a390 tx=0x7fe6640040d0 comp rx=0 tx=0).stop 2024-09-16T15:43:46.251 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:46.244+0000 7fe67a1a9640 1 -- 172.21.15.123:0/1245063411 shutdown_connections 2024-09-16T15:43:46.251 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:46.244+0000 7fe67a1a9640 1 --2- 172.21.15.123:0/1245063411 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fe654072ec0 0x7fe654075380 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:46.251 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:46.244+0000 7fe67a1a9640 1 --2- 172.21.15.123:0/1245063411 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe67415bca0 0x7fe6741f89b0 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:46.251 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:46.244+0000 7fe67a1a9640 1 --2- 172.21.15.123:0/1245063411 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fe674074600 0x7fe6740c7950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:46.251 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:46.244+0000 7fe67a1a9640 1 --2- 172.21.15.123:0/1245063411 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe6740726c0 0x7fe6740c90a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:46.251 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:46.244+0000 7fe67a1a9640 1 -- 172.21.15.123:0/1245063411 >> 172.21.15.123:0/1245063411 conn(0x7fe6740cfb10 msgr2=0x7fe6740b3d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:43:46.251 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:46.244+0000 7fe67a1a9640 1 -- 172.21.15.123:0/1245063411 shutdown_connections 2024-09-16T15:43:46.251 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:46.244+0000 7fe67a1a9640 1 -- 172.21.15.123:0/1245063411 wait complete. 2024-09-16T15:43:46.266 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-09-16T15:43:46.266 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T15:43:46.391 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:46.388+0000 7fd32cbcb640 1 -- 172.21.15.123:0/2963476272 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd32815bca0 msgr2=0x7fd3280d0630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:43:46.391 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:46.388+0000 7fd32cbcb640 1 --2- 172.21.15.123:0/2963476272 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd32815bca0 0x7fd3280d0630 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7fd318009900 tx=0x7fd31801c820 comp rx=0 tx=0).stop 2024-09-16T15:43:46.391 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:46.388+0000 7fd32cbcb640 1 -- 172.21.15.123:0/2963476272 shutdown_connections 2024-09-16T15:43:46.392 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:46.388+0000 7fd32cbcb640 1 --2- 172.21.15.123:0/2963476272 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fd328074600 0x7fd32806e750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:46.392 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:46.388+0000 7fd32cbcb640 1 --2- 172.21.15.123:0/2963476272 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd32815bca0 0x7fd3280d0630 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:46.392 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:46.388+0000 7fd32cbcb640 1 -- 172.21.15.123:0/2963476272 >> 172.21.15.123:0/2963476272 conn(0x7fd3280cfb10 msgr2=0x7fd3280cff20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:43:46.392 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:46.388+0000 7fd32cbcb640 1 -- 172.21.15.123:0/2963476272 shutdown_connections 2024-09-16T15:43:46.392 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:46.388+0000 7fd32cbcb640 1 -- 172.21.15.123:0/2963476272 wait complete. 2024-09-16T15:43:46.392 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:46.388+0000 7fd32cbcb640 1 Processor -- start 2024-09-16T15:43:46.392 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:46.388+0000 7fd32cbcb640 1 -- start start 2024-09-16T15:43:46.394 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:46.388+0000 7fd32cbcb640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd3280726c0 0x7fd3281541b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:43:46.394 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:46.388+0000 7fd32cbcb640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fd328074600 0x7fd32814f120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:43:46.394 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:46.388+0000 7fd32cbcb640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd32815bca0 0x7fd3281faf30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:43:46.395 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:46.388+0000 7fd32cbcb640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fd328168bd0 con 0x7fd3280726c0 2024-09-16T15:43:46.395 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:46.388+0000 7fd32cbcb640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fd328168a50 con 0x7fd32815bca0 2024-09-16T15:43:46.395 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:46.388+0000 7fd32cbcb640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fd328168d50 con 0x7fd328074600 2024-09-16T15:43:46.395 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:46.388+0000 7fd32659c640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd3280726c0 0x7fd3281541b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:43:46.395 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:46.388+0000 7fd32659c640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd3280726c0 0x7fd3281541b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:41678/0 (socket says 172.21.15.123:41678) 2024-09-16T15:43:46.395 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:46.388+0000 7fd32659c640 1 -- 172.21.15.123:0/2646152493 learned_addr learned my addr 172.21.15.123:0/2646152493 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:43:46.395 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:46.388+0000 7fd325d9b640 1 --2- 172.21.15.123:0/2646152493 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fd328074600 0x7fd32814f120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:43:46.395 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:46.388+0000 7fd32659c640 1 -- 172.21.15.123:0/2646152493 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fd328074600 msgr2=0x7fd32814f120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:43:46.395 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:46.388+0000 7fd32659c640 1 --2- 172.21.15.123:0/2646152493 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fd328074600 0x7fd32814f120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:46.395 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:46.388+0000 7fd32659c640 1 -- 172.21.15.123:0/2646152493 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd32815bca0 msgr2=0x7fd3281faf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:43:46.395 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:46.388+0000 7fd32659c640 1 --2- 172.21.15.123:0/2646152493 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd32815bca0 0x7fd3281faf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:46.395 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:46.388+0000 7fd32659c640 1 -- 172.21.15.123:0/2646152493 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd3281fb610 con 0x7fd3280726c0 2024-09-16T15:43:46.395 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:46.388+0000 7fd325d9b640 1 --2- 172.21.15.123:0/2646152493 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fd328074600 0x7fd32814f120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T15:43:46.396 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:46.388+0000 7fd32659c640 1 --2- 172.21.15.123:0/2646152493 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd3280726c0 0x7fd3281541b0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7fd318009850 tx=0x7fd3180a5680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:43:46.396 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:46.388+0000 7fd316ffd640 1 -- 172.21.15.123:0/2646152493 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3180b2040 con 0x7fd3280726c0 2024-09-16T15:43:46.396 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:46.388+0000 7fd316ffd640 1 -- 172.21.15.123:0/2646152493 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fd318004430 con 0x7fd3280726c0 2024-09-16T15:43:46.396 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:46.388+0000 7fd32cbcb640 1 -- 172.21.15.123:0/2646152493 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd3281fb8f0 con 0x7fd3280726c0 2024-09-16T15:43:46.396 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:46.388+0000 7fd316ffd640 1 -- 172.21.15.123:0/2646152493 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd318003860 con 0x7fd3280726c0 2024-09-16T15:43:46.396 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:46.388+0000 7fd32cbcb640 1 -- 172.21.15.123:0/2646152493 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd3280c4720 con 0x7fd3280726c0 2024-09-16T15:43:46.397 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:46.392+0000 7fd32cbcb640 1 -- 172.21.15.123:0/2646152493 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd32806e750 con 0x7fd3280726c0 2024-09-16T15:43:46.397 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:46.392+0000 7fd316ffd640 1 -- 172.21.15.123:0/2646152493 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 5) ==== 97080+0+0 (secure 0 0 0) 0x7fd318003a00 con 0x7fd3280726c0 2024-09-16T15:43:46.397 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:46.392+0000 7fd316ffd640 1 --2- 172.21.15.123:0/2646152493 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fd308072ec0 0x7fd308075380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:43:46.397 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:46.392+0000 7fd316ffd640 1 -- 172.21.15.123:0/2646152493 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(33..33 src has 1..33) ==== 5709+0+0 (secure 0 0 0) 0x7fd31812dac0 con 0x7fd3280726c0 2024-09-16T15:43:46.397 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:46.392+0000 7fd325d9b640 1 --2- 172.21.15.123:0/2646152493 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fd308072ec0 0x7fd308075380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:43:46.397 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:46.392+0000 7fd325d9b640 1 --2- 172.21.15.123:0/2646152493 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fd308072ec0 0x7fd308075380 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fd328166710 tx=0x7fd31c009040 comp rx=0 tx=0).ready entity=mgr.4135 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:43:46.400 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:46.396+0000 7fd316ffd640 1 -- 172.21.15.123:0/2646152493 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7fd3180bd060 con 0x7fd3280726c0 2024-09-16T15:43:46.564 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:46.560+0000 7fd32cbcb640 1 -- 172.21.15.123:0/2646152493 --> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fd32815f630 con 0x7fd308072ec0 2024-09-16T15:43:46.578 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:46.572+0000 7fd316ffd640 1 -- 172.21.15.123:0/2646152493 <== mgr.4135 v2:172.21.15.184:6832/3114346423 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+251957 (secure 0 0 0) 0x7fd31c055850 con 0x7fd308072ec0 2024-09-16T15:43:46.580 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-09-16T15:43:46.582 INFO:teuthology.orchestra.run.smithi123.stderr:dumped all 2024-09-16T15:43:46.583 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:46.576+0000 7fd32cbcb640 1 -- 172.21.15.123:0/2646152493 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fd308072ec0 msgr2=0x7fd308075380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:43:46.583 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:46.576+0000 7fd32cbcb640 1 --2- 172.21.15.123:0/2646152493 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fd308072ec0 0x7fd308075380 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fd328166710 tx=0x7fd31c009040 comp rx=0 tx=0).stop 2024-09-16T15:43:46.583 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:46.576+0000 7fd32cbcb640 1 -- 172.21.15.123:0/2646152493 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd3280726c0 msgr2=0x7fd3281541b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:43:46.583 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:46.576+0000 7fd32cbcb640 1 --2- 172.21.15.123:0/2646152493 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd3280726c0 0x7fd3281541b0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7fd318009850 tx=0x7fd3180a5680 comp rx=0 tx=0).stop 2024-09-16T15:43:46.583 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:46.576+0000 7fd32cbcb640 1 -- 172.21.15.123:0/2646152493 shutdown_connections 2024-09-16T15:43:46.583 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:46.576+0000 7fd32cbcb640 1 --2- 172.21.15.123:0/2646152493 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fd308072ec0 0x7fd308075380 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:46.583 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:46.576+0000 7fd32cbcb640 1 --2- 172.21.15.123:0/2646152493 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd32815bca0 0x7fd3281faf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:46.583 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:46.576+0000 7fd32cbcb640 1 --2- 172.21.15.123:0/2646152493 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fd328074600 0x7fd32814f120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:46.583 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:46.576+0000 7fd32cbcb640 1 --2- 172.21.15.123:0/2646152493 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd3280726c0 0x7fd3281541b0 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:46.583 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:46.576+0000 7fd32cbcb640 1 -- 172.21.15.123:0/2646152493 >> 172.21.15.123:0/2646152493 conn(0x7fd3280cfb10 msgr2=0x7fd3280758d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:43:46.583 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:46.576+0000 7fd32cbcb640 1 -- 172.21.15.123:0/2646152493 shutdown_connections 2024-09-16T15:43:46.583 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:46.576+0000 7fd32cbcb640 1 -- 172.21.15.123:0/2646152493 wait complete. 2024-09-16T15:43:46.597 INFO:teuthology.orchestra.run.smithi123.stdout:{"pg_ready":true,"pg_map":{"version":59,"stamp":"2024-09-16T15:43:45.262906+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":145,"num_write_kb":1642,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":110,"ondisk_log_size":110,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":138,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":220320,"kb_used_data":4080,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754754400,"statfs":{"total":773094113280,"available":772868505600,"internally_reserved":0,"allocated":4177920,"data_stored":2313724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,4],"upper_bound":32},"perf_stat":{"commit_latency_ms":45,"apply_latency_ms":45,"commit_latency_ns":45000000,"apply_latency_ns":45000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.802492"},"pg_stats":[{"pgid":"5.3f","version":"0'0","reported_seq":13,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-16T15:43:43.734474+0000","last_change":"2024-09-16T15:43:42.524818+0000","last_active":"2024-09-16T15:43:43.734474+0000","last_peered":"2024-09-16T15:43:43.734474+0000","last_clean":"2024-09-16T15:43:43.734474+0000","last_became_active":"2024-09-16T15:43:42.523640+0000","last_became_peered":"2024-09-16T15:43:42.523640+0000","last_unstale":"2024-09-16T15:43:43.734474+0000","last_undegraded":"2024-09-16T15:43:43.734474+0000","last_fullsized":"2024-09-16T15:43:43.734474+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:43:41.457712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:43:41.457712+0000","last_clean_scrub_stamp":"2024-09-16T15:43:41.457712+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T02:51:58.757703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.3e","version":"0'0","reported_seq":13,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-16T15:43:43.484595+0000","last_change":"2024-09-16T15:43:42.521224+0000","last_active":"2024-09-16T15:43:43.484595+0000","last_peered":"2024-09-16T15:43:43.484595+0000","last_clean":"2024-09-16T15:43:43.484595+0000","last_became_active":"2024-09-16T15:43:42.513486+0000","last_became_peered":"2024-09-16T15:43:42.513486+0000","last_unstale":"2024-09-16T15:43:43.484595+0000","last_undegraded":"2024-09-16T15:43:43.484595+0000","last_fullsized":"2024-09-16T15:43:43.484595+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:43:41.457712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:43:41.457712+0000","last_clean_scrub_stamp":"2024-09-16T15:43:41.457712+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:59:10.901283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.3d","version":"0'0","reported_seq":13,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-16T15:43:43.487543+0000","last_change":"2024-09-16T15:43:42.527094+0000","last_active":"2024-09-16T15:43:43.487543+0000","last_peered":"2024-09-16T15:43:43.487543+0000","last_clean":"2024-09-16T15:43:43.487543+0000","last_became_active":"2024-09-16T15:43:42.522632+0000","last_became_peered":"2024-09-16T15:43:42.522632+0000","last_unstale":"2024-09-16T15:43:43.487543+0000","last_undegraded":"2024-09-16T15:43:43.487543+0000","last_fullsized":"2024-09-16T15:43:43.487543+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:43:41.457712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:43:41.457712+0000","last_clean_scrub_stamp":"2024-09-16T15:43:41.457712+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T23:14:58.404320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3c","version":"0'0","reported_seq":13,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-16T15:43:43.713868+0000","last_change":"2024-09-16T15:43:42.515323+0000","last_active":"2024-09-16T15:43:43.713868+0000","last_peered":"2024-09-16T15:43:43.713868+0000","last_clean":"2024-09-16T15:43:43.713868+0000","last_became_active":"2024-09-16T15:43:42.513871+0000","last_became_peered":"2024-09-16T15:43:42.513871+0000","last_unstale":"2024-09-16T15:43:43.713868+0000","last_undegraded":"2024-09-16T15:43:43.713868+0000","last_fullsized":"2024-09-16T15:43:43.713868+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:43:41.457712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:43:41.457712+0000","last_clean_scrub_stamp":"2024-09-16T15:43:41.457712+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T03:42:47.543239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3b","version":"0'0","reported_seq":13,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-16T15:43:43.732207+0000","last_change":"2024-09-16T15:43:42.522044+0000","last_active":"2024-09-16T15:43:43.732207+0000","last_peered":"2024-09-16T15:43:43.732207+0000","last_clean":"2024-09-16T15:43:43.732207+0000","last_became_active":"2024-09-16T15:43:42.521770+0000","last_became_peered":"2024-09-16T15:43:42.521770+0000","last_unstale":"2024-09-16T15:43:43.732207+0000","last_undegraded":"2024-09-16T15:43:43.732207+0000","last_fullsized":"2024-09-16T15:43:43.732207+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:43:41.457712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:43:41.457712+0000","last_clean_scrub_stamp":"2024-09-16T15:43:41.457712+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T18:45:40.961435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3a","version":"0'0","reported_seq":13,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-16T15:43:43.483501+0000","last_change":"2024-09-16T15:43:42.538688+0000","last_active":"2024-09-16T15:43:43.483501+0000","last_peered":"2024-09-16T15:43:43.483501+0000","last_clean":"2024-09-16T15:43:43.483501+0000","last_became_active":"2024-09-16T15:43:42.538389+0000","last_became_peered":"2024-09-16T15:43:42.538389+0000","last_unstale":"2024-09-16T15:43:43.483501+0000","last_undegraded":"2024-09-16T15:43:43.483501+0000","last_fullsized":"2024-09-16T15:43:43.483501+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:43:41.457712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:43:41.457712+0000","last_clean_scrub_stamp":"2024-09-16T15:43:41.457712+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T23:05:07.986662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.39","version":"0'0","reported_seq":13,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-16T15:43:43.488389+0000","last_change":"2024-09-16T15:43:42.508618+0000","last_active":"2024-09-16T15:43:43.488389+0000","last_peered":"2024-09-16T15:43:43.488389+0000","last_clean":"2024-09-16T15:43:43.488389+0000","last_became_active":"2024-09-16T15:43:42.507868+0000","last_became_peered":"2024-09-16T15:43:42.507868+0000","last_unstale":"2024-09-16T15:43:43.488389+0000","last_undegraded":"2024-09-16T15:43:43.488389+0000","last_fullsized":"2024-09-16T15:43:43.488389+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:43:41.457712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:43:41.457712+0000","last_clean_scrub_stamp":"2024-09-16T15:43:41.457712+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T23:14:58.404320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.38","version":"0'0","reported_seq":13,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-16T15:43:43.485025+0000","last_change":"2024-09-16T15:43:42.500138+0000","last_active":"2024-09-16T15:43:43.485025+0000","last_peered":"2024-09-16T15:43:43.485025+0000","last_clean":"2024-09-16T15:43:43.485025+0000","last_became_active":"2024-09-16T15:43:42.497303+0000","last_became_peered":"2024-09-16T15:43:42.497303+0000","last_unstale":"2024-09-16T15:43:43.485025+0000","last_undegraded":"2024-09-16T15:43:43.485025+0000","last_fullsized":"2024-09-16T15:43:43.485025+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:43:41.457712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:43:41.457712+0000","last_clean_scrub_stamp":"2024-09-16T15:43:41.457712+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:23:11.473976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.37","version":"0'0","reported_seq":13,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-16T15:43:43.734078+0000","last_change":"2024-09-16T15:43:42.505344+0000","last_active":"2024-09-16T15:43:43.734078+0000","last_peered":"2024-09-16T15:43:43.734078+0000","last_clean":"2024-09-16T15:43:43.734078+0000","last_became_active":"2024-09-16T15:43:42.504620+0000","last_became_peered":"2024-09-16T15:43:42.504620+0000","last_unstale":"2024-09-16T15:43:43.734078+0000","last_undegraded":"2024-09-16T15:43:43.734078+0000","last_fullsized":"2024-09-16T15:43:43.734078+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:43:41.457712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:43:41.457712+0000","last_clean_scrub_stamp":"2024-09-16T15:43:41.457712+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:32:27.596625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.36","version":"0'0","reported_seq":13,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-16T15:43:43.714211+0000","last_change":"2024-09-16T15:43:42.527117+0000","last_active":"2024-09-16T15:43:43.714211+0000","last_peered":"2024-09-16T15:43:43.714211+0000","last_clean":"2024-09-16T15:43:43.714211+0000","last_became_active":"2024-09-16T15:43:42.526834+0000","last_became_peered":"2024-09-16T15:43:42.526834+0000","last_unstale":"2024-09-16T15:43:43.714211+0000","last_undegraded":"2024-09-16T15:43:43.714211+0000","last_fullsized":"2024-09-16T15:43:43.714211+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:43:41.457712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:43:41.457712+0000","last_clean_scrub_stamp":"2024-09-16T15:43:41.457712+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T22:20:03.787407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.35","version":"0'0","reported_seq":13,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-16T15:43:43.488081+0000","last_change":"2024-09-16T15:43:42.501721+0000","last_active":"2024-09-16T15:43:43.488081+0000","last_peered":"2024-09-16T15:43:43.488081+0000","last_clean":"2024-09-16T15:43:43.488081+0000","last_became_active":"2024-09-16T15:43:42.500952+0000","last_became_peered":"2024-09-16T15:43:42.500952+0000","last_unstale":"2024-09-16T15:43:43.488081+0000","last_undegraded":"2024-09-16T15:43:43.488081+0000","last_fullsized":"2024-09-16T15:43:43.488081+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:43:41.457712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:43:41.457712+0000","last_clean_scrub_stamp":"2024-09-16T15:43:41.457712+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:53:25.198570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.34","version":"0'0","reported_seq":13,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-16T15:43:43.486317+0000","last_change":"2024-09-16T15:43:42.532417+0000","last_active":"2024-09-16T15:43:43.486317+0000","last_peered":"2024-09-16T15:43:43.486317+0000","last_clean":"2024-09-16T15:43:43.486317+0000","last_became_active":"2024-09-16T15:43:42.530613+0000","last_became_peered":"2024-09-16T15:43:42.530613+0000","last_unstale":"2024-09-16T15:43:43.486317+0000","last_undegraded":"2024-09-16T15:43:43.486317+0000","last_fullsized":"2024-09-16T15:43:43.486317+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:43:41.457712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:43:41.457712+0000","last_clean_scrub_stamp":"2024-09-16T15:43:41.457712+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T02:35:21.217508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.33","version":"0'0","reported_seq":13,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-16T15:43:43.732236+0000","last_change":"2024-09-16T15:43:42.501898+0000","last_active":"2024-09-16T15:43:43.732236+0000","last_peered":"2024-09-16T15:43:43.732236+0000","last_clean":"2024-09-16T15:43:43.732236+0000","last_became_active":"2024-09-16T15:43:42.501174+0000","last_became_peered":"2024-09-16T15:43:42.501174+0000","last_unstale":"2024-09-16T15:43:43.732236+0000","last_undegraded":"2024-09-16T15:43:43.732236+0000","last_fullsized":"2024-09-16T15:43:43.732236+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:43:41.457712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:43:41.457712+0000","last_clean_scrub_stamp":"2024-09-16T15:43:41.457712+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T02:08:26.700938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.32","version":"0'0","reported_seq":13,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-16T15:43:43.484664+0000","last_change":"2024-09-16T15:43:42.511755+0000","last_active":"2024-09-16T15:43:43.484664+0000","last_peered":"2024-09-16T15:43:43.484664+0000","last_clean":"2024-09-16T15:43:43.484664+0000","last_became_active":"2024-09-16T15:43:42.510868+0000","last_became_peered":"2024-09-16T15:43:42.510868+0000","last_unstale":"2024-09-16T15:43:43.484664+0000","last_undegraded":"2024-09-16T15:43:43.484664+0000","last_fullsized":"2024-09-16T15:43:43.484664+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:43:41.457712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:43:41.457712+0000","last_clean_scrub_stamp":"2024-09-16T15:43:41.457712+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T22:59:08.176784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.31","version":"0'0","reported_seq":13,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-16T15:43:43.484562+0000","last_change":"2024-09-16T15:43:42.522783+0000","last_active":"2024-09-16T15:43:43.484562+0000","last_peered":"2024-09-16T15:43:43.484562+0000","last_clean":"2024-09-16T15:43:43.484562+0000","last_became_active":"2024-09-16T15:43:42.519423+0000","last_became_peered":"2024-09-16T15:43:42.519423+0000","last_unstale":"2024-09-16T15:43:43.484562+0000","last_undegraded":"2024-09-16T15:43:43.484562+0000","last_fullsized":"2024-09-16T15:43:43.484562+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:43:41.457712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:43:41.457712+0000","last_clean_scrub_stamp":"2024-09-16T15:43:41.457712+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:53:05.432956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.30","version":"0'0","reported_seq":13,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-16T15:43:43.486421+0000","last_change":"2024-09-16T15:43:42.525901+0000","last_active":"2024-09-16T15:43:43.486421+0000","last_peered":"2024-09-16T15:43:43.486421+0000","last_clean":"2024-09-16T15:43:43.486421+0000","last_became_active":"2024-09-16T15:43:42.525168+0000","last_became_peered":"2024-09-16T15:43:42.525168+0000","last_unstale":"2024-09-16T15:43:43.486421+0000","last_undegraded":"2024-09-16T15:43:43.486421+0000","last_fullsized":"2024-09-16T15:43:43.486421+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:43:41.457712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:43:41.457712+0000","last_clean_scrub_stamp":"2024-09-16T15:43:41.457712+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T02:35:21.217508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2f","version":"0'0","reported_seq":13,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-16T15:43:43.488346+0000","last_change":"2024-09-16T15:43:42.508558+0000","last_active":"2024-09-16T15:43:43.488346+0000","last_peered":"2024-09-16T15:43:43.488346+0000","last_clean":"2024-09-16T15:43:43.488346+0000","last_became_active":"2024-09-16T15:43:42.507653+0000","last_became_peered":"2024-09-16T15:43:42.507653+0000","last_unstale":"2024-09-16T15:43:43.488346+0000","last_undegraded":"2024-09-16T15:43:43.488346+0000","last_fullsized":"2024-09-16T15:43:43.488346+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:43:41.457712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:43:41.457712+0000","last_clean_scrub_stamp":"2024-09-16T15:43:41.457712+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T03:25:30.813074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2e","version":"0'0","reported_seq":13,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-16T15:43:43.484810+0000","last_change":"2024-09-16T15:43:42.521592+0000","last_active":"2024-09-16T15:43:43.484810+0000","last_peered":"2024-09-16T15:43:43.484810+0000","last_clean":"2024-09-16T15:43:43.484810+0000","last_became_active":"2024-09-16T15:43:42.512173+0000","last_became_peered":"2024-09-16T15:43:42.512173+0000","last_unstale":"2024-09-16T15:43:43.484810+0000","last_undegraded":"2024-09-16T15:43:43.484810+0000","last_fullsized":"2024-09-16T15:43:43.484810+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:43:41.457712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:43:41.457712+0000","last_clean_scrub_stamp":"2024-09-16T15:43:41.457712+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:02:55.957888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2d","version":"0'0","reported_seq":13,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-16T15:43:43.484494+0000","last_change":"2024-09-16T15:43:42.521526+0000","last_active":"2024-09-16T15:43:43.484494+0000","last_peered":"2024-09-16T15:43:43.484494+0000","last_clean":"2024-09-16T15:43:43.484494+0000","last_became_active":"2024-09-16T15:43:42.511688+0000","last_became_peered":"2024-09-16T15:43:42.511688+0000","last_unstale":"2024-09-16T15:43:43.484494+0000","last_undegraded":"2024-09-16T15:43:43.484494+0000","last_fullsized":"2024-09-16T15:43:43.484494+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:43:41.457712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:43:41.457712+0000","last_clean_scrub_stamp":"2024-09-16T15:43:41.457712+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T23:53:41.204959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2c","version":"0'0","reported_seq":13,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-16T15:43:43.732580+0000","last_change":"2024-09-16T15:43:42.524820+0000","last_active":"2024-09-16T15:43:43.732580+0000","last_peered":"2024-09-16T15:43:43.732580+0000","last_clean":"2024-09-16T15:43:43.732580+0000","last_became_active":"2024-09-16T15:43:42.524553+0000","last_became_peered":"2024-09-16T15:43:42.524553+0000","last_unstale":"2024-09-16T15:43:43.732580+0000","last_undegraded":"2024-09-16T15:43:43.732580+0000","last_fullsized":"2024-09-16T15:43:43.732580+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:43:41.457712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:43:41.457712+0000","last_clean_scrub_stamp":"2024-09-16T15:43:41.457712+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T01:59:56.657959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.2b","version":"0'0","reported_seq":13,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-16T15:43:43.734152+0000","last_change":"2024-09-16T15:43:42.515549+0000","last_active":"2024-09-16T15:43:43.734152+0000","last_peered":"2024-09-16T15:43:43.734152+0000","last_clean":"2024-09-16T15:43:43.734152+0000","last_became_active":"2024-09-16T15:43:42.514728+0000","last_became_peered":"2024-09-16T15:43:42.514728+0000","last_unstale":"2024-09-16T15:43:43.734152+0000","last_undegraded":"2024-09-16T15:43:43.734152+0000","last_fullsized":"2024-09-16T15:43:43.734152+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:43:41.457712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:43:41.457712+0000","last_clean_scrub_stamp":"2024-09-16T15:43:41.457712+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T18:25:52.364522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.2a","version":"0'0","reported_seq":13,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-16T15:43:43.733479+0000","last_change":"2024-09-16T15:43:42.508495+0000","last_active":"2024-09-16T15:43:43.733479+0000","last_peered":"2024-09-16T15:43:43.733479+0000","last_clean":"2024-09-16T15:43:43.733479+0000","last_became_active":"2024-09-16T15:43:42.507783+0000","last_became_peered":"2024-09-16T15:43:42.507783+0000","last_unstale":"2024-09-16T15:43:43.733479+0000","last_undegraded":"2024-09-16T15:43:43.733479+0000","last_fullsized":"2024-09-16T15:43:43.733479+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:43:41.457712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:43:41.457712+0000","last_clean_scrub_stamp":"2024-09-16T15:43:41.457712+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:58:40.375405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.29","version":"0'0","reported_seq":13,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-16T15:43:43.615739+0000","last_change":"2024-09-16T15:43:42.498261+0000","last_active":"2024-09-16T15:43:43.615739+0000","last_peered":"2024-09-16T15:43:43.615739+0000","last_clean":"2024-09-16T15:43:43.615739+0000","last_became_active":"2024-09-16T15:43:42.497465+0000","last_became_peered":"2024-09-16T15:43:42.497465+0000","last_unstale":"2024-09-16T15:43:43.615739+0000","last_undegraded":"2024-09-16T15:43:43.615739+0000","last_fullsized":"2024-09-16T15:43:43.615739+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:43:41.457712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:43:41.457712+0000","last_clean_scrub_stamp":"2024-09-16T15:43:41.457712+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:45:14.528268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.28","version":"0'0","reported_seq":13,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-16T15:43:43.484446+0000","last_change":"2024-09-16T15:43:42.518202+0000","last_active":"2024-09-16T15:43:43.484446+0000","last_peered":"2024-09-16T15:43:43.484446+0000","last_clean":"2024-09-16T15:43:43.484446+0000","last_became_active":"2024-09-16T15:43:42.517487+0000","last_became_peered":"2024-09-16T15:43:42.517487+0000","last_unstale":"2024-09-16T15:43:43.484446+0000","last_undegraded":"2024-09-16T15:43:43.484446+0000","last_fullsized":"2024-09-16T15:43:43.484446+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:43:41.457712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:43:41.457712+0000","last_clean_scrub_stamp":"2024-09-16T15:43:41.457712+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:45:15.762373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.27","version":"0'0","reported_seq":13,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-16T15:43:43.714363+0000","last_change":"2024-09-16T15:43:42.502918+0000","last_active":"2024-09-16T15:43:43.714363+0000","last_peered":"2024-09-16T15:43:43.714363+0000","last_clean":"2024-09-16T15:43:43.714363+0000","last_became_active":"2024-09-16T15:43:42.502010+0000","last_became_peered":"2024-09-16T15:43:42.502010+0000","last_unstale":"2024-09-16T15:43:43.714363+0000","last_undegraded":"2024-09-16T15:43:43.714363+0000","last_fullsized":"2024-09-16T15:43:43.714363+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:43:41.457712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:43:41.457712+0000","last_clean_scrub_stamp":"2024-09-16T15:43:41.457712+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:23:11.473976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.26","version":"0'0","reported_seq":13,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-16T15:43:43.484978+0000","last_change":"2024-09-16T15:43:42.511094+0000","last_active":"2024-09-16T15:43:43.484978+0000","last_peered":"2024-09-16T15:43:43.484978+0000","last_clean":"2024-09-16T15:43:43.484978+0000","last_became_active":"2024-09-16T15:43:42.504493+0000","last_became_peered":"2024-09-16T15:43:42.504493+0000","last_unstale":"2024-09-16T15:43:43.484978+0000","last_undegraded":"2024-09-16T15:43:43.484978+0000","last_fullsized":"2024-09-16T15:43:43.484978+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:43:41.457712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:43:41.457712+0000","last_clean_scrub_stamp":"2024-09-16T15:43:41.457712+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:39:56.071308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.25","version":"0'0","reported_seq":13,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-16T15:43:43.714836+0000","last_change":"2024-09-16T15:43:42.515442+0000","last_active":"2024-09-16T15:43:43.714836+0000","last_peered":"2024-09-16T15:43:43.714836+0000","last_clean":"2024-09-16T15:43:43.714836+0000","last_became_active":"2024-09-16T15:43:42.513955+0000","last_became_peered":"2024-09-16T15:43:42.513955+0000","last_unstale":"2024-09-16T15:43:43.714836+0000","last_undegraded":"2024-09-16T15:43:43.714836+0000","last_fullsized":"2024-09-16T15:43:43.714836+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:43:41.457712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:43:41.457712+0000","last_clean_scrub_stamp":"2024-09-16T15:43:41.457712+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:53:05.432956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.24","version":"0'0","reported_seq":13,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-16T15:43:43.486425+0000","last_change":"2024-09-16T15:43:42.523230+0000","last_active":"2024-09-16T15:43:43.486425+0000","last_peered":"2024-09-16T15:43:43.486425+0000","last_clean":"2024-09-16T15:43:43.486425+0000","last_became_active":"2024-09-16T15:43:42.522572+0000","last_became_peered":"2024-09-16T15:43:42.522572+0000","last_unstale":"2024-09-16T15:43:43.486425+0000","last_undegraded":"2024-09-16T15:43:43.486425+0000","last_fullsized":"2024-09-16T15:43:43.486425+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:43:41.457712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:43:41.457712+0000","last_clean_scrub_stamp":"2024-09-16T15:43:41.457712+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T23:39:18.158705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.23","version":"0'0","reported_seq":13,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-16T15:43:43.732806+0000","last_change":"2024-09-16T15:43:42.511612+0000","last_active":"2024-09-16T15:43:43.732806+0000","last_peered":"2024-09-16T15:43:43.732806+0000","last_clean":"2024-09-16T15:43:43.732806+0000","last_became_active":"2024-09-16T15:43:42.510806+0000","last_became_peered":"2024-09-16T15:43:42.510806+0000","last_unstale":"2024-09-16T15:43:43.732806+0000","last_undegraded":"2024-09-16T15:43:43.732806+0000","last_fullsized":"2024-09-16T15:43:43.732806+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:43:41.457712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:43:41.457712+0000","last_clean_scrub_stamp":"2024-09-16T15:43:41.457712+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T00:02:15.747411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.22","version":"0'0","reported_seq":13,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-16T15:43:43.487605+0000","last_change":"2024-09-16T15:43:42.514120+0000","last_active":"2024-09-16T15:43:43.487605+0000","last_peered":"2024-09-16T15:43:43.487605+0000","last_clean":"2024-09-16T15:43:43.487605+0000","last_became_active":"2024-09-16T15:43:42.509320+0000","last_became_peered":"2024-09-16T15:43:42.509320+0000","last_unstale":"2024-09-16T15:43:43.487605+0000","last_undegraded":"2024-09-16T15:43:43.487605+0000","last_fullsized":"2024-09-16T15:43:43.487605+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:43:41.457712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:43:41.457712+0000","last_clean_scrub_stamp":"2024-09-16T15:43:41.457712+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T23:39:18.158705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.21","version":"0'0","reported_seq":13,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-16T15:43:43.732237+0000","last_change":"2024-09-16T15:43:42.493963+0000","last_active":"2024-09-16T15:43:43.732237+0000","last_peered":"2024-09-16T15:43:43.732237+0000","last_clean":"2024-09-16T15:43:43.732237+0000","last_became_active":"2024-09-16T15:43:42.490203+0000","last_became_peered":"2024-09-16T15:43:42.490203+0000","last_unstale":"2024-09-16T15:43:43.732237+0000","last_undegraded":"2024-09-16T15:43:43.732237+0000","last_fullsized":"2024-09-16T15:43:43.732237+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:43:41.457712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:43:41.457712+0000","last_clean_scrub_stamp":"2024-09-16T15:43:41.457712+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T00:47:37.753066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.20","version":"0'0","reported_seq":13,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-16T15:43:43.485378+0000","last_change":"2024-09-16T15:43:42.501631+0000","last_active":"2024-09-16T15:43:43.485378+0000","last_peered":"2024-09-16T15:43:43.485378+0000","last_clean":"2024-09-16T15:43:43.485378+0000","last_became_active":"2024-09-16T15:43:42.500847+0000","last_became_peered":"2024-09-16T15:43:42.500847+0000","last_unstale":"2024-09-16T15:43:43.485378+0000","last_undegraded":"2024-09-16T15:43:43.485378+0000","last_fullsized":"2024-09-16T15:43:43.485378+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:43:41.457712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:43:41.457712+0000","last_clean_scrub_stamp":"2024-09-16T15:43:41.457712+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:53:05.432956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":13,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-16T15:43:43.615985+0000","last_change":"2024-09-16T15:43:42.502901+0000","last_active":"2024-09-16T15:43:43.615985+0000","last_peered":"2024-09-16T15:43:43.615985+0000","last_clean":"2024-09-16T15:43:43.615985+0000","last_became_active":"2024-09-16T15:43:42.502162+0000","last_became_peered":"2024-09-16T15:43:42.502162+0000","last_unstale":"2024-09-16T15:43:43.615985+0000","last_undegraded":"2024-09-16T15:43:43.615985+0000","last_fullsized":"2024-09-16T15:43:43.615985+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:43:41.457712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:43:41.457712+0000","last_clean_scrub_stamp":"2024-09-16T15:43:41.457712+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:25:27.174564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1f","version":"32'4","reported_seq":19,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-16T15:43:43.734645+0000","last_change":"2024-09-16T15:43:41.512316+0000","last_active":"2024-09-16T15:43:43.734645+0000","last_peered":"2024-09-16T15:43:43.734645+0000","last_clean":"2024-09-16T15:43:43.734645+0000","last_became_active":"2024-09-16T15:43:41.511559+0000","last_became_peered":"2024-09-16T15:43:41.511559+0000","last_unstale":"2024-09-16T15:43:43.734645+0000","last_undegraded":"2024-09-16T15:43:43.734645+0000","last_fullsized":"2024-09-16T15:43:43.734645+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:43:40.445567+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:43:40.445567+0000","last_clean_scrub_stamp":"2024-09-16T15:43:40.445567+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T00:47:54.458065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1f","version":"0'0","reported_seq":13,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-16T15:43:43.734537+0000","last_change":"2024-09-16T15:43:42.496030+0000","last_active":"2024-09-16T15:43:43.734537+0000","last_peered":"2024-09-16T15:43:43.734537+0000","last_clean":"2024-09-16T15:43:43.734537+0000","last_became_active":"2024-09-16T15:43:42.495281+0000","last_became_peered":"2024-09-16T15:43:42.495281+0000","last_unstale":"2024-09-16T15:43:43.734537+0000","last_undegraded":"2024-09-16T15:43:43.734537+0000","last_fullsized":"2024-09-16T15:43:43.734537+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:43:41.457712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:43:41.457712+0000","last_clean_scrub_stamp":"2024-09-16T15:43:41.457712+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T00:25:11.394386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1e","version":"32'1","reported_seq":16,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-16T15:43:43.615849+0000","last_change":"2024-09-16T15:43:41.503087+0000","last_active":"2024-09-16T15:43:43.615849+0000","last_peered":"2024-09-16T15:43:43.615849+0000","last_clean":"2024-09-16T15:43:43.615849+0000","last_became_active":"2024-09-16T15:43:41.502803+0000","last_became_peered":"2024-09-16T15:43:41.502803+0000","last_unstale":"2024-09-16T15:43:43.615849+0000","last_undegraded":"2024-09-16T15:43:43.615849+0000","last_fullsized":"2024-09-16T15:43:43.615849+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:43:40.445567+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:43:40.445567+0000","last_clean_scrub_stamp":"2024-09-16T15:43:40.445567+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T18:45:12.079394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":13,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-16T15:43:43.733308+0000","last_change":"2024-09-16T15:43:42.513223+0000","last_active":"2024-09-16T15:43:43.733308+0000","last_peered":"2024-09-16T15:43:43.733308+0000","last_clean":"2024-09-16T15:43:43.733308+0000","last_became_active":"2024-09-16T15:43:42.512493+0000","last_became_peered":"2024-09-16T15:43:42.512493+0000","last_unstale":"2024-09-16T15:43:43.733308+0000","last_undegraded":"2024-09-16T15:43:43.733308+0000","last_fullsized":"2024-09-16T15:43:43.733308+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:43:41.457712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:43:41.457712+0000","last_clean_scrub_stamp":"2024-09-16T15:43:41.457712+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T18:20:42.609654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"32'2","reported_seq":17,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-16T15:43:43.484322+0000","last_change":"2024-09-16T15:43:41.498815+0000","last_active":"2024-09-16T15:43:43.484322+0000","last_peered":"2024-09-16T15:43:43.484322+0000","last_clean":"2024-09-16T15:43:43.484322+0000","last_became_active":"2024-09-16T15:43:41.498187+0000","last_became_peered":"2024-09-16T15:43:41.498187+0000","last_unstale":"2024-09-16T15:43:43.484322+0000","last_undegraded":"2024-09-16T15:43:43.484322+0000","last_fullsized":"2024-09-16T15:43:43.484322+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:43:40.445567+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:43:40.445567+0000","last_clean_scrub_stamp":"2024-09-16T15:43:40.445567+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T23:05:21.486769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":13,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-16T15:43:43.714773+0000","last_change":"2024-09-16T15:43:42.509444+0000","last_active":"2024-09-16T15:43:43.714773+0000","last_peered":"2024-09-16T15:43:43.714773+0000","last_clean":"2024-09-16T15:43:43.714773+0000","last_became_active":"2024-09-16T15:43:42.508510+0000","last_became_peered":"2024-09-16T15:43:42.508510+0000","last_unstale":"2024-09-16T15:43:43.714773+0000","last_undegraded":"2024-09-16T15:43:43.714773+0000","last_fullsized":"2024-09-16T15:43:43.714773+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:43:41.457712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:43:41.457712+0000","last_clean_scrub_stamp":"2024-09-16T15:43:41.457712+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:45:15.762373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1c","version":"32'2","reported_seq":17,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-16T15:43:43.732327+0000","last_change":"2024-09-16T15:43:41.497006+0000","last_active":"2024-09-16T15:43:43.732327+0000","last_peered":"2024-09-16T15:43:43.732327+0000","last_clean":"2024-09-16T15:43:43.732327+0000","last_became_active":"2024-09-16T15:43:41.496696+0000","last_became_peered":"2024-09-16T15:43:41.496696+0000","last_unstale":"2024-09-16T15:43:43.732327+0000","last_undegraded":"2024-09-16T15:43:43.732327+0000","last_fullsized":"2024-09-16T15:43:43.732327+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:43:40.445567+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:43:40.445567+0000","last_clean_scrub_stamp":"2024-09-16T15:43:40.445567+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:08:07.602741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":13,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-16T15:43:43.487705+0000","last_change":"2024-09-16T15:43:42.515296+0000","last_active":"2024-09-16T15:43:43.487705+0000","last_peered":"2024-09-16T15:43:43.487705+0000","last_clean":"2024-09-16T15:43:43.487705+0000","last_became_active":"2024-09-16T15:43:42.514616+0000","last_became_peered":"2024-09-16T15:43:42.514616+0000","last_unstale":"2024-09-16T15:43:43.487705+0000","last_undegraded":"2024-09-16T15:43:43.487705+0000","last_fullsized":"2024-09-16T15:43:43.487705+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:43:41.457712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:43:41.457712+0000","last_clean_scrub_stamp":"2024-09-16T15:43:41.457712+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T01:57:17.824602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1b","version":"32'1","reported_seq":16,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-16T15:43:43.733143+0000","last_change":"2024-09-16T15:43:41.508469+0000","last_active":"2024-09-16T15:43:43.733143+0000","last_peered":"2024-09-16T15:43:43.733143+0000","last_clean":"2024-09-16T15:43:43.733143+0000","last_became_active":"2024-09-16T15:43:41.507710+0000","last_became_peered":"2024-09-16T15:43:41.507710+0000","last_unstale":"2024-09-16T15:43:43.733143+0000","last_undegraded":"2024-09-16T15:43:43.733143+0000","last_fullsized":"2024-09-16T15:43:43.733143+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:43:40.445567+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:43:40.445567+0000","last_clean_scrub_stamp":"2024-09-16T15:43:40.445567+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T00:47:54.458065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":13,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-16T15:43:43.485942+0000","last_change":"2024-09-16T15:43:42.528041+0000","last_active":"2024-09-16T15:43:43.485942+0000","last_peered":"2024-09-16T15:43:43.485942+0000","last_clean":"2024-09-16T15:43:43.485942+0000","last_became_active":"2024-09-16T15:43:42.524019+0000","last_became_peered":"2024-09-16T15:43:42.524019+0000","last_unstale":"2024-09-16T15:43:43.485942+0000","last_undegraded":"2024-09-16T15:43:43.485942+0000","last_fullsized":"2024-09-16T15:43:43.485942+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:43:41.457712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:43:41.457712+0000","last_clean_scrub_stamp":"2024-09-16T15:43:41.457712+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T23:09:25.039240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"32'1","reported_seq":16,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-16T15:43:43.733350+0000","last_change":"2024-09-16T15:43:41.505606+0000","last_active":"2024-09-16T15:43:43.733350+0000","last_peered":"2024-09-16T15:43:43.733350+0000","last_clean":"2024-09-16T15:43:43.733350+0000","last_became_active":"2024-09-16T15:43:41.501328+0000","last_became_peered":"2024-09-16T15:43:41.501328+0000","last_unstale":"2024-09-16T15:43:43.733350+0000","last_undegraded":"2024-09-16T15:43:43.733350+0000","last_fullsized":"2024-09-16T15:43:43.733350+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:43:40.445567+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:43:40.445567+0000","last_clean_scrub_stamp":"2024-09-16T15:43:40.445567+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:02:43.216562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":13,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-16T15:43:43.732398+0000","last_change":"2024-09-16T15:43:42.506325+0000","last_active":"2024-09-16T15:43:43.732398+0000","last_peered":"2024-09-16T15:43:43.732398+0000","last_clean":"2024-09-16T15:43:43.732398+0000","last_became_active":"2024-09-16T15:43:42.505588+0000","last_became_peered":"2024-09-16T15:43:42.505588+0000","last_unstale":"2024-09-16T15:43:43.732398+0000","last_undegraded":"2024-09-16T15:43:43.732398+0000","last_fullsized":"2024-09-16T15:43:43.732398+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:43:41.457712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:43:41.457712+0000","last_clean_scrub_stamp":"2024-09-16T15:43:41.457712+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T01:45:43.408312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":15,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-16T15:43:43.483256+0000","last_change":"2024-09-16T15:43:41.495106+0000","last_active":"2024-09-16T15:43:43.483256+0000","last_peered":"2024-09-16T15:43:43.483256+0000","last_clean":"2024-09-16T15:43:43.483256+0000","last_became_active":"2024-09-16T15:43:41.494152+0000","last_became_peered":"2024-09-16T15:43:41.494152+0000","last_unstale":"2024-09-16T15:43:43.483256+0000","last_undegraded":"2024-09-16T15:43:43.483256+0000","last_fullsized":"2024-09-16T15:43:43.483256+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:43:40.445567+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:43:40.445567+0000","last_clean_scrub_stamp":"2024-09-16T15:43:40.445567+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T22:41:37.724364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":13,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-16T15:43:43.713544+0000","last_change":"2024-09-16T15:43:42.524416+0000","last_active":"2024-09-16T15:43:43.713544+0000","last_peered":"2024-09-16T15:43:43.713544+0000","last_clean":"2024-09-16T15:43:43.713544+0000","last_became_active":"2024-09-16T15:43:42.523604+0000","last_became_peered":"2024-09-16T15:43:42.523604+0000","last_unstale":"2024-09-16T15:43:43.713544+0000","last_undegraded":"2024-09-16T15:43:43.713544+0000","last_fullsized":"2024-09-16T15:43:43.713544+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:43:41.457712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:43:41.457712+0000","last_clean_scrub_stamp":"2024-09-16T15:43:41.457712+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T18:57:49.412121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":15,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-16T15:43:43.732702+0000","last_change":"2024-09-16T15:43:41.505626+0000","last_active":"2024-09-16T15:43:43.732702+0000","last_peered":"2024-09-16T15:43:43.732702+0000","last_clean":"2024-09-16T15:43:43.732702+0000","last_became_active":"2024-09-16T15:43:41.500940+0000","last_became_peered":"2024-09-16T15:43:41.500940+0000","last_unstale":"2024-09-16T15:43:43.732702+0000","last_undegraded":"2024-09-16T15:43:43.732702+0000","last_fullsized":"2024-09-16T15:43:43.732702+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:43:40.445567+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:43:40.445567+0000","last_clean_scrub_stamp":"2024-09-16T15:43:40.445567+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T23:23:26.003518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":13,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-16T15:43:43.487182+0000","last_change":"2024-09-16T15:43:42.528814+0000","last_active":"2024-09-16T15:43:43.487182+0000","last_peered":"2024-09-16T15:43:43.487182+0000","last_clean":"2024-09-16T15:43:43.487182+0000","last_became_active":"2024-09-16T15:43:42.527052+0000","last_became_peered":"2024-09-16T15:43:42.527052+0000","last_unstale":"2024-09-16T15:43:43.487182+0000","last_undegraded":"2024-09-16T15:43:43.487182+0000","last_fullsized":"2024-09-16T15:43:43.487182+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:43:41.457712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:43:41.457712+0000","last_clean_scrub_stamp":"2024-09-16T15:43:41.457712+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T03:25:30.813074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":15,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-16T15:43:43.484626+0000","last_change":"2024-09-16T15:43:41.505547+0000","last_active":"2024-09-16T15:43:43.484626+0000","last_peered":"2024-09-16T15:43:43.484626+0000","last_clean":"2024-09-16T15:43:43.484626+0000","last_became_active":"2024-09-16T15:43:41.505285+0000","last_became_peered":"2024-09-16T15:43:41.505285+0000","last_unstale":"2024-09-16T15:43:43.484626+0000","last_undegraded":"2024-09-16T15:43:43.484626+0000","last_fullsized":"2024-09-16T15:43:43.484626+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:43:40.445567+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:43:40.445567+0000","last_clean_scrub_stamp":"2024-09-16T15:43:40.445567+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:29:42.437978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":13,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-16T15:43:43.484781+0000","last_change":"2024-09-16T15:43:42.521329+0000","last_active":"2024-09-16T15:43:43.484781+0000","last_peered":"2024-09-16T15:43:43.484781+0000","last_clean":"2024-09-16T15:43:43.484781+0000","last_became_active":"2024-09-16T15:43:42.512536+0000","last_became_peered":"2024-09-16T15:43:42.512536+0000","last_unstale":"2024-09-16T15:43:43.484781+0000","last_undegraded":"2024-09-16T15:43:43.484781+0000","last_fullsized":"2024-09-16T15:43:43.484781+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:43:41.457712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:43:41.457712+0000","last_clean_scrub_stamp":"2024-09-16T15:43:41.457712+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T01:12:22.003154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":15,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-16T15:43:43.616069+0000","last_change":"2024-09-16T15:43:41.499795+0000","last_active":"2024-09-16T15:43:43.616069+0000","last_peered":"2024-09-16T15:43:43.616069+0000","last_clean":"2024-09-16T15:43:43.616069+0000","last_became_active":"2024-09-16T15:43:41.499475+0000","last_became_peered":"2024-09-16T15:43:41.499475+0000","last_unstale":"2024-09-16T15:43:43.616069+0000","last_undegraded":"2024-09-16T15:43:43.616069+0000","last_fullsized":"2024-09-16T15:43:43.616069+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:43:40.445567+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:43:40.445567+0000","last_clean_scrub_stamp":"2024-09-16T15:43:40.445567+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:08:18.770146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"0'0","reported_seq":13,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-16T15:43:43.484086+0000","last_change":"2024-09-16T15:43:42.509837+0000","last_active":"2024-09-16T15:43:43.484086+0000","last_peered":"2024-09-16T15:43:43.484086+0000","last_clean":"2024-09-16T15:43:43.484086+0000","last_became_active":"2024-09-16T15:43:42.505473+0000","last_became_peered":"2024-09-16T15:43:42.505473+0000","last_unstale":"2024-09-16T15:43:43.484086+0000","last_undegraded":"2024-09-16T15:43:43.484086+0000","last_fullsized":"2024-09-16T15:43:43.484086+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:43:41.457712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:43:41.457712+0000","last_clean_scrub_stamp":"2024-09-16T15:43:41.457712+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:50:12.328928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.15","version":"32'3","reported_seq":18,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-16T15:43:43.487542+0000","last_change":"2024-09-16T15:43:41.506168+0000","last_active":"2024-09-16T15:43:43.487542+0000","last_peered":"2024-09-16T15:43:43.487542+0000","last_clean":"2024-09-16T15:43:43.487542+0000","last_became_active":"2024-09-16T15:43:41.505392+0000","last_became_peered":"2024-09-16T15:43:41.505392+0000","last_unstale":"2024-09-16T15:43:43.487542+0000","last_undegraded":"2024-09-16T15:43:43.487542+0000","last_fullsized":"2024-09-16T15:43:43.487542+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:43:40.445567+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:43:40.445567+0000","last_clean_scrub_stamp":"2024-09-16T15:43:40.445567+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T18:41:12.648868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"0'0","reported_seq":13,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-16T15:43:43.486818+0000","last_change":"2024-09-16T15:43:42.532302+0000","last_active":"2024-09-16T15:43:43.486818+0000","last_peered":"2024-09-16T15:43:43.486818+0000","last_clean":"2024-09-16T15:43:43.486818+0000","last_became_active":"2024-09-16T15:43:42.523267+0000","last_became_peered":"2024-09-16T15:43:42.523267+0000","last_unstale":"2024-09-16T15:43:43.486818+0000","last_undegraded":"2024-09-16T15:43:43.486818+0000","last_fullsized":"2024-09-16T15:43:43.486818+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:43:41.457712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:43:41.457712+0000","last_clean_scrub_stamp":"2024-09-16T15:43:41.457712+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:53:25.198570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"32'6","reported_seq":21,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-16T15:43:43.484198+0000","last_change":"2024-09-16T15:43:41.502074+0000","last_active":"2024-09-16T15:43:43.484198+0000","last_peered":"2024-09-16T15:43:43.484198+0000","last_clean":"2024-09-16T15:43:43.484198+0000","last_became_active":"2024-09-16T15:43:41.501797+0000","last_became_peered":"2024-09-16T15:43:41.501797+0000","last_unstale":"2024-09-16T15:43:43.484198+0000","last_undegraded":"2024-09-16T15:43:43.484198+0000","last_fullsized":"2024-09-16T15:43:43.484198+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:43:40.445567+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:43:40.445567+0000","last_clean_scrub_stamp":"2024-09-16T15:43:40.445567+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T00:25:10.382240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":13,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-16T15:43:43.713645+0000","last_change":"2024-09-16T15:43:42.524990+0000","last_active":"2024-09-16T15:43:43.713645+0000","last_peered":"2024-09-16T15:43:43.713645+0000","last_clean":"2024-09-16T15:43:43.713645+0000","last_became_active":"2024-09-16T15:43:42.523907+0000","last_became_peered":"2024-09-16T15:43:42.523907+0000","last_unstale":"2024-09-16T15:43:43.713645+0000","last_undegraded":"2024-09-16T15:43:43.713645+0000","last_fullsized":"2024-09-16T15:43:43.713645+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:43:41.457712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:43:41.457712+0000","last_clean_scrub_stamp":"2024-09-16T15:43:41.457712+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:23:06.229761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"32'2","reported_seq":17,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-16T15:43:43.733437+0000","last_change":"2024-09-16T15:43:41.507979+0000","last_active":"2024-09-16T15:43:43.733437+0000","last_peered":"2024-09-16T15:43:43.733437+0000","last_clean":"2024-09-16T15:43:43.733437+0000","last_became_active":"2024-09-16T15:43:41.507251+0000","last_became_peered":"2024-09-16T15:43:41.507251+0000","last_unstale":"2024-09-16T15:43:43.733437+0000","last_undegraded":"2024-09-16T15:43:43.733437+0000","last_fullsized":"2024-09-16T15:43:43.733437+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:43:40.445567+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:43:40.445567+0000","last_clean_scrub_stamp":"2024-09-16T15:43:40.445567+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:10:58.646118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":13,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-16T15:43:43.484054+0000","last_change":"2024-09-16T15:43:42.503800+0000","last_active":"2024-09-16T15:43:43.484054+0000","last_peered":"2024-09-16T15:43:43.484054+0000","last_clean":"2024-09-16T15:43:43.484054+0000","last_became_active":"2024-09-16T15:43:42.503081+0000","last_became_peered":"2024-09-16T15:43:42.503081+0000","last_unstale":"2024-09-16T15:43:43.484054+0000","last_undegraded":"2024-09-16T15:43:43.484054+0000","last_fullsized":"2024-09-16T15:43:43.484054+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:43:41.457712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:43:41.457712+0000","last_clean_scrub_stamp":"2024-09-16T15:43:41.457712+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T03:02:24.003658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":15,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-16T15:43:43.713484+0000","last_change":"2024-09-16T15:43:41.501377+0000","last_active":"2024-09-16T15:43:43.713484+0000","last_peered":"2024-09-16T15:43:43.713484+0000","last_clean":"2024-09-16T15:43:43.713484+0000","last_became_active":"2024-09-16T15:43:41.500885+0000","last_became_peered":"2024-09-16T15:43:41.500885+0000","last_unstale":"2024-09-16T15:43:43.713484+0000","last_undegraded":"2024-09-16T15:43:43.713484+0000","last_fullsized":"2024-09-16T15:43:43.713484+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:43:40.445567+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:43:40.445567+0000","last_clean_scrub_stamp":"2024-09-16T15:43:40.445567+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:08:18.770146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":13,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-16T15:43:43.485914+0000","last_change":"2024-09-16T15:43:42.515158+0000","last_active":"2024-09-16T15:43:43.485914+0000","last_peered":"2024-09-16T15:43:43.485914+0000","last_clean":"2024-09-16T15:43:43.485914+0000","last_became_active":"2024-09-16T15:43:42.502582+0000","last_became_peered":"2024-09-16T15:43:42.502582+0000","last_unstale":"2024-09-16T15:43:43.485914+0000","last_undegraded":"2024-09-16T15:43:43.485914+0000","last_fullsized":"2024-09-16T15:43:43.485914+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:43:41.457712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:43:41.457712+0000","last_clean_scrub_stamp":"2024-09-16T15:43:41.457712+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T23:09:25.039240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":15,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-16T15:43:43.483338+0000","last_change":"2024-09-16T15:43:41.510375+0000","last_active":"2024-09-16T15:43:43.483338+0000","last_peered":"2024-09-16T15:43:43.483338+0000","last_clean":"2024-09-16T15:43:43.483338+0000","last_became_active":"2024-09-16T15:43:41.510044+0000","last_became_peered":"2024-09-16T15:43:41.510044+0000","last_unstale":"2024-09-16T15:43:43.483338+0000","last_undegraded":"2024-09-16T15:43:43.483338+0000","last_fullsized":"2024-09-16T15:43:43.483338+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:43:40.445567+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:43:40.445567+0000","last_clean_scrub_stamp":"2024-09-16T15:43:40.445567+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T02:51:57.745557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":13,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-16T15:43:43.732293+0000","last_change":"2024-09-16T15:43:42.496964+0000","last_active":"2024-09-16T15:43:43.732293+0000","last_peered":"2024-09-16T15:43:43.732293+0000","last_clean":"2024-09-16T15:43:43.732293+0000","last_became_active":"2024-09-16T15:43:42.496284+0000","last_became_peered":"2024-09-16T15:43:42.496284+0000","last_unstale":"2024-09-16T15:43:43.732293+0000","last_undegraded":"2024-09-16T15:43:43.732293+0000","last_fullsized":"2024-09-16T15:43:43.732293+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:43:41.457712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:43:41.457712+0000","last_clean_scrub_stamp":"2024-09-16T15:43:41.457712+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T22:41:38.736510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.10","version":"32'8","reported_seq":23,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-16T15:43:43.483002+0000","last_change":"2024-09-16T15:43:41.500850+0000","last_active":"2024-09-16T15:43:43.483002+0000","last_peered":"2024-09-16T15:43:43.483002+0000","last_clean":"2024-09-16T15:43:43.483002+0000","last_became_active":"2024-09-16T15:43:41.500181+0000","last_became_peered":"2024-09-16T15:43:41.500181+0000","last_unstale":"2024-09-16T15:43:43.483002+0000","last_undegraded":"2024-09-16T15:43:43.483002+0000","last_fullsized":"2024-09-16T15:43:43.483002+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:43:40.445567+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:43:40.445567+0000","last_clean_scrub_stamp":"2024-09-16T15:43:40.445567+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T00:47:36.740921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"0'0","reported_seq":13,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-16T15:43:43.733240+0000","last_change":"2024-09-16T15:43:42.520243+0000","last_active":"2024-09-16T15:43:43.733240+0000","last_peered":"2024-09-16T15:43:43.733240+0000","last_clean":"2024-09-16T15:43:43.733240+0000","last_became_active":"2024-09-16T15:43:42.519538+0000","last_became_peered":"2024-09-16T15:43:42.519538+0000","last_unstale":"2024-09-16T15:43:43.733240+0000","last_undegraded":"2024-09-16T15:43:43.733240+0000","last_fullsized":"2024-09-16T15:43:43.733240+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:43:41.457712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:43:41.457712+0000","last_clean_scrub_stamp":"2024-09-16T15:43:41.457712+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T03:21:18.518406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.f","version":"32'1","reported_seq":16,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-16T15:43:43.714497+0000","last_change":"2024-09-16T15:43:41.505805+0000","last_active":"2024-09-16T15:43:43.714497+0000","last_peered":"2024-09-16T15:43:43.714497+0000","last_clean":"2024-09-16T15:43:43.714497+0000","last_became_active":"2024-09-16T15:43:41.505538+0000","last_became_peered":"2024-09-16T15:43:41.505538+0000","last_unstale":"2024-09-16T15:43:43.714497+0000","last_undegraded":"2024-09-16T15:43:43.714497+0000","last_fullsized":"2024-09-16T15:43:43.714497+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:43:40.445567+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:43:40.445567+0000","last_clean_scrub_stamp":"2024-09-16T15:43:40.445567+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T18:46:12.199768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":13,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-16T15:43:43.486566+0000","last_change":"2024-09-16T15:43:42.523995+0000","last_active":"2024-09-16T15:43:43.486566+0000","last_peered":"2024-09-16T15:43:43.486566+0000","last_clean":"2024-09-16T15:43:43.486566+0000","last_became_active":"2024-09-16T15:43:42.523379+0000","last_became_peered":"2024-09-16T15:43:42.523379+0000","last_unstale":"2024-09-16T15:43:43.486566+0000","last_undegraded":"2024-09-16T15:43:43.486566+0000","last_fullsized":"2024-09-16T15:43:43.486566+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:43:41.457712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:43:41.457712+0000","last_clean_scrub_stamp":"2024-09-16T15:43:41.457712+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T01:57:17.824602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.e","version":"32'2","reported_seq":17,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-16T15:43:43.733519+0000","last_change":"2024-09-16T15:43:41.507958+0000","last_active":"2024-09-16T15:43:43.733519+0000","last_peered":"2024-09-16T15:43:43.733519+0000","last_clean":"2024-09-16T15:43:43.733519+0000","last_became_active":"2024-09-16T15:43:41.507221+0000","last_became_peered":"2024-09-16T15:43:41.507221+0000","last_unstale":"2024-09-16T15:43:43.733519+0000","last_undegraded":"2024-09-16T15:43:43.733519+0000","last_fullsized":"2024-09-16T15:43:43.733519+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:43:40.445567+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:43:40.445567+0000","last_clean_scrub_stamp":"2024-09-16T15:43:40.445567+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T18:01:15.103311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":13,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-16T15:43:43.713969+0000","last_change":"2024-09-16T15:43:42.514591+0000","last_active":"2024-09-16T15:43:43.713969+0000","last_peered":"2024-09-16T15:43:43.713969+0000","last_clean":"2024-09-16T15:43:43.713969+0000","last_became_active":"2024-09-16T15:43:42.512955+0000","last_became_peered":"2024-09-16T15:43:42.512955+0000","last_unstale":"2024-09-16T15:43:43.713969+0000","last_undegraded":"2024-09-16T15:43:43.713969+0000","last_fullsized":"2024-09-16T15:43:43.713969+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:43:41.457712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:43:41.457712+0000","last_clean_scrub_stamp":"2024-09-16T15:43:41.457712+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T22:30:41.098254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.d","version":"32'1","reported_seq":16,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-16T15:43:43.733814+0000","last_change":"2024-09-16T15:43:41.498188+0000","last_active":"2024-09-16T15:43:43.733814+0000","last_peered":"2024-09-16T15:43:43.733814+0000","last_clean":"2024-09-16T15:43:43.733814+0000","last_became_active":"2024-09-16T15:43:41.497233+0000","last_became_peered":"2024-09-16T15:43:41.497233+0000","last_unstale":"2024-09-16T15:43:43.733814+0000","last_undegraded":"2024-09-16T15:43:43.733814+0000","last_fullsized":"2024-09-16T15:43:43.733814+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:43:40.445567+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:43:40.445567+0000","last_clean_scrub_stamp":"2024-09-16T15:43:40.445567+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T00:53:49.420412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"0'0","reported_seq":13,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-16T15:43:43.732411+0000","last_change":"2024-09-16T15:43:42.504728+0000","last_active":"2024-09-16T15:43:43.732411+0000","last_peered":"2024-09-16T15:43:43.732411+0000","last_clean":"2024-09-16T15:43:43.732411+0000","last_became_active":"2024-09-16T15:43:42.503979+0000","last_became_peered":"2024-09-16T15:43:42.503979+0000","last_unstale":"2024-09-16T15:43:43.732411+0000","last_undegraded":"2024-09-16T15:43:43.732411+0000","last_fullsized":"2024-09-16T15:43:43.732411+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:43:41.457712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:43:41.457712+0000","last_clean_scrub_stamp":"2024-09-16T15:43:41.457712+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T18:46:13.211914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":15,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-16T15:43:43.733613+0000","last_change":"2024-09-16T15:43:41.506932+0000","last_active":"2024-09-16T15:43:43.733613+0000","last_peered":"2024-09-16T15:43:43.733613+0000","last_clean":"2024-09-16T15:43:43.733613+0000","last_became_active":"2024-09-16T15:43:41.506136+0000","last_became_peered":"2024-09-16T15:43:41.506136+0000","last_unstale":"2024-09-16T15:43:43.733613+0000","last_undegraded":"2024-09-16T15:43:43.733613+0000","last_fullsized":"2024-09-16T15:43:43.733613+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:43:40.445567+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:43:40.445567+0000","last_clean_scrub_stamp":"2024-09-16T15:43:40.445567+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T02:37:07.989248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":13,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-16T15:43:43.732100+0000","last_change":"2024-09-16T15:43:42.496091+0000","last_active":"2024-09-16T15:43:43.732100+0000","last_peered":"2024-09-16T15:43:43.732100+0000","last_clean":"2024-09-16T15:43:43.732100+0000","last_became_active":"2024-09-16T15:43:42.495370+0000","last_became_peered":"2024-09-16T15:43:42.495370+0000","last_unstale":"2024-09-16T15:43:43.732100+0000","last_undegraded":"2024-09-16T15:43:43.732100+0000","last_fullsized":"2024-09-16T15:43:43.732100+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:43:41.457712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:43:41.457712+0000","last_clean_scrub_stamp":"2024-09-16T15:43:41.457712+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T18:02:56.530946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.b","version":"32'2","reported_seq":17,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-16T15:43:43.616117+0000","last_change":"2024-09-16T15:43:41.495721+0000","last_active":"2024-09-16T15:43:43.616117+0000","last_peered":"2024-09-16T15:43:43.616117+0000","last_clean":"2024-09-16T15:43:43.616117+0000","last_became_active":"2024-09-16T15:43:41.495439+0000","last_became_peered":"2024-09-16T15:43:41.495439+0000","last_unstale":"2024-09-16T15:43:43.616117+0000","last_undegraded":"2024-09-16T15:43:43.616117+0000","last_fullsized":"2024-09-16T15:43:43.616117+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:43:40.445567+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:43:40.445567+0000","last_clean_scrub_stamp":"2024-09-16T15:43:40.445567+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T18:46:12.199768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":13,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-16T15:43:43.732759+0000","last_change":"2024-09-16T15:43:42.506436+0000","last_active":"2024-09-16T15:43:43.732759+0000","last_peered":"2024-09-16T15:43:43.732759+0000","last_clean":"2024-09-16T15:43:43.732759+0000","last_became_active":"2024-09-16T15:43:42.505694+0000","last_became_peered":"2024-09-16T15:43:42.505694+0000","last_unstale":"2024-09-16T15:43:43.732759+0000","last_undegraded":"2024-09-16T15:43:43.732759+0000","last_fullsized":"2024-09-16T15:43:43.732759+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:43:41.457712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:43:41.457712+0000","last_clean_scrub_stamp":"2024-09-16T15:43:41.457712+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T18:45:13.091540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":15,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-16T15:43:43.733766+0000","last_change":"2024-09-16T15:43:41.499830+0000","last_active":"2024-09-16T15:43:43.733766+0000","last_peered":"2024-09-16T15:43:43.733766+0000","last_clean":"2024-09-16T15:43:43.733766+0000","last_became_active":"2024-09-16T15:43:41.498784+0000","last_became_peered":"2024-09-16T15:43:41.498784+0000","last_unstale":"2024-09-16T15:43:43.733766+0000","last_undegraded":"2024-09-16T15:43:43.733766+0000","last_fullsized":"2024-09-16T15:43:43.733766+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:43:40.445567+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:43:40.445567+0000","last_clean_scrub_stamp":"2024-09-16T15:43:40.445567+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T00:53:49.420412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":13,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-16T15:43:43.732087+0000","last_change":"2024-09-16T15:43:42.499649+0000","last_active":"2024-09-16T15:43:43.732087+0000","last_peered":"2024-09-16T15:43:43.732087+0000","last_clean":"2024-09-16T15:43:43.732087+0000","last_became_active":"2024-09-16T15:43:42.498576+0000","last_became_peered":"2024-09-16T15:43:42.498576+0000","last_unstale":"2024-09-16T15:43:43.732087+0000","last_undegraded":"2024-09-16T15:43:43.732087+0000","last_fullsized":"2024-09-16T15:43:43.732087+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:43:41.457712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:43:41.457712+0000","last_clean_scrub_stamp":"2024-09-16T15:43:41.457712+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T22:44:48.708241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":15,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-16T15:43:43.732581+0000","last_change":"2024-09-16T15:43:41.513191+0000","last_active":"2024-09-16T15:43:43.732581+0000","last_peered":"2024-09-16T15:43:43.732581+0000","last_clean":"2024-09-16T15:43:43.732581+0000","last_became_active":"2024-09-16T15:43:41.501991+0000","last_became_peered":"2024-09-16T15:43:41.501991+0000","last_unstale":"2024-09-16T15:43:43.732581+0000","last_undegraded":"2024-09-16T15:43:43.732581+0000","last_fullsized":"2024-09-16T15:43:43.732581+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:43:40.445567+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:43:40.445567+0000","last_clean_scrub_stamp":"2024-09-16T15:43:40.445567+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T03:11:54.723233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":13,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-16T15:43:43.485156+0000","last_change":"2024-09-16T15:43:42.508427+0000","last_active":"2024-09-16T15:43:43.485156+0000","last_peered":"2024-09-16T15:43:43.485156+0000","last_clean":"2024-09-16T15:43:43.485156+0000","last_became_active":"2024-09-16T15:43:42.507685+0000","last_became_peered":"2024-09-16T15:43:42.507685+0000","last_unstale":"2024-09-16T15:43:43.485156+0000","last_undegraded":"2024-09-16T15:43:43.485156+0000","last_fullsized":"2024-09-16T15:43:43.485156+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:43:41.457712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:43:41.457712+0000","last_clean_scrub_stamp":"2024-09-16T15:43:41.457712+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:45:15.762373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":15,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-16T15:43:43.484093+0000","last_change":"2024-09-16T15:43:41.507727+0000","last_active":"2024-09-16T15:43:43.484093+0000","last_peered":"2024-09-16T15:43:43.484093+0000","last_clean":"2024-09-16T15:43:43.484093+0000","last_became_active":"2024-09-16T15:43:41.507048+0000","last_became_peered":"2024-09-16T15:43:41.507048+0000","last_unstale":"2024-09-16T15:43:43.484093+0000","last_undegraded":"2024-09-16T15:43:43.484093+0000","last_fullsized":"2024-09-16T15:43:43.484093+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:43:40.445567+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:43:40.445567+0000","last_clean_scrub_stamp":"2024-09-16T15:43:40.445567+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:30:00.160372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":13,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-16T15:43:43.732577+0000","last_change":"2024-09-16T15:43:42.518408+0000","last_active":"2024-09-16T15:43:43.732577+0000","last_peered":"2024-09-16T15:43:43.732577+0000","last_clean":"2024-09-16T15:43:43.732577+0000","last_became_active":"2024-09-16T15:43:42.517727+0000","last_became_peered":"2024-09-16T15:43:42.517727+0000","last_unstale":"2024-09-16T15:43:43.732577+0000","last_undegraded":"2024-09-16T15:43:43.732577+0000","last_fullsized":"2024-09-16T15:43:43.732577+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:43:41.457712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:43:41.457712+0000","last_clean_scrub_stamp":"2024-09-16T15:43:41.457712+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:14:01.389644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"32'2","reported_seq":17,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-16T15:43:43.714520+0000","last_change":"2024-09-16T15:43:41.496121+0000","last_active":"2024-09-16T15:43:43.714520+0000","last_peered":"2024-09-16T15:43:43.714520+0000","last_clean":"2024-09-16T15:43:43.714520+0000","last_became_active":"2024-09-16T15:43:41.495417+0000","last_became_peered":"2024-09-16T15:43:41.495417+0000","last_unstale":"2024-09-16T15:43:43.714520+0000","last_undegraded":"2024-09-16T15:43:43.714520+0000","last_fullsized":"2024-09-16T15:43:43.714520+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:43:40.445567+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:43:40.445567+0000","last_clean_scrub_stamp":"2024-09-16T15:43:40.445567+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T01:59:55.645813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":13,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-16T15:43:43.487054+0000","last_change":"2024-09-16T15:43:42.527574+0000","last_active":"2024-09-16T15:43:43.487054+0000","last_peered":"2024-09-16T15:43:43.487054+0000","last_clean":"2024-09-16T15:43:43.487054+0000","last_became_active":"2024-09-16T15:43:42.526817+0000","last_became_peered":"2024-09-16T15:43:42.526817+0000","last_unstale":"2024-09-16T15:43:43.487054+0000","last_undegraded":"2024-09-16T15:43:43.487054+0000","last_fullsized":"2024-09-16T15:43:43.487054+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:43:41.457712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:43:41.457712+0000","last_clean_scrub_stamp":"2024-09-16T15:43:41.457712+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T00:01:42.341879+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":15,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-16T15:43:43.616270+0000","last_change":"2024-09-16T15:43:41.495632+0000","last_active":"2024-09-16T15:43:43.616270+0000","last_peered":"2024-09-16T15:43:43.616270+0000","last_clean":"2024-09-16T15:43:43.616270+0000","last_became_active":"2024-09-16T15:43:41.495343+0000","last_became_peered":"2024-09-16T15:43:41.495343+0000","last_unstale":"2024-09-16T15:43:43.616270+0000","last_undegraded":"2024-09-16T15:43:43.616270+0000","last_fullsized":"2024-09-16T15:43:43.616270+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:43:40.445567+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:43:40.445567+0000","last_clean_scrub_stamp":"2024-09-16T15:43:40.445567+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T01:30:19.395237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":13,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-16T15:43:43.615859+0000","last_change":"2024-09-16T15:43:42.498071+0000","last_active":"2024-09-16T15:43:43.615859+0000","last_peered":"2024-09-16T15:43:43.615859+0000","last_clean":"2024-09-16T15:43:43.615859+0000","last_became_active":"2024-09-16T15:43:42.497308+0000","last_became_peered":"2024-09-16T15:43:42.497308+0000","last_unstale":"2024-09-16T15:43:43.615859+0000","last_undegraded":"2024-09-16T15:43:43.615859+0000","last_fullsized":"2024-09-16T15:43:43.615859+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:43:41.457712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:43:41.457712+0000","last_clean_scrub_stamp":"2024-09-16T15:43:41.457712+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T03:04:13.235513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":15,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-16T15:43:43.714262+0000","last_change":"2024-09-16T15:43:41.498126+0000","last_active":"2024-09-16T15:43:43.714262+0000","last_peered":"2024-09-16T15:43:43.714262+0000","last_clean":"2024-09-16T15:43:43.714262+0000","last_became_active":"2024-09-16T15:43:41.497610+0000","last_became_peered":"2024-09-16T15:43:41.497610+0000","last_unstale":"2024-09-16T15:43:43.714262+0000","last_undegraded":"2024-09-16T15:43:43.714262+0000","last_fullsized":"2024-09-16T15:43:43.714262+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:43:40.445567+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:43:40.445567+0000","last_clean_scrub_stamp":"2024-09-16T15:43:40.445567+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T01:30:19.395237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":13,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-16T15:43:43.734102+0000","last_change":"2024-09-16T15:43:42.497130+0000","last_active":"2024-09-16T15:43:43.734102+0000","last_peered":"2024-09-16T15:43:43.734102+0000","last_clean":"2024-09-16T15:43:43.734102+0000","last_became_active":"2024-09-16T15:43:42.496289+0000","last_became_peered":"2024-09-16T15:43:42.496289+0000","last_unstale":"2024-09-16T15:43:43.734102+0000","last_undegraded":"2024-09-16T15:43:43.734102+0000","last_fullsized":"2024-09-16T15:43:43.734102+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:43:41.457712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:43:41.457712+0000","last_clean_scrub_stamp":"2024-09-16T15:43:41.457712+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:42:45.472879+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":15,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-16T15:43:43.616383+0000","last_change":"2024-09-16T15:43:41.492043+0000","last_active":"2024-09-16T15:43:43.616383+0000","last_peered":"2024-09-16T15:43:43.616383+0000","last_clean":"2024-09-16T15:43:43.616383+0000","last_became_active":"2024-09-16T15:43:41.491503+0000","last_became_peered":"2024-09-16T15:43:41.491503+0000","last_unstale":"2024-09-16T15:43:43.616383+0000","last_undegraded":"2024-09-16T15:43:43.616383+0000","last_fullsized":"2024-09-16T15:43:43.616383+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:43:40.445567+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:43:40.445567+0000","last_clean_scrub_stamp":"2024-09-16T15:43:40.445567+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T01:59:55.645813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":13,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-16T15:43:43.616244+0000","last_change":"2024-09-16T15:43:42.499650+0000","last_active":"2024-09-16T15:43:43.616244+0000","last_peered":"2024-09-16T15:43:43.616244+0000","last_clean":"2024-09-16T15:43:43.616244+0000","last_became_active":"2024-09-16T15:43:42.496108+0000","last_became_peered":"2024-09-16T15:43:42.496108+0000","last_unstale":"2024-09-16T15:43:43.616244+0000","last_undegraded":"2024-09-16T15:43:43.616244+0000","last_fullsized":"2024-09-16T15:43:43.616244+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:43:41.457712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:43:41.457712+0000","last_clean_scrub_stamp":"2024-09-16T15:43:41.457712+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:33:04.168915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":15,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-16T15:43:43.713727+0000","last_change":"2024-09-16T15:43:41.494711+0000","last_active":"2024-09-16T15:43:43.713727+0000","last_peered":"2024-09-16T15:43:43.713727+0000","last_clean":"2024-09-16T15:43:43.713727+0000","last_became_active":"2024-09-16T15:43:41.494462+0000","last_became_peered":"2024-09-16T15:43:41.494462+0000","last_unstale":"2024-09-16T15:43:43.713727+0000","last_undegraded":"2024-09-16T15:43:43.713727+0000","last_fullsized":"2024-09-16T15:43:43.713727+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:43:40.445567+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:43:40.445567+0000","last_clean_scrub_stamp":"2024-09-16T15:43:40.445567+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T18:45:12.079394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":13,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-16T15:43:43.483277+0000","last_change":"2024-09-16T15:43:42.503416+0000","last_active":"2024-09-16T15:43:43.483277+0000","last_peered":"2024-09-16T15:43:43.483277+0000","last_clean":"2024-09-16T15:43:43.483277+0000","last_became_active":"2024-09-16T15:43:42.502723+0000","last_became_peered":"2024-09-16T15:43:42.502723+0000","last_unstale":"2024-09-16T15:43:43.483277+0000","last_undegraded":"2024-09-16T15:43:43.483277+0000","last_fullsized":"2024-09-16T15:43:43.483277+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:43:41.457712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:43:41.457712+0000","last_clean_scrub_stamp":"2024-09-16T15:43:41.457712+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:50:08.102576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":15,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-16T15:43:43.732861+0000","last_change":"2024-09-16T15:43:41.498342+0000","last_active":"2024-09-16T15:43:43.732861+0000","last_peered":"2024-09-16T15:43:43.732861+0000","last_clean":"2024-09-16T15:43:43.732861+0000","last_became_active":"2024-09-16T15:43:41.495586+0000","last_became_peered":"2024-09-16T15:43:41.495586+0000","last_unstale":"2024-09-16T15:43:43.732861+0000","last_undegraded":"2024-09-16T15:43:43.732861+0000","last_fullsized":"2024-09-16T15:43:43.732861+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:43:40.445567+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:43:40.445567+0000","last_clean_scrub_stamp":"2024-09-16T15:43:40.445567+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:27:40.487399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":13,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-16T15:43:43.733917+0000","last_change":"2024-09-16T15:43:42.506069+0000","last_active":"2024-09-16T15:43:43.733917+0000","last_peered":"2024-09-16T15:43:43.733917+0000","last_clean":"2024-09-16T15:43:43.733917+0000","last_became_active":"2024-09-16T15:43:42.505182+0000","last_became_peered":"2024-09-16T15:43:42.505182+0000","last_unstale":"2024-09-16T15:43:43.733917+0000","last_undegraded":"2024-09-16T15:43:43.733917+0000","last_fullsized":"2024-09-16T15:43:43.733917+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:43:41.457712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:43:41.457712+0000","last_clean_scrub_stamp":"2024-09-16T15:43:41.457712+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:29:43.450124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":15,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-16T15:43:43.483100+0000","last_change":"2024-09-16T15:43:41.505133+0000","last_active":"2024-09-16T15:43:43.483100+0000","last_peered":"2024-09-16T15:43:43.483100+0000","last_clean":"2024-09-16T15:43:43.483100+0000","last_became_active":"2024-09-16T15:43:41.504809+0000","last_became_peered":"2024-09-16T15:43:41.504809+0000","last_unstale":"2024-09-16T15:43:43.483100+0000","last_undegraded":"2024-09-16T15:43:43.483100+0000","last_fullsized":"2024-09-16T15:43:43.483100+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:43:40.445567+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:43:40.445567+0000","last_clean_scrub_stamp":"2024-09-16T15:43:40.445567+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:42:44.460733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":13,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-16T15:43:43.488154+0000","last_change":"2024-09-16T15:43:42.508085+0000","last_active":"2024-09-16T15:43:43.488154+0000","last_peered":"2024-09-16T15:43:43.488154+0000","last_clean":"2024-09-16T15:43:43.488154+0000","last_became_active":"2024-09-16T15:43:42.507287+0000","last_became_peered":"2024-09-16T15:43:42.507287+0000","last_unstale":"2024-09-16T15:43:43.488154+0000","last_undegraded":"2024-09-16T15:43:43.488154+0000","last_fullsized":"2024-09-16T15:43:43.488154+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:43:41.457712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:43:41.457712+0000","last_clean_scrub_stamp":"2024-09-16T15:43:41.457712+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T00:01:42.341879+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":15,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-16T15:43:43.733846+0000","last_change":"2024-09-16T15:43:41.508780+0000","last_active":"2024-09-16T15:43:43.733846+0000","last_peered":"2024-09-16T15:43:43.733846+0000","last_clean":"2024-09-16T15:43:43.733846+0000","last_became_active":"2024-09-16T15:43:41.508020+0000","last_became_peered":"2024-09-16T15:43:41.508020+0000","last_unstale":"2024-09-16T15:43:43.733846+0000","last_undegraded":"2024-09-16T15:43:43.733846+0000","last_fullsized":"2024-09-16T15:43:43.733846+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:43:40.445567+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:43:40.445567+0000","last_clean_scrub_stamp":"2024-09-16T15:43:40.445567+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:02:43.216562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"16'72","reported_seq":134,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-16T15:43:43.486571+0000","last_change":"2024-09-16T15:42:40.881820+0000","last_active":"2024-09-16T15:43:43.486571+0000","last_peered":"2024-09-16T15:43:43.486571+0000","last_clean":"2024-09-16T15:43:43.486571+0000","last_became_active":"2024-09-16T15:42:40.879965+0000","last_became_peered":"2024-09-16T15:42:40.879965+0000","last_unstale":"2024-09-16T15:43:43.486571+0000","last_undegraded":"2024-09-16T15:43:43.486571+0000","last_fullsized":"2024-09-16T15:43:43.486571+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:42:39.728762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:42:39.728762+0000","last_clean_scrub_stamp":"2024-09-16T15:42:39.728762+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:22:18.768932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542161,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27976,"kb_used_data":944,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343864,"statfs":{"total":96636764160,"available":96608116736,"internally_reserved":0,"allocated":966656,"data_stored":733054,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542160,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27388,"kb_used_data":356,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344452,"statfs":{"total":96636764160,"available":96608718848,"internally_reserved":0,"allocated":364544,"data_stored":136912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,4],"upper_bound":32},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":5,"up_from":14,"seq":60129542161,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27396,"kb_used_data":368,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344444,"statfs":{"total":96636764160,"available":96608710656,"internally_reserved":0,"allocated":376832,"data_stored":143268,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542159,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27384,"kb_used_data":356,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344456,"statfs":{"total":96636764160,"available":96608722944,"internally_reserved":0,"allocated":364544,"data_stored":136400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":3,"up_from":14,"seq":60129542160,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27408,"kb_used_data":376,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344432,"statfs":{"total":96636764160,"available":96608698368,"internally_reserved":0,"allocated":385024,"data_stored":144230,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":14,"seq":60129542160,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27392,"kb_used_data":364,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344448,"statfs":{"total":96636764160,"available":96608714752,"internally_reserved":0,"allocated":372736,"data_stored":142666,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":14,"seq":60129542162,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27404,"kb_used_data":376,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344436,"statfs":{"total":96636764160,"available":96608702464,"internally_reserved":0,"allocated":385024,"data_stored":144230,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":14,"seq":60129542162,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27972,"kb_used_data":940,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343868,"statfs":{"total":96636764160,"available":96608120832,"internally_reserved":0,"allocated":962560,"data_stored":732964,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T15:43:46.601 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T15:43:46.759 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:46.752+0000 7ff1e872d640 1 -- 172.21.15.123:0/2979743943 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff1e00726c0 msgr2=0x7ff1e0073ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:43:46.759 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:46.752+0000 7ff1e872d640 1 --2- 172.21.15.123:0/2979743943 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff1e00726c0 0x7ff1e0073ef0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7ff1e00cd650 tx=0x7ff1d001c800 comp rx=0 tx=0).stop 2024-09-16T15:43:46.759 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:46.752+0000 7ff1e872d640 1 -- 172.21.15.123:0/2979743943 shutdown_connections 2024-09-16T15:43:46.759 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:46.752+0000 7ff1e872d640 1 --2- 172.21.15.123:0/2979743943 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff1e00726c0 0x7ff1e0073ef0 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:46.759 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:46.752+0000 7ff1e872d640 1 --2- 172.21.15.123:0/2979743943 >> v2:172.21.15.184:3300/0 conn(0x7ff1e015bca0 0x7ff1e00d0630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:46.759 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:46.752+0000 7ff1e872d640 1 -- 172.21.15.123:0/2979743943 >> 172.21.15.123:0/2979743943 conn(0x7ff1e00cfb10 msgr2=0x7ff1e00cff20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:43:46.759 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:46.752+0000 7ff1e872d640 1 -- 172.21.15.123:0/2979743943 shutdown_connections 2024-09-16T15:43:46.759 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:46.752+0000 7ff1e872d640 1 -- 172.21.15.123:0/2979743943 wait complete. 2024-09-16T15:43:46.760 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:46.756+0000 7ff1e872d640 1 Processor -- start 2024-09-16T15:43:46.760 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:46.756+0000 7ff1e872d640 1 -- start start 2024-09-16T15:43:46.761 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:46.756+0000 7ff1e872d640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7ff1e00726c0 0x7ff1e00c99b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:43:46.761 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:46.756+0000 7ff1e872d640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff1e0074430 0x7ff1e01f90f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:43:46.761 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:46.756+0000 7ff1e872d640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff1e015bca0 0x7ff1e015cff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:43:46.762 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:46.756+0000 7ff1e872d640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7ff1e0168e90 con 0x7ff1e0074430 2024-09-16T15:43:46.762 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:46.756+0000 7ff1e872d640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7ff1e0168d10 con 0x7ff1e015bca0 2024-09-16T15:43:46.762 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:46.756+0000 7ff1e872d640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7ff1e0169010 con 0x7ff1e00726c0 2024-09-16T15:43:46.762 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:46.756+0000 7ff1e5cc8640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff1e0074430 0x7ff1e01f90f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:43:46.762 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:46.756+0000 7ff1e5cc8640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff1e0074430 0x7ff1e01f90f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:41692/0 (socket says 172.21.15.123:41692) 2024-09-16T15:43:46.762 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:46.756+0000 7ff1e5cc8640 1 -- 172.21.15.123:0/1187903145 learned_addr learned my addr 172.21.15.123:0/1187903145 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:43:46.762 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:46.756+0000 7ff1e5cc8640 1 -- 172.21.15.123:0/1187903145 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7ff1e00726c0 msgr2=0x7ff1e00c99b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:43:46.762 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:46.756+0000 7ff1e5cc8640 1 --2- 172.21.15.123:0/1187903145 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7ff1e00726c0 0x7ff1e00c99b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:46.762 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:46.756+0000 7ff1e5cc8640 1 -- 172.21.15.123:0/1187903145 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff1e015bca0 msgr2=0x7ff1e015cff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:43:46.762 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:46.756+0000 7ff1e5cc8640 1 --2- 172.21.15.123:0/1187903145 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff1e015bca0 0x7ff1e015cff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:46.762 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:46.756+0000 7ff1e5cc8640 1 -- 172.21.15.123:0/1187903145 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff1e00c9ef0 con 0x7ff1e0074430 2024-09-16T15:43:46.762 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:46.756+0000 7ff1e5cc8640 1 --2- 172.21.15.123:0/1187903145 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff1e0074430 0x7ff1e01f90f0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7ff1d00b0040 tx=0x7ff1d00a5790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:43:46.763 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:46.756+0000 7ff1d6ffd640 1 -- 172.21.15.123:0/1187903145 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1d00bc070 con 0x7ff1e0074430 2024-09-16T15:43:46.763 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:46.756+0000 7ff1d6ffd640 1 -- 172.21.15.123:0/1187903145 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7ff1d0004450 con 0x7ff1e0074430 2024-09-16T15:43:46.763 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:46.756+0000 7ff1e872d640 1 -- 172.21.15.123:0/1187903145 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff1e01f9a60 con 0x7ff1e0074430 2024-09-16T15:43:46.763 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:46.756+0000 7ff1d6ffd640 1 -- 172.21.15.123:0/1187903145 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1d00b2040 con 0x7ff1e0074430 2024-09-16T15:43:46.763 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:46.756+0000 7ff1e872d640 1 -- 172.21.15.123:0/1187903145 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff1e01543f0 con 0x7ff1e0074430 2024-09-16T15:43:46.764 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:46.756+0000 7ff1e872d640 1 -- 172.21.15.123:0/1187903145 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff1e0073ef0 con 0x7ff1e0074430 2024-09-16T15:43:46.764 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:46.756+0000 7ff1d6ffd640 1 -- 172.21.15.123:0/1187903145 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 5) ==== 97080+0+0 (secure 0 0 0) 0x7ff1d0005280 con 0x7ff1e0074430 2024-09-16T15:43:46.764 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:46.760+0000 7ff1d6ffd640 1 --2- 172.21.15.123:0/1187903145 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7ff1c4072ec0 0x7ff1c4075380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:43:46.764 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:46.760+0000 7ff1d6ffd640 1 -- 172.21.15.123:0/1187903145 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(33..33 src has 1..33) ==== 5709+0+0 (secure 0 0 0) 0x7ff1d012ddf0 con 0x7ff1e0074430 2024-09-16T15:43:46.764 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:46.760+0000 7ff1e64c9640 1 --2- 172.21.15.123:0/1187903145 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7ff1c4072ec0 0x7ff1c4075380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:43:46.764 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:46.760+0000 7ff1e64c9640 1 --2- 172.21.15.123:0/1187903145 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7ff1c4072ec0 0x7ff1c4075380 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7ff1dc009690 tx=0x7ff1dc006c00 comp rx=0 tx=0).ready entity=mgr.4135 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:43:46.768 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:46.764+0000 7ff1d6ffd640 1 -- 172.21.15.123:0/1187903145 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7ff1d00164b0 con 0x7ff1e0074430 2024-09-16T15:43:46.923 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:46.916+0000 7ff1e872d640 1 -- 172.21.15.123:0/1187903145 --> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7ff1e00700e0 con 0x7ff1c4072ec0 2024-09-16T15:43:46.934 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:46.932+0000 7ff1d6ffd640 1 -- 172.21.15.123:0/1187903145 <== mgr.4135 v2:172.21.15.184:6832/3114346423 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+251957 (secure 0 0 0) 0x7ff1e00700e0 con 0x7ff1c4072ec0 2024-09-16T15:43:46.936 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-09-16T15:43:46.937 INFO:teuthology.orchestra.run.smithi123.stderr:dumped all 2024-09-16T15:43:46.938 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:46.932+0000 7ff1e872d640 1 -- 172.21.15.123:0/1187903145 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7ff1c4072ec0 msgr2=0x7ff1c4075380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:43:46.939 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:46.932+0000 7ff1e872d640 1 --2- 172.21.15.123:0/1187903145 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7ff1c4072ec0 0x7ff1c4075380 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7ff1dc009690 tx=0x7ff1dc006c00 comp rx=0 tx=0).stop 2024-09-16T15:43:46.939 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:46.932+0000 7ff1e872d640 1 -- 172.21.15.123:0/1187903145 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff1e0074430 msgr2=0x7ff1e01f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:43:46.939 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:46.932+0000 7ff1e872d640 1 --2- 172.21.15.123:0/1187903145 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff1e0074430 0x7ff1e01f90f0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7ff1d00b0040 tx=0x7ff1d00a5790 comp rx=0 tx=0).stop 2024-09-16T15:43:46.939 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:46.932+0000 7ff1e872d640 1 -- 172.21.15.123:0/1187903145 shutdown_connections 2024-09-16T15:43:46.939 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:46.932+0000 7ff1e872d640 1 --2- 172.21.15.123:0/1187903145 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7ff1c4072ec0 0x7ff1c4075380 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:46.939 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:46.932+0000 7ff1e872d640 1 --2- 172.21.15.123:0/1187903145 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff1e015bca0 0x7ff1e015cff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:46.939 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:46.932+0000 7ff1e872d640 1 --2- 172.21.15.123:0/1187903145 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff1e0074430 0x7ff1e01f90f0 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:46.939 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:46.932+0000 7ff1e872d640 1 --2- 172.21.15.123:0/1187903145 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7ff1e00726c0 0x7ff1e00c99b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:46.939 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:46.932+0000 7ff1e872d640 1 -- 172.21.15.123:0/1187903145 >> 172.21.15.123:0/1187903145 conn(0x7ff1e00cfb10 msgr2=0x7ff1e00ca140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:43:46.939 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:46.932+0000 7ff1e872d640 1 -- 172.21.15.123:0/1187903145 shutdown_connections 2024-09-16T15:43:46.939 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:46.932+0000 7ff1e872d640 1 -- 172.21.15.123:0/1187903145 wait complete. 2024-09-16T15:43:46.953 INFO:teuthology.orchestra.run.smithi123.stdout:{"pg_ready":true,"pg_map":{"version":59,"stamp":"2024-09-16T15:43:45.262906+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":145,"num_write_kb":1642,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":110,"ondisk_log_size":110,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":138,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":220320,"kb_used_data":4080,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754754400,"statfs":{"total":773094113280,"available":772868505600,"internally_reserved":0,"allocated":4177920,"data_stored":2313724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,4],"upper_bound":32},"perf_stat":{"commit_latency_ms":45,"apply_latency_ms":45,"commit_latency_ns":45000000,"apply_latency_ns":45000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.802492"},"pg_stats":[{"pgid":"5.3f","version":"0'0","reported_seq":13,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-16T15:43:43.734474+0000","last_change":"2024-09-16T15:43:42.524818+0000","last_active":"2024-09-16T15:43:43.734474+0000","last_peered":"2024-09-16T15:43:43.734474+0000","last_clean":"2024-09-16T15:43:43.734474+0000","last_became_active":"2024-09-16T15:43:42.523640+0000","last_became_peered":"2024-09-16T15:43:42.523640+0000","last_unstale":"2024-09-16T15:43:43.734474+0000","last_undegraded":"2024-09-16T15:43:43.734474+0000","last_fullsized":"2024-09-16T15:43:43.734474+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:43:41.457712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:43:41.457712+0000","last_clean_scrub_stamp":"2024-09-16T15:43:41.457712+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T02:51:58.757703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.3e","version":"0'0","reported_seq":13,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-16T15:43:43.484595+0000","last_change":"2024-09-16T15:43:42.521224+0000","last_active":"2024-09-16T15:43:43.484595+0000","last_peered":"2024-09-16T15:43:43.484595+0000","last_clean":"2024-09-16T15:43:43.484595+0000","last_became_active":"2024-09-16T15:43:42.513486+0000","last_became_peered":"2024-09-16T15:43:42.513486+0000","last_unstale":"2024-09-16T15:43:43.484595+0000","last_undegraded":"2024-09-16T15:43:43.484595+0000","last_fullsized":"2024-09-16T15:43:43.484595+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:43:41.457712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:43:41.457712+0000","last_clean_scrub_stamp":"2024-09-16T15:43:41.457712+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:59:10.901283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.3d","version":"0'0","reported_seq":13,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-16T15:43:43.487543+0000","last_change":"2024-09-16T15:43:42.527094+0000","last_active":"2024-09-16T15:43:43.487543+0000","last_peered":"2024-09-16T15:43:43.487543+0000","last_clean":"2024-09-16T15:43:43.487543+0000","last_became_active":"2024-09-16T15:43:42.522632+0000","last_became_peered":"2024-09-16T15:43:42.522632+0000","last_unstale":"2024-09-16T15:43:43.487543+0000","last_undegraded":"2024-09-16T15:43:43.487543+0000","last_fullsized":"2024-09-16T15:43:43.487543+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:43:41.457712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:43:41.457712+0000","last_clean_scrub_stamp":"2024-09-16T15:43:41.457712+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T23:14:58.404320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3c","version":"0'0","reported_seq":13,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-16T15:43:43.713868+0000","last_change":"2024-09-16T15:43:42.515323+0000","last_active":"2024-09-16T15:43:43.713868+0000","last_peered":"2024-09-16T15:43:43.713868+0000","last_clean":"2024-09-16T15:43:43.713868+0000","last_became_active":"2024-09-16T15:43:42.513871+0000","last_became_peered":"2024-09-16T15:43:42.513871+0000","last_unstale":"2024-09-16T15:43:43.713868+0000","last_undegraded":"2024-09-16T15:43:43.713868+0000","last_fullsized":"2024-09-16T15:43:43.713868+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:43:41.457712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:43:41.457712+0000","last_clean_scrub_stamp":"2024-09-16T15:43:41.457712+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T03:42:47.543239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3b","version":"0'0","reported_seq":13,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-16T15:43:43.732207+0000","last_change":"2024-09-16T15:43:42.522044+0000","last_active":"2024-09-16T15:43:43.732207+0000","last_peered":"2024-09-16T15:43:43.732207+0000","last_clean":"2024-09-16T15:43:43.732207+0000","last_became_active":"2024-09-16T15:43:42.521770+0000","last_became_peered":"2024-09-16T15:43:42.521770+0000","last_unstale":"2024-09-16T15:43:43.732207+0000","last_undegraded":"2024-09-16T15:43:43.732207+0000","last_fullsized":"2024-09-16T15:43:43.732207+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:43:41.457712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:43:41.457712+0000","last_clean_scrub_stamp":"2024-09-16T15:43:41.457712+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T18:45:40.961435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3a","version":"0'0","reported_seq":13,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-16T15:43:43.483501+0000","last_change":"2024-09-16T15:43:42.538688+0000","last_active":"2024-09-16T15:43:43.483501+0000","last_peered":"2024-09-16T15:43:43.483501+0000","last_clean":"2024-09-16T15:43:43.483501+0000","last_became_active":"2024-09-16T15:43:42.538389+0000","last_became_peered":"2024-09-16T15:43:42.538389+0000","last_unstale":"2024-09-16T15:43:43.483501+0000","last_undegraded":"2024-09-16T15:43:43.483501+0000","last_fullsized":"2024-09-16T15:43:43.483501+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:43:41.457712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:43:41.457712+0000","last_clean_scrub_stamp":"2024-09-16T15:43:41.457712+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T23:05:07.986662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.39","version":"0'0","reported_seq":13,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-16T15:43:43.488389+0000","last_change":"2024-09-16T15:43:42.508618+0000","last_active":"2024-09-16T15:43:43.488389+0000","last_peered":"2024-09-16T15:43:43.488389+0000","last_clean":"2024-09-16T15:43:43.488389+0000","last_became_active":"2024-09-16T15:43:42.507868+0000","last_became_peered":"2024-09-16T15:43:42.507868+0000","last_unstale":"2024-09-16T15:43:43.488389+0000","last_undegraded":"2024-09-16T15:43:43.488389+0000","last_fullsized":"2024-09-16T15:43:43.488389+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:43:41.457712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:43:41.457712+0000","last_clean_scrub_stamp":"2024-09-16T15:43:41.457712+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T23:14:58.404320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.38","version":"0'0","reported_seq":13,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-16T15:43:43.485025+0000","last_change":"2024-09-16T15:43:42.500138+0000","last_active":"2024-09-16T15:43:43.485025+0000","last_peered":"2024-09-16T15:43:43.485025+0000","last_clean":"2024-09-16T15:43:43.485025+0000","last_became_active":"2024-09-16T15:43:42.497303+0000","last_became_peered":"2024-09-16T15:43:42.497303+0000","last_unstale":"2024-09-16T15:43:43.485025+0000","last_undegraded":"2024-09-16T15:43:43.485025+0000","last_fullsized":"2024-09-16T15:43:43.485025+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:43:41.457712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:43:41.457712+0000","last_clean_scrub_stamp":"2024-09-16T15:43:41.457712+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:23:11.473976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.37","version":"0'0","reported_seq":13,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-16T15:43:43.734078+0000","last_change":"2024-09-16T15:43:42.505344+0000","last_active":"2024-09-16T15:43:43.734078+0000","last_peered":"2024-09-16T15:43:43.734078+0000","last_clean":"2024-09-16T15:43:43.734078+0000","last_became_active":"2024-09-16T15:43:42.504620+0000","last_became_peered":"2024-09-16T15:43:42.504620+0000","last_unstale":"2024-09-16T15:43:43.734078+0000","last_undegraded":"2024-09-16T15:43:43.734078+0000","last_fullsized":"2024-09-16T15:43:43.734078+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:43:41.457712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:43:41.457712+0000","last_clean_scrub_stamp":"2024-09-16T15:43:41.457712+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:32:27.596625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.36","version":"0'0","reported_seq":13,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-16T15:43:43.714211+0000","last_change":"2024-09-16T15:43:42.527117+0000","last_active":"2024-09-16T15:43:43.714211+0000","last_peered":"2024-09-16T15:43:43.714211+0000","last_clean":"2024-09-16T15:43:43.714211+0000","last_became_active":"2024-09-16T15:43:42.526834+0000","last_became_peered":"2024-09-16T15:43:42.526834+0000","last_unstale":"2024-09-16T15:43:43.714211+0000","last_undegraded":"2024-09-16T15:43:43.714211+0000","last_fullsized":"2024-09-16T15:43:43.714211+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:43:41.457712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:43:41.457712+0000","last_clean_scrub_stamp":"2024-09-16T15:43:41.457712+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T22:20:03.787407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.35","version":"0'0","reported_seq":13,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-16T15:43:43.488081+0000","last_change":"2024-09-16T15:43:42.501721+0000","last_active":"2024-09-16T15:43:43.488081+0000","last_peered":"2024-09-16T15:43:43.488081+0000","last_clean":"2024-09-16T15:43:43.488081+0000","last_became_active":"2024-09-16T15:43:42.500952+0000","last_became_peered":"2024-09-16T15:43:42.500952+0000","last_unstale":"2024-09-16T15:43:43.488081+0000","last_undegraded":"2024-09-16T15:43:43.488081+0000","last_fullsized":"2024-09-16T15:43:43.488081+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:43:41.457712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:43:41.457712+0000","last_clean_scrub_stamp":"2024-09-16T15:43:41.457712+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:53:25.198570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.34","version":"0'0","reported_seq":13,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-16T15:43:43.486317+0000","last_change":"2024-09-16T15:43:42.532417+0000","last_active":"2024-09-16T15:43:43.486317+0000","last_peered":"2024-09-16T15:43:43.486317+0000","last_clean":"2024-09-16T15:43:43.486317+0000","last_became_active":"2024-09-16T15:43:42.530613+0000","last_became_peered":"2024-09-16T15:43:42.530613+0000","last_unstale":"2024-09-16T15:43:43.486317+0000","last_undegraded":"2024-09-16T15:43:43.486317+0000","last_fullsized":"2024-09-16T15:43:43.486317+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:43:41.457712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:43:41.457712+0000","last_clean_scrub_stamp":"2024-09-16T15:43:41.457712+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T02:35:21.217508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.33","version":"0'0","reported_seq":13,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-16T15:43:43.732236+0000","last_change":"2024-09-16T15:43:42.501898+0000","last_active":"2024-09-16T15:43:43.732236+0000","last_peered":"2024-09-16T15:43:43.732236+0000","last_clean":"2024-09-16T15:43:43.732236+0000","last_became_active":"2024-09-16T15:43:42.501174+0000","last_became_peered":"2024-09-16T15:43:42.501174+0000","last_unstale":"2024-09-16T15:43:43.732236+0000","last_undegraded":"2024-09-16T15:43:43.732236+0000","last_fullsized":"2024-09-16T15:43:43.732236+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:43:41.457712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:43:41.457712+0000","last_clean_scrub_stamp":"2024-09-16T15:43:41.457712+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T02:08:26.700938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.32","version":"0'0","reported_seq":13,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-16T15:43:43.484664+0000","last_change":"2024-09-16T15:43:42.511755+0000","last_active":"2024-09-16T15:43:43.484664+0000","last_peered":"2024-09-16T15:43:43.484664+0000","last_clean":"2024-09-16T15:43:43.484664+0000","last_became_active":"2024-09-16T15:43:42.510868+0000","last_became_peered":"2024-09-16T15:43:42.510868+0000","last_unstale":"2024-09-16T15:43:43.484664+0000","last_undegraded":"2024-09-16T15:43:43.484664+0000","last_fullsized":"2024-09-16T15:43:43.484664+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:43:41.457712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:43:41.457712+0000","last_clean_scrub_stamp":"2024-09-16T15:43:41.457712+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T22:59:08.176784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.31","version":"0'0","reported_seq":13,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-16T15:43:43.484562+0000","last_change":"2024-09-16T15:43:42.522783+0000","last_active":"2024-09-16T15:43:43.484562+0000","last_peered":"2024-09-16T15:43:43.484562+0000","last_clean":"2024-09-16T15:43:43.484562+0000","last_became_active":"2024-09-16T15:43:42.519423+0000","last_became_peered":"2024-09-16T15:43:42.519423+0000","last_unstale":"2024-09-16T15:43:43.484562+0000","last_undegraded":"2024-09-16T15:43:43.484562+0000","last_fullsized":"2024-09-16T15:43:43.484562+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:43:41.457712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:43:41.457712+0000","last_clean_scrub_stamp":"2024-09-16T15:43:41.457712+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:53:05.432956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.30","version":"0'0","reported_seq":13,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-16T15:43:43.486421+0000","last_change":"2024-09-16T15:43:42.525901+0000","last_active":"2024-09-16T15:43:43.486421+0000","last_peered":"2024-09-16T15:43:43.486421+0000","last_clean":"2024-09-16T15:43:43.486421+0000","last_became_active":"2024-09-16T15:43:42.525168+0000","last_became_peered":"2024-09-16T15:43:42.525168+0000","last_unstale":"2024-09-16T15:43:43.486421+0000","last_undegraded":"2024-09-16T15:43:43.486421+0000","last_fullsized":"2024-09-16T15:43:43.486421+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:43:41.457712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:43:41.457712+0000","last_clean_scrub_stamp":"2024-09-16T15:43:41.457712+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T02:35:21.217508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2f","version":"0'0","reported_seq":13,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-16T15:43:43.488346+0000","last_change":"2024-09-16T15:43:42.508558+0000","last_active":"2024-09-16T15:43:43.488346+0000","last_peered":"2024-09-16T15:43:43.488346+0000","last_clean":"2024-09-16T15:43:43.488346+0000","last_became_active":"2024-09-16T15:43:42.507653+0000","last_became_peered":"2024-09-16T15:43:42.507653+0000","last_unstale":"2024-09-16T15:43:43.488346+0000","last_undegraded":"2024-09-16T15:43:43.488346+0000","last_fullsized":"2024-09-16T15:43:43.488346+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:43:41.457712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:43:41.457712+0000","last_clean_scrub_stamp":"2024-09-16T15:43:41.457712+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T03:25:30.813074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2e","version":"0'0","reported_seq":13,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-16T15:43:43.484810+0000","last_change":"2024-09-16T15:43:42.521592+0000","last_active":"2024-09-16T15:43:43.484810+0000","last_peered":"2024-09-16T15:43:43.484810+0000","last_clean":"2024-09-16T15:43:43.484810+0000","last_became_active":"2024-09-16T15:43:42.512173+0000","last_became_peered":"2024-09-16T15:43:42.512173+0000","last_unstale":"2024-09-16T15:43:43.484810+0000","last_undegraded":"2024-09-16T15:43:43.484810+0000","last_fullsized":"2024-09-16T15:43:43.484810+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:43:41.457712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:43:41.457712+0000","last_clean_scrub_stamp":"2024-09-16T15:43:41.457712+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:02:55.957888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2d","version":"0'0","reported_seq":13,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-16T15:43:43.484494+0000","last_change":"2024-09-16T15:43:42.521526+0000","last_active":"2024-09-16T15:43:43.484494+0000","last_peered":"2024-09-16T15:43:43.484494+0000","last_clean":"2024-09-16T15:43:43.484494+0000","last_became_active":"2024-09-16T15:43:42.511688+0000","last_became_peered":"2024-09-16T15:43:42.511688+0000","last_unstale":"2024-09-16T15:43:43.484494+0000","last_undegraded":"2024-09-16T15:43:43.484494+0000","last_fullsized":"2024-09-16T15:43:43.484494+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:43:41.457712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:43:41.457712+0000","last_clean_scrub_stamp":"2024-09-16T15:43:41.457712+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T23:53:41.204959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2c","version":"0'0","reported_seq":13,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-16T15:43:43.732580+0000","last_change":"2024-09-16T15:43:42.524820+0000","last_active":"2024-09-16T15:43:43.732580+0000","last_peered":"2024-09-16T15:43:43.732580+0000","last_clean":"2024-09-16T15:43:43.732580+0000","last_became_active":"2024-09-16T15:43:42.524553+0000","last_became_peered":"2024-09-16T15:43:42.524553+0000","last_unstale":"2024-09-16T15:43:43.732580+0000","last_undegraded":"2024-09-16T15:43:43.732580+0000","last_fullsized":"2024-09-16T15:43:43.732580+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:43:41.457712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:43:41.457712+0000","last_clean_scrub_stamp":"2024-09-16T15:43:41.457712+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T01:59:56.657959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.2b","version":"0'0","reported_seq":13,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-16T15:43:43.734152+0000","last_change":"2024-09-16T15:43:42.515549+0000","last_active":"2024-09-16T15:43:43.734152+0000","last_peered":"2024-09-16T15:43:43.734152+0000","last_clean":"2024-09-16T15:43:43.734152+0000","last_became_active":"2024-09-16T15:43:42.514728+0000","last_became_peered":"2024-09-16T15:43:42.514728+0000","last_unstale":"2024-09-16T15:43:43.734152+0000","last_undegraded":"2024-09-16T15:43:43.734152+0000","last_fullsized":"2024-09-16T15:43:43.734152+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:43:41.457712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:43:41.457712+0000","last_clean_scrub_stamp":"2024-09-16T15:43:41.457712+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T18:25:52.364522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.2a","version":"0'0","reported_seq":13,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-16T15:43:43.733479+0000","last_change":"2024-09-16T15:43:42.508495+0000","last_active":"2024-09-16T15:43:43.733479+0000","last_peered":"2024-09-16T15:43:43.733479+0000","last_clean":"2024-09-16T15:43:43.733479+0000","last_became_active":"2024-09-16T15:43:42.507783+0000","last_became_peered":"2024-09-16T15:43:42.507783+0000","last_unstale":"2024-09-16T15:43:43.733479+0000","last_undegraded":"2024-09-16T15:43:43.733479+0000","last_fullsized":"2024-09-16T15:43:43.733479+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:43:41.457712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:43:41.457712+0000","last_clean_scrub_stamp":"2024-09-16T15:43:41.457712+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:58:40.375405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.29","version":"0'0","reported_seq":13,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-16T15:43:43.615739+0000","last_change":"2024-09-16T15:43:42.498261+0000","last_active":"2024-09-16T15:43:43.615739+0000","last_peered":"2024-09-16T15:43:43.615739+0000","last_clean":"2024-09-16T15:43:43.615739+0000","last_became_active":"2024-09-16T15:43:42.497465+0000","last_became_peered":"2024-09-16T15:43:42.497465+0000","last_unstale":"2024-09-16T15:43:43.615739+0000","last_undegraded":"2024-09-16T15:43:43.615739+0000","last_fullsized":"2024-09-16T15:43:43.615739+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:43:41.457712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:43:41.457712+0000","last_clean_scrub_stamp":"2024-09-16T15:43:41.457712+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:45:14.528268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.28","version":"0'0","reported_seq":13,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-16T15:43:43.484446+0000","last_change":"2024-09-16T15:43:42.518202+0000","last_active":"2024-09-16T15:43:43.484446+0000","last_peered":"2024-09-16T15:43:43.484446+0000","last_clean":"2024-09-16T15:43:43.484446+0000","last_became_active":"2024-09-16T15:43:42.517487+0000","last_became_peered":"2024-09-16T15:43:42.517487+0000","last_unstale":"2024-09-16T15:43:43.484446+0000","last_undegraded":"2024-09-16T15:43:43.484446+0000","last_fullsized":"2024-09-16T15:43:43.484446+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:43:41.457712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:43:41.457712+0000","last_clean_scrub_stamp":"2024-09-16T15:43:41.457712+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:45:15.762373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.27","version":"0'0","reported_seq":13,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-16T15:43:43.714363+0000","last_change":"2024-09-16T15:43:42.502918+0000","last_active":"2024-09-16T15:43:43.714363+0000","last_peered":"2024-09-16T15:43:43.714363+0000","last_clean":"2024-09-16T15:43:43.714363+0000","last_became_active":"2024-09-16T15:43:42.502010+0000","last_became_peered":"2024-09-16T15:43:42.502010+0000","last_unstale":"2024-09-16T15:43:43.714363+0000","last_undegraded":"2024-09-16T15:43:43.714363+0000","last_fullsized":"2024-09-16T15:43:43.714363+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:43:41.457712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:43:41.457712+0000","last_clean_scrub_stamp":"2024-09-16T15:43:41.457712+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:23:11.473976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.26","version":"0'0","reported_seq":13,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-16T15:43:43.484978+0000","last_change":"2024-09-16T15:43:42.511094+0000","last_active":"2024-09-16T15:43:43.484978+0000","last_peered":"2024-09-16T15:43:43.484978+0000","last_clean":"2024-09-16T15:43:43.484978+0000","last_became_active":"2024-09-16T15:43:42.504493+0000","last_became_peered":"2024-09-16T15:43:42.504493+0000","last_unstale":"2024-09-16T15:43:43.484978+0000","last_undegraded":"2024-09-16T15:43:43.484978+0000","last_fullsized":"2024-09-16T15:43:43.484978+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:43:41.457712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:43:41.457712+0000","last_clean_scrub_stamp":"2024-09-16T15:43:41.457712+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:39:56.071308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.25","version":"0'0","reported_seq":13,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-16T15:43:43.714836+0000","last_change":"2024-09-16T15:43:42.515442+0000","last_active":"2024-09-16T15:43:43.714836+0000","last_peered":"2024-09-16T15:43:43.714836+0000","last_clean":"2024-09-16T15:43:43.714836+0000","last_became_active":"2024-09-16T15:43:42.513955+0000","last_became_peered":"2024-09-16T15:43:42.513955+0000","last_unstale":"2024-09-16T15:43:43.714836+0000","last_undegraded":"2024-09-16T15:43:43.714836+0000","last_fullsized":"2024-09-16T15:43:43.714836+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:43:41.457712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:43:41.457712+0000","last_clean_scrub_stamp":"2024-09-16T15:43:41.457712+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:53:05.432956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.24","version":"0'0","reported_seq":13,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-16T15:43:43.486425+0000","last_change":"2024-09-16T15:43:42.523230+0000","last_active":"2024-09-16T15:43:43.486425+0000","last_peered":"2024-09-16T15:43:43.486425+0000","last_clean":"2024-09-16T15:43:43.486425+0000","last_became_active":"2024-09-16T15:43:42.522572+0000","last_became_peered":"2024-09-16T15:43:42.522572+0000","last_unstale":"2024-09-16T15:43:43.486425+0000","last_undegraded":"2024-09-16T15:43:43.486425+0000","last_fullsized":"2024-09-16T15:43:43.486425+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:43:41.457712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:43:41.457712+0000","last_clean_scrub_stamp":"2024-09-16T15:43:41.457712+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T23:39:18.158705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.23","version":"0'0","reported_seq":13,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-16T15:43:43.732806+0000","last_change":"2024-09-16T15:43:42.511612+0000","last_active":"2024-09-16T15:43:43.732806+0000","last_peered":"2024-09-16T15:43:43.732806+0000","last_clean":"2024-09-16T15:43:43.732806+0000","last_became_active":"2024-09-16T15:43:42.510806+0000","last_became_peered":"2024-09-16T15:43:42.510806+0000","last_unstale":"2024-09-16T15:43:43.732806+0000","last_undegraded":"2024-09-16T15:43:43.732806+0000","last_fullsized":"2024-09-16T15:43:43.732806+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:43:41.457712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:43:41.457712+0000","last_clean_scrub_stamp":"2024-09-16T15:43:41.457712+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T00:02:15.747411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.22","version":"0'0","reported_seq":13,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-16T15:43:43.487605+0000","last_change":"2024-09-16T15:43:42.514120+0000","last_active":"2024-09-16T15:43:43.487605+0000","last_peered":"2024-09-16T15:43:43.487605+0000","last_clean":"2024-09-16T15:43:43.487605+0000","last_became_active":"2024-09-16T15:43:42.509320+0000","last_became_peered":"2024-09-16T15:43:42.509320+0000","last_unstale":"2024-09-16T15:43:43.487605+0000","last_undegraded":"2024-09-16T15:43:43.487605+0000","last_fullsized":"2024-09-16T15:43:43.487605+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:43:41.457712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:43:41.457712+0000","last_clean_scrub_stamp":"2024-09-16T15:43:41.457712+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T23:39:18.158705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.21","version":"0'0","reported_seq":13,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-16T15:43:43.732237+0000","last_change":"2024-09-16T15:43:42.493963+0000","last_active":"2024-09-16T15:43:43.732237+0000","last_peered":"2024-09-16T15:43:43.732237+0000","last_clean":"2024-09-16T15:43:43.732237+0000","last_became_active":"2024-09-16T15:43:42.490203+0000","last_became_peered":"2024-09-16T15:43:42.490203+0000","last_unstale":"2024-09-16T15:43:43.732237+0000","last_undegraded":"2024-09-16T15:43:43.732237+0000","last_fullsized":"2024-09-16T15:43:43.732237+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:43:41.457712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:43:41.457712+0000","last_clean_scrub_stamp":"2024-09-16T15:43:41.457712+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T00:47:37.753066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.20","version":"0'0","reported_seq":13,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-16T15:43:43.485378+0000","last_change":"2024-09-16T15:43:42.501631+0000","last_active":"2024-09-16T15:43:43.485378+0000","last_peered":"2024-09-16T15:43:43.485378+0000","last_clean":"2024-09-16T15:43:43.485378+0000","last_became_active":"2024-09-16T15:43:42.500847+0000","last_became_peered":"2024-09-16T15:43:42.500847+0000","last_unstale":"2024-09-16T15:43:43.485378+0000","last_undegraded":"2024-09-16T15:43:43.485378+0000","last_fullsized":"2024-09-16T15:43:43.485378+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:43:41.457712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:43:41.457712+0000","last_clean_scrub_stamp":"2024-09-16T15:43:41.457712+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:53:05.432956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":13,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-16T15:43:43.615985+0000","last_change":"2024-09-16T15:43:42.502901+0000","last_active":"2024-09-16T15:43:43.615985+0000","last_peered":"2024-09-16T15:43:43.615985+0000","last_clean":"2024-09-16T15:43:43.615985+0000","last_became_active":"2024-09-16T15:43:42.502162+0000","last_became_peered":"2024-09-16T15:43:42.502162+0000","last_unstale":"2024-09-16T15:43:43.615985+0000","last_undegraded":"2024-09-16T15:43:43.615985+0000","last_fullsized":"2024-09-16T15:43:43.615985+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:43:41.457712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:43:41.457712+0000","last_clean_scrub_stamp":"2024-09-16T15:43:41.457712+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:25:27.174564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1f","version":"32'4","reported_seq":19,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-16T15:43:43.734645+0000","last_change":"2024-09-16T15:43:41.512316+0000","last_active":"2024-09-16T15:43:43.734645+0000","last_peered":"2024-09-16T15:43:43.734645+0000","last_clean":"2024-09-16T15:43:43.734645+0000","last_became_active":"2024-09-16T15:43:41.511559+0000","last_became_peered":"2024-09-16T15:43:41.511559+0000","last_unstale":"2024-09-16T15:43:43.734645+0000","last_undegraded":"2024-09-16T15:43:43.734645+0000","last_fullsized":"2024-09-16T15:43:43.734645+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:43:40.445567+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:43:40.445567+0000","last_clean_scrub_stamp":"2024-09-16T15:43:40.445567+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T00:47:54.458065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1f","version":"0'0","reported_seq":13,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-16T15:43:43.734537+0000","last_change":"2024-09-16T15:43:42.496030+0000","last_active":"2024-09-16T15:43:43.734537+0000","last_peered":"2024-09-16T15:43:43.734537+0000","last_clean":"2024-09-16T15:43:43.734537+0000","last_became_active":"2024-09-16T15:43:42.495281+0000","last_became_peered":"2024-09-16T15:43:42.495281+0000","last_unstale":"2024-09-16T15:43:43.734537+0000","last_undegraded":"2024-09-16T15:43:43.734537+0000","last_fullsized":"2024-09-16T15:43:43.734537+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:43:41.457712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:43:41.457712+0000","last_clean_scrub_stamp":"2024-09-16T15:43:41.457712+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T00:25:11.394386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1e","version":"32'1","reported_seq":16,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-16T15:43:43.615849+0000","last_change":"2024-09-16T15:43:41.503087+0000","last_active":"2024-09-16T15:43:43.615849+0000","last_peered":"2024-09-16T15:43:43.615849+0000","last_clean":"2024-09-16T15:43:43.615849+0000","last_became_active":"2024-09-16T15:43:41.502803+0000","last_became_peered":"2024-09-16T15:43:41.502803+0000","last_unstale":"2024-09-16T15:43:43.615849+0000","last_undegraded":"2024-09-16T15:43:43.615849+0000","last_fullsized":"2024-09-16T15:43:43.615849+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:43:40.445567+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:43:40.445567+0000","last_clean_scrub_stamp":"2024-09-16T15:43:40.445567+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T18:45:12.079394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":13,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-16T15:43:43.733308+0000","last_change":"2024-09-16T15:43:42.513223+0000","last_active":"2024-09-16T15:43:43.733308+0000","last_peered":"2024-09-16T15:43:43.733308+0000","last_clean":"2024-09-16T15:43:43.733308+0000","last_became_active":"2024-09-16T15:43:42.512493+0000","last_became_peered":"2024-09-16T15:43:42.512493+0000","last_unstale":"2024-09-16T15:43:43.733308+0000","last_undegraded":"2024-09-16T15:43:43.733308+0000","last_fullsized":"2024-09-16T15:43:43.733308+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:43:41.457712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:43:41.457712+0000","last_clean_scrub_stamp":"2024-09-16T15:43:41.457712+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T18:20:42.609654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"32'2","reported_seq":17,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-16T15:43:43.484322+0000","last_change":"2024-09-16T15:43:41.498815+0000","last_active":"2024-09-16T15:43:43.484322+0000","last_peered":"2024-09-16T15:43:43.484322+0000","last_clean":"2024-09-16T15:43:43.484322+0000","last_became_active":"2024-09-16T15:43:41.498187+0000","last_became_peered":"2024-09-16T15:43:41.498187+0000","last_unstale":"2024-09-16T15:43:43.484322+0000","last_undegraded":"2024-09-16T15:43:43.484322+0000","last_fullsized":"2024-09-16T15:43:43.484322+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:43:40.445567+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:43:40.445567+0000","last_clean_scrub_stamp":"2024-09-16T15:43:40.445567+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T23:05:21.486769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":13,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-16T15:43:43.714773+0000","last_change":"2024-09-16T15:43:42.509444+0000","last_active":"2024-09-16T15:43:43.714773+0000","last_peered":"2024-09-16T15:43:43.714773+0000","last_clean":"2024-09-16T15:43:43.714773+0000","last_became_active":"2024-09-16T15:43:42.508510+0000","last_became_peered":"2024-09-16T15:43:42.508510+0000","last_unstale":"2024-09-16T15:43:43.714773+0000","last_undegraded":"2024-09-16T15:43:43.714773+0000","last_fullsized":"2024-09-16T15:43:43.714773+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:43:41.457712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:43:41.457712+0000","last_clean_scrub_stamp":"2024-09-16T15:43:41.457712+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:45:15.762373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1c","version":"32'2","reported_seq":17,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-16T15:43:43.732327+0000","last_change":"2024-09-16T15:43:41.497006+0000","last_active":"2024-09-16T15:43:43.732327+0000","last_peered":"2024-09-16T15:43:43.732327+0000","last_clean":"2024-09-16T15:43:43.732327+0000","last_became_active":"2024-09-16T15:43:41.496696+0000","last_became_peered":"2024-09-16T15:43:41.496696+0000","last_unstale":"2024-09-16T15:43:43.732327+0000","last_undegraded":"2024-09-16T15:43:43.732327+0000","last_fullsized":"2024-09-16T15:43:43.732327+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:43:40.445567+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:43:40.445567+0000","last_clean_scrub_stamp":"2024-09-16T15:43:40.445567+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:08:07.602741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":13,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-16T15:43:43.487705+0000","last_change":"2024-09-16T15:43:42.515296+0000","last_active":"2024-09-16T15:43:43.487705+0000","last_peered":"2024-09-16T15:43:43.487705+0000","last_clean":"2024-09-16T15:43:43.487705+0000","last_became_active":"2024-09-16T15:43:42.514616+0000","last_became_peered":"2024-09-16T15:43:42.514616+0000","last_unstale":"2024-09-16T15:43:43.487705+0000","last_undegraded":"2024-09-16T15:43:43.487705+0000","last_fullsized":"2024-09-16T15:43:43.487705+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:43:41.457712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:43:41.457712+0000","last_clean_scrub_stamp":"2024-09-16T15:43:41.457712+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T01:57:17.824602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1b","version":"32'1","reported_seq":16,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-16T15:43:43.733143+0000","last_change":"2024-09-16T15:43:41.508469+0000","last_active":"2024-09-16T15:43:43.733143+0000","last_peered":"2024-09-16T15:43:43.733143+0000","last_clean":"2024-09-16T15:43:43.733143+0000","last_became_active":"2024-09-16T15:43:41.507710+0000","last_became_peered":"2024-09-16T15:43:41.507710+0000","last_unstale":"2024-09-16T15:43:43.733143+0000","last_undegraded":"2024-09-16T15:43:43.733143+0000","last_fullsized":"2024-09-16T15:43:43.733143+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:43:40.445567+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:43:40.445567+0000","last_clean_scrub_stamp":"2024-09-16T15:43:40.445567+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T00:47:54.458065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":13,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-16T15:43:43.485942+0000","last_change":"2024-09-16T15:43:42.528041+0000","last_active":"2024-09-16T15:43:43.485942+0000","last_peered":"2024-09-16T15:43:43.485942+0000","last_clean":"2024-09-16T15:43:43.485942+0000","last_became_active":"2024-09-16T15:43:42.524019+0000","last_became_peered":"2024-09-16T15:43:42.524019+0000","last_unstale":"2024-09-16T15:43:43.485942+0000","last_undegraded":"2024-09-16T15:43:43.485942+0000","last_fullsized":"2024-09-16T15:43:43.485942+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:43:41.457712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:43:41.457712+0000","last_clean_scrub_stamp":"2024-09-16T15:43:41.457712+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T23:09:25.039240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"32'1","reported_seq":16,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-16T15:43:43.733350+0000","last_change":"2024-09-16T15:43:41.505606+0000","last_active":"2024-09-16T15:43:43.733350+0000","last_peered":"2024-09-16T15:43:43.733350+0000","last_clean":"2024-09-16T15:43:43.733350+0000","last_became_active":"2024-09-16T15:43:41.501328+0000","last_became_peered":"2024-09-16T15:43:41.501328+0000","last_unstale":"2024-09-16T15:43:43.733350+0000","last_undegraded":"2024-09-16T15:43:43.733350+0000","last_fullsized":"2024-09-16T15:43:43.733350+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:43:40.445567+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:43:40.445567+0000","last_clean_scrub_stamp":"2024-09-16T15:43:40.445567+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:02:43.216562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":13,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-16T15:43:43.732398+0000","last_change":"2024-09-16T15:43:42.506325+0000","last_active":"2024-09-16T15:43:43.732398+0000","last_peered":"2024-09-16T15:43:43.732398+0000","last_clean":"2024-09-16T15:43:43.732398+0000","last_became_active":"2024-09-16T15:43:42.505588+0000","last_became_peered":"2024-09-16T15:43:42.505588+0000","last_unstale":"2024-09-16T15:43:43.732398+0000","last_undegraded":"2024-09-16T15:43:43.732398+0000","last_fullsized":"2024-09-16T15:43:43.732398+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:43:41.457712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:43:41.457712+0000","last_clean_scrub_stamp":"2024-09-16T15:43:41.457712+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T01:45:43.408312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":15,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-16T15:43:43.483256+0000","last_change":"2024-09-16T15:43:41.495106+0000","last_active":"2024-09-16T15:43:43.483256+0000","last_peered":"2024-09-16T15:43:43.483256+0000","last_clean":"2024-09-16T15:43:43.483256+0000","last_became_active":"2024-09-16T15:43:41.494152+0000","last_became_peered":"2024-09-16T15:43:41.494152+0000","last_unstale":"2024-09-16T15:43:43.483256+0000","last_undegraded":"2024-09-16T15:43:43.483256+0000","last_fullsized":"2024-09-16T15:43:43.483256+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:43:40.445567+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:43:40.445567+0000","last_clean_scrub_stamp":"2024-09-16T15:43:40.445567+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T22:41:37.724364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":13,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-16T15:43:43.713544+0000","last_change":"2024-09-16T15:43:42.524416+0000","last_active":"2024-09-16T15:43:43.713544+0000","last_peered":"2024-09-16T15:43:43.713544+0000","last_clean":"2024-09-16T15:43:43.713544+0000","last_became_active":"2024-09-16T15:43:42.523604+0000","last_became_peered":"2024-09-16T15:43:42.523604+0000","last_unstale":"2024-09-16T15:43:43.713544+0000","last_undegraded":"2024-09-16T15:43:43.713544+0000","last_fullsized":"2024-09-16T15:43:43.713544+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:43:41.457712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:43:41.457712+0000","last_clean_scrub_stamp":"2024-09-16T15:43:41.457712+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T18:57:49.412121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":15,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-16T15:43:43.732702+0000","last_change":"2024-09-16T15:43:41.505626+0000","last_active":"2024-09-16T15:43:43.732702+0000","last_peered":"2024-09-16T15:43:43.732702+0000","last_clean":"2024-09-16T15:43:43.732702+0000","last_became_active":"2024-09-16T15:43:41.500940+0000","last_became_peered":"2024-09-16T15:43:41.500940+0000","last_unstale":"2024-09-16T15:43:43.732702+0000","last_undegraded":"2024-09-16T15:43:43.732702+0000","last_fullsized":"2024-09-16T15:43:43.732702+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:43:40.445567+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:43:40.445567+0000","last_clean_scrub_stamp":"2024-09-16T15:43:40.445567+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T23:23:26.003518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":13,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-16T15:43:43.487182+0000","last_change":"2024-09-16T15:43:42.528814+0000","last_active":"2024-09-16T15:43:43.487182+0000","last_peered":"2024-09-16T15:43:43.487182+0000","last_clean":"2024-09-16T15:43:43.487182+0000","last_became_active":"2024-09-16T15:43:42.527052+0000","last_became_peered":"2024-09-16T15:43:42.527052+0000","last_unstale":"2024-09-16T15:43:43.487182+0000","last_undegraded":"2024-09-16T15:43:43.487182+0000","last_fullsized":"2024-09-16T15:43:43.487182+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:43:41.457712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:43:41.457712+0000","last_clean_scrub_stamp":"2024-09-16T15:43:41.457712+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T03:25:30.813074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":15,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-16T15:43:43.484626+0000","last_change":"2024-09-16T15:43:41.505547+0000","last_active":"2024-09-16T15:43:43.484626+0000","last_peered":"2024-09-16T15:43:43.484626+0000","last_clean":"2024-09-16T15:43:43.484626+0000","last_became_active":"2024-09-16T15:43:41.505285+0000","last_became_peered":"2024-09-16T15:43:41.505285+0000","last_unstale":"2024-09-16T15:43:43.484626+0000","last_undegraded":"2024-09-16T15:43:43.484626+0000","last_fullsized":"2024-09-16T15:43:43.484626+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:43:40.445567+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:43:40.445567+0000","last_clean_scrub_stamp":"2024-09-16T15:43:40.445567+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:29:42.437978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":13,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-16T15:43:43.484781+0000","last_change":"2024-09-16T15:43:42.521329+0000","last_active":"2024-09-16T15:43:43.484781+0000","last_peered":"2024-09-16T15:43:43.484781+0000","last_clean":"2024-09-16T15:43:43.484781+0000","last_became_active":"2024-09-16T15:43:42.512536+0000","last_became_peered":"2024-09-16T15:43:42.512536+0000","last_unstale":"2024-09-16T15:43:43.484781+0000","last_undegraded":"2024-09-16T15:43:43.484781+0000","last_fullsized":"2024-09-16T15:43:43.484781+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:43:41.457712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:43:41.457712+0000","last_clean_scrub_stamp":"2024-09-16T15:43:41.457712+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T01:12:22.003154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":15,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-16T15:43:43.616069+0000","last_change":"2024-09-16T15:43:41.499795+0000","last_active":"2024-09-16T15:43:43.616069+0000","last_peered":"2024-09-16T15:43:43.616069+0000","last_clean":"2024-09-16T15:43:43.616069+0000","last_became_active":"2024-09-16T15:43:41.499475+0000","last_became_peered":"2024-09-16T15:43:41.499475+0000","last_unstale":"2024-09-16T15:43:43.616069+0000","last_undegraded":"2024-09-16T15:43:43.616069+0000","last_fullsized":"2024-09-16T15:43:43.616069+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:43:40.445567+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:43:40.445567+0000","last_clean_scrub_stamp":"2024-09-16T15:43:40.445567+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:08:18.770146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"0'0","reported_seq":13,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-16T15:43:43.484086+0000","last_change":"2024-09-16T15:43:42.509837+0000","last_active":"2024-09-16T15:43:43.484086+0000","last_peered":"2024-09-16T15:43:43.484086+0000","last_clean":"2024-09-16T15:43:43.484086+0000","last_became_active":"2024-09-16T15:43:42.505473+0000","last_became_peered":"2024-09-16T15:43:42.505473+0000","last_unstale":"2024-09-16T15:43:43.484086+0000","last_undegraded":"2024-09-16T15:43:43.484086+0000","last_fullsized":"2024-09-16T15:43:43.484086+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:43:41.457712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:43:41.457712+0000","last_clean_scrub_stamp":"2024-09-16T15:43:41.457712+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:50:12.328928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.15","version":"32'3","reported_seq":18,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-16T15:43:43.487542+0000","last_change":"2024-09-16T15:43:41.506168+0000","last_active":"2024-09-16T15:43:43.487542+0000","last_peered":"2024-09-16T15:43:43.487542+0000","last_clean":"2024-09-16T15:43:43.487542+0000","last_became_active":"2024-09-16T15:43:41.505392+0000","last_became_peered":"2024-09-16T15:43:41.505392+0000","last_unstale":"2024-09-16T15:43:43.487542+0000","last_undegraded":"2024-09-16T15:43:43.487542+0000","last_fullsized":"2024-09-16T15:43:43.487542+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:43:40.445567+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:43:40.445567+0000","last_clean_scrub_stamp":"2024-09-16T15:43:40.445567+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T18:41:12.648868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"0'0","reported_seq":13,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-16T15:43:43.486818+0000","last_change":"2024-09-16T15:43:42.532302+0000","last_active":"2024-09-16T15:43:43.486818+0000","last_peered":"2024-09-16T15:43:43.486818+0000","last_clean":"2024-09-16T15:43:43.486818+0000","last_became_active":"2024-09-16T15:43:42.523267+0000","last_became_peered":"2024-09-16T15:43:42.523267+0000","last_unstale":"2024-09-16T15:43:43.486818+0000","last_undegraded":"2024-09-16T15:43:43.486818+0000","last_fullsized":"2024-09-16T15:43:43.486818+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:43:41.457712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:43:41.457712+0000","last_clean_scrub_stamp":"2024-09-16T15:43:41.457712+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:53:25.198570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"32'6","reported_seq":21,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-16T15:43:43.484198+0000","last_change":"2024-09-16T15:43:41.502074+0000","last_active":"2024-09-16T15:43:43.484198+0000","last_peered":"2024-09-16T15:43:43.484198+0000","last_clean":"2024-09-16T15:43:43.484198+0000","last_became_active":"2024-09-16T15:43:41.501797+0000","last_became_peered":"2024-09-16T15:43:41.501797+0000","last_unstale":"2024-09-16T15:43:43.484198+0000","last_undegraded":"2024-09-16T15:43:43.484198+0000","last_fullsized":"2024-09-16T15:43:43.484198+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:43:40.445567+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:43:40.445567+0000","last_clean_scrub_stamp":"2024-09-16T15:43:40.445567+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T00:25:10.382240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":13,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-16T15:43:43.713645+0000","last_change":"2024-09-16T15:43:42.524990+0000","last_active":"2024-09-16T15:43:43.713645+0000","last_peered":"2024-09-16T15:43:43.713645+0000","last_clean":"2024-09-16T15:43:43.713645+0000","last_became_active":"2024-09-16T15:43:42.523907+0000","last_became_peered":"2024-09-16T15:43:42.523907+0000","last_unstale":"2024-09-16T15:43:43.713645+0000","last_undegraded":"2024-09-16T15:43:43.713645+0000","last_fullsized":"2024-09-16T15:43:43.713645+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:43:41.457712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:43:41.457712+0000","last_clean_scrub_stamp":"2024-09-16T15:43:41.457712+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:23:06.229761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"32'2","reported_seq":17,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-16T15:43:43.733437+0000","last_change":"2024-09-16T15:43:41.507979+0000","last_active":"2024-09-16T15:43:43.733437+0000","last_peered":"2024-09-16T15:43:43.733437+0000","last_clean":"2024-09-16T15:43:43.733437+0000","last_became_active":"2024-09-16T15:43:41.507251+0000","last_became_peered":"2024-09-16T15:43:41.507251+0000","last_unstale":"2024-09-16T15:43:43.733437+0000","last_undegraded":"2024-09-16T15:43:43.733437+0000","last_fullsized":"2024-09-16T15:43:43.733437+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:43:40.445567+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:43:40.445567+0000","last_clean_scrub_stamp":"2024-09-16T15:43:40.445567+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:10:58.646118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":13,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-16T15:43:43.484054+0000","last_change":"2024-09-16T15:43:42.503800+0000","last_active":"2024-09-16T15:43:43.484054+0000","last_peered":"2024-09-16T15:43:43.484054+0000","last_clean":"2024-09-16T15:43:43.484054+0000","last_became_active":"2024-09-16T15:43:42.503081+0000","last_became_peered":"2024-09-16T15:43:42.503081+0000","last_unstale":"2024-09-16T15:43:43.484054+0000","last_undegraded":"2024-09-16T15:43:43.484054+0000","last_fullsized":"2024-09-16T15:43:43.484054+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:43:41.457712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:43:41.457712+0000","last_clean_scrub_stamp":"2024-09-16T15:43:41.457712+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T03:02:24.003658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":15,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-16T15:43:43.713484+0000","last_change":"2024-09-16T15:43:41.501377+0000","last_active":"2024-09-16T15:43:43.713484+0000","last_peered":"2024-09-16T15:43:43.713484+0000","last_clean":"2024-09-16T15:43:43.713484+0000","last_became_active":"2024-09-16T15:43:41.500885+0000","last_became_peered":"2024-09-16T15:43:41.500885+0000","last_unstale":"2024-09-16T15:43:43.713484+0000","last_undegraded":"2024-09-16T15:43:43.713484+0000","last_fullsized":"2024-09-16T15:43:43.713484+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:43:40.445567+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:43:40.445567+0000","last_clean_scrub_stamp":"2024-09-16T15:43:40.445567+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:08:18.770146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":13,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-16T15:43:43.485914+0000","last_change":"2024-09-16T15:43:42.515158+0000","last_active":"2024-09-16T15:43:43.485914+0000","last_peered":"2024-09-16T15:43:43.485914+0000","last_clean":"2024-09-16T15:43:43.485914+0000","last_became_active":"2024-09-16T15:43:42.502582+0000","last_became_peered":"2024-09-16T15:43:42.502582+0000","last_unstale":"2024-09-16T15:43:43.485914+0000","last_undegraded":"2024-09-16T15:43:43.485914+0000","last_fullsized":"2024-09-16T15:43:43.485914+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:43:41.457712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:43:41.457712+0000","last_clean_scrub_stamp":"2024-09-16T15:43:41.457712+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T23:09:25.039240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":15,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-16T15:43:43.483338+0000","last_change":"2024-09-16T15:43:41.510375+0000","last_active":"2024-09-16T15:43:43.483338+0000","last_peered":"2024-09-16T15:43:43.483338+0000","last_clean":"2024-09-16T15:43:43.483338+0000","last_became_active":"2024-09-16T15:43:41.510044+0000","last_became_peered":"2024-09-16T15:43:41.510044+0000","last_unstale":"2024-09-16T15:43:43.483338+0000","last_undegraded":"2024-09-16T15:43:43.483338+0000","last_fullsized":"2024-09-16T15:43:43.483338+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:43:40.445567+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:43:40.445567+0000","last_clean_scrub_stamp":"2024-09-16T15:43:40.445567+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T02:51:57.745557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":13,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-16T15:43:43.732293+0000","last_change":"2024-09-16T15:43:42.496964+0000","last_active":"2024-09-16T15:43:43.732293+0000","last_peered":"2024-09-16T15:43:43.732293+0000","last_clean":"2024-09-16T15:43:43.732293+0000","last_became_active":"2024-09-16T15:43:42.496284+0000","last_became_peered":"2024-09-16T15:43:42.496284+0000","last_unstale":"2024-09-16T15:43:43.732293+0000","last_undegraded":"2024-09-16T15:43:43.732293+0000","last_fullsized":"2024-09-16T15:43:43.732293+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:43:41.457712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:43:41.457712+0000","last_clean_scrub_stamp":"2024-09-16T15:43:41.457712+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T22:41:38.736510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.10","version":"32'8","reported_seq":23,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-16T15:43:43.483002+0000","last_change":"2024-09-16T15:43:41.500850+0000","last_active":"2024-09-16T15:43:43.483002+0000","last_peered":"2024-09-16T15:43:43.483002+0000","last_clean":"2024-09-16T15:43:43.483002+0000","last_became_active":"2024-09-16T15:43:41.500181+0000","last_became_peered":"2024-09-16T15:43:41.500181+0000","last_unstale":"2024-09-16T15:43:43.483002+0000","last_undegraded":"2024-09-16T15:43:43.483002+0000","last_fullsized":"2024-09-16T15:43:43.483002+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:43:40.445567+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:43:40.445567+0000","last_clean_scrub_stamp":"2024-09-16T15:43:40.445567+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T00:47:36.740921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"0'0","reported_seq":13,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-16T15:43:43.733240+0000","last_change":"2024-09-16T15:43:42.520243+0000","last_active":"2024-09-16T15:43:43.733240+0000","last_peered":"2024-09-16T15:43:43.733240+0000","last_clean":"2024-09-16T15:43:43.733240+0000","last_became_active":"2024-09-16T15:43:42.519538+0000","last_became_peered":"2024-09-16T15:43:42.519538+0000","last_unstale":"2024-09-16T15:43:43.733240+0000","last_undegraded":"2024-09-16T15:43:43.733240+0000","last_fullsized":"2024-09-16T15:43:43.733240+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:43:41.457712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:43:41.457712+0000","last_clean_scrub_stamp":"2024-09-16T15:43:41.457712+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T03:21:18.518406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.f","version":"32'1","reported_seq":16,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-16T15:43:43.714497+0000","last_change":"2024-09-16T15:43:41.505805+0000","last_active":"2024-09-16T15:43:43.714497+0000","last_peered":"2024-09-16T15:43:43.714497+0000","last_clean":"2024-09-16T15:43:43.714497+0000","last_became_active":"2024-09-16T15:43:41.505538+0000","last_became_peered":"2024-09-16T15:43:41.505538+0000","last_unstale":"2024-09-16T15:43:43.714497+0000","last_undegraded":"2024-09-16T15:43:43.714497+0000","last_fullsized":"2024-09-16T15:43:43.714497+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:43:40.445567+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:43:40.445567+0000","last_clean_scrub_stamp":"2024-09-16T15:43:40.445567+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T18:46:12.199768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":13,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-16T15:43:43.486566+0000","last_change":"2024-09-16T15:43:42.523995+0000","last_active":"2024-09-16T15:43:43.486566+0000","last_peered":"2024-09-16T15:43:43.486566+0000","last_clean":"2024-09-16T15:43:43.486566+0000","last_became_active":"2024-09-16T15:43:42.523379+0000","last_became_peered":"2024-09-16T15:43:42.523379+0000","last_unstale":"2024-09-16T15:43:43.486566+0000","last_undegraded":"2024-09-16T15:43:43.486566+0000","last_fullsized":"2024-09-16T15:43:43.486566+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:43:41.457712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:43:41.457712+0000","last_clean_scrub_stamp":"2024-09-16T15:43:41.457712+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T01:57:17.824602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.e","version":"32'2","reported_seq":17,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-16T15:43:43.733519+0000","last_change":"2024-09-16T15:43:41.507958+0000","last_active":"2024-09-16T15:43:43.733519+0000","last_peered":"2024-09-16T15:43:43.733519+0000","last_clean":"2024-09-16T15:43:43.733519+0000","last_became_active":"2024-09-16T15:43:41.507221+0000","last_became_peered":"2024-09-16T15:43:41.507221+0000","last_unstale":"2024-09-16T15:43:43.733519+0000","last_undegraded":"2024-09-16T15:43:43.733519+0000","last_fullsized":"2024-09-16T15:43:43.733519+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:43:40.445567+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:43:40.445567+0000","last_clean_scrub_stamp":"2024-09-16T15:43:40.445567+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T18:01:15.103311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":13,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-16T15:43:43.713969+0000","last_change":"2024-09-16T15:43:42.514591+0000","last_active":"2024-09-16T15:43:43.713969+0000","last_peered":"2024-09-16T15:43:43.713969+0000","last_clean":"2024-09-16T15:43:43.713969+0000","last_became_active":"2024-09-16T15:43:42.512955+0000","last_became_peered":"2024-09-16T15:43:42.512955+0000","last_unstale":"2024-09-16T15:43:43.713969+0000","last_undegraded":"2024-09-16T15:43:43.713969+0000","last_fullsized":"2024-09-16T15:43:43.713969+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:43:41.457712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:43:41.457712+0000","last_clean_scrub_stamp":"2024-09-16T15:43:41.457712+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T22:30:41.098254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.d","version":"32'1","reported_seq":16,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-16T15:43:43.733814+0000","last_change":"2024-09-16T15:43:41.498188+0000","last_active":"2024-09-16T15:43:43.733814+0000","last_peered":"2024-09-16T15:43:43.733814+0000","last_clean":"2024-09-16T15:43:43.733814+0000","last_became_active":"2024-09-16T15:43:41.497233+0000","last_became_peered":"2024-09-16T15:43:41.497233+0000","last_unstale":"2024-09-16T15:43:43.733814+0000","last_undegraded":"2024-09-16T15:43:43.733814+0000","last_fullsized":"2024-09-16T15:43:43.733814+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:43:40.445567+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:43:40.445567+0000","last_clean_scrub_stamp":"2024-09-16T15:43:40.445567+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T00:53:49.420412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"0'0","reported_seq":13,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-16T15:43:43.732411+0000","last_change":"2024-09-16T15:43:42.504728+0000","last_active":"2024-09-16T15:43:43.732411+0000","last_peered":"2024-09-16T15:43:43.732411+0000","last_clean":"2024-09-16T15:43:43.732411+0000","last_became_active":"2024-09-16T15:43:42.503979+0000","last_became_peered":"2024-09-16T15:43:42.503979+0000","last_unstale":"2024-09-16T15:43:43.732411+0000","last_undegraded":"2024-09-16T15:43:43.732411+0000","last_fullsized":"2024-09-16T15:43:43.732411+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:43:41.457712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:43:41.457712+0000","last_clean_scrub_stamp":"2024-09-16T15:43:41.457712+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T18:46:13.211914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":15,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-16T15:43:43.733613+0000","last_change":"2024-09-16T15:43:41.506932+0000","last_active":"2024-09-16T15:43:43.733613+0000","last_peered":"2024-09-16T15:43:43.733613+0000","last_clean":"2024-09-16T15:43:43.733613+0000","last_became_active":"2024-09-16T15:43:41.506136+0000","last_became_peered":"2024-09-16T15:43:41.506136+0000","last_unstale":"2024-09-16T15:43:43.733613+0000","last_undegraded":"2024-09-16T15:43:43.733613+0000","last_fullsized":"2024-09-16T15:43:43.733613+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:43:40.445567+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:43:40.445567+0000","last_clean_scrub_stamp":"2024-09-16T15:43:40.445567+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T02:37:07.989248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":13,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-16T15:43:43.732100+0000","last_change":"2024-09-16T15:43:42.496091+0000","last_active":"2024-09-16T15:43:43.732100+0000","last_peered":"2024-09-16T15:43:43.732100+0000","last_clean":"2024-09-16T15:43:43.732100+0000","last_became_active":"2024-09-16T15:43:42.495370+0000","last_became_peered":"2024-09-16T15:43:42.495370+0000","last_unstale":"2024-09-16T15:43:43.732100+0000","last_undegraded":"2024-09-16T15:43:43.732100+0000","last_fullsized":"2024-09-16T15:43:43.732100+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:43:41.457712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:43:41.457712+0000","last_clean_scrub_stamp":"2024-09-16T15:43:41.457712+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T18:02:56.530946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.b","version":"32'2","reported_seq":17,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-16T15:43:43.616117+0000","last_change":"2024-09-16T15:43:41.495721+0000","last_active":"2024-09-16T15:43:43.616117+0000","last_peered":"2024-09-16T15:43:43.616117+0000","last_clean":"2024-09-16T15:43:43.616117+0000","last_became_active":"2024-09-16T15:43:41.495439+0000","last_became_peered":"2024-09-16T15:43:41.495439+0000","last_unstale":"2024-09-16T15:43:43.616117+0000","last_undegraded":"2024-09-16T15:43:43.616117+0000","last_fullsized":"2024-09-16T15:43:43.616117+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:43:40.445567+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:43:40.445567+0000","last_clean_scrub_stamp":"2024-09-16T15:43:40.445567+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T18:46:12.199768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":13,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-16T15:43:43.732759+0000","last_change":"2024-09-16T15:43:42.506436+0000","last_active":"2024-09-16T15:43:43.732759+0000","last_peered":"2024-09-16T15:43:43.732759+0000","last_clean":"2024-09-16T15:43:43.732759+0000","last_became_active":"2024-09-16T15:43:42.505694+0000","last_became_peered":"2024-09-16T15:43:42.505694+0000","last_unstale":"2024-09-16T15:43:43.732759+0000","last_undegraded":"2024-09-16T15:43:43.732759+0000","last_fullsized":"2024-09-16T15:43:43.732759+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:43:41.457712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:43:41.457712+0000","last_clean_scrub_stamp":"2024-09-16T15:43:41.457712+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T18:45:13.091540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":15,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-16T15:43:43.733766+0000","last_change":"2024-09-16T15:43:41.499830+0000","last_active":"2024-09-16T15:43:43.733766+0000","last_peered":"2024-09-16T15:43:43.733766+0000","last_clean":"2024-09-16T15:43:43.733766+0000","last_became_active":"2024-09-16T15:43:41.498784+0000","last_became_peered":"2024-09-16T15:43:41.498784+0000","last_unstale":"2024-09-16T15:43:43.733766+0000","last_undegraded":"2024-09-16T15:43:43.733766+0000","last_fullsized":"2024-09-16T15:43:43.733766+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:43:40.445567+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:43:40.445567+0000","last_clean_scrub_stamp":"2024-09-16T15:43:40.445567+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T00:53:49.420412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":13,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-16T15:43:43.732087+0000","last_change":"2024-09-16T15:43:42.499649+0000","last_active":"2024-09-16T15:43:43.732087+0000","last_peered":"2024-09-16T15:43:43.732087+0000","last_clean":"2024-09-16T15:43:43.732087+0000","last_became_active":"2024-09-16T15:43:42.498576+0000","last_became_peered":"2024-09-16T15:43:42.498576+0000","last_unstale":"2024-09-16T15:43:43.732087+0000","last_undegraded":"2024-09-16T15:43:43.732087+0000","last_fullsized":"2024-09-16T15:43:43.732087+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:43:41.457712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:43:41.457712+0000","last_clean_scrub_stamp":"2024-09-16T15:43:41.457712+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T22:44:48.708241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":15,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-16T15:43:43.732581+0000","last_change":"2024-09-16T15:43:41.513191+0000","last_active":"2024-09-16T15:43:43.732581+0000","last_peered":"2024-09-16T15:43:43.732581+0000","last_clean":"2024-09-16T15:43:43.732581+0000","last_became_active":"2024-09-16T15:43:41.501991+0000","last_became_peered":"2024-09-16T15:43:41.501991+0000","last_unstale":"2024-09-16T15:43:43.732581+0000","last_undegraded":"2024-09-16T15:43:43.732581+0000","last_fullsized":"2024-09-16T15:43:43.732581+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:43:40.445567+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:43:40.445567+0000","last_clean_scrub_stamp":"2024-09-16T15:43:40.445567+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T03:11:54.723233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":13,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-16T15:43:43.485156+0000","last_change":"2024-09-16T15:43:42.508427+0000","last_active":"2024-09-16T15:43:43.485156+0000","last_peered":"2024-09-16T15:43:43.485156+0000","last_clean":"2024-09-16T15:43:43.485156+0000","last_became_active":"2024-09-16T15:43:42.507685+0000","last_became_peered":"2024-09-16T15:43:42.507685+0000","last_unstale":"2024-09-16T15:43:43.485156+0000","last_undegraded":"2024-09-16T15:43:43.485156+0000","last_fullsized":"2024-09-16T15:43:43.485156+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:43:41.457712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:43:41.457712+0000","last_clean_scrub_stamp":"2024-09-16T15:43:41.457712+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:45:15.762373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":15,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-16T15:43:43.484093+0000","last_change":"2024-09-16T15:43:41.507727+0000","last_active":"2024-09-16T15:43:43.484093+0000","last_peered":"2024-09-16T15:43:43.484093+0000","last_clean":"2024-09-16T15:43:43.484093+0000","last_became_active":"2024-09-16T15:43:41.507048+0000","last_became_peered":"2024-09-16T15:43:41.507048+0000","last_unstale":"2024-09-16T15:43:43.484093+0000","last_undegraded":"2024-09-16T15:43:43.484093+0000","last_fullsized":"2024-09-16T15:43:43.484093+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:43:40.445567+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:43:40.445567+0000","last_clean_scrub_stamp":"2024-09-16T15:43:40.445567+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:30:00.160372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":13,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-16T15:43:43.732577+0000","last_change":"2024-09-16T15:43:42.518408+0000","last_active":"2024-09-16T15:43:43.732577+0000","last_peered":"2024-09-16T15:43:43.732577+0000","last_clean":"2024-09-16T15:43:43.732577+0000","last_became_active":"2024-09-16T15:43:42.517727+0000","last_became_peered":"2024-09-16T15:43:42.517727+0000","last_unstale":"2024-09-16T15:43:43.732577+0000","last_undegraded":"2024-09-16T15:43:43.732577+0000","last_fullsized":"2024-09-16T15:43:43.732577+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:43:41.457712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:43:41.457712+0000","last_clean_scrub_stamp":"2024-09-16T15:43:41.457712+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:14:01.389644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"32'2","reported_seq":17,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-16T15:43:43.714520+0000","last_change":"2024-09-16T15:43:41.496121+0000","last_active":"2024-09-16T15:43:43.714520+0000","last_peered":"2024-09-16T15:43:43.714520+0000","last_clean":"2024-09-16T15:43:43.714520+0000","last_became_active":"2024-09-16T15:43:41.495417+0000","last_became_peered":"2024-09-16T15:43:41.495417+0000","last_unstale":"2024-09-16T15:43:43.714520+0000","last_undegraded":"2024-09-16T15:43:43.714520+0000","last_fullsized":"2024-09-16T15:43:43.714520+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:43:40.445567+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:43:40.445567+0000","last_clean_scrub_stamp":"2024-09-16T15:43:40.445567+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T01:59:55.645813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":13,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-16T15:43:43.487054+0000","last_change":"2024-09-16T15:43:42.527574+0000","last_active":"2024-09-16T15:43:43.487054+0000","last_peered":"2024-09-16T15:43:43.487054+0000","last_clean":"2024-09-16T15:43:43.487054+0000","last_became_active":"2024-09-16T15:43:42.526817+0000","last_became_peered":"2024-09-16T15:43:42.526817+0000","last_unstale":"2024-09-16T15:43:43.487054+0000","last_undegraded":"2024-09-16T15:43:43.487054+0000","last_fullsized":"2024-09-16T15:43:43.487054+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:43:41.457712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:43:41.457712+0000","last_clean_scrub_stamp":"2024-09-16T15:43:41.457712+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T00:01:42.341879+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":15,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-16T15:43:43.616270+0000","last_change":"2024-09-16T15:43:41.495632+0000","last_active":"2024-09-16T15:43:43.616270+0000","last_peered":"2024-09-16T15:43:43.616270+0000","last_clean":"2024-09-16T15:43:43.616270+0000","last_became_active":"2024-09-16T15:43:41.495343+0000","last_became_peered":"2024-09-16T15:43:41.495343+0000","last_unstale":"2024-09-16T15:43:43.616270+0000","last_undegraded":"2024-09-16T15:43:43.616270+0000","last_fullsized":"2024-09-16T15:43:43.616270+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:43:40.445567+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:43:40.445567+0000","last_clean_scrub_stamp":"2024-09-16T15:43:40.445567+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T01:30:19.395237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":13,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-16T15:43:43.615859+0000","last_change":"2024-09-16T15:43:42.498071+0000","last_active":"2024-09-16T15:43:43.615859+0000","last_peered":"2024-09-16T15:43:43.615859+0000","last_clean":"2024-09-16T15:43:43.615859+0000","last_became_active":"2024-09-16T15:43:42.497308+0000","last_became_peered":"2024-09-16T15:43:42.497308+0000","last_unstale":"2024-09-16T15:43:43.615859+0000","last_undegraded":"2024-09-16T15:43:43.615859+0000","last_fullsized":"2024-09-16T15:43:43.615859+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:43:41.457712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:43:41.457712+0000","last_clean_scrub_stamp":"2024-09-16T15:43:41.457712+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T03:04:13.235513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":15,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-16T15:43:43.714262+0000","last_change":"2024-09-16T15:43:41.498126+0000","last_active":"2024-09-16T15:43:43.714262+0000","last_peered":"2024-09-16T15:43:43.714262+0000","last_clean":"2024-09-16T15:43:43.714262+0000","last_became_active":"2024-09-16T15:43:41.497610+0000","last_became_peered":"2024-09-16T15:43:41.497610+0000","last_unstale":"2024-09-16T15:43:43.714262+0000","last_undegraded":"2024-09-16T15:43:43.714262+0000","last_fullsized":"2024-09-16T15:43:43.714262+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:43:40.445567+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:43:40.445567+0000","last_clean_scrub_stamp":"2024-09-16T15:43:40.445567+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T01:30:19.395237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":13,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-16T15:43:43.734102+0000","last_change":"2024-09-16T15:43:42.497130+0000","last_active":"2024-09-16T15:43:43.734102+0000","last_peered":"2024-09-16T15:43:43.734102+0000","last_clean":"2024-09-16T15:43:43.734102+0000","last_became_active":"2024-09-16T15:43:42.496289+0000","last_became_peered":"2024-09-16T15:43:42.496289+0000","last_unstale":"2024-09-16T15:43:43.734102+0000","last_undegraded":"2024-09-16T15:43:43.734102+0000","last_fullsized":"2024-09-16T15:43:43.734102+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:43:41.457712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:43:41.457712+0000","last_clean_scrub_stamp":"2024-09-16T15:43:41.457712+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:42:45.472879+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":15,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-16T15:43:43.616383+0000","last_change":"2024-09-16T15:43:41.492043+0000","last_active":"2024-09-16T15:43:43.616383+0000","last_peered":"2024-09-16T15:43:43.616383+0000","last_clean":"2024-09-16T15:43:43.616383+0000","last_became_active":"2024-09-16T15:43:41.491503+0000","last_became_peered":"2024-09-16T15:43:41.491503+0000","last_unstale":"2024-09-16T15:43:43.616383+0000","last_undegraded":"2024-09-16T15:43:43.616383+0000","last_fullsized":"2024-09-16T15:43:43.616383+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:43:40.445567+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:43:40.445567+0000","last_clean_scrub_stamp":"2024-09-16T15:43:40.445567+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T01:59:55.645813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":13,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-16T15:43:43.616244+0000","last_change":"2024-09-16T15:43:42.499650+0000","last_active":"2024-09-16T15:43:43.616244+0000","last_peered":"2024-09-16T15:43:43.616244+0000","last_clean":"2024-09-16T15:43:43.616244+0000","last_became_active":"2024-09-16T15:43:42.496108+0000","last_became_peered":"2024-09-16T15:43:42.496108+0000","last_unstale":"2024-09-16T15:43:43.616244+0000","last_undegraded":"2024-09-16T15:43:43.616244+0000","last_fullsized":"2024-09-16T15:43:43.616244+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:43:41.457712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:43:41.457712+0000","last_clean_scrub_stamp":"2024-09-16T15:43:41.457712+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:33:04.168915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":15,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-16T15:43:43.713727+0000","last_change":"2024-09-16T15:43:41.494711+0000","last_active":"2024-09-16T15:43:43.713727+0000","last_peered":"2024-09-16T15:43:43.713727+0000","last_clean":"2024-09-16T15:43:43.713727+0000","last_became_active":"2024-09-16T15:43:41.494462+0000","last_became_peered":"2024-09-16T15:43:41.494462+0000","last_unstale":"2024-09-16T15:43:43.713727+0000","last_undegraded":"2024-09-16T15:43:43.713727+0000","last_fullsized":"2024-09-16T15:43:43.713727+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:43:40.445567+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:43:40.445567+0000","last_clean_scrub_stamp":"2024-09-16T15:43:40.445567+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T18:45:12.079394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":13,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-16T15:43:43.483277+0000","last_change":"2024-09-16T15:43:42.503416+0000","last_active":"2024-09-16T15:43:43.483277+0000","last_peered":"2024-09-16T15:43:43.483277+0000","last_clean":"2024-09-16T15:43:43.483277+0000","last_became_active":"2024-09-16T15:43:42.502723+0000","last_became_peered":"2024-09-16T15:43:42.502723+0000","last_unstale":"2024-09-16T15:43:43.483277+0000","last_undegraded":"2024-09-16T15:43:43.483277+0000","last_fullsized":"2024-09-16T15:43:43.483277+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:43:41.457712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:43:41.457712+0000","last_clean_scrub_stamp":"2024-09-16T15:43:41.457712+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:50:08.102576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":15,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-16T15:43:43.732861+0000","last_change":"2024-09-16T15:43:41.498342+0000","last_active":"2024-09-16T15:43:43.732861+0000","last_peered":"2024-09-16T15:43:43.732861+0000","last_clean":"2024-09-16T15:43:43.732861+0000","last_became_active":"2024-09-16T15:43:41.495586+0000","last_became_peered":"2024-09-16T15:43:41.495586+0000","last_unstale":"2024-09-16T15:43:43.732861+0000","last_undegraded":"2024-09-16T15:43:43.732861+0000","last_fullsized":"2024-09-16T15:43:43.732861+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:43:40.445567+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:43:40.445567+0000","last_clean_scrub_stamp":"2024-09-16T15:43:40.445567+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:27:40.487399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":13,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-16T15:43:43.733917+0000","last_change":"2024-09-16T15:43:42.506069+0000","last_active":"2024-09-16T15:43:43.733917+0000","last_peered":"2024-09-16T15:43:43.733917+0000","last_clean":"2024-09-16T15:43:43.733917+0000","last_became_active":"2024-09-16T15:43:42.505182+0000","last_became_peered":"2024-09-16T15:43:42.505182+0000","last_unstale":"2024-09-16T15:43:43.733917+0000","last_undegraded":"2024-09-16T15:43:43.733917+0000","last_fullsized":"2024-09-16T15:43:43.733917+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:43:41.457712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:43:41.457712+0000","last_clean_scrub_stamp":"2024-09-16T15:43:41.457712+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:29:43.450124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":15,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-16T15:43:43.483100+0000","last_change":"2024-09-16T15:43:41.505133+0000","last_active":"2024-09-16T15:43:43.483100+0000","last_peered":"2024-09-16T15:43:43.483100+0000","last_clean":"2024-09-16T15:43:43.483100+0000","last_became_active":"2024-09-16T15:43:41.504809+0000","last_became_peered":"2024-09-16T15:43:41.504809+0000","last_unstale":"2024-09-16T15:43:43.483100+0000","last_undegraded":"2024-09-16T15:43:43.483100+0000","last_fullsized":"2024-09-16T15:43:43.483100+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:43:40.445567+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:43:40.445567+0000","last_clean_scrub_stamp":"2024-09-16T15:43:40.445567+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:42:44.460733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":13,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-16T15:43:43.488154+0000","last_change":"2024-09-16T15:43:42.508085+0000","last_active":"2024-09-16T15:43:43.488154+0000","last_peered":"2024-09-16T15:43:43.488154+0000","last_clean":"2024-09-16T15:43:43.488154+0000","last_became_active":"2024-09-16T15:43:42.507287+0000","last_became_peered":"2024-09-16T15:43:42.507287+0000","last_unstale":"2024-09-16T15:43:43.488154+0000","last_undegraded":"2024-09-16T15:43:43.488154+0000","last_fullsized":"2024-09-16T15:43:43.488154+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:43:41.457712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:43:41.457712+0000","last_clean_scrub_stamp":"2024-09-16T15:43:41.457712+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T00:01:42.341879+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":15,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-16T15:43:43.733846+0000","last_change":"2024-09-16T15:43:41.508780+0000","last_active":"2024-09-16T15:43:43.733846+0000","last_peered":"2024-09-16T15:43:43.733846+0000","last_clean":"2024-09-16T15:43:43.733846+0000","last_became_active":"2024-09-16T15:43:41.508020+0000","last_became_peered":"2024-09-16T15:43:41.508020+0000","last_unstale":"2024-09-16T15:43:43.733846+0000","last_undegraded":"2024-09-16T15:43:43.733846+0000","last_fullsized":"2024-09-16T15:43:43.733846+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:43:40.445567+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:43:40.445567+0000","last_clean_scrub_stamp":"2024-09-16T15:43:40.445567+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:02:43.216562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"16'72","reported_seq":134,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-16T15:43:43.486571+0000","last_change":"2024-09-16T15:42:40.881820+0000","last_active":"2024-09-16T15:43:43.486571+0000","last_peered":"2024-09-16T15:43:43.486571+0000","last_clean":"2024-09-16T15:43:43.486571+0000","last_became_active":"2024-09-16T15:42:40.879965+0000","last_became_peered":"2024-09-16T15:42:40.879965+0000","last_unstale":"2024-09-16T15:43:43.486571+0000","last_undegraded":"2024-09-16T15:43:43.486571+0000","last_fullsized":"2024-09-16T15:43:43.486571+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:42:39.728762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:42:39.728762+0000","last_clean_scrub_stamp":"2024-09-16T15:42:39.728762+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:22:18.768932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542161,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27976,"kb_used_data":944,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343864,"statfs":{"total":96636764160,"available":96608116736,"internally_reserved":0,"allocated":966656,"data_stored":733054,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542160,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27388,"kb_used_data":356,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344452,"statfs":{"total":96636764160,"available":96608718848,"internally_reserved":0,"allocated":364544,"data_stored":136912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,4],"upper_bound":32},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":5,"up_from":14,"seq":60129542161,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27396,"kb_used_data":368,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344444,"statfs":{"total":96636764160,"available":96608710656,"internally_reserved":0,"allocated":376832,"data_stored":143268,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542159,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27384,"kb_used_data":356,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344456,"statfs":{"total":96636764160,"available":96608722944,"internally_reserved":0,"allocated":364544,"data_stored":136400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":3,"up_from":14,"seq":60129542160,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27408,"kb_used_data":376,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344432,"statfs":{"total":96636764160,"available":96608698368,"internally_reserved":0,"allocated":385024,"data_stored":144230,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":14,"seq":60129542160,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27392,"kb_used_data":364,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344448,"statfs":{"total":96636764160,"available":96608714752,"internally_reserved":0,"allocated":372736,"data_stored":142666,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":14,"seq":60129542162,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27404,"kb_used_data":376,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344436,"statfs":{"total":96636764160,"available":96608702464,"internally_reserved":0,"allocated":385024,"data_stored":144230,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":14,"seq":60129542162,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27972,"kb_used_data":940,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343868,"statfs":{"total":96636764160,"available":96608120832,"internally_reserved":0,"allocated":962560,"data_stored":732964,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T15:43:46.956 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-09-16T15:43:46.956 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-09-16T15:43:47.134 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:47.128+0000 7f0d89721640 1 -- 172.21.15.123:0/3095095396 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f0d840b28e0 msgr2=0x7f0d840b1610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:43:47.134 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:47.128+0000 7f0d89721640 1 --2- 172.21.15.123:0/3095095396 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f0d840b28e0 0x7f0d840b1610 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f0d7800a980 tx=0x7f0d7801daf0 comp rx=0 tx=0).stop 2024-09-16T15:43:47.134 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:47.128+0000 7f0d89721640 1 -- 172.21.15.123:0/3095095396 shutdown_connections 2024-09-16T15:43:47.134 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:47.128+0000 7f0d89721640 1 --2- 172.21.15.123:0/3095095396 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f0d840b0a50 0x7f0d840741c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:47.134 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:47.128+0000 7f0d89721640 1 --2- 172.21.15.123:0/3095095396 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f0d840b28e0 0x7f0d840b1610 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:47.135 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:47.128+0000 7f0d89721640 1 -- 172.21.15.123:0/3095095396 >> 172.21.15.123:0/3095095396 conn(0x7f0d840b1dc0 msgr2=0x7f0d840b21d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:43:47.135 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:47.128+0000 7f0d89721640 1 -- 172.21.15.123:0/3095095396 shutdown_connections 2024-09-16T15:43:47.135 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:47.128+0000 7f0d89721640 1 -- 172.21.15.123:0/3095095396 wait complete. 2024-09-16T15:43:47.135 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:47.132+0000 7f0d89721640 1 Processor -- start 2024-09-16T15:43:47.135 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:47.132+0000 7f0d89721640 1 -- start start 2024-09-16T15:43:47.137 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:47.132+0000 7f0d89721640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0d840b0a50 0x7f0d84099910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:43:47.137 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:47.132+0000 7f0d89721640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f0d840b28e0 0x7f0d840992e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:43:47.137 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:47.132+0000 7f0d89721640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f0d8415d9f0 0x7f0d840ad0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:43:47.137 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:47.132+0000 7f0d89721640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f0d84074ba0 con 0x7f0d840b0a50 2024-09-16T15:43:47.137 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:47.132+0000 7f0d89721640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f0d84074a20 con 0x7f0d840b28e0 2024-09-16T15:43:47.137 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:47.132+0000 7f0d89721640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f0d84074d20 con 0x7f0d8415d9f0 2024-09-16T15:43:47.138 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:47.132+0000 7f0d82ffd640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0d840b0a50 0x7f0d84099910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:43:47.138 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:47.132+0000 7f0d82ffd640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0d840b0a50 0x7f0d84099910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:41702/0 (socket says 172.21.15.123:41702) 2024-09-16T15:43:47.138 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:47.132+0000 7f0d82ffd640 1 -- 172.21.15.123:0/3403469802 learned_addr learned my addr 172.21.15.123:0/3403469802 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:43:47.138 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:47.132+0000 7f0d837fe640 1 --2- 172.21.15.123:0/3403469802 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f0d8415d9f0 0x7f0d840ad0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:43:47.138 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:47.132+0000 7f0d827fc640 1 --2- 172.21.15.123:0/3403469802 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f0d840b28e0 0x7f0d840992e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:43:47.138 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:47.132+0000 7f0d82ffd640 1 -- 172.21.15.123:0/3403469802 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f0d8415d9f0 msgr2=0x7f0d840ad0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:43:47.138 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:47.132+0000 7f0d82ffd640 1 --2- 172.21.15.123:0/3403469802 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f0d8415d9f0 0x7f0d840ad0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:47.138 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:47.132+0000 7f0d82ffd640 1 -- 172.21.15.123:0/3403469802 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f0d840b28e0 msgr2=0x7f0d840992e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:43:47.138 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:47.132+0000 7f0d82ffd640 1 --2- 172.21.15.123:0/3403469802 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f0d840b28e0 0x7f0d840992e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:47.138 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:47.132+0000 7f0d82ffd640 1 -- 172.21.15.123:0/3403469802 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0d840adf00 con 0x7f0d840b0a50 2024-09-16T15:43:47.138 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:47.132+0000 7f0d827fc640 1 --2- 172.21.15.123:0/3403469802 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f0d840b28e0 0x7f0d840992e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T15:43:47.138 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:47.132+0000 7f0d82ffd640 1 --2- 172.21.15.123:0/3403469802 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0d840b0a50 0x7f0d84099910 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f0d7400c430 tx=0x7f0d7400c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:43:47.138 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:47.132+0000 7f0d6b7fe640 1 -- 172.21.15.123:0/3403469802 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d74011790 con 0x7f0d840b0a50 2024-09-16T15:43:47.139 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:47.132+0000 7f0d6b7fe640 1 -- 172.21.15.123:0/3403469802 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f0d74017070 con 0x7f0d840b0a50 2024-09-16T15:43:47.139 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:47.132+0000 7f0d6b7fe640 1 -- 172.21.15.123:0/3403469802 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d74011cc0 con 0x7f0d840b0a50 2024-09-16T15:43:47.139 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:47.132+0000 7f0d89721640 1 -- 172.21.15.123:0/3403469802 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0d8406efa0 con 0x7f0d840b0a50 2024-09-16T15:43:47.139 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:47.132+0000 7f0d89721640 1 -- 172.21.15.123:0/3403469802 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0d8406f560 con 0x7f0d840b0a50 2024-09-16T15:43:47.139 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:47.132+0000 7f0d89721640 1 -- 172.21.15.123:0/3403469802 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0d840741c0 con 0x7f0d840b0a50 2024-09-16T15:43:47.139 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:47.132+0000 7f0d6b7fe640 1 -- 172.21.15.123:0/3403469802 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 5) ==== 97080+0+0 (secure 0 0 0) 0x7f0d7401c460 con 0x7f0d840b0a50 2024-09-16T15:43:47.140 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:47.136+0000 7f0d6b7fe640 1 --2- 172.21.15.123:0/3403469802 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f0d70072ef0 0x7f0d700753b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:43:47.140 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:47.136+0000 7f0d6b7fe640 1 -- 172.21.15.123:0/3403469802 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(33..33 src has 1..33) ==== 5709+0+0 (secure 0 0 0) 0x7f0d740944c0 con 0x7f0d840b0a50 2024-09-16T15:43:47.140 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:47.136+0000 7f0d827fc640 1 --2- 172.21.15.123:0/3403469802 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f0d70072ef0 0x7f0d700753b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:43:47.140 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:47.136+0000 7f0d827fc640 1 --2- 172.21.15.123:0/3403469802 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f0d70072ef0 0x7f0d700753b0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f0d780b0000 tx=0x7f0d7801c390 comp rx=0 tx=0).ready entity=mgr.4135 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:43:47.142 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:47.136+0000 7f0d6b7fe640 1 -- 172.21.15.123:0/3403469802 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f0d7405f880 con 0x7f0d840b0a50 2024-09-16T15:43:47.362 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:47.356+0000 7f0d89721640 1 -- 172.21.15.123:0/3403469802 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) -- 0x7f0d8415cbf0 con 0x7f0d840b0a50 2024-09-16T15:43:47.366 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:47.360+0000 7f0d6b7fe640 1 -- 172.21.15.123:0/3403469802 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v17) ==== 190+0+147 (secure 0 0 0) 0x7f0d7405f180 con 0x7f0d840b0a50 2024-09-16T15:43:47.366 INFO:teuthology.orchestra.run.smithi123.stdout:[client.0] 2024-09-16T15:43:47.366 INFO:teuthology.orchestra.run.smithi123.stdout: key = AQC1Uehm7GDTAhAAHWWVIyjVVnTeeozBR8DfFA== 2024-09-16T15:43:47.367 INFO:teuthology.orchestra.run.smithi123.stdout: caps mds = "allow" 2024-09-16T15:43:47.367 INFO:teuthology.orchestra.run.smithi123.stdout: caps mon = "allow r" 2024-09-16T15:43:47.367 INFO:teuthology.orchestra.run.smithi123.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-09-16T15:43:47.367 INFO:teuthology.orchestra.run.smithi123.stderr:updated caps for client.0 2024-09-16T15:43:47.369 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:47.364+0000 7f0d89721640 1 -- 172.21.15.123:0/3403469802 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f0d70072ef0 msgr2=0x7f0d700753b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:43:47.369 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:47.364+0000 7f0d89721640 1 --2- 172.21.15.123:0/3403469802 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f0d70072ef0 0x7f0d700753b0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f0d780b0000 tx=0x7f0d7801c390 comp rx=0 tx=0).stop 2024-09-16T15:43:47.369 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:47.364+0000 7f0d89721640 1 -- 172.21.15.123:0/3403469802 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0d840b0a50 msgr2=0x7f0d84099910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:43:47.370 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:47.364+0000 7f0d89721640 1 --2- 172.21.15.123:0/3403469802 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0d840b0a50 0x7f0d84099910 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f0d7400c430 tx=0x7f0d7400c8f0 comp rx=0 tx=0).stop 2024-09-16T15:43:47.370 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:47.364+0000 7f0d89721640 1 -- 172.21.15.123:0/3403469802 shutdown_connections 2024-09-16T15:43:47.370 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:47.364+0000 7f0d89721640 1 --2- 172.21.15.123:0/3403469802 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f0d70072ef0 0x7f0d700753b0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:47.370 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:47.364+0000 7f0d89721640 1 --2- 172.21.15.123:0/3403469802 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f0d8415d9f0 0x7f0d840ad0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:47.370 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:47.364+0000 7f0d89721640 1 --2- 172.21.15.123:0/3403469802 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f0d840b28e0 0x7f0d840992e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:47.370 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:47.364+0000 7f0d89721640 1 --2- 172.21.15.123:0/3403469802 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0d840b0a50 0x7f0d84099910 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:47.370 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:47.364+0000 7f0d89721640 1 -- 172.21.15.123:0/3403469802 >> 172.21.15.123:0/3403469802 conn(0x7f0d840b1dc0 msgr2=0x7f0d84157be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:43:47.370 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:47.364+0000 7f0d89721640 1 -- 172.21.15.123:0/3403469802 shutdown_connections 2024-09-16T15:43:47.370 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:47.364+0000 7f0d89721640 1 -- 172.21.15.123:0/3403469802 wait complete. 2024-09-16T15:43:47.384 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-16T15:43:47.497 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:47.492+0000 7f4bb90ba640 1 -- 172.21.15.123:0/3440425149 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4b9c003a40 msgr2=0x7f4b9c003e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:43:47.497 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:47.492+0000 7f4bb90ba640 1 --2- 172.21.15.123:0/3440425149 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4b9c003a40 0x7f4b9c003e60 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f4ba400bb10 tx=0x7f4ba401cde0 comp rx=0 tx=0).stop 2024-09-16T15:43:47.497 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:47.492+0000 7f4bb359e640 1 -- 172.21.15.123:0/3440425149 reap_dead start 2024-09-16T15:43:47.497 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:47.492+0000 7f4bb90ba640 1 -- 172.21.15.123:0/3440425149 shutdown_connections 2024-09-16T15:43:47.497 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:47.492+0000 7f4bb90ba640 1 -- 172.21.15.123:0/3440425149 >> 172.21.15.123:0/3440425149 conn(0x7f4bb40d0b40 msgr2=0x7f4bb40d0f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:43:47.497 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:47.492+0000 7f4bb90ba640 1 -- 172.21.15.123:0/3440425149 shutdown_connections 2024-09-16T15:43:47.497 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:47.492+0000 7f4bb90ba640 1 -- 172.21.15.123:0/3440425149 wait complete. 2024-09-16T15:43:47.498 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:47.492+0000 7f4bb90ba640 1 Processor -- start 2024-09-16T15:43:47.498 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:47.492+0000 7f4bb90ba640 1 -- start start 2024-09-16T15:43:47.500 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:47.492+0000 7f4bb90ba640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4b9c003a40 0x7f4bb414dd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:43:47.500 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:47.492+0000 7f4bb90ba640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f4bb414e4c0 0x7f4bb40b5410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:43:47.500 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:47.492+0000 7f4bb90ba640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4bb415abd0 0x7f4bb415b050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:43:47.500 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:47.492+0000 7f4bb90ba640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f4bb4168330 con 0x7f4b9c003a40 2024-09-16T15:43:47.500 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:47.492+0000 7f4bb90ba640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f4bb40d0920 con 0x7f4bb415abd0 2024-09-16T15:43:47.500 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:47.492+0000 7f4bb90ba640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f4bb41684b0 con 0x7f4bb414e4c0 2024-09-16T15:43:47.500 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:47.492+0000 7f4bb2d9d640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4b9c003a40 0x7f4bb414dd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:43:47.500 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:47.492+0000 7f4bb2d9d640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4b9c003a40 0x7f4bb414dd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:41720/0 (socket says 172.21.15.123:41720) 2024-09-16T15:43:47.501 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:47.492+0000 7f4bb2d9d640 1 -- 172.21.15.123:0/4099123110 learned_addr learned my addr 172.21.15.123:0/4099123110 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:43:47.501 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:47.492+0000 7f4bb359e640 1 --2- 172.21.15.123:0/4099123110 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4bb415abd0 0x7f4bb415b050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:43:47.501 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:47.496+0000 7f4bb259c640 1 --2- 172.21.15.123:0/4099123110 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f4bb414e4c0 0x7f4bb40b5410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:43:47.501 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:47.496+0000 7f4bb2d9d640 1 -- 172.21.15.123:0/4099123110 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f4bb414e4c0 msgr2=0x7f4bb40b5410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:43:47.501 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:47.496+0000 7f4bb2d9d640 1 --2- 172.21.15.123:0/4099123110 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f4bb414e4c0 0x7f4bb40b5410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:47.501 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:47.496+0000 7f4bb2d9d640 1 -- 172.21.15.123:0/4099123110 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4bb415abd0 msgr2=0x7f4bb415b050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:43:47.501 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:47.496+0000 7f4bb2d9d640 1 --2- 172.21.15.123:0/4099123110 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4bb415abd0 0x7f4bb415b050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:47.501 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:47.496+0000 7f4bb2d9d640 1 -- 172.21.15.123:0/4099123110 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4bb4152c40 con 0x7f4b9c003a40 2024-09-16T15:43:47.501 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:47.496+0000 7f4bb359e640 1 --2- 172.21.15.123:0/4099123110 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4bb415abd0 0x7f4bb415b050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T15:43:47.501 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:47.496+0000 7f4bb2d9d640 1 --2- 172.21.15.123:0/4099123110 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4b9c003a40 0x7f4bb414dd50 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f4ba40b1040 tx=0x7f4ba4005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:43:47.501 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:47.496+0000 7f4b9b7fe640 1 -- 172.21.15.123:0/4099123110 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ba40be070 con 0x7f4b9c003a40 2024-09-16T15:43:47.501 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:47.496+0000 7f4b9b7fe640 1 -- 172.21.15.123:0/4099123110 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f4ba4005130 con 0x7f4b9c003a40 2024-09-16T15:43:47.501 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:47.496+0000 7f4bb90ba640 1 -- 172.21.15.123:0/4099123110 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4bb4160790 con 0x7f4b9c003a40 2024-09-16T15:43:47.502 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:47.496+0000 7f4b9b7fe640 1 -- 172.21.15.123:0/4099123110 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ba40ab050 con 0x7f4b9c003a40 2024-09-16T15:43:47.502 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:47.496+0000 7f4bb90ba640 1 -- 172.21.15.123:0/4099123110 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4bb4070400 con 0x7f4b9c003a40 2024-09-16T15:43:47.502 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:47.496+0000 7f4bb90ba640 1 -- 172.21.15.123:0/4099123110 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4bb415cc10 con 0x7f4b9c003a40 2024-09-16T15:43:47.502 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:47.496+0000 7f4b9b7fe640 1 -- 172.21.15.123:0/4099123110 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 5) ==== 97080+0+0 (secure 0 0 0) 0x7f4ba40c1070 con 0x7f4b9c003a40 2024-09-16T15:43:47.503 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:47.496+0000 7f4b9b7fe640 1 --2- 172.21.15.123:0/4099123110 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f4ba0072ef0 0x7f4ba00753b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:43:47.503 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:47.496+0000 7f4b9b7fe640 1 -- 172.21.15.123:0/4099123110 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(33..33 src has 1..33) ==== 5709+0+0 (secure 0 0 0) 0x7f4ba412dcb0 con 0x7f4b9c003a40 2024-09-16T15:43:47.503 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:47.496+0000 7f4bb259c640 1 --2- 172.21.15.123:0/4099123110 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f4ba0072ef0 0x7f4ba00753b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:43:47.503 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:47.496+0000 7f4bb259c640 1 --2- 172.21.15.123:0/4099123110 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f4ba0072ef0 0x7f4ba00753b0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f4ba80018d0 tx=0x7f4ba8001e00 comp rx=0 tx=0).ready entity=mgr.4135 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:43:47.505 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:47.500+0000 7f4b9b7fe640 1 -- 172.21.15.123:0/4099123110 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f4ba4016af0 con 0x7f4b9c003a40 2024-09-16T15:43:47.701 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:47.696+0000 7f4bb90ba640 1 -- 172.21.15.123:0/4099123110 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f4bb415bf70 con 0x7f4b9c003a40 2024-09-16T15:43:47.701 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:47.696+0000 7f4b9b7fe640 1 -- 172.21.15.123:0/4099123110 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 18 v18) ==== 94+0+5229 (secure 0 0 0) 0x7f4ba40f8fa0 con 0x7f4b9c003a40 2024-09-16T15:43:47.702 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-09-16T15:43:47.702 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":18,"btime":"2024-09-16T15:43:45:540259+0000","default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4541,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6837/249053959","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6836","nonce":249053959},{"type":"v1","addr":"172.21.15.184:6837","nonce":249053959}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":13},{"gid":4705,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6835/1285950902","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":1285950902},{"type":"v1","addr":"172.21.15.184:6835","nonce":1285950902}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":4866,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6833/1030632114","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6832","nonce":1030632114},{"type":"v1","addr":"172.21.15.123:6833","nonce":1030632114}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11}],"filesystems":[{"mdsmap":{"epoch":18,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-16T15:43:41.851612+0000","modified":"2024-09-16T15:43:44.919014+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4869},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4869":{"gid":4869,"name":"a","rank":0,"incarnation":14,"state":"up:active","state_seq":5,"addr":"172.21.15.123:6835/822918970","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6834","nonce":822918970},{"type":"v1","addr":"172.21.15.123:6835","nonce":822918970}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4869,"qdb_cluster":[4869]},"id":2}]} 2024-09-16T15:43:47.702 INFO:teuthology.orchestra.run.smithi123.stderr:dumped fsmap epoch 18 2024-09-16T15:43:47.704 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:47.700+0000 7f4bb90ba640 1 -- 172.21.15.123:0/4099123110 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f4ba0072ef0 msgr2=0x7f4ba00753b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:43:47.704 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:47.700+0000 7f4bb90ba640 1 --2- 172.21.15.123:0/4099123110 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f4ba0072ef0 0x7f4ba00753b0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f4ba80018d0 tx=0x7f4ba8001e00 comp rx=0 tx=0).stop 2024-09-16T15:43:47.704 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:47.700+0000 7f4bb90ba640 1 -- 172.21.15.123:0/4099123110 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4b9c003a40 msgr2=0x7f4bb414dd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:43:47.704 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:47.700+0000 7f4bb90ba640 1 --2- 172.21.15.123:0/4099123110 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4b9c003a40 0x7f4bb414dd50 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f4ba40b1040 tx=0x7f4ba4005880 comp rx=0 tx=0).stop 2024-09-16T15:43:47.704 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:47.700+0000 7f4bb90ba640 1 -- 172.21.15.123:0/4099123110 shutdown_connections 2024-09-16T15:43:47.704 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:47.700+0000 7f4bb90ba640 1 --2- 172.21.15.123:0/4099123110 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f4ba0072ef0 0x7f4ba00753b0 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:47.704 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:47.700+0000 7f4bb90ba640 1 --2- 172.21.15.123:0/4099123110 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4bb415abd0 0x7f4bb415b050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:47.705 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:47.700+0000 7f4bb90ba640 1 --2- 172.21.15.123:0/4099123110 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f4bb414e4c0 0x7f4bb40b5410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:47.705 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:47.700+0000 7f4bb90ba640 1 --2- 172.21.15.123:0/4099123110 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4b9c003a40 0x7f4bb414dd50 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:47.705 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:47.700+0000 7f4bb90ba640 1 -- 172.21.15.123:0/4099123110 >> 172.21.15.123:0/4099123110 conn(0x7f4bb40d0b40 msgr2=0x7f4bb40d0f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:43:47.705 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:47.700+0000 7f4bb90ba640 1 -- 172.21.15.123:0/4099123110 shutdown_connections 2024-09-16T15:43:47.705 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:47.700+0000 7f4bb90ba640 1 -- 172.21.15.123:0/4099123110 wait complete. 2024-09-16T15:43:48.121 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 18, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-09-16T15:43:41.851612+0000', 'modified': '2024-09-16T15:43:44.919014+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 4869}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4869': {'gid': 4869, 'name': 'a', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 5, 'addr': '172.21.15.123:6835/822918970', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.123:6834', 'nonce': 822918970}, {'type': 'v1', 'addr': '172.21.15.123:6835', 'nonce': 822918970}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540719139924082687, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [5], 'metadata_pool': 4, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4869, 'qdb_cluster': [4869]} 2024-09-16T15:43:48.121 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-09-16T15:43:48.121 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2024-09-16T15:43:48.262 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:48.256+0000 7fe224b1f640 1 -- 172.21.15.123:0/1270456389 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe200003a40 msgr2=0x7fe200003e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:43:48.262 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:48.256+0000 7fe224b1f640 1 --2- 172.21.15.123:0/1270456389 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe200003a40 0x7fe200003e60 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7fe21800c610 tx=0x7fe21801d2c0 comp rx=0 tx=0).stop 2024-09-16T15:43:48.262 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:48.256+0000 7fe21ed9d640 1 -- 172.21.15.123:0/1270456389 reap_dead start 2024-09-16T15:43:48.262 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:48.256+0000 7fe224b1f640 1 -- 172.21.15.123:0/1270456389 shutdown_connections 2024-09-16T15:43:48.263 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:48.256+0000 7fe224b1f640 1 -- 172.21.15.123:0/1270456389 >> 172.21.15.123:0/1270456389 conn(0x7fe2200732a0 msgr2=0x7fe220074150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:43:48.263 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:48.256+0000 7fe224b1f640 1 -- 172.21.15.123:0/1270456389 shutdown_connections 2024-09-16T15:43:48.263 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:48.256+0000 7fe224b1f640 1 -- 172.21.15.123:0/1270456389 wait complete. 2024-09-16T15:43:48.263 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:48.260+0000 7fe224b1f640 1 Processor -- start 2024-09-16T15:43:48.263 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:48.260+0000 7fe224b1f640 1 -- start start 2024-09-16T15:43:48.265 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:48.260+0000 7fe224b1f640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe200003a40 0x7fe2200c58d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:43:48.265 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:48.260+0000 7fe224b1f640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe22009d370 0x7fe22009d7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:43:48.265 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:48.260+0000 7fe224b1f640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fe22009c810 0x7fe22009cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:43:48.265 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:48.260+0000 7fe224b1f640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fe2201682e0 con 0x7fe22009d370 2024-09-16T15:43:48.265 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:48.260+0000 7fe224b1f640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fe2200cff50 con 0x7fe200003a40 2024-09-16T15:43:48.266 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:48.260+0000 7fe224b1f640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fe220168460 con 0x7fe22009c810 2024-09-16T15:43:48.266 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:48.260+0000 7fe21dd9b640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe22009d370 0x7fe22009d7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:43:48.266 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:48.260+0000 7fe21dd9b640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe22009d370 0x7fe22009d7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:41744/0 (socket says 172.21.15.123:41744) 2024-09-16T15:43:48.266 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:48.260+0000 7fe21dd9b640 1 -- 172.21.15.123:0/2494149558 learned_addr learned my addr 172.21.15.123:0/2494149558 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:43:48.266 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:48.260+0000 7fe21e59c640 1 --2- 172.21.15.123:0/2494149558 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe200003a40 0x7fe2200c58d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:43:48.266 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:48.260+0000 7fe21ed9d640 1 --2- 172.21.15.123:0/2494149558 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fe22009c810 0x7fe22009cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:43:48.266 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:48.260+0000 7fe21dd9b640 1 -- 172.21.15.123:0/2494149558 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fe22009c810 msgr2=0x7fe22009cc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:43:48.266 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:48.260+0000 7fe21dd9b640 1 --2- 172.21.15.123:0/2494149558 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fe22009c810 0x7fe22009cc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:48.266 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:48.260+0000 7fe21dd9b640 1 -- 172.21.15.123:0/2494149558 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe200003a40 msgr2=0x7fe2200c58d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:43:48.266 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:48.260+0000 7fe21dd9b640 1 --2- 172.21.15.123:0/2494149558 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe200003a40 0x7fe2200c58d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:48.266 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:48.260+0000 7fe21dd9b640 1 -- 172.21.15.123:0/2494149558 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe2200b3db0 con 0x7fe22009d370 2024-09-16T15:43:48.266 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:48.260+0000 7fe21ed9d640 1 --2- 172.21.15.123:0/2494149558 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fe22009c810 0x7fe22009cc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T15:43:48.266 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:48.260+0000 7fe21dd9b640 1 --2- 172.21.15.123:0/2494149558 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe22009d370 0x7fe22009d7d0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7fe20800e740 tx=0x7fe20800ec00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:43:48.267 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:48.260+0000 7fe216ffd640 1 -- 172.21.15.123:0/2494149558 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe208016070 con 0x7fe22009d370 2024-09-16T15:43:48.267 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:48.260+0000 7fe216ffd640 1 -- 172.21.15.123:0/2494149558 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fe20800ee50 con 0x7fe22009d370 2024-09-16T15:43:48.267 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:48.260+0000 7fe224b1f640 1 -- 172.21.15.123:0/2494149558 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe2200b4050 con 0x7fe22009d370 2024-09-16T15:43:48.267 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:48.260+0000 7fe216ffd640 1 -- 172.21.15.123:0/2494149558 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe20800ca90 con 0x7fe22009d370 2024-09-16T15:43:48.267 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:48.260+0000 7fe224b1f640 1 -- 172.21.15.123:0/2494149558 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe2200b4850 con 0x7fe22009d370 2024-09-16T15:43:48.269 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:48.260+0000 7fe216ffd640 1 -- 172.21.15.123:0/2494149558 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 5) ==== 97080+0+0 (secure 0 0 0) 0x7fe208019070 con 0x7fe22009d370 2024-09-16T15:43:48.269 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:48.260+0000 7fe216ffd640 1 --2- 172.21.15.123:0/2494149558 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fe204072ef0 0x7fe2040753b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:43:48.269 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:48.264+0000 7fe216ffd640 1 -- 172.21.15.123:0/2494149558 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(33..33 src has 1..33) ==== 5709+0+0 (secure 0 0 0) 0x7fe208094290 con 0x7fe22009d370 2024-09-16T15:43:48.269 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:48.264+0000 7fe21e59c640 1 --2- 172.21.15.123:0/2494149558 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fe204072ef0 0x7fe2040753b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:43:48.270 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:48.264+0000 7fe227c09000 1 Processor -- start 2024-09-16T15:43:48.270 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:48.264+0000 7fe227c09000 1 -- start start 2024-09-16T15:43:48.270 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:48.264+0000 7fe21e59c640 1 --2- 172.21.15.123:0/2494149558 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fe204072ef0 0x7fe2040753b0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fe2180ac000 tx=0x7fe21801a100 comp rx=0 tx=0).ready entity=mgr.4135 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:43:48.270 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:48.264+0000 7fe227c09000 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55fd480df9d0 0x55fd480dfdb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:43:48.270 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:48.264+0000 7fe227c09000 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55fd480e02f0 0x55fd480938d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:43:48.270 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:48.264+0000 7fe227c09000 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x55fd48093e10 0x55fd480941f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:43:48.270 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:48.264+0000 7fe227c09000 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x55fd47fb43f0 con 0x55fd480e02f0 2024-09-16T15:43:48.270 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:48.264+0000 7fe227c09000 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x55fd47ef1b20 con 0x55fd480df9d0 2024-09-16T15:43:48.270 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:48.264+0000 7fe227c09000 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x55fd47ee0f50 con 0x55fd48093e10 2024-09-16T15:43:48.270 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:48.264+0000 7fe21dd9b640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55fd480e02f0 0x55fd480938d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:43:48.270 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:48.264+0000 7fe21dd9b640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55fd480e02f0 0x55fd480938d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:41750/0 (socket says 172.21.15.123:41750) 2024-09-16T15:43:48.270 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:48.264+0000 7fe21dd9b640 1 -- 172.21.15.123:0/572342615 learned_addr learned my addr 172.21.15.123:0/572342615 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:43:48.270 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:48.264+0000 7fe21e59c640 1 --2- 172.21.15.123:0/572342615 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55fd480df9d0 0x55fd480dfdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:43:48.271 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:48.264+0000 7fe21ed9d640 1 --2- 172.21.15.123:0/572342615 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x55fd48093e10 0x55fd480941f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:43:48.271 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:48.264+0000 7fe21dd9b640 1 -- 172.21.15.123:0/572342615 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x55fd48093e10 msgr2=0x55fd480941f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T15:43:48.271 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:48.264+0000 7fe21dd9b640 1 --2- 172.21.15.123:0/572342615 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x55fd48093e10 0x55fd480941f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:48.271 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:48.264+0000 7fe21dd9b640 1 -- 172.21.15.123:0/572342615 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55fd480df9d0 msgr2=0x55fd480dfdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T15:43:48.271 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:48.264+0000 7fe21dd9b640 1 --2- 172.21.15.123:0/572342615 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55fd480df9d0 0x55fd480dfdb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:48.271 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:48.264+0000 7fe21dd9b640 1 -- 172.21.15.123:0/572342615 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fd47ea8590 con 0x55fd480e02f0 2024-09-16T15:43:48.271 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:48.264+0000 7fe21e59c640 1 --2- 172.21.15.123:0/572342615 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55fd480df9d0 0x55fd480dfdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T15:43:48.271 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:48.264+0000 7fe21dd9b640 1 --2- 172.21.15.123:0/572342615 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55fd480e02f0 0x55fd480938d0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7fe208092870 tx=0x7fe20805e5e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7f69144c99c51032 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:43:48.271 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:48.264+0000 7fe20ffff640 1 -- 172.21.15.123:0/572342615 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe208020040 con 0x55fd480e02f0 2024-09-16T15:43:48.271 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:48.264+0000 7fe20ffff640 1 -- 172.21.15.123:0/572342615 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fe2080940d0 con 0x55fd480e02f0 2024-09-16T15:43:48.271 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:48.264+0000 7fe20ffff640 1 -- 172.21.15.123:0/572342615 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe208084400 con 0x55fd480e02f0 2024-09-16T15:43:48.275 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:48.264+0000 7fe227c09000 1 -- 172.21.15.123:0/572342615 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55fd480e02f0 msgr2=0x55fd480938d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:43:48.275 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:48.264+0000 7fe227c09000 1 --2- 172.21.15.123:0/572342615 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55fd480e02f0 0x55fd480938d0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7fe208092870 tx=0x7fe20805e5e0 comp rx=0 tx=0).stop 2024-09-16T15:43:48.276 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:48.264+0000 7fe227c09000 1 -- 172.21.15.123:0/572342615 shutdown_connections 2024-09-16T15:43:48.276 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:48.264+0000 7fe227c09000 1 --2- 172.21.15.123:0/572342615 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x55fd48093e10 0x55fd480941f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:48.276 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:48.264+0000 7fe227c09000 1 --2- 172.21.15.123:0/572342615 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55fd480e02f0 0x55fd480938d0 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:48.276 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:48.264+0000 7fe227c09000 1 --2- 172.21.15.123:0/572342615 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55fd480df9d0 0x55fd480dfdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:48.276 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:48.264+0000 7fe227c09000 1 -- 172.21.15.123:0/572342615 >> 172.21.15.123:0/572342615 conn(0x55fd47f7d130 msgr2=0x55fd480c2e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:43:48.276 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:48.264+0000 7fe227c09000 1 -- 172.21.15.123:0/572342615 shutdown_connections 2024-09-16T15:43:48.276 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:48.264+0000 7fe227c09000 1 -- 172.21.15.123:0/572342615 wait complete. 2024-09-16T15:43:48.276 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:48.264+0000 7fe227c09000 1 -- start start 2024-09-16T15:43:48.276 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:48.264+0000 7fe227c09000 1 Processor -- start 2024-09-16T15:43:48.276 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:48.268+0000 7fe227c09000 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x55fd48093e10 0x55fd480e6d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:43:48.276 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:48.268+0000 7fe227c09000 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55fd480df9d0 0x55fd480e72a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:43:48.276 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:48.268+0000 7fe227c09000 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55fd480e02f0 0x55fd480e77e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:43:48.276 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:48.268+0000 7fe227c09000 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x55fd47e7b5d0 con 0x55fd480df9d0 2024-09-16T15:43:48.277 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:48.268+0000 7fe227c09000 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x55fd47ee4d60 con 0x55fd480e02f0 2024-09-16T15:43:48.277 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:48.268+0000 7fe227c09000 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x55fd47e87500 con 0x55fd48093e10 2024-09-16T15:43:48.277 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:48.268+0000 7fe227c09000 10 client.0 _ll_register_callbacks cb 0x55fd47e8b810 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T15:43:48.277 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:48.268+0000 7fe21dd9b640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55fd480df9d0 0x55fd480e72a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:43:48.277 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:48.268+0000 7fe21dd9b640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55fd480df9d0 0x55fd480e72a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:41762/0 (socket says 172.21.15.123:41762) 2024-09-16T15:43:48.277 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:48.268+0000 7fe21dd9b640 1 -- 172.21.15.123:0/2652134257 learned_addr learned my addr 172.21.15.123:0/2652134257 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:43:48.277 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:48.268+0000 7fe21e59c640 1 --2- 172.21.15.123:0/2652134257 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x55fd48093e10 0x55fd480e6d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:43:48.277 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:48.268+0000 7fe21ed9d640 1 --2- 172.21.15.123:0/2652134257 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55fd480e02f0 0x55fd480e77e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:43:48.277 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:48.268+0000 7fe21dd9b640 1 -- 172.21.15.123:0/2652134257 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x55fd48093e10 msgr2=0x55fd480e6d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T15:43:48.277 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:48.268+0000 7fe21dd9b640 1 --2- 172.21.15.123:0/2652134257 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x55fd48093e10 0x55fd480e6d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:48.277 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:48.268+0000 7fe21dd9b640 1 -- 172.21.15.123:0/2652134257 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55fd480e02f0 msgr2=0x55fd480e77e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T15:43:48.277 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:48.268+0000 7fe21dd9b640 1 --2- 172.21.15.123:0/2652134257 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55fd480e02f0 0x55fd480e77e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:48.277 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:48.268+0000 7fe21dd9b640 1 -- 172.21.15.123:0/2652134257 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fd47e7f170 con 0x55fd480df9d0 2024-09-16T15:43:48.277 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:48.268+0000 7fe21e59c640 1 --2- 172.21.15.123:0/2652134257 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x55fd48093e10 0x55fd480e6d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T15:43:48.278 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:48.268+0000 7fe21dd9b640 1 --2- 172.21.15.123:0/2652134257 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55fd480df9d0 0x55fd480e72a0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7fe2080096c0 tx=0x7fe2080843f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4d26603df058e1a8 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:43:48.278 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:48.268+0000 7fe20d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.123:3300/0 2024-09-16T15:43:48.278 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:48.268+0000 7fe20d7fa640 1 -- 172.21.15.123:0/2652134257 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe208020040 con 0x55fd480df9d0 2024-09-16T15:43:48.278 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:48.268+0000 7fe20d7fa640 1 -- 172.21.15.123:0/2652134257 <== mon.0 v2:172.21.15.123:3300/0 2 ==== osd_map(33..33 src has 1..33) ==== 5709+0+0 (secure 0 0 0) 0x7fe208068370 con 0x55fd480df9d0 2024-09-16T15:43:48.278 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:48.268+0000 7fe227c09000 1 -- 172.21.15.123:0/2652134257 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55fd47e790f0 con 0x55fd480df9d0 2024-09-16T15:43:48.278 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:48.268+0000 7fe20d7fa640 1 -- 172.21.15.123:0/2652134257 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_get_version_reply(handle=1 version=18) ==== 24+0+0 (secure 0 0 0) 0x7fe208080050 con 0x55fd480df9d0 2024-09-16T15:43:48.278 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:48.268+0000 7fe227c09000 10 client.5031 fetch_fsmap learned FSMap version 18 2024-09-16T15:43:48.278 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:48.268+0000 7fe227c09000 1 -- 172.21.15.123:0/2652134257 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,fsmap=18,monmap=2+}) -- 0x55fd47f65ef0 con 0x55fd480df9d0 2024-09-16T15:43:48.278 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:48.268+0000 7fe20d7fa640 1 -- 172.21.15.123:0/2652134257 <== mon.0 v2:172.21.15.123:3300/0 4 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x55fd47e790f0 con 0x55fd480df9d0 2024-09-16T15:43:48.278 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:48.268+0000 7fe20d7fa640 1 -- 172.21.15.123:0/2652134257 <== mon.0 v2:172.21.15.123:3300/0 5 ==== fsmap(e 18) ==== 3502+0+0 (secure 0 0 0) 0x7fe20800f040 con 0x55fd480df9d0 2024-09-16T15:43:48.278 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:48.268+0000 7fe227c09000 10 client.5031 fetch_fsmap finished waiting for FSMap version 18 2024-09-16T15:43:48.278 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:48.268+0000 7fe227c09000 10 client.5031 resolve_mds: resolved name 'a' to daemon mds.a 2024-09-16T15:43:48.278 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:48.268+0000 7fe227c09000 20 client.5031 populate_metadata read hostname 'smithi123' 2024-09-16T15:43:48.279 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:48.268+0000 7fe227c09000 1 --2- 172.21.15.123:0/2652134257 >> [v2:172.21.15.123:6834/822918970,v1:172.21.15.123:6835/822918970] conn(0x55fd480ef130 0x55fd480f1520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:43:48.279 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:48.268+0000 7fe227c09000 4 client.5031 mds_command: new command op to 4869 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-16T15:43:48.279 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:48.268+0000 7fe227c09000 1 -- 172.21.15.123:0/2652134257 --> [v2:172.21.15.123:6834/822918970,v1:172.21.15.123:6835/822918970] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55fd47ef3a70 con 0x55fd480ef130 2024-09-16T15:43:48.279 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:48.268+0000 7fe21e59c640 1 --2- 172.21.15.123:0/2652134257 >> [v2:172.21.15.123:6834/822918970,v1:172.21.15.123:6835/822918970] conn(0x55fd480ef130 0x55fd480f1520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:43:48.279 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:48.268+0000 7fe20d7fa640 10 client.5031 ms_handle_connect on v2:172.21.15.123:6834/822918970 2024-09-16T15:43:48.279 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:48.268+0000 7fe21e59c640 1 --2- 172.21.15.123:0/2652134257 >> [v2:172.21.15.123:6834/822918970,v1:172.21.15.123:6835/822918970] conn(0x55fd480ef130 0x55fd480f1520 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8a5ffcd39e21a91e server_cookie=2d119b66099ccf03 in_seq=0 out_seq=0 2024-09-16T15:43:48.279 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:48.268+0000 7fe20d7fa640 1 -- 172.21.15.123:0/2652134257 <== mds.0 v2:172.21.15.123:6834/822918970 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x55fd47ef3a70 con 0x55fd480ef130 2024-09-16T15:43:48.279 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:48.268+0000 7fe20d7fa640 10 client.5031 handle_command_reply: tid=0 2024-09-16T15:43:48.279 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:48.268+0000 7fe227c09000 1 client.5031 shutdown 2024-09-16T15:43:48.279 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:48.268+0000 7fe227c09000 1 -- 172.21.15.123:0/2652134257 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55fd480df9d0 msgr2=0x55fd480e72a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:43:48.279 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:48.268+0000 7fe227c09000 1 --2- 172.21.15.123:0/2652134257 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55fd480df9d0 0x55fd480e72a0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7fe2080096c0 tx=0x7fe2080843f0 comp rx=0 tx=0).stop 2024-09-16T15:43:48.279 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:48.268+0000 7fe227c09000 1 -- 172.21.15.123:0/2652134257 shutdown_connections 2024-09-16T15:43:48.279 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:48.268+0000 7fe227c09000 1 --2- 172.21.15.123:0/2652134257 >> [v2:172.21.15.123:6834/822918970,v1:172.21.15.123:6835/822918970] conn(0x55fd480ef130 0x55fd480f1520 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:48.279 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:48.268+0000 7fe227c09000 1 --2- 172.21.15.123:0/2652134257 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55fd480e02f0 0x55fd480e77e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:48.279 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:48.268+0000 7fe227c09000 1 --2- 172.21.15.123:0/2652134257 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55fd480df9d0 0x55fd480e72a0 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:48.280 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:48.268+0000 7fe20d7fa640 0 client.5031 ms_handle_reset on v2:172.21.15.123:6834/822918970 2024-09-16T15:43:48.280 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:48.268+0000 7fe227c09000 1 --2- 172.21.15.123:0/2652134257 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x55fd48093e10 0x55fd480e6d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:48.280 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:48.268+0000 7fe227c09000 1 -- 172.21.15.123:0/2652134257 >> 172.21.15.123:0/2652134257 conn(0x55fd47f7d130 msgr2=0x55fd48092fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:43:48.280 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:48.268+0000 7fe227c09000 1 -- 172.21.15.123:0/2652134257 shutdown_connections 2024-09-16T15:43:48.280 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:48.268+0000 7fe227c09000 1 -- 172.21.15.123:0/2652134257 wait complete. 2024-09-16T15:43:48.280 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:48.268+0000 7fe227c09000 20 client.5031 trim_cache size 0 max 16384 2024-09-16T15:43:48.293 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:48.288+0000 7fe227c09000 1 Processor -- start 2024-09-16T15:43:48.293 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:48.288+0000 7fe227c09000 1 -- start start 2024-09-16T15:43:48.296 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:48.288+0000 7fe227c09000 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x55fd47f7d130 0x55fd4815a850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:43:48.296 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:48.288+0000 7fe227c09000 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55fd48093e10 0x55fd4815ad90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:43:48.296 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:48.288+0000 7fe227c09000 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55fd480df9d0 0x55fd4815b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:43:48.296 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:48.288+0000 7fe227c09000 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fe220201fa0 con 0x55fd480df9d0 2024-09-16T15:43:48.296 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:48.288+0000 7fe227c09000 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fe220159040 con 0x55fd48093e10 2024-09-16T15:43:48.296 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:48.288+0000 7fe227c09000 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fe2200b5e30 con 0x55fd47f7d130 2024-09-16T15:43:48.296 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:48.288+0000 7fe21ed9d640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55fd480df9d0 0x55fd4815b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:43:48.296 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:48.288+0000 7fe21ed9d640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55fd480df9d0 0x55fd4815b2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:41766/0 (socket says 172.21.15.123:41766) 2024-09-16T15:43:48.297 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:48.288+0000 7fe21ed9d640 1 -- 172.21.15.123:0/400496143 learned_addr learned my addr 172.21.15.123:0/400496143 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:43:48.297 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:48.288+0000 7fe21ed9d640 1 -- 172.21.15.123:0/400496143 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x55fd47f7d130 msgr2=0x55fd4815a850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T15:43:48.297 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:48.288+0000 7fe21ed9d640 1 --2- 172.21.15.123:0/400496143 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x55fd47f7d130 0x55fd4815a850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:48.297 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:48.288+0000 7fe21ed9d640 1 -- 172.21.15.123:0/400496143 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55fd48093e10 msgr2=0x55fd4815ad90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T15:43:48.297 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:48.288+0000 7fe21dd9b640 1 --2- 172.21.15.123:0/400496143 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55fd48093e10 0x55fd4815ad90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:43:48.297 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:48.288+0000 7fe21ed9d640 1 --2- 172.21.15.123:0/400496143 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55fd48093e10 0x55fd4815ad90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:48.297 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:48.288+0000 7fe21ed9d640 1 -- 172.21.15.123:0/400496143 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fd47f3cd90 con 0x55fd480df9d0 2024-09-16T15:43:48.297 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:48.288+0000 7fe21ed9d640 1 --2- 172.21.15.123:0/400496143 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55fd480df9d0 0x55fd4815b2d0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7fe22009e3c0 tx=0x7fe210001840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=645156149e910e92 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:43:48.297 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:48.288+0000 7fe20f7fe640 1 -- 172.21.15.123:0/400496143 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe210013870 con 0x55fd480df9d0 2024-09-16T15:43:48.297 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:48.288+0000 7fe20f7fe640 1 -- 172.21.15.123:0/400496143 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fe210019070 con 0x55fd480df9d0 2024-09-16T15:43:48.297 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:48.288+0000 7fe20f7fe640 1 -- 172.21.15.123:0/400496143 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe210013da0 con 0x55fd480df9d0 2024-09-16T15:43:48.298 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:48.292+0000 7fe227c09000 1 -- 172.21.15.123:0/400496143 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55fd480df9d0 msgr2=0x55fd4815b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:43:48.298 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:48.292+0000 7fe227c09000 1 --2- 172.21.15.123:0/400496143 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55fd480df9d0 0x55fd4815b2d0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7fe22009e3c0 tx=0x7fe210001840 comp rx=0 tx=0).stop 2024-09-16T15:43:48.298 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:48.292+0000 7fe227c09000 1 -- 172.21.15.123:0/400496143 shutdown_connections 2024-09-16T15:43:48.298 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:48.292+0000 7fe227c09000 1 --2- 172.21.15.123:0/400496143 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55fd480df9d0 0x55fd4815b2d0 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:48.298 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:48.292+0000 7fe227c09000 1 --2- 172.21.15.123:0/400496143 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55fd48093e10 0x55fd4815ad90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:48.298 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:48.292+0000 7fe227c09000 1 --2- 172.21.15.123:0/400496143 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x55fd47f7d130 0x55fd4815a850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:48.298 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:48.292+0000 7fe227c09000 1 -- 172.21.15.123:0/400496143 >> 172.21.15.123:0/400496143 conn(0x55fd480df0c0 msgr2=0x55fd480920a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:43:48.298 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:48.292+0000 7fe227c09000 1 -- 172.21.15.123:0/400496143 shutdown_connections 2024-09-16T15:43:48.298 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:48.292+0000 7fe227c09000 1 -- 172.21.15.123:0/400496143 wait complete. 2024-09-16T15:43:48.298 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:48.292+0000 7fe227c09000 1 -- start start 2024-09-16T15:43:48.298 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:48.292+0000 7fe227c09000 1 Processor -- start 2024-09-16T15:43:48.302 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:48.292+0000 7fe227c09000 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55fd47f7d130 0x55fd4815e6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:43:48.302 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:48.292+0000 7fe227c09000 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55fd48093e10 0x55fd4815ec30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:43:48.302 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:48.292+0000 7fe227c09000 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x55fd480df9d0 0x55fd4815f170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:43:48.302 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:48.292+0000 7fe227c09000 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x55fd47e7b5d0 con 0x55fd48093e10 2024-09-16T15:43:48.302 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:48.292+0000 7fe227c09000 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x55fd47eacec0 con 0x55fd47f7d130 2024-09-16T15:43:48.302 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:48.292+0000 7fe227c09000 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x55fd47ee4d60 con 0x55fd480df9d0 2024-09-16T15:43:48.302 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:48.292+0000 7fe227c09000 10 client.0 _ll_register_callbacks cb 0x7fe20805ae00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T15:43:48.303 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:48.292+0000 7fe21dd9b640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55fd48093e10 0x55fd4815ec30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:43:48.303 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:48.292+0000 7fe21dd9b640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55fd48093e10 0x55fd4815ec30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:41768/0 (socket says 172.21.15.123:41768) 2024-09-16T15:43:48.303 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:48.292+0000 7fe21dd9b640 1 -- 172.21.15.123:0/4037582473 learned_addr learned my addr 172.21.15.123:0/4037582473 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:43:48.303 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:48.292+0000 7fe21e59c640 1 --2- 172.21.15.123:0/4037582473 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55fd47f7d130 0x55fd4815e6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:43:48.303 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:48.292+0000 7fe21ed9d640 1 --2- 172.21.15.123:0/4037582473 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x55fd480df9d0 0x55fd4815f170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:43:48.303 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:48.292+0000 7fe21dd9b640 1 -- 172.21.15.123:0/4037582473 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x55fd480df9d0 msgr2=0x55fd4815f170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T15:43:48.303 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:48.292+0000 7fe21dd9b640 1 --2- 172.21.15.123:0/4037582473 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x55fd480df9d0 0x55fd4815f170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:48.303 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:48.292+0000 7fe21dd9b640 1 -- 172.21.15.123:0/4037582473 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55fd47f7d130 msgr2=0x55fd4815e6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T15:43:48.303 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:48.292+0000 7fe21dd9b640 1 --2- 172.21.15.123:0/4037582473 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55fd47f7d130 0x55fd4815e6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:48.303 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:48.292+0000 7fe21dd9b640 1 -- 172.21.15.123:0/4037582473 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fd47ea7a90 con 0x55fd48093e10 2024-09-16T15:43:48.303 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:48.292+0000 7fe21dd9b640 1 --2- 172.21.15.123:0/4037582473 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55fd48093e10 0x55fd4815ec30 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7fe20805c0e0 tx=0x7fe20804c5a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c65638b1f3f2d563 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:43:48.303 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:48.292+0000 7fe20d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.123:3300/0 2024-09-16T15:43:48.304 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:48.292+0000 7fe20d7fa640 1 -- 172.21.15.123:0/4037582473 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe208020070 con 0x55fd48093e10 2024-09-16T15:43:48.304 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:48.292+0000 7fe20d7fa640 1 -- 172.21.15.123:0/4037582473 <== mon.0 v2:172.21.15.123:3300/0 2 ==== osd_map(33..33 src has 1..33) ==== 5709+0+0 (secure 0 0 0) 0x7fe208068370 con 0x55fd48093e10 2024-09-16T15:43:48.304 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:48.292+0000 7fe227c09000 1 -- 172.21.15.123:0/4037582473 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe210019070 con 0x55fd48093e10 2024-09-16T15:43:48.304 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:48.292+0000 7fe20d7fa640 1 -- 172.21.15.123:0/4037582473 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_get_version_reply(handle=1 version=18) ==== 24+0+0 (secure 0 0 0) 0x7fe208074050 con 0x55fd48093e10 2024-09-16T15:43:48.304 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:48.292+0000 7fe227c09000 10 client.5037 fetch_fsmap learned FSMap version 18 2024-09-16T15:43:48.304 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:48.292+0000 7fe227c09000 1 -- 172.21.15.123:0/4037582473 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,fsmap=18,monmap=2+}) -- 0x55fd47e86050 con 0x55fd48093e10 2024-09-16T15:43:48.304 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:48.292+0000 7fe20d7fa640 1 -- 172.21.15.123:0/4037582473 <== mon.0 v2:172.21.15.123:3300/0 4 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fe210019070 con 0x55fd48093e10 2024-09-16T15:43:48.304 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:48.292+0000 7fe20d7fa640 1 -- 172.21.15.123:0/4037582473 <== mon.0 v2:172.21.15.123:3300/0 5 ==== fsmap(e 18) ==== 3502+0+0 (secure 0 0 0) 0x7fe2080801f0 con 0x55fd48093e10 2024-09-16T15:43:48.304 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:48.292+0000 7fe227c09000 10 client.5037 fetch_fsmap finished waiting for FSMap version 18 2024-09-16T15:43:48.304 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:48.292+0000 7fe227c09000 10 client.5037 resolve_mds: resolved name 'a' to daemon mds.a 2024-09-16T15:43:48.304 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:48.292+0000 7fe227c09000 20 client.5037 populate_metadata read hostname 'smithi123' 2024-09-16T15:43:48.304 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:48.292+0000 7fe227c09000 1 --2- 172.21.15.123:0/4037582473 >> [v2:172.21.15.123:6834/822918970,v1:172.21.15.123:6835/822918970] conn(0x55fd480e02f0 0x55fd480f5150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:43:48.304 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:48.292+0000 7fe227c09000 4 client.5037 mds_command: new command op to 4869 tid=0 multi_id=0 [{"prefix": "status"}] 2024-09-16T15:43:48.305 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:48.292+0000 7fe227c09000 1 -- 172.21.15.123:0/4037582473 --> [v2:172.21.15.123:6834/822918970,v1:172.21.15.123:6835/822918970] -- command(tid 0: {"prefix": "status"}) -- 0x55fd47eef520 con 0x55fd480e02f0 2024-09-16T15:43:48.305 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:48.292+0000 7fe21e59c640 1 --2- 172.21.15.123:0/4037582473 >> [v2:172.21.15.123:6834/822918970,v1:172.21.15.123:6835/822918970] conn(0x55fd480e02f0 0x55fd480f5150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:43:48.305 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:48.292+0000 7fe20d7fa640 10 client.5037 ms_handle_connect on v2:172.21.15.123:6834/822918970 2024-09-16T15:43:48.305 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:48.292+0000 7fe21e59c640 1 --2- 172.21.15.123:0/4037582473 >> [v2:172.21.15.123:6834/822918970,v1:172.21.15.123:6835/822918970] conn(0x55fd480e02f0 0x55fd480f5150 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8f7d53f39c323d9d server_cookie=2104cee38142dd6e in_seq=0 out_seq=0 2024-09-16T15:43:48.305 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:48.292+0000 7fe20d7fa640 1 -- 172.21.15.123:0/4037582473 <== mds.0 v2:172.21.15.123:6834/822918970 1 ==== command_reply(tid 0: 0 ) ==== 8+0+323 (crc 0 0 0) 0x55fd47eef520 con 0x55fd480e02f0 2024-09-16T15:43:48.305 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:48.292+0000 7fe20d7fa640 10 client.5037 handle_command_reply: tid=0 2024-09-16T15:43:48.305 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:48.292+0000 7fe227c09000 1 client.5037 shutdown 2024-09-16T15:43:48.305 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:48.292+0000 7fe227c09000 1 -- 172.21.15.123:0/4037582473 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55fd48093e10 msgr2=0x55fd4815ec30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:43:48.305 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:48.292+0000 7fe227c09000 1 --2- 172.21.15.123:0/4037582473 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55fd48093e10 0x55fd4815ec30 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7fe20805c0e0 tx=0x7fe20804c5a0 comp rx=0 tx=0).stop 2024-09-16T15:43:48.305 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:48.292+0000 7fe227c09000 1 -- 172.21.15.123:0/4037582473 shutdown_connections 2024-09-16T15:43:48.305 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:48.292+0000 7fe227c09000 1 --2- 172.21.15.123:0/4037582473 >> [v2:172.21.15.123:6834/822918970,v1:172.21.15.123:6835/822918970] conn(0x55fd480e02f0 0x55fd480f5150 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:48.306 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:48.292+0000 7fe227c09000 1 --2- 172.21.15.123:0/4037582473 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x55fd480df9d0 0x55fd4815f170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:48.306 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:48.292+0000 7fe227c09000 1 --2- 172.21.15.123:0/4037582473 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55fd48093e10 0x55fd4815ec30 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:48.306 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:48.292+0000 7fe20d7fa640 0 client.5037 ms_handle_reset on v2:172.21.15.123:6834/822918970 2024-09-16T15:43:48.306 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:48.292+0000 7fe227c09000 1 --2- 172.21.15.123:0/4037582473 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55fd47f7d130 0x55fd4815e6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:48.306 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:48.292+0000 7fe227c09000 1 -- 172.21.15.123:0/4037582473 >> 172.21.15.123:0/4037582473 conn(0x55fd480df0c0 msgr2=0x55fd480928c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:43:48.306 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:48.292+0000 7fe227c09000 1 -- 172.21.15.123:0/4037582473 shutdown_connections 2024-09-16T15:43:48.306 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:48.292+0000 7fe227c09000 1 -- 172.21.15.123:0/4037582473 wait complete. 2024-09-16T15:43:48.306 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:48.292+0000 7fe227c09000 20 client.5037 trim_cache size 0 max 16384 2024-09-16T15:43:48.306 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2024-09-16T15:43:48.307 INFO:teuthology.orchestra.run.smithi123.stdout: "cluster_fsid": "56077f89-18c8-486f-ae7e-eecc0bb7822c", 2024-09-16T15:43:48.307 INFO:teuthology.orchestra.run.smithi123.stdout: "whoami": 0, 2024-09-16T15:43:48.307 INFO:teuthology.orchestra.run.smithi123.stdout: "id": 4869, 2024-09-16T15:43:48.307 INFO:teuthology.orchestra.run.smithi123.stdout: "want_state": "up:active", 2024-09-16T15:43:48.307 INFO:teuthology.orchestra.run.smithi123.stdout: "state": "up:active", 2024-09-16T15:43:48.307 INFO:teuthology.orchestra.run.smithi123.stdout: "fs_name": "cephfs", 2024-09-16T15:43:48.307 INFO:teuthology.orchestra.run.smithi123.stdout: "rank_uptime": 5.77996189, 2024-09-16T15:43:48.307 INFO:teuthology.orchestra.run.smithi123.stdout: "mdsmap_epoch": 18, 2024-09-16T15:43:48.307 INFO:teuthology.orchestra.run.smithi123.stdout: "osdmap_epoch": 33, 2024-09-16T15:43:48.307 INFO:teuthology.orchestra.run.smithi123.stdout: "osdmap_epoch_barrier": 33, 2024-09-16T15:43:48.307 INFO:teuthology.orchestra.run.smithi123.stdout: "uptime": 19.756323347999999 2024-09-16T15:43:48.307 INFO:teuthology.orchestra.run.smithi123.stdout:} 2024-09-16T15:43:48.308 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:48.296+0000 7fe224b1f640 1 -- 172.21.15.123:0/2494149558 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fe204072ef0 msgr2=0x7fe2040753b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:43:48.308 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:48.296+0000 7fe224b1f640 1 --2- 172.21.15.123:0/2494149558 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fe204072ef0 0x7fe2040753b0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fe2180ac000 tx=0x7fe21801a100 comp rx=0 tx=0).stop 2024-09-16T15:43:48.309 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:48.296+0000 7fe224b1f640 1 -- 172.21.15.123:0/2494149558 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe22009d370 msgr2=0x7fe22009d7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:43:48.309 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:48.296+0000 7fe224b1f640 1 --2- 172.21.15.123:0/2494149558 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe22009d370 0x7fe22009d7d0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7fe20800e740 tx=0x7fe20800ec00 comp rx=0 tx=0).stop 2024-09-16T15:43:48.309 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:48.296+0000 7fe224b1f640 1 -- 172.21.15.123:0/2494149558 shutdown_connections 2024-09-16T15:43:48.309 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:48.296+0000 7fe224b1f640 1 --2- 172.21.15.123:0/2494149558 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fe204072ef0 0x7fe2040753b0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:48.309 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:48.296+0000 7fe224b1f640 1 --2- 172.21.15.123:0/2494149558 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fe22009c810 0x7fe22009cc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:48.309 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:48.296+0000 7fe224b1f640 1 --2- 172.21.15.123:0/2494149558 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe22009d370 0x7fe22009d7d0 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:48.309 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:48.296+0000 7fe224b1f640 1 --2- 172.21.15.123:0/2494149558 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe200003a40 0x7fe2200c58d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:48.309 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:48.296+0000 7fe224b1f640 1 -- 172.21.15.123:0/2494149558 >> 172.21.15.123:0/2494149558 conn(0x7fe2200732a0 msgr2=0x7fe220155c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:43:48.309 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:48.296+0000 7fe224b1f640 1 -- 172.21.15.123:0/2494149558 shutdown_connections 2024-09-16T15:43:48.309 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:48.296+0000 7fe224b1f640 1 -- 172.21.15.123:0/2494149558 wait complete. 2024-09-16T15:43:48.316 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-09-16T15:43:48.433 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:48.428+0000 7f7972095640 1 -- 172.21.15.123:0/845253241 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f796c0b25a0 msgr2=0x7f796c0d1c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:43:48.433 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:48.428+0000 7f7972095640 1 --2- 172.21.15.123:0/845253241 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f796c0b25a0 0x7f796c0d1c10 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f796c0b4820 tx=0x7f795401c830 comp rx=0 tx=0).stop 2024-09-16T15:43:48.433 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:48.428+0000 7f7972095640 1 -- 172.21.15.123:0/845253241 shutdown_connections 2024-09-16T15:43:48.433 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:48.428+0000 7f7972095640 1 --2- 172.21.15.123:0/845253241 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f796c0d0850 0x7f796c0d2150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:48.434 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:48.428+0000 7f7972095640 1 --2- 172.21.15.123:0/845253241 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f796c0b25a0 0x7f796c0d1c10 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:48.434 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:48.428+0000 7f7972095640 1 --2- 172.21.15.123:0/845253241 >> v2:172.21.15.184:3300/0 conn(0x7f796c15bda0 0x7f796c0d0310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:48.434 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:48.428+0000 7f7972095640 1 -- 172.21.15.123:0/845253241 >> 172.21.15.123:0/845253241 conn(0x7f796c074bd0 msgr2=0x7f796c09aa40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:43:48.434 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:48.428+0000 7f7972095640 1 -- 172.21.15.123:0/845253241 shutdown_connections 2024-09-16T15:43:48.434 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:48.428+0000 7f7972095640 1 -- 172.21.15.123:0/845253241 wait complete. 2024-09-16T15:43:48.434 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:48.428+0000 7f7972095640 1 Processor -- start 2024-09-16T15:43:48.435 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:48.428+0000 7f7972095640 1 -- start start 2024-09-16T15:43:48.437 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:48.432+0000 7f7972095640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f796c0b25a0 0x7f796c154fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:43:48.437 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:48.432+0000 7f7972095640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f796c0d0850 0x7f796c14ba50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:43:48.437 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:48.432+0000 7f7972095640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f796c15bda0 0x7f796c0b4af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:43:48.437 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:48.432+0000 7f7972095640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f796c0cce20 con 0x7f796c0b25a0 2024-09-16T15:43:48.437 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:48.432+0000 7f7972095640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f796c0ccca0 con 0x7f796c15bda0 2024-09-16T15:43:48.437 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:48.432+0000 7f7972095640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f796c0ccfa0 con 0x7f796c0d0850 2024-09-16T15:43:48.437 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:48.432+0000 7f796b7fe640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f796c0b25a0 0x7f796c154fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:43:48.437 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:48.432+0000 7f796b7fe640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f796c0b25a0 0x7f796c154fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:41784/0 (socket says 172.21.15.123:41784) 2024-09-16T15:43:48.437 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:48.432+0000 7f796b7fe640 1 -- 172.21.15.123:0/3851683973 learned_addr learned my addr 172.21.15.123:0/3851683973 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:43:48.438 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:48.432+0000 7f7963fff640 1 --2- 172.21.15.123:0/3851683973 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f796c0d0850 0x7f796c14ba50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:43:48.438 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:48.432+0000 7f796bfff640 1 --2- 172.21.15.123:0/3851683973 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f796c15bda0 0x7f796c0b4af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:43:48.438 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:48.432+0000 7f796b7fe640 1 -- 172.21.15.123:0/3851683973 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f796c0d0850 msgr2=0x7f796c14ba50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:43:48.438 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:48.432+0000 7f796b7fe640 1 --2- 172.21.15.123:0/3851683973 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f796c0d0850 0x7f796c14ba50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:48.438 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:48.432+0000 7f796b7fe640 1 -- 172.21.15.123:0/3851683973 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f796c15bda0 msgr2=0x7f796c0b4af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:43:48.438 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:48.432+0000 7f796b7fe640 1 --2- 172.21.15.123:0/3851683973 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f796c15bda0 0x7f796c0b4af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:48.438 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:48.432+0000 7f796b7fe640 1 -- 172.21.15.123:0/3851683973 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f796c0cb870 con 0x7f796c0b25a0 2024-09-16T15:43:48.438 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:48.432+0000 7f7963fff640 1 --2- 172.21.15.123:0/3851683973 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f796c0d0850 0x7f796c14ba50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T15:43:48.438 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:48.432+0000 7f796b7fe640 1 --2- 172.21.15.123:0/3851683973 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f796c0b25a0 0x7f796c154fc0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f795c00d950 tx=0x7f795c00de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:43:48.438 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:48.432+0000 7f7968ff9640 1 -- 172.21.15.123:0/3851683973 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f795c017070 con 0x7f796c0b25a0 2024-09-16T15:43:48.439 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:48.432+0000 7f7968ff9640 1 -- 172.21.15.123:0/3851683973 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f795c009d20 con 0x7f796c0b25a0 2024-09-16T15:43:48.439 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:48.432+0000 7f7972095640 1 -- 172.21.15.123:0/3851683973 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f796c0cbb50 con 0x7f796c0b25a0 2024-09-16T15:43:48.439 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:48.432+0000 7f7968ff9640 1 -- 172.21.15.123:0/3851683973 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f795c0106c0 con 0x7f796c0b25a0 2024-09-16T15:43:48.439 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:48.432+0000 7f7972095640 1 -- 172.21.15.123:0/3851683973 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f796c0b5030 con 0x7f796c0b25a0 2024-09-16T15:43:48.440 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:48.432+0000 7f7972095640 1 -- 172.21.15.123:0/3851683973 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f796c0c9b80 con 0x7f796c0b25a0 2024-09-16T15:43:48.440 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:48.432+0000 7f7968ff9640 1 -- 172.21.15.123:0/3851683973 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 5) ==== 97080+0+0 (secure 0 0 0) 0x7f795c01a070 con 0x7f796c0b25a0 2024-09-16T15:43:48.440 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:48.432+0000 7f7968ff9640 1 --2- 172.21.15.123:0/3851683973 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f794c072ec0 0x7f794c075380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:43:48.440 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:48.432+0000 7f7968ff9640 1 -- 172.21.15.123:0/3851683973 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(33..33 src has 1..33) ==== 5709+0+0 (secure 0 0 0) 0x7f795c093e70 con 0x7f796c0b25a0 2024-09-16T15:43:48.440 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:48.436+0000 7f7963fff640 1 --2- 172.21.15.123:0/3851683973 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f794c072ec0 0x7f794c075380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:43:48.440 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:48.436+0000 7f7963fff640 1 --2- 172.21.15.123:0/3851683973 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f794c072ec0 0x7f794c075380 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f79540af000 tx=0x7f795401b070 comp rx=0 tx=0).ready entity=mgr.4135 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:43:48.443 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:48.440+0000 7f7968ff9640 1 -- 172.21.15.123:0/3851683973 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f795c05f150 con 0x7f796c0b25a0 2024-09-16T15:43:48.647 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:48.640+0000 7f7972095640 1 -- 172.21.15.123:0/3851683973 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f796c0d0310 con 0x7f796c0b25a0 2024-09-16T15:43:48.648 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:48.644+0000 7f7968ff9640 1 -- 172.21.15.123:0/3851683973 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v18) ==== 53+0+73 (secure 0 0 0) 0x7f795c05ea50 con 0x7f796c0b25a0 2024-09-16T15:43:48.648 INFO:teuthology.orchestra.run.smithi123.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-09-16T15:43:48.650 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:48.644+0000 7f7972095640 1 -- 172.21.15.123:0/3851683973 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f794c072ec0 msgr2=0x7f794c075380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:43:48.651 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:48.644+0000 7f7972095640 1 --2- 172.21.15.123:0/3851683973 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f794c072ec0 0x7f794c075380 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f79540af000 tx=0x7f795401b070 comp rx=0 tx=0).stop 2024-09-16T15:43:48.651 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:48.644+0000 7f7972095640 1 -- 172.21.15.123:0/3851683973 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f796c0b25a0 msgr2=0x7f796c154fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:43:48.651 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:48.644+0000 7f7972095640 1 --2- 172.21.15.123:0/3851683973 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f796c0b25a0 0x7f796c154fc0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f795c00d950 tx=0x7f795c00de10 comp rx=0 tx=0).stop 2024-09-16T15:43:48.651 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:48.644+0000 7f7972095640 1 -- 172.21.15.123:0/3851683973 shutdown_connections 2024-09-16T15:43:48.651 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:48.644+0000 7f7972095640 1 --2- 172.21.15.123:0/3851683973 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f794c072ec0 0x7f794c075380 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:48.651 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:48.644+0000 7f7972095640 1 --2- 172.21.15.123:0/3851683973 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f796c15bda0 0x7f796c0b4af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:48.651 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:48.644+0000 7f7972095640 1 --2- 172.21.15.123:0/3851683973 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f796c0d0850 0x7f796c14ba50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:48.651 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:48.644+0000 7f7972095640 1 --2- 172.21.15.123:0/3851683973 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f796c0b25a0 0x7f796c154fc0 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:48.651 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:48.644+0000 7f7972095640 1 -- 172.21.15.123:0/3851683973 >> 172.21.15.123:0/3851683973 conn(0x7f796c074bd0 msgr2=0x7f796c0b0a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:43:48.651 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:48.644+0000 7f7972095640 1 -- 172.21.15.123:0/3851683973 shutdown_connections 2024-09-16T15:43:48.651 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:48.644+0000 7f7972095640 1 -- 172.21.15.123:0/3851683973 wait complete. 2024-09-16T15:43:48.666 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-09-16T15:43:48.666 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-09-16T15:43:48.666 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi123.front.sepia.ceph.com 2024-09-16T15:43:48.666 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-09-16T15:43:48.667 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-09-16T15:43:48.667 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-09-16T15:43:48.667 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-09-16T15:43:48.667 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-09-16T15:43:48.667 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-09-16T15:43:48.667 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:43:48.667 DEBUG:teuthology.orchestra.run.smithi123:> ip addr 2024-09-16T15:43:48.673 INFO:teuthology.orchestra.run.smithi123.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-09-16T15:43:48.673 INFO:teuthology.orchestra.run.smithi123.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-09-16T15:43:48.674 INFO:teuthology.orchestra.run.smithi123.stdout: inet 127.0.0.1/8 scope host lo 2024-09-16T15:43:48.674 INFO:teuthology.orchestra.run.smithi123.stdout: valid_lft forever preferred_lft forever 2024-09-16T15:43:48.674 INFO:teuthology.orchestra.run.smithi123.stdout: inet6 ::1/128 scope host 2024-09-16T15:43:48.674 INFO:teuthology.orchestra.run.smithi123.stdout: valid_lft forever preferred_lft forever 2024-09-16T15:43:48.674 INFO:teuthology.orchestra.run.smithi123.stdout:2: eno1: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-09-16T15:43:48.674 INFO:teuthology.orchestra.run.smithi123.stdout: link/ether 0c:c4:7a:d8:da:6c brd ff:ff:ff:ff:ff:ff 2024-09-16T15:43:48.674 INFO:teuthology.orchestra.run.smithi123.stdout: altname enp7s0f0 2024-09-16T15:43:48.674 INFO:teuthology.orchestra.run.smithi123.stdout:3: eno2: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-09-16T15:43:48.674 INFO:teuthology.orchestra.run.smithi123.stdout: link/ether 0c:c4:7a:d8:da:6d brd ff:ff:ff:ff:ff:ff 2024-09-16T15:43:48.674 INFO:teuthology.orchestra.run.smithi123.stdout: altname enp7s0f1 2024-09-16T15:43:48.674 INFO:teuthology.orchestra.run.smithi123.stdout:4: enp3s0f0: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-09-16T15:43:48.675 INFO:teuthology.orchestra.run.smithi123.stdout: link/ether 0c:c4:7a:8f:d3:f0 brd ff:ff:ff:ff:ff:ff 2024-09-16T15:43:48.675 INFO:teuthology.orchestra.run.smithi123.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-09-16T15:43:48.675 INFO:teuthology.orchestra.run.smithi123.stdout: link/ether 0c:c4:7a:8f:d3:f1 brd ff:ff:ff:ff:ff:ff 2024-09-16T15:43:48.675 INFO:teuthology.orchestra.run.smithi123.stdout: inet 172.21.15.123/20 metric 100 brd 172.21.15.255 scope global dynamic enp3s0f1 2024-09-16T15:43:48.675 INFO:teuthology.orchestra.run.smithi123.stdout: valid_lft 42440sec preferred_lft 42440sec 2024-09-16T15:43:48.675 INFO:teuthology.orchestra.run.smithi123.stdout: inet6 fe80::ec4:7aff:fe8f:d3f1/64 scope link 2024-09-16T15:43:48.675 INFO:teuthology.orchestra.run.smithi123.stdout: valid_lft forever preferred_lft forever 2024-09-16T15:43:48.675 INFO:teuthology.orchestra.run.smithi123.stdout:6: docker0: mtu 1500 qdisc noqueue state DOWN group default 2024-09-16T15:43:48.675 INFO:teuthology.orchestra.run.smithi123.stdout: link/ether 02:42:dd:82:a2:6e brd ff:ff:ff:ff:ff:ff 2024-09-16T15:43:48.675 INFO:teuthology.orchestra.run.smithi123.stdout: inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0 2024-09-16T15:43:48.675 INFO:teuthology.orchestra.run.smithi123.stdout: valid_lft forever preferred_lft forever 2024-09-16T15:43:48.676 INFO:teuthology.orchestra.run.smithi123.stdout:7: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-09-16T15:43:48.676 INFO:teuthology.orchestra.run.smithi123.stdout: link/ether 5e:1f:84:6d:79:ae brd ff:ff:ff:ff:ff:ff 2024-09-16T15:43:48.676 INFO:teuthology.orchestra.run.smithi123.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-09-16T15:43:48.676 INFO:teuthology.orchestra.run.smithi123.stdout: valid_lft forever preferred_lft forever 2024-09-16T15:43:48.676 INFO:teuthology.orchestra.run.smithi123.stdout: inet6 fe80::5c1f:84ff:fe6d:79ae/64 scope link 2024-09-16T15:43:48.676 INFO:teuthology.orchestra.run.smithi123.stdout: valid_lft forever preferred_lft forever 2024-09-16T15:43:48.676 INFO:teuthology.orchestra.run.smithi123.stdout:8: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-09-16T15:43:48.676 INFO:teuthology.orchestra.run.smithi123.stdout: link/ether 0e:5c:f6:1f:82:2f brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-09-16T15:43:48.676 INFO:teuthology.orchestra.run.smithi123.stdout: inet6 fe80::c5c:f6ff:fe1f:822f/64 scope link 2024-09-16T15:43:48.676 INFO:teuthology.orchestra.run.smithi123.stdout: valid_lft forever preferred_lft forever 2024-09-16T15:43:48.677 INFO:teuthology.orchestra.run.smithi123.stdout:9: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-09-16T15:43:48.677 INFO:teuthology.orchestra.run.smithi123.stdout: link/ether c6:a6:3d:69:d0:db brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-09-16T15:43:48.677 INFO:teuthology.orchestra.run.smithi123.stdout: inet6 fe80::c4a6:3dff:fe69:d0db/64 scope link 2024-09-16T15:43:48.677 INFO:teuthology.orchestra.run.smithi123.stdout: valid_lft forever preferred_lft forever 2024-09-16T15:43:48.677 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:43:48.677 DEBUG:teuthology.orchestra.run.smithi123:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-09-16T15:43:48.725 INFO:teuthology.orchestra.run.smithi123.stdout:1 2024-09-16T15:43:48.726 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:43:48.726 DEBUG:teuthology.orchestra.run.smithi123:> route 2024-09-16T15:43:48.780 INFO:teuthology.orchestra.run.smithi123.stdout:Kernel IP routing table 2024-09-16T15:43:48.780 INFO:teuthology.orchestra.run.smithi123.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-09-16T15:43:48.780 INFO:teuthology.orchestra.run.smithi123.stdout:default _gateway 0.0.0.0 UG 0 0 0 enp3s0f1 2024-09-16T15:43:48.780 INFO:teuthology.orchestra.run.smithi123.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-09-16T15:43:48.780 INFO:teuthology.orchestra.run.smithi123.stdout:158.69.67.47 _gateway 255.255.255.255 UGH 100 0 0 enp3s0f1 2024-09-16T15:43:48.780 INFO:teuthology.orchestra.run.smithi123.stdout:172.17.0.0 0.0.0.0 255.255.0.0 U 0 0 0 docker0 2024-09-16T15:43:48.780 INFO:teuthology.orchestra.run.smithi123.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-09-16T15:43:48.780 INFO:teuthology.orchestra.run.smithi123.stdout:ns1.front.sepia 0.0.0.0 255.255.255.255 UH 100 0 0 enp3s0f1 2024-09-16T15:43:48.781 INFO:teuthology.orchestra.run.smithi123.stdout:172.21.0.2 0.0.0.0 255.255.255.255 UH 100 0 0 enp3s0f1 2024-09-16T15:43:48.781 INFO:teuthology.orchestra.run.smithi123.stdout:_gateway 0.0.0.0 255.255.255.255 UH 100 0 0 enp3s0f1 2024-09-16T15:43:48.781 INFO:teuthology.orchestra.run.smithi123.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-09-16T15:43:48.781 DEBUG:teuthology.orchestra.run.smithi123:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-09-16T15:43:48.781 DEBUG:teuthology.orchestra.run.smithi123:> set -e 2024-09-16T15:43:48.781 DEBUG:teuthology.orchestra.run.smithi123:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-09-16T15:43:48.781 DEBUG:teuthology.orchestra.run.smithi123:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-09-16T15:43:48.781 DEBUG:teuthology.orchestra.run.smithi123:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-09-16T15:43:48.781 DEBUG:teuthology.orchestra.run.smithi123:> ') 2024-09-16T15:43:48.867 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:48 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-09-16T15:43:48.894 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:48 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-16T15:43:48.901 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:43:48.901 DEBUG:teuthology.orchestra.run.smithi123:> ip netns list 2024-09-16T15:43:48.948 INFO:teuthology.orchestra.run.smithi123.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-09-16T15:43:48.948 INFO:teuthology.orchestra.run.smithi123.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-09-16T15:43:48.948 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-09-16T15:43:48.949 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:43:48.949 DEBUG:teuthology.orchestra.run.smithi123:> sudo ip link set brx.0 up 2024-09-16T15:43:49.003 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2024-09-16T15:43:49.004 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-09-16T15:43:49.004 DEBUG:teuthology.orchestra.run.smithi123:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-09-16T15:43:49.051 INFO:teuthology.orchestra.run.smithi123.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-09-16T15:43:49.052 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-09-16T15:43:49.052 DEBUG:teuthology.orchestra.run.smithi123:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-09-16T15:43:49.100 DEBUG:teuthology.orchestra.run.smithi123:> sudo modprobe fuse 2024-09-16T15:43:49.157 DEBUG:teuthology.orchestra.run.smithi123:> cat /proc/self/mounts | awk '{print $2}' 2024-09-16T15:43:49.205 INFO:teuthology.orchestra.run.smithi123.stdout:/sys 2024-09-16T15:43:49.205 INFO:teuthology.orchestra.run.smithi123.stdout:/proc 2024-09-16T15:43:49.205 INFO:teuthology.orchestra.run.smithi123.stdout:/dev 2024-09-16T15:43:49.205 INFO:teuthology.orchestra.run.smithi123.stdout:/dev/pts 2024-09-16T15:43:49.205 INFO:teuthology.orchestra.run.smithi123.stdout:/run 2024-09-16T15:43:49.205 INFO:teuthology.orchestra.run.smithi123.stdout:/ 2024-09-16T15:43:49.205 INFO:teuthology.orchestra.run.smithi123.stdout:/sys/kernel/security 2024-09-16T15:43:49.205 INFO:teuthology.orchestra.run.smithi123.stdout:/dev/shm 2024-09-16T15:43:49.206 INFO:teuthology.orchestra.run.smithi123.stdout:/run/lock 2024-09-16T15:43:49.206 INFO:teuthology.orchestra.run.smithi123.stdout:/sys/fs/cgroup 2024-09-16T15:43:49.206 INFO:teuthology.orchestra.run.smithi123.stdout:/sys/fs/pstore 2024-09-16T15:43:49.206 INFO:teuthology.orchestra.run.smithi123.stdout:/sys/fs/bpf 2024-09-16T15:43:49.206 INFO:teuthology.orchestra.run.smithi123.stdout:/proc/sys/fs/binfmt_misc 2024-09-16T15:43:49.206 INFO:teuthology.orchestra.run.smithi123.stdout:/dev/hugepages 2024-09-16T15:43:49.206 INFO:teuthology.orchestra.run.smithi123.stdout:/dev/mqueue 2024-09-16T15:43:49.206 INFO:teuthology.orchestra.run.smithi123.stdout:/sys/kernel/debug 2024-09-16T15:43:49.206 INFO:teuthology.orchestra.run.smithi123.stdout:/sys/kernel/tracing 2024-09-16T15:43:49.206 INFO:teuthology.orchestra.run.smithi123.stdout:/proc/fs/nfsd 2024-09-16T15:43:49.206 INFO:teuthology.orchestra.run.smithi123.stdout:/sys/fs/fuse/connections 2024-09-16T15:43:49.206 INFO:teuthology.orchestra.run.smithi123.stdout:/sys/kernel/config 2024-09-16T15:43:49.206 INFO:teuthology.orchestra.run.smithi123.stdout:/run/credentials/systemd-sysusers.service 2024-09-16T15:43:49.206 INFO:teuthology.orchestra.run.smithi123.stdout:/run/rpc_pipefs 2024-09-16T15:43:49.207 INFO:teuthology.orchestra.run.smithi123.stdout:/run/user/1000 2024-09-16T15:43:49.207 INFO:teuthology.orchestra.run.smithi123.stdout:/var/lib/ceph 2024-09-16T15:43:49.207 INFO:teuthology.orchestra.run.smithi123.stdout:/var/lib/ceph/osd/ceph-0 2024-09-16T15:43:49.207 INFO:teuthology.orchestra.run.smithi123.stdout:/var/lib/ceph/osd/ceph-1 2024-09-16T15:43:49.207 INFO:teuthology.orchestra.run.smithi123.stdout:/var/lib/ceph/osd/ceph-2 2024-09-16T15:43:49.207 INFO:teuthology.orchestra.run.smithi123.stdout:/var/lib/ceph/osd/ceph-3 2024-09-16T15:43:49.207 INFO:teuthology.orchestra.run.smithi123.stdout:/run/netns 2024-09-16T15:43:49.207 INFO:teuthology.orchestra.run.smithi123.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-09-16T15:43:49.207 INFO:teuthology.orchestra.run.smithi123.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-09-16T15:43:49.207 INFO:teuthology.orchestra.run.smithi123.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-09-16T15:43:49.207 INFO:teuthology.orchestra.run.smithi123.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-09-16T15:43:49.208 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:43:49.208 DEBUG:teuthology.orchestra.run.smithi123:> ls /sys/fs/fuse/connections 2024-09-16T15:43:49.256 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-09-16T15:43:49.256 DEBUG:teuthology.orchestra.run.smithi123:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2024-09-16T15:43:49.301 DEBUG:teuthology.orchestra.run.smithi123:> sudo modprobe fuse 2024-09-16T15:43:49.316 DEBUG:teuthology.orchestra.run.smithi123:> cat /proc/self/mounts | awk '{print $2}' 2024-09-16T15:43:49.364 INFO:teuthology.orchestra.run.smithi123.stdout:/sys 2024-09-16T15:43:49.364 INFO:teuthology.orchestra.run.smithi123.stdout:/proc 2024-09-16T15:43:49.364 INFO:teuthology.orchestra.run.smithi123.stdout:/dev 2024-09-16T15:43:49.364 INFO:teuthology.orchestra.run.smithi123.stdout:/dev/pts 2024-09-16T15:43:49.364 INFO:teuthology.orchestra.run.smithi123.stdout:/run 2024-09-16T15:43:49.364 INFO:teuthology.orchestra.run.smithi123.stdout:/ 2024-09-16T15:43:49.364 INFO:teuthology.orchestra.run.smithi123.stdout:/sys/kernel/security 2024-09-16T15:43:49.364 INFO:teuthology.orchestra.run.smithi123.stdout:/dev/shm 2024-09-16T15:43:49.364 INFO:teuthology.orchestra.run.smithi123.stdout:/run/lock 2024-09-16T15:43:49.364 INFO:teuthology.orchestra.run.smithi123.stdout:/sys/fs/cgroup 2024-09-16T15:43:49.365 INFO:teuthology.orchestra.run.smithi123.stdout:/sys/fs/pstore 2024-09-16T15:43:49.365 INFO:teuthology.orchestra.run.smithi123.stdout:/sys/fs/bpf 2024-09-16T15:43:49.365 INFO:teuthology.orchestra.run.smithi123.stdout:/proc/sys/fs/binfmt_misc 2024-09-16T15:43:49.365 INFO:teuthology.orchestra.run.smithi123.stdout:/dev/hugepages 2024-09-16T15:43:49.365 INFO:teuthology.orchestra.run.smithi123.stdout:/dev/mqueue 2024-09-16T15:43:49.365 INFO:teuthology.orchestra.run.smithi123.stdout:/sys/kernel/debug 2024-09-16T15:43:49.365 INFO:teuthology.orchestra.run.smithi123.stdout:/sys/kernel/tracing 2024-09-16T15:43:49.365 INFO:teuthology.orchestra.run.smithi123.stdout:/proc/fs/nfsd 2024-09-16T15:43:49.365 INFO:teuthology.orchestra.run.smithi123.stdout:/sys/fs/fuse/connections 2024-09-16T15:43:49.365 INFO:teuthology.orchestra.run.smithi123.stdout:/sys/kernel/config 2024-09-16T15:43:49.365 INFO:teuthology.orchestra.run.smithi123.stdout:/run/credentials/systemd-sysusers.service 2024-09-16T15:43:49.365 INFO:teuthology.orchestra.run.smithi123.stdout:/run/rpc_pipefs 2024-09-16T15:43:49.365 INFO:teuthology.orchestra.run.smithi123.stdout:/run/user/1000 2024-09-16T15:43:49.366 INFO:teuthology.orchestra.run.smithi123.stdout:/var/lib/ceph 2024-09-16T15:43:49.366 INFO:teuthology.orchestra.run.smithi123.stdout:/var/lib/ceph/osd/ceph-0 2024-09-16T15:43:49.366 INFO:teuthology.orchestra.run.smithi123.stdout:/var/lib/ceph/osd/ceph-1 2024-09-16T15:43:49.366 INFO:teuthology.orchestra.run.smithi123.stdout:/var/lib/ceph/osd/ceph-2 2024-09-16T15:43:49.366 INFO:teuthology.orchestra.run.smithi123.stdout:/var/lib/ceph/osd/ceph-3 2024-09-16T15:43:49.366 INFO:teuthology.orchestra.run.smithi123.stdout:/run/netns 2024-09-16T15:43:49.366 INFO:teuthology.orchestra.run.smithi123.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-09-16T15:43:49.366 INFO:teuthology.orchestra.run.smithi123.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-09-16T15:43:49.366 INFO:teuthology.orchestra.run.smithi123.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-09-16T15:43:49.366 INFO:teuthology.orchestra.run.smithi123.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-09-16T15:43:49.367 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:43:49.367 DEBUG:teuthology.orchestra.run.smithi123:> ls /sys/fs/fuse/connections 2024-09-16T15:43:49.382 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi123.stderr:2024-09-16T15:43:49.376+0000 7f2fc64e0580 -1 init, newargv = 0x55d6cd751b80 newargc=15 2024-09-16T15:43:49.382 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi123.stderr:ceph-fuse[37907]: starting ceph client 2024-09-16T15:43:49.387 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi123.stderr:ceph-fuse[37907]: starting fuse 2024-09-16T15:43:50.487 DEBUG:teuthology.orchestra.run.smithi123:> sudo modprobe fuse 2024-09-16T15:43:50.505 DEBUG:teuthology.orchestra.run.smithi123:> cat /proc/self/mounts | awk '{print $2}' 2024-09-16T15:43:50.553 INFO:teuthology.orchestra.run.smithi123.stdout:/sys 2024-09-16T15:43:50.553 INFO:teuthology.orchestra.run.smithi123.stdout:/proc 2024-09-16T15:43:50.553 INFO:teuthology.orchestra.run.smithi123.stdout:/dev 2024-09-16T15:43:50.553 INFO:teuthology.orchestra.run.smithi123.stdout:/dev/pts 2024-09-16T15:43:50.553 INFO:teuthology.orchestra.run.smithi123.stdout:/run 2024-09-16T15:43:50.553 INFO:teuthology.orchestra.run.smithi123.stdout:/ 2024-09-16T15:43:50.553 INFO:teuthology.orchestra.run.smithi123.stdout:/sys/kernel/security 2024-09-16T15:43:50.553 INFO:teuthology.orchestra.run.smithi123.stdout:/dev/shm 2024-09-16T15:43:50.553 INFO:teuthology.orchestra.run.smithi123.stdout:/run/lock 2024-09-16T15:43:50.553 INFO:teuthology.orchestra.run.smithi123.stdout:/sys/fs/cgroup 2024-09-16T15:43:50.554 INFO:teuthology.orchestra.run.smithi123.stdout:/sys/fs/pstore 2024-09-16T15:43:50.554 INFO:teuthology.orchestra.run.smithi123.stdout:/sys/fs/bpf 2024-09-16T15:43:50.554 INFO:teuthology.orchestra.run.smithi123.stdout:/proc/sys/fs/binfmt_misc 2024-09-16T15:43:50.554 INFO:teuthology.orchestra.run.smithi123.stdout:/dev/hugepages 2024-09-16T15:43:50.554 INFO:teuthology.orchestra.run.smithi123.stdout:/dev/mqueue 2024-09-16T15:43:50.554 INFO:teuthology.orchestra.run.smithi123.stdout:/sys/kernel/debug 2024-09-16T15:43:50.554 INFO:teuthology.orchestra.run.smithi123.stdout:/sys/kernel/tracing 2024-09-16T15:43:50.554 INFO:teuthology.orchestra.run.smithi123.stdout:/proc/fs/nfsd 2024-09-16T15:43:50.554 INFO:teuthology.orchestra.run.smithi123.stdout:/sys/fs/fuse/connections 2024-09-16T15:43:50.554 INFO:teuthology.orchestra.run.smithi123.stdout:/sys/kernel/config 2024-09-16T15:43:50.554 INFO:teuthology.orchestra.run.smithi123.stdout:/run/credentials/systemd-sysusers.service 2024-09-16T15:43:50.554 INFO:teuthology.orchestra.run.smithi123.stdout:/run/rpc_pipefs 2024-09-16T15:43:50.554 INFO:teuthology.orchestra.run.smithi123.stdout:/run/user/1000 2024-09-16T15:43:50.554 INFO:teuthology.orchestra.run.smithi123.stdout:/var/lib/ceph 2024-09-16T15:43:50.555 INFO:teuthology.orchestra.run.smithi123.stdout:/var/lib/ceph/osd/ceph-0 2024-09-16T15:43:50.555 INFO:teuthology.orchestra.run.smithi123.stdout:/var/lib/ceph/osd/ceph-1 2024-09-16T15:43:50.555 INFO:teuthology.orchestra.run.smithi123.stdout:/var/lib/ceph/osd/ceph-2 2024-09-16T15:43:50.555 INFO:teuthology.orchestra.run.smithi123.stdout:/var/lib/ceph/osd/ceph-3 2024-09-16T15:43:50.555 INFO:teuthology.orchestra.run.smithi123.stdout:/run/netns 2024-09-16T15:43:50.555 INFO:teuthology.orchestra.run.smithi123.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-09-16T15:43:50.555 INFO:teuthology.orchestra.run.smithi123.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-09-16T15:43:50.555 INFO:teuthology.orchestra.run.smithi123.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-09-16T15:43:50.555 INFO:teuthology.orchestra.run.smithi123.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-09-16T15:43:50.555 INFO:teuthology.orchestra.run.smithi123.stdout:/home/ubuntu/cephtest/mnt.0 2024-09-16T15:43:50.556 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:43:50.556 DEBUG:teuthology.orchestra.run.smithi123:> ls /sys/fs/fuse/connections 2024-09-16T15:43:50.603 INFO:teuthology.orchestra.run.smithi123.stdout:45 2024-09-16T15:43:50.604 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [45] 2024-09-16T15:43:50.604 DEBUG:teuthology.orchestra.run.smithi123:> sudo stdin-killer -- python3 -c ' 2024-09-16T15:43:50.604 DEBUG:teuthology.orchestra.run.smithi123:> import glob 2024-09-16T15:43:50.604 DEBUG:teuthology.orchestra.run.smithi123:> import re 2024-09-16T15:43:50.604 DEBUG:teuthology.orchestra.run.smithi123:> import os 2024-09-16T15:43:50.604 DEBUG:teuthology.orchestra.run.smithi123:> import subprocess 2024-09-16T15:43:50.604 DEBUG:teuthology.orchestra.run.smithi123:> 2024-09-16T15:43:50.604 DEBUG:teuthology.orchestra.run.smithi123:> def _find_admin_socket(client_name): 2024-09-16T15:43:50.604 DEBUG:teuthology.orchestra.run.smithi123:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2024-09-16T15:43:50.604 DEBUG:teuthology.orchestra.run.smithi123:> files = glob.glob(asok_path) 2024-09-16T15:43:50.604 DEBUG:teuthology.orchestra.run.smithi123:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-09-16T15:43:50.604 DEBUG:teuthology.orchestra.run.smithi123:> 2024-09-16T15:43:50.605 DEBUG:teuthology.orchestra.run.smithi123:> # Given a non-glob path, it better be there 2024-09-16T15:43:50.605 DEBUG:teuthology.orchestra.run.smithi123:> if "*" not in asok_path: 2024-09-16T15:43:50.605 DEBUG:teuthology.orchestra.run.smithi123:> assert(len(files) == 1) 2024-09-16T15:43:50.605 DEBUG:teuthology.orchestra.run.smithi123:> return files[0] 2024-09-16T15:43:50.605 DEBUG:teuthology.orchestra.run.smithi123:> 2024-09-16T15:43:50.605 DEBUG:teuthology.orchestra.run.smithi123:> for f in files: 2024-09-16T15:43:50.605 DEBUG:teuthology.orchestra.run.smithi123:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-09-16T15:43:50.605 DEBUG:teuthology.orchestra.run.smithi123:> if os.path.exists("/proc/{0}".format(pid)): 2024-09-16T15:43:50.605 DEBUG:teuthology.orchestra.run.smithi123:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-09-16T15:43:50.605 DEBUG:teuthology.orchestra.run.smithi123:> contents = proc_f.read() 2024-09-16T15:43:50.605 DEBUG:teuthology.orchestra.run.smithi123:> if mountpoint in contents: 2024-09-16T15:43:50.605 DEBUG:teuthology.orchestra.run.smithi123:> return f 2024-09-16T15:43:50.605 DEBUG:teuthology.orchestra.run.smithi123:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-09-16T15:43:50.605 DEBUG:teuthology.orchestra.run.smithi123:> 2024-09-16T15:43:50.605 DEBUG:teuthology.orchestra.run.smithi123:> print(_find_admin_socket("client.0")) 2024-09-16T15:43:50.605 DEBUG:teuthology.orchestra.run.smithi123:> ' 2024-09-16T15:43:50.729 INFO:teuthology.orchestra.run.smithi123.stdout:/var/run/ceph/ceph-client.0.37907.asok 2024-09-16T15:43:50.733 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:50 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-16T15:43:50.741 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.37907.asok 2024-09-16T15:43:50.741 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:43:50.741 DEBUG:teuthology.orchestra.run.smithi123:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.37907.asok status 2024-09-16T15:43:50.892 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2024-09-16T15:43:50.892 INFO:teuthology.orchestra.run.smithi123.stdout: "metadata": { 2024-09-16T15:43:50.892 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph_sha1": "33f95d2ae1bcbe7eb04404303f179eeeaa6982df", 2024-09-16T15:43:50.893 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph_version": "ceph version 19.3.0-4938-g33f95d2a (33f95d2ae1bcbe7eb04404303f179eeeaa6982df) squid (dev)", 2024-09-16T15:43:50.893 INFO:teuthology.orchestra.run.smithi123.stdout: "entity_id": "0", 2024-09-16T15:43:50.893 INFO:teuthology.orchestra.run.smithi123.stdout: "hostname": "smithi123", 2024-09-16T15:43:50.893 INFO:teuthology.orchestra.run.smithi123.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2024-09-16T15:43:50.893 INFO:teuthology.orchestra.run.smithi123.stdout: "pid": "37907", 2024-09-16T15:43:50.893 INFO:teuthology.orchestra.run.smithi123.stdout: "root": "/" 2024-09-16T15:43:50.893 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:43:50.893 INFO:teuthology.orchestra.run.smithi123.stdout: "dentry_count": 0, 2024-09-16T15:43:50.893 INFO:teuthology.orchestra.run.smithi123.stdout: "dentry_pinned_count": 0, 2024-09-16T15:43:50.893 INFO:teuthology.orchestra.run.smithi123.stdout: "id": 5049, 2024-09-16T15:43:50.893 INFO:teuthology.orchestra.run.smithi123.stdout: "inst": { 2024-09-16T15:43:50.893 INFO:teuthology.orchestra.run.smithi123.stdout: "name": { 2024-09-16T15:43:50.893 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "client", 2024-09-16T15:43:50.894 INFO:teuthology.orchestra.run.smithi123.stdout: "num": 5049 2024-09-16T15:43:50.894 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:43:50.894 INFO:teuthology.orchestra.run.smithi123.stdout: "addr": { 2024-09-16T15:43:50.894 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "v1", 2024-09-16T15:43:50.894 INFO:teuthology.orchestra.run.smithi123.stdout: "addr": "192.168.0.1:0", 2024-09-16T15:43:50.894 INFO:teuthology.orchestra.run.smithi123.stdout: "nonce": 2346128290 2024-09-16T15:43:50.894 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-16T15:43:50.894 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:43:50.894 INFO:teuthology.orchestra.run.smithi123.stdout: "addr": { 2024-09-16T15:43:50.894 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "v1", 2024-09-16T15:43:50.894 INFO:teuthology.orchestra.run.smithi123.stdout: "addr": "192.168.0.1:0", 2024-09-16T15:43:50.894 INFO:teuthology.orchestra.run.smithi123.stdout: "nonce": 2346128290 2024-09-16T15:43:50.894 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:43:50.894 INFO:teuthology.orchestra.run.smithi123.stdout: "inst_str": "client.5049 192.168.0.1:0/2346128290", 2024-09-16T15:43:50.895 INFO:teuthology.orchestra.run.smithi123.stdout: "addr_str": "192.168.0.1:0/2346128290", 2024-09-16T15:43:50.895 INFO:teuthology.orchestra.run.smithi123.stdout: "inode_count": 1, 2024-09-16T15:43:50.895 INFO:teuthology.orchestra.run.smithi123.stdout: "mds_epoch": 18, 2024-09-16T15:43:50.895 INFO:teuthology.orchestra.run.smithi123.stdout: "osd_epoch": 33, 2024-09-16T15:43:50.895 INFO:teuthology.orchestra.run.smithi123.stdout: "osd_epoch_barrier": 0, 2024-09-16T15:43:50.895 INFO:teuthology.orchestra.run.smithi123.stdout: "blocklisted": false, 2024-09-16T15:43:50.895 INFO:teuthology.orchestra.run.smithi123.stdout: "fs_name": "cephfs" 2024-09-16T15:43:50.895 INFO:teuthology.orchestra.run.smithi123.stdout:} 2024-09-16T15:43:50.906 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:43:50.906 DEBUG:teuthology.orchestra.run.smithi123:> stat --file-system '--printf=%T 2024-09-16T15:43:50.906 DEBUG:teuthology.orchestra.run.smithi123:> ' -- /home/ubuntu/cephtest/mnt.0 2024-09-16T15:43:50.911 INFO:teuthology.orchestra.run.smithi123.stdout:fuseblk 2024-09-16T15:43:50.912 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2024-09-16T15:43:50.912 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:43:50.912 DEBUG:teuthology.orchestra.run.smithi123:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-09-16T15:43:50.968 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-09-16T15:43:51.138 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:51.132+0000 7fc79ada9640 1 -- 172.21.15.123:0/2782863487 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc77c004d50 msgr2=0x7fc77c0051d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:43:51.138 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:51.132+0000 7fc79ada9640 1 --2- 172.21.15.123:0/2782863487 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc77c004d50 0x7fc77c0051d0 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7fc78c009ee0 tx=0x7fc78c01d5f0 comp rx=0 tx=0).stop 2024-09-16T15:43:51.138 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:51.132+0000 7fc799346640 1 -- 172.21.15.123:0/2782863487 reap_dead start 2024-09-16T15:43:51.138 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:51.132+0000 7fc79ada9640 1 -- 172.21.15.123:0/2782863487 shutdown_connections 2024-09-16T15:43:51.138 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:51.132+0000 7fc79ada9640 1 -- 172.21.15.123:0/2782863487 >> 172.21.15.123:0/2782863487 conn(0x7fc7940d0b70 msgr2=0x7fc7940d0f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:43:51.138 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:51.132+0000 7fc79ada9640 1 -- 172.21.15.123:0/2782863487 shutdown_connections 2024-09-16T15:43:51.138 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:51.132+0000 7fc79ada9640 1 -- 172.21.15.123:0/2782863487 wait complete. 2024-09-16T15:43:51.139 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:51.136+0000 7fc79ada9640 1 Processor -- start 2024-09-16T15:43:51.139 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:51.136+0000 7fc79ada9640 1 -- start start 2024-09-16T15:43:51.141 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:51.136+0000 7fc79ada9640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc77c004d50 0x7fc7940b1ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:43:51.141 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:51.136+0000 7fc79ada9640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc79415b9f0 0x7fc794154b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:43:51.141 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:51.136+0000 7fc79ada9640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fc7940b27e0 0x7fc794155090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:43:51.141 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:51.136+0000 7fc79ada9640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fc794167060 con 0x7fc77c004d50 2024-09-16T15:43:51.141 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:51.136+0000 7fc79ada9640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fc794166ee0 con 0x7fc79415b9f0 2024-09-16T15:43:51.141 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:51.136+0000 7fc79ada9640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fc7941671e0 con 0x7fc7940b27e0 2024-09-16T15:43:51.141 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:51.136+0000 7fc798b45640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc77c004d50 0x7fc7940b1ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:43:51.141 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:51.136+0000 7fc798b45640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc77c004d50 0x7fc7940b1ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:41808/0 (socket says 172.21.15.123:41808) 2024-09-16T15:43:51.141 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:51.136+0000 7fc798b45640 1 -- 172.21.15.123:0/2303557470 learned_addr learned my addr 172.21.15.123:0/2303557470 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:43:51.141 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:51.136+0000 7fc793fff640 1 --2- 172.21.15.123:0/2303557470 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc79415b9f0 0x7fc794154b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:43:51.141 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:51.136+0000 7fc798b45640 1 -- 172.21.15.123:0/2303557470 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fc7940b27e0 msgr2=0x7fc794155090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:43:51.141 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:51.136+0000 7fc798b45640 1 --2- 172.21.15.123:0/2303557470 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fc7940b27e0 0x7fc794155090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:51.142 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:51.136+0000 7fc798b45640 1 -- 172.21.15.123:0/2303557470 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc79415b9f0 msgr2=0x7fc794154b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:43:51.142 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:51.136+0000 7fc798b45640 1 --2- 172.21.15.123:0/2303557470 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc79415b9f0 0x7fc794154b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:51.142 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:51.136+0000 7fc798b45640 1 -- 172.21.15.123:0/2303557470 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc79415bf60 con 0x7fc77c004d50 2024-09-16T15:43:51.142 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:51.136+0000 7fc798b45640 1 --2- 172.21.15.123:0/2303557470 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc77c004d50 0x7fc7940b1ed0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7fc78400d8f0 tx=0x7fc78400ddb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:43:51.142 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:51.136+0000 7fc7917fa640 1 -- 172.21.15.123:0/2303557470 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc784001d40 con 0x7fc77c004d50 2024-09-16T15:43:51.142 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:51.136+0000 7fc7917fa640 1 -- 172.21.15.123:0/2303557470 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fc784015020 con 0x7fc77c004d50 2024-09-16T15:43:51.142 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:51.136+0000 7fc7917fa640 1 -- 172.21.15.123:0/2303557470 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc78400b7b0 con 0x7fc77c004d50 2024-09-16T15:43:51.143 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:51.136+0000 7fc79ada9640 1 -- 172.21.15.123:0/2303557470 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc79415c240 con 0x7fc77c004d50 2024-09-16T15:43:51.143 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:51.136+0000 7fc79ada9640 1 -- 172.21.15.123:0/2303557470 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc79415c800 con 0x7fc77c004d50 2024-09-16T15:43:51.143 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:51.136+0000 7fc79ada9640 1 -- 172.21.15.123:0/2303557470 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc79415e830 con 0x7fc77c004d50 2024-09-16T15:43:51.143 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:51.136+0000 7fc7917fa640 1 -- 172.21.15.123:0/2303557470 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 5) ==== 97080+0+0 (secure 0 0 0) 0x7fc78400b950 con 0x7fc77c004d50 2024-09-16T15:43:51.143 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:51.140+0000 7fc7917fa640 1 --2- 172.21.15.123:0/2303557470 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fc780073090 0x7fc780075550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:43:51.144 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:51.140+0000 7fc7917fa640 1 -- 172.21.15.123:0/2303557470 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(33..33 src has 1..33) ==== 5709+0+0 (secure 0 0 0) 0x7fc784094570 con 0x7fc77c004d50 2024-09-16T15:43:51.144 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:51.140+0000 7fc793fff640 1 --2- 172.21.15.123:0/2303557470 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fc780073090 0x7fc780075550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:43:51.144 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:51.140+0000 7fc793fff640 1 --2- 172.21.15.123:0/2303557470 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fc780073090 0x7fc780075550 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fc788001250 tx=0x7fc788001540 comp rx=0 tx=0).ready entity=mgr.4135 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:43:51.147 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:51.144+0000 7fc7917fa640 1 -- 172.21.15.123:0/2303557470 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7fc78401f020 con 0x7fc77c004d50 2024-09-16T15:43:51.357 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:51.352+0000 7fc79ada9640 1 -- 172.21.15.123:0/2303557470 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7fc7940b6af0 con 0x7fc77c004d50 2024-09-16T15:43:51.358 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:51.352+0000 7fc7917fa640 1 -- 172.21.15.123:0/2303557470 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v18) ==== 53+0+73 (secure 0 0 0) 0x7fc78405f0d0 con 0x7fc77c004d50 2024-09-16T15:43:51.358 INFO:teuthology.orchestra.run.smithi123.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-09-16T15:43:51.361 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:51.356+0000 7fc79ada9640 1 -- 172.21.15.123:0/2303557470 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fc780073090 msgr2=0x7fc780075550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:43:51.361 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:51.356+0000 7fc79ada9640 1 --2- 172.21.15.123:0/2303557470 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fc780073090 0x7fc780075550 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fc788001250 tx=0x7fc788001540 comp rx=0 tx=0).stop 2024-09-16T15:43:51.361 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:51.356+0000 7fc79ada9640 1 -- 172.21.15.123:0/2303557470 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc77c004d50 msgr2=0x7fc7940b1ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:43:51.361 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:51.356+0000 7fc79ada9640 1 --2- 172.21.15.123:0/2303557470 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc77c004d50 0x7fc7940b1ed0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7fc78400d8f0 tx=0x7fc78400ddb0 comp rx=0 tx=0).stop 2024-09-16T15:43:51.361 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:51.356+0000 7fc79ada9640 1 -- 172.21.15.123:0/2303557470 shutdown_connections 2024-09-16T15:43:51.361 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:51.356+0000 7fc79ada9640 1 --2- 172.21.15.123:0/2303557470 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fc780073090 0x7fc780075550 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:51.361 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:51.356+0000 7fc79ada9640 1 --2- 172.21.15.123:0/2303557470 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fc7940b27e0 0x7fc794155090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:51.361 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:51.356+0000 7fc79ada9640 1 --2- 172.21.15.123:0/2303557470 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc79415b9f0 0x7fc794154b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:51.361 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:51.356+0000 7fc79ada9640 1 --2- 172.21.15.123:0/2303557470 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc77c004d50 0x7fc7940b1ed0 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:51.361 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:51.356+0000 7fc79ada9640 1 -- 172.21.15.123:0/2303557470 >> 172.21.15.123:0/2303557470 conn(0x7fc7940d0b70 msgr2=0x7fc7940ca360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:43:51.361 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:51.356+0000 7fc79ada9640 1 -- 172.21.15.123:0/2303557470 shutdown_connections 2024-09-16T15:43:51.361 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:51.356+0000 7fc79ada9640 1 -- 172.21.15.123:0/2303557470 wait complete. 2024-09-16T15:43:51.375 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-09-16T15:43:51.375 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-09-16T15:43:51.375 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi184.front.sepia.ceph.com 2024-09-16T15:43:51.376 INFO:tasks.cephfs.mount:self.client.name = client.1 2024-09-16T15:43:51.376 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2024-09-16T15:43:51.376 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-09-16T15:43:51.376 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-09-16T15:43:51.376 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-09-16T15:43:51.376 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.1' netns for '/home/ubuntu/cephtest/mnt.1' 2024-09-16T15:43:51.376 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:43:51.376 DEBUG:teuthology.orchestra.run.smithi184:> ip addr 2024-09-16T15:43:51.383 INFO:teuthology.orchestra.run.smithi184.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-09-16T15:43:51.383 INFO:teuthology.orchestra.run.smithi184.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-09-16T15:43:51.383 INFO:teuthology.orchestra.run.smithi184.stdout: inet 127.0.0.1/8 scope host lo 2024-09-16T15:43:51.384 INFO:teuthology.orchestra.run.smithi184.stdout: valid_lft forever preferred_lft forever 2024-09-16T15:43:51.384 INFO:teuthology.orchestra.run.smithi184.stdout: inet6 ::1/128 scope host 2024-09-16T15:43:51.384 INFO:teuthology.orchestra.run.smithi184.stdout: valid_lft forever preferred_lft forever 2024-09-16T15:43:51.384 INFO:teuthology.orchestra.run.smithi184.stdout:2: eno1: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-09-16T15:43:51.384 INFO:teuthology.orchestra.run.smithi184.stdout: link/ether 0c:c4:7a:d9:37:16 brd ff:ff:ff:ff:ff:ff 2024-09-16T15:43:51.384 INFO:teuthology.orchestra.run.smithi184.stdout: altname enp7s0f0 2024-09-16T15:43:51.384 INFO:teuthology.orchestra.run.smithi184.stdout:3: eno2: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-09-16T15:43:51.384 INFO:teuthology.orchestra.run.smithi184.stdout: link/ether 0c:c4:7a:d9:37:17 brd ff:ff:ff:ff:ff:ff 2024-09-16T15:43:51.384 INFO:teuthology.orchestra.run.smithi184.stdout: altname enp7s0f1 2024-09-16T15:43:51.384 INFO:teuthology.orchestra.run.smithi184.stdout:4: enp3s0f0: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-09-16T15:43:51.384 INFO:teuthology.orchestra.run.smithi184.stdout: link/ether 0c:c4:7a:88:7b:8e brd ff:ff:ff:ff:ff:ff 2024-09-16T15:43:51.384 INFO:teuthology.orchestra.run.smithi184.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-09-16T15:43:51.384 INFO:teuthology.orchestra.run.smithi184.stdout: link/ether 0c:c4:7a:88:7b:8f brd ff:ff:ff:ff:ff:ff 2024-09-16T15:43:51.384 INFO:teuthology.orchestra.run.smithi184.stdout: inet 172.21.15.184/20 metric 100 brd 172.21.15.255 scope global dynamic enp3s0f1 2024-09-16T15:43:51.385 INFO:teuthology.orchestra.run.smithi184.stdout: valid_lft 42426sec preferred_lft 42426sec 2024-09-16T15:43:51.385 INFO:teuthology.orchestra.run.smithi184.stdout: inet6 fe80::ec4:7aff:fe88:7b8f/64 scope link 2024-09-16T15:43:51.385 INFO:teuthology.orchestra.run.smithi184.stdout: valid_lft forever preferred_lft forever 2024-09-16T15:43:51.385 INFO:teuthology.orchestra.run.smithi184.stdout:6: docker0: mtu 1500 qdisc noqueue state DOWN group default 2024-09-16T15:43:51.385 INFO:teuthology.orchestra.run.smithi184.stdout: link/ether 02:42:5d:0a:44:b9 brd ff:ff:ff:ff:ff:ff 2024-09-16T15:43:51.385 INFO:teuthology.orchestra.run.smithi184.stdout: inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0 2024-09-16T15:43:51.385 INFO:teuthology.orchestra.run.smithi184.stdout: valid_lft forever preferred_lft forever 2024-09-16T15:43:51.385 INFO:teuthology.orchestra.run.smithi184.stdout:7: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-09-16T15:43:51.385 INFO:teuthology.orchestra.run.smithi184.stdout: link/ether 5e:1f:84:6d:79:ae brd ff:ff:ff:ff:ff:ff 2024-09-16T15:43:51.385 INFO:teuthology.orchestra.run.smithi184.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-09-16T15:43:51.385 INFO:teuthology.orchestra.run.smithi184.stdout: valid_lft forever preferred_lft forever 2024-09-16T15:43:51.385 INFO:teuthology.orchestra.run.smithi184.stdout: inet6 fe80::5c1f:84ff:fe6d:79ae/64 scope link 2024-09-16T15:43:51.385 INFO:teuthology.orchestra.run.smithi184.stdout: valid_lft forever preferred_lft forever 2024-09-16T15:43:51.385 INFO:teuthology.orchestra.run.smithi184.stdout:8: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-09-16T15:43:51.386 INFO:teuthology.orchestra.run.smithi184.stdout: link/ether 0e:5c:f6:1f:82:2f brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.1 2024-09-16T15:43:51.386 INFO:teuthology.orchestra.run.smithi184.stdout: inet6 fe80::c5c:f6ff:fe1f:822f/64 scope link 2024-09-16T15:43:51.386 INFO:teuthology.orchestra.run.smithi184.stdout: valid_lft forever preferred_lft forever 2024-09-16T15:43:51.386 INFO:teuthology.orchestra.run.smithi184.stdout:9: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-09-16T15:43:51.386 INFO:teuthology.orchestra.run.smithi184.stdout: link/ether c6:a6:3d:69:d0:db brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.3 2024-09-16T15:43:51.386 INFO:teuthology.orchestra.run.smithi184.stdout: inet6 fe80::c4a6:3dff:fe69:d0db/64 scope link 2024-09-16T15:43:51.386 INFO:teuthology.orchestra.run.smithi184.stdout: valid_lft forever preferred_lft forever 2024-09-16T15:43:51.386 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:43:51.386 DEBUG:teuthology.orchestra.run.smithi184:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-09-16T15:43:51.439 INFO:teuthology.orchestra.run.smithi184.stdout:1 2024-09-16T15:43:51.440 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:43:51.441 DEBUG:teuthology.orchestra.run.smithi184:> route 2024-09-16T15:43:51.492 INFO:teuthology.orchestra.run.smithi184.stdout:Kernel IP routing table 2024-09-16T15:43:51.492 INFO:teuthology.orchestra.run.smithi184.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-09-16T15:43:51.492 INFO:teuthology.orchestra.run.smithi184.stdout:default _gateway 0.0.0.0 UG 0 0 0 enp3s0f1 2024-09-16T15:43:51.492 INFO:teuthology.orchestra.run.smithi184.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-09-16T15:43:51.492 INFO:teuthology.orchestra.run.smithi184.stdout:158.69.67.47 _gateway 255.255.255.255 UGH 100 0 0 enp3s0f1 2024-09-16T15:43:51.492 INFO:teuthology.orchestra.run.smithi184.stdout:172.17.0.0 0.0.0.0 255.255.0.0 U 0 0 0 docker0 2024-09-16T15:43:51.492 INFO:teuthology.orchestra.run.smithi184.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-09-16T15:43:51.492 INFO:teuthology.orchestra.run.smithi184.stdout:ns1.front.sepia 0.0.0.0 255.255.255.255 UH 100 0 0 enp3s0f1 2024-09-16T15:43:51.492 INFO:teuthology.orchestra.run.smithi184.stdout:172.21.0.2 0.0.0.0 255.255.255.255 UH 100 0 0 enp3s0f1 2024-09-16T15:43:51.492 INFO:teuthology.orchestra.run.smithi184.stdout:_gateway 0.0.0.0 255.255.255.255 UH 100 0 0 enp3s0f1 2024-09-16T15:43:51.493 INFO:teuthology.orchestra.run.smithi184.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-09-16T15:43:51.493 DEBUG:teuthology.orchestra.run.smithi184:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-09-16T15:43:51.493 DEBUG:teuthology.orchestra.run.smithi184:> set -e 2024-09-16T15:43:51.493 DEBUG:teuthology.orchestra.run.smithi184:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-09-16T15:43:51.493 DEBUG:teuthology.orchestra.run.smithi184:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-09-16T15:43:51.493 DEBUG:teuthology.orchestra.run.smithi184:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-09-16T15:43:51.493 DEBUG:teuthology.orchestra.run.smithi184:> ') 2024-09-16T15:43:51.581 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:43:51 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-09-16T15:43:51.609 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:43:51 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-16T15:43:51.616 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:43:51.616 DEBUG:teuthology.orchestra.run.smithi184:> ip netns list 2024-09-16T15:43:51.621 INFO:teuthology.orchestra.run.smithi184.stdout:ceph-ns--home-ubuntu-cephtest-mnt.3 (id: 1) 2024-09-16T15:43:51.622 INFO:teuthology.orchestra.run.smithi184.stdout:ceph-ns--home-ubuntu-cephtest-mnt.1 (id: 0) 2024-09-16T15:43:51.622 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.1' netns for '/home/ubuntu/cephtest/mnt.1' 2024-09-16T15:43:51.622 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:43:51.622 DEBUG:teuthology.orchestra.run.smithi184:> sudo ip link set brx.0 up 2024-09-16T15:43:51.675 INFO:tasks.cephfs.fuse_mount:Client client.1 config is {} 2024-09-16T15:43:51.675 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-09-16T15:43:51.675 DEBUG:teuthology.orchestra.run.smithi184:> mkdir -p -v /home/ubuntu/cephtest/mnt.1 2024-09-16T15:43:51.723 INFO:teuthology.orchestra.run.smithi184.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.1' 2024-09-16T15:43:51.723 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-09-16T15:43:51.723 DEBUG:teuthology.orchestra.run.smithi184:> chmod 0000 /home/ubuntu/cephtest/mnt.1 2024-09-16T15:43:51.771 DEBUG:teuthology.orchestra.run.smithi184:> sudo modprobe fuse 2024-09-16T15:43:51.828 DEBUG:teuthology.orchestra.run.smithi184:> cat /proc/self/mounts | awk '{print $2}' 2024-09-16T15:43:51.877 INFO:teuthology.orchestra.run.smithi184.stdout:/sys 2024-09-16T15:43:51.877 INFO:teuthology.orchestra.run.smithi184.stdout:/proc 2024-09-16T15:43:51.877 INFO:teuthology.orchestra.run.smithi184.stdout:/dev 2024-09-16T15:43:51.877 INFO:teuthology.orchestra.run.smithi184.stdout:/dev/pts 2024-09-16T15:43:51.877 INFO:teuthology.orchestra.run.smithi184.stdout:/run 2024-09-16T15:43:51.877 INFO:teuthology.orchestra.run.smithi184.stdout:/ 2024-09-16T15:43:51.877 INFO:teuthology.orchestra.run.smithi184.stdout:/sys/kernel/security 2024-09-16T15:43:51.877 INFO:teuthology.orchestra.run.smithi184.stdout:/dev/shm 2024-09-16T15:43:51.877 INFO:teuthology.orchestra.run.smithi184.stdout:/run/lock 2024-09-16T15:43:51.877 INFO:teuthology.orchestra.run.smithi184.stdout:/sys/fs/cgroup 2024-09-16T15:43:51.877 INFO:teuthology.orchestra.run.smithi184.stdout:/sys/fs/pstore 2024-09-16T15:43:51.878 INFO:teuthology.orchestra.run.smithi184.stdout:/sys/fs/bpf 2024-09-16T15:43:51.878 INFO:teuthology.orchestra.run.smithi184.stdout:/proc/sys/fs/binfmt_misc 2024-09-16T15:43:51.878 INFO:teuthology.orchestra.run.smithi184.stdout:/dev/hugepages 2024-09-16T15:43:51.878 INFO:teuthology.orchestra.run.smithi184.stdout:/dev/mqueue 2024-09-16T15:43:51.878 INFO:teuthology.orchestra.run.smithi184.stdout:/sys/kernel/debug 2024-09-16T15:43:51.878 INFO:teuthology.orchestra.run.smithi184.stdout:/sys/kernel/tracing 2024-09-16T15:43:51.878 INFO:teuthology.orchestra.run.smithi184.stdout:/proc/fs/nfsd 2024-09-16T15:43:51.878 INFO:teuthology.orchestra.run.smithi184.stdout:/sys/fs/fuse/connections 2024-09-16T15:43:51.878 INFO:teuthology.orchestra.run.smithi184.stdout:/sys/kernel/config 2024-09-16T15:43:51.878 INFO:teuthology.orchestra.run.smithi184.stdout:/run/credentials/systemd-sysusers.service 2024-09-16T15:43:51.878 INFO:teuthology.orchestra.run.smithi184.stdout:/run/rpc_pipefs 2024-09-16T15:43:51.878 INFO:teuthology.orchestra.run.smithi184.stdout:/run/user/1000 2024-09-16T15:43:51.878 INFO:teuthology.orchestra.run.smithi184.stdout:/var/lib/ceph 2024-09-16T15:43:51.878 INFO:teuthology.orchestra.run.smithi184.stdout:/var/lib/ceph/osd/ceph-4 2024-09-16T15:43:51.879 INFO:teuthology.orchestra.run.smithi184.stdout:/var/lib/ceph/osd/ceph-5 2024-09-16T15:43:51.879 INFO:teuthology.orchestra.run.smithi184.stdout:/var/lib/ceph/osd/ceph-6 2024-09-16T15:43:51.879 INFO:teuthology.orchestra.run.smithi184.stdout:/var/lib/ceph/osd/ceph-7 2024-09-16T15:43:51.879 INFO:teuthology.orchestra.run.smithi184.stdout:/run/netns 2024-09-16T15:43:51.879 INFO:teuthology.orchestra.run.smithi184.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-09-16T15:43:51.879 INFO:teuthology.orchestra.run.smithi184.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-09-16T15:43:51.879 INFO:teuthology.orchestra.run.smithi184.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-09-16T15:43:51.879 INFO:teuthology.orchestra.run.smithi184.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-09-16T15:43:51.880 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:43:51.880 DEBUG:teuthology.orchestra.run.smithi184:> ls /sys/fs/fuse/connections 2024-09-16T15:43:51.927 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-09-16T15:43:51.927 DEBUG:teuthology.orchestra.run.smithi184:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.1 --id 1) 2024-09-16T15:43:51.972 DEBUG:teuthology.orchestra.run.smithi184:> sudo modprobe fuse 2024-09-16T15:43:51.986 DEBUG:teuthology.orchestra.run.smithi184:> cat /proc/self/mounts | awk '{print $2}' 2024-09-16T15:43:52.031 INFO:teuthology.orchestra.run.smithi184.stdout:/sys 2024-09-16T15:43:52.032 INFO:teuthology.orchestra.run.smithi184.stdout:/proc 2024-09-16T15:43:52.032 INFO:teuthology.orchestra.run.smithi184.stdout:/dev 2024-09-16T15:43:52.032 INFO:teuthology.orchestra.run.smithi184.stdout:/dev/pts 2024-09-16T15:43:52.032 INFO:teuthology.orchestra.run.smithi184.stdout:/run 2024-09-16T15:43:52.032 INFO:teuthology.orchestra.run.smithi184.stdout:/ 2024-09-16T15:43:52.032 INFO:teuthology.orchestra.run.smithi184.stdout:/sys/kernel/security 2024-09-16T15:43:52.032 INFO:teuthology.orchestra.run.smithi184.stdout:/dev/shm 2024-09-16T15:43:52.032 INFO:teuthology.orchestra.run.smithi184.stdout:/run/lock 2024-09-16T15:43:52.032 INFO:teuthology.orchestra.run.smithi184.stdout:/sys/fs/cgroup 2024-09-16T15:43:52.032 INFO:teuthology.orchestra.run.smithi184.stdout:/sys/fs/pstore 2024-09-16T15:43:52.032 INFO:teuthology.orchestra.run.smithi184.stdout:/sys/fs/bpf 2024-09-16T15:43:52.032 INFO:teuthology.orchestra.run.smithi184.stdout:/proc/sys/fs/binfmt_misc 2024-09-16T15:43:52.033 INFO:teuthology.orchestra.run.smithi184.stdout:/dev/hugepages 2024-09-16T15:43:52.033 INFO:teuthology.orchestra.run.smithi184.stdout:/dev/mqueue 2024-09-16T15:43:52.033 INFO:teuthology.orchestra.run.smithi184.stdout:/sys/kernel/debug 2024-09-16T15:43:52.033 INFO:teuthology.orchestra.run.smithi184.stdout:/sys/kernel/tracing 2024-09-16T15:43:52.033 INFO:teuthology.orchestra.run.smithi184.stdout:/proc/fs/nfsd 2024-09-16T15:43:52.033 INFO:teuthology.orchestra.run.smithi184.stdout:/sys/fs/fuse/connections 2024-09-16T15:43:52.033 INFO:teuthology.orchestra.run.smithi184.stdout:/sys/kernel/config 2024-09-16T15:43:52.033 INFO:teuthology.orchestra.run.smithi184.stdout:/run/credentials/systemd-sysusers.service 2024-09-16T15:43:52.033 INFO:teuthology.orchestra.run.smithi184.stdout:/run/rpc_pipefs 2024-09-16T15:43:52.033 INFO:teuthology.orchestra.run.smithi184.stdout:/run/user/1000 2024-09-16T15:43:52.033 INFO:teuthology.orchestra.run.smithi184.stdout:/var/lib/ceph 2024-09-16T15:43:52.033 INFO:teuthology.orchestra.run.smithi184.stdout:/var/lib/ceph/osd/ceph-4 2024-09-16T15:43:52.033 INFO:teuthology.orchestra.run.smithi184.stdout:/var/lib/ceph/osd/ceph-5 2024-09-16T15:43:52.033 INFO:teuthology.orchestra.run.smithi184.stdout:/var/lib/ceph/osd/ceph-6 2024-09-16T15:43:52.034 INFO:teuthology.orchestra.run.smithi184.stdout:/var/lib/ceph/osd/ceph-7 2024-09-16T15:43:52.034 INFO:teuthology.orchestra.run.smithi184.stdout:/run/netns 2024-09-16T15:43:52.034 INFO:teuthology.orchestra.run.smithi184.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-09-16T15:43:52.034 INFO:teuthology.orchestra.run.smithi184.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-09-16T15:43:52.034 INFO:teuthology.orchestra.run.smithi184.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-09-16T15:43:52.034 INFO:teuthology.orchestra.run.smithi184.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-09-16T15:43:52.035 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:43:52.035 DEBUG:teuthology.orchestra.run.smithi184:> ls /sys/fs/fuse/connections 2024-09-16T15:43:52.050 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi184.stderr:2024-09-16T15:43:52.045+0000 7fcb06a1e580 -1 init, newargv = 0x55e3b3725bb0 newargc=15 2024-09-16T15:43:52.050 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi184.stderr:ceph-fuse[34638]: starting ceph client 2024-09-16T15:43:52.058 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi184.stderr:ceph-fuse[34638]: starting fuse 2024-09-16T15:43:53.156 DEBUG:teuthology.orchestra.run.smithi184:> sudo modprobe fuse 2024-09-16T15:43:53.172 DEBUG:teuthology.orchestra.run.smithi184:> cat /proc/self/mounts | awk '{print $2}' 2024-09-16T15:43:53.225 INFO:teuthology.orchestra.run.smithi184.stdout:/sys 2024-09-16T15:43:53.225 INFO:teuthology.orchestra.run.smithi184.stdout:/proc 2024-09-16T15:43:53.225 INFO:teuthology.orchestra.run.smithi184.stdout:/dev 2024-09-16T15:43:53.225 INFO:teuthology.orchestra.run.smithi184.stdout:/dev/pts 2024-09-16T15:43:53.225 INFO:teuthology.orchestra.run.smithi184.stdout:/run 2024-09-16T15:43:53.225 INFO:teuthology.orchestra.run.smithi184.stdout:/ 2024-09-16T15:43:53.225 INFO:teuthology.orchestra.run.smithi184.stdout:/sys/kernel/security 2024-09-16T15:43:53.225 INFO:teuthology.orchestra.run.smithi184.stdout:/dev/shm 2024-09-16T15:43:53.225 INFO:teuthology.orchestra.run.smithi184.stdout:/run/lock 2024-09-16T15:43:53.225 INFO:teuthology.orchestra.run.smithi184.stdout:/sys/fs/cgroup 2024-09-16T15:43:53.226 INFO:teuthology.orchestra.run.smithi184.stdout:/sys/fs/pstore 2024-09-16T15:43:53.226 INFO:teuthology.orchestra.run.smithi184.stdout:/sys/fs/bpf 2024-09-16T15:43:53.226 INFO:teuthology.orchestra.run.smithi184.stdout:/proc/sys/fs/binfmt_misc 2024-09-16T15:43:53.226 INFO:teuthology.orchestra.run.smithi184.stdout:/dev/hugepages 2024-09-16T15:43:53.226 INFO:teuthology.orchestra.run.smithi184.stdout:/dev/mqueue 2024-09-16T15:43:53.226 INFO:teuthology.orchestra.run.smithi184.stdout:/sys/kernel/debug 2024-09-16T15:43:53.226 INFO:teuthology.orchestra.run.smithi184.stdout:/sys/kernel/tracing 2024-09-16T15:43:53.226 INFO:teuthology.orchestra.run.smithi184.stdout:/proc/fs/nfsd 2024-09-16T15:43:53.226 INFO:teuthology.orchestra.run.smithi184.stdout:/sys/fs/fuse/connections 2024-09-16T15:43:53.226 INFO:teuthology.orchestra.run.smithi184.stdout:/sys/kernel/config 2024-09-16T15:43:53.226 INFO:teuthology.orchestra.run.smithi184.stdout:/run/credentials/systemd-sysusers.service 2024-09-16T15:43:53.226 INFO:teuthology.orchestra.run.smithi184.stdout:/run/rpc_pipefs 2024-09-16T15:43:53.226 INFO:teuthology.orchestra.run.smithi184.stdout:/run/user/1000 2024-09-16T15:43:53.226 INFO:teuthology.orchestra.run.smithi184.stdout:/var/lib/ceph 2024-09-16T15:43:53.226 INFO:teuthology.orchestra.run.smithi184.stdout:/var/lib/ceph/osd/ceph-4 2024-09-16T15:43:53.227 INFO:teuthology.orchestra.run.smithi184.stdout:/var/lib/ceph/osd/ceph-5 2024-09-16T15:43:53.227 INFO:teuthology.orchestra.run.smithi184.stdout:/var/lib/ceph/osd/ceph-6 2024-09-16T15:43:53.227 INFO:teuthology.orchestra.run.smithi184.stdout:/var/lib/ceph/osd/ceph-7 2024-09-16T15:43:53.227 INFO:teuthology.orchestra.run.smithi184.stdout:/run/netns 2024-09-16T15:43:53.227 INFO:teuthology.orchestra.run.smithi184.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-09-16T15:43:53.227 INFO:teuthology.orchestra.run.smithi184.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-09-16T15:43:53.227 INFO:teuthology.orchestra.run.smithi184.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-09-16T15:43:53.227 INFO:teuthology.orchestra.run.smithi184.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-09-16T15:43:53.227 INFO:teuthology.orchestra.run.smithi184.stdout:/home/ubuntu/cephtest/mnt.1 2024-09-16T15:43:53.228 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:43:53.228 DEBUG:teuthology.orchestra.run.smithi184:> ls /sys/fs/fuse/connections 2024-09-16T15:43:53.276 INFO:teuthology.orchestra.run.smithi184.stdout:45 2024-09-16T15:43:53.276 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [45] 2024-09-16T15:43:53.276 DEBUG:teuthology.orchestra.run.smithi184:> sudo stdin-killer -- python3 -c ' 2024-09-16T15:43:53.276 DEBUG:teuthology.orchestra.run.smithi184:> import glob 2024-09-16T15:43:53.276 DEBUG:teuthology.orchestra.run.smithi184:> import re 2024-09-16T15:43:53.276 DEBUG:teuthology.orchestra.run.smithi184:> import os 2024-09-16T15:43:53.276 DEBUG:teuthology.orchestra.run.smithi184:> import subprocess 2024-09-16T15:43:53.276 DEBUG:teuthology.orchestra.run.smithi184:> 2024-09-16T15:43:53.276 DEBUG:teuthology.orchestra.run.smithi184:> def _find_admin_socket(client_name): 2024-09-16T15:43:53.277 DEBUG:teuthology.orchestra.run.smithi184:> asok_path = "/var/run/ceph/ceph-client.1.*.asok" 2024-09-16T15:43:53.277 DEBUG:teuthology.orchestra.run.smithi184:> files = glob.glob(asok_path) 2024-09-16T15:43:53.277 DEBUG:teuthology.orchestra.run.smithi184:> mountpoint = "/home/ubuntu/cephtest/mnt.1" 2024-09-16T15:43:53.277 DEBUG:teuthology.orchestra.run.smithi184:> 2024-09-16T15:43:53.277 DEBUG:teuthology.orchestra.run.smithi184:> # Given a non-glob path, it better be there 2024-09-16T15:43:53.277 DEBUG:teuthology.orchestra.run.smithi184:> if "*" not in asok_path: 2024-09-16T15:43:53.277 DEBUG:teuthology.orchestra.run.smithi184:> assert(len(files) == 1) 2024-09-16T15:43:53.277 DEBUG:teuthology.orchestra.run.smithi184:> return files[0] 2024-09-16T15:43:53.277 DEBUG:teuthology.orchestra.run.smithi184:> 2024-09-16T15:43:53.277 DEBUG:teuthology.orchestra.run.smithi184:> for f in files: 2024-09-16T15:43:53.277 DEBUG:teuthology.orchestra.run.smithi184:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-09-16T15:43:53.277 DEBUG:teuthology.orchestra.run.smithi184:> if os.path.exists("/proc/{0}".format(pid)): 2024-09-16T15:43:53.277 DEBUG:teuthology.orchestra.run.smithi184:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-09-16T15:43:53.277 DEBUG:teuthology.orchestra.run.smithi184:> contents = proc_f.read() 2024-09-16T15:43:53.278 DEBUG:teuthology.orchestra.run.smithi184:> if mountpoint in contents: 2024-09-16T15:43:53.278 DEBUG:teuthology.orchestra.run.smithi184:> return f 2024-09-16T15:43:53.278 DEBUG:teuthology.orchestra.run.smithi184:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-09-16T15:43:53.278 DEBUG:teuthology.orchestra.run.smithi184:> 2024-09-16T15:43:53.278 DEBUG:teuthology.orchestra.run.smithi184:> print(_find_admin_socket("client.1")) 2024-09-16T15:43:53.278 DEBUG:teuthology.orchestra.run.smithi184:> ' 2024-09-16T15:43:53.398 INFO:teuthology.orchestra.run.smithi184.stdout:/var/run/ceph/ceph-client.1.34638.asok 2024-09-16T15:43:53.402 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:43:53 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-16T15:43:53.409 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.1.34638.asok 2024-09-16T15:43:53.409 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:43:53.409 DEBUG:teuthology.orchestra.run.smithi184:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.1.34638.asok status 2024-09-16T15:43:53.564 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-16T15:43:53.564 INFO:teuthology.orchestra.run.smithi184.stdout: "metadata": { 2024-09-16T15:43:53.564 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph_sha1": "33f95d2ae1bcbe7eb04404303f179eeeaa6982df", 2024-09-16T15:43:53.565 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph_version": "ceph version 19.3.0-4938-g33f95d2a (33f95d2ae1bcbe7eb04404303f179eeeaa6982df) squid (dev)", 2024-09-16T15:43:53.565 INFO:teuthology.orchestra.run.smithi184.stdout: "entity_id": "1", 2024-09-16T15:43:53.565 INFO:teuthology.orchestra.run.smithi184.stdout: "hostname": "smithi184", 2024-09-16T15:43:53.565 INFO:teuthology.orchestra.run.smithi184.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.1", 2024-09-16T15:43:53.565 INFO:teuthology.orchestra.run.smithi184.stdout: "pid": "34638", 2024-09-16T15:43:53.565 INFO:teuthology.orchestra.run.smithi184.stdout: "root": "/" 2024-09-16T15:43:53.565 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-16T15:43:53.565 INFO:teuthology.orchestra.run.smithi184.stdout: "dentry_count": 0, 2024-09-16T15:43:53.565 INFO:teuthology.orchestra.run.smithi184.stdout: "dentry_pinned_count": 0, 2024-09-16T15:43:53.565 INFO:teuthology.orchestra.run.smithi184.stdout: "id": 4828, 2024-09-16T15:43:53.565 INFO:teuthology.orchestra.run.smithi184.stdout: "inst": { 2024-09-16T15:43:53.565 INFO:teuthology.orchestra.run.smithi184.stdout: "name": { 2024-09-16T15:43:53.565 INFO:teuthology.orchestra.run.smithi184.stdout: "type": "client", 2024-09-16T15:43:53.565 INFO:teuthology.orchestra.run.smithi184.stdout: "num": 4828 2024-09-16T15:43:53.566 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-16T15:43:53.566 INFO:teuthology.orchestra.run.smithi184.stdout: "addr": { 2024-09-16T15:43:53.566 INFO:teuthology.orchestra.run.smithi184.stdout: "type": "v1", 2024-09-16T15:43:53.566 INFO:teuthology.orchestra.run.smithi184.stdout: "addr": "192.168.0.1:0", 2024-09-16T15:43:53.566 INFO:teuthology.orchestra.run.smithi184.stdout: "nonce": 182167629 2024-09-16T15:43:53.566 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-16T15:43:53.566 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-16T15:43:53.566 INFO:teuthology.orchestra.run.smithi184.stdout: "addr": { 2024-09-16T15:43:53.566 INFO:teuthology.orchestra.run.smithi184.stdout: "type": "v1", 2024-09-16T15:43:53.566 INFO:teuthology.orchestra.run.smithi184.stdout: "addr": "192.168.0.1:0", 2024-09-16T15:43:53.566 INFO:teuthology.orchestra.run.smithi184.stdout: "nonce": 182167629 2024-09-16T15:43:53.566 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-16T15:43:53.566 INFO:teuthology.orchestra.run.smithi184.stdout: "inst_str": "client.4828 192.168.0.1:0/182167629", 2024-09-16T15:43:53.567 INFO:teuthology.orchestra.run.smithi184.stdout: "addr_str": "192.168.0.1:0/182167629", 2024-09-16T15:43:53.567 INFO:teuthology.orchestra.run.smithi184.stdout: "inode_count": 1, 2024-09-16T15:43:53.567 INFO:teuthology.orchestra.run.smithi184.stdout: "mds_epoch": 18, 2024-09-16T15:43:53.567 INFO:teuthology.orchestra.run.smithi184.stdout: "osd_epoch": 33, 2024-09-16T15:43:53.567 INFO:teuthology.orchestra.run.smithi184.stdout: "osd_epoch_barrier": 0, 2024-09-16T15:43:53.567 INFO:teuthology.orchestra.run.smithi184.stdout: "blocklisted": false, 2024-09-16T15:43:53.567 INFO:teuthology.orchestra.run.smithi184.stdout: "fs_name": "cephfs" 2024-09-16T15:43:53.567 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-16T15:43:53.578 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:43:53.578 DEBUG:teuthology.orchestra.run.smithi184:> stat --file-system '--printf=%T 2024-09-16T15:43:53.578 DEBUG:teuthology.orchestra.run.smithi184:> ' -- /home/ubuntu/cephtest/mnt.1 2024-09-16T15:43:53.584 INFO:teuthology.orchestra.run.smithi184.stdout:fuseblk 2024-09-16T15:43:53.584 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2024-09-16T15:43:53.584 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:43:53.584 DEBUG:teuthology.orchestra.run.smithi184:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.1 2024-09-16T15:43:53.639 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs flag set enable_multiple true --yes-i-really-mean-it 2024-09-16T15:43:53.779 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:53.772+0000 7f441bf4d640 1 -- 172.21.15.123:0/1249464371 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f44000040c0 msgr2=0x7f4400004810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:43:53.779 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:53.772+0000 7f441bf4d640 1 --2- 172.21.15.123:0/1249464371 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f44000040c0 0x7f4400004810 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f44140b9ee0 tx=0x7f440401c750 comp rx=0 tx=0).stop 2024-09-16T15:43:53.779 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:53.772+0000 7f441a4ea640 1 -- 172.21.15.123:0/1249464371 reap_dead start 2024-09-16T15:43:53.779 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:53.772+0000 7f441bf4d640 1 -- 172.21.15.123:0/1249464371 shutdown_connections 2024-09-16T15:43:53.780 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:53.772+0000 7f441bf4d640 1 -- 172.21.15.123:0/1249464371 >> 172.21.15.123:0/1249464371 conn(0x7f44140d3e30 msgr2=0x7f44140d4240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:43:53.780 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:53.772+0000 7f441bf4d640 1 -- 172.21.15.123:0/1249464371 shutdown_connections 2024-09-16T15:43:53.780 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:53.772+0000 7f441bf4d640 1 -- 172.21.15.123:0/1249464371 wait complete. 2024-09-16T15:43:53.780 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:53.776+0000 7f441bf4d640 1 Processor -- start 2024-09-16T15:43:53.780 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:53.776+0000 7f441bf4d640 1 -- start start 2024-09-16T15:43:53.782 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:53.776+0000 7f441bf4d640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f44000040c0 0x7f44140caf00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:43:53.782 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:53.776+0000 7f441bf4d640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f441414fc00 0x7f441415c680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:43:53.782 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:53.776+0000 7f441bf4d640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f441415cbc0 0x7f4414151ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:43:53.783 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:53.776+0000 7f441bf4d640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f4414166e80 con 0x7f441415cbc0 2024-09-16T15:43:53.783 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:53.776+0000 7f441bf4d640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f4414166d00 con 0x7f44000040c0 2024-09-16T15:43:53.783 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:53.776+0000 7f441bf4d640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f4414167000 con 0x7f441414fc00 2024-09-16T15:43:53.783 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:53.776+0000 7f441a4ea640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f441415cbc0 0x7f4414151ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:43:53.783 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:53.776+0000 7f441a4ea640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f441415cbc0 0x7f4414151ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:34690/0 (socket says 172.21.15.123:34690) 2024-09-16T15:43:53.783 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:53.776+0000 7f441a4ea640 1 -- 172.21.15.123:0/2839325738 learned_addr learned my addr 172.21.15.123:0/2839325738 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:43:53.783 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:53.776+0000 7f4419ce9640 1 --2- 172.21.15.123:0/2839325738 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f44000040c0 0x7f44140caf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:43:53.783 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:53.776+0000 7f44194e8640 1 --2- 172.21.15.123:0/2839325738 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f441414fc00 0x7f441415c680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:43:53.783 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:53.776+0000 7f441a4ea640 1 -- 172.21.15.123:0/2839325738 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f441414fc00 msgr2=0x7f441415c680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:43:53.783 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:53.776+0000 7f441a4ea640 1 --2- 172.21.15.123:0/2839325738 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f441414fc00 0x7f441415c680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:53.783 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:53.776+0000 7f441a4ea640 1 -- 172.21.15.123:0/2839325738 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f44000040c0 msgr2=0x7f44140caf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:43:53.783 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:53.776+0000 7f441a4ea640 1 --2- 172.21.15.123:0/2839325738 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f44000040c0 0x7f44140caf00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:53.783 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:53.776+0000 7f441a4ea640 1 -- 172.21.15.123:0/2839325738 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f44141f4f40 con 0x7f441415cbc0 2024-09-16T15:43:53.784 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:53.776+0000 7f4419ce9640 1 --2- 172.21.15.123:0/2839325738 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f44000040c0 0x7f44140caf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T15:43:53.784 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:53.776+0000 7f441a4ea640 1 --2- 172.21.15.123:0/2839325738 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f441415cbc0 0x7f4414151ee0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f441408c5a0 tx=0x7f440c00fe10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:43:53.784 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:53.776+0000 7f440a7fc640 1 -- 172.21.15.123:0/2839325738 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f440c004d30 con 0x7f441415cbc0 2024-09-16T15:43:53.784 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:53.776+0000 7f440a7fc640 1 -- 172.21.15.123:0/2839325738 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f440c00eab0 con 0x7f441415cbc0 2024-09-16T15:43:53.784 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:53.776+0000 7f441bf4d640 1 -- 172.21.15.123:0/2839325738 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f44141f5220 con 0x7f441415cbc0 2024-09-16T15:43:53.784 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:53.776+0000 7f440a7fc640 1 -- 172.21.15.123:0/2839325738 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f440c005020 con 0x7f441415cbc0 2024-09-16T15:43:53.784 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:53.776+0000 7f441bf4d640 1 -- 172.21.15.123:0/2839325738 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f44141f57e0 con 0x7f441415cbc0 2024-09-16T15:43:53.785 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:53.776+0000 7f441bf4d640 1 -- 172.21.15.123:0/2839325738 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f44141f59e0 con 0x7f441415cbc0 2024-09-16T15:43:53.785 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:53.780+0000 7f440a7fc640 1 -- 172.21.15.123:0/2839325738 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 5) ==== 97080+0+0 (secure 0 0 0) 0x7f440c019030 con 0x7f441415cbc0 2024-09-16T15:43:53.785 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:53.780+0000 7f440a7fc640 1 --2- 172.21.15.123:0/2839325738 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f43f8072ef0 0x7f43f80753b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:43:53.785 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:53.780+0000 7f440a7fc640 1 -- 172.21.15.123:0/2839325738 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(33..33 src has 1..33) ==== 5709+0+0 (secure 0 0 0) 0x7f440c094650 con 0x7f441415cbc0 2024-09-16T15:43:53.785 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:53.780+0000 7f4419ce9640 1 --2- 172.21.15.123:0/2839325738 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f43f8072ef0 0x7f43f80753b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:43:53.785 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:53.780+0000 7f4419ce9640 1 --2- 172.21.15.123:0/2839325738 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f43f8072ef0 0x7f43f80753b0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f4410000fc0 tx=0x7f4410001730 comp rx=0 tx=0).ready entity=mgr.4135 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:43:53.788 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:53.784+0000 7f440a7fc640 1 -- 172.21.15.123:0/2839325738 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f44141f59e0 con 0x7f441415cbc0 2024-09-16T15:43:53.999 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:53.992+0000 7f441bf4d640 1 -- 172.21.15.123:0/2839325738 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "fs flag set", "flag_name": "enable_multiple", "val": "true", "yes_i_really_mean_it": true} v 0) -- 0x7f44140cb440 con 0x7f441415cbc0 2024-09-16T15:43:54.326 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:54.320+0000 7f440a7fc640 1 -- 172.21.15.123:0/2839325738 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "fs flag set", "flag_name": "enable_multiple", "val": "true", "yes_i_really_mean_it": true}]=0 v19) ==== 136+0+0 (secure 0 0 0) 0x7f440c05f1d0 con 0x7f441415cbc0 2024-09-16T15:43:54.329 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:54.324+0000 7f441bf4d640 1 -- 172.21.15.123:0/2839325738 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f43f8072ef0 msgr2=0x7f43f80753b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:43:54.329 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:54.324+0000 7f441bf4d640 1 --2- 172.21.15.123:0/2839325738 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f43f8072ef0 0x7f43f80753b0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f4410000fc0 tx=0x7f4410001730 comp rx=0 tx=0).stop 2024-09-16T15:43:54.329 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:54.324+0000 7f441bf4d640 1 -- 172.21.15.123:0/2839325738 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f441415cbc0 msgr2=0x7f4414151ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:43:54.330 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:54.324+0000 7f441bf4d640 1 --2- 172.21.15.123:0/2839325738 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f441415cbc0 0x7f4414151ee0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f441408c5a0 tx=0x7f440c00fe10 comp rx=0 tx=0).stop 2024-09-16T15:43:54.330 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:54.324+0000 7f441bf4d640 1 -- 172.21.15.123:0/2839325738 shutdown_connections 2024-09-16T15:43:54.330 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:54.324+0000 7f441bf4d640 1 --2- 172.21.15.123:0/2839325738 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f43f8072ef0 0x7f43f80753b0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:54.330 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:54.324+0000 7f441bf4d640 1 --2- 172.21.15.123:0/2839325738 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f441415cbc0 0x7f4414151ee0 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:54.330 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:54.324+0000 7f441bf4d640 1 --2- 172.21.15.123:0/2839325738 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f441414fc00 0x7f441415c680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:54.330 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:54.324+0000 7f441bf4d640 1 --2- 172.21.15.123:0/2839325738 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f44000040c0 0x7f44140caf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:54.330 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:54.324+0000 7f441bf4d640 1 -- 172.21.15.123:0/2839325738 >> 172.21.15.123:0/2839325738 conn(0x7f44140d3e30 msgr2=0x7f44140b8460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:43:54.330 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:54.324+0000 7f441bf4d640 1 -- 172.21.15.123:0/2839325738 shutdown_connections 2024-09-16T15:43:54.330 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:54.324+0000 7f441bf4d640 1 -- 172.21.15.123:0/2839325738 wait complete. 2024-09-16T15:43:54.349 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2024-09-16T15:43:54.478 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:54.472+0000 7f7a18a43640 1 -- 172.21.15.123:0/744539852 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7a140cd6e0 msgr2=0x7f7a140cdb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:43:54.479 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:54.472+0000 7f7a18a43640 1 --2- 172.21.15.123:0/744539852 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7a140cd6e0 0x7f7a140cdb50 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f7a14073c20 tx=0x7f7a0801c790 comp rx=0 tx=0).stop 2024-09-16T15:43:54.479 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:54.472+0000 7f7a18a43640 1 -- 172.21.15.123:0/744539852 shutdown_connections 2024-09-16T15:43:54.479 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:54.472+0000 7f7a18a43640 1 --2- 172.21.15.123:0/744539852 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7a140cd6e0 0x7f7a140cdb50 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:54.479 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:54.472+0000 7f7a18a43640 1 -- 172.21.15.123:0/744539852 >> 172.21.15.123:0/744539852 conn(0x7f7a140cfb10 msgr2=0x7f7a140cff20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:43:54.479 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:54.472+0000 7f7a18a43640 1 -- 172.21.15.123:0/744539852 shutdown_connections 2024-09-16T15:43:54.479 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:54.472+0000 7f7a18a43640 1 -- 172.21.15.123:0/744539852 wait complete. 2024-09-16T15:43:54.479 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:54.476+0000 7f7a18a43640 1 Processor -- start 2024-09-16T15:43:54.480 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:54.476+0000 7f7a18a43640 1 -- start start 2024-09-16T15:43:54.481 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:54.476+0000 7f7a18a43640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7a140cd6e0 0x7f7a14148650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:43:54.482 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:54.476+0000 7f7a18a43640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f7a1415e160 0x7f7a14152060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:43:54.482 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:54.476+0000 7f7a18a43640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7a1415e820 0x7f7a140b37c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:43:54.482 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:54.476+0000 7f7a18a43640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f7a141697f0 con 0x7f7a140cd6e0 2024-09-16T15:43:54.482 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:54.476+0000 7f7a18a43640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f7a140cf640 con 0x7f7a1415e820 2024-09-16T15:43:54.482 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:54.476+0000 7f7a18a43640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f7a14169970 con 0x7f7a1415e160 2024-09-16T15:43:54.482 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:54.476+0000 7f7a1259c640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7a140cd6e0 0x7f7a14148650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:43:54.482 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:54.476+0000 7f7a1259c640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7a140cd6e0 0x7f7a14148650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:34710/0 (socket says 172.21.15.123:34710) 2024-09-16T15:43:54.482 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:54.476+0000 7f7a1259c640 1 -- 172.21.15.123:0/4199562087 learned_addr learned my addr 172.21.15.123:0/4199562087 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:43:54.482 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:54.476+0000 7f7a12d9d640 1 --2- 172.21.15.123:0/4199562087 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7a1415e820 0x7f7a140b37c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:43:54.482 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:54.476+0000 7f7a1259c640 1 -- 172.21.15.123:0/4199562087 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f7a1415e160 msgr2=0x7f7a14152060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:43:54.482 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:54.476+0000 7f7a1259c640 1 --2- 172.21.15.123:0/4199562087 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f7a1415e160 0x7f7a14152060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:54.482 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:54.476+0000 7f7a1259c640 1 -- 172.21.15.123:0/4199562087 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7a1415e820 msgr2=0x7f7a140b37c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:43:54.482 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:54.476+0000 7f7a1259c640 1 --2- 172.21.15.123:0/4199562087 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7a1415e820 0x7f7a140b37c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:54.483 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:54.476+0000 7f7a1259c640 1 -- 172.21.15.123:0/4199562087 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7a141525a0 con 0x7f7a140cd6e0 2024-09-16T15:43:54.483 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:54.476+0000 7f7a1259c640 1 --2- 172.21.15.123:0/4199562087 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7a140cd6e0 0x7f7a14148650 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f7a0400c7c0 tx=0x7f7a0400cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:43:54.483 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:54.476+0000 7f7a02ffd640 1 -- 172.21.15.123:0/4199562087 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a04016020 con 0x7f7a140cd6e0 2024-09-16T15:43:54.483 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:54.476+0000 7f7a02ffd640 1 -- 172.21.15.123:0/4199562087 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f7a0400ab40 con 0x7f7a140cd6e0 2024-09-16T15:43:54.483 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:54.476+0000 7f7a18a43640 1 -- 172.21.15.123:0/4199562087 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7a140b3580 con 0x7f7a140cd6e0 2024-09-16T15:43:54.483 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:54.476+0000 7f7a02ffd640 1 -- 172.21.15.123:0/4199562087 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a04012420 con 0x7f7a140cd6e0 2024-09-16T15:43:54.483 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:54.476+0000 7f7a18a43640 1 -- 172.21.15.123:0/4199562087 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7a14152c40 con 0x7f7a140cd6e0 2024-09-16T15:43:54.484 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:54.476+0000 7f7a18a43640 1 -- 172.21.15.123:0/4199562087 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7a140cdb50 con 0x7f7a140cd6e0 2024-09-16T15:43:54.484 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:54.480+0000 7f7a02ffd640 1 -- 172.21.15.123:0/4199562087 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 5) ==== 97080+0+0 (secure 0 0 0) 0x7f7a04019020 con 0x7f7a140cd6e0 2024-09-16T15:43:54.484 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:54.480+0000 7f7a02ffd640 1 --2- 172.21.15.123:0/4199562087 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f79f4072ec0 0x7f79f4075380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:43:54.484 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:54.480+0000 7f7a02ffd640 1 -- 172.21.15.123:0/4199562087 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(33..33 src has 1..33) ==== 5709+0+0 (secure 0 0 0) 0x7f7a04093f30 con 0x7f7a140cd6e0 2024-09-16T15:43:54.484 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:54.480+0000 7f7a11d9b640 1 --2- 172.21.15.123:0/4199562087 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f79f4072ec0 0x7f79f4075380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:43:54.484 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:54.480+0000 7f7a11d9b640 1 --2- 172.21.15.123:0/4199562087 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f79f4072ec0 0x7f79f4075380 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f7a080af000 tx=0x7f7a08019530 comp rx=0 tx=0).ready entity=mgr.4135 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:43:54.488 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:54.484+0000 7f7a02ffd640 1 -- 172.21.15.123:0/4199562087 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f7a04020030 con 0x7f7a140cd6e0 2024-09-16T15:43:54.669 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:54.664+0000 7f7a18a43640 1 -- 172.21.15.123:0/4199562087 --> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) -- 0x7f7a14157b80 con 0x7f79f4072ec0 2024-09-16T15:43:54.673 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:54.672+0000 7f7a02ffd640 1 -- 172.21.15.123:0/4199562087 <== mgr.4135 v2:172.21.15.184:6832/3114346423 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+41 (secure 0 0 0) 0x7f7a14157b80 con 0x7f79f4072ec0 2024-09-16T15:43:54.673 INFO:teuthology.orchestra.run.smithi123.stdout:[ 2024-09-16T15:43:54.673 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-09-16T15:43:54.673 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "cephfs" 2024-09-16T15:43:54.673 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-16T15:43:54.673 INFO:teuthology.orchestra.run.smithi123.stdout:] 2024-09-16T15:43:54.675 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:54.672+0000 7f7a18a43640 1 -- 172.21.15.123:0/4199562087 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f79f4072ec0 msgr2=0x7f79f4075380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:43:54.675 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:54.672+0000 7f7a18a43640 1 --2- 172.21.15.123:0/4199562087 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f79f4072ec0 0x7f79f4075380 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f7a080af000 tx=0x7f7a08019530 comp rx=0 tx=0).stop 2024-09-16T15:43:54.675 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:54.672+0000 7f7a18a43640 1 -- 172.21.15.123:0/4199562087 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7a140cd6e0 msgr2=0x7f7a14148650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:43:54.676 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:54.672+0000 7f7a18a43640 1 --2- 172.21.15.123:0/4199562087 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7a140cd6e0 0x7f7a14148650 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f7a0400c7c0 tx=0x7f7a0400cc80 comp rx=0 tx=0).stop 2024-09-16T15:43:54.676 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:54.672+0000 7f7a18a43640 1 -- 172.21.15.123:0/4199562087 shutdown_connections 2024-09-16T15:43:54.676 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:54.672+0000 7f7a18a43640 1 --2- 172.21.15.123:0/4199562087 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f79f4072ec0 0x7f79f4075380 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:54.676 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:54.672+0000 7f7a18a43640 1 --2- 172.21.15.123:0/4199562087 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7a1415e820 0x7f7a140b37c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:54.676 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:54.672+0000 7f7a18a43640 1 --2- 172.21.15.123:0/4199562087 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f7a1415e160 0x7f7a14152060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:54.676 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:54.672+0000 7f7a18a43640 1 --2- 172.21.15.123:0/4199562087 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7a140cd6e0 0x7f7a14148650 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:54.676 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:54.672+0000 7f7a18a43640 1 -- 172.21.15.123:0/4199562087 >> 172.21.15.123:0/4199562087 conn(0x7f7a140cfb10 msgr2=0x7f7a140af5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:43:54.676 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:54.672+0000 7f7a18a43640 1 -- 172.21.15.123:0/4199562087 shutdown_connections 2024-09-16T15:43:54.676 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:54.672+0000 7f7a18a43640 1 -- 172.21.15.123:0/4199562087 wait complete. 2024-09-16T15:43:54.691 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-09-16T15:43:54.822 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:54.816+0000 7f3d328ba640 1 -- 172.21.15.123:0/1605078697 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3d2c15bd60 msgr2=0x7f3d2c0cf850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:43:54.822 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:54.816+0000 7f3d328ba640 1 --2- 172.21.15.123:0/1605078697 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3d2c15bd60 0x7f3d2c0cf850 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f3d1c009930 tx=0x7f3d1c01f0d0 comp rx=0 tx=0).stop 2024-09-16T15:43:54.822 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:54.816+0000 7f3d328ba640 1 -- 172.21.15.123:0/1605078697 shutdown_connections 2024-09-16T15:43:54.822 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:54.816+0000 7f3d328ba640 1 --2- 172.21.15.123:0/1605078697 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f3d2c0b31a0 0x7f3d2c0d1b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:54.822 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:54.816+0000 7f3d328ba640 1 --2- 172.21.15.123:0/1605078697 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3d2c15bd60 0x7f3d2c0cf850 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:54.822 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:54.816+0000 7f3d328ba640 1 -- 172.21.15.123:0/1605078697 >> 172.21.15.123:0/1605078697 conn(0x7f3d2c09aa40 msgr2=0x7f3d2c09ae50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:43:54.822 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:54.816+0000 7f3d328ba640 1 -- 172.21.15.123:0/1605078697 shutdown_connections 2024-09-16T15:43:54.822 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:54.816+0000 7f3d328ba640 1 -- 172.21.15.123:0/1605078697 wait complete. 2024-09-16T15:43:54.823 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:54.820+0000 7f3d328ba640 1 Processor -- start 2024-09-16T15:43:54.823 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:54.820+0000 7f3d328ba640 1 -- start start 2024-09-16T15:43:54.825 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:54.820+0000 7f3d328ba640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3d2c0b31a0 0x7f3d2c0b72b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:43:54.825 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:54.820+0000 7f3d328ba640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f3d2c0d2060 0x7f3d2c15a8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:43:54.825 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:54.820+0000 7f3d328ba640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3d2c15bd60 0x7f3d2c0b1080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:43:54.825 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:54.820+0000 7f3d328ba640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f3d2c0c7c10 con 0x7f3d2c15bd60 2024-09-16T15:43:54.826 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:54.820+0000 7f3d328ba640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f3d2c15fe80 con 0x7f3d2c0b31a0 2024-09-16T15:43:54.826 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:54.820+0000 7f3d328ba640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f3d2c15e140 con 0x7f3d2c0d2060 2024-09-16T15:43:54.826 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:54.820+0000 7f3d30e57640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3d2c15bd60 0x7f3d2c0b1080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:43:54.826 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:54.820+0000 7f3d30e57640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3d2c15bd60 0x7f3d2c0b1080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:34726/0 (socket says 172.21.15.123:34726) 2024-09-16T15:43:54.826 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:54.820+0000 7f3d30e57640 1 -- 172.21.15.123:0/2659241168 learned_addr learned my addr 172.21.15.123:0/2659241168 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:43:54.826 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:54.820+0000 7f3d2b7fe640 1 --2- 172.21.15.123:0/2659241168 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f3d2c0d2060 0x7f3d2c15a8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:43:54.826 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:54.820+0000 7f3d30e57640 1 -- 172.21.15.123:0/2659241168 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f3d2c0d2060 msgr2=0x7f3d2c15a8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:43:54.826 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:54.820+0000 7f3d2bfff640 1 --2- 172.21.15.123:0/2659241168 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3d2c0b31a0 0x7f3d2c0b72b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:43:54.826 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:54.820+0000 7f3d30e57640 1 --2- 172.21.15.123:0/2659241168 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f3d2c0d2060 0x7f3d2c15a8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:54.826 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:54.820+0000 7f3d30e57640 1 -- 172.21.15.123:0/2659241168 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3d2c0b31a0 msgr2=0x7f3d2c0b72b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:43:54.826 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:54.820+0000 7f3d30e57640 1 --2- 172.21.15.123:0/2659241168 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3d2c0b31a0 0x7f3d2c0b72b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:54.826 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:54.820+0000 7f3d30e57640 1 -- 172.21.15.123:0/2659241168 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3d2c1f26d0 con 0x7f3d2c15bd60 2024-09-16T15:43:54.826 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:54.820+0000 7f3d2b7fe640 1 --2- 172.21.15.123:0/2659241168 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f3d2c0d2060 0x7f3d2c15a8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T15:43:54.826 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:54.820+0000 7f3d30e57640 1 --2- 172.21.15.123:0/2659241168 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3d2c15bd60 0x7f3d2c0b1080 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f3d2400c780 tx=0x7f3d2400cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:43:54.827 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:54.820+0000 7f3d28ff9640 1 -- 172.21.15.123:0/2659241168 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d24004b70 con 0x7f3d2c15bd60 2024-09-16T15:43:54.827 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:54.820+0000 7f3d28ff9640 1 -- 172.21.15.123:0/2659241168 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f3d24004e00 con 0x7f3d2c15bd60 2024-09-16T15:43:54.827 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:54.820+0000 7f3d328ba640 1 -- 172.21.15.123:0/2659241168 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3d2c1f29b0 con 0x7f3d2c15bd60 2024-09-16T15:43:54.827 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:54.820+0000 7f3d28ff9640 1 -- 172.21.15.123:0/2659241168 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d2400aa90 con 0x7f3d2c15bd60 2024-09-16T15:43:54.827 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:54.820+0000 7f3d328ba640 1 -- 172.21.15.123:0/2659241168 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3d2c1f2f70 con 0x7f3d2c15bd60 2024-09-16T15:43:54.828 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:54.820+0000 7f3d328ba640 1 -- 172.21.15.123:0/2659241168 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3d2c160410 con 0x7f3d2c15bd60 2024-09-16T15:43:54.828 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:54.820+0000 7f3d28ff9640 1 -- 172.21.15.123:0/2659241168 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 5) ==== 97080+0+0 (secure 0 0 0) 0x7f3d2400ac30 con 0x7f3d2c15bd60 2024-09-16T15:43:54.828 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:54.824+0000 7f3d28ff9640 1 --2- 172.21.15.123:0/2659241168 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f3d18072ec0 0x7f3d18075380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:43:54.828 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:54.824+0000 7f3d28ff9640 1 -- 172.21.15.123:0/2659241168 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(33..33 src has 1..33) ==== 5709+0+0 (secure 0 0 0) 0x7f3d240943b0 con 0x7f3d2c15bd60 2024-09-16T15:43:54.828 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:54.824+0000 7f3d2bfff640 1 --2- 172.21.15.123:0/2659241168 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f3d18072ec0 0x7f3d18075380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:43:54.828 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:54.824+0000 7f3d2bfff640 1 --2- 172.21.15.123:0/2659241168 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f3d18072ec0 0x7f3d18075380 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f3d1c0b2000 tx=0x7f3d1c01d910 comp rx=0 tx=0).ready entity=mgr.4135 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:43:54.833 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:54.828+0000 7f3d28ff9640 1 -- 172.21.15.123:0/2659241168 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f3d2405f5f0 con 0x7f3d2c15bd60 2024-09-16T15:43:55.007 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:55.004+0000 7f3d328ba640 1 -- 172.21.15.123:0/2659241168 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) -- 0x7f3d2c1f33b0 con 0x7f3d2c15bd60 2024-09-16T15:43:55.008 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:55.004+0000 7f3d28ff9640 1 -- 172.21.15.123:0/2659241168 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v28)=0 v28) ==== 122+0+0 (secure 0 0 0) 0x7f3d2405eef0 con 0x7f3d2c15bd60 2024-09-16T15:43:55.011 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:55.004+0000 7f3d328ba640 1 -- 172.21.15.123:0/2659241168 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f3d18072ec0 msgr2=0x7f3d18075380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:43:55.011 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:55.004+0000 7f3d328ba640 1 --2- 172.21.15.123:0/2659241168 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f3d18072ec0 0x7f3d18075380 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f3d1c0b2000 tx=0x7f3d1c01d910 comp rx=0 tx=0).stop 2024-09-16T15:43:55.011 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:55.004+0000 7f3d328ba640 1 -- 172.21.15.123:0/2659241168 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3d2c15bd60 msgr2=0x7f3d2c0b1080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:43:55.011 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:55.004+0000 7f3d328ba640 1 --2- 172.21.15.123:0/2659241168 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3d2c15bd60 0x7f3d2c0b1080 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f3d2400c780 tx=0x7f3d2400cc40 comp rx=0 tx=0).stop 2024-09-16T15:43:55.011 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:55.004+0000 7f3d328ba640 1 -- 172.21.15.123:0/2659241168 shutdown_connections 2024-09-16T15:43:55.011 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:55.004+0000 7f3d328ba640 1 --2- 172.21.15.123:0/2659241168 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f3d18072ec0 0x7f3d18075380 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:55.011 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:55.004+0000 7f3d328ba640 1 --2- 172.21.15.123:0/2659241168 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3d2c15bd60 0x7f3d2c0b1080 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:55.011 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:55.004+0000 7f3d328ba640 1 --2- 172.21.15.123:0/2659241168 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f3d2c0d2060 0x7f3d2c15a8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:55.011 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:55.004+0000 7f3d328ba640 1 --2- 172.21.15.123:0/2659241168 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3d2c0b31a0 0x7f3d2c0b72b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:55.011 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:55.004+0000 7f3d328ba640 1 -- 172.21.15.123:0/2659241168 >> 172.21.15.123:0/2659241168 conn(0x7f3d2c09aa40 msgr2=0x7f3d2c09ae20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:43:55.011 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:55.004+0000 7f3d328ba640 1 -- 172.21.15.123:0/2659241168 shutdown_connections 2024-09-16T15:43:55.011 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:55.004+0000 7f3d328ba640 1 -- 172.21.15.123:0/2659241168 wait complete. 2024-09-16T15:43:55.027 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:43:55.027 DEBUG:teuthology.orchestra.run.smithi123:> (cd /home/ubuntu/cephtest/mnt.0 && exec sudo mkdir -p ./volumes/subvol_grp11100/subvol54470) 2024-09-16T15:43:55.050 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:43:55.050 DEBUG:teuthology.orchestra.run.smithi123:> (cd /home/ubuntu/cephtest/mnt.0 && exec getfattr --only-values -n ceph.dir.layout.pool .) 2024-09-16T15:43:55.180 INFO:teuthology.orchestra.run.smithi123.stdout:cephfs_data 2024-09-16T15:43:55.180 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:43:55.180 DEBUG:teuthology.orchestra.run.smithi123:> (cd /home/ubuntu/cephtest/mnt.0 && exec sudo setfattr -n ceph.dir.layout.pool -v cephfs_data ./volumes/subvol_grp11100/subvol54470) 2024-09-16T15:43:55.200 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2024-09-16T15:43:55.200 DEBUG:teuthology.orchestra.run.smithi123:> sudo dd of=/home/ubuntu/cephtest/mnt.0/./volumes/subvol_grp11100/subvol54470/.meta 2024-09-16T15:43:55.263 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs subvol54470 subvol_grp11100 2024-09-16T15:43:55.450 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:55.444+0000 7f0d97592640 1 -- 172.21.15.123:0/1704454682 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0d9015dc40 msgr2=0x7f0d9015e060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:43:55.450 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:55.444+0000 7f0d97592640 1 --2- 172.21.15.123:0/1704454682 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0d9015dc40 0x7f0d9015e060 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f0d80009900 tx=0x7f0d8001c8e0 comp rx=0 tx=0).stop 2024-09-16T15:43:55.450 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:55.444+0000 7f0d97592640 1 -- 172.21.15.123:0/1704454682 shutdown_connections 2024-09-16T15:43:55.450 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:55.444+0000 7f0d97592640 1 --2- 172.21.15.123:0/1704454682 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0d9015dc40 0x7f0d9015e060 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:55.450 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:55.444+0000 7f0d97592640 1 --2- 172.21.15.123:0/1704454682 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f0d9015a1b0 0x7f0d90072fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:55.450 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:55.444+0000 7f0d97592640 1 -- 172.21.15.123:0/1704454682 >> 172.21.15.123:0/1704454682 conn(0x7f0d900b1630 msgr2=0x7f0d900b1a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:43:55.451 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:55.444+0000 7f0d97592640 1 -- 172.21.15.123:0/1704454682 shutdown_connections 2024-09-16T15:43:55.451 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:55.444+0000 7f0d97592640 1 -- 172.21.15.123:0/1704454682 wait complete. 2024-09-16T15:43:55.451 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:55.448+0000 7f0d97592640 1 Processor -- start 2024-09-16T15:43:55.451 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:55.448+0000 7f0d97592640 1 -- start start 2024-09-16T15:43:55.453 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:55.448+0000 7f0d97592640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0d9015a1b0 0x7f0d90146650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:43:55.453 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:55.448+0000 7f0d97592640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f0d9015a590 0x7f0d900b06a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:43:55.453 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:55.448+0000 7f0d97592640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f0d9015dc40 0x7f0d900ad8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:43:55.453 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:55.448+0000 7f0d97592640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f0d9015b240 con 0x7f0d9015a1b0 2024-09-16T15:43:55.453 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:55.448+0000 7f0d97592640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f0d9015b0c0 con 0x7f0d9015dc40 2024-09-16T15:43:55.453 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:55.448+0000 7f0d97592640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f0d9015b3c0 con 0x7f0d9015a590 2024-09-16T15:43:55.453 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:55.448+0000 7f0d9532e640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0d9015a1b0 0x7f0d90146650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:43:55.454 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:55.448+0000 7f0d9532e640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0d9015a1b0 0x7f0d90146650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:34750/0 (socket says 172.21.15.123:34750) 2024-09-16T15:43:55.454 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:55.448+0000 7f0d9532e640 1 -- 172.21.15.123:0/3443253132 learned_addr learned my addr 172.21.15.123:0/3443253132 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:43:55.454 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:55.448+0000 7f0d9532e640 1 -- 172.21.15.123:0/3443253132 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f0d9015a590 msgr2=0x7f0d900b06a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:43:55.454 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:55.448+0000 7f0d9532e640 1 --2- 172.21.15.123:0/3443253132 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f0d9015a590 0x7f0d900b06a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:55.454 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:55.448+0000 7f0d9532e640 1 -- 172.21.15.123:0/3443253132 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f0d9015dc40 msgr2=0x7f0d900ad8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:43:55.454 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:55.448+0000 7f0d9532e640 1 --2- 172.21.15.123:0/3443253132 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f0d9015dc40 0x7f0d900ad8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:55.454 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:55.448+0000 7f0d9532e640 1 -- 172.21.15.123:0/3443253132 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0d901f8750 con 0x7f0d9015a1b0 2024-09-16T15:43:55.454 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:55.448+0000 7f0d9532e640 1 --2- 172.21.15.123:0/3443253132 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0d9015a1b0 0x7f0d90146650 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f0d8c00c970 tx=0x7f0d8c00ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:43:55.454 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:55.448+0000 7f0d85ffb640 1 -- 172.21.15.123:0/3443253132 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d8c007bb0 con 0x7f0d9015a1b0 2024-09-16T15:43:55.454 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:55.448+0000 7f0d85ffb640 1 -- 172.21.15.123:0/3443253132 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f0d8c005020 con 0x7f0d9015a1b0 2024-09-16T15:43:55.454 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:55.448+0000 7f0d97592640 1 -- 172.21.15.123:0/3443253132 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0d900b1250 con 0x7f0d9015a1b0 2024-09-16T15:43:55.454 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:55.448+0000 7f0d85ffb640 1 -- 172.21.15.123:0/3443253132 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d8c007e60 con 0x7f0d9015a1b0 2024-09-16T15:43:55.454 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:55.448+0000 7f0d97592640 1 -- 172.21.15.123:0/3443253132 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0d901f7c70 con 0x7f0d9015a1b0 2024-09-16T15:43:55.455 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:55.448+0000 7f0d97592640 1 -- 172.21.15.123:0/3443253132 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0d90072fb0 con 0x7f0d9015a1b0 2024-09-16T15:43:55.455 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:55.448+0000 7f0d85ffb640 1 -- 172.21.15.123:0/3443253132 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 5) ==== 97080+0+0 (secure 0 0 0) 0x7f0d8c005470 con 0x7f0d9015a1b0 2024-09-16T15:43:55.455 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:55.448+0000 7f0d85ffb640 1 --2- 172.21.15.123:0/3443253132 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f0d74072ec0 0x7f0d74075380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:43:55.455 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:55.452+0000 7f0d85ffb640 1 -- 172.21.15.123:0/3443253132 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(33..33 src has 1..33) ==== 5709+0+0 (secure 0 0 0) 0x7f0d8c094390 con 0x7f0d9015a1b0 2024-09-16T15:43:55.455 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:55.452+0000 7f0d94b2d640 1 --2- 172.21.15.123:0/3443253132 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f0d74072ec0 0x7f0d74075380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:43:55.456 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:55.452+0000 7f0d94b2d640 1 --2- 172.21.15.123:0/3443253132 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f0d74072ec0 0x7f0d74075380 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f0d800af000 tx=0x7f0d80019680 comp rx=0 tx=0).ready entity=mgr.4135 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:43:55.459 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:55.456+0000 7f0d85ffb640 1 -- 172.21.15.123:0/3443253132 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f0d8c014030 con 0x7f0d9015a1b0 2024-09-16T15:43:55.646 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:55.640+0000 7f0d97592640 1 -- 172.21.15.123:0/3443253132 --> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "subvol54470", "group_name": "subvol_grp11100", "target": ["mon-mgr", ""]}) -- 0x7f0d901f7740 con 0x7f0d74072ec0 2024-09-16T15:43:55.654 INFO:tasks.ceph.mgr.x.smithi184.stderr:2024-09-16T15:43:55.649+0000 7fda6a8d4640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-16T15:43:55.654 INFO:tasks.ceph.mgr.x.smithi184.stderr:2024-09-16T15:43:55.649+0000 7fda6a8d4640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-16T15:43:55.654 INFO:tasks.ceph.mgr.x.smithi184.stderr:2024-09-16T15:43:55.649+0000 7fda6a8d4640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-16T15:43:55.654 INFO:tasks.ceph.mgr.x.smithi184.stderr:2024-09-16T15:43:55.649+0000 7fda6a8d4640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-16T15:43:55.654 INFO:tasks.ceph.mgr.x.smithi184.stderr:2024-09-16T15:43:55.649+0000 7fda6a8d4640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-16T15:43:55.784 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:55.780+0000 7f0d85ffb640 1 -- 172.21.15.123:0/3443253132 <== mgr.4135 v2:172.21.15.184:6832/3114346423 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f0d901f7740 con 0x7f0d74072ec0 2024-09-16T15:43:55.784 INFO:teuthology.orchestra.run.smithi123.stdout:/volumes/subvol_grp11100/subvol54470 2024-09-16T15:43:55.788 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:55.784+0000 7f0d97592640 1 -- 172.21.15.123:0/3443253132 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f0d74072ec0 msgr2=0x7f0d74075380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:43:55.788 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:55.784+0000 7f0d97592640 1 --2- 172.21.15.123:0/3443253132 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f0d74072ec0 0x7f0d74075380 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f0d800af000 tx=0x7f0d80019680 comp rx=0 tx=0).stop 2024-09-16T15:43:55.788 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:55.784+0000 7f0d97592640 1 -- 172.21.15.123:0/3443253132 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0d9015a1b0 msgr2=0x7f0d90146650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:43:55.788 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:55.784+0000 7f0d97592640 1 --2- 172.21.15.123:0/3443253132 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0d9015a1b0 0x7f0d90146650 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f0d8c00c970 tx=0x7f0d8c00ce30 comp rx=0 tx=0).stop 2024-09-16T15:43:55.789 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:55.784+0000 7f0d97592640 1 -- 172.21.15.123:0/3443253132 shutdown_connections 2024-09-16T15:43:55.789 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:55.784+0000 7f0d97592640 1 --2- 172.21.15.123:0/3443253132 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f0d74072ec0 0x7f0d74075380 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:55.789 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:55.784+0000 7f0d97592640 1 --2- 172.21.15.123:0/3443253132 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f0d9015dc40 0x7f0d900ad8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:55.789 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:55.784+0000 7f0d97592640 1 --2- 172.21.15.123:0/3443253132 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f0d9015a590 0x7f0d900b06a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:55.789 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:55.784+0000 7f0d97592640 1 --2- 172.21.15.123:0/3443253132 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0d9015a1b0 0x7f0d90146650 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:55.789 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:55.784+0000 7f0d97592640 1 -- 172.21.15.123:0/3443253132 >> 172.21.15.123:0/3443253132 conn(0x7f0d900b1630 msgr2=0x7f0d901594d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:43:55.789 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:55.784+0000 7f0d97592640 1 -- 172.21.15.123:0/3443253132 shutdown_connections 2024-09-16T15:43:55.789 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:55.784+0000 7f0d97592640 1 -- 172.21.15.123:0/3443253132 wait complete. 2024-09-16T15:43:55.807 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs subvol54470 subvol_grp11100 2024-09-16T15:43:55.941 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:55.936+0000 7fe975554640 1 -- 172.21.15.123:0/762396074 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe97015b540 msgr2=0x7fe97006ff30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:43:55.941 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:55.936+0000 7fe975554640 1 --2- 172.21.15.123:0/762396074 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe97015b540 0x7fe97006ff30 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7fe970051540 tx=0x7fe96401c770 comp rx=0 tx=0).stop 2024-09-16T15:43:55.941 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:55.936+0000 7fe975554640 1 -- 172.21.15.123:0/762396074 shutdown_connections 2024-09-16T15:43:55.941 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:55.936+0000 7fe975554640 1 --2- 172.21.15.123:0/762396074 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe97015b540 0x7fe97006ff30 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:55.941 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:55.936+0000 7fe975554640 1 --2- 172.21.15.123:0/762396074 >> v2:172.21.15.184:3300/0 conn(0x7fe970073c20 0x7fe97015b000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:55.941 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:55.936+0000 7fe975554640 1 -- 172.21.15.123:0/762396074 >> 172.21.15.123:0/762396074 conn(0x7fe9700b1630 msgr2=0x7fe9700b1a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:43:55.941 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:55.936+0000 7fe975554640 1 -- 172.21.15.123:0/762396074 shutdown_connections 2024-09-16T15:43:55.941 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:55.936+0000 7fe975554640 1 -- 172.21.15.123:0/762396074 wait complete. 2024-09-16T15:43:55.943 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:55.936+0000 7fe975554640 1 Processor -- start 2024-09-16T15:43:55.943 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:55.936+0000 7fe975554640 1 -- start start 2024-09-16T15:43:55.943 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:55.936+0000 7fe975554640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe970070470 0x7fe970158440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:43:55.944 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:55.936+0000 7fe975554640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe970073c20 0x7fe97006ee20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:43:55.944 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:55.936+0000 7fe975554640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fe97015b540 0x7fe97006f360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:43:55.944 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:55.936+0000 7fe975554640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fe9701681f0 con 0x7fe970070470 2024-09-16T15:43:55.944 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:55.936+0000 7fe975554640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fe970168070 con 0x7fe970073c20 2024-09-16T15:43:55.944 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:55.936+0000 7fe975554640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fe970168370 con 0x7fe97015b540 2024-09-16T15:43:55.944 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:55.936+0000 7fe96effd640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe970070470 0x7fe970158440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:43:55.944 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:55.936+0000 7fe96effd640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe970070470 0x7fe970158440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:34764/0 (socket says 172.21.15.123:34764) 2024-09-16T15:43:55.944 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:55.936+0000 7fe96effd640 1 -- 172.21.15.123:0/1195125950 learned_addr learned my addr 172.21.15.123:0/1195125950 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:43:55.944 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:55.936+0000 7fe96effd640 1 -- 172.21.15.123:0/1195125950 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fe97015b540 msgr2=0x7fe97006f360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:43:55.944 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:55.936+0000 7fe96effd640 1 --2- 172.21.15.123:0/1195125950 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fe97015b540 0x7fe97006f360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:55.944 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:55.936+0000 7fe96effd640 1 -- 172.21.15.123:0/1195125950 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe970073c20 msgr2=0x7fe97006ee20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:43:55.944 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:55.936+0000 7fe96effd640 1 --2- 172.21.15.123:0/1195125950 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe970073c20 0x7fe97006ee20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:55.945 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:55.936+0000 7fe96effd640 1 -- 172.21.15.123:0/1195125950 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe97013e010 con 0x7fe970070470 2024-09-16T15:43:55.945 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:55.936+0000 7fe96effd640 1 --2- 172.21.15.123:0/1195125950 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe970070470 0x7fe970158440 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7fe96000c970 tx=0x7fe96000ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:43:55.945 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:55.940+0000 7fe9577fe640 1 -- 172.21.15.123:0/1195125950 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe960007c70 con 0x7fe970070470 2024-09-16T15:43:55.945 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:55.940+0000 7fe9577fe640 1 -- 172.21.15.123:0/1195125950 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fe96000f450 con 0x7fe970070470 2024-09-16T15:43:55.945 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:55.940+0000 7fe9577fe640 1 -- 172.21.15.123:0/1195125950 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe96000f8a0 con 0x7fe970070470 2024-09-16T15:43:55.945 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:55.940+0000 7fe975554640 1 -- 172.21.15.123:0/1195125950 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe97013e2f0 con 0x7fe970070470 2024-09-16T15:43:55.945 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:55.940+0000 7fe975554640 1 -- 172.21.15.123:0/1195125950 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe9701653c0 con 0x7fe970070470 2024-09-16T15:43:55.946 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:55.940+0000 7fe975554640 1 -- 172.21.15.123:0/1195125950 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe970156fe0 con 0x7fe970070470 2024-09-16T15:43:55.946 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:55.940+0000 7fe9577fe640 1 -- 172.21.15.123:0/1195125950 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 5) ==== 97080+0+0 (secure 0 0 0) 0x7fe96000fa40 con 0x7fe970070470 2024-09-16T15:43:55.946 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:55.940+0000 7fe9577fe640 1 --2- 172.21.15.123:0/1195125950 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fe95c072ec0 0x7fe95c075380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:43:55.946 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:55.940+0000 7fe9577fe640 1 -- 172.21.15.123:0/1195125950 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(33..33 src has 1..33) ==== 5709+0+0 (secure 0 0 0) 0x7fe9600946b0 con 0x7fe970070470 2024-09-16T15:43:55.946 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:55.940+0000 7fe96e7fc640 1 --2- 172.21.15.123:0/1195125950 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fe95c072ec0 0x7fe95c075380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:43:55.946 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:55.940+0000 7fe96e7fc640 1 --2- 172.21.15.123:0/1195125950 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fe95c072ec0 0x7fe95c075380 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fe9640af000 tx=0x7fe964019510 comp rx=0 tx=0).ready entity=mgr.4135 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:43:55.949 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:55.944+0000 7fe9577fe640 1 -- 172.21.15.123:0/1195125950 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7fe970156fe0 con 0x7fe970070470 2024-09-16T15:43:56.150 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:56.144+0000 7fe975554640 1 -- 172.21.15.123:0/1195125950 --> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "subvol54470", "group_name": "subvol_grp11100", "target": ["mon-mgr", ""]}) -- 0x7fe97015b000 con 0x7fe95c072ec0 2024-09-16T15:43:56.161 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:56.156+0000 7fe9577fe640 1 -- 172.21.15.123:0/1195125950 <== mgr.4135 v2:172.21.15.184:6832/3114346423 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7fe97015b000 con 0x7fe95c072ec0 2024-09-16T15:43:56.161 INFO:teuthology.orchestra.run.smithi123.stdout:/volumes/subvol_grp11100/subvol54470 2024-09-16T15:43:56.163 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:56.160+0000 7fe975554640 1 -- 172.21.15.123:0/1195125950 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fe95c072ec0 msgr2=0x7fe95c075380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:43:56.163 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:56.160+0000 7fe975554640 1 --2- 172.21.15.123:0/1195125950 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fe95c072ec0 0x7fe95c075380 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fe9640af000 tx=0x7fe964019510 comp rx=0 tx=0).stop 2024-09-16T15:43:56.164 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:56.160+0000 7fe975554640 1 -- 172.21.15.123:0/1195125950 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe970070470 msgr2=0x7fe970158440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:43:56.164 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:56.160+0000 7fe975554640 1 --2- 172.21.15.123:0/1195125950 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe970070470 0x7fe970158440 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7fe96000c970 tx=0x7fe96000ce30 comp rx=0 tx=0).stop 2024-09-16T15:43:56.164 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:56.160+0000 7fe975554640 1 -- 172.21.15.123:0/1195125950 shutdown_connections 2024-09-16T15:43:56.164 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:56.160+0000 7fe975554640 1 --2- 172.21.15.123:0/1195125950 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fe95c072ec0 0x7fe95c075380 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:56.164 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:56.160+0000 7fe975554640 1 --2- 172.21.15.123:0/1195125950 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fe97015b540 0x7fe97006f360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:56.164 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:56.160+0000 7fe975554640 1 --2- 172.21.15.123:0/1195125950 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe970073c20 0x7fe97006ee20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:56.164 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:56.160+0000 7fe975554640 1 --2- 172.21.15.123:0/1195125950 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe970070470 0x7fe970158440 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:56.164 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:56.160+0000 7fe975554640 1 -- 172.21.15.123:0/1195125950 >> 172.21.15.123:0/1195125950 conn(0x7fe9700b1630 msgr2=0x7fe9700b1a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:43:56.164 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:56.160+0000 7fe975554640 1 -- 172.21.15.123:0/1195125950 shutdown_connections 2024-09-16T15:43:56.164 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:56.160+0000 7fe975554640 1 -- 172.21.15.123:0/1195125950 wait complete. 2024-09-16T15:43:56.181 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:43:56.181 DEBUG:teuthology.orchestra.run.smithi123:> (cd /home/ubuntu/cephtest/mnt.0 && exec sudo cat ./volumes/_legacy/8c8c5d083f72ca0ceaae2982cc07309d.meta) 2024-09-16T15:43:58.553 INFO:teuthology.orchestra.run.smithi123.stdout:[GLOBAL] 2024-09-16T15:43:58.553 INFO:teuthology.orchestra.run.smithi123.stdout:version = 1 2024-09-16T15:43:58.554 INFO:teuthology.orchestra.run.smithi123.stdout:type = subvolume 2024-09-16T15:43:58.554 INFO:teuthology.orchestra.run.smithi123.stdout:path = /volumes/subvol_grp11100/subvol54470 2024-09-16T15:43:58.554 INFO:teuthology.orchestra.run.smithi123.stdout:state = complete 2024-09-16T15:43:58.554 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-09-16T15:43:58.556 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume rm cephfs subvol54470 subvol_grp11100 2024-09-16T15:43:58.698 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:58.692+0000 7f0837739640 1 -- 172.21.15.123:0/3931265925 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f081c004810 msgr2=0x7f081c004c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:43:58.698 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:58.692+0000 7f0837739640 1 --2- 172.21.15.123:0/3931265925 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f081c004810 0x7f081c004c90 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f08300b9ee0 tx=0x7f082c01c710 comp rx=0 tx=0).stop 2024-09-16T15:43:58.698 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:58.692+0000 7f0837739640 1 -- 172.21.15.123:0/3931265925 shutdown_connections 2024-09-16T15:43:58.698 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:58.692+0000 7f0837739640 1 --2- 172.21.15.123:0/3931265925 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f081c004810 0x7f081c004c90 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:58.698 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:58.692+0000 7f0837739640 1 -- 172.21.15.123:0/3931265925 >> 172.21.15.123:0/3931265925 conn(0x7f08300d3e30 msgr2=0x7f08300d4240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:43:58.698 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:58.692+0000 7f0837739640 1 -- 172.21.15.123:0/3931265925 shutdown_connections 2024-09-16T15:43:58.698 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:58.692+0000 7f0837739640 1 -- 172.21.15.123:0/3931265925 wait complete. 2024-09-16T15:43:58.699 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:58.692+0000 7f0837739640 1 Processor -- start 2024-09-16T15:43:58.699 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:58.692+0000 7f0837739640 1 -- start start 2024-09-16T15:43:58.701 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:58.696+0000 7f0837739640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f081c004810 0x7f08301f50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:43:58.701 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:58.696+0000 7f0837739640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f083009fd40 0x7f083015d090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:43:58.701 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:58.696+0000 7f0837739640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f08300a13f0 0x7f08300b1150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:43:58.701 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:58.696+0000 7f0837739640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f08300d0c80 con 0x7f08300a13f0 2024-09-16T15:43:58.701 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:58.696+0000 7f0837739640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f08300d0b00 con 0x7f083009fd40 2024-09-16T15:43:58.701 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:58.696+0000 7f0837739640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f08300cf450 con 0x7f081c004810 2024-09-16T15:43:58.701 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:58.696+0000 7f0835cd6640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f08300a13f0 0x7f08300b1150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:43:58.701 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:58.696+0000 7f0835cd6640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f08300a13f0 0x7f08300b1150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:34780/0 (socket says 172.21.15.123:34780) 2024-09-16T15:43:58.701 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:58.696+0000 7f0835cd6640 1 -- 172.21.15.123:0/1869535357 learned_addr learned my addr 172.21.15.123:0/1869535357 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:43:58.701 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:58.696+0000 7f0834cd4640 1 --2- 172.21.15.123:0/1869535357 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f083009fd40 0x7f083015d090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:43:58.701 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:58.696+0000 7f08354d5640 1 --2- 172.21.15.123:0/1869535357 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f081c004810 0x7f08301f50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:43:58.702 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:58.696+0000 7f0835cd6640 1 -- 172.21.15.123:0/1869535357 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f081c004810 msgr2=0x7f08301f50e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:43:58.702 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:58.696+0000 7f0835cd6640 1 --2- 172.21.15.123:0/1869535357 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f081c004810 0x7f08301f50e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:58.702 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:58.696+0000 7f0835cd6640 1 -- 172.21.15.123:0/1869535357 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f083009fd40 msgr2=0x7f083015d090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:43:58.702 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:58.696+0000 7f0835cd6640 1 --2- 172.21.15.123:0/1869535357 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f083009fd40 0x7f083015d090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:58.702 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:58.696+0000 7f0835cd6640 1 -- 172.21.15.123:0/1869535357 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f083015d5d0 con 0x7f08300a13f0 2024-09-16T15:43:58.702 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:58.696+0000 7f0834cd4640 1 --2- 172.21.15.123:0/1869535357 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f083009fd40 0x7f083015d090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T15:43:58.702 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:58.696+0000 7f0835cd6640 1 --2- 172.21.15.123:0/1869535357 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f08300a13f0 0x7f08300b1150 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f08300ce410 tx=0x7f082800f960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:43:58.702 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:58.696+0000 7f0825ffb640 1 -- 172.21.15.123:0/1869535357 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f082801a070 con 0x7f08300a13f0 2024-09-16T15:43:58.702 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:58.696+0000 7f0825ffb640 1 -- 172.21.15.123:0/1869535357 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f08280080d0 con 0x7f08300a13f0 2024-09-16T15:43:58.702 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:58.696+0000 7f0825ffb640 1 -- 172.21.15.123:0/1869535357 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0828008520 con 0x7f08300a13f0 2024-09-16T15:43:58.702 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:58.696+0000 7f0837739640 1 -- 172.21.15.123:0/1869535357 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f08300a0630 con 0x7f08300a13f0 2024-09-16T15:43:58.702 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:58.696+0000 7f0837739640 1 -- 172.21.15.123:0/1869535357 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f08300a0bf0 con 0x7f08300a13f0 2024-09-16T15:43:58.703 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:58.696+0000 7f0837739640 1 -- 172.21.15.123:0/1869535357 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f08300a0380 con 0x7f08300a13f0 2024-09-16T15:43:58.703 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:58.696+0000 7f0825ffb640 1 -- 172.21.15.123:0/1869535357 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 6) ==== 97131+0+0 (secure 0 0 0) 0x7f082801d070 con 0x7f08300a13f0 2024-09-16T15:43:58.703 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:58.696+0000 7f0825ffb640 1 --2- 172.21.15.123:0/1869535357 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f0814072f50 0x7f0814075410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:43:58.703 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:58.700+0000 7f0825ffb640 1 -- 172.21.15.123:0/1869535357 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(33..33 src has 1..33) ==== 5709+0+0 (secure 0 0 0) 0x7f08280966f0 con 0x7f08300a13f0 2024-09-16T15:43:58.703 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:58.700+0000 7f08354d5640 1 --2- 172.21.15.123:0/1869535357 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f0814072f50 0x7f0814075410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:43:58.704 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:58.700+0000 7f08354d5640 1 --2- 172.21.15.123:0/1869535357 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f0814072f50 0x7f0814075410 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f08200058d0 tx=0x7f0820005860 comp rx=0 tx=0).ready entity=mgr.4135 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:43:58.706 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:58.700+0000 7f0825ffb640 1 -- 172.21.15.123:0/1869535357 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f0828024070 con 0x7f08300a13f0 2024-09-16T15:43:58.905 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:58.900+0000 7f0837739640 1 -- 172.21.15.123:0/1869535357 --> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] -- mgr_command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "subvol54470", "group_name": "subvol_grp11100", "target": ["mon-mgr", ""]}) -- 0x7f08300cf940 con 0x7f0814072f50 2024-09-16T15:43:58.933 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:58.928+0000 7f0825ffb640 1 -- 172.21.15.123:0/1869535357 <== mgr.4135 v2:172.21.15.184:6832/3114346423 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f08300cf940 con 0x7f0814072f50 2024-09-16T15:43:58.937 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:58.932+0000 7f0837739640 1 -- 172.21.15.123:0/1869535357 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f0814072f50 msgr2=0x7f0814075410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:43:58.937 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:58.932+0000 7f0837739640 1 --2- 172.21.15.123:0/1869535357 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f0814072f50 0x7f0814075410 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f08200058d0 tx=0x7f0820005860 comp rx=0 tx=0).stop 2024-09-16T15:43:58.937 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:58.932+0000 7f0837739640 1 -- 172.21.15.123:0/1869535357 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f08300a13f0 msgr2=0x7f08300b1150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:43:58.937 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:58.932+0000 7f0837739640 1 --2- 172.21.15.123:0/1869535357 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f08300a13f0 0x7f08300b1150 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f08300ce410 tx=0x7f082800f960 comp rx=0 tx=0).stop 2024-09-16T15:43:58.937 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:58.932+0000 7f0837739640 1 -- 172.21.15.123:0/1869535357 shutdown_connections 2024-09-16T15:43:58.937 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:58.932+0000 7f0837739640 1 --2- 172.21.15.123:0/1869535357 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f0814072f50 0x7f0814075410 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:58.937 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:58.932+0000 7f0837739640 1 --2- 172.21.15.123:0/1869535357 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f08300a13f0 0x7f08300b1150 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:58.937 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:58.932+0000 7f0837739640 1 --2- 172.21.15.123:0/1869535357 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f083009fd40 0x7f083015d090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:58.937 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:58.932+0000 7f0837739640 1 --2- 172.21.15.123:0/1869535357 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f081c004810 0x7f08301f50e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:58.938 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:58.932+0000 7f0837739640 1 -- 172.21.15.123:0/1869535357 >> 172.21.15.123:0/1869535357 conn(0x7f08300d3e30 msgr2=0x7f08300b7fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:43:58.938 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:58.932+0000 7f0837739640 1 -- 172.21.15.123:0/1869535357 shutdown_connections 2024-09-16T15:43:58.938 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:58.932+0000 7f0837739640 1 -- 172.21.15.123:0/1869535357 wait complete. 2024-09-16T15:43:58.939 INFO:tasks.ceph.mgr.x.smithi184.stderr:2024-09-16T15:43:58.933+0000 7fda6d0d9640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-16T15:43:58.939 INFO:tasks.ceph.mgr.x.smithi184.stderr:2024-09-16T15:43:58.933+0000 7fda6d0d9640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-16T15:43:58.939 INFO:tasks.ceph.mgr.x.smithi184.stderr:2024-09-16T15:43:58.933+0000 7fda6d0d9640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-16T15:43:58.939 INFO:tasks.ceph.mgr.x.smithi184.stderr:2024-09-16T15:43:58.937+0000 7fda6d0d9640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-16T15:43:58.939 INFO:tasks.ceph.mgr.x.smithi184.stderr:2024-09-16T15:43:58.937+0000 7fda6d0d9640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-16T15:43:58.956 DEBUG:teuthology.orchestra.run.smithi123:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'stat -c %h /home/ubuntu/cephtest/mnt.0/./volumes/_deleting') 2024-09-16T15:43:58.958 INFO:tasks.ceph.mgr.x.smithi184.stderr:2024-09-16T15:43:58.953+0000 7fda6c8d8640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-16T15:43:58.958 INFO:tasks.ceph.mgr.x.smithi184.stderr:2024-09-16T15:43:58.953+0000 7fda6c8d8640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-16T15:43:58.958 INFO:tasks.ceph.mgr.x.smithi184.stderr:2024-09-16T15:43:58.953+0000 7fda6c8d8640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-16T15:43:58.958 INFO:tasks.ceph.mgr.x.smithi184.stderr:2024-09-16T15:43:58.953+0000 7fda6c8d8640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-16T15:43:58.958 INFO:tasks.ceph.mgr.x.smithi184.stderr:2024-09-16T15:43:58.953+0000 7fda6c8d8640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-16T15:43:58.997 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:58 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-09-16T15:43:59.002 INFO:teuthology.orchestra.run.smithi123.stdout:2 2024-09-16T15:43:59.003 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:59 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-16T15:43:59.008 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolumegroup rm cephfs subvol_grp11100 2024-09-16T15:43:59.185 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:59.180+0000 7fcee5184640 1 -- 172.21.15.123:0/4266771152 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcee00703f0 msgr2=0x7fcee006dd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:43:59.185 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:59.180+0000 7fcee5184640 1 --2- 172.21.15.123:0/4266771152 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcee00703f0 0x7fcee006dd70 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7fcee00ce680 tx=0x7fcec801c8c0 comp rx=0 tx=0).stop 2024-09-16T15:43:59.185 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:59.180+0000 7fcee5184640 1 -- 172.21.15.123:0/4266771152 shutdown_connections 2024-09-16T15:43:59.185 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:59.180+0000 7fcee5184640 1 --2- 172.21.15.123:0/4266771152 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcee00703f0 0x7fcee006dd70 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:59.186 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:59.180+0000 7fcee5184640 1 --2- 172.21.15.123:0/4266771152 >> v2:172.21.15.184:3300/0 conn(0x7fcee0151bf0 0x7fcee006feb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:59.186 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:59.180+0000 7fcee5184640 1 -- 172.21.15.123:0/4266771152 >> 172.21.15.123:0/4266771152 conn(0x7fcee00d0bf0 msgr2=0x7fcee00d1000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:43:59.186 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:59.180+0000 7fcee5184640 1 -- 172.21.15.123:0/4266771152 shutdown_connections 2024-09-16T15:43:59.186 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:59.180+0000 7fcee5184640 1 -- 172.21.15.123:0/4266771152 wait complete. 2024-09-16T15:43:59.186 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:59.180+0000 7fcee5184640 1 Processor -- start 2024-09-16T15:43:59.186 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:59.180+0000 7fcee5184640 1 -- start start 2024-09-16T15:43:59.189 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:59.184+0000 7fcee5184640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fcee006e2b0 0x7fcee01641f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:43:59.189 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:59.184+0000 7fcee5184640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcee00703f0 0x7fcee014a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:43:59.189 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:59.184+0000 7fcee5184640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fcee0151bf0 0x7fcee0158c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:43:59.189 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:59.184+0000 7fcee5184640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fcee0166150 con 0x7fcee00703f0 2024-09-16T15:43:59.189 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:59.184+0000 7fcee5184640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fcee0165fd0 con 0x7fcee0151bf0 2024-09-16T15:43:59.189 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:59.184+0000 7fcee5184640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fcee01662d0 con 0x7fcee006e2b0 2024-09-16T15:43:59.189 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:59.184+0000 7fcede59c640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcee00703f0 0x7fcee014a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:43:59.189 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:59.184+0000 7fcede59c640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcee00703f0 0x7fcee014a2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:34792/0 (socket says 172.21.15.123:34792) 2024-09-16T15:43:59.189 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:59.184+0000 7fcede59c640 1 -- 172.21.15.123:0/1611033296 learned_addr learned my addr 172.21.15.123:0/1611033296 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:43:59.189 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:59.184+0000 7fceded9d640 1 --2- 172.21.15.123:0/1611033296 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fcee006e2b0 0x7fcee01641f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:43:59.189 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:59.184+0000 7fcedf59e640 1 --2- 172.21.15.123:0/1611033296 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fcee0151bf0 0x7fcee0158c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:43:59.190 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:59.184+0000 7fcede59c640 1 -- 172.21.15.123:0/1611033296 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fcee006e2b0 msgr2=0x7fcee01641f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:43:59.190 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:59.184+0000 7fcede59c640 1 --2- 172.21.15.123:0/1611033296 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fcee006e2b0 0x7fcee01641f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:59.190 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:59.184+0000 7fcede59c640 1 -- 172.21.15.123:0/1611033296 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fcee0151bf0 msgr2=0x7fcee0158c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:43:59.190 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:59.184+0000 7fcede59c640 1 --2- 172.21.15.123:0/1611033296 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fcee0151bf0 0x7fcee0158c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:59.190 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:59.184+0000 7fcede59c640 1 -- 172.21.15.123:0/1611033296 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcee00ce810 con 0x7fcee00703f0 2024-09-16T15:43:59.190 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:59.184+0000 7fceded9d640 1 --2- 172.21.15.123:0/1611033296 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fcee006e2b0 0x7fcee01641f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T15:43:59.190 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:59.184+0000 7fcede59c640 1 --2- 172.21.15.123:0/1611033296 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcee00703f0 0x7fcee014a2e0 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7fcec80b0040 tx=0x7fcec8005760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:43:59.190 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:59.184+0000 7fced77fe640 1 -- 172.21.15.123:0/1611033296 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcec80b2040 con 0x7fcee00703f0 2024-09-16T15:43:59.190 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:59.184+0000 7fced77fe640 1 -- 172.21.15.123:0/1611033296 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fcec80a6d10 con 0x7fcee00703f0 2024-09-16T15:43:59.190 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:59.184+0000 7fcee5184640 1 -- 172.21.15.123:0/1611033296 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcee0168120 con 0x7fcee00703f0 2024-09-16T15:43:59.190 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:59.184+0000 7fced77fe640 1 -- 172.21.15.123:0/1611033296 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcec8005290 con 0x7fcee00703f0 2024-09-16T15:43:59.190 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:59.184+0000 7fcee5184640 1 -- 172.21.15.123:0/1611033296 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcee014c240 con 0x7fcee00703f0 2024-09-16T15:43:59.191 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:59.184+0000 7fcee5184640 1 -- 172.21.15.123:0/1611033296 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcee006dd70 con 0x7fcee00703f0 2024-09-16T15:43:59.191 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:59.184+0000 7fced77fe640 1 -- 172.21.15.123:0/1611033296 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 6) ==== 97131+0+0 (secure 0 0 0) 0x7fcec80bd060 con 0x7fcee00703f0 2024-09-16T15:43:59.191 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:59.184+0000 7fced77fe640 1 --2- 172.21.15.123:0/1611033296 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fcec0072f50 0x7fcec0075410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:43:59.191 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:59.184+0000 7fced77fe640 1 -- 172.21.15.123:0/1611033296 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(33..33 src has 1..33) ==== 5709+0+0 (secure 0 0 0) 0x7fcec812d820 con 0x7fcee00703f0 2024-09-16T15:43:59.191 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:59.184+0000 7fceded9d640 1 --2- 172.21.15.123:0/1611033296 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fcec0072f50 0x7fcec0075410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:43:59.192 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:59.188+0000 7fceded9d640 1 --2- 172.21.15.123:0/1611033296 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fcec0072f50 0x7fcec0075410 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fcee00ce090 tx=0x7fced80073e0 comp rx=0 tx=0).ready entity=mgr.4135 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:43:59.195 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:59.188+0000 7fced77fe640 1 -- 172.21.15.123:0/1611033296 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7fcec80164b0 con 0x7fcee00703f0 2024-09-16T15:43:59.392 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:59.388+0000 7fcee5184640 1 -- 172.21.15.123:0/1611033296 --> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] -- mgr_command(tid 0: {"prefix": "fs subvolumegroup rm", "vol_name": "cephfs", "group_name": "subvol_grp11100", "target": ["mon-mgr", ""]}) -- 0x7fcee00c7920 con 0x7fcec0072f50 2024-09-16T15:43:59.401 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:59.396+0000 7fced77fe640 1 -- 172.21.15.123:0/1611033296 <== mgr.4135 v2:172.21.15.184:6832/3114346423 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7fcee00c7920 con 0x7fcec0072f50 2024-09-16T15:43:59.404 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:59.400+0000 7fcee5184640 1 -- 172.21.15.123:0/1611033296 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fcec0072f50 msgr2=0x7fcec0075410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:43:59.404 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:59.400+0000 7fcee5184640 1 --2- 172.21.15.123:0/1611033296 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fcec0072f50 0x7fcec0075410 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fcee00ce090 tx=0x7fced80073e0 comp rx=0 tx=0).stop 2024-09-16T15:43:59.404 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:59.400+0000 7fcee5184640 1 -- 172.21.15.123:0/1611033296 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcee00703f0 msgr2=0x7fcee014a2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:43:59.404 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:59.400+0000 7fcee5184640 1 --2- 172.21.15.123:0/1611033296 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcee00703f0 0x7fcee014a2e0 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7fcec80b0040 tx=0x7fcec8005760 comp rx=0 tx=0).stop 2024-09-16T15:43:59.404 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:59.400+0000 7fcee5184640 1 -- 172.21.15.123:0/1611033296 shutdown_connections 2024-09-16T15:43:59.405 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:59.400+0000 7fcee5184640 1 --2- 172.21.15.123:0/1611033296 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fcec0072f50 0x7fcec0075410 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:59.405 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:59.400+0000 7fcee5184640 1 --2- 172.21.15.123:0/1611033296 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fcee0151bf0 0x7fcee0158c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:59.405 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:59.400+0000 7fcee5184640 1 --2- 172.21.15.123:0/1611033296 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcee00703f0 0x7fcee014a2e0 secure :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7fcec80b0040 tx=0x7fcec8005760 comp rx=0 tx=0).stop 2024-09-16T15:43:59.405 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:59.400+0000 7fcee5184640 1 --2- 172.21.15.123:0/1611033296 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fcee006e2b0 0x7fcee01641f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:59.405 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:59.400+0000 7fcee5184640 1 -- 172.21.15.123:0/1611033296 >> 172.21.15.123:0/1611033296 conn(0x7fcee00d0bf0 msgr2=0x7fcee00d0fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:43:59.405 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:59.400+0000 7fcee5184640 1 -- 172.21.15.123:0/1611033296 shutdown_connections 2024-09-16T15:43:59.405 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:59.400+0000 7fcee5184640 1 -- 172.21.15.123:0/1611033296 wait complete. 2024-09-16T15:43:59.421 INFO:teuthology.nuke:Clearing teuthology firewall rules... 2024-09-16T15:43:59.424 DEBUG:teuthology.orchestra.run.smithi123:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-09-16T15:43:59.484 DEBUG:teuthology.orchestra.run.smithi184:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-09-16T15:43:59.556 INFO:teuthology.nuke:Cleared teuthology firewall rules. 2024-09-16T15:43:59.557 INFO:tasks.cephfs.mount:Terminating background process 2024-09-16T15:43:59.557 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2024-09-16T15:43:59.557 DEBUG:teuthology.orchestra.run.smithi123:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-16T15:43:59.563 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2024-09-16T15:43:59.563 DEBUG:teuthology.orchestra.run.smithi123:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-16T15:43:59.612 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-09-16T15:43:59.790 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:59.784+0000 7f58f81a3640 1 -- 172.21.15.123:0/3925508563 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f58d8004810 msgr2=0x7f58d8004c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:43:59.791 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:59.784+0000 7f58f81a3640 1 --2- 172.21.15.123:0/3925508563 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f58d8004810 0x7f58d8004c90 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f58e400c660 tx=0x7f58e401d2b0 comp rx=0 tx=0).stop 2024-09-16T15:43:59.791 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:59.784+0000 7f58f81a3640 1 -- 172.21.15.123:0/3925508563 shutdown_connections 2024-09-16T15:43:59.791 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:59.784+0000 7f58f81a3640 1 --2- 172.21.15.123:0/3925508563 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f58d8004810 0x7f58d8004c90 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:59.791 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:59.784+0000 7f58f81a3640 1 -- 172.21.15.123:0/3925508563 >> 172.21.15.123:0/3925508563 conn(0x7f58f00cfb10 msgr2=0x7f58f00cff20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:43:59.791 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:59.784+0000 7f58f81a3640 1 -- 172.21.15.123:0/3925508563 shutdown_connections 2024-09-16T15:43:59.791 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:59.784+0000 7f58f81a3640 1 -- 172.21.15.123:0/3925508563 wait complete. 2024-09-16T15:43:59.791 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:59.788+0000 7f58f81a3640 1 Processor -- start 2024-09-16T15:43:59.792 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:59.788+0000 7f58f81a3640 1 -- start start 2024-09-16T15:43:59.794 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:59.788+0000 7f58f81a3640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f58d8004810 0x7f58f0145b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:43:59.794 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:59.788+0000 7f58f81a3640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f58f0152610 0x7f58f00c8aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:43:59.794 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:59.788+0000 7f58f81a3640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f58f00c8fe0 0x7f58f00c9460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:43:59.794 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:59.788+0000 7f58f81a3640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f58f00cad00 con 0x7f58d8004810 2024-09-16T15:43:59.794 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:59.788+0000 7f58f81a3640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f58f00cab80 con 0x7f58f00c8fe0 2024-09-16T15:43:59.794 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:59.788+0000 7f58f81a3640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f58f00cae80 con 0x7f58f0152610 2024-09-16T15:43:59.794 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:59.788+0000 7f58f5f3f640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f58d8004810 0x7f58f0145b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:43:59.794 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:59.788+0000 7f58f5f3f640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f58d8004810 0x7f58f0145b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:34806/0 (socket says 172.21.15.123:34806) 2024-09-16T15:43:59.794 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:59.788+0000 7f58f5f3f640 1 -- 172.21.15.123:0/3226713596 learned_addr learned my addr 172.21.15.123:0/3226713596 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:43:59.794 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:59.788+0000 7f58f573e640 1 --2- 172.21.15.123:0/3226713596 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f58f0152610 0x7f58f00c8aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:43:59.794 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:59.788+0000 7f58f6740640 1 --2- 172.21.15.123:0/3226713596 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f58f00c8fe0 0x7f58f00c9460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:43:59.794 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:59.788+0000 7f58f5f3f640 1 -- 172.21.15.123:0/3226713596 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f58f0152610 msgr2=0x7f58f00c8aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:43:59.794 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:59.788+0000 7f58f5f3f640 1 --2- 172.21.15.123:0/3226713596 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f58f0152610 0x7f58f00c8aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:59.795 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:59.788+0000 7f58f5f3f640 1 -- 172.21.15.123:0/3226713596 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f58f00c8fe0 msgr2=0x7f58f00c9460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:43:59.795 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:59.788+0000 7f58f5f3f640 1 --2- 172.21.15.123:0/3226713596 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f58f00c8fe0 0x7f58f00c9460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:59.795 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:59.788+0000 7f58f5f3f640 1 -- 172.21.15.123:0/3226713596 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f58f00c7960 con 0x7f58d8004810 2024-09-16T15:43:59.795 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:59.788+0000 7f58f5f3f640 1 --2- 172.21.15.123:0/3226713596 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f58d8004810 0x7f58f0145b80 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f58e800d5b0 tx=0x7f58e800da70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:43:59.795 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:59.788+0000 7f58ee7fc640 1 -- 172.21.15.123:0/3226713596 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58e80064e0 con 0x7f58d8004810 2024-09-16T15:43:59.795 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:59.788+0000 7f58ee7fc640 1 -- 172.21.15.123:0/3226713596 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f58e8003710 con 0x7f58d8004810 2024-09-16T15:43:59.795 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:59.788+0000 7f58f81a3640 1 -- 172.21.15.123:0/3226713596 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f58f00c7c40 con 0x7f58d8004810 2024-09-16T15:43:59.795 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:59.788+0000 7f58ee7fc640 1 -- 172.21.15.123:0/3226713596 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58e8003b60 con 0x7f58d8004810 2024-09-16T15:43:59.795 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:59.788+0000 7f58f81a3640 1 -- 172.21.15.123:0/3226713596 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f58f00c8200 con 0x7f58d8004810 2024-09-16T15:43:59.796 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:59.788+0000 7f58f81a3640 1 -- 172.21.15.123:0/3226713596 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f58f006f0f0 con 0x7f58d8004810 2024-09-16T15:43:59.796 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:59.792+0000 7f58ee7fc640 1 -- 172.21.15.123:0/3226713596 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 6) ==== 97131+0+0 (secure 0 0 0) 0x7f58e8003d00 con 0x7f58d8004810 2024-09-16T15:43:59.796 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:59.792+0000 7f58ee7fc640 1 --2- 172.21.15.123:0/3226713596 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f58dc073120 0x7f58dc0755e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:43:59.796 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:59.792+0000 7f58ee7fc640 1 -- 172.21.15.123:0/3226713596 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(33..33 src has 1..33) ==== 5709+0+0 (secure 0 0 0) 0x7f58e8094210 con 0x7f58d8004810 2024-09-16T15:43:59.796 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:59.792+0000 7f58f573e640 1 --2- 172.21.15.123:0/3226713596 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f58dc073120 0x7f58dc0755e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:43:59.797 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:59.792+0000 7f58f573e640 1 --2- 172.21.15.123:0/3226713596 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f58dc073120 0x7f58dc0755e0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f58e40ac000 tx=0x7f58e401a0f0 comp rx=0 tx=0).ready entity=mgr.4135 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:43:59.800 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:59.796+0000 7f58ee7fc640 1 -- 172.21.15.123:0/3226713596 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f58e805f4b0 con 0x7f58d8004810 2024-09-16T15:43:59.979 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:59.972+0000 7f58f81a3640 1 -- 172.21.15.123:0/3226713596 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f58f00700e0 con 0x7f58d8004810 2024-09-16T15:43:59.980 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:59.976+0000 7f58ee7fc640 1 -- 172.21.15.123:0/3226713596 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v33) ==== 80+0+0 (secure 0 0 0) 0x7f58e805edb0 con 0x7f58d8004810 2024-09-16T15:43:59.980 INFO:teuthology.orchestra.run.smithi123.stderr:listed 0 entries 2024-09-16T15:43:59.982 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:59.976+0000 7f58f81a3640 1 -- 172.21.15.123:0/3226713596 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f58dc073120 msgr2=0x7f58dc0755e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:43:59.982 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:59.976+0000 7f58f81a3640 1 --2- 172.21.15.123:0/3226713596 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f58dc073120 0x7f58dc0755e0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f58e40ac000 tx=0x7f58e401a0f0 comp rx=0 tx=0).stop 2024-09-16T15:43:59.983 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:59.976+0000 7f58f81a3640 1 -- 172.21.15.123:0/3226713596 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f58d8004810 msgr2=0x7f58f0145b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:43:59.983 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:59.976+0000 7f58f81a3640 1 --2- 172.21.15.123:0/3226713596 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f58d8004810 0x7f58f0145b80 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f58e800d5b0 tx=0x7f58e800da70 comp rx=0 tx=0).stop 2024-09-16T15:43:59.983 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:59.976+0000 7f58f81a3640 1 -- 172.21.15.123:0/3226713596 shutdown_connections 2024-09-16T15:43:59.983 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:59.976+0000 7f58f81a3640 1 --2- 172.21.15.123:0/3226713596 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f58dc073120 0x7f58dc0755e0 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:59.983 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:59.976+0000 7f58f81a3640 1 --2- 172.21.15.123:0/3226713596 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f58f00c8fe0 0x7f58f00c9460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:59.983 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:59.976+0000 7f58f81a3640 1 --2- 172.21.15.123:0/3226713596 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f58f0152610 0x7f58f00c8aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:59.983 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:59.976+0000 7f58f81a3640 1 --2- 172.21.15.123:0/3226713596 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f58d8004810 0x7f58f0145b80 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:43:59.983 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:59.976+0000 7f58f81a3640 1 -- 172.21.15.123:0/3226713596 >> 172.21.15.123:0/3226713596 conn(0x7f58f00cfb10 msgr2=0x7f58f0152b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:43:59.983 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:59.976+0000 7f58f81a3640 1 -- 172.21.15.123:0/3226713596 shutdown_connections 2024-09-16T15:43:59.983 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:43:59.976+0000 7f58f81a3640 1 -- 172.21.15.123:0/3226713596 wait complete. 2024-09-16T15:43:59.997 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi123.front.sepia.ceph.com... 2024-09-16T15:43:59.998 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:43:59.998 DEBUG:teuthology.orchestra.run.smithi123:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2024-09-16T15:44:00.007 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi123.stderr:ceph-fuse[37907]: fuse finished with error 0 and tester_r 0 2024-09-16T15:44:00.009 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi123.front.sepia.ceph.com 2024-09-16T15:44:00.009 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:44:00.009 DEBUG:teuthology.orchestra.run.smithi123:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-09-16T15:44:00.167 INFO:tasks.cephfs.mount:Terminating background process 2024-09-16T15:44:00.167 DEBUG:teuthology.orchestra.run.smithi184:> set -ex 2024-09-16T15:44:00.167 DEBUG:teuthology.orchestra.run.smithi184:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-16T15:44:00.173 DEBUG:teuthology.orchestra.run.smithi184:> set -ex 2024-09-16T15:44:00.173 DEBUG:teuthology.orchestra.run.smithi184:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-16T15:44:00.219 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-09-16T15:44:00.355 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:00.348+0000 7f4e50f2f640 1 -- 172.21.15.123:0/979544833 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4e4c15e000 msgr2=0x7f4e4c0cf450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:44:00.355 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:00.348+0000 7f4e50f2f640 1 --2- 172.21.15.123:0/979544833 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4e4c15e000 0x7f4e4c0cf450 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f4e4c0cea00 tx=0x7f4e4001c8c0 comp rx=0 tx=0).stop 2024-09-16T15:44:00.355 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:00.348+0000 7f4e50f2f640 1 -- 172.21.15.123:0/979544833 shutdown_connections 2024-09-16T15:44:00.355 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:00.348+0000 7f4e50f2f640 1 --2- 172.21.15.123:0/979544833 >> v2:172.21.15.184:3300/0 conn(0x7f4e4c0cf990 0x7f4e4c0d0340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:00.355 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:00.348+0000 7f4e50f2f640 1 --2- 172.21.15.123:0/979544833 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4e4c15e000 0x7f4e4c0cf450 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:00.355 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:00.348+0000 7f4e50f2f640 1 -- 172.21.15.123:0/979544833 >> 172.21.15.123:0/979544833 conn(0x7f4e4c0732a0 msgr2=0x7f4e4c074150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:44:00.355 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:00.348+0000 7f4e50f2f640 1 -- 172.21.15.123:0/979544833 shutdown_connections 2024-09-16T15:44:00.355 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:00.348+0000 7f4e50f2f640 1 -- 172.21.15.123:0/979544833 wait complete. 2024-09-16T15:44:00.356 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:00.352+0000 7f4e50f2f640 1 Processor -- start 2024-09-16T15:44:00.356 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:00.352+0000 7f4e50f2f640 1 -- start start 2024-09-16T15:44:00.358 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:00.352+0000 7f4e50f2f640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4e4c0cf990 0x7f4e4c160d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:44:00.358 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:00.352+0000 7f4e50f2f640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4e4c15dc20 0x7f4e4c0b2730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:44:00.358 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:00.352+0000 7f4e50f2f640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f4e4c15e000 0x7f4e4c0c98e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:44:00.358 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:00.352+0000 7f4e50f2f640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f4e4c16aeb0 con 0x7f4e4c15dc20 2024-09-16T15:44:00.358 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:00.352+0000 7f4e50f2f640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f4e4c16ad30 con 0x7f4e4c0cf990 2024-09-16T15:44:00.358 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:00.352+0000 7f4e50f2f640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f4e4c16b030 con 0x7f4e4c15e000 2024-09-16T15:44:00.359 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:00.352+0000 7f4e49d9b640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4e4c15dc20 0x7f4e4c0b2730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:44:00.359 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:00.352+0000 7f4e49d9b640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4e4c15dc20 0x7f4e4c0b2730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:34824/0 (socket says 172.21.15.123:34824) 2024-09-16T15:44:00.359 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:00.352+0000 7f4e49d9b640 1 -- 172.21.15.123:0/688431348 learned_addr learned my addr 172.21.15.123:0/688431348 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:44:00.359 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:00.352+0000 7f4e4ad9d640 1 --2- 172.21.15.123:0/688431348 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f4e4c15e000 0x7f4e4c0c98e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:44:00.359 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:00.352+0000 7f4e4a59c640 1 --2- 172.21.15.123:0/688431348 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4e4c0cf990 0x7f4e4c160d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:44:00.359 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:00.352+0000 7f4e49d9b640 1 -- 172.21.15.123:0/688431348 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f4e4c15e000 msgr2=0x7f4e4c0c98e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:44:00.359 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:00.352+0000 7f4e49d9b640 1 --2- 172.21.15.123:0/688431348 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f4e4c15e000 0x7f4e4c0c98e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:00.359 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:00.352+0000 7f4e49d9b640 1 -- 172.21.15.123:0/688431348 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4e4c0cf990 msgr2=0x7f4e4c160d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:44:00.359 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:00.352+0000 7f4e49d9b640 1 --2- 172.21.15.123:0/688431348 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4e4c0cf990 0x7f4e4c160d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:00.359 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:00.352+0000 7f4e49d9b640 1 -- 172.21.15.123:0/688431348 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4e4c156190 con 0x7f4e4c15dc20 2024-09-16T15:44:00.359 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:00.352+0000 7f4e49d9b640 1 --2- 172.21.15.123:0/688431348 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4e4c15dc20 0x7f4e4c0b2730 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f4e400b0040 tx=0x7f4e400a5710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:44:00.359 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:00.352+0000 7f4e3affd640 1 -- 172.21.15.123:0/688431348 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e400b2040 con 0x7f4e4c15dc20 2024-09-16T15:44:00.359 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:00.352+0000 7f4e3affd640 1 -- 172.21.15.123:0/688431348 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f4e400043b0 con 0x7f4e4c15dc20 2024-09-16T15:44:00.359 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:00.352+0000 7f4e50f2f640 1 -- 172.21.15.123:0/688431348 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4e4c0b2c70 con 0x7f4e4c15dc20 2024-09-16T15:44:00.359 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:00.352+0000 7f4e3affd640 1 -- 172.21.15.123:0/688431348 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e40005280 con 0x7f4e4c15dc20 2024-09-16T15:44:00.360 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:00.352+0000 7f4e50f2f640 1 -- 172.21.15.123:0/688431348 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4e4c156740 con 0x7f4e4c15dc20 2024-09-16T15:44:00.360 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:00.352+0000 7f4e50f2f640 1 -- 172.21.15.123:0/688431348 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4e4c0d0340 con 0x7f4e4c15dc20 2024-09-16T15:44:00.360 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:00.356+0000 7f4e3affd640 1 -- 172.21.15.123:0/688431348 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 6) ==== 97131+0+0 (secure 0 0 0) 0x7f4e40005420 con 0x7f4e4c15dc20 2024-09-16T15:44:00.361 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:00.356+0000 7f4e3affd640 1 --2- 172.21.15.123:0/688431348 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f4e2c072f50 0x7f4e2c075410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:44:00.361 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:00.356+0000 7f4e3affd640 1 -- 172.21.15.123:0/688431348 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(33..33 src has 1..33) ==== 5709+0+0 (secure 0 0 0) 0x7f4e4012da10 con 0x7f4e4c15dc20 2024-09-16T15:44:00.361 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:00.356+0000 7f4e4a59c640 1 --2- 172.21.15.123:0/688431348 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f4e2c072f50 0x7f4e2c075410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:44:00.362 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:00.356+0000 7f4e4a59c640 1 --2- 172.21.15.123:0/688431348 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f4e2c072f50 0x7f4e2c075410 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f4e3c0016f0 tx=0x7f4e3c009040 comp rx=0 tx=0).ready entity=mgr.4135 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:44:00.364 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:00.360+0000 7f4e3affd640 1 -- 172.21.15.123:0/688431348 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f4e400164b0 con 0x7f4e4c15dc20 2024-09-16T15:44:00.372 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:00.368+0000 7f4e3affd640 1 -- 172.21.15.123:0/688431348 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mgrmap(e 7) ==== 97233+0+0 (secure 0 0 0) 0x7f4e400f4760 con 0x7f4e4c15dc20 2024-09-16T15:44:00.552 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:00.548+0000 7f4e50f2f640 1 -- 172.21.15.123:0/688431348 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f4e4c1577a0 con 0x7f4e4c15dc20 2024-09-16T15:44:00.552 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:00.548+0000 7f4e3affd640 1 -- 172.21.15.123:0/688431348 <== mon.0 v2:172.21.15.123:3300/0 8 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v33) ==== 80+0+0 (secure 0 0 0) 0x7f4e400f8bf0 con 0x7f4e4c15dc20 2024-09-16T15:44:00.552 INFO:teuthology.orchestra.run.smithi123.stderr:listed 0 entries 2024-09-16T15:44:00.555 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:00.548+0000 7f4e50f2f640 1 -- 172.21.15.123:0/688431348 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f4e2c072f50 msgr2=0x7f4e2c075410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:44:00.555 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:00.548+0000 7f4e50f2f640 1 --2- 172.21.15.123:0/688431348 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f4e2c072f50 0x7f4e2c075410 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f4e3c0016f0 tx=0x7f4e3c009040 comp rx=0 tx=0).stop 2024-09-16T15:44:00.555 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:00.548+0000 7f4e50f2f640 1 -- 172.21.15.123:0/688431348 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4e4c15dc20 msgr2=0x7f4e4c0b2730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:44:00.555 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:00.548+0000 7f4e50f2f640 1 --2- 172.21.15.123:0/688431348 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4e4c15dc20 0x7f4e4c0b2730 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f4e400b0040 tx=0x7f4e400a5710 comp rx=0 tx=0).stop 2024-09-16T15:44:00.555 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:00.552+0000 7f4e50f2f640 1 -- 172.21.15.123:0/688431348 shutdown_connections 2024-09-16T15:44:00.556 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:00.552+0000 7f4e50f2f640 1 --2- 172.21.15.123:0/688431348 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f4e2c072f50 0x7f4e2c075410 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:00.556 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:00.552+0000 7f4e50f2f640 1 --2- 172.21.15.123:0/688431348 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f4e4c15e000 0x7f4e4c0c98e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:00.556 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:00.552+0000 7f4e50f2f640 1 --2- 172.21.15.123:0/688431348 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4e4c15dc20 0x7f4e4c0b2730 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:00.556 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:00.552+0000 7f4e50f2f640 1 --2- 172.21.15.123:0/688431348 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4e4c0cf990 0x7f4e4c160d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:00.556 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:00.552+0000 7f4e50f2f640 1 -- 172.21.15.123:0/688431348 >> 172.21.15.123:0/688431348 conn(0x7f4e4c0732a0 msgr2=0x7f4e4c0b57e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:44:00.556 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:00.552+0000 7f4e50f2f640 1 -- 172.21.15.123:0/688431348 shutdown_connections 2024-09-16T15:44:00.556 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:00.552+0000 7f4e50f2f640 1 -- 172.21.15.123:0/688431348 wait complete. 2024-09-16T15:44:00.572 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi184.front.sepia.ceph.com... 2024-09-16T15:44:00.572 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:44:00.572 DEBUG:teuthology.orchestra.run.smithi184:> sudo fusermount -u /home/ubuntu/cephtest/mnt.1 2024-09-16T15:44:00.585 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi184.stderr:ceph-fuse[34638]: fuse finished with error 0 and tester_r 0 2024-09-16T15:44:00.586 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi184.front.sepia.ceph.com 2024-09-16T15:44:00.586 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:44:00.586 DEBUG:teuthology.orchestra.run.smithi184:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2024-09-16T15:44:00.656 INFO:tasks.cephfs.mount:Terminating background process 2024-09-16T15:44:00.656 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2024-09-16T15:44:00.656 DEBUG:teuthology.orchestra.run.smithi123:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-16T15:44:00.662 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16336468k,nr_inodes=4084117,mode=755,inode64 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3274684k,mode=755,inode64 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev,inode64 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=17082 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3274680k,nr_inodes=818670,mode=700,uid=1000,gid=1271,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3274684k,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-09-16T15:44:00.662 INFO:tasks.cephfs.mount:Terminating background process 2024-09-16T15:44:00.662 DEBUG:teuthology.orchestra.run.smithi184:> set -ex 2024-09-16T15:44:00.663 DEBUG:teuthology.orchestra.run.smithi184:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-16T15:44:00.704 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16336488k,nr_inodes=4084122,mode=755,inode64 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3274688k,mode=755,inode64 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev,inode64 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=17834 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3274684k,nr_inodes=818671,mode=700,uid=1000,gid=1271,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3274688k,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-09-16T15:44:00.704 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-16T15:44:00.833 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:00.828+0000 7f8e415f2640 1 -- 172.21.15.123:0/2739161216 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8e3c0d4200 msgr2=0x7f8e3c0d1b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:44:00.833 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:00.828+0000 7f8e415f2640 1 --2- 172.21.15.123:0/2739161216 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8e3c0d4200 0x7f8e3c0d1b60 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f8e2c009900 tx=0x7f8e2c01c850 comp rx=0 tx=0).stop 2024-09-16T15:44:00.833 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:00.828+0000 7f8e415f2640 1 -- 172.21.15.123:0/2739161216 shutdown_connections 2024-09-16T15:44:00.834 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:00.828+0000 7f8e415f2640 1 --2- 172.21.15.123:0/2739161216 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8e3c0d4200 0x7f8e3c0d1b60 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:00.834 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:00.828+0000 7f8e415f2640 1 --2- 172.21.15.123:0/2739161216 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f8e3c15dd60 0x7f8e3c1621d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:00.834 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:00.828+0000 7f8e415f2640 1 -- 172.21.15.123:0/2739161216 >> 172.21.15.123:0/2739161216 conn(0x7f8e3c069650 msgr2=0x7f8e3c09d170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:44:00.834 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:00.828+0000 7f8e415f2640 1 -- 172.21.15.123:0/2739161216 shutdown_connections 2024-09-16T15:44:00.834 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:00.828+0000 7f8e415f2640 1 -- 172.21.15.123:0/2739161216 wait complete. 2024-09-16T15:44:00.834 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:00.828+0000 7f8e415f2640 1 Processor -- start 2024-09-16T15:44:00.834 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:00.828+0000 7f8e415f2640 1 -- start start 2024-09-16T15:44:00.837 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:00.832+0000 7f8e415f2640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f8e3c0b3b20 0x7f8e3c154180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:44:00.837 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:00.832+0000 7f8e415f2640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f8e3c0d4200 0x7f8e3c157a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:44:00.838 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:00.832+0000 7f8e415f2640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8e3c15dd60 0x7f8e3c1583b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:44:00.838 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:00.832+0000 7f8e415f2640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f8e3c16afb0 con 0x7f8e3c15dd60 2024-09-16T15:44:00.838 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:00.832+0000 7f8e415f2640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f8e3c16ae30 con 0x7f8e3c0d4200 2024-09-16T15:44:00.838 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:00.832+0000 7f8e415f2640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f8e3c16b130 con 0x7f8e3c0b3b20 2024-09-16T15:44:00.838 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:00.832+0000 7f8e3b7fe640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8e3c15dd60 0x7f8e3c1583b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:44:00.838 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:00.832+0000 7f8e3b7fe640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8e3c15dd60 0x7f8e3c1583b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:34846/0 (socket says 172.21.15.123:34846) 2024-09-16T15:44:00.838 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:00.832+0000 7f8e3b7fe640 1 -- 172.21.15.123:0/593943352 learned_addr learned my addr 172.21.15.123:0/593943352 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:44:00.838 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:00.832+0000 7f8e3affd640 1 --2- 172.21.15.123:0/593943352 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f8e3c0b3b20 0x7f8e3c154180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:44:00.838 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:00.832+0000 7f8e3a7fc640 1 --2- 172.21.15.123:0/593943352 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f8e3c0d4200 0x7f8e3c157a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:44:00.838 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:00.832+0000 7f8e3b7fe640 1 -- 172.21.15.123:0/593943352 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f8e3c0b3b20 msgr2=0x7f8e3c154180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:44:00.838 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:00.832+0000 7f8e3b7fe640 1 --2- 172.21.15.123:0/593943352 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f8e3c0b3b20 0x7f8e3c154180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:00.839 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:00.832+0000 7f8e3b7fe640 1 -- 172.21.15.123:0/593943352 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f8e3c0d4200 msgr2=0x7f8e3c157a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:44:00.839 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:00.832+0000 7f8e3b7fe640 1 --2- 172.21.15.123:0/593943352 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f8e3c0d4200 0x7f8e3c157a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:00.839 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:00.832+0000 7f8e3b7fe640 1 -- 172.21.15.123:0/593943352 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8e3c155610 con 0x7f8e3c15dd60 2024-09-16T15:44:00.839 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:00.832+0000 7f8e3a7fc640 1 --2- 172.21.15.123:0/593943352 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f8e3c0d4200 0x7f8e3c157a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T15:44:00.839 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:00.832+0000 7f8e3b7fe640 1 --2- 172.21.15.123:0/593943352 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8e3c15dd60 0x7f8e3c1583b0 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f8e2c01af00 tx=0x7f8e2c0a5690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:44:00.839 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:00.832+0000 7f8e337fe640 1 -- 172.21.15.123:0/593943352 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e2c0ba020 con 0x7f8e3c15dd60 2024-09-16T15:44:00.839 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:00.832+0000 7f8e337fe640 1 -- 172.21.15.123:0/593943352 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f8e2c004470 con 0x7f8e3c15dd60 2024-09-16T15:44:00.839 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:00.832+0000 7f8e415f2640 1 -- 172.21.15.123:0/593943352 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8e3c1558f0 con 0x7f8e3c15dd60 2024-09-16T15:44:00.839 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:00.832+0000 7f8e337fe640 1 -- 172.21.15.123:0/593943352 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e2c0b2040 con 0x7f8e3c15dd60 2024-09-16T15:44:00.839 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:00.832+0000 7f8e415f2640 1 -- 172.21.15.123:0/593943352 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8e3c0b7160 con 0x7f8e3c15dd60 2024-09-16T15:44:00.839 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:00.832+0000 7f8e415f2640 1 -- 172.21.15.123:0/593943352 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8e3c0d1b60 con 0x7f8e3c15dd60 2024-09-16T15:44:00.840 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:00.832+0000 7f8e337fe640 1 -- 172.21.15.123:0/593943352 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 7) ==== 97233+0+0 (secure 0 0 0) 0x7f8e2c005290 con 0x7f8e3c15dd60 2024-09-16T15:44:00.840 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:00.832+0000 7f8e337fe640 1 --2- 172.21.15.123:0/593943352 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f8e1c073070 0x7f8e1c075530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:44:00.840 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:00.832+0000 7f8e337fe640 1 -- 172.21.15.123:0/593943352 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(33..33 src has 1..33) ==== 5709+0+0 (secure 0 0 0) 0x7f8e2c12e000 con 0x7f8e3c15dd60 2024-09-16T15:44:00.840 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:00.832+0000 7f8e3affd640 1 --2- 172.21.15.123:0/593943352 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f8e1c073070 0x7f8e1c075530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:44:00.840 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:00.836+0000 7f8e3affd640 1 --2- 172.21.15.123:0/593943352 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f8e1c073070 0x7f8e1c075530 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f8e3c15da00 tx=0x7f8e340095e0 comp rx=0 tx=0).ready entity=mgr.4135 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:44:00.843 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:00.840+0000 7f8e337fe640 1 -- 172.21.15.123:0/593943352 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f8e2c0c4060 con 0x7f8e3c15dd60 2024-09-16T15:44:01.043 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:01.036+0000 7f8e415f2640 1 -- 172.21.15.123:0/593943352 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f8e3c0b9e30 con 0x7f8e3c15dd60 2024-09-16T15:44:01.044 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:01.040+0000 7f8e337fe640 1 -- 172.21.15.123:0/593943352 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 19 v19) ==== 94+0+5229 (secure 0 0 0) 0x7f8e2c0164d0 con 0x7f8e3c15dd60 2024-09-16T15:44:01.044 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-09-16T15:44:01.044 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":19,"btime":"2024-09-16T15:43:54:323710+0000","default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4541,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6837/249053959","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6836","nonce":249053959},{"type":"v1","addr":"172.21.15.184:6837","nonce":249053959}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":13},{"gid":4705,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6835/1285950902","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":1285950902},{"type":"v1","addr":"172.21.15.184:6835","nonce":1285950902}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":4866,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6833/1030632114","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6832","nonce":1030632114},{"type":"v1","addr":"172.21.15.123:6833","nonce":1030632114}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11}],"filesystems":[{"mdsmap":{"epoch":18,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-16T15:43:41.851612+0000","modified":"2024-09-16T15:43:44.919014+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4869},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4869":{"gid":4869,"name":"a","rank":0,"incarnation":14,"state":"up:active","state_seq":5,"addr":"172.21.15.123:6835/822918970","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6834","nonce":822918970},{"type":"v1","addr":"172.21.15.123:6835","nonce":822918970}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4869,"qdb_cluster":[4869]},"id":2}]} 2024-09-16T15:44:01.044 INFO:teuthology.orchestra.run.smithi123.stderr:dumped fsmap epoch 19 2024-09-16T15:44:01.046 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:01.040+0000 7f8e415f2640 1 -- 172.21.15.123:0/593943352 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f8e1c073070 msgr2=0x7f8e1c075530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:44:01.046 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:01.040+0000 7f8e415f2640 1 --2- 172.21.15.123:0/593943352 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f8e1c073070 0x7f8e1c075530 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f8e3c15da00 tx=0x7f8e340095e0 comp rx=0 tx=0).stop 2024-09-16T15:44:01.046 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:01.040+0000 7f8e415f2640 1 -- 172.21.15.123:0/593943352 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8e3c15dd60 msgr2=0x7f8e3c1583b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:44:01.046 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:01.040+0000 7f8e415f2640 1 --2- 172.21.15.123:0/593943352 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8e3c15dd60 0x7f8e3c1583b0 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f8e2c01af00 tx=0x7f8e2c0a5690 comp rx=0 tx=0).stop 2024-09-16T15:44:01.046 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:01.040+0000 7f8e415f2640 1 -- 172.21.15.123:0/593943352 shutdown_connections 2024-09-16T15:44:01.047 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:01.040+0000 7f8e415f2640 1 --2- 172.21.15.123:0/593943352 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f8e1c073070 0x7f8e1c075530 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:01.047 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:01.040+0000 7f8e415f2640 1 --2- 172.21.15.123:0/593943352 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8e3c15dd60 0x7f8e3c1583b0 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:01.047 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:01.040+0000 7f8e415f2640 1 --2- 172.21.15.123:0/593943352 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f8e3c0d4200 0x7f8e3c157a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:01.047 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:01.040+0000 7f8e415f2640 1 --2- 172.21.15.123:0/593943352 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f8e3c0b3b20 0x7f8e3c154180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:01.047 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:01.040+0000 7f8e415f2640 1 -- 172.21.15.123:0/593943352 >> 172.21.15.123:0/593943352 conn(0x7f8e3c069650 msgr2=0x7f8e3c0cf8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:44:01.047 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:01.040+0000 7f8e415f2640 1 -- 172.21.15.123:0/593943352 shutdown_connections 2024-09-16T15:44:01.047 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:01.040+0000 7f8e415f2640 1 -- 172.21.15.123:0/593943352 wait complete. 2024-09-16T15:44:01.061 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-16T15:44:01.184 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:01.180+0000 7fe566c9a640 1 -- 172.21.15.123:0/1542303553 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe548003a60 msgr2=0x7fe548003e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:44:01.184 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:01.180+0000 7fe566c9a640 1 --2- 172.21.15.123:0/1542303553 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe548003a60 0x7fe548003e80 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7fe550009e30 tx=0x7fe55001d170 comp rx=0 tx=0).stop 2024-09-16T15:44:01.184 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:01.180+0000 7fe566c9a640 1 -- 172.21.15.123:0/1542303553 shutdown_connections 2024-09-16T15:44:01.184 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:01.180+0000 7fe566c9a640 1 --2- 172.21.15.123:0/1542303553 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe548003a60 0x7fe548003e80 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:01.184 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:01.180+0000 7fe566c9a640 1 -- 172.21.15.123:0/1542303553 >> 172.21.15.123:0/1542303553 conn(0x7fe560074bd0 msgr2=0x7fe56009aa40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:44:01.184 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:01.180+0000 7fe566c9a640 1 -- 172.21.15.123:0/1542303553 shutdown_connections 2024-09-16T15:44:01.184 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:01.180+0000 7fe566c9a640 1 -- 172.21.15.123:0/1542303553 wait complete. 2024-09-16T15:44:01.185 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:01.180+0000 7fe566c9a640 1 Processor -- start 2024-09-16T15:44:01.185 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:01.180+0000 7fe566c9a640 1 -- start start 2024-09-16T15:44:01.187 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:01.180+0000 7fe566c9a640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe548003a60 0x7fe5600c80b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:44:01.187 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:01.180+0000 7fe566c9a640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fe5601f8ed0 0x7fe5600cb630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:44:01.187 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:01.180+0000 7fe566c9a640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe5601f6010 0x7fe5601f6450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:44:01.187 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:01.180+0000 7fe566c9a640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fe5600cf030 con 0x7fe5601f6010 2024-09-16T15:44:01.187 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:01.180+0000 7fe566c9a640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fe5600ceeb0 con 0x7fe548003a60 2024-09-16T15:44:01.187 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:01.180+0000 7fe566c9a640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fe5600cf1b0 con 0x7fe5601f8ed0 2024-09-16T15:44:01.187 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:01.180+0000 7fe565237640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe5601f6010 0x7fe5601f6450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:44:01.187 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:01.180+0000 7fe565237640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe5601f6010 0x7fe5601f6450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:34864/0 (socket says 172.21.15.123:34864) 2024-09-16T15:44:01.188 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:01.180+0000 7fe565237640 1 -- 172.21.15.123:0/3239768990 learned_addr learned my addr 172.21.15.123:0/3239768990 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:44:01.188 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:01.180+0000 7fe55ffff640 1 --2- 172.21.15.123:0/3239768990 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fe5601f8ed0 0x7fe5600cb630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:44:01.188 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:01.180+0000 7fe564a36640 1 --2- 172.21.15.123:0/3239768990 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe548003a60 0x7fe5600c80b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:44:01.188 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:01.180+0000 7fe565237640 1 -- 172.21.15.123:0/3239768990 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fe5601f8ed0 msgr2=0x7fe5600cb630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:44:01.188 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:01.180+0000 7fe565237640 1 --2- 172.21.15.123:0/3239768990 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fe5601f8ed0 0x7fe5600cb630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:01.188 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:01.180+0000 7fe565237640 1 -- 172.21.15.123:0/3239768990 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe548003a60 msgr2=0x7fe5600c80b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:44:01.188 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:01.180+0000 7fe565237640 1 --2- 172.21.15.123:0/3239768990 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe548003a60 0x7fe5600c80b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:01.188 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:01.180+0000 7fe565237640 1 -- 172.21.15.123:0/3239768990 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe5601f9350 con 0x7fe5601f6010 2024-09-16T15:44:01.189 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:01.180+0000 7fe565237640 1 --2- 172.21.15.123:0/3239768990 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe5601f6010 0x7fe5601f6450 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7fe5600b7cd0 tx=0x7fe558001640 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:44:01.189 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:01.180+0000 7fe55d7fa640 1 -- 172.21.15.123:0/3239768990 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe558011660 con 0x7fe5601f6010 2024-09-16T15:44:01.189 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:01.180+0000 7fe55d7fa640 1 -- 172.21.15.123:0/3239768990 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fe558017070 con 0x7fe5601f6010 2024-09-16T15:44:01.189 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:01.180+0000 7fe566c9a640 1 -- 172.21.15.123:0/3239768990 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe5601f95d0 con 0x7fe5601f6010 2024-09-16T15:44:01.189 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:01.180+0000 7fe55d7fa640 1 -- 172.21.15.123:0/3239768990 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe558011bb0 con 0x7fe5601f6010 2024-09-16T15:44:01.189 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:01.180+0000 7fe566c9a640 1 -- 172.21.15.123:0/3239768990 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe5601f9b90 con 0x7fe5601f6010 2024-09-16T15:44:01.190 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:01.184+0000 7fe566c9a640 1 -- 172.21.15.123:0/3239768990 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe5601f5b40 con 0x7fe5601f6010 2024-09-16T15:44:01.190 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:01.184+0000 7fe55d7fa640 1 -- 172.21.15.123:0/3239768990 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 7) ==== 97233+0+0 (secure 0 0 0) 0x7fe55801c460 con 0x7fe5601f6010 2024-09-16T15:44:01.190 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:01.184+0000 7fe55d7fa640 1 --2- 172.21.15.123:0/3239768990 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fe54c0730a0 0x7fe54c075560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:44:01.190 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:01.184+0000 7fe55d7fa640 1 -- 172.21.15.123:0/3239768990 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(33..33 src has 1..33) ==== 5709+0+0 (secure 0 0 0) 0x7fe558094610 con 0x7fe5601f6010 2024-09-16T15:44:01.190 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:01.184+0000 7fe564a36640 1 --2- 172.21.15.123:0/3239768990 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fe54c0730a0 0x7fe54c075560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:44:01.190 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:01.184+0000 7fe564a36640 1 --2- 172.21.15.123:0/3239768990 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fe54c0730a0 0x7fe54c075560 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fe5500ae000 tx=0x7fe550019f10 comp rx=0 tx=0).ready entity=mgr.4135 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:44:01.193 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:01.188+0000 7fe55d7fa640 1 -- 172.21.15.123:0/3239768990 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7fe5601f9350 con 0x7fe5601f6010 2024-09-16T15:44:01.402 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:01.396+0000 7fe566c9a640 1 -- 172.21.15.123:0/3239768990 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe5600cdc60 con 0x7fe5601f6010 2024-09-16T15:44:01.403 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:01.396+0000 7fe55d7fa640 1 -- 172.21.15.123:0/3239768990 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 19 v19) ==== 94+0+5229 (secure 0 0 0) 0x7fe55805f4d0 con 0x7fe5601f6010 2024-09-16T15:44:01.403 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-09-16T15:44:01.403 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":19,"btime":"2024-09-16T15:43:54:323710+0000","default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4541,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6837/249053959","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6836","nonce":249053959},{"type":"v1","addr":"172.21.15.184:6837","nonce":249053959}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":13},{"gid":4705,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6835/1285950902","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":1285950902},{"type":"v1","addr":"172.21.15.184:6835","nonce":1285950902}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":4866,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6833/1030632114","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6832","nonce":1030632114},{"type":"v1","addr":"172.21.15.123:6833","nonce":1030632114}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11}],"filesystems":[{"mdsmap":{"epoch":18,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-16T15:43:41.851612+0000","modified":"2024-09-16T15:43:44.919014+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4869},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4869":{"gid":4869,"name":"a","rank":0,"incarnation":14,"state":"up:active","state_seq":5,"addr":"172.21.15.123:6835/822918970","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6834","nonce":822918970},{"type":"v1","addr":"172.21.15.123:6835","nonce":822918970}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4869,"qdb_cluster":[4869]},"id":2}]} 2024-09-16T15:44:01.403 INFO:teuthology.orchestra.run.smithi123.stderr:dumped fsmap epoch 19 2024-09-16T15:44:01.405 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:01.400+0000 7fe566c9a640 1 -- 172.21.15.123:0/3239768990 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fe54c0730a0 msgr2=0x7fe54c075560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:44:01.405 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:01.400+0000 7fe566c9a640 1 --2- 172.21.15.123:0/3239768990 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fe54c0730a0 0x7fe54c075560 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fe5500ae000 tx=0x7fe550019f10 comp rx=0 tx=0).stop 2024-09-16T15:44:01.405 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:01.400+0000 7fe566c9a640 1 -- 172.21.15.123:0/3239768990 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe5601f6010 msgr2=0x7fe5601f6450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:44:01.405 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:01.400+0000 7fe566c9a640 1 --2- 172.21.15.123:0/3239768990 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe5601f6010 0x7fe5601f6450 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7fe5600b7cd0 tx=0x7fe558001640 comp rx=0 tx=0).stop 2024-09-16T15:44:01.405 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:01.400+0000 7fe566c9a640 1 -- 172.21.15.123:0/3239768990 shutdown_connections 2024-09-16T15:44:01.405 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:01.400+0000 7fe566c9a640 1 --2- 172.21.15.123:0/3239768990 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fe54c0730a0 0x7fe54c075560 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:01.405 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:01.400+0000 7fe566c9a640 1 --2- 172.21.15.123:0/3239768990 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe5601f6010 0x7fe5601f6450 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:01.406 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:01.400+0000 7fe566c9a640 1 --2- 172.21.15.123:0/3239768990 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fe5601f8ed0 0x7fe5600cb630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:01.406 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:01.400+0000 7fe566c9a640 1 --2- 172.21.15.123:0/3239768990 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe548003a60 0x7fe5600c80b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:01.406 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:01.400+0000 7fe566c9a640 1 -- 172.21.15.123:0/3239768990 >> 172.21.15.123:0/3239768990 conn(0x7fe560074bd0 msgr2=0x7fe5600b6b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:44:01.406 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:01.400+0000 7fe566c9a640 1 -- 172.21.15.123:0/3239768990 shutdown_connections 2024-09-16T15:44:01.406 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:01.400+0000 7fe566c9a640 1 -- 172.21.15.123:0/3239768990 wait complete. 2024-09-16T15:44:01.422 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-16T15:44:01.547 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:01.540+0000 7fd7d207e640 1 -- 172.21.15.123:0/3350057500 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fd7cc06e690 msgr2=0x7fd7cc0b4530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:44:01.547 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:01.540+0000 7fd7d207e640 1 --2- 172.21.15.123:0/3350057500 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fd7cc06e690 0x7fd7cc0b4530 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fd7c4009c80 tx=0x7fd7c401cce0 comp rx=0 tx=0).stop 2024-09-16T15:44:01.547 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:01.540+0000 7fd7d207e640 1 -- 172.21.15.123:0/3350057500 shutdown_connections 2024-09-16T15:44:01.547 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:01.540+0000 7fd7d207e640 1 --2- 172.21.15.123:0/3350057500 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fd7cc06e690 0x7fd7cc0b4530 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:01.548 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:01.540+0000 7fd7d207e640 1 --2- 172.21.15.123:0/3350057500 >> v2:172.21.15.184:3300/0 conn(0x7fd7cc074050 0x7fd7cc06e150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:01.548 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:01.540+0000 7fd7d207e640 1 -- 172.21.15.123:0/3350057500 >> 172.21.15.123:0/3350057500 conn(0x7fd7cc0b2c20 msgr2=0x7fd7cc0b3030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:44:01.548 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:01.540+0000 7fd7d207e640 1 -- 172.21.15.123:0/3350057500 shutdown_connections 2024-09-16T15:44:01.548 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:01.540+0000 7fd7d207e640 1 -- 172.21.15.123:0/3350057500 wait complete. 2024-09-16T15:44:01.548 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:01.544+0000 7fd7d207e640 1 Processor -- start 2024-09-16T15:44:01.548 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:01.544+0000 7fd7d207e640 1 -- start start 2024-09-16T15:44:01.551 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:01.544+0000 7fd7d207e640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd7cc06e690 0x7fd7cc13b0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:44:01.551 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:01.544+0000 7fd7d207e640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fd7cc074050 0x7fd7cc1f6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:44:01.551 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:01.544+0000 7fd7d207e640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd7cc0d05e0 0x7fd7cc1f8c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:44:01.551 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:01.544+0000 7fd7d207e640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fd7cc167d00 con 0x7fd7cc0d05e0 2024-09-16T15:44:01.551 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:01.544+0000 7fd7d207e640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fd7cc167b80 con 0x7fd7cc06e690 2024-09-16T15:44:01.552 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:01.544+0000 7fd7d207e640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fd7cc167e80 con 0x7fd7cc074050 2024-09-16T15:44:01.552 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:01.544+0000 7fd7cbfff640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd7cc0d05e0 0x7fd7cc1f8c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:44:01.552 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:01.544+0000 7fd7cbfff640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd7cc0d05e0 0x7fd7cc1f8c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:34870/0 (socket says 172.21.15.123:34870) 2024-09-16T15:44:01.552 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:01.544+0000 7fd7cbfff640 1 -- 172.21.15.123:0/3467208887 learned_addr learned my addr 172.21.15.123:0/3467208887 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:44:01.552 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:01.544+0000 7fd7cb7fe640 1 --2- 172.21.15.123:0/3467208887 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd7cc06e690 0x7fd7cc13b0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:44:01.552 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:01.544+0000 7fd7c2ffd640 1 --2- 172.21.15.123:0/3467208887 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fd7cc074050 0x7fd7cc1f6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:44:01.552 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:01.544+0000 7fd7cbfff640 1 -- 172.21.15.123:0/3467208887 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fd7cc074050 msgr2=0x7fd7cc1f6070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:44:01.552 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:01.544+0000 7fd7cbfff640 1 --2- 172.21.15.123:0/3467208887 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fd7cc074050 0x7fd7cc1f6070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:01.552 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:01.544+0000 7fd7cbfff640 1 -- 172.21.15.123:0/3467208887 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd7cc06e690 msgr2=0x7fd7cc13b0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:44:01.552 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:01.544+0000 7fd7cbfff640 1 --2- 172.21.15.123:0/3467208887 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd7cc06e690 0x7fd7cc13b0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:01.552 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:01.544+0000 7fd7cbfff640 1 -- 172.21.15.123:0/3467208887 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd7cc1f9640 con 0x7fd7cc0d05e0 2024-09-16T15:44:01.553 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:01.544+0000 7fd7cbfff640 1 --2- 172.21.15.123:0/3467208887 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd7cc0d05e0 0x7fd7cc1f8c50 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7fd7c4009bd0 tx=0x7fd7c40092e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:44:01.553 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:01.544+0000 7fd7c8ff9640 1 -- 172.21.15.123:0/3467208887 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7c40bb070 con 0x7fd7cc0d05e0 2024-09-16T15:44:01.553 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:01.544+0000 7fd7c8ff9640 1 -- 172.21.15.123:0/3467208887 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fd7c40a5e50 con 0x7fd7cc0d05e0 2024-09-16T15:44:01.553 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:01.544+0000 7fd7c8ff9640 1 -- 172.21.15.123:0/3467208887 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7c40b2040 con 0x7fd7cc0d05e0 2024-09-16T15:44:01.553 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:01.544+0000 7fd7d207e640 1 -- 172.21.15.123:0/3467208887 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd7cc1f9900 con 0x7fd7cc0d05e0 2024-09-16T15:44:01.553 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:01.544+0000 7fd7d207e640 1 -- 172.21.15.123:0/3467208887 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd7cc1f5b50 con 0x7fd7cc0d05e0 2024-09-16T15:44:01.553 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:01.544+0000 7fd7d207e640 1 -- 172.21.15.123:0/3467208887 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd7cc1f5660 con 0x7fd7cc0d05e0 2024-09-16T15:44:01.554 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:01.548+0000 7fd7c8ff9640 1 -- 172.21.15.123:0/3467208887 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 7) ==== 97233+0+0 (secure 0 0 0) 0x7fd7c40be070 con 0x7fd7cc0d05e0 2024-09-16T15:44:01.554 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:01.548+0000 7fd7c8ff9640 1 --2- 172.21.15.123:0/3467208887 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fd7ac073070 0x7fd7ac075530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:44:01.554 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:01.548+0000 7fd7c8ff9640 1 -- 172.21.15.123:0/3467208887 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(33..33 src has 1..33) ==== 5709+0+0 (secure 0 0 0) 0x7fd7c412dc10 con 0x7fd7cc0d05e0 2024-09-16T15:44:01.554 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:01.548+0000 7fd7cb7fe640 1 --2- 172.21.15.123:0/3467208887 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fd7ac073070 0x7fd7ac075530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:44:01.554 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:01.548+0000 7fd7cb7fe640 1 --2- 172.21.15.123:0/3467208887 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fd7ac073070 0x7fd7ac075530 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fd7cc070270 tx=0x7fd7bc009040 comp rx=0 tx=0).ready entity=mgr.4135 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:44:01.556 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:01.552+0000 7fd7c8ff9640 1 -- 172.21.15.123:0/3467208887 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7fd7c4016990 con 0x7fd7cc0d05e0 2024-09-16T15:44:01.716 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:01.712+0000 7fd7d207e640 1 -- 172.21.15.123:0/3467208887 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fd7cc06fbf0 con 0x7fd7cc0d05e0 2024-09-16T15:44:01.718 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:01.712+0000 7fd7c8ff9640 1 -- 172.21.15.123:0/3467208887 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v33) ==== 74+0+16789 (secure 0 0 0) 0x7fd7c4101020 con 0x7fd7cc0d05e0 2024-09-16T15:44:01.719 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-09-16T15:44:01.719 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":33,"fsid":"56077f89-18c8-486f-ae7e-eecc0bb7822c","created":"2024-09-16T15:42:26.364892+0000","modified":"2024-09-16T15:43:43.471260+0000","last_up_change":"2024-09-16T15:42:38.707234+0000","last_in_change":"2024-09-16T15:42:29.423662+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-16T15:42:39.287764+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2024-09-16T15:43:39.775631+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"32","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data","create_time":"2024-09-16T15:43:40.809234+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"33","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9e7c1955-a607-4671-9be7-f2ef760ceeaa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6800","nonce":3882044969},{"type":"v1","addr":"172.21.15.123:6801","nonce":3882044969}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6802","nonce":3882044969},{"type":"v1","addr":"172.21.15.123:6803","nonce":3882044969}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6806","nonce":3882044969},{"type":"v1","addr":"172.21.15.123:6807","nonce":3882044969}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6804","nonce":3882044969},{"type":"v1","addr":"172.21.15.123:6805","nonce":3882044969}]},"public_addr":"172.21.15.123:6801/3882044969","cluster_addr":"172.21.15.123:6803/3882044969","heartbeat_back_addr":"172.21.15.123:6807/3882044969","heartbeat_front_addr":"172.21.15.123:6805/3882044969","state":["exists","up"]},{"osd":1,"uuid":"c540cfde-0977-4742-9b17-fd634668e010","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6816","nonce":4037001850},{"type":"v1","addr":"172.21.15.123:6817","nonce":4037001850}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6818","nonce":4037001850},{"type":"v1","addr":"172.21.15.123:6819","nonce":4037001850}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6822","nonce":4037001850},{"type":"v1","addr":"172.21.15.123:6823","nonce":4037001850}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6820","nonce":4037001850},{"type":"v1","addr":"172.21.15.123:6821","nonce":4037001850}]},"public_addr":"172.21.15.123:6817/4037001850","cluster_addr":"172.21.15.123:6819/4037001850","heartbeat_back_addr":"172.21.15.123:6823/4037001850","heartbeat_front_addr":"172.21.15.123:6821/4037001850","state":["exists","up"]},{"osd":2,"uuid":"9864a496-f169-486b-95c4-1d59162793a7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6824","nonce":1332143088},{"type":"v1","addr":"172.21.15.123:6825","nonce":1332143088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6826","nonce":1332143088},{"type":"v1","addr":"172.21.15.123:6827","nonce":1332143088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6830","nonce":1332143088},{"type":"v1","addr":"172.21.15.123:6831","nonce":1332143088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6828","nonce":1332143088},{"type":"v1","addr":"172.21.15.123:6829","nonce":1332143088}]},"public_addr":"172.21.15.123:6825/1332143088","cluster_addr":"172.21.15.123:6827/1332143088","heartbeat_back_addr":"172.21.15.123:6831/1332143088","heartbeat_front_addr":"172.21.15.123:6829/1332143088","state":["exists","up"]},{"osd":3,"uuid":"c6d1f817-f074-4325-95e1-578eba13edda","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6808","nonce":988586998},{"type":"v1","addr":"172.21.15.123:6809","nonce":988586998}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6810","nonce":988586998},{"type":"v1","addr":"172.21.15.123:6811","nonce":988586998}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6814","nonce":988586998},{"type":"v1","addr":"172.21.15.123:6815","nonce":988586998}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6812","nonce":988586998},{"type":"v1","addr":"172.21.15.123:6813","nonce":988586998}]},"public_addr":"172.21.15.123:6809/988586998","cluster_addr":"172.21.15.123:6811/988586998","heartbeat_back_addr":"172.21.15.123:6815/988586998","heartbeat_front_addr":"172.21.15.123:6813/988586998","state":["exists","up"]},{"osd":4,"uuid":"200a8df4-50ef-40a5-b48a-58981269c086","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6816","nonce":3387813890},{"type":"v1","addr":"172.21.15.184:6817","nonce":3387813890}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6818","nonce":3387813890},{"type":"v1","addr":"172.21.15.184:6819","nonce":3387813890}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6822","nonce":3387813890},{"type":"v1","addr":"172.21.15.184:6823","nonce":3387813890}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6820","nonce":3387813890},{"type":"v1","addr":"172.21.15.184:6821","nonce":3387813890}]},"public_addr":"172.21.15.184:6817/3387813890","cluster_addr":"172.21.15.184:6819/3387813890","heartbeat_back_addr":"172.21.15.184:6823/3387813890","heartbeat_front_addr":"172.21.15.184:6821/3387813890","state":["exists","up"]},{"osd":5,"uuid":"32d5a900-da3d-412c-bc19-13861a2368d8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6808","nonce":2680916017},{"type":"v1","addr":"172.21.15.184:6809","nonce":2680916017}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6810","nonce":2680916017},{"type":"v1","addr":"172.21.15.184:6811","nonce":2680916017}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6814","nonce":2680916017},{"type":"v1","addr":"172.21.15.184:6815","nonce":2680916017}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6812","nonce":2680916017},{"type":"v1","addr":"172.21.15.184:6813","nonce":2680916017}]},"public_addr":"172.21.15.184:6809/2680916017","cluster_addr":"172.21.15.184:6811/2680916017","heartbeat_back_addr":"172.21.15.184:6815/2680916017","heartbeat_front_addr":"172.21.15.184:6813/2680916017","state":["exists","up"]},{"osd":6,"uuid":"cde6f39e-aa1b-4260-ac19-774ee6aaca72","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6824","nonce":2183120457},{"type":"v1","addr":"172.21.15.184:6825","nonce":2183120457}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6826","nonce":2183120457},{"type":"v1","addr":"172.21.15.184:6827","nonce":2183120457}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6830","nonce":2183120457},{"type":"v1","addr":"172.21.15.184:6831","nonce":2183120457}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6828","nonce":2183120457},{"type":"v1","addr":"172.21.15.184:6829","nonce":2183120457}]},"public_addr":"172.21.15.184:6825/2183120457","cluster_addr":"172.21.15.184:6827/2183120457","heartbeat_back_addr":"172.21.15.184:6831/2183120457","heartbeat_front_addr":"172.21.15.184:6829/2183120457","state":["exists","up"]},{"osd":7,"uuid":"4f381778-eee1-47c6-804f-df0e6b33e313","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6800","nonce":4102957285},{"type":"v1","addr":"172.21.15.184:6801","nonce":4102957285}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6802","nonce":4102957285},{"type":"v1","addr":"172.21.15.184:6803","nonce":4102957285}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6806","nonce":4102957285},{"type":"v1","addr":"172.21.15.184:6807","nonce":4102957285}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6804","nonce":4102957285},{"type":"v1","addr":"172.21.15.184:6805","nonce":4102957285}]},"public_addr":"172.21.15.184:6801/4102957285","cluster_addr":"172.21.15.184:6803/4102957285","heartbeat_back_addr":"172.21.15.184:6807/4102957285","heartbeat_front_addr":"172.21.15.184:6805/4102957285","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:35.972210+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.222007+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.198617+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.385663+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.358935+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.156066+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.752973+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:35.682846+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T15:44:01.721 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:01.716+0000 7fd7d207e640 1 -- 172.21.15.123:0/3467208887 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fd7ac073070 msgr2=0x7fd7ac075530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:44:01.721 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:01.716+0000 7fd7d207e640 1 --2- 172.21.15.123:0/3467208887 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fd7ac073070 0x7fd7ac075530 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fd7cc070270 tx=0x7fd7bc009040 comp rx=0 tx=0).stop 2024-09-16T15:44:01.721 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:01.716+0000 7fd7d207e640 1 -- 172.21.15.123:0/3467208887 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd7cc0d05e0 msgr2=0x7fd7cc1f8c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:44:01.721 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:01.716+0000 7fd7d207e640 1 --2- 172.21.15.123:0/3467208887 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd7cc0d05e0 0x7fd7cc1f8c50 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7fd7c4009bd0 tx=0x7fd7c40092e0 comp rx=0 tx=0).stop 2024-09-16T15:44:01.721 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:01.716+0000 7fd7d207e640 1 -- 172.21.15.123:0/3467208887 shutdown_connections 2024-09-16T15:44:01.721 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:01.716+0000 7fd7d207e640 1 --2- 172.21.15.123:0/3467208887 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fd7ac073070 0x7fd7ac075530 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:01.721 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:01.716+0000 7fd7d207e640 1 --2- 172.21.15.123:0/3467208887 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd7cc0d05e0 0x7fd7cc1f8c50 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:01.721 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:01.716+0000 7fd7d207e640 1 --2- 172.21.15.123:0/3467208887 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fd7cc074050 0x7fd7cc1f6070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:01.721 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:01.716+0000 7fd7d207e640 1 --2- 172.21.15.123:0/3467208887 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd7cc06e690 0x7fd7cc13b0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:01.721 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:01.716+0000 7fd7d207e640 1 -- 172.21.15.123:0/3467208887 >> 172.21.15.123:0/3467208887 conn(0x7fd7cc0b2c20 msgr2=0x7fd7cc0b3000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:44:01.722 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:01.716+0000 7fd7d207e640 1 -- 172.21.15.123:0/3467208887 shutdown_connections 2024-09-16T15:44:01.722 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:01.716+0000 7fd7d207e640 1 -- 172.21.15.123:0/3467208887 wait complete. 2024-09-16T15:44:01.736 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-09-16T15:44:01.737 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-16T15:44:01.861 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:01.856+0000 7f037f5e7640 1 -- 172.21.15.123:0/2088525667 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f03780b3100 msgr2=0x7f03780cc1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:44:01.861 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:01.856+0000 7f037f5e7640 1 --2- 172.21.15.123:0/2088525667 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f03780b3100 0x7f03780cc1b0 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f03780c45f0 tx=0x7f036801c8c0 comp rx=0 tx=0).stop 2024-09-16T15:44:01.861 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:01.856+0000 7f037f5e7640 1 -- 172.21.15.123:0/2088525667 shutdown_connections 2024-09-16T15:44:01.861 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:01.856+0000 7f037f5e7640 1 --2- 172.21.15.123:0/2088525667 >> v2:172.21.15.184:3300/0 conn(0x7f03780cc6f0 0x7f03780cf650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:01.861 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:01.856+0000 7f037f5e7640 1 --2- 172.21.15.123:0/2088525667 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f03780b3100 0x7f03780cc1b0 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:01.861 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:01.856+0000 7f037f5e7640 1 -- 172.21.15.123:0/2088525667 >> 172.21.15.123:0/2088525667 conn(0x7f03780b2610 msgr2=0x7f03780b29f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:44:01.861 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:01.856+0000 7f037f5e7640 1 -- 172.21.15.123:0/2088525667 shutdown_connections 2024-09-16T15:44:01.861 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:01.856+0000 7f037f5e7640 1 -- 172.21.15.123:0/2088525667 wait complete. 2024-09-16T15:44:01.862 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:01.856+0000 7f037f5e7640 1 Processor -- start 2024-09-16T15:44:01.862 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:01.856+0000 7f037f5e7640 1 -- start start 2024-09-16T15:44:01.863 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:01.856+0000 7f037f5e7640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f03780b3100 0x7f0378154400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:44:01.864 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:01.856+0000 7f037f5e7640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f03780cc6f0 0x7f0378154ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:44:01.864 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:01.856+0000 7f037f5e7640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f037815ac00 0x7f0378155010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:44:01.864 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:01.856+0000 7f037f5e7640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f03781682d0 con 0x7f037815ac00 2024-09-16T15:44:01.864 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:01.856+0000 7f037f5e7640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f0378070a00 con 0x7f03780b3100 2024-09-16T15:44:01.864 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:01.856+0000 7f037f5e7640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f0378168450 con 0x7f03780cc6f0 2024-09-16T15:44:01.864 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:01.856+0000 7f037db84640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f037815ac00 0x7f0378155010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:44:01.864 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:01.856+0000 7f037db84640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f037815ac00 0x7f0378155010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:34892/0 (socket says 172.21.15.123:34892) 2024-09-16T15:44:01.864 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:01.856+0000 7f037db84640 1 -- 172.21.15.123:0/306290048 learned_addr learned my addr 172.21.15.123:0/306290048 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:44:01.864 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:01.860+0000 7f037cb82640 1 --2- 172.21.15.123:0/306290048 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f03780cc6f0 0x7f0378154ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:44:01.864 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:01.860+0000 7f037d383640 1 --2- 172.21.15.123:0/306290048 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f03780b3100 0x7f0378154400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:44:01.864 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:01.860+0000 7f037db84640 1 -- 172.21.15.123:0/306290048 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f03780cc6f0 msgr2=0x7f0378154ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:44:01.864 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:01.860+0000 7f037db84640 1 --2- 172.21.15.123:0/306290048 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f03780cc6f0 0x7f0378154ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:01.865 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:01.860+0000 7f037db84640 1 -- 172.21.15.123:0/306290048 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f03780b3100 msgr2=0x7f0378154400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:44:01.865 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:01.860+0000 7f037db84640 1 --2- 172.21.15.123:0/306290048 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f03780b3100 0x7f0378154400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:01.865 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:01.860+0000 7f037db84640 1 -- 172.21.15.123:0/306290048 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0378159b50 con 0x7f037815ac00 2024-09-16T15:44:01.865 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:01.860+0000 7f037d383640 1 --2- 172.21.15.123:0/306290048 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f03780b3100 0x7f0378154400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T15:44:01.865 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:01.860+0000 7f037db84640 1 --2- 172.21.15.123:0/306290048 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f037815ac00 0x7f0378155010 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f037000cce0 tx=0x7f037000a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:44:01.865 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:01.860+0000 7f036dffb640 1 -- 172.21.15.123:0/306290048 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f037000f450 con 0x7f037815ac00 2024-09-16T15:44:01.865 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:01.860+0000 7f036dffb640 1 -- 172.21.15.123:0/306290048 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f037000f700 con 0x7f037815ac00 2024-09-16T15:44:01.865 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:01.860+0000 7f036dffb640 1 -- 172.21.15.123:0/306290048 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f037000fb50 con 0x7f037815ac00 2024-09-16T15:44:01.865 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:01.860+0000 7f037f5e7640 1 -- 172.21.15.123:0/306290048 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0378158f40 con 0x7f037815ac00 2024-09-16T15:44:01.865 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:01.860+0000 7f037f5e7640 1 -- 172.21.15.123:0/306290048 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0378159500 con 0x7f037815ac00 2024-09-16T15:44:01.866 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:01.860+0000 7f037f5e7640 1 -- 172.21.15.123:0/306290048 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f03780cc1b0 con 0x7f037815ac00 2024-09-16T15:44:01.866 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:01.860+0000 7f036dffb640 1 -- 172.21.15.123:0/306290048 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 7) ==== 97233+0+0 (secure 0 0 0) 0x7f037000fcf0 con 0x7f037815ac00 2024-09-16T15:44:01.866 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:01.860+0000 7f036dffb640 1 --2- 172.21.15.123:0/306290048 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f035c073070 0x7f035c075530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:44:01.866 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:01.860+0000 7f036dffb640 1 -- 172.21.15.123:0/306290048 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(33..33 src has 1..33) ==== 5709+0+0 (secure 0 0 0) 0x7f0370094dd0 con 0x7f037815ac00 2024-09-16T15:44:01.866 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:01.860+0000 7f037d383640 1 --2- 172.21.15.123:0/306290048 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f035c073070 0x7f035c075530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:44:01.867 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:01.860+0000 7f037d383640 1 --2- 172.21.15.123:0/306290048 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f035c073070 0x7f035c075530 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f037400a850 tx=0x7f0374008040 comp rx=0 tx=0).ready entity=mgr.4135 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:44:01.870 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:01.864+0000 7f036dffb640 1 -- 172.21.15.123:0/306290048 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f0370014030 con 0x7f037815ac00 2024-09-16T15:44:02.078 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:02.072+0000 7f037f5e7640 1 -- 172.21.15.123:0/306290048 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f037815bf70 con 0x7f037815ac00 2024-09-16T15:44:02.079 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:02.072+0000 7f036dffb640 1 -- 172.21.15.123:0/306290048 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 19 v19) ==== 94+0+5229 (secure 0 0 0) 0x7f0370096090 con 0x7f037815ac00 2024-09-16T15:44:02.079 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-09-16T15:44:02.079 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":19,"btime":"2024-09-16T15:43:54:323710+0000","default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4541,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6837/249053959","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6836","nonce":249053959},{"type":"v1","addr":"172.21.15.184:6837","nonce":249053959}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":13},{"gid":4705,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6835/1285950902","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":1285950902},{"type":"v1","addr":"172.21.15.184:6835","nonce":1285950902}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":4866,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6833/1030632114","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6832","nonce":1030632114},{"type":"v1","addr":"172.21.15.123:6833","nonce":1030632114}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11}],"filesystems":[{"mdsmap":{"epoch":18,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-16T15:43:41.851612+0000","modified":"2024-09-16T15:43:44.919014+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4869},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4869":{"gid":4869,"name":"a","rank":0,"incarnation":14,"state":"up:active","state_seq":5,"addr":"172.21.15.123:6835/822918970","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6834","nonce":822918970},{"type":"v1","addr":"172.21.15.123:6835","nonce":822918970}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4869,"qdb_cluster":[4869]},"id":2}]} 2024-09-16T15:44:02.079 INFO:teuthology.orchestra.run.smithi123.stderr:dumped fsmap epoch 19 2024-09-16T15:44:02.081 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:02.076+0000 7f037f5e7640 1 -- 172.21.15.123:0/306290048 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f035c073070 msgr2=0x7f035c075530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:44:02.082 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:02.076+0000 7f037f5e7640 1 --2- 172.21.15.123:0/306290048 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f035c073070 0x7f035c075530 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f037400a850 tx=0x7f0374008040 comp rx=0 tx=0).stop 2024-09-16T15:44:02.082 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:02.076+0000 7f037f5e7640 1 -- 172.21.15.123:0/306290048 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f037815ac00 msgr2=0x7f0378155010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:44:02.082 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:02.076+0000 7f037f5e7640 1 --2- 172.21.15.123:0/306290048 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f037815ac00 0x7f0378155010 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f037000cce0 tx=0x7f037000a620 comp rx=0 tx=0).stop 2024-09-16T15:44:02.082 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:02.076+0000 7f037f5e7640 1 -- 172.21.15.123:0/306290048 shutdown_connections 2024-09-16T15:44:02.082 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:02.076+0000 7f037f5e7640 1 --2- 172.21.15.123:0/306290048 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f035c073070 0x7f035c075530 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:02.082 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:02.076+0000 7f037f5e7640 1 --2- 172.21.15.123:0/306290048 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f037815ac00 0x7f0378155010 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:02.082 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:02.076+0000 7f037f5e7640 1 --2- 172.21.15.123:0/306290048 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f03780cc6f0 0x7f0378154ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:02.082 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:02.076+0000 7f037f5e7640 1 --2- 172.21.15.123:0/306290048 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f03780b3100 0x7f0378154400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:02.082 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:02.076+0000 7f037f5e7640 1 -- 172.21.15.123:0/306290048 >> 172.21.15.123:0/306290048 conn(0x7f03780b2610 msgr2=0x7f0378159d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:44:02.082 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:02.076+0000 7f037f5e7640 1 -- 172.21.15.123:0/306290048 shutdown_connections 2024-09-16T15:44:02.082 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:02.076+0000 7f037f5e7640 1 -- 172.21.15.123:0/306290048 wait complete. 2024-09-16T15:44:02.098 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-16T15:44:02.220 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:02.216+0000 7f6b02151640 1 -- 172.21.15.123:0/902378942 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6ae8004810 msgr2=0x7f6ae8004c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:44:02.220 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:02.216+0000 7f6b02151640 1 --2- 172.21.15.123:0/902378942 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6ae8004810 0x7f6ae8004c90 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f6ae400c310 tx=0x7f6ae401cf80 comp rx=0 tx=0).stop 2024-09-16T15:44:02.220 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:02.216+0000 7f6b02151640 1 -- 172.21.15.123:0/902378942 shutdown_connections 2024-09-16T15:44:02.220 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:02.216+0000 7f6b02151640 1 --2- 172.21.15.123:0/902378942 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6ae8004810 0x7f6ae8004c90 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:02.220 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:02.216+0000 7f6b02151640 1 -- 172.21.15.123:0/902378942 >> 172.21.15.123:0/902378942 conn(0x7f6afc0cfb10 msgr2=0x7f6afc0cff20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:44:02.221 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:02.216+0000 7f6b02151640 1 -- 172.21.15.123:0/902378942 shutdown_connections 2024-09-16T15:44:02.221 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:02.216+0000 7f6b02151640 1 -- 172.21.15.123:0/902378942 wait complete. 2024-09-16T15:44:02.221 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:02.216+0000 7f6b02151640 1 Processor -- start 2024-09-16T15:44:02.221 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:02.216+0000 7f6b02151640 1 -- start start 2024-09-16T15:44:02.223 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:02.216+0000 7f6b02151640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f6ae8004810 0x7f6afc1e60b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:44:02.223 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:02.216+0000 7f6b02151640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6afc1f30a0 0x7f6afc0c52c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:44:02.223 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:02.216+0000 7f6b02151640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6afc0c5800 0x7f6afc15bbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:44:02.223 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:02.216+0000 7f6b02151640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f6afc06eee0 con 0x7f6afc1f30a0 2024-09-16T15:44:02.223 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:02.216+0000 7f6b02151640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f6afc06ed60 con 0x7f6afc0c5800 2024-09-16T15:44:02.224 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:02.216+0000 7f6b02151640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f6afc06f060 con 0x7f6ae8004810 2024-09-16T15:44:02.224 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:02.216+0000 7f6af2ffd640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6afc1f30a0 0x7f6afc0c52c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:44:02.224 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:02.216+0000 7f6af2ffd640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6afc1f30a0 0x7f6afc0c52c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:34908/0 (socket says 172.21.15.123:34908) 2024-09-16T15:44:02.224 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:02.216+0000 7f6af2ffd640 1 -- 172.21.15.123:0/4128936638 learned_addr learned my addr 172.21.15.123:0/4128936638 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:44:02.224 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:02.216+0000 7f6af2ffd640 1 -- 172.21.15.123:0/4128936638 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f6ae8004810 msgr2=0x7f6afc1e60b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:44:02.224 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:02.216+0000 7f6afbfff640 1 --2- 172.21.15.123:0/4128936638 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6afc0c5800 0x7f6afc15bbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:44:02.224 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:02.216+0000 7f6afb7fe640 1 --2- 172.21.15.123:0/4128936638 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f6ae8004810 0x7f6afc1e60b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:44:02.224 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:02.216+0000 7f6af2ffd640 1 --2- 172.21.15.123:0/4128936638 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f6ae8004810 0x7f6afc1e60b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:02.224 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:02.216+0000 7f6af2ffd640 1 -- 172.21.15.123:0/4128936638 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6afc0c5800 msgr2=0x7f6afc15bbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:44:02.224 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:02.216+0000 7f6af2ffd640 1 --2- 172.21.15.123:0/4128936638 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6afc0c5800 0x7f6afc15bbe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:02.224 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:02.216+0000 7f6af2ffd640 1 -- 172.21.15.123:0/4128936638 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6afc1f9440 con 0x7f6afc1f30a0 2024-09-16T15:44:02.224 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:02.216+0000 7f6af2ffd640 1 --2- 172.21.15.123:0/4128936638 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6afc1f30a0 0x7f6afc0c52c0 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f6ae400c310 tx=0x7f6ae4006cf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:44:02.224 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:02.216+0000 7f6af8ff9640 1 -- 172.21.15.123:0/4128936638 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ae40014c0 con 0x7f6afc1f30a0 2024-09-16T15:44:02.224 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:02.216+0000 7f6af8ff9640 1 -- 172.21.15.123:0/4128936638 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f6ae4001680 con 0x7f6afc1f30a0 2024-09-16T15:44:02.225 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:02.216+0000 7f6b02151640 1 -- 172.21.15.123:0/4128936638 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6afc1f9720 con 0x7f6afc1f30a0 2024-09-16T15:44:02.225 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:02.216+0000 7f6af8ff9640 1 -- 172.21.15.123:0/4128936638 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ae4006330 con 0x7f6afc1f30a0 2024-09-16T15:44:02.225 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:02.216+0000 7f6b02151640 1 -- 172.21.15.123:0/4128936638 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6afc1f9ce0 con 0x7f6afc1f30a0 2024-09-16T15:44:02.225 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:02.220+0000 7f6b02151640 1 -- 172.21.15.123:0/4128936638 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6afc15aa60 con 0x7f6afc1f30a0 2024-09-16T15:44:02.225 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:02.220+0000 7f6af8ff9640 1 -- 172.21.15.123:0/4128936638 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 7) ==== 97233+0+0 (secure 0 0 0) 0x7f6ae40a5cd0 con 0x7f6afc1f30a0 2024-09-16T15:44:02.226 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:02.220+0000 7f6af8ff9640 1 --2- 172.21.15.123:0/4128936638 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f6adc0730a0 0x7f6adc075560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:44:02.226 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:02.220+0000 7f6af8ff9640 1 -- 172.21.15.123:0/4128936638 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(33..33 src has 1..33) ==== 5709+0+0 (secure 0 0 0) 0x7f6ae412d990 con 0x7f6afc1f30a0 2024-09-16T15:44:02.226 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:02.220+0000 7f6afb7fe640 1 --2- 172.21.15.123:0/4128936638 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f6adc0730a0 0x7f6adc075560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:44:02.226 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:02.220+0000 7f6afb7fe640 1 --2- 172.21.15.123:0/4128936638 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f6adc0730a0 0x7f6adc075560 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f6afc1e7500 tx=0x7f6aec0069d0 comp rx=0 tx=0).ready entity=mgr.4135 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:44:02.229 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:02.224+0000 7f6af8ff9640 1 -- 172.21.15.123:0/4128936638 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f6ae40be060 con 0x7f6afc1f30a0 2024-09-16T15:44:02.423 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:02.420+0000 7f6b02151640 1 -- 172.21.15.123:0/4128936638 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f6afc0d06c0 con 0x7f6afc1f30a0 2024-09-16T15:44:02.424 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:02.420+0000 7f6af8ff9640 1 -- 172.21.15.123:0/4128936638 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 19 v19) ==== 94+0+5229 (secure 0 0 0) 0x7f6ae4016d30 con 0x7f6afc1f30a0 2024-09-16T15:44:02.424 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-09-16T15:44:02.424 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":19,"btime":"2024-09-16T15:43:54:323710+0000","default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4541,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6837/249053959","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6836","nonce":249053959},{"type":"v1","addr":"172.21.15.184:6837","nonce":249053959}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":13},{"gid":4705,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6835/1285950902","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":1285950902},{"type":"v1","addr":"172.21.15.184:6835","nonce":1285950902}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":4866,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6833/1030632114","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6832","nonce":1030632114},{"type":"v1","addr":"172.21.15.123:6833","nonce":1030632114}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11}],"filesystems":[{"mdsmap":{"epoch":18,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-16T15:43:41.851612+0000","modified":"2024-09-16T15:43:44.919014+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4869},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4869":{"gid":4869,"name":"a","rank":0,"incarnation":14,"state":"up:active","state_seq":5,"addr":"172.21.15.123:6835/822918970","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6834","nonce":822918970},{"type":"v1","addr":"172.21.15.123:6835","nonce":822918970}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4869,"qdb_cluster":[4869]},"id":2}]} 2024-09-16T15:44:02.425 INFO:teuthology.orchestra.run.smithi123.stderr:dumped fsmap epoch 19 2024-09-16T15:44:02.426 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:02.420+0000 7f6b02151640 1 -- 172.21.15.123:0/4128936638 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f6adc0730a0 msgr2=0x7f6adc075560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:44:02.427 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:02.420+0000 7f6b02151640 1 --2- 172.21.15.123:0/4128936638 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f6adc0730a0 0x7f6adc075560 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f6afc1e7500 tx=0x7f6aec0069d0 comp rx=0 tx=0).stop 2024-09-16T15:44:02.427 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:02.420+0000 7f6b02151640 1 -- 172.21.15.123:0/4128936638 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6afc1f30a0 msgr2=0x7f6afc0c52c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:44:02.427 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:02.420+0000 7f6b02151640 1 --2- 172.21.15.123:0/4128936638 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6afc1f30a0 0x7f6afc0c52c0 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f6ae400c310 tx=0x7f6ae4006cf0 comp rx=0 tx=0).stop 2024-09-16T15:44:02.427 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:02.420+0000 7f6b02151640 1 -- 172.21.15.123:0/4128936638 shutdown_connections 2024-09-16T15:44:02.427 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:02.420+0000 7f6b02151640 1 --2- 172.21.15.123:0/4128936638 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f6adc0730a0 0x7f6adc075560 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:02.427 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:02.420+0000 7f6b02151640 1 --2- 172.21.15.123:0/4128936638 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6afc0c5800 0x7f6afc15bbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:02.427 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:02.420+0000 7f6b02151640 1 --2- 172.21.15.123:0/4128936638 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6afc1f30a0 0x7f6afc0c52c0 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:02.427 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:02.420+0000 7f6b02151640 1 --2- 172.21.15.123:0/4128936638 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f6ae8004810 0x7f6afc1e60b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:02.427 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:02.420+0000 7f6b02151640 1 -- 172.21.15.123:0/4128936638 >> 172.21.15.123:0/4128936638 conn(0x7f6afc0cfb10 msgr2=0x7f6afc0b08d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:44:02.427 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:02.420+0000 7f6b02151640 1 -- 172.21.15.123:0/4128936638 shutdown_connections 2024-09-16T15:44:02.427 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:02.420+0000 7f6b02151640 1 -- 172.21.15.123:0/4128936638 wait complete. 2024-09-16T15:44:02.442 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-16T15:44:02.564 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:02.560+0000 7fe6f7c64640 1 -- 172.21.15.123:0/512729696 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe6f00748d0 msgr2=0x7fe6f0068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:44:02.564 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:02.560+0000 7fe6f7c64640 1 --2- 172.21.15.123:0/512729696 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe6f00748d0 0x7fe6f0068960 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7fe6f0071a50 tx=0x7fe6e001c8c0 comp rx=0 tx=0).stop 2024-09-16T15:44:02.564 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:02.560+0000 7fe6f7c64640 1 -- 172.21.15.123:0/512729696 shutdown_connections 2024-09-16T15:44:02.564 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:02.560+0000 7fe6f7c64640 1 --2- 172.21.15.123:0/512729696 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fe6f00b2910 0x7fe6f00b4c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:02.564 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:02.560+0000 7fe6f7c64640 1 --2- 172.21.15.123:0/512729696 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe6f00748d0 0x7fe6f0068960 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:02.565 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:02.560+0000 7fe6f7c64640 1 -- 172.21.15.123:0/512729696 >> 172.21.15.123:0/512729696 conn(0x7fe6f00b1df0 msgr2=0x7fe6f00b2200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:44:02.565 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:02.560+0000 7fe6f7c64640 1 -- 172.21.15.123:0/512729696 shutdown_connections 2024-09-16T15:44:02.565 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:02.560+0000 7fe6f7c64640 1 -- 172.21.15.123:0/512729696 wait complete. 2024-09-16T15:44:02.565 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:02.560+0000 7fe6f7c64640 1 Processor -- start 2024-09-16T15:44:02.565 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:02.560+0000 7fe6f7c64640 1 -- start start 2024-09-16T15:44:02.567 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:02.560+0000 7fe6f7c64640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fe6f00748d0 0x7fe6f0156300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:44:02.567 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:02.560+0000 7fe6f7c64640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe6f00b2910 0x7fe6f01fbb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:44:02.567 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:02.560+0000 7fe6f7c64640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe6f015f420 0x7fe6f00cded0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:44:02.567 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:02.560+0000 7fe6f7c64640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fe6f0168bd0 con 0x7fe6f00b2910 2024-09-16T15:44:02.567 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:02.560+0000 7fe6f7c64640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fe6f0168d50 con 0x7fe6f015f420 2024-09-16T15:44:02.567 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:02.560+0000 7fe6f7c64640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fe6f0168ed0 con 0x7fe6f00748d0 2024-09-16T15:44:02.567 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:02.560+0000 7fe6f51ff640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe6f00b2910 0x7fe6f01fbb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:44:02.567 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:02.560+0000 7fe6f51ff640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe6f00b2910 0x7fe6f01fbb10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:57830/0 (socket says 172.21.15.123:57830) 2024-09-16T15:44:02.567 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:02.560+0000 7fe6f51ff640 1 -- 172.21.15.123:0/2839343297 learned_addr learned my addr 172.21.15.123:0/2839343297 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:44:02.567 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:02.560+0000 7fe6f51ff640 1 -- 172.21.15.123:0/2839343297 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fe6f00748d0 msgr2=0x7fe6f0156300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:44:02.568 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:02.560+0000 7fe6f5a00640 1 --2- 172.21.15.123:0/2839343297 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fe6f00748d0 0x7fe6f0156300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:44:02.568 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:02.560+0000 7fe6f6201640 1 --2- 172.21.15.123:0/2839343297 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe6f015f420 0x7fe6f00cded0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:44:02.568 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:02.560+0000 7fe6f51ff640 1 --2- 172.21.15.123:0/2839343297 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fe6f00748d0 0x7fe6f0156300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:02.568 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:02.560+0000 7fe6f51ff640 1 -- 172.21.15.123:0/2839343297 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe6f015f420 msgr2=0x7fe6f00cded0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:44:02.568 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:02.560+0000 7fe6f51ff640 1 --2- 172.21.15.123:0/2839343297 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe6f015f420 0x7fe6f00cded0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:02.568 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:02.560+0000 7fe6f51ff640 1 -- 172.21.15.123:0/2839343297 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe6f0165330 con 0x7fe6f00b2910 2024-09-16T15:44:02.568 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:02.560+0000 7fe6f51ff640 1 --2- 172.21.15.123:0/2839343297 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe6f00b2910 0x7fe6f01fbb10 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7fe6e001d160 tx=0x7fe6e0005760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:44:02.568 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:02.560+0000 7fe6e67fc640 1 -- 172.21.15.123:0/2839343297 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6e00bc020 con 0x7fe6f00b2910 2024-09-16T15:44:02.568 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:02.560+0000 7fe6e67fc640 1 -- 172.21.15.123:0/2839343297 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fe6e00044b0 con 0x7fe6f00b2910 2024-09-16T15:44:02.568 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:02.560+0000 7fe6f7c64640 1 -- 172.21.15.123:0/2839343297 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe6f01fc050 con 0x7fe6f00b2910 2024-09-16T15:44:02.568 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:02.560+0000 7fe6e67fc640 1 -- 172.21.15.123:0/2839343297 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6e0005290 con 0x7fe6f00b2910 2024-09-16T15:44:02.568 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:02.560+0000 7fe6f7c64640 1 -- 172.21.15.123:0/2839343297 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe6f015b8f0 con 0x7fe6f00b2910 2024-09-16T15:44:02.569 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:02.564+0000 7fe6f7c64640 1 -- 172.21.15.123:0/2839343297 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe6f0068960 con 0x7fe6f00b2910 2024-09-16T15:44:02.569 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:02.564+0000 7fe6e67fc640 1 -- 172.21.15.123:0/2839343297 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 7) ==== 97233+0+0 (secure 0 0 0) 0x7fe6e00bf070 con 0x7fe6f00b2910 2024-09-16T15:44:02.569 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:02.564+0000 7fe6e67fc640 1 --2- 172.21.15.123:0/2839343297 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fe6d4073070 0x7fe6d4075530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:44:02.569 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:02.564+0000 7fe6e67fc640 1 -- 172.21.15.123:0/2839343297 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(33..33 src has 1..33) ==== 5709+0+0 (secure 0 0 0) 0x7fe6e012e6e0 con 0x7fe6f00b2910 2024-09-16T15:44:02.569 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:02.564+0000 7fe6f5a00640 1 --2- 172.21.15.123:0/2839343297 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fe6d4073070 0x7fe6d4075530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:44:02.570 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:02.564+0000 7fe6f5a00640 1 --2- 172.21.15.123:0/2839343297 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fe6d4073070 0x7fe6d4075530 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fe6f00c7160 tx=0x7fe6ec008040 comp rx=0 tx=0).ready entity=mgr.4135 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:44:02.573 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:02.568+0000 7fe6e67fc640 1 -- 172.21.15.123:0/2839343297 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7fe6f01fc050 con 0x7fe6f00b2910 2024-09-16T15:44:02.733 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:02.728+0000 7fe6f7c64640 1 -- 172.21.15.123:0/2839343297 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fe6f0074df0 con 0x7fe6f00b2910 2024-09-16T15:44:02.736 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:02.732+0000 7fe6e67fc640 1 -- 172.21.15.123:0/2839343297 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v33) ==== 74+0+16789 (secure 0 0 0) 0x7fe6e00164b0 con 0x7fe6f00b2910 2024-09-16T15:44:02.736 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-09-16T15:44:02.736 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":33,"fsid":"56077f89-18c8-486f-ae7e-eecc0bb7822c","created":"2024-09-16T15:42:26.364892+0000","modified":"2024-09-16T15:43:43.471260+0000","last_up_change":"2024-09-16T15:42:38.707234+0000","last_in_change":"2024-09-16T15:42:29.423662+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-16T15:42:39.287764+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2024-09-16T15:43:39.775631+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"32","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data","create_time":"2024-09-16T15:43:40.809234+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"33","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9e7c1955-a607-4671-9be7-f2ef760ceeaa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6800","nonce":3882044969},{"type":"v1","addr":"172.21.15.123:6801","nonce":3882044969}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6802","nonce":3882044969},{"type":"v1","addr":"172.21.15.123:6803","nonce":3882044969}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6806","nonce":3882044969},{"type":"v1","addr":"172.21.15.123:6807","nonce":3882044969}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6804","nonce":3882044969},{"type":"v1","addr":"172.21.15.123:6805","nonce":3882044969}]},"public_addr":"172.21.15.123:6801/3882044969","cluster_addr":"172.21.15.123:6803/3882044969","heartbeat_back_addr":"172.21.15.123:6807/3882044969","heartbeat_front_addr":"172.21.15.123:6805/3882044969","state":["exists","up"]},{"osd":1,"uuid":"c540cfde-0977-4742-9b17-fd634668e010","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6816","nonce":4037001850},{"type":"v1","addr":"172.21.15.123:6817","nonce":4037001850}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6818","nonce":4037001850},{"type":"v1","addr":"172.21.15.123:6819","nonce":4037001850}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6822","nonce":4037001850},{"type":"v1","addr":"172.21.15.123:6823","nonce":4037001850}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6820","nonce":4037001850},{"type":"v1","addr":"172.21.15.123:6821","nonce":4037001850}]},"public_addr":"172.21.15.123:6817/4037001850","cluster_addr":"172.21.15.123:6819/4037001850","heartbeat_back_addr":"172.21.15.123:6823/4037001850","heartbeat_front_addr":"172.21.15.123:6821/4037001850","state":["exists","up"]},{"osd":2,"uuid":"9864a496-f169-486b-95c4-1d59162793a7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6824","nonce":1332143088},{"type":"v1","addr":"172.21.15.123:6825","nonce":1332143088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6826","nonce":1332143088},{"type":"v1","addr":"172.21.15.123:6827","nonce":1332143088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6830","nonce":1332143088},{"type":"v1","addr":"172.21.15.123:6831","nonce":1332143088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6828","nonce":1332143088},{"type":"v1","addr":"172.21.15.123:6829","nonce":1332143088}]},"public_addr":"172.21.15.123:6825/1332143088","cluster_addr":"172.21.15.123:6827/1332143088","heartbeat_back_addr":"172.21.15.123:6831/1332143088","heartbeat_front_addr":"172.21.15.123:6829/1332143088","state":["exists","up"]},{"osd":3,"uuid":"c6d1f817-f074-4325-95e1-578eba13edda","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6808","nonce":988586998},{"type":"v1","addr":"172.21.15.123:6809","nonce":988586998}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6810","nonce":988586998},{"type":"v1","addr":"172.21.15.123:6811","nonce":988586998}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6814","nonce":988586998},{"type":"v1","addr":"172.21.15.123:6815","nonce":988586998}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6812","nonce":988586998},{"type":"v1","addr":"172.21.15.123:6813","nonce":988586998}]},"public_addr":"172.21.15.123:6809/988586998","cluster_addr":"172.21.15.123:6811/988586998","heartbeat_back_addr":"172.21.15.123:6815/988586998","heartbeat_front_addr":"172.21.15.123:6813/988586998","state":["exists","up"]},{"osd":4,"uuid":"200a8df4-50ef-40a5-b48a-58981269c086","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6816","nonce":3387813890},{"type":"v1","addr":"172.21.15.184:6817","nonce":3387813890}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6818","nonce":3387813890},{"type":"v1","addr":"172.21.15.184:6819","nonce":3387813890}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6822","nonce":3387813890},{"type":"v1","addr":"172.21.15.184:6823","nonce":3387813890}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6820","nonce":3387813890},{"type":"v1","addr":"172.21.15.184:6821","nonce":3387813890}]},"public_addr":"172.21.15.184:6817/3387813890","cluster_addr":"172.21.15.184:6819/3387813890","heartbeat_back_addr":"172.21.15.184:6823/3387813890","heartbeat_front_addr":"172.21.15.184:6821/3387813890","state":["exists","up"]},{"osd":5,"uuid":"32d5a900-da3d-412c-bc19-13861a2368d8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6808","nonce":2680916017},{"type":"v1","addr":"172.21.15.184:6809","nonce":2680916017}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6810","nonce":2680916017},{"type":"v1","addr":"172.21.15.184:6811","nonce":2680916017}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6814","nonce":2680916017},{"type":"v1","addr":"172.21.15.184:6815","nonce":2680916017}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6812","nonce":2680916017},{"type":"v1","addr":"172.21.15.184:6813","nonce":2680916017}]},"public_addr":"172.21.15.184:6809/2680916017","cluster_addr":"172.21.15.184:6811/2680916017","heartbeat_back_addr":"172.21.15.184:6815/2680916017","heartbeat_front_addr":"172.21.15.184:6813/2680916017","state":["exists","up"]},{"osd":6,"uuid":"cde6f39e-aa1b-4260-ac19-774ee6aaca72","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6824","nonce":2183120457},{"type":"v1","addr":"172.21.15.184:6825","nonce":2183120457}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6826","nonce":2183120457},{"type":"v1","addr":"172.21.15.184:6827","nonce":2183120457}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6830","nonce":2183120457},{"type":"v1","addr":"172.21.15.184:6831","nonce":2183120457}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6828","nonce":2183120457},{"type":"v1","addr":"172.21.15.184:6829","nonce":2183120457}]},"public_addr":"172.21.15.184:6825/2183120457","cluster_addr":"172.21.15.184:6827/2183120457","heartbeat_back_addr":"172.21.15.184:6831/2183120457","heartbeat_front_addr":"172.21.15.184:6829/2183120457","state":["exists","up"]},{"osd":7,"uuid":"4f381778-eee1-47c6-804f-df0e6b33e313","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6800","nonce":4102957285},{"type":"v1","addr":"172.21.15.184:6801","nonce":4102957285}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6802","nonce":4102957285},{"type":"v1","addr":"172.21.15.184:6803","nonce":4102957285}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6806","nonce":4102957285},{"type":"v1","addr":"172.21.15.184:6807","nonce":4102957285}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6804","nonce":4102957285},{"type":"v1","addr":"172.21.15.184:6805","nonce":4102957285}]},"public_addr":"172.21.15.184:6801/4102957285","cluster_addr":"172.21.15.184:6803/4102957285","heartbeat_back_addr":"172.21.15.184:6807/4102957285","heartbeat_front_addr":"172.21.15.184:6805/4102957285","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:35.972210+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.222007+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.198617+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.385663+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.358935+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.156066+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.752973+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:35.682846+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T15:44:02.738 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:02.732+0000 7fe6f7c64640 1 -- 172.21.15.123:0/2839343297 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fe6d4073070 msgr2=0x7fe6d4075530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:44:02.738 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:02.732+0000 7fe6f7c64640 1 --2- 172.21.15.123:0/2839343297 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fe6d4073070 0x7fe6d4075530 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fe6f00c7160 tx=0x7fe6ec008040 comp rx=0 tx=0).stop 2024-09-16T15:44:02.739 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:02.732+0000 7fe6f7c64640 1 -- 172.21.15.123:0/2839343297 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe6f00b2910 msgr2=0x7fe6f01fbb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:44:02.739 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:02.732+0000 7fe6f7c64640 1 --2- 172.21.15.123:0/2839343297 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe6f00b2910 0x7fe6f01fbb10 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7fe6e001d160 tx=0x7fe6e0005760 comp rx=0 tx=0).stop 2024-09-16T15:44:02.739 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:02.732+0000 7fe6f7c64640 1 -- 172.21.15.123:0/2839343297 shutdown_connections 2024-09-16T15:44:02.739 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:02.732+0000 7fe6f7c64640 1 --2- 172.21.15.123:0/2839343297 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fe6d4073070 0x7fe6d4075530 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:02.739 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:02.732+0000 7fe6f7c64640 1 --2- 172.21.15.123:0/2839343297 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe6f015f420 0x7fe6f00cded0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:02.739 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:02.732+0000 7fe6f7c64640 1 --2- 172.21.15.123:0/2839343297 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe6f00b2910 0x7fe6f01fbb10 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:02.739 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:02.732+0000 7fe6f7c64640 1 --2- 172.21.15.123:0/2839343297 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fe6f00748d0 0x7fe6f0156300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:02.739 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:02.732+0000 7fe6f7c64640 1 -- 172.21.15.123:0/2839343297 >> 172.21.15.123:0/2839343297 conn(0x7fe6f00b1df0 msgr2=0x7fe6f015da30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:44:02.739 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:02.732+0000 7fe6f7c64640 1 -- 172.21.15.123:0/2839343297 shutdown_connections 2024-09-16T15:44:02.739 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:02.732+0000 7fe6f7c64640 1 -- 172.21.15.123:0/2839343297 wait complete. 2024-09-16T15:44:02.754 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-09-16T15:44:02.872 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:02.868+0000 7f74b486e640 1 -- 172.21.15.123:0/3774948916 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f74b015bca0 msgr2=0x7f74b00d0630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:44:02.873 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:02.868+0000 7f74b486e640 1 --2- 172.21.15.123:0/3774948916 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f74b015bca0 0x7f74b00d0630 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f74b00cbd60 tx=0x7f74a001c820 comp rx=0 tx=0).stop 2024-09-16T15:44:02.873 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:02.868+0000 7f74b486e640 1 -- 172.21.15.123:0/3774948916 shutdown_connections 2024-09-16T15:44:02.873 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:02.868+0000 7f74b486e640 1 --2- 172.21.15.123:0/3774948916 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f74b015bca0 0x7f74b00d0630 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:02.873 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:02.868+0000 7f74b486e640 1 -- 172.21.15.123:0/3774948916 >> 172.21.15.123:0/3774948916 conn(0x7f74b00cfb10 msgr2=0x7f74b00cff20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:44:02.873 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:02.868+0000 7f74b486e640 1 -- 172.21.15.123:0/3774948916 shutdown_connections 2024-09-16T15:44:02.873 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:02.868+0000 7f74b486e640 1 -- 172.21.15.123:0/3774948916 wait complete. 2024-09-16T15:44:02.874 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:02.868+0000 7f74b486e640 1 Processor -- start 2024-09-16T15:44:02.874 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:02.868+0000 7f74b486e640 1 -- start start 2024-09-16T15:44:02.876 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:02.868+0000 7f74b486e640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f74b00726c0 0x7f74b00cf150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:44:02.876 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:02.868+0000 7f74b486e640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f74b0074600 0x7f74b01faf50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:44:02.876 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:02.868+0000 7f74b486e640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f74b015bca0 0x7f74b01fb490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:44:02.876 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:02.868+0000 7f74b486e640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f74b0169df0 con 0x7f74b00726c0 2024-09-16T15:44:02.876 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:02.868+0000 7f74b486e640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f74b0169c70 con 0x7f74b015bca0 2024-09-16T15:44:02.876 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:02.868+0000 7f74b486e640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f74b0169f70 con 0x7f74b0074600 2024-09-16T15:44:02.876 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:02.868+0000 7f74ae59c640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f74b00726c0 0x7f74b00cf150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:44:02.877 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:02.868+0000 7f74ae59c640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f74b00726c0 0x7f74b00cf150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:57844/0 (socket says 172.21.15.123:57844) 2024-09-16T15:44:02.877 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:02.868+0000 7f74ae59c640 1 -- 172.21.15.123:0/4126497498 learned_addr learned my addr 172.21.15.123:0/4126497498 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:44:02.877 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:02.868+0000 7f74add9b640 1 --2- 172.21.15.123:0/4126497498 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f74b0074600 0x7f74b01faf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:44:02.877 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:02.868+0000 7f74aed9d640 1 --2- 172.21.15.123:0/4126497498 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f74b015bca0 0x7f74b01fb490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:44:02.877 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:02.868+0000 7f74ae59c640 1 -- 172.21.15.123:0/4126497498 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f74b0074600 msgr2=0x7f74b01faf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:44:02.877 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:02.868+0000 7f74ae59c640 1 --2- 172.21.15.123:0/4126497498 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f74b0074600 0x7f74b01faf50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:02.877 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:02.868+0000 7f74ae59c640 1 -- 172.21.15.123:0/4126497498 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f74b015bca0 msgr2=0x7f74b01fb490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:44:02.877 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:02.868+0000 7f74ae59c640 1 --2- 172.21.15.123:0/4126497498 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f74b015bca0 0x7f74b01fb490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:02.877 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:02.872+0000 7f74ae59c640 1 -- 172.21.15.123:0/4126497498 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f74b00c5610 con 0x7f74b00726c0 2024-09-16T15:44:02.877 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:02.872+0000 7f74ae59c640 1 --2- 172.21.15.123:0/4126497498 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f74b00726c0 0x7f74b00cf150 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f74a0009850 tx=0x7f74a00a5700 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:44:02.878 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:02.872+0000 7f749effd640 1 -- 172.21.15.123:0/4126497498 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74a00b2040 con 0x7f74b00726c0 2024-09-16T15:44:02.878 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:02.872+0000 7f749effd640 1 -- 172.21.15.123:0/4126497498 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f74a0004230 con 0x7f74b00726c0 2024-09-16T15:44:02.878 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:02.872+0000 7f749effd640 1 -- 172.21.15.123:0/4126497498 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74a0003600 con 0x7f74b00726c0 2024-09-16T15:44:02.878 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:02.872+0000 7f74b486e640 1 -- 172.21.15.123:0/4126497498 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f74b00c58f0 con 0x7f74b00726c0 2024-09-16T15:44:02.878 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:02.872+0000 7f74b486e640 1 -- 172.21.15.123:0/4126497498 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f74b00c5eb0 con 0x7f74b00726c0 2024-09-16T15:44:02.878 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:02.872+0000 7f74b486e640 1 -- 172.21.15.123:0/4126497498 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f74b00d0630 con 0x7f74b00726c0 2024-09-16T15:44:02.879 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:02.872+0000 7f749effd640 1 -- 172.21.15.123:0/4126497498 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 7) ==== 97233+0+0 (secure 0 0 0) 0x7f74a00bf020 con 0x7f74b00726c0 2024-09-16T15:44:02.879 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:02.872+0000 7f749effd640 1 --2- 172.21.15.123:0/4126497498 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f7490073070 0x7f7490075530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:44:02.879 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:02.872+0000 7f749effd640 1 -- 172.21.15.123:0/4126497498 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(33..33 src has 1..33) ==== 5709+0+0 (secure 0 0 0) 0x7f74a012e710 con 0x7f74b00726c0 2024-09-16T15:44:02.879 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:02.872+0000 7f74add9b640 1 --2- 172.21.15.123:0/4126497498 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f7490073070 0x7f7490075530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:44:02.879 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:02.872+0000 7f74add9b640 1 --2- 172.21.15.123:0/4126497498 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f7490073070 0x7f7490075530 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f74a4007fd0 tx=0x7f74a4009040 comp rx=0 tx=0).ready entity=mgr.4135 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:44:02.881 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:02.876+0000 7f749effd640 1 -- 172.21.15.123:0/4126497498 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f74a00164d0 con 0x7f74b00726c0 2024-09-16T15:44:03.074 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:03.068+0000 7f74b486e640 1 -- 172.21.15.123:0/4126497498 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7f74b015f630 con 0x7f74b00726c0 2024-09-16T15:44:03.074 INFO:tasks.ceph.mon.a.smithi123.stderr:2024-09-16T15:44:03.068+0000 7f268d076640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-09-16T15:44:03.077 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:03.072+0000 7f749effd640 1 -- 172.21.15.123:0/4126497498 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v20) ==== 161+0+0 (secure 0 0 0) 0x7f74a00f95d0 con 0x7f74b00726c0 2024-09-16T15:44:03.077 INFO:teuthology.orchestra.run.smithi123.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-09-16T15:44:03.079 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:03.076+0000 7f74b486e640 1 -- 172.21.15.123:0/4126497498 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f7490073070 msgr2=0x7f7490075530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:44:03.080 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:03.076+0000 7f74b486e640 1 --2- 172.21.15.123:0/4126497498 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f7490073070 0x7f7490075530 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f74a4007fd0 tx=0x7f74a4009040 comp rx=0 tx=0).stop 2024-09-16T15:44:03.080 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:03.076+0000 7f74b486e640 1 -- 172.21.15.123:0/4126497498 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f74b00726c0 msgr2=0x7f74b00cf150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:44:03.080 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:03.076+0000 7f74b486e640 1 --2- 172.21.15.123:0/4126497498 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f74b00726c0 0x7f74b00cf150 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f74a0009850 tx=0x7f74a00a5700 comp rx=0 tx=0).stop 2024-09-16T15:44:03.080 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:03.076+0000 7f74b486e640 1 -- 172.21.15.123:0/4126497498 shutdown_connections 2024-09-16T15:44:03.080 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:03.076+0000 7f74b486e640 1 --2- 172.21.15.123:0/4126497498 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f7490073070 0x7f7490075530 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:03.080 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:03.076+0000 7f74b486e640 1 --2- 172.21.15.123:0/4126497498 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f74b015bca0 0x7f74b01fb490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:03.080 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:03.076+0000 7f74b486e640 1 --2- 172.21.15.123:0/4126497498 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f74b0074600 0x7f74b01faf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:03.080 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:03.076+0000 7f74b486e640 1 --2- 172.21.15.123:0/4126497498 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f74b00726c0 0x7f74b00cf150 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:03.080 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:03.076+0000 7f74b486e640 1 -- 172.21.15.123:0/4126497498 >> 172.21.15.123:0/4126497498 conn(0x7f74b00cfb10 msgr2=0x7f74b00b29a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:44:03.080 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:03.076+0000 7f74b486e640 1 -- 172.21.15.123:0/4126497498 shutdown_connections 2024-09-16T15:44:03.080 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:03.076+0000 7f74b486e640 1 -- 172.21.15.123:0/4126497498 wait complete. 2024-09-16T15:44:03.097 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-09-16T15:44:03.129 INFO:tasks.ceph.mds.a.smithi123.stdout:starting mds.a at 2024-09-16T15:44:03.224 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:03.220+0000 7f0deb4a3640 1 -- 172.21.15.123:0/1367219479 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0de415ddb0 msgr2=0x7f0de415e220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:44:03.224 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:03.220+0000 7f0deb4a3640 1 --2- 172.21.15.123:0/1367219479 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0de415ddb0 0x7f0de415e220 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f0de406db30 tx=0x7f0ddc01cd80 comp rx=0 tx=0).stop 2024-09-16T15:44:03.224 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:03.220+0000 7f0deb4a3640 1 -- 172.21.15.123:0/1367219479 shutdown_connections 2024-09-16T15:44:03.224 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:03.220+0000 7f0deb4a3640 1 --2- 172.21.15.123:0/1367219479 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0de415ddb0 0x7f0de415e220 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:03.224 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:03.220+0000 7f0deb4a3640 1 --2- 172.21.15.123:0/1367219479 >> v2:172.21.15.184:3300/0 conn(0x7f0de4159f70 0x7f0de4072f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:03.224 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:03.220+0000 7f0deb4a3640 1 -- 172.21.15.123:0/1367219479 >> 172.21.15.123:0/1367219479 conn(0x7f0de40b1710 msgr2=0x7f0de40b1b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:44:03.224 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:03.220+0000 7f0deb4a3640 1 -- 172.21.15.123:0/1367219479 shutdown_connections 2024-09-16T15:44:03.224 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:03.220+0000 7f0deb4a3640 1 -- 172.21.15.123:0/1367219479 wait complete. 2024-09-16T15:44:03.225 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:03.220+0000 7f0deb4a3640 1 Processor -- start 2024-09-16T15:44:03.225 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:03.220+0000 7f0deb4a3640 1 -- start start 2024-09-16T15:44:03.227 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:03.220+0000 7f0deb4a3640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f0de4159f70 0x7f0de40c9370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:44:03.227 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:03.220+0000 7f0deb4a3640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0de415a350 0x7f0de41498f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:44:03.227 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:03.220+0000 7f0deb4a3640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f0de415ddb0 0x7f0de41f90f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:44:03.227 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:03.220+0000 7f0deb4a3640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f0de4167300 con 0x7f0de415a350 2024-09-16T15:44:03.227 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:03.220+0000 7f0deb4a3640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f0de4167180 con 0x7f0de4159f70 2024-09-16T15:44:03.227 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:03.220+0000 7f0deb4a3640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f0de4167480 con 0x7f0de415ddb0 2024-09-16T15:44:03.227 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:03.220+0000 7f0de8a3e640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0de415a350 0x7f0de41498f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:44:03.227 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:03.220+0000 7f0de8a3e640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0de415a350 0x7f0de41498f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:57896/0 (socket says 172.21.15.123:57896) 2024-09-16T15:44:03.228 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:03.220+0000 7f0de8a3e640 1 -- 172.21.15.123:0/604605563 learned_addr learned my addr 172.21.15.123:0/604605563 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:44:03.228 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:03.220+0000 7f0de923f640 1 --2- 172.21.15.123:0/604605563 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f0de4159f70 0x7f0de40c9370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:44:03.228 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:03.220+0000 7f0de9a40640 1 --2- 172.21.15.123:0/604605563 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f0de415ddb0 0x7f0de41f90f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:44:03.228 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:03.220+0000 7f0de8a3e640 1 -- 172.21.15.123:0/604605563 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f0de415ddb0 msgr2=0x7f0de41f90f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:44:03.228 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:03.220+0000 7f0de8a3e640 1 --2- 172.21.15.123:0/604605563 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f0de415ddb0 0x7f0de41f90f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:03.228 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:03.220+0000 7f0de8a3e640 1 -- 172.21.15.123:0/604605563 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f0de4159f70 msgr2=0x7f0de40c9370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:44:03.228 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:03.220+0000 7f0de8a3e640 1 --2- 172.21.15.123:0/604605563 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f0de4159f70 0x7f0de40c9370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:03.228 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:03.220+0000 7f0de8a3e640 1 -- 172.21.15.123:0/604605563 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0de41f5c00 con 0x7f0de415a350 2024-09-16T15:44:03.228 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:03.220+0000 7f0de8a3e640 1 --2- 172.21.15.123:0/604605563 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0de415a350 0x7f0de41498f0 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f0dd400c480 tx=0x7f0dd400c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:44:03.228 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:03.220+0000 7f0dd9ffb640 1 -- 172.21.15.123:0/604605563 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0dd4012830 con 0x7f0de415a350 2024-09-16T15:44:03.228 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:03.220+0000 7f0dd9ffb640 1 -- 172.21.15.123:0/604605563 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f0dd4017070 con 0x7f0de415a350 2024-09-16T15:44:03.228 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:03.220+0000 7f0deb4a3640 1 -- 172.21.15.123:0/604605563 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0de41f5ee0 con 0x7f0de415a350 2024-09-16T15:44:03.229 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:03.220+0000 7f0dd9ffb640 1 -- 172.21.15.123:0/604605563 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0dd4012d80 con 0x7f0de415a350 2024-09-16T15:44:03.229 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:03.220+0000 7f0deb4a3640 1 -- 172.21.15.123:0/604605563 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0de4149e30 con 0x7f0de415a350 2024-09-16T15:44:03.229 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:03.224+0000 7f0deb4a3640 1 -- 172.21.15.123:0/604605563 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0de41f8980 con 0x7f0de415a350 2024-09-16T15:44:03.229 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:03.224+0000 7f0dd9ffb640 1 -- 172.21.15.123:0/604605563 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 7) ==== 97233+0+0 (secure 0 0 0) 0x7f0dd401c460 con 0x7f0de415a350 2024-09-16T15:44:03.230 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:03.224+0000 7f0dd9ffb640 1 --2- 172.21.15.123:0/604605563 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f0dc8073070 0x7f0dc8075530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:44:03.230 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:03.224+0000 7f0dd9ffb640 1 -- 172.21.15.123:0/604605563 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(34..34 src has 1..34) ==== 5795+0+0 (secure 0 0 0) 0x7f0dd4094100 con 0x7f0de415a350 2024-09-16T15:44:03.230 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:03.224+0000 7f0de923f640 1 --2- 172.21.15.123:0/604605563 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f0dc8073070 0x7f0dc8075530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:44:03.230 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:03.224+0000 7f0de923f640 1 --2- 172.21.15.123:0/604605563 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f0dc8073070 0x7f0dc8075530 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f0de00058d0 tx=0x7f0de0005e30 comp rx=0 tx=0).ready entity=mgr.4135 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:44:03.233 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:03.228+0000 7f0dd9ffb640 1 -- 172.21.15.123:0/604605563 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f0dd405ef50 con 0x7f0de415a350 2024-09-16T15:44:03.430 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:03.424+0000 7f0deb4a3640 1 -- 172.21.15.123:0/604605563 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) -- 0x7f0de4072ea0 con 0x7f0de415a350 2024-09-16T15:44:04.080 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:04.076+0000 7f0dd9ffb640 1 -- 172.21.15.123:0/604605563 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v21) ==== 104+0+0 (secure 0 0 0) 0x7f0dd405e850 con 0x7f0de415a350 2024-09-16T15:44:04.085 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:04.080+0000 7f0deb4a3640 1 -- 172.21.15.123:0/604605563 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f0dc8073070 msgr2=0x7f0dc8075530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:44:04.085 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:04.080+0000 7f0deb4a3640 1 --2- 172.21.15.123:0/604605563 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f0dc8073070 0x7f0dc8075530 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f0de00058d0 tx=0x7f0de0005e30 comp rx=0 tx=0).stop 2024-09-16T15:44:04.085 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:04.080+0000 7f0deb4a3640 1 -- 172.21.15.123:0/604605563 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0de415a350 msgr2=0x7f0de41498f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:44:04.085 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:04.080+0000 7f0deb4a3640 1 --2- 172.21.15.123:0/604605563 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0de415a350 0x7f0de41498f0 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f0dd400c480 tx=0x7f0dd400c940 comp rx=0 tx=0).stop 2024-09-16T15:44:04.085 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:04.080+0000 7f0deb4a3640 1 -- 172.21.15.123:0/604605563 shutdown_connections 2024-09-16T15:44:04.085 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:04.080+0000 7f0deb4a3640 1 --2- 172.21.15.123:0/604605563 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f0dc8073070 0x7f0dc8075530 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:04.085 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:04.080+0000 7f0deb4a3640 1 --2- 172.21.15.123:0/604605563 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f0de415ddb0 0x7f0de41f90f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:04.085 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:04.080+0000 7f0deb4a3640 1 --2- 172.21.15.123:0/604605563 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0de415a350 0x7f0de41498f0 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:04.085 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:04.080+0000 7f0deb4a3640 1 --2- 172.21.15.123:0/604605563 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f0de4159f70 0x7f0de40c9370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:04.085 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:04.080+0000 7f0deb4a3640 1 -- 172.21.15.123:0/604605563 >> 172.21.15.123:0/604605563 conn(0x7f0de40b1710 msgr2=0x7f0de40c5880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:44:04.085 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:04.080+0000 7f0deb4a3640 1 -- 172.21.15.123:0/604605563 shutdown_connections 2024-09-16T15:44:04.086 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:04.080+0000 7f0deb4a3640 1 -- 172.21.15.123:0/604605563 wait complete. 2024-09-16T15:44:04.107 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-09-16T15:44:04.286 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:04.280+0000 7f58de847640 1 -- 172.21.15.123:0/983534421 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f58d8073390 msgr2=0x7f58d806ff30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:44:04.286 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:04.280+0000 7f58de847640 1 --2- 172.21.15.123:0/983534421 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f58d8073390 0x7f58d806ff30 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f58d806a600 tx=0x7f58cc01c8c0 comp rx=0 tx=0).stop 2024-09-16T15:44:04.286 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:04.280+0000 7f58de847640 1 -- 172.21.15.123:0/983534421 shutdown_connections 2024-09-16T15:44:04.286 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:04.280+0000 7f58de847640 1 --2- 172.21.15.123:0/983534421 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f58d8070470 0x7f58d8157d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:04.287 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:04.280+0000 7f58de847640 1 --2- 172.21.15.123:0/983534421 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f58d8073390 0x7f58d806ff30 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:04.287 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:04.280+0000 7f58de847640 1 --2- 172.21.15.123:0/983534421 >> v2:172.21.15.184:3300/0 conn(0x7f58d80aee40 0x7f58d815b980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:04.287 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:04.280+0000 7f58de847640 1 -- 172.21.15.123:0/983534421 >> 172.21.15.123:0/983534421 conn(0x7f58d80b1630 msgr2=0x7f58d80b1a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:44:04.287 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:04.280+0000 7f58de847640 1 -- 172.21.15.123:0/983534421 shutdown_connections 2024-09-16T15:44:04.287 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:04.280+0000 7f58de847640 1 -- 172.21.15.123:0/983534421 wait complete. 2024-09-16T15:44:04.287 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:04.284+0000 7f58de847640 1 Processor -- start 2024-09-16T15:44:04.288 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:04.284+0000 7f58de847640 1 -- start start 2024-09-16T15:44:04.289 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:04.284+0000 7f58de847640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f58d8070470 0x7f58d8142350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:44:04.289 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:04.284+0000 7f58de847640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f58d8073390 0x7f58d80b4000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:44:04.289 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:04.284+0000 7f58de847640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f58d80aee40 0x7f58d80b4540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:44:04.289 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:04.284+0000 7f58de847640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f58d815ebe0 con 0x7f58d8073390 2024-09-16T15:44:04.290 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:04.284+0000 7f58de847640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f58d815ea60 con 0x7f58d8070470 2024-09-16T15:44:04.290 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:04.284+0000 7f58de847640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f58d815ed60 con 0x7f58d80aee40 2024-09-16T15:44:04.290 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:04.284+0000 7f58d77fe640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f58d8073390 0x7f58d80b4000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:44:04.290 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:04.284+0000 7f58d77fe640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f58d8073390 0x7f58d80b4000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:57914/0 (socket says 172.21.15.123:57914) 2024-09-16T15:44:04.290 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:04.284+0000 7f58d77fe640 1 -- 172.21.15.123:0/1199154461 learned_addr learned my addr 172.21.15.123:0/1199154461 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:44:04.290 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:04.284+0000 7f58dcde4640 1 --2- 172.21.15.123:0/1199154461 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f58d80aee40 0x7f58d80b4540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:44:04.290 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:04.284+0000 7f58d7fff640 1 --2- 172.21.15.123:0/1199154461 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f58d8070470 0x7f58d8142350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:44:04.290 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:04.284+0000 7f58d77fe640 1 -- 172.21.15.123:0/1199154461 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f58d80aee40 msgr2=0x7f58d80b4540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:44:04.290 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:04.284+0000 7f58d77fe640 1 --2- 172.21.15.123:0/1199154461 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f58d80aee40 0x7f58d80b4540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:04.290 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:04.284+0000 7f58d77fe640 1 -- 172.21.15.123:0/1199154461 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f58d8070470 msgr2=0x7f58d8142350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:44:04.290 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:04.284+0000 7f58d77fe640 1 --2- 172.21.15.123:0/1199154461 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f58d8070470 0x7f58d8142350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:04.290 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:04.284+0000 7f58d77fe640 1 -- 172.21.15.123:0/1199154461 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f58d80cdaf0 con 0x7f58d8073390 2024-09-16T15:44:04.291 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:04.284+0000 7f58d77fe640 1 --2- 172.21.15.123:0/1199154461 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f58d8073390 0x7f58d80b4000 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f58cc0b0040 tx=0x7f58cc004040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:44:04.291 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:04.284+0000 7f58d4ff9640 1 -- 172.21.15.123:0/1199154461 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58cc0bc070 con 0x7f58d8073390 2024-09-16T15:44:04.291 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:04.284+0000 7f58d4ff9640 1 -- 172.21.15.123:0/1199154461 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f58cc005020 con 0x7f58d8073390 2024-09-16T15:44:04.291 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:04.284+0000 7f58de847640 1 -- 172.21.15.123:0/1199154461 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f58d81fb310 con 0x7f58d8073390 2024-09-16T15:44:04.291 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:04.284+0000 7f58d4ff9640 1 -- 172.21.15.123:0/1199154461 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58cc0b2040 con 0x7f58d8073390 2024-09-16T15:44:04.291 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:04.284+0000 7f58de847640 1 -- 172.21.15.123:0/1199154461 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f58d81fb8d0 con 0x7f58d8073390 2024-09-16T15:44:04.292 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:04.284+0000 7f58de847640 1 -- 172.21.15.123:0/1199154461 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f58d81fbf90 con 0x7f58d8073390 2024-09-16T15:44:04.292 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:04.284+0000 7f58d4ff9640 1 -- 172.21.15.123:0/1199154461 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 7) ==== 97233+0+0 (secure 0 0 0) 0x7f58cc0046d0 con 0x7f58d8073390 2024-09-16T15:44:04.292 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:04.284+0000 7f58d4ff9640 1 --2- 172.21.15.123:0/1199154461 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f58c4073070 0x7f58c4075530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:44:04.292 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:04.288+0000 7f58d4ff9640 1 -- 172.21.15.123:0/1199154461 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(34..34 src has 1..34) ==== 5795+0+0 (secure 0 0 0) 0x7f58cc12e370 con 0x7f58d8073390 2024-09-16T15:44:04.292 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:04.288+0000 7f58d7fff640 1 --2- 172.21.15.123:0/1199154461 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f58c4073070 0x7f58c4075530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:44:04.292 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:04.288+0000 7f58d7fff640 1 --2- 172.21.15.123:0/1199154461 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f58c4073070 0x7f58c4075530 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f58d000b9a0 tx=0x7f58d0005c30 comp rx=0 tx=0).ready entity=mgr.4135 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:44:04.295 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:04.292+0000 7f58d4ff9640 1 -- 172.21.15.123:0/1199154461 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f58d81fbf90 con 0x7f58d8073390 2024-09-16T15:44:04.458 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:04.452+0000 7f58de847640 1 -- 172.21.15.123:0/1199154461 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) -- 0x7f58d815f240 con 0x7f58d8073390 2024-09-16T15:44:05.411 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:05.408+0000 7f58d4ff9640 1 -- 172.21.15.123:0/1199154461 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v35) ==== 181+0+0 (secure 0 0 0) 0x7f58cc0164b0 con 0x7f58d8073390 2024-09-16T15:44:05.411 INFO:teuthology.orchestra.run.smithi123.stderr:pool 'cephfs_metadata' removed 2024-09-16T15:44:05.414 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:05.408+0000 7f58de847640 1 -- 172.21.15.123:0/1199154461 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f58c4073070 msgr2=0x7f58c4075530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:44:05.414 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:05.408+0000 7f58de847640 1 --2- 172.21.15.123:0/1199154461 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f58c4073070 0x7f58c4075530 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f58d000b9a0 tx=0x7f58d0005c30 comp rx=0 tx=0).stop 2024-09-16T15:44:05.414 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:05.408+0000 7f58de847640 1 -- 172.21.15.123:0/1199154461 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f58d8073390 msgr2=0x7f58d80b4000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:44:05.414 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:05.408+0000 7f58de847640 1 --2- 172.21.15.123:0/1199154461 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f58d8073390 0x7f58d80b4000 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f58cc0b0040 tx=0x7f58cc004040 comp rx=0 tx=0).stop 2024-09-16T15:44:05.414 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:05.408+0000 7f58de847640 1 -- 172.21.15.123:0/1199154461 shutdown_connections 2024-09-16T15:44:05.414 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:05.408+0000 7f58de847640 1 --2- 172.21.15.123:0/1199154461 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f58c4073070 0x7f58c4075530 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:05.415 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:05.408+0000 7f58de847640 1 --2- 172.21.15.123:0/1199154461 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f58d80aee40 0x7f58d80b4540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:05.415 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:05.408+0000 7f58de847640 1 --2- 172.21.15.123:0/1199154461 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f58d8073390 0x7f58d80b4000 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:05.415 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:05.408+0000 7f58de847640 1 --2- 172.21.15.123:0/1199154461 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f58d8070470 0x7f58d8142350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:05.415 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:05.408+0000 7f58de847640 1 -- 172.21.15.123:0/1199154461 >> 172.21.15.123:0/1199154461 conn(0x7f58d80b1630 msgr2=0x7f58d80ad8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:44:05.415 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:05.408+0000 7f58de847640 1 -- 172.21.15.123:0/1199154461 shutdown_connections 2024-09-16T15:44:05.415 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:05.408+0000 7f58de847640 1 -- 172.21.15.123:0/1199154461 wait complete. 2024-09-16T15:44:05.440 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-09-16T15:44:05.565 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:05.560+0000 7fcfbc927640 1 -- 172.21.15.123:0/301787521 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcfb815a1b0 msgr2=0x7fcfb8072fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:44:05.566 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:05.560+0000 7fcfbc927640 1 --2- 172.21.15.123:0/301787521 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcfb815a1b0 0x7fcfb8072fb0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7fcfac009960 tx=0x7fcfac01eaf0 comp rx=0 tx=0).stop 2024-09-16T15:44:05.566 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:05.560+0000 7fcfbc927640 1 -- 172.21.15.123:0/301787521 shutdown_connections 2024-09-16T15:44:05.566 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:05.560+0000 7fcfbc927640 1 --2- 172.21.15.123:0/301787521 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fcfb80706d0 0x7fcfb815e090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:05.566 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:05.560+0000 7fcfbc927640 1 --2- 172.21.15.123:0/301787521 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcfb815a1b0 0x7fcfb8072fb0 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:05.566 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:05.560+0000 7fcfbc927640 1 -- 172.21.15.123:0/301787521 >> 172.21.15.123:0/301787521 conn(0x7fcfb80b1630 msgr2=0x7fcfb80b1a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:44:05.566 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:05.560+0000 7fcfbc927640 1 -- 172.21.15.123:0/301787521 shutdown_connections 2024-09-16T15:44:05.566 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:05.560+0000 7fcfbc927640 1 -- 172.21.15.123:0/301787521 wait complete. 2024-09-16T15:44:05.567 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:05.560+0000 7fcfbc927640 1 Processor -- start 2024-09-16T15:44:05.567 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:05.560+0000 7fcfbc927640 1 -- start start 2024-09-16T15:44:05.569 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:05.564+0000 7fcfbc927640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fcfb80706d0 0x7fcfb8161b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:44:05.569 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:05.564+0000 7fcfbc927640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fcfb815a1b0 0x7fcfb8156390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:44:05.569 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:05.564+0000 7fcfbc927640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcfb815dc40 0x7fcfb806f630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:44:05.569 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:05.564+0000 7fcfbc927640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fcfb815b230 con 0x7fcfb815dc40 2024-09-16T15:44:05.569 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:05.564+0000 7fcfbc927640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fcfb815b0b0 con 0x7fcfb80706d0 2024-09-16T15:44:05.569 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:05.564+0000 7fcfbc927640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fcfb815b3b0 con 0x7fcfb815a1b0 2024-09-16T15:44:05.570 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:05.564+0000 7fcfb6d9d640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcfb815dc40 0x7fcfb806f630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:44:05.570 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:05.564+0000 7fcfb6d9d640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcfb815dc40 0x7fcfb806f630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:57932/0 (socket says 172.21.15.123:57932) 2024-09-16T15:44:05.570 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:05.564+0000 7fcfb6d9d640 1 -- 172.21.15.123:0/3137840498 learned_addr learned my addr 172.21.15.123:0/3137840498 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:44:05.570 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:05.564+0000 7fcfb5d9b640 1 --2- 172.21.15.123:0/3137840498 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fcfb815a1b0 0x7fcfb8156390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:44:05.570 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:05.564+0000 7fcfb659c640 1 --2- 172.21.15.123:0/3137840498 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fcfb80706d0 0x7fcfb8161b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:44:05.570 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:05.564+0000 7fcfb6d9d640 1 -- 172.21.15.123:0/3137840498 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fcfb815a1b0 msgr2=0x7fcfb8156390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:44:05.570 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:05.564+0000 7fcfb6d9d640 1 --2- 172.21.15.123:0/3137840498 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fcfb815a1b0 0x7fcfb8156390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:05.570 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:05.564+0000 7fcfb6d9d640 1 -- 172.21.15.123:0/3137840498 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fcfb80706d0 msgr2=0x7fcfb8161b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:44:05.570 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:05.564+0000 7fcfb6d9d640 1 --2- 172.21.15.123:0/3137840498 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fcfb80706d0 0x7fcfb8161b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:05.570 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:05.564+0000 7fcfb6d9d640 1 -- 172.21.15.123:0/3137840498 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcfb8162390 con 0x7fcfb815dc40 2024-09-16T15:44:05.570 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:05.564+0000 7fcfb659c640 1 --2- 172.21.15.123:0/3137840498 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fcfb80706d0 0x7fcfb8161b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T15:44:05.570 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:05.564+0000 7fcfb6d9d640 1 --2- 172.21.15.123:0/3137840498 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcfb815dc40 0x7fcfb806f630 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7fcfb000ddc0 tx=0x7fcfb000b730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:44:05.570 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:05.564+0000 7fcfa6ffd640 1 -- 172.21.15.123:0/3137840498 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfb0004d30 con 0x7fcfb815dc40 2024-09-16T15:44:05.571 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:05.564+0000 7fcfa6ffd640 1 -- 172.21.15.123:0/3137840498 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fcfb0005020 con 0x7fcfb815dc40 2024-09-16T15:44:05.571 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:05.564+0000 7fcfbc927640 1 -- 172.21.15.123:0/3137840498 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcfb81f7b30 con 0x7fcfb815dc40 2024-09-16T15:44:05.571 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:05.564+0000 7fcfa6ffd640 1 -- 172.21.15.123:0/3137840498 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfb0005430 con 0x7fcfb815dc40 2024-09-16T15:44:05.571 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:05.564+0000 7fcfbc927640 1 -- 172.21.15.123:0/3137840498 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcfb81f80f0 con 0x7fcfb815dc40 2024-09-16T15:44:05.571 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:05.564+0000 7fcfbc927640 1 -- 172.21.15.123:0/3137840498 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcfb8072fb0 con 0x7fcfb815dc40 2024-09-16T15:44:05.572 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:05.564+0000 7fcfa6ffd640 1 -- 172.21.15.123:0/3137840498 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 7) ==== 97233+0+0 (secure 0 0 0) 0x7fcfb00057e0 con 0x7fcfb815dc40 2024-09-16T15:44:05.572 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:05.564+0000 7fcfa6ffd640 1 --2- 172.21.15.123:0/3137840498 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fcf98073070 0x7fcf98075530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:44:05.572 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:05.564+0000 7fcfa6ffd640 1 -- 172.21.15.123:0/3137840498 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 5357+0+0 (secure 0 0 0) 0x7fcfb0094c10 con 0x7fcfb815dc40 2024-09-16T15:44:05.572 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:05.568+0000 7fcfb659c640 1 --2- 172.21.15.123:0/3137840498 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fcf98073070 0x7fcf98075530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:44:05.572 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:05.568+0000 7fcfb659c640 1 --2- 172.21.15.123:0/3137840498 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fcf98073070 0x7fcf98075530 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fcfac0b1000 tx=0x7fcfac01d330 comp rx=0 tx=0).ready entity=mgr.4135 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:44:05.575 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:05.568+0000 7fcfa6ffd640 1 -- 172.21.15.123:0/3137840498 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7fcfb005fc30 con 0x7fcfb815dc40 2024-09-16T15:44:05.762 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:05.756+0000 7fcfbc927640 1 -- 172.21.15.123:0/3137840498 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) -- 0x7fcfb81f82c0 con 0x7fcfb815dc40 2024-09-16T15:44:06.424 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:06.420+0000 7fcfa6ffd640 1 -- 172.21.15.123:0/3137840498 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v36) ==== 169+0+0 (secure 0 0 0) 0x7fcfb005f530 con 0x7fcfb815dc40 2024-09-16T15:44:06.424 INFO:teuthology.orchestra.run.smithi123.stderr:pool 'cephfs_data' removed 2024-09-16T15:44:06.431 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:06.424+0000 7fcfbc927640 1 -- 172.21.15.123:0/3137840498 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fcf98073070 msgr2=0x7fcf98075530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:44:06.431 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:06.424+0000 7fcfbc927640 1 --2- 172.21.15.123:0/3137840498 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fcf98073070 0x7fcf98075530 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fcfac0b1000 tx=0x7fcfac01d330 comp rx=0 tx=0).stop 2024-09-16T15:44:06.431 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:06.424+0000 7fcfbc927640 1 -- 172.21.15.123:0/3137840498 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcfb815dc40 msgr2=0x7fcfb806f630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:44:06.431 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:06.424+0000 7fcfbc927640 1 --2- 172.21.15.123:0/3137840498 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcfb815dc40 0x7fcfb806f630 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7fcfb000ddc0 tx=0x7fcfb000b730 comp rx=0 tx=0).stop 2024-09-16T15:44:06.431 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:06.424+0000 7fcfbc927640 1 -- 172.21.15.123:0/3137840498 shutdown_connections 2024-09-16T15:44:06.431 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:06.424+0000 7fcfbc927640 1 --2- 172.21.15.123:0/3137840498 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fcf98073070 0x7fcf98075530 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:06.431 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:06.424+0000 7fcfbc927640 1 --2- 172.21.15.123:0/3137840498 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcfb815dc40 0x7fcfb806f630 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:06.432 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:06.424+0000 7fcfbc927640 1 --2- 172.21.15.123:0/3137840498 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fcfb815a1b0 0x7fcfb8156390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:06.432 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:06.424+0000 7fcfbc927640 1 --2- 172.21.15.123:0/3137840498 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fcfb80706d0 0x7fcfb8161b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:06.432 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:06.424+0000 7fcfbc927640 1 -- 172.21.15.123:0/3137840498 >> 172.21.15.123:0/3137840498 conn(0x7fcfb80b1630 msgr2=0x7fcfb8157700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:44:06.432 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:06.428+0000 7fcfbc927640 1 -- 172.21.15.123:0/3137840498 shutdown_connections 2024-09-16T15:44:06.432 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:06.428+0000 7fcfbc927640 1 -- 172.21.15.123:0/3137840498 wait complete. 2024-09-16T15:44:06.453 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config reset 27 2024-09-16T15:44:06.581 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:06.576+0000 7f9456546640 1 -- 172.21.15.123:0/1451912064 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f94500cdc20 msgr2=0x7f94500ca860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:44:06.582 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:06.576+0000 7f9456546640 1 --2- 172.21.15.123:0/1451912064 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f94500cdc20 0x7f94500ca860 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f9450069550 tx=0x7f944801c9d0 comp rx=0 tx=0).stop 2024-09-16T15:44:06.582 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:06.576+0000 7f9456546640 1 -- 172.21.15.123:0/1451912064 shutdown_connections 2024-09-16T15:44:06.582 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:06.576+0000 7f9456546640 1 --2- 172.21.15.123:0/1451912064 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f94500cdc20 0x7f94500ca860 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:06.582 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:06.576+0000 7f9456546640 1 --2- 172.21.15.123:0/1451912064 >> v2:172.21.15.184:3300/0 conn(0x7f9450074470 0x7f94500cd6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:06.582 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:06.576+0000 7f9456546640 1 --2- 172.21.15.123:0/1451912064 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f945015e160 0x7f9450073f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:06.582 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:06.576+0000 7f9456546640 1 -- 172.21.15.123:0/1451912064 >> 172.21.15.123:0/1451912064 conn(0x7f94500cfb10 msgr2=0x7f94500cff20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:44:06.582 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:06.576+0000 7f9456546640 1 -- 172.21.15.123:0/1451912064 shutdown_connections 2024-09-16T15:44:06.582 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:06.576+0000 7f9456546640 1 -- 172.21.15.123:0/1451912064 wait complete. 2024-09-16T15:44:06.583 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:06.576+0000 7f9456546640 1 Processor -- start 2024-09-16T15:44:06.583 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:06.576+0000 7f9456546640 1 -- start start 2024-09-16T15:44:06.585 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:06.580+0000 7f9456546640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f9450074470 0x7f9450151b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:44:06.585 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:06.580+0000 7f9456546640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f94500cdc20 0x7f94500c8d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:44:06.585 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:06.580+0000 7f9456546640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f945015e160 0x7f94500c9290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:44:06.585 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:06.580+0000 7f9456546640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f945015ecf0 con 0x7f94500cdc20 2024-09-16T15:44:06.585 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:06.580+0000 7f9456546640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f945015eb70 con 0x7f945015e160 2024-09-16T15:44:06.585 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:06.580+0000 7f9456546640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f945015ee70 con 0x7f9450074470 2024-09-16T15:44:06.585 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:06.580+0000 7f944f7fe640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f94500cdc20 0x7f94500c8d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:44:06.585 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:06.580+0000 7f944f7fe640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f94500cdc20 0x7f94500c8d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:57938/0 (socket says 172.21.15.123:57938) 2024-09-16T15:44:06.585 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:06.580+0000 7f944f7fe640 1 -- 172.21.15.123:0/3020145976 learned_addr learned my addr 172.21.15.123:0/3020145976 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:44:06.585 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:06.580+0000 7f9454ae3640 1 --2- 172.21.15.123:0/3020145976 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f945015e160 0x7f94500c9290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:44:06.585 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:06.580+0000 7f944ffff640 1 --2- 172.21.15.123:0/3020145976 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f9450074470 0x7f9450151b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:44:06.585 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:06.580+0000 7f944f7fe640 1 -- 172.21.15.123:0/3020145976 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f9450074470 msgr2=0x7f9450151b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:44:06.586 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:06.580+0000 7f944f7fe640 1 --2- 172.21.15.123:0/3020145976 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f9450074470 0x7f9450151b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:06.586 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:06.580+0000 7f944f7fe640 1 -- 172.21.15.123:0/3020145976 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f945015e160 msgr2=0x7f94500c9290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:44:06.586 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:06.580+0000 7f944f7fe640 1 --2- 172.21.15.123:0/3020145976 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f945015e160 0x7f94500c9290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:06.586 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:06.580+0000 7f944f7fe640 1 -- 172.21.15.123:0/3020145976 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9450154830 con 0x7f94500cdc20 2024-09-16T15:44:06.586 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:06.580+0000 7f9454ae3640 1 --2- 172.21.15.123:0/3020145976 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f945015e160 0x7f94500c9290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T15:44:06.586 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:06.580+0000 7f944f7fe640 1 --2- 172.21.15.123:0/3020145976 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f94500cdc20 0x7f94500c8d50 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f944400e9f0 tx=0x7f944400eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:44:06.586 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:06.580+0000 7f944cff9640 1 -- 172.21.15.123:0/3020145976 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f944400ccc0 con 0x7f94500cdc20 2024-09-16T15:44:06.586 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:06.580+0000 7f944cff9640 1 -- 172.21.15.123:0/3020145976 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f944400ce80 con 0x7f94500cdc20 2024-09-16T15:44:06.586 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:06.580+0000 7f9456546640 1 -- 172.21.15.123:0/3020145976 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f94500b4670 con 0x7f94500cdc20 2024-09-16T15:44:06.586 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:06.580+0000 7f944cff9640 1 -- 172.21.15.123:0/3020145976 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9444004280 con 0x7f94500cdc20 2024-09-16T15:44:06.586 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:06.580+0000 7f9456546640 1 -- 172.21.15.123:0/3020145976 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f94500b4c30 con 0x7f94500cdc20 2024-09-16T15:44:06.587 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:06.580+0000 7f9456546640 1 -- 172.21.15.123:0/3020145976 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f94500ca860 con 0x7f94500cdc20 2024-09-16T15:44:06.587 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:06.580+0000 7f944cff9640 1 -- 172.21.15.123:0/3020145976 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 7) ==== 97233+0+0 (secure 0 0 0) 0x7f9444003600 con 0x7f94500cdc20 2024-09-16T15:44:06.587 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:06.580+0000 7f944cff9640 1 --2- 172.21.15.123:0/3020145976 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f943c073070 0x7f943c075530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:44:06.587 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:06.580+0000 7f944cff9640 1 -- 172.21.15.123:0/3020145976 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(36..36 src has 1..36) ==== 4943+0+0 (secure 0 0 0) 0x7f9444094920 con 0x7f94500cdc20 2024-09-16T15:44:06.587 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:06.584+0000 7f944ffff640 1 --2- 172.21.15.123:0/3020145976 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f943c073070 0x7f943c075530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:44:06.588 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:06.584+0000 7f944ffff640 1 --2- 172.21.15.123:0/3020145976 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f943c073070 0x7f943c075530 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f9450156740 tx=0x7f94400091e0 comp rx=0 tx=0).ready entity=mgr.4135 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:44:06.591 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:06.588+0000 7f944cff9640 1 -- 172.21.15.123:0/3020145976 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f944405fb70 con 0x7f94500cdc20 2024-09-16T15:44:06.764 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:06.760+0000 7f9456546640 1 -- 172.21.15.123:0/3020145976 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "config reset", "num": 27} v 0) -- 0x7f9450157730 con 0x7f94500cdc20 2024-09-16T15:44:06.769 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:06.764+0000 7f944cff9640 1 -- 172.21.15.123:0/3020145976 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "config reset", "num": 27}]=0 v29) ==== 71+0+0 (secure 0 0 0) 0x7f944405f470 con 0x7f94500cdc20 2024-09-16T15:44:06.772 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:06.768+0000 7f9456546640 1 -- 172.21.15.123:0/3020145976 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f943c073070 msgr2=0x7f943c075530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:44:06.772 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:06.768+0000 7f9456546640 1 --2- 172.21.15.123:0/3020145976 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f943c073070 0x7f943c075530 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f9450156740 tx=0x7f94400091e0 comp rx=0 tx=0).stop 2024-09-16T15:44:06.772 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:06.768+0000 7f9456546640 1 -- 172.21.15.123:0/3020145976 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f94500cdc20 msgr2=0x7f94500c8d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:44:06.772 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:06.768+0000 7f9456546640 1 --2- 172.21.15.123:0/3020145976 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f94500cdc20 0x7f94500c8d50 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f944400e9f0 tx=0x7f944400eeb0 comp rx=0 tx=0).stop 2024-09-16T15:44:06.773 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:06.768+0000 7f9456546640 1 -- 172.21.15.123:0/3020145976 shutdown_connections 2024-09-16T15:44:06.773 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:06.768+0000 7f9456546640 1 --2- 172.21.15.123:0/3020145976 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f943c073070 0x7f943c075530 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:06.773 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:06.768+0000 7f9456546640 1 --2- 172.21.15.123:0/3020145976 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f945015e160 0x7f94500c9290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:06.773 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:06.768+0000 7f9456546640 1 --2- 172.21.15.123:0/3020145976 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f94500cdc20 0x7f94500c8d50 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:06.773 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:06.768+0000 7f9456546640 1 --2- 172.21.15.123:0/3020145976 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f9450074470 0x7f9450151b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:06.773 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:06.768+0000 7f9456546640 1 -- 172.21.15.123:0/3020145976 >> 172.21.15.123:0/3020145976 conn(0x7f94500cfb10 msgr2=0x7f9450072f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:44:06.773 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:06.768+0000 7f9456546640 1 -- 172.21.15.123:0/3020145976 shutdown_connections 2024-09-16T15:44:06.773 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:06.768+0000 7f9456546640 1 -- 172.21.15.123:0/3020145976 wait complete. 2024-09-16T15:44:06.789 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_volumes.TestMisc.test_binary_metafile_on_legacy_to_v1_upgrade' 2024-09-16T15:44:06.919 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:06.912+0000 7f56d0e35640 1 -- 172.21.15.123:0/3485895622 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f56cc161620 msgr2=0x7f56cc0d1da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:44:06.919 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:06.912+0000 7f56d0e35640 1 --2- 172.21.15.123:0/3485895622 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f56cc161620 0x7f56cc0d1da0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f56bc0098f0 tx=0x7f56bc01c790 comp rx=0 tx=0).stop 2024-09-16T15:44:06.919 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:06.912+0000 7f56d0e35640 1 -- 172.21.15.123:0/3485895622 shutdown_connections 2024-09-16T15:44:06.919 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:06.912+0000 7f56d0e35640 1 --2- 172.21.15.123:0/3485895622 >> v2:172.21.15.184:3300/0 conn(0x7f56cc0b0a50 0x7f56cc0b0f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:06.920 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:06.912+0000 7f56d0e35640 1 --2- 172.21.15.123:0/3485895622 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f56cc161620 0x7f56cc0d1da0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:06.920 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:06.912+0000 7f56d0e35640 1 -- 172.21.15.123:0/3485895622 >> 172.21.15.123:0/3485895622 conn(0x7f56cc0d3ea0 msgr2=0x7f56cc0d42b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:44:06.920 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:06.916+0000 7f56d0e35640 1 -- 172.21.15.123:0/3485895622 shutdown_connections 2024-09-16T15:44:06.920 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:06.916+0000 7f56d0e35640 1 -- 172.21.15.123:0/3485895622 wait complete. 2024-09-16T15:44:06.920 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:06.916+0000 7f56d0e35640 1 Processor -- start 2024-09-16T15:44:06.921 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:06.916+0000 7f56d0e35640 1 -- start start 2024-09-16T15:44:06.923 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:06.916+0000 7f56d0e35640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f56cc0b0a50 0x7f56cc14ce70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:44:06.923 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:06.916+0000 7f56d0e35640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f56cc0b2b00 0x7f56cc153e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:44:06.923 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:06.916+0000 7f56d0e35640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f56cc161620 0x7f56cc0c9140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:44:06.923 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:06.916+0000 7f56d0e35640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f56cc16b0f0 con 0x7f56cc161620 2024-09-16T15:44:06.923 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:06.916+0000 7f56d0e35640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f56cc16af70 con 0x7f56cc0b0a50 2024-09-16T15:44:06.923 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:06.916+0000 7f56d0e35640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f56cc16b270 con 0x7f56cc0b2b00 2024-09-16T15:44:06.923 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:06.916+0000 7f56cad9d640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f56cc161620 0x7f56cc0c9140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:44:06.923 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:06.916+0000 7f56cad9d640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f56cc161620 0x7f56cc0c9140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:57940/0 (socket says 172.21.15.123:57940) 2024-09-16T15:44:06.923 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:06.916+0000 7f56cad9d640 1 -- 172.21.15.123:0/2670421423 learned_addr learned my addr 172.21.15.123:0/2670421423 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:44:06.923 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:06.916+0000 7f56ca59c640 1 --2- 172.21.15.123:0/2670421423 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f56cc0b0a50 0x7f56cc14ce70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:44:06.923 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:06.916+0000 7f56c9d9b640 1 --2- 172.21.15.123:0/2670421423 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f56cc0b2b00 0x7f56cc153e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:44:06.924 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:06.916+0000 7f56cad9d640 1 -- 172.21.15.123:0/2670421423 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f56cc0b2b00 msgr2=0x7f56cc153e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:44:06.924 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:06.916+0000 7f56cad9d640 1 --2- 172.21.15.123:0/2670421423 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f56cc0b2b00 0x7f56cc153e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:06.924 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:06.916+0000 7f56cad9d640 1 -- 172.21.15.123:0/2670421423 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f56cc0b0a50 msgr2=0x7f56cc14ce70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:44:06.924 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:06.916+0000 7f56cad9d640 1 --2- 172.21.15.123:0/2670421423 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f56cc0b0a50 0x7f56cc14ce70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:06.924 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:06.916+0000 7f56cad9d640 1 -- 172.21.15.123:0/2670421423 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f56cc153a20 con 0x7f56cc161620 2024-09-16T15:44:06.924 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:06.916+0000 7f56ca59c640 1 --2- 172.21.15.123:0/2670421423 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f56cc0b0a50 0x7f56cc14ce70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T15:44:06.924 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:06.916+0000 7f56cad9d640 1 --2- 172.21.15.123:0/2670421423 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f56cc161620 0x7f56cc0c9140 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f56c400eb20 tx=0x7f56c400c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:44:06.924 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:06.916+0000 7f56baffd640 1 -- 172.21.15.123:0/2670421423 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56c4004b30 con 0x7f56cc161620 2024-09-16T15:44:06.924 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:06.916+0000 7f56baffd640 1 -- 172.21.15.123:0/2670421423 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f56c4004cf0 con 0x7f56cc161620 2024-09-16T15:44:06.924 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:06.916+0000 7f56d0e35640 1 -- 172.21.15.123:0/2670421423 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f56cc1fb040 con 0x7f56cc161620 2024-09-16T15:44:06.924 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:06.916+0000 7f56baffd640 1 -- 172.21.15.123:0/2670421423 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56c4005280 con 0x7f56cc161620 2024-09-16T15:44:06.924 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:06.916+0000 7f56d0e35640 1 -- 172.21.15.123:0/2670421423 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f56cc1fb600 con 0x7f56cc161620 2024-09-16T15:44:06.925 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:06.920+0000 7f56d0e35640 1 -- 172.21.15.123:0/2670421423 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f56cc0b0f00 con 0x7f56cc161620 2024-09-16T15:44:06.925 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:06.920+0000 7f56baffd640 1 -- 172.21.15.123:0/2670421423 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 7) ==== 97233+0+0 (secure 0 0 0) 0x7f56c4005420 con 0x7f56cc161620 2024-09-16T15:44:06.925 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:06.920+0000 7f56baffd640 1 --2- 172.21.15.123:0/2670421423 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f56ac073070 0x7f56ac075530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:44:06.925 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:06.920+0000 7f56baffd640 1 -- 172.21.15.123:0/2670421423 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(36..36 src has 1..36) ==== 4943+0+0 (secure 0 0 0) 0x7f56c4094cb0 con 0x7f56cc161620 2024-09-16T15:44:06.925 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:06.920+0000 7f56ca59c640 1 --2- 172.21.15.123:0/2670421423 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f56ac073070 0x7f56ac075530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:44:06.926 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:06.920+0000 7f56ca59c640 1 --2- 172.21.15.123:0/2670421423 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f56ac073070 0x7f56ac075530 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f56bc0af000 tx=0x7f56bc01afd0 comp rx=0 tx=0).ready entity=mgr.4135 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:44:06.929 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:06.924+0000 7f56baffd640 1 -- 172.21.15.123:0/2670421423 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f56c405fe70 con 0x7f56cc161620 2024-09-16T15:44:07.180 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:07.176+0000 7f56d0e35640 1 -- 172.21.15.123:0/2670421423 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_volumes.TestMisc.test_binary_metafile_on_legacy_to_v1_upgrade"]} v 0) -- 0x7f56cc0b3d80 con 0x7f56cc161620 2024-09-16T15:44:07.426 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:07.420+0000 7f56baffd640 1 -- 172.21.15.123:0/2670421423 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_volumes.TestMisc.test_binary_metafile_on_legacy_to_v1_upgrade"]}]=0 v89) ==== 158+0+0 (secure 0 0 0) 0x7f56c405f770 con 0x7f56cc161620 2024-09-16T15:44:07.429 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:07.424+0000 7f56d0e35640 1 -- 172.21.15.123:0/2670421423 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f56ac073070 msgr2=0x7f56ac075530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:44:07.429 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:07.424+0000 7f56d0e35640 1 --2- 172.21.15.123:0/2670421423 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f56ac073070 0x7f56ac075530 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f56bc0af000 tx=0x7f56bc01afd0 comp rx=0 tx=0).stop 2024-09-16T15:44:07.430 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:07.424+0000 7f56d0e35640 1 -- 172.21.15.123:0/2670421423 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f56cc161620 msgr2=0x7f56cc0c9140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:44:07.430 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:07.424+0000 7f56d0e35640 1 --2- 172.21.15.123:0/2670421423 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f56cc161620 0x7f56cc0c9140 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f56c400eb20 tx=0x7f56c400c2d0 comp rx=0 tx=0).stop 2024-09-16T15:44:07.430 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:07.424+0000 7f56d0e35640 1 -- 172.21.15.123:0/2670421423 shutdown_connections 2024-09-16T15:44:07.430 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:07.424+0000 7f56d0e35640 1 --2- 172.21.15.123:0/2670421423 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f56ac073070 0x7f56ac075530 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:07.430 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:07.424+0000 7f56d0e35640 1 --2- 172.21.15.123:0/2670421423 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f56cc161620 0x7f56cc0c9140 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:07.430 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:07.424+0000 7f56d0e35640 1 --2- 172.21.15.123:0/2670421423 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f56cc0b2b00 0x7f56cc153e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:07.430 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:07.424+0000 7f56d0e35640 1 --2- 172.21.15.123:0/2670421423 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f56cc0b0a50 0x7f56cc14ce70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:07.430 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:07.424+0000 7f56d0e35640 1 -- 172.21.15.123:0/2670421423 >> 172.21.15.123:0/2670421423 conn(0x7f56cc0d3ea0 msgr2=0x7f56cc158fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:44:07.430 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:07.424+0000 7f56d0e35640 1 -- 172.21.15.123:0/2670421423 shutdown_connections 2024-09-16T15:44:07.430 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:07.424+0000 7f56d0e35640 1 -- 172.21.15.123:0/2670421423 wait complete. 2024-09-16T15:44:07.448 INFO:tasks.cephfs_test_runner:Validate binary .meta file on legacy subvol root doesn't break the system ... ok 2024-09-16T15:44:07.449 INFO:tasks.cephfs_test_runner:Starting test: test_connection_expiration (tasks.cephfs.test_volumes.TestMisc) 2024-09-16T15:44:07.449 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-16T15:44:07.573 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:07.568+0000 7f1d5181b640 1 -- 172.21.15.123:0/1435023213 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1d4c15e160 msgr2=0x7f1d4c073f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:44:07.573 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:07.568+0000 7f1d5181b640 1 --2- 172.21.15.123:0/1435023213 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1d4c15e160 0x7f1d4c073f30 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f1d40009900 tx=0x7f1d4001c850 comp rx=0 tx=0).stop 2024-09-16T15:44:07.573 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:07.568+0000 7f1d5181b640 1 -- 172.21.15.123:0/1435023213 shutdown_connections 2024-09-16T15:44:07.573 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:07.568+0000 7f1d5181b640 1 --2- 172.21.15.123:0/1435023213 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f1d4c074470 0x7f1d4c0cd6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:07.573 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:07.568+0000 7f1d5181b640 1 --2- 172.21.15.123:0/1435023213 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1d4c15e160 0x7f1d4c073f30 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:07.573 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:07.568+0000 7f1d5181b640 1 -- 172.21.15.123:0/1435023213 >> 172.21.15.123:0/1435023213 conn(0x7f1d4c0cfb10 msgr2=0x7f1d4c0cff20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:44:07.573 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:07.568+0000 7f1d5181b640 1 -- 172.21.15.123:0/1435023213 shutdown_connections 2024-09-16T15:44:07.573 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:07.568+0000 7f1d5181b640 1 -- 172.21.15.123:0/1435023213 wait complete. 2024-09-16T15:44:07.574 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:07.568+0000 7f1d5181b640 1 Processor -- start 2024-09-16T15:44:07.574 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:07.568+0000 7f1d5181b640 1 -- start start 2024-09-16T15:44:07.575 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:07.568+0000 7f1d5181b640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f1d4c074470 0x7f1d4c0b3240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:44:07.575 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:07.568+0000 7f1d5181b640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1d4c0cdc20 0x7f1d4c0b5310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:44:07.576 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:07.568+0000 7f1d5181b640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1d4c15e160 0x7f1d4c151b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:44:07.576 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:07.568+0000 7f1d5181b640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f1d4c167f50 con 0x7f1d4c15e160 2024-09-16T15:44:07.576 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:07.568+0000 7f1d5181b640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f1d4c167dd0 con 0x7f1d4c0cdc20 2024-09-16T15:44:07.576 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:07.568+0000 7f1d5181b640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f1d4c1680d0 con 0x7f1d4c074470 2024-09-16T15:44:07.576 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:07.568+0000 7f1d4b7fe640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1d4c15e160 0x7f1d4c151b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:44:07.576 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:07.568+0000 7f1d4b7fe640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1d4c15e160 0x7f1d4c151b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:57962/0 (socket says 172.21.15.123:57962) 2024-09-16T15:44:07.576 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:07.568+0000 7f1d4b7fe640 1 -- 172.21.15.123:0/318920004 learned_addr learned my addr 172.21.15.123:0/318920004 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:44:07.576 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:07.568+0000 7f1d4b7fe640 1 -- 172.21.15.123:0/318920004 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f1d4c074470 msgr2=0x7f1d4c0b3240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:44:07.576 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:07.568+0000 7f1d4b7fe640 1 --2- 172.21.15.123:0/318920004 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f1d4c074470 0x7f1d4c0b3240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:07.576 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:07.568+0000 7f1d4b7fe640 1 -- 172.21.15.123:0/318920004 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1d4c0cdc20 msgr2=0x7f1d4c0b5310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:44:07.576 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:07.568+0000 7f1d4b7fe640 1 --2- 172.21.15.123:0/318920004 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1d4c0cdc20 0x7f1d4c0b5310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:07.576 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:07.568+0000 7f1d4b7fe640 1 -- 172.21.15.123:0/318920004 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1d4c15d280 con 0x7f1d4c15e160 2024-09-16T15:44:07.577 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:07.572+0000 7f1d4b7fe640 1 --2- 172.21.15.123:0/318920004 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1d4c15e160 0x7f1d4c151b70 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f1d4400c790 tx=0x7f1d4400cc50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:44:07.577 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:07.572+0000 7f1d33fff640 1 -- 172.21.15.123:0/318920004 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d44003b80 con 0x7f1d4c15e160 2024-09-16T15:44:07.577 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:07.572+0000 7f1d33fff640 1 -- 172.21.15.123:0/318920004 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f1d44003d40 con 0x7f1d4c15e160 2024-09-16T15:44:07.577 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:07.572+0000 7f1d33fff640 1 -- 172.21.15.123:0/318920004 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d4400aab0 con 0x7f1d4c15e160 2024-09-16T15:44:07.577 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:07.572+0000 7f1d5181b640 1 -- 172.21.15.123:0/318920004 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1d4c15d560 con 0x7f1d4c15e160 2024-09-16T15:44:07.577 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:07.572+0000 7f1d5181b640 1 -- 172.21.15.123:0/318920004 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1d4c0b2a40 con 0x7f1d4c15e160 2024-09-16T15:44:07.578 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:07.572+0000 7f1d5181b640 1 -- 172.21.15.123:0/318920004 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1d4c0cd6e0 con 0x7f1d4c15e160 2024-09-16T15:44:07.578 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:07.572+0000 7f1d33fff640 1 -- 172.21.15.123:0/318920004 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 7) ==== 97233+0+0 (secure 0 0 0) 0x7f1d4400ac50 con 0x7f1d4c15e160 2024-09-16T15:44:07.578 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:07.572+0000 7f1d33fff640 1 --2- 172.21.15.123:0/318920004 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f1d380730a0 0x7f1d38075560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:44:07.578 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:07.572+0000 7f1d33fff640 1 -- 172.21.15.123:0/318920004 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(36..36 src has 1..36) ==== 4943+0+0 (secure 0 0 0) 0x7f1d44093cd0 con 0x7f1d4c15e160 2024-09-16T15:44:07.578 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:07.572+0000 7f1d4affd640 1 --2- 172.21.15.123:0/318920004 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f1d380730a0 0x7f1d38075560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:44:07.578 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:07.572+0000 7f1d4affd640 1 --2- 172.21.15.123:0/318920004 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f1d380730a0 0x7f1d38075560 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f1d400af000 tx=0x7f1d400195f0 comp rx=0 tx=0).ready entity=mgr.4135 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:44:07.581 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:07.576+0000 7f1d33fff640 1 -- 172.21.15.123:0/318920004 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f1d4405ef20 con 0x7f1d4c15e160 2024-09-16T15:44:07.754 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:07.748+0000 7f1d5181b640 1 -- 172.21.15.123:0/318920004 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f1d4c06fcb0 con 0x7f1d4c15e160 2024-09-16T15:44:07.756 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:07.752+0000 7f1d33fff640 1 -- 172.21.15.123:0/318920004 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v36) ==== 74+0+13092 (secure 0 0 0) 0x7f1d4405e820 con 0x7f1d4c15e160 2024-09-16T15:44:07.756 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-09-16T15:44:07.756 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":36,"fsid":"56077f89-18c8-486f-ae7e-eecc0bb7822c","created":"2024-09-16T15:42:26.364892+0000","modified":"2024-09-16T15:44:06.411917+0000","last_up_change":"2024-09-16T15:42:38.707234+0000","last_in_change":"2024-09-16T15:42:29.423662+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-16T15:42:39.287764+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9e7c1955-a607-4671-9be7-f2ef760ceeaa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6800","nonce":3882044969},{"type":"v1","addr":"172.21.15.123:6801","nonce":3882044969}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6802","nonce":3882044969},{"type":"v1","addr":"172.21.15.123:6803","nonce":3882044969}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6806","nonce":3882044969},{"type":"v1","addr":"172.21.15.123:6807","nonce":3882044969}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6804","nonce":3882044969},{"type":"v1","addr":"172.21.15.123:6805","nonce":3882044969}]},"public_addr":"172.21.15.123:6801/3882044969","cluster_addr":"172.21.15.123:6803/3882044969","heartbeat_back_addr":"172.21.15.123:6807/3882044969","heartbeat_front_addr":"172.21.15.123:6805/3882044969","state":["exists","up"]},{"osd":1,"uuid":"c540cfde-0977-4742-9b17-fd634668e010","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6816","nonce":4037001850},{"type":"v1","addr":"172.21.15.123:6817","nonce":4037001850}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6818","nonce":4037001850},{"type":"v1","addr":"172.21.15.123:6819","nonce":4037001850}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6822","nonce":4037001850},{"type":"v1","addr":"172.21.15.123:6823","nonce":4037001850}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6820","nonce":4037001850},{"type":"v1","addr":"172.21.15.123:6821","nonce":4037001850}]},"public_addr":"172.21.15.123:6817/4037001850","cluster_addr":"172.21.15.123:6819/4037001850","heartbeat_back_addr":"172.21.15.123:6823/4037001850","heartbeat_front_addr":"172.21.15.123:6821/4037001850","state":["exists","up"]},{"osd":2,"uuid":"9864a496-f169-486b-95c4-1d59162793a7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6824","nonce":1332143088},{"type":"v1","addr":"172.21.15.123:6825","nonce":1332143088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6826","nonce":1332143088},{"type":"v1","addr":"172.21.15.123:6827","nonce":1332143088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6830","nonce":1332143088},{"type":"v1","addr":"172.21.15.123:6831","nonce":1332143088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6828","nonce":1332143088},{"type":"v1","addr":"172.21.15.123:6829","nonce":1332143088}]},"public_addr":"172.21.15.123:6825/1332143088","cluster_addr":"172.21.15.123:6827/1332143088","heartbeat_back_addr":"172.21.15.123:6831/1332143088","heartbeat_front_addr":"172.21.15.123:6829/1332143088","state":["exists","up"]},{"osd":3,"uuid":"c6d1f817-f074-4325-95e1-578eba13edda","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6808","nonce":988586998},{"type":"v1","addr":"172.21.15.123:6809","nonce":988586998}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6810","nonce":988586998},{"type":"v1","addr":"172.21.15.123:6811","nonce":988586998}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6814","nonce":988586998},{"type":"v1","addr":"172.21.15.123:6815","nonce":988586998}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6812","nonce":988586998},{"type":"v1","addr":"172.21.15.123:6813","nonce":988586998}]},"public_addr":"172.21.15.123:6809/988586998","cluster_addr":"172.21.15.123:6811/988586998","heartbeat_back_addr":"172.21.15.123:6815/988586998","heartbeat_front_addr":"172.21.15.123:6813/988586998","state":["exists","up"]},{"osd":4,"uuid":"200a8df4-50ef-40a5-b48a-58981269c086","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6816","nonce":3387813890},{"type":"v1","addr":"172.21.15.184:6817","nonce":3387813890}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6818","nonce":3387813890},{"type":"v1","addr":"172.21.15.184:6819","nonce":3387813890}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6822","nonce":3387813890},{"type":"v1","addr":"172.21.15.184:6823","nonce":3387813890}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6820","nonce":3387813890},{"type":"v1","addr":"172.21.15.184:6821","nonce":3387813890}]},"public_addr":"172.21.15.184:6817/3387813890","cluster_addr":"172.21.15.184:6819/3387813890","heartbeat_back_addr":"172.21.15.184:6823/3387813890","heartbeat_front_addr":"172.21.15.184:6821/3387813890","state":["exists","up"]},{"osd":5,"uuid":"32d5a900-da3d-412c-bc19-13861a2368d8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6808","nonce":2680916017},{"type":"v1","addr":"172.21.15.184:6809","nonce":2680916017}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6810","nonce":2680916017},{"type":"v1","addr":"172.21.15.184:6811","nonce":2680916017}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6814","nonce":2680916017},{"type":"v1","addr":"172.21.15.184:6815","nonce":2680916017}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6812","nonce":2680916017},{"type":"v1","addr":"172.21.15.184:6813","nonce":2680916017}]},"public_addr":"172.21.15.184:6809/2680916017","cluster_addr":"172.21.15.184:6811/2680916017","heartbeat_back_addr":"172.21.15.184:6815/2680916017","heartbeat_front_addr":"172.21.15.184:6813/2680916017","state":["exists","up"]},{"osd":6,"uuid":"cde6f39e-aa1b-4260-ac19-774ee6aaca72","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6824","nonce":2183120457},{"type":"v1","addr":"172.21.15.184:6825","nonce":2183120457}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6826","nonce":2183120457},{"type":"v1","addr":"172.21.15.184:6827","nonce":2183120457}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6830","nonce":2183120457},{"type":"v1","addr":"172.21.15.184:6831","nonce":2183120457}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6828","nonce":2183120457},{"type":"v1","addr":"172.21.15.184:6829","nonce":2183120457}]},"public_addr":"172.21.15.184:6825/2183120457","cluster_addr":"172.21.15.184:6827/2183120457","heartbeat_back_addr":"172.21.15.184:6831/2183120457","heartbeat_front_addr":"172.21.15.184:6829/2183120457","state":["exists","up"]},{"osd":7,"uuid":"4f381778-eee1-47c6-804f-df0e6b33e313","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6800","nonce":4102957285},{"type":"v1","addr":"172.21.15.184:6801","nonce":4102957285}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6802","nonce":4102957285},{"type":"v1","addr":"172.21.15.184:6803","nonce":4102957285}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6806","nonce":4102957285},{"type":"v1","addr":"172.21.15.184:6807","nonce":4102957285}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6804","nonce":4102957285},{"type":"v1","addr":"172.21.15.184:6805","nonce":4102957285}]},"public_addr":"172.21.15.184:6801/4102957285","cluster_addr":"172.21.15.184:6803/4102957285","heartbeat_back_addr":"172.21.15.184:6807/4102957285","heartbeat_front_addr":"172.21.15.184:6805/4102957285","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:35.972210+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.222007+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.198617+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.385663+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.358935+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.156066+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.752973+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:35.682846+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.123:6835/822918970":"2024-09-17T15:44:03.075050+0000","172.21.15.123:6834/822918970":"2024-09-17T15:44:03.075050+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T15:44:07.758 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:07.752+0000 7f1d5181b640 1 -- 172.21.15.123:0/318920004 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f1d380730a0 msgr2=0x7f1d38075560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:44:07.758 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:07.752+0000 7f1d5181b640 1 --2- 172.21.15.123:0/318920004 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f1d380730a0 0x7f1d38075560 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f1d400af000 tx=0x7f1d400195f0 comp rx=0 tx=0).stop 2024-09-16T15:44:07.758 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:07.752+0000 7f1d5181b640 1 -- 172.21.15.123:0/318920004 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1d4c15e160 msgr2=0x7f1d4c151b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:44:07.759 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:07.752+0000 7f1d5181b640 1 --2- 172.21.15.123:0/318920004 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1d4c15e160 0x7f1d4c151b70 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f1d4400c790 tx=0x7f1d4400cc50 comp rx=0 tx=0).stop 2024-09-16T15:44:07.759 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:07.752+0000 7f1d5181b640 1 -- 172.21.15.123:0/318920004 shutdown_connections 2024-09-16T15:44:07.759 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:07.752+0000 7f1d5181b640 1 --2- 172.21.15.123:0/318920004 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f1d380730a0 0x7f1d38075560 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:07.759 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:07.752+0000 7f1d5181b640 1 --2- 172.21.15.123:0/318920004 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1d4c15e160 0x7f1d4c151b70 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:07.759 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:07.752+0000 7f1d5181b640 1 --2- 172.21.15.123:0/318920004 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1d4c0cdc20 0x7f1d4c0b5310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:07.759 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:07.752+0000 7f1d5181b640 1 --2- 172.21.15.123:0/318920004 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f1d4c074470 0x7f1d4c0b3240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:07.759 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:07.752+0000 7f1d5181b640 1 -- 172.21.15.123:0/318920004 >> 172.21.15.123:0/318920004 conn(0x7f1d4c0cfb10 msgr2=0x7f1d4c164450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:44:07.759 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:07.752+0000 7f1d5181b640 1 -- 172.21.15.123:0/318920004 shutdown_connections 2024-09-16T15:44:07.759 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:07.752+0000 7f1d5181b640 1 -- 172.21.15.123:0/318920004 wait complete. 2024-09-16T15:44:07.775 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-16T15:42:39.287764+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-09-16T15:44:07.775 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-09-16T15:44:07.900 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:07.896+0000 7f87e3fa9640 1 -- 172.21.15.123:0/2199259168 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f87dc072720 msgr2=0x7f87dc0cabc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:44:07.900 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:07.896+0000 7f87e3fa9640 1 --2- 172.21.15.123:0/2199259168 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f87dc072720 0x7f87dc0cabc0 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f87cc009900 tx=0x7f87cc01c8e0 comp rx=0 tx=0).stop 2024-09-16T15:44:07.900 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:07.896+0000 7f87e3fa9640 1 -- 172.21.15.123:0/2199259168 shutdown_connections 2024-09-16T15:44:07.900 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:07.896+0000 7f87e3fa9640 1 --2- 172.21.15.123:0/2199259168 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f87dc072720 0x7f87dc0cabc0 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:07.900 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:07.896+0000 7f87e3fa9640 1 --2- 172.21.15.123:0/2199259168 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f87dc15aa00 0x7f87dc15e5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:07.900 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:07.896+0000 7f87e3fa9640 1 -- 172.21.15.123:0/2199259168 >> 172.21.15.123:0/2199259168 conn(0x7f87dc0b1710 msgr2=0x7f87dc0b1b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:44:07.901 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:07.896+0000 7f87e3fa9640 1 -- 172.21.15.123:0/2199259168 shutdown_connections 2024-09-16T15:44:07.901 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:07.896+0000 7f87e3fa9640 1 -- 172.21.15.123:0/2199259168 wait complete. 2024-09-16T15:44:07.901 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:07.896+0000 7f87e3fa9640 1 Processor -- start 2024-09-16T15:44:07.901 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:07.896+0000 7f87e3fa9640 1 -- start start 2024-09-16T15:44:07.903 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:07.896+0000 7f87e3fa9640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f87dc072720 0x7f87dc154c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:44:07.903 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:07.896+0000 7f87e3fa9640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f87dc0cc3c0 0x7f87dc15fcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:44:07.903 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:07.896+0000 7f87e3fa9640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f87dc15aa00 0x7f87dc0cd710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:44:07.903 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:07.896+0000 7f87e3fa9640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f87dc168390 con 0x7f87dc15aa00 2024-09-16T15:44:07.904 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:07.896+0000 7f87e3fa9640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f87dc06ffa0 con 0x7f87dc0cc3c0 2024-09-16T15:44:07.904 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:07.896+0000 7f87e3fa9640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f87dc168510 con 0x7f87dc072720 2024-09-16T15:44:07.904 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:07.896+0000 7f87e2546640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f87dc15aa00 0x7f87dc0cd710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:44:07.904 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:07.896+0000 7f87e2546640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f87dc15aa00 0x7f87dc0cd710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:57980/0 (socket says 172.21.15.123:57980) 2024-09-16T15:44:07.904 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:07.896+0000 7f87e2546640 1 -- 172.21.15.123:0/479342572 learned_addr learned my addr 172.21.15.123:0/479342572 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:44:07.904 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:07.896+0000 7f87e1d45640 1 --2- 172.21.15.123:0/479342572 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f87dc072720 0x7f87dc154c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:44:07.904 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:07.896+0000 7f87e1544640 1 --2- 172.21.15.123:0/479342572 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f87dc0cc3c0 0x7f87dc15fcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:44:07.904 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:07.896+0000 7f87e2546640 1 -- 172.21.15.123:0/479342572 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f87dc072720 msgr2=0x7f87dc154c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:44:07.904 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:07.896+0000 7f87e2546640 1 --2- 172.21.15.123:0/479342572 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f87dc072720 0x7f87dc154c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:07.904 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:07.896+0000 7f87e2546640 1 -- 172.21.15.123:0/479342572 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f87dc0cc3c0 msgr2=0x7f87dc15fcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:44:07.904 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:07.896+0000 7f87e2546640 1 --2- 172.21.15.123:0/479342572 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f87dc0cc3c0 0x7f87dc15fcf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:07.904 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:07.896+0000 7f87e2546640 1 -- 172.21.15.123:0/479342572 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f87dc1f9510 con 0x7f87dc15aa00 2024-09-16T15:44:07.904 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:07.896+0000 7f87e1544640 1 --2- 172.21.15.123:0/479342572 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f87dc0cc3c0 0x7f87dc15fcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T15:44:07.905 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:07.896+0000 7f87e2546640 1 --2- 172.21.15.123:0/479342572 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f87dc15aa00 0x7f87dc0cd710 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f87d400c7c0 tx=0x7f87d400cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:44:07.905 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:07.896+0000 7f87d27fc640 1 -- 172.21.15.123:0/479342572 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87d4003b80 con 0x7f87dc15aa00 2024-09-16T15:44:07.905 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:07.896+0000 7f87d27fc640 1 -- 172.21.15.123:0/479342572 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f87d4003d20 con 0x7f87dc15aa00 2024-09-16T15:44:07.905 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:07.896+0000 7f87d27fc640 1 -- 172.21.15.123:0/479342572 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87d400aab0 con 0x7f87dc15aa00 2024-09-16T15:44:07.905 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:07.896+0000 7f87e3fa9640 1 -- 172.21.15.123:0/479342572 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f87dc1f97f0 con 0x7f87dc15aa00 2024-09-16T15:44:07.905 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:07.896+0000 7f87e3fa9640 1 -- 172.21.15.123:0/479342572 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f87dc1f9db0 con 0x7f87dc15aa00 2024-09-16T15:44:07.905 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:07.900+0000 7f87e3fa9640 1 -- 172.21.15.123:0/479342572 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f87dc15e620 con 0x7f87dc15aa00 2024-09-16T15:44:07.906 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:07.900+0000 7f87d27fc640 1 -- 172.21.15.123:0/479342572 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 7) ==== 97233+0+0 (secure 0 0 0) 0x7f87d400ac50 con 0x7f87dc15aa00 2024-09-16T15:44:07.906 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:07.900+0000 7f87d27fc640 1 --2- 172.21.15.123:0/479342572 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f87c0073070 0x7f87c0075530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:44:07.906 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:07.900+0000 7f87d27fc640 1 -- 172.21.15.123:0/479342572 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(36..36 src has 1..36) ==== 4943+0+0 (secure 0 0 0) 0x7f87d4093dc0 con 0x7f87dc15aa00 2024-09-16T15:44:07.906 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:07.900+0000 7f87e1d45640 1 --2- 172.21.15.123:0/479342572 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f87c0073070 0x7f87c0075530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:44:07.906 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:07.900+0000 7f87e1d45640 1 --2- 172.21.15.123:0/479342572 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f87c0073070 0x7f87c0075530 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f87d80058d0 tx=0x7f87d8005860 comp rx=0 tx=0).ready entity=mgr.4135 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:44:07.909 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:07.904+0000 7f87d27fc640 1 -- 172.21.15.123:0/479342572 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f87d405ef80 con 0x7f87dc15aa00 2024-09-16T15:44:08.070 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:08.064+0000 7f87e3fa9640 1 -- 172.21.15.123:0/479342572 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f87dc0697b0 con 0x7f87dc15aa00 2024-09-16T15:44:08.071 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:08.064+0000 7f87d27fc640 1 -- 172.21.15.123:0/479342572 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v36) ==== 93+0+10 (secure 0 0 0) 0x7f87d405e880 con 0x7f87dc15aa00 2024-09-16T15:44:08.071 INFO:teuthology.orchestra.run.smithi123.stdout:pg_num: 1 2024-09-16T15:44:08.073 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:08.068+0000 7f87e3fa9640 1 -- 172.21.15.123:0/479342572 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f87c0073070 msgr2=0x7f87c0075530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:44:08.073 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:08.068+0000 7f87e3fa9640 1 --2- 172.21.15.123:0/479342572 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f87c0073070 0x7f87c0075530 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f87d80058d0 tx=0x7f87d8005860 comp rx=0 tx=0).stop 2024-09-16T15:44:08.073 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:08.068+0000 7f87e3fa9640 1 -- 172.21.15.123:0/479342572 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f87dc15aa00 msgr2=0x7f87dc0cd710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:44:08.073 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:08.068+0000 7f87e3fa9640 1 --2- 172.21.15.123:0/479342572 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f87dc15aa00 0x7f87dc0cd710 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f87d400c7c0 tx=0x7f87d400cc80 comp rx=0 tx=0).stop 2024-09-16T15:44:08.073 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:08.068+0000 7f87e3fa9640 1 -- 172.21.15.123:0/479342572 shutdown_connections 2024-09-16T15:44:08.073 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:08.068+0000 7f87e3fa9640 1 --2- 172.21.15.123:0/479342572 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f87c0073070 0x7f87c0075530 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:08.073 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:08.068+0000 7f87e3fa9640 1 --2- 172.21.15.123:0/479342572 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f87dc15aa00 0x7f87dc0cd710 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:08.073 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:08.068+0000 7f87e3fa9640 1 --2- 172.21.15.123:0/479342572 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f87dc0cc3c0 0x7f87dc15fcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:08.073 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:08.068+0000 7f87e3fa9640 1 --2- 172.21.15.123:0/479342572 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f87dc072720 0x7f87dc154c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:08.073 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:08.068+0000 7f87e3fa9640 1 -- 172.21.15.123:0/479342572 >> 172.21.15.123:0/479342572 conn(0x7f87dc0b1710 msgr2=0x7f87dc0cfa80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:44:08.073 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:08.068+0000 7f87e3fa9640 1 -- 172.21.15.123:0/479342572 shutdown_connections 2024-09-16T15:44:08.074 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:08.068+0000 7f87e3fa9640 1 -- 172.21.15.123:0/479342572 wait complete. 2024-09-16T15:44:08.088 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_volumes.TestMisc.test_connection_expiration' 2024-09-16T15:44:08.210 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:08.204+0000 7f5d4aa39640 1 -- 172.21.15.123:0/2235097213 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5d4415bda0 msgr2=0x7f5d440d0310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:44:08.210 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:08.204+0000 7f5d4aa39640 1 --2- 172.21.15.123:0/2235097213 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5d4415bda0 0x7f5d440d0310 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f5d34009900 tx=0x7f5d3401c850 comp rx=0 tx=0).stop 2024-09-16T15:44:08.210 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:08.204+0000 7f5d4aa39640 1 -- 172.21.15.123:0/2235097213 shutdown_connections 2024-09-16T15:44:08.210 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:08.204+0000 7f5d4aa39640 1 --2- 172.21.15.123:0/2235097213 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5d4415bda0 0x7f5d440d0310 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:08.210 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:08.204+0000 7f5d4aa39640 1 -- 172.21.15.123:0/2235097213 >> 172.21.15.123:0/2235097213 conn(0x7f5d44074bd0 msgr2=0x7f5d4409aa40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:44:08.210 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:08.204+0000 7f5d4aa39640 1 -- 172.21.15.123:0/2235097213 shutdown_connections 2024-09-16T15:44:08.210 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:08.204+0000 7f5d4aa39640 1 -- 172.21.15.123:0/2235097213 wait complete. 2024-09-16T15:44:08.211 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:08.208+0000 7f5d4aa39640 1 Processor -- start 2024-09-16T15:44:08.211 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:08.208+0000 7f5d4aa39640 1 -- start start 2024-09-16T15:44:08.213 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:08.208+0000 7f5d4aa39640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f5d440b25a0 0x7f5d441516f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:44:08.213 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:08.208+0000 7f5d4aa39640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5d440d2320 0x7f5d441f53f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:44:08.213 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:08.208+0000 7f5d4aa39640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5d4415bda0 0x7f5d441f90a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:44:08.213 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:08.208+0000 7f5d4aa39640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f5d44169da0 con 0x7f5d4415bda0 2024-09-16T15:44:08.213 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:08.208+0000 7f5d4aa39640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f5d44169c20 con 0x7f5d440d2320 2024-09-16T15:44:08.213 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:08.208+0000 7f5d4aa39640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f5d44169f20 con 0x7f5d440b25a0 2024-09-16T15:44:08.213 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:08.208+0000 7f5d48fd6640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5d4415bda0 0x7f5d441f90a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:44:08.214 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:08.208+0000 7f5d48fd6640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5d4415bda0 0x7f5d441f90a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:58004/0 (socket says 172.21.15.123:58004) 2024-09-16T15:44:08.214 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:08.208+0000 7f5d48fd6640 1 -- 172.21.15.123:0/927725778 learned_addr learned my addr 172.21.15.123:0/927725778 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:44:08.214 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:08.208+0000 7f5d437fe640 1 --2- 172.21.15.123:0/927725778 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5d440d2320 0x7f5d441f53f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:44:08.214 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:08.208+0000 7f5d43fff640 1 --2- 172.21.15.123:0/927725778 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f5d440b25a0 0x7f5d441516f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:44:08.214 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:08.208+0000 7f5d48fd6640 1 -- 172.21.15.123:0/927725778 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f5d440b25a0 msgr2=0x7f5d441516f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:44:08.214 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:08.208+0000 7f5d48fd6640 1 --2- 172.21.15.123:0/927725778 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f5d440b25a0 0x7f5d441516f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:08.214 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:08.208+0000 7f5d48fd6640 1 -- 172.21.15.123:0/927725778 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5d440d2320 msgr2=0x7f5d441f53f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:44:08.214 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:08.208+0000 7f5d48fd6640 1 --2- 172.21.15.123:0/927725778 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5d440d2320 0x7f5d441f53f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:08.214 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:08.208+0000 7f5d48fd6640 1 -- 172.21.15.123:0/927725778 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5d441f5b60 con 0x7f5d4415bda0 2024-09-16T15:44:08.214 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:08.208+0000 7f5d48fd6640 1 --2- 172.21.15.123:0/927725778 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5d4415bda0 0x7f5d441f90a0 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f5d3c00c6f0 tx=0x7f5d3c00cbb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:44:08.214 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:08.208+0000 7f5d40ff9640 1 -- 172.21.15.123:0/927725778 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d3c0039a0 con 0x7f5d4415bda0 2024-09-16T15:44:08.214 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:08.208+0000 7f5d40ff9640 1 -- 172.21.15.123:0/927725778 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f5d3c003b40 con 0x7f5d4415bda0 2024-09-16T15:44:08.214 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:08.208+0000 7f5d4aa39640 1 -- 172.21.15.123:0/927725778 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5d441f5e40 con 0x7f5d4415bda0 2024-09-16T15:44:08.215 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:08.208+0000 7f5d40ff9640 1 -- 172.21.15.123:0/927725778 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d3c00a850 con 0x7f5d4415bda0 2024-09-16T15:44:08.215 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:08.208+0000 7f5d4aa39640 1 -- 172.21.15.123:0/927725778 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5d4414c040 con 0x7f5d4415bda0 2024-09-16T15:44:08.215 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:08.208+0000 7f5d4aa39640 1 -- 172.21.15.123:0/927725778 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5d440d0310 con 0x7f5d4415bda0 2024-09-16T15:44:08.215 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:08.208+0000 7f5d40ff9640 1 -- 172.21.15.123:0/927725778 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 7) ==== 97233+0+0 (secure 0 0 0) 0x7f5d3c00a9f0 con 0x7f5d4415bda0 2024-09-16T15:44:08.216 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:08.208+0000 7f5d40ff9640 1 --2- 172.21.15.123:0/927725778 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f5d30073070 0x7f5d30075530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:44:08.216 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:08.208+0000 7f5d40ff9640 1 -- 172.21.15.123:0/927725778 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(36..36 src has 1..36) ==== 4943+0+0 (secure 0 0 0) 0x7f5d3c093c80 con 0x7f5d4415bda0 2024-09-16T15:44:08.216 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:08.212+0000 7f5d43fff640 1 --2- 172.21.15.123:0/927725778 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f5d30073070 0x7f5d30075530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:44:08.216 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:08.212+0000 7f5d43fff640 1 --2- 172.21.15.123:0/927725778 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f5d30073070 0x7f5d30075530 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f5d340af000 tx=0x7f5d3401b090 comp rx=0 tx=0).ready entity=mgr.4135 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:44:08.219 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:08.212+0000 7f5d40ff9640 1 -- 172.21.15.123:0/927725778 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f5d3c05eed0 con 0x7f5d4415bda0 2024-09-16T15:44:08.440 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:08.436+0000 7f5d4aa39640 1 -- 172.21.15.123:0/927725778 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_volumes.TestMisc.test_connection_expiration"]} v 0) -- 0x7f5d441f89e0 con 0x7f5d4415bda0 2024-09-16T15:44:08.697 DEBUG:teuthology.orchestra.run.smithi123:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-16T15:44:08.700 DEBUG:teuthology.orchestra.run.smithi184:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-16T15:44:09.438 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:09.432+0000 7f5d40ff9640 1 -- 172.21.15.123:0/927725778 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_volumes.TestMisc.test_connection_expiration"]}]=0 v91) ==== 143+0+0 (secure 0 0 0) 0x7f5d3c05e7d0 con 0x7f5d4415bda0 2024-09-16T15:44:09.442 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:09.436+0000 7f5d4aa39640 1 -- 172.21.15.123:0/927725778 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f5d30073070 msgr2=0x7f5d30075530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:44:09.443 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:09.436+0000 7f5d4aa39640 1 --2- 172.21.15.123:0/927725778 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f5d30073070 0x7f5d30075530 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f5d340af000 tx=0x7f5d3401b090 comp rx=0 tx=0).stop 2024-09-16T15:44:09.443 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:09.436+0000 7f5d4aa39640 1 -- 172.21.15.123:0/927725778 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5d4415bda0 msgr2=0x7f5d441f90a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:44:09.443 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:09.436+0000 7f5d4aa39640 1 --2- 172.21.15.123:0/927725778 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5d4415bda0 0x7f5d441f90a0 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f5d3c00c6f0 tx=0x7f5d3c00cbb0 comp rx=0 tx=0).stop 2024-09-16T15:44:09.443 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:09.436+0000 7f5d4aa39640 1 -- 172.21.15.123:0/927725778 shutdown_connections 2024-09-16T15:44:09.443 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:09.436+0000 7f5d4aa39640 1 --2- 172.21.15.123:0/927725778 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f5d30073070 0x7f5d30075530 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:09.443 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:09.436+0000 7f5d4aa39640 1 --2- 172.21.15.123:0/927725778 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5d4415bda0 0x7f5d441f90a0 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:09.443 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:09.436+0000 7f5d4aa39640 1 --2- 172.21.15.123:0/927725778 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5d440d2320 0x7f5d441f53f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:09.443 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:09.436+0000 7f5d4aa39640 1 --2- 172.21.15.123:0/927725778 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f5d440b25a0 0x7f5d441516f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:09.443 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:09.436+0000 7f5d4aa39640 1 -- 172.21.15.123:0/927725778 >> 172.21.15.123:0/927725778 conn(0x7f5d44074bd0 msgr2=0x7f5d440cea10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:44:09.443 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:09.436+0000 7f5d4aa39640 1 -- 172.21.15.123:0/927725778 shutdown_connections 2024-09-16T15:44:09.443 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:09.436+0000 7f5d4aa39640 1 -- 172.21.15.123:0/927725778 wait complete. 2024-09-16T15:44:09.461 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config dump 2024-09-16T15:44:09.583 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:09.576+0000 7f26f856f640 1 -- 172.21.15.123:0/593098939 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f26f0161620 msgr2=0x7f26f00d1da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:44:09.583 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:09.576+0000 7f26f856f640 1 --2- 172.21.15.123:0/593098939 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f26f0161620 0x7f26f00d1da0 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f26ec0098f0 tx=0x7f26ec01c790 comp rx=0 tx=0).stop 2024-09-16T15:44:09.584 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:09.580+0000 7f26f856f640 1 -- 172.21.15.123:0/593098939 shutdown_connections 2024-09-16T15:44:09.584 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:09.580+0000 7f26f856f640 1 --2- 172.21.15.123:0/593098939 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f26f00b0a50 0x7f26f00b0ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:09.584 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:09.580+0000 7f26f856f640 1 --2- 172.21.15.123:0/593098939 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f26f0161620 0x7f26f00d1da0 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:09.584 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:09.580+0000 7f26f856f640 1 -- 172.21.15.123:0/593098939 >> 172.21.15.123:0/593098939 conn(0x7f26f00d3ea0 msgr2=0x7f26f00d42b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:44:09.584 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:09.580+0000 7f26f856f640 1 -- 172.21.15.123:0/593098939 shutdown_connections 2024-09-16T15:44:09.584 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:09.580+0000 7f26f856f640 1 -- 172.21.15.123:0/593098939 wait complete. 2024-09-16T15:44:09.584 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:09.580+0000 7f26f856f640 1 Processor -- start 2024-09-16T15:44:09.585 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:09.580+0000 7f26f856f640 1 -- start start 2024-09-16T15:44:09.587 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:09.580+0000 7f26f856f640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f26f00b0a50 0x7f26f00ca2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:44:09.587 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:09.580+0000 7f26f856f640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f26f00b2b00 0x7f26f01f81f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:44:09.587 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:09.580+0000 7f26f856f640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f26f0161620 0x7f26f00b4080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:44:09.587 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:09.580+0000 7f26f856f640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f26f016aee0 con 0x7f26f00b2b00 2024-09-16T15:44:09.587 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:09.580+0000 7f26f856f640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f26f016ad60 con 0x7f26f00b0a50 2024-09-16T15:44:09.587 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:09.580+0000 7f26f856f640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f26f016b060 con 0x7f26f0161620 2024-09-16T15:44:09.587 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:09.580+0000 7f26f5b0a640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f26f00b2b00 0x7f26f01f81f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:44:09.588 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:09.580+0000 7f26f5b0a640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f26f00b2b00 0x7f26f01f81f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:58014/0 (socket says 172.21.15.123:58014) 2024-09-16T15:44:09.588 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:09.580+0000 7f26f5b0a640 1 -- 172.21.15.123:0/1684437540 learned_addr learned my addr 172.21.15.123:0/1684437540 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:44:09.588 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:09.580+0000 7f26f6b0c640 1 --2- 172.21.15.123:0/1684437540 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f26f0161620 0x7f26f00b4080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:44:09.588 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:09.580+0000 7f26f630b640 1 --2- 172.21.15.123:0/1684437540 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f26f00b0a50 0x7f26f00ca2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:44:09.588 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:09.580+0000 7f26f5b0a640 1 -- 172.21.15.123:0/1684437540 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f26f0161620 msgr2=0x7f26f00b4080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:44:09.588 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:09.580+0000 7f26f5b0a640 1 --2- 172.21.15.123:0/1684437540 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f26f0161620 0x7f26f00b4080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:09.588 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:09.580+0000 7f26f5b0a640 1 -- 172.21.15.123:0/1684437540 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f26f00b0a50 msgr2=0x7f26f00ca2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:44:09.588 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:09.580+0000 7f26f5b0a640 1 --2- 172.21.15.123:0/1684437540 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f26f00b0a50 0x7f26f00ca2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:09.588 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:09.580+0000 7f26f5b0a640 1 -- 172.21.15.123:0/1684437540 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f26f01f8730 con 0x7f26f00b2b00 2024-09-16T15:44:09.588 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:09.580+0000 7f26f5b0a640 1 --2- 172.21.15.123:0/1684437540 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f26f00b2b00 0x7f26f01f81f0 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f26e000c4e0 tx=0x7f26e000c9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:44:09.588 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:09.580+0000 7f26e6ffd640 1 -- 172.21.15.123:0/1684437540 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26e00052d0 con 0x7f26f00b2b00 2024-09-16T15:44:09.589 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:09.580+0000 7f26e6ffd640 1 -- 172.21.15.123:0/1684437540 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f26e0005470 con 0x7f26f00b2b00 2024-09-16T15:44:09.589 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:09.580+0000 7f26f856f640 1 -- 172.21.15.123:0/1684437540 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f26f00c8010 con 0x7f26f00b2b00 2024-09-16T15:44:09.589 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:09.580+0000 7f26e6ffd640 1 -- 172.21.15.123:0/1684437540 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26e0012890 con 0x7f26f00b2b00 2024-09-16T15:44:09.589 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:09.580+0000 7f26f856f640 1 -- 172.21.15.123:0/1684437540 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f26f00c88b0 con 0x7f26f00b2b00 2024-09-16T15:44:09.590 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:09.584+0000 7f26f856f640 1 -- 172.21.15.123:0/1684437540 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f26f00c8400 con 0x7f26f00b2b00 2024-09-16T15:44:09.590 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:09.584+0000 7f26e6ffd640 1 -- 172.21.15.123:0/1684437540 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 7) ==== 97233+0+0 (secure 0 0 0) 0x7f26e0003600 con 0x7f26f00b2b00 2024-09-16T15:44:09.590 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:09.584+0000 7f26e6ffd640 1 --2- 172.21.15.123:0/1684437540 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f26d40730a0 0x7f26d4075560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:44:09.590 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:09.584+0000 7f26e6ffd640 1 -- 172.21.15.123:0/1684437540 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(36..36 src has 1..36) ==== 4943+0+0 (secure 0 0 0) 0x7f26e0093b10 con 0x7f26f00b2b00 2024-09-16T15:44:09.590 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:09.584+0000 7f26f630b640 1 --2- 172.21.15.123:0/1684437540 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f26d40730a0 0x7f26d4075560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:44:09.590 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:09.584+0000 7f26f630b640 1 --2- 172.21.15.123:0/1684437540 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f26d40730a0 0x7f26d4075560 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f26ec0af000 tx=0x7f26ec01afd0 comp rx=0 tx=0).ready entity=mgr.4135 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:44:09.593 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:09.588+0000 7f26e6ffd640 1 -- 172.21.15.123:0/1684437540 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f26f01f8730 con 0x7f26f00b2b00 2024-09-16T15:44:09.751 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:09.744+0000 7f26f856f640 1 -- 172.21.15.123:0/1684437540 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "config dump"} v 0) -- 0x7f26f00d4960 con 0x7f26f00b2b00 2024-09-16T15:44:09.752 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:09.748+0000 7f26e6ffd640 1 -- 172.21.15.123:0/1684437540 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "config dump"}]=0 v29) ==== 59+0+1998 (secure 0 0 0) 0x7f26e005ecd0 con 0x7f26f00b2b00 2024-09-16T15:44:09.752 INFO:teuthology.orchestra.run.smithi123.stdout:WHO MASK LEVEL OPTION VALUE RO 2024-09-16T15:44:09.752 INFO:teuthology.orchestra.run.smithi123.stdout:mon advanced mon_op_complaint_time 120 2024-09-16T15:44:09.752 INFO:teuthology.orchestra.run.smithi123.stdout:mon dev mon_warn_on_pool_no_app false 2024-09-16T15:44:09.753 INFO:teuthology.orchestra.run.smithi123.stdout:osd advanced osd_op_complaint_time 180.000000 2024-09-16T15:44:09.753 INFO:teuthology.orchestra.run.smithi123.stdout:osd.0 basic osd_mclock_max_capacity_iops_ssd 4805.300677 2024-09-16T15:44:09.753 INFO:teuthology.orchestra.run.smithi123.stdout:osd.1 basic osd_mclock_max_capacity_iops_ssd 4806.240607 2024-09-16T15:44:09.753 INFO:teuthology.orchestra.run.smithi123.stdout:osd.2 basic osd_mclock_max_capacity_iops_ssd 4046.441686 2024-09-16T15:44:09.753 INFO:teuthology.orchestra.run.smithi123.stdout:osd.3 basic osd_mclock_max_capacity_iops_ssd 6882.576123 2024-09-16T15:44:09.753 INFO:teuthology.orchestra.run.smithi123.stdout:osd.4 basic osd_mclock_max_capacity_iops_ssd 6108.444971 2024-09-16T15:44:09.753 INFO:teuthology.orchestra.run.smithi123.stdout:osd.5 basic osd_mclock_max_capacity_iops_ssd 4995.096306 2024-09-16T15:44:09.753 INFO:teuthology.orchestra.run.smithi123.stdout:osd.6 basic osd_mclock_max_capacity_iops_ssd 6237.822095 2024-09-16T15:44:09.753 INFO:teuthology.orchestra.run.smithi123.stdout:osd.7 basic osd_mclock_max_capacity_iops_ssd 5202.385138 2024-09-16T15:44:09.753 INFO:teuthology.orchestra.run.smithi123.stdout:mds advanced debug_mds 20/20 2024-09-16T15:44:09.753 INFO:teuthology.orchestra.run.smithi123.stdout:mds advanced debug_mds_balancer 20/20 2024-09-16T15:44:09.753 INFO:teuthology.orchestra.run.smithi123.stdout:mds advanced debug_ms 2/2 2024-09-16T15:44:09.753 INFO:teuthology.orchestra.run.smithi123.stdout:mds dev mds_debug_frag true 2024-09-16T15:44:09.754 INFO:teuthology.orchestra.run.smithi123.stdout:mds dev mds_debug_scatterstat true 2024-09-16T15:44:09.754 INFO:teuthology.orchestra.run.smithi123.stdout:mds advanced mds_op_complaint_time 180.000000 2024-09-16T15:44:09.754 INFO:teuthology.orchestra.run.smithi123.stdout:mds dev mds_verify_scatter true 2024-09-16T15:44:09.754 INFO:teuthology.orchestra.run.smithi123.stdout:mds advanced osd_op_complaint_time 180.000000 2024-09-16T15:44:09.754 INFO:teuthology.orchestra.run.smithi123.stdout:mds advanced rados_mon_op_timeout 900 2024-09-16T15:44:09.754 INFO:teuthology.orchestra.run.smithi123.stdout:mds advanced rados_osd_op_timeout 900 2024-09-16T15:44:09.754 INFO:teuthology.orchestra.run.smithi123.stdout:client advanced client_mount_timeout 600 2024-09-16T15:44:09.754 INFO:teuthology.orchestra.run.smithi123.stdout:client advanced debug_client 20/20 2024-09-16T15:44:09.754 INFO:teuthology.orchestra.run.smithi123.stdout:client advanced debug_ms 1/1 2024-09-16T15:44:09.754 INFO:teuthology.orchestra.run.smithi123.stdout:client advanced rados_mon_op_timeout 900 2024-09-16T15:44:09.754 INFO:teuthology.orchestra.run.smithi123.stdout:client advanced rados_osd_op_timeout 900 2024-09-16T15:44:09.755 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:09.748+0000 7f26f856f640 1 -- 172.21.15.123:0/1684437540 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f26d40730a0 msgr2=0x7f26d4075560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:44:09.755 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:09.748+0000 7f26f856f640 1 --2- 172.21.15.123:0/1684437540 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f26d40730a0 0x7f26d4075560 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f26ec0af000 tx=0x7f26ec01afd0 comp rx=0 tx=0).stop 2024-09-16T15:44:09.756 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:09.748+0000 7f26f856f640 1 -- 172.21.15.123:0/1684437540 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f26f00b2b00 msgr2=0x7f26f01f81f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:44:09.756 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:09.748+0000 7f26f856f640 1 --2- 172.21.15.123:0/1684437540 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f26f00b2b00 0x7f26f01f81f0 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f26e000c4e0 tx=0x7f26e000c9a0 comp rx=0 tx=0).stop 2024-09-16T15:44:09.756 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:09.748+0000 7f26f856f640 1 -- 172.21.15.123:0/1684437540 shutdown_connections 2024-09-16T15:44:09.756 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:09.748+0000 7f26f856f640 1 --2- 172.21.15.123:0/1684437540 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f26d40730a0 0x7f26d4075560 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:09.756 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:09.748+0000 7f26f856f640 1 --2- 172.21.15.123:0/1684437540 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f26f0161620 0x7f26f00b4080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:09.756 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:09.748+0000 7f26f856f640 1 --2- 172.21.15.123:0/1684437540 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f26f00b2b00 0x7f26f01f81f0 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:09.756 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:09.748+0000 7f26f856f640 1 --2- 172.21.15.123:0/1684437540 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f26f00b0a50 0x7f26f00ca2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:09.756 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:09.748+0000 7f26f856f640 1 -- 172.21.15.123:0/1684437540 >> 172.21.15.123:0/1684437540 conn(0x7f26f00d3ea0 msgr2=0x7f26f00b5920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:44:09.756 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:09.748+0000 7f26f856f640 1 -- 172.21.15.123:0/1684437540 shutdown_connections 2024-09-16T15:44:09.756 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:09.748+0000 7f26f856f640 1 -- 172.21.15.123:0/1684437540 wait complete. 2024-09-16T15:44:09.770 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-09-16T15:44:09.884 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:09.880+0000 7f5487034640 1 -- 172.21.15.123:0/4259237823 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f548015a9d0 msgr2=0x7f548006ebe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:44:09.884 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:09.880+0000 7f5487034640 1 --2- 172.21.15.123:0/4259237823 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f548015a9d0 0x7f548006ebe0 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f5470009bc0 tx=0x7f547001cae0 comp rx=0 tx=0).stop 2024-09-16T15:44:09.884 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:09.880+0000 7f5487034640 1 -- 172.21.15.123:0/4259237823 shutdown_connections 2024-09-16T15:44:09.884 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:09.880+0000 7f5487034640 1 --2- 172.21.15.123:0/4259237823 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f54800ccad0 0x7f548015e9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:09.884 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:09.880+0000 7f5487034640 1 --2- 172.21.15.123:0/4259237823 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f548015a9d0 0x7f548006ebe0 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:09.885 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:09.880+0000 7f5487034640 1 -- 172.21.15.123:0/4259237823 >> 172.21.15.123:0/4259237823 conn(0x7f54800b1710 msgr2=0x7f54800b1b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:44:09.885 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:09.880+0000 7f5487034640 1 -- 172.21.15.123:0/4259237823 shutdown_connections 2024-09-16T15:44:09.885 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:09.880+0000 7f5487034640 1 -- 172.21.15.123:0/4259237823 wait complete. 2024-09-16T15:44:09.885 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:09.880+0000 7f5487034640 1 Processor -- start 2024-09-16T15:44:09.885 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:09.880+0000 7f5487034640 1 -- start start 2024-09-16T15:44:09.887 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:09.880+0000 7f5487034640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5480072c70 0x7f548014e140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:44:09.887 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:09.880+0000 7f5487034640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f54800ccad0 0x7f54800b4690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:44:09.887 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:09.880+0000 7f5487034640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f548015a9d0 0x7f54800b0740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:44:09.887 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:09.880+0000 7f5487034640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f5480167e90 con 0x7f5480072c70 2024-09-16T15:44:09.887 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:09.880+0000 7f5487034640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f5480167d10 con 0x7f548015a9d0 2024-09-16T15:44:09.888 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:09.880+0000 7f5487034640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f5480168010 con 0x7f54800ccad0 2024-09-16T15:44:09.888 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:09.880+0000 7f5484dd0640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5480072c70 0x7f548014e140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:44:09.888 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:09.880+0000 7f5484dd0640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5480072c70 0x7f548014e140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:58026/0 (socket says 172.21.15.123:58026) 2024-09-16T15:44:09.888 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:09.880+0000 7f5484dd0640 1 -- 172.21.15.123:0/3287294140 learned_addr learned my addr 172.21.15.123:0/3287294140 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:44:09.888 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:09.880+0000 7f5484dd0640 1 -- 172.21.15.123:0/3287294140 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f54800ccad0 msgr2=0x7f54800b4690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:44:09.888 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:09.880+0000 7f5484dd0640 1 --2- 172.21.15.123:0/3287294140 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f54800ccad0 0x7f54800b4690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:09.888 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:09.880+0000 7f5484dd0640 1 -- 172.21.15.123:0/3287294140 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f548015a9d0 msgr2=0x7f54800b0740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:44:09.888 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:09.880+0000 7f5484dd0640 1 --2- 172.21.15.123:0/3287294140 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f548015a9d0 0x7f54800b0740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:09.888 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:09.880+0000 7f5484dd0640 1 -- 172.21.15.123:0/3287294140 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f54800b5310 con 0x7f5480072c70 2024-09-16T15:44:09.888 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:09.880+0000 7f5484dd0640 1 --2- 172.21.15.123:0/3287294140 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5480072c70 0x7f548014e140 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f5470009b10 tx=0x7f5470009200 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:44:09.888 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:09.880+0000 7f547d7fa640 1 -- 172.21.15.123:0/3287294140 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54700b2040 con 0x7f5480072c70 2024-09-16T15:44:09.888 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:09.880+0000 7f547d7fa640 1 -- 172.21.15.123:0/3287294140 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f5470004d40 con 0x7f5480072c70 2024-09-16T15:44:09.889 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:09.880+0000 7f5487034640 1 -- 172.21.15.123:0/3287294140 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f54800b55f0 con 0x7f5480072c70 2024-09-16T15:44:09.889 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:09.880+0000 7f547d7fa640 1 -- 172.21.15.123:0/3287294140 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5470004260 con 0x7f5480072c70 2024-09-16T15:44:09.889 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:09.880+0000 7f5487034640 1 -- 172.21.15.123:0/3287294140 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5480155710 con 0x7f5480072c70 2024-09-16T15:44:09.889 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:09.884+0000 7f5487034640 1 -- 172.21.15.123:0/3287294140 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5480155500 con 0x7f5480072c70 2024-09-16T15:44:09.890 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:09.884+0000 7f547d7fa640 1 -- 172.21.15.123:0/3287294140 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 7) ==== 97233+0+0 (secure 0 0 0) 0x7f54700bd060 con 0x7f5480072c70 2024-09-16T15:44:09.890 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:09.884+0000 7f547d7fa640 1 --2- 172.21.15.123:0/3287294140 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f546c073070 0x7f546c075530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:44:09.890 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:09.884+0000 7f547d7fa640 1 -- 172.21.15.123:0/3287294140 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(36..36 src has 1..36) ==== 4943+0+0 (secure 0 0 0) 0x7f547012db80 con 0x7f5480072c70 2024-09-16T15:44:09.890 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:09.884+0000 7f547ffff640 1 --2- 172.21.15.123:0/3287294140 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f546c073070 0x7f546c075530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:44:09.890 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:09.884+0000 7f547ffff640 1 --2- 172.21.15.123:0/3287294140 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f546c073070 0x7f546c075530 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f54740046c0 tx=0x7f5474007910 comp rx=0 tx=0).ready entity=mgr.4135 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:44:09.892 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:09.888+0000 7f547d7fa640 1 -- 172.21.15.123:0/3287294140 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f5470016790 con 0x7f5480072c70 2024-09-16T15:44:10.058 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:10.052+0000 7f5487034640 1 -- 172.21.15.123:0/3287294140 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) -- 0x7f548015e9c0 con 0x7f5480072c70 2024-09-16T15:44:10.064 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:10.060+0000 7f547d7fa640 1 -- 172.21.15.123:0/3287294140 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v30)=0 v30) ==== 122+0+0 (secure 0 0 0) 0x7f54700f8dd0 con 0x7f5480072c70 2024-09-16T15:44:10.066 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:10.060+0000 7f5487034640 1 -- 172.21.15.123:0/3287294140 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f546c073070 msgr2=0x7f546c075530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:44:10.066 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:10.060+0000 7f5487034640 1 --2- 172.21.15.123:0/3287294140 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f546c073070 0x7f546c075530 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f54740046c0 tx=0x7f5474007910 comp rx=0 tx=0).stop 2024-09-16T15:44:10.066 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:10.060+0000 7f5487034640 1 -- 172.21.15.123:0/3287294140 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5480072c70 msgr2=0x7f548014e140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:44:10.066 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:10.060+0000 7f5487034640 1 --2- 172.21.15.123:0/3287294140 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5480072c70 0x7f548014e140 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f5470009b10 tx=0x7f5470009200 comp rx=0 tx=0).stop 2024-09-16T15:44:10.067 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:10.060+0000 7f5487034640 1 -- 172.21.15.123:0/3287294140 shutdown_connections 2024-09-16T15:44:10.067 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:10.060+0000 7f5487034640 1 --2- 172.21.15.123:0/3287294140 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f546c073070 0x7f546c075530 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:10.067 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:10.060+0000 7f5487034640 1 --2- 172.21.15.123:0/3287294140 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f548015a9d0 0x7f54800b0740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:10.067 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:10.060+0000 7f5487034640 1 --2- 172.21.15.123:0/3287294140 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f54800ccad0 0x7f54800b4690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:10.067 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:10.060+0000 7f5487034640 1 --2- 172.21.15.123:0/3287294140 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5480072c70 0x7f548014e140 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:10.067 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:10.060+0000 7f5487034640 1 -- 172.21.15.123:0/3287294140 >> 172.21.15.123:0/3287294140 conn(0x7f54800b1710 msgr2=0x7f548015b9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:44:10.067 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:10.060+0000 7f5487034640 1 -- 172.21.15.123:0/3287294140 shutdown_connections 2024-09-16T15:44:10.067 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:10.060+0000 7f5487034640 1 -- 172.21.15.123:0/3287294140 wait complete. 2024-09-16T15:44:10.081 INFO:teuthology.nuke:Clearing teuthology firewall rules... 2024-09-16T15:44:10.082 DEBUG:teuthology.orchestra.run.smithi123:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-09-16T15:44:10.145 DEBUG:teuthology.orchestra.run.smithi184:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-09-16T15:44:10.228 INFO:teuthology.nuke:Cleared teuthology firewall rules. 2024-09-16T15:44:10.229 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2024-09-16T15:44:10.229 DEBUG:teuthology.orchestra.run.smithi123:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-16T15:44:10.234 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16336468k,nr_inodes=4084117,mode=755,inode64 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3274684k,mode=755,inode64 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev,inode64 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=17082 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3274680k,nr_inodes=818670,mode=700,uid=1000,gid=1271,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3274684k,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-09-16T15:44:10.234 DEBUG:teuthology.orchestra.run.smithi184:> set -ex 2024-09-16T15:44:10.235 DEBUG:teuthology.orchestra.run.smithi184:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-16T15:44:10.240 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16336488k,nr_inodes=4084122,mode=755,inode64 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3274688k,mode=755,inode64 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev,inode64 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=17834 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3274684k,nr_inodes=818671,mode=700,uid=1000,gid=1271,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3274688k,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-09-16T15:44:10.240 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2024-09-16T15:44:10.240 DEBUG:teuthology.orchestra.run.smithi123:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-16T15:44:10.284 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16336468k,nr_inodes=4084117,mode=755,inode64 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3274684k,mode=755,inode64 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev,inode64 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=17082 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3274680k,nr_inodes=818670,mode=700,uid=1000,gid=1271,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3274684k,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-09-16T15:44:10.284 DEBUG:teuthology.orchestra.run.smithi184:> set -ex 2024-09-16T15:44:10.284 DEBUG:teuthology.orchestra.run.smithi184:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-16T15:44:10.290 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16336488k,nr_inodes=4084122,mode=755,inode64 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3274688k,mode=755,inode64 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev,inode64 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=17834 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3274684k,nr_inodes=818671,mode=700,uid=1000,gid=1271,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3274688k,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-09-16T15:44:10.290 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-09-16T15:44:10.290 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-16T15:44:10.457 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:10.452+0000 7fbc2277c640 1 -- 172.21.15.123:0/3166406296 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbc1c0cc8b0 msgr2=0x7fbc1c0cc090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:44:10.457 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:10.452+0000 7fbc2277c640 1 --2- 172.21.15.123:0/3166406296 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbc1c0cc8b0 0x7fbc1c0cc090 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7fbc1c072770 tx=0x7fbc1401cd80 comp rx=0 tx=0).stop 2024-09-16T15:44:10.457 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:10.452+0000 7fbc2277c640 1 -- 172.21.15.123:0/3166406296 shutdown_connections 2024-09-16T15:44:10.457 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:10.452+0000 7fbc2277c640 1 --2- 172.21.15.123:0/3166406296 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbc1c0cc8b0 0x7fbc1c0cc090 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:10.457 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:10.452+0000 7fbc2277c640 1 --2- 172.21.15.123:0/3166406296 >> v2:172.21.15.184:3300/0 conn(0x7fbc1c0ca590 0x7fbc1c0c74d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:10.457 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:10.452+0000 7fbc2277c640 1 -- 172.21.15.123:0/3166406296 >> 172.21.15.123:0/3166406296 conn(0x7fbc1c074230 msgr2=0x7fbc1c074640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:44:10.458 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:10.452+0000 7fbc2277c640 1 -- 172.21.15.123:0/3166406296 shutdown_connections 2024-09-16T15:44:10.458 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:10.452+0000 7fbc2277c640 1 -- 172.21.15.123:0/3166406296 wait complete. 2024-09-16T15:44:10.460 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:10.452+0000 7fbc2277c640 1 Processor -- start 2024-09-16T15:44:10.460 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:10.452+0000 7fbc2277c640 1 -- start start 2024-09-16T15:44:10.460 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:10.452+0000 7fbc2277c640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbc1c0ca590 0x7fbc1c1f8240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:44:10.460 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:10.452+0000 7fbc2277c640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fbc1c0cc8b0 0x7fbc1c158850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:44:10.460 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:10.452+0000 7fbc2277c640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fbc1c0cd070 0x7fbc1c15c060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:44:10.460 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:10.452+0000 7fbc2277c640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fbc1c1681c0 con 0x7fbc1c0ca590 2024-09-16T15:44:10.460 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:10.452+0000 7fbc2277c640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fbc1c15fd50 con 0x7fbc1c0cc8b0 2024-09-16T15:44:10.460 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:10.452+0000 7fbc2277c640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fbc1c168340 con 0x7fbc1c0cd070 2024-09-16T15:44:10.460 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:10.452+0000 7fbc1bfff640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbc1c0ca590 0x7fbc1c1f8240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:44:10.460 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:10.452+0000 7fbc1bfff640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbc1c0ca590 0x7fbc1c1f8240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:58046/0 (socket says 172.21.15.123:58046) 2024-09-16T15:44:10.461 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:10.452+0000 7fbc1bfff640 1 -- 172.21.15.123:0/906663053 learned_addr learned my addr 172.21.15.123:0/906663053 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:44:10.461 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:10.452+0000 7fbc20d19640 1 --2- 172.21.15.123:0/906663053 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fbc1c0cd070 0x7fbc1c15c060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:44:10.461 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:10.452+0000 7fbc1b7fe640 1 --2- 172.21.15.123:0/906663053 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fbc1c0cc8b0 0x7fbc1c158850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:44:10.461 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:10.456+0000 7fbc1bfff640 1 -- 172.21.15.123:0/906663053 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fbc1c0cd070 msgr2=0x7fbc1c15c060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:44:10.461 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:10.456+0000 7fbc1bfff640 1 --2- 172.21.15.123:0/906663053 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fbc1c0cd070 0x7fbc1c15c060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:10.461 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:10.456+0000 7fbc1bfff640 1 -- 172.21.15.123:0/906663053 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fbc1c0cc8b0 msgr2=0x7fbc1c158850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:44:10.461 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:10.456+0000 7fbc1bfff640 1 --2- 172.21.15.123:0/906663053 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fbc1c0cc8b0 0x7fbc1c158850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:10.461 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:10.456+0000 7fbc1bfff640 1 -- 172.21.15.123:0/906663053 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbc1c0c7cd0 con 0x7fbc1c0ca590 2024-09-16T15:44:10.461 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:10.456+0000 7fbc1b7fe640 1 --2- 172.21.15.123:0/906663053 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fbc1c0cc8b0 0x7fbc1c158850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T15:44:10.461 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:10.456+0000 7fbc1bfff640 1 --2- 172.21.15.123:0/906663053 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbc1c0ca590 0x7fbc1c1f8240 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7fbc0c00c3c0 tx=0x7fbc0c00c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:44:10.461 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:10.456+0000 7fbc18ff9640 1 -- 172.21.15.123:0/906663053 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc0c0052d0 con 0x7fbc1c0ca590 2024-09-16T15:44:10.462 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:10.456+0000 7fbc18ff9640 1 -- 172.21.15.123:0/906663053 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fbc0c007020 con 0x7fbc1c0ca590 2024-09-16T15:44:10.462 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:10.456+0000 7fbc2277c640 1 -- 172.21.15.123:0/906663053 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbc1c0c7f70 con 0x7fbc1c0ca590 2024-09-16T15:44:10.462 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:10.456+0000 7fbc18ff9640 1 -- 172.21.15.123:0/906663053 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc0c007470 con 0x7fbc1c0ca590 2024-09-16T15:44:10.462 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:10.456+0000 7fbc2277c640 1 -- 172.21.15.123:0/906663053 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbc1c1f7d20 con 0x7fbc1c0ca590 2024-09-16T15:44:10.463 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:10.456+0000 7fbc2277c640 1 -- 172.21.15.123:0/906663053 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbc1c0b15d0 con 0x7fbc1c0ca590 2024-09-16T15:44:10.463 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:10.456+0000 7fbc18ff9640 1 -- 172.21.15.123:0/906663053 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 7) ==== 97233+0+0 (secure 0 0 0) 0x7fbc0c0067e0 con 0x7fbc1c0ca590 2024-09-16T15:44:10.463 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:10.456+0000 7fbc18ff9640 1 --2- 172.21.15.123:0/906663053 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fbc08073070 0x7fbc08075530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:44:10.463 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:10.456+0000 7fbc18ff9640 1 -- 172.21.15.123:0/906663053 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(36..36 src has 1..36) ==== 4943+0+0 (secure 0 0 0) 0x7fbc0c0933a0 con 0x7fbc1c0ca590 2024-09-16T15:44:10.463 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:10.456+0000 7fbc1b7fe640 1 --2- 172.21.15.123:0/906663053 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fbc08073070 0x7fbc08075530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:44:10.463 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:10.456+0000 7fbc1b7fe640 1 --2- 172.21.15.123:0/906663053 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fbc08073070 0x7fbc08075530 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fbc10005fd0 tx=0x7fbc10005e80 comp rx=0 tx=0).ready entity=mgr.4135 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:44:10.466 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:10.460+0000 7fbc18ff9640 1 -- 172.21.15.123:0/906663053 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7fbc0c05e560 con 0x7fbc1c0ca590 2024-09-16T15:44:10.675 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:10.672+0000 7fbc2277c640 1 -- 172.21.15.123:0/906663053 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fbc1c0c74b0 con 0x7fbc1c0ca590 2024-09-16T15:44:10.676 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:10.672+0000 7fbc18ff9640 1 -- 172.21.15.123:0/906663053 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 21 v21) ==== 94+0+3920 (secure 0 0 0) 0x7fbc0c001090 con 0x7fbc1c0ca590 2024-09-16T15:44:10.676 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-09-16T15:44:10.676 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":21,"btime":"2024-09-16T15:44:04:077352+0000","default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4541,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6837/249053959","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6836","nonce":249053959},{"type":"v1","addr":"172.21.15.184:6837","nonce":249053959}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":13},{"gid":4705,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6835/1285950902","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":1285950902},{"type":"v1","addr":"172.21.15.184:6835","nonce":1285950902}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":4866,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6833/1030632114","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6832","nonce":1030632114},{"type":"v1","addr":"172.21.15.123:6833","nonce":1030632114}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":5181,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6835/2628436560","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6834","nonce":2628436560},{"type":"v1","addr":"172.21.15.123:6835","nonce":2628436560}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":21}],"filesystems":[]} 2024-09-16T15:44:10.677 INFO:teuthology.orchestra.run.smithi123.stderr:dumped fsmap epoch 21 2024-09-16T15:44:10.679 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:10.672+0000 7fbc2277c640 1 -- 172.21.15.123:0/906663053 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fbc08073070 msgr2=0x7fbc08075530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:44:10.679 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:10.672+0000 7fbc2277c640 1 --2- 172.21.15.123:0/906663053 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fbc08073070 0x7fbc08075530 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fbc10005fd0 tx=0x7fbc10005e80 comp rx=0 tx=0).stop 2024-09-16T15:44:10.679 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:10.672+0000 7fbc2277c640 1 -- 172.21.15.123:0/906663053 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbc1c0ca590 msgr2=0x7fbc1c1f8240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:44:10.679 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:10.672+0000 7fbc2277c640 1 --2- 172.21.15.123:0/906663053 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbc1c0ca590 0x7fbc1c1f8240 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7fbc0c00c3c0 tx=0x7fbc0c00c880 comp rx=0 tx=0).stop 2024-09-16T15:44:10.679 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:10.672+0000 7fbc2277c640 1 -- 172.21.15.123:0/906663053 shutdown_connections 2024-09-16T15:44:10.679 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:10.672+0000 7fbc2277c640 1 --2- 172.21.15.123:0/906663053 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fbc08073070 0x7fbc08075530 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:10.680 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:10.672+0000 7fbc2277c640 1 --2- 172.21.15.123:0/906663053 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fbc1c0cd070 0x7fbc1c15c060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:10.680 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:10.672+0000 7fbc2277c640 1 --2- 172.21.15.123:0/906663053 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fbc1c0cc8b0 0x7fbc1c158850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:10.680 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:10.672+0000 7fbc2277c640 1 --2- 172.21.15.123:0/906663053 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbc1c0ca590 0x7fbc1c1f8240 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:10.680 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:10.672+0000 7fbc2277c640 1 -- 172.21.15.123:0/906663053 >> 172.21.15.123:0/906663053 conn(0x7fbc1c074230 msgr2=0x7fbc1c0cb980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:44:10.680 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:10.672+0000 7fbc2277c640 1 -- 172.21.15.123:0/906663053 shutdown_connections 2024-09-16T15:44:10.680 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:10.672+0000 7fbc2277c640 1 -- 172.21.15.123:0/906663053 wait complete. 2024-09-16T15:44:10.696 INFO:tasks.ceph.mds.a:Restarting daemon 2024-09-16T15:44:10.696 INFO:tasks.ceph.mds.a:Stopping old one... 2024-09-16T15:44:10.696 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-09-16T15:44:10.696 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T15:44:10.696 INFO:tasks.ceph.mds.c:Restarting daemon 2024-09-16T15:44:10.697 INFO:tasks.ceph.mds.c:Stopping old one... 2024-09-16T15:44:10.697 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-09-16T15:44:10.697 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T15:44:10.697 INFO:tasks.ceph.mds.b:Restarting daemon 2024-09-16T15:44:10.697 INFO:tasks.ceph.mds.b:Stopping old one... 2024-09-16T15:44:10.697 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-09-16T15:44:10.698 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T15:44:10.698 INFO:tasks.ceph.mds.d:Restarting daemon 2024-09-16T15:44:10.698 INFO:tasks.ceph.mds.d:Stopping old one... 2024-09-16T15:44:10.698 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-09-16T15:44:10.698 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T15:44:10.710 INFO:tasks.ceph.mds.a:Stopped 2024-09-16T15:44:10.711 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-09-16T15:44:10.711 INFO:tasks.ceph.mds.c:Stopped 2024-09-16T15:44:10.712 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-09-16T15:44:10.713 INFO:tasks.ceph.mds.d:Stopped 2024-09-16T15:44:10.713 DEBUG:teuthology.orchestra.run.smithi184:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-09-16T15:44:10.715 INFO:tasks.ceph.mds.b:Stopped 2024-09-16T15:44:10.715 DEBUG:teuthology.orchestra.run.smithi184:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-09-16T15:44:10.756 INFO:tasks.ceph.mds.a:Started 2024-09-16T15:44:10.757 DEBUG:teuthology.parallel:result is None 2024-09-16T15:44:10.757 INFO:tasks.ceph.mds.c:Started 2024-09-16T15:44:10.757 DEBUG:teuthology.parallel:result is None 2024-09-16T15:44:10.760 INFO:tasks.ceph.mds.d:Started 2024-09-16T15:44:10.760 DEBUG:teuthology.parallel:result is None 2024-09-16T15:44:10.761 INFO:tasks.ceph.mds.b:Started 2024-09-16T15:44:10.761 DEBUG:teuthology.parallel:result is None 2024-09-16T15:44:10.761 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-09-16T15:44:10.816 INFO:tasks.ceph.mds.a.smithi123.stdout:starting mds.a at 2024-09-16T15:44:10.817 INFO:tasks.ceph.mds.c.smithi123.stdout:starting mds.c at 2024-09-16T15:44:10.829 INFO:tasks.ceph.mds.d.smithi184.stdout:starting mds.d at 2024-09-16T15:44:10.831 INFO:tasks.ceph.mds.b.smithi184.stdout:starting mds.b at 2024-09-16T15:44:10.876 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:10.872+0000 7f4e99fc9640 1 -- 172.21.15.123:0/2172116433 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4e7c0051d0 msgr2=0x7f4e7c00b9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:44:10.876 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:10.872+0000 7f4e99fc9640 1 --2- 172.21.15.123:0/2172116433 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4e7c0051d0 0x7f4e7c00b9d0 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f4e8c001dc0 tx=0x7f4e8c01cb30 comp rx=0 tx=0).stop 2024-09-16T15:44:10.876 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:10.872+0000 7f4e99fc9640 1 -- 172.21.15.123:0/2172116433 shutdown_connections 2024-09-16T15:44:10.876 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:10.872+0000 7f4e99fc9640 1 --2- 172.21.15.123:0/2172116433 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4e7c0051d0 0x7f4e7c00b9d0 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:10.876 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:10.872+0000 7f4e99fc9640 1 -- 172.21.15.123:0/2172116433 >> 172.21.15.123:0/2172116433 conn(0x7f4e940cd5d0 msgr2=0x7f4e940cd9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:44:10.876 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:10.872+0000 7f4e99fc9640 1 -- 172.21.15.123:0/2172116433 shutdown_connections 2024-09-16T15:44:10.876 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:10.872+0000 7f4e99fc9640 1 -- 172.21.15.123:0/2172116433 wait complete. 2024-09-16T15:44:10.877 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:10.872+0000 7f4e99fc9640 1 Processor -- start 2024-09-16T15:44:10.877 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:10.872+0000 7f4e99fc9640 1 -- start start 2024-09-16T15:44:10.879 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:10.872+0000 7f4e99fc9640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4e7c0051d0 0x7f4e941f83b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:44:10.879 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:10.872+0000 7f4e99fc9640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4e940c6ea0 0x7f4e94159e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:44:10.879 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:10.872+0000 7f4e99fc9640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f4e941521c0 0x7f4e94152640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:44:10.879 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:10.872+0000 7f4e99fc9640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f4e9415b120 con 0x7f4e7c0051d0 2024-09-16T15:44:10.879 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:10.872+0000 7f4e99fc9640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f4e940cbdc0 con 0x7f4e940c6ea0 2024-09-16T15:44:10.879 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:10.872+0000 7f4e99fc9640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f4e9415b2a0 con 0x7f4e941521c0 2024-09-16T15:44:10.879 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:10.872+0000 7f4e937fe640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4e7c0051d0 0x7f4e941f83b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:44:10.879 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:10.872+0000 7f4e937fe640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4e7c0051d0 0x7f4e941f83b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:58098/0 (socket says 172.21.15.123:58098) 2024-09-16T15:44:10.879 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:10.872+0000 7f4e937fe640 1 -- 172.21.15.123:0/4011524102 learned_addr learned my addr 172.21.15.123:0/4011524102 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:44:10.879 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:10.872+0000 7f4e93fff640 1 --2- 172.21.15.123:0/4011524102 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f4e941521c0 0x7f4e94152640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:44:10.879 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:10.872+0000 7f4e92ffd640 1 --2- 172.21.15.123:0/4011524102 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4e940c6ea0 0x7f4e94159e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:44:10.879 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:10.872+0000 7f4e937fe640 1 -- 172.21.15.123:0/4011524102 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f4e941521c0 msgr2=0x7f4e94152640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:44:10.880 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:10.872+0000 7f4e937fe640 1 --2- 172.21.15.123:0/4011524102 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f4e941521c0 0x7f4e94152640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:10.880 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:10.872+0000 7f4e937fe640 1 -- 172.21.15.123:0/4011524102 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4e940c6ea0 msgr2=0x7f4e94159e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:44:10.880 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:10.872+0000 7f4e937fe640 1 --2- 172.21.15.123:0/4011524102 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4e940c6ea0 0x7f4e94159e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:10.880 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:10.872+0000 7f4e937fe640 1 -- 172.21.15.123:0/4011524102 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4e941f7c20 con 0x7f4e7c0051d0 2024-09-16T15:44:10.880 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:10.872+0000 7f4e92ffd640 1 --2- 172.21.15.123:0/4011524102 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4e940c6ea0 0x7f4e94159e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T15:44:10.880 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:10.872+0000 7f4e937fe640 1 --2- 172.21.15.123:0/4011524102 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4e7c0051d0 0x7f4e941f83b0 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f4e880118d0 tx=0x7f4e88011d90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:44:10.880 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:10.872+0000 7f4e7bfff640 1 -- 172.21.15.123:0/4011524102 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e8801e070 con 0x7f4e7c0051d0 2024-09-16T15:44:10.880 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:10.872+0000 7f4e7bfff640 1 -- 172.21.15.123:0/4011524102 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f4e8800fbb0 con 0x7f4e7c0051d0 2024-09-16T15:44:10.880 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:10.872+0000 7f4e99fc9640 1 -- 172.21.15.123:0/4011524102 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4e941f4e30 con 0x7f4e7c0051d0 2024-09-16T15:44:10.880 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:10.872+0000 7f4e7bfff640 1 -- 172.21.15.123:0/4011524102 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e88019410 con 0x7f4e7c0051d0 2024-09-16T15:44:10.880 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:10.872+0000 7f4e99fc9640 1 -- 172.21.15.123:0/4011524102 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4e94152b80 con 0x7f4e7c0051d0 2024-09-16T15:44:10.881 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:10.876+0000 7f4e99fc9640 1 -- 172.21.15.123:0/4011524102 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4e94073690 con 0x7f4e7c0051d0 2024-09-16T15:44:10.881 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:10.876+0000 7f4e7bfff640 1 -- 172.21.15.123:0/4011524102 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 7) ==== 97233+0+0 (secure 0 0 0) 0x7f4e88021070 con 0x7f4e7c0051d0 2024-09-16T15:44:10.881 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:10.876+0000 7f4e7bfff640 1 --2- 172.21.15.123:0/4011524102 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f4e800730a0 0x7f4e80075560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:44:10.881 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:10.876+0000 7f4e7bfff640 1 -- 172.21.15.123:0/4011524102 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(36..36 src has 1..36) ==== 4943+0+0 (secure 0 0 0) 0x7f4e880946e0 con 0x7f4e7c0051d0 2024-09-16T15:44:10.881 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:10.876+0000 7f4e92ffd640 1 --2- 172.21.15.123:0/4011524102 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f4e800730a0 0x7f4e80075560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:44:10.882 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:10.876+0000 7f4e92ffd640 1 --2- 172.21.15.123:0/4011524102 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f4e800730a0 0x7f4e80075560 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f4e84000fc0 tx=0x7f4e840067b0 comp rx=0 tx=0).ready entity=mgr.4135 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:44:10.884 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:10.880+0000 7f4e7bfff640 1 -- 172.21.15.123:0/4011524102 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f4e8805f250 con 0x7f4e7c0051d0 2024-09-16T15:44:10.917 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~0s 2024-09-16T15:44:10.917 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.c is failed for ~0s 2024-09-16T15:44:10.917 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~0s 2024-09-16T15:44:11.060 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:11.056+0000 7f4e99fc9640 1 -- 172.21.15.123:0/4011524102 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) -- 0x7f4e940b5c10 con 0x7f4e7c0051d0 2024-09-16T15:44:11.458 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:11.452+0000 7f4e7bfff640 1 -- 172.21.15.123:0/4011524102 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v38) ==== 97+0+0 (secure 0 0 0) 0x7f4e88063c80 con 0x7f4e7c0051d0 2024-09-16T15:44:11.458 INFO:teuthology.orchestra.run.smithi123.stderr: removed all blocklist entries 2024-09-16T15:44:11.463 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:11.456+0000 7f4e99fc9640 1 -- 172.21.15.123:0/4011524102 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f4e800730a0 msgr2=0x7f4e80075560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:44:11.463 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:11.456+0000 7f4e99fc9640 1 --2- 172.21.15.123:0/4011524102 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f4e800730a0 0x7f4e80075560 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f4e84000fc0 tx=0x7f4e840067b0 comp rx=0 tx=0).stop 2024-09-16T15:44:11.463 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:11.456+0000 7f4e99fc9640 1 -- 172.21.15.123:0/4011524102 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4e7c0051d0 msgr2=0x7f4e941f83b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:44:11.463 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:11.456+0000 7f4e99fc9640 1 --2- 172.21.15.123:0/4011524102 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4e7c0051d0 0x7f4e941f83b0 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f4e880118d0 tx=0x7f4e88011d90 comp rx=0 tx=0).stop 2024-09-16T15:44:11.463 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:11.456+0000 7f4e99fc9640 1 -- 172.21.15.123:0/4011524102 shutdown_connections 2024-09-16T15:44:11.463 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:11.456+0000 7f4e99fc9640 1 --2- 172.21.15.123:0/4011524102 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f4e800730a0 0x7f4e80075560 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:11.464 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:11.456+0000 7f4e99fc9640 1 --2- 172.21.15.123:0/4011524102 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f4e941521c0 0x7f4e94152640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:11.464 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:11.456+0000 7f4e99fc9640 1 --2- 172.21.15.123:0/4011524102 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4e940c6ea0 0x7f4e94159e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:11.464 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:11.456+0000 7f4e99fc9640 1 --2- 172.21.15.123:0/4011524102 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4e7c0051d0 0x7f4e941f83b0 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:11.464 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:11.456+0000 7f4e99fc9640 1 -- 172.21.15.123:0/4011524102 >> 172.21.15.123:0/4011524102 conn(0x7f4e940cd5d0 msgr2=0x7f4e940b4c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:44:11.464 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:11.456+0000 7f4e99fc9640 1 -- 172.21.15.123:0/4011524102 shutdown_connections 2024-09-16T15:44:11.464 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:11.460+0000 7f4e99fc9640 1 -- 172.21.15.123:0/4011524102 wait complete. 2024-09-16T15:44:11.483 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-09-16T15:44:11.600 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:11.596+0000 7f322810a640 1 -- 172.21.15.123:0/3780032212 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3220161620 msgr2=0x7f32200d1da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:44:11.600 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:11.596+0000 7f322810a640 1 --2- 172.21.15.123:0/3780032212 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3220161620 0x7f32200d1da0 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f321c005fd0 tx=0x7f321c01c6d0 comp rx=0 tx=0).stop 2024-09-16T15:44:11.600 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:11.596+0000 7f322810a640 1 -- 172.21.15.123:0/3780032212 shutdown_connections 2024-09-16T15:44:11.600 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:11.596+0000 7f322810a640 1 --2- 172.21.15.123:0/3780032212 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f32200b2b00 0x7f32200d22e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:11.600 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:11.596+0000 7f322810a640 1 --2- 172.21.15.123:0/3780032212 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3220161620 0x7f32200d1da0 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:11.600 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:11.596+0000 7f322810a640 1 -- 172.21.15.123:0/3780032212 >> 172.21.15.123:0/3780032212 conn(0x7f32200d3ea0 msgr2=0x7f32200d42b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:44:11.600 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:11.596+0000 7f322810a640 1 -- 172.21.15.123:0/3780032212 shutdown_connections 2024-09-16T15:44:11.600 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:11.596+0000 7f322810a640 1 -- 172.21.15.123:0/3780032212 wait complete. 2024-09-16T15:44:11.602 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:11.596+0000 7f322810a640 1 Processor -- start 2024-09-16T15:44:11.603 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:11.596+0000 7f322810a640 1 -- start start 2024-09-16T15:44:11.603 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:11.596+0000 7f322810a640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f32200b2b00 0x7f32200c8510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:44:11.603 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:11.596+0000 7f322810a640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f32200d2820 0x7f322015c050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:44:11.603 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:11.596+0000 7f322810a640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3220161620 0x7f32200c9390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:44:11.603 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:11.596+0000 7f322810a640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f322016aee0 con 0x7f32200d2820 2024-09-16T15:44:11.603 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:11.596+0000 7f322810a640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f322016ad60 con 0x7f3220161620 2024-09-16T15:44:11.603 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:11.596+0000 7f322810a640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f322016b060 con 0x7f32200b2b00 2024-09-16T15:44:11.603 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:11.596+0000 7f32256a5640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f32200d2820 0x7f322015c050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:44:11.603 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:11.596+0000 7f32256a5640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f32200d2820 0x7f322015c050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:58118/0 (socket says 172.21.15.123:58118) 2024-09-16T15:44:11.603 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:11.596+0000 7f32256a5640 1 -- 172.21.15.123:0/2830274921 learned_addr learned my addr 172.21.15.123:0/2830274921 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:44:11.603 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:11.596+0000 7f3225ea6640 1 --2- 172.21.15.123:0/2830274921 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f32200b2b00 0x7f32200c8510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:44:11.604 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:11.596+0000 7f32256a5640 1 -- 172.21.15.123:0/2830274921 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f32200b2b00 msgr2=0x7f32200c8510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:44:11.604 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:11.596+0000 7f32256a5640 1 --2- 172.21.15.123:0/2830274921 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f32200b2b00 0x7f32200c8510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:11.604 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:11.596+0000 7f32256a5640 1 -- 172.21.15.123:0/2830274921 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3220161620 msgr2=0x7f32200c9390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:44:11.604 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:11.596+0000 7f32256a5640 1 --2- 172.21.15.123:0/2830274921 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3220161620 0x7f32200c9390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:11.604 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:11.596+0000 7f32256a5640 1 -- 172.21.15.123:0/2830274921 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3220069970 con 0x7f32200d2820 2024-09-16T15:44:11.604 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:11.596+0000 7f32256a5640 1 --2- 172.21.15.123:0/2830274921 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f32200d2820 0x7f322015c050 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f321000c970 tx=0x7f321000ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:44:11.604 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:11.596+0000 7f32167fc640 1 -- 172.21.15.123:0/2830274921 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3210007bb0 con 0x7f32200d2820 2024-09-16T15:44:11.604 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:11.596+0000 7f32167fc640 1 -- 172.21.15.123:0/2830274921 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f3210007e40 con 0x7f32200d2820 2024-09-16T15:44:11.604 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:11.596+0000 7f322810a640 1 -- 172.21.15.123:0/2830274921 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f32200c7a70 con 0x7f32200d2820 2024-09-16T15:44:11.604 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:11.596+0000 7f32167fc640 1 -- 172.21.15.123:0/2830274921 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3210005260 con 0x7f32200d2820 2024-09-16T15:44:11.604 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:11.596+0000 7f322810a640 1 -- 172.21.15.123:0/2830274921 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f32200c9f50 con 0x7f32200d2820 2024-09-16T15:44:11.605 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:11.600+0000 7f322810a640 1 -- 172.21.15.123:0/2830274921 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f32200cea00 con 0x7f32200d2820 2024-09-16T15:44:11.605 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:11.600+0000 7f32167fc640 1 -- 172.21.15.123:0/2830274921 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 7) ==== 97233+0+0 (secure 0 0 0) 0x7f3210005400 con 0x7f32200d2820 2024-09-16T15:44:11.606 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:11.600+0000 7f32167fc640 1 --2- 172.21.15.123:0/2830274921 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f3204073070 0x7f3204075530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:44:11.606 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:11.600+0000 7f32167fc640 1 -- 172.21.15.123:0/2830274921 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(40..40 src has 1..40) ==== 4857+0+0 (secure 0 0 0) 0x7f32100946d0 con 0x7f32200d2820 2024-09-16T15:44:11.606 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:11.600+0000 7f3225ea6640 1 --2- 172.21.15.123:0/2830274921 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f3204073070 0x7f3204075530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:44:11.606 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:11.600+0000 7f3225ea6640 1 --2- 172.21.15.123:0/2830274921 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f3204073070 0x7f3204075530 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f321c0ae000 tx=0x7f321c01af10 comp rx=0 tx=0).ready entity=mgr.4135 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:44:11.608 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:11.604+0000 7f32167fc640 1 -- 172.21.15.123:0/2830274921 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f3210014030 con 0x7f32200d2820 2024-09-16T15:44:11.836 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:11.832+0000 7f322810a640 1 -- 172.21.15.123:0/2830274921 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) -- 0x7f322006dcf0 con 0x7f32200d2820 2024-09-16T15:44:11.836 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:11.832+0000 7f32167fc640 1 -- 172.21.15.123:0/2830274921 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v17) ==== 80+0+5304 (secure 0 0 0) 0x7f321005f7c0 con 0x7f32200d2820 2024-09-16T15:44:11.837 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-09-16T15:44:11.837 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2024-09-16T15:44:11.837 INFO:teuthology.orchestra.run.smithi123.stdout: "auth_dump": [ 2024-09-16T15:44:11.837 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-09-16T15:44:11.837 INFO:teuthology.orchestra.run.smithi123.stdout: "entity": "mds.a", 2024-09-16T15:44:11.837 INFO:teuthology.orchestra.run.smithi123.stdout: "key": "AQC0UehmbZpOJBAAaYGyDcWrt4k0FS5P/bb+4A==", 2024-09-16T15:44:11.837 INFO:teuthology.orchestra.run.smithi123.stdout: "caps": { 2024-09-16T15:44:11.837 INFO:teuthology.orchestra.run.smithi123.stdout: "mds": "allow", 2024-09-16T15:44:11.837 INFO:teuthology.orchestra.run.smithi123.stdout: "mgr": "allow *", 2024-09-16T15:44:11.837 INFO:teuthology.orchestra.run.smithi123.stdout: "mon": "allow *", 2024-09-16T15:44:11.837 INFO:teuthology.orchestra.run.smithi123.stdout: "osd": "allow *" 2024-09-16T15:44:11.837 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-16T15:44:11.838 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:44:11.838 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-09-16T15:44:11.838 INFO:teuthology.orchestra.run.smithi123.stdout: "entity": "mds.b", 2024-09-16T15:44:11.838 INFO:teuthology.orchestra.run.smithi123.stdout: "key": "AQC0UehmF1HFMhAADcvqXNKdHZMPAbLc+se4Aw==", 2024-09-16T15:44:11.838 INFO:teuthology.orchestra.run.smithi123.stdout: "caps": { 2024-09-16T15:44:11.838 INFO:teuthology.orchestra.run.smithi123.stdout: "mds": "allow", 2024-09-16T15:44:11.838 INFO:teuthology.orchestra.run.smithi123.stdout: "mgr": "allow *", 2024-09-16T15:44:11.838 INFO:teuthology.orchestra.run.smithi123.stdout: "mon": "allow *", 2024-09-16T15:44:11.838 INFO:teuthology.orchestra.run.smithi123.stdout: "osd": "allow *" 2024-09-16T15:44:11.838 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-16T15:44:11.838 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:44:11.838 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-09-16T15:44:11.839 INFO:teuthology.orchestra.run.smithi123.stdout: "entity": "mds.c", 2024-09-16T15:44:11.839 INFO:teuthology.orchestra.run.smithi123.stdout: "key": "AQC0UehmCBjCLBAAXd85J25r7cgrXeDCG7pahg==", 2024-09-16T15:44:11.839 INFO:teuthology.orchestra.run.smithi123.stdout: "caps": { 2024-09-16T15:44:11.839 INFO:teuthology.orchestra.run.smithi123.stdout: "mds": "allow", 2024-09-16T15:44:11.839 INFO:teuthology.orchestra.run.smithi123.stdout: "mgr": "allow *", 2024-09-16T15:44:11.839 INFO:teuthology.orchestra.run.smithi123.stdout: "mon": "allow *", 2024-09-16T15:44:11.839 INFO:teuthology.orchestra.run.smithi123.stdout: "osd": "allow *" 2024-09-16T15:44:11.839 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-16T15:44:11.839 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:44:11.839 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-09-16T15:44:11.839 INFO:teuthology.orchestra.run.smithi123.stdout: "entity": "mds.d", 2024-09-16T15:44:11.839 INFO:teuthology.orchestra.run.smithi123.stdout: "key": "AQC0UehmMh3ZOBAAx0RCa3R9RMT93/icZhPGyg==", 2024-09-16T15:44:11.839 INFO:teuthology.orchestra.run.smithi123.stdout: "caps": { 2024-09-16T15:44:11.839 INFO:teuthology.orchestra.run.smithi123.stdout: "mds": "allow", 2024-09-16T15:44:11.839 INFO:teuthology.orchestra.run.smithi123.stdout: "mgr": "allow *", 2024-09-16T15:44:11.840 INFO:teuthology.orchestra.run.smithi123.stdout: "mon": "allow *", 2024-09-16T15:44:11.840 INFO:teuthology.orchestra.run.smithi123.stdout: "osd": "allow *" 2024-09-16T15:44:11.840 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-16T15:44:11.840 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:44:11.840 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-09-16T15:44:11.840 INFO:teuthology.orchestra.run.smithi123.stdout: "entity": "osd.0", 2024-09-16T15:44:11.840 INFO:teuthology.orchestra.run.smithi123.stdout: "key": "AQC4UehmT633CRAAIhpkeIBOzwJO0kiXLu0x9Q==", 2024-09-16T15:44:11.840 INFO:teuthology.orchestra.run.smithi123.stdout: "caps": { 2024-09-16T15:44:11.840 INFO:teuthology.orchestra.run.smithi123.stdout: "mgr": "allow profile osd", 2024-09-16T15:44:11.840 INFO:teuthology.orchestra.run.smithi123.stdout: "mon": "allow profile osd", 2024-09-16T15:44:11.840 INFO:teuthology.orchestra.run.smithi123.stdout: "osd": "allow *" 2024-09-16T15:44:11.840 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-16T15:44:11.840 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:44:11.841 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-09-16T15:44:11.841 INFO:teuthology.orchestra.run.smithi123.stdout: "entity": "osd.1", 2024-09-16T15:44:11.841 INFO:teuthology.orchestra.run.smithi123.stdout: "key": "AQC7UehmFwF7GBAAvOUG+rWjANKT8wm/ai4xGg==", 2024-09-16T15:44:11.841 INFO:teuthology.orchestra.run.smithi123.stdout: "caps": { 2024-09-16T15:44:11.841 INFO:teuthology.orchestra.run.smithi123.stdout: "mgr": "allow profile osd", 2024-09-16T15:44:11.841 INFO:teuthology.orchestra.run.smithi123.stdout: "mon": "allow profile osd", 2024-09-16T15:44:11.841 INFO:teuthology.orchestra.run.smithi123.stdout: "osd": "allow *" 2024-09-16T15:44:11.841 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-16T15:44:11.841 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:44:11.841 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-09-16T15:44:11.841 INFO:teuthology.orchestra.run.smithi123.stdout: "entity": "osd.2", 2024-09-16T15:44:11.841 INFO:teuthology.orchestra.run.smithi123.stdout: "key": "AQC+UehmlCwWKxAAl3iO7ZALWyaorQH2urHKCw==", 2024-09-16T15:44:11.841 INFO:teuthology.orchestra.run.smithi123.stdout: "caps": { 2024-09-16T15:44:11.841 INFO:teuthology.orchestra.run.smithi123.stdout: "mgr": "allow profile osd", 2024-09-16T15:44:11.842 INFO:teuthology.orchestra.run.smithi123.stdout: "mon": "allow profile osd", 2024-09-16T15:44:11.842 INFO:teuthology.orchestra.run.smithi123.stdout: "osd": "allow *" 2024-09-16T15:44:11.842 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-16T15:44:11.842 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:44:11.842 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-09-16T15:44:11.842 INFO:teuthology.orchestra.run.smithi123.stdout: "entity": "osd.3", 2024-09-16T15:44:11.842 INFO:teuthology.orchestra.run.smithi123.stdout: "key": "AQDCUehmD8VgAhAAneoIeGr3MflSmi+nyavj2A==", 2024-09-16T15:44:11.842 INFO:teuthology.orchestra.run.smithi123.stdout: "caps": { 2024-09-16T15:44:11.842 INFO:teuthology.orchestra.run.smithi123.stdout: "mgr": "allow profile osd", 2024-09-16T15:44:11.842 INFO:teuthology.orchestra.run.smithi123.stdout: "mon": "allow profile osd", 2024-09-16T15:44:11.842 INFO:teuthology.orchestra.run.smithi123.stdout: "osd": "allow *" 2024-09-16T15:44:11.842 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-16T15:44:11.842 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:44:11.842 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-09-16T15:44:11.842 INFO:teuthology.orchestra.run.smithi123.stdout: "entity": "osd.4", 2024-09-16T15:44:11.843 INFO:teuthology.orchestra.run.smithi123.stdout: "key": "AQDIUehmA9B9EBAAsu0X5LPx1TmatZeHzjnVig==", 2024-09-16T15:44:11.843 INFO:teuthology.orchestra.run.smithi123.stdout: "caps": { 2024-09-16T15:44:11.843 INFO:teuthology.orchestra.run.smithi123.stdout: "mgr": "allow profile osd", 2024-09-16T15:44:11.843 INFO:teuthology.orchestra.run.smithi123.stdout: "mon": "allow profile osd", 2024-09-16T15:44:11.843 INFO:teuthology.orchestra.run.smithi123.stdout: "osd": "allow *" 2024-09-16T15:44:11.843 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-16T15:44:11.843 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:44:11.843 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-09-16T15:44:11.843 INFO:teuthology.orchestra.run.smithi123.stdout: "entity": "osd.5", 2024-09-16T15:44:11.843 INFO:teuthology.orchestra.run.smithi123.stdout: "key": "AQDLUehmcugdHxAAQMSS1ZH71FUzCx5A+yNnZg==", 2024-09-16T15:44:11.843 INFO:teuthology.orchestra.run.smithi123.stdout: "caps": { 2024-09-16T15:44:11.843 INFO:teuthology.orchestra.run.smithi123.stdout: "mgr": "allow profile osd", 2024-09-16T15:44:11.843 INFO:teuthology.orchestra.run.smithi123.stdout: "mon": "allow profile osd", 2024-09-16T15:44:11.843 INFO:teuthology.orchestra.run.smithi123.stdout: "osd": "allow *" 2024-09-16T15:44:11.844 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-16T15:44:11.844 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:44:11.844 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-09-16T15:44:11.844 INFO:teuthology.orchestra.run.smithi123.stdout: "entity": "osd.6", 2024-09-16T15:44:11.844 INFO:teuthology.orchestra.run.smithi123.stdout: "key": "AQDOUehmWLwYLhAAhOlQLepcWElsVPMnHlZm4w==", 2024-09-16T15:44:11.844 INFO:teuthology.orchestra.run.smithi123.stdout: "caps": { 2024-09-16T15:44:11.844 INFO:teuthology.orchestra.run.smithi123.stdout: "mgr": "allow profile osd", 2024-09-16T15:44:11.844 INFO:teuthology.orchestra.run.smithi123.stdout: "mon": "allow profile osd", 2024-09-16T15:44:11.844 INFO:teuthology.orchestra.run.smithi123.stdout: "osd": "allow *" 2024-09-16T15:44:11.844 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-16T15:44:11.844 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:44:11.844 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-09-16T15:44:11.844 INFO:teuthology.orchestra.run.smithi123.stdout: "entity": "osd.7", 2024-09-16T15:44:11.844 INFO:teuthology.orchestra.run.smithi123.stdout: "key": "AQDSUehmipkbABAAQuL/2O7w0tsM1OKeOil97w==", 2024-09-16T15:44:11.845 INFO:teuthology.orchestra.run.smithi123.stdout: "caps": { 2024-09-16T15:44:11.845 INFO:teuthology.orchestra.run.smithi123.stdout: "mgr": "allow profile osd", 2024-09-16T15:44:11.845 INFO:teuthology.orchestra.run.smithi123.stdout: "mon": "allow profile osd", 2024-09-16T15:44:11.845 INFO:teuthology.orchestra.run.smithi123.stdout: "osd": "allow *" 2024-09-16T15:44:11.845 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-16T15:44:11.845 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:44:11.845 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-09-16T15:44:11.845 INFO:teuthology.orchestra.run.smithi123.stdout: "entity": "client.0", 2024-09-16T15:44:11.845 INFO:teuthology.orchestra.run.smithi123.stdout: "key": "AQC1Uehm7GDTAhAAHWWVIyjVVnTeeozBR8DfFA==", 2024-09-16T15:44:11.845 INFO:teuthology.orchestra.run.smithi123.stdout: "caps": { 2024-09-16T15:44:11.845 INFO:teuthology.orchestra.run.smithi123.stdout: "mds": "allow", 2024-09-16T15:44:11.845 INFO:teuthology.orchestra.run.smithi123.stdout: "mon": "allow r", 2024-09-16T15:44:11.845 INFO:teuthology.orchestra.run.smithi123.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-09-16T15:44:11.845 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-16T15:44:11.846 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:44:11.846 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-09-16T15:44:11.846 INFO:teuthology.orchestra.run.smithi123.stdout: "entity": "client.1", 2024-09-16T15:44:11.846 INFO:teuthology.orchestra.run.smithi123.stdout: "key": "AQC1UehmFbaWChAAFtRmKucm+pbKtCNvRCTNqA==", 2024-09-16T15:44:11.846 INFO:teuthology.orchestra.run.smithi123.stdout: "caps": { 2024-09-16T15:44:11.846 INFO:teuthology.orchestra.run.smithi123.stdout: "mds": "allow", 2024-09-16T15:44:11.846 INFO:teuthology.orchestra.run.smithi123.stdout: "mgr": "allow r", 2024-09-16T15:44:11.846 INFO:teuthology.orchestra.run.smithi123.stdout: "mon": "allow rw", 2024-09-16T15:44:11.846 INFO:teuthology.orchestra.run.smithi123.stdout: "osd": "allow rwx" 2024-09-16T15:44:11.846 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-16T15:44:11.846 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:44:11.846 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-09-16T15:44:11.846 INFO:teuthology.orchestra.run.smithi123.stdout: "entity": "client.2", 2024-09-16T15:44:11.846 INFO:teuthology.orchestra.run.smithi123.stdout: "key": "AQC1Uehm1ADrBxAAW/8ayNHI6Wq/YlI4ivFZiw==", 2024-09-16T15:44:11.847 INFO:teuthology.orchestra.run.smithi123.stdout: "caps": { 2024-09-16T15:44:11.847 INFO:teuthology.orchestra.run.smithi123.stdout: "mds": "allow", 2024-09-16T15:44:11.847 INFO:teuthology.orchestra.run.smithi123.stdout: "mgr": "allow r", 2024-09-16T15:44:11.847 INFO:teuthology.orchestra.run.smithi123.stdout: "mon": "allow rw", 2024-09-16T15:44:11.847 INFO:teuthology.orchestra.run.smithi123.stdout: "osd": "allow rwx" 2024-09-16T15:44:11.847 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-16T15:44:11.847 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:44:11.847 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-09-16T15:44:11.847 INFO:teuthology.orchestra.run.smithi123.stdout: "entity": "client.3", 2024-09-16T15:44:11.847 INFO:teuthology.orchestra.run.smithi123.stdout: "key": "AQC1UehmLv/3DxAAuGloBxSuza95uajpfiZ9MQ==", 2024-09-16T15:44:11.847 INFO:teuthology.orchestra.run.smithi123.stdout: "caps": { 2024-09-16T15:44:11.847 INFO:teuthology.orchestra.run.smithi123.stdout: "mds": "allow", 2024-09-16T15:44:11.847 INFO:teuthology.orchestra.run.smithi123.stdout: "mgr": "allow r", 2024-09-16T15:44:11.848 INFO:teuthology.orchestra.run.smithi123.stdout: "mon": "allow rw", 2024-09-16T15:44:11.848 INFO:teuthology.orchestra.run.smithi123.stdout: "osd": "allow rwx" 2024-09-16T15:44:11.848 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-16T15:44:11.848 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:44:11.848 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-09-16T15:44:11.848 INFO:teuthology.orchestra.run.smithi123.stdout: "entity": "client.admin", 2024-09-16T15:44:11.848 INFO:teuthology.orchestra.run.smithi123.stdout: "key": "AQC0Uehm1ZDRDBAAhyQ1BmJg1Mw6wQ04I2CbFA==", 2024-09-16T15:44:11.848 INFO:teuthology.orchestra.run.smithi123.stdout: "caps": { 2024-09-16T15:44:11.848 INFO:teuthology.orchestra.run.smithi123.stdout: "mds": "allow *", 2024-09-16T15:44:11.848 INFO:teuthology.orchestra.run.smithi123.stdout: "mgr": "allow *", 2024-09-16T15:44:11.848 INFO:teuthology.orchestra.run.smithi123.stdout: "mon": "allow *", 2024-09-16T15:44:11.848 INFO:teuthology.orchestra.run.smithi123.stdout: "osd": "allow *" 2024-09-16T15:44:11.848 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-16T15:44:11.849 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:44:11.849 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-09-16T15:44:11.849 INFO:teuthology.orchestra.run.smithi123.stdout: "entity": "mgr.x", 2024-09-16T15:44:11.849 INFO:teuthology.orchestra.run.smithi123.stdout: "key": "AQC0UehmfPilIBAAERlBbduOgAU9ba0n/1tl3g==", 2024-09-16T15:44:11.849 INFO:teuthology.orchestra.run.smithi123.stdout: "caps": { 2024-09-16T15:44:11.849 INFO:teuthology.orchestra.run.smithi123.stdout: "mds": "allow *", 2024-09-16T15:44:11.849 INFO:teuthology.orchestra.run.smithi123.stdout: "mon": "allow profile mgr", 2024-09-16T15:44:11.849 INFO:teuthology.orchestra.run.smithi123.stdout: "osd": "allow *" 2024-09-16T15:44:11.849 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-16T15:44:11.849 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:44:11.849 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-09-16T15:44:11.849 INFO:teuthology.orchestra.run.smithi123.stdout: "entity": "mgr.y", 2024-09-16T15:44:11.849 INFO:teuthology.orchestra.run.smithi123.stdout: "key": "AQC0UehmkjrlHRAA+3xbQC+SPWGWW1gND7xcAg==", 2024-09-16T15:44:11.850 INFO:teuthology.orchestra.run.smithi123.stdout: "caps": { 2024-09-16T15:44:11.850 INFO:teuthology.orchestra.run.smithi123.stdout: "mds": "allow *", 2024-09-16T15:44:11.850 INFO:teuthology.orchestra.run.smithi123.stdout: "mon": "allow profile mgr", 2024-09-16T15:44:11.850 INFO:teuthology.orchestra.run.smithi123.stdout: "osd": "allow *" 2024-09-16T15:44:11.850 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-16T15:44:11.850 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-16T15:44:11.850 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-09-16T15:44:11.850 INFO:teuthology.orchestra.run.smithi123.stdout:} 2024-09-16T15:44:11.851 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:11.832+0000 7f322810a640 1 -- 172.21.15.123:0/2830274921 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f3204073070 msgr2=0x7f3204075530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:44:11.851 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:11.832+0000 7f322810a640 1 --2- 172.21.15.123:0/2830274921 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f3204073070 0x7f3204075530 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f321c0ae000 tx=0x7f321c01af10 comp rx=0 tx=0).stop 2024-09-16T15:44:11.851 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:11.832+0000 7f322810a640 1 -- 172.21.15.123:0/2830274921 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f32200d2820 msgr2=0x7f322015c050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:44:11.851 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:11.832+0000 7f322810a640 1 --2- 172.21.15.123:0/2830274921 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f32200d2820 0x7f322015c050 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f321000c970 tx=0x7f321000ce30 comp rx=0 tx=0).stop 2024-09-16T15:44:11.851 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:11.832+0000 7f322810a640 1 -- 172.21.15.123:0/2830274921 shutdown_connections 2024-09-16T15:44:11.851 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:11.832+0000 7f322810a640 1 --2- 172.21.15.123:0/2830274921 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f3204073070 0x7f3204075530 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:11.851 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:11.832+0000 7f322810a640 1 --2- 172.21.15.123:0/2830274921 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3220161620 0x7f32200c9390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:11.851 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:11.832+0000 7f322810a640 1 --2- 172.21.15.123:0/2830274921 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f32200d2820 0x7f322015c050 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:11.852 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:11.832+0000 7f322810a640 1 --2- 172.21.15.123:0/2830274921 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f32200b2b00 0x7f32200c8510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:11.852 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:11.832+0000 7f322810a640 1 -- 172.21.15.123:0/2830274921 >> 172.21.15.123:0/2830274921 conn(0x7f32200d3ea0 msgr2=0x7f322006e4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:44:11.852 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:11.832+0000 7f322810a640 1 -- 172.21.15.123:0/2830274921 shutdown_connections 2024-09-16T15:44:11.852 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:11.832+0000 7f322810a640 1 -- 172.21.15.123:0/2830274921 wait complete. 2024-09-16T15:44:11.854 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-09-16T15:44:11.973 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:11.968+0000 7f0561743640 1 -- 172.21.15.123:0/1358033521 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f055c158120 msgr2=0x7f055c0ca320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:44:11.973 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:11.968+0000 7f0561743640 1 --2- 172.21.15.123:0/1358033521 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f055c158120 0x7f055c0ca320 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f054c009900 tx=0x7f054c01c850 comp rx=0 tx=0).stop 2024-09-16T15:44:11.973 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:11.968+0000 7f0561743640 1 -- 172.21.15.123:0/1358033521 shutdown_connections 2024-09-16T15:44:11.973 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:11.968+0000 7f0561743640 1 --2- 172.21.15.123:0/1358033521 >> v2:172.21.15.184:3300/0 conn(0x7f055c06f5f0 0x7f055c15aa60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:11.973 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:11.968+0000 7f0561743640 1 --2- 172.21.15.123:0/1358033521 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f055c158120 0x7f055c0ca320 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:11.973 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:11.968+0000 7f0561743640 1 -- 172.21.15.123:0/1358033521 >> 172.21.15.123:0/1358033521 conn(0x7f055c0cfb10 msgr2=0x7f055c0cff20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:44:11.973 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:11.968+0000 7f0561743640 1 -- 172.21.15.123:0/1358033521 shutdown_connections 2024-09-16T15:44:11.973 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:11.968+0000 7f0561743640 1 -- 172.21.15.123:0/1358033521 wait complete. 2024-09-16T15:44:11.976 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:11.968+0000 7f0561743640 1 Processor -- start 2024-09-16T15:44:11.976 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:11.968+0000 7f0561743640 1 -- start start 2024-09-16T15:44:11.976 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:11.968+0000 7f0561743640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f055c06f5f0 0x7f055c073c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:44:11.976 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:11.968+0000 7f0561743640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f055c0cf5a0 0x7f055c0aec20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:44:11.976 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:11.968+0000 7f0561743640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f055c158120 0x7f055c1563e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:44:11.976 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:11.968+0000 7f0561743640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f055c167b70 con 0x7f055c06f5f0 2024-09-16T15:44:11.976 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:11.968+0000 7f0561743640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f055c1679f0 con 0x7f055c158120 2024-09-16T15:44:11.976 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:11.968+0000 7f0561743640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f055c167cf0 con 0x7f055c0cf5a0 2024-09-16T15:44:11.976 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:11.968+0000 7f055affd640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f055c06f5f0 0x7f055c073c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:44:11.976 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:11.968+0000 7f055affd640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f055c06f5f0 0x7f055c073c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:58128/0 (socket says 172.21.15.123:58128) 2024-09-16T15:44:11.976 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:11.968+0000 7f055affd640 1 -- 172.21.15.123:0/781665228 learned_addr learned my addr 172.21.15.123:0/781665228 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:44:11.977 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:11.968+0000 7f055a7fc640 1 --2- 172.21.15.123:0/781665228 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f055c0cf5a0 0x7f055c0aec20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:44:11.977 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:11.968+0000 7f055b7fe640 1 --2- 172.21.15.123:0/781665228 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f055c158120 0x7f055c1563e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:44:11.977 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:11.968+0000 7f055affd640 1 -- 172.21.15.123:0/781665228 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f055c0cf5a0 msgr2=0x7f055c0aec20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:44:11.977 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:11.968+0000 7f055affd640 1 --2- 172.21.15.123:0/781665228 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f055c0cf5a0 0x7f055c0aec20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:11.977 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:11.968+0000 7f055affd640 1 -- 172.21.15.123:0/781665228 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f055c158120 msgr2=0x7f055c1563e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:44:11.977 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:11.968+0000 7f055affd640 1 --2- 172.21.15.123:0/781665228 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f055c158120 0x7f055c1563e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:11.977 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:11.968+0000 7f055affd640 1 -- 172.21.15.123:0/781665228 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f055c0ae040 con 0x7f055c06f5f0 2024-09-16T15:44:11.977 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:11.968+0000 7f055a7fc640 1 --2- 172.21.15.123:0/781665228 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f055c0cf5a0 0x7f055c0aec20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T15:44:11.977 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:11.972+0000 7f055affd640 1 --2- 172.21.15.123:0/781665228 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f055c06f5f0 0x7f055c073c60 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f054c009850 tx=0x7f054c005de0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:44:11.977 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:11.972+0000 7f05437fe640 1 -- 172.21.15.123:0/781665228 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f054c0bc020 con 0x7f055c06f5f0 2024-09-16T15:44:11.977 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:11.972+0000 7f05437fe640 1 -- 172.21.15.123:0/781665228 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f054c004330 con 0x7f055c06f5f0 2024-09-16T15:44:11.978 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:11.972+0000 7f0561743640 1 -- 172.21.15.123:0/781665228 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f055c0ae460 con 0x7f055c06f5f0 2024-09-16T15:44:11.978 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:11.972+0000 7f05437fe640 1 -- 172.21.15.123:0/781665228 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f054c0b2040 con 0x7f055c06f5f0 2024-09-16T15:44:11.978 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:11.972+0000 7f0561743640 1 -- 172.21.15.123:0/781665228 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f055c0add60 con 0x7f055c06f5f0 2024-09-16T15:44:11.979 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:11.972+0000 7f0561743640 1 -- 172.21.15.123:0/781665228 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f055c15aa60 con 0x7f055c06f5f0 2024-09-16T15:44:11.979 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:11.972+0000 7f05437fe640 1 -- 172.21.15.123:0/781665228 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 7) ==== 97233+0+0 (secure 0 0 0) 0x7f054c005280 con 0x7f055c06f5f0 2024-09-16T15:44:11.979 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:11.972+0000 7f05437fe640 1 --2- 172.21.15.123:0/781665228 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f05480730a0 0x7f0548075560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:44:11.979 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:11.972+0000 7f05437fe640 1 -- 172.21.15.123:0/781665228 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(40..40 src has 1..40) ==== 4857+0+0 (secure 0 0 0) 0x7f054c12e370 con 0x7f055c06f5f0 2024-09-16T15:44:11.979 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:11.972+0000 7f055a7fc640 1 --2- 172.21.15.123:0/781665228 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f05480730a0 0x7f0548075560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:44:11.979 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:11.972+0000 7f055a7fc640 1 --2- 172.21.15.123:0/781665228 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f05480730a0 0x7f0548075560 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f055c154820 tx=0x7f0550009040 comp rx=0 tx=0).ready entity=mgr.4135 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:44:11.981 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:11.976+0000 7f05437fe640 1 -- 172.21.15.123:0/781665228 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f054c0c6020 con 0x7f055c06f5f0 2024-09-16T15:44:12.143 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:12.136+0000 7f0561743640 1 -- 172.21.15.123:0/781665228 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) -- 0x7f055c0d06c0 con 0x7f055c06f5f0 2024-09-16T15:44:12.143 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:12.140+0000 7f05437fe640 1 -- 172.21.15.123:0/781665228 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v40) ==== 84+0+65 (secure 0 0 0) 0x7f054c0164d0 con 0x7f055c06f5f0 2024-09-16T15:44:12.143 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-09-16T15:44:12.144 INFO:teuthology.orchestra.run.smithi123.stdout:[ 2024-09-16T15:44:12.144 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-09-16T15:44:12.144 INFO:teuthology.orchestra.run.smithi123.stdout: "poolnum": 1, 2024-09-16T15:44:12.144 INFO:teuthology.orchestra.run.smithi123.stdout: "poolname": ".mgr" 2024-09-16T15:44:12.144 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-16T15:44:12.144 INFO:teuthology.orchestra.run.smithi123.stdout:] 2024-09-16T15:44:12.146 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:12.140+0000 7f0561743640 1 -- 172.21.15.123:0/781665228 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f05480730a0 msgr2=0x7f0548075560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:44:12.146 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:12.140+0000 7f0561743640 1 --2- 172.21.15.123:0/781665228 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f05480730a0 0x7f0548075560 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f055c154820 tx=0x7f0550009040 comp rx=0 tx=0).stop 2024-09-16T15:44:12.146 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:12.140+0000 7f0561743640 1 -- 172.21.15.123:0/781665228 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f055c06f5f0 msgr2=0x7f055c073c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:44:12.146 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:12.140+0000 7f0561743640 1 --2- 172.21.15.123:0/781665228 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f055c06f5f0 0x7f055c073c60 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f054c009850 tx=0x7f054c005de0 comp rx=0 tx=0).stop 2024-09-16T15:44:12.146 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:12.140+0000 7f0561743640 1 -- 172.21.15.123:0/781665228 shutdown_connections 2024-09-16T15:44:12.147 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:12.140+0000 7f0561743640 1 --2- 172.21.15.123:0/781665228 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f05480730a0 0x7f0548075560 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:12.147 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:12.140+0000 7f0561743640 1 --2- 172.21.15.123:0/781665228 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f055c158120 0x7f055c1563e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:12.147 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:12.140+0000 7f0561743640 1 --2- 172.21.15.123:0/781665228 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f055c0cf5a0 0x7f055c0aec20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:12.147 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:12.140+0000 7f0561743640 1 --2- 172.21.15.123:0/781665228 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f055c06f5f0 0x7f055c073c60 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:12.147 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:12.140+0000 7f0561743640 1 -- 172.21.15.123:0/781665228 >> 172.21.15.123:0/781665228 conn(0x7f055c0cfb10 msgr2=0x7f055c1535c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:44:12.147 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:12.140+0000 7f0561743640 1 -- 172.21.15.123:0/781665228 shutdown_connections 2024-09-16T15:44:12.147 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:12.140+0000 7f0561743640 1 -- 172.21.15.123:0/781665228 wait complete. 2024-09-16T15:44:12.162 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-09-16T15:44:12.162 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-09-16T15:44:12.286 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:12.280+0000 7fc65c454640 1 -- 172.21.15.123:0/2373989052 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc640003a60 msgr2=0x7fc640003e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:44:12.286 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:12.280+0000 7fc65c454640 1 --2- 172.21.15.123:0/2373989052 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc640003a60 0x7fc640003e80 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7fc650006fa0 tx=0x7fc65001cf00 comp rx=0 tx=0).stop 2024-09-16T15:44:12.286 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:12.280+0000 7fc65a9f1640 1 -- 172.21.15.123:0/2373989052 reap_dead start 2024-09-16T15:44:12.286 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:12.280+0000 7fc65c454640 1 -- 172.21.15.123:0/2373989052 shutdown_connections 2024-09-16T15:44:12.286 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:12.280+0000 7fc65c454640 1 -- 172.21.15.123:0/2373989052 >> 172.21.15.123:0/2373989052 conn(0x7fc6540b1630 msgr2=0x7fc6540b1a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:44:12.286 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:12.280+0000 7fc65c454640 1 -- 172.21.15.123:0/2373989052 shutdown_connections 2024-09-16T15:44:12.286 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:12.280+0000 7fc65c454640 1 -- 172.21.15.123:0/2373989052 wait complete. 2024-09-16T15:44:12.287 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:12.280+0000 7fc65c454640 1 Processor -- start 2024-09-16T15:44:12.287 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:12.280+0000 7fc65c454640 1 -- start start 2024-09-16T15:44:12.289 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:12.284+0000 7fc65c454640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc640003a60 0x7fc654162030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:44:12.289 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:12.284+0000 7fc65c454640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc6540addc0 0x7fc6540b0b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:44:12.289 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:12.284+0000 7fc65c454640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fc65414c120 0x7fc65414c590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:44:12.289 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:12.284+0000 7fc65c454640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fc6540b3960 con 0x7fc6540addc0 2024-09-16T15:44:12.290 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:12.284+0000 7fc65c454640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fc6540b37e0 con 0x7fc640003a60 2024-09-16T15:44:12.290 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:12.284+0000 7fc65c454640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fc6540b3ae0 con 0x7fc65414c120 2024-09-16T15:44:12.290 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:12.284+0000 7fc6599ef640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc6540addc0 0x7fc6540b0b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:44:12.290 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:12.284+0000 7fc6599ef640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc6540addc0 0x7fc6540b0b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:58160/0 (socket says 172.21.15.123:58160) 2024-09-16T15:44:12.290 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:12.284+0000 7fc6599ef640 1 -- 172.21.15.123:0/2603357810 learned_addr learned my addr 172.21.15.123:0/2603357810 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:44:12.290 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:12.284+0000 7fc65a1f0640 1 --2- 172.21.15.123:0/2603357810 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc640003a60 0x7fc654162030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:44:12.290 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:12.284+0000 7fc65a9f1640 1 --2- 172.21.15.123:0/2603357810 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fc65414c120 0x7fc65414c590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:44:12.290 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:12.284+0000 7fc6599ef640 1 -- 172.21.15.123:0/2603357810 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fc65414c120 msgr2=0x7fc65414c590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:44:12.290 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:12.284+0000 7fc6599ef640 1 --2- 172.21.15.123:0/2603357810 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fc65414c120 0x7fc65414c590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:12.290 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:12.284+0000 7fc6599ef640 1 -- 172.21.15.123:0/2603357810 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc640003a60 msgr2=0x7fc654162030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:44:12.290 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:12.284+0000 7fc6599ef640 1 --2- 172.21.15.123:0/2603357810 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc640003a60 0x7fc654162030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:12.290 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:12.284+0000 7fc6599ef640 1 -- 172.21.15.123:0/2603357810 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc6540ae5a0 con 0x7fc6540addc0 2024-09-16T15:44:12.291 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:12.284+0000 7fc65a9f1640 1 --2- 172.21.15.123:0/2603357810 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fc65414c120 0x7fc65414c590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T15:44:12.291 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:12.284+0000 7fc6599ef640 1 --2- 172.21.15.123:0/2603357810 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc6540addc0 0x7fc6540b0b80 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7fc64400f9d0 tx=0x7fc64400fe90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:44:12.291 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:12.284+0000 7fc64affd640 1 -- 172.21.15.123:0/2603357810 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc644015070 con 0x7fc6540addc0 2024-09-16T15:44:12.291 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:12.284+0000 7fc64affd640 1 -- 172.21.15.123:0/2603357810 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fc64400b8a0 con 0x7fc6540addc0 2024-09-16T15:44:12.291 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:12.284+0000 7fc64affd640 1 -- 172.21.15.123:0/2603357810 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc64400bcd0 con 0x7fc6540addc0 2024-09-16T15:44:12.291 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:12.284+0000 7fc65c454640 1 -- 172.21.15.123:0/2603357810 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc65414cad0 con 0x7fc6540addc0 2024-09-16T15:44:12.291 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:12.284+0000 7fc65c454640 1 -- 172.21.15.123:0/2603357810 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc654159b80 con 0x7fc6540addc0 2024-09-16T15:44:12.291 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:12.284+0000 7fc65c454640 1 -- 172.21.15.123:0/2603357810 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc6540b4040 con 0x7fc6540addc0 2024-09-16T15:44:12.292 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:12.284+0000 7fc64affd640 1 -- 172.21.15.123:0/2603357810 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 7) ==== 97233+0+0 (secure 0 0 0) 0x7fc644003600 con 0x7fc6540addc0 2024-09-16T15:44:12.292 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:12.284+0000 7fc64affd640 1 --2- 172.21.15.123:0/2603357810 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fc6380730a0 0x7fc638075560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:44:12.292 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:12.284+0000 7fc64affd640 1 -- 172.21.15.123:0/2603357810 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(40..40 src has 1..40) ==== 4857+0+0 (secure 0 0 0) 0x7fc644094ae0 con 0x7fc6540addc0 2024-09-16T15:44:12.292 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:12.288+0000 7fc65a1f0640 1 --2- 172.21.15.123:0/2603357810 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fc6380730a0 0x7fc638075560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:44:12.292 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:12.288+0000 7fc65a1f0640 1 --2- 172.21.15.123:0/2603357810 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fc6380730a0 0x7fc638075560 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fc6500ab000 tx=0x7fc650019d40 comp rx=0 tx=0).ready entity=mgr.4135 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:44:12.294 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:12.288+0000 7fc64affd640 1 -- 172.21.15.123:0/2603357810 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7fc64405fd80 con 0x7fc6540addc0 2024-09-16T15:44:12.456 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:12.452+0000 7fc65c454640 1 -- 172.21.15.123:0/2603357810 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) -- 0x7fc6540cf7b0 con 0x7fc6540addc0 2024-09-16T15:44:13.462 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:13.456+0000 7fc64affd640 1 -- 172.21.15.123:0/2603357810 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v41) ==== 138+0+0 (secure 0 0 0) 0x7fc64405f680 con 0x7fc6540addc0 2024-09-16T15:44:13.462 INFO:teuthology.orchestra.run.smithi123.stderr:pool 'cephfs_metadata' created 2024-09-16T15:44:13.467 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:13.460+0000 7fc65c454640 1 -- 172.21.15.123:0/2603357810 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fc6380730a0 msgr2=0x7fc638075560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:44:13.467 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:13.460+0000 7fc65c454640 1 --2- 172.21.15.123:0/2603357810 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fc6380730a0 0x7fc638075560 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fc6500ab000 tx=0x7fc650019d40 comp rx=0 tx=0).stop 2024-09-16T15:44:13.467 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:13.460+0000 7fc65c454640 1 -- 172.21.15.123:0/2603357810 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc6540addc0 msgr2=0x7fc6540b0b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:44:13.467 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:13.460+0000 7fc65c454640 1 --2- 172.21.15.123:0/2603357810 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc6540addc0 0x7fc6540b0b80 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7fc64400f9d0 tx=0x7fc64400fe90 comp rx=0 tx=0).stop 2024-09-16T15:44:13.467 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:13.460+0000 7fc65c454640 1 -- 172.21.15.123:0/2603357810 shutdown_connections 2024-09-16T15:44:13.467 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:13.460+0000 7fc65c454640 1 --2- 172.21.15.123:0/2603357810 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fc6380730a0 0x7fc638075560 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:13.467 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:13.460+0000 7fc65c454640 1 --2- 172.21.15.123:0/2603357810 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fc65414c120 0x7fc65414c590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:13.467 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:13.460+0000 7fc65c454640 1 --2- 172.21.15.123:0/2603357810 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc6540addc0 0x7fc6540b0b80 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:13.467 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:13.460+0000 7fc65c454640 1 --2- 172.21.15.123:0/2603357810 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc640003a60 0x7fc654162030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:13.467 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:13.460+0000 7fc65c454640 1 -- 172.21.15.123:0/2603357810 >> 172.21.15.123:0/2603357810 conn(0x7fc6540b1630 msgr2=0x7fc6540cee50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:44:13.468 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:13.464+0000 7fc65c454640 1 -- 172.21.15.123:0/2603357810 shutdown_connections 2024-09-16T15:44:13.468 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:13.464+0000 7fc65c454640 1 -- 172.21.15.123:0/2603357810 wait complete. 2024-09-16T15:44:13.492 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-09-16T15:44:13.629 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:13.624+0000 7f8f5fa12640 1 -- 172.21.15.123:0/4127331950 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8f580d2120 msgr2=0x7f8f580cf0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:44:13.629 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:13.624+0000 7f8f5fa12640 1 --2- 172.21.15.123:0/4127331950 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8f580d2120 0x7f8f580cf0d0 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f8f50009bc0 tx=0x7f8f5001c680 comp rx=0 tx=0).stop 2024-09-16T15:44:13.629 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:13.624+0000 7f8f5fa12640 1 -- 172.21.15.123:0/4127331950 shutdown_connections 2024-09-16T15:44:13.629 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:13.624+0000 7f8f5fa12640 1 --2- 172.21.15.123:0/4127331950 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8f580d2120 0x7f8f580cf0d0 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:13.630 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:13.624+0000 7f8f5fa12640 1 -- 172.21.15.123:0/4127331950 >> 172.21.15.123:0/4127331950 conn(0x7f8f580d3e40 msgr2=0x7f8f580d4220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:44:13.630 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:13.624+0000 7f8f5fa12640 1 -- 172.21.15.123:0/4127331950 shutdown_connections 2024-09-16T15:44:13.630 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:13.624+0000 7f8f5fa12640 1 -- 172.21.15.123:0/4127331950 wait complete. 2024-09-16T15:44:13.632 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:13.624+0000 7f8f5fa12640 1 Processor -- start 2024-09-16T15:44:13.632 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:13.624+0000 7f8f5fa12640 1 -- start start 2024-09-16T15:44:13.632 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:13.624+0000 7f8f5fa12640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f8f580d0f60 0x7f8f58155710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:44:13.632 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:13.624+0000 7f8f5fa12640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8f580d2120 0x7f8f580c9960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:44:13.632 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:13.624+0000 7f8f5fa12640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f8f581615c0 0x7f8f580c9ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:44:13.632 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:13.624+0000 7f8f5fa12640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f8f5816d5e0 con 0x7f8f580d2120 2024-09-16T15:44:13.632 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:13.628+0000 7f8f5fa12640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f8f5816d460 con 0x7f8f581615c0 2024-09-16T15:44:13.633 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:13.628+0000 7f8f5fa12640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f8f5816d760 con 0x7f8f580d0f60 2024-09-16T15:44:13.633 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:13.628+0000 7f8f5cfad640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8f580d2120 0x7f8f580c9960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:44:13.633 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:13.628+0000 7f8f5cfad640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8f580d2120 0x7f8f580c9960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:47120/0 (socket says 172.21.15.123:47120) 2024-09-16T15:44:13.633 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:13.628+0000 7f8f5cfad640 1 -- 172.21.15.123:0/1853672417 learned_addr learned my addr 172.21.15.123:0/1853672417 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:44:13.633 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:13.628+0000 7f8f5dfaf640 1 --2- 172.21.15.123:0/1853672417 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f8f581615c0 0x7f8f580c9ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:44:13.633 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:13.628+0000 7f8f5d7ae640 1 --2- 172.21.15.123:0/1853672417 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f8f580d0f60 0x7f8f58155710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:44:13.633 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:13.628+0000 7f8f5cfad640 1 -- 172.21.15.123:0/1853672417 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f8f580d0f60 msgr2=0x7f8f58155710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:44:13.633 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:13.628+0000 7f8f5cfad640 1 --2- 172.21.15.123:0/1853672417 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f8f580d0f60 0x7f8f58155710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:13.633 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:13.628+0000 7f8f5cfad640 1 -- 172.21.15.123:0/1853672417 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f8f581615c0 msgr2=0x7f8f580c9ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:44:13.633 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:13.628+0000 7f8f5cfad640 1 --2- 172.21.15.123:0/1853672417 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f8f581615c0 0x7f8f580c9ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:13.633 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:13.628+0000 7f8f5cfad640 1 -- 172.21.15.123:0/1853672417 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8f580cba30 con 0x7f8f580d2120 2024-09-16T15:44:13.633 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:13.628+0000 7f8f5d7ae640 1 --2- 172.21.15.123:0/1853672417 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f8f580d0f60 0x7f8f58155710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T15:44:13.634 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:13.628+0000 7f8f5dfaf640 1 --2- 172.21.15.123:0/1853672417 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f8f581615c0 0x7f8f580c9ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T15:44:13.634 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:13.628+0000 7f8f5cfad640 1 --2- 172.21.15.123:0/1853672417 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8f580d2120 0x7f8f580c9960 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f8f4800c730 tx=0x7f8f4800cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:44:13.634 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:13.628+0000 7f8f4dffb640 1 -- 172.21.15.123:0/1853672417 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f48017070 con 0x7f8f580d2120 2024-09-16T15:44:13.634 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:13.628+0000 7f8f4dffb640 1 -- 172.21.15.123:0/1853672417 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f8f48005360 con 0x7f8f580d2120 2024-09-16T15:44:13.634 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:13.628+0000 7f8f5fa12640 1 -- 172.21.15.123:0/1853672417 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8f580cbd10 con 0x7f8f580d2120 2024-09-16T15:44:13.634 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:13.628+0000 7f8f4dffb640 1 -- 172.21.15.123:0/1853672417 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f4800a8a0 con 0x7f8f580d2120 2024-09-16T15:44:13.634 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:13.628+0000 7f8f5fa12640 1 -- 172.21.15.123:0/1853672417 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8f580cc2d0 con 0x7f8f580d2120 2024-09-16T15:44:13.635 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:13.628+0000 7f8f5fa12640 1 -- 172.21.15.123:0/1853672417 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8f580d0850 con 0x7f8f580d2120 2024-09-16T15:44:13.635 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:13.628+0000 7f8f4dffb640 1 -- 172.21.15.123:0/1853672417 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 7) ==== 97233+0+0 (secure 0 0 0) 0x7f8f4801a070 con 0x7f8f580d2120 2024-09-16T15:44:13.635 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:13.628+0000 7f8f4dffb640 1 --2- 172.21.15.123:0/1853672417 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f8f3c073070 0x7f8f3c075530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:44:13.635 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:13.628+0000 7f8f4dffb640 1 -- 172.21.15.123:0/1853672417 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(41..41 src has 1..41) ==== 5227+0+0 (secure 0 0 0) 0x7f8f48093870 con 0x7f8f580d2120 2024-09-16T15:44:13.635 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:13.628+0000 7f8f5d7ae640 1 --2- 172.21.15.123:0/1853672417 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f8f3c073070 0x7f8f3c075530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:44:13.636 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:13.632+0000 7f8f5d7ae640 1 --2- 172.21.15.123:0/1853672417 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f8f3c073070 0x7f8f3c075530 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f8f5400a830 tx=0x7f8f54008040 comp rx=0 tx=0).ready entity=mgr.4135 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:44:13.638 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:13.632+0000 7f8f4dffb640 1 -- 172.21.15.123:0/1853672417 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f8f4805e910 con 0x7f8f580d2120 2024-09-16T15:44:13.810 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:13.804+0000 7f8f5fa12640 1 -- 172.21.15.123:0/1853672417 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) -- 0x7f8f5806d7b0 con 0x7f8f580d2120 2024-09-16T15:44:14.466 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:14.460+0000 7f8f4dffb640 1 -- 172.21.15.123:0/1853672417 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v42) ==== 170+0+0 (secure 0 0 0) 0x7f8f4805e210 con 0x7f8f580d2120 2024-09-16T15:44:14.466 INFO:teuthology.orchestra.run.smithi123.stderr:pool 'cephfs_data' created 2024-09-16T15:44:14.470 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:14.464+0000 7f8f5fa12640 1 -- 172.21.15.123:0/1853672417 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f8f3c073070 msgr2=0x7f8f3c075530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:44:14.470 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:14.464+0000 7f8f5fa12640 1 --2- 172.21.15.123:0/1853672417 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f8f3c073070 0x7f8f3c075530 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f8f5400a830 tx=0x7f8f54008040 comp rx=0 tx=0).stop 2024-09-16T15:44:14.471 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:14.464+0000 7f8f5fa12640 1 -- 172.21.15.123:0/1853672417 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8f580d2120 msgr2=0x7f8f580c9960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:44:14.471 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:14.464+0000 7f8f5fa12640 1 --2- 172.21.15.123:0/1853672417 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8f580d2120 0x7f8f580c9960 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f8f4800c730 tx=0x7f8f4800cbf0 comp rx=0 tx=0).stop 2024-09-16T15:44:14.471 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:14.464+0000 7f8f5fa12640 1 -- 172.21.15.123:0/1853672417 shutdown_connections 2024-09-16T15:44:14.471 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:14.464+0000 7f8f5fa12640 1 --2- 172.21.15.123:0/1853672417 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f8f3c073070 0x7f8f3c075530 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:14.471 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:14.464+0000 7f8f5fa12640 1 --2- 172.21.15.123:0/1853672417 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f8f581615c0 0x7f8f580c9ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:14.471 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:14.464+0000 7f8f5fa12640 1 --2- 172.21.15.123:0/1853672417 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8f580d2120 0x7f8f580c9960 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:14.471 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:14.464+0000 7f8f5fa12640 1 --2- 172.21.15.123:0/1853672417 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f8f580d0f60 0x7f8f58155710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:14.471 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:14.464+0000 7f8f5fa12640 1 -- 172.21.15.123:0/1853672417 >> 172.21.15.123:0/1853672417 conn(0x7f8f580d3e40 msgr2=0x7f8f581607e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:44:14.472 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:14.464+0000 7f8f5fa12640 1 -- 172.21.15.123:0/1853672417 shutdown_connections 2024-09-16T15:44:14.472 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:14.468+0000 7f8f5fa12640 1 -- 172.21.15.123:0/1853672417 wait complete. 2024-09-16T15:44:14.505 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-09-16T15:44:14.631 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:14.624+0000 7f43aa9de640 1 -- 172.21.15.123:0/189751585 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f43a40b2150 msgr2=0x7f43a40cfdc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:44:14.631 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:14.624+0000 7f43aa9de640 1 --2- 172.21.15.123:0/189751585 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f43a40b2150 0x7f43a40cfdc0 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f4398009840 tx=0x7f439801c820 comp rx=0 tx=0).stop 2024-09-16T15:44:14.631 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:14.624+0000 7f43aa9de640 1 -- 172.21.15.123:0/189751585 shutdown_connections 2024-09-16T15:44:14.631 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:14.624+0000 7f43aa9de640 1 --2- 172.21.15.123:0/189751585 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f43a40b2150 0x7f43a40cfdc0 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:14.632 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:14.624+0000 7f43aa9de640 1 --2- 172.21.15.123:0/189751585 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f43a415bb70 0x7f43a40b1c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:14.632 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:14.624+0000 7f43aa9de640 1 -- 172.21.15.123:0/189751585 >> 172.21.15.123:0/189751585 conn(0x7f43a409aa40 msgr2=0x7f43a409ae50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:44:14.632 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:14.628+0000 7f43aa9de640 1 -- 172.21.15.123:0/189751585 shutdown_connections 2024-09-16T15:44:14.632 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:14.628+0000 7f43aa9de640 1 -- 172.21.15.123:0/189751585 wait complete. 2024-09-16T15:44:14.632 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:14.628+0000 7f43aa9de640 1 Processor -- start 2024-09-16T15:44:14.633 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:14.628+0000 7f43aa9de640 1 -- start start 2024-09-16T15:44:14.635 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:14.628+0000 7f43aa9de640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f43a40b2150 0x7f43a4151360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:44:14.635 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:14.628+0000 7f43aa9de640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f43a40d0300 0x7f43a41fbe40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:44:14.635 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:14.628+0000 7f43aa9de640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f43a415bb70 0x7f43a4159570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:44:14.635 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:14.628+0000 7f43aa9de640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f43a4169010 con 0x7f43a40b2150 2024-09-16T15:44:14.635 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:14.628+0000 7f43aa9de640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f43a4168e90 con 0x7f43a415bb70 2024-09-16T15:44:14.635 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:14.628+0000 7f43aa9de640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f43a4169190 con 0x7f43a40d0300 2024-09-16T15:44:14.635 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:14.628+0000 7f43a3fff640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f43a40b2150 0x7f43a4151360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:44:14.635 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:14.628+0000 7f43a3fff640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f43a40b2150 0x7f43a4151360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:47136/0 (socket says 172.21.15.123:47136) 2024-09-16T15:44:14.635 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:14.628+0000 7f43a3fff640 1 -- 172.21.15.123:0/497584801 learned_addr learned my addr 172.21.15.123:0/497584801 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:44:14.636 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:14.628+0000 7f43a37fe640 1 --2- 172.21.15.123:0/497584801 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f43a40d0300 0x7f43a41fbe40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:44:14.636 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:14.628+0000 7f43a8f7b640 1 --2- 172.21.15.123:0/497584801 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f43a415bb70 0x7f43a4159570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:44:14.636 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:14.628+0000 7f43a3fff640 1 -- 172.21.15.123:0/497584801 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f43a40d0300 msgr2=0x7f43a41fbe40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:44:14.636 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:14.628+0000 7f43a3fff640 1 --2- 172.21.15.123:0/497584801 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f43a40d0300 0x7f43a41fbe40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:14.636 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:14.628+0000 7f43a3fff640 1 -- 172.21.15.123:0/497584801 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f43a415bb70 msgr2=0x7f43a4159570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:44:14.636 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:14.628+0000 7f43a3fff640 1 --2- 172.21.15.123:0/497584801 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f43a415bb70 0x7f43a4159570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:14.636 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:14.628+0000 7f43a3fff640 1 -- 172.21.15.123:0/497584801 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f43a41f8030 con 0x7f43a40b2150 2024-09-16T15:44:14.636 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:14.628+0000 7f43a37fe640 1 --2- 172.21.15.123:0/497584801 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f43a40d0300 0x7f43a41fbe40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T15:44:14.636 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:14.628+0000 7f43a8f7b640 1 --2- 172.21.15.123:0/497584801 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f43a415bb70 0x7f43a4159570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T15:44:14.636 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:14.628+0000 7f43a3fff640 1 --2- 172.21.15.123:0/497584801 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f43a40b2150 0x7f43a4151360 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f439400ccb0 tx=0x7f439400a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:44:14.636 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:14.628+0000 7f43a0ff9640 1 -- 172.21.15.123:0/497584801 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f439400f450 con 0x7f43a40b2150 2024-09-16T15:44:14.637 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:14.628+0000 7f43a0ff9640 1 -- 172.21.15.123:0/497584801 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f439400f6e0 con 0x7f43a40b2150 2024-09-16T15:44:14.637 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:14.628+0000 7f43aa9de640 1 -- 172.21.15.123:0/497584801 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f43a41f82f0 con 0x7f43a40b2150 2024-09-16T15:44:14.637 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:14.628+0000 7f43a0ff9640 1 -- 172.21.15.123:0/497584801 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f439400fb30 con 0x7f43a40b2150 2024-09-16T15:44:14.637 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:14.628+0000 7f43aa9de640 1 -- 172.21.15.123:0/497584801 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f43a41f88b0 con 0x7f43a40b2150 2024-09-16T15:44:14.637 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:14.632+0000 7f43aa9de640 1 -- 172.21.15.123:0/497584801 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f43a40cfdc0 con 0x7f43a40b2150 2024-09-16T15:44:14.638 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:14.632+0000 7f43a0ff9640 1 -- 172.21.15.123:0/497584801 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 7) ==== 97233+0+0 (secure 0 0 0) 0x7f4394018400 con 0x7f43a40b2150 2024-09-16T15:44:14.638 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:14.632+0000 7f43a0ff9640 1 --2- 172.21.15.123:0/497584801 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f4390073070 0x7f4390075530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:44:14.638 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:14.632+0000 7f43a0ff9640 1 -- 172.21.15.123:0/497584801 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(42..42 src has 1..42) ==== 5609+0+0 (secure 0 0 0) 0x7f4394095120 con 0x7f43a40b2150 2024-09-16T15:44:14.638 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:14.632+0000 7f43a37fe640 1 --2- 172.21.15.123:0/497584801 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f4390073070 0x7f4390075530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:44:14.638 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:14.632+0000 7f43a37fe640 1 --2- 172.21.15.123:0/497584801 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f4390073070 0x7f4390075530 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f43980af000 tx=0x7f439801b060 comp rx=0 tx=0).ready entity=mgr.4135 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:44:14.640 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:14.636+0000 7f43a0ff9640 1 -- 172.21.15.123:0/497584801 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f4394026020 con 0x7f43a40b2150 2024-09-16T15:44:14.852 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:14.848+0000 7f43aa9de640 1 -- 172.21.15.123:0/497584801 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) -- 0x7f43a40b0b30 con 0x7f43a40b2150 2024-09-16T15:44:15.534 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:15.528+0000 7f43a0ff9640 1 -- 172.21.15.123:0/497584801 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 6 and data pool 7 v24) ==== 172+0+0 (secure 0 0 0) 0x7f4394014030 con 0x7f43a40b2150 2024-09-16T15:44:15.541 INFO:teuthology.orchestra.run.smithi123.stderr:new fs with metadata pool 6 and data pool 7 2024-09-16T15:44:15.545 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:15.540+0000 7f43aa9de640 1 -- 172.21.15.123:0/497584801 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f4390073070 msgr2=0x7f4390075530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:44:15.545 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:15.540+0000 7f43aa9de640 1 --2- 172.21.15.123:0/497584801 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f4390073070 0x7f4390075530 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f43980af000 tx=0x7f439801b060 comp rx=0 tx=0).stop 2024-09-16T15:44:15.545 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:15.540+0000 7f43aa9de640 1 -- 172.21.15.123:0/497584801 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f43a40b2150 msgr2=0x7f43a4151360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:44:15.545 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:15.540+0000 7f43aa9de640 1 --2- 172.21.15.123:0/497584801 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f43a40b2150 0x7f43a4151360 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f439400ccb0 tx=0x7f439400a620 comp rx=0 tx=0).stop 2024-09-16T15:44:15.545 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:15.540+0000 7f43aa9de640 1 -- 172.21.15.123:0/497584801 shutdown_connections 2024-09-16T15:44:15.546 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:15.540+0000 7f43aa9de640 1 --2- 172.21.15.123:0/497584801 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f4390073070 0x7f4390075530 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:15.546 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:15.540+0000 7f43aa9de640 1 --2- 172.21.15.123:0/497584801 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f43a415bb70 0x7f43a4159570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:15.546 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:15.540+0000 7f43aa9de640 1 --2- 172.21.15.123:0/497584801 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f43a40d0300 0x7f43a41fbe40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:15.546 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:15.540+0000 7f43aa9de640 1 --2- 172.21.15.123:0/497584801 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f43a40b2150 0x7f43a4151360 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:15.546 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:15.540+0000 7f43aa9de640 1 -- 172.21.15.123:0/497584801 >> 172.21.15.123:0/497584801 conn(0x7f43a409aa40 msgr2=0x7f43a415b090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:44:15.546 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:15.540+0000 7f43aa9de640 1 -- 172.21.15.123:0/497584801 shutdown_connections 2024-09-16T15:44:15.546 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:15.540+0000 7f43aa9de640 1 -- 172.21.15.123:0/497584801 wait complete. 2024-09-16T15:44:15.571 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-16T15:44:15.692 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:15.688+0000 7fa5ae679640 1 -- 172.21.15.123:0/1474132222 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa5a8074600 msgr2=0x7fa5a806e750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:44:15.692 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:15.688+0000 7fa5ae679640 1 --2- 172.21.15.123:0/1474132222 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa5a8074600 0x7fa5a806e750 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7fa5a806df40 tx=0x7fa5a001cac0 comp rx=0 tx=0).stop 2024-09-16T15:44:15.692 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:15.688+0000 7fa5ae679640 1 -- 172.21.15.123:0/1474132222 shutdown_connections 2024-09-16T15:44:15.692 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:15.688+0000 7fa5ae679640 1 --2- 172.21.15.123:0/1474132222 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa5a8074600 0x7fa5a806e750 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:15.692 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:15.688+0000 7fa5ae679640 1 --2- 172.21.15.123:0/1474132222 >> v2:172.21.15.184:3300/0 conn(0x7fa5a80726c0 0x7fa5a80d0630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:15.692 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:15.688+0000 7fa5ae679640 1 -- 172.21.15.123:0/1474132222 >> 172.21.15.123:0/1474132222 conn(0x7fa5a80cfb10 msgr2=0x7fa5a80cff20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:44:15.692 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:15.688+0000 7fa5ae679640 1 -- 172.21.15.123:0/1474132222 shutdown_connections 2024-09-16T15:44:15.692 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:15.688+0000 7fa5ae679640 1 -- 172.21.15.123:0/1474132222 wait complete. 2024-09-16T15:44:15.693 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:15.688+0000 7fa5ae679640 1 Processor -- start 2024-09-16T15:44:15.693 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:15.688+0000 7fa5ae679640 1 -- start start 2024-09-16T15:44:15.695 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:15.688+0000 7fa5ae679640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa5a80726c0 0x7fa5a81f94a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:44:15.695 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:15.688+0000 7fa5ae679640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fa5a8074600 0x7fa5a8151940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:44:15.695 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:15.688+0000 7fa5ae679640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa5a815bca0 0x7fa5a8159f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:44:15.696 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:15.688+0000 7fa5ae679640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fa5a8168a50 con 0x7fa5a815bca0 2024-09-16T15:44:15.696 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:15.688+0000 7fa5ae679640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fa5a806f980 con 0x7fa5a80726c0 2024-09-16T15:44:15.696 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:15.688+0000 7fa5ae679640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fa5a8168bd0 con 0x7fa5a8074600 2024-09-16T15:44:15.696 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:15.688+0000 7fa5acc16640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa5a815bca0 0x7fa5a8159f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:44:15.696 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:15.688+0000 7fa5acc16640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa5a815bca0 0x7fa5a8159f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:47152/0 (socket says 172.21.15.123:47152) 2024-09-16T15:44:15.696 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:15.688+0000 7fa5acc16640 1 -- 172.21.15.123:0/1363143029 learned_addr learned my addr 172.21.15.123:0/1363143029 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:44:15.696 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:15.688+0000 7fa5acc16640 1 -- 172.21.15.123:0/1363143029 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fa5a8074600 msgr2=0x7fa5a8151940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:44:15.696 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:15.688+0000 7fa59f7fe640 1 --2- 172.21.15.123:0/1363143029 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fa5a8074600 0x7fa5a8151940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:44:15.696 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:15.688+0000 7fa5acc16640 1 --2- 172.21.15.123:0/1363143029 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fa5a8074600 0x7fa5a8151940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:15.696 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:15.688+0000 7fa5acc16640 1 -- 172.21.15.123:0/1363143029 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa5a80726c0 msgr2=0x7fa5a81f94a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:44:15.696 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:15.688+0000 7fa5a7fff640 1 --2- 172.21.15.123:0/1363143029 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa5a80726c0 0x7fa5a81f94a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:44:15.696 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:15.688+0000 7fa5acc16640 1 --2- 172.21.15.123:0/1363143029 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa5a80726c0 0x7fa5a81f94a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:15.696 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:15.688+0000 7fa5acc16640 1 -- 172.21.15.123:0/1363143029 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa5a80c7690 con 0x7fa5a815bca0 2024-09-16T15:44:15.697 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:15.688+0000 7fa5acc16640 1 --2- 172.21.15.123:0/1363143029 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa5a815bca0 0x7fa5a8159f30 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7fa5a001c530 tx=0x7fa5a0005b00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:44:15.697 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:15.688+0000 7fa5a57fa640 1 -- 172.21.15.123:0/1363143029 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5a00b2040 con 0x7fa5a815bca0 2024-09-16T15:44:15.697 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:15.688+0000 7fa5a57fa640 1 -- 172.21.15.123:0/1363143029 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fa5a00bb070 con 0x7fa5a815bca0 2024-09-16T15:44:15.697 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:15.688+0000 7fa5ae679640 1 -- 172.21.15.123:0/1363143029 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa5a80c7970 con 0x7fa5a815bca0 2024-09-16T15:44:15.697 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:15.688+0000 7fa5a57fa640 1 -- 172.21.15.123:0/1363143029 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5a0004330 con 0x7fa5a815bca0 2024-09-16T15:44:15.697 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:15.688+0000 7fa5ae679640 1 -- 172.21.15.123:0/1363143029 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa5a80c9670 con 0x7fa5a815bca0 2024-09-16T15:44:15.697 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:15.692+0000 7fa5ae679640 1 -- 172.21.15.123:0/1363143029 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa5a80c9180 con 0x7fa5a815bca0 2024-09-16T15:44:15.698 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:15.692+0000 7fa5a57fa640 1 -- 172.21.15.123:0/1363143029 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 7) ==== 97233+0+0 (secure 0 0 0) 0x7fa5a00044d0 con 0x7fa5a815bca0 2024-09-16T15:44:15.698 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:15.692+0000 7fa5a57fa640 1 --2- 172.21.15.123:0/1363143029 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fa588073070 0x7fa588075530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:44:15.698 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:15.692+0000 7fa5a57fa640 1 -- 172.21.15.123:0/1363143029 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(43..43 src has 1..43) ==== 5709+0+0 (secure 0 0 0) 0x7fa5a012eb80 con 0x7fa5a815bca0 2024-09-16T15:44:15.698 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:15.692+0000 7fa5a7fff640 1 --2- 172.21.15.123:0/1363143029 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fa588073070 0x7fa588075530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:44:15.698 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:15.692+0000 7fa5a7fff640 1 --2- 172.21.15.123:0/1363143029 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fa588073070 0x7fa588075530 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fa598001230 tx=0x7fa598009040 comp rx=0 tx=0).ready entity=mgr.4135 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:44:15.701 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:15.696+0000 7fa5a57fa640 1 -- 172.21.15.123:0/1363143029 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7fa5a0016770 con 0x7fa5a815bca0 2024-09-16T15:44:15.875 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:15.872+0000 7fa5ae679640 1 -- 172.21.15.123:0/1363143029 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fa5a80c4990 con 0x7fa5a815bca0 2024-09-16T15:44:15.878 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:15.872+0000 7fa5a57fa640 1 -- 172.21.15.123:0/1363143029 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v43) ==== 74+0+16746 (secure 0 0 0) 0x7fa5a00f9ad0 con 0x7fa5a815bca0 2024-09-16T15:44:15.878 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-09-16T15:44:15.878 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":43,"fsid":"56077f89-18c8-486f-ae7e-eecc0bb7822c","created":"2024-09-16T15:42:26.364892+0000","modified":"2024-09-16T15:44:14.853278+0000","last_up_change":"2024-09-16T15:42:38.707234+0000","last_in_change":"2024-09-16T15:42:29.423662+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-16T15:42:39.287764+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_metadata","create_time":"2024-09-16T15:44:12.463367+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"43","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_data","create_time":"2024-09-16T15:44:13.817170+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"43","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9e7c1955-a607-4671-9be7-f2ef760ceeaa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6800","nonce":3882044969},{"type":"v1","addr":"172.21.15.123:6801","nonce":3882044969}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6802","nonce":3882044969},{"type":"v1","addr":"172.21.15.123:6803","nonce":3882044969}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6806","nonce":3882044969},{"type":"v1","addr":"172.21.15.123:6807","nonce":3882044969}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6804","nonce":3882044969},{"type":"v1","addr":"172.21.15.123:6805","nonce":3882044969}]},"public_addr":"172.21.15.123:6801/3882044969","cluster_addr":"172.21.15.123:6803/3882044969","heartbeat_back_addr":"172.21.15.123:6807/3882044969","heartbeat_front_addr":"172.21.15.123:6805/3882044969","state":["exists","up"]},{"osd":1,"uuid":"c540cfde-0977-4742-9b17-fd634668e010","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6816","nonce":4037001850},{"type":"v1","addr":"172.21.15.123:6817","nonce":4037001850}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6818","nonce":4037001850},{"type":"v1","addr":"172.21.15.123:6819","nonce":4037001850}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6822","nonce":4037001850},{"type":"v1","addr":"172.21.15.123:6823","nonce":4037001850}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6820","nonce":4037001850},{"type":"v1","addr":"172.21.15.123:6821","nonce":4037001850}]},"public_addr":"172.21.15.123:6817/4037001850","cluster_addr":"172.21.15.123:6819/4037001850","heartbeat_back_addr":"172.21.15.123:6823/4037001850","heartbeat_front_addr":"172.21.15.123:6821/4037001850","state":["exists","up"]},{"osd":2,"uuid":"9864a496-f169-486b-95c4-1d59162793a7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6824","nonce":1332143088},{"type":"v1","addr":"172.21.15.123:6825","nonce":1332143088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6826","nonce":1332143088},{"type":"v1","addr":"172.21.15.123:6827","nonce":1332143088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6830","nonce":1332143088},{"type":"v1","addr":"172.21.15.123:6831","nonce":1332143088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6828","nonce":1332143088},{"type":"v1","addr":"172.21.15.123:6829","nonce":1332143088}]},"public_addr":"172.21.15.123:6825/1332143088","cluster_addr":"172.21.15.123:6827/1332143088","heartbeat_back_addr":"172.21.15.123:6831/1332143088","heartbeat_front_addr":"172.21.15.123:6829/1332143088","state":["exists","up"]},{"osd":3,"uuid":"c6d1f817-f074-4325-95e1-578eba13edda","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6808","nonce":988586998},{"type":"v1","addr":"172.21.15.123:6809","nonce":988586998}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6810","nonce":988586998},{"type":"v1","addr":"172.21.15.123:6811","nonce":988586998}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6814","nonce":988586998},{"type":"v1","addr":"172.21.15.123:6815","nonce":988586998}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6812","nonce":988586998},{"type":"v1","addr":"172.21.15.123:6813","nonce":988586998}]},"public_addr":"172.21.15.123:6809/988586998","cluster_addr":"172.21.15.123:6811/988586998","heartbeat_back_addr":"172.21.15.123:6815/988586998","heartbeat_front_addr":"172.21.15.123:6813/988586998","state":["exists","up"]},{"osd":4,"uuid":"200a8df4-50ef-40a5-b48a-58981269c086","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6816","nonce":3387813890},{"type":"v1","addr":"172.21.15.184:6817","nonce":3387813890}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6818","nonce":3387813890},{"type":"v1","addr":"172.21.15.184:6819","nonce":3387813890}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6822","nonce":3387813890},{"type":"v1","addr":"172.21.15.184:6823","nonce":3387813890}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6820","nonce":3387813890},{"type":"v1","addr":"172.21.15.184:6821","nonce":3387813890}]},"public_addr":"172.21.15.184:6817/3387813890","cluster_addr":"172.21.15.184:6819/3387813890","heartbeat_back_addr":"172.21.15.184:6823/3387813890","heartbeat_front_addr":"172.21.15.184:6821/3387813890","state":["exists","up"]},{"osd":5,"uuid":"32d5a900-da3d-412c-bc19-13861a2368d8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6808","nonce":2680916017},{"type":"v1","addr":"172.21.15.184:6809","nonce":2680916017}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6810","nonce":2680916017},{"type":"v1","addr":"172.21.15.184:6811","nonce":2680916017}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6814","nonce":2680916017},{"type":"v1","addr":"172.21.15.184:6815","nonce":2680916017}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6812","nonce":2680916017},{"type":"v1","addr":"172.21.15.184:6813","nonce":2680916017}]},"public_addr":"172.21.15.184:6809/2680916017","cluster_addr":"172.21.15.184:6811/2680916017","heartbeat_back_addr":"172.21.15.184:6815/2680916017","heartbeat_front_addr":"172.21.15.184:6813/2680916017","state":["exists","up"]},{"osd":6,"uuid":"cde6f39e-aa1b-4260-ac19-774ee6aaca72","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6824","nonce":2183120457},{"type":"v1","addr":"172.21.15.184:6825","nonce":2183120457}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6826","nonce":2183120457},{"type":"v1","addr":"172.21.15.184:6827","nonce":2183120457}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6830","nonce":2183120457},{"type":"v1","addr":"172.21.15.184:6831","nonce":2183120457}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6828","nonce":2183120457},{"type":"v1","addr":"172.21.15.184:6829","nonce":2183120457}]},"public_addr":"172.21.15.184:6825/2183120457","cluster_addr":"172.21.15.184:6827/2183120457","heartbeat_back_addr":"172.21.15.184:6831/2183120457","heartbeat_front_addr":"172.21.15.184:6829/2183120457","state":["exists","up"]},{"osd":7,"uuid":"4f381778-eee1-47c6-804f-df0e6b33e313","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6800","nonce":4102957285},{"type":"v1","addr":"172.21.15.184:6801","nonce":4102957285}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6802","nonce":4102957285},{"type":"v1","addr":"172.21.15.184:6803","nonce":4102957285}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6806","nonce":4102957285},{"type":"v1","addr":"172.21.15.184:6807","nonce":4102957285}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6804","nonce":4102957285},{"type":"v1","addr":"172.21.15.184:6805","nonce":4102957285}]},"public_addr":"172.21.15.184:6801/4102957285","cluster_addr":"172.21.15.184:6803/4102957285","heartbeat_back_addr":"172.21.15.184:6807/4102957285","heartbeat_front_addr":"172.21.15.184:6805/4102957285","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:35.972210+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.222007+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.198617+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.385663+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.358935+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.156066+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.752973+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:35.682846+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T15:44:15.880 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:15.876+0000 7fa5ae679640 1 -- 172.21.15.123:0/1363143029 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fa588073070 msgr2=0x7fa588075530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:44:15.880 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:15.876+0000 7fa5ae679640 1 --2- 172.21.15.123:0/1363143029 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fa588073070 0x7fa588075530 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fa598001230 tx=0x7fa598009040 comp rx=0 tx=0).stop 2024-09-16T15:44:15.880 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:15.876+0000 7fa5ae679640 1 -- 172.21.15.123:0/1363143029 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa5a815bca0 msgr2=0x7fa5a8159f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:44:15.881 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:15.876+0000 7fa5ae679640 1 --2- 172.21.15.123:0/1363143029 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa5a815bca0 0x7fa5a8159f30 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7fa5a001c530 tx=0x7fa5a0005b00 comp rx=0 tx=0).stop 2024-09-16T15:44:15.881 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:15.876+0000 7fa5ae679640 1 -- 172.21.15.123:0/1363143029 shutdown_connections 2024-09-16T15:44:15.881 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:15.876+0000 7fa5ae679640 1 --2- 172.21.15.123:0/1363143029 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fa588073070 0x7fa588075530 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:15.881 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:15.876+0000 7fa5ae679640 1 --2- 172.21.15.123:0/1363143029 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa5a815bca0 0x7fa5a8159f30 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:15.881 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:15.876+0000 7fa5ae679640 1 --2- 172.21.15.123:0/1363143029 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fa5a8074600 0x7fa5a8151940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:15.881 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:15.876+0000 7fa5ae679640 1 --2- 172.21.15.123:0/1363143029 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa5a80726c0 0x7fa5a81f94a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:15.881 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:15.876+0000 7fa5ae679640 1 -- 172.21.15.123:0/1363143029 >> 172.21.15.123:0/1363143029 conn(0x7fa5a80cfb10 msgr2=0x7fa5a80b44f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:44:15.881 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:15.876+0000 7fa5ae679640 1 -- 172.21.15.123:0/1363143029 shutdown_connections 2024-09-16T15:44:15.881 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:15.876+0000 7fa5ae679640 1 -- 172.21.15.123:0/1363143029 wait complete. 2024-09-16T15:44:15.898 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-16T15:44:16.020 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:16.016+0000 7f9d956ae640 1 -- 172.21.15.123:0/3693604009 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9d70003a60 msgr2=0x7f9d70003e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:44:16.020 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:16.016+0000 7f9d956ae640 1 --2- 172.21.15.123:0/3693604009 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9d70003a60 0x7f9d70003e80 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f9d8800c640 tx=0x7f9d8801d2a0 comp rx=0 tx=0).stop 2024-09-16T15:44:16.020 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:16.016+0000 7f9d8f7fe640 1 -- 172.21.15.123:0/3693604009 reap_dead start 2024-09-16T15:44:16.021 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:16.016+0000 7f9d956ae640 1 -- 172.21.15.123:0/3693604009 shutdown_connections 2024-09-16T15:44:16.021 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:16.016+0000 7f9d956ae640 1 -- 172.21.15.123:0/3693604009 >> 172.21.15.123:0/3693604009 conn(0x7f9d900cfb10 msgr2=0x7f9d900cff20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:44:16.021 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:16.016+0000 7f9d956ae640 1 -- 172.21.15.123:0/3693604009 shutdown_connections 2024-09-16T15:44:16.021 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:16.016+0000 7f9d956ae640 1 -- 172.21.15.123:0/3693604009 wait complete. 2024-09-16T15:44:16.021 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:16.016+0000 7f9d956ae640 1 Processor -- start 2024-09-16T15:44:16.021 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:16.016+0000 7f9d956ae640 1 -- start start 2024-09-16T15:44:16.023 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:16.016+0000 7f9d956ae640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f9d70003a60 0x7f9d901f7920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:44:16.023 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:16.016+0000 7f9d956ae640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9d901fb940 0x7f9d900ae0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:44:16.023 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:16.016+0000 7f9d956ae640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f9d90166660 0x7f9d900aea40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:44:16.024 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:16.016+0000 7f9d956ae640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f9d9015adf0 con 0x7f9d901fb940 2024-09-16T15:44:16.024 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:16.016+0000 7f9d956ae640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f9d9015ac70 con 0x7f9d90166660 2024-09-16T15:44:16.024 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:16.016+0000 7f9d956ae640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f9d9015af70 con 0x7f9d70003a60 2024-09-16T15:44:16.024 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:16.016+0000 7f9d8e7fc640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9d901fb940 0x7f9d900ae0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:44:16.024 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:16.016+0000 7f9d8e7fc640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9d901fb940 0x7f9d900ae0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:47162/0 (socket says 172.21.15.123:47162) 2024-09-16T15:44:16.024 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:16.016+0000 7f9d8e7fc640 1 -- 172.21.15.123:0/555962310 learned_addr learned my addr 172.21.15.123:0/555962310 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:44:16.024 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:16.016+0000 7f9d8effd640 1 --2- 172.21.15.123:0/555962310 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f9d70003a60 0x7f9d901f7920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:44:16.024 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:16.016+0000 7f9d8f7fe640 1 --2- 172.21.15.123:0/555962310 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f9d90166660 0x7f9d900aea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:44:16.024 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:16.016+0000 7f9d8e7fc640 1 -- 172.21.15.123:0/555962310 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f9d70003a60 msgr2=0x7f9d901f7920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:44:16.024 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:16.016+0000 7f9d8e7fc640 1 --2- 172.21.15.123:0/555962310 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f9d70003a60 0x7f9d901f7920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:16.024 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:16.016+0000 7f9d8e7fc640 1 -- 172.21.15.123:0/555962310 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f9d90166660 msgr2=0x7f9d900aea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:44:16.024 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:16.016+0000 7f9d8e7fc640 1 --2- 172.21.15.123:0/555962310 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f9d90166660 0x7f9d900aea40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:16.025 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:16.016+0000 7f9d8e7fc640 1 -- 172.21.15.123:0/555962310 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9d901f88f0 con 0x7f9d901fb940 2024-09-16T15:44:16.025 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:16.016+0000 7f9d8f7fe640 1 --2- 172.21.15.123:0/555962310 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f9d90166660 0x7f9d900aea40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T15:44:16.025 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:16.020+0000 7f9d8e7fc640 1 --2- 172.21.15.123:0/555962310 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9d901fb940 0x7f9d900ae0b0 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f9d78011a50 tx=0x7f9d78011f10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:44:16.026 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:16.020+0000 7f9d877fe640 1 -- 172.21.15.123:0/555962310 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d7801f070 con 0x7f9d901fb940 2024-09-16T15:44:16.026 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:16.020+0000 7f9d956ae640 1 -- 172.21.15.123:0/555962310 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9d900ae5f0 con 0x7f9d901fb940 2024-09-16T15:44:16.026 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:16.020+0000 7f9d956ae640 1 -- 172.21.15.123:0/555962310 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9d901f8330 con 0x7f9d901fb940 2024-09-16T15:44:16.026 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:16.020+0000 7f9d877fe640 1 -- 172.21.15.123:0/555962310 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f9d7800fe80 con 0x7f9d901fb940 2024-09-16T15:44:16.026 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:16.020+0000 7f9d877fe640 1 -- 172.21.15.123:0/555962310 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d78012670 con 0x7f9d901fb940 2024-09-16T15:44:16.026 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:16.020+0000 7f9d956ae640 1 -- 172.21.15.123:0/555962310 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9d680052a0 con 0x7f9d901fb940 2024-09-16T15:44:16.026 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:16.020+0000 7f9d877fe640 1 -- 172.21.15.123:0/555962310 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 7) ==== 97233+0+0 (secure 0 0 0) 0x7f9d78022070 con 0x7f9d901fb940 2024-09-16T15:44:16.027 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:16.020+0000 7f9d877fe640 1 --2- 172.21.15.123:0/555962310 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f9d74073070 0x7f9d74075530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:44:16.027 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:16.020+0000 7f9d877fe640 1 -- 172.21.15.123:0/555962310 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(43..43 src has 1..43) ==== 5709+0+0 (secure 0 0 0) 0x7f9d78096d70 con 0x7f9d901fb940 2024-09-16T15:44:16.027 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:16.020+0000 7f9d8effd640 1 --2- 172.21.15.123:0/555962310 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f9d74073070 0x7f9d74075530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:44:16.027 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:16.024+0000 7f9d8effd640 1 --2- 172.21.15.123:0/555962310 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f9d74073070 0x7f9d74075530 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f9d880ac000 tx=0x7f9d880af040 comp rx=0 tx=0).ready entity=mgr.4135 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:44:16.030 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:16.024+0000 7f9d877fe640 1 -- 172.21.15.123:0/555962310 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f9d7805f980 con 0x7f9d901fb940 2024-09-16T15:44:16.204 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:16.200+0000 7f9d956ae640 1 -- 172.21.15.123:0/555962310 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f9d68005860 con 0x7f9d901fb940 2024-09-16T15:44:16.206 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:16.200+0000 7f9d877fe640 1 -- 172.21.15.123:0/555962310 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v43) ==== 74+0+16746 (secure 0 0 0) 0x7f9d7805f280 con 0x7f9d901fb940 2024-09-16T15:44:16.206 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-09-16T15:44:16.207 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":43,"fsid":"56077f89-18c8-486f-ae7e-eecc0bb7822c","created":"2024-09-16T15:42:26.364892+0000","modified":"2024-09-16T15:44:14.853278+0000","last_up_change":"2024-09-16T15:42:38.707234+0000","last_in_change":"2024-09-16T15:42:29.423662+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-16T15:42:39.287764+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_metadata","create_time":"2024-09-16T15:44:12.463367+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"43","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_data","create_time":"2024-09-16T15:44:13.817170+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"43","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9e7c1955-a607-4671-9be7-f2ef760ceeaa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6800","nonce":3882044969},{"type":"v1","addr":"172.21.15.123:6801","nonce":3882044969}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6802","nonce":3882044969},{"type":"v1","addr":"172.21.15.123:6803","nonce":3882044969}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6806","nonce":3882044969},{"type":"v1","addr":"172.21.15.123:6807","nonce":3882044969}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6804","nonce":3882044969},{"type":"v1","addr":"172.21.15.123:6805","nonce":3882044969}]},"public_addr":"172.21.15.123:6801/3882044969","cluster_addr":"172.21.15.123:6803/3882044969","heartbeat_back_addr":"172.21.15.123:6807/3882044969","heartbeat_front_addr":"172.21.15.123:6805/3882044969","state":["exists","up"]},{"osd":1,"uuid":"c540cfde-0977-4742-9b17-fd634668e010","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6816","nonce":4037001850},{"type":"v1","addr":"172.21.15.123:6817","nonce":4037001850}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6818","nonce":4037001850},{"type":"v1","addr":"172.21.15.123:6819","nonce":4037001850}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6822","nonce":4037001850},{"type":"v1","addr":"172.21.15.123:6823","nonce":4037001850}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6820","nonce":4037001850},{"type":"v1","addr":"172.21.15.123:6821","nonce":4037001850}]},"public_addr":"172.21.15.123:6817/4037001850","cluster_addr":"172.21.15.123:6819/4037001850","heartbeat_back_addr":"172.21.15.123:6823/4037001850","heartbeat_front_addr":"172.21.15.123:6821/4037001850","state":["exists","up"]},{"osd":2,"uuid":"9864a496-f169-486b-95c4-1d59162793a7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6824","nonce":1332143088},{"type":"v1","addr":"172.21.15.123:6825","nonce":1332143088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6826","nonce":1332143088},{"type":"v1","addr":"172.21.15.123:6827","nonce":1332143088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6830","nonce":1332143088},{"type":"v1","addr":"172.21.15.123:6831","nonce":1332143088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6828","nonce":1332143088},{"type":"v1","addr":"172.21.15.123:6829","nonce":1332143088}]},"public_addr":"172.21.15.123:6825/1332143088","cluster_addr":"172.21.15.123:6827/1332143088","heartbeat_back_addr":"172.21.15.123:6831/1332143088","heartbeat_front_addr":"172.21.15.123:6829/1332143088","state":["exists","up"]},{"osd":3,"uuid":"c6d1f817-f074-4325-95e1-578eba13edda","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6808","nonce":988586998},{"type":"v1","addr":"172.21.15.123:6809","nonce":988586998}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6810","nonce":988586998},{"type":"v1","addr":"172.21.15.123:6811","nonce":988586998}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6814","nonce":988586998},{"type":"v1","addr":"172.21.15.123:6815","nonce":988586998}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6812","nonce":988586998},{"type":"v1","addr":"172.21.15.123:6813","nonce":988586998}]},"public_addr":"172.21.15.123:6809/988586998","cluster_addr":"172.21.15.123:6811/988586998","heartbeat_back_addr":"172.21.15.123:6815/988586998","heartbeat_front_addr":"172.21.15.123:6813/988586998","state":["exists","up"]},{"osd":4,"uuid":"200a8df4-50ef-40a5-b48a-58981269c086","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6816","nonce":3387813890},{"type":"v1","addr":"172.21.15.184:6817","nonce":3387813890}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6818","nonce":3387813890},{"type":"v1","addr":"172.21.15.184:6819","nonce":3387813890}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6822","nonce":3387813890},{"type":"v1","addr":"172.21.15.184:6823","nonce":3387813890}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6820","nonce":3387813890},{"type":"v1","addr":"172.21.15.184:6821","nonce":3387813890}]},"public_addr":"172.21.15.184:6817/3387813890","cluster_addr":"172.21.15.184:6819/3387813890","heartbeat_back_addr":"172.21.15.184:6823/3387813890","heartbeat_front_addr":"172.21.15.184:6821/3387813890","state":["exists","up"]},{"osd":5,"uuid":"32d5a900-da3d-412c-bc19-13861a2368d8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6808","nonce":2680916017},{"type":"v1","addr":"172.21.15.184:6809","nonce":2680916017}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6810","nonce":2680916017},{"type":"v1","addr":"172.21.15.184:6811","nonce":2680916017}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6814","nonce":2680916017},{"type":"v1","addr":"172.21.15.184:6815","nonce":2680916017}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6812","nonce":2680916017},{"type":"v1","addr":"172.21.15.184:6813","nonce":2680916017}]},"public_addr":"172.21.15.184:6809/2680916017","cluster_addr":"172.21.15.184:6811/2680916017","heartbeat_back_addr":"172.21.15.184:6815/2680916017","heartbeat_front_addr":"172.21.15.184:6813/2680916017","state":["exists","up"]},{"osd":6,"uuid":"cde6f39e-aa1b-4260-ac19-774ee6aaca72","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6824","nonce":2183120457},{"type":"v1","addr":"172.21.15.184:6825","nonce":2183120457}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6826","nonce":2183120457},{"type":"v1","addr":"172.21.15.184:6827","nonce":2183120457}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6830","nonce":2183120457},{"type":"v1","addr":"172.21.15.184:6831","nonce":2183120457}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6828","nonce":2183120457},{"type":"v1","addr":"172.21.15.184:6829","nonce":2183120457}]},"public_addr":"172.21.15.184:6825/2183120457","cluster_addr":"172.21.15.184:6827/2183120457","heartbeat_back_addr":"172.21.15.184:6831/2183120457","heartbeat_front_addr":"172.21.15.184:6829/2183120457","state":["exists","up"]},{"osd":7,"uuid":"4f381778-eee1-47c6-804f-df0e6b33e313","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6800","nonce":4102957285},{"type":"v1","addr":"172.21.15.184:6801","nonce":4102957285}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6802","nonce":4102957285},{"type":"v1","addr":"172.21.15.184:6803","nonce":4102957285}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6806","nonce":4102957285},{"type":"v1","addr":"172.21.15.184:6807","nonce":4102957285}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6804","nonce":4102957285},{"type":"v1","addr":"172.21.15.184:6805","nonce":4102957285}]},"public_addr":"172.21.15.184:6801/4102957285","cluster_addr":"172.21.15.184:6803/4102957285","heartbeat_back_addr":"172.21.15.184:6807/4102957285","heartbeat_front_addr":"172.21.15.184:6805/4102957285","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:35.972210+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.222007+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.198617+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.385663+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.358935+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.156066+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.752973+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:35.682846+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T15:44:16.209 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:16.204+0000 7f9d956ae640 1 -- 172.21.15.123:0/555962310 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f9d74073070 msgr2=0x7f9d74075530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:44:16.209 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:16.204+0000 7f9d956ae640 1 --2- 172.21.15.123:0/555962310 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f9d74073070 0x7f9d74075530 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f9d880ac000 tx=0x7f9d880af040 comp rx=0 tx=0).stop 2024-09-16T15:44:16.209 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:16.204+0000 7f9d956ae640 1 -- 172.21.15.123:0/555962310 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9d901fb940 msgr2=0x7f9d900ae0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:44:16.209 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:16.204+0000 7f9d956ae640 1 --2- 172.21.15.123:0/555962310 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9d901fb940 0x7f9d900ae0b0 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f9d78011a50 tx=0x7f9d78011f10 comp rx=0 tx=0).stop 2024-09-16T15:44:16.209 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:16.204+0000 7f9d956ae640 1 -- 172.21.15.123:0/555962310 shutdown_connections 2024-09-16T15:44:16.209 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:16.204+0000 7f9d956ae640 1 --2- 172.21.15.123:0/555962310 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f9d74073070 0x7f9d74075530 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:16.209 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:16.204+0000 7f9d956ae640 1 --2- 172.21.15.123:0/555962310 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f9d90166660 0x7f9d900aea40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:16.210 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:16.204+0000 7f9d956ae640 1 --2- 172.21.15.123:0/555962310 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9d901fb940 0x7f9d900ae0b0 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:16.210 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:16.204+0000 7f9d956ae640 1 --2- 172.21.15.123:0/555962310 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f9d70003a60 0x7f9d901f7920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:16.210 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:16.204+0000 7f9d956ae640 1 -- 172.21.15.123:0/555962310 >> 172.21.15.123:0/555962310 conn(0x7f9d900cfb10 msgr2=0x7f9d9015f810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:44:16.210 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:16.204+0000 7f9d956ae640 1 -- 172.21.15.123:0/555962310 shutdown_connections 2024-09-16T15:44:16.210 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:16.204+0000 7f9d956ae640 1 -- 172.21.15.123:0/555962310 wait complete. 2024-09-16T15:44:16.226 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-09-16T15:44:16.357 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:16.352+0000 7f39014b2640 1 -- 172.21.15.123:0/2051067171 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f38e8003a40 msgr2=0x7f38e8003e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:44:16.357 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:16.352+0000 7f39014b2640 1 --2- 172.21.15.123:0/2051067171 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f38e8003a40 0x7f38e8003e60 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f38ec006fd0 tx=0x7f38ec01d8b0 comp rx=0 tx=0).stop 2024-09-16T15:44:16.357 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:16.352+0000 7f38fb7fe640 1 -- 172.21.15.123:0/2051067171 reap_dead start 2024-09-16T15:44:16.357 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:16.352+0000 7f39014b2640 1 -- 172.21.15.123:0/2051067171 shutdown_connections 2024-09-16T15:44:16.357 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:16.352+0000 7f39014b2640 1 -- 172.21.15.123:0/2051067171 >> 172.21.15.123:0/2051067171 conn(0x7f38fc0b1710 msgr2=0x7f38fc0b1b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:44:16.357 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:16.352+0000 7f39014b2640 1 -- 172.21.15.123:0/2051067171 shutdown_connections 2024-09-16T15:44:16.357 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:16.352+0000 7f39014b2640 1 -- 172.21.15.123:0/2051067171 wait complete. 2024-09-16T15:44:16.358 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:16.352+0000 7f39014b2640 1 Processor -- start 2024-09-16T15:44:16.358 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:16.352+0000 7f39014b2640 1 -- start start 2024-09-16T15:44:16.361 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:16.356+0000 7f39014b2640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f38e8003a40 0x7f38fc0c96b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:44:16.361 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:16.356+0000 7f39014b2640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f38fc1f5610 0x7f38fc14fde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:44:16.361 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:16.356+0000 7f39014b2640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f38fc0c6c10 0x7f38fc0c70a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:44:16.361 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:16.356+0000 7f39014b2640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f38fc167540 con 0x7f38fc0c6c10 2024-09-16T15:44:16.361 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:16.356+0000 7f39014b2640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f38fc1673c0 con 0x7f38fc1f5610 2024-09-16T15:44:16.361 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:16.356+0000 7f39014b2640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f38fc1676c0 con 0x7f38e8003a40 2024-09-16T15:44:16.361 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:16.356+0000 7f38fb7fe640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f38fc0c6c10 0x7f38fc0c70a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:44:16.361 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:16.356+0000 7f38fb7fe640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f38fc0c6c10 0x7f38fc0c70a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:47186/0 (socket says 172.21.15.123:47186) 2024-09-16T15:44:16.361 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:16.356+0000 7f38fb7fe640 1 -- 172.21.15.123:0/964122929 learned_addr learned my addr 172.21.15.123:0/964122929 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:44:16.361 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:16.356+0000 7f38f27fc640 1 --2- 172.21.15.123:0/964122929 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f38fc1f5610 0x7f38fc14fde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:44:16.362 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:16.356+0000 7f38faffd640 1 --2- 172.21.15.123:0/964122929 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f38e8003a40 0x7f38fc0c96b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:44:16.362 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:16.356+0000 7f38fb7fe640 1 -- 172.21.15.123:0/964122929 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f38e8003a40 msgr2=0x7f38fc0c96b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:44:16.362 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:16.356+0000 7f38fb7fe640 1 --2- 172.21.15.123:0/964122929 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f38e8003a40 0x7f38fc0c96b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:16.362 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:16.356+0000 7f38fb7fe640 1 -- 172.21.15.123:0/964122929 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f38fc1f5610 msgr2=0x7f38fc14fde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:44:16.362 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:16.356+0000 7f38fb7fe640 1 --2- 172.21.15.123:0/964122929 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f38fc1f5610 0x7f38fc14fde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:16.362 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:16.356+0000 7f38fb7fe640 1 -- 172.21.15.123:0/964122929 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f38fc0b4150 con 0x7f38fc0c6c10 2024-09-16T15:44:16.362 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:16.356+0000 7f38faffd640 1 --2- 172.21.15.123:0/964122929 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f38e8003a40 0x7f38fc0c96b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T15:44:16.362 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:16.356+0000 7f38fb7fe640 1 --2- 172.21.15.123:0/964122929 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f38fc0c6c10 0x7f38fc0c70a0 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f38fc0749f0 tx=0x7f38f400fe20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:44:16.362 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:16.356+0000 7f38f3fff640 1 -- 172.21.15.123:0/964122929 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38f400dd10 con 0x7f38fc0c6c10 2024-09-16T15:44:16.362 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:16.356+0000 7f38f3fff640 1 -- 172.21.15.123:0/964122929 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f38f4011440 con 0x7f38fc0c6c10 2024-09-16T15:44:16.362 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:16.356+0000 7f39014b2640 1 -- 172.21.15.123:0/964122929 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f38fc1f5b10 con 0x7f38fc0c6c10 2024-09-16T15:44:16.362 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:16.356+0000 7f38f3fff640 1 -- 172.21.15.123:0/964122929 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38f4011890 con 0x7f38fc0c6c10 2024-09-16T15:44:16.363 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:16.356+0000 7f39014b2640 1 -- 172.21.15.123:0/964122929 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f38fc1f60d0 con 0x7f38fc0c6c10 2024-09-16T15:44:16.363 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:16.356+0000 7f39014b2640 1 -- 172.21.15.123:0/964122929 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f38fc0cedd0 con 0x7f38fc0c6c10 2024-09-16T15:44:16.364 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:16.356+0000 7f38f3fff640 1 -- 172.21.15.123:0/964122929 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 7) ==== 97233+0+0 (secure 0 0 0) 0x7f38f4011a30 con 0x7f38fc0c6c10 2024-09-16T15:44:16.364 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:16.356+0000 7f38f3fff640 1 --2- 172.21.15.123:0/964122929 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f38dc0730a0 0x7f38dc075560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:44:16.364 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:16.360+0000 7f38f3fff640 1 -- 172.21.15.123:0/964122929 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(43..43 src has 1..43) ==== 5709+0+0 (secure 0 0 0) 0x7f38f4094b50 con 0x7f38fc0c6c10 2024-09-16T15:44:16.364 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:16.360+0000 7f38faffd640 1 --2- 172.21.15.123:0/964122929 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f38dc0730a0 0x7f38dc075560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:44:16.364 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:16.360+0000 7f38faffd640 1 --2- 172.21.15.123:0/964122929 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f38dc0730a0 0x7f38dc075560 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f38ec0ab000 tx=0x7f38ec0a7040 comp rx=0 tx=0).ready entity=mgr.4135 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:44:16.367 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:16.364+0000 7f38f3fff640 1 -- 172.21.15.123:0/964122929 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f38f4021060 con 0x7f38fc0c6c10 2024-09-16T15:44:16.580 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:16.576+0000 7f39014b2640 1 -- 172.21.15.123:0/964122929 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) -- 0x7f38fc1510d0 con 0x7f38fc0c6c10 2024-09-16T15:44:17.474 INFO:tasks.ceph.mds.d.smithi184.stderr:2024-09-16T15:44:17.469+0000 7f4568ca1640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-09-16T15:44:17.500 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:17.496+0000 7f38f3fff640 1 -- 172.21.15.123:0/964122929 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v26) ==== 118+0+0 (secure 0 0 0) 0x7f38f405fa10 con 0x7f38fc0c6c10 2024-09-16T15:44:17.505 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:17.500+0000 7f39014b2640 1 -- 172.21.15.123:0/964122929 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f38dc0730a0 msgr2=0x7f38dc075560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:44:17.505 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:17.500+0000 7f39014b2640 1 --2- 172.21.15.123:0/964122929 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f38dc0730a0 0x7f38dc075560 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f38ec0ab000 tx=0x7f38ec0a7040 comp rx=0 tx=0).stop 2024-09-16T15:44:17.505 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:17.500+0000 7f39014b2640 1 -- 172.21.15.123:0/964122929 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f38fc0c6c10 msgr2=0x7f38fc0c70a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:44:17.505 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:17.500+0000 7f39014b2640 1 --2- 172.21.15.123:0/964122929 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f38fc0c6c10 0x7f38fc0c70a0 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f38fc0749f0 tx=0x7f38f400fe20 comp rx=0 tx=0).stop 2024-09-16T15:44:17.505 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:17.500+0000 7f39014b2640 1 -- 172.21.15.123:0/964122929 shutdown_connections 2024-09-16T15:44:17.505 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:17.500+0000 7f39014b2640 1 --2- 172.21.15.123:0/964122929 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f38dc0730a0 0x7f38dc075560 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:17.505 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:17.500+0000 7f39014b2640 1 --2- 172.21.15.123:0/964122929 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f38fc0c6c10 0x7f38fc0c70a0 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:17.505 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:17.500+0000 7f39014b2640 1 --2- 172.21.15.123:0/964122929 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f38fc1f5610 0x7f38fc14fde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:17.505 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:17.500+0000 7f39014b2640 1 --2- 172.21.15.123:0/964122929 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f38e8003a40 0x7f38fc0c96b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:17.505 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:17.500+0000 7f39014b2640 1 -- 172.21.15.123:0/964122929 >> 172.21.15.123:0/964122929 conn(0x7f38fc0b1710 msgr2=0x7f38fc15aa60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:44:17.505 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:17.500+0000 7f39014b2640 1 -- 172.21.15.123:0/964122929 shutdown_connections 2024-09-16T15:44:17.505 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:17.500+0000 7f39014b2640 1 -- 172.21.15.123:0/964122929 wait complete. 2024-09-16T15:44:17.524 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-09-16T15:44:17.524 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-09-16T15:44:17.629 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a has been restored 2024-09-16T15:44:17.629 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.c has been restored 2024-09-16T15:44:17.629 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b has been restored 2024-09-16T15:44:17.656 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:17.652+0000 7fd2db1f0640 1 -- 172.21.15.123:0/2272324560 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd2d41615c0 msgr2=0x7fd2d40d0850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:44:17.657 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:17.652+0000 7fd2db1f0640 1 --2- 172.21.15.123:0/2272324560 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd2d41615c0 0x7fd2d40d0850 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7fd2d40ce420 tx=0x7fd2c401c830 comp rx=0 tx=0).stop 2024-09-16T15:44:17.657 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:17.652+0000 7fd2db1f0640 1 -- 172.21.15.123:0/2272324560 shutdown_connections 2024-09-16T15:44:17.657 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:17.652+0000 7fd2db1f0640 1 --2- 172.21.15.123:0/2272324560 >> v2:172.21.15.184:3300/0 conn(0x7fd2d40d0d90 0x7fd2d40b3eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:17.657 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:17.652+0000 7fd2db1f0640 1 --2- 172.21.15.123:0/2272324560 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd2d41615c0 0x7fd2d40d0850 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:17.657 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:17.652+0000 7fd2db1f0640 1 -- 172.21.15.123:0/2272324560 >> 172.21.15.123:0/2272324560 conn(0x7fd2d40d3e40 msgr2=0x7fd2d40d4220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:44:17.657 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:17.652+0000 7fd2db1f0640 1 -- 172.21.15.123:0/2272324560 shutdown_connections 2024-09-16T15:44:17.657 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:17.652+0000 7fd2db1f0640 1 -- 172.21.15.123:0/2272324560 wait complete. 2024-09-16T15:44:17.658 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:17.652+0000 7fd2db1f0640 1 Processor -- start 2024-09-16T15:44:17.658 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:17.652+0000 7fd2db1f0640 1 -- start start 2024-09-16T15:44:17.659 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:17.652+0000 7fd2db1f0640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd2d40d0d90 0x7fd2d41fbd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:44:17.659 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:17.652+0000 7fd2db1f0640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd2d40d2120 0x7fd2d40b2520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:44:17.660 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:17.652+0000 7fd2db1f0640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fd2d41615c0 0x7fd2d41529e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:44:17.660 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:17.652+0000 7fd2db1f0640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fd2d40b43f0 con 0x7fd2d40d2120 2024-09-16T15:44:17.660 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:17.652+0000 7fd2db1f0640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fd2d40b3120 con 0x7fd2d40d0d90 2024-09-16T15:44:17.660 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:17.652+0000 7fd2db1f0640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fd2d40b4570 con 0x7fd2d41615c0 2024-09-16T15:44:17.660 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:17.652+0000 7fd2d3fff640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd2d40d2120 0x7fd2d40b2520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:44:17.660 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:17.652+0000 7fd2d3fff640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd2d40d2120 0x7fd2d40b2520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:47202/0 (socket says 172.21.15.123:47202) 2024-09-16T15:44:17.660 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:17.652+0000 7fd2d3fff640 1 -- 172.21.15.123:0/2150167705 learned_addr learned my addr 172.21.15.123:0/2150167705 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:44:17.660 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:17.652+0000 7fd2d3fff640 1 -- 172.21.15.123:0/2150167705 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fd2d41615c0 msgr2=0x7fd2d41529e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:44:17.660 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:17.652+0000 7fd2d3fff640 1 --2- 172.21.15.123:0/2150167705 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fd2d41615c0 0x7fd2d41529e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:17.660 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:17.652+0000 7fd2d3fff640 1 -- 172.21.15.123:0/2150167705 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd2d40d0d90 msgr2=0x7fd2d41fbd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:44:17.660 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:17.652+0000 7fd2d3fff640 1 --2- 172.21.15.123:0/2150167705 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd2d40d0d90 0x7fd2d41fbd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:17.660 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:17.652+0000 7fd2d3fff640 1 -- 172.21.15.123:0/2150167705 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd2d41539d0 con 0x7fd2d40d2120 2024-09-16T15:44:17.660 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:17.652+0000 7fd2d3fff640 1 --2- 172.21.15.123:0/2150167705 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd2d40d2120 0x7fd2d40b2520 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7fd2c800c490 tx=0x7fd2c8001e80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:44:17.661 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:17.656+0000 7fd2d17fa640 1 -- 172.21.15.123:0/2150167705 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2c8017070 con 0x7fd2d40d2120 2024-09-16T15:44:17.661 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:17.656+0000 7fd2d17fa640 1 -- 172.21.15.123:0/2150167705 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fd2c8003600 con 0x7fd2d40d2120 2024-09-16T15:44:17.661 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:17.656+0000 7fd2db1f0640 1 -- 172.21.15.123:0/2150167705 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd2d4153cb0 con 0x7fd2d40d2120 2024-09-16T15:44:17.661 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:17.656+0000 7fd2d17fa640 1 -- 172.21.15.123:0/2150167705 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2c8003a90 con 0x7fd2d40d2120 2024-09-16T15:44:17.661 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:17.656+0000 7fd2db1f0640 1 -- 172.21.15.123:0/2150167705 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd2d4154550 con 0x7fd2d40d2120 2024-09-16T15:44:17.662 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:17.656+0000 7fd2db1f0640 1 -- 172.21.15.123:0/2150167705 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd2d41fc2b0 con 0x7fd2d40d2120 2024-09-16T15:44:17.662 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:17.656+0000 7fd2d17fa640 1 -- 172.21.15.123:0/2150167705 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 7) ==== 97233+0+0 (secure 0 0 0) 0x7fd2c801a070 con 0x7fd2d40d2120 2024-09-16T15:44:17.662 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:17.656+0000 7fd2d17fa640 1 --2- 172.21.15.123:0/2150167705 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fd2c00730a0 0x7fd2c0075560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:44:17.662 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:17.656+0000 7fd2d17fa640 1 -- 172.21.15.123:0/2150167705 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(44..44 src has 1..44) ==== 5709+0+0 (secure 0 0 0) 0x7fd2c80935f0 con 0x7fd2d40d2120 2024-09-16T15:44:17.662 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:17.656+0000 7fd2d8f8c640 1 --2- 172.21.15.123:0/2150167705 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fd2c00730a0 0x7fd2c0075560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:44:17.662 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:17.656+0000 7fd2d8f8c640 1 --2- 172.21.15.123:0/2150167705 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fd2c00730a0 0x7fd2c0075560 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fd2c40af000 tx=0x7fd2c40195d0 comp rx=0 tx=0).ready entity=mgr.4135 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:44:17.665 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:17.660+0000 7fd2d17fa640 1 -- 172.21.15.123:0/2150167705 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7fd2d41fc2b0 con 0x7fd2d40d2120 2024-09-16T15:44:17.860 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:17.856+0000 7fd2db1f0640 1 -- 172.21.15.123:0/2150167705 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) -- 0x7fd2d40d0850 con 0x7fd2d40d2120 2024-09-16T15:44:18.521 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:18.516+0000 7fd2d17fa640 1 -- 172.21.15.123:0/2150167705 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v27) ==== 122+0+0 (secure 0 0 0) 0x7fd2c805e420 con 0x7fd2d40d2120 2024-09-16T15:44:18.525 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:18.520+0000 7fd2db1f0640 1 -- 172.21.15.123:0/2150167705 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fd2c00730a0 msgr2=0x7fd2c0075560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:44:18.525 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:18.520+0000 7fd2db1f0640 1 --2- 172.21.15.123:0/2150167705 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fd2c00730a0 0x7fd2c0075560 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fd2c40af000 tx=0x7fd2c40195d0 comp rx=0 tx=0).stop 2024-09-16T15:44:18.525 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:18.520+0000 7fd2db1f0640 1 -- 172.21.15.123:0/2150167705 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd2d40d2120 msgr2=0x7fd2d40b2520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:44:18.525 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:18.520+0000 7fd2db1f0640 1 --2- 172.21.15.123:0/2150167705 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd2d40d2120 0x7fd2d40b2520 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7fd2c800c490 tx=0x7fd2c8001e80 comp rx=0 tx=0).stop 2024-09-16T15:44:18.525 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:18.520+0000 7fd2db1f0640 1 -- 172.21.15.123:0/2150167705 shutdown_connections 2024-09-16T15:44:18.525 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:18.520+0000 7fd2db1f0640 1 --2- 172.21.15.123:0/2150167705 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fd2c00730a0 0x7fd2c0075560 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:18.525 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:18.520+0000 7fd2db1f0640 1 --2- 172.21.15.123:0/2150167705 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fd2d41615c0 0x7fd2d41529e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:18.526 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:18.520+0000 7fd2db1f0640 1 --2- 172.21.15.123:0/2150167705 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd2d40d2120 0x7fd2d40b2520 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:18.526 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:18.520+0000 7fd2db1f0640 1 --2- 172.21.15.123:0/2150167705 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd2d40d0d90 0x7fd2d41fbd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:18.526 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:18.520+0000 7fd2db1f0640 1 -- 172.21.15.123:0/2150167705 >> 172.21.15.123:0/2150167705 conn(0x7fd2d40d3e40 msgr2=0x7fd2d415e380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:44:18.526 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:18.520+0000 7fd2db1f0640 1 -- 172.21.15.123:0/2150167705 shutdown_connections 2024-09-16T15:44:18.526 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:18.520+0000 7fd2db1f0640 1 -- 172.21.15.123:0/2150167705 wait complete. 2024-09-16T15:44:18.543 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-09-16T15:44:18.543 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-16T15:44:18.672 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:18.668+0000 7f98f4e26640 1 -- 172.21.15.123:0/3180530867 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f98f0074600 msgr2=0x7f98f006e750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:44:18.672 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:18.668+0000 7f98f4e26640 1 --2- 172.21.15.123:0/3180530867 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f98f0074600 0x7f98f006e750 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f98e0009900 tx=0x7f98e001c820 comp rx=0 tx=0).stop 2024-09-16T15:44:18.672 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:18.668+0000 7f98f4e26640 1 -- 172.21.15.123:0/3180530867 shutdown_connections 2024-09-16T15:44:18.672 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:18.668+0000 7f98f4e26640 1 --2- 172.21.15.123:0/3180530867 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f98f0074600 0x7f98f006e750 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:18.673 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:18.668+0000 7f98f4e26640 1 --2- 172.21.15.123:0/3180530867 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f98f00726c0 0x7f98f00d0630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:18.673 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:18.668+0000 7f98f4e26640 1 -- 172.21.15.123:0/3180530867 >> 172.21.15.123:0/3180530867 conn(0x7f98f00cfb10 msgr2=0x7f98f00cff20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:44:18.673 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:18.668+0000 7f98f4e26640 1 -- 172.21.15.123:0/3180530867 shutdown_connections 2024-09-16T15:44:18.673 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:18.668+0000 7f98f4e26640 1 -- 172.21.15.123:0/3180530867 wait complete. 2024-09-16T15:44:18.673 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:18.668+0000 7f98f4e26640 1 Processor -- start 2024-09-16T15:44:18.673 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:18.668+0000 7f98f4e26640 1 -- start start 2024-09-16T15:44:18.674 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:18.668+0000 7f98f4e26640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f98f00726c0 0x7f98f014b8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:44:18.675 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:18.668+0000 7f98f4e26640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f98f0074600 0x7f98f0152f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:44:18.675 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:18.668+0000 7f98f4e26640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f98f015bca0 0x7f98f01f9600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:44:18.675 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:18.668+0000 7f98f4e26640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f98f0168a50 con 0x7f98f00726c0 2024-09-16T15:44:18.675 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:18.668+0000 7f98f4e26640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f98f006f9b0 con 0x7f98f015bca0 2024-09-16T15:44:18.675 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:18.668+0000 7f98f4e26640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f98f0168bd0 con 0x7f98f0074600 2024-09-16T15:44:18.675 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:18.668+0000 7f98ee59c640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f98f00726c0 0x7f98f014b8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:44:18.675 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:18.668+0000 7f98ee59c640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f98f00726c0 0x7f98f014b8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:47218/0 (socket says 172.21.15.123:47218) 2024-09-16T15:44:18.675 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:18.668+0000 7f98ee59c640 1 -- 172.21.15.123:0/2623250320 learned_addr learned my addr 172.21.15.123:0/2623250320 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:44:18.675 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:18.668+0000 7f98edd9b640 1 --2- 172.21.15.123:0/2623250320 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f98f0074600 0x7f98f0152f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:44:18.675 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:18.668+0000 7f98eed9d640 1 --2- 172.21.15.123:0/2623250320 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f98f015bca0 0x7f98f01f9600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:44:18.676 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:18.672+0000 7f98ee59c640 1 -- 172.21.15.123:0/2623250320 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f98f0074600 msgr2=0x7f98f0152f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:44:18.677 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:18.672+0000 7f98ee59c640 1 --2- 172.21.15.123:0/2623250320 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f98f0074600 0x7f98f0152f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:18.677 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:18.672+0000 7f98ee59c640 1 -- 172.21.15.123:0/2623250320 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f98f015bca0 msgr2=0x7f98f01f9600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:44:18.677 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:18.672+0000 7f98ee59c640 1 --2- 172.21.15.123:0/2623250320 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f98f015bca0 0x7f98f01f9600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:18.677 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:18.672+0000 7f98ee59c640 1 -- 172.21.15.123:0/2623250320 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f98f0153780 con 0x7f98f00726c0 2024-09-16T15:44:18.677 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:18.672+0000 7f98ee59c640 1 --2- 172.21.15.123:0/2623250320 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f98f00726c0 0x7f98f014b8e0 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f98e800ba80 tx=0x7f98e800bf40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:44:18.677 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:18.672+0000 7f98deffd640 1 -- 172.21.15.123:0/2623250320 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98e800cc00 con 0x7f98f00726c0 2024-09-16T15:44:18.677 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:18.672+0000 7f98deffd640 1 -- 172.21.15.123:0/2623250320 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f98e8003980 con 0x7f98f00726c0 2024-09-16T15:44:18.677 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:18.672+0000 7f98f4e26640 1 -- 172.21.15.123:0/2623250320 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f98f0153a60 con 0x7f98f00726c0 2024-09-16T15:44:18.677 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:18.672+0000 7f98deffd640 1 -- 172.21.15.123:0/2623250320 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98e8003e10 con 0x7f98f00726c0 2024-09-16T15:44:18.677 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:18.672+0000 7f98f4e26640 1 -- 172.21.15.123:0/2623250320 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f98f01f90e0 con 0x7f98f00726c0 2024-09-16T15:44:18.677 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:18.672+0000 7f98f4e26640 1 -- 172.21.15.123:0/2623250320 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f98f00d0630 con 0x7f98f00726c0 2024-09-16T15:44:18.678 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:18.672+0000 7f98deffd640 1 -- 172.21.15.123:0/2623250320 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 7) ==== 97233+0+0 (secure 0 0 0) 0x7f98e8006d10 con 0x7f98f00726c0 2024-09-16T15:44:18.678 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:18.672+0000 7f98deffd640 1 --2- 172.21.15.123:0/2623250320 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f98d0073070 0x7f98d0075530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:44:18.678 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:18.672+0000 7f98deffd640 1 -- 172.21.15.123:0/2623250320 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(44..44 src has 1..44) ==== 5709+0+0 (secure 0 0 0) 0x7f98e8094550 con 0x7f98f00726c0 2024-09-16T15:44:18.679 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:18.676+0000 7f98edd9b640 1 --2- 172.21.15.123:0/2623250320 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f98d0073070 0x7f98d0075530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:44:18.680 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:18.676+0000 7f98edd9b640 1 --2- 172.21.15.123:0/2623250320 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f98d0073070 0x7f98d0075530 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f98f0159e10 tx=0x7f98e400b390 comp rx=0 tx=0).ready entity=mgr.4135 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:44:18.683 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:18.680+0000 7f98deffd640 1 -- 172.21.15.123:0/2623250320 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f98e805ef00 con 0x7f98f00726c0 2024-09-16T15:44:18.892 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:18.888+0000 7f98f4e26640 1 -- 172.21.15.123:0/2623250320 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f98f015f630 con 0x7f98f00726c0 2024-09-16T15:44:18.893 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:18.888+0000 7f98deffd640 1 -- 172.21.15.123:0/2623250320 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) ==== 94+0+5223 (secure 0 0 0) 0x7f98e8063930 con 0x7f98f00726c0 2024-09-16T15:44:18.893 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-09-16T15:44:18.893 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":27,"btime":"2024-09-16T15:44:18:509761+0000","default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4712,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6837/288264117","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6836","nonce":288264117},{"type":"v1","addr":"172.21.15.184:6837","nonce":288264117}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":23},{"gid":5268,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6834/60194515","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6832","nonce":60194515},{"type":"v1","addr":"172.21.15.123:6834","nonce":60194515}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":23},{"gid":5271,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6835/1798373479","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6833","nonce":1798373479},{"type":"v1","addr":"172.21.15.123:6835","nonce":1798373479}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":22}],"filesystems":[{"mdsmap":{"epoch":27,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-16T15:44:14.853230+0000","modified":"2024-09-16T15:44:17.861742+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5280},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5280":{"gid":5280,"name":"d","rank":0,"incarnation":24,"state":"up:active","state_seq":3,"addr":"172.21.15.184:6835/374389412","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":374389412},{"type":"v1","addr":"172.21.15.184:6835","nonce":374389412}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5280,"qdb_cluster":[5280]},"id":3}]} 2024-09-16T15:44:18.893 INFO:teuthology.orchestra.run.smithi123.stderr:dumped fsmap epoch 27 2024-09-16T15:44:18.895 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:18.892+0000 7f98f4e26640 1 -- 172.21.15.123:0/2623250320 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f98d0073070 msgr2=0x7f98d0075530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:44:18.895 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:18.892+0000 7f98f4e26640 1 --2- 172.21.15.123:0/2623250320 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f98d0073070 0x7f98d0075530 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f98f0159e10 tx=0x7f98e400b390 comp rx=0 tx=0).stop 2024-09-16T15:44:18.895 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:18.892+0000 7f98f4e26640 1 -- 172.21.15.123:0/2623250320 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f98f00726c0 msgr2=0x7f98f014b8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:44:18.896 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:18.892+0000 7f98f4e26640 1 --2- 172.21.15.123:0/2623250320 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f98f00726c0 0x7f98f014b8e0 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f98e800ba80 tx=0x7f98e800bf40 comp rx=0 tx=0).stop 2024-09-16T15:44:18.896 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:18.892+0000 7f98f4e26640 1 -- 172.21.15.123:0/2623250320 shutdown_connections 2024-09-16T15:44:18.896 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:18.892+0000 7f98f4e26640 1 --2- 172.21.15.123:0/2623250320 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f98d0073070 0x7f98d0075530 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:18.896 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:18.892+0000 7f98f4e26640 1 --2- 172.21.15.123:0/2623250320 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f98f015bca0 0x7f98f01f9600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:18.896 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:18.892+0000 7f98f4e26640 1 --2- 172.21.15.123:0/2623250320 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f98f0074600 0x7f98f0152f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:18.896 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:18.892+0000 7f98f4e26640 1 --2- 172.21.15.123:0/2623250320 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f98f00726c0 0x7f98f014b8e0 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:18.896 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:18.892+0000 7f98f4e26640 1 -- 172.21.15.123:0/2623250320 >> 172.21.15.123:0/2623250320 conn(0x7f98f00cfb10 msgr2=0x7f98f00c3640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:44:18.896 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:18.892+0000 7f98f4e26640 1 -- 172.21.15.123:0/2623250320 shutdown_connections 2024-09-16T15:44:18.896 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:18.892+0000 7f98f4e26640 1 -- 172.21.15.123:0/2623250320 wait complete. 2024-09-16T15:44:18.912 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-16T15:44:19.027 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:19.020+0000 7fecbacdf640 1 -- 172.21.15.123:0/4256801160 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fecb4074470 msgr2=0x7fecb40cd6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:44:19.027 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:19.020+0000 7fecbacdf640 1 --2- 172.21.15.123:0/4256801160 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fecb4074470 0x7fecb40cd6e0 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7fecb4070730 tx=0x7feca801c4e0 comp rx=0 tx=0).stop 2024-09-16T15:44:19.027 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:19.020+0000 7fecbacdf640 1 -- 172.21.15.123:0/4256801160 shutdown_connections 2024-09-16T15:44:19.027 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:19.020+0000 7fecbacdf640 1 --2- 172.21.15.123:0/4256801160 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fecb40cdc20 0x7fecb40ca860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:19.027 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:19.020+0000 7fecbacdf640 1 --2- 172.21.15.123:0/4256801160 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fecb4074470 0x7fecb40cd6e0 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:19.027 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:19.020+0000 7fecbacdf640 1 --2- 172.21.15.123:0/4256801160 >> v2:172.21.15.184:3300/0 conn(0x7fecb415e160 0x7fecb4073f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:19.027 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:19.020+0000 7fecbacdf640 1 -- 172.21.15.123:0/4256801160 >> 172.21.15.123:0/4256801160 conn(0x7fecb40cfb10 msgr2=0x7fecb40cff20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:44:19.027 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:19.020+0000 7fecbacdf640 1 -- 172.21.15.123:0/4256801160 shutdown_connections 2024-09-16T15:44:19.027 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:19.020+0000 7fecbacdf640 1 -- 172.21.15.123:0/4256801160 wait complete. 2024-09-16T15:44:19.028 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:19.024+0000 7fecbacdf640 1 Processor -- start 2024-09-16T15:44:19.028 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:19.024+0000 7fecbacdf640 1 -- start start 2024-09-16T15:44:19.029 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:19.024+0000 7fecbacdf640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fecb4074470 0x7fecb4154200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:44:19.029 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:19.024+0000 7fecbacdf640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fecb40cdc20 0x7fecb41faf80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:44:19.029 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:19.024+0000 7fecbacdf640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fecb415e160 0x7fecb41fb4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:44:19.029 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:19.024+0000 7fecbacdf640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fecb415ebb0 con 0x7fecb40cdc20 2024-09-16T15:44:19.029 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:19.024+0000 7fecbacdf640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fecb415ea30 con 0x7fecb415e160 2024-09-16T15:44:19.030 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:19.024+0000 7fecbacdf640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fecb415ed30 con 0x7fecb4074470 2024-09-16T15:44:19.030 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:19.024+0000 7fecb3fff640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fecb40cdc20 0x7fecb41faf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:44:19.030 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:19.024+0000 7fecb3fff640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fecb40cdc20 0x7fecb41faf80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:47240/0 (socket says 172.21.15.123:47240) 2024-09-16T15:44:19.030 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:19.024+0000 7fecb3fff640 1 -- 172.21.15.123:0/2146560026 learned_addr learned my addr 172.21.15.123:0/2146560026 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:44:19.030 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:19.024+0000 7fecb3fff640 1 -- 172.21.15.123:0/2146560026 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fecb4074470 msgr2=0x7fecb4154200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:44:19.030 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:19.024+0000 7fecb3fff640 1 --2- 172.21.15.123:0/2146560026 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fecb4074470 0x7fecb4154200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:19.030 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:19.024+0000 7fecb3fff640 1 -- 172.21.15.123:0/2146560026 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fecb415e160 msgr2=0x7fecb41fb4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:44:19.030 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:19.024+0000 7fecb3fff640 1 --2- 172.21.15.123:0/2146560026 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fecb415e160 0x7fecb41fb4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:19.030 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:19.024+0000 7fecb3fff640 1 -- 172.21.15.123:0/2146560026 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fecb40aeaa0 con 0x7fecb40cdc20 2024-09-16T15:44:19.030 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:19.024+0000 7fecb3fff640 1 --2- 172.21.15.123:0/2146560026 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fecb40cdc20 0x7fecb41faf80 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7feca80ad040 tx=0x7feca80ab5e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:44:19.030 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:19.024+0000 7fecb17fa640 1 -- 172.21.15.123:0/2146560026 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feca80abb10 con 0x7fecb40cdc20 2024-09-16T15:44:19.030 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:19.024+0000 7fecb17fa640 1 -- 172.21.15.123:0/2146560026 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7feca80b9070 con 0x7fecb40cdc20 2024-09-16T15:44:19.030 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:19.024+0000 7fecb17fa640 1 -- 172.21.15.123:0/2146560026 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feca8005020 con 0x7fecb40cdc20 2024-09-16T15:44:19.030 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:19.024+0000 7fecbacdf640 1 -- 172.21.15.123:0/2146560026 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fecb40aed80 con 0x7fecb40cdc20 2024-09-16T15:44:19.031 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:19.024+0000 7fecbacdf640 1 -- 172.21.15.123:0/2146560026 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fecb4154740 con 0x7fecb40cdc20 2024-09-16T15:44:19.031 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:19.024+0000 7fecbacdf640 1 -- 172.21.15.123:0/2146560026 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fecb40ca860 con 0x7fecb40cdc20 2024-09-16T15:44:19.031 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:19.024+0000 7fecb17fa640 1 -- 172.21.15.123:0/2146560026 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 7) ==== 97233+0+0 (secure 0 0 0) 0x7feca80bb030 con 0x7fecb40cdc20 2024-09-16T15:44:19.031 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:19.028+0000 7fecb17fa640 1 --2- 172.21.15.123:0/2146560026 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7feca00730a0 0x7feca0075560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:44:19.031 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:19.028+0000 7fecb17fa640 1 -- 172.21.15.123:0/2146560026 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(44..44 src has 1..44) ==== 5709+0+0 (secure 0 0 0) 0x7feca812ebe0 con 0x7fecb40cdc20 2024-09-16T15:44:19.032 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:19.028+0000 7fecb8a7b640 1 --2- 172.21.15.123:0/2146560026 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7feca00730a0 0x7feca0075560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:44:19.032 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:19.028+0000 7fecb8a7b640 1 --2- 172.21.15.123:0/2146560026 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7feca00730a0 0x7feca0075560 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7feca400ab90 tx=0x7feca4009250 comp rx=0 tx=0).ready entity=mgr.4135 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:44:19.034 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:19.028+0000 7fecb17fa640 1 -- 172.21.15.123:0/2146560026 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7feca80164b0 con 0x7fecb40cdc20 2024-09-16T15:44:19.203 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:19.196+0000 7fecbacdf640 1 -- 172.21.15.123:0/2146560026 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fecb40d06c0 con 0x7fecb40cdc20 2024-09-16T15:44:19.204 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:19.200+0000 7fecb17fa640 1 -- 172.21.15.123:0/2146560026 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v44) ==== 74+0+16733 (secure 0 0 0) 0x7feca80f9b80 con 0x7fecb40cdc20 2024-09-16T15:44:19.205 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-09-16T15:44:19.205 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":44,"fsid":"56077f89-18c8-486f-ae7e-eecc0bb7822c","created":"2024-09-16T15:42:26.364892+0000","modified":"2024-09-16T15:44:16.486042+0000","last_up_change":"2024-09-16T15:42:38.707234+0000","last_in_change":"2024-09-16T15:42:29.423662+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-16T15:42:39.287764+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_metadata","create_time":"2024-09-16T15:44:12.463367+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"43","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_data","create_time":"2024-09-16T15:44:13.817170+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"44","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9e7c1955-a607-4671-9be7-f2ef760ceeaa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6800","nonce":3882044969},{"type":"v1","addr":"172.21.15.123:6801","nonce":3882044969}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6802","nonce":3882044969},{"type":"v1","addr":"172.21.15.123:6803","nonce":3882044969}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6806","nonce":3882044969},{"type":"v1","addr":"172.21.15.123:6807","nonce":3882044969}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6804","nonce":3882044969},{"type":"v1","addr":"172.21.15.123:6805","nonce":3882044969}]},"public_addr":"172.21.15.123:6801/3882044969","cluster_addr":"172.21.15.123:6803/3882044969","heartbeat_back_addr":"172.21.15.123:6807/3882044969","heartbeat_front_addr":"172.21.15.123:6805/3882044969","state":["exists","up"]},{"osd":1,"uuid":"c540cfde-0977-4742-9b17-fd634668e010","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6816","nonce":4037001850},{"type":"v1","addr":"172.21.15.123:6817","nonce":4037001850}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6818","nonce":4037001850},{"type":"v1","addr":"172.21.15.123:6819","nonce":4037001850}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6822","nonce":4037001850},{"type":"v1","addr":"172.21.15.123:6823","nonce":4037001850}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6820","nonce":4037001850},{"type":"v1","addr":"172.21.15.123:6821","nonce":4037001850}]},"public_addr":"172.21.15.123:6817/4037001850","cluster_addr":"172.21.15.123:6819/4037001850","heartbeat_back_addr":"172.21.15.123:6823/4037001850","heartbeat_front_addr":"172.21.15.123:6821/4037001850","state":["exists","up"]},{"osd":2,"uuid":"9864a496-f169-486b-95c4-1d59162793a7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6824","nonce":1332143088},{"type":"v1","addr":"172.21.15.123:6825","nonce":1332143088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6826","nonce":1332143088},{"type":"v1","addr":"172.21.15.123:6827","nonce":1332143088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6830","nonce":1332143088},{"type":"v1","addr":"172.21.15.123:6831","nonce":1332143088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6828","nonce":1332143088},{"type":"v1","addr":"172.21.15.123:6829","nonce":1332143088}]},"public_addr":"172.21.15.123:6825/1332143088","cluster_addr":"172.21.15.123:6827/1332143088","heartbeat_back_addr":"172.21.15.123:6831/1332143088","heartbeat_front_addr":"172.21.15.123:6829/1332143088","state":["exists","up"]},{"osd":3,"uuid":"c6d1f817-f074-4325-95e1-578eba13edda","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6808","nonce":988586998},{"type":"v1","addr":"172.21.15.123:6809","nonce":988586998}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6810","nonce":988586998},{"type":"v1","addr":"172.21.15.123:6811","nonce":988586998}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6814","nonce":988586998},{"type":"v1","addr":"172.21.15.123:6815","nonce":988586998}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6812","nonce":988586998},{"type":"v1","addr":"172.21.15.123:6813","nonce":988586998}]},"public_addr":"172.21.15.123:6809/988586998","cluster_addr":"172.21.15.123:6811/988586998","heartbeat_back_addr":"172.21.15.123:6815/988586998","heartbeat_front_addr":"172.21.15.123:6813/988586998","state":["exists","up"]},{"osd":4,"uuid":"200a8df4-50ef-40a5-b48a-58981269c086","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6816","nonce":3387813890},{"type":"v1","addr":"172.21.15.184:6817","nonce":3387813890}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6818","nonce":3387813890},{"type":"v1","addr":"172.21.15.184:6819","nonce":3387813890}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6822","nonce":3387813890},{"type":"v1","addr":"172.21.15.184:6823","nonce":3387813890}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6820","nonce":3387813890},{"type":"v1","addr":"172.21.15.184:6821","nonce":3387813890}]},"public_addr":"172.21.15.184:6817/3387813890","cluster_addr":"172.21.15.184:6819/3387813890","heartbeat_back_addr":"172.21.15.184:6823/3387813890","heartbeat_front_addr":"172.21.15.184:6821/3387813890","state":["exists","up"]},{"osd":5,"uuid":"32d5a900-da3d-412c-bc19-13861a2368d8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6808","nonce":2680916017},{"type":"v1","addr":"172.21.15.184:6809","nonce":2680916017}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6810","nonce":2680916017},{"type":"v1","addr":"172.21.15.184:6811","nonce":2680916017}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6814","nonce":2680916017},{"type":"v1","addr":"172.21.15.184:6815","nonce":2680916017}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6812","nonce":2680916017},{"type":"v1","addr":"172.21.15.184:6813","nonce":2680916017}]},"public_addr":"172.21.15.184:6809/2680916017","cluster_addr":"172.21.15.184:6811/2680916017","heartbeat_back_addr":"172.21.15.184:6815/2680916017","heartbeat_front_addr":"172.21.15.184:6813/2680916017","state":["exists","up"]},{"osd":6,"uuid":"cde6f39e-aa1b-4260-ac19-774ee6aaca72","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6824","nonce":2183120457},{"type":"v1","addr":"172.21.15.184:6825","nonce":2183120457}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6826","nonce":2183120457},{"type":"v1","addr":"172.21.15.184:6827","nonce":2183120457}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6830","nonce":2183120457},{"type":"v1","addr":"172.21.15.184:6831","nonce":2183120457}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6828","nonce":2183120457},{"type":"v1","addr":"172.21.15.184:6829","nonce":2183120457}]},"public_addr":"172.21.15.184:6825/2183120457","cluster_addr":"172.21.15.184:6827/2183120457","heartbeat_back_addr":"172.21.15.184:6831/2183120457","heartbeat_front_addr":"172.21.15.184:6829/2183120457","state":["exists","up"]},{"osd":7,"uuid":"4f381778-eee1-47c6-804f-df0e6b33e313","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6800","nonce":4102957285},{"type":"v1","addr":"172.21.15.184:6801","nonce":4102957285}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6802","nonce":4102957285},{"type":"v1","addr":"172.21.15.184:6803","nonce":4102957285}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6806","nonce":4102957285},{"type":"v1","addr":"172.21.15.184:6807","nonce":4102957285}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6804","nonce":4102957285},{"type":"v1","addr":"172.21.15.184:6805","nonce":4102957285}]},"public_addr":"172.21.15.184:6801/4102957285","cluster_addr":"172.21.15.184:6803/4102957285","heartbeat_back_addr":"172.21.15.184:6807/4102957285","heartbeat_front_addr":"172.21.15.184:6805/4102957285","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:35.972210+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.222007+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.198617+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.385663+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.358935+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.156066+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.752973+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:35.682846+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T15:44:19.207 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:19.200+0000 7fecbacdf640 1 -- 172.21.15.123:0/2146560026 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7feca00730a0 msgr2=0x7feca0075560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:44:19.207 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:19.200+0000 7fecbacdf640 1 --2- 172.21.15.123:0/2146560026 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7feca00730a0 0x7feca0075560 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7feca400ab90 tx=0x7feca4009250 comp rx=0 tx=0).stop 2024-09-16T15:44:19.207 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:19.200+0000 7fecbacdf640 1 -- 172.21.15.123:0/2146560026 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fecb40cdc20 msgr2=0x7fecb41faf80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:44:19.207 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:19.200+0000 7fecbacdf640 1 --2- 172.21.15.123:0/2146560026 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fecb40cdc20 0x7fecb41faf80 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7feca80ad040 tx=0x7feca80ab5e0 comp rx=0 tx=0).stop 2024-09-16T15:44:19.207 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:19.200+0000 7fecbacdf640 1 -- 172.21.15.123:0/2146560026 shutdown_connections 2024-09-16T15:44:19.207 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:19.200+0000 7fecbacdf640 1 --2- 172.21.15.123:0/2146560026 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7feca00730a0 0x7feca0075560 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:19.207 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:19.200+0000 7fecbacdf640 1 --2- 172.21.15.123:0/2146560026 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fecb415e160 0x7fecb41fb4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:19.207 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:19.200+0000 7fecbacdf640 1 --2- 172.21.15.123:0/2146560026 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fecb40cdc20 0x7fecb41faf80 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:19.208 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:19.200+0000 7fecbacdf640 1 --2- 172.21.15.123:0/2146560026 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fecb4074470 0x7fecb4154200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:19.208 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:19.200+0000 7fecbacdf640 1 -- 172.21.15.123:0/2146560026 >> 172.21.15.123:0/2146560026 conn(0x7fecb40cfb10 msgr2=0x7fecb40b49b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:44:19.208 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:19.200+0000 7fecbacdf640 1 -- 172.21.15.123:0/2146560026 shutdown_connections 2024-09-16T15:44:19.208 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:19.200+0000 7fecbacdf640 1 -- 172.21.15.123:0/2146560026 wait complete. 2024-09-16T15:44:19.222 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-09-16T15:44:19.223 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T15:44:19.336 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:19.332+0000 7f0f6aa7b640 1 -- 172.21.15.123:0/4080489247 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0f4c004810 msgr2=0x7f0f4c004c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:44:19.336 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:19.332+0000 7f0f6aa7b640 1 --2- 172.21.15.123:0/4080489247 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0f4c004810 0x7f0f4c004c90 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f0f6000c730 tx=0x7f0f6001c6b0 comp rx=0 tx=0).stop 2024-09-16T15:44:19.336 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:19.332+0000 7f0f6aa7b640 1 -- 172.21.15.123:0/4080489247 shutdown_connections 2024-09-16T15:44:19.336 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:19.332+0000 7f0f6aa7b640 1 --2- 172.21.15.123:0/4080489247 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0f4c004810 0x7f0f4c004c90 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:19.336 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:19.332+0000 7f0f6aa7b640 1 -- 172.21.15.123:0/4080489247 >> 172.21.15.123:0/4080489247 conn(0x7f0f640d3ea0 msgr2=0x7f0f640d42b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:44:19.336 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:19.332+0000 7f0f6aa7b640 1 -- 172.21.15.123:0/4080489247 shutdown_connections 2024-09-16T15:44:19.336 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:19.332+0000 7f0f6aa7b640 1 -- 172.21.15.123:0/4080489247 wait complete. 2024-09-16T15:44:19.337 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:19.332+0000 7f0f6aa7b640 1 Processor -- start 2024-09-16T15:44:19.337 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:19.332+0000 7f0f6aa7b640 1 -- start start 2024-09-16T15:44:19.339 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:19.332+0000 7f0f6aa7b640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f0f4c004810 0x7f0f640b8e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:44:19.339 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:19.332+0000 7f0f6aa7b640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0f640b9340 0x7f0f6414b140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:44:19.339 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:19.332+0000 7f0f6aa7b640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f0f6414b680 0x7f0f640c9e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:44:19.339 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:19.332+0000 7f0f6aa7b640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f0f640d2be0 con 0x7f0f640b9340 2024-09-16T15:44:19.339 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:19.332+0000 7f0f6aa7b640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f0f640d2a60 con 0x7f0f6414b680 2024-09-16T15:44:19.339 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:19.332+0000 7f0f6aa7b640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f0f640b0e30 con 0x7f0f4c004810 2024-09-16T15:44:19.339 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:19.332+0000 7f0f5bfff640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0f640b9340 0x7f0f6414b140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:44:19.339 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:19.332+0000 7f0f5bfff640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0f640b9340 0x7f0f6414b140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:47250/0 (socket says 172.21.15.123:47250) 2024-09-16T15:44:19.339 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:19.332+0000 7f0f5bfff640 1 -- 172.21.15.123:0/2513376718 learned_addr learned my addr 172.21.15.123:0/2513376718 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:44:19.340 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:19.332+0000 7f0f68817640 1 --2- 172.21.15.123:0/2513376718 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f0f4c004810 0x7f0f640b8e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:44:19.340 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:19.332+0000 7f0f69018640 1 --2- 172.21.15.123:0/2513376718 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f0f6414b680 0x7f0f640c9e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:44:19.340 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:19.332+0000 7f0f5bfff640 1 -- 172.21.15.123:0/2513376718 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f0f4c004810 msgr2=0x7f0f640b8e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:44:19.340 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:19.332+0000 7f0f5bfff640 1 --2- 172.21.15.123:0/2513376718 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f0f4c004810 0x7f0f640b8e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:19.340 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:19.332+0000 7f0f5bfff640 1 -- 172.21.15.123:0/2513376718 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f0f6414b680 msgr2=0x7f0f640c9e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:44:19.340 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:19.332+0000 7f0f5bfff640 1 --2- 172.21.15.123:0/2513376718 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f0f6414b680 0x7f0f640c9e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:19.340 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:19.332+0000 7f0f5bfff640 1 -- 172.21.15.123:0/2513376718 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0f640b7dc0 con 0x7f0f640b9340 2024-09-16T15:44:19.340 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:19.332+0000 7f0f5bfff640 1 --2- 172.21.15.123:0/2513376718 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0f640b9340 0x7f0f6414b140 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f0f60000c10 tx=0x7f0f60004c90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:44:19.340 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:19.332+0000 7f0f597fa640 1 -- 172.21.15.123:0/2513376718 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f600ad070 con 0x7f0f640b9340 2024-09-16T15:44:19.340 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:19.332+0000 7f0f597fa640 1 -- 172.21.15.123:0/2513376718 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f0f60006500 con 0x7f0f640b9340 2024-09-16T15:44:19.340 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:19.332+0000 7f0f6aa7b640 1 -- 172.21.15.123:0/2513376718 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0f640b8060 con 0x7f0f640b9340 2024-09-16T15:44:19.340 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:19.332+0000 7f0f597fa640 1 -- 172.21.15.123:0/2513376718 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f6000c300 con 0x7f0f640b9340 2024-09-16T15:44:19.340 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:19.332+0000 7f0f6aa7b640 1 -- 172.21.15.123:0/2513376718 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0f640b8620 con 0x7f0f640b9340 2024-09-16T15:44:19.341 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:19.336+0000 7f0f6aa7b640 1 -- 172.21.15.123:0/2513376718 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0f640b1340 con 0x7f0f640b9340 2024-09-16T15:44:19.341 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:19.336+0000 7f0f597fa640 1 -- 172.21.15.123:0/2513376718 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 7) ==== 97233+0+0 (secure 0 0 0) 0x7f0f600bb060 con 0x7f0f640b9340 2024-09-16T15:44:19.341 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:19.336+0000 7f0f597fa640 1 --2- 172.21.15.123:0/2513376718 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f0f50073070 0x7f0f50075530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:44:19.341 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:19.336+0000 7f0f597fa640 1 -- 172.21.15.123:0/2513376718 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(44..44 src has 1..44) ==== 5709+0+0 (secure 0 0 0) 0x7f0f6012d620 con 0x7f0f640b9340 2024-09-16T15:44:19.341 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:19.336+0000 7f0f68817640 1 --2- 172.21.15.123:0/2513376718 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f0f50073070 0x7f0f50075530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:44:19.342 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:19.336+0000 7f0f68817640 1 --2- 172.21.15.123:0/2513376718 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f0f50073070 0x7f0f50075530 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f0f5400cb70 tx=0x7f0f54005e50 comp rx=0 tx=0).ready entity=mgr.4135 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:44:19.345 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:19.340+0000 7f0f597fa640 1 -- 172.21.15.123:0/2513376718 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f0f600163f0 con 0x7f0f640b9340 2024-09-16T15:44:19.513 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:19.508+0000 7f0f6aa7b640 1 -- 172.21.15.123:0/2513376718 --> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f0f640cf540 con 0x7f0f50073070 2024-09-16T15:44:19.524 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:19.520+0000 7f0f597fa640 1 -- 172.21.15.123:0/2513376718 <== mgr.4135 v2:172.21.15.184:6832/3114346423 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+239845 (secure 0 0 0) 0x7f0f640cf540 con 0x7f0f50073070 2024-09-16T15:44:19.527 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-09-16T15:44:19.528 INFO:teuthology.orchestra.run.smithi123.stderr:dumped all 2024-09-16T15:44:19.529 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:19.524+0000 7f0f6aa7b640 1 -- 172.21.15.123:0/2513376718 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f0f50073070 msgr2=0x7f0f50075530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:44:19.529 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:19.524+0000 7f0f6aa7b640 1 --2- 172.21.15.123:0/2513376718 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f0f50073070 0x7f0f50075530 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f0f5400cb70 tx=0x7f0f54005e50 comp rx=0 tx=0).stop 2024-09-16T15:44:19.529 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:19.524+0000 7f0f6aa7b640 1 -- 172.21.15.123:0/2513376718 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0f640b9340 msgr2=0x7f0f6414b140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:44:19.530 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:19.524+0000 7f0f6aa7b640 1 --2- 172.21.15.123:0/2513376718 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0f640b9340 0x7f0f6414b140 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f0f60000c10 tx=0x7f0f60004c90 comp rx=0 tx=0).stop 2024-09-16T15:44:19.530 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:19.524+0000 7f0f6aa7b640 1 -- 172.21.15.123:0/2513376718 shutdown_connections 2024-09-16T15:44:19.530 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:19.524+0000 7f0f6aa7b640 1 --2- 172.21.15.123:0/2513376718 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f0f50073070 0x7f0f50075530 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:19.530 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:19.524+0000 7f0f6aa7b640 1 --2- 172.21.15.123:0/2513376718 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f0f6414b680 0x7f0f640c9e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:19.530 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:19.524+0000 7f0f6aa7b640 1 --2- 172.21.15.123:0/2513376718 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0f640b9340 0x7f0f6414b140 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:19.530 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:19.524+0000 7f0f6aa7b640 1 --2- 172.21.15.123:0/2513376718 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f0f4c004810 0x7f0f640b8e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:19.530 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:19.524+0000 7f0f6aa7b640 1 -- 172.21.15.123:0/2513376718 >> 172.21.15.123:0/2513376718 conn(0x7f0f640d3ea0 msgr2=0x7f0f640b4b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:44:19.530 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:19.524+0000 7f0f6aa7b640 1 -- 172.21.15.123:0/2513376718 shutdown_connections 2024-09-16T15:44:19.530 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:19.524+0000 7f0f6aa7b640 1 -- 172.21.15.123:0/2513376718 wait complete. 2024-09-16T15:44:19.546 INFO:teuthology.orchestra.run.smithi123.stdout:{"pg_ready":true,"pg_map":{"version":87,"stamp":"2024-09-16T15:44:19.274293+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":66,"acting":66,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":9,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":7,"kb":754974720,"kb_used":225260,"kb_used_data":4848,"kb_used_omap":12,"kb_used_meta":220211,"kb_avail":754749460,"statfs":{"total":773094113280,"available":772863447040,"internally_reserved":0,"allocated":4964352,"data_stored":2660670,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":225496625},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":13,"apply_latency_ms":13,"commit_latency_ns":13000000,"apply_latency_ns":13000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.810966"},"pg_stats":[{"pgid":"7.3f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T15:44:14.461845+0000","last_change":"2024-09-16T15:44:14.461845+0000","last_active":"2024-09-16T15:44:14.461845+0000","last_peered":"2024-09-16T15:44:14.461845+0000","last_clean":"2024-09-16T15:44:14.461845+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:44:14.461845+0000","last_undegraded":"2024-09-16T15:44:14.461845+0000","last_fullsized":"2024-09-16T15:44:14.461845+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_clean_scrub_stamp":"2024-09-16T15:44:14.461845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.3e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T15:44:14.461845+0000","last_change":"2024-09-16T15:44:14.461845+0000","last_active":"2024-09-16T15:44:14.461845+0000","last_peered":"2024-09-16T15:44:14.461845+0000","last_clean":"2024-09-16T15:44:14.461845+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:44:14.461845+0000","last_undegraded":"2024-09-16T15:44:14.461845+0000","last_fullsized":"2024-09-16T15:44:14.461845+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_clean_scrub_stamp":"2024-09-16T15:44:14.461845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.3d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T15:44:14.461845+0000","last_change":"2024-09-16T15:44:14.461845+0000","last_active":"2024-09-16T15:44:14.461845+0000","last_peered":"2024-09-16T15:44:14.461845+0000","last_clean":"2024-09-16T15:44:14.461845+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:44:14.461845+0000","last_undegraded":"2024-09-16T15:44:14.461845+0000","last_fullsized":"2024-09-16T15:44:14.461845+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_clean_scrub_stamp":"2024-09-16T15:44:14.461845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.3c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T15:44:14.461845+0000","last_change":"2024-09-16T15:44:14.461845+0000","last_active":"2024-09-16T15:44:14.461845+0000","last_peered":"2024-09-16T15:44:14.461845+0000","last_clean":"2024-09-16T15:44:14.461845+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:44:14.461845+0000","last_undegraded":"2024-09-16T15:44:14.461845+0000","last_fullsized":"2024-09-16T15:44:14.461845+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_clean_scrub_stamp":"2024-09-16T15:44:14.461845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.3b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T15:44:14.461845+0000","last_change":"2024-09-16T15:44:14.461845+0000","last_active":"2024-09-16T15:44:14.461845+0000","last_peered":"2024-09-16T15:44:14.461845+0000","last_clean":"2024-09-16T15:44:14.461845+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:44:14.461845+0000","last_undegraded":"2024-09-16T15:44:14.461845+0000","last_fullsized":"2024-09-16T15:44:14.461845+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_clean_scrub_stamp":"2024-09-16T15:44:14.461845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.3a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T15:44:14.461845+0000","last_change":"2024-09-16T15:44:14.461845+0000","last_active":"2024-09-16T15:44:14.461845+0000","last_peered":"2024-09-16T15:44:14.461845+0000","last_clean":"2024-09-16T15:44:14.461845+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:44:14.461845+0000","last_undegraded":"2024-09-16T15:44:14.461845+0000","last_fullsized":"2024-09-16T15:44:14.461845+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_clean_scrub_stamp":"2024-09-16T15:44:14.461845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.39","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T15:44:14.461845+0000","last_change":"2024-09-16T15:44:14.461845+0000","last_active":"2024-09-16T15:44:14.461845+0000","last_peered":"2024-09-16T15:44:14.461845+0000","last_clean":"2024-09-16T15:44:14.461845+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:44:14.461845+0000","last_undegraded":"2024-09-16T15:44:14.461845+0000","last_fullsized":"2024-09-16T15:44:14.461845+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_clean_scrub_stamp":"2024-09-16T15:44:14.461845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.38","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T15:44:14.461845+0000","last_change":"2024-09-16T15:44:14.461845+0000","last_active":"2024-09-16T15:44:14.461845+0000","last_peered":"2024-09-16T15:44:14.461845+0000","last_clean":"2024-09-16T15:44:14.461845+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:44:14.461845+0000","last_undegraded":"2024-09-16T15:44:14.461845+0000","last_fullsized":"2024-09-16T15:44:14.461845+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_clean_scrub_stamp":"2024-09-16T15:44:14.461845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.37","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T15:44:14.461845+0000","last_change":"2024-09-16T15:44:14.461845+0000","last_active":"2024-09-16T15:44:14.461845+0000","last_peered":"2024-09-16T15:44:14.461845+0000","last_clean":"2024-09-16T15:44:14.461845+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:44:14.461845+0000","last_undegraded":"2024-09-16T15:44:14.461845+0000","last_fullsized":"2024-09-16T15:44:14.461845+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_clean_scrub_stamp":"2024-09-16T15:44:14.461845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.36","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T15:44:14.461845+0000","last_change":"2024-09-16T15:44:14.461845+0000","last_active":"2024-09-16T15:44:14.461845+0000","last_peered":"2024-09-16T15:44:14.461845+0000","last_clean":"2024-09-16T15:44:14.461845+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:44:14.461845+0000","last_undegraded":"2024-09-16T15:44:14.461845+0000","last_fullsized":"2024-09-16T15:44:14.461845+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_clean_scrub_stamp":"2024-09-16T15:44:14.461845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.35","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T15:44:14.461845+0000","last_change":"2024-09-16T15:44:14.461845+0000","last_active":"2024-09-16T15:44:14.461845+0000","last_peered":"2024-09-16T15:44:14.461845+0000","last_clean":"2024-09-16T15:44:14.461845+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:44:14.461845+0000","last_undegraded":"2024-09-16T15:44:14.461845+0000","last_fullsized":"2024-09-16T15:44:14.461845+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_clean_scrub_stamp":"2024-09-16T15:44:14.461845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.34","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T15:44:14.461845+0000","last_change":"2024-09-16T15:44:14.461845+0000","last_active":"2024-09-16T15:44:14.461845+0000","last_peered":"2024-09-16T15:44:14.461845+0000","last_clean":"2024-09-16T15:44:14.461845+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:44:14.461845+0000","last_undegraded":"2024-09-16T15:44:14.461845+0000","last_fullsized":"2024-09-16T15:44:14.461845+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_clean_scrub_stamp":"2024-09-16T15:44:14.461845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.33","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T15:44:14.461845+0000","last_change":"2024-09-16T15:44:14.461845+0000","last_active":"2024-09-16T15:44:14.461845+0000","last_peered":"2024-09-16T15:44:14.461845+0000","last_clean":"2024-09-16T15:44:14.461845+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:44:14.461845+0000","last_undegraded":"2024-09-16T15:44:14.461845+0000","last_fullsized":"2024-09-16T15:44:14.461845+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_clean_scrub_stamp":"2024-09-16T15:44:14.461845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.32","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T15:44:14.461845+0000","last_change":"2024-09-16T15:44:14.461845+0000","last_active":"2024-09-16T15:44:14.461845+0000","last_peered":"2024-09-16T15:44:14.461845+0000","last_clean":"2024-09-16T15:44:14.461845+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:44:14.461845+0000","last_undegraded":"2024-09-16T15:44:14.461845+0000","last_fullsized":"2024-09-16T15:44:14.461845+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_clean_scrub_stamp":"2024-09-16T15:44:14.461845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.31","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T15:44:14.461845+0000","last_change":"2024-09-16T15:44:14.461845+0000","last_active":"2024-09-16T15:44:14.461845+0000","last_peered":"2024-09-16T15:44:14.461845+0000","last_clean":"2024-09-16T15:44:14.461845+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:44:14.461845+0000","last_undegraded":"2024-09-16T15:44:14.461845+0000","last_fullsized":"2024-09-16T15:44:14.461845+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_clean_scrub_stamp":"2024-09-16T15:44:14.461845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.30","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T15:44:14.461845+0000","last_change":"2024-09-16T15:44:14.461845+0000","last_active":"2024-09-16T15:44:14.461845+0000","last_peered":"2024-09-16T15:44:14.461845+0000","last_clean":"2024-09-16T15:44:14.461845+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:44:14.461845+0000","last_undegraded":"2024-09-16T15:44:14.461845+0000","last_fullsized":"2024-09-16T15:44:14.461845+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_clean_scrub_stamp":"2024-09-16T15:44:14.461845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.2f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T15:44:14.461845+0000","last_change":"2024-09-16T15:44:14.461845+0000","last_active":"2024-09-16T15:44:14.461845+0000","last_peered":"2024-09-16T15:44:14.461845+0000","last_clean":"2024-09-16T15:44:14.461845+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:44:14.461845+0000","last_undegraded":"2024-09-16T15:44:14.461845+0000","last_fullsized":"2024-09-16T15:44:14.461845+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_clean_scrub_stamp":"2024-09-16T15:44:14.461845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.2e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T15:44:14.461845+0000","last_change":"2024-09-16T15:44:14.461845+0000","last_active":"2024-09-16T15:44:14.461845+0000","last_peered":"2024-09-16T15:44:14.461845+0000","last_clean":"2024-09-16T15:44:14.461845+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:44:14.461845+0000","last_undegraded":"2024-09-16T15:44:14.461845+0000","last_fullsized":"2024-09-16T15:44:14.461845+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_clean_scrub_stamp":"2024-09-16T15:44:14.461845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.2d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T15:44:14.461845+0000","last_change":"2024-09-16T15:44:14.461845+0000","last_active":"2024-09-16T15:44:14.461845+0000","last_peered":"2024-09-16T15:44:14.461845+0000","last_clean":"2024-09-16T15:44:14.461845+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:44:14.461845+0000","last_undegraded":"2024-09-16T15:44:14.461845+0000","last_fullsized":"2024-09-16T15:44:14.461845+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_clean_scrub_stamp":"2024-09-16T15:44:14.461845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.2c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T15:44:14.461845+0000","last_change":"2024-09-16T15:44:14.461845+0000","last_active":"2024-09-16T15:44:14.461845+0000","last_peered":"2024-09-16T15:44:14.461845+0000","last_clean":"2024-09-16T15:44:14.461845+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:44:14.461845+0000","last_undegraded":"2024-09-16T15:44:14.461845+0000","last_fullsized":"2024-09-16T15:44:14.461845+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_clean_scrub_stamp":"2024-09-16T15:44:14.461845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.2b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T15:44:14.461845+0000","last_change":"2024-09-16T15:44:14.461845+0000","last_active":"2024-09-16T15:44:14.461845+0000","last_peered":"2024-09-16T15:44:14.461845+0000","last_clean":"2024-09-16T15:44:14.461845+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:44:14.461845+0000","last_undegraded":"2024-09-16T15:44:14.461845+0000","last_fullsized":"2024-09-16T15:44:14.461845+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_clean_scrub_stamp":"2024-09-16T15:44:14.461845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.2a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T15:44:14.461845+0000","last_change":"2024-09-16T15:44:14.461845+0000","last_active":"2024-09-16T15:44:14.461845+0000","last_peered":"2024-09-16T15:44:14.461845+0000","last_clean":"2024-09-16T15:44:14.461845+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:44:14.461845+0000","last_undegraded":"2024-09-16T15:44:14.461845+0000","last_fullsized":"2024-09-16T15:44:14.461845+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_clean_scrub_stamp":"2024-09-16T15:44:14.461845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.29","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T15:44:14.461845+0000","last_change":"2024-09-16T15:44:14.461845+0000","last_active":"2024-09-16T15:44:14.461845+0000","last_peered":"2024-09-16T15:44:14.461845+0000","last_clean":"2024-09-16T15:44:14.461845+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:44:14.461845+0000","last_undegraded":"2024-09-16T15:44:14.461845+0000","last_fullsized":"2024-09-16T15:44:14.461845+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_clean_scrub_stamp":"2024-09-16T15:44:14.461845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.28","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T15:44:14.461845+0000","last_change":"2024-09-16T15:44:14.461845+0000","last_active":"2024-09-16T15:44:14.461845+0000","last_peered":"2024-09-16T15:44:14.461845+0000","last_clean":"2024-09-16T15:44:14.461845+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:44:14.461845+0000","last_undegraded":"2024-09-16T15:44:14.461845+0000","last_fullsized":"2024-09-16T15:44:14.461845+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_clean_scrub_stamp":"2024-09-16T15:44:14.461845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.27","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T15:44:14.461845+0000","last_change":"2024-09-16T15:44:14.461845+0000","last_active":"2024-09-16T15:44:14.461845+0000","last_peered":"2024-09-16T15:44:14.461845+0000","last_clean":"2024-09-16T15:44:14.461845+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:44:14.461845+0000","last_undegraded":"2024-09-16T15:44:14.461845+0000","last_fullsized":"2024-09-16T15:44:14.461845+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_clean_scrub_stamp":"2024-09-16T15:44:14.461845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.26","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T15:44:14.461845+0000","last_change":"2024-09-16T15:44:14.461845+0000","last_active":"2024-09-16T15:44:14.461845+0000","last_peered":"2024-09-16T15:44:14.461845+0000","last_clean":"2024-09-16T15:44:14.461845+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:44:14.461845+0000","last_undegraded":"2024-09-16T15:44:14.461845+0000","last_fullsized":"2024-09-16T15:44:14.461845+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_clean_scrub_stamp":"2024-09-16T15:44:14.461845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.25","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T15:44:14.461845+0000","last_change":"2024-09-16T15:44:14.461845+0000","last_active":"2024-09-16T15:44:14.461845+0000","last_peered":"2024-09-16T15:44:14.461845+0000","last_clean":"2024-09-16T15:44:14.461845+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:44:14.461845+0000","last_undegraded":"2024-09-16T15:44:14.461845+0000","last_fullsized":"2024-09-16T15:44:14.461845+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_clean_scrub_stamp":"2024-09-16T15:44:14.461845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.24","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T15:44:14.461845+0000","last_change":"2024-09-16T15:44:14.461845+0000","last_active":"2024-09-16T15:44:14.461845+0000","last_peered":"2024-09-16T15:44:14.461845+0000","last_clean":"2024-09-16T15:44:14.461845+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:44:14.461845+0000","last_undegraded":"2024-09-16T15:44:14.461845+0000","last_fullsized":"2024-09-16T15:44:14.461845+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_clean_scrub_stamp":"2024-09-16T15:44:14.461845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.23","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T15:44:14.461845+0000","last_change":"2024-09-16T15:44:14.461845+0000","last_active":"2024-09-16T15:44:14.461845+0000","last_peered":"2024-09-16T15:44:14.461845+0000","last_clean":"2024-09-16T15:44:14.461845+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:44:14.461845+0000","last_undegraded":"2024-09-16T15:44:14.461845+0000","last_fullsized":"2024-09-16T15:44:14.461845+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_clean_scrub_stamp":"2024-09-16T15:44:14.461845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.22","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T15:44:14.461845+0000","last_change":"2024-09-16T15:44:14.461845+0000","last_active":"2024-09-16T15:44:14.461845+0000","last_peered":"2024-09-16T15:44:14.461845+0000","last_clean":"2024-09-16T15:44:14.461845+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:44:14.461845+0000","last_undegraded":"2024-09-16T15:44:14.461845+0000","last_fullsized":"2024-09-16T15:44:14.461845+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_clean_scrub_stamp":"2024-09-16T15:44:14.461845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.21","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T15:44:14.461845+0000","last_change":"2024-09-16T15:44:14.461845+0000","last_active":"2024-09-16T15:44:14.461845+0000","last_peered":"2024-09-16T15:44:14.461845+0000","last_clean":"2024-09-16T15:44:14.461845+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:44:14.461845+0000","last_undegraded":"2024-09-16T15:44:14.461845+0000","last_fullsized":"2024-09-16T15:44:14.461845+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_clean_scrub_stamp":"2024-09-16T15:44:14.461845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.20","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T15:44:14.461845+0000","last_change":"2024-09-16T15:44:14.461845+0000","last_active":"2024-09-16T15:44:14.461845+0000","last_peered":"2024-09-16T15:44:14.461845+0000","last_clean":"2024-09-16T15:44:14.461845+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:44:14.461845+0000","last_undegraded":"2024-09-16T15:44:14.461845+0000","last_fullsized":"2024-09-16T15:44:14.461845+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_clean_scrub_stamp":"2024-09-16T15:44:14.461845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T15:44:14.461845+0000","last_change":"2024-09-16T15:44:14.461845+0000","last_active":"2024-09-16T15:44:14.461845+0000","last_peered":"2024-09-16T15:44:14.461845+0000","last_clean":"2024-09-16T15:44:14.461845+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:44:14.461845+0000","last_undegraded":"2024-09-16T15:44:14.461845+0000","last_fullsized":"2024-09-16T15:44:14.461845+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_clean_scrub_stamp":"2024-09-16T15:44:14.461845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":1,"reported_epoch":41,"state":"creating+peering","last_fresh":"2024-09-16T15:44:13.470227+0000","last_change":"2024-09-16T15:44:13.470227+0000","last_active":"2024-09-16T15:44:13.453120+0000","last_peered":"2024-09-16T15:44:13.453120+0000","last_clean":"2024-09-16T15:44:13.453120+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:44:13.470227+0000","last_undegraded":"2024-09-16T15:44:13.470227+0000","last_fullsized":"2024-09-16T15:44:13.470227+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:13.453120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:13.453120+0000","last_clean_scrub_stamp":"2024-09-16T15:44:13.453120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T15:44:14.461845+0000","last_change":"2024-09-16T15:44:14.461845+0000","last_active":"2024-09-16T15:44:14.461845+0000","last_peered":"2024-09-16T15:44:14.461845+0000","last_clean":"2024-09-16T15:44:14.461845+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:44:14.461845+0000","last_undegraded":"2024-09-16T15:44:14.461845+0000","last_fullsized":"2024-09-16T15:44:14.461845+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_clean_scrub_stamp":"2024-09-16T15:44:14.461845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":1,"reported_epoch":41,"state":"creating+peering","last_fresh":"2024-09-16T15:44:13.469543+0000","last_change":"2024-09-16T15:44:13.469543+0000","last_active":"2024-09-16T15:44:13.453120+0000","last_peered":"2024-09-16T15:44:13.453120+0000","last_clean":"2024-09-16T15:44:13.453120+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:44:13.469543+0000","last_undegraded":"2024-09-16T15:44:13.469543+0000","last_fullsized":"2024-09-16T15:44:13.469543+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:13.453120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:13.453120+0000","last_clean_scrub_stamp":"2024-09-16T15:44:13.453120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T15:44:14.461845+0000","last_change":"2024-09-16T15:44:14.461845+0000","last_active":"2024-09-16T15:44:14.461845+0000","last_peered":"2024-09-16T15:44:14.461845+0000","last_clean":"2024-09-16T15:44:14.461845+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:44:14.461845+0000","last_undegraded":"2024-09-16T15:44:14.461845+0000","last_fullsized":"2024-09-16T15:44:14.461845+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_clean_scrub_stamp":"2024-09-16T15:44:14.461845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":1,"reported_epoch":41,"state":"creating+peering","last_fresh":"2024-09-16T15:44:13.472025+0000","last_change":"2024-09-16T15:44:13.472025+0000","last_active":"2024-09-16T15:44:13.453120+0000","last_peered":"2024-09-16T15:44:13.453120+0000","last_clean":"2024-09-16T15:44:13.453120+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:44:13.472025+0000","last_undegraded":"2024-09-16T15:44:13.472025+0000","last_fullsized":"2024-09-16T15:44:13.472025+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:13.453120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:13.453120+0000","last_clean_scrub_stamp":"2024-09-16T15:44:13.453120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T15:44:14.461845+0000","last_change":"2024-09-16T15:44:14.461845+0000","last_active":"2024-09-16T15:44:14.461845+0000","last_peered":"2024-09-16T15:44:14.461845+0000","last_clean":"2024-09-16T15:44:14.461845+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:44:14.461845+0000","last_undegraded":"2024-09-16T15:44:14.461845+0000","last_fullsized":"2024-09-16T15:44:14.461845+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_clean_scrub_stamp":"2024-09-16T15:44:14.461845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.1c","version":"0'0","reported_seq":1,"reported_epoch":41,"state":"creating+peering","last_fresh":"2024-09-16T15:44:13.475916+0000","last_change":"2024-09-16T15:44:13.475916+0000","last_active":"2024-09-16T15:44:13.453120+0000","last_peered":"2024-09-16T15:44:13.453120+0000","last_clean":"2024-09-16T15:44:13.453120+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:44:13.475916+0000","last_undegraded":"2024-09-16T15:44:13.475916+0000","last_fullsized":"2024-09-16T15:44:13.475916+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:13.453120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:13.453120+0000","last_clean_scrub_stamp":"2024-09-16T15:44:13.453120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T15:44:14.461845+0000","last_change":"2024-09-16T15:44:14.461845+0000","last_active":"2024-09-16T15:44:14.461845+0000","last_peered":"2024-09-16T15:44:14.461845+0000","last_clean":"2024-09-16T15:44:14.461845+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:44:14.461845+0000","last_undegraded":"2024-09-16T15:44:14.461845+0000","last_fullsized":"2024-09-16T15:44:14.461845+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_clean_scrub_stamp":"2024-09-16T15:44:14.461845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":1,"reported_epoch":41,"state":"creating+peering","last_fresh":"2024-09-16T15:44:13.470521+0000","last_change":"2024-09-16T15:44:13.470521+0000","last_active":"2024-09-16T15:44:13.453120+0000","last_peered":"2024-09-16T15:44:13.453120+0000","last_clean":"2024-09-16T15:44:13.453120+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:44:13.470521+0000","last_undegraded":"2024-09-16T15:44:13.470521+0000","last_fullsized":"2024-09-16T15:44:13.470521+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:13.453120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:13.453120+0000","last_clean_scrub_stamp":"2024-09-16T15:44:13.453120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T15:44:14.461845+0000","last_change":"2024-09-16T15:44:14.461845+0000","last_active":"2024-09-16T15:44:14.461845+0000","last_peered":"2024-09-16T15:44:14.461845+0000","last_clean":"2024-09-16T15:44:14.461845+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:44:14.461845+0000","last_undegraded":"2024-09-16T15:44:14.461845+0000","last_fullsized":"2024-09-16T15:44:14.461845+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_clean_scrub_stamp":"2024-09-16T15:44:14.461845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":1,"reported_epoch":41,"state":"creating+peering","last_fresh":"2024-09-16T15:44:13.468698+0000","last_change":"2024-09-16T15:44:13.468698+0000","last_active":"2024-09-16T15:44:13.453120+0000","last_peered":"2024-09-16T15:44:13.453120+0000","last_clean":"2024-09-16T15:44:13.453120+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:44:13.468698+0000","last_undegraded":"2024-09-16T15:44:13.468698+0000","last_fullsized":"2024-09-16T15:44:13.468698+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:13.453120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:13.453120+0000","last_clean_scrub_stamp":"2024-09-16T15:44:13.453120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T15:44:14.461845+0000","last_change":"2024-09-16T15:44:14.461845+0000","last_active":"2024-09-16T15:44:14.461845+0000","last_peered":"2024-09-16T15:44:14.461845+0000","last_clean":"2024-09-16T15:44:14.461845+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:44:14.461845+0000","last_undegraded":"2024-09-16T15:44:14.461845+0000","last_fullsized":"2024-09-16T15:44:14.461845+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_clean_scrub_stamp":"2024-09-16T15:44:14.461845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":1,"reported_epoch":41,"state":"creating+peering","last_fresh":"2024-09-16T15:44:13.478336+0000","last_change":"2024-09-16T15:44:13.478336+0000","last_active":"2024-09-16T15:44:13.453120+0000","last_peered":"2024-09-16T15:44:13.453120+0000","last_clean":"2024-09-16T15:44:13.453120+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:44:13.478336+0000","last_undegraded":"2024-09-16T15:44:13.478336+0000","last_fullsized":"2024-09-16T15:44:13.478336+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:13.453120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:13.453120+0000","last_clean_scrub_stamp":"2024-09-16T15:44:13.453120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T15:44:14.461845+0000","last_change":"2024-09-16T15:44:14.461845+0000","last_active":"2024-09-16T15:44:14.461845+0000","last_peered":"2024-09-16T15:44:14.461845+0000","last_clean":"2024-09-16T15:44:14.461845+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:44:14.461845+0000","last_undegraded":"2024-09-16T15:44:14.461845+0000","last_fullsized":"2024-09-16T15:44:14.461845+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_clean_scrub_stamp":"2024-09-16T15:44:14.461845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":1,"reported_epoch":41,"state":"creating+peering","last_fresh":"2024-09-16T15:44:13.475717+0000","last_change":"2024-09-16T15:44:13.475717+0000","last_active":"2024-09-16T15:44:13.453120+0000","last_peered":"2024-09-16T15:44:13.453120+0000","last_clean":"2024-09-16T15:44:13.453120+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:44:13.475717+0000","last_undegraded":"2024-09-16T15:44:13.475717+0000","last_fullsized":"2024-09-16T15:44:13.475717+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:13.453120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:13.453120+0000","last_clean_scrub_stamp":"2024-09-16T15:44:13.453120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T15:44:14.461845+0000","last_change":"2024-09-16T15:44:14.461845+0000","last_active":"2024-09-16T15:44:14.461845+0000","last_peered":"2024-09-16T15:44:14.461845+0000","last_clean":"2024-09-16T15:44:14.461845+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:44:14.461845+0000","last_undegraded":"2024-09-16T15:44:14.461845+0000","last_fullsized":"2024-09-16T15:44:14.461845+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_clean_scrub_stamp":"2024-09-16T15:44:14.461845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":1,"reported_epoch":41,"state":"creating+peering","last_fresh":"2024-09-16T15:44:13.468269+0000","last_change":"2024-09-16T15:44:13.468269+0000","last_active":"2024-09-16T15:44:13.453120+0000","last_peered":"2024-09-16T15:44:13.453120+0000","last_clean":"2024-09-16T15:44:13.453120+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:44:13.468269+0000","last_undegraded":"2024-09-16T15:44:13.468269+0000","last_fullsized":"2024-09-16T15:44:13.468269+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:13.453120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:13.453120+0000","last_clean_scrub_stamp":"2024-09-16T15:44:13.453120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T15:44:14.461845+0000","last_change":"2024-09-16T15:44:14.461845+0000","last_active":"2024-09-16T15:44:14.461845+0000","last_peered":"2024-09-16T15:44:14.461845+0000","last_clean":"2024-09-16T15:44:14.461845+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:44:14.461845+0000","last_undegraded":"2024-09-16T15:44:14.461845+0000","last_fullsized":"2024-09-16T15:44:14.461845+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_clean_scrub_stamp":"2024-09-16T15:44:14.461845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":1,"reported_epoch":41,"state":"creating+peering","last_fresh":"2024-09-16T15:44:13.473942+0000","last_change":"2024-09-16T15:44:13.473942+0000","last_active":"2024-09-16T15:44:13.453120+0000","last_peered":"2024-09-16T15:44:13.453120+0000","last_clean":"2024-09-16T15:44:13.453120+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:44:13.473942+0000","last_undegraded":"2024-09-16T15:44:13.473942+0000","last_fullsized":"2024-09-16T15:44:13.473942+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:13.453120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:13.453120+0000","last_clean_scrub_stamp":"2024-09-16T15:44:13.453120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T15:44:14.461845+0000","last_change":"2024-09-16T15:44:14.461845+0000","last_active":"2024-09-16T15:44:14.461845+0000","last_peered":"2024-09-16T15:44:14.461845+0000","last_clean":"2024-09-16T15:44:14.461845+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:44:14.461845+0000","last_undegraded":"2024-09-16T15:44:14.461845+0000","last_fullsized":"2024-09-16T15:44:14.461845+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_clean_scrub_stamp":"2024-09-16T15:44:14.461845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":1,"reported_epoch":41,"state":"creating+peering","last_fresh":"2024-09-16T15:44:13.476299+0000","last_change":"2024-09-16T15:44:13.476299+0000","last_active":"2024-09-16T15:44:13.453120+0000","last_peered":"2024-09-16T15:44:13.453120+0000","last_clean":"2024-09-16T15:44:13.453120+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:44:13.476299+0000","last_undegraded":"2024-09-16T15:44:13.476299+0000","last_fullsized":"2024-09-16T15:44:13.476299+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:13.453120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:13.453120+0000","last_clean_scrub_stamp":"2024-09-16T15:44:13.453120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T15:44:14.461845+0000","last_change":"2024-09-16T15:44:14.461845+0000","last_active":"2024-09-16T15:44:14.461845+0000","last_peered":"2024-09-16T15:44:14.461845+0000","last_clean":"2024-09-16T15:44:14.461845+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:44:14.461845+0000","last_undegraded":"2024-09-16T15:44:14.461845+0000","last_fullsized":"2024-09-16T15:44:14.461845+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_clean_scrub_stamp":"2024-09-16T15:44:14.461845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":1,"reported_epoch":41,"state":"creating+peering","last_fresh":"2024-09-16T15:44:13.470832+0000","last_change":"2024-09-16T15:44:13.470832+0000","last_active":"2024-09-16T15:44:13.453120+0000","last_peered":"2024-09-16T15:44:13.453120+0000","last_clean":"2024-09-16T15:44:13.453120+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:44:13.470832+0000","last_undegraded":"2024-09-16T15:44:13.470832+0000","last_fullsized":"2024-09-16T15:44:13.470832+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:13.453120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:13.453120+0000","last_clean_scrub_stamp":"2024-09-16T15:44:13.453120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T15:44:14.461845+0000","last_change":"2024-09-16T15:44:14.461845+0000","last_active":"2024-09-16T15:44:14.461845+0000","last_peered":"2024-09-16T15:44:14.461845+0000","last_clean":"2024-09-16T15:44:14.461845+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:44:14.461845+0000","last_undegraded":"2024-09-16T15:44:14.461845+0000","last_fullsized":"2024-09-16T15:44:14.461845+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_clean_scrub_stamp":"2024-09-16T15:44:14.461845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":1,"reported_epoch":41,"state":"creating+peering","last_fresh":"2024-09-16T15:44:13.469026+0000","last_change":"2024-09-16T15:44:13.469026+0000","last_active":"2024-09-16T15:44:13.453120+0000","last_peered":"2024-09-16T15:44:13.453120+0000","last_clean":"2024-09-16T15:44:13.453120+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:44:13.469026+0000","last_undegraded":"2024-09-16T15:44:13.469026+0000","last_fullsized":"2024-09-16T15:44:13.469026+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:13.453120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:13.453120+0000","last_clean_scrub_stamp":"2024-09-16T15:44:13.453120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T15:44:14.461845+0000","last_change":"2024-09-16T15:44:14.461845+0000","last_active":"2024-09-16T15:44:14.461845+0000","last_peered":"2024-09-16T15:44:14.461845+0000","last_clean":"2024-09-16T15:44:14.461845+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:44:14.461845+0000","last_undegraded":"2024-09-16T15:44:14.461845+0000","last_fullsized":"2024-09-16T15:44:14.461845+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_clean_scrub_stamp":"2024-09-16T15:44:14.461845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":1,"reported_epoch":41,"state":"creating+peering","last_fresh":"2024-09-16T15:44:13.474765+0000","last_change":"2024-09-16T15:44:13.474765+0000","last_active":"2024-09-16T15:44:13.453120+0000","last_peered":"2024-09-16T15:44:13.453120+0000","last_clean":"2024-09-16T15:44:13.453120+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:44:13.474765+0000","last_undegraded":"2024-09-16T15:44:13.474765+0000","last_fullsized":"2024-09-16T15:44:13.474765+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:13.453120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:13.453120+0000","last_clean_scrub_stamp":"2024-09-16T15:44:13.453120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T15:44:14.461845+0000","last_change":"2024-09-16T15:44:14.461845+0000","last_active":"2024-09-16T15:44:14.461845+0000","last_peered":"2024-09-16T15:44:14.461845+0000","last_clean":"2024-09-16T15:44:14.461845+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:44:14.461845+0000","last_undegraded":"2024-09-16T15:44:14.461845+0000","last_fullsized":"2024-09-16T15:44:14.461845+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_clean_scrub_stamp":"2024-09-16T15:44:14.461845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":1,"reported_epoch":41,"state":"creating+peering","last_fresh":"2024-09-16T15:44:13.468318+0000","last_change":"2024-09-16T15:44:13.468318+0000","last_active":"2024-09-16T15:44:13.453120+0000","last_peered":"2024-09-16T15:44:13.453120+0000","last_clean":"2024-09-16T15:44:13.453120+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:44:13.468318+0000","last_undegraded":"2024-09-16T15:44:13.468318+0000","last_fullsized":"2024-09-16T15:44:13.468318+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:13.453120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:13.453120+0000","last_clean_scrub_stamp":"2024-09-16T15:44:13.453120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T15:44:14.461845+0000","last_change":"2024-09-16T15:44:14.461845+0000","last_active":"2024-09-16T15:44:14.461845+0000","last_peered":"2024-09-16T15:44:14.461845+0000","last_clean":"2024-09-16T15:44:14.461845+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:44:14.461845+0000","last_undegraded":"2024-09-16T15:44:14.461845+0000","last_fullsized":"2024-09-16T15:44:14.461845+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_clean_scrub_stamp":"2024-09-16T15:44:14.461845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":1,"reported_epoch":41,"state":"creating+peering","last_fresh":"2024-09-16T15:44:13.473996+0000","last_change":"2024-09-16T15:44:13.473996+0000","last_active":"2024-09-16T15:44:13.453120+0000","last_peered":"2024-09-16T15:44:13.453120+0000","last_clean":"2024-09-16T15:44:13.453120+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:44:13.473996+0000","last_undegraded":"2024-09-16T15:44:13.473996+0000","last_fullsized":"2024-09-16T15:44:13.473996+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:13.453120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:13.453120+0000","last_clean_scrub_stamp":"2024-09-16T15:44:13.453120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T15:44:14.461845+0000","last_change":"2024-09-16T15:44:14.461845+0000","last_active":"2024-09-16T15:44:14.461845+0000","last_peered":"2024-09-16T15:44:14.461845+0000","last_clean":"2024-09-16T15:44:14.461845+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:44:14.461845+0000","last_undegraded":"2024-09-16T15:44:14.461845+0000","last_fullsized":"2024-09-16T15:44:14.461845+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_clean_scrub_stamp":"2024-09-16T15:44:14.461845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":1,"reported_epoch":41,"state":"creating+peering","last_fresh":"2024-09-16T15:44:13.471259+0000","last_change":"2024-09-16T15:44:13.471259+0000","last_active":"2024-09-16T15:44:13.453120+0000","last_peered":"2024-09-16T15:44:13.453120+0000","last_clean":"2024-09-16T15:44:13.453120+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:44:13.471259+0000","last_undegraded":"2024-09-16T15:44:13.471259+0000","last_fullsized":"2024-09-16T15:44:13.471259+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:13.453120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:13.453120+0000","last_clean_scrub_stamp":"2024-09-16T15:44:13.453120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T15:44:14.461845+0000","last_change":"2024-09-16T15:44:14.461845+0000","last_active":"2024-09-16T15:44:14.461845+0000","last_peered":"2024-09-16T15:44:14.461845+0000","last_clean":"2024-09-16T15:44:14.461845+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:44:14.461845+0000","last_undegraded":"2024-09-16T15:44:14.461845+0000","last_fullsized":"2024-09-16T15:44:14.461845+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_clean_scrub_stamp":"2024-09-16T15:44:14.461845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":1,"reported_epoch":41,"state":"creating+peering","last_fresh":"2024-09-16T15:44:13.468343+0000","last_change":"2024-09-16T15:44:13.468343+0000","last_active":"2024-09-16T15:44:13.453120+0000","last_peered":"2024-09-16T15:44:13.453120+0000","last_clean":"2024-09-16T15:44:13.453120+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:44:13.468343+0000","last_undegraded":"2024-09-16T15:44:13.468343+0000","last_fullsized":"2024-09-16T15:44:13.468343+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:13.453120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:13.453120+0000","last_clean_scrub_stamp":"2024-09-16T15:44:13.453120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T15:44:14.461845+0000","last_change":"2024-09-16T15:44:14.461845+0000","last_active":"2024-09-16T15:44:14.461845+0000","last_peered":"2024-09-16T15:44:14.461845+0000","last_clean":"2024-09-16T15:44:14.461845+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:44:14.461845+0000","last_undegraded":"2024-09-16T15:44:14.461845+0000","last_fullsized":"2024-09-16T15:44:14.461845+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_clean_scrub_stamp":"2024-09-16T15:44:14.461845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":1,"reported_epoch":41,"state":"creating+peering","last_fresh":"2024-09-16T15:44:13.478680+0000","last_change":"2024-09-16T15:44:13.478680+0000","last_active":"2024-09-16T15:44:13.453120+0000","last_peered":"2024-09-16T15:44:13.453120+0000","last_clean":"2024-09-16T15:44:13.453120+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:44:13.478680+0000","last_undegraded":"2024-09-16T15:44:13.478680+0000","last_fullsized":"2024-09-16T15:44:13.478680+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:13.453120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:13.453120+0000","last_clean_scrub_stamp":"2024-09-16T15:44:13.453120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T15:44:14.461845+0000","last_change":"2024-09-16T15:44:14.461845+0000","last_active":"2024-09-16T15:44:14.461845+0000","last_peered":"2024-09-16T15:44:14.461845+0000","last_clean":"2024-09-16T15:44:14.461845+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:44:14.461845+0000","last_undegraded":"2024-09-16T15:44:14.461845+0000","last_fullsized":"2024-09-16T15:44:14.461845+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_clean_scrub_stamp":"2024-09-16T15:44:14.461845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":1,"reported_epoch":41,"state":"creating+peering","last_fresh":"2024-09-16T15:44:13.468140+0000","last_change":"2024-09-16T15:44:13.468140+0000","last_active":"2024-09-16T15:44:13.453120+0000","last_peered":"2024-09-16T15:44:13.453120+0000","last_clean":"2024-09-16T15:44:13.453120+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:44:13.468140+0000","last_undegraded":"2024-09-16T15:44:13.468140+0000","last_fullsized":"2024-09-16T15:44:13.468140+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:13.453120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:13.453120+0000","last_clean_scrub_stamp":"2024-09-16T15:44:13.453120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T15:44:14.461845+0000","last_change":"2024-09-16T15:44:14.461845+0000","last_active":"2024-09-16T15:44:14.461845+0000","last_peered":"2024-09-16T15:44:14.461845+0000","last_clean":"2024-09-16T15:44:14.461845+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:44:14.461845+0000","last_undegraded":"2024-09-16T15:44:14.461845+0000","last_fullsized":"2024-09-16T15:44:14.461845+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_clean_scrub_stamp":"2024-09-16T15:44:14.461845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":1,"reported_epoch":41,"state":"creating+peering","last_fresh":"2024-09-16T15:44:13.468299+0000","last_change":"2024-09-16T15:44:13.468299+0000","last_active":"2024-09-16T15:44:13.453120+0000","last_peered":"2024-09-16T15:44:13.453120+0000","last_clean":"2024-09-16T15:44:13.453120+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:44:13.468299+0000","last_undegraded":"2024-09-16T15:44:13.468299+0000","last_fullsized":"2024-09-16T15:44:13.468299+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:13.453120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:13.453120+0000","last_clean_scrub_stamp":"2024-09-16T15:44:13.453120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T15:44:14.461845+0000","last_change":"2024-09-16T15:44:14.461845+0000","last_active":"2024-09-16T15:44:14.461845+0000","last_peered":"2024-09-16T15:44:14.461845+0000","last_clean":"2024-09-16T15:44:14.461845+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:44:14.461845+0000","last_undegraded":"2024-09-16T15:44:14.461845+0000","last_fullsized":"2024-09-16T15:44:14.461845+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_clean_scrub_stamp":"2024-09-16T15:44:14.461845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":1,"reported_epoch":41,"state":"creating+peering","last_fresh":"2024-09-16T15:44:13.473999+0000","last_change":"2024-09-16T15:44:13.473999+0000","last_active":"2024-09-16T15:44:13.453120+0000","last_peered":"2024-09-16T15:44:13.453120+0000","last_clean":"2024-09-16T15:44:13.453120+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:44:13.473999+0000","last_undegraded":"2024-09-16T15:44:13.473999+0000","last_fullsized":"2024-09-16T15:44:13.473999+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:13.453120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:13.453120+0000","last_clean_scrub_stamp":"2024-09-16T15:44:13.453120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T15:44:14.461845+0000","last_change":"2024-09-16T15:44:14.461845+0000","last_active":"2024-09-16T15:44:14.461845+0000","last_peered":"2024-09-16T15:44:14.461845+0000","last_clean":"2024-09-16T15:44:14.461845+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:44:14.461845+0000","last_undegraded":"2024-09-16T15:44:14.461845+0000","last_fullsized":"2024-09-16T15:44:14.461845+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_clean_scrub_stamp":"2024-09-16T15:44:14.461845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":1,"reported_epoch":41,"state":"creating+peering","last_fresh":"2024-09-16T15:44:13.473367+0000","last_change":"2024-09-16T15:44:13.473367+0000","last_active":"2024-09-16T15:44:13.453120+0000","last_peered":"2024-09-16T15:44:13.453120+0000","last_clean":"2024-09-16T15:44:13.453120+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:44:13.473367+0000","last_undegraded":"2024-09-16T15:44:13.473367+0000","last_fullsized":"2024-09-16T15:44:13.473367+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:13.453120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:13.453120+0000","last_clean_scrub_stamp":"2024-09-16T15:44:13.453120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T15:44:14.461845+0000","last_change":"2024-09-16T15:44:14.461845+0000","last_active":"2024-09-16T15:44:14.461845+0000","last_peered":"2024-09-16T15:44:14.461845+0000","last_clean":"2024-09-16T15:44:14.461845+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:44:14.461845+0000","last_undegraded":"2024-09-16T15:44:14.461845+0000","last_fullsized":"2024-09-16T15:44:14.461845+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_clean_scrub_stamp":"2024-09-16T15:44:14.461845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":1,"reported_epoch":41,"state":"creating+peering","last_fresh":"2024-09-16T15:44:13.474597+0000","last_change":"2024-09-16T15:44:13.474597+0000","last_active":"2024-09-16T15:44:13.453120+0000","last_peered":"2024-09-16T15:44:13.453120+0000","last_clean":"2024-09-16T15:44:13.453120+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:44:13.474597+0000","last_undegraded":"2024-09-16T15:44:13.474597+0000","last_fullsized":"2024-09-16T15:44:13.474597+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:13.453120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:13.453120+0000","last_clean_scrub_stamp":"2024-09-16T15:44:13.453120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T15:44:14.461845+0000","last_change":"2024-09-16T15:44:14.461845+0000","last_active":"2024-09-16T15:44:14.461845+0000","last_peered":"2024-09-16T15:44:14.461845+0000","last_clean":"2024-09-16T15:44:14.461845+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:44:14.461845+0000","last_undegraded":"2024-09-16T15:44:14.461845+0000","last_fullsized":"2024-09-16T15:44:14.461845+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_clean_scrub_stamp":"2024-09-16T15:44:14.461845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":1,"reported_epoch":41,"state":"creating+peering","last_fresh":"2024-09-16T15:44:13.468102+0000","last_change":"2024-09-16T15:44:13.468102+0000","last_active":"2024-09-16T15:44:13.453120+0000","last_peered":"2024-09-16T15:44:13.453120+0000","last_clean":"2024-09-16T15:44:13.453120+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:44:13.468102+0000","last_undegraded":"2024-09-16T15:44:13.468102+0000","last_fullsized":"2024-09-16T15:44:13.468102+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:13.453120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:13.453120+0000","last_clean_scrub_stamp":"2024-09-16T15:44:13.453120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T15:44:14.461845+0000","last_change":"2024-09-16T15:44:14.461845+0000","last_active":"2024-09-16T15:44:14.461845+0000","last_peered":"2024-09-16T15:44:14.461845+0000","last_clean":"2024-09-16T15:44:14.461845+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:44:14.461845+0000","last_undegraded":"2024-09-16T15:44:14.461845+0000","last_fullsized":"2024-09-16T15:44:14.461845+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_clean_scrub_stamp":"2024-09-16T15:44:14.461845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":1,"reported_epoch":41,"state":"creating+peering","last_fresh":"2024-09-16T15:44:13.474821+0000","last_change":"2024-09-16T15:44:13.474821+0000","last_active":"2024-09-16T15:44:13.453120+0000","last_peered":"2024-09-16T15:44:13.453120+0000","last_clean":"2024-09-16T15:44:13.453120+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:44:13.474821+0000","last_undegraded":"2024-09-16T15:44:13.474821+0000","last_fullsized":"2024-09-16T15:44:13.474821+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:13.453120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:13.453120+0000","last_clean_scrub_stamp":"2024-09-16T15:44:13.453120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T15:44:14.461845+0000","last_change":"2024-09-16T15:44:14.461845+0000","last_active":"2024-09-16T15:44:14.461845+0000","last_peered":"2024-09-16T15:44:14.461845+0000","last_clean":"2024-09-16T15:44:14.461845+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:44:14.461845+0000","last_undegraded":"2024-09-16T15:44:14.461845+0000","last_fullsized":"2024-09-16T15:44:14.461845+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_clean_scrub_stamp":"2024-09-16T15:44:14.461845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":1,"reported_epoch":41,"state":"creating+peering","last_fresh":"2024-09-16T15:44:13.472012+0000","last_change":"2024-09-16T15:44:13.472012+0000","last_active":"2024-09-16T15:44:13.453120+0000","last_peered":"2024-09-16T15:44:13.453120+0000","last_clean":"2024-09-16T15:44:13.453120+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:44:13.472012+0000","last_undegraded":"2024-09-16T15:44:13.472012+0000","last_fullsized":"2024-09-16T15:44:13.472012+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:13.453120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:13.453120+0000","last_clean_scrub_stamp":"2024-09-16T15:44:13.453120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T15:44:14.461845+0000","last_change":"2024-09-16T15:44:14.461845+0000","last_active":"2024-09-16T15:44:14.461845+0000","last_peered":"2024-09-16T15:44:14.461845+0000","last_clean":"2024-09-16T15:44:14.461845+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:44:14.461845+0000","last_undegraded":"2024-09-16T15:44:14.461845+0000","last_fullsized":"2024-09-16T15:44:14.461845+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_clean_scrub_stamp":"2024-09-16T15:44:14.461845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":1,"reported_epoch":41,"state":"creating+peering","last_fresh":"2024-09-16T15:44:13.474616+0000","last_change":"2024-09-16T15:44:13.474616+0000","last_active":"2024-09-16T15:44:13.453120+0000","last_peered":"2024-09-16T15:44:13.453120+0000","last_clean":"2024-09-16T15:44:13.453120+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:44:13.474616+0000","last_undegraded":"2024-09-16T15:44:13.474616+0000","last_fullsized":"2024-09-16T15:44:13.474616+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:13.453120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:13.453120+0000","last_clean_scrub_stamp":"2024-09-16T15:44:13.453120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T15:44:14.461845+0000","last_change":"2024-09-16T15:44:14.461845+0000","last_active":"2024-09-16T15:44:14.461845+0000","last_peered":"2024-09-16T15:44:14.461845+0000","last_clean":"2024-09-16T15:44:14.461845+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:44:14.461845+0000","last_undegraded":"2024-09-16T15:44:14.461845+0000","last_fullsized":"2024-09-16T15:44:14.461845+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_clean_scrub_stamp":"2024-09-16T15:44:14.461845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":1,"reported_epoch":41,"state":"creating+peering","last_fresh":"2024-09-16T15:44:13.467248+0000","last_change":"2024-09-16T15:44:13.467248+0000","last_active":"2024-09-16T15:44:13.453120+0000","last_peered":"2024-09-16T15:44:13.453120+0000","last_clean":"2024-09-16T15:44:13.453120+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:44:13.467248+0000","last_undegraded":"2024-09-16T15:44:13.467248+0000","last_fullsized":"2024-09-16T15:44:13.467248+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:13.453120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:13.453120+0000","last_clean_scrub_stamp":"2024-09-16T15:44:13.453120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T15:44:14.461845+0000","last_change":"2024-09-16T15:44:14.461845+0000","last_active":"2024-09-16T15:44:14.461845+0000","last_peered":"2024-09-16T15:44:14.461845+0000","last_clean":"2024-09-16T15:44:14.461845+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:44:14.461845+0000","last_undegraded":"2024-09-16T15:44:14.461845+0000","last_fullsized":"2024-09-16T15:44:14.461845+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_clean_scrub_stamp":"2024-09-16T15:44:14.461845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":1,"reported_epoch":41,"state":"creating+peering","last_fresh":"2024-09-16T15:44:13.472810+0000","last_change":"2024-09-16T15:44:13.472810+0000","last_active":"2024-09-16T15:44:13.453120+0000","last_peered":"2024-09-16T15:44:13.453120+0000","last_clean":"2024-09-16T15:44:13.453120+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:44:13.472810+0000","last_undegraded":"2024-09-16T15:44:13.472810+0000","last_fullsized":"2024-09-16T15:44:13.472810+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:13.453120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:13.453120+0000","last_clean_scrub_stamp":"2024-09-16T15:44:13.453120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T15:44:14.461845+0000","last_change":"2024-09-16T15:44:14.461845+0000","last_active":"2024-09-16T15:44:14.461845+0000","last_peered":"2024-09-16T15:44:14.461845+0000","last_clean":"2024-09-16T15:44:14.461845+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:44:14.461845+0000","last_undegraded":"2024-09-16T15:44:14.461845+0000","last_fullsized":"2024-09-16T15:44:14.461845+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_clean_scrub_stamp":"2024-09-16T15:44:14.461845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":1,"reported_epoch":41,"state":"creating+peering","last_fresh":"2024-09-16T15:44:13.473352+0000","last_change":"2024-09-16T15:44:13.473352+0000","last_active":"2024-09-16T15:44:13.453120+0000","last_peered":"2024-09-16T15:44:13.453120+0000","last_clean":"2024-09-16T15:44:13.453120+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:44:13.473352+0000","last_undegraded":"2024-09-16T15:44:13.473352+0000","last_fullsized":"2024-09-16T15:44:13.473352+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:13.453120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:13.453120+0000","last_clean_scrub_stamp":"2024-09-16T15:44:13.453120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T15:44:14.461845+0000","last_change":"2024-09-16T15:44:14.461845+0000","last_active":"2024-09-16T15:44:14.461845+0000","last_peered":"2024-09-16T15:44:14.461845+0000","last_clean":"2024-09-16T15:44:14.461845+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:44:14.461845+0000","last_undegraded":"2024-09-16T15:44:14.461845+0000","last_fullsized":"2024-09-16T15:44:14.461845+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_clean_scrub_stamp":"2024-09-16T15:44:14.461845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":1,"reported_epoch":41,"state":"creating+peering","last_fresh":"2024-09-16T15:44:13.476429+0000","last_change":"2024-09-16T15:44:13.476429+0000","last_active":"2024-09-16T15:44:13.453120+0000","last_peered":"2024-09-16T15:44:13.453120+0000","last_clean":"2024-09-16T15:44:13.453120+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:44:13.476429+0000","last_undegraded":"2024-09-16T15:44:13.476429+0000","last_fullsized":"2024-09-16T15:44:13.476429+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:13.453120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:13.453120+0000","last_clean_scrub_stamp":"2024-09-16T15:44:13.453120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"16'72","reported_seq":147,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-16T15:44:13.477262+0000","last_change":"2024-09-16T15:42:40.881820+0000","last_active":"2024-09-16T15:44:13.477262+0000","last_peered":"2024-09-16T15:44:13.477262+0000","last_clean":"2024-09-16T15:44:13.477262+0000","last_became_active":"2024-09-16T15:42:40.879965+0000","last_became_peered":"2024-09-16T15:42:40.879965+0000","last_unstale":"2024-09-16T15:44:13.477262+0000","last_undegraded":"2024-09-16T15:44:13.477262+0000","last_fullsized":"2024-09-16T15:44:13.477262+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:42:39.728762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:42:39.728762+0000","last_clean_scrub_stamp":"2024-09-16T15:42:39.728762+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:22:18.768932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542167,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28072,"kb_used_data":1044,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343768,"statfs":{"total":96636764160,"available":96608018432,"internally_reserved":0,"allocated":1069056,"data_stored":776814,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542166,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27492,"kb_used_data":452,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344348,"statfs":{"total":96636764160,"available":96608612352,"internally_reserved":0,"allocated":462848,"data_stored":180629,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":5,"up_from":14,"seq":60129542167,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27492,"kb_used_data":464,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344348,"statfs":{"total":96636764160,"available":96608612352,"internally_reserved":0,"allocated":475136,"data_stored":186446,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542165,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27492,"kb_used_data":464,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344348,"statfs":{"total":96636764160,"available":96608612352,"internally_reserved":0,"allocated":475136,"data_stored":186446,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":14,"seq":60129542166,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31652,"kb_used_data":464,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340188,"statfs":{"total":96636764160,"available":96604352512,"internally_reserved":0,"allocated":475136,"data_stored":186446,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":14,"seq":60129542166,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27508,"kb_used_data":464,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344332,"statfs":{"total":96636764160,"available":96608595968,"internally_reserved":0,"allocated":475136,"data_stored":186446,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":14,"seq":60129542168,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27480,"kb_used_data":452,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344360,"statfs":{"total":96636764160,"available":96608624640,"internally_reserved":0,"allocated":462848,"data_stored":180629,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":14,"seq":60129542168,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28072,"kb_used_data":1044,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343768,"statfs":{"total":96636764160,"available":96608018432,"internally_reserved":0,"allocated":1069056,"data_stored":776814,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T15:44:19.550 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T15:44:19.707 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:19.700+0000 7f7565b99640 1 -- 172.21.15.123:0/3534004082 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7560161520 msgr2=0x7f75600d0840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:44:19.707 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:19.700+0000 7f7565b99640 1 --2- 172.21.15.123:0/3534004082 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7560161520 0x7f75600d0840 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f7550009840 tx=0x7f755001c820 comp rx=0 tx=0).stop 2024-09-16T15:44:19.707 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:19.700+0000 7f7565b99640 1 -- 172.21.15.123:0/3534004082 shutdown_connections 2024-09-16T15:44:19.707 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:19.700+0000 7f7565b99640 1 --2- 172.21.15.123:0/3534004082 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f75600d23c0 0x7f75600d0d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:19.707 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:19.700+0000 7f7565b99640 1 --2- 172.21.15.123:0/3534004082 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7560161520 0x7f75600d0840 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:19.707 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:19.700+0000 7f7565b99640 1 -- 172.21.15.123:0/3534004082 >> 172.21.15.123:0/3534004082 conn(0x7f75600d3e30 msgr2=0x7f75600d4240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:44:19.707 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:19.700+0000 7f7565b99640 1 -- 172.21.15.123:0/3534004082 shutdown_connections 2024-09-16T15:44:19.708 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:19.700+0000 7f7565b99640 1 -- 172.21.15.123:0/3534004082 wait complete. 2024-09-16T15:44:19.708 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:19.704+0000 7f7565b99640 1 Processor -- start 2024-09-16T15:44:19.708 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:19.704+0000 7f7565b99640 1 -- start start 2024-09-16T15:44:19.710 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:19.704+0000 7f7565b99640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f75600d23c0 0x7f756015ac00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:44:19.710 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:19.704+0000 7f7565b99640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f75600d4950 0x7f756015ded0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:44:19.710 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:19.704+0000 7f7565b99640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7560161520 0x7f756015e410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:44:19.710 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:19.704+0000 7f7565b99640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f756016b2f0 con 0x7f75600d4950 2024-09-16T15:44:19.710 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:19.704+0000 7f7565b99640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f756016b170 con 0x7f7560161520 2024-09-16T15:44:19.710 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:19.704+0000 7f7565b99640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f756016b470 con 0x7f75600d23c0 2024-09-16T15:44:19.710 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:19.704+0000 7f755effd640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f75600d4950 0x7f756015ded0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:44:19.710 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:19.704+0000 7f755effd640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f75600d4950 0x7f756015ded0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:47270/0 (socket says 172.21.15.123:47270) 2024-09-16T15:44:19.710 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:19.704+0000 7f755effd640 1 -- 172.21.15.123:0/1469078098 learned_addr learned my addr 172.21.15.123:0/1469078098 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:44:19.710 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:19.704+0000 7f755f7fe640 1 --2- 172.21.15.123:0/1469078098 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f75600d23c0 0x7f756015ac00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:44:19.711 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:19.704+0000 7f755effd640 1 -- 172.21.15.123:0/1469078098 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f75600d23c0 msgr2=0x7f756015ac00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:44:19.711 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:19.704+0000 7f755effd640 1 --2- 172.21.15.123:0/1469078098 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f75600d23c0 0x7f756015ac00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:19.711 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:19.704+0000 7f755effd640 1 -- 172.21.15.123:0/1469078098 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7560161520 msgr2=0x7f756015e410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:44:19.711 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:19.704+0000 7f755effd640 1 --2- 172.21.15.123:0/1469078098 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7560161520 0x7f756015e410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:19.711 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:19.704+0000 7f755effd640 1 -- 172.21.15.123:0/1469078098 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f75600c8ea0 con 0x7f75600d4950 2024-09-16T15:44:19.711 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:19.704+0000 7f755effd640 1 --2- 172.21.15.123:0/1469078098 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f75600d4950 0x7f756015ded0 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f755400d980 tx=0x7f755400de40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:44:19.711 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:19.704+0000 7f7547fff640 1 -- 172.21.15.123:0/1469078098 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7554017070 con 0x7f75600d4950 2024-09-16T15:44:19.711 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:19.704+0000 7f7547fff640 1 -- 172.21.15.123:0/1469078098 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f7554009d20 con 0x7f75600d4950 2024-09-16T15:44:19.711 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:19.704+0000 7f7565b99640 1 -- 172.21.15.123:0/1469078098 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f75600cd680 con 0x7f75600d4950 2024-09-16T15:44:19.711 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:19.704+0000 7f7547fff640 1 -- 172.21.15.123:0/1469078098 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7554004280 con 0x7f75600d4950 2024-09-16T15:44:19.711 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:19.704+0000 7f7565b99640 1 -- 172.21.15.123:0/1469078098 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f75600c9420 con 0x7f75600d4950 2024-09-16T15:44:19.712 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:19.704+0000 7f7565b99640 1 -- 172.21.15.123:0/1469078098 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f75600cea40 con 0x7f75600d4950 2024-09-16T15:44:19.712 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:19.708+0000 7f7547fff640 1 -- 172.21.15.123:0/1469078098 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 7) ==== 97233+0+0 (secure 0 0 0) 0x7f7554003600 con 0x7f75600d4950 2024-09-16T15:44:19.712 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:19.708+0000 7f7547fff640 1 --2- 172.21.15.123:0/1469078098 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f754c073070 0x7f754c075530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:44:19.712 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:19.708+0000 7f7547fff640 1 -- 172.21.15.123:0/1469078098 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(44..44 src has 1..44) ==== 5709+0+0 (secure 0 0 0) 0x7f75540943e0 con 0x7f75600d4950 2024-09-16T15:44:19.712 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:19.708+0000 7f755f7fe640 1 --2- 172.21.15.123:0/1469078098 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f754c073070 0x7f754c075530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:44:19.713 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:19.708+0000 7f755f7fe640 1 --2- 172.21.15.123:0/1469078098 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f754c073070 0x7f754c075530 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f75500af000 tx=0x7f755001b060 comp rx=0 tx=0).ready entity=mgr.4135 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:44:19.716 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:19.712+0000 7f7547fff640 1 -- 172.21.15.123:0/1469078098 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f755405f220 con 0x7f75600d4950 2024-09-16T15:44:19.884 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:19.880+0000 7f7565b99640 1 -- 172.21.15.123:0/1469078098 --> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f75600760f0 con 0x7f754c073070 2024-09-16T15:44:19.894 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:19.888+0000 7f7547fff640 1 -- 172.21.15.123:0/1469078098 <== mgr.4135 v2:172.21.15.184:6832/3114346423 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+239845 (secure 0 0 0) 0x7f75600760f0 con 0x7f754c073070 2024-09-16T15:44:19.897 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-09-16T15:44:19.898 INFO:teuthology.orchestra.run.smithi123.stderr:dumped all 2024-09-16T15:44:19.899 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:19.892+0000 7f7565b99640 1 -- 172.21.15.123:0/1469078098 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f754c073070 msgr2=0x7f754c075530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:44:19.899 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:19.892+0000 7f7565b99640 1 --2- 172.21.15.123:0/1469078098 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f754c073070 0x7f754c075530 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f75500af000 tx=0x7f755001b060 comp rx=0 tx=0).stop 2024-09-16T15:44:19.899 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:19.892+0000 7f7565b99640 1 -- 172.21.15.123:0/1469078098 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f75600d4950 msgr2=0x7f756015ded0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:44:19.899 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:19.892+0000 7f7565b99640 1 --2- 172.21.15.123:0/1469078098 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f75600d4950 0x7f756015ded0 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f755400d980 tx=0x7f755400de40 comp rx=0 tx=0).stop 2024-09-16T15:44:19.899 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:19.892+0000 7f7565b99640 1 -- 172.21.15.123:0/1469078098 shutdown_connections 2024-09-16T15:44:19.899 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:19.892+0000 7f7565b99640 1 --2- 172.21.15.123:0/1469078098 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f754c073070 0x7f754c075530 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:19.899 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:19.892+0000 7f7565b99640 1 --2- 172.21.15.123:0/1469078098 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7560161520 0x7f756015e410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:19.899 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:19.892+0000 7f7565b99640 1 --2- 172.21.15.123:0/1469078098 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f75600d4950 0x7f756015ded0 unknown :-1 s=CLOSED pgs=455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:19.899 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:19.892+0000 7f7565b99640 1 --2- 172.21.15.123:0/1469078098 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f75600d23c0 0x7f756015ac00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:19.899 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:19.892+0000 7f7565b99640 1 -- 172.21.15.123:0/1469078098 >> 172.21.15.123:0/1469078098 conn(0x7f75600d3e30 msgr2=0x7f756006eb20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:44:19.899 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:19.892+0000 7f7565b99640 1 -- 172.21.15.123:0/1469078098 shutdown_connections 2024-09-16T15:44:19.900 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:19.892+0000 7f7565b99640 1 -- 172.21.15.123:0/1469078098 wait complete. 2024-09-16T15:44:19.915 INFO:teuthology.orchestra.run.smithi123.stdout:{"pg_ready":true,"pg_map":{"version":87,"stamp":"2024-09-16T15:44:19.274293+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":66,"acting":66,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":9,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":7,"kb":754974720,"kb_used":225260,"kb_used_data":4848,"kb_used_omap":12,"kb_used_meta":220211,"kb_avail":754749460,"statfs":{"total":773094113280,"available":772863447040,"internally_reserved":0,"allocated":4964352,"data_stored":2660670,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":225496625},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":13,"apply_latency_ms":13,"commit_latency_ns":13000000,"apply_latency_ns":13000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.810966"},"pg_stats":[{"pgid":"7.3f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T15:44:14.461845+0000","last_change":"2024-09-16T15:44:14.461845+0000","last_active":"2024-09-16T15:44:14.461845+0000","last_peered":"2024-09-16T15:44:14.461845+0000","last_clean":"2024-09-16T15:44:14.461845+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:44:14.461845+0000","last_undegraded":"2024-09-16T15:44:14.461845+0000","last_fullsized":"2024-09-16T15:44:14.461845+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_clean_scrub_stamp":"2024-09-16T15:44:14.461845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.3e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T15:44:14.461845+0000","last_change":"2024-09-16T15:44:14.461845+0000","last_active":"2024-09-16T15:44:14.461845+0000","last_peered":"2024-09-16T15:44:14.461845+0000","last_clean":"2024-09-16T15:44:14.461845+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:44:14.461845+0000","last_undegraded":"2024-09-16T15:44:14.461845+0000","last_fullsized":"2024-09-16T15:44:14.461845+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_clean_scrub_stamp":"2024-09-16T15:44:14.461845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.3d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T15:44:14.461845+0000","last_change":"2024-09-16T15:44:14.461845+0000","last_active":"2024-09-16T15:44:14.461845+0000","last_peered":"2024-09-16T15:44:14.461845+0000","last_clean":"2024-09-16T15:44:14.461845+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:44:14.461845+0000","last_undegraded":"2024-09-16T15:44:14.461845+0000","last_fullsized":"2024-09-16T15:44:14.461845+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_clean_scrub_stamp":"2024-09-16T15:44:14.461845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.3c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T15:44:14.461845+0000","last_change":"2024-09-16T15:44:14.461845+0000","last_active":"2024-09-16T15:44:14.461845+0000","last_peered":"2024-09-16T15:44:14.461845+0000","last_clean":"2024-09-16T15:44:14.461845+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:44:14.461845+0000","last_undegraded":"2024-09-16T15:44:14.461845+0000","last_fullsized":"2024-09-16T15:44:14.461845+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_clean_scrub_stamp":"2024-09-16T15:44:14.461845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.3b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T15:44:14.461845+0000","last_change":"2024-09-16T15:44:14.461845+0000","last_active":"2024-09-16T15:44:14.461845+0000","last_peered":"2024-09-16T15:44:14.461845+0000","last_clean":"2024-09-16T15:44:14.461845+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:44:14.461845+0000","last_undegraded":"2024-09-16T15:44:14.461845+0000","last_fullsized":"2024-09-16T15:44:14.461845+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_clean_scrub_stamp":"2024-09-16T15:44:14.461845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.3a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T15:44:14.461845+0000","last_change":"2024-09-16T15:44:14.461845+0000","last_active":"2024-09-16T15:44:14.461845+0000","last_peered":"2024-09-16T15:44:14.461845+0000","last_clean":"2024-09-16T15:44:14.461845+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:44:14.461845+0000","last_undegraded":"2024-09-16T15:44:14.461845+0000","last_fullsized":"2024-09-16T15:44:14.461845+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_clean_scrub_stamp":"2024-09-16T15:44:14.461845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.39","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T15:44:14.461845+0000","last_change":"2024-09-16T15:44:14.461845+0000","last_active":"2024-09-16T15:44:14.461845+0000","last_peered":"2024-09-16T15:44:14.461845+0000","last_clean":"2024-09-16T15:44:14.461845+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:44:14.461845+0000","last_undegraded":"2024-09-16T15:44:14.461845+0000","last_fullsized":"2024-09-16T15:44:14.461845+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_clean_scrub_stamp":"2024-09-16T15:44:14.461845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.38","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T15:44:14.461845+0000","last_change":"2024-09-16T15:44:14.461845+0000","last_active":"2024-09-16T15:44:14.461845+0000","last_peered":"2024-09-16T15:44:14.461845+0000","last_clean":"2024-09-16T15:44:14.461845+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:44:14.461845+0000","last_undegraded":"2024-09-16T15:44:14.461845+0000","last_fullsized":"2024-09-16T15:44:14.461845+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_clean_scrub_stamp":"2024-09-16T15:44:14.461845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.37","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T15:44:14.461845+0000","last_change":"2024-09-16T15:44:14.461845+0000","last_active":"2024-09-16T15:44:14.461845+0000","last_peered":"2024-09-16T15:44:14.461845+0000","last_clean":"2024-09-16T15:44:14.461845+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:44:14.461845+0000","last_undegraded":"2024-09-16T15:44:14.461845+0000","last_fullsized":"2024-09-16T15:44:14.461845+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_clean_scrub_stamp":"2024-09-16T15:44:14.461845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.36","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T15:44:14.461845+0000","last_change":"2024-09-16T15:44:14.461845+0000","last_active":"2024-09-16T15:44:14.461845+0000","last_peered":"2024-09-16T15:44:14.461845+0000","last_clean":"2024-09-16T15:44:14.461845+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:44:14.461845+0000","last_undegraded":"2024-09-16T15:44:14.461845+0000","last_fullsized":"2024-09-16T15:44:14.461845+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_clean_scrub_stamp":"2024-09-16T15:44:14.461845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.35","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T15:44:14.461845+0000","last_change":"2024-09-16T15:44:14.461845+0000","last_active":"2024-09-16T15:44:14.461845+0000","last_peered":"2024-09-16T15:44:14.461845+0000","last_clean":"2024-09-16T15:44:14.461845+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:44:14.461845+0000","last_undegraded":"2024-09-16T15:44:14.461845+0000","last_fullsized":"2024-09-16T15:44:14.461845+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_clean_scrub_stamp":"2024-09-16T15:44:14.461845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.34","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T15:44:14.461845+0000","last_change":"2024-09-16T15:44:14.461845+0000","last_active":"2024-09-16T15:44:14.461845+0000","last_peered":"2024-09-16T15:44:14.461845+0000","last_clean":"2024-09-16T15:44:14.461845+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:44:14.461845+0000","last_undegraded":"2024-09-16T15:44:14.461845+0000","last_fullsized":"2024-09-16T15:44:14.461845+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_clean_scrub_stamp":"2024-09-16T15:44:14.461845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.33","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T15:44:14.461845+0000","last_change":"2024-09-16T15:44:14.461845+0000","last_active":"2024-09-16T15:44:14.461845+0000","last_peered":"2024-09-16T15:44:14.461845+0000","last_clean":"2024-09-16T15:44:14.461845+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:44:14.461845+0000","last_undegraded":"2024-09-16T15:44:14.461845+0000","last_fullsized":"2024-09-16T15:44:14.461845+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_clean_scrub_stamp":"2024-09-16T15:44:14.461845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.32","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T15:44:14.461845+0000","last_change":"2024-09-16T15:44:14.461845+0000","last_active":"2024-09-16T15:44:14.461845+0000","last_peered":"2024-09-16T15:44:14.461845+0000","last_clean":"2024-09-16T15:44:14.461845+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:44:14.461845+0000","last_undegraded":"2024-09-16T15:44:14.461845+0000","last_fullsized":"2024-09-16T15:44:14.461845+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_clean_scrub_stamp":"2024-09-16T15:44:14.461845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.31","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T15:44:14.461845+0000","last_change":"2024-09-16T15:44:14.461845+0000","last_active":"2024-09-16T15:44:14.461845+0000","last_peered":"2024-09-16T15:44:14.461845+0000","last_clean":"2024-09-16T15:44:14.461845+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:44:14.461845+0000","last_undegraded":"2024-09-16T15:44:14.461845+0000","last_fullsized":"2024-09-16T15:44:14.461845+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_clean_scrub_stamp":"2024-09-16T15:44:14.461845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.30","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T15:44:14.461845+0000","last_change":"2024-09-16T15:44:14.461845+0000","last_active":"2024-09-16T15:44:14.461845+0000","last_peered":"2024-09-16T15:44:14.461845+0000","last_clean":"2024-09-16T15:44:14.461845+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:44:14.461845+0000","last_undegraded":"2024-09-16T15:44:14.461845+0000","last_fullsized":"2024-09-16T15:44:14.461845+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_clean_scrub_stamp":"2024-09-16T15:44:14.461845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.2f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T15:44:14.461845+0000","last_change":"2024-09-16T15:44:14.461845+0000","last_active":"2024-09-16T15:44:14.461845+0000","last_peered":"2024-09-16T15:44:14.461845+0000","last_clean":"2024-09-16T15:44:14.461845+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:44:14.461845+0000","last_undegraded":"2024-09-16T15:44:14.461845+0000","last_fullsized":"2024-09-16T15:44:14.461845+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_clean_scrub_stamp":"2024-09-16T15:44:14.461845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.2e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T15:44:14.461845+0000","last_change":"2024-09-16T15:44:14.461845+0000","last_active":"2024-09-16T15:44:14.461845+0000","last_peered":"2024-09-16T15:44:14.461845+0000","last_clean":"2024-09-16T15:44:14.461845+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:44:14.461845+0000","last_undegraded":"2024-09-16T15:44:14.461845+0000","last_fullsized":"2024-09-16T15:44:14.461845+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_clean_scrub_stamp":"2024-09-16T15:44:14.461845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.2d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T15:44:14.461845+0000","last_change":"2024-09-16T15:44:14.461845+0000","last_active":"2024-09-16T15:44:14.461845+0000","last_peered":"2024-09-16T15:44:14.461845+0000","last_clean":"2024-09-16T15:44:14.461845+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:44:14.461845+0000","last_undegraded":"2024-09-16T15:44:14.461845+0000","last_fullsized":"2024-09-16T15:44:14.461845+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_clean_scrub_stamp":"2024-09-16T15:44:14.461845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.2c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T15:44:14.461845+0000","last_change":"2024-09-16T15:44:14.461845+0000","last_active":"2024-09-16T15:44:14.461845+0000","last_peered":"2024-09-16T15:44:14.461845+0000","last_clean":"2024-09-16T15:44:14.461845+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:44:14.461845+0000","last_undegraded":"2024-09-16T15:44:14.461845+0000","last_fullsized":"2024-09-16T15:44:14.461845+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_clean_scrub_stamp":"2024-09-16T15:44:14.461845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.2b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T15:44:14.461845+0000","last_change":"2024-09-16T15:44:14.461845+0000","last_active":"2024-09-16T15:44:14.461845+0000","last_peered":"2024-09-16T15:44:14.461845+0000","last_clean":"2024-09-16T15:44:14.461845+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:44:14.461845+0000","last_undegraded":"2024-09-16T15:44:14.461845+0000","last_fullsized":"2024-09-16T15:44:14.461845+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_clean_scrub_stamp":"2024-09-16T15:44:14.461845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.2a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T15:44:14.461845+0000","last_change":"2024-09-16T15:44:14.461845+0000","last_active":"2024-09-16T15:44:14.461845+0000","last_peered":"2024-09-16T15:44:14.461845+0000","last_clean":"2024-09-16T15:44:14.461845+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:44:14.461845+0000","last_undegraded":"2024-09-16T15:44:14.461845+0000","last_fullsized":"2024-09-16T15:44:14.461845+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_clean_scrub_stamp":"2024-09-16T15:44:14.461845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.29","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T15:44:14.461845+0000","last_change":"2024-09-16T15:44:14.461845+0000","last_active":"2024-09-16T15:44:14.461845+0000","last_peered":"2024-09-16T15:44:14.461845+0000","last_clean":"2024-09-16T15:44:14.461845+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:44:14.461845+0000","last_undegraded":"2024-09-16T15:44:14.461845+0000","last_fullsized":"2024-09-16T15:44:14.461845+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_clean_scrub_stamp":"2024-09-16T15:44:14.461845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.28","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T15:44:14.461845+0000","last_change":"2024-09-16T15:44:14.461845+0000","last_active":"2024-09-16T15:44:14.461845+0000","last_peered":"2024-09-16T15:44:14.461845+0000","last_clean":"2024-09-16T15:44:14.461845+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:44:14.461845+0000","last_undegraded":"2024-09-16T15:44:14.461845+0000","last_fullsized":"2024-09-16T15:44:14.461845+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_clean_scrub_stamp":"2024-09-16T15:44:14.461845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.27","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T15:44:14.461845+0000","last_change":"2024-09-16T15:44:14.461845+0000","last_active":"2024-09-16T15:44:14.461845+0000","last_peered":"2024-09-16T15:44:14.461845+0000","last_clean":"2024-09-16T15:44:14.461845+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:44:14.461845+0000","last_undegraded":"2024-09-16T15:44:14.461845+0000","last_fullsized":"2024-09-16T15:44:14.461845+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_clean_scrub_stamp":"2024-09-16T15:44:14.461845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.26","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T15:44:14.461845+0000","last_change":"2024-09-16T15:44:14.461845+0000","last_active":"2024-09-16T15:44:14.461845+0000","last_peered":"2024-09-16T15:44:14.461845+0000","last_clean":"2024-09-16T15:44:14.461845+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:44:14.461845+0000","last_undegraded":"2024-09-16T15:44:14.461845+0000","last_fullsized":"2024-09-16T15:44:14.461845+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_clean_scrub_stamp":"2024-09-16T15:44:14.461845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.25","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T15:44:14.461845+0000","last_change":"2024-09-16T15:44:14.461845+0000","last_active":"2024-09-16T15:44:14.461845+0000","last_peered":"2024-09-16T15:44:14.461845+0000","last_clean":"2024-09-16T15:44:14.461845+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:44:14.461845+0000","last_undegraded":"2024-09-16T15:44:14.461845+0000","last_fullsized":"2024-09-16T15:44:14.461845+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_clean_scrub_stamp":"2024-09-16T15:44:14.461845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.24","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T15:44:14.461845+0000","last_change":"2024-09-16T15:44:14.461845+0000","last_active":"2024-09-16T15:44:14.461845+0000","last_peered":"2024-09-16T15:44:14.461845+0000","last_clean":"2024-09-16T15:44:14.461845+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:44:14.461845+0000","last_undegraded":"2024-09-16T15:44:14.461845+0000","last_fullsized":"2024-09-16T15:44:14.461845+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_clean_scrub_stamp":"2024-09-16T15:44:14.461845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.23","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T15:44:14.461845+0000","last_change":"2024-09-16T15:44:14.461845+0000","last_active":"2024-09-16T15:44:14.461845+0000","last_peered":"2024-09-16T15:44:14.461845+0000","last_clean":"2024-09-16T15:44:14.461845+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:44:14.461845+0000","last_undegraded":"2024-09-16T15:44:14.461845+0000","last_fullsized":"2024-09-16T15:44:14.461845+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_clean_scrub_stamp":"2024-09-16T15:44:14.461845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.22","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T15:44:14.461845+0000","last_change":"2024-09-16T15:44:14.461845+0000","last_active":"2024-09-16T15:44:14.461845+0000","last_peered":"2024-09-16T15:44:14.461845+0000","last_clean":"2024-09-16T15:44:14.461845+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:44:14.461845+0000","last_undegraded":"2024-09-16T15:44:14.461845+0000","last_fullsized":"2024-09-16T15:44:14.461845+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_clean_scrub_stamp":"2024-09-16T15:44:14.461845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.21","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T15:44:14.461845+0000","last_change":"2024-09-16T15:44:14.461845+0000","last_active":"2024-09-16T15:44:14.461845+0000","last_peered":"2024-09-16T15:44:14.461845+0000","last_clean":"2024-09-16T15:44:14.461845+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:44:14.461845+0000","last_undegraded":"2024-09-16T15:44:14.461845+0000","last_fullsized":"2024-09-16T15:44:14.461845+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_clean_scrub_stamp":"2024-09-16T15:44:14.461845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.20","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T15:44:14.461845+0000","last_change":"2024-09-16T15:44:14.461845+0000","last_active":"2024-09-16T15:44:14.461845+0000","last_peered":"2024-09-16T15:44:14.461845+0000","last_clean":"2024-09-16T15:44:14.461845+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:44:14.461845+0000","last_undegraded":"2024-09-16T15:44:14.461845+0000","last_fullsized":"2024-09-16T15:44:14.461845+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_clean_scrub_stamp":"2024-09-16T15:44:14.461845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T15:44:14.461845+0000","last_change":"2024-09-16T15:44:14.461845+0000","last_active":"2024-09-16T15:44:14.461845+0000","last_peered":"2024-09-16T15:44:14.461845+0000","last_clean":"2024-09-16T15:44:14.461845+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:44:14.461845+0000","last_undegraded":"2024-09-16T15:44:14.461845+0000","last_fullsized":"2024-09-16T15:44:14.461845+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_clean_scrub_stamp":"2024-09-16T15:44:14.461845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":1,"reported_epoch":41,"state":"creating+peering","last_fresh":"2024-09-16T15:44:13.470227+0000","last_change":"2024-09-16T15:44:13.470227+0000","last_active":"2024-09-16T15:44:13.453120+0000","last_peered":"2024-09-16T15:44:13.453120+0000","last_clean":"2024-09-16T15:44:13.453120+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:44:13.470227+0000","last_undegraded":"2024-09-16T15:44:13.470227+0000","last_fullsized":"2024-09-16T15:44:13.470227+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:13.453120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:13.453120+0000","last_clean_scrub_stamp":"2024-09-16T15:44:13.453120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T15:44:14.461845+0000","last_change":"2024-09-16T15:44:14.461845+0000","last_active":"2024-09-16T15:44:14.461845+0000","last_peered":"2024-09-16T15:44:14.461845+0000","last_clean":"2024-09-16T15:44:14.461845+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:44:14.461845+0000","last_undegraded":"2024-09-16T15:44:14.461845+0000","last_fullsized":"2024-09-16T15:44:14.461845+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_clean_scrub_stamp":"2024-09-16T15:44:14.461845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":1,"reported_epoch":41,"state":"creating+peering","last_fresh":"2024-09-16T15:44:13.469543+0000","last_change":"2024-09-16T15:44:13.469543+0000","last_active":"2024-09-16T15:44:13.453120+0000","last_peered":"2024-09-16T15:44:13.453120+0000","last_clean":"2024-09-16T15:44:13.453120+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:44:13.469543+0000","last_undegraded":"2024-09-16T15:44:13.469543+0000","last_fullsized":"2024-09-16T15:44:13.469543+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:13.453120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:13.453120+0000","last_clean_scrub_stamp":"2024-09-16T15:44:13.453120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T15:44:14.461845+0000","last_change":"2024-09-16T15:44:14.461845+0000","last_active":"2024-09-16T15:44:14.461845+0000","last_peered":"2024-09-16T15:44:14.461845+0000","last_clean":"2024-09-16T15:44:14.461845+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:44:14.461845+0000","last_undegraded":"2024-09-16T15:44:14.461845+0000","last_fullsized":"2024-09-16T15:44:14.461845+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_clean_scrub_stamp":"2024-09-16T15:44:14.461845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":1,"reported_epoch":41,"state":"creating+peering","last_fresh":"2024-09-16T15:44:13.472025+0000","last_change":"2024-09-16T15:44:13.472025+0000","last_active":"2024-09-16T15:44:13.453120+0000","last_peered":"2024-09-16T15:44:13.453120+0000","last_clean":"2024-09-16T15:44:13.453120+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:44:13.472025+0000","last_undegraded":"2024-09-16T15:44:13.472025+0000","last_fullsized":"2024-09-16T15:44:13.472025+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:13.453120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:13.453120+0000","last_clean_scrub_stamp":"2024-09-16T15:44:13.453120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T15:44:14.461845+0000","last_change":"2024-09-16T15:44:14.461845+0000","last_active":"2024-09-16T15:44:14.461845+0000","last_peered":"2024-09-16T15:44:14.461845+0000","last_clean":"2024-09-16T15:44:14.461845+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:44:14.461845+0000","last_undegraded":"2024-09-16T15:44:14.461845+0000","last_fullsized":"2024-09-16T15:44:14.461845+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_clean_scrub_stamp":"2024-09-16T15:44:14.461845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.1c","version":"0'0","reported_seq":1,"reported_epoch":41,"state":"creating+peering","last_fresh":"2024-09-16T15:44:13.475916+0000","last_change":"2024-09-16T15:44:13.475916+0000","last_active":"2024-09-16T15:44:13.453120+0000","last_peered":"2024-09-16T15:44:13.453120+0000","last_clean":"2024-09-16T15:44:13.453120+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:44:13.475916+0000","last_undegraded":"2024-09-16T15:44:13.475916+0000","last_fullsized":"2024-09-16T15:44:13.475916+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:13.453120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:13.453120+0000","last_clean_scrub_stamp":"2024-09-16T15:44:13.453120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T15:44:14.461845+0000","last_change":"2024-09-16T15:44:14.461845+0000","last_active":"2024-09-16T15:44:14.461845+0000","last_peered":"2024-09-16T15:44:14.461845+0000","last_clean":"2024-09-16T15:44:14.461845+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:44:14.461845+0000","last_undegraded":"2024-09-16T15:44:14.461845+0000","last_fullsized":"2024-09-16T15:44:14.461845+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_clean_scrub_stamp":"2024-09-16T15:44:14.461845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":1,"reported_epoch":41,"state":"creating+peering","last_fresh":"2024-09-16T15:44:13.470521+0000","last_change":"2024-09-16T15:44:13.470521+0000","last_active":"2024-09-16T15:44:13.453120+0000","last_peered":"2024-09-16T15:44:13.453120+0000","last_clean":"2024-09-16T15:44:13.453120+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:44:13.470521+0000","last_undegraded":"2024-09-16T15:44:13.470521+0000","last_fullsized":"2024-09-16T15:44:13.470521+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:13.453120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:13.453120+0000","last_clean_scrub_stamp":"2024-09-16T15:44:13.453120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T15:44:14.461845+0000","last_change":"2024-09-16T15:44:14.461845+0000","last_active":"2024-09-16T15:44:14.461845+0000","last_peered":"2024-09-16T15:44:14.461845+0000","last_clean":"2024-09-16T15:44:14.461845+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:44:14.461845+0000","last_undegraded":"2024-09-16T15:44:14.461845+0000","last_fullsized":"2024-09-16T15:44:14.461845+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_clean_scrub_stamp":"2024-09-16T15:44:14.461845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":1,"reported_epoch":41,"state":"creating+peering","last_fresh":"2024-09-16T15:44:13.468698+0000","last_change":"2024-09-16T15:44:13.468698+0000","last_active":"2024-09-16T15:44:13.453120+0000","last_peered":"2024-09-16T15:44:13.453120+0000","last_clean":"2024-09-16T15:44:13.453120+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:44:13.468698+0000","last_undegraded":"2024-09-16T15:44:13.468698+0000","last_fullsized":"2024-09-16T15:44:13.468698+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:13.453120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:13.453120+0000","last_clean_scrub_stamp":"2024-09-16T15:44:13.453120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T15:44:14.461845+0000","last_change":"2024-09-16T15:44:14.461845+0000","last_active":"2024-09-16T15:44:14.461845+0000","last_peered":"2024-09-16T15:44:14.461845+0000","last_clean":"2024-09-16T15:44:14.461845+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:44:14.461845+0000","last_undegraded":"2024-09-16T15:44:14.461845+0000","last_fullsized":"2024-09-16T15:44:14.461845+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_clean_scrub_stamp":"2024-09-16T15:44:14.461845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":1,"reported_epoch":41,"state":"creating+peering","last_fresh":"2024-09-16T15:44:13.478336+0000","last_change":"2024-09-16T15:44:13.478336+0000","last_active":"2024-09-16T15:44:13.453120+0000","last_peered":"2024-09-16T15:44:13.453120+0000","last_clean":"2024-09-16T15:44:13.453120+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:44:13.478336+0000","last_undegraded":"2024-09-16T15:44:13.478336+0000","last_fullsized":"2024-09-16T15:44:13.478336+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:13.453120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:13.453120+0000","last_clean_scrub_stamp":"2024-09-16T15:44:13.453120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T15:44:14.461845+0000","last_change":"2024-09-16T15:44:14.461845+0000","last_active":"2024-09-16T15:44:14.461845+0000","last_peered":"2024-09-16T15:44:14.461845+0000","last_clean":"2024-09-16T15:44:14.461845+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:44:14.461845+0000","last_undegraded":"2024-09-16T15:44:14.461845+0000","last_fullsized":"2024-09-16T15:44:14.461845+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_clean_scrub_stamp":"2024-09-16T15:44:14.461845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":1,"reported_epoch":41,"state":"creating+peering","last_fresh":"2024-09-16T15:44:13.475717+0000","last_change":"2024-09-16T15:44:13.475717+0000","last_active":"2024-09-16T15:44:13.453120+0000","last_peered":"2024-09-16T15:44:13.453120+0000","last_clean":"2024-09-16T15:44:13.453120+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:44:13.475717+0000","last_undegraded":"2024-09-16T15:44:13.475717+0000","last_fullsized":"2024-09-16T15:44:13.475717+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:13.453120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:13.453120+0000","last_clean_scrub_stamp":"2024-09-16T15:44:13.453120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T15:44:14.461845+0000","last_change":"2024-09-16T15:44:14.461845+0000","last_active":"2024-09-16T15:44:14.461845+0000","last_peered":"2024-09-16T15:44:14.461845+0000","last_clean":"2024-09-16T15:44:14.461845+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:44:14.461845+0000","last_undegraded":"2024-09-16T15:44:14.461845+0000","last_fullsized":"2024-09-16T15:44:14.461845+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_clean_scrub_stamp":"2024-09-16T15:44:14.461845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":1,"reported_epoch":41,"state":"creating+peering","last_fresh":"2024-09-16T15:44:13.468269+0000","last_change":"2024-09-16T15:44:13.468269+0000","last_active":"2024-09-16T15:44:13.453120+0000","last_peered":"2024-09-16T15:44:13.453120+0000","last_clean":"2024-09-16T15:44:13.453120+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:44:13.468269+0000","last_undegraded":"2024-09-16T15:44:13.468269+0000","last_fullsized":"2024-09-16T15:44:13.468269+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:13.453120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:13.453120+0000","last_clean_scrub_stamp":"2024-09-16T15:44:13.453120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T15:44:14.461845+0000","last_change":"2024-09-16T15:44:14.461845+0000","last_active":"2024-09-16T15:44:14.461845+0000","last_peered":"2024-09-16T15:44:14.461845+0000","last_clean":"2024-09-16T15:44:14.461845+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:44:14.461845+0000","last_undegraded":"2024-09-16T15:44:14.461845+0000","last_fullsized":"2024-09-16T15:44:14.461845+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_clean_scrub_stamp":"2024-09-16T15:44:14.461845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":1,"reported_epoch":41,"state":"creating+peering","last_fresh":"2024-09-16T15:44:13.473942+0000","last_change":"2024-09-16T15:44:13.473942+0000","last_active":"2024-09-16T15:44:13.453120+0000","last_peered":"2024-09-16T15:44:13.453120+0000","last_clean":"2024-09-16T15:44:13.453120+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:44:13.473942+0000","last_undegraded":"2024-09-16T15:44:13.473942+0000","last_fullsized":"2024-09-16T15:44:13.473942+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:13.453120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:13.453120+0000","last_clean_scrub_stamp":"2024-09-16T15:44:13.453120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T15:44:14.461845+0000","last_change":"2024-09-16T15:44:14.461845+0000","last_active":"2024-09-16T15:44:14.461845+0000","last_peered":"2024-09-16T15:44:14.461845+0000","last_clean":"2024-09-16T15:44:14.461845+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:44:14.461845+0000","last_undegraded":"2024-09-16T15:44:14.461845+0000","last_fullsized":"2024-09-16T15:44:14.461845+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_clean_scrub_stamp":"2024-09-16T15:44:14.461845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":1,"reported_epoch":41,"state":"creating+peering","last_fresh":"2024-09-16T15:44:13.476299+0000","last_change":"2024-09-16T15:44:13.476299+0000","last_active":"2024-09-16T15:44:13.453120+0000","last_peered":"2024-09-16T15:44:13.453120+0000","last_clean":"2024-09-16T15:44:13.453120+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:44:13.476299+0000","last_undegraded":"2024-09-16T15:44:13.476299+0000","last_fullsized":"2024-09-16T15:44:13.476299+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:13.453120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:13.453120+0000","last_clean_scrub_stamp":"2024-09-16T15:44:13.453120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T15:44:14.461845+0000","last_change":"2024-09-16T15:44:14.461845+0000","last_active":"2024-09-16T15:44:14.461845+0000","last_peered":"2024-09-16T15:44:14.461845+0000","last_clean":"2024-09-16T15:44:14.461845+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:44:14.461845+0000","last_undegraded":"2024-09-16T15:44:14.461845+0000","last_fullsized":"2024-09-16T15:44:14.461845+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_clean_scrub_stamp":"2024-09-16T15:44:14.461845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":1,"reported_epoch":41,"state":"creating+peering","last_fresh":"2024-09-16T15:44:13.470832+0000","last_change":"2024-09-16T15:44:13.470832+0000","last_active":"2024-09-16T15:44:13.453120+0000","last_peered":"2024-09-16T15:44:13.453120+0000","last_clean":"2024-09-16T15:44:13.453120+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:44:13.470832+0000","last_undegraded":"2024-09-16T15:44:13.470832+0000","last_fullsized":"2024-09-16T15:44:13.470832+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:13.453120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:13.453120+0000","last_clean_scrub_stamp":"2024-09-16T15:44:13.453120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T15:44:14.461845+0000","last_change":"2024-09-16T15:44:14.461845+0000","last_active":"2024-09-16T15:44:14.461845+0000","last_peered":"2024-09-16T15:44:14.461845+0000","last_clean":"2024-09-16T15:44:14.461845+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:44:14.461845+0000","last_undegraded":"2024-09-16T15:44:14.461845+0000","last_fullsized":"2024-09-16T15:44:14.461845+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_clean_scrub_stamp":"2024-09-16T15:44:14.461845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":1,"reported_epoch":41,"state":"creating+peering","last_fresh":"2024-09-16T15:44:13.469026+0000","last_change":"2024-09-16T15:44:13.469026+0000","last_active":"2024-09-16T15:44:13.453120+0000","last_peered":"2024-09-16T15:44:13.453120+0000","last_clean":"2024-09-16T15:44:13.453120+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:44:13.469026+0000","last_undegraded":"2024-09-16T15:44:13.469026+0000","last_fullsized":"2024-09-16T15:44:13.469026+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:13.453120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:13.453120+0000","last_clean_scrub_stamp":"2024-09-16T15:44:13.453120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T15:44:14.461845+0000","last_change":"2024-09-16T15:44:14.461845+0000","last_active":"2024-09-16T15:44:14.461845+0000","last_peered":"2024-09-16T15:44:14.461845+0000","last_clean":"2024-09-16T15:44:14.461845+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:44:14.461845+0000","last_undegraded":"2024-09-16T15:44:14.461845+0000","last_fullsized":"2024-09-16T15:44:14.461845+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_clean_scrub_stamp":"2024-09-16T15:44:14.461845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":1,"reported_epoch":41,"state":"creating+peering","last_fresh":"2024-09-16T15:44:13.474765+0000","last_change":"2024-09-16T15:44:13.474765+0000","last_active":"2024-09-16T15:44:13.453120+0000","last_peered":"2024-09-16T15:44:13.453120+0000","last_clean":"2024-09-16T15:44:13.453120+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:44:13.474765+0000","last_undegraded":"2024-09-16T15:44:13.474765+0000","last_fullsized":"2024-09-16T15:44:13.474765+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:13.453120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:13.453120+0000","last_clean_scrub_stamp":"2024-09-16T15:44:13.453120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T15:44:14.461845+0000","last_change":"2024-09-16T15:44:14.461845+0000","last_active":"2024-09-16T15:44:14.461845+0000","last_peered":"2024-09-16T15:44:14.461845+0000","last_clean":"2024-09-16T15:44:14.461845+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:44:14.461845+0000","last_undegraded":"2024-09-16T15:44:14.461845+0000","last_fullsized":"2024-09-16T15:44:14.461845+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_clean_scrub_stamp":"2024-09-16T15:44:14.461845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":1,"reported_epoch":41,"state":"creating+peering","last_fresh":"2024-09-16T15:44:13.468318+0000","last_change":"2024-09-16T15:44:13.468318+0000","last_active":"2024-09-16T15:44:13.453120+0000","last_peered":"2024-09-16T15:44:13.453120+0000","last_clean":"2024-09-16T15:44:13.453120+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:44:13.468318+0000","last_undegraded":"2024-09-16T15:44:13.468318+0000","last_fullsized":"2024-09-16T15:44:13.468318+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:13.453120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:13.453120+0000","last_clean_scrub_stamp":"2024-09-16T15:44:13.453120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T15:44:14.461845+0000","last_change":"2024-09-16T15:44:14.461845+0000","last_active":"2024-09-16T15:44:14.461845+0000","last_peered":"2024-09-16T15:44:14.461845+0000","last_clean":"2024-09-16T15:44:14.461845+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:44:14.461845+0000","last_undegraded":"2024-09-16T15:44:14.461845+0000","last_fullsized":"2024-09-16T15:44:14.461845+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_clean_scrub_stamp":"2024-09-16T15:44:14.461845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":1,"reported_epoch":41,"state":"creating+peering","last_fresh":"2024-09-16T15:44:13.473996+0000","last_change":"2024-09-16T15:44:13.473996+0000","last_active":"2024-09-16T15:44:13.453120+0000","last_peered":"2024-09-16T15:44:13.453120+0000","last_clean":"2024-09-16T15:44:13.453120+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:44:13.473996+0000","last_undegraded":"2024-09-16T15:44:13.473996+0000","last_fullsized":"2024-09-16T15:44:13.473996+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:13.453120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:13.453120+0000","last_clean_scrub_stamp":"2024-09-16T15:44:13.453120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T15:44:14.461845+0000","last_change":"2024-09-16T15:44:14.461845+0000","last_active":"2024-09-16T15:44:14.461845+0000","last_peered":"2024-09-16T15:44:14.461845+0000","last_clean":"2024-09-16T15:44:14.461845+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:44:14.461845+0000","last_undegraded":"2024-09-16T15:44:14.461845+0000","last_fullsized":"2024-09-16T15:44:14.461845+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_clean_scrub_stamp":"2024-09-16T15:44:14.461845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":1,"reported_epoch":41,"state":"creating+peering","last_fresh":"2024-09-16T15:44:13.471259+0000","last_change":"2024-09-16T15:44:13.471259+0000","last_active":"2024-09-16T15:44:13.453120+0000","last_peered":"2024-09-16T15:44:13.453120+0000","last_clean":"2024-09-16T15:44:13.453120+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:44:13.471259+0000","last_undegraded":"2024-09-16T15:44:13.471259+0000","last_fullsized":"2024-09-16T15:44:13.471259+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:13.453120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:13.453120+0000","last_clean_scrub_stamp":"2024-09-16T15:44:13.453120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T15:44:14.461845+0000","last_change":"2024-09-16T15:44:14.461845+0000","last_active":"2024-09-16T15:44:14.461845+0000","last_peered":"2024-09-16T15:44:14.461845+0000","last_clean":"2024-09-16T15:44:14.461845+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:44:14.461845+0000","last_undegraded":"2024-09-16T15:44:14.461845+0000","last_fullsized":"2024-09-16T15:44:14.461845+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_clean_scrub_stamp":"2024-09-16T15:44:14.461845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":1,"reported_epoch":41,"state":"creating+peering","last_fresh":"2024-09-16T15:44:13.468343+0000","last_change":"2024-09-16T15:44:13.468343+0000","last_active":"2024-09-16T15:44:13.453120+0000","last_peered":"2024-09-16T15:44:13.453120+0000","last_clean":"2024-09-16T15:44:13.453120+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:44:13.468343+0000","last_undegraded":"2024-09-16T15:44:13.468343+0000","last_fullsized":"2024-09-16T15:44:13.468343+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:13.453120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:13.453120+0000","last_clean_scrub_stamp":"2024-09-16T15:44:13.453120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T15:44:14.461845+0000","last_change":"2024-09-16T15:44:14.461845+0000","last_active":"2024-09-16T15:44:14.461845+0000","last_peered":"2024-09-16T15:44:14.461845+0000","last_clean":"2024-09-16T15:44:14.461845+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:44:14.461845+0000","last_undegraded":"2024-09-16T15:44:14.461845+0000","last_fullsized":"2024-09-16T15:44:14.461845+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_clean_scrub_stamp":"2024-09-16T15:44:14.461845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":1,"reported_epoch":41,"state":"creating+peering","last_fresh":"2024-09-16T15:44:13.478680+0000","last_change":"2024-09-16T15:44:13.478680+0000","last_active":"2024-09-16T15:44:13.453120+0000","last_peered":"2024-09-16T15:44:13.453120+0000","last_clean":"2024-09-16T15:44:13.453120+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:44:13.478680+0000","last_undegraded":"2024-09-16T15:44:13.478680+0000","last_fullsized":"2024-09-16T15:44:13.478680+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:13.453120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:13.453120+0000","last_clean_scrub_stamp":"2024-09-16T15:44:13.453120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T15:44:14.461845+0000","last_change":"2024-09-16T15:44:14.461845+0000","last_active":"2024-09-16T15:44:14.461845+0000","last_peered":"2024-09-16T15:44:14.461845+0000","last_clean":"2024-09-16T15:44:14.461845+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:44:14.461845+0000","last_undegraded":"2024-09-16T15:44:14.461845+0000","last_fullsized":"2024-09-16T15:44:14.461845+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_clean_scrub_stamp":"2024-09-16T15:44:14.461845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":1,"reported_epoch":41,"state":"creating+peering","last_fresh":"2024-09-16T15:44:13.468140+0000","last_change":"2024-09-16T15:44:13.468140+0000","last_active":"2024-09-16T15:44:13.453120+0000","last_peered":"2024-09-16T15:44:13.453120+0000","last_clean":"2024-09-16T15:44:13.453120+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:44:13.468140+0000","last_undegraded":"2024-09-16T15:44:13.468140+0000","last_fullsized":"2024-09-16T15:44:13.468140+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:13.453120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:13.453120+0000","last_clean_scrub_stamp":"2024-09-16T15:44:13.453120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T15:44:14.461845+0000","last_change":"2024-09-16T15:44:14.461845+0000","last_active":"2024-09-16T15:44:14.461845+0000","last_peered":"2024-09-16T15:44:14.461845+0000","last_clean":"2024-09-16T15:44:14.461845+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:44:14.461845+0000","last_undegraded":"2024-09-16T15:44:14.461845+0000","last_fullsized":"2024-09-16T15:44:14.461845+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_clean_scrub_stamp":"2024-09-16T15:44:14.461845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":1,"reported_epoch":41,"state":"creating+peering","last_fresh":"2024-09-16T15:44:13.468299+0000","last_change":"2024-09-16T15:44:13.468299+0000","last_active":"2024-09-16T15:44:13.453120+0000","last_peered":"2024-09-16T15:44:13.453120+0000","last_clean":"2024-09-16T15:44:13.453120+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:44:13.468299+0000","last_undegraded":"2024-09-16T15:44:13.468299+0000","last_fullsized":"2024-09-16T15:44:13.468299+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:13.453120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:13.453120+0000","last_clean_scrub_stamp":"2024-09-16T15:44:13.453120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T15:44:14.461845+0000","last_change":"2024-09-16T15:44:14.461845+0000","last_active":"2024-09-16T15:44:14.461845+0000","last_peered":"2024-09-16T15:44:14.461845+0000","last_clean":"2024-09-16T15:44:14.461845+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:44:14.461845+0000","last_undegraded":"2024-09-16T15:44:14.461845+0000","last_fullsized":"2024-09-16T15:44:14.461845+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_clean_scrub_stamp":"2024-09-16T15:44:14.461845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":1,"reported_epoch":41,"state":"creating+peering","last_fresh":"2024-09-16T15:44:13.473999+0000","last_change":"2024-09-16T15:44:13.473999+0000","last_active":"2024-09-16T15:44:13.453120+0000","last_peered":"2024-09-16T15:44:13.453120+0000","last_clean":"2024-09-16T15:44:13.453120+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:44:13.473999+0000","last_undegraded":"2024-09-16T15:44:13.473999+0000","last_fullsized":"2024-09-16T15:44:13.473999+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:13.453120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:13.453120+0000","last_clean_scrub_stamp":"2024-09-16T15:44:13.453120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T15:44:14.461845+0000","last_change":"2024-09-16T15:44:14.461845+0000","last_active":"2024-09-16T15:44:14.461845+0000","last_peered":"2024-09-16T15:44:14.461845+0000","last_clean":"2024-09-16T15:44:14.461845+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:44:14.461845+0000","last_undegraded":"2024-09-16T15:44:14.461845+0000","last_fullsized":"2024-09-16T15:44:14.461845+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_clean_scrub_stamp":"2024-09-16T15:44:14.461845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":1,"reported_epoch":41,"state":"creating+peering","last_fresh":"2024-09-16T15:44:13.473367+0000","last_change":"2024-09-16T15:44:13.473367+0000","last_active":"2024-09-16T15:44:13.453120+0000","last_peered":"2024-09-16T15:44:13.453120+0000","last_clean":"2024-09-16T15:44:13.453120+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:44:13.473367+0000","last_undegraded":"2024-09-16T15:44:13.473367+0000","last_fullsized":"2024-09-16T15:44:13.473367+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:13.453120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:13.453120+0000","last_clean_scrub_stamp":"2024-09-16T15:44:13.453120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T15:44:14.461845+0000","last_change":"2024-09-16T15:44:14.461845+0000","last_active":"2024-09-16T15:44:14.461845+0000","last_peered":"2024-09-16T15:44:14.461845+0000","last_clean":"2024-09-16T15:44:14.461845+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:44:14.461845+0000","last_undegraded":"2024-09-16T15:44:14.461845+0000","last_fullsized":"2024-09-16T15:44:14.461845+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_clean_scrub_stamp":"2024-09-16T15:44:14.461845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":1,"reported_epoch":41,"state":"creating+peering","last_fresh":"2024-09-16T15:44:13.474597+0000","last_change":"2024-09-16T15:44:13.474597+0000","last_active":"2024-09-16T15:44:13.453120+0000","last_peered":"2024-09-16T15:44:13.453120+0000","last_clean":"2024-09-16T15:44:13.453120+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:44:13.474597+0000","last_undegraded":"2024-09-16T15:44:13.474597+0000","last_fullsized":"2024-09-16T15:44:13.474597+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:13.453120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:13.453120+0000","last_clean_scrub_stamp":"2024-09-16T15:44:13.453120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T15:44:14.461845+0000","last_change":"2024-09-16T15:44:14.461845+0000","last_active":"2024-09-16T15:44:14.461845+0000","last_peered":"2024-09-16T15:44:14.461845+0000","last_clean":"2024-09-16T15:44:14.461845+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:44:14.461845+0000","last_undegraded":"2024-09-16T15:44:14.461845+0000","last_fullsized":"2024-09-16T15:44:14.461845+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_clean_scrub_stamp":"2024-09-16T15:44:14.461845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":1,"reported_epoch":41,"state":"creating+peering","last_fresh":"2024-09-16T15:44:13.468102+0000","last_change":"2024-09-16T15:44:13.468102+0000","last_active":"2024-09-16T15:44:13.453120+0000","last_peered":"2024-09-16T15:44:13.453120+0000","last_clean":"2024-09-16T15:44:13.453120+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:44:13.468102+0000","last_undegraded":"2024-09-16T15:44:13.468102+0000","last_fullsized":"2024-09-16T15:44:13.468102+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:13.453120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:13.453120+0000","last_clean_scrub_stamp":"2024-09-16T15:44:13.453120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T15:44:14.461845+0000","last_change":"2024-09-16T15:44:14.461845+0000","last_active":"2024-09-16T15:44:14.461845+0000","last_peered":"2024-09-16T15:44:14.461845+0000","last_clean":"2024-09-16T15:44:14.461845+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:44:14.461845+0000","last_undegraded":"2024-09-16T15:44:14.461845+0000","last_fullsized":"2024-09-16T15:44:14.461845+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_clean_scrub_stamp":"2024-09-16T15:44:14.461845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":1,"reported_epoch":41,"state":"creating+peering","last_fresh":"2024-09-16T15:44:13.474821+0000","last_change":"2024-09-16T15:44:13.474821+0000","last_active":"2024-09-16T15:44:13.453120+0000","last_peered":"2024-09-16T15:44:13.453120+0000","last_clean":"2024-09-16T15:44:13.453120+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:44:13.474821+0000","last_undegraded":"2024-09-16T15:44:13.474821+0000","last_fullsized":"2024-09-16T15:44:13.474821+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:13.453120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:13.453120+0000","last_clean_scrub_stamp":"2024-09-16T15:44:13.453120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T15:44:14.461845+0000","last_change":"2024-09-16T15:44:14.461845+0000","last_active":"2024-09-16T15:44:14.461845+0000","last_peered":"2024-09-16T15:44:14.461845+0000","last_clean":"2024-09-16T15:44:14.461845+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:44:14.461845+0000","last_undegraded":"2024-09-16T15:44:14.461845+0000","last_fullsized":"2024-09-16T15:44:14.461845+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_clean_scrub_stamp":"2024-09-16T15:44:14.461845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":1,"reported_epoch":41,"state":"creating+peering","last_fresh":"2024-09-16T15:44:13.472012+0000","last_change":"2024-09-16T15:44:13.472012+0000","last_active":"2024-09-16T15:44:13.453120+0000","last_peered":"2024-09-16T15:44:13.453120+0000","last_clean":"2024-09-16T15:44:13.453120+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:44:13.472012+0000","last_undegraded":"2024-09-16T15:44:13.472012+0000","last_fullsized":"2024-09-16T15:44:13.472012+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:13.453120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:13.453120+0000","last_clean_scrub_stamp":"2024-09-16T15:44:13.453120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T15:44:14.461845+0000","last_change":"2024-09-16T15:44:14.461845+0000","last_active":"2024-09-16T15:44:14.461845+0000","last_peered":"2024-09-16T15:44:14.461845+0000","last_clean":"2024-09-16T15:44:14.461845+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:44:14.461845+0000","last_undegraded":"2024-09-16T15:44:14.461845+0000","last_fullsized":"2024-09-16T15:44:14.461845+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_clean_scrub_stamp":"2024-09-16T15:44:14.461845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":1,"reported_epoch":41,"state":"creating+peering","last_fresh":"2024-09-16T15:44:13.474616+0000","last_change":"2024-09-16T15:44:13.474616+0000","last_active":"2024-09-16T15:44:13.453120+0000","last_peered":"2024-09-16T15:44:13.453120+0000","last_clean":"2024-09-16T15:44:13.453120+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:44:13.474616+0000","last_undegraded":"2024-09-16T15:44:13.474616+0000","last_fullsized":"2024-09-16T15:44:13.474616+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:13.453120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:13.453120+0000","last_clean_scrub_stamp":"2024-09-16T15:44:13.453120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T15:44:14.461845+0000","last_change":"2024-09-16T15:44:14.461845+0000","last_active":"2024-09-16T15:44:14.461845+0000","last_peered":"2024-09-16T15:44:14.461845+0000","last_clean":"2024-09-16T15:44:14.461845+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:44:14.461845+0000","last_undegraded":"2024-09-16T15:44:14.461845+0000","last_fullsized":"2024-09-16T15:44:14.461845+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_clean_scrub_stamp":"2024-09-16T15:44:14.461845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":1,"reported_epoch":41,"state":"creating+peering","last_fresh":"2024-09-16T15:44:13.467248+0000","last_change":"2024-09-16T15:44:13.467248+0000","last_active":"2024-09-16T15:44:13.453120+0000","last_peered":"2024-09-16T15:44:13.453120+0000","last_clean":"2024-09-16T15:44:13.453120+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:44:13.467248+0000","last_undegraded":"2024-09-16T15:44:13.467248+0000","last_fullsized":"2024-09-16T15:44:13.467248+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:13.453120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:13.453120+0000","last_clean_scrub_stamp":"2024-09-16T15:44:13.453120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T15:44:14.461845+0000","last_change":"2024-09-16T15:44:14.461845+0000","last_active":"2024-09-16T15:44:14.461845+0000","last_peered":"2024-09-16T15:44:14.461845+0000","last_clean":"2024-09-16T15:44:14.461845+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:44:14.461845+0000","last_undegraded":"2024-09-16T15:44:14.461845+0000","last_fullsized":"2024-09-16T15:44:14.461845+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_clean_scrub_stamp":"2024-09-16T15:44:14.461845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":1,"reported_epoch":41,"state":"creating+peering","last_fresh":"2024-09-16T15:44:13.472810+0000","last_change":"2024-09-16T15:44:13.472810+0000","last_active":"2024-09-16T15:44:13.453120+0000","last_peered":"2024-09-16T15:44:13.453120+0000","last_clean":"2024-09-16T15:44:13.453120+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:44:13.472810+0000","last_undegraded":"2024-09-16T15:44:13.472810+0000","last_fullsized":"2024-09-16T15:44:13.472810+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:13.453120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:13.453120+0000","last_clean_scrub_stamp":"2024-09-16T15:44:13.453120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T15:44:14.461845+0000","last_change":"2024-09-16T15:44:14.461845+0000","last_active":"2024-09-16T15:44:14.461845+0000","last_peered":"2024-09-16T15:44:14.461845+0000","last_clean":"2024-09-16T15:44:14.461845+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:44:14.461845+0000","last_undegraded":"2024-09-16T15:44:14.461845+0000","last_fullsized":"2024-09-16T15:44:14.461845+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_clean_scrub_stamp":"2024-09-16T15:44:14.461845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":1,"reported_epoch":41,"state":"creating+peering","last_fresh":"2024-09-16T15:44:13.473352+0000","last_change":"2024-09-16T15:44:13.473352+0000","last_active":"2024-09-16T15:44:13.453120+0000","last_peered":"2024-09-16T15:44:13.453120+0000","last_clean":"2024-09-16T15:44:13.453120+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:44:13.473352+0000","last_undegraded":"2024-09-16T15:44:13.473352+0000","last_fullsized":"2024-09-16T15:44:13.473352+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:13.453120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:13.453120+0000","last_clean_scrub_stamp":"2024-09-16T15:44:13.453120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T15:44:14.461845+0000","last_change":"2024-09-16T15:44:14.461845+0000","last_active":"2024-09-16T15:44:14.461845+0000","last_peered":"2024-09-16T15:44:14.461845+0000","last_clean":"2024-09-16T15:44:14.461845+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:44:14.461845+0000","last_undegraded":"2024-09-16T15:44:14.461845+0000","last_fullsized":"2024-09-16T15:44:14.461845+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_clean_scrub_stamp":"2024-09-16T15:44:14.461845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":1,"reported_epoch":41,"state":"creating+peering","last_fresh":"2024-09-16T15:44:13.476429+0000","last_change":"2024-09-16T15:44:13.476429+0000","last_active":"2024-09-16T15:44:13.453120+0000","last_peered":"2024-09-16T15:44:13.453120+0000","last_clean":"2024-09-16T15:44:13.453120+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:44:13.476429+0000","last_undegraded":"2024-09-16T15:44:13.476429+0000","last_fullsized":"2024-09-16T15:44:13.476429+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:13.453120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:13.453120+0000","last_clean_scrub_stamp":"2024-09-16T15:44:13.453120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"16'72","reported_seq":147,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-16T15:44:13.477262+0000","last_change":"2024-09-16T15:42:40.881820+0000","last_active":"2024-09-16T15:44:13.477262+0000","last_peered":"2024-09-16T15:44:13.477262+0000","last_clean":"2024-09-16T15:44:13.477262+0000","last_became_active":"2024-09-16T15:42:40.879965+0000","last_became_peered":"2024-09-16T15:42:40.879965+0000","last_unstale":"2024-09-16T15:44:13.477262+0000","last_undegraded":"2024-09-16T15:44:13.477262+0000","last_fullsized":"2024-09-16T15:44:13.477262+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:42:39.728762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:42:39.728762+0000","last_clean_scrub_stamp":"2024-09-16T15:42:39.728762+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:22:18.768932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542167,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28072,"kb_used_data":1044,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343768,"statfs":{"total":96636764160,"available":96608018432,"internally_reserved":0,"allocated":1069056,"data_stored":776814,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542166,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27492,"kb_used_data":452,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344348,"statfs":{"total":96636764160,"available":96608612352,"internally_reserved":0,"allocated":462848,"data_stored":180629,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":5,"up_from":14,"seq":60129542167,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27492,"kb_used_data":464,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344348,"statfs":{"total":96636764160,"available":96608612352,"internally_reserved":0,"allocated":475136,"data_stored":186446,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542165,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27492,"kb_used_data":464,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344348,"statfs":{"total":96636764160,"available":96608612352,"internally_reserved":0,"allocated":475136,"data_stored":186446,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":14,"seq":60129542166,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31652,"kb_used_data":464,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340188,"statfs":{"total":96636764160,"available":96604352512,"internally_reserved":0,"allocated":475136,"data_stored":186446,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":14,"seq":60129542166,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27508,"kb_used_data":464,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344332,"statfs":{"total":96636764160,"available":96608595968,"internally_reserved":0,"allocated":475136,"data_stored":186446,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":14,"seq":60129542168,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27480,"kb_used_data":452,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344360,"statfs":{"total":96636764160,"available":96608624640,"internally_reserved":0,"allocated":462848,"data_stored":180629,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":14,"seq":60129542168,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28072,"kb_used_data":1044,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343768,"statfs":{"total":96636764160,"available":96608018432,"internally_reserved":0,"allocated":1069056,"data_stored":776814,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T15:44:19.918 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T15:44:20.091 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:20.084+0000 7fc88147d640 1 -- 172.21.15.123:0/2177735246 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc87c0d2820 msgr2=0x7fc87c0cea00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:44:20.091 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:20.084+0000 7fc88147d640 1 --2- 172.21.15.123:0/2177735246 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc87c0d2820 0x7fc87c0cea00 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7fc87c06f930 tx=0x7fc87401cc30 comp rx=0 tx=0).stop 2024-09-16T15:44:20.091 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:20.088+0000 7fc88147d640 1 -- 172.21.15.123:0/2177735246 shutdown_connections 2024-09-16T15:44:20.091 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:20.088+0000 7fc88147d640 1 --2- 172.21.15.123:0/2177735246 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc87c0d2820 0x7fc87c0cea00 unknown :-1 s=CLOSED pgs=456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:20.092 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:20.088+0000 7fc88147d640 1 --2- 172.21.15.123:0/2177735246 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fc87c0b2b00 0x7fc87c0d22e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:20.092 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:20.088+0000 7fc88147d640 1 --2- 172.21.15.123:0/2177735246 >> v2:172.21.15.184:3300/0 conn(0x7fc87c161620 0x7fc87c0d1da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:20.092 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:20.088+0000 7fc88147d640 1 -- 172.21.15.123:0/2177735246 >> 172.21.15.123:0/2177735246 conn(0x7fc87c0d3ea0 msgr2=0x7fc87c0d42b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:44:20.092 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:20.088+0000 7fc88147d640 1 -- 172.21.15.123:0/2177735246 shutdown_connections 2024-09-16T15:44:20.092 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:20.088+0000 7fc88147d640 1 -- 172.21.15.123:0/2177735246 wait complete. 2024-09-16T15:44:20.092 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:20.088+0000 7fc88147d640 1 Processor -- start 2024-09-16T15:44:20.092 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:20.088+0000 7fc88147d640 1 -- start start 2024-09-16T15:44:20.094 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:20.088+0000 7fc88147d640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc87c0b2b00 0x7fc87c14fdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:44:20.095 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:20.088+0000 7fc88147d640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc87c0d2820 0x7fc87c0c9db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:44:20.095 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:20.088+0000 7fc88147d640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fc87c161620 0x7fc87c0cd1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:44:20.095 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:20.088+0000 7fc88147d640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fc87c0b10e0 con 0x7fc87c0b2b00 2024-09-16T15:44:20.095 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:20.088+0000 7fc88147d640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fc87c0b0f60 con 0x7fc87c0d2820 2024-09-16T15:44:20.095 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:20.088+0000 7fc88147d640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fc87c0b1260 con 0x7fc87c161620 2024-09-16T15:44:20.095 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:20.088+0000 7fc87affd640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc87c0b2b00 0x7fc87c14fdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:44:20.095 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:20.088+0000 7fc87affd640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc87c0b2b00 0x7fc87c14fdf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:47282/0 (socket says 172.21.15.123:47282) 2024-09-16T15:44:20.095 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:20.088+0000 7fc87affd640 1 -- 172.21.15.123:0/2074664909 learned_addr learned my addr 172.21.15.123:0/2074664909 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:44:20.095 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:20.088+0000 7fc87b7fe640 1 --2- 172.21.15.123:0/2074664909 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fc87c161620 0x7fc87c0cd1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:44:20.095 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:20.088+0000 7fc8727fc640 1 --2- 172.21.15.123:0/2074664909 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc87c0d2820 0x7fc87c0c9db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:44:20.095 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:20.088+0000 7fc87affd640 1 -- 172.21.15.123:0/2074664909 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fc87c161620 msgr2=0x7fc87c0cd1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:44:20.095 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:20.088+0000 7fc87affd640 1 --2- 172.21.15.123:0/2074664909 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fc87c161620 0x7fc87c0cd1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:20.095 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:20.088+0000 7fc87affd640 1 -- 172.21.15.123:0/2074664909 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc87c0d2820 msgr2=0x7fc87c0c9db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:44:20.096 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:20.088+0000 7fc87affd640 1 --2- 172.21.15.123:0/2074664909 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc87c0d2820 0x7fc87c0c9db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:20.096 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:20.088+0000 7fc87affd640 1 -- 172.21.15.123:0/2074664909 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc87c158fe0 con 0x7fc87c0b2b00 2024-09-16T15:44:20.096 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:20.088+0000 7fc87affd640 1 --2- 172.21.15.123:0/2074664909 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc87c0b2b00 0x7fc87c14fdf0 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7fc86c00c970 tx=0x7fc86c00ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:44:20.096 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:20.088+0000 7fc873fff640 1 -- 172.21.15.123:0/2074664909 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc86c007bb0 con 0x7fc87c0b2b00 2024-09-16T15:44:20.096 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:20.088+0000 7fc873fff640 1 -- 172.21.15.123:0/2074664909 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fc86c007e40 con 0x7fc87c0b2b00 2024-09-16T15:44:20.096 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:20.088+0000 7fc88147d640 1 -- 172.21.15.123:0/2074664909 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc87c1592c0 con 0x7fc87c0b2b00 2024-09-16T15:44:20.096 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:20.088+0000 7fc873fff640 1 -- 172.21.15.123:0/2074664909 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc86c005270 con 0x7fc87c0b2b00 2024-09-16T15:44:20.096 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:20.088+0000 7fc88147d640 1 -- 172.21.15.123:0/2074664909 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc87c159880 con 0x7fc87c0b2b00 2024-09-16T15:44:20.097 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:20.092+0000 7fc88147d640 1 -- 172.21.15.123:0/2074664909 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc87c0cea00 con 0x7fc87c0b2b00 2024-09-16T15:44:20.097 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:20.092+0000 7fc873fff640 1 -- 172.21.15.123:0/2074664909 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 7) ==== 97233+0+0 (secure 0 0 0) 0x7fc86c005410 con 0x7fc87c0b2b00 2024-09-16T15:44:20.097 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:20.092+0000 7fc873fff640 1 --2- 172.21.15.123:0/2074664909 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fc85c073070 0x7fc85c075530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:44:20.097 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:20.092+0000 7fc873fff640 1 -- 172.21.15.123:0/2074664909 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(44..44 src has 1..44) ==== 5709+0+0 (secure 0 0 0) 0x7fc86c094700 con 0x7fc87c0b2b00 2024-09-16T15:44:20.097 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:20.092+0000 7fc8727fc640 1 --2- 172.21.15.123:0/2074664909 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fc85c073070 0x7fc85c075530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:44:20.097 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:20.092+0000 7fc8727fc640 1 --2- 172.21.15.123:0/2074664909 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fc85c073070 0x7fc85c075530 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fc864005fd0 tx=0x7fc86400c480 comp rx=0 tx=0).ready entity=mgr.4135 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:44:20.100 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:20.096+0000 7fc873fff640 1 -- 172.21.15.123:0/2074664909 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7fc86c014070 con 0x7fc87c0b2b00 2024-09-16T15:44:20.267 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:20.260+0000 7fc88147d640 1 -- 172.21.15.123:0/2074664909 --> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fc87c15ed80 con 0x7fc85c073070 2024-09-16T15:44:20.277 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:20.276+0000 7fc873fff640 1 -- 172.21.15.123:0/2074664909 <== mgr.4135 v2:172.21.15.184:6832/3114346423 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+239845 (secure 0 0 0) 0x7fc87c15ed80 con 0x7fc85c073070 2024-09-16T15:44:20.280 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-09-16T15:44:20.282 INFO:teuthology.orchestra.run.smithi123.stderr:dumped all 2024-09-16T15:44:20.283 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:20.276+0000 7fc88147d640 1 -- 172.21.15.123:0/2074664909 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fc85c073070 msgr2=0x7fc85c075530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:44:20.283 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:20.276+0000 7fc88147d640 1 --2- 172.21.15.123:0/2074664909 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fc85c073070 0x7fc85c075530 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fc864005fd0 tx=0x7fc86400c480 comp rx=0 tx=0).stop 2024-09-16T15:44:20.283 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:20.276+0000 7fc88147d640 1 -- 172.21.15.123:0/2074664909 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc87c0b2b00 msgr2=0x7fc87c14fdf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:44:20.283 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:20.276+0000 7fc88147d640 1 --2- 172.21.15.123:0/2074664909 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc87c0b2b00 0x7fc87c14fdf0 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7fc86c00c970 tx=0x7fc86c00ce30 comp rx=0 tx=0).stop 2024-09-16T15:44:20.283 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:20.276+0000 7fc88147d640 1 -- 172.21.15.123:0/2074664909 shutdown_connections 2024-09-16T15:44:20.283 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:20.276+0000 7fc88147d640 1 --2- 172.21.15.123:0/2074664909 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fc85c073070 0x7fc85c075530 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:20.283 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:20.276+0000 7fc88147d640 1 --2- 172.21.15.123:0/2074664909 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fc87c161620 0x7fc87c0cd1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:20.283 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:20.276+0000 7fc88147d640 1 --2- 172.21.15.123:0/2074664909 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc87c0d2820 0x7fc87c0c9db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:20.283 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:20.276+0000 7fc88147d640 1 --2- 172.21.15.123:0/2074664909 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc87c0b2b00 0x7fc87c14fdf0 unknown :-1 s=CLOSED pgs=457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:20.284 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:20.276+0000 7fc88147d640 1 -- 172.21.15.123:0/2074664909 >> 172.21.15.123:0/2074664909 conn(0x7fc87c0d3ea0 msgr2=0x7fc87c0d4380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:44:20.284 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:20.276+0000 7fc88147d640 1 -- 172.21.15.123:0/2074664909 shutdown_connections 2024-09-16T15:44:20.284 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:20.276+0000 7fc88147d640 1 -- 172.21.15.123:0/2074664909 wait complete. 2024-09-16T15:44:20.297 INFO:teuthology.orchestra.run.smithi123.stdout:{"pg_ready":true,"pg_map":{"version":87,"stamp":"2024-09-16T15:44:19.274293+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":66,"acting":66,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":9,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":7,"kb":754974720,"kb_used":225260,"kb_used_data":4848,"kb_used_omap":12,"kb_used_meta":220211,"kb_avail":754749460,"statfs":{"total":773094113280,"available":772863447040,"internally_reserved":0,"allocated":4964352,"data_stored":2660670,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":225496625},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":13,"apply_latency_ms":13,"commit_latency_ns":13000000,"apply_latency_ns":13000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.810966"},"pg_stats":[{"pgid":"7.3f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T15:44:14.461845+0000","last_change":"2024-09-16T15:44:14.461845+0000","last_active":"2024-09-16T15:44:14.461845+0000","last_peered":"2024-09-16T15:44:14.461845+0000","last_clean":"2024-09-16T15:44:14.461845+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:44:14.461845+0000","last_undegraded":"2024-09-16T15:44:14.461845+0000","last_fullsized":"2024-09-16T15:44:14.461845+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_clean_scrub_stamp":"2024-09-16T15:44:14.461845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.3e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T15:44:14.461845+0000","last_change":"2024-09-16T15:44:14.461845+0000","last_active":"2024-09-16T15:44:14.461845+0000","last_peered":"2024-09-16T15:44:14.461845+0000","last_clean":"2024-09-16T15:44:14.461845+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:44:14.461845+0000","last_undegraded":"2024-09-16T15:44:14.461845+0000","last_fullsized":"2024-09-16T15:44:14.461845+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_clean_scrub_stamp":"2024-09-16T15:44:14.461845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.3d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T15:44:14.461845+0000","last_change":"2024-09-16T15:44:14.461845+0000","last_active":"2024-09-16T15:44:14.461845+0000","last_peered":"2024-09-16T15:44:14.461845+0000","last_clean":"2024-09-16T15:44:14.461845+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:44:14.461845+0000","last_undegraded":"2024-09-16T15:44:14.461845+0000","last_fullsized":"2024-09-16T15:44:14.461845+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_clean_scrub_stamp":"2024-09-16T15:44:14.461845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.3c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T15:44:14.461845+0000","last_change":"2024-09-16T15:44:14.461845+0000","last_active":"2024-09-16T15:44:14.461845+0000","last_peered":"2024-09-16T15:44:14.461845+0000","last_clean":"2024-09-16T15:44:14.461845+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:44:14.461845+0000","last_undegraded":"2024-09-16T15:44:14.461845+0000","last_fullsized":"2024-09-16T15:44:14.461845+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_clean_scrub_stamp":"2024-09-16T15:44:14.461845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.3b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T15:44:14.461845+0000","last_change":"2024-09-16T15:44:14.461845+0000","last_active":"2024-09-16T15:44:14.461845+0000","last_peered":"2024-09-16T15:44:14.461845+0000","last_clean":"2024-09-16T15:44:14.461845+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:44:14.461845+0000","last_undegraded":"2024-09-16T15:44:14.461845+0000","last_fullsized":"2024-09-16T15:44:14.461845+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_clean_scrub_stamp":"2024-09-16T15:44:14.461845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.3a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T15:44:14.461845+0000","last_change":"2024-09-16T15:44:14.461845+0000","last_active":"2024-09-16T15:44:14.461845+0000","last_peered":"2024-09-16T15:44:14.461845+0000","last_clean":"2024-09-16T15:44:14.461845+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:44:14.461845+0000","last_undegraded":"2024-09-16T15:44:14.461845+0000","last_fullsized":"2024-09-16T15:44:14.461845+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_clean_scrub_stamp":"2024-09-16T15:44:14.461845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.39","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T15:44:14.461845+0000","last_change":"2024-09-16T15:44:14.461845+0000","last_active":"2024-09-16T15:44:14.461845+0000","last_peered":"2024-09-16T15:44:14.461845+0000","last_clean":"2024-09-16T15:44:14.461845+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:44:14.461845+0000","last_undegraded":"2024-09-16T15:44:14.461845+0000","last_fullsized":"2024-09-16T15:44:14.461845+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_clean_scrub_stamp":"2024-09-16T15:44:14.461845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.38","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T15:44:14.461845+0000","last_change":"2024-09-16T15:44:14.461845+0000","last_active":"2024-09-16T15:44:14.461845+0000","last_peered":"2024-09-16T15:44:14.461845+0000","last_clean":"2024-09-16T15:44:14.461845+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:44:14.461845+0000","last_undegraded":"2024-09-16T15:44:14.461845+0000","last_fullsized":"2024-09-16T15:44:14.461845+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_clean_scrub_stamp":"2024-09-16T15:44:14.461845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.37","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T15:44:14.461845+0000","last_change":"2024-09-16T15:44:14.461845+0000","last_active":"2024-09-16T15:44:14.461845+0000","last_peered":"2024-09-16T15:44:14.461845+0000","last_clean":"2024-09-16T15:44:14.461845+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:44:14.461845+0000","last_undegraded":"2024-09-16T15:44:14.461845+0000","last_fullsized":"2024-09-16T15:44:14.461845+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_clean_scrub_stamp":"2024-09-16T15:44:14.461845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.36","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T15:44:14.461845+0000","last_change":"2024-09-16T15:44:14.461845+0000","last_active":"2024-09-16T15:44:14.461845+0000","last_peered":"2024-09-16T15:44:14.461845+0000","last_clean":"2024-09-16T15:44:14.461845+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:44:14.461845+0000","last_undegraded":"2024-09-16T15:44:14.461845+0000","last_fullsized":"2024-09-16T15:44:14.461845+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_clean_scrub_stamp":"2024-09-16T15:44:14.461845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.35","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T15:44:14.461845+0000","last_change":"2024-09-16T15:44:14.461845+0000","last_active":"2024-09-16T15:44:14.461845+0000","last_peered":"2024-09-16T15:44:14.461845+0000","last_clean":"2024-09-16T15:44:14.461845+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:44:14.461845+0000","last_undegraded":"2024-09-16T15:44:14.461845+0000","last_fullsized":"2024-09-16T15:44:14.461845+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_clean_scrub_stamp":"2024-09-16T15:44:14.461845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.34","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T15:44:14.461845+0000","last_change":"2024-09-16T15:44:14.461845+0000","last_active":"2024-09-16T15:44:14.461845+0000","last_peered":"2024-09-16T15:44:14.461845+0000","last_clean":"2024-09-16T15:44:14.461845+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:44:14.461845+0000","last_undegraded":"2024-09-16T15:44:14.461845+0000","last_fullsized":"2024-09-16T15:44:14.461845+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_clean_scrub_stamp":"2024-09-16T15:44:14.461845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.33","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T15:44:14.461845+0000","last_change":"2024-09-16T15:44:14.461845+0000","last_active":"2024-09-16T15:44:14.461845+0000","last_peered":"2024-09-16T15:44:14.461845+0000","last_clean":"2024-09-16T15:44:14.461845+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:44:14.461845+0000","last_undegraded":"2024-09-16T15:44:14.461845+0000","last_fullsized":"2024-09-16T15:44:14.461845+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_clean_scrub_stamp":"2024-09-16T15:44:14.461845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.32","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T15:44:14.461845+0000","last_change":"2024-09-16T15:44:14.461845+0000","last_active":"2024-09-16T15:44:14.461845+0000","last_peered":"2024-09-16T15:44:14.461845+0000","last_clean":"2024-09-16T15:44:14.461845+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:44:14.461845+0000","last_undegraded":"2024-09-16T15:44:14.461845+0000","last_fullsized":"2024-09-16T15:44:14.461845+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_clean_scrub_stamp":"2024-09-16T15:44:14.461845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.31","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T15:44:14.461845+0000","last_change":"2024-09-16T15:44:14.461845+0000","last_active":"2024-09-16T15:44:14.461845+0000","last_peered":"2024-09-16T15:44:14.461845+0000","last_clean":"2024-09-16T15:44:14.461845+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:44:14.461845+0000","last_undegraded":"2024-09-16T15:44:14.461845+0000","last_fullsized":"2024-09-16T15:44:14.461845+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_clean_scrub_stamp":"2024-09-16T15:44:14.461845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.30","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T15:44:14.461845+0000","last_change":"2024-09-16T15:44:14.461845+0000","last_active":"2024-09-16T15:44:14.461845+0000","last_peered":"2024-09-16T15:44:14.461845+0000","last_clean":"2024-09-16T15:44:14.461845+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:44:14.461845+0000","last_undegraded":"2024-09-16T15:44:14.461845+0000","last_fullsized":"2024-09-16T15:44:14.461845+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_clean_scrub_stamp":"2024-09-16T15:44:14.461845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.2f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T15:44:14.461845+0000","last_change":"2024-09-16T15:44:14.461845+0000","last_active":"2024-09-16T15:44:14.461845+0000","last_peered":"2024-09-16T15:44:14.461845+0000","last_clean":"2024-09-16T15:44:14.461845+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:44:14.461845+0000","last_undegraded":"2024-09-16T15:44:14.461845+0000","last_fullsized":"2024-09-16T15:44:14.461845+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_clean_scrub_stamp":"2024-09-16T15:44:14.461845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.2e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T15:44:14.461845+0000","last_change":"2024-09-16T15:44:14.461845+0000","last_active":"2024-09-16T15:44:14.461845+0000","last_peered":"2024-09-16T15:44:14.461845+0000","last_clean":"2024-09-16T15:44:14.461845+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:44:14.461845+0000","last_undegraded":"2024-09-16T15:44:14.461845+0000","last_fullsized":"2024-09-16T15:44:14.461845+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_clean_scrub_stamp":"2024-09-16T15:44:14.461845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.2d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T15:44:14.461845+0000","last_change":"2024-09-16T15:44:14.461845+0000","last_active":"2024-09-16T15:44:14.461845+0000","last_peered":"2024-09-16T15:44:14.461845+0000","last_clean":"2024-09-16T15:44:14.461845+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:44:14.461845+0000","last_undegraded":"2024-09-16T15:44:14.461845+0000","last_fullsized":"2024-09-16T15:44:14.461845+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_clean_scrub_stamp":"2024-09-16T15:44:14.461845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.2c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T15:44:14.461845+0000","last_change":"2024-09-16T15:44:14.461845+0000","last_active":"2024-09-16T15:44:14.461845+0000","last_peered":"2024-09-16T15:44:14.461845+0000","last_clean":"2024-09-16T15:44:14.461845+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:44:14.461845+0000","last_undegraded":"2024-09-16T15:44:14.461845+0000","last_fullsized":"2024-09-16T15:44:14.461845+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_clean_scrub_stamp":"2024-09-16T15:44:14.461845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.2b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T15:44:14.461845+0000","last_change":"2024-09-16T15:44:14.461845+0000","last_active":"2024-09-16T15:44:14.461845+0000","last_peered":"2024-09-16T15:44:14.461845+0000","last_clean":"2024-09-16T15:44:14.461845+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:44:14.461845+0000","last_undegraded":"2024-09-16T15:44:14.461845+0000","last_fullsized":"2024-09-16T15:44:14.461845+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_clean_scrub_stamp":"2024-09-16T15:44:14.461845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.2a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T15:44:14.461845+0000","last_change":"2024-09-16T15:44:14.461845+0000","last_active":"2024-09-16T15:44:14.461845+0000","last_peered":"2024-09-16T15:44:14.461845+0000","last_clean":"2024-09-16T15:44:14.461845+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:44:14.461845+0000","last_undegraded":"2024-09-16T15:44:14.461845+0000","last_fullsized":"2024-09-16T15:44:14.461845+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_clean_scrub_stamp":"2024-09-16T15:44:14.461845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.29","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T15:44:14.461845+0000","last_change":"2024-09-16T15:44:14.461845+0000","last_active":"2024-09-16T15:44:14.461845+0000","last_peered":"2024-09-16T15:44:14.461845+0000","last_clean":"2024-09-16T15:44:14.461845+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:44:14.461845+0000","last_undegraded":"2024-09-16T15:44:14.461845+0000","last_fullsized":"2024-09-16T15:44:14.461845+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_clean_scrub_stamp":"2024-09-16T15:44:14.461845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.28","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T15:44:14.461845+0000","last_change":"2024-09-16T15:44:14.461845+0000","last_active":"2024-09-16T15:44:14.461845+0000","last_peered":"2024-09-16T15:44:14.461845+0000","last_clean":"2024-09-16T15:44:14.461845+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:44:14.461845+0000","last_undegraded":"2024-09-16T15:44:14.461845+0000","last_fullsized":"2024-09-16T15:44:14.461845+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_clean_scrub_stamp":"2024-09-16T15:44:14.461845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.27","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T15:44:14.461845+0000","last_change":"2024-09-16T15:44:14.461845+0000","last_active":"2024-09-16T15:44:14.461845+0000","last_peered":"2024-09-16T15:44:14.461845+0000","last_clean":"2024-09-16T15:44:14.461845+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:44:14.461845+0000","last_undegraded":"2024-09-16T15:44:14.461845+0000","last_fullsized":"2024-09-16T15:44:14.461845+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_clean_scrub_stamp":"2024-09-16T15:44:14.461845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.26","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T15:44:14.461845+0000","last_change":"2024-09-16T15:44:14.461845+0000","last_active":"2024-09-16T15:44:14.461845+0000","last_peered":"2024-09-16T15:44:14.461845+0000","last_clean":"2024-09-16T15:44:14.461845+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:44:14.461845+0000","last_undegraded":"2024-09-16T15:44:14.461845+0000","last_fullsized":"2024-09-16T15:44:14.461845+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_clean_scrub_stamp":"2024-09-16T15:44:14.461845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.25","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T15:44:14.461845+0000","last_change":"2024-09-16T15:44:14.461845+0000","last_active":"2024-09-16T15:44:14.461845+0000","last_peered":"2024-09-16T15:44:14.461845+0000","last_clean":"2024-09-16T15:44:14.461845+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:44:14.461845+0000","last_undegraded":"2024-09-16T15:44:14.461845+0000","last_fullsized":"2024-09-16T15:44:14.461845+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_clean_scrub_stamp":"2024-09-16T15:44:14.461845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.24","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T15:44:14.461845+0000","last_change":"2024-09-16T15:44:14.461845+0000","last_active":"2024-09-16T15:44:14.461845+0000","last_peered":"2024-09-16T15:44:14.461845+0000","last_clean":"2024-09-16T15:44:14.461845+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:44:14.461845+0000","last_undegraded":"2024-09-16T15:44:14.461845+0000","last_fullsized":"2024-09-16T15:44:14.461845+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_clean_scrub_stamp":"2024-09-16T15:44:14.461845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.23","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T15:44:14.461845+0000","last_change":"2024-09-16T15:44:14.461845+0000","last_active":"2024-09-16T15:44:14.461845+0000","last_peered":"2024-09-16T15:44:14.461845+0000","last_clean":"2024-09-16T15:44:14.461845+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:44:14.461845+0000","last_undegraded":"2024-09-16T15:44:14.461845+0000","last_fullsized":"2024-09-16T15:44:14.461845+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_clean_scrub_stamp":"2024-09-16T15:44:14.461845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.22","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T15:44:14.461845+0000","last_change":"2024-09-16T15:44:14.461845+0000","last_active":"2024-09-16T15:44:14.461845+0000","last_peered":"2024-09-16T15:44:14.461845+0000","last_clean":"2024-09-16T15:44:14.461845+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:44:14.461845+0000","last_undegraded":"2024-09-16T15:44:14.461845+0000","last_fullsized":"2024-09-16T15:44:14.461845+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_clean_scrub_stamp":"2024-09-16T15:44:14.461845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.21","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T15:44:14.461845+0000","last_change":"2024-09-16T15:44:14.461845+0000","last_active":"2024-09-16T15:44:14.461845+0000","last_peered":"2024-09-16T15:44:14.461845+0000","last_clean":"2024-09-16T15:44:14.461845+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:44:14.461845+0000","last_undegraded":"2024-09-16T15:44:14.461845+0000","last_fullsized":"2024-09-16T15:44:14.461845+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_clean_scrub_stamp":"2024-09-16T15:44:14.461845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.20","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T15:44:14.461845+0000","last_change":"2024-09-16T15:44:14.461845+0000","last_active":"2024-09-16T15:44:14.461845+0000","last_peered":"2024-09-16T15:44:14.461845+0000","last_clean":"2024-09-16T15:44:14.461845+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:44:14.461845+0000","last_undegraded":"2024-09-16T15:44:14.461845+0000","last_fullsized":"2024-09-16T15:44:14.461845+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_clean_scrub_stamp":"2024-09-16T15:44:14.461845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T15:44:14.461845+0000","last_change":"2024-09-16T15:44:14.461845+0000","last_active":"2024-09-16T15:44:14.461845+0000","last_peered":"2024-09-16T15:44:14.461845+0000","last_clean":"2024-09-16T15:44:14.461845+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:44:14.461845+0000","last_undegraded":"2024-09-16T15:44:14.461845+0000","last_fullsized":"2024-09-16T15:44:14.461845+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_clean_scrub_stamp":"2024-09-16T15:44:14.461845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":1,"reported_epoch":41,"state":"creating+peering","last_fresh":"2024-09-16T15:44:13.470227+0000","last_change":"2024-09-16T15:44:13.470227+0000","last_active":"2024-09-16T15:44:13.453120+0000","last_peered":"2024-09-16T15:44:13.453120+0000","last_clean":"2024-09-16T15:44:13.453120+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:44:13.470227+0000","last_undegraded":"2024-09-16T15:44:13.470227+0000","last_fullsized":"2024-09-16T15:44:13.470227+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:13.453120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:13.453120+0000","last_clean_scrub_stamp":"2024-09-16T15:44:13.453120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T15:44:14.461845+0000","last_change":"2024-09-16T15:44:14.461845+0000","last_active":"2024-09-16T15:44:14.461845+0000","last_peered":"2024-09-16T15:44:14.461845+0000","last_clean":"2024-09-16T15:44:14.461845+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:44:14.461845+0000","last_undegraded":"2024-09-16T15:44:14.461845+0000","last_fullsized":"2024-09-16T15:44:14.461845+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_clean_scrub_stamp":"2024-09-16T15:44:14.461845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":1,"reported_epoch":41,"state":"creating+peering","last_fresh":"2024-09-16T15:44:13.469543+0000","last_change":"2024-09-16T15:44:13.469543+0000","last_active":"2024-09-16T15:44:13.453120+0000","last_peered":"2024-09-16T15:44:13.453120+0000","last_clean":"2024-09-16T15:44:13.453120+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:44:13.469543+0000","last_undegraded":"2024-09-16T15:44:13.469543+0000","last_fullsized":"2024-09-16T15:44:13.469543+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:13.453120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:13.453120+0000","last_clean_scrub_stamp":"2024-09-16T15:44:13.453120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T15:44:14.461845+0000","last_change":"2024-09-16T15:44:14.461845+0000","last_active":"2024-09-16T15:44:14.461845+0000","last_peered":"2024-09-16T15:44:14.461845+0000","last_clean":"2024-09-16T15:44:14.461845+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:44:14.461845+0000","last_undegraded":"2024-09-16T15:44:14.461845+0000","last_fullsized":"2024-09-16T15:44:14.461845+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_clean_scrub_stamp":"2024-09-16T15:44:14.461845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":1,"reported_epoch":41,"state":"creating+peering","last_fresh":"2024-09-16T15:44:13.472025+0000","last_change":"2024-09-16T15:44:13.472025+0000","last_active":"2024-09-16T15:44:13.453120+0000","last_peered":"2024-09-16T15:44:13.453120+0000","last_clean":"2024-09-16T15:44:13.453120+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:44:13.472025+0000","last_undegraded":"2024-09-16T15:44:13.472025+0000","last_fullsized":"2024-09-16T15:44:13.472025+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:13.453120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:13.453120+0000","last_clean_scrub_stamp":"2024-09-16T15:44:13.453120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T15:44:14.461845+0000","last_change":"2024-09-16T15:44:14.461845+0000","last_active":"2024-09-16T15:44:14.461845+0000","last_peered":"2024-09-16T15:44:14.461845+0000","last_clean":"2024-09-16T15:44:14.461845+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:44:14.461845+0000","last_undegraded":"2024-09-16T15:44:14.461845+0000","last_fullsized":"2024-09-16T15:44:14.461845+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_clean_scrub_stamp":"2024-09-16T15:44:14.461845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.1c","version":"0'0","reported_seq":1,"reported_epoch":41,"state":"creating+peering","last_fresh":"2024-09-16T15:44:13.475916+0000","last_change":"2024-09-16T15:44:13.475916+0000","last_active":"2024-09-16T15:44:13.453120+0000","last_peered":"2024-09-16T15:44:13.453120+0000","last_clean":"2024-09-16T15:44:13.453120+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:44:13.475916+0000","last_undegraded":"2024-09-16T15:44:13.475916+0000","last_fullsized":"2024-09-16T15:44:13.475916+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:13.453120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:13.453120+0000","last_clean_scrub_stamp":"2024-09-16T15:44:13.453120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T15:44:14.461845+0000","last_change":"2024-09-16T15:44:14.461845+0000","last_active":"2024-09-16T15:44:14.461845+0000","last_peered":"2024-09-16T15:44:14.461845+0000","last_clean":"2024-09-16T15:44:14.461845+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:44:14.461845+0000","last_undegraded":"2024-09-16T15:44:14.461845+0000","last_fullsized":"2024-09-16T15:44:14.461845+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_clean_scrub_stamp":"2024-09-16T15:44:14.461845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":1,"reported_epoch":41,"state":"creating+peering","last_fresh":"2024-09-16T15:44:13.470521+0000","last_change":"2024-09-16T15:44:13.470521+0000","last_active":"2024-09-16T15:44:13.453120+0000","last_peered":"2024-09-16T15:44:13.453120+0000","last_clean":"2024-09-16T15:44:13.453120+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:44:13.470521+0000","last_undegraded":"2024-09-16T15:44:13.470521+0000","last_fullsized":"2024-09-16T15:44:13.470521+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:13.453120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:13.453120+0000","last_clean_scrub_stamp":"2024-09-16T15:44:13.453120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T15:44:14.461845+0000","last_change":"2024-09-16T15:44:14.461845+0000","last_active":"2024-09-16T15:44:14.461845+0000","last_peered":"2024-09-16T15:44:14.461845+0000","last_clean":"2024-09-16T15:44:14.461845+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:44:14.461845+0000","last_undegraded":"2024-09-16T15:44:14.461845+0000","last_fullsized":"2024-09-16T15:44:14.461845+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_clean_scrub_stamp":"2024-09-16T15:44:14.461845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":1,"reported_epoch":41,"state":"creating+peering","last_fresh":"2024-09-16T15:44:13.468698+0000","last_change":"2024-09-16T15:44:13.468698+0000","last_active":"2024-09-16T15:44:13.453120+0000","last_peered":"2024-09-16T15:44:13.453120+0000","last_clean":"2024-09-16T15:44:13.453120+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:44:13.468698+0000","last_undegraded":"2024-09-16T15:44:13.468698+0000","last_fullsized":"2024-09-16T15:44:13.468698+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:13.453120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:13.453120+0000","last_clean_scrub_stamp":"2024-09-16T15:44:13.453120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T15:44:14.461845+0000","last_change":"2024-09-16T15:44:14.461845+0000","last_active":"2024-09-16T15:44:14.461845+0000","last_peered":"2024-09-16T15:44:14.461845+0000","last_clean":"2024-09-16T15:44:14.461845+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:44:14.461845+0000","last_undegraded":"2024-09-16T15:44:14.461845+0000","last_fullsized":"2024-09-16T15:44:14.461845+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_clean_scrub_stamp":"2024-09-16T15:44:14.461845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":1,"reported_epoch":41,"state":"creating+peering","last_fresh":"2024-09-16T15:44:13.478336+0000","last_change":"2024-09-16T15:44:13.478336+0000","last_active":"2024-09-16T15:44:13.453120+0000","last_peered":"2024-09-16T15:44:13.453120+0000","last_clean":"2024-09-16T15:44:13.453120+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:44:13.478336+0000","last_undegraded":"2024-09-16T15:44:13.478336+0000","last_fullsized":"2024-09-16T15:44:13.478336+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:13.453120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:13.453120+0000","last_clean_scrub_stamp":"2024-09-16T15:44:13.453120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T15:44:14.461845+0000","last_change":"2024-09-16T15:44:14.461845+0000","last_active":"2024-09-16T15:44:14.461845+0000","last_peered":"2024-09-16T15:44:14.461845+0000","last_clean":"2024-09-16T15:44:14.461845+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:44:14.461845+0000","last_undegraded":"2024-09-16T15:44:14.461845+0000","last_fullsized":"2024-09-16T15:44:14.461845+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_clean_scrub_stamp":"2024-09-16T15:44:14.461845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":1,"reported_epoch":41,"state":"creating+peering","last_fresh":"2024-09-16T15:44:13.475717+0000","last_change":"2024-09-16T15:44:13.475717+0000","last_active":"2024-09-16T15:44:13.453120+0000","last_peered":"2024-09-16T15:44:13.453120+0000","last_clean":"2024-09-16T15:44:13.453120+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:44:13.475717+0000","last_undegraded":"2024-09-16T15:44:13.475717+0000","last_fullsized":"2024-09-16T15:44:13.475717+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:13.453120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:13.453120+0000","last_clean_scrub_stamp":"2024-09-16T15:44:13.453120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T15:44:14.461845+0000","last_change":"2024-09-16T15:44:14.461845+0000","last_active":"2024-09-16T15:44:14.461845+0000","last_peered":"2024-09-16T15:44:14.461845+0000","last_clean":"2024-09-16T15:44:14.461845+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:44:14.461845+0000","last_undegraded":"2024-09-16T15:44:14.461845+0000","last_fullsized":"2024-09-16T15:44:14.461845+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_clean_scrub_stamp":"2024-09-16T15:44:14.461845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":1,"reported_epoch":41,"state":"creating+peering","last_fresh":"2024-09-16T15:44:13.468269+0000","last_change":"2024-09-16T15:44:13.468269+0000","last_active":"2024-09-16T15:44:13.453120+0000","last_peered":"2024-09-16T15:44:13.453120+0000","last_clean":"2024-09-16T15:44:13.453120+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:44:13.468269+0000","last_undegraded":"2024-09-16T15:44:13.468269+0000","last_fullsized":"2024-09-16T15:44:13.468269+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:13.453120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:13.453120+0000","last_clean_scrub_stamp":"2024-09-16T15:44:13.453120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T15:44:14.461845+0000","last_change":"2024-09-16T15:44:14.461845+0000","last_active":"2024-09-16T15:44:14.461845+0000","last_peered":"2024-09-16T15:44:14.461845+0000","last_clean":"2024-09-16T15:44:14.461845+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:44:14.461845+0000","last_undegraded":"2024-09-16T15:44:14.461845+0000","last_fullsized":"2024-09-16T15:44:14.461845+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_clean_scrub_stamp":"2024-09-16T15:44:14.461845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":1,"reported_epoch":41,"state":"creating+peering","last_fresh":"2024-09-16T15:44:13.473942+0000","last_change":"2024-09-16T15:44:13.473942+0000","last_active":"2024-09-16T15:44:13.453120+0000","last_peered":"2024-09-16T15:44:13.453120+0000","last_clean":"2024-09-16T15:44:13.453120+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:44:13.473942+0000","last_undegraded":"2024-09-16T15:44:13.473942+0000","last_fullsized":"2024-09-16T15:44:13.473942+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:13.453120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:13.453120+0000","last_clean_scrub_stamp":"2024-09-16T15:44:13.453120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T15:44:14.461845+0000","last_change":"2024-09-16T15:44:14.461845+0000","last_active":"2024-09-16T15:44:14.461845+0000","last_peered":"2024-09-16T15:44:14.461845+0000","last_clean":"2024-09-16T15:44:14.461845+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:44:14.461845+0000","last_undegraded":"2024-09-16T15:44:14.461845+0000","last_fullsized":"2024-09-16T15:44:14.461845+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_clean_scrub_stamp":"2024-09-16T15:44:14.461845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":1,"reported_epoch":41,"state":"creating+peering","last_fresh":"2024-09-16T15:44:13.476299+0000","last_change":"2024-09-16T15:44:13.476299+0000","last_active":"2024-09-16T15:44:13.453120+0000","last_peered":"2024-09-16T15:44:13.453120+0000","last_clean":"2024-09-16T15:44:13.453120+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:44:13.476299+0000","last_undegraded":"2024-09-16T15:44:13.476299+0000","last_fullsized":"2024-09-16T15:44:13.476299+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:13.453120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:13.453120+0000","last_clean_scrub_stamp":"2024-09-16T15:44:13.453120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T15:44:14.461845+0000","last_change":"2024-09-16T15:44:14.461845+0000","last_active":"2024-09-16T15:44:14.461845+0000","last_peered":"2024-09-16T15:44:14.461845+0000","last_clean":"2024-09-16T15:44:14.461845+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:44:14.461845+0000","last_undegraded":"2024-09-16T15:44:14.461845+0000","last_fullsized":"2024-09-16T15:44:14.461845+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_clean_scrub_stamp":"2024-09-16T15:44:14.461845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":1,"reported_epoch":41,"state":"creating+peering","last_fresh":"2024-09-16T15:44:13.470832+0000","last_change":"2024-09-16T15:44:13.470832+0000","last_active":"2024-09-16T15:44:13.453120+0000","last_peered":"2024-09-16T15:44:13.453120+0000","last_clean":"2024-09-16T15:44:13.453120+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:44:13.470832+0000","last_undegraded":"2024-09-16T15:44:13.470832+0000","last_fullsized":"2024-09-16T15:44:13.470832+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:13.453120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:13.453120+0000","last_clean_scrub_stamp":"2024-09-16T15:44:13.453120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T15:44:14.461845+0000","last_change":"2024-09-16T15:44:14.461845+0000","last_active":"2024-09-16T15:44:14.461845+0000","last_peered":"2024-09-16T15:44:14.461845+0000","last_clean":"2024-09-16T15:44:14.461845+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:44:14.461845+0000","last_undegraded":"2024-09-16T15:44:14.461845+0000","last_fullsized":"2024-09-16T15:44:14.461845+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_clean_scrub_stamp":"2024-09-16T15:44:14.461845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":1,"reported_epoch":41,"state":"creating+peering","last_fresh":"2024-09-16T15:44:13.469026+0000","last_change":"2024-09-16T15:44:13.469026+0000","last_active":"2024-09-16T15:44:13.453120+0000","last_peered":"2024-09-16T15:44:13.453120+0000","last_clean":"2024-09-16T15:44:13.453120+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:44:13.469026+0000","last_undegraded":"2024-09-16T15:44:13.469026+0000","last_fullsized":"2024-09-16T15:44:13.469026+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:13.453120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:13.453120+0000","last_clean_scrub_stamp":"2024-09-16T15:44:13.453120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T15:44:14.461845+0000","last_change":"2024-09-16T15:44:14.461845+0000","last_active":"2024-09-16T15:44:14.461845+0000","last_peered":"2024-09-16T15:44:14.461845+0000","last_clean":"2024-09-16T15:44:14.461845+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:44:14.461845+0000","last_undegraded":"2024-09-16T15:44:14.461845+0000","last_fullsized":"2024-09-16T15:44:14.461845+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_clean_scrub_stamp":"2024-09-16T15:44:14.461845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":1,"reported_epoch":41,"state":"creating+peering","last_fresh":"2024-09-16T15:44:13.474765+0000","last_change":"2024-09-16T15:44:13.474765+0000","last_active":"2024-09-16T15:44:13.453120+0000","last_peered":"2024-09-16T15:44:13.453120+0000","last_clean":"2024-09-16T15:44:13.453120+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:44:13.474765+0000","last_undegraded":"2024-09-16T15:44:13.474765+0000","last_fullsized":"2024-09-16T15:44:13.474765+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:13.453120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:13.453120+0000","last_clean_scrub_stamp":"2024-09-16T15:44:13.453120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T15:44:14.461845+0000","last_change":"2024-09-16T15:44:14.461845+0000","last_active":"2024-09-16T15:44:14.461845+0000","last_peered":"2024-09-16T15:44:14.461845+0000","last_clean":"2024-09-16T15:44:14.461845+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:44:14.461845+0000","last_undegraded":"2024-09-16T15:44:14.461845+0000","last_fullsized":"2024-09-16T15:44:14.461845+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_clean_scrub_stamp":"2024-09-16T15:44:14.461845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":1,"reported_epoch":41,"state":"creating+peering","last_fresh":"2024-09-16T15:44:13.468318+0000","last_change":"2024-09-16T15:44:13.468318+0000","last_active":"2024-09-16T15:44:13.453120+0000","last_peered":"2024-09-16T15:44:13.453120+0000","last_clean":"2024-09-16T15:44:13.453120+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:44:13.468318+0000","last_undegraded":"2024-09-16T15:44:13.468318+0000","last_fullsized":"2024-09-16T15:44:13.468318+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:13.453120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:13.453120+0000","last_clean_scrub_stamp":"2024-09-16T15:44:13.453120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T15:44:14.461845+0000","last_change":"2024-09-16T15:44:14.461845+0000","last_active":"2024-09-16T15:44:14.461845+0000","last_peered":"2024-09-16T15:44:14.461845+0000","last_clean":"2024-09-16T15:44:14.461845+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:44:14.461845+0000","last_undegraded":"2024-09-16T15:44:14.461845+0000","last_fullsized":"2024-09-16T15:44:14.461845+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_clean_scrub_stamp":"2024-09-16T15:44:14.461845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":1,"reported_epoch":41,"state":"creating+peering","last_fresh":"2024-09-16T15:44:13.473996+0000","last_change":"2024-09-16T15:44:13.473996+0000","last_active":"2024-09-16T15:44:13.453120+0000","last_peered":"2024-09-16T15:44:13.453120+0000","last_clean":"2024-09-16T15:44:13.453120+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:44:13.473996+0000","last_undegraded":"2024-09-16T15:44:13.473996+0000","last_fullsized":"2024-09-16T15:44:13.473996+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:13.453120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:13.453120+0000","last_clean_scrub_stamp":"2024-09-16T15:44:13.453120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T15:44:14.461845+0000","last_change":"2024-09-16T15:44:14.461845+0000","last_active":"2024-09-16T15:44:14.461845+0000","last_peered":"2024-09-16T15:44:14.461845+0000","last_clean":"2024-09-16T15:44:14.461845+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:44:14.461845+0000","last_undegraded":"2024-09-16T15:44:14.461845+0000","last_fullsized":"2024-09-16T15:44:14.461845+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_clean_scrub_stamp":"2024-09-16T15:44:14.461845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":1,"reported_epoch":41,"state":"creating+peering","last_fresh":"2024-09-16T15:44:13.471259+0000","last_change":"2024-09-16T15:44:13.471259+0000","last_active":"2024-09-16T15:44:13.453120+0000","last_peered":"2024-09-16T15:44:13.453120+0000","last_clean":"2024-09-16T15:44:13.453120+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:44:13.471259+0000","last_undegraded":"2024-09-16T15:44:13.471259+0000","last_fullsized":"2024-09-16T15:44:13.471259+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:13.453120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:13.453120+0000","last_clean_scrub_stamp":"2024-09-16T15:44:13.453120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T15:44:14.461845+0000","last_change":"2024-09-16T15:44:14.461845+0000","last_active":"2024-09-16T15:44:14.461845+0000","last_peered":"2024-09-16T15:44:14.461845+0000","last_clean":"2024-09-16T15:44:14.461845+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:44:14.461845+0000","last_undegraded":"2024-09-16T15:44:14.461845+0000","last_fullsized":"2024-09-16T15:44:14.461845+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_clean_scrub_stamp":"2024-09-16T15:44:14.461845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":1,"reported_epoch":41,"state":"creating+peering","last_fresh":"2024-09-16T15:44:13.468343+0000","last_change":"2024-09-16T15:44:13.468343+0000","last_active":"2024-09-16T15:44:13.453120+0000","last_peered":"2024-09-16T15:44:13.453120+0000","last_clean":"2024-09-16T15:44:13.453120+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:44:13.468343+0000","last_undegraded":"2024-09-16T15:44:13.468343+0000","last_fullsized":"2024-09-16T15:44:13.468343+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:13.453120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:13.453120+0000","last_clean_scrub_stamp":"2024-09-16T15:44:13.453120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T15:44:14.461845+0000","last_change":"2024-09-16T15:44:14.461845+0000","last_active":"2024-09-16T15:44:14.461845+0000","last_peered":"2024-09-16T15:44:14.461845+0000","last_clean":"2024-09-16T15:44:14.461845+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:44:14.461845+0000","last_undegraded":"2024-09-16T15:44:14.461845+0000","last_fullsized":"2024-09-16T15:44:14.461845+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_clean_scrub_stamp":"2024-09-16T15:44:14.461845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":1,"reported_epoch":41,"state":"creating+peering","last_fresh":"2024-09-16T15:44:13.478680+0000","last_change":"2024-09-16T15:44:13.478680+0000","last_active":"2024-09-16T15:44:13.453120+0000","last_peered":"2024-09-16T15:44:13.453120+0000","last_clean":"2024-09-16T15:44:13.453120+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:44:13.478680+0000","last_undegraded":"2024-09-16T15:44:13.478680+0000","last_fullsized":"2024-09-16T15:44:13.478680+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:13.453120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:13.453120+0000","last_clean_scrub_stamp":"2024-09-16T15:44:13.453120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T15:44:14.461845+0000","last_change":"2024-09-16T15:44:14.461845+0000","last_active":"2024-09-16T15:44:14.461845+0000","last_peered":"2024-09-16T15:44:14.461845+0000","last_clean":"2024-09-16T15:44:14.461845+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:44:14.461845+0000","last_undegraded":"2024-09-16T15:44:14.461845+0000","last_fullsized":"2024-09-16T15:44:14.461845+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_clean_scrub_stamp":"2024-09-16T15:44:14.461845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":1,"reported_epoch":41,"state":"creating+peering","last_fresh":"2024-09-16T15:44:13.468140+0000","last_change":"2024-09-16T15:44:13.468140+0000","last_active":"2024-09-16T15:44:13.453120+0000","last_peered":"2024-09-16T15:44:13.453120+0000","last_clean":"2024-09-16T15:44:13.453120+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:44:13.468140+0000","last_undegraded":"2024-09-16T15:44:13.468140+0000","last_fullsized":"2024-09-16T15:44:13.468140+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:13.453120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:13.453120+0000","last_clean_scrub_stamp":"2024-09-16T15:44:13.453120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T15:44:14.461845+0000","last_change":"2024-09-16T15:44:14.461845+0000","last_active":"2024-09-16T15:44:14.461845+0000","last_peered":"2024-09-16T15:44:14.461845+0000","last_clean":"2024-09-16T15:44:14.461845+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:44:14.461845+0000","last_undegraded":"2024-09-16T15:44:14.461845+0000","last_fullsized":"2024-09-16T15:44:14.461845+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_clean_scrub_stamp":"2024-09-16T15:44:14.461845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":1,"reported_epoch":41,"state":"creating+peering","last_fresh":"2024-09-16T15:44:13.468299+0000","last_change":"2024-09-16T15:44:13.468299+0000","last_active":"2024-09-16T15:44:13.453120+0000","last_peered":"2024-09-16T15:44:13.453120+0000","last_clean":"2024-09-16T15:44:13.453120+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:44:13.468299+0000","last_undegraded":"2024-09-16T15:44:13.468299+0000","last_fullsized":"2024-09-16T15:44:13.468299+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:13.453120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:13.453120+0000","last_clean_scrub_stamp":"2024-09-16T15:44:13.453120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T15:44:14.461845+0000","last_change":"2024-09-16T15:44:14.461845+0000","last_active":"2024-09-16T15:44:14.461845+0000","last_peered":"2024-09-16T15:44:14.461845+0000","last_clean":"2024-09-16T15:44:14.461845+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:44:14.461845+0000","last_undegraded":"2024-09-16T15:44:14.461845+0000","last_fullsized":"2024-09-16T15:44:14.461845+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_clean_scrub_stamp":"2024-09-16T15:44:14.461845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":1,"reported_epoch":41,"state":"creating+peering","last_fresh":"2024-09-16T15:44:13.473999+0000","last_change":"2024-09-16T15:44:13.473999+0000","last_active":"2024-09-16T15:44:13.453120+0000","last_peered":"2024-09-16T15:44:13.453120+0000","last_clean":"2024-09-16T15:44:13.453120+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:44:13.473999+0000","last_undegraded":"2024-09-16T15:44:13.473999+0000","last_fullsized":"2024-09-16T15:44:13.473999+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:13.453120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:13.453120+0000","last_clean_scrub_stamp":"2024-09-16T15:44:13.453120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T15:44:14.461845+0000","last_change":"2024-09-16T15:44:14.461845+0000","last_active":"2024-09-16T15:44:14.461845+0000","last_peered":"2024-09-16T15:44:14.461845+0000","last_clean":"2024-09-16T15:44:14.461845+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:44:14.461845+0000","last_undegraded":"2024-09-16T15:44:14.461845+0000","last_fullsized":"2024-09-16T15:44:14.461845+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_clean_scrub_stamp":"2024-09-16T15:44:14.461845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":1,"reported_epoch":41,"state":"creating+peering","last_fresh":"2024-09-16T15:44:13.473367+0000","last_change":"2024-09-16T15:44:13.473367+0000","last_active":"2024-09-16T15:44:13.453120+0000","last_peered":"2024-09-16T15:44:13.453120+0000","last_clean":"2024-09-16T15:44:13.453120+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:44:13.473367+0000","last_undegraded":"2024-09-16T15:44:13.473367+0000","last_fullsized":"2024-09-16T15:44:13.473367+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:13.453120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:13.453120+0000","last_clean_scrub_stamp":"2024-09-16T15:44:13.453120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T15:44:14.461845+0000","last_change":"2024-09-16T15:44:14.461845+0000","last_active":"2024-09-16T15:44:14.461845+0000","last_peered":"2024-09-16T15:44:14.461845+0000","last_clean":"2024-09-16T15:44:14.461845+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:44:14.461845+0000","last_undegraded":"2024-09-16T15:44:14.461845+0000","last_fullsized":"2024-09-16T15:44:14.461845+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_clean_scrub_stamp":"2024-09-16T15:44:14.461845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":1,"reported_epoch":41,"state":"creating+peering","last_fresh":"2024-09-16T15:44:13.474597+0000","last_change":"2024-09-16T15:44:13.474597+0000","last_active":"2024-09-16T15:44:13.453120+0000","last_peered":"2024-09-16T15:44:13.453120+0000","last_clean":"2024-09-16T15:44:13.453120+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:44:13.474597+0000","last_undegraded":"2024-09-16T15:44:13.474597+0000","last_fullsized":"2024-09-16T15:44:13.474597+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:13.453120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:13.453120+0000","last_clean_scrub_stamp":"2024-09-16T15:44:13.453120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T15:44:14.461845+0000","last_change":"2024-09-16T15:44:14.461845+0000","last_active":"2024-09-16T15:44:14.461845+0000","last_peered":"2024-09-16T15:44:14.461845+0000","last_clean":"2024-09-16T15:44:14.461845+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:44:14.461845+0000","last_undegraded":"2024-09-16T15:44:14.461845+0000","last_fullsized":"2024-09-16T15:44:14.461845+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_clean_scrub_stamp":"2024-09-16T15:44:14.461845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":1,"reported_epoch":41,"state":"creating+peering","last_fresh":"2024-09-16T15:44:13.468102+0000","last_change":"2024-09-16T15:44:13.468102+0000","last_active":"2024-09-16T15:44:13.453120+0000","last_peered":"2024-09-16T15:44:13.453120+0000","last_clean":"2024-09-16T15:44:13.453120+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:44:13.468102+0000","last_undegraded":"2024-09-16T15:44:13.468102+0000","last_fullsized":"2024-09-16T15:44:13.468102+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:13.453120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:13.453120+0000","last_clean_scrub_stamp":"2024-09-16T15:44:13.453120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T15:44:14.461845+0000","last_change":"2024-09-16T15:44:14.461845+0000","last_active":"2024-09-16T15:44:14.461845+0000","last_peered":"2024-09-16T15:44:14.461845+0000","last_clean":"2024-09-16T15:44:14.461845+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:44:14.461845+0000","last_undegraded":"2024-09-16T15:44:14.461845+0000","last_fullsized":"2024-09-16T15:44:14.461845+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_clean_scrub_stamp":"2024-09-16T15:44:14.461845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":1,"reported_epoch":41,"state":"creating+peering","last_fresh":"2024-09-16T15:44:13.474821+0000","last_change":"2024-09-16T15:44:13.474821+0000","last_active":"2024-09-16T15:44:13.453120+0000","last_peered":"2024-09-16T15:44:13.453120+0000","last_clean":"2024-09-16T15:44:13.453120+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:44:13.474821+0000","last_undegraded":"2024-09-16T15:44:13.474821+0000","last_fullsized":"2024-09-16T15:44:13.474821+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:13.453120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:13.453120+0000","last_clean_scrub_stamp":"2024-09-16T15:44:13.453120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T15:44:14.461845+0000","last_change":"2024-09-16T15:44:14.461845+0000","last_active":"2024-09-16T15:44:14.461845+0000","last_peered":"2024-09-16T15:44:14.461845+0000","last_clean":"2024-09-16T15:44:14.461845+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:44:14.461845+0000","last_undegraded":"2024-09-16T15:44:14.461845+0000","last_fullsized":"2024-09-16T15:44:14.461845+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_clean_scrub_stamp":"2024-09-16T15:44:14.461845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":1,"reported_epoch":41,"state":"creating+peering","last_fresh":"2024-09-16T15:44:13.472012+0000","last_change":"2024-09-16T15:44:13.472012+0000","last_active":"2024-09-16T15:44:13.453120+0000","last_peered":"2024-09-16T15:44:13.453120+0000","last_clean":"2024-09-16T15:44:13.453120+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:44:13.472012+0000","last_undegraded":"2024-09-16T15:44:13.472012+0000","last_fullsized":"2024-09-16T15:44:13.472012+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:13.453120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:13.453120+0000","last_clean_scrub_stamp":"2024-09-16T15:44:13.453120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T15:44:14.461845+0000","last_change":"2024-09-16T15:44:14.461845+0000","last_active":"2024-09-16T15:44:14.461845+0000","last_peered":"2024-09-16T15:44:14.461845+0000","last_clean":"2024-09-16T15:44:14.461845+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:44:14.461845+0000","last_undegraded":"2024-09-16T15:44:14.461845+0000","last_fullsized":"2024-09-16T15:44:14.461845+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_clean_scrub_stamp":"2024-09-16T15:44:14.461845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":1,"reported_epoch":41,"state":"creating+peering","last_fresh":"2024-09-16T15:44:13.474616+0000","last_change":"2024-09-16T15:44:13.474616+0000","last_active":"2024-09-16T15:44:13.453120+0000","last_peered":"2024-09-16T15:44:13.453120+0000","last_clean":"2024-09-16T15:44:13.453120+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:44:13.474616+0000","last_undegraded":"2024-09-16T15:44:13.474616+0000","last_fullsized":"2024-09-16T15:44:13.474616+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:13.453120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:13.453120+0000","last_clean_scrub_stamp":"2024-09-16T15:44:13.453120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T15:44:14.461845+0000","last_change":"2024-09-16T15:44:14.461845+0000","last_active":"2024-09-16T15:44:14.461845+0000","last_peered":"2024-09-16T15:44:14.461845+0000","last_clean":"2024-09-16T15:44:14.461845+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:44:14.461845+0000","last_undegraded":"2024-09-16T15:44:14.461845+0000","last_fullsized":"2024-09-16T15:44:14.461845+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_clean_scrub_stamp":"2024-09-16T15:44:14.461845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":1,"reported_epoch":41,"state":"creating+peering","last_fresh":"2024-09-16T15:44:13.467248+0000","last_change":"2024-09-16T15:44:13.467248+0000","last_active":"2024-09-16T15:44:13.453120+0000","last_peered":"2024-09-16T15:44:13.453120+0000","last_clean":"2024-09-16T15:44:13.453120+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:44:13.467248+0000","last_undegraded":"2024-09-16T15:44:13.467248+0000","last_fullsized":"2024-09-16T15:44:13.467248+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:13.453120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:13.453120+0000","last_clean_scrub_stamp":"2024-09-16T15:44:13.453120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T15:44:14.461845+0000","last_change":"2024-09-16T15:44:14.461845+0000","last_active":"2024-09-16T15:44:14.461845+0000","last_peered":"2024-09-16T15:44:14.461845+0000","last_clean":"2024-09-16T15:44:14.461845+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:44:14.461845+0000","last_undegraded":"2024-09-16T15:44:14.461845+0000","last_fullsized":"2024-09-16T15:44:14.461845+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_clean_scrub_stamp":"2024-09-16T15:44:14.461845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":1,"reported_epoch":41,"state":"creating+peering","last_fresh":"2024-09-16T15:44:13.472810+0000","last_change":"2024-09-16T15:44:13.472810+0000","last_active":"2024-09-16T15:44:13.453120+0000","last_peered":"2024-09-16T15:44:13.453120+0000","last_clean":"2024-09-16T15:44:13.453120+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:44:13.472810+0000","last_undegraded":"2024-09-16T15:44:13.472810+0000","last_fullsized":"2024-09-16T15:44:13.472810+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:13.453120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:13.453120+0000","last_clean_scrub_stamp":"2024-09-16T15:44:13.453120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T15:44:14.461845+0000","last_change":"2024-09-16T15:44:14.461845+0000","last_active":"2024-09-16T15:44:14.461845+0000","last_peered":"2024-09-16T15:44:14.461845+0000","last_clean":"2024-09-16T15:44:14.461845+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:44:14.461845+0000","last_undegraded":"2024-09-16T15:44:14.461845+0000","last_fullsized":"2024-09-16T15:44:14.461845+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_clean_scrub_stamp":"2024-09-16T15:44:14.461845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":1,"reported_epoch":41,"state":"creating+peering","last_fresh":"2024-09-16T15:44:13.473352+0000","last_change":"2024-09-16T15:44:13.473352+0000","last_active":"2024-09-16T15:44:13.453120+0000","last_peered":"2024-09-16T15:44:13.453120+0000","last_clean":"2024-09-16T15:44:13.453120+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:44:13.473352+0000","last_undegraded":"2024-09-16T15:44:13.473352+0000","last_fullsized":"2024-09-16T15:44:13.473352+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:13.453120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:13.453120+0000","last_clean_scrub_stamp":"2024-09-16T15:44:13.453120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T15:44:14.461845+0000","last_change":"2024-09-16T15:44:14.461845+0000","last_active":"2024-09-16T15:44:14.461845+0000","last_peered":"2024-09-16T15:44:14.461845+0000","last_clean":"2024-09-16T15:44:14.461845+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:44:14.461845+0000","last_undegraded":"2024-09-16T15:44:14.461845+0000","last_fullsized":"2024-09-16T15:44:14.461845+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_clean_scrub_stamp":"2024-09-16T15:44:14.461845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":1,"reported_epoch":41,"state":"creating+peering","last_fresh":"2024-09-16T15:44:13.476429+0000","last_change":"2024-09-16T15:44:13.476429+0000","last_active":"2024-09-16T15:44:13.453120+0000","last_peered":"2024-09-16T15:44:13.453120+0000","last_clean":"2024-09-16T15:44:13.453120+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:44:13.476429+0000","last_undegraded":"2024-09-16T15:44:13.476429+0000","last_fullsized":"2024-09-16T15:44:13.476429+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:13.453120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:13.453120+0000","last_clean_scrub_stamp":"2024-09-16T15:44:13.453120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"16'72","reported_seq":147,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-16T15:44:13.477262+0000","last_change":"2024-09-16T15:42:40.881820+0000","last_active":"2024-09-16T15:44:13.477262+0000","last_peered":"2024-09-16T15:44:13.477262+0000","last_clean":"2024-09-16T15:44:13.477262+0000","last_became_active":"2024-09-16T15:42:40.879965+0000","last_became_peered":"2024-09-16T15:42:40.879965+0000","last_unstale":"2024-09-16T15:44:13.477262+0000","last_undegraded":"2024-09-16T15:44:13.477262+0000","last_fullsized":"2024-09-16T15:44:13.477262+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:42:39.728762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:42:39.728762+0000","last_clean_scrub_stamp":"2024-09-16T15:42:39.728762+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:22:18.768932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542167,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28072,"kb_used_data":1044,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343768,"statfs":{"total":96636764160,"available":96608018432,"internally_reserved":0,"allocated":1069056,"data_stored":776814,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542166,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27492,"kb_used_data":452,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344348,"statfs":{"total":96636764160,"available":96608612352,"internally_reserved":0,"allocated":462848,"data_stored":180629,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":5,"up_from":14,"seq":60129542167,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27492,"kb_used_data":464,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344348,"statfs":{"total":96636764160,"available":96608612352,"internally_reserved":0,"allocated":475136,"data_stored":186446,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542165,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27492,"kb_used_data":464,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344348,"statfs":{"total":96636764160,"available":96608612352,"internally_reserved":0,"allocated":475136,"data_stored":186446,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":14,"seq":60129542166,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31652,"kb_used_data":464,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340188,"statfs":{"total":96636764160,"available":96604352512,"internally_reserved":0,"allocated":475136,"data_stored":186446,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":14,"seq":60129542166,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27508,"kb_used_data":464,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344332,"statfs":{"total":96636764160,"available":96608595968,"internally_reserved":0,"allocated":475136,"data_stored":186446,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":14,"seq":60129542168,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27480,"kb_used_data":452,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344360,"statfs":{"total":96636764160,"available":96608624640,"internally_reserved":0,"allocated":462848,"data_stored":180629,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":14,"seq":60129542168,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28072,"kb_used_data":1044,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343768,"statfs":{"total":96636764160,"available":96608018432,"internally_reserved":0,"allocated":1069056,"data_stored":776814,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T15:44:20.301 INFO:tasks.ceph.ceph_manager.ceph:PG 7.3f is not active+clean 2024-09-16T15:44:20.301 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '7.3f', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-09-16T15:44:14.461845+0000', 'last_change': '2024-09-16T15:44:14.461845+0000', 'last_active': '2024-09-16T15:44:14.461845+0000', 'last_peered': '2024-09-16T15:44:14.461845+0000', 'last_clean': '2024-09-16T15:44:14.461845+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-16T15:44:14.461845+0000', 'last_undegraded': '2024-09-16T15:44:14.461845+0000', 'last_fullsized': '2024-09-16T15:44:14.461845+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T15:44:14.461845+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T15:44:14.461845+0000', 'last_clean_scrub_stamp': '2024-09-16T15:44:14.461845+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-09-16T15:44:20.301 INFO:tasks.ceph.ceph_manager.ceph:PG 7.3e is not active+clean 2024-09-16T15:44:20.301 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '7.3e', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-09-16T15:44:14.461845+0000', 'last_change': '2024-09-16T15:44:14.461845+0000', 'last_active': '2024-09-16T15:44:14.461845+0000', 'last_peered': '2024-09-16T15:44:14.461845+0000', 'last_clean': '2024-09-16T15:44:14.461845+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-16T15:44:14.461845+0000', 'last_undegraded': '2024-09-16T15:44:14.461845+0000', 'last_fullsized': '2024-09-16T15:44:14.461845+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T15:44:14.461845+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T15:44:14.461845+0000', 'last_clean_scrub_stamp': '2024-09-16T15:44:14.461845+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-09-16T15:44:20.301 INFO:tasks.ceph.ceph_manager.ceph:PG 7.3d is not active+clean 2024-09-16T15:44:20.301 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '7.3d', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-09-16T15:44:14.461845+0000', 'last_change': '2024-09-16T15:44:14.461845+0000', 'last_active': '2024-09-16T15:44:14.461845+0000', 'last_peered': '2024-09-16T15:44:14.461845+0000', 'last_clean': '2024-09-16T15:44:14.461845+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-16T15:44:14.461845+0000', 'last_undegraded': '2024-09-16T15:44:14.461845+0000', 'last_fullsized': '2024-09-16T15:44:14.461845+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T15:44:14.461845+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T15:44:14.461845+0000', 'last_clean_scrub_stamp': '2024-09-16T15:44:14.461845+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-09-16T15:44:20.301 INFO:tasks.ceph.ceph_manager.ceph:PG 7.3c is not active+clean 2024-09-16T15:44:20.301 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '7.3c', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-09-16T15:44:14.461845+0000', 'last_change': '2024-09-16T15:44:14.461845+0000', 'last_active': '2024-09-16T15:44:14.461845+0000', 'last_peered': '2024-09-16T15:44:14.461845+0000', 'last_clean': '2024-09-16T15:44:14.461845+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-16T15:44:14.461845+0000', 'last_undegraded': '2024-09-16T15:44:14.461845+0000', 'last_fullsized': '2024-09-16T15:44:14.461845+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T15:44:14.461845+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T15:44:14.461845+0000', 'last_clean_scrub_stamp': '2024-09-16T15:44:14.461845+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-09-16T15:44:20.301 INFO:tasks.ceph.ceph_manager.ceph:PG 7.3b is not active+clean 2024-09-16T15:44:20.301 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '7.3b', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-09-16T15:44:14.461845+0000', 'last_change': '2024-09-16T15:44:14.461845+0000', 'last_active': '2024-09-16T15:44:14.461845+0000', 'last_peered': '2024-09-16T15:44:14.461845+0000', 'last_clean': '2024-09-16T15:44:14.461845+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-16T15:44:14.461845+0000', 'last_undegraded': '2024-09-16T15:44:14.461845+0000', 'last_fullsized': '2024-09-16T15:44:14.461845+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T15:44:14.461845+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T15:44:14.461845+0000', 'last_clean_scrub_stamp': '2024-09-16T15:44:14.461845+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-09-16T15:44:20.301 INFO:tasks.ceph.ceph_manager.ceph:PG 7.3a is not active+clean 2024-09-16T15:44:20.301 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '7.3a', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-09-16T15:44:14.461845+0000', 'last_change': '2024-09-16T15:44:14.461845+0000', 'last_active': '2024-09-16T15:44:14.461845+0000', 'last_peered': '2024-09-16T15:44:14.461845+0000', 'last_clean': '2024-09-16T15:44:14.461845+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-16T15:44:14.461845+0000', 'last_undegraded': '2024-09-16T15:44:14.461845+0000', 'last_fullsized': '2024-09-16T15:44:14.461845+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T15:44:14.461845+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T15:44:14.461845+0000', 'last_clean_scrub_stamp': '2024-09-16T15:44:14.461845+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-09-16T15:44:20.302 INFO:tasks.ceph.ceph_manager.ceph:PG 7.39 is not active+clean 2024-09-16T15:44:20.302 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '7.39', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-09-16T15:44:14.461845+0000', 'last_change': '2024-09-16T15:44:14.461845+0000', 'last_active': '2024-09-16T15:44:14.461845+0000', 'last_peered': '2024-09-16T15:44:14.461845+0000', 'last_clean': '2024-09-16T15:44:14.461845+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-16T15:44:14.461845+0000', 'last_undegraded': '2024-09-16T15:44:14.461845+0000', 'last_fullsized': '2024-09-16T15:44:14.461845+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T15:44:14.461845+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T15:44:14.461845+0000', 'last_clean_scrub_stamp': '2024-09-16T15:44:14.461845+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-09-16T15:44:20.302 INFO:tasks.ceph.ceph_manager.ceph:PG 7.38 is not active+clean 2024-09-16T15:44:20.302 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '7.38', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-09-16T15:44:14.461845+0000', 'last_change': '2024-09-16T15:44:14.461845+0000', 'last_active': '2024-09-16T15:44:14.461845+0000', 'last_peered': '2024-09-16T15:44:14.461845+0000', 'last_clean': '2024-09-16T15:44:14.461845+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-16T15:44:14.461845+0000', 'last_undegraded': '2024-09-16T15:44:14.461845+0000', 'last_fullsized': '2024-09-16T15:44:14.461845+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T15:44:14.461845+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T15:44:14.461845+0000', 'last_clean_scrub_stamp': '2024-09-16T15:44:14.461845+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-09-16T15:44:20.302 INFO:tasks.ceph.ceph_manager.ceph:PG 7.37 is not active+clean 2024-09-16T15:44:20.302 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '7.37', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-09-16T15:44:14.461845+0000', 'last_change': '2024-09-16T15:44:14.461845+0000', 'last_active': '2024-09-16T15:44:14.461845+0000', 'last_peered': '2024-09-16T15:44:14.461845+0000', 'last_clean': '2024-09-16T15:44:14.461845+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-16T15:44:14.461845+0000', 'last_undegraded': '2024-09-16T15:44:14.461845+0000', 'last_fullsized': '2024-09-16T15:44:14.461845+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T15:44:14.461845+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T15:44:14.461845+0000', 'last_clean_scrub_stamp': '2024-09-16T15:44:14.461845+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-09-16T15:44:20.302 INFO:tasks.ceph.ceph_manager.ceph:PG 7.36 is not active+clean 2024-09-16T15:44:20.302 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '7.36', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-09-16T15:44:14.461845+0000', 'last_change': '2024-09-16T15:44:14.461845+0000', 'last_active': '2024-09-16T15:44:14.461845+0000', 'last_peered': '2024-09-16T15:44:14.461845+0000', 'last_clean': '2024-09-16T15:44:14.461845+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-16T15:44:14.461845+0000', 'last_undegraded': '2024-09-16T15:44:14.461845+0000', 'last_fullsized': '2024-09-16T15:44:14.461845+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T15:44:14.461845+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T15:44:14.461845+0000', 'last_clean_scrub_stamp': '2024-09-16T15:44:14.461845+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-09-16T15:44:20.302 INFO:tasks.ceph.ceph_manager.ceph:PG 7.35 is not active+clean 2024-09-16T15:44:20.302 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '7.35', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-09-16T15:44:14.461845+0000', 'last_change': '2024-09-16T15:44:14.461845+0000', 'last_active': '2024-09-16T15:44:14.461845+0000', 'last_peered': '2024-09-16T15:44:14.461845+0000', 'last_clean': '2024-09-16T15:44:14.461845+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-16T15:44:14.461845+0000', 'last_undegraded': '2024-09-16T15:44:14.461845+0000', 'last_fullsized': '2024-09-16T15:44:14.461845+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T15:44:14.461845+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T15:44:14.461845+0000', 'last_clean_scrub_stamp': '2024-09-16T15:44:14.461845+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-09-16T15:44:20.302 INFO:tasks.ceph.ceph_manager.ceph:PG 7.34 is not active+clean 2024-09-16T15:44:20.302 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '7.34', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-09-16T15:44:14.461845+0000', 'last_change': '2024-09-16T15:44:14.461845+0000', 'last_active': '2024-09-16T15:44:14.461845+0000', 'last_peered': '2024-09-16T15:44:14.461845+0000', 'last_clean': '2024-09-16T15:44:14.461845+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-16T15:44:14.461845+0000', 'last_undegraded': '2024-09-16T15:44:14.461845+0000', 'last_fullsized': '2024-09-16T15:44:14.461845+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T15:44:14.461845+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T15:44:14.461845+0000', 'last_clean_scrub_stamp': '2024-09-16T15:44:14.461845+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-09-16T15:44:20.303 INFO:tasks.ceph.ceph_manager.ceph:PG 7.33 is not active+clean 2024-09-16T15:44:20.303 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '7.33', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-09-16T15:44:14.461845+0000', 'last_change': '2024-09-16T15:44:14.461845+0000', 'last_active': '2024-09-16T15:44:14.461845+0000', 'last_peered': '2024-09-16T15:44:14.461845+0000', 'last_clean': '2024-09-16T15:44:14.461845+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-16T15:44:14.461845+0000', 'last_undegraded': '2024-09-16T15:44:14.461845+0000', 'last_fullsized': '2024-09-16T15:44:14.461845+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T15:44:14.461845+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T15:44:14.461845+0000', 'last_clean_scrub_stamp': '2024-09-16T15:44:14.461845+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-09-16T15:44:20.303 INFO:tasks.ceph.ceph_manager.ceph:PG 7.32 is not active+clean 2024-09-16T15:44:20.303 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '7.32', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-09-16T15:44:14.461845+0000', 'last_change': '2024-09-16T15:44:14.461845+0000', 'last_active': '2024-09-16T15:44:14.461845+0000', 'last_peered': '2024-09-16T15:44:14.461845+0000', 'last_clean': '2024-09-16T15:44:14.461845+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-16T15:44:14.461845+0000', 'last_undegraded': '2024-09-16T15:44:14.461845+0000', 'last_fullsized': '2024-09-16T15:44:14.461845+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T15:44:14.461845+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T15:44:14.461845+0000', 'last_clean_scrub_stamp': '2024-09-16T15:44:14.461845+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-09-16T15:44:20.303 INFO:tasks.ceph.ceph_manager.ceph:PG 7.31 is not active+clean 2024-09-16T15:44:20.303 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '7.31', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-09-16T15:44:14.461845+0000', 'last_change': '2024-09-16T15:44:14.461845+0000', 'last_active': '2024-09-16T15:44:14.461845+0000', 'last_peered': '2024-09-16T15:44:14.461845+0000', 'last_clean': '2024-09-16T15:44:14.461845+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-16T15:44:14.461845+0000', 'last_undegraded': '2024-09-16T15:44:14.461845+0000', 'last_fullsized': '2024-09-16T15:44:14.461845+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T15:44:14.461845+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T15:44:14.461845+0000', 'last_clean_scrub_stamp': '2024-09-16T15:44:14.461845+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-09-16T15:44:20.303 INFO:tasks.ceph.ceph_manager.ceph:PG 7.30 is not active+clean 2024-09-16T15:44:20.303 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '7.30', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-09-16T15:44:14.461845+0000', 'last_change': '2024-09-16T15:44:14.461845+0000', 'last_active': '2024-09-16T15:44:14.461845+0000', 'last_peered': '2024-09-16T15:44:14.461845+0000', 'last_clean': '2024-09-16T15:44:14.461845+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-16T15:44:14.461845+0000', 'last_undegraded': '2024-09-16T15:44:14.461845+0000', 'last_fullsized': '2024-09-16T15:44:14.461845+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T15:44:14.461845+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T15:44:14.461845+0000', 'last_clean_scrub_stamp': '2024-09-16T15:44:14.461845+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-09-16T15:44:20.303 INFO:tasks.ceph.ceph_manager.ceph:PG 7.2f is not active+clean 2024-09-16T15:44:20.303 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '7.2f', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-09-16T15:44:14.461845+0000', 'last_change': '2024-09-16T15:44:14.461845+0000', 'last_active': '2024-09-16T15:44:14.461845+0000', 'last_peered': '2024-09-16T15:44:14.461845+0000', 'last_clean': '2024-09-16T15:44:14.461845+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-16T15:44:14.461845+0000', 'last_undegraded': '2024-09-16T15:44:14.461845+0000', 'last_fullsized': '2024-09-16T15:44:14.461845+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T15:44:14.461845+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T15:44:14.461845+0000', 'last_clean_scrub_stamp': '2024-09-16T15:44:14.461845+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-09-16T15:44:20.303 INFO:tasks.ceph.ceph_manager.ceph:PG 7.2e is not active+clean 2024-09-16T15:44:20.303 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '7.2e', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-09-16T15:44:14.461845+0000', 'last_change': '2024-09-16T15:44:14.461845+0000', 'last_active': '2024-09-16T15:44:14.461845+0000', 'last_peered': '2024-09-16T15:44:14.461845+0000', 'last_clean': '2024-09-16T15:44:14.461845+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-16T15:44:14.461845+0000', 'last_undegraded': '2024-09-16T15:44:14.461845+0000', 'last_fullsized': '2024-09-16T15:44:14.461845+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T15:44:14.461845+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T15:44:14.461845+0000', 'last_clean_scrub_stamp': '2024-09-16T15:44:14.461845+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-09-16T15:44:20.303 INFO:tasks.ceph.ceph_manager.ceph:PG 7.2d is not active+clean 2024-09-16T15:44:20.304 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '7.2d', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-09-16T15:44:14.461845+0000', 'last_change': '2024-09-16T15:44:14.461845+0000', 'last_active': '2024-09-16T15:44:14.461845+0000', 'last_peered': '2024-09-16T15:44:14.461845+0000', 'last_clean': '2024-09-16T15:44:14.461845+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-16T15:44:14.461845+0000', 'last_undegraded': '2024-09-16T15:44:14.461845+0000', 'last_fullsized': '2024-09-16T15:44:14.461845+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T15:44:14.461845+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T15:44:14.461845+0000', 'last_clean_scrub_stamp': '2024-09-16T15:44:14.461845+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-09-16T15:44:20.304 INFO:tasks.ceph.ceph_manager.ceph:PG 7.2c is not active+clean 2024-09-16T15:44:20.304 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '7.2c', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-09-16T15:44:14.461845+0000', 'last_change': '2024-09-16T15:44:14.461845+0000', 'last_active': '2024-09-16T15:44:14.461845+0000', 'last_peered': '2024-09-16T15:44:14.461845+0000', 'last_clean': '2024-09-16T15:44:14.461845+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-16T15:44:14.461845+0000', 'last_undegraded': '2024-09-16T15:44:14.461845+0000', 'last_fullsized': '2024-09-16T15:44:14.461845+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T15:44:14.461845+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T15:44:14.461845+0000', 'last_clean_scrub_stamp': '2024-09-16T15:44:14.461845+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-09-16T15:44:20.304 INFO:tasks.ceph.ceph_manager.ceph:PG 7.2b is not active+clean 2024-09-16T15:44:20.304 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '7.2b', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-09-16T15:44:14.461845+0000', 'last_change': '2024-09-16T15:44:14.461845+0000', 'last_active': '2024-09-16T15:44:14.461845+0000', 'last_peered': '2024-09-16T15:44:14.461845+0000', 'last_clean': '2024-09-16T15:44:14.461845+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-16T15:44:14.461845+0000', 'last_undegraded': '2024-09-16T15:44:14.461845+0000', 'last_fullsized': '2024-09-16T15:44:14.461845+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T15:44:14.461845+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T15:44:14.461845+0000', 'last_clean_scrub_stamp': '2024-09-16T15:44:14.461845+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-09-16T15:44:20.304 INFO:tasks.ceph.ceph_manager.ceph:PG 7.2a is not active+clean 2024-09-16T15:44:20.304 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '7.2a', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-09-16T15:44:14.461845+0000', 'last_change': '2024-09-16T15:44:14.461845+0000', 'last_active': '2024-09-16T15:44:14.461845+0000', 'last_peered': '2024-09-16T15:44:14.461845+0000', 'last_clean': '2024-09-16T15:44:14.461845+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-16T15:44:14.461845+0000', 'last_undegraded': '2024-09-16T15:44:14.461845+0000', 'last_fullsized': '2024-09-16T15:44:14.461845+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T15:44:14.461845+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T15:44:14.461845+0000', 'last_clean_scrub_stamp': '2024-09-16T15:44:14.461845+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-09-16T15:44:20.304 INFO:tasks.ceph.ceph_manager.ceph:PG 7.29 is not active+clean 2024-09-16T15:44:20.304 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '7.29', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-09-16T15:44:14.461845+0000', 'last_change': '2024-09-16T15:44:14.461845+0000', 'last_active': '2024-09-16T15:44:14.461845+0000', 'last_peered': '2024-09-16T15:44:14.461845+0000', 'last_clean': '2024-09-16T15:44:14.461845+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-16T15:44:14.461845+0000', 'last_undegraded': '2024-09-16T15:44:14.461845+0000', 'last_fullsized': '2024-09-16T15:44:14.461845+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T15:44:14.461845+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T15:44:14.461845+0000', 'last_clean_scrub_stamp': '2024-09-16T15:44:14.461845+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-09-16T15:44:20.304 INFO:tasks.ceph.ceph_manager.ceph:PG 7.28 is not active+clean 2024-09-16T15:44:20.304 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '7.28', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-09-16T15:44:14.461845+0000', 'last_change': '2024-09-16T15:44:14.461845+0000', 'last_active': '2024-09-16T15:44:14.461845+0000', 'last_peered': '2024-09-16T15:44:14.461845+0000', 'last_clean': '2024-09-16T15:44:14.461845+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-16T15:44:14.461845+0000', 'last_undegraded': '2024-09-16T15:44:14.461845+0000', 'last_fullsized': '2024-09-16T15:44:14.461845+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T15:44:14.461845+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T15:44:14.461845+0000', 'last_clean_scrub_stamp': '2024-09-16T15:44:14.461845+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-09-16T15:44:20.304 INFO:tasks.ceph.ceph_manager.ceph:PG 7.27 is not active+clean 2024-09-16T15:44:20.305 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '7.27', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-09-16T15:44:14.461845+0000', 'last_change': '2024-09-16T15:44:14.461845+0000', 'last_active': '2024-09-16T15:44:14.461845+0000', 'last_peered': '2024-09-16T15:44:14.461845+0000', 'last_clean': '2024-09-16T15:44:14.461845+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-16T15:44:14.461845+0000', 'last_undegraded': '2024-09-16T15:44:14.461845+0000', 'last_fullsized': '2024-09-16T15:44:14.461845+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T15:44:14.461845+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T15:44:14.461845+0000', 'last_clean_scrub_stamp': '2024-09-16T15:44:14.461845+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-09-16T15:44:20.305 INFO:tasks.ceph.ceph_manager.ceph:PG 7.26 is not active+clean 2024-09-16T15:44:20.305 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '7.26', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-09-16T15:44:14.461845+0000', 'last_change': '2024-09-16T15:44:14.461845+0000', 'last_active': '2024-09-16T15:44:14.461845+0000', 'last_peered': '2024-09-16T15:44:14.461845+0000', 'last_clean': '2024-09-16T15:44:14.461845+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-16T15:44:14.461845+0000', 'last_undegraded': '2024-09-16T15:44:14.461845+0000', 'last_fullsized': '2024-09-16T15:44:14.461845+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T15:44:14.461845+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T15:44:14.461845+0000', 'last_clean_scrub_stamp': '2024-09-16T15:44:14.461845+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-09-16T15:44:20.305 INFO:tasks.ceph.ceph_manager.ceph:PG 7.25 is not active+clean 2024-09-16T15:44:20.305 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '7.25', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-09-16T15:44:14.461845+0000', 'last_change': '2024-09-16T15:44:14.461845+0000', 'last_active': '2024-09-16T15:44:14.461845+0000', 'last_peered': '2024-09-16T15:44:14.461845+0000', 'last_clean': '2024-09-16T15:44:14.461845+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-16T15:44:14.461845+0000', 'last_undegraded': '2024-09-16T15:44:14.461845+0000', 'last_fullsized': '2024-09-16T15:44:14.461845+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T15:44:14.461845+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T15:44:14.461845+0000', 'last_clean_scrub_stamp': '2024-09-16T15:44:14.461845+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-09-16T15:44:20.305 INFO:tasks.ceph.ceph_manager.ceph:PG 7.24 is not active+clean 2024-09-16T15:44:20.305 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '7.24', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-09-16T15:44:14.461845+0000', 'last_change': '2024-09-16T15:44:14.461845+0000', 'last_active': '2024-09-16T15:44:14.461845+0000', 'last_peered': '2024-09-16T15:44:14.461845+0000', 'last_clean': '2024-09-16T15:44:14.461845+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-16T15:44:14.461845+0000', 'last_undegraded': '2024-09-16T15:44:14.461845+0000', 'last_fullsized': '2024-09-16T15:44:14.461845+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T15:44:14.461845+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T15:44:14.461845+0000', 'last_clean_scrub_stamp': '2024-09-16T15:44:14.461845+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-09-16T15:44:20.305 INFO:tasks.ceph.ceph_manager.ceph:PG 7.23 is not active+clean 2024-09-16T15:44:20.305 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '7.23', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-09-16T15:44:14.461845+0000', 'last_change': '2024-09-16T15:44:14.461845+0000', 'last_active': '2024-09-16T15:44:14.461845+0000', 'last_peered': '2024-09-16T15:44:14.461845+0000', 'last_clean': '2024-09-16T15:44:14.461845+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-16T15:44:14.461845+0000', 'last_undegraded': '2024-09-16T15:44:14.461845+0000', 'last_fullsized': '2024-09-16T15:44:14.461845+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T15:44:14.461845+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T15:44:14.461845+0000', 'last_clean_scrub_stamp': '2024-09-16T15:44:14.461845+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-09-16T15:44:20.305 INFO:tasks.ceph.ceph_manager.ceph:PG 7.22 is not active+clean 2024-09-16T15:44:20.305 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '7.22', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-09-16T15:44:14.461845+0000', 'last_change': '2024-09-16T15:44:14.461845+0000', 'last_active': '2024-09-16T15:44:14.461845+0000', 'last_peered': '2024-09-16T15:44:14.461845+0000', 'last_clean': '2024-09-16T15:44:14.461845+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-16T15:44:14.461845+0000', 'last_undegraded': '2024-09-16T15:44:14.461845+0000', 'last_fullsized': '2024-09-16T15:44:14.461845+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T15:44:14.461845+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T15:44:14.461845+0000', 'last_clean_scrub_stamp': '2024-09-16T15:44:14.461845+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-09-16T15:44:20.306 INFO:tasks.ceph.ceph_manager.ceph:PG 7.21 is not active+clean 2024-09-16T15:44:20.306 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '7.21', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-09-16T15:44:14.461845+0000', 'last_change': '2024-09-16T15:44:14.461845+0000', 'last_active': '2024-09-16T15:44:14.461845+0000', 'last_peered': '2024-09-16T15:44:14.461845+0000', 'last_clean': '2024-09-16T15:44:14.461845+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-16T15:44:14.461845+0000', 'last_undegraded': '2024-09-16T15:44:14.461845+0000', 'last_fullsized': '2024-09-16T15:44:14.461845+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T15:44:14.461845+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T15:44:14.461845+0000', 'last_clean_scrub_stamp': '2024-09-16T15:44:14.461845+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-09-16T15:44:20.306 INFO:tasks.ceph.ceph_manager.ceph:PG 7.20 is not active+clean 2024-09-16T15:44:20.306 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '7.20', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-09-16T15:44:14.461845+0000', 'last_change': '2024-09-16T15:44:14.461845+0000', 'last_active': '2024-09-16T15:44:14.461845+0000', 'last_peered': '2024-09-16T15:44:14.461845+0000', 'last_clean': '2024-09-16T15:44:14.461845+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-16T15:44:14.461845+0000', 'last_undegraded': '2024-09-16T15:44:14.461845+0000', 'last_fullsized': '2024-09-16T15:44:14.461845+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T15:44:14.461845+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T15:44:14.461845+0000', 'last_clean_scrub_stamp': '2024-09-16T15:44:14.461845+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-09-16T15:44:20.306 INFO:tasks.ceph.ceph_manager.ceph:PG 7.1e is not active+clean 2024-09-16T15:44:20.306 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '7.1e', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-09-16T15:44:14.461845+0000', 'last_change': '2024-09-16T15:44:14.461845+0000', 'last_active': '2024-09-16T15:44:14.461845+0000', 'last_peered': '2024-09-16T15:44:14.461845+0000', 'last_clean': '2024-09-16T15:44:14.461845+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-16T15:44:14.461845+0000', 'last_undegraded': '2024-09-16T15:44:14.461845+0000', 'last_fullsized': '2024-09-16T15:44:14.461845+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T15:44:14.461845+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T15:44:14.461845+0000', 'last_clean_scrub_stamp': '2024-09-16T15:44:14.461845+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-09-16T15:44:20.306 INFO:tasks.ceph.ceph_manager.ceph:PG 6.1f is not active+clean 2024-09-16T15:44:20.306 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '6.1f', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 41, 'state': 'creating+peering', 'last_fresh': '2024-09-16T15:44:13.470227+0000', 'last_change': '2024-09-16T15:44:13.470227+0000', 'last_active': '2024-09-16T15:44:13.453120+0000', 'last_peered': '2024-09-16T15:44:13.453120+0000', 'last_clean': '2024-09-16T15:44:13.453120+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-16T15:44:13.470227+0000', 'last_undegraded': '2024-09-16T15:44:13.470227+0000', 'last_fullsized': '2024-09-16T15:44:13.470227+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 41, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T15:44:13.453120+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T15:44:13.453120+0000', 'last_clean_scrub_stamp': '2024-09-16T15:44:13.453120+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 6], 'acting': [3, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T15:44:20.306 INFO:tasks.ceph.ceph_manager.ceph:PG 7.1f is not active+clean 2024-09-16T15:44:20.306 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '7.1f', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-09-16T15:44:14.461845+0000', 'last_change': '2024-09-16T15:44:14.461845+0000', 'last_active': '2024-09-16T15:44:14.461845+0000', 'last_peered': '2024-09-16T15:44:14.461845+0000', 'last_clean': '2024-09-16T15:44:14.461845+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-16T15:44:14.461845+0000', 'last_undegraded': '2024-09-16T15:44:14.461845+0000', 'last_fullsized': '2024-09-16T15:44:14.461845+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T15:44:14.461845+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T15:44:14.461845+0000', 'last_clean_scrub_stamp': '2024-09-16T15:44:14.461845+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-09-16T15:44:20.306 INFO:tasks.ceph.ceph_manager.ceph:PG 6.1e is not active+clean 2024-09-16T15:44:20.307 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '6.1e', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 41, 'state': 'creating+peering', 'last_fresh': '2024-09-16T15:44:13.469543+0000', 'last_change': '2024-09-16T15:44:13.469543+0000', 'last_active': '2024-09-16T15:44:13.453120+0000', 'last_peered': '2024-09-16T15:44:13.453120+0000', 'last_clean': '2024-09-16T15:44:13.453120+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-16T15:44:13.469543+0000', 'last_undegraded': '2024-09-16T15:44:13.469543+0000', 'last_fullsized': '2024-09-16T15:44:13.469543+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 41, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T15:44:13.453120+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T15:44:13.453120+0000', 'last_clean_scrub_stamp': '2024-09-16T15:44:13.453120+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 6], 'acting': [4, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-09-16T15:44:20.307 INFO:tasks.ceph.ceph_manager.ceph:PG 7.1c is not active+clean 2024-09-16T15:44:20.307 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '7.1c', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-09-16T15:44:14.461845+0000', 'last_change': '2024-09-16T15:44:14.461845+0000', 'last_active': '2024-09-16T15:44:14.461845+0000', 'last_peered': '2024-09-16T15:44:14.461845+0000', 'last_clean': '2024-09-16T15:44:14.461845+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-16T15:44:14.461845+0000', 'last_undegraded': '2024-09-16T15:44:14.461845+0000', 'last_fullsized': '2024-09-16T15:44:14.461845+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T15:44:14.461845+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T15:44:14.461845+0000', 'last_clean_scrub_stamp': '2024-09-16T15:44:14.461845+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-09-16T15:44:20.307 INFO:tasks.ceph.ceph_manager.ceph:PG 6.1d is not active+clean 2024-09-16T15:44:20.307 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '6.1d', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 41, 'state': 'creating+peering', 'last_fresh': '2024-09-16T15:44:13.472025+0000', 'last_change': '2024-09-16T15:44:13.472025+0000', 'last_active': '2024-09-16T15:44:13.453120+0000', 'last_peered': '2024-09-16T15:44:13.453120+0000', 'last_clean': '2024-09-16T15:44:13.453120+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-16T15:44:13.472025+0000', 'last_undegraded': '2024-09-16T15:44:13.472025+0000', 'last_fullsized': '2024-09-16T15:44:13.472025+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 41, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T15:44:13.453120+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T15:44:13.453120+0000', 'last_clean_scrub_stamp': '2024-09-16T15:44:13.453120+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 5], 'acting': [1, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-09-16T15:44:20.307 INFO:tasks.ceph.ceph_manager.ceph:PG 7.1d is not active+clean 2024-09-16T15:44:20.307 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '7.1d', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-09-16T15:44:14.461845+0000', 'last_change': '2024-09-16T15:44:14.461845+0000', 'last_active': '2024-09-16T15:44:14.461845+0000', 'last_peered': '2024-09-16T15:44:14.461845+0000', 'last_clean': '2024-09-16T15:44:14.461845+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-16T15:44:14.461845+0000', 'last_undegraded': '2024-09-16T15:44:14.461845+0000', 'last_fullsized': '2024-09-16T15:44:14.461845+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T15:44:14.461845+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T15:44:14.461845+0000', 'last_clean_scrub_stamp': '2024-09-16T15:44:14.461845+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-09-16T15:44:20.307 INFO:tasks.ceph.ceph_manager.ceph:PG 6.1c is not active+clean 2024-09-16T15:44:20.307 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '6.1c', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 41, 'state': 'creating+peering', 'last_fresh': '2024-09-16T15:44:13.475916+0000', 'last_change': '2024-09-16T15:44:13.475916+0000', 'last_active': '2024-09-16T15:44:13.453120+0000', 'last_peered': '2024-09-16T15:44:13.453120+0000', 'last_clean': '2024-09-16T15:44:13.453120+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-16T15:44:13.475916+0000', 'last_undegraded': '2024-09-16T15:44:13.475916+0000', 'last_fullsized': '2024-09-16T15:44:13.475916+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 41, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T15:44:13.453120+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T15:44:13.453120+0000', 'last_clean_scrub_stamp': '2024-09-16T15:44:13.453120+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 5], 'acting': [7, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-09-16T15:44:20.307 INFO:tasks.ceph.ceph_manager.ceph:PG 7.1a is not active+clean 2024-09-16T15:44:20.307 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '7.1a', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-09-16T15:44:14.461845+0000', 'last_change': '2024-09-16T15:44:14.461845+0000', 'last_active': '2024-09-16T15:44:14.461845+0000', 'last_peered': '2024-09-16T15:44:14.461845+0000', 'last_clean': '2024-09-16T15:44:14.461845+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-16T15:44:14.461845+0000', 'last_undegraded': '2024-09-16T15:44:14.461845+0000', 'last_fullsized': '2024-09-16T15:44:14.461845+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T15:44:14.461845+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T15:44:14.461845+0000', 'last_clean_scrub_stamp': '2024-09-16T15:44:14.461845+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-09-16T15:44:20.307 INFO:tasks.ceph.ceph_manager.ceph:PG 6.1b is not active+clean 2024-09-16T15:44:20.307 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '6.1b', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 41, 'state': 'creating+peering', 'last_fresh': '2024-09-16T15:44:13.470521+0000', 'last_change': '2024-09-16T15:44:13.470521+0000', 'last_active': '2024-09-16T15:44:13.453120+0000', 'last_peered': '2024-09-16T15:44:13.453120+0000', 'last_clean': '2024-09-16T15:44:13.453120+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-16T15:44:13.470521+0000', 'last_undegraded': '2024-09-16T15:44:13.470521+0000', 'last_fullsized': '2024-09-16T15:44:13.470521+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 41, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T15:44:13.453120+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T15:44:13.453120+0000', 'last_clean_scrub_stamp': '2024-09-16T15:44:13.453120+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 7], 'acting': [3, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T15:44:20.308 INFO:tasks.ceph.ceph_manager.ceph:PG 7.1b is not active+clean 2024-09-16T15:44:20.308 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '7.1b', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-09-16T15:44:14.461845+0000', 'last_change': '2024-09-16T15:44:14.461845+0000', 'last_active': '2024-09-16T15:44:14.461845+0000', 'last_peered': '2024-09-16T15:44:14.461845+0000', 'last_clean': '2024-09-16T15:44:14.461845+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-16T15:44:14.461845+0000', 'last_undegraded': '2024-09-16T15:44:14.461845+0000', 'last_fullsized': '2024-09-16T15:44:14.461845+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T15:44:14.461845+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T15:44:14.461845+0000', 'last_clean_scrub_stamp': '2024-09-16T15:44:14.461845+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-09-16T15:44:20.308 INFO:tasks.ceph.ceph_manager.ceph:PG 6.1a is not active+clean 2024-09-16T15:44:20.308 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '6.1a', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 41, 'state': 'creating+peering', 'last_fresh': '2024-09-16T15:44:13.468698+0000', 'last_change': '2024-09-16T15:44:13.468698+0000', 'last_active': '2024-09-16T15:44:13.453120+0000', 'last_peered': '2024-09-16T15:44:13.453120+0000', 'last_clean': '2024-09-16T15:44:13.453120+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-16T15:44:13.468698+0000', 'last_undegraded': '2024-09-16T15:44:13.468698+0000', 'last_fullsized': '2024-09-16T15:44:13.468698+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 41, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T15:44:13.453120+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T15:44:13.453120+0000', 'last_clean_scrub_stamp': '2024-09-16T15:44:13.453120+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 5], 'acting': [4, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-09-16T15:44:20.308 INFO:tasks.ceph.ceph_manager.ceph:PG 7.18 is not active+clean 2024-09-16T15:44:20.308 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '7.18', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-09-16T15:44:14.461845+0000', 'last_change': '2024-09-16T15:44:14.461845+0000', 'last_active': '2024-09-16T15:44:14.461845+0000', 'last_peered': '2024-09-16T15:44:14.461845+0000', 'last_clean': '2024-09-16T15:44:14.461845+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-16T15:44:14.461845+0000', 'last_undegraded': '2024-09-16T15:44:14.461845+0000', 'last_fullsized': '2024-09-16T15:44:14.461845+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T15:44:14.461845+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T15:44:14.461845+0000', 'last_clean_scrub_stamp': '2024-09-16T15:44:14.461845+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-09-16T15:44:20.308 INFO:tasks.ceph.ceph_manager.ceph:PG 6.19 is not active+clean 2024-09-16T15:44:20.308 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '6.19', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 41, 'state': 'creating+peering', 'last_fresh': '2024-09-16T15:44:13.478336+0000', 'last_change': '2024-09-16T15:44:13.478336+0000', 'last_active': '2024-09-16T15:44:13.453120+0000', 'last_peered': '2024-09-16T15:44:13.453120+0000', 'last_clean': '2024-09-16T15:44:13.453120+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-16T15:44:13.478336+0000', 'last_undegraded': '2024-09-16T15:44:13.478336+0000', 'last_fullsized': '2024-09-16T15:44:13.478336+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 41, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T15:44:13.453120+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T15:44:13.453120+0000', 'last_clean_scrub_stamp': '2024-09-16T15:44:13.453120+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 1], 'acting': [5, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-09-16T15:44:20.308 INFO:tasks.ceph.ceph_manager.ceph:PG 7.19 is not active+clean 2024-09-16T15:44:20.308 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '7.19', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-09-16T15:44:14.461845+0000', 'last_change': '2024-09-16T15:44:14.461845+0000', 'last_active': '2024-09-16T15:44:14.461845+0000', 'last_peered': '2024-09-16T15:44:14.461845+0000', 'last_clean': '2024-09-16T15:44:14.461845+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-16T15:44:14.461845+0000', 'last_undegraded': '2024-09-16T15:44:14.461845+0000', 'last_fullsized': '2024-09-16T15:44:14.461845+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T15:44:14.461845+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T15:44:14.461845+0000', 'last_clean_scrub_stamp': '2024-09-16T15:44:14.461845+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-09-16T15:44:20.308 INFO:tasks.ceph.ceph_manager.ceph:PG 6.18 is not active+clean 2024-09-16T15:44:20.308 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '6.18', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 41, 'state': 'creating+peering', 'last_fresh': '2024-09-16T15:44:13.475717+0000', 'last_change': '2024-09-16T15:44:13.475717+0000', 'last_active': '2024-09-16T15:44:13.453120+0000', 'last_peered': '2024-09-16T15:44:13.453120+0000', 'last_clean': '2024-09-16T15:44:13.453120+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-16T15:44:13.475717+0000', 'last_undegraded': '2024-09-16T15:44:13.475717+0000', 'last_fullsized': '2024-09-16T15:44:13.475717+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 41, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T15:44:13.453120+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T15:44:13.453120+0000', 'last_clean_scrub_stamp': '2024-09-16T15:44:13.453120+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T15:44:20.309 INFO:tasks.ceph.ceph_manager.ceph:PG 7.16 is not active+clean 2024-09-16T15:44:20.309 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '7.16', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-09-16T15:44:14.461845+0000', 'last_change': '2024-09-16T15:44:14.461845+0000', 'last_active': '2024-09-16T15:44:14.461845+0000', 'last_peered': '2024-09-16T15:44:14.461845+0000', 'last_clean': '2024-09-16T15:44:14.461845+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-16T15:44:14.461845+0000', 'last_undegraded': '2024-09-16T15:44:14.461845+0000', 'last_fullsized': '2024-09-16T15:44:14.461845+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T15:44:14.461845+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T15:44:14.461845+0000', 'last_clean_scrub_stamp': '2024-09-16T15:44:14.461845+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-09-16T15:44:20.309 INFO:tasks.ceph.ceph_manager.ceph:PG 6.17 is not active+clean 2024-09-16T15:44:20.309 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '6.17', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 41, 'state': 'creating+peering', 'last_fresh': '2024-09-16T15:44:13.468269+0000', 'last_change': '2024-09-16T15:44:13.468269+0000', 'last_active': '2024-09-16T15:44:13.453120+0000', 'last_peered': '2024-09-16T15:44:13.453120+0000', 'last_clean': '2024-09-16T15:44:13.453120+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-16T15:44:13.468269+0000', 'last_undegraded': '2024-09-16T15:44:13.468269+0000', 'last_fullsized': '2024-09-16T15:44:13.468269+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 41, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T15:44:13.453120+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T15:44:13.453120+0000', 'last_clean_scrub_stamp': '2024-09-16T15:44:13.453120+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 2], 'acting': [4, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-09-16T15:44:20.309 INFO:tasks.ceph.ceph_manager.ceph:PG 7.17 is not active+clean 2024-09-16T15:44:20.309 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '7.17', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-09-16T15:44:14.461845+0000', 'last_change': '2024-09-16T15:44:14.461845+0000', 'last_active': '2024-09-16T15:44:14.461845+0000', 'last_peered': '2024-09-16T15:44:14.461845+0000', 'last_clean': '2024-09-16T15:44:14.461845+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-16T15:44:14.461845+0000', 'last_undegraded': '2024-09-16T15:44:14.461845+0000', 'last_fullsized': '2024-09-16T15:44:14.461845+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T15:44:14.461845+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T15:44:14.461845+0000', 'last_clean_scrub_stamp': '2024-09-16T15:44:14.461845+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-09-16T15:44:20.309 INFO:tasks.ceph.ceph_manager.ceph:PG 6.16 is not active+clean 2024-09-16T15:44:20.309 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '6.16', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 41, 'state': 'creating+peering', 'last_fresh': '2024-09-16T15:44:13.473942+0000', 'last_change': '2024-09-16T15:44:13.473942+0000', 'last_active': '2024-09-16T15:44:13.453120+0000', 'last_peered': '2024-09-16T15:44:13.453120+0000', 'last_clean': '2024-09-16T15:44:13.453120+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-16T15:44:13.473942+0000', 'last_undegraded': '2024-09-16T15:44:13.473942+0000', 'last_fullsized': '2024-09-16T15:44:13.473942+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 41, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T15:44:13.453120+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T15:44:13.453120+0000', 'last_clean_scrub_stamp': '2024-09-16T15:44:13.453120+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 7], 'acting': [0, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T15:44:20.309 INFO:tasks.ceph.ceph_manager.ceph:PG 7.14 is not active+clean 2024-09-16T15:44:20.309 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '7.14', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-09-16T15:44:14.461845+0000', 'last_change': '2024-09-16T15:44:14.461845+0000', 'last_active': '2024-09-16T15:44:14.461845+0000', 'last_peered': '2024-09-16T15:44:14.461845+0000', 'last_clean': '2024-09-16T15:44:14.461845+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-16T15:44:14.461845+0000', 'last_undegraded': '2024-09-16T15:44:14.461845+0000', 'last_fullsized': '2024-09-16T15:44:14.461845+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T15:44:14.461845+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T15:44:14.461845+0000', 'last_clean_scrub_stamp': '2024-09-16T15:44:14.461845+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-09-16T15:44:20.309 INFO:tasks.ceph.ceph_manager.ceph:PG 6.15 is not active+clean 2024-09-16T15:44:20.309 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '6.15', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 41, 'state': 'creating+peering', 'last_fresh': '2024-09-16T15:44:13.476299+0000', 'last_change': '2024-09-16T15:44:13.476299+0000', 'last_active': '2024-09-16T15:44:13.453120+0000', 'last_peered': '2024-09-16T15:44:13.453120+0000', 'last_clean': '2024-09-16T15:44:13.453120+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-16T15:44:13.476299+0000', 'last_undegraded': '2024-09-16T15:44:13.476299+0000', 'last_fullsized': '2024-09-16T15:44:13.476299+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 41, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T15:44:13.453120+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T15:44:13.453120+0000', 'last_clean_scrub_stamp': '2024-09-16T15:44:13.453120+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 6], 'acting': [7, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-09-16T15:44:20.310 INFO:tasks.ceph.ceph_manager.ceph:PG 7.15 is not active+clean 2024-09-16T15:44:20.310 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '7.15', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-09-16T15:44:14.461845+0000', 'last_change': '2024-09-16T15:44:14.461845+0000', 'last_active': '2024-09-16T15:44:14.461845+0000', 'last_peered': '2024-09-16T15:44:14.461845+0000', 'last_clean': '2024-09-16T15:44:14.461845+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-16T15:44:14.461845+0000', 'last_undegraded': '2024-09-16T15:44:14.461845+0000', 'last_fullsized': '2024-09-16T15:44:14.461845+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T15:44:14.461845+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T15:44:14.461845+0000', 'last_clean_scrub_stamp': '2024-09-16T15:44:14.461845+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-09-16T15:44:20.310 INFO:tasks.ceph.ceph_manager.ceph:PG 6.14 is not active+clean 2024-09-16T15:44:20.310 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '6.14', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 41, 'state': 'creating+peering', 'last_fresh': '2024-09-16T15:44:13.470832+0000', 'last_change': '2024-09-16T15:44:13.470832+0000', 'last_active': '2024-09-16T15:44:13.453120+0000', 'last_peered': '2024-09-16T15:44:13.453120+0000', 'last_clean': '2024-09-16T15:44:13.453120+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-16T15:44:13.470832+0000', 'last_undegraded': '2024-09-16T15:44:13.470832+0000', 'last_fullsized': '2024-09-16T15:44:13.470832+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 41, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T15:44:13.453120+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T15:44:13.453120+0000', 'last_clean_scrub_stamp': '2024-09-16T15:44:13.453120+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 4], 'acting': [2, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T15:44:20.310 INFO:tasks.ceph.ceph_manager.ceph:PG 7.12 is not active+clean 2024-09-16T15:44:20.310 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '7.12', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-09-16T15:44:14.461845+0000', 'last_change': '2024-09-16T15:44:14.461845+0000', 'last_active': '2024-09-16T15:44:14.461845+0000', 'last_peered': '2024-09-16T15:44:14.461845+0000', 'last_clean': '2024-09-16T15:44:14.461845+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-16T15:44:14.461845+0000', 'last_undegraded': '2024-09-16T15:44:14.461845+0000', 'last_fullsized': '2024-09-16T15:44:14.461845+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T15:44:14.461845+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T15:44:14.461845+0000', 'last_clean_scrub_stamp': '2024-09-16T15:44:14.461845+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-09-16T15:44:20.310 INFO:tasks.ceph.ceph_manager.ceph:PG 6.13 is not active+clean 2024-09-16T15:44:20.310 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '6.13', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 41, 'state': 'creating+peering', 'last_fresh': '2024-09-16T15:44:13.469026+0000', 'last_change': '2024-09-16T15:44:13.469026+0000', 'last_active': '2024-09-16T15:44:13.453120+0000', 'last_peered': '2024-09-16T15:44:13.453120+0000', 'last_clean': '2024-09-16T15:44:13.453120+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-16T15:44:13.469026+0000', 'last_undegraded': '2024-09-16T15:44:13.469026+0000', 'last_fullsized': '2024-09-16T15:44:13.469026+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 41, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T15:44:13.453120+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T15:44:13.453120+0000', 'last_clean_scrub_stamp': '2024-09-16T15:44:13.453120+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0], 'acting': [3, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T15:44:20.310 INFO:tasks.ceph.ceph_manager.ceph:PG 7.13 is not active+clean 2024-09-16T15:44:20.310 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '7.13', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-09-16T15:44:14.461845+0000', 'last_change': '2024-09-16T15:44:14.461845+0000', 'last_active': '2024-09-16T15:44:14.461845+0000', 'last_peered': '2024-09-16T15:44:14.461845+0000', 'last_clean': '2024-09-16T15:44:14.461845+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-16T15:44:14.461845+0000', 'last_undegraded': '2024-09-16T15:44:14.461845+0000', 'last_fullsized': '2024-09-16T15:44:14.461845+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T15:44:14.461845+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T15:44:14.461845+0000', 'last_clean_scrub_stamp': '2024-09-16T15:44:14.461845+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-09-16T15:44:20.310 INFO:tasks.ceph.ceph_manager.ceph:PG 6.12 is not active+clean 2024-09-16T15:44:20.310 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '6.12', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 41, 'state': 'creating+peering', 'last_fresh': '2024-09-16T15:44:13.474765+0000', 'last_change': '2024-09-16T15:44:13.474765+0000', 'last_active': '2024-09-16T15:44:13.453120+0000', 'last_peered': '2024-09-16T15:44:13.453120+0000', 'last_clean': '2024-09-16T15:44:13.453120+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-16T15:44:13.474765+0000', 'last_undegraded': '2024-09-16T15:44:13.474765+0000', 'last_fullsized': '2024-09-16T15:44:13.474765+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 41, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T15:44:13.453120+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T15:44:13.453120+0000', 'last_clean_scrub_stamp': '2024-09-16T15:44:13.453120+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 2], 'acting': [7, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-09-16T15:44:20.310 INFO:tasks.ceph.ceph_manager.ceph:PG 7.10 is not active+clean 2024-09-16T15:44:20.311 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '7.10', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-09-16T15:44:14.461845+0000', 'last_change': '2024-09-16T15:44:14.461845+0000', 'last_active': '2024-09-16T15:44:14.461845+0000', 'last_peered': '2024-09-16T15:44:14.461845+0000', 'last_clean': '2024-09-16T15:44:14.461845+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-16T15:44:14.461845+0000', 'last_undegraded': '2024-09-16T15:44:14.461845+0000', 'last_fullsized': '2024-09-16T15:44:14.461845+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T15:44:14.461845+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T15:44:14.461845+0000', 'last_clean_scrub_stamp': '2024-09-16T15:44:14.461845+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-09-16T15:44:20.311 INFO:tasks.ceph.ceph_manager.ceph:PG 6.11 is not active+clean 2024-09-16T15:44:20.311 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '6.11', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 41, 'state': 'creating+peering', 'last_fresh': '2024-09-16T15:44:13.468318+0000', 'last_change': '2024-09-16T15:44:13.468318+0000', 'last_active': '2024-09-16T15:44:13.453120+0000', 'last_peered': '2024-09-16T15:44:13.453120+0000', 'last_clean': '2024-09-16T15:44:13.453120+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-16T15:44:13.468318+0000', 'last_undegraded': '2024-09-16T15:44:13.468318+0000', 'last_fullsized': '2024-09-16T15:44:13.468318+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 41, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T15:44:13.453120+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T15:44:13.453120+0000', 'last_clean_scrub_stamp': '2024-09-16T15:44:13.453120+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0], 'acting': [3, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T15:44:20.311 INFO:tasks.ceph.ceph_manager.ceph:PG 7.11 is not active+clean 2024-09-16T15:44:20.311 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '7.11', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-09-16T15:44:14.461845+0000', 'last_change': '2024-09-16T15:44:14.461845+0000', 'last_active': '2024-09-16T15:44:14.461845+0000', 'last_peered': '2024-09-16T15:44:14.461845+0000', 'last_clean': '2024-09-16T15:44:14.461845+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-16T15:44:14.461845+0000', 'last_undegraded': '2024-09-16T15:44:14.461845+0000', 'last_fullsized': '2024-09-16T15:44:14.461845+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T15:44:14.461845+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T15:44:14.461845+0000', 'last_clean_scrub_stamp': '2024-09-16T15:44:14.461845+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-09-16T15:44:20.311 INFO:tasks.ceph.ceph_manager.ceph:PG 6.10 is not active+clean 2024-09-16T15:44:20.311 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '6.10', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 41, 'state': 'creating+peering', 'last_fresh': '2024-09-16T15:44:13.473996+0000', 'last_change': '2024-09-16T15:44:13.473996+0000', 'last_active': '2024-09-16T15:44:13.453120+0000', 'last_peered': '2024-09-16T15:44:13.453120+0000', 'last_clean': '2024-09-16T15:44:13.453120+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-16T15:44:13.473996+0000', 'last_undegraded': '2024-09-16T15:44:13.473996+0000', 'last_fullsized': '2024-09-16T15:44:13.473996+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 41, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T15:44:13.453120+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T15:44:13.453120+0000', 'last_clean_scrub_stamp': '2024-09-16T15:44:13.453120+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 5], 'acting': [0, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T15:44:20.311 INFO:tasks.ceph.ceph_manager.ceph:PG 7.e is not active+clean 2024-09-16T15:44:20.311 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '7.e', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-09-16T15:44:14.461845+0000', 'last_change': '2024-09-16T15:44:14.461845+0000', 'last_active': '2024-09-16T15:44:14.461845+0000', 'last_peered': '2024-09-16T15:44:14.461845+0000', 'last_clean': '2024-09-16T15:44:14.461845+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-16T15:44:14.461845+0000', 'last_undegraded': '2024-09-16T15:44:14.461845+0000', 'last_fullsized': '2024-09-16T15:44:14.461845+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T15:44:14.461845+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T15:44:14.461845+0000', 'last_clean_scrub_stamp': '2024-09-16T15:44:14.461845+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-09-16T15:44:20.311 INFO:tasks.ceph.ceph_manager.ceph:PG 6.f is not active+clean 2024-09-16T15:44:20.311 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '6.f', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 41, 'state': 'creating+peering', 'last_fresh': '2024-09-16T15:44:13.471259+0000', 'last_change': '2024-09-16T15:44:13.471259+0000', 'last_active': '2024-09-16T15:44:13.453120+0000', 'last_peered': '2024-09-16T15:44:13.453120+0000', 'last_clean': '2024-09-16T15:44:13.453120+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-16T15:44:13.471259+0000', 'last_undegraded': '2024-09-16T15:44:13.471259+0000', 'last_fullsized': '2024-09-16T15:44:13.471259+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 41, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T15:44:13.453120+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T15:44:13.453120+0000', 'last_clean_scrub_stamp': '2024-09-16T15:44:13.453120+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T15:44:20.311 INFO:tasks.ceph.ceph_manager.ceph:PG 7.f is not active+clean 2024-09-16T15:44:20.311 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '7.f', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-09-16T15:44:14.461845+0000', 'last_change': '2024-09-16T15:44:14.461845+0000', 'last_active': '2024-09-16T15:44:14.461845+0000', 'last_peered': '2024-09-16T15:44:14.461845+0000', 'last_clean': '2024-09-16T15:44:14.461845+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-16T15:44:14.461845+0000', 'last_undegraded': '2024-09-16T15:44:14.461845+0000', 'last_fullsized': '2024-09-16T15:44:14.461845+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T15:44:14.461845+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T15:44:14.461845+0000', 'last_clean_scrub_stamp': '2024-09-16T15:44:14.461845+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-09-16T15:44:20.312 INFO:tasks.ceph.ceph_manager.ceph:PG 6.e is not active+clean 2024-09-16T15:44:20.312 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '6.e', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 41, 'state': 'creating+peering', 'last_fresh': '2024-09-16T15:44:13.468343+0000', 'last_change': '2024-09-16T15:44:13.468343+0000', 'last_active': '2024-09-16T15:44:13.453120+0000', 'last_peered': '2024-09-16T15:44:13.453120+0000', 'last_clean': '2024-09-16T15:44:13.453120+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-16T15:44:13.468343+0000', 'last_undegraded': '2024-09-16T15:44:13.468343+0000', 'last_fullsized': '2024-09-16T15:44:13.468343+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 41, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T15:44:13.453120+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T15:44:13.453120+0000', 'last_clean_scrub_stamp': '2024-09-16T15:44:13.453120+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 1], 'acting': [4, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-09-16T15:44:20.312 INFO:tasks.ceph.ceph_manager.ceph:PG 7.c is not active+clean 2024-09-16T15:44:20.312 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '7.c', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-09-16T15:44:14.461845+0000', 'last_change': '2024-09-16T15:44:14.461845+0000', 'last_active': '2024-09-16T15:44:14.461845+0000', 'last_peered': '2024-09-16T15:44:14.461845+0000', 'last_clean': '2024-09-16T15:44:14.461845+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-16T15:44:14.461845+0000', 'last_undegraded': '2024-09-16T15:44:14.461845+0000', 'last_fullsized': '2024-09-16T15:44:14.461845+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T15:44:14.461845+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T15:44:14.461845+0000', 'last_clean_scrub_stamp': '2024-09-16T15:44:14.461845+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-09-16T15:44:20.312 INFO:tasks.ceph.ceph_manager.ceph:PG 6.d is not active+clean 2024-09-16T15:44:20.312 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '6.d', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 41, 'state': 'creating+peering', 'last_fresh': '2024-09-16T15:44:13.478680+0000', 'last_change': '2024-09-16T15:44:13.478680+0000', 'last_active': '2024-09-16T15:44:13.453120+0000', 'last_peered': '2024-09-16T15:44:13.453120+0000', 'last_clean': '2024-09-16T15:44:13.453120+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-16T15:44:13.478680+0000', 'last_undegraded': '2024-09-16T15:44:13.478680+0000', 'last_fullsized': '2024-09-16T15:44:13.478680+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 41, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T15:44:13.453120+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T15:44:13.453120+0000', 'last_clean_scrub_stamp': '2024-09-16T15:44:13.453120+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 1], 'acting': [5, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-09-16T15:44:20.312 INFO:tasks.ceph.ceph_manager.ceph:PG 7.d is not active+clean 2024-09-16T15:44:20.312 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '7.d', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-09-16T15:44:14.461845+0000', 'last_change': '2024-09-16T15:44:14.461845+0000', 'last_active': '2024-09-16T15:44:14.461845+0000', 'last_peered': '2024-09-16T15:44:14.461845+0000', 'last_clean': '2024-09-16T15:44:14.461845+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-16T15:44:14.461845+0000', 'last_undegraded': '2024-09-16T15:44:14.461845+0000', 'last_fullsized': '2024-09-16T15:44:14.461845+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T15:44:14.461845+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T15:44:14.461845+0000', 'last_clean_scrub_stamp': '2024-09-16T15:44:14.461845+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-09-16T15:44:20.312 INFO:tasks.ceph.ceph_manager.ceph:PG 6.c is not active+clean 2024-09-16T15:44:20.312 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '6.c', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 41, 'state': 'creating+peering', 'last_fresh': '2024-09-16T15:44:13.468140+0000', 'last_change': '2024-09-16T15:44:13.468140+0000', 'last_active': '2024-09-16T15:44:13.453120+0000', 'last_peered': '2024-09-16T15:44:13.453120+0000', 'last_clean': '2024-09-16T15:44:13.453120+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-16T15:44:13.468140+0000', 'last_undegraded': '2024-09-16T15:44:13.468140+0000', 'last_fullsized': '2024-09-16T15:44:13.468140+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 41, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T15:44:13.453120+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T15:44:13.453120+0000', 'last_clean_scrub_stamp': '2024-09-16T15:44:13.453120+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 6], 'acting': [3, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T15:44:20.312 INFO:tasks.ceph.ceph_manager.ceph:PG 7.a is not active+clean 2024-09-16T15:44:20.313 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '7.a', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-09-16T15:44:14.461845+0000', 'last_change': '2024-09-16T15:44:14.461845+0000', 'last_active': '2024-09-16T15:44:14.461845+0000', 'last_peered': '2024-09-16T15:44:14.461845+0000', 'last_clean': '2024-09-16T15:44:14.461845+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-16T15:44:14.461845+0000', 'last_undegraded': '2024-09-16T15:44:14.461845+0000', 'last_fullsized': '2024-09-16T15:44:14.461845+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T15:44:14.461845+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T15:44:14.461845+0000', 'last_clean_scrub_stamp': '2024-09-16T15:44:14.461845+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-09-16T15:44:20.313 INFO:tasks.ceph.ceph_manager.ceph:PG 6.b is not active+clean 2024-09-16T15:44:20.313 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '6.b', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 41, 'state': 'creating+peering', 'last_fresh': '2024-09-16T15:44:13.468299+0000', 'last_change': '2024-09-16T15:44:13.468299+0000', 'last_active': '2024-09-16T15:44:13.453120+0000', 'last_peered': '2024-09-16T15:44:13.453120+0000', 'last_clean': '2024-09-16T15:44:13.453120+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-16T15:44:13.468299+0000', 'last_undegraded': '2024-09-16T15:44:13.468299+0000', 'last_fullsized': '2024-09-16T15:44:13.468299+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 41, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T15:44:13.453120+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T15:44:13.453120+0000', 'last_clean_scrub_stamp': '2024-09-16T15:44:13.453120+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 7], 'acting': [3, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T15:44:20.313 INFO:tasks.ceph.ceph_manager.ceph:PG 7.b is not active+clean 2024-09-16T15:44:20.313 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '7.b', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-09-16T15:44:14.461845+0000', 'last_change': '2024-09-16T15:44:14.461845+0000', 'last_active': '2024-09-16T15:44:14.461845+0000', 'last_peered': '2024-09-16T15:44:14.461845+0000', 'last_clean': '2024-09-16T15:44:14.461845+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-16T15:44:14.461845+0000', 'last_undegraded': '2024-09-16T15:44:14.461845+0000', 'last_fullsized': '2024-09-16T15:44:14.461845+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T15:44:14.461845+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T15:44:14.461845+0000', 'last_clean_scrub_stamp': '2024-09-16T15:44:14.461845+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-09-16T15:44:20.313 INFO:tasks.ceph.ceph_manager.ceph:PG 6.a is not active+clean 2024-09-16T15:44:20.313 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '6.a', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 41, 'state': 'creating+peering', 'last_fresh': '2024-09-16T15:44:13.473999+0000', 'last_change': '2024-09-16T15:44:13.473999+0000', 'last_active': '2024-09-16T15:44:13.453120+0000', 'last_peered': '2024-09-16T15:44:13.453120+0000', 'last_clean': '2024-09-16T15:44:13.453120+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-16T15:44:13.473999+0000', 'last_undegraded': '2024-09-16T15:44:13.473999+0000', 'last_fullsized': '2024-09-16T15:44:13.473999+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 41, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T15:44:13.453120+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T15:44:13.453120+0000', 'last_clean_scrub_stamp': '2024-09-16T15:44:13.453120+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 6], 'acting': [5, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-09-16T15:44:20.313 INFO:tasks.ceph.ceph_manager.ceph:PG 7.8 is not active+clean 2024-09-16T15:44:20.313 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '7.8', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-09-16T15:44:14.461845+0000', 'last_change': '2024-09-16T15:44:14.461845+0000', 'last_active': '2024-09-16T15:44:14.461845+0000', 'last_peered': '2024-09-16T15:44:14.461845+0000', 'last_clean': '2024-09-16T15:44:14.461845+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-16T15:44:14.461845+0000', 'last_undegraded': '2024-09-16T15:44:14.461845+0000', 'last_fullsized': '2024-09-16T15:44:14.461845+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T15:44:14.461845+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T15:44:14.461845+0000', 'last_clean_scrub_stamp': '2024-09-16T15:44:14.461845+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-09-16T15:44:20.313 INFO:tasks.ceph.ceph_manager.ceph:PG 6.9 is not active+clean 2024-09-16T15:44:20.313 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '6.9', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 41, 'state': 'creating+peering', 'last_fresh': '2024-09-16T15:44:13.473367+0000', 'last_change': '2024-09-16T15:44:13.473367+0000', 'last_active': '2024-09-16T15:44:13.453120+0000', 'last_peered': '2024-09-16T15:44:13.453120+0000', 'last_clean': '2024-09-16T15:44:13.453120+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-16T15:44:13.473367+0000', 'last_undegraded': '2024-09-16T15:44:13.473367+0000', 'last_fullsized': '2024-09-16T15:44:13.473367+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 41, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T15:44:13.453120+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T15:44:13.453120+0000', 'last_clean_scrub_stamp': '2024-09-16T15:44:13.453120+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 7], 'acting': [0, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T15:44:20.313 INFO:tasks.ceph.ceph_manager.ceph:PG 7.9 is not active+clean 2024-09-16T15:44:20.314 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '7.9', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-09-16T15:44:14.461845+0000', 'last_change': '2024-09-16T15:44:14.461845+0000', 'last_active': '2024-09-16T15:44:14.461845+0000', 'last_peered': '2024-09-16T15:44:14.461845+0000', 'last_clean': '2024-09-16T15:44:14.461845+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-16T15:44:14.461845+0000', 'last_undegraded': '2024-09-16T15:44:14.461845+0000', 'last_fullsized': '2024-09-16T15:44:14.461845+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T15:44:14.461845+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T15:44:14.461845+0000', 'last_clean_scrub_stamp': '2024-09-16T15:44:14.461845+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-09-16T15:44:20.314 INFO:tasks.ceph.ceph_manager.ceph:PG 6.8 is not active+clean 2024-09-16T15:44:20.314 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '6.8', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 41, 'state': 'creating+peering', 'last_fresh': '2024-09-16T15:44:13.474597+0000', 'last_change': '2024-09-16T15:44:13.474597+0000', 'last_active': '2024-09-16T15:44:13.453120+0000', 'last_peered': '2024-09-16T15:44:13.453120+0000', 'last_clean': '2024-09-16T15:44:13.453120+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-16T15:44:13.474597+0000', 'last_undegraded': '2024-09-16T15:44:13.474597+0000', 'last_fullsized': '2024-09-16T15:44:13.474597+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 41, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T15:44:13.453120+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T15:44:13.453120+0000', 'last_clean_scrub_stamp': '2024-09-16T15:44:13.453120+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 2], 'acting': [7, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-09-16T15:44:20.314 INFO:tasks.ceph.ceph_manager.ceph:PG 7.7 is not active+clean 2024-09-16T15:44:20.314 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '7.7', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-09-16T15:44:14.461845+0000', 'last_change': '2024-09-16T15:44:14.461845+0000', 'last_active': '2024-09-16T15:44:14.461845+0000', 'last_peered': '2024-09-16T15:44:14.461845+0000', 'last_clean': '2024-09-16T15:44:14.461845+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-16T15:44:14.461845+0000', 'last_undegraded': '2024-09-16T15:44:14.461845+0000', 'last_fullsized': '2024-09-16T15:44:14.461845+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T15:44:14.461845+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T15:44:14.461845+0000', 'last_clean_scrub_stamp': '2024-09-16T15:44:14.461845+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-09-16T15:44:20.314 INFO:tasks.ceph.ceph_manager.ceph:PG 6.6 is not active+clean 2024-09-16T15:44:20.314 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '6.6', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 41, 'state': 'creating+peering', 'last_fresh': '2024-09-16T15:44:13.468102+0000', 'last_change': '2024-09-16T15:44:13.468102+0000', 'last_active': '2024-09-16T15:44:13.453120+0000', 'last_peered': '2024-09-16T15:44:13.453120+0000', 'last_clean': '2024-09-16T15:44:13.453120+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-16T15:44:13.468102+0000', 'last_undegraded': '2024-09-16T15:44:13.468102+0000', 'last_fullsized': '2024-09-16T15:44:13.468102+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 41, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T15:44:13.453120+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T15:44:13.453120+0000', 'last_clean_scrub_stamp': '2024-09-16T15:44:13.453120+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 4], 'acting': [3, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T15:44:20.314 INFO:tasks.ceph.ceph_manager.ceph:PG 7.4 is not active+clean 2024-09-16T15:44:20.314 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '7.4', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-09-16T15:44:14.461845+0000', 'last_change': '2024-09-16T15:44:14.461845+0000', 'last_active': '2024-09-16T15:44:14.461845+0000', 'last_peered': '2024-09-16T15:44:14.461845+0000', 'last_clean': '2024-09-16T15:44:14.461845+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-16T15:44:14.461845+0000', 'last_undegraded': '2024-09-16T15:44:14.461845+0000', 'last_fullsized': '2024-09-16T15:44:14.461845+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T15:44:14.461845+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T15:44:14.461845+0000', 'last_clean_scrub_stamp': '2024-09-16T15:44:14.461845+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-09-16T15:44:20.314 INFO:tasks.ceph.ceph_manager.ceph:PG 6.5 is not active+clean 2024-09-16T15:44:20.314 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '6.5', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 41, 'state': 'creating+peering', 'last_fresh': '2024-09-16T15:44:13.474821+0000', 'last_change': '2024-09-16T15:44:13.474821+0000', 'last_active': '2024-09-16T15:44:13.453120+0000', 'last_peered': '2024-09-16T15:44:13.453120+0000', 'last_clean': '2024-09-16T15:44:13.453120+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-16T15:44:13.474821+0000', 'last_undegraded': '2024-09-16T15:44:13.474821+0000', 'last_fullsized': '2024-09-16T15:44:13.474821+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 41, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T15:44:13.453120+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T15:44:13.453120+0000', 'last_clean_scrub_stamp': '2024-09-16T15:44:13.453120+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 6], 'acting': [7, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-09-16T15:44:20.314 INFO:tasks.ceph.ceph_manager.ceph:PG 7.5 is not active+clean 2024-09-16T15:44:20.314 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '7.5', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-09-16T15:44:14.461845+0000', 'last_change': '2024-09-16T15:44:14.461845+0000', 'last_active': '2024-09-16T15:44:14.461845+0000', 'last_peered': '2024-09-16T15:44:14.461845+0000', 'last_clean': '2024-09-16T15:44:14.461845+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-16T15:44:14.461845+0000', 'last_undegraded': '2024-09-16T15:44:14.461845+0000', 'last_fullsized': '2024-09-16T15:44:14.461845+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T15:44:14.461845+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T15:44:14.461845+0000', 'last_clean_scrub_stamp': '2024-09-16T15:44:14.461845+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-09-16T15:44:20.315 INFO:tasks.ceph.ceph_manager.ceph:PG 6.4 is not active+clean 2024-09-16T15:44:20.315 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '6.4', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 41, 'state': 'creating+peering', 'last_fresh': '2024-09-16T15:44:13.472012+0000', 'last_change': '2024-09-16T15:44:13.472012+0000', 'last_active': '2024-09-16T15:44:13.453120+0000', 'last_peered': '2024-09-16T15:44:13.453120+0000', 'last_clean': '2024-09-16T15:44:13.453120+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-16T15:44:13.472012+0000', 'last_undegraded': '2024-09-16T15:44:13.472012+0000', 'last_fullsized': '2024-09-16T15:44:13.472012+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 41, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T15:44:13.453120+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T15:44:13.453120+0000', 'last_clean_scrub_stamp': '2024-09-16T15:44:13.453120+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 5], 'acting': [1, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-09-16T15:44:20.315 INFO:tasks.ceph.ceph_manager.ceph:PG 7.2 is not active+clean 2024-09-16T15:44:20.315 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '7.2', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-09-16T15:44:14.461845+0000', 'last_change': '2024-09-16T15:44:14.461845+0000', 'last_active': '2024-09-16T15:44:14.461845+0000', 'last_peered': '2024-09-16T15:44:14.461845+0000', 'last_clean': '2024-09-16T15:44:14.461845+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-16T15:44:14.461845+0000', 'last_undegraded': '2024-09-16T15:44:14.461845+0000', 'last_fullsized': '2024-09-16T15:44:14.461845+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T15:44:14.461845+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T15:44:14.461845+0000', 'last_clean_scrub_stamp': '2024-09-16T15:44:14.461845+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-09-16T15:44:20.315 INFO:tasks.ceph.ceph_manager.ceph:PG 6.3 is not active+clean 2024-09-16T15:44:20.315 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '6.3', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 41, 'state': 'creating+peering', 'last_fresh': '2024-09-16T15:44:13.474616+0000', 'last_change': '2024-09-16T15:44:13.474616+0000', 'last_active': '2024-09-16T15:44:13.453120+0000', 'last_peered': '2024-09-16T15:44:13.453120+0000', 'last_clean': '2024-09-16T15:44:13.453120+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-16T15:44:13.474616+0000', 'last_undegraded': '2024-09-16T15:44:13.474616+0000', 'last_fullsized': '2024-09-16T15:44:13.474616+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 41, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T15:44:13.453120+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T15:44:13.453120+0000', 'last_clean_scrub_stamp': '2024-09-16T15:44:13.453120+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 6], 'acting': [7, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-09-16T15:44:20.315 INFO:tasks.ceph.ceph_manager.ceph:PG 7.3 is not active+clean 2024-09-16T15:44:20.315 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '7.3', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-09-16T15:44:14.461845+0000', 'last_change': '2024-09-16T15:44:14.461845+0000', 'last_active': '2024-09-16T15:44:14.461845+0000', 'last_peered': '2024-09-16T15:44:14.461845+0000', 'last_clean': '2024-09-16T15:44:14.461845+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-16T15:44:14.461845+0000', 'last_undegraded': '2024-09-16T15:44:14.461845+0000', 'last_fullsized': '2024-09-16T15:44:14.461845+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T15:44:14.461845+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T15:44:14.461845+0000', 'last_clean_scrub_stamp': '2024-09-16T15:44:14.461845+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-09-16T15:44:20.315 INFO:tasks.ceph.ceph_manager.ceph:PG 6.2 is not active+clean 2024-09-16T15:44:20.315 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '6.2', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 41, 'state': 'creating+peering', 'last_fresh': '2024-09-16T15:44:13.467248+0000', 'last_change': '2024-09-16T15:44:13.467248+0000', 'last_active': '2024-09-16T15:44:13.453120+0000', 'last_peered': '2024-09-16T15:44:13.453120+0000', 'last_clean': '2024-09-16T15:44:13.453120+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-16T15:44:13.467248+0000', 'last_undegraded': '2024-09-16T15:44:13.467248+0000', 'last_fullsized': '2024-09-16T15:44:13.467248+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 41, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T15:44:13.453120+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T15:44:13.453120+0000', 'last_clean_scrub_stamp': '2024-09-16T15:44:13.453120+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 3], 'acting': [4, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-09-16T15:44:20.315 INFO:tasks.ceph.ceph_manager.ceph:PG 7.0 is not active+clean 2024-09-16T15:44:20.315 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '7.0', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-09-16T15:44:14.461845+0000', 'last_change': '2024-09-16T15:44:14.461845+0000', 'last_active': '2024-09-16T15:44:14.461845+0000', 'last_peered': '2024-09-16T15:44:14.461845+0000', 'last_clean': '2024-09-16T15:44:14.461845+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-16T15:44:14.461845+0000', 'last_undegraded': '2024-09-16T15:44:14.461845+0000', 'last_fullsized': '2024-09-16T15:44:14.461845+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T15:44:14.461845+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T15:44:14.461845+0000', 'last_clean_scrub_stamp': '2024-09-16T15:44:14.461845+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-09-16T15:44:20.316 INFO:tasks.ceph.ceph_manager.ceph:PG 6.1 is not active+clean 2024-09-16T15:44:20.316 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '6.1', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 41, 'state': 'creating+peering', 'last_fresh': '2024-09-16T15:44:13.472810+0000', 'last_change': '2024-09-16T15:44:13.472810+0000', 'last_active': '2024-09-16T15:44:13.453120+0000', 'last_peered': '2024-09-16T15:44:13.453120+0000', 'last_clean': '2024-09-16T15:44:13.453120+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-16T15:44:13.472810+0000', 'last_undegraded': '2024-09-16T15:44:13.472810+0000', 'last_fullsized': '2024-09-16T15:44:13.472810+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 41, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T15:44:13.453120+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T15:44:13.453120+0000', 'last_clean_scrub_stamp': '2024-09-16T15:44:13.453120+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 6], 'acting': [1, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-09-16T15:44:20.316 INFO:tasks.ceph.ceph_manager.ceph:PG 7.1 is not active+clean 2024-09-16T15:44:20.316 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '7.1', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-09-16T15:44:14.461845+0000', 'last_change': '2024-09-16T15:44:14.461845+0000', 'last_active': '2024-09-16T15:44:14.461845+0000', 'last_peered': '2024-09-16T15:44:14.461845+0000', 'last_clean': '2024-09-16T15:44:14.461845+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-16T15:44:14.461845+0000', 'last_undegraded': '2024-09-16T15:44:14.461845+0000', 'last_fullsized': '2024-09-16T15:44:14.461845+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T15:44:14.461845+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T15:44:14.461845+0000', 'last_clean_scrub_stamp': '2024-09-16T15:44:14.461845+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-09-16T15:44:20.316 INFO:tasks.ceph.ceph_manager.ceph:PG 6.0 is not active+clean 2024-09-16T15:44:20.316 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '6.0', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 41, 'state': 'creating+peering', 'last_fresh': '2024-09-16T15:44:13.473352+0000', 'last_change': '2024-09-16T15:44:13.473352+0000', 'last_active': '2024-09-16T15:44:13.453120+0000', 'last_peered': '2024-09-16T15:44:13.453120+0000', 'last_clean': '2024-09-16T15:44:13.453120+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-16T15:44:13.473352+0000', 'last_undegraded': '2024-09-16T15:44:13.473352+0000', 'last_fullsized': '2024-09-16T15:44:13.473352+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 41, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T15:44:13.453120+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T15:44:13.453120+0000', 'last_clean_scrub_stamp': '2024-09-16T15:44:13.453120+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3], 'acting': [0, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T15:44:20.316 INFO:tasks.ceph.ceph_manager.ceph:PG 7.6 is not active+clean 2024-09-16T15:44:20.316 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '7.6', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-09-16T15:44:14.461845+0000', 'last_change': '2024-09-16T15:44:14.461845+0000', 'last_active': '2024-09-16T15:44:14.461845+0000', 'last_peered': '2024-09-16T15:44:14.461845+0000', 'last_clean': '2024-09-16T15:44:14.461845+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-16T15:44:14.461845+0000', 'last_undegraded': '2024-09-16T15:44:14.461845+0000', 'last_fullsized': '2024-09-16T15:44:14.461845+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T15:44:14.461845+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T15:44:14.461845+0000', 'last_clean_scrub_stamp': '2024-09-16T15:44:14.461845+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-09-16T15:44:20.316 INFO:tasks.ceph.ceph_manager.ceph:PG 6.7 is not active+clean 2024-09-16T15:44:20.316 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '6.7', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 41, 'state': 'creating+peering', 'last_fresh': '2024-09-16T15:44:13.476429+0000', 'last_change': '2024-09-16T15:44:13.476429+0000', 'last_active': '2024-09-16T15:44:13.453120+0000', 'last_peered': '2024-09-16T15:44:13.453120+0000', 'last_clean': '2024-09-16T15:44:13.453120+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-16T15:44:13.476429+0000', 'last_undegraded': '2024-09-16T15:44:13.476429+0000', 'last_fullsized': '2024-09-16T15:44:13.476429+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 41, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T15:44:13.453120+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T15:44:13.453120+0000', 'last_clean_scrub_stamp': '2024-09-16T15:44:13.453120+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 3], 'acting': [5, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-09-16T15:44:20.317 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T15:44:20.467 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:20.460+0000 7f56c80dc640 1 -- 172.21.15.123:0/1386045040 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f56ac0052f0 msgr2=0x7f56ac00b9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:44:20.467 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:20.460+0000 7f56c80dc640 1 --2- 172.21.15.123:0/1386045040 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f56ac0052f0 0x7f56ac00b9d0 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f56b800c640 tx=0x7f56b801d240 comp rx=0 tx=0).stop 2024-09-16T15:44:20.467 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:20.464+0000 7f56c80dc640 1 -- 172.21.15.123:0/1386045040 shutdown_connections 2024-09-16T15:44:20.468 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:20.464+0000 7f56c80dc640 1 --2- 172.21.15.123:0/1386045040 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f56ac0052f0 0x7f56ac00b9d0 unknown :-1 s=CLOSED pgs=459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:20.468 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:20.464+0000 7f56c80dc640 1 -- 172.21.15.123:0/1386045040 >> 172.21.15.123:0/1386045040 conn(0x7f56c00cfa90 msgr2=0x7f56c00cfea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:44:20.468 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:20.464+0000 7f56c80dc640 1 -- 172.21.15.123:0/1386045040 shutdown_connections 2024-09-16T15:44:20.468 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:20.464+0000 7f56c80dc640 1 -- 172.21.15.123:0/1386045040 wait complete. 2024-09-16T15:44:20.468 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:20.464+0000 7f56c80dc640 1 Processor -- start 2024-09-16T15:44:20.468 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:20.464+0000 7f56c80dc640 1 -- start start 2024-09-16T15:44:20.470 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:20.464+0000 7f56c80dc640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f56ac0052f0 0x7f56c0151490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:44:20.470 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:20.464+0000 7f56c80dc640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f56c00b41b0 0x7f56c00b4630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:44:20.471 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:20.464+0000 7f56c80dc640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f56c00b2a10 0x7f56c00b2e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:44:20.471 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:20.464+0000 7f56c80dc640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f56c00ce4c0 con 0x7f56c00b2a10 2024-09-16T15:44:20.471 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:20.464+0000 7f56c80dc640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f56c00ce340 con 0x7f56c00b41b0 2024-09-16T15:44:20.471 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:20.464+0000 7f56c80dc640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f56c00ce640 con 0x7f56ac0052f0 2024-09-16T15:44:20.471 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:20.464+0000 7f56c6679640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f56c00b2a10 0x7f56c00b2e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:44:20.471 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:20.464+0000 7f56c6679640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f56c00b2a10 0x7f56c00b2e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:47292/0 (socket says 172.21.15.123:47292) 2024-09-16T15:44:20.471 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:20.464+0000 7f56c6679640 1 -- 172.21.15.123:0/1009262550 learned_addr learned my addr 172.21.15.123:0/1009262550 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:44:20.471 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:20.464+0000 7f56c5e78640 1 --2- 172.21.15.123:0/1009262550 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f56ac0052f0 0x7f56c0151490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:44:20.471 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:20.464+0000 7f56c5677640 1 --2- 172.21.15.123:0/1009262550 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f56c00b41b0 0x7f56c00b4630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:44:20.471 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:20.464+0000 7f56c6679640 1 -- 172.21.15.123:0/1009262550 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f56ac0052f0 msgr2=0x7f56c0151490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:44:20.471 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:20.464+0000 7f56c6679640 1 --2- 172.21.15.123:0/1009262550 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f56ac0052f0 0x7f56c0151490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:20.471 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:20.464+0000 7f56c6679640 1 -- 172.21.15.123:0/1009262550 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f56c00b41b0 msgr2=0x7f56c00b4630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:44:20.471 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:20.464+0000 7f56c6679640 1 --2- 172.21.15.123:0/1009262550 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f56c00b41b0 0x7f56c00b4630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:20.472 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:20.464+0000 7f56c6679640 1 -- 172.21.15.123:0/1009262550 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f56c00b4b70 con 0x7f56c00b2a10 2024-09-16T15:44:20.472 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:20.464+0000 7f56c5677640 1 --2- 172.21.15.123:0/1009262550 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f56c00b41b0 0x7f56c00b4630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T15:44:20.472 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:20.464+0000 7f56c6679640 1 --2- 172.21.15.123:0/1009262550 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f56c00b2a10 0x7f56c00b2e50 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f56b801dae0 tx=0x7f56b8006dd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:44:20.472 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:20.464+0000 7f56b67fc640 1 -- 172.21.15.123:0/1009262550 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56b8003980 con 0x7f56c00b2a10 2024-09-16T15:44:20.472 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:20.464+0000 7f56b67fc640 1 -- 172.21.15.123:0/1009262550 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f56b8003c10 con 0x7f56c00b2a10 2024-09-16T15:44:20.472 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:20.464+0000 7f56c80dc640 1 -- 172.21.15.123:0/1009262550 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f56c00b3740 con 0x7f56c00b2a10 2024-09-16T15:44:20.472 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:20.464+0000 7f56b67fc640 1 -- 172.21.15.123:0/1009262550 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56b8006070 con 0x7f56c00b2a10 2024-09-16T15:44:20.472 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:20.464+0000 7f56c80dc640 1 -- 172.21.15.123:0/1009262550 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f56c00b3d00 con 0x7f56c00b2a10 2024-09-16T15:44:20.472 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:20.464+0000 7f56c80dc640 1 -- 172.21.15.123:0/1009262550 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f56c00cb6d0 con 0x7f56c00b2a10 2024-09-16T15:44:20.473 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:20.468+0000 7f56b67fc640 1 -- 172.21.15.123:0/1009262550 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 7) ==== 97233+0+0 (secure 0 0 0) 0x7f56b8006210 con 0x7f56c00b2a10 2024-09-16T15:44:20.473 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:20.468+0000 7f56b67fc640 1 --2- 172.21.15.123:0/1009262550 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f56a40730a0 0x7f56a4075560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:44:20.473 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:20.468+0000 7f56b67fc640 1 -- 172.21.15.123:0/1009262550 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(44..44 src has 1..44) ==== 5709+0+0 (secure 0 0 0) 0x7f56b812dfd0 con 0x7f56c00b2a10 2024-09-16T15:44:20.473 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:20.468+0000 7f56c5e78640 1 --2- 172.21.15.123:0/1009262550 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f56a40730a0 0x7f56a4075560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:44:20.473 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:20.468+0000 7f56c5e78640 1 --2- 172.21.15.123:0/1009262550 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f56a40730a0 0x7f56a4075560 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f56bc0057d0 tx=0x7f56bc005e10 comp rx=0 tx=0).ready entity=mgr.4135 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:44:20.476 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:20.472+0000 7f56b67fc640 1 -- 172.21.15.123:0/1009262550 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f56c00b4b70 con 0x7f56c00b2a10 2024-09-16T15:44:20.704 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:20.700+0000 7f56c80dc640 1 -- 172.21.15.123:0/1009262550 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "status", "format": "json"} v 0) -- 0x7f56c00d09a0 con 0x7f56c00b2a10 2024-09-16T15:44:20.704 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:20.700+0000 7f56b67fc640 1 -- 172.21.15.123:0/1009262550 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json"}]=0 v0) ==== 72+0+1570 (secure 0 0 0) 0x7f56b8016ff0 con 0x7f56c00b2a10 2024-09-16T15:44:20.705 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-09-16T15:44:20.705 INFO:teuthology.orchestra.run.smithi123.stdout:{"fsid":"56077f89-18c8-486f-ae7e-eecc0bb7822c","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":114,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":44,"num_osds":8,"num_up_osds":8,"osd_up_since":1726501358,"num_in_osds":8,"osd_in_since":1726501349,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"unknown","count":64},{"state_name":"creating+peering","count":32},{"state_name":"active+clean","count":1}],"num_pgs":97,"num_pools":3,"num_objects":2,"data_bytes":590368,"bytes_used":230666240,"bytes_avail":772863447040,"bytes_total":773094113280,"unknown_pgs_ratio":0.65979379415512085,"inactive_pgs_ratio":0.32989689707756042},"fsmap":{"epoch":27,"btime":"2024-09-16T15:44:18:509761+0000","id":3,"up":1,"in":1,"max":1,"by_rank":[{"filesystem_id":3,"rank":0,"name":"d","status":"up:active","gid":5280}],"up:standby":3},"mgrmap":{"available":true,"num_standbys":1,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":5,"modified":"2024-09-16T15:44:03.269517+0000","services":{"mds":{"daemons":{"summary":"","b":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"c":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"d":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-09-16T15:44:20.707 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:20.700+0000 7f56c80dc640 1 -- 172.21.15.123:0/1009262550 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f56a40730a0 msgr2=0x7f56a4075560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:44:20.707 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:20.700+0000 7f56c80dc640 1 --2- 172.21.15.123:0/1009262550 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f56a40730a0 0x7f56a4075560 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f56bc0057d0 tx=0x7f56bc005e10 comp rx=0 tx=0).stop 2024-09-16T15:44:20.707 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:20.700+0000 7f56c80dc640 1 -- 172.21.15.123:0/1009262550 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f56c00b2a10 msgr2=0x7f56c00b2e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:44:20.707 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:20.700+0000 7f56c80dc640 1 --2- 172.21.15.123:0/1009262550 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f56c00b2a10 0x7f56c00b2e50 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f56b801dae0 tx=0x7f56b8006dd0 comp rx=0 tx=0).stop 2024-09-16T15:44:20.707 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:20.700+0000 7f56c80dc640 1 -- 172.21.15.123:0/1009262550 shutdown_connections 2024-09-16T15:44:20.708 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:20.700+0000 7f56c80dc640 1 --2- 172.21.15.123:0/1009262550 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f56a40730a0 0x7f56a4075560 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:20.708 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:20.700+0000 7f56c80dc640 1 --2- 172.21.15.123:0/1009262550 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f56c00b2a10 0x7f56c00b2e50 unknown :-1 s=CLOSED pgs=460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:20.708 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:20.700+0000 7f56c80dc640 1 --2- 172.21.15.123:0/1009262550 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f56c00b41b0 0x7f56c00b4630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:20.708 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:20.700+0000 7f56c80dc640 1 --2- 172.21.15.123:0/1009262550 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f56ac0052f0 0x7f56c0151490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:20.708 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:20.700+0000 7f56c80dc640 1 -- 172.21.15.123:0/1009262550 >> 172.21.15.123:0/1009262550 conn(0x7f56c00cfa90 msgr2=0x7f56c00c4fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:44:20.708 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:20.700+0000 7f56c80dc640 1 -- 172.21.15.123:0/1009262550 shutdown_connections 2024-09-16T15:44:20.708 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:20.700+0000 7f56c80dc640 1 -- 172.21.15.123:0/1009262550 wait complete. 2024-09-16T15:44:20.722 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2024-09-16T15:44:20.722 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T15:44:20.837 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:20.832+0000 7fd73fb17640 1 -- 172.21.15.123:0/402332065 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd720005920 msgr2=0x7fd720005da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:44:20.837 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:20.832+0000 7fd73fb17640 1 --2- 172.21.15.123:0/402332065 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd720005920 0x7fd720005da0 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7fd72c00c2e0 tx=0x7fd72c01cea0 comp rx=0 tx=0).stop 2024-09-16T15:44:20.837 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:20.832+0000 7fd73fb17640 1 -- 172.21.15.123:0/402332065 shutdown_connections 2024-09-16T15:44:20.837 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:20.832+0000 7fd73fb17640 1 --2- 172.21.15.123:0/402332065 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd720005920 0x7fd720005da0 unknown :-1 s=CLOSED pgs=462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:20.837 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:20.832+0000 7fd73fb17640 1 -- 172.21.15.123:0/402332065 >> 172.21.15.123:0/402332065 conn(0x7fd7380b1df0 msgr2=0x7fd7380b2200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:44:20.837 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:20.832+0000 7fd73fb17640 1 -- 172.21.15.123:0/402332065 shutdown_connections 2024-09-16T15:44:20.837 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:20.832+0000 7fd73fb17640 1 -- 172.21.15.123:0/402332065 wait complete. 2024-09-16T15:44:20.838 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:20.832+0000 7fd73fb17640 1 Processor -- start 2024-09-16T15:44:20.838 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:20.832+0000 7fd73fb17640 1 -- start start 2024-09-16T15:44:20.840 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:20.832+0000 7fd73fb17640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd720005920 0x7fd73809db80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:44:20.840 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:20.832+0000 7fd73fb17640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd7381ef720 0x7fd7381f59e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:44:20.840 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:20.832+0000 7fd73fb17640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fd7381efba0 0x7fd73815c690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:44:20.840 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:20.832+0000 7fd73fb17640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fd7380ad500 con 0x7fd7381ef720 2024-09-16T15:44:20.840 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:20.832+0000 7fd73fb17640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fd7380ad380 con 0x7fd720005920 2024-09-16T15:44:20.840 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:20.832+0000 7fd73fb17640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fd7380ad680 con 0x7fd7381efba0 2024-09-16T15:44:20.840 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:20.832+0000 7fd73d0b2640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd7381ef720 0x7fd7381f59e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:44:20.840 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:20.832+0000 7fd73d0b2640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd7381ef720 0x7fd7381f59e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:47306/0 (socket says 172.21.15.123:47306) 2024-09-16T15:44:20.840 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:20.832+0000 7fd73d0b2640 1 -- 172.21.15.123:0/3920642515 learned_addr learned my addr 172.21.15.123:0/3920642515 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:44:20.840 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:20.832+0000 7fd73e0b4640 1 --2- 172.21.15.123:0/3920642515 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fd7381efba0 0x7fd73815c690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:44:20.840 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:20.832+0000 7fd73d8b3640 1 --2- 172.21.15.123:0/3920642515 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd720005920 0x7fd73809db80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:44:20.841 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:20.832+0000 7fd73d0b2640 1 -- 172.21.15.123:0/3920642515 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fd7381efba0 msgr2=0x7fd73815c690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:44:20.841 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:20.832+0000 7fd73d0b2640 1 --2- 172.21.15.123:0/3920642515 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fd7381efba0 0x7fd73815c690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:20.841 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:20.832+0000 7fd73d0b2640 1 -- 172.21.15.123:0/3920642515 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd720005920 msgr2=0x7fd73809db80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:44:20.841 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:20.832+0000 7fd73d0b2640 1 --2- 172.21.15.123:0/3920642515 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd720005920 0x7fd73809db80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:20.841 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:20.832+0000 7fd73d0b2640 1 -- 172.21.15.123:0/3920642515 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd738157f60 con 0x7fd7381ef720 2024-09-16T15:44:20.841 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:20.832+0000 7fd73e0b4640 1 --2- 172.21.15.123:0/3920642515 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fd7381efba0 0x7fd73815c690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T15:44:20.841 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:20.832+0000 7fd73d8b3640 1 --2- 172.21.15.123:0/3920642515 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd720005920 0x7fd73809db80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T15:44:20.841 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:20.836+0000 7fd73d0b2640 1 --2- 172.21.15.123:0/3920642515 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd7381ef720 0x7fd7381f59e0 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7fd72c01b5a0 tx=0x7fd72c004c50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:44:20.841 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:20.836+0000 7fd7367fc640 1 -- 172.21.15.123:0/3920642515 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd72c006c60 con 0x7fd7381ef720 2024-09-16T15:44:20.841 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:20.836+0000 7fd7367fc640 1 -- 172.21.15.123:0/3920642515 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fd72c004e40 con 0x7fd7381ef720 2024-09-16T15:44:20.841 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:20.836+0000 7fd7367fc640 1 -- 172.21.15.123:0/3920642515 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd72c001ce0 con 0x7fd7381ef720 2024-09-16T15:44:20.841 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:20.836+0000 7fd73fb17640 1 -- 172.21.15.123:0/3920642515 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd73809e0c0 con 0x7fd7381ef720 2024-09-16T15:44:20.841 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:20.836+0000 7fd73fb17640 1 -- 172.21.15.123:0/3920642515 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd7381f6740 con 0x7fd7381ef720 2024-09-16T15:44:20.842 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:20.836+0000 7fd73fb17640 1 -- 172.21.15.123:0/3920642515 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd7380af8c0 con 0x7fd7381ef720 2024-09-16T15:44:20.842 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:20.836+0000 7fd7367fc640 1 -- 172.21.15.123:0/3920642515 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 7) ==== 97233+0+0 (secure 0 0 0) 0x7fd72c006090 con 0x7fd7381ef720 2024-09-16T15:44:20.842 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:20.836+0000 7fd7367fc640 1 --2- 172.21.15.123:0/3920642515 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fd724073070 0x7fd724075530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:44:20.842 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:20.836+0000 7fd7367fc640 1 -- 172.21.15.123:0/3920642515 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(44..44 src has 1..44) ==== 5709+0+0 (secure 0 0 0) 0x7fd72c12e920 con 0x7fd7381ef720 2024-09-16T15:44:20.842 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:20.836+0000 7fd73d8b3640 1 --2- 172.21.15.123:0/3920642515 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fd724073070 0x7fd724075530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:44:20.843 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:20.836+0000 7fd73d8b3640 1 --2- 172.21.15.123:0/3920642515 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fd724073070 0x7fd724075530 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fd738074c90 tx=0x7fd730006da0 comp rx=0 tx=0).ready entity=mgr.4135 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:44:20.846 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:20.840+0000 7fd7367fc640 1 -- 172.21.15.123:0/3920642515 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7fd72c016d10 con 0x7fd7381ef720 2024-09-16T15:44:21.001 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:20.996+0000 7fd73fb17640 1 -- 172.21.15.123:0/3920642515 --> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fd7380b5f80 con 0x7fd724073070 2024-09-16T15:44:21.012 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:21.008+0000 7fd7367fc640 1 -- 172.21.15.123:0/3920642515 <== mgr.4135 v2:172.21.15.184:6832/3114346423 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+239845 (secure 0 0 0) 0x7fd7380b5f80 con 0x7fd724073070 2024-09-16T15:44:21.014 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-09-16T15:44:21.015 INFO:teuthology.orchestra.run.smithi123.stderr:dumped all 2024-09-16T15:44:21.016 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:21.012+0000 7fd73fb17640 1 -- 172.21.15.123:0/3920642515 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fd724073070 msgr2=0x7fd724075530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:44:21.016 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:21.012+0000 7fd73fb17640 1 --2- 172.21.15.123:0/3920642515 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fd724073070 0x7fd724075530 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fd738074c90 tx=0x7fd730006da0 comp rx=0 tx=0).stop 2024-09-16T15:44:21.016 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:21.012+0000 7fd73fb17640 1 -- 172.21.15.123:0/3920642515 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd7381ef720 msgr2=0x7fd7381f59e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:44:21.017 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:21.012+0000 7fd73fb17640 1 --2- 172.21.15.123:0/3920642515 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd7381ef720 0x7fd7381f59e0 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7fd72c01b5a0 tx=0x7fd72c004c50 comp rx=0 tx=0).stop 2024-09-16T15:44:21.017 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:21.012+0000 7fd73fb17640 1 -- 172.21.15.123:0/3920642515 shutdown_connections 2024-09-16T15:44:21.017 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:21.012+0000 7fd73fb17640 1 --2- 172.21.15.123:0/3920642515 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fd724073070 0x7fd724075530 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:21.017 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:21.012+0000 7fd73fb17640 1 --2- 172.21.15.123:0/3920642515 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fd7381efba0 0x7fd73815c690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:21.017 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:21.012+0000 7fd73fb17640 1 --2- 172.21.15.123:0/3920642515 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd7381ef720 0x7fd7381f59e0 unknown :-1 s=CLOSED pgs=463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:21.017 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:21.012+0000 7fd73fb17640 1 --2- 172.21.15.123:0/3920642515 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd720005920 0x7fd73809db80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:21.017 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:21.012+0000 7fd73fb17640 1 -- 172.21.15.123:0/3920642515 >> 172.21.15.123:0/3920642515 conn(0x7fd7380b1df0 msgr2=0x7fd7380b3ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:44:21.017 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:21.012+0000 7fd73fb17640 1 -- 172.21.15.123:0/3920642515 shutdown_connections 2024-09-16T15:44:21.017 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:21.012+0000 7fd73fb17640 1 -- 172.21.15.123:0/3920642515 wait complete. 2024-09-16T15:44:21.031 INFO:teuthology.orchestra.run.smithi123.stdout:{"pg_ready":true,"pg_map":{"version":87,"stamp":"2024-09-16T15:44:19.274293+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":66,"acting":66,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":9,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":7,"kb":754974720,"kb_used":225260,"kb_used_data":4848,"kb_used_omap":12,"kb_used_meta":220211,"kb_avail":754749460,"statfs":{"total":773094113280,"available":772863447040,"internally_reserved":0,"allocated":4964352,"data_stored":2660670,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":225496625},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":13,"apply_latency_ms":13,"commit_latency_ns":13000000,"apply_latency_ns":13000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.810966"},"pg_stats":[{"pgid":"7.3f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T15:44:14.461845+0000","last_change":"2024-09-16T15:44:14.461845+0000","last_active":"2024-09-16T15:44:14.461845+0000","last_peered":"2024-09-16T15:44:14.461845+0000","last_clean":"2024-09-16T15:44:14.461845+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:44:14.461845+0000","last_undegraded":"2024-09-16T15:44:14.461845+0000","last_fullsized":"2024-09-16T15:44:14.461845+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_clean_scrub_stamp":"2024-09-16T15:44:14.461845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.3e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T15:44:14.461845+0000","last_change":"2024-09-16T15:44:14.461845+0000","last_active":"2024-09-16T15:44:14.461845+0000","last_peered":"2024-09-16T15:44:14.461845+0000","last_clean":"2024-09-16T15:44:14.461845+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:44:14.461845+0000","last_undegraded":"2024-09-16T15:44:14.461845+0000","last_fullsized":"2024-09-16T15:44:14.461845+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_clean_scrub_stamp":"2024-09-16T15:44:14.461845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.3d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T15:44:14.461845+0000","last_change":"2024-09-16T15:44:14.461845+0000","last_active":"2024-09-16T15:44:14.461845+0000","last_peered":"2024-09-16T15:44:14.461845+0000","last_clean":"2024-09-16T15:44:14.461845+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:44:14.461845+0000","last_undegraded":"2024-09-16T15:44:14.461845+0000","last_fullsized":"2024-09-16T15:44:14.461845+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_clean_scrub_stamp":"2024-09-16T15:44:14.461845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.3c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T15:44:14.461845+0000","last_change":"2024-09-16T15:44:14.461845+0000","last_active":"2024-09-16T15:44:14.461845+0000","last_peered":"2024-09-16T15:44:14.461845+0000","last_clean":"2024-09-16T15:44:14.461845+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:44:14.461845+0000","last_undegraded":"2024-09-16T15:44:14.461845+0000","last_fullsized":"2024-09-16T15:44:14.461845+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_clean_scrub_stamp":"2024-09-16T15:44:14.461845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.3b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T15:44:14.461845+0000","last_change":"2024-09-16T15:44:14.461845+0000","last_active":"2024-09-16T15:44:14.461845+0000","last_peered":"2024-09-16T15:44:14.461845+0000","last_clean":"2024-09-16T15:44:14.461845+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:44:14.461845+0000","last_undegraded":"2024-09-16T15:44:14.461845+0000","last_fullsized":"2024-09-16T15:44:14.461845+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_clean_scrub_stamp":"2024-09-16T15:44:14.461845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.3a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T15:44:14.461845+0000","last_change":"2024-09-16T15:44:14.461845+0000","last_active":"2024-09-16T15:44:14.461845+0000","last_peered":"2024-09-16T15:44:14.461845+0000","last_clean":"2024-09-16T15:44:14.461845+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:44:14.461845+0000","last_undegraded":"2024-09-16T15:44:14.461845+0000","last_fullsized":"2024-09-16T15:44:14.461845+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_clean_scrub_stamp":"2024-09-16T15:44:14.461845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.39","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T15:44:14.461845+0000","last_change":"2024-09-16T15:44:14.461845+0000","last_active":"2024-09-16T15:44:14.461845+0000","last_peered":"2024-09-16T15:44:14.461845+0000","last_clean":"2024-09-16T15:44:14.461845+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:44:14.461845+0000","last_undegraded":"2024-09-16T15:44:14.461845+0000","last_fullsized":"2024-09-16T15:44:14.461845+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_clean_scrub_stamp":"2024-09-16T15:44:14.461845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.38","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T15:44:14.461845+0000","last_change":"2024-09-16T15:44:14.461845+0000","last_active":"2024-09-16T15:44:14.461845+0000","last_peered":"2024-09-16T15:44:14.461845+0000","last_clean":"2024-09-16T15:44:14.461845+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:44:14.461845+0000","last_undegraded":"2024-09-16T15:44:14.461845+0000","last_fullsized":"2024-09-16T15:44:14.461845+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_clean_scrub_stamp":"2024-09-16T15:44:14.461845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.37","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T15:44:14.461845+0000","last_change":"2024-09-16T15:44:14.461845+0000","last_active":"2024-09-16T15:44:14.461845+0000","last_peered":"2024-09-16T15:44:14.461845+0000","last_clean":"2024-09-16T15:44:14.461845+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:44:14.461845+0000","last_undegraded":"2024-09-16T15:44:14.461845+0000","last_fullsized":"2024-09-16T15:44:14.461845+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_clean_scrub_stamp":"2024-09-16T15:44:14.461845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.36","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T15:44:14.461845+0000","last_change":"2024-09-16T15:44:14.461845+0000","last_active":"2024-09-16T15:44:14.461845+0000","last_peered":"2024-09-16T15:44:14.461845+0000","last_clean":"2024-09-16T15:44:14.461845+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:44:14.461845+0000","last_undegraded":"2024-09-16T15:44:14.461845+0000","last_fullsized":"2024-09-16T15:44:14.461845+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_clean_scrub_stamp":"2024-09-16T15:44:14.461845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.35","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T15:44:14.461845+0000","last_change":"2024-09-16T15:44:14.461845+0000","last_active":"2024-09-16T15:44:14.461845+0000","last_peered":"2024-09-16T15:44:14.461845+0000","last_clean":"2024-09-16T15:44:14.461845+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:44:14.461845+0000","last_undegraded":"2024-09-16T15:44:14.461845+0000","last_fullsized":"2024-09-16T15:44:14.461845+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_clean_scrub_stamp":"2024-09-16T15:44:14.461845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.34","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T15:44:14.461845+0000","last_change":"2024-09-16T15:44:14.461845+0000","last_active":"2024-09-16T15:44:14.461845+0000","last_peered":"2024-09-16T15:44:14.461845+0000","last_clean":"2024-09-16T15:44:14.461845+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:44:14.461845+0000","last_undegraded":"2024-09-16T15:44:14.461845+0000","last_fullsized":"2024-09-16T15:44:14.461845+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_clean_scrub_stamp":"2024-09-16T15:44:14.461845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.33","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T15:44:14.461845+0000","last_change":"2024-09-16T15:44:14.461845+0000","last_active":"2024-09-16T15:44:14.461845+0000","last_peered":"2024-09-16T15:44:14.461845+0000","last_clean":"2024-09-16T15:44:14.461845+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:44:14.461845+0000","last_undegraded":"2024-09-16T15:44:14.461845+0000","last_fullsized":"2024-09-16T15:44:14.461845+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_clean_scrub_stamp":"2024-09-16T15:44:14.461845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.32","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T15:44:14.461845+0000","last_change":"2024-09-16T15:44:14.461845+0000","last_active":"2024-09-16T15:44:14.461845+0000","last_peered":"2024-09-16T15:44:14.461845+0000","last_clean":"2024-09-16T15:44:14.461845+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:44:14.461845+0000","last_undegraded":"2024-09-16T15:44:14.461845+0000","last_fullsized":"2024-09-16T15:44:14.461845+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_clean_scrub_stamp":"2024-09-16T15:44:14.461845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.31","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T15:44:14.461845+0000","last_change":"2024-09-16T15:44:14.461845+0000","last_active":"2024-09-16T15:44:14.461845+0000","last_peered":"2024-09-16T15:44:14.461845+0000","last_clean":"2024-09-16T15:44:14.461845+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:44:14.461845+0000","last_undegraded":"2024-09-16T15:44:14.461845+0000","last_fullsized":"2024-09-16T15:44:14.461845+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_clean_scrub_stamp":"2024-09-16T15:44:14.461845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.30","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T15:44:14.461845+0000","last_change":"2024-09-16T15:44:14.461845+0000","last_active":"2024-09-16T15:44:14.461845+0000","last_peered":"2024-09-16T15:44:14.461845+0000","last_clean":"2024-09-16T15:44:14.461845+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:44:14.461845+0000","last_undegraded":"2024-09-16T15:44:14.461845+0000","last_fullsized":"2024-09-16T15:44:14.461845+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_clean_scrub_stamp":"2024-09-16T15:44:14.461845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.2f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T15:44:14.461845+0000","last_change":"2024-09-16T15:44:14.461845+0000","last_active":"2024-09-16T15:44:14.461845+0000","last_peered":"2024-09-16T15:44:14.461845+0000","last_clean":"2024-09-16T15:44:14.461845+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:44:14.461845+0000","last_undegraded":"2024-09-16T15:44:14.461845+0000","last_fullsized":"2024-09-16T15:44:14.461845+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_clean_scrub_stamp":"2024-09-16T15:44:14.461845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.2e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T15:44:14.461845+0000","last_change":"2024-09-16T15:44:14.461845+0000","last_active":"2024-09-16T15:44:14.461845+0000","last_peered":"2024-09-16T15:44:14.461845+0000","last_clean":"2024-09-16T15:44:14.461845+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:44:14.461845+0000","last_undegraded":"2024-09-16T15:44:14.461845+0000","last_fullsized":"2024-09-16T15:44:14.461845+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_clean_scrub_stamp":"2024-09-16T15:44:14.461845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.2d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T15:44:14.461845+0000","last_change":"2024-09-16T15:44:14.461845+0000","last_active":"2024-09-16T15:44:14.461845+0000","last_peered":"2024-09-16T15:44:14.461845+0000","last_clean":"2024-09-16T15:44:14.461845+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:44:14.461845+0000","last_undegraded":"2024-09-16T15:44:14.461845+0000","last_fullsized":"2024-09-16T15:44:14.461845+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_clean_scrub_stamp":"2024-09-16T15:44:14.461845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.2c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T15:44:14.461845+0000","last_change":"2024-09-16T15:44:14.461845+0000","last_active":"2024-09-16T15:44:14.461845+0000","last_peered":"2024-09-16T15:44:14.461845+0000","last_clean":"2024-09-16T15:44:14.461845+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:44:14.461845+0000","last_undegraded":"2024-09-16T15:44:14.461845+0000","last_fullsized":"2024-09-16T15:44:14.461845+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_clean_scrub_stamp":"2024-09-16T15:44:14.461845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.2b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T15:44:14.461845+0000","last_change":"2024-09-16T15:44:14.461845+0000","last_active":"2024-09-16T15:44:14.461845+0000","last_peered":"2024-09-16T15:44:14.461845+0000","last_clean":"2024-09-16T15:44:14.461845+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:44:14.461845+0000","last_undegraded":"2024-09-16T15:44:14.461845+0000","last_fullsized":"2024-09-16T15:44:14.461845+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_clean_scrub_stamp":"2024-09-16T15:44:14.461845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.2a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T15:44:14.461845+0000","last_change":"2024-09-16T15:44:14.461845+0000","last_active":"2024-09-16T15:44:14.461845+0000","last_peered":"2024-09-16T15:44:14.461845+0000","last_clean":"2024-09-16T15:44:14.461845+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:44:14.461845+0000","last_undegraded":"2024-09-16T15:44:14.461845+0000","last_fullsized":"2024-09-16T15:44:14.461845+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_clean_scrub_stamp":"2024-09-16T15:44:14.461845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.29","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T15:44:14.461845+0000","last_change":"2024-09-16T15:44:14.461845+0000","last_active":"2024-09-16T15:44:14.461845+0000","last_peered":"2024-09-16T15:44:14.461845+0000","last_clean":"2024-09-16T15:44:14.461845+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:44:14.461845+0000","last_undegraded":"2024-09-16T15:44:14.461845+0000","last_fullsized":"2024-09-16T15:44:14.461845+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_clean_scrub_stamp":"2024-09-16T15:44:14.461845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.28","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T15:44:14.461845+0000","last_change":"2024-09-16T15:44:14.461845+0000","last_active":"2024-09-16T15:44:14.461845+0000","last_peered":"2024-09-16T15:44:14.461845+0000","last_clean":"2024-09-16T15:44:14.461845+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:44:14.461845+0000","last_undegraded":"2024-09-16T15:44:14.461845+0000","last_fullsized":"2024-09-16T15:44:14.461845+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_clean_scrub_stamp":"2024-09-16T15:44:14.461845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.27","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T15:44:14.461845+0000","last_change":"2024-09-16T15:44:14.461845+0000","last_active":"2024-09-16T15:44:14.461845+0000","last_peered":"2024-09-16T15:44:14.461845+0000","last_clean":"2024-09-16T15:44:14.461845+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:44:14.461845+0000","last_undegraded":"2024-09-16T15:44:14.461845+0000","last_fullsized":"2024-09-16T15:44:14.461845+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_clean_scrub_stamp":"2024-09-16T15:44:14.461845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.26","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T15:44:14.461845+0000","last_change":"2024-09-16T15:44:14.461845+0000","last_active":"2024-09-16T15:44:14.461845+0000","last_peered":"2024-09-16T15:44:14.461845+0000","last_clean":"2024-09-16T15:44:14.461845+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:44:14.461845+0000","last_undegraded":"2024-09-16T15:44:14.461845+0000","last_fullsized":"2024-09-16T15:44:14.461845+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_clean_scrub_stamp":"2024-09-16T15:44:14.461845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.25","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T15:44:14.461845+0000","last_change":"2024-09-16T15:44:14.461845+0000","last_active":"2024-09-16T15:44:14.461845+0000","last_peered":"2024-09-16T15:44:14.461845+0000","last_clean":"2024-09-16T15:44:14.461845+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:44:14.461845+0000","last_undegraded":"2024-09-16T15:44:14.461845+0000","last_fullsized":"2024-09-16T15:44:14.461845+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_clean_scrub_stamp":"2024-09-16T15:44:14.461845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.24","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T15:44:14.461845+0000","last_change":"2024-09-16T15:44:14.461845+0000","last_active":"2024-09-16T15:44:14.461845+0000","last_peered":"2024-09-16T15:44:14.461845+0000","last_clean":"2024-09-16T15:44:14.461845+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:44:14.461845+0000","last_undegraded":"2024-09-16T15:44:14.461845+0000","last_fullsized":"2024-09-16T15:44:14.461845+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_clean_scrub_stamp":"2024-09-16T15:44:14.461845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.23","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T15:44:14.461845+0000","last_change":"2024-09-16T15:44:14.461845+0000","last_active":"2024-09-16T15:44:14.461845+0000","last_peered":"2024-09-16T15:44:14.461845+0000","last_clean":"2024-09-16T15:44:14.461845+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:44:14.461845+0000","last_undegraded":"2024-09-16T15:44:14.461845+0000","last_fullsized":"2024-09-16T15:44:14.461845+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_clean_scrub_stamp":"2024-09-16T15:44:14.461845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.22","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T15:44:14.461845+0000","last_change":"2024-09-16T15:44:14.461845+0000","last_active":"2024-09-16T15:44:14.461845+0000","last_peered":"2024-09-16T15:44:14.461845+0000","last_clean":"2024-09-16T15:44:14.461845+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:44:14.461845+0000","last_undegraded":"2024-09-16T15:44:14.461845+0000","last_fullsized":"2024-09-16T15:44:14.461845+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_clean_scrub_stamp":"2024-09-16T15:44:14.461845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.21","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T15:44:14.461845+0000","last_change":"2024-09-16T15:44:14.461845+0000","last_active":"2024-09-16T15:44:14.461845+0000","last_peered":"2024-09-16T15:44:14.461845+0000","last_clean":"2024-09-16T15:44:14.461845+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:44:14.461845+0000","last_undegraded":"2024-09-16T15:44:14.461845+0000","last_fullsized":"2024-09-16T15:44:14.461845+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_clean_scrub_stamp":"2024-09-16T15:44:14.461845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.20","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T15:44:14.461845+0000","last_change":"2024-09-16T15:44:14.461845+0000","last_active":"2024-09-16T15:44:14.461845+0000","last_peered":"2024-09-16T15:44:14.461845+0000","last_clean":"2024-09-16T15:44:14.461845+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:44:14.461845+0000","last_undegraded":"2024-09-16T15:44:14.461845+0000","last_fullsized":"2024-09-16T15:44:14.461845+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_clean_scrub_stamp":"2024-09-16T15:44:14.461845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T15:44:14.461845+0000","last_change":"2024-09-16T15:44:14.461845+0000","last_active":"2024-09-16T15:44:14.461845+0000","last_peered":"2024-09-16T15:44:14.461845+0000","last_clean":"2024-09-16T15:44:14.461845+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:44:14.461845+0000","last_undegraded":"2024-09-16T15:44:14.461845+0000","last_fullsized":"2024-09-16T15:44:14.461845+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_clean_scrub_stamp":"2024-09-16T15:44:14.461845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":1,"reported_epoch":41,"state":"creating+peering","last_fresh":"2024-09-16T15:44:13.470227+0000","last_change":"2024-09-16T15:44:13.470227+0000","last_active":"2024-09-16T15:44:13.453120+0000","last_peered":"2024-09-16T15:44:13.453120+0000","last_clean":"2024-09-16T15:44:13.453120+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:44:13.470227+0000","last_undegraded":"2024-09-16T15:44:13.470227+0000","last_fullsized":"2024-09-16T15:44:13.470227+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:13.453120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:13.453120+0000","last_clean_scrub_stamp":"2024-09-16T15:44:13.453120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T15:44:14.461845+0000","last_change":"2024-09-16T15:44:14.461845+0000","last_active":"2024-09-16T15:44:14.461845+0000","last_peered":"2024-09-16T15:44:14.461845+0000","last_clean":"2024-09-16T15:44:14.461845+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:44:14.461845+0000","last_undegraded":"2024-09-16T15:44:14.461845+0000","last_fullsized":"2024-09-16T15:44:14.461845+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_clean_scrub_stamp":"2024-09-16T15:44:14.461845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":1,"reported_epoch":41,"state":"creating+peering","last_fresh":"2024-09-16T15:44:13.469543+0000","last_change":"2024-09-16T15:44:13.469543+0000","last_active":"2024-09-16T15:44:13.453120+0000","last_peered":"2024-09-16T15:44:13.453120+0000","last_clean":"2024-09-16T15:44:13.453120+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:44:13.469543+0000","last_undegraded":"2024-09-16T15:44:13.469543+0000","last_fullsized":"2024-09-16T15:44:13.469543+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:13.453120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:13.453120+0000","last_clean_scrub_stamp":"2024-09-16T15:44:13.453120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T15:44:14.461845+0000","last_change":"2024-09-16T15:44:14.461845+0000","last_active":"2024-09-16T15:44:14.461845+0000","last_peered":"2024-09-16T15:44:14.461845+0000","last_clean":"2024-09-16T15:44:14.461845+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:44:14.461845+0000","last_undegraded":"2024-09-16T15:44:14.461845+0000","last_fullsized":"2024-09-16T15:44:14.461845+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_clean_scrub_stamp":"2024-09-16T15:44:14.461845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":1,"reported_epoch":41,"state":"creating+peering","last_fresh":"2024-09-16T15:44:13.472025+0000","last_change":"2024-09-16T15:44:13.472025+0000","last_active":"2024-09-16T15:44:13.453120+0000","last_peered":"2024-09-16T15:44:13.453120+0000","last_clean":"2024-09-16T15:44:13.453120+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:44:13.472025+0000","last_undegraded":"2024-09-16T15:44:13.472025+0000","last_fullsized":"2024-09-16T15:44:13.472025+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:13.453120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:13.453120+0000","last_clean_scrub_stamp":"2024-09-16T15:44:13.453120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T15:44:14.461845+0000","last_change":"2024-09-16T15:44:14.461845+0000","last_active":"2024-09-16T15:44:14.461845+0000","last_peered":"2024-09-16T15:44:14.461845+0000","last_clean":"2024-09-16T15:44:14.461845+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:44:14.461845+0000","last_undegraded":"2024-09-16T15:44:14.461845+0000","last_fullsized":"2024-09-16T15:44:14.461845+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_clean_scrub_stamp":"2024-09-16T15:44:14.461845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.1c","version":"0'0","reported_seq":1,"reported_epoch":41,"state":"creating+peering","last_fresh":"2024-09-16T15:44:13.475916+0000","last_change":"2024-09-16T15:44:13.475916+0000","last_active":"2024-09-16T15:44:13.453120+0000","last_peered":"2024-09-16T15:44:13.453120+0000","last_clean":"2024-09-16T15:44:13.453120+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:44:13.475916+0000","last_undegraded":"2024-09-16T15:44:13.475916+0000","last_fullsized":"2024-09-16T15:44:13.475916+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:13.453120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:13.453120+0000","last_clean_scrub_stamp":"2024-09-16T15:44:13.453120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T15:44:14.461845+0000","last_change":"2024-09-16T15:44:14.461845+0000","last_active":"2024-09-16T15:44:14.461845+0000","last_peered":"2024-09-16T15:44:14.461845+0000","last_clean":"2024-09-16T15:44:14.461845+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:44:14.461845+0000","last_undegraded":"2024-09-16T15:44:14.461845+0000","last_fullsized":"2024-09-16T15:44:14.461845+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_clean_scrub_stamp":"2024-09-16T15:44:14.461845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":1,"reported_epoch":41,"state":"creating+peering","last_fresh":"2024-09-16T15:44:13.470521+0000","last_change":"2024-09-16T15:44:13.470521+0000","last_active":"2024-09-16T15:44:13.453120+0000","last_peered":"2024-09-16T15:44:13.453120+0000","last_clean":"2024-09-16T15:44:13.453120+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:44:13.470521+0000","last_undegraded":"2024-09-16T15:44:13.470521+0000","last_fullsized":"2024-09-16T15:44:13.470521+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:13.453120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:13.453120+0000","last_clean_scrub_stamp":"2024-09-16T15:44:13.453120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T15:44:14.461845+0000","last_change":"2024-09-16T15:44:14.461845+0000","last_active":"2024-09-16T15:44:14.461845+0000","last_peered":"2024-09-16T15:44:14.461845+0000","last_clean":"2024-09-16T15:44:14.461845+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:44:14.461845+0000","last_undegraded":"2024-09-16T15:44:14.461845+0000","last_fullsized":"2024-09-16T15:44:14.461845+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_clean_scrub_stamp":"2024-09-16T15:44:14.461845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":1,"reported_epoch":41,"state":"creating+peering","last_fresh":"2024-09-16T15:44:13.468698+0000","last_change":"2024-09-16T15:44:13.468698+0000","last_active":"2024-09-16T15:44:13.453120+0000","last_peered":"2024-09-16T15:44:13.453120+0000","last_clean":"2024-09-16T15:44:13.453120+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:44:13.468698+0000","last_undegraded":"2024-09-16T15:44:13.468698+0000","last_fullsized":"2024-09-16T15:44:13.468698+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:13.453120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:13.453120+0000","last_clean_scrub_stamp":"2024-09-16T15:44:13.453120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T15:44:14.461845+0000","last_change":"2024-09-16T15:44:14.461845+0000","last_active":"2024-09-16T15:44:14.461845+0000","last_peered":"2024-09-16T15:44:14.461845+0000","last_clean":"2024-09-16T15:44:14.461845+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:44:14.461845+0000","last_undegraded":"2024-09-16T15:44:14.461845+0000","last_fullsized":"2024-09-16T15:44:14.461845+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_clean_scrub_stamp":"2024-09-16T15:44:14.461845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":1,"reported_epoch":41,"state":"creating+peering","last_fresh":"2024-09-16T15:44:13.478336+0000","last_change":"2024-09-16T15:44:13.478336+0000","last_active":"2024-09-16T15:44:13.453120+0000","last_peered":"2024-09-16T15:44:13.453120+0000","last_clean":"2024-09-16T15:44:13.453120+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:44:13.478336+0000","last_undegraded":"2024-09-16T15:44:13.478336+0000","last_fullsized":"2024-09-16T15:44:13.478336+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:13.453120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:13.453120+0000","last_clean_scrub_stamp":"2024-09-16T15:44:13.453120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T15:44:14.461845+0000","last_change":"2024-09-16T15:44:14.461845+0000","last_active":"2024-09-16T15:44:14.461845+0000","last_peered":"2024-09-16T15:44:14.461845+0000","last_clean":"2024-09-16T15:44:14.461845+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:44:14.461845+0000","last_undegraded":"2024-09-16T15:44:14.461845+0000","last_fullsized":"2024-09-16T15:44:14.461845+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_clean_scrub_stamp":"2024-09-16T15:44:14.461845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":1,"reported_epoch":41,"state":"creating+peering","last_fresh":"2024-09-16T15:44:13.475717+0000","last_change":"2024-09-16T15:44:13.475717+0000","last_active":"2024-09-16T15:44:13.453120+0000","last_peered":"2024-09-16T15:44:13.453120+0000","last_clean":"2024-09-16T15:44:13.453120+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:44:13.475717+0000","last_undegraded":"2024-09-16T15:44:13.475717+0000","last_fullsized":"2024-09-16T15:44:13.475717+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:13.453120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:13.453120+0000","last_clean_scrub_stamp":"2024-09-16T15:44:13.453120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T15:44:14.461845+0000","last_change":"2024-09-16T15:44:14.461845+0000","last_active":"2024-09-16T15:44:14.461845+0000","last_peered":"2024-09-16T15:44:14.461845+0000","last_clean":"2024-09-16T15:44:14.461845+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:44:14.461845+0000","last_undegraded":"2024-09-16T15:44:14.461845+0000","last_fullsized":"2024-09-16T15:44:14.461845+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_clean_scrub_stamp":"2024-09-16T15:44:14.461845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":1,"reported_epoch":41,"state":"creating+peering","last_fresh":"2024-09-16T15:44:13.468269+0000","last_change":"2024-09-16T15:44:13.468269+0000","last_active":"2024-09-16T15:44:13.453120+0000","last_peered":"2024-09-16T15:44:13.453120+0000","last_clean":"2024-09-16T15:44:13.453120+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:44:13.468269+0000","last_undegraded":"2024-09-16T15:44:13.468269+0000","last_fullsized":"2024-09-16T15:44:13.468269+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:13.453120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:13.453120+0000","last_clean_scrub_stamp":"2024-09-16T15:44:13.453120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T15:44:14.461845+0000","last_change":"2024-09-16T15:44:14.461845+0000","last_active":"2024-09-16T15:44:14.461845+0000","last_peered":"2024-09-16T15:44:14.461845+0000","last_clean":"2024-09-16T15:44:14.461845+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:44:14.461845+0000","last_undegraded":"2024-09-16T15:44:14.461845+0000","last_fullsized":"2024-09-16T15:44:14.461845+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_clean_scrub_stamp":"2024-09-16T15:44:14.461845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":1,"reported_epoch":41,"state":"creating+peering","last_fresh":"2024-09-16T15:44:13.473942+0000","last_change":"2024-09-16T15:44:13.473942+0000","last_active":"2024-09-16T15:44:13.453120+0000","last_peered":"2024-09-16T15:44:13.453120+0000","last_clean":"2024-09-16T15:44:13.453120+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:44:13.473942+0000","last_undegraded":"2024-09-16T15:44:13.473942+0000","last_fullsized":"2024-09-16T15:44:13.473942+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:13.453120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:13.453120+0000","last_clean_scrub_stamp":"2024-09-16T15:44:13.453120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T15:44:14.461845+0000","last_change":"2024-09-16T15:44:14.461845+0000","last_active":"2024-09-16T15:44:14.461845+0000","last_peered":"2024-09-16T15:44:14.461845+0000","last_clean":"2024-09-16T15:44:14.461845+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:44:14.461845+0000","last_undegraded":"2024-09-16T15:44:14.461845+0000","last_fullsized":"2024-09-16T15:44:14.461845+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_clean_scrub_stamp":"2024-09-16T15:44:14.461845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":1,"reported_epoch":41,"state":"creating+peering","last_fresh":"2024-09-16T15:44:13.476299+0000","last_change":"2024-09-16T15:44:13.476299+0000","last_active":"2024-09-16T15:44:13.453120+0000","last_peered":"2024-09-16T15:44:13.453120+0000","last_clean":"2024-09-16T15:44:13.453120+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:44:13.476299+0000","last_undegraded":"2024-09-16T15:44:13.476299+0000","last_fullsized":"2024-09-16T15:44:13.476299+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:13.453120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:13.453120+0000","last_clean_scrub_stamp":"2024-09-16T15:44:13.453120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T15:44:14.461845+0000","last_change":"2024-09-16T15:44:14.461845+0000","last_active":"2024-09-16T15:44:14.461845+0000","last_peered":"2024-09-16T15:44:14.461845+0000","last_clean":"2024-09-16T15:44:14.461845+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:44:14.461845+0000","last_undegraded":"2024-09-16T15:44:14.461845+0000","last_fullsized":"2024-09-16T15:44:14.461845+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_clean_scrub_stamp":"2024-09-16T15:44:14.461845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":1,"reported_epoch":41,"state":"creating+peering","last_fresh":"2024-09-16T15:44:13.470832+0000","last_change":"2024-09-16T15:44:13.470832+0000","last_active":"2024-09-16T15:44:13.453120+0000","last_peered":"2024-09-16T15:44:13.453120+0000","last_clean":"2024-09-16T15:44:13.453120+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:44:13.470832+0000","last_undegraded":"2024-09-16T15:44:13.470832+0000","last_fullsized":"2024-09-16T15:44:13.470832+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:13.453120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:13.453120+0000","last_clean_scrub_stamp":"2024-09-16T15:44:13.453120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T15:44:14.461845+0000","last_change":"2024-09-16T15:44:14.461845+0000","last_active":"2024-09-16T15:44:14.461845+0000","last_peered":"2024-09-16T15:44:14.461845+0000","last_clean":"2024-09-16T15:44:14.461845+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:44:14.461845+0000","last_undegraded":"2024-09-16T15:44:14.461845+0000","last_fullsized":"2024-09-16T15:44:14.461845+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_clean_scrub_stamp":"2024-09-16T15:44:14.461845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":1,"reported_epoch":41,"state":"creating+peering","last_fresh":"2024-09-16T15:44:13.469026+0000","last_change":"2024-09-16T15:44:13.469026+0000","last_active":"2024-09-16T15:44:13.453120+0000","last_peered":"2024-09-16T15:44:13.453120+0000","last_clean":"2024-09-16T15:44:13.453120+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:44:13.469026+0000","last_undegraded":"2024-09-16T15:44:13.469026+0000","last_fullsized":"2024-09-16T15:44:13.469026+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:13.453120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:13.453120+0000","last_clean_scrub_stamp":"2024-09-16T15:44:13.453120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T15:44:14.461845+0000","last_change":"2024-09-16T15:44:14.461845+0000","last_active":"2024-09-16T15:44:14.461845+0000","last_peered":"2024-09-16T15:44:14.461845+0000","last_clean":"2024-09-16T15:44:14.461845+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:44:14.461845+0000","last_undegraded":"2024-09-16T15:44:14.461845+0000","last_fullsized":"2024-09-16T15:44:14.461845+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_clean_scrub_stamp":"2024-09-16T15:44:14.461845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":1,"reported_epoch":41,"state":"creating+peering","last_fresh":"2024-09-16T15:44:13.474765+0000","last_change":"2024-09-16T15:44:13.474765+0000","last_active":"2024-09-16T15:44:13.453120+0000","last_peered":"2024-09-16T15:44:13.453120+0000","last_clean":"2024-09-16T15:44:13.453120+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:44:13.474765+0000","last_undegraded":"2024-09-16T15:44:13.474765+0000","last_fullsized":"2024-09-16T15:44:13.474765+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:13.453120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:13.453120+0000","last_clean_scrub_stamp":"2024-09-16T15:44:13.453120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T15:44:14.461845+0000","last_change":"2024-09-16T15:44:14.461845+0000","last_active":"2024-09-16T15:44:14.461845+0000","last_peered":"2024-09-16T15:44:14.461845+0000","last_clean":"2024-09-16T15:44:14.461845+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:44:14.461845+0000","last_undegraded":"2024-09-16T15:44:14.461845+0000","last_fullsized":"2024-09-16T15:44:14.461845+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_clean_scrub_stamp":"2024-09-16T15:44:14.461845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":1,"reported_epoch":41,"state":"creating+peering","last_fresh":"2024-09-16T15:44:13.468318+0000","last_change":"2024-09-16T15:44:13.468318+0000","last_active":"2024-09-16T15:44:13.453120+0000","last_peered":"2024-09-16T15:44:13.453120+0000","last_clean":"2024-09-16T15:44:13.453120+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:44:13.468318+0000","last_undegraded":"2024-09-16T15:44:13.468318+0000","last_fullsized":"2024-09-16T15:44:13.468318+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:13.453120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:13.453120+0000","last_clean_scrub_stamp":"2024-09-16T15:44:13.453120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T15:44:14.461845+0000","last_change":"2024-09-16T15:44:14.461845+0000","last_active":"2024-09-16T15:44:14.461845+0000","last_peered":"2024-09-16T15:44:14.461845+0000","last_clean":"2024-09-16T15:44:14.461845+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:44:14.461845+0000","last_undegraded":"2024-09-16T15:44:14.461845+0000","last_fullsized":"2024-09-16T15:44:14.461845+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_clean_scrub_stamp":"2024-09-16T15:44:14.461845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":1,"reported_epoch":41,"state":"creating+peering","last_fresh":"2024-09-16T15:44:13.473996+0000","last_change":"2024-09-16T15:44:13.473996+0000","last_active":"2024-09-16T15:44:13.453120+0000","last_peered":"2024-09-16T15:44:13.453120+0000","last_clean":"2024-09-16T15:44:13.453120+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:44:13.473996+0000","last_undegraded":"2024-09-16T15:44:13.473996+0000","last_fullsized":"2024-09-16T15:44:13.473996+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:13.453120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:13.453120+0000","last_clean_scrub_stamp":"2024-09-16T15:44:13.453120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T15:44:14.461845+0000","last_change":"2024-09-16T15:44:14.461845+0000","last_active":"2024-09-16T15:44:14.461845+0000","last_peered":"2024-09-16T15:44:14.461845+0000","last_clean":"2024-09-16T15:44:14.461845+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:44:14.461845+0000","last_undegraded":"2024-09-16T15:44:14.461845+0000","last_fullsized":"2024-09-16T15:44:14.461845+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_clean_scrub_stamp":"2024-09-16T15:44:14.461845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":1,"reported_epoch":41,"state":"creating+peering","last_fresh":"2024-09-16T15:44:13.471259+0000","last_change":"2024-09-16T15:44:13.471259+0000","last_active":"2024-09-16T15:44:13.453120+0000","last_peered":"2024-09-16T15:44:13.453120+0000","last_clean":"2024-09-16T15:44:13.453120+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:44:13.471259+0000","last_undegraded":"2024-09-16T15:44:13.471259+0000","last_fullsized":"2024-09-16T15:44:13.471259+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:13.453120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:13.453120+0000","last_clean_scrub_stamp":"2024-09-16T15:44:13.453120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T15:44:14.461845+0000","last_change":"2024-09-16T15:44:14.461845+0000","last_active":"2024-09-16T15:44:14.461845+0000","last_peered":"2024-09-16T15:44:14.461845+0000","last_clean":"2024-09-16T15:44:14.461845+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:44:14.461845+0000","last_undegraded":"2024-09-16T15:44:14.461845+0000","last_fullsized":"2024-09-16T15:44:14.461845+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_clean_scrub_stamp":"2024-09-16T15:44:14.461845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":1,"reported_epoch":41,"state":"creating+peering","last_fresh":"2024-09-16T15:44:13.468343+0000","last_change":"2024-09-16T15:44:13.468343+0000","last_active":"2024-09-16T15:44:13.453120+0000","last_peered":"2024-09-16T15:44:13.453120+0000","last_clean":"2024-09-16T15:44:13.453120+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:44:13.468343+0000","last_undegraded":"2024-09-16T15:44:13.468343+0000","last_fullsized":"2024-09-16T15:44:13.468343+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:13.453120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:13.453120+0000","last_clean_scrub_stamp":"2024-09-16T15:44:13.453120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T15:44:14.461845+0000","last_change":"2024-09-16T15:44:14.461845+0000","last_active":"2024-09-16T15:44:14.461845+0000","last_peered":"2024-09-16T15:44:14.461845+0000","last_clean":"2024-09-16T15:44:14.461845+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:44:14.461845+0000","last_undegraded":"2024-09-16T15:44:14.461845+0000","last_fullsized":"2024-09-16T15:44:14.461845+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_clean_scrub_stamp":"2024-09-16T15:44:14.461845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":1,"reported_epoch":41,"state":"creating+peering","last_fresh":"2024-09-16T15:44:13.478680+0000","last_change":"2024-09-16T15:44:13.478680+0000","last_active":"2024-09-16T15:44:13.453120+0000","last_peered":"2024-09-16T15:44:13.453120+0000","last_clean":"2024-09-16T15:44:13.453120+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:44:13.478680+0000","last_undegraded":"2024-09-16T15:44:13.478680+0000","last_fullsized":"2024-09-16T15:44:13.478680+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:13.453120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:13.453120+0000","last_clean_scrub_stamp":"2024-09-16T15:44:13.453120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T15:44:14.461845+0000","last_change":"2024-09-16T15:44:14.461845+0000","last_active":"2024-09-16T15:44:14.461845+0000","last_peered":"2024-09-16T15:44:14.461845+0000","last_clean":"2024-09-16T15:44:14.461845+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:44:14.461845+0000","last_undegraded":"2024-09-16T15:44:14.461845+0000","last_fullsized":"2024-09-16T15:44:14.461845+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_clean_scrub_stamp":"2024-09-16T15:44:14.461845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":1,"reported_epoch":41,"state":"creating+peering","last_fresh":"2024-09-16T15:44:13.468140+0000","last_change":"2024-09-16T15:44:13.468140+0000","last_active":"2024-09-16T15:44:13.453120+0000","last_peered":"2024-09-16T15:44:13.453120+0000","last_clean":"2024-09-16T15:44:13.453120+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:44:13.468140+0000","last_undegraded":"2024-09-16T15:44:13.468140+0000","last_fullsized":"2024-09-16T15:44:13.468140+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:13.453120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:13.453120+0000","last_clean_scrub_stamp":"2024-09-16T15:44:13.453120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T15:44:14.461845+0000","last_change":"2024-09-16T15:44:14.461845+0000","last_active":"2024-09-16T15:44:14.461845+0000","last_peered":"2024-09-16T15:44:14.461845+0000","last_clean":"2024-09-16T15:44:14.461845+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:44:14.461845+0000","last_undegraded":"2024-09-16T15:44:14.461845+0000","last_fullsized":"2024-09-16T15:44:14.461845+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_clean_scrub_stamp":"2024-09-16T15:44:14.461845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":1,"reported_epoch":41,"state":"creating+peering","last_fresh":"2024-09-16T15:44:13.468299+0000","last_change":"2024-09-16T15:44:13.468299+0000","last_active":"2024-09-16T15:44:13.453120+0000","last_peered":"2024-09-16T15:44:13.453120+0000","last_clean":"2024-09-16T15:44:13.453120+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:44:13.468299+0000","last_undegraded":"2024-09-16T15:44:13.468299+0000","last_fullsized":"2024-09-16T15:44:13.468299+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:13.453120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:13.453120+0000","last_clean_scrub_stamp":"2024-09-16T15:44:13.453120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T15:44:14.461845+0000","last_change":"2024-09-16T15:44:14.461845+0000","last_active":"2024-09-16T15:44:14.461845+0000","last_peered":"2024-09-16T15:44:14.461845+0000","last_clean":"2024-09-16T15:44:14.461845+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:44:14.461845+0000","last_undegraded":"2024-09-16T15:44:14.461845+0000","last_fullsized":"2024-09-16T15:44:14.461845+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_clean_scrub_stamp":"2024-09-16T15:44:14.461845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":1,"reported_epoch":41,"state":"creating+peering","last_fresh":"2024-09-16T15:44:13.473999+0000","last_change":"2024-09-16T15:44:13.473999+0000","last_active":"2024-09-16T15:44:13.453120+0000","last_peered":"2024-09-16T15:44:13.453120+0000","last_clean":"2024-09-16T15:44:13.453120+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:44:13.473999+0000","last_undegraded":"2024-09-16T15:44:13.473999+0000","last_fullsized":"2024-09-16T15:44:13.473999+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:13.453120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:13.453120+0000","last_clean_scrub_stamp":"2024-09-16T15:44:13.453120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T15:44:14.461845+0000","last_change":"2024-09-16T15:44:14.461845+0000","last_active":"2024-09-16T15:44:14.461845+0000","last_peered":"2024-09-16T15:44:14.461845+0000","last_clean":"2024-09-16T15:44:14.461845+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:44:14.461845+0000","last_undegraded":"2024-09-16T15:44:14.461845+0000","last_fullsized":"2024-09-16T15:44:14.461845+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_clean_scrub_stamp":"2024-09-16T15:44:14.461845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":1,"reported_epoch":41,"state":"creating+peering","last_fresh":"2024-09-16T15:44:13.473367+0000","last_change":"2024-09-16T15:44:13.473367+0000","last_active":"2024-09-16T15:44:13.453120+0000","last_peered":"2024-09-16T15:44:13.453120+0000","last_clean":"2024-09-16T15:44:13.453120+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:44:13.473367+0000","last_undegraded":"2024-09-16T15:44:13.473367+0000","last_fullsized":"2024-09-16T15:44:13.473367+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:13.453120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:13.453120+0000","last_clean_scrub_stamp":"2024-09-16T15:44:13.453120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T15:44:14.461845+0000","last_change":"2024-09-16T15:44:14.461845+0000","last_active":"2024-09-16T15:44:14.461845+0000","last_peered":"2024-09-16T15:44:14.461845+0000","last_clean":"2024-09-16T15:44:14.461845+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:44:14.461845+0000","last_undegraded":"2024-09-16T15:44:14.461845+0000","last_fullsized":"2024-09-16T15:44:14.461845+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_clean_scrub_stamp":"2024-09-16T15:44:14.461845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":1,"reported_epoch":41,"state":"creating+peering","last_fresh":"2024-09-16T15:44:13.474597+0000","last_change":"2024-09-16T15:44:13.474597+0000","last_active":"2024-09-16T15:44:13.453120+0000","last_peered":"2024-09-16T15:44:13.453120+0000","last_clean":"2024-09-16T15:44:13.453120+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:44:13.474597+0000","last_undegraded":"2024-09-16T15:44:13.474597+0000","last_fullsized":"2024-09-16T15:44:13.474597+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:13.453120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:13.453120+0000","last_clean_scrub_stamp":"2024-09-16T15:44:13.453120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T15:44:14.461845+0000","last_change":"2024-09-16T15:44:14.461845+0000","last_active":"2024-09-16T15:44:14.461845+0000","last_peered":"2024-09-16T15:44:14.461845+0000","last_clean":"2024-09-16T15:44:14.461845+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:44:14.461845+0000","last_undegraded":"2024-09-16T15:44:14.461845+0000","last_fullsized":"2024-09-16T15:44:14.461845+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_clean_scrub_stamp":"2024-09-16T15:44:14.461845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":1,"reported_epoch":41,"state":"creating+peering","last_fresh":"2024-09-16T15:44:13.468102+0000","last_change":"2024-09-16T15:44:13.468102+0000","last_active":"2024-09-16T15:44:13.453120+0000","last_peered":"2024-09-16T15:44:13.453120+0000","last_clean":"2024-09-16T15:44:13.453120+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:44:13.468102+0000","last_undegraded":"2024-09-16T15:44:13.468102+0000","last_fullsized":"2024-09-16T15:44:13.468102+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:13.453120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:13.453120+0000","last_clean_scrub_stamp":"2024-09-16T15:44:13.453120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T15:44:14.461845+0000","last_change":"2024-09-16T15:44:14.461845+0000","last_active":"2024-09-16T15:44:14.461845+0000","last_peered":"2024-09-16T15:44:14.461845+0000","last_clean":"2024-09-16T15:44:14.461845+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:44:14.461845+0000","last_undegraded":"2024-09-16T15:44:14.461845+0000","last_fullsized":"2024-09-16T15:44:14.461845+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_clean_scrub_stamp":"2024-09-16T15:44:14.461845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":1,"reported_epoch":41,"state":"creating+peering","last_fresh":"2024-09-16T15:44:13.474821+0000","last_change":"2024-09-16T15:44:13.474821+0000","last_active":"2024-09-16T15:44:13.453120+0000","last_peered":"2024-09-16T15:44:13.453120+0000","last_clean":"2024-09-16T15:44:13.453120+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:44:13.474821+0000","last_undegraded":"2024-09-16T15:44:13.474821+0000","last_fullsized":"2024-09-16T15:44:13.474821+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:13.453120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:13.453120+0000","last_clean_scrub_stamp":"2024-09-16T15:44:13.453120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T15:44:14.461845+0000","last_change":"2024-09-16T15:44:14.461845+0000","last_active":"2024-09-16T15:44:14.461845+0000","last_peered":"2024-09-16T15:44:14.461845+0000","last_clean":"2024-09-16T15:44:14.461845+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:44:14.461845+0000","last_undegraded":"2024-09-16T15:44:14.461845+0000","last_fullsized":"2024-09-16T15:44:14.461845+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_clean_scrub_stamp":"2024-09-16T15:44:14.461845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":1,"reported_epoch":41,"state":"creating+peering","last_fresh":"2024-09-16T15:44:13.472012+0000","last_change":"2024-09-16T15:44:13.472012+0000","last_active":"2024-09-16T15:44:13.453120+0000","last_peered":"2024-09-16T15:44:13.453120+0000","last_clean":"2024-09-16T15:44:13.453120+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:44:13.472012+0000","last_undegraded":"2024-09-16T15:44:13.472012+0000","last_fullsized":"2024-09-16T15:44:13.472012+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:13.453120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:13.453120+0000","last_clean_scrub_stamp":"2024-09-16T15:44:13.453120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T15:44:14.461845+0000","last_change":"2024-09-16T15:44:14.461845+0000","last_active":"2024-09-16T15:44:14.461845+0000","last_peered":"2024-09-16T15:44:14.461845+0000","last_clean":"2024-09-16T15:44:14.461845+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:44:14.461845+0000","last_undegraded":"2024-09-16T15:44:14.461845+0000","last_fullsized":"2024-09-16T15:44:14.461845+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_clean_scrub_stamp":"2024-09-16T15:44:14.461845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":1,"reported_epoch":41,"state":"creating+peering","last_fresh":"2024-09-16T15:44:13.474616+0000","last_change":"2024-09-16T15:44:13.474616+0000","last_active":"2024-09-16T15:44:13.453120+0000","last_peered":"2024-09-16T15:44:13.453120+0000","last_clean":"2024-09-16T15:44:13.453120+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:44:13.474616+0000","last_undegraded":"2024-09-16T15:44:13.474616+0000","last_fullsized":"2024-09-16T15:44:13.474616+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:13.453120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:13.453120+0000","last_clean_scrub_stamp":"2024-09-16T15:44:13.453120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T15:44:14.461845+0000","last_change":"2024-09-16T15:44:14.461845+0000","last_active":"2024-09-16T15:44:14.461845+0000","last_peered":"2024-09-16T15:44:14.461845+0000","last_clean":"2024-09-16T15:44:14.461845+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:44:14.461845+0000","last_undegraded":"2024-09-16T15:44:14.461845+0000","last_fullsized":"2024-09-16T15:44:14.461845+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_clean_scrub_stamp":"2024-09-16T15:44:14.461845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":1,"reported_epoch":41,"state":"creating+peering","last_fresh":"2024-09-16T15:44:13.467248+0000","last_change":"2024-09-16T15:44:13.467248+0000","last_active":"2024-09-16T15:44:13.453120+0000","last_peered":"2024-09-16T15:44:13.453120+0000","last_clean":"2024-09-16T15:44:13.453120+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:44:13.467248+0000","last_undegraded":"2024-09-16T15:44:13.467248+0000","last_fullsized":"2024-09-16T15:44:13.467248+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:13.453120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:13.453120+0000","last_clean_scrub_stamp":"2024-09-16T15:44:13.453120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T15:44:14.461845+0000","last_change":"2024-09-16T15:44:14.461845+0000","last_active":"2024-09-16T15:44:14.461845+0000","last_peered":"2024-09-16T15:44:14.461845+0000","last_clean":"2024-09-16T15:44:14.461845+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:44:14.461845+0000","last_undegraded":"2024-09-16T15:44:14.461845+0000","last_fullsized":"2024-09-16T15:44:14.461845+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_clean_scrub_stamp":"2024-09-16T15:44:14.461845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":1,"reported_epoch":41,"state":"creating+peering","last_fresh":"2024-09-16T15:44:13.472810+0000","last_change":"2024-09-16T15:44:13.472810+0000","last_active":"2024-09-16T15:44:13.453120+0000","last_peered":"2024-09-16T15:44:13.453120+0000","last_clean":"2024-09-16T15:44:13.453120+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:44:13.472810+0000","last_undegraded":"2024-09-16T15:44:13.472810+0000","last_fullsized":"2024-09-16T15:44:13.472810+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:13.453120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:13.453120+0000","last_clean_scrub_stamp":"2024-09-16T15:44:13.453120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T15:44:14.461845+0000","last_change":"2024-09-16T15:44:14.461845+0000","last_active":"2024-09-16T15:44:14.461845+0000","last_peered":"2024-09-16T15:44:14.461845+0000","last_clean":"2024-09-16T15:44:14.461845+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:44:14.461845+0000","last_undegraded":"2024-09-16T15:44:14.461845+0000","last_fullsized":"2024-09-16T15:44:14.461845+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_clean_scrub_stamp":"2024-09-16T15:44:14.461845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":1,"reported_epoch":41,"state":"creating+peering","last_fresh":"2024-09-16T15:44:13.473352+0000","last_change":"2024-09-16T15:44:13.473352+0000","last_active":"2024-09-16T15:44:13.453120+0000","last_peered":"2024-09-16T15:44:13.453120+0000","last_clean":"2024-09-16T15:44:13.453120+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:44:13.473352+0000","last_undegraded":"2024-09-16T15:44:13.473352+0000","last_fullsized":"2024-09-16T15:44:13.473352+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:13.453120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:13.453120+0000","last_clean_scrub_stamp":"2024-09-16T15:44:13.453120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T15:44:14.461845+0000","last_change":"2024-09-16T15:44:14.461845+0000","last_active":"2024-09-16T15:44:14.461845+0000","last_peered":"2024-09-16T15:44:14.461845+0000","last_clean":"2024-09-16T15:44:14.461845+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:44:14.461845+0000","last_undegraded":"2024-09-16T15:44:14.461845+0000","last_fullsized":"2024-09-16T15:44:14.461845+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_clean_scrub_stamp":"2024-09-16T15:44:14.461845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":1,"reported_epoch":41,"state":"creating+peering","last_fresh":"2024-09-16T15:44:13.476429+0000","last_change":"2024-09-16T15:44:13.476429+0000","last_active":"2024-09-16T15:44:13.453120+0000","last_peered":"2024-09-16T15:44:13.453120+0000","last_clean":"2024-09-16T15:44:13.453120+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:44:13.476429+0000","last_undegraded":"2024-09-16T15:44:13.476429+0000","last_fullsized":"2024-09-16T15:44:13.476429+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:13.453120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:13.453120+0000","last_clean_scrub_stamp":"2024-09-16T15:44:13.453120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"16'72","reported_seq":147,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-16T15:44:13.477262+0000","last_change":"2024-09-16T15:42:40.881820+0000","last_active":"2024-09-16T15:44:13.477262+0000","last_peered":"2024-09-16T15:44:13.477262+0000","last_clean":"2024-09-16T15:44:13.477262+0000","last_became_active":"2024-09-16T15:42:40.879965+0000","last_became_peered":"2024-09-16T15:42:40.879965+0000","last_unstale":"2024-09-16T15:44:13.477262+0000","last_undegraded":"2024-09-16T15:44:13.477262+0000","last_fullsized":"2024-09-16T15:44:13.477262+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:42:39.728762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:42:39.728762+0000","last_clean_scrub_stamp":"2024-09-16T15:42:39.728762+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:22:18.768932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542167,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28072,"kb_used_data":1044,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343768,"statfs":{"total":96636764160,"available":96608018432,"internally_reserved":0,"allocated":1069056,"data_stored":776814,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542166,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27492,"kb_used_data":452,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344348,"statfs":{"total":96636764160,"available":96608612352,"internally_reserved":0,"allocated":462848,"data_stored":180629,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":5,"up_from":14,"seq":60129542167,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27492,"kb_used_data":464,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344348,"statfs":{"total":96636764160,"available":96608612352,"internally_reserved":0,"allocated":475136,"data_stored":186446,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542165,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27492,"kb_used_data":464,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344348,"statfs":{"total":96636764160,"available":96608612352,"internally_reserved":0,"allocated":475136,"data_stored":186446,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":14,"seq":60129542166,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31652,"kb_used_data":464,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340188,"statfs":{"total":96636764160,"available":96604352512,"internally_reserved":0,"allocated":475136,"data_stored":186446,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":14,"seq":60129542166,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27508,"kb_used_data":464,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344332,"statfs":{"total":96636764160,"available":96608595968,"internally_reserved":0,"allocated":475136,"data_stored":186446,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":14,"seq":60129542168,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27480,"kb_used_data":452,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344360,"statfs":{"total":96636764160,"available":96608624640,"internally_reserved":0,"allocated":462848,"data_stored":180629,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":14,"seq":60129542168,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28072,"kb_used_data":1044,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343768,"statfs":{"total":96636764160,"available":96608018432,"internally_reserved":0,"allocated":1069056,"data_stored":776814,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T15:44:24.035 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T15:44:24.169 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:24.163+0000 7f49db6e4640 1 -- 172.21.15.123:0/1313128902 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f49d4074600 msgr2=0x7f49d40d0630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:44:24.169 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:24.163+0000 7f49db6e4640 1 --2- 172.21.15.123:0/1313128902 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f49d4074600 0x7f49d40d0630 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7f49d40cbd60 tx=0x7f49cc01cae0 comp rx=0 tx=0).stop 2024-09-16T15:44:24.169 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:24.163+0000 7f49db6e4640 1 -- 172.21.15.123:0/1313128902 shutdown_connections 2024-09-16T15:44:24.169 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:24.163+0000 7f49db6e4640 1 --2- 172.21.15.123:0/1313128902 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f49d4074600 0x7f49d40d0630 unknown :-1 s=CLOSED pgs=465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:24.169 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:24.163+0000 7f49db6e4640 1 -- 172.21.15.123:0/1313128902 >> 172.21.15.123:0/1313128902 conn(0x7f49d40cfb10 msgr2=0x7f49d40cff20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:44:24.169 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:24.163+0000 7f49db6e4640 1 -- 172.21.15.123:0/1313128902 shutdown_connections 2024-09-16T15:44:24.169 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:24.163+0000 7f49db6e4640 1 -- 172.21.15.123:0/1313128902 wait complete. 2024-09-16T15:44:24.170 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:24.163+0000 7f49db6e4640 1 Processor -- start 2024-09-16T15:44:24.170 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:24.163+0000 7f49db6e4640 1 -- start start 2024-09-16T15:44:24.172 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:24.163+0000 7f49db6e4640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f49d40726c0 0x7f49d414f920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:44:24.172 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:24.163+0000 7f49db6e4640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f49d4074600 0x7f49d40c5cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:44:24.172 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:24.163+0000 7f49db6e4640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f49d415bca0 0x7f49d40b3920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:44:24.172 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:24.163+0000 7f49db6e4640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f49d4169c70 con 0x7f49d415bca0 2024-09-16T15:44:24.172 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:24.163+0000 7f49db6e4640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f49d406fa80 con 0x7f49d40726c0 2024-09-16T15:44:24.172 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:24.163+0000 7f49db6e4640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f49d4169df0 con 0x7f49d4074600 2024-09-16T15:44:24.172 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:24.163+0000 7f49d9c81640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f49d415bca0 0x7f49d40b3920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:44:24.172 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:24.163+0000 7f49d9c81640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f49d415bca0 0x7f49d40b3920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:38798/0 (socket says 172.21.15.123:38798) 2024-09-16T15:44:24.172 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:24.163+0000 7f49d9c81640 1 -- 172.21.15.123:0/2794506733 learned_addr learned my addr 172.21.15.123:0/2794506733 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:44:24.172 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:24.163+0000 7f49d9c81640 1 -- 172.21.15.123:0/2794506733 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f49d4074600 msgr2=0x7f49d40c5cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:44:24.172 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:24.163+0000 7f49d9c81640 1 --2- 172.21.15.123:0/2794506733 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f49d4074600 0x7f49d40c5cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:24.172 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:24.163+0000 7f49d9c81640 1 -- 172.21.15.123:0/2794506733 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f49d40726c0 msgr2=0x7f49d414f920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:44:24.173 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:24.163+0000 7f49d9c81640 1 --2- 172.21.15.123:0/2794506733 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f49d40726c0 0x7f49d414f920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:24.173 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:24.163+0000 7f49d9c81640 1 -- 172.21.15.123:0/2794506733 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f49d415d250 con 0x7f49d415bca0 2024-09-16T15:44:24.173 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:24.167+0000 7f49d9c81640 1 --2- 172.21.15.123:0/2794506733 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f49d415bca0 0x7f49d40b3920 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f49cc0986d0 tx=0x7f49cc005b00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:44:24.173 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:24.167+0000 7f49c9ffb640 1 -- 172.21.15.123:0/2794506733 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49cc0b2040 con 0x7f49d415bca0 2024-09-16T15:44:24.173 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:24.167+0000 7f49c9ffb640 1 -- 172.21.15.123:0/2794506733 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f49cc0bb070 con 0x7f49d415bca0 2024-09-16T15:44:24.173 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:24.167+0000 7f49db6e4640 1 -- 172.21.15.123:0/2794506733 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f49d415d4f0 con 0x7f49d415bca0 2024-09-16T15:44:24.173 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:24.167+0000 7f49c9ffb640 1 -- 172.21.15.123:0/2794506733 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49cc004330 con 0x7f49d415bca0 2024-09-16T15:44:24.173 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:24.167+0000 7f49db6e4640 1 -- 172.21.15.123:0/2794506733 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f49d415dab0 con 0x7f49d415bca0 2024-09-16T15:44:24.174 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:24.167+0000 7f49db6e4640 1 -- 172.21.15.123:0/2794506733 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f49d40d0630 con 0x7f49d415bca0 2024-09-16T15:44:24.174 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:24.167+0000 7f49c9ffb640 1 -- 172.21.15.123:0/2794506733 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 7) ==== 97233+0+0 (secure 0 0 0) 0x7f49cc0044d0 con 0x7f49d415bca0 2024-09-16T15:44:24.174 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:24.167+0000 7f49c9ffb640 1 --2- 172.21.15.123:0/2794506733 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f49b8073070 0x7f49b8075530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:44:24.174 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:24.167+0000 7f49c9ffb640 1 -- 172.21.15.123:0/2794506733 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(44..44 src has 1..44) ==== 5709+0+0 (secure 0 0 0) 0x7f49cc12f790 con 0x7f49d415bca0 2024-09-16T15:44:24.174 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:24.167+0000 7f49d9480640 1 --2- 172.21.15.123:0/2794506733 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f49b8073070 0x7f49b8075530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:44:24.175 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:24.167+0000 7f49d9480640 1 --2- 172.21.15.123:0/2794506733 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f49b8073070 0x7f49b8075530 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f49c40046c0 tx=0x7f49c4007910 comp rx=0 tx=0).ready entity=mgr.4135 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:44:24.177 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:24.171+0000 7f49c9ffb640 1 -- 172.21.15.123:0/2794506733 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f49cc016790 con 0x7f49d415bca0 2024-09-16T15:44:24.350 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:24.343+0000 7f49db6e4640 1 -- 172.21.15.123:0/2794506733 --> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f49d415f630 con 0x7f49b8073070 2024-09-16T15:44:24.361 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:24.355+0000 7f49c9ffb640 1 -- 172.21.15.123:0/2794506733 <== mgr.4135 v2:172.21.15.184:6832/3114346423 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+251948 (secure 0 0 0) 0x7f49d415f630 con 0x7f49b8073070 2024-09-16T15:44:24.364 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-09-16T15:44:24.365 INFO:teuthology.orchestra.run.smithi123.stderr:dumped all 2024-09-16T15:44:24.366 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:24.359+0000 7f49db6e4640 1 -- 172.21.15.123:0/2794506733 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f49b8073070 msgr2=0x7f49b8075530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:44:24.366 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:24.359+0000 7f49db6e4640 1 --2- 172.21.15.123:0/2794506733 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f49b8073070 0x7f49b8075530 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f49c40046c0 tx=0x7f49c4007910 comp rx=0 tx=0).stop 2024-09-16T15:44:24.366 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:24.359+0000 7f49db6e4640 1 -- 172.21.15.123:0/2794506733 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f49d415bca0 msgr2=0x7f49d40b3920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:44:24.366 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:24.359+0000 7f49db6e4640 1 --2- 172.21.15.123:0/2794506733 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f49d415bca0 0x7f49d40b3920 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f49cc0986d0 tx=0x7f49cc005b00 comp rx=0 tx=0).stop 2024-09-16T15:44:24.366 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:24.359+0000 7f49db6e4640 1 -- 172.21.15.123:0/2794506733 shutdown_connections 2024-09-16T15:44:24.366 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:24.359+0000 7f49db6e4640 1 --2- 172.21.15.123:0/2794506733 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f49b8073070 0x7f49b8075530 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:24.367 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:24.359+0000 7f49db6e4640 1 --2- 172.21.15.123:0/2794506733 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f49d415bca0 0x7f49d40b3920 unknown :-1 s=CLOSED pgs=466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:24.367 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:24.359+0000 7f49db6e4640 1 --2- 172.21.15.123:0/2794506733 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f49d4074600 0x7f49d40c5cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:24.367 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:24.359+0000 7f49db6e4640 1 --2- 172.21.15.123:0/2794506733 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f49d40726c0 0x7f49d414f920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:24.367 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:24.359+0000 7f49db6e4640 1 -- 172.21.15.123:0/2794506733 >> 172.21.15.123:0/2794506733 conn(0x7f49d40cfb10 msgr2=0x7f49d40cfef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:44:24.367 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:24.359+0000 7f49db6e4640 1 -- 172.21.15.123:0/2794506733 shutdown_connections 2024-09-16T15:44:24.367 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:24.359+0000 7f49db6e4640 1 -- 172.21.15.123:0/2794506733 wait complete. 2024-09-16T15:44:24.382 INFO:teuthology.orchestra.run.smithi123.stdout:{"pg_ready":true,"pg_map":{"version":89,"stamp":"2024-09-16T15:44:23.275714+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":146,"num_write_kb":1642,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":111,"ondisk_log_size":111,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":188,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":225640,"kb_used_data":5224,"kb_used_omap":12,"kb_used_meta":220211,"kb_avail":754749080,"statfs":{"total":773094113280,"available":772863057920,"internally_reserved":0,"allocated":5349376,"data_stored":2831860,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":225496625},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":48,"apply_latency_ms":48,"commit_latency_ns":48000000,"apply_latency_ns":48000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002627"},"pg_stats":[{"pgid":"7.3f","version":"0'0","reported_seq":13,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T15:44:16.653977+0000","last_change":"2024-09-16T15:44:15.526109+0000","last_active":"2024-09-16T15:44:16.653977+0000","last_peered":"2024-09-16T15:44:16.653977+0000","last_clean":"2024-09-16T15:44:16.653977+0000","last_became_active":"2024-09-16T15:44:15.524737+0000","last_became_peered":"2024-09-16T15:44:15.524737+0000","last_unstale":"2024-09-16T15:44:16.653977+0000","last_undegraded":"2024-09-16T15:44:16.653977+0000","last_fullsized":"2024-09-16T15:44:16.653977+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_clean_scrub_stamp":"2024-09-16T15:44:14.461845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T00:30:57.600139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.3e","version":"0'0","reported_seq":13,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T15:44:16.654005+0000","last_change":"2024-09-16T15:44:15.524142+0000","last_active":"2024-09-16T15:44:16.654005+0000","last_peered":"2024-09-16T15:44:16.654005+0000","last_clean":"2024-09-16T15:44:16.654005+0000","last_became_active":"2024-09-16T15:44:15.522926+0000","last_became_peered":"2024-09-16T15:44:15.522926+0000","last_unstale":"2024-09-16T15:44:16.654005+0000","last_undegraded":"2024-09-16T15:44:16.654005+0000","last_fullsized":"2024-09-16T15:44:16.654005+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_clean_scrub_stamp":"2024-09-16T15:44:14.461845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T22:16:40.046596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.3d","version":"0'0","reported_seq":13,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T15:44:16.504414+0000","last_change":"2024-09-16T15:44:15.536821+0000","last_active":"2024-09-16T15:44:16.504414+0000","last_peered":"2024-09-16T15:44:16.504414+0000","last_clean":"2024-09-16T15:44:16.504414+0000","last_became_active":"2024-09-16T15:44:15.535951+0000","last_became_peered":"2024-09-16T15:44:15.535951+0000","last_unstale":"2024-09-16T15:44:16.504414+0000","last_undegraded":"2024-09-16T15:44:16.504414+0000","last_fullsized":"2024-09-16T15:44:16.504414+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_clean_scrub_stamp":"2024-09-16T15:44:14.461845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T23:53:09.825776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.3c","version":"0'0","reported_seq":13,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T15:44:16.653615+0000","last_change":"2024-09-16T15:44:15.527552+0000","last_active":"2024-09-16T15:44:16.653615+0000","last_peered":"2024-09-16T15:44:16.653615+0000","last_clean":"2024-09-16T15:44:16.653615+0000","last_became_active":"2024-09-16T15:44:15.524264+0000","last_became_peered":"2024-09-16T15:44:15.524264+0000","last_unstale":"2024-09-16T15:44:16.653615+0000","last_undegraded":"2024-09-16T15:44:16.653615+0000","last_fullsized":"2024-09-16T15:44:16.653615+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_clean_scrub_stamp":"2024-09-16T15:44:14.461845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T01:56:32.093012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.3b","version":"0'0","reported_seq":13,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T15:44:16.654558+0000","last_change":"2024-09-16T15:44:15.523439+0000","last_active":"2024-09-16T15:44:16.654558+0000","last_peered":"2024-09-16T15:44:16.654558+0000","last_clean":"2024-09-16T15:44:16.654558+0000","last_became_active":"2024-09-16T15:44:15.522751+0000","last_became_peered":"2024-09-16T15:44:15.522751+0000","last_unstale":"2024-09-16T15:44:16.654558+0000","last_undegraded":"2024-09-16T15:44:16.654558+0000","last_fullsized":"2024-09-16T15:44:16.654558+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_clean_scrub_stamp":"2024-09-16T15:44:14.461845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:06:32.379714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.3a","version":"0'0","reported_seq":13,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T15:44:16.549474+0000","last_change":"2024-09-16T15:44:15.531426+0000","last_active":"2024-09-16T15:44:16.549474+0000","last_peered":"2024-09-16T15:44:16.549474+0000","last_clean":"2024-09-16T15:44:16.549474+0000","last_became_active":"2024-09-16T15:44:15.527326+0000","last_became_peered":"2024-09-16T15:44:15.527326+0000","last_unstale":"2024-09-16T15:44:16.549474+0000","last_undegraded":"2024-09-16T15:44:16.549474+0000","last_fullsized":"2024-09-16T15:44:16.549474+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_clean_scrub_stamp":"2024-09-16T15:44:14.461845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T22:55:43.022402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.39","version":"0'0","reported_seq":13,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T15:44:16.653090+0000","last_change":"2024-09-16T15:44:15.540612+0000","last_active":"2024-09-16T15:44:16.653090+0000","last_peered":"2024-09-16T15:44:16.653090+0000","last_clean":"2024-09-16T15:44:16.653090+0000","last_became_active":"2024-09-16T15:44:15.540100+0000","last_became_peered":"2024-09-16T15:44:15.540100+0000","last_unstale":"2024-09-16T15:44:16.653090+0000","last_undegraded":"2024-09-16T15:44:16.653090+0000","last_fullsized":"2024-09-16T15:44:16.653090+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_clean_scrub_stamp":"2024-09-16T15:44:14.461845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T00:19:50.260206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.38","version":"0'0","reported_seq":13,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T15:44:16.652747+0000","last_change":"2024-09-16T15:44:15.523239+0000","last_active":"2024-09-16T15:44:16.652747+0000","last_peered":"2024-09-16T15:44:16.652747+0000","last_clean":"2024-09-16T15:44:16.652747+0000","last_became_active":"2024-09-16T15:44:15.522523+0000","last_became_peered":"2024-09-16T15:44:15.522523+0000","last_unstale":"2024-09-16T15:44:16.652747+0000","last_undegraded":"2024-09-16T15:44:16.652747+0000","last_fullsized":"2024-09-16T15:44:16.652747+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_clean_scrub_stamp":"2024-09-16T15:44:14.461845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T02:25:02.835896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.37","version":"0'0","reported_seq":13,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T15:44:16.653772+0000","last_change":"2024-09-16T15:44:15.536161+0000","last_active":"2024-09-16T15:44:16.653772+0000","last_peered":"2024-09-16T15:44:16.653772+0000","last_clean":"2024-09-16T15:44:16.653772+0000","last_became_active":"2024-09-16T15:44:15.535362+0000","last_became_peered":"2024-09-16T15:44:15.535362+0000","last_unstale":"2024-09-16T15:44:16.653772+0000","last_undegraded":"2024-09-16T15:44:16.653772+0000","last_fullsized":"2024-09-16T15:44:16.653772+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_clean_scrub_stamp":"2024-09-16T15:44:14.461845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:02:53.783922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.36","version":"0'0","reported_seq":13,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T15:44:16.654196+0000","last_change":"2024-09-16T15:44:15.528153+0000","last_active":"2024-09-16T15:44:16.654196+0000","last_peered":"2024-09-16T15:44:16.654196+0000","last_clean":"2024-09-16T15:44:16.654196+0000","last_became_active":"2024-09-16T15:44:15.526351+0000","last_became_peered":"2024-09-16T15:44:15.526351+0000","last_unstale":"2024-09-16T15:44:16.654196+0000","last_undegraded":"2024-09-16T15:44:16.654196+0000","last_fullsized":"2024-09-16T15:44:16.654196+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_clean_scrub_stamp":"2024-09-16T15:44:14.461845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T03:26:57.808666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.35","version":"0'0","reported_seq":13,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T15:44:16.654399+0000","last_change":"2024-09-16T15:44:15.518972+0000","last_active":"2024-09-16T15:44:16.654399+0000","last_peered":"2024-09-16T15:44:16.654399+0000","last_clean":"2024-09-16T15:44:16.654399+0000","last_became_active":"2024-09-16T15:44:15.518111+0000","last_became_peered":"2024-09-16T15:44:15.518111+0000","last_unstale":"2024-09-16T15:44:16.654399+0000","last_undegraded":"2024-09-16T15:44:16.654399+0000","last_fullsized":"2024-09-16T15:44:16.654399+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_clean_scrub_stamp":"2024-09-16T15:44:14.461845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T18:07:17.331288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.34","version":"0'0","reported_seq":13,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T15:44:16.654584+0000","last_change":"2024-09-16T15:44:15.541939+0000","last_active":"2024-09-16T15:44:16.654584+0000","last_peered":"2024-09-16T15:44:16.654584+0000","last_clean":"2024-09-16T15:44:16.654584+0000","last_became_active":"2024-09-16T15:44:15.541252+0000","last_became_peered":"2024-09-16T15:44:15.541252+0000","last_unstale":"2024-09-16T15:44:16.654584+0000","last_undegraded":"2024-09-16T15:44:16.654584+0000","last_fullsized":"2024-09-16T15:44:16.654584+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_clean_scrub_stamp":"2024-09-16T15:44:14.461845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T18:07:17.331288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.33","version":"0'0","reported_seq":13,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T15:44:16.504706+0000","last_change":"2024-09-16T15:44:15.531436+0000","last_active":"2024-09-16T15:44:16.504706+0000","last_peered":"2024-09-16T15:44:16.504706+0000","last_clean":"2024-09-16T15:44:16.504706+0000","last_became_active":"2024-09-16T15:44:15.530653+0000","last_became_peered":"2024-09-16T15:44:15.530653+0000","last_unstale":"2024-09-16T15:44:16.504706+0000","last_undegraded":"2024-09-16T15:44:16.504706+0000","last_fullsized":"2024-09-16T15:44:16.504706+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_clean_scrub_stamp":"2024-09-16T15:44:14.461845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:45:48.766505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.32","version":"0'0","reported_seq":13,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T15:44:16.654309+0000","last_change":"2024-09-16T15:44:15.519038+0000","last_active":"2024-09-16T15:44:16.654309+0000","last_peered":"2024-09-16T15:44:16.654309+0000","last_clean":"2024-09-16T15:44:16.654309+0000","last_became_active":"2024-09-16T15:44:15.518308+0000","last_became_peered":"2024-09-16T15:44:15.518308+0000","last_unstale":"2024-09-16T15:44:16.654309+0000","last_undegraded":"2024-09-16T15:44:16.654309+0000","last_fullsized":"2024-09-16T15:44:16.654309+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_clean_scrub_stamp":"2024-09-16T15:44:14.461845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T22:16:40.046596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.31","version":"0'0","reported_seq":13,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T15:44:16.550444+0000","last_change":"2024-09-16T15:44:15.535530+0000","last_active":"2024-09-16T15:44:16.550444+0000","last_peered":"2024-09-16T15:44:16.550444+0000","last_clean":"2024-09-16T15:44:16.550444+0000","last_became_active":"2024-09-16T15:44:15.526632+0000","last_became_peered":"2024-09-16T15:44:15.526632+0000","last_unstale":"2024-09-16T15:44:16.550444+0000","last_undegraded":"2024-09-16T15:44:16.550444+0000","last_fullsized":"2024-09-16T15:44:16.550444+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_clean_scrub_stamp":"2024-09-16T15:44:14.461845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:33:36.756308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.30","version":"0'0","reported_seq":13,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T15:44:16.504279+0000","last_change":"2024-09-16T15:44:15.522618+0000","last_active":"2024-09-16T15:44:16.504279+0000","last_peered":"2024-09-16T15:44:16.504279+0000","last_clean":"2024-09-16T15:44:16.504279+0000","last_became_active":"2024-09-16T15:44:15.521829+0000","last_became_peered":"2024-09-16T15:44:15.521829+0000","last_unstale":"2024-09-16T15:44:16.504279+0000","last_undegraded":"2024-09-16T15:44:16.504279+0000","last_fullsized":"2024-09-16T15:44:16.504279+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_clean_scrub_stamp":"2024-09-16T15:44:14.461845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:28:22.393754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.2f","version":"0'0","reported_seq":13,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T15:44:16.506606+0000","last_change":"2024-09-16T15:44:15.519972+0000","last_active":"2024-09-16T15:44:16.506606+0000","last_peered":"2024-09-16T15:44:16.506606+0000","last_clean":"2024-09-16T15:44:16.506606+0000","last_became_active":"2024-09-16T15:44:15.519204+0000","last_became_peered":"2024-09-16T15:44:15.519204+0000","last_unstale":"2024-09-16T15:44:16.506606+0000","last_undegraded":"2024-09-16T15:44:16.506606+0000","last_fullsized":"2024-09-16T15:44:16.506606+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_clean_scrub_stamp":"2024-09-16T15:44:14.461845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:49:26.583843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.2e","version":"0'0","reported_seq":13,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T15:44:16.504513+0000","last_change":"2024-09-16T15:44:15.524571+0000","last_active":"2024-09-16T15:44:16.504513+0000","last_peered":"2024-09-16T15:44:16.504513+0000","last_clean":"2024-09-16T15:44:16.504513+0000","last_became_active":"2024-09-16T15:44:15.523759+0000","last_became_peered":"2024-09-16T15:44:15.523759+0000","last_unstale":"2024-09-16T15:44:16.504513+0000","last_undegraded":"2024-09-16T15:44:16.504513+0000","last_fullsized":"2024-09-16T15:44:16.504513+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_clean_scrub_stamp":"2024-09-16T15:44:14.461845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T22:55:38.725563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.2d","version":"0'0","reported_seq":13,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T15:44:16.507582+0000","last_change":"2024-09-16T15:44:15.517379+0000","last_active":"2024-09-16T15:44:16.507582+0000","last_peered":"2024-09-16T15:44:16.507582+0000","last_clean":"2024-09-16T15:44:16.507582+0000","last_became_active":"2024-09-16T15:44:15.516655+0000","last_became_peered":"2024-09-16T15:44:15.516655+0000","last_unstale":"2024-09-16T15:44:16.507582+0000","last_undegraded":"2024-09-16T15:44:16.507582+0000","last_fullsized":"2024-09-16T15:44:16.507582+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_clean_scrub_stamp":"2024-09-16T15:44:14.461845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T02:15:51.750452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.2c","version":"0'0","reported_seq":13,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T15:44:16.504717+0000","last_change":"2024-09-16T15:44:15.523283+0000","last_active":"2024-09-16T15:44:16.504717+0000","last_peered":"2024-09-16T15:44:16.504717+0000","last_clean":"2024-09-16T15:44:16.504717+0000","last_became_active":"2024-09-16T15:44:15.522536+0000","last_became_peered":"2024-09-16T15:44:15.522536+0000","last_unstale":"2024-09-16T15:44:16.504717+0000","last_undegraded":"2024-09-16T15:44:16.504717+0000","last_fullsized":"2024-09-16T15:44:16.504717+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_clean_scrub_stamp":"2024-09-16T15:44:14.461845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:44:45.134236+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.2b","version":"0'0","reported_seq":13,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T15:44:16.549164+0000","last_change":"2024-09-16T15:44:15.533693+0000","last_active":"2024-09-16T15:44:16.549164+0000","last_peered":"2024-09-16T15:44:16.549164+0000","last_clean":"2024-09-16T15:44:16.549164+0000","last_became_active":"2024-09-16T15:44:15.532884+0000","last_became_peered":"2024-09-16T15:44:15.532884+0000","last_unstale":"2024-09-16T15:44:16.549164+0000","last_undegraded":"2024-09-16T15:44:16.549164+0000","last_fullsized":"2024-09-16T15:44:16.549164+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_clean_scrub_stamp":"2024-09-16T15:44:14.461845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T22:11:10.028732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.2a","version":"0'0","reported_seq":13,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T15:44:16.653376+0000","last_change":"2024-09-16T15:44:15.528197+0000","last_active":"2024-09-16T15:44:16.653376+0000","last_peered":"2024-09-16T15:44:16.653376+0000","last_clean":"2024-09-16T15:44:16.653376+0000","last_became_active":"2024-09-16T15:44:15.526285+0000","last_became_peered":"2024-09-16T15:44:15.526285+0000","last_unstale":"2024-09-16T15:44:16.653376+0000","last_undegraded":"2024-09-16T15:44:16.653376+0000","last_fullsized":"2024-09-16T15:44:16.653376+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_clean_scrub_stamp":"2024-09-16T15:44:14.461845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T00:45:50.374355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.29","version":"0'0","reported_seq":13,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T15:44:16.504878+0000","last_change":"2024-09-16T15:44:15.530572+0000","last_active":"2024-09-16T15:44:16.504878+0000","last_peered":"2024-09-16T15:44:16.504878+0000","last_clean":"2024-09-16T15:44:16.504878+0000","last_became_active":"2024-09-16T15:44:15.529746+0000","last_became_peered":"2024-09-16T15:44:15.529746+0000","last_unstale":"2024-09-16T15:44:16.504878+0000","last_undegraded":"2024-09-16T15:44:16.504878+0000","last_fullsized":"2024-09-16T15:44:16.504878+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_clean_scrub_stamp":"2024-09-16T15:44:14.461845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:58:18.168632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.28","version":"0'0","reported_seq":13,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T15:44:16.504824+0000","last_change":"2024-09-16T15:44:15.530199+0000","last_active":"2024-09-16T15:44:16.504824+0000","last_peered":"2024-09-16T15:44:16.504824+0000","last_clean":"2024-09-16T15:44:16.504824+0000","last_became_active":"2024-09-16T15:44:15.529031+0000","last_became_peered":"2024-09-16T15:44:15.529031+0000","last_unstale":"2024-09-16T15:44:16.504824+0000","last_undegraded":"2024-09-16T15:44:16.504824+0000","last_fullsized":"2024-09-16T15:44:16.504824+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_clean_scrub_stamp":"2024-09-16T15:44:14.461845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T00:14:42.823128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.27","version":"0'0","reported_seq":13,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T15:44:16.507983+0000","last_change":"2024-09-16T15:44:15.529188+0000","last_active":"2024-09-16T15:44:16.507983+0000","last_peered":"2024-09-16T15:44:16.507983+0000","last_clean":"2024-09-16T15:44:16.507983+0000","last_became_active":"2024-09-16T15:44:15.528471+0000","last_became_peered":"2024-09-16T15:44:15.528471+0000","last_unstale":"2024-09-16T15:44:16.507983+0000","last_undegraded":"2024-09-16T15:44:16.507983+0000","last_fullsized":"2024-09-16T15:44:16.507983+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_clean_scrub_stamp":"2024-09-16T15:44:14.461845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:13:05.113843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.26","version":"0'0","reported_seq":13,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T15:44:16.504712+0000","last_change":"2024-09-16T15:44:15.524846+0000","last_active":"2024-09-16T15:44:16.504712+0000","last_peered":"2024-09-16T15:44:16.504712+0000","last_clean":"2024-09-16T15:44:16.504712+0000","last_became_active":"2024-09-16T15:44:15.524169+0000","last_became_peered":"2024-09-16T15:44:15.524169+0000","last_unstale":"2024-09-16T15:44:16.504712+0000","last_undegraded":"2024-09-16T15:44:16.504712+0000","last_fullsized":"2024-09-16T15:44:16.504712+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_clean_scrub_stamp":"2024-09-16T15:44:14.461845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T23:46:12.488007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.25","version":"0'0","reported_seq":13,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T15:44:16.504971+0000","last_change":"2024-09-16T15:44:15.514789+0000","last_active":"2024-09-16T15:44:16.504971+0000","last_peered":"2024-09-16T15:44:16.504971+0000","last_clean":"2024-09-16T15:44:16.504971+0000","last_became_active":"2024-09-16T15:44:15.514042+0000","last_became_peered":"2024-09-16T15:44:15.514042+0000","last_unstale":"2024-09-16T15:44:16.504971+0000","last_undegraded":"2024-09-16T15:44:16.504971+0000","last_fullsized":"2024-09-16T15:44:16.504971+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_clean_scrub_stamp":"2024-09-16T15:44:14.461845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T22:59:43.722404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.24","version":"0'0","reported_seq":13,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T15:44:16.654868+0000","last_change":"2024-09-16T15:44:15.529450+0000","last_active":"2024-09-16T15:44:16.654868+0000","last_peered":"2024-09-16T15:44:16.654868+0000","last_clean":"2024-09-16T15:44:16.654868+0000","last_became_active":"2024-09-16T15:44:15.528610+0000","last_became_peered":"2024-09-16T15:44:15.528610+0000","last_unstale":"2024-09-16T15:44:16.654868+0000","last_undegraded":"2024-09-16T15:44:16.654868+0000","last_fullsized":"2024-09-16T15:44:16.654868+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_clean_scrub_stamp":"2024-09-16T15:44:14.461845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T02:48:45.627033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.23","version":"0'0","reported_seq":13,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T15:44:16.505612+0000","last_change":"2024-09-16T15:44:15.517061+0000","last_active":"2024-09-16T15:44:16.505612+0000","last_peered":"2024-09-16T15:44:16.505612+0000","last_clean":"2024-09-16T15:44:16.505612+0000","last_became_active":"2024-09-16T15:44:15.516256+0000","last_became_peered":"2024-09-16T15:44:15.516256+0000","last_unstale":"2024-09-16T15:44:16.505612+0000","last_undegraded":"2024-09-16T15:44:16.505612+0000","last_fullsized":"2024-09-16T15:44:16.505612+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_clean_scrub_stamp":"2024-09-16T15:44:14.461845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T02:12:09.311304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.22","version":"0'0","reported_seq":13,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T15:44:16.503988+0000","last_change":"2024-09-16T15:44:15.526992+0000","last_active":"2024-09-16T15:44:16.503988+0000","last_peered":"2024-09-16T15:44:16.503988+0000","last_clean":"2024-09-16T15:44:16.503988+0000","last_became_active":"2024-09-16T15:44:15.526257+0000","last_became_peered":"2024-09-16T15:44:15.526257+0000","last_unstale":"2024-09-16T15:44:16.503988+0000","last_undegraded":"2024-09-16T15:44:16.503988+0000","last_fullsized":"2024-09-16T15:44:16.503988+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_clean_scrub_stamp":"2024-09-16T15:44:14.461845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T00:02:31.408206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.21","version":"0'0","reported_seq":13,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T15:44:16.549394+0000","last_change":"2024-09-16T15:44:15.528315+0000","last_active":"2024-09-16T15:44:16.549394+0000","last_peered":"2024-09-16T15:44:16.549394+0000","last_clean":"2024-09-16T15:44:16.549394+0000","last_became_active":"2024-09-16T15:44:15.524497+0000","last_became_peered":"2024-09-16T15:44:15.524497+0000","last_unstale":"2024-09-16T15:44:16.549394+0000","last_undegraded":"2024-09-16T15:44:16.549394+0000","last_fullsized":"2024-09-16T15:44:16.549394+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_clean_scrub_stamp":"2024-09-16T15:44:14.461845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T02:15:15.130011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.20","version":"0'0","reported_seq":13,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T15:44:16.504136+0000","last_change":"2024-09-16T15:44:15.523536+0000","last_active":"2024-09-16T15:44:16.504136+0000","last_peered":"2024-09-16T15:44:16.504136+0000","last_clean":"2024-09-16T15:44:16.504136+0000","last_became_active":"2024-09-16T15:44:15.522805+0000","last_became_peered":"2024-09-16T15:44:15.522805+0000","last_unstale":"2024-09-16T15:44:16.504136+0000","last_undegraded":"2024-09-16T15:44:16.504136+0000","last_fullsized":"2024-09-16T15:44:16.504136+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_clean_scrub_stamp":"2024-09-16T15:44:14.461845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T22:20:36.791540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.1e","version":"0'0","reported_seq":13,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T15:44:16.652217+0000","last_change":"2024-09-16T15:44:15.523074+0000","last_active":"2024-09-16T15:44:16.652217+0000","last_peered":"2024-09-16T15:44:16.652217+0000","last_clean":"2024-09-16T15:44:16.652217+0000","last_became_active":"2024-09-16T15:44:15.522357+0000","last_became_peered":"2024-09-16T15:44:15.522357+0000","last_unstale":"2024-09-16T15:44:16.652217+0000","last_undegraded":"2024-09-16T15:44:16.652217+0000","last_fullsized":"2024-09-16T15:44:16.652217+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_clean_scrub_stamp":"2024-09-16T15:44:14.461845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:30:43.225602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"43'4","reported_seq":19,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T15:44:16.549830+0000","last_change":"2024-09-16T15:44:14.501315+0000","last_active":"2024-09-16T15:44:16.549830+0000","last_peered":"2024-09-16T15:44:16.549830+0000","last_clean":"2024-09-16T15:44:16.549830+0000","last_became_active":"2024-09-16T15:44:14.501012+0000","last_became_peered":"2024-09-16T15:44:14.501012+0000","last_unstale":"2024-09-16T15:44:16.549830+0000","last_undegraded":"2024-09-16T15:44:16.549830+0000","last_fullsized":"2024-09-16T15:44:16.549830+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:13.453120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:13.453120+0000","last_clean_scrub_stamp":"2024-09-16T15:44:13.453120+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T02:38:28.887006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1f","version":"0'0","reported_seq":13,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T15:44:16.507990+0000","last_change":"2024-09-16T15:44:15.509152+0000","last_active":"2024-09-16T15:44:16.507990+0000","last_peered":"2024-09-16T15:44:16.507990+0000","last_clean":"2024-09-16T15:44:16.507990+0000","last_became_active":"2024-09-16T15:44:15.508322+0000","last_became_peered":"2024-09-16T15:44:15.508322+0000","last_unstale":"2024-09-16T15:44:16.507990+0000","last_undegraded":"2024-09-16T15:44:16.507990+0000","last_fullsized":"2024-09-16T15:44:16.507990+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_clean_scrub_stamp":"2024-09-16T15:44:14.461845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T18:52:15.728160+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1e","version":"43'1","reported_seq":16,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T15:44:16.506148+0000","last_change":"2024-09-16T15:44:14.495200+0000","last_active":"2024-09-16T15:44:16.506148+0000","last_peered":"2024-09-16T15:44:16.506148+0000","last_clean":"2024-09-16T15:44:16.506148+0000","last_became_active":"2024-09-16T15:44:14.493576+0000","last_became_peered":"2024-09-16T15:44:14.493576+0000","last_unstale":"2024-09-16T15:44:16.506148+0000","last_undegraded":"2024-09-16T15:44:16.506148+0000","last_fullsized":"2024-09-16T15:44:16.506148+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:13.453120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:13.453120+0000","last_clean_scrub_stamp":"2024-09-16T15:44:13.453120+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T00:14:41.814403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.1c","version":"0'0","reported_seq":13,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T15:44:16.654896+0000","last_change":"2024-09-16T15:44:15.525864+0000","last_active":"2024-09-16T15:44:16.654896+0000","last_peered":"2024-09-16T15:44:16.654896+0000","last_clean":"2024-09-16T15:44:16.654896+0000","last_became_active":"2024-09-16T15:44:15.525107+0000","last_became_peered":"2024-09-16T15:44:15.525107+0000","last_unstale":"2024-09-16T15:44:16.654896+0000","last_undegraded":"2024-09-16T15:44:16.654896+0000","last_fullsized":"2024-09-16T15:44:16.654896+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_clean_scrub_stamp":"2024-09-16T15:44:14.461845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T22:45:33.988227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1d","version":"43'2","reported_seq":17,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T15:44:16.652366+0000","last_change":"2024-09-16T15:44:14.505997+0000","last_active":"2024-09-16T15:44:16.652366+0000","last_peered":"2024-09-16T15:44:16.652366+0000","last_clean":"2024-09-16T15:44:16.652366+0000","last_became_active":"2024-09-16T15:44:14.505669+0000","last_became_peered":"2024-09-16T15:44:14.505669+0000","last_unstale":"2024-09-16T15:44:16.652366+0000","last_undegraded":"2024-09-16T15:44:16.652366+0000","last_fullsized":"2024-09-16T15:44:16.652366+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:13.453120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:13.453120+0000","last_clean_scrub_stamp":"2024-09-16T15:44:13.453120+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:43:50.112041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.1d","version":"0'0","reported_seq":13,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T15:44:16.652706+0000","last_change":"2024-09-16T15:44:15.516477+0000","last_active":"2024-09-16T15:44:16.652706+0000","last_peered":"2024-09-16T15:44:16.652706+0000","last_clean":"2024-09-16T15:44:16.652706+0000","last_became_active":"2024-09-16T15:44:15.514781+0000","last_became_peered":"2024-09-16T15:44:15.514781+0000","last_unstale":"2024-09-16T15:44:16.652706+0000","last_undegraded":"2024-09-16T15:44:16.652706+0000","last_fullsized":"2024-09-16T15:44:16.652706+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_clean_scrub_stamp":"2024-09-16T15:44:14.461845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T23:26:07.131961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1c","version":"43'2","reported_seq":17,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T15:44:16.653316+0000","last_change":"2024-09-16T15:44:14.517767+0000","last_active":"2024-09-16T15:44:16.653316+0000","last_peered":"2024-09-16T15:44:16.653316+0000","last_clean":"2024-09-16T15:44:16.653316+0000","last_became_active":"2024-09-16T15:44:14.516935+0000","last_became_peered":"2024-09-16T15:44:14.516935+0000","last_unstale":"2024-09-16T15:44:16.653316+0000","last_undegraded":"2024-09-16T15:44:16.653316+0000","last_fullsized":"2024-09-16T15:44:16.653316+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:13.453120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:13.453120+0000","last_clean_scrub_stamp":"2024-09-16T15:44:13.453120+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T00:29:58.706775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1a","version":"0'0","reported_seq":13,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T15:44:16.504044+0000","last_change":"2024-09-16T15:44:15.510672+0000","last_active":"2024-09-16T15:44:16.504044+0000","last_peered":"2024-09-16T15:44:16.504044+0000","last_clean":"2024-09-16T15:44:16.504044+0000","last_became_active":"2024-09-16T15:44:15.509938+0000","last_became_peered":"2024-09-16T15:44:15.509938+0000","last_unstale":"2024-09-16T15:44:16.504044+0000","last_undegraded":"2024-09-16T15:44:16.504044+0000","last_fullsized":"2024-09-16T15:44:16.504044+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_clean_scrub_stamp":"2024-09-16T15:44:14.461845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T03:39:29.740476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"43'1","reported_seq":16,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T15:44:16.549949+0000","last_change":"2024-09-16T15:44:14.508039+0000","last_active":"2024-09-16T15:44:16.549949+0000","last_peered":"2024-09-16T15:44:16.549949+0000","last_clean":"2024-09-16T15:44:16.549949+0000","last_became_active":"2024-09-16T15:44:14.507779+0000","last_became_peered":"2024-09-16T15:44:14.507779+0000","last_unstale":"2024-09-16T15:44:16.549949+0000","last_undegraded":"2024-09-16T15:44:16.549949+0000","last_fullsized":"2024-09-16T15:44:16.549949+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:13.453120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:13.453120+0000","last_clean_scrub_stamp":"2024-09-16T15:44:13.453120+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:42:38.616006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1b","version":"0'0","reported_seq":13,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T15:44:16.653620+0000","last_change":"2024-09-16T15:44:15.536540+0000","last_active":"2024-09-16T15:44:16.653620+0000","last_peered":"2024-09-16T15:44:16.653620+0000","last_clean":"2024-09-16T15:44:16.653620+0000","last_became_active":"2024-09-16T15:44:15.535847+0000","last_became_peered":"2024-09-16T15:44:15.535847+0000","last_unstale":"2024-09-16T15:44:16.653620+0000","last_undegraded":"2024-09-16T15:44:16.653620+0000","last_fullsized":"2024-09-16T15:44:16.653620+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_clean_scrub_stamp":"2024-09-16T15:44:14.461845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:46:26.359711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1a","version":"43'1","reported_seq":16,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T15:44:16.504180+0000","last_change":"2024-09-16T15:44:14.511271+0000","last_active":"2024-09-16T15:44:16.504180+0000","last_peered":"2024-09-16T15:44:16.504180+0000","last_clean":"2024-09-16T15:44:16.504180+0000","last_became_active":"2024-09-16T15:44:14.510446+0000","last_became_peered":"2024-09-16T15:44:14.510446+0000","last_unstale":"2024-09-16T15:44:16.504180+0000","last_undegraded":"2024-09-16T15:44:16.504180+0000","last_fullsized":"2024-09-16T15:44:16.504180+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:13.453120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:13.453120+0000","last_clean_scrub_stamp":"2024-09-16T15:44:13.453120+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:48:57.991477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.18","version":"0'0","reported_seq":13,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T15:44:16.652817+0000","last_change":"2024-09-16T15:44:15.536188+0000","last_active":"2024-09-16T15:44:16.652817+0000","last_peered":"2024-09-16T15:44:16.652817+0000","last_clean":"2024-09-16T15:44:16.652817+0000","last_became_active":"2024-09-16T15:44:15.535809+0000","last_became_peered":"2024-09-16T15:44:15.535809+0000","last_unstale":"2024-09-16T15:44:16.652817+0000","last_undegraded":"2024-09-16T15:44:16.652817+0000","last_fullsized":"2024-09-16T15:44:16.652817+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_clean_scrub_stamp":"2024-09-16T15:44:14.461845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T23:55:49.668059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":15,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T15:44:16.653873+0000","last_change":"2024-09-16T15:44:14.506260+0000","last_active":"2024-09-16T15:44:16.653873+0000","last_peered":"2024-09-16T15:44:16.653873+0000","last_clean":"2024-09-16T15:44:16.653873+0000","last_became_active":"2024-09-16T15:44:14.505486+0000","last_became_peered":"2024-09-16T15:44:14.505486+0000","last_unstale":"2024-09-16T15:44:16.653873+0000","last_undegraded":"2024-09-16T15:44:16.653873+0000","last_fullsized":"2024-09-16T15:44:16.653873+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:13.453120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:13.453120+0000","last_clean_scrub_stamp":"2024-09-16T15:44:13.453120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T03:37:47.775188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.19","version":"0'0","reported_seq":13,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T15:44:16.504464+0000","last_change":"2024-09-16T15:44:15.531418+0000","last_active":"2024-09-16T15:44:16.504464+0000","last_peered":"2024-09-16T15:44:16.504464+0000","last_clean":"2024-09-16T15:44:16.504464+0000","last_became_active":"2024-09-16T15:44:15.530395+0000","last_became_peered":"2024-09-16T15:44:15.530395+0000","last_unstale":"2024-09-16T15:44:16.504464+0000","last_undegraded":"2024-09-16T15:44:16.504464+0000","last_fullsized":"2024-09-16T15:44:16.504464+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_clean_scrub_stamp":"2024-09-16T15:44:14.461845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T02:31:15.361033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":15,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T15:44:16.652883+0000","last_change":"2024-09-16T15:44:14.488839+0000","last_active":"2024-09-16T15:44:16.652883+0000","last_peered":"2024-09-16T15:44:16.652883+0000","last_clean":"2024-09-16T15:44:16.652883+0000","last_became_active":"2024-09-16T15:44:14.488540+0000","last_became_peered":"2024-09-16T15:44:14.488540+0000","last_unstale":"2024-09-16T15:44:16.652883+0000","last_undegraded":"2024-09-16T15:44:16.652883+0000","last_fullsized":"2024-09-16T15:44:16.652883+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:13.453120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:13.453120+0000","last_clean_scrub_stamp":"2024-09-16T15:44:13.453120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:50:46.631666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.16","version":"0'0","reported_seq":13,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T15:44:16.506346+0000","last_change":"2024-09-16T15:44:15.534329+0000","last_active":"2024-09-16T15:44:16.506346+0000","last_peered":"2024-09-16T15:44:16.506346+0000","last_clean":"2024-09-16T15:44:16.506346+0000","last_became_active":"2024-09-16T15:44:15.533566+0000","last_became_peered":"2024-09-16T15:44:15.533566+0000","last_unstale":"2024-09-16T15:44:16.506346+0000","last_undegraded":"2024-09-16T15:44:16.506346+0000","last_fullsized":"2024-09-16T15:44:16.506346+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_clean_scrub_stamp":"2024-09-16T15:44:14.461845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:11:20.170152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":15,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T15:44:16.506777+0000","last_change":"2024-09-16T15:44:14.490740+0000","last_active":"2024-09-16T15:44:16.506777+0000","last_peered":"2024-09-16T15:44:16.506777+0000","last_clean":"2024-09-16T15:44:16.506777+0000","last_became_active":"2024-09-16T15:44:14.489894+0000","last_became_peered":"2024-09-16T15:44:14.489894+0000","last_unstale":"2024-09-16T15:44:16.506777+0000","last_undegraded":"2024-09-16T15:44:16.506777+0000","last_fullsized":"2024-09-16T15:44:16.506777+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:13.453120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:13.453120+0000","last_clean_scrub_stamp":"2024-09-16T15:44:13.453120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T02:31:14.352308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.17","version":"0'0","reported_seq":13,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T15:44:16.654031+0000","last_change":"2024-09-16T15:44:15.526710+0000","last_active":"2024-09-16T15:44:16.654031+0000","last_peered":"2024-09-16T15:44:16.654031+0000","last_clean":"2024-09-16T15:44:16.654031+0000","last_became_active":"2024-09-16T15:44:15.525977+0000","last_became_peered":"2024-09-16T15:44:15.525977+0000","last_unstale":"2024-09-16T15:44:16.654031+0000","last_undegraded":"2024-09-16T15:44:16.654031+0000","last_fullsized":"2024-09-16T15:44:16.654031+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_clean_scrub_stamp":"2024-09-16T15:44:14.461845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T03:36:28.770144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":15,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T15:44:16.654360+0000","last_change":"2024-09-16T15:44:14.503126+0000","last_active":"2024-09-16T15:44:16.654360+0000","last_peered":"2024-09-16T15:44:16.654360+0000","last_clean":"2024-09-16T15:44:16.654360+0000","last_became_active":"2024-09-16T15:44:14.502758+0000","last_became_peered":"2024-09-16T15:44:14.502758+0000","last_unstale":"2024-09-16T15:44:16.654360+0000","last_undegraded":"2024-09-16T15:44:16.654360+0000","last_fullsized":"2024-09-16T15:44:16.654360+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:13.453120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:13.453120+0000","last_clean_scrub_stamp":"2024-09-16T15:44:13.453120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T22:36:40.097279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.14","version":"0'0","reported_seq":13,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T15:44:16.550558+0000","last_change":"2024-09-16T15:44:15.542103+0000","last_active":"2024-09-16T15:44:16.550558+0000","last_peered":"2024-09-16T15:44:16.550558+0000","last_clean":"2024-09-16T15:44:16.550558+0000","last_became_active":"2024-09-16T15:44:15.541491+0000","last_became_peered":"2024-09-16T15:44:15.541491+0000","last_unstale":"2024-09-16T15:44:16.550558+0000","last_undegraded":"2024-09-16T15:44:16.550558+0000","last_fullsized":"2024-09-16T15:44:16.550558+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_clean_scrub_stamp":"2024-09-16T15:44:14.461845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T01:42:34.206609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"43'3","reported_seq":18,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T15:44:16.653262+0000","last_change":"2024-09-16T15:44:14.517210+0000","last_active":"2024-09-16T15:44:16.653262+0000","last_peered":"2024-09-16T15:44:16.653262+0000","last_clean":"2024-09-16T15:44:16.653262+0000","last_became_active":"2024-09-16T15:44:14.516535+0000","last_became_peered":"2024-09-16T15:44:14.516535+0000","last_unstale":"2024-09-16T15:44:16.653262+0000","last_undegraded":"2024-09-16T15:44:16.653262+0000","last_fullsized":"2024-09-16T15:44:16.653262+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:13.453120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:13.453120+0000","last_clean_scrub_stamp":"2024-09-16T15:44:13.453120+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T01:56:31.084287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.15","version":"0'0","reported_seq":13,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T15:44:16.652950+0000","last_change":"2024-09-16T15:44:15.540960+0000","last_active":"2024-09-16T15:44:16.652950+0000","last_peered":"2024-09-16T15:44:16.652950+0000","last_clean":"2024-09-16T15:44:16.652950+0000","last_became_active":"2024-09-16T15:44:15.540281+0000","last_became_peered":"2024-09-16T15:44:15.540281+0000","last_unstale":"2024-09-16T15:44:16.652950+0000","last_undegraded":"2024-09-16T15:44:16.652950+0000","last_fullsized":"2024-09-16T15:44:16.652950+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_clean_scrub_stamp":"2024-09-16T15:44:14.461845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:05:49.391359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.14","version":"43'6","reported_seq":21,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T15:44:16.504331+0000","last_change":"2024-09-16T15:44:14.492992+0000","last_active":"2024-09-16T15:44:16.504331+0000","last_peered":"2024-09-16T15:44:16.504331+0000","last_clean":"2024-09-16T15:44:16.504331+0000","last_became_active":"2024-09-16T15:44:14.492582+0000","last_became_peered":"2024-09-16T15:44:14.492582+0000","last_unstale":"2024-09-16T15:44:16.504331+0000","last_undegraded":"2024-09-16T15:44:16.504331+0000","last_fullsized":"2024-09-16T15:44:16.504331+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:13.453120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:13.453120+0000","last_clean_scrub_stamp":"2024-09-16T15:44:13.453120+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:45:46.523676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.12","version":"0'0","reported_seq":13,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T15:44:16.505918+0000","last_change":"2024-09-16T15:44:15.522400+0000","last_active":"2024-09-16T15:44:16.505918+0000","last_peered":"2024-09-16T15:44:16.505918+0000","last_clean":"2024-09-16T15:44:16.505918+0000","last_became_active":"2024-09-16T15:44:15.521571+0000","last_became_peered":"2024-09-16T15:44:15.521571+0000","last_unstale":"2024-09-16T15:44:16.505918+0000","last_undegraded":"2024-09-16T15:44:16.505918+0000","last_fullsized":"2024-09-16T15:44:16.505918+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_clean_scrub_stamp":"2024-09-16T15:44:14.461845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T02:53:19.885694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.13","version":"43'2","reported_seq":17,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T15:44:16.549959+0000","last_change":"2024-09-16T15:44:14.485688+0000","last_active":"2024-09-16T15:44:16.549959+0000","last_peered":"2024-09-16T15:44:16.549959+0000","last_clean":"2024-09-16T15:44:16.549959+0000","last_became_active":"2024-09-16T15:44:14.485333+0000","last_became_peered":"2024-09-16T15:44:14.485333+0000","last_unstale":"2024-09-16T15:44:16.549959+0000","last_undegraded":"2024-09-16T15:44:16.549959+0000","last_fullsized":"2024-09-16T15:44:16.549959+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:13.453120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:13.453120+0000","last_clean_scrub_stamp":"2024-09-16T15:44:13.453120+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T03:06:47.174818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.13","version":"0'0","reported_seq":13,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T15:44:16.654289+0000","last_change":"2024-09-16T15:44:15.527389+0000","last_active":"2024-09-16T15:44:16.654289+0000","last_peered":"2024-09-16T15:44:16.654289+0000","last_clean":"2024-09-16T15:44:16.654289+0000","last_became_active":"2024-09-16T15:44:15.526183+0000","last_became_peered":"2024-09-16T15:44:15.526183+0000","last_unstale":"2024-09-16T15:44:16.654289+0000","last_undegraded":"2024-09-16T15:44:16.654289+0000","last_fullsized":"2024-09-16T15:44:16.654289+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_clean_scrub_stamp":"2024-09-16T15:44:14.461845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T22:51:23.104520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":15,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T15:44:16.653537+0000","last_change":"2024-09-16T15:44:14.515153+0000","last_active":"2024-09-16T15:44:16.653537+0000","last_peered":"2024-09-16T15:44:16.653537+0000","last_clean":"2024-09-16T15:44:16.653537+0000","last_became_active":"2024-09-16T15:44:14.514343+0000","last_became_peered":"2024-09-16T15:44:14.514343+0000","last_unstale":"2024-09-16T15:44:16.653537+0000","last_undegraded":"2024-09-16T15:44:16.653537+0000","last_fullsized":"2024-09-16T15:44:16.653537+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:13.453120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:13.453120+0000","last_clean_scrub_stamp":"2024-09-16T15:44:13.453120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:14:32.419022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.10","version":"0'0","reported_seq":13,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T15:44:16.549177+0000","last_change":"2024-09-16T15:44:15.531666+0000","last_active":"2024-09-16T15:44:16.549177+0000","last_peered":"2024-09-16T15:44:16.549177+0000","last_clean":"2024-09-16T15:44:16.549177+0000","last_became_active":"2024-09-16T15:44:15.530873+0000","last_became_peered":"2024-09-16T15:44:15.530873+0000","last_unstale":"2024-09-16T15:44:16.549177+0000","last_undegraded":"2024-09-16T15:44:16.549177+0000","last_fullsized":"2024-09-16T15:44:16.549177+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_clean_scrub_stamp":"2024-09-16T15:44:14.461845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T18:25:12.406909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":15,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T15:44:16.549547+0000","last_change":"2024-09-16T15:44:14.482723+0000","last_active":"2024-09-16T15:44:16.549547+0000","last_peered":"2024-09-16T15:44:16.549547+0000","last_clean":"2024-09-16T15:44:16.549547+0000","last_became_active":"2024-09-16T15:44:14.482303+0000","last_became_peered":"2024-09-16T15:44:14.482303+0000","last_unstale":"2024-09-16T15:44:16.549547+0000","last_undegraded":"2024-09-16T15:44:16.549547+0000","last_fullsized":"2024-09-16T15:44:16.549547+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:13.453120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:13.453120+0000","last_clean_scrub_stamp":"2024-09-16T15:44:13.453120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:37:16.748939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.11","version":"0'0","reported_seq":13,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T15:44:16.653156+0000","last_change":"2024-09-16T15:44:15.537537+0000","last_active":"2024-09-16T15:44:16.653156+0000","last_peered":"2024-09-16T15:44:16.653156+0000","last_clean":"2024-09-16T15:44:16.653156+0000","last_became_active":"2024-09-16T15:44:15.535360+0000","last_became_peered":"2024-09-16T15:44:15.535360+0000","last_unstale":"2024-09-16T15:44:16.653156+0000","last_undegraded":"2024-09-16T15:44:16.653156+0000","last_fullsized":"2024-09-16T15:44:16.653156+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_clean_scrub_stamp":"2024-09-16T15:44:14.461845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T00:22:32.889458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.10","version":"43'8","reported_seq":23,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T15:44:16.652982+0000","last_change":"2024-09-16T15:44:14.506263+0000","last_active":"2024-09-16T15:44:16.652982+0000","last_peered":"2024-09-16T15:44:16.652982+0000","last_clean":"2024-09-16T15:44:16.652982+0000","last_became_active":"2024-09-16T15:44:14.505781+0000","last_became_peered":"2024-09-16T15:44:14.505781+0000","last_unstale":"2024-09-16T15:44:16.652982+0000","last_undegraded":"2024-09-16T15:44:16.652982+0000","last_fullsized":"2024-09-16T15:44:16.652982+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:13.453120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:13.453120+0000","last_clean_scrub_stamp":"2024-09-16T15:44:13.453120+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T00:56:16.749454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.e","version":"0'0","reported_seq":13,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T15:44:16.505163+0000","last_change":"2024-09-16T15:44:15.523301+0000","last_active":"2024-09-16T15:44:16.505163+0000","last_peered":"2024-09-16T15:44:16.505163+0000","last_clean":"2024-09-16T15:44:16.505163+0000","last_became_active":"2024-09-16T15:44:15.522573+0000","last_became_peered":"2024-09-16T15:44:15.522573+0000","last_unstale":"2024-09-16T15:44:16.505163+0000","last_undegraded":"2024-09-16T15:44:16.505163+0000","last_fullsized":"2024-09-16T15:44:16.505163+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_clean_scrub_stamp":"2024-09-16T15:44:14.461845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T02:35:54.221641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.f","version":"43'1","reported_seq":16,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T15:44:16.504522+0000","last_change":"2024-09-16T15:44:14.484260+0000","last_active":"2024-09-16T15:44:16.504522+0000","last_peered":"2024-09-16T15:44:16.504522+0000","last_clean":"2024-09-16T15:44:16.504522+0000","last_became_active":"2024-09-16T15:44:14.483881+0000","last_became_peered":"2024-09-16T15:44:14.483881+0000","last_unstale":"2024-09-16T15:44:16.504522+0000","last_undegraded":"2024-09-16T15:44:16.504522+0000","last_fullsized":"2024-09-16T15:44:16.504522+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:13.453120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:13.453120+0000","last_clean_scrub_stamp":"2024-09-16T15:44:13.453120+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T03:04:45.230921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.f","version":"0'0","reported_seq":13,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T15:44:16.654411+0000","last_change":"2024-09-16T15:44:15.526866+0000","last_active":"2024-09-16T15:44:16.654411+0000","last_peered":"2024-09-16T15:44:16.654411+0000","last_clean":"2024-09-16T15:44:16.654411+0000","last_became_active":"2024-09-16T15:44:15.525312+0000","last_became_peered":"2024-09-16T15:44:15.525312+0000","last_unstale":"2024-09-16T15:44:16.654411+0000","last_undegraded":"2024-09-16T15:44:16.654411+0000","last_fullsized":"2024-09-16T15:44:16.654411+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_clean_scrub_stamp":"2024-09-16T15:44:14.461845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T02:48:45.627033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.e","version":"43'2","reported_seq":17,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T15:44:16.506432+0000","last_change":"2024-09-16T15:44:14.493083+0000","last_active":"2024-09-16T15:44:16.506432+0000","last_peered":"2024-09-16T15:44:16.506432+0000","last_clean":"2024-09-16T15:44:16.506432+0000","last_became_active":"2024-09-16T15:44:14.490551+0000","last_became_peered":"2024-09-16T15:44:14.490551+0000","last_unstale":"2024-09-16T15:44:16.506432+0000","last_undegraded":"2024-09-16T15:44:16.506432+0000","last_fullsized":"2024-09-16T15:44:16.506432+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:13.453120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:13.453120+0000","last_clean_scrub_stamp":"2024-09-16T15:44:13.453120+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:58:17.159907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.c","version":"0'0","reported_seq":13,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T15:44:16.507179+0000","last_change":"2024-09-16T15:44:15.523446+0000","last_active":"2024-09-16T15:44:16.507179+0000","last_peered":"2024-09-16T15:44:16.507179+0000","last_clean":"2024-09-16T15:44:16.507179+0000","last_became_active":"2024-09-16T15:44:15.522397+0000","last_became_peered":"2024-09-16T15:44:15.522397+0000","last_unstale":"2024-09-16T15:44:16.507179+0000","last_undegraded":"2024-09-16T15:44:16.507179+0000","last_fullsized":"2024-09-16T15:44:16.507179+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_clean_scrub_stamp":"2024-09-16T15:44:14.461845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T22:19:41.560457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.d","version":"43'1","reported_seq":16,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T15:44:16.654109+0000","last_change":"2024-09-16T15:44:14.501835+0000","last_active":"2024-09-16T15:44:16.654109+0000","last_peered":"2024-09-16T15:44:16.654109+0000","last_clean":"2024-09-16T15:44:16.654109+0000","last_became_active":"2024-09-16T15:44:14.501070+0000","last_became_peered":"2024-09-16T15:44:14.501070+0000","last_unstale":"2024-09-16T15:44:16.654109+0000","last_undegraded":"2024-09-16T15:44:16.654109+0000","last_fullsized":"2024-09-16T15:44:16.654109+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:13.453120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:13.453120+0000","last_clean_scrub_stamp":"2024-09-16T15:44:13.453120+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:48:48.524239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.d","version":"0'0","reported_seq":13,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T15:44:16.507609+0000","last_change":"2024-09-16T15:44:15.521313+0000","last_active":"2024-09-16T15:44:16.507609+0000","last_peered":"2024-09-16T15:44:16.507609+0000","last_clean":"2024-09-16T15:44:16.507609+0000","last_became_active":"2024-09-16T15:44:15.517446+0000","last_became_peered":"2024-09-16T15:44:15.517446+0000","last_unstale":"2024-09-16T15:44:16.507609+0000","last_undegraded":"2024-09-16T15:44:16.507609+0000","last_fullsized":"2024-09-16T15:44:16.507609+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_clean_scrub_stamp":"2024-09-16T15:44:14.461845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:44:09.363616+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":15,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T15:44:16.550258+0000","last_change":"2024-09-16T15:44:14.496657+0000","last_active":"2024-09-16T15:44:16.550258+0000","last_peered":"2024-09-16T15:44:16.550258+0000","last_clean":"2024-09-16T15:44:16.550258+0000","last_became_active":"2024-09-16T15:44:14.496434+0000","last_became_peered":"2024-09-16T15:44:14.496434+0000","last_unstale":"2024-09-16T15:44:16.550258+0000","last_undegraded":"2024-09-16T15:44:16.550258+0000","last_fullsized":"2024-09-16T15:44:16.550258+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:13.453120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:13.453120+0000","last_clean_scrub_stamp":"2024-09-16T15:44:13.453120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:30:11.074942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.a","version":"0'0","reported_seq":13,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T15:44:16.549518+0000","last_change":"2024-09-16T15:44:15.537013+0000","last_active":"2024-09-16T15:44:16.549518+0000","last_peered":"2024-09-16T15:44:16.549518+0000","last_clean":"2024-09-16T15:44:16.549518+0000","last_became_active":"2024-09-16T15:44:15.536560+0000","last_became_peered":"2024-09-16T15:44:15.536560+0000","last_unstale":"2024-09-16T15:44:16.549518+0000","last_undegraded":"2024-09-16T15:44:16.549518+0000","last_fullsized":"2024-09-16T15:44:16.549518+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_clean_scrub_stamp":"2024-09-16T15:44:14.461845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:30:39.696556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.b","version":"43'2","reported_seq":17,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T15:44:16.550286+0000","last_change":"2024-09-16T15:44:14.508444+0000","last_active":"2024-09-16T15:44:16.550286+0000","last_peered":"2024-09-16T15:44:16.550286+0000","last_clean":"2024-09-16T15:44:16.550286+0000","last_became_active":"2024-09-16T15:44:14.507754+0000","last_became_peered":"2024-09-16T15:44:14.507754+0000","last_unstale":"2024-09-16T15:44:16.550286+0000","last_undegraded":"2024-09-16T15:44:16.550286+0000","last_fullsized":"2024-09-16T15:44:16.550286+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:13.453120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:13.453120+0000","last_clean_scrub_stamp":"2024-09-16T15:44:13.453120+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:39:14.010886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.b","version":"0'0","reported_seq":13,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T15:44:16.652071+0000","last_change":"2024-09-16T15:44:15.505327+0000","last_active":"2024-09-16T15:44:16.652071+0000","last_peered":"2024-09-16T15:44:16.652071+0000","last_clean":"2024-09-16T15:44:16.652071+0000","last_became_active":"2024-09-16T15:44:15.504567+0000","last_became_peered":"2024-09-16T15:44:15.504567+0000","last_unstale":"2024-09-16T15:44:16.652071+0000","last_undegraded":"2024-09-16T15:44:16.652071+0000","last_fullsized":"2024-09-16T15:44:16.652071+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_clean_scrub_stamp":"2024-09-16T15:44:14.461845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:59:23.737060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":15,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T15:44:16.655125+0000","last_change":"2024-09-16T15:44:14.502790+0000","last_active":"2024-09-16T15:44:16.655125+0000","last_peered":"2024-09-16T15:44:16.655125+0000","last_clean":"2024-09-16T15:44:16.655125+0000","last_became_active":"2024-09-16T15:44:14.502001+0000","last_became_peered":"2024-09-16T15:44:14.502001+0000","last_unstale":"2024-09-16T15:44:16.655125+0000","last_undegraded":"2024-09-16T15:44:16.655125+0000","last_fullsized":"2024-09-16T15:44:16.655125+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:13.453120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:13.453120+0000","last_clean_scrub_stamp":"2024-09-16T15:44:13.453120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T01:15:18.086313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.8","version":"0'0","reported_seq":13,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T15:44:16.504441+0000","last_change":"2024-09-16T15:44:15.528347+0000","last_active":"2024-09-16T15:44:16.504441+0000","last_peered":"2024-09-16T15:44:16.504441+0000","last_clean":"2024-09-16T15:44:16.504441+0000","last_became_active":"2024-09-16T15:44:15.527658+0000","last_became_peered":"2024-09-16T15:44:15.527658+0000","last_unstale":"2024-09-16T15:44:16.504441+0000","last_undegraded":"2024-09-16T15:44:16.504441+0000","last_fullsized":"2024-09-16T15:44:16.504441+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_clean_scrub_stamp":"2024-09-16T15:44:14.461845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:23:44.478108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":15,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T15:44:16.653278+0000","last_change":"2024-09-16T15:44:14.505897+0000","last_active":"2024-09-16T15:44:16.653278+0000","last_peered":"2024-09-16T15:44:16.653278+0000","last_clean":"2024-09-16T15:44:16.653278+0000","last_became_active":"2024-09-16T15:44:14.505337+0000","last_became_peered":"2024-09-16T15:44:14.505337+0000","last_unstale":"2024-09-16T15:44:16.653278+0000","last_undegraded":"2024-09-16T15:44:16.653278+0000","last_fullsized":"2024-09-16T15:44:16.653278+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:13.453120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:13.453120+0000","last_clean_scrub_stamp":"2024-09-16T15:44:13.453120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:20:28.620435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.9","version":"0'0","reported_seq":13,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T15:44:16.652951+0000","last_change":"2024-09-16T15:44:15.519005+0000","last_active":"2024-09-16T15:44:16.652951+0000","last_peered":"2024-09-16T15:44:16.652951+0000","last_clean":"2024-09-16T15:44:16.652951+0000","last_became_active":"2024-09-16T15:44:15.518183+0000","last_became_peered":"2024-09-16T15:44:15.518183+0000","last_unstale":"2024-09-16T15:44:16.652951+0000","last_undegraded":"2024-09-16T15:44:16.652951+0000","last_fullsized":"2024-09-16T15:44:16.652951+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_clean_scrub_stamp":"2024-09-16T15:44:14.461845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:30:38.888326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":15,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T15:44:16.652207+0000","last_change":"2024-09-16T15:44:14.509676+0000","last_active":"2024-09-16T15:44:16.652207+0000","last_peered":"2024-09-16T15:44:16.652207+0000","last_clean":"2024-09-16T15:44:16.652207+0000","last_became_active":"2024-09-16T15:44:14.508922+0000","last_became_peered":"2024-09-16T15:44:14.508922+0000","last_unstale":"2024-09-16T15:44:16.652207+0000","last_undegraded":"2024-09-16T15:44:16.652207+0000","last_fullsized":"2024-09-16T15:44:16.652207+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:13.453120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:13.453120+0000","last_clean_scrub_stamp":"2024-09-16T15:44:13.453120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:48:48.524239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.7","version":"0'0","reported_seq":13,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T15:44:16.508313+0000","last_change":"2024-09-16T15:44:15.522383+0000","last_active":"2024-09-16T15:44:16.508313+0000","last_peered":"2024-09-16T15:44:16.508313+0000","last_clean":"2024-09-16T15:44:16.508313+0000","last_became_active":"2024-09-16T15:44:15.521677+0000","last_became_peered":"2024-09-16T15:44:15.521677+0000","last_unstale":"2024-09-16T15:44:16.508313+0000","last_undegraded":"2024-09-16T15:44:16.508313+0000","last_fullsized":"2024-09-16T15:44:16.508313+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_clean_scrub_stamp":"2024-09-16T15:44:14.461845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:58:42.373023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.6","version":"44'1","reported_seq":16,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T15:44:16.549512+0000","last_change":"2024-09-16T15:44:14.492127+0000","last_active":"2024-09-16T15:44:16.549512+0000","last_peered":"2024-09-16T15:44:16.549512+0000","last_clean":"2024-09-16T15:44:16.549512+0000","last_became_active":"2024-09-16T15:44:14.491705+0000","last_became_peered":"2024-09-16T15:44:14.491705+0000","last_unstale":"2024-09-16T15:44:16.549512+0000","last_undegraded":"2024-09-16T15:44:16.549512+0000","last_fullsized":"2024-09-16T15:44:16.549512+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:13.453120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:13.453120+0000","last_clean_scrub_stamp":"2024-09-16T15:44:13.453120+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T18:58:30.998947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.4","version":"0'0","reported_seq":13,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T15:44:16.654140+0000","last_change":"2024-09-16T15:44:15.529545+0000","last_active":"2024-09-16T15:44:16.654140+0000","last_peered":"2024-09-16T15:44:16.654140+0000","last_clean":"2024-09-16T15:44:16.654140+0000","last_became_active":"2024-09-16T15:44:15.528768+0000","last_became_peered":"2024-09-16T15:44:15.528768+0000","last_unstale":"2024-09-16T15:44:16.654140+0000","last_undegraded":"2024-09-16T15:44:16.654140+0000","last_fullsized":"2024-09-16T15:44:16.654140+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_clean_scrub_stamp":"2024-09-16T15:44:14.461845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:41:16.229234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":15,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T15:44:16.653545+0000","last_change":"2024-09-16T15:44:14.518542+0000","last_active":"2024-09-16T15:44:16.653545+0000","last_peered":"2024-09-16T15:44:16.653545+0000","last_clean":"2024-09-16T15:44:16.653545+0000","last_became_active":"2024-09-16T15:44:14.517771+0000","last_became_peered":"2024-09-16T15:44:14.517771+0000","last_unstale":"2024-09-16T15:44:16.653545+0000","last_undegraded":"2024-09-16T15:44:16.653545+0000","last_fullsized":"2024-09-16T15:44:16.653545+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:13.453120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:13.453120+0000","last_clean_scrub_stamp":"2024-09-16T15:44:13.453120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:01:58.080704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.5","version":"0'0","reported_seq":13,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T15:44:16.653096+0000","last_change":"2024-09-16T15:44:15.534669+0000","last_active":"2024-09-16T15:44:16.653096+0000","last_peered":"2024-09-16T15:44:16.653096+0000","last_clean":"2024-09-16T15:44:16.653096+0000","last_became_active":"2024-09-16T15:44:15.533906+0000","last_became_peered":"2024-09-16T15:44:15.533906+0000","last_unstale":"2024-09-16T15:44:16.653096+0000","last_undegraded":"2024-09-16T15:44:16.653096+0000","last_fullsized":"2024-09-16T15:44:16.653096+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_clean_scrub_stamp":"2024-09-16T15:44:14.461845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T18:17:57.818319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":15,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T15:44:16.651470+0000","last_change":"2024-09-16T15:44:14.506407+0000","last_active":"2024-09-16T15:44:16.651470+0000","last_peered":"2024-09-16T15:44:16.651470+0000","last_clean":"2024-09-16T15:44:16.651470+0000","last_became_active":"2024-09-16T15:44:14.505653+0000","last_became_peered":"2024-09-16T15:44:14.505653+0000","last_unstale":"2024-09-16T15:44:16.651470+0000","last_undegraded":"2024-09-16T15:44:16.651470+0000","last_fullsized":"2024-09-16T15:44:16.651470+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:13.453120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:13.453120+0000","last_clean_scrub_stamp":"2024-09-16T15:44:13.453120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:54:55.171756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.2","version":"0'0","reported_seq":13,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T15:44:16.550033+0000","last_change":"2024-09-16T15:44:15.522529+0000","last_active":"2024-09-16T15:44:16.550033+0000","last_peered":"2024-09-16T15:44:16.550033+0000","last_clean":"2024-09-16T15:44:16.550033+0000","last_became_active":"2024-09-16T15:44:15.521784+0000","last_became_peered":"2024-09-16T15:44:15.521784+0000","last_unstale":"2024-09-16T15:44:16.550033+0000","last_undegraded":"2024-09-16T15:44:16.550033+0000","last_fullsized":"2024-09-16T15:44:16.550033+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_clean_scrub_stamp":"2024-09-16T15:44:14.461845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:08:45.487688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":15,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T15:44:16.653189+0000","last_change":"2024-09-16T15:44:14.517206+0000","last_active":"2024-09-16T15:44:16.653189+0000","last_peered":"2024-09-16T15:44:16.653189+0000","last_clean":"2024-09-16T15:44:16.653189+0000","last_became_active":"2024-09-16T15:44:14.516154+0000","last_became_peered":"2024-09-16T15:44:14.516154+0000","last_unstale":"2024-09-16T15:44:16.653189+0000","last_undegraded":"2024-09-16T15:44:16.653189+0000","last_fullsized":"2024-09-16T15:44:16.653189+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:13.453120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:13.453120+0000","last_clean_scrub_stamp":"2024-09-16T15:44:13.453120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T00:26:35.189372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.3","version":"0'0","reported_seq":13,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T15:44:16.549762+0000","last_change":"2024-09-16T15:44:15.537688+0000","last_active":"2024-09-16T15:44:16.549762+0000","last_peered":"2024-09-16T15:44:16.549762+0000","last_clean":"2024-09-16T15:44:16.549762+0000","last_became_active":"2024-09-16T15:44:15.536928+0000","last_became_peered":"2024-09-16T15:44:15.536928+0000","last_unstale":"2024-09-16T15:44:16.549762+0000","last_undegraded":"2024-09-16T15:44:16.549762+0000","last_fullsized":"2024-09-16T15:44:16.549762+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_clean_scrub_stamp":"2024-09-16T15:44:14.461845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:28:18.899130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":15,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T15:44:16.504248+0000","last_change":"2024-09-16T15:44:14.485689+0000","last_active":"2024-09-16T15:44:16.504248+0000","last_peered":"2024-09-16T15:44:16.504248+0000","last_clean":"2024-09-16T15:44:16.504248+0000","last_became_active":"2024-09-16T15:44:14.484809+0000","last_became_peered":"2024-09-16T15:44:14.484809+0000","last_unstale":"2024-09-16T15:44:16.504248+0000","last_undegraded":"2024-09-16T15:44:16.504248+0000","last_fullsized":"2024-09-16T15:44:16.504248+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:13.453120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:13.453120+0000","last_clean_scrub_stamp":"2024-09-16T15:44:13.453120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:28:21.385029+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.0","version":"0'0","reported_seq":13,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T15:44:16.653143+0000","last_change":"2024-09-16T15:44:15.519845+0000","last_active":"2024-09-16T15:44:16.653143+0000","last_peered":"2024-09-16T15:44:16.653143+0000","last_clean":"2024-09-16T15:44:16.653143+0000","last_became_active":"2024-09-16T15:44:15.519060+0000","last_became_peered":"2024-09-16T15:44:15.519060+0000","last_unstale":"2024-09-16T15:44:16.653143+0000","last_undegraded":"2024-09-16T15:44:16.653143+0000","last_fullsized":"2024-09-16T15:44:16.653143+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_clean_scrub_stamp":"2024-09-16T15:44:14.461845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T18:00:03.385301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":15,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T15:44:16.651501+0000","last_change":"2024-09-16T15:44:14.498147+0000","last_active":"2024-09-16T15:44:16.651501+0000","last_peered":"2024-09-16T15:44:16.651501+0000","last_clean":"2024-09-16T15:44:16.651501+0000","last_became_active":"2024-09-16T15:44:14.497623+0000","last_became_peered":"2024-09-16T15:44:14.497623+0000","last_unstale":"2024-09-16T15:44:16.651501+0000","last_undegraded":"2024-09-16T15:44:16.651501+0000","last_fullsized":"2024-09-16T15:44:16.651501+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:13.453120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:13.453120+0000","last_clean_scrub_stamp":"2024-09-16T15:44:13.453120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T00:56:50.471500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.1","version":"0'0","reported_seq":13,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T15:44:16.505707+0000","last_change":"2024-09-16T15:44:15.532828+0000","last_active":"2024-09-16T15:44:16.505707+0000","last_peered":"2024-09-16T15:44:16.505707+0000","last_clean":"2024-09-16T15:44:16.505707+0000","last_became_active":"2024-09-16T15:44:15.531936+0000","last_became_peered":"2024-09-16T15:44:15.531936+0000","last_unstale":"2024-09-16T15:44:16.505707+0000","last_undegraded":"2024-09-16T15:44:16.505707+0000","last_fullsized":"2024-09-16T15:44:16.505707+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_clean_scrub_stamp":"2024-09-16T15:44:14.461845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T00:52:44.476756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":15,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T15:44:16.653109+0000","last_change":"2024-09-16T15:44:14.489736+0000","last_active":"2024-09-16T15:44:16.653109+0000","last_peered":"2024-09-16T15:44:16.653109+0000","last_clean":"2024-09-16T15:44:16.653109+0000","last_became_active":"2024-09-16T15:44:14.489399+0000","last_became_peered":"2024-09-16T15:44:14.489399+0000","last_unstale":"2024-09-16T15:44:16.653109+0000","last_undegraded":"2024-09-16T15:44:16.653109+0000","last_fullsized":"2024-09-16T15:44:16.653109+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:13.453120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:13.453120+0000","last_clean_scrub_stamp":"2024-09-16T15:44:13.453120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:33:25.416197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.6","version":"0'0","reported_seq":13,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T15:44:16.654396+0000","last_change":"2024-09-16T15:44:15.529347+0000","last_active":"2024-09-16T15:44:16.654396+0000","last_peered":"2024-09-16T15:44:16.654396+0000","last_clean":"2024-09-16T15:44:16.654396+0000","last_became_active":"2024-09-16T15:44:15.527433+0000","last_became_peered":"2024-09-16T15:44:15.527433+0000","last_unstale":"2024-09-16T15:44:16.654396+0000","last_undegraded":"2024-09-16T15:44:16.654396+0000","last_fullsized":"2024-09-16T15:44:16.654396+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:14.461845+0000","last_clean_scrub_stamp":"2024-09-16T15:44:14.461845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:06:52.956916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.7","version":"43'2","reported_seq":17,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T15:44:16.654489+0000","last_change":"2024-09-16T15:44:14.499055+0000","last_active":"2024-09-16T15:44:16.654489+0000","last_peered":"2024-09-16T15:44:16.654489+0000","last_clean":"2024-09-16T15:44:16.654489+0000","last_became_active":"2024-09-16T15:44:14.498247+0000","last_became_peered":"2024-09-16T15:44:14.498247+0000","last_unstale":"2024-09-16T15:44:16.654489+0000","last_undegraded":"2024-09-16T15:44:16.654489+0000","last_fullsized":"2024-09-16T15:44:16.654489+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:44:13.453120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:44:13.453120+0000","last_clean_scrub_stamp":"2024-09-16T15:44:13.453120+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:14:32.419022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"16'72","reported_seq":153,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T15:44:16.652344+0000","last_change":"2024-09-16T15:42:40.881820+0000","last_active":"2024-09-16T15:44:16.652344+0000","last_peered":"2024-09-16T15:44:16.652344+0000","last_clean":"2024-09-16T15:44:16.652344+0000","last_became_active":"2024-09-16T15:42:40.879965+0000","last_became_peered":"2024-09-16T15:42:40.879965+0000","last_unstale":"2024-09-16T15:44:16.652344+0000","last_undegraded":"2024-09-16T15:44:16.652344+0000","last_fullsized":"2024-09-16T15:44:16.652344+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:42:39.728762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:42:39.728762+0000","last_clean_scrub_stamp":"2024-09-16T15:42:39.728762+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:22:18.768932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542168,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28116,"kb_used_data":1084,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343724,"statfs":{"total":96636764160,"available":96607973376,"internally_reserved":0,"allocated":1110016,"data_stored":796246,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542167,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27544,"kb_used_data":508,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344296,"statfs":{"total":96636764160,"available":96608559104,"internally_reserved":0,"allocated":520192,"data_stored":206460,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":5,"up_from":14,"seq":60129542168,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27532,"kb_used_data":504,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344308,"statfs":{"total":96636764160,"available":96608571392,"internally_reserved":0,"allocated":516096,"data_stored":205858,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542166,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27544,"kb_used_data":512,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344296,"statfs":{"total":96636764160,"available":96608559104,"internally_reserved":0,"allocated":524288,"data_stored":207400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":3,"up_from":14,"seq":60129542167,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31704,"kb_used_data":512,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340136,"statfs":{"total":96636764160,"available":96604299264,"internally_reserved":0,"allocated":524288,"data_stored":206426,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":2,"up_from":14,"seq":60129542167,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27552,"kb_used_data":516,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344288,"statfs":{"total":96636764160,"available":96608550912,"internally_reserved":0,"allocated":528384,"data_stored":207422,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":1,"up_from":14,"seq":60129542169,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27532,"kb_used_data":504,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344308,"statfs":{"total":96636764160,"available":96608571392,"internally_reserved":0,"allocated":516096,"data_stored":205858,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":0,"up_from":14,"seq":60129542169,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28116,"kb_used_data":1084,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343724,"statfs":{"total":96636764160,"available":96607973376,"internally_reserved":0,"allocated":1110016,"data_stored":796190,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T15:44:24.386 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-09-16T15:44:24.386 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-09-16T15:44:24.562 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:24.555+0000 7f7945203640 1 -- 172.21.15.123:0/479653923 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f79400cc3c0 msgr2=0x7f79400cc850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:44:24.562 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:24.555+0000 7f7945203640 1 --2- 172.21.15.123:0/479653923 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f79400cc3c0 0x7f79400cc850 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f7938009bc0 tx=0x7f793801cba0 comp rx=0 tx=0).stop 2024-09-16T15:44:24.562 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:24.555+0000 7f7945203640 1 -- 172.21.15.123:0/479653923 shutdown_connections 2024-09-16T15:44:24.562 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:24.555+0000 7f7945203640 1 --2- 172.21.15.123:0/479653923 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f79400cc3c0 0x7f79400cc850 unknown :-1 s=CLOSED pgs=468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:24.562 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:24.555+0000 7f7945203640 1 --2- 172.21.15.123:0/479653923 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f7940072720 0x7f794015e5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:24.562 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:24.555+0000 7f7945203640 1 -- 172.21.15.123:0/479653923 >> 172.21.15.123:0/479653923 conn(0x7f79400b1710 msgr2=0x7f79400b1b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:44:24.562 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:24.555+0000 7f7945203640 1 -- 172.21.15.123:0/479653923 shutdown_connections 2024-09-16T15:44:24.562 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:24.555+0000 7f7945203640 1 -- 172.21.15.123:0/479653923 wait complete. 2024-09-16T15:44:24.563 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:24.555+0000 7f7945203640 1 Processor -- start 2024-09-16T15:44:24.563 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:24.555+0000 7f7945203640 1 -- start start 2024-09-16T15:44:24.563 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:24.559+0000 7f7945203640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7940072720 0x7f7940155be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:44:24.564 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:24.559+0000 7f7945203640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f79400cc3c0 0x7f7940159020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:44:24.564 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:24.559+0000 7f7945203640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f794015aa00 0x7f7940159560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:44:24.564 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:24.559+0000 7f7945203640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f7940167af0 con 0x7f794015aa00 2024-09-16T15:44:24.564 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:24.559+0000 7f7945203640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f7940167970 con 0x7f7940072720 2024-09-16T15:44:24.564 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:24.559+0000 7f7945203640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f7940167c70 con 0x7f79400cc3c0 2024-09-16T15:44:24.564 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:24.559+0000 7f793f59e640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f794015aa00 0x7f7940159560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:44:24.564 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:24.559+0000 7f793f59e640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f794015aa00 0x7f7940159560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:38826/0 (socket says 172.21.15.123:38826) 2024-09-16T15:44:24.564 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:24.559+0000 7f793f59e640 1 -- 172.21.15.123:0/970774609 learned_addr learned my addr 172.21.15.123:0/970774609 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:44:24.565 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:24.559+0000 7f793f59e640 1 -- 172.21.15.123:0/970774609 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f79400cc3c0 msgr2=0x7f7940159020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:44:24.565 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:24.559+0000 7f793e59c640 1 --2- 172.21.15.123:0/970774609 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f79400cc3c0 0x7f7940159020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:44:24.566 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:24.559+0000 7f793ed9d640 1 --2- 172.21.15.123:0/970774609 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7940072720 0x7f7940155be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:44:24.566 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:24.559+0000 7f793f59e640 1 --2- 172.21.15.123:0/970774609 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f79400cc3c0 0x7f7940159020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:24.566 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:24.559+0000 7f793f59e640 1 -- 172.21.15.123:0/970774609 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7940072720 msgr2=0x7f7940155be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:44:24.566 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:24.559+0000 7f793f59e640 1 --2- 172.21.15.123:0/970774609 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7940072720 0x7f7940155be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:24.566 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:24.559+0000 7f793f59e640 1 -- 172.21.15.123:0/970774609 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f79401f8f80 con 0x7f794015aa00 2024-09-16T15:44:24.566 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:24.559+0000 7f793ed9d640 1 --2- 172.21.15.123:0/970774609 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7940072720 0x7f7940155be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T15:44:24.566 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:24.559+0000 7f793f59e640 1 --2- 172.21.15.123:0/970774609 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f794015aa00 0x7f7940159560 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f79380b0040 tx=0x7f7938005b00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:44:24.566 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:24.559+0000 7f79277fe640 1 -- 172.21.15.123:0/970774609 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79380b2040 con 0x7f794015aa00 2024-09-16T15:44:24.566 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:24.559+0000 7f79277fe640 1 -- 172.21.15.123:0/970774609 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f7938004040 con 0x7f794015aa00 2024-09-16T15:44:24.566 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:24.559+0000 7f7945203640 1 -- 172.21.15.123:0/970774609 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f79401f9260 con 0x7f794015aa00 2024-09-16T15:44:24.566 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:24.559+0000 7f79277fe640 1 -- 172.21.15.123:0/970774609 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7938004430 con 0x7f794015aa00 2024-09-16T15:44:24.566 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:24.559+0000 7f7945203640 1 -- 172.21.15.123:0/970774609 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f79401f9820 con 0x7f794015aa00 2024-09-16T15:44:24.566 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:24.559+0000 7f7945203640 1 -- 172.21.15.123:0/970774609 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f794015e5d0 con 0x7f794015aa00 2024-09-16T15:44:24.567 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:24.559+0000 7f79277fe640 1 -- 172.21.15.123:0/970774609 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 7) ==== 97233+0+0 (secure 0 0 0) 0x7f79380bd030 con 0x7f794015aa00 2024-09-16T15:44:24.567 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:24.559+0000 7f79277fe640 1 --2- 172.21.15.123:0/970774609 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f792c073070 0x7f792c075530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:44:24.567 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:24.563+0000 7f79277fe640 1 -- 172.21.15.123:0/970774609 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(44..44 src has 1..44) ==== 5709+0+0 (secure 0 0 0) 0x7f793812e0a0 con 0x7f794015aa00 2024-09-16T15:44:24.567 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:24.563+0000 7f793ed9d640 1 --2- 172.21.15.123:0/970774609 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f792c073070 0x7f792c075530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:44:24.568 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:24.563+0000 7f793ed9d640 1 --2- 172.21.15.123:0/970774609 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f792c073070 0x7f792c075530 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f79400c6460 tx=0x7f7930009040 comp rx=0 tx=0).ready entity=mgr.4135 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:44:24.571 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:24.567+0000 7f79277fe640 1 -- 172.21.15.123:0/970774609 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f79380c4060 con 0x7f794015aa00 2024-09-16T15:44:24.808 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:24.803+0000 7f7945203640 1 -- 172.21.15.123:0/970774609 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) -- 0x7f79400cd440 con 0x7f794015aa00 2024-09-16T15:44:24.812 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:24.807+0000 7f79277fe640 1 -- 172.21.15.123:0/970774609 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v18) ==== 190+0+147 (secure 0 0 0) 0x7f79380a5e20 con 0x7f794015aa00 2024-09-16T15:44:24.812 INFO:teuthology.orchestra.run.smithi123.stdout:[client.0] 2024-09-16T15:44:24.812 INFO:teuthology.orchestra.run.smithi123.stdout: key = AQC1Uehm7GDTAhAAHWWVIyjVVnTeeozBR8DfFA== 2024-09-16T15:44:24.812 INFO:teuthology.orchestra.run.smithi123.stdout: caps mds = "allow" 2024-09-16T15:44:24.812 INFO:teuthology.orchestra.run.smithi123.stdout: caps mon = "allow r" 2024-09-16T15:44:24.812 INFO:teuthology.orchestra.run.smithi123.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-09-16T15:44:24.813 INFO:teuthology.orchestra.run.smithi123.stderr:updated caps for client.0 2024-09-16T15:44:24.815 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:24.807+0000 7f7945203640 1 -- 172.21.15.123:0/970774609 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f792c073070 msgr2=0x7f792c075530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:44:24.815 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:24.807+0000 7f7945203640 1 --2- 172.21.15.123:0/970774609 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f792c073070 0x7f792c075530 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f79400c6460 tx=0x7f7930009040 comp rx=0 tx=0).stop 2024-09-16T15:44:24.815 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:24.807+0000 7f7945203640 1 -- 172.21.15.123:0/970774609 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f794015aa00 msgr2=0x7f7940159560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:44:24.815 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:24.807+0000 7f7945203640 1 --2- 172.21.15.123:0/970774609 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f794015aa00 0x7f7940159560 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f79380b0040 tx=0x7f7938005b00 comp rx=0 tx=0).stop 2024-09-16T15:44:24.815 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:24.807+0000 7f7945203640 1 -- 172.21.15.123:0/970774609 shutdown_connections 2024-09-16T15:44:24.815 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:24.811+0000 7f7945203640 1 --2- 172.21.15.123:0/970774609 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f792c073070 0x7f792c075530 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:24.816 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:24.811+0000 7f7945203640 1 --2- 172.21.15.123:0/970774609 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f794015aa00 0x7f7940159560 unknown :-1 s=CLOSED pgs=469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:24.816 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:24.811+0000 7f7945203640 1 --2- 172.21.15.123:0/970774609 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f79400cc3c0 0x7f7940159020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:24.816 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:24.811+0000 7f7945203640 1 --2- 172.21.15.123:0/970774609 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7940072720 0x7f7940155be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:24.816 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:24.811+0000 7f7945203640 1 -- 172.21.15.123:0/970774609 >> 172.21.15.123:0/970774609 conn(0x7f79400b1710 msgr2=0x7f79400b5080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:44:24.816 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:24.811+0000 7f7945203640 1 -- 172.21.15.123:0/970774609 shutdown_connections 2024-09-16T15:44:24.816 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:24.811+0000 7f7945203640 1 -- 172.21.15.123:0/970774609 wait complete. 2024-09-16T15:44:24.831 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-16T15:44:24.945 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:24.939+0000 7ff2a2564640 1 -- 172.21.15.123:0/935595549 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff29c161620 msgr2=0x7ff29c0d1da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:44:24.945 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:24.939+0000 7ff2a2564640 1 --2- 172.21.15.123:0/935595549 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff29c161620 0x7ff29c0d1da0 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7ff29c0cffa0 tx=0x7ff28c01c750 comp rx=0 tx=0).stop 2024-09-16T15:44:24.945 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:24.939+0000 7ff2a2564640 1 -- 172.21.15.123:0/935595549 shutdown_connections 2024-09-16T15:44:24.945 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:24.939+0000 7ff2a2564640 1 --2- 172.21.15.123:0/935595549 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7ff29c0d2820 0x7ff29c0cea00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:24.945 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:24.939+0000 7ff2a2564640 1 --2- 172.21.15.123:0/935595549 >> v2:172.21.15.184:3300/0 conn(0x7ff29c0b2b00 0x7ff29c0d22e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:24.946 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:24.939+0000 7ff2a2564640 1 --2- 172.21.15.123:0/935595549 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff29c161620 0x7ff29c0d1da0 unknown :-1 s=CLOSED pgs=470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:24.946 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:24.939+0000 7ff2a2564640 1 -- 172.21.15.123:0/935595549 >> 172.21.15.123:0/935595549 conn(0x7ff29c0d3ea0 msgr2=0x7ff29c0d42b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:44:24.946 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:24.939+0000 7ff2a2564640 1 -- 172.21.15.123:0/935595549 shutdown_connections 2024-09-16T15:44:24.946 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:24.939+0000 7ff2a2564640 1 -- 172.21.15.123:0/935595549 wait complete. 2024-09-16T15:44:24.946 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:24.939+0000 7ff2a2564640 1 Processor -- start 2024-09-16T15:44:24.946 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:24.939+0000 7ff2a2564640 1 -- start start 2024-09-16T15:44:24.949 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:24.943+0000 7ff2a2564640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff29c0b2b00 0x7ff29c158e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:44:24.949 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:24.943+0000 7ff2a2564640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7ff29c0d2820 0x7ff29c153630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:44:24.949 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:24.943+0000 7ff2a2564640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff29c161620 0x7ff29c159f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:44:24.949 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:24.943+0000 7ff2a2564640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7ff29c0b1030 con 0x7ff29c161620 2024-09-16T15:44:24.949 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:24.943+0000 7ff2a2564640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7ff29c0b0eb0 con 0x7ff29c0b2b00 2024-09-16T15:44:24.949 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:24.943+0000 7ff2a2564640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7ff29c0b11b0 con 0x7ff29c0d2820 2024-09-16T15:44:24.949 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:24.943+0000 7ff2a0b01640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff29c161620 0x7ff29c159f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:44:24.949 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:24.943+0000 7ff2a0b01640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff29c161620 0x7ff29c159f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:38834/0 (socket says 172.21.15.123:38834) 2024-09-16T15:44:24.952 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:24.943+0000 7ff2a0b01640 1 -- 172.21.15.123:0/3581808358 learned_addr learned my addr 172.21.15.123:0/3581808358 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:44:24.952 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:24.943+0000 7ff2a0b01640 1 -- 172.21.15.123:0/3581808358 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7ff29c0d2820 msgr2=0x7ff29c153630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:44:24.952 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:24.943+0000 7ff2a0b01640 1 --2- 172.21.15.123:0/3581808358 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7ff29c0d2820 0x7ff29c153630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:24.952 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:24.943+0000 7ff2a0b01640 1 -- 172.21.15.123:0/3581808358 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff29c0b2b00 msgr2=0x7ff29c158e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:44:24.952 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:24.943+0000 7ff2a0b01640 1 --2- 172.21.15.123:0/3581808358 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff29c0b2b00 0x7ff29c158e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:24.952 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:24.943+0000 7ff2a0b01640 1 -- 172.21.15.123:0/3581808358 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff29c157be0 con 0x7ff29c161620 2024-09-16T15:44:24.952 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:24.943+0000 7ff2a0b01640 1 --2- 172.21.15.123:0/3581808358 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff29c161620 0x7ff29c159f70 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7ff29400ea20 tx=0x7ff29400eee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:44:24.953 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:24.943+0000 7ff298ff9640 1 -- 172.21.15.123:0/3581808358 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff29400cde0 con 0x7ff29c161620 2024-09-16T15:44:24.953 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:24.943+0000 7ff298ff9640 1 -- 172.21.15.123:0/3581808358 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7ff294004c40 con 0x7ff29c161620 2024-09-16T15:44:24.953 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:24.943+0000 7ff298ff9640 1 -- 172.21.15.123:0/3581808358 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff294005280 con 0x7ff29c161620 2024-09-16T15:44:24.953 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:24.943+0000 7ff2a2564640 1 -- 172.21.15.123:0/3581808358 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff29c157ec0 con 0x7ff29c161620 2024-09-16T15:44:24.953 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:24.943+0000 7ff2a2564640 1 -- 172.21.15.123:0/3581808358 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff29c1589a0 con 0x7ff29c161620 2024-09-16T15:44:24.954 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:24.943+0000 7ff2a2564640 1 -- 172.21.15.123:0/3581808358 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff29c0cea00 con 0x7ff29c161620 2024-09-16T15:44:24.954 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:24.943+0000 7ff298ff9640 1 -- 172.21.15.123:0/3581808358 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 7) ==== 97233+0+0 (secure 0 0 0) 0x7ff294005420 con 0x7ff29c161620 2024-09-16T15:44:24.954 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:24.943+0000 7ff298ff9640 1 --2- 172.21.15.123:0/3581808358 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7ff27c0730d0 0x7ff27c075590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:44:24.954 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:24.943+0000 7ff298ff9640 1 -- 172.21.15.123:0/3581808358 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(44..44 src has 1..44) ==== 5709+0+0 (secure 0 0 0) 0x7ff294094b00 con 0x7ff29c161620 2024-09-16T15:44:24.954 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:24.943+0000 7ff29bfff640 1 --2- 172.21.15.123:0/3581808358 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7ff27c0730d0 0x7ff27c075590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:44:24.954 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:24.947+0000 7ff29bfff640 1 --2- 172.21.15.123:0/3581808358 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7ff27c0730d0 0x7ff27c075590 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7ff28c0af000 tx=0x7ff28c0194f0 comp rx=0 tx=0).ready entity=mgr.4135 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:44:24.954 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:24.947+0000 7ff298ff9640 1 -- 172.21.15.123:0/3581808358 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7ff294016030 con 0x7ff29c161620 2024-09-16T15:44:25.155 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:25.151+0000 7ff2a2564640 1 -- 172.21.15.123:0/3581808358 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff29c0b9be0 con 0x7ff29c161620 2024-09-16T15:44:25.156 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:25.151+0000 7ff298ff9640 1 -- 172.21.15.123:0/3581808358 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) ==== 94+0+5223 (secure 0 0 0) 0x7ff29405fa50 con 0x7ff29c161620 2024-09-16T15:44:25.157 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-09-16T15:44:25.157 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":27,"btime":"2024-09-16T15:44:18:509761+0000","default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4712,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6837/288264117","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6836","nonce":288264117},{"type":"v1","addr":"172.21.15.184:6837","nonce":288264117}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":23},{"gid":5268,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6834/60194515","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6832","nonce":60194515},{"type":"v1","addr":"172.21.15.123:6834","nonce":60194515}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":23},{"gid":5271,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6835/1798373479","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6833","nonce":1798373479},{"type":"v1","addr":"172.21.15.123:6835","nonce":1798373479}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":22}],"filesystems":[{"mdsmap":{"epoch":27,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-16T15:44:14.853230+0000","modified":"2024-09-16T15:44:17.861742+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5280},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5280":{"gid":5280,"name":"d","rank":0,"incarnation":24,"state":"up:active","state_seq":3,"addr":"172.21.15.184:6835/374389412","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":374389412},{"type":"v1","addr":"172.21.15.184:6835","nonce":374389412}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5280,"qdb_cluster":[5280]},"id":3}]} 2024-09-16T15:44:25.157 INFO:teuthology.orchestra.run.smithi123.stderr:dumped fsmap epoch 27 2024-09-16T15:44:25.159 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:25.151+0000 7ff2a2564640 1 -- 172.21.15.123:0/3581808358 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7ff27c0730d0 msgr2=0x7ff27c075590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:44:25.159 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:25.151+0000 7ff2a2564640 1 --2- 172.21.15.123:0/3581808358 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7ff27c0730d0 0x7ff27c075590 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7ff28c0af000 tx=0x7ff28c0194f0 comp rx=0 tx=0).stop 2024-09-16T15:44:25.159 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:25.151+0000 7ff2a2564640 1 -- 172.21.15.123:0/3581808358 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff29c161620 msgr2=0x7ff29c159f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:44:25.159 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:25.151+0000 7ff2a2564640 1 --2- 172.21.15.123:0/3581808358 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff29c161620 0x7ff29c159f70 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7ff29400ea20 tx=0x7ff29400eee0 comp rx=0 tx=0).stop 2024-09-16T15:44:25.159 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:25.151+0000 7ff2a2564640 1 -- 172.21.15.123:0/3581808358 shutdown_connections 2024-09-16T15:44:25.159 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:25.151+0000 7ff2a2564640 1 --2- 172.21.15.123:0/3581808358 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7ff27c0730d0 0x7ff27c075590 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:25.159 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:25.151+0000 7ff2a2564640 1 --2- 172.21.15.123:0/3581808358 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff29c161620 0x7ff29c159f70 unknown :-1 s=CLOSED pgs=471 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:25.159 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:25.151+0000 7ff2a2564640 1 --2- 172.21.15.123:0/3581808358 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7ff29c0d2820 0x7ff29c153630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:25.159 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:25.151+0000 7ff2a2564640 1 --2- 172.21.15.123:0/3581808358 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff29c0b2b00 0x7ff29c158e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:25.159 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:25.151+0000 7ff2a2564640 1 -- 172.21.15.123:0/3581808358 >> 172.21.15.123:0/3581808358 conn(0x7ff29c0d3ea0 msgr2=0x7ff29c0b16a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:44:25.160 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:25.151+0000 7ff2a2564640 1 -- 172.21.15.123:0/3581808358 shutdown_connections 2024-09-16T15:44:25.160 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:25.151+0000 7ff2a2564640 1 -- 172.21.15.123:0/3581808358 wait complete. 2024-09-16T15:44:25.576 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 27, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-09-16T15:44:14.853230+0000', 'modified': '2024-09-16T15:44:17.861742+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 5280}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_5280': {'gid': 5280, 'name': 'd', 'rank': 0, 'incarnation': 24, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.184:6835/374389412', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.184:6834', 'nonce': 374389412}, {'type': 'v1', 'addr': '172.21.15.184:6835', 'nonce': 374389412}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540719139924082687, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [7], 'metadata_pool': 6, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 5280, 'qdb_cluster': [5280]} 2024-09-16T15:44:25.577 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-09-16T15:44:25.577 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.d status 2024-09-16T15:44:25.711 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:25.703+0000 7f226d3ae640 1 -- 172.21.15.123:0/2525703426 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f22680cf5a0 msgr2=0x7f226806eee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:44:25.711 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:25.703+0000 7f226d3ae640 1 --2- 172.21.15.123:0/2525703426 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f22680cf5a0 0x7f226806eee0 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f225c00a950 tx=0x7f225c01da50 comp rx=0 tx=0).stop 2024-09-16T15:44:25.711 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:25.703+0000 7f226d3ae640 1 -- 172.21.15.123:0/2525703426 shutdown_connections 2024-09-16T15:44:25.712 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:25.703+0000 7f226d3ae640 1 --2- 172.21.15.123:0/2525703426 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f226815ec40 0x7f226806f420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:25.712 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:25.703+0000 7f226d3ae640 1 --2- 172.21.15.123:0/2525703426 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f22680cf5a0 0x7f226806eee0 unknown :-1 s=CLOSED pgs=473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:25.712 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:25.703+0000 7f226d3ae640 1 -- 172.21.15.123:0/2525703426 >> 172.21.15.123:0/2525703426 conn(0x7f22680cfb10 msgr2=0x7f22680cff20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:44:25.712 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:25.703+0000 7f226d3ae640 1 -- 172.21.15.123:0/2525703426 shutdown_connections 2024-09-16T15:44:25.712 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:25.703+0000 7f226d3ae640 1 -- 172.21.15.123:0/2525703426 wait complete. 2024-09-16T15:44:25.714 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:25.707+0000 7f226d3ae640 1 Processor -- start 2024-09-16T15:44:25.714 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:25.707+0000 7f226d3ae640 1 -- start start 2024-09-16T15:44:25.714 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:25.707+0000 7f226d3ae640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f22680cf5a0 0x7f22680b2ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:44:25.714 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:25.707+0000 7f226d3ae640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2268158120 0x7f2268159bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:44:25.714 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:25.707+0000 7f226d3ae640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f226815ec40 0x7f226815a130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:44:25.715 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:25.707+0000 7f226d3ae640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f226815b140 con 0x7f226815ec40 2024-09-16T15:44:25.715 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:25.707+0000 7f226d3ae640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f226815afc0 con 0x7f2268158120 2024-09-16T15:44:25.715 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:25.707+0000 7f226d3ae640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f226815b2c0 con 0x7f22680cf5a0 2024-09-16T15:44:25.715 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:25.707+0000 7f22677fe640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f226815ec40 0x7f226815a130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:44:25.715 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:25.707+0000 7f22677fe640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f226815ec40 0x7f226815a130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:38852/0 (socket says 172.21.15.123:38852) 2024-09-16T15:44:25.715 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:25.707+0000 7f22677fe640 1 -- 172.21.15.123:0/4090023046 learned_addr learned my addr 172.21.15.123:0/4090023046 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:44:25.715 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:25.707+0000 7f22667fc640 1 --2- 172.21.15.123:0/4090023046 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2268158120 0x7f2268159bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:44:25.715 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:25.707+0000 7f2266ffd640 1 --2- 172.21.15.123:0/4090023046 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f22680cf5a0 0x7f22680b2ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:44:25.715 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:25.707+0000 7f22677fe640 1 -- 172.21.15.123:0/4090023046 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f22680cf5a0 msgr2=0x7f22680b2ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:44:25.715 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:25.707+0000 7f22677fe640 1 --2- 172.21.15.123:0/4090023046 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f22680cf5a0 0x7f22680b2ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:25.715 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:25.707+0000 7f22677fe640 1 -- 172.21.15.123:0/4090023046 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2268158120 msgr2=0x7f2268159bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:44:25.715 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:25.707+0000 7f22677fe640 1 --2- 172.21.15.123:0/4090023046 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2268158120 0x7f2268159bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:25.716 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:25.707+0000 7f22677fe640 1 -- 172.21.15.123:0/4090023046 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f22681f5840 con 0x7f226815ec40 2024-09-16T15:44:25.716 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:25.707+0000 7f2266ffd640 1 --2- 172.21.15.123:0/4090023046 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f22680cf5a0 0x7f22680b2ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T15:44:25.716 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:25.707+0000 7f22677fe640 1 --2- 172.21.15.123:0/4090023046 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f226815ec40 0x7f226815a130 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f226000e970 tx=0x7f226000ee30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:44:25.716 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:25.707+0000 7f224f7fe640 1 -- 172.21.15.123:0/4090023046 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2260004d30 con 0x7f226815ec40 2024-09-16T15:44:25.716 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:25.707+0000 7f224f7fe640 1 -- 172.21.15.123:0/4090023046 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f226000cb20 con 0x7f226815ec40 2024-09-16T15:44:25.716 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:25.707+0000 7f226d3ae640 1 -- 172.21.15.123:0/4090023046 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f22680b0ce0 con 0x7f226815ec40 2024-09-16T15:44:25.716 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:25.707+0000 7f224f7fe640 1 -- 172.21.15.123:0/4090023046 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2260005020 con 0x7f226815ec40 2024-09-16T15:44:25.716 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:25.707+0000 7f226d3ae640 1 -- 172.21.15.123:0/4090023046 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f22681f5ae0 con 0x7f226815ec40 2024-09-16T15:44:25.719 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:25.711+0000 7f224f7fe640 1 -- 172.21.15.123:0/4090023046 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 7) ==== 97233+0+0 (secure 0 0 0) 0x7f22600051c0 con 0x7f226815ec40 2024-09-16T15:44:25.719 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:25.711+0000 7f224f7fe640 1 --2- 172.21.15.123:0/4090023046 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f2254073070 0x7f2254075530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:44:25.720 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:25.711+0000 7f224f7fe640 1 -- 172.21.15.123:0/4090023046 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(44..44 src has 1..44) ==== 5709+0+0 (secure 0 0 0) 0x7f2260094bc0 con 0x7f226815ec40 2024-09-16T15:44:25.720 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:25.711+0000 7f2266ffd640 1 --2- 172.21.15.123:0/4090023046 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f2254073070 0x7f2254075530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:44:25.720 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:25.711+0000 7f2270498000 1 Processor -- start 2024-09-16T15:44:25.720 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:25.711+0000 7f2270498000 1 -- start start 2024-09-16T15:44:25.720 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:25.711+0000 7f2266ffd640 1 --2- 172.21.15.123:0/4090023046 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f2254073070 0x7f2254075530 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f2258000f70 tx=0x7f2258009040 comp rx=0 tx=0).ready entity=mgr.4135 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:44:25.720 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:25.711+0000 7f2270498000 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55f9756dc9d0 0x55f9756dcdb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:44:25.720 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:25.711+0000 7f2270498000 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x55f9756dd2f0 0x55f9756908d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:44:25.720 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:25.711+0000 7f2270498000 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55f975690e10 0x55f9756911f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:44:25.720 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:25.711+0000 7f2270498000 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x55f9755b13f0 con 0x55f975690e10 2024-09-16T15:44:25.720 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:25.711+0000 7f2270498000 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x55f9754eeb20 con 0x55f9756dc9d0 2024-09-16T15:44:25.720 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:25.711+0000 7f2270498000 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x55f9754ddf50 con 0x55f9756dd2f0 2024-09-16T15:44:25.720 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:25.711+0000 7f22677fe640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55f975690e10 0x55f9756911f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:44:25.721 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:25.711+0000 7f22677fe640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55f975690e10 0x55f9756911f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:38862/0 (socket says 172.21.15.123:38862) 2024-09-16T15:44:25.721 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:25.711+0000 7f22677fe640 1 -- 172.21.15.123:0/3849790718 learned_addr learned my addr 172.21.15.123:0/3849790718 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:44:25.721 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:25.711+0000 7f2266ffd640 1 --2- 172.21.15.123:0/3849790718 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55f9756dc9d0 0x55f9756dcdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:44:25.721 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:25.711+0000 7f22667fc640 1 --2- 172.21.15.123:0/3849790718 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x55f9756dd2f0 0x55f9756908d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:44:25.721 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:25.711+0000 7f22677fe640 1 -- 172.21.15.123:0/3849790718 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x55f9756dd2f0 msgr2=0x55f9756908d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T15:44:25.721 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:25.711+0000 7f22677fe640 1 --2- 172.21.15.123:0/3849790718 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x55f9756dd2f0 0x55f9756908d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:25.721 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:25.711+0000 7f22677fe640 1 -- 172.21.15.123:0/3849790718 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55f9756dc9d0 msgr2=0x55f9756dcdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T15:44:25.721 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:25.711+0000 7f22677fe640 1 --2- 172.21.15.123:0/3849790718 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55f9756dc9d0 0x55f9756dcdb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:25.721 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:25.711+0000 7f22677fe640 1 -- 172.21.15.123:0/3849790718 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f9754a5590 con 0x55f975690e10 2024-09-16T15:44:25.721 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:25.711+0000 7f22677fe640 1 --2- 172.21.15.123:0/3849790718 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55f975690e10 0x55f9756911f0 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f226000e940 tx=0x7f226005f280 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9ba28c87aae8a280 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:44:25.721 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:25.711+0000 7f224cff9640 1 -- 172.21.15.123:0/3849790718 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f226001f030 con 0x55f975690e10 2024-09-16T15:44:25.721 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:25.711+0000 7f224cff9640 1 -- 172.21.15.123:0/3849790718 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f2260093250 con 0x55f975690e10 2024-09-16T15:44:25.722 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:25.711+0000 7f224cff9640 1 -- 172.21.15.123:0/3849790718 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f226005e7e0 con 0x55f975690e10 2024-09-16T15:44:25.722 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:25.711+0000 7f2270498000 1 -- 172.21.15.123:0/3849790718 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55f975690e10 msgr2=0x55f9756911f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:44:25.722 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:25.711+0000 7f2270498000 1 --2- 172.21.15.123:0/3849790718 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55f975690e10 0x55f9756911f0 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f226000e940 tx=0x7f226005f280 comp rx=0 tx=0).stop 2024-09-16T15:44:25.722 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:25.711+0000 7f2270498000 1 -- 172.21.15.123:0/3849790718 shutdown_connections 2024-09-16T15:44:25.722 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:25.711+0000 7f2270498000 1 --2- 172.21.15.123:0/3849790718 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55f975690e10 0x55f9756911f0 unknown :-1 s=CLOSED pgs=475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:25.722 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:25.711+0000 7f2270498000 1 --2- 172.21.15.123:0/3849790718 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x55f9756dd2f0 0x55f9756908d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:25.722 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:25.711+0000 7f2270498000 1 --2- 172.21.15.123:0/3849790718 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55f9756dc9d0 0x55f9756dcdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:25.722 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:25.711+0000 7f2270498000 1 -- 172.21.15.123:0/3849790718 >> 172.21.15.123:0/3849790718 conn(0x55f97557a130 msgr2=0x55f9756bfe80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:44:25.722 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:25.715+0000 7f2270498000 1 -- 172.21.15.123:0/3849790718 shutdown_connections 2024-09-16T15:44:25.722 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:25.715+0000 7f2270498000 1 -- 172.21.15.123:0/3849790718 wait complete. 2024-09-16T15:44:25.722 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:25.715+0000 7f2270498000 1 -- start start 2024-09-16T15:44:25.722 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:25.715+0000 7f2270498000 1 Processor -- start 2024-09-16T15:44:25.726 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:25.715+0000 7f2270498000 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x55f975690e10 0x55f9756bd6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:44:25.726 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:25.715+0000 7f2270498000 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55f9756dc9d0 0x55f9756bde70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:44:25.726 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:25.715+0000 7f2270498000 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55f9756dd2f0 0x55f9756be3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:44:25.726 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:25.715+0000 7f2270498000 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x55f9754785d0 con 0x55f9756dc9d0 2024-09-16T15:44:25.726 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:25.715+0000 7f2270498000 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x55f9754e1d60 con 0x55f9756dd2f0 2024-09-16T15:44:25.726 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:25.715+0000 7f2270498000 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x55f975484500 con 0x55f975690e10 2024-09-16T15:44:25.727 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:25.715+0000 7f2270498000 10 client.0 _ll_register_callbacks cb 0x55f975488810 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T15:44:25.727 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:25.715+0000 7f22667fc640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55f9756dc9d0 0x55f9756bde70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:44:25.727 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:25.715+0000 7f22667fc640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55f9756dc9d0 0x55f9756bde70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:38874/0 (socket says 172.21.15.123:38874) 2024-09-16T15:44:25.727 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:25.715+0000 7f22667fc640 1 -- 172.21.15.123:0/3320395765 learned_addr learned my addr 172.21.15.123:0/3320395765 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:44:25.727 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:25.715+0000 7f2266ffd640 1 --2- 172.21.15.123:0/3320395765 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x55f975690e10 0x55f9756bd6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:44:25.727 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:25.715+0000 7f22677fe640 1 --2- 172.21.15.123:0/3320395765 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55f9756dd2f0 0x55f9756be3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:44:25.727 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:25.715+0000 7f22667fc640 1 -- 172.21.15.123:0/3320395765 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x55f975690e10 msgr2=0x55f9756bd6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T15:44:25.727 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:25.715+0000 7f22667fc640 1 --2- 172.21.15.123:0/3320395765 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x55f975690e10 0x55f9756bd6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:25.727 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:25.715+0000 7f22667fc640 1 -- 172.21.15.123:0/3320395765 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55f9756dd2f0 msgr2=0x55f9756be3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T15:44:25.727 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:25.715+0000 7f22667fc640 1 --2- 172.21.15.123:0/3320395765 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55f9756dd2f0 0x55f9756be3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:25.727 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:25.715+0000 7f22667fc640 1 -- 172.21.15.123:0/3320395765 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f97547c170 con 0x55f9756dc9d0 2024-09-16T15:44:25.727 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:25.715+0000 7f22677fe640 1 --2- 172.21.15.123:0/3320395765 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55f9756dd2f0 0x55f9756be3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T15:44:25.728 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:25.715+0000 7f22667fc640 1 --2- 172.21.15.123:0/3320395765 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55f9756dc9d0 0x55f9756bde70 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f225c01c100 tx=0x7f225c0045e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=515f2e8eba57bac1 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:44:25.728 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:25.715+0000 7f2245ffb640 10 client.0 ms_handle_connect on v2:172.21.15.123:3300/0 2024-09-16T15:44:25.728 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:25.715+0000 7f2245ffb640 1 -- 172.21.15.123:0/3320395765 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f225c0bb020 con 0x55f9756dc9d0 2024-09-16T15:44:25.728 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:25.715+0000 7f2270498000 1 -- 172.21.15.123:0/3320395765 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55f9754760f0 con 0x55f9756dc9d0 2024-09-16T15:44:25.728 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:25.715+0000 7f2245ffb640 1 -- 172.21.15.123:0/3320395765 <== mon.0 v2:172.21.15.123:3300/0 2 ==== osd_map(44..44 src has 1..44) ==== 5709+0+0 (secure 0 0 0) 0x7f225c005240 con 0x55f9756dc9d0 2024-09-16T15:44:25.728 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:25.715+0000 7f2245ffb640 1 -- 172.21.15.123:0/3320395765 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f225c0c2050 con 0x55f9756dc9d0 2024-09-16T15:44:25.728 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:25.715+0000 7f2270498000 10 client.5436 fetch_fsmap learned FSMap version 27 2024-09-16T15:44:25.728 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:25.715+0000 7f2270498000 1 -- 172.21.15.123:0/3320395765 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=2+}) -- 0x55f975562ef0 con 0x55f9756dc9d0 2024-09-16T15:44:25.728 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:25.715+0000 7f2245ffb640 1 -- 172.21.15.123:0/3320395765 <== mon.0 v2:172.21.15.123:3300/0 4 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x55f9754760f0 con 0x55f9756dc9d0 2024-09-16T15:44:25.728 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:25.715+0000 7f2245ffb640 1 -- 172.21.15.123:0/3320395765 <== mon.0 v2:172.21.15.123:3300/0 5 ==== fsmap(e 27) ==== 3502+0+0 (secure 0 0 0) 0x7f225c0c5020 con 0x55f9756dc9d0 2024-09-16T15:44:25.728 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:25.715+0000 7f2270498000 10 client.5436 fetch_fsmap finished waiting for FSMap version 27 2024-09-16T15:44:25.728 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:25.715+0000 7f2270498000 10 client.5436 resolve_mds: resolved name 'd' to daemon mds.d 2024-09-16T15:44:25.728 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:25.715+0000 7f2270498000 20 client.5436 populate_metadata read hostname 'smithi123' 2024-09-16T15:44:25.729 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:25.715+0000 7f2270498000 1 --2- 172.21.15.123:0/3320395765 >> [v2:172.21.15.184:6834/374389412,v1:172.21.15.184:6835/374389412] conn(0x55f9756bf0f0 0x55f9756eec90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:44:25.729 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:25.715+0000 7f2270498000 4 client.5436 mds_command: new command op to 5280 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-16T15:44:25.729 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:25.715+0000 7f2270498000 1 -- 172.21.15.123:0/3320395765 --> [v2:172.21.15.184:6834/374389412,v1:172.21.15.184:6835/374389412] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f9754f0a70 con 0x55f9756bf0f0 2024-09-16T15:44:25.729 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:25.715+0000 7f2266ffd640 1 --2- 172.21.15.123:0/3320395765 >> [v2:172.21.15.184:6834/374389412,v1:172.21.15.184:6835/374389412] conn(0x55f9756bf0f0 0x55f9756eec90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:44:25.729 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:25.719+0000 7f2245ffb640 10 client.5436 ms_handle_connect on v2:172.21.15.184:6834/374389412 2024-09-16T15:44:25.729 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:25.719+0000 7f2266ffd640 1 --2- 172.21.15.123:0/3320395765 >> [v2:172.21.15.184:6834/374389412,v1:172.21.15.184:6835/374389412] conn(0x55f9756bf0f0 0x55f9756eec90 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e8579c237e3d0d39 server_cookie=50a2f57932b93949 in_seq=0 out_seq=0 2024-09-16T15:44:25.729 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:25.719+0000 7f2245ffb640 1 -- 172.21.15.123:0/3320395765 <== mds.0 v2:172.21.15.184:6834/374389412 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x55f9754f0a70 con 0x55f9756bf0f0 2024-09-16T15:44:25.729 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:25.719+0000 7f2245ffb640 10 client.5436 handle_command_reply: tid=0 2024-09-16T15:44:25.729 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:25.719+0000 7f2270498000 1 client.5436 shutdown 2024-09-16T15:44:25.729 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:25.719+0000 7f2270498000 1 -- 172.21.15.123:0/3320395765 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55f9756dc9d0 msgr2=0x55f9756bde70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:44:25.729 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:25.719+0000 7f2270498000 1 --2- 172.21.15.123:0/3320395765 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55f9756dc9d0 0x55f9756bde70 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f225c01c100 tx=0x7f225c0045e0 comp rx=0 tx=0).stop 2024-09-16T15:44:25.729 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:25.719+0000 7f2270498000 1 -- 172.21.15.123:0/3320395765 shutdown_connections 2024-09-16T15:44:25.729 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:25.719+0000 7f2270498000 1 --2- 172.21.15.123:0/3320395765 >> [v2:172.21.15.184:6834/374389412,v1:172.21.15.184:6835/374389412] conn(0x55f9756bf0f0 0x55f9756eec90 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:25.729 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:25.719+0000 7f2270498000 1 --2- 172.21.15.123:0/3320395765 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55f9756dd2f0 0x55f9756be3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:25.730 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:25.719+0000 7f2270498000 1 --2- 172.21.15.123:0/3320395765 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55f9756dc9d0 0x55f9756bde70 unknown :-1 s=CLOSED pgs=476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:25.730 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:25.719+0000 7f2245ffb640 0 client.5436 ms_handle_reset on v2:172.21.15.184:6834/374389412 2024-09-16T15:44:25.730 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:25.719+0000 7f2270498000 1 --2- 172.21.15.123:0/3320395765 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x55f975690e10 0x55f9756bd6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:25.730 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:25.719+0000 7f2270498000 1 -- 172.21.15.123:0/3320395765 >> 172.21.15.123:0/3320395765 conn(0x55f97557a130 msgr2=0x55f97568ffd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:44:25.730 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:25.719+0000 7f2270498000 1 -- 172.21.15.123:0/3320395765 shutdown_connections 2024-09-16T15:44:25.730 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:25.719+0000 7f2270498000 1 -- 172.21.15.123:0/3320395765 wait complete. 2024-09-16T15:44:25.730 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:25.719+0000 7f2270498000 20 client.5436 trim_cache size 0 max 16384 2024-09-16T15:44:25.743 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:25.739+0000 7f2270498000 1 Processor -- start 2024-09-16T15:44:25.743 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:25.739+0000 7f2270498000 1 -- start start 2024-09-16T15:44:25.745 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:25.739+0000 7f2270498000 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55f975690e10 0x55f97574ebc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:44:25.745 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:25.739+0000 7f2270498000 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x55f9756dc9d0 0x55f97574f100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:44:25.745 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:25.739+0000 7f2270498000 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55f9756dd2f0 0x55f97574f640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:44:25.745 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:25.739+0000 7f2270498000 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f2268201af0 con 0x55f9756dd2f0 2024-09-16T15:44:25.745 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:25.739+0000 7f2270498000 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f2268152cd0 con 0x55f975690e10 2024-09-16T15:44:25.745 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:25.739+0000 7f2270498000 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f2268152030 con 0x55f9756dc9d0 2024-09-16T15:44:25.745 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:25.739+0000 7f22677fe640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55f9756dd2f0 0x55f97574f640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:44:25.745 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:25.739+0000 7f22677fe640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55f9756dd2f0 0x55f97574f640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:38876/0 (socket says 172.21.15.123:38876) 2024-09-16T15:44:25.745 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:25.739+0000 7f22677fe640 1 -- 172.21.15.123:0/3732678087 learned_addr learned my addr 172.21.15.123:0/3732678087 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:44:25.745 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:25.739+0000 7f22667fc640 1 --2- 172.21.15.123:0/3732678087 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x55f9756dc9d0 0x55f97574f100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:44:25.745 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:25.739+0000 7f2266ffd640 1 --2- 172.21.15.123:0/3732678087 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55f975690e10 0x55f97574ebc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:44:25.746 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:25.739+0000 7f22677fe640 1 -- 172.21.15.123:0/3732678087 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x55f9756dc9d0 msgr2=0x55f97574f100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T15:44:25.746 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:25.739+0000 7f22677fe640 1 --2- 172.21.15.123:0/3732678087 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x55f9756dc9d0 0x55f97574f100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:25.746 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:25.739+0000 7f22677fe640 1 -- 172.21.15.123:0/3732678087 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55f975690e10 msgr2=0x55f97574ebc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T15:44:25.746 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:25.739+0000 7f22677fe640 1 --2- 172.21.15.123:0/3732678087 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55f975690e10 0x55f97574ebc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:25.746 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:25.739+0000 7f22677fe640 1 -- 172.21.15.123:0/3732678087 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f975539d90 con 0x55f9756dd2f0 2024-09-16T15:44:25.746 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:25.739+0000 7f22667fc640 1 --2- 172.21.15.123:0/3732678087 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x55f9756dc9d0 0x55f97574f100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T15:44:25.746 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:25.739+0000 7f22677fe640 1 --2- 172.21.15.123:0/3732678087 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55f9756dd2f0 0x55f97574f640 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f226005f030 tx=0x7f226001c040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=32057c5bc439bd34 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:44:25.746 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:25.739+0000 7f2247fff640 1 -- 172.21.15.123:0/3732678087 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f226001f040 con 0x55f9756dd2f0 2024-09-16T15:44:25.746 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:25.739+0000 7f2247fff640 1 -- 172.21.15.123:0/3732678087 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f2260068d10 con 0x55f9756dd2f0 2024-09-16T15:44:25.746 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:25.739+0000 7f2247fff640 1 -- 172.21.15.123:0/3732678087 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f226005e840 con 0x55f9756dd2f0 2024-09-16T15:44:25.751 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:25.739+0000 7f2270498000 1 -- 172.21.15.123:0/3732678087 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55f9756dd2f0 msgr2=0x55f97574f640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:44:25.751 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:25.739+0000 7f2270498000 1 --2- 172.21.15.123:0/3732678087 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55f9756dd2f0 0x55f97574f640 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f226005f030 tx=0x7f226001c040 comp rx=0 tx=0).stop 2024-09-16T15:44:25.751 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:25.739+0000 7f2270498000 1 -- 172.21.15.123:0/3732678087 shutdown_connections 2024-09-16T15:44:25.751 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:25.739+0000 7f2270498000 1 --2- 172.21.15.123:0/3732678087 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55f9756dd2f0 0x55f97574f640 unknown :-1 s=CLOSED pgs=477 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:25.751 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:25.739+0000 7f2270498000 1 --2- 172.21.15.123:0/3732678087 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x55f9756dc9d0 0x55f97574f100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:25.751 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:25.739+0000 7f2270498000 1 --2- 172.21.15.123:0/3732678087 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55f975690e10 0x55f97574ebc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:25.752 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:25.739+0000 7f2270498000 1 -- 172.21.15.123:0/3732678087 >> 172.21.15.123:0/3732678087 conn(0x55f97557a130 msgr2=0x55f9756e9fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:44:25.752 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:25.739+0000 7f2270498000 1 -- 172.21.15.123:0/3732678087 shutdown_connections 2024-09-16T15:44:25.752 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:25.739+0000 7f2270498000 1 -- 172.21.15.123:0/3732678087 wait complete. 2024-09-16T15:44:25.752 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:25.739+0000 7f2270498000 1 -- start start 2024-09-16T15:44:25.752 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:25.739+0000 7f2270498000 1 Processor -- start 2024-09-16T15:44:25.752 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:25.739+0000 7f2270498000 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x55f975690e10 0x55f9757529f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:44:25.752 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:25.739+0000 7f2270498000 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55f9756dc9d0 0x55f975752f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:44:25.752 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:25.739+0000 7f2270498000 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55f9756dd2f0 0x55f975753470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:44:25.752 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:25.739+0000 7f2270498000 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x55f9756f3ce0 con 0x55f9756dd2f0 2024-09-16T15:44:25.752 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:25.739+0000 7f2270498000 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x55f9754a9ec0 con 0x55f9756dc9d0 2024-09-16T15:44:25.752 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:25.739+0000 7f2270498000 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x55f9754e1070 con 0x55f975690e10 2024-09-16T15:44:25.752 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:25.739+0000 7f2270498000 10 client.0 _ll_register_callbacks cb 0x7f225c0b1070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T15:44:25.752 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:25.739+0000 7f22677fe640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55f9756dd2f0 0x55f975753470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:44:25.753 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:25.739+0000 7f22677fe640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55f9756dd2f0 0x55f975753470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:38880/0 (socket says 172.21.15.123:38880) 2024-09-16T15:44:25.753 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:25.739+0000 7f22677fe640 1 -- 172.21.15.123:0/3196078693 learned_addr learned my addr 172.21.15.123:0/3196078693 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:44:25.753 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:25.739+0000 7f2266ffd640 1 --2- 172.21.15.123:0/3196078693 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x55f975690e10 0x55f9757529f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:44:25.753 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:25.743+0000 7f22667fc640 1 --2- 172.21.15.123:0/3196078693 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55f9756dc9d0 0x55f975752f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:44:25.753 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:25.743+0000 7f22677fe640 1 -- 172.21.15.123:0/3196078693 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x55f975690e10 msgr2=0x55f9757529f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T15:44:25.753 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:25.743+0000 7f22677fe640 1 --2- 172.21.15.123:0/3196078693 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x55f975690e10 0x55f9757529f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:25.753 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:25.743+0000 7f22677fe640 1 -- 172.21.15.123:0/3196078693 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55f9756dc9d0 msgr2=0x55f975752f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T15:44:25.753 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:25.743+0000 7f22677fe640 1 --2- 172.21.15.123:0/3196078693 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55f9756dc9d0 0x55f975752f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:25.753 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:25.743+0000 7f22677fe640 1 -- 172.21.15.123:0/3196078693 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f9754a4a90 con 0x55f9756dd2f0 2024-09-16T15:44:25.753 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:25.743+0000 7f22667fc640 1 --2- 172.21.15.123:0/3196078693 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55f9756dc9d0 0x55f975752f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T15:44:25.753 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:25.743+0000 7f22677fe640 1 --2- 172.21.15.123:0/3196078693 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55f9756dd2f0 0x55f975753470 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f226005f310 tx=0x7f226005af00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e1523b96b646f643 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:44:25.753 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:25.743+0000 7f2245ffb640 10 client.0 ms_handle_connect on v2:172.21.15.123:3300/0 2024-09-16T15:44:25.753 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:25.743+0000 7f2245ffb640 1 -- 172.21.15.123:0/3196078693 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f226006d060 con 0x55f9756dd2f0 2024-09-16T15:44:25.754 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:25.743+0000 7f2245ffb640 1 -- 172.21.15.123:0/3196078693 <== mon.0 v2:172.21.15.123:3300/0 2 ==== osd_map(44..44 src has 1..44) ==== 5709+0+0 (secure 0 0 0) 0x7f2260066070 con 0x55f9756dd2f0 2024-09-16T15:44:25.754 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:25.743+0000 7f2270498000 1 -- 172.21.15.123:0/3196078693 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2260068d10 con 0x55f9756dd2f0 2024-09-16T15:44:25.754 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:25.743+0000 7f2245ffb640 1 -- 172.21.15.123:0/3196078693 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f226001f070 con 0x55f9756dd2f0 2024-09-16T15:44:25.754 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:25.743+0000 7f2270498000 10 client.5442 fetch_fsmap learned FSMap version 27 2024-09-16T15:44:25.754 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:25.743+0000 7f2270498000 1 -- 172.21.15.123:0/3196078693 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=2+}) -- 0x55f975483050 con 0x55f9756dd2f0 2024-09-16T15:44:25.754 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:25.743+0000 7f2245ffb640 1 -- 172.21.15.123:0/3196078693 <== mon.0 v2:172.21.15.123:3300/0 4 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f2260068d10 con 0x55f9756dd2f0 2024-09-16T15:44:25.754 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:25.743+0000 7f2245ffb640 1 -- 172.21.15.123:0/3196078693 <== mon.0 v2:172.21.15.123:3300/0 5 ==== fsmap(e 27) ==== 3502+0+0 (secure 0 0 0) 0x7f226000f040 con 0x55f9756dd2f0 2024-09-16T15:44:25.754 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:25.743+0000 7f2270498000 10 client.5442 fetch_fsmap finished waiting for FSMap version 27 2024-09-16T15:44:25.754 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:25.743+0000 7f2270498000 10 client.5442 resolve_mds: resolved name 'd' to daemon mds.d 2024-09-16T15:44:25.754 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:25.743+0000 7f2270498000 20 client.5442 populate_metadata read hostname 'smithi123' 2024-09-16T15:44:25.754 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:25.743+0000 7f2270498000 1 --2- 172.21.15.123:0/3196078693 >> [v2:172.21.15.184:6834/374389412,v1:172.21.15.184:6835/374389412] conn(0x55f9756f0b80 0x55f9756f0f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:44:25.754 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:25.743+0000 7f2270498000 4 client.5442 mds_command: new command op to 5280 tid=0 multi_id=0 [{"prefix": "status"}] 2024-09-16T15:44:25.754 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:25.743+0000 7f2270498000 1 -- 172.21.15.123:0/3196078693 --> [v2:172.21.15.184:6834/374389412,v1:172.21.15.184:6835/374389412] -- command(tid 0: {"prefix": "status"}) -- 0x55f9754ec520 con 0x55f9756f0b80 2024-09-16T15:44:25.755 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:25.743+0000 7f2266ffd640 1 --2- 172.21.15.123:0/3196078693 >> [v2:172.21.15.184:6834/374389412,v1:172.21.15.184:6835/374389412] conn(0x55f9756f0b80 0x55f9756f0f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:44:25.755 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:25.743+0000 7f2245ffb640 10 client.5442 ms_handle_connect on v2:172.21.15.184:6834/374389412 2024-09-16T15:44:25.755 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:25.743+0000 7f2266ffd640 1 --2- 172.21.15.123:0/3196078693 >> [v2:172.21.15.184:6834/374389412,v1:172.21.15.184:6835/374389412] conn(0x55f9756f0b80 0x55f9756f0f60 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d06e1ae49a363 server_cookie=2eeba73d26b3420f in_seq=0 out_seq=0 2024-09-16T15:44:25.755 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:25.743+0000 7f2245ffb640 1 -- 172.21.15.123:0/3196078693 <== mds.0 v2:172.21.15.184:6834/374389412 1 ==== command_reply(tid 0: 0 ) ==== 8+0+319 (crc 0 0 0) 0x55f9754ec520 con 0x55f9756f0b80 2024-09-16T15:44:25.755 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:25.743+0000 7f2245ffb640 10 client.5442 handle_command_reply: tid=0 2024-09-16T15:44:25.755 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:25.743+0000 7f2270498000 1 client.5442 shutdown 2024-09-16T15:44:25.755 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:25.743+0000 7f2270498000 1 -- 172.21.15.123:0/3196078693 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55f9756dd2f0 msgr2=0x55f975753470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:44:25.755 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:25.743+0000 7f2270498000 1 --2- 172.21.15.123:0/3196078693 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55f9756dd2f0 0x55f975753470 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f226005f310 tx=0x7f226005af00 comp rx=0 tx=0).stop 2024-09-16T15:44:25.755 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:25.743+0000 7f2270498000 1 -- 172.21.15.123:0/3196078693 shutdown_connections 2024-09-16T15:44:25.755 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:25.743+0000 7f2270498000 1 --2- 172.21.15.123:0/3196078693 >> [v2:172.21.15.184:6834/374389412,v1:172.21.15.184:6835/374389412] conn(0x55f9756f0b80 0x55f9756f0f60 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:25.755 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:25.743+0000 7f2270498000 1 --2- 172.21.15.123:0/3196078693 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55f9756dd2f0 0x55f975753470 unknown :-1 s=CLOSED pgs=478 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:25.755 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:25.743+0000 7f2270498000 1 --2- 172.21.15.123:0/3196078693 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55f9756dc9d0 0x55f975752f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:25.755 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:25.743+0000 7f2245ffb640 0 client.5442 ms_handle_reset on v2:172.21.15.184:6834/374389412 2024-09-16T15:44:25.756 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:25.743+0000 7f2270498000 1 --2- 172.21.15.123:0/3196078693 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x55f975690e10 0x55f9757529f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:25.756 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:25.743+0000 7f2270498000 1 -- 172.21.15.123:0/3196078693 >> 172.21.15.123:0/3196078693 conn(0x55f97557a130 msgr2=0x55f975698ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:44:25.756 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:25.743+0000 7f2270498000 1 -- 172.21.15.123:0/3196078693 shutdown_connections 2024-09-16T15:44:25.756 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:25.743+0000 7f2270498000 1 -- 172.21.15.123:0/3196078693 wait complete. 2024-09-16T15:44:25.756 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:25.743+0000 7f2270498000 20 client.5442 trim_cache size 0 max 16384 2024-09-16T15:44:25.756 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2024-09-16T15:44:25.756 INFO:teuthology.orchestra.run.smithi123.stdout: "cluster_fsid": "56077f89-18c8-486f-ae7e-eecc0bb7822c", 2024-09-16T15:44:25.756 INFO:teuthology.orchestra.run.smithi123.stdout: "whoami": 0, 2024-09-16T15:44:25.756 INFO:teuthology.orchestra.run.smithi123.stdout: "id": 5280, 2024-09-16T15:44:25.757 INFO:teuthology.orchestra.run.smithi123.stdout: "want_state": "up:active", 2024-09-16T15:44:25.757 INFO:teuthology.orchestra.run.smithi123.stdout: "state": "up:active", 2024-09-16T15:44:25.757 INFO:teuthology.orchestra.run.smithi123.stdout: "fs_name": "cephfs", 2024-09-16T15:44:25.757 INFO:teuthology.orchestra.run.smithi123.stdout: "rank_uptime": 10.207987329, 2024-09-16T15:44:25.757 INFO:teuthology.orchestra.run.smithi123.stdout: "mdsmap_epoch": 27, 2024-09-16T15:44:25.757 INFO:teuthology.orchestra.run.smithi123.stdout: "osdmap_epoch": 44, 2024-09-16T15:44:25.757 INFO:teuthology.orchestra.run.smithi123.stdout: "osdmap_epoch_barrier": 44, 2024-09-16T15:44:25.757 INFO:teuthology.orchestra.run.smithi123.stdout: "uptime": 14.918359425 2024-09-16T15:44:25.757 INFO:teuthology.orchestra.run.smithi123.stdout:} 2024-09-16T15:44:25.758 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:25.743+0000 7f226d3ae640 1 -- 172.21.15.123:0/4090023046 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f2254073070 msgr2=0x7f2254075530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:44:25.758 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:25.743+0000 7f226d3ae640 1 --2- 172.21.15.123:0/4090023046 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f2254073070 0x7f2254075530 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f2258000f70 tx=0x7f2258009040 comp rx=0 tx=0).stop 2024-09-16T15:44:25.758 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:25.743+0000 7f226d3ae640 1 -- 172.21.15.123:0/4090023046 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f226815ec40 msgr2=0x7f226815a130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:44:25.758 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:25.743+0000 7f226d3ae640 1 --2- 172.21.15.123:0/4090023046 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f226815ec40 0x7f226815a130 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f226000e970 tx=0x7f226000ee30 comp rx=0 tx=0).stop 2024-09-16T15:44:25.758 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:25.743+0000 7f226d3ae640 1 -- 172.21.15.123:0/4090023046 shutdown_connections 2024-09-16T15:44:25.759 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:25.743+0000 7f226d3ae640 1 --2- 172.21.15.123:0/4090023046 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f2254073070 0x7f2254075530 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:25.759 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:25.743+0000 7f226d3ae640 1 --2- 172.21.15.123:0/4090023046 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f226815ec40 0x7f226815a130 unknown :-1 s=CLOSED pgs=474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:25.759 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:25.743+0000 7f226d3ae640 1 --2- 172.21.15.123:0/4090023046 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2268158120 0x7f2268159bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:25.759 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:25.743+0000 7f226d3ae640 1 --2- 172.21.15.123:0/4090023046 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f22680cf5a0 0x7f22680b2ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:25.759 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:25.743+0000 7f226d3ae640 1 -- 172.21.15.123:0/4090023046 >> 172.21.15.123:0/4090023046 conn(0x7f22680cfb10 msgr2=0x7f2268156d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:44:25.759 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:25.743+0000 7f226d3ae640 1 -- 172.21.15.123:0/4090023046 shutdown_connections 2024-09-16T15:44:25.759 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:25.743+0000 7f226d3ae640 1 -- 172.21.15.123:0/4090023046 wait complete. 2024-09-16T15:44:25.768 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-09-16T15:44:25.884 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:25.879+0000 7efee4409640 1 -- 172.21.15.123:0/1863581246 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7efedc0726c0 msgr2=0x7efedc073ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:44:25.884 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:25.879+0000 7efee4409640 1 --2- 172.21.15.123:0/1863581246 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7efedc0726c0 0x7efedc073ef0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7efecc009840 tx=0x7efecc01c760 comp rx=0 tx=0).stop 2024-09-16T15:44:25.885 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:25.879+0000 7efee4409640 1 -- 172.21.15.123:0/1863581246 shutdown_connections 2024-09-16T15:44:25.885 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:25.879+0000 7efee4409640 1 --2- 172.21.15.123:0/1863581246 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7efedc0726c0 0x7efedc073ef0 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:25.885 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:25.879+0000 7efee4409640 1 --2- 172.21.15.123:0/1863581246 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7efedc15bca0 0x7efedc0d0630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:25.885 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:25.879+0000 7efee4409640 1 -- 172.21.15.123:0/1863581246 >> 172.21.15.123:0/1863581246 conn(0x7efedc0cfb10 msgr2=0x7efedc0cff20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:44:25.885 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:25.879+0000 7efee4409640 1 -- 172.21.15.123:0/1863581246 shutdown_connections 2024-09-16T15:44:25.885 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:25.879+0000 7efee4409640 1 -- 172.21.15.123:0/1863581246 wait complete. 2024-09-16T15:44:25.885 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:25.879+0000 7efee4409640 1 Processor -- start 2024-09-16T15:44:25.886 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:25.879+0000 7efee4409640 1 -- start start 2024-09-16T15:44:25.888 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:25.879+0000 7efee4409640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7efedc0726c0 0x7efedc14fce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:44:25.888 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:25.879+0000 7efee4409640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7efedc074430 0x7efedc158880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:44:25.888 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:25.879+0000 7efee4409640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7efedc15bca0 0x7efedc158dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:44:25.888 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:25.879+0000 7efee4409640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7efedc168ef0 con 0x7efedc074430 2024-09-16T15:44:25.888 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:25.879+0000 7efee4409640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7efedc168d70 con 0x7efedc15bca0 2024-09-16T15:44:25.888 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:25.879+0000 7efee4409640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7efedc169070 con 0x7efedc0726c0 2024-09-16T15:44:25.888 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:25.879+0000 7efee19a4640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7efedc074430 0x7efedc158880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:44:25.888 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:25.879+0000 7efee19a4640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7efedc074430 0x7efedc158880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:38894/0 (socket says 172.21.15.123:38894) 2024-09-16T15:44:25.888 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:25.879+0000 7efee19a4640 1 -- 172.21.15.123:0/2238590355 learned_addr learned my addr 172.21.15.123:0/2238590355 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:44:25.888 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:25.879+0000 7efee21a5640 1 --2- 172.21.15.123:0/2238590355 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7efedc0726c0 0x7efedc14fce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:44:25.888 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:25.879+0000 7efee29a6640 1 --2- 172.21.15.123:0/2238590355 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7efedc15bca0 0x7efedc158dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:44:25.888 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:25.879+0000 7efee19a4640 1 -- 172.21.15.123:0/2238590355 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7efedc0726c0 msgr2=0x7efedc14fce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:44:25.889 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:25.879+0000 7efee19a4640 1 --2- 172.21.15.123:0/2238590355 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7efedc0726c0 0x7efedc14fce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:25.889 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:25.879+0000 7efee19a4640 1 -- 172.21.15.123:0/2238590355 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7efedc15bca0 msgr2=0x7efedc158dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:44:25.889 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:25.879+0000 7efee19a4640 1 --2- 172.21.15.123:0/2238590355 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7efedc15bca0 0x7efedc158dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:25.889 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:25.879+0000 7efee19a4640 1 -- 172.21.15.123:0/2238590355 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efedc150bc0 con 0x7efedc074430 2024-09-16T15:44:25.889 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:25.879+0000 7efee29a6640 1 --2- 172.21.15.123:0/2238590355 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7efedc15bca0 0x7efedc158dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T15:44:25.889 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:25.883+0000 7efee19a4640 1 --2- 172.21.15.123:0/2238590355 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7efedc074430 0x7efedc158880 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7efecc01a390 tx=0x7efecc005dd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:44:25.889 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:25.883+0000 7efed2ffd640 1 -- 172.21.15.123:0/2238590355 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efecc0b2040 con 0x7efedc074430 2024-09-16T15:44:25.889 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:25.883+0000 7efed2ffd640 1 -- 172.21.15.123:0/2238590355 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7efecc004320 con 0x7efedc074430 2024-09-16T15:44:25.889 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:25.883+0000 7efee4409640 1 -- 172.21.15.123:0/2238590355 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efedc1506e0 con 0x7efedc074430 2024-09-16T15:44:25.889 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:25.883+0000 7efed2ffd640 1 -- 172.21.15.123:0/2238590355 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efecc0044f0 con 0x7efedc074430 2024-09-16T15:44:25.889 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:25.883+0000 7efee4409640 1 -- 172.21.15.123:0/2238590355 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efedc150220 con 0x7efedc074430 2024-09-16T15:44:25.890 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:25.883+0000 7efee4409640 1 -- 172.21.15.123:0/2238590355 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efedc150920 con 0x7efedc074430 2024-09-16T15:44:25.890 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:25.883+0000 7efed2ffd640 1 -- 172.21.15.123:0/2238590355 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 7) ==== 97233+0+0 (secure 0 0 0) 0x7efecc005280 con 0x7efedc074430 2024-09-16T15:44:25.890 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:25.883+0000 7efed2ffd640 1 --2- 172.21.15.123:0/2238590355 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7efec0073070 0x7efec0075530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:44:25.890 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:25.883+0000 7efed2ffd640 1 -- 172.21.15.123:0/2238590355 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(44..44 src has 1..44) ==== 5709+0+0 (secure 0 0 0) 0x7efecc12d9d0 con 0x7efedc074430 2024-09-16T15:44:25.891 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:25.883+0000 7efee21a5640 1 --2- 172.21.15.123:0/2238590355 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7efec0073070 0x7efec0075530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:44:25.891 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:25.883+0000 7efee21a5640 1 --2- 172.21.15.123:0/2238590355 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7efec0073070 0x7efec0075530 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7efedc153c00 tx=0x7efed8009500 comp rx=0 tx=0).ready entity=mgr.4135 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:44:25.894 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:25.887+0000 7efed2ffd640 1 -- 172.21.15.123:0/2238590355 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7efecc0bd030 con 0x7efedc074430 2024-09-16T15:44:26.091 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:26.087+0000 7efee4409640 1 -- 172.21.15.123:0/2238590355 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7efedc073ef0 con 0x7efedc074430 2024-09-16T15:44:26.092 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:26.087+0000 7efed2ffd640 1 -- 172.21.15.123:0/2238590355 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v27) ==== 53+0+73 (secure 0 0 0) 0x7efecc016410 con 0x7efedc074430 2024-09-16T15:44:26.092 INFO:teuthology.orchestra.run.smithi123.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-09-16T15:44:26.094 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:26.087+0000 7efee4409640 1 -- 172.21.15.123:0/2238590355 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7efec0073070 msgr2=0x7efec0075530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:44:26.094 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:26.087+0000 7efee4409640 1 --2- 172.21.15.123:0/2238590355 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7efec0073070 0x7efec0075530 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7efedc153c00 tx=0x7efed8009500 comp rx=0 tx=0).stop 2024-09-16T15:44:26.094 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:26.087+0000 7efee4409640 1 -- 172.21.15.123:0/2238590355 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7efedc074430 msgr2=0x7efedc158880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:44:26.094 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:26.087+0000 7efee4409640 1 --2- 172.21.15.123:0/2238590355 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7efedc074430 0x7efedc158880 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7efecc01a390 tx=0x7efecc005dd0 comp rx=0 tx=0).stop 2024-09-16T15:44:26.094 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:26.087+0000 7efee4409640 1 -- 172.21.15.123:0/2238590355 shutdown_connections 2024-09-16T15:44:26.094 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:26.087+0000 7efee4409640 1 --2- 172.21.15.123:0/2238590355 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7efec0073070 0x7efec0075530 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:26.095 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:26.087+0000 7efee4409640 1 --2- 172.21.15.123:0/2238590355 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7efedc15bca0 0x7efedc158dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:26.095 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:26.087+0000 7efee4409640 1 --2- 172.21.15.123:0/2238590355 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7efedc074430 0x7efedc158880 unknown :-1 s=CLOSED pgs=479 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:26.095 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:26.087+0000 7efee4409640 1 --2- 172.21.15.123:0/2238590355 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7efedc0726c0 0x7efedc14fce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:26.095 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:26.087+0000 7efee4409640 1 -- 172.21.15.123:0/2238590355 >> 172.21.15.123:0/2238590355 conn(0x7efedc0cfb10 msgr2=0x7efedc06eee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:44:26.095 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:26.087+0000 7efee4409640 1 -- 172.21.15.123:0/2238590355 shutdown_connections 2024-09-16T15:44:26.095 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:26.087+0000 7efee4409640 1 -- 172.21.15.123:0/2238590355 wait complete. 2024-09-16T15:44:26.109 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-09-16T15:44:26.109 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-09-16T15:44:26.109 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi123.front.sepia.ceph.com 2024-09-16T15:44:26.109 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-09-16T15:44:26.110 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-09-16T15:44:26.110 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-09-16T15:44:26.110 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-09-16T15:44:26.110 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-09-16T15:44:26.110 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-09-16T15:44:26.110 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:44:26.110 DEBUG:teuthology.orchestra.run.smithi123:> ip addr 2024-09-16T15:44:26.115 INFO:teuthology.orchestra.run.smithi123.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-09-16T15:44:26.116 INFO:teuthology.orchestra.run.smithi123.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-09-16T15:44:26.116 INFO:teuthology.orchestra.run.smithi123.stdout: inet 127.0.0.1/8 scope host lo 2024-09-16T15:44:26.116 INFO:teuthology.orchestra.run.smithi123.stdout: valid_lft forever preferred_lft forever 2024-09-16T15:44:26.116 INFO:teuthology.orchestra.run.smithi123.stdout: inet6 ::1/128 scope host 2024-09-16T15:44:26.116 INFO:teuthology.orchestra.run.smithi123.stdout: valid_lft forever preferred_lft forever 2024-09-16T15:44:26.116 INFO:teuthology.orchestra.run.smithi123.stdout:2: eno1: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-09-16T15:44:26.116 INFO:teuthology.orchestra.run.smithi123.stdout: link/ether 0c:c4:7a:d8:da:6c brd ff:ff:ff:ff:ff:ff 2024-09-16T15:44:26.116 INFO:teuthology.orchestra.run.smithi123.stdout: altname enp7s0f0 2024-09-16T15:44:26.116 INFO:teuthology.orchestra.run.smithi123.stdout:3: eno2: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-09-16T15:44:26.116 INFO:teuthology.orchestra.run.smithi123.stdout: link/ether 0c:c4:7a:d8:da:6d brd ff:ff:ff:ff:ff:ff 2024-09-16T15:44:26.116 INFO:teuthology.orchestra.run.smithi123.stdout: altname enp7s0f1 2024-09-16T15:44:26.116 INFO:teuthology.orchestra.run.smithi123.stdout:4: enp3s0f0: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-09-16T15:44:26.117 INFO:teuthology.orchestra.run.smithi123.stdout: link/ether 0c:c4:7a:8f:d3:f0 brd ff:ff:ff:ff:ff:ff 2024-09-16T15:44:26.117 INFO:teuthology.orchestra.run.smithi123.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-09-16T15:44:26.117 INFO:teuthology.orchestra.run.smithi123.stdout: link/ether 0c:c4:7a:8f:d3:f1 brd ff:ff:ff:ff:ff:ff 2024-09-16T15:44:26.117 INFO:teuthology.orchestra.run.smithi123.stdout: inet 172.21.15.123/20 metric 100 brd 172.21.15.255 scope global dynamic enp3s0f1 2024-09-16T15:44:26.117 INFO:teuthology.orchestra.run.smithi123.stdout: valid_lft 42402sec preferred_lft 42402sec 2024-09-16T15:44:26.117 INFO:teuthology.orchestra.run.smithi123.stdout: inet6 fe80::ec4:7aff:fe8f:d3f1/64 scope link 2024-09-16T15:44:26.117 INFO:teuthology.orchestra.run.smithi123.stdout: valid_lft forever preferred_lft forever 2024-09-16T15:44:26.117 INFO:teuthology.orchestra.run.smithi123.stdout:6: docker0: mtu 1500 qdisc noqueue state DOWN group default 2024-09-16T15:44:26.117 INFO:teuthology.orchestra.run.smithi123.stdout: link/ether 02:42:dd:82:a2:6e brd ff:ff:ff:ff:ff:ff 2024-09-16T15:44:26.117 INFO:teuthology.orchestra.run.smithi123.stdout: inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0 2024-09-16T15:44:26.117 INFO:teuthology.orchestra.run.smithi123.stdout: valid_lft forever preferred_lft forever 2024-09-16T15:44:26.117 INFO:teuthology.orchestra.run.smithi123.stdout:7: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-09-16T15:44:26.117 INFO:teuthology.orchestra.run.smithi123.stdout: link/ether 5e:1f:84:6d:79:ae brd ff:ff:ff:ff:ff:ff 2024-09-16T15:44:26.118 INFO:teuthology.orchestra.run.smithi123.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-09-16T15:44:26.118 INFO:teuthology.orchestra.run.smithi123.stdout: valid_lft forever preferred_lft forever 2024-09-16T15:44:26.118 INFO:teuthology.orchestra.run.smithi123.stdout: inet6 fe80::5c1f:84ff:fe6d:79ae/64 scope link 2024-09-16T15:44:26.118 INFO:teuthology.orchestra.run.smithi123.stdout: valid_lft forever preferred_lft forever 2024-09-16T15:44:26.118 INFO:teuthology.orchestra.run.smithi123.stdout:8: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-09-16T15:44:26.118 INFO:teuthology.orchestra.run.smithi123.stdout: link/ether 0e:5c:f6:1f:82:2f brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-09-16T15:44:26.118 INFO:teuthology.orchestra.run.smithi123.stdout: inet6 fe80::c5c:f6ff:fe1f:822f/64 scope link 2024-09-16T15:44:26.118 INFO:teuthology.orchestra.run.smithi123.stdout: valid_lft forever preferred_lft forever 2024-09-16T15:44:26.118 INFO:teuthology.orchestra.run.smithi123.stdout:9: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-09-16T15:44:26.118 INFO:teuthology.orchestra.run.smithi123.stdout: link/ether c6:a6:3d:69:d0:db brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-09-16T15:44:26.118 INFO:teuthology.orchestra.run.smithi123.stdout: inet6 fe80::c4a6:3dff:fe69:d0db/64 scope link 2024-09-16T15:44:26.118 INFO:teuthology.orchestra.run.smithi123.stdout: valid_lft forever preferred_lft forever 2024-09-16T15:44:26.119 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:44:26.119 DEBUG:teuthology.orchestra.run.smithi123:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-09-16T15:44:26.168 INFO:teuthology.orchestra.run.smithi123.stdout:1 2024-09-16T15:44:26.169 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:44:26.169 DEBUG:teuthology.orchestra.run.smithi123:> route 2024-09-16T15:44:26.219 INFO:teuthology.orchestra.run.smithi123.stdout:Kernel IP routing table 2024-09-16T15:44:26.219 INFO:teuthology.orchestra.run.smithi123.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-09-16T15:44:26.220 INFO:teuthology.orchestra.run.smithi123.stdout:default _gateway 0.0.0.0 UG 0 0 0 enp3s0f1 2024-09-16T15:44:26.220 INFO:teuthology.orchestra.run.smithi123.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-09-16T15:44:26.220 INFO:teuthology.orchestra.run.smithi123.stdout:158.69.67.47 _gateway 255.255.255.255 UGH 100 0 0 enp3s0f1 2024-09-16T15:44:26.220 INFO:teuthology.orchestra.run.smithi123.stdout:172.17.0.0 0.0.0.0 255.255.0.0 U 0 0 0 docker0 2024-09-16T15:44:26.220 INFO:teuthology.orchestra.run.smithi123.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-09-16T15:44:26.220 INFO:teuthology.orchestra.run.smithi123.stdout:ns1.front.sepia 0.0.0.0 255.255.255.255 UH 100 0 0 enp3s0f1 2024-09-16T15:44:26.220 INFO:teuthology.orchestra.run.smithi123.stdout:172.21.0.2 0.0.0.0 255.255.255.255 UH 100 0 0 enp3s0f1 2024-09-16T15:44:26.220 INFO:teuthology.orchestra.run.smithi123.stdout:_gateway 0.0.0.0 255.255.255.255 UH 100 0 0 enp3s0f1 2024-09-16T15:44:26.220 INFO:teuthology.orchestra.run.smithi123.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-09-16T15:44:26.220 DEBUG:teuthology.orchestra.run.smithi123:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-09-16T15:44:26.220 DEBUG:teuthology.orchestra.run.smithi123:> set -e 2024-09-16T15:44:26.221 DEBUG:teuthology.orchestra.run.smithi123:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-09-16T15:44:26.221 DEBUG:teuthology.orchestra.run.smithi123:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-09-16T15:44:26.221 DEBUG:teuthology.orchestra.run.smithi123:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-09-16T15:44:26.221 DEBUG:teuthology.orchestra.run.smithi123:> ') 2024-09-16T15:44:26.306 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:26 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-09-16T15:44:26.334 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:26 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-16T15:44:26.340 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:44:26.341 DEBUG:teuthology.orchestra.run.smithi123:> ip netns list 2024-09-16T15:44:26.388 INFO:teuthology.orchestra.run.smithi123.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-09-16T15:44:26.388 INFO:teuthology.orchestra.run.smithi123.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-09-16T15:44:26.388 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-09-16T15:44:26.388 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:44:26.389 DEBUG:teuthology.orchestra.run.smithi123:> sudo ip link set brx.0 up 2024-09-16T15:44:26.443 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2024-09-16T15:44:26.444 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-09-16T15:44:26.444 DEBUG:teuthology.orchestra.run.smithi123:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-09-16T15:44:26.493 INFO:teuthology.orchestra.run.smithi123.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-09-16T15:44:26.493 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-09-16T15:44:26.493 DEBUG:teuthology.orchestra.run.smithi123:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-09-16T15:44:26.540 DEBUG:teuthology.orchestra.run.smithi123:> sudo modprobe fuse 2024-09-16T15:44:26.597 DEBUG:teuthology.orchestra.run.smithi123:> cat /proc/self/mounts | awk '{print $2}' 2024-09-16T15:44:26.645 INFO:teuthology.orchestra.run.smithi123.stdout:/sys 2024-09-16T15:44:26.645 INFO:teuthology.orchestra.run.smithi123.stdout:/proc 2024-09-16T15:44:26.645 INFO:teuthology.orchestra.run.smithi123.stdout:/dev 2024-09-16T15:44:26.645 INFO:teuthology.orchestra.run.smithi123.stdout:/dev/pts 2024-09-16T15:44:26.645 INFO:teuthology.orchestra.run.smithi123.stdout:/run 2024-09-16T15:44:26.645 INFO:teuthology.orchestra.run.smithi123.stdout:/ 2024-09-16T15:44:26.645 INFO:teuthology.orchestra.run.smithi123.stdout:/sys/kernel/security 2024-09-16T15:44:26.646 INFO:teuthology.orchestra.run.smithi123.stdout:/dev/shm 2024-09-16T15:44:26.646 INFO:teuthology.orchestra.run.smithi123.stdout:/run/lock 2024-09-16T15:44:26.646 INFO:teuthology.orchestra.run.smithi123.stdout:/sys/fs/cgroup 2024-09-16T15:44:26.646 INFO:teuthology.orchestra.run.smithi123.stdout:/sys/fs/pstore 2024-09-16T15:44:26.646 INFO:teuthology.orchestra.run.smithi123.stdout:/sys/fs/bpf 2024-09-16T15:44:26.646 INFO:teuthology.orchestra.run.smithi123.stdout:/proc/sys/fs/binfmt_misc 2024-09-16T15:44:26.646 INFO:teuthology.orchestra.run.smithi123.stdout:/dev/hugepages 2024-09-16T15:44:26.646 INFO:teuthology.orchestra.run.smithi123.stdout:/dev/mqueue 2024-09-16T15:44:26.646 INFO:teuthology.orchestra.run.smithi123.stdout:/sys/kernel/debug 2024-09-16T15:44:26.646 INFO:teuthology.orchestra.run.smithi123.stdout:/sys/kernel/tracing 2024-09-16T15:44:26.646 INFO:teuthology.orchestra.run.smithi123.stdout:/proc/fs/nfsd 2024-09-16T15:44:26.646 INFO:teuthology.orchestra.run.smithi123.stdout:/sys/fs/fuse/connections 2024-09-16T15:44:26.646 INFO:teuthology.orchestra.run.smithi123.stdout:/sys/kernel/config 2024-09-16T15:44:26.647 INFO:teuthology.orchestra.run.smithi123.stdout:/run/credentials/systemd-sysusers.service 2024-09-16T15:44:26.647 INFO:teuthology.orchestra.run.smithi123.stdout:/run/rpc_pipefs 2024-09-16T15:44:26.647 INFO:teuthology.orchestra.run.smithi123.stdout:/run/user/1000 2024-09-16T15:44:26.647 INFO:teuthology.orchestra.run.smithi123.stdout:/var/lib/ceph 2024-09-16T15:44:26.647 INFO:teuthology.orchestra.run.smithi123.stdout:/var/lib/ceph/osd/ceph-0 2024-09-16T15:44:26.647 INFO:teuthology.orchestra.run.smithi123.stdout:/var/lib/ceph/osd/ceph-1 2024-09-16T15:44:26.647 INFO:teuthology.orchestra.run.smithi123.stdout:/var/lib/ceph/osd/ceph-2 2024-09-16T15:44:26.647 INFO:teuthology.orchestra.run.smithi123.stdout:/var/lib/ceph/osd/ceph-3 2024-09-16T15:44:26.647 INFO:teuthology.orchestra.run.smithi123.stdout:/run/netns 2024-09-16T15:44:26.647 INFO:teuthology.orchestra.run.smithi123.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-09-16T15:44:26.647 INFO:teuthology.orchestra.run.smithi123.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-09-16T15:44:26.647 INFO:teuthology.orchestra.run.smithi123.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-09-16T15:44:26.647 INFO:teuthology.orchestra.run.smithi123.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-09-16T15:44:26.648 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:44:26.648 DEBUG:teuthology.orchestra.run.smithi123:> ls /sys/fs/fuse/connections 2024-09-16T15:44:26.696 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-09-16T15:44:26.696 DEBUG:teuthology.orchestra.run.smithi123:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2024-09-16T15:44:26.741 DEBUG:teuthology.orchestra.run.smithi123:> sudo modprobe fuse 2024-09-16T15:44:26.754 DEBUG:teuthology.orchestra.run.smithi123:> cat /proc/self/mounts | awk '{print $2}' 2024-09-16T15:44:26.804 INFO:teuthology.orchestra.run.smithi123.stdout:/sys 2024-09-16T15:44:26.804 INFO:teuthology.orchestra.run.smithi123.stdout:/proc 2024-09-16T15:44:26.804 INFO:teuthology.orchestra.run.smithi123.stdout:/dev 2024-09-16T15:44:26.804 INFO:teuthology.orchestra.run.smithi123.stdout:/dev/pts 2024-09-16T15:44:26.804 INFO:teuthology.orchestra.run.smithi123.stdout:/run 2024-09-16T15:44:26.804 INFO:teuthology.orchestra.run.smithi123.stdout:/ 2024-09-16T15:44:26.804 INFO:teuthology.orchestra.run.smithi123.stdout:/sys/kernel/security 2024-09-16T15:44:26.804 INFO:teuthology.orchestra.run.smithi123.stdout:/dev/shm 2024-09-16T15:44:26.805 INFO:teuthology.orchestra.run.smithi123.stdout:/run/lock 2024-09-16T15:44:26.805 INFO:teuthology.orchestra.run.smithi123.stdout:/sys/fs/cgroup 2024-09-16T15:44:26.805 INFO:teuthology.orchestra.run.smithi123.stdout:/sys/fs/pstore 2024-09-16T15:44:26.805 INFO:teuthology.orchestra.run.smithi123.stdout:/sys/fs/bpf 2024-09-16T15:44:26.805 INFO:teuthology.orchestra.run.smithi123.stdout:/proc/sys/fs/binfmt_misc 2024-09-16T15:44:26.805 INFO:teuthology.orchestra.run.smithi123.stdout:/dev/hugepages 2024-09-16T15:44:26.805 INFO:teuthology.orchestra.run.smithi123.stdout:/dev/mqueue 2024-09-16T15:44:26.805 INFO:teuthology.orchestra.run.smithi123.stdout:/sys/kernel/debug 2024-09-16T15:44:26.805 INFO:teuthology.orchestra.run.smithi123.stdout:/sys/kernel/tracing 2024-09-16T15:44:26.805 INFO:teuthology.orchestra.run.smithi123.stdout:/proc/fs/nfsd 2024-09-16T15:44:26.805 INFO:teuthology.orchestra.run.smithi123.stdout:/sys/fs/fuse/connections 2024-09-16T15:44:26.805 INFO:teuthology.orchestra.run.smithi123.stdout:/sys/kernel/config 2024-09-16T15:44:26.805 INFO:teuthology.orchestra.run.smithi123.stdout:/run/credentials/systemd-sysusers.service 2024-09-16T15:44:26.806 INFO:teuthology.orchestra.run.smithi123.stdout:/run/rpc_pipefs 2024-09-16T15:44:26.806 INFO:teuthology.orchestra.run.smithi123.stdout:/run/user/1000 2024-09-16T15:44:26.806 INFO:teuthology.orchestra.run.smithi123.stdout:/var/lib/ceph 2024-09-16T15:44:26.806 INFO:teuthology.orchestra.run.smithi123.stdout:/var/lib/ceph/osd/ceph-0 2024-09-16T15:44:26.806 INFO:teuthology.orchestra.run.smithi123.stdout:/var/lib/ceph/osd/ceph-1 2024-09-16T15:44:26.806 INFO:teuthology.orchestra.run.smithi123.stdout:/var/lib/ceph/osd/ceph-2 2024-09-16T15:44:26.806 INFO:teuthology.orchestra.run.smithi123.stdout:/var/lib/ceph/osd/ceph-3 2024-09-16T15:44:26.806 INFO:teuthology.orchestra.run.smithi123.stdout:/run/netns 2024-09-16T15:44:26.806 INFO:teuthology.orchestra.run.smithi123.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-09-16T15:44:26.806 INFO:teuthology.orchestra.run.smithi123.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-09-16T15:44:26.806 INFO:teuthology.orchestra.run.smithi123.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-09-16T15:44:26.806 INFO:teuthology.orchestra.run.smithi123.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-09-16T15:44:26.807 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:44:26.807 DEBUG:teuthology.orchestra.run.smithi123:> ls /sys/fs/fuse/connections 2024-09-16T15:44:26.821 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi123.stderr:2024-09-16T15:44:26.815+0000 7fe94ba73580 -1 init, newargv = 0x55b636a7f150 newargc=15 2024-09-16T15:44:26.821 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi123.stderr:ceph-fuse[39423]: starting ceph client 2024-09-16T15:44:26.828 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi123.stderr:ceph-fuse[39423]: starting fuse 2024-09-16T15:44:27.927 DEBUG:teuthology.orchestra.run.smithi123:> sudo modprobe fuse 2024-09-16T15:44:27.942 DEBUG:teuthology.orchestra.run.smithi123:> cat /proc/self/mounts | awk '{print $2}' 2024-09-16T15:44:27.990 INFO:teuthology.orchestra.run.smithi123.stdout:/sys 2024-09-16T15:44:27.990 INFO:teuthology.orchestra.run.smithi123.stdout:/proc 2024-09-16T15:44:27.990 INFO:teuthology.orchestra.run.smithi123.stdout:/dev 2024-09-16T15:44:27.990 INFO:teuthology.orchestra.run.smithi123.stdout:/dev/pts 2024-09-16T15:44:27.990 INFO:teuthology.orchestra.run.smithi123.stdout:/run 2024-09-16T15:44:27.990 INFO:teuthology.orchestra.run.smithi123.stdout:/ 2024-09-16T15:44:27.990 INFO:teuthology.orchestra.run.smithi123.stdout:/sys/kernel/security 2024-09-16T15:44:27.990 INFO:teuthology.orchestra.run.smithi123.stdout:/dev/shm 2024-09-16T15:44:27.990 INFO:teuthology.orchestra.run.smithi123.stdout:/run/lock 2024-09-16T15:44:27.990 INFO:teuthology.orchestra.run.smithi123.stdout:/sys/fs/cgroup 2024-09-16T15:44:27.990 INFO:teuthology.orchestra.run.smithi123.stdout:/sys/fs/pstore 2024-09-16T15:44:27.990 INFO:teuthology.orchestra.run.smithi123.stdout:/sys/fs/bpf 2024-09-16T15:44:27.991 INFO:teuthology.orchestra.run.smithi123.stdout:/proc/sys/fs/binfmt_misc 2024-09-16T15:44:27.991 INFO:teuthology.orchestra.run.smithi123.stdout:/dev/hugepages 2024-09-16T15:44:27.991 INFO:teuthology.orchestra.run.smithi123.stdout:/dev/mqueue 2024-09-16T15:44:27.991 INFO:teuthology.orchestra.run.smithi123.stdout:/sys/kernel/debug 2024-09-16T15:44:27.991 INFO:teuthology.orchestra.run.smithi123.stdout:/sys/kernel/tracing 2024-09-16T15:44:27.991 INFO:teuthology.orchestra.run.smithi123.stdout:/proc/fs/nfsd 2024-09-16T15:44:27.991 INFO:teuthology.orchestra.run.smithi123.stdout:/sys/fs/fuse/connections 2024-09-16T15:44:27.991 INFO:teuthology.orchestra.run.smithi123.stdout:/sys/kernel/config 2024-09-16T15:44:27.991 INFO:teuthology.orchestra.run.smithi123.stdout:/run/credentials/systemd-sysusers.service 2024-09-16T15:44:27.991 INFO:teuthology.orchestra.run.smithi123.stdout:/run/rpc_pipefs 2024-09-16T15:44:27.991 INFO:teuthology.orchestra.run.smithi123.stdout:/run/user/1000 2024-09-16T15:44:27.991 INFO:teuthology.orchestra.run.smithi123.stdout:/var/lib/ceph 2024-09-16T15:44:27.991 INFO:teuthology.orchestra.run.smithi123.stdout:/var/lib/ceph/osd/ceph-0 2024-09-16T15:44:27.991 INFO:teuthology.orchestra.run.smithi123.stdout:/var/lib/ceph/osd/ceph-1 2024-09-16T15:44:27.992 INFO:teuthology.orchestra.run.smithi123.stdout:/var/lib/ceph/osd/ceph-2 2024-09-16T15:44:27.992 INFO:teuthology.orchestra.run.smithi123.stdout:/var/lib/ceph/osd/ceph-3 2024-09-16T15:44:27.992 INFO:teuthology.orchestra.run.smithi123.stdout:/run/netns 2024-09-16T15:44:27.992 INFO:teuthology.orchestra.run.smithi123.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-09-16T15:44:27.992 INFO:teuthology.orchestra.run.smithi123.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-09-16T15:44:27.992 INFO:teuthology.orchestra.run.smithi123.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-09-16T15:44:27.992 INFO:teuthology.orchestra.run.smithi123.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-09-16T15:44:27.992 INFO:teuthology.orchestra.run.smithi123.stdout:/home/ubuntu/cephtest/mnt.0 2024-09-16T15:44:27.992 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:44:27.992 DEBUG:teuthology.orchestra.run.smithi123:> ls /sys/fs/fuse/connections 2024-09-16T15:44:28.036 INFO:teuthology.orchestra.run.smithi123.stdout:45 2024-09-16T15:44:28.036 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [45] 2024-09-16T15:44:28.036 DEBUG:teuthology.orchestra.run.smithi123:> sudo stdin-killer -- python3 -c ' 2024-09-16T15:44:28.036 DEBUG:teuthology.orchestra.run.smithi123:> import glob 2024-09-16T15:44:28.036 DEBUG:teuthology.orchestra.run.smithi123:> import re 2024-09-16T15:44:28.037 DEBUG:teuthology.orchestra.run.smithi123:> import os 2024-09-16T15:44:28.037 DEBUG:teuthology.orchestra.run.smithi123:> import subprocess 2024-09-16T15:44:28.037 DEBUG:teuthology.orchestra.run.smithi123:> 2024-09-16T15:44:28.037 DEBUG:teuthology.orchestra.run.smithi123:> def _find_admin_socket(client_name): 2024-09-16T15:44:28.037 DEBUG:teuthology.orchestra.run.smithi123:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2024-09-16T15:44:28.037 DEBUG:teuthology.orchestra.run.smithi123:> files = glob.glob(asok_path) 2024-09-16T15:44:28.037 DEBUG:teuthology.orchestra.run.smithi123:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-09-16T15:44:28.037 DEBUG:teuthology.orchestra.run.smithi123:> 2024-09-16T15:44:28.037 DEBUG:teuthology.orchestra.run.smithi123:> # Given a non-glob path, it better be there 2024-09-16T15:44:28.037 DEBUG:teuthology.orchestra.run.smithi123:> if "*" not in asok_path: 2024-09-16T15:44:28.037 DEBUG:teuthology.orchestra.run.smithi123:> assert(len(files) == 1) 2024-09-16T15:44:28.037 DEBUG:teuthology.orchestra.run.smithi123:> return files[0] 2024-09-16T15:44:28.037 DEBUG:teuthology.orchestra.run.smithi123:> 2024-09-16T15:44:28.037 DEBUG:teuthology.orchestra.run.smithi123:> for f in files: 2024-09-16T15:44:28.038 DEBUG:teuthology.orchestra.run.smithi123:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-09-16T15:44:28.038 DEBUG:teuthology.orchestra.run.smithi123:> if os.path.exists("/proc/{0}".format(pid)): 2024-09-16T15:44:28.038 DEBUG:teuthology.orchestra.run.smithi123:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-09-16T15:44:28.038 DEBUG:teuthology.orchestra.run.smithi123:> contents = proc_f.read() 2024-09-16T15:44:28.038 DEBUG:teuthology.orchestra.run.smithi123:> if mountpoint in contents: 2024-09-16T15:44:28.038 DEBUG:teuthology.orchestra.run.smithi123:> return f 2024-09-16T15:44:28.038 DEBUG:teuthology.orchestra.run.smithi123:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-09-16T15:44:28.038 DEBUG:teuthology.orchestra.run.smithi123:> 2024-09-16T15:44:28.038 DEBUG:teuthology.orchestra.run.smithi123:> print(_find_admin_socket("client.0")) 2024-09-16T15:44:28.038 DEBUG:teuthology.orchestra.run.smithi123:> ' 2024-09-16T15:44:28.161 INFO:teuthology.orchestra.run.smithi123.stdout:/var/run/ceph/ceph-client.0.39423.asok 2024-09-16T15:44:28.164 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:28 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-16T15:44:28.172 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.39423.asok 2024-09-16T15:44:28.172 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:44:28.172 DEBUG:teuthology.orchestra.run.smithi123:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.39423.asok status 2024-09-16T15:44:28.329 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2024-09-16T15:44:28.330 INFO:teuthology.orchestra.run.smithi123.stdout: "metadata": { 2024-09-16T15:44:28.330 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph_sha1": "33f95d2ae1bcbe7eb04404303f179eeeaa6982df", 2024-09-16T15:44:28.330 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph_version": "ceph version 19.3.0-4938-g33f95d2a (33f95d2ae1bcbe7eb04404303f179eeeaa6982df) squid (dev)", 2024-09-16T15:44:28.330 INFO:teuthology.orchestra.run.smithi123.stdout: "entity_id": "0", 2024-09-16T15:44:28.330 INFO:teuthology.orchestra.run.smithi123.stdout: "hostname": "smithi123", 2024-09-16T15:44:28.330 INFO:teuthology.orchestra.run.smithi123.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2024-09-16T15:44:28.330 INFO:teuthology.orchestra.run.smithi123.stdout: "pid": "39423", 2024-09-16T15:44:28.330 INFO:teuthology.orchestra.run.smithi123.stdout: "root": "/" 2024-09-16T15:44:28.330 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:44:28.330 INFO:teuthology.orchestra.run.smithi123.stdout: "dentry_count": 0, 2024-09-16T15:44:28.330 INFO:teuthology.orchestra.run.smithi123.stdout: "dentry_pinned_count": 0, 2024-09-16T15:44:28.330 INFO:teuthology.orchestra.run.smithi123.stdout: "id": 5451, 2024-09-16T15:44:28.331 INFO:teuthology.orchestra.run.smithi123.stdout: "inst": { 2024-09-16T15:44:28.331 INFO:teuthology.orchestra.run.smithi123.stdout: "name": { 2024-09-16T15:44:28.331 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "client", 2024-09-16T15:44:28.331 INFO:teuthology.orchestra.run.smithi123.stdout: "num": 5451 2024-09-16T15:44:28.331 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:44:28.331 INFO:teuthology.orchestra.run.smithi123.stdout: "addr": { 2024-09-16T15:44:28.331 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "v1", 2024-09-16T15:44:28.331 INFO:teuthology.orchestra.run.smithi123.stdout: "addr": "192.168.0.1:0", 2024-09-16T15:44:28.331 INFO:teuthology.orchestra.run.smithi123.stdout: "nonce": 3346624420 2024-09-16T15:44:28.331 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-16T15:44:28.331 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:44:28.331 INFO:teuthology.orchestra.run.smithi123.stdout: "addr": { 2024-09-16T15:44:28.332 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "v1", 2024-09-16T15:44:28.332 INFO:teuthology.orchestra.run.smithi123.stdout: "addr": "192.168.0.1:0", 2024-09-16T15:44:28.332 INFO:teuthology.orchestra.run.smithi123.stdout: "nonce": 3346624420 2024-09-16T15:44:28.332 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:44:28.332 INFO:teuthology.orchestra.run.smithi123.stdout: "inst_str": "client.5451 192.168.0.1:0/3346624420", 2024-09-16T15:44:28.332 INFO:teuthology.orchestra.run.smithi123.stdout: "addr_str": "192.168.0.1:0/3346624420", 2024-09-16T15:44:28.332 INFO:teuthology.orchestra.run.smithi123.stdout: "inode_count": 1, 2024-09-16T15:44:28.332 INFO:teuthology.orchestra.run.smithi123.stdout: "mds_epoch": 27, 2024-09-16T15:44:28.332 INFO:teuthology.orchestra.run.smithi123.stdout: "osd_epoch": 44, 2024-09-16T15:44:28.332 INFO:teuthology.orchestra.run.smithi123.stdout: "osd_epoch_barrier": 0, 2024-09-16T15:44:28.332 INFO:teuthology.orchestra.run.smithi123.stdout: "blocklisted": false, 2024-09-16T15:44:28.332 INFO:teuthology.orchestra.run.smithi123.stdout: "fs_name": "cephfs" 2024-09-16T15:44:28.332 INFO:teuthology.orchestra.run.smithi123.stdout:} 2024-09-16T15:44:28.344 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:44:28.345 DEBUG:teuthology.orchestra.run.smithi123:> stat --file-system '--printf=%T 2024-09-16T15:44:28.345 DEBUG:teuthology.orchestra.run.smithi123:> ' -- /home/ubuntu/cephtest/mnt.0 2024-09-16T15:44:28.351 INFO:teuthology.orchestra.run.smithi123.stdout:fuseblk 2024-09-16T15:44:28.351 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2024-09-16T15:44:28.351 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:44:28.351 DEBUG:teuthology.orchestra.run.smithi123:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-09-16T15:44:28.409 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-09-16T15:44:28.585 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:28.579+0000 7f8bc3116640 1 -- 172.21.15.123:0/249293467 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8ba4003a10 msgr2=0x7f8ba4003e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:44:28.585 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:28.579+0000 7f8bc3116640 1 --2- 172.21.15.123:0/249293467 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8ba4003a10 0x7f8ba4003e30 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f8bb0009b60 tx=0x7f8bb001d240 comp rx=0 tx=0).stop 2024-09-16T15:44:28.585 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:28.579+0000 7f8bc16b3640 1 -- 172.21.15.123:0/249293467 reap_dead start 2024-09-16T15:44:28.585 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:28.579+0000 7f8bc3116640 1 -- 172.21.15.123:0/249293467 shutdown_connections 2024-09-16T15:44:28.585 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:28.579+0000 7f8bc3116640 1 -- 172.21.15.123:0/249293467 >> 172.21.15.123:0/249293467 conn(0x7f8bbc0cfb10 msgr2=0x7f8bbc0cff20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:44:28.585 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:28.579+0000 7f8bc3116640 1 -- 172.21.15.123:0/249293467 shutdown_connections 2024-09-16T15:44:28.585 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:28.579+0000 7f8bc3116640 1 -- 172.21.15.123:0/249293467 wait complete. 2024-09-16T15:44:28.586 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:28.579+0000 7f8bc3116640 1 Processor -- start 2024-09-16T15:44:28.586 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:28.579+0000 7f8bc3116640 1 -- start start 2024-09-16T15:44:28.588 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:28.579+0000 7f8bc3116640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f8ba4003a10 0x7f8bbc1f72f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:44:28.588 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:28.579+0000 7f8bc3116640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8bbc162b40 0x7f8bbc0ce040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:44:28.588 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:28.579+0000 7f8bc3116640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f8bbc0ce580 0x7f8bbc163b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:44:28.588 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:28.579+0000 7f8bc3116640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f8bbc166a40 con 0x7f8bbc162b40 2024-09-16T15:44:28.588 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:28.579+0000 7f8bc3116640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f8bbc06f950 con 0x7f8ba4003a10 2024-09-16T15:44:28.588 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:28.579+0000 7f8bc3116640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f8bbc166bc0 con 0x7f8bbc0ce580 2024-09-16T15:44:28.588 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:28.579+0000 7f8bbbfff640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8bbc162b40 0x7f8bbc0ce040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:44:28.588 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:28.579+0000 7f8bbbfff640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8bbc162b40 0x7f8bbc0ce040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:38918/0 (socket says 172.21.15.123:38918) 2024-09-16T15:44:28.588 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:28.579+0000 7f8bbbfff640 1 -- 172.21.15.123:0/1450894235 learned_addr learned my addr 172.21.15.123:0/1450894235 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:44:28.588 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:28.579+0000 7f8bc0eb2640 1 --2- 172.21.15.123:0/1450894235 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f8ba4003a10 0x7f8bbc1f72f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:44:28.589 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:28.579+0000 7f8bc16b3640 1 --2- 172.21.15.123:0/1450894235 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f8bbc0ce580 0x7f8bbc163b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:44:28.589 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:28.579+0000 7f8bbbfff640 1 -- 172.21.15.123:0/1450894235 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f8bbc0ce580 msgr2=0x7f8bbc163b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:44:28.589 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:28.579+0000 7f8bbbfff640 1 --2- 172.21.15.123:0/1450894235 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f8bbc0ce580 0x7f8bbc163b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:28.589 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:28.579+0000 7f8bbbfff640 1 -- 172.21.15.123:0/1450894235 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f8ba4003a10 msgr2=0x7f8bbc1f72f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:44:28.589 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:28.579+0000 7f8bbbfff640 1 --2- 172.21.15.123:0/1450894235 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f8ba4003a10 0x7f8bbc1f72f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:28.589 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:28.583+0000 7f8bbbfff640 1 -- 172.21.15.123:0/1450894235 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8bbc0cea20 con 0x7f8bbc162b40 2024-09-16T15:44:28.589 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:28.583+0000 7f8bbbfff640 1 --2- 172.21.15.123:0/1450894235 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8bbc162b40 0x7f8bbc0ce040 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f8bac00d950 tx=0x7f8bac00de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:44:28.589 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:28.583+0000 7f8bb97fa640 1 -- 172.21.15.123:0/1450894235 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8bac00b830 con 0x7f8bbc162b40 2024-09-16T15:44:28.589 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:28.583+0000 7f8bb97fa640 1 -- 172.21.15.123:0/1450894235 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f8bac00bac0 con 0x7f8bbc162b40 2024-09-16T15:44:28.589 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:28.583+0000 7f8bc3116640 1 -- 172.21.15.123:0/1450894235 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8bbc0ced00 con 0x7f8bbc162b40 2024-09-16T15:44:28.589 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:28.583+0000 7f8bb97fa640 1 -- 172.21.15.123:0/1450894235 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8bac003600 con 0x7f8bbc162b40 2024-09-16T15:44:28.589 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:28.583+0000 7f8bc3116640 1 -- 172.21.15.123:0/1450894235 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8bbc0cf2c0 con 0x7f8bbc162b40 2024-09-16T15:44:28.590 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:28.583+0000 7f8bc3116640 1 -- 172.21.15.123:0/1450894235 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8bbc06e750 con 0x7f8bbc162b40 2024-09-16T15:44:28.590 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:28.583+0000 7f8bb97fa640 1 -- 172.21.15.123:0/1450894235 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 7) ==== 97233+0+0 (secure 0 0 0) 0x7f8bac0037a0 con 0x7f8bbc162b40 2024-09-16T15:44:28.590 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:28.583+0000 7f8bb97fa640 1 --2- 172.21.15.123:0/1450894235 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f8ba8073070 0x7f8ba8075530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:44:28.590 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:28.583+0000 7f8bb97fa640 1 -- 172.21.15.123:0/1450894235 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(44..44 src has 1..44) ==== 5709+0+0 (secure 0 0 0) 0x7f8bac095560 con 0x7f8bbc162b40 2024-09-16T15:44:28.591 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:28.583+0000 7f8bc0eb2640 1 --2- 172.21.15.123:0/1450894235 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f8ba8073070 0x7f8ba8075530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:44:28.591 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:28.587+0000 7f8bc0eb2640 1 --2- 172.21.15.123:0/1450894235 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f8ba8073070 0x7f8ba8075530 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f8bb00ae000 tx=0x7f8bb0019fe0 comp rx=0 tx=0).ready entity=mgr.4135 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:44:28.594 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:28.587+0000 7f8bb97fa640 1 -- 172.21.15.123:0/1450894235 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f8bac060390 con 0x7f8bbc162b40 2024-09-16T15:44:28.804 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:28.799+0000 7f8bc3116640 1 -- 172.21.15.123:0/1450894235 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f8bbc15f630 con 0x7f8bbc162b40 2024-09-16T15:44:28.805 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:28.799+0000 7f8bb97fa640 1 -- 172.21.15.123:0/1450894235 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v27) ==== 53+0+73 (secure 0 0 0) 0x7f8bac05fc90 con 0x7f8bbc162b40 2024-09-16T15:44:28.805 INFO:teuthology.orchestra.run.smithi123.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-09-16T15:44:28.807 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:28.799+0000 7f8bc3116640 1 -- 172.21.15.123:0/1450894235 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f8ba8073070 msgr2=0x7f8ba8075530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:44:28.807 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:28.799+0000 7f8bc3116640 1 --2- 172.21.15.123:0/1450894235 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f8ba8073070 0x7f8ba8075530 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f8bb00ae000 tx=0x7f8bb0019fe0 comp rx=0 tx=0).stop 2024-09-16T15:44:28.808 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:28.803+0000 7f8bc3116640 1 -- 172.21.15.123:0/1450894235 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8bbc162b40 msgr2=0x7f8bbc0ce040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:44:28.808 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:28.803+0000 7f8bc3116640 1 --2- 172.21.15.123:0/1450894235 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8bbc162b40 0x7f8bbc0ce040 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f8bac00d950 tx=0x7f8bac00de10 comp rx=0 tx=0).stop 2024-09-16T15:44:28.808 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:28.803+0000 7f8bc3116640 1 -- 172.21.15.123:0/1450894235 shutdown_connections 2024-09-16T15:44:28.808 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:28.803+0000 7f8bc3116640 1 --2- 172.21.15.123:0/1450894235 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f8ba8073070 0x7f8ba8075530 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:28.808 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:28.803+0000 7f8bc3116640 1 --2- 172.21.15.123:0/1450894235 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f8bbc0ce580 0x7f8bbc163b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:28.808 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:28.803+0000 7f8bc3116640 1 --2- 172.21.15.123:0/1450894235 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8bbc162b40 0x7f8bbc0ce040 unknown :-1 s=CLOSED pgs=485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:28.808 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:28.803+0000 7f8bc3116640 1 --2- 172.21.15.123:0/1450894235 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f8ba4003a10 0x7f8bbc1f72f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:28.808 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:28.803+0000 7f8bc3116640 1 -- 172.21.15.123:0/1450894235 >> 172.21.15.123:0/1450894235 conn(0x7f8bbc0cfb10 msgr2=0x7f8bbc0b4220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:44:28.808 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:28.803+0000 7f8bc3116640 1 -- 172.21.15.123:0/1450894235 shutdown_connections 2024-09-16T15:44:28.808 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:28.803+0000 7f8bc3116640 1 -- 172.21.15.123:0/1450894235 wait complete. 2024-09-16T15:44:28.824 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-09-16T15:44:28.824 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-09-16T15:44:28.824 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi184.front.sepia.ceph.com 2024-09-16T15:44:28.825 INFO:tasks.cephfs.mount:self.client.name = client.1 2024-09-16T15:44:28.825 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2024-09-16T15:44:28.825 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-09-16T15:44:28.825 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-09-16T15:44:28.825 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-09-16T15:44:28.825 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.1' netns for '/home/ubuntu/cephtest/mnt.1' 2024-09-16T15:44:28.825 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:44:28.825 DEBUG:teuthology.orchestra.run.smithi184:> ip addr 2024-09-16T15:44:28.832 INFO:teuthology.orchestra.run.smithi184.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-09-16T15:44:28.833 INFO:teuthology.orchestra.run.smithi184.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-09-16T15:44:28.833 INFO:teuthology.orchestra.run.smithi184.stdout: inet 127.0.0.1/8 scope host lo 2024-09-16T15:44:28.833 INFO:teuthology.orchestra.run.smithi184.stdout: valid_lft forever preferred_lft forever 2024-09-16T15:44:28.833 INFO:teuthology.orchestra.run.smithi184.stdout: inet6 ::1/128 scope host 2024-09-16T15:44:28.833 INFO:teuthology.orchestra.run.smithi184.stdout: valid_lft forever preferred_lft forever 2024-09-16T15:44:28.833 INFO:teuthology.orchestra.run.smithi184.stdout:2: eno1: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-09-16T15:44:28.833 INFO:teuthology.orchestra.run.smithi184.stdout: link/ether 0c:c4:7a:d9:37:16 brd ff:ff:ff:ff:ff:ff 2024-09-16T15:44:28.833 INFO:teuthology.orchestra.run.smithi184.stdout: altname enp7s0f0 2024-09-16T15:44:28.833 INFO:teuthology.orchestra.run.smithi184.stdout:3: eno2: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-09-16T15:44:28.833 INFO:teuthology.orchestra.run.smithi184.stdout: link/ether 0c:c4:7a:d9:37:17 brd ff:ff:ff:ff:ff:ff 2024-09-16T15:44:28.833 INFO:teuthology.orchestra.run.smithi184.stdout: altname enp7s0f1 2024-09-16T15:44:28.833 INFO:teuthology.orchestra.run.smithi184.stdout:4: enp3s0f0: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-09-16T15:44:28.833 INFO:teuthology.orchestra.run.smithi184.stdout: link/ether 0c:c4:7a:88:7b:8e brd ff:ff:ff:ff:ff:ff 2024-09-16T15:44:28.834 INFO:teuthology.orchestra.run.smithi184.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-09-16T15:44:28.834 INFO:teuthology.orchestra.run.smithi184.stdout: link/ether 0c:c4:7a:88:7b:8f brd ff:ff:ff:ff:ff:ff 2024-09-16T15:44:28.834 INFO:teuthology.orchestra.run.smithi184.stdout: inet 172.21.15.184/20 metric 100 brd 172.21.15.255 scope global dynamic enp3s0f1 2024-09-16T15:44:28.834 INFO:teuthology.orchestra.run.smithi184.stdout: valid_lft 42388sec preferred_lft 42388sec 2024-09-16T15:44:28.834 INFO:teuthology.orchestra.run.smithi184.stdout: inet6 fe80::ec4:7aff:fe88:7b8f/64 scope link 2024-09-16T15:44:28.834 INFO:teuthology.orchestra.run.smithi184.stdout: valid_lft forever preferred_lft forever 2024-09-16T15:44:28.834 INFO:teuthology.orchestra.run.smithi184.stdout:6: docker0: mtu 1500 qdisc noqueue state DOWN group default 2024-09-16T15:44:28.834 INFO:teuthology.orchestra.run.smithi184.stdout: link/ether 02:42:5d:0a:44:b9 brd ff:ff:ff:ff:ff:ff 2024-09-16T15:44:28.834 INFO:teuthology.orchestra.run.smithi184.stdout: inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0 2024-09-16T15:44:28.834 INFO:teuthology.orchestra.run.smithi184.stdout: valid_lft forever preferred_lft forever 2024-09-16T15:44:28.834 INFO:teuthology.orchestra.run.smithi184.stdout:7: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-09-16T15:44:28.834 INFO:teuthology.orchestra.run.smithi184.stdout: link/ether 5e:1f:84:6d:79:ae brd ff:ff:ff:ff:ff:ff 2024-09-16T15:44:28.834 INFO:teuthology.orchestra.run.smithi184.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-09-16T15:44:28.834 INFO:teuthology.orchestra.run.smithi184.stdout: valid_lft forever preferred_lft forever 2024-09-16T15:44:28.834 INFO:teuthology.orchestra.run.smithi184.stdout: inet6 fe80::5c1f:84ff:fe6d:79ae/64 scope link 2024-09-16T15:44:28.835 INFO:teuthology.orchestra.run.smithi184.stdout: valid_lft forever preferred_lft forever 2024-09-16T15:44:28.835 INFO:teuthology.orchestra.run.smithi184.stdout:8: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-09-16T15:44:28.835 INFO:teuthology.orchestra.run.smithi184.stdout: link/ether 0e:5c:f6:1f:82:2f brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.1 2024-09-16T15:44:28.835 INFO:teuthology.orchestra.run.smithi184.stdout: inet6 fe80::c5c:f6ff:fe1f:822f/64 scope link 2024-09-16T15:44:28.835 INFO:teuthology.orchestra.run.smithi184.stdout: valid_lft forever preferred_lft forever 2024-09-16T15:44:28.835 INFO:teuthology.orchestra.run.smithi184.stdout:9: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-09-16T15:44:28.835 INFO:teuthology.orchestra.run.smithi184.stdout: link/ether c6:a6:3d:69:d0:db brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.3 2024-09-16T15:44:28.835 INFO:teuthology.orchestra.run.smithi184.stdout: inet6 fe80::c4a6:3dff:fe69:d0db/64 scope link 2024-09-16T15:44:28.835 INFO:teuthology.orchestra.run.smithi184.stdout: valid_lft forever preferred_lft forever 2024-09-16T15:44:28.835 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:44:28.835 DEBUG:teuthology.orchestra.run.smithi184:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-09-16T15:44:28.891 INFO:teuthology.orchestra.run.smithi184.stdout:1 2024-09-16T15:44:28.893 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:44:28.893 DEBUG:teuthology.orchestra.run.smithi184:> route 2024-09-16T15:44:28.944 INFO:teuthology.orchestra.run.smithi184.stdout:Kernel IP routing table 2024-09-16T15:44:28.945 INFO:teuthology.orchestra.run.smithi184.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-09-16T15:44:28.945 INFO:teuthology.orchestra.run.smithi184.stdout:default _gateway 0.0.0.0 UG 0 0 0 enp3s0f1 2024-09-16T15:44:28.945 INFO:teuthology.orchestra.run.smithi184.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-09-16T15:44:28.945 INFO:teuthology.orchestra.run.smithi184.stdout:158.69.67.47 _gateway 255.255.255.255 UGH 100 0 0 enp3s0f1 2024-09-16T15:44:28.945 INFO:teuthology.orchestra.run.smithi184.stdout:172.17.0.0 0.0.0.0 255.255.0.0 U 0 0 0 docker0 2024-09-16T15:44:28.945 INFO:teuthology.orchestra.run.smithi184.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-09-16T15:44:28.945 INFO:teuthology.orchestra.run.smithi184.stdout:ns1.front.sepia 0.0.0.0 255.255.255.255 UH 100 0 0 enp3s0f1 2024-09-16T15:44:28.945 INFO:teuthology.orchestra.run.smithi184.stdout:172.21.0.2 0.0.0.0 255.255.255.255 UH 100 0 0 enp3s0f1 2024-09-16T15:44:28.945 INFO:teuthology.orchestra.run.smithi184.stdout:_gateway 0.0.0.0 255.255.255.255 UH 100 0 0 enp3s0f1 2024-09-16T15:44:28.945 INFO:teuthology.orchestra.run.smithi184.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-09-16T15:44:28.946 DEBUG:teuthology.orchestra.run.smithi184:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-09-16T15:44:28.946 DEBUG:teuthology.orchestra.run.smithi184:> set -e 2024-09-16T15:44:28.946 DEBUG:teuthology.orchestra.run.smithi184:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-09-16T15:44:28.946 DEBUG:teuthology.orchestra.run.smithi184:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-09-16T15:44:28.946 DEBUG:teuthology.orchestra.run.smithi184:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-09-16T15:44:28.946 DEBUG:teuthology.orchestra.run.smithi184:> ') 2024-09-16T15:44:29.036 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:44:29 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-09-16T15:44:29.065 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:44:29 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-16T15:44:29.072 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:44:29.072 DEBUG:teuthology.orchestra.run.smithi184:> ip netns list 2024-09-16T15:44:29.123 INFO:teuthology.orchestra.run.smithi184.stdout:ceph-ns--home-ubuntu-cephtest-mnt.3 (id: 1) 2024-09-16T15:44:29.124 INFO:teuthology.orchestra.run.smithi184.stdout:ceph-ns--home-ubuntu-cephtest-mnt.1 (id: 0) 2024-09-16T15:44:29.124 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.1' netns for '/home/ubuntu/cephtest/mnt.1' 2024-09-16T15:44:29.124 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:44:29.124 DEBUG:teuthology.orchestra.run.smithi184:> sudo ip link set brx.0 up 2024-09-16T15:44:29.179 INFO:tasks.cephfs.fuse_mount:Client client.1 config is {} 2024-09-16T15:44:29.179 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-09-16T15:44:29.180 DEBUG:teuthology.orchestra.run.smithi184:> mkdir -p -v /home/ubuntu/cephtest/mnt.1 2024-09-16T15:44:29.227 INFO:teuthology.orchestra.run.smithi184.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.1' 2024-09-16T15:44:29.228 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-09-16T15:44:29.228 DEBUG:teuthology.orchestra.run.smithi184:> chmod 0000 /home/ubuntu/cephtest/mnt.1 2024-09-16T15:44:29.275 DEBUG:teuthology.orchestra.run.smithi184:> sudo modprobe fuse 2024-09-16T15:44:29.333 DEBUG:teuthology.orchestra.run.smithi184:> cat /proc/self/mounts | awk '{print $2}' 2024-09-16T15:44:29.380 INFO:teuthology.orchestra.run.smithi184.stdout:/sys 2024-09-16T15:44:29.380 INFO:teuthology.orchestra.run.smithi184.stdout:/proc 2024-09-16T15:44:29.380 INFO:teuthology.orchestra.run.smithi184.stdout:/dev 2024-09-16T15:44:29.380 INFO:teuthology.orchestra.run.smithi184.stdout:/dev/pts 2024-09-16T15:44:29.380 INFO:teuthology.orchestra.run.smithi184.stdout:/run 2024-09-16T15:44:29.380 INFO:teuthology.orchestra.run.smithi184.stdout:/ 2024-09-16T15:44:29.380 INFO:teuthology.orchestra.run.smithi184.stdout:/sys/kernel/security 2024-09-16T15:44:29.381 INFO:teuthology.orchestra.run.smithi184.stdout:/dev/shm 2024-09-16T15:44:29.381 INFO:teuthology.orchestra.run.smithi184.stdout:/run/lock 2024-09-16T15:44:29.381 INFO:teuthology.orchestra.run.smithi184.stdout:/sys/fs/cgroup 2024-09-16T15:44:29.381 INFO:teuthology.orchestra.run.smithi184.stdout:/sys/fs/pstore 2024-09-16T15:44:29.381 INFO:teuthology.orchestra.run.smithi184.stdout:/sys/fs/bpf 2024-09-16T15:44:29.381 INFO:teuthology.orchestra.run.smithi184.stdout:/proc/sys/fs/binfmt_misc 2024-09-16T15:44:29.381 INFO:teuthology.orchestra.run.smithi184.stdout:/dev/hugepages 2024-09-16T15:44:29.381 INFO:teuthology.orchestra.run.smithi184.stdout:/dev/mqueue 2024-09-16T15:44:29.381 INFO:teuthology.orchestra.run.smithi184.stdout:/sys/kernel/debug 2024-09-16T15:44:29.381 INFO:teuthology.orchestra.run.smithi184.stdout:/sys/kernel/tracing 2024-09-16T15:44:29.381 INFO:teuthology.orchestra.run.smithi184.stdout:/proc/fs/nfsd 2024-09-16T15:44:29.381 INFO:teuthology.orchestra.run.smithi184.stdout:/sys/fs/fuse/connections 2024-09-16T15:44:29.381 INFO:teuthology.orchestra.run.smithi184.stdout:/sys/kernel/config 2024-09-16T15:44:29.381 INFO:teuthology.orchestra.run.smithi184.stdout:/run/credentials/systemd-sysusers.service 2024-09-16T15:44:29.382 INFO:teuthology.orchestra.run.smithi184.stdout:/run/rpc_pipefs 2024-09-16T15:44:29.382 INFO:teuthology.orchestra.run.smithi184.stdout:/run/user/1000 2024-09-16T15:44:29.382 INFO:teuthology.orchestra.run.smithi184.stdout:/var/lib/ceph 2024-09-16T15:44:29.382 INFO:teuthology.orchestra.run.smithi184.stdout:/var/lib/ceph/osd/ceph-4 2024-09-16T15:44:29.382 INFO:teuthology.orchestra.run.smithi184.stdout:/var/lib/ceph/osd/ceph-5 2024-09-16T15:44:29.382 INFO:teuthology.orchestra.run.smithi184.stdout:/var/lib/ceph/osd/ceph-6 2024-09-16T15:44:29.382 INFO:teuthology.orchestra.run.smithi184.stdout:/var/lib/ceph/osd/ceph-7 2024-09-16T15:44:29.382 INFO:teuthology.orchestra.run.smithi184.stdout:/run/netns 2024-09-16T15:44:29.382 INFO:teuthology.orchestra.run.smithi184.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-09-16T15:44:29.382 INFO:teuthology.orchestra.run.smithi184.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-09-16T15:44:29.382 INFO:teuthology.orchestra.run.smithi184.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-09-16T15:44:29.382 INFO:teuthology.orchestra.run.smithi184.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-09-16T15:44:29.383 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:44:29.383 DEBUG:teuthology.orchestra.run.smithi184:> ls /sys/fs/fuse/connections 2024-09-16T15:44:29.431 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-09-16T15:44:29.432 DEBUG:teuthology.orchestra.run.smithi184:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.1 --id 1) 2024-09-16T15:44:29.476 DEBUG:teuthology.orchestra.run.smithi184:> sudo modprobe fuse 2024-09-16T15:44:29.492 DEBUG:teuthology.orchestra.run.smithi184:> cat /proc/self/mounts | awk '{print $2}' 2024-09-16T15:44:29.544 INFO:teuthology.orchestra.run.smithi184.stdout:/sys 2024-09-16T15:44:29.544 INFO:teuthology.orchestra.run.smithi184.stdout:/proc 2024-09-16T15:44:29.544 INFO:teuthology.orchestra.run.smithi184.stdout:/dev 2024-09-16T15:44:29.544 INFO:teuthology.orchestra.run.smithi184.stdout:/dev/pts 2024-09-16T15:44:29.544 INFO:teuthology.orchestra.run.smithi184.stdout:/run 2024-09-16T15:44:29.544 INFO:teuthology.orchestra.run.smithi184.stdout:/ 2024-09-16T15:44:29.544 INFO:teuthology.orchestra.run.smithi184.stdout:/sys/kernel/security 2024-09-16T15:44:29.544 INFO:teuthology.orchestra.run.smithi184.stdout:/dev/shm 2024-09-16T15:44:29.544 INFO:teuthology.orchestra.run.smithi184.stdout:/run/lock 2024-09-16T15:44:29.545 INFO:teuthology.orchestra.run.smithi184.stdout:/sys/fs/cgroup 2024-09-16T15:44:29.545 INFO:teuthology.orchestra.run.smithi184.stdout:/sys/fs/pstore 2024-09-16T15:44:29.545 INFO:teuthology.orchestra.run.smithi184.stdout:/sys/fs/bpf 2024-09-16T15:44:29.545 INFO:teuthology.orchestra.run.smithi184.stdout:/proc/sys/fs/binfmt_misc 2024-09-16T15:44:29.545 INFO:teuthology.orchestra.run.smithi184.stdout:/dev/hugepages 2024-09-16T15:44:29.545 INFO:teuthology.orchestra.run.smithi184.stdout:/dev/mqueue 2024-09-16T15:44:29.545 INFO:teuthology.orchestra.run.smithi184.stdout:/sys/kernel/debug 2024-09-16T15:44:29.545 INFO:teuthology.orchestra.run.smithi184.stdout:/sys/kernel/tracing 2024-09-16T15:44:29.545 INFO:teuthology.orchestra.run.smithi184.stdout:/proc/fs/nfsd 2024-09-16T15:44:29.545 INFO:teuthology.orchestra.run.smithi184.stdout:/sys/fs/fuse/connections 2024-09-16T15:44:29.545 INFO:teuthology.orchestra.run.smithi184.stdout:/sys/kernel/config 2024-09-16T15:44:29.545 INFO:teuthology.orchestra.run.smithi184.stdout:/run/credentials/systemd-sysusers.service 2024-09-16T15:44:29.545 INFO:teuthology.orchestra.run.smithi184.stdout:/run/rpc_pipefs 2024-09-16T15:44:29.545 INFO:teuthology.orchestra.run.smithi184.stdout:/run/user/1000 2024-09-16T15:44:29.546 INFO:teuthology.orchestra.run.smithi184.stdout:/var/lib/ceph 2024-09-16T15:44:29.546 INFO:teuthology.orchestra.run.smithi184.stdout:/var/lib/ceph/osd/ceph-4 2024-09-16T15:44:29.546 INFO:teuthology.orchestra.run.smithi184.stdout:/var/lib/ceph/osd/ceph-5 2024-09-16T15:44:29.546 INFO:teuthology.orchestra.run.smithi184.stdout:/var/lib/ceph/osd/ceph-6 2024-09-16T15:44:29.546 INFO:teuthology.orchestra.run.smithi184.stdout:/var/lib/ceph/osd/ceph-7 2024-09-16T15:44:29.546 INFO:teuthology.orchestra.run.smithi184.stdout:/run/netns 2024-09-16T15:44:29.546 INFO:teuthology.orchestra.run.smithi184.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-09-16T15:44:29.546 INFO:teuthology.orchestra.run.smithi184.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-09-16T15:44:29.546 INFO:teuthology.orchestra.run.smithi184.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-09-16T15:44:29.546 INFO:teuthology.orchestra.run.smithi184.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-09-16T15:44:29.546 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:44:29.546 DEBUG:teuthology.orchestra.run.smithi184:> ls /sys/fs/fuse/connections 2024-09-16T15:44:29.558 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi184.stderr:2024-09-16T15:44:29.552+0000 7fad09897580 -1 init, newargv = 0x55bb8b448c10 newargc=15 2024-09-16T15:44:29.558 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi184.stderr:ceph-fuse[34843]: starting ceph client 2024-09-16T15:44:29.564 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi184.stderr:ceph-fuse[34843]: starting fuse 2024-09-16T15:44:30.664 DEBUG:teuthology.orchestra.run.smithi184:> sudo modprobe fuse 2024-09-16T15:44:30.681 DEBUG:teuthology.orchestra.run.smithi184:> cat /proc/self/mounts | awk '{print $2}' 2024-09-16T15:44:30.729 INFO:teuthology.orchestra.run.smithi184.stdout:/sys 2024-09-16T15:44:30.729 INFO:teuthology.orchestra.run.smithi184.stdout:/proc 2024-09-16T15:44:30.729 INFO:teuthology.orchestra.run.smithi184.stdout:/dev 2024-09-16T15:44:30.729 INFO:teuthology.orchestra.run.smithi184.stdout:/dev/pts 2024-09-16T15:44:30.729 INFO:teuthology.orchestra.run.smithi184.stdout:/run 2024-09-16T15:44:30.729 INFO:teuthology.orchestra.run.smithi184.stdout:/ 2024-09-16T15:44:30.729 INFO:teuthology.orchestra.run.smithi184.stdout:/sys/kernel/security 2024-09-16T15:44:30.729 INFO:teuthology.orchestra.run.smithi184.stdout:/dev/shm 2024-09-16T15:44:30.729 INFO:teuthology.orchestra.run.smithi184.stdout:/run/lock 2024-09-16T15:44:30.729 INFO:teuthology.orchestra.run.smithi184.stdout:/sys/fs/cgroup 2024-09-16T15:44:30.729 INFO:teuthology.orchestra.run.smithi184.stdout:/sys/fs/pstore 2024-09-16T15:44:30.730 INFO:teuthology.orchestra.run.smithi184.stdout:/sys/fs/bpf 2024-09-16T15:44:30.730 INFO:teuthology.orchestra.run.smithi184.stdout:/proc/sys/fs/binfmt_misc 2024-09-16T15:44:30.730 INFO:teuthology.orchestra.run.smithi184.stdout:/dev/hugepages 2024-09-16T15:44:30.730 INFO:teuthology.orchestra.run.smithi184.stdout:/dev/mqueue 2024-09-16T15:44:30.730 INFO:teuthology.orchestra.run.smithi184.stdout:/sys/kernel/debug 2024-09-16T15:44:30.730 INFO:teuthology.orchestra.run.smithi184.stdout:/sys/kernel/tracing 2024-09-16T15:44:30.730 INFO:teuthology.orchestra.run.smithi184.stdout:/proc/fs/nfsd 2024-09-16T15:44:30.730 INFO:teuthology.orchestra.run.smithi184.stdout:/sys/fs/fuse/connections 2024-09-16T15:44:30.730 INFO:teuthology.orchestra.run.smithi184.stdout:/sys/kernel/config 2024-09-16T15:44:30.730 INFO:teuthology.orchestra.run.smithi184.stdout:/run/credentials/systemd-sysusers.service 2024-09-16T15:44:30.730 INFO:teuthology.orchestra.run.smithi184.stdout:/run/rpc_pipefs 2024-09-16T15:44:30.730 INFO:teuthology.orchestra.run.smithi184.stdout:/run/user/1000 2024-09-16T15:44:30.730 INFO:teuthology.orchestra.run.smithi184.stdout:/var/lib/ceph 2024-09-16T15:44:30.730 INFO:teuthology.orchestra.run.smithi184.stdout:/var/lib/ceph/osd/ceph-4 2024-09-16T15:44:30.731 INFO:teuthology.orchestra.run.smithi184.stdout:/var/lib/ceph/osd/ceph-5 2024-09-16T15:44:30.731 INFO:teuthology.orchestra.run.smithi184.stdout:/var/lib/ceph/osd/ceph-6 2024-09-16T15:44:30.731 INFO:teuthology.orchestra.run.smithi184.stdout:/var/lib/ceph/osd/ceph-7 2024-09-16T15:44:30.731 INFO:teuthology.orchestra.run.smithi184.stdout:/run/netns 2024-09-16T15:44:30.731 INFO:teuthology.orchestra.run.smithi184.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-09-16T15:44:30.731 INFO:teuthology.orchestra.run.smithi184.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-09-16T15:44:30.731 INFO:teuthology.orchestra.run.smithi184.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-09-16T15:44:30.731 INFO:teuthology.orchestra.run.smithi184.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-09-16T15:44:30.731 INFO:teuthology.orchestra.run.smithi184.stdout:/home/ubuntu/cephtest/mnt.1 2024-09-16T15:44:30.732 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:44:30.732 DEBUG:teuthology.orchestra.run.smithi184:> ls /sys/fs/fuse/connections 2024-09-16T15:44:30.783 INFO:teuthology.orchestra.run.smithi184.stdout:45 2024-09-16T15:44:30.783 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [45] 2024-09-16T15:44:30.784 DEBUG:teuthology.orchestra.run.smithi184:> sudo stdin-killer -- python3 -c ' 2024-09-16T15:44:30.784 DEBUG:teuthology.orchestra.run.smithi184:> import glob 2024-09-16T15:44:30.784 DEBUG:teuthology.orchestra.run.smithi184:> import re 2024-09-16T15:44:30.784 DEBUG:teuthology.orchestra.run.smithi184:> import os 2024-09-16T15:44:30.784 DEBUG:teuthology.orchestra.run.smithi184:> import subprocess 2024-09-16T15:44:30.784 DEBUG:teuthology.orchestra.run.smithi184:> 2024-09-16T15:44:30.784 DEBUG:teuthology.orchestra.run.smithi184:> def _find_admin_socket(client_name): 2024-09-16T15:44:30.784 DEBUG:teuthology.orchestra.run.smithi184:> asok_path = "/var/run/ceph/ceph-client.1.*.asok" 2024-09-16T15:44:30.784 DEBUG:teuthology.orchestra.run.smithi184:> files = glob.glob(asok_path) 2024-09-16T15:44:30.784 DEBUG:teuthology.orchestra.run.smithi184:> mountpoint = "/home/ubuntu/cephtest/mnt.1" 2024-09-16T15:44:30.784 DEBUG:teuthology.orchestra.run.smithi184:> 2024-09-16T15:44:30.784 DEBUG:teuthology.orchestra.run.smithi184:> # Given a non-glob path, it better be there 2024-09-16T15:44:30.784 DEBUG:teuthology.orchestra.run.smithi184:> if "*" not in asok_path: 2024-09-16T15:44:30.784 DEBUG:teuthology.orchestra.run.smithi184:> assert(len(files) == 1) 2024-09-16T15:44:30.785 DEBUG:teuthology.orchestra.run.smithi184:> return files[0] 2024-09-16T15:44:30.785 DEBUG:teuthology.orchestra.run.smithi184:> 2024-09-16T15:44:30.785 DEBUG:teuthology.orchestra.run.smithi184:> for f in files: 2024-09-16T15:44:30.785 DEBUG:teuthology.orchestra.run.smithi184:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-09-16T15:44:30.785 DEBUG:teuthology.orchestra.run.smithi184:> if os.path.exists("/proc/{0}".format(pid)): 2024-09-16T15:44:30.785 DEBUG:teuthology.orchestra.run.smithi184:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-09-16T15:44:30.785 DEBUG:teuthology.orchestra.run.smithi184:> contents = proc_f.read() 2024-09-16T15:44:30.785 DEBUG:teuthology.orchestra.run.smithi184:> if mountpoint in contents: 2024-09-16T15:44:30.785 DEBUG:teuthology.orchestra.run.smithi184:> return f 2024-09-16T15:44:30.785 DEBUG:teuthology.orchestra.run.smithi184:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-09-16T15:44:30.785 DEBUG:teuthology.orchestra.run.smithi184:> 2024-09-16T15:44:30.785 DEBUG:teuthology.orchestra.run.smithi184:> print(_find_admin_socket("client.1")) 2024-09-16T15:44:30.785 DEBUG:teuthology.orchestra.run.smithi184:> ' 2024-09-16T15:44:30.909 INFO:teuthology.orchestra.run.smithi184.stdout:/var/run/ceph/ceph-client.1.34843.asok 2024-09-16T15:44:30.913 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:44:30 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-16T15:44:30.921 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.1.34843.asok 2024-09-16T15:44:30.922 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:44:30.922 DEBUG:teuthology.orchestra.run.smithi184:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.1.34843.asok status 2024-09-16T15:44:31.072 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-16T15:44:31.072 INFO:teuthology.orchestra.run.smithi184.stdout: "metadata": { 2024-09-16T15:44:31.072 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph_sha1": "33f95d2ae1bcbe7eb04404303f179eeeaa6982df", 2024-09-16T15:44:31.072 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph_version": "ceph version 19.3.0-4938-g33f95d2a (33f95d2ae1bcbe7eb04404303f179eeeaa6982df) squid (dev)", 2024-09-16T15:44:31.072 INFO:teuthology.orchestra.run.smithi184.stdout: "entity_id": "1", 2024-09-16T15:44:31.072 INFO:teuthology.orchestra.run.smithi184.stdout: "hostname": "smithi184", 2024-09-16T15:44:31.072 INFO:teuthology.orchestra.run.smithi184.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.1", 2024-09-16T15:44:31.072 INFO:teuthology.orchestra.run.smithi184.stdout: "pid": "34843", 2024-09-16T15:44:31.072 INFO:teuthology.orchestra.run.smithi184.stdout: "root": "/" 2024-09-16T15:44:31.072 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-16T15:44:31.072 INFO:teuthology.orchestra.run.smithi184.stdout: "dentry_count": 0, 2024-09-16T15:44:31.072 INFO:teuthology.orchestra.run.smithi184.stdout: "dentry_pinned_count": 0, 2024-09-16T15:44:31.073 INFO:teuthology.orchestra.run.smithi184.stdout: "id": 5023, 2024-09-16T15:44:31.073 INFO:teuthology.orchestra.run.smithi184.stdout: "inst": { 2024-09-16T15:44:31.073 INFO:teuthology.orchestra.run.smithi184.stdout: "name": { 2024-09-16T15:44:31.073 INFO:teuthology.orchestra.run.smithi184.stdout: "type": "client", 2024-09-16T15:44:31.073 INFO:teuthology.orchestra.run.smithi184.stdout: "num": 5023 2024-09-16T15:44:31.073 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-16T15:44:31.073 INFO:teuthology.orchestra.run.smithi184.stdout: "addr": { 2024-09-16T15:44:31.073 INFO:teuthology.orchestra.run.smithi184.stdout: "type": "v1", 2024-09-16T15:44:31.073 INFO:teuthology.orchestra.run.smithi184.stdout: "addr": "192.168.0.1:0", 2024-09-16T15:44:31.073 INFO:teuthology.orchestra.run.smithi184.stdout: "nonce": 4235670546 2024-09-16T15:44:31.073 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-16T15:44:31.073 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-16T15:44:31.073 INFO:teuthology.orchestra.run.smithi184.stdout: "addr": { 2024-09-16T15:44:31.073 INFO:teuthology.orchestra.run.smithi184.stdout: "type": "v1", 2024-09-16T15:44:31.073 INFO:teuthology.orchestra.run.smithi184.stdout: "addr": "192.168.0.1:0", 2024-09-16T15:44:31.074 INFO:teuthology.orchestra.run.smithi184.stdout: "nonce": 4235670546 2024-09-16T15:44:31.074 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-16T15:44:31.074 INFO:teuthology.orchestra.run.smithi184.stdout: "inst_str": "client.5023 192.168.0.1:0/4235670546", 2024-09-16T15:44:31.074 INFO:teuthology.orchestra.run.smithi184.stdout: "addr_str": "192.168.0.1:0/4235670546", 2024-09-16T15:44:31.074 INFO:teuthology.orchestra.run.smithi184.stdout: "inode_count": 1, 2024-09-16T15:44:31.074 INFO:teuthology.orchestra.run.smithi184.stdout: "mds_epoch": 27, 2024-09-16T15:44:31.074 INFO:teuthology.orchestra.run.smithi184.stdout: "osd_epoch": 44, 2024-09-16T15:44:31.074 INFO:teuthology.orchestra.run.smithi184.stdout: "osd_epoch_barrier": 0, 2024-09-16T15:44:31.074 INFO:teuthology.orchestra.run.smithi184.stdout: "blocklisted": false, 2024-09-16T15:44:31.074 INFO:teuthology.orchestra.run.smithi184.stdout: "fs_name": "cephfs" 2024-09-16T15:44:31.074 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-16T15:44:31.087 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:44:31.087 DEBUG:teuthology.orchestra.run.smithi184:> stat --file-system '--printf=%T 2024-09-16T15:44:31.087 DEBUG:teuthology.orchestra.run.smithi184:> ' -- /home/ubuntu/cephtest/mnt.1 2024-09-16T15:44:31.093 INFO:teuthology.orchestra.run.smithi184.stdout:fuseblk 2024-09-16T15:44:31.093 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2024-09-16T15:44:31.093 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:44:31.094 DEBUG:teuthology.orchestra.run.smithi184:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.1 2024-09-16T15:44:31.146 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs flag set enable_multiple true --yes-i-really-mean-it 2024-09-16T15:44:31.292 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:31.287+0000 7f9158865640 1 -- 172.21.15.123:0/2947369844 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9140004010 msgr2=0x7f9140004810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:44:31.292 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:31.287+0000 7f9158865640 1 --2- 172.21.15.123:0/2947369844 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9140004010 0x7f9140004810 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7f914c001dc0 tx=0x7f914c01cb30 comp rx=0 tx=0).stop 2024-09-16T15:44:31.292 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:31.287+0000 7f9152d9d640 1 -- 172.21.15.123:0/2947369844 reap_dead start 2024-09-16T15:44:31.292 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:31.287+0000 7f9158865640 1 -- 172.21.15.123:0/2947369844 shutdown_connections 2024-09-16T15:44:31.292 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:31.287+0000 7f9158865640 1 -- 172.21.15.123:0/2947369844 >> 172.21.15.123:0/2947369844 conn(0x7f915409aa40 msgr2=0x7f915409ae50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:44:31.292 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:31.287+0000 7f9158865640 1 -- 172.21.15.123:0/2947369844 shutdown_connections 2024-09-16T15:44:31.292 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:31.287+0000 7f9158865640 1 -- 172.21.15.123:0/2947369844 wait complete. 2024-09-16T15:44:31.293 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:31.287+0000 7f9158865640 1 Processor -- start 2024-09-16T15:44:31.293 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:31.287+0000 7f9158865640 1 -- start start 2024-09-16T15:44:31.295 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:31.287+0000 7f9158865640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f9140004010 0x7f915415e940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:44:31.295 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:31.287+0000 7f9158865640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f91541f2700 0x7f91541f6690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:44:31.295 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:31.287+0000 7f9158865640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f9154158fb0 0x7f91540af820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:44:31.295 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:31.287+0000 7f9158865640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f91540cca00 con 0x7f91541f2700 2024-09-16T15:44:31.295 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:31.287+0000 7f9158865640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f91540cc880 con 0x7f9140004010 2024-09-16T15:44:31.295 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:31.287+0000 7f9158865640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f91540ccb80 con 0x7f9154158fb0 2024-09-16T15:44:31.295 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:31.287+0000 7f9151d9b640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f91541f2700 0x7f91541f6690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:44:31.296 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:31.287+0000 7f9151d9b640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f91541f2700 0x7f91541f6690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:38942/0 (socket says 172.21.15.123:38942) 2024-09-16T15:44:31.296 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:31.287+0000 7f9151d9b640 1 -- 172.21.15.123:0/101987432 learned_addr learned my addr 172.21.15.123:0/101987432 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:44:31.296 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:31.287+0000 7f915259c640 1 --2- 172.21.15.123:0/101987432 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f9140004010 0x7f915415e940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:44:31.296 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:31.287+0000 7f9152d9d640 1 --2- 172.21.15.123:0/101987432 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f9154158fb0 0x7f91540af820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:44:31.296 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:31.287+0000 7f9151d9b640 1 -- 172.21.15.123:0/101987432 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f9154158fb0 msgr2=0x7f91540af820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:44:31.296 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:31.287+0000 7f9151d9b640 1 --2- 172.21.15.123:0/101987432 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f9154158fb0 0x7f91540af820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:31.296 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:31.287+0000 7f9151d9b640 1 -- 172.21.15.123:0/101987432 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f9140004010 msgr2=0x7f915415e940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:44:31.296 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:31.287+0000 7f9151d9b640 1 --2- 172.21.15.123:0/101987432 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f9140004010 0x7f915415e940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:31.296 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:31.287+0000 7f9151d9b640 1 -- 172.21.15.123:0/101987432 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f91541f2b90 con 0x7f91541f2700 2024-09-16T15:44:31.296 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:31.287+0000 7f9151d9b640 1 --2- 172.21.15.123:0/101987432 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f91541f2700 0x7f91541f6690 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7f914c01b230 tx=0x7f914c006cb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:44:31.296 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:31.287+0000 7f914affd640 1 -- 172.21.15.123:0/101987432 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f914c003b80 con 0x7f91541f2700 2024-09-16T15:44:31.296 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:31.287+0000 7f914affd640 1 -- 172.21.15.123:0/101987432 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f914c003e50 con 0x7f91541f2700 2024-09-16T15:44:31.296 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:31.287+0000 7f914affd640 1 -- 172.21.15.123:0/101987432 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f914c006330 con 0x7f91541f2700 2024-09-16T15:44:31.297 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:31.287+0000 7f9158865640 1 -- 172.21.15.123:0/101987432 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f91541f2e70 con 0x7f91541f2700 2024-09-16T15:44:31.297 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:31.287+0000 7f9158865640 1 -- 172.21.15.123:0/101987432 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f91541f3430 con 0x7f91541f2700 2024-09-16T15:44:31.298 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:31.291+0000 7f9158865640 1 -- 172.21.15.123:0/101987432 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f91540cd0e0 con 0x7f91541f2700 2024-09-16T15:44:31.298 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:31.291+0000 7f914affd640 1 -- 172.21.15.123:0/101987432 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 7) ==== 97233+0+0 (secure 0 0 0) 0x7f914c0064d0 con 0x7f91541f2700 2024-09-16T15:44:31.298 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:31.291+0000 7f914affd640 1 --2- 172.21.15.123:0/101987432 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f9134073070 0x7f9134075530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:44:31.298 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:31.291+0000 7f914affd640 1 -- 172.21.15.123:0/101987432 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(44..44 src has 1..44) ==== 5709+0+0 (secure 0 0 0) 0x7f914c12e030 con 0x7f91541f2700 2024-09-16T15:44:31.298 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:31.291+0000 7f915259c640 1 --2- 172.21.15.123:0/101987432 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f9134073070 0x7f9134075530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:44:31.298 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:31.291+0000 7f915259c640 1 --2- 172.21.15.123:0/101987432 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f9134073070 0x7f9134075530 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f913c0046e0 tx=0x7f913c001500 comp rx=0 tx=0).ready entity=mgr.4135 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:44:31.301 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:31.295+0000 7f914affd640 1 -- 172.21.15.123:0/101987432 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f914c016890 con 0x7f91541f2700 2024-09-16T15:44:31.517 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:31.511+0000 7f9158865640 1 -- 172.21.15.123:0/101987432 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "fs flag set", "flag_name": "enable_multiple", "val": "true", "yes_i_really_mean_it": true} v 0) -- 0x7f91540b3400 con 0x7f91541f2700 2024-09-16T15:44:32.375 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:32.371+0000 7f914affd640 1 -- 172.21.15.123:0/101987432 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "fs flag set", "flag_name": "enable_multiple", "val": "true", "yes_i_really_mean_it": true}]=0 v28) ==== 136+0+0 (secure 0 0 0) 0x7f914c0f8f80 con 0x7f91541f2700 2024-09-16T15:44:32.378 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:32.375+0000 7f9158865640 1 -- 172.21.15.123:0/101987432 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f9134073070 msgr2=0x7f9134075530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:44:32.378 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:32.375+0000 7f9158865640 1 --2- 172.21.15.123:0/101987432 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f9134073070 0x7f9134075530 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f913c0046e0 tx=0x7f913c001500 comp rx=0 tx=0).stop 2024-09-16T15:44:32.378 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:32.375+0000 7f9158865640 1 -- 172.21.15.123:0/101987432 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f91541f2700 msgr2=0x7f91541f6690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:44:32.378 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:32.375+0000 7f9158865640 1 --2- 172.21.15.123:0/101987432 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f91541f2700 0x7f91541f6690 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7f914c01b230 tx=0x7f914c006cb0 comp rx=0 tx=0).stop 2024-09-16T15:44:32.378 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:32.375+0000 7f9158865640 1 -- 172.21.15.123:0/101987432 shutdown_connections 2024-09-16T15:44:32.378 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:32.375+0000 7f9158865640 1 --2- 172.21.15.123:0/101987432 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f9134073070 0x7f9134075530 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:32.378 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:32.375+0000 7f9158865640 1 --2- 172.21.15.123:0/101987432 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f9154158fb0 0x7f91540af820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:32.378 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:32.375+0000 7f9158865640 1 --2- 172.21.15.123:0/101987432 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f91541f2700 0x7f91541f6690 unknown :-1 s=CLOSED pgs=489 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:32.378 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:32.375+0000 7f9158865640 1 --2- 172.21.15.123:0/101987432 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f9140004010 0x7f915415e940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:32.378 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:32.375+0000 7f9158865640 1 -- 172.21.15.123:0/101987432 >> 172.21.15.123:0/101987432 conn(0x7f915409aa40 msgr2=0x7f91540719e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:44:32.379 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:32.375+0000 7f9158865640 1 -- 172.21.15.123:0/101987432 shutdown_connections 2024-09-16T15:44:32.379 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:32.375+0000 7f9158865640 1 -- 172.21.15.123:0/101987432 wait complete. 2024-09-16T15:44:32.399 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2024-09-16T15:44:32.529 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:32.527+0000 7f23c0827640 1 -- 172.21.15.123:0/4284303435 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f23bc0726c0 msgr2=0x7f23bc073ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:44:32.530 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:32.527+0000 7f23c0827640 1 --2- 172.21.15.123:0/4284303435 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f23bc0726c0 0x7f23bc073ef0 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7f23bc0cd650 tx=0x7f23b001c540 comp rx=0 tx=0).stop 2024-09-16T15:44:32.530 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:32.527+0000 7f23c0827640 1 -- 172.21.15.123:0/4284303435 shutdown_connections 2024-09-16T15:44:32.530 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:32.527+0000 7f23c0827640 1 --2- 172.21.15.123:0/4284303435 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f23bc074430 0x7f23bc0ca160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:32.530 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:32.527+0000 7f23c0827640 1 --2- 172.21.15.123:0/4284303435 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f23bc0726c0 0x7f23bc073ef0 unknown :-1 s=CLOSED pgs=490 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:32.530 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:32.527+0000 7f23c0827640 1 --2- 172.21.15.123:0/4284303435 >> v2:172.21.15.184:3300/0 conn(0x7f23bc15bca0 0x7f23bc0d0630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:32.530 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:32.527+0000 7f23c0827640 1 -- 172.21.15.123:0/4284303435 >> 172.21.15.123:0/4284303435 conn(0x7f23bc0cfb10 msgr2=0x7f23bc0cff20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:44:32.530 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:32.527+0000 7f23c0827640 1 -- 172.21.15.123:0/4284303435 shutdown_connections 2024-09-16T15:44:32.530 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:32.527+0000 7f23c0827640 1 -- 172.21.15.123:0/4284303435 wait complete. 2024-09-16T15:44:32.532 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:32.527+0000 7f23c0827640 1 Processor -- start 2024-09-16T15:44:32.532 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:32.527+0000 7f23c0827640 1 -- start start 2024-09-16T15:44:32.533 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:32.531+0000 7f23c0827640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f23bc0726c0 0x7f23bc14b8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:44:32.533 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:32.531+0000 7f23c0827640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f23bc074430 0x7f23bc14cc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:44:32.533 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:32.531+0000 7f23c0827640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f23bc15bca0 0x7f23bc165c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:44:32.533 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:32.531+0000 7f23c0827640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f23bc06ec50 con 0x7f23bc0726c0 2024-09-16T15:44:32.533 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:32.531+0000 7f23c0827640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f23bc06ead0 con 0x7f23bc15bca0 2024-09-16T15:44:32.533 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:32.531+0000 7f23c0827640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f23bc06edd0 con 0x7f23bc074430 2024-09-16T15:44:32.533 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:32.531+0000 7f23ba59c640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f23bc0726c0 0x7f23bc14b8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:44:32.533 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:32.531+0000 7f23ba59c640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f23bc0726c0 0x7f23bc14b8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:54956/0 (socket says 172.21.15.123:54956) 2024-09-16T15:44:32.533 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:32.531+0000 7f23ba59c640 1 -- 172.21.15.123:0/729328448 learned_addr learned my addr 172.21.15.123:0/729328448 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:44:32.533 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:32.531+0000 7f23b9d9b640 1 --2- 172.21.15.123:0/729328448 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f23bc074430 0x7f23bc14cc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:44:32.534 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:32.531+0000 7f23bad9d640 1 --2- 172.21.15.123:0/729328448 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f23bc15bca0 0x7f23bc165c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:44:32.534 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:32.531+0000 7f23ba59c640 1 -- 172.21.15.123:0/729328448 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f23bc074430 msgr2=0x7f23bc14cc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:44:32.534 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:32.531+0000 7f23ba59c640 1 --2- 172.21.15.123:0/729328448 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f23bc074430 0x7f23bc14cc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:32.534 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:32.531+0000 7f23ba59c640 1 -- 172.21.15.123:0/729328448 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f23bc15bca0 msgr2=0x7f23bc165c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:44:32.534 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:32.531+0000 7f23ba59c640 1 --2- 172.21.15.123:0/729328448 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f23bc15bca0 0x7f23bc165c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:32.534 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:32.531+0000 7f23ba59c640 1 -- 172.21.15.123:0/729328448 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f23bc0cee80 con 0x7f23bc0726c0 2024-09-16T15:44:32.534 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:32.531+0000 7f23ba59c640 1 --2- 172.21.15.123:0/729328448 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f23bc0726c0 0x7f23bc14b8b0 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7f23ac00c940 tx=0x7f23ac00ce00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:44:32.534 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:32.531+0000 7f23aaffd640 1 -- 172.21.15.123:0/729328448 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f23ac004220 con 0x7f23bc0726c0 2024-09-16T15:44:32.534 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:32.531+0000 7f23aaffd640 1 -- 172.21.15.123:0/729328448 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f23ac007950 con 0x7f23bc0726c0 2024-09-16T15:44:32.534 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:32.531+0000 7f23c0827640 1 -- 172.21.15.123:0/729328448 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f23bc164e80 con 0x7f23bc0726c0 2024-09-16T15:44:32.535 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:32.531+0000 7f23aaffd640 1 -- 172.21.15.123:0/729328448 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f23ac0044b0 con 0x7f23bc0726c0 2024-09-16T15:44:32.535 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:32.531+0000 7f23c0827640 1 -- 172.21.15.123:0/729328448 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f23bc165440 con 0x7f23bc0726c0 2024-09-16T15:44:32.536 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:32.531+0000 7f23c0827640 1 -- 172.21.15.123:0/729328448 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f23bc15f630 con 0x7f23bc0726c0 2024-09-16T15:44:32.536 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:32.531+0000 7f23aaffd640 1 -- 172.21.15.123:0/729328448 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 7) ==== 97233+0+0 (secure 0 0 0) 0x7f23ac003600 con 0x7f23bc0726c0 2024-09-16T15:44:32.536 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:32.531+0000 7f23aaffd640 1 --2- 172.21.15.123:0/729328448 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f239c073070 0x7f239c075530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:44:32.536 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:32.531+0000 7f23aaffd640 1 -- 172.21.15.123:0/729328448 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(46..46 src has 1..46) ==== 5767+0+0 (secure 0 0 0) 0x7f23ac094370 con 0x7f23bc0726c0 2024-09-16T15:44:32.536 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:32.531+0000 7f23b9d9b640 1 --2- 172.21.15.123:0/729328448 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f239c073070 0x7f239c075530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:44:32.536 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:32.535+0000 7f23b9d9b640 1 --2- 172.21.15.123:0/729328448 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f239c073070 0x7f239c075530 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f23b00af000 tx=0x7f23b001ad80 comp rx=0 tx=0).ready entity=mgr.4135 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:44:32.538 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:32.535+0000 7f23aaffd640 1 -- 172.21.15.123:0/729328448 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f23ac014030 con 0x7f23bc0726c0 2024-09-16T15:44:32.734 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:32.731+0000 7f23c0827640 1 -- 172.21.15.123:0/729328448 --> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) -- 0x7f23bc0700e0 con 0x7f239c073070 2024-09-16T15:44:32.737 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:32.735+0000 7f23aaffd640 1 -- 172.21.15.123:0/729328448 <== mgr.4135 v2:172.21.15.184:6832/3114346423 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+41 (secure 0 0 0) 0x7f23bc0700e0 con 0x7f239c073070 2024-09-16T15:44:32.737 INFO:teuthology.orchestra.run.smithi123.stdout:[ 2024-09-16T15:44:32.738 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-09-16T15:44:32.738 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "cephfs" 2024-09-16T15:44:32.738 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-16T15:44:32.738 INFO:teuthology.orchestra.run.smithi123.stdout:] 2024-09-16T15:44:32.740 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:32.739+0000 7f23c0827640 1 -- 172.21.15.123:0/729328448 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f239c073070 msgr2=0x7f239c075530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:44:32.740 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:32.739+0000 7f23c0827640 1 --2- 172.21.15.123:0/729328448 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f239c073070 0x7f239c075530 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f23b00af000 tx=0x7f23b001ad80 comp rx=0 tx=0).stop 2024-09-16T15:44:32.740 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:32.739+0000 7f23c0827640 1 -- 172.21.15.123:0/729328448 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f23bc0726c0 msgr2=0x7f23bc14b8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:44:32.740 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:32.739+0000 7f23c0827640 1 --2- 172.21.15.123:0/729328448 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f23bc0726c0 0x7f23bc14b8b0 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7f23ac00c940 tx=0x7f23ac00ce00 comp rx=0 tx=0).stop 2024-09-16T15:44:32.740 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:32.739+0000 7f23c0827640 1 -- 172.21.15.123:0/729328448 shutdown_connections 2024-09-16T15:44:32.740 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:32.739+0000 7f23c0827640 1 --2- 172.21.15.123:0/729328448 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f239c073070 0x7f239c075530 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:32.740 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:32.739+0000 7f23c0827640 1 --2- 172.21.15.123:0/729328448 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f23bc15bca0 0x7f23bc165c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:32.740 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:32.739+0000 7f23c0827640 1 --2- 172.21.15.123:0/729328448 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f23bc074430 0x7f23bc14cc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:32.740 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:32.739+0000 7f23c0827640 1 --2- 172.21.15.123:0/729328448 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f23bc0726c0 0x7f23bc14b8b0 unknown :-1 s=CLOSED pgs=491 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:32.741 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:32.739+0000 7f23c0827640 1 -- 172.21.15.123:0/729328448 >> 172.21.15.123:0/729328448 conn(0x7f23bc0cfb10 msgr2=0x7f23bc0c3b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:44:32.741 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:32.739+0000 7f23c0827640 1 -- 172.21.15.123:0/729328448 shutdown_connections 2024-09-16T15:44:32.741 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:32.739+0000 7f23c0827640 1 -- 172.21.15.123:0/729328448 wait complete. 2024-09-16T15:44:32.756 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-09-16T15:44:32.880 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:32.879+0000 7fca7f5d0640 1 -- 172.21.15.123:0/2722149533 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fca780cc1a0 msgr2=0x7fca7815ddc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:44:32.881 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:32.879+0000 7fca7f5d0640 1 --2- 172.21.15.123:0/2722149533 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fca780cc1a0 0x7fca7815ddc0 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7fca70009c80 tx=0x7fca7001cce0 comp rx=0 tx=0).stop 2024-09-16T15:44:32.881 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:32.879+0000 7fca7f5d0640 1 -- 172.21.15.123:0/2722149533 shutdown_connections 2024-09-16T15:44:32.881 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:32.879+0000 7fca7f5d0640 1 --2- 172.21.15.123:0/2722149533 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fca780cc1a0 0x7fca7815ddc0 unknown :-1 s=CLOSED pgs=492 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:32.881 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:32.879+0000 7fca7f5d0640 1 --2- 172.21.15.123:0/2722149533 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fca780ca650 0x7fca780cbc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:32.881 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:32.879+0000 7fca7f5d0640 1 -- 172.21.15.123:0/2722149533 >> 172.21.15.123:0/2722149533 conn(0x7fca780b1710 msgr2=0x7fca780b1b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:44:32.881 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:32.879+0000 7fca7f5d0640 1 -- 172.21.15.123:0/2722149533 shutdown_connections 2024-09-16T15:44:32.881 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:32.879+0000 7fca7f5d0640 1 -- 172.21.15.123:0/2722149533 wait complete. 2024-09-16T15:44:32.883 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:32.879+0000 7fca7f5d0640 1 Processor -- start 2024-09-16T15:44:32.884 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:32.879+0000 7fca7f5d0640 1 -- start start 2024-09-16T15:44:32.884 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:32.879+0000 7fca7f5d0640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fca780ca650 0x7fca781f70b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:44:32.884 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:32.879+0000 7fca7f5d0640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fca7815a170 0x7fca7814a6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:44:32.884 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:32.879+0000 7fca7f5d0640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fca781f75f0 0x7fca7814abe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:44:32.884 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:32.879+0000 7fca7f5d0640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fca7806eca0 con 0x7fca7815a170 2024-09-16T15:44:32.884 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:32.879+0000 7fca7f5d0640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fca7806eb20 con 0x7fca781f75f0 2024-09-16T15:44:32.884 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:32.879+0000 7fca7f5d0640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fca7806ee20 con 0x7fca780ca650 2024-09-16T15:44:32.884 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:32.879+0000 7fca7cb6b640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fca7815a170 0x7fca7814a6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:44:32.884 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:32.879+0000 7fca7cb6b640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fca7815a170 0x7fca7814a6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:54978/0 (socket says 172.21.15.123:54978) 2024-09-16T15:44:32.884 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:32.879+0000 7fca7cb6b640 1 -- 172.21.15.123:0/3080164982 learned_addr learned my addr 172.21.15.123:0/3080164982 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:44:32.885 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:32.879+0000 7fca7db6d640 1 --2- 172.21.15.123:0/3080164982 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fca781f75f0 0x7fca7814abe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:44:32.885 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:32.879+0000 7fca7d36c640 1 --2- 172.21.15.123:0/3080164982 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fca780ca650 0x7fca781f70b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:44:32.885 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:32.879+0000 7fca7cb6b640 1 -- 172.21.15.123:0/3080164982 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fca780ca650 msgr2=0x7fca781f70b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:44:32.885 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:32.879+0000 7fca7cb6b640 1 --2- 172.21.15.123:0/3080164982 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fca780ca650 0x7fca781f70b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:32.885 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:32.879+0000 7fca7cb6b640 1 -- 172.21.15.123:0/3080164982 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fca781f75f0 msgr2=0x7fca7814abe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:44:32.885 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:32.879+0000 7fca7cb6b640 1 --2- 172.21.15.123:0/3080164982 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fca781f75f0 0x7fca7814abe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:32.885 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:32.879+0000 7fca7cb6b640 1 -- 172.21.15.123:0/3080164982 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fca781f7b50 con 0x7fca7815a170 2024-09-16T15:44:32.885 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:32.883+0000 7fca7cb6b640 1 --2- 172.21.15.123:0/3080164982 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fca7815a170 0x7fca7814a6a0 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7fca6800dbd0 tx=0x7fca6800a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:44:32.885 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:32.883+0000 7fca6dffb640 1 -- 172.21.15.123:0/3080164982 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca68005220 con 0x7fca7815a170 2024-09-16T15:44:32.885 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:32.883+0000 7fca6dffb640 1 -- 172.21.15.123:0/3080164982 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fca6800ae50 con 0x7fca7815a170 2024-09-16T15:44:32.885 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:32.883+0000 7fca7f5d0640 1 -- 172.21.15.123:0/3080164982 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fca781f7e30 con 0x7fca7815a170 2024-09-16T15:44:32.885 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:32.883+0000 7fca6dffb640 1 -- 172.21.15.123:0/3080164982 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca680047b0 con 0x7fca7815a170 2024-09-16T15:44:32.886 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:32.883+0000 7fca7f5d0640 1 -- 172.21.15.123:0/3080164982 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fca781f83f0 con 0x7fca7815a170 2024-09-16T15:44:32.886 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:32.883+0000 7fca7f5d0640 1 -- 172.21.15.123:0/3080164982 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fca780cbc60 con 0x7fca7815a170 2024-09-16T15:44:32.886 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:32.883+0000 7fca6dffb640 1 -- 172.21.15.123:0/3080164982 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 7) ==== 97233+0+0 (secure 0 0 0) 0x7fca68004950 con 0x7fca7815a170 2024-09-16T15:44:32.887 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:32.883+0000 7fca6dffb640 1 --2- 172.21.15.123:0/3080164982 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fca5c0730a0 0x7fca5c075560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:44:32.887 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:32.883+0000 7fca6dffb640 1 -- 172.21.15.123:0/3080164982 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(46..46 src has 1..46) ==== 5767+0+0 (secure 0 0 0) 0x7fca68094890 con 0x7fca7815a170 2024-09-16T15:44:32.887 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:32.883+0000 7fca7d36c640 1 --2- 172.21.15.123:0/3080164982 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fca5c0730a0 0x7fca5c075560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:44:32.887 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:32.883+0000 7fca7d36c640 1 --2- 172.21.15.123:0/3080164982 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fca5c0730a0 0x7fca5c075560 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fca74001650 tx=0x7fca74009040 comp rx=0 tx=0).ready entity=mgr.4135 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:44:32.889 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:32.887+0000 7fca6dffb640 1 -- 172.21.15.123:0/3080164982 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7fca6805f710 con 0x7fca7815a170 2024-09-16T15:44:33.061 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:33.059+0000 7fca7f5d0640 1 -- 172.21.15.123:0/3080164982 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) -- 0x7fca780697b0 con 0x7fca7815a170 2024-09-16T15:44:33.062 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:33.059+0000 7fca6dffb640 1 -- 172.21.15.123:0/3080164982 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v30)=0 v30) ==== 122+0+0 (secure 0 0 0) 0x7fca6805f010 con 0x7fca7815a170 2024-09-16T15:44:33.064 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:33.063+0000 7fca7f5d0640 1 -- 172.21.15.123:0/3080164982 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fca5c0730a0 msgr2=0x7fca5c075560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:44:33.065 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:33.063+0000 7fca7f5d0640 1 --2- 172.21.15.123:0/3080164982 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fca5c0730a0 0x7fca5c075560 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fca74001650 tx=0x7fca74009040 comp rx=0 tx=0).stop 2024-09-16T15:44:33.065 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:33.063+0000 7fca7f5d0640 1 -- 172.21.15.123:0/3080164982 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fca7815a170 msgr2=0x7fca7814a6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:44:33.065 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:33.063+0000 7fca7f5d0640 1 --2- 172.21.15.123:0/3080164982 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fca7815a170 0x7fca7814a6a0 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7fca6800dbd0 tx=0x7fca6800a620 comp rx=0 tx=0).stop 2024-09-16T15:44:33.065 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:33.063+0000 7fca7f5d0640 1 -- 172.21.15.123:0/3080164982 shutdown_connections 2024-09-16T15:44:33.065 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:33.063+0000 7fca7f5d0640 1 --2- 172.21.15.123:0/3080164982 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fca5c0730a0 0x7fca5c075560 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:33.065 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:33.063+0000 7fca7f5d0640 1 --2- 172.21.15.123:0/3080164982 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fca781f75f0 0x7fca7814abe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:33.065 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:33.063+0000 7fca7f5d0640 1 --2- 172.21.15.123:0/3080164982 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fca7815a170 0x7fca7814a6a0 unknown :-1 s=CLOSED pgs=493 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:33.065 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:33.063+0000 7fca7f5d0640 1 --2- 172.21.15.123:0/3080164982 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fca780ca650 0x7fca781f70b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:33.065 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:33.063+0000 7fca7f5d0640 1 -- 172.21.15.123:0/3080164982 >> 172.21.15.123:0/3080164982 conn(0x7fca780b1710 msgr2=0x7fca7806faf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:44:33.065 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:33.063+0000 7fca7f5d0640 1 -- 172.21.15.123:0/3080164982 shutdown_connections 2024-09-16T15:44:33.065 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:33.063+0000 7fca7f5d0640 1 -- 172.21.15.123:0/3080164982 wait complete. 2024-09-16T15:44:33.081 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2024-09-16T15:44:33.082 DEBUG:teuthology.orchestra.run.smithi123:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-16T15:44:33.087 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-09-16T15:44:33.257 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:33.255+0000 7fc935595640 1 -- 172.21.15.123:0/565004109 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc93015bda0 msgr2=0x7fc9300d0310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:44:33.257 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:33.255+0000 7fc935595640 1 --2- 172.21.15.123:0/565004109 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc93015bda0 0x7fc9300d0310 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7fc924009900 tx=0x7fc92401c850 comp rx=0 tx=0).stop 2024-09-16T15:44:33.257 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:33.255+0000 7fc935595640 1 -- 172.21.15.123:0/565004109 shutdown_connections 2024-09-16T15:44:33.257 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:33.255+0000 7fc935595640 1 --2- 172.21.15.123:0/565004109 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fc9300d2320 0x7fc9300ccab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:33.257 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:33.255+0000 7fc935595640 1 --2- 172.21.15.123:0/565004109 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc93015bda0 0x7fc9300d0310 unknown :-1 s=CLOSED pgs=495 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:33.257 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:33.255+0000 7fc935595640 1 -- 172.21.15.123:0/565004109 >> 172.21.15.123:0/565004109 conn(0x7fc930074bd0 msgr2=0x7fc93009aa40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:44:33.257 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:33.255+0000 7fc935595640 1 -- 172.21.15.123:0/565004109 shutdown_connections 2024-09-16T15:44:33.257 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:33.255+0000 7fc935595640 1 -- 172.21.15.123:0/565004109 wait complete. 2024-09-16T15:44:33.259 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:33.255+0000 7fc935595640 1 Processor -- start 2024-09-16T15:44:33.259 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:33.255+0000 7fc935595640 1 -- start start 2024-09-16T15:44:33.259 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:33.255+0000 7fc935595640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fc9300b25a0 0x7fc9300ce4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:44:33.260 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:33.255+0000 7fc935595640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc9300d2320 0x7fc93015aa60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:44:33.260 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:33.255+0000 7fc935595640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc93015bda0 0x7fc9300cb700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:44:33.260 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:33.255+0000 7fc935595640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fc930168c00 con 0x7fc93015bda0 2024-09-16T15:44:33.260 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:33.255+0000 7fc935595640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fc930168a80 con 0x7fc9300d2320 2024-09-16T15:44:33.260 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:33.255+0000 7fc935595640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fc930168d80 con 0x7fc9300b25a0 2024-09-16T15:44:33.260 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:33.255+0000 7fc92f7fe640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc93015bda0 0x7fc9300cb700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:44:33.260 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:33.255+0000 7fc92f7fe640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc93015bda0 0x7fc9300cb700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:55010/0 (socket says 172.21.15.123:55010) 2024-09-16T15:44:33.260 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:33.255+0000 7fc92f7fe640 1 -- 172.21.15.123:0/2146610675 learned_addr learned my addr 172.21.15.123:0/2146610675 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:44:33.260 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:33.255+0000 7fc92effd640 1 --2- 172.21.15.123:0/2146610675 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fc9300b25a0 0x7fc9300ce4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:44:33.260 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:33.255+0000 7fc92e7fc640 1 --2- 172.21.15.123:0/2146610675 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc9300d2320 0x7fc93015aa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:44:33.260 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:33.255+0000 7fc92f7fe640 1 -- 172.21.15.123:0/2146610675 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fc9300b25a0 msgr2=0x7fc9300ce4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:44:33.260 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:33.255+0000 7fc92f7fe640 1 --2- 172.21.15.123:0/2146610675 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fc9300b25a0 0x7fc9300ce4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:33.260 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:33.255+0000 7fc92f7fe640 1 -- 172.21.15.123:0/2146610675 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc9300d2320 msgr2=0x7fc93015aa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:44:33.261 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:33.255+0000 7fc92f7fe640 1 --2- 172.21.15.123:0/2146610675 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc9300d2320 0x7fc93015aa60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:33.261 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:33.255+0000 7fc92f7fe640 1 -- 172.21.15.123:0/2146610675 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc930155590 con 0x7fc93015bda0 2024-09-16T15:44:33.261 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:33.255+0000 7fc92effd640 1 --2- 172.21.15.123:0/2146610675 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fc9300b25a0 0x7fc9300ce4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T15:44:33.261 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:33.255+0000 7fc92e7fc640 1 --2- 172.21.15.123:0/2146610675 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc9300d2320 0x7fc93015aa60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T15:44:33.261 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:33.259+0000 7fc92f7fe640 1 --2- 172.21.15.123:0/2146610675 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc93015bda0 0x7fc9300cb700 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7fc92800cce0 tx=0x7fc92800a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:44:33.261 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:33.259+0000 7fc9177fe640 1 -- 172.21.15.123:0/2146610675 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc928003980 con 0x7fc93015bda0 2024-09-16T15:44:33.261 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:33.259+0000 7fc9177fe640 1 -- 172.21.15.123:0/2146610675 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fc928003c10 con 0x7fc93015bda0 2024-09-16T15:44:33.261 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:33.259+0000 7fc935595640 1 -- 172.21.15.123:0/2146610675 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc9300c6000 con 0x7fc93015bda0 2024-09-16T15:44:33.261 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:33.259+0000 7fc9177fe640 1 -- 172.21.15.123:0/2146610675 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc928004030 con 0x7fc93015bda0 2024-09-16T15:44:33.261 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:33.259+0000 7fc935595640 1 -- 172.21.15.123:0/2146610675 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc9300c65c0 con 0x7fc93015bda0 2024-09-16T15:44:33.262 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:33.259+0000 7fc935595640 1 -- 172.21.15.123:0/2146610675 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc9300cca90 con 0x7fc93015bda0 2024-09-16T15:44:33.262 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:33.259+0000 7fc9177fe640 1 -- 172.21.15.123:0/2146610675 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 7) ==== 97233+0+0 (secure 0 0 0) 0x7fc9280041d0 con 0x7fc93015bda0 2024-09-16T15:44:33.262 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:33.259+0000 7fc9177fe640 1 --2- 172.21.15.123:0/2146610675 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fc91c073070 0x7fc91c075530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:44:33.262 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:33.259+0000 7fc9177fe640 1 -- 172.21.15.123:0/2146610675 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(46..46 src has 1..46) ==== 5767+0+0 (secure 0 0 0) 0x7fc928094a30 con 0x7fc93015bda0 2024-09-16T15:44:33.262 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:33.259+0000 7fc92effd640 1 --2- 172.21.15.123:0/2146610675 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fc91c073070 0x7fc91c075530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:44:33.262 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:33.259+0000 7fc92effd640 1 --2- 172.21.15.123:0/2146610675 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fc91c073070 0x7fc91c075530 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fc9240af000 tx=0x7fc92401b090 comp rx=0 tx=0).ready entity=mgr.4135 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:44:33.265 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:33.263+0000 7fc9177fe640 1 -- 172.21.15.123:0/2146610675 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7fc930155590 con 0x7fc93015bda0 2024-09-16T15:44:33.437 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:33.435+0000 7fc935595640 1 -- 172.21.15.123:0/2146610675 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7fc9300721f0 con 0x7fc93015bda0 2024-09-16T15:44:33.437 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:33.435+0000 7fc9177fe640 1 -- 172.21.15.123:0/2146610675 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v46) ==== 80+0+0 (secure 0 0 0) 0x7fc928014070 con 0x7fc93015bda0 2024-09-16T15:44:33.437 INFO:teuthology.orchestra.run.smithi123.stderr:listed 0 entries 2024-09-16T15:44:33.439 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:33.435+0000 7fc935595640 1 -- 172.21.15.123:0/2146610675 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fc91c073070 msgr2=0x7fc91c075530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:44:33.439 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:33.435+0000 7fc935595640 1 --2- 172.21.15.123:0/2146610675 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fc91c073070 0x7fc91c075530 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fc9240af000 tx=0x7fc92401b090 comp rx=0 tx=0).stop 2024-09-16T15:44:33.440 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:33.435+0000 7fc935595640 1 -- 172.21.15.123:0/2146610675 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc93015bda0 msgr2=0x7fc9300cb700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:44:33.440 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:33.435+0000 7fc935595640 1 --2- 172.21.15.123:0/2146610675 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc93015bda0 0x7fc9300cb700 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7fc92800cce0 tx=0x7fc92800a620 comp rx=0 tx=0).stop 2024-09-16T15:44:33.440 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:33.435+0000 7fc935595640 1 -- 172.21.15.123:0/2146610675 shutdown_connections 2024-09-16T15:44:33.440 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:33.435+0000 7fc935595640 1 --2- 172.21.15.123:0/2146610675 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fc91c073070 0x7fc91c075530 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:33.440 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:33.435+0000 7fc935595640 1 --2- 172.21.15.123:0/2146610675 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc93015bda0 0x7fc9300cb700 unknown :-1 s=CLOSED pgs=496 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:33.440 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:33.439+0000 7fc935595640 1 --2- 172.21.15.123:0/2146610675 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc9300d2320 0x7fc93015aa60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:33.440 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:33.439+0000 7fc935595640 1 --2- 172.21.15.123:0/2146610675 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fc9300b25a0 0x7fc9300ce4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:33.440 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:33.439+0000 7fc935595640 1 -- 172.21.15.123:0/2146610675 >> 172.21.15.123:0/2146610675 conn(0x7fc930074bd0 msgr2=0x7fc9300d1c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:44:33.440 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:33.439+0000 7fc935595640 1 -- 172.21.15.123:0/2146610675 shutdown_connections 2024-09-16T15:44:33.440 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:33.439+0000 7fc935595640 1 -- 172.21.15.123:0/2146610675 wait complete. 2024-09-16T15:44:33.455 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2024-09-16T15:44:33.455 DEBUG:teuthology.orchestra.run.smithi123:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-16T15:44:33.461 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-09-16T15:44:33.625 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:33.623+0000 7f8445ca4640 1 -- 172.21.15.123:0/969823235 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8428004b20 msgr2=0x7f8428004f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:44:33.625 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:33.623+0000 7f8445ca4640 1 --2- 172.21.15.123:0/969823235 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8428004b20 0x7f8428004f40 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7f8430009b60 tx=0x7f843001d2b0 comp rx=0 tx=0).stop 2024-09-16T15:44:33.625 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:33.623+0000 7f843ffff640 1 -- 172.21.15.123:0/969823235 reap_dead start 2024-09-16T15:44:33.625 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:33.623+0000 7f8445ca4640 1 -- 172.21.15.123:0/969823235 shutdown_connections 2024-09-16T15:44:33.625 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:33.623+0000 7f8445ca4640 1 -- 172.21.15.123:0/969823235 >> 172.21.15.123:0/969823235 conn(0x7f84400cfb10 msgr2=0x7f84400cff20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:44:33.625 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:33.623+0000 7f8445ca4640 1 -- 172.21.15.123:0/969823235 shutdown_connections 2024-09-16T15:44:33.625 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:33.623+0000 7f8445ca4640 1 -- 172.21.15.123:0/969823235 wait complete. 2024-09-16T15:44:33.626 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:33.623+0000 7f8445ca4640 1 Processor -- start 2024-09-16T15:44:33.626 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:33.623+0000 7f8445ca4640 1 -- start start 2024-09-16T15:44:33.628 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:33.623+0000 7f8445ca4640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f8428004b20 0x7f8440154d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:44:33.628 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:33.623+0000 7f8445ca4640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f8440073760 0x7f8440073be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:44:33.628 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:33.623+0000 7f8445ca4640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f84401588a0 0x7f8440158d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:44:33.628 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:33.623+0000 7f8445ca4640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f844015b1e0 con 0x7f84401588a0 2024-09-16T15:44:33.628 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:33.623+0000 7f8445ca4640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f844015b060 con 0x7f8440073760 2024-09-16T15:44:33.628 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:33.623+0000 7f8445ca4640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f844015b360 con 0x7f8428004b20 2024-09-16T15:44:33.628 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:33.623+0000 7f843ffff640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f84401588a0 0x7f8440158d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:44:33.629 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:33.623+0000 7f843ffff640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f84401588a0 0x7f8440158d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:55022/0 (socket says 172.21.15.123:55022) 2024-09-16T15:44:33.629 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:33.623+0000 7f843ffff640 1 -- 172.21.15.123:0/776123379 learned_addr learned my addr 172.21.15.123:0/776123379 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:44:33.629 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:33.623+0000 7f843f7fe640 1 --2- 172.21.15.123:0/776123379 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f8428004b20 0x7f8440154d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:44:33.629 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:33.623+0000 7f843effd640 1 --2- 172.21.15.123:0/776123379 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f8440073760 0x7f8440073be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:44:33.629 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:33.627+0000 7f843ffff640 1 -- 172.21.15.123:0/776123379 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f8428004b20 msgr2=0x7f8440154d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:44:33.629 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:33.627+0000 7f843ffff640 1 --2- 172.21.15.123:0/776123379 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f8428004b20 0x7f8440154d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:33.629 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:33.627+0000 7f843ffff640 1 -- 172.21.15.123:0/776123379 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f8440073760 msgr2=0x7f8440073be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:44:33.629 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:33.627+0000 7f843ffff640 1 --2- 172.21.15.123:0/776123379 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f8440073760 0x7f8440073be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:33.629 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:33.627+0000 7f843ffff640 1 -- 172.21.15.123:0/776123379 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8440155270 con 0x7f84401588a0 2024-09-16T15:44:33.629 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:33.627+0000 7f843ffff640 1 --2- 172.21.15.123:0/776123379 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f84401588a0 0x7f8440158d50 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7f8440069550 tx=0x7f843800ee10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:44:33.629 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:33.627+0000 7f8427fff640 1 -- 172.21.15.123:0/776123379 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8438004d30 con 0x7f84401588a0 2024-09-16T15:44:33.629 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:33.627+0000 7f8427fff640 1 -- 172.21.15.123:0/776123379 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f843800cad0 con 0x7f84401588a0 2024-09-16T15:44:33.630 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:33.627+0000 7f8445ca4640 1 -- 172.21.15.123:0/776123379 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f84401fbdf0 con 0x7f84401588a0 2024-09-16T15:44:33.630 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:33.627+0000 7f8427fff640 1 -- 172.21.15.123:0/776123379 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8438003980 con 0x7f84401588a0 2024-09-16T15:44:33.630 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:33.627+0000 7f8445ca4640 1 -- 172.21.15.123:0/776123379 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f84401fb310 con 0x7f84401588a0 2024-09-16T15:44:33.630 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:33.627+0000 7f8445ca4640 1 -- 172.21.15.123:0/776123379 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8440074120 con 0x7f84401588a0 2024-09-16T15:44:33.630 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:33.627+0000 7f8427fff640 1 -- 172.21.15.123:0/776123379 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 7) ==== 97233+0+0 (secure 0 0 0) 0x7f8438010430 con 0x7f84401588a0 2024-09-16T15:44:33.631 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:33.627+0000 7f8427fff640 1 --2- 172.21.15.123:0/776123379 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f842c073070 0x7f842c075530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:44:33.631 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:33.627+0000 7f8427fff640 1 -- 172.21.15.123:0/776123379 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(46..46 src has 1..46) ==== 5767+0+0 (secure 0 0 0) 0x7f84380947c0 con 0x7f84401588a0 2024-09-16T15:44:33.631 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:33.627+0000 7f843f7fe640 1 --2- 172.21.15.123:0/776123379 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f842c073070 0x7f842c075530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:44:33.631 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:33.627+0000 7f843f7fe640 1 --2- 172.21.15.123:0/776123379 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f842c073070 0x7f842c075530 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f84300ae000 tx=0x7f843001a050 comp rx=0 tx=0).ready entity=mgr.4135 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:44:33.634 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:33.631+0000 7f8427fff640 1 -- 172.21.15.123:0/776123379 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f843805f6d0 con 0x7f84401588a0 2024-09-16T15:44:33.805 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:33.803+0000 7f8445ca4640 1 -- 172.21.15.123:0/776123379 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f84400ca320 con 0x7f84401588a0 2024-09-16T15:44:33.805 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:33.803+0000 7f8427fff640 1 -- 172.21.15.123:0/776123379 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v46) ==== 80+0+0 (secure 0 0 0) 0x7f843805efd0 con 0x7f84401588a0 2024-09-16T15:44:33.805 INFO:teuthology.orchestra.run.smithi123.stderr:listed 0 entries 2024-09-16T15:44:33.808 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:33.807+0000 7f8445ca4640 1 -- 172.21.15.123:0/776123379 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f842c073070 msgr2=0x7f842c075530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:44:33.808 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:33.807+0000 7f8445ca4640 1 --2- 172.21.15.123:0/776123379 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f842c073070 0x7f842c075530 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f84300ae000 tx=0x7f843001a050 comp rx=0 tx=0).stop 2024-09-16T15:44:33.808 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:33.807+0000 7f8445ca4640 1 -- 172.21.15.123:0/776123379 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f84401588a0 msgr2=0x7f8440158d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:44:33.808 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:33.807+0000 7f8445ca4640 1 --2- 172.21.15.123:0/776123379 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f84401588a0 0x7f8440158d50 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7f8440069550 tx=0x7f843800ee10 comp rx=0 tx=0).stop 2024-09-16T15:44:33.808 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:33.807+0000 7f8445ca4640 1 -- 172.21.15.123:0/776123379 shutdown_connections 2024-09-16T15:44:33.808 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:33.807+0000 7f8445ca4640 1 --2- 172.21.15.123:0/776123379 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f842c073070 0x7f842c075530 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:33.808 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:33.807+0000 7f8445ca4640 1 --2- 172.21.15.123:0/776123379 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f84401588a0 0x7f8440158d50 unknown :-1 s=CLOSED pgs=499 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:33.809 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:33.807+0000 7f8445ca4640 1 --2- 172.21.15.123:0/776123379 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f8440073760 0x7f8440073be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:33.809 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:33.807+0000 7f8445ca4640 1 --2- 172.21.15.123:0/776123379 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f8428004b20 0x7f8440154d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:33.809 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:33.807+0000 7f8445ca4640 1 -- 172.21.15.123:0/776123379 >> 172.21.15.123:0/776123379 conn(0x7f84400cfb10 msgr2=0x7f84400ae960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:44:33.809 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:33.807+0000 7f8445ca4640 1 -- 172.21.15.123:0/776123379 shutdown_connections 2024-09-16T15:44:33.809 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:33.807+0000 7f8445ca4640 1 -- 172.21.15.123:0/776123379 wait complete. 2024-09-16T15:44:33.824 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi123.front.sepia.ceph.com... 2024-09-16T15:44:33.824 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:44:33.824 DEBUG:teuthology.orchestra.run.smithi123:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2024-09-16T15:44:33.834 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi123.stderr:ceph-fuse[39423]: fuse finished with error 0 and tester_r 0 2024-09-16T15:44:33.836 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T15:44:38.810 DEBUG:teuthology.orchestra.run.smithi123:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-16T15:44:38.813 DEBUG:teuthology.orchestra.run.smithi184:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-16T15:44:39.937 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi123.front.sepia.ceph.com 2024-09-16T15:44:39.938 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:44:39.938 DEBUG:teuthology.orchestra.run.smithi123:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-09-16T15:44:39.945 DEBUG:teuthology.orchestra.run.smithi184:> set -ex 2024-09-16T15:44:39.946 DEBUG:teuthology.orchestra.run.smithi184:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-16T15:44:39.952 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-09-16T15:44:40.126 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:40.123+0000 7f87956d7640 1 -- 172.21.15.123:0/1085471584 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f879015bca0 msgr2=0x7f87900d0630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:44:40.126 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:40.123+0000 7f87956d7640 1 --2- 172.21.15.123:0/1085471584 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f879015bca0 0x7f87900d0630 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7f879006df40 tx=0x7f877801c800 comp rx=0 tx=0).stop 2024-09-16T15:44:40.126 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:40.123+0000 7f87956d7640 1 -- 172.21.15.123:0/1085471584 shutdown_connections 2024-09-16T15:44:40.126 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:40.123+0000 7f87956d7640 1 --2- 172.21.15.123:0/1085471584 >> v2:172.21.15.184:3300/0 conn(0x7f87900726c0 0x7f8790073ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:40.127 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:40.123+0000 7f87956d7640 1 --2- 172.21.15.123:0/1085471584 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f879015bca0 0x7f87900d0630 unknown :-1 s=CLOSED pgs=500 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:40.127 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:40.123+0000 7f87956d7640 1 -- 172.21.15.123:0/1085471584 >> 172.21.15.123:0/1085471584 conn(0x7f87900cfb10 msgr2=0x7f87900cff20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:44:40.127 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:40.123+0000 7f87956d7640 1 -- 172.21.15.123:0/1085471584 shutdown_connections 2024-09-16T15:44:40.127 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:40.123+0000 7f87956d7640 1 -- 172.21.15.123:0/1085471584 wait complete. 2024-09-16T15:44:40.127 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:40.127+0000 7f87956d7640 1 Processor -- start 2024-09-16T15:44:40.127 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:40.127+0000 7f87956d7640 1 -- start start 2024-09-16T15:44:40.129 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:40.127+0000 7f87956d7640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f87900726c0 0x7f8790164e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:44:40.129 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:40.127+0000 7f87956d7640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f8790074430 0x7f879015ed60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:44:40.129 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:40.127+0000 7f87956d7640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f879015bca0 0x7f87900cef50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:44:40.129 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:40.127+0000 7f87956d7640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f8790168e10 con 0x7f87900726c0 2024-09-16T15:44:40.129 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:40.127+0000 7f87956d7640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f8790168c90 con 0x7f879015bca0 2024-09-16T15:44:40.130 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:40.127+0000 7f87956d7640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f8790168f90 con 0x7f8790074430 2024-09-16T15:44:40.130 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:40.127+0000 7f878effd640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f87900726c0 0x7f8790164e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:44:40.130 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:40.127+0000 7f878effd640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f87900726c0 0x7f8790164e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:55032/0 (socket says 172.21.15.123:55032) 2024-09-16T15:44:40.130 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:40.127+0000 7f878effd640 1 -- 172.21.15.123:0/2445992229 learned_addr learned my addr 172.21.15.123:0/2445992229 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:44:40.130 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:40.127+0000 7f878effd640 1 -- 172.21.15.123:0/2445992229 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f8790074430 msgr2=0x7f879015ed60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:44:40.130 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:40.127+0000 7f878effd640 1 --2- 172.21.15.123:0/2445992229 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f8790074430 0x7f879015ed60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:40.130 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:40.127+0000 7f878f7fe640 1 --2- 172.21.15.123:0/2445992229 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f879015bca0 0x7f87900cef50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:44:40.130 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:40.127+0000 7f878effd640 1 -- 172.21.15.123:0/2445992229 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f879015bca0 msgr2=0x7f87900cef50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:44:40.130 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:40.127+0000 7f878effd640 1 --2- 172.21.15.123:0/2445992229 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f879015bca0 0x7f87900cef50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:40.130 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:40.127+0000 7f878effd640 1 -- 172.21.15.123:0/2445992229 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8790165340 con 0x7f87900726c0 2024-09-16T15:44:40.130 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:40.127+0000 7f878effd640 1 --2- 172.21.15.123:0/2445992229 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f87900726c0 0x7f8790164e00 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7f87780b0040 tx=0x7f87780a5790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:44:40.130 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:40.127+0000 7f87877fe640 1 -- 172.21.15.123:0/2445992229 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87780bc020 con 0x7f87900726c0 2024-09-16T15:44:40.130 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:40.127+0000 7f87877fe640 1 -- 172.21.15.123:0/2445992229 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f8778004430 con 0x7f87900726c0 2024-09-16T15:44:40.131 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:40.127+0000 7f87956d7640 1 -- 172.21.15.123:0/2445992229 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f879015c4e0 con 0x7f87900726c0 2024-09-16T15:44:40.131 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:40.127+0000 7f87877fe640 1 -- 172.21.15.123:0/2445992229 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87780b2040 con 0x7f87900726c0 2024-09-16T15:44:40.131 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:40.127+0000 7f87956d7640 1 -- 172.21.15.123:0/2445992229 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f879015caa0 con 0x7f87900726c0 2024-09-16T15:44:40.131 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:40.127+0000 7f87956d7640 1 -- 172.21.15.123:0/2445992229 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8790073ef0 con 0x7f87900726c0 2024-09-16T15:44:40.132 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:40.127+0000 7f87877fe640 1 -- 172.21.15.123:0/2445992229 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 7) ==== 97233+0+0 (secure 0 0 0) 0x7f87780bf070 con 0x7f87900726c0 2024-09-16T15:44:40.132 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:40.131+0000 7f87877fe640 1 --2- 172.21.15.123:0/2445992229 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f87700730a0 0x7f8770075560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:44:40.132 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:40.131+0000 7f87877fe640 1 -- 172.21.15.123:0/2445992229 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(46..46 src has 1..46) ==== 5767+0+0 (secure 0 0 0) 0x7f877812e1e0 con 0x7f87900726c0 2024-09-16T15:44:40.132 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:40.131+0000 7f8787fff640 1 --2- 172.21.15.123:0/2445992229 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f87700730a0 0x7f8770075560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:44:40.132 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:40.131+0000 7f8787fff640 1 --2- 172.21.15.123:0/2445992229 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f87700730a0 0x7f8770075560 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f87900cec60 tx=0x7f8780005c80 comp rx=0 tx=0).ready entity=mgr.4135 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:44:40.135 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:40.131+0000 7f87877fe640 1 -- 172.21.15.123:0/2445992229 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f87780164b0 con 0x7f87900726c0 2024-09-16T15:44:40.320 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:40.319+0000 7f87956d7640 1 -- 172.21.15.123:0/2445992229 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f879015f630 con 0x7f87900726c0 2024-09-16T15:44:40.320 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:40.319+0000 7f87877fe640 1 -- 172.21.15.123:0/2445992229 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v46) ==== 80+0+0 (secure 0 0 0) 0x7f87780f91d0 con 0x7f87900726c0 2024-09-16T15:44:40.321 INFO:teuthology.orchestra.run.smithi123.stderr:listed 0 entries 2024-09-16T15:44:40.323 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:40.319+0000 7f87956d7640 1 -- 172.21.15.123:0/2445992229 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f87700730a0 msgr2=0x7f8770075560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:44:40.324 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:40.319+0000 7f87956d7640 1 --2- 172.21.15.123:0/2445992229 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f87700730a0 0x7f8770075560 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f87900cec60 tx=0x7f8780005c80 comp rx=0 tx=0).stop 2024-09-16T15:44:40.324 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:40.319+0000 7f87956d7640 1 -- 172.21.15.123:0/2445992229 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f87900726c0 msgr2=0x7f8790164e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:44:40.324 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:40.319+0000 7f87956d7640 1 --2- 172.21.15.123:0/2445992229 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f87900726c0 0x7f8790164e00 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7f87780b0040 tx=0x7f87780a5790 comp rx=0 tx=0).stop 2024-09-16T15:44:40.324 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:40.319+0000 7f87956d7640 1 -- 172.21.15.123:0/2445992229 shutdown_connections 2024-09-16T15:44:40.324 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:40.319+0000 7f87956d7640 1 --2- 172.21.15.123:0/2445992229 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f87700730a0 0x7f8770075560 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:40.324 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:40.319+0000 7f87956d7640 1 --2- 172.21.15.123:0/2445992229 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f879015bca0 0x7f87900cef50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:40.324 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:40.319+0000 7f87956d7640 1 --2- 172.21.15.123:0/2445992229 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f8790074430 0x7f879015ed60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:40.324 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:40.319+0000 7f87956d7640 1 --2- 172.21.15.123:0/2445992229 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f87900726c0 0x7f8790164e00 unknown :-1 s=CLOSED pgs=501 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:40.324 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:40.319+0000 7f87956d7640 1 -- 172.21.15.123:0/2445992229 >> 172.21.15.123:0/2445992229 conn(0x7f87900cfb10 msgr2=0x7f87900ca140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:44:40.324 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:40.323+0000 7f87956d7640 1 -- 172.21.15.123:0/2445992229 shutdown_connections 2024-09-16T15:44:40.324 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:40.323+0000 7f87956d7640 1 -- 172.21.15.123:0/2445992229 wait complete. 2024-09-16T15:44:40.340 DEBUG:teuthology.orchestra.run.smithi184:> set -ex 2024-09-16T15:44:40.341 DEBUG:teuthology.orchestra.run.smithi184:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-16T15:44:40.347 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-09-16T15:44:40.477 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:40.475+0000 7f5b82b49640 1 -- 172.21.15.123:0/3887336863 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5b7c15e160 msgr2=0x7f5b7c073f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:44:40.477 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:40.475+0000 7f5b82b49640 1 --2- 172.21.15.123:0/3887336863 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5b7c15e160 0x7f5b7c073f30 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7f5b7c0cf1f0 tx=0x7f5b6c01c6e0 comp rx=0 tx=0).stop 2024-09-16T15:44:40.478 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:40.475+0000 7f5b82b49640 1 -- 172.21.15.123:0/3887336863 shutdown_connections 2024-09-16T15:44:40.478 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:40.475+0000 7f5b82b49640 1 --2- 172.21.15.123:0/3887336863 >> v2:172.21.15.184:3300/0 conn(0x7f5b7c074470 0x7f5b7c0cd6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:40.478 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:40.475+0000 7f5b82b49640 1 --2- 172.21.15.123:0/3887336863 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5b7c15e160 0x7f5b7c073f30 unknown :-1 s=CLOSED pgs=503 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:40.478 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:40.475+0000 7f5b82b49640 1 -- 172.21.15.123:0/3887336863 >> 172.21.15.123:0/3887336863 conn(0x7f5b7c0cfb10 msgr2=0x7f5b7c0cff20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:44:40.478 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:40.475+0000 7f5b82b49640 1 -- 172.21.15.123:0/3887336863 shutdown_connections 2024-09-16T15:44:40.478 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:40.475+0000 7f5b82b49640 1 -- 172.21.15.123:0/3887336863 wait complete. 2024-09-16T15:44:40.478 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:40.475+0000 7f5b82b49640 1 Processor -- start 2024-09-16T15:44:40.478 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:40.475+0000 7f5b82b49640 1 -- start start 2024-09-16T15:44:40.480 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:40.479+0000 7f5b82b49640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f5b7c074470 0x7f5b7c14f910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:44:40.480 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:40.479+0000 7f5b82b49640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5b7c0cdc20 0x7f5b7c0c6be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:44:40.480 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:40.479+0000 7f5b82b49640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5b7c15e160 0x7f5b7c1508f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:44:40.480 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:40.479+0000 7f5b82b49640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f5b7c167dd0 con 0x7f5b7c0cdc20 2024-09-16T15:44:40.480 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:40.479+0000 7f5b82b49640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f5b7c0cf620 con 0x7f5b7c15e160 2024-09-16T15:44:40.480 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:40.479+0000 7f5b82b49640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f5b7c167f50 con 0x7f5b7c074470 2024-09-16T15:44:40.480 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:40.479+0000 7f5b7bfff640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5b7c0cdc20 0x7f5b7c0c6be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:44:40.481 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:40.479+0000 7f5b7bfff640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5b7c0cdc20 0x7f5b7c0c6be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:55060/0 (socket says 172.21.15.123:55060) 2024-09-16T15:44:40.481 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:40.479+0000 7f5b7bfff640 1 -- 172.21.15.123:0/1884845584 learned_addr learned my addr 172.21.15.123:0/1884845584 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:44:40.481 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:40.479+0000 7f5b810e6640 1 --2- 172.21.15.123:0/1884845584 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5b7c15e160 0x7f5b7c1508f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:44:40.481 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:40.479+0000 7f5b7bfff640 1 -- 172.21.15.123:0/1884845584 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f5b7c074470 msgr2=0x7f5b7c14f910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:44:40.481 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:40.479+0000 7f5b7bfff640 1 --2- 172.21.15.123:0/1884845584 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f5b7c074470 0x7f5b7c14f910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:40.481 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:40.479+0000 7f5b7bfff640 1 -- 172.21.15.123:0/1884845584 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5b7c15e160 msgr2=0x7f5b7c1508f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:44:40.481 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:40.479+0000 7f5b7bfff640 1 --2- 172.21.15.123:0/1884845584 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5b7c15e160 0x7f5b7c1508f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:40.481 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:40.479+0000 7f5b7bfff640 1 -- 172.21.15.123:0/1884845584 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5b7c1511d0 con 0x7f5b7c0cdc20 2024-09-16T15:44:40.481 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:40.479+0000 7f5b7bfff640 1 --2- 172.21.15.123:0/1884845584 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5b7c0cdc20 0x7f5b7c0c6be0 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7f5b7000c9a0 tx=0x7f5b7000ce60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:44:40.481 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:40.479+0000 7f5b797fa640 1 -- 172.21.15.123:0/1884845584 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b70007c70 con 0x7f5b7c0cdc20 2024-09-16T15:44:40.481 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:40.479+0000 7f5b797fa640 1 -- 172.21.15.123:0/1884845584 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f5b7000f450 con 0x7f5b7c0cdc20 2024-09-16T15:44:40.481 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:40.479+0000 7f5b82b49640 1 -- 172.21.15.123:0/1884845584 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5b7c156270 con 0x7f5b7c0cdc20 2024-09-16T15:44:40.481 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:40.479+0000 7f5b797fa640 1 -- 172.21.15.123:0/1884845584 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b7000f8a0 con 0x7f5b7c0cdc20 2024-09-16T15:44:40.482 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:40.479+0000 7f5b82b49640 1 -- 172.21.15.123:0/1884845584 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5b7c156830 con 0x7f5b7c0cdc20 2024-09-16T15:44:40.482 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:40.479+0000 7f5b82b49640 1 -- 172.21.15.123:0/1884845584 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5b7c0cd6e0 con 0x7f5b7c0cdc20 2024-09-16T15:44:40.482 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:40.479+0000 7f5b797fa640 1 -- 172.21.15.123:0/1884845584 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 7) ==== 97233+0+0 (secure 0 0 0) 0x7f5b7000fa40 con 0x7f5b7c0cdc20 2024-09-16T15:44:40.482 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:40.479+0000 7f5b797fa640 1 --2- 172.21.15.123:0/1884845584 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f5b68073070 0x7f5b68075530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:44:40.482 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:40.479+0000 7f5b797fa640 1 -- 172.21.15.123:0/1884845584 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(46..46 src has 1..46) ==== 5767+0+0 (secure 0 0 0) 0x7f5b70094b20 con 0x7f5b7c0cdc20 2024-09-16T15:44:40.482 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:40.479+0000 7f5b808e5640 1 --2- 172.21.15.123:0/1884845584 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f5b68073070 0x7f5b68075530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:44:40.483 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:40.483+0000 7f5b808e5640 1 --2- 172.21.15.123:0/1884845584 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f5b68073070 0x7f5b68075530 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f5b6c0af000 tx=0x7f5b6c019480 comp rx=0 tx=0).ready entity=mgr.4135 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:44:40.486 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:40.483+0000 7f5b797fa640 1 -- 172.21.15.123:0/1884845584 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f5b70014030 con 0x7f5b7c0cdc20 2024-09-16T15:44:40.663 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:40.663+0000 7f5b82b49640 1 -- 172.21.15.123:0/1884845584 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f5b7c0ccb60 con 0x7f5b7c0cdc20 2024-09-16T15:44:40.664 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:40.663+0000 7f5b797fa640 1 -- 172.21.15.123:0/1884845584 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v46) ==== 80+0+0 (secure 0 0 0) 0x7f5b7005f920 con 0x7f5b7c0cdc20 2024-09-16T15:44:40.664 INFO:teuthology.orchestra.run.smithi123.stderr:listed 0 entries 2024-09-16T15:44:40.667 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:40.663+0000 7f5b82b49640 1 -- 172.21.15.123:0/1884845584 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f5b68073070 msgr2=0x7f5b68075530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:44:40.667 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:40.663+0000 7f5b82b49640 1 --2- 172.21.15.123:0/1884845584 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f5b68073070 0x7f5b68075530 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f5b6c0af000 tx=0x7f5b6c019480 comp rx=0 tx=0).stop 2024-09-16T15:44:40.667 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:40.663+0000 7f5b82b49640 1 -- 172.21.15.123:0/1884845584 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5b7c0cdc20 msgr2=0x7f5b7c0c6be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:44:40.667 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:40.663+0000 7f5b82b49640 1 --2- 172.21.15.123:0/1884845584 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5b7c0cdc20 0x7f5b7c0c6be0 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7f5b7000c9a0 tx=0x7f5b7000ce60 comp rx=0 tx=0).stop 2024-09-16T15:44:40.667 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:40.663+0000 7f5b82b49640 1 -- 172.21.15.123:0/1884845584 shutdown_connections 2024-09-16T15:44:40.667 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:40.663+0000 7f5b82b49640 1 --2- 172.21.15.123:0/1884845584 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f5b68073070 0x7f5b68075530 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:40.667 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:40.663+0000 7f5b82b49640 1 --2- 172.21.15.123:0/1884845584 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5b7c15e160 0x7f5b7c1508f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:40.667 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:40.663+0000 7f5b82b49640 1 --2- 172.21.15.123:0/1884845584 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5b7c0cdc20 0x7f5b7c0c6be0 unknown :-1 s=CLOSED pgs=504 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:40.667 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:40.663+0000 7f5b82b49640 1 --2- 172.21.15.123:0/1884845584 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f5b7c074470 0x7f5b7c14f910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:40.667 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:40.663+0000 7f5b82b49640 1 -- 172.21.15.123:0/1884845584 >> 172.21.15.123:0/1884845584 conn(0x7f5b7c0cfb10 msgr2=0x7f5b7c075ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:44:40.667 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:40.663+0000 7f5b82b49640 1 -- 172.21.15.123:0/1884845584 shutdown_connections 2024-09-16T15:44:40.667 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:40.663+0000 7f5b82b49640 1 -- 172.21.15.123:0/1884845584 wait complete. 2024-09-16T15:44:40.682 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi184.front.sepia.ceph.com... 2024-09-16T15:44:40.682 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:44:40.683 DEBUG:teuthology.orchestra.run.smithi184:> sudo fusermount -u /home/ubuntu/cephtest/mnt.1 2024-09-16T15:44:40.696 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi184.stderr:ceph-fuse[34843]: fuse finished with error 0 and tester_r 0 2024-09-16T15:44:40.697 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T15:44:40.765 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi184.front.sepia.ceph.com 2024-09-16T15:44:40.765 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:44:40.766 DEBUG:teuthology.orchestra.run.smithi184:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2024-09-16T15:44:40.772 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-16T15:44:40.902 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:40.899+0000 7f31782eb640 1 -- 172.21.15.123:0/708682449 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f317015e820 msgr2=0x7f317015ec90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:44:40.902 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:40.899+0000 7f31782eb640 1 --2- 172.21.15.123:0/708682449 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f317015e820 0x7f317015ec90 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7f31700cf1f0 tx=0x7f316001c770 comp rx=0 tx=0).stop 2024-09-16T15:44:40.902 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:40.899+0000 7f31782eb640 1 -- 172.21.15.123:0/708682449 shutdown_connections 2024-09-16T15:44:40.902 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:40.899+0000 7f31782eb640 1 --2- 172.21.15.123:0/708682449 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f317015e820 0x7f317015ec90 unknown :-1 s=CLOSED pgs=506 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:40.902 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:40.899+0000 7f31782eb640 1 --2- 172.21.15.123:0/708682449 >> v2:172.21.15.184:3300/0 conn(0x7f31700cd6e0 0x7f31700cdb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:40.902 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:40.899+0000 7f31782eb640 1 -- 172.21.15.123:0/708682449 >> 172.21.15.123:0/708682449 conn(0x7f31700cfb10 msgr2=0x7f31700cff20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:44:40.902 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:40.899+0000 7f31782eb640 1 -- 172.21.15.123:0/708682449 shutdown_connections 2024-09-16T15:44:40.902 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:40.899+0000 7f31782eb640 1 -- 172.21.15.123:0/708682449 wait complete. 2024-09-16T15:44:40.903 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:40.899+0000 7f31782eb640 1 Processor -- start 2024-09-16T15:44:40.903 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:40.899+0000 7f31782eb640 1 -- start start 2024-09-16T15:44:40.904 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:40.903+0000 7f31782eb640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f31700cd6e0 0x7f3170150220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:44:40.905 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:40.903+0000 7f31782eb640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f317015e160 0x7f31700b4b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:44:40.905 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:40.903+0000 7f31782eb640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f317015e820 0x7f31700b2af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:44:40.905 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:40.903+0000 7f31782eb640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f31700cf5b0 con 0x7f31700cd6e0 2024-09-16T15:44:40.905 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:40.903+0000 7f31782eb640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f31700cf430 con 0x7f317015e160 2024-09-16T15:44:40.905 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:40.903+0000 7f31782eb640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f31701683d0 con 0x7f317015e820 2024-09-16T15:44:40.905 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:40.903+0000 7f3176087640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f31700cd6e0 0x7f3170150220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:44:40.905 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:40.903+0000 7f3176087640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f31700cd6e0 0x7f3170150220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:55078/0 (socket says 172.21.15.123:55078) 2024-09-16T15:44:40.905 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:40.903+0000 7f3176087640 1 -- 172.21.15.123:0/2619000476 learned_addr learned my addr 172.21.15.123:0/2619000476 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:44:40.905 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:40.903+0000 7f3176888640 1 --2- 172.21.15.123:0/2619000476 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f317015e820 0x7f31700b2af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:44:40.905 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:40.903+0000 7f3175886640 1 --2- 172.21.15.123:0/2619000476 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f317015e160 0x7f31700b4b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:44:40.905 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:40.903+0000 7f3176087640 1 -- 172.21.15.123:0/2619000476 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f317015e820 msgr2=0x7f31700b2af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:44:40.905 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:40.903+0000 7f3176087640 1 --2- 172.21.15.123:0/2619000476 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f317015e820 0x7f31700b2af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:40.905 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:40.903+0000 7f3176087640 1 -- 172.21.15.123:0/2619000476 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f317015e160 msgr2=0x7f31700b4b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:44:40.906 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:40.903+0000 7f3176087640 1 --2- 172.21.15.123:0/2619000476 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f317015e160 0x7f31700b4b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:40.906 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:40.903+0000 7f3176087640 1 -- 172.21.15.123:0/2619000476 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f31700b5060 con 0x7f31700cd6e0 2024-09-16T15:44:40.906 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:40.903+0000 7f3176087640 1 --2- 172.21.15.123:0/2619000476 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f31700cd6e0 0x7f3170150220 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7f316800cba0 tx=0x7f316800cc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:44:40.906 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:40.903+0000 7f316effd640 1 -- 172.21.15.123:0/2619000476 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3168016020 con 0x7f31700cd6e0 2024-09-16T15:44:40.906 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:40.903+0000 7f316effd640 1 -- 172.21.15.123:0/2619000476 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f316800ab40 con 0x7f31700cd6e0 2024-09-16T15:44:40.906 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:40.903+0000 7f31782eb640 1 -- 172.21.15.123:0/2619000476 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f31700b58c0 con 0x7f31700cd6e0 2024-09-16T15:44:40.906 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:40.903+0000 7f316effd640 1 -- 172.21.15.123:0/2619000476 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3168012420 con 0x7f31700cd6e0 2024-09-16T15:44:40.906 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:40.903+0000 7f31782eb640 1 -- 172.21.15.123:0/2619000476 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f31700b55e0 con 0x7f31700cd6e0 2024-09-16T15:44:40.907 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:40.903+0000 7f31782eb640 1 -- 172.21.15.123:0/2619000476 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f31700cdb50 con 0x7f31700cd6e0 2024-09-16T15:44:40.907 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:40.903+0000 7f316effd640 1 -- 172.21.15.123:0/2619000476 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 7) ==== 97233+0+0 (secure 0 0 0) 0x7f3168019020 con 0x7f31700cd6e0 2024-09-16T15:44:40.907 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:40.903+0000 7f316effd640 1 --2- 172.21.15.123:0/2619000476 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f315c073070 0x7f315c075530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:44:40.907 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:40.903+0000 7f316effd640 1 -- 172.21.15.123:0/2619000476 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(46..46 src has 1..46) ==== 5767+0+0 (secure 0 0 0) 0x7f3168094930 con 0x7f31700cd6e0 2024-09-16T15:44:40.907 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:40.903+0000 7f3175886640 1 --2- 172.21.15.123:0/2619000476 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f315c073070 0x7f315c075530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:44:40.907 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:40.907+0000 7f3175886640 1 --2- 172.21.15.123:0/2619000476 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f315c073070 0x7f315c075530 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f31701f8c50 tx=0x7f3164008040 comp rx=0 tx=0).ready entity=mgr.4135 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:44:40.910 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:40.907+0000 7f316effd640 1 -- 172.21.15.123:0/2619000476 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f3168020020 con 0x7f31700cd6e0 2024-09-16T15:44:41.119 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:41.115+0000 7f31782eb640 1 -- 172.21.15.123:0/2619000476 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f31700d06c0 con 0x7f31700cd6e0 2024-09-16T15:44:41.120 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:41.119+0000 7f316effd640 1 -- 172.21.15.123:0/2619000476 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) ==== 94+0+5223 (secure 0 0 0) 0x7f316805f890 con 0x7f31700cd6e0 2024-09-16T15:44:41.120 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-09-16T15:44:41.120 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":28,"btime":"2024-09-16T15:44:32:363512+0000","default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4712,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6837/288264117","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6836","nonce":288264117},{"type":"v1","addr":"172.21.15.184:6837","nonce":288264117}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":23},{"gid":5268,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6834/60194515","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6832","nonce":60194515},{"type":"v1","addr":"172.21.15.123:6834","nonce":60194515}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":23},{"gid":5271,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6835/1798373479","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6833","nonce":1798373479},{"type":"v1","addr":"172.21.15.123:6835","nonce":1798373479}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":22}],"filesystems":[{"mdsmap":{"epoch":27,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-16T15:44:14.853230+0000","modified":"2024-09-16T15:44:17.861742+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5280},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5280":{"gid":5280,"name":"d","rank":0,"incarnation":24,"state":"up:active","state_seq":3,"addr":"172.21.15.184:6835/374389412","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":374389412},{"type":"v1","addr":"172.21.15.184:6835","nonce":374389412}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5280,"qdb_cluster":[5280]},"id":3}]} 2024-09-16T15:44:41.121 INFO:teuthology.orchestra.run.smithi123.stderr:dumped fsmap epoch 28 2024-09-16T15:44:41.123 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:41.119+0000 7f31782eb640 1 -- 172.21.15.123:0/2619000476 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f315c073070 msgr2=0x7f315c075530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:44:41.123 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:41.119+0000 7f31782eb640 1 --2- 172.21.15.123:0/2619000476 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f315c073070 0x7f315c075530 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f31701f8c50 tx=0x7f3164008040 comp rx=0 tx=0).stop 2024-09-16T15:44:41.123 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:41.119+0000 7f31782eb640 1 -- 172.21.15.123:0/2619000476 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f31700cd6e0 msgr2=0x7f3170150220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:44:41.123 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:41.119+0000 7f31782eb640 1 --2- 172.21.15.123:0/2619000476 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f31700cd6e0 0x7f3170150220 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7f316800cba0 tx=0x7f316800cc90 comp rx=0 tx=0).stop 2024-09-16T15:44:41.123 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:41.119+0000 7f31782eb640 1 -- 172.21.15.123:0/2619000476 shutdown_connections 2024-09-16T15:44:41.123 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:41.119+0000 7f31782eb640 1 --2- 172.21.15.123:0/2619000476 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f315c073070 0x7f315c075530 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:41.124 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:41.119+0000 7f31782eb640 1 --2- 172.21.15.123:0/2619000476 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f317015e820 0x7f31700b2af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:41.124 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:41.119+0000 7f31782eb640 1 --2- 172.21.15.123:0/2619000476 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f317015e160 0x7f31700b4b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:41.124 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:41.119+0000 7f31782eb640 1 --2- 172.21.15.123:0/2619000476 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f31700cd6e0 0x7f3170150220 unknown :-1 s=CLOSED pgs=507 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:41.124 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:41.119+0000 7f31782eb640 1 -- 172.21.15.123:0/2619000476 >> 172.21.15.123:0/2619000476 conn(0x7f31700cfb10 msgr2=0x7f31700b0160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:44:41.124 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:41.119+0000 7f31782eb640 1 -- 172.21.15.123:0/2619000476 shutdown_connections 2024-09-16T15:44:41.124 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:41.119+0000 7f31782eb640 1 -- 172.21.15.123:0/2619000476 wait complete. 2024-09-16T15:44:41.140 DEBUG:teuthology.orchestra.run.smithi184:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.d.asok --format=json session ls 2024-09-16T15:44:41.290 INFO:teuthology.orchestra.run.smithi184.stdout:[] 2024-09-16T15:44:41.291 DEBUG:tasks.cephfs.filesystem:_json_asok output [] 2024-09-16T15:44:41.291 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume create cephfs subvol56410 2024-09-16T15:44:41.428 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:41.427+0000 7fef523f1640 1 -- 172.21.15.123:0/641315694 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fef4c15ddc0 msgr2=0x7fef4c15e270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:44:41.429 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:41.427+0000 7fef523f1640 1 --2- 172.21.15.123:0/641315694 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fef4c15ddc0 0x7fef4c15e270 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7fef44009bc0 tx=0x7fef4401cb10 comp rx=0 tx=0).stop 2024-09-16T15:44:41.429 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:41.427+0000 7fef523f1640 1 -- 172.21.15.123:0/641315694 shutdown_connections 2024-09-16T15:44:41.429 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:41.427+0000 7fef523f1640 1 --2- 172.21.15.123:0/641315694 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fef4c15ddc0 0x7fef4c15e270 unknown :-1 s=CLOSED pgs=509 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:41.429 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:41.427+0000 7fef523f1640 1 --2- 172.21.15.123:0/641315694 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fef4c15a170 0x7fef4c072f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:41.429 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:41.427+0000 7fef523f1640 1 -- 172.21.15.123:0/641315694 >> 172.21.15.123:0/641315694 conn(0x7fef4c0b1710 msgr2=0x7fef4c0b1b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:44:41.429 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:41.427+0000 7fef523f1640 1 -- 172.21.15.123:0/641315694 shutdown_connections 2024-09-16T15:44:41.429 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:41.427+0000 7fef523f1640 1 -- 172.21.15.123:0/641315694 wait complete. 2024-09-16T15:44:41.430 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:41.427+0000 7fef523f1640 1 Processor -- start 2024-09-16T15:44:41.430 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:41.427+0000 7fef523f1640 1 -- start start 2024-09-16T15:44:41.432 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:41.427+0000 7fef523f1640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fef4c0ca650 0x7fef4c154450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:44:41.432 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:41.427+0000 7fef523f1640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fef4c15a170 0x7fef4c0b3420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:44:41.432 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:41.427+0000 7fef523f1640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fef4c15ddc0 0x7fef4c151b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:44:41.432 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:41.427+0000 7fef523f1640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fef4c0cc740 con 0x7fef4c15a170 2024-09-16T15:44:41.432 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:41.427+0000 7fef523f1640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fef4c0cc5c0 con 0x7fef4c15ddc0 2024-09-16T15:44:41.432 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:41.427+0000 7fef523f1640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fef4c0cc8c0 con 0x7fef4c0ca650 2024-09-16T15:44:41.432 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:41.427+0000 7fef4b7fe640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fef4c15a170 0x7fef4c0b3420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:44:41.432 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:41.427+0000 7fef4b7fe640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fef4c15a170 0x7fef4c0b3420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:55094/0 (socket says 172.21.15.123:55094) 2024-09-16T15:44:41.433 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:41.427+0000 7fef4b7fe640 1 -- 172.21.15.123:0/1703284043 learned_addr learned my addr 172.21.15.123:0/1703284043 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:44:41.433 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:41.427+0000 7fef4bfff640 1 --2- 172.21.15.123:0/1703284043 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fef4c0ca650 0x7fef4c154450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:44:41.433 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:41.427+0000 7fef5098e640 1 --2- 172.21.15.123:0/1703284043 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fef4c15ddc0 0x7fef4c151b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:44:41.433 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:41.427+0000 7fef4b7fe640 1 -- 172.21.15.123:0/1703284043 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fef4c0ca650 msgr2=0x7fef4c154450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:44:41.433 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:41.427+0000 7fef4b7fe640 1 --2- 172.21.15.123:0/1703284043 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fef4c0ca650 0x7fef4c154450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:41.433 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:41.427+0000 7fef4b7fe640 1 -- 172.21.15.123:0/1703284043 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fef4c15ddc0 msgr2=0x7fef4c151b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:44:41.433 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:41.427+0000 7fef4b7fe640 1 --2- 172.21.15.123:0/1703284043 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fef4c15ddc0 0x7fef4c151b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:41.433 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:41.427+0000 7fef4b7fe640 1 -- 172.21.15.123:0/1703284043 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fef4c152050 con 0x7fef4c15a170 2024-09-16T15:44:41.433 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:41.427+0000 7fef5098e640 1 --2- 172.21.15.123:0/1703284043 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fef4c15ddc0 0x7fef4c151b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T15:44:41.433 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:41.431+0000 7fef4b7fe640 1 --2- 172.21.15.123:0/1703284043 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fef4c15a170 0x7fef4c0b3420 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7fef4000c480 tx=0x7fef4000c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:44:41.433 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:41.431+0000 7fef48ff9640 1 -- 172.21.15.123:0/1703284043 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef40016020 con 0x7fef4c15a170 2024-09-16T15:44:41.433 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:41.431+0000 7fef48ff9640 1 -- 172.21.15.123:0/1703284043 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fef40012920 con 0x7fef4c15a170 2024-09-16T15:44:41.434 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:41.431+0000 7fef523f1640 1 -- 172.21.15.123:0/1703284043 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fef4c154a30 con 0x7fef4c15a170 2024-09-16T15:44:41.434 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:41.431+0000 7fef48ff9640 1 -- 172.21.15.123:0/1703284043 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef40012db0 con 0x7fef4c15a170 2024-09-16T15:44:41.434 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:41.431+0000 7fef523f1640 1 -- 172.21.15.123:0/1703284043 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fef4c153160 con 0x7fef4c15a170 2024-09-16T15:44:41.434 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:41.431+0000 7fef523f1640 1 -- 172.21.15.123:0/1703284043 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fef4c072f30 con 0x7fef4c15a170 2024-09-16T15:44:41.435 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:41.431+0000 7fef48ff9640 1 -- 172.21.15.123:0/1703284043 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 7) ==== 97233+0+0 (secure 0 0 0) 0x7fef40019020 con 0x7fef4c15a170 2024-09-16T15:44:41.435 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:41.431+0000 7fef48ff9640 1 --2- 172.21.15.123:0/1703284043 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fef38073070 0x7fef38075530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:44:41.435 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:41.431+0000 7fef48ff9640 1 -- 172.21.15.123:0/1703284043 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(46..46 src has 1..46) ==== 5767+0+0 (secure 0 0 0) 0x7fef40094620 con 0x7fef4c15a170 2024-09-16T15:44:41.435 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:41.431+0000 7fef4bfff640 1 --2- 172.21.15.123:0/1703284043 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fef38073070 0x7fef38075530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:44:41.435 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:41.431+0000 7fef4bfff640 1 --2- 172.21.15.123:0/1703284043 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fef38073070 0x7fef38075530 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fef3c005fd0 tx=0x7fef3c00a4e0 comp rx=0 tx=0).ready entity=mgr.4135 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:44:41.438 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:41.435+0000 7fef48ff9640 1 -- 172.21.15.123:0/1703284043 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7fef40020030 con 0x7fef4c15a170 2024-09-16T15:44:41.629 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:41.627+0000 7fef523f1640 1 -- 172.21.15.123:0/1703284043 --> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "subvol56410", "target": ["mon-mgr", ""]}) -- 0x7fef4c15e980 con 0x7fef38073070 2024-09-16T15:44:41.641 INFO:tasks.ceph.mgr.x.smithi184.stderr:2024-09-16T15:44:41.636+0000 7fda6a8d4640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-16T15:44:41.641 INFO:tasks.ceph.mgr.x.smithi184.stderr:2024-09-16T15:44:41.636+0000 7fda6a8d4640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-16T15:44:41.641 INFO:tasks.ceph.mgr.x.smithi184.stderr:2024-09-16T15:44:41.636+0000 7fda6a8d4640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-16T15:44:41.641 INFO:tasks.ceph.mgr.x.smithi184.stderr:2024-09-16T15:44:41.636+0000 7fda6a8d4640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-16T15:44:41.641 INFO:tasks.ceph.mgr.x.smithi184.stderr:2024-09-16T15:44:41.636+0000 7fda6a8d4640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-16T15:44:41.691 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:41.691+0000 7fef48ff9640 1 -- 172.21.15.123:0/1703284043 <== mgr.4135 v2:172.21.15.184:6832/3114346423 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7fef4c15e980 con 0x7fef38073070 2024-09-16T15:44:41.695 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:41.691+0000 7fef523f1640 1 -- 172.21.15.123:0/1703284043 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fef38073070 msgr2=0x7fef38075530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:44:41.695 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:41.691+0000 7fef523f1640 1 --2- 172.21.15.123:0/1703284043 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fef38073070 0x7fef38075530 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fef3c005fd0 tx=0x7fef3c00a4e0 comp rx=0 tx=0).stop 2024-09-16T15:44:41.695 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:41.691+0000 7fef523f1640 1 -- 172.21.15.123:0/1703284043 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fef4c15a170 msgr2=0x7fef4c0b3420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:44:41.695 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:41.691+0000 7fef523f1640 1 --2- 172.21.15.123:0/1703284043 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fef4c15a170 0x7fef4c0b3420 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7fef4000c480 tx=0x7fef4000c940 comp rx=0 tx=0).stop 2024-09-16T15:44:41.695 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:41.691+0000 7fef523f1640 1 -- 172.21.15.123:0/1703284043 shutdown_connections 2024-09-16T15:44:41.695 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:41.691+0000 7fef523f1640 1 --2- 172.21.15.123:0/1703284043 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fef38073070 0x7fef38075530 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:41.695 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:41.691+0000 7fef523f1640 1 --2- 172.21.15.123:0/1703284043 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fef4c15ddc0 0x7fef4c151b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:41.695 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:41.691+0000 7fef523f1640 1 --2- 172.21.15.123:0/1703284043 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fef4c15a170 0x7fef4c0b3420 unknown :-1 s=CLOSED pgs=510 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:41.696 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:41.691+0000 7fef523f1640 1 --2- 172.21.15.123:0/1703284043 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fef4c0ca650 0x7fef4c154450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:41.696 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:41.691+0000 7fef523f1640 1 -- 172.21.15.123:0/1703284043 >> 172.21.15.123:0/1703284043 conn(0x7fef4c0b1710 msgr2=0x7fef4c158dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:44:41.696 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:41.691+0000 7fef523f1640 1 -- 172.21.15.123:0/1703284043 shutdown_connections 2024-09-16T15:44:41.696 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:41.691+0000 7fef523f1640 1 -- 172.21.15.123:0/1703284043 wait complete. 2024-09-16T15:44:41.712 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-16T15:44:41.840 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:41.839+0000 7f16d5ede640 1 -- 172.21.15.123:0/1166912203 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f16b8004e70 msgr2=0x7f16b800b9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:44:41.840 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:41.839+0000 7f16d5ede640 1 --2- 172.21.15.123:0/1166912203 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f16b8004e70 0x7f16b800b9d0 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7f16c800c690 tx=0x7f16c801d2e0 comp rx=0 tx=0).stop 2024-09-16T15:44:41.840 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:41.839+0000 7f16cffff640 1 -- 172.21.15.123:0/1166912203 reap_dead start 2024-09-16T15:44:41.840 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:41.839+0000 7f16d5ede640 1 -- 172.21.15.123:0/1166912203 shutdown_connections 2024-09-16T15:44:41.840 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:41.839+0000 7f16d5ede640 1 -- 172.21.15.123:0/1166912203 >> 172.21.15.123:0/1166912203 conn(0x7f16d00cfb10 msgr2=0x7f16d00cff20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:44:41.840 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:41.839+0000 7f16d5ede640 1 -- 172.21.15.123:0/1166912203 shutdown_connections 2024-09-16T15:44:41.840 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:41.839+0000 7f16d5ede640 1 -- 172.21.15.123:0/1166912203 wait complete. 2024-09-16T15:44:41.841 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:41.839+0000 7f16d5ede640 1 Processor -- start 2024-09-16T15:44:41.841 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:41.839+0000 7f16d5ede640 1 -- start start 2024-09-16T15:44:41.843 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:41.839+0000 7f16d5ede640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f16b8004e70 0x7f16d0154970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:44:41.843 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:41.839+0000 7f16d5ede640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f16d01598e0 0x7f16d015f160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:44:41.843 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:41.839+0000 7f16d5ede640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f16d00aecb0 0x7f16d015c330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:44:41.844 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:41.839+0000 7f16d5ede640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f16d006f630 con 0x7f16d01598e0 2024-09-16T15:44:41.844 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:41.839+0000 7f16d5ede640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f16d006f4b0 con 0x7f16b8004e70 2024-09-16T15:44:41.844 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:41.839+0000 7f16d5ede640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f16d006f7b0 con 0x7f16d00aecb0 2024-09-16T15:44:41.844 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:41.839+0000 7f16cf7fe640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f16b8004e70 0x7f16d0154970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:44:41.844 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:41.839+0000 7f16ceffd640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f16d01598e0 0x7f16d015f160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:44:41.844 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:41.839+0000 7f16cf7fe640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f16b8004e70 0x7f16d0154970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.123:42132/0 (socket says 172.21.15.123:42132) 2024-09-16T15:44:41.844 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:41.839+0000 7f16cf7fe640 1 -- 172.21.15.123:0/3547742411 learned_addr learned my addr 172.21.15.123:0/3547742411 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:44:41.844 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:41.839+0000 7f16cffff640 1 --2- 172.21.15.123:0/3547742411 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f16d00aecb0 0x7f16d015c330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:44:41.844 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:41.839+0000 7f16ceffd640 1 -- 172.21.15.123:0/3547742411 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f16d00aecb0 msgr2=0x7f16d015c330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:44:41.844 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:41.839+0000 7f16ceffd640 1 --2- 172.21.15.123:0/3547742411 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f16d00aecb0 0x7f16d015c330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:41.844 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:41.839+0000 7f16ceffd640 1 -- 172.21.15.123:0/3547742411 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f16b8004e70 msgr2=0x7f16d0154970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:44:41.844 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:41.839+0000 7f16ceffd640 1 --2- 172.21.15.123:0/3547742411 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f16b8004e70 0x7f16d0154970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:41.845 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:41.839+0000 7f16ceffd640 1 -- 172.21.15.123:0/3547742411 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f16d0159e30 con 0x7f16d01598e0 2024-09-16T15:44:41.845 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:41.839+0000 7f16ceffd640 1 --2- 172.21.15.123:0/3547742411 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f16d01598e0 0x7f16d015f160 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7f16c400ebc0 tx=0x7f16c400c740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:44:41.845 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:41.839+0000 7f16b7fff640 1 -- 172.21.15.123:0/3547742411 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16c4005220 con 0x7f16d01598e0 2024-09-16T15:44:41.845 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:41.839+0000 7f16b7fff640 1 -- 172.21.15.123:0/3547742411 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f16c40054b0 con 0x7f16d01598e0 2024-09-16T15:44:41.845 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:41.839+0000 7f16b7fff640 1 -- 172.21.15.123:0/3547742411 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16c4005660 con 0x7f16d01598e0 2024-09-16T15:44:41.845 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:41.843+0000 7f16d5ede640 1 -- 172.21.15.123:0/3547742411 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f16d015a110 con 0x7f16d01598e0 2024-09-16T15:44:41.845 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:41.843+0000 7f16d5ede640 1 -- 172.21.15.123:0/3547742411 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f16d015a6d0 con 0x7f16d01598e0 2024-09-16T15:44:41.846 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:41.843+0000 7f16d5ede640 1 -- 172.21.15.123:0/3547742411 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f16d015a8d0 con 0x7f16d01598e0 2024-09-16T15:44:41.846 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:41.843+0000 7f16b7fff640 1 -- 172.21.15.123:0/3547742411 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 7) ==== 97233+0+0 (secure 0 0 0) 0x7f16c40049f0 con 0x7f16d01598e0 2024-09-16T15:44:41.846 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:41.843+0000 7f16b7fff640 1 --2- 172.21.15.123:0/3547742411 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f16bc0730a0 0x7f16bc075560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:44:41.846 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:41.843+0000 7f16b7fff640 1 -- 172.21.15.123:0/3547742411 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(46..46 src has 1..46) ==== 5767+0+0 (secure 0 0 0) 0x7f16c4094890 con 0x7f16d01598e0 2024-09-16T15:44:41.846 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:41.843+0000 7f16cf7fe640 1 --2- 172.21.15.123:0/3547742411 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f16bc0730a0 0x7f16bc075560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:44:41.847 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:41.843+0000 7f16cf7fe640 1 --2- 172.21.15.123:0/3547742411 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f16bc0730a0 0x7f16bc075560 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f16c0006680 tx=0x7f16c00011a0 comp rx=0 tx=0).ready entity=mgr.4135 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:44:41.849 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:41.847+0000 7f16b7fff640 1 -- 172.21.15.123:0/3547742411 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f16d015a8d0 con 0x7f16d01598e0 2024-09-16T15:44:42.049 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:42.047+0000 7f16d5ede640 1 -- 172.21.15.123:0/3547742411 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f16d01fb300 con 0x7f16d01598e0 2024-09-16T15:44:42.050 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:42.047+0000 7f16b7fff640 1 -- 172.21.15.123:0/3547742411 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) ==== 94+0+5223 (secure 0 0 0) 0x7f16c4013070 con 0x7f16d01598e0 2024-09-16T15:44:42.050 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-09-16T15:44:42.050 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":28,"btime":"2024-09-16T15:44:32:363512+0000","default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4712,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6837/288264117","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6836","nonce":288264117},{"type":"v1","addr":"172.21.15.184:6837","nonce":288264117}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":23},{"gid":5268,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6834/60194515","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6832","nonce":60194515},{"type":"v1","addr":"172.21.15.123:6834","nonce":60194515}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":23},{"gid":5271,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6835/1798373479","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6833","nonce":1798373479},{"type":"v1","addr":"172.21.15.123:6835","nonce":1798373479}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":22}],"filesystems":[{"mdsmap":{"epoch":27,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-16T15:44:14.853230+0000","modified":"2024-09-16T15:44:17.861742+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5280},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5280":{"gid":5280,"name":"d","rank":0,"incarnation":24,"state":"up:active","state_seq":3,"addr":"172.21.15.184:6835/374389412","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":374389412},{"type":"v1","addr":"172.21.15.184:6835","nonce":374389412}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5280,"qdb_cluster":[5280]},"id":3}]} 2024-09-16T15:44:42.051 INFO:teuthology.orchestra.run.smithi123.stderr:dumped fsmap epoch 28 2024-09-16T15:44:42.052 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:42.051+0000 7f16d5ede640 1 -- 172.21.15.123:0/3547742411 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f16bc0730a0 msgr2=0x7f16bc075560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:44:42.053 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:42.051+0000 7f16d5ede640 1 --2- 172.21.15.123:0/3547742411 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f16bc0730a0 0x7f16bc075560 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f16c0006680 tx=0x7f16c00011a0 comp rx=0 tx=0).stop 2024-09-16T15:44:42.053 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:42.051+0000 7f16d5ede640 1 -- 172.21.15.123:0/3547742411 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f16d01598e0 msgr2=0x7f16d015f160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:44:42.053 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:42.051+0000 7f16d5ede640 1 --2- 172.21.15.123:0/3547742411 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f16d01598e0 0x7f16d015f160 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7f16c400ebc0 tx=0x7f16c400c740 comp rx=0 tx=0).stop 2024-09-16T15:44:42.053 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:42.051+0000 7f16d5ede640 1 -- 172.21.15.123:0/3547742411 shutdown_connections 2024-09-16T15:44:42.053 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:42.051+0000 7f16d5ede640 1 --2- 172.21.15.123:0/3547742411 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f16bc0730a0 0x7f16bc075560 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:42.053 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:42.051+0000 7f16d5ede640 1 --2- 172.21.15.123:0/3547742411 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f16d00aecb0 0x7f16d015c330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:42.053 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:42.051+0000 7f16d5ede640 1 --2- 172.21.15.123:0/3547742411 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f16d01598e0 0x7f16d015f160 unknown :-1 s=CLOSED pgs=513 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:42.053 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:42.051+0000 7f16d5ede640 1 --2- 172.21.15.123:0/3547742411 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f16b8004e70 0x7f16d0154970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:42.053 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:42.051+0000 7f16d5ede640 1 -- 172.21.15.123:0/3547742411 >> 172.21.15.123:0/3547742411 conn(0x7f16d00cfb10 msgr2=0x7f16d015ba20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:44:42.053 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:42.051+0000 7f16d5ede640 1 -- 172.21.15.123:0/3547742411 shutdown_connections 2024-09-16T15:44:42.053 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:42.051+0000 7f16d5ede640 1 -- 172.21.15.123:0/3547742411 wait complete. 2024-09-16T15:44:42.068 DEBUG:teuthology.orchestra.run.smithi184:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.d.asok --format=json session ls 2024-09-16T15:44:42.218 INFO:teuthology.orchestra.run.smithi184.stdout:[{"id":5047,"entity":{"name":{"type":"client","num":5047},"addr":{"type":"any","addr":"172.21.15.184:0","nonce":908570948}},"state":"open","num_leases":1,"num_caps":7,"request_load_avg":15,"uptime":0.559986231,"requests_in_flight":2,"num_completed_requests":0,"num_completed_flushes":1,"reconnecting":false,"recall_caps":{"value":0,"halflife":60},"release_caps":{"value":0,"halflife":60},"recall_caps_throttle":{"value":0,"halflife":1.5},"recall_caps_throttle2o":{"value":0,"halflife":0.5},"session_cache_liveness":{"value":21.972393797718311,"halflife":300},"cap_acquisition":{"value":0,"halflife":30},"last_trim_completed_requests_tid":15,"last_trim_completed_flushes_tid":2,"delegated_inos":[{"start":"0x10000000005","length":500}],"inst":"client.5047 172.21.15.184:0/908570948","completed_requests":[],"prealloc_inos":[{"start":"0x10000000005","length":996}],"client_metadata":{"client_features":{"feature_bits":"0x00000000003fffff"},"metric_spec":{"metric_flags":{"feature_bits":"0x000000000000ffff"}},"ceph_sha1":"33f95d2ae1bcbe7eb04404303f179eeeaa6982df","ceph_version":"ceph version 19.3.0-4938-g33f95d2a (33f95d2ae1bcbe7eb04404303f179eeeaa6982df) squid (dev)","entity_id":"x","hostname":"smithi184","pid":"28840","root":"/"}}] 2024-09-16T15:44:42.219 DEBUG:tasks.cephfs.filesystem:_json_asok output [ { "cap_acquisition": { "halflife": 30, "value": 0 }, "client_metadata": { "ceph_sha1": "33f95d2ae1bcbe7eb04404303f179eeeaa6982df", "ceph_version": "ceph version 19.3.0-4938-g33f95d2a (33f95d2ae1bcbe7eb04404303f179eeeaa6982df) squid (dev)", "client_features": { "feature_bits": "0x00000000003fffff" }, "entity_id": "x", "hostname": "smithi184", "metric_spec": { "metric_flags": { "feature_bits": "0x000000000000ffff" } }, "pid": "28840", "root": "/" }, "completed_requests": [], "delegated_inos": [ { "length": 500, "start": "0x10000000005" } ], "entity": { "addr": { "addr": "172.21.15.184:0", "nonce": 908570948, "type": "any" }, "name": { "num": 5047, "type": "client" } }, "id": 5047, "inst": "client.5047 172.21.15.184:0/908570948", "last_trim_completed_flushes_tid": 2, "last_trim_completed_requests_tid": 15, "num_caps": 7, "num_completed_flushes": 1, "num_completed_requests": 0, "num_leases": 1, "prealloc_inos": [ { "length": 996, "start": "0x10000000005" } ], "recall_caps": { "halflife": 60, "value": 0 }, "recall_caps_throttle": { "halflife": 1.5, "value": 0 }, "recall_caps_throttle2o": { "halflife": 0.5, "value": 0 }, "reconnecting": false, "release_caps": { "halflife": 60, "value": 0 }, "request_load_avg": 15, "requests_in_flight": 2, "session_cache_liveness": { "halflife": 300, "value": 21.97239379771831 }, "state": "open", "uptime": 0.559986231 } ] 2024-09-16T15:44:42.219 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-16T15:44:42.350 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:42.347+0000 7f5018a3a640 1 -- 172.21.15.123:0/2085333551 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f50140726c0 msgr2=0x7f5014073ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:44:42.350 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:42.347+0000 7f5018a3a640 1 --2- 172.21.15.123:0/2085333551 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f50140726c0 0x7f5014073ef0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f5004009c80 tx=0x7f500401cce0 comp rx=0 tx=0).stop 2024-09-16T15:44:42.350 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:42.347+0000 7f5018a3a640 1 -- 172.21.15.123:0/2085333551 shutdown_connections 2024-09-16T15:44:42.350 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:42.347+0000 7f5018a3a640 1 --2- 172.21.15.123:0/2085333551 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f50140726c0 0x7f5014073ef0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:42.350 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:42.347+0000 7f5018a3a640 1 --2- 172.21.15.123:0/2085333551 >> v2:172.21.15.184:3300/0 conn(0x7f501415bca0 0x7f50140d0630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:42.350 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:42.347+0000 7f5018a3a640 1 -- 172.21.15.123:0/2085333551 >> 172.21.15.123:0/2085333551 conn(0x7f50140cfb10 msgr2=0x7f50140cff20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:44:42.350 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:42.347+0000 7f5018a3a640 1 -- 172.21.15.123:0/2085333551 shutdown_connections 2024-09-16T15:44:42.350 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:42.347+0000 7f5018a3a640 1 -- 172.21.15.123:0/2085333551 wait complete. 2024-09-16T15:44:42.351 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:42.347+0000 7f5018a3a640 1 Processor -- start 2024-09-16T15:44:42.351 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:42.347+0000 7f5018a3a640 1 -- start start 2024-09-16T15:44:42.353 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:42.351+0000 7f5018a3a640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f50140726c0 0x7f5014164e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:44:42.353 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:42.351+0000 7f5018a3a640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5014074430 0x7f5014165340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:44:42.353 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:42.351+0000 7f5018a3a640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f501415bca0 0x7f50140c7010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:44:42.353 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:42.351+0000 7f5018a3a640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f5014168e70 con 0x7f501415bca0 2024-09-16T15:44:42.353 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:42.351+0000 7f5018a3a640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f5014168cf0 con 0x7f5014074430 2024-09-16T15:44:42.353 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:42.351+0000 7f5018a3a640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f5014168ff0 con 0x7f50140726c0 2024-09-16T15:44:42.353 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:42.351+0000 7f5012d9d640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f501415bca0 0x7f50140c7010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:44:42.353 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:42.351+0000 7f5012d9d640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f501415bca0 0x7f50140c7010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:55116/0 (socket says 172.21.15.123:55116) 2024-09-16T15:44:42.353 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:42.351+0000 7f5012d9d640 1 -- 172.21.15.123:0/903342840 learned_addr learned my addr 172.21.15.123:0/903342840 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:44:42.353 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:42.351+0000 7f5011d9b640 1 --2- 172.21.15.123:0/903342840 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5014074430 0x7f5014165340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:44:42.354 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:42.351+0000 7f501259c640 1 --2- 172.21.15.123:0/903342840 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f50140726c0 0x7f5014164e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:44:42.354 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:42.351+0000 7f5012d9d640 1 -- 172.21.15.123:0/903342840 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f50140726c0 msgr2=0x7f5014164e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:44:42.354 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:42.351+0000 7f5012d9d640 1 --2- 172.21.15.123:0/903342840 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f50140726c0 0x7f5014164e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:42.354 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:42.351+0000 7f5012d9d640 1 -- 172.21.15.123:0/903342840 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5014074430 msgr2=0x7f5014165340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:44:42.354 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:42.351+0000 7f5012d9d640 1 --2- 172.21.15.123:0/903342840 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5014074430 0x7f5014165340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:42.354 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:42.351+0000 7f5012d9d640 1 -- 172.21.15.123:0/903342840 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f50140c7c90 con 0x7f501415bca0 2024-09-16T15:44:42.354 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:42.351+0000 7f5011d9b640 1 --2- 172.21.15.123:0/903342840 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5014074430 0x7f5014165340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T15:44:42.354 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:42.351+0000 7f5012d9d640 1 --2- 172.21.15.123:0/903342840 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f501415bca0 0x7f50140c7010 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7f500c00c400 tx=0x7f500c00c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:44:42.354 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:42.351+0000 7f500affd640 1 -- 172.21.15.123:0/903342840 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f500c017070 con 0x7f501415bca0 2024-09-16T15:44:42.354 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:42.351+0000 7f500affd640 1 -- 172.21.15.123:0/903342840 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f500c0053c0 con 0x7f501415bca0 2024-09-16T15:44:42.354 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:42.351+0000 7f500affd640 1 -- 172.21.15.123:0/903342840 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f500c003860 con 0x7f501415bca0 2024-09-16T15:44:42.354 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:42.351+0000 7f5018a3a640 1 -- 172.21.15.123:0/903342840 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f50141fb670 con 0x7f501415bca0 2024-09-16T15:44:42.354 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:42.351+0000 7f5018a3a640 1 -- 172.21.15.123:0/903342840 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f50141fbc30 con 0x7f501415bca0 2024-09-16T15:44:42.355 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:42.351+0000 7f5018a3a640 1 -- 172.21.15.123:0/903342840 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5014073ef0 con 0x7f501415bca0 2024-09-16T15:44:42.355 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:42.351+0000 7f500affd640 1 -- 172.21.15.123:0/903342840 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 7) ==== 97233+0+0 (secure 0 0 0) 0x7f500c01a070 con 0x7f501415bca0 2024-09-16T15:44:42.355 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:42.355+0000 7f500affd640 1 --2- 172.21.15.123:0/903342840 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f4ff40730a0 0x7f4ff4075560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:44:42.355 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:42.355+0000 7f500affd640 1 -- 172.21.15.123:0/903342840 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(46..46 src has 1..46) ==== 5767+0+0 (secure 0 0 0) 0x7f500c093530 con 0x7f501415bca0 2024-09-16T15:44:42.356 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:42.355+0000 7f501259c640 1 --2- 172.21.15.123:0/903342840 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f4ff40730a0 0x7f4ff4075560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:44:42.356 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:42.355+0000 7f501259c640 1 --2- 172.21.15.123:0/903342840 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f4ff40730a0 0x7f4ff4075560 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f50141659c0 tx=0x7f4ffc0091e0 comp rx=0 tx=0).ready entity=mgr.4135 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:44:42.359 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:42.359+0000 7f500affd640 1 -- 172.21.15.123:0/903342840 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f500c05e440 con 0x7f501415bca0 2024-09-16T15:44:42.555 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:42.551+0000 7f5018a3a640 1 -- 172.21.15.123:0/903342840 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f50140700e0 con 0x7f501415bca0 2024-09-16T15:44:42.556 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:42.555+0000 7f500affd640 1 -- 172.21.15.123:0/903342840 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) ==== 94+0+5223 (secure 0 0 0) 0x7f500c05dd40 con 0x7f501415bca0 2024-09-16T15:44:42.556 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-09-16T15:44:42.556 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":28,"btime":"2024-09-16T15:44:32:363512+0000","default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4712,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6837/288264117","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6836","nonce":288264117},{"type":"v1","addr":"172.21.15.184:6837","nonce":288264117}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":23},{"gid":5268,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6834/60194515","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6832","nonce":60194515},{"type":"v1","addr":"172.21.15.123:6834","nonce":60194515}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":23},{"gid":5271,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6835/1798373479","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6833","nonce":1798373479},{"type":"v1","addr":"172.21.15.123:6835","nonce":1798373479}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":22}],"filesystems":[{"mdsmap":{"epoch":27,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-16T15:44:14.853230+0000","modified":"2024-09-16T15:44:17.861742+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5280},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5280":{"gid":5280,"name":"d","rank":0,"incarnation":24,"state":"up:active","state_seq":3,"addr":"172.21.15.184:6835/374389412","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":374389412},{"type":"v1","addr":"172.21.15.184:6835","nonce":374389412}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5280,"qdb_cluster":[5280]},"id":3}]} 2024-09-16T15:44:42.556 INFO:teuthology.orchestra.run.smithi123.stderr:dumped fsmap epoch 28 2024-09-16T15:44:42.559 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:42.555+0000 7f5018a3a640 1 -- 172.21.15.123:0/903342840 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f4ff40730a0 msgr2=0x7f4ff4075560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:44:42.559 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:42.555+0000 7f5018a3a640 1 --2- 172.21.15.123:0/903342840 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f4ff40730a0 0x7f4ff4075560 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f50141659c0 tx=0x7f4ffc0091e0 comp rx=0 tx=0).stop 2024-09-16T15:44:42.559 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:42.555+0000 7f5018a3a640 1 -- 172.21.15.123:0/903342840 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f501415bca0 msgr2=0x7f50140c7010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:44:42.560 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:42.555+0000 7f5018a3a640 1 --2- 172.21.15.123:0/903342840 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f501415bca0 0x7f50140c7010 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7f500c00c400 tx=0x7f500c00c8c0 comp rx=0 tx=0).stop 2024-09-16T15:44:42.560 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:42.555+0000 7f5018a3a640 1 -- 172.21.15.123:0/903342840 shutdown_connections 2024-09-16T15:44:42.560 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:42.555+0000 7f5018a3a640 1 --2- 172.21.15.123:0/903342840 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f4ff40730a0 0x7f4ff4075560 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:42.560 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:42.555+0000 7f5018a3a640 1 --2- 172.21.15.123:0/903342840 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f501415bca0 0x7f50140c7010 unknown :-1 s=CLOSED pgs=514 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:42.560 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:42.555+0000 7f5018a3a640 1 --2- 172.21.15.123:0/903342840 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5014074430 0x7f5014165340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:42.560 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:42.555+0000 7f5018a3a640 1 --2- 172.21.15.123:0/903342840 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f50140726c0 0x7f5014164e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:42.560 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:42.555+0000 7f5018a3a640 1 -- 172.21.15.123:0/903342840 >> 172.21.15.123:0/903342840 conn(0x7f50140cfb10 msgr2=0x7f50141662b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:44:42.560 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:42.559+0000 7f5018a3a640 1 -- 172.21.15.123:0/903342840 shutdown_connections 2024-09-16T15:44:42.560 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:42.559+0000 7f5018a3a640 1 -- 172.21.15.123:0/903342840 wait complete. 2024-09-16T15:44:42.575 DEBUG:teuthology.orchestra.run.smithi184:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.d.asok --format=json session ls 2024-09-16T15:44:42.723 INFO:teuthology.orchestra.run.smithi184.stdout:[{"id":5047,"entity":{"name":{"type":"client","num":5047},"addr":{"type":"any","addr":"172.21.15.184:0","nonce":908570948}},"state":"open","num_leases":1,"num_caps":7,"request_load_avg":15,"uptime":1.0639741700000001,"requests_in_flight":2,"num_completed_requests":0,"num_completed_flushes":1,"reconnecting":false,"recall_caps":{"value":0,"halflife":60},"release_caps":{"value":0,"halflife":60},"recall_caps_throttle":{"value":0,"halflife":1.5},"recall_caps_throttle2o":{"value":0,"halflife":0.5},"session_cache_liveness":{"value":21.946822728422081,"halflife":300},"cap_acquisition":{"value":0,"halflife":30},"last_trim_completed_requests_tid":15,"last_trim_completed_flushes_tid":2,"delegated_inos":[{"start":"0x10000000005","length":500}],"inst":"client.5047 172.21.15.184:0/908570948","completed_requests":[],"prealloc_inos":[{"start":"0x10000000005","length":996}],"client_metadata":{"client_features":{"feature_bits":"0x00000000003fffff"},"metric_spec":{"metric_flags":{"feature_bits":"0x000000000000ffff"}},"ceph_sha1":"33f95d2ae1bcbe7eb04404303f179eeeaa6982df","ceph_version":"ceph version 19.3.0-4938-g33f95d2a (33f95d2ae1bcbe7eb04404303f179eeeaa6982df) squid (dev)","entity_id":"x","hostname":"smithi184","pid":"28840","root":"/"}}] 2024-09-16T15:44:42.724 DEBUG:tasks.cephfs.filesystem:_json_asok output [ { "cap_acquisition": { "halflife": 30, "value": 0 }, "client_metadata": { "ceph_sha1": "33f95d2ae1bcbe7eb04404303f179eeeaa6982df", "ceph_version": "ceph version 19.3.0-4938-g33f95d2a (33f95d2ae1bcbe7eb04404303f179eeeaa6982df) squid (dev)", "client_features": { "feature_bits": "0x00000000003fffff" }, "entity_id": "x", "hostname": "smithi184", "metric_spec": { "metric_flags": { "feature_bits": "0x000000000000ffff" } }, "pid": "28840", "root": "/" }, "completed_requests": [], "delegated_inos": [ { "length": 500, "start": "0x10000000005" } ], "entity": { "addr": { "addr": "172.21.15.184:0", "nonce": 908570948, "type": "any" }, "name": { "num": 5047, "type": "client" } }, "id": 5047, "inst": "client.5047 172.21.15.184:0/908570948", "last_trim_completed_flushes_tid": 2, "last_trim_completed_requests_tid": 15, "num_caps": 7, "num_completed_flushes": 1, "num_completed_requests": 0, "num_leases": 1, "prealloc_inos": [ { "length": 996, "start": "0x10000000005" } ], "recall_caps": { "halflife": 60, "value": 0 }, "recall_caps_throttle": { "halflife": 1.5, "value": 0 }, "recall_caps_throttle2o": { "halflife": 0.5, "value": 0 }, "reconnecting": false, "release_caps": { "halflife": 60, "value": 0 }, "request_load_avg": 15, "requests_in_flight": 2, "session_cache_liveness": { "halflife": 300, "value": 21.94682272842208 }, "state": "open", "uptime": 1.06397417 } ] 2024-09-16T15:44:42.724 DEBUG:tasks.ceph_test_case:wait_until_true: waiting (timeout=90 retry_count=0)... 2024-09-16T15:44:47.724 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-16T15:44:47.864 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:47.863+0000 7fe445e42640 1 -- 172.21.15.123:0/3617528165 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe428004810 msgr2=0x7fe428004c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:44:47.864 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:47.863+0000 7fe445e42640 1 --2- 172.21.15.123:0/3617528165 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe428004810 0x7fe428004c90 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7fe434006fd0 tx=0x7fe43401cf60 comp rx=0 tx=0).stop 2024-09-16T15:44:47.864 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:47.863+0000 7fe445e42640 1 -- 172.21.15.123:0/3617528165 shutdown_connections 2024-09-16T15:44:47.864 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:47.863+0000 7fe445e42640 1 --2- 172.21.15.123:0/3617528165 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe428004810 0x7fe428004c90 unknown :-1 s=CLOSED pgs=516 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:47.865 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:47.863+0000 7fe445e42640 1 -- 172.21.15.123:0/3617528165 >> 172.21.15.123:0/3617528165 conn(0x7fe440069650 msgr2=0x7fe44009d170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:44:47.865 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:47.863+0000 7fe445e42640 1 -- 172.21.15.123:0/3617528165 shutdown_connections 2024-09-16T15:44:47.865 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:47.863+0000 7fe445e42640 1 -- 172.21.15.123:0/3617528165 wait complete. 2024-09-16T15:44:47.865 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:47.863+0000 7fe445e42640 1 Processor -- start 2024-09-16T15:44:47.865 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:47.863+0000 7fe445e42640 1 -- start start 2024-09-16T15:44:47.867 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:47.863+0000 7fe445e42640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fe428004810 0x7fe4400c8260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:44:47.867 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:47.863+0000 7fe445e42640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe4400c79e0 0x7fe4400ced90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:44:47.867 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:47.863+0000 7fe445e42640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe44015cd20 0x7fe44015d1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:44:47.867 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:47.863+0000 7fe445e42640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fe4400d1440 con 0x7fe44015cd20 2024-09-16T15:44:47.867 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:47.863+0000 7fe445e42640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fe4400d12c0 con 0x7fe4400c79e0 2024-09-16T15:44:47.867 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:47.863+0000 7fe445e42640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fe4400d15c0 con 0x7fe428004810 2024-09-16T15:44:47.868 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:47.863+0000 7fe43ffff640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe44015cd20 0x7fe44015d1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:44:47.868 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:47.863+0000 7fe43ffff640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe44015cd20 0x7fe44015d1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:52084/0 (socket says 172.21.15.123:52084) 2024-09-16T15:44:47.868 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:47.863+0000 7fe43ffff640 1 -- 172.21.15.123:0/1330558855 learned_addr learned my addr 172.21.15.123:0/1330558855 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:44:47.868 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:47.863+0000 7fe43effd640 1 --2- 172.21.15.123:0/1330558855 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe4400c79e0 0x7fe4400ced90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:44:47.868 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:47.863+0000 7fe43f7fe640 1 --2- 172.21.15.123:0/1330558855 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fe428004810 0x7fe4400c8260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:44:47.868 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:47.863+0000 7fe43ffff640 1 -- 172.21.15.123:0/1330558855 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fe428004810 msgr2=0x7fe4400c8260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:44:47.868 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:47.863+0000 7fe43ffff640 1 --2- 172.21.15.123:0/1330558855 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fe428004810 0x7fe4400c8260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:47.868 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:47.863+0000 7fe43ffff640 1 -- 172.21.15.123:0/1330558855 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe4400c79e0 msgr2=0x7fe4400ced90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:44:47.868 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:47.863+0000 7fe43ffff640 1 --2- 172.21.15.123:0/1330558855 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe4400c79e0 0x7fe4400ced90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:47.868 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:47.863+0000 7fe43ffff640 1 -- 172.21.15.123:0/1330558855 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe4401f7d60 con 0x7fe44015cd20 2024-09-16T15:44:47.868 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:47.863+0000 7fe43f7fe640 1 --2- 172.21.15.123:0/1330558855 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fe428004810 0x7fe4400c8260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T15:44:47.868 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:47.863+0000 7fe43ffff640 1 --2- 172.21.15.123:0/1330558855 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe44015cd20 0x7fe44015d1a0 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7fe4400d19a0 tx=0x7fe43800db50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:44:47.868 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:47.863+0000 7fe427fff640 1 -- 172.21.15.123:0/1330558855 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe438003b80 con 0x7fe44015cd20 2024-09-16T15:44:47.869 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:47.863+0000 7fe427fff640 1 -- 172.21.15.123:0/1330558855 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fe438015070 con 0x7fe44015cd20 2024-09-16T15:44:47.869 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:47.863+0000 7fe445e42640 1 -- 172.21.15.123:0/1330558855 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe4400c88f0 con 0x7fe44015cd20 2024-09-16T15:44:47.869 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:47.863+0000 7fe427fff640 1 -- 172.21.15.123:0/1330558855 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe43800ba00 con 0x7fe44015cd20 2024-09-16T15:44:47.869 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:47.863+0000 7fe445e42640 1 -- 172.21.15.123:0/1330558855 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe4401f8300 con 0x7fe44015cd20 2024-09-16T15:44:47.870 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:47.867+0000 7fe445e42640 1 -- 172.21.15.123:0/1330558855 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe4400d1de0 con 0x7fe44015cd20 2024-09-16T15:44:47.870 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:47.867+0000 7fe427fff640 1 -- 172.21.15.123:0/1330558855 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 8) ==== 97284+0+0 (secure 0 0 0) 0x7fe43800bba0 con 0x7fe44015cd20 2024-09-16T15:44:47.870 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:47.867+0000 7fe427fff640 1 --2- 172.21.15.123:0/1330558855 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fe42c073130 0x7fe42c0755f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:44:47.870 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:47.867+0000 7fe427fff640 1 -- 172.21.15.123:0/1330558855 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(46..46 src has 1..46) ==== 5767+0+0 (secure 0 0 0) 0x7fe4380943c0 con 0x7fe44015cd20 2024-09-16T15:44:47.870 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:47.867+0000 7fe43f7fe640 1 --2- 172.21.15.123:0/1330558855 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fe42c073130 0x7fe42c0755f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:44:47.870 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:47.867+0000 7fe43f7fe640 1 --2- 172.21.15.123:0/1330558855 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fe42c073130 0x7fe42c0755f0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fe43000dc70 tx=0x7fe43000d650 comp rx=0 tx=0).ready entity=mgr.4135 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:44:47.873 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:47.871+0000 7fe427fff640 1 -- 172.21.15.123:0/1330558855 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7fe43805f160 con 0x7fe44015cd20 2024-09-16T15:44:48.080 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:48.079+0000 7fe445e42640 1 -- 172.21.15.123:0/1330558855 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe4400b97b0 con 0x7fe44015cd20 2024-09-16T15:44:48.081 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:48.079+0000 7fe427fff640 1 -- 172.21.15.123:0/1330558855 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) ==== 94+0+5223 (secure 0 0 0) 0x7fe43805ea60 con 0x7fe44015cd20 2024-09-16T15:44:48.081 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-09-16T15:44:48.081 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":28,"btime":"2024-09-16T15:44:32:363512+0000","default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4712,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6837/288264117","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6836","nonce":288264117},{"type":"v1","addr":"172.21.15.184:6837","nonce":288264117}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":23},{"gid":5268,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6834/60194515","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6832","nonce":60194515},{"type":"v1","addr":"172.21.15.123:6834","nonce":60194515}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":23},{"gid":5271,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6835/1798373479","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6833","nonce":1798373479},{"type":"v1","addr":"172.21.15.123:6835","nonce":1798373479}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":22}],"filesystems":[{"mdsmap":{"epoch":27,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-16T15:44:14.853230+0000","modified":"2024-09-16T15:44:17.861742+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5280},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5280":{"gid":5280,"name":"d","rank":0,"incarnation":24,"state":"up:active","state_seq":3,"addr":"172.21.15.184:6835/374389412","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":374389412},{"type":"v1","addr":"172.21.15.184:6835","nonce":374389412}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5280,"qdb_cluster":[5280]},"id":3}]} 2024-09-16T15:44:48.082 INFO:teuthology.orchestra.run.smithi123.stderr:dumped fsmap epoch 28 2024-09-16T15:44:48.084 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:48.083+0000 7fe445e42640 1 -- 172.21.15.123:0/1330558855 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fe42c073130 msgr2=0x7fe42c0755f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:44:48.084 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:48.083+0000 7fe445e42640 1 --2- 172.21.15.123:0/1330558855 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fe42c073130 0x7fe42c0755f0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fe43000dc70 tx=0x7fe43000d650 comp rx=0 tx=0).stop 2024-09-16T15:44:48.084 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:48.083+0000 7fe445e42640 1 -- 172.21.15.123:0/1330558855 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe44015cd20 msgr2=0x7fe44015d1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:44:48.084 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:48.083+0000 7fe445e42640 1 --2- 172.21.15.123:0/1330558855 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe44015cd20 0x7fe44015d1a0 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7fe4400d19a0 tx=0x7fe43800db50 comp rx=0 tx=0).stop 2024-09-16T15:44:48.084 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:48.083+0000 7fe445e42640 1 -- 172.21.15.123:0/1330558855 shutdown_connections 2024-09-16T15:44:48.084 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:48.083+0000 7fe445e42640 1 --2- 172.21.15.123:0/1330558855 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fe42c073130 0x7fe42c0755f0 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:48.084 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:48.083+0000 7fe445e42640 1 --2- 172.21.15.123:0/1330558855 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe44015cd20 0x7fe44015d1a0 unknown :-1 s=CLOSED pgs=517 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:48.085 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:48.083+0000 7fe445e42640 1 --2- 172.21.15.123:0/1330558855 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe4400c79e0 0x7fe4400ced90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:48.085 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:48.083+0000 7fe445e42640 1 --2- 172.21.15.123:0/1330558855 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fe428004810 0x7fe4400c8260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:48.085 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:48.083+0000 7fe445e42640 1 -- 172.21.15.123:0/1330558855 >> 172.21.15.123:0/1330558855 conn(0x7fe440069650 msgr2=0x7fe44006ec60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:44:48.085 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:48.083+0000 7fe445e42640 1 -- 172.21.15.123:0/1330558855 shutdown_connections 2024-09-16T15:44:48.085 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:48.083+0000 7fe445e42640 1 -- 172.21.15.123:0/1330558855 wait complete. 2024-09-16T15:44:48.101 DEBUG:teuthology.orchestra.run.smithi184:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.d.asok --format=json session ls 2024-09-16T15:44:48.249 INFO:teuthology.orchestra.run.smithi184.stdout:[{"id":5047,"entity":{"name":{"type":"client","num":5047},"addr":{"type":"any","addr":"172.21.15.184:0","nonce":908570948}},"state":"open","num_leases":1,"num_caps":7,"request_load_avg":14,"uptime":6.5918505569999999,"requests_in_flight":0,"num_completed_requests":2,"num_completed_flushes":1,"reconnecting":false,"recall_caps":{"value":0,"halflife":60},"release_caps":{"value":0,"halflife":60},"recall_caps_throttle":{"value":0,"halflife":1.5},"recall_caps_throttle2o":{"value":0,"halflife":0.5},"session_cache_liveness":{"value":21.668298032262285,"halflife":300},"cap_acquisition":{"value":0,"halflife":30},"last_trim_completed_requests_tid":15,"last_trim_completed_flushes_tid":2,"delegated_inos":[{"start":"0x10000000005","length":500}],"inst":"client.5047 172.21.15.184:0/908570948","completed_requests":[{"tid":15,"created_ino":"0x0"},{"tid":16,"created_ino":"0x100000001f9"}],"prealloc_inos":[{"start":"0x10000000005","length":500},{"start":"0x100000001fa","length":1000}],"client_metadata":{"client_features":{"feature_bits":"0x00000000003fffff"},"metric_spec":{"metric_flags":{"feature_bits":"0x000000000000ffff"}},"ceph_sha1":"33f95d2ae1bcbe7eb04404303f179eeeaa6982df","ceph_version":"ceph version 19.3.0-4938-g33f95d2a (33f95d2ae1bcbe7eb04404303f179eeeaa6982df) squid (dev)","entity_id":"x","hostname":"smithi184","pid":"28840","root":"/"}}] 2024-09-16T15:44:48.250 DEBUG:tasks.cephfs.filesystem:_json_asok output [ { "cap_acquisition": { "halflife": 30, "value": 0 }, "client_metadata": { "ceph_sha1": "33f95d2ae1bcbe7eb04404303f179eeeaa6982df", "ceph_version": "ceph version 19.3.0-4938-g33f95d2a (33f95d2ae1bcbe7eb04404303f179eeeaa6982df) squid (dev)", "client_features": { "feature_bits": "0x00000000003fffff" }, "entity_id": "x", "hostname": "smithi184", "metric_spec": { "metric_flags": { "feature_bits": "0x000000000000ffff" } }, "pid": "28840", "root": "/" }, "completed_requests": [ { "created_ino": "0x0", "tid": 15 }, { "created_ino": "0x100000001f9", "tid": 16 } ], "delegated_inos": [ { "length": 500, "start": "0x10000000005" } ], "entity": { "addr": { "addr": "172.21.15.184:0", "nonce": 908570948, "type": "any" }, "name": { "num": 5047, "type": "client" } }, "id": 5047, "inst": "client.5047 172.21.15.184:0/908570948", "last_trim_completed_flushes_tid": 2, "last_trim_completed_requests_tid": 15, "num_caps": 7, "num_completed_flushes": 1, "num_completed_requests": 2, "num_leases": 1, "prealloc_inos": [ { "length": 500, "start": "0x10000000005" }, { "length": 1000, "start": "0x100000001fa" } ], "recall_caps": { "halflife": 60, "value": 0 }, "recall_caps_throttle": { "halflife": 1.5, "value": 0 }, "recall_caps_throttle2o": { "halflife": 0.5, "value": 0 }, "reconnecting": false, "release_caps": { "halflife": 60, "value": 0 }, "request_load_avg": 14, "requests_in_flight": 0, "session_cache_liveness": { "halflife": 300, "value": 21.668298032262285 }, "state": "open", "uptime": 6.591850557 } ] 2024-09-16T15:44:48.250 DEBUG:tasks.ceph_test_case:wait_until_true: waiting (timeout=90 retry_count=0)... 2024-09-16T15:44:53.251 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-16T15:44:53.384 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:53.383+0000 7ff5891bb640 1 -- 172.21.15.123:0/3497873157 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff5840cddf0 msgr2=0x7ff5840ad8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:44:53.384 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:53.383+0000 7ff5891bb640 1 --2- 172.21.15.123:0/3497873157 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff5840cddf0 0x7ff5840ad8c0 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7ff57c009bc0 tx=0x7ff57c01cba0 comp rx=0 tx=0).stop 2024-09-16T15:44:53.385 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:53.383+0000 7ff5891bb640 1 -- 172.21.15.123:0/3497873157 shutdown_connections 2024-09-16T15:44:53.385 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:53.383+0000 7ff5891bb640 1 --2- 172.21.15.123:0/3497873157 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff5840cddf0 0x7ff5840ad8c0 unknown :-1 s=CLOSED pgs=519 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:53.385 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:53.383+0000 7ff5891bb640 1 --2- 172.21.15.123:0/3497873157 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7ff58415de50 0x7ff584162470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:53.385 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:53.383+0000 7ff5891bb640 1 -- 172.21.15.123:0/3497873157 >> 172.21.15.123:0/3497873157 conn(0x7ff5840cfb10 msgr2=0x7ff5840cff20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:44:53.385 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:53.383+0000 7ff5891bb640 1 -- 172.21.15.123:0/3497873157 shutdown_connections 2024-09-16T15:44:53.385 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:53.383+0000 7ff5891bb640 1 -- 172.21.15.123:0/3497873157 wait complete. 2024-09-16T15:44:53.385 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:53.383+0000 7ff5891bb640 1 Processor -- start 2024-09-16T15:44:53.386 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:53.383+0000 7ff5891bb640 1 -- start start 2024-09-16T15:44:53.387 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:53.383+0000 7ff5891bb640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff5840cddf0 0x7ff5841597c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:44:53.387 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:53.383+0000 7ff5891bb640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7ff58415de50 0x7ff584144ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:44:53.387 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:53.383+0000 7ff5891bb640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff58415e230 0x7ff584160120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:44:53.387 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:53.383+0000 7ff5891bb640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7ff58416af80 con 0x7ff5840cddf0 2024-09-16T15:44:53.387 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:53.383+0000 7ff5891bb640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7ff58416ae00 con 0x7ff58415e230 2024-09-16T15:44:53.388 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:53.383+0000 7ff5891bb640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7ff58416b100 con 0x7ff58415de50 2024-09-16T15:44:53.388 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:53.383+0000 7ff582d9d640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff5840cddf0 0x7ff5841597c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:44:53.388 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:53.383+0000 7ff582d9d640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff5840cddf0 0x7ff5841597c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:41716/0 (socket says 172.21.15.123:41716) 2024-09-16T15:44:53.388 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:53.383+0000 7ff582d9d640 1 -- 172.21.15.123:0/601007700 learned_addr learned my addr 172.21.15.123:0/601007700 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:44:53.388 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:53.383+0000 7ff57bfff640 1 --2- 172.21.15.123:0/601007700 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7ff58415de50 0x7ff584144ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:44:53.388 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:53.383+0000 7ff58359e640 1 --2- 172.21.15.123:0/601007700 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff58415e230 0x7ff584160120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:44:53.388 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:53.383+0000 7ff582d9d640 1 -- 172.21.15.123:0/601007700 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7ff58415de50 msgr2=0x7ff584144ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:44:53.388 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:53.383+0000 7ff582d9d640 1 --2- 172.21.15.123:0/601007700 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7ff58415de50 0x7ff584144ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:53.388 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:53.383+0000 7ff582d9d640 1 -- 172.21.15.123:0/601007700 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff58415e230 msgr2=0x7ff584160120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:44:53.388 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:53.383+0000 7ff582d9d640 1 --2- 172.21.15.123:0/601007700 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff58415e230 0x7ff584160120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:53.388 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:53.383+0000 7ff582d9d640 1 -- 172.21.15.123:0/601007700 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff584154150 con 0x7ff5840cddf0 2024-09-16T15:44:53.388 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:53.383+0000 7ff582d9d640 1 --2- 172.21.15.123:0/601007700 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff5840cddf0 0x7ff5841597c0 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7ff57400d950 tx=0x7ff57400de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:44:53.388 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:53.383+0000 7ff57b7fe640 1 -- 172.21.15.123:0/601007700 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff574017070 con 0x7ff5840cddf0 2024-09-16T15:44:53.388 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:53.383+0000 7ff57b7fe640 1 -- 172.21.15.123:0/601007700 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7ff574009d20 con 0x7ff5840cddf0 2024-09-16T15:44:53.389 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:53.383+0000 7ff5891bb640 1 -- 172.21.15.123:0/601007700 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff584154430 con 0x7ff5840cddf0 2024-09-16T15:44:53.389 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:53.387+0000 7ff57b7fe640 1 -- 172.21.15.123:0/601007700 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5740106c0 con 0x7ff5840cddf0 2024-09-16T15:44:53.389 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:53.387+0000 7ff5891bb640 1 -- 172.21.15.123:0/601007700 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff58415f920 con 0x7ff5840cddf0 2024-09-16T15:44:53.389 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:53.387+0000 7ff5891bb640 1 -- 172.21.15.123:0/601007700 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff5840ad8c0 con 0x7ff5840cddf0 2024-09-16T15:44:53.389 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:53.387+0000 7ff57b7fe640 1 -- 172.21.15.123:0/601007700 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 8) ==== 97284+0+0 (secure 0 0 0) 0x7ff574010860 con 0x7ff5840cddf0 2024-09-16T15:44:53.390 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:53.387+0000 7ff57b7fe640 1 --2- 172.21.15.123:0/601007700 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7ff564073100 0x7ff5640755c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:44:53.390 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:53.387+0000 7ff57b7fe640 1 -- 172.21.15.123:0/601007700 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(46..46 src has 1..46) ==== 5767+0+0 (secure 0 0 0) 0x7ff574094790 con 0x7ff5840cddf0 2024-09-16T15:44:53.390 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:53.387+0000 7ff57bfff640 1 --2- 172.21.15.123:0/601007700 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7ff564073100 0x7ff5640755c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:44:53.390 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:53.387+0000 7ff57bfff640 1 --2- 172.21.15.123:0/601007700 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7ff564073100 0x7ff5640755c0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7ff58415d6a0 tx=0x7ff56c009040 comp rx=0 tx=0).ready entity=mgr.4135 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:44:53.393 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:53.391+0000 7ff57b7fe640 1 -- 172.21.15.123:0/601007700 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7ff57405f530 con 0x7ff5840cddf0 2024-09-16T15:44:53.609 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:53.607+0000 7ff5891bb640 1 -- 172.21.15.123:0/601007700 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff5840d04b0 con 0x7ff5840cddf0 2024-09-16T15:44:53.610 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:53.607+0000 7ff57b7fe640 1 -- 172.21.15.123:0/601007700 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) ==== 94+0+5223 (secure 0 0 0) 0x7ff57405ee30 con 0x7ff5840cddf0 2024-09-16T15:44:53.611 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-09-16T15:44:53.611 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":28,"btime":"2024-09-16T15:44:32:363512+0000","default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4712,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6837/288264117","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6836","nonce":288264117},{"type":"v1","addr":"172.21.15.184:6837","nonce":288264117}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":23},{"gid":5268,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6834/60194515","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6832","nonce":60194515},{"type":"v1","addr":"172.21.15.123:6834","nonce":60194515}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":23},{"gid":5271,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6835/1798373479","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6833","nonce":1798373479},{"type":"v1","addr":"172.21.15.123:6835","nonce":1798373479}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":22}],"filesystems":[{"mdsmap":{"epoch":27,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-16T15:44:14.853230+0000","modified":"2024-09-16T15:44:17.861742+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5280},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5280":{"gid":5280,"name":"d","rank":0,"incarnation":24,"state":"up:active","state_seq":3,"addr":"172.21.15.184:6835/374389412","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":374389412},{"type":"v1","addr":"172.21.15.184:6835","nonce":374389412}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5280,"qdb_cluster":[5280]},"id":3}]} 2024-09-16T15:44:53.611 INFO:teuthology.orchestra.run.smithi123.stderr:dumped fsmap epoch 28 2024-09-16T15:44:53.613 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:53.611+0000 7ff5891bb640 1 -- 172.21.15.123:0/601007700 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7ff564073100 msgr2=0x7ff5640755c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:44:53.613 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:53.611+0000 7ff5891bb640 1 --2- 172.21.15.123:0/601007700 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7ff564073100 0x7ff5640755c0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7ff58415d6a0 tx=0x7ff56c009040 comp rx=0 tx=0).stop 2024-09-16T15:44:53.613 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:53.611+0000 7ff5891bb640 1 -- 172.21.15.123:0/601007700 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff5840cddf0 msgr2=0x7ff5841597c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:44:53.613 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:53.611+0000 7ff5891bb640 1 --2- 172.21.15.123:0/601007700 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff5840cddf0 0x7ff5841597c0 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7ff57400d950 tx=0x7ff57400de10 comp rx=0 tx=0).stop 2024-09-16T15:44:53.613 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:53.611+0000 7ff5891bb640 1 -- 172.21.15.123:0/601007700 shutdown_connections 2024-09-16T15:44:53.613 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:53.611+0000 7ff5891bb640 1 --2- 172.21.15.123:0/601007700 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7ff564073100 0x7ff5640755c0 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:53.613 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:53.611+0000 7ff5891bb640 1 --2- 172.21.15.123:0/601007700 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff58415e230 0x7ff584160120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:53.613 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:53.611+0000 7ff5891bb640 1 --2- 172.21.15.123:0/601007700 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7ff58415de50 0x7ff584144ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:53.614 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:53.611+0000 7ff5891bb640 1 --2- 172.21.15.123:0/601007700 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff5840cddf0 0x7ff5841597c0 unknown :-1 s=CLOSED pgs=520 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:53.614 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:53.611+0000 7ff5891bb640 1 -- 172.21.15.123:0/601007700 >> 172.21.15.123:0/601007700 conn(0x7ff5840cfb10 msgr2=0x7ff584073dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:44:53.614 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:53.611+0000 7ff5891bb640 1 -- 172.21.15.123:0/601007700 shutdown_connections 2024-09-16T15:44:53.614 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:53.611+0000 7ff5891bb640 1 -- 172.21.15.123:0/601007700 wait complete. 2024-09-16T15:44:53.630 DEBUG:teuthology.orchestra.run.smithi184:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.d.asok --format=json session ls 2024-09-16T15:44:53.779 INFO:teuthology.orchestra.run.smithi184.stdout:[{"id":5047,"entity":{"name":{"type":"client","num":5047},"addr":{"type":"any","addr":"172.21.15.184:0","nonce":908570948}},"state":"open","num_leases":1,"num_caps":7,"request_load_avg":13,"uptime":12.119738352000001,"requests_in_flight":0,"num_completed_requests":2,"num_completed_flushes":1,"reconnecting":false,"recall_caps":{"value":0,"halflife":60},"release_caps":{"value":0,"halflife":60},"recall_caps_throttle":{"value":0,"halflife":1.5},"recall_caps_throttle2o":{"value":0,"halflife":0.5},"session_cache_liveness":{"value":21.393307498282351,"halflife":300},"cap_acquisition":{"value":0,"halflife":30},"last_trim_completed_requests_tid":15,"last_trim_completed_flushes_tid":2,"delegated_inos":[{"start":"0x10000000005","length":500}],"inst":"client.5047 172.21.15.184:0/908570948","completed_requests":[{"tid":15,"created_ino":"0x0"},{"tid":16,"created_ino":"0x100000001f9"}],"prealloc_inos":[{"start":"0x10000000005","length":500},{"start":"0x100000001fa","length":1000}],"client_metadata":{"client_features":{"feature_bits":"0x00000000003fffff"},"metric_spec":{"metric_flags":{"feature_bits":"0x000000000000ffff"}},"ceph_sha1":"33f95d2ae1bcbe7eb04404303f179eeeaa6982df","ceph_version":"ceph version 19.3.0-4938-g33f95d2a (33f95d2ae1bcbe7eb04404303f179eeeaa6982df) squid (dev)","entity_id":"x","hostname":"smithi184","pid":"28840","root":"/"}}] 2024-09-16T15:44:53.779 DEBUG:tasks.cephfs.filesystem:_json_asok output [ { "cap_acquisition": { "halflife": 30, "value": 0 }, "client_metadata": { "ceph_sha1": "33f95d2ae1bcbe7eb04404303f179eeeaa6982df", "ceph_version": "ceph version 19.3.0-4938-g33f95d2a (33f95d2ae1bcbe7eb04404303f179eeeaa6982df) squid (dev)", "client_features": { "feature_bits": "0x00000000003fffff" }, "entity_id": "x", "hostname": "smithi184", "metric_spec": { "metric_flags": { "feature_bits": "0x000000000000ffff" } }, "pid": "28840", "root": "/" }, "completed_requests": [ { "created_ino": "0x0", "tid": 15 }, { "created_ino": "0x100000001f9", "tid": 16 } ], "delegated_inos": [ { "length": 500, "start": "0x10000000005" } ], "entity": { "addr": { "addr": "172.21.15.184:0", "nonce": 908570948, "type": "any" }, "name": { "num": 5047, "type": "client" } }, "id": 5047, "inst": "client.5047 172.21.15.184:0/908570948", "last_trim_completed_flushes_tid": 2, "last_trim_completed_requests_tid": 15, "num_caps": 7, "num_completed_flushes": 1, "num_completed_requests": 2, "num_leases": 1, "prealloc_inos": [ { "length": 500, "start": "0x10000000005" }, { "length": 1000, "start": "0x100000001fa" } ], "recall_caps": { "halflife": 60, "value": 0 }, "recall_caps_throttle": { "halflife": 1.5, "value": 0 }, "recall_caps_throttle2o": { "halflife": 0.5, "value": 0 }, "reconnecting": false, "release_caps": { "halflife": 60, "value": 0 }, "request_load_avg": 13, "requests_in_flight": 0, "session_cache_liveness": { "halflife": 300, "value": 21.39330749828235 }, "state": "open", "uptime": 12.119738352 } ] 2024-09-16T15:44:53.779 DEBUG:tasks.ceph_test_case:wait_until_true: waiting (timeout=90 retry_count=0)... 2024-09-16T15:44:58.781 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-16T15:44:58.917 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:58.915+0000 7f5b0a9d6640 1 -- 172.21.15.123:0/411529045 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5b040b2b00 msgr2=0x7f5b040d22e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:44:58.917 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:58.915+0000 7f5b0a9d6640 1 --2- 172.21.15.123:0/411529045 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5b040b2b00 0x7f5b040d22e0 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7f5b0406f930 tx=0x7f5af401c7a0 comp rx=0 tx=0).stop 2024-09-16T15:44:58.917 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:58.915+0000 7f5b0a9d6640 1 -- 172.21.15.123:0/411529045 shutdown_connections 2024-09-16T15:44:58.917 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:58.915+0000 7f5b0a9d6640 1 --2- 172.21.15.123:0/411529045 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5b040b2b00 0x7f5b040d22e0 unknown :-1 s=CLOSED pgs=521 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:58.918 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:58.915+0000 7f5b0a9d6640 1 --2- 172.21.15.123:0/411529045 >> v2:172.21.15.184:3300/0 conn(0x7f5b04161620 0x7f5b040d1da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:58.918 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:58.915+0000 7f5b0a9d6640 1 -- 172.21.15.123:0/411529045 >> 172.21.15.123:0/411529045 conn(0x7f5b040d3ea0 msgr2=0x7f5b040d42b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:44:58.918 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:58.915+0000 7f5b0a9d6640 1 -- 172.21.15.123:0/411529045 shutdown_connections 2024-09-16T15:44:58.918 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:58.915+0000 7f5b0a9d6640 1 -- 172.21.15.123:0/411529045 wait complete. 2024-09-16T15:44:58.920 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:58.915+0000 7f5b0a9d6640 1 Processor -- start 2024-09-16T15:44:58.920 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:58.915+0000 7f5b0a9d6640 1 -- start start 2024-09-16T15:44:58.921 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:58.915+0000 7f5b0a9d6640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5b040d2820 0x7f5b04149f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:44:58.921 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:58.915+0000 7f5b0a9d6640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f5b04161620 0x7f5b0406e980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:44:58.921 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:58.915+0000 7f5b0a9d6640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5b040b3940 0x7f5b040b3dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:44:58.921 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:58.915+0000 7f5b0a9d6640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f5b0416aee0 con 0x7f5b040d2820 2024-09-16T15:44:58.921 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:58.915+0000 7f5b0a9d6640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f5b0416ad60 con 0x7f5b040b3940 2024-09-16T15:44:58.921 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:58.915+0000 7f5b0a9d6640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f5b0416b060 con 0x7f5b04161620 2024-09-16T15:44:58.921 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:58.915+0000 7f5b03fff640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5b040d2820 0x7f5b04149f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:44:58.921 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:58.919+0000 7f5b03fff640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5b040d2820 0x7f5b04149f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:41740/0 (socket says 172.21.15.123:41740) 2024-09-16T15:44:58.921 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:58.919+0000 7f5b03fff640 1 -- 172.21.15.123:0/131779951 learned_addr learned my addr 172.21.15.123:0/131779951 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:44:58.921 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:58.919+0000 7f5b037fe640 1 --2- 172.21.15.123:0/131779951 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f5b04161620 0x7f5b0406e980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:44:58.921 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:58.919+0000 7f5b08f73640 1 --2- 172.21.15.123:0/131779951 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5b040b3940 0x7f5b040b3dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:44:58.921 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:58.919+0000 7f5b03fff640 1 -- 172.21.15.123:0/131779951 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f5b04161620 msgr2=0x7f5b0406e980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:44:58.921 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:58.919+0000 7f5b03fff640 1 --2- 172.21.15.123:0/131779951 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f5b04161620 0x7f5b0406e980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:58.921 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:58.919+0000 7f5b03fff640 1 -- 172.21.15.123:0/131779951 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5b040b3940 msgr2=0x7f5b040b3dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:44:58.922 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:58.919+0000 7f5b03fff640 1 --2- 172.21.15.123:0/131779951 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5b040b3940 0x7f5b040b3dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:58.922 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:58.919+0000 7f5b03fff640 1 -- 172.21.15.123:0/131779951 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5b040c7cd0 con 0x7f5b040d2820 2024-09-16T15:44:58.922 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:58.919+0000 7f5b03fff640 1 --2- 172.21.15.123:0/131779951 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5b040d2820 0x7f5b04149f30 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7f5af800c970 tx=0x7f5af800ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:44:58.922 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:58.919+0000 7f5b00ff9640 1 -- 172.21.15.123:0/131779951 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5af8007bb0 con 0x7f5b040d2820 2024-09-16T15:44:58.922 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:58.919+0000 7f5b00ff9640 1 -- 172.21.15.123:0/131779951 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f5af8007e40 con 0x7f5b040d2820 2024-09-16T15:44:58.922 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:58.919+0000 7f5b0a9d6640 1 -- 172.21.15.123:0/131779951 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5b0415c130 con 0x7f5b040d2820 2024-09-16T15:44:58.922 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:58.919+0000 7f5b00ff9640 1 -- 172.21.15.123:0/131779951 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5af8005290 con 0x7f5b040d2820 2024-09-16T15:44:58.922 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:58.919+0000 7f5b0a9d6640 1 -- 172.21.15.123:0/131779951 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5b041f8700 con 0x7f5b040d2820 2024-09-16T15:44:58.922 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:58.919+0000 7f5b0a9d6640 1 -- 172.21.15.123:0/131779951 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5b040cea00 con 0x7f5b040d2820 2024-09-16T15:44:58.923 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:58.919+0000 7f5b00ff9640 1 -- 172.21.15.123:0/131779951 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 8) ==== 97284+0+0 (secure 0 0 0) 0x7f5af8005430 con 0x7f5b040d2820 2024-09-16T15:44:58.923 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:58.919+0000 7f5b00ff9640 1 --2- 172.21.15.123:0/131779951 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f5af0073100 0x7f5af00755c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:44:58.923 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:58.919+0000 7f5b00ff9640 1 -- 172.21.15.123:0/131779951 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(46..46 src has 1..46) ==== 5767+0+0 (secure 0 0 0) 0x7f5af80947e0 con 0x7f5b040d2820 2024-09-16T15:44:58.923 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:58.919+0000 7f5b037fe640 1 --2- 172.21.15.123:0/131779951 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f5af0073100 0x7f5af00755c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:44:58.923 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:58.923+0000 7f5b037fe640 1 --2- 172.21.15.123:0/131779951 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f5af0073100 0x7f5af00755c0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f5af40af000 tx=0x7f5af401afe0 comp rx=0 tx=0).ready entity=mgr.4135 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:44:58.926 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:58.923+0000 7f5b00ff9640 1 -- 172.21.15.123:0/131779951 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f5af8014070 con 0x7f5b040d2820 2024-09-16T15:44:59.136 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:59.135+0000 7f5b0a9d6640 1 -- 172.21.15.123:0/131779951 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f5b0415f9f0 con 0x7f5b040d2820 2024-09-16T15:44:59.137 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:59.135+0000 7f5b00ff9640 1 -- 172.21.15.123:0/131779951 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) ==== 94+0+5223 (secure 0 0 0) 0x7f5af805f610 con 0x7f5b040d2820 2024-09-16T15:44:59.137 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-09-16T15:44:59.138 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":28,"btime":"2024-09-16T15:44:32:363512+0000","default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4712,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6837/288264117","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6836","nonce":288264117},{"type":"v1","addr":"172.21.15.184:6837","nonce":288264117}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":23},{"gid":5268,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6834/60194515","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6832","nonce":60194515},{"type":"v1","addr":"172.21.15.123:6834","nonce":60194515}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":23},{"gid":5271,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6835/1798373479","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6833","nonce":1798373479},{"type":"v1","addr":"172.21.15.123:6835","nonce":1798373479}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":22}],"filesystems":[{"mdsmap":{"epoch":27,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-16T15:44:14.853230+0000","modified":"2024-09-16T15:44:17.861742+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5280},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5280":{"gid":5280,"name":"d","rank":0,"incarnation":24,"state":"up:active","state_seq":3,"addr":"172.21.15.184:6835/374389412","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":374389412},{"type":"v1","addr":"172.21.15.184:6835","nonce":374389412}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5280,"qdb_cluster":[5280]},"id":3}]} 2024-09-16T15:44:59.138 INFO:teuthology.orchestra.run.smithi123.stderr:dumped fsmap epoch 28 2024-09-16T15:44:59.140 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:59.139+0000 7f5b0a9d6640 1 -- 172.21.15.123:0/131779951 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f5af0073100 msgr2=0x7f5af00755c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:44:59.140 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:59.139+0000 7f5b0a9d6640 1 --2- 172.21.15.123:0/131779951 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f5af0073100 0x7f5af00755c0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f5af40af000 tx=0x7f5af401afe0 comp rx=0 tx=0).stop 2024-09-16T15:44:59.140 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:59.139+0000 7f5b0a9d6640 1 -- 172.21.15.123:0/131779951 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5b040d2820 msgr2=0x7f5b04149f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:44:59.140 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:59.139+0000 7f5b0a9d6640 1 --2- 172.21.15.123:0/131779951 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5b040d2820 0x7f5b04149f30 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7f5af800c970 tx=0x7f5af800ce30 comp rx=0 tx=0).stop 2024-09-16T15:44:59.140 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:59.139+0000 7f5b0a9d6640 1 -- 172.21.15.123:0/131779951 shutdown_connections 2024-09-16T15:44:59.140 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:59.139+0000 7f5b0a9d6640 1 --2- 172.21.15.123:0/131779951 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f5af0073100 0x7f5af00755c0 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:59.140 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:59.139+0000 7f5b0a9d6640 1 --2- 172.21.15.123:0/131779951 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5b040b3940 0x7f5b040b3dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:59.140 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:59.139+0000 7f5b0a9d6640 1 --2- 172.21.15.123:0/131779951 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f5b04161620 0x7f5b0406e980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:59.141 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:59.139+0000 7f5b0a9d6640 1 --2- 172.21.15.123:0/131779951 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5b040d2820 0x7f5b04149f30 unknown :-1 s=CLOSED pgs=522 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:44:59.141 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:59.139+0000 7f5b0a9d6640 1 -- 172.21.15.123:0/131779951 >> 172.21.15.123:0/131779951 conn(0x7f5b040d3ea0 msgr2=0x7f5b0415d800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:44:59.141 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:59.139+0000 7f5b0a9d6640 1 -- 172.21.15.123:0/131779951 shutdown_connections 2024-09-16T15:44:59.141 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:44:59.139+0000 7f5b0a9d6640 1 -- 172.21.15.123:0/131779951 wait complete. 2024-09-16T15:44:59.157 DEBUG:teuthology.orchestra.run.smithi184:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.d.asok --format=json session ls 2024-09-16T15:44:59.308 INFO:teuthology.orchestra.run.smithi184.stdout:[{"id":5047,"entity":{"name":{"type":"client","num":5047},"addr":{"type":"any","addr":"172.21.15.184:0","nonce":908570948}},"state":"open","num_leases":1,"num_caps":7,"request_load_avg":13,"uptime":17.64763627,"requests_in_flight":0,"num_completed_requests":2,"num_completed_flushes":1,"reconnecting":false,"recall_caps":{"value":0,"halflife":60},"release_caps":{"value":0,"halflife":60},"recall_caps_throttle":{"value":0,"halflife":1.5},"recall_caps_throttle2o":{"value":0,"halflife":0.5},"session_cache_liveness":{"value":21.121806351844253,"halflife":300},"cap_acquisition":{"value":0,"halflife":30},"last_trim_completed_requests_tid":15,"last_trim_completed_flushes_tid":2,"delegated_inos":[{"start":"0x10000000005","length":500}],"inst":"client.5047 172.21.15.184:0/908570948","completed_requests":[{"tid":15,"created_ino":"0x0"},{"tid":16,"created_ino":"0x100000001f9"}],"prealloc_inos":[{"start":"0x10000000005","length":500},{"start":"0x100000001fa","length":1000}],"client_metadata":{"client_features":{"feature_bits":"0x00000000003fffff"},"metric_spec":{"metric_flags":{"feature_bits":"0x000000000000ffff"}},"ceph_sha1":"33f95d2ae1bcbe7eb04404303f179eeeaa6982df","ceph_version":"ceph version 19.3.0-4938-g33f95d2a (33f95d2ae1bcbe7eb04404303f179eeeaa6982df) squid (dev)","entity_id":"x","hostname":"smithi184","pid":"28840","root":"/"}}] 2024-09-16T15:44:59.309 DEBUG:tasks.cephfs.filesystem:_json_asok output [ { "cap_acquisition": { "halflife": 30, "value": 0 }, "client_metadata": { "ceph_sha1": "33f95d2ae1bcbe7eb04404303f179eeeaa6982df", "ceph_version": "ceph version 19.3.0-4938-g33f95d2a (33f95d2ae1bcbe7eb04404303f179eeeaa6982df) squid (dev)", "client_features": { "feature_bits": "0x00000000003fffff" }, "entity_id": "x", "hostname": "smithi184", "metric_spec": { "metric_flags": { "feature_bits": "0x000000000000ffff" } }, "pid": "28840", "root": "/" }, "completed_requests": [ { "created_ino": "0x0", "tid": 15 }, { "created_ino": "0x100000001f9", "tid": 16 } ], "delegated_inos": [ { "length": 500, "start": "0x10000000005" } ], "entity": { "addr": { "addr": "172.21.15.184:0", "nonce": 908570948, "type": "any" }, "name": { "num": 5047, "type": "client" } }, "id": 5047, "inst": "client.5047 172.21.15.184:0/908570948", "last_trim_completed_flushes_tid": 2, "last_trim_completed_requests_tid": 15, "num_caps": 7, "num_completed_flushes": 1, "num_completed_requests": 2, "num_leases": 1, "prealloc_inos": [ { "length": 500, "start": "0x10000000005" }, { "length": 1000, "start": "0x100000001fa" } ], "recall_caps": { "halflife": 60, "value": 0 }, "recall_caps_throttle": { "halflife": 1.5, "value": 0 }, "recall_caps_throttle2o": { "halflife": 0.5, "value": 0 }, "reconnecting": false, "release_caps": { "halflife": 60, "value": 0 }, "request_load_avg": 13, "requests_in_flight": 0, "session_cache_liveness": { "halflife": 300, "value": 21.121806351844253 }, "state": "open", "uptime": 17.64763627 } ] 2024-09-16T15:44:59.309 DEBUG:tasks.ceph_test_case:wait_until_true: waiting (timeout=90 retry_count=0)... 2024-09-16T15:45:04.310 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-16T15:45:04.445 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:04.442+0000 7fb305fe6640 1 -- 172.21.15.123:0/2234564316 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb30015bca0 msgr2=0x7fb3000d0630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:45:04.445 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:04.442+0000 7fb305fe6640 1 --2- 172.21.15.123:0/2234564316 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb30015bca0 0x7fb3000d0630 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7fb300076230 tx=0x7fb2f401c540 comp rx=0 tx=0).stop 2024-09-16T15:45:04.445 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:04.442+0000 7fb305fe6640 1 -- 172.21.15.123:0/2234564316 shutdown_connections 2024-09-16T15:45:04.445 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:04.442+0000 7fb305fe6640 1 --2- 172.21.15.123:0/2234564316 >> v2:172.21.15.184:3300/0 conn(0x7fb300074430 0x7fb3000ca160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:45:04.445 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:04.442+0000 7fb305fe6640 1 --2- 172.21.15.123:0/2234564316 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fb3000726c0 0x7fb300073ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:45:04.445 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:04.442+0000 7fb305fe6640 1 --2- 172.21.15.123:0/2234564316 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb30015bca0 0x7fb3000d0630 unknown :-1 s=CLOSED pgs=523 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:45:04.445 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:04.442+0000 7fb305fe6640 1 -- 172.21.15.123:0/2234564316 >> 172.21.15.123:0/2234564316 conn(0x7fb3000cfb10 msgr2=0x7fb3000cff20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:45:04.445 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:04.442+0000 7fb305fe6640 1 -- 172.21.15.123:0/2234564316 shutdown_connections 2024-09-16T15:45:04.446 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:04.442+0000 7fb305fe6640 1 -- 172.21.15.123:0/2234564316 wait complete. 2024-09-16T15:45:04.448 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:04.442+0000 7fb305fe6640 1 Processor -- start 2024-09-16T15:45:04.448 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:04.442+0000 7fb305fe6640 1 -- start start 2024-09-16T15:45:04.448 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:04.442+0000 7fb305fe6640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb3000726c0 0x7fb300150560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:45:04.449 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:04.442+0000 7fb305fe6640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fb300074430 0x7fb3000c87a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:45:04.449 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:04.442+0000 7fb305fe6640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb30015bca0 0x7fb3001558d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:45:04.449 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:04.442+0000 7fb305fe6640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fb30006ec20 con 0x7fb3000726c0 2024-09-16T15:45:04.449 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:04.442+0000 7fb305fe6640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fb30006eaa0 con 0x7fb30015bca0 2024-09-16T15:45:04.449 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:04.442+0000 7fb305fe6640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fb30006eda0 con 0x7fb300074430 2024-09-16T15:45:04.449 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:04.442+0000 7fb2ff7fe640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb3000726c0 0x7fb300150560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:45:04.449 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:04.442+0000 7fb2ff7fe640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb3000726c0 0x7fb300150560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:35588/0 (socket says 172.21.15.123:35588) 2024-09-16T15:45:04.449 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:04.442+0000 7fb2ff7fe640 1 -- 172.21.15.123:0/812241739 learned_addr learned my addr 172.21.15.123:0/812241739 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:45:04.449 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:04.442+0000 7fb2feffd640 1 --2- 172.21.15.123:0/812241739 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fb300074430 0x7fb3000c87a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:45:04.449 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:04.442+0000 7fb2ff7fe640 1 -- 172.21.15.123:0/812241739 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fb300074430 msgr2=0x7fb3000c87a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:45:04.449 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:04.442+0000 7fb2ff7fe640 1 --2- 172.21.15.123:0/812241739 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fb300074430 0x7fb3000c87a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:45:04.450 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:04.442+0000 7fb2ff7fe640 1 -- 172.21.15.123:0/812241739 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb30015bca0 msgr2=0x7fb3001558d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:45:04.450 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:04.442+0000 7fb2fffff640 1 --2- 172.21.15.123:0/812241739 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb30015bca0 0x7fb3001558d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:45:04.450 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:04.442+0000 7fb2ff7fe640 1 --2- 172.21.15.123:0/812241739 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb30015bca0 0x7fb3001558d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:45:04.450 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:04.442+0000 7fb2ff7fe640 1 -- 172.21.15.123:0/812241739 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb3000c8ce0 con 0x7fb3000726c0 2024-09-16T15:45:04.450 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:04.442+0000 7fb2feffd640 1 --2- 172.21.15.123:0/812241739 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fb300074430 0x7fb3000c87a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T15:45:04.450 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:04.442+0000 7fb2fffff640 1 --2- 172.21.15.123:0/812241739 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb30015bca0 0x7fb3001558d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T15:45:04.450 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:04.442+0000 7fb2ff7fe640 1 --2- 172.21.15.123:0/812241739 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb3000726c0 0x7fb300150560 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7fb2f401ca20 tx=0x7fb2f40a86a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:45:04.450 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:04.446+0000 7fb2e7fff640 1 -- 172.21.15.123:0/812241739 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2f4004340 con 0x7fb3000726c0 2024-09-16T15:45:04.450 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:04.446+0000 7fb2e7fff640 1 -- 172.21.15.123:0/812241739 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fb2f4005030 con 0x7fb3000726c0 2024-09-16T15:45:04.450 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:04.446+0000 7fb305fe6640 1 -- 172.21.15.123:0/812241739 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb3000c8f80 con 0x7fb3000726c0 2024-09-16T15:45:04.451 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:04.446+0000 7fb2e7fff640 1 -- 172.21.15.123:0/812241739 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2f4005480 con 0x7fb3000726c0 2024-09-16T15:45:04.451 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:04.446+0000 7fb305fe6640 1 -- 172.21.15.123:0/812241739 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb3001550d0 con 0x7fb3000726c0 2024-09-16T15:45:04.451 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:04.446+0000 7fb305fe6640 1 -- 172.21.15.123:0/812241739 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb3000ca160 con 0x7fb3000726c0 2024-09-16T15:45:04.451 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:04.446+0000 7fb2e7fff640 1 -- 172.21.15.123:0/812241739 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 8) ==== 97284+0+0 (secure 0 0 0) 0x7fb2f40046d0 con 0x7fb3000726c0 2024-09-16T15:45:04.451 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:04.446+0000 7fb2e7fff640 1 --2- 172.21.15.123:0/812241739 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fb2ec073100 0x7fb2ec0755c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:45:04.452 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:04.446+0000 7fb2e7fff640 1 -- 172.21.15.123:0/812241739 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(46..46 src has 1..46) ==== 5767+0+0 (secure 0 0 0) 0x7fb2f412eba0 con 0x7fb3000726c0 2024-09-16T15:45:04.452 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:04.446+0000 7fb2feffd640 1 --2- 172.21.15.123:0/812241739 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fb2ec073100 0x7fb2ec0755c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:45:04.452 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:04.446+0000 7fb2feffd640 1 --2- 172.21.15.123:0/812241739 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fb2ec073100 0x7fb2ec0755c0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fb2f0005950 tx=0x7fb2f0005ca0 comp rx=0 tx=0).ready entity=mgr.4135 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:45:04.453 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:04.450+0000 7fb2e7fff640 1 -- 172.21.15.123:0/812241739 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7fb2f40ac030 con 0x7fb3000726c0 2024-09-16T15:45:04.656 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:04.654+0000 7fb305fe6640 1 -- 172.21.15.123:0/812241739 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb30015f630 con 0x7fb3000726c0 2024-09-16T15:45:04.657 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:04.654+0000 7fb2e7fff640 1 -- 172.21.15.123:0/812241739 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) ==== 94+0+5223 (secure 0 0 0) 0x7fb2f40164b0 con 0x7fb3000726c0 2024-09-16T15:45:04.658 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-09-16T15:45:04.658 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":28,"btime":"2024-09-16T15:44:32:363512+0000","default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4712,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6837/288264117","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6836","nonce":288264117},{"type":"v1","addr":"172.21.15.184:6837","nonce":288264117}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":23},{"gid":5268,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6834/60194515","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6832","nonce":60194515},{"type":"v1","addr":"172.21.15.123:6834","nonce":60194515}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":23},{"gid":5271,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6835/1798373479","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6833","nonce":1798373479},{"type":"v1","addr":"172.21.15.123:6835","nonce":1798373479}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":22}],"filesystems":[{"mdsmap":{"epoch":27,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-16T15:44:14.853230+0000","modified":"2024-09-16T15:44:17.861742+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5280},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5280":{"gid":5280,"name":"d","rank":0,"incarnation":24,"state":"up:active","state_seq":3,"addr":"172.21.15.184:6835/374389412","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":374389412},{"type":"v1","addr":"172.21.15.184:6835","nonce":374389412}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5280,"qdb_cluster":[5280]},"id":3}]} 2024-09-16T15:45:04.658 INFO:teuthology.orchestra.run.smithi123.stderr:dumped fsmap epoch 28 2024-09-16T15:45:04.659 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:04.658+0000 7fb305fe6640 1 -- 172.21.15.123:0/812241739 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fb2ec073100 msgr2=0x7fb2ec0755c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:45:04.660 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:04.658+0000 7fb305fe6640 1 --2- 172.21.15.123:0/812241739 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fb2ec073100 0x7fb2ec0755c0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fb2f0005950 tx=0x7fb2f0005ca0 comp rx=0 tx=0).stop 2024-09-16T15:45:04.660 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:04.658+0000 7fb305fe6640 1 -- 172.21.15.123:0/812241739 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb3000726c0 msgr2=0x7fb300150560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:45:04.660 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:04.658+0000 7fb305fe6640 1 --2- 172.21.15.123:0/812241739 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb3000726c0 0x7fb300150560 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7fb2f401ca20 tx=0x7fb2f40a86a0 comp rx=0 tx=0).stop 2024-09-16T15:45:04.660 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:04.658+0000 7fb305fe6640 1 -- 172.21.15.123:0/812241739 shutdown_connections 2024-09-16T15:45:04.660 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:04.658+0000 7fb305fe6640 1 --2- 172.21.15.123:0/812241739 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fb2ec073100 0x7fb2ec0755c0 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:45:04.660 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:04.658+0000 7fb305fe6640 1 --2- 172.21.15.123:0/812241739 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb30015bca0 0x7fb3001558d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:45:04.660 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:04.658+0000 7fb305fe6640 1 --2- 172.21.15.123:0/812241739 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fb300074430 0x7fb3000c87a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:45:04.660 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:04.658+0000 7fb305fe6640 1 --2- 172.21.15.123:0/812241739 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb3000726c0 0x7fb300150560 unknown :-1 s=CLOSED pgs=524 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:45:04.660 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:04.658+0000 7fb305fe6640 1 -- 172.21.15.123:0/812241739 >> 172.21.15.123:0/812241739 conn(0x7fb3000cfb10 msgr2=0x7fb3000b51d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:45:04.660 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:04.658+0000 7fb305fe6640 1 -- 172.21.15.123:0/812241739 shutdown_connections 2024-09-16T15:45:04.660 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:04.658+0000 7fb305fe6640 1 -- 172.21.15.123:0/812241739 wait complete. 2024-09-16T15:45:04.675 DEBUG:teuthology.orchestra.run.smithi184:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.d.asok --format=json session ls 2024-09-16T15:45:04.826 INFO:teuthology.orchestra.run.smithi184.stdout:[{"id":5047,"entity":{"name":{"type":"client","num":5047},"addr":{"type":"any","addr":"172.21.15.184:0","nonce":908570948}},"state":"open","num_leases":1,"num_caps":7,"request_load_avg":12,"uptime":23.167542780000002,"requests_in_flight":0,"num_completed_requests":2,"num_completed_flushes":1,"reconnecting":false,"recall_caps":{"value":0,"halflife":60},"release_caps":{"value":0,"halflife":60},"recall_caps_throttle":{"value":0,"halflife":1.5},"recall_caps_throttle2o":{"value":0,"halflife":0.5},"session_cache_liveness":{"value":20.854135858449677,"halflife":300},"cap_acquisition":{"value":0,"halflife":30},"last_trim_completed_requests_tid":15,"last_trim_completed_flushes_tid":2,"delegated_inos":[{"start":"0x10000000005","length":500}],"inst":"client.5047 172.21.15.184:0/908570948","completed_requests":[{"tid":15,"created_ino":"0x0"},{"tid":16,"created_ino":"0x100000001f9"}],"prealloc_inos":[{"start":"0x10000000005","length":500},{"start":"0x100000001fa","length":1000}],"client_metadata":{"client_features":{"feature_bits":"0x00000000003fffff"},"metric_spec":{"metric_flags":{"feature_bits":"0x000000000000ffff"}},"ceph_sha1":"33f95d2ae1bcbe7eb04404303f179eeeaa6982df","ceph_version":"ceph version 19.3.0-4938-g33f95d2a (33f95d2ae1bcbe7eb04404303f179eeeaa6982df) squid (dev)","entity_id":"x","hostname":"smithi184","pid":"28840","root":"/"}}] 2024-09-16T15:45:04.826 DEBUG:tasks.cephfs.filesystem:_json_asok output [ { "cap_acquisition": { "halflife": 30, "value": 0 }, "client_metadata": { "ceph_sha1": "33f95d2ae1bcbe7eb04404303f179eeeaa6982df", "ceph_version": "ceph version 19.3.0-4938-g33f95d2a (33f95d2ae1bcbe7eb04404303f179eeeaa6982df) squid (dev)", "client_features": { "feature_bits": "0x00000000003fffff" }, "entity_id": "x", "hostname": "smithi184", "metric_spec": { "metric_flags": { "feature_bits": "0x000000000000ffff" } }, "pid": "28840", "root": "/" }, "completed_requests": [ { "created_ino": "0x0", "tid": 15 }, { "created_ino": "0x100000001f9", "tid": 16 } ], "delegated_inos": [ { "length": 500, "start": "0x10000000005" } ], "entity": { "addr": { "addr": "172.21.15.184:0", "nonce": 908570948, "type": "any" }, "name": { "num": 5047, "type": "client" } }, "id": 5047, "inst": "client.5047 172.21.15.184:0/908570948", "last_trim_completed_flushes_tid": 2, "last_trim_completed_requests_tid": 15, "num_caps": 7, "num_completed_flushes": 1, "num_completed_requests": 2, "num_leases": 1, "prealloc_inos": [ { "length": 500, "start": "0x10000000005" }, { "length": 1000, "start": "0x100000001fa" } ], "recall_caps": { "halflife": 60, "value": 0 }, "recall_caps_throttle": { "halflife": 1.5, "value": 0 }, "recall_caps_throttle2o": { "halflife": 0.5, "value": 0 }, "reconnecting": false, "release_caps": { "halflife": 60, "value": 0 }, "request_load_avg": 12, "requests_in_flight": 0, "session_cache_liveness": { "halflife": 300, "value": 20.854135858449677 }, "state": "open", "uptime": 23.16754278 } ] 2024-09-16T15:45:04.826 DEBUG:tasks.ceph_test_case:wait_until_true: waiting (timeout=90 retry_count=0)... 2024-09-16T15:45:09.009 DEBUG:teuthology.orchestra.run.smithi123:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-16T15:45:09.012 DEBUG:teuthology.orchestra.run.smithi184:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-16T15:45:09.827 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-16T15:45:09.966 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:09.962+0000 7fa985586640 1 -- 172.21.15.123:0/3582150659 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa98015bca0 msgr2=0x7fa9800d0630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:45:09.966 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:09.962+0000 7fa985586640 1 --2- 172.21.15.123:0/3582150659 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa98015bca0 0x7fa9800d0630 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7fa970009840 tx=0x7fa97001c760 comp rx=0 tx=0).stop 2024-09-16T15:45:09.966 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:09.962+0000 7fa985586640 1 -- 172.21.15.123:0/3582150659 shutdown_connections 2024-09-16T15:45:09.966 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:09.962+0000 7fa985586640 1 --2- 172.21.15.123:0/3582150659 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fa980074600 0x7fa98006e750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:45:09.966 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:09.962+0000 7fa985586640 1 --2- 172.21.15.123:0/3582150659 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa98015bca0 0x7fa9800d0630 unknown :-1 s=CLOSED pgs=526 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:45:09.966 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:09.962+0000 7fa985586640 1 -- 172.21.15.123:0/3582150659 >> 172.21.15.123:0/3582150659 conn(0x7fa9800cfb10 msgr2=0x7fa9800cff20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:45:09.966 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:09.962+0000 7fa985586640 1 -- 172.21.15.123:0/3582150659 shutdown_connections 2024-09-16T15:45:09.966 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:09.962+0000 7fa985586640 1 -- 172.21.15.123:0/3582150659 wait complete. 2024-09-16T15:45:09.967 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:09.962+0000 7fa985586640 1 Processor -- start 2024-09-16T15:45:09.967 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:09.962+0000 7fa985586640 1 -- start start 2024-09-16T15:45:09.968 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:09.966+0000 7fa985586640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fa9800726c0 0x7fa98014e870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:45:09.969 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:09.966+0000 7fa985586640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa980074600 0x7fa98014dd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:45:09.969 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:09.966+0000 7fa985586640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa98015bca0 0x7fa9800c3fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:45:09.969 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:09.966+0000 7fa985586640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fa980168bd0 con 0x7fa980074600 2024-09-16T15:45:09.969 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:09.966+0000 7fa985586640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fa980168a50 con 0x7fa98015bca0 2024-09-16T15:45:09.969 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:09.966+0000 7fa985586640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fa980168d50 con 0x7fa9800726c0 2024-09-16T15:45:09.969 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:09.966+0000 7fa97e7fc640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa980074600 0x7fa98014dd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:45:09.969 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:09.966+0000 7fa97e7fc640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa980074600 0x7fa98014dd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:35604/0 (socket says 172.21.15.123:35604) 2024-09-16T15:45:09.969 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:09.966+0000 7fa97e7fc640 1 -- 172.21.15.123:0/2719511214 learned_addr learned my addr 172.21.15.123:0/2719511214 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:45:09.969 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:09.966+0000 7fa97effd640 1 --2- 172.21.15.123:0/2719511214 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fa9800726c0 0x7fa98014e870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:45:09.969 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:09.966+0000 7fa97f7fe640 1 --2- 172.21.15.123:0/2719511214 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa98015bca0 0x7fa9800c3fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:45:09.969 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:09.966+0000 7fa97e7fc640 1 -- 172.21.15.123:0/2719511214 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fa9800726c0 msgr2=0x7fa98014e870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:45:09.969 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:09.966+0000 7fa97e7fc640 1 --2- 172.21.15.123:0/2719511214 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fa9800726c0 0x7fa98014e870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:45:09.969 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:09.966+0000 7fa97e7fc640 1 -- 172.21.15.123:0/2719511214 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa98015bca0 msgr2=0x7fa9800c3fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:45:09.970 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:09.966+0000 7fa97e7fc640 1 --2- 172.21.15.123:0/2719511214 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa98015bca0 0x7fa9800c3fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:45:09.970 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:09.966+0000 7fa97e7fc640 1 -- 172.21.15.123:0/2719511214 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa98014e590 con 0x7fa980074600 2024-09-16T15:45:09.970 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:09.966+0000 7fa97f7fe640 1 --2- 172.21.15.123:0/2719511214 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa98015bca0 0x7fa9800c3fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T15:45:09.970 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:09.966+0000 7fa97e7fc640 1 --2- 172.21.15.123:0/2719511214 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa980074600 0x7fa98014dd50 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7fa97400c450 tx=0x7fa97400c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:45:09.970 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:09.966+0000 7fa9677fe640 1 -- 172.21.15.123:0/2719511214 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa974005480 con 0x7fa980074600 2024-09-16T15:45:09.970 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:09.966+0000 7fa9677fe640 1 -- 172.21.15.123:0/2719511214 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fa974003600 con 0x7fa980074600 2024-09-16T15:45:09.970 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:09.966+0000 7fa985586640 1 -- 172.21.15.123:0/2719511214 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa980166120 con 0x7fa980074600 2024-09-16T15:45:09.970 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:09.966+0000 7fa9677fe640 1 -- 172.21.15.123:0/2719511214 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa974003a30 con 0x7fa980074600 2024-09-16T15:45:09.970 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:09.966+0000 7fa985586640 1 -- 172.21.15.123:0/2719511214 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa9801561a0 con 0x7fa980074600 2024-09-16T15:45:09.971 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:09.966+0000 7fa985586640 1 -- 172.21.15.123:0/2719511214 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa98006e750 con 0x7fa980074600 2024-09-16T15:45:09.971 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:09.966+0000 7fa9677fe640 1 -- 172.21.15.123:0/2719511214 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 8) ==== 97284+0+0 (secure 0 0 0) 0x7fa974003bd0 con 0x7fa980074600 2024-09-16T15:45:09.971 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:09.966+0000 7fa9677fe640 1 --2- 172.21.15.123:0/2719511214 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fa96c073130 0x7fa96c0755f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:45:09.971 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:09.966+0000 7fa9677fe640 1 -- 172.21.15.123:0/2719511214 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(46..46 src has 1..46) ==== 5767+0+0 (secure 0 0 0) 0x7fa974093e20 con 0x7fa980074600 2024-09-16T15:45:09.971 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:09.970+0000 7fa97effd640 1 --2- 172.21.15.123:0/2719511214 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fa96c073130 0x7fa96c0755f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:45:09.972 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:09.970+0000 7fa97effd640 1 --2- 172.21.15.123:0/2719511214 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fa96c073130 0x7fa96c0755f0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fa9700af000 tx=0x7fa97001afa0 comp rx=0 tx=0).ready entity=mgr.4135 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:45:09.975 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:09.970+0000 7fa9677fe640 1 -- 172.21.15.123:0/2719511214 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7fa97405ec50 con 0x7fa980074600 2024-09-16T15:45:10.182 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:10.178+0000 7fa985586640 1 -- 172.21.15.123:0/2719511214 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa98015f630 con 0x7fa980074600 2024-09-16T15:45:10.182 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:10.178+0000 7fa9677fe640 1 -- 172.21.15.123:0/2719511214 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) ==== 94+0+5223 (secure 0 0 0) 0x7fa97405e550 con 0x7fa980074600 2024-09-16T15:45:10.183 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-09-16T15:45:10.183 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":28,"btime":"2024-09-16T15:44:32:363512+0000","default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4712,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6837/288264117","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6836","nonce":288264117},{"type":"v1","addr":"172.21.15.184:6837","nonce":288264117}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":23},{"gid":5268,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6834/60194515","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6832","nonce":60194515},{"type":"v1","addr":"172.21.15.123:6834","nonce":60194515}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":23},{"gid":5271,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6835/1798373479","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6833","nonce":1798373479},{"type":"v1","addr":"172.21.15.123:6835","nonce":1798373479}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":22}],"filesystems":[{"mdsmap":{"epoch":27,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-16T15:44:14.853230+0000","modified":"2024-09-16T15:44:17.861742+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5280},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5280":{"gid":5280,"name":"d","rank":0,"incarnation":24,"state":"up:active","state_seq":3,"addr":"172.21.15.184:6835/374389412","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":374389412},{"type":"v1","addr":"172.21.15.184:6835","nonce":374389412}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5280,"qdb_cluster":[5280]},"id":3}]} 2024-09-16T15:45:10.183 INFO:teuthology.orchestra.run.smithi123.stderr:dumped fsmap epoch 28 2024-09-16T15:45:10.185 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:10.182+0000 7fa985586640 1 -- 172.21.15.123:0/2719511214 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fa96c073130 msgr2=0x7fa96c0755f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:45:10.185 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:10.182+0000 7fa985586640 1 --2- 172.21.15.123:0/2719511214 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fa96c073130 0x7fa96c0755f0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fa9700af000 tx=0x7fa97001afa0 comp rx=0 tx=0).stop 2024-09-16T15:45:10.185 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:10.182+0000 7fa985586640 1 -- 172.21.15.123:0/2719511214 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa980074600 msgr2=0x7fa98014dd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:45:10.185 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:10.182+0000 7fa985586640 1 --2- 172.21.15.123:0/2719511214 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa980074600 0x7fa98014dd50 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7fa97400c450 tx=0x7fa97400c910 comp rx=0 tx=0).stop 2024-09-16T15:45:10.185 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:10.182+0000 7fa985586640 1 -- 172.21.15.123:0/2719511214 shutdown_connections 2024-09-16T15:45:10.186 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:10.182+0000 7fa985586640 1 --2- 172.21.15.123:0/2719511214 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fa96c073130 0x7fa96c0755f0 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:45:10.186 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:10.182+0000 7fa985586640 1 --2- 172.21.15.123:0/2719511214 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa98015bca0 0x7fa9800c3fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:45:10.186 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:10.182+0000 7fa985586640 1 --2- 172.21.15.123:0/2719511214 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa980074600 0x7fa98014dd50 unknown :-1 s=CLOSED pgs=527 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:45:10.186 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:10.182+0000 7fa985586640 1 --2- 172.21.15.123:0/2719511214 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fa9800726c0 0x7fa98014e870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:45:10.186 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:10.182+0000 7fa985586640 1 -- 172.21.15.123:0/2719511214 >> 172.21.15.123:0/2719511214 conn(0x7fa9800cfb10 msgr2=0x7fa9800758d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:45:10.186 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:10.182+0000 7fa985586640 1 -- 172.21.15.123:0/2719511214 shutdown_connections 2024-09-16T15:45:10.186 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:10.182+0000 7fa985586640 1 -- 172.21.15.123:0/2719511214 wait complete. 2024-09-16T15:45:10.200 DEBUG:teuthology.orchestra.run.smithi184:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.d.asok --format=json session ls 2024-09-16T15:45:10.352 INFO:teuthology.orchestra.run.smithi184.stdout:[{"id":5047,"entity":{"name":{"type":"client","num":5047},"addr":{"type":"any","addr":"172.21.15.184:0","nonce":908570948}},"state":"open","num_leases":1,"num_caps":7,"request_load_avg":11,"uptime":28.691456900999999,"requests_in_flight":0,"num_completed_requests":2,"num_completed_flushes":1,"reconnecting":false,"recall_caps":{"value":0,"halflife":60},"release_caps":{"value":0,"halflife":60},"recall_caps_throttle":{"value":0,"halflife":1.5},"recall_caps_throttle2o":{"value":0,"halflife":0.5},"session_cache_liveness":{"value":20.589666823291296,"halflife":300},"cap_acquisition":{"value":0,"halflife":30},"last_trim_completed_requests_tid":15,"last_trim_completed_flushes_tid":2,"delegated_inos":[{"start":"0x10000000005","length":500}],"inst":"client.5047 172.21.15.184:0/908570948","completed_requests":[{"tid":15,"created_ino":"0x0"},{"tid":16,"created_ino":"0x100000001f9"}],"prealloc_inos":[{"start":"0x10000000005","length":500},{"start":"0x100000001fa","length":1000}],"client_metadata":{"client_features":{"feature_bits":"0x00000000003fffff"},"metric_spec":{"metric_flags":{"feature_bits":"0x000000000000ffff"}},"ceph_sha1":"33f95d2ae1bcbe7eb04404303f179eeeaa6982df","ceph_version":"ceph version 19.3.0-4938-g33f95d2a (33f95d2ae1bcbe7eb04404303f179eeeaa6982df) squid (dev)","entity_id":"x","hostname":"smithi184","pid":"28840","root":"/"}}] 2024-09-16T15:45:10.353 DEBUG:tasks.cephfs.filesystem:_json_asok output [ { "cap_acquisition": { "halflife": 30, "value": 0 }, "client_metadata": { "ceph_sha1": "33f95d2ae1bcbe7eb04404303f179eeeaa6982df", "ceph_version": "ceph version 19.3.0-4938-g33f95d2a (33f95d2ae1bcbe7eb04404303f179eeeaa6982df) squid (dev)", "client_features": { "feature_bits": "0x00000000003fffff" }, "entity_id": "x", "hostname": "smithi184", "metric_spec": { "metric_flags": { "feature_bits": "0x000000000000ffff" } }, "pid": "28840", "root": "/" }, "completed_requests": [ { "created_ino": "0x0", "tid": 15 }, { "created_ino": "0x100000001f9", "tid": 16 } ], "delegated_inos": [ { "length": 500, "start": "0x10000000005" } ], "entity": { "addr": { "addr": "172.21.15.184:0", "nonce": 908570948, "type": "any" }, "name": { "num": 5047, "type": "client" } }, "id": 5047, "inst": "client.5047 172.21.15.184:0/908570948", "last_trim_completed_flushes_tid": 2, "last_trim_completed_requests_tid": 15, "num_caps": 7, "num_completed_flushes": 1, "num_completed_requests": 2, "num_leases": 1, "prealloc_inos": [ { "length": 500, "start": "0x10000000005" }, { "length": 1000, "start": "0x100000001fa" } ], "recall_caps": { "halflife": 60, "value": 0 }, "recall_caps_throttle": { "halflife": 1.5, "value": 0 }, "recall_caps_throttle2o": { "halflife": 0.5, "value": 0 }, "reconnecting": false, "release_caps": { "halflife": 60, "value": 0 }, "request_load_avg": 11, "requests_in_flight": 0, "session_cache_liveness": { "halflife": 300, "value": 20.589666823291296 }, "state": "open", "uptime": 28.691456901 } ] 2024-09-16T15:45:10.353 DEBUG:tasks.ceph_test_case:wait_until_true: waiting (timeout=90 retry_count=0)... 2024-09-16T15:45:15.354 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-16T15:45:15.493 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:15.490+0000 7f1e01769640 1 -- 172.21.15.123:0/2244045208 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1dfc0726c0 msgr2=0x7f1dfc073ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:45:15.493 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:15.490+0000 7f1e01769640 1 --2- 172.21.15.123:0/2244045208 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1dfc0726c0 0x7f1dfc073ef0 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7f1dfc0cd650 tx=0x7f1df001c660 comp rx=0 tx=0).stop 2024-09-16T15:45:15.493 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:15.490+0000 7f1e01769640 1 -- 172.21.15.123:0/2244045208 shutdown_connections 2024-09-16T15:45:15.493 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:15.490+0000 7f1e01769640 1 --2- 172.21.15.123:0/2244045208 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1dfc0726c0 0x7f1dfc073ef0 unknown :-1 s=CLOSED pgs=529 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:45:15.493 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:15.490+0000 7f1e01769640 1 --2- 172.21.15.123:0/2244045208 >> v2:172.21.15.184:3300/0 conn(0x7f1dfc15bca0 0x7f1dfc0d0630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:45:15.494 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:15.490+0000 7f1e01769640 1 -- 172.21.15.123:0/2244045208 >> 172.21.15.123:0/2244045208 conn(0x7f1dfc0cfb10 msgr2=0x7f1dfc0cff20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:45:15.494 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:15.490+0000 7f1e01769640 1 -- 172.21.15.123:0/2244045208 shutdown_connections 2024-09-16T15:45:15.494 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:15.490+0000 7f1e01769640 1 -- 172.21.15.123:0/2244045208 wait complete. 2024-09-16T15:45:15.494 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:15.490+0000 7f1e01769640 1 Processor -- start 2024-09-16T15:45:15.494 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:15.490+0000 7f1e01769640 1 -- start start 2024-09-16T15:45:15.496 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:15.494+0000 7f1e01769640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1dfc0726c0 0x7f1dfc1663e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:45:15.496 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:15.494+0000 7f1e01769640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1dfc074430 0x7f1dfc0cf160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:45:15.496 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:15.494+0000 7f1e01769640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f1dfc15bca0 0x7f1dfc0c5440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:45:15.496 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:15.494+0000 7f1e01769640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f1dfc168e90 con 0x7f1dfc0726c0 2024-09-16T15:45:15.496 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:15.494+0000 7f1e01769640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f1dfc168d10 con 0x7f1dfc074430 2024-09-16T15:45:15.497 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:15.494+0000 7f1e01769640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f1dfc169010 con 0x7f1dfc15bca0 2024-09-16T15:45:15.497 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:15.494+0000 7f1dfaffd640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1dfc0726c0 0x7f1dfc1663e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:45:15.497 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:15.494+0000 7f1dfaffd640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1dfc0726c0 0x7f1dfc1663e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:38986/0 (socket says 172.21.15.123:38986) 2024-09-16T15:45:15.497 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:15.494+0000 7f1dfaffd640 1 -- 172.21.15.123:0/1927523852 learned_addr learned my addr 172.21.15.123:0/1927523852 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:45:15.497 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:15.494+0000 7f1dfaffd640 1 -- 172.21.15.123:0/1927523852 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f1dfc15bca0 msgr2=0x7f1dfc0c5440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:45:15.497 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:15.494+0000 7f1dfaffd640 1 --2- 172.21.15.123:0/1927523852 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f1dfc15bca0 0x7f1dfc0c5440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:45:15.497 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:15.494+0000 7f1dfaffd640 1 -- 172.21.15.123:0/1927523852 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1dfc074430 msgr2=0x7f1dfc0cf160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:45:15.497 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:15.494+0000 7f1dfa7fc640 1 --2- 172.21.15.123:0/1927523852 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1dfc074430 0x7f1dfc0cf160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:45:15.497 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:15.494+0000 7f1dfaffd640 1 --2- 172.21.15.123:0/1927523852 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1dfc074430 0x7f1dfc0cf160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:45:15.497 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:15.494+0000 7f1dfaffd640 1 -- 172.21.15.123:0/1927523852 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1dfc15e700 con 0x7f1dfc0726c0 2024-09-16T15:45:15.497 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:15.494+0000 7f1dfaffd640 1 --2- 172.21.15.123:0/1927523852 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1dfc0726c0 0x7f1dfc1663e0 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7f1dec0058d0 tx=0x7f1dec00dde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:45:15.497 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:15.494+0000 7f1de37fe640 1 -- 172.21.15.123:0/1927523852 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1dec017070 con 0x7f1dfc0726c0 2024-09-16T15:45:15.498 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:15.494+0000 7f1de37fe640 1 -- 172.21.15.123:0/1927523852 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f1dec009d20 con 0x7f1dfc0726c0 2024-09-16T15:45:15.498 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:15.494+0000 7f1e01769640 1 -- 172.21.15.123:0/1927523852 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1dfc15e9a0 con 0x7f1dfc0726c0 2024-09-16T15:45:15.498 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:15.494+0000 7f1de37fe640 1 -- 172.21.15.123:0/1927523852 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1dec0106c0 con 0x7f1dfc0726c0 2024-09-16T15:45:15.498 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:15.494+0000 7f1e01769640 1 -- 172.21.15.123:0/1927523852 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1dfc15ef60 con 0x7f1dfc0726c0 2024-09-16T15:45:15.498 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:15.494+0000 7f1e01769640 1 -- 172.21.15.123:0/1927523852 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1dfc073ef0 con 0x7f1dfc0726c0 2024-09-16T15:45:15.499 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:15.494+0000 7f1de37fe640 1 -- 172.21.15.123:0/1927523852 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 8) ==== 97284+0+0 (secure 0 0 0) 0x7f1dec01a070 con 0x7f1dfc0726c0 2024-09-16T15:45:15.499 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:15.494+0000 7f1de37fe640 1 --2- 172.21.15.123:0/1927523852 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f1de8073100 0x7f1de80755c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:45:15.499 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:15.494+0000 7f1de37fe640 1 -- 172.21.15.123:0/1927523852 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(46..46 src has 1..46) ==== 5767+0+0 (secure 0 0 0) 0x7f1dec094460 con 0x7f1dfc0726c0 2024-09-16T15:45:15.499 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:15.494+0000 7f1dfa7fc640 1 --2- 172.21.15.123:0/1927523852 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f1de8073100 0x7f1de80755c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:45:15.499 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:15.498+0000 7f1dfa7fc640 1 --2- 172.21.15.123:0/1927523852 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f1de8073100 0x7f1de80755c0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f1df00af000 tx=0x7f1df001aea0 comp rx=0 tx=0).ready entity=mgr.4135 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:45:15.502 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:15.498+0000 7f1de37fe640 1 -- 172.21.15.123:0/1927523852 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f1dec05f290 con 0x7f1dfc0726c0 2024-09-16T15:45:15.705 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:15.702+0000 7f1e01769640 1 -- 172.21.15.123:0/1927523852 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f1dfc0700e0 con 0x7f1dfc0726c0 2024-09-16T15:45:15.706 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:15.702+0000 7f1de37fe640 1 -- 172.21.15.123:0/1927523852 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) ==== 94+0+5223 (secure 0 0 0) 0x7f1dec05eb90 con 0x7f1dfc0726c0 2024-09-16T15:45:15.706 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-09-16T15:45:15.706 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":28,"btime":"2024-09-16T15:44:32:363512+0000","default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4712,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6837/288264117","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6836","nonce":288264117},{"type":"v1","addr":"172.21.15.184:6837","nonce":288264117}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":23},{"gid":5268,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6834/60194515","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6832","nonce":60194515},{"type":"v1","addr":"172.21.15.123:6834","nonce":60194515}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":23},{"gid":5271,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6835/1798373479","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6833","nonce":1798373479},{"type":"v1","addr":"172.21.15.123:6835","nonce":1798373479}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":22}],"filesystems":[{"mdsmap":{"epoch":27,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-16T15:44:14.853230+0000","modified":"2024-09-16T15:44:17.861742+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5280},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5280":{"gid":5280,"name":"d","rank":0,"incarnation":24,"state":"up:active","state_seq":3,"addr":"172.21.15.184:6835/374389412","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":374389412},{"type":"v1","addr":"172.21.15.184:6835","nonce":374389412}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5280,"qdb_cluster":[5280]},"id":3}]} 2024-09-16T15:45:15.706 INFO:teuthology.orchestra.run.smithi123.stderr:dumped fsmap epoch 28 2024-09-16T15:45:15.708 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:15.706+0000 7f1e01769640 1 -- 172.21.15.123:0/1927523852 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f1de8073100 msgr2=0x7f1de80755c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:45:15.708 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:15.706+0000 7f1e01769640 1 --2- 172.21.15.123:0/1927523852 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f1de8073100 0x7f1de80755c0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f1df00af000 tx=0x7f1df001aea0 comp rx=0 tx=0).stop 2024-09-16T15:45:15.709 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:15.706+0000 7f1e01769640 1 -- 172.21.15.123:0/1927523852 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1dfc0726c0 msgr2=0x7f1dfc1663e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:45:15.709 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:15.706+0000 7f1e01769640 1 --2- 172.21.15.123:0/1927523852 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1dfc0726c0 0x7f1dfc1663e0 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7f1dec0058d0 tx=0x7f1dec00dde0 comp rx=0 tx=0).stop 2024-09-16T15:45:15.709 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:15.706+0000 7f1e01769640 1 -- 172.21.15.123:0/1927523852 shutdown_connections 2024-09-16T15:45:15.709 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:15.706+0000 7f1e01769640 1 --2- 172.21.15.123:0/1927523852 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f1de8073100 0x7f1de80755c0 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:45:15.709 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:15.706+0000 7f1e01769640 1 --2- 172.21.15.123:0/1927523852 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f1dfc15bca0 0x7f1dfc0c5440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:45:15.709 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:15.706+0000 7f1e01769640 1 --2- 172.21.15.123:0/1927523852 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1dfc074430 0x7f1dfc0cf160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:45:15.709 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:15.706+0000 7f1e01769640 1 --2- 172.21.15.123:0/1927523852 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1dfc0726c0 0x7f1dfc1663e0 unknown :-1 s=CLOSED pgs=530 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:45:15.709 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:15.706+0000 7f1e01769640 1 -- 172.21.15.123:0/1927523852 >> 172.21.15.123:0/1927523852 conn(0x7f1dfc0cfb10 msgr2=0x7f1dfc0b3700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:45:15.709 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:15.706+0000 7f1e01769640 1 -- 172.21.15.123:0/1927523852 shutdown_connections 2024-09-16T15:45:15.709 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:15.706+0000 7f1e01769640 1 -- 172.21.15.123:0/1927523852 wait complete. 2024-09-16T15:45:15.724 DEBUG:teuthology.orchestra.run.smithi184:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.d.asok --format=json session ls 2024-09-16T15:45:15.875 INFO:teuthology.orchestra.run.smithi184.stdout:[{"id":5047,"entity":{"name":{"type":"client","num":5047},"addr":{"type":"any","addr":"172.21.15.184:0","nonce":908570948}},"state":"open","num_leases":0,"num_caps":7,"request_load_avg":10,"uptime":34.215376268999997,"requests_in_flight":0,"num_completed_requests":2,"num_completed_flushes":1,"reconnecting":false,"recall_caps":{"value":0,"halflife":60},"release_caps":{"value":0,"halflife":60},"recall_caps_throttle":{"value":0,"halflife":1.5},"recall_caps_throttle2o":{"value":0,"halflife":0.5},"session_cache_liveness":{"value":20.328551498476742,"halflife":300},"cap_acquisition":{"value":0,"halflife":30},"last_trim_completed_requests_tid":15,"last_trim_completed_flushes_tid":2,"delegated_inos":[{"start":"0x10000000005","length":500}],"inst":"client.5047 172.21.15.184:0/908570948","completed_requests":[{"tid":15,"created_ino":"0x0"},{"tid":16,"created_ino":"0x100000001f9"}],"prealloc_inos":[{"start":"0x10000000005","length":500},{"start":"0x100000001fa","length":1000}],"client_metadata":{"client_features":{"feature_bits":"0x00000000003fffff"},"metric_spec":{"metric_flags":{"feature_bits":"0x000000000000ffff"}},"ceph_sha1":"33f95d2ae1bcbe7eb04404303f179eeeaa6982df","ceph_version":"ceph version 19.3.0-4938-g33f95d2a (33f95d2ae1bcbe7eb04404303f179eeeaa6982df) squid (dev)","entity_id":"x","hostname":"smithi184","pid":"28840","root":"/"}}] 2024-09-16T15:45:15.875 DEBUG:tasks.cephfs.filesystem:_json_asok output [ { "cap_acquisition": { "halflife": 30, "value": 0 }, "client_metadata": { "ceph_sha1": "33f95d2ae1bcbe7eb04404303f179eeeaa6982df", "ceph_version": "ceph version 19.3.0-4938-g33f95d2a (33f95d2ae1bcbe7eb04404303f179eeeaa6982df) squid (dev)", "client_features": { "feature_bits": "0x00000000003fffff" }, "entity_id": "x", "hostname": "smithi184", "metric_spec": { "metric_flags": { "feature_bits": "0x000000000000ffff" } }, "pid": "28840", "root": "/" }, "completed_requests": [ { "created_ino": "0x0", "tid": 15 }, { "created_ino": "0x100000001f9", "tid": 16 } ], "delegated_inos": [ { "length": 500, "start": "0x10000000005" } ], "entity": { "addr": { "addr": "172.21.15.184:0", "nonce": 908570948, "type": "any" }, "name": { "num": 5047, "type": "client" } }, "id": 5047, "inst": "client.5047 172.21.15.184:0/908570948", "last_trim_completed_flushes_tid": 2, "last_trim_completed_requests_tid": 15, "num_caps": 7, "num_completed_flushes": 1, "num_completed_requests": 2, "num_leases": 0, "prealloc_inos": [ { "length": 500, "start": "0x10000000005" }, { "length": 1000, "start": "0x100000001fa" } ], "recall_caps": { "halflife": 60, "value": 0 }, "recall_caps_throttle": { "halflife": 1.5, "value": 0 }, "recall_caps_throttle2o": { "halflife": 0.5, "value": 0 }, "reconnecting": false, "release_caps": { "halflife": 60, "value": 0 }, "request_load_avg": 10, "requests_in_flight": 0, "session_cache_liveness": { "halflife": 300, "value": 20.32855149847674 }, "state": "open", "uptime": 34.215376269 } ] 2024-09-16T15:45:15.875 DEBUG:tasks.ceph_test_case:wait_until_true: waiting (timeout=90 retry_count=0)... 2024-09-16T15:45:20.877 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-16T15:45:21.013 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:21.010+0000 7f9071524640 1 -- 172.21.15.123:0/1373270401 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9054003a10 msgr2=0x7f9054003e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:45:21.013 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:21.010+0000 7f9071524640 1 --2- 172.21.15.123:0/1373270401 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9054003a10 0x7f9054003e30 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7f905c00c310 tx=0x7f905c01cf90 comp rx=0 tx=0).stop 2024-09-16T15:45:21.013 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:21.010+0000 7f906b7fe640 1 -- 172.21.15.123:0/1373270401 reap_dead start 2024-09-16T15:45:21.013 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:21.010+0000 7f9071524640 1 -- 172.21.15.123:0/1373270401 shutdown_connections 2024-09-16T15:45:21.013 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:21.010+0000 7f9071524640 1 -- 172.21.15.123:0/1373270401 >> 172.21.15.123:0/1373270401 conn(0x7f906c0cfb10 msgr2=0x7f906c0cff20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:45:21.013 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:21.010+0000 7f9071524640 1 -- 172.21.15.123:0/1373270401 shutdown_connections 2024-09-16T15:45:21.013 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:21.010+0000 7f9071524640 1 -- 172.21.15.123:0/1373270401 wait complete. 2024-09-16T15:45:21.014 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:21.010+0000 7f9071524640 1 Processor -- start 2024-09-16T15:45:21.014 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:21.010+0000 7f9071524640 1 -- start start 2024-09-16T15:45:21.016 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:21.010+0000 7f9071524640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f9054003a10 0x7f906c0c9b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:45:21.016 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:21.010+0000 7f9071524640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f906c0c9630 0x7f906c0b2b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:45:21.016 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:21.010+0000 7f9071524640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f906c0b4970 0x7f906c15e700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:45:21.016 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:21.010+0000 7f9071524640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f906c06faa0 con 0x7f906c0c9630 2024-09-16T15:45:21.016 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:21.010+0000 7f9071524640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f906c06f920 con 0x7f9054003a10 2024-09-16T15:45:21.016 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:21.010+0000 7f9071524640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f906c166a40 con 0x7f906c0b4970 2024-09-16T15:45:21.016 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:21.014+0000 7f906a7fc640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f906c0c9630 0x7f906c0b2b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:45:21.016 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:21.014+0000 7f906a7fc640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f906c0c9630 0x7f906c0b2b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:39014/0 (socket says 172.21.15.123:39014) 2024-09-16T15:45:21.017 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:21.014+0000 7f906a7fc640 1 -- 172.21.15.123:0/3358811331 learned_addr learned my addr 172.21.15.123:0/3358811331 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:45:21.017 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:21.014+0000 7f906affd640 1 --2- 172.21.15.123:0/3358811331 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f9054003a10 0x7f906c0c9b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:45:21.017 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:21.014+0000 7f906b7fe640 1 --2- 172.21.15.123:0/3358811331 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f906c0b4970 0x7f906c15e700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:45:21.017 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:21.014+0000 7f906a7fc640 1 -- 172.21.15.123:0/3358811331 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f906c0b4970 msgr2=0x7f906c15e700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:45:21.017 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:21.014+0000 7f906a7fc640 1 --2- 172.21.15.123:0/3358811331 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f906c0b4970 0x7f906c15e700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:45:21.017 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:21.014+0000 7f906a7fc640 1 -- 172.21.15.123:0/3358811331 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f9054003a10 msgr2=0x7f906c0c9b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:45:21.017 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:21.014+0000 7f906a7fc640 1 --2- 172.21.15.123:0/3358811331 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f9054003a10 0x7f906c0c9b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:45:21.017 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:21.014+0000 7f906a7fc640 1 -- 172.21.15.123:0/3358811331 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f906c166840 con 0x7f906c0c9630 2024-09-16T15:45:21.017 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:21.014+0000 7f906b7fe640 1 --2- 172.21.15.123:0/3358811331 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f906c0b4970 0x7f906c15e700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T15:45:21.017 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:21.014+0000 7f906a7fc640 1 --2- 172.21.15.123:0/3358811331 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f906c0c9630 0x7f906c0b2b30 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7f906000e760 tx=0x7f906000ec20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:45:21.017 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:21.014+0000 7f90537fe640 1 -- 172.21.15.123:0/3358811331 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f906000cc60 con 0x7f906c0c9630 2024-09-16T15:45:21.017 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:21.014+0000 7f90537fe640 1 -- 172.21.15.123:0/3358811331 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f906000ee70 con 0x7f906c0c9630 2024-09-16T15:45:21.018 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:21.014+0000 7f9071524640 1 -- 172.21.15.123:0/3358811331 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f906c0b12a0 con 0x7f906c0c9630 2024-09-16T15:45:21.018 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:21.014+0000 7f90537fe640 1 -- 172.21.15.123:0/3358811331 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9060011690 con 0x7f906c0c9630 2024-09-16T15:45:21.018 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:21.014+0000 7f9071524640 1 -- 172.21.15.123:0/3358811331 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f906c15cdd0 con 0x7f906c0c9630 2024-09-16T15:45:21.019 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:21.014+0000 7f9071524640 1 -- 172.21.15.123:0/3358811331 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f906c06e750 con 0x7f906c0c9630 2024-09-16T15:45:21.019 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:21.014+0000 7f90537fe640 1 -- 172.21.15.123:0/3358811331 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 8) ==== 97284+0+0 (secure 0 0 0) 0x7f9060011830 con 0x7f906c0c9630 2024-09-16T15:45:21.019 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:21.014+0000 7f90537fe640 1 --2- 172.21.15.123:0/3358811331 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f9058073100 0x7f90580755c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:45:21.019 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:21.014+0000 7f90537fe640 1 -- 172.21.15.123:0/3358811331 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(46..46 src has 1..46) ==== 5767+0+0 (secure 0 0 0) 0x7f9060094f50 con 0x7f906c0c9630 2024-09-16T15:45:21.019 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:21.014+0000 7f906affd640 1 --2- 172.21.15.123:0/3358811331 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f9058073100 0x7f90580755c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:45:21.019 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:21.018+0000 7f906affd640 1 --2- 172.21.15.123:0/3358811331 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f9058073100 0x7f90580755c0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f905c0ac000 tx=0x7f905c019dd0 comp rx=0 tx=0).ready entity=mgr.4135 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:45:21.022 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:21.018+0000 7f90537fe640 1 -- 172.21.15.123:0/3358811331 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f9060016030 con 0x7f906c0c9630 2024-09-16T15:45:21.232 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:21.230+0000 7f9071524640 1 -- 172.21.15.123:0/3358811331 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f906c0700e0 con 0x7f906c0c9630 2024-09-16T15:45:21.232 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:21.230+0000 7f90537fe640 1 -- 172.21.15.123:0/3358811331 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) ==== 94+0+5223 (secure 0 0 0) 0x7f906005fcf0 con 0x7f906c0c9630 2024-09-16T15:45:21.233 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-09-16T15:45:21.233 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":28,"btime":"2024-09-16T15:44:32:363512+0000","default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4712,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6837/288264117","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6836","nonce":288264117},{"type":"v1","addr":"172.21.15.184:6837","nonce":288264117}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":23},{"gid":5268,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6834/60194515","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6832","nonce":60194515},{"type":"v1","addr":"172.21.15.123:6834","nonce":60194515}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":23},{"gid":5271,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6835/1798373479","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6833","nonce":1798373479},{"type":"v1","addr":"172.21.15.123:6835","nonce":1798373479}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":22}],"filesystems":[{"mdsmap":{"epoch":27,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-16T15:44:14.853230+0000","modified":"2024-09-16T15:44:17.861742+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5280},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5280":{"gid":5280,"name":"d","rank":0,"incarnation":24,"state":"up:active","state_seq":3,"addr":"172.21.15.184:6835/374389412","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":374389412},{"type":"v1","addr":"172.21.15.184:6835","nonce":374389412}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5280,"qdb_cluster":[5280]},"id":3}]} 2024-09-16T15:45:21.233 INFO:teuthology.orchestra.run.smithi123.stderr:dumped fsmap epoch 28 2024-09-16T15:45:21.235 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:21.230+0000 7f9071524640 1 -- 172.21.15.123:0/3358811331 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f9058073100 msgr2=0x7f90580755c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:45:21.235 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:21.230+0000 7f9071524640 1 --2- 172.21.15.123:0/3358811331 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f9058073100 0x7f90580755c0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f905c0ac000 tx=0x7f905c019dd0 comp rx=0 tx=0).stop 2024-09-16T15:45:21.235 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:21.230+0000 7f9071524640 1 -- 172.21.15.123:0/3358811331 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f906c0c9630 msgr2=0x7f906c0b2b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:45:21.235 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:21.230+0000 7f9071524640 1 --2- 172.21.15.123:0/3358811331 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f906c0c9630 0x7f906c0b2b30 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7f906000e760 tx=0x7f906000ec20 comp rx=0 tx=0).stop 2024-09-16T15:45:21.235 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:21.234+0000 7f9071524640 1 -- 172.21.15.123:0/3358811331 shutdown_connections 2024-09-16T15:45:21.235 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:21.234+0000 7f9071524640 1 --2- 172.21.15.123:0/3358811331 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f9058073100 0x7f90580755c0 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:45:21.235 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:21.234+0000 7f9071524640 1 --2- 172.21.15.123:0/3358811331 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f906c0b4970 0x7f906c15e700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:45:21.236 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:21.234+0000 7f9071524640 1 --2- 172.21.15.123:0/3358811331 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f906c0c9630 0x7f906c0b2b30 unknown :-1 s=CLOSED pgs=533 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:45:21.236 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:21.234+0000 7f9071524640 1 --2- 172.21.15.123:0/3358811331 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f9054003a10 0x7f906c0c9b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:45:21.236 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:21.234+0000 7f9071524640 1 -- 172.21.15.123:0/3358811331 >> 172.21.15.123:0/3358811331 conn(0x7f906c0cfb10 msgr2=0x7f906c162210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:45:21.236 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:21.234+0000 7f9071524640 1 -- 172.21.15.123:0/3358811331 shutdown_connections 2024-09-16T15:45:21.236 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:21.234+0000 7f9071524640 1 -- 172.21.15.123:0/3358811331 wait complete. 2024-09-16T15:45:21.252 DEBUG:teuthology.orchestra.run.smithi184:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.d.asok --format=json session ls 2024-09-16T15:45:21.404 INFO:teuthology.orchestra.run.smithi184.stdout:[{"id":5047,"entity":{"name":{"type":"client","num":5047},"addr":{"type":"any","addr":"172.21.15.184:0","nonce":908570948}},"state":"open","num_leases":0,"num_caps":7,"request_load_avg":10,"uptime":39.743300640999998,"requests_in_flight":0,"num_completed_requests":2,"num_completed_flushes":1,"reconnecting":false,"recall_caps":{"value":0,"halflife":60},"release_caps":{"value":0,"halflife":60},"recall_caps_throttle":{"value":0,"halflife":1.5},"recall_caps_throttle2o":{"value":0,"halflife":0.5},"session_cache_liveness":{"value":20.070561878036475,"halflife":300},"cap_acquisition":{"value":0,"halflife":30},"last_trim_completed_requests_tid":15,"last_trim_completed_flushes_tid":2,"delegated_inos":[{"start":"0x10000000005","length":500}],"inst":"client.5047 172.21.15.184:0/908570948","completed_requests":[{"tid":15,"created_ino":"0x0"},{"tid":16,"created_ino":"0x100000001f9"}],"prealloc_inos":[{"start":"0x10000000005","length":500},{"start":"0x100000001fa","length":1000}],"client_metadata":{"client_features":{"feature_bits":"0x00000000003fffff"},"metric_spec":{"metric_flags":{"feature_bits":"0x000000000000ffff"}},"ceph_sha1":"33f95d2ae1bcbe7eb04404303f179eeeaa6982df","ceph_version":"ceph version 19.3.0-4938-g33f95d2a (33f95d2ae1bcbe7eb04404303f179eeeaa6982df) squid (dev)","entity_id":"x","hostname":"smithi184","pid":"28840","root":"/"}}] 2024-09-16T15:45:21.404 DEBUG:tasks.cephfs.filesystem:_json_asok output [ { "cap_acquisition": { "halflife": 30, "value": 0 }, "client_metadata": { "ceph_sha1": "33f95d2ae1bcbe7eb04404303f179eeeaa6982df", "ceph_version": "ceph version 19.3.0-4938-g33f95d2a (33f95d2ae1bcbe7eb04404303f179eeeaa6982df) squid (dev)", "client_features": { "feature_bits": "0x00000000003fffff" }, "entity_id": "x", "hostname": "smithi184", "metric_spec": { "metric_flags": { "feature_bits": "0x000000000000ffff" } }, "pid": "28840", "root": "/" }, "completed_requests": [ { "created_ino": "0x0", "tid": 15 }, { "created_ino": "0x100000001f9", "tid": 16 } ], "delegated_inos": [ { "length": 500, "start": "0x10000000005" } ], "entity": { "addr": { "addr": "172.21.15.184:0", "nonce": 908570948, "type": "any" }, "name": { "num": 5047, "type": "client" } }, "id": 5047, "inst": "client.5047 172.21.15.184:0/908570948", "last_trim_completed_flushes_tid": 2, "last_trim_completed_requests_tid": 15, "num_caps": 7, "num_completed_flushes": 1, "num_completed_requests": 2, "num_leases": 0, "prealloc_inos": [ { "length": 500, "start": "0x10000000005" }, { "length": 1000, "start": "0x100000001fa" } ], "recall_caps": { "halflife": 60, "value": 0 }, "recall_caps_throttle": { "halflife": 1.5, "value": 0 }, "recall_caps_throttle2o": { "halflife": 0.5, "value": 0 }, "reconnecting": false, "release_caps": { "halflife": 60, "value": 0 }, "request_load_avg": 10, "requests_in_flight": 0, "session_cache_liveness": { "halflife": 300, "value": 20.070561878036475 }, "state": "open", "uptime": 39.743300641 } ] 2024-09-16T15:45:21.405 DEBUG:tasks.ceph_test_case:wait_until_true: waiting (timeout=90 retry_count=0)... 2024-09-16T15:45:26.406 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-16T15:45:26.544 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:26.542+0000 7f4da96cd640 1 -- 172.21.15.123:0/733027393 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4da40cdb50 msgr2=0x7f4da415ef30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:45:26.544 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:26.542+0000 7f4da96cd640 1 --2- 172.21.15.123:0/733027393 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4da40cdb50 0x7f4da415ef30 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7f4da4073c20 tx=0x7f4d9c01cb10 comp rx=0 tx=0).stop 2024-09-16T15:45:26.544 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:26.542+0000 7f4da96cd640 1 -- 172.21.15.123:0/733027393 shutdown_connections 2024-09-16T15:45:26.544 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:26.542+0000 7f4da96cd640 1 --2- 172.21.15.123:0/733027393 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4da40cdb50 0x7f4da415ef30 unknown :-1 s=CLOSED pgs=535 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:45:26.544 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:26.542+0000 7f4da96cd640 1 -- 172.21.15.123:0/733027393 >> 172.21.15.123:0/733027393 conn(0x7f4da40cfb10 msgr2=0x7f4da40cff20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:45:26.544 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:26.542+0000 7f4da96cd640 1 -- 172.21.15.123:0/733027393 shutdown_connections 2024-09-16T15:45:26.544 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:26.542+0000 7f4da96cd640 1 -- 172.21.15.123:0/733027393 wait complete. 2024-09-16T15:45:26.545 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:26.542+0000 7f4da96cd640 1 Processor -- start 2024-09-16T15:45:26.545 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:26.542+0000 7f4da96cd640 1 -- start start 2024-09-16T15:45:26.547 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:26.542+0000 7f4da96cd640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4da40cd6e0 0x7f4da41519c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:45:26.547 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:26.542+0000 7f4da96cd640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4da40cdb50 0x7f4da40b29a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:45:26.547 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:26.542+0000 7f4da96cd640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f4da415e160 0x7f4da415c850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:45:26.547 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:26.542+0000 7f4da96cd640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f4da40cf640 con 0x7f4da40cdb50 2024-09-16T15:45:26.547 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:26.542+0000 7f4da96cd640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f4da40747d0 con 0x7f4da40cd6e0 2024-09-16T15:45:26.547 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:26.542+0000 7f4da96cd640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f4da41697f0 con 0x7f4da415e160 2024-09-16T15:45:26.547 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:26.542+0000 7f4d9a7fc640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4da40cdb50 0x7f4da40b29a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:45:26.547 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:26.542+0000 7f4d9a7fc640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4da40cdb50 0x7f4da40b29a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:47352/0 (socket says 172.21.15.123:47352) 2024-09-16T15:45:26.547 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:26.542+0000 7f4d9a7fc640 1 -- 172.21.15.123:0/2574262264 learned_addr learned my addr 172.21.15.123:0/2574262264 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:45:26.547 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:26.542+0000 7f4da2ffd640 1 --2- 172.21.15.123:0/2574262264 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4da40cd6e0 0x7f4da41519c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:45:26.547 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:26.542+0000 7f4da37fe640 1 --2- 172.21.15.123:0/2574262264 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f4da415e160 0x7f4da415c850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:45:26.548 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:26.542+0000 7f4d9a7fc640 1 -- 172.21.15.123:0/2574262264 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f4da415e160 msgr2=0x7f4da415c850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:45:26.548 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:26.542+0000 7f4d9a7fc640 1 --2- 172.21.15.123:0/2574262264 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f4da415e160 0x7f4da415c850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:45:26.548 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:26.542+0000 7f4d9a7fc640 1 -- 172.21.15.123:0/2574262264 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4da40cd6e0 msgr2=0x7f4da41519c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:45:26.548 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:26.542+0000 7f4d9a7fc640 1 --2- 172.21.15.123:0/2574262264 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4da40cd6e0 0x7f4da41519c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:45:26.548 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:26.542+0000 7f4d9a7fc640 1 -- 172.21.15.123:0/2574262264 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4da4151f00 con 0x7f4da40cdb50 2024-09-16T15:45:26.548 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:26.542+0000 7f4da37fe640 1 --2- 172.21.15.123:0/2574262264 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f4da415e160 0x7f4da415c850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T15:45:26.548 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:26.542+0000 7f4d9a7fc640 1 --2- 172.21.15.123:0/2574262264 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4da40cdb50 0x7f4da40b29a0 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7f4d8c00c4c0 tx=0x7f4d8c00c980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:45:26.548 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:26.542+0000 7f4d9bfff640 1 -- 172.21.15.123:0/2574262264 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d8c016020 con 0x7f4da40cdb50 2024-09-16T15:45:26.548 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:26.542+0000 7f4d9bfff640 1 -- 172.21.15.123:0/2574262264 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f4d8c012940 con 0x7f4da40cdb50 2024-09-16T15:45:26.548 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:26.542+0000 7f4da96cd640 1 -- 172.21.15.123:0/2574262264 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4da415d980 con 0x7f4da40cdb50 2024-09-16T15:45:26.548 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:26.542+0000 7f4d9bfff640 1 -- 172.21.15.123:0/2574262264 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d8c012d90 con 0x7f4da40cdb50 2024-09-16T15:45:26.548 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:26.542+0000 7f4da96cd640 1 -- 172.21.15.123:0/2574262264 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4da415d3c0 con 0x7f4da40cdb50 2024-09-16T15:45:26.549 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:26.546+0000 7f4da96cd640 1 -- 172.21.15.123:0/2574262264 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4da4073f30 con 0x7f4da40cdb50 2024-09-16T15:45:26.549 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:26.546+0000 7f4d9bfff640 1 -- 172.21.15.123:0/2574262264 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 8) ==== 97284+0+0 (secure 0 0 0) 0x7f4d8c019020 con 0x7f4da40cdb50 2024-09-16T15:45:26.549 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:26.546+0000 7f4d9bfff640 1 --2- 172.21.15.123:0/2574262264 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f4d84073100 0x7f4d840755c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:45:26.549 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:26.546+0000 7f4d9bfff640 1 -- 172.21.15.123:0/2574262264 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(46..46 src has 1..46) ==== 5767+0+0 (secure 0 0 0) 0x7f4d8c0948b0 con 0x7f4da40cdb50 2024-09-16T15:45:26.549 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:26.546+0000 7f4da2ffd640 1 --2- 172.21.15.123:0/2574262264 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f4d84073100 0x7f4d840755c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:45:26.550 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:26.546+0000 7f4da2ffd640 1 --2- 172.21.15.123:0/2574262264 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f4d84073100 0x7f4d840755c0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f4d94000c10 tx=0x7f4d94009040 comp rx=0 tx=0).ready entity=mgr.4135 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:45:26.552 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:26.550+0000 7f4d9bfff640 1 -- 172.21.15.123:0/2574262264 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f4d8c020020 con 0x7f4da40cdb50 2024-09-16T15:45:26.760 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:26.758+0000 7f4da96cd640 1 -- 172.21.15.123:0/2574262264 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f4da40d06c0 con 0x7f4da40cdb50 2024-09-16T15:45:26.762 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:26.758+0000 7f4d9bfff640 1 -- 172.21.15.123:0/2574262264 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) ==== 94+0+5223 (secure 0 0 0) 0x7f4da4151f00 con 0x7f4da40cdb50 2024-09-16T15:45:26.762 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-09-16T15:45:26.762 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":28,"btime":"2024-09-16T15:44:32:363512+0000","default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4712,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6837/288264117","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6836","nonce":288264117},{"type":"v1","addr":"172.21.15.184:6837","nonce":288264117}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":23},{"gid":5268,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6834/60194515","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6832","nonce":60194515},{"type":"v1","addr":"172.21.15.123:6834","nonce":60194515}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":23},{"gid":5271,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6835/1798373479","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6833","nonce":1798373479},{"type":"v1","addr":"172.21.15.123:6835","nonce":1798373479}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":22}],"filesystems":[{"mdsmap":{"epoch":27,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-16T15:44:14.853230+0000","modified":"2024-09-16T15:44:17.861742+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5280},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5280":{"gid":5280,"name":"d","rank":0,"incarnation":24,"state":"up:active","state_seq":3,"addr":"172.21.15.184:6835/374389412","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":374389412},{"type":"v1","addr":"172.21.15.184:6835","nonce":374389412}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5280,"qdb_cluster":[5280]},"id":3}]} 2024-09-16T15:45:26.762 INFO:teuthology.orchestra.run.smithi123.stderr:dumped fsmap epoch 28 2024-09-16T15:45:26.764 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:26.762+0000 7f4da96cd640 1 -- 172.21.15.123:0/2574262264 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f4d84073100 msgr2=0x7f4d840755c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:45:26.764 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:26.762+0000 7f4da96cd640 1 --2- 172.21.15.123:0/2574262264 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f4d84073100 0x7f4d840755c0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f4d94000c10 tx=0x7f4d94009040 comp rx=0 tx=0).stop 2024-09-16T15:45:26.764 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:26.762+0000 7f4da96cd640 1 -- 172.21.15.123:0/2574262264 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4da40cdb50 msgr2=0x7f4da40b29a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:45:26.764 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:26.762+0000 7f4da96cd640 1 --2- 172.21.15.123:0/2574262264 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4da40cdb50 0x7f4da40b29a0 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7f4d8c00c4c0 tx=0x7f4d8c00c980 comp rx=0 tx=0).stop 2024-09-16T15:45:26.764 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:26.762+0000 7f4da96cd640 1 -- 172.21.15.123:0/2574262264 shutdown_connections 2024-09-16T15:45:26.765 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:26.762+0000 7f4da96cd640 1 --2- 172.21.15.123:0/2574262264 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f4d84073100 0x7f4d840755c0 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:45:26.765 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:26.762+0000 7f4da96cd640 1 --2- 172.21.15.123:0/2574262264 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f4da415e160 0x7f4da415c850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:45:26.765 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:26.762+0000 7f4da96cd640 1 --2- 172.21.15.123:0/2574262264 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4da40cdb50 0x7f4da40b29a0 unknown :-1 s=CLOSED pgs=536 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:45:26.765 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:26.762+0000 7f4da96cd640 1 --2- 172.21.15.123:0/2574262264 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4da40cd6e0 0x7f4da41519c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:45:26.765 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:26.762+0000 7f4da96cd640 1 -- 172.21.15.123:0/2574262264 >> 172.21.15.123:0/2574262264 conn(0x7f4da40cfb10 msgr2=0x7f4da4159d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:45:26.765 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:26.762+0000 7f4da96cd640 1 -- 172.21.15.123:0/2574262264 shutdown_connections 2024-09-16T15:45:26.765 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:26.762+0000 7f4da96cd640 1 -- 172.21.15.123:0/2574262264 wait complete. 2024-09-16T15:45:26.781 DEBUG:teuthology.orchestra.run.smithi184:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.d.asok --format=json session ls 2024-09-16T15:45:26.930 INFO:teuthology.orchestra.run.smithi184.stdout:[{"id":5047,"entity":{"name":{"type":"client","num":5047},"addr":{"type":"any","addr":"172.21.15.184:0","nonce":908570948}},"state":"open","num_leases":0,"num_caps":7,"request_load_avg":9,"uptime":45.271229159000001,"requests_in_flight":0,"num_completed_requests":2,"num_completed_flushes":1,"reconnecting":false,"recall_caps":{"value":0,"halflife":60},"release_caps":{"value":0,"halflife":60},"recall_caps_throttle":{"value":0,"halflife":1.5},"recall_caps_throttle2o":{"value":0,"halflife":0.5},"session_cache_liveness":{"value":19.81584621370947,"halflife":300},"cap_acquisition":{"value":0,"halflife":30},"last_trim_completed_requests_tid":15,"last_trim_completed_flushes_tid":2,"delegated_inos":[{"start":"0x10000000005","length":500}],"inst":"client.5047 172.21.15.184:0/908570948","completed_requests":[{"tid":15,"created_ino":"0x0"},{"tid":16,"created_ino":"0x100000001f9"}],"prealloc_inos":[{"start":"0x10000000005","length":500},{"start":"0x100000001fa","length":1000}],"client_metadata":{"client_features":{"feature_bits":"0x00000000003fffff"},"metric_spec":{"metric_flags":{"feature_bits":"0x000000000000ffff"}},"ceph_sha1":"33f95d2ae1bcbe7eb04404303f179eeeaa6982df","ceph_version":"ceph version 19.3.0-4938-g33f95d2a (33f95d2ae1bcbe7eb04404303f179eeeaa6982df) squid (dev)","entity_id":"x","hostname":"smithi184","pid":"28840","root":"/"}}] 2024-09-16T15:45:26.930 DEBUG:tasks.cephfs.filesystem:_json_asok output [ { "cap_acquisition": { "halflife": 30, "value": 0 }, "client_metadata": { "ceph_sha1": "33f95d2ae1bcbe7eb04404303f179eeeaa6982df", "ceph_version": "ceph version 19.3.0-4938-g33f95d2a (33f95d2ae1bcbe7eb04404303f179eeeaa6982df) squid (dev)", "client_features": { "feature_bits": "0x00000000003fffff" }, "entity_id": "x", "hostname": "smithi184", "metric_spec": { "metric_flags": { "feature_bits": "0x000000000000ffff" } }, "pid": "28840", "root": "/" }, "completed_requests": [ { "created_ino": "0x0", "tid": 15 }, { "created_ino": "0x100000001f9", "tid": 16 } ], "delegated_inos": [ { "length": 500, "start": "0x10000000005" } ], "entity": { "addr": { "addr": "172.21.15.184:0", "nonce": 908570948, "type": "any" }, "name": { "num": 5047, "type": "client" } }, "id": 5047, "inst": "client.5047 172.21.15.184:0/908570948", "last_trim_completed_flushes_tid": 2, "last_trim_completed_requests_tid": 15, "num_caps": 7, "num_completed_flushes": 1, "num_completed_requests": 2, "num_leases": 0, "prealloc_inos": [ { "length": 500, "start": "0x10000000005" }, { "length": 1000, "start": "0x100000001fa" } ], "recall_caps": { "halflife": 60, "value": 0 }, "recall_caps_throttle": { "halflife": 1.5, "value": 0 }, "recall_caps_throttle2o": { "halflife": 0.5, "value": 0 }, "reconnecting": false, "release_caps": { "halflife": 60, "value": 0 }, "request_load_avg": 9, "requests_in_flight": 0, "session_cache_liveness": { "halflife": 300, "value": 19.81584621370947 }, "state": "open", "uptime": 45.271229159 } ] 2024-09-16T15:45:26.930 DEBUG:tasks.ceph_test_case:wait_until_true: waiting (timeout=90 retry_count=0)... 2024-09-16T15:45:31.931 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-16T15:45:32.068 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:32.066+0000 7fce947d3640 1 -- 172.21.15.123:0/3470671757 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fce8c15bca0 msgr2=0x7fce8c0d0630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:45:32.068 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:32.066+0000 7fce947d3640 1 --2- 172.21.15.123:0/3470671757 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fce8c15bca0 0x7fce8c0d0630 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7fce88009840 tx=0x7fce8801c760 comp rx=0 tx=0).stop 2024-09-16T15:45:32.068 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:32.066+0000 7fce947d3640 1 -- 172.21.15.123:0/3470671757 shutdown_connections 2024-09-16T15:45:32.068 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:32.066+0000 7fce947d3640 1 --2- 172.21.15.123:0/3470671757 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fce8c0726c0 0x7fce8c073ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:45:32.068 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:32.066+0000 7fce947d3640 1 --2- 172.21.15.123:0/3470671757 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fce8c15bca0 0x7fce8c0d0630 unknown :-1 s=CLOSED pgs=538 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:45:32.068 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:32.066+0000 7fce947d3640 1 -- 172.21.15.123:0/3470671757 >> 172.21.15.123:0/3470671757 conn(0x7fce8c0cfb10 msgr2=0x7fce8c0cff20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:45:32.068 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:32.066+0000 7fce947d3640 1 -- 172.21.15.123:0/3470671757 shutdown_connections 2024-09-16T15:45:32.068 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:32.066+0000 7fce947d3640 1 -- 172.21.15.123:0/3470671757 wait complete. 2024-09-16T15:45:32.069 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:32.066+0000 7fce947d3640 1 Processor -- start 2024-09-16T15:45:32.069 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:32.066+0000 7fce947d3640 1 -- start start 2024-09-16T15:45:32.071 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:32.066+0000 7fce947d3640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fce8c0726c0 0x7fce8c14e310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:45:32.071 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:32.066+0000 7fce947d3640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fce8c074430 0x7fce8c14e850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:45:32.071 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:32.066+0000 7fce947d3640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fce8c15bca0 0x7fce8c15c550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:45:32.071 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:32.066+0000 7fce947d3640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fce8c168ef0 con 0x7fce8c074430 2024-09-16T15:45:32.071 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:32.066+0000 7fce947d3640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fce8c168d70 con 0x7fce8c15bca0 2024-09-16T15:45:32.071 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:32.066+0000 7fce947d3640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fce8c169070 con 0x7fce8c0726c0 2024-09-16T15:45:32.071 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:32.066+0000 7fce91d6e640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fce8c074430 0x7fce8c14e850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:45:32.071 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:32.066+0000 7fce91d6e640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fce8c074430 0x7fce8c14e850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:47380/0 (socket says 172.21.15.123:47380) 2024-09-16T15:45:32.071 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:32.066+0000 7fce91d6e640 1 -- 172.21.15.123:0/4111080648 learned_addr learned my addr 172.21.15.123:0/4111080648 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:45:32.071 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:32.066+0000 7fce9256f640 1 --2- 172.21.15.123:0/4111080648 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fce8c0726c0 0x7fce8c14e310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:45:32.071 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:32.066+0000 7fce92d70640 1 --2- 172.21.15.123:0/4111080648 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fce8c15bca0 0x7fce8c15c550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:45:32.071 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:32.066+0000 7fce91d6e640 1 -- 172.21.15.123:0/4111080648 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fce8c0726c0 msgr2=0x7fce8c14e310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:45:32.071 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:32.066+0000 7fce91d6e640 1 --2- 172.21.15.123:0/4111080648 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fce8c0726c0 0x7fce8c14e310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:45:32.072 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:32.066+0000 7fce91d6e640 1 -- 172.21.15.123:0/4111080648 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fce8c15bca0 msgr2=0x7fce8c15c550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:45:32.072 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:32.066+0000 7fce91d6e640 1 --2- 172.21.15.123:0/4111080648 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fce8c15bca0 0x7fce8c15c550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:45:32.072 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:32.066+0000 7fce91d6e640 1 -- 172.21.15.123:0/4111080648 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fce8c15e9c0 con 0x7fce8c074430 2024-09-16T15:45:32.072 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:32.066+0000 7fce9256f640 1 --2- 172.21.15.123:0/4111080648 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fce8c0726c0 0x7fce8c14e310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T15:45:32.072 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:32.066+0000 7fce91d6e640 1 --2- 172.21.15.123:0/4111080648 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fce8c074430 0x7fce8c14e850 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7fce7c00b580 tx=0x7fce7c00ba40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:45:32.072 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:32.066+0000 7fce82ffd640 1 -- 172.21.15.123:0/4111080648 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce7c014070 con 0x7fce8c074430 2024-09-16T15:45:32.072 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:32.066+0000 7fce82ffd640 1 -- 172.21.15.123:0/4111080648 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fce7c010720 con 0x7fce8c074430 2024-09-16T15:45:32.072 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:32.066+0000 7fce947d3640 1 -- 172.21.15.123:0/4111080648 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fce8c15d220 con 0x7fce8c074430 2024-09-16T15:45:32.072 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:32.066+0000 7fce82ffd640 1 -- 172.21.15.123:0/4111080648 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce7c010b70 con 0x7fce8c074430 2024-09-16T15:45:32.072 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:32.066+0000 7fce947d3640 1 -- 172.21.15.123:0/4111080648 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fce8c15ef20 con 0x7fce8c074430 2024-09-16T15:45:32.073 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:32.070+0000 7fce947d3640 1 -- 172.21.15.123:0/4111080648 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fce8c073ef0 con 0x7fce8c074430 2024-09-16T15:45:32.073 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:32.070+0000 7fce82ffd640 1 -- 172.21.15.123:0/4111080648 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 8) ==== 97284+0+0 (secure 0 0 0) 0x7fce7c017070 con 0x7fce8c074430 2024-09-16T15:45:32.073 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:32.070+0000 7fce82ffd640 1 --2- 172.21.15.123:0/4111080648 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fce70073100 0x7fce700755c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:45:32.073 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:32.070+0000 7fce82ffd640 1 -- 172.21.15.123:0/4111080648 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(46..46 src has 1..46) ==== 5767+0+0 (secure 0 0 0) 0x7fce7c093e00 con 0x7fce8c074430 2024-09-16T15:45:32.073 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:32.070+0000 7fce9256f640 1 --2- 172.21.15.123:0/4111080648 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fce70073100 0x7fce700755c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:45:32.073 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:32.070+0000 7fce9256f640 1 --2- 172.21.15.123:0/4111080648 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fce70073100 0x7fce700755c0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fce880af000 tx=0x7fce8801afa0 comp rx=0 tx=0).ready entity=mgr.4135 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:45:32.077 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:32.074+0000 7fce82ffd640 1 -- 172.21.15.123:0/4111080648 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7fce8c15e9c0 con 0x7fce8c074430 2024-09-16T15:45:32.286 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:32.282+0000 7fce947d3640 1 -- 172.21.15.123:0/4111080648 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fce8c0c33e0 con 0x7fce8c074430 2024-09-16T15:45:32.287 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:32.286+0000 7fce82ffd640 1 -- 172.21.15.123:0/4111080648 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) ==== 94+0+5223 (secure 0 0 0) 0x7fce7c05eb10 con 0x7fce8c074430 2024-09-16T15:45:32.288 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-09-16T15:45:32.288 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":28,"btime":"2024-09-16T15:44:32:363512+0000","default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4712,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6837/288264117","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6836","nonce":288264117},{"type":"v1","addr":"172.21.15.184:6837","nonce":288264117}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":23},{"gid":5268,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6834/60194515","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6832","nonce":60194515},{"type":"v1","addr":"172.21.15.123:6834","nonce":60194515}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":23},{"gid":5271,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6835/1798373479","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6833","nonce":1798373479},{"type":"v1","addr":"172.21.15.123:6835","nonce":1798373479}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":22}],"filesystems":[{"mdsmap":{"epoch":27,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-16T15:44:14.853230+0000","modified":"2024-09-16T15:44:17.861742+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5280},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5280":{"gid":5280,"name":"d","rank":0,"incarnation":24,"state":"up:active","state_seq":3,"addr":"172.21.15.184:6835/374389412","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":374389412},{"type":"v1","addr":"172.21.15.184:6835","nonce":374389412}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5280,"qdb_cluster":[5280]},"id":3}]} 2024-09-16T15:45:32.288 INFO:teuthology.orchestra.run.smithi123.stderr:dumped fsmap epoch 28 2024-09-16T15:45:32.290 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:32.286+0000 7fce947d3640 1 -- 172.21.15.123:0/4111080648 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fce70073100 msgr2=0x7fce700755c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:45:32.290 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:32.286+0000 7fce947d3640 1 --2- 172.21.15.123:0/4111080648 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fce70073100 0x7fce700755c0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fce880af000 tx=0x7fce8801afa0 comp rx=0 tx=0).stop 2024-09-16T15:45:32.290 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:32.286+0000 7fce947d3640 1 -- 172.21.15.123:0/4111080648 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fce8c074430 msgr2=0x7fce8c14e850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:45:32.290 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:32.286+0000 7fce947d3640 1 --2- 172.21.15.123:0/4111080648 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fce8c074430 0x7fce8c14e850 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7fce7c00b580 tx=0x7fce7c00ba40 comp rx=0 tx=0).stop 2024-09-16T15:45:32.290 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:32.286+0000 7fce947d3640 1 -- 172.21.15.123:0/4111080648 shutdown_connections 2024-09-16T15:45:32.290 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:32.286+0000 7fce947d3640 1 --2- 172.21.15.123:0/4111080648 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fce70073100 0x7fce700755c0 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:45:32.290 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:32.286+0000 7fce947d3640 1 --2- 172.21.15.123:0/4111080648 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fce8c15bca0 0x7fce8c15c550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:45:32.290 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:32.286+0000 7fce947d3640 1 --2- 172.21.15.123:0/4111080648 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fce8c074430 0x7fce8c14e850 unknown :-1 s=CLOSED pgs=539 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:45:32.290 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:32.286+0000 7fce947d3640 1 --2- 172.21.15.123:0/4111080648 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fce8c0726c0 0x7fce8c14e310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:45:32.290 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:32.286+0000 7fce947d3640 1 -- 172.21.15.123:0/4111080648 >> 172.21.15.123:0/4111080648 conn(0x7fce8c0cfb10 msgr2=0x7fce8c0ca140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:45:32.291 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:32.286+0000 7fce947d3640 1 -- 172.21.15.123:0/4111080648 shutdown_connections 2024-09-16T15:45:32.291 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:32.286+0000 7fce947d3640 1 -- 172.21.15.123:0/4111080648 wait complete. 2024-09-16T15:45:32.307 DEBUG:teuthology.orchestra.run.smithi184:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.d.asok --format=json session ls 2024-09-16T15:45:32.458 INFO:teuthology.orchestra.run.smithi184.stdout:[{"id":5047,"entity":{"name":{"type":"client","num":5047},"addr":{"type":"any","addr":"172.21.15.184:0","nonce":908570948}},"state":"open","num_leases":0,"num_caps":7,"request_load_avg":8,"uptime":50.799088253999997,"requests_in_flight":0,"num_completed_requests":2,"num_completed_flushes":1,"reconnecting":false,"recall_caps":{"value":0,"halflife":60},"release_caps":{"value":0,"halflife":60},"recall_caps_throttle":{"value":0,"halflife":1.5},"recall_caps_throttle2o":{"value":0,"halflife":0.5},"session_cache_liveness":{"value":19.564366286101198,"halflife":300},"cap_acquisition":{"value":0,"halflife":30},"last_trim_completed_requests_tid":15,"last_trim_completed_flushes_tid":2,"delegated_inos":[{"start":"0x10000000005","length":500}],"inst":"client.5047 172.21.15.184:0/908570948","completed_requests":[{"tid":15,"created_ino":"0x0"},{"tid":16,"created_ino":"0x100000001f9"}],"prealloc_inos":[{"start":"0x10000000005","length":500},{"start":"0x100000001fa","length":1000}],"client_metadata":{"client_features":{"feature_bits":"0x00000000003fffff"},"metric_spec":{"metric_flags":{"feature_bits":"0x000000000000ffff"}},"ceph_sha1":"33f95d2ae1bcbe7eb04404303f179eeeaa6982df","ceph_version":"ceph version 19.3.0-4938-g33f95d2a (33f95d2ae1bcbe7eb04404303f179eeeaa6982df) squid (dev)","entity_id":"x","hostname":"smithi184","pid":"28840","root":"/"}}] 2024-09-16T15:45:32.458 DEBUG:tasks.cephfs.filesystem:_json_asok output [ { "cap_acquisition": { "halflife": 30, "value": 0 }, "client_metadata": { "ceph_sha1": "33f95d2ae1bcbe7eb04404303f179eeeaa6982df", "ceph_version": "ceph version 19.3.0-4938-g33f95d2a (33f95d2ae1bcbe7eb04404303f179eeeaa6982df) squid (dev)", "client_features": { "feature_bits": "0x00000000003fffff" }, "entity_id": "x", "hostname": "smithi184", "metric_spec": { "metric_flags": { "feature_bits": "0x000000000000ffff" } }, "pid": "28840", "root": "/" }, "completed_requests": [ { "created_ino": "0x0", "tid": 15 }, { "created_ino": "0x100000001f9", "tid": 16 } ], "delegated_inos": [ { "length": 500, "start": "0x10000000005" } ], "entity": { "addr": { "addr": "172.21.15.184:0", "nonce": 908570948, "type": "any" }, "name": { "num": 5047, "type": "client" } }, "id": 5047, "inst": "client.5047 172.21.15.184:0/908570948", "last_trim_completed_flushes_tid": 2, "last_trim_completed_requests_tid": 15, "num_caps": 7, "num_completed_flushes": 1, "num_completed_requests": 2, "num_leases": 0, "prealloc_inos": [ { "length": 500, "start": "0x10000000005" }, { "length": 1000, "start": "0x100000001fa" } ], "recall_caps": { "halflife": 60, "value": 0 }, "recall_caps_throttle": { "halflife": 1.5, "value": 0 }, "recall_caps_throttle2o": { "halflife": 0.5, "value": 0 }, "reconnecting": false, "release_caps": { "halflife": 60, "value": 0 }, "request_load_avg": 8, "requests_in_flight": 0, "session_cache_liveness": { "halflife": 300, "value": 19.564366286101198 }, "state": "open", "uptime": 50.799088254 } ] 2024-09-16T15:45:32.458 DEBUG:tasks.ceph_test_case:wait_until_true: waiting (timeout=90 retry_count=0)... 2024-09-16T15:45:37.459 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-16T15:45:37.593 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:37.590+0000 7fd93b1c6640 1 -- 172.21.15.123:0/3922262499 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd934158120 msgr2=0x7fd9340ca320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:45:37.593 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:37.590+0000 7fd93b1c6640 1 --2- 172.21.15.123:0/3922262499 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd934158120 0x7fd9340ca320 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7fd928009840 tx=0x7fd92801c790 comp rx=0 tx=0).stop 2024-09-16T15:45:37.593 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:37.590+0000 7fd93b1c6640 1 -- 172.21.15.123:0/3922262499 shutdown_connections 2024-09-16T15:45:37.593 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:37.590+0000 7fd93b1c6640 1 --2- 172.21.15.123:0/3922262499 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd934158120 0x7fd9340ca320 unknown :-1 s=CLOSED pgs=541 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:45:37.593 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:37.590+0000 7fd93b1c6640 1 -- 172.21.15.123:0/3922262499 >> 172.21.15.123:0/3922262499 conn(0x7fd9340cfb10 msgr2=0x7fd9340cff20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:45:37.593 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:37.590+0000 7fd93b1c6640 1 -- 172.21.15.123:0/3922262499 shutdown_connections 2024-09-16T15:45:37.593 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:37.590+0000 7fd93b1c6640 1 -- 172.21.15.123:0/3922262499 wait complete. 2024-09-16T15:45:37.594 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:37.590+0000 7fd93b1c6640 1 Processor -- start 2024-09-16T15:45:37.594 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:37.590+0000 7fd93b1c6640 1 -- start start 2024-09-16T15:45:37.596 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:37.590+0000 7fd93b1c6640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd93406f5f0 0x7fd9340b4450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:45:37.596 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:37.590+0000 7fd93b1c6640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd9340cf5a0 0x7fd9341fb960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:45:37.596 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:37.590+0000 7fd93b1c6640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fd934158120 0x7fd9340cc520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:45:37.596 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:37.590+0000 7fd93b1c6640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fd934168560 con 0x7fd93406f5f0 2024-09-16T15:45:37.596 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:37.590+0000 7fd93b1c6640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fd9341683e0 con 0x7fd9340cf5a0 2024-09-16T15:45:37.596 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:37.590+0000 7fd93b1c6640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fd9341686e0 con 0x7fd934158120 2024-09-16T15:45:37.596 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:37.590+0000 7fd938f62640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd93406f5f0 0x7fd9340b4450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:45:37.596 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:37.594+0000 7fd938f62640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd93406f5f0 0x7fd9340b4450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:34056/0 (socket says 172.21.15.123:34056) 2024-09-16T15:45:37.597 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:37.594+0000 7fd938f62640 1 -- 172.21.15.123:0/431012932 learned_addr learned my addr 172.21.15.123:0/431012932 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:45:37.597 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:37.594+0000 7fd933fff640 1 --2- 172.21.15.123:0/431012932 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd9340cf5a0 0x7fd9341fb960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:45:37.597 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:37.594+0000 7fd939763640 1 --2- 172.21.15.123:0/431012932 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fd934158120 0x7fd9340cc520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:45:37.597 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:37.594+0000 7fd938f62640 1 -- 172.21.15.123:0/431012932 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fd934158120 msgr2=0x7fd9340cc520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:45:37.597 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:37.594+0000 7fd938f62640 1 --2- 172.21.15.123:0/431012932 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fd934158120 0x7fd9340cc520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:45:37.597 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:37.594+0000 7fd938f62640 1 -- 172.21.15.123:0/431012932 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd9340cf5a0 msgr2=0x7fd9341fb960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:45:37.597 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:37.594+0000 7fd938f62640 1 --2- 172.21.15.123:0/431012932 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd9340cf5a0 0x7fd9341fb960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:45:37.597 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:37.594+0000 7fd938f62640 1 -- 172.21.15.123:0/431012932 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd9340b4b50 con 0x7fd93406f5f0 2024-09-16T15:45:37.597 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:37.594+0000 7fd939763640 1 --2- 172.21.15.123:0/431012932 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fd934158120 0x7fd9340cc520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T15:45:37.597 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:37.594+0000 7fd938f62640 1 --2- 172.21.15.123:0/431012932 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd93406f5f0 0x7fd9340b4450 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7fd928000c10 tx=0x7fd9280aafd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:45:37.597 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:37.594+0000 7fd9317fa640 1 -- 172.21.15.123:0/431012932 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9280a8a80 con 0x7fd93406f5f0 2024-09-16T15:45:37.597 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:37.594+0000 7fd9317fa640 1 -- 172.21.15.123:0/431012932 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fd9280a8c20 con 0x7fd93406f5f0 2024-09-16T15:45:37.597 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:37.594+0000 7fd9317fa640 1 -- 172.21.15.123:0/431012932 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9280051c0 con 0x7fd93406f5f0 2024-09-16T15:45:37.598 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:37.594+0000 7fd93b1c6640 1 -- 172.21.15.123:0/431012932 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd9340b4e30 con 0x7fd93406f5f0 2024-09-16T15:45:37.598 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:37.594+0000 7fd93b1c6640 1 -- 172.21.15.123:0/431012932 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd9340b5110 con 0x7fd93406f5f0 2024-09-16T15:45:37.598 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:37.594+0000 7fd93b1c6640 1 -- 172.21.15.123:0/431012932 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd9340ca320 con 0x7fd93406f5f0 2024-09-16T15:45:37.599 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:37.594+0000 7fd9317fa640 1 -- 172.21.15.123:0/431012932 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 8) ==== 97284+0+0 (secure 0 0 0) 0x7fd928005380 con 0x7fd93406f5f0 2024-09-16T15:45:37.599 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:37.594+0000 7fd9317fa640 1 --2- 172.21.15.123:0/431012932 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fd920073100 0x7fd9200755c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:45:37.599 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:37.594+0000 7fd9317fa640 1 -- 172.21.15.123:0/431012932 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(46..46 src has 1..46) ==== 5767+0+0 (secure 0 0 0) 0x7fd92812e900 con 0x7fd93406f5f0 2024-09-16T15:45:37.599 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:37.594+0000 7fd933fff640 1 --2- 172.21.15.123:0/431012932 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fd920073100 0x7fd9200755c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:45:37.599 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:37.598+0000 7fd933fff640 1 --2- 172.21.15.123:0/431012932 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fd920073100 0x7fd9200755c0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fd9340cf260 tx=0x7fd924009040 comp rx=0 tx=0).ready entity=mgr.4135 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:45:37.602 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:37.598+0000 7fd9317fa640 1 -- 172.21.15.123:0/431012932 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7fd9280ac030 con 0x7fd93406f5f0 2024-09-16T15:45:37.800 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:37.798+0000 7fd93b1c6640 1 -- 172.21.15.123:0/431012932 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd934157590 con 0x7fd93406f5f0 2024-09-16T15:45:37.801 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:37.798+0000 7fd9317fa640 1 -- 172.21.15.123:0/431012932 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) ==== 94+0+5223 (secure 0 0 0) 0x7fd928016410 con 0x7fd93406f5f0 2024-09-16T15:45:37.801 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-09-16T15:45:37.801 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":28,"btime":"2024-09-16T15:44:32:363512+0000","default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4712,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6837/288264117","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6836","nonce":288264117},{"type":"v1","addr":"172.21.15.184:6837","nonce":288264117}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":23},{"gid":5268,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6834/60194515","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6832","nonce":60194515},{"type":"v1","addr":"172.21.15.123:6834","nonce":60194515}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":23},{"gid":5271,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6835/1798373479","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6833","nonce":1798373479},{"type":"v1","addr":"172.21.15.123:6835","nonce":1798373479}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":22}],"filesystems":[{"mdsmap":{"epoch":27,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-16T15:44:14.853230+0000","modified":"2024-09-16T15:44:17.861742+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5280},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5280":{"gid":5280,"name":"d","rank":0,"incarnation":24,"state":"up:active","state_seq":3,"addr":"172.21.15.184:6835/374389412","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":374389412},{"type":"v1","addr":"172.21.15.184:6835","nonce":374389412}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5280,"qdb_cluster":[5280]},"id":3}]} 2024-09-16T15:45:37.801 INFO:teuthology.orchestra.run.smithi123.stderr:dumped fsmap epoch 28 2024-09-16T15:45:37.803 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:37.802+0000 7fd93b1c6640 1 -- 172.21.15.123:0/431012932 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fd920073100 msgr2=0x7fd9200755c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:45:37.804 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:37.802+0000 7fd93b1c6640 1 --2- 172.21.15.123:0/431012932 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fd920073100 0x7fd9200755c0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fd9340cf260 tx=0x7fd924009040 comp rx=0 tx=0).stop 2024-09-16T15:45:37.804 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:37.802+0000 7fd93b1c6640 1 -- 172.21.15.123:0/431012932 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd93406f5f0 msgr2=0x7fd9340b4450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:45:37.804 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:37.802+0000 7fd93b1c6640 1 --2- 172.21.15.123:0/431012932 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd93406f5f0 0x7fd9340b4450 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7fd928000c10 tx=0x7fd9280aafd0 comp rx=0 tx=0).stop 2024-09-16T15:45:37.804 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:37.802+0000 7fd93b1c6640 1 -- 172.21.15.123:0/431012932 shutdown_connections 2024-09-16T15:45:37.804 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:37.802+0000 7fd93b1c6640 1 --2- 172.21.15.123:0/431012932 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fd920073100 0x7fd9200755c0 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:45:37.804 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:37.802+0000 7fd93b1c6640 1 --2- 172.21.15.123:0/431012932 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fd934158120 0x7fd9340cc520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:45:37.804 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:37.802+0000 7fd93b1c6640 1 --2- 172.21.15.123:0/431012932 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd9340cf5a0 0x7fd9341fb960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:45:37.804 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:37.802+0000 7fd93b1c6640 1 --2- 172.21.15.123:0/431012932 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd93406f5f0 0x7fd9340b4450 unknown :-1 s=CLOSED pgs=542 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:45:37.804 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:37.802+0000 7fd93b1c6640 1 -- 172.21.15.123:0/431012932 >> 172.21.15.123:0/431012932 conn(0x7fd9340cfb10 msgr2=0x7fd934072a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:45:37.804 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:37.802+0000 7fd93b1c6640 1 -- 172.21.15.123:0/431012932 shutdown_connections 2024-09-16T15:45:37.804 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:37.802+0000 7fd93b1c6640 1 -- 172.21.15.123:0/431012932 wait complete. 2024-09-16T15:45:37.819 DEBUG:teuthology.orchestra.run.smithi184:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.d.asok --format=json session ls 2024-09-16T15:45:37.970 INFO:teuthology.orchestra.run.smithi184.stdout:[{"id":5047,"entity":{"name":{"type":"client","num":5047},"addr":{"type":"any","addr":"172.21.15.184:0","nonce":908570948}},"state":"open","num_leases":0,"num_caps":7,"request_load_avg":8,"uptime":56.314945022000003,"requests_in_flight":0,"num_completed_requests":2,"num_completed_flushes":1,"reconnecting":false,"recall_caps":{"value":0,"halflife":60},"release_caps":{"value":0,"halflife":60},"recall_caps_throttle":{"value":0,"halflife":1.5},"recall_caps_throttle2o":{"value":0,"halflife":0.5},"session_cache_liveness":{"value":19.316613519154032,"halflife":300},"cap_acquisition":{"value":0,"halflife":30},"last_trim_completed_requests_tid":15,"last_trim_completed_flushes_tid":2,"delegated_inos":[{"start":"0x10000000005","length":500}],"inst":"client.5047 172.21.15.184:0/908570948","completed_requests":[{"tid":15,"created_ino":"0x0"},{"tid":16,"created_ino":"0x100000001f9"}],"prealloc_inos":[{"start":"0x10000000005","length":500},{"start":"0x100000001fa","length":1000}],"client_metadata":{"client_features":{"feature_bits":"0x00000000003fffff"},"metric_spec":{"metric_flags":{"feature_bits":"0x000000000000ffff"}},"ceph_sha1":"33f95d2ae1bcbe7eb04404303f179eeeaa6982df","ceph_version":"ceph version 19.3.0-4938-g33f95d2a (33f95d2ae1bcbe7eb04404303f179eeeaa6982df) squid (dev)","entity_id":"x","hostname":"smithi184","pid":"28840","root":"/"}}] 2024-09-16T15:45:37.970 DEBUG:tasks.cephfs.filesystem:_json_asok output [ { "cap_acquisition": { "halflife": 30, "value": 0 }, "client_metadata": { "ceph_sha1": "33f95d2ae1bcbe7eb04404303f179eeeaa6982df", "ceph_version": "ceph version 19.3.0-4938-g33f95d2a (33f95d2ae1bcbe7eb04404303f179eeeaa6982df) squid (dev)", "client_features": { "feature_bits": "0x00000000003fffff" }, "entity_id": "x", "hostname": "smithi184", "metric_spec": { "metric_flags": { "feature_bits": "0x000000000000ffff" } }, "pid": "28840", "root": "/" }, "completed_requests": [ { "created_ino": "0x0", "tid": 15 }, { "created_ino": "0x100000001f9", "tid": 16 } ], "delegated_inos": [ { "length": 500, "start": "0x10000000005" } ], "entity": { "addr": { "addr": "172.21.15.184:0", "nonce": 908570948, "type": "any" }, "name": { "num": 5047, "type": "client" } }, "id": 5047, "inst": "client.5047 172.21.15.184:0/908570948", "last_trim_completed_flushes_tid": 2, "last_trim_completed_requests_tid": 15, "num_caps": 7, "num_completed_flushes": 1, "num_completed_requests": 2, "num_leases": 0, "prealloc_inos": [ { "length": 500, "start": "0x10000000005" }, { "length": 1000, "start": "0x100000001fa" } ], "recall_caps": { "halflife": 60, "value": 0 }, "recall_caps_throttle": { "halflife": 1.5, "value": 0 }, "recall_caps_throttle2o": { "halflife": 0.5, "value": 0 }, "reconnecting": false, "release_caps": { "halflife": 60, "value": 0 }, "request_load_avg": 8, "requests_in_flight": 0, "session_cache_liveness": { "halflife": 300, "value": 19.316613519154032 }, "state": "open", "uptime": 56.314945022 } ] 2024-09-16T15:45:37.970 DEBUG:tasks.ceph_test_case:wait_until_true: waiting (timeout=90 retry_count=0)... 2024-09-16T15:45:39.057 DEBUG:teuthology.orchestra.run.smithi123:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-16T15:45:39.059 DEBUG:teuthology.orchestra.run.smithi184:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-16T15:45:42.971 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-16T15:45:43.105 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:43.102+0000 7f3bc8661640 1 -- 172.21.15.123:0/2238374035 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3bac004810 msgr2=0x7f3bac004c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:45:43.105 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:43.102+0000 7f3bc8661640 1 --2- 172.21.15.123:0/2238374035 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3bac004810 0x7f3bac004c90 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7f3bb0009b60 tx=0x7f3bb001d280 comp rx=0 tx=0).stop 2024-09-16T15:45:43.105 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:43.102+0000 7f3bc6bfe640 1 -- 172.21.15.123:0/2238374035 reap_dead start 2024-09-16T15:45:43.105 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:43.102+0000 7f3bc8661640 1 -- 172.21.15.123:0/2238374035 shutdown_connections 2024-09-16T15:45:43.106 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:43.102+0000 7f3bc8661640 1 -- 172.21.15.123:0/2238374035 >> 172.21.15.123:0/2238374035 conn(0x7f3bc00cfb10 msgr2=0x7f3bc00cff20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:45:43.106 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:43.102+0000 7f3bc8661640 1 -- 172.21.15.123:0/2238374035 shutdown_connections 2024-09-16T15:45:43.106 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:43.102+0000 7f3bc8661640 1 -- 172.21.15.123:0/2238374035 wait complete. 2024-09-16T15:45:43.106 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:43.102+0000 7f3bc8661640 1 Processor -- start 2024-09-16T15:45:43.106 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:43.102+0000 7f3bc8661640 1 -- start start 2024-09-16T15:45:43.109 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:43.106+0000 7f3bc8661640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3bc015ab50 0x7f3bc0158ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:45:43.109 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:43.106+0000 7f3bc8661640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3bc01f7e40 0x7f3bc00cf050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:45:43.109 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:43.106+0000 7f3bc8661640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f3bc0157d90 0x7f3bc0158210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:45:43.109 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:43.106+0000 7f3bc8661640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f3bc0166c50 con 0x7f3bc01f7e40 2024-09-16T15:45:43.109 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:43.106+0000 7f3bc8661640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f3bc0074930 con 0x7f3bc015ab50 2024-09-16T15:45:43.109 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:43.106+0000 7f3bc8661640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f3bc0166dd0 con 0x7f3bc0157d90 2024-09-16T15:45:43.109 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:43.106+0000 7f3bc5bfc640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3bc01f7e40 0x7f3bc00cf050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:45:43.109 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:43.106+0000 7f3bc5bfc640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3bc01f7e40 0x7f3bc00cf050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:56720/0 (socket says 172.21.15.123:56720) 2024-09-16T15:45:43.109 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:43.106+0000 7f3bc5bfc640 1 -- 172.21.15.123:0/3478782489 learned_addr learned my addr 172.21.15.123:0/3478782489 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:45:43.109 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:43.106+0000 7f3bc63fd640 1 --2- 172.21.15.123:0/3478782489 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3bc015ab50 0x7f3bc0158ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:45:43.109 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:43.106+0000 7f3bc6bfe640 1 --2- 172.21.15.123:0/3478782489 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f3bc0157d90 0x7f3bc0158210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:45:43.110 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:43.106+0000 7f3bc5bfc640 1 -- 172.21.15.123:0/3478782489 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f3bc0157d90 msgr2=0x7f3bc0158210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:45:43.110 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:43.106+0000 7f3bc5bfc640 1 --2- 172.21.15.123:0/3478782489 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f3bc0157d90 0x7f3bc0158210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:45:43.110 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:43.106+0000 7f3bc5bfc640 1 -- 172.21.15.123:0/3478782489 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3bc015ab50 msgr2=0x7f3bc0158ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:45:43.110 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:43.106+0000 7f3bc5bfc640 1 --2- 172.21.15.123:0/3478782489 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3bc015ab50 0x7f3bc0158ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:45:43.110 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:43.106+0000 7f3bc5bfc640 1 -- 172.21.15.123:0/3478782489 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3bc01f8390 con 0x7f3bc01f7e40 2024-09-16T15:45:43.110 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:43.106+0000 7f3bc6bfe640 1 --2- 172.21.15.123:0/3478782489 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f3bc0157d90 0x7f3bc0158210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T15:45:43.110 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:43.106+0000 7f3bc5bfc640 1 --2- 172.21.15.123:0/3478782489 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3bc01f7e40 0x7f3bc00cf050 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7f3bb0009a80 tx=0x7f3bb0005640 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:45:43.110 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:43.106+0000 7f3bb6ffd640 1 -- 172.21.15.123:0/3478782489 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3bb00ba020 con 0x7f3bc01f7e40 2024-09-16T15:45:43.110 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:43.106+0000 7f3bb6ffd640 1 -- 172.21.15.123:0/3478782489 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f3bb00ac8a0 con 0x7f3bc01f7e40 2024-09-16T15:45:43.110 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:43.106+0000 7f3bc8661640 1 -- 172.21.15.123:0/3478782489 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3bc01f8670 con 0x7f3bc01f7e40 2024-09-16T15:45:43.110 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:43.106+0000 7f3bb6ffd640 1 -- 172.21.15.123:0/3478782489 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3bb00b1040 con 0x7f3bc01f7e40 2024-09-16T15:45:43.110 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:43.106+0000 7f3bc8661640 1 -- 172.21.15.123:0/3478782489 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3bc01f8c30 con 0x7f3bc01f7e40 2024-09-16T15:45:43.110 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:43.106+0000 7f3bc8661640 1 -- 172.21.15.123:0/3478782489 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3bc0158750 con 0x7f3bc01f7e40 2024-09-16T15:45:43.111 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:43.106+0000 7f3bb6ffd640 1 -- 172.21.15.123:0/3478782489 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 8) ==== 97284+0+0 (secure 0 0 0) 0x7f3bb00bd020 con 0x7f3bc01f7e40 2024-09-16T15:45:43.111 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:43.106+0000 7f3bb6ffd640 1 --2- 172.21.15.123:0/3478782489 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f3ba4073130 0x7f3ba40755f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:45:43.111 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:43.106+0000 7f3bb6ffd640 1 -- 172.21.15.123:0/3478782489 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(46..46 src has 1..46) ==== 5767+0+0 (secure 0 0 0) 0x7f3bb012efd0 con 0x7f3bc01f7e40 2024-09-16T15:45:43.111 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:43.106+0000 7f3bc63fd640 1 --2- 172.21.15.123:0/3478782489 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f3ba4073130 0x7f3ba40755f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:45:43.112 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:43.110+0000 7f3bc63fd640 1 --2- 172.21.15.123:0/3478782489 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f3ba4073130 0x7f3ba40755f0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f3bbc009470 tx=0x7f3bbc0093a0 comp rx=0 tx=0).ready entity=mgr.4135 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:45:43.114 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:43.110+0000 7f3bb6ffd640 1 -- 172.21.15.123:0/3478782489 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f3bb00c4030 con 0x7f3bc01f7e40 2024-09-16T15:45:43.314 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:43.310+0000 7f3bc8661640 1 -- 172.21.15.123:0/3478782489 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f3bc00d0630 con 0x7f3bc01f7e40 2024-09-16T15:45:43.315 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:43.310+0000 7f3bb6ffd640 1 -- 172.21.15.123:0/3478782489 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) ==== 94+0+5223 (secure 0 0 0) 0x7f3bb0016fc0 con 0x7f3bc01f7e40 2024-09-16T15:45:43.315 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-09-16T15:45:43.315 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":28,"btime":"2024-09-16T15:44:32:363512+0000","default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4712,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6837/288264117","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6836","nonce":288264117},{"type":"v1","addr":"172.21.15.184:6837","nonce":288264117}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":23},{"gid":5268,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6834/60194515","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6832","nonce":60194515},{"type":"v1","addr":"172.21.15.123:6834","nonce":60194515}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":23},{"gid":5271,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6835/1798373479","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6833","nonce":1798373479},{"type":"v1","addr":"172.21.15.123:6835","nonce":1798373479}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":22}],"filesystems":[{"mdsmap":{"epoch":27,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-16T15:44:14.853230+0000","modified":"2024-09-16T15:44:17.861742+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5280},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5280":{"gid":5280,"name":"d","rank":0,"incarnation":24,"state":"up:active","state_seq":3,"addr":"172.21.15.184:6835/374389412","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":374389412},{"type":"v1","addr":"172.21.15.184:6835","nonce":374389412}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5280,"qdb_cluster":[5280]},"id":3}]} 2024-09-16T15:45:43.315 INFO:teuthology.orchestra.run.smithi123.stderr:dumped fsmap epoch 28 2024-09-16T15:45:43.317 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:43.314+0000 7f3bc8661640 1 -- 172.21.15.123:0/3478782489 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f3ba4073130 msgr2=0x7f3ba40755f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:45:43.317 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:43.314+0000 7f3bc8661640 1 --2- 172.21.15.123:0/3478782489 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f3ba4073130 0x7f3ba40755f0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f3bbc009470 tx=0x7f3bbc0093a0 comp rx=0 tx=0).stop 2024-09-16T15:45:43.317 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:43.314+0000 7f3bc8661640 1 -- 172.21.15.123:0/3478782489 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3bc01f7e40 msgr2=0x7f3bc00cf050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:45:43.318 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:43.314+0000 7f3bc8661640 1 --2- 172.21.15.123:0/3478782489 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3bc01f7e40 0x7f3bc00cf050 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7f3bb0009a80 tx=0x7f3bb0005640 comp rx=0 tx=0).stop 2024-09-16T15:45:43.318 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:43.314+0000 7f3bc8661640 1 -- 172.21.15.123:0/3478782489 shutdown_connections 2024-09-16T15:45:43.318 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:43.314+0000 7f3bc8661640 1 --2- 172.21.15.123:0/3478782489 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f3ba4073130 0x7f3ba40755f0 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:45:43.318 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:43.314+0000 7f3bc8661640 1 --2- 172.21.15.123:0/3478782489 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f3bc0157d90 0x7f3bc0158210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:45:43.318 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:43.314+0000 7f3bc8661640 1 --2- 172.21.15.123:0/3478782489 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3bc01f7e40 0x7f3bc00cf050 unknown :-1 s=CLOSED pgs=545 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:45:43.318 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:43.314+0000 7f3bc8661640 1 --2- 172.21.15.123:0/3478782489 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3bc015ab50 0x7f3bc0158ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:45:43.318 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:43.314+0000 7f3bc8661640 1 -- 172.21.15.123:0/3478782489 >> 172.21.15.123:0/3478782489 conn(0x7f3bc00cfb10 msgr2=0x7f3bc00cff20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:45:43.318 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:43.314+0000 7f3bc8661640 1 -- 172.21.15.123:0/3478782489 shutdown_connections 2024-09-16T15:45:43.318 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:43.314+0000 7f3bc8661640 1 -- 172.21.15.123:0/3478782489 wait complete. 2024-09-16T15:45:43.333 DEBUG:teuthology.orchestra.run.smithi184:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.d.asok --format=json session ls 2024-09-16T15:45:43.485 INFO:teuthology.orchestra.run.smithi184.stdout:[{"id":5047,"entity":{"name":{"type":"client","num":5047},"addr":{"type":"any","addr":"172.21.15.184:0","nonce":908570948}},"state":"open","num_leases":0,"num_caps":7,"request_load_avg":7,"uptime":61.826816977,"requests_in_flight":0,"num_completed_requests":2,"num_completed_flushes":1,"reconnecting":false,"recall_caps":{"value":0,"halflife":60},"release_caps":{"value":0,"halflife":60},"recall_caps_throttle":{"value":0,"halflife":1.5},"recall_caps_throttle2o":{"value":0,"halflife":0.5},"session_cache_liveness":{"value":19.072173756229446,"halflife":300},"cap_acquisition":{"value":0,"halflife":30},"last_trim_completed_requests_tid":15,"last_trim_completed_flushes_tid":2,"delegated_inos":[{"start":"0x10000000005","length":500}],"inst":"client.5047 172.21.15.184:0/908570948","completed_requests":[{"tid":15,"created_ino":"0x0"},{"tid":16,"created_ino":"0x100000001f9"}],"prealloc_inos":[{"start":"0x10000000005","length":500},{"start":"0x100000001fa","length":1000}],"client_metadata":{"client_features":{"feature_bits":"0x00000000003fffff"},"metric_spec":{"metric_flags":{"feature_bits":"0x000000000000ffff"}},"ceph_sha1":"33f95d2ae1bcbe7eb04404303f179eeeaa6982df","ceph_version":"ceph version 19.3.0-4938-g33f95d2a (33f95d2ae1bcbe7eb04404303f179eeeaa6982df) squid (dev)","entity_id":"x","hostname":"smithi184","pid":"28840","root":"/"}}] 2024-09-16T15:45:43.485 DEBUG:tasks.cephfs.filesystem:_json_asok output [ { "cap_acquisition": { "halflife": 30, "value": 0 }, "client_metadata": { "ceph_sha1": "33f95d2ae1bcbe7eb04404303f179eeeaa6982df", "ceph_version": "ceph version 19.3.0-4938-g33f95d2a (33f95d2ae1bcbe7eb04404303f179eeeaa6982df) squid (dev)", "client_features": { "feature_bits": "0x00000000003fffff" }, "entity_id": "x", "hostname": "smithi184", "metric_spec": { "metric_flags": { "feature_bits": "0x000000000000ffff" } }, "pid": "28840", "root": "/" }, "completed_requests": [ { "created_ino": "0x0", "tid": 15 }, { "created_ino": "0x100000001f9", "tid": 16 } ], "delegated_inos": [ { "length": 500, "start": "0x10000000005" } ], "entity": { "addr": { "addr": "172.21.15.184:0", "nonce": 908570948, "type": "any" }, "name": { "num": 5047, "type": "client" } }, "id": 5047, "inst": "client.5047 172.21.15.184:0/908570948", "last_trim_completed_flushes_tid": 2, "last_trim_completed_requests_tid": 15, "num_caps": 7, "num_completed_flushes": 1, "num_completed_requests": 2, "num_leases": 0, "prealloc_inos": [ { "length": 500, "start": "0x10000000005" }, { "length": 1000, "start": "0x100000001fa" } ], "recall_caps": { "halflife": 60, "value": 0 }, "recall_caps_throttle": { "halflife": 1.5, "value": 0 }, "recall_caps_throttle2o": { "halflife": 0.5, "value": 0 }, "reconnecting": false, "release_caps": { "halflife": 60, "value": 0 }, "request_load_avg": 7, "requests_in_flight": 0, "session_cache_liveness": { "halflife": 300, "value": 19.072173756229446 }, "state": "open", "uptime": 61.826816977 } ] 2024-09-16T15:45:43.486 DEBUG:tasks.ceph_test_case:wait_until_true: waiting (timeout=90 retry_count=0)... 2024-09-16T15:45:48.487 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-16T15:45:48.621 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:48.618+0000 7fbfaf741640 1 -- 172.21.15.123:0/2963917943 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbfa80726c0 msgr2=0x7fbfa8073ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:45:48.621 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:48.618+0000 7fbfaf741640 1 --2- 172.21.15.123:0/2963917943 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbfa80726c0 0x7fbfa8073ef0 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7fbf98009840 tx=0x7fbf9801c760 comp rx=0 tx=0).stop 2024-09-16T15:45:48.621 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:48.618+0000 7fbfaf741640 1 -- 172.21.15.123:0/2963917943 shutdown_connections 2024-09-16T15:45:48.621 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:48.618+0000 7fbfaf741640 1 --2- 172.21.15.123:0/2963917943 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbfa80726c0 0x7fbfa8073ef0 unknown :-1 s=CLOSED pgs=547 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:45:48.621 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:48.618+0000 7fbfaf741640 1 --2- 172.21.15.123:0/2963917943 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fbfa815bca0 0x7fbfa80d0630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:45:48.621 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:48.618+0000 7fbfaf741640 1 -- 172.21.15.123:0/2963917943 >> 172.21.15.123:0/2963917943 conn(0x7fbfa80cfb10 msgr2=0x7fbfa80cff20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:45:48.621 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:48.618+0000 7fbfaf741640 1 -- 172.21.15.123:0/2963917943 shutdown_connections 2024-09-16T15:45:48.621 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:48.618+0000 7fbfaf741640 1 -- 172.21.15.123:0/2963917943 wait complete. 2024-09-16T15:45:48.622 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:48.618+0000 7fbfaf741640 1 Processor -- start 2024-09-16T15:45:48.622 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:48.618+0000 7fbfaf741640 1 -- start start 2024-09-16T15:45:48.624 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:48.618+0000 7fbfaf741640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fbfa80726c0 0x7fbfa8156e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:45:48.624 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:48.618+0000 7fbfaf741640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fbfa8074430 0x7fbfa80c5950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:45:48.624 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:48.618+0000 7fbfaf741640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbfa815bca0 0x7fbfa80c5e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:45:48.624 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:48.618+0000 7fbfaf741640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fbfa8168e70 con 0x7fbfa815bca0 2024-09-16T15:45:48.624 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:48.618+0000 7fbfaf741640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fbfa8168cf0 con 0x7fbfa80726c0 2024-09-16T15:45:48.624 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:48.618+0000 7fbfaf741640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fbfa8168ff0 con 0x7fbfa8074430 2024-09-16T15:45:48.624 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:48.622+0000 7fbfadcde640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbfa815bca0 0x7fbfa80c5e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:45:48.624 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:48.622+0000 7fbfadcde640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbfa815bca0 0x7fbfa80c5e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:56746/0 (socket says 172.21.15.123:56746) 2024-09-16T15:45:48.624 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:48.622+0000 7fbfadcde640 1 -- 172.21.15.123:0/4275463585 learned_addr learned my addr 172.21.15.123:0/4275463585 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:45:48.625 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:48.622+0000 7fbfaccdc640 1 --2- 172.21.15.123:0/4275463585 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fbfa8074430 0x7fbfa80c5950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:45:48.625 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:48.622+0000 7fbfad4dd640 1 --2- 172.21.15.123:0/4275463585 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fbfa80726c0 0x7fbfa8156e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:45:48.625 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:48.622+0000 7fbfadcde640 1 -- 172.21.15.123:0/4275463585 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fbfa8074430 msgr2=0x7fbfa80c5950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:45:48.625 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:48.622+0000 7fbfadcde640 1 --2- 172.21.15.123:0/4275463585 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fbfa8074430 0x7fbfa80c5950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:45:48.625 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:48.622+0000 7fbfadcde640 1 -- 172.21.15.123:0/4275463585 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fbfa80726c0 msgr2=0x7fbfa8156e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:45:48.625 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:48.622+0000 7fbfadcde640 1 --2- 172.21.15.123:0/4275463585 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fbfa80726c0 0x7fbfa8156e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:45:48.625 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:48.622+0000 7fbfadcde640 1 -- 172.21.15.123:0/4275463585 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbfa81506f0 con 0x7fbfa815bca0 2024-09-16T15:45:48.625 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:48.622+0000 7fbfadcde640 1 --2- 172.21.15.123:0/4275463585 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbfa815bca0 0x7fbfa80c5e90 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7fbfa000cbc0 tx=0x7fbfa000ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:45:48.625 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:48.622+0000 7fbf9dffb640 1 -- 172.21.15.123:0/4275463585 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbfa0016020 con 0x7fbfa815bca0 2024-09-16T15:45:48.625 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:48.622+0000 7fbf9dffb640 1 -- 172.21.15.123:0/4275463585 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fbfa000ab40 con 0x7fbfa815bca0 2024-09-16T15:45:48.625 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:48.622+0000 7fbf9dffb640 1 -- 172.21.15.123:0/4275463585 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbfa0012420 con 0x7fbfa815bca0 2024-09-16T15:45:48.625 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:48.622+0000 7fbfaf741640 1 -- 172.21.15.123:0/4275463585 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbfa81509d0 con 0x7fbfa815bca0 2024-09-16T15:45:48.626 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:48.622+0000 7fbfaf741640 1 -- 172.21.15.123:0/4275463585 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbfa814cc90 con 0x7fbfa815bca0 2024-09-16T15:45:48.626 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:48.622+0000 7fbfaf741640 1 -- 172.21.15.123:0/4275463585 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbfa8073ef0 con 0x7fbfa815bca0 2024-09-16T15:45:48.626 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:48.622+0000 7fbf9dffb640 1 -- 172.21.15.123:0/4275463585 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 8) ==== 97284+0+0 (secure 0 0 0) 0x7fbfa0019020 con 0x7fbfa815bca0 2024-09-16T15:45:48.626 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:48.622+0000 7fbf9dffb640 1 --2- 172.21.15.123:0/4275463585 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fbf8c073100 0x7fbf8c0755c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:45:48.627 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:48.622+0000 7fbf9dffb640 1 -- 172.21.15.123:0/4275463585 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(46..46 src has 1..46) ==== 5767+0+0 (secure 0 0 0) 0x7fbfa0094620 con 0x7fbfa815bca0 2024-09-16T15:45:48.627 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:48.622+0000 7fbfad4dd640 1 --2- 172.21.15.123:0/4275463585 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fbf8c073100 0x7fbf8c0755c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:45:48.627 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:48.626+0000 7fbfad4dd640 1 --2- 172.21.15.123:0/4275463585 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fbf8c073100 0x7fbf8c0755c0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fbfa4005fd0 tx=0x7fbfa400c460 comp rx=0 tx=0).ready entity=mgr.4135 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:45:48.630 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:48.626+0000 7fbf9dffb640 1 -- 172.21.15.123:0/4275463585 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7fbfa0020020 con 0x7fbfa815bca0 2024-09-16T15:45:48.831 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:48.826+0000 7fbfaf741640 1 -- 172.21.15.123:0/4275463585 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fbfa815f630 con 0x7fbfa815bca0 2024-09-16T15:45:48.831 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:48.830+0000 7fbf9dffb640 1 -- 172.21.15.123:0/4275463585 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) ==== 94+0+5223 (secure 0 0 0) 0x7fbfa005f450 con 0x7fbfa815bca0 2024-09-16T15:45:48.832 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-09-16T15:45:48.832 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":28,"btime":"2024-09-16T15:44:32:363512+0000","default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4712,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6837/288264117","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6836","nonce":288264117},{"type":"v1","addr":"172.21.15.184:6837","nonce":288264117}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":23},{"gid":5268,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6834/60194515","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6832","nonce":60194515},{"type":"v1","addr":"172.21.15.123:6834","nonce":60194515}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":23},{"gid":5271,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6835/1798373479","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6833","nonce":1798373479},{"type":"v1","addr":"172.21.15.123:6835","nonce":1798373479}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":22}],"filesystems":[{"mdsmap":{"epoch":27,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-16T15:44:14.853230+0000","modified":"2024-09-16T15:44:17.861742+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5280},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5280":{"gid":5280,"name":"d","rank":0,"incarnation":24,"state":"up:active","state_seq":3,"addr":"172.21.15.184:6835/374389412","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":374389412},{"type":"v1","addr":"172.21.15.184:6835","nonce":374389412}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5280,"qdb_cluster":[5280]},"id":3}]} 2024-09-16T15:45:48.832 INFO:teuthology.orchestra.run.smithi123.stderr:dumped fsmap epoch 28 2024-09-16T15:45:48.834 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:48.830+0000 7fbfaf741640 1 -- 172.21.15.123:0/4275463585 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fbf8c073100 msgr2=0x7fbf8c0755c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:45:48.834 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:48.830+0000 7fbfaf741640 1 --2- 172.21.15.123:0/4275463585 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fbf8c073100 0x7fbf8c0755c0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fbfa4005fd0 tx=0x7fbfa400c460 comp rx=0 tx=0).stop 2024-09-16T15:45:48.834 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:48.830+0000 7fbfaf741640 1 -- 172.21.15.123:0/4275463585 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbfa815bca0 msgr2=0x7fbfa80c5e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:45:48.834 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:48.830+0000 7fbfaf741640 1 --2- 172.21.15.123:0/4275463585 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbfa815bca0 0x7fbfa80c5e90 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7fbfa000cbc0 tx=0x7fbfa000ccb0 comp rx=0 tx=0).stop 2024-09-16T15:45:48.834 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:48.830+0000 7fbfaf741640 1 -- 172.21.15.123:0/4275463585 shutdown_connections 2024-09-16T15:45:48.834 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:48.830+0000 7fbfaf741640 1 --2- 172.21.15.123:0/4275463585 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fbf8c073100 0x7fbf8c0755c0 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:45:48.834 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:48.830+0000 7fbfaf741640 1 --2- 172.21.15.123:0/4275463585 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbfa815bca0 0x7fbfa80c5e90 unknown :-1 s=CLOSED pgs=548 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:45:48.834 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:48.830+0000 7fbfaf741640 1 --2- 172.21.15.123:0/4275463585 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fbfa8074430 0x7fbfa80c5950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:45:48.834 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:48.830+0000 7fbfaf741640 1 --2- 172.21.15.123:0/4275463585 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fbfa80726c0 0x7fbfa8156e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:45:48.834 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:48.830+0000 7fbfaf741640 1 -- 172.21.15.123:0/4275463585 >> 172.21.15.123:0/4275463585 conn(0x7fbfa80cfb10 msgr2=0x7fbfa81560c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:45:48.835 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:48.830+0000 7fbfaf741640 1 -- 172.21.15.123:0/4275463585 shutdown_connections 2024-09-16T15:45:48.835 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:48.830+0000 7fbfaf741640 1 -- 172.21.15.123:0/4275463585 wait complete. 2024-09-16T15:45:48.849 DEBUG:teuthology.orchestra.run.smithi184:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.d.asok --format=json session ls 2024-09-16T15:45:49.001 INFO:teuthology.orchestra.run.smithi184.stdout:[{"id":5047,"entity":{"name":{"type":"client","num":5047},"addr":{"type":"any","addr":"172.21.15.184:0","nonce":908570948}},"state":"open","num_leases":0,"num_caps":7,"request_load_avg":7,"uptime":67.346701874000004,"requests_in_flight":0,"num_completed_requests":2,"num_completed_flushes":1,"reconnecting":false,"recall_caps":{"value":0,"halflife":60},"release_caps":{"value":0,"halflife":60},"recall_caps_throttle":{"value":0,"halflife":1.5},"recall_caps_throttle2o":{"value":0,"halflife":0.5},"session_cache_liveness":{"value":18.830478599436731,"halflife":300},"cap_acquisition":{"value":0,"halflife":30},"last_trim_completed_requests_tid":15,"last_trim_completed_flushes_tid":2,"delegated_inos":[{"start":"0x10000000005","length":500}],"inst":"client.5047 172.21.15.184:0/908570948","completed_requests":[{"tid":15,"created_ino":"0x0"},{"tid":16,"created_ino":"0x100000001f9"}],"prealloc_inos":[{"start":"0x10000000005","length":500},{"start":"0x100000001fa","length":1000}],"client_metadata":{"client_features":{"feature_bits":"0x00000000003fffff"},"metric_spec":{"metric_flags":{"feature_bits":"0x000000000000ffff"}},"ceph_sha1":"33f95d2ae1bcbe7eb04404303f179eeeaa6982df","ceph_version":"ceph version 19.3.0-4938-g33f95d2a (33f95d2ae1bcbe7eb04404303f179eeeaa6982df) squid (dev)","entity_id":"x","hostname":"smithi184","pid":"28840","root":"/"}}] 2024-09-16T15:45:49.002 DEBUG:tasks.cephfs.filesystem:_json_asok output [ { "cap_acquisition": { "halflife": 30, "value": 0 }, "client_metadata": { "ceph_sha1": "33f95d2ae1bcbe7eb04404303f179eeeaa6982df", "ceph_version": "ceph version 19.3.0-4938-g33f95d2a (33f95d2ae1bcbe7eb04404303f179eeeaa6982df) squid (dev)", "client_features": { "feature_bits": "0x00000000003fffff" }, "entity_id": "x", "hostname": "smithi184", "metric_spec": { "metric_flags": { "feature_bits": "0x000000000000ffff" } }, "pid": "28840", "root": "/" }, "completed_requests": [ { "created_ino": "0x0", "tid": 15 }, { "created_ino": "0x100000001f9", "tid": 16 } ], "delegated_inos": [ { "length": 500, "start": "0x10000000005" } ], "entity": { "addr": { "addr": "172.21.15.184:0", "nonce": 908570948, "type": "any" }, "name": { "num": 5047, "type": "client" } }, "id": 5047, "inst": "client.5047 172.21.15.184:0/908570948", "last_trim_completed_flushes_tid": 2, "last_trim_completed_requests_tid": 15, "num_caps": 7, "num_completed_flushes": 1, "num_completed_requests": 2, "num_leases": 0, "prealloc_inos": [ { "length": 500, "start": "0x10000000005" }, { "length": 1000, "start": "0x100000001fa" } ], "recall_caps": { "halflife": 60, "value": 0 }, "recall_caps_throttle": { "halflife": 1.5, "value": 0 }, "recall_caps_throttle2o": { "halflife": 0.5, "value": 0 }, "reconnecting": false, "release_caps": { "halflife": 60, "value": 0 }, "request_load_avg": 7, "requests_in_flight": 0, "session_cache_liveness": { "halflife": 300, "value": 18.83047859943673 }, "state": "open", "uptime": 67.346701874 } ] 2024-09-16T15:45:49.002 DEBUG:tasks.ceph_test_case:wait_until_true: waiting (timeout=90 retry_count=0)... 2024-09-16T15:45:54.003 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-16T15:45:54.140 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:54.137+0000 7f5868cf3640 1 -- 172.21.15.123:0/894842755 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f58640cf5a0 msgr2=0x7f586406eee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:45:54.140 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:54.137+0000 7f5868cf3640 1 --2- 172.21.15.123:0/894842755 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f58640cf5a0 0x7f586406eee0 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7f5864076280 tx=0x7f585401c7a0 comp rx=0 tx=0).stop 2024-09-16T15:45:54.140 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:54.137+0000 7f5868cf3640 1 -- 172.21.15.123:0/894842755 shutdown_connections 2024-09-16T15:45:54.140 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:54.137+0000 7f5868cf3640 1 --2- 172.21.15.123:0/894842755 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f58640cf5a0 0x7f586406eee0 unknown :-1 s=CLOSED pgs=549 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:45:54.140 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:54.137+0000 7f5868cf3640 1 --2- 172.21.15.123:0/894842755 >> v2:172.21.15.184:3300/0 conn(0x7f5864158120 0x7f58640ca320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:45:54.140 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:54.137+0000 7f5868cf3640 1 -- 172.21.15.123:0/894842755 >> 172.21.15.123:0/894842755 conn(0x7f58640cfb10 msgr2=0x7f58640cff20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:45:54.140 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:54.137+0000 7f5868cf3640 1 -- 172.21.15.123:0/894842755 shutdown_connections 2024-09-16T15:45:54.140 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:54.137+0000 7f5868cf3640 1 -- 172.21.15.123:0/894842755 wait complete. 2024-09-16T15:45:54.141 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:54.137+0000 7f5868cf3640 1 Processor -- start 2024-09-16T15:45:54.141 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:54.137+0000 7f5868cf3640 1 -- start start 2024-09-16T15:45:54.143 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:54.137+0000 7f5868cf3640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f58640cf5a0 0x7f5864149150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:45:54.143 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:54.137+0000 7f5868cf3640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5864158120 0x7f58640b29a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:45:54.143 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:54.137+0000 7f5868cf3640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f586415ec40 0x7f586414c710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:45:54.143 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:54.137+0000 7f5868cf3640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f5864167970 con 0x7f5864158120 2024-09-16T15:45:54.143 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:54.137+0000 7f5868cf3640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f58641677f0 con 0x7f586415ec40 2024-09-16T15:45:54.143 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:54.137+0000 7f5868cf3640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f5864167af0 con 0x7f58640cf5a0 2024-09-16T15:45:54.143 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:54.137+0000 7f5861d9b640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5864158120 0x7f58640b29a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:45:54.143 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:54.137+0000 7f5861d9b640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5864158120 0x7f58640b29a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:37086/0 (socket says 172.21.15.123:37086) 2024-09-16T15:45:54.143 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:54.137+0000 7f5861d9b640 1 -- 172.21.15.123:0/3590666210 learned_addr learned my addr 172.21.15.123:0/3590666210 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:45:54.143 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:54.137+0000 7f5862d9d640 1 --2- 172.21.15.123:0/3590666210 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f586415ec40 0x7f586414c710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:45:54.143 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:54.137+0000 7f586259c640 1 --2- 172.21.15.123:0/3590666210 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f58640cf5a0 0x7f5864149150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:45:54.143 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:54.137+0000 7f5861d9b640 1 -- 172.21.15.123:0/3590666210 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f58640cf5a0 msgr2=0x7f5864149150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:45:54.144 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:54.137+0000 7f5861d9b640 1 --2- 172.21.15.123:0/3590666210 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f58640cf5a0 0x7f5864149150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:45:54.144 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:54.137+0000 7f5861d9b640 1 -- 172.21.15.123:0/3590666210 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f586415ec40 msgr2=0x7f586414c710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:45:54.144 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:54.137+0000 7f5861d9b640 1 --2- 172.21.15.123:0/3590666210 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f586415ec40 0x7f586414c710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:45:54.144 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:54.137+0000 7f5861d9b640 1 -- 172.21.15.123:0/3590666210 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f58640b4c70 con 0x7f5864158120 2024-09-16T15:45:54.144 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:54.137+0000 7f5862d9d640 1 --2- 172.21.15.123:0/3590666210 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f586415ec40 0x7f586414c710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T15:45:54.144 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:54.137+0000 7f5861d9b640 1 --2- 172.21.15.123:0/3590666210 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5864158120 0x7f58640b29a0 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7f5854009850 tx=0x7f58540a6540 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:45:54.144 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:54.137+0000 7f5852ffd640 1 -- 172.21.15.123:0/3590666210 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58540ac070 con 0x7f5864158120 2024-09-16T15:45:54.144 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:54.137+0000 7f5852ffd640 1 -- 172.21.15.123:0/3590666210 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f5854004340 con 0x7f5864158120 2024-09-16T15:45:54.144 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:54.137+0000 7f5868cf3640 1 -- 172.21.15.123:0/3590666210 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f586414a0a0 con 0x7f5864158120 2024-09-16T15:45:54.144 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:54.137+0000 7f5852ffd640 1 -- 172.21.15.123:0/3590666210 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58540053a0 con 0x7f5864158120 2024-09-16T15:45:54.144 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:54.137+0000 7f5868cf3640 1 -- 172.21.15.123:0/3590666210 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f586414a360 con 0x7f5864158120 2024-09-16T15:45:54.145 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:54.141+0000 7f5868cf3640 1 -- 172.21.15.123:0/3590666210 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f586406eee0 con 0x7f5864158120 2024-09-16T15:45:54.145 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:54.141+0000 7f5852ffd640 1 -- 172.21.15.123:0/3590666210 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 8) ==== 97284+0+0 (secure 0 0 0) 0x7f58540af070 con 0x7f5864158120 2024-09-16T15:45:54.145 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:54.141+0000 7f5852ffd640 1 --2- 172.21.15.123:0/3590666210 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f5844073130 0x7f58440755f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:45:54.145 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:54.141+0000 7f5852ffd640 1 -- 172.21.15.123:0/3590666210 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(46..46 src has 1..46) ==== 5767+0+0 (secure 0 0 0) 0x7f585412e500 con 0x7f5864158120 2024-09-16T15:45:54.145 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:54.141+0000 7f586259c640 1 --2- 172.21.15.123:0/3590666210 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f5844073130 0x7f58440755f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:45:54.146 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:54.141+0000 7f586259c640 1 --2- 172.21.15.123:0/3590666210 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f5844073130 0x7f58440755f0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f58580058d0 tx=0x7f5858005c70 comp rx=0 tx=0).ready entity=mgr.4135 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:45:54.149 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:54.145+0000 7f5852ffd640 1 -- 172.21.15.123:0/3590666210 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f58540164b0 con 0x7f5864158120 2024-09-16T15:45:54.357 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:54.353+0000 7f5868cf3640 1 -- 172.21.15.123:0/3590666210 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f586414a5a0 con 0x7f5864158120 2024-09-16T15:45:54.358 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:54.353+0000 7f5852ffd640 1 -- 172.21.15.123:0/3590666210 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) ==== 94+0+5223 (secure 0 0 0) 0x7f58540f92a0 con 0x7f5864158120 2024-09-16T15:45:54.358 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-09-16T15:45:54.358 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":28,"btime":"2024-09-16T15:44:32:363512+0000","default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4712,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6837/288264117","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6836","nonce":288264117},{"type":"v1","addr":"172.21.15.184:6837","nonce":288264117}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":23},{"gid":5268,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6834/60194515","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6832","nonce":60194515},{"type":"v1","addr":"172.21.15.123:6834","nonce":60194515}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":23},{"gid":5271,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6835/1798373479","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6833","nonce":1798373479},{"type":"v1","addr":"172.21.15.123:6835","nonce":1798373479}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":22}],"filesystems":[{"mdsmap":{"epoch":27,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-16T15:44:14.853230+0000","modified":"2024-09-16T15:44:17.861742+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5280},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5280":{"gid":5280,"name":"d","rank":0,"incarnation":24,"state":"up:active","state_seq":3,"addr":"172.21.15.184:6835/374389412","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":374389412},{"type":"v1","addr":"172.21.15.184:6835","nonce":374389412}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5280,"qdb_cluster":[5280]},"id":3}]} 2024-09-16T15:45:54.358 INFO:teuthology.orchestra.run.smithi123.stderr:dumped fsmap epoch 28 2024-09-16T15:45:54.360 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:54.357+0000 7f5868cf3640 1 -- 172.21.15.123:0/3590666210 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f5844073130 msgr2=0x7f58440755f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:45:54.361 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:54.357+0000 7f5868cf3640 1 --2- 172.21.15.123:0/3590666210 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f5844073130 0x7f58440755f0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f58580058d0 tx=0x7f5858005c70 comp rx=0 tx=0).stop 2024-09-16T15:45:54.361 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:54.357+0000 7f5868cf3640 1 -- 172.21.15.123:0/3590666210 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5864158120 msgr2=0x7f58640b29a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:45:54.361 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:54.357+0000 7f5868cf3640 1 --2- 172.21.15.123:0/3590666210 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5864158120 0x7f58640b29a0 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7f5854009850 tx=0x7f58540a6540 comp rx=0 tx=0).stop 2024-09-16T15:45:54.361 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:54.357+0000 7f5868cf3640 1 -- 172.21.15.123:0/3590666210 shutdown_connections 2024-09-16T15:45:54.361 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:54.357+0000 7f5868cf3640 1 --2- 172.21.15.123:0/3590666210 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f5844073130 0x7f58440755f0 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:45:54.361 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:54.357+0000 7f5868cf3640 1 --2- 172.21.15.123:0/3590666210 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f586415ec40 0x7f586414c710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:45:54.361 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:54.357+0000 7f5868cf3640 1 --2- 172.21.15.123:0/3590666210 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5864158120 0x7f58640b29a0 unknown :-1 s=CLOSED pgs=550 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:45:54.361 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:54.357+0000 7f5868cf3640 1 --2- 172.21.15.123:0/3590666210 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f58640cf5a0 0x7f5864149150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:45:54.361 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:54.357+0000 7f5868cf3640 1 -- 172.21.15.123:0/3590666210 >> 172.21.15.123:0/3590666210 conn(0x7f58640cfb10 msgr2=0x7f586406fc40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:45:54.361 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:54.357+0000 7f5868cf3640 1 -- 172.21.15.123:0/3590666210 shutdown_connections 2024-09-16T15:45:54.361 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:54.357+0000 7f5868cf3640 1 -- 172.21.15.123:0/3590666210 wait complete. 2024-09-16T15:45:54.377 DEBUG:teuthology.orchestra.run.smithi184:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.d.asok --format=json session ls 2024-09-16T15:45:54.528 INFO:teuthology.orchestra.run.smithi184.stdout:[{"id":5047,"entity":{"name":{"type":"client","num":5047},"addr":{"type":"any","addr":"172.21.15.184:0","nonce":908570948}},"state":"open","num_leases":0,"num_caps":7,"request_load_avg":6,"uptime":72.870596785999993,"requests_in_flight":0,"num_completed_requests":2,"num_completed_flushes":1,"reconnecting":false,"recall_caps":{"value":0,"halflife":60},"release_caps":{"value":0,"halflife":60},"recall_caps_throttle":{"value":0,"halflife":1.5},"recall_caps_throttle2o":{"value":0,"halflife":0.5},"session_cache_liveness":{"value":18.591674108423902,"halflife":300},"cap_acquisition":{"value":0,"halflife":30},"last_trim_completed_requests_tid":15,"last_trim_completed_flushes_tid":2,"delegated_inos":[{"start":"0x10000000005","length":500}],"inst":"client.5047 172.21.15.184:0/908570948","completed_requests":[{"tid":15,"created_ino":"0x0"},{"tid":16,"created_ino":"0x100000001f9"}],"prealloc_inos":[{"start":"0x10000000005","length":500},{"start":"0x100000001fa","length":1000}],"client_metadata":{"client_features":{"feature_bits":"0x00000000003fffff"},"metric_spec":{"metric_flags":{"feature_bits":"0x000000000000ffff"}},"ceph_sha1":"33f95d2ae1bcbe7eb04404303f179eeeaa6982df","ceph_version":"ceph version 19.3.0-4938-g33f95d2a (33f95d2ae1bcbe7eb04404303f179eeeaa6982df) squid (dev)","entity_id":"x","hostname":"smithi184","pid":"28840","root":"/"}}] 2024-09-16T15:45:54.546 DEBUG:tasks.cephfs.filesystem:_json_asok output [ { "cap_acquisition": { "halflife": 30, "value": 0 }, "client_metadata": { "ceph_sha1": "33f95d2ae1bcbe7eb04404303f179eeeaa6982df", "ceph_version": "ceph version 19.3.0-4938-g33f95d2a (33f95d2ae1bcbe7eb04404303f179eeeaa6982df) squid (dev)", "client_features": { "feature_bits": "0x00000000003fffff" }, "entity_id": "x", "hostname": "smithi184", "metric_spec": { "metric_flags": { "feature_bits": "0x000000000000ffff" } }, "pid": "28840", "root": "/" }, "completed_requests": [ { "created_ino": "0x0", "tid": 15 }, { "created_ino": "0x100000001f9", "tid": 16 } ], "delegated_inos": [ { "length": 500, "start": "0x10000000005" } ], "entity": { "addr": { "addr": "172.21.15.184:0", "nonce": 908570948, "type": "any" }, "name": { "num": 5047, "type": "client" } }, "id": 5047, "inst": "client.5047 172.21.15.184:0/908570948", "last_trim_completed_flushes_tid": 2, "last_trim_completed_requests_tid": 15, "num_caps": 7, "num_completed_flushes": 1, "num_completed_requests": 2, "num_leases": 0, "prealloc_inos": [ { "length": 500, "start": "0x10000000005" }, { "length": 1000, "start": "0x100000001fa" } ], "recall_caps": { "halflife": 60, "value": 0 }, "recall_caps_throttle": { "halflife": 1.5, "value": 0 }, "recall_caps_throttle2o": { "halflife": 0.5, "value": 0 }, "reconnecting": false, "release_caps": { "halflife": 60, "value": 0 }, "request_load_avg": 6, "requests_in_flight": 0, "session_cache_liveness": { "halflife": 300, "value": 18.591674108423902 }, "state": "open", "uptime": 72.870596786 } ] 2024-09-16T15:45:54.546 DEBUG:tasks.ceph_test_case:wait_until_true: waiting (timeout=90 retry_count=0)... 2024-09-16T15:45:59.547 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-16T15:45:59.682 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:59.677+0000 7fa0eea06640 1 -- 172.21.15.123:0/2984124570 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa0e815e160 msgr2=0x7fa0e8073f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:45:59.682 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:59.677+0000 7fa0eea06640 1 --2- 172.21.15.123:0/2984124570 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa0e815e160 0x7fa0e8073f30 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7fa0e80708f0 tx=0x7fa0dc01c7a0 comp rx=0 tx=0).stop 2024-09-16T15:45:59.682 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:59.677+0000 7fa0eea06640 1 -- 172.21.15.123:0/2984124570 shutdown_connections 2024-09-16T15:45:59.682 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:59.677+0000 7fa0eea06640 1 --2- 172.21.15.123:0/2984124570 >> v2:172.21.15.184:3300/0 conn(0x7fa0e815e820 0x7fa0e815ec90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:45:59.682 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:59.677+0000 7fa0eea06640 1 --2- 172.21.15.123:0/2984124570 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa0e815e160 0x7fa0e8073f30 unknown :-1 s=CLOSED pgs=551 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:45:59.682 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:59.677+0000 7fa0eea06640 1 -- 172.21.15.123:0/2984124570 >> 172.21.15.123:0/2984124570 conn(0x7fa0e80cfb10 msgr2=0x7fa0e80cff20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:45:59.682 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:59.677+0000 7fa0eea06640 1 -- 172.21.15.123:0/2984124570 shutdown_connections 2024-09-16T15:45:59.682 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:59.677+0000 7fa0eea06640 1 -- 172.21.15.123:0/2984124570 wait complete. 2024-09-16T15:45:59.683 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:59.677+0000 7fa0eea06640 1 Processor -- start 2024-09-16T15:45:59.683 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:59.677+0000 7fa0eea06640 1 -- start start 2024-09-16T15:45:59.685 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:59.681+0000 7fa0eea06640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fa0e8074470 0x7fa0e8149050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:45:59.685 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:59.681+0000 7fa0eea06640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa0e815e160 0x7fa0e81555f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:45:59.685 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:59.681+0000 7fa0eea06640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa0e815e820 0x7fa0e8155dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:45:59.685 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:59.681+0000 7fa0eea06640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fa0e80cf430 con 0x7fa0e815e820 2024-09-16T15:45:59.685 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:59.681+0000 7fa0eea06640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fa0e815f7d0 con 0x7fa0e815e160 2024-09-16T15:45:59.685 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:59.681+0000 7fa0eea06640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fa0e80cf5b0 con 0x7fa0e8074470 2024-09-16T15:45:59.685 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:59.681+0000 7fa0ecfa3640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa0e815e820 0x7fa0e8155dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:45:59.686 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:59.681+0000 7fa0ecfa3640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa0e815e820 0x7fa0e8155dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:37106/0 (socket says 172.21.15.123:37106) 2024-09-16T15:45:59.686 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:59.681+0000 7fa0ecfa3640 1 -- 172.21.15.123:0/1350577729 learned_addr learned my addr 172.21.15.123:0/1350577729 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:45:59.686 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:59.681+0000 7fa0e77fe640 1 --2- 172.21.15.123:0/1350577729 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa0e815e160 0x7fa0e81555f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:45:59.686 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:59.681+0000 7fa0e7fff640 1 --2- 172.21.15.123:0/1350577729 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fa0e8074470 0x7fa0e8149050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:45:59.686 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:59.681+0000 7fa0ecfa3640 1 -- 172.21.15.123:0/1350577729 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fa0e8074470 msgr2=0x7fa0e8149050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:45:59.686 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:59.681+0000 7fa0ecfa3640 1 --2- 172.21.15.123:0/1350577729 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fa0e8074470 0x7fa0e8149050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:45:59.686 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:59.681+0000 7fa0ecfa3640 1 -- 172.21.15.123:0/1350577729 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa0e815e160 msgr2=0x7fa0e81555f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:45:59.686 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:59.681+0000 7fa0ecfa3640 1 --2- 172.21.15.123:0/1350577729 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa0e815e160 0x7fa0e81555f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:45:59.686 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:59.681+0000 7fa0ecfa3640 1 -- 172.21.15.123:0/1350577729 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa0e8149590 con 0x7fa0e815e820 2024-09-16T15:45:59.686 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:59.681+0000 7fa0e7fff640 1 --2- 172.21.15.123:0/1350577729 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fa0e8074470 0x7fa0e8149050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T15:45:59.686 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:59.681+0000 7fa0ecfa3640 1 --2- 172.21.15.123:0/1350577729 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa0e815e820 0x7fa0e8155dd0 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7fa0e000cce0 tx=0x7fa0e000a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:45:59.687 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:59.681+0000 7fa0e4ff9640 1 -- 172.21.15.123:0/1350577729 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0e000f450 con 0x7fa0e815e820 2024-09-16T15:45:59.687 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:59.681+0000 7fa0e4ff9640 1 -- 172.21.15.123:0/1350577729 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fa0e000f6e0 con 0x7fa0e815e820 2024-09-16T15:45:59.687 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:59.681+0000 7fa0eea06640 1 -- 172.21.15.123:0/1350577729 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa0e8164300 con 0x7fa0e815e820 2024-09-16T15:45:59.687 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:59.681+0000 7fa0e4ff9640 1 -- 172.21.15.123:0/1350577729 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0e000fb30 con 0x7fa0e815e820 2024-09-16T15:45:59.687 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:59.681+0000 7fa0eea06640 1 -- 172.21.15.123:0/1350577729 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa0e81648c0 con 0x7fa0e815e820 2024-09-16T15:45:59.687 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:59.681+0000 7fa0eea06640 1 -- 172.21.15.123:0/1350577729 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa0e8073210 con 0x7fa0e815e820 2024-09-16T15:45:59.688 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:59.681+0000 7fa0e4ff9640 1 -- 172.21.15.123:0/1350577729 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 8) ==== 97284+0+0 (secure 0 0 0) 0x7fa0e000fcd0 con 0x7fa0e815e820 2024-09-16T15:45:59.688 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:59.681+0000 7fa0e4ff9640 1 --2- 172.21.15.123:0/1350577729 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fa0d4073100 0x7fa0d40755c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:45:59.688 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:59.681+0000 7fa0e4ff9640 1 -- 172.21.15.123:0/1350577729 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(46..46 src has 1..46) ==== 5767+0+0 (secure 0 0 0) 0x7fa0e0094e80 con 0x7fa0e815e820 2024-09-16T15:45:59.688 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:59.685+0000 7fa0e7fff640 1 --2- 172.21.15.123:0/1350577729 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fa0d4073100 0x7fa0d40755c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:45:59.688 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:59.685+0000 7fa0e7fff640 1 --2- 172.21.15.123:0/1350577729 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fa0d4073100 0x7fa0d40755c0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fa0dc0af000 tx=0x7fa0dc01afe0 comp rx=0 tx=0).ready entity=mgr.4135 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:45:59.691 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:59.689+0000 7fa0e4ff9640 1 -- 172.21.15.123:0/1350577729 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7fa0e0014030 con 0x7fa0e815e820 2024-09-16T15:45:59.894 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:59.889+0000 7fa0eea06640 1 -- 172.21.15.123:0/1350577729 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa0e80d06c0 con 0x7fa0e815e820 2024-09-16T15:45:59.895 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:59.893+0000 7fa0e4ff9640 1 -- 172.21.15.123:0/1350577729 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) ==== 94+0+5223 (secure 0 0 0) 0x7fa0e005fc20 con 0x7fa0e815e820 2024-09-16T15:45:59.895 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-09-16T15:45:59.896 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":28,"btime":"2024-09-16T15:44:32:363512+0000","default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4712,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6837/288264117","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6836","nonce":288264117},{"type":"v1","addr":"172.21.15.184:6837","nonce":288264117}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":23},{"gid":5268,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6834/60194515","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6832","nonce":60194515},{"type":"v1","addr":"172.21.15.123:6834","nonce":60194515}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":23},{"gid":5271,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6835/1798373479","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6833","nonce":1798373479},{"type":"v1","addr":"172.21.15.123:6835","nonce":1798373479}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":22}],"filesystems":[{"mdsmap":{"epoch":27,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-16T15:44:14.853230+0000","modified":"2024-09-16T15:44:17.861742+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5280},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5280":{"gid":5280,"name":"d","rank":0,"incarnation":24,"state":"up:active","state_seq":3,"addr":"172.21.15.184:6835/374389412","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":374389412},{"type":"v1","addr":"172.21.15.184:6835","nonce":374389412}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5280,"qdb_cluster":[5280]},"id":3}]} 2024-09-16T15:45:59.896 INFO:teuthology.orchestra.run.smithi123.stderr:dumped fsmap epoch 28 2024-09-16T15:45:59.898 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:59.893+0000 7fa0eea06640 1 -- 172.21.15.123:0/1350577729 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fa0d4073100 msgr2=0x7fa0d40755c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:45:59.898 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:59.893+0000 7fa0eea06640 1 --2- 172.21.15.123:0/1350577729 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fa0d4073100 0x7fa0d40755c0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fa0dc0af000 tx=0x7fa0dc01afe0 comp rx=0 tx=0).stop 2024-09-16T15:45:59.898 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:59.893+0000 7fa0eea06640 1 -- 172.21.15.123:0/1350577729 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa0e815e820 msgr2=0x7fa0e8155dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:45:59.898 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:59.893+0000 7fa0eea06640 1 --2- 172.21.15.123:0/1350577729 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa0e815e820 0x7fa0e8155dd0 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7fa0e000cce0 tx=0x7fa0e000a620 comp rx=0 tx=0).stop 2024-09-16T15:45:59.898 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:59.893+0000 7fa0eea06640 1 -- 172.21.15.123:0/1350577729 shutdown_connections 2024-09-16T15:45:59.898 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:59.893+0000 7fa0eea06640 1 --2- 172.21.15.123:0/1350577729 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fa0d4073100 0x7fa0d40755c0 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:45:59.898 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:59.893+0000 7fa0eea06640 1 --2- 172.21.15.123:0/1350577729 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa0e815e820 0x7fa0e8155dd0 unknown :-1 s=CLOSED pgs=552 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:45:59.898 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:59.893+0000 7fa0eea06640 1 --2- 172.21.15.123:0/1350577729 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa0e815e160 0x7fa0e81555f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:45:59.898 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:59.893+0000 7fa0eea06640 1 --2- 172.21.15.123:0/1350577729 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fa0e8074470 0x7fa0e8149050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:45:59.899 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:59.893+0000 7fa0eea06640 1 -- 172.21.15.123:0/1350577729 >> 172.21.15.123:0/1350577729 conn(0x7fa0e80cfb10 msgr2=0x7fa0e80cd6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:45:59.899 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:59.893+0000 7fa0eea06640 1 -- 172.21.15.123:0/1350577729 shutdown_connections 2024-09-16T15:45:59.899 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:45:59.893+0000 7fa0eea06640 1 -- 172.21.15.123:0/1350577729 wait complete. 2024-09-16T15:45:59.914 DEBUG:teuthology.orchestra.run.smithi184:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.d.asok --format=json session ls 2024-09-16T15:46:00.065 INFO:teuthology.orchestra.run.smithi184.stdout:[{"id":5047,"entity":{"name":{"type":"client","num":5047},"addr":{"type":"any","addr":"172.21.15.184:0","nonce":908570948}},"state":"open","num_leases":0,"num_caps":7,"request_load_avg":6,"uptime":78.406500597000004,"requests_in_flight":0,"num_completed_requests":2,"num_completed_flushes":1,"reconnecting":false,"recall_caps":{"value":0,"halflife":60},"release_caps":{"value":0,"halflife":60},"recall_caps_throttle":{"value":0,"halflife":1.5},"recall_caps_throttle2o":{"value":0,"halflife":0.5},"session_cache_liveness":{"value":18.355388785911035,"halflife":300},"cap_acquisition":{"value":0,"halflife":30},"last_trim_completed_requests_tid":15,"last_trim_completed_flushes_tid":2,"delegated_inos":[{"start":"0x10000000005","length":500}],"inst":"client.5047 172.21.15.184:0/908570948","completed_requests":[{"tid":15,"created_ino":"0x0"},{"tid":16,"created_ino":"0x100000001f9"}],"prealloc_inos":[{"start":"0x10000000005","length":500},{"start":"0x100000001fa","length":1000}],"client_metadata":{"client_features":{"feature_bits":"0x00000000003fffff"},"metric_spec":{"metric_flags":{"feature_bits":"0x000000000000ffff"}},"ceph_sha1":"33f95d2ae1bcbe7eb04404303f179eeeaa6982df","ceph_version":"ceph version 19.3.0-4938-g33f95d2a (33f95d2ae1bcbe7eb04404303f179eeeaa6982df) squid (dev)","entity_id":"x","hostname":"smithi184","pid":"28840","root":"/"}}] 2024-09-16T15:46:00.066 DEBUG:tasks.cephfs.filesystem:_json_asok output [ { "cap_acquisition": { "halflife": 30, "value": 0 }, "client_metadata": { "ceph_sha1": "33f95d2ae1bcbe7eb04404303f179eeeaa6982df", "ceph_version": "ceph version 19.3.0-4938-g33f95d2a (33f95d2ae1bcbe7eb04404303f179eeeaa6982df) squid (dev)", "client_features": { "feature_bits": "0x00000000003fffff" }, "entity_id": "x", "hostname": "smithi184", "metric_spec": { "metric_flags": { "feature_bits": "0x000000000000ffff" } }, "pid": "28840", "root": "/" }, "completed_requests": [ { "created_ino": "0x0", "tid": 15 }, { "created_ino": "0x100000001f9", "tid": 16 } ], "delegated_inos": [ { "length": 500, "start": "0x10000000005" } ], "entity": { "addr": { "addr": "172.21.15.184:0", "nonce": 908570948, "type": "any" }, "name": { "num": 5047, "type": "client" } }, "id": 5047, "inst": "client.5047 172.21.15.184:0/908570948", "last_trim_completed_flushes_tid": 2, "last_trim_completed_requests_tid": 15, "num_caps": 7, "num_completed_flushes": 1, "num_completed_requests": 2, "num_leases": 0, "prealloc_inos": [ { "length": 500, "start": "0x10000000005" }, { "length": 1000, "start": "0x100000001fa" } ], "recall_caps": { "halflife": 60, "value": 0 }, "recall_caps_throttle": { "halflife": 1.5, "value": 0 }, "recall_caps_throttle2o": { "halflife": 0.5, "value": 0 }, "reconnecting": false, "release_caps": { "halflife": 60, "value": 0 }, "request_load_avg": 6, "requests_in_flight": 0, "session_cache_liveness": { "halflife": 300, "value": 18.355388785911035 }, "state": "open", "uptime": 78.406500597 } ] 2024-09-16T15:46:00.066 DEBUG:tasks.ceph_test_case:wait_until_true: waiting (timeout=90 retry_count=0)... 2024-09-16T15:46:05.066 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-16T15:46:05.200 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:05.197+0000 7f5b89953640 1 -- 172.21.15.123:0/3781598840 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f5b84074600 msgr2=0x7f5b8406e750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:46:05.200 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:05.197+0000 7f5b89953640 1 --2- 172.21.15.123:0/3781598840 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f5b84074600 0x7f5b8406e750 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f5b7c009bc0 tx=0x7f5b7c01cae0 comp rx=0 tx=0).stop 2024-09-16T15:46:05.201 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:05.197+0000 7f5b89953640 1 -- 172.21.15.123:0/3781598840 shutdown_connections 2024-09-16T15:46:05.201 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:05.197+0000 7f5b89953640 1 --2- 172.21.15.123:0/3781598840 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f5b84074600 0x7f5b8406e750 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:05.201 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:05.197+0000 7f5b89953640 1 --2- 172.21.15.123:0/3781598840 >> v2:172.21.15.184:3300/0 conn(0x7f5b8415bca0 0x7f5b840d0630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:05.201 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:05.197+0000 7f5b89953640 1 -- 172.21.15.123:0/3781598840 >> 172.21.15.123:0/3781598840 conn(0x7f5b840cfb10 msgr2=0x7f5b840cff20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:46:05.201 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:05.197+0000 7f5b89953640 1 -- 172.21.15.123:0/3781598840 shutdown_connections 2024-09-16T15:46:05.201 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:05.197+0000 7f5b89953640 1 -- 172.21.15.123:0/3781598840 wait complete. 2024-09-16T15:46:05.201 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:05.197+0000 7f5b89953640 1 Processor -- start 2024-09-16T15:46:05.202 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:05.197+0000 7f5b89953640 1 -- start start 2024-09-16T15:46:05.203 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:05.197+0000 7f5b89953640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5b840726c0 0x7f5b841f8240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:46:05.203 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:05.197+0000 7f5b89953640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f5b84074600 0x7f5b8415e980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:46:05.203 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:05.197+0000 7f5b89953640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5b8415bca0 0x7f5b840cdbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:46:05.204 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:05.197+0000 7f5b89953640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f5b84168a50 con 0x7f5b840726c0 2024-09-16T15:46:05.204 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:05.197+0000 7f5b89953640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f5b8406fa20 con 0x7f5b8415bca0 2024-09-16T15:46:05.204 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:05.197+0000 7f5b89953640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f5b84168bd0 con 0x7f5b84074600 2024-09-16T15:46:05.204 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:05.197+0000 7f5b82ffd640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5b840726c0 0x7f5b841f8240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:46:05.204 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:05.197+0000 7f5b82ffd640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5b840726c0 0x7f5b841f8240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:52040/0 (socket says 172.21.15.123:52040) 2024-09-16T15:46:05.204 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:05.197+0000 7f5b82ffd640 1 -- 172.21.15.123:0/1334288835 learned_addr learned my addr 172.21.15.123:0/1334288835 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:46:05.204 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:05.197+0000 7f5b827fc640 1 --2- 172.21.15.123:0/1334288835 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f5b84074600 0x7f5b8415e980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:46:05.204 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:05.197+0000 7f5b837fe640 1 --2- 172.21.15.123:0/1334288835 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5b8415bca0 0x7f5b840cdbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:46:05.204 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:05.197+0000 7f5b82ffd640 1 -- 172.21.15.123:0/1334288835 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f5b84074600 msgr2=0x7f5b8415e980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:46:05.204 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:05.197+0000 7f5b82ffd640 1 --2- 172.21.15.123:0/1334288835 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f5b84074600 0x7f5b8415e980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:05.204 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:05.197+0000 7f5b82ffd640 1 -- 172.21.15.123:0/1334288835 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5b8415bca0 msgr2=0x7f5b840cdbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:46:05.204 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:05.197+0000 7f5b82ffd640 1 --2- 172.21.15.123:0/1334288835 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5b8415bca0 0x7f5b840cdbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:05.205 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:05.197+0000 7f5b82ffd640 1 -- 172.21.15.123:0/1334288835 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5b8415f180 con 0x7f5b840726c0 2024-09-16T15:46:05.205 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:05.201+0000 7f5b82ffd640 1 --2- 172.21.15.123:0/1334288835 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5b840726c0 0x7f5b841f8240 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7f5b7400e7d0 tx=0x7f5b7400ec90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:46:05.205 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:05.201+0000 7f5b6bfff640 1 -- 172.21.15.123:0/1334288835 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b7400ccc0 con 0x7f5b840726c0 2024-09-16T15:46:05.205 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:05.201+0000 7f5b6bfff640 1 -- 172.21.15.123:0/1334288835 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f5b74010430 con 0x7f5b840726c0 2024-09-16T15:46:05.205 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:05.201+0000 7f5b6bfff640 1 -- 172.21.15.123:0/1334288835 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b74010860 con 0x7f5b840726c0 2024-09-16T15:46:05.205 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:05.201+0000 7f5b89953640 1 -- 172.21.15.123:0/1334288835 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5b8415eec0 con 0x7f5b840726c0 2024-09-16T15:46:05.205 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:05.201+0000 7f5b89953640 1 -- 172.21.15.123:0/1334288835 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5b8415d090 con 0x7f5b840726c0 2024-09-16T15:46:05.206 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:05.201+0000 7f5b89953640 1 -- 172.21.15.123:0/1334288835 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5b8406e750 con 0x7f5b840726c0 2024-09-16T15:46:05.206 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:05.201+0000 7f5b6bfff640 1 -- 172.21.15.123:0/1334288835 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 9) ==== 97233+0+0 (secure 0 0 0) 0x7f5b74010a00 con 0x7f5b840726c0 2024-09-16T15:46:05.206 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:05.201+0000 7f5b6bfff640 1 --2- 172.21.15.123:0/1334288835 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f5b700730a0 0x7f5b70075560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:46:05.206 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:05.201+0000 7f5b6bfff640 1 -- 172.21.15.123:0/1334288835 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(46..46 src has 1..46) ==== 5767+0+0 (secure 0 0 0) 0x7f5b74094cc0 con 0x7f5b840726c0 2024-09-16T15:46:05.206 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:05.201+0000 7f5b827fc640 1 --2- 172.21.15.123:0/1334288835 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f5b700730a0 0x7f5b70075560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:46:05.206 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:05.201+0000 7f5b827fc640 1 --2- 172.21.15.123:0/1334288835 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f5b700730a0 0x7f5b70075560 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f5b840b47b0 tx=0x7f5b78009040 comp rx=0 tx=0).ready entity=mgr.4135 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:46:05.210 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:05.205+0000 7f5b6bfff640 1 -- 172.21.15.123:0/1334288835 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f5b7405fb40 con 0x7f5b840726c0 2024-09-16T15:46:05.413 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:05.409+0000 7f5b89953640 1 -- 172.21.15.123:0/1334288835 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f5b8415f630 con 0x7f5b840726c0 2024-09-16T15:46:05.415 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:05.413+0000 7f5b6bfff640 1 -- 172.21.15.123:0/1334288835 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) ==== 94+0+5223 (secure 0 0 0) 0x7f5b7405f440 con 0x7f5b840726c0 2024-09-16T15:46:05.415 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-09-16T15:46:05.416 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":28,"btime":"2024-09-16T15:44:32:363512+0000","default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4712,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6837/288264117","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6836","nonce":288264117},{"type":"v1","addr":"172.21.15.184:6837","nonce":288264117}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":23},{"gid":5268,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6834/60194515","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6832","nonce":60194515},{"type":"v1","addr":"172.21.15.123:6834","nonce":60194515}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":23},{"gid":5271,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6835/1798373479","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6833","nonce":1798373479},{"type":"v1","addr":"172.21.15.123:6835","nonce":1798373479}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":22}],"filesystems":[{"mdsmap":{"epoch":27,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-16T15:44:14.853230+0000","modified":"2024-09-16T15:44:17.861742+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5280},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5280":{"gid":5280,"name":"d","rank":0,"incarnation":24,"state":"up:active","state_seq":3,"addr":"172.21.15.184:6835/374389412","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":374389412},{"type":"v1","addr":"172.21.15.184:6835","nonce":374389412}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5280,"qdb_cluster":[5280]},"id":3}]} 2024-09-16T15:46:05.416 INFO:teuthology.orchestra.run.smithi123.stderr:dumped fsmap epoch 28 2024-09-16T15:46:05.418 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:05.413+0000 7f5b89953640 1 -- 172.21.15.123:0/1334288835 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f5b700730a0 msgr2=0x7f5b70075560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:46:05.418 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:05.413+0000 7f5b89953640 1 --2- 172.21.15.123:0/1334288835 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f5b700730a0 0x7f5b70075560 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f5b840b47b0 tx=0x7f5b78009040 comp rx=0 tx=0).stop 2024-09-16T15:46:05.418 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:05.413+0000 7f5b89953640 1 -- 172.21.15.123:0/1334288835 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5b840726c0 msgr2=0x7f5b841f8240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:46:05.418 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:05.413+0000 7f5b89953640 1 --2- 172.21.15.123:0/1334288835 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5b840726c0 0x7f5b841f8240 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7f5b7400e7d0 tx=0x7f5b7400ec90 comp rx=0 tx=0).stop 2024-09-16T15:46:05.418 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:05.413+0000 7f5b89953640 1 -- 172.21.15.123:0/1334288835 shutdown_connections 2024-09-16T15:46:05.418 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:05.413+0000 7f5b89953640 1 --2- 172.21.15.123:0/1334288835 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f5b700730a0 0x7f5b70075560 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:05.418 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:05.413+0000 7f5b89953640 1 --2- 172.21.15.123:0/1334288835 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5b8415bca0 0x7f5b840cdbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:05.418 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:05.413+0000 7f5b89953640 1 --2- 172.21.15.123:0/1334288835 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f5b84074600 0x7f5b8415e980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:05.418 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:05.413+0000 7f5b89953640 1 --2- 172.21.15.123:0/1334288835 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5b840726c0 0x7f5b841f8240 unknown :-1 s=CLOSED pgs=553 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:05.418 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:05.413+0000 7f5b89953640 1 -- 172.21.15.123:0/1334288835 >> 172.21.15.123:0/1334288835 conn(0x7f5b840cfb10 msgr2=0x7f5b840c31c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:46:05.418 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:05.413+0000 7f5b89953640 1 -- 172.21.15.123:0/1334288835 shutdown_connections 2024-09-16T15:46:05.419 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:05.413+0000 7f5b89953640 1 -- 172.21.15.123:0/1334288835 wait complete. 2024-09-16T15:46:05.433 DEBUG:teuthology.orchestra.run.smithi184:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.d.asok --format=json session ls 2024-09-16T15:46:05.582 INFO:teuthology.orchestra.run.smithi184.stdout:[] 2024-09-16T15:46:05.582 DEBUG:tasks.cephfs.filesystem:_json_asok output [] 2024-09-16T15:46:05.583 DEBUG:tasks.ceph_test_case:wait_until_true: success in 75s and 0 retries 2024-09-16T15:46:05.583 INFO:teuthology.nuke:Clearing teuthology firewall rules... 2024-09-16T15:46:05.583 DEBUG:teuthology.orchestra.run.smithi123:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-09-16T15:46:05.661 DEBUG:teuthology.orchestra.run.smithi184:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-09-16T15:46:05.736 INFO:teuthology.nuke:Cleared teuthology firewall rules. 2024-09-16T15:46:05.736 INFO:tasks.cephfs.mount:Terminating background process 2024-09-16T15:46:05.737 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2024-09-16T15:46:05.737 DEBUG:teuthology.orchestra.run.smithi123:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-16T15:46:05.743 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16336468k,nr_inodes=4084117,mode=755,inode64 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3274684k,mode=755,inode64 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev,inode64 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=17082 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3274680k,nr_inodes=818670,mode=700,uid=1000,gid=1271,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3274684k,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-09-16T15:46:05.743 INFO:tasks.cephfs.mount:Terminating background process 2024-09-16T15:46:05.743 DEBUG:teuthology.orchestra.run.smithi184:> set -ex 2024-09-16T15:46:05.743 DEBUG:teuthology.orchestra.run.smithi184:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-16T15:46:05.749 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16336488k,nr_inodes=4084122,mode=755,inode64 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3274688k,mode=755,inode64 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev,inode64 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=17834 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3274684k,nr_inodes=818671,mode=700,uid=1000,gid=1271,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3274688k,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-09-16T15:46:05.749 INFO:tasks.cephfs.mount:Terminating background process 2024-09-16T15:46:05.749 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2024-09-16T15:46:05.749 DEBUG:teuthology.orchestra.run.smithi123:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-16T15:46:05.792 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16336468k,nr_inodes=4084117,mode=755,inode64 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3274684k,mode=755,inode64 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev,inode64 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=17082 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3274680k,nr_inodes=818670,mode=700,uid=1000,gid=1271,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3274684k,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-09-16T15:46:05.792 INFO:tasks.cephfs.mount:Terminating background process 2024-09-16T15:46:05.792 DEBUG:teuthology.orchestra.run.smithi184:> set -ex 2024-09-16T15:46:05.792 DEBUG:teuthology.orchestra.run.smithi184:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-16T15:46:05.799 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16336488k,nr_inodes=4084122,mode=755,inode64 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3274688k,mode=755,inode64 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev,inode64 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=17834 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3274684k,nr_inodes=818671,mode=700,uid=1000,gid=1271,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3274688k,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-09-16T15:46:05.799 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-16T15:46:05.970 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:05.965+0000 7f7e6fb30640 1 -- 172.21.15.123:0/1814734115 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7e54004e70 msgr2=0x7f7e5400b9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:46:05.970 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:05.965+0000 7f7e6fb30640 1 --2- 172.21.15.123:0/1814734115 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7e54004e70 0x7f7e5400b9d0 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7f7e6000c640 tx=0x7f7e6001f9a0 comp rx=0 tx=0).stop 2024-09-16T15:46:05.970 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:05.965+0000 7f7e6e0cd640 1 -- 172.21.15.123:0/1814734115 reap_dead start 2024-09-16T15:46:05.970 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:05.965+0000 7f7e6fb30640 1 -- 172.21.15.123:0/1814734115 shutdown_connections 2024-09-16T15:46:05.970 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:05.965+0000 7f7e6fb30640 1 -- 172.21.15.123:0/1814734115 >> 172.21.15.123:0/1814734115 conn(0x7f7e680cfb10 msgr2=0x7f7e680cff20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:46:05.970 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:05.965+0000 7f7e6fb30640 1 -- 172.21.15.123:0/1814734115 shutdown_connections 2024-09-16T15:46:05.970 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:05.965+0000 7f7e6fb30640 1 -- 172.21.15.123:0/1814734115 wait complete. 2024-09-16T15:46:05.971 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:05.969+0000 7f7e6fb30640 1 Processor -- start 2024-09-16T15:46:05.971 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:05.969+0000 7f7e6fb30640 1 -- start start 2024-09-16T15:46:05.973 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:05.969+0000 7f7e6fb30640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7e54004e70 0x7f7e680b0e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:46:05.973 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:05.969+0000 7f7e6fb30640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7e681f3750 0x7f7e68159f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:46:05.973 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:05.969+0000 7f7e6fb30640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f7e6815a4a0 0x7f7e6814ac50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:46:05.973 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:05.969+0000 7f7e6fb30640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f7e680d0a10 con 0x7f7e681f3750 2024-09-16T15:46:05.973 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:05.969+0000 7f7e6fb30640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f7e680d0890 con 0x7f7e54004e70 2024-09-16T15:46:05.973 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:05.969+0000 7f7e6fb30640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f7e6806ee60 con 0x7f7e6815a4a0 2024-09-16T15:46:05.973 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:05.969+0000 7f7e6d0cb640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7e681f3750 0x7f7e68159f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:46:05.973 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:05.969+0000 7f7e6d0cb640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7e681f3750 0x7f7e68159f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:52060/0 (socket says 172.21.15.123:52060) 2024-09-16T15:46:05.973 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:05.969+0000 7f7e6d0cb640 1 -- 172.21.15.123:0/754271292 learned_addr learned my addr 172.21.15.123:0/754271292 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:46:05.973 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:05.969+0000 7f7e6d8cc640 1 --2- 172.21.15.123:0/754271292 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7e54004e70 0x7f7e680b0e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:46:05.974 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:05.969+0000 7f7e6e0cd640 1 --2- 172.21.15.123:0/754271292 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f7e6815a4a0 0x7f7e6814ac50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:46:05.974 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:05.969+0000 7f7e6d0cb640 1 -- 172.21.15.123:0/754271292 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f7e6815a4a0 msgr2=0x7f7e6814ac50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:46:05.974 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:05.969+0000 7f7e6d0cb640 1 --2- 172.21.15.123:0/754271292 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f7e6815a4a0 0x7f7e6814ac50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:05.974 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:05.969+0000 7f7e6d0cb640 1 -- 172.21.15.123:0/754271292 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7e54004e70 msgr2=0x7f7e680b0e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:46:05.974 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:05.969+0000 7f7e6d0cb640 1 --2- 172.21.15.123:0/754271292 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7e54004e70 0x7f7e680b0e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:05.974 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:05.969+0000 7f7e6d0cb640 1 -- 172.21.15.123:0/754271292 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7e6814b960 con 0x7f7e681f3750 2024-09-16T15:46:05.974 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:05.969+0000 7f7e6d8cc640 1 --2- 172.21.15.123:0/754271292 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7e54004e70 0x7f7e680b0e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T15:46:05.974 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:05.969+0000 7f7e6d0cb640 1 --2- 172.21.15.123:0/754271292 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7e681f3750 0x7f7e68159f60 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7f7e64003ef0 tx=0x7f7e64001640 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:46:05.974 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:05.969+0000 7f7e5e7fc640 1 -- 172.21.15.123:0/754271292 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e64011640 con 0x7f7e681f3750 2024-09-16T15:46:05.974 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:05.969+0000 7f7e5e7fc640 1 -- 172.21.15.123:0/754271292 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f7e64016020 con 0x7f7e681f3750 2024-09-16T15:46:05.974 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:05.969+0000 7f7e6fb30640 1 -- 172.21.15.123:0/754271292 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7e681f2c70 con 0x7f7e681f3750 2024-09-16T15:46:05.974 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:05.969+0000 7f7e5e7fc640 1 -- 172.21.15.123:0/754271292 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e64011a80 con 0x7f7e681f3750 2024-09-16T15:46:05.974 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:05.969+0000 7f7e6fb30640 1 -- 172.21.15.123:0/754271292 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7e681f3230 con 0x7f7e681f3750 2024-09-16T15:46:05.975 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:05.969+0000 7f7e6fb30640 1 -- 172.21.15.123:0/754271292 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7e681f2a60 con 0x7f7e681f3750 2024-09-16T15:46:05.975 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:05.969+0000 7f7e5e7fc640 1 -- 172.21.15.123:0/754271292 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 9) ==== 97233+0+0 (secure 0 0 0) 0x7f7e64011c20 con 0x7f7e681f3750 2024-09-16T15:46:05.976 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:05.969+0000 7f7e5e7fc640 1 --2- 172.21.15.123:0/754271292 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f7e4c0730a0 0x7f7e4c075560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:46:05.976 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:05.969+0000 7f7e5e7fc640 1 -- 172.21.15.123:0/754271292 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(46..46 src has 1..46) ==== 5767+0+0 (secure 0 0 0) 0x7f7e64094ee0 con 0x7f7e681f3750 2024-09-16T15:46:05.976 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:05.973+0000 7f7e6d8cc640 1 --2- 172.21.15.123:0/754271292 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f7e4c0730a0 0x7f7e4c075560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:46:05.976 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:05.973+0000 7f7e6d8cc640 1 --2- 172.21.15.123:0/754271292 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f7e4c0730a0 0x7f7e4c075560 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f7e6815f0e0 tx=0x7f7e5800b260 comp rx=0 tx=0).ready entity=mgr.4135 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:46:05.978 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:05.973+0000 7f7e5e7fc640 1 -- 172.21.15.123:0/754271292 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f7e64020020 con 0x7f7e681f3750 2024-09-16T15:46:06.169 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:06.165+0000 7f7e6fb30640 1 -- 172.21.15.123:0/754271292 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f7e6815a880 con 0x7f7e681f3750 2024-09-16T15:46:06.170 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:06.165+0000 7f7e5e7fc640 1 -- 172.21.15.123:0/754271292 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) ==== 94+0+5223 (secure 0 0 0) 0x7f7e6405fdf0 con 0x7f7e681f3750 2024-09-16T15:46:06.170 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-09-16T15:46:06.170 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":28,"btime":"2024-09-16T15:44:32:363512+0000","default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4712,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6837/288264117","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6836","nonce":288264117},{"type":"v1","addr":"172.21.15.184:6837","nonce":288264117}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":23},{"gid":5268,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6834/60194515","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6832","nonce":60194515},{"type":"v1","addr":"172.21.15.123:6834","nonce":60194515}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":23},{"gid":5271,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6835/1798373479","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6833","nonce":1798373479},{"type":"v1","addr":"172.21.15.123:6835","nonce":1798373479}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":22}],"filesystems":[{"mdsmap":{"epoch":27,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-16T15:44:14.853230+0000","modified":"2024-09-16T15:44:17.861742+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5280},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5280":{"gid":5280,"name":"d","rank":0,"incarnation":24,"state":"up:active","state_seq":3,"addr":"172.21.15.184:6835/374389412","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":374389412},{"type":"v1","addr":"172.21.15.184:6835","nonce":374389412}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5280,"qdb_cluster":[5280]},"id":3}]} 2024-09-16T15:46:06.171 INFO:teuthology.orchestra.run.smithi123.stderr:dumped fsmap epoch 28 2024-09-16T15:46:06.173 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:06.169+0000 7f7e6fb30640 1 -- 172.21.15.123:0/754271292 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f7e4c0730a0 msgr2=0x7f7e4c075560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:46:06.173 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:06.169+0000 7f7e6fb30640 1 --2- 172.21.15.123:0/754271292 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f7e4c0730a0 0x7f7e4c075560 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f7e6815f0e0 tx=0x7f7e5800b260 comp rx=0 tx=0).stop 2024-09-16T15:46:06.173 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:06.169+0000 7f7e6fb30640 1 -- 172.21.15.123:0/754271292 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7e681f3750 msgr2=0x7f7e68159f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:46:06.173 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:06.169+0000 7f7e6fb30640 1 --2- 172.21.15.123:0/754271292 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7e681f3750 0x7f7e68159f60 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7f7e64003ef0 tx=0x7f7e64001640 comp rx=0 tx=0).stop 2024-09-16T15:46:06.173 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:06.169+0000 7f7e6fb30640 1 -- 172.21.15.123:0/754271292 shutdown_connections 2024-09-16T15:46:06.173 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:06.169+0000 7f7e6fb30640 1 --2- 172.21.15.123:0/754271292 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f7e4c0730a0 0x7f7e4c075560 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:06.173 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:06.169+0000 7f7e6fb30640 1 --2- 172.21.15.123:0/754271292 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f7e6815a4a0 0x7f7e6814ac50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:06.173 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:06.169+0000 7f7e6fb30640 1 --2- 172.21.15.123:0/754271292 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7e681f3750 0x7f7e68159f60 unknown :-1 s=CLOSED pgs=556 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:06.173 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:06.169+0000 7f7e6fb30640 1 --2- 172.21.15.123:0/754271292 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7e54004e70 0x7f7e680b0e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:06.173 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:06.169+0000 7f7e6fb30640 1 -- 172.21.15.123:0/754271292 >> 172.21.15.123:0/754271292 conn(0x7f7e680cfb10 msgr2=0x7f7e680c59c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:46:06.173 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:06.169+0000 7f7e6fb30640 1 -- 172.21.15.123:0/754271292 shutdown_connections 2024-09-16T15:46:06.173 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:06.169+0000 7f7e6fb30640 1 -- 172.21.15.123:0/754271292 wait complete. 2024-09-16T15:46:06.188 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-16T15:46:06.308 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:06.305+0000 7fe2ab266640 1 -- 172.21.15.123:0/3602814106 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe2900052f0 msgr2=0x7fe29000b9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:46:06.308 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:06.305+0000 7fe2ab266640 1 --2- 172.21.15.123:0/3602814106 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe2900052f0 0x7fe29000b9d0 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7fe29c009e30 tx=0x7fe29c01d1b0 comp rx=0 tx=0).stop 2024-09-16T15:46:06.308 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:06.305+0000 7fe2ab266640 1 -- 172.21.15.123:0/3602814106 shutdown_connections 2024-09-16T15:46:06.308 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:06.305+0000 7fe2ab266640 1 --2- 172.21.15.123:0/3602814106 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe2900052f0 0x7fe29000b9d0 unknown :-1 s=CLOSED pgs=558 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:06.308 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:06.305+0000 7fe2ab266640 1 -- 172.21.15.123:0/3602814106 >> 172.21.15.123:0/3602814106 conn(0x7fe2a40cfb10 msgr2=0x7fe2a40cff20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:46:06.308 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:06.305+0000 7fe2ab266640 1 -- 172.21.15.123:0/3602814106 shutdown_connections 2024-09-16T15:46:06.308 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:06.305+0000 7fe2ab266640 1 -- 172.21.15.123:0/3602814106 wait complete. 2024-09-16T15:46:06.309 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:06.305+0000 7fe2ab266640 1 Processor -- start 2024-09-16T15:46:06.309 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:06.305+0000 7fe2ab266640 1 -- start start 2024-09-16T15:46:06.311 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:06.305+0000 7fe2ab266640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fe2900052f0 0x7fe2a40adb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:46:06.311 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:06.305+0000 7fe2ab266640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe2a40afe40 0x7fe2a414ce40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:46:06.311 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:06.305+0000 7fe2ab266640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe2a415e4e0 0x7fe2a415e960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:46:06.311 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:06.305+0000 7fe2ab266640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fe2a40d09c0 con 0x7fe2a40afe40 2024-09-16T15:46:06.311 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:06.305+0000 7fe2ab266640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fe2a40d0840 con 0x7fe2a415e4e0 2024-09-16T15:46:06.311 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:06.305+0000 7fe2ab266640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fe2a415c190 con 0x7fe2900052f0 2024-09-16T15:46:06.312 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:06.305+0000 7fe2a8801640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe2a40afe40 0x7fe2a414ce40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:46:06.312 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:06.305+0000 7fe2a8801640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe2a40afe40 0x7fe2a414ce40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:52090/0 (socket says 172.21.15.123:52090) 2024-09-16T15:46:06.312 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:06.305+0000 7fe2a8801640 1 -- 172.21.15.123:0/3853586576 learned_addr learned my addr 172.21.15.123:0/3853586576 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:46:06.312 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:06.305+0000 7fe2a9002640 1 --2- 172.21.15.123:0/3853586576 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fe2900052f0 0x7fe2a40adb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:46:06.312 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:06.305+0000 7fe2a9803640 1 --2- 172.21.15.123:0/3853586576 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe2a415e4e0 0x7fe2a415e960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:46:06.312 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:06.305+0000 7fe2a8801640 1 -- 172.21.15.123:0/3853586576 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fe2900052f0 msgr2=0x7fe2a40adb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:46:06.312 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:06.305+0000 7fe2a8801640 1 --2- 172.21.15.123:0/3853586576 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fe2900052f0 0x7fe2a40adb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:06.312 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:06.305+0000 7fe2a8801640 1 -- 172.21.15.123:0/3853586576 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe2a415e4e0 msgr2=0x7fe2a415e960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:46:06.312 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:06.305+0000 7fe2a8801640 1 --2- 172.21.15.123:0/3853586576 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe2a415e4e0 0x7fe2a415e960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:06.312 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:06.305+0000 7fe2a8801640 1 -- 172.21.15.123:0/3853586576 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe2a40b0690 con 0x7fe2a40afe40 2024-09-16T15:46:06.312 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:06.305+0000 7fe2a9002640 1 --2- 172.21.15.123:0/3853586576 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fe2900052f0 0x7fe2a40adb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T15:46:06.312 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:06.305+0000 7fe2a9803640 1 --2- 172.21.15.123:0/3853586576 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe2a415e4e0 0x7fe2a415e960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T15:46:06.312 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:06.305+0000 7fe2a8801640 1 --2- 172.21.15.123:0/3853586576 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe2a40afe40 0x7fe2a414ce40 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7fe2a0001490 tx=0x7fe2a000fa30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:46:06.313 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:06.305+0000 7fe299ffb640 1 -- 172.21.15.123:0/3853586576 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2a0017070 con 0x7fe2a40afe40 2024-09-16T15:46:06.313 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:06.305+0000 7fe299ffb640 1 -- 172.21.15.123:0/3853586576 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fe2a0004310 con 0x7fe2a40afe40 2024-09-16T15:46:06.313 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:06.305+0000 7fe2ab266640 1 -- 172.21.15.123:0/3853586576 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe2a40b0970 con 0x7fe2a40afe40 2024-09-16T15:46:06.313 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:06.305+0000 7fe299ffb640 1 -- 172.21.15.123:0/3853586576 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2a0011900 con 0x7fe2a40afe40 2024-09-16T15:46:06.313 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:06.305+0000 7fe2ab266640 1 -- 172.21.15.123:0/3853586576 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe2a40b0f30 con 0x7fe2a40afe40 2024-09-16T15:46:06.314 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:06.309+0000 7fe2ab266640 1 -- 172.21.15.123:0/3853586576 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe2a415eea0 con 0x7fe2a40afe40 2024-09-16T15:46:06.314 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:06.309+0000 7fe299ffb640 1 -- 172.21.15.123:0/3853586576 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 9) ==== 97233+0+0 (secure 0 0 0) 0x7fe2a0011aa0 con 0x7fe2a40afe40 2024-09-16T15:46:06.314 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:06.309+0000 7fe299ffb640 1 --2- 172.21.15.123:0/3853586576 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fe288073070 0x7fe288075530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:46:06.314 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:06.309+0000 7fe299ffb640 1 -- 172.21.15.123:0/3853586576 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(46..46 src has 1..46) ==== 5767+0+0 (secure 0 0 0) 0x7fe2a00972b0 con 0x7fe2a40afe40 2024-09-16T15:46:06.314 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:06.309+0000 7fe2a9002640 1 --2- 172.21.15.123:0/3853586576 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fe288073070 0x7fe288075530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:46:06.314 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:06.309+0000 7fe2a9002640 1 --2- 172.21.15.123:0/3853586576 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fe288073070 0x7fe288075530 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7fe294006ec0 tx=0x7fe2940061f0 comp rx=0 tx=0).ready entity=mgr.4135 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:46:06.317 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:06.313+0000 7fe299ffb640 1 -- 172.21.15.123:0/3853586576 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7fe2a0062020 con 0x7fe2a40afe40 2024-09-16T15:46:06.521 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:06.517+0000 7fe2ab266640 1 -- 172.21.15.123:0/3853586576 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe2a4074270 con 0x7fe2a40afe40 2024-09-16T15:46:06.521 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:06.517+0000 7fe299ffb640 1 -- 172.21.15.123:0/3853586576 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) ==== 94+0+5223 (secure 0 0 0) 0x7fe2a00970d0 con 0x7fe2a40afe40 2024-09-16T15:46:06.521 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-09-16T15:46:06.522 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":28,"btime":"2024-09-16T15:44:32:363512+0000","default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4712,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6837/288264117","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6836","nonce":288264117},{"type":"v1","addr":"172.21.15.184:6837","nonce":288264117}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":23},{"gid":5268,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6834/60194515","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6832","nonce":60194515},{"type":"v1","addr":"172.21.15.123:6834","nonce":60194515}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":23},{"gid":5271,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6835/1798373479","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6833","nonce":1798373479},{"type":"v1","addr":"172.21.15.123:6835","nonce":1798373479}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":22}],"filesystems":[{"mdsmap":{"epoch":27,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-16T15:44:14.853230+0000","modified":"2024-09-16T15:44:17.861742+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5280},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5280":{"gid":5280,"name":"d","rank":0,"incarnation":24,"state":"up:active","state_seq":3,"addr":"172.21.15.184:6835/374389412","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":374389412},{"type":"v1","addr":"172.21.15.184:6835","nonce":374389412}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5280,"qdb_cluster":[5280]},"id":3}]} 2024-09-16T15:46:06.522 INFO:teuthology.orchestra.run.smithi123.stderr:dumped fsmap epoch 28 2024-09-16T15:46:06.524 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:06.521+0000 7fe2ab266640 1 -- 172.21.15.123:0/3853586576 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fe288073070 msgr2=0x7fe288075530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:46:06.524 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:06.521+0000 7fe2ab266640 1 --2- 172.21.15.123:0/3853586576 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fe288073070 0x7fe288075530 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7fe294006ec0 tx=0x7fe2940061f0 comp rx=0 tx=0).stop 2024-09-16T15:46:06.524 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:06.521+0000 7fe2ab266640 1 -- 172.21.15.123:0/3853586576 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe2a40afe40 msgr2=0x7fe2a414ce40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:46:06.525 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:06.521+0000 7fe2ab266640 1 --2- 172.21.15.123:0/3853586576 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe2a40afe40 0x7fe2a414ce40 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7fe2a0001490 tx=0x7fe2a000fa30 comp rx=0 tx=0).stop 2024-09-16T15:46:06.525 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:06.521+0000 7fe2ab266640 1 -- 172.21.15.123:0/3853586576 shutdown_connections 2024-09-16T15:46:06.525 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:06.521+0000 7fe2ab266640 1 --2- 172.21.15.123:0/3853586576 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fe288073070 0x7fe288075530 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:06.525 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:06.521+0000 7fe2ab266640 1 --2- 172.21.15.123:0/3853586576 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe2a415e4e0 0x7fe2a415e960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:06.525 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:06.521+0000 7fe2ab266640 1 --2- 172.21.15.123:0/3853586576 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe2a40afe40 0x7fe2a414ce40 unknown :-1 s=CLOSED pgs=559 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:06.526 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:06.521+0000 7fe2ab266640 1 --2- 172.21.15.123:0/3853586576 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fe2900052f0 0x7fe2a40adb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:06.526 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:06.521+0000 7fe2ab266640 1 -- 172.21.15.123:0/3853586576 >> 172.21.15.123:0/3853586576 conn(0x7fe2a40cfb10 msgr2=0x7fe2a40c94e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:46:06.526 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:06.521+0000 7fe2ab266640 1 -- 172.21.15.123:0/3853586576 shutdown_connections 2024-09-16T15:46:06.526 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:06.521+0000 7fe2ab266640 1 -- 172.21.15.123:0/3853586576 wait complete. 2024-09-16T15:46:06.538 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-16T15:46:06.661 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:06.657+0000 7f7ef96a1640 1 -- 172.21.15.123:0/3240102787 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7ef415f420 msgr2=0x7f7ef4068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:46:06.661 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:06.657+0000 7f7ef96a1640 1 --2- 172.21.15.123:0/3240102787 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7ef415f420 0x7f7ef4068960 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7f7eec009870 tx=0x7f7eec01c7c0 comp rx=0 tx=0).stop 2024-09-16T15:46:06.661 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:06.657+0000 7f7ef96a1640 1 -- 172.21.15.123:0/3240102787 shutdown_connections 2024-09-16T15:46:06.661 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:06.657+0000 7f7ef96a1640 1 --2- 172.21.15.123:0/3240102787 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f7ef40b2910 0x7f7ef40741c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:06.661 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:06.657+0000 7f7ef96a1640 1 --2- 172.21.15.123:0/3240102787 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7ef415f420 0x7f7ef4068960 unknown :-1 s=CLOSED pgs=561 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:06.662 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:06.657+0000 7f7ef96a1640 1 -- 172.21.15.123:0/3240102787 >> 172.21.15.123:0/3240102787 conn(0x7f7ef40b1df0 msgr2=0x7f7ef40b2200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:46:06.662 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:06.657+0000 7f7ef96a1640 1 -- 172.21.15.123:0/3240102787 shutdown_connections 2024-09-16T15:46:06.662 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:06.657+0000 7f7ef96a1640 1 -- 172.21.15.123:0/3240102787 wait complete. 2024-09-16T15:46:06.662 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:06.657+0000 7f7ef96a1640 1 Processor -- start 2024-09-16T15:46:06.662 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:06.657+0000 7f7ef96a1640 1 -- start start 2024-09-16T15:46:06.664 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:06.661+0000 7f7ef96a1640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f7ef4074700 0x7f7ef4099640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:46:06.664 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:06.661+0000 7f7ef96a1640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7ef40b2910 0x7f7ef41546f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:46:06.664 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:06.661+0000 7f7ef96a1640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7ef415f420 0x7f7ef4098840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:46:06.664 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:06.661+0000 7f7ef96a1640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f7ef41690e0 con 0x7f7ef415f420 2024-09-16T15:46:06.664 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:06.661+0000 7f7ef96a1640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f7ef4168f60 con 0x7f7ef40b2910 2024-09-16T15:46:06.664 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:06.661+0000 7f7ef96a1640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f7ef4169260 con 0x7f7ef4074700 2024-09-16T15:46:06.664 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:06.661+0000 7f7ef37fe640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7ef415f420 0x7f7ef4098840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:46:06.665 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:06.661+0000 7f7ef37fe640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7ef415f420 0x7f7ef4098840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:52108/0 (socket says 172.21.15.123:52108) 2024-09-16T15:46:06.665 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:06.661+0000 7f7ef37fe640 1 -- 172.21.15.123:0/486351278 learned_addr learned my addr 172.21.15.123:0/486351278 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:46:06.665 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:06.661+0000 7f7ef2ffd640 1 --2- 172.21.15.123:0/486351278 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f7ef4074700 0x7f7ef4099640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:46:06.665 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:06.661+0000 7f7ef37fe640 1 -- 172.21.15.123:0/486351278 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f7ef4074700 msgr2=0x7f7ef4099640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:46:06.665 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:06.661+0000 7f7ef37fe640 1 --2- 172.21.15.123:0/486351278 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f7ef4074700 0x7f7ef4099640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:06.665 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:06.661+0000 7f7ef37fe640 1 -- 172.21.15.123:0/486351278 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7ef40b2910 msgr2=0x7f7ef41546f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:46:06.665 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:06.661+0000 7f7ef37fe640 1 --2- 172.21.15.123:0/486351278 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7ef40b2910 0x7f7ef41546f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:06.665 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:06.661+0000 7f7ef37fe640 1 -- 172.21.15.123:0/486351278 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7ef40c5f00 con 0x7f7ef415f420 2024-09-16T15:46:06.665 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:06.661+0000 7f7ef37fe640 1 --2- 172.21.15.123:0/486351278 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7ef415f420 0x7f7ef4098840 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7f7ee400cbf0 tx=0x7f7ee400cce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:46:06.665 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:06.661+0000 7f7eeb7fe640 1 -- 172.21.15.123:0/486351278 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ee4016020 con 0x7f7ef415f420 2024-09-16T15:46:06.665 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:06.661+0000 7f7eeb7fe640 1 -- 172.21.15.123:0/486351278 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f7ee400ab40 con 0x7f7ef415f420 2024-09-16T15:46:06.665 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:06.661+0000 7f7ef96a1640 1 -- 172.21.15.123:0/486351278 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7ef4099360 con 0x7f7ef415f420 2024-09-16T15:46:06.665 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:06.661+0000 7f7eeb7fe640 1 -- 172.21.15.123:0/486351278 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ee4012420 con 0x7f7ef415f420 2024-09-16T15:46:06.666 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:06.661+0000 7f7ef96a1640 1 -- 172.21.15.123:0/486351278 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7ef4154c30 con 0x7f7ef415f420 2024-09-16T15:46:06.666 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:06.661+0000 7f7ef96a1640 1 -- 172.21.15.123:0/486351278 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7ef40741c0 con 0x7f7ef415f420 2024-09-16T15:46:06.666 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:06.661+0000 7f7eeb7fe640 1 -- 172.21.15.123:0/486351278 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 9) ==== 97233+0+0 (secure 0 0 0) 0x7f7ee4019020 con 0x7f7ef415f420 2024-09-16T15:46:06.666 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:06.661+0000 7f7eeb7fe640 1 --2- 172.21.15.123:0/486351278 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f7ed40730a0 0x7f7ed4075560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:46:06.666 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:06.661+0000 7f7eeb7fe640 1 -- 172.21.15.123:0/486351278 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(46..46 src has 1..46) ==== 5767+0+0 (secure 0 0 0) 0x7f7ee4094160 con 0x7f7ef415f420 2024-09-16T15:46:06.667 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:06.661+0000 7f7ef2ffd640 1 --2- 172.21.15.123:0/486351278 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f7ed40730a0 0x7f7ed4075560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:46:06.667 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:06.665+0000 7f7ef2ffd640 1 --2- 172.21.15.123:0/486351278 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f7ed40730a0 0x7f7ed4075560 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f7eec0af000 tx=0x7f7eec01b060 comp rx=0 tx=0).ready entity=mgr.4135 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:46:06.670 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:06.665+0000 7f7eeb7fe640 1 -- 172.21.15.123:0/486351278 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f7ee4020030 con 0x7f7ef415f420 2024-09-16T15:46:06.841 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:06.837+0000 7f7ef96a1640 1 -- 172.21.15.123:0/486351278 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f7ef409e3a0 con 0x7f7ef415f420 2024-09-16T15:46:06.843 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:06.841+0000 7f7eeb7fe640 1 -- 172.21.15.123:0/486351278 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v46) ==== 74+0+16885 (secure 0 0 0) 0x7f7ee405f070 con 0x7f7ef415f420 2024-09-16T15:46:06.844 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-09-16T15:46:06.844 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":46,"fsid":"56077f89-18c8-486f-ae7e-eecc0bb7822c","created":"2024-09-16T15:42:26.364892+0000","modified":"2024-09-16T15:44:32.363194+0000","last_up_change":"2024-09-16T15:42:38.707234+0000","last_in_change":"2024-09-16T15:42:29.423662+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-16T15:42:39.287764+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_metadata","create_time":"2024-09-16T15:44:12.463367+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"43","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_data","create_time":"2024-09-16T15:44:13.817170+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"44","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9e7c1955-a607-4671-9be7-f2ef760ceeaa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6800","nonce":3882044969},{"type":"v1","addr":"172.21.15.123:6801","nonce":3882044969}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6802","nonce":3882044969},{"type":"v1","addr":"172.21.15.123:6803","nonce":3882044969}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6806","nonce":3882044969},{"type":"v1","addr":"172.21.15.123:6807","nonce":3882044969}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6804","nonce":3882044969},{"type":"v1","addr":"172.21.15.123:6805","nonce":3882044969}]},"public_addr":"172.21.15.123:6801/3882044969","cluster_addr":"172.21.15.123:6803/3882044969","heartbeat_back_addr":"172.21.15.123:6807/3882044969","heartbeat_front_addr":"172.21.15.123:6805/3882044969","state":["exists","up"]},{"osd":1,"uuid":"c540cfde-0977-4742-9b17-fd634668e010","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6816","nonce":4037001850},{"type":"v1","addr":"172.21.15.123:6817","nonce":4037001850}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6818","nonce":4037001850},{"type":"v1","addr":"172.21.15.123:6819","nonce":4037001850}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6822","nonce":4037001850},{"type":"v1","addr":"172.21.15.123:6823","nonce":4037001850}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6820","nonce":4037001850},{"type":"v1","addr":"172.21.15.123:6821","nonce":4037001850}]},"public_addr":"172.21.15.123:6817/4037001850","cluster_addr":"172.21.15.123:6819/4037001850","heartbeat_back_addr":"172.21.15.123:6823/4037001850","heartbeat_front_addr":"172.21.15.123:6821/4037001850","state":["exists","up"]},{"osd":2,"uuid":"9864a496-f169-486b-95c4-1d59162793a7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6824","nonce":1332143088},{"type":"v1","addr":"172.21.15.123:6825","nonce":1332143088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6826","nonce":1332143088},{"type":"v1","addr":"172.21.15.123:6827","nonce":1332143088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6830","nonce":1332143088},{"type":"v1","addr":"172.21.15.123:6831","nonce":1332143088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6828","nonce":1332143088},{"type":"v1","addr":"172.21.15.123:6829","nonce":1332143088}]},"public_addr":"172.21.15.123:6825/1332143088","cluster_addr":"172.21.15.123:6827/1332143088","heartbeat_back_addr":"172.21.15.123:6831/1332143088","heartbeat_front_addr":"172.21.15.123:6829/1332143088","state":["exists","up"]},{"osd":3,"uuid":"c6d1f817-f074-4325-95e1-578eba13edda","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6808","nonce":988586998},{"type":"v1","addr":"172.21.15.123:6809","nonce":988586998}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6810","nonce":988586998},{"type":"v1","addr":"172.21.15.123:6811","nonce":988586998}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6814","nonce":988586998},{"type":"v1","addr":"172.21.15.123:6815","nonce":988586998}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6812","nonce":988586998},{"type":"v1","addr":"172.21.15.123:6813","nonce":988586998}]},"public_addr":"172.21.15.123:6809/988586998","cluster_addr":"172.21.15.123:6811/988586998","heartbeat_back_addr":"172.21.15.123:6815/988586998","heartbeat_front_addr":"172.21.15.123:6813/988586998","state":["exists","up"]},{"osd":4,"uuid":"200a8df4-50ef-40a5-b48a-58981269c086","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6816","nonce":3387813890},{"type":"v1","addr":"172.21.15.184:6817","nonce":3387813890}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6818","nonce":3387813890},{"type":"v1","addr":"172.21.15.184:6819","nonce":3387813890}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6822","nonce":3387813890},{"type":"v1","addr":"172.21.15.184:6823","nonce":3387813890}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6820","nonce":3387813890},{"type":"v1","addr":"172.21.15.184:6821","nonce":3387813890}]},"public_addr":"172.21.15.184:6817/3387813890","cluster_addr":"172.21.15.184:6819/3387813890","heartbeat_back_addr":"172.21.15.184:6823/3387813890","heartbeat_front_addr":"172.21.15.184:6821/3387813890","state":["exists","up"]},{"osd":5,"uuid":"32d5a900-da3d-412c-bc19-13861a2368d8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6808","nonce":2680916017},{"type":"v1","addr":"172.21.15.184:6809","nonce":2680916017}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6810","nonce":2680916017},{"type":"v1","addr":"172.21.15.184:6811","nonce":2680916017}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6814","nonce":2680916017},{"type":"v1","addr":"172.21.15.184:6815","nonce":2680916017}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6812","nonce":2680916017},{"type":"v1","addr":"172.21.15.184:6813","nonce":2680916017}]},"public_addr":"172.21.15.184:6809/2680916017","cluster_addr":"172.21.15.184:6811/2680916017","heartbeat_back_addr":"172.21.15.184:6815/2680916017","heartbeat_front_addr":"172.21.15.184:6813/2680916017","state":["exists","up"]},{"osd":6,"uuid":"cde6f39e-aa1b-4260-ac19-774ee6aaca72","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6824","nonce":2183120457},{"type":"v1","addr":"172.21.15.184:6825","nonce":2183120457}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6826","nonce":2183120457},{"type":"v1","addr":"172.21.15.184:6827","nonce":2183120457}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6830","nonce":2183120457},{"type":"v1","addr":"172.21.15.184:6831","nonce":2183120457}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6828","nonce":2183120457},{"type":"v1","addr":"172.21.15.184:6829","nonce":2183120457}]},"public_addr":"172.21.15.184:6825/2183120457","cluster_addr":"172.21.15.184:6827/2183120457","heartbeat_back_addr":"172.21.15.184:6831/2183120457","heartbeat_front_addr":"172.21.15.184:6829/2183120457","state":["exists","up"]},{"osd":7,"uuid":"4f381778-eee1-47c6-804f-df0e6b33e313","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6800","nonce":4102957285},{"type":"v1","addr":"172.21.15.184:6801","nonce":4102957285}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6802","nonce":4102957285},{"type":"v1","addr":"172.21.15.184:6803","nonce":4102957285}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6806","nonce":4102957285},{"type":"v1","addr":"172.21.15.184:6807","nonce":4102957285}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6804","nonce":4102957285},{"type":"v1","addr":"172.21.15.184:6805","nonce":4102957285}]},"public_addr":"172.21.15.184:6801/4102957285","cluster_addr":"172.21.15.184:6803/4102957285","heartbeat_back_addr":"172.21.15.184:6807/4102957285","heartbeat_front_addr":"172.21.15.184:6805/4102957285","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:35.972210+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.222007+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.198617+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.385663+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.358935+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.156066+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.752973+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:35.682846+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"7.2c","mappings":[{"from":0,"to":4}]},{"pgid":"7.37","mappings":[{"from":0,"to":5}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T15:46:06.846 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:06.841+0000 7f7ef96a1640 1 -- 172.21.15.123:0/486351278 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f7ed40730a0 msgr2=0x7f7ed4075560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:46:06.846 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:06.841+0000 7f7ef96a1640 1 --2- 172.21.15.123:0/486351278 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f7ed40730a0 0x7f7ed4075560 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f7eec0af000 tx=0x7f7eec01b060 comp rx=0 tx=0).stop 2024-09-16T15:46:06.846 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:06.841+0000 7f7ef96a1640 1 -- 172.21.15.123:0/486351278 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7ef415f420 msgr2=0x7f7ef4098840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:46:06.846 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:06.841+0000 7f7ef96a1640 1 --2- 172.21.15.123:0/486351278 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7ef415f420 0x7f7ef4098840 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7f7ee400cbf0 tx=0x7f7ee400cce0 comp rx=0 tx=0).stop 2024-09-16T15:46:06.846 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:06.841+0000 7f7ef96a1640 1 -- 172.21.15.123:0/486351278 shutdown_connections 2024-09-16T15:46:06.846 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:06.841+0000 7f7ef96a1640 1 --2- 172.21.15.123:0/486351278 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f7ed40730a0 0x7f7ed4075560 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:06.846 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:06.841+0000 7f7ef96a1640 1 --2- 172.21.15.123:0/486351278 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7ef415f420 0x7f7ef4098840 unknown :-1 s=CLOSED pgs=562 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:06.846 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:06.841+0000 7f7ef96a1640 1 --2- 172.21.15.123:0/486351278 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7ef40b2910 0x7f7ef41546f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:06.846 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:06.841+0000 7f7ef96a1640 1 --2- 172.21.15.123:0/486351278 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f7ef4074700 0x7f7ef4099640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:06.846 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:06.841+0000 7f7ef96a1640 1 -- 172.21.15.123:0/486351278 >> 172.21.15.123:0/486351278 conn(0x7f7ef40b1df0 msgr2=0x7f7ef406ee70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:46:06.846 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:06.841+0000 7f7ef96a1640 1 -- 172.21.15.123:0/486351278 shutdown_connections 2024-09-16T15:46:06.847 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:06.841+0000 7f7ef96a1640 1 -- 172.21.15.123:0/486351278 wait complete. 2024-09-16T15:46:06.862 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-09-16T15:46:06.862 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-16T15:46:06.983 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:06.981+0000 7f7dacf11640 1 -- 172.21.15.123:0/3926007819 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7da80b1a60 msgr2=0x7f7da8168d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:46:06.983 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:06.981+0000 7f7dacf11640 1 --2- 172.21.15.123:0/3926007819 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7da80b1a60 0x7f7da8168d50 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7f7da0009bc0 tx=0x7f7da001ca50 comp rx=0 tx=0).stop 2024-09-16T15:46:06.983 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:06.981+0000 7f7dacf11640 1 -- 172.21.15.123:0/3926007819 shutdown_connections 2024-09-16T15:46:06.983 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:06.981+0000 7f7dacf11640 1 --2- 172.21.15.123:0/3926007819 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7da80b1a60 0x7f7da8168d50 unknown :-1 s=CLOSED pgs=564 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:06.983 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:06.981+0000 7f7dacf11640 1 -- 172.21.15.123:0/3926007819 >> 172.21.15.123:0/3926007819 conn(0x7f7da80d3ea0 msgr2=0x7f7da80d42b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:46:06.983 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:06.981+0000 7f7dacf11640 1 -- 172.21.15.123:0/3926007819 shutdown_connections 2024-09-16T15:46:06.984 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:06.981+0000 7f7dacf11640 1 -- 172.21.15.123:0/3926007819 wait complete. 2024-09-16T15:46:06.984 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:06.981+0000 7f7dacf11640 1 Processor -- start 2024-09-16T15:46:06.984 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:06.981+0000 7f7dacf11640 1 -- start start 2024-09-16T15:46:06.986 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:06.981+0000 7f7dacf11640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7da80b1a60 0x7f7da80b3480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:46:06.986 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:06.981+0000 7f7dacf11640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f7da80b2b00 0x7f7da814f470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:46:06.986 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:06.981+0000 7f7dacf11640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7da8161620 0x7f7da80b4b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:46:06.986 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:06.981+0000 7f7dacf11640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f7da816bf50 con 0x7f7da80b1a60 2024-09-16T15:46:06.987 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:06.981+0000 7f7dacf11640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f7da816bdd0 con 0x7f7da8161620 2024-09-16T15:46:06.987 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:06.981+0000 7f7dacf11640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f7da816c0d0 con 0x7f7da80b2b00 2024-09-16T15:46:06.987 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:06.981+0000 7f7da659c640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7da80b1a60 0x7f7da80b3480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:46:06.987 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:06.981+0000 7f7da659c640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7da80b1a60 0x7f7da80b3480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:52124/0 (socket says 172.21.15.123:52124) 2024-09-16T15:46:06.987 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:06.981+0000 7f7da659c640 1 -- 172.21.15.123:0/3659432719 learned_addr learned my addr 172.21.15.123:0/3659432719 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:46:06.987 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:06.981+0000 7f7da6d9d640 1 --2- 172.21.15.123:0/3659432719 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7da8161620 0x7f7da80b4b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:46:06.987 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:06.981+0000 7f7da5d9b640 1 --2- 172.21.15.123:0/3659432719 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f7da80b2b00 0x7f7da814f470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:46:06.987 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:06.981+0000 7f7da659c640 1 -- 172.21.15.123:0/3659432719 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f7da80b2b00 msgr2=0x7f7da814f470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:46:06.987 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:06.981+0000 7f7da659c640 1 --2- 172.21.15.123:0/3659432719 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f7da80b2b00 0x7f7da814f470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:06.987 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:06.981+0000 7f7da659c640 1 -- 172.21.15.123:0/3659432719 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7da8161620 msgr2=0x7f7da80b4b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:46:06.987 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:06.981+0000 7f7da659c640 1 --2- 172.21.15.123:0/3659432719 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7da8161620 0x7f7da80b4b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:06.987 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:06.981+0000 7f7da659c640 1 -- 172.21.15.123:0/3659432719 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7da80b3a60 con 0x7f7da80b1a60 2024-09-16T15:46:06.988 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:06.981+0000 7f7da5d9b640 1 --2- 172.21.15.123:0/3659432719 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f7da80b2b00 0x7f7da814f470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T15:46:06.988 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:06.981+0000 7f7da6d9d640 1 --2- 172.21.15.123:0/3659432719 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7da8161620 0x7f7da80b4b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T15:46:06.988 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:06.981+0000 7f7da659c640 1 --2- 172.21.15.123:0/3659432719 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7da80b1a60 0x7f7da80b3480 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7f7d9800c490 tx=0x7f7d9800c950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:46:06.988 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:06.981+0000 7f7d9effd640 1 -- 172.21.15.123:0/3659432719 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d98017070 con 0x7f7da80b1a60 2024-09-16T15:46:06.988 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:06.981+0000 7f7d9effd640 1 -- 172.21.15.123:0/3659432719 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f7d98003600 con 0x7f7da80b1a60 2024-09-16T15:46:06.988 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:06.981+0000 7f7dacf11640 1 -- 172.21.15.123:0/3659432719 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7da80b3d40 con 0x7f7da80b1a60 2024-09-16T15:46:06.988 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:06.981+0000 7f7d9effd640 1 -- 172.21.15.123:0/3659432719 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d98003a90 con 0x7f7da80b1a60 2024-09-16T15:46:06.988 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:06.981+0000 7f7dacf11640 1 -- 172.21.15.123:0/3659432719 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7da80b4300 con 0x7f7da80b1a60 2024-09-16T15:46:06.989 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:06.985+0000 7f7dacf11640 1 -- 172.21.15.123:0/3659432719 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7da80b4740 con 0x7f7da80b1a60 2024-09-16T15:46:06.989 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:06.985+0000 7f7d9effd640 1 -- 172.21.15.123:0/3659432719 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 9) ==== 97233+0+0 (secure 0 0 0) 0x7f7d9801a070 con 0x7f7da80b1a60 2024-09-16T15:46:06.989 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:06.985+0000 7f7d9effd640 1 --2- 172.21.15.123:0/3659432719 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f7d88073070 0x7f7d88075530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:46:06.989 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:06.985+0000 7f7d9effd640 1 -- 172.21.15.123:0/3659432719 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(46..46 src has 1..46) ==== 5767+0+0 (secure 0 0 0) 0x7f7d98093570 con 0x7f7da80b1a60 2024-09-16T15:46:06.989 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:06.985+0000 7f7da5d9b640 1 --2- 172.21.15.123:0/3659432719 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f7d88073070 0x7f7d88075530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:46:06.989 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:06.985+0000 7f7da5d9b640 1 --2- 172.21.15.123:0/3659432719 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f7d88073070 0x7f7d88075530 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f7da8151840 tx=0x7f7d90009040 comp rx=0 tx=0).ready entity=mgr.4135 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:46:06.992 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:06.989+0000 7f7d9effd640 1 -- 172.21.15.123:0/3659432719 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f7d9805e3f0 con 0x7f7da80b1a60 2024-09-16T15:46:07.183 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:07.181+0000 7f7dacf11640 1 -- 172.21.15.123:0/3659432719 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f7da80d2560 con 0x7f7da80b1a60 2024-09-16T15:46:07.183 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:07.181+0000 7f7d9effd640 1 -- 172.21.15.123:0/3659432719 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) ==== 94+0+5223 (secure 0 0 0) 0x7f7d9805dcf0 con 0x7f7da80b1a60 2024-09-16T15:46:07.184 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-09-16T15:46:07.184 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":28,"btime":"2024-09-16T15:44:32:363512+0000","default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4712,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6837/288264117","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6836","nonce":288264117},{"type":"v1","addr":"172.21.15.184:6837","nonce":288264117}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":23},{"gid":5268,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6834/60194515","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6832","nonce":60194515},{"type":"v1","addr":"172.21.15.123:6834","nonce":60194515}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":23},{"gid":5271,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6835/1798373479","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6833","nonce":1798373479},{"type":"v1","addr":"172.21.15.123:6835","nonce":1798373479}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":22}],"filesystems":[{"mdsmap":{"epoch":27,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-16T15:44:14.853230+0000","modified":"2024-09-16T15:44:17.861742+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5280},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5280":{"gid":5280,"name":"d","rank":0,"incarnation":24,"state":"up:active","state_seq":3,"addr":"172.21.15.184:6835/374389412","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":374389412},{"type":"v1","addr":"172.21.15.184:6835","nonce":374389412}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5280,"qdb_cluster":[5280]},"id":3}]} 2024-09-16T15:46:07.184 INFO:teuthology.orchestra.run.smithi123.stderr:dumped fsmap epoch 28 2024-09-16T15:46:07.186 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:07.181+0000 7f7dacf11640 1 -- 172.21.15.123:0/3659432719 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f7d88073070 msgr2=0x7f7d88075530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:46:07.186 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:07.181+0000 7f7dacf11640 1 --2- 172.21.15.123:0/3659432719 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f7d88073070 0x7f7d88075530 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f7da8151840 tx=0x7f7d90009040 comp rx=0 tx=0).stop 2024-09-16T15:46:07.186 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:07.181+0000 7f7dacf11640 1 -- 172.21.15.123:0/3659432719 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7da80b1a60 msgr2=0x7f7da80b3480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:46:07.186 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:07.181+0000 7f7dacf11640 1 --2- 172.21.15.123:0/3659432719 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7da80b1a60 0x7f7da80b3480 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7f7d9800c490 tx=0x7f7d9800c950 comp rx=0 tx=0).stop 2024-09-16T15:46:07.186 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:07.181+0000 7f7dacf11640 1 -- 172.21.15.123:0/3659432719 shutdown_connections 2024-09-16T15:46:07.186 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:07.181+0000 7f7dacf11640 1 --2- 172.21.15.123:0/3659432719 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f7d88073070 0x7f7d88075530 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:07.186 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:07.181+0000 7f7dacf11640 1 --2- 172.21.15.123:0/3659432719 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7da8161620 0x7f7da80b4b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:07.186 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:07.181+0000 7f7dacf11640 1 --2- 172.21.15.123:0/3659432719 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f7da80b2b00 0x7f7da814f470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:07.186 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:07.181+0000 7f7dacf11640 1 --2- 172.21.15.123:0/3659432719 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7da80b1a60 0x7f7da80b3480 unknown :-1 s=CLOSED pgs=565 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:07.187 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:07.181+0000 7f7dacf11640 1 -- 172.21.15.123:0/3659432719 >> 172.21.15.123:0/3659432719 conn(0x7f7da80d3ea0 msgr2=0x7f7da806db00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:46:07.187 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:07.181+0000 7f7dacf11640 1 -- 172.21.15.123:0/3659432719 shutdown_connections 2024-09-16T15:46:07.187 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:07.181+0000 7f7dacf11640 1 -- 172.21.15.123:0/3659432719 wait complete. 2024-09-16T15:46:07.201 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-16T15:46:07.312 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:07.309+0000 7f49abb8a640 1 -- 172.21.15.123:0/1581645384 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f49a40b0a50 msgr2=0x7f49a40b0ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:46:07.312 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:07.309+0000 7f49abb8a640 1 --2- 172.21.15.123:0/1581645384 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f49a40b0a50 0x7f49a40b0ea0 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7f49a406f930 tx=0x7f499c01cc30 comp rx=0 tx=0).stop 2024-09-16T15:46:07.312 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:07.309+0000 7f49abb8a640 1 -- 172.21.15.123:0/1581645384 shutdown_connections 2024-09-16T15:46:07.312 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:07.309+0000 7f49abb8a640 1 --2- 172.21.15.123:0/1581645384 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f49a40b0a50 0x7f49a40b0ea0 unknown :-1 s=CLOSED pgs=566 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:07.312 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:07.309+0000 7f49abb8a640 1 --2- 172.21.15.123:0/1581645384 >> v2:172.21.15.184:3300/0 conn(0x7f49a4161620 0x7f49a40d1da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:07.312 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:07.309+0000 7f49abb8a640 1 -- 172.21.15.123:0/1581645384 >> 172.21.15.123:0/1581645384 conn(0x7f49a40d3ea0 msgr2=0x7f49a40d42b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:46:07.312 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:07.309+0000 7f49abb8a640 1 -- 172.21.15.123:0/1581645384 shutdown_connections 2024-09-16T15:46:07.313 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:07.309+0000 7f49abb8a640 1 -- 172.21.15.123:0/1581645384 wait complete. 2024-09-16T15:46:07.313 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:07.309+0000 7f49abb8a640 1 Processor -- start 2024-09-16T15:46:07.313 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:07.309+0000 7f49abb8a640 1 -- start start 2024-09-16T15:46:07.315 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:07.309+0000 7f49abb8a640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f49a40b0a50 0x7f49a41532a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:46:07.315 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:07.309+0000 7f49abb8a640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f49a40b2b00 0x7f49a415ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:46:07.315 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:07.309+0000 7f49abb8a640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f49a4161620 0x7f49a415f380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:46:07.315 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:07.309+0000 7f49abb8a640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f49a416b0f0 con 0x7f49a4161620 2024-09-16T15:46:07.316 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:07.309+0000 7f49abb8a640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f49a416af70 con 0x7f49a40b2b00 2024-09-16T15:46:07.316 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:07.309+0000 7f49abb8a640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f49a416b270 con 0x7f49a40b0a50 2024-09-16T15:46:07.316 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:07.309+0000 7f49aa127640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f49a4161620 0x7f49a415f380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:46:07.316 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:07.309+0000 7f49aa127640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f49a4161620 0x7f49a415f380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:52136/0 (socket says 172.21.15.123:52136) 2024-09-16T15:46:07.316 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:07.309+0000 7f49aa127640 1 -- 172.21.15.123:0/997283704 learned_addr learned my addr 172.21.15.123:0/997283704 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:46:07.316 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:07.309+0000 7f49a9125640 1 --2- 172.21.15.123:0/997283704 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f49a40b2b00 0x7f49a415ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:46:07.316 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:07.309+0000 7f49a9926640 1 --2- 172.21.15.123:0/997283704 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f49a40b0a50 0x7f49a41532a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:46:07.316 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:07.309+0000 7f49aa127640 1 -- 172.21.15.123:0/997283704 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f49a40b0a50 msgr2=0x7f49a41532a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:46:07.316 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:07.309+0000 7f49aa127640 1 --2- 172.21.15.123:0/997283704 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f49a40b0a50 0x7f49a41532a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:07.316 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:07.309+0000 7f49aa127640 1 -- 172.21.15.123:0/997283704 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f49a40b2b00 msgr2=0x7f49a415ee40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:46:07.316 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:07.309+0000 7f49aa127640 1 --2- 172.21.15.123:0/997283704 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f49a40b2b00 0x7f49a415ee40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:07.316 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:07.309+0000 7f49aa127640 1 -- 172.21.15.123:0/997283704 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f49a41524e0 con 0x7f49a4161620 2024-09-16T15:46:07.316 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:07.309+0000 7f49a9926640 1 --2- 172.21.15.123:0/997283704 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f49a40b0a50 0x7f49a41532a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T15:46:07.316 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:07.309+0000 7f49a9125640 1 --2- 172.21.15.123:0/997283704 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f49a40b2b00 0x7f49a415ee40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T15:46:07.317 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:07.309+0000 7f49aa127640 1 --2- 172.21.15.123:0/997283704 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f49a4161620 0x7f49a415f380 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7f499c009bd0 tx=0x7f499c0a5bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:46:07.317 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:07.309+0000 7f499a7fc640 1 -- 172.21.15.123:0/997283704 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f499c0bb070 con 0x7f49a4161620 2024-09-16T15:46:07.317 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:07.309+0000 7f499a7fc640 1 -- 172.21.15.123:0/997283704 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f499c004d40 con 0x7f49a4161620 2024-09-16T15:46:07.317 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:07.309+0000 7f49abb8a640 1 -- 172.21.15.123:0/997283704 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f49a41527c0 con 0x7f49a4161620 2024-09-16T15:46:07.317 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:07.309+0000 7f499a7fc640 1 -- 172.21.15.123:0/997283704 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f499c0b2040 con 0x7f49a4161620 2024-09-16T15:46:07.317 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:07.313+0000 7f49abb8a640 1 -- 172.21.15.123:0/997283704 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f49a415c8c0 con 0x7f49a4161620 2024-09-16T15:46:07.317 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:07.313+0000 7f49abb8a640 1 -- 172.21.15.123:0/997283704 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f49a40b0ea0 con 0x7f49a4161620 2024-09-16T15:46:07.318 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:07.313+0000 7f499a7fc640 1 -- 172.21.15.123:0/997283704 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 9) ==== 97233+0+0 (secure 0 0 0) 0x7f499c004320 con 0x7f49a4161620 2024-09-16T15:46:07.318 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:07.313+0000 7f499a7fc640 1 --2- 172.21.15.123:0/997283704 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f4988073070 0x7f4988075530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:46:07.318 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:07.313+0000 7f499a7fc640 1 -- 172.21.15.123:0/997283704 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(46..46 src has 1..46) ==== 5767+0+0 (secure 0 0 0) 0x7f499c12e180 con 0x7f49a4161620 2024-09-16T15:46:07.318 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:07.313+0000 7f49a9926640 1 --2- 172.21.15.123:0/997283704 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f4988073070 0x7f4988075530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:46:07.318 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:07.313+0000 7f49a9926640 1 --2- 172.21.15.123:0/997283704 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f4988073070 0x7f4988075530 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f49a0005870 tx=0x7f49a0005c90 comp rx=0 tx=0).ready entity=mgr.4135 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:46:07.321 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:07.317+0000 7f499a7fc640 1 -- 172.21.15.123:0/997283704 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f499c016970 con 0x7f49a4161620 2024-09-16T15:46:07.515 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:07.513+0000 7f49abb8a640 1 -- 172.21.15.123:0/997283704 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f49a41598e0 con 0x7f49a4161620 2024-09-16T15:46:07.516 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:07.513+0000 7f499a7fc640 1 -- 172.21.15.123:0/997283704 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) ==== 94+0+5223 (secure 0 0 0) 0x7f499c0f9090 con 0x7f49a4161620 2024-09-16T15:46:07.516 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-09-16T15:46:07.516 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":28,"btime":"2024-09-16T15:44:32:363512+0000","default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4712,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6837/288264117","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6836","nonce":288264117},{"type":"v1","addr":"172.21.15.184:6837","nonce":288264117}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":23},{"gid":5268,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6834/60194515","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6832","nonce":60194515},{"type":"v1","addr":"172.21.15.123:6834","nonce":60194515}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":23},{"gid":5271,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6835/1798373479","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6833","nonce":1798373479},{"type":"v1","addr":"172.21.15.123:6835","nonce":1798373479}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":22}],"filesystems":[{"mdsmap":{"epoch":27,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-16T15:44:14.853230+0000","modified":"2024-09-16T15:44:17.861742+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5280},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5280":{"gid":5280,"name":"d","rank":0,"incarnation":24,"state":"up:active","state_seq":3,"addr":"172.21.15.184:6835/374389412","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":374389412},{"type":"v1","addr":"172.21.15.184:6835","nonce":374389412}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5280,"qdb_cluster":[5280]},"id":3}]} 2024-09-16T15:46:07.516 INFO:teuthology.orchestra.run.smithi123.stderr:dumped fsmap epoch 28 2024-09-16T15:46:07.519 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:07.513+0000 7f49abb8a640 1 -- 172.21.15.123:0/997283704 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f4988073070 msgr2=0x7f4988075530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:46:07.519 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:07.513+0000 7f49abb8a640 1 --2- 172.21.15.123:0/997283704 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f4988073070 0x7f4988075530 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f49a0005870 tx=0x7f49a0005c90 comp rx=0 tx=0).stop 2024-09-16T15:46:07.519 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:07.513+0000 7f49abb8a640 1 -- 172.21.15.123:0/997283704 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f49a4161620 msgr2=0x7f49a415f380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:46:07.519 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:07.513+0000 7f49abb8a640 1 --2- 172.21.15.123:0/997283704 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f49a4161620 0x7f49a415f380 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7f499c009bd0 tx=0x7f499c0a5bb0 comp rx=0 tx=0).stop 2024-09-16T15:46:07.519 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:07.513+0000 7f49abb8a640 1 -- 172.21.15.123:0/997283704 shutdown_connections 2024-09-16T15:46:07.519 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:07.513+0000 7f49abb8a640 1 --2- 172.21.15.123:0/997283704 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f4988073070 0x7f4988075530 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:07.519 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:07.513+0000 7f49abb8a640 1 --2- 172.21.15.123:0/997283704 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f49a4161620 0x7f49a415f380 unknown :-1 s=CLOSED pgs=567 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:07.519 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:07.513+0000 7f49abb8a640 1 --2- 172.21.15.123:0/997283704 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f49a40b2b00 0x7f49a415ee40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:07.519 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:07.513+0000 7f49abb8a640 1 --2- 172.21.15.123:0/997283704 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f49a40b0a50 0x7f49a41532a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:07.519 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:07.513+0000 7f49abb8a640 1 -- 172.21.15.123:0/997283704 >> 172.21.15.123:0/997283704 conn(0x7f49a40d3ea0 msgr2=0x7f49a415a0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:46:07.519 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:07.513+0000 7f49abb8a640 1 -- 172.21.15.123:0/997283704 shutdown_connections 2024-09-16T15:46:07.519 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:07.513+0000 7f49abb8a640 1 -- 172.21.15.123:0/997283704 wait complete. 2024-09-16T15:46:07.535 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-16T15:46:07.653 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:07.649+0000 7f95be23d640 1 -- 172.21.15.123:0/773907564 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f95a0004810 msgr2=0x7f95a0004c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:46:07.653 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:07.649+0000 7f95be23d640 1 --2- 172.21.15.123:0/773907564 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f95a0004810 0x7f95a0004c90 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7f95ac009b60 tx=0x7f95ac01d220 comp rx=0 tx=0).stop 2024-09-16T15:46:07.653 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:07.649+0000 7f95be23d640 1 -- 172.21.15.123:0/773907564 shutdown_connections 2024-09-16T15:46:07.653 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:07.649+0000 7f95be23d640 1 --2- 172.21.15.123:0/773907564 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f95a0004810 0x7f95a0004c90 unknown :-1 s=CLOSED pgs=569 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:07.654 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:07.649+0000 7f95be23d640 1 -- 172.21.15.123:0/773907564 >> 172.21.15.123:0/773907564 conn(0x7f95b80cfb10 msgr2=0x7f95b80cff20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:46:07.654 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:07.649+0000 7f95be23d640 1 -- 172.21.15.123:0/773907564 shutdown_connections 2024-09-16T15:46:07.654 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:07.649+0000 7f95be23d640 1 -- 172.21.15.123:0/773907564 wait complete. 2024-09-16T15:46:07.654 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:07.649+0000 7f95be23d640 1 Processor -- start 2024-09-16T15:46:07.654 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:07.649+0000 7f95be23d640 1 -- start start 2024-09-16T15:46:07.656 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:07.653+0000 7f95be23d640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f95a0004810 0x7f95b80b4d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:46:07.656 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:07.653+0000 7f95be23d640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f95b8166a00 0x7f95b81f8b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:46:07.656 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:07.653+0000 7f95be23d640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f95b81675b0 0x7f95b81fb780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:46:07.656 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:07.653+0000 7f95be23d640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f95b815ec30 con 0x7f95a0004810 2024-09-16T15:46:07.656 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:07.653+0000 7f95be23d640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f95b80cf640 con 0x7f95b8166a00 2024-09-16T15:46:07.656 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:07.653+0000 7f95be23d640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f95b815edb0 con 0x7f95b81675b0 2024-09-16T15:46:07.656 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:07.653+0000 7f95b77fe640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f95a0004810 0x7f95b80b4d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:46:07.656 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:07.653+0000 7f95b77fe640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f95a0004810 0x7f95b80b4d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:52156/0 (socket says 172.21.15.123:52156) 2024-09-16T15:46:07.657 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:07.653+0000 7f95b77fe640 1 -- 172.21.15.123:0/3199295330 learned_addr learned my addr 172.21.15.123:0/3199295330 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:46:07.657 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:07.653+0000 7f95b77fe640 1 -- 172.21.15.123:0/3199295330 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f95b81675b0 msgr2=0x7f95b81fb780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:46:07.657 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:07.653+0000 7f95b77fe640 1 --2- 172.21.15.123:0/3199295330 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f95b81675b0 0x7f95b81fb780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:07.657 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:07.653+0000 7f95b77fe640 1 -- 172.21.15.123:0/3199295330 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f95b8166a00 msgr2=0x7f95b81f8b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:46:07.657 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:07.653+0000 7f95b77fe640 1 --2- 172.21.15.123:0/3199295330 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f95b8166a00 0x7f95b81f8b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:07.657 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:07.653+0000 7f95b77fe640 1 -- 172.21.15.123:0/3199295330 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f95b80b3cc0 con 0x7f95a0004810 2024-09-16T15:46:07.657 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:07.653+0000 7f95b77fe640 1 --2- 172.21.15.123:0/3199295330 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f95a0004810 0x7f95b80b4d00 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7f95a8003e50 tx=0x7f95a8001640 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:46:07.657 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:07.653+0000 7f959ffff640 1 -- 172.21.15.123:0/3199295330 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95a8015070 con 0x7f95a0004810 2024-09-16T15:46:07.657 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:07.653+0000 7f959ffff640 1 -- 172.21.15.123:0/3199295330 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f95a8011840 con 0x7f95a0004810 2024-09-16T15:46:07.657 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:07.653+0000 7f95be23d640 1 -- 172.21.15.123:0/3199295330 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f95b80b3f60 con 0x7f95a0004810 2024-09-16T15:46:07.657 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:07.653+0000 7f959ffff640 1 -- 172.21.15.123:0/3199295330 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95a8001d50 con 0x7f95a0004810 2024-09-16T15:46:07.657 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:07.653+0000 7f95be23d640 1 -- 172.21.15.123:0/3199295330 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f95b80b4800 con 0x7f95a0004810 2024-09-16T15:46:07.658 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:07.653+0000 7f95be23d640 1 -- 172.21.15.123:0/3199295330 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f95b815f310 con 0x7f95a0004810 2024-09-16T15:46:07.658 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:07.653+0000 7f959ffff640 1 -- 172.21.15.123:0/3199295330 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 9) ==== 97233+0+0 (secure 0 0 0) 0x7f95a8018070 con 0x7f95a0004810 2024-09-16T15:46:07.658 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:07.653+0000 7f959ffff640 1 --2- 172.21.15.123:0/3199295330 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f95a40730a0 0x7f95a4075560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:46:07.658 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:07.653+0000 7f959ffff640 1 -- 172.21.15.123:0/3199295330 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(46..46 src has 1..46) ==== 5767+0+0 (secure 0 0 0) 0x7f95a8094d10 con 0x7f95a0004810 2024-09-16T15:46:07.658 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:07.653+0000 7f95b6ffd640 1 --2- 172.21.15.123:0/3199295330 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f95a40730a0 0x7f95a4075560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:46:07.659 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:07.657+0000 7f95b6ffd640 1 --2- 172.21.15.123:0/3199295330 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f95a40730a0 0x7f95a4075560 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f95ac0ae000 tx=0x7f95a0005840 comp rx=0 tx=0).ready entity=mgr.4135 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:46:07.662 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:07.657+0000 7f959ffff640 1 -- 172.21.15.123:0/3199295330 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f95a805fb90 con 0x7f95a0004810 2024-09-16T15:46:07.834 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:07.829+0000 7f95be23d640 1 -- 172.21.15.123:0/3199295330 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f95b80b4130 con 0x7f95a0004810 2024-09-16T15:46:07.837 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:07.833+0000 7f959ffff640 1 -- 172.21.15.123:0/3199295330 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v46) ==== 74+0+16885 (secure 0 0 0) 0x7f95a805f490 con 0x7f95a0004810 2024-09-16T15:46:07.837 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-09-16T15:46:07.837 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":46,"fsid":"56077f89-18c8-486f-ae7e-eecc0bb7822c","created":"2024-09-16T15:42:26.364892+0000","modified":"2024-09-16T15:44:32.363194+0000","last_up_change":"2024-09-16T15:42:38.707234+0000","last_in_change":"2024-09-16T15:42:29.423662+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-16T15:42:39.287764+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_metadata","create_time":"2024-09-16T15:44:12.463367+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"43","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_data","create_time":"2024-09-16T15:44:13.817170+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"44","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9e7c1955-a607-4671-9be7-f2ef760ceeaa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6800","nonce":3882044969},{"type":"v1","addr":"172.21.15.123:6801","nonce":3882044969}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6802","nonce":3882044969},{"type":"v1","addr":"172.21.15.123:6803","nonce":3882044969}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6806","nonce":3882044969},{"type":"v1","addr":"172.21.15.123:6807","nonce":3882044969}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6804","nonce":3882044969},{"type":"v1","addr":"172.21.15.123:6805","nonce":3882044969}]},"public_addr":"172.21.15.123:6801/3882044969","cluster_addr":"172.21.15.123:6803/3882044969","heartbeat_back_addr":"172.21.15.123:6807/3882044969","heartbeat_front_addr":"172.21.15.123:6805/3882044969","state":["exists","up"]},{"osd":1,"uuid":"c540cfde-0977-4742-9b17-fd634668e010","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6816","nonce":4037001850},{"type":"v1","addr":"172.21.15.123:6817","nonce":4037001850}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6818","nonce":4037001850},{"type":"v1","addr":"172.21.15.123:6819","nonce":4037001850}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6822","nonce":4037001850},{"type":"v1","addr":"172.21.15.123:6823","nonce":4037001850}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6820","nonce":4037001850},{"type":"v1","addr":"172.21.15.123:6821","nonce":4037001850}]},"public_addr":"172.21.15.123:6817/4037001850","cluster_addr":"172.21.15.123:6819/4037001850","heartbeat_back_addr":"172.21.15.123:6823/4037001850","heartbeat_front_addr":"172.21.15.123:6821/4037001850","state":["exists","up"]},{"osd":2,"uuid":"9864a496-f169-486b-95c4-1d59162793a7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6824","nonce":1332143088},{"type":"v1","addr":"172.21.15.123:6825","nonce":1332143088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6826","nonce":1332143088},{"type":"v1","addr":"172.21.15.123:6827","nonce":1332143088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6830","nonce":1332143088},{"type":"v1","addr":"172.21.15.123:6831","nonce":1332143088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6828","nonce":1332143088},{"type":"v1","addr":"172.21.15.123:6829","nonce":1332143088}]},"public_addr":"172.21.15.123:6825/1332143088","cluster_addr":"172.21.15.123:6827/1332143088","heartbeat_back_addr":"172.21.15.123:6831/1332143088","heartbeat_front_addr":"172.21.15.123:6829/1332143088","state":["exists","up"]},{"osd":3,"uuid":"c6d1f817-f074-4325-95e1-578eba13edda","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6808","nonce":988586998},{"type":"v1","addr":"172.21.15.123:6809","nonce":988586998}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6810","nonce":988586998},{"type":"v1","addr":"172.21.15.123:6811","nonce":988586998}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6814","nonce":988586998},{"type":"v1","addr":"172.21.15.123:6815","nonce":988586998}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6812","nonce":988586998},{"type":"v1","addr":"172.21.15.123:6813","nonce":988586998}]},"public_addr":"172.21.15.123:6809/988586998","cluster_addr":"172.21.15.123:6811/988586998","heartbeat_back_addr":"172.21.15.123:6815/988586998","heartbeat_front_addr":"172.21.15.123:6813/988586998","state":["exists","up"]},{"osd":4,"uuid":"200a8df4-50ef-40a5-b48a-58981269c086","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6816","nonce":3387813890},{"type":"v1","addr":"172.21.15.184:6817","nonce":3387813890}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6818","nonce":3387813890},{"type":"v1","addr":"172.21.15.184:6819","nonce":3387813890}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6822","nonce":3387813890},{"type":"v1","addr":"172.21.15.184:6823","nonce":3387813890}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6820","nonce":3387813890},{"type":"v1","addr":"172.21.15.184:6821","nonce":3387813890}]},"public_addr":"172.21.15.184:6817/3387813890","cluster_addr":"172.21.15.184:6819/3387813890","heartbeat_back_addr":"172.21.15.184:6823/3387813890","heartbeat_front_addr":"172.21.15.184:6821/3387813890","state":["exists","up"]},{"osd":5,"uuid":"32d5a900-da3d-412c-bc19-13861a2368d8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6808","nonce":2680916017},{"type":"v1","addr":"172.21.15.184:6809","nonce":2680916017}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6810","nonce":2680916017},{"type":"v1","addr":"172.21.15.184:6811","nonce":2680916017}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6814","nonce":2680916017},{"type":"v1","addr":"172.21.15.184:6815","nonce":2680916017}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6812","nonce":2680916017},{"type":"v1","addr":"172.21.15.184:6813","nonce":2680916017}]},"public_addr":"172.21.15.184:6809/2680916017","cluster_addr":"172.21.15.184:6811/2680916017","heartbeat_back_addr":"172.21.15.184:6815/2680916017","heartbeat_front_addr":"172.21.15.184:6813/2680916017","state":["exists","up"]},{"osd":6,"uuid":"cde6f39e-aa1b-4260-ac19-774ee6aaca72","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6824","nonce":2183120457},{"type":"v1","addr":"172.21.15.184:6825","nonce":2183120457}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6826","nonce":2183120457},{"type":"v1","addr":"172.21.15.184:6827","nonce":2183120457}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6830","nonce":2183120457},{"type":"v1","addr":"172.21.15.184:6831","nonce":2183120457}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6828","nonce":2183120457},{"type":"v1","addr":"172.21.15.184:6829","nonce":2183120457}]},"public_addr":"172.21.15.184:6825/2183120457","cluster_addr":"172.21.15.184:6827/2183120457","heartbeat_back_addr":"172.21.15.184:6831/2183120457","heartbeat_front_addr":"172.21.15.184:6829/2183120457","state":["exists","up"]},{"osd":7,"uuid":"4f381778-eee1-47c6-804f-df0e6b33e313","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6800","nonce":4102957285},{"type":"v1","addr":"172.21.15.184:6801","nonce":4102957285}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6802","nonce":4102957285},{"type":"v1","addr":"172.21.15.184:6803","nonce":4102957285}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6806","nonce":4102957285},{"type":"v1","addr":"172.21.15.184:6807","nonce":4102957285}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6804","nonce":4102957285},{"type":"v1","addr":"172.21.15.184:6805","nonce":4102957285}]},"public_addr":"172.21.15.184:6801/4102957285","cluster_addr":"172.21.15.184:6803/4102957285","heartbeat_back_addr":"172.21.15.184:6807/4102957285","heartbeat_front_addr":"172.21.15.184:6805/4102957285","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:35.972210+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.222007+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.198617+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.385663+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.358935+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.156066+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.752973+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:35.682846+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"7.2c","mappings":[{"from":0,"to":4}]},{"pgid":"7.37","mappings":[{"from":0,"to":5}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T15:46:07.839 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:07.837+0000 7f95be23d640 1 -- 172.21.15.123:0/3199295330 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f95a40730a0 msgr2=0x7f95a4075560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:46:07.839 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:07.837+0000 7f95be23d640 1 --2- 172.21.15.123:0/3199295330 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f95a40730a0 0x7f95a4075560 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f95ac0ae000 tx=0x7f95a0005840 comp rx=0 tx=0).stop 2024-09-16T15:46:07.840 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:07.837+0000 7f95be23d640 1 -- 172.21.15.123:0/3199295330 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f95a0004810 msgr2=0x7f95b80b4d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:46:07.840 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:07.837+0000 7f95be23d640 1 --2- 172.21.15.123:0/3199295330 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f95a0004810 0x7f95b80b4d00 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7f95a8003e50 tx=0x7f95a8001640 comp rx=0 tx=0).stop 2024-09-16T15:46:07.840 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:07.837+0000 7f95be23d640 1 -- 172.21.15.123:0/3199295330 shutdown_connections 2024-09-16T15:46:07.840 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:07.837+0000 7f95be23d640 1 --2- 172.21.15.123:0/3199295330 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f95a40730a0 0x7f95a4075560 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:07.840 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:07.837+0000 7f95be23d640 1 --2- 172.21.15.123:0/3199295330 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f95b81675b0 0x7f95b81fb780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:07.840 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:07.837+0000 7f95be23d640 1 --2- 172.21.15.123:0/3199295330 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f95b8166a00 0x7f95b81f8b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:07.840 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:07.837+0000 7f95be23d640 1 --2- 172.21.15.123:0/3199295330 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f95a0004810 0x7f95b80b4d00 unknown :-1 s=CLOSED pgs=570 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:07.840 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:07.837+0000 7f95be23d640 1 -- 172.21.15.123:0/3199295330 >> 172.21.15.123:0/3199295330 conn(0x7f95b80cfb10 msgr2=0x7f95b8156880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:46:07.840 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:07.837+0000 7f95be23d640 1 -- 172.21.15.123:0/3199295330 shutdown_connections 2024-09-16T15:46:07.840 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:07.837+0000 7f95be23d640 1 -- 172.21.15.123:0/3199295330 wait complete. 2024-09-16T15:46:07.856 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-09-16T15:46:07.979 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:07.973+0000 7efef659d640 1 -- 172.21.15.123:0/1609669023 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7efed80052f0 msgr2=0x7efed800b9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:46:07.979 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:07.973+0000 7efef659d640 1 --2- 172.21.15.123:0/1609669023 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7efed80052f0 0x7efed800b9d0 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7efee8009ee0 tx=0x7efee801d630 comp rx=0 tx=0).stop 2024-09-16T15:46:07.979 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:07.973+0000 7efef659d640 1 -- 172.21.15.123:0/1609669023 shutdown_connections 2024-09-16T15:46:07.979 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:07.973+0000 7efef659d640 1 --2- 172.21.15.123:0/1609669023 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7efed80052f0 0x7efed800b9d0 unknown :-1 s=CLOSED pgs=572 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:07.979 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:07.973+0000 7efef659d640 1 -- 172.21.15.123:0/1609669023 >> 172.21.15.123:0/1609669023 conn(0x7efef00732a0 msgr2=0x7efef0074150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:46:07.979 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:07.973+0000 7efef659d640 1 -- 172.21.15.123:0/1609669023 shutdown_connections 2024-09-16T15:46:07.979 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:07.973+0000 7efef659d640 1 -- 172.21.15.123:0/1609669023 wait complete. 2024-09-16T15:46:07.980 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:07.977+0000 7efef659d640 1 Processor -- start 2024-09-16T15:46:07.980 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:07.977+0000 7efef659d640 1 -- start start 2024-09-16T15:46:07.982 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:07.977+0000 7efef659d640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7efed80052f0 0x7efef01f47c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:46:07.982 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:07.977+0000 7efef659d640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7efef00ca4c0 0x7efef01f50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:46:07.983 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:07.977+0000 7efef659d640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7efef0153f90 0x7efef0156230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:46:07.983 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:07.977+0000 7efef659d640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7efef0068fa0 con 0x7efef0153f90 2024-09-16T15:46:07.983 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:07.977+0000 7efef659d640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7efef0068e20 con 0x7efef00ca4c0 2024-09-16T15:46:07.983 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:07.977+0000 7efef659d640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7efef00ce2b0 con 0x7efed80052f0 2024-09-16T15:46:07.983 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:07.977+0000 7efef4b3a640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7efef0153f90 0x7efef0156230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:46:07.983 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:07.977+0000 7efef4b3a640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7efef0153f90 0x7efef0156230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:52168/0 (socket says 172.21.15.123:52168) 2024-09-16T15:46:07.983 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:07.977+0000 7efef4b3a640 1 -- 172.21.15.123:0/3091562145 learned_addr learned my addr 172.21.15.123:0/3091562145 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:46:07.983 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:07.977+0000 7efeef7fe640 1 --2- 172.21.15.123:0/3091562145 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7efef00ca4c0 0x7efef01f50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:46:07.983 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:07.977+0000 7efeeffff640 1 --2- 172.21.15.123:0/3091562145 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7efed80052f0 0x7efef01f47c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:46:07.983 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:07.977+0000 7efef4b3a640 1 -- 172.21.15.123:0/3091562145 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7efed80052f0 msgr2=0x7efef01f47c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:46:07.983 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:07.977+0000 7efef4b3a640 1 --2- 172.21.15.123:0/3091562145 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7efed80052f0 0x7efef01f47c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:07.984 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:07.977+0000 7efef4b3a640 1 -- 172.21.15.123:0/3091562145 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7efef00ca4c0 msgr2=0x7efef01f50d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:46:07.984 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:07.977+0000 7efef4b3a640 1 --2- 172.21.15.123:0/3091562145 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7efef00ca4c0 0x7efef01f50d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:07.984 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:07.977+0000 7efef4b3a640 1 -- 172.21.15.123:0/3091562145 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efef0154490 con 0x7efef0153f90 2024-09-16T15:46:07.984 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:07.977+0000 7efeef7fe640 1 --2- 172.21.15.123:0/3091562145 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7efef00ca4c0 0x7efef01f50d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T15:46:07.984 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:07.977+0000 7efef4b3a640 1 --2- 172.21.15.123:0/3091562145 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7efef0153f90 0x7efef0156230 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7efee801d460 tx=0x7efee8004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:46:07.984 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:07.977+0000 7efeecff9640 1 -- 172.21.15.123:0/3091562145 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efee8009ab0 con 0x7efef0153f90 2024-09-16T15:46:07.984 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:07.977+0000 7efeecff9640 1 -- 172.21.15.123:0/3091562145 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7efee80b1070 con 0x7efef0153f90 2024-09-16T15:46:07.984 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:07.977+0000 7efef659d640 1 -- 172.21.15.123:0/3091562145 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efef0154770 con 0x7efef0153f90 2024-09-16T15:46:07.984 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:07.977+0000 7efeecff9640 1 -- 172.21.15.123:0/3091562145 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efee8009200 con 0x7efef0153f90 2024-09-16T15:46:07.984 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:07.977+0000 7efef659d640 1 -- 172.21.15.123:0/3091562145 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efef0154d30 con 0x7efef0153f90 2024-09-16T15:46:07.984 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:07.977+0000 7efef659d640 1 -- 172.21.15.123:0/3091562145 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efef00cf800 con 0x7efef0153f90 2024-09-16T15:46:07.985 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:07.981+0000 7efeecff9640 1 -- 172.21.15.123:0/3091562145 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 9) ==== 97233+0+0 (secure 0 0 0) 0x7efee8004a50 con 0x7efef0153f90 2024-09-16T15:46:07.985 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:07.981+0000 7efeecff9640 1 --2- 172.21.15.123:0/3091562145 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7efedc0730a0 0x7efedc075560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:46:07.986 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:07.981+0000 7efeecff9640 1 -- 172.21.15.123:0/3091562145 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(46..46 src has 1..46) ==== 5767+0+0 (secure 0 0 0) 0x7efee812e130 con 0x7efef0153f90 2024-09-16T15:46:07.986 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:07.981+0000 7efeeffff640 1 --2- 172.21.15.123:0/3091562145 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7efedc0730a0 0x7efedc075560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:46:07.986 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:07.981+0000 7efeeffff640 1 --2- 172.21.15.123:0/3091562145 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7efedc0730a0 0x7efedc075560 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7efee0001680 tx=0x7efee0009040 comp rx=0 tx=0).ready entity=mgr.4135 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:46:07.988 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:07.985+0000 7efeecff9640 1 -- 172.21.15.123:0/3091562145 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7efee8017340 con 0x7efef0153f90 2024-09-16T15:46:08.187 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:08.185+0000 7efef659d640 1 -- 172.21.15.123:0/3091562145 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7efef00b28c0 con 0x7efef0153f90 2024-09-16T15:46:08.188 INFO:tasks.ceph.mon.a.smithi123.stderr:2024-09-16T15:46:08.185+0000 7f268d076640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-09-16T15:46:08.190 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:08.185+0000 7efeecff9640 1 -- 172.21.15.123:0/3091562145 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v29) ==== 161+0+0 (secure 0 0 0) 0x7efee80f8fb0 con 0x7efef0153f90 2024-09-16T15:46:08.190 INFO:teuthology.orchestra.run.smithi123.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-09-16T15:46:08.192 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:08.189+0000 7efef659d640 1 -- 172.21.15.123:0/3091562145 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7efedc0730a0 msgr2=0x7efedc075560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:46:08.193 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:08.189+0000 7efef659d640 1 --2- 172.21.15.123:0/3091562145 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7efedc0730a0 0x7efedc075560 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7efee0001680 tx=0x7efee0009040 comp rx=0 tx=0).stop 2024-09-16T15:46:08.193 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:08.189+0000 7efef659d640 1 -- 172.21.15.123:0/3091562145 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7efef0153f90 msgr2=0x7efef0156230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:46:08.193 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:08.189+0000 7efef659d640 1 --2- 172.21.15.123:0/3091562145 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7efef0153f90 0x7efef0156230 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7efee801d460 tx=0x7efee8004600 comp rx=0 tx=0).stop 2024-09-16T15:46:08.193 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:08.189+0000 7efef659d640 1 -- 172.21.15.123:0/3091562145 shutdown_connections 2024-09-16T15:46:08.193 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:08.189+0000 7efef659d640 1 --2- 172.21.15.123:0/3091562145 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7efedc0730a0 0x7efedc075560 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:08.193 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:08.189+0000 7efef659d640 1 --2- 172.21.15.123:0/3091562145 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7efef0153f90 0x7efef0156230 unknown :-1 s=CLOSED pgs=573 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:08.193 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:08.189+0000 7efef659d640 1 --2- 172.21.15.123:0/3091562145 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7efef00ca4c0 0x7efef01f50d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:08.193 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:08.189+0000 7efef659d640 1 --2- 172.21.15.123:0/3091562145 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7efed80052f0 0x7efef01f47c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:08.193 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:08.189+0000 7efef659d640 1 -- 172.21.15.123:0/3091562145 >> 172.21.15.123:0/3091562145 conn(0x7efef00732a0 msgr2=0x7efef00cbd70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:46:08.194 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:08.189+0000 7efef659d640 1 -- 172.21.15.123:0/3091562145 shutdown_connections 2024-09-16T15:46:08.194 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:08.189+0000 7efef659d640 1 -- 172.21.15.123:0/3091562145 wait complete. 2024-09-16T15:46:08.211 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-09-16T15:46:08.283 INFO:tasks.ceph.mds.d.smithi184.stdout:starting mds.d at 2024-09-16T15:46:08.323 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:08.317+0000 7fd4764dc640 1 -- 172.21.15.123:0/4036327088 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd47015a350 msgr2=0x7fd47006e3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:46:08.323 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:08.317+0000 7fd4764dc640 1 --2- 172.21.15.123:0/4036327088 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd47015a350 0x7fd47006e3b0 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7fd47006db30 tx=0x7fd46001cd80 comp rx=0 tx=0).stop 2024-09-16T15:46:08.323 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:08.317+0000 7fd4764dc640 1 -- 172.21.15.123:0/4036327088 shutdown_connections 2024-09-16T15:46:08.323 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:08.317+0000 7fd4764dc640 1 --2- 172.21.15.123:0/4036327088 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd47015a350 0x7fd47006e3b0 unknown :-1 s=CLOSED pgs=575 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:08.323 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:08.317+0000 7fd4764dc640 1 --2- 172.21.15.123:0/4036327088 >> v2:172.21.15.184:3300/0 conn(0x7fd470159f70 0x7fd470072f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:08.323 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:08.317+0000 7fd4764dc640 1 -- 172.21.15.123:0/4036327088 >> 172.21.15.123:0/4036327088 conn(0x7fd4700b1710 msgr2=0x7fd4700b1b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:46:08.323 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:08.317+0000 7fd4764dc640 1 -- 172.21.15.123:0/4036327088 shutdown_connections 2024-09-16T15:46:08.323 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:08.317+0000 7fd4764dc640 1 -- 172.21.15.123:0/4036327088 wait complete. 2024-09-16T15:46:08.325 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:08.321+0000 7fd4764dc640 1 Processor -- start 2024-09-16T15:46:08.325 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:08.321+0000 7fd4764dc640 1 -- start start 2024-09-16T15:46:08.326 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:08.321+0000 7fd4764dc640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd47006e8f0 0x7fd470150600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:46:08.326 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:08.321+0000 7fd4764dc640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fd470159f70 0x7fd4701541a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:46:08.326 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:08.321+0000 7fd4764dc640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd47015a350 0x7fd4701546e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:46:08.326 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:08.321+0000 7fd4764dc640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fd4700cc740 con 0x7fd47006e8f0 2024-09-16T15:46:08.326 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:08.321+0000 7fd4764dc640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fd4700cc5c0 con 0x7fd47015a350 2024-09-16T15:46:08.326 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:08.321+0000 7fd4764dc640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fd4700cc8c0 con 0x7fd470159f70 2024-09-16T15:46:08.326 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:08.321+0000 7fd46ffff640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd47006e8f0 0x7fd470150600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:46:08.326 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:08.321+0000 7fd46ffff640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd47006e8f0 0x7fd470150600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:52184/0 (socket says 172.21.15.123:52184) 2024-09-16T15:46:08.326 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:08.321+0000 7fd46ffff640 1 -- 172.21.15.123:0/947877488 learned_addr learned my addr 172.21.15.123:0/947877488 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:46:08.326 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:08.321+0000 7fd46f7fe640 1 --2- 172.21.15.123:0/947877488 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fd470159f70 0x7fd4701541a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:46:08.326 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:08.321+0000 7fd474a79640 1 --2- 172.21.15.123:0/947877488 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd47015a350 0x7fd4701546e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:46:08.326 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:08.321+0000 7fd46ffff640 1 -- 172.21.15.123:0/947877488 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fd470159f70 msgr2=0x7fd4701541a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:46:08.327 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:08.321+0000 7fd46ffff640 1 --2- 172.21.15.123:0/947877488 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fd470159f70 0x7fd4701541a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:08.327 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:08.321+0000 7fd46ffff640 1 -- 172.21.15.123:0/947877488 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd47015a350 msgr2=0x7fd4701546e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:46:08.327 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:08.321+0000 7fd46ffff640 1 --2- 172.21.15.123:0/947877488 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd47015a350 0x7fd4701546e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:08.327 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:08.321+0000 7fd46ffff640 1 -- 172.21.15.123:0/947877488 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd4700c8dc0 con 0x7fd47006e8f0 2024-09-16T15:46:08.327 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:08.321+0000 7fd46ffff640 1 --2- 172.21.15.123:0/947877488 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd47006e8f0 0x7fd470150600 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7fd46400c970 tx=0x7fd46400ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:46:08.327 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:08.321+0000 7fd46cff9640 1 -- 172.21.15.123:0/947877488 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd464007bb0 con 0x7fd47006e8f0 2024-09-16T15:46:08.327 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:08.321+0000 7fd46cff9640 1 -- 172.21.15.123:0/947877488 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fd464007e40 con 0x7fd47006e8f0 2024-09-16T15:46:08.327 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:08.321+0000 7fd4764dc640 1 -- 172.21.15.123:0/947877488 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd4700c8ae0 con 0x7fd47006e8f0 2024-09-16T15:46:08.327 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:08.321+0000 7fd46cff9640 1 -- 172.21.15.123:0/947877488 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd464005260 con 0x7fd47006e8f0 2024-09-16T15:46:08.327 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:08.321+0000 7fd4764dc640 1 -- 172.21.15.123:0/947877488 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd4701571d0 con 0x7fd47006e8f0 2024-09-16T15:46:08.328 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:08.321+0000 7fd4764dc640 1 -- 172.21.15.123:0/947877488 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd4701596b0 con 0x7fd47006e8f0 2024-09-16T15:46:08.328 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:08.325+0000 7fd46cff9640 1 -- 172.21.15.123:0/947877488 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 9) ==== 97233+0+0 (secure 0 0 0) 0x7fd464005400 con 0x7fd47006e8f0 2024-09-16T15:46:08.328 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:08.325+0000 7fd46cff9640 1 --2- 172.21.15.123:0/947877488 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fd45c073070 0x7fd45c075530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:46:08.328 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:08.325+0000 7fd46cff9640 1 -- 172.21.15.123:0/947877488 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(47..47 src has 1..47) ==== 5853+0+0 (secure 0 0 0) 0x7fd4640948f0 con 0x7fd47006e8f0 2024-09-16T15:46:08.328 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:08.325+0000 7fd46f7fe640 1 --2- 172.21.15.123:0/947877488 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fd45c073070 0x7fd45c075530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:46:08.329 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:08.325+0000 7fd46f7fe640 1 --2- 172.21.15.123:0/947877488 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fd45c073070 0x7fd45c075530 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7fd4600af000 tx=0x7fd46001b5c0 comp rx=0 tx=0).ready entity=mgr.4135 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:46:08.332 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:08.329+0000 7fd46cff9640 1 -- 172.21.15.123:0/947877488 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7fd464014030 con 0x7fd47006e8f0 2024-09-16T15:46:08.527 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:08.525+0000 7fd4764dc640 1 -- 172.21.15.123:0/947877488 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) -- 0x7fd470072ea0 con 0x7fd47006e8f0 2024-09-16T15:46:09.193 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:09.189+0000 7fd46cff9640 1 -- 172.21.15.123:0/947877488 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v30) ==== 104+0+0 (secure 0 0 0) 0x7fd4640940a0 con 0x7fd47006e8f0 2024-09-16T15:46:09.198 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:09.193+0000 7fd4764dc640 1 -- 172.21.15.123:0/947877488 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fd45c073070 msgr2=0x7fd45c075530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:46:09.198 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:09.193+0000 7fd4764dc640 1 --2- 172.21.15.123:0/947877488 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fd45c073070 0x7fd45c075530 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7fd4600af000 tx=0x7fd46001b5c0 comp rx=0 tx=0).stop 2024-09-16T15:46:09.198 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:09.193+0000 7fd4764dc640 1 -- 172.21.15.123:0/947877488 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd47006e8f0 msgr2=0x7fd470150600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:46:09.198 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:09.193+0000 7fd4764dc640 1 --2- 172.21.15.123:0/947877488 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd47006e8f0 0x7fd470150600 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7fd46400c970 tx=0x7fd46400ce30 comp rx=0 tx=0).stop 2024-09-16T15:46:09.199 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:09.193+0000 7fd4764dc640 1 -- 172.21.15.123:0/947877488 shutdown_connections 2024-09-16T15:46:09.199 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:09.193+0000 7fd4764dc640 1 --2- 172.21.15.123:0/947877488 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fd45c073070 0x7fd45c075530 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:09.199 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:09.193+0000 7fd4764dc640 1 --2- 172.21.15.123:0/947877488 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd47015a350 0x7fd4701546e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:09.199 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:09.193+0000 7fd4764dc640 1 --2- 172.21.15.123:0/947877488 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fd470159f70 0x7fd4701541a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:09.199 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:09.193+0000 7fd4764dc640 1 --2- 172.21.15.123:0/947877488 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd47006e8f0 0x7fd470150600 unknown :-1 s=CLOSED pgs=576 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:09.199 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:09.193+0000 7fd4764dc640 1 -- 172.21.15.123:0/947877488 >> 172.21.15.123:0/947877488 conn(0x7fd4700b1710 msgr2=0x7fd4700ceb40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:46:09.199 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:09.193+0000 7fd4764dc640 1 -- 172.21.15.123:0/947877488 shutdown_connections 2024-09-16T15:46:09.199 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:09.193+0000 7fd4764dc640 1 -- 172.21.15.123:0/947877488 wait complete. 2024-09-16T15:46:09.218 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-09-16T15:46:09.252 DEBUG:teuthology.orchestra.run.smithi123:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-16T15:46:09.268 DEBUG:teuthology.orchestra.run.smithi184:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-16T15:46:09.343 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:09.337+0000 7fad9d113640 1 -- 172.21.15.123:0/940628599 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fad980d2a10 msgr2=0x7fad980d0fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:46:09.343 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:09.337+0000 7fad9d113640 1 --2- 172.21.15.123:0/940628599 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fad980d2a10 0x7fad980d0fe0 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7fad90009c80 tx=0x7fad9001cd60 comp rx=0 tx=0).stop 2024-09-16T15:46:09.343 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:09.337+0000 7fad9d113640 1 -- 172.21.15.123:0/940628599 shutdown_connections 2024-09-16T15:46:09.343 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:09.337+0000 7fad9d113640 1 --2- 172.21.15.123:0/940628599 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fad980d2a10 0x7fad980d0fe0 unknown :-1 s=CLOSED pgs=578 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:09.343 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:09.337+0000 7fad9d113640 1 --2- 172.21.15.123:0/940628599 >> v2:172.21.15.184:3300/0 conn(0x7fad98161390 0x7fad980d0630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:09.343 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:09.337+0000 7fad9d113640 1 -- 172.21.15.123:0/940628599 >> 172.21.15.123:0/940628599 conn(0x7fad980d3e30 msgr2=0x7fad980d4240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:46:09.343 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:09.337+0000 7fad9d113640 1 -- 172.21.15.123:0/940628599 shutdown_connections 2024-09-16T15:46:09.343 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:09.337+0000 7fad9d113640 1 -- 172.21.15.123:0/940628599 wait complete. 2024-09-16T15:46:09.344 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:09.341+0000 7fad9d113640 1 Processor -- start 2024-09-16T15:46:09.344 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:09.341+0000 7fad9d113640 1 -- start start 2024-09-16T15:46:09.346 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:09.341+0000 7fad9d113640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fad980d0b70 0x7fad981522c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:46:09.346 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:09.341+0000 7fad9d113640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fad980d2a10 0x7fad980b9410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:46:09.346 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:09.341+0000 7fad9d113640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fad98161390 0x7fad9806e000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:46:09.346 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:09.341+0000 7fad9d113640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fad9816ad60 con 0x7fad980d0b70 2024-09-16T15:46:09.346 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:09.341+0000 7fad9d113640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fad980b4ea0 con 0x7fad980d2a10 2024-09-16T15:46:09.347 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:09.341+0000 7fad9d113640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fad9816aee0 con 0x7fad98161390 2024-09-16T15:46:09.347 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:09.341+0000 7fad96d9d640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fad980d0b70 0x7fad981522c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:46:09.347 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:09.341+0000 7fad96d9d640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fad980d0b70 0x7fad981522c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:52204/0 (socket says 172.21.15.123:52204) 2024-09-16T15:46:09.347 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:09.341+0000 7fad96d9d640 1 -- 172.21.15.123:0/3892979432 learned_addr learned my addr 172.21.15.123:0/3892979432 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:46:09.347 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:09.341+0000 7fad9659c640 1 --2- 172.21.15.123:0/3892979432 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fad980d2a10 0x7fad980b9410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:46:09.347 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:09.341+0000 7fad9759e640 1 --2- 172.21.15.123:0/3892979432 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fad98161390 0x7fad9806e000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:46:09.347 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:09.341+0000 7fad96d9d640 1 -- 172.21.15.123:0/3892979432 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fad98161390 msgr2=0x7fad9806e000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:46:09.347 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:09.341+0000 7fad96d9d640 1 --2- 172.21.15.123:0/3892979432 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fad98161390 0x7fad9806e000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:09.347 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:09.341+0000 7fad96d9d640 1 -- 172.21.15.123:0/3892979432 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fad980d2a10 msgr2=0x7fad980b9410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:46:09.347 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:09.341+0000 7fad96d9d640 1 --2- 172.21.15.123:0/3892979432 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fad980d2a10 0x7fad980b9410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:09.347 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:09.341+0000 7fad96d9d640 1 -- 172.21.15.123:0/3892979432 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fad980ccac0 con 0x7fad980d0b70 2024-09-16T15:46:09.348 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:09.341+0000 7fad9659c640 1 --2- 172.21.15.123:0/3892979432 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fad980d2a10 0x7fad980b9410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T15:46:09.348 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:09.341+0000 7fad96d9d640 1 --2- 172.21.15.123:0/3892979432 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fad980d0b70 0x7fad981522c0 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7fad880058d0 tx=0x7fad8800dde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:46:09.348 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:09.341+0000 7fad7f7fe640 1 -- 172.21.15.123:0/3892979432 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad88017070 con 0x7fad980d0b70 2024-09-16T15:46:09.348 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:09.341+0000 7fad7f7fe640 1 -- 172.21.15.123:0/3892979432 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fad88009d20 con 0x7fad980d0b70 2024-09-16T15:46:09.348 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:09.341+0000 7fad9d113640 1 -- 172.21.15.123:0/3892979432 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fad980ccda0 con 0x7fad980d0b70 2024-09-16T15:46:09.348 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:09.341+0000 7fad7f7fe640 1 -- 172.21.15.123:0/3892979432 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad880106c0 con 0x7fad980d0b70 2024-09-16T15:46:09.348 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:09.341+0000 7fad9d113640 1 -- 172.21.15.123:0/3892979432 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fad980cd640 con 0x7fad980d0b70 2024-09-16T15:46:09.348 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:09.341+0000 7fad9d113640 1 -- 172.21.15.123:0/3892979432 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fad980d0fe0 con 0x7fad980d0b70 2024-09-16T15:46:09.349 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:09.345+0000 7fad7f7fe640 1 -- 172.21.15.123:0/3892979432 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 9) ==== 97233+0+0 (secure 0 0 0) 0x7fad8801a070 con 0x7fad980d0b70 2024-09-16T15:46:09.349 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:09.345+0000 7fad7f7fe640 1 --2- 172.21.15.123:0/3892979432 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fad84073070 0x7fad84075530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:46:09.349 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:09.345+0000 7fad7f7fe640 1 -- 172.21.15.123:0/3892979432 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(47..47 src has 1..47) ==== 5853+0+0 (secure 0 0 0) 0x7fad880947d0 con 0x7fad980d0b70 2024-09-16T15:46:09.349 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:09.345+0000 7fad9659c640 1 --2- 172.21.15.123:0/3892979432 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fad84073070 0x7fad84075530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:46:09.349 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:09.345+0000 7fad9659c640 1 --2- 172.21.15.123:0/3892979432 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fad84073070 0x7fad84075530 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7fad980a1f00 tx=0x7fad8c009040 comp rx=0 tx=0).ready entity=mgr.4135 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:46:09.351 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:09.349+0000 7fad7f7fe640 1 -- 172.21.15.123:0/3892979432 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7fad8805f6e0 con 0x7fad980d0b70 2024-09-16T15:46:09.514 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:09.509+0000 7fad9d113640 1 -- 172.21.15.123:0/3892979432 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) -- 0x7fad980ccf70 con 0x7fad980d0b70 2024-09-16T15:46:10.430 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:10.425+0000 7fad7f7fe640 1 -- 172.21.15.123:0/3892979432 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v48) ==== 181+0+0 (secure 0 0 0) 0x7fad8805efe0 con 0x7fad980d0b70 2024-09-16T15:46:10.431 INFO:teuthology.orchestra.run.smithi123.stderr:pool 'cephfs_metadata' removed 2024-09-16T15:46:10.437 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:10.433+0000 7fad9d113640 1 -- 172.21.15.123:0/3892979432 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fad84073070 msgr2=0x7fad84075530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:46:10.438 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:10.433+0000 7fad9d113640 1 --2- 172.21.15.123:0/3892979432 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fad84073070 0x7fad84075530 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7fad980a1f00 tx=0x7fad8c009040 comp rx=0 tx=0).stop 2024-09-16T15:46:10.438 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:10.433+0000 7fad9d113640 1 -- 172.21.15.123:0/3892979432 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fad980d0b70 msgr2=0x7fad981522c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:46:10.438 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:10.433+0000 7fad9d113640 1 --2- 172.21.15.123:0/3892979432 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fad980d0b70 0x7fad981522c0 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7fad880058d0 tx=0x7fad8800dde0 comp rx=0 tx=0).stop 2024-09-16T15:46:10.438 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:10.433+0000 7fad9d113640 1 -- 172.21.15.123:0/3892979432 shutdown_connections 2024-09-16T15:46:10.438 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:10.433+0000 7fad9d113640 1 --2- 172.21.15.123:0/3892979432 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fad84073070 0x7fad84075530 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:10.438 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:10.433+0000 7fad9d113640 1 --2- 172.21.15.123:0/3892979432 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fad98161390 0x7fad9806e000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:10.438 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:10.433+0000 7fad9d113640 1 --2- 172.21.15.123:0/3892979432 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fad980d2a10 0x7fad980b9410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:10.438 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:10.433+0000 7fad9d113640 1 --2- 172.21.15.123:0/3892979432 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fad980d0b70 0x7fad981522c0 unknown :-1 s=CLOSED pgs=579 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:10.438 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:10.433+0000 7fad9d113640 1 -- 172.21.15.123:0/3892979432 >> 172.21.15.123:0/3892979432 conn(0x7fad980d3e30 msgr2=0x7fad980b09d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:46:10.438 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:10.433+0000 7fad9d113640 1 -- 172.21.15.123:0/3892979432 shutdown_connections 2024-09-16T15:46:10.438 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:10.433+0000 7fad9d113640 1 -- 172.21.15.123:0/3892979432 wait complete. 2024-09-16T15:46:10.458 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-09-16T15:46:10.589 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:10.585+0000 7fc1bcddc640 1 -- 172.21.15.123:0/3147692799 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc1b80b0a80 msgr2=0x7fc1b806e3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:46:10.590 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:10.585+0000 7fc1bcddc640 1 --2- 172.21.15.123:0/3147692799 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc1b80b0a80 0x7fc1b806e3c0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7fc1ac009840 tx=0x7fc1ac01c700 comp rx=0 tx=0).stop 2024-09-16T15:46:10.590 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:10.585+0000 7fc1bcddc640 1 -- 172.21.15.123:0/3147692799 shutdown_connections 2024-09-16T15:46:10.590 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:10.585+0000 7fc1bcddc640 1 --2- 172.21.15.123:0/3147692799 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc1b80b0a80 0x7fc1b806e3c0 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:10.590 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:10.585+0000 7fc1bcddc640 1 --2- 172.21.15.123:0/3147692799 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fc1b8161470 0x7fc1b806dd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:10.590 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:10.585+0000 7fc1bcddc640 1 -- 172.21.15.123:0/3147692799 >> 172.21.15.123:0/3147692799 conn(0x7fc1b80d1bd0 msgr2=0x7fc1b80d1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:46:10.590 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:10.585+0000 7fc1bcddc640 1 -- 172.21.15.123:0/3147692799 shutdown_connections 2024-09-16T15:46:10.590 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:10.585+0000 7fc1bcddc640 1 -- 172.21.15.123:0/3147692799 wait complete. 2024-09-16T15:46:10.591 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:10.585+0000 7fc1bcddc640 1 Processor -- start 2024-09-16T15:46:10.591 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:10.585+0000 7fc1bcddc640 1 -- start start 2024-09-16T15:46:10.593 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:10.589+0000 7fc1bcddc640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc1b806e900 0x7fc1b80b6050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:46:10.593 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:10.589+0000 7fc1bcddc640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fc1b80b0a80 0x7fc1b80caa40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:46:10.593 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:10.589+0000 7fc1bcddc640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc1b8161470 0x7fc1b80c8ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:46:10.593 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:10.589+0000 7fc1bcddc640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fc1b816afc0 con 0x7fc1b806e900 2024-09-16T15:46:10.593 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:10.589+0000 7fc1bcddc640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fc1b816ae40 con 0x7fc1b8161470 2024-09-16T15:46:10.593 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:10.589+0000 7fc1bcddc640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fc1b816b140 con 0x7fc1b80b0a80 2024-09-16T15:46:10.593 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:10.589+0000 7fc1b659c640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc1b806e900 0x7fc1b80b6050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:46:10.593 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:10.589+0000 7fc1b659c640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc1b806e900 0x7fc1b80b6050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:52220/0 (socket says 172.21.15.123:52220) 2024-09-16T15:46:10.593 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:10.589+0000 7fc1b659c640 1 -- 172.21.15.123:0/2679463500 learned_addr learned my addr 172.21.15.123:0/2679463500 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:46:10.594 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:10.589+0000 7fc1b5d9b640 1 --2- 172.21.15.123:0/2679463500 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fc1b80b0a80 0x7fc1b80caa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:46:10.594 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:10.589+0000 7fc1b6d9d640 1 --2- 172.21.15.123:0/2679463500 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc1b8161470 0x7fc1b80c8ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:46:10.594 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:10.589+0000 7fc1b659c640 1 -- 172.21.15.123:0/2679463500 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fc1b80b0a80 msgr2=0x7fc1b80caa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:46:10.594 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:10.589+0000 7fc1b659c640 1 --2- 172.21.15.123:0/2679463500 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fc1b80b0a80 0x7fc1b80caa40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:10.594 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:10.589+0000 7fc1b659c640 1 -- 172.21.15.123:0/2679463500 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc1b8161470 msgr2=0x7fc1b80c8ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:46:10.594 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:10.589+0000 7fc1b659c640 1 --2- 172.21.15.123:0/2679463500 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc1b8161470 0x7fc1b80c8ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:10.594 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:10.589+0000 7fc1b659c640 1 -- 172.21.15.123:0/2679463500 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc1b81fb570 con 0x7fc1b806e900 2024-09-16T15:46:10.594 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:10.589+0000 7fc1b6d9d640 1 --2- 172.21.15.123:0/2679463500 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc1b8161470 0x7fc1b80c8ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T15:46:10.594 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:10.589+0000 7fc1b659c640 1 --2- 172.21.15.123:0/2679463500 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc1b806e900 0x7fc1b80b6050 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7fc1a800e7a0 tx=0x7fc1a800ec60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:46:10.594 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:10.589+0000 7fc1a6ffd640 1 -- 172.21.15.123:0/2679463500 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1a800ccc0 con 0x7fc1b806e900 2024-09-16T15:46:10.594 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:10.589+0000 7fc1a6ffd640 1 -- 172.21.15.123:0/2679463500 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fc1a8010430 con 0x7fc1b806e900 2024-09-16T15:46:10.594 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:10.589+0000 7fc1a6ffd640 1 -- 172.21.15.123:0/2679463500 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1a8010880 con 0x7fc1b806e900 2024-09-16T15:46:10.595 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:10.589+0000 7fc1bcddc640 1 -- 172.21.15.123:0/2679463500 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc1b81f88a0 con 0x7fc1b806e900 2024-09-16T15:46:10.595 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:10.589+0000 7fc1bcddc640 1 -- 172.21.15.123:0/2679463500 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc1b80c9f60 con 0x7fc1b806e900 2024-09-16T15:46:10.596 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:10.589+0000 7fc1bcddc640 1 -- 172.21.15.123:0/2679463500 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc1b806e3c0 con 0x7fc1b806e900 2024-09-16T15:46:10.596 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:10.589+0000 7fc1a6ffd640 1 -- 172.21.15.123:0/2679463500 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 9) ==== 97233+0+0 (secure 0 0 0) 0x7fc1a801f020 con 0x7fc1b806e900 2024-09-16T15:46:10.596 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:10.593+0000 7fc1a6ffd640 1 --2- 172.21.15.123:0/2679463500 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fc198073070 0x7fc198075530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:46:10.596 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:10.593+0000 7fc1a6ffd640 1 -- 172.21.15.123:0/2679463500 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(48..48 src has 1..48) ==== 5415+0+0 (secure 0 0 0) 0x7fc1a8094930 con 0x7fc1b806e900 2024-09-16T15:46:10.596 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:10.593+0000 7fc1b5d9b640 1 --2- 172.21.15.123:0/2679463500 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fc198073070 0x7fc198075530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:46:10.596 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:10.593+0000 7fc1b5d9b640 1 --2- 172.21.15.123:0/2679463500 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fc198073070 0x7fc198075530 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7fc1ac0af000 tx=0x7fc1ac01af40 comp rx=0 tx=0).ready entity=mgr.4135 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:46:10.599 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:10.597+0000 7fc1a6ffd640 1 -- 172.21.15.123:0/2679463500 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7fc1b81fb570 con 0x7fc1b806e900 2024-09-16T15:46:10.773 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:10.769+0000 7fc1bcddc640 1 -- 172.21.15.123:0/2679463500 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) -- 0x7fc1b80d16f0 con 0x7fc1b806e900 2024-09-16T15:46:11.455 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:11.453+0000 7fc1a6ffd640 1 -- 172.21.15.123:0/2679463500 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v49) ==== 169+0+0 (secure 0 0 0) 0x7fc1a805f920 con 0x7fc1b806e900 2024-09-16T15:46:11.455 INFO:teuthology.orchestra.run.smithi123.stderr:pool 'cephfs_data' removed 2024-09-16T15:46:11.458 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:11.453+0000 7fc1bcddc640 1 -- 172.21.15.123:0/2679463500 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fc198073070 msgr2=0x7fc198075530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:46:11.458 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:11.453+0000 7fc1bcddc640 1 --2- 172.21.15.123:0/2679463500 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fc198073070 0x7fc198075530 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7fc1ac0af000 tx=0x7fc1ac01af40 comp rx=0 tx=0).stop 2024-09-16T15:46:11.458 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:11.453+0000 7fc1bcddc640 1 -- 172.21.15.123:0/2679463500 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc1b806e900 msgr2=0x7fc1b80b6050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:46:11.458 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:11.453+0000 7fc1bcddc640 1 --2- 172.21.15.123:0/2679463500 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc1b806e900 0x7fc1b80b6050 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7fc1a800e7a0 tx=0x7fc1a800ec60 comp rx=0 tx=0).stop 2024-09-16T15:46:11.458 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:11.453+0000 7fc1bcddc640 1 -- 172.21.15.123:0/2679463500 shutdown_connections 2024-09-16T15:46:11.459 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:11.453+0000 7fc1bcddc640 1 --2- 172.21.15.123:0/2679463500 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fc198073070 0x7fc198075530 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:11.459 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:11.453+0000 7fc1bcddc640 1 --2- 172.21.15.123:0/2679463500 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc1b8161470 0x7fc1b80c8ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:11.459 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:11.453+0000 7fc1bcddc640 1 --2- 172.21.15.123:0/2679463500 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fc1b80b0a80 0x7fc1b80caa40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:11.459 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:11.453+0000 7fc1bcddc640 1 --2- 172.21.15.123:0/2679463500 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc1b806e900 0x7fc1b80b6050 unknown :-1 s=CLOSED pgs=580 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:11.459 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:11.453+0000 7fc1bcddc640 1 -- 172.21.15.123:0/2679463500 >> 172.21.15.123:0/2679463500 conn(0x7fc1b80d1bd0 msgr2=0x7fc1b80ce450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:46:11.459 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:11.453+0000 7fc1bcddc640 1 -- 172.21.15.123:0/2679463500 shutdown_connections 2024-09-16T15:46:11.459 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:11.453+0000 7fc1bcddc640 1 -- 172.21.15.123:0/2679463500 wait complete. 2024-09-16T15:46:11.481 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config reset 27 2024-09-16T15:46:11.609 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:11.605+0000 7f76430dc640 1 -- 172.21.15.123:0/1942115013 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f763c0d2030 msgr2=0x7f763c161d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:46:11.609 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:11.605+0000 7f76430dc640 1 --2- 172.21.15.123:0/1942115013 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f763c0d2030 0x7f763c161d40 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7f762c009900 tx=0x7f762c01c870 comp rx=0 tx=0).stop 2024-09-16T15:46:11.609 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:11.605+0000 7f76430dc640 1 -- 172.21.15.123:0/1942115013 shutdown_connections 2024-09-16T15:46:11.609 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:11.605+0000 7f76430dc640 1 --2- 172.21.15.123:0/1942115013 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f763c0d2410 0x7f763c1627c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:11.609 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:11.605+0000 7f76430dc640 1 --2- 172.21.15.123:0/1942115013 >> v2:172.21.15.184:3300/0 conn(0x7f763c0ce7d0 0x7f763c162280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:11.609 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:11.605+0000 7f76430dc640 1 --2- 172.21.15.123:0/1942115013 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f763c0d2030 0x7f763c161d40 unknown :-1 s=CLOSED pgs=581 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:11.610 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:11.605+0000 7f76430dc640 1 -- 172.21.15.123:0/1942115013 >> 172.21.15.123:0/1942115013 conn(0x7f763c0d3e60 msgr2=0x7f763c0d4240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:46:11.610 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:11.605+0000 7f76430dc640 1 -- 172.21.15.123:0/1942115013 shutdown_connections 2024-09-16T15:46:11.610 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:11.605+0000 7f76430dc640 1 -- 172.21.15.123:0/1942115013 wait complete. 2024-09-16T15:46:11.612 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:11.605+0000 7f76430dc640 1 Processor -- start 2024-09-16T15:46:11.612 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:11.605+0000 7f76430dc640 1 -- start start 2024-09-16T15:46:11.612 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:11.605+0000 7f76430dc640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f763c0ce7d0 0x7f763c153a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:46:11.612 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:11.605+0000 7f76430dc640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f763c0d2030 0x7f763c160ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:46:11.612 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:11.605+0000 7f76430dc640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f763c0d2410 0x7f763c161400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:46:11.612 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:11.605+0000 7f76430dc640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f763c0b3bd0 con 0x7f763c0ce7d0 2024-09-16T15:46:11.612 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:11.605+0000 7f76430dc640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f763c0b3a50 con 0x7f763c0d2030 2024-09-16T15:46:11.612 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:11.605+0000 7f76430dc640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f763c0b3d50 con 0x7f763c0d2410 2024-09-16T15:46:11.612 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:11.609+0000 7f7640e78640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f763c0ce7d0 0x7f763c153a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:46:11.612 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:11.609+0000 7f7640e78640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f763c0ce7d0 0x7f763c153a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:52238/0 (socket says 172.21.15.123:52238) 2024-09-16T15:46:11.612 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:11.609+0000 7f7640e78640 1 -- 172.21.15.123:0/3444658723 learned_addr learned my addr 172.21.15.123:0/3444658723 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:46:11.613 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:11.609+0000 7f763bfff640 1 --2- 172.21.15.123:0/3444658723 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f763c0d2030 0x7f763c160ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:46:11.613 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:11.609+0000 7f7641679640 1 --2- 172.21.15.123:0/3444658723 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f763c0d2410 0x7f763c161400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:46:11.613 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:11.609+0000 7f7640e78640 1 -- 172.21.15.123:0/3444658723 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f763c0d2410 msgr2=0x7f763c161400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:46:11.613 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:11.609+0000 7f7640e78640 1 --2- 172.21.15.123:0/3444658723 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f763c0d2410 0x7f763c161400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:11.613 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:11.609+0000 7f7640e78640 1 -- 172.21.15.123:0/3444658723 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f763c0d2030 msgr2=0x7f763c160ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:46:11.613 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:11.609+0000 7f7640e78640 1 --2- 172.21.15.123:0/3444658723 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f763c0d2030 0x7f763c160ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:11.613 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:11.609+0000 7f7640e78640 1 -- 172.21.15.123:0/3444658723 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f763c153f50 con 0x7f763c0ce7d0 2024-09-16T15:46:11.613 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:11.609+0000 7f7641679640 1 --2- 172.21.15.123:0/3444658723 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f763c0d2410 0x7f763c161400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T15:46:11.613 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:11.609+0000 7f763bfff640 1 --2- 172.21.15.123:0/3444658723 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f763c0d2030 0x7f763c160ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T15:46:11.613 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:11.609+0000 7f7640e78640 1 --2- 172.21.15.123:0/3444658723 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f763c0ce7d0 0x7f763c153a10 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7f762c009850 tx=0x7f762c004040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:46:11.613 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:11.609+0000 7f76397fa640 1 -- 172.21.15.123:0/3444658723 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f762c0b2040 con 0x7f763c0ce7d0 2024-09-16T15:46:11.613 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:11.609+0000 7f76397fa640 1 -- 172.21.15.123:0/3444658723 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f762c005030 con 0x7f763c0ce7d0 2024-09-16T15:46:11.613 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:11.609+0000 7f76430dc640 1 -- 172.21.15.123:0/3444658723 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f763c0cc9c0 con 0x7f763c0ce7d0 2024-09-16T15:46:11.614 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:11.609+0000 7f76397fa640 1 -- 172.21.15.123:0/3444658723 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f762c005460 con 0x7f763c0ce7d0 2024-09-16T15:46:11.614 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:11.609+0000 7f76430dc640 1 -- 172.21.15.123:0/3444658723 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f763c0cb460 con 0x7f763c0ce7d0 2024-09-16T15:46:11.614 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:11.609+0000 7f76430dc640 1 -- 172.21.15.123:0/3444658723 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f763c0b4270 con 0x7f763c0ce7d0 2024-09-16T15:46:11.614 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:11.609+0000 7f76397fa640 1 -- 172.21.15.123:0/3444658723 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 9) ==== 97233+0+0 (secure 0 0 0) 0x7f762c0046d0 con 0x7f763c0ce7d0 2024-09-16T15:46:11.614 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:11.609+0000 7f76397fa640 1 --2- 172.21.15.123:0/3444658723 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f76280730a0 0x7f7628075560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:46:11.615 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:11.609+0000 7f76397fa640 1 -- 172.21.15.123:0/3444658723 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(49..49 src has 1..49) ==== 4943+0+0 (secure 0 0 0) 0x7f762c12e050 con 0x7f763c0ce7d0 2024-09-16T15:46:11.615 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:11.609+0000 7f763bfff640 1 --2- 172.21.15.123:0/3444658723 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f76280730a0 0x7f7628075560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:46:11.615 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:11.609+0000 7f763bfff640 1 --2- 172.21.15.123:0/3444658723 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f76280730a0 0x7f7628075560 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f7630005fd0 tx=0x7f7630005d60 comp rx=0 tx=0).ready entity=mgr.4135 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:46:11.617 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:11.613+0000 7f76397fa640 1 -- 172.21.15.123:0/3444658723 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f762c016490 con 0x7f763c0ce7d0 2024-09-16T15:46:11.790 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:11.785+0000 7f76430dc640 1 -- 172.21.15.123:0/3444658723 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "config reset", "num": 27} v 0) -- 0x7f763c0b5a60 con 0x7f763c0ce7d0 2024-09-16T15:46:11.794 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:11.789+0000 7f76397fa640 1 -- 172.21.15.123:0/3444658723 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "config reset", "num": 27}]=0 v31) ==== 71+0+0 (secure 0 0 0) 0x7f762c005600 con 0x7f763c0ce7d0 2024-09-16T15:46:11.798 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:11.793+0000 7f76430dc640 1 -- 172.21.15.123:0/3444658723 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f76280730a0 msgr2=0x7f7628075560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:46:11.798 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:11.793+0000 7f76430dc640 1 --2- 172.21.15.123:0/3444658723 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f76280730a0 0x7f7628075560 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f7630005fd0 tx=0x7f7630005d60 comp rx=0 tx=0).stop 2024-09-16T15:46:11.798 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:11.793+0000 7f76430dc640 1 -- 172.21.15.123:0/3444658723 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f763c0ce7d0 msgr2=0x7f763c153a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:46:11.798 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:11.793+0000 7f76430dc640 1 --2- 172.21.15.123:0/3444658723 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f763c0ce7d0 0x7f763c153a10 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7f762c009850 tx=0x7f762c004040 comp rx=0 tx=0).stop 2024-09-16T15:46:11.798 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:11.793+0000 7f76430dc640 1 -- 172.21.15.123:0/3444658723 shutdown_connections 2024-09-16T15:46:11.798 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:11.793+0000 7f76430dc640 1 --2- 172.21.15.123:0/3444658723 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f76280730a0 0x7f7628075560 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:11.798 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:11.793+0000 7f76430dc640 1 --2- 172.21.15.123:0/3444658723 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f763c0d2410 0x7f763c161400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:11.798 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:11.793+0000 7f76430dc640 1 --2- 172.21.15.123:0/3444658723 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f763c0d2030 0x7f763c160ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:11.799 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:11.793+0000 7f76430dc640 1 --2- 172.21.15.123:0/3444658723 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f763c0ce7d0 0x7f763c153a10 unknown :-1 s=CLOSED pgs=582 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:11.799 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:11.793+0000 7f76430dc640 1 -- 172.21.15.123:0/3444658723 >> 172.21.15.123:0/3444658723 conn(0x7f763c0d3e60 msgr2=0x7f763c0b6260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:46:11.799 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:11.793+0000 7f76430dc640 1 -- 172.21.15.123:0/3444658723 shutdown_connections 2024-09-16T15:46:11.799 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:11.793+0000 7f76430dc640 1 -- 172.21.15.123:0/3444658723 wait complete. 2024-09-16T15:46:11.815 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_volumes.TestMisc.test_connection_expiration' 2024-09-16T15:46:11.936 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:11.933+0000 7f4c97f26640 1 -- 172.21.15.123:0/267684761 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4c900d23c0 msgr2=0x7f4c900b7b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:46:11.936 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:11.933+0000 7f4c97f26640 1 --2- 172.21.15.123:0/267684761 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4c900d23c0 0x7f4c900b7b60 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7f4c88009bc0 tx=0x7f4c8801cb40 comp rx=0 tx=0).stop 2024-09-16T15:46:11.936 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:11.933+0000 7f4c97f26640 1 -- 172.21.15.123:0/267684761 shutdown_connections 2024-09-16T15:46:11.936 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:11.933+0000 7f4c97f26640 1 --2- 172.21.15.123:0/267684761 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4c900d23c0 0x7f4c900b7b60 unknown :-1 s=CLOSED pgs=584 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:11.936 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:11.933+0000 7f4c97f26640 1 --2- 172.21.15.123:0/267684761 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f4c900d0f50 0x7f4c900b7620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:11.936 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:11.933+0000 7f4c97f26640 1 -- 172.21.15.123:0/267684761 >> 172.21.15.123:0/267684761 conn(0x7f4c900d3e30 msgr2=0x7f4c900d4240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:46:11.936 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:11.933+0000 7f4c97f26640 1 -- 172.21.15.123:0/267684761 shutdown_connections 2024-09-16T15:46:11.937 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:11.933+0000 7f4c97f26640 1 -- 172.21.15.123:0/267684761 wait complete. 2024-09-16T15:46:11.937 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:11.933+0000 7f4c97f26640 1 Processor -- start 2024-09-16T15:46:11.937 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:11.933+0000 7f4c97f26640 1 -- start start 2024-09-16T15:46:11.939 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:11.933+0000 7f4c97f26640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4c900d0f50 0x7f4c900c8d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:46:11.939 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:11.933+0000 7f4c97f26640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4c900d23c0 0x7f4c900cd950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:46:11.939 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:11.933+0000 7f4c97f26640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f4c90161520 0x7f4c9015b280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:46:11.939 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:11.933+0000 7f4c97f26640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f4c9016b400 con 0x7f4c900d0f50 2024-09-16T15:46:11.939 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:11.933+0000 7f4c97f26640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f4c9016b280 con 0x7f4c900d23c0 2024-09-16T15:46:11.939 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:11.933+0000 7f4c97f26640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f4c9016b580 con 0x7f4c90161520 2024-09-16T15:46:11.939 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:11.933+0000 7f4c95cc2640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4c900d0f50 0x7f4c900c8d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:46:11.939 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:11.933+0000 7f4c95cc2640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4c900d0f50 0x7f4c900c8d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:52260/0 (socket says 172.21.15.123:52260) 2024-09-16T15:46:11.939 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:11.933+0000 7f4c95cc2640 1 -- 172.21.15.123:0/177252312 learned_addr learned my addr 172.21.15.123:0/177252312 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:46:11.939 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:11.933+0000 7f4c964c3640 1 --2- 172.21.15.123:0/177252312 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f4c90161520 0x7f4c9015b280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:46:11.939 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:11.933+0000 7f4c954c1640 1 --2- 172.21.15.123:0/177252312 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4c900d23c0 0x7f4c900cd950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:46:11.939 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:11.933+0000 7f4c95cc2640 1 -- 172.21.15.123:0/177252312 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f4c90161520 msgr2=0x7f4c9015b280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:46:11.940 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:11.933+0000 7f4c95cc2640 1 --2- 172.21.15.123:0/177252312 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f4c90161520 0x7f4c9015b280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:11.940 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:11.933+0000 7f4c95cc2640 1 -- 172.21.15.123:0/177252312 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4c900d23c0 msgr2=0x7f4c900cd950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:46:11.940 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:11.933+0000 7f4c95cc2640 1 --2- 172.21.15.123:0/177252312 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4c900d23c0 0x7f4c900cd950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:11.940 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:11.933+0000 7f4c95cc2640 1 -- 172.21.15.123:0/177252312 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4c900c9370 con 0x7f4c900d0f50 2024-09-16T15:46:11.940 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:11.933+0000 7f4c964c3640 1 --2- 172.21.15.123:0/177252312 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f4c90161520 0x7f4c9015b280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T15:46:11.940 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:11.933+0000 7f4c95cc2640 1 --2- 172.21.15.123:0/177252312 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4c900d0f50 0x7f4c900c8d90 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7f4c8c00c430 tx=0x7f4c8c00c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:46:11.940 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:11.933+0000 7f4c867fc640 1 -- 172.21.15.123:0/177252312 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c8c011790 con 0x7f4c900d0f50 2024-09-16T15:46:11.940 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:11.933+0000 7f4c867fc640 1 -- 172.21.15.123:0/177252312 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f4c8c017070 con 0x7f4c900d0f50 2024-09-16T15:46:11.940 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:11.933+0000 7f4c867fc640 1 -- 172.21.15.123:0/177252312 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c8c011ce0 con 0x7f4c900d0f50 2024-09-16T15:46:11.940 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:11.933+0000 7f4c97f26640 1 -- 172.21.15.123:0/177252312 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4c900c9650 con 0x7f4c900d0f50 2024-09-16T15:46:11.940 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:11.933+0000 7f4c97f26640 1 -- 172.21.15.123:0/177252312 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4c900c9c10 con 0x7f4c900d0f50 2024-09-16T15:46:11.941 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:11.937+0000 7f4c97f26640 1 -- 172.21.15.123:0/177252312 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4c900ca040 con 0x7f4c900d0f50 2024-09-16T15:46:11.941 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:11.937+0000 7f4c867fc640 1 -- 172.21.15.123:0/177252312 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 9) ==== 97233+0+0 (secure 0 0 0) 0x7f4c8c01c460 con 0x7f4c900d0f50 2024-09-16T15:46:11.941 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:11.937+0000 7f4c867fc640 1 --2- 172.21.15.123:0/177252312 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f4c740730a0 0x7f4c74075560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:46:11.941 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:11.937+0000 7f4c867fc640 1 -- 172.21.15.123:0/177252312 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(49..49 src has 1..49) ==== 4943+0+0 (secure 0 0 0) 0x7f4c8c093c40 con 0x7f4c900d0f50 2024-09-16T15:46:11.941 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:11.937+0000 7f4c954c1640 1 --2- 172.21.15.123:0/177252312 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f4c740730a0 0x7f4c74075560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:46:11.942 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:11.937+0000 7f4c954c1640 1 --2- 172.21.15.123:0/177252312 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f4c740730a0 0x7f4c74075560 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f4c800058d0 tx=0x7f4c80005860 comp rx=0 tx=0).ready entity=mgr.4135 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:46:11.945 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:11.941+0000 7f4c867fc640 1 -- 172.21.15.123:0/177252312 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f4c8c05ef50 con 0x7f4c900d0f50 2024-09-16T15:46:12.178 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:12.173+0000 7f4c97f26640 1 -- 172.21.15.123:0/177252312 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_volumes.TestMisc.test_connection_expiration"]} v 0) -- 0x7f4c900d44c0 con 0x7f4c900d0f50 2024-09-16T15:46:12.457 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:12.453+0000 7f4c867fc640 1 -- 172.21.15.123:0/177252312 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_volumes.TestMisc.test_connection_expiration"]}]=0 v173) ==== 140+0+0 (secure 0 0 0) 0x7f4c8c05e850 con 0x7f4c900d0f50 2024-09-16T15:46:12.460 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:12.457+0000 7f4c97f26640 1 -- 172.21.15.123:0/177252312 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f4c740730a0 msgr2=0x7f4c74075560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:46:12.460 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:12.457+0000 7f4c97f26640 1 --2- 172.21.15.123:0/177252312 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f4c740730a0 0x7f4c74075560 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f4c800058d0 tx=0x7f4c80005860 comp rx=0 tx=0).stop 2024-09-16T15:46:12.460 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:12.457+0000 7f4c97f26640 1 -- 172.21.15.123:0/177252312 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4c900d0f50 msgr2=0x7f4c900c8d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:46:12.460 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:12.457+0000 7f4c97f26640 1 --2- 172.21.15.123:0/177252312 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4c900d0f50 0x7f4c900c8d90 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7f4c8c00c430 tx=0x7f4c8c00c8f0 comp rx=0 tx=0).stop 2024-09-16T15:46:12.460 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:12.457+0000 7f4c97f26640 1 -- 172.21.15.123:0/177252312 shutdown_connections 2024-09-16T15:46:12.461 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:12.457+0000 7f4c97f26640 1 --2- 172.21.15.123:0/177252312 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f4c740730a0 0x7f4c74075560 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:12.461 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:12.457+0000 7f4c97f26640 1 --2- 172.21.15.123:0/177252312 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f4c90161520 0x7f4c9015b280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:12.461 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:12.457+0000 7f4c97f26640 1 --2- 172.21.15.123:0/177252312 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4c900d23c0 0x7f4c900cd950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:12.461 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:12.457+0000 7f4c97f26640 1 --2- 172.21.15.123:0/177252312 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4c900d0f50 0x7f4c900c8d90 unknown :-1 s=CLOSED pgs=585 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:12.461 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:12.457+0000 7f4c97f26640 1 -- 172.21.15.123:0/177252312 >> 172.21.15.123:0/177252312 conn(0x7f4c900d3e30 msgr2=0x7f4c9006d410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:46:12.461 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:12.457+0000 7f4c97f26640 1 -- 172.21.15.123:0/177252312 shutdown_connections 2024-09-16T15:46:12.461 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:12.457+0000 7f4c97f26640 1 -- 172.21.15.123:0/177252312 wait complete. 2024-09-16T15:46:12.479 INFO:tasks.cephfs_test_runner:test_connection_expiration (tasks.cephfs.test_volumes.TestMisc) ... ok 2024-09-16T15:46:12.480 INFO:tasks.cephfs_test_runner:Starting test: test_malicious_metafile_on_legacy_to_v1_upgrade (tasks.cephfs.test_volumes.TestMisc) Validate handcrafted .meta file on legacy subvol root doesn't break the system 2024-09-16T15:46:12.480 INFO:tasks.cephfs_test_runner:test_malicious_metafile_on_legacy_to_v1_upgrade (tasks.cephfs.test_volumes.TestMisc) 2024-09-16T15:46:12.480 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-16T15:46:12.607 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:12.605+0000 7f4950328640 1 -- 172.21.15.123:0/767884504 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4934004d50 msgr2=0x7f49340051d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:46:12.607 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:12.605+0000 7f4950328640 1 --2- 172.21.15.123:0/767884504 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4934004d50 0x7f49340051d0 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7f4940009ee0 tx=0x7f494001d630 comp rx=0 tx=0).stop 2024-09-16T15:46:12.607 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:12.605+0000 7f494e8c5640 1 -- 172.21.15.123:0/767884504 reap_dead start 2024-09-16T15:46:12.608 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:12.605+0000 7f4950328640 1 -- 172.21.15.123:0/767884504 shutdown_connections 2024-09-16T15:46:12.608 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:12.605+0000 7f4950328640 1 -- 172.21.15.123:0/767884504 >> 172.21.15.123:0/767884504 conn(0x7f49480cfb10 msgr2=0x7f49480cff20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:46:12.608 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:12.605+0000 7f4950328640 1 -- 172.21.15.123:0/767884504 shutdown_connections 2024-09-16T15:46:12.608 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:12.605+0000 7f4950328640 1 -- 172.21.15.123:0/767884504 wait complete. 2024-09-16T15:46:12.608 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:12.605+0000 7f4950328640 1 Processor -- start 2024-09-16T15:46:12.608 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:12.605+0000 7f4950328640 1 -- start start 2024-09-16T15:46:12.610 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:12.605+0000 7f4950328640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f49480c9ae0 0x7f49480c4ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:46:12.611 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:12.605+0000 7f4950328640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f49480c5530 0x7f49480c7370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:46:12.611 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:12.605+0000 7f4950328640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f49480c78b0 0x7f49481f5ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:46:12.611 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:12.605+0000 7f4950328640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f49480d09c0 con 0x7f49480c78b0 2024-09-16T15:46:12.611 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:12.605+0000 7f4950328640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f49480d0840 con 0x7f49480c9ae0 2024-09-16T15:46:12.611 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:12.605+0000 7f4950328640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f494806ec90 con 0x7f49480c5530 2024-09-16T15:46:12.611 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:12.605+0000 7f494e8c5640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f49480c78b0 0x7f49481f5ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:46:12.611 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:12.605+0000 7f494e8c5640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f49480c78b0 0x7f49481f5ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:41992/0 (socket says 172.21.15.123:41992) 2024-09-16T15:46:12.611 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:12.605+0000 7f494e8c5640 1 -- 172.21.15.123:0/3264221171 learned_addr learned my addr 172.21.15.123:0/3264221171 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:46:12.611 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:12.605+0000 7f494e0c4640 1 --2- 172.21.15.123:0/3264221171 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f49480c9ae0 0x7f49480c4ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:46:12.611 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:12.605+0000 7f494d8c3640 1 --2- 172.21.15.123:0/3264221171 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f49480c5530 0x7f49480c7370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:46:12.611 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:12.605+0000 7f494e8c5640 1 -- 172.21.15.123:0/3264221171 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f49480c5530 msgr2=0x7f49480c7370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:46:12.611 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:12.605+0000 7f494e8c5640 1 --2- 172.21.15.123:0/3264221171 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f49480c5530 0x7f49480c7370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:12.611 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:12.605+0000 7f494e8c5640 1 -- 172.21.15.123:0/3264221171 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f49480c9ae0 msgr2=0x7f49480c4ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:46:12.612 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:12.605+0000 7f494e8c5640 1 --2- 172.21.15.123:0/3264221171 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f49480c9ae0 0x7f49480c4ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:12.612 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:12.605+0000 7f494e8c5640 1 -- 172.21.15.123:0/3264221171 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f49480c9ec0 con 0x7f49480c78b0 2024-09-16T15:46:12.612 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:12.605+0000 7f494e8c5640 1 --2- 172.21.15.123:0/3264221171 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f49480c78b0 0x7f49481f5ab0 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7f494001d0a0 tx=0x7f4940004f30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:46:12.612 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:12.605+0000 7f493effd640 1 -- 172.21.15.123:0/3264221171 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49400ad070 con 0x7f49480c78b0 2024-09-16T15:46:12.612 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:12.605+0000 7f493effd640 1 -- 172.21.15.123:0/3264221171 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f4940003980 con 0x7f49480c78b0 2024-09-16T15:46:12.612 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:12.605+0000 7f4950328640 1 -- 172.21.15.123:0/3264221171 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f49480c7d30 con 0x7f49480c78b0 2024-09-16T15:46:12.612 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:12.605+0000 7f493effd640 1 -- 172.21.15.123:0/3264221171 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4940009200 con 0x7f49480c78b0 2024-09-16T15:46:12.612 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:12.605+0000 7f4950328640 1 -- 172.21.15.123:0/3264221171 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f49481f2780 con 0x7f49480c78b0 2024-09-16T15:46:12.613 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:12.609+0000 7f4950328640 1 -- 172.21.15.123:0/3264221171 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f494806e750 con 0x7f49480c78b0 2024-09-16T15:46:12.613 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:12.609+0000 7f493effd640 1 -- 172.21.15.123:0/3264221171 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 9) ==== 97233+0+0 (secure 0 0 0) 0x7f49400b0070 con 0x7f49480c78b0 2024-09-16T15:46:12.613 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:12.609+0000 7f493effd640 1 --2- 172.21.15.123:0/3264221171 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f492c0730a0 0x7f492c075560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:46:12.613 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:12.609+0000 7f493effd640 1 -- 172.21.15.123:0/3264221171 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(49..49 src has 1..49) ==== 4943+0+0 (secure 0 0 0) 0x7f494012eab0 con 0x7f49480c78b0 2024-09-16T15:46:12.613 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:12.609+0000 7f494e0c4640 1 --2- 172.21.15.123:0/3264221171 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f492c0730a0 0x7f492c075560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:46:12.614 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:12.609+0000 7f494e0c4640 1 --2- 172.21.15.123:0/3264221171 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f492c0730a0 0x7f492c075560 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f493800fa40 tx=0x7f49380015d0 comp rx=0 tx=0).ready entity=mgr.4135 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:46:12.615 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:12.613+0000 7f493effd640 1 -- 172.21.15.123:0/3264221171 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f49400b7070 con 0x7f49480c78b0 2024-09-16T15:46:12.789 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:12.785+0000 7f4950328640 1 -- 172.21.15.123:0/3264221171 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f494815f630 con 0x7f49480c78b0 2024-09-16T15:46:12.791 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:12.785+0000 7f493effd640 1 -- 172.21.15.123:0/3264221171 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v49) ==== 74+0+13092 (secure 0 0 0) 0x7f4940017340 con 0x7f49480c78b0 2024-09-16T15:46:12.791 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-09-16T15:46:12.791 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":49,"fsid":"56077f89-18c8-486f-ae7e-eecc0bb7822c","created":"2024-09-16T15:42:26.364892+0000","modified":"2024-09-16T15:46:11.444625+0000","last_up_change":"2024-09-16T15:42:38.707234+0000","last_in_change":"2024-09-16T15:42:29.423662+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-16T15:42:39.287764+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9e7c1955-a607-4671-9be7-f2ef760ceeaa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6800","nonce":3882044969},{"type":"v1","addr":"172.21.15.123:6801","nonce":3882044969}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6802","nonce":3882044969},{"type":"v1","addr":"172.21.15.123:6803","nonce":3882044969}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6806","nonce":3882044969},{"type":"v1","addr":"172.21.15.123:6807","nonce":3882044969}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6804","nonce":3882044969},{"type":"v1","addr":"172.21.15.123:6805","nonce":3882044969}]},"public_addr":"172.21.15.123:6801/3882044969","cluster_addr":"172.21.15.123:6803/3882044969","heartbeat_back_addr":"172.21.15.123:6807/3882044969","heartbeat_front_addr":"172.21.15.123:6805/3882044969","state":["exists","up"]},{"osd":1,"uuid":"c540cfde-0977-4742-9b17-fd634668e010","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6816","nonce":4037001850},{"type":"v1","addr":"172.21.15.123:6817","nonce":4037001850}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6818","nonce":4037001850},{"type":"v1","addr":"172.21.15.123:6819","nonce":4037001850}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6822","nonce":4037001850},{"type":"v1","addr":"172.21.15.123:6823","nonce":4037001850}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6820","nonce":4037001850},{"type":"v1","addr":"172.21.15.123:6821","nonce":4037001850}]},"public_addr":"172.21.15.123:6817/4037001850","cluster_addr":"172.21.15.123:6819/4037001850","heartbeat_back_addr":"172.21.15.123:6823/4037001850","heartbeat_front_addr":"172.21.15.123:6821/4037001850","state":["exists","up"]},{"osd":2,"uuid":"9864a496-f169-486b-95c4-1d59162793a7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6824","nonce":1332143088},{"type":"v1","addr":"172.21.15.123:6825","nonce":1332143088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6826","nonce":1332143088},{"type":"v1","addr":"172.21.15.123:6827","nonce":1332143088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6830","nonce":1332143088},{"type":"v1","addr":"172.21.15.123:6831","nonce":1332143088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6828","nonce":1332143088},{"type":"v1","addr":"172.21.15.123:6829","nonce":1332143088}]},"public_addr":"172.21.15.123:6825/1332143088","cluster_addr":"172.21.15.123:6827/1332143088","heartbeat_back_addr":"172.21.15.123:6831/1332143088","heartbeat_front_addr":"172.21.15.123:6829/1332143088","state":["exists","up"]},{"osd":3,"uuid":"c6d1f817-f074-4325-95e1-578eba13edda","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6808","nonce":988586998},{"type":"v1","addr":"172.21.15.123:6809","nonce":988586998}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6810","nonce":988586998},{"type":"v1","addr":"172.21.15.123:6811","nonce":988586998}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6814","nonce":988586998},{"type":"v1","addr":"172.21.15.123:6815","nonce":988586998}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6812","nonce":988586998},{"type":"v1","addr":"172.21.15.123:6813","nonce":988586998}]},"public_addr":"172.21.15.123:6809/988586998","cluster_addr":"172.21.15.123:6811/988586998","heartbeat_back_addr":"172.21.15.123:6815/988586998","heartbeat_front_addr":"172.21.15.123:6813/988586998","state":["exists","up"]},{"osd":4,"uuid":"200a8df4-50ef-40a5-b48a-58981269c086","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6816","nonce":3387813890},{"type":"v1","addr":"172.21.15.184:6817","nonce":3387813890}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6818","nonce":3387813890},{"type":"v1","addr":"172.21.15.184:6819","nonce":3387813890}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6822","nonce":3387813890},{"type":"v1","addr":"172.21.15.184:6823","nonce":3387813890}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6820","nonce":3387813890},{"type":"v1","addr":"172.21.15.184:6821","nonce":3387813890}]},"public_addr":"172.21.15.184:6817/3387813890","cluster_addr":"172.21.15.184:6819/3387813890","heartbeat_back_addr":"172.21.15.184:6823/3387813890","heartbeat_front_addr":"172.21.15.184:6821/3387813890","state":["exists","up"]},{"osd":5,"uuid":"32d5a900-da3d-412c-bc19-13861a2368d8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6808","nonce":2680916017},{"type":"v1","addr":"172.21.15.184:6809","nonce":2680916017}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6810","nonce":2680916017},{"type":"v1","addr":"172.21.15.184:6811","nonce":2680916017}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6814","nonce":2680916017},{"type":"v1","addr":"172.21.15.184:6815","nonce":2680916017}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6812","nonce":2680916017},{"type":"v1","addr":"172.21.15.184:6813","nonce":2680916017}]},"public_addr":"172.21.15.184:6809/2680916017","cluster_addr":"172.21.15.184:6811/2680916017","heartbeat_back_addr":"172.21.15.184:6815/2680916017","heartbeat_front_addr":"172.21.15.184:6813/2680916017","state":["exists","up"]},{"osd":6,"uuid":"cde6f39e-aa1b-4260-ac19-774ee6aaca72","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6824","nonce":2183120457},{"type":"v1","addr":"172.21.15.184:6825","nonce":2183120457}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6826","nonce":2183120457},{"type":"v1","addr":"172.21.15.184:6827","nonce":2183120457}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6830","nonce":2183120457},{"type":"v1","addr":"172.21.15.184:6831","nonce":2183120457}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6828","nonce":2183120457},{"type":"v1","addr":"172.21.15.184:6829","nonce":2183120457}]},"public_addr":"172.21.15.184:6825/2183120457","cluster_addr":"172.21.15.184:6827/2183120457","heartbeat_back_addr":"172.21.15.184:6831/2183120457","heartbeat_front_addr":"172.21.15.184:6829/2183120457","state":["exists","up"]},{"osd":7,"uuid":"4f381778-eee1-47c6-804f-df0e6b33e313","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6800","nonce":4102957285},{"type":"v1","addr":"172.21.15.184:6801","nonce":4102957285}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6802","nonce":4102957285},{"type":"v1","addr":"172.21.15.184:6803","nonce":4102957285}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6806","nonce":4102957285},{"type":"v1","addr":"172.21.15.184:6807","nonce":4102957285}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6804","nonce":4102957285},{"type":"v1","addr":"172.21.15.184:6805","nonce":4102957285}]},"public_addr":"172.21.15.184:6801/4102957285","cluster_addr":"172.21.15.184:6803/4102957285","heartbeat_back_addr":"172.21.15.184:6807/4102957285","heartbeat_front_addr":"172.21.15.184:6805/4102957285","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:35.972210+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.222007+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.198617+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.385663+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.358935+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.156066+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.752973+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:35.682846+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.184:6835/374389412":"2024-09-17T15:46:08.188349+0000","172.21.15.184:6834/374389412":"2024-09-17T15:46:08.188349+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T15:46:12.793 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:12.789+0000 7f4950328640 1 -- 172.21.15.123:0/3264221171 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f492c0730a0 msgr2=0x7f492c075560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:46:12.793 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:12.789+0000 7f4950328640 1 --2- 172.21.15.123:0/3264221171 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f492c0730a0 0x7f492c075560 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f493800fa40 tx=0x7f49380015d0 comp rx=0 tx=0).stop 2024-09-16T15:46:12.793 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:12.789+0000 7f4950328640 1 -- 172.21.15.123:0/3264221171 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f49480c78b0 msgr2=0x7f49481f5ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:46:12.793 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:12.789+0000 7f4950328640 1 --2- 172.21.15.123:0/3264221171 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f49480c78b0 0x7f49481f5ab0 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7f494001d0a0 tx=0x7f4940004f30 comp rx=0 tx=0).stop 2024-09-16T15:46:12.794 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:12.789+0000 7f4950328640 1 -- 172.21.15.123:0/3264221171 shutdown_connections 2024-09-16T15:46:12.794 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:12.789+0000 7f4950328640 1 --2- 172.21.15.123:0/3264221171 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f492c0730a0 0x7f492c075560 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:12.794 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:12.789+0000 7f4950328640 1 --2- 172.21.15.123:0/3264221171 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f49480c78b0 0x7f49481f5ab0 unknown :-1 s=CLOSED pgs=588 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:12.794 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:12.789+0000 7f4950328640 1 --2- 172.21.15.123:0/3264221171 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f49480c5530 0x7f49480c7370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:12.794 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:12.789+0000 7f4950328640 1 --2- 172.21.15.123:0/3264221171 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f49480c9ae0 0x7f49480c4ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:12.794 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:12.789+0000 7f4950328640 1 -- 172.21.15.123:0/3264221171 >> 172.21.15.123:0/3264221171 conn(0x7f49480cfb10 msgr2=0x7f4948073ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:46:12.794 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:12.789+0000 7f4950328640 1 -- 172.21.15.123:0/3264221171 shutdown_connections 2024-09-16T15:46:12.794 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:12.789+0000 7f4950328640 1 -- 172.21.15.123:0/3264221171 wait complete. 2024-09-16T15:46:12.809 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-16T15:42:39.287764+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-09-16T15:46:12.809 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-09-16T15:46:12.927 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:12.925+0000 7f2ab0b1b640 1 -- 172.21.15.123:0/2111885277 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2a94006400 msgr2=0x7f2a9400cae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:46:12.927 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:12.925+0000 7f2ab0b1b640 1 --2- 172.21.15.123:0/2111885277 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2a94006400 0x7f2a9400cae0 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7f2aac070a20 tx=0x7f2aa401f460 comp rx=0 tx=0).stop 2024-09-16T15:46:12.928 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:12.925+0000 7f2ab0b1b640 1 -- 172.21.15.123:0/2111885277 shutdown_connections 2024-09-16T15:46:12.928 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:12.925+0000 7f2ab0b1b640 1 --2- 172.21.15.123:0/2111885277 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2a94006400 0x7f2a9400cae0 unknown :-1 s=CLOSED pgs=590 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:12.928 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:12.925+0000 7f2ab0b1b640 1 -- 172.21.15.123:0/2111885277 >> 172.21.15.123:0/2111885277 conn(0x7f2aac0b1710 msgr2=0x7f2aac0b1b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:46:12.928 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:12.925+0000 7f2ab0b1b640 1 -- 172.21.15.123:0/2111885277 shutdown_connections 2024-09-16T15:46:12.928 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:12.925+0000 7f2ab0b1b640 1 -- 172.21.15.123:0/2111885277 wait complete. 2024-09-16T15:46:12.928 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:12.925+0000 7f2ab0b1b640 1 Processor -- start 2024-09-16T15:46:12.928 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:12.925+0000 7f2ab0b1b640 1 -- start start 2024-09-16T15:46:12.930 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:12.925+0000 7f2ab0b1b640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2a94006400 0x7f2aac0c7c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:46:12.930 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:12.925+0000 7f2ab0b1b640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2aac1f3ab0 0x7f2aac0c3630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:46:12.930 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:12.925+0000 7f2ab0b1b640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f2aac0c3b70 0x7f2aac141330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:46:12.931 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:12.925+0000 7f2ab0b1b640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f2aac06e530 con 0x7f2aac1f3ab0 2024-09-16T15:46:12.931 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:12.925+0000 7f2ab0b1b640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f2aac06e3b0 con 0x7f2a94006400 2024-09-16T15:46:12.931 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:12.925+0000 7f2ab0b1b640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f2aac06e6b0 con 0x7f2aac0c3b70 2024-09-16T15:46:12.931 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:12.925+0000 7f2aa9d9b640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2aac1f3ab0 0x7f2aac0c3630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:46:12.931 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:12.925+0000 7f2aa9d9b640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2aac1f3ab0 0x7f2aac0c3630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:42010/0 (socket says 172.21.15.123:42010) 2024-09-16T15:46:12.931 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:12.925+0000 7f2aa9d9b640 1 -- 172.21.15.123:0/3410327481 learned_addr learned my addr 172.21.15.123:0/3410327481 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:46:12.931 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:12.925+0000 7f2aaa59c640 1 --2- 172.21.15.123:0/3410327481 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2a94006400 0x7f2aac0c7c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:46:12.931 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:12.925+0000 7f2aaad9d640 1 --2- 172.21.15.123:0/3410327481 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f2aac0c3b70 0x7f2aac141330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:46:12.931 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:12.925+0000 7f2aa9d9b640 1 -- 172.21.15.123:0/3410327481 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f2aac0c3b70 msgr2=0x7f2aac141330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:46:12.931 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:12.925+0000 7f2aa9d9b640 1 --2- 172.21.15.123:0/3410327481 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f2aac0c3b70 0x7f2aac141330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:12.931 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:12.925+0000 7f2aa9d9b640 1 -- 172.21.15.123:0/3410327481 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2a94006400 msgr2=0x7f2aac0c7c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:46:12.931 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:12.925+0000 7f2aa9d9b640 1 --2- 172.21.15.123:0/3410327481 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2a94006400 0x7f2aac0c7c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:12.931 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:12.925+0000 7f2aa9d9b640 1 -- 172.21.15.123:0/3410327481 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2aac0c5500 con 0x7f2aac1f3ab0 2024-09-16T15:46:12.931 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:12.925+0000 7f2aaad9d640 1 --2- 172.21.15.123:0/3410327481 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f2aac0c3b70 0x7f2aac141330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T15:46:12.932 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:12.925+0000 7f2aa9d9b640 1 --2- 172.21.15.123:0/3410327481 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2aac1f3ab0 0x7f2aac0c3630 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7f2aa0004e00 tx=0x7f2aa0001640 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:46:12.932 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:12.925+0000 7f2a9affd640 1 -- 172.21.15.123:0/3410327481 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2aa0015020 con 0x7f2aac1f3ab0 2024-09-16T15:46:12.932 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:12.925+0000 7f2a9affd640 1 -- 172.21.15.123:0/3410327481 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f2aa0011960 con 0x7f2aac1f3ab0 2024-09-16T15:46:12.932 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:12.925+0000 7f2a9affd640 1 -- 172.21.15.123:0/3410327481 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2aa0011d90 con 0x7f2aac1f3ab0 2024-09-16T15:46:12.932 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:12.925+0000 7f2ab0b1b640 1 -- 172.21.15.123:0/3410327481 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2aac1f3f30 con 0x7f2aac1f3ab0 2024-09-16T15:46:12.932 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:12.925+0000 7f2ab0b1b640 1 -- 172.21.15.123:0/3410327481 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2aac1f44c0 con 0x7f2aac1f3ab0 2024-09-16T15:46:12.933 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:12.929+0000 7f2ab0b1b640 1 -- 172.21.15.123:0/3410327481 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2aac073090 con 0x7f2aac1f3ab0 2024-09-16T15:46:12.933 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:12.929+0000 7f2a9affd640 1 -- 172.21.15.123:0/3410327481 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 9) ==== 97233+0+0 (secure 0 0 0) 0x7f2aa0018020 con 0x7f2aac1f3ab0 2024-09-16T15:46:12.933 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:12.929+0000 7f2a9affd640 1 --2- 172.21.15.123:0/3410327481 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f2a8c073070 0x7f2a8c075530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:46:12.933 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:12.929+0000 7f2a9affd640 1 -- 172.21.15.123:0/3410327481 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(49..49 src has 1..49) ==== 4943+0+0 (secure 0 0 0) 0x7f2aa0094a30 con 0x7f2aac1f3ab0 2024-09-16T15:46:12.933 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:12.929+0000 7f2aaa59c640 1 --2- 172.21.15.123:0/3410327481 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f2a8c073070 0x7f2a8c075530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:46:12.933 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:12.929+0000 7f2aaa59c640 1 --2- 172.21.15.123:0/3410327481 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f2a8c073070 0x7f2a8c075530 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f2a9c0015d0 tx=0x7f2a9c0014a0 comp rx=0 tx=0).ready entity=mgr.4135 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:46:12.936 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:12.933+0000 7f2a9affd640 1 -- 172.21.15.123:0/3410327481 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f2aa001f020 con 0x7f2aac1f3ab0 2024-09-16T15:46:13.093 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:13.089+0000 7f2ab0b1b640 1 -- 172.21.15.123:0/3410327481 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f2aac0b3100 con 0x7f2aac1f3ab0 2024-09-16T15:46:13.094 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:13.089+0000 7f2a9affd640 1 -- 172.21.15.123:0/3410327481 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v49) ==== 93+0+10 (secure 0 0 0) 0x7f2aac0c5500 con 0x7f2aac1f3ab0 2024-09-16T15:46:13.094 INFO:teuthology.orchestra.run.smithi123.stdout:pg_num: 1 2024-09-16T15:46:13.096 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:13.093+0000 7f2ab0b1b640 1 -- 172.21.15.123:0/3410327481 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f2a8c073070 msgr2=0x7f2a8c075530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:46:13.096 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:13.093+0000 7f2ab0b1b640 1 --2- 172.21.15.123:0/3410327481 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f2a8c073070 0x7f2a8c075530 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f2a9c0015d0 tx=0x7f2a9c0014a0 comp rx=0 tx=0).stop 2024-09-16T15:46:13.096 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:13.093+0000 7f2ab0b1b640 1 -- 172.21.15.123:0/3410327481 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2aac1f3ab0 msgr2=0x7f2aac0c3630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:46:13.097 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:13.093+0000 7f2ab0b1b640 1 --2- 172.21.15.123:0/3410327481 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2aac1f3ab0 0x7f2aac0c3630 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7f2aa0004e00 tx=0x7f2aa0001640 comp rx=0 tx=0).stop 2024-09-16T15:46:13.097 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:13.093+0000 7f2ab0b1b640 1 -- 172.21.15.123:0/3410327481 shutdown_connections 2024-09-16T15:46:13.097 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:13.093+0000 7f2ab0b1b640 1 --2- 172.21.15.123:0/3410327481 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f2a8c073070 0x7f2a8c075530 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:13.097 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:13.093+0000 7f2ab0b1b640 1 --2- 172.21.15.123:0/3410327481 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f2aac0c3b70 0x7f2aac141330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:13.097 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:13.093+0000 7f2ab0b1b640 1 --2- 172.21.15.123:0/3410327481 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2aac1f3ab0 0x7f2aac0c3630 unknown :-1 s=CLOSED pgs=591 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:13.097 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:13.093+0000 7f2ab0b1b640 1 --2- 172.21.15.123:0/3410327481 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2a94006400 0x7f2aac0c7c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:13.097 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:13.093+0000 7f2ab0b1b640 1 -- 172.21.15.123:0/3410327481 >> 172.21.15.123:0/3410327481 conn(0x7f2aac0b1710 msgr2=0x7f2aac0b5450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:46:13.097 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:13.093+0000 7f2ab0b1b640 1 -- 172.21.15.123:0/3410327481 shutdown_connections 2024-09-16T15:46:13.097 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:13.093+0000 7f2ab0b1b640 1 -- 172.21.15.123:0/3410327481 wait complete. 2024-09-16T15:46:13.111 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_volumes.TestMisc.test_malicious_metafile_on_legacy_to_v1_upgrade' 2024-09-16T15:46:13.224 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:13.221+0000 7f8948cd8640 1 -- 172.21.15.123:0/1045798923 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f894415e160 msgr2=0x7f8944073f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:46:13.225 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:13.221+0000 7f8948cd8640 1 --2- 172.21.15.123:0/1045798923 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f894415e160 0x7f8944073f30 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7f89440708f0 tx=0x7f893401c6e0 comp rx=0 tx=0).stop 2024-09-16T15:46:13.225 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:13.221+0000 7f8948cd8640 1 -- 172.21.15.123:0/1045798923 shutdown_connections 2024-09-16T15:46:13.225 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:13.221+0000 7f8948cd8640 1 --2- 172.21.15.123:0/1045798923 >> v2:172.21.15.184:3300/0 conn(0x7f894415e820 0x7f894415ec90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:13.225 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:13.221+0000 7f8948cd8640 1 --2- 172.21.15.123:0/1045798923 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f894415e160 0x7f8944073f30 unknown :-1 s=CLOSED pgs=593 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:13.225 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:13.221+0000 7f8948cd8640 1 -- 172.21.15.123:0/1045798923 >> 172.21.15.123:0/1045798923 conn(0x7f89440cfb10 msgr2=0x7f89440cff20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:46:13.225 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:13.221+0000 7f8948cd8640 1 -- 172.21.15.123:0/1045798923 shutdown_connections 2024-09-16T15:46:13.225 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:13.221+0000 7f8948cd8640 1 -- 172.21.15.123:0/1045798923 wait complete. 2024-09-16T15:46:13.226 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:13.221+0000 7f8948cd8640 1 Processor -- start 2024-09-16T15:46:13.226 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:13.221+0000 7f8948cd8640 1 -- start start 2024-09-16T15:46:13.228 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:13.225+0000 7f8948cd8640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8944074470 0x7f89440c6be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:46:13.228 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:13.225+0000 7f8948cd8640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f894415e160 0x7f8944164fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:46:13.228 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:13.225+0000 7f8948cd8640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f894415e820 0x7f894414f2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:46:13.229 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:13.225+0000 7f8948cd8640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f89440cf5b0 con 0x7f8944074470 2024-09-16T15:46:13.229 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:13.225+0000 7f8948cd8640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f89440cf430 con 0x7f894415e820 2024-09-16T15:46:13.229 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:13.225+0000 7f8948cd8640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f89441683d0 con 0x7f894415e160 2024-09-16T15:46:13.229 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:13.225+0000 7f894259c640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8944074470 0x7f89440c6be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:46:13.229 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:13.225+0000 7f894259c640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8944074470 0x7f89440c6be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:42032/0 (socket says 172.21.15.123:42032) 2024-09-16T15:46:13.229 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:13.225+0000 7f894259c640 1 -- 172.21.15.123:0/3550699082 learned_addr learned my addr 172.21.15.123:0/3550699082 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:46:13.229 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:13.225+0000 7f894259c640 1 -- 172.21.15.123:0/3550699082 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f894415e160 msgr2=0x7f8944164fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:46:13.229 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:13.225+0000 7f894259c640 1 --2- 172.21.15.123:0/3550699082 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f894415e160 0x7f8944164fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:13.229 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:13.225+0000 7f894259c640 1 -- 172.21.15.123:0/3550699082 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f894415e820 msgr2=0x7f894414f2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:46:13.229 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:13.225+0000 7f894259c640 1 --2- 172.21.15.123:0/3550699082 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f894415e820 0x7f894414f2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:13.229 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:13.225+0000 7f894259c640 1 -- 172.21.15.123:0/3550699082 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f89440b2bb0 con 0x7f8944074470 2024-09-16T15:46:13.229 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:13.225+0000 7f894259c640 1 --2- 172.21.15.123:0/3550699082 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8944074470 0x7f89440c6be0 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7f8934000c10 tx=0x7f89340a5670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:46:13.229 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:13.225+0000 7f893affd640 1 -- 172.21.15.123:0/3550699082 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89340b2040 con 0x7f8944074470 2024-09-16T15:46:13.230 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:13.225+0000 7f893affd640 1 -- 172.21.15.123:0/3550699082 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f8934004340 con 0x7f8944074470 2024-09-16T15:46:13.230 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:13.225+0000 7f8948cd8640 1 -- 172.21.15.123:0/3550699082 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f89440b2e90 con 0x7f8944074470 2024-09-16T15:46:13.230 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:13.225+0000 7f893affd640 1 -- 172.21.15.123:0/3550699082 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89340044f0 con 0x7f8944074470 2024-09-16T15:46:13.230 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:13.225+0000 7f8948cd8640 1 -- 172.21.15.123:0/3550699082 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8944165520 con 0x7f8944074470 2024-09-16T15:46:13.230 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:13.225+0000 7f8948cd8640 1 -- 172.21.15.123:0/3550699082 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8944073f30 con 0x7f8944074470 2024-09-16T15:46:13.230 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:13.225+0000 7f893affd640 1 -- 172.21.15.123:0/3550699082 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 9) ==== 97233+0+0 (secure 0 0 0) 0x7f8934003840 con 0x7f8944074470 2024-09-16T15:46:13.231 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:13.225+0000 7f893affd640 1 --2- 172.21.15.123:0/3550699082 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f8924073070 0x7f8924075530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:46:13.231 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:13.225+0000 7f893affd640 1 -- 172.21.15.123:0/3550699082 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(49..49 src has 1..49) ==== 4943+0+0 (secure 0 0 0) 0x7f893412e120 con 0x7f8944074470 2024-09-16T15:46:13.231 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:13.225+0000 7f8941d9b640 1 --2- 172.21.15.123:0/3550699082 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f8924073070 0x7f8924075530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:46:13.231 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:13.229+0000 7f8941d9b640 1 --2- 172.21.15.123:0/3550699082 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f8924073070 0x7f8924075530 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f89440c9a70 tx=0x7f892c007a00 comp rx=0 tx=0).ready entity=mgr.4135 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:46:13.234 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:13.229+0000 7f893affd640 1 -- 172.21.15.123:0/3550699082 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f89340163f0 con 0x7f8944074470 2024-09-16T15:46:13.467 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:13.465+0000 7f8948cd8640 1 -- 172.21.15.123:0/3550699082 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_volumes.TestMisc.test_malicious_metafile_on_legacy_to_v1_upgrade"]} v 0) -- 0x7f89440d06c0 con 0x7f8944074470 2024-09-16T15:46:14.468 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:14.465+0000 7f893affd640 1 -- 172.21.15.123:0/3550699082 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_volumes.TestMisc.test_malicious_metafile_on_legacy_to_v1_upgrade"]}]=0 v175) ==== 164+0+0 (secure 0 0 0) 0x7f89340f9370 con 0x7f8944074470 2024-09-16T15:46:14.471 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:14.469+0000 7f8948cd8640 1 -- 172.21.15.123:0/3550699082 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f8924073070 msgr2=0x7f8924075530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:46:14.471 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:14.469+0000 7f8948cd8640 1 --2- 172.21.15.123:0/3550699082 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f8924073070 0x7f8924075530 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f89440c9a70 tx=0x7f892c007a00 comp rx=0 tx=0).stop 2024-09-16T15:46:14.471 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:14.469+0000 7f8948cd8640 1 -- 172.21.15.123:0/3550699082 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8944074470 msgr2=0x7f89440c6be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:46:14.472 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:14.469+0000 7f8948cd8640 1 --2- 172.21.15.123:0/3550699082 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8944074470 0x7f89440c6be0 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7f8934000c10 tx=0x7f89340a5670 comp rx=0 tx=0).stop 2024-09-16T15:46:14.472 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:14.469+0000 7f8948cd8640 1 -- 172.21.15.123:0/3550699082 shutdown_connections 2024-09-16T15:46:14.472 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:14.469+0000 7f8948cd8640 1 --2- 172.21.15.123:0/3550699082 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f8924073070 0x7f8924075530 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:14.472 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:14.469+0000 7f8948cd8640 1 --2- 172.21.15.123:0/3550699082 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f894415e820 0x7f894414f2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:14.472 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:14.469+0000 7f8948cd8640 1 --2- 172.21.15.123:0/3550699082 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f894415e160 0x7f8944164fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:14.472 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:14.469+0000 7f8948cd8640 1 --2- 172.21.15.123:0/3550699082 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8944074470 0x7f89440c6be0 unknown :-1 s=CLOSED pgs=594 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:14.472 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:14.469+0000 7f8948cd8640 1 -- 172.21.15.123:0/3550699082 >> 172.21.15.123:0/3550699082 conn(0x7f89440cfb10 msgr2=0x7f8944153da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:46:14.472 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:14.469+0000 7f8948cd8640 1 -- 172.21.15.123:0/3550699082 shutdown_connections 2024-09-16T15:46:14.472 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:14.469+0000 7f8948cd8640 1 -- 172.21.15.123:0/3550699082 wait complete. 2024-09-16T15:46:14.493 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config dump 2024-09-16T15:46:14.619 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:14.617+0000 7effc8b10640 1 -- 172.21.15.123:0/1118625465 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7effc40726c0 msgr2=0x7effc40d0630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:46:14.620 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:14.617+0000 7effc8b10640 1 --2- 172.21.15.123:0/1118625465 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7effc40726c0 0x7effc40d0630 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7effb4009bc0 tx=0x7effb401cae0 comp rx=0 tx=0).stop 2024-09-16T15:46:14.620 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:14.617+0000 7effc8b10640 1 -- 172.21.15.123:0/1118625465 shutdown_connections 2024-09-16T15:46:14.620 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:14.617+0000 7effc8b10640 1 --2- 172.21.15.123:0/1118625465 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7effc4074600 0x7effc406e750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:14.620 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:14.617+0000 7effc8b10640 1 --2- 172.21.15.123:0/1118625465 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7effc40726c0 0x7effc40d0630 unknown :-1 s=CLOSED pgs=596 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:14.620 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:14.617+0000 7effc8b10640 1 -- 172.21.15.123:0/1118625465 >> 172.21.15.123:0/1118625465 conn(0x7effc40cfb10 msgr2=0x7effc40cff20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:46:14.620 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:14.617+0000 7effc8b10640 1 -- 172.21.15.123:0/1118625465 shutdown_connections 2024-09-16T15:46:14.620 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:14.617+0000 7effc8b10640 1 -- 172.21.15.123:0/1118625465 wait complete. 2024-09-16T15:46:14.621 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:14.617+0000 7effc8b10640 1 Processor -- start 2024-09-16T15:46:14.621 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:14.617+0000 7effc8b10640 1 -- start start 2024-09-16T15:46:14.623 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:14.617+0000 7effc8b10640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7effc40726c0 0x7effc41f6330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:46:14.623 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:14.617+0000 7effc8b10640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7effc4074600 0x7effc4154ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:46:14.623 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:14.617+0000 7effc8b10640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7effc415bca0 0x7effc41655e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:46:14.623 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:14.617+0000 7effc8b10640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7effc4168bd0 con 0x7effc4074600 2024-09-16T15:46:14.623 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:14.617+0000 7effc8b10640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7effc4168a50 con 0x7effc40726c0 2024-09-16T15:46:14.623 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:14.617+0000 7effc8b10640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7effc4168d50 con 0x7effc415bca0 2024-09-16T15:46:14.623 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:14.617+0000 7effc1d9b640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7effc4074600 0x7effc4154ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:46:14.623 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:14.617+0000 7effc1d9b640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7effc4074600 0x7effc4154ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:42058/0 (socket says 172.21.15.123:42058) 2024-09-16T15:46:14.624 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:14.617+0000 7effc1d9b640 1 -- 172.21.15.123:0/2402749144 learned_addr learned my addr 172.21.15.123:0/2402749144 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:46:14.624 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:14.617+0000 7effc2d9d640 1 --2- 172.21.15.123:0/2402749144 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7effc415bca0 0x7effc41655e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:46:14.624 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:14.617+0000 7effc259c640 1 --2- 172.21.15.123:0/2402749144 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7effc40726c0 0x7effc41f6330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:46:14.624 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:14.617+0000 7effc1d9b640 1 -- 172.21.15.123:0/2402749144 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7effc415bca0 msgr2=0x7effc41655e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:46:14.624 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:14.617+0000 7effc1d9b640 1 --2- 172.21.15.123:0/2402749144 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7effc415bca0 0x7effc41655e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:14.624 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:14.617+0000 7effc1d9b640 1 -- 172.21.15.123:0/2402749144 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7effc40726c0 msgr2=0x7effc41f6330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:46:14.624 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:14.617+0000 7effc1d9b640 1 --2- 172.21.15.123:0/2402749144 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7effc40726c0 0x7effc41f6330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:14.624 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:14.617+0000 7effc1d9b640 1 -- 172.21.15.123:0/2402749144 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7effc40c7990 con 0x7effc4074600 2024-09-16T15:46:14.624 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:14.617+0000 7effc2d9d640 1 --2- 172.21.15.123:0/2402749144 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7effc415bca0 0x7effc41655e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T15:46:14.624 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:14.617+0000 7effc1d9b640 1 --2- 172.21.15.123:0/2402749144 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7effc4074600 0x7effc4154ca0 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7effb4009b10 tx=0x7effb40a5b00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:46:14.624 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:14.621+0000 7effb2ffd640 1 -- 172.21.15.123:0/2402749144 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7effb40bb070 con 0x7effc4074600 2024-09-16T15:46:14.624 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:14.621+0000 7effb2ffd640 1 -- 172.21.15.123:0/2402749144 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7effb4004e00 con 0x7effc4074600 2024-09-16T15:46:14.624 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:14.621+0000 7effb2ffd640 1 -- 172.21.15.123:0/2402749144 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7effb40b2040 con 0x7effc4074600 2024-09-16T15:46:14.625 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:14.621+0000 7effc8b10640 1 -- 172.21.15.123:0/2402749144 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7effc41f5850 con 0x7effc4074600 2024-09-16T15:46:14.625 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:14.621+0000 7effc8b10640 1 -- 172.21.15.123:0/2402749144 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7effc41f5e10 con 0x7effc4074600 2024-09-16T15:46:14.625 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:14.621+0000 7effc8b10640 1 -- 172.21.15.123:0/2402749144 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7effc40d0630 con 0x7effc4074600 2024-09-16T15:46:14.626 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:14.621+0000 7effb2ffd640 1 -- 172.21.15.123:0/2402749144 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 9) ==== 97233+0+0 (secure 0 0 0) 0x7effb40a5b70 con 0x7effc4074600 2024-09-16T15:46:14.626 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:14.621+0000 7effb2ffd640 1 --2- 172.21.15.123:0/2402749144 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7effa40730a0 0x7effa4075560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:46:14.626 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:14.621+0000 7effb2ffd640 1 -- 172.21.15.123:0/2402749144 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(49..49 src has 1..49) ==== 4943+0+0 (secure 0 0 0) 0x7effb412ddf0 con 0x7effc4074600 2024-09-16T15:46:14.626 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:14.621+0000 7effc259c640 1 --2- 172.21.15.123:0/2402749144 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7effa40730a0 0x7effa4075560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:46:14.627 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:14.625+0000 7effc259c640 1 --2- 172.21.15.123:0/2402749144 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7effa40730a0 0x7effa4075560 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7effb8001570 tx=0x7effb8009040 comp rx=0 tx=0).ready entity=mgr.4135 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:46:14.630 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:14.625+0000 7effb2ffd640 1 -- 172.21.15.123:0/2402749144 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7effb40f20b0 con 0x7effc4074600 2024-09-16T15:46:14.788 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:14.785+0000 7effc8b10640 1 -- 172.21.15.123:0/2402749144 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "config dump"} v 0) -- 0x7effc41551e0 con 0x7effc4074600 2024-09-16T15:46:14.789 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:14.785+0000 7effb2ffd640 1 -- 172.21.15.123:0/2402749144 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "config dump"}]=0 v31) ==== 59+0+1998 (secure 0 0 0) 0x7effb4016790 con 0x7effc4074600 2024-09-16T15:46:14.789 INFO:teuthology.orchestra.run.smithi123.stdout:WHO MASK LEVEL OPTION VALUE RO 2024-09-16T15:46:14.789 INFO:teuthology.orchestra.run.smithi123.stdout:mon advanced mon_op_complaint_time 120 2024-09-16T15:46:14.789 INFO:teuthology.orchestra.run.smithi123.stdout:mon dev mon_warn_on_pool_no_app false 2024-09-16T15:46:14.790 INFO:teuthology.orchestra.run.smithi123.stdout:osd advanced osd_op_complaint_time 180.000000 2024-09-16T15:46:14.790 INFO:teuthology.orchestra.run.smithi123.stdout:osd.0 basic osd_mclock_max_capacity_iops_ssd 4805.300677 2024-09-16T15:46:14.790 INFO:teuthology.orchestra.run.smithi123.stdout:osd.1 basic osd_mclock_max_capacity_iops_ssd 4806.240607 2024-09-16T15:46:14.790 INFO:teuthology.orchestra.run.smithi123.stdout:osd.2 basic osd_mclock_max_capacity_iops_ssd 4046.441686 2024-09-16T15:46:14.790 INFO:teuthology.orchestra.run.smithi123.stdout:osd.3 basic osd_mclock_max_capacity_iops_ssd 6882.576123 2024-09-16T15:46:14.790 INFO:teuthology.orchestra.run.smithi123.stdout:osd.4 basic osd_mclock_max_capacity_iops_ssd 6108.444971 2024-09-16T15:46:14.790 INFO:teuthology.orchestra.run.smithi123.stdout:osd.5 basic osd_mclock_max_capacity_iops_ssd 4995.096306 2024-09-16T15:46:14.790 INFO:teuthology.orchestra.run.smithi123.stdout:osd.6 basic osd_mclock_max_capacity_iops_ssd 6237.822095 2024-09-16T15:46:14.790 INFO:teuthology.orchestra.run.smithi123.stdout:osd.7 basic osd_mclock_max_capacity_iops_ssd 5202.385138 2024-09-16T15:46:14.790 INFO:teuthology.orchestra.run.smithi123.stdout:mds advanced debug_mds 20/20 2024-09-16T15:46:14.790 INFO:teuthology.orchestra.run.smithi123.stdout:mds advanced debug_mds_balancer 20/20 2024-09-16T15:46:14.790 INFO:teuthology.orchestra.run.smithi123.stdout:mds advanced debug_ms 2/2 2024-09-16T15:46:14.790 INFO:teuthology.orchestra.run.smithi123.stdout:mds dev mds_debug_frag true 2024-09-16T15:46:14.791 INFO:teuthology.orchestra.run.smithi123.stdout:mds dev mds_debug_scatterstat true 2024-09-16T15:46:14.791 INFO:teuthology.orchestra.run.smithi123.stdout:mds advanced mds_op_complaint_time 180.000000 2024-09-16T15:46:14.791 INFO:teuthology.orchestra.run.smithi123.stdout:mds dev mds_verify_scatter true 2024-09-16T15:46:14.791 INFO:teuthology.orchestra.run.smithi123.stdout:mds advanced osd_op_complaint_time 180.000000 2024-09-16T15:46:14.791 INFO:teuthology.orchestra.run.smithi123.stdout:mds advanced rados_mon_op_timeout 900 2024-09-16T15:46:14.791 INFO:teuthology.orchestra.run.smithi123.stdout:mds advanced rados_osd_op_timeout 900 2024-09-16T15:46:14.791 INFO:teuthology.orchestra.run.smithi123.stdout:client advanced client_mount_timeout 600 2024-09-16T15:46:14.791 INFO:teuthology.orchestra.run.smithi123.stdout:client advanced debug_client 20/20 2024-09-16T15:46:14.791 INFO:teuthology.orchestra.run.smithi123.stdout:client advanced debug_ms 1/1 2024-09-16T15:46:14.791 INFO:teuthology.orchestra.run.smithi123.stdout:client advanced rados_mon_op_timeout 900 2024-09-16T15:46:14.791 INFO:teuthology.orchestra.run.smithi123.stdout:client advanced rados_osd_op_timeout 900 2024-09-16T15:46:14.792 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:14.789+0000 7effc8b10640 1 -- 172.21.15.123:0/2402749144 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7effa40730a0 msgr2=0x7effa4075560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:46:14.792 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:14.789+0000 7effc8b10640 1 --2- 172.21.15.123:0/2402749144 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7effa40730a0 0x7effa4075560 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7effb8001570 tx=0x7effb8009040 comp rx=0 tx=0).stop 2024-09-16T15:46:14.792 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:14.789+0000 7effc8b10640 1 -- 172.21.15.123:0/2402749144 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7effc4074600 msgr2=0x7effc4154ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:46:14.792 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:14.789+0000 7effc8b10640 1 --2- 172.21.15.123:0/2402749144 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7effc4074600 0x7effc4154ca0 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7effb4009b10 tx=0x7effb40a5b00 comp rx=0 tx=0).stop 2024-09-16T15:46:14.792 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:14.789+0000 7effc8b10640 1 -- 172.21.15.123:0/2402749144 shutdown_connections 2024-09-16T15:46:14.792 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:14.789+0000 7effc8b10640 1 --2- 172.21.15.123:0/2402749144 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7effa40730a0 0x7effa4075560 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:14.793 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:14.789+0000 7effc8b10640 1 --2- 172.21.15.123:0/2402749144 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7effc415bca0 0x7effc41655e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:14.793 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:14.789+0000 7effc8b10640 1 --2- 172.21.15.123:0/2402749144 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7effc4074600 0x7effc4154ca0 unknown :-1 s=CLOSED pgs=597 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:14.793 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:14.789+0000 7effc8b10640 1 --2- 172.21.15.123:0/2402749144 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7effc40726c0 0x7effc41f6330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:14.793 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:14.789+0000 7effc8b10640 1 -- 172.21.15.123:0/2402749144 >> 172.21.15.123:0/2402749144 conn(0x7effc40cfb10 msgr2=0x7effc40c33f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:46:14.793 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:14.789+0000 7effc8b10640 1 -- 172.21.15.123:0/2402749144 shutdown_connections 2024-09-16T15:46:14.793 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:14.789+0000 7effc8b10640 1 -- 172.21.15.123:0/2402749144 wait complete. 2024-09-16T15:46:14.807 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-09-16T15:46:14.924 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:14.921+0000 7f4974da8640 1 -- 172.21.15.123:0/3040968730 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4958004810 msgr2=0x7f4958004c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:46:14.925 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:14.921+0000 7f4974da8640 1 --2- 172.21.15.123:0/3040968730 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4958004810 0x7f4958004c90 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7f496400c2e0 tx=0x7f496401cef0 comp rx=0 tx=0).stop 2024-09-16T15:46:14.925 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:14.921+0000 7f4974da8640 1 -- 172.21.15.123:0/3040968730 shutdown_connections 2024-09-16T15:46:14.925 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:14.921+0000 7f4974da8640 1 --2- 172.21.15.123:0/3040968730 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4958004810 0x7f4958004c90 unknown :-1 s=CLOSED pgs=599 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:14.925 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:14.921+0000 7f4974da8640 1 -- 172.21.15.123:0/3040968730 >> 172.21.15.123:0/3040968730 conn(0x7f49700b1710 msgr2=0x7f49700b1b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:46:14.925 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:14.921+0000 7f4974da8640 1 -- 172.21.15.123:0/3040968730 shutdown_connections 2024-09-16T15:46:14.925 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:14.921+0000 7f4974da8640 1 -- 172.21.15.123:0/3040968730 wait complete. 2024-09-16T15:46:14.925 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:14.921+0000 7f4974da8640 1 Processor -- start 2024-09-16T15:46:14.926 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:14.921+0000 7f4974da8640 1 -- start start 2024-09-16T15:46:14.927 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:14.921+0000 7f4974da8640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4958004810 0x7f49701f8bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:46:14.927 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:14.921+0000 7f4974da8640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f49701f9110 0x7f49701f55e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:46:14.928 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:14.921+0000 7f4974da8640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f49701f5b20 0x7f49701f5fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:46:14.928 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:14.921+0000 7f4974da8640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f49700728a0 con 0x7f4958004810 2024-09-16T15:46:14.928 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:14.921+0000 7f4974da8640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f4970072720 con 0x7f49701f9110 2024-09-16T15:46:14.928 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:14.921+0000 7f4974da8640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f4970072a20 con 0x7f49701f5b20 2024-09-16T15:46:14.928 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:14.921+0000 7f496e59c640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4958004810 0x7f49701f8bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:46:14.928 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:14.921+0000 7f496e59c640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4958004810 0x7f49701f8bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:42082/0 (socket says 172.21.15.123:42082) 2024-09-16T15:46:14.928 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:14.921+0000 7f496e59c640 1 -- 172.21.15.123:0/1590407084 learned_addr learned my addr 172.21.15.123:0/1590407084 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:46:14.928 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:14.921+0000 7f496ed9d640 1 --2- 172.21.15.123:0/1590407084 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f49701f5b20 0x7f49701f5fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:46:14.928 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:14.921+0000 7f496dd9b640 1 --2- 172.21.15.123:0/1590407084 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f49701f9110 0x7f49701f55e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:46:14.928 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:14.921+0000 7f496e59c640 1 -- 172.21.15.123:0/1590407084 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f49701f5b20 msgr2=0x7f49701f5fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:46:14.928 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:14.921+0000 7f496e59c640 1 --2- 172.21.15.123:0/1590407084 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f49701f5b20 0x7f49701f5fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:14.929 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:14.921+0000 7f496e59c640 1 -- 172.21.15.123:0/1590407084 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f49701f9110 msgr2=0x7f49701f55e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:46:14.929 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:14.921+0000 7f496e59c640 1 --2- 172.21.15.123:0/1590407084 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f49701f9110 0x7f49701f55e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:14.929 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:14.921+0000 7f496e59c640 1 -- 172.21.15.123:0/1590407084 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f49700c4bd0 con 0x7f4958004810 2024-09-16T15:46:14.929 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:14.921+0000 7f496dd9b640 1 --2- 172.21.15.123:0/1590407084 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f49701f9110 0x7f49701f55e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T15:46:14.929 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:14.925+0000 7f496e59c640 1 --2- 172.21.15.123:0/1590407084 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4958004810 0x7f49701f8bd0 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7f496000e9f0 tx=0x7f496000eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:46:14.929 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:14.925+0000 7f495effd640 1 -- 172.21.15.123:0/1590407084 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f496000cc60 con 0x7f4958004810 2024-09-16T15:46:14.929 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:14.925+0000 7f495effd640 1 -- 172.21.15.123:0/1590407084 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f4960011470 con 0x7f4958004810 2024-09-16T15:46:14.929 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:14.925+0000 7f4974da8640 1 -- 172.21.15.123:0/1590407084 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f49700c4eb0 con 0x7f4958004810 2024-09-16T15:46:14.929 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:14.925+0000 7f495effd640 1 -- 172.21.15.123:0/1590407084 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4960011860 con 0x7f4958004810 2024-09-16T15:46:14.929 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:14.925+0000 7f4974da8640 1 -- 172.21.15.123:0/1590407084 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f49701f9a20 con 0x7f4958004810 2024-09-16T15:46:14.930 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:14.925+0000 7f4974da8640 1 -- 172.21.15.123:0/1590407084 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4970072f80 con 0x7f4958004810 2024-09-16T15:46:14.930 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:14.925+0000 7f495effd640 1 -- 172.21.15.123:0/1590407084 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 9) ==== 97233+0+0 (secure 0 0 0) 0x7f4960011a00 con 0x7f4958004810 2024-09-16T15:46:14.930 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:14.925+0000 7f495effd640 1 --2- 172.21.15.123:0/1590407084 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f4950073070 0x7f4950075530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:46:14.930 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:14.925+0000 7f495effd640 1 -- 172.21.15.123:0/1590407084 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(49..49 src has 1..49) ==== 4943+0+0 (secure 0 0 0) 0x7f4960094b40 con 0x7f4958004810 2024-09-16T15:46:14.930 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:14.925+0000 7f496dd9b640 1 --2- 172.21.15.123:0/1590407084 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f4950073070 0x7f4950075530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:46:14.930 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:14.925+0000 7f496dd9b640 1 --2- 172.21.15.123:0/1590407084 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f4950073070 0x7f4950075530 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f49640ac000 tx=0x7f4964019d30 comp rx=0 tx=0).ready entity=mgr.4135 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:46:14.933 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:14.929+0000 7f495effd640 1 -- 172.21.15.123:0/1590407084 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f4960016030 con 0x7f4958004810 2024-09-16T15:46:15.091 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:15.089+0000 7f4974da8640 1 -- 172.21.15.123:0/1590407084 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) -- 0x7f49701f94f0 con 0x7f4958004810 2024-09-16T15:46:15.097 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:15.093+0000 7f495effd640 1 -- 172.21.15.123:0/1590407084 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v32)=0 v32) ==== 122+0+0 (secure 0 0 0) 0x7f496005fd00 con 0x7f4958004810 2024-09-16T15:46:15.100 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:15.097+0000 7f4974da8640 1 -- 172.21.15.123:0/1590407084 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f4950073070 msgr2=0x7f4950075530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:46:15.100 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:15.097+0000 7f4974da8640 1 --2- 172.21.15.123:0/1590407084 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f4950073070 0x7f4950075530 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f49640ac000 tx=0x7f4964019d30 comp rx=0 tx=0).stop 2024-09-16T15:46:15.100 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:15.097+0000 7f4974da8640 1 -- 172.21.15.123:0/1590407084 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4958004810 msgr2=0x7f49701f8bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:46:15.100 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:15.097+0000 7f4974da8640 1 --2- 172.21.15.123:0/1590407084 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4958004810 0x7f49701f8bd0 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7f496000e9f0 tx=0x7f496000eeb0 comp rx=0 tx=0).stop 2024-09-16T15:46:15.100 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:15.097+0000 7f4974da8640 1 -- 172.21.15.123:0/1590407084 shutdown_connections 2024-09-16T15:46:15.100 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:15.097+0000 7f4974da8640 1 --2- 172.21.15.123:0/1590407084 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f4950073070 0x7f4950075530 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:15.100 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:15.097+0000 7f4974da8640 1 --2- 172.21.15.123:0/1590407084 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f49701f5b20 0x7f49701f5fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:15.100 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:15.097+0000 7f4974da8640 1 --2- 172.21.15.123:0/1590407084 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f49701f9110 0x7f49701f55e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:15.100 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:15.097+0000 7f4974da8640 1 --2- 172.21.15.123:0/1590407084 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4958004810 0x7f49701f8bd0 unknown :-1 s=CLOSED pgs=600 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:15.100 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:15.097+0000 7f4974da8640 1 -- 172.21.15.123:0/1590407084 >> 172.21.15.123:0/1590407084 conn(0x7f49700b1710 msgr2=0x7f49700b2a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:46:15.100 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:15.097+0000 7f4974da8640 1 -- 172.21.15.123:0/1590407084 shutdown_connections 2024-09-16T15:46:15.100 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:15.097+0000 7f4974da8640 1 -- 172.21.15.123:0/1590407084 wait complete. 2024-09-16T15:46:15.114 INFO:teuthology.nuke:Clearing teuthology firewall rules... 2024-09-16T15:46:15.114 DEBUG:teuthology.orchestra.run.smithi123:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-09-16T15:46:15.180 DEBUG:teuthology.orchestra.run.smithi184:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-09-16T15:46:15.240 INFO:teuthology.nuke:Cleared teuthology firewall rules. 2024-09-16T15:46:15.241 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2024-09-16T15:46:15.241 DEBUG:teuthology.orchestra.run.smithi123:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-16T15:46:15.246 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16336468k,nr_inodes=4084117,mode=755,inode64 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3274684k,mode=755,inode64 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev,inode64 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=17082 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3274680k,nr_inodes=818670,mode=700,uid=1000,gid=1271,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3274684k,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-09-16T15:46:15.246 DEBUG:teuthology.orchestra.run.smithi184:> set -ex 2024-09-16T15:46:15.247 DEBUG:teuthology.orchestra.run.smithi184:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-16T15:46:15.252 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16336488k,nr_inodes=4084122,mode=755,inode64 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3274688k,mode=755,inode64 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev,inode64 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=17834 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3274684k,nr_inodes=818671,mode=700,uid=1000,gid=1271,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3274688k,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-09-16T15:46:15.252 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2024-09-16T15:46:15.252 DEBUG:teuthology.orchestra.run.smithi123:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-16T15:46:15.295 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16336468k,nr_inodes=4084117,mode=755,inode64 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3274684k,mode=755,inode64 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev,inode64 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=17082 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3274680k,nr_inodes=818670,mode=700,uid=1000,gid=1271,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3274684k,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-09-16T15:46:15.295 DEBUG:teuthology.orchestra.run.smithi184:> set -ex 2024-09-16T15:46:15.295 DEBUG:teuthology.orchestra.run.smithi184:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-16T15:46:15.301 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16336488k,nr_inodes=4084122,mode=755,inode64 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3274688k,mode=755,inode64 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev,inode64 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=17834 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3274684k,nr_inodes=818671,mode=700,uid=1000,gid=1271,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3274688k,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-09-16T15:46:15.301 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-09-16T15:46:15.302 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-16T15:46:15.467 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:15.465+0000 7fca446bb640 1 -- 172.21.15.123:0/2804404394 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fca3c074430 msgr2=0x7fca3c0ca160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:46:15.468 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:15.465+0000 7fca446bb640 1 --2- 172.21.15.123:0/2804404394 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fca3c074430 0x7fca3c0ca160 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7fca3c069550 tx=0x7fca3401ca00 comp rx=0 tx=0).stop 2024-09-16T15:46:15.468 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:15.465+0000 7fca446bb640 1 -- 172.21.15.123:0/2804404394 shutdown_connections 2024-09-16T15:46:15.468 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:15.465+0000 7fca446bb640 1 --2- 172.21.15.123:0/2804404394 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fca3c074430 0x7fca3c0ca160 unknown :-1 s=CLOSED pgs=601 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:15.468 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:15.465+0000 7fca446bb640 1 --2- 172.21.15.123:0/2804404394 >> v2:172.21.15.184:3300/0 conn(0x7fca3c0726c0 0x7fca3c073ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:15.468 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:15.465+0000 7fca446bb640 1 --2- 172.21.15.123:0/2804404394 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fca3c15bca0 0x7fca3c0d0630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:15.468 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:15.465+0000 7fca446bb640 1 -- 172.21.15.123:0/2804404394 >> 172.21.15.123:0/2804404394 conn(0x7fca3c0cfb10 msgr2=0x7fca3c0cff20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:46:15.468 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:15.465+0000 7fca446bb640 1 -- 172.21.15.123:0/2804404394 shutdown_connections 2024-09-16T15:46:15.468 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:15.465+0000 7fca446bb640 1 -- 172.21.15.123:0/2804404394 wait complete. 2024-09-16T15:46:15.469 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:15.465+0000 7fca446bb640 1 Processor -- start 2024-09-16T15:46:15.469 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:15.465+0000 7fca446bb640 1 -- start start 2024-09-16T15:46:15.471 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:15.465+0000 7fca446bb640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fca3c0726c0 0x7fca3c155fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:46:15.471 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:15.465+0000 7fca446bb640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fca3c074430 0x7fca3c15d6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:46:15.471 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:15.465+0000 7fca446bb640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fca3c15bca0 0x7fca3c0c74c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:46:15.471 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:15.465+0000 7fca446bb640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fca3c06ec70 con 0x7fca3c074430 2024-09-16T15:46:15.471 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:15.465+0000 7fca446bb640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fca3c06eaf0 con 0x7fca3c0726c0 2024-09-16T15:46:15.471 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:15.465+0000 7fca446bb640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fca3c06edf0 con 0x7fca3c15bca0 2024-09-16T15:46:15.471 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:15.465+0000 7fca41c56640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fca3c074430 0x7fca3c15d6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:46:15.471 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:15.465+0000 7fca41c56640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fca3c074430 0x7fca3c15d6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:42092/0 (socket says 172.21.15.123:42092) 2024-09-16T15:46:15.471 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:15.465+0000 7fca41c56640 1 -- 172.21.15.123:0/3588693474 learned_addr learned my addr 172.21.15.123:0/3588693474 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:46:15.471 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:15.465+0000 7fca42c58640 1 --2- 172.21.15.123:0/3588693474 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fca3c15bca0 0x7fca3c0c74c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:46:15.471 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:15.465+0000 7fca42457640 1 --2- 172.21.15.123:0/3588693474 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fca3c0726c0 0x7fca3c155fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:46:15.471 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:15.465+0000 7fca41c56640 1 -- 172.21.15.123:0/3588693474 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fca3c15bca0 msgr2=0x7fca3c0c74c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:46:15.472 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:15.465+0000 7fca41c56640 1 --2- 172.21.15.123:0/3588693474 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fca3c15bca0 0x7fca3c0c74c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:15.472 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:15.465+0000 7fca41c56640 1 -- 172.21.15.123:0/3588693474 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fca3c0726c0 msgr2=0x7fca3c155fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:46:15.472 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:15.465+0000 7fca41c56640 1 --2- 172.21.15.123:0/3588693474 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fca3c0726c0 0x7fca3c155fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:15.472 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:15.465+0000 7fca41c56640 1 -- 172.21.15.123:0/3588693474 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fca3c0cee70 con 0x7fca3c074430 2024-09-16T15:46:15.472 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:15.465+0000 7fca42c58640 1 --2- 172.21.15.123:0/3588693474 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fca3c15bca0 0x7fca3c0c74c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T15:46:15.472 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:15.465+0000 7fca41c56640 1 --2- 172.21.15.123:0/3588693474 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fca3c074430 0x7fca3c15d6b0 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7fca2c00d770 tx=0x7fca2c00dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:46:15.472 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:15.465+0000 7fca32ffd640 1 -- 172.21.15.123:0/3588693474 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca2c017070 con 0x7fca3c074430 2024-09-16T15:46:15.472 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:15.465+0000 7fca32ffd640 1 -- 172.21.15.123:0/3588693474 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fca2c009d20 con 0x7fca3c074430 2024-09-16T15:46:15.472 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:15.465+0000 7fca446bb640 1 -- 172.21.15.123:0/3588693474 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fca3c0cf150 con 0x7fca3c074430 2024-09-16T15:46:15.472 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:15.465+0000 7fca32ffd640 1 -- 172.21.15.123:0/3588693474 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca2c0106c0 con 0x7fca3c074430 2024-09-16T15:46:15.472 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:15.465+0000 7fca446bb640 1 -- 172.21.15.123:0/3588693474 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fca3c15dbf0 con 0x7fca3c074430 2024-09-16T15:46:15.473 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:15.469+0000 7fca446bb640 1 -- 172.21.15.123:0/3588693474 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fca3c0ca160 con 0x7fca3c074430 2024-09-16T15:46:15.473 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:15.469+0000 7fca32ffd640 1 -- 172.21.15.123:0/3588693474 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 9) ==== 97233+0+0 (secure 0 0 0) 0x7fca2c01a070 con 0x7fca3c074430 2024-09-16T15:46:15.473 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:15.469+0000 7fca32ffd640 1 --2- 172.21.15.123:0/3588693474 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fca20073070 0x7fca20075530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:46:15.473 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:15.469+0000 7fca42457640 1 --2- 172.21.15.123:0/3588693474 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fca20073070 0x7fca20075530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:46:15.473 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:15.469+0000 7fca32ffd640 1 -- 172.21.15.123:0/3588693474 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(49..49 src has 1..49) ==== 4943+0+0 (secure 0 0 0) 0x7fca2c05af00 con 0x7fca3c074430 2024-09-16T15:46:15.474 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:15.469+0000 7fca42457640 1 --2- 172.21.15.123:0/3588693474 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fca20073070 0x7fca20075530 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7fca3c165880 tx=0x7fca380091e0 comp rx=0 tx=0).ready entity=mgr.4135 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:46:15.477 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:15.473+0000 7fca32ffd640 1 -- 172.21.15.123:0/3588693474 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7fca2c05f390 con 0x7fca3c074430 2024-09-16T15:46:15.672 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:15.669+0000 7fca446bb640 1 -- 172.21.15.123:0/3588693474 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fca3c15f630 con 0x7fca3c074430 2024-09-16T15:46:15.673 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:15.669+0000 7fca32ffd640 1 -- 172.21.15.123:0/3588693474 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 30 v30) ==== 94+0+3914 (secure 0 0 0) 0x7fca2c092e20 con 0x7fca3c074430 2024-09-16T15:46:15.673 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-09-16T15:46:15.673 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":30,"btime":"2024-09-16T15:46:09:190722+0000","default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4712,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6837/288264117","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6836","nonce":288264117},{"type":"v1","addr":"172.21.15.184:6837","nonce":288264117}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":23},{"gid":4859,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6835/1271790919","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":1271790919},{"type":"v1","addr":"172.21.15.184:6835","nonce":1271790919}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":30},{"gid":5268,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6834/60194515","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6832","nonce":60194515},{"type":"v1","addr":"172.21.15.123:6834","nonce":60194515}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":23},{"gid":5271,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6835/1798373479","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6833","nonce":1798373479},{"type":"v1","addr":"172.21.15.123:6835","nonce":1798373479}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":22}],"filesystems":[]} 2024-09-16T15:46:15.673 INFO:teuthology.orchestra.run.smithi123.stderr:dumped fsmap epoch 30 2024-09-16T15:46:15.675 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:15.673+0000 7fca446bb640 1 -- 172.21.15.123:0/3588693474 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fca20073070 msgr2=0x7fca20075530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:46:15.675 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:15.673+0000 7fca446bb640 1 --2- 172.21.15.123:0/3588693474 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fca20073070 0x7fca20075530 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7fca3c165880 tx=0x7fca380091e0 comp rx=0 tx=0).stop 2024-09-16T15:46:15.675 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:15.673+0000 7fca446bb640 1 -- 172.21.15.123:0/3588693474 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fca3c074430 msgr2=0x7fca3c15d6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:46:15.675 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:15.673+0000 7fca446bb640 1 --2- 172.21.15.123:0/3588693474 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fca3c074430 0x7fca3c15d6b0 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7fca2c00d770 tx=0x7fca2c00dc30 comp rx=0 tx=0).stop 2024-09-16T15:46:15.675 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:15.673+0000 7fca446bb640 1 -- 172.21.15.123:0/3588693474 shutdown_connections 2024-09-16T15:46:15.676 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:15.673+0000 7fca446bb640 1 --2- 172.21.15.123:0/3588693474 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fca20073070 0x7fca20075530 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:15.676 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:15.673+0000 7fca446bb640 1 --2- 172.21.15.123:0/3588693474 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fca3c15bca0 0x7fca3c0c74c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:15.676 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:15.673+0000 7fca446bb640 1 --2- 172.21.15.123:0/3588693474 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fca3c074430 0x7fca3c15d6b0 unknown :-1 s=CLOSED pgs=602 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:15.676 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:15.673+0000 7fca446bb640 1 --2- 172.21.15.123:0/3588693474 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fca3c0726c0 0x7fca3c155fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:15.676 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:15.673+0000 7fca446bb640 1 -- 172.21.15.123:0/3588693474 >> 172.21.15.123:0/3588693474 conn(0x7fca3c0cfb10 msgr2=0x7fca3c15a4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:46:15.676 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:15.673+0000 7fca446bb640 1 -- 172.21.15.123:0/3588693474 shutdown_connections 2024-09-16T15:46:15.676 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:15.673+0000 7fca446bb640 1 -- 172.21.15.123:0/3588693474 wait complete. 2024-09-16T15:46:15.691 INFO:tasks.ceph.mds.a:Restarting daemon 2024-09-16T15:46:15.691 INFO:tasks.ceph.mds.a:Stopping old one... 2024-09-16T15:46:15.691 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-09-16T15:46:15.691 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T15:46:15.691 INFO:tasks.ceph.mds.c:Restarting daemon 2024-09-16T15:46:15.691 INFO:tasks.ceph.mds.c:Stopping old one... 2024-09-16T15:46:15.692 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-09-16T15:46:15.692 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T15:46:15.692 INFO:tasks.ceph.mds.b:Restarting daemon 2024-09-16T15:46:15.692 INFO:tasks.ceph.mds.b:Stopping old one... 2024-09-16T15:46:15.692 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-09-16T15:46:15.692 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T15:46:15.692 INFO:tasks.ceph.mds.d:Restarting daemon 2024-09-16T15:46:15.693 INFO:tasks.ceph.mds.d:Stopping old one... 2024-09-16T15:46:15.693 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-09-16T15:46:15.693 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T15:46:15.701 INFO:tasks.ceph.mds.a:Stopped 2024-09-16T15:46:15.702 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-09-16T15:46:15.702 INFO:tasks.ceph.mds.c:Stopped 2024-09-16T15:46:15.702 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-09-16T15:46:15.708 INFO:tasks.ceph.mds.b:Stopped 2024-09-16T15:46:15.708 DEBUG:teuthology.orchestra.run.smithi184:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-09-16T15:46:15.710 INFO:tasks.ceph.mds.d:Stopped 2024-09-16T15:46:15.710 DEBUG:teuthology.orchestra.run.smithi184:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-09-16T15:46:15.744 INFO:tasks.ceph.mds.a:Started 2024-09-16T15:46:15.744 DEBUG:teuthology.parallel:result is None 2024-09-16T15:46:15.745 INFO:tasks.ceph.mds.c:Started 2024-09-16T15:46:15.745 DEBUG:teuthology.parallel:result is None 2024-09-16T15:46:15.755 INFO:tasks.ceph.mds.b:Started 2024-09-16T15:46:15.756 DEBUG:teuthology.parallel:result is None 2024-09-16T15:46:15.757 INFO:tasks.ceph.mds.d:Started 2024-09-16T15:46:15.757 DEBUG:teuthology.parallel:result is None 2024-09-16T15:46:15.757 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-09-16T15:46:15.802 INFO:tasks.ceph.mds.a.smithi123.stdout:starting mds.a at 2024-09-16T15:46:15.803 INFO:tasks.ceph.mds.c.smithi123.stdout:starting mds.c at 2024-09-16T15:46:15.824 INFO:tasks.ceph.mds.d.smithi184.stdout:starting mds.d at 2024-09-16T15:46:15.826 INFO:tasks.ceph.mds.b.smithi184.stdout:starting mds.b at 2024-09-16T15:46:15.878 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:15.873+0000 7f402cf3e640 1 -- 172.21.15.123:0/2059168810 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f40280cdb50 msgr2=0x7f402815ef30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:46:15.878 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:15.873+0000 7f402cf3e640 1 --2- 172.21.15.123:0/2059168810 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f40280cdb50 0x7f402815ef30 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7f4028073c20 tx=0x7f402001cb10 comp rx=0 tx=0).stop 2024-09-16T15:46:15.878 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:15.873+0000 7f402cf3e640 1 -- 172.21.15.123:0/2059168810 shutdown_connections 2024-09-16T15:46:15.878 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:15.873+0000 7f402cf3e640 1 --2- 172.21.15.123:0/2059168810 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f40280cdb50 0x7f402815ef30 unknown :-1 s=CLOSED pgs=611 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:15.878 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:15.873+0000 7f402cf3e640 1 -- 172.21.15.123:0/2059168810 >> 172.21.15.123:0/2059168810 conn(0x7f40280cfb10 msgr2=0x7f40280cff20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:46:15.878 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:15.873+0000 7f402cf3e640 1 -- 172.21.15.123:0/2059168810 shutdown_connections 2024-09-16T15:46:15.878 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:15.873+0000 7f402cf3e640 1 -- 172.21.15.123:0/2059168810 wait complete. 2024-09-16T15:46:15.879 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:15.873+0000 7f402cf3e640 1 Processor -- start 2024-09-16T15:46:15.879 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:15.873+0000 7f402cf3e640 1 -- start start 2024-09-16T15:46:15.881 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:15.877+0000 7f402cf3e640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f40280cd6e0 0x7f402814db30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:46:15.881 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:15.877+0000 7f402cf3e640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f40280cdb50 0x7f40280affb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:46:15.881 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:15.877+0000 7f402cf3e640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f402815e160 0x7f40280b04f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:46:15.881 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:15.877+0000 7f402cf3e640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f40281697f0 con 0x7f402815e160 2024-09-16T15:46:15.881 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:15.877+0000 7f402cf3e640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f40280cf640 con 0x7f40280cdb50 2024-09-16T15:46:15.881 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:15.877+0000 7f402cf3e640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f4028169970 con 0x7f40280cd6e0 2024-09-16T15:46:15.881 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:15.877+0000 7f4026d9d640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f402815e160 0x7f40280b04f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:46:15.881 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:15.877+0000 7f4026d9d640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f402815e160 0x7f40280b04f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:42146/0 (socket says 172.21.15.123:42146) 2024-09-16T15:46:15.881 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:15.877+0000 7f4026d9d640 1 -- 172.21.15.123:0/3913476744 learned_addr learned my addr 172.21.15.123:0/3913476744 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:46:15.881 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:15.877+0000 7f402659c640 1 --2- 172.21.15.123:0/3913476744 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f40280cd6e0 0x7f402814db30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:46:15.881 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:15.877+0000 7f4025d9b640 1 --2- 172.21.15.123:0/3913476744 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f40280cdb50 0x7f40280affb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:46:15.881 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:15.877+0000 7f4026d9d640 1 -- 172.21.15.123:0/3913476744 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f40280cd6e0 msgr2=0x7f402814db30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:46:15.882 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:15.877+0000 7f4026d9d640 1 --2- 172.21.15.123:0/3913476744 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f40280cd6e0 0x7f402814db30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:15.882 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:15.877+0000 7f4026d9d640 1 -- 172.21.15.123:0/3913476744 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f40280cdb50 msgr2=0x7f40280affb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:46:15.882 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:15.877+0000 7f4026d9d640 1 --2- 172.21.15.123:0/3913476744 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f40280cdb50 0x7f40280affb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:15.882 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:15.877+0000 7f4026d9d640 1 -- 172.21.15.123:0/3913476744 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4028151a40 con 0x7f402815e160 2024-09-16T15:46:15.882 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:15.877+0000 7f4025d9b640 1 --2- 172.21.15.123:0/3913476744 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f40280cdb50 0x7f40280affb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T15:46:15.882 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:15.877+0000 7f4026d9d640 1 --2- 172.21.15.123:0/3913476744 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f402815e160 0x7f40280b04f0 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7f4020098610 tx=0x7f40200a5890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:46:15.882 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:15.877+0000 7f401effd640 1 -- 172.21.15.123:0/3913476744 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40200b2040 con 0x7f402815e160 2024-09-16T15:46:15.882 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:15.877+0000 7f401effd640 1 -- 172.21.15.123:0/3913476744 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f40200bb070 con 0x7f402815e160 2024-09-16T15:46:15.882 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:15.877+0000 7f402cf3e640 1 -- 172.21.15.123:0/3913476744 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f40280af390 con 0x7f402815e160 2024-09-16T15:46:15.882 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:15.877+0000 7f401effd640 1 -- 172.21.15.123:0/3913476744 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4020004670 con 0x7f402815e160 2024-09-16T15:46:15.882 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:15.877+0000 7f402cf3e640 1 -- 172.21.15.123:0/3913476744 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f40280aedd0 con 0x7f402815e160 2024-09-16T15:46:15.883 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:15.877+0000 7f402cf3e640 1 -- 172.21.15.123:0/3913476744 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4028073f30 con 0x7f402815e160 2024-09-16T15:46:15.883 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:15.877+0000 7f401effd640 1 -- 172.21.15.123:0/3913476744 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 9) ==== 97233+0+0 (secure 0 0 0) 0x7f40200c0460 con 0x7f402815e160 2024-09-16T15:46:15.883 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:15.877+0000 7f401effd640 1 --2- 172.21.15.123:0/3913476744 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f4008073070 0x7f4008075530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:46:15.883 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:15.877+0000 7f401effd640 1 -- 172.21.15.123:0/3913476744 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(49..49 src has 1..49) ==== 4943+0+0 (secure 0 0 0) 0x7f402012ea40 con 0x7f402815e160 2024-09-16T15:46:15.883 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:15.877+0000 7f402659c640 1 --2- 172.21.15.123:0/3913476744 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f4008073070 0x7f4008075530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:46:15.883 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:15.881+0000 7f402659c640 1 --2- 172.21.15.123:0/3913476744 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f4008073070 0x7f4008075530 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f4018007fd0 tx=0x7f4018009040 comp rx=0 tx=0).ready entity=mgr.4135 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:46:15.886 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:15.881+0000 7f401effd640 1 -- 172.21.15.123:0/3913476744 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f4020016790 con 0x7f402815e160 2024-09-16T15:46:16.072 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:16.069+0000 7f402cf3e640 1 -- 172.21.15.123:0/3913476744 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) -- 0x7f40280d06c0 con 0x7f402815e160 2024-09-16T15:46:17.120 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:17.117+0000 7f401effd640 1 -- 172.21.15.123:0/3913476744 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v54) ==== 97+0+0 (secure 0 0 0) 0x7f402012ee20 con 0x7f402815e160 2024-09-16T15:46:17.120 INFO:teuthology.orchestra.run.smithi123.stderr: removed all blocklist entries 2024-09-16T15:46:17.125 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:17.121+0000 7f402cf3e640 1 -- 172.21.15.123:0/3913476744 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f4008073070 msgr2=0x7f4008075530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:46:17.125 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:17.121+0000 7f402cf3e640 1 --2- 172.21.15.123:0/3913476744 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f4008073070 0x7f4008075530 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f4018007fd0 tx=0x7f4018009040 comp rx=0 tx=0).stop 2024-09-16T15:46:17.125 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:17.121+0000 7f402cf3e640 1 -- 172.21.15.123:0/3913476744 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f402815e160 msgr2=0x7f40280b04f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:46:17.125 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:17.121+0000 7f402cf3e640 1 --2- 172.21.15.123:0/3913476744 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f402815e160 0x7f40280b04f0 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7f4020098610 tx=0x7f40200a5890 comp rx=0 tx=0).stop 2024-09-16T15:46:17.125 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:17.121+0000 7f402cf3e640 1 -- 172.21.15.123:0/3913476744 shutdown_connections 2024-09-16T15:46:17.125 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:17.121+0000 7f402cf3e640 1 --2- 172.21.15.123:0/3913476744 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f4008073070 0x7f4008075530 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:17.125 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:17.121+0000 7f402cf3e640 1 --2- 172.21.15.123:0/3913476744 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f402815e160 0x7f40280b04f0 unknown :-1 s=CLOSED pgs=612 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:17.125 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:17.121+0000 7f402cf3e640 1 --2- 172.21.15.123:0/3913476744 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f40280cdb50 0x7f40280affb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:17.126 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:17.121+0000 7f402cf3e640 1 --2- 172.21.15.123:0/3913476744 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f40280cd6e0 0x7f402814db30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:17.126 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:17.121+0000 7f402cf3e640 1 -- 172.21.15.123:0/3913476744 >> 172.21.15.123:0/3913476744 conn(0x7f40280cfb10 msgr2=0x7f402806f700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:46:17.126 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:17.121+0000 7f402cf3e640 1 -- 172.21.15.123:0/3913476744 shutdown_connections 2024-09-16T15:46:17.126 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:17.121+0000 7f402cf3e640 1 -- 172.21.15.123:0/3913476744 wait complete. 2024-09-16T15:46:17.144 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-09-16T15:46:17.311 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:17.309+0000 7f8e81063640 1 -- 172.21.15.123:0/29471710 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8e7c0cd390 msgr2=0x7f8e7c0cd7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:46:17.311 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:17.309+0000 7f8e81063640 1 --2- 172.21.15.123:0/29471710 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8e7c0cd390 0x7f8e7c0cd7a0 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7f8e6c009840 tx=0x7f8e6c01c7f0 comp rx=0 tx=0).stop 2024-09-16T15:46:17.311 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:17.309+0000 7f8e81063640 1 -- 172.21.15.123:0/29471710 shutdown_connections 2024-09-16T15:46:17.312 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:17.309+0000 7f8e81063640 1 --2- 172.21.15.123:0/29471710 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f8e7c0b0eb0 0x7f8e7c0b12d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:17.312 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:17.309+0000 7f8e81063640 1 --2- 172.21.15.123:0/29471710 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8e7c0cd390 0x7f8e7c0cd7a0 unknown :-1 s=CLOSED pgs=614 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:17.312 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:17.309+0000 7f8e81063640 1 -- 172.21.15.123:0/29471710 >> 172.21.15.123:0/29471710 conn(0x7f8e7c074210 msgr2=0x7f8e7c0745f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:46:17.312 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:17.309+0000 7f8e81063640 1 -- 172.21.15.123:0/29471710 shutdown_connections 2024-09-16T15:46:17.312 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:17.309+0000 7f8e81063640 1 -- 172.21.15.123:0/29471710 wait complete. 2024-09-16T15:46:17.312 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:17.309+0000 7f8e81063640 1 Processor -- start 2024-09-16T15:46:17.312 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:17.309+0000 7f8e81063640 1 -- start start 2024-09-16T15:46:17.314 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:17.309+0000 7f8e81063640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f8e7c0b0eb0 0x7f8e7c0b4e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:46:17.314 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:17.309+0000 7f8e81063640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8e7c0cd390 0x7f8e7c0b53d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:46:17.314 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:17.309+0000 7f8e81063640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f8e7c15dd80 0x7f8e7c0b3180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:46:17.314 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:17.309+0000 7f8e81063640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f8e7c16b0f0 con 0x7f8e7c0cd390 2024-09-16T15:46:17.314 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:17.309+0000 7f8e81063640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f8e7c16af70 con 0x7f8e7c15dd80 2024-09-16T15:46:17.315 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:17.309+0000 7f8e81063640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f8e7c16b270 con 0x7f8e7c0b0eb0 2024-09-16T15:46:17.315 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:17.309+0000 7f8e7a59c640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8e7c0cd390 0x7f8e7c0b53d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:46:17.315 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:17.309+0000 7f8e7a59c640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8e7c0cd390 0x7f8e7c0b53d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:42178/0 (socket says 172.21.15.123:42178) 2024-09-16T15:46:17.315 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:17.309+0000 7f8e7a59c640 1 -- 172.21.15.123:0/803449578 learned_addr learned my addr 172.21.15.123:0/803449578 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:46:17.315 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:17.309+0000 7f8e7b59e640 1 --2- 172.21.15.123:0/803449578 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f8e7c15dd80 0x7f8e7c0b3180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:46:17.315 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:17.309+0000 7f8e7ad9d640 1 --2- 172.21.15.123:0/803449578 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f8e7c0b0eb0 0x7f8e7c0b4e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:46:17.315 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:17.309+0000 7f8e7a59c640 1 -- 172.21.15.123:0/803449578 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f8e7c0b0eb0 msgr2=0x7f8e7c0b4e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:46:17.315 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:17.309+0000 7f8e7a59c640 1 --2- 172.21.15.123:0/803449578 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f8e7c0b0eb0 0x7f8e7c0b4e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:17.315 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:17.309+0000 7f8e7a59c640 1 -- 172.21.15.123:0/803449578 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f8e7c15dd80 msgr2=0x7f8e7c0b3180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:46:17.315 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:17.309+0000 7f8e7a59c640 1 --2- 172.21.15.123:0/803449578 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f8e7c15dd80 0x7f8e7c0b3180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:17.315 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:17.309+0000 7f8e7a59c640 1 -- 172.21.15.123:0/803449578 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8e7c0b5a40 con 0x7f8e7c0cd390 2024-09-16T15:46:17.315 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:17.309+0000 7f8e7a59c640 1 --2- 172.21.15.123:0/803449578 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8e7c0cd390 0x7f8e7c0b53d0 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7f8e6c01aea0 tx=0x7f8e6c0a5840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:46:17.316 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:17.309+0000 7f8e737fe640 1 -- 172.21.15.123:0/803449578 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e6c0b2040 con 0x7f8e7c0cd390 2024-09-16T15:46:17.316 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:17.309+0000 7f8e737fe640 1 -- 172.21.15.123:0/803449578 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f8e6c0044b0 con 0x7f8e7c0cd390 2024-09-16T15:46:17.316 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:17.309+0000 7f8e81063640 1 -- 172.21.15.123:0/803449578 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8e7c15a2c0 con 0x7f8e7c0cd390 2024-09-16T15:46:17.316 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:17.309+0000 7f8e737fe640 1 -- 172.21.15.123:0/803449578 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e6c005260 con 0x7f8e7c0cd390 2024-09-16T15:46:17.316 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:17.309+0000 7f8e81063640 1 -- 172.21.15.123:0/803449578 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8e7c0b4200 con 0x7f8e7c0cd390 2024-09-16T15:46:17.317 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:17.313+0000 7f8e81063640 1 -- 172.21.15.123:0/803449578 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8e7c0cd7a0 con 0x7f8e7c0cd390 2024-09-16T15:46:17.317 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:17.313+0000 7f8e737fe640 1 -- 172.21.15.123:0/803449578 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 9) ==== 97233+0+0 (secure 0 0 0) 0x7f8e6c0be060 con 0x7f8e7c0cd390 2024-09-16T15:46:17.317 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:17.313+0000 7f8e737fe640 1 --2- 172.21.15.123:0/803449578 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f8e5c073070 0x7f8e5c075530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:46:17.317 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:17.313+0000 7f8e737fe640 1 -- 172.21.15.123:0/803449578 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(54..54 src has 1..54) ==== 4857+0+0 (secure 0 0 0) 0x7f8e6c12d750 con 0x7f8e7c0cd390 2024-09-16T15:46:17.317 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:17.313+0000 7f8e7ad9d640 1 --2- 172.21.15.123:0/803449578 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f8e5c073070 0x7f8e5c075530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:46:17.317 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:17.313+0000 7f8e7ad9d640 1 --2- 172.21.15.123:0/803449578 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f8e5c073070 0x7f8e5c075530 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f8e740015d0 tx=0x7f8e74009040 comp rx=0 tx=0).ready entity=mgr.4135 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:46:17.320 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:17.317+0000 7f8e737fe640 1 -- 172.21.15.123:0/803449578 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f8e6c016410 con 0x7f8e7c0cd390 2024-09-16T15:46:17.551 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:17.549+0000 7f8e81063640 1 -- 172.21.15.123:0/803449578 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) -- 0x7f8e7c074da0 con 0x7f8e7c0cd390 2024-09-16T15:46:17.552 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:17.549+0000 7f8e737fe640 1 -- 172.21.15.123:0/803449578 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v18) ==== 80+0+5304 (secure 0 0 0) 0x7f8e6c0f89f0 con 0x7f8e7c0cd390 2024-09-16T15:46:17.552 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-09-16T15:46:17.552 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2024-09-16T15:46:17.553 INFO:teuthology.orchestra.run.smithi123.stdout: "auth_dump": [ 2024-09-16T15:46:17.553 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-09-16T15:46:17.553 INFO:teuthology.orchestra.run.smithi123.stdout: "entity": "mds.a", 2024-09-16T15:46:17.553 INFO:teuthology.orchestra.run.smithi123.stdout: "key": "AQC0UehmbZpOJBAAaYGyDcWrt4k0FS5P/bb+4A==", 2024-09-16T15:46:17.553 INFO:teuthology.orchestra.run.smithi123.stdout: "caps": { 2024-09-16T15:46:17.553 INFO:teuthology.orchestra.run.smithi123.stdout: "mds": "allow", 2024-09-16T15:46:17.553 INFO:teuthology.orchestra.run.smithi123.stdout: "mgr": "allow *", 2024-09-16T15:46:17.553 INFO:teuthology.orchestra.run.smithi123.stdout: "mon": "allow *", 2024-09-16T15:46:17.553 INFO:teuthology.orchestra.run.smithi123.stdout: "osd": "allow *" 2024-09-16T15:46:17.553 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-16T15:46:17.553 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:46:17.553 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-09-16T15:46:17.553 INFO:teuthology.orchestra.run.smithi123.stdout: "entity": "mds.b", 2024-09-16T15:46:17.553 INFO:teuthology.orchestra.run.smithi123.stdout: "key": "AQC0UehmF1HFMhAADcvqXNKdHZMPAbLc+se4Aw==", 2024-09-16T15:46:17.554 INFO:teuthology.orchestra.run.smithi123.stdout: "caps": { 2024-09-16T15:46:17.554 INFO:teuthology.orchestra.run.smithi123.stdout: "mds": "allow", 2024-09-16T15:46:17.554 INFO:teuthology.orchestra.run.smithi123.stdout: "mgr": "allow *", 2024-09-16T15:46:17.554 INFO:teuthology.orchestra.run.smithi123.stdout: "mon": "allow *", 2024-09-16T15:46:17.554 INFO:teuthology.orchestra.run.smithi123.stdout: "osd": "allow *" 2024-09-16T15:46:17.554 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-16T15:46:17.554 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:46:17.554 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-09-16T15:46:17.554 INFO:teuthology.orchestra.run.smithi123.stdout: "entity": "mds.c", 2024-09-16T15:46:17.554 INFO:teuthology.orchestra.run.smithi123.stdout: "key": "AQC0UehmCBjCLBAAXd85J25r7cgrXeDCG7pahg==", 2024-09-16T15:46:17.554 INFO:teuthology.orchestra.run.smithi123.stdout: "caps": { 2024-09-16T15:46:17.554 INFO:teuthology.orchestra.run.smithi123.stdout: "mds": "allow", 2024-09-16T15:46:17.554 INFO:teuthology.orchestra.run.smithi123.stdout: "mgr": "allow *", 2024-09-16T15:46:17.554 INFO:teuthology.orchestra.run.smithi123.stdout: "mon": "allow *", 2024-09-16T15:46:17.554 INFO:teuthology.orchestra.run.smithi123.stdout: "osd": "allow *" 2024-09-16T15:46:17.555 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-16T15:46:17.555 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:46:17.555 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-09-16T15:46:17.555 INFO:teuthology.orchestra.run.smithi123.stdout: "entity": "mds.d", 2024-09-16T15:46:17.555 INFO:teuthology.orchestra.run.smithi123.stdout: "key": "AQC0UehmMh3ZOBAAx0RCa3R9RMT93/icZhPGyg==", 2024-09-16T15:46:17.555 INFO:teuthology.orchestra.run.smithi123.stdout: "caps": { 2024-09-16T15:46:17.555 INFO:teuthology.orchestra.run.smithi123.stdout: "mds": "allow", 2024-09-16T15:46:17.555 INFO:teuthology.orchestra.run.smithi123.stdout: "mgr": "allow *", 2024-09-16T15:46:17.555 INFO:teuthology.orchestra.run.smithi123.stdout: "mon": "allow *", 2024-09-16T15:46:17.555 INFO:teuthology.orchestra.run.smithi123.stdout: "osd": "allow *" 2024-09-16T15:46:17.555 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-16T15:46:17.555 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:46:17.555 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-09-16T15:46:17.555 INFO:teuthology.orchestra.run.smithi123.stdout: "entity": "osd.0", 2024-09-16T15:46:17.555 INFO:teuthology.orchestra.run.smithi123.stdout: "key": "AQC4UehmT633CRAAIhpkeIBOzwJO0kiXLu0x9Q==", 2024-09-16T15:46:17.555 INFO:teuthology.orchestra.run.smithi123.stdout: "caps": { 2024-09-16T15:46:17.556 INFO:teuthology.orchestra.run.smithi123.stdout: "mgr": "allow profile osd", 2024-09-16T15:46:17.556 INFO:teuthology.orchestra.run.smithi123.stdout: "mon": "allow profile osd", 2024-09-16T15:46:17.556 INFO:teuthology.orchestra.run.smithi123.stdout: "osd": "allow *" 2024-09-16T15:46:17.556 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-16T15:46:17.556 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:46:17.556 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-09-16T15:46:17.556 INFO:teuthology.orchestra.run.smithi123.stdout: "entity": "osd.1", 2024-09-16T15:46:17.556 INFO:teuthology.orchestra.run.smithi123.stdout: "key": "AQC7UehmFwF7GBAAvOUG+rWjANKT8wm/ai4xGg==", 2024-09-16T15:46:17.556 INFO:teuthology.orchestra.run.smithi123.stdout: "caps": { 2024-09-16T15:46:17.556 INFO:teuthology.orchestra.run.smithi123.stdout: "mgr": "allow profile osd", 2024-09-16T15:46:17.556 INFO:teuthology.orchestra.run.smithi123.stdout: "mon": "allow profile osd", 2024-09-16T15:46:17.556 INFO:teuthology.orchestra.run.smithi123.stdout: "osd": "allow *" 2024-09-16T15:46:17.556 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-16T15:46:17.556 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:46:17.557 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-09-16T15:46:17.557 INFO:teuthology.orchestra.run.smithi123.stdout: "entity": "osd.2", 2024-09-16T15:46:17.557 INFO:teuthology.orchestra.run.smithi123.stdout: "key": "AQC+UehmlCwWKxAAl3iO7ZALWyaorQH2urHKCw==", 2024-09-16T15:46:17.557 INFO:teuthology.orchestra.run.smithi123.stdout: "caps": { 2024-09-16T15:46:17.557 INFO:teuthology.orchestra.run.smithi123.stdout: "mgr": "allow profile osd", 2024-09-16T15:46:17.557 INFO:teuthology.orchestra.run.smithi123.stdout: "mon": "allow profile osd", 2024-09-16T15:46:17.557 INFO:teuthology.orchestra.run.smithi123.stdout: "osd": "allow *" 2024-09-16T15:46:17.557 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-16T15:46:17.557 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:46:17.557 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-09-16T15:46:17.557 INFO:teuthology.orchestra.run.smithi123.stdout: "entity": "osd.3", 2024-09-16T15:46:17.557 INFO:teuthology.orchestra.run.smithi123.stdout: "key": "AQDCUehmD8VgAhAAneoIeGr3MflSmi+nyavj2A==", 2024-09-16T15:46:17.557 INFO:teuthology.orchestra.run.smithi123.stdout: "caps": { 2024-09-16T15:46:17.558 INFO:teuthology.orchestra.run.smithi123.stdout: "mgr": "allow profile osd", 2024-09-16T15:46:17.558 INFO:teuthology.orchestra.run.smithi123.stdout: "mon": "allow profile osd", 2024-09-16T15:46:17.558 INFO:teuthology.orchestra.run.smithi123.stdout: "osd": "allow *" 2024-09-16T15:46:17.558 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-16T15:46:17.558 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:46:17.558 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-09-16T15:46:17.558 INFO:teuthology.orchestra.run.smithi123.stdout: "entity": "osd.4", 2024-09-16T15:46:17.558 INFO:teuthology.orchestra.run.smithi123.stdout: "key": "AQDIUehmA9B9EBAAsu0X5LPx1TmatZeHzjnVig==", 2024-09-16T15:46:17.558 INFO:teuthology.orchestra.run.smithi123.stdout: "caps": { 2024-09-16T15:46:17.558 INFO:teuthology.orchestra.run.smithi123.stdout: "mgr": "allow profile osd", 2024-09-16T15:46:17.558 INFO:teuthology.orchestra.run.smithi123.stdout: "mon": "allow profile osd", 2024-09-16T15:46:17.558 INFO:teuthology.orchestra.run.smithi123.stdout: "osd": "allow *" 2024-09-16T15:46:17.558 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-16T15:46:17.558 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:46:17.558 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-09-16T15:46:17.559 INFO:teuthology.orchestra.run.smithi123.stdout: "entity": "osd.5", 2024-09-16T15:46:17.559 INFO:teuthology.orchestra.run.smithi123.stdout: "key": "AQDLUehmcugdHxAAQMSS1ZH71FUzCx5A+yNnZg==", 2024-09-16T15:46:17.559 INFO:teuthology.orchestra.run.smithi123.stdout: "caps": { 2024-09-16T15:46:17.559 INFO:teuthology.orchestra.run.smithi123.stdout: "mgr": "allow profile osd", 2024-09-16T15:46:17.559 INFO:teuthology.orchestra.run.smithi123.stdout: "mon": "allow profile osd", 2024-09-16T15:46:17.559 INFO:teuthology.orchestra.run.smithi123.stdout: "osd": "allow *" 2024-09-16T15:46:17.559 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-16T15:46:17.559 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:46:17.559 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-09-16T15:46:17.559 INFO:teuthology.orchestra.run.smithi123.stdout: "entity": "osd.6", 2024-09-16T15:46:17.559 INFO:teuthology.orchestra.run.smithi123.stdout: "key": "AQDOUehmWLwYLhAAhOlQLepcWElsVPMnHlZm4w==", 2024-09-16T15:46:17.559 INFO:teuthology.orchestra.run.smithi123.stdout: "caps": { 2024-09-16T15:46:17.559 INFO:teuthology.orchestra.run.smithi123.stdout: "mgr": "allow profile osd", 2024-09-16T15:46:17.559 INFO:teuthology.orchestra.run.smithi123.stdout: "mon": "allow profile osd", 2024-09-16T15:46:17.559 INFO:teuthology.orchestra.run.smithi123.stdout: "osd": "allow *" 2024-09-16T15:46:17.560 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-16T15:46:17.560 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:46:17.560 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-09-16T15:46:17.560 INFO:teuthology.orchestra.run.smithi123.stdout: "entity": "osd.7", 2024-09-16T15:46:17.560 INFO:teuthology.orchestra.run.smithi123.stdout: "key": "AQDSUehmipkbABAAQuL/2O7w0tsM1OKeOil97w==", 2024-09-16T15:46:17.560 INFO:teuthology.orchestra.run.smithi123.stdout: "caps": { 2024-09-16T15:46:17.560 INFO:teuthology.orchestra.run.smithi123.stdout: "mgr": "allow profile osd", 2024-09-16T15:46:17.560 INFO:teuthology.orchestra.run.smithi123.stdout: "mon": "allow profile osd", 2024-09-16T15:46:17.560 INFO:teuthology.orchestra.run.smithi123.stdout: "osd": "allow *" 2024-09-16T15:46:17.560 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-16T15:46:17.560 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:46:17.560 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-09-16T15:46:17.560 INFO:teuthology.orchestra.run.smithi123.stdout: "entity": "client.0", 2024-09-16T15:46:17.560 INFO:teuthology.orchestra.run.smithi123.stdout: "key": "AQC1Uehm7GDTAhAAHWWVIyjVVnTeeozBR8DfFA==", 2024-09-16T15:46:17.561 INFO:teuthology.orchestra.run.smithi123.stdout: "caps": { 2024-09-16T15:46:17.561 INFO:teuthology.orchestra.run.smithi123.stdout: "mds": "allow", 2024-09-16T15:46:17.561 INFO:teuthology.orchestra.run.smithi123.stdout: "mon": "allow r", 2024-09-16T15:46:17.561 INFO:teuthology.orchestra.run.smithi123.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-09-16T15:46:17.561 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-16T15:46:17.561 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:46:17.561 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-09-16T15:46:17.561 INFO:teuthology.orchestra.run.smithi123.stdout: "entity": "client.1", 2024-09-16T15:46:17.561 INFO:teuthology.orchestra.run.smithi123.stdout: "key": "AQC1UehmFbaWChAAFtRmKucm+pbKtCNvRCTNqA==", 2024-09-16T15:46:17.561 INFO:teuthology.orchestra.run.smithi123.stdout: "caps": { 2024-09-16T15:46:17.561 INFO:teuthology.orchestra.run.smithi123.stdout: "mds": "allow", 2024-09-16T15:46:17.561 INFO:teuthology.orchestra.run.smithi123.stdout: "mgr": "allow r", 2024-09-16T15:46:17.561 INFO:teuthology.orchestra.run.smithi123.stdout: "mon": "allow rw", 2024-09-16T15:46:17.561 INFO:teuthology.orchestra.run.smithi123.stdout: "osd": "allow rwx" 2024-09-16T15:46:17.561 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-16T15:46:17.562 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:46:17.562 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-09-16T15:46:17.562 INFO:teuthology.orchestra.run.smithi123.stdout: "entity": "client.2", 2024-09-16T15:46:17.562 INFO:teuthology.orchestra.run.smithi123.stdout: "key": "AQC1Uehm1ADrBxAAW/8ayNHI6Wq/YlI4ivFZiw==", 2024-09-16T15:46:17.562 INFO:teuthology.orchestra.run.smithi123.stdout: "caps": { 2024-09-16T15:46:17.562 INFO:teuthology.orchestra.run.smithi123.stdout: "mds": "allow", 2024-09-16T15:46:17.562 INFO:teuthology.orchestra.run.smithi123.stdout: "mgr": "allow r", 2024-09-16T15:46:17.562 INFO:teuthology.orchestra.run.smithi123.stdout: "mon": "allow rw", 2024-09-16T15:46:17.562 INFO:teuthology.orchestra.run.smithi123.stdout: "osd": "allow rwx" 2024-09-16T15:46:17.562 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-16T15:46:17.562 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:46:17.562 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-09-16T15:46:17.562 INFO:teuthology.orchestra.run.smithi123.stdout: "entity": "client.3", 2024-09-16T15:46:17.562 INFO:teuthology.orchestra.run.smithi123.stdout: "key": "AQC1UehmLv/3DxAAuGloBxSuza95uajpfiZ9MQ==", 2024-09-16T15:46:17.563 INFO:teuthology.orchestra.run.smithi123.stdout: "caps": { 2024-09-16T15:46:17.563 INFO:teuthology.orchestra.run.smithi123.stdout: "mds": "allow", 2024-09-16T15:46:17.563 INFO:teuthology.orchestra.run.smithi123.stdout: "mgr": "allow r", 2024-09-16T15:46:17.563 INFO:teuthology.orchestra.run.smithi123.stdout: "mon": "allow rw", 2024-09-16T15:46:17.563 INFO:teuthology.orchestra.run.smithi123.stdout: "osd": "allow rwx" 2024-09-16T15:46:17.563 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-16T15:46:17.563 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:46:17.563 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-09-16T15:46:17.563 INFO:teuthology.orchestra.run.smithi123.stdout: "entity": "client.admin", 2024-09-16T15:46:17.563 INFO:teuthology.orchestra.run.smithi123.stdout: "key": "AQC0Uehm1ZDRDBAAhyQ1BmJg1Mw6wQ04I2CbFA==", 2024-09-16T15:46:17.563 INFO:teuthology.orchestra.run.smithi123.stdout: "caps": { 2024-09-16T15:46:17.563 INFO:teuthology.orchestra.run.smithi123.stdout: "mds": "allow *", 2024-09-16T15:46:17.563 INFO:teuthology.orchestra.run.smithi123.stdout: "mgr": "allow *", 2024-09-16T15:46:17.564 INFO:teuthology.orchestra.run.smithi123.stdout: "mon": "allow *", 2024-09-16T15:46:17.564 INFO:teuthology.orchestra.run.smithi123.stdout: "osd": "allow *" 2024-09-16T15:46:17.564 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-16T15:46:17.564 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:46:17.564 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-09-16T15:46:17.564 INFO:teuthology.orchestra.run.smithi123.stdout: "entity": "mgr.x", 2024-09-16T15:46:17.564 INFO:teuthology.orchestra.run.smithi123.stdout: "key": "AQC0UehmfPilIBAAERlBbduOgAU9ba0n/1tl3g==", 2024-09-16T15:46:17.564 INFO:teuthology.orchestra.run.smithi123.stdout: "caps": { 2024-09-16T15:46:17.564 INFO:teuthology.orchestra.run.smithi123.stdout: "mds": "allow *", 2024-09-16T15:46:17.564 INFO:teuthology.orchestra.run.smithi123.stdout: "mon": "allow profile mgr", 2024-09-16T15:46:17.564 INFO:teuthology.orchestra.run.smithi123.stdout: "osd": "allow *" 2024-09-16T15:46:17.564 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-16T15:46:17.564 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:46:17.564 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-09-16T15:46:17.564 INFO:teuthology.orchestra.run.smithi123.stdout: "entity": "mgr.y", 2024-09-16T15:46:17.565 INFO:teuthology.orchestra.run.smithi123.stdout: "key": "AQC0UehmkjrlHRAA+3xbQC+SPWGWW1gND7xcAg==", 2024-09-16T15:46:17.565 INFO:teuthology.orchestra.run.smithi123.stdout: "caps": { 2024-09-16T15:46:17.565 INFO:teuthology.orchestra.run.smithi123.stdout: "mds": "allow *", 2024-09-16T15:46:17.565 INFO:teuthology.orchestra.run.smithi123.stdout: "mon": "allow profile mgr", 2024-09-16T15:46:17.565 INFO:teuthology.orchestra.run.smithi123.stdout: "osd": "allow *" 2024-09-16T15:46:17.565 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-16T15:46:17.565 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-16T15:46:17.565 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-09-16T15:46:17.565 INFO:teuthology.orchestra.run.smithi123.stdout:} 2024-09-16T15:46:17.566 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:17.549+0000 7f8e81063640 1 -- 172.21.15.123:0/803449578 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f8e5c073070 msgr2=0x7f8e5c075530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:46:17.566 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:17.549+0000 7f8e81063640 1 --2- 172.21.15.123:0/803449578 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f8e5c073070 0x7f8e5c075530 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f8e740015d0 tx=0x7f8e74009040 comp rx=0 tx=0).stop 2024-09-16T15:46:17.566 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:17.549+0000 7f8e81063640 1 -- 172.21.15.123:0/803449578 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8e7c0cd390 msgr2=0x7f8e7c0b53d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:46:17.566 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:17.549+0000 7f8e81063640 1 --2- 172.21.15.123:0/803449578 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8e7c0cd390 0x7f8e7c0b53d0 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7f8e6c01aea0 tx=0x7f8e6c0a5840 comp rx=0 tx=0).stop 2024-09-16T15:46:17.566 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:17.549+0000 7f8e81063640 1 -- 172.21.15.123:0/803449578 shutdown_connections 2024-09-16T15:46:17.566 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:17.549+0000 7f8e81063640 1 --2- 172.21.15.123:0/803449578 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f8e5c073070 0x7f8e5c075530 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:17.566 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:17.549+0000 7f8e81063640 1 --2- 172.21.15.123:0/803449578 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f8e7c15dd80 0x7f8e7c0b3180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:17.567 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:17.549+0000 7f8e81063640 1 --2- 172.21.15.123:0/803449578 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8e7c0cd390 0x7f8e7c0b53d0 unknown :-1 s=CLOSED pgs=615 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:17.567 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:17.549+0000 7f8e81063640 1 --2- 172.21.15.123:0/803449578 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f8e7c0b0eb0 0x7f8e7c0b4e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:17.567 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:17.549+0000 7f8e81063640 1 -- 172.21.15.123:0/803449578 >> 172.21.15.123:0/803449578 conn(0x7f8e7c074210 msgr2=0x7f8e7c0cfd80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:46:17.567 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:17.553+0000 7f8e81063640 1 -- 172.21.15.123:0/803449578 shutdown_connections 2024-09-16T15:46:17.567 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:17.553+0000 7f8e81063640 1 -- 172.21.15.123:0/803449578 wait complete. 2024-09-16T15:46:17.570 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-09-16T15:46:17.683 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:17.681+0000 7fa61b8be640 1 -- 172.21.15.123:0/3957675270 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa6140cdb50 msgr2=0x7fa61415ef30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:46:17.683 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:17.681+0000 7fa61b8be640 1 --2- 172.21.15.123:0/3957675270 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa6140cdb50 0x7fa61415ef30 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7fa614073c20 tx=0x7fa60c01c9a0 comp rx=0 tx=0).stop 2024-09-16T15:46:17.683 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:17.681+0000 7fa61b8be640 1 -- 172.21.15.123:0/3957675270 shutdown_connections 2024-09-16T15:46:17.684 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:17.681+0000 7fa61b8be640 1 --2- 172.21.15.123:0/3957675270 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa6140cdb50 0x7fa61415ef30 unknown :-1 s=CLOSED pgs=617 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:17.684 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:17.681+0000 7fa61b8be640 1 -- 172.21.15.123:0/3957675270 >> 172.21.15.123:0/3957675270 conn(0x7fa6140cfb10 msgr2=0x7fa6140cff20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:46:17.684 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:17.681+0000 7fa61b8be640 1 -- 172.21.15.123:0/3957675270 shutdown_connections 2024-09-16T15:46:17.684 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:17.681+0000 7fa61b8be640 1 -- 172.21.15.123:0/3957675270 wait complete. 2024-09-16T15:46:17.684 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:17.681+0000 7fa61b8be640 1 Processor -- start 2024-09-16T15:46:17.684 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:17.681+0000 7fa61b8be640 1 -- start start 2024-09-16T15:46:17.686 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:17.681+0000 7fa61b8be640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fa6140cd6e0 0x7fa61414cf90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:46:17.686 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:17.681+0000 7fa61b8be640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa6140cdb50 0x7fa614159e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:46:17.686 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:17.681+0000 7fa61b8be640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa61415e160 0x7fa614157540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:46:17.687 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:17.681+0000 7fa61b8be640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fa6141697f0 con 0x7fa61415e160 2024-09-16T15:46:17.687 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:17.681+0000 7fa61b8be640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fa6140cf640 con 0x7fa6140cdb50 2024-09-16T15:46:17.687 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:17.681+0000 7fa61b8be640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fa614169970 con 0x7fa6140cd6e0 2024-09-16T15:46:17.687 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:17.681+0000 7fa619e5b640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa61415e160 0x7fa614157540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:46:17.687 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:17.681+0000 7fa619e5b640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa61415e160 0x7fa614157540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:42198/0 (socket says 172.21.15.123:42198) 2024-09-16T15:46:17.687 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:17.681+0000 7fa619e5b640 1 -- 172.21.15.123:0/472460526 learned_addr learned my addr 172.21.15.123:0/472460526 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:46:17.687 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:17.681+0000 7fa61965a640 1 --2- 172.21.15.123:0/472460526 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fa6140cd6e0 0x7fa61414cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:46:17.687 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:17.681+0000 7fa618e59640 1 --2- 172.21.15.123:0/472460526 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa6140cdb50 0x7fa614159e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:46:17.687 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:17.681+0000 7fa619e5b640 1 -- 172.21.15.123:0/472460526 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fa6140cd6e0 msgr2=0x7fa61414cf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:46:17.687 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:17.681+0000 7fa619e5b640 1 --2- 172.21.15.123:0/472460526 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fa6140cd6e0 0x7fa61414cf90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:17.687 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:17.681+0000 7fa619e5b640 1 -- 172.21.15.123:0/472460526 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa6140cdb50 msgr2=0x7fa614159e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:46:17.687 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:17.681+0000 7fa619e5b640 1 --2- 172.21.15.123:0/472460526 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa6140cdb50 0x7fa614159e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:17.688 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:17.681+0000 7fa619e5b640 1 -- 172.21.15.123:0/472460526 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa61415a3c0 con 0x7fa61415e160 2024-09-16T15:46:17.688 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:17.681+0000 7fa618e59640 1 --2- 172.21.15.123:0/472460526 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa6140cdb50 0x7fa614159e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T15:46:17.688 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:17.681+0000 7fa619e5b640 1 --2- 172.21.15.123:0/472460526 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa61415e160 0x7fa614157540 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7fa60c01d240 tx=0x7fa60c003a10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:46:17.688 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:17.681+0000 7fa609ffb640 1 -- 172.21.15.123:0/472460526 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa60c0bb070 con 0x7fa61415e160 2024-09-16T15:46:17.688 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:17.681+0000 7fa609ffb640 1 -- 172.21.15.123:0/472460526 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fa60c004e00 con 0x7fa61415e160 2024-09-16T15:46:17.688 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:17.681+0000 7fa61b8be640 1 -- 172.21.15.123:0/472460526 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa61414c1d0 con 0x7fa61415e160 2024-09-16T15:46:17.688 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:17.681+0000 7fa609ffb640 1 -- 172.21.15.123:0/472460526 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa60c0b2040 con 0x7fa61415e160 2024-09-16T15:46:17.688 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:17.681+0000 7fa61b8be640 1 -- 172.21.15.123:0/472460526 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa61414c790 con 0x7fa61415e160 2024-09-16T15:46:17.688 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:17.681+0000 7fa61b8be640 1 -- 172.21.15.123:0/472460526 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa614073f30 con 0x7fa61415e160 2024-09-16T15:46:17.689 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:17.685+0000 7fa609ffb640 1 -- 172.21.15.123:0/472460526 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 9) ==== 97233+0+0 (secure 0 0 0) 0x7fa60c0be070 con 0x7fa61415e160 2024-09-16T15:46:17.689 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:17.685+0000 7fa609ffb640 1 --2- 172.21.15.123:0/472460526 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fa5f8073070 0x7fa5f8075530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:46:17.689 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:17.685+0000 7fa609ffb640 1 -- 172.21.15.123:0/472460526 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(54..54 src has 1..54) ==== 4857+0+0 (secure 0 0 0) 0x7fa60c12d950 con 0x7fa61415e160 2024-09-16T15:46:17.689 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:17.685+0000 7fa61965a640 1 --2- 172.21.15.123:0/472460526 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fa5f8073070 0x7fa5f8075530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:46:17.689 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:17.685+0000 7fa61965a640 1 --2- 172.21.15.123:0/472460526 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fa5f8073070 0x7fa5f8075530 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7fa61000a750 tx=0x7fa610008040 comp rx=0 tx=0).ready entity=mgr.4135 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:46:17.692 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:17.689+0000 7fa609ffb640 1 -- 172.21.15.123:0/472460526 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7fa60c0166b0 con 0x7fa61415e160 2024-09-16T15:46:17.850 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:17.845+0000 7fa61b8be640 1 -- 172.21.15.123:0/472460526 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) -- 0x7fa6140d06c0 con 0x7fa61415e160 2024-09-16T15:46:17.851 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:17.849+0000 7fa609ffb640 1 -- 172.21.15.123:0/472460526 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v54) ==== 84+0+65 (secure 0 0 0) 0x7fa60c0f8bf0 con 0x7fa61415e160 2024-09-16T15:46:17.851 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-09-16T15:46:17.851 INFO:teuthology.orchestra.run.smithi123.stdout:[ 2024-09-16T15:46:17.851 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-09-16T15:46:17.851 INFO:teuthology.orchestra.run.smithi123.stdout: "poolnum": 1, 2024-09-16T15:46:17.851 INFO:teuthology.orchestra.run.smithi123.stdout: "poolname": ".mgr" 2024-09-16T15:46:17.851 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-16T15:46:17.852 INFO:teuthology.orchestra.run.smithi123.stdout:] 2024-09-16T15:46:17.853 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:17.849+0000 7fa61b8be640 1 -- 172.21.15.123:0/472460526 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fa5f8073070 msgr2=0x7fa5f8075530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:46:17.853 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:17.849+0000 7fa61b8be640 1 --2- 172.21.15.123:0/472460526 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fa5f8073070 0x7fa5f8075530 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7fa61000a750 tx=0x7fa610008040 comp rx=0 tx=0).stop 2024-09-16T15:46:17.854 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:17.849+0000 7fa61b8be640 1 -- 172.21.15.123:0/472460526 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa61415e160 msgr2=0x7fa614157540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:46:17.854 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:17.849+0000 7fa61b8be640 1 --2- 172.21.15.123:0/472460526 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa61415e160 0x7fa614157540 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7fa60c01d240 tx=0x7fa60c003a10 comp rx=0 tx=0).stop 2024-09-16T15:46:17.854 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:17.849+0000 7fa61b8be640 1 -- 172.21.15.123:0/472460526 shutdown_connections 2024-09-16T15:46:17.854 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:17.849+0000 7fa61b8be640 1 --2- 172.21.15.123:0/472460526 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fa5f8073070 0x7fa5f8075530 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:17.854 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:17.849+0000 7fa61b8be640 1 --2- 172.21.15.123:0/472460526 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa61415e160 0x7fa614157540 unknown :-1 s=CLOSED pgs=618 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:17.854 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:17.849+0000 7fa61b8be640 1 --2- 172.21.15.123:0/472460526 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa6140cdb50 0x7fa614159e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:17.854 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:17.849+0000 7fa61b8be640 1 --2- 172.21.15.123:0/472460526 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fa6140cd6e0 0x7fa61414cf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:17.854 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:17.849+0000 7fa61b8be640 1 -- 172.21.15.123:0/472460526 >> 172.21.15.123:0/472460526 conn(0x7fa6140cfb10 msgr2=0x7fa6140b0e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:46:17.854 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:17.849+0000 7fa61b8be640 1 -- 172.21.15.123:0/472460526 shutdown_connections 2024-09-16T15:46:17.854 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:17.849+0000 7fa61b8be640 1 -- 172.21.15.123:0/472460526 wait complete. 2024-09-16T15:46:17.869 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-09-16T15:46:17.869 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-09-16T15:46:17.986 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:17.981+0000 7f322bbfd640 1 -- 172.21.15.123:0/3418827101 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f32240b3260 msgr2=0x7f32240b36f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:46:17.986 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:17.981+0000 7f322bbfd640 1 --2- 172.21.15.123:0/3418827101 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f32240b3260 0x7f32240b36f0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f321c009bc0 tx=0x7f321c01cb10 comp rx=0 tx=0).stop 2024-09-16T15:46:17.987 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:17.981+0000 7f322bbfd640 1 -- 172.21.15.123:0/3418827101 shutdown_connections 2024-09-16T15:46:17.987 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:17.981+0000 7f322bbfd640 1 --2- 172.21.15.123:0/3418827101 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f32240b3260 0x7f32240b36f0 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:17.987 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:17.981+0000 7f322bbfd640 1 --2- 172.21.15.123:0/3418827101 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f322406ff30 0x7f3224070390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:17.987 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:17.981+0000 7f322bbfd640 1 -- 172.21.15.123:0/3418827101 >> 172.21.15.123:0/3418827101 conn(0x7f32240b1630 msgr2=0x7f32240b1a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:46:17.987 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:17.981+0000 7f322bbfd640 1 -- 172.21.15.123:0/3418827101 shutdown_connections 2024-09-16T15:46:17.987 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:17.981+0000 7f322bbfd640 1 -- 172.21.15.123:0/3418827101 wait complete. 2024-09-16T15:46:17.987 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:17.985+0000 7f322bbfd640 1 Processor -- start 2024-09-16T15:46:17.987 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:17.985+0000 7f322bbfd640 1 -- start start 2024-09-16T15:46:17.989 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:17.985+0000 7f322bbfd640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f322406ff30 0x7f32241f9060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:46:17.989 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:17.985+0000 7f322bbfd640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3224073c20 0x7f32241490a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:46:17.989 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:17.985+0000 7f322bbfd640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f32240b3260 0x7f32240cb3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:46:17.989 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:17.985+0000 7f322bbfd640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f3224167eb0 con 0x7f3224073c20 2024-09-16T15:46:17.990 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:17.985+0000 7f322bbfd640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f3224167d30 con 0x7f322406ff30 2024-09-16T15:46:17.990 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:17.985+0000 7f322bbfd640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f3224168030 con 0x7f32240b3260 2024-09-16T15:46:17.990 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:17.985+0000 7f3229198640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3224073c20 0x7f32241490a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:46:17.990 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:17.985+0000 7f3229198640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3224073c20 0x7f32241490a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:42200/0 (socket says 172.21.15.123:42200) 2024-09-16T15:46:17.990 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:17.985+0000 7f3229198640 1 -- 172.21.15.123:0/1865213815 learned_addr learned my addr 172.21.15.123:0/1865213815 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:46:17.990 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:17.985+0000 7f322a19a640 1 --2- 172.21.15.123:0/1865213815 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f32240b3260 0x7f32240cb3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:46:17.990 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:17.985+0000 7f3229999640 1 --2- 172.21.15.123:0/1865213815 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f322406ff30 0x7f32241f9060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:46:17.990 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:17.985+0000 7f3229198640 1 -- 172.21.15.123:0/1865213815 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f32240b3260 msgr2=0x7f32240cb3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:46:17.990 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:17.985+0000 7f3229198640 1 --2- 172.21.15.123:0/1865213815 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f32240b3260 0x7f32240cb3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:17.990 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:17.985+0000 7f3229198640 1 -- 172.21.15.123:0/1865213815 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f322406ff30 msgr2=0x7f32241f9060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:46:17.990 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:17.985+0000 7f3229198640 1 --2- 172.21.15.123:0/1865213815 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f322406ff30 0x7f32241f9060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:17.990 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:17.985+0000 7f3229198640 1 -- 172.21.15.123:0/1865213815 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f32241495e0 con 0x7f3224073c20 2024-09-16T15:46:17.990 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:17.985+0000 7f3229198640 1 --2- 172.21.15.123:0/1865213815 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3224073c20 0x7f32241490a0 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7f321400e9f0 tx=0x7f321400eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:46:17.991 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:17.985+0000 7f321a7fc640 1 -- 172.21.15.123:0/1865213815 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f321400cd80 con 0x7f3224073c20 2024-09-16T15:46:17.991 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:17.985+0000 7f321a7fc640 1 -- 172.21.15.123:0/1865213815 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f3214010430 con 0x7f3224073c20 2024-09-16T15:46:17.991 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:17.985+0000 7f322bbfd640 1 -- 172.21.15.123:0/1865213815 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f32241537e0 con 0x7f3224073c20 2024-09-16T15:46:17.991 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:17.985+0000 7f321a7fc640 1 -- 172.21.15.123:0/1865213815 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3214010860 con 0x7f3224073c20 2024-09-16T15:46:17.991 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:17.985+0000 7f322bbfd640 1 -- 172.21.15.123:0/1865213815 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3224153da0 con 0x7f3224073c20 2024-09-16T15:46:17.992 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:17.985+0000 7f322bbfd640 1 -- 172.21.15.123:0/1865213815 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f32241545a0 con 0x7f3224073c20 2024-09-16T15:46:17.992 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:17.989+0000 7f321a7fc640 1 -- 172.21.15.123:0/1865213815 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 9) ==== 97233+0+0 (secure 0 0 0) 0x7f3214010a00 con 0x7f3224073c20 2024-09-16T15:46:17.992 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:17.989+0000 7f321a7fc640 1 --2- 172.21.15.123:0/1865213815 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f3208073070 0x7f3208075530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:46:17.992 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:17.989+0000 7f321a7fc640 1 -- 172.21.15.123:0/1865213815 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(54..54 src has 1..54) ==== 4857+0+0 (secure 0 0 0) 0x7f3214094a80 con 0x7f3224073c20 2024-09-16T15:46:17.992 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:17.989+0000 7f3229999640 1 --2- 172.21.15.123:0/1865213815 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f3208073070 0x7f3208075530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:46:17.992 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:17.989+0000 7f3229999640 1 --2- 172.21.15.123:0/1865213815 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f3208073070 0x7f3208075530 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f3220001690 tx=0x7f3220009040 comp rx=0 tx=0).ready entity=mgr.4135 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:46:17.995 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:17.993+0000 7f321a7fc640 1 -- 172.21.15.123:0/1865213815 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f321405fd60 con 0x7f3224073c20 2024-09-16T15:46:18.165 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:18.161+0000 7f322bbfd640 1 -- 172.21.15.123:0/1865213815 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) -- 0x7f3224070390 con 0x7f3224073c20 2024-09-16T15:46:19.125 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:19.121+0000 7f321a7fc640 1 -- 172.21.15.123:0/1865213815 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v55) ==== 138+0+0 (secure 0 0 0) 0x7f321405f660 con 0x7f3224073c20 2024-09-16T15:46:19.125 INFO:teuthology.orchestra.run.smithi123.stderr:pool 'cephfs_metadata' created 2024-09-16T15:46:19.128 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:19.125+0000 7f322bbfd640 1 -- 172.21.15.123:0/1865213815 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f3208073070 msgr2=0x7f3208075530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:46:19.129 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:19.125+0000 7f322bbfd640 1 --2- 172.21.15.123:0/1865213815 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f3208073070 0x7f3208075530 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f3220001690 tx=0x7f3220009040 comp rx=0 tx=0).stop 2024-09-16T15:46:19.129 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:19.125+0000 7f322bbfd640 1 -- 172.21.15.123:0/1865213815 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3224073c20 msgr2=0x7f32241490a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:46:19.129 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:19.125+0000 7f322bbfd640 1 --2- 172.21.15.123:0/1865213815 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3224073c20 0x7f32241490a0 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7f321400e9f0 tx=0x7f321400eeb0 comp rx=0 tx=0).stop 2024-09-16T15:46:19.129 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:19.125+0000 7f322bbfd640 1 -- 172.21.15.123:0/1865213815 shutdown_connections 2024-09-16T15:46:19.129 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:19.125+0000 7f322bbfd640 1 --2- 172.21.15.123:0/1865213815 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f3208073070 0x7f3208075530 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:19.129 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:19.125+0000 7f322bbfd640 1 --2- 172.21.15.123:0/1865213815 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f32240b3260 0x7f32240cb3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:19.129 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:19.125+0000 7f322bbfd640 1 --2- 172.21.15.123:0/1865213815 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3224073c20 0x7f32241490a0 unknown :-1 s=CLOSED pgs=619 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:19.129 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:19.125+0000 7f322bbfd640 1 --2- 172.21.15.123:0/1865213815 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f322406ff30 0x7f32241f9060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:19.129 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:19.125+0000 7f322bbfd640 1 -- 172.21.15.123:0/1865213815 >> 172.21.15.123:0/1865213815 conn(0x7f32240b1630 msgr2=0x7f322406f750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:46:19.129 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:19.125+0000 7f322bbfd640 1 -- 172.21.15.123:0/1865213815 shutdown_connections 2024-09-16T15:46:19.129 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:19.125+0000 7f322bbfd640 1 -- 172.21.15.123:0/1865213815 wait complete. 2024-09-16T15:46:19.153 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-09-16T15:46:19.290 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:19.285+0000 7f0d7c5dc640 1 -- 172.21.15.123:0/894845736 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0d60004810 msgr2=0x7f0d60004c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:46:19.290 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:19.285+0000 7f0d7c5dc640 1 --2- 172.21.15.123:0/894845736 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0d60004810 0x7f0d60004c90 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7f0d6400c2e0 tx=0x7f0d6401f6c0 comp rx=0 tx=0).stop 2024-09-16T15:46:19.290 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:19.285+0000 7f0d7c5dc640 1 -- 172.21.15.123:0/894845736 shutdown_connections 2024-09-16T15:46:19.290 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:19.285+0000 7f0d7c5dc640 1 --2- 172.21.15.123:0/894845736 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0d60004810 0x7f0d60004c90 unknown :-1 s=CLOSED pgs=621 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:19.290 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:19.285+0000 7f0d7c5dc640 1 -- 172.21.15.123:0/894845736 >> 172.21.15.123:0/894845736 conn(0x7f0d740b1710 msgr2=0x7f0d740b1b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:46:19.290 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:19.285+0000 7f0d7c5dc640 1 -- 172.21.15.123:0/894845736 shutdown_connections 2024-09-16T15:46:19.290 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:19.285+0000 7f0d7c5dc640 1 -- 172.21.15.123:0/894845736 wait complete. 2024-09-16T15:46:19.291 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:19.285+0000 7f0d7c5dc640 1 Processor -- start 2024-09-16T15:46:19.291 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:19.289+0000 7f0d7c5dc640 1 -- start start 2024-09-16T15:46:19.292 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:19.289+0000 7f0d7c5dc640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0d60004810 0x7f0d74155040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:46:19.293 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:19.289+0000 7f0d7c5dc640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f0d741586f0 0x7f0d7414aef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:46:19.293 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:19.289+0000 7f0d7c5dc640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f0d740c8650 0x7f0d740c8ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:46:19.293 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:19.289+0000 7f0d7c5dc640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f0d7415e960 con 0x7f0d60004810 2024-09-16T15:46:19.293 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:19.289+0000 7f0d7c5dc640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f0d7415e7e0 con 0x7f0d740c8650 2024-09-16T15:46:19.293 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:19.289+0000 7f0d7c5dc640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f0d7415ae90 con 0x7f0d741586f0 2024-09-16T15:46:19.293 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:19.289+0000 7f0d7a378640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0d60004810 0x7f0d74155040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:46:19.293 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:19.289+0000 7f0d7a378640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0d60004810 0x7f0d74155040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:42220/0 (socket says 172.21.15.123:42220) 2024-09-16T15:46:19.293 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:19.289+0000 7f0d7a378640 1 -- 172.21.15.123:0/2858411999 learned_addr learned my addr 172.21.15.123:0/2858411999 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:46:19.293 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:19.289+0000 7f0d7ab79640 1 --2- 172.21.15.123:0/2858411999 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f0d740c8650 0x7f0d740c8ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:46:19.293 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:19.289+0000 7f0d79b77640 1 --2- 172.21.15.123:0/2858411999 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f0d741586f0 0x7f0d7414aef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:46:19.293 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:19.289+0000 7f0d7a378640 1 -- 172.21.15.123:0/2858411999 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f0d741586f0 msgr2=0x7f0d7414aef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:46:19.293 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:19.289+0000 7f0d7a378640 1 --2- 172.21.15.123:0/2858411999 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f0d741586f0 0x7f0d7414aef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:19.293 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:19.289+0000 7f0d7a378640 1 -- 172.21.15.123:0/2858411999 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f0d740c8650 msgr2=0x7f0d740c8ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:46:19.294 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:19.289+0000 7f0d7a378640 1 --2- 172.21.15.123:0/2858411999 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f0d740c8650 0x7f0d740c8ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:19.294 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:19.289+0000 7f0d7a378640 1 -- 172.21.15.123:0/2858411999 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0d74158db0 con 0x7f0d60004810 2024-09-16T15:46:19.294 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:19.289+0000 7f0d7a378640 1 --2- 172.21.15.123:0/2858411999 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0d60004810 0x7f0d74155040 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7f0d70004ee0 tx=0x7f0d70001640 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:46:19.294 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:19.289+0000 7f0d6affd640 1 -- 172.21.15.123:0/2858411999 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d70015020 con 0x7f0d60004810 2024-09-16T15:46:19.294 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:19.289+0000 7f0d6affd640 1 -- 172.21.15.123:0/2858411999 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f0d70011960 con 0x7f0d60004810 2024-09-16T15:46:19.294 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:19.289+0000 7f0d6affd640 1 -- 172.21.15.123:0/2858411999 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d70011d90 con 0x7f0d60004810 2024-09-16T15:46:19.294 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:19.289+0000 7f0d7c5dc640 1 -- 172.21.15.123:0/2858411999 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0d740c76c0 con 0x7f0d60004810 2024-09-16T15:46:19.294 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:19.289+0000 7f0d7c5dc640 1 -- 172.21.15.123:0/2858411999 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0d740c7c80 con 0x7f0d60004810 2024-09-16T15:46:19.295 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:19.289+0000 7f0d7c5dc640 1 -- 172.21.15.123:0/2858411999 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0d740cc7a0 con 0x7f0d60004810 2024-09-16T15:46:19.295 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:19.289+0000 7f0d6affd640 1 -- 172.21.15.123:0/2858411999 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 9) ==== 97233+0+0 (secure 0 0 0) 0x7f0d70018020 con 0x7f0d60004810 2024-09-16T15:46:19.295 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:19.293+0000 7f0d6affd640 1 --2- 172.21.15.123:0/2858411999 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f0d580730a0 0x7f0d58075560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:46:19.295 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:19.293+0000 7f0d6affd640 1 -- 172.21.15.123:0/2858411999 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(55..55 src has 1..55) ==== 5227+0+0 (secure 0 0 0) 0x7f0d70094f90 con 0x7f0d60004810 2024-09-16T15:46:19.295 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:19.293+0000 7f0d79b77640 1 --2- 172.21.15.123:0/2858411999 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f0d580730a0 0x7f0d58075560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:46:19.296 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:19.293+0000 7f0d79b77640 1 --2- 172.21.15.123:0/2858411999 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f0d580730a0 0x7f0d58075560 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f0d640b0000 tx=0x7f0d64006040 comp rx=0 tx=0).ready entity=mgr.4135 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:46:19.299 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:19.297+0000 7f0d6affd640 1 -- 172.21.15.123:0/2858411999 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f0d7001f020 con 0x7f0d60004810 2024-09-16T15:46:19.470 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:19.465+0000 7f0d7c5dc640 1 -- 172.21.15.123:0/2858411999 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) -- 0x7f0d74155b30 con 0x7f0d60004810 2024-09-16T15:46:20.137 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:20.133+0000 7f0d6affd640 1 -- 172.21.15.123:0/2858411999 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v56) ==== 170+0+0 (secure 0 0 0) 0x7f0d700601a0 con 0x7f0d60004810 2024-09-16T15:46:20.137 INFO:teuthology.orchestra.run.smithi123.stderr:pool 'cephfs_data' created 2024-09-16T15:46:20.142 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:20.137+0000 7f0d7c5dc640 1 -- 172.21.15.123:0/2858411999 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f0d580730a0 msgr2=0x7f0d58075560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:46:20.142 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:20.137+0000 7f0d7c5dc640 1 --2- 172.21.15.123:0/2858411999 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f0d580730a0 0x7f0d58075560 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f0d640b0000 tx=0x7f0d64006040 comp rx=0 tx=0).stop 2024-09-16T15:46:20.142 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:20.137+0000 7f0d7c5dc640 1 -- 172.21.15.123:0/2858411999 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0d60004810 msgr2=0x7f0d74155040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:46:20.142 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:20.137+0000 7f0d7c5dc640 1 --2- 172.21.15.123:0/2858411999 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0d60004810 0x7f0d74155040 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7f0d70004ee0 tx=0x7f0d70001640 comp rx=0 tx=0).stop 2024-09-16T15:46:20.142 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:20.137+0000 7f0d7c5dc640 1 -- 172.21.15.123:0/2858411999 shutdown_connections 2024-09-16T15:46:20.143 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:20.137+0000 7f0d7c5dc640 1 --2- 172.21.15.123:0/2858411999 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f0d580730a0 0x7f0d58075560 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:20.143 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:20.137+0000 7f0d7c5dc640 1 --2- 172.21.15.123:0/2858411999 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f0d740c8650 0x7f0d740c8ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:20.143 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:20.137+0000 7f0d7c5dc640 1 --2- 172.21.15.123:0/2858411999 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f0d741586f0 0x7f0d7414aef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:20.143 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:20.137+0000 7f0d7c5dc640 1 --2- 172.21.15.123:0/2858411999 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0d60004810 0x7f0d74155040 unknown :-1 s=CLOSED pgs=622 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:20.143 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:20.137+0000 7f0d7c5dc640 1 -- 172.21.15.123:0/2858411999 >> 172.21.15.123:0/2858411999 conn(0x7f0d740b1710 msgr2=0x7f0d7415b340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:46:20.143 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:20.137+0000 7f0d7c5dc640 1 -- 172.21.15.123:0/2858411999 shutdown_connections 2024-09-16T15:46:20.143 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:20.137+0000 7f0d7c5dc640 1 -- 172.21.15.123:0/2858411999 wait complete. 2024-09-16T15:46:20.172 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-09-16T15:46:20.297 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:20.293+0000 7f62e10fb640 1 -- 172.21.15.123:0/1826795103 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f62dc0d2120 msgr2=0x7f62dc0cef00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:46:20.297 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:20.293+0000 7f62e10fb640 1 --2- 172.21.15.123:0/1826795103 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f62dc0d2120 0x7f62dc0cef00 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7f62cc009c80 tx=0x7f62cc01cc80 comp rx=0 tx=0).stop 2024-09-16T15:46:20.297 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:20.293+0000 7f62e10fb640 1 -- 172.21.15.123:0/1826795103 shutdown_connections 2024-09-16T15:46:20.297 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:20.293+0000 7f62e10fb640 1 --2- 172.21.15.123:0/1826795103 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f62dc0d2120 0x7f62dc0cef00 unknown :-1 s=CLOSED pgs=624 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:20.297 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:20.293+0000 7f62e10fb640 1 --2- 172.21.15.123:0/1826795103 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f62dc0d0f60 0x7f62dc0ce9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:20.297 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:20.293+0000 7f62e10fb640 1 -- 172.21.15.123:0/1826795103 >> 172.21.15.123:0/1826795103 conn(0x7f62dc0d3e40 msgr2=0x7f62dc0d4220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:46:20.297 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:20.293+0000 7f62e10fb640 1 -- 172.21.15.123:0/1826795103 shutdown_connections 2024-09-16T15:46:20.297 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:20.293+0000 7f62e10fb640 1 -- 172.21.15.123:0/1826795103 wait complete. 2024-09-16T15:46:20.300 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:20.293+0000 7f62e10fb640 1 Processor -- start 2024-09-16T15:46:20.300 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:20.293+0000 7f62e10fb640 1 -- start start 2024-09-16T15:46:20.300 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:20.293+0000 7f62e10fb640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f62dc0d0f60 0x7f62dc0b9860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:46:20.300 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:20.293+0000 7f62e10fb640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f62dc0d2120 0x7f62dc150d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:46:20.300 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:20.293+0000 7f62e10fb640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f62dc1615c0 0x7f62dc151e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:46:20.300 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:20.293+0000 7f62e10fb640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f62dc16b590 con 0x7f62dc0d2120 2024-09-16T15:46:20.300 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:20.293+0000 7f62e10fb640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f62dc16b410 con 0x7f62dc0d0f60 2024-09-16T15:46:20.301 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:20.293+0000 7f62e10fb640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f62dc16b710 con 0x7f62dc1615c0 2024-09-16T15:46:20.301 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:20.293+0000 7f62da59c640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f62dc0d2120 0x7f62dc150d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:46:20.301 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:20.293+0000 7f62da59c640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f62dc0d2120 0x7f62dc150d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:42250/0 (socket says 172.21.15.123:42250) 2024-09-16T15:46:20.301 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:20.293+0000 7f62da59c640 1 -- 172.21.15.123:0/3882971278 learned_addr learned my addr 172.21.15.123:0/3882971278 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:46:20.301 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:20.297+0000 7f62dad9d640 1 --2- 172.21.15.123:0/3882971278 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f62dc0d0f60 0x7f62dc0b9860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:46:20.301 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:20.297+0000 7f62da59c640 1 -- 172.21.15.123:0/3882971278 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f62dc1615c0 msgr2=0x7f62dc151e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:46:20.301 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:20.297+0000 7f62db59e640 1 --2- 172.21.15.123:0/3882971278 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f62dc1615c0 0x7f62dc151e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:46:20.301 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:20.297+0000 7f62da59c640 1 --2- 172.21.15.123:0/3882971278 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f62dc1615c0 0x7f62dc151e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:20.301 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:20.297+0000 7f62da59c640 1 -- 172.21.15.123:0/3882971278 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f62dc0d0f60 msgr2=0x7f62dc0b9860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:46:20.301 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:20.297+0000 7f62da59c640 1 --2- 172.21.15.123:0/3882971278 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f62dc0d0f60 0x7f62dc0b9860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:20.301 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:20.297+0000 7f62da59c640 1 -- 172.21.15.123:0/3882971278 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f62dc0b8560 con 0x7f62dc0d2120 2024-09-16T15:46:20.301 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:20.297+0000 7f62da59c640 1 --2- 172.21.15.123:0/3882971278 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f62dc0d2120 0x7f62dc150d10 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7f62d000c8c0 tx=0x7f62d000cd80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:46:20.302 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:20.297+0000 7f62c37fe640 1 -- 172.21.15.123:0/3882971278 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62d0004230 con 0x7f62dc0d2120 2024-09-16T15:46:20.302 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:20.297+0000 7f62c37fe640 1 -- 172.21.15.123:0/3882971278 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f62d00044c0 con 0x7f62dc0d2120 2024-09-16T15:46:20.302 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:20.297+0000 7f62e10fb640 1 -- 172.21.15.123:0/3882971278 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f62dc151250 con 0x7f62dc0d2120 2024-09-16T15:46:20.302 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:20.297+0000 7f62c37fe640 1 -- 172.21.15.123:0/3882971278 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62d0007bf0 con 0x7f62dc0d2120 2024-09-16T15:46:20.302 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:20.297+0000 7f62e10fb640 1 -- 172.21.15.123:0/3882971278 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f62dc0b8aa0 con 0x7f62dc0d2120 2024-09-16T15:46:20.302 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:20.297+0000 7f62e10fb640 1 -- 172.21.15.123:0/3882971278 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f62dc0d0850 con 0x7f62dc0d2120 2024-09-16T15:46:20.303 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:20.297+0000 7f62c37fe640 1 -- 172.21.15.123:0/3882971278 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 9) ==== 97233+0+0 (secure 0 0 0) 0x7f62d0003600 con 0x7f62dc0d2120 2024-09-16T15:46:20.303 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:20.297+0000 7f62c37fe640 1 --2- 172.21.15.123:0/3882971278 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f62c8073070 0x7f62c8075530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:46:20.303 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:20.297+0000 7f62c37fe640 1 -- 172.21.15.123:0/3882971278 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(56..56 src has 1..56) ==== 5609+0+0 (secure 0 0 0) 0x7f62d0094780 con 0x7f62dc0d2120 2024-09-16T15:46:20.303 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:20.297+0000 7f62dad9d640 1 --2- 172.21.15.123:0/3882971278 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f62c8073070 0x7f62c8075530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:46:20.303 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:20.297+0000 7f62dad9d640 1 --2- 172.21.15.123:0/3882971278 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f62c8073070 0x7f62c8075530 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f62d4001650 tx=0x7f62d4009040 comp rx=0 tx=0).ready entity=mgr.4135 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:46:20.306 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:20.301+0000 7f62c37fe640 1 -- 172.21.15.123:0/3882971278 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f62dc151250 con 0x7f62dc0d2120 2024-09-16T15:46:20.514 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:20.509+0000 7f62e10fb640 1 -- 172.21.15.123:0/3882971278 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) -- 0x7f62dc0d4500 con 0x7f62dc0d2120 2024-09-16T15:46:21.140 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:21.137+0000 7f62c37fe640 1 -- 172.21.15.123:0/3882971278 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 8 and data pool 9 v32) ==== 172+0+0 (secure 0 0 0) 0x7f62dc0b8560 con 0x7f62dc0d2120 2024-09-16T15:46:21.140 INFO:teuthology.orchestra.run.smithi123.stderr:new fs with metadata pool 8 and data pool 9 2024-09-16T15:46:21.144 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:21.141+0000 7f62e10fb640 1 -- 172.21.15.123:0/3882971278 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f62c8073070 msgr2=0x7f62c8075530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:46:21.145 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:21.141+0000 7f62e10fb640 1 --2- 172.21.15.123:0/3882971278 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f62c8073070 0x7f62c8075530 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f62d4001650 tx=0x7f62d4009040 comp rx=0 tx=0).stop 2024-09-16T15:46:21.148 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:21.141+0000 7f62e10fb640 1 -- 172.21.15.123:0/3882971278 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f62dc0d2120 msgr2=0x7f62dc150d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:46:21.148 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:21.141+0000 7f62e10fb640 1 --2- 172.21.15.123:0/3882971278 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f62dc0d2120 0x7f62dc150d10 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7f62d000c8c0 tx=0x7f62d000cd80 comp rx=0 tx=0).stop 2024-09-16T15:46:21.148 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:21.141+0000 7f62e10fb640 1 -- 172.21.15.123:0/3882971278 shutdown_connections 2024-09-16T15:46:21.148 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:21.141+0000 7f62e10fb640 1 --2- 172.21.15.123:0/3882971278 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f62c8073070 0x7f62c8075530 secure :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f62d4001650 tx=0x7f62d4009040 comp rx=0 tx=0).stop 2024-09-16T15:46:21.148 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:21.141+0000 7f62e10fb640 1 --2- 172.21.15.123:0/3882971278 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f62dc1615c0 0x7f62dc151e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:21.148 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:21.141+0000 7f62e10fb640 1 --2- 172.21.15.123:0/3882971278 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f62dc0d2120 0x7f62dc150d10 unknown :-1 s=CLOSED pgs=625 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:21.148 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:21.141+0000 7f62e10fb640 1 --2- 172.21.15.123:0/3882971278 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f62dc0d0f60 0x7f62dc0b9860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:21.148 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:21.141+0000 7f62e10fb640 1 -- 172.21.15.123:0/3882971278 >> 172.21.15.123:0/3882971278 conn(0x7f62dc0d3e40 msgr2=0x7f62dc0b2450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:46:21.149 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:21.141+0000 7f62e10fb640 1 -- 172.21.15.123:0/3882971278 shutdown_connections 2024-09-16T15:46:21.149 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:21.141+0000 7f62e10fb640 1 -- 172.21.15.123:0/3882971278 wait complete. 2024-09-16T15:46:21.181 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-16T15:46:21.299 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:21.293+0000 7f869e571640 1 -- 172.21.15.123:0/896501493 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f86800053d0 msgr2=0x7f868000b9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:46:21.299 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:21.293+0000 7f869e571640 1 --2- 172.21.15.123:0/896501493 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f86800053d0 0x7f868000b9d0 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7f86900043e0 tx=0x7f869001c970 comp rx=0 tx=0).stop 2024-09-16T15:46:21.299 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:21.293+0000 7f869e571640 1 -- 172.21.15.123:0/896501493 shutdown_connections 2024-09-16T15:46:21.299 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:21.293+0000 7f869e571640 1 --2- 172.21.15.123:0/896501493 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f86800053d0 0x7f868000b9d0 unknown :-1 s=CLOSED pgs=627 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:21.299 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:21.293+0000 7f869e571640 1 -- 172.21.15.123:0/896501493 >> 172.21.15.123:0/896501493 conn(0x7f86980d3ea0 msgr2=0x7f86980d42b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:46:21.299 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:21.293+0000 7f869e571640 1 -- 172.21.15.123:0/896501493 shutdown_connections 2024-09-16T15:46:21.299 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:21.293+0000 7f869e571640 1 -- 172.21.15.123:0/896501493 wait complete. 2024-09-16T15:46:21.300 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:21.297+0000 7f869e571640 1 Processor -- start 2024-09-16T15:46:21.300 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:21.297+0000 7f869e571640 1 -- start start 2024-09-16T15:46:21.302 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:21.297+0000 7f869e571640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f86800053d0 0x7f8698155f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:46:21.303 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:21.297+0000 7f869e571640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f869815ba50 0x7f86980cbd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:46:21.303 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:21.297+0000 7f869e571640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f86980ceec0 0x7f86980cf340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:46:21.303 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:21.297+0000 7f869e571640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f86980d1da0 con 0x7f86980ceec0 2024-09-16T15:46:21.303 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:21.297+0000 7f869e571640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f86980b12c0 con 0x7f869815ba50 2024-09-16T15:46:21.303 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:21.297+0000 7f869e571640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f86980d1f20 con 0x7f86800053d0 2024-09-16T15:46:21.303 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:21.297+0000 7f869cb0e640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f86980ceec0 0x7f86980cf340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:46:21.303 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:21.297+0000 7f869cb0e640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f86980ceec0 0x7f86980cf340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:42260/0 (socket says 172.21.15.123:42260) 2024-09-16T15:46:21.303 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:21.297+0000 7f869cb0e640 1 -- 172.21.15.123:0/3941751507 learned_addr learned my addr 172.21.15.123:0/3941751507 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:46:21.303 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:21.297+0000 7f86977fe640 1 --2- 172.21.15.123:0/3941751507 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f869815ba50 0x7f86980cbd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:46:21.303 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:21.297+0000 7f8697fff640 1 --2- 172.21.15.123:0/3941751507 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f86800053d0 0x7f8698155f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:46:21.304 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:21.297+0000 7f869cb0e640 1 -- 172.21.15.123:0/3941751507 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f86800053d0 msgr2=0x7f8698155f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:46:21.304 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:21.297+0000 7f869cb0e640 1 --2- 172.21.15.123:0/3941751507 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f86800053d0 0x7f8698155f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:21.304 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:21.297+0000 7f869cb0e640 1 -- 172.21.15.123:0/3941751507 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f869815ba50 msgr2=0x7f86980cbd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:46:21.304 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:21.297+0000 7f869cb0e640 1 --2- 172.21.15.123:0/3941751507 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f869815ba50 0x7f86980cbd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:21.304 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:21.297+0000 7f869cb0e640 1 -- 172.21.15.123:0/3941751507 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f86980d4680 con 0x7f86980ceec0 2024-09-16T15:46:21.304 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:21.297+0000 7f86977fe640 1 --2- 172.21.15.123:0/3941751507 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f869815ba50 0x7f86980cbd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T15:46:21.304 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:21.297+0000 7f869cb0e640 1 --2- 172.21.15.123:0/3941751507 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f86980ceec0 0x7f86980cf340 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7f8690006040 tx=0x7f8690006cb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:46:21.304 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:21.297+0000 7f8694ff9640 1 -- 172.21.15.123:0/3941751507 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f86900a58e0 con 0x7f86980ceec0 2024-09-16T15:46:21.304 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:21.297+0000 7f8694ff9640 1 -- 172.21.15.123:0/3941751507 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f86900b3030 con 0x7f86980ceec0 2024-09-16T15:46:21.304 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:21.297+0000 7f869e571640 1 -- 172.21.15.123:0/3941751507 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f86981faa90 con 0x7f86980ceec0 2024-09-16T15:46:21.304 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:21.297+0000 7f8694ff9640 1 -- 172.21.15.123:0/3941751507 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f86900a5e30 con 0x7f86980ceec0 2024-09-16T15:46:21.305 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:21.297+0000 7f869e571640 1 -- 172.21.15.123:0/3941751507 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f86981fb050 con 0x7f86980ceec0 2024-09-16T15:46:21.305 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:21.297+0000 7f869e571640 1 -- 172.21.15.123:0/3941751507 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f86980d2480 con 0x7f86980ceec0 2024-09-16T15:46:21.305 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:21.301+0000 7f8694ff9640 1 -- 172.21.15.123:0/3941751507 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 9) ==== 97233+0+0 (secure 0 0 0) 0x7f8690006070 con 0x7f86980ceec0 2024-09-16T15:46:21.305 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:21.301+0000 7f8694ff9640 1 --2- 172.21.15.123:0/3941751507 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f86840730a0 0x7f8684075560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:46:21.306 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:21.301+0000 7f8694ff9640 1 -- 172.21.15.123:0/3941751507 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(57..57 src has 1..57) ==== 5709+0+0 (secure 0 0 0) 0x7f869012e250 con 0x7f86980ceec0 2024-09-16T15:46:21.306 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:21.301+0000 7f8697fff640 1 --2- 172.21.15.123:0/3941751507 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f86840730a0 0x7f8684075560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:46:21.306 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:21.301+0000 7f8697fff640 1 --2- 172.21.15.123:0/3941751507 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f86840730a0 0x7f8684075560 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f8688001ec0 tx=0x7f86880012f0 comp rx=0 tx=0).ready entity=mgr.4135 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:46:21.307 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:21.305+0000 7f8694ff9640 1 -- 172.21.15.123:0/3941751507 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f8690016770 con 0x7f86980ceec0 2024-09-16T15:46:21.481 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:21.477+0000 7f869e571640 1 -- 172.21.15.123:0/3941751507 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f86980cc270 con 0x7f86980ceec0 2024-09-16T15:46:21.482 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:21.477+0000 7f8694ff9640 1 -- 172.21.15.123:0/3941751507 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v57) ==== 74+0+16798 (secure 0 0 0) 0x7f86900f9390 con 0x7f86980ceec0 2024-09-16T15:46:21.483 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-09-16T15:46:21.483 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":57,"fsid":"56077f89-18c8-486f-ae7e-eecc0bb7822c","created":"2024-09-16T15:42:26.364892+0000","modified":"2024-09-16T15:46:20.516069+0000","last_up_change":"2024-09-16T15:42:38.707234+0000","last_in_change":"2024-09-16T15:42:29.423662+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-16T15:42:39.287764+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_metadata","create_time":"2024-09-16T15:46:18.173124+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"57","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_data","create_time":"2024-09-16T15:46:19.478016+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"57","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9e7c1955-a607-4671-9be7-f2ef760ceeaa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6800","nonce":3882044969},{"type":"v1","addr":"172.21.15.123:6801","nonce":3882044969}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6802","nonce":3882044969},{"type":"v1","addr":"172.21.15.123:6803","nonce":3882044969}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6806","nonce":3882044969},{"type":"v1","addr":"172.21.15.123:6807","nonce":3882044969}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6804","nonce":3882044969},{"type":"v1","addr":"172.21.15.123:6805","nonce":3882044969}]},"public_addr":"172.21.15.123:6801/3882044969","cluster_addr":"172.21.15.123:6803/3882044969","heartbeat_back_addr":"172.21.15.123:6807/3882044969","heartbeat_front_addr":"172.21.15.123:6805/3882044969","state":["exists","up"]},{"osd":1,"uuid":"c540cfde-0977-4742-9b17-fd634668e010","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6816","nonce":4037001850},{"type":"v1","addr":"172.21.15.123:6817","nonce":4037001850}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6818","nonce":4037001850},{"type":"v1","addr":"172.21.15.123:6819","nonce":4037001850}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6822","nonce":4037001850},{"type":"v1","addr":"172.21.15.123:6823","nonce":4037001850}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6820","nonce":4037001850},{"type":"v1","addr":"172.21.15.123:6821","nonce":4037001850}]},"public_addr":"172.21.15.123:6817/4037001850","cluster_addr":"172.21.15.123:6819/4037001850","heartbeat_back_addr":"172.21.15.123:6823/4037001850","heartbeat_front_addr":"172.21.15.123:6821/4037001850","state":["exists","up"]},{"osd":2,"uuid":"9864a496-f169-486b-95c4-1d59162793a7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6824","nonce":1332143088},{"type":"v1","addr":"172.21.15.123:6825","nonce":1332143088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6826","nonce":1332143088},{"type":"v1","addr":"172.21.15.123:6827","nonce":1332143088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6830","nonce":1332143088},{"type":"v1","addr":"172.21.15.123:6831","nonce":1332143088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6828","nonce":1332143088},{"type":"v1","addr":"172.21.15.123:6829","nonce":1332143088}]},"public_addr":"172.21.15.123:6825/1332143088","cluster_addr":"172.21.15.123:6827/1332143088","heartbeat_back_addr":"172.21.15.123:6831/1332143088","heartbeat_front_addr":"172.21.15.123:6829/1332143088","state":["exists","up"]},{"osd":3,"uuid":"c6d1f817-f074-4325-95e1-578eba13edda","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6808","nonce":988586998},{"type":"v1","addr":"172.21.15.123:6809","nonce":988586998}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6810","nonce":988586998},{"type":"v1","addr":"172.21.15.123:6811","nonce":988586998}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6814","nonce":988586998},{"type":"v1","addr":"172.21.15.123:6815","nonce":988586998}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6812","nonce":988586998},{"type":"v1","addr":"172.21.15.123:6813","nonce":988586998}]},"public_addr":"172.21.15.123:6809/988586998","cluster_addr":"172.21.15.123:6811/988586998","heartbeat_back_addr":"172.21.15.123:6815/988586998","heartbeat_front_addr":"172.21.15.123:6813/988586998","state":["exists","up"]},{"osd":4,"uuid":"200a8df4-50ef-40a5-b48a-58981269c086","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6816","nonce":3387813890},{"type":"v1","addr":"172.21.15.184:6817","nonce":3387813890}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6818","nonce":3387813890},{"type":"v1","addr":"172.21.15.184:6819","nonce":3387813890}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6822","nonce":3387813890},{"type":"v1","addr":"172.21.15.184:6823","nonce":3387813890}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6820","nonce":3387813890},{"type":"v1","addr":"172.21.15.184:6821","nonce":3387813890}]},"public_addr":"172.21.15.184:6817/3387813890","cluster_addr":"172.21.15.184:6819/3387813890","heartbeat_back_addr":"172.21.15.184:6823/3387813890","heartbeat_front_addr":"172.21.15.184:6821/3387813890","state":["exists","up"]},{"osd":5,"uuid":"32d5a900-da3d-412c-bc19-13861a2368d8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6808","nonce":2680916017},{"type":"v1","addr":"172.21.15.184:6809","nonce":2680916017}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6810","nonce":2680916017},{"type":"v1","addr":"172.21.15.184:6811","nonce":2680916017}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6814","nonce":2680916017},{"type":"v1","addr":"172.21.15.184:6815","nonce":2680916017}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6812","nonce":2680916017},{"type":"v1","addr":"172.21.15.184:6813","nonce":2680916017}]},"public_addr":"172.21.15.184:6809/2680916017","cluster_addr":"172.21.15.184:6811/2680916017","heartbeat_back_addr":"172.21.15.184:6815/2680916017","heartbeat_front_addr":"172.21.15.184:6813/2680916017","state":["exists","up"]},{"osd":6,"uuid":"cde6f39e-aa1b-4260-ac19-774ee6aaca72","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6824","nonce":2183120457},{"type":"v1","addr":"172.21.15.184:6825","nonce":2183120457}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6826","nonce":2183120457},{"type":"v1","addr":"172.21.15.184:6827","nonce":2183120457}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6830","nonce":2183120457},{"type":"v1","addr":"172.21.15.184:6831","nonce":2183120457}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6828","nonce":2183120457},{"type":"v1","addr":"172.21.15.184:6829","nonce":2183120457}]},"public_addr":"172.21.15.184:6825/2183120457","cluster_addr":"172.21.15.184:6827/2183120457","heartbeat_back_addr":"172.21.15.184:6831/2183120457","heartbeat_front_addr":"172.21.15.184:6829/2183120457","state":["exists","up"]},{"osd":7,"uuid":"4f381778-eee1-47c6-804f-df0e6b33e313","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6800","nonce":4102957285},{"type":"v1","addr":"172.21.15.184:6801","nonce":4102957285}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6802","nonce":4102957285},{"type":"v1","addr":"172.21.15.184:6803","nonce":4102957285}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6806","nonce":4102957285},{"type":"v1","addr":"172.21.15.184:6807","nonce":4102957285}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6804","nonce":4102957285},{"type":"v1","addr":"172.21.15.184:6805","nonce":4102957285}]},"public_addr":"172.21.15.184:6801/4102957285","cluster_addr":"172.21.15.184:6803/4102957285","heartbeat_back_addr":"172.21.15.184:6807/4102957285","heartbeat_front_addr":"172.21.15.184:6805/4102957285","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:35.972210+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.222007+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.198617+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.385663+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.358935+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.156066+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.752973+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:35.682846+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T15:46:21.485 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:21.481+0000 7f869e571640 1 -- 172.21.15.123:0/3941751507 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f86840730a0 msgr2=0x7f8684075560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:46:21.485 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:21.481+0000 7f869e571640 1 --2- 172.21.15.123:0/3941751507 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f86840730a0 0x7f8684075560 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f8688001ec0 tx=0x7f86880012f0 comp rx=0 tx=0).stop 2024-09-16T15:46:21.485 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:21.481+0000 7f869e571640 1 -- 172.21.15.123:0/3941751507 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f86980ceec0 msgr2=0x7f86980cf340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:46:21.485 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:21.481+0000 7f869e571640 1 --2- 172.21.15.123:0/3941751507 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f86980ceec0 0x7f86980cf340 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7f8690006040 tx=0x7f8690006cb0 comp rx=0 tx=0).stop 2024-09-16T15:46:21.485 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:21.481+0000 7f869e571640 1 -- 172.21.15.123:0/3941751507 shutdown_connections 2024-09-16T15:46:21.485 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:21.481+0000 7f869e571640 1 --2- 172.21.15.123:0/3941751507 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f86840730a0 0x7f8684075560 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:21.485 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:21.481+0000 7f869e571640 1 --2- 172.21.15.123:0/3941751507 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f86980ceec0 0x7f86980cf340 unknown :-1 s=CLOSED pgs=628 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:21.486 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:21.481+0000 7f869e571640 1 --2- 172.21.15.123:0/3941751507 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f869815ba50 0x7f86980cbd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:21.486 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:21.481+0000 7f869e571640 1 --2- 172.21.15.123:0/3941751507 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f86800053d0 0x7f8698155f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:21.486 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:21.481+0000 7f869e571640 1 -- 172.21.15.123:0/3941751507 >> 172.21.15.123:0/3941751507 conn(0x7f86980d3ea0 msgr2=0x7f869815f460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:46:21.486 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:21.481+0000 7f869e571640 1 -- 172.21.15.123:0/3941751507 shutdown_connections 2024-09-16T15:46:21.486 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:21.481+0000 7f869e571640 1 -- 172.21.15.123:0/3941751507 wait complete. 2024-09-16T15:46:21.502 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-16T15:46:21.625 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:21.621+0000 7fc306e40640 1 -- 172.21.15.123:0/1953114720 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc2ec004050 msgr2=0x7fc2ec004810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:46:21.625 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:21.621+0000 7fc306e40640 1 --2- 172.21.15.123:0/1953114720 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc2ec004050 0x7fc2ec004810 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7fc2e800c640 tx=0x7fc2e801d320 comp rx=0 tx=0).stop 2024-09-16T15:46:21.625 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:21.621+0000 7fc3053dd640 1 -- 172.21.15.123:0/1953114720 reap_dead start 2024-09-16T15:46:21.625 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:21.621+0000 7fc306e40640 1 -- 172.21.15.123:0/1953114720 shutdown_connections 2024-09-16T15:46:21.625 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:21.621+0000 7fc306e40640 1 -- 172.21.15.123:0/1953114720 >> 172.21.15.123:0/1953114720 conn(0x7fc3000cfb10 msgr2=0x7fc3000cff20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:46:21.625 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:21.621+0000 7fc306e40640 1 -- 172.21.15.123:0/1953114720 shutdown_connections 2024-09-16T15:46:21.625 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:21.621+0000 7fc306e40640 1 -- 172.21.15.123:0/1953114720 wait complete. 2024-09-16T15:46:21.626 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:21.621+0000 7fc306e40640 1 Processor -- start 2024-09-16T15:46:21.626 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:21.621+0000 7fc306e40640 1 -- start start 2024-09-16T15:46:21.629 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:21.625+0000 7fc306e40640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc2ec004050 0x7fc3001507a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:46:21.629 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:21.625+0000 7fc306e40640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fc300150eb0 0x7fc300072ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:46:21.629 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:21.625+0000 7fc306e40640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc3001513e0 0x7fc3000730e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:46:21.629 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:21.625+0000 7fc306e40640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fc30015b190 con 0x7fc3001513e0 2024-09-16T15:46:21.629 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:21.625+0000 7fc306e40640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fc30015b010 con 0x7fc2ec004050 2024-09-16T15:46:21.629 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:21.625+0000 7fc306e40640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fc30015b310 con 0x7fc300150eb0 2024-09-16T15:46:21.630 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:21.625+0000 7fc3053dd640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc3001513e0 0x7fc3000730e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:46:21.630 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:21.625+0000 7fc3053dd640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc3001513e0 0x7fc3000730e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:42278/0 (socket says 172.21.15.123:42278) 2024-09-16T15:46:21.630 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:21.625+0000 7fc3053dd640 1 -- 172.21.15.123:0/1412846550 learned_addr learned my addr 172.21.15.123:0/1412846550 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:46:21.630 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:21.625+0000 7fc304bdc640 1 --2- 172.21.15.123:0/1412846550 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc2ec004050 0x7fc3001507a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:46:21.630 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:21.625+0000 7fc2fffff640 1 --2- 172.21.15.123:0/1412846550 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fc300150eb0 0x7fc300072ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:46:21.630 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:21.625+0000 7fc3053dd640 1 -- 172.21.15.123:0/1412846550 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fc300150eb0 msgr2=0x7fc300072ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:46:21.630 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:21.625+0000 7fc3053dd640 1 --2- 172.21.15.123:0/1412846550 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fc300150eb0 0x7fc300072ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:21.630 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:21.625+0000 7fc3053dd640 1 -- 172.21.15.123:0/1412846550 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc2ec004050 msgr2=0x7fc3001507a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:46:21.630 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:21.625+0000 7fc3053dd640 1 --2- 172.21.15.123:0/1412846550 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc2ec004050 0x7fc3001507a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:21.630 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:21.625+0000 7fc3053dd640 1 -- 172.21.15.123:0/1412846550 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc3000c9150 con 0x7fc3001513e0 2024-09-16T15:46:21.630 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:21.625+0000 7fc2fffff640 1 --2- 172.21.15.123:0/1412846550 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fc300150eb0 0x7fc300072ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T15:46:21.630 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:21.625+0000 7fc3053dd640 1 --2- 172.21.15.123:0/1412846550 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc3001513e0 0x7fc3000730e0 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7fc300069550 tx=0x7fc2f000e9b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:46:21.631 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:21.625+0000 7fc2fd7fa640 1 -- 172.21.15.123:0/1412846550 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2f0015030 con 0x7fc3001513e0 2024-09-16T15:46:21.631 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:21.625+0000 7fc2fd7fa640 1 -- 172.21.15.123:0/1412846550 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fc2f0004350 con 0x7fc3001513e0 2024-09-16T15:46:21.631 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:21.625+0000 7fc306e40640 1 -- 172.21.15.123:0/1412846550 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc3000c9430 con 0x7fc3001513e0 2024-09-16T15:46:21.631 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:21.625+0000 7fc2fd7fa640 1 -- 172.21.15.123:0/1412846550 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2f0005280 con 0x7fc3001513e0 2024-09-16T15:46:21.631 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:21.625+0000 7fc306e40640 1 -- 172.21.15.123:0/1412846550 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc3000c99f0 con 0x7fc3001513e0 2024-09-16T15:46:21.631 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:21.625+0000 7fc306e40640 1 -- 172.21.15.123:0/1412846550 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc3000ca320 con 0x7fc3001513e0 2024-09-16T15:46:21.632 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:21.625+0000 7fc2fd7fa640 1 -- 172.21.15.123:0/1412846550 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 9) ==== 97233+0+0 (secure 0 0 0) 0x7fc2f0018070 con 0x7fc3001513e0 2024-09-16T15:46:21.632 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:21.625+0000 7fc2fd7fa640 1 --2- 172.21.15.123:0/1412846550 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fc2e00730a0 0x7fc2e0075560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:46:21.632 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:21.629+0000 7fc2fd7fa640 1 -- 172.21.15.123:0/1412846550 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(57..57 src has 1..57) ==== 5709+0+0 (secure 0 0 0) 0x7fc2f0094ad0 con 0x7fc3001513e0 2024-09-16T15:46:21.632 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:21.629+0000 7fc304bdc640 1 --2- 172.21.15.123:0/1412846550 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fc2e00730a0 0x7fc2e0075560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:46:21.632 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:21.629+0000 7fc304bdc640 1 --2- 172.21.15.123:0/1412846550 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fc2e00730a0 0x7fc2e0075560 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7fc2f8005d10 tx=0x7fc2f8005c80 comp rx=0 tx=0).ready entity=mgr.4135 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:46:21.635 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:21.633+0000 7fc2fd7fa640 1 -- 172.21.15.123:0/1412846550 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7fc2f0094e20 con 0x7fc3001513e0 2024-09-16T15:46:21.808 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:21.805+0000 7fc306e40640 1 -- 172.21.15.123:0/1412846550 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fc3000d06c0 con 0x7fc3001513e0 2024-09-16T15:46:21.809 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:21.805+0000 7fc2fd7fa640 1 -- 172.21.15.123:0/1412846550 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v57) ==== 74+0+16798 (secure 0 0 0) 0x7fc2f005fb00 con 0x7fc3001513e0 2024-09-16T15:46:21.810 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-09-16T15:46:21.810 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":57,"fsid":"56077f89-18c8-486f-ae7e-eecc0bb7822c","created":"2024-09-16T15:42:26.364892+0000","modified":"2024-09-16T15:46:20.516069+0000","last_up_change":"2024-09-16T15:42:38.707234+0000","last_in_change":"2024-09-16T15:42:29.423662+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-16T15:42:39.287764+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_metadata","create_time":"2024-09-16T15:46:18.173124+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"57","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_data","create_time":"2024-09-16T15:46:19.478016+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"57","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9e7c1955-a607-4671-9be7-f2ef760ceeaa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6800","nonce":3882044969},{"type":"v1","addr":"172.21.15.123:6801","nonce":3882044969}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6802","nonce":3882044969},{"type":"v1","addr":"172.21.15.123:6803","nonce":3882044969}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6806","nonce":3882044969},{"type":"v1","addr":"172.21.15.123:6807","nonce":3882044969}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6804","nonce":3882044969},{"type":"v1","addr":"172.21.15.123:6805","nonce":3882044969}]},"public_addr":"172.21.15.123:6801/3882044969","cluster_addr":"172.21.15.123:6803/3882044969","heartbeat_back_addr":"172.21.15.123:6807/3882044969","heartbeat_front_addr":"172.21.15.123:6805/3882044969","state":["exists","up"]},{"osd":1,"uuid":"c540cfde-0977-4742-9b17-fd634668e010","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6816","nonce":4037001850},{"type":"v1","addr":"172.21.15.123:6817","nonce":4037001850}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6818","nonce":4037001850},{"type":"v1","addr":"172.21.15.123:6819","nonce":4037001850}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6822","nonce":4037001850},{"type":"v1","addr":"172.21.15.123:6823","nonce":4037001850}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6820","nonce":4037001850},{"type":"v1","addr":"172.21.15.123:6821","nonce":4037001850}]},"public_addr":"172.21.15.123:6817/4037001850","cluster_addr":"172.21.15.123:6819/4037001850","heartbeat_back_addr":"172.21.15.123:6823/4037001850","heartbeat_front_addr":"172.21.15.123:6821/4037001850","state":["exists","up"]},{"osd":2,"uuid":"9864a496-f169-486b-95c4-1d59162793a7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6824","nonce":1332143088},{"type":"v1","addr":"172.21.15.123:6825","nonce":1332143088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6826","nonce":1332143088},{"type":"v1","addr":"172.21.15.123:6827","nonce":1332143088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6830","nonce":1332143088},{"type":"v1","addr":"172.21.15.123:6831","nonce":1332143088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6828","nonce":1332143088},{"type":"v1","addr":"172.21.15.123:6829","nonce":1332143088}]},"public_addr":"172.21.15.123:6825/1332143088","cluster_addr":"172.21.15.123:6827/1332143088","heartbeat_back_addr":"172.21.15.123:6831/1332143088","heartbeat_front_addr":"172.21.15.123:6829/1332143088","state":["exists","up"]},{"osd":3,"uuid":"c6d1f817-f074-4325-95e1-578eba13edda","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6808","nonce":988586998},{"type":"v1","addr":"172.21.15.123:6809","nonce":988586998}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6810","nonce":988586998},{"type":"v1","addr":"172.21.15.123:6811","nonce":988586998}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6814","nonce":988586998},{"type":"v1","addr":"172.21.15.123:6815","nonce":988586998}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6812","nonce":988586998},{"type":"v1","addr":"172.21.15.123:6813","nonce":988586998}]},"public_addr":"172.21.15.123:6809/988586998","cluster_addr":"172.21.15.123:6811/988586998","heartbeat_back_addr":"172.21.15.123:6815/988586998","heartbeat_front_addr":"172.21.15.123:6813/988586998","state":["exists","up"]},{"osd":4,"uuid":"200a8df4-50ef-40a5-b48a-58981269c086","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6816","nonce":3387813890},{"type":"v1","addr":"172.21.15.184:6817","nonce":3387813890}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6818","nonce":3387813890},{"type":"v1","addr":"172.21.15.184:6819","nonce":3387813890}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6822","nonce":3387813890},{"type":"v1","addr":"172.21.15.184:6823","nonce":3387813890}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6820","nonce":3387813890},{"type":"v1","addr":"172.21.15.184:6821","nonce":3387813890}]},"public_addr":"172.21.15.184:6817/3387813890","cluster_addr":"172.21.15.184:6819/3387813890","heartbeat_back_addr":"172.21.15.184:6823/3387813890","heartbeat_front_addr":"172.21.15.184:6821/3387813890","state":["exists","up"]},{"osd":5,"uuid":"32d5a900-da3d-412c-bc19-13861a2368d8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6808","nonce":2680916017},{"type":"v1","addr":"172.21.15.184:6809","nonce":2680916017}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6810","nonce":2680916017},{"type":"v1","addr":"172.21.15.184:6811","nonce":2680916017}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6814","nonce":2680916017},{"type":"v1","addr":"172.21.15.184:6815","nonce":2680916017}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6812","nonce":2680916017},{"type":"v1","addr":"172.21.15.184:6813","nonce":2680916017}]},"public_addr":"172.21.15.184:6809/2680916017","cluster_addr":"172.21.15.184:6811/2680916017","heartbeat_back_addr":"172.21.15.184:6815/2680916017","heartbeat_front_addr":"172.21.15.184:6813/2680916017","state":["exists","up"]},{"osd":6,"uuid":"cde6f39e-aa1b-4260-ac19-774ee6aaca72","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6824","nonce":2183120457},{"type":"v1","addr":"172.21.15.184:6825","nonce":2183120457}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6826","nonce":2183120457},{"type":"v1","addr":"172.21.15.184:6827","nonce":2183120457}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6830","nonce":2183120457},{"type":"v1","addr":"172.21.15.184:6831","nonce":2183120457}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6828","nonce":2183120457},{"type":"v1","addr":"172.21.15.184:6829","nonce":2183120457}]},"public_addr":"172.21.15.184:6825/2183120457","cluster_addr":"172.21.15.184:6827/2183120457","heartbeat_back_addr":"172.21.15.184:6831/2183120457","heartbeat_front_addr":"172.21.15.184:6829/2183120457","state":["exists","up"]},{"osd":7,"uuid":"4f381778-eee1-47c6-804f-df0e6b33e313","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6800","nonce":4102957285},{"type":"v1","addr":"172.21.15.184:6801","nonce":4102957285}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6802","nonce":4102957285},{"type":"v1","addr":"172.21.15.184:6803","nonce":4102957285}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6806","nonce":4102957285},{"type":"v1","addr":"172.21.15.184:6807","nonce":4102957285}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6804","nonce":4102957285},{"type":"v1","addr":"172.21.15.184:6805","nonce":4102957285}]},"public_addr":"172.21.15.184:6801/4102957285","cluster_addr":"172.21.15.184:6803/4102957285","heartbeat_back_addr":"172.21.15.184:6807/4102957285","heartbeat_front_addr":"172.21.15.184:6805/4102957285","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:35.972210+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.222007+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.198617+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.385663+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.358935+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.156066+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.752973+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:35.682846+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T15:46:21.812 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:21.809+0000 7fc306e40640 1 -- 172.21.15.123:0/1412846550 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fc2e00730a0 msgr2=0x7fc2e0075560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:46:21.812 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:21.809+0000 7fc306e40640 1 --2- 172.21.15.123:0/1412846550 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fc2e00730a0 0x7fc2e0075560 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7fc2f8005d10 tx=0x7fc2f8005c80 comp rx=0 tx=0).stop 2024-09-16T15:46:21.812 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:21.809+0000 7fc306e40640 1 -- 172.21.15.123:0/1412846550 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc3001513e0 msgr2=0x7fc3000730e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:46:21.812 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:21.809+0000 7fc306e40640 1 --2- 172.21.15.123:0/1412846550 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc3001513e0 0x7fc3000730e0 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7fc300069550 tx=0x7fc2f000e9b0 comp rx=0 tx=0).stop 2024-09-16T15:46:21.813 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:21.809+0000 7fc306e40640 1 -- 172.21.15.123:0/1412846550 shutdown_connections 2024-09-16T15:46:21.813 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:21.809+0000 7fc306e40640 1 --2- 172.21.15.123:0/1412846550 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fc2e00730a0 0x7fc2e0075560 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:21.813 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:21.809+0000 7fc306e40640 1 --2- 172.21.15.123:0/1412846550 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc3001513e0 0x7fc3000730e0 unknown :-1 s=CLOSED pgs=631 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:21.813 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:21.809+0000 7fc306e40640 1 --2- 172.21.15.123:0/1412846550 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fc300150eb0 0x7fc300072ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:21.813 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:21.809+0000 7fc306e40640 1 --2- 172.21.15.123:0/1412846550 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc2ec004050 0x7fc3001507a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:21.813 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:21.809+0000 7fc306e40640 1 -- 172.21.15.123:0/1412846550 >> 172.21.15.123:0/1412846550 conn(0x7fc3000cfb10 msgr2=0x7fc30015ccb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:46:21.813 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:21.809+0000 7fc306e40640 1 -- 172.21.15.123:0/1412846550 shutdown_connections 2024-09-16T15:46:21.813 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:21.809+0000 7fc306e40640 1 -- 172.21.15.123:0/1412846550 wait complete. 2024-09-16T15:46:21.830 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-09-16T15:46:21.956 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:21.953+0000 7fef3b278640 1 -- 172.21.15.123:0/899941855 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fef20003fe0 msgr2=0x7fef20004810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:46:21.957 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:21.953+0000 7fef3b278640 1 --2- 172.21.15.123:0/899941855 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fef20003fe0 0x7fef20004810 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7fef2400c2e0 tx=0x7fef2401f6c0 comp rx=0 tx=0).stop 2024-09-16T15:46:21.957 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:21.953+0000 7fef39815640 1 -- 172.21.15.123:0/899941855 reap_dead start 2024-09-16T15:46:21.957 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:21.953+0000 7fef3b278640 1 -- 172.21.15.123:0/899941855 shutdown_connections 2024-09-16T15:46:21.957 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:21.953+0000 7fef3b278640 1 -- 172.21.15.123:0/899941855 >> 172.21.15.123:0/899941855 conn(0x7fef340d0b70 msgr2=0x7fef340d0f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:46:21.957 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:21.953+0000 7fef3b278640 1 -- 172.21.15.123:0/899941855 shutdown_connections 2024-09-16T15:46:21.957 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:21.953+0000 7fef3b278640 1 -- 172.21.15.123:0/899941855 wait complete. 2024-09-16T15:46:21.960 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:21.953+0000 7fef3b278640 1 Processor -- start 2024-09-16T15:46:21.960 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:21.953+0000 7fef3b278640 1 -- start start 2024-09-16T15:46:21.960 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:21.953+0000 7fef3b278640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fef20003fe0 0x7fef3413aa00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:46:21.960 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:21.953+0000 7fef3b278640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fef3415b7d0 0x7fef340ad8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:46:21.960 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:21.953+0000 7fef3b278640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fef340b4dd0 0x7fef340ae610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:46:21.961 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:21.953+0000 7fef3b278640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fef3406cbf0 con 0x7fef20003fe0 2024-09-16T15:46:21.961 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:21.953+0000 7fef3b278640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fef34145870 con 0x7fef340b4dd0 2024-09-16T15:46:21.961 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:21.953+0000 7fef3b278640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fef3406cd70 con 0x7fef3415b7d0 2024-09-16T15:46:21.961 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:21.953+0000 7fef39014640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fef20003fe0 0x7fef3413aa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:46:21.961 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:21.953+0000 7fef39014640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fef20003fe0 0x7fef3413aa00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:42286/0 (socket says 172.21.15.123:42286) 2024-09-16T15:46:21.961 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:21.953+0000 7fef39014640 1 -- 172.21.15.123:0/1654505034 learned_addr learned my addr 172.21.15.123:0/1654505034 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:46:21.961 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:21.953+0000 7fef38813640 1 --2- 172.21.15.123:0/1654505034 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fef3415b7d0 0x7fef340ad8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:46:21.961 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:21.953+0000 7fef39815640 1 --2- 172.21.15.123:0/1654505034 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fef340b4dd0 0x7fef340ae610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:46:21.961 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:21.953+0000 7fef39014640 1 -- 172.21.15.123:0/1654505034 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fef3415b7d0 msgr2=0x7fef340ad8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:46:21.961 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:21.953+0000 7fef39014640 1 --2- 172.21.15.123:0/1654505034 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fef3415b7d0 0x7fef340ad8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:21.961 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:21.953+0000 7fef39014640 1 -- 172.21.15.123:0/1654505034 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fef340b4dd0 msgr2=0x7fef340ae610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:46:21.961 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:21.953+0000 7fef39014640 1 --2- 172.21.15.123:0/1654505034 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fef340b4dd0 0x7fef340ae610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:21.962 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:21.953+0000 7fef39014640 1 -- 172.21.15.123:0/1654505034 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fef3413dd00 con 0x7fef20003fe0 2024-09-16T15:46:21.962 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:21.957+0000 7fef39014640 1 --2- 172.21.15.123:0/1654505034 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fef20003fe0 0x7fef3413aa00 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7fef3000e970 tx=0x7fef3000ee30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:46:21.962 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:21.957+0000 7fef29ffb640 1 -- 172.21.15.123:0/1654505034 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef3000ccc0 con 0x7fef20003fe0 2024-09-16T15:46:21.962 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:21.957+0000 7fef29ffb640 1 -- 172.21.15.123:0/1654505034 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fef3000ce60 con 0x7fef20003fe0 2024-09-16T15:46:21.962 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:21.957+0000 7fef3b278640 1 -- 172.21.15.123:0/1654505034 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fef3413dfe0 con 0x7fef20003fe0 2024-09-16T15:46:21.962 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:21.957+0000 7fef29ffb640 1 -- 172.21.15.123:0/1654505034 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef30005260 con 0x7fef20003fe0 2024-09-16T15:46:21.964 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:21.957+0000 7fef3b278640 1 -- 172.21.15.123:0/1654505034 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fef340b45d0 con 0x7fef20003fe0 2024-09-16T15:46:21.964 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:21.957+0000 7fef3b278640 1 -- 172.21.15.123:0/1654505034 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fef3406d0f0 con 0x7fef20003fe0 2024-09-16T15:46:21.965 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:21.957+0000 7fef29ffb640 1 -- 172.21.15.123:0/1654505034 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 9) ==== 97233+0+0 (secure 0 0 0) 0x7fef30005400 con 0x7fef20003fe0 2024-09-16T15:46:21.965 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:21.957+0000 7fef29ffb640 1 --2- 172.21.15.123:0/1654505034 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fef18073070 0x7fef18075530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:46:21.965 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:21.957+0000 7fef29ffb640 1 -- 172.21.15.123:0/1654505034 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(57..57 src has 1..57) ==== 5709+0+0 (secure 0 0 0) 0x7fef30094ab0 con 0x7fef20003fe0 2024-09-16T15:46:21.965 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:21.957+0000 7fef38813640 1 --2- 172.21.15.123:0/1654505034 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fef18073070 0x7fef18075530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:46:21.965 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:21.957+0000 7fef38813640 1 --2- 172.21.15.123:0/1654505034 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fef18073070 0x7fef18075530 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7fef240b0000 tx=0x7fef2401c500 comp rx=0 tx=0).ready entity=mgr.4135 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:46:21.965 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:21.961+0000 7fef29ffb640 1 -- 172.21.15.123:0/1654505034 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7fef3005fa00 con 0x7fef20003fe0 2024-09-16T15:46:22.176 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:22.173+0000 7fef3b278640 1 -- 172.21.15.123:0/1654505034 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) -- 0x7fef34141e70 con 0x7fef20003fe0 2024-09-16T15:46:23.186 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:23.181+0000 7fef29ffb640 1 -- 172.21.15.123:0/1654505034 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v35) ==== 118+0+0 (secure 0 0 0) 0x7fef3005f300 con 0x7fef20003fe0 2024-09-16T15:46:23.189 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:23.185+0000 7fef3b278640 1 -- 172.21.15.123:0/1654505034 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fef18073070 msgr2=0x7fef18075530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:46:23.190 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:23.185+0000 7fef3b278640 1 --2- 172.21.15.123:0/1654505034 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fef18073070 0x7fef18075530 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7fef240b0000 tx=0x7fef2401c500 comp rx=0 tx=0).stop 2024-09-16T15:46:23.190 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:23.185+0000 7fef3b278640 1 -- 172.21.15.123:0/1654505034 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fef20003fe0 msgr2=0x7fef3413aa00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:46:23.190 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:23.185+0000 7fef3b278640 1 --2- 172.21.15.123:0/1654505034 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fef20003fe0 0x7fef3413aa00 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7fef3000e970 tx=0x7fef3000ee30 comp rx=0 tx=0).stop 2024-09-16T15:46:23.190 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:23.185+0000 7fef3b278640 1 -- 172.21.15.123:0/1654505034 shutdown_connections 2024-09-16T15:46:23.190 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:23.185+0000 7fef3b278640 1 --2- 172.21.15.123:0/1654505034 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fef18073070 0x7fef18075530 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:23.190 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:23.185+0000 7fef3b278640 1 --2- 172.21.15.123:0/1654505034 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fef340b4dd0 0x7fef340ae610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:23.190 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:23.185+0000 7fef3b278640 1 --2- 172.21.15.123:0/1654505034 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fef3415b7d0 0x7fef340ad8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:23.190 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:23.185+0000 7fef3b278640 1 --2- 172.21.15.123:0/1654505034 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fef20003fe0 0x7fef3413aa00 unknown :-1 s=CLOSED pgs=634 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:23.190 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:23.185+0000 7fef3b278640 1 -- 172.21.15.123:0/1654505034 >> 172.21.15.123:0/1654505034 conn(0x7fef340d0b70 msgr2=0x7fef34140800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:46:23.190 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:23.185+0000 7fef3b278640 1 -- 172.21.15.123:0/1654505034 shutdown_connections 2024-09-16T15:46:23.190 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:23.185+0000 7fef3b278640 1 -- 172.21.15.123:0/1654505034 wait complete. 2024-09-16T15:46:23.209 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-09-16T15:46:23.209 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-09-16T15:46:23.341 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:23.337+0000 7f223a71c640 1 -- 172.21.15.123:0/3014809908 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f22340c3230 msgr2=0x7f22340cedd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:46:23.341 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:23.337+0000 7f223a71c640 1 --2- 172.21.15.123:0/3014809908 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f22340c3230 0x7f22340cedd0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f221c009840 tx=0x7f221c01c820 comp rx=0 tx=0).stop 2024-09-16T15:46:23.341 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:23.337+0000 7f223a71c640 1 -- 172.21.15.123:0/3014809908 shutdown_connections 2024-09-16T15:46:23.341 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:23.337+0000 7f223a71c640 1 --2- 172.21.15.123:0/3014809908 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f22340c3230 0x7f22340cedd0 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:23.342 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:23.337+0000 7f223a71c640 1 --2- 172.21.15.123:0/3014809908 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f223415b860 0x7f223406fca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:23.342 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:23.337+0000 7f223a71c640 1 -- 172.21.15.123:0/3014809908 >> 172.21.15.123:0/3014809908 conn(0x7f22340b1710 msgr2=0x7f22340b1b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:46:23.342 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:23.337+0000 7f223a71c640 1 -- 172.21.15.123:0/3014809908 shutdown_connections 2024-09-16T15:46:23.342 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:23.337+0000 7f223a71c640 1 -- 172.21.15.123:0/3014809908 wait complete. 2024-09-16T15:46:23.343 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:23.341+0000 7f223a71c640 1 Processor -- start 2024-09-16T15:46:23.343 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:23.341+0000 7f223a71c640 1 -- start start 2024-09-16T15:46:23.345 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:23.341+0000 7f223a71c640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f22340701e0 0x7f2234143f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:46:23.345 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:23.341+0000 7f223a71c640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f22340c3230 0x7f22340c6c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:46:23.345 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:23.341+0000 7f223a71c640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f223415b860 0x7f22340c7150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:46:23.346 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:23.341+0000 7f223a71c640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f2234169190 con 0x7f223415b860 2024-09-16T15:46:23.346 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:23.341+0000 7f223a71c640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f2234169010 con 0x7f22340701e0 2024-09-16T15:46:23.346 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:23.341+0000 7f223a71c640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f2234169310 con 0x7f22340c3230 2024-09-16T15:46:23.346 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:23.341+0000 7f2238cb9640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f223415b860 0x7f22340c7150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:46:23.346 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:23.341+0000 7f2238cb9640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f223415b860 0x7f22340c7150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:54614/0 (socket says 172.21.15.123:54614) 2024-09-16T15:46:23.346 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:23.341+0000 7f2238cb9640 1 -- 172.21.15.123:0/1460983970 learned_addr learned my addr 172.21.15.123:0/1460983970 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:46:23.346 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:23.341+0000 7f2233fff640 1 --2- 172.21.15.123:0/1460983970 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f22340701e0 0x7f2234143f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:46:23.346 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:23.341+0000 7f222bfff640 1 --2- 172.21.15.123:0/1460983970 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f22340c3230 0x7f22340c6c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:46:23.346 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:23.341+0000 7f2238cb9640 1 -- 172.21.15.123:0/1460983970 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f22340c3230 msgr2=0x7f22340c6c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:46:23.346 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:23.341+0000 7f2238cb9640 1 --2- 172.21.15.123:0/1460983970 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f22340c3230 0x7f22340c6c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:23.347 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:23.341+0000 7f2238cb9640 1 -- 172.21.15.123:0/1460983970 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f22340701e0 msgr2=0x7f2234143f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:46:23.347 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:23.341+0000 7f2238cb9640 1 --2- 172.21.15.123:0/1460983970 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f22340701e0 0x7f2234143f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:23.347 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:23.341+0000 7f2238cb9640 1 -- 172.21.15.123:0/1460983970 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f22340c2650 con 0x7f223415b860 2024-09-16T15:46:23.347 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:23.341+0000 7f222bfff640 1 --2- 172.21.15.123:0/1460983970 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f22340c3230 0x7f22340c6c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T15:46:23.347 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:23.341+0000 7f2238cb9640 1 --2- 172.21.15.123:0/1460983970 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f223415b860 0x7f22340c7150 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7f222c00c7a0 tx=0x7f222c00cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:46:23.347 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:23.341+0000 7f22317fa640 1 -- 172.21.15.123:0/1460983970 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f222c003b80 con 0x7f223415b860 2024-09-16T15:46:23.347 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:23.341+0000 7f22317fa640 1 -- 172.21.15.123:0/1460983970 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f222c003d40 con 0x7f223415b860 2024-09-16T15:46:23.347 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:23.341+0000 7f223a71c640 1 -- 172.21.15.123:0/1460983970 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f22340c2200 con 0x7f223415b860 2024-09-16T15:46:23.347 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:23.341+0000 7f22317fa640 1 -- 172.21.15.123:0/1460983970 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f222c00aab0 con 0x7f223415b860 2024-09-16T15:46:23.347 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:23.341+0000 7f223a71c640 1 -- 172.21.15.123:0/1460983970 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2234144450 con 0x7f223415b860 2024-09-16T15:46:23.347 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:23.341+0000 7f223a71c640 1 -- 172.21.15.123:0/1460983970 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f22340cedd0 con 0x7f223415b860 2024-09-16T15:46:23.348 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:23.341+0000 7f22317fa640 1 -- 172.21.15.123:0/1460983970 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 9) ==== 97233+0+0 (secure 0 0 0) 0x7f222c00ac50 con 0x7f223415b860 2024-09-16T15:46:23.348 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:23.345+0000 7f22317fa640 1 --2- 172.21.15.123:0/1460983970 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f2214073070 0x7f2214075530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:46:23.348 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:23.345+0000 7f22317fa640 1 -- 172.21.15.123:0/1460983970 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(58..58 src has 1..58) ==== 5709+0+0 (secure 0 0 0) 0x7f222c094060 con 0x7f223415b860 2024-09-16T15:46:23.349 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:23.345+0000 7f2233fff640 1 --2- 172.21.15.123:0/1460983970 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f2214073070 0x7f2214075530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:46:23.349 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:23.345+0000 7f2233fff640 1 --2- 172.21.15.123:0/1460983970 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f2214073070 0x7f2214075530 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f2224009ad0 tx=0x7f2224009260 comp rx=0 tx=0).ready entity=mgr.4135 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:46:23.351 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:23.349+0000 7f22317fa640 1 -- 172.21.15.123:0/1460983970 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f222c05ef20 con 0x7f223415b860 2024-09-16T15:46:23.561 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:23.557+0000 7f223a71c640 1 -- 172.21.15.123:0/1460983970 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) -- 0x7f22340c96f0 con 0x7f223415b860 2024-09-16T15:46:24.194 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:24.189+0000 7f22317fa640 1 -- 172.21.15.123:0/1460983970 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v36) ==== 122+0+0 (secure 0 0 0) 0x7f222c05e820 con 0x7f223415b860 2024-09-16T15:46:24.198 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:24.193+0000 7f223a71c640 1 -- 172.21.15.123:0/1460983970 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f2214073070 msgr2=0x7f2214075530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:46:24.198 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:24.193+0000 7f223a71c640 1 --2- 172.21.15.123:0/1460983970 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f2214073070 0x7f2214075530 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f2224009ad0 tx=0x7f2224009260 comp rx=0 tx=0).stop 2024-09-16T15:46:24.198 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:24.193+0000 7f223a71c640 1 -- 172.21.15.123:0/1460983970 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f223415b860 msgr2=0x7f22340c7150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:46:24.199 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:24.193+0000 7f223a71c640 1 --2- 172.21.15.123:0/1460983970 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f223415b860 0x7f22340c7150 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7f222c00c7a0 tx=0x7f222c00cc60 comp rx=0 tx=0).stop 2024-09-16T15:46:24.199 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:24.193+0000 7f223a71c640 1 -- 172.21.15.123:0/1460983970 shutdown_connections 2024-09-16T15:46:24.199 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:24.193+0000 7f223a71c640 1 --2- 172.21.15.123:0/1460983970 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f2214073070 0x7f2214075530 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:24.199 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:24.193+0000 7f223a71c640 1 --2- 172.21.15.123:0/1460983970 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f223415b860 0x7f22340c7150 unknown :-1 s=CLOSED pgs=635 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:24.199 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:24.193+0000 7f223a71c640 1 --2- 172.21.15.123:0/1460983970 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f22340c3230 0x7f22340c6c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:24.199 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:24.193+0000 7f223a71c640 1 --2- 172.21.15.123:0/1460983970 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f22340701e0 0x7f2234143f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:24.199 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:24.193+0000 7f223a71c640 1 -- 172.21.15.123:0/1460983970 >> 172.21.15.123:0/1460983970 conn(0x7f22340b1710 msgr2=0x7f2234147c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:46:24.199 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:24.193+0000 7f223a71c640 1 -- 172.21.15.123:0/1460983970 shutdown_connections 2024-09-16T15:46:24.200 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:24.193+0000 7f223a71c640 1 -- 172.21.15.123:0/1460983970 wait complete. 2024-09-16T15:46:24.220 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-09-16T15:46:24.221 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-16T15:46:24.352 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:24.349+0000 7f46a1677640 1 -- 172.21.15.123:0/102311179 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4684004810 msgr2=0x7f4684004c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:46:24.352 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:24.349+0000 7f46a1677640 1 --2- 172.21.15.123:0/102311179 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4684004810 0x7f4684004c90 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7f46900016f0 tx=0x7f469001c7b0 comp rx=0 tx=0).stop 2024-09-16T15:46:24.352 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:24.349+0000 7f469b7fe640 1 -- 172.21.15.123:0/102311179 reap_dead start 2024-09-16T15:46:24.352 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:24.349+0000 7f46a1677640 1 -- 172.21.15.123:0/102311179 shutdown_connections 2024-09-16T15:46:24.352 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:24.349+0000 7f46a1677640 1 -- 172.21.15.123:0/102311179 >> 172.21.15.123:0/102311179 conn(0x7f469c0cfb10 msgr2=0x7f469c0cff20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:46:24.352 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:24.349+0000 7f46a1677640 1 -- 172.21.15.123:0/102311179 shutdown_connections 2024-09-16T15:46:24.353 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:24.349+0000 7f46a1677640 1 -- 172.21.15.123:0/102311179 wait complete. 2024-09-16T15:46:24.357 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:24.349+0000 7f46a1677640 1 Processor -- start 2024-09-16T15:46:24.357 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:24.349+0000 7f46a1677640 1 -- start start 2024-09-16T15:46:24.357 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:24.349+0000 7f46a1677640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4684004810 0x7f469c150960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:46:24.357 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:24.349+0000 7f46a1677640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f469c150010 0x7f469c13d8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:46:24.357 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:24.349+0000 7f46a1677640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f469c0c2440 0x7f469c0c28c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:46:24.357 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:24.349+0000 7f46a1677640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f469c166bc0 con 0x7f4684004810 2024-09-16T15:46:24.357 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:24.349+0000 7f46a1677640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f469c0cf640 con 0x7f469c0c2440 2024-09-16T15:46:24.358 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:24.349+0000 7f46a1677640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f469c166d40 con 0x7f469c150010 2024-09-16T15:46:24.358 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:24.349+0000 7f469affd640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4684004810 0x7f469c150960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:46:24.358 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:24.349+0000 7f469affd640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4684004810 0x7f469c150960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:54646/0 (socket says 172.21.15.123:54646) 2024-09-16T15:46:24.358 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:24.349+0000 7f469affd640 1 -- 172.21.15.123:0/1134678303 learned_addr learned my addr 172.21.15.123:0/1134678303 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:46:24.358 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:24.349+0000 7f469b7fe640 1 --2- 172.21.15.123:0/1134678303 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f469c0c2440 0x7f469c0c28c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:46:24.358 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:24.349+0000 7f469affd640 1 -- 172.21.15.123:0/1134678303 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f469c150010 msgr2=0x7f469c13d8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:46:24.358 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:24.349+0000 7f469a7fc640 1 --2- 172.21.15.123:0/1134678303 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f469c150010 0x7f469c13d8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:46:24.359 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:24.349+0000 7f469affd640 1 --2- 172.21.15.123:0/1134678303 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f469c150010 0x7f469c13d8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:24.359 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:24.349+0000 7f469affd640 1 -- 172.21.15.123:0/1134678303 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f469c0c2440 msgr2=0x7f469c0c28c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:46:24.359 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:24.349+0000 7f469affd640 1 --2- 172.21.15.123:0/1134678303 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f469c0c2440 0x7f469c0c28c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:24.359 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:24.349+0000 7f469affd640 1 -- 172.21.15.123:0/1134678303 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f469c140850 con 0x7f4684004810 2024-09-16T15:46:24.359 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:24.349+0000 7f469b7fe640 1 --2- 172.21.15.123:0/1134678303 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f469c0c2440 0x7f469c0c28c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T15:46:24.359 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:24.349+0000 7f469affd640 1 --2- 172.21.15.123:0/1134678303 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4684004810 0x7f469c150960 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7f468c00d920 tx=0x7f468c00dd20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:46:24.360 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:24.349+0000 7f46837fe640 1 -- 172.21.15.123:0/1134678303 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f468c00b830 con 0x7f4684004810 2024-09-16T15:46:24.360 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:24.349+0000 7f46837fe640 1 -- 172.21.15.123:0/1134678303 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f468c00bac0 con 0x7f4684004810 2024-09-16T15:46:24.360 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:24.349+0000 7f46a1677640 1 -- 172.21.15.123:0/1134678303 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f469c1523f0 con 0x7f4684004810 2024-09-16T15:46:24.360 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:24.349+0000 7f46837fe640 1 -- 172.21.15.123:0/1134678303 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f468c003600 con 0x7f4684004810 2024-09-16T15:46:24.360 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:24.349+0000 7f46a1677640 1 -- 172.21.15.123:0/1134678303 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f469c1529b0 con 0x7f4684004810 2024-09-16T15:46:24.360 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:24.353+0000 7f46a1677640 1 -- 172.21.15.123:0/1134678303 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f469c15e820 con 0x7f4684004810 2024-09-16T15:46:24.361 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:24.353+0000 7f46837fe640 1 -- 172.21.15.123:0/1134678303 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 9) ==== 97233+0+0 (secure 0 0 0) 0x7f468c0037a0 con 0x7f4684004810 2024-09-16T15:46:24.361 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:24.353+0000 7f46837fe640 1 --2- 172.21.15.123:0/1134678303 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f4688073070 0x7f4688075530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:46:24.361 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:24.353+0000 7f46837fe640 1 -- 172.21.15.123:0/1134678303 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(58..58 src has 1..58) ==== 5709+0+0 (secure 0 0 0) 0x7f468c0948b0 con 0x7f4684004810 2024-09-16T15:46:24.361 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:24.353+0000 7f469a7fc640 1 --2- 172.21.15.123:0/1134678303 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f4688073070 0x7f4688075530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:46:24.362 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:24.353+0000 7f469a7fc640 1 --2- 172.21.15.123:0/1134678303 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f4688073070 0x7f4688075530 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f46900016c0 tx=0x7f46900a8040 comp rx=0 tx=0).ready entity=mgr.4135 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:46:24.362 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:24.357+0000 7f46837fe640 1 -- 172.21.15.123:0/1134678303 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f468c05f8c0 con 0x7f4684004810 2024-09-16T15:46:24.572 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:24.569+0000 7f46a1677640 1 -- 172.21.15.123:0/1134678303 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f469c0d06c0 con 0x7f4684004810 2024-09-16T15:46:24.573 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:24.569+0000 7f46837fe640 1 -- 172.21.15.123:0/1134678303 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 36 v36) ==== 94+0+5232 (secure 0 0 0) 0x7f468c05f1c0 con 0x7f4684004810 2024-09-16T15:46:24.573 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-09-16T15:46:24.573 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":36,"btime":"2024-09-16T15:46:24:187144+0000","default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4901,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6835/4036392311","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":4036392311},{"type":"v1","addr":"172.21.15.184:6835","nonce":4036392311}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":31},{"gid":5203,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6837/436207465","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6836","nonce":436207465},{"type":"v1","addr":"172.21.15.184:6837","nonce":436207465}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":31},{"gid":5796,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6834/4239187724","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6832","nonce":4239187724},{"type":"v1","addr":"172.21.15.123:6834","nonce":4239187724}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":31}],"filesystems":[{"mdsmap":{"epoch":36,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-16T15:46:20.516000+0000","modified":"2024-09-16T15:46:23.563031+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5799},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5799":{"gid":5799,"name":"c","rank":0,"incarnation":32,"state":"up:active","state_seq":3,"addr":"172.21.15.123:6835/3589624203","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6833","nonce":3589624203},{"type":"v1","addr":"172.21.15.123:6835","nonce":3589624203}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[9],"metadata_pool":8,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5799,"qdb_cluster":[5799]},"id":4}]} 2024-09-16T15:46:24.574 INFO:teuthology.orchestra.run.smithi123.stderr:dumped fsmap epoch 36 2024-09-16T15:46:24.576 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:24.573+0000 7f46a1677640 1 -- 172.21.15.123:0/1134678303 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f4688073070 msgr2=0x7f4688075530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:46:24.576 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:24.573+0000 7f46a1677640 1 --2- 172.21.15.123:0/1134678303 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f4688073070 0x7f4688075530 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f46900016c0 tx=0x7f46900a8040 comp rx=0 tx=0).stop 2024-09-16T15:46:24.576 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:24.573+0000 7f46a1677640 1 -- 172.21.15.123:0/1134678303 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4684004810 msgr2=0x7f469c150960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:46:24.576 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:24.573+0000 7f46a1677640 1 --2- 172.21.15.123:0/1134678303 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4684004810 0x7f469c150960 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7f468c00d920 tx=0x7f468c00dd20 comp rx=0 tx=0).stop 2024-09-16T15:46:24.576 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:24.573+0000 7f46a1677640 1 -- 172.21.15.123:0/1134678303 shutdown_connections 2024-09-16T15:46:24.576 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:24.573+0000 7f46a1677640 1 --2- 172.21.15.123:0/1134678303 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f4688073070 0x7f4688075530 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:24.576 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:24.573+0000 7f46a1677640 1 --2- 172.21.15.123:0/1134678303 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f469c0c2440 0x7f469c0c28c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:24.577 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:24.573+0000 7f46a1677640 1 --2- 172.21.15.123:0/1134678303 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f469c150010 0x7f469c13d8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:24.577 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:24.573+0000 7f46a1677640 1 --2- 172.21.15.123:0/1134678303 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4684004810 0x7f469c150960 unknown :-1 s=CLOSED pgs=638 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:24.577 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:24.573+0000 7f46a1677640 1 -- 172.21.15.123:0/1134678303 >> 172.21.15.123:0/1134678303 conn(0x7f469c0cfb10 msgr2=0x7f469c0b5120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:46:24.577 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:24.573+0000 7f46a1677640 1 -- 172.21.15.123:0/1134678303 shutdown_connections 2024-09-16T15:46:24.577 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:24.573+0000 7f46a1677640 1 -- 172.21.15.123:0/1134678303 wait complete. 2024-09-16T15:46:24.592 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-16T15:46:24.714 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:24.709+0000 7f40e7a7c640 1 -- 172.21.15.123:0/3917971317 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f40e00cf5a0 msgr2=0x7f40e006eee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:46:24.714 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:24.709+0000 7f40e7a7c640 1 --2- 172.21.15.123:0/3917971317 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f40e00cf5a0 0x7f40e006eee0 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7f40e0076280 tx=0x7f40d001dad0 comp rx=0 tx=0).stop 2024-09-16T15:46:24.714 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:24.709+0000 7f40e7a7c640 1 -- 172.21.15.123:0/3917971317 shutdown_connections 2024-09-16T15:46:24.715 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:24.709+0000 7f40e7a7c640 1 --2- 172.21.15.123:0/3917971317 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f40e00cf5a0 0x7f40e006eee0 unknown :-1 s=CLOSED pgs=640 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:24.715 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:24.709+0000 7f40e7a7c640 1 --2- 172.21.15.123:0/3917971317 >> v2:172.21.15.184:3300/0 conn(0x7f40e0158120 0x7f40e00ca320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:24.715 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:24.709+0000 7f40e7a7c640 1 -- 172.21.15.123:0/3917971317 >> 172.21.15.123:0/3917971317 conn(0x7f40e00cfb10 msgr2=0x7f40e00cff20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:46:24.715 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:24.709+0000 7f40e7a7c640 1 -- 172.21.15.123:0/3917971317 shutdown_connections 2024-09-16T15:46:24.715 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:24.709+0000 7f40e7a7c640 1 -- 172.21.15.123:0/3917971317 wait complete. 2024-09-16T15:46:24.716 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:24.713+0000 7f40e7a7c640 1 Processor -- start 2024-09-16T15:46:24.716 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:24.713+0000 7f40e7a7c640 1 -- start start 2024-09-16T15:46:24.718 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:24.713+0000 7f40e7a7c640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f40e00cf5a0 0x7f40e015aa70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:46:24.719 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:24.713+0000 7f40e7a7c640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f40e0158120 0x7f40e01f9590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:46:24.719 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:24.713+0000 7f40e7a7c640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f40e015ec40 0x7f40e0149890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:46:24.719 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:24.713+0000 7f40e7a7c640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f40e015b740 con 0x7f40e00cf5a0 2024-09-16T15:46:24.719 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:24.713+0000 7f40e7a7c640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f40e015b5c0 con 0x7f40e0158120 2024-09-16T15:46:24.719 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:24.713+0000 7f40e7a7c640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f40e01673d0 con 0x7f40e015ec40 2024-09-16T15:46:24.719 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:24.713+0000 7f40e5818640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f40e00cf5a0 0x7f40e015aa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:46:24.719 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:24.713+0000 7f40e5818640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f40e00cf5a0 0x7f40e015aa70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:54668/0 (socket says 172.21.15.123:54668) 2024-09-16T15:46:24.719 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:24.713+0000 7f40e5818640 1 -- 172.21.15.123:0/3911583535 learned_addr learned my addr 172.21.15.123:0/3911583535 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:46:24.719 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:24.713+0000 7f40e5017640 1 --2- 172.21.15.123:0/3911583535 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f40e0158120 0x7f40e01f9590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:46:24.720 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:24.713+0000 7f40e6019640 1 --2- 172.21.15.123:0/3911583535 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f40e015ec40 0x7f40e0149890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:46:24.720 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:24.713+0000 7f40e5818640 1 -- 172.21.15.123:0/3911583535 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f40e015ec40 msgr2=0x7f40e0149890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:46:24.720 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:24.713+0000 7f40e5818640 1 --2- 172.21.15.123:0/3911583535 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f40e015ec40 0x7f40e0149890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:24.720 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:24.713+0000 7f40e5818640 1 -- 172.21.15.123:0/3911583535 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f40e0158120 msgr2=0x7f40e01f9590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:46:24.720 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:24.713+0000 7f40e5818640 1 --2- 172.21.15.123:0/3911583535 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f40e0158120 0x7f40e01f9590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:24.720 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:24.713+0000 7f40e5818640 1 -- 172.21.15.123:0/3911583535 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f40e00c6050 con 0x7f40e00cf5a0 2024-09-16T15:46:24.720 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:24.713+0000 7f40e5818640 1 --2- 172.21.15.123:0/3911583535 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f40e00cf5a0 0x7f40e015aa70 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7f40dc00d950 tx=0x7f40dc00de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:46:24.720 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:24.713+0000 7f40d67fc640 1 -- 172.21.15.123:0/3911583535 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40dc017070 con 0x7f40e00cf5a0 2024-09-16T15:46:24.721 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:24.713+0000 7f40d67fc640 1 -- 172.21.15.123:0/3911583535 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f40dc009d20 con 0x7f40e00cf5a0 2024-09-16T15:46:24.721 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:24.713+0000 7f40e7a7c640 1 -- 172.21.15.123:0/3911583535 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f40e00c6310 con 0x7f40e00cf5a0 2024-09-16T15:46:24.721 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:24.713+0000 7f40d67fc640 1 -- 172.21.15.123:0/3911583535 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40dc0106c0 con 0x7f40e00cf5a0 2024-09-16T15:46:24.721 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:24.713+0000 7f40e7a7c640 1 -- 172.21.15.123:0/3911583535 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f40e00c5790 con 0x7f40e00cf5a0 2024-09-16T15:46:24.721 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:24.713+0000 7f40e7a7c640 1 -- 172.21.15.123:0/3911583535 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f40e006eee0 con 0x7f40e00cf5a0 2024-09-16T15:46:24.722 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:24.717+0000 7f40d67fc640 1 -- 172.21.15.123:0/3911583535 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 9) ==== 97233+0+0 (secure 0 0 0) 0x7f40dc010860 con 0x7f40e00cf5a0 2024-09-16T15:46:24.722 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:24.717+0000 7f40d67fc640 1 --2- 172.21.15.123:0/3911583535 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f40c4073070 0x7f40c4075530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:46:24.722 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:24.717+0000 7f40d67fc640 1 -- 172.21.15.123:0/3911583535 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(58..58 src has 1..58) ==== 5709+0+0 (secure 0 0 0) 0x7f40dc0949b0 con 0x7f40e00cf5a0 2024-09-16T15:46:24.722 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:24.717+0000 7f40e5017640 1 --2- 172.21.15.123:0/3911583535 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f40c4073070 0x7f40c4075530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:46:24.722 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:24.717+0000 7f40e5017640 1 --2- 172.21.15.123:0/3911583535 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f40c4073070 0x7f40c4075530 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f40d00b0000 tx=0x7f40d001c310 comp rx=0 tx=0).ready entity=mgr.4135 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:46:24.723 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:24.721+0000 7f40d67fc640 1 -- 172.21.15.123:0/3911583535 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f40dc05f9e0 con 0x7f40e00cf5a0 2024-09-16T15:46:24.896 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:24.893+0000 7f40e7a7c640 1 -- 172.21.15.123:0/3911583535 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f40e00d06c0 con 0x7f40e00cf5a0 2024-09-16T15:46:24.897 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:24.893+0000 7f40d67fc640 1 -- 172.21.15.123:0/3911583535 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v58) ==== 74+0+16785 (secure 0 0 0) 0x7f40dc05f2e0 con 0x7f40e00cf5a0 2024-09-16T15:46:24.898 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-09-16T15:46:24.898 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":58,"fsid":"56077f89-18c8-486f-ae7e-eecc0bb7822c","created":"2024-09-16T15:42:26.364892+0000","modified":"2024-09-16T15:46:22.140001+0000","last_up_change":"2024-09-16T15:42:38.707234+0000","last_in_change":"2024-09-16T15:42:29.423662+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-16T15:42:39.287764+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_metadata","create_time":"2024-09-16T15:46:18.173124+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"57","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_data","create_time":"2024-09-16T15:46:19.478016+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"58","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9e7c1955-a607-4671-9be7-f2ef760ceeaa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6800","nonce":3882044969},{"type":"v1","addr":"172.21.15.123:6801","nonce":3882044969}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6802","nonce":3882044969},{"type":"v1","addr":"172.21.15.123:6803","nonce":3882044969}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6806","nonce":3882044969},{"type":"v1","addr":"172.21.15.123:6807","nonce":3882044969}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6804","nonce":3882044969},{"type":"v1","addr":"172.21.15.123:6805","nonce":3882044969}]},"public_addr":"172.21.15.123:6801/3882044969","cluster_addr":"172.21.15.123:6803/3882044969","heartbeat_back_addr":"172.21.15.123:6807/3882044969","heartbeat_front_addr":"172.21.15.123:6805/3882044969","state":["exists","up"]},{"osd":1,"uuid":"c540cfde-0977-4742-9b17-fd634668e010","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6816","nonce":4037001850},{"type":"v1","addr":"172.21.15.123:6817","nonce":4037001850}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6818","nonce":4037001850},{"type":"v1","addr":"172.21.15.123:6819","nonce":4037001850}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6822","nonce":4037001850},{"type":"v1","addr":"172.21.15.123:6823","nonce":4037001850}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6820","nonce":4037001850},{"type":"v1","addr":"172.21.15.123:6821","nonce":4037001850}]},"public_addr":"172.21.15.123:6817/4037001850","cluster_addr":"172.21.15.123:6819/4037001850","heartbeat_back_addr":"172.21.15.123:6823/4037001850","heartbeat_front_addr":"172.21.15.123:6821/4037001850","state":["exists","up"]},{"osd":2,"uuid":"9864a496-f169-486b-95c4-1d59162793a7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6824","nonce":1332143088},{"type":"v1","addr":"172.21.15.123:6825","nonce":1332143088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6826","nonce":1332143088},{"type":"v1","addr":"172.21.15.123:6827","nonce":1332143088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6830","nonce":1332143088},{"type":"v1","addr":"172.21.15.123:6831","nonce":1332143088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6828","nonce":1332143088},{"type":"v1","addr":"172.21.15.123:6829","nonce":1332143088}]},"public_addr":"172.21.15.123:6825/1332143088","cluster_addr":"172.21.15.123:6827/1332143088","heartbeat_back_addr":"172.21.15.123:6831/1332143088","heartbeat_front_addr":"172.21.15.123:6829/1332143088","state":["exists","up"]},{"osd":3,"uuid":"c6d1f817-f074-4325-95e1-578eba13edda","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6808","nonce":988586998},{"type":"v1","addr":"172.21.15.123:6809","nonce":988586998}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6810","nonce":988586998},{"type":"v1","addr":"172.21.15.123:6811","nonce":988586998}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6814","nonce":988586998},{"type":"v1","addr":"172.21.15.123:6815","nonce":988586998}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6812","nonce":988586998},{"type":"v1","addr":"172.21.15.123:6813","nonce":988586998}]},"public_addr":"172.21.15.123:6809/988586998","cluster_addr":"172.21.15.123:6811/988586998","heartbeat_back_addr":"172.21.15.123:6815/988586998","heartbeat_front_addr":"172.21.15.123:6813/988586998","state":["exists","up"]},{"osd":4,"uuid":"200a8df4-50ef-40a5-b48a-58981269c086","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6816","nonce":3387813890},{"type":"v1","addr":"172.21.15.184:6817","nonce":3387813890}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6818","nonce":3387813890},{"type":"v1","addr":"172.21.15.184:6819","nonce":3387813890}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6822","nonce":3387813890},{"type":"v1","addr":"172.21.15.184:6823","nonce":3387813890}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6820","nonce":3387813890},{"type":"v1","addr":"172.21.15.184:6821","nonce":3387813890}]},"public_addr":"172.21.15.184:6817/3387813890","cluster_addr":"172.21.15.184:6819/3387813890","heartbeat_back_addr":"172.21.15.184:6823/3387813890","heartbeat_front_addr":"172.21.15.184:6821/3387813890","state":["exists","up"]},{"osd":5,"uuid":"32d5a900-da3d-412c-bc19-13861a2368d8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6808","nonce":2680916017},{"type":"v1","addr":"172.21.15.184:6809","nonce":2680916017}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6810","nonce":2680916017},{"type":"v1","addr":"172.21.15.184:6811","nonce":2680916017}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6814","nonce":2680916017},{"type":"v1","addr":"172.21.15.184:6815","nonce":2680916017}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6812","nonce":2680916017},{"type":"v1","addr":"172.21.15.184:6813","nonce":2680916017}]},"public_addr":"172.21.15.184:6809/2680916017","cluster_addr":"172.21.15.184:6811/2680916017","heartbeat_back_addr":"172.21.15.184:6815/2680916017","heartbeat_front_addr":"172.21.15.184:6813/2680916017","state":["exists","up"]},{"osd":6,"uuid":"cde6f39e-aa1b-4260-ac19-774ee6aaca72","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6824","nonce":2183120457},{"type":"v1","addr":"172.21.15.184:6825","nonce":2183120457}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6826","nonce":2183120457},{"type":"v1","addr":"172.21.15.184:6827","nonce":2183120457}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6830","nonce":2183120457},{"type":"v1","addr":"172.21.15.184:6831","nonce":2183120457}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6828","nonce":2183120457},{"type":"v1","addr":"172.21.15.184:6829","nonce":2183120457}]},"public_addr":"172.21.15.184:6825/2183120457","cluster_addr":"172.21.15.184:6827/2183120457","heartbeat_back_addr":"172.21.15.184:6831/2183120457","heartbeat_front_addr":"172.21.15.184:6829/2183120457","state":["exists","up"]},{"osd":7,"uuid":"4f381778-eee1-47c6-804f-df0e6b33e313","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6800","nonce":4102957285},{"type":"v1","addr":"172.21.15.184:6801","nonce":4102957285}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6802","nonce":4102957285},{"type":"v1","addr":"172.21.15.184:6803","nonce":4102957285}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6806","nonce":4102957285},{"type":"v1","addr":"172.21.15.184:6807","nonce":4102957285}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6804","nonce":4102957285},{"type":"v1","addr":"172.21.15.184:6805","nonce":4102957285}]},"public_addr":"172.21.15.184:6801/4102957285","cluster_addr":"172.21.15.184:6803/4102957285","heartbeat_back_addr":"172.21.15.184:6807/4102957285","heartbeat_front_addr":"172.21.15.184:6805/4102957285","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:35.972210+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.222007+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.198617+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.385663+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.358935+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.156066+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.752973+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:35.682846+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T15:46:24.901 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:24.897+0000 7f40e7a7c640 1 -- 172.21.15.123:0/3911583535 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f40c4073070 msgr2=0x7f40c4075530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:46:24.901 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:24.897+0000 7f40e7a7c640 1 --2- 172.21.15.123:0/3911583535 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f40c4073070 0x7f40c4075530 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f40d00b0000 tx=0x7f40d001c310 comp rx=0 tx=0).stop 2024-09-16T15:46:24.901 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:24.897+0000 7f40e7a7c640 1 -- 172.21.15.123:0/3911583535 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f40e00cf5a0 msgr2=0x7f40e015aa70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:46:24.901 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:24.897+0000 7f40e7a7c640 1 --2- 172.21.15.123:0/3911583535 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f40e00cf5a0 0x7f40e015aa70 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7f40dc00d950 tx=0x7f40dc00de10 comp rx=0 tx=0).stop 2024-09-16T15:46:24.901 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:24.897+0000 7f40e7a7c640 1 -- 172.21.15.123:0/3911583535 shutdown_connections 2024-09-16T15:46:24.901 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:24.897+0000 7f40e7a7c640 1 --2- 172.21.15.123:0/3911583535 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f40c4073070 0x7f40c4075530 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:24.901 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:24.897+0000 7f40e7a7c640 1 --2- 172.21.15.123:0/3911583535 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f40e015ec40 0x7f40e0149890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:24.901 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:24.897+0000 7f40e7a7c640 1 --2- 172.21.15.123:0/3911583535 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f40e0158120 0x7f40e01f9590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:24.902 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:24.897+0000 7f40e7a7c640 1 --2- 172.21.15.123:0/3911583535 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f40e00cf5a0 0x7f40e015aa70 unknown :-1 s=CLOSED pgs=641 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:24.902 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:24.897+0000 7f40e7a7c640 1 -- 172.21.15.123:0/3911583535 >> 172.21.15.123:0/3911583535 conn(0x7f40e00cfb10 msgr2=0x7f40e0156d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:46:24.902 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:24.897+0000 7f40e7a7c640 1 -- 172.21.15.123:0/3911583535 shutdown_connections 2024-09-16T15:46:24.902 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:24.897+0000 7f40e7a7c640 1 -- 172.21.15.123:0/3911583535 wait complete. 2024-09-16T15:46:24.916 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-09-16T15:46:24.917 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T15:46:25.046 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:25.041+0000 7f2373d15640 1 -- 172.21.15.123:0/3676047647 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f236c0cdc20 msgr2=0x7f236c0ca860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:46:25.046 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:25.041+0000 7f2373d15640 1 --2- 172.21.15.123:0/3676047647 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f236c0cdc20 0x7f236c0ca860 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7f236c070730 tx=0x7f236401ca30 comp rx=0 tx=0).stop 2024-09-16T15:46:25.046 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:25.041+0000 7f2373d15640 1 -- 172.21.15.123:0/3676047647 shutdown_connections 2024-09-16T15:46:25.046 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:25.041+0000 7f2373d15640 1 --2- 172.21.15.123:0/3676047647 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f236c0cdc20 0x7f236c0ca860 unknown :-1 s=CLOSED pgs=642 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:25.047 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:25.041+0000 7f2373d15640 1 --2- 172.21.15.123:0/3676047647 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f236c074470 0x7f236c0cd6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:25.047 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:25.041+0000 7f2373d15640 1 --2- 172.21.15.123:0/3676047647 >> v2:172.21.15.184:3300/0 conn(0x7f236c15e160 0x7f236c073f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:25.047 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:25.041+0000 7f2373d15640 1 -- 172.21.15.123:0/3676047647 >> 172.21.15.123:0/3676047647 conn(0x7f236c0cfb10 msgr2=0x7f236c0cff20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:46:25.047 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:25.041+0000 7f2373d15640 1 -- 172.21.15.123:0/3676047647 shutdown_connections 2024-09-16T15:46:25.047 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:25.041+0000 7f2373d15640 1 -- 172.21.15.123:0/3676047647 wait complete. 2024-09-16T15:46:25.049 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:25.045+0000 7f2373d15640 1 Processor -- start 2024-09-16T15:46:25.049 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:25.045+0000 7f2373d15640 1 -- start start 2024-09-16T15:46:25.049 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:25.045+0000 7f2373d15640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f236c074470 0x7f236c149f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:46:25.050 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:25.045+0000 7f2373d15640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f236c0cdc20 0x7f236c14a760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:46:25.050 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:25.045+0000 7f2373d15640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f236c15e160 0x7f236c15c840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:46:25.050 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:25.045+0000 7f2373d15640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f236c15ebb0 con 0x7f236c15e160 2024-09-16T15:46:25.050 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:25.045+0000 7f2373d15640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f236c15ea30 con 0x7f236c074470 2024-09-16T15:46:25.050 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:25.045+0000 7f2373d15640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f236c15ed30 con 0x7f236c0cdc20 2024-09-16T15:46:25.050 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:25.045+0000 7f23722b2640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f236c15e160 0x7f236c15c840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:46:25.050 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:25.045+0000 7f23722b2640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f236c15e160 0x7f236c15c840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:54674/0 (socket says 172.21.15.123:54674) 2024-09-16T15:46:25.050 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:25.045+0000 7f23722b2640 1 -- 172.21.15.123:0/2967068505 learned_addr learned my addr 172.21.15.123:0/2967068505 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:46:25.050 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:25.045+0000 7f2371ab1640 1 --2- 172.21.15.123:0/2967068505 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f236c074470 0x7f236c149f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:46:25.050 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:25.045+0000 7f23712b0640 1 --2- 172.21.15.123:0/2967068505 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f236c0cdc20 0x7f236c14a760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:46:25.051 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:25.045+0000 7f23722b2640 1 -- 172.21.15.123:0/2967068505 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f236c0cdc20 msgr2=0x7f236c14a760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:46:25.051 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:25.045+0000 7f23722b2640 1 --2- 172.21.15.123:0/2967068505 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f236c0cdc20 0x7f236c14a760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:25.051 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:25.045+0000 7f23722b2640 1 -- 172.21.15.123:0/2967068505 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f236c074470 msgr2=0x7f236c149f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:46:25.051 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:25.045+0000 7f23722b2640 1 --2- 172.21.15.123:0/2967068505 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f236c074470 0x7f236c149f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:25.051 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:25.045+0000 7f23722b2640 1 -- 172.21.15.123:0/2967068505 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f236c0b2e20 con 0x7f236c15e160 2024-09-16T15:46:25.051 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:25.045+0000 7f2371ab1640 1 --2- 172.21.15.123:0/2967068505 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f236c074470 0x7f236c149f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T15:46:25.051 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:25.045+0000 7f23722b2640 1 --2- 172.21.15.123:0/2967068505 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f236c15e160 0x7f236c15c840 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7f236401d2d0 tx=0x7f2364003f60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:46:25.051 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:25.045+0000 7f23627fc640 1 -- 172.21.15.123:0/2967068505 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f23640bb070 con 0x7f236c15e160 2024-09-16T15:46:25.051 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:25.045+0000 7f23627fc640 1 -- 172.21.15.123:0/2967068505 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f2364004d40 con 0x7f236c15e160 2024-09-16T15:46:25.051 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:25.045+0000 7f2373d15640 1 -- 172.21.15.123:0/2967068505 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f236c14b500 con 0x7f236c15e160 2024-09-16T15:46:25.052 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:25.045+0000 7f23627fc640 1 -- 172.21.15.123:0/2967068505 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f23640b2040 con 0x7f236c15e160 2024-09-16T15:46:25.052 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:25.045+0000 7f2373d15640 1 -- 172.21.15.123:0/2967068505 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f236c0c98c0 con 0x7f236c15e160 2024-09-16T15:46:25.052 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:25.045+0000 7f2373d15640 1 -- 172.21.15.123:0/2967068505 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f236c0ca860 con 0x7f236c15e160 2024-09-16T15:46:25.053 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:25.045+0000 7f23627fc640 1 -- 172.21.15.123:0/2967068505 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 9) ==== 97233+0+0 (secure 0 0 0) 0x7f23640be070 con 0x7f236c15e160 2024-09-16T15:46:25.053 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:25.045+0000 7f23627fc640 1 --2- 172.21.15.123:0/2967068505 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f23500730a0 0x7f2350075560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:46:25.053 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:25.049+0000 7f23627fc640 1 -- 172.21.15.123:0/2967068505 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(58..58 src has 1..58) ==== 5709+0+0 (secure 0 0 0) 0x7f236412df50 con 0x7f236c15e160 2024-09-16T15:46:25.053 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:25.049+0000 7f2371ab1640 1 --2- 172.21.15.123:0/2967068505 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f23500730a0 0x7f2350075560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:46:25.053 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:25.049+0000 7f2371ab1640 1 --2- 172.21.15.123:0/2967068505 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f23500730a0 0x7f2350075560 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f2368005fd0 tx=0x7f2368005dc0 comp rx=0 tx=0).ready entity=mgr.4135 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:46:25.054 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:25.049+0000 7f23627fc640 1 -- 172.21.15.123:0/2967068505 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f2364016970 con 0x7f236c15e160 2024-09-16T15:46:25.211 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:25.209+0000 7f2373d15640 1 -- 172.21.15.123:0/2967068505 --> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f236c0d06c0 con 0x7f23500730a0 2024-09-16T15:46:25.223 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:25.221+0000 7f23627fc640 1 -- 172.21.15.123:0/2967068505 <== mgr.4135 v2:172.21.15.184:6832/3114346423 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+239846 (secure 0 0 0) 0x7f236c0d06c0 con 0x7f23500730a0 2024-09-16T15:46:25.239 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-09-16T15:46:25.241 INFO:teuthology.orchestra.run.smithi123.stderr:dumped all 2024-09-16T15:46:25.243 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:25.225+0000 7f2373d15640 1 -- 172.21.15.123:0/2967068505 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f23500730a0 msgr2=0x7f2350075560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:46:25.243 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:25.225+0000 7f2373d15640 1 --2- 172.21.15.123:0/2967068505 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f23500730a0 0x7f2350075560 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f2368005fd0 tx=0x7f2368005dc0 comp rx=0 tx=0).stop 2024-09-16T15:46:25.243 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:25.225+0000 7f2373d15640 1 -- 172.21.15.123:0/2967068505 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f236c15e160 msgr2=0x7f236c15c840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:46:25.243 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:25.225+0000 7f2373d15640 1 --2- 172.21.15.123:0/2967068505 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f236c15e160 0x7f236c15c840 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7f236401d2d0 tx=0x7f2364003f60 comp rx=0 tx=0).stop 2024-09-16T15:46:25.243 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:25.225+0000 7f2373d15640 1 -- 172.21.15.123:0/2967068505 shutdown_connections 2024-09-16T15:46:25.243 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:25.225+0000 7f2373d15640 1 --2- 172.21.15.123:0/2967068505 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f23500730a0 0x7f2350075560 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:25.243 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:25.225+0000 7f2373d15640 1 --2- 172.21.15.123:0/2967068505 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f236c15e160 0x7f236c15c840 unknown :-1 s=CLOSED pgs=643 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:25.243 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:25.225+0000 7f2373d15640 1 --2- 172.21.15.123:0/2967068505 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f236c0cdc20 0x7f236c14a760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:25.243 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:25.225+0000 7f2373d15640 1 --2- 172.21.15.123:0/2967068505 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f236c074470 0x7f236c149f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:25.244 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:25.225+0000 7f2373d15640 1 -- 172.21.15.123:0/2967068505 >> 172.21.15.123:0/2967068505 conn(0x7f236c0cfb10 msgr2=0x7f236c0c4180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:46:25.244 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:25.225+0000 7f2373d15640 1 -- 172.21.15.123:0/2967068505 shutdown_connections 2024-09-16T15:46:25.244 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:25.225+0000 7f2373d15640 1 -- 172.21.15.123:0/2967068505 wait complete. 2024-09-16T15:46:25.244 INFO:teuthology.orchestra.run.smithi123.stdout:{"pg_ready":true,"pg_map":{"version":163,"stamp":"2024-09-16T15:46:23.317907+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":66,"acting":66,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":2,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":7,"kb":754974720,"kb_used":234960,"kb_used_data":6208,"kb_used_omap":12,"kb_used_meta":228531,"kb_avail":754739760,"statfs":{"total":773094113280,"available":772853514240,"internally_reserved":0,"allocated":6356992,"data_stored":3282722,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":234016305},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":16,"apply_latency_ms":16,"commit_latency_ns":16000000,"apply_latency_ns":16000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.190708"},"pg_stats":[{"pgid":"9.3f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T15:46:20.126241+0000","last_change":"2024-09-16T15:46:20.126241+0000","last_active":"2024-09-16T15:46:20.126241+0000","last_peered":"2024-09-16T15:46:20.126241+0000","last_clean":"2024-09-16T15:46:20.126241+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:46:20.126241+0000","last_undegraded":"2024-09-16T15:46:20.126241+0000","last_fullsized":"2024-09-16T15:46:20.126241+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:20.126241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:20.126241+0000","last_clean_scrub_stamp":"2024-09-16T15:46:20.126241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.3e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T15:46:20.126241+0000","last_change":"2024-09-16T15:46:20.126241+0000","last_active":"2024-09-16T15:46:20.126241+0000","last_peered":"2024-09-16T15:46:20.126241+0000","last_clean":"2024-09-16T15:46:20.126241+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:46:20.126241+0000","last_undegraded":"2024-09-16T15:46:20.126241+0000","last_fullsized":"2024-09-16T15:46:20.126241+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:20.126241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:20.126241+0000","last_clean_scrub_stamp":"2024-09-16T15:46:20.126241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.3d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T15:46:20.126241+0000","last_change":"2024-09-16T15:46:20.126241+0000","last_active":"2024-09-16T15:46:20.126241+0000","last_peered":"2024-09-16T15:46:20.126241+0000","last_clean":"2024-09-16T15:46:20.126241+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:46:20.126241+0000","last_undegraded":"2024-09-16T15:46:20.126241+0000","last_fullsized":"2024-09-16T15:46:20.126241+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:20.126241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:20.126241+0000","last_clean_scrub_stamp":"2024-09-16T15:46:20.126241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.3c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T15:46:20.126241+0000","last_change":"2024-09-16T15:46:20.126241+0000","last_active":"2024-09-16T15:46:20.126241+0000","last_peered":"2024-09-16T15:46:20.126241+0000","last_clean":"2024-09-16T15:46:20.126241+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:46:20.126241+0000","last_undegraded":"2024-09-16T15:46:20.126241+0000","last_fullsized":"2024-09-16T15:46:20.126241+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:20.126241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:20.126241+0000","last_clean_scrub_stamp":"2024-09-16T15:46:20.126241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.3b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T15:46:20.126241+0000","last_change":"2024-09-16T15:46:20.126241+0000","last_active":"2024-09-16T15:46:20.126241+0000","last_peered":"2024-09-16T15:46:20.126241+0000","last_clean":"2024-09-16T15:46:20.126241+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:46:20.126241+0000","last_undegraded":"2024-09-16T15:46:20.126241+0000","last_fullsized":"2024-09-16T15:46:20.126241+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:20.126241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:20.126241+0000","last_clean_scrub_stamp":"2024-09-16T15:46:20.126241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.3a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T15:46:20.126241+0000","last_change":"2024-09-16T15:46:20.126241+0000","last_active":"2024-09-16T15:46:20.126241+0000","last_peered":"2024-09-16T15:46:20.126241+0000","last_clean":"2024-09-16T15:46:20.126241+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:46:20.126241+0000","last_undegraded":"2024-09-16T15:46:20.126241+0000","last_fullsized":"2024-09-16T15:46:20.126241+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:20.126241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:20.126241+0000","last_clean_scrub_stamp":"2024-09-16T15:46:20.126241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.39","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T15:46:20.126241+0000","last_change":"2024-09-16T15:46:20.126241+0000","last_active":"2024-09-16T15:46:20.126241+0000","last_peered":"2024-09-16T15:46:20.126241+0000","last_clean":"2024-09-16T15:46:20.126241+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:46:20.126241+0000","last_undegraded":"2024-09-16T15:46:20.126241+0000","last_fullsized":"2024-09-16T15:46:20.126241+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:20.126241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:20.126241+0000","last_clean_scrub_stamp":"2024-09-16T15:46:20.126241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.38","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T15:46:20.126241+0000","last_change":"2024-09-16T15:46:20.126241+0000","last_active":"2024-09-16T15:46:20.126241+0000","last_peered":"2024-09-16T15:46:20.126241+0000","last_clean":"2024-09-16T15:46:20.126241+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:46:20.126241+0000","last_undegraded":"2024-09-16T15:46:20.126241+0000","last_fullsized":"2024-09-16T15:46:20.126241+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:20.126241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:20.126241+0000","last_clean_scrub_stamp":"2024-09-16T15:46:20.126241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.37","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T15:46:20.126241+0000","last_change":"2024-09-16T15:46:20.126241+0000","last_active":"2024-09-16T15:46:20.126241+0000","last_peered":"2024-09-16T15:46:20.126241+0000","last_clean":"2024-09-16T15:46:20.126241+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:46:20.126241+0000","last_undegraded":"2024-09-16T15:46:20.126241+0000","last_fullsized":"2024-09-16T15:46:20.126241+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:20.126241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:20.126241+0000","last_clean_scrub_stamp":"2024-09-16T15:46:20.126241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.36","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T15:46:20.126241+0000","last_change":"2024-09-16T15:46:20.126241+0000","last_active":"2024-09-16T15:46:20.126241+0000","last_peered":"2024-09-16T15:46:20.126241+0000","last_clean":"2024-09-16T15:46:20.126241+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:46:20.126241+0000","last_undegraded":"2024-09-16T15:46:20.126241+0000","last_fullsized":"2024-09-16T15:46:20.126241+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:20.126241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:20.126241+0000","last_clean_scrub_stamp":"2024-09-16T15:46:20.126241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.35","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T15:46:20.126241+0000","last_change":"2024-09-16T15:46:20.126241+0000","last_active":"2024-09-16T15:46:20.126241+0000","last_peered":"2024-09-16T15:46:20.126241+0000","last_clean":"2024-09-16T15:46:20.126241+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:46:20.126241+0000","last_undegraded":"2024-09-16T15:46:20.126241+0000","last_fullsized":"2024-09-16T15:46:20.126241+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:20.126241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:20.126241+0000","last_clean_scrub_stamp":"2024-09-16T15:46:20.126241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.34","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T15:46:20.126241+0000","last_change":"2024-09-16T15:46:20.126241+0000","last_active":"2024-09-16T15:46:20.126241+0000","last_peered":"2024-09-16T15:46:20.126241+0000","last_clean":"2024-09-16T15:46:20.126241+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:46:20.126241+0000","last_undegraded":"2024-09-16T15:46:20.126241+0000","last_fullsized":"2024-09-16T15:46:20.126241+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:20.126241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:20.126241+0000","last_clean_scrub_stamp":"2024-09-16T15:46:20.126241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.33","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T15:46:20.126241+0000","last_change":"2024-09-16T15:46:20.126241+0000","last_active":"2024-09-16T15:46:20.126241+0000","last_peered":"2024-09-16T15:46:20.126241+0000","last_clean":"2024-09-16T15:46:20.126241+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:46:20.126241+0000","last_undegraded":"2024-09-16T15:46:20.126241+0000","last_fullsized":"2024-09-16T15:46:20.126241+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:20.126241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:20.126241+0000","last_clean_scrub_stamp":"2024-09-16T15:46:20.126241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.32","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T15:46:20.126241+0000","last_change":"2024-09-16T15:46:20.126241+0000","last_active":"2024-09-16T15:46:20.126241+0000","last_peered":"2024-09-16T15:46:20.126241+0000","last_clean":"2024-09-16T15:46:20.126241+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:46:20.126241+0000","last_undegraded":"2024-09-16T15:46:20.126241+0000","last_fullsized":"2024-09-16T15:46:20.126241+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:20.126241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:20.126241+0000","last_clean_scrub_stamp":"2024-09-16T15:46:20.126241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.31","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T15:46:20.126241+0000","last_change":"2024-09-16T15:46:20.126241+0000","last_active":"2024-09-16T15:46:20.126241+0000","last_peered":"2024-09-16T15:46:20.126241+0000","last_clean":"2024-09-16T15:46:20.126241+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:46:20.126241+0000","last_undegraded":"2024-09-16T15:46:20.126241+0000","last_fullsized":"2024-09-16T15:46:20.126241+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:20.126241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:20.126241+0000","last_clean_scrub_stamp":"2024-09-16T15:46:20.126241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.30","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T15:46:20.126241+0000","last_change":"2024-09-16T15:46:20.126241+0000","last_active":"2024-09-16T15:46:20.126241+0000","last_peered":"2024-09-16T15:46:20.126241+0000","last_clean":"2024-09-16T15:46:20.126241+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:46:20.126241+0000","last_undegraded":"2024-09-16T15:46:20.126241+0000","last_fullsized":"2024-09-16T15:46:20.126241+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:20.126241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:20.126241+0000","last_clean_scrub_stamp":"2024-09-16T15:46:20.126241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.2f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T15:46:20.126241+0000","last_change":"2024-09-16T15:46:20.126241+0000","last_active":"2024-09-16T15:46:20.126241+0000","last_peered":"2024-09-16T15:46:20.126241+0000","last_clean":"2024-09-16T15:46:20.126241+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:46:20.126241+0000","last_undegraded":"2024-09-16T15:46:20.126241+0000","last_fullsized":"2024-09-16T15:46:20.126241+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:20.126241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:20.126241+0000","last_clean_scrub_stamp":"2024-09-16T15:46:20.126241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.2e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T15:46:20.126241+0000","last_change":"2024-09-16T15:46:20.126241+0000","last_active":"2024-09-16T15:46:20.126241+0000","last_peered":"2024-09-16T15:46:20.126241+0000","last_clean":"2024-09-16T15:46:20.126241+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:46:20.126241+0000","last_undegraded":"2024-09-16T15:46:20.126241+0000","last_fullsized":"2024-09-16T15:46:20.126241+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:20.126241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:20.126241+0000","last_clean_scrub_stamp":"2024-09-16T15:46:20.126241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.2d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T15:46:20.126241+0000","last_change":"2024-09-16T15:46:20.126241+0000","last_active":"2024-09-16T15:46:20.126241+0000","last_peered":"2024-09-16T15:46:20.126241+0000","last_clean":"2024-09-16T15:46:20.126241+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:46:20.126241+0000","last_undegraded":"2024-09-16T15:46:20.126241+0000","last_fullsized":"2024-09-16T15:46:20.126241+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:20.126241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:20.126241+0000","last_clean_scrub_stamp":"2024-09-16T15:46:20.126241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.2c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T15:46:20.126241+0000","last_change":"2024-09-16T15:46:20.126241+0000","last_active":"2024-09-16T15:46:20.126241+0000","last_peered":"2024-09-16T15:46:20.126241+0000","last_clean":"2024-09-16T15:46:20.126241+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:46:20.126241+0000","last_undegraded":"2024-09-16T15:46:20.126241+0000","last_fullsized":"2024-09-16T15:46:20.126241+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:20.126241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:20.126241+0000","last_clean_scrub_stamp":"2024-09-16T15:46:20.126241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.2b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T15:46:20.126241+0000","last_change":"2024-09-16T15:46:20.126241+0000","last_active":"2024-09-16T15:46:20.126241+0000","last_peered":"2024-09-16T15:46:20.126241+0000","last_clean":"2024-09-16T15:46:20.126241+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:46:20.126241+0000","last_undegraded":"2024-09-16T15:46:20.126241+0000","last_fullsized":"2024-09-16T15:46:20.126241+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:20.126241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:20.126241+0000","last_clean_scrub_stamp":"2024-09-16T15:46:20.126241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.2a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T15:46:20.126241+0000","last_change":"2024-09-16T15:46:20.126241+0000","last_active":"2024-09-16T15:46:20.126241+0000","last_peered":"2024-09-16T15:46:20.126241+0000","last_clean":"2024-09-16T15:46:20.126241+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:46:20.126241+0000","last_undegraded":"2024-09-16T15:46:20.126241+0000","last_fullsized":"2024-09-16T15:46:20.126241+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:20.126241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:20.126241+0000","last_clean_scrub_stamp":"2024-09-16T15:46:20.126241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.29","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T15:46:20.126241+0000","last_change":"2024-09-16T15:46:20.126241+0000","last_active":"2024-09-16T15:46:20.126241+0000","last_peered":"2024-09-16T15:46:20.126241+0000","last_clean":"2024-09-16T15:46:20.126241+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:46:20.126241+0000","last_undegraded":"2024-09-16T15:46:20.126241+0000","last_fullsized":"2024-09-16T15:46:20.126241+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:20.126241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:20.126241+0000","last_clean_scrub_stamp":"2024-09-16T15:46:20.126241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.28","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T15:46:20.126241+0000","last_change":"2024-09-16T15:46:20.126241+0000","last_active":"2024-09-16T15:46:20.126241+0000","last_peered":"2024-09-16T15:46:20.126241+0000","last_clean":"2024-09-16T15:46:20.126241+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:46:20.126241+0000","last_undegraded":"2024-09-16T15:46:20.126241+0000","last_fullsized":"2024-09-16T15:46:20.126241+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:20.126241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:20.126241+0000","last_clean_scrub_stamp":"2024-09-16T15:46:20.126241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.27","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T15:46:20.126241+0000","last_change":"2024-09-16T15:46:20.126241+0000","last_active":"2024-09-16T15:46:20.126241+0000","last_peered":"2024-09-16T15:46:20.126241+0000","last_clean":"2024-09-16T15:46:20.126241+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:46:20.126241+0000","last_undegraded":"2024-09-16T15:46:20.126241+0000","last_fullsized":"2024-09-16T15:46:20.126241+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:20.126241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:20.126241+0000","last_clean_scrub_stamp":"2024-09-16T15:46:20.126241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.26","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T15:46:20.126241+0000","last_change":"2024-09-16T15:46:20.126241+0000","last_active":"2024-09-16T15:46:20.126241+0000","last_peered":"2024-09-16T15:46:20.126241+0000","last_clean":"2024-09-16T15:46:20.126241+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:46:20.126241+0000","last_undegraded":"2024-09-16T15:46:20.126241+0000","last_fullsized":"2024-09-16T15:46:20.126241+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:20.126241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:20.126241+0000","last_clean_scrub_stamp":"2024-09-16T15:46:20.126241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.25","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T15:46:20.126241+0000","last_change":"2024-09-16T15:46:20.126241+0000","last_active":"2024-09-16T15:46:20.126241+0000","last_peered":"2024-09-16T15:46:20.126241+0000","last_clean":"2024-09-16T15:46:20.126241+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:46:20.126241+0000","last_undegraded":"2024-09-16T15:46:20.126241+0000","last_fullsized":"2024-09-16T15:46:20.126241+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:20.126241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:20.126241+0000","last_clean_scrub_stamp":"2024-09-16T15:46:20.126241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.24","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T15:46:20.126241+0000","last_change":"2024-09-16T15:46:20.126241+0000","last_active":"2024-09-16T15:46:20.126241+0000","last_peered":"2024-09-16T15:46:20.126241+0000","last_clean":"2024-09-16T15:46:20.126241+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:46:20.126241+0000","last_undegraded":"2024-09-16T15:46:20.126241+0000","last_fullsized":"2024-09-16T15:46:20.126241+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:20.126241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:20.126241+0000","last_clean_scrub_stamp":"2024-09-16T15:46:20.126241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.23","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T15:46:20.126241+0000","last_change":"2024-09-16T15:46:20.126241+0000","last_active":"2024-09-16T15:46:20.126241+0000","last_peered":"2024-09-16T15:46:20.126241+0000","last_clean":"2024-09-16T15:46:20.126241+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:46:20.126241+0000","last_undegraded":"2024-09-16T15:46:20.126241+0000","last_fullsized":"2024-09-16T15:46:20.126241+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:20.126241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:20.126241+0000","last_clean_scrub_stamp":"2024-09-16T15:46:20.126241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.22","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T15:46:20.126241+0000","last_change":"2024-09-16T15:46:20.126241+0000","last_active":"2024-09-16T15:46:20.126241+0000","last_peered":"2024-09-16T15:46:20.126241+0000","last_clean":"2024-09-16T15:46:20.126241+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:46:20.126241+0000","last_undegraded":"2024-09-16T15:46:20.126241+0000","last_fullsized":"2024-09-16T15:46:20.126241+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:20.126241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:20.126241+0000","last_clean_scrub_stamp":"2024-09-16T15:46:20.126241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.21","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T15:46:20.126241+0000","last_change":"2024-09-16T15:46:20.126241+0000","last_active":"2024-09-16T15:46:20.126241+0000","last_peered":"2024-09-16T15:46:20.126241+0000","last_clean":"2024-09-16T15:46:20.126241+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:46:20.126241+0000","last_undegraded":"2024-09-16T15:46:20.126241+0000","last_fullsized":"2024-09-16T15:46:20.126241+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:20.126241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:20.126241+0000","last_clean_scrub_stamp":"2024-09-16T15:46:20.126241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.20","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T15:46:20.126241+0000","last_change":"2024-09-16T15:46:20.126241+0000","last_active":"2024-09-16T15:46:20.126241+0000","last_peered":"2024-09-16T15:46:20.126241+0000","last_clean":"2024-09-16T15:46:20.126241+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:46:20.126241+0000","last_undegraded":"2024-09-16T15:46:20.126241+0000","last_fullsized":"2024-09-16T15:46:20.126241+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:20.126241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:20.126241+0000","last_clean_scrub_stamp":"2024-09-16T15:46:20.126241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T15:46:20.126241+0000","last_change":"2024-09-16T15:46:20.126241+0000","last_active":"2024-09-16T15:46:20.126241+0000","last_peered":"2024-09-16T15:46:20.126241+0000","last_clean":"2024-09-16T15:46:20.126241+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:46:20.126241+0000","last_undegraded":"2024-09-16T15:46:20.126241+0000","last_fullsized":"2024-09-16T15:46:20.126241+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:20.126241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:20.126241+0000","last_clean_scrub_stamp":"2024-09-16T15:46:20.126241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.1f","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-09-16T15:46:19.136556+0000","last_change":"2024-09-16T15:46:19.136556+0000","last_active":"2024-09-16T15:46:19.122499+0000","last_peered":"2024-09-16T15:46:19.122499+0000","last_clean":"2024-09-16T15:46:19.122499+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:46:19.136556+0000","last_undegraded":"2024-09-16T15:46:19.136556+0000","last_fullsized":"2024-09-16T15:46:19.136556+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:19.122499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:19.122499+0000","last_clean_scrub_stamp":"2024-09-16T15:46:19.122499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T15:46:20.126241+0000","last_change":"2024-09-16T15:46:20.126241+0000","last_active":"2024-09-16T15:46:20.126241+0000","last_peered":"2024-09-16T15:46:20.126241+0000","last_clean":"2024-09-16T15:46:20.126241+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:46:20.126241+0000","last_undegraded":"2024-09-16T15:46:20.126241+0000","last_fullsized":"2024-09-16T15:46:20.126241+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:20.126241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:20.126241+0000","last_clean_scrub_stamp":"2024-09-16T15:46:20.126241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.1e","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-09-16T15:46:19.138290+0000","last_change":"2024-09-16T15:46:19.138290+0000","last_active":"2024-09-16T15:46:19.122499+0000","last_peered":"2024-09-16T15:46:19.122499+0000","last_clean":"2024-09-16T15:46:19.122499+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:46:19.138290+0000","last_undegraded":"2024-09-16T15:46:19.138290+0000","last_fullsized":"2024-09-16T15:46:19.138290+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:19.122499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:19.122499+0000","last_clean_scrub_stamp":"2024-09-16T15:46:19.122499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T15:46:20.126241+0000","last_change":"2024-09-16T15:46:20.126241+0000","last_active":"2024-09-16T15:46:20.126241+0000","last_peered":"2024-09-16T15:46:20.126241+0000","last_clean":"2024-09-16T15:46:20.126241+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:46:20.126241+0000","last_undegraded":"2024-09-16T15:46:20.126241+0000","last_fullsized":"2024-09-16T15:46:20.126241+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:20.126241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:20.126241+0000","last_clean_scrub_stamp":"2024-09-16T15:46:20.126241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.1d","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-09-16T15:46:19.135936+0000","last_change":"2024-09-16T15:46:19.135936+0000","last_active":"2024-09-16T15:46:19.122499+0000","last_peered":"2024-09-16T15:46:19.122499+0000","last_clean":"2024-09-16T15:46:19.122499+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:46:19.135936+0000","last_undegraded":"2024-09-16T15:46:19.135936+0000","last_fullsized":"2024-09-16T15:46:19.135936+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:19.122499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:19.122499+0000","last_clean_scrub_stamp":"2024-09-16T15:46:19.122499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T15:46:20.126241+0000","last_change":"2024-09-16T15:46:20.126241+0000","last_active":"2024-09-16T15:46:20.126241+0000","last_peered":"2024-09-16T15:46:20.126241+0000","last_clean":"2024-09-16T15:46:20.126241+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:46:20.126241+0000","last_undegraded":"2024-09-16T15:46:20.126241+0000","last_fullsized":"2024-09-16T15:46:20.126241+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:20.126241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:20.126241+0000","last_clean_scrub_stamp":"2024-09-16T15:46:20.126241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.1c","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-09-16T15:46:19.138613+0000","last_change":"2024-09-16T15:46:19.138613+0000","last_active":"2024-09-16T15:46:19.122499+0000","last_peered":"2024-09-16T15:46:19.122499+0000","last_clean":"2024-09-16T15:46:19.122499+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:46:19.138613+0000","last_undegraded":"2024-09-16T15:46:19.138613+0000","last_fullsized":"2024-09-16T15:46:19.138613+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:19.122499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:19.122499+0000","last_clean_scrub_stamp":"2024-09-16T15:46:19.122499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T15:46:20.126241+0000","last_change":"2024-09-16T15:46:20.126241+0000","last_active":"2024-09-16T15:46:20.126241+0000","last_peered":"2024-09-16T15:46:20.126241+0000","last_clean":"2024-09-16T15:46:20.126241+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:46:20.126241+0000","last_undegraded":"2024-09-16T15:46:20.126241+0000","last_fullsized":"2024-09-16T15:46:20.126241+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:20.126241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:20.126241+0000","last_clean_scrub_stamp":"2024-09-16T15:46:20.126241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.1b","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-09-16T15:46:19.131031+0000","last_change":"2024-09-16T15:46:19.131031+0000","last_active":"2024-09-16T15:46:19.122499+0000","last_peered":"2024-09-16T15:46:19.122499+0000","last_clean":"2024-09-16T15:46:19.122499+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:46:19.131031+0000","last_undegraded":"2024-09-16T15:46:19.131031+0000","last_fullsized":"2024-09-16T15:46:19.131031+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:19.122499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:19.122499+0000","last_clean_scrub_stamp":"2024-09-16T15:46:19.122499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T15:46:20.126241+0000","last_change":"2024-09-16T15:46:20.126241+0000","last_active":"2024-09-16T15:46:20.126241+0000","last_peered":"2024-09-16T15:46:20.126241+0000","last_clean":"2024-09-16T15:46:20.126241+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:46:20.126241+0000","last_undegraded":"2024-09-16T15:46:20.126241+0000","last_fullsized":"2024-09-16T15:46:20.126241+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:20.126241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:20.126241+0000","last_clean_scrub_stamp":"2024-09-16T15:46:20.126241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.1a","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-09-16T15:46:19.135039+0000","last_change":"2024-09-16T15:46:19.135039+0000","last_active":"2024-09-16T15:46:19.122499+0000","last_peered":"2024-09-16T15:46:19.122499+0000","last_clean":"2024-09-16T15:46:19.122499+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:46:19.135039+0000","last_undegraded":"2024-09-16T15:46:19.135039+0000","last_fullsized":"2024-09-16T15:46:19.135039+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:19.122499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:19.122499+0000","last_clean_scrub_stamp":"2024-09-16T15:46:19.122499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T15:46:20.126241+0000","last_change":"2024-09-16T15:46:20.126241+0000","last_active":"2024-09-16T15:46:20.126241+0000","last_peered":"2024-09-16T15:46:20.126241+0000","last_clean":"2024-09-16T15:46:20.126241+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:46:20.126241+0000","last_undegraded":"2024-09-16T15:46:20.126241+0000","last_fullsized":"2024-09-16T15:46:20.126241+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:20.126241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:20.126241+0000","last_clean_scrub_stamp":"2024-09-16T15:46:20.126241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.19","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-09-16T15:46:19.137984+0000","last_change":"2024-09-16T15:46:19.137984+0000","last_active":"2024-09-16T15:46:19.122499+0000","last_peered":"2024-09-16T15:46:19.122499+0000","last_clean":"2024-09-16T15:46:19.122499+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:46:19.137984+0000","last_undegraded":"2024-09-16T15:46:19.137984+0000","last_fullsized":"2024-09-16T15:46:19.137984+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:19.122499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:19.122499+0000","last_clean_scrub_stamp":"2024-09-16T15:46:19.122499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T15:46:20.126241+0000","last_change":"2024-09-16T15:46:20.126241+0000","last_active":"2024-09-16T15:46:20.126241+0000","last_peered":"2024-09-16T15:46:20.126241+0000","last_clean":"2024-09-16T15:46:20.126241+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:46:20.126241+0000","last_undegraded":"2024-09-16T15:46:20.126241+0000","last_fullsized":"2024-09-16T15:46:20.126241+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:20.126241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:20.126241+0000","last_clean_scrub_stamp":"2024-09-16T15:46:20.126241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.18","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-09-16T15:46:19.131031+0000","last_change":"2024-09-16T15:46:19.131031+0000","last_active":"2024-09-16T15:46:19.122499+0000","last_peered":"2024-09-16T15:46:19.122499+0000","last_clean":"2024-09-16T15:46:19.122499+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:46:19.131031+0000","last_undegraded":"2024-09-16T15:46:19.131031+0000","last_fullsized":"2024-09-16T15:46:19.131031+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:19.122499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:19.122499+0000","last_clean_scrub_stamp":"2024-09-16T15:46:19.122499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T15:46:20.126241+0000","last_change":"2024-09-16T15:46:20.126241+0000","last_active":"2024-09-16T15:46:20.126241+0000","last_peered":"2024-09-16T15:46:20.126241+0000","last_clean":"2024-09-16T15:46:20.126241+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:46:20.126241+0000","last_undegraded":"2024-09-16T15:46:20.126241+0000","last_fullsized":"2024-09-16T15:46:20.126241+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:20.126241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:20.126241+0000","last_clean_scrub_stamp":"2024-09-16T15:46:20.126241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.17","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-09-16T15:46:19.139400+0000","last_change":"2024-09-16T15:46:19.139400+0000","last_active":"2024-09-16T15:46:19.122499+0000","last_peered":"2024-09-16T15:46:19.122499+0000","last_clean":"2024-09-16T15:46:19.122499+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:46:19.139400+0000","last_undegraded":"2024-09-16T15:46:19.139400+0000","last_fullsized":"2024-09-16T15:46:19.139400+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:19.122499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:19.122499+0000","last_clean_scrub_stamp":"2024-09-16T15:46:19.122499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T15:46:20.126241+0000","last_change":"2024-09-16T15:46:20.126241+0000","last_active":"2024-09-16T15:46:20.126241+0000","last_peered":"2024-09-16T15:46:20.126241+0000","last_clean":"2024-09-16T15:46:20.126241+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:46:20.126241+0000","last_undegraded":"2024-09-16T15:46:20.126241+0000","last_fullsized":"2024-09-16T15:46:20.126241+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:20.126241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:20.126241+0000","last_clean_scrub_stamp":"2024-09-16T15:46:20.126241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.16","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-09-16T15:46:19.137549+0000","last_change":"2024-09-16T15:46:19.137549+0000","last_active":"2024-09-16T15:46:19.122499+0000","last_peered":"2024-09-16T15:46:19.122499+0000","last_clean":"2024-09-16T15:46:19.122499+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:46:19.137549+0000","last_undegraded":"2024-09-16T15:46:19.137549+0000","last_fullsized":"2024-09-16T15:46:19.137549+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:19.122499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:19.122499+0000","last_clean_scrub_stamp":"2024-09-16T15:46:19.122499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T15:46:20.126241+0000","last_change":"2024-09-16T15:46:20.126241+0000","last_active":"2024-09-16T15:46:20.126241+0000","last_peered":"2024-09-16T15:46:20.126241+0000","last_clean":"2024-09-16T15:46:20.126241+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:46:20.126241+0000","last_undegraded":"2024-09-16T15:46:20.126241+0000","last_fullsized":"2024-09-16T15:46:20.126241+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:20.126241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:20.126241+0000","last_clean_scrub_stamp":"2024-09-16T15:46:20.126241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.15","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-09-16T15:46:19.140113+0000","last_change":"2024-09-16T15:46:19.140113+0000","last_active":"2024-09-16T15:46:19.122499+0000","last_peered":"2024-09-16T15:46:19.122499+0000","last_clean":"2024-09-16T15:46:19.122499+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:46:19.140113+0000","last_undegraded":"2024-09-16T15:46:19.140113+0000","last_fullsized":"2024-09-16T15:46:19.140113+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:19.122499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:19.122499+0000","last_clean_scrub_stamp":"2024-09-16T15:46:19.122499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T15:46:20.126241+0000","last_change":"2024-09-16T15:46:20.126241+0000","last_active":"2024-09-16T15:46:20.126241+0000","last_peered":"2024-09-16T15:46:20.126241+0000","last_clean":"2024-09-16T15:46:20.126241+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:46:20.126241+0000","last_undegraded":"2024-09-16T15:46:20.126241+0000","last_fullsized":"2024-09-16T15:46:20.126241+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:20.126241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:20.126241+0000","last_clean_scrub_stamp":"2024-09-16T15:46:20.126241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.14","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-09-16T15:46:19.131010+0000","last_change":"2024-09-16T15:46:19.131010+0000","last_active":"2024-09-16T15:46:19.122499+0000","last_peered":"2024-09-16T15:46:19.122499+0000","last_clean":"2024-09-16T15:46:19.122499+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:46:19.131010+0000","last_undegraded":"2024-09-16T15:46:19.131010+0000","last_fullsized":"2024-09-16T15:46:19.131010+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:19.122499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:19.122499+0000","last_clean_scrub_stamp":"2024-09-16T15:46:19.122499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T15:46:20.126241+0000","last_change":"2024-09-16T15:46:20.126241+0000","last_active":"2024-09-16T15:46:20.126241+0000","last_peered":"2024-09-16T15:46:20.126241+0000","last_clean":"2024-09-16T15:46:20.126241+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:46:20.126241+0000","last_undegraded":"2024-09-16T15:46:20.126241+0000","last_fullsized":"2024-09-16T15:46:20.126241+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:20.126241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:20.126241+0000","last_clean_scrub_stamp":"2024-09-16T15:46:20.126241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.13","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-09-16T15:46:19.140267+0000","last_change":"2024-09-16T15:46:19.140267+0000","last_active":"2024-09-16T15:46:19.122499+0000","last_peered":"2024-09-16T15:46:19.122499+0000","last_clean":"2024-09-16T15:46:19.122499+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:46:19.140267+0000","last_undegraded":"2024-09-16T15:46:19.140267+0000","last_fullsized":"2024-09-16T15:46:19.140267+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:19.122499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:19.122499+0000","last_clean_scrub_stamp":"2024-09-16T15:46:19.122499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T15:46:20.126241+0000","last_change":"2024-09-16T15:46:20.126241+0000","last_active":"2024-09-16T15:46:20.126241+0000","last_peered":"2024-09-16T15:46:20.126241+0000","last_clean":"2024-09-16T15:46:20.126241+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:46:20.126241+0000","last_undegraded":"2024-09-16T15:46:20.126241+0000","last_fullsized":"2024-09-16T15:46:20.126241+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:20.126241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:20.126241+0000","last_clean_scrub_stamp":"2024-09-16T15:46:20.126241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.12","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-09-16T15:46:19.134111+0000","last_change":"2024-09-16T15:46:19.134111+0000","last_active":"2024-09-16T15:46:19.122499+0000","last_peered":"2024-09-16T15:46:19.122499+0000","last_clean":"2024-09-16T15:46:19.122499+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:46:19.134111+0000","last_undegraded":"2024-09-16T15:46:19.134111+0000","last_fullsized":"2024-09-16T15:46:19.134111+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:19.122499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:19.122499+0000","last_clean_scrub_stamp":"2024-09-16T15:46:19.122499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T15:46:20.126241+0000","last_change":"2024-09-16T15:46:20.126241+0000","last_active":"2024-09-16T15:46:20.126241+0000","last_peered":"2024-09-16T15:46:20.126241+0000","last_clean":"2024-09-16T15:46:20.126241+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:46:20.126241+0000","last_undegraded":"2024-09-16T15:46:20.126241+0000","last_fullsized":"2024-09-16T15:46:20.126241+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:20.126241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:20.126241+0000","last_clean_scrub_stamp":"2024-09-16T15:46:20.126241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.11","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-09-16T15:46:19.136788+0000","last_change":"2024-09-16T15:46:19.136788+0000","last_active":"2024-09-16T15:46:19.122499+0000","last_peered":"2024-09-16T15:46:19.122499+0000","last_clean":"2024-09-16T15:46:19.122499+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:46:19.136788+0000","last_undegraded":"2024-09-16T15:46:19.136788+0000","last_fullsized":"2024-09-16T15:46:19.136788+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:19.122499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:19.122499+0000","last_clean_scrub_stamp":"2024-09-16T15:46:19.122499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T15:46:20.126241+0000","last_change":"2024-09-16T15:46:20.126241+0000","last_active":"2024-09-16T15:46:20.126241+0000","last_peered":"2024-09-16T15:46:20.126241+0000","last_clean":"2024-09-16T15:46:20.126241+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:46:20.126241+0000","last_undegraded":"2024-09-16T15:46:20.126241+0000","last_fullsized":"2024-09-16T15:46:20.126241+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:20.126241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:20.126241+0000","last_clean_scrub_stamp":"2024-09-16T15:46:20.126241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.10","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-09-16T15:46:19.139056+0000","last_change":"2024-09-16T15:46:19.139056+0000","last_active":"2024-09-16T15:46:19.122499+0000","last_peered":"2024-09-16T15:46:19.122499+0000","last_clean":"2024-09-16T15:46:19.122499+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:46:19.139056+0000","last_undegraded":"2024-09-16T15:46:19.139056+0000","last_fullsized":"2024-09-16T15:46:19.139056+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:19.122499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:19.122499+0000","last_clean_scrub_stamp":"2024-09-16T15:46:19.122499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T15:46:20.126241+0000","last_change":"2024-09-16T15:46:20.126241+0000","last_active":"2024-09-16T15:46:20.126241+0000","last_peered":"2024-09-16T15:46:20.126241+0000","last_clean":"2024-09-16T15:46:20.126241+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:46:20.126241+0000","last_undegraded":"2024-09-16T15:46:20.126241+0000","last_fullsized":"2024-09-16T15:46:20.126241+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:20.126241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:20.126241+0000","last_clean_scrub_stamp":"2024-09-16T15:46:20.126241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.f","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-09-16T15:46:19.135250+0000","last_change":"2024-09-16T15:46:19.135250+0000","last_active":"2024-09-16T15:46:19.122499+0000","last_peered":"2024-09-16T15:46:19.122499+0000","last_clean":"2024-09-16T15:46:19.122499+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:46:19.135250+0000","last_undegraded":"2024-09-16T15:46:19.135250+0000","last_fullsized":"2024-09-16T15:46:19.135250+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:19.122499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:19.122499+0000","last_clean_scrub_stamp":"2024-09-16T15:46:19.122499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T15:46:20.126241+0000","last_change":"2024-09-16T15:46:20.126241+0000","last_active":"2024-09-16T15:46:20.126241+0000","last_peered":"2024-09-16T15:46:20.126241+0000","last_clean":"2024-09-16T15:46:20.126241+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:46:20.126241+0000","last_undegraded":"2024-09-16T15:46:20.126241+0000","last_fullsized":"2024-09-16T15:46:20.126241+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:20.126241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:20.126241+0000","last_clean_scrub_stamp":"2024-09-16T15:46:20.126241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.e","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-09-16T15:46:19.136335+0000","last_change":"2024-09-16T15:46:19.136335+0000","last_active":"2024-09-16T15:46:19.122499+0000","last_peered":"2024-09-16T15:46:19.122499+0000","last_clean":"2024-09-16T15:46:19.122499+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:46:19.136335+0000","last_undegraded":"2024-09-16T15:46:19.136335+0000","last_fullsized":"2024-09-16T15:46:19.136335+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:19.122499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:19.122499+0000","last_clean_scrub_stamp":"2024-09-16T15:46:19.122499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T15:46:20.126241+0000","last_change":"2024-09-16T15:46:20.126241+0000","last_active":"2024-09-16T15:46:20.126241+0000","last_peered":"2024-09-16T15:46:20.126241+0000","last_clean":"2024-09-16T15:46:20.126241+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:46:20.126241+0000","last_undegraded":"2024-09-16T15:46:20.126241+0000","last_fullsized":"2024-09-16T15:46:20.126241+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:20.126241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:20.126241+0000","last_clean_scrub_stamp":"2024-09-16T15:46:20.126241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.d","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-09-16T15:46:19.136813+0000","last_change":"2024-09-16T15:46:19.136813+0000","last_active":"2024-09-16T15:46:19.122499+0000","last_peered":"2024-09-16T15:46:19.122499+0000","last_clean":"2024-09-16T15:46:19.122499+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:46:19.136813+0000","last_undegraded":"2024-09-16T15:46:19.136813+0000","last_fullsized":"2024-09-16T15:46:19.136813+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:19.122499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:19.122499+0000","last_clean_scrub_stamp":"2024-09-16T15:46:19.122499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T15:46:20.126241+0000","last_change":"2024-09-16T15:46:20.126241+0000","last_active":"2024-09-16T15:46:20.126241+0000","last_peered":"2024-09-16T15:46:20.126241+0000","last_clean":"2024-09-16T15:46:20.126241+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:46:20.126241+0000","last_undegraded":"2024-09-16T15:46:20.126241+0000","last_fullsized":"2024-09-16T15:46:20.126241+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:20.126241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:20.126241+0000","last_clean_scrub_stamp":"2024-09-16T15:46:20.126241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.c","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-09-16T15:46:19.135494+0000","last_change":"2024-09-16T15:46:19.135494+0000","last_active":"2024-09-16T15:46:19.122499+0000","last_peered":"2024-09-16T15:46:19.122499+0000","last_clean":"2024-09-16T15:46:19.122499+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:46:19.135494+0000","last_undegraded":"2024-09-16T15:46:19.135494+0000","last_fullsized":"2024-09-16T15:46:19.135494+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:19.122499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:19.122499+0000","last_clean_scrub_stamp":"2024-09-16T15:46:19.122499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T15:46:20.126241+0000","last_change":"2024-09-16T15:46:20.126241+0000","last_active":"2024-09-16T15:46:20.126241+0000","last_peered":"2024-09-16T15:46:20.126241+0000","last_clean":"2024-09-16T15:46:20.126241+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:46:20.126241+0000","last_undegraded":"2024-09-16T15:46:20.126241+0000","last_fullsized":"2024-09-16T15:46:20.126241+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:20.126241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:20.126241+0000","last_clean_scrub_stamp":"2024-09-16T15:46:20.126241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.b","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-09-16T15:46:19.135577+0000","last_change":"2024-09-16T15:46:19.135577+0000","last_active":"2024-09-16T15:46:19.122499+0000","last_peered":"2024-09-16T15:46:19.122499+0000","last_clean":"2024-09-16T15:46:19.122499+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:46:19.135577+0000","last_undegraded":"2024-09-16T15:46:19.135577+0000","last_fullsized":"2024-09-16T15:46:19.135577+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:19.122499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:19.122499+0000","last_clean_scrub_stamp":"2024-09-16T15:46:19.122499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T15:46:20.126241+0000","last_change":"2024-09-16T15:46:20.126241+0000","last_active":"2024-09-16T15:46:20.126241+0000","last_peered":"2024-09-16T15:46:20.126241+0000","last_clean":"2024-09-16T15:46:20.126241+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:46:20.126241+0000","last_undegraded":"2024-09-16T15:46:20.126241+0000","last_fullsized":"2024-09-16T15:46:20.126241+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:20.126241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:20.126241+0000","last_clean_scrub_stamp":"2024-09-16T15:46:20.126241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.a","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-09-16T15:46:19.132713+0000","last_change":"2024-09-16T15:46:19.132713+0000","last_active":"2024-09-16T15:46:19.122499+0000","last_peered":"2024-09-16T15:46:19.122499+0000","last_clean":"2024-09-16T15:46:19.122499+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:46:19.132713+0000","last_undegraded":"2024-09-16T15:46:19.132713+0000","last_fullsized":"2024-09-16T15:46:19.132713+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:19.122499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:19.122499+0000","last_clean_scrub_stamp":"2024-09-16T15:46:19.122499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T15:46:20.126241+0000","last_change":"2024-09-16T15:46:20.126241+0000","last_active":"2024-09-16T15:46:20.126241+0000","last_peered":"2024-09-16T15:46:20.126241+0000","last_clean":"2024-09-16T15:46:20.126241+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:46:20.126241+0000","last_undegraded":"2024-09-16T15:46:20.126241+0000","last_fullsized":"2024-09-16T15:46:20.126241+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:20.126241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:20.126241+0000","last_clean_scrub_stamp":"2024-09-16T15:46:20.126241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.8","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-09-16T15:46:19.136738+0000","last_change":"2024-09-16T15:46:19.136738+0000","last_active":"2024-09-16T15:46:19.122499+0000","last_peered":"2024-09-16T15:46:19.122499+0000","last_clean":"2024-09-16T15:46:19.122499+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:46:19.136738+0000","last_undegraded":"2024-09-16T15:46:19.136738+0000","last_fullsized":"2024-09-16T15:46:19.136738+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:19.122499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:19.122499+0000","last_clean_scrub_stamp":"2024-09-16T15:46:19.122499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T15:46:20.126241+0000","last_change":"2024-09-16T15:46:20.126241+0000","last_active":"2024-09-16T15:46:20.126241+0000","last_peered":"2024-09-16T15:46:20.126241+0000","last_clean":"2024-09-16T15:46:20.126241+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:46:20.126241+0000","last_undegraded":"2024-09-16T15:46:20.126241+0000","last_fullsized":"2024-09-16T15:46:20.126241+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:20.126241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:20.126241+0000","last_clean_scrub_stamp":"2024-09-16T15:46:20.126241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.7","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-09-16T15:46:19.132337+0000","last_change":"2024-09-16T15:46:19.132337+0000","last_active":"2024-09-16T15:46:19.122499+0000","last_peered":"2024-09-16T15:46:19.122499+0000","last_clean":"2024-09-16T15:46:19.122499+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:46:19.132337+0000","last_undegraded":"2024-09-16T15:46:19.132337+0000","last_fullsized":"2024-09-16T15:46:19.132337+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:19.122499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:19.122499+0000","last_clean_scrub_stamp":"2024-09-16T15:46:19.122499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T15:46:20.126241+0000","last_change":"2024-09-16T15:46:20.126241+0000","last_active":"2024-09-16T15:46:20.126241+0000","last_peered":"2024-09-16T15:46:20.126241+0000","last_clean":"2024-09-16T15:46:20.126241+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:46:20.126241+0000","last_undegraded":"2024-09-16T15:46:20.126241+0000","last_fullsized":"2024-09-16T15:46:20.126241+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:20.126241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:20.126241+0000","last_clean_scrub_stamp":"2024-09-16T15:46:20.126241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.6","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-09-16T15:46:19.135757+0000","last_change":"2024-09-16T15:46:19.135757+0000","last_active":"2024-09-16T15:46:19.122499+0000","last_peered":"2024-09-16T15:46:19.122499+0000","last_clean":"2024-09-16T15:46:19.122499+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:46:19.135757+0000","last_undegraded":"2024-09-16T15:46:19.135757+0000","last_fullsized":"2024-09-16T15:46:19.135757+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:19.122499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:19.122499+0000","last_clean_scrub_stamp":"2024-09-16T15:46:19.122499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T15:46:20.126241+0000","last_change":"2024-09-16T15:46:20.126241+0000","last_active":"2024-09-16T15:46:20.126241+0000","last_peered":"2024-09-16T15:46:20.126241+0000","last_clean":"2024-09-16T15:46:20.126241+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:46:20.126241+0000","last_undegraded":"2024-09-16T15:46:20.126241+0000","last_fullsized":"2024-09-16T15:46:20.126241+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:20.126241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:20.126241+0000","last_clean_scrub_stamp":"2024-09-16T15:46:20.126241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.5","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-09-16T15:46:19.134043+0000","last_change":"2024-09-16T15:46:19.134043+0000","last_active":"2024-09-16T15:46:19.122499+0000","last_peered":"2024-09-16T15:46:19.122499+0000","last_clean":"2024-09-16T15:46:19.122499+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:46:19.134043+0000","last_undegraded":"2024-09-16T15:46:19.134043+0000","last_fullsized":"2024-09-16T15:46:19.134043+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:19.122499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:19.122499+0000","last_clean_scrub_stamp":"2024-09-16T15:46:19.122499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T15:46:20.126241+0000","last_change":"2024-09-16T15:46:20.126241+0000","last_active":"2024-09-16T15:46:20.126241+0000","last_peered":"2024-09-16T15:46:20.126241+0000","last_clean":"2024-09-16T15:46:20.126241+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:46:20.126241+0000","last_undegraded":"2024-09-16T15:46:20.126241+0000","last_fullsized":"2024-09-16T15:46:20.126241+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:20.126241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:20.126241+0000","last_clean_scrub_stamp":"2024-09-16T15:46:20.126241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.4","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-09-16T15:46:19.136795+0000","last_change":"2024-09-16T15:46:19.136795+0000","last_active":"2024-09-16T15:46:19.122499+0000","last_peered":"2024-09-16T15:46:19.122499+0000","last_clean":"2024-09-16T15:46:19.122499+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:46:19.136795+0000","last_undegraded":"2024-09-16T15:46:19.136795+0000","last_fullsized":"2024-09-16T15:46:19.136795+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:19.122499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:19.122499+0000","last_clean_scrub_stamp":"2024-09-16T15:46:19.122499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T15:46:20.126241+0000","last_change":"2024-09-16T15:46:20.126241+0000","last_active":"2024-09-16T15:46:20.126241+0000","last_peered":"2024-09-16T15:46:20.126241+0000","last_clean":"2024-09-16T15:46:20.126241+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:46:20.126241+0000","last_undegraded":"2024-09-16T15:46:20.126241+0000","last_fullsized":"2024-09-16T15:46:20.126241+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:20.126241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:20.126241+0000","last_clean_scrub_stamp":"2024-09-16T15:46:20.126241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.3","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-09-16T15:46:19.132004+0000","last_change":"2024-09-16T15:46:19.132004+0000","last_active":"2024-09-16T15:46:19.122499+0000","last_peered":"2024-09-16T15:46:19.122499+0000","last_clean":"2024-09-16T15:46:19.122499+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:46:19.132004+0000","last_undegraded":"2024-09-16T15:46:19.132004+0000","last_fullsized":"2024-09-16T15:46:19.132004+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:19.122499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:19.122499+0000","last_clean_scrub_stamp":"2024-09-16T15:46:19.122499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T15:46:20.126241+0000","last_change":"2024-09-16T15:46:20.126241+0000","last_active":"2024-09-16T15:46:20.126241+0000","last_peered":"2024-09-16T15:46:20.126241+0000","last_clean":"2024-09-16T15:46:20.126241+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:46:20.126241+0000","last_undegraded":"2024-09-16T15:46:20.126241+0000","last_fullsized":"2024-09-16T15:46:20.126241+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:20.126241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:20.126241+0000","last_clean_scrub_stamp":"2024-09-16T15:46:20.126241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.2","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-09-16T15:46:19.130973+0000","last_change":"2024-09-16T15:46:19.130973+0000","last_active":"2024-09-16T15:46:19.122499+0000","last_peered":"2024-09-16T15:46:19.122499+0000","last_clean":"2024-09-16T15:46:19.122499+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:46:19.130973+0000","last_undegraded":"2024-09-16T15:46:19.130973+0000","last_fullsized":"2024-09-16T15:46:19.130973+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:19.122499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:19.122499+0000","last_clean_scrub_stamp":"2024-09-16T15:46:19.122499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T15:46:20.126241+0000","last_change":"2024-09-16T15:46:20.126241+0000","last_active":"2024-09-16T15:46:20.126241+0000","last_peered":"2024-09-16T15:46:20.126241+0000","last_clean":"2024-09-16T15:46:20.126241+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:46:20.126241+0000","last_undegraded":"2024-09-16T15:46:20.126241+0000","last_fullsized":"2024-09-16T15:46:20.126241+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:20.126241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:20.126241+0000","last_clean_scrub_stamp":"2024-09-16T15:46:20.126241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.1","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-09-16T15:46:19.132292+0000","last_change":"2024-09-16T15:46:19.132292+0000","last_active":"2024-09-16T15:46:19.122499+0000","last_peered":"2024-09-16T15:46:19.122499+0000","last_clean":"2024-09-16T15:46:19.122499+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:46:19.132292+0000","last_undegraded":"2024-09-16T15:46:19.132292+0000","last_fullsized":"2024-09-16T15:46:19.132292+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:19.122499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:19.122499+0000","last_clean_scrub_stamp":"2024-09-16T15:46:19.122499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T15:46:20.126241+0000","last_change":"2024-09-16T15:46:20.126241+0000","last_active":"2024-09-16T15:46:20.126241+0000","last_peered":"2024-09-16T15:46:20.126241+0000","last_clean":"2024-09-16T15:46:20.126241+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:46:20.126241+0000","last_undegraded":"2024-09-16T15:46:20.126241+0000","last_fullsized":"2024-09-16T15:46:20.126241+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:20.126241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:20.126241+0000","last_clean_scrub_stamp":"2024-09-16T15:46:20.126241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.0","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-09-16T15:46:19.137049+0000","last_change":"2024-09-16T15:46:19.137049+0000","last_active":"2024-09-16T15:46:19.122499+0000","last_peered":"2024-09-16T15:46:19.122499+0000","last_clean":"2024-09-16T15:46:19.122499+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:46:19.137049+0000","last_undegraded":"2024-09-16T15:46:19.137049+0000","last_fullsized":"2024-09-16T15:46:19.137049+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:19.122499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:19.122499+0000","last_clean_scrub_stamp":"2024-09-16T15:46:19.122499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T15:46:20.126241+0000","last_change":"2024-09-16T15:46:20.126241+0000","last_active":"2024-09-16T15:46:20.126241+0000","last_peered":"2024-09-16T15:46:20.126241+0000","last_clean":"2024-09-16T15:46:20.126241+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:46:20.126241+0000","last_undegraded":"2024-09-16T15:46:20.126241+0000","last_fullsized":"2024-09-16T15:46:20.126241+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:20.126241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:20.126241+0000","last_clean_scrub_stamp":"2024-09-16T15:46:20.126241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.9","version":"0'0","reported_seq":1,"reported_epoch":55,"state":"creating+peering","last_fresh":"2024-09-16T15:46:19.134061+0000","last_change":"2024-09-16T15:46:19.134061+0000","last_active":"2024-09-16T15:46:19.122499+0000","last_peered":"2024-09-16T15:46:19.122499+0000","last_clean":"2024-09-16T15:46:19.122499+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:46:19.134061+0000","last_undegraded":"2024-09-16T15:46:19.134061+0000","last_fullsized":"2024-09-16T15:46:19.134061+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:19.122499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:19.122499+0000","last_clean_scrub_stamp":"2024-09-16T15:46:19.122499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"16'72","reported_seq":175,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-16T15:46:19.142673+0000","last_change":"2024-09-16T15:42:40.881820+0000","last_active":"2024-09-16T15:46:19.142673+0000","last_peered":"2024-09-16T15:46:19.142673+0000","last_clean":"2024-09-16T15:46:19.142673+0000","last_became_active":"2024-09-16T15:42:40.879965+0000","last_became_peered":"2024-09-16T15:42:40.879965+0000","last_unstale":"2024-09-16T15:46:19.142673+0000","last_undegraded":"2024-09-16T15:46:19.142673+0000","last_fullsized":"2024-09-16T15:46:19.142673+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:42:39.728762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:42:39.728762+0000","last_clean_scrub_stamp":"2024-09-16T15:42:39.728762+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:22:18.768932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":9,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},{"poolid":8,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542192,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28248,"kb_used_data":1208,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343592,"statfs":{"total":96636764160,"available":96607838208,"internally_reserved":0,"allocated":1236992,"data_stored":851662,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542191,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31832,"kb_used_data":628,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340008,"statfs":{"total":96636764160,"available":96604168192,"internally_reserved":0,"allocated":643072,"data_stored":261294,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":14,"seq":60129542192,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27656,"kb_used_data":628,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344184,"statfs":{"total":96636764160,"available":96608444416,"internally_reserved":0,"allocated":643072,"data_stored":261294,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542190,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31828,"kb_used_data":640,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340012,"statfs":{"total":96636764160,"available":96604172288,"internally_reserved":0,"allocated":655360,"data_stored":267111,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":14,"seq":60129542191,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31824,"kb_used_data":628,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340016,"statfs":{"total":96636764160,"available":96604176384,"internally_reserved":0,"allocated":643072,"data_stored":261294,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":2,"up_from":14,"seq":60129542191,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27656,"kb_used_data":628,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344184,"statfs":{"total":96636764160,"available":96608444416,"internally_reserved":0,"allocated":643072,"data_stored":261294,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":14,"seq":60129542193,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27672,"kb_used_data":640,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344168,"statfs":{"total":96636764160,"available":96608428032,"internally_reserved":0,"allocated":655360,"data_stored":267111,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":14,"seq":60129542193,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28244,"kb_used_data":1208,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343596,"statfs":{"total":96636764160,"available":96607842304,"internally_reserved":0,"allocated":1236992,"data_stored":851662,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T15:46:25.299 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T15:46:25.427 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:25.425+0000 7f0292a8d640 1 -- 172.21.15.123:0/744929227 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f028c0cf5a0 msgr2=0x7f028c06eee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:46:25.427 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:25.425+0000 7f0292a8d640 1 --2- 172.21.15.123:0/744929227 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f028c0cf5a0 0x7f028c06eee0 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7f028c076280 tx=0x7f027c01c7a0 comp rx=0 tx=0).stop 2024-09-16T15:46:25.427 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:25.425+0000 7f0292a8d640 1 -- 172.21.15.123:0/744929227 shutdown_connections 2024-09-16T15:46:25.428 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:25.425+0000 7f0292a8d640 1 --2- 172.21.15.123:0/744929227 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f028c0cf5a0 0x7f028c06eee0 unknown :-1 s=CLOSED pgs=644 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:25.428 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:25.425+0000 7f0292a8d640 1 --2- 172.21.15.123:0/744929227 >> v2:172.21.15.184:3300/0 conn(0x7f028c158120 0x7f028c0ca320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:25.428 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:25.425+0000 7f0292a8d640 1 -- 172.21.15.123:0/744929227 >> 172.21.15.123:0/744929227 conn(0x7f028c0cfb10 msgr2=0x7f028c0cff20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:46:25.428 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:25.425+0000 7f0292a8d640 1 -- 172.21.15.123:0/744929227 shutdown_connections 2024-09-16T15:46:25.428 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:25.425+0000 7f0292a8d640 1 -- 172.21.15.123:0/744929227 wait complete. 2024-09-16T15:46:25.428 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:25.425+0000 7f0292a8d640 1 Processor -- start 2024-09-16T15:46:25.429 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:25.425+0000 7f0292a8d640 1 -- start start 2024-09-16T15:46:25.433 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:25.425+0000 7f0292a8d640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f028c0cf5a0 0x7f028c168f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:46:25.434 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:25.425+0000 7f0292a8d640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f028c158120 0x7f028c167e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:46:25.434 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:25.425+0000 7f0292a8d640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f028c15ec40 0x7f028c0b0620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:46:25.434 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:25.425+0000 7f0292a8d640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f028c15b720 con 0x7f028c15ec40 2024-09-16T15:46:25.434 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:25.425+0000 7f0292a8d640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f028c15b5a0 con 0x7f028c0cf5a0 2024-09-16T15:46:25.434 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:25.425+0000 7f0292a8d640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f028c15b8a0 con 0x7f028c158120 2024-09-16T15:46:25.434 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:25.425+0000 7f029102a640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f028c15ec40 0x7f028c0b0620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:46:25.434 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:25.425+0000 7f029102a640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f028c15ec40 0x7f028c0b0620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:54680/0 (socket says 172.21.15.123:54680) 2024-09-16T15:46:25.434 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:25.425+0000 7f029102a640 1 -- 172.21.15.123:0/2723381858 learned_addr learned my addr 172.21.15.123:0/2723381858 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:46:25.434 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:25.425+0000 7f0290829640 1 --2- 172.21.15.123:0/2723381858 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f028c0cf5a0 0x7f028c168f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:46:25.434 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:25.425+0000 7f028bfff640 1 --2- 172.21.15.123:0/2723381858 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f028c158120 0x7f028c167e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:46:25.434 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:25.425+0000 7f029102a640 1 -- 172.21.15.123:0/2723381858 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f028c158120 msgr2=0x7f028c167e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:46:25.435 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:25.425+0000 7f029102a640 1 --2- 172.21.15.123:0/2723381858 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f028c158120 0x7f028c167e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:25.435 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:25.425+0000 7f029102a640 1 -- 172.21.15.123:0/2723381858 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f028c0cf5a0 msgr2=0x7f028c168f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:46:25.435 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:25.425+0000 7f029102a640 1 --2- 172.21.15.123:0/2723381858 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f028c0cf5a0 0x7f028c168f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:25.435 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:25.425+0000 7f029102a640 1 -- 172.21.15.123:0/2723381858 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f028c169440 con 0x7f028c15ec40 2024-09-16T15:46:25.435 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:25.425+0000 7f0290829640 1 --2- 172.21.15.123:0/2723381858 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f028c0cf5a0 0x7f028c168f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T15:46:25.435 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:25.425+0000 7f029102a640 1 --2- 172.21.15.123:0/2723381858 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f028c15ec40 0x7f028c0b0620 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7f028400b970 tx=0x7f028400be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:46:25.435 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:25.425+0000 7f02897fa640 1 -- 172.21.15.123:0/2723381858 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0284019070 con 0x7f028c15ec40 2024-09-16T15:46:25.435 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:25.425+0000 7f02897fa640 1 -- 172.21.15.123:0/2723381858 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f028400cbd0 con 0x7f028c15ec40 2024-09-16T15:46:25.435 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:25.425+0000 7f0292a8d640 1 -- 172.21.15.123:0/2723381858 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f028c1683c0 con 0x7f028c15ec40 2024-09-16T15:46:25.435 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:25.425+0000 7f02897fa640 1 -- 172.21.15.123:0/2723381858 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02840146a0 con 0x7f028c15ec40 2024-09-16T15:46:25.435 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:25.425+0000 7f0292a8d640 1 -- 172.21.15.123:0/2723381858 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f028c1fb970 con 0x7f028c15ec40 2024-09-16T15:46:25.435 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:25.429+0000 7f0292a8d640 1 -- 172.21.15.123:0/2723381858 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f028c06eee0 con 0x7f028c15ec40 2024-09-16T15:46:25.436 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:25.429+0000 7f02897fa640 1 -- 172.21.15.123:0/2723381858 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 9) ==== 97233+0+0 (secure 0 0 0) 0x7f028401c070 con 0x7f028c15ec40 2024-09-16T15:46:25.436 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:25.429+0000 7f02897fa640 1 --2- 172.21.15.123:0/2723381858 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f0278073070 0x7f0278075530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:46:25.436 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:25.429+0000 7f02897fa640 1 -- 172.21.15.123:0/2723381858 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(58..58 src has 1..58) ==== 5709+0+0 (secure 0 0 0) 0x7f0284093ef0 con 0x7f028c15ec40 2024-09-16T15:46:25.436 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:25.429+0000 7f0290829640 1 --2- 172.21.15.123:0/2723381858 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f0278073070 0x7f0278075530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:46:25.436 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:25.429+0000 7f0290829640 1 --2- 172.21.15.123:0/2723381858 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f0278073070 0x7f0278075530 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f0280005fd0 tx=0x7f0280005d60 comp rx=0 tx=0).ready entity=mgr.4135 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:46:25.436 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:25.433+0000 7f02897fa640 1 -- 172.21.15.123:0/2723381858 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f028405ee40 con 0x7f028c15ec40 2024-09-16T15:46:25.603 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:25.601+0000 7f0292a8d640 1 -- 172.21.15.123:0/2723381858 --> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f028c15db00 con 0x7f0278073070 2024-09-16T15:46:25.615 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:25.613+0000 7f02897fa640 1 -- 172.21.15.123:0/2723381858 <== mgr.4135 v2:172.21.15.184:6832/3114346423 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+251950 (secure 0 0 0) 0x7f028c15db00 con 0x7f0278073070 2024-09-16T15:46:25.618 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-09-16T15:46:25.619 INFO:teuthology.orchestra.run.smithi123.stderr:dumped all 2024-09-16T15:46:25.620 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:25.617+0000 7f0292a8d640 1 -- 172.21.15.123:0/2723381858 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f0278073070 msgr2=0x7f0278075530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:46:25.621 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:25.617+0000 7f0292a8d640 1 --2- 172.21.15.123:0/2723381858 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f0278073070 0x7f0278075530 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f0280005fd0 tx=0x7f0280005d60 comp rx=0 tx=0).stop 2024-09-16T15:46:25.621 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:25.617+0000 7f0292a8d640 1 -- 172.21.15.123:0/2723381858 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f028c15ec40 msgr2=0x7f028c0b0620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:46:25.621 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:25.617+0000 7f0292a8d640 1 --2- 172.21.15.123:0/2723381858 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f028c15ec40 0x7f028c0b0620 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7f028400b970 tx=0x7f028400be30 comp rx=0 tx=0).stop 2024-09-16T15:46:25.621 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:25.617+0000 7f0292a8d640 1 -- 172.21.15.123:0/2723381858 shutdown_connections 2024-09-16T15:46:25.621 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:25.617+0000 7f0292a8d640 1 --2- 172.21.15.123:0/2723381858 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f0278073070 0x7f0278075530 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:25.621 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:25.617+0000 7f0292a8d640 1 --2- 172.21.15.123:0/2723381858 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f028c15ec40 0x7f028c0b0620 unknown :-1 s=CLOSED pgs=645 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:25.621 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:25.617+0000 7f0292a8d640 1 --2- 172.21.15.123:0/2723381858 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f028c158120 0x7f028c167e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:25.621 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:25.617+0000 7f0292a8d640 1 --2- 172.21.15.123:0/2723381858 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f028c0cf5a0 0x7f028c168f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:25.621 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:25.617+0000 7f0292a8d640 1 -- 172.21.15.123:0/2723381858 >> 172.21.15.123:0/2723381858 conn(0x7f028c0cfb10 msgr2=0x7f028c156080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:46:25.621 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:25.617+0000 7f0292a8d640 1 -- 172.21.15.123:0/2723381858 shutdown_connections 2024-09-16T15:46:25.621 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:25.617+0000 7f0292a8d640 1 -- 172.21.15.123:0/2723381858 wait complete. 2024-09-16T15:46:25.636 INFO:teuthology.orchestra.run.smithi123.stdout:{"pg_ready":true,"pg_map":{"version":164,"stamp":"2024-09-16T15:46:25.318483+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":146,"num_write_kb":1642,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":111,"ondisk_log_size":111,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":149,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":235372,"kb_used_data":6632,"kb_used_omap":12,"kb_used_meta":228531,"kb_avail":754739348,"statfs":{"total":773094113280,"available":772853092352,"internally_reserved":0,"allocated":6791168,"data_stored":3477180,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":234016305},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":53,"apply_latency_ms":53,"commit_latency_ns":53000000,"apply_latency_ns":53000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001799"},"pg_stats":[{"pgid":"9.3f","version":"0'0","reported_seq":13,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-16T15:46:22.664500+0000","last_change":"2024-09-16T15:46:21.167413+0000","last_active":"2024-09-16T15:46:22.664500+0000","last_peered":"2024-09-16T15:46:22.664500+0000","last_clean":"2024-09-16T15:46:22.664500+0000","last_became_active":"2024-09-16T15:46:21.167152+0000","last_became_peered":"2024-09-16T15:46:21.167152+0000","last_unstale":"2024-09-16T15:46:22.664500+0000","last_undegraded":"2024-09-16T15:46:22.664500+0000","last_fullsized":"2024-09-16T15:46:22.664500+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:20.126241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:20.126241+0000","last_clean_scrub_stamp":"2024-09-16T15:46:20.126241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T00:24:38.553854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.3e","version":"0'0","reported_seq":13,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-16T15:46:22.665429+0000","last_change":"2024-09-16T15:46:21.162606+0000","last_active":"2024-09-16T15:46:22.665429+0000","last_peered":"2024-09-16T15:46:22.665429+0000","last_clean":"2024-09-16T15:46:22.665429+0000","last_became_active":"2024-09-16T15:46:21.162339+0000","last_became_peered":"2024-09-16T15:46:21.162339+0000","last_unstale":"2024-09-16T15:46:22.665429+0000","last_undegraded":"2024-09-16T15:46:22.665429+0000","last_fullsized":"2024-09-16T15:46:22.665429+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:20.126241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:20.126241+0000","last_clean_scrub_stamp":"2024-09-16T15:46:20.126241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:18:52.432458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.3d","version":"0'0","reported_seq":13,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-16T15:46:22.154836+0000","last_change":"2024-09-16T15:46:21.200650+0000","last_active":"2024-09-16T15:46:22.154836+0000","last_peered":"2024-09-16T15:46:22.154836+0000","last_clean":"2024-09-16T15:46:22.154836+0000","last_became_active":"2024-09-16T15:46:21.199232+0000","last_became_peered":"2024-09-16T15:46:21.199232+0000","last_unstale":"2024-09-16T15:46:22.154836+0000","last_undegraded":"2024-09-16T15:46:22.154836+0000","last_fullsized":"2024-09-16T15:46:22.154836+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:20.126241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:20.126241+0000","last_clean_scrub_stamp":"2024-09-16T15:46:20.126241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T18:43:19.978239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.3c","version":"0'0","reported_seq":13,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-16T15:46:22.662538+0000","last_change":"2024-09-16T15:46:21.180171+0000","last_active":"2024-09-16T15:46:22.662538+0000","last_peered":"2024-09-16T15:46:22.662538+0000","last_clean":"2024-09-16T15:46:22.662538+0000","last_became_active":"2024-09-16T15:46:21.179341+0000","last_became_peered":"2024-09-16T15:46:21.179341+0000","last_unstale":"2024-09-16T15:46:22.662538+0000","last_undegraded":"2024-09-16T15:46:22.662538+0000","last_fullsized":"2024-09-16T15:46:22.662538+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:20.126241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:20.126241+0000","last_clean_scrub_stamp":"2024-09-16T15:46:20.126241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T02:30:25.147666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.3b","version":"0'0","reported_seq":13,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-16T15:46:22.662779+0000","last_change":"2024-09-16T15:46:21.173252+0000","last_active":"2024-09-16T15:46:22.662779+0000","last_peered":"2024-09-16T15:46:22.662779+0000","last_clean":"2024-09-16T15:46:22.662779+0000","last_became_active":"2024-09-16T15:46:21.172965+0000","last_became_peered":"2024-09-16T15:46:21.172965+0000","last_unstale":"2024-09-16T15:46:22.662779+0000","last_undegraded":"2024-09-16T15:46:22.662779+0000","last_fullsized":"2024-09-16T15:46:22.662779+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:20.126241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:20.126241+0000","last_clean_scrub_stamp":"2024-09-16T15:46:20.126241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:07:55.055754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.3a","version":"0'0","reported_seq":13,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-16T15:46:22.664735+0000","last_change":"2024-09-16T15:46:21.155302+0000","last_active":"2024-09-16T15:46:22.664735+0000","last_peered":"2024-09-16T15:46:22.664735+0000","last_clean":"2024-09-16T15:46:22.664735+0000","last_became_active":"2024-09-16T15:46:21.154929+0000","last_became_peered":"2024-09-16T15:46:21.154929+0000","last_unstale":"2024-09-16T15:46:22.664735+0000","last_undegraded":"2024-09-16T15:46:22.664735+0000","last_fullsized":"2024-09-16T15:46:22.664735+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:20.126241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:20.126241+0000","last_clean_scrub_stamp":"2024-09-16T15:46:20.126241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:43:07.519678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.39","version":"0'0","reported_seq":13,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-16T15:46:22.150756+0000","last_change":"2024-09-16T15:46:21.187697+0000","last_active":"2024-09-16T15:46:22.150756+0000","last_peered":"2024-09-16T15:46:22.150756+0000","last_clean":"2024-09-16T15:46:22.150756+0000","last_became_active":"2024-09-16T15:46:21.186764+0000","last_became_peered":"2024-09-16T15:46:21.186764+0000","last_unstale":"2024-09-16T15:46:22.150756+0000","last_undegraded":"2024-09-16T15:46:22.150756+0000","last_fullsized":"2024-09-16T15:46:22.150756+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:20.126241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:20.126241+0000","last_clean_scrub_stamp":"2024-09-16T15:46:20.126241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:50:27.021076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.38","version":"0'0","reported_seq":13,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-16T15:46:22.663289+0000","last_change":"2024-09-16T15:46:21.174046+0000","last_active":"2024-09-16T15:46:22.663289+0000","last_peered":"2024-09-16T15:46:22.663289+0000","last_clean":"2024-09-16T15:46:22.663289+0000","last_became_active":"2024-09-16T15:46:21.173593+0000","last_became_peered":"2024-09-16T15:46:21.173593+0000","last_unstale":"2024-09-16T15:46:22.663289+0000","last_undegraded":"2024-09-16T15:46:22.663289+0000","last_fullsized":"2024-09-16T15:46:22.663289+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:20.126241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:20.126241+0000","last_clean_scrub_stamp":"2024-09-16T15:46:20.126241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:21:13.633938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.37","version":"0'0","reported_seq":13,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-16T15:46:22.152348+0000","last_change":"2024-09-16T15:46:21.153022+0000","last_active":"2024-09-16T15:46:22.152348+0000","last_peered":"2024-09-16T15:46:22.152348+0000","last_clean":"2024-09-16T15:46:22.152348+0000","last_became_active":"2024-09-16T15:46:21.152211+0000","last_became_peered":"2024-09-16T15:46:21.152211+0000","last_unstale":"2024-09-16T15:46:22.152348+0000","last_undegraded":"2024-09-16T15:46:22.152348+0000","last_fullsized":"2024-09-16T15:46:22.152348+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:20.126241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:20.126241+0000","last_clean_scrub_stamp":"2024-09-16T15:46:20.126241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:00:24.268395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.36","version":"0'0","reported_seq":13,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-16T15:46:22.146272+0000","last_change":"2024-09-16T15:46:21.162511+0000","last_active":"2024-09-16T15:46:22.146272+0000","last_peered":"2024-09-16T15:46:22.146272+0000","last_clean":"2024-09-16T15:46:22.146272+0000","last_became_active":"2024-09-16T15:46:21.162257+0000","last_became_peered":"2024-09-16T15:46:21.162257+0000","last_unstale":"2024-09-16T15:46:22.146272+0000","last_undegraded":"2024-09-16T15:46:22.146272+0000","last_fullsized":"2024-09-16T15:46:22.146272+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:20.126241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:20.126241+0000","last_clean_scrub_stamp":"2024-09-16T15:46:20.126241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:52:19.636331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.35","version":"0'0","reported_seq":13,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-16T15:46:22.145910+0000","last_change":"2024-09-16T15:46:21.161816+0000","last_active":"2024-09-16T15:46:22.145910+0000","last_peered":"2024-09-16T15:46:22.145910+0000","last_clean":"2024-09-16T15:46:22.145910+0000","last_became_active":"2024-09-16T15:46:21.161564+0000","last_became_peered":"2024-09-16T15:46:21.161564+0000","last_unstale":"2024-09-16T15:46:22.145910+0000","last_undegraded":"2024-09-16T15:46:22.145910+0000","last_fullsized":"2024-09-16T15:46:22.145910+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:20.126241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:20.126241+0000","last_clean_scrub_stamp":"2024-09-16T15:46:20.126241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:04:00.585526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.34","version":"0'0","reported_seq":13,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-16T15:46:22.663090+0000","last_change":"2024-09-16T15:46:21.183846+0000","last_active":"2024-09-16T15:46:22.663090+0000","last_peered":"2024-09-16T15:46:22.663090+0000","last_clean":"2024-09-16T15:46:22.663090+0000","last_became_active":"2024-09-16T15:46:21.178122+0000","last_became_peered":"2024-09-16T15:46:21.178122+0000","last_unstale":"2024-09-16T15:46:22.663090+0000","last_undegraded":"2024-09-16T15:46:22.663090+0000","last_fullsized":"2024-09-16T15:46:22.663090+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:20.126241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:20.126241+0000","last_clean_scrub_stamp":"2024-09-16T15:46:20.126241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T01:59:02.801025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.33","version":"0'0","reported_seq":13,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-16T15:46:22.151672+0000","last_change":"2024-09-16T15:46:21.161955+0000","last_active":"2024-09-16T15:46:22.151672+0000","last_peered":"2024-09-16T15:46:22.151672+0000","last_clean":"2024-09-16T15:46:22.151672+0000","last_became_active":"2024-09-16T15:46:21.161191+0000","last_became_peered":"2024-09-16T15:46:21.161191+0000","last_unstale":"2024-09-16T15:46:22.151672+0000","last_undegraded":"2024-09-16T15:46:22.151672+0000","last_fullsized":"2024-09-16T15:46:22.151672+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:20.126241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:20.126241+0000","last_clean_scrub_stamp":"2024-09-16T15:46:20.126241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:25:08.511212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.32","version":"0'0","reported_seq":13,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-16T15:46:22.152317+0000","last_change":"2024-09-16T15:46:21.155938+0000","last_active":"2024-09-16T15:46:22.152317+0000","last_peered":"2024-09-16T15:46:22.152317+0000","last_clean":"2024-09-16T15:46:22.152317+0000","last_became_active":"2024-09-16T15:46:21.155203+0000","last_became_peered":"2024-09-16T15:46:21.155203+0000","last_unstale":"2024-09-16T15:46:22.152317+0000","last_undegraded":"2024-09-16T15:46:22.152317+0000","last_fullsized":"2024-09-16T15:46:22.152317+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:20.126241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:20.126241+0000","last_clean_scrub_stamp":"2024-09-16T15:46:20.126241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T22:44:00.307565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.31","version":"0'0","reported_seq":13,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-16T15:46:22.151045+0000","last_change":"2024-09-16T15:46:21.157161+0000","last_active":"2024-09-16T15:46:22.151045+0000","last_peered":"2024-09-16T15:46:22.151045+0000","last_clean":"2024-09-16T15:46:22.151045+0000","last_became_active":"2024-09-16T15:46:21.156478+0000","last_became_peered":"2024-09-16T15:46:21.156478+0000","last_unstale":"2024-09-16T15:46:22.151045+0000","last_undegraded":"2024-09-16T15:46:22.151045+0000","last_fullsized":"2024-09-16T15:46:22.151045+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:20.126241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:20.126241+0000","last_clean_scrub_stamp":"2024-09-16T15:46:20.126241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:22:28.911078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.30","version":"0'0","reported_seq":13,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-16T15:46:22.663372+0000","last_change":"2024-09-16T15:46:21.177696+0000","last_active":"2024-09-16T15:46:22.663372+0000","last_peered":"2024-09-16T15:46:22.663372+0000","last_clean":"2024-09-16T15:46:22.663372+0000","last_became_active":"2024-09-16T15:46:21.177416+0000","last_became_peered":"2024-09-16T15:46:21.177416+0000","last_unstale":"2024-09-16T15:46:22.663372+0000","last_undegraded":"2024-09-16T15:46:22.663372+0000","last_fullsized":"2024-09-16T15:46:22.663372+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:20.126241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:20.126241+0000","last_clean_scrub_stamp":"2024-09-16T15:46:20.126241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T00:28:52.221941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.2f","version":"0'0","reported_seq":13,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-16T15:46:22.666015+0000","last_change":"2024-09-16T15:46:21.182852+0000","last_active":"2024-09-16T15:46:22.666015+0000","last_peered":"2024-09-16T15:46:22.666015+0000","last_clean":"2024-09-16T15:46:22.666015+0000","last_became_active":"2024-09-16T15:46:21.182624+0000","last_became_peered":"2024-09-16T15:46:21.182624+0000","last_unstale":"2024-09-16T15:46:22.666015+0000","last_undegraded":"2024-09-16T15:46:22.666015+0000","last_fullsized":"2024-09-16T15:46:22.666015+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:20.126241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:20.126241+0000","last_clean_scrub_stamp":"2024-09-16T15:46:20.126241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:12:19.497427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.2e","version":"0'0","reported_seq":13,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-16T15:46:22.155531+0000","last_change":"2024-09-16T15:46:21.187715+0000","last_active":"2024-09-16T15:46:22.155531+0000","last_peered":"2024-09-16T15:46:22.155531+0000","last_clean":"2024-09-16T15:46:22.155531+0000","last_became_active":"2024-09-16T15:46:21.186958+0000","last_became_peered":"2024-09-16T15:46:21.186958+0000","last_unstale":"2024-09-16T15:46:22.155531+0000","last_undegraded":"2024-09-16T15:46:22.155531+0000","last_fullsized":"2024-09-16T15:46:22.155531+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:20.126241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:20.126241+0000","last_clean_scrub_stamp":"2024-09-16T15:46:20.126241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T18:08:12.890914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.2d","version":"0'0","reported_seq":13,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-16T15:46:22.663828+0000","last_change":"2024-09-16T15:46:21.156659+0000","last_active":"2024-09-16T15:46:22.663828+0000","last_peered":"2024-09-16T15:46:22.663828+0000","last_clean":"2024-09-16T15:46:22.663828+0000","last_became_active":"2024-09-16T15:46:21.156319+0000","last_became_peered":"2024-09-16T15:46:21.156319+0000","last_unstale":"2024-09-16T15:46:22.663828+0000","last_undegraded":"2024-09-16T15:46:22.663828+0000","last_fullsized":"2024-09-16T15:46:22.663828+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:20.126241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:20.126241+0000","last_clean_scrub_stamp":"2024-09-16T15:46:20.126241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T00:21:55.924601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.2c","version":"0'0","reported_seq":13,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-16T15:46:22.664984+0000","last_change":"2024-09-16T15:46:21.162663+0000","last_active":"2024-09-16T15:46:22.664984+0000","last_peered":"2024-09-16T15:46:22.664984+0000","last_clean":"2024-09-16T15:46:22.664984+0000","last_became_active":"2024-09-16T15:46:21.160598+0000","last_became_peered":"2024-09-16T15:46:21.160598+0000","last_unstale":"2024-09-16T15:46:22.664984+0000","last_undegraded":"2024-09-16T15:46:22.664984+0000","last_fullsized":"2024-09-16T15:46:22.664984+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:20.126241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:20.126241+0000","last_clean_scrub_stamp":"2024-09-16T15:46:20.126241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:16:11.408338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.2b","version":"0'0","reported_seq":13,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-16T15:46:22.153601+0000","last_change":"2024-09-16T15:46:21.182806+0000","last_active":"2024-09-16T15:46:22.153601+0000","last_peered":"2024-09-16T15:46:22.153601+0000","last_clean":"2024-09-16T15:46:22.153601+0000","last_became_active":"2024-09-16T15:46:21.180957+0000","last_became_peered":"2024-09-16T15:46:21.180957+0000","last_unstale":"2024-09-16T15:46:22.153601+0000","last_undegraded":"2024-09-16T15:46:22.153601+0000","last_fullsized":"2024-09-16T15:46:22.153601+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:20.126241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:20.126241+0000","last_clean_scrub_stamp":"2024-09-16T15:46:20.126241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:18:03.272489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.2a","version":"0'0","reported_seq":13,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-16T15:46:22.663106+0000","last_change":"2024-09-16T15:46:21.161312+0000","last_active":"2024-09-16T15:46:22.663106+0000","last_peered":"2024-09-16T15:46:22.663106+0000","last_clean":"2024-09-16T15:46:22.663106+0000","last_became_active":"2024-09-16T15:46:21.161024+0000","last_became_peered":"2024-09-16T15:46:21.161024+0000","last_unstale":"2024-09-16T15:46:22.663106+0000","last_undegraded":"2024-09-16T15:46:22.663106+0000","last_fullsized":"2024-09-16T15:46:22.663106+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:20.126241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:20.126241+0000","last_clean_scrub_stamp":"2024-09-16T15:46:20.126241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T01:41:01.034633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.29","version":"0'0","reported_seq":13,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-16T15:46:22.151392+0000","last_change":"2024-09-16T15:46:21.177986+0000","last_active":"2024-09-16T15:46:22.151392+0000","last_peered":"2024-09-16T15:46:22.151392+0000","last_clean":"2024-09-16T15:46:22.151392+0000","last_became_active":"2024-09-16T15:46:21.175183+0000","last_became_peered":"2024-09-16T15:46:21.175183+0000","last_unstale":"2024-09-16T15:46:22.151392+0000","last_undegraded":"2024-09-16T15:46:22.151392+0000","last_fullsized":"2024-09-16T15:46:22.151392+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:20.126241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:20.126241+0000","last_clean_scrub_stamp":"2024-09-16T15:46:20.126241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:06:33.559242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.28","version":"0'0","reported_seq":13,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-16T15:46:22.150337+0000","last_change":"2024-09-16T15:46:21.161928+0000","last_active":"2024-09-16T15:46:22.150337+0000","last_peered":"2024-09-16T15:46:22.150337+0000","last_clean":"2024-09-16T15:46:22.150337+0000","last_became_active":"2024-09-16T15:46:21.161036+0000","last_became_peered":"2024-09-16T15:46:21.161036+0000","last_unstale":"2024-09-16T15:46:22.150337+0000","last_undegraded":"2024-09-16T15:46:22.150337+0000","last_fullsized":"2024-09-16T15:46:22.150337+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:20.126241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:20.126241+0000","last_clean_scrub_stamp":"2024-09-16T15:46:20.126241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T02:08:34.316151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.27","version":"0'0","reported_seq":13,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-16T15:46:22.665039+0000","last_change":"2024-09-16T15:46:21.166313+0000","last_active":"2024-09-16T15:46:22.665039+0000","last_peered":"2024-09-16T15:46:22.665039+0000","last_clean":"2024-09-16T15:46:22.665039+0000","last_became_active":"2024-09-16T15:46:21.166032+0000","last_became_peered":"2024-09-16T15:46:21.166032+0000","last_unstale":"2024-09-16T15:46:22.665039+0000","last_undegraded":"2024-09-16T15:46:22.665039+0000","last_fullsized":"2024-09-16T15:46:22.665039+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:20.126241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:20.126241+0000","last_clean_scrub_stamp":"2024-09-16T15:46:20.126241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T00:33:03.264535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.26","version":"0'0","reported_seq":13,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-16T15:46:22.662841+0000","last_change":"2024-09-16T15:46:21.198851+0000","last_active":"2024-09-16T15:46:22.662841+0000","last_peered":"2024-09-16T15:46:22.662841+0000","last_clean":"2024-09-16T15:46:22.662841+0000","last_became_active":"2024-09-16T15:46:21.198155+0000","last_became_peered":"2024-09-16T15:46:21.198155+0000","last_unstale":"2024-09-16T15:46:22.662841+0000","last_undegraded":"2024-09-16T15:46:22.662841+0000","last_fullsized":"2024-09-16T15:46:22.662841+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:20.126241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:20.126241+0000","last_clean_scrub_stamp":"2024-09-16T15:46:20.126241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T02:48:46.778283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.25","version":"0'0","reported_seq":13,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-16T15:46:22.666213+0000","last_change":"2024-09-16T15:46:21.177637+0000","last_active":"2024-09-16T15:46:22.666213+0000","last_peered":"2024-09-16T15:46:22.666213+0000","last_clean":"2024-09-16T15:46:22.666213+0000","last_became_active":"2024-09-16T15:46:21.177367+0000","last_became_peered":"2024-09-16T15:46:21.177367+0000","last_unstale":"2024-09-16T15:46:22.666213+0000","last_undegraded":"2024-09-16T15:46:22.666213+0000","last_fullsized":"2024-09-16T15:46:22.666213+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:20.126241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:20.126241+0000","last_clean_scrub_stamp":"2024-09-16T15:46:20.126241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:04:16.851627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.24","version":"0'0","reported_seq":13,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-16T15:46:22.664357+0000","last_change":"2024-09-16T15:46:21.181008+0000","last_active":"2024-09-16T15:46:22.664357+0000","last_peered":"2024-09-16T15:46:22.664357+0000","last_clean":"2024-09-16T15:46:22.664357+0000","last_became_active":"2024-09-16T15:46:21.180757+0000","last_became_peered":"2024-09-16T15:46:21.180757+0000","last_unstale":"2024-09-16T15:46:22.664357+0000","last_undegraded":"2024-09-16T15:46:22.664357+0000","last_fullsized":"2024-09-16T15:46:22.664357+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:20.126241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:20.126241+0000","last_clean_scrub_stamp":"2024-09-16T15:46:20.126241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T22:41:13.974671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.23","version":"0'0","reported_seq":13,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-16T15:46:22.665323+0000","last_change":"2024-09-16T15:46:21.186868+0000","last_active":"2024-09-16T15:46:22.665323+0000","last_peered":"2024-09-16T15:46:22.665323+0000","last_clean":"2024-09-16T15:46:22.665323+0000","last_became_active":"2024-09-16T15:46:21.186578+0000","last_became_peered":"2024-09-16T15:46:21.186578+0000","last_unstale":"2024-09-16T15:46:22.665323+0000","last_undegraded":"2024-09-16T15:46:22.665323+0000","last_fullsized":"2024-09-16T15:46:22.665323+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:20.126241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:20.126241+0000","last_clean_scrub_stamp":"2024-09-16T15:46:20.126241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:40:54.364802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.22","version":"0'0","reported_seq":13,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-16T15:46:22.150475+0000","last_change":"2024-09-16T15:46:21.194681+0000","last_active":"2024-09-16T15:46:22.150475+0000","last_peered":"2024-09-16T15:46:22.150475+0000","last_clean":"2024-09-16T15:46:22.150475+0000","last_became_active":"2024-09-16T15:46:21.193938+0000","last_became_peered":"2024-09-16T15:46:21.193938+0000","last_unstale":"2024-09-16T15:46:22.150475+0000","last_undegraded":"2024-09-16T15:46:22.150475+0000","last_fullsized":"2024-09-16T15:46:22.150475+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:20.126241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:20.126241+0000","last_clean_scrub_stamp":"2024-09-16T15:46:20.126241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T02:30:25.147666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.21","version":"0'0","reported_seq":13,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-16T15:46:22.662394+0000","last_change":"2024-09-16T15:46:21.165474+0000","last_active":"2024-09-16T15:46:22.662394+0000","last_peered":"2024-09-16T15:46:22.662394+0000","last_clean":"2024-09-16T15:46:22.662394+0000","last_became_active":"2024-09-16T15:46:21.165198+0000","last_became_peered":"2024-09-16T15:46:21.165198+0000","last_unstale":"2024-09-16T15:46:22.662394+0000","last_undegraded":"2024-09-16T15:46:22.662394+0000","last_fullsized":"2024-09-16T15:46:22.662394+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:20.126241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:20.126241+0000","last_clean_scrub_stamp":"2024-09-16T15:46:20.126241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T03:30:20.497891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.20","version":"0'0","reported_seq":13,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-16T15:46:22.663963+0000","last_change":"2024-09-16T15:46:21.161635+0000","last_active":"2024-09-16T15:46:22.663963+0000","last_peered":"2024-09-16T15:46:22.663963+0000","last_clean":"2024-09-16T15:46:22.663963+0000","last_became_active":"2024-09-16T15:46:21.161375+0000","last_became_peered":"2024-09-16T15:46:21.161375+0000","last_unstale":"2024-09-16T15:46:22.663963+0000","last_undegraded":"2024-09-16T15:46:22.663963+0000","last_fullsized":"2024-09-16T15:46:22.663963+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:20.126241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:20.126241+0000","last_clean_scrub_stamp":"2024-09-16T15:46:20.126241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:02:08.119880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1e","version":"0'0","reported_seq":13,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-16T15:46:22.666224+0000","last_change":"2024-09-16T15:46:21.169712+0000","last_active":"2024-09-16T15:46:22.666224+0000","last_peered":"2024-09-16T15:46:22.666224+0000","last_clean":"2024-09-16T15:46:22.666224+0000","last_became_active":"2024-09-16T15:46:21.169475+0000","last_became_peered":"2024-09-16T15:46:21.169475+0000","last_unstale":"2024-09-16T15:46:22.666224+0000","last_undegraded":"2024-09-16T15:46:22.666224+0000","last_fullsized":"2024-09-16T15:46:22.666224+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:20.126241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:20.126241+0000","last_clean_scrub_stamp":"2024-09-16T15:46:20.126241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T18:43:19.978239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1f","version":"57'4","reported_seq":19,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-16T15:46:22.665576+0000","last_change":"2024-09-16T15:46:20.166540+0000","last_active":"2024-09-16T15:46:22.665576+0000","last_peered":"2024-09-16T15:46:22.665576+0000","last_clean":"2024-09-16T15:46:22.665576+0000","last_became_active":"2024-09-16T15:46:20.166174+0000","last_became_peered":"2024-09-16T15:46:20.166174+0000","last_unstale":"2024-09-16T15:46:22.665576+0000","last_undegraded":"2024-09-16T15:46:22.665576+0000","last_fullsized":"2024-09-16T15:46:22.665576+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:19.122499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:19.122499+0000","last_clean_scrub_stamp":"2024-09-16T15:46:19.122499+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T01:00:38.654626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1f","version":"0'0","reported_seq":13,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-16T15:46:22.146667+0000","last_change":"2024-09-16T15:46:21.168697+0000","last_active":"2024-09-16T15:46:22.146667+0000","last_peered":"2024-09-16T15:46:22.146667+0000","last_clean":"2024-09-16T15:46:22.146667+0000","last_became_active":"2024-09-16T15:46:21.168423+0000","last_became_peered":"2024-09-16T15:46:21.168423+0000","last_unstale":"2024-09-16T15:46:22.146667+0000","last_undegraded":"2024-09-16T15:46:22.146667+0000","last_fullsized":"2024-09-16T15:46:22.146667+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:20.126241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:20.126241+0000","last_clean_scrub_stamp":"2024-09-16T15:46:20.126241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T01:52:56.318281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.1e","version":"57'1","reported_seq":16,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-16T15:46:22.665160+0000","last_change":"2024-09-16T15:46:20.173570+0000","last_active":"2024-09-16T15:46:22.665160+0000","last_peered":"2024-09-16T15:46:22.665160+0000","last_clean":"2024-09-16T15:46:22.665160+0000","last_became_active":"2024-09-16T15:46:20.173087+0000","last_became_peered":"2024-09-16T15:46:20.173087+0000","last_unstale":"2024-09-16T15:46:22.665160+0000","last_undegraded":"2024-09-16T15:46:22.665160+0000","last_fullsized":"2024-09-16T15:46:22.665160+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:19.122499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:19.122499+0000","last_clean_scrub_stamp":"2024-09-16T15:46:19.122499+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:45:44.517948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.1c","version":"0'0","reported_seq":13,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-16T15:46:22.665043+0000","last_change":"2024-09-16T15:46:21.163880+0000","last_active":"2024-09-16T15:46:22.665043+0000","last_peered":"2024-09-16T15:46:22.665043+0000","last_clean":"2024-09-16T15:46:22.665043+0000","last_became_active":"2024-09-16T15:46:21.163599+0000","last_became_peered":"2024-09-16T15:46:21.163599+0000","last_unstale":"2024-09-16T15:46:22.665043+0000","last_undegraded":"2024-09-16T15:46:22.665043+0000","last_fullsized":"2024-09-16T15:46:22.665043+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:20.126241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:20.126241+0000","last_clean_scrub_stamp":"2024-09-16T15:46:20.126241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:45:05.277490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1d","version":"57'2","reported_seq":17,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-16T15:46:22.666435+0000","last_change":"2024-09-16T15:46:20.165693+0000","last_active":"2024-09-16T15:46:22.666435+0000","last_peered":"2024-09-16T15:46:22.666435+0000","last_clean":"2024-09-16T15:46:22.666435+0000","last_became_active":"2024-09-16T15:46:20.165450+0000","last_became_peered":"2024-09-16T15:46:20.165450+0000","last_unstale":"2024-09-16T15:46:22.666435+0000","last_undegraded":"2024-09-16T15:46:22.666435+0000","last_fullsized":"2024-09-16T15:46:22.666435+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:19.122499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:19.122499+0000","last_clean_scrub_stamp":"2024-09-16T15:46:19.122499+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:57:48.935049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1d","version":"0'0","reported_seq":13,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-16T15:46:22.666243+0000","last_change":"2024-09-16T15:46:21.186815+0000","last_active":"2024-09-16T15:46:22.666243+0000","last_peered":"2024-09-16T15:46:22.666243+0000","last_clean":"2024-09-16T15:46:22.666243+0000","last_became_active":"2024-09-16T15:46:21.186510+0000","last_became_peered":"2024-09-16T15:46:21.186510+0000","last_unstale":"2024-09-16T15:46:22.666243+0000","last_undegraded":"2024-09-16T15:46:22.666243+0000","last_fullsized":"2024-09-16T15:46:22.666243+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:20.126241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:20.126241+0000","last_clean_scrub_stamp":"2024-09-16T15:46:20.126241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T02:03:17.281882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1c","version":"57'2","reported_seq":17,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-16T15:46:22.662702+0000","last_change":"2024-09-16T15:46:20.177385+0000","last_active":"2024-09-16T15:46:22.662702+0000","last_peered":"2024-09-16T15:46:22.662702+0000","last_clean":"2024-09-16T15:46:22.662702+0000","last_became_active":"2024-09-16T15:46:20.176639+0000","last_became_peered":"2024-09-16T15:46:20.176639+0000","last_unstale":"2024-09-16T15:46:22.662702+0000","last_undegraded":"2024-09-16T15:46:22.662702+0000","last_fullsized":"2024-09-16T15:46:22.662702+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:19.122499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:19.122499+0000","last_clean_scrub_stamp":"2024-09-16T15:46:19.122499+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T02:08:40.864048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.1a","version":"0'0","reported_seq":13,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-16T15:46:22.666143+0000","last_change":"2024-09-16T15:46:21.156283+0000","last_active":"2024-09-16T15:46:22.666143+0000","last_peered":"2024-09-16T15:46:22.666143+0000","last_clean":"2024-09-16T15:46:22.666143+0000","last_became_active":"2024-09-16T15:46:21.155996+0000","last_became_peered":"2024-09-16T15:46:21.155996+0000","last_unstale":"2024-09-16T15:46:22.666143+0000","last_undegraded":"2024-09-16T15:46:22.666143+0000","last_fullsized":"2024-09-16T15:46:22.666143+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:20.126241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:20.126241+0000","last_clean_scrub_stamp":"2024-09-16T15:46:20.126241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T22:14:11.307873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1b","version":"57'1","reported_seq":16,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-16T15:46:22.152104+0000","last_change":"2024-09-16T15:46:20.170745+0000","last_active":"2024-09-16T15:46:22.152104+0000","last_peered":"2024-09-16T15:46:22.152104+0000","last_clean":"2024-09-16T15:46:22.152104+0000","last_became_active":"2024-09-16T15:46:20.170035+0000","last_became_peered":"2024-09-16T15:46:20.170035+0000","last_unstale":"2024-09-16T15:46:22.152104+0000","last_undegraded":"2024-09-16T15:46:22.152104+0000","last_fullsized":"2024-09-16T15:46:22.152104+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:19.122499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:19.122499+0000","last_clean_scrub_stamp":"2024-09-16T15:46:19.122499+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:16:04.639067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.1b","version":"0'0","reported_seq":13,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-16T15:46:22.665004+0000","last_change":"2024-09-16T15:46:21.157399+0000","last_active":"2024-09-16T15:46:22.665004+0000","last_peered":"2024-09-16T15:46:22.665004+0000","last_clean":"2024-09-16T15:46:22.665004+0000","last_became_active":"2024-09-16T15:46:21.157099+0000","last_became_peered":"2024-09-16T15:46:21.157099+0000","last_unstale":"2024-09-16T15:46:22.665004+0000","last_undegraded":"2024-09-16T15:46:22.665004+0000","last_fullsized":"2024-09-16T15:46:22.665004+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:20.126241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:20.126241+0000","last_clean_scrub_stamp":"2024-09-16T15:46:20.126241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T22:06:15.943463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1a","version":"57'1","reported_seq":16,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-16T15:46:22.664912+0000","last_change":"2024-09-16T15:46:20.163808+0000","last_active":"2024-09-16T15:46:22.664912+0000","last_peered":"2024-09-16T15:46:22.664912+0000","last_clean":"2024-09-16T15:46:22.664912+0000","last_became_active":"2024-09-16T15:46:20.163537+0000","last_became_peered":"2024-09-16T15:46:20.163537+0000","last_unstale":"2024-09-16T15:46:22.664912+0000","last_undegraded":"2024-09-16T15:46:22.664912+0000","last_fullsized":"2024-09-16T15:46:22.664912+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:19.122499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:19.122499+0000","last_clean_scrub_stamp":"2024-09-16T15:46:19.122499+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:31:11.780154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.18","version":"0'0","reported_seq":13,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-16T15:46:22.150484+0000","last_change":"2024-09-16T15:46:21.190438+0000","last_active":"2024-09-16T15:46:22.150484+0000","last_peered":"2024-09-16T15:46:22.150484+0000","last_clean":"2024-09-16T15:46:22.150484+0000","last_became_active":"2024-09-16T15:46:21.189789+0000","last_became_peered":"2024-09-16T15:46:21.189789+0000","last_unstale":"2024-09-16T15:46:22.150484+0000","last_undegraded":"2024-09-16T15:46:22.150484+0000","last_fullsized":"2024-09-16T15:46:22.150484+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:20.126241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:20.126241+0000","last_clean_scrub_stamp":"2024-09-16T15:46:20.126241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T22:23:55.468615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.19","version":"0'0","reported_seq":15,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-16T15:46:22.661633+0000","last_change":"2024-09-16T15:46:20.179167+0000","last_active":"2024-09-16T15:46:22.661633+0000","last_peered":"2024-09-16T15:46:22.661633+0000","last_clean":"2024-09-16T15:46:22.661633+0000","last_became_active":"2024-09-16T15:46:20.178523+0000","last_became_peered":"2024-09-16T15:46:20.178523+0000","last_unstale":"2024-09-16T15:46:22.661633+0000","last_undegraded":"2024-09-16T15:46:22.661633+0000","last_fullsized":"2024-09-16T15:46:22.661633+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:19.122499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:19.122499+0000","last_clean_scrub_stamp":"2024-09-16T15:46:19.122499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T01:00:38.654626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.19","version":"0'0","reported_seq":13,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-16T15:46:22.663826+0000","last_change":"2024-09-16T15:46:21.167226+0000","last_active":"2024-09-16T15:46:22.663826+0000","last_peered":"2024-09-16T15:46:22.663826+0000","last_clean":"2024-09-16T15:46:22.663826+0000","last_became_active":"2024-09-16T15:46:21.166941+0000","last_became_peered":"2024-09-16T15:46:21.166941+0000","last_unstale":"2024-09-16T15:46:22.663826+0000","last_undegraded":"2024-09-16T15:46:22.663826+0000","last_fullsized":"2024-09-16T15:46:22.663826+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:20.126241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:20.126241+0000","last_clean_scrub_stamp":"2024-09-16T15:46:20.126241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:35:15.637361+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.18","version":"0'0","reported_seq":15,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-16T15:46:22.150295+0000","last_change":"2024-09-16T15:46:20.168940+0000","last_active":"2024-09-16T15:46:22.150295+0000","last_peered":"2024-09-16T15:46:22.150295+0000","last_clean":"2024-09-16T15:46:22.150295+0000","last_became_active":"2024-09-16T15:46:20.168009+0000","last_became_peered":"2024-09-16T15:46:20.168009+0000","last_unstale":"2024-09-16T15:46:22.150295+0000","last_undegraded":"2024-09-16T15:46:22.150295+0000","last_fullsized":"2024-09-16T15:46:22.150295+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:19.122499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:19.122499+0000","last_clean_scrub_stamp":"2024-09-16T15:46:19.122499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T01:07:24.376384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.16","version":"0'0","reported_seq":13,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-16T15:46:22.152529+0000","last_change":"2024-09-16T15:46:21.167649+0000","last_active":"2024-09-16T15:46:22.152529+0000","last_peered":"2024-09-16T15:46:22.152529+0000","last_clean":"2024-09-16T15:46:22.152529+0000","last_became_active":"2024-09-16T15:46:21.166944+0000","last_became_peered":"2024-09-16T15:46:21.166944+0000","last_unstale":"2024-09-16T15:46:22.152529+0000","last_undegraded":"2024-09-16T15:46:22.152529+0000","last_fullsized":"2024-09-16T15:46:22.152529+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:20.126241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:20.126241+0000","last_clean_scrub_stamp":"2024-09-16T15:46:20.126241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:44:33.231565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.17","version":"0'0","reported_seq":15,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-16T15:46:22.154621+0000","last_change":"2024-09-16T15:46:20.177276+0000","last_active":"2024-09-16T15:46:22.154621+0000","last_peered":"2024-09-16T15:46:22.154621+0000","last_clean":"2024-09-16T15:46:22.154621+0000","last_became_active":"2024-09-16T15:46:20.176556+0000","last_became_peered":"2024-09-16T15:46:20.176556+0000","last_unstale":"2024-09-16T15:46:22.154621+0000","last_undegraded":"2024-09-16T15:46:22.154621+0000","last_fullsized":"2024-09-16T15:46:22.154621+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:19.122499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:19.122499+0000","last_clean_scrub_stamp":"2024-09-16T15:46:19.122499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T02:08:40.864048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.17","version":"0'0","reported_seq":13,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-16T15:46:22.155274+0000","last_change":"2024-09-16T15:46:21.194067+0000","last_active":"2024-09-16T15:46:22.155274+0000","last_peered":"2024-09-16T15:46:22.155274+0000","last_clean":"2024-09-16T15:46:22.155274+0000","last_became_active":"2024-09-16T15:46:21.193450+0000","last_became_peered":"2024-09-16T15:46:21.193450+0000","last_unstale":"2024-09-16T15:46:22.155274+0000","last_undegraded":"2024-09-16T15:46:22.155274+0000","last_fullsized":"2024-09-16T15:46:22.155274+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:20.126241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:20.126241+0000","last_clean_scrub_stamp":"2024-09-16T15:46:20.126241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T22:04:25.863794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.16","version":"0'0","reported_seq":15,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-16T15:46:22.665459+0000","last_change":"2024-09-16T15:46:20.167134+0000","last_active":"2024-09-16T15:46:22.665459+0000","last_peered":"2024-09-16T15:46:22.665459+0000","last_clean":"2024-09-16T15:46:22.665459+0000","last_became_active":"2024-09-16T15:46:20.166782+0000","last_became_peered":"2024-09-16T15:46:20.166782+0000","last_unstale":"2024-09-16T15:46:22.665459+0000","last_undegraded":"2024-09-16T15:46:22.665459+0000","last_fullsized":"2024-09-16T15:46:22.665459+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:19.122499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:19.122499+0000","last_clean_scrub_stamp":"2024-09-16T15:46:19.122499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T00:57:14.010694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.14","version":"0'0","reported_seq":13,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-16T15:46:22.146260+0000","last_change":"2024-09-16T15:46:21.188473+0000","last_active":"2024-09-16T15:46:22.146260+0000","last_peered":"2024-09-16T15:46:22.146260+0000","last_clean":"2024-09-16T15:46:22.146260+0000","last_became_active":"2024-09-16T15:46:21.188158+0000","last_became_peered":"2024-09-16T15:46:21.188158+0000","last_unstale":"2024-09-16T15:46:22.146260+0000","last_undegraded":"2024-09-16T15:46:22.146260+0000","last_fullsized":"2024-09-16T15:46:22.146260+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:20.126241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:20.126241+0000","last_clean_scrub_stamp":"2024-09-16T15:46:20.126241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T02:15:23.142882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.15","version":"57'3","reported_seq":18,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-16T15:46:22.155019+0000","last_change":"2024-09-16T15:46:20.183097+0000","last_active":"2024-09-16T15:46:22.155019+0000","last_peered":"2024-09-16T15:46:22.155019+0000","last_clean":"2024-09-16T15:46:22.155019+0000","last_became_active":"2024-09-16T15:46:20.182459+0000","last_became_peered":"2024-09-16T15:46:20.182459+0000","last_unstale":"2024-09-16T15:46:22.155019+0000","last_undegraded":"2024-09-16T15:46:22.155019+0000","last_fullsized":"2024-09-16T15:46:22.155019+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:19.122499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:19.122499+0000","last_clean_scrub_stamp":"2024-09-16T15:46:19.122499+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T02:34:10.362794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.15","version":"0'0","reported_seq":13,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-16T15:46:22.154685+0000","last_change":"2024-09-16T15:46:21.201304+0000","last_active":"2024-09-16T15:46:22.154685+0000","last_peered":"2024-09-16T15:46:22.154685+0000","last_clean":"2024-09-16T15:46:22.154685+0000","last_became_active":"2024-09-16T15:46:21.200098+0000","last_became_peered":"2024-09-16T15:46:21.200098+0000","last_unstale":"2024-09-16T15:46:22.154685+0000","last_undegraded":"2024-09-16T15:46:22.154685+0000","last_fullsized":"2024-09-16T15:46:22.154685+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:20.126241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:20.126241+0000","last_clean_scrub_stamp":"2024-09-16T15:46:20.126241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T01:59:02.801025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.14","version":"57'6","reported_seq":21,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-16T15:46:22.152650+0000","last_change":"2024-09-16T15:46:20.169021+0000","last_active":"2024-09-16T15:46:22.152650+0000","last_peered":"2024-09-16T15:46:22.152650+0000","last_clean":"2024-09-16T15:46:22.152650+0000","last_became_active":"2024-09-16T15:46:20.168286+0000","last_became_peered":"2024-09-16T15:46:20.168286+0000","last_unstale":"2024-09-16T15:46:22.152650+0000","last_undegraded":"2024-09-16T15:46:22.152650+0000","last_fullsized":"2024-09-16T15:46:22.152650+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:19.122499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:19.122499+0000","last_clean_scrub_stamp":"2024-09-16T15:46:19.122499+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:59:30.864433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.12","version":"0'0","reported_seq":13,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-16T15:46:22.662004+0000","last_change":"2024-09-16T15:46:21.196486+0000","last_active":"2024-09-16T15:46:22.662004+0000","last_peered":"2024-09-16T15:46:22.662004+0000","last_clean":"2024-09-16T15:46:22.662004+0000","last_became_active":"2024-09-16T15:46:21.195784+0000","last_became_peered":"2024-09-16T15:46:21.195784+0000","last_unstale":"2024-09-16T15:46:22.662004+0000","last_undegraded":"2024-09-16T15:46:22.662004+0000","last_fullsized":"2024-09-16T15:46:22.662004+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:20.126241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:20.126241+0000","last_clean_scrub_stamp":"2024-09-16T15:46:20.126241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:20:45.355620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.13","version":"57'2","reported_seq":17,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-16T15:46:22.664638+0000","last_change":"2024-09-16T15:46:20.165007+0000","last_active":"2024-09-16T15:46:22.664638+0000","last_peered":"2024-09-16T15:46:22.664638+0000","last_clean":"2024-09-16T15:46:22.664638+0000","last_became_active":"2024-09-16T15:46:20.164615+0000","last_became_peered":"2024-09-16T15:46:20.164615+0000","last_unstale":"2024-09-16T15:46:22.664638+0000","last_undegraded":"2024-09-16T15:46:22.664638+0000","last_fullsized":"2024-09-16T15:46:22.664638+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:19.122499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:19.122499+0000","last_clean_scrub_stamp":"2024-09-16T15:46:19.122499+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:28:49.947491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.13","version":"0'0","reported_seq":13,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-16T15:46:22.154239+0000","last_change":"2024-09-16T15:46:21.202708+0000","last_active":"2024-09-16T15:46:22.154239+0000","last_peered":"2024-09-16T15:46:22.154239+0000","last_clean":"2024-09-16T15:46:22.154239+0000","last_became_active":"2024-09-16T15:46:21.201956+0000","last_became_peered":"2024-09-16T15:46:21.201956+0000","last_unstale":"2024-09-16T15:46:22.154239+0000","last_undegraded":"2024-09-16T15:46:22.154239+0000","last_fullsized":"2024-09-16T15:46:22.154239+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:20.126241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:20.126241+0000","last_clean_scrub_stamp":"2024-09-16T15:46:20.126241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:16:25.611352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.12","version":"0'0","reported_seq":15,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-16T15:46:22.150716+0000","last_change":"2024-09-16T15:46:20.174615+0000","last_active":"2024-09-16T15:46:22.150716+0000","last_peered":"2024-09-16T15:46:22.150716+0000","last_clean":"2024-09-16T15:46:22.150716+0000","last_became_active":"2024-09-16T15:46:20.173929+0000","last_became_peered":"2024-09-16T15:46:20.173929+0000","last_unstale":"2024-09-16T15:46:22.150716+0000","last_undegraded":"2024-09-16T15:46:22.150716+0000","last_fullsized":"2024-09-16T15:46:22.150716+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:19.122499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:19.122499+0000","last_clean_scrub_stamp":"2024-09-16T15:46:19.122499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:24:33.895442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.10","version":"0'0","reported_seq":13,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-16T15:46:22.150023+0000","last_change":"2024-09-16T15:46:21.183360+0000","last_active":"2024-09-16T15:46:22.150023+0000","last_peered":"2024-09-16T15:46:22.150023+0000","last_clean":"2024-09-16T15:46:22.150023+0000","last_became_active":"2024-09-16T15:46:21.181481+0000","last_became_peered":"2024-09-16T15:46:21.181481+0000","last_unstale":"2024-09-16T15:46:22.150023+0000","last_undegraded":"2024-09-16T15:46:22.150023+0000","last_fullsized":"2024-09-16T15:46:22.150023+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:20.126241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:20.126241+0000","last_clean_scrub_stamp":"2024-09-16T15:46:20.126241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:23:37.624787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.11","version":"0'0","reported_seq":15,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-16T15:46:22.661641+0000","last_change":"2024-09-16T15:46:20.177564+0000","last_active":"2024-09-16T15:46:22.661641+0000","last_peered":"2024-09-16T15:46:22.661641+0000","last_clean":"2024-09-16T15:46:22.661641+0000","last_became_active":"2024-09-16T15:46:20.176740+0000","last_became_peered":"2024-09-16T15:46:20.176740+0000","last_unstale":"2024-09-16T15:46:22.661641+0000","last_undegraded":"2024-09-16T15:46:22.661641+0000","last_fullsized":"2024-09-16T15:46:22.661641+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:19.122499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:19.122499+0000","last_clean_scrub_stamp":"2024-09-16T15:46:19.122499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:57:48.935049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.11","version":"0'0","reported_seq":13,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-16T15:46:22.150873+0000","last_change":"2024-09-16T15:46:21.178824+0000","last_active":"2024-09-16T15:46:22.150873+0000","last_peered":"2024-09-16T15:46:22.150873+0000","last_clean":"2024-09-16T15:46:22.150873+0000","last_became_active":"2024-09-16T15:46:21.176997+0000","last_became_peered":"2024-09-16T15:46:21.176997+0000","last_unstale":"2024-09-16T15:46:22.150873+0000","last_undegraded":"2024-09-16T15:46:22.150873+0000","last_fullsized":"2024-09-16T15:46:22.150873+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:20.126241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:20.126241+0000","last_clean_scrub_stamp":"2024-09-16T15:46:20.126241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T02:48:46.778283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.10","version":"57'8","reported_seq":23,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-16T15:46:22.150244+0000","last_change":"2024-09-16T15:46:20.178911+0000","last_active":"2024-09-16T15:46:22.150244+0000","last_peered":"2024-09-16T15:46:22.150244+0000","last_clean":"2024-09-16T15:46:22.150244+0000","last_became_active":"2024-09-16T15:46:20.178208+0000","last_became_peered":"2024-09-16T15:46:20.178208+0000","last_unstale":"2024-09-16T15:46:22.150244+0000","last_undegraded":"2024-09-16T15:46:22.150244+0000","last_fullsized":"2024-09-16T15:46:22.150244+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:19.122499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:19.122499+0000","last_clean_scrub_stamp":"2024-09-16T15:46:19.122499+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T01:00:38.654626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.e","version":"0'0","reported_seq":13,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-16T15:46:22.155466+0000","last_change":"2024-09-16T15:46:21.196972+0000","last_active":"2024-09-16T15:46:22.155466+0000","last_peered":"2024-09-16T15:46:22.155466+0000","last_clean":"2024-09-16T15:46:22.155466+0000","last_became_active":"2024-09-16T15:46:21.195963+0000","last_became_peered":"2024-09-16T15:46:21.195963+0000","last_unstale":"2024-09-16T15:46:22.155466+0000","last_undegraded":"2024-09-16T15:46:22.155466+0000","last_fullsized":"2024-09-16T15:46:22.155466+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:20.126241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:20.126241+0000","last_clean_scrub_stamp":"2024-09-16T15:46:20.126241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:42:02.964189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.f","version":"57'1","reported_seq":16,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-16T15:46:22.665910+0000","last_change":"2024-09-16T15:46:20.169229+0000","last_active":"2024-09-16T15:46:22.665910+0000","last_peered":"2024-09-16T15:46:22.665910+0000","last_clean":"2024-09-16T15:46:22.665910+0000","last_became_active":"2024-09-16T15:46:20.168849+0000","last_became_peered":"2024-09-16T15:46:20.168849+0000","last_unstale":"2024-09-16T15:46:22.665910+0000","last_undegraded":"2024-09-16T15:46:22.665910+0000","last_fullsized":"2024-09-16T15:46:22.665910+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:19.122499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:19.122499+0000","last_clean_scrub_stamp":"2024-09-16T15:46:19.122499+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T03:01:41.067722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.f","version":"0'0","reported_seq":13,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-16T15:46:22.662823+0000","last_change":"2024-09-16T15:46:21.196313+0000","last_active":"2024-09-16T15:46:22.662823+0000","last_peered":"2024-09-16T15:46:22.662823+0000","last_clean":"2024-09-16T15:46:22.662823+0000","last_became_active":"2024-09-16T15:46:21.195676+0000","last_became_peered":"2024-09-16T15:46:21.195676+0000","last_unstale":"2024-09-16T15:46:22.662823+0000","last_undegraded":"2024-09-16T15:46:22.662823+0000","last_fullsized":"2024-09-16T15:46:22.662823+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:20.126241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:20.126241+0000","last_clean_scrub_stamp":"2024-09-16T15:46:20.126241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:06:33.559242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.e","version":"57'2","reported_seq":17,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-16T15:46:22.666329+0000","last_change":"2024-09-16T15:46:20.169741+0000","last_active":"2024-09-16T15:46:22.666329+0000","last_peered":"2024-09-16T15:46:22.666329+0000","last_clean":"2024-09-16T15:46:22.666329+0000","last_became_active":"2024-09-16T15:46:20.168807+0000","last_became_peered":"2024-09-16T15:46:20.168807+0000","last_unstale":"2024-09-16T15:46:22.666329+0000","last_undegraded":"2024-09-16T15:46:22.666329+0000","last_fullsized":"2024-09-16T15:46:22.666329+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:19.122499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:19.122499+0000","last_clean_scrub_stamp":"2024-09-16T15:46:19.122499+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T18:40:39.863914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.c","version":"0'0","reported_seq":13,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-16T15:46:22.665649+0000","last_change":"2024-09-16T15:46:21.188376+0000","last_active":"2024-09-16T15:46:22.665649+0000","last_peered":"2024-09-16T15:46:22.665649+0000","last_clean":"2024-09-16T15:46:22.665649+0000","last_became_active":"2024-09-16T15:46:21.188115+0000","last_became_peered":"2024-09-16T15:46:21.188115+0000","last_unstale":"2024-09-16T15:46:22.665649+0000","last_undegraded":"2024-09-16T15:46:22.665649+0000","last_fullsized":"2024-09-16T15:46:22.665649+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:20.126241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:20.126241+0000","last_clean_scrub_stamp":"2024-09-16T15:46:20.126241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:32:53.156685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.d","version":"57'1","reported_seq":16,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-16T15:46:22.662444+0000","last_change":"2024-09-16T15:46:20.177143+0000","last_active":"2024-09-16T15:46:22.662444+0000","last_peered":"2024-09-16T15:46:22.662444+0000","last_clean":"2024-09-16T15:46:22.662444+0000","last_became_active":"2024-09-16T15:46:20.176500+0000","last_became_peered":"2024-09-16T15:46:20.176500+0000","last_unstale":"2024-09-16T15:46:22.662444+0000","last_undegraded":"2024-09-16T15:46:22.662444+0000","last_fullsized":"2024-09-16T15:46:22.662444+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:19.122499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:19.122499+0000","last_clean_scrub_stamp":"2024-09-16T15:46:19.122499+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T18:50:44.561861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.d","version":"0'0","reported_seq":13,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-16T15:46:22.155403+0000","last_change":"2024-09-16T15:46:21.205417+0000","last_active":"2024-09-16T15:46:22.155403+0000","last_peered":"2024-09-16T15:46:22.155403+0000","last_clean":"2024-09-16T15:46:22.155403+0000","last_became_active":"2024-09-16T15:46:21.200693+0000","last_became_peered":"2024-09-16T15:46:21.200693+0000","last_unstale":"2024-09-16T15:46:22.155403+0000","last_undegraded":"2024-09-16T15:46:22.155403+0000","last_fullsized":"2024-09-16T15:46:22.155403+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:20.126241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:20.126241+0000","last_clean_scrub_stamp":"2024-09-16T15:46:20.126241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:06:49.089844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.c","version":"0'0","reported_seq":15,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-16T15:46:22.665203+0000","last_change":"2024-09-16T15:46:20.159441+0000","last_active":"2024-09-16T15:46:22.665203+0000","last_peered":"2024-09-16T15:46:22.665203+0000","last_clean":"2024-09-16T15:46:22.665203+0000","last_became_active":"2024-09-16T15:46:20.159041+0000","last_became_peered":"2024-09-16T15:46:20.159041+0000","last_unstale":"2024-09-16T15:46:22.665203+0000","last_undegraded":"2024-09-16T15:46:22.665203+0000","last_fullsized":"2024-09-16T15:46:22.665203+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:19.122499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:19.122499+0000","last_clean_scrub_stamp":"2024-09-16T15:46:19.122499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:54:25.988820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.a","version":"0'0","reported_seq":13,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-16T15:46:22.662058+0000","last_change":"2024-09-16T15:46:21.194389+0000","last_active":"2024-09-16T15:46:22.662058+0000","last_peered":"2024-09-16T15:46:22.662058+0000","last_clean":"2024-09-16T15:46:22.662058+0000","last_became_active":"2024-09-16T15:46:21.193684+0000","last_became_peered":"2024-09-16T15:46:21.193684+0000","last_unstale":"2024-09-16T15:46:22.662058+0000","last_undegraded":"2024-09-16T15:46:22.662058+0000","last_fullsized":"2024-09-16T15:46:22.662058+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:20.126241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:20.126241+0000","last_clean_scrub_stamp":"2024-09-16T15:46:20.126241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:47:48.587754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.b","version":"57'2","reported_seq":17,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-16T15:46:22.665498+0000","last_change":"2024-09-16T15:46:20.176249+0000","last_active":"2024-09-16T15:46:22.665498+0000","last_peered":"2024-09-16T15:46:22.665498+0000","last_clean":"2024-09-16T15:46:22.665498+0000","last_became_active":"2024-09-16T15:46:20.175919+0000","last_became_peered":"2024-09-16T15:46:20.175919+0000","last_unstale":"2024-09-16T15:46:22.665498+0000","last_undegraded":"2024-09-16T15:46:22.665498+0000","last_fullsized":"2024-09-16T15:46:22.665498+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:19.122499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:19.122499+0000","last_clean_scrub_stamp":"2024-09-16T15:46:19.122499+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:34:21.204574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.b","version":"0'0","reported_seq":13,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-16T15:46:22.665233+0000","last_change":"2024-09-16T15:46:21.162542+0000","last_active":"2024-09-16T15:46:22.665233+0000","last_peered":"2024-09-16T15:46:22.665233+0000","last_clean":"2024-09-16T15:46:22.665233+0000","last_became_active":"2024-09-16T15:46:21.162102+0000","last_became_peered":"2024-09-16T15:46:21.162102+0000","last_unstale":"2024-09-16T15:46:22.665233+0000","last_undegraded":"2024-09-16T15:46:22.665233+0000","last_fullsized":"2024-09-16T15:46:22.665233+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:20.126241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:20.126241+0000","last_clean_scrub_stamp":"2024-09-16T15:46:20.126241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:40:25.014018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.a","version":"0'0","reported_seq":15,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-16T15:46:22.145395+0000","last_change":"2024-09-16T15:46:20.169259+0000","last_active":"2024-09-16T15:46:22.145395+0000","last_peered":"2024-09-16T15:46:22.145395+0000","last_clean":"2024-09-16T15:46:22.145395+0000","last_became_active":"2024-09-16T15:46:20.168879+0000","last_became_peered":"2024-09-16T15:46:20.168879+0000","last_unstale":"2024-09-16T15:46:22.145395+0000","last_undegraded":"2024-09-16T15:46:22.145395+0000","last_fullsized":"2024-09-16T15:46:22.145395+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:19.122499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:19.122499+0000","last_clean_scrub_stamp":"2024-09-16T15:46:19.122499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T23:05:35.126527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.9","version":"0'0","reported_seq":13,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-16T15:46:22.664622+0000","last_change":"2024-09-16T15:46:21.168450+0000","last_active":"2024-09-16T15:46:22.664622+0000","last_peered":"2024-09-16T15:46:22.664622+0000","last_clean":"2024-09-16T15:46:22.664622+0000","last_became_active":"2024-09-16T15:46:21.168028+0000","last_became_peered":"2024-09-16T15:46:21.168028+0000","last_unstale":"2024-09-16T15:46:22.664622+0000","last_undegraded":"2024-09-16T15:46:22.664622+0000","last_fullsized":"2024-09-16T15:46:22.664622+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:20.126241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:20.126241+0000","last_clean_scrub_stamp":"2024-09-16T15:46:20.126241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:41:54.960196+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.8","version":"0'0","reported_seq":15,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-16T15:46:22.661449+0000","last_change":"2024-09-16T15:46:20.166278+0000","last_active":"2024-09-16T15:46:22.661449+0000","last_peered":"2024-09-16T15:46:22.661449+0000","last_clean":"2024-09-16T15:46:22.661449+0000","last_became_active":"2024-09-16T15:46:20.165547+0000","last_became_peered":"2024-09-16T15:46:20.165547+0000","last_unstale":"2024-09-16T15:46:22.661449+0000","last_undegraded":"2024-09-16T15:46:22.661449+0000","last_fullsized":"2024-09-16T15:46:22.661449+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:19.122499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:19.122499+0000","last_clean_scrub_stamp":"2024-09-16T15:46:19.122499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:54:25.988820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.6","version":"0'0","reported_seq":13,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-16T15:46:22.662831+0000","last_change":"2024-09-16T15:46:21.174448+0000","last_active":"2024-09-16T15:46:22.662831+0000","last_peered":"2024-09-16T15:46:22.662831+0000","last_clean":"2024-09-16T15:46:22.662831+0000","last_became_active":"2024-09-16T15:46:21.172190+0000","last_became_peered":"2024-09-16T15:46:21.172190+0000","last_unstale":"2024-09-16T15:46:22.662831+0000","last_undegraded":"2024-09-16T15:46:22.662831+0000","last_fullsized":"2024-09-16T15:46:22.662831+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:20.126241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:20.126241+0000","last_clean_scrub_stamp":"2024-09-16T15:46:20.126241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:41:54.960196+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.7","version":"57'2","reported_seq":17,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-16T15:46:22.146815+0000","last_change":"2024-09-16T15:46:20.158835+0000","last_active":"2024-09-16T15:46:22.146815+0000","last_peered":"2024-09-16T15:46:22.146815+0000","last_clean":"2024-09-16T15:46:22.146815+0000","last_became_active":"2024-09-16T15:46:20.158347+0000","last_became_peered":"2024-09-16T15:46:20.158347+0000","last_unstale":"2024-09-16T15:46:22.146815+0000","last_undegraded":"2024-09-16T15:46:22.146815+0000","last_fullsized":"2024-09-16T15:46:22.146815+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:19.122499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:19.122499+0000","last_clean_scrub_stamp":"2024-09-16T15:46:19.122499+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T18:25:41.227749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.7","version":"0'0","reported_seq":13,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-16T15:46:22.663005+0000","last_change":"2024-09-16T15:46:21.178341+0000","last_active":"2024-09-16T15:46:22.663005+0000","last_peered":"2024-09-16T15:46:22.663005+0000","last_clean":"2024-09-16T15:46:22.663005+0000","last_became_active":"2024-09-16T15:46:21.177509+0000","last_became_peered":"2024-09-16T15:46:21.177509+0000","last_unstale":"2024-09-16T15:46:22.663005+0000","last_undegraded":"2024-09-16T15:46:22.663005+0000","last_fullsized":"2024-09-16T15:46:22.663005+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:20.126241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:20.126241+0000","last_clean_scrub_stamp":"2024-09-16T15:46:20.126241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:16:25.611352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.6","version":"58'1","reported_seq":16,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-16T15:46:22.666493+0000","last_change":"2024-09-16T15:46:20.166118+0000","last_active":"2024-09-16T15:46:22.666493+0000","last_peered":"2024-09-16T15:46:22.666493+0000","last_clean":"2024-09-16T15:46:22.666493+0000","last_became_active":"2024-09-16T15:46:20.165857+0000","last_became_peered":"2024-09-16T15:46:20.165857+0000","last_unstale":"2024-09-16T15:46:22.666493+0000","last_undegraded":"2024-09-16T15:46:22.666493+0000","last_fullsized":"2024-09-16T15:46:22.666493+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:19.122499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:19.122499+0000","last_clean_scrub_stamp":"2024-09-16T15:46:19.122499+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T00:24:50.840767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.4","version":"0'0","reported_seq":13,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-16T15:46:22.146765+0000","last_change":"2024-09-16T15:46:21.170344+0000","last_active":"2024-09-16T15:46:22.146765+0000","last_peered":"2024-09-16T15:46:22.146765+0000","last_clean":"2024-09-16T15:46:22.146765+0000","last_became_active":"2024-09-16T15:46:21.170101+0000","last_became_peered":"2024-09-16T15:46:21.170101+0000","last_unstale":"2024-09-16T15:46:22.146765+0000","last_undegraded":"2024-09-16T15:46:22.146765+0000","last_fullsized":"2024-09-16T15:46:22.146765+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:20.126241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:20.126241+0000","last_clean_scrub_stamp":"2024-09-16T15:46:20.126241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:22:00.182757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.5","version":"0'0","reported_seq":15,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-16T15:46:22.151504+0000","last_change":"2024-09-16T15:46:20.179032+0000","last_active":"2024-09-16T15:46:22.151504+0000","last_peered":"2024-09-16T15:46:22.151504+0000","last_clean":"2024-09-16T15:46:22.151504+0000","last_became_active":"2024-09-16T15:46:20.178401+0000","last_became_peered":"2024-09-16T15:46:20.178401+0000","last_unstale":"2024-09-16T15:46:22.151504+0000","last_undegraded":"2024-09-16T15:46:22.151504+0000","last_fullsized":"2024-09-16T15:46:22.151504+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:19.122499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:19.122499+0000","last_clean_scrub_stamp":"2024-09-16T15:46:19.122499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T00:16:11.944406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.5","version":"0'0","reported_seq":13,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-16T15:46:22.666665+0000","last_change":"2024-09-16T15:46:21.186703+0000","last_active":"2024-09-16T15:46:22.666665+0000","last_peered":"2024-09-16T15:46:22.666665+0000","last_clean":"2024-09-16T15:46:22.666665+0000","last_became_active":"2024-09-16T15:46:21.186412+0000","last_became_peered":"2024-09-16T15:46:21.186412+0000","last_unstale":"2024-09-16T15:46:22.666665+0000","last_undegraded":"2024-09-16T15:46:22.666665+0000","last_fullsized":"2024-09-16T15:46:22.666665+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:20.126241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:20.126241+0000","last_clean_scrub_stamp":"2024-09-16T15:46:20.126241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T23:59:34.433527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.4","version":"0'0","reported_seq":15,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-16T15:46:22.662973+0000","last_change":"2024-09-16T15:46:20.182719+0000","last_active":"2024-09-16T15:46:22.662973+0000","last_peered":"2024-09-16T15:46:22.662973+0000","last_clean":"2024-09-16T15:46:22.662973+0000","last_became_active":"2024-09-16T15:46:20.182231+0000","last_became_peered":"2024-09-16T15:46:20.182231+0000","last_unstale":"2024-09-16T15:46:22.662973+0000","last_undegraded":"2024-09-16T15:46:22.662973+0000","last_fullsized":"2024-09-16T15:46:22.662973+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:19.122499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:19.122499+0000","last_clean_scrub_stamp":"2024-09-16T15:46:19.122499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T02:34:10.362794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.2","version":"0'0","reported_seq":13,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-16T15:46:22.663908+0000","last_change":"2024-09-16T15:46:21.163961+0000","last_active":"2024-09-16T15:46:22.663908+0000","last_peered":"2024-09-16T15:46:22.663908+0000","last_clean":"2024-09-16T15:46:22.663908+0000","last_became_active":"2024-09-16T15:46:21.163691+0000","last_became_peered":"2024-09-16T15:46:21.163691+0000","last_unstale":"2024-09-16T15:46:22.663908+0000","last_undegraded":"2024-09-16T15:46:22.663908+0000","last_fullsized":"2024-09-16T15:46:22.663908+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:20.126241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:20.126241+0000","last_clean_scrub_stamp":"2024-09-16T15:46:20.126241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T22:23:47.858055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.3","version":"0'0","reported_seq":15,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-16T15:46:22.145560+0000","last_change":"2024-09-16T15:46:20.166298+0000","last_active":"2024-09-16T15:46:22.145560+0000","last_peered":"2024-09-16T15:46:22.145560+0000","last_clean":"2024-09-16T15:46:22.145560+0000","last_became_active":"2024-09-16T15:46:20.165923+0000","last_became_peered":"2024-09-16T15:46:20.165923+0000","last_unstale":"2024-09-16T15:46:22.145560+0000","last_undegraded":"2024-09-16T15:46:22.145560+0000","last_fullsized":"2024-09-16T15:46:22.145560+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:19.122499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:19.122499+0000","last_clean_scrub_stamp":"2024-09-16T15:46:19.122499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T02:55:31.723658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.3","version":"0'0","reported_seq":13,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-16T15:46:22.154816+0000","last_change":"2024-09-16T15:46:21.177222+0000","last_active":"2024-09-16T15:46:22.154816+0000","last_peered":"2024-09-16T15:46:22.154816+0000","last_clean":"2024-09-16T15:46:22.154816+0000","last_became_active":"2024-09-16T15:46:21.176461+0000","last_became_peered":"2024-09-16T15:46:21.176461+0000","last_unstale":"2024-09-16T15:46:22.154816+0000","last_undegraded":"2024-09-16T15:46:22.154816+0000","last_fullsized":"2024-09-16T15:46:22.154816+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:20.126241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:20.126241+0000","last_clean_scrub_stamp":"2024-09-16T15:46:20.126241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T02:22:43.541298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.2","version":"0'0","reported_seq":15,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-16T15:46:22.152484+0000","last_change":"2024-09-16T15:46:20.176443+0000","last_active":"2024-09-16T15:46:22.152484+0000","last_peered":"2024-09-16T15:46:22.152484+0000","last_clean":"2024-09-16T15:46:22.152484+0000","last_became_active":"2024-09-16T15:46:20.175637+0000","last_became_peered":"2024-09-16T15:46:20.175637+0000","last_unstale":"2024-09-16T15:46:22.152484+0000","last_undegraded":"2024-09-16T15:46:22.152484+0000","last_fullsized":"2024-09-16T15:46:22.152484+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:19.122499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:19.122499+0000","last_clean_scrub_stamp":"2024-09-16T15:46:19.122499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T18:05:26.933531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.0","version":"0'0","reported_seq":13,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-16T15:46:22.145480+0000","last_change":"2024-09-16T15:46:21.176827+0000","last_active":"2024-09-16T15:46:22.145480+0000","last_peered":"2024-09-16T15:46:22.145480+0000","last_clean":"2024-09-16T15:46:22.145480+0000","last_became_active":"2024-09-16T15:46:21.176432+0000","last_became_peered":"2024-09-16T15:46:21.176432+0000","last_unstale":"2024-09-16T15:46:22.145480+0000","last_undegraded":"2024-09-16T15:46:22.145480+0000","last_fullsized":"2024-09-16T15:46:22.145480+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:20.126241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:20.126241+0000","last_clean_scrub_stamp":"2024-09-16T15:46:20.126241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:44:35.339565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.1","version":"0'0","reported_seq":15,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-16T15:46:22.145280+0000","last_change":"2024-09-16T15:46:20.162876+0000","last_active":"2024-09-16T15:46:22.145280+0000","last_peered":"2024-09-16T15:46:22.145280+0000","last_clean":"2024-09-16T15:46:22.145280+0000","last_became_active":"2024-09-16T15:46:20.162545+0000","last_became_peered":"2024-09-16T15:46:20.162545+0000","last_unstale":"2024-09-16T15:46:22.145280+0000","last_undegraded":"2024-09-16T15:46:22.145280+0000","last_fullsized":"2024-09-16T15:46:22.145280+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:19.122499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:19.122499+0000","last_clean_scrub_stamp":"2024-09-16T15:46:19.122499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:14:07.111865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.1","version":"0'0","reported_seq":13,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-16T15:46:22.663836+0000","last_change":"2024-09-16T15:46:21.167192+0000","last_active":"2024-09-16T15:46:22.663836+0000","last_peered":"2024-09-16T15:46:22.663836+0000","last_clean":"2024-09-16T15:46:22.663836+0000","last_became_active":"2024-09-16T15:46:21.166925+0000","last_became_peered":"2024-09-16T15:46:21.166925+0000","last_unstale":"2024-09-16T15:46:22.663836+0000","last_undegraded":"2024-09-16T15:46:22.663836+0000","last_fullsized":"2024-09-16T15:46:22.663836+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:20.126241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:20.126241+0000","last_clean_scrub_stamp":"2024-09-16T15:46:20.126241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T18:01:09.069133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.0","version":"0'0","reported_seq":15,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-16T15:46:22.663449+0000","last_change":"2024-09-16T15:46:20.170495+0000","last_active":"2024-09-16T15:46:22.663449+0000","last_peered":"2024-09-16T15:46:22.663449+0000","last_clean":"2024-09-16T15:46:22.663449+0000","last_became_active":"2024-09-16T15:46:20.170270+0000","last_became_peered":"2024-09-16T15:46:20.170270+0000","last_unstale":"2024-09-16T15:46:22.663449+0000","last_undegraded":"2024-09-16T15:46:22.663449+0000","last_fullsized":"2024-09-16T15:46:22.663449+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:19.122499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:19.122499+0000","last_clean_scrub_stamp":"2024-09-16T15:46:19.122499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T01:37:23.276947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.8","version":"0'0","reported_seq":13,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-16T15:46:22.663435+0000","last_change":"2024-09-16T15:46:21.155164+0000","last_active":"2024-09-16T15:46:22.663435+0000","last_peered":"2024-09-16T15:46:22.663435+0000","last_clean":"2024-09-16T15:46:22.663435+0000","last_became_active":"2024-09-16T15:46:21.154880+0000","last_became_peered":"2024-09-16T15:46:21.154880+0000","last_unstale":"2024-09-16T15:46:22.663435+0000","last_undegraded":"2024-09-16T15:46:22.663435+0000","last_fullsized":"2024-09-16T15:46:22.663435+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:20.126241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:20.126241+0000","last_clean_scrub_stamp":"2024-09-16T15:46:20.126241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:36:54.944442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.9","version":"0'0","reported_seq":15,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-16T15:46:22.149854+0000","last_change":"2024-09-16T15:46:20.171339+0000","last_active":"2024-09-16T15:46:22.149854+0000","last_peered":"2024-09-16T15:46:22.149854+0000","last_clean":"2024-09-16T15:46:22.149854+0000","last_became_active":"2024-09-16T15:46:20.170566+0000","last_became_peered":"2024-09-16T15:46:20.170566+0000","last_unstale":"2024-09-16T15:46:22.149854+0000","last_undegraded":"2024-09-16T15:46:22.149854+0000","last_fullsized":"2024-09-16T15:46:22.149854+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:19.122499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:19.122499+0000","last_clean_scrub_stamp":"2024-09-16T15:46:19.122499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:48:30.271405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"16'72","reported_seq":181,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-16T15:46:22.150542+0000","last_change":"2024-09-16T15:42:40.881820+0000","last_active":"2024-09-16T15:46:22.150542+0000","last_peered":"2024-09-16T15:46:22.150542+0000","last_clean":"2024-09-16T15:46:22.150542+0000","last_became_active":"2024-09-16T15:42:40.879965+0000","last_became_peered":"2024-09-16T15:42:40.879965+0000","last_unstale":"2024-09-16T15:46:22.150542+0000","last_undegraded":"2024-09-16T15:46:22.150542+0000","last_fullsized":"2024-09-16T15:46:22.150542+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:42:39.728762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:42:39.728762+0000","last_clean_scrub_stamp":"2024-09-16T15:42:39.728762+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:22:18.768932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":9,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":8,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542193,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28292,"kb_used_data":1264,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343548,"statfs":{"total":96636764160,"available":96607793152,"internally_reserved":0,"allocated":1294336,"data_stored":876981,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542192,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31868,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339972,"statfs":{"total":96636764160,"available":96604131328,"internally_reserved":0,"allocated":692224,"data_stored":286453,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":12,"apply_latency_ms":12,"commit_latency_ns":12000000,"apply_latency_ns":12000000},"alerts":[]},{"osd":5,"up_from":14,"seq":60129542193,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27720,"kb_used_data":684,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344120,"statfs":{"total":96636764160,"available":96608378880,"internally_reserved":0,"allocated":700416,"data_stored":287105,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542191,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31880,"kb_used_data":688,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339960,"statfs":{"total":96636764160,"available":96604119040,"internally_reserved":0,"allocated":704512,"data_stored":288065,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[]},{"osd":3,"up_from":14,"seq":60129542192,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31876,"kb_used_data":684,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339964,"statfs":{"total":96636764160,"available":96604123136,"internally_reserved":0,"allocated":700416,"data_stored":286509,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":2,"up_from":14,"seq":60129542192,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27720,"kb_used_data":684,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344120,"statfs":{"total":96636764160,"available":96608378880,"internally_reserved":0,"allocated":700416,"data_stored":286509,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":14,"seq":60129542194,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27716,"kb_used_data":680,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344124,"statfs":{"total":96636764160,"available":96608382976,"internally_reserved":0,"allocated":696320,"data_stored":286543,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":0,"up_from":14,"seq":60129542194,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28300,"kb_used_data":1272,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343540,"statfs":{"total":96636764160,"available":96607784960,"internally_reserved":0,"allocated":1302528,"data_stored":879015,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2194,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T15:46:25.639 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-09-16T15:46:25.640 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-09-16T15:46:25.809 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:25.805+0000 7fadf77d3640 1 -- 172.21.15.123:0/1317679750 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fadf00c3230 msgr2=0x7fadf00cedd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:46:25.810 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:25.805+0000 7fadf77d3640 1 --2- 172.21.15.123:0/1317679750 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fadf00c3230 0x7fadf00cedd0 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7fadf00b27d0 tx=0x7fade401c8c0 comp rx=0 tx=0).stop 2024-09-16T15:46:25.810 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:25.805+0000 7fadf77d3640 1 -- 172.21.15.123:0/1317679750 shutdown_connections 2024-09-16T15:46:25.810 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:25.805+0000 7fadf77d3640 1 --2- 172.21.15.123:0/1317679750 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fadf00701e0 0x7fadf00cf310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:25.810 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:25.805+0000 7fadf77d3640 1 --2- 172.21.15.123:0/1317679750 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fadf00c3230 0x7fadf00cedd0 unknown :-1 s=CLOSED pgs=646 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:25.810 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:25.805+0000 7fadf77d3640 1 --2- 172.21.15.123:0/1317679750 >> v2:172.21.15.184:3300/0 conn(0x7fadf015b860 0x7fadf006fca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:25.810 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:25.805+0000 7fadf77d3640 1 -- 172.21.15.123:0/1317679750 >> 172.21.15.123:0/1317679750 conn(0x7fadf00b1710 msgr2=0x7fadf00b1b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:46:25.810 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:25.805+0000 7fadf77d3640 1 -- 172.21.15.123:0/1317679750 shutdown_connections 2024-09-16T15:46:25.810 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:25.805+0000 7fadf77d3640 1 -- 172.21.15.123:0/1317679750 wait complete. 2024-09-16T15:46:25.811 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:25.805+0000 7fadf77d3640 1 Processor -- start 2024-09-16T15:46:25.811 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:25.809+0000 7fadf77d3640 1 -- start start 2024-09-16T15:46:25.813 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:25.809+0000 7fadf77d3640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fadf00701e0 0x7fadf0147e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:46:25.813 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:25.809+0000 7fadf77d3640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fadf00c3230 0x7fadf00b2af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:46:25.813 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:25.809+0000 7fadf77d3640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fadf015b860 0x7fadf00b3030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:46:25.813 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:25.809+0000 7fadf77d3640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fadf015fa40 con 0x7fadf015b860 2024-09-16T15:46:25.814 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:25.809+0000 7fadf77d3640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fadf015f8c0 con 0x7fadf00701e0 2024-09-16T15:46:25.814 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:25.809+0000 7fadf77d3640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fadf015fbc0 con 0x7fadf00c3230 2024-09-16T15:46:25.814 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:25.809+0000 7fadf5d70640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fadf015b860 0x7fadf00b3030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:46:25.814 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:25.809+0000 7fadf5d70640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fadf015b860 0x7fadf00b3030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:54708/0 (socket says 172.21.15.123:54708) 2024-09-16T15:46:25.814 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:25.809+0000 7fadf5d70640 1 -- 172.21.15.123:0/1154079601 learned_addr learned my addr 172.21.15.123:0/1154079601 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:46:25.814 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:25.809+0000 7fadf4d6e640 1 --2- 172.21.15.123:0/1154079601 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fadf00c3230 0x7fadf00b2af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:46:25.814 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:25.809+0000 7fadf556f640 1 --2- 172.21.15.123:0/1154079601 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fadf00701e0 0x7fadf0147e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:46:25.814 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:25.809+0000 7fadf5d70640 1 -- 172.21.15.123:0/1154079601 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fadf00c3230 msgr2=0x7fadf00b2af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:46:25.814 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:25.809+0000 7fadf5d70640 1 --2- 172.21.15.123:0/1154079601 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fadf00c3230 0x7fadf00b2af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:25.814 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:25.809+0000 7fadf5d70640 1 -- 172.21.15.123:0/1154079601 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fadf00701e0 msgr2=0x7fadf0147e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:46:25.814 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:25.809+0000 7fadf5d70640 1 --2- 172.21.15.123:0/1154079601 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fadf00701e0 0x7fadf0147e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:25.815 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:25.809+0000 7fadf5d70640 1 -- 172.21.15.123:0/1154079601 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fadf006c730 con 0x7fadf015b860 2024-09-16T15:46:25.815 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:25.809+0000 7fadf556f640 1 --2- 172.21.15.123:0/1154079601 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fadf00701e0 0x7fadf0147e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T15:46:25.815 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:25.809+0000 7fadf4d6e640 1 --2- 172.21.15.123:0/1154079601 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fadf00c3230 0x7fadf00b2af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T15:46:25.815 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:25.809+0000 7fadf5d70640 1 --2- 172.21.15.123:0/1154079601 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fadf015b860 0x7fadf00b3030 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7fade800ea50 tx=0x7fade800ef10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:46:25.815 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:25.809+0000 7fade1ffb640 1 -- 172.21.15.123:0/1154079601 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fade800cdc0 con 0x7fadf015b860 2024-09-16T15:46:25.815 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:25.809+0000 7fade1ffb640 1 -- 172.21.15.123:0/1154079601 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fade8004c40 con 0x7fadf015b860 2024-09-16T15:46:25.815 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:25.809+0000 7fadf77d3640 1 -- 172.21.15.123:0/1154079601 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fadf01537f0 con 0x7fadf015b860 2024-09-16T15:46:25.815 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:25.809+0000 7fade1ffb640 1 -- 172.21.15.123:0/1154079601 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fade8005280 con 0x7fadf015b860 2024-09-16T15:46:25.815 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:25.809+0000 7fadf77d3640 1 -- 172.21.15.123:0/1154079601 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fadf0164540 con 0x7fadf015b860 2024-09-16T15:46:25.815 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:25.809+0000 7fadf77d3640 1 -- 172.21.15.123:0/1154079601 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fadf006fca0 con 0x7fadf015b860 2024-09-16T15:46:25.816 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:25.809+0000 7fade1ffb640 1 -- 172.21.15.123:0/1154079601 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 9) ==== 97233+0+0 (secure 0 0 0) 0x7fade8005420 con 0x7fadf015b860 2024-09-16T15:46:25.816 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:25.809+0000 7fade1ffb640 1 --2- 172.21.15.123:0/1154079601 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fadd4073070 0x7fadd4075530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:46:25.816 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:25.809+0000 7fade1ffb640 1 -- 172.21.15.123:0/1154079601 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(58..58 src has 1..58) ==== 5709+0+0 (secure 0 0 0) 0x7fade8094cd0 con 0x7fadf015b860 2024-09-16T15:46:25.817 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:25.813+0000 7fadf556f640 1 --2- 172.21.15.123:0/1154079601 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fadd4073070 0x7fadd4075530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:46:25.817 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:25.813+0000 7fadf556f640 1 --2- 172.21.15.123:0/1154079601 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fadd4073070 0x7fadd4075530 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7fadec005d90 tx=0x7fadec005d20 comp rx=0 tx=0).ready entity=mgr.4135 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:46:25.818 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:25.817+0000 7fade1ffb640 1 -- 172.21.15.123:0/1154079601 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7fade8016030 con 0x7fadf015b860 2024-09-16T15:46:26.057 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:26.053+0000 7fadf77d3640 1 -- 172.21.15.123:0/1154079601 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) -- 0x7fadf0074700 con 0x7fadf015b860 2024-09-16T15:46:26.060 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:26.057+0000 7fade1ffb640 1 -- 172.21.15.123:0/1154079601 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v19) ==== 190+0+147 (secure 0 0 0) 0x7fade805faf0 con 0x7fadf015b860 2024-09-16T15:46:26.060 INFO:teuthology.orchestra.run.smithi123.stdout:[client.0] 2024-09-16T15:46:26.061 INFO:teuthology.orchestra.run.smithi123.stdout: key = AQC1Uehm7GDTAhAAHWWVIyjVVnTeeozBR8DfFA== 2024-09-16T15:46:26.061 INFO:teuthology.orchestra.run.smithi123.stdout: caps mds = "allow" 2024-09-16T15:46:26.061 INFO:teuthology.orchestra.run.smithi123.stdout: caps mon = "allow r" 2024-09-16T15:46:26.061 INFO:teuthology.orchestra.run.smithi123.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-09-16T15:46:26.061 INFO:teuthology.orchestra.run.smithi123.stderr:updated caps for client.0 2024-09-16T15:46:26.063 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:26.057+0000 7fadf77d3640 1 -- 172.21.15.123:0/1154079601 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fadd4073070 msgr2=0x7fadd4075530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:46:26.063 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:26.057+0000 7fadf77d3640 1 --2- 172.21.15.123:0/1154079601 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fadd4073070 0x7fadd4075530 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7fadec005d90 tx=0x7fadec005d20 comp rx=0 tx=0).stop 2024-09-16T15:46:26.063 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:26.057+0000 7fadf77d3640 1 -- 172.21.15.123:0/1154079601 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fadf015b860 msgr2=0x7fadf00b3030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:46:26.063 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:26.057+0000 7fadf77d3640 1 --2- 172.21.15.123:0/1154079601 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fadf015b860 0x7fadf00b3030 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7fade800ea50 tx=0x7fade800ef10 comp rx=0 tx=0).stop 2024-09-16T15:46:26.063 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:26.061+0000 7fadf77d3640 1 -- 172.21.15.123:0/1154079601 shutdown_connections 2024-09-16T15:46:26.064 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:26.061+0000 7fadf77d3640 1 --2- 172.21.15.123:0/1154079601 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fadd4073070 0x7fadd4075530 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:26.064 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:26.061+0000 7fadf77d3640 1 --2- 172.21.15.123:0/1154079601 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fadf015b860 0x7fadf00b3030 unknown :-1 s=CLOSED pgs=647 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:26.064 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:26.061+0000 7fadf77d3640 1 --2- 172.21.15.123:0/1154079601 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fadf00c3230 0x7fadf00b2af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:26.064 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:26.061+0000 7fadf77d3640 1 --2- 172.21.15.123:0/1154079601 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fadf00701e0 0x7fadf0147e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:26.064 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:26.061+0000 7fadf77d3640 1 -- 172.21.15.123:0/1154079601 >> 172.21.15.123:0/1154079601 conn(0x7fadf00b1710 msgr2=0x7fadf0152ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:46:26.064 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:26.061+0000 7fadf77d3640 1 -- 172.21.15.123:0/1154079601 shutdown_connections 2024-09-16T15:46:26.064 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:26.061+0000 7fadf77d3640 1 -- 172.21.15.123:0/1154079601 wait complete. 2024-09-16T15:46:26.080 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-16T15:46:26.202 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:26.197+0000 7f2bd2e76640 1 -- 172.21.15.123:0/3722957541 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2bcc15bd20 msgr2=0x7f2bcc0b23b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:46:26.202 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:26.197+0000 7f2bd2e76640 1 --2- 172.21.15.123:0/3722957541 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2bcc15bd20 0x7f2bcc0b23b0 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7f2bcc0c8b30 tx=0x7f2bbc01c830 comp rx=0 tx=0).stop 2024-09-16T15:46:26.202 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:26.197+0000 7f2bd2e76640 1 -- 172.21.15.123:0/3722957541 shutdown_connections 2024-09-16T15:46:26.202 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:26.197+0000 7f2bd2e76640 1 --2- 172.21.15.123:0/3722957541 >> v2:172.21.15.184:3300/0 conn(0x7f2bcc0d0430 0x7f2bcc0d1b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:26.202 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:26.197+0000 7f2bd2e76640 1 --2- 172.21.15.123:0/3722957541 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2bcc15bd20 0x7f2bcc0b23b0 unknown :-1 s=CLOSED pgs=648 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:26.202 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:26.197+0000 7f2bd2e76640 1 -- 172.21.15.123:0/3722957541 >> 172.21.15.123:0/3722957541 conn(0x7f2bcc09aa40 msgr2=0x7f2bcc09ae50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:46:26.202 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:26.197+0000 7f2bd2e76640 1 -- 172.21.15.123:0/3722957541 shutdown_connections 2024-09-16T15:46:26.203 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:26.197+0000 7f2bd2e76640 1 -- 172.21.15.123:0/3722957541 wait complete. 2024-09-16T15:46:26.203 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:26.201+0000 7f2bd2e76640 1 Processor -- start 2024-09-16T15:46:26.203 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:26.201+0000 7f2bd2e76640 1 -- start start 2024-09-16T15:46:26.205 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:26.201+0000 7f2bd2e76640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2bcc0d0430 0x7f2bcc06f840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:46:26.205 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:26.201+0000 7f2bd2e76640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2bcc0d2090 0x7f2bcc06e7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:46:26.205 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:26.201+0000 7f2bd2e76640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f2bcc15bd20 0x7f2bcc164c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:46:26.205 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:26.201+0000 7f2bd2e76640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f2bcc168e00 con 0x7f2bcc0d0430 2024-09-16T15:46:26.205 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:26.201+0000 7f2bd2e76640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f2bcc168c80 con 0x7f2bcc0d2090 2024-09-16T15:46:26.205 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:26.201+0000 7f2bd2e76640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f2bcc168f80 con 0x7f2bcc15bd20 2024-09-16T15:46:26.205 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:26.201+0000 7f2bd0c12640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2bcc0d0430 0x7f2bcc06f840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:46:26.205 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:26.201+0000 7f2bd0c12640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2bcc0d0430 0x7f2bcc06f840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:54730/0 (socket says 172.21.15.123:54730) 2024-09-16T15:46:26.205 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:26.201+0000 7f2bd0c12640 1 -- 172.21.15.123:0/4191144616 learned_addr learned my addr 172.21.15.123:0/4191144616 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:46:26.205 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:26.201+0000 7f2bcbfff640 1 --2- 172.21.15.123:0/4191144616 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2bcc0d2090 0x7f2bcc06e7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:46:26.205 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:26.201+0000 7f2bd1413640 1 --2- 172.21.15.123:0/4191144616 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f2bcc15bd20 0x7f2bcc164c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:46:26.206 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:26.201+0000 7f2bd0c12640 1 -- 172.21.15.123:0/4191144616 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f2bcc15bd20 msgr2=0x7f2bcc164c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:46:26.206 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:26.201+0000 7f2bd0c12640 1 --2- 172.21.15.123:0/4191144616 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f2bcc15bd20 0x7f2bcc164c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:26.206 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:26.201+0000 7f2bd0c12640 1 -- 172.21.15.123:0/4191144616 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2bcc0d2090 msgr2=0x7f2bcc06e7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:46:26.206 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:26.201+0000 7f2bd0c12640 1 --2- 172.21.15.123:0/4191144616 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2bcc0d2090 0x7f2bcc06e7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:26.206 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:26.201+0000 7f2bd0c12640 1 -- 172.21.15.123:0/4191144616 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2bcc06ed20 con 0x7f2bcc0d0430 2024-09-16T15:46:26.206 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:26.201+0000 7f2bd0c12640 1 --2- 172.21.15.123:0/4191144616 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2bcc0d0430 0x7f2bcc06f840 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7f2bbc0b0040 tx=0x7f2bbc0a5620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:46:26.207 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:26.201+0000 7f2bc97fa640 1 -- 172.21.15.123:0/4191144616 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2bbc0b2040 con 0x7f2bcc0d0430 2024-09-16T15:46:26.207 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:26.201+0000 7f2bc97fa640 1 -- 172.21.15.123:0/4191144616 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f2bbc0a6cd0 con 0x7f2bcc0d0430 2024-09-16T15:46:26.207 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:26.201+0000 7f2bd2e76640 1 -- 172.21.15.123:0/4191144616 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2bcc1f8d20 con 0x7f2bcc0d0430 2024-09-16T15:46:26.207 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:26.201+0000 7f2bc97fa640 1 -- 172.21.15.123:0/4191144616 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2bbc003860 con 0x7f2bcc0d0430 2024-09-16T15:46:26.207 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:26.201+0000 7f2bd2e76640 1 -- 172.21.15.123:0/4191144616 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2bcc15e7c0 con 0x7f2bcc0d0430 2024-09-16T15:46:26.208 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:26.205+0000 7f2bd2e76640 1 -- 172.21.15.123:0/4191144616 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2bcc06d6a0 con 0x7f2bcc0d0430 2024-09-16T15:46:26.208 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:26.205+0000 7f2bc97fa640 1 -- 172.21.15.123:0/4191144616 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 9) ==== 97233+0+0 (secure 0 0 0) 0x7f2bbc0bf060 con 0x7f2bcc0d0430 2024-09-16T15:46:26.208 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:26.205+0000 7f2bc97fa640 1 --2- 172.21.15.123:0/4191144616 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f2bb4073070 0x7f2bb4075530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:46:26.208 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:26.205+0000 7f2bc97fa640 1 -- 172.21.15.123:0/4191144616 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(58..58 src has 1..58) ==== 5709+0+0 (secure 0 0 0) 0x7f2bbc12e110 con 0x7f2bcc0d0430 2024-09-16T15:46:26.208 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:26.205+0000 7f2bcbfff640 1 --2- 172.21.15.123:0/4191144616 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f2bb4073070 0x7f2bb4075530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:46:26.209 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:26.205+0000 7f2bcbfff640 1 --2- 172.21.15.123:0/4191144616 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f2bb4073070 0x7f2bb4075530 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f2bcc0b7cc0 tx=0x7f2bb800a3b0 comp rx=0 tx=0).ready entity=mgr.4135 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:46:26.212 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:26.209+0000 7f2bc97fa640 1 -- 172.21.15.123:0/4191144616 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f2bbc0164b0 con 0x7f2bcc0d0430 2024-09-16T15:46:26.406 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:26.401+0000 7f2bd2e76640 1 -- 172.21.15.123:0/4191144616 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f2bcc166620 con 0x7f2bcc0d0430 2024-09-16T15:46:26.407 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:26.405+0000 7f2bc97fa640 1 -- 172.21.15.123:0/4191144616 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 36 v36) ==== 94+0+5232 (secure 0 0 0) 0x7f2bbc0f8fe0 con 0x7f2bcc0d0430 2024-09-16T15:46:26.407 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-09-16T15:46:26.407 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":36,"btime":"2024-09-16T15:46:24:187144+0000","default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4901,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6835/4036392311","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":4036392311},{"type":"v1","addr":"172.21.15.184:6835","nonce":4036392311}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":31},{"gid":5203,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6837/436207465","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6836","nonce":436207465},{"type":"v1","addr":"172.21.15.184:6837","nonce":436207465}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":31},{"gid":5796,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6834/4239187724","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6832","nonce":4239187724},{"type":"v1","addr":"172.21.15.123:6834","nonce":4239187724}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":31}],"filesystems":[{"mdsmap":{"epoch":36,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-16T15:46:20.516000+0000","modified":"2024-09-16T15:46:23.563031+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5799},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5799":{"gid":5799,"name":"c","rank":0,"incarnation":32,"state":"up:active","state_seq":3,"addr":"172.21.15.123:6835/3589624203","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6833","nonce":3589624203},{"type":"v1","addr":"172.21.15.123:6835","nonce":3589624203}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[9],"metadata_pool":8,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5799,"qdb_cluster":[5799]},"id":4}]} 2024-09-16T15:46:26.408 INFO:teuthology.orchestra.run.smithi123.stderr:dumped fsmap epoch 36 2024-09-16T15:46:26.410 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:26.405+0000 7f2bd2e76640 1 -- 172.21.15.123:0/4191144616 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f2bb4073070 msgr2=0x7f2bb4075530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:46:26.410 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:26.405+0000 7f2bd2e76640 1 --2- 172.21.15.123:0/4191144616 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f2bb4073070 0x7f2bb4075530 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f2bcc0b7cc0 tx=0x7f2bb800a3b0 comp rx=0 tx=0).stop 2024-09-16T15:46:26.410 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:26.405+0000 7f2bd2e76640 1 -- 172.21.15.123:0/4191144616 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2bcc0d0430 msgr2=0x7f2bcc06f840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:46:26.410 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:26.405+0000 7f2bd2e76640 1 --2- 172.21.15.123:0/4191144616 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2bcc0d0430 0x7f2bcc06f840 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7f2bbc0b0040 tx=0x7f2bbc0a5620 comp rx=0 tx=0).stop 2024-09-16T15:46:26.410 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:26.405+0000 7f2bd2e76640 1 -- 172.21.15.123:0/4191144616 shutdown_connections 2024-09-16T15:46:26.410 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:26.405+0000 7f2bd2e76640 1 --2- 172.21.15.123:0/4191144616 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f2bb4073070 0x7f2bb4075530 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:26.410 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:26.405+0000 7f2bd2e76640 1 --2- 172.21.15.123:0/4191144616 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f2bcc15bd20 0x7f2bcc164c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:26.410 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:26.405+0000 7f2bd2e76640 1 --2- 172.21.15.123:0/4191144616 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2bcc0d2090 0x7f2bcc06e7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:26.410 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:26.405+0000 7f2bd2e76640 1 --2- 172.21.15.123:0/4191144616 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2bcc0d0430 0x7f2bcc06f840 unknown :-1 s=CLOSED pgs=649 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:26.410 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:26.405+0000 7f2bd2e76640 1 -- 172.21.15.123:0/4191144616 >> 172.21.15.123:0/4191144616 conn(0x7f2bcc09aa40 msgr2=0x7f2bcc0716d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:46:26.411 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:26.405+0000 7f2bd2e76640 1 -- 172.21.15.123:0/4191144616 shutdown_connections 2024-09-16T15:46:26.411 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:26.405+0000 7f2bd2e76640 1 -- 172.21.15.123:0/4191144616 wait complete. 2024-09-16T15:46:26.487 INFO:tasks.ceph.mds.c.smithi123.stderr:2024-09-16T15:46:26.485+0000 7fb6088fa640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-09-16T15:46:26.828 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 36, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-09-16T15:46:20.516000+0000', 'modified': '2024-09-16T15:46:23.563031+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 5799}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_5799': {'gid': 5799, 'name': 'c', 'rank': 0, 'incarnation': 32, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.123:6835/3589624203', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.123:6833', 'nonce': 3589624203}, {'type': 'v1', 'addr': '172.21.15.123:6835', 'nonce': 3589624203}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540719139924082687, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [9], 'metadata_pool': 8, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 5799, 'qdb_cluster': [5799]} 2024-09-16T15:46:26.828 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-09-16T15:46:26.828 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-09-16T15:46:26.961 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:26.957+0000 7fef4f66c640 1 -- 172.21.15.123:0/3796181559 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fef480b2b00 msgr2=0x7fef480d24b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:46:26.961 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:26.957+0000 7fef4f66c640 1 --2- 172.21.15.123:0/3796181559 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fef480b2b00 0x7fef480d24b0 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7fef38009840 tx=0x7fef3801c6d0 comp rx=0 tx=0).stop 2024-09-16T15:46:26.961 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:26.957+0000 7fef4f66c640 1 -- 172.21.15.123:0/3796181559 shutdown_connections 2024-09-16T15:46:26.961 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:26.957+0000 7fef4f66c640 1 --2- 172.21.15.123:0/3796181559 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fef480b0a50 0x7fef480b0ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:26.962 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:26.957+0000 7fef4f66c640 1 --2- 172.21.15.123:0/3796181559 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fef480b2b00 0x7fef480d24b0 unknown :-1 s=CLOSED pgs=651 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:26.962 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:26.957+0000 7fef4f66c640 1 -- 172.21.15.123:0/3796181559 >> 172.21.15.123:0/3796181559 conn(0x7fef480d3ea0 msgr2=0x7fef480d42b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:46:26.962 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:26.957+0000 7fef4f66c640 1 -- 172.21.15.123:0/3796181559 shutdown_connections 2024-09-16T15:46:26.962 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:26.957+0000 7fef4f66c640 1 -- 172.21.15.123:0/3796181559 wait complete. 2024-09-16T15:46:26.962 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:26.957+0000 7fef4f66c640 1 Processor -- start 2024-09-16T15:46:26.962 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:26.957+0000 7fef4f66c640 1 -- start start 2024-09-16T15:46:26.964 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:26.961+0000 7fef4f66c640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fef480b0a50 0x7fef481554d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:46:26.964 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:26.961+0000 7fef4f66c640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fef480b2b00 0x7fef4815bcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:46:26.964 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:26.961+0000 7fef4f66c640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fef48161620 0x7fef4815c200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:46:26.964 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:26.961+0000 7fef4f66c640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fef4816b0f0 con 0x7fef480b0a50 2024-09-16T15:46:26.964 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:26.961+0000 7fef4f66c640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fef4816af70 con 0x7fef480b2b00 2024-09-16T15:46:26.964 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:26.961+0000 7fef4f66c640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fef4816b270 con 0x7fef48161620 2024-09-16T15:46:26.964 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:26.961+0000 7fef4d408640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fef480b0a50 0x7fef481554d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:46:26.965 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:26.961+0000 7fef4d408640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fef480b0a50 0x7fef481554d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:54754/0 (socket says 172.21.15.123:54754) 2024-09-16T15:46:26.965 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:26.961+0000 7fef4d408640 1 -- 172.21.15.123:0/1427987952 learned_addr learned my addr 172.21.15.123:0/1427987952 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:46:26.965 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:26.961+0000 7fef4cc07640 1 --2- 172.21.15.123:0/1427987952 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fef480b2b00 0x7fef4815bcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:46:26.965 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:26.961+0000 7fef4d408640 1 -- 172.21.15.123:0/1427987952 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fef48161620 msgr2=0x7fef4815c200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:46:26.965 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:26.961+0000 7fef4d408640 1 --2- 172.21.15.123:0/1427987952 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fef48161620 0x7fef4815c200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:26.965 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:26.961+0000 7fef4d408640 1 -- 172.21.15.123:0/1427987952 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fef480b2b00 msgr2=0x7fef4815bcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:46:26.965 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:26.961+0000 7fef4d408640 1 --2- 172.21.15.123:0/1427987952 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fef480b2b00 0x7fef4815bcc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:26.965 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:26.961+0000 7fef4d408640 1 -- 172.21.15.123:0/1427987952 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fef480c8910 con 0x7fef480b0a50 2024-09-16T15:46:26.965 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:26.961+0000 7fef4cc07640 1 --2- 172.21.15.123:0/1427987952 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fef480b2b00 0x7fef4815bcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T15:46:26.965 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:26.961+0000 7fef4d408640 1 --2- 172.21.15.123:0/1427987952 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fef480b0a50 0x7fef481554d0 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7fef4400c430 tx=0x7fef4400c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:46:26.965 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:26.961+0000 7fef3dffb640 1 -- 172.21.15.123:0/1427987952 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef44017070 con 0x7fef480b0a50 2024-09-16T15:46:26.965 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:26.961+0000 7fef3dffb640 1 -- 172.21.15.123:0/1427987952 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fef44003600 con 0x7fef480b0a50 2024-09-16T15:46:26.965 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:26.961+0000 7fef4f66c640 1 -- 172.21.15.123:0/1427987952 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fef480ca850 con 0x7fef480b0a50 2024-09-16T15:46:26.966 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:26.961+0000 7fef3dffb640 1 -- 172.21.15.123:0/1427987952 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef44003a30 con 0x7fef480b0a50 2024-09-16T15:46:26.966 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:26.961+0000 7fef4f66c640 1 -- 172.21.15.123:0/1427987952 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fef480c8e90 con 0x7fef480b0a50 2024-09-16T15:46:26.967 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:26.961+0000 7fef3dffb640 1 -- 172.21.15.123:0/1427987952 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 9) ==== 97233+0+0 (secure 0 0 0) 0x7fef4401a070 con 0x7fef480b0a50 2024-09-16T15:46:26.967 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:26.961+0000 7fef3dffb640 1 --2- 172.21.15.123:0/1427987952 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fef2c073070 0x7fef2c075530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:46:26.967 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:26.961+0000 7fef3dffb640 1 -- 172.21.15.123:0/1427987952 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(58..58 src has 1..58) ==== 5709+0+0 (secure 0 0 0) 0x7fef44093420 con 0x7fef480b0a50 2024-09-16T15:46:26.967 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:26.961+0000 7fef4cc07640 1 --2- 172.21.15.123:0/1427987952 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fef2c073070 0x7fef2c075530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:46:26.967 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:26.961+0000 7fef52756000 1 Processor -- start 2024-09-16T15:46:26.967 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:26.961+0000 7fef52756000 1 -- start start 2024-09-16T15:46:26.967 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:26.965+0000 7fef4cc07640 1 --2- 172.21.15.123:0/1427987952 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fef2c073070 0x7fef2c075530 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7fef380af000 tx=0x7fef3801af10 comp rx=0 tx=0).ready entity=mgr.4135 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:46:26.969 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:26.965+0000 7fef52756000 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x560d4ac509d0 0x560d4ac50db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:46:26.969 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:26.965+0000 7fef52756000 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x560d4ac512f0 0x560d4ac048d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:46:26.969 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:26.965+0000 7fef52756000 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x560d4ac04e10 0x560d4ac051f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:46:26.970 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:26.965+0000 7fef52756000 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x560d4ab253f0 con 0x560d4ac509d0 2024-09-16T15:46:26.970 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:26.965+0000 7fef52756000 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x560d4aa62b20 con 0x560d4ac04e10 2024-09-16T15:46:26.970 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:26.965+0000 7fef52756000 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x560d4aa51f50 con 0x560d4ac512f0 2024-09-16T15:46:26.970 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:26.965+0000 7fef4d408640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x560d4ac509d0 0x560d4ac50db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:46:26.970 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:26.965+0000 7fef4d408640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x560d4ac509d0 0x560d4ac50db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:54762/0 (socket says 172.21.15.123:54762) 2024-09-16T15:46:26.970 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:26.965+0000 7fef4d408640 1 -- 172.21.15.123:0/3423470123 learned_addr learned my addr 172.21.15.123:0/3423470123 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:46:26.970 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:26.965+0000 7fef4cc07640 1 --2- 172.21.15.123:0/3423470123 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x560d4ac512f0 0x560d4ac048d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:46:26.970 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:26.965+0000 7fef4d408640 1 -- 172.21.15.123:0/3423470123 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x560d4ac512f0 msgr2=0x560d4ac048d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T15:46:26.970 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:26.965+0000 7fef4dc09640 1 --2- 172.21.15.123:0/3423470123 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x560d4ac04e10 0x560d4ac051f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:46:26.970 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:26.965+0000 7fef4d408640 1 --2- 172.21.15.123:0/3423470123 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x560d4ac512f0 0x560d4ac048d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:26.970 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:26.965+0000 7fef4d408640 1 -- 172.21.15.123:0/3423470123 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x560d4ac04e10 msgr2=0x560d4ac051f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T15:46:26.970 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:26.965+0000 7fef4d408640 1 --2- 172.21.15.123:0/3423470123 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x560d4ac04e10 0x560d4ac051f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:26.971 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:26.965+0000 7fef4d408640 1 -- 172.21.15.123:0/3423470123 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560d4aa19590 con 0x560d4ac509d0 2024-09-16T15:46:26.971 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:26.965+0000 7fef4d408640 1 --2- 172.21.15.123:0/3423470123 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x560d4ac509d0 0x560d4ac50db0 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7fef4400c400 tx=0x7fef4405d780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=43fbc48a7c5afbde server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:46:26.971 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:26.965+0000 7fef32ffd640 1 -- 172.21.15.123:0/3423470123 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef4405dc70 con 0x560d4ac509d0 2024-09-16T15:46:26.971 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:26.965+0000 7fef32ffd640 1 -- 172.21.15.123:0/3423470123 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fef44062030 con 0x560d4ac509d0 2024-09-16T15:46:26.971 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:26.965+0000 7fef32ffd640 1 -- 172.21.15.123:0/3423470123 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef44062750 con 0x560d4ac509d0 2024-09-16T15:46:26.971 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:26.965+0000 7fef52756000 1 -- 172.21.15.123:0/3423470123 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x560d4ac509d0 msgr2=0x560d4ac50db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:46:26.971 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:26.965+0000 7fef52756000 1 --2- 172.21.15.123:0/3423470123 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x560d4ac509d0 0x560d4ac50db0 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7fef4400c400 tx=0x7fef4405d780 comp rx=0 tx=0).stop 2024-09-16T15:46:26.971 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:26.965+0000 7fef52756000 1 -- 172.21.15.123:0/3423470123 shutdown_connections 2024-09-16T15:46:26.971 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:26.965+0000 7fef52756000 1 --2- 172.21.15.123:0/3423470123 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x560d4ac04e10 0x560d4ac051f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:26.971 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:26.965+0000 7fef52756000 1 --2- 172.21.15.123:0/3423470123 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x560d4ac512f0 0x560d4ac048d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:26.971 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:26.965+0000 7fef52756000 1 --2- 172.21.15.123:0/3423470123 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x560d4ac509d0 0x560d4ac50db0 unknown :-1 s=CLOSED pgs=653 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:26.971 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:26.965+0000 7fef52756000 1 -- 172.21.15.123:0/3423470123 >> 172.21.15.123:0/3423470123 conn(0x560d4aaee130 msgr2=0x560d4ac33e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:46:26.971 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:26.965+0000 7fef52756000 1 -- 172.21.15.123:0/3423470123 shutdown_connections 2024-09-16T15:46:26.972 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:26.965+0000 7fef52756000 1 -- 172.21.15.123:0/3423470123 wait complete. 2024-09-16T15:46:26.972 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:26.965+0000 7fef52756000 1 -- start start 2024-09-16T15:46:26.972 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:26.965+0000 7fef52756000 1 Processor -- start 2024-09-16T15:46:26.975 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:26.965+0000 7fef52756000 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x560d4ac04e10 0x560d4ac316a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:46:26.975 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:26.965+0000 7fef52756000 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x560d4ac509d0 0x560d4ac31e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:46:26.975 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:26.965+0000 7fef52756000 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x560d4ac512f0 0x560d4ac323b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:46:26.975 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:26.965+0000 7fef52756000 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x560d4a9ec5d0 con 0x560d4ac512f0 2024-09-16T15:46:26.975 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:26.965+0000 7fef52756000 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x560d4aa55d60 con 0x560d4ac509d0 2024-09-16T15:46:26.975 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:26.965+0000 7fef52756000 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x560d4a9f8500 con 0x560d4ac04e10 2024-09-16T15:46:26.975 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:26.965+0000 7fef52756000 10 client.0 _ll_register_callbacks cb 0x560d4a9fc810 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T15:46:26.975 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:26.965+0000 7fef4dc09640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x560d4ac512f0 0x560d4ac323b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:46:26.975 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:26.965+0000 7fef4dc09640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x560d4ac512f0 0x560d4ac323b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:54778/0 (socket says 172.21.15.123:54778) 2024-09-16T15:46:26.975 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:26.965+0000 7fef4dc09640 1 -- 172.21.15.123:0/3728054209 learned_addr learned my addr 172.21.15.123:0/3728054209 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:46:26.975 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:26.965+0000 7fef4d408640 1 --2- 172.21.15.123:0/3728054209 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x560d4ac04e10 0x560d4ac316a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:46:26.976 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:26.965+0000 7fef4cc07640 1 --2- 172.21.15.123:0/3728054209 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x560d4ac509d0 0x560d4ac31e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:46:26.976 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:26.969+0000 7fef4dc09640 1 -- 172.21.15.123:0/3728054209 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x560d4ac04e10 msgr2=0x560d4ac316a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T15:46:26.976 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:26.969+0000 7fef4dc09640 1 --2- 172.21.15.123:0/3728054209 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x560d4ac04e10 0x560d4ac316a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:26.976 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:26.969+0000 7fef4dc09640 1 -- 172.21.15.123:0/3728054209 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x560d4ac509d0 msgr2=0x560d4ac31e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T15:46:26.976 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:26.969+0000 7fef4dc09640 1 --2- 172.21.15.123:0/3728054209 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x560d4ac509d0 0x560d4ac31e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:26.976 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:26.969+0000 7fef4dc09640 1 -- 172.21.15.123:0/3728054209 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560d4a9f0170 con 0x560d4ac512f0 2024-09-16T15:46:26.976 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:26.969+0000 7fef4dc09640 1 --2- 172.21.15.123:0/3728054209 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x560d4ac512f0 0x560d4ac323b0 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7fef40009f70 tx=0x7fef4000f7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f4ddcfa4c9634201 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:46:26.976 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:26.969+0000 7fef27fff640 10 client.0 ms_handle_connect on v2:172.21.15.123:3300/0 2024-09-16T15:46:26.976 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:26.969+0000 7fef27fff640 1 -- 172.21.15.123:0/3728054209 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef40004d30 con 0x560d4ac512f0 2024-09-16T15:46:26.976 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:26.969+0000 7fef52756000 1 -- 172.21.15.123:0/3728054209 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x560d4a9ea0f0 con 0x560d4ac512f0 2024-09-16T15:46:26.976 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:26.969+0000 7fef27fff640 1 -- 172.21.15.123:0/3728054209 <== mon.0 v2:172.21.15.123:3300/0 2 ==== osd_map(58..58 src has 1..58) ==== 5709+0+0 (secure 0 0 0) 0x7fef40011650 con 0x560d4ac512f0 2024-09-16T15:46:26.976 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:26.969+0000 7fef27fff640 1 -- 172.21.15.123:0/3728054209 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7fef40021050 con 0x560d4ac512f0 2024-09-16T15:46:26.976 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:26.969+0000 7fef52756000 10 client.5931 fetch_fsmap learned FSMap version 36 2024-09-16T15:46:26.976 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:26.969+0000 7fef52756000 1 -- 172.21.15.123:0/3728054209 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=2+}) -- 0x560d4aad6ef0 con 0x560d4ac512f0 2024-09-16T15:46:26.977 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:26.969+0000 7fef27fff640 1 -- 172.21.15.123:0/3728054209 <== mon.0 v2:172.21.15.123:3300/0 4 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x560d4a9ea0f0 con 0x560d4ac512f0 2024-09-16T15:46:26.977 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:26.969+0000 7fef27fff640 1 -- 172.21.15.123:0/3728054209 <== mon.0 v2:172.21.15.123:3300/0 5 ==== fsmap(e 36) ==== 3502+0+0 (secure 0 0 0) 0x7fef40024020 con 0x560d4ac512f0 2024-09-16T15:46:26.977 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:26.969+0000 7fef52756000 10 client.5931 fetch_fsmap finished waiting for FSMap version 36 2024-09-16T15:46:26.977 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:26.969+0000 7fef52756000 10 client.5931 resolve_mds: resolved name 'c' to daemon mds.c 2024-09-16T15:46:26.977 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:26.969+0000 7fef52756000 20 client.5931 populate_metadata read hostname 'smithi123' 2024-09-16T15:46:26.977 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:26.969+0000 7fef52756000 1 --2- 172.21.15.123:0/3728054209 >> [v2:172.21.15.123:6833/3589624203,v1:172.21.15.123:6835/3589624203] conn(0x560d4ac330f0 0x560d4ac62c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:46:26.977 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:26.969+0000 7fef52756000 4 client.5931 mds_command: new command op to 5799 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-16T15:46:26.977 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:26.969+0000 7fef52756000 1 -- 172.21.15.123:0/3728054209 --> [v2:172.21.15.123:6833/3589624203,v1:172.21.15.123:6835/3589624203] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560d4aa64a70 con 0x560d4ac330f0 2024-09-16T15:46:26.977 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:26.969+0000 7fef4d408640 1 --2- 172.21.15.123:0/3728054209 >> [v2:172.21.15.123:6833/3589624203,v1:172.21.15.123:6835/3589624203] conn(0x560d4ac330f0 0x560d4ac62c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:46:26.977 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:26.969+0000 7fef4d408640 1 --2- 172.21.15.123:0/3728054209 >> [v2:172.21.15.123:6833/3589624203,v1:172.21.15.123:6835/3589624203] conn(0x560d4ac330f0 0x560d4ac62c90 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ec45e23b8efb9c31 server_cookie=bacc8a2a29d67c96 in_seq=0 out_seq=0 2024-09-16T15:46:26.977 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:26.969+0000 7fef27fff640 10 client.5931 ms_handle_connect on v2:172.21.15.123:6833/3589624203 2024-09-16T15:46:26.977 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:26.969+0000 7fef27fff640 1 -- 172.21.15.123:0/3728054209 <== mds.0 v2:172.21.15.123:6833/3589624203 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x560d4aa64a70 con 0x560d4ac330f0 2024-09-16T15:46:26.977 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:26.969+0000 7fef27fff640 10 client.5931 handle_command_reply: tid=0 2024-09-16T15:46:26.977 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:26.969+0000 7fef52756000 1 client.5931 shutdown 2024-09-16T15:46:26.978 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:26.969+0000 7fef52756000 1 -- 172.21.15.123:0/3728054209 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x560d4ac512f0 msgr2=0x560d4ac323b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:46:26.978 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:26.969+0000 7fef52756000 1 --2- 172.21.15.123:0/3728054209 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x560d4ac512f0 0x560d4ac323b0 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7fef40009f70 tx=0x7fef4000f7f0 comp rx=0 tx=0).stop 2024-09-16T15:46:26.978 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:26.969+0000 7fef52756000 1 -- 172.21.15.123:0/3728054209 shutdown_connections 2024-09-16T15:46:26.978 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:26.969+0000 7fef52756000 1 --2- 172.21.15.123:0/3728054209 >> [v2:172.21.15.123:6833/3589624203,v1:172.21.15.123:6835/3589624203] conn(0x560d4ac330f0 0x560d4ac62c90 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:26.978 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:26.969+0000 7fef52756000 1 --2- 172.21.15.123:0/3728054209 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x560d4ac512f0 0x560d4ac323b0 unknown :-1 s=CLOSED pgs=654 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:26.978 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:26.969+0000 7fef27fff640 0 client.5931 ms_handle_reset on v2:172.21.15.123:6833/3589624203 2024-09-16T15:46:26.978 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:26.969+0000 7fef52756000 1 --2- 172.21.15.123:0/3728054209 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x560d4ac509d0 0x560d4ac31e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:26.978 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:26.969+0000 7fef52756000 1 --2- 172.21.15.123:0/3728054209 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x560d4ac04e10 0x560d4ac316a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:26.978 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:26.969+0000 7fef52756000 1 -- 172.21.15.123:0/3728054209 >> 172.21.15.123:0/3728054209 conn(0x560d4aaee130 msgr2=0x560d4ac03fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:46:26.978 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:26.969+0000 7fef52756000 1 -- 172.21.15.123:0/3728054209 shutdown_connections 2024-09-16T15:46:26.978 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:26.969+0000 7fef52756000 1 -- 172.21.15.123:0/3728054209 wait complete. 2024-09-16T15:46:26.978 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:26.969+0000 7fef52756000 20 client.5931 trim_cache size 0 max 16384 2024-09-16T15:46:26.992 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:26.989+0000 7fef52756000 1 Processor -- start 2024-09-16T15:46:26.992 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:26.989+0000 7fef52756000 1 -- start start 2024-09-16T15:46:26.993 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:26.989+0000 7fef52756000 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x560d4ac04e10 0x560d4acc3400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:46:26.993 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:26.989+0000 7fef52756000 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x560d4ac509d0 0x560d4acc3940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:46:26.993 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:26.989+0000 7fef52756000 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x560d4ac512f0 0x560d4acc3e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:46:26.994 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:26.989+0000 7fef52756000 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fef48203cd0 con 0x560d4ac04e10 2024-09-16T15:46:26.994 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:26.989+0000 7fef52756000 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fef481f8460 con 0x560d4ac512f0 2024-09-16T15:46:26.994 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:26.989+0000 7fef52756000 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fef481f77f0 con 0x560d4ac509d0 2024-09-16T15:46:26.994 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:26.989+0000 7fef4d408640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x560d4ac04e10 0x560d4acc3400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:46:26.994 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:26.989+0000 7fef4d408640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x560d4ac04e10 0x560d4acc3400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:54794/0 (socket says 172.21.15.123:54794) 2024-09-16T15:46:26.994 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:26.989+0000 7fef4d408640 1 -- 172.21.15.123:0/3189397140 learned_addr learned my addr 172.21.15.123:0/3189397140 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:46:26.994 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:26.989+0000 7fef4dc09640 1 --2- 172.21.15.123:0/3189397140 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x560d4ac512f0 0x560d4acc3e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:46:26.994 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:26.989+0000 7fef4d408640 1 -- 172.21.15.123:0/3189397140 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x560d4ac509d0 msgr2=0x560d4acc3940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T15:46:26.994 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:26.989+0000 7fef4d408640 1 --2- 172.21.15.123:0/3189397140 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x560d4ac509d0 0x560d4acc3940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:26.994 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:26.989+0000 7fef4d408640 1 -- 172.21.15.123:0/3189397140 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x560d4ac512f0 msgr2=0x560d4acc3e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T15:46:26.994 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:26.989+0000 7fef4d408640 1 --2- 172.21.15.123:0/3189397140 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x560d4ac512f0 0x560d4acc3e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:26.995 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:26.989+0000 7fef4d408640 1 -- 172.21.15.123:0/3189397140 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560d4aaadd90 con 0x560d4ac04e10 2024-09-16T15:46:26.995 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:26.989+0000 7fef4d408640 1 --2- 172.21.15.123:0/3189397140 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x560d4ac04e10 0x560d4acc3400 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7fef4405dc00 tx=0x7fef44077f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=36860d63db9ac622 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:46:26.995 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:26.989+0000 7fef327fc640 1 -- 172.21.15.123:0/3189397140 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef4405dc70 con 0x560d4ac04e10 2024-09-16T15:46:26.995 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:26.989+0000 7fef327fc640 1 -- 172.21.15.123:0/3189397140 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fef44062040 con 0x560d4ac04e10 2024-09-16T15:46:26.995 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:26.989+0000 7fef327fc640 1 -- 172.21.15.123:0/3189397140 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef440918b0 con 0x560d4ac04e10 2024-09-16T15:46:26.999 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:26.989+0000 7fef52756000 1 -- 172.21.15.123:0/3189397140 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x560d4ac04e10 msgr2=0x560d4acc3400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:46:26.999 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:26.989+0000 7fef52756000 1 --2- 172.21.15.123:0/3189397140 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x560d4ac04e10 0x560d4acc3400 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7fef4405dc00 tx=0x7fef44077f90 comp rx=0 tx=0).stop 2024-09-16T15:46:26.999 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:26.989+0000 7fef52756000 1 -- 172.21.15.123:0/3189397140 shutdown_connections 2024-09-16T15:46:26.999 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:26.989+0000 7fef52756000 1 --2- 172.21.15.123:0/3189397140 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x560d4ac512f0 0x560d4acc3e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:26.999 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:26.989+0000 7fef52756000 1 --2- 172.21.15.123:0/3189397140 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x560d4ac509d0 0x560d4acc3940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:26.999 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:26.989+0000 7fef52756000 1 --2- 172.21.15.123:0/3189397140 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x560d4ac04e10 0x560d4acc3400 unknown :-1 s=CLOSED pgs=655 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:27.000 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:26.989+0000 7fef52756000 1 -- 172.21.15.123:0/3189397140 >> 172.21.15.123:0/3189397140 conn(0x560d4aaee130 msgr2=0x560d4ac0c2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:46:27.000 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:26.989+0000 7fef52756000 1 -- 172.21.15.123:0/3189397140 shutdown_connections 2024-09-16T15:46:27.000 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:26.989+0000 7fef52756000 1 -- 172.21.15.123:0/3189397140 wait complete. 2024-09-16T15:46:27.000 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:26.989+0000 7fef52756000 1 -- start start 2024-09-16T15:46:27.000 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:26.989+0000 7fef52756000 1 Processor -- start 2024-09-16T15:46:27.000 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:26.993+0000 7fef52756000 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x560d4ac04e10 0x560d4acc7020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:46:27.000 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:26.993+0000 7fef52756000 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x560d4ac509d0 0x560d4acc7560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:46:27.000 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:26.993+0000 7fef52756000 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x560d4ac512f0 0x560d4acc7aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:46:27.000 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:26.993+0000 7fef52756000 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x560d4ab11730 con 0x560d4ac04e10 2024-09-16T15:46:27.000 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:26.993+0000 7fef52756000 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x560d4aa5a390 con 0x560d4ac512f0 2024-09-16T15:46:27.000 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:26.993+0000 7fef52756000 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x560d4aad6a60 con 0x560d4ac509d0 2024-09-16T15:46:27.000 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:26.993+0000 7fef52756000 10 client.0 _ll_register_callbacks cb 0x7fef40009580 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T15:46:27.001 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:26.993+0000 7fef4d408640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x560d4ac04e10 0x560d4acc7020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:46:27.001 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:26.993+0000 7fef4d408640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x560d4ac04e10 0x560d4acc7020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:54796/0 (socket says 172.21.15.123:54796) 2024-09-16T15:46:27.001 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:26.993+0000 7fef4d408640 1 -- 172.21.15.123:0/1144125433 learned_addr learned my addr 172.21.15.123:0/1144125433 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:46:27.001 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:26.993+0000 7fef4dc09640 1 --2- 172.21.15.123:0/1144125433 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x560d4ac512f0 0x560d4acc7aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:46:27.001 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:26.993+0000 7fef4cc07640 1 --2- 172.21.15.123:0/1144125433 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x560d4ac509d0 0x560d4acc7560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:46:27.001 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:26.993+0000 7fef4d408640 1 -- 172.21.15.123:0/1144125433 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x560d4ac509d0 msgr2=0x560d4acc7560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T15:46:27.001 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:26.993+0000 7fef4d408640 1 --2- 172.21.15.123:0/1144125433 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x560d4ac509d0 0x560d4acc7560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:27.001 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:26.993+0000 7fef4d408640 1 -- 172.21.15.123:0/1144125433 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x560d4ac512f0 msgr2=0x560d4acc7aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T15:46:27.001 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:26.993+0000 7fef4d408640 1 --2- 172.21.15.123:0/1144125433 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x560d4ac512f0 0x560d4acc7aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:27.001 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:26.993+0000 7fef4d408640 1 -- 172.21.15.123:0/1144125433 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560d4aa18a90 con 0x560d4ac04e10 2024-09-16T15:46:27.001 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:26.993+0000 7fef4cc07640 1 --2- 172.21.15.123:0/1144125433 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x560d4ac509d0 0x560d4acc7560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T15:46:27.001 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:26.993+0000 7fef4d408640 1 --2- 172.21.15.123:0/1144125433 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x560d4ac04e10 0x560d4acc7020 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7fef4400cdf0 tx=0x7fef44001010 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1dd6784adf8d86f0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:46:27.001 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:26.993+0000 7fef27fff640 10 client.0 ms_handle_connect on v2:172.21.15.123:3300/0 2024-09-16T15:46:27.002 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:26.993+0000 7fef27fff640 1 -- 172.21.15.123:0/1144125433 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef44077020 con 0x560d4ac04e10 2024-09-16T15:46:27.002 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:26.993+0000 7fef27fff640 1 -- 172.21.15.123:0/1144125433 <== mon.0 v2:172.21.15.123:3300/0 2 ==== osd_map(58..58 src has 1..58) ==== 5709+0+0 (secure 0 0 0) 0x7fef440918b0 con 0x560d4ac04e10 2024-09-16T15:46:27.002 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:26.993+0000 7fef52756000 1 -- 172.21.15.123:0/1144125433 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fef44062040 con 0x560d4ac04e10 2024-09-16T15:46:27.002 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:26.993+0000 7fef27fff640 1 -- 172.21.15.123:0/1144125433 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7fef44091ab0 con 0x560d4ac04e10 2024-09-16T15:46:27.002 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:26.993+0000 7fef52756000 10 client.5937 fetch_fsmap learned FSMap version 36 2024-09-16T15:46:27.002 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:26.993+0000 7fef52756000 1 -- 172.21.15.123:0/1144125433 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=2+}) -- 0x560d4a9f7050 con 0x560d4ac04e10 2024-09-16T15:46:27.002 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:26.993+0000 7fef27fff640 1 -- 172.21.15.123:0/1144125433 <== mon.0 v2:172.21.15.123:3300/0 4 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fef44062040 con 0x560d4ac04e10 2024-09-16T15:46:27.002 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:26.993+0000 7fef27fff640 1 -- 172.21.15.123:0/1144125433 <== mon.0 v2:172.21.15.123:3300/0 5 ==== fsmap(e 36) ==== 3502+0+0 (secure 0 0 0) 0x7fef4407f0b0 con 0x560d4ac04e10 2024-09-16T15:46:27.002 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:26.993+0000 7fef52756000 10 client.5937 fetch_fsmap finished waiting for FSMap version 36 2024-09-16T15:46:27.002 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:26.993+0000 7fef52756000 10 client.5937 resolve_mds: resolved name 'c' to daemon mds.c 2024-09-16T15:46:27.002 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:26.993+0000 7fef52756000 20 client.5937 populate_metadata read hostname 'smithi123' 2024-09-16T15:46:27.002 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:26.993+0000 7fef52756000 1 --2- 172.21.15.123:0/1144125433 >> [v2:172.21.15.123:6833/3589624203,v1:172.21.15.123:6835/3589624203] conn(0x560d4ac65420 0x560d4ac65800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:46:27.002 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:26.993+0000 7fef52756000 4 client.5937 mds_command: new command op to 5799 tid=0 multi_id=0 [{"prefix": "status"}] 2024-09-16T15:46:27.003 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:26.993+0000 7fef52756000 1 -- 172.21.15.123:0/1144125433 --> [v2:172.21.15.123:6833/3589624203,v1:172.21.15.123:6835/3589624203] -- command(tid 0: {"prefix": "status"}) -- 0x560d4aa60520 con 0x560d4ac65420 2024-09-16T15:46:27.003 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:26.993+0000 7fef4cc07640 1 --2- 172.21.15.123:0/1144125433 >> [v2:172.21.15.123:6833/3589624203,v1:172.21.15.123:6835/3589624203] conn(0x560d4ac65420 0x560d4ac65800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:46:27.003 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:26.993+0000 7fef27fff640 10 client.5937 ms_handle_connect on v2:172.21.15.123:6833/3589624203 2024-09-16T15:46:27.003 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:26.993+0000 7fef4cc07640 1 --2- 172.21.15.123:0/1144125433 >> [v2:172.21.15.123:6833/3589624203,v1:172.21.15.123:6835/3589624203] conn(0x560d4ac65420 0x560d4ac65800 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=30ee474f135ebf53 server_cookie=4311209f26a6165a in_seq=0 out_seq=0 2024-09-16T15:46:27.003 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:26.993+0000 7fef27fff640 1 -- 172.21.15.123:0/1144125433 <== mds.0 v2:172.21.15.123:6833/3589624203 1 ==== command_reply(tid 0: 0 ) ==== 8+0+325 (crc 0 0 0) 0x560d4aa60520 con 0x560d4ac65420 2024-09-16T15:46:27.003 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:26.993+0000 7fef27fff640 10 client.5937 handle_command_reply: tid=0 2024-09-16T15:46:27.003 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:26.993+0000 7fef52756000 1 client.5937 shutdown 2024-09-16T15:46:27.003 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:26.993+0000 7fef52756000 1 -- 172.21.15.123:0/1144125433 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x560d4ac04e10 msgr2=0x560d4acc7020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:46:27.003 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:26.993+0000 7fef52756000 1 --2- 172.21.15.123:0/1144125433 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x560d4ac04e10 0x560d4acc7020 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7fef4400cdf0 tx=0x7fef44001010 comp rx=0 tx=0).stop 2024-09-16T15:46:27.003 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:26.993+0000 7fef52756000 1 -- 172.21.15.123:0/1144125433 shutdown_connections 2024-09-16T15:46:27.003 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:26.993+0000 7fef52756000 1 --2- 172.21.15.123:0/1144125433 >> [v2:172.21.15.123:6833/3589624203,v1:172.21.15.123:6835/3589624203] conn(0x560d4ac65420 0x560d4ac65800 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:27.003 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:26.993+0000 7fef52756000 1 --2- 172.21.15.123:0/1144125433 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x560d4ac512f0 0x560d4acc7aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:27.003 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:26.993+0000 7fef52756000 1 --2- 172.21.15.123:0/1144125433 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x560d4ac509d0 0x560d4acc7560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:27.003 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:26.993+0000 7fef52756000 1 --2- 172.21.15.123:0/1144125433 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x560d4ac04e10 0x560d4acc7020 unknown :-1 s=CLOSED pgs=656 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:27.004 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:26.993+0000 7fef27fff640 0 client.5937 ms_handle_reset on v2:172.21.15.123:6833/3589624203 2024-09-16T15:46:27.004 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:26.993+0000 7fef52756000 1 -- 172.21.15.123:0/1144125433 >> 172.21.15.123:0/1144125433 conn(0x560d4aaee130 msgr2=0x560d4ac4f5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:46:27.004 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:26.993+0000 7fef52756000 1 -- 172.21.15.123:0/1144125433 shutdown_connections 2024-09-16T15:46:27.004 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:26.993+0000 7fef52756000 1 -- 172.21.15.123:0/1144125433 wait complete. 2024-09-16T15:46:27.004 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:26.993+0000 7fef52756000 20 client.5937 trim_cache size 0 max 16384 2024-09-16T15:46:27.004 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2024-09-16T15:46:27.004 INFO:teuthology.orchestra.run.smithi123.stdout: "cluster_fsid": "56077f89-18c8-486f-ae7e-eecc0bb7822c", 2024-09-16T15:46:27.004 INFO:teuthology.orchestra.run.smithi123.stdout: "whoami": 0, 2024-09-16T15:46:27.004 INFO:teuthology.orchestra.run.smithi123.stdout: "id": 5799, 2024-09-16T15:46:27.004 INFO:teuthology.orchestra.run.smithi123.stdout: "want_state": "up:active", 2024-09-16T15:46:27.005 INFO:teuthology.orchestra.run.smithi123.stdout: "state": "up:active", 2024-09-16T15:46:27.005 INFO:teuthology.orchestra.run.smithi123.stdout: "fs_name": "cephfs", 2024-09-16T15:46:27.005 INFO:teuthology.orchestra.run.smithi123.stdout: "rank_uptime": 5.8567181589999997, 2024-09-16T15:46:27.005 INFO:teuthology.orchestra.run.smithi123.stdout: "mdsmap_epoch": 36, 2024-09-16T15:46:27.005 INFO:teuthology.orchestra.run.smithi123.stdout: "osdmap_epoch": 58, 2024-09-16T15:46:27.005 INFO:teuthology.orchestra.run.smithi123.stdout: "osdmap_epoch_barrier": 58, 2024-09-16T15:46:27.005 INFO:teuthology.orchestra.run.smithi123.stdout: "uptime": 11.193088954 2024-09-16T15:46:27.005 INFO:teuthology.orchestra.run.smithi123.stdout:} 2024-09-16T15:46:27.006 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:26.993+0000 7fef4f66c640 1 -- 172.21.15.123:0/1427987952 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fef2c073070 msgr2=0x7fef2c075530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:46:27.006 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:26.993+0000 7fef4f66c640 1 --2- 172.21.15.123:0/1427987952 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fef2c073070 0x7fef2c075530 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7fef380af000 tx=0x7fef3801af10 comp rx=0 tx=0).stop 2024-09-16T15:46:27.006 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:26.993+0000 7fef4f66c640 1 -- 172.21.15.123:0/1427987952 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fef480b0a50 msgr2=0x7fef481554d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:46:27.006 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:26.993+0000 7fef4f66c640 1 --2- 172.21.15.123:0/1427987952 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fef480b0a50 0x7fef481554d0 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7fef4400c430 tx=0x7fef4400c8f0 comp rx=0 tx=0).stop 2024-09-16T15:46:27.006 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:26.993+0000 7fef4f66c640 1 -- 172.21.15.123:0/1427987952 shutdown_connections 2024-09-16T15:46:27.006 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:26.993+0000 7fef4f66c640 1 --2- 172.21.15.123:0/1427987952 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fef2c073070 0x7fef2c075530 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:27.006 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:26.993+0000 7fef4f66c640 1 --2- 172.21.15.123:0/1427987952 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fef48161620 0x7fef4815c200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:27.006 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:26.993+0000 7fef4f66c640 1 --2- 172.21.15.123:0/1427987952 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fef480b2b00 0x7fef4815bcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:27.007 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:26.993+0000 7fef4f66c640 1 --2- 172.21.15.123:0/1427987952 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fef480b0a50 0x7fef481554d0 unknown :-1 s=CLOSED pgs=652 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:27.007 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:26.993+0000 7fef4f66c640 1 -- 172.21.15.123:0/1427987952 >> 172.21.15.123:0/1427987952 conn(0x7fef480d3ea0 msgr2=0x7fef480d1da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:46:27.007 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:26.993+0000 7fef4f66c640 1 -- 172.21.15.123:0/1427987952 shutdown_connections 2024-09-16T15:46:27.007 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:26.993+0000 7fef4f66c640 1 -- 172.21.15.123:0/1427987952 wait complete. 2024-09-16T15:46:27.014 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-09-16T15:46:27.123 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:27.121+0000 7f1d31c47640 1 -- 172.21.15.123:0/3967050785 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1d2c15add0 msgr2=0x7f1d2c0cf9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:46:27.123 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:27.121+0000 7f1d31c47640 1 --2- 172.21.15.123:0/3967050785 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1d2c15add0 0x7f1d2c0cf9d0 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7f1d20009900 tx=0x7f1d2001c8e0 comp rx=0 tx=0).stop 2024-09-16T15:46:27.123 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:27.121+0000 7f1d31c47640 1 -- 172.21.15.123:0/3967050785 shutdown_connections 2024-09-16T15:46:27.123 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:27.121+0000 7f1d31c47640 1 --2- 172.21.15.123:0/3967050785 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f1d2c073d70 0x7f1d2c06fa00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:27.123 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:27.121+0000 7f1d31c47640 1 --2- 172.21.15.123:0/3967050785 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1d2c15add0 0x7f1d2c0cf9d0 unknown :-1 s=CLOSED pgs=657 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:27.123 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:27.121+0000 7f1d31c47640 1 -- 172.21.15.123:0/3967050785 >> 172.21.15.123:0/3967050785 conn(0x7f1d2c0d0b40 msgr2=0x7f1d2c0d0f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:46:27.124 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:27.121+0000 7f1d31c47640 1 -- 172.21.15.123:0/3967050785 shutdown_connections 2024-09-16T15:46:27.124 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:27.121+0000 7f1d31c47640 1 -- 172.21.15.123:0/3967050785 wait complete. 2024-09-16T15:46:27.124 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:27.121+0000 7f1d31c47640 1 Processor -- start 2024-09-16T15:46:27.124 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:27.121+0000 7f1d31c47640 1 -- start start 2024-09-16T15:46:27.126 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:27.121+0000 7f1d31c47640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1d2c073d70 0x7f1d2c1582d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:46:27.126 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:27.121+0000 7f1d31c47640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f1d2c0cff10 0x7f1d2c15df30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:46:27.126 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:27.121+0000 7f1d31c47640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1d2c15add0 0x7f1d2c0c9de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:46:27.126 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:27.121+0000 7f1d31c47640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f1d2c167a60 con 0x7f1d2c15add0 2024-09-16T15:46:27.126 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:27.121+0000 7f1d31c47640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f1d2c167be0 con 0x7f1d2c073d70 2024-09-16T15:46:27.126 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:27.121+0000 7f1d31c47640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f1d2c167d60 con 0x7f1d2c0cff10 2024-09-16T15:46:27.126 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:27.121+0000 7f1d2bfff640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1d2c15add0 0x7f1d2c0c9de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:46:27.126 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:27.121+0000 7f1d2bfff640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1d2c15add0 0x7f1d2c0c9de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:54826/0 (socket says 172.21.15.123:54826) 2024-09-16T15:46:27.126 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:27.121+0000 7f1d2bfff640 1 -- 172.21.15.123:0/3020905448 learned_addr learned my addr 172.21.15.123:0/3020905448 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:46:27.126 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:27.121+0000 7f1d2affd640 1 --2- 172.21.15.123:0/3020905448 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f1d2c0cff10 0x7f1d2c15df30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:46:27.126 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:27.121+0000 7f1d2b7fe640 1 --2- 172.21.15.123:0/3020905448 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1d2c073d70 0x7f1d2c1582d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:46:27.127 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:27.121+0000 7f1d2bfff640 1 -- 172.21.15.123:0/3020905448 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f1d2c0cff10 msgr2=0x7f1d2c15df30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:46:27.127 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:27.121+0000 7f1d2bfff640 1 --2- 172.21.15.123:0/3020905448 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f1d2c0cff10 0x7f1d2c15df30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:27.127 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:27.121+0000 7f1d2bfff640 1 -- 172.21.15.123:0/3020905448 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1d2c073d70 msgr2=0x7f1d2c1582d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:46:27.127 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:27.121+0000 7f1d2bfff640 1 --2- 172.21.15.123:0/3020905448 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1d2c073d70 0x7f1d2c1582d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:27.127 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:27.121+0000 7f1d2bfff640 1 -- 172.21.15.123:0/3020905448 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1d2c158810 con 0x7f1d2c15add0 2024-09-16T15:46:27.127 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:27.121+0000 7f1d2bfff640 1 --2- 172.21.15.123:0/3020905448 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1d2c15add0 0x7f1d2c0c9de0 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7f1d2400ba10 tx=0x7f1d2400bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:46:27.127 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:27.121+0000 7f1d13fff640 1 -- 172.21.15.123:0/3020905448 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d24019070 con 0x7f1d2c15add0 2024-09-16T15:46:27.127 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:27.121+0000 7f1d13fff640 1 -- 172.21.15.123:0/3020905448 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f1d2400cc50 con 0x7f1d2c15add0 2024-09-16T15:46:27.127 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:27.121+0000 7f1d31c47640 1 -- 172.21.15.123:0/3020905448 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1d2c162c40 con 0x7f1d2c15add0 2024-09-16T15:46:27.127 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:27.121+0000 7f1d13fff640 1 -- 172.21.15.123:0/3020905448 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d240146a0 con 0x7f1d2c15add0 2024-09-16T15:46:27.127 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:27.121+0000 7f1d31c47640 1 -- 172.21.15.123:0/3020905448 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1d2c1631c0 con 0x7f1d2c15add0 2024-09-16T15:46:27.128 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:27.121+0000 7f1d31c47640 1 -- 172.21.15.123:0/3020905448 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1d2c163980 con 0x7f1d2c15add0 2024-09-16T15:46:27.128 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:27.125+0000 7f1d13fff640 1 -- 172.21.15.123:0/3020905448 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 9) ==== 97233+0+0 (secure 0 0 0) 0x7f1d2401c070 con 0x7f1d2c15add0 2024-09-16T15:46:27.128 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:27.125+0000 7f1d13fff640 1 --2- 172.21.15.123:0/3020905448 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f1d18073070 0x7f1d18075530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:46:27.128 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:27.125+0000 7f1d13fff640 1 -- 172.21.15.123:0/3020905448 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(58..58 src has 1..58) ==== 5709+0+0 (secure 0 0 0) 0x7f1d240942e0 con 0x7f1d2c15add0 2024-09-16T15:46:27.128 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:27.125+0000 7f1d2b7fe640 1 --2- 172.21.15.123:0/3020905448 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f1d18073070 0x7f1d18075530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:46:27.129 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:27.125+0000 7f1d2b7fe640 1 --2- 172.21.15.123:0/3020905448 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f1d18073070 0x7f1d18075530 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f1d200af000 tx=0x7f1d2001b120 comp rx=0 tx=0).ready entity=mgr.4135 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:46:27.131 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:27.129+0000 7f1d13fff640 1 -- 172.21.15.123:0/3020905448 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f1d2c163980 con 0x7f1d2c15add0 2024-09-16T15:46:27.325 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:27.321+0000 7f1d31c47640 1 -- 172.21.15.123:0/3020905448 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f1d2c06fa00 con 0x7f1d2c15add0 2024-09-16T15:46:27.326 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:27.321+0000 7f1d13fff640 1 -- 172.21.15.123:0/3020905448 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v36) ==== 53+0+73 (secure 0 0 0) 0x7f1d24023020 con 0x7f1d2c15add0 2024-09-16T15:46:27.326 INFO:teuthology.orchestra.run.smithi123.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-09-16T15:46:27.328 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:27.325+0000 7f1d31c47640 1 -- 172.21.15.123:0/3020905448 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f1d18073070 msgr2=0x7f1d18075530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:46:27.328 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:27.325+0000 7f1d31c47640 1 --2- 172.21.15.123:0/3020905448 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f1d18073070 0x7f1d18075530 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f1d200af000 tx=0x7f1d2001b120 comp rx=0 tx=0).stop 2024-09-16T15:46:27.328 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:27.325+0000 7f1d31c47640 1 -- 172.21.15.123:0/3020905448 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1d2c15add0 msgr2=0x7f1d2c0c9de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:46:27.329 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:27.325+0000 7f1d31c47640 1 --2- 172.21.15.123:0/3020905448 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1d2c15add0 0x7f1d2c0c9de0 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7f1d2400ba10 tx=0x7f1d2400bed0 comp rx=0 tx=0).stop 2024-09-16T15:46:27.329 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:27.325+0000 7f1d31c47640 1 -- 172.21.15.123:0/3020905448 shutdown_connections 2024-09-16T15:46:27.329 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:27.325+0000 7f1d31c47640 1 --2- 172.21.15.123:0/3020905448 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f1d18073070 0x7f1d18075530 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:27.329 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:27.325+0000 7f1d31c47640 1 --2- 172.21.15.123:0/3020905448 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1d2c15add0 0x7f1d2c0c9de0 unknown :-1 s=CLOSED pgs=658 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:27.329 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:27.325+0000 7f1d31c47640 1 --2- 172.21.15.123:0/3020905448 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f1d2c0cff10 0x7f1d2c15df30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:27.329 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:27.325+0000 7f1d31c47640 1 --2- 172.21.15.123:0/3020905448 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1d2c073d70 0x7f1d2c1582d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:27.329 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:27.325+0000 7f1d31c47640 1 -- 172.21.15.123:0/3020905448 >> 172.21.15.123:0/3020905448 conn(0x7f1d2c0d0b40 msgr2=0x7f1d2c0cf160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:46:27.329 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:27.325+0000 7f1d31c47640 1 -- 172.21.15.123:0/3020905448 shutdown_connections 2024-09-16T15:46:27.329 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:27.325+0000 7f1d31c47640 1 -- 172.21.15.123:0/3020905448 wait complete. 2024-09-16T15:46:27.343 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-09-16T15:46:27.343 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-09-16T15:46:27.343 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi123.front.sepia.ceph.com 2024-09-16T15:46:27.343 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-09-16T15:46:27.344 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-09-16T15:46:27.344 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-09-16T15:46:27.344 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-09-16T15:46:27.344 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-09-16T15:46:27.344 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-09-16T15:46:27.344 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:46:27.344 DEBUG:teuthology.orchestra.run.smithi123:> ip addr 2024-09-16T15:46:27.350 INFO:teuthology.orchestra.run.smithi123.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-09-16T15:46:27.350 INFO:teuthology.orchestra.run.smithi123.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-09-16T15:46:27.350 INFO:teuthology.orchestra.run.smithi123.stdout: inet 127.0.0.1/8 scope host lo 2024-09-16T15:46:27.350 INFO:teuthology.orchestra.run.smithi123.stdout: valid_lft forever preferred_lft forever 2024-09-16T15:46:27.350 INFO:teuthology.orchestra.run.smithi123.stdout: inet6 ::1/128 scope host 2024-09-16T15:46:27.350 INFO:teuthology.orchestra.run.smithi123.stdout: valid_lft forever preferred_lft forever 2024-09-16T15:46:27.351 INFO:teuthology.orchestra.run.smithi123.stdout:2: eno1: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-09-16T15:46:27.351 INFO:teuthology.orchestra.run.smithi123.stdout: link/ether 0c:c4:7a:d8:da:6c brd ff:ff:ff:ff:ff:ff 2024-09-16T15:46:27.351 INFO:teuthology.orchestra.run.smithi123.stdout: altname enp7s0f0 2024-09-16T15:46:27.351 INFO:teuthology.orchestra.run.smithi123.stdout:3: eno2: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-09-16T15:46:27.351 INFO:teuthology.orchestra.run.smithi123.stdout: link/ether 0c:c4:7a:d8:da:6d brd ff:ff:ff:ff:ff:ff 2024-09-16T15:46:27.351 INFO:teuthology.orchestra.run.smithi123.stdout: altname enp7s0f1 2024-09-16T15:46:27.351 INFO:teuthology.orchestra.run.smithi123.stdout:4: enp3s0f0: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-09-16T15:46:27.351 INFO:teuthology.orchestra.run.smithi123.stdout: link/ether 0c:c4:7a:8f:d3:f0 brd ff:ff:ff:ff:ff:ff 2024-09-16T15:46:27.351 INFO:teuthology.orchestra.run.smithi123.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-09-16T15:46:27.351 INFO:teuthology.orchestra.run.smithi123.stdout: link/ether 0c:c4:7a:8f:d3:f1 brd ff:ff:ff:ff:ff:ff 2024-09-16T15:46:27.351 INFO:teuthology.orchestra.run.smithi123.stdout: inet 172.21.15.123/20 metric 100 brd 172.21.15.255 scope global dynamic enp3s0f1 2024-09-16T15:46:27.351 INFO:teuthology.orchestra.run.smithi123.stdout: valid_lft 42281sec preferred_lft 42281sec 2024-09-16T15:46:27.351 INFO:teuthology.orchestra.run.smithi123.stdout: inet6 fe80::ec4:7aff:fe8f:d3f1/64 scope link 2024-09-16T15:46:27.351 INFO:teuthology.orchestra.run.smithi123.stdout: valid_lft forever preferred_lft forever 2024-09-16T15:46:27.351 INFO:teuthology.orchestra.run.smithi123.stdout:6: docker0: mtu 1500 qdisc noqueue state DOWN group default 2024-09-16T15:46:27.352 INFO:teuthology.orchestra.run.smithi123.stdout: link/ether 02:42:dd:82:a2:6e brd ff:ff:ff:ff:ff:ff 2024-09-16T15:46:27.352 INFO:teuthology.orchestra.run.smithi123.stdout: inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0 2024-09-16T15:46:27.352 INFO:teuthology.orchestra.run.smithi123.stdout: valid_lft forever preferred_lft forever 2024-09-16T15:46:27.352 INFO:teuthology.orchestra.run.smithi123.stdout:7: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-09-16T15:46:27.352 INFO:teuthology.orchestra.run.smithi123.stdout: link/ether 5e:1f:84:6d:79:ae brd ff:ff:ff:ff:ff:ff 2024-09-16T15:46:27.352 INFO:teuthology.orchestra.run.smithi123.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-09-16T15:46:27.352 INFO:teuthology.orchestra.run.smithi123.stdout: valid_lft forever preferred_lft forever 2024-09-16T15:46:27.352 INFO:teuthology.orchestra.run.smithi123.stdout: inet6 fe80::5c1f:84ff:fe6d:79ae/64 scope link 2024-09-16T15:46:27.352 INFO:teuthology.orchestra.run.smithi123.stdout: valid_lft forever preferred_lft forever 2024-09-16T15:46:27.352 INFO:teuthology.orchestra.run.smithi123.stdout:8: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-09-16T15:46:27.352 INFO:teuthology.orchestra.run.smithi123.stdout: link/ether 0e:5c:f6:1f:82:2f brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-09-16T15:46:27.352 INFO:teuthology.orchestra.run.smithi123.stdout: inet6 fe80::c5c:f6ff:fe1f:822f/64 scope link 2024-09-16T15:46:27.352 INFO:teuthology.orchestra.run.smithi123.stdout: valid_lft forever preferred_lft forever 2024-09-16T15:46:27.352 INFO:teuthology.orchestra.run.smithi123.stdout:9: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-09-16T15:46:27.353 INFO:teuthology.orchestra.run.smithi123.stdout: link/ether c6:a6:3d:69:d0:db brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-09-16T15:46:27.353 INFO:teuthology.orchestra.run.smithi123.stdout: inet6 fe80::c4a6:3dff:fe69:d0db/64 scope link 2024-09-16T15:46:27.353 INFO:teuthology.orchestra.run.smithi123.stdout: valid_lft forever preferred_lft forever 2024-09-16T15:46:27.353 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:46:27.353 DEBUG:teuthology.orchestra.run.smithi123:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-09-16T15:46:27.403 INFO:teuthology.orchestra.run.smithi123.stdout:1 2024-09-16T15:46:27.404 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:46:27.405 DEBUG:teuthology.orchestra.run.smithi123:> route 2024-09-16T15:46:27.454 INFO:teuthology.orchestra.run.smithi123.stdout:Kernel IP routing table 2024-09-16T15:46:27.454 INFO:teuthology.orchestra.run.smithi123.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-09-16T15:46:27.454 INFO:teuthology.orchestra.run.smithi123.stdout:default _gateway 0.0.0.0 UG 0 0 0 enp3s0f1 2024-09-16T15:46:27.454 INFO:teuthology.orchestra.run.smithi123.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-09-16T15:46:27.454 INFO:teuthology.orchestra.run.smithi123.stdout:158.69.67.47 _gateway 255.255.255.255 UGH 100 0 0 enp3s0f1 2024-09-16T15:46:27.454 INFO:teuthology.orchestra.run.smithi123.stdout:172.17.0.0 0.0.0.0 255.255.0.0 U 0 0 0 docker0 2024-09-16T15:46:27.454 INFO:teuthology.orchestra.run.smithi123.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-09-16T15:46:27.454 INFO:teuthology.orchestra.run.smithi123.stdout:ns1.front.sepia 0.0.0.0 255.255.255.255 UH 100 0 0 enp3s0f1 2024-09-16T15:46:27.454 INFO:teuthology.orchestra.run.smithi123.stdout:172.21.0.2 0.0.0.0 255.255.255.255 UH 100 0 0 enp3s0f1 2024-09-16T15:46:27.455 INFO:teuthology.orchestra.run.smithi123.stdout:_gateway 0.0.0.0 255.255.255.255 UH 100 0 0 enp3s0f1 2024-09-16T15:46:27.455 INFO:teuthology.orchestra.run.smithi123.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-09-16T15:46:27.455 DEBUG:teuthology.orchestra.run.smithi123:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-09-16T15:46:27.455 DEBUG:teuthology.orchestra.run.smithi123:> set -e 2024-09-16T15:46:27.455 DEBUG:teuthology.orchestra.run.smithi123:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-09-16T15:46:27.455 DEBUG:teuthology.orchestra.run.smithi123:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-09-16T15:46:27.455 DEBUG:teuthology.orchestra.run.smithi123:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-09-16T15:46:27.455 DEBUG:teuthology.orchestra.run.smithi123:> ') 2024-09-16T15:46:27.537 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:27 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-09-16T15:46:27.567 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:27 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-16T15:46:27.574 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:46:27.574 DEBUG:teuthology.orchestra.run.smithi123:> ip netns list 2024-09-16T15:46:27.619 INFO:teuthology.orchestra.run.smithi123.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-09-16T15:46:27.619 INFO:teuthology.orchestra.run.smithi123.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-09-16T15:46:27.619 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-09-16T15:46:27.619 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:46:27.619 DEBUG:teuthology.orchestra.run.smithi123:> sudo ip link set brx.0 up 2024-09-16T15:46:27.677 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2024-09-16T15:46:27.677 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-09-16T15:46:27.677 DEBUG:teuthology.orchestra.run.smithi123:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-09-16T15:46:27.727 INFO:teuthology.orchestra.run.smithi123.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-09-16T15:46:27.727 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-09-16T15:46:27.727 DEBUG:teuthology.orchestra.run.smithi123:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-09-16T15:46:27.775 DEBUG:teuthology.orchestra.run.smithi123:> sudo modprobe fuse 2024-09-16T15:46:27.833 DEBUG:teuthology.orchestra.run.smithi123:> cat /proc/self/mounts | awk '{print $2}' 2024-09-16T15:46:27.882 INFO:teuthology.orchestra.run.smithi123.stdout:/sys 2024-09-16T15:46:27.882 INFO:teuthology.orchestra.run.smithi123.stdout:/proc 2024-09-16T15:46:27.882 INFO:teuthology.orchestra.run.smithi123.stdout:/dev 2024-09-16T15:46:27.882 INFO:teuthology.orchestra.run.smithi123.stdout:/dev/pts 2024-09-16T15:46:27.882 INFO:teuthology.orchestra.run.smithi123.stdout:/run 2024-09-16T15:46:27.882 INFO:teuthology.orchestra.run.smithi123.stdout:/ 2024-09-16T15:46:27.883 INFO:teuthology.orchestra.run.smithi123.stdout:/sys/kernel/security 2024-09-16T15:46:27.883 INFO:teuthology.orchestra.run.smithi123.stdout:/dev/shm 2024-09-16T15:46:27.883 INFO:teuthology.orchestra.run.smithi123.stdout:/run/lock 2024-09-16T15:46:27.883 INFO:teuthology.orchestra.run.smithi123.stdout:/sys/fs/cgroup 2024-09-16T15:46:27.883 INFO:teuthology.orchestra.run.smithi123.stdout:/sys/fs/pstore 2024-09-16T15:46:27.883 INFO:teuthology.orchestra.run.smithi123.stdout:/sys/fs/bpf 2024-09-16T15:46:27.883 INFO:teuthology.orchestra.run.smithi123.stdout:/proc/sys/fs/binfmt_misc 2024-09-16T15:46:27.883 INFO:teuthology.orchestra.run.smithi123.stdout:/dev/hugepages 2024-09-16T15:46:27.883 INFO:teuthology.orchestra.run.smithi123.stdout:/dev/mqueue 2024-09-16T15:46:27.883 INFO:teuthology.orchestra.run.smithi123.stdout:/sys/kernel/debug 2024-09-16T15:46:27.883 INFO:teuthology.orchestra.run.smithi123.stdout:/sys/kernel/tracing 2024-09-16T15:46:27.883 INFO:teuthology.orchestra.run.smithi123.stdout:/proc/fs/nfsd 2024-09-16T15:46:27.883 INFO:teuthology.orchestra.run.smithi123.stdout:/sys/fs/fuse/connections 2024-09-16T15:46:27.883 INFO:teuthology.orchestra.run.smithi123.stdout:/sys/kernel/config 2024-09-16T15:46:27.884 INFO:teuthology.orchestra.run.smithi123.stdout:/run/credentials/systemd-sysusers.service 2024-09-16T15:46:27.884 INFO:teuthology.orchestra.run.smithi123.stdout:/run/rpc_pipefs 2024-09-16T15:46:27.884 INFO:teuthology.orchestra.run.smithi123.stdout:/run/user/1000 2024-09-16T15:46:27.884 INFO:teuthology.orchestra.run.smithi123.stdout:/var/lib/ceph 2024-09-16T15:46:27.884 INFO:teuthology.orchestra.run.smithi123.stdout:/var/lib/ceph/osd/ceph-0 2024-09-16T15:46:27.884 INFO:teuthology.orchestra.run.smithi123.stdout:/var/lib/ceph/osd/ceph-1 2024-09-16T15:46:27.884 INFO:teuthology.orchestra.run.smithi123.stdout:/var/lib/ceph/osd/ceph-2 2024-09-16T15:46:27.884 INFO:teuthology.orchestra.run.smithi123.stdout:/var/lib/ceph/osd/ceph-3 2024-09-16T15:46:27.884 INFO:teuthology.orchestra.run.smithi123.stdout:/run/netns 2024-09-16T15:46:27.884 INFO:teuthology.orchestra.run.smithi123.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-09-16T15:46:27.884 INFO:teuthology.orchestra.run.smithi123.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-09-16T15:46:27.884 INFO:teuthology.orchestra.run.smithi123.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-09-16T15:46:27.884 INFO:teuthology.orchestra.run.smithi123.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-09-16T15:46:27.885 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:46:27.885 DEBUG:teuthology.orchestra.run.smithi123:> ls /sys/fs/fuse/connections 2024-09-16T15:46:27.935 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-09-16T15:46:27.936 DEBUG:teuthology.orchestra.run.smithi123:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2024-09-16T15:46:27.980 DEBUG:teuthology.orchestra.run.smithi123:> sudo modprobe fuse 2024-09-16T15:46:27.994 DEBUG:teuthology.orchestra.run.smithi123:> cat /proc/self/mounts | awk '{print $2}' 2024-09-16T15:46:28.040 INFO:teuthology.orchestra.run.smithi123.stdout:/sys 2024-09-16T15:46:28.040 INFO:teuthology.orchestra.run.smithi123.stdout:/proc 2024-09-16T15:46:28.040 INFO:teuthology.orchestra.run.smithi123.stdout:/dev 2024-09-16T15:46:28.040 INFO:teuthology.orchestra.run.smithi123.stdout:/dev/pts 2024-09-16T15:46:28.040 INFO:teuthology.orchestra.run.smithi123.stdout:/run 2024-09-16T15:46:28.040 INFO:teuthology.orchestra.run.smithi123.stdout:/ 2024-09-16T15:46:28.040 INFO:teuthology.orchestra.run.smithi123.stdout:/sys/kernel/security 2024-09-16T15:46:28.041 INFO:teuthology.orchestra.run.smithi123.stdout:/dev/shm 2024-09-16T15:46:28.041 INFO:teuthology.orchestra.run.smithi123.stdout:/run/lock 2024-09-16T15:46:28.041 INFO:teuthology.orchestra.run.smithi123.stdout:/sys/fs/cgroup 2024-09-16T15:46:28.041 INFO:teuthology.orchestra.run.smithi123.stdout:/sys/fs/pstore 2024-09-16T15:46:28.041 INFO:teuthology.orchestra.run.smithi123.stdout:/sys/fs/bpf 2024-09-16T15:46:28.041 INFO:teuthology.orchestra.run.smithi123.stdout:/proc/sys/fs/binfmt_misc 2024-09-16T15:46:28.041 INFO:teuthology.orchestra.run.smithi123.stdout:/dev/hugepages 2024-09-16T15:46:28.041 INFO:teuthology.orchestra.run.smithi123.stdout:/dev/mqueue 2024-09-16T15:46:28.041 INFO:teuthology.orchestra.run.smithi123.stdout:/sys/kernel/debug 2024-09-16T15:46:28.041 INFO:teuthology.orchestra.run.smithi123.stdout:/sys/kernel/tracing 2024-09-16T15:46:28.041 INFO:teuthology.orchestra.run.smithi123.stdout:/proc/fs/nfsd 2024-09-16T15:46:28.041 INFO:teuthology.orchestra.run.smithi123.stdout:/sys/fs/fuse/connections 2024-09-16T15:46:28.041 INFO:teuthology.orchestra.run.smithi123.stdout:/sys/kernel/config 2024-09-16T15:46:28.041 INFO:teuthology.orchestra.run.smithi123.stdout:/run/credentials/systemd-sysusers.service 2024-09-16T15:46:28.042 INFO:teuthology.orchestra.run.smithi123.stdout:/run/rpc_pipefs 2024-09-16T15:46:28.042 INFO:teuthology.orchestra.run.smithi123.stdout:/run/user/1000 2024-09-16T15:46:28.042 INFO:teuthology.orchestra.run.smithi123.stdout:/var/lib/ceph 2024-09-16T15:46:28.042 INFO:teuthology.orchestra.run.smithi123.stdout:/var/lib/ceph/osd/ceph-0 2024-09-16T15:46:28.042 INFO:teuthology.orchestra.run.smithi123.stdout:/var/lib/ceph/osd/ceph-1 2024-09-16T15:46:28.042 INFO:teuthology.orchestra.run.smithi123.stdout:/var/lib/ceph/osd/ceph-2 2024-09-16T15:46:28.042 INFO:teuthology.orchestra.run.smithi123.stdout:/var/lib/ceph/osd/ceph-3 2024-09-16T15:46:28.042 INFO:teuthology.orchestra.run.smithi123.stdout:/run/netns 2024-09-16T15:46:28.042 INFO:teuthology.orchestra.run.smithi123.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-09-16T15:46:28.042 INFO:teuthology.orchestra.run.smithi123.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-09-16T15:46:28.042 INFO:teuthology.orchestra.run.smithi123.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-09-16T15:46:28.042 INFO:teuthology.orchestra.run.smithi123.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-09-16T15:46:28.043 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:46:28.043 DEBUG:teuthology.orchestra.run.smithi123:> ls /sys/fs/fuse/connections 2024-09-16T15:46:28.061 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi123.stderr:2024-09-16T15:46:28.057+0000 7fd544516580 -1 init, newargv = 0x55a98a66ea30 newargc=15 2024-09-16T15:46:28.061 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi123.stderr:ceph-fuse[41267]: starting ceph client 2024-09-16T15:46:28.067 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi123.stderr:ceph-fuse[41267]: starting fuse 2024-09-16T15:46:29.167 DEBUG:teuthology.orchestra.run.smithi123:> sudo modprobe fuse 2024-09-16T15:46:29.181 DEBUG:teuthology.orchestra.run.smithi123:> cat /proc/self/mounts | awk '{print $2}' 2024-09-16T15:46:29.230 INFO:teuthology.orchestra.run.smithi123.stdout:/sys 2024-09-16T15:46:29.230 INFO:teuthology.orchestra.run.smithi123.stdout:/proc 2024-09-16T15:46:29.230 INFO:teuthology.orchestra.run.smithi123.stdout:/dev 2024-09-16T15:46:29.230 INFO:teuthology.orchestra.run.smithi123.stdout:/dev/pts 2024-09-16T15:46:29.230 INFO:teuthology.orchestra.run.smithi123.stdout:/run 2024-09-16T15:46:29.230 INFO:teuthology.orchestra.run.smithi123.stdout:/ 2024-09-16T15:46:29.230 INFO:teuthology.orchestra.run.smithi123.stdout:/sys/kernel/security 2024-09-16T15:46:29.230 INFO:teuthology.orchestra.run.smithi123.stdout:/dev/shm 2024-09-16T15:46:29.230 INFO:teuthology.orchestra.run.smithi123.stdout:/run/lock 2024-09-16T15:46:29.231 INFO:teuthology.orchestra.run.smithi123.stdout:/sys/fs/cgroup 2024-09-16T15:46:29.231 INFO:teuthology.orchestra.run.smithi123.stdout:/sys/fs/pstore 2024-09-16T15:46:29.231 INFO:teuthology.orchestra.run.smithi123.stdout:/sys/fs/bpf 2024-09-16T15:46:29.231 INFO:teuthology.orchestra.run.smithi123.stdout:/proc/sys/fs/binfmt_misc 2024-09-16T15:46:29.231 INFO:teuthology.orchestra.run.smithi123.stdout:/dev/hugepages 2024-09-16T15:46:29.231 INFO:teuthology.orchestra.run.smithi123.stdout:/dev/mqueue 2024-09-16T15:46:29.231 INFO:teuthology.orchestra.run.smithi123.stdout:/sys/kernel/debug 2024-09-16T15:46:29.231 INFO:teuthology.orchestra.run.smithi123.stdout:/sys/kernel/tracing 2024-09-16T15:46:29.231 INFO:teuthology.orchestra.run.smithi123.stdout:/proc/fs/nfsd 2024-09-16T15:46:29.231 INFO:teuthology.orchestra.run.smithi123.stdout:/sys/fs/fuse/connections 2024-09-16T15:46:29.231 INFO:teuthology.orchestra.run.smithi123.stdout:/sys/kernel/config 2024-09-16T15:46:29.231 INFO:teuthology.orchestra.run.smithi123.stdout:/run/credentials/systemd-sysusers.service 2024-09-16T15:46:29.231 INFO:teuthology.orchestra.run.smithi123.stdout:/run/rpc_pipefs 2024-09-16T15:46:29.232 INFO:teuthology.orchestra.run.smithi123.stdout:/run/user/1000 2024-09-16T15:46:29.232 INFO:teuthology.orchestra.run.smithi123.stdout:/var/lib/ceph 2024-09-16T15:46:29.232 INFO:teuthology.orchestra.run.smithi123.stdout:/var/lib/ceph/osd/ceph-0 2024-09-16T15:46:29.232 INFO:teuthology.orchestra.run.smithi123.stdout:/var/lib/ceph/osd/ceph-1 2024-09-16T15:46:29.232 INFO:teuthology.orchestra.run.smithi123.stdout:/var/lib/ceph/osd/ceph-2 2024-09-16T15:46:29.232 INFO:teuthology.orchestra.run.smithi123.stdout:/var/lib/ceph/osd/ceph-3 2024-09-16T15:46:29.232 INFO:teuthology.orchestra.run.smithi123.stdout:/run/netns 2024-09-16T15:46:29.232 INFO:teuthology.orchestra.run.smithi123.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-09-16T15:46:29.232 INFO:teuthology.orchestra.run.smithi123.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-09-16T15:46:29.232 INFO:teuthology.orchestra.run.smithi123.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-09-16T15:46:29.232 INFO:teuthology.orchestra.run.smithi123.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-09-16T15:46:29.232 INFO:teuthology.orchestra.run.smithi123.stdout:/home/ubuntu/cephtest/mnt.0 2024-09-16T15:46:29.233 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:46:29.233 DEBUG:teuthology.orchestra.run.smithi123:> ls /sys/fs/fuse/connections 2024-09-16T15:46:29.280 INFO:teuthology.orchestra.run.smithi123.stdout:45 2024-09-16T15:46:29.280 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [45] 2024-09-16T15:46:29.280 DEBUG:teuthology.orchestra.run.smithi123:> sudo stdin-killer -- python3 -c ' 2024-09-16T15:46:29.280 DEBUG:teuthology.orchestra.run.smithi123:> import glob 2024-09-16T15:46:29.280 DEBUG:teuthology.orchestra.run.smithi123:> import re 2024-09-16T15:46:29.280 DEBUG:teuthology.orchestra.run.smithi123:> import os 2024-09-16T15:46:29.280 DEBUG:teuthology.orchestra.run.smithi123:> import subprocess 2024-09-16T15:46:29.280 DEBUG:teuthology.orchestra.run.smithi123:> 2024-09-16T15:46:29.280 DEBUG:teuthology.orchestra.run.smithi123:> def _find_admin_socket(client_name): 2024-09-16T15:46:29.281 DEBUG:teuthology.orchestra.run.smithi123:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2024-09-16T15:46:29.281 DEBUG:teuthology.orchestra.run.smithi123:> files = glob.glob(asok_path) 2024-09-16T15:46:29.281 DEBUG:teuthology.orchestra.run.smithi123:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-09-16T15:46:29.281 DEBUG:teuthology.orchestra.run.smithi123:> 2024-09-16T15:46:29.281 DEBUG:teuthology.orchestra.run.smithi123:> # Given a non-glob path, it better be there 2024-09-16T15:46:29.281 DEBUG:teuthology.orchestra.run.smithi123:> if "*" not in asok_path: 2024-09-16T15:46:29.281 DEBUG:teuthology.orchestra.run.smithi123:> assert(len(files) == 1) 2024-09-16T15:46:29.281 DEBUG:teuthology.orchestra.run.smithi123:> return files[0] 2024-09-16T15:46:29.281 DEBUG:teuthology.orchestra.run.smithi123:> 2024-09-16T15:46:29.281 DEBUG:teuthology.orchestra.run.smithi123:> for f in files: 2024-09-16T15:46:29.281 DEBUG:teuthology.orchestra.run.smithi123:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-09-16T15:46:29.281 DEBUG:teuthology.orchestra.run.smithi123:> if os.path.exists("/proc/{0}".format(pid)): 2024-09-16T15:46:29.281 DEBUG:teuthology.orchestra.run.smithi123:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-09-16T15:46:29.281 DEBUG:teuthology.orchestra.run.smithi123:> contents = proc_f.read() 2024-09-16T15:46:29.281 DEBUG:teuthology.orchestra.run.smithi123:> if mountpoint in contents: 2024-09-16T15:46:29.282 DEBUG:teuthology.orchestra.run.smithi123:> return f 2024-09-16T15:46:29.282 DEBUG:teuthology.orchestra.run.smithi123:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-09-16T15:46:29.282 DEBUG:teuthology.orchestra.run.smithi123:> 2024-09-16T15:46:29.282 DEBUG:teuthology.orchestra.run.smithi123:> print(_find_admin_socket("client.0")) 2024-09-16T15:46:29.282 DEBUG:teuthology.orchestra.run.smithi123:> ' 2024-09-16T15:46:29.403 INFO:teuthology.orchestra.run.smithi123.stdout:/var/run/ceph/ceph-client.0.41267.asok 2024-09-16T15:46:29.407 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:29 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-16T15:46:29.417 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.41267.asok 2024-09-16T15:46:29.417 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:46:29.417 DEBUG:teuthology.orchestra.run.smithi123:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.41267.asok status 2024-09-16T15:46:29.571 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2024-09-16T15:46:29.571 INFO:teuthology.orchestra.run.smithi123.stdout: "metadata": { 2024-09-16T15:46:29.572 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph_sha1": "33f95d2ae1bcbe7eb04404303f179eeeaa6982df", 2024-09-16T15:46:29.572 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph_version": "ceph version 19.3.0-4938-g33f95d2a (33f95d2ae1bcbe7eb04404303f179eeeaa6982df) squid (dev)", 2024-09-16T15:46:29.572 INFO:teuthology.orchestra.run.smithi123.stdout: "entity_id": "0", 2024-09-16T15:46:29.572 INFO:teuthology.orchestra.run.smithi123.stdout: "hostname": "smithi123", 2024-09-16T15:46:29.572 INFO:teuthology.orchestra.run.smithi123.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2024-09-16T15:46:29.572 INFO:teuthology.orchestra.run.smithi123.stdout: "pid": "41267", 2024-09-16T15:46:29.572 INFO:teuthology.orchestra.run.smithi123.stdout: "root": "/" 2024-09-16T15:46:29.572 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:46:29.572 INFO:teuthology.orchestra.run.smithi123.stdout: "dentry_count": 0, 2024-09-16T15:46:29.572 INFO:teuthology.orchestra.run.smithi123.stdout: "dentry_pinned_count": 0, 2024-09-16T15:46:29.572 INFO:teuthology.orchestra.run.smithi123.stdout: "id": 5952, 2024-09-16T15:46:29.572 INFO:teuthology.orchestra.run.smithi123.stdout: "inst": { 2024-09-16T15:46:29.572 INFO:teuthology.orchestra.run.smithi123.stdout: "name": { 2024-09-16T15:46:29.573 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "client", 2024-09-16T15:46:29.573 INFO:teuthology.orchestra.run.smithi123.stdout: "num": 5952 2024-09-16T15:46:29.573 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:46:29.573 INFO:teuthology.orchestra.run.smithi123.stdout: "addr": { 2024-09-16T15:46:29.573 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "v1", 2024-09-16T15:46:29.573 INFO:teuthology.orchestra.run.smithi123.stdout: "addr": "192.168.0.1:0", 2024-09-16T15:46:29.573 INFO:teuthology.orchestra.run.smithi123.stdout: "nonce": 4074585735 2024-09-16T15:46:29.573 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-16T15:46:29.573 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:46:29.573 INFO:teuthology.orchestra.run.smithi123.stdout: "addr": { 2024-09-16T15:46:29.573 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "v1", 2024-09-16T15:46:29.573 INFO:teuthology.orchestra.run.smithi123.stdout: "addr": "192.168.0.1:0", 2024-09-16T15:46:29.573 INFO:teuthology.orchestra.run.smithi123.stdout: "nonce": 4074585735 2024-09-16T15:46:29.573 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:46:29.574 INFO:teuthology.orchestra.run.smithi123.stdout: "inst_str": "client.5952 192.168.0.1:0/4074585735", 2024-09-16T15:46:29.574 INFO:teuthology.orchestra.run.smithi123.stdout: "addr_str": "192.168.0.1:0/4074585735", 2024-09-16T15:46:29.574 INFO:teuthology.orchestra.run.smithi123.stdout: "inode_count": 1, 2024-09-16T15:46:29.574 INFO:teuthology.orchestra.run.smithi123.stdout: "mds_epoch": 36, 2024-09-16T15:46:29.574 INFO:teuthology.orchestra.run.smithi123.stdout: "osd_epoch": 58, 2024-09-16T15:46:29.574 INFO:teuthology.orchestra.run.smithi123.stdout: "osd_epoch_barrier": 0, 2024-09-16T15:46:29.574 INFO:teuthology.orchestra.run.smithi123.stdout: "blocklisted": false, 2024-09-16T15:46:29.574 INFO:teuthology.orchestra.run.smithi123.stdout: "fs_name": "cephfs" 2024-09-16T15:46:29.574 INFO:teuthology.orchestra.run.smithi123.stdout:} 2024-09-16T15:46:29.585 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:46:29.585 DEBUG:teuthology.orchestra.run.smithi123:> stat --file-system '--printf=%T 2024-09-16T15:46:29.585 DEBUG:teuthology.orchestra.run.smithi123:> ' -- /home/ubuntu/cephtest/mnt.0 2024-09-16T15:46:29.591 INFO:teuthology.orchestra.run.smithi123.stdout:fuseblk 2024-09-16T15:46:29.591 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2024-09-16T15:46:29.591 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:46:29.591 DEBUG:teuthology.orchestra.run.smithi123:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-09-16T15:46:29.651 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-09-16T15:46:29.827 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:29.821+0000 7fb95fdc6640 1 -- 172.21.15.123:0/659175171 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb944005920 msgr2=0x7fb944005da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:46:29.827 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:29.821+0000 7fb95fdc6640 1 --2- 172.21.15.123:0/659175171 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb944005920 0x7fb944005da0 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7fb95806fc10 tx=0x7fb94801cc60 comp rx=0 tx=0).stop 2024-09-16T15:46:29.827 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:29.825+0000 7fb95e363640 1 -- 172.21.15.123:0/659175171 reap_dead start 2024-09-16T15:46:29.827 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:29.825+0000 7fb95fdc6640 1 -- 172.21.15.123:0/659175171 shutdown_connections 2024-09-16T15:46:29.827 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:29.825+0000 7fb95fdc6640 1 -- 172.21.15.123:0/659175171 >> 172.21.15.123:0/659175171 conn(0x7fb9580cfb10 msgr2=0x7fb9580cff20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:46:29.827 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:29.825+0000 7fb95fdc6640 1 -- 172.21.15.123:0/659175171 shutdown_connections 2024-09-16T15:46:29.827 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:29.825+0000 7fb95fdc6640 1 -- 172.21.15.123:0/659175171 wait complete. 2024-09-16T15:46:29.828 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:29.825+0000 7fb95fdc6640 1 Processor -- start 2024-09-16T15:46:29.828 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:29.825+0000 7fb95fdc6640 1 -- start start 2024-09-16T15:46:29.830 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:29.825+0000 7fb95fdc6640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fb944005920 0x7fb95814e110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:46:29.830 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:29.825+0000 7fb95fdc6640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb95814f270 0x7fb9580af610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:46:29.830 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:29.825+0000 7fb95fdc6640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb95814ed20 0x7fb9580afb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:46:29.830 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:29.825+0000 7fb95fdc6640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fb95815abe0 con 0x7fb95814f270 2024-09-16T15:46:29.830 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:29.825+0000 7fb95fdc6640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fb95815aa60 con 0x7fb95814ed20 2024-09-16T15:46:29.830 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:29.825+0000 7fb95fdc6640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fb95815ad60 con 0x7fb944005920 2024-09-16T15:46:29.830 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:29.825+0000 7fb95d361640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb95814f270 0x7fb9580af610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:46:29.830 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:29.825+0000 7fb95d361640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb95814f270 0x7fb9580af610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:54842/0 (socket says 172.21.15.123:54842) 2024-09-16T15:46:29.831 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:29.825+0000 7fb95d361640 1 -- 172.21.15.123:0/3625799620 learned_addr learned my addr 172.21.15.123:0/3625799620 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:46:29.831 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:29.825+0000 7fb95e363640 1 --2- 172.21.15.123:0/3625799620 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb95814ed20 0x7fb9580afb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:46:29.831 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:29.825+0000 7fb95d361640 1 -- 172.21.15.123:0/3625799620 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fb944005920 msgr2=0x7fb95814e110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:46:29.831 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:29.825+0000 7fb95db62640 1 --2- 172.21.15.123:0/3625799620 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fb944005920 0x7fb95814e110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:46:29.831 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:29.825+0000 7fb95d361640 1 --2- 172.21.15.123:0/3625799620 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fb944005920 0x7fb95814e110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:29.831 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:29.825+0000 7fb95d361640 1 -- 172.21.15.123:0/3625799620 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb95814ed20 msgr2=0x7fb9580afb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:46:29.831 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:29.825+0000 7fb95d361640 1 --2- 172.21.15.123:0/3625799620 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb95814ed20 0x7fb9580afb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:29.831 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:29.825+0000 7fb95d361640 1 -- 172.21.15.123:0/3625799620 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb95815cb60 con 0x7fb95814f270 2024-09-16T15:46:29.831 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:29.825+0000 7fb95d361640 1 --2- 172.21.15.123:0/3625799620 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb95814f270 0x7fb9580af610 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7fb948001ec0 tx=0x7fb9480056a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:46:29.831 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:29.825+0000 7fb94e7fc640 1 -- 172.21.15.123:0/3625799620 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9480ab040 con 0x7fb95814f270 2024-09-16T15:46:29.831 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:29.825+0000 7fb94e7fc640 1 -- 172.21.15.123:0/3625799620 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fb9480046d0 con 0x7fb95814f270 2024-09-16T15:46:29.831 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:29.825+0000 7fb95fdc6640 1 -- 172.21.15.123:0/3625799620 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb95815ce40 con 0x7fb95814f270 2024-09-16T15:46:29.831 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:29.825+0000 7fb94e7fc640 1 -- 172.21.15.123:0/3625799620 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb948004b20 con 0x7fb95814f270 2024-09-16T15:46:29.831 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:29.825+0000 7fb95fdc6640 1 -- 172.21.15.123:0/3625799620 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb95814e7d0 con 0x7fb95814f270 2024-09-16T15:46:29.832 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:29.829+0000 7fb95fdc6640 1 -- 172.21.15.123:0/3625799620 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb95806f420 con 0x7fb95814f270 2024-09-16T15:46:29.832 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:29.829+0000 7fb94e7fc640 1 -- 172.21.15.123:0/3625799620 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 9) ==== 97233+0+0 (secure 0 0 0) 0x7fb9480c0030 con 0x7fb95814f270 2024-09-16T15:46:29.832 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:29.829+0000 7fb94e7fc640 1 --2- 172.21.15.123:0/3625799620 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fb93c073070 0x7fb93c075530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:46:29.833 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:29.829+0000 7fb94e7fc640 1 -- 172.21.15.123:0/3625799620 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(58..58 src has 1..58) ==== 5709+0+0 (secure 0 0 0) 0x7fb94812e700 con 0x7fb95814f270 2024-09-16T15:46:29.833 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:29.829+0000 7fb95db62640 1 --2- 172.21.15.123:0/3625799620 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fb93c073070 0x7fb93c075530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:46:29.833 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:29.829+0000 7fb95db62640 1 --2- 172.21.15.123:0/3625799620 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fb93c073070 0x7fb93c075530 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7fb954000fc0 tx=0x7fb9540014b0 comp rx=0 tx=0).ready entity=mgr.4135 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:46:29.836 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:29.833+0000 7fb94e7fc640 1 -- 172.21.15.123:0/3625799620 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7fb9480c7060 con 0x7fb95814f270 2024-09-16T15:46:30.047 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:30.045+0000 7fb95fdc6640 1 -- 172.21.15.123:0/3625799620 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7fb9580c3350 con 0x7fb95814f270 2024-09-16T15:46:30.047 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:30.045+0000 7fb94e7fc640 1 -- 172.21.15.123:0/3625799620 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v36) ==== 53+0+73 (secure 0 0 0) 0x7fb948016970 con 0x7fb95814f270 2024-09-16T15:46:30.048 INFO:teuthology.orchestra.run.smithi123.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-09-16T15:46:30.050 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:30.045+0000 7fb95fdc6640 1 -- 172.21.15.123:0/3625799620 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fb93c073070 msgr2=0x7fb93c075530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:46:30.050 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:30.045+0000 7fb95fdc6640 1 --2- 172.21.15.123:0/3625799620 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fb93c073070 0x7fb93c075530 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7fb954000fc0 tx=0x7fb9540014b0 comp rx=0 tx=0).stop 2024-09-16T15:46:30.050 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:30.045+0000 7fb95fdc6640 1 -- 172.21.15.123:0/3625799620 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb95814f270 msgr2=0x7fb9580af610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:46:30.050 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:30.045+0000 7fb95fdc6640 1 --2- 172.21.15.123:0/3625799620 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb95814f270 0x7fb9580af610 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7fb948001ec0 tx=0x7fb9480056a0 comp rx=0 tx=0).stop 2024-09-16T15:46:30.050 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:30.045+0000 7fb95fdc6640 1 -- 172.21.15.123:0/3625799620 shutdown_connections 2024-09-16T15:46:30.050 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:30.045+0000 7fb95fdc6640 1 --2- 172.21.15.123:0/3625799620 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fb93c073070 0x7fb93c075530 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:30.051 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:30.045+0000 7fb95fdc6640 1 --2- 172.21.15.123:0/3625799620 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb95814ed20 0x7fb9580afb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:30.051 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:30.045+0000 7fb95fdc6640 1 --2- 172.21.15.123:0/3625799620 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb95814f270 0x7fb9580af610 unknown :-1 s=CLOSED pgs=664 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:30.051 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:30.045+0000 7fb95fdc6640 1 --2- 172.21.15.123:0/3625799620 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fb944005920 0x7fb95814e110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:30.051 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:30.045+0000 7fb95fdc6640 1 -- 172.21.15.123:0/3625799620 >> 172.21.15.123:0/3625799620 conn(0x7fb9580cfb10 msgr2=0x7fb958156fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:46:30.051 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:30.045+0000 7fb95fdc6640 1 -- 172.21.15.123:0/3625799620 shutdown_connections 2024-09-16T15:46:30.051 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:30.045+0000 7fb95fdc6640 1 -- 172.21.15.123:0/3625799620 wait complete. 2024-09-16T15:46:30.065 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-09-16T15:46:30.065 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-09-16T15:46:30.066 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi184.front.sepia.ceph.com 2024-09-16T15:46:30.066 INFO:tasks.cephfs.mount:self.client.name = client.1 2024-09-16T15:46:30.066 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2024-09-16T15:46:30.066 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-09-16T15:46:30.066 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-09-16T15:46:30.066 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-09-16T15:46:30.066 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.1' netns for '/home/ubuntu/cephtest/mnt.1' 2024-09-16T15:46:30.066 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:46:30.066 DEBUG:teuthology.orchestra.run.smithi184:> ip addr 2024-09-16T15:46:30.073 INFO:teuthology.orchestra.run.smithi184.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-09-16T15:46:30.073 INFO:teuthology.orchestra.run.smithi184.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-09-16T15:46:30.074 INFO:teuthology.orchestra.run.smithi184.stdout: inet 127.0.0.1/8 scope host lo 2024-09-16T15:46:30.074 INFO:teuthology.orchestra.run.smithi184.stdout: valid_lft forever preferred_lft forever 2024-09-16T15:46:30.074 INFO:teuthology.orchestra.run.smithi184.stdout: inet6 ::1/128 scope host 2024-09-16T15:46:30.074 INFO:teuthology.orchestra.run.smithi184.stdout: valid_lft forever preferred_lft forever 2024-09-16T15:46:30.074 INFO:teuthology.orchestra.run.smithi184.stdout:2: eno1: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-09-16T15:46:30.074 INFO:teuthology.orchestra.run.smithi184.stdout: link/ether 0c:c4:7a:d9:37:16 brd ff:ff:ff:ff:ff:ff 2024-09-16T15:46:30.074 INFO:teuthology.orchestra.run.smithi184.stdout: altname enp7s0f0 2024-09-16T15:46:30.074 INFO:teuthology.orchestra.run.smithi184.stdout:3: eno2: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-09-16T15:46:30.074 INFO:teuthology.orchestra.run.smithi184.stdout: link/ether 0c:c4:7a:d9:37:17 brd ff:ff:ff:ff:ff:ff 2024-09-16T15:46:30.074 INFO:teuthology.orchestra.run.smithi184.stdout: altname enp7s0f1 2024-09-16T15:46:30.074 INFO:teuthology.orchestra.run.smithi184.stdout:4: enp3s0f0: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-09-16T15:46:30.074 INFO:teuthology.orchestra.run.smithi184.stdout: link/ether 0c:c4:7a:88:7b:8e brd ff:ff:ff:ff:ff:ff 2024-09-16T15:46:30.074 INFO:teuthology.orchestra.run.smithi184.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-09-16T15:46:30.075 INFO:teuthology.orchestra.run.smithi184.stdout: link/ether 0c:c4:7a:88:7b:8f brd ff:ff:ff:ff:ff:ff 2024-09-16T15:46:30.075 INFO:teuthology.orchestra.run.smithi184.stdout: inet 172.21.15.184/20 metric 100 brd 172.21.15.255 scope global dynamic enp3s0f1 2024-09-16T15:46:30.075 INFO:teuthology.orchestra.run.smithi184.stdout: valid_lft 42267sec preferred_lft 42267sec 2024-09-16T15:46:30.075 INFO:teuthology.orchestra.run.smithi184.stdout: inet6 fe80::ec4:7aff:fe88:7b8f/64 scope link 2024-09-16T15:46:30.075 INFO:teuthology.orchestra.run.smithi184.stdout: valid_lft forever preferred_lft forever 2024-09-16T15:46:30.075 INFO:teuthology.orchestra.run.smithi184.stdout:6: docker0: mtu 1500 qdisc noqueue state DOWN group default 2024-09-16T15:46:30.075 INFO:teuthology.orchestra.run.smithi184.stdout: link/ether 02:42:5d:0a:44:b9 brd ff:ff:ff:ff:ff:ff 2024-09-16T15:46:30.075 INFO:teuthology.orchestra.run.smithi184.stdout: inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0 2024-09-16T15:46:30.075 INFO:teuthology.orchestra.run.smithi184.stdout: valid_lft forever preferred_lft forever 2024-09-16T15:46:30.075 INFO:teuthology.orchestra.run.smithi184.stdout:7: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-09-16T15:46:30.075 INFO:teuthology.orchestra.run.smithi184.stdout: link/ether 5e:1f:84:6d:79:ae brd ff:ff:ff:ff:ff:ff 2024-09-16T15:46:30.075 INFO:teuthology.orchestra.run.smithi184.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-09-16T15:46:30.075 INFO:teuthology.orchestra.run.smithi184.stdout: valid_lft forever preferred_lft forever 2024-09-16T15:46:30.075 INFO:teuthology.orchestra.run.smithi184.stdout: inet6 fe80::5c1f:84ff:fe6d:79ae/64 scope link 2024-09-16T15:46:30.076 INFO:teuthology.orchestra.run.smithi184.stdout: valid_lft forever preferred_lft forever 2024-09-16T15:46:30.076 INFO:teuthology.orchestra.run.smithi184.stdout:8: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-09-16T15:46:30.076 INFO:teuthology.orchestra.run.smithi184.stdout: link/ether 0e:5c:f6:1f:82:2f brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.1 2024-09-16T15:46:30.076 INFO:teuthology.orchestra.run.smithi184.stdout: inet6 fe80::c5c:f6ff:fe1f:822f/64 scope link 2024-09-16T15:46:30.076 INFO:teuthology.orchestra.run.smithi184.stdout: valid_lft forever preferred_lft forever 2024-09-16T15:46:30.076 INFO:teuthology.orchestra.run.smithi184.stdout:9: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-09-16T15:46:30.076 INFO:teuthology.orchestra.run.smithi184.stdout: link/ether c6:a6:3d:69:d0:db brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.3 2024-09-16T15:46:30.076 INFO:teuthology.orchestra.run.smithi184.stdout: inet6 fe80::c4a6:3dff:fe69:d0db/64 scope link 2024-09-16T15:46:30.076 INFO:teuthology.orchestra.run.smithi184.stdout: valid_lft forever preferred_lft forever 2024-09-16T15:46:30.076 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:46:30.076 DEBUG:teuthology.orchestra.run.smithi184:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-09-16T15:46:30.127 INFO:teuthology.orchestra.run.smithi184.stdout:1 2024-09-16T15:46:30.128 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:46:30.129 DEBUG:teuthology.orchestra.run.smithi184:> route 2024-09-16T15:46:30.180 INFO:teuthology.orchestra.run.smithi184.stdout:Kernel IP routing table 2024-09-16T15:46:30.180 INFO:teuthology.orchestra.run.smithi184.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-09-16T15:46:30.180 INFO:teuthology.orchestra.run.smithi184.stdout:default _gateway 0.0.0.0 UG 0 0 0 enp3s0f1 2024-09-16T15:46:30.180 INFO:teuthology.orchestra.run.smithi184.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-09-16T15:46:30.180 INFO:teuthology.orchestra.run.smithi184.stdout:158.69.67.47 _gateway 255.255.255.255 UGH 100 0 0 enp3s0f1 2024-09-16T15:46:30.180 INFO:teuthology.orchestra.run.smithi184.stdout:172.17.0.0 0.0.0.0 255.255.0.0 U 0 0 0 docker0 2024-09-16T15:46:30.180 INFO:teuthology.orchestra.run.smithi184.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-09-16T15:46:30.180 INFO:teuthology.orchestra.run.smithi184.stdout:ns1.front.sepia 0.0.0.0 255.255.255.255 UH 100 0 0 enp3s0f1 2024-09-16T15:46:30.180 INFO:teuthology.orchestra.run.smithi184.stdout:172.21.0.2 0.0.0.0 255.255.255.255 UH 100 0 0 enp3s0f1 2024-09-16T15:46:30.180 INFO:teuthology.orchestra.run.smithi184.stdout:_gateway 0.0.0.0 255.255.255.255 UH 100 0 0 enp3s0f1 2024-09-16T15:46:30.181 INFO:teuthology.orchestra.run.smithi184.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-09-16T15:46:30.181 DEBUG:teuthology.orchestra.run.smithi184:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-09-16T15:46:30.181 DEBUG:teuthology.orchestra.run.smithi184:> set -e 2024-09-16T15:46:30.181 DEBUG:teuthology.orchestra.run.smithi184:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-09-16T15:46:30.181 DEBUG:teuthology.orchestra.run.smithi184:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-09-16T15:46:30.181 DEBUG:teuthology.orchestra.run.smithi184:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-09-16T15:46:30.181 DEBUG:teuthology.orchestra.run.smithi184:> ') 2024-09-16T15:46:30.268 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:46:30 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-09-16T15:46:30.297 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:46:30 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-16T15:46:30.304 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:46:30.305 DEBUG:teuthology.orchestra.run.smithi184:> ip netns list 2024-09-16T15:46:30.309 INFO:teuthology.orchestra.run.smithi184.stdout:ceph-ns--home-ubuntu-cephtest-mnt.3 (id: 1) 2024-09-16T15:46:30.309 INFO:teuthology.orchestra.run.smithi184.stdout:ceph-ns--home-ubuntu-cephtest-mnt.1 (id: 0) 2024-09-16T15:46:30.309 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.1' netns for '/home/ubuntu/cephtest/mnt.1' 2024-09-16T15:46:30.310 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:46:30.310 DEBUG:teuthology.orchestra.run.smithi184:> sudo ip link set brx.0 up 2024-09-16T15:46:30.361 INFO:tasks.cephfs.fuse_mount:Client client.1 config is {} 2024-09-16T15:46:30.361 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-09-16T15:46:30.361 DEBUG:teuthology.orchestra.run.smithi184:> mkdir -p -v /home/ubuntu/cephtest/mnt.1 2024-09-16T15:46:30.411 INFO:teuthology.orchestra.run.smithi184.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.1' 2024-09-16T15:46:30.412 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-09-16T15:46:30.412 DEBUG:teuthology.orchestra.run.smithi184:> chmod 0000 /home/ubuntu/cephtest/mnt.1 2024-09-16T15:46:30.459 DEBUG:teuthology.orchestra.run.smithi184:> sudo modprobe fuse 2024-09-16T15:46:30.517 DEBUG:teuthology.orchestra.run.smithi184:> cat /proc/self/mounts | awk '{print $2}' 2024-09-16T15:46:30.564 INFO:teuthology.orchestra.run.smithi184.stdout:/sys 2024-09-16T15:46:30.565 INFO:teuthology.orchestra.run.smithi184.stdout:/proc 2024-09-16T15:46:30.565 INFO:teuthology.orchestra.run.smithi184.stdout:/dev 2024-09-16T15:46:30.565 INFO:teuthology.orchestra.run.smithi184.stdout:/dev/pts 2024-09-16T15:46:30.565 INFO:teuthology.orchestra.run.smithi184.stdout:/run 2024-09-16T15:46:30.565 INFO:teuthology.orchestra.run.smithi184.stdout:/ 2024-09-16T15:46:30.565 INFO:teuthology.orchestra.run.smithi184.stdout:/sys/kernel/security 2024-09-16T15:46:30.565 INFO:teuthology.orchestra.run.smithi184.stdout:/dev/shm 2024-09-16T15:46:30.565 INFO:teuthology.orchestra.run.smithi184.stdout:/run/lock 2024-09-16T15:46:30.565 INFO:teuthology.orchestra.run.smithi184.stdout:/sys/fs/cgroup 2024-09-16T15:46:30.565 INFO:teuthology.orchestra.run.smithi184.stdout:/sys/fs/pstore 2024-09-16T15:46:30.565 INFO:teuthology.orchestra.run.smithi184.stdout:/sys/fs/bpf 2024-09-16T15:46:30.565 INFO:teuthology.orchestra.run.smithi184.stdout:/proc/sys/fs/binfmt_misc 2024-09-16T15:46:30.566 INFO:teuthology.orchestra.run.smithi184.stdout:/dev/hugepages 2024-09-16T15:46:30.566 INFO:teuthology.orchestra.run.smithi184.stdout:/dev/mqueue 2024-09-16T15:46:30.566 INFO:teuthology.orchestra.run.smithi184.stdout:/sys/kernel/debug 2024-09-16T15:46:30.566 INFO:teuthology.orchestra.run.smithi184.stdout:/sys/kernel/tracing 2024-09-16T15:46:30.566 INFO:teuthology.orchestra.run.smithi184.stdout:/proc/fs/nfsd 2024-09-16T15:46:30.566 INFO:teuthology.orchestra.run.smithi184.stdout:/sys/fs/fuse/connections 2024-09-16T15:46:30.566 INFO:teuthology.orchestra.run.smithi184.stdout:/sys/kernel/config 2024-09-16T15:46:30.566 INFO:teuthology.orchestra.run.smithi184.stdout:/run/credentials/systemd-sysusers.service 2024-09-16T15:46:30.566 INFO:teuthology.orchestra.run.smithi184.stdout:/run/rpc_pipefs 2024-09-16T15:46:30.566 INFO:teuthology.orchestra.run.smithi184.stdout:/run/user/1000 2024-09-16T15:46:30.566 INFO:teuthology.orchestra.run.smithi184.stdout:/var/lib/ceph 2024-09-16T15:46:30.566 INFO:teuthology.orchestra.run.smithi184.stdout:/var/lib/ceph/osd/ceph-4 2024-09-16T15:46:30.566 INFO:teuthology.orchestra.run.smithi184.stdout:/var/lib/ceph/osd/ceph-5 2024-09-16T15:46:30.566 INFO:teuthology.orchestra.run.smithi184.stdout:/var/lib/ceph/osd/ceph-6 2024-09-16T15:46:30.566 INFO:teuthology.orchestra.run.smithi184.stdout:/var/lib/ceph/osd/ceph-7 2024-09-16T15:46:30.567 INFO:teuthology.orchestra.run.smithi184.stdout:/run/netns 2024-09-16T15:46:30.567 INFO:teuthology.orchestra.run.smithi184.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-09-16T15:46:30.567 INFO:teuthology.orchestra.run.smithi184.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-09-16T15:46:30.567 INFO:teuthology.orchestra.run.smithi184.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-09-16T15:46:30.567 INFO:teuthology.orchestra.run.smithi184.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-09-16T15:46:30.568 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:46:30.568 DEBUG:teuthology.orchestra.run.smithi184:> ls /sys/fs/fuse/connections 2024-09-16T15:46:30.615 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-09-16T15:46:30.615 DEBUG:teuthology.orchestra.run.smithi184:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.1 --id 1) 2024-09-16T15:46:30.660 DEBUG:teuthology.orchestra.run.smithi184:> sudo modprobe fuse 2024-09-16T15:46:30.676 DEBUG:teuthology.orchestra.run.smithi184:> cat /proc/self/mounts | awk '{print $2}' 2024-09-16T15:46:30.723 INFO:teuthology.orchestra.run.smithi184.stdout:/sys 2024-09-16T15:46:30.724 INFO:teuthology.orchestra.run.smithi184.stdout:/proc 2024-09-16T15:46:30.724 INFO:teuthology.orchestra.run.smithi184.stdout:/dev 2024-09-16T15:46:30.724 INFO:teuthology.orchestra.run.smithi184.stdout:/dev/pts 2024-09-16T15:46:30.724 INFO:teuthology.orchestra.run.smithi184.stdout:/run 2024-09-16T15:46:30.724 INFO:teuthology.orchestra.run.smithi184.stdout:/ 2024-09-16T15:46:30.724 INFO:teuthology.orchestra.run.smithi184.stdout:/sys/kernel/security 2024-09-16T15:46:30.724 INFO:teuthology.orchestra.run.smithi184.stdout:/dev/shm 2024-09-16T15:46:30.724 INFO:teuthology.orchestra.run.smithi184.stdout:/run/lock 2024-09-16T15:46:30.724 INFO:teuthology.orchestra.run.smithi184.stdout:/sys/fs/cgroup 2024-09-16T15:46:30.724 INFO:teuthology.orchestra.run.smithi184.stdout:/sys/fs/pstore 2024-09-16T15:46:30.724 INFO:teuthology.orchestra.run.smithi184.stdout:/sys/fs/bpf 2024-09-16T15:46:30.724 INFO:teuthology.orchestra.run.smithi184.stdout:/proc/sys/fs/binfmt_misc 2024-09-16T15:46:30.724 INFO:teuthology.orchestra.run.smithi184.stdout:/dev/hugepages 2024-09-16T15:46:30.725 INFO:teuthology.orchestra.run.smithi184.stdout:/dev/mqueue 2024-09-16T15:46:30.725 INFO:teuthology.orchestra.run.smithi184.stdout:/sys/kernel/debug 2024-09-16T15:46:30.725 INFO:teuthology.orchestra.run.smithi184.stdout:/sys/kernel/tracing 2024-09-16T15:46:30.725 INFO:teuthology.orchestra.run.smithi184.stdout:/proc/fs/nfsd 2024-09-16T15:46:30.725 INFO:teuthology.orchestra.run.smithi184.stdout:/sys/fs/fuse/connections 2024-09-16T15:46:30.725 INFO:teuthology.orchestra.run.smithi184.stdout:/sys/kernel/config 2024-09-16T15:46:30.725 INFO:teuthology.orchestra.run.smithi184.stdout:/run/credentials/systemd-sysusers.service 2024-09-16T15:46:30.725 INFO:teuthology.orchestra.run.smithi184.stdout:/run/rpc_pipefs 2024-09-16T15:46:30.725 INFO:teuthology.orchestra.run.smithi184.stdout:/run/user/1000 2024-09-16T15:46:30.725 INFO:teuthology.orchestra.run.smithi184.stdout:/var/lib/ceph 2024-09-16T15:46:30.725 INFO:teuthology.orchestra.run.smithi184.stdout:/var/lib/ceph/osd/ceph-4 2024-09-16T15:46:30.725 INFO:teuthology.orchestra.run.smithi184.stdout:/var/lib/ceph/osd/ceph-5 2024-09-16T15:46:30.725 INFO:teuthology.orchestra.run.smithi184.stdout:/var/lib/ceph/osd/ceph-6 2024-09-16T15:46:30.725 INFO:teuthology.orchestra.run.smithi184.stdout:/var/lib/ceph/osd/ceph-7 2024-09-16T15:46:30.726 INFO:teuthology.orchestra.run.smithi184.stdout:/run/netns 2024-09-16T15:46:30.726 INFO:teuthology.orchestra.run.smithi184.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-09-16T15:46:30.726 INFO:teuthology.orchestra.run.smithi184.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-09-16T15:46:30.726 INFO:teuthology.orchestra.run.smithi184.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-09-16T15:46:30.726 INFO:teuthology.orchestra.run.smithi184.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-09-16T15:46:30.727 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:46:30.727 DEBUG:teuthology.orchestra.run.smithi184:> ls /sys/fs/fuse/connections 2024-09-16T15:46:30.737 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi184.stderr:2024-09-16T15:46:30.734+0000 7fdde9463580 -1 init, newargv = 0x55f85ca2adc0 newargc=15 2024-09-16T15:46:30.738 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi184.stderr:ceph-fuse[35114]: starting ceph client 2024-09-16T15:46:30.745 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi184.stderr:ceph-fuse[35114]: starting fuse 2024-09-16T15:46:31.844 DEBUG:teuthology.orchestra.run.smithi184:> sudo modprobe fuse 2024-09-16T15:46:31.860 DEBUG:teuthology.orchestra.run.smithi184:> cat /proc/self/mounts | awk '{print $2}' 2024-09-16T15:46:31.909 INFO:teuthology.orchestra.run.smithi184.stdout:/sys 2024-09-16T15:46:31.909 INFO:teuthology.orchestra.run.smithi184.stdout:/proc 2024-09-16T15:46:31.909 INFO:teuthology.orchestra.run.smithi184.stdout:/dev 2024-09-16T15:46:31.909 INFO:teuthology.orchestra.run.smithi184.stdout:/dev/pts 2024-09-16T15:46:31.909 INFO:teuthology.orchestra.run.smithi184.stdout:/run 2024-09-16T15:46:31.909 INFO:teuthology.orchestra.run.smithi184.stdout:/ 2024-09-16T15:46:31.909 INFO:teuthology.orchestra.run.smithi184.stdout:/sys/kernel/security 2024-09-16T15:46:31.909 INFO:teuthology.orchestra.run.smithi184.stdout:/dev/shm 2024-09-16T15:46:31.909 INFO:teuthology.orchestra.run.smithi184.stdout:/run/lock 2024-09-16T15:46:31.909 INFO:teuthology.orchestra.run.smithi184.stdout:/sys/fs/cgroup 2024-09-16T15:46:31.909 INFO:teuthology.orchestra.run.smithi184.stdout:/sys/fs/pstore 2024-09-16T15:46:31.909 INFO:teuthology.orchestra.run.smithi184.stdout:/sys/fs/bpf 2024-09-16T15:46:31.909 INFO:teuthology.orchestra.run.smithi184.stdout:/proc/sys/fs/binfmt_misc 2024-09-16T15:46:31.910 INFO:teuthology.orchestra.run.smithi184.stdout:/dev/hugepages 2024-09-16T15:46:31.910 INFO:teuthology.orchestra.run.smithi184.stdout:/dev/mqueue 2024-09-16T15:46:31.910 INFO:teuthology.orchestra.run.smithi184.stdout:/sys/kernel/debug 2024-09-16T15:46:31.910 INFO:teuthology.orchestra.run.smithi184.stdout:/sys/kernel/tracing 2024-09-16T15:46:31.910 INFO:teuthology.orchestra.run.smithi184.stdout:/proc/fs/nfsd 2024-09-16T15:46:31.910 INFO:teuthology.orchestra.run.smithi184.stdout:/sys/fs/fuse/connections 2024-09-16T15:46:31.910 INFO:teuthology.orchestra.run.smithi184.stdout:/sys/kernel/config 2024-09-16T15:46:31.910 INFO:teuthology.orchestra.run.smithi184.stdout:/run/credentials/systemd-sysusers.service 2024-09-16T15:46:31.910 INFO:teuthology.orchestra.run.smithi184.stdout:/run/rpc_pipefs 2024-09-16T15:46:31.910 INFO:teuthology.orchestra.run.smithi184.stdout:/run/user/1000 2024-09-16T15:46:31.910 INFO:teuthology.orchestra.run.smithi184.stdout:/var/lib/ceph 2024-09-16T15:46:31.910 INFO:teuthology.orchestra.run.smithi184.stdout:/var/lib/ceph/osd/ceph-4 2024-09-16T15:46:31.910 INFO:teuthology.orchestra.run.smithi184.stdout:/var/lib/ceph/osd/ceph-5 2024-09-16T15:46:31.910 INFO:teuthology.orchestra.run.smithi184.stdout:/var/lib/ceph/osd/ceph-6 2024-09-16T15:46:31.910 INFO:teuthology.orchestra.run.smithi184.stdout:/var/lib/ceph/osd/ceph-7 2024-09-16T15:46:31.911 INFO:teuthology.orchestra.run.smithi184.stdout:/run/netns 2024-09-16T15:46:31.911 INFO:teuthology.orchestra.run.smithi184.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-09-16T15:46:31.911 INFO:teuthology.orchestra.run.smithi184.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-09-16T15:46:31.911 INFO:teuthology.orchestra.run.smithi184.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-09-16T15:46:31.911 INFO:teuthology.orchestra.run.smithi184.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-09-16T15:46:31.911 INFO:teuthology.orchestra.run.smithi184.stdout:/home/ubuntu/cephtest/mnt.1 2024-09-16T15:46:31.912 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:46:31.912 DEBUG:teuthology.orchestra.run.smithi184:> ls /sys/fs/fuse/connections 2024-09-16T15:46:31.959 INFO:teuthology.orchestra.run.smithi184.stdout:45 2024-09-16T15:46:31.959 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [45] 2024-09-16T15:46:31.960 DEBUG:teuthology.orchestra.run.smithi184:> sudo stdin-killer -- python3 -c ' 2024-09-16T15:46:31.960 DEBUG:teuthology.orchestra.run.smithi184:> import glob 2024-09-16T15:46:31.960 DEBUG:teuthology.orchestra.run.smithi184:> import re 2024-09-16T15:46:31.960 DEBUG:teuthology.orchestra.run.smithi184:> import os 2024-09-16T15:46:31.960 DEBUG:teuthology.orchestra.run.smithi184:> import subprocess 2024-09-16T15:46:31.960 DEBUG:teuthology.orchestra.run.smithi184:> 2024-09-16T15:46:31.960 DEBUG:teuthology.orchestra.run.smithi184:> def _find_admin_socket(client_name): 2024-09-16T15:46:31.960 DEBUG:teuthology.orchestra.run.smithi184:> asok_path = "/var/run/ceph/ceph-client.1.*.asok" 2024-09-16T15:46:31.960 DEBUG:teuthology.orchestra.run.smithi184:> files = glob.glob(asok_path) 2024-09-16T15:46:31.960 DEBUG:teuthology.orchestra.run.smithi184:> mountpoint = "/home/ubuntu/cephtest/mnt.1" 2024-09-16T15:46:31.960 DEBUG:teuthology.orchestra.run.smithi184:> 2024-09-16T15:46:31.960 DEBUG:teuthology.orchestra.run.smithi184:> # Given a non-glob path, it better be there 2024-09-16T15:46:31.960 DEBUG:teuthology.orchestra.run.smithi184:> if "*" not in asok_path: 2024-09-16T15:46:31.961 DEBUG:teuthology.orchestra.run.smithi184:> assert(len(files) == 1) 2024-09-16T15:46:31.961 DEBUG:teuthology.orchestra.run.smithi184:> return files[0] 2024-09-16T15:46:31.961 DEBUG:teuthology.orchestra.run.smithi184:> 2024-09-16T15:46:31.961 DEBUG:teuthology.orchestra.run.smithi184:> for f in files: 2024-09-16T15:46:31.961 DEBUG:teuthology.orchestra.run.smithi184:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-09-16T15:46:31.961 DEBUG:teuthology.orchestra.run.smithi184:> if os.path.exists("/proc/{0}".format(pid)): 2024-09-16T15:46:31.961 DEBUG:teuthology.orchestra.run.smithi184:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-09-16T15:46:31.962 DEBUG:teuthology.orchestra.run.smithi184:> contents = proc_f.read() 2024-09-16T15:46:31.962 DEBUG:teuthology.orchestra.run.smithi184:> if mountpoint in contents: 2024-09-16T15:46:31.962 DEBUG:teuthology.orchestra.run.smithi184:> return f 2024-09-16T15:46:31.962 DEBUG:teuthology.orchestra.run.smithi184:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-09-16T15:46:31.962 DEBUG:teuthology.orchestra.run.smithi184:> 2024-09-16T15:46:31.962 DEBUG:teuthology.orchestra.run.smithi184:> print(_find_admin_socket("client.1")) 2024-09-16T15:46:31.963 DEBUG:teuthology.orchestra.run.smithi184:> ' 2024-09-16T15:46:32.084 INFO:teuthology.orchestra.run.smithi184.stdout:/var/run/ceph/ceph-client.1.35114.asok 2024-09-16T15:46:32.088 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:46:32 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-16T15:46:32.096 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.1.35114.asok 2024-09-16T15:46:32.096 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:46:32.096 DEBUG:teuthology.orchestra.run.smithi184:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.1.35114.asok status 2024-09-16T15:46:32.246 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-16T15:46:32.246 INFO:teuthology.orchestra.run.smithi184.stdout: "metadata": { 2024-09-16T15:46:32.246 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph_sha1": "33f95d2ae1bcbe7eb04404303f179eeeaa6982df", 2024-09-16T15:46:32.246 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph_version": "ceph version 19.3.0-4938-g33f95d2a (33f95d2ae1bcbe7eb04404303f179eeeaa6982df) squid (dev)", 2024-09-16T15:46:32.246 INFO:teuthology.orchestra.run.smithi184.stdout: "entity_id": "1", 2024-09-16T15:46:32.246 INFO:teuthology.orchestra.run.smithi184.stdout: "hostname": "smithi184", 2024-09-16T15:46:32.246 INFO:teuthology.orchestra.run.smithi184.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.1", 2024-09-16T15:46:32.246 INFO:teuthology.orchestra.run.smithi184.stdout: "pid": "35114", 2024-09-16T15:46:32.246 INFO:teuthology.orchestra.run.smithi184.stdout: "root": "/" 2024-09-16T15:46:32.246 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-16T15:46:32.246 INFO:teuthology.orchestra.run.smithi184.stdout: "dentry_count": 0, 2024-09-16T15:46:32.247 INFO:teuthology.orchestra.run.smithi184.stdout: "dentry_pinned_count": 0, 2024-09-16T15:46:32.247 INFO:teuthology.orchestra.run.smithi184.stdout: "id": 5293, 2024-09-16T15:46:32.247 INFO:teuthology.orchestra.run.smithi184.stdout: "inst": { 2024-09-16T15:46:32.247 INFO:teuthology.orchestra.run.smithi184.stdout: "name": { 2024-09-16T15:46:32.247 INFO:teuthology.orchestra.run.smithi184.stdout: "type": "client", 2024-09-16T15:46:32.247 INFO:teuthology.orchestra.run.smithi184.stdout: "num": 5293 2024-09-16T15:46:32.247 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-16T15:46:32.247 INFO:teuthology.orchestra.run.smithi184.stdout: "addr": { 2024-09-16T15:46:32.247 INFO:teuthology.orchestra.run.smithi184.stdout: "type": "v1", 2024-09-16T15:46:32.247 INFO:teuthology.orchestra.run.smithi184.stdout: "addr": "192.168.0.1:0", 2024-09-16T15:46:32.247 INFO:teuthology.orchestra.run.smithi184.stdout: "nonce": 800573190 2024-09-16T15:46:32.247 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-16T15:46:32.247 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-16T15:46:32.247 INFO:teuthology.orchestra.run.smithi184.stdout: "addr": { 2024-09-16T15:46:32.248 INFO:teuthology.orchestra.run.smithi184.stdout: "type": "v1", 2024-09-16T15:46:32.248 INFO:teuthology.orchestra.run.smithi184.stdout: "addr": "192.168.0.1:0", 2024-09-16T15:46:32.248 INFO:teuthology.orchestra.run.smithi184.stdout: "nonce": 800573190 2024-09-16T15:46:32.248 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-16T15:46:32.248 INFO:teuthology.orchestra.run.smithi184.stdout: "inst_str": "client.5293 192.168.0.1:0/800573190", 2024-09-16T15:46:32.248 INFO:teuthology.orchestra.run.smithi184.stdout: "addr_str": "192.168.0.1:0/800573190", 2024-09-16T15:46:32.248 INFO:teuthology.orchestra.run.smithi184.stdout: "inode_count": 1, 2024-09-16T15:46:32.248 INFO:teuthology.orchestra.run.smithi184.stdout: "mds_epoch": 36, 2024-09-16T15:46:32.248 INFO:teuthology.orchestra.run.smithi184.stdout: "osd_epoch": 58, 2024-09-16T15:46:32.248 INFO:teuthology.orchestra.run.smithi184.stdout: "osd_epoch_barrier": 0, 2024-09-16T15:46:32.248 INFO:teuthology.orchestra.run.smithi184.stdout: "blocklisted": false, 2024-09-16T15:46:32.248 INFO:teuthology.orchestra.run.smithi184.stdout: "fs_name": "cephfs" 2024-09-16T15:46:32.248 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-16T15:46:32.261 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:46:32.261 DEBUG:teuthology.orchestra.run.smithi184:> stat --file-system '--printf=%T 2024-09-16T15:46:32.261 DEBUG:teuthology.orchestra.run.smithi184:> ' -- /home/ubuntu/cephtest/mnt.1 2024-09-16T15:46:32.267 INFO:teuthology.orchestra.run.smithi184.stdout:fuseblk 2024-09-16T15:46:32.267 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2024-09-16T15:46:32.267 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:46:32.267 DEBUG:teuthology.orchestra.run.smithi184:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.1 2024-09-16T15:46:32.324 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs flag set enable_multiple true --yes-i-really-mean-it 2024-09-16T15:46:32.463 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:32.461+0000 7f02802cf640 1 -- 172.21.15.123:0/3145337946 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f02780d21a0 msgr2=0x7f02780d0590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:46:32.463 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:32.461+0000 7f02802cf640 1 --2- 172.21.15.123:0/3145337946 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f02780d21a0 0x7f02780d0590 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7f027815ced0 tx=0x7f026801c820 comp rx=0 tx=0).stop 2024-09-16T15:46:32.463 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:32.461+0000 7f02802cf640 1 -- 172.21.15.123:0/3145337946 shutdown_connections 2024-09-16T15:46:32.463 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:32.461+0000 7f02802cf640 1 --2- 172.21.15.123:0/3145337946 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f02780d21a0 0x7f02780d0590 unknown :-1 s=CLOSED pgs=666 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:32.463 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:32.461+0000 7f02802cf640 1 -- 172.21.15.123:0/3145337946 >> 172.21.15.123:0/3145337946 conn(0x7f027809aa40 msgr2=0x7f027809ae50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:46:32.464 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:32.461+0000 7f02802cf640 1 -- 172.21.15.123:0/3145337946 shutdown_connections 2024-09-16T15:46:32.464 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:32.461+0000 7f02802cf640 1 -- 172.21.15.123:0/3145337946 wait complete. 2024-09-16T15:46:32.464 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:32.461+0000 7f02802cf640 1 Processor -- start 2024-09-16T15:46:32.464 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:32.461+0000 7f02802cf640 1 -- start start 2024-09-16T15:46:32.466 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:32.461+0000 7f02802cf640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f02780b2840 0x7f027806f3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:46:32.466 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:32.461+0000 7f02802cf640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f02780d21a0 0x7f02780b69f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:46:32.467 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:32.461+0000 7f02802cf640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f027815c230 0x7f027815e2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:46:32.467 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:32.461+0000 7f02802cf640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f0278169fd0 con 0x7f02780b2840 2024-09-16T15:46:32.467 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:32.461+0000 7f02802cf640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f0278169e50 con 0x7f027815c230 2024-09-16T15:46:32.467 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:32.461+0000 7f02802cf640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f027816a150 con 0x7f02780d21a0 2024-09-16T15:46:32.467 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:32.461+0000 7f027e06b640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f02780b2840 0x7f027806f3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:46:32.467 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:32.461+0000 7f027e06b640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f02780b2840 0x7f027806f3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:39376/0 (socket says 172.21.15.123:39376) 2024-09-16T15:46:32.467 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:32.461+0000 7f027e06b640 1 -- 172.21.15.123:0/2168762792 learned_addr learned my addr 172.21.15.123:0/2168762792 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:46:32.467 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:32.461+0000 7f027e86c640 1 --2- 172.21.15.123:0/2168762792 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f027815c230 0x7f027815e2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:46:32.467 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:32.461+0000 7f027e06b640 1 -- 172.21.15.123:0/2168762792 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f02780d21a0 msgr2=0x7f02780b69f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:46:32.467 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:32.461+0000 7f027e06b640 1 --2- 172.21.15.123:0/2168762792 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f02780d21a0 0x7f02780b69f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:32.468 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:32.461+0000 7f027e06b640 1 -- 172.21.15.123:0/2168762792 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f027815c230 msgr2=0x7f027815e2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:46:32.468 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:32.461+0000 7f027e06b640 1 --2- 172.21.15.123:0/2168762792 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f027815c230 0x7f027815e2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:32.468 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:32.461+0000 7f027e06b640 1 -- 172.21.15.123:0/2168762792 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f027806f900 con 0x7f02780b2840 2024-09-16T15:46:32.468 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:32.461+0000 7f027e86c640 1 --2- 172.21.15.123:0/2168762792 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f027815c230 0x7f027815e2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T15:46:32.468 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:32.461+0000 7f027e06b640 1 --2- 172.21.15.123:0/2168762792 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f02780b2840 0x7f027806f3c0 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7f027400c500 tx=0x7f027400c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:46:32.468 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:32.461+0000 7f026effd640 1 -- 172.21.15.123:0/2168762792 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0274017070 con 0x7f02780b2840 2024-09-16T15:46:32.468 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:32.461+0000 7f026effd640 1 -- 172.21.15.123:0/2168762792 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f02740053c0 con 0x7f02780b2840 2024-09-16T15:46:32.468 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:32.461+0000 7f02802cf640 1 -- 172.21.15.123:0/2168762792 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f027806fbe0 con 0x7f02780b2840 2024-09-16T15:46:32.468 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:32.461+0000 7f026effd640 1 -- 172.21.15.123:0/2168762792 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0274012890 con 0x7f02780b2840 2024-09-16T15:46:32.468 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:32.461+0000 7f02802cf640 1 -- 172.21.15.123:0/2168762792 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f02780b64d0 con 0x7f02780b2840 2024-09-16T15:46:32.468 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:32.461+0000 7f02802cf640 1 -- 172.21.15.123:0/2168762792 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f02780d0590 con 0x7f02780b2840 2024-09-16T15:46:32.469 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:32.465+0000 7f026effd640 1 -- 172.21.15.123:0/2168762792 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 9) ==== 97233+0+0 (secure 0 0 0) 0x7f027401a070 con 0x7f02780b2840 2024-09-16T15:46:32.469 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:32.465+0000 7f026effd640 1 --2- 172.21.15.123:0/2168762792 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f025c073070 0x7f025c075530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:46:32.469 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:32.465+0000 7f026effd640 1 -- 172.21.15.123:0/2168762792 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(58..58 src has 1..58) ==== 5709+0+0 (secure 0 0 0) 0x7f0274093610 con 0x7f02780b2840 2024-09-16T15:46:32.469 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:32.465+0000 7f027d86a640 1 --2- 172.21.15.123:0/2168762792 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f025c073070 0x7f025c075530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:46:32.470 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:32.465+0000 7f027d86a640 1 --2- 172.21.15.123:0/2168762792 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f025c073070 0x7f025c075530 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f02680af000 tx=0x7f02680195c0 comp rx=0 tx=0).ready entity=mgr.4135 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:46:32.472 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:32.469+0000 7f026effd640 1 -- 172.21.15.123:0/2168762792 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f027806fbe0 con 0x7f02780b2840 2024-09-16T15:46:32.681 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:32.677+0000 7f02802cf640 1 -- 172.21.15.123:0/2168762792 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "fs flag set", "flag_name": "enable_multiple", "val": "true", "yes_i_really_mean_it": true} v 0) -- 0x7f0278074a00 con 0x7f02780b2840 2024-09-16T15:46:33.387 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:33.385+0000 7f026effd640 1 -- 172.21.15.123:0/2168762792 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "fs flag set", "flag_name": "enable_multiple", "val": "true", "yes_i_really_mean_it": true}]=0 v37) ==== 136+0+0 (secure 0 0 0) 0x7f027806f900 con 0x7f02780b2840 2024-09-16T15:46:33.391 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:33.385+0000 7f02802cf640 1 -- 172.21.15.123:0/2168762792 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f025c073070 msgr2=0x7f025c075530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:46:33.391 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:33.385+0000 7f02802cf640 1 --2- 172.21.15.123:0/2168762792 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f025c073070 0x7f025c075530 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f02680af000 tx=0x7f02680195c0 comp rx=0 tx=0).stop 2024-09-16T15:46:33.391 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:33.385+0000 7f02802cf640 1 -- 172.21.15.123:0/2168762792 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f02780b2840 msgr2=0x7f027806f3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:46:33.391 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:33.385+0000 7f02802cf640 1 --2- 172.21.15.123:0/2168762792 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f02780b2840 0x7f027806f3c0 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7f027400c500 tx=0x7f027400c9c0 comp rx=0 tx=0).stop 2024-09-16T15:46:33.391 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:33.385+0000 7f02802cf640 1 -- 172.21.15.123:0/2168762792 shutdown_connections 2024-09-16T15:46:33.391 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:33.385+0000 7f02802cf640 1 --2- 172.21.15.123:0/2168762792 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f025c073070 0x7f025c075530 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:33.391 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:33.385+0000 7f02802cf640 1 --2- 172.21.15.123:0/2168762792 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f027815c230 0x7f027815e2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:33.391 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:33.385+0000 7f02802cf640 1 --2- 172.21.15.123:0/2168762792 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f02780d21a0 0x7f02780b69f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:33.391 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:33.385+0000 7f02802cf640 1 --2- 172.21.15.123:0/2168762792 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f02780b2840 0x7f027806f3c0 unknown :-1 s=CLOSED pgs=667 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:33.391 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:33.385+0000 7f02802cf640 1 -- 172.21.15.123:0/2168762792 >> 172.21.15.123:0/2168762792 conn(0x7f027809aa40 msgr2=0x7f0278167300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:46:33.391 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:33.389+0000 7f02802cf640 1 -- 172.21.15.123:0/2168762792 shutdown_connections 2024-09-16T15:46:33.391 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:33.389+0000 7f02802cf640 1 -- 172.21.15.123:0/2168762792 wait complete. 2024-09-16T15:46:33.410 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2024-09-16T15:46:33.536 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:33.533+0000 7fd041b17640 1 -- 172.21.15.123:0/4071274514 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd03c15bca0 msgr2=0x7fd03c0d0630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:46:33.537 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:33.533+0000 7fd041b17640 1 --2- 172.21.15.123:0/4071274514 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd03c15bca0 0x7fd03c0d0630 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7fd03c076230 tx=0x7fd03401ca00 comp rx=0 tx=0).stop 2024-09-16T15:46:33.537 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:33.533+0000 7fd041b17640 1 -- 172.21.15.123:0/4071274514 shutdown_connections 2024-09-16T15:46:33.537 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:33.533+0000 7fd041b17640 1 --2- 172.21.15.123:0/4071274514 >> v2:172.21.15.184:3300/0 conn(0x7fd03c074430 0x7fd03c0ca160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:33.537 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:33.533+0000 7fd041b17640 1 --2- 172.21.15.123:0/4071274514 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fd03c0726c0 0x7fd03c073ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:33.537 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:33.533+0000 7fd041b17640 1 --2- 172.21.15.123:0/4071274514 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd03c15bca0 0x7fd03c0d0630 unknown :-1 s=CLOSED pgs=668 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:33.537 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:33.533+0000 7fd041b17640 1 -- 172.21.15.123:0/4071274514 >> 172.21.15.123:0/4071274514 conn(0x7fd03c0cfb10 msgr2=0x7fd03c0cff20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:46:33.537 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:33.533+0000 7fd041b17640 1 -- 172.21.15.123:0/4071274514 shutdown_connections 2024-09-16T15:46:33.537 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:33.533+0000 7fd041b17640 1 -- 172.21.15.123:0/4071274514 wait complete. 2024-09-16T15:46:33.539 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:33.533+0000 7fd041b17640 1 Processor -- start 2024-09-16T15:46:33.540 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:33.533+0000 7fd041b17640 1 -- start start 2024-09-16T15:46:33.540 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:33.533+0000 7fd041b17640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd03c0726c0 0x7fd03c0c8180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:46:33.540 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:33.533+0000 7fd041b17640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd03c074430 0x7fd03c151120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:46:33.540 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:33.533+0000 7fd041b17640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fd03c15bca0 0x7fd03c151850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:46:33.540 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:33.533+0000 7fd041b17640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fd03c06ec20 con 0x7fd03c0726c0 2024-09-16T15:46:33.540 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:33.533+0000 7fd041b17640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fd03c06eaa0 con 0x7fd03c074430 2024-09-16T15:46:33.540 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:33.533+0000 7fd041b17640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fd03c06eda0 con 0x7fd03c15bca0 2024-09-16T15:46:33.540 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:33.533+0000 7fd03b7fe640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd03c0726c0 0x7fd03c0c8180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:46:33.540 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:33.533+0000 7fd03b7fe640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd03c0726c0 0x7fd03c0c8180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:39396/0 (socket says 172.21.15.123:39396) 2024-09-16T15:46:33.540 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:33.533+0000 7fd03b7fe640 1 -- 172.21.15.123:0/1730253703 learned_addr learned my addr 172.21.15.123:0/1730253703 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:46:33.541 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:33.537+0000 7fd03affd640 1 --2- 172.21.15.123:0/1730253703 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd03c074430 0x7fd03c151120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:46:33.541 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:33.537+0000 7fd03bfff640 1 --2- 172.21.15.123:0/1730253703 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fd03c15bca0 0x7fd03c151850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:46:33.541 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:33.537+0000 7fd03b7fe640 1 -- 172.21.15.123:0/1730253703 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fd03c15bca0 msgr2=0x7fd03c151850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:46:33.541 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:33.537+0000 7fd03b7fe640 1 --2- 172.21.15.123:0/1730253703 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fd03c15bca0 0x7fd03c151850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:33.541 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:33.537+0000 7fd03b7fe640 1 -- 172.21.15.123:0/1730253703 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd03c074430 msgr2=0x7fd03c151120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:46:33.541 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:33.537+0000 7fd03b7fe640 1 --2- 172.21.15.123:0/1730253703 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd03c074430 0x7fd03c151120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:33.541 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:33.537+0000 7fd03b7fe640 1 -- 172.21.15.123:0/1730253703 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd03c151660 con 0x7fd03c0726c0 2024-09-16T15:46:33.541 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:33.537+0000 7fd03bfff640 1 --2- 172.21.15.123:0/1730253703 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fd03c15bca0 0x7fd03c151850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T15:46:33.541 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:33.537+0000 7fd03b7fe640 1 --2- 172.21.15.123:0/1730253703 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd03c0726c0 0x7fd03c0c8180 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7fd034000f90 tx=0x7fd0340a5880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:46:33.541 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:33.537+0000 7fd033fff640 1 -- 172.21.15.123:0/1730253703 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0340bb070 con 0x7fd03c0726c0 2024-09-16T15:46:33.541 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:33.537+0000 7fd033fff640 1 -- 172.21.15.123:0/1730253703 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fd034004e00 con 0x7fd03c0726c0 2024-09-16T15:46:33.541 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:33.537+0000 7fd041b17640 1 -- 172.21.15.123:0/1730253703 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd03c0b49b0 con 0x7fd03c0726c0 2024-09-16T15:46:33.541 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:33.537+0000 7fd033fff640 1 -- 172.21.15.123:0/1730253703 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0340b2040 con 0x7fd03c0726c0 2024-09-16T15:46:33.541 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:33.537+0000 7fd041b17640 1 -- 172.21.15.123:0/1730253703 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd03c1551a0 con 0x7fd03c0726c0 2024-09-16T15:46:33.542 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:33.537+0000 7fd041b17640 1 -- 172.21.15.123:0/1730253703 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd03c0ca160 con 0x7fd03c0726c0 2024-09-16T15:46:33.542 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:33.537+0000 7fd033fff640 1 -- 172.21.15.123:0/1730253703 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 9) ==== 97233+0+0 (secure 0 0 0) 0x7fd0340be070 con 0x7fd03c0726c0 2024-09-16T15:46:33.543 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:33.537+0000 7fd033fff640 1 --2- 172.21.15.123:0/1730253703 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fd01c073070 0x7fd01c075530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:46:33.543 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:33.537+0000 7fd033fff640 1 -- 172.21.15.123:0/1730253703 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(58..58 src has 1..58) ==== 5709+0+0 (secure 0 0 0) 0x7fd03412e220 con 0x7fd03c0726c0 2024-09-16T15:46:33.543 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:33.537+0000 7fd03affd640 1 --2- 172.21.15.123:0/1730253703 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fd01c073070 0x7fd01c075530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:46:33.543 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:33.537+0000 7fd03affd640 1 --2- 172.21.15.123:0/1730253703 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fd01c073070 0x7fd01c075530 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7fd03c158f70 tx=0x7fd02400a4e0 comp rx=0 tx=0).ready entity=mgr.4135 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:46:33.545 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:33.541+0000 7fd033fff640 1 -- 172.21.15.123:0/1730253703 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7fd034016970 con 0x7fd03c0726c0 2024-09-16T15:46:33.735 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:33.733+0000 7fd041b17640 1 -- 172.21.15.123:0/1730253703 --> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) -- 0x7fd03c0700e0 con 0x7fd01c073070 2024-09-16T15:46:33.738 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:33.733+0000 7fd033fff640 1 -- 172.21.15.123:0/1730253703 <== mgr.4135 v2:172.21.15.184:6832/3114346423 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+41 (secure 0 0 0) 0x7fd03c0700e0 con 0x7fd01c073070 2024-09-16T15:46:33.739 INFO:teuthology.orchestra.run.smithi123.stdout:[ 2024-09-16T15:46:33.739 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-09-16T15:46:33.739 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "cephfs" 2024-09-16T15:46:33.739 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-16T15:46:33.739 INFO:teuthology.orchestra.run.smithi123.stdout:] 2024-09-16T15:46:33.741 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:33.737+0000 7fd041b17640 1 -- 172.21.15.123:0/1730253703 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fd01c073070 msgr2=0x7fd01c075530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:46:33.741 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:33.737+0000 7fd041b17640 1 --2- 172.21.15.123:0/1730253703 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fd01c073070 0x7fd01c075530 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7fd03c158f70 tx=0x7fd02400a4e0 comp rx=0 tx=0).stop 2024-09-16T15:46:33.741 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:33.737+0000 7fd041b17640 1 -- 172.21.15.123:0/1730253703 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd03c0726c0 msgr2=0x7fd03c0c8180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:46:33.741 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:33.737+0000 7fd041b17640 1 --2- 172.21.15.123:0/1730253703 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd03c0726c0 0x7fd03c0c8180 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7fd034000f90 tx=0x7fd0340a5880 comp rx=0 tx=0).stop 2024-09-16T15:46:33.741 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:33.737+0000 7fd041b17640 1 -- 172.21.15.123:0/1730253703 shutdown_connections 2024-09-16T15:46:33.741 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:33.737+0000 7fd041b17640 1 --2- 172.21.15.123:0/1730253703 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fd01c073070 0x7fd01c075530 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:33.741 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:33.737+0000 7fd041b17640 1 --2- 172.21.15.123:0/1730253703 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fd03c15bca0 0x7fd03c151850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:33.742 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:33.737+0000 7fd041b17640 1 --2- 172.21.15.123:0/1730253703 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd03c074430 0x7fd03c151120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:33.742 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:33.737+0000 7fd041b17640 1 --2- 172.21.15.123:0/1730253703 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd03c0726c0 0x7fd03c0c8180 unknown :-1 s=CLOSED pgs=669 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:33.742 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:33.737+0000 7fd041b17640 1 -- 172.21.15.123:0/1730253703 >> 172.21.15.123:0/1730253703 conn(0x7fd03c0cfb10 msgr2=0x7fd03c0c3480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:46:33.742 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:33.737+0000 7fd041b17640 1 -- 172.21.15.123:0/1730253703 shutdown_connections 2024-09-16T15:46:33.742 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:33.737+0000 7fd041b17640 1 -- 172.21.15.123:0/1730253703 wait complete. 2024-09-16T15:46:33.756 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-09-16T15:46:33.883 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:33.881+0000 7fd2492b1640 1 -- 172.21.15.123:0/1784604605 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd2440d0510 msgr2=0x7fd2440d0980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:46:33.883 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:33.881+0000 7fd2492b1640 1 --2- 172.21.15.123:0/1784604605 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd2440d0510 0x7fd2440d0980 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7fd23c009c80 tx=0x7fd23c01cda0 comp rx=0 tx=0).stop 2024-09-16T15:46:33.883 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:33.881+0000 7fd2492b1640 1 -- 172.21.15.123:0/1784604605 shutdown_connections 2024-09-16T15:46:33.883 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:33.881+0000 7fd2492b1640 1 --2- 172.21.15.123:0/1784604605 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd2440d0510 0x7fd2440d0980 unknown :-1 s=CLOSED pgs=670 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:33.884 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:33.881+0000 7fd2492b1640 1 --2- 172.21.15.123:0/1784604605 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fd2440cc450 0x7fd24415b410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:33.884 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:33.881+0000 7fd2492b1640 1 -- 172.21.15.123:0/1784604605 >> 172.21.15.123:0/1784604605 conn(0x7fd24406d670 msgr2=0x7fd2440743a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:46:33.884 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:33.881+0000 7fd2492b1640 1 -- 172.21.15.123:0/1784604605 shutdown_connections 2024-09-16T15:46:33.884 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:33.881+0000 7fd2492b1640 1 -- 172.21.15.123:0/1784604605 wait complete. 2024-09-16T15:46:33.885 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:33.881+0000 7fd2492b1640 1 Processor -- start 2024-09-16T15:46:33.886 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:33.881+0000 7fd2492b1640 1 -- start start 2024-09-16T15:46:33.886 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:33.881+0000 7fd2492b1640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fd2440cc070 0x7fd244099540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:46:33.886 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:33.881+0000 7fd2492b1640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd2440cc450 0x7fd2440c9eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:46:33.886 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:33.881+0000 7fd2492b1640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd2440d0510 0x7fd244152310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:46:33.886 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:33.881+0000 7fd2492b1640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fd24415a4f0 con 0x7fd2440d0510 2024-09-16T15:46:33.886 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:33.881+0000 7fd2492b1640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fd24415a370 con 0x7fd2440cc450 2024-09-16T15:46:33.886 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:33.881+0000 7fd2492b1640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fd24415a670 con 0x7fd2440cc070 2024-09-16T15:46:33.886 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:33.881+0000 7fd2437fe640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd2440d0510 0x7fd244152310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:46:33.886 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:33.881+0000 7fd2437fe640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd2440d0510 0x7fd244152310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:39406/0 (socket says 172.21.15.123:39406) 2024-09-16T15:46:33.886 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:33.881+0000 7fd2437fe640 1 -- 172.21.15.123:0/38940481 learned_addr learned my addr 172.21.15.123:0/38940481 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:46:33.887 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:33.881+0000 7fd23bfff640 1 --2- 172.21.15.123:0/38940481 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd2440cc450 0x7fd2440c9eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:46:33.887 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:33.881+0000 7fd2437fe640 1 -- 172.21.15.123:0/38940481 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fd2440cc070 msgr2=0x7fd244099540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:46:33.887 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:33.881+0000 7fd2437fe640 1 --2- 172.21.15.123:0/38940481 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fd2440cc070 0x7fd244099540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:33.887 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:33.881+0000 7fd2437fe640 1 -- 172.21.15.123:0/38940481 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd2440cc450 msgr2=0x7fd2440c9eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:46:33.887 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:33.881+0000 7fd2437fe640 1 --2- 172.21.15.123:0/38940481 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd2440cc450 0x7fd2440c9eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:33.887 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:33.881+0000 7fd2437fe640 1 -- 172.21.15.123:0/38940481 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd2440c78a0 con 0x7fd2440d0510 2024-09-16T15:46:33.887 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:33.881+0000 7fd2437fe640 1 --2- 172.21.15.123:0/38940481 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd2440d0510 0x7fd244152310 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7fd23c009bd0 tx=0x7fd23c009350 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:46:33.887 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:33.881+0000 7fd23b7fe640 1 -- 172.21.15.123:0/38940481 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd23c0bb070 con 0x7fd2440d0510 2024-09-16T15:46:33.887 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:33.881+0000 7fd23b7fe640 1 -- 172.21.15.123:0/38940481 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fd23c004d40 con 0x7fd2440d0510 2024-09-16T15:46:33.887 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:33.881+0000 7fd2492b1640 1 -- 172.21.15.123:0/38940481 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd2440b5f30 con 0x7fd2440d0510 2024-09-16T15:46:33.887 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:33.881+0000 7fd23b7fe640 1 -- 172.21.15.123:0/38940481 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd23c0b2040 con 0x7fd2440d0510 2024-09-16T15:46:33.887 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:33.881+0000 7fd2492b1640 1 -- 172.21.15.123:0/38940481 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd2440b04d0 con 0x7fd2440d0510 2024-09-16T15:46:33.888 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:33.881+0000 7fd2492b1640 1 -- 172.21.15.123:0/38940481 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd24415b410 con 0x7fd2440d0510 2024-09-16T15:46:33.888 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:33.885+0000 7fd23b7fe640 1 -- 172.21.15.123:0/38940481 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 9) ==== 97233+0+0 (secure 0 0 0) 0x7fd23c0be070 con 0x7fd2440d0510 2024-09-16T15:46:33.888 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:33.885+0000 7fd23b7fe640 1 --2- 172.21.15.123:0/38940481 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fd2240730a0 0x7fd224075560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:46:33.888 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:33.885+0000 7fd23b7fe640 1 -- 172.21.15.123:0/38940481 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(58..58 src has 1..58) ==== 5709+0+0 (secure 0 0 0) 0x7fd23c12dd70 con 0x7fd2440d0510 2024-09-16T15:46:33.889 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:33.885+0000 7fd242ffd640 1 --2- 172.21.15.123:0/38940481 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fd2240730a0 0x7fd224075560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:46:33.889 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:33.885+0000 7fd242ffd640 1 --2- 172.21.15.123:0/38940481 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fd2240730a0 0x7fd224075560 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7fd234003cf0 tx=0x7fd2340079c0 comp rx=0 tx=0).ready entity=mgr.4135 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:46:33.892 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:33.889+0000 7fd23b7fe640 1 -- 172.21.15.123:0/38940481 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7fd23c016990 con 0x7fd2440d0510 2024-09-16T15:46:34.064 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:34.061+0000 7fd2492b1640 1 -- 172.21.15.123:0/38940481 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) -- 0x7fd2440d11d0 con 0x7fd2440d0510 2024-09-16T15:46:34.065 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:34.061+0000 7fd23b7fe640 1 -- 172.21.15.123:0/38940481 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v32)=0 v32) ==== 122+0+0 (secure 0 0 0) 0x7fd23c0f8ba0 con 0x7fd2440d0510 2024-09-16T15:46:34.067 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:34.065+0000 7fd2492b1640 1 -- 172.21.15.123:0/38940481 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fd2240730a0 msgr2=0x7fd224075560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:46:34.067 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:34.065+0000 7fd2492b1640 1 --2- 172.21.15.123:0/38940481 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fd2240730a0 0x7fd224075560 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7fd234003cf0 tx=0x7fd2340079c0 comp rx=0 tx=0).stop 2024-09-16T15:46:34.067 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:34.065+0000 7fd2492b1640 1 -- 172.21.15.123:0/38940481 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd2440d0510 msgr2=0x7fd244152310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:46:34.068 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:34.065+0000 7fd2492b1640 1 --2- 172.21.15.123:0/38940481 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd2440d0510 0x7fd244152310 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7fd23c009bd0 tx=0x7fd23c009350 comp rx=0 tx=0).stop 2024-09-16T15:46:34.068 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:34.065+0000 7fd2492b1640 1 -- 172.21.15.123:0/38940481 shutdown_connections 2024-09-16T15:46:34.068 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:34.065+0000 7fd2492b1640 1 --2- 172.21.15.123:0/38940481 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fd2240730a0 0x7fd224075560 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:34.068 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:34.065+0000 7fd2492b1640 1 --2- 172.21.15.123:0/38940481 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd2440d0510 0x7fd244152310 unknown :-1 s=CLOSED pgs=671 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:34.068 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:34.065+0000 7fd2492b1640 1 --2- 172.21.15.123:0/38940481 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd2440cc450 0x7fd2440c9eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:34.068 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:34.065+0000 7fd2492b1640 1 --2- 172.21.15.123:0/38940481 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fd2440cc070 0x7fd244099540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:34.068 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:34.065+0000 7fd2492b1640 1 -- 172.21.15.123:0/38940481 >> 172.21.15.123:0/38940481 conn(0x7fd24406d670 msgr2=0x7fd244159320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:46:34.068 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:34.065+0000 7fd2492b1640 1 -- 172.21.15.123:0/38940481 shutdown_connections 2024-09-16T15:46:34.068 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:34.065+0000 7fd2492b1640 1 -- 172.21.15.123:0/38940481 wait complete. 2024-09-16T15:46:34.084 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:46:34.084 DEBUG:teuthology.orchestra.run.smithi123:> (cd /home/ubuntu/cephtest/mnt.0 && exec sudo mkdir -p ./volumes/_nogroup/subvol14360) 2024-09-16T15:46:34.106 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:46:34.106 DEBUG:teuthology.orchestra.run.smithi123:> (cd /home/ubuntu/cephtest/mnt.0 && exec getfattr --only-values -n ceph.dir.layout.pool .) 2024-09-16T15:46:34.153 INFO:teuthology.orchestra.run.smithi123.stdout:cephfs_data 2024-09-16T15:46:34.154 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:46:34.154 DEBUG:teuthology.orchestra.run.smithi123:> (cd /home/ubuntu/cephtest/mnt.0 && exec sudo setfattr -n ceph.dir.layout.pool -v cephfs_data ./volumes/_nogroup/subvol14360) 2024-09-16T15:46:34.213 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume create cephfs subvol2021 2024-09-16T15:46:34.385 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:34.381+0000 7f3f50de2640 1 -- 172.21.15.123:0/1833956069 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3f38003a10 msgr2=0x7f3f38003e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:46:34.385 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:34.381+0000 7f3f50de2640 1 --2- 172.21.15.123:0/1833956069 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3f38003a10 0x7f3f38003e10 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7f3f44009ef0 tx=0x7f3f4401d1a0 comp rx=0 tx=0).stop 2024-09-16T15:46:34.385 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:34.381+0000 7f3f50de2640 1 -- 172.21.15.123:0/1833956069 shutdown_connections 2024-09-16T15:46:34.385 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:34.381+0000 7f3f50de2640 1 --2- 172.21.15.123:0/1833956069 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3f38003a10 0x7f3f38003e10 unknown :-1 s=CLOSED pgs=673 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:34.385 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:34.381+0000 7f3f50de2640 1 -- 172.21.15.123:0/1833956069 >> 172.21.15.123:0/1833956069 conn(0x7f3f4c0b1710 msgr2=0x7f3f4c0b1b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:46:34.385 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:34.381+0000 7f3f50de2640 1 -- 172.21.15.123:0/1833956069 shutdown_connections 2024-09-16T15:46:34.385 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:34.381+0000 7f3f50de2640 1 -- 172.21.15.123:0/1833956069 wait complete. 2024-09-16T15:46:34.386 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:34.381+0000 7f3f50de2640 1 Processor -- start 2024-09-16T15:46:34.386 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:34.381+0000 7f3f50de2640 1 -- start start 2024-09-16T15:46:34.388 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:34.381+0000 7f3f50de2640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3f38003a10 0x7f3f4c151970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:46:34.388 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:34.381+0000 7f3f50de2640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f3f4c151f80 0x7f3f4c1fb380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:46:34.388 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:34.381+0000 7f3f50de2640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3f4c0c8f70 0x7f3f4c1fb8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:46:34.388 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:34.381+0000 7f3f50de2640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f3f4c15f560 con 0x7f3f38003a10 2024-09-16T15:46:34.388 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:34.381+0000 7f3f50de2640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f3f4c15f3e0 con 0x7f3f4c0c8f70 2024-09-16T15:46:34.388 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:34.381+0000 7f3f50de2640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f3f4c06ebe0 con 0x7f3f4c151f80 2024-09-16T15:46:34.389 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:34.385+0000 7f3f4a59c640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3f38003a10 0x7f3f4c151970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:46:34.389 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:34.385+0000 7f3f4a59c640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3f38003a10 0x7f3f4c151970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:39416/0 (socket says 172.21.15.123:39416) 2024-09-16T15:46:34.389 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:34.385+0000 7f3f4a59c640 1 -- 172.21.15.123:0/2338049691 learned_addr learned my addr 172.21.15.123:0/2338049691 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:46:34.389 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:34.385+0000 7f3f49d9b640 1 --2- 172.21.15.123:0/2338049691 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f3f4c151f80 0x7f3f4c1fb380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:46:34.389 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:34.385+0000 7f3f4ad9d640 1 --2- 172.21.15.123:0/2338049691 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3f4c0c8f70 0x7f3f4c1fb8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:46:34.389 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:34.385+0000 7f3f4a59c640 1 -- 172.21.15.123:0/2338049691 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f3f4c151f80 msgr2=0x7f3f4c1fb380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:46:34.389 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:34.385+0000 7f3f4a59c640 1 --2- 172.21.15.123:0/2338049691 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f3f4c151f80 0x7f3f4c1fb380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:34.389 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:34.385+0000 7f3f4a59c640 1 -- 172.21.15.123:0/2338049691 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3f4c0c8f70 msgr2=0x7f3f4c1fb8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:46:34.389 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:34.385+0000 7f3f4a59c640 1 --2- 172.21.15.123:0/2338049691 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3f4c0c8f70 0x7f3f4c1fb8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:34.389 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:34.385+0000 7f3f4a59c640 1 -- 172.21.15.123:0/2338049691 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3f4c0c9770 con 0x7f3f38003a10 2024-09-16T15:46:34.390 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:34.385+0000 7f3f49d9b640 1 --2- 172.21.15.123:0/2338049691 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f3f4c151f80 0x7f3f4c1fb380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T15:46:34.390 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:34.385+0000 7f3f4a59c640 1 --2- 172.21.15.123:0/2338049691 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3f38003a10 0x7f3f4c151970 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7f3f440ae040 tx=0x7f3f440aca00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:46:34.390 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:34.385+0000 7f3f42ffd640 1 -- 172.21.15.123:0/2338049691 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f440bb030 con 0x7f3f38003a10 2024-09-16T15:46:34.390 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:34.385+0000 7f3f50de2640 1 -- 172.21.15.123:0/2338049691 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3f4c1fbf50 con 0x7f3f38003a10 2024-09-16T15:46:34.390 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:34.385+0000 7f3f42ffd640 1 -- 172.21.15.123:0/2338049691 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f3f44003980 con 0x7f3f38003a10 2024-09-16T15:46:34.390 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:34.385+0000 7f3f42ffd640 1 -- 172.21.15.123:0/2338049691 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f440ace10 con 0x7f3f38003a10 2024-09-16T15:46:34.390 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:34.385+0000 7f3f50de2640 1 -- 172.21.15.123:0/2338049691 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3f4c152740 con 0x7f3f38003a10 2024-09-16T15:46:34.390 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:34.385+0000 7f3f50de2640 1 -- 172.21.15.123:0/2338049691 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3f4c0c9df0 con 0x7f3f38003a10 2024-09-16T15:46:34.391 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:34.385+0000 7f3f42ffd640 1 -- 172.21.15.123:0/2338049691 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 9) ==== 97233+0+0 (secure 0 0 0) 0x7f3f440b5040 con 0x7f3f38003a10 2024-09-16T15:46:34.391 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:34.385+0000 7f3f42ffd640 1 --2- 172.21.15.123:0/2338049691 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f3f2c0730a0 0x7f3f2c075560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:46:34.391 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:34.385+0000 7f3f42ffd640 1 -- 172.21.15.123:0/2338049691 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(58..58 src has 1..58) ==== 5709+0+0 (secure 0 0 0) 0x7f3f4412f360 con 0x7f3f38003a10 2024-09-16T15:46:34.391 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:34.385+0000 7f3f49d9b640 1 --2- 172.21.15.123:0/2338049691 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f3f2c0730a0 0x7f3f2c075560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:46:34.391 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:34.389+0000 7f3f49d9b640 1 --2- 172.21.15.123:0/2338049691 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f3f2c0730a0 0x7f3f2c075560 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f3f4c1529d0 tx=0x7f3f34009040 comp rx=0 tx=0).ready entity=mgr.4135 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:46:34.395 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:34.393+0000 7f3f42ffd640 1 -- 172.21.15.123:0/2338049691 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f3f4c0c9df0 con 0x7f3f38003a10 2024-09-16T15:46:34.590 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:34.585+0000 7f3f50de2640 1 -- 172.21.15.123:0/2338049691 --> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "subvol2021", "target": ["mon-mgr", ""]}) -- 0x7f3f4c0ca9a0 con 0x7f3f2c0730a0 2024-09-16T15:46:34.600 INFO:tasks.ceph.mgr.x.smithi184.stderr:2024-09-16T15:46:34.598+0000 7fda6a8d4640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-16T15:46:34.600 INFO:tasks.ceph.mgr.x.smithi184.stderr:2024-09-16T15:46:34.598+0000 7fda6a8d4640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-16T15:46:34.601 INFO:tasks.ceph.mgr.x.smithi184.stderr:2024-09-16T15:46:34.598+0000 7fda6a8d4640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-16T15:46:34.601 INFO:tasks.ceph.mgr.x.smithi184.stderr:2024-09-16T15:46:34.598+0000 7fda6a8d4640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-16T15:46:34.601 INFO:tasks.ceph.mgr.x.smithi184.stderr:2024-09-16T15:46:34.598+0000 7fda6a8d4640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-16T15:46:34.662 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:34.657+0000 7f3f42ffd640 1 -- 172.21.15.123:0/2338049691 <== mgr.4135 v2:172.21.15.184:6832/3114346423 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f3f4c0ca9a0 con 0x7f3f2c0730a0 2024-09-16T15:46:34.665 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:34.661+0000 7f3f50de2640 1 -- 172.21.15.123:0/2338049691 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f3f2c0730a0 msgr2=0x7f3f2c075560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:46:34.665 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:34.661+0000 7f3f50de2640 1 --2- 172.21.15.123:0/2338049691 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f3f2c0730a0 0x7f3f2c075560 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f3f4c1529d0 tx=0x7f3f34009040 comp rx=0 tx=0).stop 2024-09-16T15:46:34.665 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:34.661+0000 7f3f50de2640 1 -- 172.21.15.123:0/2338049691 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3f38003a10 msgr2=0x7f3f4c151970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:46:34.665 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:34.661+0000 7f3f50de2640 1 --2- 172.21.15.123:0/2338049691 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3f38003a10 0x7f3f4c151970 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7f3f440ae040 tx=0x7f3f440aca00 comp rx=0 tx=0).stop 2024-09-16T15:46:34.665 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:34.661+0000 7f3f50de2640 1 -- 172.21.15.123:0/2338049691 shutdown_connections 2024-09-16T15:46:34.666 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:34.661+0000 7f3f50de2640 1 --2- 172.21.15.123:0/2338049691 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f3f2c0730a0 0x7f3f2c075560 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:34.666 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:34.661+0000 7f3f50de2640 1 --2- 172.21.15.123:0/2338049691 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3f4c0c8f70 0x7f3f4c1fb8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:34.666 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:34.661+0000 7f3f50de2640 1 --2- 172.21.15.123:0/2338049691 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f3f4c151f80 0x7f3f4c1fb380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:34.666 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:34.661+0000 7f3f50de2640 1 --2- 172.21.15.123:0/2338049691 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3f38003a10 0x7f3f4c151970 unknown :-1 s=CLOSED pgs=674 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:34.666 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:34.661+0000 7f3f50de2640 1 -- 172.21.15.123:0/2338049691 >> 172.21.15.123:0/2338049691 conn(0x7f3f4c0b1710 msgr2=0x7f3f4c15c540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:46:34.666 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:34.661+0000 7f3f50de2640 1 -- 172.21.15.123:0/2338049691 shutdown_connections 2024-09-16T15:46:34.666 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:34.661+0000 7f3f50de2640 1 -- 172.21.15.123:0/2338049691 wait complete. 2024-09-16T15:46:34.680 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:46:34.680 DEBUG:teuthology.orchestra.run.smithi123:> (cd /home/ubuntu/cephtest/mnt.0 && exec sudo cp ./volumes/_nogroup/subvol2021/.meta ./volumes/_nogroup/subvol14360) 2024-09-16T15:46:35.827 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs subvol14360 2024-09-16T15:46:35.964 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:35.961+0000 7f2bffcad640 1 -- 172.21.15.123:0/2455426303 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2be4003a60 msgr2=0x7f2be4003e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:46:35.964 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:35.961+0000 7f2bffcad640 1 --2- 172.21.15.123:0/2455426303 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2be4003a60 0x7f2be4003e80 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7f2be800bb10 tx=0x7f2be801cda0 comp rx=0 tx=0).stop 2024-09-16T15:46:35.964 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:35.961+0000 7f2bffcad640 1 -- 172.21.15.123:0/2455426303 shutdown_connections 2024-09-16T15:46:35.965 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:35.961+0000 7f2bffcad640 1 --2- 172.21.15.123:0/2455426303 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2be4003a60 0x7f2be4003e80 unknown :-1 s=CLOSED pgs=676 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:35.965 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:35.961+0000 7f2bffcad640 1 -- 172.21.15.123:0/2455426303 >> 172.21.15.123:0/2455426303 conn(0x7f2bf80b1710 msgr2=0x7f2bf80b1b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:46:35.965 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:35.961+0000 7f2bffcad640 1 -- 172.21.15.123:0/2455426303 shutdown_connections 2024-09-16T15:46:35.965 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:35.961+0000 7f2bffcad640 1 -- 172.21.15.123:0/2455426303 wait complete. 2024-09-16T15:46:35.965 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:35.961+0000 7f2bffcad640 1 Processor -- start 2024-09-16T15:46:35.965 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:35.961+0000 7f2bffcad640 1 -- start start 2024-09-16T15:46:35.968 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:35.961+0000 7f2bffcad640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2be4003a60 0x7f2bf814e040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:46:35.969 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:35.961+0000 7f2bffcad640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2bf81f7900 0x7f2bf81fa710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:46:35.969 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:35.961+0000 7f2bffcad640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f2bf80c8c20 0x7f2bf80c90d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:46:35.969 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:35.961+0000 7f2bffcad640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f2bf8072ea0 con 0x7f2bf81f7900 2024-09-16T15:46:35.969 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:35.961+0000 7f2bffcad640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f2bf8072d20 con 0x7f2be4003a60 2024-09-16T15:46:35.969 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:35.961+0000 7f2bffcad640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f2bf8073020 con 0x7f2bf80c8c20 2024-09-16T15:46:35.969 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:35.961+0000 7f2bfd248640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2bf81f7900 0x7f2bf81fa710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:46:35.969 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:35.961+0000 7f2bfd248640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2bf81f7900 0x7f2bf81fa710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:39426/0 (socket says 172.21.15.123:39426) 2024-09-16T15:46:35.969 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:35.961+0000 7f2bfd248640 1 -- 172.21.15.123:0/2680963825 learned_addr learned my addr 172.21.15.123:0/2680963825 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:46:35.969 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:35.961+0000 7f2bfda49640 1 --2- 172.21.15.123:0/2680963825 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2be4003a60 0x7f2bf814e040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:46:35.969 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:35.961+0000 7f2bfe24a640 1 --2- 172.21.15.123:0/2680963825 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f2bf80c8c20 0x7f2bf80c90d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:46:35.969 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:35.965+0000 7f2bfd248640 1 -- 172.21.15.123:0/2680963825 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f2bf80c8c20 msgr2=0x7f2bf80c90d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:46:35.970 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:35.965+0000 7f2bfd248640 1 --2- 172.21.15.123:0/2680963825 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f2bf80c8c20 0x7f2bf80c90d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:35.970 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:35.965+0000 7f2bfd248640 1 -- 172.21.15.123:0/2680963825 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2be4003a60 msgr2=0x7f2bf814e040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:46:35.970 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:35.965+0000 7f2bfd248640 1 --2- 172.21.15.123:0/2680963825 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2be4003a60 0x7f2bf814e040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:35.970 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:35.965+0000 7f2bfd248640 1 -- 172.21.15.123:0/2680963825 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2bf814e580 con 0x7f2bf81f7900 2024-09-16T15:46:35.970 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:35.965+0000 7f2bfd248640 1 --2- 172.21.15.123:0/2680963825 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2bf81f7900 0x7f2bf81fa710 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7f2bf4003ed0 tx=0x7f2bf4001640 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:46:35.970 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:35.965+0000 7f2bee7fc640 1 -- 172.21.15.123:0/2680963825 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2bf4015070 con 0x7f2bf81f7900 2024-09-16T15:46:35.970 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:35.965+0000 7f2bee7fc640 1 -- 172.21.15.123:0/2680963825 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f2bf4011840 con 0x7f2bf81f7900 2024-09-16T15:46:35.970 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:35.965+0000 7f2bffcad640 1 -- 172.21.15.123:0/2680963825 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2bf81fac50 con 0x7f2bf81f7900 2024-09-16T15:46:35.970 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:35.965+0000 7f2bee7fc640 1 -- 172.21.15.123:0/2680963825 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2bf4011c90 con 0x7f2bf81f7900 2024-09-16T15:46:35.970 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:35.965+0000 7f2bffcad640 1 -- 172.21.15.123:0/2680963825 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2bf80c4b80 con 0x7f2bf81f7900 2024-09-16T15:46:35.970 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:35.965+0000 7f2bffcad640 1 -- 172.21.15.123:0/2680963825 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2bf815f700 con 0x7f2bf81f7900 2024-09-16T15:46:35.970 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:35.965+0000 7f2bee7fc640 1 -- 172.21.15.123:0/2680963825 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 9) ==== 97233+0+0 (secure 0 0 0) 0x7f2bf4018070 con 0x7f2bf81f7900 2024-09-16T15:46:35.971 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:35.965+0000 7f2bee7fc640 1 --2- 172.21.15.123:0/2680963825 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f2bdc073070 0x7f2bdc075530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:46:35.971 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:35.965+0000 7f2bee7fc640 1 -- 172.21.15.123:0/2680963825 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(58..58 src has 1..58) ==== 5709+0+0 (secure 0 0 0) 0x7f2bf4094a70 con 0x7f2bf81f7900 2024-09-16T15:46:35.971 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:35.965+0000 7f2bfda49640 1 --2- 172.21.15.123:0/2680963825 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f2bdc073070 0x7f2bdc075530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:46:35.971 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:35.965+0000 7f2bfda49640 1 --2- 172.21.15.123:0/2680963825 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f2bdc073070 0x7f2bdc075530 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f2be80b0000 tx=0x7f2be8019b40 comp rx=0 tx=0).ready entity=mgr.4135 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:46:35.973 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:35.969+0000 7f2bee7fc640 1 -- 172.21.15.123:0/2680963825 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f2bf405faa0 con 0x7f2bf81f7900 2024-09-16T15:46:36.157 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:36.153+0000 7f2bffcad640 1 -- 172.21.15.123:0/2680963825 --> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "subvol14360", "target": ["mon-mgr", ""]}) -- 0x7f2bf815d6f0 con 0x7f2bdc073070 2024-09-16T15:46:36.221 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:36.217+0000 7f2bee7fc640 1 -- 172.21.15.123:0/2680963825 <== mgr.4135 v2:172.21.15.184:6832/3114346423 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+30 (secure 0 0 0) 0x7f2bf815d6f0 con 0x7f2bdc073070 2024-09-16T15:46:36.221 INFO:teuthology.orchestra.run.smithi123.stdout:/volumes/_nogroup/subvol14360 2024-09-16T15:46:36.223 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:36.221+0000 7f2bffcad640 1 -- 172.21.15.123:0/2680963825 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f2bdc073070 msgr2=0x7f2bdc075530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:46:36.224 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:36.221+0000 7f2bffcad640 1 --2- 172.21.15.123:0/2680963825 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f2bdc073070 0x7f2bdc075530 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f2be80b0000 tx=0x7f2be8019b40 comp rx=0 tx=0).stop 2024-09-16T15:46:36.224 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:36.221+0000 7f2bffcad640 1 -- 172.21.15.123:0/2680963825 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2bf81f7900 msgr2=0x7f2bf81fa710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:46:36.224 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:36.221+0000 7f2bffcad640 1 --2- 172.21.15.123:0/2680963825 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2bf81f7900 0x7f2bf81fa710 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7f2bf4003ed0 tx=0x7f2bf4001640 comp rx=0 tx=0).stop 2024-09-16T15:46:36.224 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:36.221+0000 7f2bffcad640 1 -- 172.21.15.123:0/2680963825 shutdown_connections 2024-09-16T15:46:36.224 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:36.221+0000 7f2bffcad640 1 --2- 172.21.15.123:0/2680963825 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f2bdc073070 0x7f2bdc075530 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:36.224 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:36.221+0000 7f2bffcad640 1 --2- 172.21.15.123:0/2680963825 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f2bf80c8c20 0x7f2bf80c90d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:36.224 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:36.221+0000 7f2bffcad640 1 --2- 172.21.15.123:0/2680963825 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2bf81f7900 0x7f2bf81fa710 unknown :-1 s=CLOSED pgs=677 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:36.224 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:36.221+0000 7f2bffcad640 1 --2- 172.21.15.123:0/2680963825 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2be4003a60 0x7f2bf814e040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:36.224 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:36.221+0000 7f2bffcad640 1 -- 172.21.15.123:0/2680963825 >> 172.21.15.123:0/2680963825 conn(0x7f2bf80b1710 msgr2=0x7f2bf8157110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:46:36.224 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:36.221+0000 7f2bffcad640 1 -- 172.21.15.123:0/2680963825 shutdown_connections 2024-09-16T15:46:36.224 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:36.221+0000 7f2bffcad640 1 -- 172.21.15.123:0/2680963825 wait complete. 2024-09-16T15:46:36.239 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs subvol14360 2024-09-16T15:46:36.363 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:36.361+0000 7f91e8f74640 1 -- 172.21.15.123:0/2670879701 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f91e4072720 msgr2=0x7f91e40cabc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:46:36.363 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:36.361+0000 7f91e8f74640 1 --2- 172.21.15.123:0/2670879701 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f91e4072720 0x7f91e40cabc0 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7f91cc009840 tx=0x7f91cc01c820 comp rx=0 tx=0).stop 2024-09-16T15:46:36.363 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:36.361+0000 7f91e8f74640 1 -- 172.21.15.123:0/2670879701 shutdown_connections 2024-09-16T15:46:36.363 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:36.361+0000 7f91e8f74640 1 --2- 172.21.15.123:0/2670879701 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f91e4072720 0x7f91e40cabc0 unknown :-1 s=CLOSED pgs=679 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:36.363 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:36.361+0000 7f91e8f74640 1 --2- 172.21.15.123:0/2670879701 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f91e415aa00 0x7f91e415e5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:36.363 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:36.361+0000 7f91e8f74640 1 -- 172.21.15.123:0/2670879701 >> 172.21.15.123:0/2670879701 conn(0x7f91e40b1710 msgr2=0x7f91e40b1b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:46:36.363 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:36.361+0000 7f91e8f74640 1 -- 172.21.15.123:0/2670879701 shutdown_connections 2024-09-16T15:46:36.363 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:36.361+0000 7f91e8f74640 1 -- 172.21.15.123:0/2670879701 wait complete. 2024-09-16T15:46:36.364 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:36.361+0000 7f91e8f74640 1 Processor -- start 2024-09-16T15:46:36.364 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:36.361+0000 7f91e8f74640 1 -- start start 2024-09-16T15:46:36.366 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:36.361+0000 7f91e8f74640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f91e4072720 0x7f91e41f9b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:46:36.366 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:36.361+0000 7f91e8f74640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f91e40cc3c0 0x7f91e41f5eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:46:36.366 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:36.361+0000 7f91e8f74640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f91e415aa00 0x7f91e40af680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:46:36.366 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:36.361+0000 7f91e8f74640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f91e4168510 con 0x7f91e40cc3c0 2024-09-16T15:46:36.366 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:36.361+0000 7f91e8f74640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f91e4168390 con 0x7f91e4072720 2024-09-16T15:46:36.367 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:36.361+0000 7f91e8f74640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f91e4168690 con 0x7f91e415aa00 2024-09-16T15:46:36.367 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:36.361+0000 7f91d9d9b640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f91e40cc3c0 0x7f91e41f5eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:46:36.367 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:36.361+0000 7f91d9d9b640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f91e40cc3c0 0x7f91e41f5eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:39442/0 (socket says 172.21.15.123:39442) 2024-09-16T15:46:36.367 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:36.361+0000 7f91d9d9b640 1 -- 172.21.15.123:0/4237922477 learned_addr learned my addr 172.21.15.123:0/4237922477 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:46:36.367 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:36.361+0000 7f91e2d9d640 1 --2- 172.21.15.123:0/4237922477 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f91e415aa00 0x7f91e40af680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:46:36.367 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:36.361+0000 7f91e259c640 1 --2- 172.21.15.123:0/4237922477 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f91e4072720 0x7f91e41f9b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:46:36.367 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:36.361+0000 7f91d9d9b640 1 -- 172.21.15.123:0/4237922477 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f91e415aa00 msgr2=0x7f91e40af680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:46:36.367 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:36.361+0000 7f91d9d9b640 1 --2- 172.21.15.123:0/4237922477 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f91e415aa00 0x7f91e40af680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:36.367 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:36.361+0000 7f91d9d9b640 1 -- 172.21.15.123:0/4237922477 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f91e4072720 msgr2=0x7f91e41f9b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:46:36.367 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:36.361+0000 7f91d9d9b640 1 --2- 172.21.15.123:0/4237922477 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f91e4072720 0x7f91e41f9b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:36.367 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:36.361+0000 7f91d9d9b640 1 -- 172.21.15.123:0/4237922477 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f91e40cb8b0 con 0x7f91e40cc3c0 2024-09-16T15:46:36.368 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:36.361+0000 7f91e2d9d640 1 --2- 172.21.15.123:0/4237922477 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f91e415aa00 0x7f91e40af680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T15:46:36.368 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:36.361+0000 7f91d9d9b640 1 --2- 172.21.15.123:0/4237922477 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f91e40cc3c0 0x7f91e41f5eb0 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7f91cc009790 tx=0x7f91cc004040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:46:36.368 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:36.361+0000 7f91db7fe640 1 -- 172.21.15.123:0/4237922477 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91cc0042f0 con 0x7f91e40cc3c0 2024-09-16T15:46:36.368 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:36.361+0000 7f91db7fe640 1 -- 172.21.15.123:0/4237922477 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f91cc005030 con 0x7f91e40cc3c0 2024-09-16T15:46:36.368 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:36.361+0000 7f91db7fe640 1 -- 172.21.15.123:0/4237922477 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91cc005460 con 0x7f91e40cc3c0 2024-09-16T15:46:36.368 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:36.361+0000 7f91e8f74640 1 -- 172.21.15.123:0/4237922477 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f91e41fa0b0 con 0x7f91e40cc3c0 2024-09-16T15:46:36.368 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:36.361+0000 7f91e8f74640 1 -- 172.21.15.123:0/4237922477 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f91e41fa350 con 0x7f91e40cc3c0 2024-09-16T15:46:36.369 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:36.365+0000 7f91e8f74640 1 -- 172.21.15.123:0/4237922477 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f91e4145960 con 0x7f91e40cc3c0 2024-09-16T15:46:36.369 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:36.365+0000 7f91db7fe640 1 -- 172.21.15.123:0/4237922477 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 9) ==== 97233+0+0 (secure 0 0 0) 0x7f91cc0c0020 con 0x7f91e40cc3c0 2024-09-16T15:46:36.369 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:36.365+0000 7f91db7fe640 1 --2- 172.21.15.123:0/4237922477 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f91c4073070 0x7f91c4075530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:46:36.369 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:36.365+0000 7f91db7fe640 1 -- 172.21.15.123:0/4237922477 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(58..58 src has 1..58) ==== 5709+0+0 (secure 0 0 0) 0x7f91cc12ea10 con 0x7f91e40cc3c0 2024-09-16T15:46:36.369 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:36.365+0000 7f91e259c640 1 --2- 172.21.15.123:0/4237922477 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f91c4073070 0x7f91c4075530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:46:36.369 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:36.365+0000 7f91e259c640 1 --2- 172.21.15.123:0/4237922477 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f91c4073070 0x7f91c4075530 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f91d40058d0 tx=0x7f91d4005bd0 comp rx=0 tx=0).ready entity=mgr.4135 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:46:36.372 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:36.369+0000 7f91db7fe640 1 -- 172.21.15.123:0/4237922477 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f91cc0c7020 con 0x7f91e40cc3c0 2024-09-16T15:46:36.406 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:36.401+0000 7f91db7fe640 1 -- 172.21.15.123:0/4237922477 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mgrmap(e 10) ==== 97284+0+0 (secure 0 0 0) 0x7f91cc0f54d0 con 0x7f91e40cc3c0 2024-09-16T15:46:36.575 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:36.573+0000 7f91e8f74640 1 -- 172.21.15.123:0/4237922477 --> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "subvol14360", "target": ["mon-mgr", ""]}) -- 0x7f91e415e050 con 0x7f91c4073070 2024-09-16T15:46:36.586 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:36.581+0000 7f91db7fe640 1 -- 172.21.15.123:0/4237922477 <== mgr.4135 v2:172.21.15.184:6832/3114346423 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+30 (secure 0 0 0) 0x7f91e415e050 con 0x7f91c4073070 2024-09-16T15:46:36.586 INFO:teuthology.orchestra.run.smithi123.stdout:/volumes/_nogroup/subvol14360 2024-09-16T15:46:36.589 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:36.585+0000 7f91e8f74640 1 -- 172.21.15.123:0/4237922477 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f91c4073070 msgr2=0x7f91c4075530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:46:36.589 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:36.585+0000 7f91e8f74640 1 --2- 172.21.15.123:0/4237922477 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f91c4073070 0x7f91c4075530 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f91d40058d0 tx=0x7f91d4005bd0 comp rx=0 tx=0).stop 2024-09-16T15:46:36.589 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:36.585+0000 7f91e8f74640 1 -- 172.21.15.123:0/4237922477 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f91e40cc3c0 msgr2=0x7f91e41f5eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:46:36.589 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:36.585+0000 7f91e8f74640 1 --2- 172.21.15.123:0/4237922477 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f91e40cc3c0 0x7f91e41f5eb0 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7f91cc009790 tx=0x7f91cc004040 comp rx=0 tx=0).stop 2024-09-16T15:46:36.589 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:36.585+0000 7f91e8f74640 1 -- 172.21.15.123:0/4237922477 shutdown_connections 2024-09-16T15:46:36.589 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:36.585+0000 7f91e8f74640 1 --2- 172.21.15.123:0/4237922477 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f91c4073070 0x7f91c4075530 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:36.589 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:36.585+0000 7f91e8f74640 1 --2- 172.21.15.123:0/4237922477 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f91e415aa00 0x7f91e40af680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:36.589 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:36.585+0000 7f91e8f74640 1 --2- 172.21.15.123:0/4237922477 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f91e40cc3c0 0x7f91e41f5eb0 unknown :-1 s=CLOSED pgs=680 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:36.590 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:36.585+0000 7f91e8f74640 1 --2- 172.21.15.123:0/4237922477 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f91e4072720 0x7f91e41f9b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:36.590 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:36.585+0000 7f91e8f74640 1 -- 172.21.15.123:0/4237922477 >> 172.21.15.123:0/4237922477 conn(0x7f91e40b1710 msgr2=0x7f91e40b05e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:46:36.590 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:36.585+0000 7f91e8f74640 1 -- 172.21.15.123:0/4237922477 shutdown_connections 2024-09-16T15:46:36.590 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:36.585+0000 7f91e8f74640 1 -- 172.21.15.123:0/4237922477 wait complete. 2024-09-16T15:46:36.605 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:46:36.605 DEBUG:teuthology.orchestra.run.smithi123:> (cd /home/ubuntu/cephtest/mnt.0 && exec sudo cat ./volumes/_legacy/ae66263a86b37857cc4c20cbb05f1555.meta) 2024-09-16T15:46:39.402 DEBUG:teuthology.orchestra.run.smithi123:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-16T15:46:39.405 DEBUG:teuthology.orchestra.run.smithi184:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-16T15:46:40.819 INFO:teuthology.orchestra.run.smithi123.stdout:[GLOBAL] 2024-09-16T15:46:40.819 INFO:teuthology.orchestra.run.smithi123.stdout:version = 1 2024-09-16T15:46:40.820 INFO:teuthology.orchestra.run.smithi123.stdout:type = subvolume 2024-09-16T15:46:40.820 INFO:teuthology.orchestra.run.smithi123.stdout:path = /volumes/_nogroup/subvol14360 2024-09-16T15:46:40.820 INFO:teuthology.orchestra.run.smithi123.stdout:state = complete 2024-09-16T15:46:40.820 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-09-16T15:46:40.822 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume authorize cephfs subvol14360 alice 2024-09-16T15:46:40.963 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:40.957+0000 7f6c64607640 1 -- 172.21.15.123:0/779005774 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6c5c15a1b0 msgr2=0x7f6c5c072fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:46:40.963 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:40.957+0000 7f6c64607640 1 --2- 172.21.15.123:0/779005774 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6c5c15a1b0 0x7f6c5c072fb0 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7f6c5800a950 tx=0x7f6c58020360 comp rx=0 tx=0).stop 2024-09-16T15:46:40.963 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:40.957+0000 7f6c64607640 1 -- 172.21.15.123:0/779005774 shutdown_connections 2024-09-16T15:46:40.963 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:40.957+0000 7f6c64607640 1 --2- 172.21.15.123:0/779005774 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f6c5c15dc40 0x7f6c5c15e090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:40.963 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:40.957+0000 7f6c64607640 1 --2- 172.21.15.123:0/779005774 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6c5c15a1b0 0x7f6c5c072fb0 unknown :-1 s=CLOSED pgs=682 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:40.963 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:40.957+0000 7f6c64607640 1 -- 172.21.15.123:0/779005774 >> 172.21.15.123:0/779005774 conn(0x7f6c5c0b1630 msgr2=0x7f6c5c0b1a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:46:40.963 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:40.957+0000 7f6c64607640 1 -- 172.21.15.123:0/779005774 shutdown_connections 2024-09-16T15:46:40.963 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:40.957+0000 7f6c64607640 1 -- 172.21.15.123:0/779005774 wait complete. 2024-09-16T15:46:40.964 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:40.961+0000 7f6c64607640 1 Processor -- start 2024-09-16T15:46:40.964 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:40.961+0000 7f6c64607640 1 -- start start 2024-09-16T15:46:40.967 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:40.961+0000 7f6c64607640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6c5c15a1b0 0x7f6c5c1e7700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:46:40.967 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:40.961+0000 7f6c64607640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6c5c15a590 0x7f6c5c1483e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:46:40.967 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:40.961+0000 7f6c64607640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f6c5c15dc40 0x7f6c5c0afdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:46:40.967 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:40.961+0000 7f6c64607640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f6c5c0706d0 con 0x7f6c5c15a1b0 2024-09-16T15:46:40.967 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:40.961+0000 7f6c64607640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f6c5c15eb40 con 0x7f6c5c15a590 2024-09-16T15:46:40.967 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:40.961+0000 7f6c64607640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f6c5c070850 con 0x7f6c5c15dc40 2024-09-16T15:46:40.967 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:40.961+0000 7f6c623a3640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6c5c15a1b0 0x7f6c5c1e7700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:46:40.967 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:40.961+0000 7f6c623a3640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6c5c15a1b0 0x7f6c5c1e7700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:39466/0 (socket says 172.21.15.123:39466) 2024-09-16T15:46:40.967 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:40.961+0000 7f6c623a3640 1 -- 172.21.15.123:0/642114832 learned_addr learned my addr 172.21.15.123:0/642114832 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:46:40.967 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:40.961+0000 7f6c62ba4640 1 --2- 172.21.15.123:0/642114832 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f6c5c15dc40 0x7f6c5c0afdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:46:40.968 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:40.961+0000 7f6c61ba2640 1 --2- 172.21.15.123:0/642114832 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6c5c15a590 0x7f6c5c1483e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:46:40.968 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:40.961+0000 7f6c623a3640 1 -- 172.21.15.123:0/642114832 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f6c5c15dc40 msgr2=0x7f6c5c0afdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:46:40.968 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:40.961+0000 7f6c623a3640 1 --2- 172.21.15.123:0/642114832 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f6c5c15dc40 0x7f6c5c0afdb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:40.968 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:40.961+0000 7f6c623a3640 1 -- 172.21.15.123:0/642114832 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6c5c15a590 msgr2=0x7f6c5c1483e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:46:40.968 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:40.961+0000 7f6c623a3640 1 --2- 172.21.15.123:0/642114832 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6c5c15a590 0x7f6c5c1483e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:40.968 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:40.961+0000 7f6c623a3640 1 -- 172.21.15.123:0/642114832 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6c5c1e44a0 con 0x7f6c5c15a1b0 2024-09-16T15:46:40.968 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:40.961+0000 7f6c62ba4640 1 --2- 172.21.15.123:0/642114832 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f6c5c15dc40 0x7f6c5c0afdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T15:46:40.968 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:40.961+0000 7f6c623a3640 1 --2- 172.21.15.123:0/642114832 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6c5c15a1b0 0x7f6c5c1e7700 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7f6c58000c10 tx=0x7f6c58005d40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:46:40.968 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:40.961+0000 7f6c52ffd640 1 -- 172.21.15.123:0/642114832 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c580b5070 con 0x7f6c5c15a1b0 2024-09-16T15:46:40.968 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:40.961+0000 7f6c52ffd640 1 -- 172.21.15.123:0/642114832 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f6c580b6040 con 0x7f6c5c15a1b0 2024-09-16T15:46:40.968 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:40.961+0000 7f6c52ffd640 1 -- 172.21.15.123:0/642114832 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c580b0d10 con 0x7f6c5c15a1b0 2024-09-16T15:46:40.968 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:40.965+0000 7f6c64607640 1 -- 172.21.15.123:0/642114832 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6c5c1e4780 con 0x7f6c5c15a1b0 2024-09-16T15:46:40.968 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:40.965+0000 7f6c64607640 1 -- 172.21.15.123:0/642114832 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6c5c1e4fc0 con 0x7f6c5c15a1b0 2024-09-16T15:46:40.969 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:40.965+0000 7f6c64607640 1 -- 172.21.15.123:0/642114832 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6c5c15b300 con 0x7f6c5c15a1b0 2024-09-16T15:46:40.969 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:40.965+0000 7f6c52ffd640 1 -- 172.21.15.123:0/642114832 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 10) ==== 97284+0+0 (secure 0 0 0) 0x7f6c58005020 con 0x7f6c5c15a1b0 2024-09-16T15:46:40.969 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:40.965+0000 7f6c52ffd640 1 --2- 172.21.15.123:0/642114832 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f6c40073100 0x7f6c400755c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:46:40.969 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:40.965+0000 7f6c52ffd640 1 -- 172.21.15.123:0/642114832 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(58..58 src has 1..58) ==== 5709+0+0 (secure 0 0 0) 0x7f6c58131a20 con 0x7f6c5c15a1b0 2024-09-16T15:46:40.970 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:40.965+0000 7f6c61ba2640 1 --2- 172.21.15.123:0/642114832 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f6c40073100 0x7f6c400755c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:46:40.970 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:40.965+0000 7f6c61ba2640 1 --2- 172.21.15.123:0/642114832 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f6c40073100 0x7f6c400755c0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f6c4c0058d0 tx=0x7f6c4c005860 comp rx=0 tx=0).ready entity=mgr.4135 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:46:40.972 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:40.969+0000 7f6c52ffd640 1 -- 172.21.15.123:0/642114832 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f6c58019ec0 con 0x7f6c5c15a1b0 2024-09-16T15:46:41.174 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:41.169+0000 7f6c64607640 1 -- 172.21.15.123:0/642114832 --> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] -- mgr_command(tid 0: {"prefix": "fs subvolume authorize", "vol_name": "cephfs", "sub_name": "subvol14360", "auth_id": "alice", "target": ["mon-mgr", ""]}) -- 0x7f6c5c15cb40 con 0x7f6c40073100 2024-09-16T15:46:41.195 INFO:tasks.ceph.mgr.x.smithi184.stderr:Creating meta for ID alice with tenant None 2024-09-16T15:46:41.273 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:41.269+0000 7f6c52ffd640 1 -- 172.21.15.123:0/642114832 <== mgr.4135 v2:172.21.15.184:6832/3114346423 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f6c5c15cb40 con 0x7f6c40073100 2024-09-16T15:46:41.276 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:41.273+0000 7f6c64607640 1 -- 172.21.15.123:0/642114832 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f6c40073100 msgr2=0x7f6c400755c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:46:41.276 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:41.273+0000 7f6c64607640 1 --2- 172.21.15.123:0/642114832 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f6c40073100 0x7f6c400755c0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f6c4c0058d0 tx=0x7f6c4c005860 comp rx=0 tx=0).stop 2024-09-16T15:46:41.276 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:41.273+0000 7f6c64607640 1 -- 172.21.15.123:0/642114832 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6c5c15a1b0 msgr2=0x7f6c5c1e7700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:46:41.276 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:41.273+0000 7f6c64607640 1 --2- 172.21.15.123:0/642114832 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6c5c15a1b0 0x7f6c5c1e7700 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7f6c58000c10 tx=0x7f6c58005d40 comp rx=0 tx=0).stop 2024-09-16T15:46:41.276 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:41.273+0000 7f6c64607640 1 -- 172.21.15.123:0/642114832 shutdown_connections 2024-09-16T15:46:41.277 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:41.273+0000 7f6c64607640 1 --2- 172.21.15.123:0/642114832 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f6c40073100 0x7f6c400755c0 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:41.277 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:41.273+0000 7f6c64607640 1 --2- 172.21.15.123:0/642114832 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f6c5c15dc40 0x7f6c5c0afdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:41.277 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:41.273+0000 7f6c64607640 1 --2- 172.21.15.123:0/642114832 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6c5c15a590 0x7f6c5c1483e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:41.277 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:41.273+0000 7f6c64607640 1 --2- 172.21.15.123:0/642114832 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6c5c15a1b0 0x7f6c5c1e7700 unknown :-1 s=CLOSED pgs=683 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:41.277 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:41.273+0000 7f6c64607640 1 -- 172.21.15.123:0/642114832 >> 172.21.15.123:0/642114832 conn(0x7f6c5c0b1630 msgr2=0x7f6c5c1596b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:46:41.277 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:41.273+0000 7f6c64607640 1 -- 172.21.15.123:0/642114832 shutdown_connections 2024-09-16T15:46:41.277 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:41.273+0000 7f6c64607640 1 -- 172.21.15.123:0/642114832 wait complete. 2024-09-16T15:46:41.295 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth get client.alice --format=json-pretty 2024-09-16T15:46:41.425 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:41.421+0000 7ff6b201a640 1 -- 172.21.15.123:0/3057522110 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff698004da0 msgr2=0x7ff698005220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:46:41.425 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:41.421+0000 7ff6b201a640 1 --2- 172.21.15.123:0/3057522110 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff698004da0 0x7ff698005220 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7ff6a4005fd0 tx=0x7ff6a401cd80 comp rx=0 tx=0).stop 2024-09-16T15:46:41.425 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:41.421+0000 7ff6abfff640 1 -- 172.21.15.123:0/3057522110 reap_dead start 2024-09-16T15:46:41.425 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:41.421+0000 7ff6b201a640 1 -- 172.21.15.123:0/3057522110 shutdown_connections 2024-09-16T15:46:41.425 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:41.421+0000 7ff6b201a640 1 -- 172.21.15.123:0/3057522110 >> 172.21.15.123:0/3057522110 conn(0x7ff6ac0d1c70 msgr2=0x7ff6ac0d2050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:46:41.426 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:41.421+0000 7ff6b201a640 1 -- 172.21.15.123:0/3057522110 shutdown_connections 2024-09-16T15:46:41.426 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:41.421+0000 7ff6b201a640 1 -- 172.21.15.123:0/3057522110 wait complete. 2024-09-16T15:46:41.426 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:41.421+0000 7ff6b201a640 1 Processor -- start 2024-09-16T15:46:41.426 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:41.421+0000 7ff6b201a640 1 -- start start 2024-09-16T15:46:41.428 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:41.425+0000 7ff6b201a640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff6ac0cbbe0 0x7ff6ac14e6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:46:41.428 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:41.425+0000 7ff6b201a640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7ff6ac0b7150 0x7ff6ac1f2620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:46:41.428 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:41.425+0000 7ff6b201a640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff6ac0b5af0 0x7ff6ac1f2b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:46:41.429 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:41.425+0000 7ff6b201a640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7ff6ac0cd960 con 0x7ff6ac0cbbe0 2024-09-16T15:46:41.429 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:41.425+0000 7ff6b201a640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7ff6ac0cd7e0 con 0x7ff6ac0b5af0 2024-09-16T15:46:41.429 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:41.425+0000 7ff6b201a640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7ff6ac0cdae0 con 0x7ff6ac0b7150 2024-09-16T15:46:41.429 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:41.425+0000 7ff6ab7fe640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff6ac0cbbe0 0x7ff6ac14e6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:46:41.429 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:41.425+0000 7ff6ab7fe640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff6ac0cbbe0 0x7ff6ac14e6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:39490/0 (socket says 172.21.15.123:39490) 2024-09-16T15:46:41.429 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:41.425+0000 7ff6ab7fe640 1 -- 172.21.15.123:0/3818594997 learned_addr learned my addr 172.21.15.123:0/3818594997 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:46:41.429 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:41.425+0000 7ff6abfff640 1 --2- 172.21.15.123:0/3818594997 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff6ac0b5af0 0x7ff6ac1f2b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:46:41.429 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:41.425+0000 7ff6aaffd640 1 --2- 172.21.15.123:0/3818594997 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7ff6ac0b7150 0x7ff6ac1f2620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:46:41.429 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:41.425+0000 7ff6ab7fe640 1 -- 172.21.15.123:0/3818594997 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7ff6ac0b7150 msgr2=0x7ff6ac1f2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:46:41.429 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:41.425+0000 7ff6ab7fe640 1 --2- 172.21.15.123:0/3818594997 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7ff6ac0b7150 0x7ff6ac1f2620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:41.429 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:41.425+0000 7ff6ab7fe640 1 -- 172.21.15.123:0/3818594997 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff6ac0b5af0 msgr2=0x7ff6ac1f2b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:46:41.429 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:41.425+0000 7ff6ab7fe640 1 --2- 172.21.15.123:0/3818594997 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff6ac0b5af0 0x7ff6ac1f2b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:41.430 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:41.425+0000 7ff6ab7fe640 1 -- 172.21.15.123:0/3818594997 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff6ac14ebf0 con 0x7ff6ac0cbbe0 2024-09-16T15:46:41.430 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:41.425+0000 7ff6ab7fe640 1 --2- 172.21.15.123:0/3818594997 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff6ac0cbbe0 0x7ff6ac14e6b0 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7ff69c00c800 tx=0x7ff69c00ccc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:46:41.430 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:41.425+0000 7ff6a3fff640 1 -- 172.21.15.123:0/3818594997 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff69c003b80 con 0x7ff6ac0cbbe0 2024-09-16T15:46:41.430 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:41.425+0000 7ff6a3fff640 1 -- 172.21.15.123:0/3818594997 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7ff69c003d20 con 0x7ff6ac0cbbe0 2024-09-16T15:46:41.430 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:41.425+0000 7ff6b201a640 1 -- 172.21.15.123:0/3818594997 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff6ac0b60b0 con 0x7ff6ac0cbbe0 2024-09-16T15:46:41.430 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:41.425+0000 7ff6a3fff640 1 -- 172.21.15.123:0/3818594997 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff69c00aab0 con 0x7ff6ac0cbbe0 2024-09-16T15:46:41.430 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:41.425+0000 7ff6b201a640 1 -- 172.21.15.123:0/3818594997 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff6ac0b6670 con 0x7ff6ac0cbbe0 2024-09-16T15:46:41.431 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:41.425+0000 7ff6b201a640 1 -- 172.21.15.123:0/3818594997 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff6ac0b0cf0 con 0x7ff6ac0cbbe0 2024-09-16T15:46:41.431 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:41.425+0000 7ff6a3fff640 1 -- 172.21.15.123:0/3818594997 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 10) ==== 97284+0+0 (secure 0 0 0) 0x7ff69c00ac50 con 0x7ff6ac0cbbe0 2024-09-16T15:46:41.431 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:41.425+0000 7ff6a3fff640 1 --2- 172.21.15.123:0/3818594997 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7ff68c073100 0x7ff68c0755c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:46:41.431 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:41.425+0000 7ff6a3fff640 1 -- 172.21.15.123:0/3818594997 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(58..58 src has 1..58) ==== 5709+0+0 (secure 0 0 0) 0x7ff69c094290 con 0x7ff6ac0cbbe0 2024-09-16T15:46:41.431 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:41.425+0000 7ff6aaffd640 1 --2- 172.21.15.123:0/3818594997 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7ff68c073100 0x7ff68c0755c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:46:41.431 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:41.429+0000 7ff6aaffd640 1 --2- 172.21.15.123:0/3818594997 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7ff68c073100 0x7ff68c0755c0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7ff694000fc0 tx=0x7ff694003600 comp rx=0 tx=0).ready entity=mgr.4135 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:46:41.434 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:41.429+0000 7ff6a3fff640 1 -- 172.21.15.123:0/3818594997 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7ff69c05f070 con 0x7ff6ac0cbbe0 2024-09-16T15:46:41.667 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:41.665+0000 7ff6b201a640 1 -- 172.21.15.123:0/3818594997 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "auth get", "entity": "client.alice", "format": "json-pretty"} v 0) -- 0x7ff6ac15e830 con 0x7ff6ac0cbbe0 2024-09-16T15:46:41.667 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:41.665+0000 7ff6a3fff640 1 -- 172.21.15.123:0/3818594997 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "auth get", "entity": "client.alice", "format": "json-pretty"}]=0 v20) ==== 107+0+280 (secure 0 0 0) 0x7ff69c05e970 con 0x7ff6ac0cbbe0 2024-09-16T15:46:41.667 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-09-16T15:46:41.668 INFO:teuthology.orchestra.run.smithi123.stdout:[ 2024-09-16T15:46:41.668 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-09-16T15:46:41.668 INFO:teuthology.orchestra.run.smithi123.stdout: "entity": "client.alice", 2024-09-16T15:46:41.668 INFO:teuthology.orchestra.run.smithi123.stdout: "key": "AQDhUuhmPcuGDRAAbzuLPE1eqyta2/Ouy8g7mw==", 2024-09-16T15:46:41.668 INFO:teuthology.orchestra.run.smithi123.stdout: "caps": { 2024-09-16T15:46:41.668 INFO:teuthology.orchestra.run.smithi123.stdout: "mds": "allow rw path=/volumes/_nogroup/subvol14360", 2024-09-16T15:46:41.668 INFO:teuthology.orchestra.run.smithi123.stdout: "mon": "allow r", 2024-09-16T15:46:41.668 INFO:teuthology.orchestra.run.smithi123.stdout: "osd": "allow rw pool=cephfs_data" 2024-09-16T15:46:41.668 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-16T15:46:41.668 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-16T15:46:41.668 INFO:teuthology.orchestra.run.smithi123.stdout:] 2024-09-16T15:46:41.670 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:41.665+0000 7ff6b201a640 1 -- 172.21.15.123:0/3818594997 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7ff68c073100 msgr2=0x7ff68c0755c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:46:41.670 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:41.665+0000 7ff6b201a640 1 --2- 172.21.15.123:0/3818594997 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7ff68c073100 0x7ff68c0755c0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7ff694000fc0 tx=0x7ff694003600 comp rx=0 tx=0).stop 2024-09-16T15:46:41.670 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:41.665+0000 7ff6b201a640 1 -- 172.21.15.123:0/3818594997 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff6ac0cbbe0 msgr2=0x7ff6ac14e6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:46:41.670 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:41.665+0000 7ff6b201a640 1 --2- 172.21.15.123:0/3818594997 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff6ac0cbbe0 0x7ff6ac14e6b0 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7ff69c00c800 tx=0x7ff69c00ccc0 comp rx=0 tx=0).stop 2024-09-16T15:46:41.670 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:41.665+0000 7ff6b201a640 1 -- 172.21.15.123:0/3818594997 shutdown_connections 2024-09-16T15:46:41.670 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:41.665+0000 7ff6b201a640 1 --2- 172.21.15.123:0/3818594997 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7ff68c073100 0x7ff68c0755c0 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:41.670 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:41.665+0000 7ff6b201a640 1 --2- 172.21.15.123:0/3818594997 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff6ac0b5af0 0x7ff6ac1f2b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:41.670 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:41.665+0000 7ff6b201a640 1 --2- 172.21.15.123:0/3818594997 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7ff6ac0b7150 0x7ff6ac1f2620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:41.671 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:41.665+0000 7ff6b201a640 1 --2- 172.21.15.123:0/3818594997 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff6ac0cbbe0 0x7ff6ac14e6b0 unknown :-1 s=CLOSED pgs=686 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:41.671 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:41.665+0000 7ff6b201a640 1 -- 172.21.15.123:0/3818594997 >> 172.21.15.123:0/3818594997 conn(0x7ff6ac0d1c70 msgr2=0x7ff6ac071820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:46:41.671 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:41.665+0000 7ff6b201a640 1 -- 172.21.15.123:0/3818594997 shutdown_connections 2024-09-16T15:46:41.671 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:41.665+0000 7ff6b201a640 1 -- 172.21.15.123:0/3818594997 wait complete. 2024-09-16T15:46:41.685 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume rm cephfs subvol14360 2024-09-16T15:46:41.812 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:41.809+0000 7fbfc9d1f640 1 -- 172.21.15.123:0/1609432416 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbfc4072720 msgr2=0x7fbfc40cabc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:46:41.812 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:41.809+0000 7fbfc9d1f640 1 --2- 172.21.15.123:0/1609432416 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbfc4072720 0x7fbfc40cabc0 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7fbfc406db90 tx=0x7fbfb801c8c0 comp rx=0 tx=0).stop 2024-09-16T15:46:41.812 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:41.809+0000 7fbfc9d1f640 1 -- 172.21.15.123:0/1609432416 shutdown_connections 2024-09-16T15:46:41.812 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:41.809+0000 7fbfc9d1f640 1 --2- 172.21.15.123:0/1609432416 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fbfc40cc3c0 0x7fbfc40cc850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:41.812 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:41.809+0000 7fbfc9d1f640 1 --2- 172.21.15.123:0/1609432416 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbfc4072720 0x7fbfc40cabc0 unknown :-1 s=CLOSED pgs=687 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:41.812 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:41.809+0000 7fbfc9d1f640 1 --2- 172.21.15.123:0/1609432416 >> v2:172.21.15.184:3300/0 conn(0x7fbfc415aa00 0x7fbfc415e5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:41.813 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:41.809+0000 7fbfc9d1f640 1 -- 172.21.15.123:0/1609432416 >> 172.21.15.123:0/1609432416 conn(0x7fbfc40b1710 msgr2=0x7fbfc40b1b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:46:41.813 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:41.809+0000 7fbfc9d1f640 1 -- 172.21.15.123:0/1609432416 shutdown_connections 2024-09-16T15:46:41.813 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:41.809+0000 7fbfc9d1f640 1 -- 172.21.15.123:0/1609432416 wait complete. 2024-09-16T15:46:41.813 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:41.809+0000 7fbfc9d1f640 1 Processor -- start 2024-09-16T15:46:41.813 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:41.809+0000 7fbfc9d1f640 1 -- start start 2024-09-16T15:46:41.815 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:41.809+0000 7fbfc9d1f640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fbfc4072720 0x7fbfc41598d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:46:41.815 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:41.809+0000 7fbfc9d1f640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbfc40cc3c0 0x7fbfc40c7a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:46:41.815 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:41.809+0000 7fbfc9d1f640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fbfc415aa00 0x7fbfc40b2a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:46:41.815 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:41.809+0000 7fbfc9d1f640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fbfc406fe70 con 0x7fbfc40cc3c0 2024-09-16T15:46:41.815 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:41.809+0000 7fbfc9d1f640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fbfc40ce820 con 0x7fbfc4072720 2024-09-16T15:46:41.816 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:41.809+0000 7fbfc9d1f640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fbfc40c7570 con 0x7fbfc415aa00 2024-09-16T15:46:41.816 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:41.809+0000 7fbfc2ffd640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbfc40cc3c0 0x7fbfc40c7a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:46:41.816 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:41.809+0000 7fbfc2ffd640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbfc40cc3c0 0x7fbfc40c7a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:39504/0 (socket says 172.21.15.123:39504) 2024-09-16T15:46:41.816 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:41.809+0000 7fbfc2ffd640 1 -- 172.21.15.123:0/2359333131 learned_addr learned my addr 172.21.15.123:0/2359333131 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:46:41.816 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:41.809+0000 7fbfc2ffd640 1 -- 172.21.15.123:0/2359333131 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fbfc415aa00 msgr2=0x7fbfc40b2a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:46:41.816 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:41.809+0000 7fbfc2ffd640 1 --2- 172.21.15.123:0/2359333131 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fbfc415aa00 0x7fbfc40b2a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:41.816 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:41.809+0000 7fbfc37fe640 1 --2- 172.21.15.123:0/2359333131 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fbfc4072720 0x7fbfc41598d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:46:41.816 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:41.809+0000 7fbfc2ffd640 1 -- 172.21.15.123:0/2359333131 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fbfc4072720 msgr2=0x7fbfc41598d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:46:41.816 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:41.809+0000 7fbfc2ffd640 1 --2- 172.21.15.123:0/2359333131 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fbfc4072720 0x7fbfc41598d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:41.816 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:41.809+0000 7fbfc2ffd640 1 -- 172.21.15.123:0/2359333131 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbfc406a170 con 0x7fbfc40cc3c0 2024-09-16T15:46:41.817 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:41.809+0000 7fbfc2ffd640 1 --2- 172.21.15.123:0/2359333131 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbfc40cc3c0 0x7fbfc40c7a00 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7fbfb80b0040 tx=0x7fbfb8004220 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:46:41.817 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:41.813+0000 7fbfabfff640 1 -- 172.21.15.123:0/2359333131 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbfb80b2040 con 0x7fbfc40cc3c0 2024-09-16T15:46:41.817 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:41.813+0000 7fbfabfff640 1 -- 172.21.15.123:0/2359333131 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fbfb8005180 con 0x7fbfc40cc3c0 2024-09-16T15:46:41.817 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:41.813+0000 7fbfc9d1f640 1 -- 172.21.15.123:0/2359333131 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbfc40c6cb0 con 0x7fbfc40cc3c0 2024-09-16T15:46:41.817 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:41.813+0000 7fbfabfff640 1 -- 172.21.15.123:0/2359333131 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbfb80055d0 con 0x7fbfc40cc3c0 2024-09-16T15:46:41.817 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:41.813+0000 7fbfc9d1f640 1 -- 172.21.15.123:0/2359333131 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbfc40c7200 con 0x7fbfc40cc3c0 2024-09-16T15:46:41.817 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:41.813+0000 7fbfc9d1f640 1 -- 172.21.15.123:0/2359333131 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbfc40cc850 con 0x7fbfc40cc3c0 2024-09-16T15:46:41.818 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:41.813+0000 7fbfabfff640 1 -- 172.21.15.123:0/2359333131 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 10) ==== 97284+0+0 (secure 0 0 0) 0x7fbfb80046d0 con 0x7fbfc40cc3c0 2024-09-16T15:46:41.818 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:41.813+0000 7fbfabfff640 1 --2- 172.21.15.123:0/2359333131 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fbfb0073100 0x7fbfb00755c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:46:41.818 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:41.813+0000 7fbfabfff640 1 -- 172.21.15.123:0/2359333131 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(58..58 src has 1..58) ==== 5709+0+0 (secure 0 0 0) 0x7fbfb812d780 con 0x7fbfc40cc3c0 2024-09-16T15:46:41.818 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:41.813+0000 7fbfc37fe640 1 --2- 172.21.15.123:0/2359333131 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fbfb0073100 0x7fbfb00755c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:46:41.818 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:41.813+0000 7fbfc37fe640 1 --2- 172.21.15.123:0/2359333131 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fbfb0073100 0x7fbfb00755c0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7fbfb4005fd0 tx=0x7fbfb4005bc0 comp rx=0 tx=0).ready entity=mgr.4135 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:46:41.821 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:41.817+0000 7fbfabfff640 1 -- 172.21.15.123:0/2359333131 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7fbfb80164b0 con 0x7fbfc40cc3c0 2024-09-16T15:46:42.003 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:42.001+0000 7fbfc9d1f640 1 -- 172.21.15.123:0/2359333131 --> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] -- mgr_command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "subvol14360", "target": ["mon-mgr", ""]}) -- 0x7fbfc415de60 con 0x7fbfb0073100 2024-09-16T15:46:42.031 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:42.029+0000 7fbfabfff640 1 -- 172.21.15.123:0/2359333131 <== mgr.4135 v2:172.21.15.184:6832/3114346423 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7fbfc415de60 con 0x7fbfb0073100 2024-09-16T15:46:42.034 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:42.029+0000 7fbfc9d1f640 1 -- 172.21.15.123:0/2359333131 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fbfb0073100 msgr2=0x7fbfb00755c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:46:42.034 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:42.029+0000 7fbfc9d1f640 1 --2- 172.21.15.123:0/2359333131 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fbfb0073100 0x7fbfb00755c0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7fbfb4005fd0 tx=0x7fbfb4005bc0 comp rx=0 tx=0).stop 2024-09-16T15:46:42.034 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:42.029+0000 7fbfc9d1f640 1 -- 172.21.15.123:0/2359333131 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbfc40cc3c0 msgr2=0x7fbfc40c7a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:46:42.034 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:42.029+0000 7fbfc9d1f640 1 --2- 172.21.15.123:0/2359333131 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbfc40cc3c0 0x7fbfc40c7a00 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7fbfb80b0040 tx=0x7fbfb8004220 comp rx=0 tx=0).stop 2024-09-16T15:46:42.034 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:42.029+0000 7fbfc9d1f640 1 -- 172.21.15.123:0/2359333131 shutdown_connections 2024-09-16T15:46:42.034 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:42.029+0000 7fbfc9d1f640 1 --2- 172.21.15.123:0/2359333131 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fbfb0073100 0x7fbfb00755c0 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:42.034 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:42.029+0000 7fbfc9d1f640 1 --2- 172.21.15.123:0/2359333131 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fbfc415aa00 0x7fbfc40b2a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:42.034 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:42.029+0000 7fbfc9d1f640 1 --2- 172.21.15.123:0/2359333131 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbfc40cc3c0 0x7fbfc40c7a00 unknown :-1 s=CLOSED pgs=688 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:42.034 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:42.029+0000 7fbfc9d1f640 1 --2- 172.21.15.123:0/2359333131 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fbfc4072720 0x7fbfc41598d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:42.034 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:42.029+0000 7fbfc9d1f640 1 -- 172.21.15.123:0/2359333131 >> 172.21.15.123:0/2359333131 conn(0x7fbfc40b1710 msgr2=0x7fbfc40af680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:46:42.034 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:42.029+0000 7fbfc9d1f640 1 -- 172.21.15.123:0/2359333131 shutdown_connections 2024-09-16T15:46:42.034 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:42.029+0000 7fbfc9d1f640 1 -- 172.21.15.123:0/2359333131 wait complete. 2024-09-16T15:46:42.036 INFO:tasks.ceph.mgr.x.smithi184.stderr:2024-09-16T15:46:42.034+0000 7fda6c8d8640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-16T15:46:42.036 INFO:tasks.ceph.mgr.x.smithi184.stderr:2024-09-16T15:46:42.034+0000 7fda6c8d8640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-16T15:46:42.036 INFO:tasks.ceph.mgr.x.smithi184.stderr:2024-09-16T15:46:42.034+0000 7fda6c8d8640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-16T15:46:42.037 INFO:tasks.ceph.mgr.x.smithi184.stderr:2024-09-16T15:46:42.034+0000 7fda6c8d8640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-16T15:46:42.037 INFO:tasks.ceph.mgr.x.smithi184.stderr:2024-09-16T15:46:42.034+0000 7fda6c8d8640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-16T15:46:42.050 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume rm cephfs subvol2021 2024-09-16T15:46:42.054 INFO:tasks.ceph.mgr.x.smithi184.stderr:2024-09-16T15:46:42.050+0000 7fda6d0d9640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-16T15:46:42.054 INFO:tasks.ceph.mgr.x.smithi184.stderr:2024-09-16T15:46:42.050+0000 7fda6d0d9640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-16T15:46:42.054 INFO:tasks.ceph.mgr.x.smithi184.stderr:2024-09-16T15:46:42.050+0000 7fda6d0d9640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-16T15:46:42.054 INFO:tasks.ceph.mgr.x.smithi184.stderr:2024-09-16T15:46:42.050+0000 7fda6d0d9640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-16T15:46:42.054 INFO:tasks.ceph.mgr.x.smithi184.stderr:2024-09-16T15:46:42.050+0000 7fda6d0d9640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-16T15:46:42.185 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:42.181+0000 7f5da3968640 1 -- 172.21.15.123:0/1098038287 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5d9c0cfdc0 msgr2=0x7f5d9c0d0230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:46:42.185 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:42.181+0000 7f5da3968640 1 --2- 172.21.15.123:0/1098038287 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5d9c0cfdc0 0x7f5d9c0d0230 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7f5d8c009840 tx=0x7f5d8c01c820 comp rx=0 tx=0).stop 2024-09-16T15:46:42.185 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:42.181+0000 7f5da3968640 1 -- 172.21.15.123:0/1098038287 shutdown_connections 2024-09-16T15:46:42.185 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:42.181+0000 7f5da3968640 1 --2- 172.21.15.123:0/1098038287 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f5d9c0aed20 0x7f5d9c0af140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:42.185 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:42.181+0000 7f5da3968640 1 --2- 172.21.15.123:0/1098038287 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5d9c0cfdc0 0x7f5d9c0d0230 unknown :-1 s=CLOSED pgs=690 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:42.185 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:42.181+0000 7f5da3968640 1 -- 172.21.15.123:0/1098038287 >> 172.21.15.123:0/1098038287 conn(0x7f5d9c09aa40 msgr2=0x7f5d9c09ae50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:46:42.186 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:42.181+0000 7f5da3968640 1 -- 172.21.15.123:0/1098038287 shutdown_connections 2024-09-16T15:46:42.186 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:42.181+0000 7f5da3968640 1 -- 172.21.15.123:0/1098038287 wait complete. 2024-09-16T15:46:42.186 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:42.181+0000 7f5da3968640 1 Processor -- start 2024-09-16T15:46:42.186 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:42.181+0000 7f5da3968640 1 -- start start 2024-09-16T15:46:42.187 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:42.181+0000 7f5da3968640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5d9c0aed20 0x7f5d9c0b73d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:46:42.187 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:42.181+0000 7f5da3968640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5d9c0cfdc0 0x7f5d9c15c470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:46:42.187 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:42.181+0000 7f5da3968640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f5d9c15bb70 0x7f5d9c15ac40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:46:42.188 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:42.181+0000 7f5da3968640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f5d9c168e60 con 0x7f5d9c0cfdc0 2024-09-16T15:46:42.188 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:42.181+0000 7f5da3968640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f5d9c168ce0 con 0x7f5d9c0aed20 2024-09-16T15:46:42.188 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:42.181+0000 7f5da3968640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f5d9c168fe0 con 0x7f5d9c15bb70 2024-09-16T15:46:42.188 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:42.181+0000 7f5da0f03640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5d9c0cfdc0 0x7f5d9c15c470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:46:42.188 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:42.181+0000 7f5da0f03640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5d9c0cfdc0 0x7f5d9c15c470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:39524/0 (socket says 172.21.15.123:39524) 2024-09-16T15:46:42.188 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:42.181+0000 7f5da0f03640 1 -- 172.21.15.123:0/2785366895 learned_addr learned my addr 172.21.15.123:0/2785366895 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:46:42.188 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:42.181+0000 7f5da1f05640 1 --2- 172.21.15.123:0/2785366895 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f5d9c15bb70 0x7f5d9c15ac40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:46:42.188 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:42.181+0000 7f5da0f03640 1 -- 172.21.15.123:0/2785366895 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f5d9c15bb70 msgr2=0x7f5d9c15ac40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:46:42.188 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:42.181+0000 7f5da0f03640 1 --2- 172.21.15.123:0/2785366895 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f5d9c15bb70 0x7f5d9c15ac40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:42.188 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:42.181+0000 7f5da1704640 1 --2- 172.21.15.123:0/2785366895 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5d9c0aed20 0x7f5d9c0b73d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:46:42.188 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:42.181+0000 7f5da0f03640 1 -- 172.21.15.123:0/2785366895 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5d9c0aed20 msgr2=0x7f5d9c0b73d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:46:42.188 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:42.181+0000 7f5da0f03640 1 --2- 172.21.15.123:0/2785366895 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5d9c0aed20 0x7f5d9c0b73d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:42.188 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:42.181+0000 7f5da0f03640 1 -- 172.21.15.123:0/2785366895 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5d9c15cea0 con 0x7f5d9c0cfdc0 2024-09-16T15:46:42.189 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:42.181+0000 7f5da0f03640 1 --2- 172.21.15.123:0/2785366895 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5d9c0cfdc0 0x7f5d9c15c470 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7f5d8c01aed0 tx=0x7f5d8c0a67b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:46:42.189 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:42.185+0000 7f5d91ffb640 1 -- 172.21.15.123:0/2785366895 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d8c0041c0 con 0x7f5d9c0cfdc0 2024-09-16T15:46:42.189 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:42.185+0000 7f5d91ffb640 1 -- 172.21.15.123:0/2785366895 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f5d8c004450 con 0x7f5d9c0cfdc0 2024-09-16T15:46:42.189 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:42.185+0000 7f5d91ffb640 1 -- 172.21.15.123:0/2785366895 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d8c0052b0 con 0x7f5d9c0cfdc0 2024-09-16T15:46:42.189 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:42.185+0000 7f5da3968640 1 -- 172.21.15.123:0/2785366895 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5d9c1f8ff0 con 0x7f5d9c0cfdc0 2024-09-16T15:46:42.189 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:42.185+0000 7f5da3968640 1 -- 172.21.15.123:0/2785366895 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5d9c1f95b0 con 0x7f5d9c0cfdc0 2024-09-16T15:46:42.190 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:42.185+0000 7f5da3968640 1 -- 172.21.15.123:0/2785366895 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5d9c0b0720 con 0x7f5d9c0cfdc0 2024-09-16T15:46:42.190 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:42.185+0000 7f5d91ffb640 1 -- 172.21.15.123:0/2785366895 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 10) ==== 97284+0+0 (secure 0 0 0) 0x7f5d8c0c1030 con 0x7f5d9c0cfdc0 2024-09-16T15:46:42.190 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:42.185+0000 7f5d91ffb640 1 --2- 172.21.15.123:0/2785366895 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f5d80073100 0x7f5d800755c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:46:42.190 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:42.185+0000 7f5d91ffb640 1 -- 172.21.15.123:0/2785366895 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(58..58 src has 1..58) ==== 5709+0+0 (secure 0 0 0) 0x7f5d8c12ea20 con 0x7f5d9c0cfdc0 2024-09-16T15:46:42.190 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:42.185+0000 7f5da1704640 1 --2- 172.21.15.123:0/2785366895 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f5d80073100 0x7f5d800755c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:46:42.190 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:42.185+0000 7f5da1704640 1 --2- 172.21.15.123:0/2785366895 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f5d80073100 0x7f5d800755c0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f5d98001570 tx=0x7f5d98009040 comp rx=0 tx=0).ready entity=mgr.4135 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:46:42.193 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:42.189+0000 7f5d91ffb640 1 -- 172.21.15.123:0/2785366895 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f5d8c0c8060 con 0x7f5d9c0cfdc0 2024-09-16T15:46:42.391 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:42.389+0000 7f5da3968640 1 -- 172.21.15.123:0/2785366895 --> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] -- mgr_command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "subvol2021", "target": ["mon-mgr", ""]}) -- 0x7f5d9c0b0440 con 0x7f5d80073100 2024-09-16T15:46:42.430 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:42.425+0000 7f5d91ffb640 1 -- 172.21.15.123:0/2785366895 <== mgr.4135 v2:172.21.15.184:6832/3114346423 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f5d9c0b0440 con 0x7f5d80073100 2024-09-16T15:46:42.433 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:42.429+0000 7f5da3968640 1 -- 172.21.15.123:0/2785366895 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f5d80073100 msgr2=0x7f5d800755c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:46:42.433 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:42.429+0000 7f5da3968640 1 --2- 172.21.15.123:0/2785366895 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f5d80073100 0x7f5d800755c0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f5d98001570 tx=0x7f5d98009040 comp rx=0 tx=0).stop 2024-09-16T15:46:42.433 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:42.429+0000 7f5da3968640 1 -- 172.21.15.123:0/2785366895 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5d9c0cfdc0 msgr2=0x7f5d9c15c470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:46:42.433 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:42.429+0000 7f5da3968640 1 --2- 172.21.15.123:0/2785366895 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5d9c0cfdc0 0x7f5d9c15c470 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7f5d8c01aed0 tx=0x7f5d8c0a67b0 comp rx=0 tx=0).stop 2024-09-16T15:46:42.433 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:42.429+0000 7f5da3968640 1 -- 172.21.15.123:0/2785366895 shutdown_connections 2024-09-16T15:46:42.434 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:42.429+0000 7f5da3968640 1 --2- 172.21.15.123:0/2785366895 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f5d80073100 0x7f5d800755c0 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:42.434 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:42.429+0000 7f5da3968640 1 --2- 172.21.15.123:0/2785366895 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f5d9c15bb70 0x7f5d9c15ac40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:42.434 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:42.429+0000 7f5da3968640 1 --2- 172.21.15.123:0/2785366895 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5d9c0cfdc0 0x7f5d9c15c470 unknown :-1 s=CLOSED pgs=691 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:42.434 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:42.429+0000 7f5da3968640 1 --2- 172.21.15.123:0/2785366895 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5d9c0aed20 0x7f5d9c0b73d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:42.434 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:42.429+0000 7f5da3968640 1 -- 172.21.15.123:0/2785366895 >> 172.21.15.123:0/2785366895 conn(0x7f5d9c09aa40 msgr2=0x7f5d9c0b0c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:46:42.434 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:42.429+0000 7f5da3968640 1 -- 172.21.15.123:0/2785366895 shutdown_connections 2024-09-16T15:46:42.434 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:42.429+0000 7f5da3968640 1 -- 172.21.15.123:0/2785366895 wait complete. 2024-09-16T15:46:42.452 DEBUG:teuthology.orchestra.run.smithi123:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'stat -c %h /home/ubuntu/cephtest/mnt.0/./volumes/_deleting') 2024-09-16T15:46:42.492 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:42 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-09-16T15:46:42.500 INFO:teuthology.orchestra.run.smithi123.stdout:2 2024-09-16T15:46:42.500 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:42 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-16T15:46:42.505 INFO:teuthology.nuke:Clearing teuthology firewall rules... 2024-09-16T15:46:42.506 DEBUG:teuthology.orchestra.run.smithi123:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-09-16T15:46:42.612 DEBUG:teuthology.orchestra.run.smithi184:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-09-16T15:46:42.684 INFO:teuthology.nuke:Cleared teuthology firewall rules. 2024-09-16T15:46:42.684 INFO:tasks.cephfs.mount:Terminating background process 2024-09-16T15:46:42.685 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2024-09-16T15:46:42.685 DEBUG:teuthology.orchestra.run.smithi123:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-16T15:46:42.690 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2024-09-16T15:46:42.690 DEBUG:teuthology.orchestra.run.smithi123:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-16T15:46:42.740 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-09-16T15:46:42.913 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:42.909+0000 7f9e37369640 1 -- 172.21.15.123:0/561616051 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9e300b2b00 msgr2=0x7f9e300d24b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:46:42.913 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:42.909+0000 7f9e37369640 1 --2- 172.21.15.123:0/561616051 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9e300b2b00 0x7f9e300d24b0 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7f9e2c009900 tx=0x7f9e2c01c6d0 comp rx=0 tx=0).stop 2024-09-16T15:46:42.913 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:42.909+0000 7f9e37369640 1 -- 172.21.15.123:0/561616051 shutdown_connections 2024-09-16T15:46:42.913 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:42.909+0000 7f9e37369640 1 --2- 172.21.15.123:0/561616051 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9e300b2b00 0x7f9e300d24b0 unknown :-1 s=CLOSED pgs=693 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:42.913 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:42.909+0000 7f9e37369640 1 -- 172.21.15.123:0/561616051 >> 172.21.15.123:0/561616051 conn(0x7f9e300d3ea0 msgr2=0x7f9e300d42b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:46:42.913 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:42.909+0000 7f9e37369640 1 -- 172.21.15.123:0/561616051 shutdown_connections 2024-09-16T15:46:42.913 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:42.909+0000 7f9e37369640 1 -- 172.21.15.123:0/561616051 wait complete. 2024-09-16T15:46:42.914 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:42.909+0000 7f9e37369640 1 Processor -- start 2024-09-16T15:46:42.914 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:42.909+0000 7f9e37369640 1 -- start start 2024-09-16T15:46:42.916 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:42.913+0000 7f9e37369640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9e300b0a50 0x7f9e300cc4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:46:42.916 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:42.913+0000 7f9e37369640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f9e300b2b00 0x7f9e30155220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:46:42.916 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:42.913+0000 7f9e37369640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f9e30161620 0x7f9e300d1da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:46:42.916 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:42.913+0000 7f9e37369640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f9e3016c100 con 0x7f9e300b0a50 2024-09-16T15:46:42.916 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:42.913+0000 7f9e37369640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f9e3016bf80 con 0x7f9e300b2b00 2024-09-16T15:46:42.916 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:42.913+0000 7f9e37369640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f9e3016c280 con 0x7f9e30161620 2024-09-16T15:46:42.916 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:42.913+0000 7f9e35105640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9e300b0a50 0x7f9e300cc4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:46:42.916 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:42.913+0000 7f9e35105640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9e300b0a50 0x7f9e300cc4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:54480/0 (socket says 172.21.15.123:54480) 2024-09-16T15:46:42.916 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:42.913+0000 7f9e35105640 1 -- 172.21.15.123:0/880794492 learned_addr learned my addr 172.21.15.123:0/880794492 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:46:42.917 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:42.913+0000 7f9e34904640 1 --2- 172.21.15.123:0/880794492 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f9e300b2b00 0x7f9e30155220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:46:42.917 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:42.913+0000 7f9e35105640 1 -- 172.21.15.123:0/880794492 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f9e30161620 msgr2=0x7f9e300d1da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:46:42.917 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:42.913+0000 7f9e35105640 1 --2- 172.21.15.123:0/880794492 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f9e30161620 0x7f9e300d1da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:42.917 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:42.913+0000 7f9e35105640 1 -- 172.21.15.123:0/880794492 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f9e300b2b00 msgr2=0x7f9e30155220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:46:42.917 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:42.913+0000 7f9e35105640 1 --2- 172.21.15.123:0/880794492 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f9e300b2b00 0x7f9e30155220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:42.917 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:42.913+0000 7f9e35105640 1 -- 172.21.15.123:0/880794492 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9e300ccd00 con 0x7f9e300b0a50 2024-09-16T15:46:42.917 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:42.913+0000 7f9e34904640 1 --2- 172.21.15.123:0/880794492 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f9e300b2b00 0x7f9e30155220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T15:46:42.917 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:42.913+0000 7f9e35105640 1 --2- 172.21.15.123:0/880794492 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9e300b0a50 0x7f9e300cc4b0 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7f9e2000b420 tx=0x7f9e2000b8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:46:42.917 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:42.913+0000 7f9e25ffb640 1 -- 172.21.15.123:0/880794492 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e20011830 con 0x7f9e300b0a50 2024-09-16T15:46:42.917 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:42.913+0000 7f9e25ffb640 1 -- 172.21.15.123:0/880794492 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f9e20016070 con 0x7f9e300b0a50 2024-09-16T15:46:42.917 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:42.913+0000 7f9e37369640 1 -- 172.21.15.123:0/880794492 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9e300ccfe0 con 0x7f9e300b0a50 2024-09-16T15:46:42.917 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:42.913+0000 7f9e25ffb640 1 -- 172.21.15.123:0/880794492 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e20011d80 con 0x7f9e300b0a50 2024-09-16T15:46:42.918 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:42.913+0000 7f9e37369640 1 -- 172.21.15.123:0/880794492 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9e300cd5a0 con 0x7f9e300b0a50 2024-09-16T15:46:42.918 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:42.913+0000 7f9e37369640 1 -- 172.21.15.123:0/880794492 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9e300cda90 con 0x7f9e300b0a50 2024-09-16T15:46:42.918 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:42.913+0000 7f9e25ffb640 1 -- 172.21.15.123:0/880794492 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 10) ==== 97284+0+0 (secure 0 0 0) 0x7f9e20005970 con 0x7f9e300b0a50 2024-09-16T15:46:42.918 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:42.913+0000 7f9e25ffb640 1 --2- 172.21.15.123:0/880794492 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f9e14073100 0x7f9e140755c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:46:42.919 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:42.913+0000 7f9e25ffb640 1 -- 172.21.15.123:0/880794492 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(58..58 src has 1..58) ==== 5709+0+0 (secure 0 0 0) 0x7f9e200946c0 con 0x7f9e300b0a50 2024-09-16T15:46:42.919 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:42.913+0000 7f9e34904640 1 --2- 172.21.15.123:0/880794492 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f9e14073100 0x7f9e140755c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:46:42.919 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:42.917+0000 7f9e34904640 1 --2- 172.21.15.123:0/880794492 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f9e14073100 0x7f9e140755c0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f9e2c0af000 tx=0x7f9e2c01af10 comp rx=0 tx=0).ready entity=mgr.4135 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:46:42.921 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:42.917+0000 7f9e25ffb640 1 -- 172.21.15.123:0/880794492 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f9e2005f530 con 0x7f9e300b0a50 2024-09-16T15:46:43.106 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:43.100+0000 7f9e37369640 1 -- 172.21.15.123:0/880794492 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f9e300b0ea0 con 0x7f9e300b0a50 2024-09-16T15:46:43.107 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:43.100+0000 7f9e25ffb640 1 -- 172.21.15.123:0/880794492 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v58) ==== 80+0+0 (secure 0 0 0) 0x7f9e2005ee30 con 0x7f9e300b0a50 2024-09-16T15:46:43.107 INFO:teuthology.orchestra.run.smithi123.stderr:listed 0 entries 2024-09-16T15:46:43.110 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:43.104+0000 7f9e37369640 1 -- 172.21.15.123:0/880794492 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f9e14073100 msgr2=0x7f9e140755c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:46:43.110 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:43.104+0000 7f9e37369640 1 --2- 172.21.15.123:0/880794492 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f9e14073100 0x7f9e140755c0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f9e2c0af000 tx=0x7f9e2c01af10 comp rx=0 tx=0).stop 2024-09-16T15:46:43.110 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:43.104+0000 7f9e37369640 1 -- 172.21.15.123:0/880794492 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9e300b0a50 msgr2=0x7f9e300cc4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:46:43.110 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:43.104+0000 7f9e37369640 1 --2- 172.21.15.123:0/880794492 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9e300b0a50 0x7f9e300cc4b0 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7f9e2000b420 tx=0x7f9e2000b8e0 comp rx=0 tx=0).stop 2024-09-16T15:46:43.110 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:43.104+0000 7f9e37369640 1 -- 172.21.15.123:0/880794492 shutdown_connections 2024-09-16T15:46:43.110 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:43.104+0000 7f9e37369640 1 --2- 172.21.15.123:0/880794492 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f9e14073100 0x7f9e140755c0 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:43.110 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:43.104+0000 7f9e37369640 1 --2- 172.21.15.123:0/880794492 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f9e30161620 0x7f9e300d1da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:43.110 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:43.104+0000 7f9e37369640 1 --2- 172.21.15.123:0/880794492 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f9e300b2b00 0x7f9e30155220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:43.110 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:43.104+0000 7f9e37369640 1 --2- 172.21.15.123:0/880794492 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9e300b0a50 0x7f9e300cc4b0 unknown :-1 s=CLOSED pgs=694 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:43.111 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:43.104+0000 7f9e37369640 1 -- 172.21.15.123:0/880794492 >> 172.21.15.123:0/880794492 conn(0x7f9e300d3ea0 msgr2=0x7f9e300b4950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:46:43.111 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:43.104+0000 7f9e37369640 1 -- 172.21.15.123:0/880794492 shutdown_connections 2024-09-16T15:46:43.111 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:43.104+0000 7f9e37369640 1 -- 172.21.15.123:0/880794492 wait complete. 2024-09-16T15:46:43.125 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi123.front.sepia.ceph.com... 2024-09-16T15:46:43.125 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:46:43.125 DEBUG:teuthology.orchestra.run.smithi123:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2024-09-16T15:46:43.137 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi123.stderr:ceph-fuse[41267]: fuse finished with error 0 and tester_r 0 2024-09-16T15:46:43.138 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi123.front.sepia.ceph.com 2024-09-16T15:46:43.138 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:46:43.138 DEBUG:teuthology.orchestra.run.smithi123:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-09-16T15:46:43.267 INFO:tasks.cephfs.mount:Terminating background process 2024-09-16T15:46:43.267 DEBUG:teuthology.orchestra.run.smithi184:> set -ex 2024-09-16T15:46:43.267 DEBUG:teuthology.orchestra.run.smithi184:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-16T15:46:43.273 DEBUG:teuthology.orchestra.run.smithi184:> set -ex 2024-09-16T15:46:43.274 DEBUG:teuthology.orchestra.run.smithi184:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-16T15:46:43.320 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-09-16T15:46:43.451 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:43.448+0000 7f68f1ef7640 1 -- 172.21.15.123:0/850828455 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f68d8004810 msgr2=0x7f68d8004c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:46:43.451 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:43.448+0000 7f68f1ef7640 1 --2- 172.21.15.123:0/850828455 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f68d8004810 0x7f68d8004c90 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7f68d4009b60 tx=0x7f68d401d270 comp rx=0 tx=0).stop 2024-09-16T15:46:43.451 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:43.448+0000 7f68f1ef7640 1 -- 172.21.15.123:0/850828455 shutdown_connections 2024-09-16T15:46:43.451 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:43.448+0000 7f68f1ef7640 1 --2- 172.21.15.123:0/850828455 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f68d8004810 0x7f68d8004c90 unknown :-1 s=CLOSED pgs=696 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:43.452 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:43.448+0000 7f68f1ef7640 1 -- 172.21.15.123:0/850828455 >> 172.21.15.123:0/850828455 conn(0x7f68ec09aa40 msgr2=0x7f68ec09ae50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:46:43.452 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:43.448+0000 7f68f1ef7640 1 -- 172.21.15.123:0/850828455 shutdown_connections 2024-09-16T15:46:43.452 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:43.448+0000 7f68f1ef7640 1 -- 172.21.15.123:0/850828455 wait complete. 2024-09-16T15:46:43.452 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:43.448+0000 7f68f1ef7640 1 Processor -- start 2024-09-16T15:46:43.452 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:43.448+0000 7f68f1ef7640 1 -- start start 2024-09-16T15:46:43.454 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:43.448+0000 7f68f1ef7640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f68d8004810 0x7f68ec1f8ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:46:43.454 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:43.448+0000 7f68f1ef7640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f68ec0b4a70 0x7f68ec151e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:46:43.454 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:43.448+0000 7f68f1ef7640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f68ec0715c0 0x7f68ec071a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:46:43.454 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:43.448+0000 7f68f1ef7640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f68ec160390 con 0x7f68ec0b4a70 2024-09-16T15:46:43.455 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:43.448+0000 7f68f1ef7640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f68ec160210 con 0x7f68d8004810 2024-09-16T15:46:43.455 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:43.448+0000 7f68f1ef7640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f68ec160510 con 0x7f68ec0715c0 2024-09-16T15:46:43.455 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:43.448+0000 7f68eaffd640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f68ec0b4a70 0x7f68ec151e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:46:43.455 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:43.448+0000 7f68eaffd640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f68ec0b4a70 0x7f68ec151e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:54502/0 (socket says 172.21.15.123:54502) 2024-09-16T15:46:43.455 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:43.448+0000 7f68eaffd640 1 -- 172.21.15.123:0/2255090063 learned_addr learned my addr 172.21.15.123:0/2255090063 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:46:43.455 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:43.448+0000 7f68eb7fe640 1 --2- 172.21.15.123:0/2255090063 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f68d8004810 0x7f68ec1f8ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:46:43.455 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:43.448+0000 7f68ebfff640 1 --2- 172.21.15.123:0/2255090063 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f68ec0715c0 0x7f68ec071a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:46:43.455 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:43.448+0000 7f68eaffd640 1 -- 172.21.15.123:0/2255090063 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f68ec0715c0 msgr2=0x7f68ec071a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:46:43.455 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:43.448+0000 7f68eaffd640 1 --2- 172.21.15.123:0/2255090063 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f68ec0715c0 0x7f68ec071a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:43.455 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:43.448+0000 7f68eaffd640 1 -- 172.21.15.123:0/2255090063 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f68d8004810 msgr2=0x7f68ec1f8ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:46:43.455 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:43.448+0000 7f68eaffd640 1 --2- 172.21.15.123:0/2255090063 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f68d8004810 0x7f68ec1f8ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:43.455 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:43.448+0000 7f68eaffd640 1 -- 172.21.15.123:0/2255090063 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f68ec0c49b0 con 0x7f68ec0b4a70 2024-09-16T15:46:43.455 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:43.448+0000 7f68eaffd640 1 --2- 172.21.15.123:0/2255090063 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f68ec0b4a70 0x7f68ec151e10 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7f68d40af040 tx=0x7f68d40056a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:46:43.456 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:43.448+0000 7f68e3fff640 1 -- 172.21.15.123:0/2255090063 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68d40b1040 con 0x7f68ec0b4a70 2024-09-16T15:46:43.456 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:43.448+0000 7f68e3fff640 1 -- 172.21.15.123:0/2255090063 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f68d40acaa0 con 0x7f68ec0b4a70 2024-09-16T15:46:43.456 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:43.448+0000 7f68e3fff640 1 -- 172.21.15.123:0/2255090063 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68d40046d0 con 0x7f68ec0b4a70 2024-09-16T15:46:43.456 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:43.448+0000 7f68f1ef7640 1 -- 172.21.15.123:0/2255090063 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f68ec0c4c70 con 0x7f68ec0b4a70 2024-09-16T15:46:43.456 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:43.448+0000 7f68f1ef7640 1 -- 172.21.15.123:0/2255090063 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f68ec0b75c0 con 0x7f68ec0b4a70 2024-09-16T15:46:43.457 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:43.452+0000 7f68f1ef7640 1 -- 172.21.15.123:0/2255090063 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f68ec06d6a0 con 0x7f68ec0b4a70 2024-09-16T15:46:43.457 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:43.452+0000 7f68e3fff640 1 -- 172.21.15.123:0/2255090063 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 10) ==== 97284+0+0 (secure 0 0 0) 0x7f68d40bf060 con 0x7f68ec0b4a70 2024-09-16T15:46:43.457 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:43.452+0000 7f68e3fff640 1 --2- 172.21.15.123:0/2255090063 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f68cc073100 0x7f68cc0755c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:46:43.457 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:43.452+0000 7f68e3fff640 1 -- 172.21.15.123:0/2255090063 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(58..58 src has 1..58) ==== 5709+0+0 (secure 0 0 0) 0x7f68d412e230 con 0x7f68ec0b4a70 2024-09-16T15:46:43.457 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:43.452+0000 7f68eb7fe640 1 --2- 172.21.15.123:0/2255090063 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f68cc073100 0x7f68cc0755c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:46:43.457 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:43.452+0000 7f68eb7fe640 1 --2- 172.21.15.123:0/2255090063 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f68cc073100 0x7f68cc0755c0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f68e40058d0 tx=0x7f68e400b4d0 comp rx=0 tx=0).ready entity=mgr.4135 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:46:43.460 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:43.456+0000 7f68e3fff640 1 -- 172.21.15.123:0/2255090063 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f68d4016f80 con 0x7f68ec0b4a70 2024-09-16T15:46:43.647 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:43.640+0000 7f68f1ef7640 1 -- 172.21.15.123:0/2255090063 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f68ec0aef60 con 0x7f68ec0b4a70 2024-09-16T15:46:43.647 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:43.644+0000 7f68e3fff640 1 -- 172.21.15.123:0/2255090063 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v58) ==== 80+0+0 (secure 0 0 0) 0x7f68d40f90e0 con 0x7f68ec0b4a70 2024-09-16T15:46:43.647 INFO:teuthology.orchestra.run.smithi123.stderr:listed 0 entries 2024-09-16T15:46:43.650 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:43.644+0000 7f68f1ef7640 1 -- 172.21.15.123:0/2255090063 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f68cc073100 msgr2=0x7f68cc0755c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:46:43.650 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:43.644+0000 7f68f1ef7640 1 --2- 172.21.15.123:0/2255090063 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f68cc073100 0x7f68cc0755c0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f68e40058d0 tx=0x7f68e400b4d0 comp rx=0 tx=0).stop 2024-09-16T15:46:43.650 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:43.644+0000 7f68f1ef7640 1 -- 172.21.15.123:0/2255090063 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f68ec0b4a70 msgr2=0x7f68ec151e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:46:43.650 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:43.644+0000 7f68f1ef7640 1 --2- 172.21.15.123:0/2255090063 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f68ec0b4a70 0x7f68ec151e10 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7f68d40af040 tx=0x7f68d40056a0 comp rx=0 tx=0).stop 2024-09-16T15:46:43.650 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:43.644+0000 7f68f1ef7640 1 -- 172.21.15.123:0/2255090063 shutdown_connections 2024-09-16T15:46:43.650 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:43.644+0000 7f68f1ef7640 1 --2- 172.21.15.123:0/2255090063 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f68cc073100 0x7f68cc0755c0 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:43.650 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:43.644+0000 7f68f1ef7640 1 --2- 172.21.15.123:0/2255090063 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f68ec0715c0 0x7f68ec071a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:43.650 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:43.644+0000 7f68f1ef7640 1 --2- 172.21.15.123:0/2255090063 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f68ec0b4a70 0x7f68ec151e10 unknown :-1 s=CLOSED pgs=697 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:43.650 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:43.644+0000 7f68f1ef7640 1 --2- 172.21.15.123:0/2255090063 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f68d8004810 0x7f68ec1f8ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:43.650 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:43.644+0000 7f68f1ef7640 1 -- 172.21.15.123:0/2255090063 >> 172.21.15.123:0/2255090063 conn(0x7f68ec09aa40 msgr2=0x7f68ec0ce200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:46:43.651 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:43.644+0000 7f68f1ef7640 1 -- 172.21.15.123:0/2255090063 shutdown_connections 2024-09-16T15:46:43.651 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:43.644+0000 7f68f1ef7640 1 -- 172.21.15.123:0/2255090063 wait complete. 2024-09-16T15:46:43.666 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi184.front.sepia.ceph.com... 2024-09-16T15:46:43.666 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:46:43.667 DEBUG:teuthology.orchestra.run.smithi184:> sudo fusermount -u /home/ubuntu/cephtest/mnt.1 2024-09-16T15:46:43.679 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi184.stderr:ceph-fuse[35114]: fuse finished with error 0 and tester_r 0 2024-09-16T15:46:43.681 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi184.front.sepia.ceph.com 2024-09-16T15:46:43.681 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:46:43.681 DEBUG:teuthology.orchestra.run.smithi184:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2024-09-16T15:46:43.743 INFO:tasks.cephfs.mount:Terminating background process 2024-09-16T15:46:43.743 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2024-09-16T15:46:43.743 DEBUG:teuthology.orchestra.run.smithi123:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-16T15:46:43.749 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16336468k,nr_inodes=4084117,mode=755,inode64 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3274684k,mode=755,inode64 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev,inode64 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=17082 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3274680k,nr_inodes=818670,mode=700,uid=1000,gid=1271,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3274684k,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-09-16T15:46:43.750 INFO:tasks.cephfs.mount:Terminating background process 2024-09-16T15:46:43.750 DEBUG:teuthology.orchestra.run.smithi184:> set -ex 2024-09-16T15:46:43.750 DEBUG:teuthology.orchestra.run.smithi184:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-16T15:46:43.792 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16336488k,nr_inodes=4084122,mode=755,inode64 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3274688k,mode=755,inode64 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev,inode64 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=17834 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3274684k,nr_inodes=818671,mode=700,uid=1000,gid=1271,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3274688k,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-09-16T15:46:43.793 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-16T15:46:43.927 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:43.924+0000 7f11f3569640 1 -- 172.21.15.123:0/2218356748 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f11ec0b0a50 msgr2=0x7f11ec0b0ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:46:43.928 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:43.924+0000 7f11f3569640 1 --2- 172.21.15.123:0/2218356748 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f11ec0b0a50 0x7f11ec0b0ea0 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x7f11ec06f930 tx=0x7f11e401c770 comp rx=0 tx=0).stop 2024-09-16T15:46:43.928 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:43.924+0000 7f11f3569640 1 -- 172.21.15.123:0/2218356748 shutdown_connections 2024-09-16T15:46:43.928 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:43.924+0000 7f11f3569640 1 --2- 172.21.15.123:0/2218356748 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f11ec0b0a50 0x7f11ec0b0ea0 unknown :-1 s=CLOSED pgs=698 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:43.928 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:43.924+0000 7f11f3569640 1 --2- 172.21.15.123:0/2218356748 >> v2:172.21.15.184:3300/0 conn(0x7f11ec161620 0x7f11ec0d1da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:43.928 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:43.924+0000 7f11f3569640 1 -- 172.21.15.123:0/2218356748 >> 172.21.15.123:0/2218356748 conn(0x7f11ec0d3ea0 msgr2=0x7f11ec0d42b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:46:43.928 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:43.924+0000 7f11f3569640 1 -- 172.21.15.123:0/2218356748 shutdown_connections 2024-09-16T15:46:43.928 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:43.924+0000 7f11f3569640 1 -- 172.21.15.123:0/2218356748 wait complete. 2024-09-16T15:46:43.928 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:43.924+0000 7f11f3569640 1 Processor -- start 2024-09-16T15:46:43.929 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:43.924+0000 7f11f3569640 1 -- start start 2024-09-16T15:46:43.931 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:43.924+0000 7f11f3569640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f11ec0b0a50 0x7f11ec153960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:46:43.931 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:43.924+0000 7f11f3569640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f11ec0b2b00 0x7f11ec0ca480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:46:43.931 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:43.924+0000 7f11f3569640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f11ec161620 0x7f11ec0b76e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:46:43.931 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:43.924+0000 7f11f3569640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f11ec16b0f0 con 0x7f11ec161620 2024-09-16T15:46:43.931 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:43.924+0000 7f11f3569640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f11ec16af70 con 0x7f11ec0b0a50 2024-09-16T15:46:43.931 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:43.924+0000 7f11f3569640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f11ec16b270 con 0x7f11ec0b2b00 2024-09-16T15:46:43.931 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:43.924+0000 7f11f1b06640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f11ec161620 0x7f11ec0b76e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:46:43.931 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:43.924+0000 7f11f1b06640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f11ec161620 0x7f11ec0b76e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:54514/0 (socket says 172.21.15.123:54514) 2024-09-16T15:46:43.931 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:43.924+0000 7f11f1b06640 1 -- 172.21.15.123:0/2108799532 learned_addr learned my addr 172.21.15.123:0/2108799532 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:46:43.931 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:43.924+0000 7f11f1305640 1 --2- 172.21.15.123:0/2108799532 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f11ec0b0a50 0x7f11ec153960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:46:43.931 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:43.924+0000 7f11f0b04640 1 --2- 172.21.15.123:0/2108799532 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f11ec0b2b00 0x7f11ec0ca480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:46:43.931 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:43.924+0000 7f11f1b06640 1 -- 172.21.15.123:0/2108799532 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f11ec0b2b00 msgr2=0x7f11ec0ca480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:46:43.932 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:43.924+0000 7f11f1b06640 1 --2- 172.21.15.123:0/2108799532 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f11ec0b2b00 0x7f11ec0ca480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:43.932 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:43.924+0000 7f11f1b06640 1 -- 172.21.15.123:0/2108799532 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f11ec0b0a50 msgr2=0x7f11ec153960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:46:43.932 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:43.924+0000 7f11f1b06640 1 --2- 172.21.15.123:0/2108799532 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f11ec0b0a50 0x7f11ec153960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:43.932 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:43.924+0000 7f11f1b06640 1 -- 172.21.15.123:0/2108799532 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f11ec152ba0 con 0x7f11ec161620 2024-09-16T15:46:43.932 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:43.924+0000 7f11f1305640 1 --2- 172.21.15.123:0/2108799532 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f11ec0b0a50 0x7f11ec153960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T15:46:43.932 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:43.924+0000 7f11f1b06640 1 --2- 172.21.15.123:0/2108799532 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f11ec161620 0x7f11ec0b76e0 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7f11e4009850 tx=0x7f11e40a56e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:46:43.932 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:43.924+0000 7f11e1ffb640 1 -- 172.21.15.123:0/2108799532 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11e40ba020 con 0x7f11ec161620 2024-09-16T15:46:43.932 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:43.924+0000 7f11e1ffb640 1 -- 172.21.15.123:0/2108799532 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f11e4005030 con 0x7f11ec161620 2024-09-16T15:46:43.932 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:43.924+0000 7f11f3569640 1 -- 172.21.15.123:0/2108799532 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f11ec152e80 con 0x7f11ec161620 2024-09-16T15:46:43.932 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:43.924+0000 7f11e1ffb640 1 -- 172.21.15.123:0/2108799532 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11e40b2040 con 0x7f11ec161620 2024-09-16T15:46:43.932 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:43.924+0000 7f11f3569640 1 -- 172.21.15.123:0/2108799532 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f11ec1521e0 con 0x7f11ec161620 2024-09-16T15:46:43.933 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:43.928+0000 7f11f3569640 1 -- 172.21.15.123:0/2108799532 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f11ec0b0ea0 con 0x7f11ec161620 2024-09-16T15:46:43.933 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:43.928+0000 7f11e1ffb640 1 -- 172.21.15.123:0/2108799532 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 10) ==== 97284+0+0 (secure 0 0 0) 0x7f11e4005480 con 0x7f11ec161620 2024-09-16T15:46:43.933 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:43.928+0000 7f11e1ffb640 1 --2- 172.21.15.123:0/2108799532 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f11d0073100 0x7f11d00755c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:46:43.933 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:43.928+0000 7f11e1ffb640 1 -- 172.21.15.123:0/2108799532 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(58..58 src has 1..58) ==== 5709+0+0 (secure 0 0 0) 0x7f11e412e270 con 0x7f11ec161620 2024-09-16T15:46:43.934 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:43.928+0000 7f11f1305640 1 --2- 172.21.15.123:0/2108799532 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f11d0073100 0x7f11d00755c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:46:43.934 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:43.928+0000 7f11f1305640 1 --2- 172.21.15.123:0/2108799532 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f11d0073100 0x7f11d00755c0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f11e800ae00 tx=0x7f11e800a7e0 comp rx=0 tx=0).ready entity=mgr.4135 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:46:43.936 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:43.932+0000 7f11e1ffb640 1 -- 172.21.15.123:0/2108799532 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f11e40c4060 con 0x7f11ec161620 2024-09-16T15:46:44.145 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:44.140+0000 7f11f3569640 1 -- 172.21.15.123:0/2108799532 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f11ec1594f0 con 0x7f11ec161620 2024-09-16T15:46:44.146 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:44.140+0000 7f11e1ffb640 1 -- 172.21.15.123:0/2108799532 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 37 v37) ==== 94+0+5232 (secure 0 0 0) 0x7f11e40164b0 con 0x7f11ec161620 2024-09-16T15:46:44.146 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-09-16T15:46:44.147 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":37,"btime":"2024-09-16T15:46:33:380459+0000","default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4901,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6835/4036392311","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":4036392311},{"type":"v1","addr":"172.21.15.184:6835","nonce":4036392311}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":31},{"gid":5203,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6837/436207465","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6836","nonce":436207465},{"type":"v1","addr":"172.21.15.184:6837","nonce":436207465}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":31},{"gid":5796,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6834/4239187724","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6832","nonce":4239187724},{"type":"v1","addr":"172.21.15.123:6834","nonce":4239187724}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":31}],"filesystems":[{"mdsmap":{"epoch":36,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-16T15:46:20.516000+0000","modified":"2024-09-16T15:46:23.563031+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5799},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5799":{"gid":5799,"name":"c","rank":0,"incarnation":32,"state":"up:active","state_seq":3,"addr":"172.21.15.123:6835/3589624203","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6833","nonce":3589624203},{"type":"v1","addr":"172.21.15.123:6835","nonce":3589624203}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[9],"metadata_pool":8,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5799,"qdb_cluster":[5799]},"id":4}]} 2024-09-16T15:46:44.147 INFO:teuthology.orchestra.run.smithi123.stderr:dumped fsmap epoch 37 2024-09-16T15:46:44.149 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:44.144+0000 7f11f3569640 1 -- 172.21.15.123:0/2108799532 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f11d0073100 msgr2=0x7f11d00755c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:46:44.149 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:44.144+0000 7f11f3569640 1 --2- 172.21.15.123:0/2108799532 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f11d0073100 0x7f11d00755c0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f11e800ae00 tx=0x7f11e800a7e0 comp rx=0 tx=0).stop 2024-09-16T15:46:44.149 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:44.144+0000 7f11f3569640 1 -- 172.21.15.123:0/2108799532 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f11ec161620 msgr2=0x7f11ec0b76e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:46:44.149 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:44.144+0000 7f11f3569640 1 --2- 172.21.15.123:0/2108799532 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f11ec161620 0x7f11ec0b76e0 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7f11e4009850 tx=0x7f11e40a56e0 comp rx=0 tx=0).stop 2024-09-16T15:46:44.149 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:44.144+0000 7f11f3569640 1 -- 172.21.15.123:0/2108799532 shutdown_connections 2024-09-16T15:46:44.149 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:44.144+0000 7f11f3569640 1 --2- 172.21.15.123:0/2108799532 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f11d0073100 0x7f11d00755c0 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:44.149 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:44.144+0000 7f11f3569640 1 --2- 172.21.15.123:0/2108799532 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f11ec161620 0x7f11ec0b76e0 unknown :-1 s=CLOSED pgs=699 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:44.149 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:44.144+0000 7f11f3569640 1 --2- 172.21.15.123:0/2108799532 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f11ec0b2b00 0x7f11ec0ca480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:44.149 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:44.144+0000 7f11f3569640 1 --2- 172.21.15.123:0/2108799532 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f11ec0b0a50 0x7f11ec153960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:44.149 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:44.144+0000 7f11f3569640 1 -- 172.21.15.123:0/2108799532 >> 172.21.15.123:0/2108799532 conn(0x7f11ec0d3ea0 msgr2=0x7f11ec159cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:46:44.150 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:44.144+0000 7f11f3569640 1 -- 172.21.15.123:0/2108799532 shutdown_connections 2024-09-16T15:46:44.150 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:44.144+0000 7f11f3569640 1 -- 172.21.15.123:0/2108799532 wait complete. 2024-09-16T15:46:44.168 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-16T15:46:44.292 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:44.288+0000 7f64370f3640 1 -- 172.21.15.123:0/3509242307 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6418003a40 msgr2=0x7f6418003e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:46:44.292 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:44.288+0000 7f64370f3640 1 --2- 172.21.15.123:0/3509242307 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6418003a40 0x7f6418003e60 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7f64300d4ad0 tx=0x7f642c01c730 comp rx=0 tx=0).stop 2024-09-16T15:46:44.292 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:44.288+0000 7f6435690640 1 -- 172.21.15.123:0/3509242307 reap_dead start 2024-09-16T15:46:44.292 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:44.288+0000 7f64370f3640 1 -- 172.21.15.123:0/3509242307 shutdown_connections 2024-09-16T15:46:44.292 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:44.288+0000 7f64370f3640 1 -- 172.21.15.123:0/3509242307 >> 172.21.15.123:0/3509242307 conn(0x7f6430069650 msgr2=0x7f643009d170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:46:44.292 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:44.288+0000 7f64370f3640 1 -- 172.21.15.123:0/3509242307 shutdown_connections 2024-09-16T15:46:44.293 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:44.288+0000 7f64370f3640 1 -- 172.21.15.123:0/3509242307 wait complete. 2024-09-16T15:46:44.293 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:44.288+0000 7f64370f3640 1 Processor -- start 2024-09-16T15:46:44.293 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:44.288+0000 7f64370f3640 1 -- start start 2024-09-16T15:46:44.295 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:44.288+0000 7f64370f3640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f6418003a40 0x7f643015acc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:46:44.295 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:44.288+0000 7f64370f3640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f64300b9350 0x7f64300b97d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:46:44.295 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:44.288+0000 7f64370f3640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f64300c8980 0x7f64300c8de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:46:44.295 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:44.288+0000 7f64370f3640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f6430169eb0 con 0x7f64300c8980 2024-09-16T15:46:44.295 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:44.288+0000 7f64370f3640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f6430169d30 con 0x7f64300b9350 2024-09-16T15:46:44.295 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:44.288+0000 7f64370f3640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f643016a030 con 0x7f6418003a40 2024-09-16T15:46:44.295 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:44.288+0000 7f6435690640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f64300c8980 0x7f64300c8de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:46:44.295 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:44.288+0000 7f6435690640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f64300c8980 0x7f64300c8de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:54538/0 (socket says 172.21.15.123:54538) 2024-09-16T15:46:44.296 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:44.288+0000 7f6435690640 1 -- 172.21.15.123:0/2139611255 learned_addr learned my addr 172.21.15.123:0/2139611255 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:46:44.296 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:44.288+0000 7f6434e8f640 1 --2- 172.21.15.123:0/2139611255 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f6418003a40 0x7f643015acc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:46:44.296 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:44.288+0000 7f6427fff640 1 --2- 172.21.15.123:0/2139611255 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f64300b9350 0x7f64300b97d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:46:44.296 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:44.288+0000 7f6435690640 1 -- 172.21.15.123:0/2139611255 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f6418003a40 msgr2=0x7f643015acc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:46:44.296 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:44.288+0000 7f6435690640 1 --2- 172.21.15.123:0/2139611255 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f6418003a40 0x7f643015acc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:44.296 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:44.288+0000 7f6435690640 1 -- 172.21.15.123:0/2139611255 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f64300b9350 msgr2=0x7f64300b97d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:46:44.296 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:44.288+0000 7f6435690640 1 --2- 172.21.15.123:0/2139611255 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f64300b9350 0x7f64300b97d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:44.296 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:44.288+0000 7f6435690640 1 -- 172.21.15.123:0/2139611255 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f64300c9320 con 0x7f64300c8980 2024-09-16T15:46:44.296 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:44.292+0000 7f6435690640 1 --2- 172.21.15.123:0/2139611255 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f64300c8980 0x7f64300c8de0 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7f64300d2fe0 tx=0x7f6428009b00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:46:44.296 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:44.292+0000 7f64257fa640 1 -- 172.21.15.123:0/2139611255 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6428011910 con 0x7f64300c8980 2024-09-16T15:46:44.296 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:44.292+0000 7f64257fa640 1 -- 172.21.15.123:0/2139611255 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f6428011ba0 con 0x7f64300c8980 2024-09-16T15:46:44.296 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:44.292+0000 7f64370f3640 1 -- 172.21.15.123:0/2139611255 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f64300c7b10 con 0x7f64300c8980 2024-09-16T15:46:44.296 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:44.292+0000 7f64257fa640 1 -- 172.21.15.123:0/2139611255 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6428010430 con 0x7f64300c8980 2024-09-16T15:46:44.297 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:44.292+0000 7f64370f3640 1 -- 172.21.15.123:0/2139611255 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f64300c80d0 con 0x7f64300c8980 2024-09-16T15:46:44.297 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:44.292+0000 7f64370f3640 1 -- 172.21.15.123:0/2139611255 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f64300d3af0 con 0x7f64300c8980 2024-09-16T15:46:44.297 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:44.292+0000 7f64257fa640 1 -- 172.21.15.123:0/2139611255 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 10) ==== 97284+0+0 (secure 0 0 0) 0x7f64280105f0 con 0x7f64300c8980 2024-09-16T15:46:44.298 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:44.292+0000 7f64257fa640 1 --2- 172.21.15.123:0/2139611255 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f641c073100 0x7f641c0755c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:46:44.298 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:44.292+0000 7f64257fa640 1 -- 172.21.15.123:0/2139611255 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(58..58 src has 1..58) ==== 5709+0+0 (secure 0 0 0) 0x7f6428095350 con 0x7f64300c8980 2024-09-16T15:46:44.298 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:44.292+0000 7f6434e8f640 1 --2- 172.21.15.123:0/2139611255 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f641c073100 0x7f641c0755c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:46:44.298 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:44.292+0000 7f6434e8f640 1 --2- 172.21.15.123:0/2139611255 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f641c073100 0x7f641c0755c0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f6418004790 tx=0x7f642c019520 comp rx=0 tx=0).ready entity=mgr.4135 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:46:44.301 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:44.296+0000 7f64257fa640 1 -- 172.21.15.123:0/2139611255 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f6428015030 con 0x7f64300c8980 2024-09-16T15:46:44.466 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:44.460+0000 7f64257fa640 1 -- 172.21.15.123:0/2139611255 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mgrmap(e 11) ==== 97386+0+0 (secure 0 0 0) 0x7f642805be10 con 0x7f64300c8980 2024-09-16T15:46:44.512 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:44.508+0000 7f64370f3640 1 -- 172.21.15.123:0/2139611255 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f643006e2f0 con 0x7f64300c8980 2024-09-16T15:46:44.513 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:44.508+0000 7f64257fa640 1 -- 172.21.15.123:0/2139611255 <== mon.0 v2:172.21.15.123:3300/0 8 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 37 v37) ==== 94+0+5232 (secure 0 0 0) 0x7f64280213a0 con 0x7f64300c8980 2024-09-16T15:46:44.513 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-09-16T15:46:44.513 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":37,"btime":"2024-09-16T15:46:33:380459+0000","default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4901,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6835/4036392311","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":4036392311},{"type":"v1","addr":"172.21.15.184:6835","nonce":4036392311}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":31},{"gid":5203,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6837/436207465","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6836","nonce":436207465},{"type":"v1","addr":"172.21.15.184:6837","nonce":436207465}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":31},{"gid":5796,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6834/4239187724","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6832","nonce":4239187724},{"type":"v1","addr":"172.21.15.123:6834","nonce":4239187724}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":31}],"filesystems":[{"mdsmap":{"epoch":36,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-16T15:46:20.516000+0000","modified":"2024-09-16T15:46:23.563031+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5799},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5799":{"gid":5799,"name":"c","rank":0,"incarnation":32,"state":"up:active","state_seq":3,"addr":"172.21.15.123:6835/3589624203","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6833","nonce":3589624203},{"type":"v1","addr":"172.21.15.123:6835","nonce":3589624203}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[9],"metadata_pool":8,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5799,"qdb_cluster":[5799]},"id":4}]} 2024-09-16T15:46:44.513 INFO:teuthology.orchestra.run.smithi123.stderr:dumped fsmap epoch 37 2024-09-16T15:46:44.515 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:44.512+0000 7f64370f3640 1 -- 172.21.15.123:0/2139611255 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f641c073100 msgr2=0x7f641c0755c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:46:44.515 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:44.512+0000 7f64370f3640 1 --2- 172.21.15.123:0/2139611255 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f641c073100 0x7f641c0755c0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f6418004790 tx=0x7f642c019520 comp rx=0 tx=0).stop 2024-09-16T15:46:44.515 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:44.512+0000 7f64370f3640 1 -- 172.21.15.123:0/2139611255 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f64300c8980 msgr2=0x7f64300c8de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:46:44.515 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:44.512+0000 7f64370f3640 1 --2- 172.21.15.123:0/2139611255 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f64300c8980 0x7f64300c8de0 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7f64300d2fe0 tx=0x7f6428009b00 comp rx=0 tx=0).stop 2024-09-16T15:46:44.515 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:44.512+0000 7f64370f3640 1 -- 172.21.15.123:0/2139611255 shutdown_connections 2024-09-16T15:46:44.516 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:44.512+0000 7f64370f3640 1 --2- 172.21.15.123:0/2139611255 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f641c073100 0x7f641c0755c0 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:44.516 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:44.512+0000 7f64370f3640 1 --2- 172.21.15.123:0/2139611255 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f64300c8980 0x7f64300c8de0 unknown :-1 s=CLOSED pgs=702 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:44.516 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:44.512+0000 7f64370f3640 1 --2- 172.21.15.123:0/2139611255 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f64300b9350 0x7f64300b97d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:44.516 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:44.512+0000 7f64370f3640 1 --2- 172.21.15.123:0/2139611255 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f6418003a40 0x7f643015acc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:44.516 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:44.512+0000 7f64370f3640 1 -- 172.21.15.123:0/2139611255 >> 172.21.15.123:0/2139611255 conn(0x7f6430069650 msgr2=0x7f64300d0650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:46:44.516 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:44.512+0000 7f64370f3640 1 -- 172.21.15.123:0/2139611255 shutdown_connections 2024-09-16T15:46:44.516 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:44.512+0000 7f64370f3640 1 -- 172.21.15.123:0/2139611255 wait complete. 2024-09-16T15:46:44.531 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-16T15:46:44.656 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:44.652+0000 7f5c1999f640 1 -- 172.21.15.123:0/3737178157 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5c14074600 msgr2=0x7f5c140d0630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:46:44.656 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:44.652+0000 7f5c1999f640 1 --2- 172.21.15.123:0/3737178157 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5c14074600 0x7f5c140d0630 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7f5c140cbd60 tx=0x7f5c0c01c820 comp rx=0 tx=0).stop 2024-09-16T15:46:44.656 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:44.652+0000 7f5c1999f640 1 -- 172.21.15.123:0/3737178157 shutdown_connections 2024-09-16T15:46:44.657 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:44.652+0000 7f5c1999f640 1 --2- 172.21.15.123:0/3737178157 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5c14074600 0x7f5c140d0630 unknown :-1 s=CLOSED pgs=704 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:44.657 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:44.652+0000 7f5c1999f640 1 -- 172.21.15.123:0/3737178157 >> 172.21.15.123:0/3737178157 conn(0x7f5c140cfb10 msgr2=0x7f5c140cff20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:46:44.657 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:44.652+0000 7f5c1999f640 1 -- 172.21.15.123:0/3737178157 shutdown_connections 2024-09-16T15:46:44.657 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:44.652+0000 7f5c1999f640 1 -- 172.21.15.123:0/3737178157 wait complete. 2024-09-16T15:46:44.657 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:44.652+0000 7f5c1999f640 1 Processor -- start 2024-09-16T15:46:44.657 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:44.652+0000 7f5c1999f640 1 -- start start 2024-09-16T15:46:44.659 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:44.652+0000 7f5c1999f640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5c140726c0 0x7f5c14155bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:46:44.660 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:44.652+0000 7f5c1999f640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f5c14074600 0x7f5c140c9550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:46:44.660 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:44.652+0000 7f5c1999f640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5c1415bca0 0x7f5c14159f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:46:44.660 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:44.652+0000 7f5c1999f640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f5c14169df0 con 0x7f5c140726c0 2024-09-16T15:46:44.660 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:44.652+0000 7f5c1999f640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f5c14169c70 con 0x7f5c1415bca0 2024-09-16T15:46:44.660 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:44.652+0000 7f5c1999f640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f5c14169f70 con 0x7f5c14074600 2024-09-16T15:46:44.660 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:44.656+0000 7f5c12ffd640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5c140726c0 0x7f5c14155bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:46:44.660 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:44.656+0000 7f5c12ffd640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5c140726c0 0x7f5c14155bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:54558/0 (socket says 172.21.15.123:54558) 2024-09-16T15:46:44.660 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:44.656+0000 7f5c12ffd640 1 -- 172.21.15.123:0/1952094480 learned_addr learned my addr 172.21.15.123:0/1952094480 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:46:44.660 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:44.656+0000 7f5c12ffd640 1 -- 172.21.15.123:0/1952094480 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f5c14074600 msgr2=0x7f5c140c9550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:46:44.660 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:44.656+0000 7f5c12ffd640 1 --2- 172.21.15.123:0/1952094480 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f5c14074600 0x7f5c140c9550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:44.660 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:44.656+0000 7f5c12ffd640 1 -- 172.21.15.123:0/1952094480 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5c1415bca0 msgr2=0x7f5c14159f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:46:44.661 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:44.656+0000 7f5c12ffd640 1 --2- 172.21.15.123:0/1952094480 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5c1415bca0 0x7f5c14159f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:44.661 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:44.656+0000 7f5c12ffd640 1 -- 172.21.15.123:0/1952094480 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5c140c84b0 con 0x7f5c140726c0 2024-09-16T15:46:44.661 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:44.656+0000 7f5c12ffd640 1 --2- 172.21.15.123:0/1952094480 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5c140726c0 0x7f5c14155bb0 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7f5c0400b440 tx=0x7f5c0400b900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:46:44.661 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:44.656+0000 7f5bfbfff640 1 -- 172.21.15.123:0/1952094480 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c04015020 con 0x7f5c140726c0 2024-09-16T15:46:44.661 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:44.656+0000 7f5bfbfff640 1 -- 172.21.15.123:0/1952094480 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f5c04011980 con 0x7f5c140726c0 2024-09-16T15:46:44.661 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:44.656+0000 7f5c1999f640 1 -- 172.21.15.123:0/1952094480 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5c140c8790 con 0x7f5c140726c0 2024-09-16T15:46:44.661 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:44.656+0000 7f5bfbfff640 1 -- 172.21.15.123:0/1952094480 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c04011dd0 con 0x7f5c140726c0 2024-09-16T15:46:44.661 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:44.656+0000 7f5c1999f640 1 -- 172.21.15.123:0/1952094480 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5c140c8d50 con 0x7f5c140726c0 2024-09-16T15:46:44.662 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:44.656+0000 7f5c1999f640 1 -- 172.21.15.123:0/1952094480 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5c140d0630 con 0x7f5c140726c0 2024-09-16T15:46:44.662 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:44.656+0000 7f5bfbfff640 1 -- 172.21.15.123:0/1952094480 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 11) ==== 97386+0+0 (secure 0 0 0) 0x7f5c04018020 con 0x7f5c140726c0 2024-09-16T15:46:44.662 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:44.656+0000 7f5bfbfff640 1 --2- 172.21.15.123:0/1952094480 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f5c00073220 0x7f5c000756e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:46:44.662 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:44.656+0000 7f5bfbfff640 1 -- 172.21.15.123:0/1952094480 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(58..58 src has 1..58) ==== 5709+0+0 (secure 0 0 0) 0x7f5c040945c0 con 0x7f5c140726c0 2024-09-16T15:46:44.662 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:44.656+0000 7f5c127fc640 1 --2- 172.21.15.123:0/1952094480 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f5c00073220 0x7f5c000756e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:46:44.662 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:44.660+0000 7f5c127fc640 1 --2- 172.21.15.123:0/1952094480 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f5c00073220 0x7f5c000756e0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f5c14166b20 tx=0x7f5c08007c60 comp rx=0 tx=0).ready entity=mgr.4135 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:46:44.665 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:44.660+0000 7f5bfbfff640 1 -- 172.21.15.123:0/1952094480 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f5c0401f020 con 0x7f5c140726c0 2024-09-16T15:46:44.840 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:44.836+0000 7f5c1999f640 1 -- 172.21.15.123:0/1952094480 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f5c1415f630 con 0x7f5c140726c0 2024-09-16T15:46:44.842 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:44.840+0000 7f5bfbfff640 1 -- 172.21.15.123:0/1952094480 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v58) ==== 74+0+16785 (secure 0 0 0) 0x7f5c0400d040 con 0x7f5c140726c0 2024-09-16T15:46:44.843 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-09-16T15:46:44.843 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":58,"fsid":"56077f89-18c8-486f-ae7e-eecc0bb7822c","created":"2024-09-16T15:42:26.364892+0000","modified":"2024-09-16T15:46:22.140001+0000","last_up_change":"2024-09-16T15:42:38.707234+0000","last_in_change":"2024-09-16T15:42:29.423662+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-16T15:42:39.287764+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_metadata","create_time":"2024-09-16T15:46:18.173124+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"57","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_data","create_time":"2024-09-16T15:46:19.478016+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"58","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9e7c1955-a607-4671-9be7-f2ef760ceeaa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6800","nonce":3882044969},{"type":"v1","addr":"172.21.15.123:6801","nonce":3882044969}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6802","nonce":3882044969},{"type":"v1","addr":"172.21.15.123:6803","nonce":3882044969}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6806","nonce":3882044969},{"type":"v1","addr":"172.21.15.123:6807","nonce":3882044969}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6804","nonce":3882044969},{"type":"v1","addr":"172.21.15.123:6805","nonce":3882044969}]},"public_addr":"172.21.15.123:6801/3882044969","cluster_addr":"172.21.15.123:6803/3882044969","heartbeat_back_addr":"172.21.15.123:6807/3882044969","heartbeat_front_addr":"172.21.15.123:6805/3882044969","state":["exists","up"]},{"osd":1,"uuid":"c540cfde-0977-4742-9b17-fd634668e010","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6816","nonce":4037001850},{"type":"v1","addr":"172.21.15.123:6817","nonce":4037001850}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6818","nonce":4037001850},{"type":"v1","addr":"172.21.15.123:6819","nonce":4037001850}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6822","nonce":4037001850},{"type":"v1","addr":"172.21.15.123:6823","nonce":4037001850}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6820","nonce":4037001850},{"type":"v1","addr":"172.21.15.123:6821","nonce":4037001850}]},"public_addr":"172.21.15.123:6817/4037001850","cluster_addr":"172.21.15.123:6819/4037001850","heartbeat_back_addr":"172.21.15.123:6823/4037001850","heartbeat_front_addr":"172.21.15.123:6821/4037001850","state":["exists","up"]},{"osd":2,"uuid":"9864a496-f169-486b-95c4-1d59162793a7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6824","nonce":1332143088},{"type":"v1","addr":"172.21.15.123:6825","nonce":1332143088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6826","nonce":1332143088},{"type":"v1","addr":"172.21.15.123:6827","nonce":1332143088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6830","nonce":1332143088},{"type":"v1","addr":"172.21.15.123:6831","nonce":1332143088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6828","nonce":1332143088},{"type":"v1","addr":"172.21.15.123:6829","nonce":1332143088}]},"public_addr":"172.21.15.123:6825/1332143088","cluster_addr":"172.21.15.123:6827/1332143088","heartbeat_back_addr":"172.21.15.123:6831/1332143088","heartbeat_front_addr":"172.21.15.123:6829/1332143088","state":["exists","up"]},{"osd":3,"uuid":"c6d1f817-f074-4325-95e1-578eba13edda","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6808","nonce":988586998},{"type":"v1","addr":"172.21.15.123:6809","nonce":988586998}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6810","nonce":988586998},{"type":"v1","addr":"172.21.15.123:6811","nonce":988586998}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6814","nonce":988586998},{"type":"v1","addr":"172.21.15.123:6815","nonce":988586998}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6812","nonce":988586998},{"type":"v1","addr":"172.21.15.123:6813","nonce":988586998}]},"public_addr":"172.21.15.123:6809/988586998","cluster_addr":"172.21.15.123:6811/988586998","heartbeat_back_addr":"172.21.15.123:6815/988586998","heartbeat_front_addr":"172.21.15.123:6813/988586998","state":["exists","up"]},{"osd":4,"uuid":"200a8df4-50ef-40a5-b48a-58981269c086","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6816","nonce":3387813890},{"type":"v1","addr":"172.21.15.184:6817","nonce":3387813890}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6818","nonce":3387813890},{"type":"v1","addr":"172.21.15.184:6819","nonce":3387813890}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6822","nonce":3387813890},{"type":"v1","addr":"172.21.15.184:6823","nonce":3387813890}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6820","nonce":3387813890},{"type":"v1","addr":"172.21.15.184:6821","nonce":3387813890}]},"public_addr":"172.21.15.184:6817/3387813890","cluster_addr":"172.21.15.184:6819/3387813890","heartbeat_back_addr":"172.21.15.184:6823/3387813890","heartbeat_front_addr":"172.21.15.184:6821/3387813890","state":["exists","up"]},{"osd":5,"uuid":"32d5a900-da3d-412c-bc19-13861a2368d8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6808","nonce":2680916017},{"type":"v1","addr":"172.21.15.184:6809","nonce":2680916017}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6810","nonce":2680916017},{"type":"v1","addr":"172.21.15.184:6811","nonce":2680916017}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6814","nonce":2680916017},{"type":"v1","addr":"172.21.15.184:6815","nonce":2680916017}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6812","nonce":2680916017},{"type":"v1","addr":"172.21.15.184:6813","nonce":2680916017}]},"public_addr":"172.21.15.184:6809/2680916017","cluster_addr":"172.21.15.184:6811/2680916017","heartbeat_back_addr":"172.21.15.184:6815/2680916017","heartbeat_front_addr":"172.21.15.184:6813/2680916017","state":["exists","up"]},{"osd":6,"uuid":"cde6f39e-aa1b-4260-ac19-774ee6aaca72","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6824","nonce":2183120457},{"type":"v1","addr":"172.21.15.184:6825","nonce":2183120457}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6826","nonce":2183120457},{"type":"v1","addr":"172.21.15.184:6827","nonce":2183120457}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6830","nonce":2183120457},{"type":"v1","addr":"172.21.15.184:6831","nonce":2183120457}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6828","nonce":2183120457},{"type":"v1","addr":"172.21.15.184:6829","nonce":2183120457}]},"public_addr":"172.21.15.184:6825/2183120457","cluster_addr":"172.21.15.184:6827/2183120457","heartbeat_back_addr":"172.21.15.184:6831/2183120457","heartbeat_front_addr":"172.21.15.184:6829/2183120457","state":["exists","up"]},{"osd":7,"uuid":"4f381778-eee1-47c6-804f-df0e6b33e313","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6800","nonce":4102957285},{"type":"v1","addr":"172.21.15.184:6801","nonce":4102957285}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6802","nonce":4102957285},{"type":"v1","addr":"172.21.15.184:6803","nonce":4102957285}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6806","nonce":4102957285},{"type":"v1","addr":"172.21.15.184:6807","nonce":4102957285}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6804","nonce":4102957285},{"type":"v1","addr":"172.21.15.184:6805","nonce":4102957285}]},"public_addr":"172.21.15.184:6801/4102957285","cluster_addr":"172.21.15.184:6803/4102957285","heartbeat_back_addr":"172.21.15.184:6807/4102957285","heartbeat_front_addr":"172.21.15.184:6805/4102957285","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:35.972210+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.222007+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.198617+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.385663+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.358935+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.156066+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.752973+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:35.682846+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T15:46:44.845 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:44.840+0000 7f5c1999f640 1 -- 172.21.15.123:0/1952094480 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f5c00073220 msgr2=0x7f5c000756e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:46:44.845 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:44.840+0000 7f5c1999f640 1 --2- 172.21.15.123:0/1952094480 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f5c00073220 0x7f5c000756e0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f5c14166b20 tx=0x7f5c08007c60 comp rx=0 tx=0).stop 2024-09-16T15:46:44.845 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:44.840+0000 7f5c1999f640 1 -- 172.21.15.123:0/1952094480 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5c140726c0 msgr2=0x7f5c14155bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:46:44.845 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:44.840+0000 7f5c1999f640 1 --2- 172.21.15.123:0/1952094480 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5c140726c0 0x7f5c14155bb0 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7f5c0400b440 tx=0x7f5c0400b900 comp rx=0 tx=0).stop 2024-09-16T15:46:44.846 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:44.840+0000 7f5c1999f640 1 -- 172.21.15.123:0/1952094480 shutdown_connections 2024-09-16T15:46:44.846 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:44.840+0000 7f5c1999f640 1 --2- 172.21.15.123:0/1952094480 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f5c00073220 0x7f5c000756e0 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:44.846 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:44.840+0000 7f5c1999f640 1 --2- 172.21.15.123:0/1952094480 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5c1415bca0 0x7f5c14159f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:44.846 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:44.840+0000 7f5c1999f640 1 --2- 172.21.15.123:0/1952094480 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f5c14074600 0x7f5c140c9550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:44.846 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:44.840+0000 7f5c1999f640 1 --2- 172.21.15.123:0/1952094480 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5c140726c0 0x7f5c14155bb0 unknown :-1 s=CLOSED pgs=705 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:44.846 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:44.840+0000 7f5c1999f640 1 -- 172.21.15.123:0/1952094480 >> 172.21.15.123:0/1952094480 conn(0x7f5c140cfb10 msgr2=0x7f5c140ca960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:46:44.846 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:44.840+0000 7f5c1999f640 1 -- 172.21.15.123:0/1952094480 shutdown_connections 2024-09-16T15:46:44.846 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:44.840+0000 7f5c1999f640 1 -- 172.21.15.123:0/1952094480 wait complete. 2024-09-16T15:46:44.861 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-09-16T15:46:44.862 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-16T15:46:44.988 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:44.984+0000 7f4b721ed640 1 -- 172.21.15.123:0/2044965498 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4b6c158120 msgr2=0x7f4b6c0ca320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:46:44.988 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:44.984+0000 7f4b721ed640 1 --2- 172.21.15.123:0/2044965498 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4b6c158120 0x7f4b6c0ca320 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7f4b5c009900 tx=0x7f4b5c01c850 comp rx=0 tx=0).stop 2024-09-16T15:46:44.988 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:44.984+0000 7f4b721ed640 1 -- 172.21.15.123:0/2044965498 shutdown_connections 2024-09-16T15:46:44.988 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:44.984+0000 7f4b721ed640 1 --2- 172.21.15.123:0/2044965498 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f4b6c06f5f0 0x7f4b6c15aa60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:44.988 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:44.984+0000 7f4b721ed640 1 --2- 172.21.15.123:0/2044965498 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4b6c158120 0x7f4b6c0ca320 unknown :-1 s=CLOSED pgs=706 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:44.988 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:44.984+0000 7f4b721ed640 1 -- 172.21.15.123:0/2044965498 >> 172.21.15.123:0/2044965498 conn(0x7f4b6c0cfb10 msgr2=0x7f4b6c0cff20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:46:44.988 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:44.984+0000 7f4b721ed640 1 -- 172.21.15.123:0/2044965498 shutdown_connections 2024-09-16T15:46:44.988 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:44.984+0000 7f4b721ed640 1 -- 172.21.15.123:0/2044965498 wait complete. 2024-09-16T15:46:44.989 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:44.984+0000 7f4b721ed640 1 Processor -- start 2024-09-16T15:46:44.989 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:44.984+0000 7f4b721ed640 1 -- start start 2024-09-16T15:46:44.991 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:44.984+0000 7f4b721ed640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4b6c06f5f0 0x7f4b6c14f920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:46:44.991 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:44.984+0000 7f4b721ed640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f4b6c0cf5a0 0x7f4b6c1559a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:46:44.991 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:44.984+0000 7f4b721ed640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4b6c158120 0x7f4b6c15d090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:46:44.991 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:44.984+0000 7f4b721ed640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f4b6c15b720 con 0x7f4b6c158120 2024-09-16T15:46:44.991 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:44.984+0000 7f4b721ed640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f4b6c15b5a0 con 0x7f4b6c06f5f0 2024-09-16T15:46:44.991 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:44.984+0000 7f4b721ed640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f4b6c15b8a0 con 0x7f4b6c0cf5a0 2024-09-16T15:46:44.991 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:44.984+0000 7f4b6bfff640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4b6c158120 0x7f4b6c15d090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:46:44.991 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:44.984+0000 7f4b6bfff640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4b6c158120 0x7f4b6c15d090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:54574/0 (socket says 172.21.15.123:54574) 2024-09-16T15:46:44.991 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:44.984+0000 7f4b6bfff640 1 -- 172.21.15.123:0/4095522732 learned_addr learned my addr 172.21.15.123:0/4095522732 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:46:44.992 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:44.984+0000 7f4b6b7fe640 1 --2- 172.21.15.123:0/4095522732 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4b6c06f5f0 0x7f4b6c14f920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:46:44.992 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:44.984+0000 7f4b6affd640 1 --2- 172.21.15.123:0/4095522732 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f4b6c0cf5a0 0x7f4b6c1559a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:46:44.992 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:44.984+0000 7f4b6bfff640 1 -- 172.21.15.123:0/4095522732 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f4b6c0cf5a0 msgr2=0x7f4b6c1559a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:46:44.992 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:44.984+0000 7f4b6bfff640 1 --2- 172.21.15.123:0/4095522732 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f4b6c0cf5a0 0x7f4b6c1559a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:44.992 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:44.984+0000 7f4b6bfff640 1 -- 172.21.15.123:0/4095522732 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4b6c06f5f0 msgr2=0x7f4b6c14f920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:46:44.992 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:44.984+0000 7f4b6bfff640 1 --2- 172.21.15.123:0/4095522732 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4b6c06f5f0 0x7f4b6c14f920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:44.992 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:44.984+0000 7f4b6bfff640 1 -- 172.21.15.123:0/4095522732 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4b6c156460 con 0x7f4b6c158120 2024-09-16T15:46:44.992 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:44.984+0000 7f4b6bfff640 1 --2- 172.21.15.123:0/4095522732 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4b6c158120 0x7f4b6c15d090 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7f4b6400dd80 tx=0x7f4b6400b730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:46:44.992 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:44.988+0000 7f4b53fff640 1 -- 172.21.15.123:0/4095522732 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b64004d30 con 0x7f4b6c158120 2024-09-16T15:46:44.992 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:44.988+0000 7f4b53fff640 1 -- 172.21.15.123:0/4095522732 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f4b6400be40 con 0x7f4b6c158120 2024-09-16T15:46:44.992 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:44.988+0000 7f4b53fff640 1 -- 172.21.15.123:0/4095522732 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b6400e8c0 con 0x7f4b6c158120 2024-09-16T15:46:44.992 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:44.988+0000 7f4b721ed640 1 -- 172.21.15.123:0/4095522732 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4b6c156740 con 0x7f4b6c158120 2024-09-16T15:46:44.993 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:44.988+0000 7f4b721ed640 1 -- 172.21.15.123:0/4095522732 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4b6c072980 con 0x7f4b6c158120 2024-09-16T15:46:44.993 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:44.988+0000 7f4b721ed640 1 -- 172.21.15.123:0/4095522732 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4b6c072b80 con 0x7f4b6c158120 2024-09-16T15:46:44.993 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:44.988+0000 7f4b53fff640 1 -- 172.21.15.123:0/4095522732 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 11) ==== 97386+0+0 (secure 0 0 0) 0x7f4b6400ea60 con 0x7f4b6c158120 2024-09-16T15:46:44.994 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:44.988+0000 7f4b53fff640 1 --2- 172.21.15.123:0/4095522732 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f4b58073220 0x7f4b580756e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:46:44.994 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:44.988+0000 7f4b53fff640 1 -- 172.21.15.123:0/4095522732 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(58..58 src has 1..58) ==== 5709+0+0 (secure 0 0 0) 0x7f4b640950b0 con 0x7f4b6c158120 2024-09-16T15:46:44.994 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:44.988+0000 7f4b6b7fe640 1 --2- 172.21.15.123:0/4095522732 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f4b58073220 0x7f4b580756e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:46:44.994 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:44.988+0000 7f4b6b7fe640 1 --2- 172.21.15.123:0/4095522732 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f4b58073220 0x7f4b580756e0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f4b5c0ae000 tx=0x7f4b5c01b090 comp rx=0 tx=0).ready entity=mgr.4135 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:46:44.997 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:44.992+0000 7f4b53fff640 1 -- 172.21.15.123:0/4095522732 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f4b6c072b80 con 0x7f4b6c158120 2024-09-16T15:46:45.210 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:45.204+0000 7f4b721ed640 1 -- 172.21.15.123:0/4095522732 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f4b6c15aa60 con 0x7f4b6c158120 2024-09-16T15:46:45.211 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:45.204+0000 7f4b53fff640 1 -- 172.21.15.123:0/4095522732 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 37 v37) ==== 94+0+5232 (secure 0 0 0) 0x7f4b6405fd70 con 0x7f4b6c158120 2024-09-16T15:46:45.211 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-09-16T15:46:45.211 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":37,"btime":"2024-09-16T15:46:33:380459+0000","default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4901,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6835/4036392311","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":4036392311},{"type":"v1","addr":"172.21.15.184:6835","nonce":4036392311}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":31},{"gid":5203,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6837/436207465","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6836","nonce":436207465},{"type":"v1","addr":"172.21.15.184:6837","nonce":436207465}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":31},{"gid":5796,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6834/4239187724","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6832","nonce":4239187724},{"type":"v1","addr":"172.21.15.123:6834","nonce":4239187724}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":31}],"filesystems":[{"mdsmap":{"epoch":36,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-16T15:46:20.516000+0000","modified":"2024-09-16T15:46:23.563031+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5799},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5799":{"gid":5799,"name":"c","rank":0,"incarnation":32,"state":"up:active","state_seq":3,"addr":"172.21.15.123:6835/3589624203","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6833","nonce":3589624203},{"type":"v1","addr":"172.21.15.123:6835","nonce":3589624203}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[9],"metadata_pool":8,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5799,"qdb_cluster":[5799]},"id":4}]} 2024-09-16T15:46:45.211 INFO:teuthology.orchestra.run.smithi123.stderr:dumped fsmap epoch 37 2024-09-16T15:46:45.213 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:45.208+0000 7f4b721ed640 1 -- 172.21.15.123:0/4095522732 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f4b58073220 msgr2=0x7f4b580756e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:46:45.213 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:45.208+0000 7f4b721ed640 1 --2- 172.21.15.123:0/4095522732 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f4b58073220 0x7f4b580756e0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f4b5c0ae000 tx=0x7f4b5c01b090 comp rx=0 tx=0).stop 2024-09-16T15:46:45.213 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:45.208+0000 7f4b721ed640 1 -- 172.21.15.123:0/4095522732 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4b6c158120 msgr2=0x7f4b6c15d090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:46:45.213 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:45.208+0000 7f4b721ed640 1 --2- 172.21.15.123:0/4095522732 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4b6c158120 0x7f4b6c15d090 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7f4b6400dd80 tx=0x7f4b6400b730 comp rx=0 tx=0).stop 2024-09-16T15:46:45.213 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:45.208+0000 7f4b721ed640 1 -- 172.21.15.123:0/4095522732 shutdown_connections 2024-09-16T15:46:45.213 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:45.208+0000 7f4b721ed640 1 --2- 172.21.15.123:0/4095522732 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f4b58073220 0x7f4b580756e0 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:45.213 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:45.208+0000 7f4b721ed640 1 --2- 172.21.15.123:0/4095522732 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4b6c158120 0x7f4b6c15d090 unknown :-1 s=CLOSED pgs=707 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:45.214 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:45.208+0000 7f4b721ed640 1 --2- 172.21.15.123:0/4095522732 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f4b6c0cf5a0 0x7f4b6c1559a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:45.214 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:45.208+0000 7f4b721ed640 1 --2- 172.21.15.123:0/4095522732 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4b6c06f5f0 0x7f4b6c14f920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:45.214 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:45.208+0000 7f4b721ed640 1 -- 172.21.15.123:0/4095522732 >> 172.21.15.123:0/4095522732 conn(0x7f4b6c0cfb10 msgr2=0x7f4b6c15c180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:46:45.214 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:45.208+0000 7f4b721ed640 1 -- 172.21.15.123:0/4095522732 shutdown_connections 2024-09-16T15:46:45.214 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:45.208+0000 7f4b721ed640 1 -- 172.21.15.123:0/4095522732 wait complete. 2024-09-16T15:46:45.229 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-16T15:46:45.351 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:45.344+0000 7f930eb77640 1 -- 172.21.15.123:0/4214322220 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f930815dd60 msgr2=0x7f93081621d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:46:45.351 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:45.344+0000 7f930eb77640 1 --2- 172.21.15.123:0/4214322220 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f930815dd60 0x7f93081621d0 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7f930808e110 tx=0x7f92f801c510 comp rx=0 tx=0).stop 2024-09-16T15:46:45.351 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:45.344+0000 7f930eb77640 1 -- 172.21.15.123:0/4214322220 shutdown_connections 2024-09-16T15:46:45.351 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:45.344+0000 7f930eb77640 1 --2- 172.21.15.123:0/4214322220 >> v2:172.21.15.184:3300/0 conn(0x7f93080d4030 0x7f93080d08a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:45.351 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:45.344+0000 7f930eb77640 1 --2- 172.21.15.123:0/4214322220 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f93080b3b20 0x7f93080d3af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:45.351 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:45.344+0000 7f930eb77640 1 --2- 172.21.15.123:0/4214322220 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f930815dd60 0x7f93081621d0 unknown :-1 s=CLOSED pgs=708 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:45.351 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:45.344+0000 7f930eb77640 1 -- 172.21.15.123:0/4214322220 >> 172.21.15.123:0/4214322220 conn(0x7f9308069650 msgr2=0x7f930809d170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:46:45.351 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:45.344+0000 7f930eb77640 1 -- 172.21.15.123:0/4214322220 shutdown_connections 2024-09-16T15:46:45.351 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:45.344+0000 7f930eb77640 1 -- 172.21.15.123:0/4214322220 wait complete. 2024-09-16T15:46:45.352 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:45.348+0000 7f930eb77640 1 Processor -- start 2024-09-16T15:46:45.352 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:45.348+0000 7f930eb77640 1 -- start start 2024-09-16T15:46:45.354 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:45.348+0000 7f930eb77640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f93080b3b20 0x7f93081559c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:46:45.354 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:45.348+0000 7f930eb77640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f93080d4030 0x7f93080c90a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:46:45.354 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:45.348+0000 7f930eb77640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f930815dd60 0x7f930806e580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:46:45.354 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:45.348+0000 7f930eb77640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f93080d2280 con 0x7f930815dd60 2024-09-16T15:46:45.354 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:45.348+0000 7f930eb77640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f93080d2100 con 0x7f93080b3b20 2024-09-16T15:46:45.354 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:45.348+0000 7f930eb77640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f93080d2400 con 0x7f93080d4030 2024-09-16T15:46:45.354 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:45.348+0000 7f930d114640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f930815dd60 0x7f930806e580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:46:45.354 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:45.348+0000 7f930d114640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f930815dd60 0x7f930806e580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:54592/0 (socket says 172.21.15.123:54592) 2024-09-16T15:46:45.354 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:45.348+0000 7f930d114640 1 -- 172.21.15.123:0/3982221554 learned_addr learned my addr 172.21.15.123:0/3982221554 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:46:45.354 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:45.348+0000 7f930d114640 1 -- 172.21.15.123:0/3982221554 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f93080d4030 msgr2=0x7f93080c90a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:46:45.355 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:45.348+0000 7f930c913640 1 --2- 172.21.15.123:0/3982221554 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f93080b3b20 0x7f93081559c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:46:45.355 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:45.348+0000 7f9307fff640 1 --2- 172.21.15.123:0/3982221554 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f93080d4030 0x7f93080c90a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:46:45.355 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:45.348+0000 7f930d114640 1 --2- 172.21.15.123:0/3982221554 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f93080d4030 0x7f93080c90a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:45.355 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:45.348+0000 7f930d114640 1 -- 172.21.15.123:0/3982221554 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f93080b3b20 msgr2=0x7f93081559c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:46:45.355 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:45.348+0000 7f930d114640 1 --2- 172.21.15.123:0/3982221554 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f93080b3b20 0x7f93081559c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:45.355 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:45.348+0000 7f930d114640 1 -- 172.21.15.123:0/3982221554 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9308155f00 con 0x7f930815dd60 2024-09-16T15:46:45.355 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:45.348+0000 7f930c913640 1 --2- 172.21.15.123:0/3982221554 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f93080b3b20 0x7f93081559c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T15:46:45.355 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:45.348+0000 7f930d114640 1 --2- 172.21.15.123:0/3982221554 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f930815dd60 0x7f930806e580 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7f930000b9a0 tx=0x7f930000be60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:46:45.355 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:45.348+0000 7f93057fa640 1 -- 172.21.15.123:0/3982221554 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9300019070 con 0x7f930815dd60 2024-09-16T15:46:45.355 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:45.348+0000 7f93057fa640 1 -- 172.21.15.123:0/3982221554 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f930000ccb0 con 0x7f930815dd60 2024-09-16T15:46:45.355 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:45.348+0000 7f93057fa640 1 -- 172.21.15.123:0/3982221554 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93000146a0 con 0x7f930815dd60 2024-09-16T15:46:45.355 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:45.348+0000 7f930eb77640 1 -- 172.21.15.123:0/3982221554 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f93080c9b60 con 0x7f930815dd60 2024-09-16T15:46:45.355 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:45.348+0000 7f930eb77640 1 -- 172.21.15.123:0/3982221554 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f93080ca120 con 0x7f930815dd60 2024-09-16T15:46:45.355 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:45.348+0000 7f930eb77640 1 -- 172.21.15.123:0/3982221554 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f93080b2db0 con 0x7f930815dd60 2024-09-16T15:46:45.356 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:45.352+0000 7f93057fa640 1 -- 172.21.15.123:0/3982221554 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 11) ==== 97386+0+0 (secure 0 0 0) 0x7f930001c070 con 0x7f930815dd60 2024-09-16T15:46:45.356 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:45.352+0000 7f93057fa640 1 --2- 172.21.15.123:0/3982221554 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f92e8073220 0x7f92e80756e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:46:45.356 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:45.352+0000 7f93057fa640 1 -- 172.21.15.123:0/3982221554 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(58..58 src has 1..58) ==== 5709+0+0 (secure 0 0 0) 0x7f93000941d0 con 0x7f930815dd60 2024-09-16T15:46:45.356 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:45.352+0000 7f930c913640 1 --2- 172.21.15.123:0/3982221554 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f92e8073220 0x7f92e80756e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:46:45.357 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:45.352+0000 7f930c913640 1 --2- 172.21.15.123:0/3982221554 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f92e8073220 0x7f92e80756e0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f92f80ae000 tx=0x7f92f801adb0 comp rx=0 tx=0).ready entity=mgr.4135 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:46:45.359 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:45.356+0000 7f93057fa640 1 -- 172.21.15.123:0/3982221554 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f930005eec0 con 0x7f930815dd60 2024-09-16T15:46:45.556 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:45.552+0000 7f930eb77640 1 -- 172.21.15.123:0/3982221554 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f93080b27f0 con 0x7f930815dd60 2024-09-16T15:46:45.557 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:45.552+0000 7f93057fa640 1 -- 172.21.15.123:0/3982221554 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 37 v37) ==== 94+0+5232 (secure 0 0 0) 0x7f930005e7c0 con 0x7f930815dd60 2024-09-16T15:46:45.557 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-09-16T15:46:45.557 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":37,"btime":"2024-09-16T15:46:33:380459+0000","default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4901,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6835/4036392311","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":4036392311},{"type":"v1","addr":"172.21.15.184:6835","nonce":4036392311}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":31},{"gid":5203,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6837/436207465","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6836","nonce":436207465},{"type":"v1","addr":"172.21.15.184:6837","nonce":436207465}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":31},{"gid":5796,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6834/4239187724","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6832","nonce":4239187724},{"type":"v1","addr":"172.21.15.123:6834","nonce":4239187724}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":31}],"filesystems":[{"mdsmap":{"epoch":36,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-16T15:46:20.516000+0000","modified":"2024-09-16T15:46:23.563031+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5799},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5799":{"gid":5799,"name":"c","rank":0,"incarnation":32,"state":"up:active","state_seq":3,"addr":"172.21.15.123:6835/3589624203","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6833","nonce":3589624203},{"type":"v1","addr":"172.21.15.123:6835","nonce":3589624203}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[9],"metadata_pool":8,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5799,"qdb_cluster":[5799]},"id":4}]} 2024-09-16T15:46:45.558 INFO:teuthology.orchestra.run.smithi123.stderr:dumped fsmap epoch 37 2024-09-16T15:46:45.560 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:45.556+0000 7f930eb77640 1 -- 172.21.15.123:0/3982221554 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f92e8073220 msgr2=0x7f92e80756e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:46:45.560 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:45.556+0000 7f930eb77640 1 --2- 172.21.15.123:0/3982221554 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f92e8073220 0x7f92e80756e0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f92f80ae000 tx=0x7f92f801adb0 comp rx=0 tx=0).stop 2024-09-16T15:46:45.560 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:45.556+0000 7f930eb77640 1 -- 172.21.15.123:0/3982221554 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f930815dd60 msgr2=0x7f930806e580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:46:45.560 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:45.556+0000 7f930eb77640 1 --2- 172.21.15.123:0/3982221554 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f930815dd60 0x7f930806e580 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7f930000b9a0 tx=0x7f930000be60 comp rx=0 tx=0).stop 2024-09-16T15:46:45.560 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:45.556+0000 7f930eb77640 1 -- 172.21.15.123:0/3982221554 shutdown_connections 2024-09-16T15:46:45.560 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:45.556+0000 7f930eb77640 1 --2- 172.21.15.123:0/3982221554 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f92e8073220 0x7f92e80756e0 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:45.560 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:45.556+0000 7f930eb77640 1 --2- 172.21.15.123:0/3982221554 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f930815dd60 0x7f930806e580 unknown :-1 s=CLOSED pgs=709 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:45.560 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:45.556+0000 7f930eb77640 1 --2- 172.21.15.123:0/3982221554 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f93080d4030 0x7f93080c90a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:45.561 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:45.556+0000 7f930eb77640 1 --2- 172.21.15.123:0/3982221554 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f93080b3b20 0x7f93081559c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:45.561 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:45.556+0000 7f930eb77640 1 -- 172.21.15.123:0/3982221554 >> 172.21.15.123:0/3982221554 conn(0x7f9308069650 msgr2=0x7f93080ccf90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:46:45.561 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:45.556+0000 7f930eb77640 1 -- 172.21.15.123:0/3982221554 shutdown_connections 2024-09-16T15:46:45.561 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:45.556+0000 7f930eb77640 1 -- 172.21.15.123:0/3982221554 wait complete. 2024-09-16T15:46:45.575 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-16T15:46:45.700 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:45.696+0000 7fc11f9a2640 1 -- 172.21.15.123:0/2564128783 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc11815e160 msgr2=0x7fc118073f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:46:45.700 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:45.696+0000 7fc11f9a2640 1 --2- 172.21.15.123:0/2564128783 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc11815e160 0x7fc118073f30 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7fc110009bc0 tx=0x7fc11001ca80 comp rx=0 tx=0).stop 2024-09-16T15:46:45.700 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:45.696+0000 7fc11f9a2640 1 -- 172.21.15.123:0/2564128783 shutdown_connections 2024-09-16T15:46:45.700 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:45.696+0000 7fc11f9a2640 1 --2- 172.21.15.123:0/2564128783 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fc11815e820 0x7fc11815ecb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:45.700 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:45.696+0000 7fc11f9a2640 1 --2- 172.21.15.123:0/2564128783 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc11815e160 0x7fc118073f30 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:45.700 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:45.696+0000 7fc11f9a2640 1 -- 172.21.15.123:0/2564128783 >> 172.21.15.123:0/2564128783 conn(0x7fc1180cfb10 msgr2=0x7fc1180cff20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:46:45.700 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:45.696+0000 7fc11f9a2640 1 -- 172.21.15.123:0/2564128783 shutdown_connections 2024-09-16T15:46:45.700 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:45.696+0000 7fc11f9a2640 1 -- 172.21.15.123:0/2564128783 wait complete. 2024-09-16T15:46:45.701 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:45.696+0000 7fc11f9a2640 1 Processor -- start 2024-09-16T15:46:45.701 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:45.696+0000 7fc11f9a2640 1 -- start start 2024-09-16T15:46:45.703 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:45.696+0000 7fc11f9a2640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fc118074470 0x7fc1180b4080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:46:45.703 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:45.696+0000 7fc11f9a2640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc11815e160 0x7fc1180cc5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:46:45.703 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:45.696+0000 7fc11f9a2640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc11815e820 0x7fc11814fed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:46:45.703 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:45.696+0000 7fc11f9a2640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fc1180cf430 con 0x7fc11815e160 2024-09-16T15:46:45.703 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:45.696+0000 7fc11f9a2640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fc11815f790 con 0x7fc11815e820 2024-09-16T15:46:45.703 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:45.696+0000 7fc11f9a2640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fc1180cf5b0 con 0x7fc118074470 2024-09-16T15:46:45.703 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:45.696+0000 7fc11cf3d640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc11815e160 0x7fc1180cc5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:46:45.703 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:45.696+0000 7fc11cf3d640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc11815e160 0x7fc1180cc5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:54594/0 (socket says 172.21.15.123:54594) 2024-09-16T15:46:45.703 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:45.696+0000 7fc11cf3d640 1 -- 172.21.15.123:0/2064714211 learned_addr learned my addr 172.21.15.123:0/2064714211 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:46:45.703 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:45.696+0000 7fc11df3f640 1 --2- 172.21.15.123:0/2064714211 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc11815e820 0x7fc11814fed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:46:45.703 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:45.696+0000 7fc11d73e640 1 --2- 172.21.15.123:0/2064714211 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fc118074470 0x7fc1180b4080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:46:45.703 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:45.696+0000 7fc11cf3d640 1 -- 172.21.15.123:0/2064714211 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fc118074470 msgr2=0x7fc1180b4080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:46:45.704 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:45.696+0000 7fc11cf3d640 1 --2- 172.21.15.123:0/2064714211 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fc118074470 0x7fc1180b4080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:45.704 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:45.696+0000 7fc11cf3d640 1 -- 172.21.15.123:0/2064714211 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc11815e820 msgr2=0x7fc11814fed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:46:45.704 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:45.696+0000 7fc11cf3d640 1 --2- 172.21.15.123:0/2064714211 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc11815e820 0x7fc11814fed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:45.704 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:45.696+0000 7fc11cf3d640 1 -- 172.21.15.123:0/2064714211 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc1180b45c0 con 0x7fc11815e160 2024-09-16T15:46:45.704 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:45.696+0000 7fc11d73e640 1 --2- 172.21.15.123:0/2064714211 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fc118074470 0x7fc1180b4080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T15:46:45.704 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:45.696+0000 7fc11cf3d640 1 --2- 172.21.15.123:0/2064714211 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc11815e160 0x7fc1180cc5f0 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7fc10800c500 tx=0x7fc10800c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:46:45.704 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:45.696+0000 7fc10dffb640 1 -- 172.21.15.123:0/2064714211 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc108017070 con 0x7fc11815e160 2024-09-16T15:46:45.704 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:45.700+0000 7fc10dffb640 1 -- 172.21.15.123:0/2064714211 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fc1080053c0 con 0x7fc11815e160 2024-09-16T15:46:45.704 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:45.700+0000 7fc11f9a2640 1 -- 172.21.15.123:0/2064714211 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc1180c70b0 con 0x7fc11815e160 2024-09-16T15:46:45.704 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:45.700+0000 7fc10dffb640 1 -- 172.21.15.123:0/2064714211 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc108012890 con 0x7fc11815e160 2024-09-16T15:46:45.704 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:45.700+0000 7fc11f9a2640 1 -- 172.21.15.123:0/2064714211 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc1180c7670 con 0x7fc11815e160 2024-09-16T15:46:45.705 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:45.700+0000 7fc11f9a2640 1 -- 172.21.15.123:0/2064714211 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc118073f30 con 0x7fc11815e160 2024-09-16T15:46:45.705 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:45.700+0000 7fc10dffb640 1 -- 172.21.15.123:0/2064714211 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 11) ==== 97386+0+0 (secure 0 0 0) 0x7fc10801a070 con 0x7fc11815e160 2024-09-16T15:46:45.705 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:45.700+0000 7fc10dffb640 1 --2- 172.21.15.123:0/2064714211 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fc0fc073220 0x7fc0fc0756e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:46:45.705 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:45.700+0000 7fc10dffb640 1 -- 172.21.15.123:0/2064714211 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(58..58 src has 1..58) ==== 5709+0+0 (secure 0 0 0) 0x7fc1080936f0 con 0x7fc11815e160 2024-09-16T15:46:45.705 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:45.700+0000 7fc11d73e640 1 --2- 172.21.15.123:0/2064714211 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fc0fc073220 0x7fc0fc0756e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:46:45.706 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:45.700+0000 7fc11d73e640 1 --2- 172.21.15.123:0/2064714211 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fc0fc073220 0x7fc0fc0756e0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7fc1100af000 tx=0x7fc11001b2c0 comp rx=0 tx=0).ready entity=mgr.4135 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:46:45.709 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:45.704+0000 7fc10dffb640 1 -- 172.21.15.123:0/2064714211 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7fc10805e4b0 con 0x7fc11815e160 2024-09-16T15:46:45.886 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:45.880+0000 7fc11f9a2640 1 -- 172.21.15.123:0/2064714211 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fc1180b0090 con 0x7fc11815e160 2024-09-16T15:46:45.888 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:45.884+0000 7fc10dffb640 1 -- 172.21.15.123:0/2064714211 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v58) ==== 74+0+16785 (secure 0 0 0) 0x7fc10805ddb0 con 0x7fc11815e160 2024-09-16T15:46:45.888 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-09-16T15:46:45.889 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":58,"fsid":"56077f89-18c8-486f-ae7e-eecc0bb7822c","created":"2024-09-16T15:42:26.364892+0000","modified":"2024-09-16T15:46:22.140001+0000","last_up_change":"2024-09-16T15:42:38.707234+0000","last_in_change":"2024-09-16T15:42:29.423662+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-16T15:42:39.287764+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_metadata","create_time":"2024-09-16T15:46:18.173124+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"57","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_data","create_time":"2024-09-16T15:46:19.478016+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"58","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9e7c1955-a607-4671-9be7-f2ef760ceeaa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6800","nonce":3882044969},{"type":"v1","addr":"172.21.15.123:6801","nonce":3882044969}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6802","nonce":3882044969},{"type":"v1","addr":"172.21.15.123:6803","nonce":3882044969}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6806","nonce":3882044969},{"type":"v1","addr":"172.21.15.123:6807","nonce":3882044969}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6804","nonce":3882044969},{"type":"v1","addr":"172.21.15.123:6805","nonce":3882044969}]},"public_addr":"172.21.15.123:6801/3882044969","cluster_addr":"172.21.15.123:6803/3882044969","heartbeat_back_addr":"172.21.15.123:6807/3882044969","heartbeat_front_addr":"172.21.15.123:6805/3882044969","state":["exists","up"]},{"osd":1,"uuid":"c540cfde-0977-4742-9b17-fd634668e010","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6816","nonce":4037001850},{"type":"v1","addr":"172.21.15.123:6817","nonce":4037001850}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6818","nonce":4037001850},{"type":"v1","addr":"172.21.15.123:6819","nonce":4037001850}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6822","nonce":4037001850},{"type":"v1","addr":"172.21.15.123:6823","nonce":4037001850}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6820","nonce":4037001850},{"type":"v1","addr":"172.21.15.123:6821","nonce":4037001850}]},"public_addr":"172.21.15.123:6817/4037001850","cluster_addr":"172.21.15.123:6819/4037001850","heartbeat_back_addr":"172.21.15.123:6823/4037001850","heartbeat_front_addr":"172.21.15.123:6821/4037001850","state":["exists","up"]},{"osd":2,"uuid":"9864a496-f169-486b-95c4-1d59162793a7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6824","nonce":1332143088},{"type":"v1","addr":"172.21.15.123:6825","nonce":1332143088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6826","nonce":1332143088},{"type":"v1","addr":"172.21.15.123:6827","nonce":1332143088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6830","nonce":1332143088},{"type":"v1","addr":"172.21.15.123:6831","nonce":1332143088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6828","nonce":1332143088},{"type":"v1","addr":"172.21.15.123:6829","nonce":1332143088}]},"public_addr":"172.21.15.123:6825/1332143088","cluster_addr":"172.21.15.123:6827/1332143088","heartbeat_back_addr":"172.21.15.123:6831/1332143088","heartbeat_front_addr":"172.21.15.123:6829/1332143088","state":["exists","up"]},{"osd":3,"uuid":"c6d1f817-f074-4325-95e1-578eba13edda","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6808","nonce":988586998},{"type":"v1","addr":"172.21.15.123:6809","nonce":988586998}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6810","nonce":988586998},{"type":"v1","addr":"172.21.15.123:6811","nonce":988586998}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6814","nonce":988586998},{"type":"v1","addr":"172.21.15.123:6815","nonce":988586998}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6812","nonce":988586998},{"type":"v1","addr":"172.21.15.123:6813","nonce":988586998}]},"public_addr":"172.21.15.123:6809/988586998","cluster_addr":"172.21.15.123:6811/988586998","heartbeat_back_addr":"172.21.15.123:6815/988586998","heartbeat_front_addr":"172.21.15.123:6813/988586998","state":["exists","up"]},{"osd":4,"uuid":"200a8df4-50ef-40a5-b48a-58981269c086","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6816","nonce":3387813890},{"type":"v1","addr":"172.21.15.184:6817","nonce":3387813890}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6818","nonce":3387813890},{"type":"v1","addr":"172.21.15.184:6819","nonce":3387813890}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6822","nonce":3387813890},{"type":"v1","addr":"172.21.15.184:6823","nonce":3387813890}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6820","nonce":3387813890},{"type":"v1","addr":"172.21.15.184:6821","nonce":3387813890}]},"public_addr":"172.21.15.184:6817/3387813890","cluster_addr":"172.21.15.184:6819/3387813890","heartbeat_back_addr":"172.21.15.184:6823/3387813890","heartbeat_front_addr":"172.21.15.184:6821/3387813890","state":["exists","up"]},{"osd":5,"uuid":"32d5a900-da3d-412c-bc19-13861a2368d8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6808","nonce":2680916017},{"type":"v1","addr":"172.21.15.184:6809","nonce":2680916017}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6810","nonce":2680916017},{"type":"v1","addr":"172.21.15.184:6811","nonce":2680916017}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6814","nonce":2680916017},{"type":"v1","addr":"172.21.15.184:6815","nonce":2680916017}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6812","nonce":2680916017},{"type":"v1","addr":"172.21.15.184:6813","nonce":2680916017}]},"public_addr":"172.21.15.184:6809/2680916017","cluster_addr":"172.21.15.184:6811/2680916017","heartbeat_back_addr":"172.21.15.184:6815/2680916017","heartbeat_front_addr":"172.21.15.184:6813/2680916017","state":["exists","up"]},{"osd":6,"uuid":"cde6f39e-aa1b-4260-ac19-774ee6aaca72","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6824","nonce":2183120457},{"type":"v1","addr":"172.21.15.184:6825","nonce":2183120457}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6826","nonce":2183120457},{"type":"v1","addr":"172.21.15.184:6827","nonce":2183120457}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6830","nonce":2183120457},{"type":"v1","addr":"172.21.15.184:6831","nonce":2183120457}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6828","nonce":2183120457},{"type":"v1","addr":"172.21.15.184:6829","nonce":2183120457}]},"public_addr":"172.21.15.184:6825/2183120457","cluster_addr":"172.21.15.184:6827/2183120457","heartbeat_back_addr":"172.21.15.184:6831/2183120457","heartbeat_front_addr":"172.21.15.184:6829/2183120457","state":["exists","up"]},{"osd":7,"uuid":"4f381778-eee1-47c6-804f-df0e6b33e313","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6800","nonce":4102957285},{"type":"v1","addr":"172.21.15.184:6801","nonce":4102957285}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6802","nonce":4102957285},{"type":"v1","addr":"172.21.15.184:6803","nonce":4102957285}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6806","nonce":4102957285},{"type":"v1","addr":"172.21.15.184:6807","nonce":4102957285}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6804","nonce":4102957285},{"type":"v1","addr":"172.21.15.184:6805","nonce":4102957285}]},"public_addr":"172.21.15.184:6801/4102957285","cluster_addr":"172.21.15.184:6803/4102957285","heartbeat_back_addr":"172.21.15.184:6807/4102957285","heartbeat_front_addr":"172.21.15.184:6805/4102957285","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:35.972210+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.222007+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.198617+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.385663+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.358935+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.156066+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.752973+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:35.682846+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T15:46:45.891 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:45.884+0000 7fc11f9a2640 1 -- 172.21.15.123:0/2064714211 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fc0fc073220 msgr2=0x7fc0fc0756e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:46:45.891 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:45.884+0000 7fc11f9a2640 1 --2- 172.21.15.123:0/2064714211 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fc0fc073220 0x7fc0fc0756e0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7fc1100af000 tx=0x7fc11001b2c0 comp rx=0 tx=0).stop 2024-09-16T15:46:45.891 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:45.884+0000 7fc11f9a2640 1 -- 172.21.15.123:0/2064714211 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc11815e160 msgr2=0x7fc1180cc5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:46:45.891 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:45.884+0000 7fc11f9a2640 1 --2- 172.21.15.123:0/2064714211 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc11815e160 0x7fc1180cc5f0 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7fc10800c500 tx=0x7fc10800c9c0 comp rx=0 tx=0).stop 2024-09-16T15:46:45.891 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:45.884+0000 7fc11f9a2640 1 -- 172.21.15.123:0/2064714211 shutdown_connections 2024-09-16T15:46:45.891 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:45.884+0000 7fc11f9a2640 1 --2- 172.21.15.123:0/2064714211 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fc0fc073220 0x7fc0fc0756e0 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:45.891 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:45.888+0000 7fc11f9a2640 1 --2- 172.21.15.123:0/2064714211 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc11815e820 0x7fc11814fed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:45.891 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:45.888+0000 7fc11f9a2640 1 --2- 172.21.15.123:0/2064714211 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc11815e160 0x7fc1180cc5f0 unknown :-1 s=CLOSED pgs=710 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:45.891 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:45.888+0000 7fc11f9a2640 1 --2- 172.21.15.123:0/2064714211 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fc118074470 0x7fc1180b4080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:45.891 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:45.888+0000 7fc11f9a2640 1 -- 172.21.15.123:0/2064714211 >> 172.21.15.123:0/2064714211 conn(0x7fc1180cfb10 msgr2=0x7fc118072ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:46:45.891 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:45.888+0000 7fc11f9a2640 1 -- 172.21.15.123:0/2064714211 shutdown_connections 2024-09-16T15:46:45.892 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:45.888+0000 7fc11f9a2640 1 -- 172.21.15.123:0/2064714211 wait complete. 2024-09-16T15:46:45.909 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-09-16T15:46:46.037 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:46.032+0000 7f00a7124640 1 -- 172.21.15.123:0/3119426058 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f00a00d0850 msgr2=0x7f00a00d2150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:46:46.038 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:46.032+0000 7f00a7124640 1 --2- 172.21.15.123:0/3119426058 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f00a00d0850 0x7f00a00d2150 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7f00a00730a0 tx=0x7f009001c830 comp rx=0 tx=0).stop 2024-09-16T15:46:46.038 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:46.032+0000 7f00a7124640 1 -- 172.21.15.123:0/3119426058 shutdown_connections 2024-09-16T15:46:46.038 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:46.032+0000 7f00a7124640 1 --2- 172.21.15.123:0/3119426058 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f00a00d0850 0x7f00a00d2150 unknown :-1 s=CLOSED pgs=711 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:46.038 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:46.032+0000 7f00a7124640 1 --2- 172.21.15.123:0/3119426058 >> v2:172.21.15.184:3300/0 conn(0x7f00a00b25a0 0x7f00a00d1c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:46.038 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:46.032+0000 7f00a7124640 1 --2- 172.21.15.123:0/3119426058 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f00a015bda0 0x7f00a00d0310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:46.038 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:46.032+0000 7f00a7124640 1 -- 172.21.15.123:0/3119426058 >> 172.21.15.123:0/3119426058 conn(0x7f00a0074bd0 msgr2=0x7f00a009aa40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:46:46.038 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:46.032+0000 7f00a7124640 1 -- 172.21.15.123:0/3119426058 shutdown_connections 2024-09-16T15:46:46.038 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:46.032+0000 7f00a7124640 1 -- 172.21.15.123:0/3119426058 wait complete. 2024-09-16T15:46:46.038 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:46.032+0000 7f00a7124640 1 Processor -- start 2024-09-16T15:46:46.039 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:46.032+0000 7f00a7124640 1 -- start start 2024-09-16T15:46:46.040 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:46.036+0000 7f00a7124640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f00a00b25a0 0x7f00a014b3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:46:46.040 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:46.036+0000 7f00a7124640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f00a00d0850 0x7f00a00ca220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:46:46.040 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:46.036+0000 7f00a7124640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f00a015bda0 0x7f00a00ca760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:46:46.040 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:46.036+0000 7f00a7124640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f00a00cce20 con 0x7f00a00d0850 2024-09-16T15:46:46.041 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:46.036+0000 7f00a7124640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f00a00ccca0 con 0x7f00a015bda0 2024-09-16T15:46:46.041 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:46.036+0000 7f00a7124640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f00a00ccfa0 con 0x7f00a00b25a0 2024-09-16T15:46:46.041 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:46.036+0000 7f009ffff640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f00a00d0850 0x7f00a00ca220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:46:46.041 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:46.036+0000 7f009ffff640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f00a00d0850 0x7f00a00ca220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:54620/0 (socket says 172.21.15.123:54620) 2024-09-16T15:46:46.041 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:46.036+0000 7f009ffff640 1 -- 172.21.15.123:0/1232290189 learned_addr learned my addr 172.21.15.123:0/1232290189 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:46:46.041 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:46.036+0000 7f009ffff640 1 -- 172.21.15.123:0/1232290189 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f00a00b25a0 msgr2=0x7f00a014b3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:46:46.041 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:46.036+0000 7f00a56c1640 1 --2- 172.21.15.123:0/1232290189 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f00a015bda0 0x7f00a00ca760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:46:46.041 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:46.036+0000 7f009ffff640 1 --2- 172.21.15.123:0/1232290189 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f00a00b25a0 0x7f00a014b3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:46.041 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:46.036+0000 7f009ffff640 1 -- 172.21.15.123:0/1232290189 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f00a015bda0 msgr2=0x7f00a00ca760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:46:46.041 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:46.036+0000 7f009ffff640 1 --2- 172.21.15.123:0/1232290189 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f00a015bda0 0x7f00a00ca760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:46.041 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:46.036+0000 7f009ffff640 1 -- 172.21.15.123:0/1232290189 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f00a00cb6b0 con 0x7f00a00d0850 2024-09-16T15:46:46.041 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:46.036+0000 7f009ffff640 1 --2- 172.21.15.123:0/1232290189 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f00a00d0850 0x7f00a00ca220 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7f008800ead0 tx=0x7f008800ef90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:46:46.041 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:46.036+0000 7f009d7fa640 1 -- 172.21.15.123:0/1232290189 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f008800ce40 con 0x7f00a00d0850 2024-09-16T15:46:46.041 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:46.036+0000 7f009d7fa640 1 -- 172.21.15.123:0/1232290189 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f0088011560 con 0x7f00a00d0850 2024-09-16T15:46:46.042 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:46.036+0000 7f00a7124640 1 -- 172.21.15.123:0/1232290189 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f00a00cae10 con 0x7f00a00d0850 2024-09-16T15:46:46.042 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:46.036+0000 7f009d7fa640 1 -- 172.21.15.123:0/1232290189 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0088011970 con 0x7f00a00d0850 2024-09-16T15:46:46.042 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:46.036+0000 7f00a7124640 1 -- 172.21.15.123:0/1232290189 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f00a00cb3d0 con 0x7f00a00d0850 2024-09-16T15:46:46.042 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:46.036+0000 7f00a7124640 1 -- 172.21.15.123:0/1232290189 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f00a00d0310 con 0x7f00a00d0850 2024-09-16T15:46:46.042 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:46.036+0000 7f009d7fa640 1 -- 172.21.15.123:0/1232290189 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 11) ==== 97386+0+0 (secure 0 0 0) 0x7f0088011b10 con 0x7f00a00d0850 2024-09-16T15:46:46.042 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:46.036+0000 7f009d7fa640 1 --2- 172.21.15.123:0/1232290189 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f0080073220 0x7f00800756e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:46:46.043 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:46.036+0000 7f009d7fa640 1 -- 172.21.15.123:0/1232290189 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(58..58 src has 1..58) ==== 5709+0+0 (secure 0 0 0) 0x7f00880952a0 con 0x7f00a00d0850 2024-09-16T15:46:46.043 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:46.036+0000 7f00a4ec0640 1 --2- 172.21.15.123:0/1232290189 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f0080073220 0x7f00800756e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:46:46.043 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:46.040+0000 7f00a4ec0640 1 --2- 172.21.15.123:0/1232290189 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f0080073220 0x7f00800756e0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f009800a890 tx=0x7f0098005b70 comp rx=0 tx=0).ready entity=mgr.4135 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:46:46.046 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:46.040+0000 7f009d7fa640 1 -- 172.21.15.123:0/1232290189 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f0088016030 con 0x7f00a00d0850 2024-09-16T15:46:46.255 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:46.252+0000 7f00a7124640 1 -- 172.21.15.123:0/1232290189 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7f00a015a450 con 0x7f00a00d0850 2024-09-16T15:46:46.256 INFO:tasks.ceph.mon.a.smithi123.stderr:2024-09-16T15:46:46.252+0000 7f268d076640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-09-16T15:46:46.259 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:46.256+0000 7f009d7fa640 1 -- 172.21.15.123:0/1232290189 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v38) ==== 161+0+0 (secure 0 0 0) 0x7f008805fe90 con 0x7f00a00d0850 2024-09-16T15:46:46.259 INFO:teuthology.orchestra.run.smithi123.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-09-16T15:46:46.261 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:46.256+0000 7f00a7124640 1 -- 172.21.15.123:0/1232290189 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f0080073220 msgr2=0x7f00800756e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:46:46.262 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:46.256+0000 7f00a7124640 1 --2- 172.21.15.123:0/1232290189 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f0080073220 0x7f00800756e0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f009800a890 tx=0x7f0098005b70 comp rx=0 tx=0).stop 2024-09-16T15:46:46.262 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:46.256+0000 7f00a7124640 1 -- 172.21.15.123:0/1232290189 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f00a00d0850 msgr2=0x7f00a00ca220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:46:46.262 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:46.256+0000 7f00a7124640 1 --2- 172.21.15.123:0/1232290189 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f00a00d0850 0x7f00a00ca220 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7f008800ead0 tx=0x7f008800ef90 comp rx=0 tx=0).stop 2024-09-16T15:46:46.262 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:46.256+0000 7f00a7124640 1 -- 172.21.15.123:0/1232290189 shutdown_connections 2024-09-16T15:46:46.262 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:46.256+0000 7f00a7124640 1 --2- 172.21.15.123:0/1232290189 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f0080073220 0x7f00800756e0 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:46.263 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:46.256+0000 7f00a7124640 1 --2- 172.21.15.123:0/1232290189 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f00a015bda0 0x7f00a00ca760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:46.263 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:46.256+0000 7f00a7124640 1 --2- 172.21.15.123:0/1232290189 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f00a00d0850 0x7f00a00ca220 unknown :-1 s=CLOSED pgs=712 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:46.263 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:46.256+0000 7f00a7124640 1 --2- 172.21.15.123:0/1232290189 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f00a00b25a0 0x7f00a014b3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:46.263 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:46.256+0000 7f00a7124640 1 -- 172.21.15.123:0/1232290189 >> 172.21.15.123:0/1232290189 conn(0x7f00a0074bd0 msgr2=0x7f00a00af600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:46:46.263 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:46.260+0000 7f00a7124640 1 -- 172.21.15.123:0/1232290189 shutdown_connections 2024-09-16T15:46:46.263 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:46.260+0000 7f00a7124640 1 -- 172.21.15.123:0/1232290189 wait complete. 2024-09-16T15:46:46.282 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-09-16T15:46:46.313 INFO:tasks.ceph.mds.c.smithi123.stdout:starting mds.c at 2024-09-16T15:46:46.418 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:46.412+0000 7f6768958640 1 -- 172.21.15.123:0/2581753451 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6764072c70 msgr2=0x7f67640cc3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:46:46.419 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:46.412+0000 7f6768958640 1 --2- 172.21.15.123:0/2581753451 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6764072c70 0x7f67640cc3c0 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7f67640b2730 tx=0x7f675801c800 comp rx=0 tx=0).stop 2024-09-16T15:46:46.419 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:46.412+0000 7f6768958640 1 -- 172.21.15.123:0/2581753451 shutdown_connections 2024-09-16T15:46:46.419 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:46.412+0000 7f6768958640 1 --2- 172.21.15.123:0/2581753451 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f6764072720 0x7f67640cc900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:46.419 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:46.412+0000 7f6768958640 1 --2- 172.21.15.123:0/2581753451 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6764072c70 0x7f67640cc3c0 unknown :-1 s=CLOSED pgs=716 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:46.419 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:46.412+0000 7f6768958640 1 --2- 172.21.15.123:0/2581753451 >> v2:172.21.15.184:3300/0 conn(0x7f676415a9d0 0x7f676406ebe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:46.419 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:46.412+0000 7f6768958640 1 -- 172.21.15.123:0/2581753451 >> 172.21.15.123:0/2581753451 conn(0x7f67640b1710 msgr2=0x7f67640b1b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:46:46.419 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:46.412+0000 7f6768958640 1 -- 172.21.15.123:0/2581753451 shutdown_connections 2024-09-16T15:46:46.419 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:46.412+0000 7f6768958640 1 -- 172.21.15.123:0/2581753451 wait complete. 2024-09-16T15:46:46.420 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:46.416+0000 7f6768958640 1 Processor -- start 2024-09-16T15:46:46.420 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:46.416+0000 7f6768958640 1 -- start start 2024-09-16T15:46:46.422 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:46.416+0000 7f6768958640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6764072720 0x7f6764157d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:46:46.422 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:46.416+0000 7f6768958640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f6764072c70 0x7f676414abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:46:46.422 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:46.416+0000 7f6768958640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f676415a9d0 0x7f67640b2f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:46:46.422 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:46.416+0000 7f6768958640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f676415f5c0 con 0x7f676415a9d0 2024-09-16T15:46:46.422 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:46.416+0000 7f6768958640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f676415f440 con 0x7f6764072720 2024-09-16T15:46:46.422 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:46.416+0000 7f6768958640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f67640ce820 con 0x7f6764072c70 2024-09-16T15:46:46.422 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:46.416+0000 7f6762d9d640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f676415a9d0 0x7f67640b2f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:46:46.422 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:46.416+0000 7f6762d9d640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f676415a9d0 0x7f67640b2f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:54670/0 (socket says 172.21.15.123:54670) 2024-09-16T15:46:46.422 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:46.416+0000 7f6762d9d640 1 -- 172.21.15.123:0/1701395362 learned_addr learned my addr 172.21.15.123:0/1701395362 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:46:46.422 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:46.416+0000 7f676259c640 1 --2- 172.21.15.123:0/1701395362 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6764072720 0x7f6764157d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:46:46.422 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:46.416+0000 7f6761d9b640 1 --2- 172.21.15.123:0/1701395362 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f6764072c70 0x7f676414abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:46:46.422 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:46.416+0000 7f6762d9d640 1 -- 172.21.15.123:0/1701395362 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f6764072c70 msgr2=0x7f676414abc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:46:46.423 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:46.416+0000 7f6762d9d640 1 --2- 172.21.15.123:0/1701395362 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f6764072c70 0x7f676414abc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:46.423 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:46.416+0000 7f6762d9d640 1 -- 172.21.15.123:0/1701395362 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6764072720 msgr2=0x7f6764157d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:46:46.423 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:46.416+0000 7f6762d9d640 1 --2- 172.21.15.123:0/1701395362 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6764072720 0x7f6764157d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:46.423 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:46.416+0000 7f6762d9d640 1 -- 172.21.15.123:0/1701395362 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f67640c89b0 con 0x7f676415a9d0 2024-09-16T15:46:46.423 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:46.416+0000 7f6761d9b640 1 --2- 172.21.15.123:0/1701395362 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f6764072c70 0x7f676414abc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T15:46:46.423 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:46.416+0000 7f6762d9d640 1 --2- 172.21.15.123:0/1701395362 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f676415a9d0 0x7f67640b2f60 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7f675c00c760 tx=0x7f675c00cc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:46:46.423 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:46.416+0000 7f6752ffd640 1 -- 172.21.15.123:0/1701395362 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f675c007b50 con 0x7f676415a9d0 2024-09-16T15:46:46.423 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:46.416+0000 7f6752ffd640 1 -- 172.21.15.123:0/1701395362 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f675c007e00 con 0x7f676415a9d0 2024-09-16T15:46:46.423 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:46.416+0000 7f6768958640 1 -- 172.21.15.123:0/1701395362 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f676414b6a0 con 0x7f676415a9d0 2024-09-16T15:46:46.423 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:46.416+0000 7f6752ffd640 1 -- 172.21.15.123:0/1701395362 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f675c00f6b0 con 0x7f676415a9d0 2024-09-16T15:46:46.423 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:46.416+0000 7f6768958640 1 -- 172.21.15.123:0/1701395362 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6764156a70 con 0x7f676415a9d0 2024-09-16T15:46:46.424 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:46.416+0000 7f6768958640 1 -- 172.21.15.123:0/1701395362 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f676406ebe0 con 0x7f676415a9d0 2024-09-16T15:46:46.424 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:46.420+0000 7f6752ffd640 1 -- 172.21.15.123:0/1701395362 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 11) ==== 97386+0+0 (secure 0 0 0) 0x7f675c00f850 con 0x7f676415a9d0 2024-09-16T15:46:46.424 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:46.420+0000 7f6752ffd640 1 --2- 172.21.15.123:0/1701395362 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f6744073220 0x7f67440756e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:46:46.424 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:46.420+0000 7f6752ffd640 1 -- 172.21.15.123:0/1701395362 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(59..59 src has 1..59) ==== 5795+0+0 (secure 0 0 0) 0x7f675c094d10 con 0x7f676415a9d0 2024-09-16T15:46:46.424 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:46.420+0000 7f676259c640 1 --2- 172.21.15.123:0/1701395362 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f6744073220 0x7f67440756e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:46:46.425 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:46.420+0000 7f676259c640 1 --2- 172.21.15.123:0/1701395362 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f6744073220 0x7f67440756e0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f6754005fd0 tx=0x7f67540077f0 comp rx=0 tx=0).ready entity=mgr.4135 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:46:46.428 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:46.424+0000 7f6752ffd640 1 -- 172.21.15.123:0/1701395362 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f675c014030 con 0x7f676415a9d0 2024-09-16T15:46:46.641 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:46.636+0000 7f6768958640 1 -- 172.21.15.123:0/1701395362 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) -- 0x7f67640ca9a0 con 0x7f676415a9d0 2024-09-16T15:46:47.262 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:47.256+0000 7f6752ffd640 1 -- 172.21.15.123:0/1701395362 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v39) ==== 104+0+0 (secure 0 0 0) 0x7f675c05f8e0 con 0x7f676415a9d0 2024-09-16T15:46:47.265 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:47.260+0000 7f6768958640 1 -- 172.21.15.123:0/1701395362 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f6744073220 msgr2=0x7f67440756e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:46:47.265 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:47.260+0000 7f6768958640 1 --2- 172.21.15.123:0/1701395362 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f6744073220 0x7f67440756e0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f6754005fd0 tx=0x7f67540077f0 comp rx=0 tx=0).stop 2024-09-16T15:46:47.265 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:47.260+0000 7f6768958640 1 -- 172.21.15.123:0/1701395362 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f676415a9d0 msgr2=0x7f67640b2f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:46:47.265 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:47.260+0000 7f6768958640 1 --2- 172.21.15.123:0/1701395362 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f676415a9d0 0x7f67640b2f60 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7f675c00c760 tx=0x7f675c00cc20 comp rx=0 tx=0).stop 2024-09-16T15:46:47.266 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:47.260+0000 7f6768958640 1 -- 172.21.15.123:0/1701395362 shutdown_connections 2024-09-16T15:46:47.266 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:47.260+0000 7f6768958640 1 --2- 172.21.15.123:0/1701395362 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f6744073220 0x7f67440756e0 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:47.266 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:47.260+0000 7f6768958640 1 --2- 172.21.15.123:0/1701395362 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f676415a9d0 0x7f67640b2f60 unknown :-1 s=CLOSED pgs=717 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:47.266 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:47.260+0000 7f6768958640 1 --2- 172.21.15.123:0/1701395362 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f6764072c70 0x7f676414abc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:47.266 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:47.260+0000 7f6768958640 1 --2- 172.21.15.123:0/1701395362 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6764072720 0x7f6764157d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:47.266 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:47.260+0000 7f6768958640 1 -- 172.21.15.123:0/1701395362 >> 172.21.15.123:0/1701395362 conn(0x7f67640b1710 msgr2=0x7f676415bb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:46:47.266 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:47.260+0000 7f6768958640 1 -- 172.21.15.123:0/1701395362 shutdown_connections 2024-09-16T15:46:47.266 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:47.260+0000 7f6768958640 1 -- 172.21.15.123:0/1701395362 wait complete. 2024-09-16T15:46:47.290 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-09-16T15:46:47.468 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:47.464+0000 7fc86030b640 1 -- 172.21.15.123:0/80080234 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc8440052a0 msgr2=0x7fc84400b9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:46:47.469 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:47.464+0000 7fc86030b640 1 --2- 172.21.15.123:0/80080234 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc8440052a0 0x7fc84400b9d0 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7fc85000c640 tx=0x7fc85001fa40 comp rx=0 tx=0).stop 2024-09-16T15:46:47.469 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:47.464+0000 7fc85e8a8640 1 -- 172.21.15.123:0/80080234 reap_dead start 2024-09-16T15:46:47.469 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:47.464+0000 7fc86030b640 1 -- 172.21.15.123:0/80080234 shutdown_connections 2024-09-16T15:46:47.469 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:47.464+0000 7fc86030b640 1 -- 172.21.15.123:0/80080234 >> 172.21.15.123:0/80080234 conn(0x7fc8580b1630 msgr2=0x7fc8580b1a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:46:47.469 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:47.464+0000 7fc86030b640 1 -- 172.21.15.123:0/80080234 shutdown_connections 2024-09-16T15:46:47.469 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:47.464+0000 7fc86030b640 1 -- 172.21.15.123:0/80080234 wait complete. 2024-09-16T15:46:47.470 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:47.464+0000 7fc86030b640 1 Processor -- start 2024-09-16T15:46:47.470 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:47.464+0000 7fc86030b640 1 -- start start 2024-09-16T15:46:47.472 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:47.464+0000 7fc86030b640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fc8440052a0 0x7fc85814d5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:46:47.472 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:47.464+0000 7fc86030b640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc8581545c0 0x7fc858154a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:46:47.472 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:47.464+0000 7fc86030b640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc85815df30 0x7fc85815e3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:46:47.472 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:47.464+0000 7fc86030b640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fc858070760 con 0x7fc8581545c0 2024-09-16T15:46:47.472 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:47.464+0000 7fc86030b640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fc8580ceff0 con 0x7fc85815df30 2024-09-16T15:46:47.472 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:47.464+0000 7fc86030b640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fc8580b2a30 con 0x7fc8440052a0 2024-09-16T15:46:47.472 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:47.464+0000 7fc85d8a6640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc8581545c0 0x7fc858154a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:46:47.473 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:47.464+0000 7fc85d8a6640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc8581545c0 0x7fc858154a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:54682/0 (socket says 172.21.15.123:54682) 2024-09-16T15:46:47.473 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:47.464+0000 7fc85d8a6640 1 -- 172.21.15.123:0/981336541 learned_addr learned my addr 172.21.15.123:0/981336541 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:46:47.473 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:47.464+0000 7fc85e8a8640 1 --2- 172.21.15.123:0/981336541 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc85815df30 0x7fc85815e3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:46:47.473 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:47.464+0000 7fc85e0a7640 1 --2- 172.21.15.123:0/981336541 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fc8440052a0 0x7fc85814d5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:46:47.473 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:47.468+0000 7fc85d8a6640 1 -- 172.21.15.123:0/981336541 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fc8440052a0 msgr2=0x7fc85814d5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:46:47.473 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:47.468+0000 7fc85d8a6640 1 --2- 172.21.15.123:0/981336541 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fc8440052a0 0x7fc85814d5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:47.473 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:47.468+0000 7fc85d8a6640 1 -- 172.21.15.123:0/981336541 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc85815df30 msgr2=0x7fc85815e3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:46:47.473 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:47.468+0000 7fc85d8a6640 1 --2- 172.21.15.123:0/981336541 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc85815df30 0x7fc85815e3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:47.473 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:47.468+0000 7fc85d8a6640 1 -- 172.21.15.123:0/981336541 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc858072b90 con 0x7fc8581545c0 2024-09-16T15:46:47.473 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:47.468+0000 7fc85e8a8640 1 --2- 172.21.15.123:0/981336541 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc85815df30 0x7fc85815e3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T15:46:47.473 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:47.468+0000 7fc85e0a7640 1 --2- 172.21.15.123:0/981336541 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fc8440052a0 0x7fc85814d5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T15:46:47.473 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:47.468+0000 7fc85d8a6640 1 --2- 172.21.15.123:0/981336541 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc8581545c0 0x7fc858154a20 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7fc84800d630 tx=0x7fc84800daf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:46:47.473 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:47.468+0000 7fc84effd640 1 -- 172.21.15.123:0/981336541 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc848017070 con 0x7fc8581545c0 2024-09-16T15:46:47.474 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:47.468+0000 7fc84effd640 1 -- 172.21.15.123:0/981336541 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fc848009a90 con 0x7fc8581545c0 2024-09-16T15:46:47.474 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:47.468+0000 7fc86030b640 1 -- 172.21.15.123:0/981336541 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc8580b2cd0 con 0x7fc8581545c0 2024-09-16T15:46:47.474 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:47.468+0000 7fc84effd640 1 -- 172.21.15.123:0/981336541 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc848010460 con 0x7fc8581545c0 2024-09-16T15:46:47.474 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:47.468+0000 7fc86030b640 1 -- 172.21.15.123:0/981336541 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc8580b3290 con 0x7fc8581545c0 2024-09-16T15:46:47.474 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:47.468+0000 7fc86030b640 1 -- 172.21.15.123:0/981336541 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc8580766e0 con 0x7fc8581545c0 2024-09-16T15:46:47.476 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:47.472+0000 7fc84effd640 1 -- 172.21.15.123:0/981336541 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 11) ==== 97386+0+0 (secure 0 0 0) 0x7fc84801a070 con 0x7fc8581545c0 2024-09-16T15:46:47.476 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:47.472+0000 7fc84effd640 1 --2- 172.21.15.123:0/981336541 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fc83c073220 0x7fc83c0756e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:46:47.476 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:47.472+0000 7fc84effd640 1 -- 172.21.15.123:0/981336541 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(59..59 src has 1..59) ==== 5795+0+0 (secure 0 0 0) 0x7fc848094ab0 con 0x7fc8581545c0 2024-09-16T15:46:47.476 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:47.472+0000 7fc85e0a7640 1 --2- 172.21.15.123:0/981336541 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fc83c073220 0x7fc83c0756e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:46:47.477 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:47.472+0000 7fc85e0a7640 1 --2- 172.21.15.123:0/981336541 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fc83c073220 0x7fc83c0756e0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7fc854001770 tx=0x7fc854006aa0 comp rx=0 tx=0).ready entity=mgr.4135 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:46:47.479 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:47.476+0000 7fc84effd640 1 -- 172.21.15.123:0/981336541 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7fc84805f6d0 con 0x7fc8581545c0 2024-09-16T15:46:47.656 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:47.652+0000 7fc86030b640 1 -- 172.21.15.123:0/981336541 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) -- 0x7fc858074220 con 0x7fc8581545c0 2024-09-16T15:46:48.475 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:48.468+0000 7fc84effd640 1 -- 172.21.15.123:0/981336541 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v60) ==== 181+0+0 (secure 0 0 0) 0x7fc84805efd0 con 0x7fc8581545c0 2024-09-16T15:46:48.475 INFO:teuthology.orchestra.run.smithi123.stderr:pool 'cephfs_metadata' removed 2024-09-16T15:46:48.481 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:48.476+0000 7fc86030b640 1 -- 172.21.15.123:0/981336541 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fc83c073220 msgr2=0x7fc83c0756e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:46:48.481 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:48.476+0000 7fc86030b640 1 --2- 172.21.15.123:0/981336541 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fc83c073220 0x7fc83c0756e0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7fc854001770 tx=0x7fc854006aa0 comp rx=0 tx=0).stop 2024-09-16T15:46:48.481 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:48.476+0000 7fc86030b640 1 -- 172.21.15.123:0/981336541 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc8581545c0 msgr2=0x7fc858154a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:46:48.481 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:48.476+0000 7fc86030b640 1 --2- 172.21.15.123:0/981336541 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc8581545c0 0x7fc858154a20 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7fc84800d630 tx=0x7fc84800daf0 comp rx=0 tx=0).stop 2024-09-16T15:46:48.482 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:48.476+0000 7fc86030b640 1 -- 172.21.15.123:0/981336541 shutdown_connections 2024-09-16T15:46:48.482 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:48.476+0000 7fc86030b640 1 --2- 172.21.15.123:0/981336541 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fc83c073220 0x7fc83c0756e0 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:48.482 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:48.476+0000 7fc86030b640 1 --2- 172.21.15.123:0/981336541 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc85815df30 0x7fc85815e3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:48.482 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:48.476+0000 7fc86030b640 1 --2- 172.21.15.123:0/981336541 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc8581545c0 0x7fc858154a20 unknown :-1 s=CLOSED pgs=720 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:48.482 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:48.476+0000 7fc86030b640 1 --2- 172.21.15.123:0/981336541 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fc8440052a0 0x7fc85814d5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:48.482 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:48.476+0000 7fc86030b640 1 -- 172.21.15.123:0/981336541 >> 172.21.15.123:0/981336541 conn(0x7fc8580b1630 msgr2=0x7fc858158110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:46:48.482 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:48.476+0000 7fc86030b640 1 -- 172.21.15.123:0/981336541 shutdown_connections 2024-09-16T15:46:48.482 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:48.476+0000 7fc86030b640 1 -- 172.21.15.123:0/981336541 wait complete. 2024-09-16T15:46:48.505 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-09-16T15:46:48.642 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:48.636+0000 7f8b690ed640 1 -- 172.21.15.123:0/3698475723 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8b4c003ab0 msgr2=0x7f8b4c003ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:46:48.642 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:48.636+0000 7f8b690ed640 1 --2- 172.21.15.123:0/3698475723 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8b4c003ab0 0x7f8b4c003ed0 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7f8b5400bb10 tx=0x7f8b5401cd70 comp rx=0 tx=0).stop 2024-09-16T15:46:48.642 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:48.636+0000 7f8b690ed640 1 -- 172.21.15.123:0/3698475723 shutdown_connections 2024-09-16T15:46:48.642 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:48.636+0000 7f8b690ed640 1 --2- 172.21.15.123:0/3698475723 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8b4c003ab0 0x7f8b4c003ed0 unknown :-1 s=CLOSED pgs=722 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:48.643 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:48.636+0000 7f8b690ed640 1 -- 172.21.15.123:0/3698475723 >> 172.21.15.123:0/3698475723 conn(0x7f8b640d3e30 msgr2=0x7f8b640d4240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:46:48.643 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:48.636+0000 7f8b690ed640 1 -- 172.21.15.123:0/3698475723 shutdown_connections 2024-09-16T15:46:48.643 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:48.636+0000 7f8b690ed640 1 -- 172.21.15.123:0/3698475723 wait complete. 2024-09-16T15:46:48.644 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:48.640+0000 7f8b690ed640 1 Processor -- start 2024-09-16T15:46:48.644 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:48.640+0000 7f8b690ed640 1 -- start start 2024-09-16T15:46:48.646 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:48.640+0000 7f8b690ed640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8b4c003ab0 0x7f8b641f7bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:46:48.646 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:48.640+0000 7f8b690ed640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f8b64159260 0x7f8b64157a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:46:48.646 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:48.640+0000 7f8b690ed640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f8b64157f60 0x7f8b641586c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:46:48.647 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:48.640+0000 7f8b690ed640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f8b640d25a0 con 0x7f8b4c003ab0 2024-09-16T15:46:48.647 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:48.640+0000 7f8b690ed640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f8b640d2420 con 0x7f8b64157f60 2024-09-16T15:46:48.647 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:48.640+0000 7f8b690ed640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f8b640d2720 con 0x7f8b64159260 2024-09-16T15:46:48.647 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:48.640+0000 7f8b62d9d640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8b4c003ab0 0x7f8b641f7bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:46:48.647 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:48.640+0000 7f8b62d9d640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8b4c003ab0 0x7f8b641f7bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:54696/0 (socket says 172.21.15.123:54696) 2024-09-16T15:46:48.647 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:48.640+0000 7f8b62d9d640 1 -- 172.21.15.123:0/88277687 learned_addr learned my addr 172.21.15.123:0/88277687 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:46:48.647 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:48.640+0000 7f8b6359e640 1 --2- 172.21.15.123:0/88277687 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f8b64157f60 0x7f8b641586c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:46:48.647 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:48.640+0000 7f8b6259c640 1 --2- 172.21.15.123:0/88277687 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f8b64159260 0x7f8b64157a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:46:48.647 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:48.640+0000 7f8b62d9d640 1 -- 172.21.15.123:0/88277687 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f8b64159260 msgr2=0x7f8b64157a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:46:48.647 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:48.640+0000 7f8b62d9d640 1 --2- 172.21.15.123:0/88277687 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f8b64159260 0x7f8b64157a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:48.647 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:48.640+0000 7f8b62d9d640 1 -- 172.21.15.123:0/88277687 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f8b64157f60 msgr2=0x7f8b641586c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:46:48.647 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:48.640+0000 7f8b62d9d640 1 --2- 172.21.15.123:0/88277687 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f8b64157f60 0x7f8b641586c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:48.648 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:48.640+0000 7f8b62d9d640 1 -- 172.21.15.123:0/88277687 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8b64159ca0 con 0x7f8b4c003ab0 2024-09-16T15:46:48.648 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:48.640+0000 7f8b6259c640 1 --2- 172.21.15.123:0/88277687 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f8b64159260 0x7f8b64157a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T15:46:48.648 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:48.640+0000 7f8b6359e640 1 --2- 172.21.15.123:0/88277687 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f8b64157f60 0x7f8b641586c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T15:46:48.648 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:48.640+0000 7f8b62d9d640 1 --2- 172.21.15.123:0/88277687 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8b4c003ab0 0x7f8b641f7bc0 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7f8b5400ba60 tx=0x7f8b54004620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:46:48.648 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:48.640+0000 7f8b4b7fe640 1 -- 172.21.15.123:0/88277687 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b540be070 con 0x7f8b4c003ab0 2024-09-16T15:46:48.648 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:48.640+0000 7f8b4b7fe640 1 -- 172.21.15.123:0/88277687 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f8b54005330 con 0x7f8b4c003ab0 2024-09-16T15:46:48.648 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:48.640+0000 7f8b690ed640 1 -- 172.21.15.123:0/88277687 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8b64159f80 con 0x7f8b4c003ab0 2024-09-16T15:46:48.648 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:48.640+0000 7f8b4b7fe640 1 -- 172.21.15.123:0/88277687 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b540ab050 con 0x7f8b4c003ab0 2024-09-16T15:46:48.648 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:48.640+0000 7f8b690ed640 1 -- 172.21.15.123:0/88277687 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8b6415a540 con 0x7f8b4c003ab0 2024-09-16T15:46:48.649 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:48.644+0000 7f8b690ed640 1 -- 172.21.15.123:0/88277687 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8b640b3e40 con 0x7f8b4c003ab0 2024-09-16T15:46:48.649 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:48.644+0000 7f8b4b7fe640 1 -- 172.21.15.123:0/88277687 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 11) ==== 97386+0+0 (secure 0 0 0) 0x7f8b540c1070 con 0x7f8b4c003ab0 2024-09-16T15:46:48.649 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:48.644+0000 7f8b4b7fe640 1 --2- 172.21.15.123:0/88277687 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f8b50073250 0x7f8b50075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:46:48.649 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:48.644+0000 7f8b4b7fe640 1 -- 172.21.15.123:0/88277687 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(60..60 src has 1..60) ==== 5357+0+0 (secure 0 0 0) 0x7f8b5412e4d0 con 0x7f8b4c003ab0 2024-09-16T15:46:48.649 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:48.644+0000 7f8b6259c640 1 --2- 172.21.15.123:0/88277687 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f8b50073250 0x7f8b50075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:46:48.650 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:48.644+0000 7f8b6259c640 1 --2- 172.21.15.123:0/88277687 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f8b50073250 0x7f8b50075710 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f8b58001ec0 tx=0x7f8b58000f60 comp rx=0 tx=0).ready entity=mgr.4135 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:46:48.652 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:48.648+0000 7f8b4b7fe640 1 -- 172.21.15.123:0/88277687 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f8b540a5e20 con 0x7f8b4c003ab0 2024-09-16T15:46:48.823 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:48.820+0000 7f8b690ed640 1 -- 172.21.15.123:0/88277687 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) -- 0x7f8b640b4580 con 0x7f8b4c003ab0 2024-09-16T15:46:49.497 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:49.492+0000 7f8b4b7fe640 1 -- 172.21.15.123:0/88277687 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v61) ==== 169+0+0 (secure 0 0 0) 0x7f8b54016ab0 con 0x7f8b4c003ab0 2024-09-16T15:46:49.497 INFO:teuthology.orchestra.run.smithi123.stderr:pool 'cephfs_data' removed 2024-09-16T15:46:49.501 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:49.496+0000 7f8b690ed640 1 -- 172.21.15.123:0/88277687 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f8b50073250 msgr2=0x7f8b50075710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:46:49.501 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:49.496+0000 7f8b690ed640 1 --2- 172.21.15.123:0/88277687 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f8b50073250 0x7f8b50075710 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f8b58001ec0 tx=0x7f8b58000f60 comp rx=0 tx=0).stop 2024-09-16T15:46:49.501 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:49.496+0000 7f8b690ed640 1 -- 172.21.15.123:0/88277687 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8b4c003ab0 msgr2=0x7f8b641f7bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:46:49.501 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:49.496+0000 7f8b690ed640 1 --2- 172.21.15.123:0/88277687 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8b4c003ab0 0x7f8b641f7bc0 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7f8b5400ba60 tx=0x7f8b54004620 comp rx=0 tx=0).stop 2024-09-16T15:46:49.501 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:49.496+0000 7f8b690ed640 1 -- 172.21.15.123:0/88277687 shutdown_connections 2024-09-16T15:46:49.501 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:49.496+0000 7f8b690ed640 1 --2- 172.21.15.123:0/88277687 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f8b50073250 0x7f8b50075710 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:49.501 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:49.496+0000 7f8b690ed640 1 --2- 172.21.15.123:0/88277687 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f8b64157f60 0x7f8b641586c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:49.501 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:49.496+0000 7f8b690ed640 1 --2- 172.21.15.123:0/88277687 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f8b64159260 0x7f8b64157a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:49.502 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:49.496+0000 7f8b690ed640 1 --2- 172.21.15.123:0/88277687 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8b4c003ab0 0x7f8b641f7bc0 unknown :-1 s=CLOSED pgs=723 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:49.502 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:49.496+0000 7f8b690ed640 1 -- 172.21.15.123:0/88277687 >> 172.21.15.123:0/88277687 conn(0x7f8b640d3e30 msgr2=0x7f8b640b4d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:46:49.502 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:49.496+0000 7f8b690ed640 1 -- 172.21.15.123:0/88277687 shutdown_connections 2024-09-16T15:46:49.502 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:49.496+0000 7f8b690ed640 1 -- 172.21.15.123:0/88277687 wait complete. 2024-09-16T15:46:49.528 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config reset 27 2024-09-16T15:46:49.659 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:49.656+0000 7fe772791640 1 -- 172.21.15.123:0/3208586811 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe76c158120 msgr2=0x7fe76c0ca320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:46:49.659 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:49.656+0000 7fe772791640 1 --2- 172.21.15.123:0/3208586811 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe76c158120 0x7fe76c0ca320 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7fe754009840 tx=0x7fe75401c790 comp rx=0 tx=0).stop 2024-09-16T15:46:49.659 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:49.656+0000 7fe772791640 1 -- 172.21.15.123:0/3208586811 shutdown_connections 2024-09-16T15:46:49.659 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:49.656+0000 7fe772791640 1 --2- 172.21.15.123:0/3208586811 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe76c158120 0x7fe76c0ca320 unknown :-1 s=CLOSED pgs=725 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:49.659 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:49.656+0000 7fe772791640 1 -- 172.21.15.123:0/3208586811 >> 172.21.15.123:0/3208586811 conn(0x7fe76c0cfb10 msgr2=0x7fe76c0cff20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:46:49.659 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:49.656+0000 7fe772791640 1 -- 172.21.15.123:0/3208586811 shutdown_connections 2024-09-16T15:46:49.659 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:49.656+0000 7fe772791640 1 -- 172.21.15.123:0/3208586811 wait complete. 2024-09-16T15:46:49.660 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:49.656+0000 7fe772791640 1 Processor -- start 2024-09-16T15:46:49.660 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:49.656+0000 7fe772791640 1 -- start start 2024-09-16T15:46:49.662 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:49.656+0000 7fe772791640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fe76c06f5f0 0x7fe76c154ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:46:49.662 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:49.656+0000 7fe772791640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe76c0cf5a0 0x7fe76c1fc0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:46:49.662 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:49.656+0000 7fe772791640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe76c158120 0x7fe76c0b47a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:46:49.662 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:49.656+0000 7fe772791640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fe76c1683e0 con 0x7fe76c0cf5a0 2024-09-16T15:46:49.662 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:49.656+0000 7fe772791640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fe76c15b830 con 0x7fe76c158120 2024-09-16T15:46:49.663 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:49.656+0000 7fe772791640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fe76c168560 con 0x7fe76c06f5f0 2024-09-16T15:46:49.663 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:49.656+0000 7fe76b7fe640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe76c0cf5a0 0x7fe76c1fc0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:46:49.663 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:49.656+0000 7fe76b7fe640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe76c0cf5a0 0x7fe76c1fc0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:54726/0 (socket says 172.21.15.123:54726) 2024-09-16T15:46:49.663 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:49.656+0000 7fe76b7fe640 1 -- 172.21.15.123:0/1155162968 learned_addr learned my addr 172.21.15.123:0/1155162968 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:46:49.663 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:49.656+0000 7fe770d2e640 1 --2- 172.21.15.123:0/1155162968 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe76c158120 0x7fe76c0b47a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:46:49.663 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:49.656+0000 7fe76b7fe640 1 -- 172.21.15.123:0/1155162968 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fe76c06f5f0 msgr2=0x7fe76c154ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:46:49.663 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:49.656+0000 7fe76bfff640 1 --2- 172.21.15.123:0/1155162968 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fe76c06f5f0 0x7fe76c154ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:46:49.663 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:49.656+0000 7fe76b7fe640 1 --2- 172.21.15.123:0/1155162968 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fe76c06f5f0 0x7fe76c154ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:49.663 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:49.656+0000 7fe76b7fe640 1 -- 172.21.15.123:0/1155162968 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe76c158120 msgr2=0x7fe76c0b47a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:46:49.663 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:49.656+0000 7fe76b7fe640 1 --2- 172.21.15.123:0/1155162968 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe76c158120 0x7fe76c0b47a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:49.663 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:49.656+0000 7fe76b7fe640 1 -- 172.21.15.123:0/1155162968 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe76c0b3d80 con 0x7fe76c0cf5a0 2024-09-16T15:46:49.663 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:49.656+0000 7fe770d2e640 1 --2- 172.21.15.123:0/1155162968 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe76c158120 0x7fe76c0b47a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T15:46:49.663 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:49.656+0000 7fe76b7fe640 1 --2- 172.21.15.123:0/1155162968 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe76c0cf5a0 0x7fe76c1fc0c0 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7fe75c00cbe0 tx=0x7fe75c00ccd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:46:49.663 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:49.656+0000 7fe768ff9640 1 -- 172.21.15.123:0/1155162968 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe75c016020 con 0x7fe76c0cf5a0 2024-09-16T15:46:49.664 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:49.656+0000 7fe768ff9640 1 -- 172.21.15.123:0/1155162968 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fe75c00ab60 con 0x7fe76c0cf5a0 2024-09-16T15:46:49.664 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:49.656+0000 7fe772791640 1 -- 172.21.15.123:0/1155162968 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe76c0b45b0 con 0x7fe76c0cf5a0 2024-09-16T15:46:49.664 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:49.656+0000 7fe768ff9640 1 -- 172.21.15.123:0/1155162968 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe75c012420 con 0x7fe76c0cf5a0 2024-09-16T15:46:49.664 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:49.656+0000 7fe772791640 1 -- 172.21.15.123:0/1155162968 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe76c0b3ff0 con 0x7fe76c0cf5a0 2024-09-16T15:46:49.664 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:49.660+0000 7fe772791640 1 -- 172.21.15.123:0/1155162968 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe76c0ca320 con 0x7fe76c0cf5a0 2024-09-16T15:46:49.665 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:49.660+0000 7fe768ff9640 1 -- 172.21.15.123:0/1155162968 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 11) ==== 97386+0+0 (secure 0 0 0) 0x7fe75c019020 con 0x7fe76c0cf5a0 2024-09-16T15:46:49.665 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:49.660+0000 7fe768ff9640 1 --2- 172.21.15.123:0/1155162968 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fe74c0731c0 0x7fe74c075680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:46:49.665 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:49.660+0000 7fe768ff9640 1 -- 172.21.15.123:0/1155162968 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(61..61 src has 1..61) ==== 4943+0+0 (secure 0 0 0) 0x7fe75c0942d0 con 0x7fe76c0cf5a0 2024-09-16T15:46:49.665 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:49.660+0000 7fe76bfff640 1 --2- 172.21.15.123:0/1155162968 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fe74c0731c0 0x7fe74c075680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:46:49.665 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:49.660+0000 7fe76bfff640 1 --2- 172.21.15.123:0/1155162968 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fe74c0731c0 0x7fe74c075680 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7fe7540af000 tx=0x7fe75401afd0 comp rx=0 tx=0).ready entity=mgr.4135 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:46:49.668 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:49.664+0000 7fe768ff9640 1 -- 172.21.15.123:0/1155162968 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7fe75c020020 con 0x7fe76c0cf5a0 2024-09-16T15:46:49.842 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:49.836+0000 7fe772791640 1 -- 172.21.15.123:0/1155162968 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "config reset", "num": 27} v 0) -- 0x7fe76c0d06c0 con 0x7fe76c0cf5a0 2024-09-16T15:46:49.848 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:49.844+0000 7fe768ff9640 1 -- 172.21.15.123:0/1155162968 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "config reset", "num": 27}]=0 v33) ==== 71+0+0 (secure 0 0 0) 0x7fe75c05f470 con 0x7fe76c0cf5a0 2024-09-16T15:46:49.851 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:49.848+0000 7fe772791640 1 -- 172.21.15.123:0/1155162968 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fe74c0731c0 msgr2=0x7fe74c075680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:46:49.851 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:49.848+0000 7fe772791640 1 --2- 172.21.15.123:0/1155162968 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fe74c0731c0 0x7fe74c075680 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7fe7540af000 tx=0x7fe75401afd0 comp rx=0 tx=0).stop 2024-09-16T15:46:49.851 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:49.848+0000 7fe772791640 1 -- 172.21.15.123:0/1155162968 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe76c0cf5a0 msgr2=0x7fe76c1fc0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:46:49.852 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:49.848+0000 7fe772791640 1 --2- 172.21.15.123:0/1155162968 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe76c0cf5a0 0x7fe76c1fc0c0 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7fe75c00cbe0 tx=0x7fe75c00ccd0 comp rx=0 tx=0).stop 2024-09-16T15:46:49.852 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:49.848+0000 7fe772791640 1 -- 172.21.15.123:0/1155162968 shutdown_connections 2024-09-16T15:46:49.852 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:49.848+0000 7fe772791640 1 --2- 172.21.15.123:0/1155162968 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fe74c0731c0 0x7fe74c075680 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:49.852 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:49.848+0000 7fe772791640 1 --2- 172.21.15.123:0/1155162968 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe76c158120 0x7fe76c0b47a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:49.852 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:49.848+0000 7fe772791640 1 --2- 172.21.15.123:0/1155162968 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe76c0cf5a0 0x7fe76c1fc0c0 unknown :-1 s=CLOSED pgs=726 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:49.852 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:49.848+0000 7fe772791640 1 --2- 172.21.15.123:0/1155162968 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fe76c06f5f0 0x7fe76c154ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:49.852 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:49.848+0000 7fe772791640 1 -- 172.21.15.123:0/1155162968 >> 172.21.15.123:0/1155162968 conn(0x7fe76c0cfb10 msgr2=0x7fe76c156e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:46:49.852 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:49.848+0000 7fe772791640 1 -- 172.21.15.123:0/1155162968 shutdown_connections 2024-09-16T15:46:49.852 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:49.848+0000 7fe772791640 1 -- 172.21.15.123:0/1155162968 wait complete. 2024-09-16T15:46:49.868 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_volumes.TestMisc.test_malicious_metafile_on_legacy_to_v1_upgrade' 2024-09-16T15:46:50.005 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:50.000+0000 7fc313497640 1 -- 172.21.15.123:0/3116434840 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc30c0b3100 msgr2=0x7fc30c0cc1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:46:50.005 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:50.000+0000 7fc313497640 1 --2- 172.21.15.123:0/3116434840 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc30c0b3100 0x7fc30c0cc1b0 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 crypto rx=0x7fc30c051540 tx=0x7fc2fc01c8c0 comp rx=0 tx=0).stop 2024-09-16T15:46:50.005 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:50.000+0000 7fc313497640 1 -- 172.21.15.123:0/3116434840 shutdown_connections 2024-09-16T15:46:50.005 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:50.000+0000 7fc313497640 1 --2- 172.21.15.123:0/3116434840 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fc30c070830 0x7fc30c0cc6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:50.005 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:50.000+0000 7fc313497640 1 --2- 172.21.15.123:0/3116434840 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc30c0b3100 0x7fc30c0cc1b0 unknown :-1 s=CLOSED pgs=727 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:50.005 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:50.000+0000 7fc313497640 1 --2- 172.21.15.123:0/3116434840 >> v2:172.21.15.184:3300/0 conn(0x7fc30c15ac00 0x7fc30c0702f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:50.005 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:50.000+0000 7fc313497640 1 -- 172.21.15.123:0/3116434840 >> 172.21.15.123:0/3116434840 conn(0x7fc30c0b2610 msgr2=0x7fc30c0b29f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:46:50.006 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:50.000+0000 7fc313497640 1 -- 172.21.15.123:0/3116434840 shutdown_connections 2024-09-16T15:46:50.006 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:50.000+0000 7fc313497640 1 -- 172.21.15.123:0/3116434840 wait complete. 2024-09-16T15:46:50.007 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:50.000+0000 7fc313497640 1 Processor -- start 2024-09-16T15:46:50.007 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:50.000+0000 7fc313497640 1 -- start start 2024-09-16T15:46:50.008 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:50.000+0000 7fc313497640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc30c070830 0x7fc30c165390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:46:50.008 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:50.000+0000 7fc313497640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc30c0b3100 0x7fc30c0ca020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:46:50.008 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:50.004+0000 7fc313497640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fc30c15ac00 0x7fc30c148dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:46:50.008 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:50.004+0000 7fc313497640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fc30c0cb5b0 con 0x7fc30c0b3100 2024-09-16T15:46:50.008 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:50.004+0000 7fc313497640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fc30c0cb430 con 0x7fc30c070830 2024-09-16T15:46:50.008 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:50.004+0000 7fc313497640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fc30c0cb730 con 0x7fc30c15ac00 2024-09-16T15:46:50.008 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:50.004+0000 7fc310a32640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc30c0b3100 0x7fc30c0ca020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:46:50.008 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:50.004+0000 7fc310a32640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc30c0b3100 0x7fc30c0ca020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:54734/0 (socket says 172.21.15.123:54734) 2024-09-16T15:46:50.008 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:50.004+0000 7fc310a32640 1 -- 172.21.15.123:0/3912658523 learned_addr learned my addr 172.21.15.123:0/3912658523 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:46:50.008 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:50.004+0000 7fc311233640 1 --2- 172.21.15.123:0/3912658523 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc30c070830 0x7fc30c165390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:46:50.008 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:50.004+0000 7fc310a32640 1 -- 172.21.15.123:0/3912658523 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fc30c15ac00 msgr2=0x7fc30c148dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:46:50.008 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:50.004+0000 7fc311a34640 1 --2- 172.21.15.123:0/3912658523 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fc30c15ac00 0x7fc30c148dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:46:50.008 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:50.004+0000 7fc310a32640 1 --2- 172.21.15.123:0/3912658523 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fc30c15ac00 0x7fc30c148dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:50.009 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:50.004+0000 7fc310a32640 1 -- 172.21.15.123:0/3912658523 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc30c070830 msgr2=0x7fc30c165390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:46:50.009 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:50.004+0000 7fc310a32640 1 --2- 172.21.15.123:0/3912658523 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc30c070830 0x7fc30c165390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:50.009 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:50.004+0000 7fc310a32640 1 -- 172.21.15.123:0/3912658523 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc30c1658d0 con 0x7fc30c0b3100 2024-09-16T15:46:50.009 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:50.004+0000 7fc310a32640 1 --2- 172.21.15.123:0/3912658523 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc30c0b3100 0x7fc30c0ca020 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7fc2fc01af70 tx=0x7fc2fc0ac4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:46:50.009 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:50.004+0000 7fc301ffb640 1 -- 172.21.15.123:0/3912658523 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2fc0b1070 con 0x7fc30c0b3100 2024-09-16T15:46:50.009 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:50.004+0000 7fc301ffb640 1 -- 172.21.15.123:0/3912658523 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fc2fc0b2040 con 0x7fc30c0b3100 2024-09-16T15:46:50.009 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:50.004+0000 7fc301ffb640 1 -- 172.21.15.123:0/3912658523 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2fc0acbe0 con 0x7fc30c0b3100 2024-09-16T15:46:50.010 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:50.004+0000 7fc313497640 1 -- 172.21.15.123:0/3912658523 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc30c160b20 con 0x7fc30c0b3100 2024-09-16T15:46:50.010 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:50.004+0000 7fc313497640 1 -- 172.21.15.123:0/3912658523 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc30c1610e0 con 0x7fc30c0b3100 2024-09-16T15:46:50.010 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:50.004+0000 7fc313497640 1 -- 172.21.15.123:0/3912658523 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc30c0702f0 con 0x7fc30c0b3100 2024-09-16T15:46:50.010 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:50.004+0000 7fc301ffb640 1 -- 172.21.15.123:0/3912658523 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 11) ==== 97386+0+0 (secure 0 0 0) 0x7fc2fc0bd030 con 0x7fc30c0b3100 2024-09-16T15:46:50.010 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:50.004+0000 7fc301ffb640 1 --2- 172.21.15.123:0/3912658523 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fc2f0073250 0x7fc2f0075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:46:50.011 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:50.004+0000 7fc301ffb640 1 -- 172.21.15.123:0/3912658523 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(61..61 src has 1..61) ==== 4943+0+0 (secure 0 0 0) 0x7fc2fc12d850 con 0x7fc30c0b3100 2024-09-16T15:46:50.011 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:50.004+0000 7fc311233640 1 --2- 172.21.15.123:0/3912658523 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fc2f0073250 0x7fc2f0075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:46:50.011 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:50.008+0000 7fc311233640 1 --2- 172.21.15.123:0/3912658523 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fc2f0073250 0x7fc2f0075710 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7fc308005fd0 tx=0x7fc308005d10 comp rx=0 tx=0).ready entity=mgr.4135 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:46:50.014 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:50.008+0000 7fc301ffb640 1 -- 172.21.15.123:0/3912658523 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7fc30c1658d0 con 0x7fc30c0b3100 2024-09-16T15:46:50.239 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:50.236+0000 7fc313497640 1 -- 172.21.15.123:0/3912658523 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_volumes.TestMisc.test_malicious_metafile_on_legacy_to_v1_upgrade"]} v 0) -- 0x7fc30c0b6aa0 con 0x7fc30c0b3100 2024-09-16T15:46:50.500 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:50.496+0000 7fc301ffb640 1 -- 172.21.15.123:0/3912658523 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_volumes.TestMisc.test_malicious_metafile_on_legacy_to_v1_upgrade"]}]=0 v207) ==== 161+0+0 (secure 0 0 0) 0x7fc2fc0164b0 con 0x7fc30c0b3100 2024-09-16T15:46:50.504 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:50.500+0000 7fc313497640 1 -- 172.21.15.123:0/3912658523 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fc2f0073250 msgr2=0x7fc2f0075710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:46:50.504 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:50.500+0000 7fc313497640 1 --2- 172.21.15.123:0/3912658523 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fc2f0073250 0x7fc2f0075710 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7fc308005fd0 tx=0x7fc308005d10 comp rx=0 tx=0).stop 2024-09-16T15:46:50.504 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:50.500+0000 7fc313497640 1 -- 172.21.15.123:0/3912658523 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc30c0b3100 msgr2=0x7fc30c0ca020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:46:50.504 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:50.500+0000 7fc313497640 1 --2- 172.21.15.123:0/3912658523 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc30c0b3100 0x7fc30c0ca020 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7fc2fc01af70 tx=0x7fc2fc0ac4a0 comp rx=0 tx=0).stop 2024-09-16T15:46:50.504 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:50.500+0000 7fc313497640 1 -- 172.21.15.123:0/3912658523 shutdown_connections 2024-09-16T15:46:50.505 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:50.500+0000 7fc313497640 1 --2- 172.21.15.123:0/3912658523 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fc2f0073250 0x7fc2f0075710 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:50.505 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:50.500+0000 7fc313497640 1 --2- 172.21.15.123:0/3912658523 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fc30c15ac00 0x7fc30c148dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:50.505 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:50.500+0000 7fc313497640 1 --2- 172.21.15.123:0/3912658523 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc30c0b3100 0x7fc30c0ca020 unknown :-1 s=CLOSED pgs=728 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:50.505 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:50.500+0000 7fc313497640 1 --2- 172.21.15.123:0/3912658523 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc30c070830 0x7fc30c165390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:50.505 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:50.500+0000 7fc313497640 1 -- 172.21.15.123:0/3912658523 >> 172.21.15.123:0/3912658523 conn(0x7fc30c0b2610 msgr2=0x7fc30c15d4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:46:50.505 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:50.500+0000 7fc313497640 1 -- 172.21.15.123:0/3912658523 shutdown_connections 2024-09-16T15:46:50.505 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:50.500+0000 7fc313497640 1 -- 172.21.15.123:0/3912658523 wait complete. 2024-09-16T15:46:50.525 INFO:tasks.cephfs_test_runner:Validate handcrafted .meta file on legacy subvol root doesn't break the system ... ok 2024-09-16T15:46:50.525 INFO:tasks.cephfs_test_runner:Starting test: test_mgr_eviction (tasks.cephfs.test_volumes.TestMisc) 2024-09-16T15:46:50.525 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-16T15:46:50.653 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:50.648+0000 7f5581cd0640 1 -- 172.21.15.123:0/3149980123 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f557c074470 msgr2=0x7f557c0cd6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:46:50.653 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:50.648+0000 7f5581cd0640 1 --2- 172.21.15.123:0/3149980123 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f557c074470 0x7f557c0cd6e0 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7f557c070730 tx=0x7f557001c7a0 comp rx=0 tx=0).stop 2024-09-16T15:46:50.653 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:50.648+0000 7f5581cd0640 1 -- 172.21.15.123:0/3149980123 shutdown_connections 2024-09-16T15:46:50.653 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:50.648+0000 7f5581cd0640 1 --2- 172.21.15.123:0/3149980123 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f557c074470 0x7f557c0cd6e0 unknown :-1 s=CLOSED pgs=729 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:50.653 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:50.648+0000 7f5581cd0640 1 --2- 172.21.15.123:0/3149980123 >> v2:172.21.15.184:3300/0 conn(0x7f557c15e160 0x7f557c073f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:50.653 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:50.648+0000 7f5581cd0640 1 -- 172.21.15.123:0/3149980123 >> 172.21.15.123:0/3149980123 conn(0x7f557c0cfb10 msgr2=0x7f557c0cff20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:46:50.654 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:50.648+0000 7f5581cd0640 1 -- 172.21.15.123:0/3149980123 shutdown_connections 2024-09-16T15:46:50.654 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:50.648+0000 7f5581cd0640 1 -- 172.21.15.123:0/3149980123 wait complete. 2024-09-16T15:46:50.654 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:50.648+0000 7f5581cd0640 1 Processor -- start 2024-09-16T15:46:50.654 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:50.648+0000 7f5581cd0640 1 -- start start 2024-09-16T15:46:50.656 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:50.652+0000 7f5581cd0640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f557c074470 0x7f557c157e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:46:50.656 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:50.652+0000 7f5581cd0640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f557c0cdc20 0x7f557c0c8f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:46:50.656 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:50.652+0000 7f5581cd0640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f557c15e160 0x7f557c0c9470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:46:50.656 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:50.652+0000 7f5581cd0640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f557c167d40 con 0x7f557c074470 2024-09-16T15:46:50.656 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:50.652+0000 7f5581cd0640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f557c167bc0 con 0x7f557c15e160 2024-09-16T15:46:50.657 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:50.652+0000 7f5581cd0640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f557c167ec0 con 0x7f557c0cdc20 2024-09-16T15:46:50.657 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:50.652+0000 7f557b7fe640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f557c074470 0x7f557c157e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:46:50.657 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:50.652+0000 7f557b7fe640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f557c074470 0x7f557c157e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:54744/0 (socket says 172.21.15.123:54744) 2024-09-16T15:46:50.657 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:50.652+0000 7f557b7fe640 1 -- 172.21.15.123:0/2613661427 learned_addr learned my addr 172.21.15.123:0/2613661427 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:46:50.657 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:50.652+0000 7f557bfff640 1 --2- 172.21.15.123:0/2613661427 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f557c15e160 0x7f557c0c9470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:46:50.657 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:50.652+0000 7f557affd640 1 --2- 172.21.15.123:0/2613661427 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f557c0cdc20 0x7f557c0c8f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:46:50.657 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:50.652+0000 7f557b7fe640 1 -- 172.21.15.123:0/2613661427 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f557c0cdc20 msgr2=0x7f557c0c8f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:46:50.657 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:50.652+0000 7f557b7fe640 1 --2- 172.21.15.123:0/2613661427 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f557c0cdc20 0x7f557c0c8f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:50.657 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:50.652+0000 7f557b7fe640 1 -- 172.21.15.123:0/2613661427 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f557c15e160 msgr2=0x7f557c0c9470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:46:50.657 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:50.652+0000 7f557b7fe640 1 --2- 172.21.15.123:0/2613661427 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f557c15e160 0x7f557c0c9470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:50.658 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:50.652+0000 7f557b7fe640 1 -- 172.21.15.123:0/2613661427 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f557c06fdd0 con 0x7f557c074470 2024-09-16T15:46:50.658 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:50.652+0000 7f557affd640 1 --2- 172.21.15.123:0/2613661427 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f557c0cdc20 0x7f557c0c8f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T15:46:50.658 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:50.652+0000 7f557b7fe640 1 --2- 172.21.15.123:0/2613661427 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f557c074470 0x7f557c157e60 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 crypto rx=0x7f556c00d780 tx=0x7f556c00dc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:46:50.658 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:50.652+0000 7f5563fff640 1 -- 172.21.15.123:0/2613661427 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f556c017070 con 0x7f557c074470 2024-09-16T15:46:50.658 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:50.652+0000 7f5563fff640 1 -- 172.21.15.123:0/2613661427 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f556c00de90 con 0x7f557c074470 2024-09-16T15:46:50.658 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:50.652+0000 7f5563fff640 1 -- 172.21.15.123:0/2613661427 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f556c004020 con 0x7f557c074470 2024-09-16T15:46:50.658 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:50.652+0000 7f5581cd0640 1 -- 172.21.15.123:0/2613661427 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f557c06f010 con 0x7f557c074470 2024-09-16T15:46:50.658 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:50.652+0000 7f5581cd0640 1 -- 172.21.15.123:0/2613661427 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f557c06f5d0 con 0x7f557c074470 2024-09-16T15:46:50.659 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:50.652+0000 7f5581cd0640 1 -- 172.21.15.123:0/2613661427 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f557c0cd6e0 con 0x7f557c074470 2024-09-16T15:46:50.659 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:50.652+0000 7f5563fff640 1 -- 172.21.15.123:0/2613661427 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 11) ==== 97386+0+0 (secure 0 0 0) 0x7f556c0041c0 con 0x7f557c074470 2024-09-16T15:46:50.659 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:50.652+0000 7f5563fff640 1 --2- 172.21.15.123:0/2613661427 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f5568073220 0x7f55680756e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:46:50.659 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:50.652+0000 7f5563fff640 1 -- 172.21.15.123:0/2613661427 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(61..61 src has 1..61) ==== 4943+0+0 (secure 0 0 0) 0x7f556c094530 con 0x7f557c074470 2024-09-16T15:46:50.660 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:50.652+0000 7f557affd640 1 --2- 172.21.15.123:0/2613661427 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f5568073220 0x7f55680756e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:46:50.660 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:50.656+0000 7f557affd640 1 --2- 172.21.15.123:0/2613661427 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f5568073220 0x7f55680756e0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f55700af000 tx=0x7f557001afe0 comp rx=0 tx=0).ready entity=mgr.4135 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:46:50.662 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:50.656+0000 7f5563fff640 1 -- 172.21.15.123:0/2613661427 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f556c020020 con 0x7f557c074470 2024-09-16T15:46:50.834 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:50.828+0000 7f5581cd0640 1 -- 172.21.15.123:0/2613661427 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f557c0c3350 con 0x7f557c074470 2024-09-16T15:46:50.835 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:50.832+0000 7f5563fff640 1 -- 172.21.15.123:0/2613661427 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v61) ==== 74+0+13094 (secure 0 0 0) 0x7f556c05f5f0 con 0x7f557c074470 2024-09-16T15:46:50.835 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-09-16T15:46:50.836 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":61,"fsid":"56077f89-18c8-486f-ae7e-eecc0bb7822c","created":"2024-09-16T15:42:26.364892+0000","modified":"2024-09-16T15:46:49.487604+0000","last_up_change":"2024-09-16T15:42:38.707234+0000","last_in_change":"2024-09-16T15:42:29.423662+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-16T15:42:39.287764+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9e7c1955-a607-4671-9be7-f2ef760ceeaa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6800","nonce":3882044969},{"type":"v1","addr":"172.21.15.123:6801","nonce":3882044969}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6802","nonce":3882044969},{"type":"v1","addr":"172.21.15.123:6803","nonce":3882044969}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6806","nonce":3882044969},{"type":"v1","addr":"172.21.15.123:6807","nonce":3882044969}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6804","nonce":3882044969},{"type":"v1","addr":"172.21.15.123:6805","nonce":3882044969}]},"public_addr":"172.21.15.123:6801/3882044969","cluster_addr":"172.21.15.123:6803/3882044969","heartbeat_back_addr":"172.21.15.123:6807/3882044969","heartbeat_front_addr":"172.21.15.123:6805/3882044969","state":["exists","up"]},{"osd":1,"uuid":"c540cfde-0977-4742-9b17-fd634668e010","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6816","nonce":4037001850},{"type":"v1","addr":"172.21.15.123:6817","nonce":4037001850}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6818","nonce":4037001850},{"type":"v1","addr":"172.21.15.123:6819","nonce":4037001850}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6822","nonce":4037001850},{"type":"v1","addr":"172.21.15.123:6823","nonce":4037001850}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6820","nonce":4037001850},{"type":"v1","addr":"172.21.15.123:6821","nonce":4037001850}]},"public_addr":"172.21.15.123:6817/4037001850","cluster_addr":"172.21.15.123:6819/4037001850","heartbeat_back_addr":"172.21.15.123:6823/4037001850","heartbeat_front_addr":"172.21.15.123:6821/4037001850","state":["exists","up"]},{"osd":2,"uuid":"9864a496-f169-486b-95c4-1d59162793a7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6824","nonce":1332143088},{"type":"v1","addr":"172.21.15.123:6825","nonce":1332143088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6826","nonce":1332143088},{"type":"v1","addr":"172.21.15.123:6827","nonce":1332143088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6830","nonce":1332143088},{"type":"v1","addr":"172.21.15.123:6831","nonce":1332143088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6828","nonce":1332143088},{"type":"v1","addr":"172.21.15.123:6829","nonce":1332143088}]},"public_addr":"172.21.15.123:6825/1332143088","cluster_addr":"172.21.15.123:6827/1332143088","heartbeat_back_addr":"172.21.15.123:6831/1332143088","heartbeat_front_addr":"172.21.15.123:6829/1332143088","state":["exists","up"]},{"osd":3,"uuid":"c6d1f817-f074-4325-95e1-578eba13edda","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6808","nonce":988586998},{"type":"v1","addr":"172.21.15.123:6809","nonce":988586998}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6810","nonce":988586998},{"type":"v1","addr":"172.21.15.123:6811","nonce":988586998}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6814","nonce":988586998},{"type":"v1","addr":"172.21.15.123:6815","nonce":988586998}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6812","nonce":988586998},{"type":"v1","addr":"172.21.15.123:6813","nonce":988586998}]},"public_addr":"172.21.15.123:6809/988586998","cluster_addr":"172.21.15.123:6811/988586998","heartbeat_back_addr":"172.21.15.123:6815/988586998","heartbeat_front_addr":"172.21.15.123:6813/988586998","state":["exists","up"]},{"osd":4,"uuid":"200a8df4-50ef-40a5-b48a-58981269c086","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6816","nonce":3387813890},{"type":"v1","addr":"172.21.15.184:6817","nonce":3387813890}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6818","nonce":3387813890},{"type":"v1","addr":"172.21.15.184:6819","nonce":3387813890}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6822","nonce":3387813890},{"type":"v1","addr":"172.21.15.184:6823","nonce":3387813890}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6820","nonce":3387813890},{"type":"v1","addr":"172.21.15.184:6821","nonce":3387813890}]},"public_addr":"172.21.15.184:6817/3387813890","cluster_addr":"172.21.15.184:6819/3387813890","heartbeat_back_addr":"172.21.15.184:6823/3387813890","heartbeat_front_addr":"172.21.15.184:6821/3387813890","state":["exists","up"]},{"osd":5,"uuid":"32d5a900-da3d-412c-bc19-13861a2368d8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6808","nonce":2680916017},{"type":"v1","addr":"172.21.15.184:6809","nonce":2680916017}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6810","nonce":2680916017},{"type":"v1","addr":"172.21.15.184:6811","nonce":2680916017}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6814","nonce":2680916017},{"type":"v1","addr":"172.21.15.184:6815","nonce":2680916017}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6812","nonce":2680916017},{"type":"v1","addr":"172.21.15.184:6813","nonce":2680916017}]},"public_addr":"172.21.15.184:6809/2680916017","cluster_addr":"172.21.15.184:6811/2680916017","heartbeat_back_addr":"172.21.15.184:6815/2680916017","heartbeat_front_addr":"172.21.15.184:6813/2680916017","state":["exists","up"]},{"osd":6,"uuid":"cde6f39e-aa1b-4260-ac19-774ee6aaca72","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6824","nonce":2183120457},{"type":"v1","addr":"172.21.15.184:6825","nonce":2183120457}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6826","nonce":2183120457},{"type":"v1","addr":"172.21.15.184:6827","nonce":2183120457}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6830","nonce":2183120457},{"type":"v1","addr":"172.21.15.184:6831","nonce":2183120457}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6828","nonce":2183120457},{"type":"v1","addr":"172.21.15.184:6829","nonce":2183120457}]},"public_addr":"172.21.15.184:6825/2183120457","cluster_addr":"172.21.15.184:6827/2183120457","heartbeat_back_addr":"172.21.15.184:6831/2183120457","heartbeat_front_addr":"172.21.15.184:6829/2183120457","state":["exists","up"]},{"osd":7,"uuid":"4f381778-eee1-47c6-804f-df0e6b33e313","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6800","nonce":4102957285},{"type":"v1","addr":"172.21.15.184:6801","nonce":4102957285}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6802","nonce":4102957285},{"type":"v1","addr":"172.21.15.184:6803","nonce":4102957285}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6806","nonce":4102957285},{"type":"v1","addr":"172.21.15.184:6807","nonce":4102957285}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6804","nonce":4102957285},{"type":"v1","addr":"172.21.15.184:6805","nonce":4102957285}]},"public_addr":"172.21.15.184:6801/4102957285","cluster_addr":"172.21.15.184:6803/4102957285","heartbeat_back_addr":"172.21.15.184:6807/4102957285","heartbeat_front_addr":"172.21.15.184:6805/4102957285","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:35.972210+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.222007+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.198617+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.385663+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.358935+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.156066+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.752973+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:35.682846+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.123:6835/3589624203":"2024-09-17T15:46:46.256890+0000","172.21.15.123:6833/3589624203":"2024-09-17T15:46:46.256890+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T15:46:50.838 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:50.832+0000 7f5581cd0640 1 -- 172.21.15.123:0/2613661427 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f5568073220 msgr2=0x7f55680756e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:46:50.838 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:50.832+0000 7f5581cd0640 1 --2- 172.21.15.123:0/2613661427 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f5568073220 0x7f55680756e0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f55700af000 tx=0x7f557001afe0 comp rx=0 tx=0).stop 2024-09-16T15:46:50.838 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:50.832+0000 7f5581cd0640 1 -- 172.21.15.123:0/2613661427 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f557c074470 msgr2=0x7f557c157e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:46:50.838 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:50.832+0000 7f5581cd0640 1 --2- 172.21.15.123:0/2613661427 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f557c074470 0x7f557c157e60 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 crypto rx=0x7f556c00d780 tx=0x7f556c00dc40 comp rx=0 tx=0).stop 2024-09-16T15:46:50.838 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:50.832+0000 7f5581cd0640 1 -- 172.21.15.123:0/2613661427 shutdown_connections 2024-09-16T15:46:50.839 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:50.832+0000 7f5581cd0640 1 --2- 172.21.15.123:0/2613661427 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f5568073220 0x7f55680756e0 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:50.839 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:50.832+0000 7f5581cd0640 1 --2- 172.21.15.123:0/2613661427 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f557c15e160 0x7f557c0c9470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:50.839 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:50.832+0000 7f5581cd0640 1 --2- 172.21.15.123:0/2613661427 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f557c0cdc20 0x7f557c0c8f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:50.839 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:50.832+0000 7f5581cd0640 1 --2- 172.21.15.123:0/2613661427 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f557c074470 0x7f557c157e60 unknown :-1 s=CLOSED pgs=730 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:50.839 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:50.832+0000 7f5581cd0640 1 -- 172.21.15.123:0/2613661427 >> 172.21.15.123:0/2613661427 conn(0x7f557c0cfb10 msgr2=0x7f557c0af610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:46:50.839 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:50.832+0000 7f5581cd0640 1 -- 172.21.15.123:0/2613661427 shutdown_connections 2024-09-16T15:46:50.839 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:50.832+0000 7f5581cd0640 1 -- 172.21.15.123:0/2613661427 wait complete. 2024-09-16T15:46:50.855 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-16T15:42:39.287764+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-09-16T15:46:50.855 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-09-16T15:46:50.980 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:50.976+0000 7f047d236640 1 -- 172.21.15.123:0/3340109043 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0478068d80 msgr2=0x7f04780cc330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:46:50.981 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:50.976+0000 7f047d236640 1 --2- 172.21.15.123:0/3340109043 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0478068d80 0x7f04780cc330 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7f0468009840 tx=0x7f046801c7f0 comp rx=0 tx=0).stop 2024-09-16T15:46:50.981 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:50.976+0000 7f047d236640 1 -- 172.21.15.123:0/3340109043 shutdown_connections 2024-09-16T15:46:50.981 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:50.976+0000 7f047d236640 1 --2- 172.21.15.123:0/3340109043 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0478068d80 0x7f04780cc330 unknown :-1 s=CLOSED pgs=732 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:50.981 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:50.976+0000 7f047d236640 1 -- 172.21.15.123:0/3340109043 >> 172.21.15.123:0/3340109043 conn(0x7f04780741a0 msgr2=0x7f04780745b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:46:50.981 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:50.976+0000 7f047d236640 1 -- 172.21.15.123:0/3340109043 shutdown_connections 2024-09-16T15:46:50.981 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:50.976+0000 7f047d236640 1 -- 172.21.15.123:0/3340109043 wait complete. 2024-09-16T15:46:50.982 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:50.976+0000 7f047d236640 1 Processor -- start 2024-09-16T15:46:50.982 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:50.976+0000 7f047d236640 1 -- start start 2024-09-16T15:46:50.983 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:50.976+0000 7f047d236640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0478068d80 0x7f047815b3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:46:50.984 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:50.976+0000 7f047d236640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f04780b0ef0 0x7f0478150c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:46:50.984 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:50.976+0000 7f047d236640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f0478099300 0x7f04781fb5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:46:50.984 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:50.976+0000 7f047d236640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f047816a530 con 0x7f0478068d80 2024-09-16T15:46:50.984 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:50.976+0000 7f047d236640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f047816a3b0 con 0x7f04780b0ef0 2024-09-16T15:46:50.984 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:50.976+0000 7f047d236640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f047816a6b0 con 0x7f0478099300 2024-09-16T15:46:50.984 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:50.976+0000 7f0476d9d640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0478068d80 0x7f047815b3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:46:50.984 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:50.976+0000 7f0476d9d640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0478068d80 0x7f047815b3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:54774/0 (socket says 172.21.15.123:54774) 2024-09-16T15:46:50.984 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:50.976+0000 7f0476d9d640 1 -- 172.21.15.123:0/672853525 learned_addr learned my addr 172.21.15.123:0/672853525 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:46:50.984 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:50.976+0000 7f047759e640 1 --2- 172.21.15.123:0/672853525 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f0478099300 0x7f04781fb5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:46:50.984 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:50.976+0000 7f047659c640 1 --2- 172.21.15.123:0/672853525 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f04780b0ef0 0x7f0478150c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:46:50.984 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:50.976+0000 7f0476d9d640 1 -- 172.21.15.123:0/672853525 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f0478099300 msgr2=0x7f04781fb5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:46:50.984 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:50.980+0000 7f0476d9d640 1 --2- 172.21.15.123:0/672853525 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f0478099300 0x7f04781fb5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:50.985 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:50.980+0000 7f0476d9d640 1 -- 172.21.15.123:0/672853525 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f04780b0ef0 msgr2=0x7f0478150c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:46:50.985 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:50.980+0000 7f0476d9d640 1 --2- 172.21.15.123:0/672853525 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f04780b0ef0 0x7f0478150c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:50.985 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:50.980+0000 7f0476d9d640 1 -- 172.21.15.123:0/672853525 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0478151190 con 0x7f0478068d80 2024-09-16T15:46:50.985 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:50.980+0000 7f0476d9d640 1 --2- 172.21.15.123:0/672853525 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0478068d80 0x7f047815b3d0 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 crypto rx=0x7f047000cba0 tx=0x7f047000cc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:46:50.985 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:50.980+0000 7f045f7fe640 1 -- 172.21.15.123:0/672853525 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0470016020 con 0x7f0478068d80 2024-09-16T15:46:50.985 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:50.980+0000 7f045f7fe640 1 -- 172.21.15.123:0/672853525 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f047000aae0 con 0x7f0478068d80 2024-09-16T15:46:50.985 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:50.980+0000 7f047d236640 1 -- 172.21.15.123:0/672853525 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0478152bb0 con 0x7f0478068d80 2024-09-16T15:46:50.985 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:50.980+0000 7f045f7fe640 1 -- 172.21.15.123:0/672853525 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0470012420 con 0x7f0478068d80 2024-09-16T15:46:50.985 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:50.980+0000 7f047d236640 1 -- 172.21.15.123:0/672853525 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f04781f8e60 con 0x7f0478068d80 2024-09-16T15:46:50.986 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:50.980+0000 7f047d236640 1 -- 172.21.15.123:0/672853525 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f04780b12d0 con 0x7f0478068d80 2024-09-16T15:46:50.986 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:50.980+0000 7f045f7fe640 1 -- 172.21.15.123:0/672853525 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 11) ==== 97386+0+0 (secure 0 0 0) 0x7f0470019020 con 0x7f0478068d80 2024-09-16T15:46:50.986 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:50.980+0000 7f045f7fe640 1 --2- 172.21.15.123:0/672853525 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f0464073250 0x7f0464075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:46:50.986 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:50.980+0000 7f045f7fe640 1 -- 172.21.15.123:0/672853525 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(61..61 src has 1..61) ==== 4943+0+0 (secure 0 0 0) 0x7f0470094390 con 0x7f0478068d80 2024-09-16T15:46:50.986 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:50.980+0000 7f047659c640 1 --2- 172.21.15.123:0/672853525 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f0464073250 0x7f0464075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:46:50.986 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:50.980+0000 7f047659c640 1 --2- 172.21.15.123:0/672853525 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f0464073250 0x7f0464075710 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f047815a950 tx=0x7f046c008040 comp rx=0 tx=0).ready entity=mgr.4135 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:46:50.990 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:50.984+0000 7f045f7fe640 1 -- 172.21.15.123:0/672853525 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f0470020020 con 0x7f0478068d80 2024-09-16T15:46:51.162 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:51.156+0000 7f047d236640 1 -- 172.21.15.123:0/672853525 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f047806e510 con 0x7f0478068d80 2024-09-16T15:46:51.163 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:51.160+0000 7f045f7fe640 1 -- 172.21.15.123:0/672853525 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v61) ==== 93+0+10 (secure 0 0 0) 0x7f047005f450 con 0x7f0478068d80 2024-09-16T15:46:51.163 INFO:teuthology.orchestra.run.smithi123.stdout:pg_num: 1 2024-09-16T15:46:51.165 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:51.160+0000 7f047d236640 1 -- 172.21.15.123:0/672853525 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f0464073250 msgr2=0x7f0464075710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:46:51.166 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:51.160+0000 7f047d236640 1 --2- 172.21.15.123:0/672853525 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f0464073250 0x7f0464075710 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f047815a950 tx=0x7f046c008040 comp rx=0 tx=0).stop 2024-09-16T15:46:51.166 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:51.160+0000 7f047d236640 1 -- 172.21.15.123:0/672853525 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0478068d80 msgr2=0x7f047815b3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:46:51.166 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:51.160+0000 7f047d236640 1 --2- 172.21.15.123:0/672853525 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0478068d80 0x7f047815b3d0 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 crypto rx=0x7f047000cba0 tx=0x7f047000cc90 comp rx=0 tx=0).stop 2024-09-16T15:46:51.166 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:51.160+0000 7f047d236640 1 -- 172.21.15.123:0/672853525 shutdown_connections 2024-09-16T15:46:51.166 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:51.160+0000 7f047d236640 1 --2- 172.21.15.123:0/672853525 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f0464073250 0x7f0464075710 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:51.166 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:51.160+0000 7f047d236640 1 --2- 172.21.15.123:0/672853525 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f0478099300 0x7f04781fb5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:51.166 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:51.160+0000 7f047d236640 1 --2- 172.21.15.123:0/672853525 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f04780b0ef0 0x7f0478150c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:51.166 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:51.160+0000 7f047d236640 1 --2- 172.21.15.123:0/672853525 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0478068d80 0x7f047815b3d0 unknown :-1 s=CLOSED pgs=733 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:51.166 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:51.160+0000 7f047d236640 1 -- 172.21.15.123:0/672853525 >> 172.21.15.123:0/672853525 conn(0x7f04780741a0 msgr2=0x7f04780cdf00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:46:51.166 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:51.160+0000 7f047d236640 1 -- 172.21.15.123:0/672853525 shutdown_connections 2024-09-16T15:46:51.166 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:51.160+0000 7f047d236640 1 -- 172.21.15.123:0/672853525 wait complete. 2024-09-16T15:46:51.180 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_volumes.TestMisc.test_mgr_eviction' 2024-09-16T15:46:51.308 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:51.304+0000 7f52bf219640 1 -- 172.21.15.123:0/590677318 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f52a0004810 msgr2=0x7f52a0004c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:46:51.308 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:51.304+0000 7f52bf219640 1 --2- 172.21.15.123:0/590677318 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f52a0004810 0x7f52a0004c90 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x7f52ac0016f0 tx=0x7f52ac01c720 comp rx=0 tx=0).stop 2024-09-16T15:46:51.309 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:51.304+0000 7f52bf219640 1 -- 172.21.15.123:0/590677318 shutdown_connections 2024-09-16T15:46:51.309 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:51.304+0000 7f52bf219640 1 --2- 172.21.15.123:0/590677318 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f52a0004810 0x7f52a0004c90 unknown :-1 s=CLOSED pgs=735 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:51.309 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:51.304+0000 7f52bf219640 1 -- 172.21.15.123:0/590677318 >> 172.21.15.123:0/590677318 conn(0x7f52b80cfb10 msgr2=0x7f52b80cff20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:46:51.309 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:51.304+0000 7f52bf219640 1 -- 172.21.15.123:0/590677318 shutdown_connections 2024-09-16T15:46:51.309 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:51.304+0000 7f52bf219640 1 -- 172.21.15.123:0/590677318 wait complete. 2024-09-16T15:46:51.309 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:51.304+0000 7f52bf219640 1 Processor -- start 2024-09-16T15:46:51.309 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:51.304+0000 7f52bf219640 1 -- start start 2024-09-16T15:46:51.312 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:51.304+0000 7f52bf219640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f52a0004810 0x7f52b80c7d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:46:51.312 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:51.304+0000 7f52bf219640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f52b8153280 0x7f52b8153c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:46:51.312 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:51.304+0000 7f52bf219640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f52b81541c0 0x7f52b80ce680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:46:51.312 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:51.304+0000 7f52bf219640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f52b80cab80 con 0x7f52b81541c0 2024-09-16T15:46:51.312 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:51.304+0000 7f52bf219640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f52b815c1c0 con 0x7f52a0004810 2024-09-16T15:46:51.312 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:51.304+0000 7f52bf219640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f52b80cad00 con 0x7f52b8153280 2024-09-16T15:46:51.312 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:51.304+0000 7f52bd7b6640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f52b81541c0 0x7f52b80ce680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:46:51.312 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:51.304+0000 7f52bd7b6640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f52b81541c0 0x7f52b80ce680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:54798/0 (socket says 172.21.15.123:54798) 2024-09-16T15:46:51.312 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:51.304+0000 7f52bd7b6640 1 -- 172.21.15.123:0/816559621 learned_addr learned my addr 172.21.15.123:0/816559621 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:46:51.312 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:51.304+0000 7f52bcfb5640 1 --2- 172.21.15.123:0/816559621 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f52a0004810 0x7f52b80c7d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:46:51.312 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:51.304+0000 7f52b7fff640 1 --2- 172.21.15.123:0/816559621 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f52b8153280 0x7f52b8153c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:46:51.312 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:51.304+0000 7f52bd7b6640 1 -- 172.21.15.123:0/816559621 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f52b8153280 msgr2=0x7f52b8153c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:46:51.313 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:51.304+0000 7f52bd7b6640 1 --2- 172.21.15.123:0/816559621 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f52b8153280 0x7f52b8153c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:51.313 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:51.304+0000 7f52bd7b6640 1 -- 172.21.15.123:0/816559621 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f52a0004810 msgr2=0x7f52b80c7d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:46:51.313 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:51.304+0000 7f52bd7b6640 1 --2- 172.21.15.123:0/816559621 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f52a0004810 0x7f52b80c7d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:51.313 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:51.304+0000 7f52bd7b6640 1 -- 172.21.15.123:0/816559621 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f52b80cd8c0 con 0x7f52b81541c0 2024-09-16T15:46:51.313 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:51.304+0000 7f52bcfb5640 1 --2- 172.21.15.123:0/816559621 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f52a0004810 0x7f52b80c7d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T15:46:51.313 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:51.308+0000 7f52bd7b6640 1 --2- 172.21.15.123:0/816559621 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f52b81541c0 0x7f52b80ce680 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 crypto rx=0x7f52b81602a0 tx=0x7f52b000d8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:46:51.313 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:51.308+0000 7f52b57fa640 1 -- 172.21.15.123:0/816559621 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52b0005280 con 0x7f52b81541c0 2024-09-16T15:46:51.313 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:51.308+0000 7f52b57fa640 1 -- 172.21.15.123:0/816559621 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f52b0005440 con 0x7f52b81541c0 2024-09-16T15:46:51.313 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:51.308+0000 7f52bf219640 1 -- 172.21.15.123:0/816559621 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f52b80cdba0 con 0x7f52b81541c0 2024-09-16T15:46:51.313 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:51.308+0000 7f52b57fa640 1 -- 172.21.15.123:0/816559621 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52b0003840 con 0x7f52b81541c0 2024-09-16T15:46:51.313 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:51.308+0000 7f52bf219640 1 -- 172.21.15.123:0/816559621 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f52b80ce160 con 0x7f52b81541c0 2024-09-16T15:46:51.313 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:51.308+0000 7f52bf219640 1 -- 172.21.15.123:0/816559621 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f52b806e8c0 con 0x7f52b81541c0 2024-09-16T15:46:51.314 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:51.308+0000 7f52b57fa640 1 -- 172.21.15.123:0/816559621 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 11) ==== 97386+0+0 (secure 0 0 0) 0x7f52b0013430 con 0x7f52b81541c0 2024-09-16T15:46:51.314 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:51.308+0000 7f52b57fa640 1 --2- 172.21.15.123:0/816559621 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f52a4073250 0x7f52a4075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:46:51.314 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:51.308+0000 7f52b57fa640 1 -- 172.21.15.123:0/816559621 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(61..61 src has 1..61) ==== 4943+0+0 (secure 0 0 0) 0x7f52b0093ba0 con 0x7f52b81541c0 2024-09-16T15:46:51.314 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:51.308+0000 7f52bcfb5640 1 --2- 172.21.15.123:0/816559621 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f52a4073250 0x7f52a4075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:46:51.314 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:51.308+0000 7f52bcfb5640 1 --2- 172.21.15.123:0/816559621 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f52a4073250 0x7f52a4075710 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f52a8005b50 tx=0x7f52a8009ae0 comp rx=0 tx=0).ready entity=mgr.4135 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:46:51.316 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:51.312+0000 7f52b57fa640 1 -- 172.21.15.123:0/816559621 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f52b0093e20 con 0x7f52b81541c0 2024-09-16T15:46:51.555 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:51.552+0000 7f52bf219640 1 -- 172.21.15.123:0/816559621 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_volumes.TestMisc.test_mgr_eviction"]} v 0) -- 0x7f52b80ce330 con 0x7f52b81541c0 2024-09-16T15:46:52.511 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:52.508+0000 7f52b57fa640 1 -- 172.21.15.123:0/816559621 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_volumes.TestMisc.test_mgr_eviction"]}]=0 v209) ==== 134+0+0 (secure 0 0 0) 0x7f52b005ec60 con 0x7f52b81541c0 2024-09-16T15:46:52.515 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:52.508+0000 7f52bf219640 1 -- 172.21.15.123:0/816559621 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f52a4073250 msgr2=0x7f52a4075710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:46:52.515 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:52.508+0000 7f52bf219640 1 --2- 172.21.15.123:0/816559621 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f52a4073250 0x7f52a4075710 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f52a8005b50 tx=0x7f52a8009ae0 comp rx=0 tx=0).stop 2024-09-16T15:46:52.515 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:52.508+0000 7f52bf219640 1 -- 172.21.15.123:0/816559621 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f52b81541c0 msgr2=0x7f52b80ce680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:46:52.515 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:52.508+0000 7f52bf219640 1 --2- 172.21.15.123:0/816559621 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f52b81541c0 0x7f52b80ce680 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 crypto rx=0x7f52b81602a0 tx=0x7f52b000d8b0 comp rx=0 tx=0).stop 2024-09-16T15:46:52.515 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:52.508+0000 7f52bf219640 1 -- 172.21.15.123:0/816559621 shutdown_connections 2024-09-16T15:46:52.515 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:52.508+0000 7f52bf219640 1 --2- 172.21.15.123:0/816559621 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f52a4073250 0x7f52a4075710 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:52.515 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:52.508+0000 7f52bf219640 1 --2- 172.21.15.123:0/816559621 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f52b81541c0 0x7f52b80ce680 unknown :-1 s=CLOSED pgs=736 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:52.515 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:52.508+0000 7f52bf219640 1 --2- 172.21.15.123:0/816559621 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f52b8153280 0x7f52b8153c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:52.515 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:52.508+0000 7f52bf219640 1 --2- 172.21.15.123:0/816559621 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f52a0004810 0x7f52b80c7d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:52.515 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:52.508+0000 7f52bf219640 1 -- 172.21.15.123:0/816559621 >> 172.21.15.123:0/816559621 conn(0x7f52b80cfb10 msgr2=0x7f52b80c5250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:46:52.516 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:52.508+0000 7f52bf219640 1 -- 172.21.15.123:0/816559621 shutdown_connections 2024-09-16T15:46:52.516 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:52.512+0000 7f52bf219640 1 -- 172.21.15.123:0/816559621 wait complete. 2024-09-16T15:46:52.537 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config dump 2024-09-16T15:46:52.668 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:52.664+0000 7f7355c68640 1 -- 172.21.15.123:0/2845120145 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f735015a1b0 msgr2=0x7f735015a620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:46:52.669 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:52.664+0000 7f7355c68640 1 --2- 172.21.15.123:0/2845120145 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f735015a1b0 0x7f735015a620 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7f7348009c80 tx=0x7f734801cd70 comp rx=0 tx=0).stop 2024-09-16T15:46:52.669 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:52.664+0000 7f7355c68640 1 -- 172.21.15.123:0/2845120145 shutdown_connections 2024-09-16T15:46:52.669 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:52.664+0000 7f7355c68640 1 --2- 172.21.15.123:0/2845120145 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f735015a1b0 0x7f735015a620 unknown :-1 s=CLOSED pgs=737 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:52.669 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:52.664+0000 7f7355c68640 1 --2- 172.21.15.123:0/2845120145 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f7350070290 0x7f73500af4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:52.669 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:52.664+0000 7f7355c68640 1 -- 172.21.15.123:0/2845120145 >> 172.21.15.123:0/2845120145 conn(0x7f73500b1a90 msgr2=0x7f73500b1ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:46:52.669 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:52.664+0000 7f7355c68640 1 -- 172.21.15.123:0/2845120145 shutdown_connections 2024-09-16T15:46:52.669 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:52.664+0000 7f7355c68640 1 -- 172.21.15.123:0/2845120145 wait complete. 2024-09-16T15:46:52.669 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:52.664+0000 7f7355c68640 1 Processor -- start 2024-09-16T15:46:52.669 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:52.664+0000 7f7355c68640 1 -- start start 2024-09-16T15:46:52.671 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:52.664+0000 7f7355c68640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7350070290 0x7f73501f72b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:46:52.671 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:52.664+0000 7f7355c68640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f73500741f0 0x7f73500b3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:46:52.672 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:52.664+0000 7f7355c68640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f735015a1b0 0x7f7350163db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:46:52.672 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:52.664+0000 7f7355c68640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f73500ae140 con 0x7f7350070290 2024-09-16T15:46:52.672 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:52.664+0000 7f7355c68640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f73500adfc0 con 0x7f735015a1b0 2024-09-16T15:46:52.672 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:52.664+0000 7f7355c68640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f73500ae2c0 con 0x7f73500741f0 2024-09-16T15:46:52.672 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:52.664+0000 7f734f7fe640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7350070290 0x7f73501f72b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:46:52.672 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:52.664+0000 7f734f7fe640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7350070290 0x7f73501f72b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:39958/0 (socket says 172.21.15.123:39958) 2024-09-16T15:46:52.672 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:52.664+0000 7f734f7fe640 1 -- 172.21.15.123:0/671459568 learned_addr learned my addr 172.21.15.123:0/671459568 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:46:52.672 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:52.668+0000 7f734ffff640 1 --2- 172.21.15.123:0/671459568 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f735015a1b0 0x7f7350163db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:46:52.672 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:52.668+0000 7f734f7fe640 1 -- 172.21.15.123:0/671459568 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f73500741f0 msgr2=0x7f73500b3ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:46:52.672 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:52.668+0000 7f734f7fe640 1 --2- 172.21.15.123:0/671459568 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f73500741f0 0x7f73500b3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:52.672 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:52.668+0000 7f734f7fe640 1 -- 172.21.15.123:0/671459568 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f735015a1b0 msgr2=0x7f7350163db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:46:52.673 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:52.668+0000 7f734f7fe640 1 --2- 172.21.15.123:0/671459568 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f735015a1b0 0x7f7350163db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:52.673 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:52.668+0000 7f734f7fe640 1 -- 172.21.15.123:0/671459568 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7350165d50 con 0x7f7350070290 2024-09-16T15:46:52.673 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:52.668+0000 7f734f7fe640 1 --2- 172.21.15.123:0/671459568 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7350070290 0x7f73501f72b0 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7f734000c500 tx=0x7f734000c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:46:52.673 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:52.668+0000 7f7337fff640 1 -- 172.21.15.123:0/671459568 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7340017070 con 0x7f7350070290 2024-09-16T15:46:52.673 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:52.668+0000 7f7337fff640 1 -- 172.21.15.123:0/671459568 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f73400053c0 con 0x7f7350070290 2024-09-16T15:46:52.673 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:52.668+0000 7f7337fff640 1 -- 172.21.15.123:0/671459568 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7340012890 con 0x7f7350070290 2024-09-16T15:46:52.673 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:52.668+0000 7f7355c68640 1 -- 172.21.15.123:0/671459568 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7350166030 con 0x7f7350070290 2024-09-16T15:46:52.673 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:52.668+0000 7f7355c68640 1 -- 172.21.15.123:0/671459568 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7350165a20 con 0x7f7350070290 2024-09-16T15:46:52.674 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:52.668+0000 7f7355c68640 1 -- 172.21.15.123:0/671459568 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f73500af4a0 con 0x7f7350070290 2024-09-16T15:46:52.674 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:52.668+0000 7f7337fff640 1 -- 172.21.15.123:0/671459568 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 11) ==== 97386+0+0 (secure 0 0 0) 0x7f734001a070 con 0x7f7350070290 2024-09-16T15:46:52.674 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:52.668+0000 7f7337fff640 1 --2- 172.21.15.123:0/671459568 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f733c073220 0x7f733c0756e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:46:52.674 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:52.668+0000 7f7337fff640 1 -- 172.21.15.123:0/671459568 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(61..61 src has 1..61) ==== 4943+0+0 (secure 0 0 0) 0x7f7340093410 con 0x7f7350070290 2024-09-16T15:46:52.674 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:52.668+0000 7f734effd640 1 --2- 172.21.15.123:0/671459568 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f733c073220 0x7f733c0756e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:46:52.674 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:52.668+0000 7f734effd640 1 --2- 172.21.15.123:0/671459568 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f733c073220 0x7f733c0756e0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f735009da10 tx=0x7f7344005c80 comp rx=0 tx=0).ready entity=mgr.4135 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:46:52.678 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:52.672+0000 7f7337fff640 1 -- 172.21.15.123:0/671459568 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f734005e490 con 0x7f7350070290 2024-09-16T15:46:52.855 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:52.852+0000 7f7355c68640 1 -- 172.21.15.123:0/671459568 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "config dump"} v 0) -- 0x7f73500b3460 con 0x7f7350070290 2024-09-16T15:46:52.856 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:52.852+0000 7f7337fff640 1 -- 172.21.15.123:0/671459568 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "config dump"}]=0 v33) ==== 59+0+1998 (secure 0 0 0) 0x7f734005dd90 con 0x7f7350070290 2024-09-16T15:46:52.856 INFO:teuthology.orchestra.run.smithi123.stdout:WHO MASK LEVEL OPTION VALUE RO 2024-09-16T15:46:52.856 INFO:teuthology.orchestra.run.smithi123.stdout:mon advanced mon_op_complaint_time 120 2024-09-16T15:46:52.857 INFO:teuthology.orchestra.run.smithi123.stdout:mon dev mon_warn_on_pool_no_app false 2024-09-16T15:46:52.857 INFO:teuthology.orchestra.run.smithi123.stdout:osd advanced osd_op_complaint_time 180.000000 2024-09-16T15:46:52.857 INFO:teuthology.orchestra.run.smithi123.stdout:osd.0 basic osd_mclock_max_capacity_iops_ssd 4805.300677 2024-09-16T15:46:52.857 INFO:teuthology.orchestra.run.smithi123.stdout:osd.1 basic osd_mclock_max_capacity_iops_ssd 4806.240607 2024-09-16T15:46:52.857 INFO:teuthology.orchestra.run.smithi123.stdout:osd.2 basic osd_mclock_max_capacity_iops_ssd 4046.441686 2024-09-16T15:46:52.857 INFO:teuthology.orchestra.run.smithi123.stdout:osd.3 basic osd_mclock_max_capacity_iops_ssd 6882.576123 2024-09-16T15:46:52.857 INFO:teuthology.orchestra.run.smithi123.stdout:osd.4 basic osd_mclock_max_capacity_iops_ssd 6108.444971 2024-09-16T15:46:52.857 INFO:teuthology.orchestra.run.smithi123.stdout:osd.5 basic osd_mclock_max_capacity_iops_ssd 4995.096306 2024-09-16T15:46:52.857 INFO:teuthology.orchestra.run.smithi123.stdout:osd.6 basic osd_mclock_max_capacity_iops_ssd 6237.822095 2024-09-16T15:46:52.857 INFO:teuthology.orchestra.run.smithi123.stdout:osd.7 basic osd_mclock_max_capacity_iops_ssd 5202.385138 2024-09-16T15:46:52.857 INFO:teuthology.orchestra.run.smithi123.stdout:mds advanced debug_mds 20/20 2024-09-16T15:46:52.857 INFO:teuthology.orchestra.run.smithi123.stdout:mds advanced debug_mds_balancer 20/20 2024-09-16T15:46:52.857 INFO:teuthology.orchestra.run.smithi123.stdout:mds advanced debug_ms 2/2 2024-09-16T15:46:52.857 INFO:teuthology.orchestra.run.smithi123.stdout:mds dev mds_debug_frag true 2024-09-16T15:46:52.857 INFO:teuthology.orchestra.run.smithi123.stdout:mds dev mds_debug_scatterstat true 2024-09-16T15:46:52.858 INFO:teuthology.orchestra.run.smithi123.stdout:mds advanced mds_op_complaint_time 180.000000 2024-09-16T15:46:52.858 INFO:teuthology.orchestra.run.smithi123.stdout:mds dev mds_verify_scatter true 2024-09-16T15:46:52.858 INFO:teuthology.orchestra.run.smithi123.stdout:mds advanced osd_op_complaint_time 180.000000 2024-09-16T15:46:52.858 INFO:teuthology.orchestra.run.smithi123.stdout:mds advanced rados_mon_op_timeout 900 2024-09-16T15:46:52.858 INFO:teuthology.orchestra.run.smithi123.stdout:mds advanced rados_osd_op_timeout 900 2024-09-16T15:46:52.858 INFO:teuthology.orchestra.run.smithi123.stdout:client advanced client_mount_timeout 600 2024-09-16T15:46:52.858 INFO:teuthology.orchestra.run.smithi123.stdout:client advanced debug_client 20/20 2024-09-16T15:46:52.858 INFO:teuthology.orchestra.run.smithi123.stdout:client advanced debug_ms 1/1 2024-09-16T15:46:52.858 INFO:teuthology.orchestra.run.smithi123.stdout:client advanced rados_mon_op_timeout 900 2024-09-16T15:46:52.858 INFO:teuthology.orchestra.run.smithi123.stdout:client advanced rados_osd_op_timeout 900 2024-09-16T15:46:52.859 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:52.852+0000 7f7355c68640 1 -- 172.21.15.123:0/671459568 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f733c073220 msgr2=0x7f733c0756e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:46:52.859 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:52.852+0000 7f7355c68640 1 --2- 172.21.15.123:0/671459568 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f733c073220 0x7f733c0756e0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f735009da10 tx=0x7f7344005c80 comp rx=0 tx=0).stop 2024-09-16T15:46:52.859 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:52.852+0000 7f7355c68640 1 -- 172.21.15.123:0/671459568 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7350070290 msgr2=0x7f73501f72b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:46:52.859 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:52.852+0000 7f7355c68640 1 --2- 172.21.15.123:0/671459568 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7350070290 0x7f73501f72b0 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7f734000c500 tx=0x7f734000c9c0 comp rx=0 tx=0).stop 2024-09-16T15:46:52.859 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:52.852+0000 7f7355c68640 1 -- 172.21.15.123:0/671459568 shutdown_connections 2024-09-16T15:46:52.859 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:52.852+0000 7f7355c68640 1 --2- 172.21.15.123:0/671459568 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f733c073220 0x7f733c0756e0 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:52.859 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:52.856+0000 7f7355c68640 1 --2- 172.21.15.123:0/671459568 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f735015a1b0 0x7f7350163db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:52.859 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:52.856+0000 7f7355c68640 1 --2- 172.21.15.123:0/671459568 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f73500741f0 0x7f73500b3ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:52.860 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:52.856+0000 7f7355c68640 1 --2- 172.21.15.123:0/671459568 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7350070290 0x7f73501f72b0 unknown :-1 s=CLOSED pgs=738 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:52.860 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:52.856+0000 7f7355c68640 1 -- 172.21.15.123:0/671459568 >> 172.21.15.123:0/671459568 conn(0x7f73500b1a90 msgr2=0x7f73500b1e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:46:52.860 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:52.856+0000 7f7355c68640 1 -- 172.21.15.123:0/671459568 shutdown_connections 2024-09-16T15:46:52.860 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:52.856+0000 7f7355c68640 1 -- 172.21.15.123:0/671459568 wait complete. 2024-09-16T15:46:52.876 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-09-16T15:46:53.008 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:53.004+0000 7fcfdcfc3640 1 -- 172.21.15.123:0/1130216019 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcfd80b5780 msgr2=0x7fcfd80ce990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:46:53.008 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:53.004+0000 7fcfdcfc3640 1 --2- 172.21.15.123:0/1130216019 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcfd80b5780 0x7fcfd80ce990 secure :-1 s=READY pgs=739 cs=0 l=1 rev1=1 crypto rx=0x7fcfc0009900 tx=0x7fcfc001c850 comp rx=0 tx=0).stop 2024-09-16T15:46:53.008 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:53.004+0000 7fcfdcfc3640 1 -- 172.21.15.123:0/1130216019 shutdown_connections 2024-09-16T15:46:53.008 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:53.004+0000 7fcfdcfc3640 1 --2- 172.21.15.123:0/1130216019 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fcfd80d0f60 0x7fcfd80ceed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:53.009 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:53.004+0000 7fcfdcfc3640 1 --2- 172.21.15.123:0/1130216019 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcfd80b5780 0x7fcfd80ce990 unknown :-1 s=CLOSED pgs=739 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:53.009 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:53.004+0000 7fcfdcfc3640 1 -- 172.21.15.123:0/1130216019 >> 172.21.15.123:0/1130216019 conn(0x7fcfd809d3d0 msgr2=0x7fcfd80d40f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:46:53.009 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:53.004+0000 7fcfdcfc3640 1 -- 172.21.15.123:0/1130216019 shutdown_connections 2024-09-16T15:46:53.009 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:53.004+0000 7fcfdcfc3640 1 -- 172.21.15.123:0/1130216019 wait complete. 2024-09-16T15:46:53.009 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:53.004+0000 7fcfdcfc3640 1 Processor -- start 2024-09-16T15:46:53.009 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:53.004+0000 7fcfdcfc3640 1 -- start start 2024-09-16T15:46:53.011 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:53.004+0000 7fcfdcfc3640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fcfd80b5780 0x7fcfd8153ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:46:53.011 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:53.004+0000 7fcfdcfc3640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcfd80d0b80 0x7fcfd80cd660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:46:53.011 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:53.004+0000 7fcfdcfc3640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fcfd80d0f60 0x7fcfd80cdba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:46:53.012 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:53.004+0000 7fcfdcfc3640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fcfd8162b20 con 0x7fcfd80d0b80 2024-09-16T15:46:53.012 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:53.004+0000 7fcfdcfc3640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fcfd81629a0 con 0x7fcfd80b5780 2024-09-16T15:46:53.012 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:53.004+0000 7fcfdcfc3640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fcfd8162ca0 con 0x7fcfd80d0f60 2024-09-16T15:46:53.012 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:53.004+0000 7fcfcdd9b640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcfd80d0b80 0x7fcfd80cd660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:46:53.012 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:53.004+0000 7fcfcdd9b640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcfd80d0b80 0x7fcfd80cd660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:39984/0 (socket says 172.21.15.123:39984) 2024-09-16T15:46:53.012 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:53.004+0000 7fcfcdd9b640 1 -- 172.21.15.123:0/1123795483 learned_addr learned my addr 172.21.15.123:0/1123795483 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:46:53.012 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:53.004+0000 7fcfd659c640 1 --2- 172.21.15.123:0/1123795483 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fcfd80b5780 0x7fcfd8153ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:46:53.012 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:53.004+0000 7fcfd6d9d640 1 --2- 172.21.15.123:0/1123795483 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fcfd80d0f60 0x7fcfd80cdba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:46:53.012 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:53.004+0000 7fcfcdd9b640 1 -- 172.21.15.123:0/1123795483 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fcfd80d0f60 msgr2=0x7fcfd80cdba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:46:53.012 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:53.004+0000 7fcfcdd9b640 1 --2- 172.21.15.123:0/1123795483 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fcfd80d0f60 0x7fcfd80cdba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:53.012 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:53.004+0000 7fcfcdd9b640 1 -- 172.21.15.123:0/1123795483 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fcfd80b5780 msgr2=0x7fcfd8153ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:46:53.012 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:53.004+0000 7fcfcdd9b640 1 --2- 172.21.15.123:0/1123795483 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fcfd80b5780 0x7fcfd8153ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:53.012 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:53.004+0000 7fcfcdd9b640 1 -- 172.21.15.123:0/1123795483 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcfd80ccae0 con 0x7fcfd80d0b80 2024-09-16T15:46:53.013 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:53.008+0000 7fcfcdd9b640 1 --2- 172.21.15.123:0/1123795483 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcfd80d0b80 0x7fcfd80cd660 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7fcfc0009850 tx=0x7fcfc00a5590 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:46:53.013 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:53.008+0000 7fcfcf7fe640 1 -- 172.21.15.123:0/1123795483 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfc00ba020 con 0x7fcfd80d0b80 2024-09-16T15:46:53.013 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:53.008+0000 7fcfcf7fe640 1 -- 172.21.15.123:0/1123795483 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fcfc0004470 con 0x7fcfd80d0b80 2024-09-16T15:46:53.013 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:53.008+0000 7fcfdcfc3640 1 -- 172.21.15.123:0/1123795483 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcfd80cc800 con 0x7fcfd80d0b80 2024-09-16T15:46:53.013 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:53.008+0000 7fcfcf7fe640 1 -- 172.21.15.123:0/1123795483 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfc00b2040 con 0x7fcfd80d0b80 2024-09-16T15:46:53.013 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:53.008+0000 7fcfdcfc3640 1 -- 172.21.15.123:0/1123795483 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcfd815b710 con 0x7fcfd80d0b80 2024-09-16T15:46:53.014 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:53.008+0000 7fcfdcfc3640 1 -- 172.21.15.123:0/1123795483 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcfd80b43c0 con 0x7fcfd80d0b80 2024-09-16T15:46:53.014 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:53.008+0000 7fcfcf7fe640 1 -- 172.21.15.123:0/1123795483 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 11) ==== 97386+0+0 (secure 0 0 0) 0x7fcfc00bd070 con 0x7fcfd80d0b80 2024-09-16T15:46:53.014 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:53.008+0000 7fcfcf7fe640 1 --2- 172.21.15.123:0/1123795483 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fcfb8073220 0x7fcfb80756e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:46:53.014 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:53.008+0000 7fcfcf7fe640 1 -- 172.21.15.123:0/1123795483 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(61..61 src has 1..61) ==== 4943+0+0 (secure 0 0 0) 0x7fcfc012d970 con 0x7fcfd80d0b80 2024-09-16T15:46:53.014 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:53.008+0000 7fcfd659c640 1 --2- 172.21.15.123:0/1123795483 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fcfb8073220 0x7fcfb80756e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:46:53.014 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:53.008+0000 7fcfd659c640 1 --2- 172.21.15.123:0/1123795483 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fcfb8073220 0x7fcfb80756e0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7fcfc8000f70 tx=0x7fcfc8009040 comp rx=0 tx=0).ready entity=mgr.4135 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:46:53.017 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:53.012+0000 7fcfcf7fe640 1 -- 172.21.15.123:0/1123795483 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7fcfc00164d0 con 0x7fcfd80d0b80 2024-09-16T15:46:53.185 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:53.180+0000 7fcfdcfc3640 1 -- 172.21.15.123:0/1123795483 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) -- 0x7fcfd80cf030 con 0x7fcfd80d0b80 2024-09-16T15:46:53.190 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:53.184+0000 7fcfcf7fe640 1 -- 172.21.15.123:0/1123795483 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v34)=0 v34) ==== 122+0+0 (secure 0 0 0) 0x7fcfc00f89b0 con 0x7fcfd80d0b80 2024-09-16T15:46:53.193 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:53.188+0000 7fcfdcfc3640 1 -- 172.21.15.123:0/1123795483 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fcfb8073220 msgr2=0x7fcfb80756e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:46:53.194 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:53.188+0000 7fcfdcfc3640 1 --2- 172.21.15.123:0/1123795483 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fcfb8073220 0x7fcfb80756e0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7fcfc8000f70 tx=0x7fcfc8009040 comp rx=0 tx=0).stop 2024-09-16T15:46:53.194 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:53.188+0000 7fcfdcfc3640 1 -- 172.21.15.123:0/1123795483 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcfd80d0b80 msgr2=0x7fcfd80cd660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:46:53.194 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:53.188+0000 7fcfdcfc3640 1 --2- 172.21.15.123:0/1123795483 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcfd80d0b80 0x7fcfd80cd660 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7fcfc0009850 tx=0x7fcfc00a5590 comp rx=0 tx=0).stop 2024-09-16T15:46:53.194 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:53.188+0000 7fcfdcfc3640 1 -- 172.21.15.123:0/1123795483 shutdown_connections 2024-09-16T15:46:53.194 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:53.188+0000 7fcfdcfc3640 1 --2- 172.21.15.123:0/1123795483 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fcfb8073220 0x7fcfb80756e0 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:53.194 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:53.188+0000 7fcfdcfc3640 1 --2- 172.21.15.123:0/1123795483 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fcfd80d0f60 0x7fcfd80cdba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:53.194 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:53.188+0000 7fcfdcfc3640 1 --2- 172.21.15.123:0/1123795483 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcfd80d0b80 0x7fcfd80cd660 unknown :-1 s=CLOSED pgs=740 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:53.194 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:53.188+0000 7fcfdcfc3640 1 --2- 172.21.15.123:0/1123795483 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fcfd80b5780 0x7fcfd8153ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:53.194 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:53.188+0000 7fcfdcfc3640 1 -- 172.21.15.123:0/1123795483 >> 172.21.15.123:0/1123795483 conn(0x7fcfd809d3d0 msgr2=0x7fcfd80b0e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:46:53.194 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:53.188+0000 7fcfdcfc3640 1 -- 172.21.15.123:0/1123795483 shutdown_connections 2024-09-16T15:46:53.194 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:53.188+0000 7fcfdcfc3640 1 -- 172.21.15.123:0/1123795483 wait complete. 2024-09-16T15:46:53.209 INFO:teuthology.nuke:Clearing teuthology firewall rules... 2024-09-16T15:46:53.209 DEBUG:teuthology.orchestra.run.smithi123:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-09-16T15:46:53.296 DEBUG:teuthology.orchestra.run.smithi184:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-09-16T15:46:53.360 INFO:teuthology.nuke:Cleared teuthology firewall rules. 2024-09-16T15:46:53.361 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2024-09-16T15:46:53.361 DEBUG:teuthology.orchestra.run.smithi123:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-16T15:46:53.366 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16336468k,nr_inodes=4084117,mode=755,inode64 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3274684k,mode=755,inode64 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev,inode64 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=17082 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3274680k,nr_inodes=818670,mode=700,uid=1000,gid=1271,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3274684k,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-09-16T15:46:53.366 DEBUG:teuthology.orchestra.run.smithi184:> set -ex 2024-09-16T15:46:53.367 DEBUG:teuthology.orchestra.run.smithi184:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-16T15:46:53.373 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16336488k,nr_inodes=4084122,mode=755,inode64 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3274688k,mode=755,inode64 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev,inode64 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=17834 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3274684k,nr_inodes=818671,mode=700,uid=1000,gid=1271,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3274688k,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-09-16T15:46:53.373 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2024-09-16T15:46:53.373 DEBUG:teuthology.orchestra.run.smithi123:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-16T15:46:53.416 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16336468k,nr_inodes=4084117,mode=755,inode64 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3274684k,mode=755,inode64 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev,inode64 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=17082 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3274680k,nr_inodes=818670,mode=700,uid=1000,gid=1271,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3274684k,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-09-16T15:46:53.417 DEBUG:teuthology.orchestra.run.smithi184:> set -ex 2024-09-16T15:46:53.417 DEBUG:teuthology.orchestra.run.smithi184:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-16T15:46:53.422 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16336488k,nr_inodes=4084122,mode=755,inode64 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3274688k,mode=755,inode64 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev,inode64 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=17834 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3274684k,nr_inodes=818671,mode=700,uid=1000,gid=1271,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3274688k,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-09-16T15:46:53.422 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-09-16T15:46:53.422 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-16T15:46:53.588 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:53.584+0000 7ff919b4d640 1 -- 172.21.15.123:0/419012173 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff9140b0d60 msgr2=0x7ff9140d0280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:46:53.588 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:53.584+0000 7ff919b4d640 1 --2- 172.21.15.123:0/419012173 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff9140b0d60 0x7ff9140d0280 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 crypto rx=0x7ff904009840 tx=0x7ff90401c820 comp rx=0 tx=0).stop 2024-09-16T15:46:53.589 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:53.584+0000 7ff919b4d640 1 -- 172.21.15.123:0/419012173 shutdown_connections 2024-09-16T15:46:53.589 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:53.584+0000 7ff919b4d640 1 --2- 172.21.15.123:0/419012173 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff9140b0d60 0x7ff9140d0280 unknown :-1 s=CLOSED pgs=742 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:53.589 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:53.584+0000 7ff919b4d640 1 --2- 172.21.15.123:0/419012173 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7ff9140ccb10 0x7ff9140cf180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:53.589 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:53.584+0000 7ff919b4d640 1 -- 172.21.15.123:0/419012173 >> 172.21.15.123:0/419012173 conn(0x7ff9140d1ca0 msgr2=0x7ff9140d20b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:46:53.589 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:53.584+0000 7ff919b4d640 1 -- 172.21.15.123:0/419012173 shutdown_connections 2024-09-16T15:46:53.589 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:53.584+0000 7ff919b4d640 1 -- 172.21.15.123:0/419012173 wait complete. 2024-09-16T15:46:53.589 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:53.584+0000 7ff919b4d640 1 Processor -- start 2024-09-16T15:46:53.589 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:53.584+0000 7ff919b4d640 1 -- start start 2024-09-16T15:46:53.591 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:53.584+0000 7ff919b4d640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff9140b0d60 0x7ff914148b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:46:53.591 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:53.584+0000 7ff919b4d640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff9140ccb10 0x7ff914071880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:46:53.591 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:53.584+0000 7ff919b4d640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7ff9140d07c0 0x7ff914071dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:46:53.592 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:53.584+0000 7ff919b4d640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7ff914168b40 con 0x7ff9140ccb10 2024-09-16T15:46:53.592 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:53.584+0000 7ff919b4d640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7ff9141689c0 con 0x7ff9140b0d60 2024-09-16T15:46:53.592 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:53.584+0000 7ff919b4d640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7ff914168cc0 con 0x7ff9140d07c0 2024-09-16T15:46:53.592 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:53.584+0000 7ff912ffd640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff9140ccb10 0x7ff914071880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:46:53.592 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:53.584+0000 7ff912ffd640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff9140ccb10 0x7ff914071880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:40002/0 (socket says 172.21.15.123:40002) 2024-09-16T15:46:53.592 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:53.584+0000 7ff912ffd640 1 -- 172.21.15.123:0/3065002021 learned_addr learned my addr 172.21.15.123:0/3065002021 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:46:53.592 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:53.584+0000 7ff913fff640 1 --2- 172.21.15.123:0/3065002021 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7ff9140d07c0 0x7ff914071dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:46:53.592 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:53.584+0000 7ff9137fe640 1 --2- 172.21.15.123:0/3065002021 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff9140b0d60 0x7ff914148b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:46:53.592 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:53.584+0000 7ff912ffd640 1 -- 172.21.15.123:0/3065002021 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7ff9140d07c0 msgr2=0x7ff914071dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:46:53.592 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:53.584+0000 7ff912ffd640 1 --2- 172.21.15.123:0/3065002021 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7ff9140d07c0 0x7ff914071dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:53.592 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:53.584+0000 7ff912ffd640 1 -- 172.21.15.123:0/3065002021 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff9140b0d60 msgr2=0x7ff914148b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:46:53.592 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:53.584+0000 7ff912ffd640 1 --2- 172.21.15.123:0/3065002021 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff9140b0d60 0x7ff914148b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:53.592 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:53.584+0000 7ff912ffd640 1 -- 172.21.15.123:0/3065002021 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff9141f5320 con 0x7ff9140ccb10 2024-09-16T15:46:53.593 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:53.588+0000 7ff912ffd640 1 --2- 172.21.15.123:0/3065002021 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff9140ccb10 0x7ff914071880 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7ff904009790 tx=0x7ff9040a5670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:46:53.593 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:53.588+0000 7ff90bfff640 1 -- 172.21.15.123:0/3065002021 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9040bc070 con 0x7ff9140ccb10 2024-09-16T15:46:53.593 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:53.588+0000 7ff90bfff640 1 -- 172.21.15.123:0/3065002021 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7ff904004470 con 0x7ff9140ccb10 2024-09-16T15:46:53.593 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:53.588+0000 7ff919b4d640 1 -- 172.21.15.123:0/3065002021 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff9141f5600 con 0x7ff9140ccb10 2024-09-16T15:46:53.593 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:53.588+0000 7ff90bfff640 1 -- 172.21.15.123:0/3065002021 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9040b2040 con 0x7ff9140ccb10 2024-09-16T15:46:53.593 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:53.588+0000 7ff919b4d640 1 -- 172.21.15.123:0/3065002021 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff9141f5bc0 con 0x7ff9140ccb10 2024-09-16T15:46:53.594 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:53.588+0000 7ff919b4d640 1 -- 172.21.15.123:0/3065002021 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff9141f5e00 con 0x7ff9140ccb10 2024-09-16T15:46:53.594 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:53.588+0000 7ff90bfff640 1 -- 172.21.15.123:0/3065002021 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 11) ==== 97386+0+0 (secure 0 0 0) 0x7ff9040bf070 con 0x7ff9140ccb10 2024-09-16T15:46:53.594 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:53.588+0000 7ff90bfff640 1 --2- 172.21.15.123:0/3065002021 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7ff8f4073220 0x7ff8f40756e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:46:53.594 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:53.588+0000 7ff90bfff640 1 -- 172.21.15.123:0/3065002021 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(61..61 src has 1..61) ==== 4943+0+0 (secure 0 0 0) 0x7ff90412dbe0 con 0x7ff9140ccb10 2024-09-16T15:46:53.594 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:53.588+0000 7ff9137fe640 1 --2- 172.21.15.123:0/3065002021 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7ff8f4073220 0x7ff8f40756e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:46:53.594 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:53.588+0000 7ff9137fe640 1 --2- 172.21.15.123:0/3065002021 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7ff8f4073220 0x7ff8f40756e0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7ff8fc0058d0 tx=0x7ff8fc005860 comp rx=0 tx=0).ready entity=mgr.4135 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:46:53.597 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:53.592+0000 7ff90bfff640 1 -- 172.21.15.123:0/3065002021 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7ff904016410 con 0x7ff9140ccb10 2024-09-16T15:46:53.789 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:53.784+0000 7ff919b4d640 1 -- 172.21.15.123:0/3065002021 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff9140d0280 con 0x7ff9140ccb10 2024-09-16T15:46:53.790 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:53.784+0000 7ff90bfff640 1 -- 172.21.15.123:0/3065002021 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 39 v39) ==== 94+0+3920 (secure 0 0 0) 0x7ff9040f8ba0 con 0x7ff9140ccb10 2024-09-16T15:46:53.790 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-09-16T15:46:53.790 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":39,"btime":"2024-09-16T15:46:47:259834+0000","default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4901,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6835/4036392311","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":4036392311},{"type":"v1","addr":"172.21.15.184:6835","nonce":4036392311}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":31},{"gid":5203,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6837/436207465","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6836","nonce":436207465},{"type":"v1","addr":"172.21.15.184:6837","nonce":436207465}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":31},{"gid":5796,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6834/4239187724","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6832","nonce":4239187724},{"type":"v1","addr":"172.21.15.123:6834","nonce":4239187724}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":31},{"gid":6117,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6835/1152745726","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6833","nonce":1152745726},{"type":"v1","addr":"172.21.15.123:6835","nonce":1152745726}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":39}],"filesystems":[]} 2024-09-16T15:46:53.790 INFO:teuthology.orchestra.run.smithi123.stderr:dumped fsmap epoch 39 2024-09-16T15:46:53.792 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:53.788+0000 7ff919b4d640 1 -- 172.21.15.123:0/3065002021 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7ff8f4073220 msgr2=0x7ff8f40756e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:46:53.792 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:53.788+0000 7ff919b4d640 1 --2- 172.21.15.123:0/3065002021 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7ff8f4073220 0x7ff8f40756e0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7ff8fc0058d0 tx=0x7ff8fc005860 comp rx=0 tx=0).stop 2024-09-16T15:46:53.792 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:53.788+0000 7ff919b4d640 1 -- 172.21.15.123:0/3065002021 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff9140ccb10 msgr2=0x7ff914071880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:46:53.792 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:53.788+0000 7ff919b4d640 1 --2- 172.21.15.123:0/3065002021 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff9140ccb10 0x7ff914071880 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7ff904009790 tx=0x7ff9040a5670 comp rx=0 tx=0).stop 2024-09-16T15:46:53.792 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:53.788+0000 7ff919b4d640 1 -- 172.21.15.123:0/3065002021 shutdown_connections 2024-09-16T15:46:53.792 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:53.788+0000 7ff919b4d640 1 --2- 172.21.15.123:0/3065002021 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7ff8f4073220 0x7ff8f40756e0 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:53.793 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:53.788+0000 7ff919b4d640 1 --2- 172.21.15.123:0/3065002021 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7ff9140d07c0 0x7ff914071dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:53.793 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:53.788+0000 7ff919b4d640 1 --2- 172.21.15.123:0/3065002021 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff9140ccb10 0x7ff914071880 unknown :-1 s=CLOSED pgs=743 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:53.793 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:53.788+0000 7ff919b4d640 1 --2- 172.21.15.123:0/3065002021 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff9140b0d60 0x7ff914148b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:53.793 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:53.788+0000 7ff919b4d640 1 -- 172.21.15.123:0/3065002021 >> 172.21.15.123:0/3065002021 conn(0x7ff9140d1ca0 msgr2=0x7ff9140ccef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:46:53.793 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:53.788+0000 7ff919b4d640 1 -- 172.21.15.123:0/3065002021 shutdown_connections 2024-09-16T15:46:53.793 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:53.788+0000 7ff919b4d640 1 -- 172.21.15.123:0/3065002021 wait complete. 2024-09-16T15:46:53.807 INFO:tasks.ceph.mds.a:Restarting daemon 2024-09-16T15:46:53.808 INFO:tasks.ceph.mds.a:Stopping old one... 2024-09-16T15:46:53.808 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-09-16T15:46:53.808 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T15:46:53.808 INFO:tasks.ceph.mds.c:Restarting daemon 2024-09-16T15:46:53.808 INFO:tasks.ceph.mds.c:Stopping old one... 2024-09-16T15:46:53.808 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-09-16T15:46:53.808 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T15:46:53.809 INFO:tasks.ceph.mds.b:Restarting daemon 2024-09-16T15:46:53.809 INFO:tasks.ceph.mds.b:Stopping old one... 2024-09-16T15:46:53.809 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-09-16T15:46:53.809 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T15:46:53.809 INFO:tasks.ceph.mds.d:Restarting daemon 2024-09-16T15:46:53.809 INFO:tasks.ceph.mds.d:Stopping old one... 2024-09-16T15:46:53.809 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-09-16T15:46:53.810 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T15:46:53.816 INFO:tasks.ceph.mds.a:Stopped 2024-09-16T15:46:53.817 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-09-16T15:46:53.818 INFO:tasks.ceph.mds.c:Stopped 2024-09-16T15:46:53.818 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-09-16T15:46:53.832 INFO:tasks.ceph.mds.b:Stopped 2024-09-16T15:46:53.832 DEBUG:teuthology.orchestra.run.smithi184:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-09-16T15:46:53.833 INFO:tasks.ceph.mds.d:Stopped 2024-09-16T15:46:53.833 DEBUG:teuthology.orchestra.run.smithi184:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-09-16T15:46:53.859 INFO:tasks.ceph.mds.a:Started 2024-09-16T15:46:53.860 DEBUG:teuthology.parallel:result is None 2024-09-16T15:46:53.860 INFO:tasks.ceph.mds.c:Started 2024-09-16T15:46:53.860 DEBUG:teuthology.parallel:result is None 2024-09-16T15:46:53.875 INFO:tasks.ceph.mds.b:Started 2024-09-16T15:46:53.876 DEBUG:teuthology.parallel:result is None 2024-09-16T15:46:53.877 INFO:tasks.ceph.mds.d:Started 2024-09-16T15:46:53.877 DEBUG:teuthology.parallel:result is None 2024-09-16T15:46:53.877 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-09-16T15:46:53.918 INFO:tasks.ceph.mds.c.smithi123.stdout:starting mds.c at 2024-09-16T15:46:53.922 INFO:tasks.ceph.mds.a.smithi123.stdout:starting mds.a at 2024-09-16T15:46:53.949 INFO:tasks.ceph.mds.d.smithi184.stdout:starting mds.d at 2024-09-16T15:46:53.949 INFO:tasks.ceph.mds.b.smithi184.stdout:starting mds.b at 2024-09-16T15:46:53.992 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:53.988+0000 7f872b5c4640 1 -- 172.21.15.123:0/2289132786 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f87240d2150 msgr2=0x7f87240ccab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:46:53.992 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:53.988+0000 7f872b5c4640 1 --2- 172.21.15.123:0/2289132786 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f87240d2150 0x7f87240ccab0 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 crypto rx=0x7f87240cc2a0 tx=0x7f871c01ccd0 comp rx=0 tx=0).stop 2024-09-16T15:46:53.992 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:53.988+0000 7f872b5c4640 1 -- 172.21.15.123:0/2289132786 shutdown_connections 2024-09-16T15:46:53.992 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:53.988+0000 7f872b5c4640 1 --2- 172.21.15.123:0/2289132786 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f87240d2150 0x7f87240ccab0 unknown :-1 s=CLOSED pgs=750 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:53.992 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:53.988+0000 7f872b5c4640 1 --2- 172.21.15.123:0/2289132786 >> v2:172.21.15.184:3300/0 conn(0x7f87240b25a0 0x7f87240d1c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:53.992 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:53.988+0000 7f872b5c4640 1 -- 172.21.15.123:0/2289132786 >> 172.21.15.123:0/2289132786 conn(0x7f8724074bd0 msgr2=0x7f872409aa40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:46:53.992 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:53.988+0000 7f872b5c4640 1 -- 172.21.15.123:0/2289132786 shutdown_connections 2024-09-16T15:46:53.992 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:53.988+0000 7f872b5c4640 1 -- 172.21.15.123:0/2289132786 wait complete. 2024-09-16T15:46:53.995 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:53.988+0000 7f872b5c4640 1 Processor -- start 2024-09-16T15:46:53.995 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:53.988+0000 7f872b5c4640 1 -- start start 2024-09-16T15:46:53.995 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:53.988+0000 7f872b5c4640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f87240b25a0 0x7f872414d6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:46:53.995 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:53.988+0000 7f872b5c4640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f87240d2150 0x7f872415ade0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:46:53.995 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:53.988+0000 7f872b5c4640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f872415bda0 0x7f872415b320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:46:53.995 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:53.988+0000 7f872b5c4640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f87241689f0 con 0x7f872415bda0 2024-09-16T15:46:53.995 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:53.988+0000 7f872b5c4640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f8724168870 con 0x7f87240b25a0 2024-09-16T15:46:53.995 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:53.988+0000 7f872b5c4640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f8724168b70 con 0x7f87240d2150 2024-09-16T15:46:53.996 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:53.988+0000 7f8729b61640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f872415bda0 0x7f872415b320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:46:53.996 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:53.988+0000 7f8729b61640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f872415bda0 0x7f872415b320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:40054/0 (socket says 172.21.15.123:40054) 2024-09-16T15:46:53.996 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:53.988+0000 7f8729b61640 1 -- 172.21.15.123:0/3820614174 learned_addr learned my addr 172.21.15.123:0/3820614174 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:46:53.996 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:53.988+0000 7f8728b5f640 1 --2- 172.21.15.123:0/3820614174 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f87240d2150 0x7f872415ade0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:46:53.996 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:53.988+0000 7f8729360640 1 --2- 172.21.15.123:0/3820614174 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f87240b25a0 0x7f872414d6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:46:53.996 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:53.988+0000 7f8729b61640 1 -- 172.21.15.123:0/3820614174 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f87240d2150 msgr2=0x7f872415ade0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:46:53.996 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:53.988+0000 7f8729b61640 1 --2- 172.21.15.123:0/3820614174 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f87240d2150 0x7f872415ade0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:53.996 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:53.988+0000 7f8729b61640 1 -- 172.21.15.123:0/3820614174 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f87240b25a0 msgr2=0x7f872414d6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:46:53.996 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:53.988+0000 7f8729b61640 1 --2- 172.21.15.123:0/3820614174 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f87240b25a0 0x7f872414d6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:53.996 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:53.988+0000 7f8729b61640 1 -- 172.21.15.123:0/3820614174 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f872406f790 con 0x7f872415bda0 2024-09-16T15:46:53.996 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:53.988+0000 7f8729b61640 1 --2- 172.21.15.123:0/3820614174 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f872415bda0 0x7f872415b320 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7f871c01b380 tx=0x7f871c0a5960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:46:53.996 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:53.988+0000 7f8719ffb640 1 -- 172.21.15.123:0/3820614174 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f871c0bb070 con 0x7f872415bda0 2024-09-16T15:46:53.997 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:53.988+0000 7f8719ffb640 1 -- 172.21.15.123:0/3820614174 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f871c004b40 con 0x7f872415bda0 2024-09-16T15:46:53.997 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:53.988+0000 7f872b5c4640 1 -- 172.21.15.123:0/3820614174 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f872406ebb0 con 0x7f872415bda0 2024-09-16T15:46:53.997 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:53.988+0000 7f8719ffb640 1 -- 172.21.15.123:0/3820614174 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f871c0b2040 con 0x7f872415bda0 2024-09-16T15:46:53.997 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:53.988+0000 7f872b5c4640 1 -- 172.21.15.123:0/3820614174 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f87241593a0 con 0x7f872415bda0 2024-09-16T15:46:53.997 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:53.992+0000 7f872b5c4640 1 -- 172.21.15.123:0/3820614174 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f87240d1c10 con 0x7f872415bda0 2024-09-16T15:46:53.998 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:53.992+0000 7f8719ffb640 1 -- 172.21.15.123:0/3820614174 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 11) ==== 97386+0+0 (secure 0 0 0) 0x7f871c0040c0 con 0x7f872415bda0 2024-09-16T15:46:53.998 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:53.992+0000 7f8719ffb640 1 --2- 172.21.15.123:0/3820614174 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f8708073220 0x7f87080756e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:46:53.998 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:53.992+0000 7f8719ffb640 1 -- 172.21.15.123:0/3820614174 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(61..61 src has 1..61) ==== 4943+0+0 (secure 0 0 0) 0x7f871c12dee0 con 0x7f872415bda0 2024-09-16T15:46:53.998 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:53.992+0000 7f8729360640 1 --2- 172.21.15.123:0/3820614174 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f8708073220 0x7f87080756e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:46:53.998 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:53.992+0000 7f8729360640 1 --2- 172.21.15.123:0/3820614174 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f8708073220 0x7f87080756e0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f871400b810 tx=0x7f8714008040 comp rx=0 tx=0).ready entity=mgr.4135 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:46:54.000 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:53.996+0000 7f8719ffb640 1 -- 172.21.15.123:0/3820614174 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f871c016950 con 0x7f872415bda0 2024-09-16T15:46:54.174 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:54.168+0000 7f872b5c4640 1 -- 172.21.15.123:0/3820614174 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) -- 0x7f8724159660 con 0x7f872415bda0 2024-09-16T15:46:55.216 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:55.212+0000 7f8719ffb640 1 -- 172.21.15.123:0/3820614174 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v66) ==== 97+0+0 (secure 0 0 0) 0x7f871c0f8890 con 0x7f872415bda0 2024-09-16T15:46:55.216 INFO:teuthology.orchestra.run.smithi123.stderr: removed all blocklist entries 2024-09-16T15:46:55.219 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:55.216+0000 7f872b5c4640 1 -- 172.21.15.123:0/3820614174 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f8708073220 msgr2=0x7f87080756e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:46:55.219 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:55.216+0000 7f872b5c4640 1 --2- 172.21.15.123:0/3820614174 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f8708073220 0x7f87080756e0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f871400b810 tx=0x7f8714008040 comp rx=0 tx=0).stop 2024-09-16T15:46:55.219 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:55.216+0000 7f872b5c4640 1 -- 172.21.15.123:0/3820614174 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f872415bda0 msgr2=0x7f872415b320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:46:55.219 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:55.216+0000 7f872b5c4640 1 --2- 172.21.15.123:0/3820614174 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f872415bda0 0x7f872415b320 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7f871c01b380 tx=0x7f871c0a5960 comp rx=0 tx=0).stop 2024-09-16T15:46:55.220 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:55.216+0000 7f872b5c4640 1 -- 172.21.15.123:0/3820614174 shutdown_connections 2024-09-16T15:46:55.220 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:55.216+0000 7f872b5c4640 1 --2- 172.21.15.123:0/3820614174 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f8708073220 0x7f87080756e0 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:55.220 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:55.216+0000 7f872b5c4640 1 --2- 172.21.15.123:0/3820614174 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f872415bda0 0x7f872415b320 unknown :-1 s=CLOSED pgs=751 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:55.220 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:55.216+0000 7f872b5c4640 1 --2- 172.21.15.123:0/3820614174 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f87240d2150 0x7f872415ade0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:55.220 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:55.216+0000 7f872b5c4640 1 --2- 172.21.15.123:0/3820614174 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f87240b25a0 0x7f872414d6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:55.220 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:55.216+0000 7f872b5c4640 1 -- 172.21.15.123:0/3820614174 >> 172.21.15.123:0/3820614174 conn(0x7f8724074bd0 msgr2=0x7f87240cea60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:46:55.220 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:55.216+0000 7f872b5c4640 1 -- 172.21.15.123:0/3820614174 shutdown_connections 2024-09-16T15:46:55.220 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:55.216+0000 7f872b5c4640 1 -- 172.21.15.123:0/3820614174 wait complete. 2024-09-16T15:46:55.240 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-09-16T15:46:55.412 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:55.408+0000 7f4ca3f0c640 1 -- 172.21.15.123:0/1491154650 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4c9c0726c0 msgr2=0x7f4c9c073ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:46:55.412 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:55.408+0000 7f4ca3f0c640 1 --2- 172.21.15.123:0/1491154650 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4c9c0726c0 0x7f4c9c073ef0 secure :-1 s=READY pgs=753 cs=0 l=1 rev1=1 crypto rx=0x7f4c9c0cd650 tx=0x7f4c8c01c740 comp rx=0 tx=0).stop 2024-09-16T15:46:55.412 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:55.408+0000 7f4ca3f0c640 1 -- 172.21.15.123:0/1491154650 shutdown_connections 2024-09-16T15:46:55.412 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:55.408+0000 7f4ca3f0c640 1 --2- 172.21.15.123:0/1491154650 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4c9c0726c0 0x7f4c9c073ef0 unknown :-1 s=CLOSED pgs=753 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:55.412 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:55.408+0000 7f4ca3f0c640 1 --2- 172.21.15.123:0/1491154650 >> v2:172.21.15.184:3300/0 conn(0x7f4c9c15bca0 0x7f4c9c0d0630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:55.412 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:55.408+0000 7f4ca3f0c640 1 -- 172.21.15.123:0/1491154650 >> 172.21.15.123:0/1491154650 conn(0x7f4c9c0cfb10 msgr2=0x7f4c9c0cff20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:46:55.412 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:55.408+0000 7f4ca3f0c640 1 -- 172.21.15.123:0/1491154650 shutdown_connections 2024-09-16T15:46:55.412 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:55.408+0000 7f4ca3f0c640 1 -- 172.21.15.123:0/1491154650 wait complete. 2024-09-16T15:46:55.415 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:55.408+0000 7f4ca3f0c640 1 Processor -- start 2024-09-16T15:46:55.415 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:55.408+0000 7f4ca3f0c640 1 -- start start 2024-09-16T15:46:55.415 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:55.408+0000 7f4ca3f0c640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f4c9c0726c0 0x7f4c9c152950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:46:55.415 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:55.408+0000 7f4ca3f0c640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4c9c074430 0x7f4c9c151d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:46:55.415 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:55.408+0000 7f4ca3f0c640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4c9c15bca0 0x7f4c9c15a060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:46:55.415 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:55.408+0000 7f4ca3f0c640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f4c9c168e30 con 0x7f4c9c074430 2024-09-16T15:46:55.415 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:55.408+0000 7f4ca3f0c640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f4c9c168cb0 con 0x7f4c9c15bca0 2024-09-16T15:46:55.415 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:55.408+0000 7f4ca3f0c640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f4c9c168fb0 con 0x7f4c9c0726c0 2024-09-16T15:46:55.416 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:55.408+0000 7f4ca14a7640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4c9c074430 0x7f4c9c151d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:46:55.416 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:55.408+0000 7f4ca14a7640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4c9c074430 0x7f4c9c151d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:40076/0 (socket says 172.21.15.123:40076) 2024-09-16T15:46:55.416 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:55.408+0000 7f4ca14a7640 1 -- 172.21.15.123:0/525406277 learned_addr learned my addr 172.21.15.123:0/525406277 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:46:55.416 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:55.408+0000 7f4ca1ca8640 1 --2- 172.21.15.123:0/525406277 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f4c9c0726c0 0x7f4c9c152950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:46:55.416 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:55.408+0000 7f4ca14a7640 1 -- 172.21.15.123:0/525406277 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f4c9c0726c0 msgr2=0x7f4c9c152950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:46:55.416 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:55.408+0000 7f4ca24a9640 1 --2- 172.21.15.123:0/525406277 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4c9c15bca0 0x7f4c9c15a060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:46:55.416 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:55.408+0000 7f4ca14a7640 1 --2- 172.21.15.123:0/525406277 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f4c9c0726c0 0x7f4c9c152950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:55.416 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:55.408+0000 7f4ca14a7640 1 -- 172.21.15.123:0/525406277 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4c9c15bca0 msgr2=0x7f4c9c15a060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:46:55.416 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:55.408+0000 7f4ca14a7640 1 --2- 172.21.15.123:0/525406277 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4c9c15bca0 0x7f4c9c15a060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:55.416 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:55.408+0000 7f4ca14a7640 1 -- 172.21.15.123:0/525406277 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4c9c15d7d0 con 0x7f4c9c074430 2024-09-16T15:46:55.416 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:55.408+0000 7f4ca1ca8640 1 --2- 172.21.15.123:0/525406277 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f4c9c0726c0 0x7f4c9c152950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T15:46:55.417 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:55.408+0000 7f4ca24a9640 1 --2- 172.21.15.123:0/525406277 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4c9c15bca0 0x7f4c9c15a060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T15:46:55.417 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:55.408+0000 7f4ca14a7640 1 --2- 172.21.15.123:0/525406277 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4c9c074430 0x7f4c9c151d00 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 crypto rx=0x7f4c8c01cc20 tx=0x7f4c8c0a5680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:46:55.417 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:55.408+0000 7f4c927fc640 1 -- 172.21.15.123:0/525406277 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c8c0b2040 con 0x7f4c9c074430 2024-09-16T15:46:55.417 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:55.408+0000 7f4c927fc640 1 -- 172.21.15.123:0/525406277 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f4c8c0041b0 con 0x7f4c9c074430 2024-09-16T15:46:55.417 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:55.408+0000 7f4ca3f0c640 1 -- 172.21.15.123:0/525406277 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4c9c15efe0 con 0x7f4c9c074430 2024-09-16T15:46:55.417 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:55.408+0000 7f4c927fc640 1 -- 172.21.15.123:0/525406277 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c8c003600 con 0x7f4c9c074430 2024-09-16T15:46:55.417 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:55.408+0000 7f4ca3f0c640 1 -- 172.21.15.123:0/525406277 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4c9c15cfd0 con 0x7f4c9c074430 2024-09-16T15:46:55.417 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:55.412+0000 7f4ca3f0c640 1 -- 172.21.15.123:0/525406277 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4c9c073ef0 con 0x7f4c9c074430 2024-09-16T15:46:55.418 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:55.412+0000 7f4c927fc640 1 -- 172.21.15.123:0/525406277 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 11) ==== 97386+0+0 (secure 0 0 0) 0x7f4c8c0bd030 con 0x7f4c9c074430 2024-09-16T15:46:55.418 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:55.412+0000 7f4c927fc640 1 --2- 172.21.15.123:0/525406277 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f4c80073250 0x7f4c80075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:46:55.418 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:55.412+0000 7f4c927fc640 1 -- 172.21.15.123:0/525406277 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(66..66 src has 1..66) ==== 4857+0+0 (secure 0 0 0) 0x7f4c8c12de00 con 0x7f4c9c074430 2024-09-16T15:46:55.418 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:55.412+0000 7f4ca1ca8640 1 --2- 172.21.15.123:0/525406277 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f4c80073250 0x7f4c80075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:46:55.418 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:55.412+0000 7f4ca1ca8640 1 --2- 172.21.15.123:0/525406277 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f4c80073250 0x7f4c80075710 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f4c9c165c10 tx=0x7f4c98005d20 comp rx=0 tx=0).ready entity=mgr.4135 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:46:55.421 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:55.416+0000 7f4c927fc640 1 -- 172.21.15.123:0/525406277 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f4c8c0c4060 con 0x7f4c9c074430 2024-09-16T15:46:55.652 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:55.648+0000 7f4ca3f0c640 1 -- 172.21.15.123:0/525406277 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) -- 0x7f4c9c15f630 con 0x7f4c9c074430 2024-09-16T15:46:55.653 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:55.648+0000 7f4c927fc640 1 -- 172.21.15.123:0/525406277 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v20) ==== 80+0+5617 (secure 0 0 0) 0x7f4c8c0163f0 con 0x7f4c9c074430 2024-09-16T15:46:55.653 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-09-16T15:46:55.653 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2024-09-16T15:46:55.653 INFO:teuthology.orchestra.run.smithi123.stdout: "auth_dump": [ 2024-09-16T15:46:55.653 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-09-16T15:46:55.654 INFO:teuthology.orchestra.run.smithi123.stdout: "entity": "mds.a", 2024-09-16T15:46:55.654 INFO:teuthology.orchestra.run.smithi123.stdout: "key": "AQC0UehmbZpOJBAAaYGyDcWrt4k0FS5P/bb+4A==", 2024-09-16T15:46:55.654 INFO:teuthology.orchestra.run.smithi123.stdout: "caps": { 2024-09-16T15:46:55.654 INFO:teuthology.orchestra.run.smithi123.stdout: "mds": "allow", 2024-09-16T15:46:55.654 INFO:teuthology.orchestra.run.smithi123.stdout: "mgr": "allow *", 2024-09-16T15:46:55.654 INFO:teuthology.orchestra.run.smithi123.stdout: "mon": "allow *", 2024-09-16T15:46:55.654 INFO:teuthology.orchestra.run.smithi123.stdout: "osd": "allow *" 2024-09-16T15:46:55.654 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-16T15:46:55.654 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:46:55.654 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-09-16T15:46:55.655 INFO:teuthology.orchestra.run.smithi123.stdout: "entity": "mds.b", 2024-09-16T15:46:55.655 INFO:teuthology.orchestra.run.smithi123.stdout: "key": "AQC0UehmF1HFMhAADcvqXNKdHZMPAbLc+se4Aw==", 2024-09-16T15:46:55.655 INFO:teuthology.orchestra.run.smithi123.stdout: "caps": { 2024-09-16T15:46:55.655 INFO:teuthology.orchestra.run.smithi123.stdout: "mds": "allow", 2024-09-16T15:46:55.655 INFO:teuthology.orchestra.run.smithi123.stdout: "mgr": "allow *", 2024-09-16T15:46:55.655 INFO:teuthology.orchestra.run.smithi123.stdout: "mon": "allow *", 2024-09-16T15:46:55.655 INFO:teuthology.orchestra.run.smithi123.stdout: "osd": "allow *" 2024-09-16T15:46:55.655 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-16T15:46:55.655 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:46:55.655 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-09-16T15:46:55.655 INFO:teuthology.orchestra.run.smithi123.stdout: "entity": "mds.c", 2024-09-16T15:46:55.655 INFO:teuthology.orchestra.run.smithi123.stdout: "key": "AQC0UehmCBjCLBAAXd85J25r7cgrXeDCG7pahg==", 2024-09-16T15:46:55.656 INFO:teuthology.orchestra.run.smithi123.stdout: "caps": { 2024-09-16T15:46:55.656 INFO:teuthology.orchestra.run.smithi123.stdout: "mds": "allow", 2024-09-16T15:46:55.656 INFO:teuthology.orchestra.run.smithi123.stdout: "mgr": "allow *", 2024-09-16T15:46:55.656 INFO:teuthology.orchestra.run.smithi123.stdout: "mon": "allow *", 2024-09-16T15:46:55.656 INFO:teuthology.orchestra.run.smithi123.stdout: "osd": "allow *" 2024-09-16T15:46:55.656 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-16T15:46:55.656 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:46:55.656 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-09-16T15:46:55.656 INFO:teuthology.orchestra.run.smithi123.stdout: "entity": "mds.d", 2024-09-16T15:46:55.656 INFO:teuthology.orchestra.run.smithi123.stdout: "key": "AQC0UehmMh3ZOBAAx0RCa3R9RMT93/icZhPGyg==", 2024-09-16T15:46:55.656 INFO:teuthology.orchestra.run.smithi123.stdout: "caps": { 2024-09-16T15:46:55.656 INFO:teuthology.orchestra.run.smithi123.stdout: "mds": "allow", 2024-09-16T15:46:55.657 INFO:teuthology.orchestra.run.smithi123.stdout: "mgr": "allow *", 2024-09-16T15:46:55.657 INFO:teuthology.orchestra.run.smithi123.stdout: "mon": "allow *", 2024-09-16T15:46:55.657 INFO:teuthology.orchestra.run.smithi123.stdout: "osd": "allow *" 2024-09-16T15:46:55.657 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-16T15:46:55.657 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:46:55.657 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-09-16T15:46:55.657 INFO:teuthology.orchestra.run.smithi123.stdout: "entity": "osd.0", 2024-09-16T15:46:55.657 INFO:teuthology.orchestra.run.smithi123.stdout: "key": "AQC4UehmT633CRAAIhpkeIBOzwJO0kiXLu0x9Q==", 2024-09-16T15:46:55.657 INFO:teuthology.orchestra.run.smithi123.stdout: "caps": { 2024-09-16T15:46:55.657 INFO:teuthology.orchestra.run.smithi123.stdout: "mgr": "allow profile osd", 2024-09-16T15:46:55.657 INFO:teuthology.orchestra.run.smithi123.stdout: "mon": "allow profile osd", 2024-09-16T15:46:55.657 INFO:teuthology.orchestra.run.smithi123.stdout: "osd": "allow *" 2024-09-16T15:46:55.658 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-16T15:46:55.658 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:46:55.658 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-09-16T15:46:55.658 INFO:teuthology.orchestra.run.smithi123.stdout: "entity": "osd.1", 2024-09-16T15:46:55.658 INFO:teuthology.orchestra.run.smithi123.stdout: "key": "AQC7UehmFwF7GBAAvOUG+rWjANKT8wm/ai4xGg==", 2024-09-16T15:46:55.658 INFO:teuthology.orchestra.run.smithi123.stdout: "caps": { 2024-09-16T15:46:55.658 INFO:teuthology.orchestra.run.smithi123.stdout: "mgr": "allow profile osd", 2024-09-16T15:46:55.658 INFO:teuthology.orchestra.run.smithi123.stdout: "mon": "allow profile osd", 2024-09-16T15:46:55.658 INFO:teuthology.orchestra.run.smithi123.stdout: "osd": "allow *" 2024-09-16T15:46:55.658 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-16T15:46:55.658 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:46:55.658 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-09-16T15:46:55.659 INFO:teuthology.orchestra.run.smithi123.stdout: "entity": "osd.2", 2024-09-16T15:46:55.659 INFO:teuthology.orchestra.run.smithi123.stdout: "key": "AQC+UehmlCwWKxAAl3iO7ZALWyaorQH2urHKCw==", 2024-09-16T15:46:55.659 INFO:teuthology.orchestra.run.smithi123.stdout: "caps": { 2024-09-16T15:46:55.659 INFO:teuthology.orchestra.run.smithi123.stdout: "mgr": "allow profile osd", 2024-09-16T15:46:55.659 INFO:teuthology.orchestra.run.smithi123.stdout: "mon": "allow profile osd", 2024-09-16T15:46:55.659 INFO:teuthology.orchestra.run.smithi123.stdout: "osd": "allow *" 2024-09-16T15:46:55.659 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-16T15:46:55.659 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:46:55.659 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-09-16T15:46:55.659 INFO:teuthology.orchestra.run.smithi123.stdout: "entity": "osd.3", 2024-09-16T15:46:55.659 INFO:teuthology.orchestra.run.smithi123.stdout: "key": "AQDCUehmD8VgAhAAneoIeGr3MflSmi+nyavj2A==", 2024-09-16T15:46:55.659 INFO:teuthology.orchestra.run.smithi123.stdout: "caps": { 2024-09-16T15:46:55.659 INFO:teuthology.orchestra.run.smithi123.stdout: "mgr": "allow profile osd", 2024-09-16T15:46:55.660 INFO:teuthology.orchestra.run.smithi123.stdout: "mon": "allow profile osd", 2024-09-16T15:46:55.660 INFO:teuthology.orchestra.run.smithi123.stdout: "osd": "allow *" 2024-09-16T15:46:55.660 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-16T15:46:55.660 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:46:55.660 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-09-16T15:46:55.660 INFO:teuthology.orchestra.run.smithi123.stdout: "entity": "osd.4", 2024-09-16T15:46:55.660 INFO:teuthology.orchestra.run.smithi123.stdout: "key": "AQDIUehmA9B9EBAAsu0X5LPx1TmatZeHzjnVig==", 2024-09-16T15:46:55.660 INFO:teuthology.orchestra.run.smithi123.stdout: "caps": { 2024-09-16T15:46:55.660 INFO:teuthology.orchestra.run.smithi123.stdout: "mgr": "allow profile osd", 2024-09-16T15:46:55.660 INFO:teuthology.orchestra.run.smithi123.stdout: "mon": "allow profile osd", 2024-09-16T15:46:55.660 INFO:teuthology.orchestra.run.smithi123.stdout: "osd": "allow *" 2024-09-16T15:46:55.660 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-16T15:46:55.660 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:46:55.661 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-09-16T15:46:55.661 INFO:teuthology.orchestra.run.smithi123.stdout: "entity": "osd.5", 2024-09-16T15:46:55.661 INFO:teuthology.orchestra.run.smithi123.stdout: "key": "AQDLUehmcugdHxAAQMSS1ZH71FUzCx5A+yNnZg==", 2024-09-16T15:46:55.661 INFO:teuthology.orchestra.run.smithi123.stdout: "caps": { 2024-09-16T15:46:55.661 INFO:teuthology.orchestra.run.smithi123.stdout: "mgr": "allow profile osd", 2024-09-16T15:46:55.661 INFO:teuthology.orchestra.run.smithi123.stdout: "mon": "allow profile osd", 2024-09-16T15:46:55.661 INFO:teuthology.orchestra.run.smithi123.stdout: "osd": "allow *" 2024-09-16T15:46:55.661 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-16T15:46:55.661 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:46:55.661 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-09-16T15:46:55.661 INFO:teuthology.orchestra.run.smithi123.stdout: "entity": "osd.6", 2024-09-16T15:46:55.661 INFO:teuthology.orchestra.run.smithi123.stdout: "key": "AQDOUehmWLwYLhAAhOlQLepcWElsVPMnHlZm4w==", 2024-09-16T15:46:55.662 INFO:teuthology.orchestra.run.smithi123.stdout: "caps": { 2024-09-16T15:46:55.662 INFO:teuthology.orchestra.run.smithi123.stdout: "mgr": "allow profile osd", 2024-09-16T15:46:55.662 INFO:teuthology.orchestra.run.smithi123.stdout: "mon": "allow profile osd", 2024-09-16T15:46:55.662 INFO:teuthology.orchestra.run.smithi123.stdout: "osd": "allow *" 2024-09-16T15:46:55.662 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-16T15:46:55.662 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:46:55.662 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-09-16T15:46:55.662 INFO:teuthology.orchestra.run.smithi123.stdout: "entity": "osd.7", 2024-09-16T15:46:55.662 INFO:teuthology.orchestra.run.smithi123.stdout: "key": "AQDSUehmipkbABAAQuL/2O7w0tsM1OKeOil97w==", 2024-09-16T15:46:55.662 INFO:teuthology.orchestra.run.smithi123.stdout: "caps": { 2024-09-16T15:46:55.662 INFO:teuthology.orchestra.run.smithi123.stdout: "mgr": "allow profile osd", 2024-09-16T15:46:55.662 INFO:teuthology.orchestra.run.smithi123.stdout: "mon": "allow profile osd", 2024-09-16T15:46:55.663 INFO:teuthology.orchestra.run.smithi123.stdout: "osd": "allow *" 2024-09-16T15:46:55.663 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-16T15:46:55.663 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:46:55.663 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-09-16T15:46:55.663 INFO:teuthology.orchestra.run.smithi123.stdout: "entity": "client.0", 2024-09-16T15:46:55.663 INFO:teuthology.orchestra.run.smithi123.stdout: "key": "AQC1Uehm7GDTAhAAHWWVIyjVVnTeeozBR8DfFA==", 2024-09-16T15:46:55.663 INFO:teuthology.orchestra.run.smithi123.stdout: "caps": { 2024-09-16T15:46:55.663 INFO:teuthology.orchestra.run.smithi123.stdout: "mds": "allow", 2024-09-16T15:46:55.663 INFO:teuthology.orchestra.run.smithi123.stdout: "mon": "allow r", 2024-09-16T15:46:55.663 INFO:teuthology.orchestra.run.smithi123.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-09-16T15:46:55.663 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-16T15:46:55.663 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:46:55.664 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-09-16T15:46:55.664 INFO:teuthology.orchestra.run.smithi123.stdout: "entity": "client.1", 2024-09-16T15:46:55.664 INFO:teuthology.orchestra.run.smithi123.stdout: "key": "AQC1UehmFbaWChAAFtRmKucm+pbKtCNvRCTNqA==", 2024-09-16T15:46:55.664 INFO:teuthology.orchestra.run.smithi123.stdout: "caps": { 2024-09-16T15:46:55.664 INFO:teuthology.orchestra.run.smithi123.stdout: "mds": "allow", 2024-09-16T15:46:55.664 INFO:teuthology.orchestra.run.smithi123.stdout: "mgr": "allow r", 2024-09-16T15:46:55.664 INFO:teuthology.orchestra.run.smithi123.stdout: "mon": "allow rw", 2024-09-16T15:46:55.664 INFO:teuthology.orchestra.run.smithi123.stdout: "osd": "allow rwx" 2024-09-16T15:46:55.664 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-16T15:46:55.664 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:46:55.664 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-09-16T15:46:55.664 INFO:teuthology.orchestra.run.smithi123.stdout: "entity": "client.2", 2024-09-16T15:46:55.664 INFO:teuthology.orchestra.run.smithi123.stdout: "key": "AQC1Uehm1ADrBxAAW/8ayNHI6Wq/YlI4ivFZiw==", 2024-09-16T15:46:55.665 INFO:teuthology.orchestra.run.smithi123.stdout: "caps": { 2024-09-16T15:46:55.665 INFO:teuthology.orchestra.run.smithi123.stdout: "mds": "allow", 2024-09-16T15:46:55.665 INFO:teuthology.orchestra.run.smithi123.stdout: "mgr": "allow r", 2024-09-16T15:46:55.665 INFO:teuthology.orchestra.run.smithi123.stdout: "mon": "allow rw", 2024-09-16T15:46:55.665 INFO:teuthology.orchestra.run.smithi123.stdout: "osd": "allow rwx" 2024-09-16T15:46:55.665 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-16T15:46:55.665 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:46:55.665 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-09-16T15:46:55.665 INFO:teuthology.orchestra.run.smithi123.stdout: "entity": "client.3", 2024-09-16T15:46:55.665 INFO:teuthology.orchestra.run.smithi123.stdout: "key": "AQC1UehmLv/3DxAAuGloBxSuza95uajpfiZ9MQ==", 2024-09-16T15:46:55.665 INFO:teuthology.orchestra.run.smithi123.stdout: "caps": { 2024-09-16T15:46:55.665 INFO:teuthology.orchestra.run.smithi123.stdout: "mds": "allow", 2024-09-16T15:46:55.666 INFO:teuthology.orchestra.run.smithi123.stdout: "mgr": "allow r", 2024-09-16T15:46:55.666 INFO:teuthology.orchestra.run.smithi123.stdout: "mon": "allow rw", 2024-09-16T15:46:55.666 INFO:teuthology.orchestra.run.smithi123.stdout: "osd": "allow rwx" 2024-09-16T15:46:55.666 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-16T15:46:55.666 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:46:55.666 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-09-16T15:46:55.666 INFO:teuthology.orchestra.run.smithi123.stdout: "entity": "client.admin", 2024-09-16T15:46:55.666 INFO:teuthology.orchestra.run.smithi123.stdout: "key": "AQC0Uehm1ZDRDBAAhyQ1BmJg1Mw6wQ04I2CbFA==", 2024-09-16T15:46:55.666 INFO:teuthology.orchestra.run.smithi123.stdout: "caps": { 2024-09-16T15:46:55.666 INFO:teuthology.orchestra.run.smithi123.stdout: "mds": "allow *", 2024-09-16T15:46:55.666 INFO:teuthology.orchestra.run.smithi123.stdout: "mgr": "allow *", 2024-09-16T15:46:55.666 INFO:teuthology.orchestra.run.smithi123.stdout: "mon": "allow *", 2024-09-16T15:46:55.666 INFO:teuthology.orchestra.run.smithi123.stdout: "osd": "allow *" 2024-09-16T15:46:55.667 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-16T15:46:55.667 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:46:55.667 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-09-16T15:46:55.667 INFO:teuthology.orchestra.run.smithi123.stdout: "entity": "client.alice", 2024-09-16T15:46:55.667 INFO:teuthology.orchestra.run.smithi123.stdout: "key": "AQDhUuhmPcuGDRAAbzuLPE1eqyta2/Ouy8g7mw==", 2024-09-16T15:46:55.667 INFO:teuthology.orchestra.run.smithi123.stdout: "caps": { 2024-09-16T15:46:55.667 INFO:teuthology.orchestra.run.smithi123.stdout: "mds": "allow rw path=/volumes/_nogroup/subvol14360", 2024-09-16T15:46:55.667 INFO:teuthology.orchestra.run.smithi123.stdout: "mon": "allow r", 2024-09-16T15:46:55.667 INFO:teuthology.orchestra.run.smithi123.stdout: "osd": "allow rw pool=cephfs_data" 2024-09-16T15:46:55.667 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-16T15:46:55.667 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:46:55.667 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-09-16T15:46:55.668 INFO:teuthology.orchestra.run.smithi123.stdout: "entity": "mgr.x", 2024-09-16T15:46:55.668 INFO:teuthology.orchestra.run.smithi123.stdout: "key": "AQC0UehmfPilIBAAERlBbduOgAU9ba0n/1tl3g==", 2024-09-16T15:46:55.668 INFO:teuthology.orchestra.run.smithi123.stdout: "caps": { 2024-09-16T15:46:55.668 INFO:teuthology.orchestra.run.smithi123.stdout: "mds": "allow *", 2024-09-16T15:46:55.668 INFO:teuthology.orchestra.run.smithi123.stdout: "mon": "allow profile mgr", 2024-09-16T15:46:55.668 INFO:teuthology.orchestra.run.smithi123.stdout: "osd": "allow *" 2024-09-16T15:46:55.668 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-16T15:46:55.668 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:46:55.668 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-09-16T15:46:55.668 INFO:teuthology.orchestra.run.smithi123.stdout: "entity": "mgr.y", 2024-09-16T15:46:55.668 INFO:teuthology.orchestra.run.smithi123.stdout: "key": "AQC0UehmkjrlHRAA+3xbQC+SPWGWW1gND7xcAg==", 2024-09-16T15:46:55.668 INFO:teuthology.orchestra.run.smithi123.stdout: "caps": { 2024-09-16T15:46:55.668 INFO:teuthology.orchestra.run.smithi123.stdout: "mds": "allow *", 2024-09-16T15:46:55.669 INFO:teuthology.orchestra.run.smithi123.stdout: "mon": "allow profile mgr", 2024-09-16T15:46:55.669 INFO:teuthology.orchestra.run.smithi123.stdout: "osd": "allow *" 2024-09-16T15:46:55.669 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-16T15:46:55.669 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-16T15:46:55.669 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-09-16T15:46:55.669 INFO:teuthology.orchestra.run.smithi123.stdout:} 2024-09-16T15:46:55.670 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:55.652+0000 7f4ca3f0c640 1 -- 172.21.15.123:0/525406277 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f4c80073250 msgr2=0x7f4c80075710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:46:55.670 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:55.652+0000 7f4ca3f0c640 1 --2- 172.21.15.123:0/525406277 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f4c80073250 0x7f4c80075710 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f4c9c165c10 tx=0x7f4c98005d20 comp rx=0 tx=0).stop 2024-09-16T15:46:55.670 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:55.652+0000 7f4ca3f0c640 1 -- 172.21.15.123:0/525406277 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4c9c074430 msgr2=0x7f4c9c151d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:46:55.671 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:55.652+0000 7f4ca3f0c640 1 --2- 172.21.15.123:0/525406277 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4c9c074430 0x7f4c9c151d00 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 crypto rx=0x7f4c8c01cc20 tx=0x7f4c8c0a5680 comp rx=0 tx=0).stop 2024-09-16T15:46:55.671 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:55.652+0000 7f4ca3f0c640 1 -- 172.21.15.123:0/525406277 shutdown_connections 2024-09-16T15:46:55.671 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:55.652+0000 7f4ca3f0c640 1 --2- 172.21.15.123:0/525406277 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f4c80073250 0x7f4c80075710 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:55.671 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:55.652+0000 7f4ca3f0c640 1 --2- 172.21.15.123:0/525406277 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4c9c15bca0 0x7f4c9c15a060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:55.671 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:55.652+0000 7f4ca3f0c640 1 --2- 172.21.15.123:0/525406277 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4c9c074430 0x7f4c9c151d00 unknown :-1 s=CLOSED pgs=754 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:55.671 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:55.652+0000 7f4ca3f0c640 1 --2- 172.21.15.123:0/525406277 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f4c9c0726c0 0x7f4c9c152950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:55.671 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:55.652+0000 7f4ca3f0c640 1 -- 172.21.15.123:0/525406277 >> 172.21.15.123:0/525406277 conn(0x7f4c9c0cfb10 msgr2=0x7f4c9c06eee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:46:55.671 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:55.652+0000 7f4ca3f0c640 1 -- 172.21.15.123:0/525406277 shutdown_connections 2024-09-16T15:46:55.671 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:55.652+0000 7f4ca3f0c640 1 -- 172.21.15.123:0/525406277 wait complete. 2024-09-16T15:46:55.673 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.alice 2024-09-16T15:46:55.786 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:55.780+0000 7fd8380b3640 1 -- 172.21.15.123:0/2420315341 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd81c0064c0 msgr2=0x7fd81c00cb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:46:55.787 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:55.780+0000 7fd8380b3640 1 --2- 172.21.15.123:0/2420315341 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd81c0064c0 0x7fd81c00cb20 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 crypto rx=0x7fd828001dc0 tx=0x7fd82801cb30 comp rx=0 tx=0).stop 2024-09-16T15:46:55.787 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:55.780+0000 7fd8380b3640 1 -- 172.21.15.123:0/2420315341 shutdown_connections 2024-09-16T15:46:55.787 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:55.780+0000 7fd8380b3640 1 --2- 172.21.15.123:0/2420315341 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd81c0064c0 0x7fd81c00cb20 unknown :-1 s=CLOSED pgs=756 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:55.787 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:55.780+0000 7fd8380b3640 1 -- 172.21.15.123:0/2420315341 >> 172.21.15.123:0/2420315341 conn(0x7fd8300732a0 msgr2=0x7fd830074150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:46:55.787 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:55.780+0000 7fd8380b3640 1 -- 172.21.15.123:0/2420315341 shutdown_connections 2024-09-16T15:46:55.787 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:55.780+0000 7fd8380b3640 1 -- 172.21.15.123:0/2420315341 wait complete. 2024-09-16T15:46:55.787 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:55.784+0000 7fd8380b3640 1 Processor -- start 2024-09-16T15:46:55.788 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:55.784+0000 7fd8380b3640 1 -- start start 2024-09-16T15:46:55.790 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:55.784+0000 7fd8380b3640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fd81c0064c0 0x7fd83015bfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:46:55.790 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:55.784+0000 7fd8380b3640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd8301f5c30 0x7fd83015b850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:46:55.790 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:55.784+0000 7fd8380b3640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd83009c4c0 0x7fd83009b2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:46:55.790 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:55.784+0000 7fd8380b3640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fd8300cdf10 con 0x7fd8301f5c30 2024-09-16T15:46:55.790 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:55.784+0000 7fd8380b3640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fd830070d60 con 0x7fd83009c4c0 2024-09-16T15:46:55.790 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:55.784+0000 7fd8380b3640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fd8300ce090 con 0x7fd81c0064c0 2024-09-16T15:46:55.790 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:55.784+0000 7fd83564e640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd8301f5c30 0x7fd83015b850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:46:55.791 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:55.784+0000 7fd83564e640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd8301f5c30 0x7fd83015b850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:40102/0 (socket says 172.21.15.123:40102) 2024-09-16T15:46:55.791 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:55.784+0000 7fd83564e640 1 -- 172.21.15.123:0/1012350003 learned_addr learned my addr 172.21.15.123:0/1012350003 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:46:55.791 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:55.784+0000 7fd835e4f640 1 --2- 172.21.15.123:0/1012350003 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fd81c0064c0 0x7fd83015bfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:46:55.791 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:55.784+0000 7fd836650640 1 --2- 172.21.15.123:0/1012350003 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd83009c4c0 0x7fd83009b2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:46:55.791 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:55.784+0000 7fd83564e640 1 -- 172.21.15.123:0/1012350003 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fd81c0064c0 msgr2=0x7fd83015bfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:46:55.791 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:55.784+0000 7fd83564e640 1 --2- 172.21.15.123:0/1012350003 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fd81c0064c0 0x7fd83015bfa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:55.791 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:55.784+0000 7fd83564e640 1 -- 172.21.15.123:0/1012350003 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd83009c4c0 msgr2=0x7fd83009b2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:46:55.791 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:55.784+0000 7fd83564e640 1 --2- 172.21.15.123:0/1012350003 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd83009c4c0 0x7fd83009b2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:55.791 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:55.784+0000 7fd83564e640 1 -- 172.21.15.123:0/1012350003 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd83015c830 con 0x7fd8301f5c30 2024-09-16T15:46:55.791 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:55.784+0000 7fd83564e640 1 --2- 172.21.15.123:0/1012350003 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd8301f5c30 0x7fd83015b850 secure :-1 s=READY pgs=757 cs=0 l=1 rev1=1 crypto rx=0x7fd82000d650 tx=0x7fd82000db10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:46:55.791 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:55.784+0000 7fd8267fc640 1 -- 172.21.15.123:0/1012350003 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd820017070 con 0x7fd8301f5c30 2024-09-16T15:46:55.791 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:55.784+0000 7fd8267fc640 1 -- 172.21.15.123:0/1012350003 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fd820009b10 con 0x7fd8301f5c30 2024-09-16T15:46:55.791 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:55.784+0000 7fd8380b3640 1 -- 172.21.15.123:0/1012350003 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd8301f60b0 con 0x7fd8301f5c30 2024-09-16T15:46:55.792 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:55.784+0000 7fd8267fc640 1 -- 172.21.15.123:0/1012350003 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd820010460 con 0x7fd8301f5c30 2024-09-16T15:46:55.792 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:55.784+0000 7fd8380b3640 1 -- 172.21.15.123:0/1012350003 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd83009a250 con 0x7fd8301f5c30 2024-09-16T15:46:55.792 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:55.784+0000 7fd8380b3640 1 -- 172.21.15.123:0/1012350003 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd83015de30 con 0x7fd8301f5c30 2024-09-16T15:46:55.792 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:55.788+0000 7fd8267fc640 1 -- 172.21.15.123:0/1012350003 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 11) ==== 97386+0+0 (secure 0 0 0) 0x7fd82001a070 con 0x7fd8301f5c30 2024-09-16T15:46:55.793 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:55.788+0000 7fd8267fc640 1 --2- 172.21.15.123:0/1012350003 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fd814073280 0x7fd814075740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:46:55.793 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:55.788+0000 7fd8267fc640 1 -- 172.21.15.123:0/1012350003 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(66..66 src has 1..66) ==== 4857+0+0 (secure 0 0 0) 0x7fd820094300 con 0x7fd8301f5c30 2024-09-16T15:46:55.793 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:55.788+0000 7fd835e4f640 1 --2- 172.21.15.123:0/1012350003 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fd814073280 0x7fd814075740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:46:55.793 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:55.788+0000 7fd835e4f640 1 --2- 172.21.15.123:0/1012350003 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fd814073280 0x7fd814075740 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7fd82c001250 tx=0x7fd82c001700 comp rx=0 tx=0).ready entity=mgr.4135 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:46:55.796 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:55.792+0000 7fd8267fc640 1 -- 172.21.15.123:0/1012350003 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7fd8301f60b0 con 0x7fd8301f5c30 2024-09-16T15:46:56.015 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:56.012+0000 7fd8380b3640 1 -- 172.21.15.123:0/1012350003 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.alice"} v 0) -- 0x7fd8300b1580 con 0x7fd8301f5c30 2024-09-16T15:46:56.017 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:56.012+0000 7fd8267fc640 1 -- 172.21.15.123:0/1012350003 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.alice"}]=0 v21) ==== 82+0+0 (secure 0 0 0) 0x7fd82005ebb0 con 0x7fd8301f5c30 2024-09-16T15:46:56.020 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:56.016+0000 7fd8380b3640 1 -- 172.21.15.123:0/1012350003 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fd814073280 msgr2=0x7fd814075740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:46:56.021 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:56.016+0000 7fd8380b3640 1 --2- 172.21.15.123:0/1012350003 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fd814073280 0x7fd814075740 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7fd82c001250 tx=0x7fd82c001700 comp rx=0 tx=0).stop 2024-09-16T15:46:56.021 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:56.016+0000 7fd8380b3640 1 -- 172.21.15.123:0/1012350003 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd8301f5c30 msgr2=0x7fd83015b850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:46:56.021 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:56.016+0000 7fd8380b3640 1 --2- 172.21.15.123:0/1012350003 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd8301f5c30 0x7fd83015b850 secure :-1 s=READY pgs=757 cs=0 l=1 rev1=1 crypto rx=0x7fd82000d650 tx=0x7fd82000db10 comp rx=0 tx=0).stop 2024-09-16T15:46:56.021 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:56.016+0000 7fd8380b3640 1 -- 172.21.15.123:0/1012350003 shutdown_connections 2024-09-16T15:46:56.021 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:56.016+0000 7fd8380b3640 1 --2- 172.21.15.123:0/1012350003 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fd814073280 0x7fd814075740 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:56.021 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:56.016+0000 7fd8380b3640 1 --2- 172.21.15.123:0/1012350003 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd83009c4c0 0x7fd83009b2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:56.021 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:56.016+0000 7fd8380b3640 1 --2- 172.21.15.123:0/1012350003 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd8301f5c30 0x7fd83015b850 unknown :-1 s=CLOSED pgs=757 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:56.021 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:56.016+0000 7fd8380b3640 1 --2- 172.21.15.123:0/1012350003 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fd81c0064c0 0x7fd83015bfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:56.021 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:56.016+0000 7fd8380b3640 1 -- 172.21.15.123:0/1012350003 >> 172.21.15.123:0/1012350003 conn(0x7fd8300732a0 msgr2=0x7fd8300c4cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:46:56.021 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:56.016+0000 7fd8380b3640 1 -- 172.21.15.123:0/1012350003 shutdown_connections 2024-09-16T15:46:56.021 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:56.016+0000 7fd8380b3640 1 -- 172.21.15.123:0/1012350003 wait complete. 2024-09-16T15:46:56.036 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-09-16T15:46:56.148 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:56.144+0000 7fa2911f1640 1 -- 172.21.15.123:0/2865349454 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa274004810 msgr2=0x7fa274004c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:46:56.148 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:56.144+0000 7fa2911f1640 1 --2- 172.21.15.123:0/2865349454 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa274004810 0x7fa274004c90 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 crypto rx=0x7fa27c00bb10 tx=0x7fa27c01cd70 comp rx=0 tx=0).stop 2024-09-16T15:46:56.149 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:56.144+0000 7fa28b59e640 1 -- 172.21.15.123:0/2865349454 reap_dead start 2024-09-16T15:46:56.149 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:56.144+0000 7fa2911f1640 1 -- 172.21.15.123:0/2865349454 shutdown_connections 2024-09-16T15:46:56.149 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:56.144+0000 7fa2911f1640 1 -- 172.21.15.123:0/2865349454 >> 172.21.15.123:0/2865349454 conn(0x7fa28c0cfb10 msgr2=0x7fa28c0cff20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:46:56.149 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:56.144+0000 7fa2911f1640 1 -- 172.21.15.123:0/2865349454 shutdown_connections 2024-09-16T15:46:56.149 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:56.144+0000 7fa2911f1640 1 -- 172.21.15.123:0/2865349454 wait complete. 2024-09-16T15:46:56.149 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:56.144+0000 7fa2911f1640 1 Processor -- start 2024-09-16T15:46:56.149 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:56.144+0000 7fa2911f1640 1 -- start start 2024-09-16T15:46:56.151 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:56.144+0000 7fa2911f1640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fa274004810 0x7fa28c0c8b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:46:56.152 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:56.144+0000 7fa2911f1640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa28c158950 0x7fa28c157da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:46:56.152 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:56.144+0000 7fa2911f1640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa28c0b1290 0x7fa28c1582e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:46:56.152 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:56.144+0000 7fa2911f1640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fa28c167bd0 con 0x7fa28c0b1290 2024-09-16T15:46:56.152 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:56.144+0000 7fa2911f1640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fa28c167a50 con 0x7fa28c158950 2024-09-16T15:46:56.152 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:56.144+0000 7fa2911f1640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fa28c167d50 con 0x7fa274004810 2024-09-16T15:46:56.152 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:56.144+0000 7fa28b59e640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa28c0b1290 0x7fa28c1582e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:46:56.152 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:56.144+0000 7fa28b59e640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa28c0b1290 0x7fa28c1582e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:40114/0 (socket says 172.21.15.123:40114) 2024-09-16T15:46:56.152 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:56.144+0000 7fa28b59e640 1 -- 172.21.15.123:0/933813084 learned_addr learned my addr 172.21.15.123:0/933813084 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:46:56.152 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:56.144+0000 7fa28a59c640 1 --2- 172.21.15.123:0/933813084 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa28c158950 0x7fa28c157da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:46:56.152 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:56.144+0000 7fa28ad9d640 1 --2- 172.21.15.123:0/933813084 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fa274004810 0x7fa28c0c8b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:46:56.152 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:56.144+0000 7fa28b59e640 1 -- 172.21.15.123:0/933813084 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fa274004810 msgr2=0x7fa28c0c8b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:46:56.152 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:56.144+0000 7fa28b59e640 1 --2- 172.21.15.123:0/933813084 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fa274004810 0x7fa28c0c8b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:56.152 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:56.144+0000 7fa28b59e640 1 -- 172.21.15.123:0/933813084 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa28c158950 msgr2=0x7fa28c157da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:46:56.153 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:56.144+0000 7fa28b59e640 1 --2- 172.21.15.123:0/933813084 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa28c158950 0x7fa28c157da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:56.153 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:56.144+0000 7fa28b59e640 1 -- 172.21.15.123:0/933813084 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa28c15e950 con 0x7fa28c0b1290 2024-09-16T15:46:56.153 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:56.144+0000 7fa28ad9d640 1 --2- 172.21.15.123:0/933813084 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fa274004810 0x7fa28c0c8b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T15:46:56.153 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:56.144+0000 7fa28a59c640 1 --2- 172.21.15.123:0/933813084 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa28c158950 0x7fa28c157da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T15:46:56.153 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:56.144+0000 7fa28b59e640 1 --2- 172.21.15.123:0/933813084 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa28c0b1290 0x7fa28c1582e0 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 crypto rx=0x7fa28c1602a0 tx=0x7fa28400db50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:46:56.153 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:56.144+0000 7fa2737fe640 1 -- 172.21.15.123:0/933813084 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa284003b80 con 0x7fa28c0b1290 2024-09-16T15:46:56.153 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:56.148+0000 7fa2737fe640 1 -- 172.21.15.123:0/933813084 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fa284003e10 con 0x7fa28c0b1290 2024-09-16T15:46:56.153 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:56.148+0000 7fa2911f1640 1 -- 172.21.15.123:0/933813084 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa28c15ec30 con 0x7fa28c0b1290 2024-09-16T15:46:56.153 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:56.148+0000 7fa2737fe640 1 -- 172.21.15.123:0/933813084 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa284001c90 con 0x7fa28c0b1290 2024-09-16T15:46:56.153 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:56.148+0000 7fa2911f1640 1 -- 172.21.15.123:0/933813084 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa28c0c90b0 con 0x7fa28c0b1290 2024-09-16T15:46:56.153 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:56.148+0000 7fa2911f1640 1 -- 172.21.15.123:0/933813084 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa28c06e750 con 0x7fa28c0b1290 2024-09-16T15:46:56.154 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:56.148+0000 7fa2737fe640 1 -- 172.21.15.123:0/933813084 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 11) ==== 97386+0+0 (secure 0 0 0) 0x7fa284001e30 con 0x7fa28c0b1290 2024-09-16T15:46:56.154 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:56.148+0000 7fa2737fe640 1 --2- 172.21.15.123:0/933813084 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fa278073220 0x7fa2780756e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:46:56.154 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:56.148+0000 7fa2737fe640 1 -- 172.21.15.123:0/933813084 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(66..66 src has 1..66) ==== 4857+0+0 (secure 0 0 0) 0x7fa284094550 con 0x7fa28c0b1290 2024-09-16T15:46:56.154 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:56.148+0000 7fa28ad9d640 1 --2- 172.21.15.123:0/933813084 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fa278073220 0x7fa2780756e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:46:56.154 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:56.148+0000 7fa28ad9d640 1 --2- 172.21.15.123:0/933813084 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fa278073220 0x7fa2780756e0 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7fa280001870 tx=0x7fa280001db0 comp rx=0 tx=0).ready entity=mgr.4135 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:46:56.157 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:56.152+0000 7fa2737fe640 1 -- 172.21.15.123:0/933813084 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7fa28405f4e0 con 0x7fa28c0b1290 2024-09-16T15:46:56.324 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:56.320+0000 7fa2911f1640 1 -- 172.21.15.123:0/933813084 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) -- 0x7fa28c0c5fc0 con 0x7fa28c0b1290 2024-09-16T15:46:56.324 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:56.320+0000 7fa2737fe640 1 -- 172.21.15.123:0/933813084 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v66) ==== 84+0+65 (secure 0 0 0) 0x7fa28405ede0 con 0x7fa28c0b1290 2024-09-16T15:46:56.325 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-09-16T15:46:56.325 INFO:teuthology.orchestra.run.smithi123.stdout:[ 2024-09-16T15:46:56.325 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-09-16T15:46:56.325 INFO:teuthology.orchestra.run.smithi123.stdout: "poolnum": 1, 2024-09-16T15:46:56.325 INFO:teuthology.orchestra.run.smithi123.stdout: "poolname": ".mgr" 2024-09-16T15:46:56.325 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-16T15:46:56.325 INFO:teuthology.orchestra.run.smithi123.stdout:] 2024-09-16T15:46:56.327 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:56.320+0000 7fa2911f1640 1 -- 172.21.15.123:0/933813084 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fa278073220 msgr2=0x7fa2780756e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:46:56.327 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:56.320+0000 7fa2911f1640 1 --2- 172.21.15.123:0/933813084 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fa278073220 0x7fa2780756e0 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7fa280001870 tx=0x7fa280001db0 comp rx=0 tx=0).stop 2024-09-16T15:46:56.327 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:56.320+0000 7fa2911f1640 1 -- 172.21.15.123:0/933813084 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa28c0b1290 msgr2=0x7fa28c1582e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:46:56.327 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:56.320+0000 7fa2911f1640 1 --2- 172.21.15.123:0/933813084 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa28c0b1290 0x7fa28c1582e0 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 crypto rx=0x7fa28c1602a0 tx=0x7fa28400db50 comp rx=0 tx=0).stop 2024-09-16T15:46:56.327 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:56.320+0000 7fa2911f1640 1 -- 172.21.15.123:0/933813084 shutdown_connections 2024-09-16T15:46:56.327 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:56.320+0000 7fa2911f1640 1 --2- 172.21.15.123:0/933813084 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fa278073220 0x7fa2780756e0 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:56.327 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:56.320+0000 7fa2911f1640 1 --2- 172.21.15.123:0/933813084 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa28c0b1290 0x7fa28c1582e0 unknown :-1 s=CLOSED pgs=760 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:56.327 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:56.320+0000 7fa2911f1640 1 --2- 172.21.15.123:0/933813084 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa28c158950 0x7fa28c157da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:56.327 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:56.320+0000 7fa2911f1640 1 --2- 172.21.15.123:0/933813084 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fa274004810 0x7fa28c0c8b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:56.327 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:56.320+0000 7fa2911f1640 1 -- 172.21.15.123:0/933813084 >> 172.21.15.123:0/933813084 conn(0x7fa28c0cfb10 msgr2=0x7fa28c162300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:46:56.327 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:56.320+0000 7fa2911f1640 1 -- 172.21.15.123:0/933813084 shutdown_connections 2024-09-16T15:46:56.327 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:56.320+0000 7fa2911f1640 1 -- 172.21.15.123:0/933813084 wait complete. 2024-09-16T15:46:56.342 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-09-16T15:46:56.343 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-09-16T15:46:56.461 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:56.456+0000 7f8228a48640 1 -- 172.21.15.123:0/487039669 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8210003a10 msgr2=0x7f8210003e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:46:56.461 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:56.456+0000 7f8228a48640 1 --2- 172.21.15.123:0/487039669 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8210003a10 0x7f8210003e30 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 crypto rx=0x7f821400b9e0 tx=0x7f821401cc40 comp rx=0 tx=0).stop 2024-09-16T15:46:56.461 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:56.456+0000 7f8222d9d640 1 -- 172.21.15.123:0/487039669 reap_dead start 2024-09-16T15:46:56.461 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:56.456+0000 7f8228a48640 1 -- 172.21.15.123:0/487039669 shutdown_connections 2024-09-16T15:46:56.461 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:56.456+0000 7f8228a48640 1 -- 172.21.15.123:0/487039669 >> 172.21.15.123:0/487039669 conn(0x7f82240d3e30 msgr2=0x7f82240d4240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:46:56.461 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:56.456+0000 7f8228a48640 1 -- 172.21.15.123:0/487039669 shutdown_connections 2024-09-16T15:46:56.461 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:56.456+0000 7f8228a48640 1 -- 172.21.15.123:0/487039669 wait complete. 2024-09-16T15:46:56.462 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:56.456+0000 7f8228a48640 1 Processor -- start 2024-09-16T15:46:56.462 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:56.456+0000 7f8228a48640 1 -- start start 2024-09-16T15:46:56.464 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:56.456+0000 7f8228a48640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f8210003a10 0x7f822415cf00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:46:56.464 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:56.456+0000 7f8228a48640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f82240b9390 0x7f82240b84b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:46:56.464 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:56.456+0000 7f8228a48640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f82240b9810 0x7f8224164ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:46:56.464 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:56.456+0000 7f8228a48640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f8224168f30 con 0x7f82240b9390 2024-09-16T15:46:56.464 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:56.456+0000 7f8228a48640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f8224168db0 con 0x7f82240b9810 2024-09-16T15:46:56.464 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:56.456+0000 7f8228a48640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f82241690b0 con 0x7f8210003a10 2024-09-16T15:46:56.464 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:56.456+0000 7f8219d9b640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f82240b9390 0x7f82240b84b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:46:56.465 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:56.456+0000 7f8219d9b640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f82240b9390 0x7f82240b84b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:40124/0 (socket says 172.21.15.123:40124) 2024-09-16T15:46:56.465 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:56.456+0000 7f8219d9b640 1 -- 172.21.15.123:0/2563318410 learned_addr learned my addr 172.21.15.123:0/2563318410 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:46:56.465 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:56.456+0000 7f822259c640 1 --2- 172.21.15.123:0/2563318410 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f8210003a10 0x7f822415cf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:46:56.465 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:56.460+0000 7f8219d9b640 1 -- 172.21.15.123:0/2563318410 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f8210003a10 msgr2=0x7f822415cf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:46:56.465 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:56.460+0000 7f8219d9b640 1 --2- 172.21.15.123:0/2563318410 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f8210003a10 0x7f822415cf00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:56.465 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:56.460+0000 7f8222d9d640 1 --2- 172.21.15.123:0/2563318410 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f82240b9810 0x7f8224164ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:46:56.465 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:56.460+0000 7f8219d9b640 1 -- 172.21.15.123:0/2563318410 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f82240b9810 msgr2=0x7f8224164ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:46:56.465 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:56.460+0000 7f8219d9b640 1 --2- 172.21.15.123:0/2563318410 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f82240b9810 0x7f8224164ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:56.465 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:56.460+0000 7f8219d9b640 1 -- 172.21.15.123:0/2563318410 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f822415d440 con 0x7f82240b9390 2024-09-16T15:46:56.465 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:56.460+0000 7f8222d9d640 1 --2- 172.21.15.123:0/2563318410 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f82240b9810 0x7f8224164ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T15:46:56.465 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:56.460+0000 7f8219d9b640 1 --2- 172.21.15.123:0/2563318410 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f82240b9390 0x7f82240b84b0 secure :-1 s=READY pgs=763 cs=0 l=1 rev1=1 crypto rx=0x7f820c00e9c0 tx=0x7f820c00ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:46:56.465 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:56.460+0000 7f821b7fe640 1 -- 172.21.15.123:0/2563318410 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f820c00cc00 con 0x7f82240b9390 2024-09-16T15:46:56.465 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:56.460+0000 7f821b7fe640 1 -- 172.21.15.123:0/2563318410 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f820c00ce90 con 0x7f82240b9390 2024-09-16T15:46:56.466 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:56.460+0000 7f8228a48640 1 -- 172.21.15.123:0/2563318410 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8224167050 con 0x7f82240b9390 2024-09-16T15:46:56.466 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:56.460+0000 7f821b7fe640 1 -- 172.21.15.123:0/2563318410 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f820c005260 con 0x7f82240b9390 2024-09-16T15:46:56.466 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:56.460+0000 7f8228a48640 1 -- 172.21.15.123:0/2563318410 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8224167610 con 0x7f82240b9390 2024-09-16T15:46:56.466 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:56.460+0000 7f8228a48640 1 -- 172.21.15.123:0/2563318410 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f82240d0fb0 con 0x7f82240b9390 2024-09-16T15:46:56.466 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:56.460+0000 7f821b7fe640 1 -- 172.21.15.123:0/2563318410 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 11) ==== 97386+0+0 (secure 0 0 0) 0x7f820c005400 con 0x7f82240b9390 2024-09-16T15:46:56.467 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:56.460+0000 7f821b7fe640 1 --2- 172.21.15.123:0/2563318410 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f8204073220 0x7f82040756e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:46:56.467 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:56.460+0000 7f821b7fe640 1 -- 172.21.15.123:0/2563318410 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(66..66 src has 1..66) ==== 4857+0+0 (secure 0 0 0) 0x7f820c094a30 con 0x7f82240b9390 2024-09-16T15:46:56.467 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:56.460+0000 7f822259c640 1 --2- 172.21.15.123:0/2563318410 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f8204073220 0x7f82040756e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:46:56.467 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:56.460+0000 7f822259c640 1 --2- 172.21.15.123:0/2563318410 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f8204073220 0x7f82040756e0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f821400b9e0 tx=0x7f82140a7040 comp rx=0 tx=0).ready entity=mgr.4135 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:46:56.469 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:56.464+0000 7f821b7fe640 1 -- 172.21.15.123:0/2563318410 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f820c016030 con 0x7f82240b9390 2024-09-16T15:46:56.632 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:56.628+0000 7f8228a48640 1 -- 172.21.15.123:0/2563318410 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) -- 0x7f82240b40d0 con 0x7f82240b9390 2024-09-16T15:46:57.541 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:57.536+0000 7f821b7fe640 1 -- 172.21.15.123:0/2563318410 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v67) ==== 138+0+0 (secure 0 0 0) 0x7f820c05f920 con 0x7f82240b9390 2024-09-16T15:46:57.541 INFO:teuthology.orchestra.run.smithi123.stderr:pool 'cephfs_metadata' created 2024-09-16T15:46:57.544 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:57.540+0000 7f8228a48640 1 -- 172.21.15.123:0/2563318410 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f8204073220 msgr2=0x7f82040756e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:46:57.544 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:57.540+0000 7f8228a48640 1 --2- 172.21.15.123:0/2563318410 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f8204073220 0x7f82040756e0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f821400b9e0 tx=0x7f82140a7040 comp rx=0 tx=0).stop 2024-09-16T15:46:57.544 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:57.540+0000 7f8228a48640 1 -- 172.21.15.123:0/2563318410 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f82240b9390 msgr2=0x7f82240b84b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:46:57.544 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:57.540+0000 7f8228a48640 1 --2- 172.21.15.123:0/2563318410 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f82240b9390 0x7f82240b84b0 secure :-1 s=READY pgs=763 cs=0 l=1 rev1=1 crypto rx=0x7f820c00e9c0 tx=0x7f820c00ee80 comp rx=0 tx=0).stop 2024-09-16T15:46:57.544 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:57.540+0000 7f8228a48640 1 -- 172.21.15.123:0/2563318410 shutdown_connections 2024-09-16T15:46:57.545 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:57.540+0000 7f8228a48640 1 --2- 172.21.15.123:0/2563318410 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f8204073220 0x7f82040756e0 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:57.545 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:57.540+0000 7f8228a48640 1 --2- 172.21.15.123:0/2563318410 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f82240b9810 0x7f8224164ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:57.545 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:57.540+0000 7f8228a48640 1 --2- 172.21.15.123:0/2563318410 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f82240b9390 0x7f82240b84b0 unknown :-1 s=CLOSED pgs=763 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:57.545 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:57.540+0000 7f8228a48640 1 --2- 172.21.15.123:0/2563318410 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f8210003a10 0x7f822415cf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:57.545 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:57.540+0000 7f8228a48640 1 -- 172.21.15.123:0/2563318410 >> 172.21.15.123:0/2563318410 conn(0x7f82240d3e30 msgr2=0x7f82241601f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:46:57.545 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:57.540+0000 7f8228a48640 1 -- 172.21.15.123:0/2563318410 shutdown_connections 2024-09-16T15:46:57.545 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:57.540+0000 7f8228a48640 1 -- 172.21.15.123:0/2563318410 wait complete. 2024-09-16T15:46:57.568 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-09-16T15:46:57.706 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:57.700+0000 7fed077a1640 1 -- 172.21.15.123:0/590687539 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fed0015a170 msgr2=0x7fed00072f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:46:57.706 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:57.700+0000 7fed077a1640 1 --2- 172.21.15.123:0/590687539 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fed0015a170 0x7fed00072f30 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 crypto rx=0x7fecfc009840 tx=0x7fecfc01c760 comp rx=0 tx=0).stop 2024-09-16T15:46:57.706 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:57.700+0000 7fed077a1640 1 -- 172.21.15.123:0/590687539 shutdown_connections 2024-09-16T15:46:57.706 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:57.700+0000 7fed077a1640 1 --2- 172.21.15.123:0/590687539 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fed000ca650 0x7fed000cbc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:57.707 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:57.700+0000 7fed077a1640 1 --2- 172.21.15.123:0/590687539 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fed0015a170 0x7fed00072f30 unknown :-1 s=CLOSED pgs=765 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:57.707 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:57.700+0000 7fed077a1640 1 -- 172.21.15.123:0/590687539 >> 172.21.15.123:0/590687539 conn(0x7fed000b1710 msgr2=0x7fed000b1b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:46:57.707 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:57.700+0000 7fed077a1640 1 -- 172.21.15.123:0/590687539 shutdown_connections 2024-09-16T15:46:57.707 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:57.700+0000 7fed077a1640 1 -- 172.21.15.123:0/590687539 wait complete. 2024-09-16T15:46:57.707 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:57.704+0000 7fed077a1640 1 Processor -- start 2024-09-16T15:46:57.707 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:57.704+0000 7fed077a1640 1 -- start start 2024-09-16T15:46:57.709 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:57.704+0000 7fed077a1640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fed000ca650 0x7fed0014df40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:46:57.709 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:57.704+0000 7fed077a1640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fed0015a170 0x7fed00156d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:46:57.709 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:57.704+0000 7fed077a1640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fed00151b80 0x7fed0006fed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:46:57.710 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:57.704+0000 7fed077a1640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fed0006e830 con 0x7fed00151b80 2024-09-16T15:46:57.710 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:57.704+0000 7fed077a1640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fed0006e6b0 con 0x7fed000ca650 2024-09-16T15:46:57.710 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:57.704+0000 7fed077a1640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fed0006e9b0 con 0x7fed0015a170 2024-09-16T15:46:57.710 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:57.704+0000 7fed05d3e640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fed00151b80 0x7fed0006fed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:46:57.710 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:57.704+0000 7fed05d3e640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fed00151b80 0x7fed0006fed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:40146/0 (socket says 172.21.15.123:40146) 2024-09-16T15:46:57.710 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:57.704+0000 7fed05d3e640 1 -- 172.21.15.123:0/2004566472 learned_addr learned my addr 172.21.15.123:0/2004566472 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:46:57.710 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:57.704+0000 7fed04d3c640 1 --2- 172.21.15.123:0/2004566472 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fed0015a170 0x7fed00156d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:46:57.710 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:57.704+0000 7fed05d3e640 1 -- 172.21.15.123:0/2004566472 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fed0015a170 msgr2=0x7fed00156d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:46:57.711 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:57.704+0000 7fed0553d640 1 --2- 172.21.15.123:0/2004566472 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fed000ca650 0x7fed0014df40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:46:57.711 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:57.704+0000 7fed05d3e640 1 --2- 172.21.15.123:0/2004566472 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fed0015a170 0x7fed00156d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:57.711 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:57.704+0000 7fed05d3e640 1 -- 172.21.15.123:0/2004566472 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fed000ca650 msgr2=0x7fed0014df40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:46:57.711 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:57.704+0000 7fed05d3e640 1 --2- 172.21.15.123:0/2004566472 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fed000ca650 0x7fed0014df40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:57.711 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:57.704+0000 7fed05d3e640 1 -- 172.21.15.123:0/2004566472 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fed0006ee30 con 0x7fed00151b80 2024-09-16T15:46:57.711 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:57.704+0000 7fed04d3c640 1 --2- 172.21.15.123:0/2004566472 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fed0015a170 0x7fed00156d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T15:46:57.711 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:57.704+0000 7fed05d3e640 1 --2- 172.21.15.123:0/2004566472 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fed00151b80 0x7fed0006fed0 secure :-1 s=READY pgs=766 cs=0 l=1 rev1=1 crypto rx=0x7fecf800cbc0 tx=0x7fecf800ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:46:57.711 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:57.704+0000 7fecf5ffb640 1 -- 172.21.15.123:0/2004566472 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fecf8016020 con 0x7fed00151b80 2024-09-16T15:46:57.711 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:57.704+0000 7fecf5ffb640 1 -- 172.21.15.123:0/2004566472 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fecf800ab40 con 0x7fed00151b80 2024-09-16T15:46:57.711 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:57.704+0000 7fed077a1640 1 -- 172.21.15.123:0/2004566472 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fed0006f110 con 0x7fed00151b80 2024-09-16T15:46:57.711 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:57.704+0000 7fecf5ffb640 1 -- 172.21.15.123:0/2004566472 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fecf8012420 con 0x7fed00151b80 2024-09-16T15:46:57.711 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:57.704+0000 7fed077a1640 1 -- 172.21.15.123:0/2004566472 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fed0006f6d0 con 0x7fed00151b80 2024-09-16T15:46:57.712 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:57.704+0000 7fed077a1640 1 -- 172.21.15.123:0/2004566472 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fed000cbc60 con 0x7fed00151b80 2024-09-16T15:46:57.712 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:57.708+0000 7fecf5ffb640 1 -- 172.21.15.123:0/2004566472 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 11) ==== 97386+0+0 (secure 0 0 0) 0x7fecf8019020 con 0x7fed00151b80 2024-09-16T15:46:57.712 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:57.708+0000 7fecf5ffb640 1 --2- 172.21.15.123:0/2004566472 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fece4073250 0x7fece4075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:46:57.712 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:57.708+0000 7fecf5ffb640 1 -- 172.21.15.123:0/2004566472 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(67..67 src has 1..67) ==== 5227+0+0 (secure 0 0 0) 0x7fecf8094970 con 0x7fed00151b80 2024-09-16T15:46:57.712 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:57.708+0000 7fed0553d640 1 --2- 172.21.15.123:0/2004566472 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fece4073250 0x7fece4075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:46:57.713 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:57.708+0000 7fed0553d640 1 --2- 172.21.15.123:0/2004566472 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fece4073250 0x7fece4075710 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7fecfc0af000 tx=0x7fecfc01afa0 comp rx=0 tx=0).ready entity=mgr.4135 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:46:57.715 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:57.712+0000 7fecf5ffb640 1 -- 172.21.15.123:0/2004566472 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7fecf8020030 con 0x7fed00151b80 2024-09-16T15:46:57.887 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:57.880+0000 7fed077a1640 1 -- 172.21.15.123:0/2004566472 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) -- 0x7fed0014e530 con 0x7fed00151b80 2024-09-16T15:46:58.552 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:58.548+0000 7fecf5ffb640 1 -- 172.21.15.123:0/2004566472 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v68) ==== 170+0+0 (secure 0 0 0) 0x7fecf805f980 con 0x7fed00151b80 2024-09-16T15:46:58.552 INFO:teuthology.orchestra.run.smithi123.stderr:pool 'cephfs_data' created 2024-09-16T15:46:58.559 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:58.556+0000 7fed077a1640 1 -- 172.21.15.123:0/2004566472 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fece4073250 msgr2=0x7fece4075710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:46:58.559 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:58.556+0000 7fed077a1640 1 --2- 172.21.15.123:0/2004566472 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fece4073250 0x7fece4075710 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7fecfc0af000 tx=0x7fecfc01afa0 comp rx=0 tx=0).stop 2024-09-16T15:46:58.559 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:58.556+0000 7fed077a1640 1 -- 172.21.15.123:0/2004566472 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fed00151b80 msgr2=0x7fed0006fed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:46:58.559 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:58.556+0000 7fed077a1640 1 --2- 172.21.15.123:0/2004566472 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fed00151b80 0x7fed0006fed0 secure :-1 s=READY pgs=766 cs=0 l=1 rev1=1 crypto rx=0x7fecf800cbc0 tx=0x7fecf800ccb0 comp rx=0 tx=0).stop 2024-09-16T15:46:58.563 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:58.556+0000 7fed077a1640 1 -- 172.21.15.123:0/2004566472 shutdown_connections 2024-09-16T15:46:58.563 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:58.556+0000 7fed077a1640 1 --2- 172.21.15.123:0/2004566472 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fece4073250 0x7fece4075710 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:58.563 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:58.556+0000 7fed077a1640 1 --2- 172.21.15.123:0/2004566472 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fed00151b80 0x7fed0006fed0 unknown :-1 s=CLOSED pgs=766 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:58.563 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:58.556+0000 7fed077a1640 1 --2- 172.21.15.123:0/2004566472 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fed0015a170 0x7fed00156d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:58.563 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:58.556+0000 7fed077a1640 1 --2- 172.21.15.123:0/2004566472 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fed000ca650 0x7fed0014df40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:58.563 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:58.556+0000 7fed077a1640 1 -- 172.21.15.123:0/2004566472 >> 172.21.15.123:0/2004566472 conn(0x7fed000b1710 msgr2=0x7fed000cff80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:46:58.563 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:58.556+0000 7fed077a1640 1 -- 172.21.15.123:0/2004566472 shutdown_connections 2024-09-16T15:46:58.564 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:58.560+0000 7fed077a1640 1 -- 172.21.15.123:0/2004566472 wait complete. 2024-09-16T15:46:58.592 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-09-16T15:46:58.711 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:58.704+0000 7f5c8c119640 1 -- 172.21.15.123:0/3387822691 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5c8415ef00 msgr2=0x7f5c840cf5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:46:58.711 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:58.704+0000 7f5c8c119640 1 --2- 172.21.15.123:0/3387822691 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5c8415ef00 0x7f5c840cf5f0 secure :-1 s=READY pgs=767 cs=0 l=1 rev1=1 crypto rx=0x7f5c74009900 tx=0x7f5c7401c8b0 comp rx=0 tx=0).stop 2024-09-16T15:46:58.711 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:58.704+0000 7f5c8c119640 1 -- 172.21.15.123:0/3387822691 shutdown_connections 2024-09-16T15:46:58.711 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:58.704+0000 7f5c8c119640 1 --2- 172.21.15.123:0/3387822691 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f5c8406dd00 0x7f5c8406e160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:58.711 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:58.704+0000 7f5c8c119640 1 --2- 172.21.15.123:0/3387822691 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5c8415ef00 0x7f5c840cf5f0 unknown :-1 s=CLOSED pgs=767 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:58.711 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:58.704+0000 7f5c8c119640 1 -- 172.21.15.123:0/3387822691 >> 172.21.15.123:0/3387822691 conn(0x7f5c840d0bc0 msgr2=0x7f5c840d0fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:46:58.711 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:58.704+0000 7f5c8c119640 1 -- 172.21.15.123:0/3387822691 shutdown_connections 2024-09-16T15:46:58.711 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:58.704+0000 7f5c8c119640 1 -- 172.21.15.123:0/3387822691 wait complete. 2024-09-16T15:46:58.712 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:58.708+0000 7f5c8c119640 1 Processor -- start 2024-09-16T15:46:58.712 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:58.708+0000 7f5c8c119640 1 -- start start 2024-09-16T15:46:58.713 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:58.708+0000 7f5c8c119640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5c8406dd00 0x7f5c8414bf60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:46:58.714 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:58.708+0000 7f5c8c119640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5c8415b050 0x7f5c841fa790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:46:58.714 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:58.708+0000 7f5c8c119640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f5c8415ef00 0x7f5c8414e970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:46:58.714 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:58.708+0000 7f5c8c119640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f5c84154930 con 0x7f5c8415b050 2024-09-16T15:46:58.714 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:58.708+0000 7f5c8c119640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f5c841547b0 con 0x7f5c8406dd00 2024-09-16T15:46:58.714 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:58.708+0000 7f5c8c119640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f5c84154ab0 con 0x7f5c8415ef00 2024-09-16T15:46:58.714 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:58.708+0000 7f5c896b4640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5c8415b050 0x7f5c841fa790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:46:58.714 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:58.708+0000 7f5c896b4640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5c8415b050 0x7f5c841fa790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:40176/0 (socket says 172.21.15.123:40176) 2024-09-16T15:46:58.714 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:58.708+0000 7f5c896b4640 1 -- 172.21.15.123:0/3425565796 learned_addr learned my addr 172.21.15.123:0/3425565796 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:46:58.714 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:58.708+0000 7f5c8a6b6640 1 --2- 172.21.15.123:0/3425565796 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f5c8415ef00 0x7f5c8414e970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:46:58.714 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:58.708+0000 7f5c89eb5640 1 --2- 172.21.15.123:0/3425565796 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5c8406dd00 0x7f5c8414bf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:46:58.714 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:58.708+0000 7f5c896b4640 1 -- 172.21.15.123:0/3425565796 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f5c8415ef00 msgr2=0x7f5c8414e970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:46:58.714 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:58.708+0000 7f5c896b4640 1 --2- 172.21.15.123:0/3425565796 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f5c8415ef00 0x7f5c8414e970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:58.714 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:58.708+0000 7f5c896b4640 1 -- 172.21.15.123:0/3425565796 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5c8406dd00 msgr2=0x7f5c8414bf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:46:58.715 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:58.708+0000 7f5c896b4640 1 --2- 172.21.15.123:0/3425565796 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5c8406dd00 0x7f5c8414bf60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:58.715 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:58.708+0000 7f5c896b4640 1 -- 172.21.15.123:0/3425565796 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5c841fafc0 con 0x7f5c8415b050 2024-09-16T15:46:58.715 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:58.708+0000 7f5c89eb5640 1 --2- 172.21.15.123:0/3425565796 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5c8406dd00 0x7f5c8414bf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T15:46:58.715 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:58.708+0000 7f5c896b4640 1 --2- 172.21.15.123:0/3425565796 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5c8415b050 0x7f5c841fa790 secure :-1 s=READY pgs=768 cs=0 l=1 rev1=1 crypto rx=0x7f5c8000b580 tx=0x7f5c8000ba40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:46:58.715 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:58.708+0000 7f5c7a7fc640 1 -- 172.21.15.123:0/3425565796 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c80014070 con 0x7f5c8415b050 2024-09-16T15:46:58.715 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:58.708+0000 7f5c7a7fc640 1 -- 172.21.15.123:0/3425565796 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f5c80010980 con 0x7f5c8415b050 2024-09-16T15:46:58.715 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:58.708+0000 7f5c8c119640 1 -- 172.21.15.123:0/3425565796 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5c841fb2a0 con 0x7f5c8415b050 2024-09-16T15:46:58.715 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:58.708+0000 7f5c7a7fc640 1 -- 172.21.15.123:0/3425565796 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c80010dd0 con 0x7f5c8415b050 2024-09-16T15:46:58.715 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:58.708+0000 7f5c8c119640 1 -- 172.21.15.123:0/3425565796 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5c840c5c30 con 0x7f5c8415b050 2024-09-16T15:46:58.716 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:58.708+0000 7f5c8c119640 1 -- 172.21.15.123:0/3425565796 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5c840c5e30 con 0x7f5c8415b050 2024-09-16T15:46:58.716 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:58.712+0000 7f5c7a7fc640 1 -- 172.21.15.123:0/3425565796 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 11) ==== 97386+0+0 (secure 0 0 0) 0x7f5c80017070 con 0x7f5c8415b050 2024-09-16T15:46:58.716 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:58.712+0000 7f5c7a7fc640 1 --2- 172.21.15.123:0/3425565796 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f5c68073190 0x7f5c68075650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:46:58.716 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:58.712+0000 7f5c7a7fc640 1 -- 172.21.15.123:0/3425565796 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(68..68 src has 1..68) ==== 5609+0+0 (secure 0 0 0) 0x7f5c800945b0 con 0x7f5c8415b050 2024-09-16T15:46:58.716 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:58.712+0000 7f5c89eb5640 1 --2- 172.21.15.123:0/3425565796 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f5c68073190 0x7f5c68075650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:46:58.716 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:58.712+0000 7f5c89eb5640 1 --2- 172.21.15.123:0/3425565796 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f5c68073190 0x7f5c68075650 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f5c740af000 tx=0x7f5c7401b0f0 comp rx=0 tx=0).ready entity=mgr.4135 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:46:58.719 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:58.716+0000 7f5c7a7fc640 1 -- 172.21.15.123:0/3425565796 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f5c840c5e30 con 0x7f5c8415b050 2024-09-16T15:46:58.915 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:58.912+0000 7f5c8c119640 1 -- 172.21.15.123:0/3425565796 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) -- 0x7f5c8415b7f0 con 0x7f5c8415b050 2024-09-16T15:46:59.584 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:59.580+0000 7f5c7a7fc640 1 -- 172.21.15.123:0/3425565796 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 10 and data pool 11 v41) ==== 174+0+0 (secure 0 0 0) 0x7f5c8005f3b0 con 0x7f5c8415b050 2024-09-16T15:46:59.585 INFO:teuthology.orchestra.run.smithi123.stderr:new fs with metadata pool 10 and data pool 11 2024-09-16T15:46:59.600 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:59.596+0000 7f5c8c119640 1 -- 172.21.15.123:0/3425565796 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f5c68073190 msgr2=0x7f5c68075650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:46:59.601 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:59.596+0000 7f5c8c119640 1 --2- 172.21.15.123:0/3425565796 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f5c68073190 0x7f5c68075650 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f5c740af000 tx=0x7f5c7401b0f0 comp rx=0 tx=0).stop 2024-09-16T15:46:59.601 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:59.596+0000 7f5c8c119640 1 -- 172.21.15.123:0/3425565796 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5c8415b050 msgr2=0x7f5c841fa790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:46:59.601 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:59.596+0000 7f5c8c119640 1 --2- 172.21.15.123:0/3425565796 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5c8415b050 0x7f5c841fa790 secure :-1 s=READY pgs=768 cs=0 l=1 rev1=1 crypto rx=0x7f5c8000b580 tx=0x7f5c8000ba40 comp rx=0 tx=0).stop 2024-09-16T15:46:59.601 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:59.596+0000 7f5c8c119640 1 -- 172.21.15.123:0/3425565796 shutdown_connections 2024-09-16T15:46:59.601 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:59.596+0000 7f5c8c119640 1 --2- 172.21.15.123:0/3425565796 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f5c68073190 0x7f5c68075650 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:59.601 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:59.596+0000 7f5c8c119640 1 --2- 172.21.15.123:0/3425565796 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f5c8415ef00 0x7f5c8414e970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:59.601 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:59.596+0000 7f5c8c119640 1 --2- 172.21.15.123:0/3425565796 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5c8415b050 0x7f5c841fa790 unknown :-1 s=CLOSED pgs=768 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:59.601 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:59.596+0000 7f5c8c119640 1 --2- 172.21.15.123:0/3425565796 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5c8406dd00 0x7f5c8414bf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:59.601 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:59.596+0000 7f5c8c119640 1 -- 172.21.15.123:0/3425565796 >> 172.21.15.123:0/3425565796 conn(0x7f5c840d0bc0 msgr2=0x7f5c84159f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:46:59.601 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:59.596+0000 7f5c8c119640 1 -- 172.21.15.123:0/3425565796 shutdown_connections 2024-09-16T15:46:59.602 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:59.596+0000 7f5c8c119640 1 -- 172.21.15.123:0/3425565796 wait complete. 2024-09-16T15:46:59.627 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-16T15:46:59.749 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:59.744+0000 7f30e9880640 1 -- 172.21.15.123:0/3527815456 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f30cc004810 msgr2=0x7f30cc004c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:46:59.749 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:59.744+0000 7f30e9880640 1 --2- 172.21.15.123:0/3527815456 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f30cc004810 0x7f30cc004c90 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 crypto rx=0x7f30d800bb10 tx=0x7f30d801cdb0 comp rx=0 tx=0).stop 2024-09-16T15:46:59.749 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:59.744+0000 7f30e37fe640 1 -- 172.21.15.123:0/3527815456 reap_dead start 2024-09-16T15:46:59.749 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:59.744+0000 7f30e9880640 1 -- 172.21.15.123:0/3527815456 shutdown_connections 2024-09-16T15:46:59.749 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:59.744+0000 7f30e9880640 1 -- 172.21.15.123:0/3527815456 >> 172.21.15.123:0/3527815456 conn(0x7f30e40cfb10 msgr2=0x7f30e40cff20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:46:59.749 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:59.744+0000 7f30e9880640 1 -- 172.21.15.123:0/3527815456 shutdown_connections 2024-09-16T15:46:59.749 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:59.744+0000 7f30e9880640 1 -- 172.21.15.123:0/3527815456 wait complete. 2024-09-16T15:46:59.750 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:59.744+0000 7f30e9880640 1 Processor -- start 2024-09-16T15:46:59.750 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:59.744+0000 7f30e9880640 1 -- start start 2024-09-16T15:46:59.751 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:59.744+0000 7f30e9880640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f30cc004810 0x7f30e4156190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:46:59.752 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:59.744+0000 7f30e9880640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f30e40b1070 0x7f30e41f6e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:46:59.752 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:59.744+0000 7f30e9880640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f30e41f7340 0x7f30e40b29a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:46:59.752 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:59.744+0000 7f30e9880640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f30e4166c50 con 0x7f30e40b1070 2024-09-16T15:46:59.752 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:59.744+0000 7f30e9880640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f30e4074980 con 0x7f30cc004810 2024-09-16T15:46:59.752 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:59.744+0000 7f30e9880640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f30e4166dd0 con 0x7f30e41f7340 2024-09-16T15:46:59.752 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:59.744+0000 7f30e27fc640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f30e40b1070 0x7f30e41f6e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:46:59.752 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:59.748+0000 7f30e27fc640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f30e40b1070 0x7f30e41f6e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:40190/0 (socket says 172.21.15.123:40190) 2024-09-16T15:46:59.752 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:59.748+0000 7f30e27fc640 1 -- 172.21.15.123:0/4265462623 learned_addr learned my addr 172.21.15.123:0/4265462623 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:46:59.752 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:59.748+0000 7f30e2ffd640 1 --2- 172.21.15.123:0/4265462623 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f30cc004810 0x7f30e4156190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:46:59.752 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:59.748+0000 7f30e37fe640 1 --2- 172.21.15.123:0/4265462623 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f30e41f7340 0x7f30e40b29a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:46:59.752 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:59.748+0000 7f30e27fc640 1 -- 172.21.15.123:0/4265462623 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f30e41f7340 msgr2=0x7f30e40b29a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:46:59.752 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:59.748+0000 7f30e27fc640 1 --2- 172.21.15.123:0/4265462623 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f30e41f7340 0x7f30e40b29a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:59.752 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:59.748+0000 7f30e27fc640 1 -- 172.21.15.123:0/4265462623 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f30cc004810 msgr2=0x7f30e4156190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:46:59.753 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:59.748+0000 7f30e27fc640 1 --2- 172.21.15.123:0/4265462623 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f30cc004810 0x7f30e4156190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:59.753 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:59.748+0000 7f30e27fc640 1 -- 172.21.15.123:0/4265462623 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f30e41566d0 con 0x7f30e40b1070 2024-09-16T15:46:59.753 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:59.748+0000 7f30e2ffd640 1 --2- 172.21.15.123:0/4265462623 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f30cc004810 0x7f30e4156190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T15:46:59.753 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:59.748+0000 7f30e27fc640 1 --2- 172.21.15.123:0/4265462623 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f30e40b1070 0x7f30e41f6e00 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 crypto rx=0x7f30d80b1040 tx=0x7f30d8005630 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:46:59.753 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:59.748+0000 7f30cbfff640 1 -- 172.21.15.123:0/4265462623 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30d80ab040 con 0x7f30e40b1070 2024-09-16T15:46:59.753 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:59.748+0000 7f30cbfff640 1 -- 172.21.15.123:0/4265462623 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f30d80a5ba0 con 0x7f30e40b1070 2024-09-16T15:46:59.753 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:59.748+0000 7f30e9880640 1 -- 172.21.15.123:0/4265462623 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f30e41f6260 con 0x7f30e40b1070 2024-09-16T15:46:59.753 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:59.748+0000 7f30cbfff640 1 -- 172.21.15.123:0/4265462623 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30d8004260 con 0x7f30e40b1070 2024-09-16T15:46:59.753 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:59.748+0000 7f30e9880640 1 -- 172.21.15.123:0/4265462623 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f30e41f6820 con 0x7f30e40b1070 2024-09-16T15:46:59.754 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:59.748+0000 7f30e9880640 1 -- 172.21.15.123:0/4265462623 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f30e4073ef0 con 0x7f30e40b1070 2024-09-16T15:46:59.754 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:59.748+0000 7f30cbfff640 1 -- 172.21.15.123:0/4265462623 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 11) ==== 97386+0+0 (secure 0 0 0) 0x7f30d80c0020 con 0x7f30e40b1070 2024-09-16T15:46:59.754 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:59.748+0000 7f30cbfff640 1 --2- 172.21.15.123:0/4265462623 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f30d0073250 0x7f30d0075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:46:59.754 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:59.748+0000 7f30cbfff640 1 -- 172.21.15.123:0/4265462623 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(69..69 src has 1..69) ==== 5709+0+0 (secure 0 0 0) 0x7f30d812eb60 con 0x7f30e40b1070 2024-09-16T15:46:59.754 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:59.748+0000 7f30e2ffd640 1 --2- 172.21.15.123:0/4265462623 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f30d0073250 0x7f30d0075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:46:59.754 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:59.752+0000 7f30e2ffd640 1 --2- 172.21.15.123:0/4265462623 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f30d0073250 0x7f30d0075710 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f30d4005bd0 tx=0x7f30d400e420 comp rx=0 tx=0).ready entity=mgr.4135 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:46:59.758 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:59.752+0000 7f30cbfff640 1 -- 172.21.15.123:0/4265462623 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f30d8016af0 con 0x7f30e40b1070 2024-09-16T15:46:59.930 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:59.924+0000 7f30e9880640 1 -- 172.21.15.123:0/4265462623 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f30e40700e0 con 0x7f30e40b1070 2024-09-16T15:46:59.932 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:59.928+0000 7f30cbfff640 1 -- 172.21.15.123:0/4265462623 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v69) ==== 74+0+16801 (secure 0 0 0) 0x7f30d80f97c0 con 0x7f30e40b1070 2024-09-16T15:46:59.932 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-09-16T15:46:59.932 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":69,"fsid":"56077f89-18c8-486f-ae7e-eecc0bb7822c","created":"2024-09-16T15:42:26.364892+0000","modified":"2024-09-16T15:46:58.916559+0000","last_up_change":"2024-09-16T15:42:38.707234+0000","last_in_change":"2024-09-16T15:42:29.423662+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-16T15:42:39.287764+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_metadata","create_time":"2024-09-16T15:46:56.639820+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"69","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_data","create_time":"2024-09-16T15:46:57.893968+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"69","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9e7c1955-a607-4671-9be7-f2ef760ceeaa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6800","nonce":3882044969},{"type":"v1","addr":"172.21.15.123:6801","nonce":3882044969}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6802","nonce":3882044969},{"type":"v1","addr":"172.21.15.123:6803","nonce":3882044969}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6806","nonce":3882044969},{"type":"v1","addr":"172.21.15.123:6807","nonce":3882044969}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6804","nonce":3882044969},{"type":"v1","addr":"172.21.15.123:6805","nonce":3882044969}]},"public_addr":"172.21.15.123:6801/3882044969","cluster_addr":"172.21.15.123:6803/3882044969","heartbeat_back_addr":"172.21.15.123:6807/3882044969","heartbeat_front_addr":"172.21.15.123:6805/3882044969","state":["exists","up"]},{"osd":1,"uuid":"c540cfde-0977-4742-9b17-fd634668e010","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6816","nonce":4037001850},{"type":"v1","addr":"172.21.15.123:6817","nonce":4037001850}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6818","nonce":4037001850},{"type":"v1","addr":"172.21.15.123:6819","nonce":4037001850}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6822","nonce":4037001850},{"type":"v1","addr":"172.21.15.123:6823","nonce":4037001850}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6820","nonce":4037001850},{"type":"v1","addr":"172.21.15.123:6821","nonce":4037001850}]},"public_addr":"172.21.15.123:6817/4037001850","cluster_addr":"172.21.15.123:6819/4037001850","heartbeat_back_addr":"172.21.15.123:6823/4037001850","heartbeat_front_addr":"172.21.15.123:6821/4037001850","state":["exists","up"]},{"osd":2,"uuid":"9864a496-f169-486b-95c4-1d59162793a7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6824","nonce":1332143088},{"type":"v1","addr":"172.21.15.123:6825","nonce":1332143088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6826","nonce":1332143088},{"type":"v1","addr":"172.21.15.123:6827","nonce":1332143088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6830","nonce":1332143088},{"type":"v1","addr":"172.21.15.123:6831","nonce":1332143088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6828","nonce":1332143088},{"type":"v1","addr":"172.21.15.123:6829","nonce":1332143088}]},"public_addr":"172.21.15.123:6825/1332143088","cluster_addr":"172.21.15.123:6827/1332143088","heartbeat_back_addr":"172.21.15.123:6831/1332143088","heartbeat_front_addr":"172.21.15.123:6829/1332143088","state":["exists","up"]},{"osd":3,"uuid":"c6d1f817-f074-4325-95e1-578eba13edda","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6808","nonce":988586998},{"type":"v1","addr":"172.21.15.123:6809","nonce":988586998}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6810","nonce":988586998},{"type":"v1","addr":"172.21.15.123:6811","nonce":988586998}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6814","nonce":988586998},{"type":"v1","addr":"172.21.15.123:6815","nonce":988586998}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6812","nonce":988586998},{"type":"v1","addr":"172.21.15.123:6813","nonce":988586998}]},"public_addr":"172.21.15.123:6809/988586998","cluster_addr":"172.21.15.123:6811/988586998","heartbeat_back_addr":"172.21.15.123:6815/988586998","heartbeat_front_addr":"172.21.15.123:6813/988586998","state":["exists","up"]},{"osd":4,"uuid":"200a8df4-50ef-40a5-b48a-58981269c086","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6816","nonce":3387813890},{"type":"v1","addr":"172.21.15.184:6817","nonce":3387813890}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6818","nonce":3387813890},{"type":"v1","addr":"172.21.15.184:6819","nonce":3387813890}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6822","nonce":3387813890},{"type":"v1","addr":"172.21.15.184:6823","nonce":3387813890}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6820","nonce":3387813890},{"type":"v1","addr":"172.21.15.184:6821","nonce":3387813890}]},"public_addr":"172.21.15.184:6817/3387813890","cluster_addr":"172.21.15.184:6819/3387813890","heartbeat_back_addr":"172.21.15.184:6823/3387813890","heartbeat_front_addr":"172.21.15.184:6821/3387813890","state":["exists","up"]},{"osd":5,"uuid":"32d5a900-da3d-412c-bc19-13861a2368d8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6808","nonce":2680916017},{"type":"v1","addr":"172.21.15.184:6809","nonce":2680916017}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6810","nonce":2680916017},{"type":"v1","addr":"172.21.15.184:6811","nonce":2680916017}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6814","nonce":2680916017},{"type":"v1","addr":"172.21.15.184:6815","nonce":2680916017}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6812","nonce":2680916017},{"type":"v1","addr":"172.21.15.184:6813","nonce":2680916017}]},"public_addr":"172.21.15.184:6809/2680916017","cluster_addr":"172.21.15.184:6811/2680916017","heartbeat_back_addr":"172.21.15.184:6815/2680916017","heartbeat_front_addr":"172.21.15.184:6813/2680916017","state":["exists","up"]},{"osd":6,"uuid":"cde6f39e-aa1b-4260-ac19-774ee6aaca72","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6824","nonce":2183120457},{"type":"v1","addr":"172.21.15.184:6825","nonce":2183120457}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6826","nonce":2183120457},{"type":"v1","addr":"172.21.15.184:6827","nonce":2183120457}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6830","nonce":2183120457},{"type":"v1","addr":"172.21.15.184:6831","nonce":2183120457}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6828","nonce":2183120457},{"type":"v1","addr":"172.21.15.184:6829","nonce":2183120457}]},"public_addr":"172.21.15.184:6825/2183120457","cluster_addr":"172.21.15.184:6827/2183120457","heartbeat_back_addr":"172.21.15.184:6831/2183120457","heartbeat_front_addr":"172.21.15.184:6829/2183120457","state":["exists","up"]},{"osd":7,"uuid":"4f381778-eee1-47c6-804f-df0e6b33e313","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6800","nonce":4102957285},{"type":"v1","addr":"172.21.15.184:6801","nonce":4102957285}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6802","nonce":4102957285},{"type":"v1","addr":"172.21.15.184:6803","nonce":4102957285}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6806","nonce":4102957285},{"type":"v1","addr":"172.21.15.184:6807","nonce":4102957285}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6804","nonce":4102957285},{"type":"v1","addr":"172.21.15.184:6805","nonce":4102957285}]},"public_addr":"172.21.15.184:6801/4102957285","cluster_addr":"172.21.15.184:6803/4102957285","heartbeat_back_addr":"172.21.15.184:6807/4102957285","heartbeat_front_addr":"172.21.15.184:6805/4102957285","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:35.972210+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.222007+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.198617+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.385663+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.358935+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.156066+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.752973+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:35.682846+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T15:46:59.934 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:59.928+0000 7f30e9880640 1 -- 172.21.15.123:0/4265462623 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f30d0073250 msgr2=0x7f30d0075710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:46:59.934 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:59.928+0000 7f30e9880640 1 --2- 172.21.15.123:0/4265462623 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f30d0073250 0x7f30d0075710 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f30d4005bd0 tx=0x7f30d400e420 comp rx=0 tx=0).stop 2024-09-16T15:46:59.934 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:59.928+0000 7f30e9880640 1 -- 172.21.15.123:0/4265462623 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f30e40b1070 msgr2=0x7f30e41f6e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:46:59.934 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:59.928+0000 7f30e9880640 1 --2- 172.21.15.123:0/4265462623 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f30e40b1070 0x7f30e41f6e00 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 crypto rx=0x7f30d80b1040 tx=0x7f30d8005630 comp rx=0 tx=0).stop 2024-09-16T15:46:59.935 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:59.928+0000 7f30e9880640 1 -- 172.21.15.123:0/4265462623 shutdown_connections 2024-09-16T15:46:59.935 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:59.928+0000 7f30e9880640 1 --2- 172.21.15.123:0/4265462623 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f30d0073250 0x7f30d0075710 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:59.935 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:59.928+0000 7f30e9880640 1 --2- 172.21.15.123:0/4265462623 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f30e41f7340 0x7f30e40b29a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:59.935 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:59.928+0000 7f30e9880640 1 --2- 172.21.15.123:0/4265462623 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f30e40b1070 0x7f30e41f6e00 unknown :-1 s=CLOSED pgs=771 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:59.935 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:59.928+0000 7f30e9880640 1 --2- 172.21.15.123:0/4265462623 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f30cc004810 0x7f30e4156190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:46:59.935 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:59.928+0000 7f30e9880640 1 -- 172.21.15.123:0/4265462623 >> 172.21.15.123:0/4265462623 conn(0x7f30e40cfb10 msgr2=0x7f30e40cb170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:46:59.935 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:59.928+0000 7f30e9880640 1 -- 172.21.15.123:0/4265462623 shutdown_connections 2024-09-16T15:46:59.935 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:46:59.928+0000 7f30e9880640 1 -- 172.21.15.123:0/4265462623 wait complete. 2024-09-16T15:46:59.952 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-16T15:47:00.073 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:00.068+0000 7f871a4d4640 1 -- 172.21.15.123:0/2829287667 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f8714161520 msgr2=0x7f87140cba60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:47:00.073 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:00.068+0000 7f871a4d4640 1 --2- 172.21.15.123:0/2829287667 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f8714161520 0x7f87140cba60 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f8708009900 tx=0x7f870801c7c0 comp rx=0 tx=0).stop 2024-09-16T15:47:00.073 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:00.068+0000 7f871a4d4640 1 -- 172.21.15.123:0/2829287667 shutdown_connections 2024-09-16T15:47:00.073 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:00.068+0000 7f871a4d4640 1 --2- 172.21.15.123:0/2829287667 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f87140cc830 0x7f87140741c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:00.074 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:00.068+0000 7f871a4d4640 1 --2- 172.21.15.123:0/2829287667 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f8714161520 0x7f87140cba60 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:00.074 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:00.068+0000 7f871a4d4640 1 -- 172.21.15.123:0/2829287667 >> 172.21.15.123:0/2829287667 conn(0x7f87140b1df0 msgr2=0x7f87140b2200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:47:00.074 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:00.068+0000 7f871a4d4640 1 -- 172.21.15.123:0/2829287667 shutdown_connections 2024-09-16T15:47:00.074 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:00.068+0000 7f871a4d4640 1 -- 172.21.15.123:0/2829287667 wait complete. 2024-09-16T15:47:00.074 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:00.068+0000 7f871a4d4640 1 Processor -- start 2024-09-16T15:47:00.074 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:00.068+0000 7f871a4d4640 1 -- start start 2024-09-16T15:47:00.076 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:00.072+0000 7f871a4d4640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f87140b2910 0x7f8714099af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:47:00.076 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:00.072+0000 7f871a4d4640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f87140cc830 0x7f871406ee70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:47:00.076 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:00.072+0000 7f871a4d4640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f8714161520 0x7f871406f3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:47:00.077 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:00.072+0000 7f871a4d4640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f871416b610 con 0x7f87140b2910 2024-09-16T15:47:00.077 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:00.072+0000 7f871a4d4640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f8714074a80 con 0x7f8714161520 2024-09-16T15:47:00.077 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:00.072+0000 7f871a4d4640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f871416b790 con 0x7f87140cc830 2024-09-16T15:47:00.077 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:00.072+0000 7f8713fff640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f87140b2910 0x7f8714099af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:47:00.077 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:00.072+0000 7f8713fff640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f87140b2910 0x7f8714099af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:40198/0 (socket says 172.21.15.123:40198) 2024-09-16T15:47:00.077 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:00.072+0000 7f8713fff640 1 -- 172.21.15.123:0/346016106 learned_addr learned my addr 172.21.15.123:0/346016106 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:47:00.077 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:00.072+0000 7f87137fe640 1 --2- 172.21.15.123:0/346016106 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f87140cc830 0x7f871406ee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:47:00.077 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:00.072+0000 7f8718a71640 1 --2- 172.21.15.123:0/346016106 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f8714161520 0x7f871406f3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:47:00.077 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:00.072+0000 7f8713fff640 1 -- 172.21.15.123:0/346016106 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f87140cc830 msgr2=0x7f871406ee70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:47:00.077 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:00.072+0000 7f8713fff640 1 --2- 172.21.15.123:0/346016106 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f87140cc830 0x7f871406ee70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:00.077 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:00.072+0000 7f8713fff640 1 -- 172.21.15.123:0/346016106 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f8714161520 msgr2=0x7f871406f3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:47:00.077 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:00.072+0000 7f8713fff640 1 --2- 172.21.15.123:0/346016106 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f8714161520 0x7f871406f3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:00.077 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:00.072+0000 7f8713fff640 1 -- 172.21.15.123:0/346016106 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f87140c8d00 con 0x7f87140b2910 2024-09-16T15:47:00.078 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:00.072+0000 7f87137fe640 1 --2- 172.21.15.123:0/346016106 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f87140cc830 0x7f871406ee70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T15:47:00.078 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:00.072+0000 7f8713fff640 1 --2- 172.21.15.123:0/346016106 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f87140b2910 0x7f8714099af0 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 crypto rx=0x7f8708009850 tx=0x7f87080a5790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:47:00.078 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:00.072+0000 7f8710ff9640 1 -- 172.21.15.123:0/346016106 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87080b2040 con 0x7f87140b2910 2024-09-16T15:47:00.078 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:00.072+0000 7f8710ff9640 1 -- 172.21.15.123:0/346016106 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f8708005020 con 0x7f87140b2910 2024-09-16T15:47:00.078 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:00.072+0000 7f871a4d4640 1 -- 172.21.15.123:0/346016106 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8714157580 con 0x7f87140b2910 2024-09-16T15:47:00.078 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:00.072+0000 7f8710ff9640 1 -- 172.21.15.123:0/346016106 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87080044f0 con 0x7f87140b2910 2024-09-16T15:47:00.078 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:00.072+0000 7f871a4d4640 1 -- 172.21.15.123:0/346016106 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8714157b40 con 0x7f87140b2910 2024-09-16T15:47:00.079 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:00.072+0000 7f871a4d4640 1 -- 172.21.15.123:0/346016106 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f87140741c0 con 0x7f87140b2910 2024-09-16T15:47:00.079 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:00.072+0000 7f8710ff9640 1 -- 172.21.15.123:0/346016106 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 11) ==== 97386+0+0 (secure 0 0 0) 0x7f8708005430 con 0x7f87140b2910 2024-09-16T15:47:00.079 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:00.072+0000 7f8710ff9640 1 --2- 172.21.15.123:0/346016106 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f8700073220 0x7f87000756e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:47:00.079 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:00.072+0000 7f8710ff9640 1 -- 172.21.15.123:0/346016106 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(69..69 src has 1..69) ==== 5709+0+0 (secure 0 0 0) 0x7f870812dfc0 con 0x7f87140b2910 2024-09-16T15:47:00.079 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:00.076+0000 7f87137fe640 1 --2- 172.21.15.123:0/346016106 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f8700073220 0x7f87000756e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:47:00.079 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:00.076+0000 7f87137fe640 1 --2- 172.21.15.123:0/346016106 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f8700073220 0x7f87000756e0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f8714160d60 tx=0x7f870c0091e0 comp rx=0 tx=0).ready entity=mgr.4135 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:47:00.082 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:00.076+0000 7f8710ff9640 1 -- 172.21.15.123:0/346016106 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f87080164d0 con 0x7f87140b2910 2024-09-16T15:47:00.253 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:00.248+0000 7f871a4d4640 1 -- 172.21.15.123:0/346016106 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f871406dd90 con 0x7f87140b2910 2024-09-16T15:47:00.255 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:00.252+0000 7f8710ff9640 1 -- 172.21.15.123:0/346016106 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v69) ==== 74+0+16801 (secure 0 0 0) 0x7f87080f8e80 con 0x7f87140b2910 2024-09-16T15:47:00.256 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-09-16T15:47:00.256 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":69,"fsid":"56077f89-18c8-486f-ae7e-eecc0bb7822c","created":"2024-09-16T15:42:26.364892+0000","modified":"2024-09-16T15:46:58.916559+0000","last_up_change":"2024-09-16T15:42:38.707234+0000","last_in_change":"2024-09-16T15:42:29.423662+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-16T15:42:39.287764+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_metadata","create_time":"2024-09-16T15:46:56.639820+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"69","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_data","create_time":"2024-09-16T15:46:57.893968+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"69","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9e7c1955-a607-4671-9be7-f2ef760ceeaa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6800","nonce":3882044969},{"type":"v1","addr":"172.21.15.123:6801","nonce":3882044969}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6802","nonce":3882044969},{"type":"v1","addr":"172.21.15.123:6803","nonce":3882044969}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6806","nonce":3882044969},{"type":"v1","addr":"172.21.15.123:6807","nonce":3882044969}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6804","nonce":3882044969},{"type":"v1","addr":"172.21.15.123:6805","nonce":3882044969}]},"public_addr":"172.21.15.123:6801/3882044969","cluster_addr":"172.21.15.123:6803/3882044969","heartbeat_back_addr":"172.21.15.123:6807/3882044969","heartbeat_front_addr":"172.21.15.123:6805/3882044969","state":["exists","up"]},{"osd":1,"uuid":"c540cfde-0977-4742-9b17-fd634668e010","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6816","nonce":4037001850},{"type":"v1","addr":"172.21.15.123:6817","nonce":4037001850}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6818","nonce":4037001850},{"type":"v1","addr":"172.21.15.123:6819","nonce":4037001850}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6822","nonce":4037001850},{"type":"v1","addr":"172.21.15.123:6823","nonce":4037001850}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6820","nonce":4037001850},{"type":"v1","addr":"172.21.15.123:6821","nonce":4037001850}]},"public_addr":"172.21.15.123:6817/4037001850","cluster_addr":"172.21.15.123:6819/4037001850","heartbeat_back_addr":"172.21.15.123:6823/4037001850","heartbeat_front_addr":"172.21.15.123:6821/4037001850","state":["exists","up"]},{"osd":2,"uuid":"9864a496-f169-486b-95c4-1d59162793a7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6824","nonce":1332143088},{"type":"v1","addr":"172.21.15.123:6825","nonce":1332143088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6826","nonce":1332143088},{"type":"v1","addr":"172.21.15.123:6827","nonce":1332143088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6830","nonce":1332143088},{"type":"v1","addr":"172.21.15.123:6831","nonce":1332143088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6828","nonce":1332143088},{"type":"v1","addr":"172.21.15.123:6829","nonce":1332143088}]},"public_addr":"172.21.15.123:6825/1332143088","cluster_addr":"172.21.15.123:6827/1332143088","heartbeat_back_addr":"172.21.15.123:6831/1332143088","heartbeat_front_addr":"172.21.15.123:6829/1332143088","state":["exists","up"]},{"osd":3,"uuid":"c6d1f817-f074-4325-95e1-578eba13edda","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6808","nonce":988586998},{"type":"v1","addr":"172.21.15.123:6809","nonce":988586998}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6810","nonce":988586998},{"type":"v1","addr":"172.21.15.123:6811","nonce":988586998}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6814","nonce":988586998},{"type":"v1","addr":"172.21.15.123:6815","nonce":988586998}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6812","nonce":988586998},{"type":"v1","addr":"172.21.15.123:6813","nonce":988586998}]},"public_addr":"172.21.15.123:6809/988586998","cluster_addr":"172.21.15.123:6811/988586998","heartbeat_back_addr":"172.21.15.123:6815/988586998","heartbeat_front_addr":"172.21.15.123:6813/988586998","state":["exists","up"]},{"osd":4,"uuid":"200a8df4-50ef-40a5-b48a-58981269c086","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6816","nonce":3387813890},{"type":"v1","addr":"172.21.15.184:6817","nonce":3387813890}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6818","nonce":3387813890},{"type":"v1","addr":"172.21.15.184:6819","nonce":3387813890}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6822","nonce":3387813890},{"type":"v1","addr":"172.21.15.184:6823","nonce":3387813890}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6820","nonce":3387813890},{"type":"v1","addr":"172.21.15.184:6821","nonce":3387813890}]},"public_addr":"172.21.15.184:6817/3387813890","cluster_addr":"172.21.15.184:6819/3387813890","heartbeat_back_addr":"172.21.15.184:6823/3387813890","heartbeat_front_addr":"172.21.15.184:6821/3387813890","state":["exists","up"]},{"osd":5,"uuid":"32d5a900-da3d-412c-bc19-13861a2368d8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6808","nonce":2680916017},{"type":"v1","addr":"172.21.15.184:6809","nonce":2680916017}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6810","nonce":2680916017},{"type":"v1","addr":"172.21.15.184:6811","nonce":2680916017}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6814","nonce":2680916017},{"type":"v1","addr":"172.21.15.184:6815","nonce":2680916017}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6812","nonce":2680916017},{"type":"v1","addr":"172.21.15.184:6813","nonce":2680916017}]},"public_addr":"172.21.15.184:6809/2680916017","cluster_addr":"172.21.15.184:6811/2680916017","heartbeat_back_addr":"172.21.15.184:6815/2680916017","heartbeat_front_addr":"172.21.15.184:6813/2680916017","state":["exists","up"]},{"osd":6,"uuid":"cde6f39e-aa1b-4260-ac19-774ee6aaca72","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6824","nonce":2183120457},{"type":"v1","addr":"172.21.15.184:6825","nonce":2183120457}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6826","nonce":2183120457},{"type":"v1","addr":"172.21.15.184:6827","nonce":2183120457}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6830","nonce":2183120457},{"type":"v1","addr":"172.21.15.184:6831","nonce":2183120457}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6828","nonce":2183120457},{"type":"v1","addr":"172.21.15.184:6829","nonce":2183120457}]},"public_addr":"172.21.15.184:6825/2183120457","cluster_addr":"172.21.15.184:6827/2183120457","heartbeat_back_addr":"172.21.15.184:6831/2183120457","heartbeat_front_addr":"172.21.15.184:6829/2183120457","state":["exists","up"]},{"osd":7,"uuid":"4f381778-eee1-47c6-804f-df0e6b33e313","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6800","nonce":4102957285},{"type":"v1","addr":"172.21.15.184:6801","nonce":4102957285}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6802","nonce":4102957285},{"type":"v1","addr":"172.21.15.184:6803","nonce":4102957285}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6806","nonce":4102957285},{"type":"v1","addr":"172.21.15.184:6807","nonce":4102957285}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6804","nonce":4102957285},{"type":"v1","addr":"172.21.15.184:6805","nonce":4102957285}]},"public_addr":"172.21.15.184:6801/4102957285","cluster_addr":"172.21.15.184:6803/4102957285","heartbeat_back_addr":"172.21.15.184:6807/4102957285","heartbeat_front_addr":"172.21.15.184:6805/4102957285","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:35.972210+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.222007+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.198617+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.385663+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.358935+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.156066+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.752973+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:35.682846+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T15:47:00.258 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:00.252+0000 7f871a4d4640 1 -- 172.21.15.123:0/346016106 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f8700073220 msgr2=0x7f87000756e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:47:00.258 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:00.252+0000 7f871a4d4640 1 --2- 172.21.15.123:0/346016106 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f8700073220 0x7f87000756e0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f8714160d60 tx=0x7f870c0091e0 comp rx=0 tx=0).stop 2024-09-16T15:47:00.258 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:00.252+0000 7f871a4d4640 1 -- 172.21.15.123:0/346016106 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f87140b2910 msgr2=0x7f8714099af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:47:00.258 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:00.252+0000 7f871a4d4640 1 --2- 172.21.15.123:0/346016106 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f87140b2910 0x7f8714099af0 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 crypto rx=0x7f8708009850 tx=0x7f87080a5790 comp rx=0 tx=0).stop 2024-09-16T15:47:00.258 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:00.252+0000 7f871a4d4640 1 -- 172.21.15.123:0/346016106 shutdown_connections 2024-09-16T15:47:00.258 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:00.252+0000 7f871a4d4640 1 --2- 172.21.15.123:0/346016106 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f8700073220 0x7f87000756e0 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:00.258 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:00.252+0000 7f871a4d4640 1 --2- 172.21.15.123:0/346016106 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f8714161520 0x7f871406f3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:00.258 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:00.252+0000 7f871a4d4640 1 --2- 172.21.15.123:0/346016106 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f87140cc830 0x7f871406ee70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:00.258 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:00.252+0000 7f871a4d4640 1 --2- 172.21.15.123:0/346016106 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f87140b2910 0x7f8714099af0 unknown :-1 s=CLOSED pgs=772 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:00.259 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:00.252+0000 7f871a4d4640 1 -- 172.21.15.123:0/346016106 >> 172.21.15.123:0/346016106 conn(0x7f87140b1df0 msgr2=0x7f871415fda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:47:00.259 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:00.252+0000 7f871a4d4640 1 -- 172.21.15.123:0/346016106 shutdown_connections 2024-09-16T15:47:00.259 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:00.252+0000 7f871a4d4640 1 -- 172.21.15.123:0/346016106 wait complete. 2024-09-16T15:47:00.274 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-09-16T15:47:00.387 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:00.380+0000 7f5b878bd640 1 -- 172.21.15.123:0/1483323105 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5b6c003a10 msgr2=0x7f5b6c003e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:47:00.387 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:00.380+0000 7f5b878bd640 1 --2- 172.21.15.123:0/1483323105 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5b6c003a10 0x7f5b6c003e30 secure :-1 s=READY pgs=774 cs=0 l=1 rev1=1 crypto rx=0x7f5b7c00c290 tx=0x7f5b7c01cf90 comp rx=0 tx=0).stop 2024-09-16T15:47:00.387 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:00.380+0000 7f5b85e5a640 1 -- 172.21.15.123:0/1483323105 reap_dead start 2024-09-16T15:47:00.387 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:00.380+0000 7f5b878bd640 1 -- 172.21.15.123:0/1483323105 shutdown_connections 2024-09-16T15:47:00.387 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:00.380+0000 7f5b878bd640 1 -- 172.21.15.123:0/1483323105 >> 172.21.15.123:0/1483323105 conn(0x7f5b800d0bf0 msgr2=0x7f5b800d1000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:47:00.387 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:00.384+0000 7f5b878bd640 1 -- 172.21.15.123:0/1483323105 shutdown_connections 2024-09-16T15:47:00.387 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:00.384+0000 7f5b878bd640 1 -- 172.21.15.123:0/1483323105 wait complete. 2024-09-16T15:47:00.389 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:00.384+0000 7f5b878bd640 1 Processor -- start 2024-09-16T15:47:00.389 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:00.384+0000 7f5b878bd640 1 -- start start 2024-09-16T15:47:00.391 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:00.384+0000 7f5b878bd640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5b6c003a10 0x7f5b80151100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:47:00.391 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:00.384+0000 7f5b878bd640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f5b8014f9a0 0x7f5b80153580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:47:00.392 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:00.384+0000 7f5b878bd640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5b80153ac0 0x7f5b800b0790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:47:00.392 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:00.384+0000 7f5b878bd640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f5b800c7330 con 0x7f5b80153ac0 2024-09-16T15:47:00.392 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:00.384+0000 7f5b878bd640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f5b800c71b0 con 0x7f5b6c003a10 2024-09-16T15:47:00.392 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:00.384+0000 7f5b878bd640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f5b800c74b0 con 0x7f5b8014f9a0 2024-09-16T15:47:00.392 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:00.384+0000 7f5b85e5a640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5b80153ac0 0x7f5b800b0790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:47:00.392 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:00.384+0000 7f5b85659640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5b6c003a10 0x7f5b80151100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:47:00.392 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:00.384+0000 7f5b85e5a640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5b80153ac0 0x7f5b800b0790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:40214/0 (socket says 172.21.15.123:40214) 2024-09-16T15:47:00.392 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:00.384+0000 7f5b85e5a640 1 -- 172.21.15.123:0/2350179459 learned_addr learned my addr 172.21.15.123:0/2350179459 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:47:00.392 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:00.384+0000 7f5b84e58640 1 --2- 172.21.15.123:0/2350179459 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f5b8014f9a0 0x7f5b80153580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:47:00.392 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:00.384+0000 7f5b85e5a640 1 -- 172.21.15.123:0/2350179459 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f5b8014f9a0 msgr2=0x7f5b80153580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:47:00.392 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:00.384+0000 7f5b85e5a640 1 --2- 172.21.15.123:0/2350179459 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f5b8014f9a0 0x7f5b80153580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:00.392 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:00.384+0000 7f5b85e5a640 1 -- 172.21.15.123:0/2350179459 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5b6c003a10 msgr2=0x7f5b80151100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:47:00.393 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:00.384+0000 7f5b85e5a640 1 --2- 172.21.15.123:0/2350179459 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5b6c003a10 0x7f5b80151100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:00.393 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:00.384+0000 7f5b85e5a640 1 -- 172.21.15.123:0/2350179459 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5b80150360 con 0x7f5b80153ac0 2024-09-16T15:47:00.393 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:00.388+0000 7f5b85e5a640 1 --2- 172.21.15.123:0/2350179459 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5b80153ac0 0x7f5b800b0790 secure :-1 s=READY pgs=775 cs=0 l=1 rev1=1 crypto rx=0x7f5b7800fbd0 tx=0x7f5b7800b690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:47:00.393 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:00.388+0000 7f5b75ffb640 1 -- 172.21.15.123:0/2350179459 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b780183f0 con 0x7f5b80153ac0 2024-09-16T15:47:00.393 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:00.388+0000 7f5b75ffb640 1 -- 172.21.15.123:0/2350179459 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f5b78018590 con 0x7f5b80153ac0 2024-09-16T15:47:00.393 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:00.388+0000 7f5b878bd640 1 -- 172.21.15.123:0/2350179459 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5b80154070 con 0x7f5b80153ac0 2024-09-16T15:47:00.393 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:00.388+0000 7f5b75ffb640 1 -- 172.21.15.123:0/2350179459 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b780189e0 con 0x7f5b80153ac0 2024-09-16T15:47:00.393 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:00.388+0000 7f5b878bd640 1 -- 172.21.15.123:0/2350179459 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5b80150900 con 0x7f5b80153ac0 2024-09-16T15:47:00.393 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:00.388+0000 7f5b878bd640 1 -- 172.21.15.123:0/2350179459 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5b8006dd70 con 0x7f5b80153ac0 2024-09-16T15:47:00.394 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:00.388+0000 7f5b75ffb640 1 -- 172.21.15.123:0/2350179459 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 11) ==== 97386+0+0 (secure 0 0 0) 0x7f5b78021400 con 0x7f5b80153ac0 2024-09-16T15:47:00.394 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:00.388+0000 7f5b75ffb640 1 --2- 172.21.15.123:0/2350179459 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f5b64073220 0x7f5b640756e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:47:00.394 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:00.388+0000 7f5b75ffb640 1 -- 172.21.15.123:0/2350179459 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(69..69 src has 1..69) ==== 5709+0+0 (secure 0 0 0) 0x7f5b78096b40 con 0x7f5b80153ac0 2024-09-16T15:47:00.394 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:00.388+0000 7f5b85659640 1 --2- 172.21.15.123:0/2350179459 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f5b64073220 0x7f5b640756e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:47:00.394 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:00.388+0000 7f5b85659640 1 --2- 172.21.15.123:0/2350179459 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f5b64073220 0x7f5b640756e0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f5b7c0ab000 tx=0x7f5b7c019de0 comp rx=0 tx=0).ready entity=mgr.4135 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:47:00.397 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:00.392+0000 7f5b75ffb640 1 -- 172.21.15.123:0/2350179459 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f5b78061800 con 0x7f5b80153ac0 2024-09-16T15:47:00.591 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:00.588+0000 7f5b878bd640 1 -- 172.21.15.123:0/2350179459 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) -- 0x7f5b8015dd30 con 0x7f5b80153ac0 2024-09-16T15:47:01.606 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:01.600+0000 7f5b75ffb640 1 -- 172.21.15.123:0/2350179459 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v43) ==== 118+0+0 (secure 0 0 0) 0x7f5b78061100 con 0x7f5b80153ac0 2024-09-16T15:47:01.610 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:01.604+0000 7f5b878bd640 1 -- 172.21.15.123:0/2350179459 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f5b64073220 msgr2=0x7f5b640756e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:47:01.610 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:01.604+0000 7f5b878bd640 1 --2- 172.21.15.123:0/2350179459 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f5b64073220 0x7f5b640756e0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f5b7c0ab000 tx=0x7f5b7c019de0 comp rx=0 tx=0).stop 2024-09-16T15:47:01.610 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:01.604+0000 7f5b878bd640 1 -- 172.21.15.123:0/2350179459 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5b80153ac0 msgr2=0x7f5b800b0790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:47:01.610 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:01.604+0000 7f5b878bd640 1 --2- 172.21.15.123:0/2350179459 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5b80153ac0 0x7f5b800b0790 secure :-1 s=READY pgs=775 cs=0 l=1 rev1=1 crypto rx=0x7f5b7800fbd0 tx=0x7f5b7800b690 comp rx=0 tx=0).stop 2024-09-16T15:47:01.610 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:01.604+0000 7f5b878bd640 1 -- 172.21.15.123:0/2350179459 shutdown_connections 2024-09-16T15:47:01.610 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:01.604+0000 7f5b878bd640 1 --2- 172.21.15.123:0/2350179459 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f5b64073220 0x7f5b640756e0 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:01.610 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:01.604+0000 7f5b878bd640 1 --2- 172.21.15.123:0/2350179459 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5b80153ac0 0x7f5b800b0790 unknown :-1 s=CLOSED pgs=775 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:01.610 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:01.604+0000 7f5b878bd640 1 --2- 172.21.15.123:0/2350179459 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f5b8014f9a0 0x7f5b80153580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:01.611 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:01.604+0000 7f5b878bd640 1 --2- 172.21.15.123:0/2350179459 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5b6c003a10 0x7f5b80151100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:01.611 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:01.604+0000 7f5b878bd640 1 -- 172.21.15.123:0/2350179459 >> 172.21.15.123:0/2350179459 conn(0x7f5b800d0bf0 msgr2=0x7f5b80149420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:47:01.611 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:01.604+0000 7f5b878bd640 1 -- 172.21.15.123:0/2350179459 shutdown_connections 2024-09-16T15:47:01.611 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:01.604+0000 7f5b878bd640 1 -- 172.21.15.123:0/2350179459 wait complete. 2024-09-16T15:47:01.629 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-09-16T15:47:01.629 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-09-16T15:47:01.759 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:01.756+0000 7f6a83d6c640 1 -- 172.21.15.123:0/2410247458 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6a7c15b860 msgr2=0x7f6a7c06fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:47:01.759 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:01.756+0000 7f6a83d6c640 1 --2- 172.21.15.123:0/2410247458 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6a7c15b860 0x7f6a7c06fca0 secure :-1 s=READY pgs=776 cs=0 l=1 rev1=1 crypto rx=0x7f6a7c0cd620 tx=0x7f6a7401c8c0 comp rx=0 tx=0).stop 2024-09-16T15:47:01.759 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:01.756+0000 7f6a83d6c640 1 -- 172.21.15.123:0/2410247458 shutdown_connections 2024-09-16T15:47:01.759 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:01.756+0000 7f6a83d6c640 1 --2- 172.21.15.123:0/2410247458 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f6a7c0701e0 0x7f6a7c0cf310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:01.760 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:01.756+0000 7f6a83d6c640 1 --2- 172.21.15.123:0/2410247458 >> v2:172.21.15.184:3300/0 conn(0x7f6a7c0c3230 0x7f6a7c0cedd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:01.760 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:01.756+0000 7f6a83d6c640 1 --2- 172.21.15.123:0/2410247458 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6a7c15b860 0x7f6a7c06fca0 unknown :-1 s=CLOSED pgs=776 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:01.760 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:01.756+0000 7f6a83d6c640 1 -- 172.21.15.123:0/2410247458 >> 172.21.15.123:0/2410247458 conn(0x7f6a7c0b1710 msgr2=0x7f6a7c0b1b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:47:01.760 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:01.756+0000 7f6a83d6c640 1 -- 172.21.15.123:0/2410247458 shutdown_connections 2024-09-16T15:47:01.760 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:01.756+0000 7f6a83d6c640 1 -- 172.21.15.123:0/2410247458 wait complete. 2024-09-16T15:47:01.760 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:01.756+0000 7f6a83d6c640 1 Processor -- start 2024-09-16T15:47:01.760 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:01.756+0000 7f6a83d6c640 1 -- start start 2024-09-16T15:47:01.763 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:01.756+0000 7f6a83d6c640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6a7c0701e0 0x7f6a7c1481e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:47:01.763 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:01.756+0000 7f6a83d6c640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f6a7c0c3230 0x7f6a7c150f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:47:01.763 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:01.756+0000 7f6a83d6c640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6a7c15b860 0x7f6a7c0c75f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:47:01.763 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:01.756+0000 7f6a83d6c640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f6a7c15fa40 con 0x7f6a7c0701e0 2024-09-16T15:47:01.763 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:01.756+0000 7f6a83d6c640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f6a7c15f8c0 con 0x7f6a7c15b860 2024-09-16T15:47:01.763 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:01.756+0000 7f6a83d6c640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f6a7c15fbc0 con 0x7f6a7c0c3230 2024-09-16T15:47:01.763 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:01.756+0000 7f6a81b08640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6a7c0701e0 0x7f6a7c1481e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:47:01.763 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:01.756+0000 7f6a81b08640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6a7c0701e0 0x7f6a7c1481e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:40232/0 (socket says 172.21.15.123:40232) 2024-09-16T15:47:01.763 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:01.756+0000 7f6a81b08640 1 -- 172.21.15.123:0/2619118398 learned_addr learned my addr 172.21.15.123:0/2619118398 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:47:01.763 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:01.756+0000 7f6a81307640 1 --2- 172.21.15.123:0/2619118398 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f6a7c0c3230 0x7f6a7c150f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:47:01.763 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:01.756+0000 7f6a82309640 1 --2- 172.21.15.123:0/2619118398 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6a7c15b860 0x7f6a7c0c75f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:47:01.763 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:01.756+0000 7f6a81b08640 1 -- 172.21.15.123:0/2619118398 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f6a7c0c3230 msgr2=0x7f6a7c150f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:47:01.763 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:01.756+0000 7f6a81b08640 1 --2- 172.21.15.123:0/2619118398 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f6a7c0c3230 0x7f6a7c150f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:01.764 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:01.756+0000 7f6a81b08640 1 -- 172.21.15.123:0/2619118398 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6a7c15b860 msgr2=0x7f6a7c0c75f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:47:01.764 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:01.756+0000 7f6a81b08640 1 --2- 172.21.15.123:0/2619118398 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6a7c15b860 0x7f6a7c0c75f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:01.764 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:01.756+0000 7f6a81b08640 1 -- 172.21.15.123:0/2619118398 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6a7c148720 con 0x7f6a7c0701e0 2024-09-16T15:47:01.764 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:01.756+0000 7f6a81307640 1 --2- 172.21.15.123:0/2619118398 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f6a7c0c3230 0x7f6a7c150f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T15:47:01.764 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:01.756+0000 7f6a82309640 1 --2- 172.21.15.123:0/2619118398 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6a7c15b860 0x7f6a7c0c75f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T15:47:01.764 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:01.756+0000 7f6a81b08640 1 --2- 172.21.15.123:0/2619118398 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6a7c0701e0 0x7f6a7c1481e0 secure :-1 s=READY pgs=777 cs=0 l=1 rev1=1 crypto rx=0x7f6a740b0040 tx=0x7f6a740a57c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:47:01.764 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:01.756+0000 7f6a7a7fc640 1 -- 172.21.15.123:0/2619118398 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a740b2040 con 0x7f6a7c0701e0 2024-09-16T15:47:01.764 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:01.756+0000 7f6a7a7fc640 1 -- 172.21.15.123:0/2619118398 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f6a740044b0 con 0x7f6a7c0701e0 2024-09-16T15:47:01.764 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:01.756+0000 7f6a83d6c640 1 -- 172.21.15.123:0/2619118398 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6a7c0c99a0 con 0x7f6a7c0701e0 2024-09-16T15:47:01.764 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:01.756+0000 7f6a7a7fc640 1 -- 172.21.15.123:0/2619118398 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a74003860 con 0x7f6a7c0701e0 2024-09-16T15:47:01.765 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:01.756+0000 7f6a83d6c640 1 -- 172.21.15.123:0/2619118398 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6a7c0c6ef0 con 0x7f6a7c0701e0 2024-09-16T15:47:01.765 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:01.760+0000 7f6a83d6c640 1 -- 172.21.15.123:0/2619118398 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6a7c150820 con 0x7f6a7c0701e0 2024-09-16T15:47:01.765 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:01.760+0000 7f6a7a7fc640 1 -- 172.21.15.123:0/2619118398 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 11) ==== 97386+0+0 (secure 0 0 0) 0x7f6a74003a00 con 0x7f6a7c0701e0 2024-09-16T15:47:01.765 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:01.760+0000 7f6a7a7fc640 1 --2- 172.21.15.123:0/2619118398 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f6a68073250 0x7f6a68075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:47:01.766 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:01.760+0000 7f6a7a7fc640 1 -- 172.21.15.123:0/2619118398 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(70..70 src has 1..70) ==== 5709+0+0 (secure 0 0 0) 0x7f6a7412e260 con 0x7f6a7c0701e0 2024-09-16T15:47:01.766 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:01.760+0000 7f6a81307640 1 --2- 172.21.15.123:0/2619118398 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f6a68073250 0x7f6a68075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:47:01.766 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:01.760+0000 7f6a81307640 1 --2- 172.21.15.123:0/2619118398 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f6a68073250 0x7f6a68075710 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f6a70005cd0 tx=0x7f6a700082f0 comp rx=0 tx=0).ready entity=mgr.4135 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:47:01.768 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:01.764+0000 7f6a7a7fc640 1 -- 172.21.15.123:0/2619118398 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f6a740bd060 con 0x7f6a7c0701e0 2024-09-16T15:47:01.963 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:01.956+0000 7f6a83d6c640 1 -- 172.21.15.123:0/2619118398 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) -- 0x7f6a7c06fca0 con 0x7f6a7c0701e0 2024-09-16T15:47:02.608 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:02.604+0000 7f6a7a7fc640 1 -- 172.21.15.123:0/2619118398 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v44) ==== 122+0+0 (secure 0 0 0) 0x7f6a740164b0 con 0x7f6a7c0701e0 2024-09-16T15:47:02.612 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:02.608+0000 7f6a83d6c640 1 -- 172.21.15.123:0/2619118398 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f6a68073250 msgr2=0x7f6a68075710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:47:02.612 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:02.608+0000 7f6a83d6c640 1 --2- 172.21.15.123:0/2619118398 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f6a68073250 0x7f6a68075710 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f6a70005cd0 tx=0x7f6a700082f0 comp rx=0 tx=0).stop 2024-09-16T15:47:02.612 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:02.608+0000 7f6a83d6c640 1 -- 172.21.15.123:0/2619118398 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6a7c0701e0 msgr2=0x7f6a7c1481e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:47:02.613 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:02.608+0000 7f6a83d6c640 1 --2- 172.21.15.123:0/2619118398 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6a7c0701e0 0x7f6a7c1481e0 secure :-1 s=READY pgs=777 cs=0 l=1 rev1=1 crypto rx=0x7f6a740b0040 tx=0x7f6a740a57c0 comp rx=0 tx=0).stop 2024-09-16T15:47:02.613 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:02.608+0000 7f6a83d6c640 1 -- 172.21.15.123:0/2619118398 shutdown_connections 2024-09-16T15:47:02.613 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:02.608+0000 7f6a83d6c640 1 --2- 172.21.15.123:0/2619118398 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f6a68073250 0x7f6a68075710 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:02.613 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:02.608+0000 7f6a83d6c640 1 --2- 172.21.15.123:0/2619118398 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6a7c15b860 0x7f6a7c0c75f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:02.613 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:02.608+0000 7f6a83d6c640 1 --2- 172.21.15.123:0/2619118398 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f6a7c0c3230 0x7f6a7c150f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:02.613 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:02.608+0000 7f6a83d6c640 1 --2- 172.21.15.123:0/2619118398 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6a7c0701e0 0x7f6a7c1481e0 unknown :-1 s=CLOSED pgs=777 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:02.613 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:02.608+0000 7f6a83d6c640 1 -- 172.21.15.123:0/2619118398 >> 172.21.15.123:0/2619118398 conn(0x7f6a7c0b1710 msgr2=0x7f6a7c15c130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:47:02.613 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:02.608+0000 7f6a83d6c640 1 -- 172.21.15.123:0/2619118398 shutdown_connections 2024-09-16T15:47:02.613 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:02.608+0000 7f6a83d6c640 1 -- 172.21.15.123:0/2619118398 wait complete. 2024-09-16T15:47:02.632 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-09-16T15:47:02.632 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-16T15:47:02.758 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:02.752+0000 7fa47f217640 1 -- 172.21.15.123:0/743303337 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa47815bca0 msgr2=0x7fa4780d0630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:47:02.758 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:02.752+0000 7fa47f217640 1 --2- 172.21.15.123:0/743303337 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa47815bca0 0x7fa4780d0630 secure :-1 s=READY pgs=779 cs=0 l=1 rev1=1 crypto rx=0x7fa47806c970 tx=0x7fa46801c740 comp rx=0 tx=0).stop 2024-09-16T15:47:02.758 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:02.752+0000 7fa47f217640 1 -- 172.21.15.123:0/743303337 shutdown_connections 2024-09-16T15:47:02.758 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:02.752+0000 7fa47f217640 1 --2- 172.21.15.123:0/743303337 >> v2:172.21.15.184:3300/0 conn(0x7fa478074600 0x7fa47806e750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:02.758 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:02.752+0000 7fa47f217640 1 --2- 172.21.15.123:0/743303337 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa47815bca0 0x7fa4780d0630 unknown :-1 s=CLOSED pgs=779 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:02.758 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:02.752+0000 7fa47f217640 1 -- 172.21.15.123:0/743303337 >> 172.21.15.123:0/743303337 conn(0x7fa4780cfb10 msgr2=0x7fa4780cff20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:47:02.758 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:02.752+0000 7fa47f217640 1 -- 172.21.15.123:0/743303337 shutdown_connections 2024-09-16T15:47:02.758 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:02.752+0000 7fa47f217640 1 -- 172.21.15.123:0/743303337 wait complete. 2024-09-16T15:47:02.759 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:02.756+0000 7fa47f217640 1 Processor -- start 2024-09-16T15:47:02.759 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:02.756+0000 7fa47f217640 1 -- start start 2024-09-16T15:47:02.761 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:02.756+0000 7fa47f217640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa4780726c0 0x7fa4780cd6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:47:02.761 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:02.756+0000 7fa47f217640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fa478074600 0x7fa47815e8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:47:02.761 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:02.756+0000 7fa47f217640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa47815bca0 0x7fa4780c31a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:47:02.761 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:02.756+0000 7fa47f217640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fa478168bd0 con 0x7fa47815bca0 2024-09-16T15:47:02.761 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:02.756+0000 7fa47f217640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fa478168a50 con 0x7fa4780726c0 2024-09-16T15:47:02.761 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:02.756+0000 7fa47f217640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fa478168d50 con 0x7fa478074600 2024-09-16T15:47:02.762 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:02.756+0000 7fa47d7b4640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa47815bca0 0x7fa4780c31a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:47:02.762 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:02.756+0000 7fa47d7b4640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa47815bca0 0x7fa4780c31a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:46794/0 (socket says 172.21.15.123:46794) 2024-09-16T15:47:02.762 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:02.756+0000 7fa47d7b4640 1 -- 172.21.15.123:0/1903690883 learned_addr learned my addr 172.21.15.123:0/1903690883 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:47:02.762 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:02.756+0000 7fa47cfb3640 1 --2- 172.21.15.123:0/1903690883 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa4780726c0 0x7fa4780cd6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:47:02.762 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:02.756+0000 7fa477fff640 1 --2- 172.21.15.123:0/1903690883 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fa478074600 0x7fa47815e8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:47:02.762 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:02.756+0000 7fa47d7b4640 1 -- 172.21.15.123:0/1903690883 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fa478074600 msgr2=0x7fa47815e8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:47:02.762 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:02.756+0000 7fa47d7b4640 1 --2- 172.21.15.123:0/1903690883 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fa478074600 0x7fa47815e8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:02.762 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:02.756+0000 7fa47d7b4640 1 -- 172.21.15.123:0/1903690883 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa4780726c0 msgr2=0x7fa4780cd6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:47:02.762 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:02.756+0000 7fa47d7b4640 1 --2- 172.21.15.123:0/1903690883 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa4780726c0 0x7fa4780cd6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:02.762 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:02.756+0000 7fa47d7b4640 1 -- 172.21.15.123:0/1903690883 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa47815c300 con 0x7fa47815bca0 2024-09-16T15:47:02.762 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:02.756+0000 7fa477fff640 1 --2- 172.21.15.123:0/1903690883 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fa478074600 0x7fa47815e8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T15:47:02.762 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:02.756+0000 7fa47cfb3640 1 --2- 172.21.15.123:0/1903690883 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa4780726c0 0x7fa4780cd6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T15:47:02.762 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:02.756+0000 7fa47d7b4640 1 --2- 172.21.15.123:0/1903690883 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa47815bca0 0x7fa4780c31a0 secure :-1 s=READY pgs=780 cs=0 l=1 rev1=1 crypto rx=0x7fa47000ea20 tx=0x7fa47000eee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:47:02.763 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:02.756+0000 7fa4757fa640 1 -- 172.21.15.123:0/1903690883 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa47000cde0 con 0x7fa47815bca0 2024-09-16T15:47:02.763 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:02.756+0000 7fa4757fa640 1 -- 172.21.15.123:0/1903690883 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fa470004c40 con 0x7fa47815bca0 2024-09-16T15:47:02.763 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:02.756+0000 7fa47f217640 1 -- 172.21.15.123:0/1903690883 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa47815c5e0 con 0x7fa47815bca0 2024-09-16T15:47:02.763 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:02.756+0000 7fa4757fa640 1 -- 172.21.15.123:0/1903690883 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa470005280 con 0x7fa47815bca0 2024-09-16T15:47:02.763 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:02.756+0000 7fa47f217640 1 -- 172.21.15.123:0/1903690883 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa47815cba0 con 0x7fa47815bca0 2024-09-16T15:47:02.764 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:02.756+0000 7fa47f217640 1 -- 172.21.15.123:0/1903690883 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa47806e750 con 0x7fa47815bca0 2024-09-16T15:47:02.764 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:02.756+0000 7fa4757fa640 1 -- 172.21.15.123:0/1903690883 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 11) ==== 97386+0+0 (secure 0 0 0) 0x7fa470005420 con 0x7fa47815bca0 2024-09-16T15:47:02.764 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:02.760+0000 7fa4757fa640 1 --2- 172.21.15.123:0/1903690883 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fa464073250 0x7fa464075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:47:02.764 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:02.760+0000 7fa4757fa640 1 -- 172.21.15.123:0/1903690883 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(70..70 src has 1..70) ==== 5709+0+0 (secure 0 0 0) 0x7fa470094f00 con 0x7fa47815bca0 2024-09-16T15:47:02.764 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:02.760+0000 7fa47cfb3640 1 --2- 172.21.15.123:0/1903690883 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fa464073250 0x7fa464075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:47:02.765 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:02.760+0000 7fa47cfb3640 1 --2- 172.21.15.123:0/1903690883 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fa464073250 0x7fa464075710 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7fa4680af000 tx=0x7fa46801af80 comp rx=0 tx=0).ready entity=mgr.4135 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:47:02.767 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:02.764+0000 7fa4757fa640 1 -- 172.21.15.123:0/1903690883 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7fa470016030 con 0x7fa47815bca0 2024-09-16T15:47:02.978 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:02.972+0000 7fa47f217640 1 -- 172.21.15.123:0/1903690883 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa4780700e0 con 0x7fa47815bca0 2024-09-16T15:47:02.979 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:02.972+0000 7fa4757fa640 1 -- 172.21.15.123:0/1903690883 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 44 v44) ==== 94+0+5234 (secure 0 0 0) 0x7fa47005fb30 con 0x7fa47815bca0 2024-09-16T15:47:02.979 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-09-16T15:47:02.979 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":44,"btime":"2024-09-16T15:47:02:606606+0000","default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5048,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6837/1498917352","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6835","nonce":1498917352},{"type":"v1","addr":"172.21.15.184:6837","nonce":1498917352}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":40},{"gid":6198,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6833/1715139909","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6832","nonce":1715139909},{"type":"v1","addr":"172.21.15.123:6833","nonce":1715139909}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":40},{"gid":6201,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6835/815923338","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6834","nonce":815923338},{"type":"v1","addr":"172.21.15.123:6835","nonce":815923338}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":40}],"filesystems":[{"mdsmap":{"epoch":44,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-16T15:46:58.916477+0000","modified":"2024-09-16T15:47:01.963874+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6216},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6216":{"gid":6216,"name":"d","rank":0,"incarnation":41,"state":"up:active","state_seq":3,"addr":"172.21.15.184:6836/3137795986","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":3137795986},{"type":"v1","addr":"172.21.15.184:6836","nonce":3137795986}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6216,"qdb_cluster":[6216]},"id":5}]} 2024-09-16T15:47:02.979 INFO:teuthology.orchestra.run.smithi123.stderr:dumped fsmap epoch 44 2024-09-16T15:47:02.981 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:02.976+0000 7fa47f217640 1 -- 172.21.15.123:0/1903690883 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fa464073250 msgr2=0x7fa464075710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:47:02.981 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:02.976+0000 7fa47f217640 1 --2- 172.21.15.123:0/1903690883 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fa464073250 0x7fa464075710 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7fa4680af000 tx=0x7fa46801af80 comp rx=0 tx=0).stop 2024-09-16T15:47:02.982 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:02.976+0000 7fa47f217640 1 -- 172.21.15.123:0/1903690883 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa47815bca0 msgr2=0x7fa4780c31a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:47:02.982 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:02.976+0000 7fa47f217640 1 --2- 172.21.15.123:0/1903690883 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa47815bca0 0x7fa4780c31a0 secure :-1 s=READY pgs=780 cs=0 l=1 rev1=1 crypto rx=0x7fa47000ea20 tx=0x7fa47000eee0 comp rx=0 tx=0).stop 2024-09-16T15:47:02.982 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:02.976+0000 7fa47f217640 1 -- 172.21.15.123:0/1903690883 shutdown_connections 2024-09-16T15:47:02.982 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:02.976+0000 7fa47f217640 1 --2- 172.21.15.123:0/1903690883 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fa464073250 0x7fa464075710 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:02.982 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:02.976+0000 7fa47f217640 1 --2- 172.21.15.123:0/1903690883 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa47815bca0 0x7fa4780c31a0 unknown :-1 s=CLOSED pgs=780 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:02.982 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:02.976+0000 7fa47f217640 1 --2- 172.21.15.123:0/1903690883 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fa478074600 0x7fa47815e8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:02.982 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:02.976+0000 7fa47f217640 1 --2- 172.21.15.123:0/1903690883 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa4780726c0 0x7fa4780cd6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:02.982 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:02.976+0000 7fa47f217640 1 -- 172.21.15.123:0/1903690883 >> 172.21.15.123:0/1903690883 conn(0x7fa4780cfb10 msgr2=0x7fa478155340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:47:02.982 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:02.976+0000 7fa47f217640 1 -- 172.21.15.123:0/1903690883 shutdown_connections 2024-09-16T15:47:02.982 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:02.976+0000 7fa47f217640 1 -- 172.21.15.123:0/1903690883 wait complete. 2024-09-16T15:47:02.998 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-16T15:47:03.116 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:03.112+0000 7fd7273f6640 1 -- 172.21.15.123:0/772450883 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd70c0051d0 msgr2=0x7fd70c00b9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:47:03.116 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:03.112+0000 7fd7273f6640 1 --2- 172.21.15.123:0/772450883 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd70c0051d0 0x7fd70c00b9d0 secure :-1 s=READY pgs=782 cs=0 l=1 rev1=1 crypto rx=0x7fd718009e30 tx=0x7fd71801d1b0 comp rx=0 tx=0).stop 2024-09-16T15:47:03.117 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:03.112+0000 7fd725993640 1 -- 172.21.15.123:0/772450883 reap_dead start 2024-09-16T15:47:03.117 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:03.112+0000 7fd7273f6640 1 -- 172.21.15.123:0/772450883 shutdown_connections 2024-09-16T15:47:03.117 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:03.112+0000 7fd7273f6640 1 -- 172.21.15.123:0/772450883 >> 172.21.15.123:0/772450883 conn(0x7fd7200cfb10 msgr2=0x7fd7200cff20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:47:03.117 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:03.112+0000 7fd7273f6640 1 -- 172.21.15.123:0/772450883 shutdown_connections 2024-09-16T15:47:03.117 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:03.112+0000 7fd7273f6640 1 -- 172.21.15.123:0/772450883 wait complete. 2024-09-16T15:47:03.119 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:03.112+0000 7fd7273f6640 1 Processor -- start 2024-09-16T15:47:03.119 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:03.112+0000 7fd7273f6640 1 -- start start 2024-09-16T15:47:03.119 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:03.112+0000 7fd7273f6640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd720154f60 0x7fd72015c270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:47:03.119 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:03.112+0000 7fd7273f6640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd72015cbf0 0x7fd72015e880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:47:03.119 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:03.112+0000 7fd7273f6640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fd7200c7c30 0x7fd72015edc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:47:03.119 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:03.112+0000 7fd7273f6640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fd7200d09c0 con 0x7fd72015cbf0 2024-09-16T15:47:03.119 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:03.112+0000 7fd7273f6640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fd7200d0840 con 0x7fd720154f60 2024-09-16T15:47:03.119 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:03.112+0000 7fd7273f6640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fd72006ec90 con 0x7fd7200c7c30 2024-09-16T15:47:03.120 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:03.112+0000 7fd724991640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd72015cbf0 0x7fd72015e880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:47:03.120 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:03.112+0000 7fd724991640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd72015cbf0 0x7fd72015e880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:46812/0 (socket says 172.21.15.123:46812) 2024-09-16T15:47:03.120 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:03.112+0000 7fd724991640 1 -- 172.21.15.123:0/3807487861 learned_addr learned my addr 172.21.15.123:0/3807487861 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:47:03.120 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:03.112+0000 7fd725192640 1 --2- 172.21.15.123:0/3807487861 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd720154f60 0x7fd72015c270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:47:03.120 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:03.112+0000 7fd725993640 1 --2- 172.21.15.123:0/3807487861 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fd7200c7c30 0x7fd72015edc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:47:03.120 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:03.112+0000 7fd724991640 1 -- 172.21.15.123:0/3807487861 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fd7200c7c30 msgr2=0x7fd72015edc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:47:03.120 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:03.112+0000 7fd724991640 1 --2- 172.21.15.123:0/3807487861 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fd7200c7c30 0x7fd72015edc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:03.120 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:03.112+0000 7fd724991640 1 -- 172.21.15.123:0/3807487861 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd720154f60 msgr2=0x7fd72015c270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:47:03.120 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:03.112+0000 7fd724991640 1 --2- 172.21.15.123:0/3807487861 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd720154f60 0x7fd72015c270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:03.120 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:03.112+0000 7fd724991640 1 -- 172.21.15.123:0/3807487861 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd72015d080 con 0x7fd72015cbf0 2024-09-16T15:47:03.120 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:03.112+0000 7fd725192640 1 --2- 172.21.15.123:0/3807487861 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd720154f60 0x7fd72015c270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T15:47:03.120 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:03.112+0000 7fd724991640 1 --2- 172.21.15.123:0/3807487861 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd72015cbf0 0x7fd72015e880 secure :-1 s=READY pgs=783 cs=0 l=1 rev1=1 crypto rx=0x7fd71000f710 tx=0x7fd71000fbd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:47:03.120 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:03.112+0000 7fd715ffb640 1 -- 172.21.15.123:0/3807487861 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd710017070 con 0x7fd72015cbf0 2024-09-16T15:47:03.121 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:03.112+0000 7fd715ffb640 1 -- 172.21.15.123:0/3807487861 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fd71000acc0 con 0x7fd72015cbf0 2024-09-16T15:47:03.121 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:03.112+0000 7fd715ffb640 1 -- 172.21.15.123:0/3807487861 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7100126d0 con 0x7fd72015cbf0 2024-09-16T15:47:03.121 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:03.116+0000 7fd7273f6640 1 -- 172.21.15.123:0/3807487861 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd7200b0c40 con 0x7fd72015cbf0 2024-09-16T15:47:03.121 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:03.116+0000 7fd7273f6640 1 -- 172.21.15.123:0/3807487861 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd72015c7b0 con 0x7fd72015cbf0 2024-09-16T15:47:03.122 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:03.116+0000 7fd7273f6640 1 -- 172.21.15.123:0/3807487861 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd7200b1300 con 0x7fd72015cbf0 2024-09-16T15:47:03.122 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:03.116+0000 7fd715ffb640 1 -- 172.21.15.123:0/3807487861 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 11) ==== 97386+0+0 (secure 0 0 0) 0x7fd71001a070 con 0x7fd72015cbf0 2024-09-16T15:47:03.122 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:03.116+0000 7fd715ffb640 1 --2- 172.21.15.123:0/3807487861 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fd704073220 0x7fd7040756e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:47:03.122 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:03.116+0000 7fd715ffb640 1 -- 172.21.15.123:0/3807487861 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(70..70 src has 1..70) ==== 5709+0+0 (secure 0 0 0) 0x7fd710097670 con 0x7fd72015cbf0 2024-09-16T15:47:03.122 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:03.116+0000 7fd725192640 1 --2- 172.21.15.123:0/3807487861 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fd704073220 0x7fd7040756e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:47:03.122 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:03.116+0000 7fd725192640 1 --2- 172.21.15.123:0/3807487861 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fd704073220 0x7fd7040756e0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7fd7200cd650 tx=0x7fd71c005d30 comp rx=0 tx=0).ready entity=mgr.4135 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:47:03.125 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:03.120+0000 7fd715ffb640 1 -- 172.21.15.123:0/3807487861 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7fd71000fe20 con 0x7fd72015cbf0 2024-09-16T15:47:03.292 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:03.288+0000 7fd7273f6640 1 -- 172.21.15.123:0/3807487861 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fd7200ce0f0 con 0x7fd72015cbf0 2024-09-16T15:47:03.295 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:03.292+0000 7fd715ffb640 1 -- 172.21.15.123:0/3807487861 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v70) ==== 74+0+16788 (secure 0 0 0) 0x7fd710062330 con 0x7fd72015cbf0 2024-09-16T15:47:03.295 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-09-16T15:47:03.295 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":70,"fsid":"56077f89-18c8-486f-ae7e-eecc0bb7822c","created":"2024-09-16T15:42:26.364892+0000","modified":"2024-09-16T15:47:00.584449+0000","last_up_change":"2024-09-16T15:42:38.707234+0000","last_in_change":"2024-09-16T15:42:29.423662+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-16T15:42:39.287764+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_metadata","create_time":"2024-09-16T15:46:56.639820+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"69","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_data","create_time":"2024-09-16T15:46:57.893968+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"70","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9e7c1955-a607-4671-9be7-f2ef760ceeaa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6800","nonce":3882044969},{"type":"v1","addr":"172.21.15.123:6801","nonce":3882044969}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6802","nonce":3882044969},{"type":"v1","addr":"172.21.15.123:6803","nonce":3882044969}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6806","nonce":3882044969},{"type":"v1","addr":"172.21.15.123:6807","nonce":3882044969}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6804","nonce":3882044969},{"type":"v1","addr":"172.21.15.123:6805","nonce":3882044969}]},"public_addr":"172.21.15.123:6801/3882044969","cluster_addr":"172.21.15.123:6803/3882044969","heartbeat_back_addr":"172.21.15.123:6807/3882044969","heartbeat_front_addr":"172.21.15.123:6805/3882044969","state":["exists","up"]},{"osd":1,"uuid":"c540cfde-0977-4742-9b17-fd634668e010","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6816","nonce":4037001850},{"type":"v1","addr":"172.21.15.123:6817","nonce":4037001850}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6818","nonce":4037001850},{"type":"v1","addr":"172.21.15.123:6819","nonce":4037001850}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6822","nonce":4037001850},{"type":"v1","addr":"172.21.15.123:6823","nonce":4037001850}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6820","nonce":4037001850},{"type":"v1","addr":"172.21.15.123:6821","nonce":4037001850}]},"public_addr":"172.21.15.123:6817/4037001850","cluster_addr":"172.21.15.123:6819/4037001850","heartbeat_back_addr":"172.21.15.123:6823/4037001850","heartbeat_front_addr":"172.21.15.123:6821/4037001850","state":["exists","up"]},{"osd":2,"uuid":"9864a496-f169-486b-95c4-1d59162793a7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6824","nonce":1332143088},{"type":"v1","addr":"172.21.15.123:6825","nonce":1332143088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6826","nonce":1332143088},{"type":"v1","addr":"172.21.15.123:6827","nonce":1332143088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6830","nonce":1332143088},{"type":"v1","addr":"172.21.15.123:6831","nonce":1332143088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6828","nonce":1332143088},{"type":"v1","addr":"172.21.15.123:6829","nonce":1332143088}]},"public_addr":"172.21.15.123:6825/1332143088","cluster_addr":"172.21.15.123:6827/1332143088","heartbeat_back_addr":"172.21.15.123:6831/1332143088","heartbeat_front_addr":"172.21.15.123:6829/1332143088","state":["exists","up"]},{"osd":3,"uuid":"c6d1f817-f074-4325-95e1-578eba13edda","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6808","nonce":988586998},{"type":"v1","addr":"172.21.15.123:6809","nonce":988586998}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6810","nonce":988586998},{"type":"v1","addr":"172.21.15.123:6811","nonce":988586998}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6814","nonce":988586998},{"type":"v1","addr":"172.21.15.123:6815","nonce":988586998}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6812","nonce":988586998},{"type":"v1","addr":"172.21.15.123:6813","nonce":988586998}]},"public_addr":"172.21.15.123:6809/988586998","cluster_addr":"172.21.15.123:6811/988586998","heartbeat_back_addr":"172.21.15.123:6815/988586998","heartbeat_front_addr":"172.21.15.123:6813/988586998","state":["exists","up"]},{"osd":4,"uuid":"200a8df4-50ef-40a5-b48a-58981269c086","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6816","nonce":3387813890},{"type":"v1","addr":"172.21.15.184:6817","nonce":3387813890}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6818","nonce":3387813890},{"type":"v1","addr":"172.21.15.184:6819","nonce":3387813890}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6822","nonce":3387813890},{"type":"v1","addr":"172.21.15.184:6823","nonce":3387813890}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6820","nonce":3387813890},{"type":"v1","addr":"172.21.15.184:6821","nonce":3387813890}]},"public_addr":"172.21.15.184:6817/3387813890","cluster_addr":"172.21.15.184:6819/3387813890","heartbeat_back_addr":"172.21.15.184:6823/3387813890","heartbeat_front_addr":"172.21.15.184:6821/3387813890","state":["exists","up"]},{"osd":5,"uuid":"32d5a900-da3d-412c-bc19-13861a2368d8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6808","nonce":2680916017},{"type":"v1","addr":"172.21.15.184:6809","nonce":2680916017}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6810","nonce":2680916017},{"type":"v1","addr":"172.21.15.184:6811","nonce":2680916017}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6814","nonce":2680916017},{"type":"v1","addr":"172.21.15.184:6815","nonce":2680916017}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6812","nonce":2680916017},{"type":"v1","addr":"172.21.15.184:6813","nonce":2680916017}]},"public_addr":"172.21.15.184:6809/2680916017","cluster_addr":"172.21.15.184:6811/2680916017","heartbeat_back_addr":"172.21.15.184:6815/2680916017","heartbeat_front_addr":"172.21.15.184:6813/2680916017","state":["exists","up"]},{"osd":6,"uuid":"cde6f39e-aa1b-4260-ac19-774ee6aaca72","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6824","nonce":2183120457},{"type":"v1","addr":"172.21.15.184:6825","nonce":2183120457}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6826","nonce":2183120457},{"type":"v1","addr":"172.21.15.184:6827","nonce":2183120457}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6830","nonce":2183120457},{"type":"v1","addr":"172.21.15.184:6831","nonce":2183120457}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6828","nonce":2183120457},{"type":"v1","addr":"172.21.15.184:6829","nonce":2183120457}]},"public_addr":"172.21.15.184:6825/2183120457","cluster_addr":"172.21.15.184:6827/2183120457","heartbeat_back_addr":"172.21.15.184:6831/2183120457","heartbeat_front_addr":"172.21.15.184:6829/2183120457","state":["exists","up"]},{"osd":7,"uuid":"4f381778-eee1-47c6-804f-df0e6b33e313","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6800","nonce":4102957285},{"type":"v1","addr":"172.21.15.184:6801","nonce":4102957285}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6802","nonce":4102957285},{"type":"v1","addr":"172.21.15.184:6803","nonce":4102957285}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6806","nonce":4102957285},{"type":"v1","addr":"172.21.15.184:6807","nonce":4102957285}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6804","nonce":4102957285},{"type":"v1","addr":"172.21.15.184:6805","nonce":4102957285}]},"public_addr":"172.21.15.184:6801/4102957285","cluster_addr":"172.21.15.184:6803/4102957285","heartbeat_back_addr":"172.21.15.184:6807/4102957285","heartbeat_front_addr":"172.21.15.184:6805/4102957285","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:35.972210+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.222007+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.198617+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.385663+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.358935+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.156066+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.752973+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:35.682846+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T15:47:03.297 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:03.292+0000 7fd7273f6640 1 -- 172.21.15.123:0/3807487861 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fd704073220 msgr2=0x7fd7040756e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:47:03.297 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:03.292+0000 7fd7273f6640 1 --2- 172.21.15.123:0/3807487861 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fd704073220 0x7fd7040756e0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7fd7200cd650 tx=0x7fd71c005d30 comp rx=0 tx=0).stop 2024-09-16T15:47:03.298 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:03.292+0000 7fd7273f6640 1 -- 172.21.15.123:0/3807487861 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd72015cbf0 msgr2=0x7fd72015e880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:47:03.298 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:03.292+0000 7fd7273f6640 1 --2- 172.21.15.123:0/3807487861 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd72015cbf0 0x7fd72015e880 secure :-1 s=READY pgs=783 cs=0 l=1 rev1=1 crypto rx=0x7fd71000f710 tx=0x7fd71000fbd0 comp rx=0 tx=0).stop 2024-09-16T15:47:03.298 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:03.292+0000 7fd7273f6640 1 -- 172.21.15.123:0/3807487861 shutdown_connections 2024-09-16T15:47:03.298 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:03.292+0000 7fd7273f6640 1 --2- 172.21.15.123:0/3807487861 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fd704073220 0x7fd7040756e0 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:03.298 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:03.292+0000 7fd7273f6640 1 --2- 172.21.15.123:0/3807487861 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fd7200c7c30 0x7fd72015edc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:03.298 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:03.292+0000 7fd7273f6640 1 --2- 172.21.15.123:0/3807487861 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd72015cbf0 0x7fd72015e880 unknown :-1 s=CLOSED pgs=783 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:03.298 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:03.292+0000 7fd7273f6640 1 --2- 172.21.15.123:0/3807487861 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd720154f60 0x7fd72015c270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:03.298 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:03.292+0000 7fd7273f6640 1 -- 172.21.15.123:0/3807487861 >> 172.21.15.123:0/3807487861 conn(0x7fd7200cfb10 msgr2=0x7fd7200c5d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:47:03.298 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:03.292+0000 7fd7273f6640 1 -- 172.21.15.123:0/3807487861 shutdown_connections 2024-09-16T15:47:03.298 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:03.292+0000 7fd7273f6640 1 -- 172.21.15.123:0/3807487861 wait complete. 2024-09-16T15:47:03.313 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-09-16T15:47:03.314 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T15:47:03.428 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:03.424+0000 7f91ded8c640 1 -- 172.21.15.123:0/1602992459 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f91c80064e0 msgr2=0x7f91c800cae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:47:03.429 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:03.424+0000 7f91ded8c640 1 --2- 172.21.15.123:0/1602992459 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f91c80064e0 0x7f91c800cae0 secure :-1 s=READY pgs=785 cs=0 l=1 rev1=1 crypto rx=0x7f91d8072670 tx=0x7f91d001d150 comp rx=0 tx=0).stop 2024-09-16T15:47:03.429 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:03.424+0000 7f91ded8c640 1 -- 172.21.15.123:0/1602992459 shutdown_connections 2024-09-16T15:47:03.429 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:03.424+0000 7f91ded8c640 1 --2- 172.21.15.123:0/1602992459 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f91c80064e0 0x7f91c800cae0 unknown :-1 s=CLOSED pgs=785 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:03.429 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:03.424+0000 7f91ded8c640 1 -- 172.21.15.123:0/1602992459 >> 172.21.15.123:0/1602992459 conn(0x7f91d80cfb10 msgr2=0x7f91d80cff20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:47:03.429 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:03.424+0000 7f91ded8c640 1 -- 172.21.15.123:0/1602992459 shutdown_connections 2024-09-16T15:47:03.429 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:03.424+0000 7f91ded8c640 1 -- 172.21.15.123:0/1602992459 wait complete. 2024-09-16T15:47:03.429 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:03.424+0000 7f91ded8c640 1 Processor -- start 2024-09-16T15:47:03.430 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:03.424+0000 7f91ded8c640 1 -- start start 2024-09-16T15:47:03.432 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:03.424+0000 7f91ded8c640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f91c80064e0 0x7f91d814ef90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:47:03.432 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:03.424+0000 7f91ded8c640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f91d80739c0 0x7f91d8073e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:47:03.432 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:03.424+0000 7f91ded8c640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f91d81e8130 0x7f91d81e85b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:47:03.432 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:03.424+0000 7f91ded8c640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f91d80ca4a0 con 0x7f91d80739c0 2024-09-16T15:47:03.432 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:03.424+0000 7f91ded8c640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f91d80ca320 con 0x7f91d81e8130 2024-09-16T15:47:03.432 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:03.424+0000 7f91ded8c640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f91d80ca620 con 0x7f91c80064e0 2024-09-16T15:47:03.432 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:03.424+0000 7f91d7fff640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f91d80739c0 0x7f91d8073e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:47:03.432 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:03.424+0000 7f91d7fff640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f91d80739c0 0x7f91d8073e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:46828/0 (socket says 172.21.15.123:46828) 2024-09-16T15:47:03.432 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:03.424+0000 7f91d7fff640 1 -- 172.21.15.123:0/4134489404 learned_addr learned my addr 172.21.15.123:0/4134489404 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:47:03.433 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:03.424+0000 7f91dcb28640 1 --2- 172.21.15.123:0/4134489404 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f91c80064e0 0x7f91d814ef90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:47:03.433 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:03.424+0000 7f91dd329640 1 --2- 172.21.15.123:0/4134489404 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f91d81e8130 0x7f91d81e85b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:47:03.433 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:03.424+0000 7f91d7fff640 1 -- 172.21.15.123:0/4134489404 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f91c80064e0 msgr2=0x7f91d814ef90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:47:03.433 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:03.424+0000 7f91d7fff640 1 --2- 172.21.15.123:0/4134489404 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f91c80064e0 0x7f91d814ef90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:03.433 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:03.424+0000 7f91d7fff640 1 -- 172.21.15.123:0/4134489404 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f91d81e8130 msgr2=0x7f91d81e85b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:47:03.433 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:03.424+0000 7f91d7fff640 1 --2- 172.21.15.123:0/4134489404 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f91d81e8130 0x7f91d81e85b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:03.433 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:03.424+0000 7f91d7fff640 1 -- 172.21.15.123:0/4134489404 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f91d8159b20 con 0x7f91d80739c0 2024-09-16T15:47:03.433 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:03.424+0000 7f91dd329640 1 --2- 172.21.15.123:0/4134489404 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f91d81e8130 0x7f91d81e85b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T15:47:03.434 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:03.428+0000 7f91d7fff640 1 --2- 172.21.15.123:0/4134489404 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f91d80739c0 0x7f91d8073e40 secure :-1 s=READY pgs=786 cs=0 l=1 rev1=1 crypto rx=0x7f91c4004d20 tx=0x7f91c4001640 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:47:03.434 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:03.428+0000 7f91d57fa640 1 -- 172.21.15.123:0/4134489404 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91c40107d0 con 0x7f91d80739c0 2024-09-16T15:47:03.434 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:03.428+0000 7f91d57fa640 1 -- 172.21.15.123:0/4134489404 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f91c4016070 con 0x7f91d80739c0 2024-09-16T15:47:03.434 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:03.428+0000 7f91ded8c640 1 -- 172.21.15.123:0/4134489404 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f91d815ab80 con 0x7f91d80739c0 2024-09-16T15:47:03.434 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:03.428+0000 7f91d57fa640 1 -- 172.21.15.123:0/4134489404 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91c4010d20 con 0x7f91d80739c0 2024-09-16T15:47:03.434 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:03.428+0000 7f91ded8c640 1 -- 172.21.15.123:0/4134489404 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f91d815a0a0 con 0x7f91d80739c0 2024-09-16T15:47:03.434 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:03.428+0000 7f91ded8c640 1 -- 172.21.15.123:0/4134489404 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f91d81e8af0 con 0x7f91d80739c0 2024-09-16T15:47:03.435 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:03.428+0000 7f91d57fa640 1 -- 172.21.15.123:0/4134489404 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 11) ==== 97386+0+0 (secure 0 0 0) 0x7f91c401b460 con 0x7f91d80739c0 2024-09-16T15:47:03.435 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:03.428+0000 7f91d57fa640 1 --2- 172.21.15.123:0/4134489404 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f91bc073250 0x7f91bc075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:47:03.435 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:03.428+0000 7f91d57fa640 1 -- 172.21.15.123:0/4134489404 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(70..70 src has 1..70) ==== 5709+0+0 (secure 0 0 0) 0x7f91c4094bc0 con 0x7f91d80739c0 2024-09-16T15:47:03.435 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:03.428+0000 7f91dcb28640 1 --2- 172.21.15.123:0/4134489404 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f91bc073250 0x7f91bc075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:47:03.435 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:03.428+0000 7f91dcb28640 1 --2- 172.21.15.123:0/4134489404 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f91bc073250 0x7f91bc075710 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f91cc001ca0 tx=0x7f91cc000fc0 comp rx=0 tx=0).ready entity=mgr.4135 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:47:03.437 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:03.432+0000 7f91d57fa640 1 -- 172.21.15.123:0/4134489404 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f91c405f2a0 con 0x7f91d80739c0 2024-09-16T15:47:03.595 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:03.592+0000 7f91ded8c640 1 -- 172.21.15.123:0/4134489404 --> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f91d80d06c0 con 0x7f91bc073250 2024-09-16T15:47:03.606 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:03.604+0000 7f91d57fa640 1 -- 172.21.15.123:0/4134489404 <== mgr.4135 v2:172.21.15.184:6832/3114346423 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+246774 (secure 0 0 0) 0x7f91d80d06c0 con 0x7f91bc073250 2024-09-16T15:47:03.609 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-09-16T15:47:03.611 INFO:teuthology.orchestra.run.smithi123.stderr:dumped all 2024-09-16T15:47:03.612 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:03.604+0000 7f91ded8c640 1 -- 172.21.15.123:0/4134489404 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f91bc073250 msgr2=0x7f91bc075710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:47:03.612 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:03.604+0000 7f91ded8c640 1 --2- 172.21.15.123:0/4134489404 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f91bc073250 0x7f91bc075710 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f91cc001ca0 tx=0x7f91cc000fc0 comp rx=0 tx=0).stop 2024-09-16T15:47:03.612 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:03.604+0000 7f91ded8c640 1 -- 172.21.15.123:0/4134489404 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f91d80739c0 msgr2=0x7f91d8073e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:47:03.613 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:03.604+0000 7f91ded8c640 1 --2- 172.21.15.123:0/4134489404 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f91d80739c0 0x7f91d8073e40 secure :-1 s=READY pgs=786 cs=0 l=1 rev1=1 crypto rx=0x7f91c4004d20 tx=0x7f91c4001640 comp rx=0 tx=0).stop 2024-09-16T15:47:03.613 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:03.604+0000 7f91ded8c640 1 -- 172.21.15.123:0/4134489404 shutdown_connections 2024-09-16T15:47:03.613 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:03.604+0000 7f91ded8c640 1 --2- 172.21.15.123:0/4134489404 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f91bc073250 0x7f91bc075710 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:03.613 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:03.604+0000 7f91ded8c640 1 --2- 172.21.15.123:0/4134489404 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f91d81e8130 0x7f91d81e85b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:03.613 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:03.604+0000 7f91ded8c640 1 --2- 172.21.15.123:0/4134489404 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f91d80739c0 0x7f91d8073e40 unknown :-1 s=CLOSED pgs=786 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:03.613 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:03.604+0000 7f91ded8c640 1 --2- 172.21.15.123:0/4134489404 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f91c80064e0 0x7f91d814ef90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:03.613 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:03.604+0000 7f91ded8c640 1 -- 172.21.15.123:0/4134489404 >> 172.21.15.123:0/4134489404 conn(0x7f91d80cfb10 msgr2=0x7f91d8152110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:47:03.613 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:03.604+0000 7f91ded8c640 1 -- 172.21.15.123:0/4134489404 shutdown_connections 2024-09-16T15:47:03.613 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:03.604+0000 7f91ded8c640 1 -- 172.21.15.123:0/4134489404 wait complete. 2024-09-16T15:47:03.626 INFO:teuthology.orchestra.run.smithi123.stdout:{"pg_ready":true,"pg_map":{"version":195,"stamp":"2024-09-16T15:47:03.331208+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":76,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":240444,"kb_used_data":7556,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":754734276,"statfs":{"total":773094113280,"available":772847898624,"internally_reserved":0,"allocated":7737344,"data_stored":3897055,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":238276145},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":41,"apply_latency_ms":41,"commit_latency_ns":41000000,"apply_latency_ns":41000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.788118"},"pg_stats":[{"pgid":"11.3f","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-09-16T15:46:58.580896+0000","last_change":"2024-09-16T15:46:58.580896+0000","last_active":"2024-09-16T15:46:58.541934+0000","last_peered":"2024-09-16T15:46:58.541934+0000","last_clean":"2024-09-16T15:46:58.541934+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:46:58.580896+0000","last_undegraded":"2024-09-16T15:46:58.580896+0000","last_fullsized":"2024-09-16T15:46:58.580896+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_clean_scrub_stamp":"2024-09-16T15:46:58.541934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.3e","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-09-16T15:46:58.580879+0000","last_change":"2024-09-16T15:46:58.580879+0000","last_active":"2024-09-16T15:46:58.541934+0000","last_peered":"2024-09-16T15:46:58.541934+0000","last_clean":"2024-09-16T15:46:58.541934+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:46:58.580879+0000","last_undegraded":"2024-09-16T15:46:58.580879+0000","last_fullsized":"2024-09-16T15:46:58.580879+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_clean_scrub_stamp":"2024-09-16T15:46:58.541934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.3d","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-09-16T15:46:58.565500+0000","last_change":"2024-09-16T15:46:58.565500+0000","last_active":"2024-09-16T15:46:58.541934+0000","last_peered":"2024-09-16T15:46:58.541934+0000","last_clean":"2024-09-16T15:46:58.541934+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:46:58.565500+0000","last_undegraded":"2024-09-16T15:46:58.565500+0000","last_fullsized":"2024-09-16T15:46:58.565500+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_clean_scrub_stamp":"2024-09-16T15:46:58.541934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.3c","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-09-16T15:46:58.560961+0000","last_change":"2024-09-16T15:46:58.560961+0000","last_active":"2024-09-16T15:46:58.541934+0000","last_peered":"2024-09-16T15:46:58.541934+0000","last_clean":"2024-09-16T15:46:58.541934+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:46:58.560961+0000","last_undegraded":"2024-09-16T15:46:58.560961+0000","last_fullsized":"2024-09-16T15:46:58.560961+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_clean_scrub_stamp":"2024-09-16T15:46:58.541934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.3b","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-09-16T15:46:58.561715+0000","last_change":"2024-09-16T15:46:58.561715+0000","last_active":"2024-09-16T15:46:58.541934+0000","last_peered":"2024-09-16T15:46:58.541934+0000","last_clean":"2024-09-16T15:46:58.541934+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:46:58.561715+0000","last_undegraded":"2024-09-16T15:46:58.561715+0000","last_fullsized":"2024-09-16T15:46:58.561715+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_clean_scrub_stamp":"2024-09-16T15:46:58.541934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.3a","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-09-16T15:46:58.578627+0000","last_change":"2024-09-16T15:46:58.578627+0000","last_active":"2024-09-16T15:46:58.541934+0000","last_peered":"2024-09-16T15:46:58.541934+0000","last_clean":"2024-09-16T15:46:58.541934+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:46:58.578627+0000","last_undegraded":"2024-09-16T15:46:58.578627+0000","last_fullsized":"2024-09-16T15:46:58.578627+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_clean_scrub_stamp":"2024-09-16T15:46:58.541934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.39","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-09-16T15:46:58.570892+0000","last_change":"2024-09-16T15:46:58.570892+0000","last_active":"2024-09-16T15:46:58.541934+0000","last_peered":"2024-09-16T15:46:58.541934+0000","last_clean":"2024-09-16T15:46:58.541934+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:46:58.570892+0000","last_undegraded":"2024-09-16T15:46:58.570892+0000","last_fullsized":"2024-09-16T15:46:58.570892+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_clean_scrub_stamp":"2024-09-16T15:46:58.541934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.38","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-09-16T15:46:58.584780+0000","last_change":"2024-09-16T15:46:58.584780+0000","last_active":"2024-09-16T15:46:58.541934+0000","last_peered":"2024-09-16T15:46:58.541934+0000","last_clean":"2024-09-16T15:46:58.541934+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:46:58.584780+0000","last_undegraded":"2024-09-16T15:46:58.584780+0000","last_fullsized":"2024-09-16T15:46:58.584780+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_clean_scrub_stamp":"2024-09-16T15:46:58.541934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.37","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-09-16T15:46:58.561912+0000","last_change":"2024-09-16T15:46:58.561912+0000","last_active":"2024-09-16T15:46:58.541934+0000","last_peered":"2024-09-16T15:46:58.541934+0000","last_clean":"2024-09-16T15:46:58.541934+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:46:58.561912+0000","last_undegraded":"2024-09-16T15:46:58.561912+0000","last_fullsized":"2024-09-16T15:46:58.561912+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_clean_scrub_stamp":"2024-09-16T15:46:58.541934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.36","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-09-16T15:46:58.562601+0000","last_change":"2024-09-16T15:46:58.562601+0000","last_active":"2024-09-16T15:46:58.541934+0000","last_peered":"2024-09-16T15:46:58.541934+0000","last_clean":"2024-09-16T15:46:58.541934+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:46:58.562601+0000","last_undegraded":"2024-09-16T15:46:58.562601+0000","last_fullsized":"2024-09-16T15:46:58.562601+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_clean_scrub_stamp":"2024-09-16T15:46:58.541934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.35","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-09-16T15:46:58.580321+0000","last_change":"2024-09-16T15:46:58.580321+0000","last_active":"2024-09-16T15:46:58.541934+0000","last_peered":"2024-09-16T15:46:58.541934+0000","last_clean":"2024-09-16T15:46:58.541934+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:46:58.580321+0000","last_undegraded":"2024-09-16T15:46:58.580321+0000","last_fullsized":"2024-09-16T15:46:58.580321+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_clean_scrub_stamp":"2024-09-16T15:46:58.541934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.34","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-09-16T15:46:58.560127+0000","last_change":"2024-09-16T15:46:58.560127+0000","last_active":"2024-09-16T15:46:58.541934+0000","last_peered":"2024-09-16T15:46:58.541934+0000","last_clean":"2024-09-16T15:46:58.541934+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:46:58.560127+0000","last_undegraded":"2024-09-16T15:46:58.560127+0000","last_fullsized":"2024-09-16T15:46:58.560127+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_clean_scrub_stamp":"2024-09-16T15:46:58.541934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.33","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-09-16T15:46:58.562521+0000","last_change":"2024-09-16T15:46:58.562521+0000","last_active":"2024-09-16T15:46:58.541934+0000","last_peered":"2024-09-16T15:46:58.541934+0000","last_clean":"2024-09-16T15:46:58.541934+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:46:58.562521+0000","last_undegraded":"2024-09-16T15:46:58.562521+0000","last_fullsized":"2024-09-16T15:46:58.562521+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_clean_scrub_stamp":"2024-09-16T15:46:58.541934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.32","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-09-16T15:46:58.561306+0000","last_change":"2024-09-16T15:46:58.561306+0000","last_active":"2024-09-16T15:46:58.541934+0000","last_peered":"2024-09-16T15:46:58.541934+0000","last_clean":"2024-09-16T15:46:58.541934+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:46:58.561306+0000","last_undegraded":"2024-09-16T15:46:58.561306+0000","last_fullsized":"2024-09-16T15:46:58.561306+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_clean_scrub_stamp":"2024-09-16T15:46:58.541934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.31","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-09-16T15:46:58.566448+0000","last_change":"2024-09-16T15:46:58.566448+0000","last_active":"2024-09-16T15:46:58.541934+0000","last_peered":"2024-09-16T15:46:58.541934+0000","last_clean":"2024-09-16T15:46:58.541934+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:46:58.566448+0000","last_undegraded":"2024-09-16T15:46:58.566448+0000","last_fullsized":"2024-09-16T15:46:58.566448+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_clean_scrub_stamp":"2024-09-16T15:46:58.541934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.30","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-09-16T15:46:58.560118+0000","last_change":"2024-09-16T15:46:58.560118+0000","last_active":"2024-09-16T15:46:58.541934+0000","last_peered":"2024-09-16T15:46:58.541934+0000","last_clean":"2024-09-16T15:46:58.541934+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:46:58.560118+0000","last_undegraded":"2024-09-16T15:46:58.560118+0000","last_fullsized":"2024-09-16T15:46:58.560118+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_clean_scrub_stamp":"2024-09-16T15:46:58.541934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.2f","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-09-16T15:46:58.561316+0000","last_change":"2024-09-16T15:46:58.561316+0000","last_active":"2024-09-16T15:46:58.541934+0000","last_peered":"2024-09-16T15:46:58.541934+0000","last_clean":"2024-09-16T15:46:58.541934+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:46:58.561316+0000","last_undegraded":"2024-09-16T15:46:58.561316+0000","last_fullsized":"2024-09-16T15:46:58.561316+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_clean_scrub_stamp":"2024-09-16T15:46:58.541934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.2e","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-09-16T15:46:58.559376+0000","last_change":"2024-09-16T15:46:58.559376+0000","last_active":"2024-09-16T15:46:58.541934+0000","last_peered":"2024-09-16T15:46:58.541934+0000","last_clean":"2024-09-16T15:46:58.541934+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:46:58.559376+0000","last_undegraded":"2024-09-16T15:46:58.559376+0000","last_fullsized":"2024-09-16T15:46:58.559376+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_clean_scrub_stamp":"2024-09-16T15:46:58.541934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.2d","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-09-16T15:46:58.580101+0000","last_change":"2024-09-16T15:46:58.580101+0000","last_active":"2024-09-16T15:46:58.541934+0000","last_peered":"2024-09-16T15:46:58.541934+0000","last_clean":"2024-09-16T15:46:58.541934+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:46:58.580101+0000","last_undegraded":"2024-09-16T15:46:58.580101+0000","last_fullsized":"2024-09-16T15:46:58.580101+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_clean_scrub_stamp":"2024-09-16T15:46:58.541934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.2c","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-09-16T15:46:58.562539+0000","last_change":"2024-09-16T15:46:58.562539+0000","last_active":"2024-09-16T15:46:58.541934+0000","last_peered":"2024-09-16T15:46:58.541934+0000","last_clean":"2024-09-16T15:46:58.541934+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:46:58.562539+0000","last_undegraded":"2024-09-16T15:46:58.562539+0000","last_fullsized":"2024-09-16T15:46:58.562539+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_clean_scrub_stamp":"2024-09-16T15:46:58.541934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.2b","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-09-16T15:46:58.564602+0000","last_change":"2024-09-16T15:46:58.564602+0000","last_active":"2024-09-16T15:46:58.541934+0000","last_peered":"2024-09-16T15:46:58.541934+0000","last_clean":"2024-09-16T15:46:58.541934+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:46:58.564602+0000","last_undegraded":"2024-09-16T15:46:58.564602+0000","last_fullsized":"2024-09-16T15:46:58.564602+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_clean_scrub_stamp":"2024-09-16T15:46:58.541934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.2a","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-09-16T15:46:58.562954+0000","last_change":"2024-09-16T15:46:58.562954+0000","last_active":"2024-09-16T15:46:58.541934+0000","last_peered":"2024-09-16T15:46:58.541934+0000","last_clean":"2024-09-16T15:46:58.541934+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:46:58.562954+0000","last_undegraded":"2024-09-16T15:46:58.562954+0000","last_fullsized":"2024-09-16T15:46:58.562954+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_clean_scrub_stamp":"2024-09-16T15:46:58.541934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.29","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-09-16T15:46:58.578047+0000","last_change":"2024-09-16T15:46:58.578047+0000","last_active":"2024-09-16T15:46:58.541934+0000","last_peered":"2024-09-16T15:46:58.541934+0000","last_clean":"2024-09-16T15:46:58.541934+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:46:58.578047+0000","last_undegraded":"2024-09-16T15:46:58.578047+0000","last_fullsized":"2024-09-16T15:46:58.578047+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_clean_scrub_stamp":"2024-09-16T15:46:58.541934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.28","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-09-16T15:46:58.575628+0000","last_change":"2024-09-16T15:46:58.575628+0000","last_active":"2024-09-16T15:46:58.541934+0000","last_peered":"2024-09-16T15:46:58.541934+0000","last_clean":"2024-09-16T15:46:58.541934+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:46:58.575628+0000","last_undegraded":"2024-09-16T15:46:58.575628+0000","last_fullsized":"2024-09-16T15:46:58.575628+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_clean_scrub_stamp":"2024-09-16T15:46:58.541934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.27","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-09-16T15:46:58.562122+0000","last_change":"2024-09-16T15:46:58.562122+0000","last_active":"2024-09-16T15:46:58.541934+0000","last_peered":"2024-09-16T15:46:58.541934+0000","last_clean":"2024-09-16T15:46:58.541934+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:46:58.562122+0000","last_undegraded":"2024-09-16T15:46:58.562122+0000","last_fullsized":"2024-09-16T15:46:58.562122+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_clean_scrub_stamp":"2024-09-16T15:46:58.541934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.26","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-09-16T15:46:58.571812+0000","last_change":"2024-09-16T15:46:58.571812+0000","last_active":"2024-09-16T15:46:58.541934+0000","last_peered":"2024-09-16T15:46:58.541934+0000","last_clean":"2024-09-16T15:46:58.541934+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:46:58.571812+0000","last_undegraded":"2024-09-16T15:46:58.571812+0000","last_fullsized":"2024-09-16T15:46:58.571812+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_clean_scrub_stamp":"2024-09-16T15:46:58.541934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.25","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-09-16T15:46:58.564846+0000","last_change":"2024-09-16T15:46:58.564846+0000","last_active":"2024-09-16T15:46:58.541934+0000","last_peered":"2024-09-16T15:46:58.541934+0000","last_clean":"2024-09-16T15:46:58.541934+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:46:58.564846+0000","last_undegraded":"2024-09-16T15:46:58.564846+0000","last_fullsized":"2024-09-16T15:46:58.564846+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_clean_scrub_stamp":"2024-09-16T15:46:58.541934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.24","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-09-16T15:46:58.560367+0000","last_change":"2024-09-16T15:46:58.560367+0000","last_active":"2024-09-16T15:46:58.541934+0000","last_peered":"2024-09-16T15:46:58.541934+0000","last_clean":"2024-09-16T15:46:58.541934+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:46:58.560367+0000","last_undegraded":"2024-09-16T15:46:58.560367+0000","last_fullsized":"2024-09-16T15:46:58.560367+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_clean_scrub_stamp":"2024-09-16T15:46:58.541934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.23","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-09-16T15:46:58.568304+0000","last_change":"2024-09-16T15:46:58.568304+0000","last_active":"2024-09-16T15:46:58.541934+0000","last_peered":"2024-09-16T15:46:58.541934+0000","last_clean":"2024-09-16T15:46:58.541934+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:46:58.568304+0000","last_undegraded":"2024-09-16T15:46:58.568304+0000","last_fullsized":"2024-09-16T15:46:58.568304+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_clean_scrub_stamp":"2024-09-16T15:46:58.541934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.22","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-09-16T15:46:58.569453+0000","last_change":"2024-09-16T15:46:58.569453+0000","last_active":"2024-09-16T15:46:58.541934+0000","last_peered":"2024-09-16T15:46:58.541934+0000","last_clean":"2024-09-16T15:46:58.541934+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:46:58.569453+0000","last_undegraded":"2024-09-16T15:46:58.569453+0000","last_fullsized":"2024-09-16T15:46:58.569453+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_clean_scrub_stamp":"2024-09-16T15:46:58.541934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.21","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-09-16T15:46:58.569512+0000","last_change":"2024-09-16T15:46:58.569512+0000","last_active":"2024-09-16T15:46:58.541934+0000","last_peered":"2024-09-16T15:46:58.541934+0000","last_clean":"2024-09-16T15:46:58.541934+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:46:58.569512+0000","last_undegraded":"2024-09-16T15:46:58.569512+0000","last_fullsized":"2024-09-16T15:46:58.569512+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_clean_scrub_stamp":"2024-09-16T15:46:58.541934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.20","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-09-16T15:46:58.582821+0000","last_change":"2024-09-16T15:46:58.582821+0000","last_active":"2024-09-16T15:46:58.541934+0000","last_peered":"2024-09-16T15:46:58.541934+0000","last_clean":"2024-09-16T15:46:58.541934+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:46:58.582821+0000","last_undegraded":"2024-09-16T15:46:58.582821+0000","last_fullsized":"2024-09-16T15:46:58.582821+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_clean_scrub_stamp":"2024-09-16T15:46:58.541934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.1e","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-09-16T15:46:58.575147+0000","last_change":"2024-09-16T15:46:58.575147+0000","last_active":"2024-09-16T15:46:58.541934+0000","last_peered":"2024-09-16T15:46:58.541934+0000","last_clean":"2024-09-16T15:46:58.541934+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:46:58.575147+0000","last_undegraded":"2024-09-16T15:46:58.575147+0000","last_fullsized":"2024-09-16T15:46:58.575147+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_clean_scrub_stamp":"2024-09-16T15:46:58.541934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.1f","version":"0'0","reported_seq":10,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-16T15:46:58.599998+0000","last_change":"2024-09-16T15:46:58.596071+0000","last_active":"2024-09-16T15:46:58.599998+0000","last_peered":"2024-09-16T15:46:58.599998+0000","last_clean":"2024-09-16T15:46:58.599998+0000","last_became_active":"2024-09-16T15:46:58.595309+0000","last_became_peered":"2024-09-16T15:46:58.595309+0000","last_unstale":"2024-09-16T15:46:58.599998+0000","last_undegraded":"2024-09-16T15:46:58.599998+0000","last_fullsized":"2024-09-16T15:46:58.599998+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:57.531046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:57.531046+0000","last_clean_scrub_stamp":"2024-09-16T15:46:57.531046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:37:47.205740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.1f","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-09-16T15:46:58.560943+0000","last_change":"2024-09-16T15:46:58.560943+0000","last_active":"2024-09-16T15:46:58.541934+0000","last_peered":"2024-09-16T15:46:58.541934+0000","last_clean":"2024-09-16T15:46:58.541934+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:46:58.560943+0000","last_undegraded":"2024-09-16T15:46:58.560943+0000","last_fullsized":"2024-09-16T15:46:58.560943+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_clean_scrub_stamp":"2024-09-16T15:46:58.541934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.1e","version":"0'0","reported_seq":10,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-16T15:46:58.577768+0000","last_change":"2024-09-16T15:46:58.572883+0000","last_active":"2024-09-16T15:46:58.577768+0000","last_peered":"2024-09-16T15:46:58.577768+0000","last_clean":"2024-09-16T15:46:58.577768+0000","last_became_active":"2024-09-16T15:46:58.572508+0000","last_became_peered":"2024-09-16T15:46:58.572508+0000","last_unstale":"2024-09-16T15:46:58.577768+0000","last_undegraded":"2024-09-16T15:46:58.577768+0000","last_fullsized":"2024-09-16T15:46:58.577768+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:57.531046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:57.531046+0000","last_clean_scrub_stamp":"2024-09-16T15:46:57.531046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:16:24.337739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1c","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-09-16T15:46:58.583638+0000","last_change":"2024-09-16T15:46:58.583638+0000","last_active":"2024-09-16T15:46:58.541934+0000","last_peered":"2024-09-16T15:46:58.541934+0000","last_clean":"2024-09-16T15:46:58.541934+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:46:58.583638+0000","last_undegraded":"2024-09-16T15:46:58.583638+0000","last_fullsized":"2024-09-16T15:46:58.583638+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_clean_scrub_stamp":"2024-09-16T15:46:58.541934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.1d","version":"0'0","reported_seq":10,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-16T15:46:58.604745+0000","last_change":"2024-09-16T15:46:58.598772+0000","last_active":"2024-09-16T15:46:58.604745+0000","last_peered":"2024-09-16T15:46:58.604745+0000","last_clean":"2024-09-16T15:46:58.604745+0000","last_became_active":"2024-09-16T15:46:58.597889+0000","last_became_peered":"2024-09-16T15:46:58.597889+0000","last_unstale":"2024-09-16T15:46:58.604745+0000","last_undegraded":"2024-09-16T15:46:58.604745+0000","last_fullsized":"2024-09-16T15:46:58.604745+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:57.531046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:57.531046+0000","last_clean_scrub_stamp":"2024-09-16T15:46:57.531046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:20:50.612777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1d","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-09-16T15:46:58.579199+0000","last_change":"2024-09-16T15:46:58.579199+0000","last_active":"2024-09-16T15:46:58.541934+0000","last_peered":"2024-09-16T15:46:58.541934+0000","last_clean":"2024-09-16T15:46:58.541934+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:46:58.579199+0000","last_undegraded":"2024-09-16T15:46:58.579199+0000","last_fullsized":"2024-09-16T15:46:58.579199+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_clean_scrub_stamp":"2024-09-16T15:46:58.541934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.1c","version":"0'0","reported_seq":10,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-16T15:46:58.595606+0000","last_change":"2024-09-16T15:46:58.588644+0000","last_active":"2024-09-16T15:46:58.595606+0000","last_peered":"2024-09-16T15:46:58.595606+0000","last_clean":"2024-09-16T15:46:58.595606+0000","last_became_active":"2024-09-16T15:46:58.587899+0000","last_became_peered":"2024-09-16T15:46:58.587899+0000","last_unstale":"2024-09-16T15:46:58.595606+0000","last_undegraded":"2024-09-16T15:46:58.595606+0000","last_fullsized":"2024-09-16T15:46:58.595606+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:57.531046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:57.531046+0000","last_clean_scrub_stamp":"2024-09-16T15:46:57.531046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T00:36:13.054311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.1a","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-09-16T15:46:58.568314+0000","last_change":"2024-09-16T15:46:58.568314+0000","last_active":"2024-09-16T15:46:58.541934+0000","last_peered":"2024-09-16T15:46:58.541934+0000","last_clean":"2024-09-16T15:46:58.541934+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:46:58.568314+0000","last_undegraded":"2024-09-16T15:46:58.568314+0000","last_fullsized":"2024-09-16T15:46:58.568314+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_clean_scrub_stamp":"2024-09-16T15:46:58.541934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.1b","version":"0'0","reported_seq":10,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-16T15:46:58.594471+0000","last_change":"2024-09-16T15:46:58.591621+0000","last_active":"2024-09-16T15:46:58.594471+0000","last_peered":"2024-09-16T15:46:58.594471+0000","last_clean":"2024-09-16T15:46:58.594471+0000","last_became_active":"2024-09-16T15:46:58.591311+0000","last_became_peered":"2024-09-16T15:46:58.591311+0000","last_unstale":"2024-09-16T15:46:58.594471+0000","last_undegraded":"2024-09-16T15:46:58.594471+0000","last_fullsized":"2024-09-16T15:46:58.594471+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:57.531046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:57.531046+0000","last_clean_scrub_stamp":"2024-09-16T15:46:57.531046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:55:01.184561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.1b","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-09-16T15:46:58.563604+0000","last_change":"2024-09-16T15:46:58.563604+0000","last_active":"2024-09-16T15:46:58.541934+0000","last_peered":"2024-09-16T15:46:58.541934+0000","last_clean":"2024-09-16T15:46:58.541934+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:46:58.563604+0000","last_undegraded":"2024-09-16T15:46:58.563604+0000","last_fullsized":"2024-09-16T15:46:58.563604+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_clean_scrub_stamp":"2024-09-16T15:46:58.541934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.1a","version":"0'0","reported_seq":10,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-16T15:46:58.604119+0000","last_change":"2024-09-16T15:46:58.600655+0000","last_active":"2024-09-16T15:46:58.604119+0000","last_peered":"2024-09-16T15:46:58.604119+0000","last_clean":"2024-09-16T15:46:58.604119+0000","last_became_active":"2024-09-16T15:46:58.599216+0000","last_became_peered":"2024-09-16T15:46:58.599216+0000","last_unstale":"2024-09-16T15:46:58.604119+0000","last_undegraded":"2024-09-16T15:46:58.604119+0000","last_fullsized":"2024-09-16T15:46:58.604119+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:57.531046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:57.531046+0000","last_clean_scrub_stamp":"2024-09-16T15:46:57.531046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:13:18.615772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.18","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-09-16T15:46:58.562911+0000","last_change":"2024-09-16T15:46:58.562911+0000","last_active":"2024-09-16T15:46:58.541934+0000","last_peered":"2024-09-16T15:46:58.541934+0000","last_clean":"2024-09-16T15:46:58.541934+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:46:58.562911+0000","last_undegraded":"2024-09-16T15:46:58.562911+0000","last_fullsized":"2024-09-16T15:46:58.562911+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_clean_scrub_stamp":"2024-09-16T15:46:58.541934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.19","version":"0'0","reported_seq":10,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-16T15:46:58.589251+0000","last_change":"2024-09-16T15:46:58.584364+0000","last_active":"2024-09-16T15:46:58.589251+0000","last_peered":"2024-09-16T15:46:58.589251+0000","last_clean":"2024-09-16T15:46:58.589251+0000","last_became_active":"2024-09-16T15:46:58.583975+0000","last_became_peered":"2024-09-16T15:46:58.583975+0000","last_unstale":"2024-09-16T15:46:58.589251+0000","last_undegraded":"2024-09-16T15:46:58.589251+0000","last_fullsized":"2024-09-16T15:46:58.589251+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:57.531046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:57.531046+0000","last_clean_scrub_stamp":"2024-09-16T15:46:57.531046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T00:25:29.249313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.19","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-09-16T15:46:58.566064+0000","last_change":"2024-09-16T15:46:58.566064+0000","last_active":"2024-09-16T15:46:58.541934+0000","last_peered":"2024-09-16T15:46:58.541934+0000","last_clean":"2024-09-16T15:46:58.541934+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:46:58.566064+0000","last_undegraded":"2024-09-16T15:46:58.566064+0000","last_fullsized":"2024-09-16T15:46:58.566064+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_clean_scrub_stamp":"2024-09-16T15:46:58.541934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.18","version":"0'0","reported_seq":10,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-16T15:46:58.574992+0000","last_change":"2024-09-16T15:46:58.570577+0000","last_active":"2024-09-16T15:46:58.574992+0000","last_peered":"2024-09-16T15:46:58.574992+0000","last_clean":"2024-09-16T15:46:58.574992+0000","last_became_active":"2024-09-16T15:46:58.570223+0000","last_became_peered":"2024-09-16T15:46:58.570223+0000","last_unstale":"2024-09-16T15:46:58.574992+0000","last_undegraded":"2024-09-16T15:46:58.574992+0000","last_fullsized":"2024-09-16T15:46:58.574992+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:57.531046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:57.531046+0000","last_clean_scrub_stamp":"2024-09-16T15:46:57.531046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:11:59.055640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.16","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-09-16T15:46:58.570594+0000","last_change":"2024-09-16T15:46:58.570594+0000","last_active":"2024-09-16T15:46:58.541934+0000","last_peered":"2024-09-16T15:46:58.541934+0000","last_clean":"2024-09-16T15:46:58.541934+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:46:58.570594+0000","last_undegraded":"2024-09-16T15:46:58.570594+0000","last_fullsized":"2024-09-16T15:46:58.570594+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_clean_scrub_stamp":"2024-09-16T15:46:58.541934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.17","version":"0'0","reported_seq":10,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-16T15:46:58.590618+0000","last_change":"2024-09-16T15:46:58.576463+0000","last_active":"2024-09-16T15:46:58.590618+0000","last_peered":"2024-09-16T15:46:58.590618+0000","last_clean":"2024-09-16T15:46:58.590618+0000","last_became_active":"2024-09-16T15:46:58.573574+0000","last_became_peered":"2024-09-16T15:46:58.573574+0000","last_unstale":"2024-09-16T15:46:58.590618+0000","last_undegraded":"2024-09-16T15:46:58.590618+0000","last_fullsized":"2024-09-16T15:46:58.590618+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:57.531046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:57.531046+0000","last_clean_scrub_stamp":"2024-09-16T15:46:57.531046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T01:24:15.136386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.17","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-09-16T15:46:58.579574+0000","last_change":"2024-09-16T15:46:58.579574+0000","last_active":"2024-09-16T15:46:58.541934+0000","last_peered":"2024-09-16T15:46:58.541934+0000","last_clean":"2024-09-16T15:46:58.541934+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:46:58.579574+0000","last_undegraded":"2024-09-16T15:46:58.579574+0000","last_fullsized":"2024-09-16T15:46:58.579574+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_clean_scrub_stamp":"2024-09-16T15:46:58.541934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.16","version":"0'0","reported_seq":10,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-16T15:46:58.602617+0000","last_change":"2024-09-16T15:46:58.597162+0000","last_active":"2024-09-16T15:46:58.602617+0000","last_peered":"2024-09-16T15:46:58.602617+0000","last_clean":"2024-09-16T15:46:58.602617+0000","last_became_active":"2024-09-16T15:46:58.596469+0000","last_became_peered":"2024-09-16T15:46:58.596469+0000","last_unstale":"2024-09-16T15:46:58.602617+0000","last_undegraded":"2024-09-16T15:46:58.602617+0000","last_fullsized":"2024-09-16T15:46:58.602617+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:57.531046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:57.531046+0000","last_clean_scrub_stamp":"2024-09-16T15:46:57.531046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:22:59.268994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.14","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-09-16T15:46:58.582525+0000","last_change":"2024-09-16T15:46:58.582525+0000","last_active":"2024-09-16T15:46:58.541934+0000","last_peered":"2024-09-16T15:46:58.541934+0000","last_clean":"2024-09-16T15:46:58.541934+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:46:58.582525+0000","last_undegraded":"2024-09-16T15:46:58.582525+0000","last_fullsized":"2024-09-16T15:46:58.582525+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_clean_scrub_stamp":"2024-09-16T15:46:58.541934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.15","version":"0'0","reported_seq":10,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-16T15:46:58.588947+0000","last_change":"2024-09-16T15:46:58.583907+0000","last_active":"2024-09-16T15:46:58.588947+0000","last_peered":"2024-09-16T15:46:58.588947+0000","last_clean":"2024-09-16T15:46:58.588947+0000","last_became_active":"2024-09-16T15:46:58.583530+0000","last_became_peered":"2024-09-16T15:46:58.583530+0000","last_unstale":"2024-09-16T15:46:58.588947+0000","last_undegraded":"2024-09-16T15:46:58.588947+0000","last_fullsized":"2024-09-16T15:46:58.588947+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:57.531046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:57.531046+0000","last_clean_scrub_stamp":"2024-09-16T15:46:57.531046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T01:06:36.276569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.15","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-09-16T15:46:58.563631+0000","last_change":"2024-09-16T15:46:58.563631+0000","last_active":"2024-09-16T15:46:58.541934+0000","last_peered":"2024-09-16T15:46:58.541934+0000","last_clean":"2024-09-16T15:46:58.541934+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:46:58.563631+0000","last_undegraded":"2024-09-16T15:46:58.563631+0000","last_fullsized":"2024-09-16T15:46:58.563631+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_clean_scrub_stamp":"2024-09-16T15:46:58.541934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.14","version":"0'0","reported_seq":10,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-16T15:46:58.575157+0000","last_change":"2024-09-16T15:46:58.570712+0000","last_active":"2024-09-16T15:46:58.575157+0000","last_peered":"2024-09-16T15:46:58.575157+0000","last_clean":"2024-09-16T15:46:58.575157+0000","last_became_active":"2024-09-16T15:46:58.570432+0000","last_became_peered":"2024-09-16T15:46:58.570432+0000","last_unstale":"2024-09-16T15:46:58.575157+0000","last_undegraded":"2024-09-16T15:46:58.575157+0000","last_fullsized":"2024-09-16T15:46:58.575157+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:57.531046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:57.531046+0000","last_clean_scrub_stamp":"2024-09-16T15:46:57.531046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:58:27.343595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.12","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-09-16T15:46:58.574297+0000","last_change":"2024-09-16T15:46:58.574297+0000","last_active":"2024-09-16T15:46:58.541934+0000","last_peered":"2024-09-16T15:46:58.541934+0000","last_clean":"2024-09-16T15:46:58.541934+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:46:58.574297+0000","last_undegraded":"2024-09-16T15:46:58.574297+0000","last_fullsized":"2024-09-16T15:46:58.574297+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_clean_scrub_stamp":"2024-09-16T15:46:58.541934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.13","version":"0'0","reported_seq":10,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-16T15:46:58.603021+0000","last_change":"2024-09-16T15:46:58.596492+0000","last_active":"2024-09-16T15:46:58.603021+0000","last_peered":"2024-09-16T15:46:58.603021+0000","last_clean":"2024-09-16T15:46:58.603021+0000","last_became_active":"2024-09-16T15:46:58.596172+0000","last_became_peered":"2024-09-16T15:46:58.596172+0000","last_unstale":"2024-09-16T15:46:58.603021+0000","last_undegraded":"2024-09-16T15:46:58.603021+0000","last_fullsized":"2024-09-16T15:46:58.603021+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:57.531046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:57.531046+0000","last_clean_scrub_stamp":"2024-09-16T15:46:57.531046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T02:09:19.272595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.13","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-09-16T15:46:58.569492+0000","last_change":"2024-09-16T15:46:58.569492+0000","last_active":"2024-09-16T15:46:58.541934+0000","last_peered":"2024-09-16T15:46:58.541934+0000","last_clean":"2024-09-16T15:46:58.541934+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:46:58.569492+0000","last_undegraded":"2024-09-16T15:46:58.569492+0000","last_fullsized":"2024-09-16T15:46:58.569492+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_clean_scrub_stamp":"2024-09-16T15:46:58.541934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.12","version":"0'0","reported_seq":10,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-16T15:46:58.602887+0000","last_change":"2024-09-16T15:46:58.595177+0000","last_active":"2024-09-16T15:46:58.602887+0000","last_peered":"2024-09-16T15:46:58.602887+0000","last_clean":"2024-09-16T15:46:58.602887+0000","last_became_active":"2024-09-16T15:46:58.593818+0000","last_became_peered":"2024-09-16T15:46:58.593818+0000","last_unstale":"2024-09-16T15:46:58.602887+0000","last_undegraded":"2024-09-16T15:46:58.602887+0000","last_fullsized":"2024-09-16T15:46:58.602887+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:57.531046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:57.531046+0000","last_clean_scrub_stamp":"2024-09-16T15:46:57.531046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T01:59:03.824917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.10","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-09-16T15:46:58.562563+0000","last_change":"2024-09-16T15:46:58.562563+0000","last_active":"2024-09-16T15:46:58.541934+0000","last_peered":"2024-09-16T15:46:58.541934+0000","last_clean":"2024-09-16T15:46:58.541934+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:46:58.562563+0000","last_undegraded":"2024-09-16T15:46:58.562563+0000","last_fullsized":"2024-09-16T15:46:58.562563+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_clean_scrub_stamp":"2024-09-16T15:46:58.541934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.11","version":"0'0","reported_seq":10,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-16T15:46:58.595218+0000","last_change":"2024-09-16T15:46:58.584495+0000","last_active":"2024-09-16T15:46:58.595218+0000","last_peered":"2024-09-16T15:46:58.595218+0000","last_clean":"2024-09-16T15:46:58.595218+0000","last_became_active":"2024-09-16T15:46:58.583741+0000","last_became_peered":"2024-09-16T15:46:58.583741+0000","last_unstale":"2024-09-16T15:46:58.595218+0000","last_undegraded":"2024-09-16T15:46:58.595218+0000","last_fullsized":"2024-09-16T15:46:58.595218+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:57.531046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:57.531046+0000","last_clean_scrub_stamp":"2024-09-16T15:46:57.531046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T01:58:08.477199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.11","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-09-16T15:46:58.560818+0000","last_change":"2024-09-16T15:46:58.560818+0000","last_active":"2024-09-16T15:46:58.541934+0000","last_peered":"2024-09-16T15:46:58.541934+0000","last_clean":"2024-09-16T15:46:58.541934+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:46:58.560818+0000","last_undegraded":"2024-09-16T15:46:58.560818+0000","last_fullsized":"2024-09-16T15:46:58.560818+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_clean_scrub_stamp":"2024-09-16T15:46:58.541934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.10","version":"0'0","reported_seq":10,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-16T15:46:58.604683+0000","last_change":"2024-09-16T15:46:58.597568+0000","last_active":"2024-09-16T15:46:58.604683+0000","last_peered":"2024-09-16T15:46:58.604683+0000","last_clean":"2024-09-16T15:46:58.604683+0000","last_became_active":"2024-09-16T15:46:58.596806+0000","last_became_peered":"2024-09-16T15:46:58.596806+0000","last_unstale":"2024-09-16T15:46:58.604683+0000","last_undegraded":"2024-09-16T15:46:58.604683+0000","last_fullsized":"2024-09-16T15:46:58.604683+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:57.531046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:57.531046+0000","last_clean_scrub_stamp":"2024-09-16T15:46:57.531046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:33:54.157169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.e","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-09-16T15:46:58.561429+0000","last_change":"2024-09-16T15:46:58.561429+0000","last_active":"2024-09-16T15:46:58.541934+0000","last_peered":"2024-09-16T15:46:58.541934+0000","last_clean":"2024-09-16T15:46:58.541934+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:46:58.561429+0000","last_undegraded":"2024-09-16T15:46:58.561429+0000","last_fullsized":"2024-09-16T15:46:58.561429+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_clean_scrub_stamp":"2024-09-16T15:46:58.541934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.f","version":"0'0","reported_seq":10,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-16T15:46:58.602564+0000","last_change":"2024-09-16T15:46:58.595767+0000","last_active":"2024-09-16T15:46:58.602564+0000","last_peered":"2024-09-16T15:46:58.602564+0000","last_clean":"2024-09-16T15:46:58.602564+0000","last_became_active":"2024-09-16T15:46:58.594655+0000","last_became_peered":"2024-09-16T15:46:58.594655+0000","last_unstale":"2024-09-16T15:46:58.602564+0000","last_undegraded":"2024-09-16T15:46:58.602564+0000","last_fullsized":"2024-09-16T15:46:58.602564+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:57.531046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:57.531046+0000","last_clean_scrub_stamp":"2024-09-16T15:46:57.531046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:56:28.530413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.f","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-09-16T15:46:58.578428+0000","last_change":"2024-09-16T15:46:58.578428+0000","last_active":"2024-09-16T15:46:58.541934+0000","last_peered":"2024-09-16T15:46:58.541934+0000","last_clean":"2024-09-16T15:46:58.541934+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:46:58.578428+0000","last_undegraded":"2024-09-16T15:46:58.578428+0000","last_fullsized":"2024-09-16T15:46:58.578428+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_clean_scrub_stamp":"2024-09-16T15:46:58.541934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.e","version":"0'0","reported_seq":10,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-16T15:46:58.604653+0000","last_change":"2024-09-16T15:46:58.597152+0000","last_active":"2024-09-16T15:46:58.604653+0000","last_peered":"2024-09-16T15:46:58.604653+0000","last_clean":"2024-09-16T15:46:58.604653+0000","last_became_active":"2024-09-16T15:46:58.596435+0000","last_became_peered":"2024-09-16T15:46:58.596435+0000","last_unstale":"2024-09-16T15:46:58.604653+0000","last_undegraded":"2024-09-16T15:46:58.604653+0000","last_fullsized":"2024-09-16T15:46:58.604653+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:57.531046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:57.531046+0000","last_clean_scrub_stamp":"2024-09-16T15:46:57.531046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:11:50.475616+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.c","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-09-16T15:46:58.559398+0000","last_change":"2024-09-16T15:46:58.559398+0000","last_active":"2024-09-16T15:46:58.541934+0000","last_peered":"2024-09-16T15:46:58.541934+0000","last_clean":"2024-09-16T15:46:58.541934+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:46:58.559398+0000","last_undegraded":"2024-09-16T15:46:58.559398+0000","last_fullsized":"2024-09-16T15:46:58.559398+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_clean_scrub_stamp":"2024-09-16T15:46:58.541934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.d","version":"0'0","reported_seq":10,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-16T15:46:58.602764+0000","last_change":"2024-09-16T15:46:58.598349+0000","last_active":"2024-09-16T15:46:58.602764+0000","last_peered":"2024-09-16T15:46:58.602764+0000","last_clean":"2024-09-16T15:46:58.602764+0000","last_became_active":"2024-09-16T15:46:58.595349+0000","last_became_peered":"2024-09-16T15:46:58.595349+0000","last_unstale":"2024-09-16T15:46:58.602764+0000","last_undegraded":"2024-09-16T15:46:58.602764+0000","last_fullsized":"2024-09-16T15:46:58.602764+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:57.531046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:57.531046+0000","last_clean_scrub_stamp":"2024-09-16T15:46:57.531046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T01:43:29.245355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.d","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-09-16T15:46:58.563246+0000","last_change":"2024-09-16T15:46:58.563246+0000","last_active":"2024-09-16T15:46:58.541934+0000","last_peered":"2024-09-16T15:46:58.541934+0000","last_clean":"2024-09-16T15:46:58.541934+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:46:58.563246+0000","last_undegraded":"2024-09-16T15:46:58.563246+0000","last_fullsized":"2024-09-16T15:46:58.563246+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_clean_scrub_stamp":"2024-09-16T15:46:58.541934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.c","version":"0'0","reported_seq":10,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-16T15:46:58.599912+0000","last_change":"2024-09-16T15:46:58.596863+0000","last_active":"2024-09-16T15:46:58.599912+0000","last_peered":"2024-09-16T15:46:58.599912+0000","last_clean":"2024-09-16T15:46:58.599912+0000","last_became_active":"2024-09-16T15:46:58.596143+0000","last_became_peered":"2024-09-16T15:46:58.596143+0000","last_unstale":"2024-09-16T15:46:58.599912+0000","last_undegraded":"2024-09-16T15:46:58.599912+0000","last_fullsized":"2024-09-16T15:46:58.599912+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:57.531046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:57.531046+0000","last_clean_scrub_stamp":"2024-09-16T15:46:57.531046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:50:55.009663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.b","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-09-16T15:46:58.567734+0000","last_change":"2024-09-16T15:46:58.567734+0000","last_active":"2024-09-16T15:46:58.541934+0000","last_peered":"2024-09-16T15:46:58.541934+0000","last_clean":"2024-09-16T15:46:58.541934+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:46:58.567734+0000","last_undegraded":"2024-09-16T15:46:58.567734+0000","last_fullsized":"2024-09-16T15:46:58.567734+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_clean_scrub_stamp":"2024-09-16T15:46:58.541934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.a","version":"0'0","reported_seq":10,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-16T15:46:58.597818+0000","last_change":"2024-09-16T15:46:58.585877+0000","last_active":"2024-09-16T15:46:58.597818+0000","last_peered":"2024-09-16T15:46:58.597818+0000","last_clean":"2024-09-16T15:46:58.597818+0000","last_became_active":"2024-09-16T15:46:58.585158+0000","last_became_peered":"2024-09-16T15:46:58.585158+0000","last_unstale":"2024-09-16T15:46:58.597818+0000","last_undegraded":"2024-09-16T15:46:58.597818+0000","last_fullsized":"2024-09-16T15:46:58.597818+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:57.531046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:57.531046+0000","last_clean_scrub_stamp":"2024-09-16T15:46:57.531046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T01:00:40.058766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.8","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-09-16T15:46:58.564210+0000","last_change":"2024-09-16T15:46:58.564210+0000","last_active":"2024-09-16T15:46:58.541934+0000","last_peered":"2024-09-16T15:46:58.541934+0000","last_clean":"2024-09-16T15:46:58.541934+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:46:58.564210+0000","last_undegraded":"2024-09-16T15:46:58.564210+0000","last_fullsized":"2024-09-16T15:46:58.564210+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_clean_scrub_stamp":"2024-09-16T15:46:58.541934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.9","version":"0'0","reported_seq":10,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-16T15:46:58.583297+0000","last_change":"2024-09-16T15:46:58.576781+0000","last_active":"2024-09-16T15:46:58.583297+0000","last_peered":"2024-09-16T15:46:58.583297+0000","last_clean":"2024-09-16T15:46:58.583297+0000","last_became_active":"2024-09-16T15:46:58.576081+0000","last_became_peered":"2024-09-16T15:46:58.576081+0000","last_unstale":"2024-09-16T15:46:58.583297+0000","last_undegraded":"2024-09-16T15:46:58.583297+0000","last_fullsized":"2024-09-16T15:46:58.583297+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:57.531046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:57.531046+0000","last_clean_scrub_stamp":"2024-09-16T15:46:57.531046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T22:41:00.960336+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.9","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-09-16T15:46:58.564335+0000","last_change":"2024-09-16T15:46:58.564335+0000","last_active":"2024-09-16T15:46:58.541934+0000","last_peered":"2024-09-16T15:46:58.541934+0000","last_clean":"2024-09-16T15:46:58.541934+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:46:58.564335+0000","last_undegraded":"2024-09-16T15:46:58.564335+0000","last_fullsized":"2024-09-16T15:46:58.564335+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_clean_scrub_stamp":"2024-09-16T15:46:58.541934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.8","version":"0'0","reported_seq":10,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-16T15:46:58.577818+0000","last_change":"2024-09-16T15:46:58.572736+0000","last_active":"2024-09-16T15:46:58.577818+0000","last_peered":"2024-09-16T15:46:58.577818+0000","last_clean":"2024-09-16T15:46:58.577818+0000","last_became_active":"2024-09-16T15:46:58.572311+0000","last_became_peered":"2024-09-16T15:46:58.572311+0000","last_unstale":"2024-09-16T15:46:58.577818+0000","last_undegraded":"2024-09-16T15:46:58.577818+0000","last_fullsized":"2024-09-16T15:46:58.577818+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:57.531046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:57.531046+0000","last_clean_scrub_stamp":"2024-09-16T15:46:57.531046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:48:10.471983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.6","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-09-16T15:46:58.562758+0000","last_change":"2024-09-16T15:46:58.562758+0000","last_active":"2024-09-16T15:46:58.541934+0000","last_peered":"2024-09-16T15:46:58.541934+0000","last_clean":"2024-09-16T15:46:58.541934+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:46:58.562758+0000","last_undegraded":"2024-09-16T15:46:58.562758+0000","last_fullsized":"2024-09-16T15:46:58.562758+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_clean_scrub_stamp":"2024-09-16T15:46:58.541934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.7","version":"0'0","reported_seq":10,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-16T15:46:58.594095+0000","last_change":"2024-09-16T15:46:58.579355+0000","last_active":"2024-09-16T15:46:58.594095+0000","last_peered":"2024-09-16T15:46:58.594095+0000","last_clean":"2024-09-16T15:46:58.594095+0000","last_became_active":"2024-09-16T15:46:58.578380+0000","last_became_peered":"2024-09-16T15:46:58.578380+0000","last_unstale":"2024-09-16T15:46:58.594095+0000","last_undegraded":"2024-09-16T15:46:58.594095+0000","last_fullsized":"2024-09-16T15:46:58.594095+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:57.531046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:57.531046+0000","last_clean_scrub_stamp":"2024-09-16T15:46:57.531046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T23:40:10.473815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.7","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-09-16T15:46:58.569698+0000","last_change":"2024-09-16T15:46:58.569698+0000","last_active":"2024-09-16T15:46:58.541934+0000","last_peered":"2024-09-16T15:46:58.541934+0000","last_clean":"2024-09-16T15:46:58.541934+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:46:58.569698+0000","last_undegraded":"2024-09-16T15:46:58.569698+0000","last_fullsized":"2024-09-16T15:46:58.569698+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_clean_scrub_stamp":"2024-09-16T15:46:58.541934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.6","version":"0'0","reported_seq":10,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-16T15:46:58.582810+0000","last_change":"2024-09-16T15:46:58.577011+0000","last_active":"2024-09-16T15:46:58.582810+0000","last_peered":"2024-09-16T15:46:58.582810+0000","last_clean":"2024-09-16T15:46:58.582810+0000","last_became_active":"2024-09-16T15:46:58.576294+0000","last_became_peered":"2024-09-16T15:46:58.576294+0000","last_unstale":"2024-09-16T15:46:58.582810+0000","last_undegraded":"2024-09-16T15:46:58.582810+0000","last_fullsized":"2024-09-16T15:46:58.582810+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:57.531046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:57.531046+0000","last_clean_scrub_stamp":"2024-09-16T15:46:57.531046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T03:31:45.091717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.4","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-09-16T15:46:58.574408+0000","last_change":"2024-09-16T15:46:58.574408+0000","last_active":"2024-09-16T15:46:58.541934+0000","last_peered":"2024-09-16T15:46:58.541934+0000","last_clean":"2024-09-16T15:46:58.541934+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:46:58.574408+0000","last_undegraded":"2024-09-16T15:46:58.574408+0000","last_fullsized":"2024-09-16T15:46:58.574408+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_clean_scrub_stamp":"2024-09-16T15:46:58.541934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.5","version":"0'0","reported_seq":10,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-16T15:46:58.578312+0000","last_change":"2024-09-16T15:46:58.575511+0000","last_active":"2024-09-16T15:46:58.578312+0000","last_peered":"2024-09-16T15:46:58.578312+0000","last_clean":"2024-09-16T15:46:58.578312+0000","last_became_active":"2024-09-16T15:46:58.575218+0000","last_became_peered":"2024-09-16T15:46:58.575218+0000","last_unstale":"2024-09-16T15:46:58.578312+0000","last_undegraded":"2024-09-16T15:46:58.578312+0000","last_fullsized":"2024-09-16T15:46:58.578312+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:57.531046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:57.531046+0000","last_clean_scrub_stamp":"2024-09-16T15:46:57.531046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:37:03.438301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.5","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-09-16T15:46:58.578126+0000","last_change":"2024-09-16T15:46:58.578126+0000","last_active":"2024-09-16T15:46:58.541934+0000","last_peered":"2024-09-16T15:46:58.541934+0000","last_clean":"2024-09-16T15:46:58.541934+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:46:58.578126+0000","last_undegraded":"2024-09-16T15:46:58.578126+0000","last_fullsized":"2024-09-16T15:46:58.578126+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_clean_scrub_stamp":"2024-09-16T15:46:58.541934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.4","version":"0'0","reported_seq":10,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-16T15:46:58.588834+0000","last_change":"2024-09-16T15:46:58.583703+0000","last_active":"2024-09-16T15:46:58.588834+0000","last_peered":"2024-09-16T15:46:58.588834+0000","last_clean":"2024-09-16T15:46:58.588834+0000","last_became_active":"2024-09-16T15:46:58.583384+0000","last_became_peered":"2024-09-16T15:46:58.583384+0000","last_unstale":"2024-09-16T15:46:58.588834+0000","last_undegraded":"2024-09-16T15:46:58.588834+0000","last_fullsized":"2024-09-16T15:46:58.588834+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:57.531046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:57.531046+0000","last_clean_scrub_stamp":"2024-09-16T15:46:57.531046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T23:06:05.405748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.2","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-09-16T15:46:58.560788+0000","last_change":"2024-09-16T15:46:58.560788+0000","last_active":"2024-09-16T15:46:58.541934+0000","last_peered":"2024-09-16T15:46:58.541934+0000","last_clean":"2024-09-16T15:46:58.541934+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:46:58.560788+0000","last_undegraded":"2024-09-16T15:46:58.560788+0000","last_fullsized":"2024-09-16T15:46:58.560788+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_clean_scrub_stamp":"2024-09-16T15:46:58.541934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.3","version":"0'0","reported_seq":10,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-16T15:46:58.577602+0000","last_change":"2024-09-16T15:46:58.573698+0000","last_active":"2024-09-16T15:46:58.577602+0000","last_peered":"2024-09-16T15:46:58.577602+0000","last_clean":"2024-09-16T15:46:58.577602+0000","last_became_active":"2024-09-16T15:46:58.573391+0000","last_became_peered":"2024-09-16T15:46:58.573391+0000","last_unstale":"2024-09-16T15:46:58.577602+0000","last_undegraded":"2024-09-16T15:46:58.577602+0000","last_fullsized":"2024-09-16T15:46:58.577602+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:57.531046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:57.531046+0000","last_clean_scrub_stamp":"2024-09-16T15:46:57.531046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T00:58:45.221067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.3","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-09-16T15:46:58.586430+0000","last_change":"2024-09-16T15:46:58.586430+0000","last_active":"2024-09-16T15:46:58.541934+0000","last_peered":"2024-09-16T15:46:58.541934+0000","last_clean":"2024-09-16T15:46:58.541934+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:46:58.586430+0000","last_undegraded":"2024-09-16T15:46:58.586430+0000","last_fullsized":"2024-09-16T15:46:58.586430+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_clean_scrub_stamp":"2024-09-16T15:46:58.541934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.2","version":"0'0","reported_seq":10,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-16T15:46:58.577483+0000","last_change":"2024-09-16T15:46:58.574227+0000","last_active":"2024-09-16T15:46:58.577483+0000","last_peered":"2024-09-16T15:46:58.577483+0000","last_clean":"2024-09-16T15:46:58.577483+0000","last_became_active":"2024-09-16T15:46:58.573898+0000","last_became_peered":"2024-09-16T15:46:58.573898+0000","last_unstale":"2024-09-16T15:46:58.577483+0000","last_undegraded":"2024-09-16T15:46:58.577483+0000","last_fullsized":"2024-09-16T15:46:58.577483+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:57.531046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:57.531046+0000","last_clean_scrub_stamp":"2024-09-16T15:46:57.531046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T18:41:18.272460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.0","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-09-16T15:46:58.559237+0000","last_change":"2024-09-16T15:46:58.559237+0000","last_active":"2024-09-16T15:46:58.541934+0000","last_peered":"2024-09-16T15:46:58.541934+0000","last_clean":"2024-09-16T15:46:58.541934+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:46:58.559237+0000","last_undegraded":"2024-09-16T15:46:58.559237+0000","last_fullsized":"2024-09-16T15:46:58.559237+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_clean_scrub_stamp":"2024-09-16T15:46:58.541934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.1","version":"0'0","reported_seq":10,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-16T15:46:58.604741+0000","last_change":"2024-09-16T15:46:58.598614+0000","last_active":"2024-09-16T15:46:58.604741+0000","last_peered":"2024-09-16T15:46:58.604741+0000","last_clean":"2024-09-16T15:46:58.604741+0000","last_became_active":"2024-09-16T15:46:58.597950+0000","last_became_peered":"2024-09-16T15:46:58.597950+0000","last_unstale":"2024-09-16T15:46:58.604741+0000","last_undegraded":"2024-09-16T15:46:58.604741+0000","last_fullsized":"2024-09-16T15:46:58.604741+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:57.531046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:57.531046+0000","last_clean_scrub_stamp":"2024-09-16T15:46:57.531046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T23:50:07.979345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-09-16T15:46:58.565520+0000","last_change":"2024-09-16T15:46:58.565520+0000","last_active":"2024-09-16T15:46:58.541934+0000","last_peered":"2024-09-16T15:46:58.541934+0000","last_clean":"2024-09-16T15:46:58.541934+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:46:58.565520+0000","last_undegraded":"2024-09-16T15:46:58.565520+0000","last_fullsized":"2024-09-16T15:46:58.565520+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_clean_scrub_stamp":"2024-09-16T15:46:58.541934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.0","version":"0'0","reported_seq":10,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-16T15:46:58.602740+0000","last_change":"2024-09-16T15:46:58.599504+0000","last_active":"2024-09-16T15:46:58.602740+0000","last_peered":"2024-09-16T15:46:58.602740+0000","last_clean":"2024-09-16T15:46:58.602740+0000","last_became_active":"2024-09-16T15:46:58.598743+0000","last_became_peered":"2024-09-16T15:46:58.598743+0000","last_unstale":"2024-09-16T15:46:58.602740+0000","last_undegraded":"2024-09-16T15:46:58.602740+0000","last_fullsized":"2024-09-16T15:46:58.602740+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:57.531046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:57.531046+0000","last_clean_scrub_stamp":"2024-09-16T15:46:57.531046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T18:45:22.195277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.a","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-09-16T15:46:58.564305+0000","last_change":"2024-09-16T15:46:58.564305+0000","last_active":"2024-09-16T15:46:58.541934+0000","last_peered":"2024-09-16T15:46:58.541934+0000","last_clean":"2024-09-16T15:46:58.541934+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:46:58.564305+0000","last_undegraded":"2024-09-16T15:46:58.564305+0000","last_fullsized":"2024-09-16T15:46:58.564305+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_clean_scrub_stamp":"2024-09-16T15:46:58.541934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.b","version":"0'0","reported_seq":10,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-16T15:46:58.604289+0000","last_change":"2024-09-16T15:46:58.601513+0000","last_active":"2024-09-16T15:46:58.604289+0000","last_peered":"2024-09-16T15:46:58.604289+0000","last_clean":"2024-09-16T15:46:58.604289+0000","last_became_active":"2024-09-16T15:46:58.600581+0000","last_became_peered":"2024-09-16T15:46:58.600581+0000","last_unstale":"2024-09-16T15:46:58.604289+0000","last_undegraded":"2024-09-16T15:46:58.604289+0000","last_fullsized":"2024-09-16T15:46:58.604289+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:57.531046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:57.531046+0000","last_clean_scrub_stamp":"2024-09-16T15:46:57.531046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T23:00:41.658258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"16'72","reported_seq":200,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-16T15:46:58.585260+0000","last_change":"2024-09-16T15:42:40.881820+0000","last_active":"2024-09-16T15:46:58.585260+0000","last_peered":"2024-09-16T15:46:58.585260+0000","last_clean":"2024-09-16T15:46:58.585260+0000","last_became_active":"2024-09-16T15:42:40.879965+0000","last_became_peered":"2024-09-16T15:42:40.879965+0000","last_unstale":"2024-09-16T15:46:58.585260+0000","last_undegraded":"2024-09-16T15:46:58.585260+0000","last_fullsized":"2024-09-16T15:46:58.585260+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:42:39.728762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:42:39.728762+0000","last_clean_scrub_stamp":"2024-09-16T15:42:39.728762+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:22:18.768932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":11,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":10,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542200,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28416,"kb_used_data":1384,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343424,"statfs":{"total":96636764160,"available":96607666176,"internally_reserved":0,"allocated":1417216,"data_stored":932261,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542199,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31996,"kb_used_data":804,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339844,"statfs":{"total":96636764160,"available":96604000256,"internally_reserved":0,"allocated":823296,"data_stored":341893,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":5,"up_from":14,"seq":60129542200,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27824,"kb_used_data":792,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344016,"statfs":{"total":96636764160,"available":96608272384,"internally_reserved":0,"allocated":811008,"data_stored":335618,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542198,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31980,"kb_used_data":792,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339860,"statfs":{"total":96636764160,"available":96604016640,"internally_reserved":0,"allocated":811008,"data_stored":335618,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":3,"up_from":14,"seq":60129542199,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31996,"kb_used_data":804,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339844,"statfs":{"total":96636764160,"available":96604000256,"internally_reserved":0,"allocated":823296,"data_stored":341893,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":14,"seq":60129542199,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27820,"kb_used_data":792,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344020,"statfs":{"total":96636764160,"available":96608276480,"internally_reserved":0,"allocated":811008,"data_stored":335618,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":14,"seq":60129542201,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27840,"kb_used_data":804,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344000,"statfs":{"total":96636764160,"available":96608256000,"internally_reserved":0,"allocated":823296,"data_stored":341893,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":14,"seq":60129542201,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32572,"kb_used_data":1384,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339268,"statfs":{"total":96636764160,"available":96603410432,"internally_reserved":0,"allocated":1417216,"data_stored":932261,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T15:47:03.631 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T15:47:03.792 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:03.788+0000 7f585203e640 1 -- 172.21.15.123:0/253911421 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5834004810 msgr2=0x7f5834004c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:47:03.793 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:03.788+0000 7f585203e640 1 --2- 172.21.15.123:0/253911421 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5834004810 0x7f5834004c90 secure :-1 s=READY pgs=788 cs=0 l=1 rev1=1 crypto rx=0x7f5840009b60 tx=0x7f584001d270 comp rx=0 tx=0).stop 2024-09-16T15:47:03.793 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:03.788+0000 7f584bfff640 1 -- 172.21.15.123:0/253911421 reap_dead start 2024-09-16T15:47:03.793 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:03.788+0000 7f585203e640 1 -- 172.21.15.123:0/253911421 shutdown_connections 2024-09-16T15:47:03.793 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:03.788+0000 7f585203e640 1 -- 172.21.15.123:0/253911421 >> 172.21.15.123:0/253911421 conn(0x7f584c072150 msgr2=0x7f584c072560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:47:03.793 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:03.788+0000 7f585203e640 1 -- 172.21.15.123:0/253911421 shutdown_connections 2024-09-16T15:47:03.793 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:03.788+0000 7f585203e640 1 -- 172.21.15.123:0/253911421 wait complete. 2024-09-16T15:47:03.796 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:03.788+0000 7f585203e640 1 Processor -- start 2024-09-16T15:47:03.796 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:03.788+0000 7f585203e640 1 -- start start 2024-09-16T15:47:03.796 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:03.788+0000 7f585203e640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f5834004810 0x7f584c14ef70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:47:03.796 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:03.788+0000 7f585203e640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f584c144530 0x7f584c14d370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:47:03.796 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:03.788+0000 7f585203e640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f584c14d8b0 0x7f584c15ae40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:47:03.796 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:03.788+0000 7f585203e640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f584c15cf90 con 0x7f584c14d8b0 2024-09-16T15:47:03.796 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:03.788+0000 7f585203e640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f584c15ce10 con 0x7f584c144530 2024-09-16T15:47:03.796 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:03.788+0000 7f585203e640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f584c0cdb50 con 0x7f5834004810 2024-09-16T15:47:03.796 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:03.788+0000 7f584bfff640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f584c14d8b0 0x7f584c15ae40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:47:03.796 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:03.788+0000 7f584bfff640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f584c14d8b0 0x7f584c15ae40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:46840/0 (socket says 172.21.15.123:46840) 2024-09-16T15:47:03.797 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:03.788+0000 7f584bfff640 1 -- 172.21.15.123:0/3307278997 learned_addr learned my addr 172.21.15.123:0/3307278997 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:47:03.797 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:03.792+0000 7f584b7fe640 1 --2- 172.21.15.123:0/3307278997 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f5834004810 0x7f584c14ef70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:47:03.797 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:03.792+0000 7f584affd640 1 --2- 172.21.15.123:0/3307278997 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f584c144530 0x7f584c14d370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:47:03.797 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:03.792+0000 7f584bfff640 1 -- 172.21.15.123:0/3307278997 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f5834004810 msgr2=0x7f584c14ef70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:47:03.797 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:03.792+0000 7f584bfff640 1 --2- 172.21.15.123:0/3307278997 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f5834004810 0x7f584c14ef70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:03.797 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:03.792+0000 7f584bfff640 1 -- 172.21.15.123:0/3307278997 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f584c144530 msgr2=0x7f584c14d370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:47:03.797 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:03.792+0000 7f584bfff640 1 --2- 172.21.15.123:0/3307278997 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f584c144530 0x7f584c14d370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:03.797 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:03.792+0000 7f584bfff640 1 -- 172.21.15.123:0/3307278997 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f584c1f8300 con 0x7f584c14d8b0 2024-09-16T15:47:03.797 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:03.792+0000 7f584affd640 1 --2- 172.21.15.123:0/3307278997 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f584c144530 0x7f584c14d370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T15:47:03.797 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:03.792+0000 7f584bfff640 1 --2- 172.21.15.123:0/3307278997 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f584c14d8b0 0x7f584c15ae40 secure :-1 s=READY pgs=789 cs=0 l=1 rev1=1 crypto rx=0x7f584c0c5690 tx=0x7f584400fb90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:47:03.797 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:03.792+0000 7f5833fff640 1 -- 172.21.15.123:0/3307278997 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f584400d810 con 0x7f584c14d8b0 2024-09-16T15:47:03.798 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:03.792+0000 7f5833fff640 1 -- 172.21.15.123:0/3307278997 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f584400daa0 con 0x7f584c14d8b0 2024-09-16T15:47:03.798 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:03.792+0000 7f585203e640 1 -- 172.21.15.123:0/3307278997 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f584c1f85e0 con 0x7f584c14d8b0 2024-09-16T15:47:03.798 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:03.792+0000 7f5833fff640 1 -- 172.21.15.123:0/3307278997 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f584400a0d0 con 0x7f584c14d8b0 2024-09-16T15:47:03.798 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:03.792+0000 7f585203e640 1 -- 172.21.15.123:0/3307278997 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f584c1f8ba0 con 0x7f584c14d8b0 2024-09-16T15:47:03.799 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:03.792+0000 7f585203e640 1 -- 172.21.15.123:0/3307278997 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f584c1607d0 con 0x7f584c14d8b0 2024-09-16T15:47:03.799 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:03.792+0000 7f5833fff640 1 -- 172.21.15.123:0/3307278997 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 11) ==== 97386+0+0 (secure 0 0 0) 0x7f5844014070 con 0x7f584c14d8b0 2024-09-16T15:47:03.799 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:03.792+0000 7f5833fff640 1 --2- 172.21.15.123:0/3307278997 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f5838073250 0x7f5838075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:47:03.799 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:03.792+0000 7f5833fff640 1 -- 172.21.15.123:0/3307278997 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(70..70 src has 1..70) ==== 5709+0+0 (secure 0 0 0) 0x7f5844096bf0 con 0x7f584c14d8b0 2024-09-16T15:47:03.799 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:03.792+0000 7f584b7fe640 1 --2- 172.21.15.123:0/3307278997 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f5838073250 0x7f5838075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:47:03.800 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:03.796+0000 7f584b7fe640 1 --2- 172.21.15.123:0/3307278997 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f5838073250 0x7f5838075710 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f583c0099f0 tx=0x7f583c0091e0 comp rx=0 tx=0).ready entity=mgr.4135 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:47:03.802 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:03.796+0000 7f5833fff640 1 -- 172.21.15.123:0/3307278997 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f584401f020 con 0x7f584c14d8b0 2024-09-16T15:47:03.965 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:03.960+0000 7f585203e640 1 -- 172.21.15.123:0/3307278997 --> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f584c072970 con 0x7f5838073250 2024-09-16T15:47:03.977 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:03.972+0000 7f5833fff640 1 -- 172.21.15.123:0/3307278997 <== mgr.4135 v2:172.21.15.184:6832/3114346423 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+246774 (secure 0 0 0) 0x7f584c072970 con 0x7f5838073250 2024-09-16T15:47:03.980 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-09-16T15:47:03.981 INFO:teuthology.orchestra.run.smithi123.stderr:dumped all 2024-09-16T15:47:03.982 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:03.976+0000 7f585203e640 1 -- 172.21.15.123:0/3307278997 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f5838073250 msgr2=0x7f5838075710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:47:03.982 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:03.976+0000 7f585203e640 1 --2- 172.21.15.123:0/3307278997 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f5838073250 0x7f5838075710 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f583c0099f0 tx=0x7f583c0091e0 comp rx=0 tx=0).stop 2024-09-16T15:47:03.982 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:03.976+0000 7f585203e640 1 -- 172.21.15.123:0/3307278997 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f584c14d8b0 msgr2=0x7f584c15ae40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:47:03.982 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:03.976+0000 7f585203e640 1 --2- 172.21.15.123:0/3307278997 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f584c14d8b0 0x7f584c15ae40 secure :-1 s=READY pgs=789 cs=0 l=1 rev1=1 crypto rx=0x7f584c0c5690 tx=0x7f584400fb90 comp rx=0 tx=0).stop 2024-09-16T15:47:03.982 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:03.976+0000 7f585203e640 1 -- 172.21.15.123:0/3307278997 shutdown_connections 2024-09-16T15:47:03.983 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:03.976+0000 7f585203e640 1 --2- 172.21.15.123:0/3307278997 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f5838073250 0x7f5838075710 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:03.983 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:03.976+0000 7f585203e640 1 --2- 172.21.15.123:0/3307278997 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f584c14d8b0 0x7f584c15ae40 unknown :-1 s=CLOSED pgs=789 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:03.983 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:03.976+0000 7f585203e640 1 --2- 172.21.15.123:0/3307278997 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f584c144530 0x7f584c14d370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:03.983 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:03.976+0000 7f585203e640 1 --2- 172.21.15.123:0/3307278997 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f5834004810 0x7f584c14ef70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:03.983 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:03.976+0000 7f585203e640 1 -- 172.21.15.123:0/3307278997 >> 172.21.15.123:0/3307278997 conn(0x7f584c072150 msgr2=0x7f584c1600c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:47:03.983 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:03.976+0000 7f585203e640 1 -- 172.21.15.123:0/3307278997 shutdown_connections 2024-09-16T15:47:03.983 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:03.976+0000 7f585203e640 1 -- 172.21.15.123:0/3307278997 wait complete. 2024-09-16T15:47:03.996 INFO:teuthology.orchestra.run.smithi123.stdout:{"pg_ready":true,"pg_map":{"version":195,"stamp":"2024-09-16T15:47:03.331208+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":76,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":240444,"kb_used_data":7556,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":754734276,"statfs":{"total":773094113280,"available":772847898624,"internally_reserved":0,"allocated":7737344,"data_stored":3897055,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":238276145},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":41,"apply_latency_ms":41,"commit_latency_ns":41000000,"apply_latency_ns":41000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.788118"},"pg_stats":[{"pgid":"11.3f","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-09-16T15:46:58.580896+0000","last_change":"2024-09-16T15:46:58.580896+0000","last_active":"2024-09-16T15:46:58.541934+0000","last_peered":"2024-09-16T15:46:58.541934+0000","last_clean":"2024-09-16T15:46:58.541934+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:46:58.580896+0000","last_undegraded":"2024-09-16T15:46:58.580896+0000","last_fullsized":"2024-09-16T15:46:58.580896+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_clean_scrub_stamp":"2024-09-16T15:46:58.541934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.3e","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-09-16T15:46:58.580879+0000","last_change":"2024-09-16T15:46:58.580879+0000","last_active":"2024-09-16T15:46:58.541934+0000","last_peered":"2024-09-16T15:46:58.541934+0000","last_clean":"2024-09-16T15:46:58.541934+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:46:58.580879+0000","last_undegraded":"2024-09-16T15:46:58.580879+0000","last_fullsized":"2024-09-16T15:46:58.580879+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_clean_scrub_stamp":"2024-09-16T15:46:58.541934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.3d","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-09-16T15:46:58.565500+0000","last_change":"2024-09-16T15:46:58.565500+0000","last_active":"2024-09-16T15:46:58.541934+0000","last_peered":"2024-09-16T15:46:58.541934+0000","last_clean":"2024-09-16T15:46:58.541934+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:46:58.565500+0000","last_undegraded":"2024-09-16T15:46:58.565500+0000","last_fullsized":"2024-09-16T15:46:58.565500+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_clean_scrub_stamp":"2024-09-16T15:46:58.541934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.3c","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-09-16T15:46:58.560961+0000","last_change":"2024-09-16T15:46:58.560961+0000","last_active":"2024-09-16T15:46:58.541934+0000","last_peered":"2024-09-16T15:46:58.541934+0000","last_clean":"2024-09-16T15:46:58.541934+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:46:58.560961+0000","last_undegraded":"2024-09-16T15:46:58.560961+0000","last_fullsized":"2024-09-16T15:46:58.560961+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_clean_scrub_stamp":"2024-09-16T15:46:58.541934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.3b","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-09-16T15:46:58.561715+0000","last_change":"2024-09-16T15:46:58.561715+0000","last_active":"2024-09-16T15:46:58.541934+0000","last_peered":"2024-09-16T15:46:58.541934+0000","last_clean":"2024-09-16T15:46:58.541934+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:46:58.561715+0000","last_undegraded":"2024-09-16T15:46:58.561715+0000","last_fullsized":"2024-09-16T15:46:58.561715+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_clean_scrub_stamp":"2024-09-16T15:46:58.541934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.3a","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-09-16T15:46:58.578627+0000","last_change":"2024-09-16T15:46:58.578627+0000","last_active":"2024-09-16T15:46:58.541934+0000","last_peered":"2024-09-16T15:46:58.541934+0000","last_clean":"2024-09-16T15:46:58.541934+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:46:58.578627+0000","last_undegraded":"2024-09-16T15:46:58.578627+0000","last_fullsized":"2024-09-16T15:46:58.578627+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_clean_scrub_stamp":"2024-09-16T15:46:58.541934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.39","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-09-16T15:46:58.570892+0000","last_change":"2024-09-16T15:46:58.570892+0000","last_active":"2024-09-16T15:46:58.541934+0000","last_peered":"2024-09-16T15:46:58.541934+0000","last_clean":"2024-09-16T15:46:58.541934+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:46:58.570892+0000","last_undegraded":"2024-09-16T15:46:58.570892+0000","last_fullsized":"2024-09-16T15:46:58.570892+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_clean_scrub_stamp":"2024-09-16T15:46:58.541934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.38","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-09-16T15:46:58.584780+0000","last_change":"2024-09-16T15:46:58.584780+0000","last_active":"2024-09-16T15:46:58.541934+0000","last_peered":"2024-09-16T15:46:58.541934+0000","last_clean":"2024-09-16T15:46:58.541934+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:46:58.584780+0000","last_undegraded":"2024-09-16T15:46:58.584780+0000","last_fullsized":"2024-09-16T15:46:58.584780+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_clean_scrub_stamp":"2024-09-16T15:46:58.541934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.37","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-09-16T15:46:58.561912+0000","last_change":"2024-09-16T15:46:58.561912+0000","last_active":"2024-09-16T15:46:58.541934+0000","last_peered":"2024-09-16T15:46:58.541934+0000","last_clean":"2024-09-16T15:46:58.541934+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:46:58.561912+0000","last_undegraded":"2024-09-16T15:46:58.561912+0000","last_fullsized":"2024-09-16T15:46:58.561912+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_clean_scrub_stamp":"2024-09-16T15:46:58.541934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.36","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-09-16T15:46:58.562601+0000","last_change":"2024-09-16T15:46:58.562601+0000","last_active":"2024-09-16T15:46:58.541934+0000","last_peered":"2024-09-16T15:46:58.541934+0000","last_clean":"2024-09-16T15:46:58.541934+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:46:58.562601+0000","last_undegraded":"2024-09-16T15:46:58.562601+0000","last_fullsized":"2024-09-16T15:46:58.562601+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_clean_scrub_stamp":"2024-09-16T15:46:58.541934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.35","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-09-16T15:46:58.580321+0000","last_change":"2024-09-16T15:46:58.580321+0000","last_active":"2024-09-16T15:46:58.541934+0000","last_peered":"2024-09-16T15:46:58.541934+0000","last_clean":"2024-09-16T15:46:58.541934+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:46:58.580321+0000","last_undegraded":"2024-09-16T15:46:58.580321+0000","last_fullsized":"2024-09-16T15:46:58.580321+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_clean_scrub_stamp":"2024-09-16T15:46:58.541934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.34","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-09-16T15:46:58.560127+0000","last_change":"2024-09-16T15:46:58.560127+0000","last_active":"2024-09-16T15:46:58.541934+0000","last_peered":"2024-09-16T15:46:58.541934+0000","last_clean":"2024-09-16T15:46:58.541934+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:46:58.560127+0000","last_undegraded":"2024-09-16T15:46:58.560127+0000","last_fullsized":"2024-09-16T15:46:58.560127+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_clean_scrub_stamp":"2024-09-16T15:46:58.541934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.33","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-09-16T15:46:58.562521+0000","last_change":"2024-09-16T15:46:58.562521+0000","last_active":"2024-09-16T15:46:58.541934+0000","last_peered":"2024-09-16T15:46:58.541934+0000","last_clean":"2024-09-16T15:46:58.541934+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:46:58.562521+0000","last_undegraded":"2024-09-16T15:46:58.562521+0000","last_fullsized":"2024-09-16T15:46:58.562521+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_clean_scrub_stamp":"2024-09-16T15:46:58.541934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.32","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-09-16T15:46:58.561306+0000","last_change":"2024-09-16T15:46:58.561306+0000","last_active":"2024-09-16T15:46:58.541934+0000","last_peered":"2024-09-16T15:46:58.541934+0000","last_clean":"2024-09-16T15:46:58.541934+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:46:58.561306+0000","last_undegraded":"2024-09-16T15:46:58.561306+0000","last_fullsized":"2024-09-16T15:46:58.561306+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_clean_scrub_stamp":"2024-09-16T15:46:58.541934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.31","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-09-16T15:46:58.566448+0000","last_change":"2024-09-16T15:46:58.566448+0000","last_active":"2024-09-16T15:46:58.541934+0000","last_peered":"2024-09-16T15:46:58.541934+0000","last_clean":"2024-09-16T15:46:58.541934+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:46:58.566448+0000","last_undegraded":"2024-09-16T15:46:58.566448+0000","last_fullsized":"2024-09-16T15:46:58.566448+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_clean_scrub_stamp":"2024-09-16T15:46:58.541934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.30","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-09-16T15:46:58.560118+0000","last_change":"2024-09-16T15:46:58.560118+0000","last_active":"2024-09-16T15:46:58.541934+0000","last_peered":"2024-09-16T15:46:58.541934+0000","last_clean":"2024-09-16T15:46:58.541934+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:46:58.560118+0000","last_undegraded":"2024-09-16T15:46:58.560118+0000","last_fullsized":"2024-09-16T15:46:58.560118+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_clean_scrub_stamp":"2024-09-16T15:46:58.541934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.2f","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-09-16T15:46:58.561316+0000","last_change":"2024-09-16T15:46:58.561316+0000","last_active":"2024-09-16T15:46:58.541934+0000","last_peered":"2024-09-16T15:46:58.541934+0000","last_clean":"2024-09-16T15:46:58.541934+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:46:58.561316+0000","last_undegraded":"2024-09-16T15:46:58.561316+0000","last_fullsized":"2024-09-16T15:46:58.561316+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_clean_scrub_stamp":"2024-09-16T15:46:58.541934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.2e","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-09-16T15:46:58.559376+0000","last_change":"2024-09-16T15:46:58.559376+0000","last_active":"2024-09-16T15:46:58.541934+0000","last_peered":"2024-09-16T15:46:58.541934+0000","last_clean":"2024-09-16T15:46:58.541934+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:46:58.559376+0000","last_undegraded":"2024-09-16T15:46:58.559376+0000","last_fullsized":"2024-09-16T15:46:58.559376+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_clean_scrub_stamp":"2024-09-16T15:46:58.541934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.2d","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-09-16T15:46:58.580101+0000","last_change":"2024-09-16T15:46:58.580101+0000","last_active":"2024-09-16T15:46:58.541934+0000","last_peered":"2024-09-16T15:46:58.541934+0000","last_clean":"2024-09-16T15:46:58.541934+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:46:58.580101+0000","last_undegraded":"2024-09-16T15:46:58.580101+0000","last_fullsized":"2024-09-16T15:46:58.580101+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_clean_scrub_stamp":"2024-09-16T15:46:58.541934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.2c","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-09-16T15:46:58.562539+0000","last_change":"2024-09-16T15:46:58.562539+0000","last_active":"2024-09-16T15:46:58.541934+0000","last_peered":"2024-09-16T15:46:58.541934+0000","last_clean":"2024-09-16T15:46:58.541934+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:46:58.562539+0000","last_undegraded":"2024-09-16T15:46:58.562539+0000","last_fullsized":"2024-09-16T15:46:58.562539+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_clean_scrub_stamp":"2024-09-16T15:46:58.541934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.2b","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-09-16T15:46:58.564602+0000","last_change":"2024-09-16T15:46:58.564602+0000","last_active":"2024-09-16T15:46:58.541934+0000","last_peered":"2024-09-16T15:46:58.541934+0000","last_clean":"2024-09-16T15:46:58.541934+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:46:58.564602+0000","last_undegraded":"2024-09-16T15:46:58.564602+0000","last_fullsized":"2024-09-16T15:46:58.564602+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_clean_scrub_stamp":"2024-09-16T15:46:58.541934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.2a","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-09-16T15:46:58.562954+0000","last_change":"2024-09-16T15:46:58.562954+0000","last_active":"2024-09-16T15:46:58.541934+0000","last_peered":"2024-09-16T15:46:58.541934+0000","last_clean":"2024-09-16T15:46:58.541934+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:46:58.562954+0000","last_undegraded":"2024-09-16T15:46:58.562954+0000","last_fullsized":"2024-09-16T15:46:58.562954+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_clean_scrub_stamp":"2024-09-16T15:46:58.541934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.29","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-09-16T15:46:58.578047+0000","last_change":"2024-09-16T15:46:58.578047+0000","last_active":"2024-09-16T15:46:58.541934+0000","last_peered":"2024-09-16T15:46:58.541934+0000","last_clean":"2024-09-16T15:46:58.541934+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:46:58.578047+0000","last_undegraded":"2024-09-16T15:46:58.578047+0000","last_fullsized":"2024-09-16T15:46:58.578047+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_clean_scrub_stamp":"2024-09-16T15:46:58.541934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.28","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-09-16T15:46:58.575628+0000","last_change":"2024-09-16T15:46:58.575628+0000","last_active":"2024-09-16T15:46:58.541934+0000","last_peered":"2024-09-16T15:46:58.541934+0000","last_clean":"2024-09-16T15:46:58.541934+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:46:58.575628+0000","last_undegraded":"2024-09-16T15:46:58.575628+0000","last_fullsized":"2024-09-16T15:46:58.575628+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_clean_scrub_stamp":"2024-09-16T15:46:58.541934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.27","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-09-16T15:46:58.562122+0000","last_change":"2024-09-16T15:46:58.562122+0000","last_active":"2024-09-16T15:46:58.541934+0000","last_peered":"2024-09-16T15:46:58.541934+0000","last_clean":"2024-09-16T15:46:58.541934+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:46:58.562122+0000","last_undegraded":"2024-09-16T15:46:58.562122+0000","last_fullsized":"2024-09-16T15:46:58.562122+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_clean_scrub_stamp":"2024-09-16T15:46:58.541934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.26","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-09-16T15:46:58.571812+0000","last_change":"2024-09-16T15:46:58.571812+0000","last_active":"2024-09-16T15:46:58.541934+0000","last_peered":"2024-09-16T15:46:58.541934+0000","last_clean":"2024-09-16T15:46:58.541934+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:46:58.571812+0000","last_undegraded":"2024-09-16T15:46:58.571812+0000","last_fullsized":"2024-09-16T15:46:58.571812+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_clean_scrub_stamp":"2024-09-16T15:46:58.541934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.25","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-09-16T15:46:58.564846+0000","last_change":"2024-09-16T15:46:58.564846+0000","last_active":"2024-09-16T15:46:58.541934+0000","last_peered":"2024-09-16T15:46:58.541934+0000","last_clean":"2024-09-16T15:46:58.541934+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:46:58.564846+0000","last_undegraded":"2024-09-16T15:46:58.564846+0000","last_fullsized":"2024-09-16T15:46:58.564846+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_clean_scrub_stamp":"2024-09-16T15:46:58.541934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.24","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-09-16T15:46:58.560367+0000","last_change":"2024-09-16T15:46:58.560367+0000","last_active":"2024-09-16T15:46:58.541934+0000","last_peered":"2024-09-16T15:46:58.541934+0000","last_clean":"2024-09-16T15:46:58.541934+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:46:58.560367+0000","last_undegraded":"2024-09-16T15:46:58.560367+0000","last_fullsized":"2024-09-16T15:46:58.560367+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_clean_scrub_stamp":"2024-09-16T15:46:58.541934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.23","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-09-16T15:46:58.568304+0000","last_change":"2024-09-16T15:46:58.568304+0000","last_active":"2024-09-16T15:46:58.541934+0000","last_peered":"2024-09-16T15:46:58.541934+0000","last_clean":"2024-09-16T15:46:58.541934+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:46:58.568304+0000","last_undegraded":"2024-09-16T15:46:58.568304+0000","last_fullsized":"2024-09-16T15:46:58.568304+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_clean_scrub_stamp":"2024-09-16T15:46:58.541934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.22","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-09-16T15:46:58.569453+0000","last_change":"2024-09-16T15:46:58.569453+0000","last_active":"2024-09-16T15:46:58.541934+0000","last_peered":"2024-09-16T15:46:58.541934+0000","last_clean":"2024-09-16T15:46:58.541934+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:46:58.569453+0000","last_undegraded":"2024-09-16T15:46:58.569453+0000","last_fullsized":"2024-09-16T15:46:58.569453+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_clean_scrub_stamp":"2024-09-16T15:46:58.541934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.21","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-09-16T15:46:58.569512+0000","last_change":"2024-09-16T15:46:58.569512+0000","last_active":"2024-09-16T15:46:58.541934+0000","last_peered":"2024-09-16T15:46:58.541934+0000","last_clean":"2024-09-16T15:46:58.541934+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:46:58.569512+0000","last_undegraded":"2024-09-16T15:46:58.569512+0000","last_fullsized":"2024-09-16T15:46:58.569512+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_clean_scrub_stamp":"2024-09-16T15:46:58.541934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.20","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-09-16T15:46:58.582821+0000","last_change":"2024-09-16T15:46:58.582821+0000","last_active":"2024-09-16T15:46:58.541934+0000","last_peered":"2024-09-16T15:46:58.541934+0000","last_clean":"2024-09-16T15:46:58.541934+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:46:58.582821+0000","last_undegraded":"2024-09-16T15:46:58.582821+0000","last_fullsized":"2024-09-16T15:46:58.582821+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_clean_scrub_stamp":"2024-09-16T15:46:58.541934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.1e","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-09-16T15:46:58.575147+0000","last_change":"2024-09-16T15:46:58.575147+0000","last_active":"2024-09-16T15:46:58.541934+0000","last_peered":"2024-09-16T15:46:58.541934+0000","last_clean":"2024-09-16T15:46:58.541934+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:46:58.575147+0000","last_undegraded":"2024-09-16T15:46:58.575147+0000","last_fullsized":"2024-09-16T15:46:58.575147+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_clean_scrub_stamp":"2024-09-16T15:46:58.541934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.1f","version":"0'0","reported_seq":10,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-16T15:46:58.599998+0000","last_change":"2024-09-16T15:46:58.596071+0000","last_active":"2024-09-16T15:46:58.599998+0000","last_peered":"2024-09-16T15:46:58.599998+0000","last_clean":"2024-09-16T15:46:58.599998+0000","last_became_active":"2024-09-16T15:46:58.595309+0000","last_became_peered":"2024-09-16T15:46:58.595309+0000","last_unstale":"2024-09-16T15:46:58.599998+0000","last_undegraded":"2024-09-16T15:46:58.599998+0000","last_fullsized":"2024-09-16T15:46:58.599998+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:57.531046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:57.531046+0000","last_clean_scrub_stamp":"2024-09-16T15:46:57.531046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:37:47.205740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.1f","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-09-16T15:46:58.560943+0000","last_change":"2024-09-16T15:46:58.560943+0000","last_active":"2024-09-16T15:46:58.541934+0000","last_peered":"2024-09-16T15:46:58.541934+0000","last_clean":"2024-09-16T15:46:58.541934+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:46:58.560943+0000","last_undegraded":"2024-09-16T15:46:58.560943+0000","last_fullsized":"2024-09-16T15:46:58.560943+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_clean_scrub_stamp":"2024-09-16T15:46:58.541934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.1e","version":"0'0","reported_seq":10,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-16T15:46:58.577768+0000","last_change":"2024-09-16T15:46:58.572883+0000","last_active":"2024-09-16T15:46:58.577768+0000","last_peered":"2024-09-16T15:46:58.577768+0000","last_clean":"2024-09-16T15:46:58.577768+0000","last_became_active":"2024-09-16T15:46:58.572508+0000","last_became_peered":"2024-09-16T15:46:58.572508+0000","last_unstale":"2024-09-16T15:46:58.577768+0000","last_undegraded":"2024-09-16T15:46:58.577768+0000","last_fullsized":"2024-09-16T15:46:58.577768+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:57.531046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:57.531046+0000","last_clean_scrub_stamp":"2024-09-16T15:46:57.531046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:16:24.337739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1c","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-09-16T15:46:58.583638+0000","last_change":"2024-09-16T15:46:58.583638+0000","last_active":"2024-09-16T15:46:58.541934+0000","last_peered":"2024-09-16T15:46:58.541934+0000","last_clean":"2024-09-16T15:46:58.541934+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:46:58.583638+0000","last_undegraded":"2024-09-16T15:46:58.583638+0000","last_fullsized":"2024-09-16T15:46:58.583638+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_clean_scrub_stamp":"2024-09-16T15:46:58.541934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.1d","version":"0'0","reported_seq":10,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-16T15:46:58.604745+0000","last_change":"2024-09-16T15:46:58.598772+0000","last_active":"2024-09-16T15:46:58.604745+0000","last_peered":"2024-09-16T15:46:58.604745+0000","last_clean":"2024-09-16T15:46:58.604745+0000","last_became_active":"2024-09-16T15:46:58.597889+0000","last_became_peered":"2024-09-16T15:46:58.597889+0000","last_unstale":"2024-09-16T15:46:58.604745+0000","last_undegraded":"2024-09-16T15:46:58.604745+0000","last_fullsized":"2024-09-16T15:46:58.604745+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:57.531046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:57.531046+0000","last_clean_scrub_stamp":"2024-09-16T15:46:57.531046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:20:50.612777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1d","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-09-16T15:46:58.579199+0000","last_change":"2024-09-16T15:46:58.579199+0000","last_active":"2024-09-16T15:46:58.541934+0000","last_peered":"2024-09-16T15:46:58.541934+0000","last_clean":"2024-09-16T15:46:58.541934+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:46:58.579199+0000","last_undegraded":"2024-09-16T15:46:58.579199+0000","last_fullsized":"2024-09-16T15:46:58.579199+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_clean_scrub_stamp":"2024-09-16T15:46:58.541934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.1c","version":"0'0","reported_seq":10,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-16T15:46:58.595606+0000","last_change":"2024-09-16T15:46:58.588644+0000","last_active":"2024-09-16T15:46:58.595606+0000","last_peered":"2024-09-16T15:46:58.595606+0000","last_clean":"2024-09-16T15:46:58.595606+0000","last_became_active":"2024-09-16T15:46:58.587899+0000","last_became_peered":"2024-09-16T15:46:58.587899+0000","last_unstale":"2024-09-16T15:46:58.595606+0000","last_undegraded":"2024-09-16T15:46:58.595606+0000","last_fullsized":"2024-09-16T15:46:58.595606+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:57.531046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:57.531046+0000","last_clean_scrub_stamp":"2024-09-16T15:46:57.531046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T00:36:13.054311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.1a","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-09-16T15:46:58.568314+0000","last_change":"2024-09-16T15:46:58.568314+0000","last_active":"2024-09-16T15:46:58.541934+0000","last_peered":"2024-09-16T15:46:58.541934+0000","last_clean":"2024-09-16T15:46:58.541934+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:46:58.568314+0000","last_undegraded":"2024-09-16T15:46:58.568314+0000","last_fullsized":"2024-09-16T15:46:58.568314+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_clean_scrub_stamp":"2024-09-16T15:46:58.541934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.1b","version":"0'0","reported_seq":10,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-16T15:46:58.594471+0000","last_change":"2024-09-16T15:46:58.591621+0000","last_active":"2024-09-16T15:46:58.594471+0000","last_peered":"2024-09-16T15:46:58.594471+0000","last_clean":"2024-09-16T15:46:58.594471+0000","last_became_active":"2024-09-16T15:46:58.591311+0000","last_became_peered":"2024-09-16T15:46:58.591311+0000","last_unstale":"2024-09-16T15:46:58.594471+0000","last_undegraded":"2024-09-16T15:46:58.594471+0000","last_fullsized":"2024-09-16T15:46:58.594471+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:57.531046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:57.531046+0000","last_clean_scrub_stamp":"2024-09-16T15:46:57.531046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:55:01.184561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.1b","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-09-16T15:46:58.563604+0000","last_change":"2024-09-16T15:46:58.563604+0000","last_active":"2024-09-16T15:46:58.541934+0000","last_peered":"2024-09-16T15:46:58.541934+0000","last_clean":"2024-09-16T15:46:58.541934+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:46:58.563604+0000","last_undegraded":"2024-09-16T15:46:58.563604+0000","last_fullsized":"2024-09-16T15:46:58.563604+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_clean_scrub_stamp":"2024-09-16T15:46:58.541934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.1a","version":"0'0","reported_seq":10,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-16T15:46:58.604119+0000","last_change":"2024-09-16T15:46:58.600655+0000","last_active":"2024-09-16T15:46:58.604119+0000","last_peered":"2024-09-16T15:46:58.604119+0000","last_clean":"2024-09-16T15:46:58.604119+0000","last_became_active":"2024-09-16T15:46:58.599216+0000","last_became_peered":"2024-09-16T15:46:58.599216+0000","last_unstale":"2024-09-16T15:46:58.604119+0000","last_undegraded":"2024-09-16T15:46:58.604119+0000","last_fullsized":"2024-09-16T15:46:58.604119+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:57.531046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:57.531046+0000","last_clean_scrub_stamp":"2024-09-16T15:46:57.531046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:13:18.615772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.18","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-09-16T15:46:58.562911+0000","last_change":"2024-09-16T15:46:58.562911+0000","last_active":"2024-09-16T15:46:58.541934+0000","last_peered":"2024-09-16T15:46:58.541934+0000","last_clean":"2024-09-16T15:46:58.541934+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:46:58.562911+0000","last_undegraded":"2024-09-16T15:46:58.562911+0000","last_fullsized":"2024-09-16T15:46:58.562911+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_clean_scrub_stamp":"2024-09-16T15:46:58.541934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.19","version":"0'0","reported_seq":10,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-16T15:46:58.589251+0000","last_change":"2024-09-16T15:46:58.584364+0000","last_active":"2024-09-16T15:46:58.589251+0000","last_peered":"2024-09-16T15:46:58.589251+0000","last_clean":"2024-09-16T15:46:58.589251+0000","last_became_active":"2024-09-16T15:46:58.583975+0000","last_became_peered":"2024-09-16T15:46:58.583975+0000","last_unstale":"2024-09-16T15:46:58.589251+0000","last_undegraded":"2024-09-16T15:46:58.589251+0000","last_fullsized":"2024-09-16T15:46:58.589251+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:57.531046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:57.531046+0000","last_clean_scrub_stamp":"2024-09-16T15:46:57.531046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T00:25:29.249313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.19","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-09-16T15:46:58.566064+0000","last_change":"2024-09-16T15:46:58.566064+0000","last_active":"2024-09-16T15:46:58.541934+0000","last_peered":"2024-09-16T15:46:58.541934+0000","last_clean":"2024-09-16T15:46:58.541934+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:46:58.566064+0000","last_undegraded":"2024-09-16T15:46:58.566064+0000","last_fullsized":"2024-09-16T15:46:58.566064+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_clean_scrub_stamp":"2024-09-16T15:46:58.541934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.18","version":"0'0","reported_seq":10,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-16T15:46:58.574992+0000","last_change":"2024-09-16T15:46:58.570577+0000","last_active":"2024-09-16T15:46:58.574992+0000","last_peered":"2024-09-16T15:46:58.574992+0000","last_clean":"2024-09-16T15:46:58.574992+0000","last_became_active":"2024-09-16T15:46:58.570223+0000","last_became_peered":"2024-09-16T15:46:58.570223+0000","last_unstale":"2024-09-16T15:46:58.574992+0000","last_undegraded":"2024-09-16T15:46:58.574992+0000","last_fullsized":"2024-09-16T15:46:58.574992+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:57.531046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:57.531046+0000","last_clean_scrub_stamp":"2024-09-16T15:46:57.531046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:11:59.055640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.16","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-09-16T15:46:58.570594+0000","last_change":"2024-09-16T15:46:58.570594+0000","last_active":"2024-09-16T15:46:58.541934+0000","last_peered":"2024-09-16T15:46:58.541934+0000","last_clean":"2024-09-16T15:46:58.541934+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:46:58.570594+0000","last_undegraded":"2024-09-16T15:46:58.570594+0000","last_fullsized":"2024-09-16T15:46:58.570594+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_clean_scrub_stamp":"2024-09-16T15:46:58.541934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.17","version":"0'0","reported_seq":10,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-16T15:46:58.590618+0000","last_change":"2024-09-16T15:46:58.576463+0000","last_active":"2024-09-16T15:46:58.590618+0000","last_peered":"2024-09-16T15:46:58.590618+0000","last_clean":"2024-09-16T15:46:58.590618+0000","last_became_active":"2024-09-16T15:46:58.573574+0000","last_became_peered":"2024-09-16T15:46:58.573574+0000","last_unstale":"2024-09-16T15:46:58.590618+0000","last_undegraded":"2024-09-16T15:46:58.590618+0000","last_fullsized":"2024-09-16T15:46:58.590618+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:57.531046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:57.531046+0000","last_clean_scrub_stamp":"2024-09-16T15:46:57.531046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T01:24:15.136386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.17","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-09-16T15:46:58.579574+0000","last_change":"2024-09-16T15:46:58.579574+0000","last_active":"2024-09-16T15:46:58.541934+0000","last_peered":"2024-09-16T15:46:58.541934+0000","last_clean":"2024-09-16T15:46:58.541934+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:46:58.579574+0000","last_undegraded":"2024-09-16T15:46:58.579574+0000","last_fullsized":"2024-09-16T15:46:58.579574+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_clean_scrub_stamp":"2024-09-16T15:46:58.541934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.16","version":"0'0","reported_seq":10,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-16T15:46:58.602617+0000","last_change":"2024-09-16T15:46:58.597162+0000","last_active":"2024-09-16T15:46:58.602617+0000","last_peered":"2024-09-16T15:46:58.602617+0000","last_clean":"2024-09-16T15:46:58.602617+0000","last_became_active":"2024-09-16T15:46:58.596469+0000","last_became_peered":"2024-09-16T15:46:58.596469+0000","last_unstale":"2024-09-16T15:46:58.602617+0000","last_undegraded":"2024-09-16T15:46:58.602617+0000","last_fullsized":"2024-09-16T15:46:58.602617+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:57.531046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:57.531046+0000","last_clean_scrub_stamp":"2024-09-16T15:46:57.531046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:22:59.268994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.14","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-09-16T15:46:58.582525+0000","last_change":"2024-09-16T15:46:58.582525+0000","last_active":"2024-09-16T15:46:58.541934+0000","last_peered":"2024-09-16T15:46:58.541934+0000","last_clean":"2024-09-16T15:46:58.541934+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:46:58.582525+0000","last_undegraded":"2024-09-16T15:46:58.582525+0000","last_fullsized":"2024-09-16T15:46:58.582525+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_clean_scrub_stamp":"2024-09-16T15:46:58.541934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.15","version":"0'0","reported_seq":10,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-16T15:46:58.588947+0000","last_change":"2024-09-16T15:46:58.583907+0000","last_active":"2024-09-16T15:46:58.588947+0000","last_peered":"2024-09-16T15:46:58.588947+0000","last_clean":"2024-09-16T15:46:58.588947+0000","last_became_active":"2024-09-16T15:46:58.583530+0000","last_became_peered":"2024-09-16T15:46:58.583530+0000","last_unstale":"2024-09-16T15:46:58.588947+0000","last_undegraded":"2024-09-16T15:46:58.588947+0000","last_fullsized":"2024-09-16T15:46:58.588947+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:57.531046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:57.531046+0000","last_clean_scrub_stamp":"2024-09-16T15:46:57.531046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T01:06:36.276569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.15","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-09-16T15:46:58.563631+0000","last_change":"2024-09-16T15:46:58.563631+0000","last_active":"2024-09-16T15:46:58.541934+0000","last_peered":"2024-09-16T15:46:58.541934+0000","last_clean":"2024-09-16T15:46:58.541934+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:46:58.563631+0000","last_undegraded":"2024-09-16T15:46:58.563631+0000","last_fullsized":"2024-09-16T15:46:58.563631+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_clean_scrub_stamp":"2024-09-16T15:46:58.541934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.14","version":"0'0","reported_seq":10,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-16T15:46:58.575157+0000","last_change":"2024-09-16T15:46:58.570712+0000","last_active":"2024-09-16T15:46:58.575157+0000","last_peered":"2024-09-16T15:46:58.575157+0000","last_clean":"2024-09-16T15:46:58.575157+0000","last_became_active":"2024-09-16T15:46:58.570432+0000","last_became_peered":"2024-09-16T15:46:58.570432+0000","last_unstale":"2024-09-16T15:46:58.575157+0000","last_undegraded":"2024-09-16T15:46:58.575157+0000","last_fullsized":"2024-09-16T15:46:58.575157+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:57.531046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:57.531046+0000","last_clean_scrub_stamp":"2024-09-16T15:46:57.531046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:58:27.343595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.12","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-09-16T15:46:58.574297+0000","last_change":"2024-09-16T15:46:58.574297+0000","last_active":"2024-09-16T15:46:58.541934+0000","last_peered":"2024-09-16T15:46:58.541934+0000","last_clean":"2024-09-16T15:46:58.541934+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:46:58.574297+0000","last_undegraded":"2024-09-16T15:46:58.574297+0000","last_fullsized":"2024-09-16T15:46:58.574297+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_clean_scrub_stamp":"2024-09-16T15:46:58.541934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.13","version":"0'0","reported_seq":10,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-16T15:46:58.603021+0000","last_change":"2024-09-16T15:46:58.596492+0000","last_active":"2024-09-16T15:46:58.603021+0000","last_peered":"2024-09-16T15:46:58.603021+0000","last_clean":"2024-09-16T15:46:58.603021+0000","last_became_active":"2024-09-16T15:46:58.596172+0000","last_became_peered":"2024-09-16T15:46:58.596172+0000","last_unstale":"2024-09-16T15:46:58.603021+0000","last_undegraded":"2024-09-16T15:46:58.603021+0000","last_fullsized":"2024-09-16T15:46:58.603021+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:57.531046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:57.531046+0000","last_clean_scrub_stamp":"2024-09-16T15:46:57.531046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T02:09:19.272595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.13","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-09-16T15:46:58.569492+0000","last_change":"2024-09-16T15:46:58.569492+0000","last_active":"2024-09-16T15:46:58.541934+0000","last_peered":"2024-09-16T15:46:58.541934+0000","last_clean":"2024-09-16T15:46:58.541934+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:46:58.569492+0000","last_undegraded":"2024-09-16T15:46:58.569492+0000","last_fullsized":"2024-09-16T15:46:58.569492+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_clean_scrub_stamp":"2024-09-16T15:46:58.541934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.12","version":"0'0","reported_seq":10,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-16T15:46:58.602887+0000","last_change":"2024-09-16T15:46:58.595177+0000","last_active":"2024-09-16T15:46:58.602887+0000","last_peered":"2024-09-16T15:46:58.602887+0000","last_clean":"2024-09-16T15:46:58.602887+0000","last_became_active":"2024-09-16T15:46:58.593818+0000","last_became_peered":"2024-09-16T15:46:58.593818+0000","last_unstale":"2024-09-16T15:46:58.602887+0000","last_undegraded":"2024-09-16T15:46:58.602887+0000","last_fullsized":"2024-09-16T15:46:58.602887+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:57.531046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:57.531046+0000","last_clean_scrub_stamp":"2024-09-16T15:46:57.531046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T01:59:03.824917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.10","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-09-16T15:46:58.562563+0000","last_change":"2024-09-16T15:46:58.562563+0000","last_active":"2024-09-16T15:46:58.541934+0000","last_peered":"2024-09-16T15:46:58.541934+0000","last_clean":"2024-09-16T15:46:58.541934+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:46:58.562563+0000","last_undegraded":"2024-09-16T15:46:58.562563+0000","last_fullsized":"2024-09-16T15:46:58.562563+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_clean_scrub_stamp":"2024-09-16T15:46:58.541934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.11","version":"0'0","reported_seq":10,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-16T15:46:58.595218+0000","last_change":"2024-09-16T15:46:58.584495+0000","last_active":"2024-09-16T15:46:58.595218+0000","last_peered":"2024-09-16T15:46:58.595218+0000","last_clean":"2024-09-16T15:46:58.595218+0000","last_became_active":"2024-09-16T15:46:58.583741+0000","last_became_peered":"2024-09-16T15:46:58.583741+0000","last_unstale":"2024-09-16T15:46:58.595218+0000","last_undegraded":"2024-09-16T15:46:58.595218+0000","last_fullsized":"2024-09-16T15:46:58.595218+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:57.531046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:57.531046+0000","last_clean_scrub_stamp":"2024-09-16T15:46:57.531046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T01:58:08.477199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.11","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-09-16T15:46:58.560818+0000","last_change":"2024-09-16T15:46:58.560818+0000","last_active":"2024-09-16T15:46:58.541934+0000","last_peered":"2024-09-16T15:46:58.541934+0000","last_clean":"2024-09-16T15:46:58.541934+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:46:58.560818+0000","last_undegraded":"2024-09-16T15:46:58.560818+0000","last_fullsized":"2024-09-16T15:46:58.560818+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_clean_scrub_stamp":"2024-09-16T15:46:58.541934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.10","version":"0'0","reported_seq":10,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-16T15:46:58.604683+0000","last_change":"2024-09-16T15:46:58.597568+0000","last_active":"2024-09-16T15:46:58.604683+0000","last_peered":"2024-09-16T15:46:58.604683+0000","last_clean":"2024-09-16T15:46:58.604683+0000","last_became_active":"2024-09-16T15:46:58.596806+0000","last_became_peered":"2024-09-16T15:46:58.596806+0000","last_unstale":"2024-09-16T15:46:58.604683+0000","last_undegraded":"2024-09-16T15:46:58.604683+0000","last_fullsized":"2024-09-16T15:46:58.604683+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:57.531046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:57.531046+0000","last_clean_scrub_stamp":"2024-09-16T15:46:57.531046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:33:54.157169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.e","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-09-16T15:46:58.561429+0000","last_change":"2024-09-16T15:46:58.561429+0000","last_active":"2024-09-16T15:46:58.541934+0000","last_peered":"2024-09-16T15:46:58.541934+0000","last_clean":"2024-09-16T15:46:58.541934+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:46:58.561429+0000","last_undegraded":"2024-09-16T15:46:58.561429+0000","last_fullsized":"2024-09-16T15:46:58.561429+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_clean_scrub_stamp":"2024-09-16T15:46:58.541934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.f","version":"0'0","reported_seq":10,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-16T15:46:58.602564+0000","last_change":"2024-09-16T15:46:58.595767+0000","last_active":"2024-09-16T15:46:58.602564+0000","last_peered":"2024-09-16T15:46:58.602564+0000","last_clean":"2024-09-16T15:46:58.602564+0000","last_became_active":"2024-09-16T15:46:58.594655+0000","last_became_peered":"2024-09-16T15:46:58.594655+0000","last_unstale":"2024-09-16T15:46:58.602564+0000","last_undegraded":"2024-09-16T15:46:58.602564+0000","last_fullsized":"2024-09-16T15:46:58.602564+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:57.531046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:57.531046+0000","last_clean_scrub_stamp":"2024-09-16T15:46:57.531046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:56:28.530413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.f","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-09-16T15:46:58.578428+0000","last_change":"2024-09-16T15:46:58.578428+0000","last_active":"2024-09-16T15:46:58.541934+0000","last_peered":"2024-09-16T15:46:58.541934+0000","last_clean":"2024-09-16T15:46:58.541934+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:46:58.578428+0000","last_undegraded":"2024-09-16T15:46:58.578428+0000","last_fullsized":"2024-09-16T15:46:58.578428+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_clean_scrub_stamp":"2024-09-16T15:46:58.541934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.e","version":"0'0","reported_seq":10,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-16T15:46:58.604653+0000","last_change":"2024-09-16T15:46:58.597152+0000","last_active":"2024-09-16T15:46:58.604653+0000","last_peered":"2024-09-16T15:46:58.604653+0000","last_clean":"2024-09-16T15:46:58.604653+0000","last_became_active":"2024-09-16T15:46:58.596435+0000","last_became_peered":"2024-09-16T15:46:58.596435+0000","last_unstale":"2024-09-16T15:46:58.604653+0000","last_undegraded":"2024-09-16T15:46:58.604653+0000","last_fullsized":"2024-09-16T15:46:58.604653+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:57.531046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:57.531046+0000","last_clean_scrub_stamp":"2024-09-16T15:46:57.531046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:11:50.475616+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.c","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-09-16T15:46:58.559398+0000","last_change":"2024-09-16T15:46:58.559398+0000","last_active":"2024-09-16T15:46:58.541934+0000","last_peered":"2024-09-16T15:46:58.541934+0000","last_clean":"2024-09-16T15:46:58.541934+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:46:58.559398+0000","last_undegraded":"2024-09-16T15:46:58.559398+0000","last_fullsized":"2024-09-16T15:46:58.559398+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_clean_scrub_stamp":"2024-09-16T15:46:58.541934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.d","version":"0'0","reported_seq":10,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-16T15:46:58.602764+0000","last_change":"2024-09-16T15:46:58.598349+0000","last_active":"2024-09-16T15:46:58.602764+0000","last_peered":"2024-09-16T15:46:58.602764+0000","last_clean":"2024-09-16T15:46:58.602764+0000","last_became_active":"2024-09-16T15:46:58.595349+0000","last_became_peered":"2024-09-16T15:46:58.595349+0000","last_unstale":"2024-09-16T15:46:58.602764+0000","last_undegraded":"2024-09-16T15:46:58.602764+0000","last_fullsized":"2024-09-16T15:46:58.602764+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:57.531046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:57.531046+0000","last_clean_scrub_stamp":"2024-09-16T15:46:57.531046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T01:43:29.245355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.d","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-09-16T15:46:58.563246+0000","last_change":"2024-09-16T15:46:58.563246+0000","last_active":"2024-09-16T15:46:58.541934+0000","last_peered":"2024-09-16T15:46:58.541934+0000","last_clean":"2024-09-16T15:46:58.541934+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:46:58.563246+0000","last_undegraded":"2024-09-16T15:46:58.563246+0000","last_fullsized":"2024-09-16T15:46:58.563246+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_clean_scrub_stamp":"2024-09-16T15:46:58.541934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.c","version":"0'0","reported_seq":10,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-16T15:46:58.599912+0000","last_change":"2024-09-16T15:46:58.596863+0000","last_active":"2024-09-16T15:46:58.599912+0000","last_peered":"2024-09-16T15:46:58.599912+0000","last_clean":"2024-09-16T15:46:58.599912+0000","last_became_active":"2024-09-16T15:46:58.596143+0000","last_became_peered":"2024-09-16T15:46:58.596143+0000","last_unstale":"2024-09-16T15:46:58.599912+0000","last_undegraded":"2024-09-16T15:46:58.599912+0000","last_fullsized":"2024-09-16T15:46:58.599912+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:57.531046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:57.531046+0000","last_clean_scrub_stamp":"2024-09-16T15:46:57.531046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:50:55.009663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.b","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-09-16T15:46:58.567734+0000","last_change":"2024-09-16T15:46:58.567734+0000","last_active":"2024-09-16T15:46:58.541934+0000","last_peered":"2024-09-16T15:46:58.541934+0000","last_clean":"2024-09-16T15:46:58.541934+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:46:58.567734+0000","last_undegraded":"2024-09-16T15:46:58.567734+0000","last_fullsized":"2024-09-16T15:46:58.567734+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_clean_scrub_stamp":"2024-09-16T15:46:58.541934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.a","version":"0'0","reported_seq":10,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-16T15:46:58.597818+0000","last_change":"2024-09-16T15:46:58.585877+0000","last_active":"2024-09-16T15:46:58.597818+0000","last_peered":"2024-09-16T15:46:58.597818+0000","last_clean":"2024-09-16T15:46:58.597818+0000","last_became_active":"2024-09-16T15:46:58.585158+0000","last_became_peered":"2024-09-16T15:46:58.585158+0000","last_unstale":"2024-09-16T15:46:58.597818+0000","last_undegraded":"2024-09-16T15:46:58.597818+0000","last_fullsized":"2024-09-16T15:46:58.597818+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:57.531046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:57.531046+0000","last_clean_scrub_stamp":"2024-09-16T15:46:57.531046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T01:00:40.058766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.8","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-09-16T15:46:58.564210+0000","last_change":"2024-09-16T15:46:58.564210+0000","last_active":"2024-09-16T15:46:58.541934+0000","last_peered":"2024-09-16T15:46:58.541934+0000","last_clean":"2024-09-16T15:46:58.541934+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:46:58.564210+0000","last_undegraded":"2024-09-16T15:46:58.564210+0000","last_fullsized":"2024-09-16T15:46:58.564210+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_clean_scrub_stamp":"2024-09-16T15:46:58.541934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.9","version":"0'0","reported_seq":10,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-16T15:46:58.583297+0000","last_change":"2024-09-16T15:46:58.576781+0000","last_active":"2024-09-16T15:46:58.583297+0000","last_peered":"2024-09-16T15:46:58.583297+0000","last_clean":"2024-09-16T15:46:58.583297+0000","last_became_active":"2024-09-16T15:46:58.576081+0000","last_became_peered":"2024-09-16T15:46:58.576081+0000","last_unstale":"2024-09-16T15:46:58.583297+0000","last_undegraded":"2024-09-16T15:46:58.583297+0000","last_fullsized":"2024-09-16T15:46:58.583297+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:57.531046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:57.531046+0000","last_clean_scrub_stamp":"2024-09-16T15:46:57.531046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T22:41:00.960336+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.9","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-09-16T15:46:58.564335+0000","last_change":"2024-09-16T15:46:58.564335+0000","last_active":"2024-09-16T15:46:58.541934+0000","last_peered":"2024-09-16T15:46:58.541934+0000","last_clean":"2024-09-16T15:46:58.541934+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:46:58.564335+0000","last_undegraded":"2024-09-16T15:46:58.564335+0000","last_fullsized":"2024-09-16T15:46:58.564335+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_clean_scrub_stamp":"2024-09-16T15:46:58.541934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.8","version":"0'0","reported_seq":10,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-16T15:46:58.577818+0000","last_change":"2024-09-16T15:46:58.572736+0000","last_active":"2024-09-16T15:46:58.577818+0000","last_peered":"2024-09-16T15:46:58.577818+0000","last_clean":"2024-09-16T15:46:58.577818+0000","last_became_active":"2024-09-16T15:46:58.572311+0000","last_became_peered":"2024-09-16T15:46:58.572311+0000","last_unstale":"2024-09-16T15:46:58.577818+0000","last_undegraded":"2024-09-16T15:46:58.577818+0000","last_fullsized":"2024-09-16T15:46:58.577818+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:57.531046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:57.531046+0000","last_clean_scrub_stamp":"2024-09-16T15:46:57.531046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:48:10.471983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.6","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-09-16T15:46:58.562758+0000","last_change":"2024-09-16T15:46:58.562758+0000","last_active":"2024-09-16T15:46:58.541934+0000","last_peered":"2024-09-16T15:46:58.541934+0000","last_clean":"2024-09-16T15:46:58.541934+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:46:58.562758+0000","last_undegraded":"2024-09-16T15:46:58.562758+0000","last_fullsized":"2024-09-16T15:46:58.562758+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_clean_scrub_stamp":"2024-09-16T15:46:58.541934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.7","version":"0'0","reported_seq":10,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-16T15:46:58.594095+0000","last_change":"2024-09-16T15:46:58.579355+0000","last_active":"2024-09-16T15:46:58.594095+0000","last_peered":"2024-09-16T15:46:58.594095+0000","last_clean":"2024-09-16T15:46:58.594095+0000","last_became_active":"2024-09-16T15:46:58.578380+0000","last_became_peered":"2024-09-16T15:46:58.578380+0000","last_unstale":"2024-09-16T15:46:58.594095+0000","last_undegraded":"2024-09-16T15:46:58.594095+0000","last_fullsized":"2024-09-16T15:46:58.594095+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:57.531046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:57.531046+0000","last_clean_scrub_stamp":"2024-09-16T15:46:57.531046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T23:40:10.473815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.7","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-09-16T15:46:58.569698+0000","last_change":"2024-09-16T15:46:58.569698+0000","last_active":"2024-09-16T15:46:58.541934+0000","last_peered":"2024-09-16T15:46:58.541934+0000","last_clean":"2024-09-16T15:46:58.541934+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:46:58.569698+0000","last_undegraded":"2024-09-16T15:46:58.569698+0000","last_fullsized":"2024-09-16T15:46:58.569698+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_clean_scrub_stamp":"2024-09-16T15:46:58.541934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.6","version":"0'0","reported_seq":10,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-16T15:46:58.582810+0000","last_change":"2024-09-16T15:46:58.577011+0000","last_active":"2024-09-16T15:46:58.582810+0000","last_peered":"2024-09-16T15:46:58.582810+0000","last_clean":"2024-09-16T15:46:58.582810+0000","last_became_active":"2024-09-16T15:46:58.576294+0000","last_became_peered":"2024-09-16T15:46:58.576294+0000","last_unstale":"2024-09-16T15:46:58.582810+0000","last_undegraded":"2024-09-16T15:46:58.582810+0000","last_fullsized":"2024-09-16T15:46:58.582810+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:57.531046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:57.531046+0000","last_clean_scrub_stamp":"2024-09-16T15:46:57.531046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T03:31:45.091717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.4","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-09-16T15:46:58.574408+0000","last_change":"2024-09-16T15:46:58.574408+0000","last_active":"2024-09-16T15:46:58.541934+0000","last_peered":"2024-09-16T15:46:58.541934+0000","last_clean":"2024-09-16T15:46:58.541934+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:46:58.574408+0000","last_undegraded":"2024-09-16T15:46:58.574408+0000","last_fullsized":"2024-09-16T15:46:58.574408+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_clean_scrub_stamp":"2024-09-16T15:46:58.541934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.5","version":"0'0","reported_seq":10,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-16T15:46:58.578312+0000","last_change":"2024-09-16T15:46:58.575511+0000","last_active":"2024-09-16T15:46:58.578312+0000","last_peered":"2024-09-16T15:46:58.578312+0000","last_clean":"2024-09-16T15:46:58.578312+0000","last_became_active":"2024-09-16T15:46:58.575218+0000","last_became_peered":"2024-09-16T15:46:58.575218+0000","last_unstale":"2024-09-16T15:46:58.578312+0000","last_undegraded":"2024-09-16T15:46:58.578312+0000","last_fullsized":"2024-09-16T15:46:58.578312+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:57.531046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:57.531046+0000","last_clean_scrub_stamp":"2024-09-16T15:46:57.531046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:37:03.438301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.5","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-09-16T15:46:58.578126+0000","last_change":"2024-09-16T15:46:58.578126+0000","last_active":"2024-09-16T15:46:58.541934+0000","last_peered":"2024-09-16T15:46:58.541934+0000","last_clean":"2024-09-16T15:46:58.541934+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:46:58.578126+0000","last_undegraded":"2024-09-16T15:46:58.578126+0000","last_fullsized":"2024-09-16T15:46:58.578126+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_clean_scrub_stamp":"2024-09-16T15:46:58.541934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.4","version":"0'0","reported_seq":10,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-16T15:46:58.588834+0000","last_change":"2024-09-16T15:46:58.583703+0000","last_active":"2024-09-16T15:46:58.588834+0000","last_peered":"2024-09-16T15:46:58.588834+0000","last_clean":"2024-09-16T15:46:58.588834+0000","last_became_active":"2024-09-16T15:46:58.583384+0000","last_became_peered":"2024-09-16T15:46:58.583384+0000","last_unstale":"2024-09-16T15:46:58.588834+0000","last_undegraded":"2024-09-16T15:46:58.588834+0000","last_fullsized":"2024-09-16T15:46:58.588834+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:57.531046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:57.531046+0000","last_clean_scrub_stamp":"2024-09-16T15:46:57.531046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T23:06:05.405748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.2","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-09-16T15:46:58.560788+0000","last_change":"2024-09-16T15:46:58.560788+0000","last_active":"2024-09-16T15:46:58.541934+0000","last_peered":"2024-09-16T15:46:58.541934+0000","last_clean":"2024-09-16T15:46:58.541934+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:46:58.560788+0000","last_undegraded":"2024-09-16T15:46:58.560788+0000","last_fullsized":"2024-09-16T15:46:58.560788+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_clean_scrub_stamp":"2024-09-16T15:46:58.541934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.3","version":"0'0","reported_seq":10,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-16T15:46:58.577602+0000","last_change":"2024-09-16T15:46:58.573698+0000","last_active":"2024-09-16T15:46:58.577602+0000","last_peered":"2024-09-16T15:46:58.577602+0000","last_clean":"2024-09-16T15:46:58.577602+0000","last_became_active":"2024-09-16T15:46:58.573391+0000","last_became_peered":"2024-09-16T15:46:58.573391+0000","last_unstale":"2024-09-16T15:46:58.577602+0000","last_undegraded":"2024-09-16T15:46:58.577602+0000","last_fullsized":"2024-09-16T15:46:58.577602+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:57.531046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:57.531046+0000","last_clean_scrub_stamp":"2024-09-16T15:46:57.531046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T00:58:45.221067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.3","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-09-16T15:46:58.586430+0000","last_change":"2024-09-16T15:46:58.586430+0000","last_active":"2024-09-16T15:46:58.541934+0000","last_peered":"2024-09-16T15:46:58.541934+0000","last_clean":"2024-09-16T15:46:58.541934+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:46:58.586430+0000","last_undegraded":"2024-09-16T15:46:58.586430+0000","last_fullsized":"2024-09-16T15:46:58.586430+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_clean_scrub_stamp":"2024-09-16T15:46:58.541934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.2","version":"0'0","reported_seq":10,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-16T15:46:58.577483+0000","last_change":"2024-09-16T15:46:58.574227+0000","last_active":"2024-09-16T15:46:58.577483+0000","last_peered":"2024-09-16T15:46:58.577483+0000","last_clean":"2024-09-16T15:46:58.577483+0000","last_became_active":"2024-09-16T15:46:58.573898+0000","last_became_peered":"2024-09-16T15:46:58.573898+0000","last_unstale":"2024-09-16T15:46:58.577483+0000","last_undegraded":"2024-09-16T15:46:58.577483+0000","last_fullsized":"2024-09-16T15:46:58.577483+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:57.531046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:57.531046+0000","last_clean_scrub_stamp":"2024-09-16T15:46:57.531046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T18:41:18.272460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.0","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-09-16T15:46:58.559237+0000","last_change":"2024-09-16T15:46:58.559237+0000","last_active":"2024-09-16T15:46:58.541934+0000","last_peered":"2024-09-16T15:46:58.541934+0000","last_clean":"2024-09-16T15:46:58.541934+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:46:58.559237+0000","last_undegraded":"2024-09-16T15:46:58.559237+0000","last_fullsized":"2024-09-16T15:46:58.559237+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_clean_scrub_stamp":"2024-09-16T15:46:58.541934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.1","version":"0'0","reported_seq":10,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-16T15:46:58.604741+0000","last_change":"2024-09-16T15:46:58.598614+0000","last_active":"2024-09-16T15:46:58.604741+0000","last_peered":"2024-09-16T15:46:58.604741+0000","last_clean":"2024-09-16T15:46:58.604741+0000","last_became_active":"2024-09-16T15:46:58.597950+0000","last_became_peered":"2024-09-16T15:46:58.597950+0000","last_unstale":"2024-09-16T15:46:58.604741+0000","last_undegraded":"2024-09-16T15:46:58.604741+0000","last_fullsized":"2024-09-16T15:46:58.604741+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:57.531046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:57.531046+0000","last_clean_scrub_stamp":"2024-09-16T15:46:57.531046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T23:50:07.979345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-09-16T15:46:58.565520+0000","last_change":"2024-09-16T15:46:58.565520+0000","last_active":"2024-09-16T15:46:58.541934+0000","last_peered":"2024-09-16T15:46:58.541934+0000","last_clean":"2024-09-16T15:46:58.541934+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:46:58.565520+0000","last_undegraded":"2024-09-16T15:46:58.565520+0000","last_fullsized":"2024-09-16T15:46:58.565520+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_clean_scrub_stamp":"2024-09-16T15:46:58.541934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.0","version":"0'0","reported_seq":10,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-16T15:46:58.602740+0000","last_change":"2024-09-16T15:46:58.599504+0000","last_active":"2024-09-16T15:46:58.602740+0000","last_peered":"2024-09-16T15:46:58.602740+0000","last_clean":"2024-09-16T15:46:58.602740+0000","last_became_active":"2024-09-16T15:46:58.598743+0000","last_became_peered":"2024-09-16T15:46:58.598743+0000","last_unstale":"2024-09-16T15:46:58.602740+0000","last_undegraded":"2024-09-16T15:46:58.602740+0000","last_fullsized":"2024-09-16T15:46:58.602740+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:57.531046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:57.531046+0000","last_clean_scrub_stamp":"2024-09-16T15:46:57.531046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T18:45:22.195277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.a","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-09-16T15:46:58.564305+0000","last_change":"2024-09-16T15:46:58.564305+0000","last_active":"2024-09-16T15:46:58.541934+0000","last_peered":"2024-09-16T15:46:58.541934+0000","last_clean":"2024-09-16T15:46:58.541934+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:46:58.564305+0000","last_undegraded":"2024-09-16T15:46:58.564305+0000","last_fullsized":"2024-09-16T15:46:58.564305+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_clean_scrub_stamp":"2024-09-16T15:46:58.541934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.b","version":"0'0","reported_seq":10,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-16T15:46:58.604289+0000","last_change":"2024-09-16T15:46:58.601513+0000","last_active":"2024-09-16T15:46:58.604289+0000","last_peered":"2024-09-16T15:46:58.604289+0000","last_clean":"2024-09-16T15:46:58.604289+0000","last_became_active":"2024-09-16T15:46:58.600581+0000","last_became_peered":"2024-09-16T15:46:58.600581+0000","last_unstale":"2024-09-16T15:46:58.604289+0000","last_undegraded":"2024-09-16T15:46:58.604289+0000","last_fullsized":"2024-09-16T15:46:58.604289+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:57.531046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:57.531046+0000","last_clean_scrub_stamp":"2024-09-16T15:46:57.531046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T23:00:41.658258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"16'72","reported_seq":200,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-16T15:46:58.585260+0000","last_change":"2024-09-16T15:42:40.881820+0000","last_active":"2024-09-16T15:46:58.585260+0000","last_peered":"2024-09-16T15:46:58.585260+0000","last_clean":"2024-09-16T15:46:58.585260+0000","last_became_active":"2024-09-16T15:42:40.879965+0000","last_became_peered":"2024-09-16T15:42:40.879965+0000","last_unstale":"2024-09-16T15:46:58.585260+0000","last_undegraded":"2024-09-16T15:46:58.585260+0000","last_fullsized":"2024-09-16T15:46:58.585260+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:42:39.728762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:42:39.728762+0000","last_clean_scrub_stamp":"2024-09-16T15:42:39.728762+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:22:18.768932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":11,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":10,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542200,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28416,"kb_used_data":1384,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343424,"statfs":{"total":96636764160,"available":96607666176,"internally_reserved":0,"allocated":1417216,"data_stored":932261,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542199,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31996,"kb_used_data":804,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339844,"statfs":{"total":96636764160,"available":96604000256,"internally_reserved":0,"allocated":823296,"data_stored":341893,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":5,"up_from":14,"seq":60129542200,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27824,"kb_used_data":792,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344016,"statfs":{"total":96636764160,"available":96608272384,"internally_reserved":0,"allocated":811008,"data_stored":335618,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542198,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31980,"kb_used_data":792,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339860,"statfs":{"total":96636764160,"available":96604016640,"internally_reserved":0,"allocated":811008,"data_stored":335618,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":3,"up_from":14,"seq":60129542199,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31996,"kb_used_data":804,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339844,"statfs":{"total":96636764160,"available":96604000256,"internally_reserved":0,"allocated":823296,"data_stored":341893,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":14,"seq":60129542199,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27820,"kb_used_data":792,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344020,"statfs":{"total":96636764160,"available":96608276480,"internally_reserved":0,"allocated":811008,"data_stored":335618,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":14,"seq":60129542201,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27840,"kb_used_data":804,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344000,"statfs":{"total":96636764160,"available":96608256000,"internally_reserved":0,"allocated":823296,"data_stored":341893,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":14,"seq":60129542201,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32572,"kb_used_data":1384,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339268,"statfs":{"total":96636764160,"available":96603410432,"internally_reserved":0,"allocated":1417216,"data_stored":932261,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T15:47:04.000 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T15:47:04.161 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:04.156+0000 7f9f08a32640 1 -- 172.21.15.123:0/3184287144 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f9f040cd6e0 msgr2=0x7f9f040cdb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:47:04.161 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:04.156+0000 7f9f08a32640 1 --2- 172.21.15.123:0/3184287144 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f9f040cd6e0 0x7f9f040cdb20 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f9ef0009840 tx=0x7f9ef001c790 comp rx=0 tx=0).stop 2024-09-16T15:47:04.161 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:04.156+0000 7f9f08a32640 1 -- 172.21.15.123:0/3184287144 shutdown_connections 2024-09-16T15:47:04.162 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:04.156+0000 7f9f08a32640 1 --2- 172.21.15.123:0/3184287144 >> v2:172.21.15.184:3300/0 conn(0x7f9f0415e820 0x7f9f0415ecd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:04.162 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:04.156+0000 7f9f08a32640 1 --2- 172.21.15.123:0/3184287144 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f9f040cd6e0 0x7f9f040cdb20 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:04.162 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:04.156+0000 7f9f08a32640 1 -- 172.21.15.123:0/3184287144 >> 172.21.15.123:0/3184287144 conn(0x7f9f040cfb10 msgr2=0x7f9f040cff20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:47:04.162 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:04.156+0000 7f9f08a32640 1 -- 172.21.15.123:0/3184287144 shutdown_connections 2024-09-16T15:47:04.162 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:04.156+0000 7f9f08a32640 1 -- 172.21.15.123:0/3184287144 wait complete. 2024-09-16T15:47:04.162 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:04.156+0000 7f9f08a32640 1 Processor -- start 2024-09-16T15:47:04.162 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:04.156+0000 7f9f08a32640 1 -- start start 2024-09-16T15:47:04.165 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:04.160+0000 7f9f08a32640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f9f040cd6e0 0x7f9f0414fb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:47:04.165 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:04.160+0000 7f9f08a32640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f9f0415e160 0x7f9f041f5e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:47:04.165 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:04.160+0000 7f9f08a32640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9f0415e820 0x7f9f040ca820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:47:04.165 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:04.160+0000 7f9f08a32640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f9f040cf5b0 con 0x7f9f0415e820 2024-09-16T15:47:04.165 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:04.160+0000 7f9f08a32640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f9f040cf430 con 0x7f9f0415e160 2024-09-16T15:47:04.165 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:04.160+0000 7f9f08a32640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f9f041663c0 con 0x7f9f040cd6e0 2024-09-16T15:47:04.165 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:04.160+0000 7f9f02d9d640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9f0415e820 0x7f9f040ca820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:47:04.165 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:04.160+0000 7f9f02d9d640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9f0415e820 0x7f9f040ca820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:46854/0 (socket says 172.21.15.123:46854) 2024-09-16T15:47:04.165 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:04.160+0000 7f9f02d9d640 1 -- 172.21.15.123:0/3967624105 learned_addr learned my addr 172.21.15.123:0/3967624105 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:47:04.165 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:04.160+0000 7f9f0259c640 1 --2- 172.21.15.123:0/3967624105 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f9f040cd6e0 0x7f9f0414fb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:47:04.166 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:04.160+0000 7f9f01d9b640 1 --2- 172.21.15.123:0/3967624105 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f9f0415e160 0x7f9f041f5e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:47:04.166 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:04.160+0000 7f9f02d9d640 1 -- 172.21.15.123:0/3967624105 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f9f040cd6e0 msgr2=0x7f9f0414fb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:47:04.166 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:04.160+0000 7f9f02d9d640 1 --2- 172.21.15.123:0/3967624105 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f9f040cd6e0 0x7f9f0414fb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:04.166 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:04.160+0000 7f9f02d9d640 1 -- 172.21.15.123:0/3967624105 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f9f0415e160 msgr2=0x7f9f041f5e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:47:04.166 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:04.160+0000 7f9f02d9d640 1 --2- 172.21.15.123:0/3967624105 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f9f0415e160 0x7f9f041f5e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:04.166 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:04.160+0000 7f9f02d9d640 1 -- 172.21.15.123:0/3967624105 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9f0415b650 con 0x7f9f0415e820 2024-09-16T15:47:04.166 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:04.160+0000 7f9f0259c640 1 --2- 172.21.15.123:0/3967624105 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f9f040cd6e0 0x7f9f0414fb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T15:47:04.166 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:04.160+0000 7f9f02d9d640 1 --2- 172.21.15.123:0/3967624105 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9f0415e820 0x7f9f040ca820 secure :-1 s=READY pgs=790 cs=0 l=1 rev1=1 crypto rx=0x7f9efc00ec70 tx=0x7f9efc00c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:47:04.166 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:04.160+0000 7f9efaffd640 1 -- 172.21.15.123:0/3967624105 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9efc003b80 con 0x7f9f0415e820 2024-09-16T15:47:04.167 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:04.160+0000 7f9efaffd640 1 -- 172.21.15.123:0/3967624105 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f9efc003d20 con 0x7f9f0415e820 2024-09-16T15:47:04.167 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:04.160+0000 7f9f08a32640 1 -- 172.21.15.123:0/3967624105 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9f0415a9d0 con 0x7f9f0415e820 2024-09-16T15:47:04.167 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:04.160+0000 7f9efaffd640 1 -- 172.21.15.123:0/3967624105 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9efc004280 con 0x7f9f0415e820 2024-09-16T15:47:04.167 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:04.160+0000 7f9f08a32640 1 -- 172.21.15.123:0/3967624105 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9f0415af90 con 0x7f9f0415e820 2024-09-16T15:47:04.167 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:04.160+0000 7f9f08a32640 1 -- 172.21.15.123:0/3967624105 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9f04166870 con 0x7f9f0415e820 2024-09-16T15:47:04.168 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:04.160+0000 7f9efaffd640 1 -- 172.21.15.123:0/3967624105 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 11) ==== 97386+0+0 (secure 0 0 0) 0x7f9efc004420 con 0x7f9f0415e820 2024-09-16T15:47:04.168 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:04.160+0000 7f9efaffd640 1 --2- 172.21.15.123:0/3967624105 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f9eec073220 0x7f9eec0756e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:47:04.168 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:04.160+0000 7f9efaffd640 1 -- 172.21.15.123:0/3967624105 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(70..70 src has 1..70) ==== 5709+0+0 (secure 0 0 0) 0x7f9efc095230 con 0x7f9f0415e820 2024-09-16T15:47:04.168 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:04.164+0000 7f9f0259c640 1 --2- 172.21.15.123:0/3967624105 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f9eec073220 0x7f9eec0756e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:47:04.168 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:04.164+0000 7f9f0259c640 1 --2- 172.21.15.123:0/3967624105 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f9eec073220 0x7f9eec0756e0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f9ef4007a60 tx=0x7f9ef4009040 comp rx=0 tx=0).ready entity=mgr.4135 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:47:04.170 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:04.164+0000 7f9efaffd640 1 -- 172.21.15.123:0/3967624105 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f9efc05fe60 con 0x7f9f0415e820 2024-09-16T15:47:04.326 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:04.320+0000 7f9f08a32640 1 -- 172.21.15.123:0/3967624105 --> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f9f04156840 con 0x7f9eec073220 2024-09-16T15:47:04.337 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:04.332+0000 7f9efaffd640 1 -- 172.21.15.123:0/3967624105 <== mgr.4135 v2:172.21.15.184:6832/3114346423 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+246774 (secure 0 0 0) 0x7f9f04156840 con 0x7f9eec073220 2024-09-16T15:47:04.339 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-09-16T15:47:04.341 INFO:teuthology.orchestra.run.smithi123.stderr:dumped all 2024-09-16T15:47:04.342 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:04.336+0000 7f9f08a32640 1 -- 172.21.15.123:0/3967624105 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f9eec073220 msgr2=0x7f9eec0756e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:47:04.342 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:04.336+0000 7f9f08a32640 1 --2- 172.21.15.123:0/3967624105 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f9eec073220 0x7f9eec0756e0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f9ef4007a60 tx=0x7f9ef4009040 comp rx=0 tx=0).stop 2024-09-16T15:47:04.342 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:04.336+0000 7f9f08a32640 1 -- 172.21.15.123:0/3967624105 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9f0415e820 msgr2=0x7f9f040ca820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:47:04.342 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:04.336+0000 7f9f08a32640 1 --2- 172.21.15.123:0/3967624105 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9f0415e820 0x7f9f040ca820 secure :-1 s=READY pgs=790 cs=0 l=1 rev1=1 crypto rx=0x7f9efc00ec70 tx=0x7f9efc00c6a0 comp rx=0 tx=0).stop 2024-09-16T15:47:04.343 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:04.336+0000 7f9f08a32640 1 -- 172.21.15.123:0/3967624105 shutdown_connections 2024-09-16T15:47:04.343 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:04.336+0000 7f9f08a32640 1 --2- 172.21.15.123:0/3967624105 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f9eec073220 0x7f9eec0756e0 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:04.343 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:04.336+0000 7f9f08a32640 1 --2- 172.21.15.123:0/3967624105 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9f0415e820 0x7f9f040ca820 unknown :-1 s=CLOSED pgs=790 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:04.343 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:04.336+0000 7f9f08a32640 1 --2- 172.21.15.123:0/3967624105 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f9f0415e160 0x7f9f041f5e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:04.343 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:04.336+0000 7f9f08a32640 1 --2- 172.21.15.123:0/3967624105 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f9f040cd6e0 0x7f9f0414fb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:04.343 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:04.336+0000 7f9f08a32640 1 -- 172.21.15.123:0/3967624105 >> 172.21.15.123:0/3967624105 conn(0x7f9f040cfb10 msgr2=0x7f9f040c5e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:47:04.343 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:04.336+0000 7f9f08a32640 1 -- 172.21.15.123:0/3967624105 shutdown_connections 2024-09-16T15:47:04.343 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:04.336+0000 7f9f08a32640 1 -- 172.21.15.123:0/3967624105 wait complete. 2024-09-16T15:47:04.357 INFO:teuthology.orchestra.run.smithi123.stdout:{"pg_ready":true,"pg_map":{"version":195,"stamp":"2024-09-16T15:47:03.331208+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":76,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":240444,"kb_used_data":7556,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":754734276,"statfs":{"total":773094113280,"available":772847898624,"internally_reserved":0,"allocated":7737344,"data_stored":3897055,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":238276145},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":41,"apply_latency_ms":41,"commit_latency_ns":41000000,"apply_latency_ns":41000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.788118"},"pg_stats":[{"pgid":"11.3f","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-09-16T15:46:58.580896+0000","last_change":"2024-09-16T15:46:58.580896+0000","last_active":"2024-09-16T15:46:58.541934+0000","last_peered":"2024-09-16T15:46:58.541934+0000","last_clean":"2024-09-16T15:46:58.541934+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:46:58.580896+0000","last_undegraded":"2024-09-16T15:46:58.580896+0000","last_fullsized":"2024-09-16T15:46:58.580896+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_clean_scrub_stamp":"2024-09-16T15:46:58.541934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.3e","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-09-16T15:46:58.580879+0000","last_change":"2024-09-16T15:46:58.580879+0000","last_active":"2024-09-16T15:46:58.541934+0000","last_peered":"2024-09-16T15:46:58.541934+0000","last_clean":"2024-09-16T15:46:58.541934+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:46:58.580879+0000","last_undegraded":"2024-09-16T15:46:58.580879+0000","last_fullsized":"2024-09-16T15:46:58.580879+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_clean_scrub_stamp":"2024-09-16T15:46:58.541934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.3d","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-09-16T15:46:58.565500+0000","last_change":"2024-09-16T15:46:58.565500+0000","last_active":"2024-09-16T15:46:58.541934+0000","last_peered":"2024-09-16T15:46:58.541934+0000","last_clean":"2024-09-16T15:46:58.541934+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:46:58.565500+0000","last_undegraded":"2024-09-16T15:46:58.565500+0000","last_fullsized":"2024-09-16T15:46:58.565500+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_clean_scrub_stamp":"2024-09-16T15:46:58.541934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.3c","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-09-16T15:46:58.560961+0000","last_change":"2024-09-16T15:46:58.560961+0000","last_active":"2024-09-16T15:46:58.541934+0000","last_peered":"2024-09-16T15:46:58.541934+0000","last_clean":"2024-09-16T15:46:58.541934+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:46:58.560961+0000","last_undegraded":"2024-09-16T15:46:58.560961+0000","last_fullsized":"2024-09-16T15:46:58.560961+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_clean_scrub_stamp":"2024-09-16T15:46:58.541934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.3b","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-09-16T15:46:58.561715+0000","last_change":"2024-09-16T15:46:58.561715+0000","last_active":"2024-09-16T15:46:58.541934+0000","last_peered":"2024-09-16T15:46:58.541934+0000","last_clean":"2024-09-16T15:46:58.541934+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:46:58.561715+0000","last_undegraded":"2024-09-16T15:46:58.561715+0000","last_fullsized":"2024-09-16T15:46:58.561715+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_clean_scrub_stamp":"2024-09-16T15:46:58.541934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.3a","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-09-16T15:46:58.578627+0000","last_change":"2024-09-16T15:46:58.578627+0000","last_active":"2024-09-16T15:46:58.541934+0000","last_peered":"2024-09-16T15:46:58.541934+0000","last_clean":"2024-09-16T15:46:58.541934+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:46:58.578627+0000","last_undegraded":"2024-09-16T15:46:58.578627+0000","last_fullsized":"2024-09-16T15:46:58.578627+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_clean_scrub_stamp":"2024-09-16T15:46:58.541934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.39","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-09-16T15:46:58.570892+0000","last_change":"2024-09-16T15:46:58.570892+0000","last_active":"2024-09-16T15:46:58.541934+0000","last_peered":"2024-09-16T15:46:58.541934+0000","last_clean":"2024-09-16T15:46:58.541934+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:46:58.570892+0000","last_undegraded":"2024-09-16T15:46:58.570892+0000","last_fullsized":"2024-09-16T15:46:58.570892+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_clean_scrub_stamp":"2024-09-16T15:46:58.541934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.38","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-09-16T15:46:58.584780+0000","last_change":"2024-09-16T15:46:58.584780+0000","last_active":"2024-09-16T15:46:58.541934+0000","last_peered":"2024-09-16T15:46:58.541934+0000","last_clean":"2024-09-16T15:46:58.541934+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:46:58.584780+0000","last_undegraded":"2024-09-16T15:46:58.584780+0000","last_fullsized":"2024-09-16T15:46:58.584780+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_clean_scrub_stamp":"2024-09-16T15:46:58.541934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.37","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-09-16T15:46:58.561912+0000","last_change":"2024-09-16T15:46:58.561912+0000","last_active":"2024-09-16T15:46:58.541934+0000","last_peered":"2024-09-16T15:46:58.541934+0000","last_clean":"2024-09-16T15:46:58.541934+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:46:58.561912+0000","last_undegraded":"2024-09-16T15:46:58.561912+0000","last_fullsized":"2024-09-16T15:46:58.561912+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_clean_scrub_stamp":"2024-09-16T15:46:58.541934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.36","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-09-16T15:46:58.562601+0000","last_change":"2024-09-16T15:46:58.562601+0000","last_active":"2024-09-16T15:46:58.541934+0000","last_peered":"2024-09-16T15:46:58.541934+0000","last_clean":"2024-09-16T15:46:58.541934+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:46:58.562601+0000","last_undegraded":"2024-09-16T15:46:58.562601+0000","last_fullsized":"2024-09-16T15:46:58.562601+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_clean_scrub_stamp":"2024-09-16T15:46:58.541934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.35","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-09-16T15:46:58.580321+0000","last_change":"2024-09-16T15:46:58.580321+0000","last_active":"2024-09-16T15:46:58.541934+0000","last_peered":"2024-09-16T15:46:58.541934+0000","last_clean":"2024-09-16T15:46:58.541934+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:46:58.580321+0000","last_undegraded":"2024-09-16T15:46:58.580321+0000","last_fullsized":"2024-09-16T15:46:58.580321+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_clean_scrub_stamp":"2024-09-16T15:46:58.541934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.34","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-09-16T15:46:58.560127+0000","last_change":"2024-09-16T15:46:58.560127+0000","last_active":"2024-09-16T15:46:58.541934+0000","last_peered":"2024-09-16T15:46:58.541934+0000","last_clean":"2024-09-16T15:46:58.541934+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:46:58.560127+0000","last_undegraded":"2024-09-16T15:46:58.560127+0000","last_fullsized":"2024-09-16T15:46:58.560127+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_clean_scrub_stamp":"2024-09-16T15:46:58.541934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.33","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-09-16T15:46:58.562521+0000","last_change":"2024-09-16T15:46:58.562521+0000","last_active":"2024-09-16T15:46:58.541934+0000","last_peered":"2024-09-16T15:46:58.541934+0000","last_clean":"2024-09-16T15:46:58.541934+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:46:58.562521+0000","last_undegraded":"2024-09-16T15:46:58.562521+0000","last_fullsized":"2024-09-16T15:46:58.562521+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_clean_scrub_stamp":"2024-09-16T15:46:58.541934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.32","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-09-16T15:46:58.561306+0000","last_change":"2024-09-16T15:46:58.561306+0000","last_active":"2024-09-16T15:46:58.541934+0000","last_peered":"2024-09-16T15:46:58.541934+0000","last_clean":"2024-09-16T15:46:58.541934+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:46:58.561306+0000","last_undegraded":"2024-09-16T15:46:58.561306+0000","last_fullsized":"2024-09-16T15:46:58.561306+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_clean_scrub_stamp":"2024-09-16T15:46:58.541934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.31","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-09-16T15:46:58.566448+0000","last_change":"2024-09-16T15:46:58.566448+0000","last_active":"2024-09-16T15:46:58.541934+0000","last_peered":"2024-09-16T15:46:58.541934+0000","last_clean":"2024-09-16T15:46:58.541934+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:46:58.566448+0000","last_undegraded":"2024-09-16T15:46:58.566448+0000","last_fullsized":"2024-09-16T15:46:58.566448+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_clean_scrub_stamp":"2024-09-16T15:46:58.541934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.30","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-09-16T15:46:58.560118+0000","last_change":"2024-09-16T15:46:58.560118+0000","last_active":"2024-09-16T15:46:58.541934+0000","last_peered":"2024-09-16T15:46:58.541934+0000","last_clean":"2024-09-16T15:46:58.541934+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:46:58.560118+0000","last_undegraded":"2024-09-16T15:46:58.560118+0000","last_fullsized":"2024-09-16T15:46:58.560118+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_clean_scrub_stamp":"2024-09-16T15:46:58.541934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.2f","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-09-16T15:46:58.561316+0000","last_change":"2024-09-16T15:46:58.561316+0000","last_active":"2024-09-16T15:46:58.541934+0000","last_peered":"2024-09-16T15:46:58.541934+0000","last_clean":"2024-09-16T15:46:58.541934+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:46:58.561316+0000","last_undegraded":"2024-09-16T15:46:58.561316+0000","last_fullsized":"2024-09-16T15:46:58.561316+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_clean_scrub_stamp":"2024-09-16T15:46:58.541934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.2e","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-09-16T15:46:58.559376+0000","last_change":"2024-09-16T15:46:58.559376+0000","last_active":"2024-09-16T15:46:58.541934+0000","last_peered":"2024-09-16T15:46:58.541934+0000","last_clean":"2024-09-16T15:46:58.541934+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:46:58.559376+0000","last_undegraded":"2024-09-16T15:46:58.559376+0000","last_fullsized":"2024-09-16T15:46:58.559376+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_clean_scrub_stamp":"2024-09-16T15:46:58.541934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.2d","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-09-16T15:46:58.580101+0000","last_change":"2024-09-16T15:46:58.580101+0000","last_active":"2024-09-16T15:46:58.541934+0000","last_peered":"2024-09-16T15:46:58.541934+0000","last_clean":"2024-09-16T15:46:58.541934+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:46:58.580101+0000","last_undegraded":"2024-09-16T15:46:58.580101+0000","last_fullsized":"2024-09-16T15:46:58.580101+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_clean_scrub_stamp":"2024-09-16T15:46:58.541934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.2c","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-09-16T15:46:58.562539+0000","last_change":"2024-09-16T15:46:58.562539+0000","last_active":"2024-09-16T15:46:58.541934+0000","last_peered":"2024-09-16T15:46:58.541934+0000","last_clean":"2024-09-16T15:46:58.541934+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:46:58.562539+0000","last_undegraded":"2024-09-16T15:46:58.562539+0000","last_fullsized":"2024-09-16T15:46:58.562539+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_clean_scrub_stamp":"2024-09-16T15:46:58.541934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.2b","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-09-16T15:46:58.564602+0000","last_change":"2024-09-16T15:46:58.564602+0000","last_active":"2024-09-16T15:46:58.541934+0000","last_peered":"2024-09-16T15:46:58.541934+0000","last_clean":"2024-09-16T15:46:58.541934+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:46:58.564602+0000","last_undegraded":"2024-09-16T15:46:58.564602+0000","last_fullsized":"2024-09-16T15:46:58.564602+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_clean_scrub_stamp":"2024-09-16T15:46:58.541934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.2a","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-09-16T15:46:58.562954+0000","last_change":"2024-09-16T15:46:58.562954+0000","last_active":"2024-09-16T15:46:58.541934+0000","last_peered":"2024-09-16T15:46:58.541934+0000","last_clean":"2024-09-16T15:46:58.541934+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:46:58.562954+0000","last_undegraded":"2024-09-16T15:46:58.562954+0000","last_fullsized":"2024-09-16T15:46:58.562954+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_clean_scrub_stamp":"2024-09-16T15:46:58.541934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.29","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-09-16T15:46:58.578047+0000","last_change":"2024-09-16T15:46:58.578047+0000","last_active":"2024-09-16T15:46:58.541934+0000","last_peered":"2024-09-16T15:46:58.541934+0000","last_clean":"2024-09-16T15:46:58.541934+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:46:58.578047+0000","last_undegraded":"2024-09-16T15:46:58.578047+0000","last_fullsized":"2024-09-16T15:46:58.578047+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_clean_scrub_stamp":"2024-09-16T15:46:58.541934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.28","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-09-16T15:46:58.575628+0000","last_change":"2024-09-16T15:46:58.575628+0000","last_active":"2024-09-16T15:46:58.541934+0000","last_peered":"2024-09-16T15:46:58.541934+0000","last_clean":"2024-09-16T15:46:58.541934+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:46:58.575628+0000","last_undegraded":"2024-09-16T15:46:58.575628+0000","last_fullsized":"2024-09-16T15:46:58.575628+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_clean_scrub_stamp":"2024-09-16T15:46:58.541934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.27","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-09-16T15:46:58.562122+0000","last_change":"2024-09-16T15:46:58.562122+0000","last_active":"2024-09-16T15:46:58.541934+0000","last_peered":"2024-09-16T15:46:58.541934+0000","last_clean":"2024-09-16T15:46:58.541934+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:46:58.562122+0000","last_undegraded":"2024-09-16T15:46:58.562122+0000","last_fullsized":"2024-09-16T15:46:58.562122+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_clean_scrub_stamp":"2024-09-16T15:46:58.541934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.26","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-09-16T15:46:58.571812+0000","last_change":"2024-09-16T15:46:58.571812+0000","last_active":"2024-09-16T15:46:58.541934+0000","last_peered":"2024-09-16T15:46:58.541934+0000","last_clean":"2024-09-16T15:46:58.541934+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:46:58.571812+0000","last_undegraded":"2024-09-16T15:46:58.571812+0000","last_fullsized":"2024-09-16T15:46:58.571812+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_clean_scrub_stamp":"2024-09-16T15:46:58.541934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.25","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-09-16T15:46:58.564846+0000","last_change":"2024-09-16T15:46:58.564846+0000","last_active":"2024-09-16T15:46:58.541934+0000","last_peered":"2024-09-16T15:46:58.541934+0000","last_clean":"2024-09-16T15:46:58.541934+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:46:58.564846+0000","last_undegraded":"2024-09-16T15:46:58.564846+0000","last_fullsized":"2024-09-16T15:46:58.564846+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_clean_scrub_stamp":"2024-09-16T15:46:58.541934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.24","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-09-16T15:46:58.560367+0000","last_change":"2024-09-16T15:46:58.560367+0000","last_active":"2024-09-16T15:46:58.541934+0000","last_peered":"2024-09-16T15:46:58.541934+0000","last_clean":"2024-09-16T15:46:58.541934+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:46:58.560367+0000","last_undegraded":"2024-09-16T15:46:58.560367+0000","last_fullsized":"2024-09-16T15:46:58.560367+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_clean_scrub_stamp":"2024-09-16T15:46:58.541934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.23","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-09-16T15:46:58.568304+0000","last_change":"2024-09-16T15:46:58.568304+0000","last_active":"2024-09-16T15:46:58.541934+0000","last_peered":"2024-09-16T15:46:58.541934+0000","last_clean":"2024-09-16T15:46:58.541934+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:46:58.568304+0000","last_undegraded":"2024-09-16T15:46:58.568304+0000","last_fullsized":"2024-09-16T15:46:58.568304+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_clean_scrub_stamp":"2024-09-16T15:46:58.541934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.22","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-09-16T15:46:58.569453+0000","last_change":"2024-09-16T15:46:58.569453+0000","last_active":"2024-09-16T15:46:58.541934+0000","last_peered":"2024-09-16T15:46:58.541934+0000","last_clean":"2024-09-16T15:46:58.541934+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:46:58.569453+0000","last_undegraded":"2024-09-16T15:46:58.569453+0000","last_fullsized":"2024-09-16T15:46:58.569453+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_clean_scrub_stamp":"2024-09-16T15:46:58.541934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.21","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-09-16T15:46:58.569512+0000","last_change":"2024-09-16T15:46:58.569512+0000","last_active":"2024-09-16T15:46:58.541934+0000","last_peered":"2024-09-16T15:46:58.541934+0000","last_clean":"2024-09-16T15:46:58.541934+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:46:58.569512+0000","last_undegraded":"2024-09-16T15:46:58.569512+0000","last_fullsized":"2024-09-16T15:46:58.569512+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_clean_scrub_stamp":"2024-09-16T15:46:58.541934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.20","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-09-16T15:46:58.582821+0000","last_change":"2024-09-16T15:46:58.582821+0000","last_active":"2024-09-16T15:46:58.541934+0000","last_peered":"2024-09-16T15:46:58.541934+0000","last_clean":"2024-09-16T15:46:58.541934+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:46:58.582821+0000","last_undegraded":"2024-09-16T15:46:58.582821+0000","last_fullsized":"2024-09-16T15:46:58.582821+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_clean_scrub_stamp":"2024-09-16T15:46:58.541934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.1e","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-09-16T15:46:58.575147+0000","last_change":"2024-09-16T15:46:58.575147+0000","last_active":"2024-09-16T15:46:58.541934+0000","last_peered":"2024-09-16T15:46:58.541934+0000","last_clean":"2024-09-16T15:46:58.541934+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:46:58.575147+0000","last_undegraded":"2024-09-16T15:46:58.575147+0000","last_fullsized":"2024-09-16T15:46:58.575147+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_clean_scrub_stamp":"2024-09-16T15:46:58.541934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.1f","version":"0'0","reported_seq":10,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-16T15:46:58.599998+0000","last_change":"2024-09-16T15:46:58.596071+0000","last_active":"2024-09-16T15:46:58.599998+0000","last_peered":"2024-09-16T15:46:58.599998+0000","last_clean":"2024-09-16T15:46:58.599998+0000","last_became_active":"2024-09-16T15:46:58.595309+0000","last_became_peered":"2024-09-16T15:46:58.595309+0000","last_unstale":"2024-09-16T15:46:58.599998+0000","last_undegraded":"2024-09-16T15:46:58.599998+0000","last_fullsized":"2024-09-16T15:46:58.599998+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:57.531046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:57.531046+0000","last_clean_scrub_stamp":"2024-09-16T15:46:57.531046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:37:47.205740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.1f","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-09-16T15:46:58.560943+0000","last_change":"2024-09-16T15:46:58.560943+0000","last_active":"2024-09-16T15:46:58.541934+0000","last_peered":"2024-09-16T15:46:58.541934+0000","last_clean":"2024-09-16T15:46:58.541934+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:46:58.560943+0000","last_undegraded":"2024-09-16T15:46:58.560943+0000","last_fullsized":"2024-09-16T15:46:58.560943+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_clean_scrub_stamp":"2024-09-16T15:46:58.541934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.1e","version":"0'0","reported_seq":10,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-16T15:46:58.577768+0000","last_change":"2024-09-16T15:46:58.572883+0000","last_active":"2024-09-16T15:46:58.577768+0000","last_peered":"2024-09-16T15:46:58.577768+0000","last_clean":"2024-09-16T15:46:58.577768+0000","last_became_active":"2024-09-16T15:46:58.572508+0000","last_became_peered":"2024-09-16T15:46:58.572508+0000","last_unstale":"2024-09-16T15:46:58.577768+0000","last_undegraded":"2024-09-16T15:46:58.577768+0000","last_fullsized":"2024-09-16T15:46:58.577768+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:57.531046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:57.531046+0000","last_clean_scrub_stamp":"2024-09-16T15:46:57.531046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:16:24.337739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1c","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-09-16T15:46:58.583638+0000","last_change":"2024-09-16T15:46:58.583638+0000","last_active":"2024-09-16T15:46:58.541934+0000","last_peered":"2024-09-16T15:46:58.541934+0000","last_clean":"2024-09-16T15:46:58.541934+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:46:58.583638+0000","last_undegraded":"2024-09-16T15:46:58.583638+0000","last_fullsized":"2024-09-16T15:46:58.583638+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_clean_scrub_stamp":"2024-09-16T15:46:58.541934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.1d","version":"0'0","reported_seq":10,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-16T15:46:58.604745+0000","last_change":"2024-09-16T15:46:58.598772+0000","last_active":"2024-09-16T15:46:58.604745+0000","last_peered":"2024-09-16T15:46:58.604745+0000","last_clean":"2024-09-16T15:46:58.604745+0000","last_became_active":"2024-09-16T15:46:58.597889+0000","last_became_peered":"2024-09-16T15:46:58.597889+0000","last_unstale":"2024-09-16T15:46:58.604745+0000","last_undegraded":"2024-09-16T15:46:58.604745+0000","last_fullsized":"2024-09-16T15:46:58.604745+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:57.531046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:57.531046+0000","last_clean_scrub_stamp":"2024-09-16T15:46:57.531046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:20:50.612777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1d","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-09-16T15:46:58.579199+0000","last_change":"2024-09-16T15:46:58.579199+0000","last_active":"2024-09-16T15:46:58.541934+0000","last_peered":"2024-09-16T15:46:58.541934+0000","last_clean":"2024-09-16T15:46:58.541934+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:46:58.579199+0000","last_undegraded":"2024-09-16T15:46:58.579199+0000","last_fullsized":"2024-09-16T15:46:58.579199+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_clean_scrub_stamp":"2024-09-16T15:46:58.541934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.1c","version":"0'0","reported_seq":10,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-16T15:46:58.595606+0000","last_change":"2024-09-16T15:46:58.588644+0000","last_active":"2024-09-16T15:46:58.595606+0000","last_peered":"2024-09-16T15:46:58.595606+0000","last_clean":"2024-09-16T15:46:58.595606+0000","last_became_active":"2024-09-16T15:46:58.587899+0000","last_became_peered":"2024-09-16T15:46:58.587899+0000","last_unstale":"2024-09-16T15:46:58.595606+0000","last_undegraded":"2024-09-16T15:46:58.595606+0000","last_fullsized":"2024-09-16T15:46:58.595606+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:57.531046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:57.531046+0000","last_clean_scrub_stamp":"2024-09-16T15:46:57.531046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T00:36:13.054311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.1a","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-09-16T15:46:58.568314+0000","last_change":"2024-09-16T15:46:58.568314+0000","last_active":"2024-09-16T15:46:58.541934+0000","last_peered":"2024-09-16T15:46:58.541934+0000","last_clean":"2024-09-16T15:46:58.541934+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:46:58.568314+0000","last_undegraded":"2024-09-16T15:46:58.568314+0000","last_fullsized":"2024-09-16T15:46:58.568314+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_clean_scrub_stamp":"2024-09-16T15:46:58.541934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.1b","version":"0'0","reported_seq":10,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-16T15:46:58.594471+0000","last_change":"2024-09-16T15:46:58.591621+0000","last_active":"2024-09-16T15:46:58.594471+0000","last_peered":"2024-09-16T15:46:58.594471+0000","last_clean":"2024-09-16T15:46:58.594471+0000","last_became_active":"2024-09-16T15:46:58.591311+0000","last_became_peered":"2024-09-16T15:46:58.591311+0000","last_unstale":"2024-09-16T15:46:58.594471+0000","last_undegraded":"2024-09-16T15:46:58.594471+0000","last_fullsized":"2024-09-16T15:46:58.594471+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:57.531046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:57.531046+0000","last_clean_scrub_stamp":"2024-09-16T15:46:57.531046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:55:01.184561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.1b","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-09-16T15:46:58.563604+0000","last_change":"2024-09-16T15:46:58.563604+0000","last_active":"2024-09-16T15:46:58.541934+0000","last_peered":"2024-09-16T15:46:58.541934+0000","last_clean":"2024-09-16T15:46:58.541934+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:46:58.563604+0000","last_undegraded":"2024-09-16T15:46:58.563604+0000","last_fullsized":"2024-09-16T15:46:58.563604+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_clean_scrub_stamp":"2024-09-16T15:46:58.541934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.1a","version":"0'0","reported_seq":10,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-16T15:46:58.604119+0000","last_change":"2024-09-16T15:46:58.600655+0000","last_active":"2024-09-16T15:46:58.604119+0000","last_peered":"2024-09-16T15:46:58.604119+0000","last_clean":"2024-09-16T15:46:58.604119+0000","last_became_active":"2024-09-16T15:46:58.599216+0000","last_became_peered":"2024-09-16T15:46:58.599216+0000","last_unstale":"2024-09-16T15:46:58.604119+0000","last_undegraded":"2024-09-16T15:46:58.604119+0000","last_fullsized":"2024-09-16T15:46:58.604119+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:57.531046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:57.531046+0000","last_clean_scrub_stamp":"2024-09-16T15:46:57.531046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:13:18.615772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.18","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-09-16T15:46:58.562911+0000","last_change":"2024-09-16T15:46:58.562911+0000","last_active":"2024-09-16T15:46:58.541934+0000","last_peered":"2024-09-16T15:46:58.541934+0000","last_clean":"2024-09-16T15:46:58.541934+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:46:58.562911+0000","last_undegraded":"2024-09-16T15:46:58.562911+0000","last_fullsized":"2024-09-16T15:46:58.562911+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_clean_scrub_stamp":"2024-09-16T15:46:58.541934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.19","version":"0'0","reported_seq":10,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-16T15:46:58.589251+0000","last_change":"2024-09-16T15:46:58.584364+0000","last_active":"2024-09-16T15:46:58.589251+0000","last_peered":"2024-09-16T15:46:58.589251+0000","last_clean":"2024-09-16T15:46:58.589251+0000","last_became_active":"2024-09-16T15:46:58.583975+0000","last_became_peered":"2024-09-16T15:46:58.583975+0000","last_unstale":"2024-09-16T15:46:58.589251+0000","last_undegraded":"2024-09-16T15:46:58.589251+0000","last_fullsized":"2024-09-16T15:46:58.589251+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:57.531046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:57.531046+0000","last_clean_scrub_stamp":"2024-09-16T15:46:57.531046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T00:25:29.249313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.19","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-09-16T15:46:58.566064+0000","last_change":"2024-09-16T15:46:58.566064+0000","last_active":"2024-09-16T15:46:58.541934+0000","last_peered":"2024-09-16T15:46:58.541934+0000","last_clean":"2024-09-16T15:46:58.541934+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:46:58.566064+0000","last_undegraded":"2024-09-16T15:46:58.566064+0000","last_fullsized":"2024-09-16T15:46:58.566064+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_clean_scrub_stamp":"2024-09-16T15:46:58.541934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.18","version":"0'0","reported_seq":10,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-16T15:46:58.574992+0000","last_change":"2024-09-16T15:46:58.570577+0000","last_active":"2024-09-16T15:46:58.574992+0000","last_peered":"2024-09-16T15:46:58.574992+0000","last_clean":"2024-09-16T15:46:58.574992+0000","last_became_active":"2024-09-16T15:46:58.570223+0000","last_became_peered":"2024-09-16T15:46:58.570223+0000","last_unstale":"2024-09-16T15:46:58.574992+0000","last_undegraded":"2024-09-16T15:46:58.574992+0000","last_fullsized":"2024-09-16T15:46:58.574992+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:57.531046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:57.531046+0000","last_clean_scrub_stamp":"2024-09-16T15:46:57.531046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:11:59.055640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.16","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-09-16T15:46:58.570594+0000","last_change":"2024-09-16T15:46:58.570594+0000","last_active":"2024-09-16T15:46:58.541934+0000","last_peered":"2024-09-16T15:46:58.541934+0000","last_clean":"2024-09-16T15:46:58.541934+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:46:58.570594+0000","last_undegraded":"2024-09-16T15:46:58.570594+0000","last_fullsized":"2024-09-16T15:46:58.570594+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_clean_scrub_stamp":"2024-09-16T15:46:58.541934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.17","version":"0'0","reported_seq":10,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-16T15:46:58.590618+0000","last_change":"2024-09-16T15:46:58.576463+0000","last_active":"2024-09-16T15:46:58.590618+0000","last_peered":"2024-09-16T15:46:58.590618+0000","last_clean":"2024-09-16T15:46:58.590618+0000","last_became_active":"2024-09-16T15:46:58.573574+0000","last_became_peered":"2024-09-16T15:46:58.573574+0000","last_unstale":"2024-09-16T15:46:58.590618+0000","last_undegraded":"2024-09-16T15:46:58.590618+0000","last_fullsized":"2024-09-16T15:46:58.590618+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:57.531046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:57.531046+0000","last_clean_scrub_stamp":"2024-09-16T15:46:57.531046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T01:24:15.136386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.17","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-09-16T15:46:58.579574+0000","last_change":"2024-09-16T15:46:58.579574+0000","last_active":"2024-09-16T15:46:58.541934+0000","last_peered":"2024-09-16T15:46:58.541934+0000","last_clean":"2024-09-16T15:46:58.541934+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:46:58.579574+0000","last_undegraded":"2024-09-16T15:46:58.579574+0000","last_fullsized":"2024-09-16T15:46:58.579574+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_clean_scrub_stamp":"2024-09-16T15:46:58.541934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.16","version":"0'0","reported_seq":10,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-16T15:46:58.602617+0000","last_change":"2024-09-16T15:46:58.597162+0000","last_active":"2024-09-16T15:46:58.602617+0000","last_peered":"2024-09-16T15:46:58.602617+0000","last_clean":"2024-09-16T15:46:58.602617+0000","last_became_active":"2024-09-16T15:46:58.596469+0000","last_became_peered":"2024-09-16T15:46:58.596469+0000","last_unstale":"2024-09-16T15:46:58.602617+0000","last_undegraded":"2024-09-16T15:46:58.602617+0000","last_fullsized":"2024-09-16T15:46:58.602617+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:57.531046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:57.531046+0000","last_clean_scrub_stamp":"2024-09-16T15:46:57.531046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:22:59.268994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.14","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-09-16T15:46:58.582525+0000","last_change":"2024-09-16T15:46:58.582525+0000","last_active":"2024-09-16T15:46:58.541934+0000","last_peered":"2024-09-16T15:46:58.541934+0000","last_clean":"2024-09-16T15:46:58.541934+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:46:58.582525+0000","last_undegraded":"2024-09-16T15:46:58.582525+0000","last_fullsized":"2024-09-16T15:46:58.582525+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_clean_scrub_stamp":"2024-09-16T15:46:58.541934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.15","version":"0'0","reported_seq":10,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-16T15:46:58.588947+0000","last_change":"2024-09-16T15:46:58.583907+0000","last_active":"2024-09-16T15:46:58.588947+0000","last_peered":"2024-09-16T15:46:58.588947+0000","last_clean":"2024-09-16T15:46:58.588947+0000","last_became_active":"2024-09-16T15:46:58.583530+0000","last_became_peered":"2024-09-16T15:46:58.583530+0000","last_unstale":"2024-09-16T15:46:58.588947+0000","last_undegraded":"2024-09-16T15:46:58.588947+0000","last_fullsized":"2024-09-16T15:46:58.588947+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:57.531046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:57.531046+0000","last_clean_scrub_stamp":"2024-09-16T15:46:57.531046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T01:06:36.276569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.15","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-09-16T15:46:58.563631+0000","last_change":"2024-09-16T15:46:58.563631+0000","last_active":"2024-09-16T15:46:58.541934+0000","last_peered":"2024-09-16T15:46:58.541934+0000","last_clean":"2024-09-16T15:46:58.541934+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:46:58.563631+0000","last_undegraded":"2024-09-16T15:46:58.563631+0000","last_fullsized":"2024-09-16T15:46:58.563631+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_clean_scrub_stamp":"2024-09-16T15:46:58.541934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.14","version":"0'0","reported_seq":10,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-16T15:46:58.575157+0000","last_change":"2024-09-16T15:46:58.570712+0000","last_active":"2024-09-16T15:46:58.575157+0000","last_peered":"2024-09-16T15:46:58.575157+0000","last_clean":"2024-09-16T15:46:58.575157+0000","last_became_active":"2024-09-16T15:46:58.570432+0000","last_became_peered":"2024-09-16T15:46:58.570432+0000","last_unstale":"2024-09-16T15:46:58.575157+0000","last_undegraded":"2024-09-16T15:46:58.575157+0000","last_fullsized":"2024-09-16T15:46:58.575157+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:57.531046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:57.531046+0000","last_clean_scrub_stamp":"2024-09-16T15:46:57.531046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:58:27.343595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.12","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-09-16T15:46:58.574297+0000","last_change":"2024-09-16T15:46:58.574297+0000","last_active":"2024-09-16T15:46:58.541934+0000","last_peered":"2024-09-16T15:46:58.541934+0000","last_clean":"2024-09-16T15:46:58.541934+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:46:58.574297+0000","last_undegraded":"2024-09-16T15:46:58.574297+0000","last_fullsized":"2024-09-16T15:46:58.574297+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_clean_scrub_stamp":"2024-09-16T15:46:58.541934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.13","version":"0'0","reported_seq":10,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-16T15:46:58.603021+0000","last_change":"2024-09-16T15:46:58.596492+0000","last_active":"2024-09-16T15:46:58.603021+0000","last_peered":"2024-09-16T15:46:58.603021+0000","last_clean":"2024-09-16T15:46:58.603021+0000","last_became_active":"2024-09-16T15:46:58.596172+0000","last_became_peered":"2024-09-16T15:46:58.596172+0000","last_unstale":"2024-09-16T15:46:58.603021+0000","last_undegraded":"2024-09-16T15:46:58.603021+0000","last_fullsized":"2024-09-16T15:46:58.603021+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:57.531046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:57.531046+0000","last_clean_scrub_stamp":"2024-09-16T15:46:57.531046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T02:09:19.272595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.13","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-09-16T15:46:58.569492+0000","last_change":"2024-09-16T15:46:58.569492+0000","last_active":"2024-09-16T15:46:58.541934+0000","last_peered":"2024-09-16T15:46:58.541934+0000","last_clean":"2024-09-16T15:46:58.541934+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:46:58.569492+0000","last_undegraded":"2024-09-16T15:46:58.569492+0000","last_fullsized":"2024-09-16T15:46:58.569492+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_clean_scrub_stamp":"2024-09-16T15:46:58.541934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.12","version":"0'0","reported_seq":10,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-16T15:46:58.602887+0000","last_change":"2024-09-16T15:46:58.595177+0000","last_active":"2024-09-16T15:46:58.602887+0000","last_peered":"2024-09-16T15:46:58.602887+0000","last_clean":"2024-09-16T15:46:58.602887+0000","last_became_active":"2024-09-16T15:46:58.593818+0000","last_became_peered":"2024-09-16T15:46:58.593818+0000","last_unstale":"2024-09-16T15:46:58.602887+0000","last_undegraded":"2024-09-16T15:46:58.602887+0000","last_fullsized":"2024-09-16T15:46:58.602887+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:57.531046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:57.531046+0000","last_clean_scrub_stamp":"2024-09-16T15:46:57.531046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T01:59:03.824917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.10","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-09-16T15:46:58.562563+0000","last_change":"2024-09-16T15:46:58.562563+0000","last_active":"2024-09-16T15:46:58.541934+0000","last_peered":"2024-09-16T15:46:58.541934+0000","last_clean":"2024-09-16T15:46:58.541934+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:46:58.562563+0000","last_undegraded":"2024-09-16T15:46:58.562563+0000","last_fullsized":"2024-09-16T15:46:58.562563+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_clean_scrub_stamp":"2024-09-16T15:46:58.541934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.11","version":"0'0","reported_seq":10,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-16T15:46:58.595218+0000","last_change":"2024-09-16T15:46:58.584495+0000","last_active":"2024-09-16T15:46:58.595218+0000","last_peered":"2024-09-16T15:46:58.595218+0000","last_clean":"2024-09-16T15:46:58.595218+0000","last_became_active":"2024-09-16T15:46:58.583741+0000","last_became_peered":"2024-09-16T15:46:58.583741+0000","last_unstale":"2024-09-16T15:46:58.595218+0000","last_undegraded":"2024-09-16T15:46:58.595218+0000","last_fullsized":"2024-09-16T15:46:58.595218+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:57.531046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:57.531046+0000","last_clean_scrub_stamp":"2024-09-16T15:46:57.531046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T01:58:08.477199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.11","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-09-16T15:46:58.560818+0000","last_change":"2024-09-16T15:46:58.560818+0000","last_active":"2024-09-16T15:46:58.541934+0000","last_peered":"2024-09-16T15:46:58.541934+0000","last_clean":"2024-09-16T15:46:58.541934+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:46:58.560818+0000","last_undegraded":"2024-09-16T15:46:58.560818+0000","last_fullsized":"2024-09-16T15:46:58.560818+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_clean_scrub_stamp":"2024-09-16T15:46:58.541934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.10","version":"0'0","reported_seq":10,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-16T15:46:58.604683+0000","last_change":"2024-09-16T15:46:58.597568+0000","last_active":"2024-09-16T15:46:58.604683+0000","last_peered":"2024-09-16T15:46:58.604683+0000","last_clean":"2024-09-16T15:46:58.604683+0000","last_became_active":"2024-09-16T15:46:58.596806+0000","last_became_peered":"2024-09-16T15:46:58.596806+0000","last_unstale":"2024-09-16T15:46:58.604683+0000","last_undegraded":"2024-09-16T15:46:58.604683+0000","last_fullsized":"2024-09-16T15:46:58.604683+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:57.531046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:57.531046+0000","last_clean_scrub_stamp":"2024-09-16T15:46:57.531046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:33:54.157169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.e","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-09-16T15:46:58.561429+0000","last_change":"2024-09-16T15:46:58.561429+0000","last_active":"2024-09-16T15:46:58.541934+0000","last_peered":"2024-09-16T15:46:58.541934+0000","last_clean":"2024-09-16T15:46:58.541934+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:46:58.561429+0000","last_undegraded":"2024-09-16T15:46:58.561429+0000","last_fullsized":"2024-09-16T15:46:58.561429+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_clean_scrub_stamp":"2024-09-16T15:46:58.541934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.f","version":"0'0","reported_seq":10,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-16T15:46:58.602564+0000","last_change":"2024-09-16T15:46:58.595767+0000","last_active":"2024-09-16T15:46:58.602564+0000","last_peered":"2024-09-16T15:46:58.602564+0000","last_clean":"2024-09-16T15:46:58.602564+0000","last_became_active":"2024-09-16T15:46:58.594655+0000","last_became_peered":"2024-09-16T15:46:58.594655+0000","last_unstale":"2024-09-16T15:46:58.602564+0000","last_undegraded":"2024-09-16T15:46:58.602564+0000","last_fullsized":"2024-09-16T15:46:58.602564+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:57.531046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:57.531046+0000","last_clean_scrub_stamp":"2024-09-16T15:46:57.531046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:56:28.530413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.f","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-09-16T15:46:58.578428+0000","last_change":"2024-09-16T15:46:58.578428+0000","last_active":"2024-09-16T15:46:58.541934+0000","last_peered":"2024-09-16T15:46:58.541934+0000","last_clean":"2024-09-16T15:46:58.541934+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:46:58.578428+0000","last_undegraded":"2024-09-16T15:46:58.578428+0000","last_fullsized":"2024-09-16T15:46:58.578428+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_clean_scrub_stamp":"2024-09-16T15:46:58.541934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.e","version":"0'0","reported_seq":10,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-16T15:46:58.604653+0000","last_change":"2024-09-16T15:46:58.597152+0000","last_active":"2024-09-16T15:46:58.604653+0000","last_peered":"2024-09-16T15:46:58.604653+0000","last_clean":"2024-09-16T15:46:58.604653+0000","last_became_active":"2024-09-16T15:46:58.596435+0000","last_became_peered":"2024-09-16T15:46:58.596435+0000","last_unstale":"2024-09-16T15:46:58.604653+0000","last_undegraded":"2024-09-16T15:46:58.604653+0000","last_fullsized":"2024-09-16T15:46:58.604653+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:57.531046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:57.531046+0000","last_clean_scrub_stamp":"2024-09-16T15:46:57.531046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:11:50.475616+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.c","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-09-16T15:46:58.559398+0000","last_change":"2024-09-16T15:46:58.559398+0000","last_active":"2024-09-16T15:46:58.541934+0000","last_peered":"2024-09-16T15:46:58.541934+0000","last_clean":"2024-09-16T15:46:58.541934+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:46:58.559398+0000","last_undegraded":"2024-09-16T15:46:58.559398+0000","last_fullsized":"2024-09-16T15:46:58.559398+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_clean_scrub_stamp":"2024-09-16T15:46:58.541934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.d","version":"0'0","reported_seq":10,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-16T15:46:58.602764+0000","last_change":"2024-09-16T15:46:58.598349+0000","last_active":"2024-09-16T15:46:58.602764+0000","last_peered":"2024-09-16T15:46:58.602764+0000","last_clean":"2024-09-16T15:46:58.602764+0000","last_became_active":"2024-09-16T15:46:58.595349+0000","last_became_peered":"2024-09-16T15:46:58.595349+0000","last_unstale":"2024-09-16T15:46:58.602764+0000","last_undegraded":"2024-09-16T15:46:58.602764+0000","last_fullsized":"2024-09-16T15:46:58.602764+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:57.531046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:57.531046+0000","last_clean_scrub_stamp":"2024-09-16T15:46:57.531046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T01:43:29.245355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.d","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-09-16T15:46:58.563246+0000","last_change":"2024-09-16T15:46:58.563246+0000","last_active":"2024-09-16T15:46:58.541934+0000","last_peered":"2024-09-16T15:46:58.541934+0000","last_clean":"2024-09-16T15:46:58.541934+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:46:58.563246+0000","last_undegraded":"2024-09-16T15:46:58.563246+0000","last_fullsized":"2024-09-16T15:46:58.563246+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_clean_scrub_stamp":"2024-09-16T15:46:58.541934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.c","version":"0'0","reported_seq":10,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-16T15:46:58.599912+0000","last_change":"2024-09-16T15:46:58.596863+0000","last_active":"2024-09-16T15:46:58.599912+0000","last_peered":"2024-09-16T15:46:58.599912+0000","last_clean":"2024-09-16T15:46:58.599912+0000","last_became_active":"2024-09-16T15:46:58.596143+0000","last_became_peered":"2024-09-16T15:46:58.596143+0000","last_unstale":"2024-09-16T15:46:58.599912+0000","last_undegraded":"2024-09-16T15:46:58.599912+0000","last_fullsized":"2024-09-16T15:46:58.599912+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:57.531046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:57.531046+0000","last_clean_scrub_stamp":"2024-09-16T15:46:57.531046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:50:55.009663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.b","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-09-16T15:46:58.567734+0000","last_change":"2024-09-16T15:46:58.567734+0000","last_active":"2024-09-16T15:46:58.541934+0000","last_peered":"2024-09-16T15:46:58.541934+0000","last_clean":"2024-09-16T15:46:58.541934+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:46:58.567734+0000","last_undegraded":"2024-09-16T15:46:58.567734+0000","last_fullsized":"2024-09-16T15:46:58.567734+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_clean_scrub_stamp":"2024-09-16T15:46:58.541934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.a","version":"0'0","reported_seq":10,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-16T15:46:58.597818+0000","last_change":"2024-09-16T15:46:58.585877+0000","last_active":"2024-09-16T15:46:58.597818+0000","last_peered":"2024-09-16T15:46:58.597818+0000","last_clean":"2024-09-16T15:46:58.597818+0000","last_became_active":"2024-09-16T15:46:58.585158+0000","last_became_peered":"2024-09-16T15:46:58.585158+0000","last_unstale":"2024-09-16T15:46:58.597818+0000","last_undegraded":"2024-09-16T15:46:58.597818+0000","last_fullsized":"2024-09-16T15:46:58.597818+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:57.531046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:57.531046+0000","last_clean_scrub_stamp":"2024-09-16T15:46:57.531046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T01:00:40.058766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.8","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-09-16T15:46:58.564210+0000","last_change":"2024-09-16T15:46:58.564210+0000","last_active":"2024-09-16T15:46:58.541934+0000","last_peered":"2024-09-16T15:46:58.541934+0000","last_clean":"2024-09-16T15:46:58.541934+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:46:58.564210+0000","last_undegraded":"2024-09-16T15:46:58.564210+0000","last_fullsized":"2024-09-16T15:46:58.564210+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_clean_scrub_stamp":"2024-09-16T15:46:58.541934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.9","version":"0'0","reported_seq":10,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-16T15:46:58.583297+0000","last_change":"2024-09-16T15:46:58.576781+0000","last_active":"2024-09-16T15:46:58.583297+0000","last_peered":"2024-09-16T15:46:58.583297+0000","last_clean":"2024-09-16T15:46:58.583297+0000","last_became_active":"2024-09-16T15:46:58.576081+0000","last_became_peered":"2024-09-16T15:46:58.576081+0000","last_unstale":"2024-09-16T15:46:58.583297+0000","last_undegraded":"2024-09-16T15:46:58.583297+0000","last_fullsized":"2024-09-16T15:46:58.583297+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:57.531046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:57.531046+0000","last_clean_scrub_stamp":"2024-09-16T15:46:57.531046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T22:41:00.960336+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.9","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-09-16T15:46:58.564335+0000","last_change":"2024-09-16T15:46:58.564335+0000","last_active":"2024-09-16T15:46:58.541934+0000","last_peered":"2024-09-16T15:46:58.541934+0000","last_clean":"2024-09-16T15:46:58.541934+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:46:58.564335+0000","last_undegraded":"2024-09-16T15:46:58.564335+0000","last_fullsized":"2024-09-16T15:46:58.564335+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_clean_scrub_stamp":"2024-09-16T15:46:58.541934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.8","version":"0'0","reported_seq":10,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-16T15:46:58.577818+0000","last_change":"2024-09-16T15:46:58.572736+0000","last_active":"2024-09-16T15:46:58.577818+0000","last_peered":"2024-09-16T15:46:58.577818+0000","last_clean":"2024-09-16T15:46:58.577818+0000","last_became_active":"2024-09-16T15:46:58.572311+0000","last_became_peered":"2024-09-16T15:46:58.572311+0000","last_unstale":"2024-09-16T15:46:58.577818+0000","last_undegraded":"2024-09-16T15:46:58.577818+0000","last_fullsized":"2024-09-16T15:46:58.577818+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:57.531046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:57.531046+0000","last_clean_scrub_stamp":"2024-09-16T15:46:57.531046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:48:10.471983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.6","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-09-16T15:46:58.562758+0000","last_change":"2024-09-16T15:46:58.562758+0000","last_active":"2024-09-16T15:46:58.541934+0000","last_peered":"2024-09-16T15:46:58.541934+0000","last_clean":"2024-09-16T15:46:58.541934+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:46:58.562758+0000","last_undegraded":"2024-09-16T15:46:58.562758+0000","last_fullsized":"2024-09-16T15:46:58.562758+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_clean_scrub_stamp":"2024-09-16T15:46:58.541934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.7","version":"0'0","reported_seq":10,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-16T15:46:58.594095+0000","last_change":"2024-09-16T15:46:58.579355+0000","last_active":"2024-09-16T15:46:58.594095+0000","last_peered":"2024-09-16T15:46:58.594095+0000","last_clean":"2024-09-16T15:46:58.594095+0000","last_became_active":"2024-09-16T15:46:58.578380+0000","last_became_peered":"2024-09-16T15:46:58.578380+0000","last_unstale":"2024-09-16T15:46:58.594095+0000","last_undegraded":"2024-09-16T15:46:58.594095+0000","last_fullsized":"2024-09-16T15:46:58.594095+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:57.531046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:57.531046+0000","last_clean_scrub_stamp":"2024-09-16T15:46:57.531046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T23:40:10.473815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.7","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-09-16T15:46:58.569698+0000","last_change":"2024-09-16T15:46:58.569698+0000","last_active":"2024-09-16T15:46:58.541934+0000","last_peered":"2024-09-16T15:46:58.541934+0000","last_clean":"2024-09-16T15:46:58.541934+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:46:58.569698+0000","last_undegraded":"2024-09-16T15:46:58.569698+0000","last_fullsized":"2024-09-16T15:46:58.569698+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_clean_scrub_stamp":"2024-09-16T15:46:58.541934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.6","version":"0'0","reported_seq":10,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-16T15:46:58.582810+0000","last_change":"2024-09-16T15:46:58.577011+0000","last_active":"2024-09-16T15:46:58.582810+0000","last_peered":"2024-09-16T15:46:58.582810+0000","last_clean":"2024-09-16T15:46:58.582810+0000","last_became_active":"2024-09-16T15:46:58.576294+0000","last_became_peered":"2024-09-16T15:46:58.576294+0000","last_unstale":"2024-09-16T15:46:58.582810+0000","last_undegraded":"2024-09-16T15:46:58.582810+0000","last_fullsized":"2024-09-16T15:46:58.582810+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:57.531046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:57.531046+0000","last_clean_scrub_stamp":"2024-09-16T15:46:57.531046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T03:31:45.091717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.4","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-09-16T15:46:58.574408+0000","last_change":"2024-09-16T15:46:58.574408+0000","last_active":"2024-09-16T15:46:58.541934+0000","last_peered":"2024-09-16T15:46:58.541934+0000","last_clean":"2024-09-16T15:46:58.541934+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:46:58.574408+0000","last_undegraded":"2024-09-16T15:46:58.574408+0000","last_fullsized":"2024-09-16T15:46:58.574408+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_clean_scrub_stamp":"2024-09-16T15:46:58.541934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.5","version":"0'0","reported_seq":10,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-16T15:46:58.578312+0000","last_change":"2024-09-16T15:46:58.575511+0000","last_active":"2024-09-16T15:46:58.578312+0000","last_peered":"2024-09-16T15:46:58.578312+0000","last_clean":"2024-09-16T15:46:58.578312+0000","last_became_active":"2024-09-16T15:46:58.575218+0000","last_became_peered":"2024-09-16T15:46:58.575218+0000","last_unstale":"2024-09-16T15:46:58.578312+0000","last_undegraded":"2024-09-16T15:46:58.578312+0000","last_fullsized":"2024-09-16T15:46:58.578312+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:57.531046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:57.531046+0000","last_clean_scrub_stamp":"2024-09-16T15:46:57.531046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:37:03.438301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.5","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-09-16T15:46:58.578126+0000","last_change":"2024-09-16T15:46:58.578126+0000","last_active":"2024-09-16T15:46:58.541934+0000","last_peered":"2024-09-16T15:46:58.541934+0000","last_clean":"2024-09-16T15:46:58.541934+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:46:58.578126+0000","last_undegraded":"2024-09-16T15:46:58.578126+0000","last_fullsized":"2024-09-16T15:46:58.578126+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_clean_scrub_stamp":"2024-09-16T15:46:58.541934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.4","version":"0'0","reported_seq":10,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-16T15:46:58.588834+0000","last_change":"2024-09-16T15:46:58.583703+0000","last_active":"2024-09-16T15:46:58.588834+0000","last_peered":"2024-09-16T15:46:58.588834+0000","last_clean":"2024-09-16T15:46:58.588834+0000","last_became_active":"2024-09-16T15:46:58.583384+0000","last_became_peered":"2024-09-16T15:46:58.583384+0000","last_unstale":"2024-09-16T15:46:58.588834+0000","last_undegraded":"2024-09-16T15:46:58.588834+0000","last_fullsized":"2024-09-16T15:46:58.588834+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:57.531046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:57.531046+0000","last_clean_scrub_stamp":"2024-09-16T15:46:57.531046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T23:06:05.405748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.2","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-09-16T15:46:58.560788+0000","last_change":"2024-09-16T15:46:58.560788+0000","last_active":"2024-09-16T15:46:58.541934+0000","last_peered":"2024-09-16T15:46:58.541934+0000","last_clean":"2024-09-16T15:46:58.541934+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:46:58.560788+0000","last_undegraded":"2024-09-16T15:46:58.560788+0000","last_fullsized":"2024-09-16T15:46:58.560788+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_clean_scrub_stamp":"2024-09-16T15:46:58.541934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.3","version":"0'0","reported_seq":10,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-16T15:46:58.577602+0000","last_change":"2024-09-16T15:46:58.573698+0000","last_active":"2024-09-16T15:46:58.577602+0000","last_peered":"2024-09-16T15:46:58.577602+0000","last_clean":"2024-09-16T15:46:58.577602+0000","last_became_active":"2024-09-16T15:46:58.573391+0000","last_became_peered":"2024-09-16T15:46:58.573391+0000","last_unstale":"2024-09-16T15:46:58.577602+0000","last_undegraded":"2024-09-16T15:46:58.577602+0000","last_fullsized":"2024-09-16T15:46:58.577602+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:57.531046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:57.531046+0000","last_clean_scrub_stamp":"2024-09-16T15:46:57.531046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T00:58:45.221067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.3","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-09-16T15:46:58.586430+0000","last_change":"2024-09-16T15:46:58.586430+0000","last_active":"2024-09-16T15:46:58.541934+0000","last_peered":"2024-09-16T15:46:58.541934+0000","last_clean":"2024-09-16T15:46:58.541934+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:46:58.586430+0000","last_undegraded":"2024-09-16T15:46:58.586430+0000","last_fullsized":"2024-09-16T15:46:58.586430+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_clean_scrub_stamp":"2024-09-16T15:46:58.541934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.2","version":"0'0","reported_seq":10,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-16T15:46:58.577483+0000","last_change":"2024-09-16T15:46:58.574227+0000","last_active":"2024-09-16T15:46:58.577483+0000","last_peered":"2024-09-16T15:46:58.577483+0000","last_clean":"2024-09-16T15:46:58.577483+0000","last_became_active":"2024-09-16T15:46:58.573898+0000","last_became_peered":"2024-09-16T15:46:58.573898+0000","last_unstale":"2024-09-16T15:46:58.577483+0000","last_undegraded":"2024-09-16T15:46:58.577483+0000","last_fullsized":"2024-09-16T15:46:58.577483+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:57.531046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:57.531046+0000","last_clean_scrub_stamp":"2024-09-16T15:46:57.531046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T18:41:18.272460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.0","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-09-16T15:46:58.559237+0000","last_change":"2024-09-16T15:46:58.559237+0000","last_active":"2024-09-16T15:46:58.541934+0000","last_peered":"2024-09-16T15:46:58.541934+0000","last_clean":"2024-09-16T15:46:58.541934+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:46:58.559237+0000","last_undegraded":"2024-09-16T15:46:58.559237+0000","last_fullsized":"2024-09-16T15:46:58.559237+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_clean_scrub_stamp":"2024-09-16T15:46:58.541934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.1","version":"0'0","reported_seq":10,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-16T15:46:58.604741+0000","last_change":"2024-09-16T15:46:58.598614+0000","last_active":"2024-09-16T15:46:58.604741+0000","last_peered":"2024-09-16T15:46:58.604741+0000","last_clean":"2024-09-16T15:46:58.604741+0000","last_became_active":"2024-09-16T15:46:58.597950+0000","last_became_peered":"2024-09-16T15:46:58.597950+0000","last_unstale":"2024-09-16T15:46:58.604741+0000","last_undegraded":"2024-09-16T15:46:58.604741+0000","last_fullsized":"2024-09-16T15:46:58.604741+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:57.531046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:57.531046+0000","last_clean_scrub_stamp":"2024-09-16T15:46:57.531046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T23:50:07.979345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-09-16T15:46:58.565520+0000","last_change":"2024-09-16T15:46:58.565520+0000","last_active":"2024-09-16T15:46:58.541934+0000","last_peered":"2024-09-16T15:46:58.541934+0000","last_clean":"2024-09-16T15:46:58.541934+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:46:58.565520+0000","last_undegraded":"2024-09-16T15:46:58.565520+0000","last_fullsized":"2024-09-16T15:46:58.565520+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_clean_scrub_stamp":"2024-09-16T15:46:58.541934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.0","version":"0'0","reported_seq":10,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-16T15:46:58.602740+0000","last_change":"2024-09-16T15:46:58.599504+0000","last_active":"2024-09-16T15:46:58.602740+0000","last_peered":"2024-09-16T15:46:58.602740+0000","last_clean":"2024-09-16T15:46:58.602740+0000","last_became_active":"2024-09-16T15:46:58.598743+0000","last_became_peered":"2024-09-16T15:46:58.598743+0000","last_unstale":"2024-09-16T15:46:58.602740+0000","last_undegraded":"2024-09-16T15:46:58.602740+0000","last_fullsized":"2024-09-16T15:46:58.602740+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:57.531046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:57.531046+0000","last_clean_scrub_stamp":"2024-09-16T15:46:57.531046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T18:45:22.195277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.a","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-09-16T15:46:58.564305+0000","last_change":"2024-09-16T15:46:58.564305+0000","last_active":"2024-09-16T15:46:58.541934+0000","last_peered":"2024-09-16T15:46:58.541934+0000","last_clean":"2024-09-16T15:46:58.541934+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:46:58.564305+0000","last_undegraded":"2024-09-16T15:46:58.564305+0000","last_fullsized":"2024-09-16T15:46:58.564305+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_clean_scrub_stamp":"2024-09-16T15:46:58.541934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.b","version":"0'0","reported_seq":10,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-16T15:46:58.604289+0000","last_change":"2024-09-16T15:46:58.601513+0000","last_active":"2024-09-16T15:46:58.604289+0000","last_peered":"2024-09-16T15:46:58.604289+0000","last_clean":"2024-09-16T15:46:58.604289+0000","last_became_active":"2024-09-16T15:46:58.600581+0000","last_became_peered":"2024-09-16T15:46:58.600581+0000","last_unstale":"2024-09-16T15:46:58.604289+0000","last_undegraded":"2024-09-16T15:46:58.604289+0000","last_fullsized":"2024-09-16T15:46:58.604289+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:57.531046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:57.531046+0000","last_clean_scrub_stamp":"2024-09-16T15:46:57.531046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T23:00:41.658258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"16'72","reported_seq":200,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-16T15:46:58.585260+0000","last_change":"2024-09-16T15:42:40.881820+0000","last_active":"2024-09-16T15:46:58.585260+0000","last_peered":"2024-09-16T15:46:58.585260+0000","last_clean":"2024-09-16T15:46:58.585260+0000","last_became_active":"2024-09-16T15:42:40.879965+0000","last_became_peered":"2024-09-16T15:42:40.879965+0000","last_unstale":"2024-09-16T15:46:58.585260+0000","last_undegraded":"2024-09-16T15:46:58.585260+0000","last_fullsized":"2024-09-16T15:46:58.585260+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:42:39.728762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:42:39.728762+0000","last_clean_scrub_stamp":"2024-09-16T15:42:39.728762+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:22:18.768932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":11,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":10,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542200,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28416,"kb_used_data":1384,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343424,"statfs":{"total":96636764160,"available":96607666176,"internally_reserved":0,"allocated":1417216,"data_stored":932261,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542199,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31996,"kb_used_data":804,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339844,"statfs":{"total":96636764160,"available":96604000256,"internally_reserved":0,"allocated":823296,"data_stored":341893,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":5,"up_from":14,"seq":60129542200,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27824,"kb_used_data":792,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344016,"statfs":{"total":96636764160,"available":96608272384,"internally_reserved":0,"allocated":811008,"data_stored":335618,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542198,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31980,"kb_used_data":792,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339860,"statfs":{"total":96636764160,"available":96604016640,"internally_reserved":0,"allocated":811008,"data_stored":335618,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":3,"up_from":14,"seq":60129542199,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31996,"kb_used_data":804,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339844,"statfs":{"total":96636764160,"available":96604000256,"internally_reserved":0,"allocated":823296,"data_stored":341893,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":14,"seq":60129542199,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27820,"kb_used_data":792,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344020,"statfs":{"total":96636764160,"available":96608276480,"internally_reserved":0,"allocated":811008,"data_stored":335618,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":14,"seq":60129542201,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27840,"kb_used_data":804,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344000,"statfs":{"total":96636764160,"available":96608256000,"internally_reserved":0,"allocated":823296,"data_stored":341893,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":14,"seq":60129542201,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32572,"kb_used_data":1384,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339268,"statfs":{"total":96636764160,"available":96603410432,"internally_reserved":0,"allocated":1417216,"data_stored":932261,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T15:47:04.361 INFO:tasks.ceph.ceph_manager.ceph:PG 11.3f is not active+clean 2024-09-16T15:47:04.361 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '11.3f', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 68, 'state': 'creating+peering', 'last_fresh': '2024-09-16T15:46:58.580896+0000', 'last_change': '2024-09-16T15:46:58.580896+0000', 'last_active': '2024-09-16T15:46:58.541934+0000', 'last_peered': '2024-09-16T15:46:58.541934+0000', 'last_clean': '2024-09-16T15:46:58.541934+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-16T15:46:58.580896+0000', 'last_undegraded': '2024-09-16T15:46:58.580896+0000', 'last_fullsized': '2024-09-16T15:46:58.580896+0000', 'mapping_epoch': 68, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 68, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T15:46:58.541934+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T15:46:58.541934+0000', 'last_clean_scrub_stamp': '2024-09-16T15:46:58.541934+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 2], 'acting': [7, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-09-16T15:47:04.361 INFO:tasks.ceph.ceph_manager.ceph:PG 11.3e is not active+clean 2024-09-16T15:47:04.362 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '11.3e', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 68, 'state': 'creating+peering', 'last_fresh': '2024-09-16T15:46:58.580879+0000', 'last_change': '2024-09-16T15:46:58.580879+0000', 'last_active': '2024-09-16T15:46:58.541934+0000', 'last_peered': '2024-09-16T15:46:58.541934+0000', 'last_clean': '2024-09-16T15:46:58.541934+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-16T15:46:58.580879+0000', 'last_undegraded': '2024-09-16T15:46:58.580879+0000', 'last_fullsized': '2024-09-16T15:46:58.580879+0000', 'mapping_epoch': 68, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 68, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T15:46:58.541934+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T15:46:58.541934+0000', 'last_clean_scrub_stamp': '2024-09-16T15:46:58.541934+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 4], 'acting': [6, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-09-16T15:47:04.362 INFO:tasks.ceph.ceph_manager.ceph:PG 11.3d is not active+clean 2024-09-16T15:47:04.362 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '11.3d', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 68, 'state': 'creating+peering', 'last_fresh': '2024-09-16T15:46:58.565500+0000', 'last_change': '2024-09-16T15:46:58.565500+0000', 'last_active': '2024-09-16T15:46:58.541934+0000', 'last_peered': '2024-09-16T15:46:58.541934+0000', 'last_clean': '2024-09-16T15:46:58.541934+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-16T15:46:58.565500+0000', 'last_undegraded': '2024-09-16T15:46:58.565500+0000', 'last_fullsized': '2024-09-16T15:46:58.565500+0000', 'mapping_epoch': 68, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 68, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T15:46:58.541934+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T15:46:58.541934+0000', 'last_clean_scrub_stamp': '2024-09-16T15:46:58.541934+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0], 'acting': [3, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T15:47:04.362 INFO:tasks.ceph.ceph_manager.ceph:PG 11.3c is not active+clean 2024-09-16T15:47:04.362 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '11.3c', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 68, 'state': 'creating+peering', 'last_fresh': '2024-09-16T15:46:58.560961+0000', 'last_change': '2024-09-16T15:46:58.560961+0000', 'last_active': '2024-09-16T15:46:58.541934+0000', 'last_peered': '2024-09-16T15:46:58.541934+0000', 'last_clean': '2024-09-16T15:46:58.541934+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-16T15:46:58.560961+0000', 'last_undegraded': '2024-09-16T15:46:58.560961+0000', 'last_fullsized': '2024-09-16T15:46:58.560961+0000', 'mapping_epoch': 68, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 68, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T15:46:58.541934+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T15:46:58.541934+0000', 'last_clean_scrub_stamp': '2024-09-16T15:46:58.541934+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 6], 'acting': [1, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-09-16T15:47:04.362 INFO:tasks.ceph.ceph_manager.ceph:PG 11.3b is not active+clean 2024-09-16T15:47:04.362 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '11.3b', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 68, 'state': 'creating+peering', 'last_fresh': '2024-09-16T15:46:58.561715+0000', 'last_change': '2024-09-16T15:46:58.561715+0000', 'last_active': '2024-09-16T15:46:58.541934+0000', 'last_peered': '2024-09-16T15:46:58.541934+0000', 'last_clean': '2024-09-16T15:46:58.541934+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-16T15:46:58.561715+0000', 'last_undegraded': '2024-09-16T15:46:58.561715+0000', 'last_fullsized': '2024-09-16T15:46:58.561715+0000', 'mapping_epoch': 68, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 68, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T15:46:58.541934+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T15:46:58.541934+0000', 'last_clean_scrub_stamp': '2024-09-16T15:46:58.541934+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 7], 'acting': [2, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T15:47:04.362 INFO:tasks.ceph.ceph_manager.ceph:PG 11.3a is not active+clean 2024-09-16T15:47:04.362 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '11.3a', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 68, 'state': 'creating+peering', 'last_fresh': '2024-09-16T15:46:58.578627+0000', 'last_change': '2024-09-16T15:46:58.578627+0000', 'last_active': '2024-09-16T15:46:58.541934+0000', 'last_peered': '2024-09-16T15:46:58.541934+0000', 'last_clean': '2024-09-16T15:46:58.541934+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-16T15:46:58.578627+0000', 'last_undegraded': '2024-09-16T15:46:58.578627+0000', 'last_fullsized': '2024-09-16T15:46:58.578627+0000', 'mapping_epoch': 68, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 68, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T15:46:58.541934+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T15:46:58.541934+0000', 'last_clean_scrub_stamp': '2024-09-16T15:46:58.541934+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 4], 'acting': [6, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-09-16T15:47:04.363 INFO:tasks.ceph.ceph_manager.ceph:PG 11.39 is not active+clean 2024-09-16T15:47:04.363 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '11.39', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 68, 'state': 'creating+peering', 'last_fresh': '2024-09-16T15:46:58.570892+0000', 'last_change': '2024-09-16T15:46:58.570892+0000', 'last_active': '2024-09-16T15:46:58.541934+0000', 'last_peered': '2024-09-16T15:46:58.541934+0000', 'last_clean': '2024-09-16T15:46:58.541934+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-16T15:46:58.570892+0000', 'last_undegraded': '2024-09-16T15:46:58.570892+0000', 'last_fullsized': '2024-09-16T15:46:58.570892+0000', 'mapping_epoch': 68, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 68, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T15:46:58.541934+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T15:46:58.541934+0000', 'last_clean_scrub_stamp': '2024-09-16T15:46:58.541934+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 1], 'acting': [5, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-09-16T15:47:04.363 INFO:tasks.ceph.ceph_manager.ceph:PG 11.38 is not active+clean 2024-09-16T15:47:04.363 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '11.38', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 68, 'state': 'creating+peering', 'last_fresh': '2024-09-16T15:46:58.584780+0000', 'last_change': '2024-09-16T15:46:58.584780+0000', 'last_active': '2024-09-16T15:46:58.541934+0000', 'last_peered': '2024-09-16T15:46:58.541934+0000', 'last_clean': '2024-09-16T15:46:58.541934+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-16T15:46:58.584780+0000', 'last_undegraded': '2024-09-16T15:46:58.584780+0000', 'last_fullsized': '2024-09-16T15:46:58.584780+0000', 'mapping_epoch': 68, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 68, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T15:46:58.541934+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T15:46:58.541934+0000', 'last_clean_scrub_stamp': '2024-09-16T15:46:58.541934+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 1], 'acting': [6, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-09-16T15:47:04.363 INFO:tasks.ceph.ceph_manager.ceph:PG 11.37 is not active+clean 2024-09-16T15:47:04.363 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '11.37', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 68, 'state': 'creating+peering', 'last_fresh': '2024-09-16T15:46:58.561912+0000', 'last_change': '2024-09-16T15:46:58.561912+0000', 'last_active': '2024-09-16T15:46:58.541934+0000', 'last_peered': '2024-09-16T15:46:58.541934+0000', 'last_clean': '2024-09-16T15:46:58.541934+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-16T15:46:58.561912+0000', 'last_undegraded': '2024-09-16T15:46:58.561912+0000', 'last_fullsized': '2024-09-16T15:46:58.561912+0000', 'mapping_epoch': 68, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 68, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T15:46:58.541934+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T15:46:58.541934+0000', 'last_clean_scrub_stamp': '2024-09-16T15:46:58.541934+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T15:47:04.363 INFO:tasks.ceph.ceph_manager.ceph:PG 11.36 is not active+clean 2024-09-16T15:47:04.363 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '11.36', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 68, 'state': 'creating+peering', 'last_fresh': '2024-09-16T15:46:58.562601+0000', 'last_change': '2024-09-16T15:46:58.562601+0000', 'last_active': '2024-09-16T15:46:58.541934+0000', 'last_peered': '2024-09-16T15:46:58.541934+0000', 'last_clean': '2024-09-16T15:46:58.541934+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-16T15:46:58.562601+0000', 'last_undegraded': '2024-09-16T15:46:58.562601+0000', 'last_fullsized': '2024-09-16T15:46:58.562601+0000', 'mapping_epoch': 68, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 68, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T15:46:58.541934+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T15:46:58.541934+0000', 'last_clean_scrub_stamp': '2024-09-16T15:46:58.541934+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 2], 'acting': [4, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-09-16T15:47:04.363 INFO:tasks.ceph.ceph_manager.ceph:PG 11.35 is not active+clean 2024-09-16T15:47:04.364 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '11.35', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 68, 'state': 'creating+peering', 'last_fresh': '2024-09-16T15:46:58.580321+0000', 'last_change': '2024-09-16T15:46:58.580321+0000', 'last_active': '2024-09-16T15:46:58.541934+0000', 'last_peered': '2024-09-16T15:46:58.541934+0000', 'last_clean': '2024-09-16T15:46:58.541934+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-16T15:46:58.580321+0000', 'last_undegraded': '2024-09-16T15:46:58.580321+0000', 'last_fullsized': '2024-09-16T15:46:58.580321+0000', 'mapping_epoch': 68, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 68, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T15:46:58.541934+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T15:46:58.541934+0000', 'last_clean_scrub_stamp': '2024-09-16T15:46:58.541934+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 3], 'acting': [7, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-09-16T15:47:04.364 INFO:tasks.ceph.ceph_manager.ceph:PG 11.34 is not active+clean 2024-09-16T15:47:04.364 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '11.34', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 68, 'state': 'creating+peering', 'last_fresh': '2024-09-16T15:46:58.560127+0000', 'last_change': '2024-09-16T15:46:58.560127+0000', 'last_active': '2024-09-16T15:46:58.541934+0000', 'last_peered': '2024-09-16T15:46:58.541934+0000', 'last_clean': '2024-09-16T15:46:58.541934+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-16T15:46:58.560127+0000', 'last_undegraded': '2024-09-16T15:46:58.560127+0000', 'last_fullsized': '2024-09-16T15:46:58.560127+0000', 'mapping_epoch': 68, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 68, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T15:46:58.541934+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T15:46:58.541934+0000', 'last_clean_scrub_stamp': '2024-09-16T15:46:58.541934+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [1, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-09-16T15:47:04.364 INFO:tasks.ceph.ceph_manager.ceph:PG 11.33 is not active+clean 2024-09-16T15:47:04.364 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '11.33', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 68, 'state': 'creating+peering', 'last_fresh': '2024-09-16T15:46:58.562521+0000', 'last_change': '2024-09-16T15:46:58.562521+0000', 'last_active': '2024-09-16T15:46:58.541934+0000', 'last_peered': '2024-09-16T15:46:58.541934+0000', 'last_clean': '2024-09-16T15:46:58.541934+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-16T15:46:58.562521+0000', 'last_undegraded': '2024-09-16T15:46:58.562521+0000', 'last_fullsized': '2024-09-16T15:46:58.562521+0000', 'mapping_epoch': 68, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 68, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T15:46:58.541934+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T15:46:58.541934+0000', 'last_clean_scrub_stamp': '2024-09-16T15:46:58.541934+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 6], 'acting': [0, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T15:47:04.364 INFO:tasks.ceph.ceph_manager.ceph:PG 11.32 is not active+clean 2024-09-16T15:47:04.364 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '11.32', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 68, 'state': 'creating+peering', 'last_fresh': '2024-09-16T15:46:58.561306+0000', 'last_change': '2024-09-16T15:46:58.561306+0000', 'last_active': '2024-09-16T15:46:58.541934+0000', 'last_peered': '2024-09-16T15:46:58.541934+0000', 'last_clean': '2024-09-16T15:46:58.541934+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-16T15:46:58.561306+0000', 'last_undegraded': '2024-09-16T15:46:58.561306+0000', 'last_fullsized': '2024-09-16T15:46:58.561306+0000', 'mapping_epoch': 68, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 68, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T15:46:58.541934+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T15:46:58.541934+0000', 'last_clean_scrub_stamp': '2024-09-16T15:46:58.541934+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 5], 'acting': [2, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T15:47:04.364 INFO:tasks.ceph.ceph_manager.ceph:PG 11.31 is not active+clean 2024-09-16T15:47:04.365 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '11.31', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 68, 'state': 'creating+peering', 'last_fresh': '2024-09-16T15:46:58.566448+0000', 'last_change': '2024-09-16T15:46:58.566448+0000', 'last_active': '2024-09-16T15:46:58.541934+0000', 'last_peered': '2024-09-16T15:46:58.541934+0000', 'last_clean': '2024-09-16T15:46:58.541934+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-16T15:46:58.566448+0000', 'last_undegraded': '2024-09-16T15:46:58.566448+0000', 'last_fullsized': '2024-09-16T15:46:58.566448+0000', 'mapping_epoch': 68, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 68, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T15:46:58.541934+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T15:46:58.541934+0000', 'last_clean_scrub_stamp': '2024-09-16T15:46:58.541934+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 2], 'acting': [0, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T15:47:04.365 INFO:tasks.ceph.ceph_manager.ceph:PG 11.30 is not active+clean 2024-09-16T15:47:04.365 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '11.30', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 68, 'state': 'creating+peering', 'last_fresh': '2024-09-16T15:46:58.560118+0000', 'last_change': '2024-09-16T15:46:58.560118+0000', 'last_active': '2024-09-16T15:46:58.541934+0000', 'last_peered': '2024-09-16T15:46:58.541934+0000', 'last_clean': '2024-09-16T15:46:58.541934+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-16T15:46:58.560118+0000', 'last_undegraded': '2024-09-16T15:46:58.560118+0000', 'last_fullsized': '2024-09-16T15:46:58.560118+0000', 'mapping_epoch': 68, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 68, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T15:46:58.541934+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T15:46:58.541934+0000', 'last_clean_scrub_stamp': '2024-09-16T15:46:58.541934+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 6], 'acting': [4, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-09-16T15:47:04.365 INFO:tasks.ceph.ceph_manager.ceph:PG 11.2f is not active+clean 2024-09-16T15:47:04.365 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '11.2f', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 68, 'state': 'creating+peering', 'last_fresh': '2024-09-16T15:46:58.561316+0000', 'last_change': '2024-09-16T15:46:58.561316+0000', 'last_active': '2024-09-16T15:46:58.541934+0000', 'last_peered': '2024-09-16T15:46:58.541934+0000', 'last_clean': '2024-09-16T15:46:58.541934+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-16T15:46:58.561316+0000', 'last_undegraded': '2024-09-16T15:46:58.561316+0000', 'last_fullsized': '2024-09-16T15:46:58.561316+0000', 'mapping_epoch': 68, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 68, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T15:46:58.541934+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T15:46:58.541934+0000', 'last_clean_scrub_stamp': '2024-09-16T15:46:58.541934+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T15:47:04.365 INFO:tasks.ceph.ceph_manager.ceph:PG 11.2e is not active+clean 2024-09-16T15:47:04.365 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '11.2e', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 68, 'state': 'creating+peering', 'last_fresh': '2024-09-16T15:46:58.559376+0000', 'last_change': '2024-09-16T15:46:58.559376+0000', 'last_active': '2024-09-16T15:46:58.541934+0000', 'last_peered': '2024-09-16T15:46:58.541934+0000', 'last_clean': '2024-09-16T15:46:58.541934+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-16T15:46:58.559376+0000', 'last_undegraded': '2024-09-16T15:46:58.559376+0000', 'last_fullsized': '2024-09-16T15:46:58.559376+0000', 'mapping_epoch': 68, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 68, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T15:46:58.541934+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T15:46:58.541934+0000', 'last_clean_scrub_stamp': '2024-09-16T15:46:58.541934+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 6], 'acting': [1, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-09-16T15:47:04.365 INFO:tasks.ceph.ceph_manager.ceph:PG 11.2d is not active+clean 2024-09-16T15:47:04.365 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '11.2d', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 68, 'state': 'creating+peering', 'last_fresh': '2024-09-16T15:46:58.580101+0000', 'last_change': '2024-09-16T15:46:58.580101+0000', 'last_active': '2024-09-16T15:46:58.541934+0000', 'last_peered': '2024-09-16T15:46:58.541934+0000', 'last_clean': '2024-09-16T15:46:58.541934+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-16T15:46:58.580101+0000', 'last_undegraded': '2024-09-16T15:46:58.580101+0000', 'last_fullsized': '2024-09-16T15:46:58.580101+0000', 'mapping_epoch': 68, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 68, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T15:46:58.541934+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T15:46:58.541934+0000', 'last_clean_scrub_stamp': '2024-09-16T15:46:58.541934+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 1], 'acting': [6, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-09-16T15:47:04.365 INFO:tasks.ceph.ceph_manager.ceph:PG 11.2c is not active+clean 2024-09-16T15:47:04.366 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '11.2c', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 68, 'state': 'creating+peering', 'last_fresh': '2024-09-16T15:46:58.562539+0000', 'last_change': '2024-09-16T15:46:58.562539+0000', 'last_active': '2024-09-16T15:46:58.541934+0000', 'last_peered': '2024-09-16T15:46:58.541934+0000', 'last_clean': '2024-09-16T15:46:58.541934+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-16T15:46:58.562539+0000', 'last_undegraded': '2024-09-16T15:46:58.562539+0000', 'last_fullsized': '2024-09-16T15:46:58.562539+0000', 'mapping_epoch': 68, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 68, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T15:46:58.541934+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T15:46:58.541934+0000', 'last_clean_scrub_stamp': '2024-09-16T15:46:58.541934+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 5], 'acting': [0, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T15:47:04.366 INFO:tasks.ceph.ceph_manager.ceph:PG 11.2b is not active+clean 2024-09-16T15:47:04.366 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '11.2b', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 68, 'state': 'creating+peering', 'last_fresh': '2024-09-16T15:46:58.564602+0000', 'last_change': '2024-09-16T15:46:58.564602+0000', 'last_active': '2024-09-16T15:46:58.541934+0000', 'last_peered': '2024-09-16T15:46:58.541934+0000', 'last_clean': '2024-09-16T15:46:58.541934+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-16T15:46:58.564602+0000', 'last_undegraded': '2024-09-16T15:46:58.564602+0000', 'last_fullsized': '2024-09-16T15:46:58.564602+0000', 'mapping_epoch': 68, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 68, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T15:46:58.541934+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T15:46:58.541934+0000', 'last_clean_scrub_stamp': '2024-09-16T15:46:58.541934+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 0], 'acting': [4, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-09-16T15:47:04.366 INFO:tasks.ceph.ceph_manager.ceph:PG 11.2a is not active+clean 2024-09-16T15:47:04.366 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '11.2a', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 68, 'state': 'creating+peering', 'last_fresh': '2024-09-16T15:46:58.562954+0000', 'last_change': '2024-09-16T15:46:58.562954+0000', 'last_active': '2024-09-16T15:46:58.541934+0000', 'last_peered': '2024-09-16T15:46:58.541934+0000', 'last_clean': '2024-09-16T15:46:58.541934+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-16T15:46:58.562954+0000', 'last_undegraded': '2024-09-16T15:46:58.562954+0000', 'last_fullsized': '2024-09-16T15:46:58.562954+0000', 'mapping_epoch': 68, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 68, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T15:46:58.541934+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T15:46:58.541934+0000', 'last_clean_scrub_stamp': '2024-09-16T15:46:58.541934+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 2], 'acting': [4, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-09-16T15:47:04.366 INFO:tasks.ceph.ceph_manager.ceph:PG 11.29 is not active+clean 2024-09-16T15:47:04.366 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '11.29', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 68, 'state': 'creating+peering', 'last_fresh': '2024-09-16T15:46:58.578047+0000', 'last_change': '2024-09-16T15:46:58.578047+0000', 'last_active': '2024-09-16T15:46:58.541934+0000', 'last_peered': '2024-09-16T15:46:58.541934+0000', 'last_clean': '2024-09-16T15:46:58.541934+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-16T15:46:58.578047+0000', 'last_undegraded': '2024-09-16T15:46:58.578047+0000', 'last_fullsized': '2024-09-16T15:46:58.578047+0000', 'mapping_epoch': 68, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 68, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T15:46:58.541934+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T15:46:58.541934+0000', 'last_clean_scrub_stamp': '2024-09-16T15:46:58.541934+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 5], 'acting': [7, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-09-16T15:47:04.366 INFO:tasks.ceph.ceph_manager.ceph:PG 11.28 is not active+clean 2024-09-16T15:47:04.366 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '11.28', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 68, 'state': 'creating+peering', 'last_fresh': '2024-09-16T15:46:58.575628+0000', 'last_change': '2024-09-16T15:46:58.575628+0000', 'last_active': '2024-09-16T15:46:58.541934+0000', 'last_peered': '2024-09-16T15:46:58.541934+0000', 'last_clean': '2024-09-16T15:46:58.541934+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-16T15:46:58.575628+0000', 'last_undegraded': '2024-09-16T15:46:58.575628+0000', 'last_fullsized': '2024-09-16T15:46:58.575628+0000', 'mapping_epoch': 68, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 68, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T15:46:58.541934+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T15:46:58.541934+0000', 'last_clean_scrub_stamp': '2024-09-16T15:46:58.541934+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 6], 'acting': [7, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-09-16T15:47:04.366 INFO:tasks.ceph.ceph_manager.ceph:PG 11.27 is not active+clean 2024-09-16T15:47:04.367 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '11.27', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 68, 'state': 'creating+peering', 'last_fresh': '2024-09-16T15:46:58.562122+0000', 'last_change': '2024-09-16T15:46:58.562122+0000', 'last_active': '2024-09-16T15:46:58.541934+0000', 'last_peered': '2024-09-16T15:46:58.541934+0000', 'last_clean': '2024-09-16T15:46:58.541934+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-16T15:46:58.562122+0000', 'last_undegraded': '2024-09-16T15:46:58.562122+0000', 'last_fullsized': '2024-09-16T15:46:58.562122+0000', 'mapping_epoch': 68, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 68, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T15:46:58.541934+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T15:46:58.541934+0000', 'last_clean_scrub_stamp': '2024-09-16T15:46:58.541934+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 7], 'acting': [4, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-09-16T15:47:04.367 INFO:tasks.ceph.ceph_manager.ceph:PG 11.26 is not active+clean 2024-09-16T15:47:04.367 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '11.26', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 68, 'state': 'creating+peering', 'last_fresh': '2024-09-16T15:46:58.571812+0000', 'last_change': '2024-09-16T15:46:58.571812+0000', 'last_active': '2024-09-16T15:46:58.541934+0000', 'last_peered': '2024-09-16T15:46:58.541934+0000', 'last_clean': '2024-09-16T15:46:58.541934+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-16T15:46:58.571812+0000', 'last_undegraded': '2024-09-16T15:46:58.571812+0000', 'last_fullsized': '2024-09-16T15:46:58.571812+0000', 'mapping_epoch': 68, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 68, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T15:46:58.541934+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T15:46:58.541934+0000', 'last_clean_scrub_stamp': '2024-09-16T15:46:58.541934+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 6], 'acting': [5, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-09-16T15:47:04.367 INFO:tasks.ceph.ceph_manager.ceph:PG 11.25 is not active+clean 2024-09-16T15:47:04.367 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '11.25', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 68, 'state': 'creating+peering', 'last_fresh': '2024-09-16T15:46:58.564846+0000', 'last_change': '2024-09-16T15:46:58.564846+0000', 'last_active': '2024-09-16T15:46:58.541934+0000', 'last_peered': '2024-09-16T15:46:58.541934+0000', 'last_clean': '2024-09-16T15:46:58.541934+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-16T15:46:58.564846+0000', 'last_undegraded': '2024-09-16T15:46:58.564846+0000', 'last_fullsized': '2024-09-16T15:46:58.564846+0000', 'mapping_epoch': 68, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 68, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T15:46:58.541934+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T15:46:58.541934+0000', 'last_clean_scrub_stamp': '2024-09-16T15:46:58.541934+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 7], 'acting': [3, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T15:47:04.367 INFO:tasks.ceph.ceph_manager.ceph:PG 11.24 is not active+clean 2024-09-16T15:47:04.367 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '11.24', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 68, 'state': 'creating+peering', 'last_fresh': '2024-09-16T15:46:58.560367+0000', 'last_change': '2024-09-16T15:46:58.560367+0000', 'last_active': '2024-09-16T15:46:58.541934+0000', 'last_peered': '2024-09-16T15:46:58.541934+0000', 'last_clean': '2024-09-16T15:46:58.541934+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-16T15:46:58.560367+0000', 'last_undegraded': '2024-09-16T15:46:58.560367+0000', 'last_fullsized': '2024-09-16T15:46:58.560367+0000', 'mapping_epoch': 68, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 68, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T15:46:58.541934+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T15:46:58.541934+0000', 'last_clean_scrub_stamp': '2024-09-16T15:46:58.541934+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 2], 'acting': [4, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-09-16T15:47:04.368 INFO:tasks.ceph.ceph_manager.ceph:PG 11.23 is not active+clean 2024-09-16T15:47:04.368 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '11.23', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 68, 'state': 'creating+peering', 'last_fresh': '2024-09-16T15:46:58.568304+0000', 'last_change': '2024-09-16T15:46:58.568304+0000', 'last_active': '2024-09-16T15:46:58.541934+0000', 'last_peered': '2024-09-16T15:46:58.541934+0000', 'last_clean': '2024-09-16T15:46:58.541934+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-16T15:46:58.568304+0000', 'last_undegraded': '2024-09-16T15:46:58.568304+0000', 'last_fullsized': '2024-09-16T15:46:58.568304+0000', 'mapping_epoch': 68, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 68, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T15:46:58.541934+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T15:46:58.541934+0000', 'last_clean_scrub_stamp': '2024-09-16T15:46:58.541934+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 5], 'acting': [3, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T15:47:04.368 INFO:tasks.ceph.ceph_manager.ceph:PG 11.22 is not active+clean 2024-09-16T15:47:04.368 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '11.22', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 68, 'state': 'creating+peering', 'last_fresh': '2024-09-16T15:46:58.569453+0000', 'last_change': '2024-09-16T15:46:58.569453+0000', 'last_active': '2024-09-16T15:46:58.541934+0000', 'last_peered': '2024-09-16T15:46:58.541934+0000', 'last_clean': '2024-09-16T15:46:58.541934+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-16T15:46:58.569453+0000', 'last_undegraded': '2024-09-16T15:46:58.569453+0000', 'last_fullsized': '2024-09-16T15:46:58.569453+0000', 'mapping_epoch': 68, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 68, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T15:46:58.541934+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T15:46:58.541934+0000', 'last_clean_scrub_stamp': '2024-09-16T15:46:58.541934+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 6], 'acting': [5, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-09-16T15:47:04.368 INFO:tasks.ceph.ceph_manager.ceph:PG 11.21 is not active+clean 2024-09-16T15:47:04.368 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '11.21', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 68, 'state': 'creating+peering', 'last_fresh': '2024-09-16T15:46:58.569512+0000', 'last_change': '2024-09-16T15:46:58.569512+0000', 'last_active': '2024-09-16T15:46:58.541934+0000', 'last_peered': '2024-09-16T15:46:58.541934+0000', 'last_clean': '2024-09-16T15:46:58.541934+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-16T15:46:58.569512+0000', 'last_undegraded': '2024-09-16T15:46:58.569512+0000', 'last_fullsized': '2024-09-16T15:46:58.569512+0000', 'mapping_epoch': 68, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 68, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T15:46:58.541934+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T15:46:58.541934+0000', 'last_clean_scrub_stamp': '2024-09-16T15:46:58.541934+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 2], 'acting': [5, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-09-16T15:47:04.368 INFO:tasks.ceph.ceph_manager.ceph:PG 11.20 is not active+clean 2024-09-16T15:47:04.368 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '11.20', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 68, 'state': 'creating+peering', 'last_fresh': '2024-09-16T15:46:58.582821+0000', 'last_change': '2024-09-16T15:46:58.582821+0000', 'last_active': '2024-09-16T15:46:58.541934+0000', 'last_peered': '2024-09-16T15:46:58.541934+0000', 'last_clean': '2024-09-16T15:46:58.541934+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-16T15:46:58.582821+0000', 'last_undegraded': '2024-09-16T15:46:58.582821+0000', 'last_fullsized': '2024-09-16T15:46:58.582821+0000', 'mapping_epoch': 68, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 68, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T15:46:58.541934+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T15:46:58.541934+0000', 'last_clean_scrub_stamp': '2024-09-16T15:46:58.541934+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 3], 'acting': [6, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-09-16T15:47:04.368 INFO:tasks.ceph.ceph_manager.ceph:PG 11.1e is not active+clean 2024-09-16T15:47:04.368 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '11.1e', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 68, 'state': 'creating+peering', 'last_fresh': '2024-09-16T15:46:58.575147+0000', 'last_change': '2024-09-16T15:46:58.575147+0000', 'last_active': '2024-09-16T15:46:58.541934+0000', 'last_peered': '2024-09-16T15:46:58.541934+0000', 'last_clean': '2024-09-16T15:46:58.541934+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-16T15:46:58.575147+0000', 'last_undegraded': '2024-09-16T15:46:58.575147+0000', 'last_fullsized': '2024-09-16T15:46:58.575147+0000', 'mapping_epoch': 68, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 68, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T15:46:58.541934+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T15:46:58.541934+0000', 'last_clean_scrub_stamp': '2024-09-16T15:46:58.541934+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 6], 'acting': [7, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-09-16T15:47:04.369 INFO:tasks.ceph.ceph_manager.ceph:PG 11.1f is not active+clean 2024-09-16T15:47:04.369 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '11.1f', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 68, 'state': 'creating+peering', 'last_fresh': '2024-09-16T15:46:58.560943+0000', 'last_change': '2024-09-16T15:46:58.560943+0000', 'last_active': '2024-09-16T15:46:58.541934+0000', 'last_peered': '2024-09-16T15:46:58.541934+0000', 'last_clean': '2024-09-16T15:46:58.541934+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-16T15:46:58.560943+0000', 'last_undegraded': '2024-09-16T15:46:58.560943+0000', 'last_fullsized': '2024-09-16T15:46:58.560943+0000', 'mapping_epoch': 68, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 68, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T15:46:58.541934+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T15:46:58.541934+0000', 'last_clean_scrub_stamp': '2024-09-16T15:46:58.541934+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T15:47:04.369 INFO:tasks.ceph.ceph_manager.ceph:PG 11.1c is not active+clean 2024-09-16T15:47:04.369 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '11.1c', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 68, 'state': 'creating+peering', 'last_fresh': '2024-09-16T15:46:58.583638+0000', 'last_change': '2024-09-16T15:46:58.583638+0000', 'last_active': '2024-09-16T15:46:58.541934+0000', 'last_peered': '2024-09-16T15:46:58.541934+0000', 'last_clean': '2024-09-16T15:46:58.541934+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-16T15:46:58.583638+0000', 'last_undegraded': '2024-09-16T15:46:58.583638+0000', 'last_fullsized': '2024-09-16T15:46:58.583638+0000', 'mapping_epoch': 68, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 68, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T15:46:58.541934+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T15:46:58.541934+0000', 'last_clean_scrub_stamp': '2024-09-16T15:46:58.541934+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 7], 'acting': [6, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-09-16T15:47:04.369 INFO:tasks.ceph.ceph_manager.ceph:PG 11.1d is not active+clean 2024-09-16T15:47:04.369 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '11.1d', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 68, 'state': 'creating+peering', 'last_fresh': '2024-09-16T15:46:58.579199+0000', 'last_change': '2024-09-16T15:46:58.579199+0000', 'last_active': '2024-09-16T15:46:58.541934+0000', 'last_peered': '2024-09-16T15:46:58.541934+0000', 'last_clean': '2024-09-16T15:46:58.541934+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-16T15:46:58.579199+0000', 'last_undegraded': '2024-09-16T15:46:58.579199+0000', 'last_fullsized': '2024-09-16T15:46:58.579199+0000', 'mapping_epoch': 68, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 68, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T15:46:58.541934+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T15:46:58.541934+0000', 'last_clean_scrub_stamp': '2024-09-16T15:46:58.541934+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 0], 'acting': [7, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-09-16T15:47:04.369 INFO:tasks.ceph.ceph_manager.ceph:PG 11.1a is not active+clean 2024-09-16T15:47:04.369 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '11.1a', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 68, 'state': 'creating+peering', 'last_fresh': '2024-09-16T15:46:58.568314+0000', 'last_change': '2024-09-16T15:46:58.568314+0000', 'last_active': '2024-09-16T15:46:58.541934+0000', 'last_peered': '2024-09-16T15:46:58.541934+0000', 'last_clean': '2024-09-16T15:46:58.541934+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-16T15:46:58.568314+0000', 'last_undegraded': '2024-09-16T15:46:58.568314+0000', 'last_fullsized': '2024-09-16T15:46:58.568314+0000', 'mapping_epoch': 68, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 68, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T15:46:58.541934+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T15:46:58.541934+0000', 'last_clean_scrub_stamp': '2024-09-16T15:46:58.541934+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 3], 'acting': [5, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-09-16T15:47:04.369 INFO:tasks.ceph.ceph_manager.ceph:PG 11.1b is not active+clean 2024-09-16T15:47:04.370 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '11.1b', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 68, 'state': 'creating+peering', 'last_fresh': '2024-09-16T15:46:58.563604+0000', 'last_change': '2024-09-16T15:46:58.563604+0000', 'last_active': '2024-09-16T15:46:58.541934+0000', 'last_peered': '2024-09-16T15:46:58.541934+0000', 'last_clean': '2024-09-16T15:46:58.541934+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-16T15:46:58.563604+0000', 'last_undegraded': '2024-09-16T15:46:58.563604+0000', 'last_fullsized': '2024-09-16T15:46:58.563604+0000', 'mapping_epoch': 68, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 68, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T15:46:58.541934+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T15:46:58.541934+0000', 'last_clean_scrub_stamp': '2024-09-16T15:46:58.541934+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 7], 'acting': [4, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-09-16T15:47:04.370 INFO:tasks.ceph.ceph_manager.ceph:PG 11.18 is not active+clean 2024-09-16T15:47:04.370 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '11.18', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 68, 'state': 'creating+peering', 'last_fresh': '2024-09-16T15:46:58.562911+0000', 'last_change': '2024-09-16T15:46:58.562911+0000', 'last_active': '2024-09-16T15:46:58.541934+0000', 'last_peered': '2024-09-16T15:46:58.541934+0000', 'last_clean': '2024-09-16T15:46:58.541934+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-16T15:46:58.562911+0000', 'last_undegraded': '2024-09-16T15:46:58.562911+0000', 'last_fullsized': '2024-09-16T15:46:58.562911+0000', 'mapping_epoch': 68, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 68, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T15:46:58.541934+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T15:46:58.541934+0000', 'last_clean_scrub_stamp': '2024-09-16T15:46:58.541934+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 4], 'acting': [2, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T15:47:04.370 INFO:tasks.ceph.ceph_manager.ceph:PG 11.19 is not active+clean 2024-09-16T15:47:04.370 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '11.19', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 68, 'state': 'creating+peering', 'last_fresh': '2024-09-16T15:46:58.566064+0000', 'last_change': '2024-09-16T15:46:58.566064+0000', 'last_active': '2024-09-16T15:46:58.541934+0000', 'last_peered': '2024-09-16T15:46:58.541934+0000', 'last_clean': '2024-09-16T15:46:58.541934+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-16T15:46:58.566064+0000', 'last_undegraded': '2024-09-16T15:46:58.566064+0000', 'last_fullsized': '2024-09-16T15:46:58.566064+0000', 'mapping_epoch': 68, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 68, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T15:46:58.541934+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T15:46:58.541934+0000', 'last_clean_scrub_stamp': '2024-09-16T15:46:58.541934+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 7], 'acting': [0, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T15:47:04.370 INFO:tasks.ceph.ceph_manager.ceph:PG 11.16 is not active+clean 2024-09-16T15:47:04.370 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '11.16', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 68, 'state': 'creating+peering', 'last_fresh': '2024-09-16T15:46:58.570594+0000', 'last_change': '2024-09-16T15:46:58.570594+0000', 'last_active': '2024-09-16T15:46:58.541934+0000', 'last_peered': '2024-09-16T15:46:58.541934+0000', 'last_clean': '2024-09-16T15:46:58.541934+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-16T15:46:58.570594+0000', 'last_undegraded': '2024-09-16T15:46:58.570594+0000', 'last_fullsized': '2024-09-16T15:46:58.570594+0000', 'mapping_epoch': 68, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 68, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T15:46:58.541934+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T15:46:58.541934+0000', 'last_clean_scrub_stamp': '2024-09-16T15:46:58.541934+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 7], 'acting': [5, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-09-16T15:47:04.370 INFO:tasks.ceph.ceph_manager.ceph:PG 11.17 is not active+clean 2024-09-16T15:47:04.370 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '11.17', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 68, 'state': 'creating+peering', 'last_fresh': '2024-09-16T15:46:58.579574+0000', 'last_change': '2024-09-16T15:46:58.579574+0000', 'last_active': '2024-09-16T15:46:58.541934+0000', 'last_peered': '2024-09-16T15:46:58.541934+0000', 'last_clean': '2024-09-16T15:46:58.541934+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-16T15:46:58.579574+0000', 'last_undegraded': '2024-09-16T15:46:58.579574+0000', 'last_fullsized': '2024-09-16T15:46:58.579574+0000', 'mapping_epoch': 68, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 68, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T15:46:58.541934+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T15:46:58.541934+0000', 'last_clean_scrub_stamp': '2024-09-16T15:46:58.541934+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 6], 'acting': [7, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-09-16T15:47:04.371 INFO:tasks.ceph.ceph_manager.ceph:PG 11.14 is not active+clean 2024-09-16T15:47:04.371 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '11.14', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 68, 'state': 'creating+peering', 'last_fresh': '2024-09-16T15:46:58.582525+0000', 'last_change': '2024-09-16T15:46:58.582525+0000', 'last_active': '2024-09-16T15:46:58.541934+0000', 'last_peered': '2024-09-16T15:46:58.541934+0000', 'last_clean': '2024-09-16T15:46:58.541934+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-16T15:46:58.582525+0000', 'last_undegraded': '2024-09-16T15:46:58.582525+0000', 'last_fullsized': '2024-09-16T15:46:58.582525+0000', 'mapping_epoch': 68, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 68, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T15:46:58.541934+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T15:46:58.541934+0000', 'last_clean_scrub_stamp': '2024-09-16T15:46:58.541934+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 4], 'acting': [6, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-09-16T15:47:04.371 INFO:tasks.ceph.ceph_manager.ceph:PG 11.15 is not active+clean 2024-09-16T15:47:04.371 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '11.15', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 68, 'state': 'creating+peering', 'last_fresh': '2024-09-16T15:46:58.563631+0000', 'last_change': '2024-09-16T15:46:58.563631+0000', 'last_active': '2024-09-16T15:46:58.541934+0000', 'last_peered': '2024-09-16T15:46:58.541934+0000', 'last_clean': '2024-09-16T15:46:58.541934+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-16T15:46:58.563631+0000', 'last_undegraded': '2024-09-16T15:46:58.563631+0000', 'last_fullsized': '2024-09-16T15:46:58.563631+0000', 'mapping_epoch': 68, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 68, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T15:46:58.541934+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T15:46:58.541934+0000', 'last_clean_scrub_stamp': '2024-09-16T15:46:58.541934+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T15:47:04.371 INFO:tasks.ceph.ceph_manager.ceph:PG 11.12 is not active+clean 2024-09-16T15:47:04.371 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '11.12', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 68, 'state': 'creating+peering', 'last_fresh': '2024-09-16T15:46:58.574297+0000', 'last_change': '2024-09-16T15:46:58.574297+0000', 'last_active': '2024-09-16T15:46:58.541934+0000', 'last_peered': '2024-09-16T15:46:58.541934+0000', 'last_clean': '2024-09-16T15:46:58.541934+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-16T15:46:58.574297+0000', 'last_undegraded': '2024-09-16T15:46:58.574297+0000', 'last_fullsized': '2024-09-16T15:46:58.574297+0000', 'mapping_epoch': 68, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 68, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T15:46:58.541934+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T15:46:58.541934+0000', 'last_clean_scrub_stamp': '2024-09-16T15:46:58.541934+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 2], 'acting': [7, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-09-16T15:47:04.371 INFO:tasks.ceph.ceph_manager.ceph:PG 11.13 is not active+clean 2024-09-16T15:47:04.371 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '11.13', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 68, 'state': 'creating+peering', 'last_fresh': '2024-09-16T15:46:58.569492+0000', 'last_change': '2024-09-16T15:46:58.569492+0000', 'last_active': '2024-09-16T15:46:58.541934+0000', 'last_peered': '2024-09-16T15:46:58.541934+0000', 'last_clean': '2024-09-16T15:46:58.541934+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-16T15:46:58.569492+0000', 'last_undegraded': '2024-09-16T15:46:58.569492+0000', 'last_fullsized': '2024-09-16T15:46:58.569492+0000', 'mapping_epoch': 68, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 68, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T15:46:58.541934+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T15:46:58.541934+0000', 'last_clean_scrub_stamp': '2024-09-16T15:46:58.541934+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 4], 'acting': [5, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-09-16T15:47:04.371 INFO:tasks.ceph.ceph_manager.ceph:PG 11.10 is not active+clean 2024-09-16T15:47:04.371 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '11.10', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 68, 'state': 'creating+peering', 'last_fresh': '2024-09-16T15:46:58.562563+0000', 'last_change': '2024-09-16T15:46:58.562563+0000', 'last_active': '2024-09-16T15:46:58.541934+0000', 'last_peered': '2024-09-16T15:46:58.541934+0000', 'last_clean': '2024-09-16T15:46:58.541934+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-16T15:46:58.562563+0000', 'last_undegraded': '2024-09-16T15:46:58.562563+0000', 'last_fullsized': '2024-09-16T15:46:58.562563+0000', 'mapping_epoch': 68, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 68, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T15:46:58.541934+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T15:46:58.541934+0000', 'last_clean_scrub_stamp': '2024-09-16T15:46:58.541934+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 6], 'acting': [0, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T15:47:04.372 INFO:tasks.ceph.ceph_manager.ceph:PG 11.11 is not active+clean 2024-09-16T15:47:04.372 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '11.11', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 68, 'state': 'creating+peering', 'last_fresh': '2024-09-16T15:46:58.560818+0000', 'last_change': '2024-09-16T15:46:58.560818+0000', 'last_active': '2024-09-16T15:46:58.541934+0000', 'last_peered': '2024-09-16T15:46:58.541934+0000', 'last_clean': '2024-09-16T15:46:58.541934+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-16T15:46:58.560818+0000', 'last_undegraded': '2024-09-16T15:46:58.560818+0000', 'last_fullsized': '2024-09-16T15:46:58.560818+0000', 'mapping_epoch': 68, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 68, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T15:46:58.541934+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T15:46:58.541934+0000', 'last_clean_scrub_stamp': '2024-09-16T15:46:58.541934+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T15:47:04.372 INFO:tasks.ceph.ceph_manager.ceph:PG 11.e is not active+clean 2024-09-16T15:47:04.372 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '11.e', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 68, 'state': 'creating+peering', 'last_fresh': '2024-09-16T15:46:58.561429+0000', 'last_change': '2024-09-16T15:46:58.561429+0000', 'last_active': '2024-09-16T15:46:58.541934+0000', 'last_peered': '2024-09-16T15:46:58.541934+0000', 'last_clean': '2024-09-16T15:46:58.541934+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-16T15:46:58.561429+0000', 'last_undegraded': '2024-09-16T15:46:58.561429+0000', 'last_fullsized': '2024-09-16T15:46:58.561429+0000', 'mapping_epoch': 68, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 68, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T15:46:58.541934+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T15:46:58.541934+0000', 'last_clean_scrub_stamp': '2024-09-16T15:46:58.541934+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 1], 'acting': [4, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-09-16T15:47:04.372 INFO:tasks.ceph.ceph_manager.ceph:PG 11.f is not active+clean 2024-09-16T15:47:04.372 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '11.f', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 68, 'state': 'creating+peering', 'last_fresh': '2024-09-16T15:46:58.578428+0000', 'last_change': '2024-09-16T15:46:58.578428+0000', 'last_active': '2024-09-16T15:46:58.541934+0000', 'last_peered': '2024-09-16T15:46:58.541934+0000', 'last_clean': '2024-09-16T15:46:58.541934+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-16T15:46:58.578428+0000', 'last_undegraded': '2024-09-16T15:46:58.578428+0000', 'last_fullsized': '2024-09-16T15:46:58.578428+0000', 'mapping_epoch': 68, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 68, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T15:46:58.541934+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T15:46:58.541934+0000', 'last_clean_scrub_stamp': '2024-09-16T15:46:58.541934+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 1], 'acting': [7, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-09-16T15:47:04.372 INFO:tasks.ceph.ceph_manager.ceph:PG 11.c is not active+clean 2024-09-16T15:47:04.372 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '11.c', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 68, 'state': 'creating+peering', 'last_fresh': '2024-09-16T15:46:58.559398+0000', 'last_change': '2024-09-16T15:46:58.559398+0000', 'last_active': '2024-09-16T15:46:58.541934+0000', 'last_peered': '2024-09-16T15:46:58.541934+0000', 'last_clean': '2024-09-16T15:46:58.541934+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-16T15:46:58.559398+0000', 'last_undegraded': '2024-09-16T15:46:58.559398+0000', 'last_fullsized': '2024-09-16T15:46:58.559398+0000', 'mapping_epoch': 68, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 68, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T15:46:58.541934+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T15:46:58.541934+0000', 'last_clean_scrub_stamp': '2024-09-16T15:46:58.541934+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [1, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-09-16T15:47:04.372 INFO:tasks.ceph.ceph_manager.ceph:PG 11.d is not active+clean 2024-09-16T15:47:04.372 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '11.d', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 68, 'state': 'creating+peering', 'last_fresh': '2024-09-16T15:46:58.563246+0000', 'last_change': '2024-09-16T15:46:58.563246+0000', 'last_active': '2024-09-16T15:46:58.541934+0000', 'last_peered': '2024-09-16T15:46:58.541934+0000', 'last_clean': '2024-09-16T15:46:58.541934+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-16T15:46:58.563246+0000', 'last_undegraded': '2024-09-16T15:46:58.563246+0000', 'last_fullsized': '2024-09-16T15:46:58.563246+0000', 'mapping_epoch': 68, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 68, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T15:46:58.541934+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T15:46:58.541934+0000', 'last_clean_scrub_stamp': '2024-09-16T15:46:58.541934+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 4], 'acting': [2, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T15:47:04.373 INFO:tasks.ceph.ceph_manager.ceph:PG 11.b is not active+clean 2024-09-16T15:47:04.373 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '11.b', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 68, 'state': 'creating+peering', 'last_fresh': '2024-09-16T15:46:58.567734+0000', 'last_change': '2024-09-16T15:46:58.567734+0000', 'last_active': '2024-09-16T15:46:58.541934+0000', 'last_peered': '2024-09-16T15:46:58.541934+0000', 'last_clean': '2024-09-16T15:46:58.541934+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-16T15:46:58.567734+0000', 'last_undegraded': '2024-09-16T15:46:58.567734+0000', 'last_fullsized': '2024-09-16T15:46:58.567734+0000', 'mapping_epoch': 68, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 68, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T15:46:58.541934+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T15:46:58.541934+0000', 'last_clean_scrub_stamp': '2024-09-16T15:46:58.541934+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1], 'acting': [3, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T15:47:04.373 INFO:tasks.ceph.ceph_manager.ceph:PG 11.8 is not active+clean 2024-09-16T15:47:04.373 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '11.8', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 68, 'state': 'creating+peering', 'last_fresh': '2024-09-16T15:46:58.564210+0000', 'last_change': '2024-09-16T15:46:58.564210+0000', 'last_active': '2024-09-16T15:46:58.541934+0000', 'last_peered': '2024-09-16T15:46:58.541934+0000', 'last_clean': '2024-09-16T15:46:58.541934+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-16T15:46:58.564210+0000', 'last_undegraded': '2024-09-16T15:46:58.564210+0000', 'last_fullsized': '2024-09-16T15:46:58.564210+0000', 'mapping_epoch': 68, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 68, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T15:46:58.541934+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T15:46:58.541934+0000', 'last_clean_scrub_stamp': '2024-09-16T15:46:58.541934+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 6], 'acting': [3, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T15:47:04.373 INFO:tasks.ceph.ceph_manager.ceph:PG 11.9 is not active+clean 2024-09-16T15:47:04.373 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '11.9', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 68, 'state': 'creating+peering', 'last_fresh': '2024-09-16T15:46:58.564335+0000', 'last_change': '2024-09-16T15:46:58.564335+0000', 'last_active': '2024-09-16T15:46:58.541934+0000', 'last_peered': '2024-09-16T15:46:58.541934+0000', 'last_clean': '2024-09-16T15:46:58.541934+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-16T15:46:58.564335+0000', 'last_undegraded': '2024-09-16T15:46:58.564335+0000', 'last_fullsized': '2024-09-16T15:46:58.564335+0000', 'mapping_epoch': 68, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 68, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T15:46:58.541934+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T15:46:58.541934+0000', 'last_clean_scrub_stamp': '2024-09-16T15:46:58.541934+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 6], 'acting': [3, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T15:47:04.374 INFO:tasks.ceph.ceph_manager.ceph:PG 11.6 is not active+clean 2024-09-16T15:47:04.374 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '11.6', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 68, 'state': 'creating+peering', 'last_fresh': '2024-09-16T15:46:58.562758+0000', 'last_change': '2024-09-16T15:46:58.562758+0000', 'last_active': '2024-09-16T15:46:58.541934+0000', 'last_peered': '2024-09-16T15:46:58.541934+0000', 'last_clean': '2024-09-16T15:46:58.541934+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-16T15:46:58.562758+0000', 'last_undegraded': '2024-09-16T15:46:58.562758+0000', 'last_fullsized': '2024-09-16T15:46:58.562758+0000', 'mapping_epoch': 68, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 68, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T15:46:58.541934+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T15:46:58.541934+0000', 'last_clean_scrub_stamp': '2024-09-16T15:46:58.541934+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 2], 'acting': [0, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T15:47:04.374 INFO:tasks.ceph.ceph_manager.ceph:PG 11.7 is not active+clean 2024-09-16T15:47:04.374 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '11.7', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 68, 'state': 'creating+peering', 'last_fresh': '2024-09-16T15:46:58.569698+0000', 'last_change': '2024-09-16T15:46:58.569698+0000', 'last_active': '2024-09-16T15:46:58.541934+0000', 'last_peered': '2024-09-16T15:46:58.541934+0000', 'last_clean': '2024-09-16T15:46:58.541934+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-16T15:46:58.569698+0000', 'last_undegraded': '2024-09-16T15:46:58.569698+0000', 'last_fullsized': '2024-09-16T15:46:58.569698+0000', 'mapping_epoch': 68, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 68, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T15:46:58.541934+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T15:46:58.541934+0000', 'last_clean_scrub_stamp': '2024-09-16T15:46:58.541934+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 1], 'acting': [5, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-09-16T15:47:04.374 INFO:tasks.ceph.ceph_manager.ceph:PG 11.4 is not active+clean 2024-09-16T15:47:04.374 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '11.4', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 68, 'state': 'creating+peering', 'last_fresh': '2024-09-16T15:46:58.574408+0000', 'last_change': '2024-09-16T15:46:58.574408+0000', 'last_active': '2024-09-16T15:46:58.541934+0000', 'last_peered': '2024-09-16T15:46:58.541934+0000', 'last_clean': '2024-09-16T15:46:58.541934+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-16T15:46:58.574408+0000', 'last_undegraded': '2024-09-16T15:46:58.574408+0000', 'last_fullsized': '2024-09-16T15:46:58.574408+0000', 'mapping_epoch': 68, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 68, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T15:46:58.541934+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T15:46:58.541934+0000', 'last_clean_scrub_stamp': '2024-09-16T15:46:58.541934+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 6], 'acting': [7, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-09-16T15:47:04.374 INFO:tasks.ceph.ceph_manager.ceph:PG 11.5 is not active+clean 2024-09-16T15:47:04.374 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '11.5', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 68, 'state': 'creating+peering', 'last_fresh': '2024-09-16T15:46:58.578126+0000', 'last_change': '2024-09-16T15:46:58.578126+0000', 'last_active': '2024-09-16T15:46:58.541934+0000', 'last_peered': '2024-09-16T15:46:58.541934+0000', 'last_clean': '2024-09-16T15:46:58.541934+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-16T15:46:58.578126+0000', 'last_undegraded': '2024-09-16T15:46:58.578126+0000', 'last_fullsized': '2024-09-16T15:46:58.578126+0000', 'mapping_epoch': 68, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 68, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T15:46:58.541934+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T15:46:58.541934+0000', 'last_clean_scrub_stamp': '2024-09-16T15:46:58.541934+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 5], 'acting': [7, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-09-16T15:47:04.375 INFO:tasks.ceph.ceph_manager.ceph:PG 11.2 is not active+clean 2024-09-16T15:47:04.375 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '11.2', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 68, 'state': 'creating+peering', 'last_fresh': '2024-09-16T15:46:58.560788+0000', 'last_change': '2024-09-16T15:46:58.560788+0000', 'last_active': '2024-09-16T15:46:58.541934+0000', 'last_peered': '2024-09-16T15:46:58.541934+0000', 'last_clean': '2024-09-16T15:46:58.541934+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-16T15:46:58.560788+0000', 'last_undegraded': '2024-09-16T15:46:58.560788+0000', 'last_fullsized': '2024-09-16T15:46:58.560788+0000', 'mapping_epoch': 68, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 68, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T15:46:58.541934+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T15:46:58.541934+0000', 'last_clean_scrub_stamp': '2024-09-16T15:46:58.541934+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T15:47:04.375 INFO:tasks.ceph.ceph_manager.ceph:PG 11.3 is not active+clean 2024-09-16T15:47:04.375 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '11.3', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 68, 'state': 'creating+peering', 'last_fresh': '2024-09-16T15:46:58.586430+0000', 'last_change': '2024-09-16T15:46:58.586430+0000', 'last_active': '2024-09-16T15:46:58.541934+0000', 'last_peered': '2024-09-16T15:46:58.541934+0000', 'last_clean': '2024-09-16T15:46:58.541934+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-16T15:46:58.586430+0000', 'last_undegraded': '2024-09-16T15:46:58.586430+0000', 'last_fullsized': '2024-09-16T15:46:58.586430+0000', 'mapping_epoch': 68, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 68, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T15:46:58.541934+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T15:46:58.541934+0000', 'last_clean_scrub_stamp': '2024-09-16T15:46:58.541934+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 7], 'acting': [6, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-09-16T15:47:04.375 INFO:tasks.ceph.ceph_manager.ceph:PG 11.0 is not active+clean 2024-09-16T15:47:04.375 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '11.0', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 68, 'state': 'creating+peering', 'last_fresh': '2024-09-16T15:46:58.559237+0000', 'last_change': '2024-09-16T15:46:58.559237+0000', 'last_active': '2024-09-16T15:46:58.541934+0000', 'last_peered': '2024-09-16T15:46:58.541934+0000', 'last_clean': '2024-09-16T15:46:58.541934+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-16T15:46:58.559237+0000', 'last_undegraded': '2024-09-16T15:46:58.559237+0000', 'last_fullsized': '2024-09-16T15:46:58.559237+0000', 'mapping_epoch': 68, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 68, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T15:46:58.541934+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T15:46:58.541934+0000', 'last_clean_scrub_stamp': '2024-09-16T15:46:58.541934+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3], 'acting': [1, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-09-16T15:47:04.375 INFO:tasks.ceph.ceph_manager.ceph:PG 11.1 is not active+clean 2024-09-16T15:47:04.375 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '11.1', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 68, 'state': 'creating+peering', 'last_fresh': '2024-09-16T15:46:58.565520+0000', 'last_change': '2024-09-16T15:46:58.565520+0000', 'last_active': '2024-09-16T15:46:58.541934+0000', 'last_peered': '2024-09-16T15:46:58.541934+0000', 'last_clean': '2024-09-16T15:46:58.541934+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-16T15:46:58.565520+0000', 'last_undegraded': '2024-09-16T15:46:58.565520+0000', 'last_fullsized': '2024-09-16T15:46:58.565520+0000', 'mapping_epoch': 68, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 68, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T15:46:58.541934+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T15:46:58.541934+0000', 'last_clean_scrub_stamp': '2024-09-16T15:46:58.541934+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 3], 'acting': [6, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-09-16T15:47:04.375 INFO:tasks.ceph.ceph_manager.ceph:PG 11.a is not active+clean 2024-09-16T15:47:04.376 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '11.a', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 68, 'state': 'creating+peering', 'last_fresh': '2024-09-16T15:46:58.564305+0000', 'last_change': '2024-09-16T15:46:58.564305+0000', 'last_active': '2024-09-16T15:46:58.541934+0000', 'last_peered': '2024-09-16T15:46:58.541934+0000', 'last_clean': '2024-09-16T15:46:58.541934+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-16T15:46:58.564305+0000', 'last_undegraded': '2024-09-16T15:46:58.564305+0000', 'last_fullsized': '2024-09-16T15:46:58.564305+0000', 'mapping_epoch': 68, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 68, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T15:46:58.541934+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T15:46:58.541934+0000', 'last_clean_scrub_stamp': '2024-09-16T15:46:58.541934+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 6], 'acting': [3, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T15:47:04.376 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T15:47:04.518 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:04.512+0000 7f387b2fb640 1 -- 172.21.15.123:0/3739699869 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3874151400 msgr2=0x7f38740c5f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:47:04.518 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:04.512+0000 7f387b2fb640 1 --2- 172.21.15.123:0/3739699869 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3874151400 0x7f38740c5f20 secure :-1 s=READY pgs=792 cs=0 l=1 rev1=1 crypto rx=0x7f3870009840 tx=0x7f387001f010 comp rx=0 tx=0).stop 2024-09-16T15:47:04.518 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:04.512+0000 7f387b2fb640 1 -- 172.21.15.123:0/3739699869 shutdown_connections 2024-09-16T15:47:04.518 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:04.512+0000 7f387b2fb640 1 --2- 172.21.15.123:0/3739699869 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f38740ce010 0x7f38740ca200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:04.518 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:04.512+0000 7f387b2fb640 1 --2- 172.21.15.123:0/3739699869 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3874151400 0x7f38740c5f20 unknown :-1 s=CLOSED pgs=792 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:04.518 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:04.512+0000 7f387b2fb640 1 -- 172.21.15.123:0/3739699869 >> 172.21.15.123:0/3739699869 conn(0x7f38740cfa90 msgr2=0x7f38740cfea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:47:04.518 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:04.512+0000 7f387b2fb640 1 -- 172.21.15.123:0/3739699869 shutdown_connections 2024-09-16T15:47:04.518 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:04.512+0000 7f387b2fb640 1 -- 172.21.15.123:0/3739699869 wait complete. 2024-09-16T15:47:04.519 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:04.516+0000 7f387b2fb640 1 Processor -- start 2024-09-16T15:47:04.519 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:04.516+0000 7f387b2fb640 1 -- start start 2024-09-16T15:47:04.521 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:04.516+0000 7f387b2fb640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f38740cbb90 0x7f387414a9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:47:04.521 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:04.516+0000 7f387b2fb640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f38740ce010 0x7f3874166990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:47:04.521 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:04.516+0000 7f387b2fb640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f3874151400 0x7f38740c8c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:47:04.521 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:04.516+0000 7f387b2fb640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f387406e6e0 con 0x7f38740cbb90 2024-09-16T15:47:04.521 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:04.516+0000 7f387b2fb640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f38740cc2d0 con 0x7f38740ce010 2024-09-16T15:47:04.521 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:04.516+0000 7f387b2fb640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f387406e860 con 0x7f3874151400 2024-09-16T15:47:04.521 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:04.516+0000 7f3879097640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f38740cbb90 0x7f387414a9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:47:04.521 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:04.516+0000 7f3879097640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f38740cbb90 0x7f387414a9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:46872/0 (socket says 172.21.15.123:46872) 2024-09-16T15:47:04.521 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:04.516+0000 7f3879097640 1 -- 172.21.15.123:0/2494629942 learned_addr learned my addr 172.21.15.123:0/2494629942 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:47:04.521 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:04.516+0000 7f3879097640 1 -- 172.21.15.123:0/2494629942 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f3874151400 msgr2=0x7f38740c8c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:47:04.521 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:04.516+0000 7f3879097640 1 --2- 172.21.15.123:0/2494629942 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f3874151400 0x7f38740c8c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:04.522 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:04.516+0000 7f3879097640 1 -- 172.21.15.123:0/2494629942 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f38740ce010 msgr2=0x7f3874166990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:47:04.522 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:04.516+0000 7f3879097640 1 --2- 172.21.15.123:0/2494629942 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f38740ce010 0x7f3874166990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:04.522 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:04.516+0000 7f3879097640 1 -- 172.21.15.123:0/2494629942 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3874149f00 con 0x7f38740cbb90 2024-09-16T15:47:04.522 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:04.516+0000 7f3879097640 1 --2- 172.21.15.123:0/2494629942 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f38740cbb90 0x7f387414a9a0 secure :-1 s=READY pgs=793 cs=0 l=1 rev1=1 crypto rx=0x7f3870009930 tx=0x7f3870005d20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:47:04.522 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:04.516+0000 7f3869ffb640 1 -- 172.21.15.123:0/2494629942 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38700b5040 con 0x7f38740cbb90 2024-09-16T15:47:04.522 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:04.516+0000 7f3869ffb640 1 -- 172.21.15.123:0/2494629942 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f3870003600 con 0x7f38740cbb90 2024-09-16T15:47:04.522 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:04.516+0000 7f387b2fb640 1 -- 172.21.15.123:0/2494629942 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f38740c9a80 con 0x7f38740cbb90 2024-09-16T15:47:04.522 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:04.516+0000 7f3869ffb640 1 -- 172.21.15.123:0/2494629942 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38700039d0 con 0x7f38740cbb90 2024-09-16T15:47:04.522 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:04.516+0000 7f387b2fb640 1 -- 172.21.15.123:0/2494629942 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f387414a1a0 con 0x7f38740cbb90 2024-09-16T15:47:04.523 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:04.516+0000 7f387b2fb640 1 -- 172.21.15.123:0/2494629942 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f387406ed10 con 0x7f38740cbb90 2024-09-16T15:47:04.523 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:04.516+0000 7f3869ffb640 1 -- 172.21.15.123:0/2494629942 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 11) ==== 97386+0+0 (secure 0 0 0) 0x7f38700c0030 con 0x7f38740cbb90 2024-09-16T15:47:04.523 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:04.516+0000 7f3869ffb640 1 --2- 172.21.15.123:0/2494629942 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f3858073220 0x7f38580756e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:47:04.523 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:04.516+0000 7f3869ffb640 1 -- 172.21.15.123:0/2494629942 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(70..70 src has 1..70) ==== 5709+0+0 (secure 0 0 0) 0x7f3870131700 con 0x7f38740cbb90 2024-09-16T15:47:04.523 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:04.520+0000 7f3878896640 1 --2- 172.21.15.123:0/2494629942 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f3858073220 0x7f38580756e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:47:04.524 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:04.520+0000 7f3878896640 1 --2- 172.21.15.123:0/2494629942 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f3858073220 0x7f38580756e0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f3864007fd0 tx=0x7f38640078e0 comp rx=0 tx=0).ready entity=mgr.4135 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:47:04.527 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:04.524+0000 7f3869ffb640 1 -- 172.21.15.123:0/2494629942 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f38700c7020 con 0x7f38740cbb90 2024-09-16T15:47:04.528 INFO:tasks.ceph.mds.d.smithi184.stderr:2024-09-16T15:47:04.525+0000 7fcc85957640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-09-16T15:47:04.744 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:04.740+0000 7f387b2fb640 1 -- 172.21.15.123:0/2494629942 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "status", "format": "json"} v 0) -- 0x7f387415ba00 con 0x7f38740cbb90 2024-09-16T15:47:04.745 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:04.740+0000 7f3869ffb640 1 -- 172.21.15.123:0/2494629942 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json"}]=0 v0) ==== 72+0+1076 (secure 0 0 0) 0x7f3870018b70 con 0x7f38740cbb90 2024-09-16T15:47:04.745 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-09-16T15:47:04.745 INFO:teuthology.orchestra.run.smithi123.stdout:{"fsid":"56077f89-18c8-486f-ae7e-eecc0bb7822c","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":278,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":70,"num_osds":8,"num_up_osds":8,"osd_up_since":1726501358,"num_in_osds":8,"osd_in_since":1726501349,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"creating+peering","count":64},{"state_name":"active+clean","count":33}],"num_pgs":97,"num_pools":3,"num_objects":2,"data_bytes":590368,"bytes_used":246214656,"bytes_avail":772847898624,"bytes_total":773094113280,"inactive_pgs_ratio":0.65979379415512085},"fsmap":{"epoch":44,"btime":"2024-09-16T15:47:02:606606+0000","id":5,"up":1,"in":1,"max":1,"by_rank":[{"filesystem_id":5,"rank":0,"name":"d","status":"up:active","gid":6216}],"up:standby":3},"mgrmap":{"available":true,"num_standbys":1,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-16T15:45:23.297865+0000","services":{}},"progress_events":{}} 2024-09-16T15:47:04.747 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:04.744+0000 7f387b2fb640 1 -- 172.21.15.123:0/2494629942 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f3858073220 msgr2=0x7f38580756e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:47:04.747 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:04.744+0000 7f387b2fb640 1 --2- 172.21.15.123:0/2494629942 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f3858073220 0x7f38580756e0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f3864007fd0 tx=0x7f38640078e0 comp rx=0 tx=0).stop 2024-09-16T15:47:04.747 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:04.744+0000 7f387b2fb640 1 -- 172.21.15.123:0/2494629942 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f38740cbb90 msgr2=0x7f387414a9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:47:04.747 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:04.744+0000 7f387b2fb640 1 --2- 172.21.15.123:0/2494629942 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f38740cbb90 0x7f387414a9a0 secure :-1 s=READY pgs=793 cs=0 l=1 rev1=1 crypto rx=0x7f3870009930 tx=0x7f3870005d20 comp rx=0 tx=0).stop 2024-09-16T15:47:04.747 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:04.744+0000 7f387b2fb640 1 -- 172.21.15.123:0/2494629942 shutdown_connections 2024-09-16T15:47:04.747 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:04.744+0000 7f387b2fb640 1 --2- 172.21.15.123:0/2494629942 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f3858073220 0x7f38580756e0 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:04.748 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:04.744+0000 7f387b2fb640 1 --2- 172.21.15.123:0/2494629942 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f3874151400 0x7f38740c8c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:04.748 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:04.744+0000 7f387b2fb640 1 --2- 172.21.15.123:0/2494629942 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f38740ce010 0x7f3874166990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:04.748 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:04.744+0000 7f387b2fb640 1 --2- 172.21.15.123:0/2494629942 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f38740cbb90 0x7f387414a9a0 unknown :-1 s=CLOSED pgs=793 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:04.748 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:04.744+0000 7f387b2fb640 1 -- 172.21.15.123:0/2494629942 >> 172.21.15.123:0/2494629942 conn(0x7f38740cfa90 msgr2=0x7f38740ccc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:47:04.748 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:04.744+0000 7f387b2fb640 1 -- 172.21.15.123:0/2494629942 shutdown_connections 2024-09-16T15:47:04.748 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:04.744+0000 7f387b2fb640 1 -- 172.21.15.123:0/2494629942 wait complete. 2024-09-16T15:47:04.762 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2024-09-16T15:47:04.762 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T15:47:04.884 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:04.880+0000 7fa389736640 1 -- 172.21.15.123:0/2147731213 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa3840726c0 msgr2=0x7fa3840d0630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:47:04.884 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:04.880+0000 7fa389736640 1 --2- 172.21.15.123:0/2147731213 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa3840726c0 0x7fa3840d0630 secure :-1 s=READY pgs=794 cs=0 l=1 rev1=1 crypto rx=0x7fa36c009900 tx=0x7fa36c01c820 comp rx=0 tx=0).stop 2024-09-16T15:47:04.885 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:04.880+0000 7fa389736640 1 -- 172.21.15.123:0/2147731213 shutdown_connections 2024-09-16T15:47:04.885 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:04.880+0000 7fa389736640 1 --2- 172.21.15.123:0/2147731213 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fa384074600 0x7fa38406e750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:04.885 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:04.880+0000 7fa389736640 1 --2- 172.21.15.123:0/2147731213 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa3840726c0 0x7fa3840d0630 unknown :-1 s=CLOSED pgs=794 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:04.885 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:04.880+0000 7fa389736640 1 -- 172.21.15.123:0/2147731213 >> 172.21.15.123:0/2147731213 conn(0x7fa3840cfb10 msgr2=0x7fa3840cff20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:47:04.885 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:04.880+0000 7fa389736640 1 -- 172.21.15.123:0/2147731213 shutdown_connections 2024-09-16T15:47:04.885 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:04.880+0000 7fa389736640 1 -- 172.21.15.123:0/2147731213 wait complete. 2024-09-16T15:47:04.885 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:04.880+0000 7fa389736640 1 Processor -- start 2024-09-16T15:47:04.886 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:04.880+0000 7fa389736640 1 -- start start 2024-09-16T15:47:04.888 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:04.880+0000 7fa389736640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa3840726c0 0x7fa384153db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:47:04.888 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:04.880+0000 7fa389736640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fa384074600 0x7fa3840cf0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:47:04.888 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:04.880+0000 7fa389736640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa38415bca0 0x7fa3840b2f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:47:04.888 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:04.880+0000 7fa389736640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fa384168bd0 con 0x7fa38415bca0 2024-09-16T15:47:04.888 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:04.880+0000 7fa389736640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fa384168a50 con 0x7fa3840726c0 2024-09-16T15:47:04.888 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:04.880+0000 7fa389736640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fa384168d50 con 0x7fa384074600 2024-09-16T15:47:04.888 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:04.880+0000 7fa3837fe640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa38415bca0 0x7fa3840b2f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:47:04.889 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:04.880+0000 7fa3837fe640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa38415bca0 0x7fa3840b2f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:46886/0 (socket says 172.21.15.123:46886) 2024-09-16T15:47:04.889 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:04.880+0000 7fa3837fe640 1 -- 172.21.15.123:0/1976497940 learned_addr learned my addr 172.21.15.123:0/1976497940 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:47:04.889 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:04.884+0000 7fa382ffd640 1 --2- 172.21.15.123:0/1976497940 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa3840726c0 0x7fa384153db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:47:04.889 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:04.884+0000 7fa37a7fc640 1 --2- 172.21.15.123:0/1976497940 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fa384074600 0x7fa3840cf0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:47:04.889 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:04.884+0000 7fa3837fe640 1 -- 172.21.15.123:0/1976497940 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fa384074600 msgr2=0x7fa3840cf0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:47:04.889 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:04.884+0000 7fa3837fe640 1 --2- 172.21.15.123:0/1976497940 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fa384074600 0x7fa3840cf0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:04.889 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:04.884+0000 7fa3837fe640 1 -- 172.21.15.123:0/1976497940 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa3840726c0 msgr2=0x7fa384153db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:47:04.889 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:04.884+0000 7fa3837fe640 1 --2- 172.21.15.123:0/1976497940 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa3840726c0 0x7fa384153db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:04.889 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:04.884+0000 7fa3837fe640 1 -- 172.21.15.123:0/1976497940 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa3841fad20 con 0x7fa38415bca0 2024-09-16T15:47:04.889 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:04.884+0000 7fa37a7fc640 1 --2- 172.21.15.123:0/1976497940 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fa384074600 0x7fa3840cf0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T15:47:04.889 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:04.884+0000 7fa3837fe640 1 --2- 172.21.15.123:0/1976497940 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa38415bca0 0x7fa3840b2f40 secure :-1 s=READY pgs=795 cs=0 l=1 rev1=1 crypto rx=0x7fa37c00ccb0 tx=0x7fa37c00a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:47:04.890 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:04.884+0000 7fa37bfff640 1 -- 172.21.15.123:0/1976497940 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa37c004b30 con 0x7fa38415bca0 2024-09-16T15:47:04.890 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:04.884+0000 7fa37bfff640 1 -- 172.21.15.123:0/1976497940 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fa37c004de0 con 0x7fa38415bca0 2024-09-16T15:47:04.890 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:04.884+0000 7fa389736640 1 -- 172.21.15.123:0/1976497940 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa3841fb000 con 0x7fa38415bca0 2024-09-16T15:47:04.890 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:04.884+0000 7fa37bfff640 1 -- 172.21.15.123:0/1976497940 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa37c005280 con 0x7fa38415bca0 2024-09-16T15:47:04.890 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:04.884+0000 7fa389736640 1 -- 172.21.15.123:0/1976497940 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa3841fb5c0 con 0x7fa38415bca0 2024-09-16T15:47:04.891 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:04.884+0000 7fa389736640 1 -- 172.21.15.123:0/1976497940 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa3840d0630 con 0x7fa38415bca0 2024-09-16T15:47:04.891 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:04.884+0000 7fa37bfff640 1 -- 172.21.15.123:0/1976497940 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 11) ==== 97386+0+0 (secure 0 0 0) 0x7fa37c005420 con 0x7fa38415bca0 2024-09-16T15:47:04.891 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:04.884+0000 7fa37bfff640 1 --2- 172.21.15.123:0/1976497940 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fa364073250 0x7fa364075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:47:04.891 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:04.884+0000 7fa37bfff640 1 -- 172.21.15.123:0/1976497940 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(70..70 src has 1..70) ==== 5709+0+0 (secure 0 0 0) 0x7fa37c0950d0 con 0x7fa38415bca0 2024-09-16T15:47:04.891 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:04.884+0000 7fa382ffd640 1 --2- 172.21.15.123:0/1976497940 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fa364073250 0x7fa364075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:47:04.891 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:04.888+0000 7fa382ffd640 1 --2- 172.21.15.123:0/1976497940 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fa364073250 0x7fa364075710 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7fa3840cd8b0 tx=0x7fa374008040 comp rx=0 tx=0).ready entity=mgr.4135 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:47:04.894 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:04.888+0000 7fa37bfff640 1 -- 172.21.15.123:0/1976497940 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7fa37c014030 con 0x7fa38415bca0 2024-09-16T15:47:05.058 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:05.052+0000 7fa389736640 1 -- 172.21.15.123:0/1976497940 --> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fa3840d0230 con 0x7fa364073250 2024-09-16T15:47:05.069 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:05.068+0000 7fa37bfff640 1 -- 172.21.15.123:0/1976497940 <== mgr.4135 v2:172.21.15.184:6832/3114346423 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+246774 (secure 0 0 0) 0x7fa3840d0230 con 0x7fa364073250 2024-09-16T15:47:05.071 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-09-16T15:47:05.073 INFO:teuthology.orchestra.run.smithi123.stderr:dumped all 2024-09-16T15:47:05.074 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:05.068+0000 7fa389736640 1 -- 172.21.15.123:0/1976497940 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fa364073250 msgr2=0x7fa364075710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:47:05.074 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:05.068+0000 7fa389736640 1 --2- 172.21.15.123:0/1976497940 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fa364073250 0x7fa364075710 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7fa3840cd8b0 tx=0x7fa374008040 comp rx=0 tx=0).stop 2024-09-16T15:47:05.074 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:05.068+0000 7fa389736640 1 -- 172.21.15.123:0/1976497940 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa38415bca0 msgr2=0x7fa3840b2f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:47:05.074 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:05.068+0000 7fa389736640 1 --2- 172.21.15.123:0/1976497940 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa38415bca0 0x7fa3840b2f40 secure :-1 s=READY pgs=795 cs=0 l=1 rev1=1 crypto rx=0x7fa37c00ccb0 tx=0x7fa37c00a620 comp rx=0 tx=0).stop 2024-09-16T15:47:05.074 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:05.068+0000 7fa389736640 1 -- 172.21.15.123:0/1976497940 shutdown_connections 2024-09-16T15:47:05.074 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:05.068+0000 7fa389736640 1 --2- 172.21.15.123:0/1976497940 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fa364073250 0x7fa364075710 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:05.074 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:05.068+0000 7fa389736640 1 --2- 172.21.15.123:0/1976497940 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa38415bca0 0x7fa3840b2f40 unknown :-1 s=CLOSED pgs=795 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:05.074 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:05.068+0000 7fa389736640 1 --2- 172.21.15.123:0/1976497940 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fa384074600 0x7fa3840cf0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:05.075 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:05.068+0000 7fa389736640 1 --2- 172.21.15.123:0/1976497940 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa3840726c0 0x7fa384153db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:05.075 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:05.068+0000 7fa389736640 1 -- 172.21.15.123:0/1976497940 >> 172.21.15.123:0/1976497940 conn(0x7fa3840cfb10 msgr2=0x7fa3840b4350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:47:05.075 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:05.068+0000 7fa389736640 1 -- 172.21.15.123:0/1976497940 shutdown_connections 2024-09-16T15:47:05.075 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:05.068+0000 7fa389736640 1 -- 172.21.15.123:0/1976497940 wait complete. 2024-09-16T15:47:05.089 INFO:teuthology.orchestra.run.smithi123.stdout:{"pg_ready":true,"pg_map":{"version":195,"stamp":"2024-09-16T15:47:03.331208+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":76,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":240444,"kb_used_data":7556,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":754734276,"statfs":{"total":773094113280,"available":772847898624,"internally_reserved":0,"allocated":7737344,"data_stored":3897055,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":238276145},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":41,"apply_latency_ms":41,"commit_latency_ns":41000000,"apply_latency_ns":41000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.788118"},"pg_stats":[{"pgid":"11.3f","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-09-16T15:46:58.580896+0000","last_change":"2024-09-16T15:46:58.580896+0000","last_active":"2024-09-16T15:46:58.541934+0000","last_peered":"2024-09-16T15:46:58.541934+0000","last_clean":"2024-09-16T15:46:58.541934+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:46:58.580896+0000","last_undegraded":"2024-09-16T15:46:58.580896+0000","last_fullsized":"2024-09-16T15:46:58.580896+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_clean_scrub_stamp":"2024-09-16T15:46:58.541934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.3e","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-09-16T15:46:58.580879+0000","last_change":"2024-09-16T15:46:58.580879+0000","last_active":"2024-09-16T15:46:58.541934+0000","last_peered":"2024-09-16T15:46:58.541934+0000","last_clean":"2024-09-16T15:46:58.541934+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:46:58.580879+0000","last_undegraded":"2024-09-16T15:46:58.580879+0000","last_fullsized":"2024-09-16T15:46:58.580879+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_clean_scrub_stamp":"2024-09-16T15:46:58.541934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.3d","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-09-16T15:46:58.565500+0000","last_change":"2024-09-16T15:46:58.565500+0000","last_active":"2024-09-16T15:46:58.541934+0000","last_peered":"2024-09-16T15:46:58.541934+0000","last_clean":"2024-09-16T15:46:58.541934+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:46:58.565500+0000","last_undegraded":"2024-09-16T15:46:58.565500+0000","last_fullsized":"2024-09-16T15:46:58.565500+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_clean_scrub_stamp":"2024-09-16T15:46:58.541934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.3c","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-09-16T15:46:58.560961+0000","last_change":"2024-09-16T15:46:58.560961+0000","last_active":"2024-09-16T15:46:58.541934+0000","last_peered":"2024-09-16T15:46:58.541934+0000","last_clean":"2024-09-16T15:46:58.541934+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:46:58.560961+0000","last_undegraded":"2024-09-16T15:46:58.560961+0000","last_fullsized":"2024-09-16T15:46:58.560961+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_clean_scrub_stamp":"2024-09-16T15:46:58.541934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.3b","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-09-16T15:46:58.561715+0000","last_change":"2024-09-16T15:46:58.561715+0000","last_active":"2024-09-16T15:46:58.541934+0000","last_peered":"2024-09-16T15:46:58.541934+0000","last_clean":"2024-09-16T15:46:58.541934+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:46:58.561715+0000","last_undegraded":"2024-09-16T15:46:58.561715+0000","last_fullsized":"2024-09-16T15:46:58.561715+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_clean_scrub_stamp":"2024-09-16T15:46:58.541934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.3a","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-09-16T15:46:58.578627+0000","last_change":"2024-09-16T15:46:58.578627+0000","last_active":"2024-09-16T15:46:58.541934+0000","last_peered":"2024-09-16T15:46:58.541934+0000","last_clean":"2024-09-16T15:46:58.541934+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:46:58.578627+0000","last_undegraded":"2024-09-16T15:46:58.578627+0000","last_fullsized":"2024-09-16T15:46:58.578627+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_clean_scrub_stamp":"2024-09-16T15:46:58.541934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.39","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-09-16T15:46:58.570892+0000","last_change":"2024-09-16T15:46:58.570892+0000","last_active":"2024-09-16T15:46:58.541934+0000","last_peered":"2024-09-16T15:46:58.541934+0000","last_clean":"2024-09-16T15:46:58.541934+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:46:58.570892+0000","last_undegraded":"2024-09-16T15:46:58.570892+0000","last_fullsized":"2024-09-16T15:46:58.570892+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_clean_scrub_stamp":"2024-09-16T15:46:58.541934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.38","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-09-16T15:46:58.584780+0000","last_change":"2024-09-16T15:46:58.584780+0000","last_active":"2024-09-16T15:46:58.541934+0000","last_peered":"2024-09-16T15:46:58.541934+0000","last_clean":"2024-09-16T15:46:58.541934+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:46:58.584780+0000","last_undegraded":"2024-09-16T15:46:58.584780+0000","last_fullsized":"2024-09-16T15:46:58.584780+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_clean_scrub_stamp":"2024-09-16T15:46:58.541934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.37","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-09-16T15:46:58.561912+0000","last_change":"2024-09-16T15:46:58.561912+0000","last_active":"2024-09-16T15:46:58.541934+0000","last_peered":"2024-09-16T15:46:58.541934+0000","last_clean":"2024-09-16T15:46:58.541934+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:46:58.561912+0000","last_undegraded":"2024-09-16T15:46:58.561912+0000","last_fullsized":"2024-09-16T15:46:58.561912+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_clean_scrub_stamp":"2024-09-16T15:46:58.541934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.36","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-09-16T15:46:58.562601+0000","last_change":"2024-09-16T15:46:58.562601+0000","last_active":"2024-09-16T15:46:58.541934+0000","last_peered":"2024-09-16T15:46:58.541934+0000","last_clean":"2024-09-16T15:46:58.541934+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:46:58.562601+0000","last_undegraded":"2024-09-16T15:46:58.562601+0000","last_fullsized":"2024-09-16T15:46:58.562601+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_clean_scrub_stamp":"2024-09-16T15:46:58.541934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.35","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-09-16T15:46:58.580321+0000","last_change":"2024-09-16T15:46:58.580321+0000","last_active":"2024-09-16T15:46:58.541934+0000","last_peered":"2024-09-16T15:46:58.541934+0000","last_clean":"2024-09-16T15:46:58.541934+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:46:58.580321+0000","last_undegraded":"2024-09-16T15:46:58.580321+0000","last_fullsized":"2024-09-16T15:46:58.580321+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_clean_scrub_stamp":"2024-09-16T15:46:58.541934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.34","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-09-16T15:46:58.560127+0000","last_change":"2024-09-16T15:46:58.560127+0000","last_active":"2024-09-16T15:46:58.541934+0000","last_peered":"2024-09-16T15:46:58.541934+0000","last_clean":"2024-09-16T15:46:58.541934+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:46:58.560127+0000","last_undegraded":"2024-09-16T15:46:58.560127+0000","last_fullsized":"2024-09-16T15:46:58.560127+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_clean_scrub_stamp":"2024-09-16T15:46:58.541934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.33","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-09-16T15:46:58.562521+0000","last_change":"2024-09-16T15:46:58.562521+0000","last_active":"2024-09-16T15:46:58.541934+0000","last_peered":"2024-09-16T15:46:58.541934+0000","last_clean":"2024-09-16T15:46:58.541934+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:46:58.562521+0000","last_undegraded":"2024-09-16T15:46:58.562521+0000","last_fullsized":"2024-09-16T15:46:58.562521+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_clean_scrub_stamp":"2024-09-16T15:46:58.541934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.32","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-09-16T15:46:58.561306+0000","last_change":"2024-09-16T15:46:58.561306+0000","last_active":"2024-09-16T15:46:58.541934+0000","last_peered":"2024-09-16T15:46:58.541934+0000","last_clean":"2024-09-16T15:46:58.541934+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:46:58.561306+0000","last_undegraded":"2024-09-16T15:46:58.561306+0000","last_fullsized":"2024-09-16T15:46:58.561306+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_clean_scrub_stamp":"2024-09-16T15:46:58.541934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.31","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-09-16T15:46:58.566448+0000","last_change":"2024-09-16T15:46:58.566448+0000","last_active":"2024-09-16T15:46:58.541934+0000","last_peered":"2024-09-16T15:46:58.541934+0000","last_clean":"2024-09-16T15:46:58.541934+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:46:58.566448+0000","last_undegraded":"2024-09-16T15:46:58.566448+0000","last_fullsized":"2024-09-16T15:46:58.566448+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_clean_scrub_stamp":"2024-09-16T15:46:58.541934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.30","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-09-16T15:46:58.560118+0000","last_change":"2024-09-16T15:46:58.560118+0000","last_active":"2024-09-16T15:46:58.541934+0000","last_peered":"2024-09-16T15:46:58.541934+0000","last_clean":"2024-09-16T15:46:58.541934+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:46:58.560118+0000","last_undegraded":"2024-09-16T15:46:58.560118+0000","last_fullsized":"2024-09-16T15:46:58.560118+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_clean_scrub_stamp":"2024-09-16T15:46:58.541934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.2f","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-09-16T15:46:58.561316+0000","last_change":"2024-09-16T15:46:58.561316+0000","last_active":"2024-09-16T15:46:58.541934+0000","last_peered":"2024-09-16T15:46:58.541934+0000","last_clean":"2024-09-16T15:46:58.541934+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:46:58.561316+0000","last_undegraded":"2024-09-16T15:46:58.561316+0000","last_fullsized":"2024-09-16T15:46:58.561316+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_clean_scrub_stamp":"2024-09-16T15:46:58.541934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.2e","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-09-16T15:46:58.559376+0000","last_change":"2024-09-16T15:46:58.559376+0000","last_active":"2024-09-16T15:46:58.541934+0000","last_peered":"2024-09-16T15:46:58.541934+0000","last_clean":"2024-09-16T15:46:58.541934+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:46:58.559376+0000","last_undegraded":"2024-09-16T15:46:58.559376+0000","last_fullsized":"2024-09-16T15:46:58.559376+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_clean_scrub_stamp":"2024-09-16T15:46:58.541934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.2d","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-09-16T15:46:58.580101+0000","last_change":"2024-09-16T15:46:58.580101+0000","last_active":"2024-09-16T15:46:58.541934+0000","last_peered":"2024-09-16T15:46:58.541934+0000","last_clean":"2024-09-16T15:46:58.541934+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:46:58.580101+0000","last_undegraded":"2024-09-16T15:46:58.580101+0000","last_fullsized":"2024-09-16T15:46:58.580101+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_clean_scrub_stamp":"2024-09-16T15:46:58.541934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.2c","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-09-16T15:46:58.562539+0000","last_change":"2024-09-16T15:46:58.562539+0000","last_active":"2024-09-16T15:46:58.541934+0000","last_peered":"2024-09-16T15:46:58.541934+0000","last_clean":"2024-09-16T15:46:58.541934+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:46:58.562539+0000","last_undegraded":"2024-09-16T15:46:58.562539+0000","last_fullsized":"2024-09-16T15:46:58.562539+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_clean_scrub_stamp":"2024-09-16T15:46:58.541934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.2b","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-09-16T15:46:58.564602+0000","last_change":"2024-09-16T15:46:58.564602+0000","last_active":"2024-09-16T15:46:58.541934+0000","last_peered":"2024-09-16T15:46:58.541934+0000","last_clean":"2024-09-16T15:46:58.541934+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:46:58.564602+0000","last_undegraded":"2024-09-16T15:46:58.564602+0000","last_fullsized":"2024-09-16T15:46:58.564602+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_clean_scrub_stamp":"2024-09-16T15:46:58.541934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.2a","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-09-16T15:46:58.562954+0000","last_change":"2024-09-16T15:46:58.562954+0000","last_active":"2024-09-16T15:46:58.541934+0000","last_peered":"2024-09-16T15:46:58.541934+0000","last_clean":"2024-09-16T15:46:58.541934+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:46:58.562954+0000","last_undegraded":"2024-09-16T15:46:58.562954+0000","last_fullsized":"2024-09-16T15:46:58.562954+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_clean_scrub_stamp":"2024-09-16T15:46:58.541934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.29","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-09-16T15:46:58.578047+0000","last_change":"2024-09-16T15:46:58.578047+0000","last_active":"2024-09-16T15:46:58.541934+0000","last_peered":"2024-09-16T15:46:58.541934+0000","last_clean":"2024-09-16T15:46:58.541934+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:46:58.578047+0000","last_undegraded":"2024-09-16T15:46:58.578047+0000","last_fullsized":"2024-09-16T15:46:58.578047+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_clean_scrub_stamp":"2024-09-16T15:46:58.541934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.28","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-09-16T15:46:58.575628+0000","last_change":"2024-09-16T15:46:58.575628+0000","last_active":"2024-09-16T15:46:58.541934+0000","last_peered":"2024-09-16T15:46:58.541934+0000","last_clean":"2024-09-16T15:46:58.541934+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:46:58.575628+0000","last_undegraded":"2024-09-16T15:46:58.575628+0000","last_fullsized":"2024-09-16T15:46:58.575628+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_clean_scrub_stamp":"2024-09-16T15:46:58.541934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.27","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-09-16T15:46:58.562122+0000","last_change":"2024-09-16T15:46:58.562122+0000","last_active":"2024-09-16T15:46:58.541934+0000","last_peered":"2024-09-16T15:46:58.541934+0000","last_clean":"2024-09-16T15:46:58.541934+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:46:58.562122+0000","last_undegraded":"2024-09-16T15:46:58.562122+0000","last_fullsized":"2024-09-16T15:46:58.562122+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_clean_scrub_stamp":"2024-09-16T15:46:58.541934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.26","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-09-16T15:46:58.571812+0000","last_change":"2024-09-16T15:46:58.571812+0000","last_active":"2024-09-16T15:46:58.541934+0000","last_peered":"2024-09-16T15:46:58.541934+0000","last_clean":"2024-09-16T15:46:58.541934+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:46:58.571812+0000","last_undegraded":"2024-09-16T15:46:58.571812+0000","last_fullsized":"2024-09-16T15:46:58.571812+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_clean_scrub_stamp":"2024-09-16T15:46:58.541934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.25","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-09-16T15:46:58.564846+0000","last_change":"2024-09-16T15:46:58.564846+0000","last_active":"2024-09-16T15:46:58.541934+0000","last_peered":"2024-09-16T15:46:58.541934+0000","last_clean":"2024-09-16T15:46:58.541934+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:46:58.564846+0000","last_undegraded":"2024-09-16T15:46:58.564846+0000","last_fullsized":"2024-09-16T15:46:58.564846+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_clean_scrub_stamp":"2024-09-16T15:46:58.541934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.24","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-09-16T15:46:58.560367+0000","last_change":"2024-09-16T15:46:58.560367+0000","last_active":"2024-09-16T15:46:58.541934+0000","last_peered":"2024-09-16T15:46:58.541934+0000","last_clean":"2024-09-16T15:46:58.541934+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:46:58.560367+0000","last_undegraded":"2024-09-16T15:46:58.560367+0000","last_fullsized":"2024-09-16T15:46:58.560367+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_clean_scrub_stamp":"2024-09-16T15:46:58.541934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.23","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-09-16T15:46:58.568304+0000","last_change":"2024-09-16T15:46:58.568304+0000","last_active":"2024-09-16T15:46:58.541934+0000","last_peered":"2024-09-16T15:46:58.541934+0000","last_clean":"2024-09-16T15:46:58.541934+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:46:58.568304+0000","last_undegraded":"2024-09-16T15:46:58.568304+0000","last_fullsized":"2024-09-16T15:46:58.568304+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_clean_scrub_stamp":"2024-09-16T15:46:58.541934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.22","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-09-16T15:46:58.569453+0000","last_change":"2024-09-16T15:46:58.569453+0000","last_active":"2024-09-16T15:46:58.541934+0000","last_peered":"2024-09-16T15:46:58.541934+0000","last_clean":"2024-09-16T15:46:58.541934+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:46:58.569453+0000","last_undegraded":"2024-09-16T15:46:58.569453+0000","last_fullsized":"2024-09-16T15:46:58.569453+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_clean_scrub_stamp":"2024-09-16T15:46:58.541934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.21","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-09-16T15:46:58.569512+0000","last_change":"2024-09-16T15:46:58.569512+0000","last_active":"2024-09-16T15:46:58.541934+0000","last_peered":"2024-09-16T15:46:58.541934+0000","last_clean":"2024-09-16T15:46:58.541934+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:46:58.569512+0000","last_undegraded":"2024-09-16T15:46:58.569512+0000","last_fullsized":"2024-09-16T15:46:58.569512+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_clean_scrub_stamp":"2024-09-16T15:46:58.541934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.20","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-09-16T15:46:58.582821+0000","last_change":"2024-09-16T15:46:58.582821+0000","last_active":"2024-09-16T15:46:58.541934+0000","last_peered":"2024-09-16T15:46:58.541934+0000","last_clean":"2024-09-16T15:46:58.541934+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:46:58.582821+0000","last_undegraded":"2024-09-16T15:46:58.582821+0000","last_fullsized":"2024-09-16T15:46:58.582821+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_clean_scrub_stamp":"2024-09-16T15:46:58.541934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.1e","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-09-16T15:46:58.575147+0000","last_change":"2024-09-16T15:46:58.575147+0000","last_active":"2024-09-16T15:46:58.541934+0000","last_peered":"2024-09-16T15:46:58.541934+0000","last_clean":"2024-09-16T15:46:58.541934+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:46:58.575147+0000","last_undegraded":"2024-09-16T15:46:58.575147+0000","last_fullsized":"2024-09-16T15:46:58.575147+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_clean_scrub_stamp":"2024-09-16T15:46:58.541934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.1f","version":"0'0","reported_seq":10,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-16T15:46:58.599998+0000","last_change":"2024-09-16T15:46:58.596071+0000","last_active":"2024-09-16T15:46:58.599998+0000","last_peered":"2024-09-16T15:46:58.599998+0000","last_clean":"2024-09-16T15:46:58.599998+0000","last_became_active":"2024-09-16T15:46:58.595309+0000","last_became_peered":"2024-09-16T15:46:58.595309+0000","last_unstale":"2024-09-16T15:46:58.599998+0000","last_undegraded":"2024-09-16T15:46:58.599998+0000","last_fullsized":"2024-09-16T15:46:58.599998+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:57.531046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:57.531046+0000","last_clean_scrub_stamp":"2024-09-16T15:46:57.531046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:37:47.205740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.1f","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-09-16T15:46:58.560943+0000","last_change":"2024-09-16T15:46:58.560943+0000","last_active":"2024-09-16T15:46:58.541934+0000","last_peered":"2024-09-16T15:46:58.541934+0000","last_clean":"2024-09-16T15:46:58.541934+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:46:58.560943+0000","last_undegraded":"2024-09-16T15:46:58.560943+0000","last_fullsized":"2024-09-16T15:46:58.560943+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_clean_scrub_stamp":"2024-09-16T15:46:58.541934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.1e","version":"0'0","reported_seq":10,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-16T15:46:58.577768+0000","last_change":"2024-09-16T15:46:58.572883+0000","last_active":"2024-09-16T15:46:58.577768+0000","last_peered":"2024-09-16T15:46:58.577768+0000","last_clean":"2024-09-16T15:46:58.577768+0000","last_became_active":"2024-09-16T15:46:58.572508+0000","last_became_peered":"2024-09-16T15:46:58.572508+0000","last_unstale":"2024-09-16T15:46:58.577768+0000","last_undegraded":"2024-09-16T15:46:58.577768+0000","last_fullsized":"2024-09-16T15:46:58.577768+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:57.531046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:57.531046+0000","last_clean_scrub_stamp":"2024-09-16T15:46:57.531046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:16:24.337739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1c","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-09-16T15:46:58.583638+0000","last_change":"2024-09-16T15:46:58.583638+0000","last_active":"2024-09-16T15:46:58.541934+0000","last_peered":"2024-09-16T15:46:58.541934+0000","last_clean":"2024-09-16T15:46:58.541934+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:46:58.583638+0000","last_undegraded":"2024-09-16T15:46:58.583638+0000","last_fullsized":"2024-09-16T15:46:58.583638+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_clean_scrub_stamp":"2024-09-16T15:46:58.541934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.1d","version":"0'0","reported_seq":10,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-16T15:46:58.604745+0000","last_change":"2024-09-16T15:46:58.598772+0000","last_active":"2024-09-16T15:46:58.604745+0000","last_peered":"2024-09-16T15:46:58.604745+0000","last_clean":"2024-09-16T15:46:58.604745+0000","last_became_active":"2024-09-16T15:46:58.597889+0000","last_became_peered":"2024-09-16T15:46:58.597889+0000","last_unstale":"2024-09-16T15:46:58.604745+0000","last_undegraded":"2024-09-16T15:46:58.604745+0000","last_fullsized":"2024-09-16T15:46:58.604745+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:57.531046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:57.531046+0000","last_clean_scrub_stamp":"2024-09-16T15:46:57.531046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:20:50.612777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1d","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-09-16T15:46:58.579199+0000","last_change":"2024-09-16T15:46:58.579199+0000","last_active":"2024-09-16T15:46:58.541934+0000","last_peered":"2024-09-16T15:46:58.541934+0000","last_clean":"2024-09-16T15:46:58.541934+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:46:58.579199+0000","last_undegraded":"2024-09-16T15:46:58.579199+0000","last_fullsized":"2024-09-16T15:46:58.579199+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_clean_scrub_stamp":"2024-09-16T15:46:58.541934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.1c","version":"0'0","reported_seq":10,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-16T15:46:58.595606+0000","last_change":"2024-09-16T15:46:58.588644+0000","last_active":"2024-09-16T15:46:58.595606+0000","last_peered":"2024-09-16T15:46:58.595606+0000","last_clean":"2024-09-16T15:46:58.595606+0000","last_became_active":"2024-09-16T15:46:58.587899+0000","last_became_peered":"2024-09-16T15:46:58.587899+0000","last_unstale":"2024-09-16T15:46:58.595606+0000","last_undegraded":"2024-09-16T15:46:58.595606+0000","last_fullsized":"2024-09-16T15:46:58.595606+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:57.531046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:57.531046+0000","last_clean_scrub_stamp":"2024-09-16T15:46:57.531046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T00:36:13.054311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.1a","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-09-16T15:46:58.568314+0000","last_change":"2024-09-16T15:46:58.568314+0000","last_active":"2024-09-16T15:46:58.541934+0000","last_peered":"2024-09-16T15:46:58.541934+0000","last_clean":"2024-09-16T15:46:58.541934+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:46:58.568314+0000","last_undegraded":"2024-09-16T15:46:58.568314+0000","last_fullsized":"2024-09-16T15:46:58.568314+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_clean_scrub_stamp":"2024-09-16T15:46:58.541934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.1b","version":"0'0","reported_seq":10,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-16T15:46:58.594471+0000","last_change":"2024-09-16T15:46:58.591621+0000","last_active":"2024-09-16T15:46:58.594471+0000","last_peered":"2024-09-16T15:46:58.594471+0000","last_clean":"2024-09-16T15:46:58.594471+0000","last_became_active":"2024-09-16T15:46:58.591311+0000","last_became_peered":"2024-09-16T15:46:58.591311+0000","last_unstale":"2024-09-16T15:46:58.594471+0000","last_undegraded":"2024-09-16T15:46:58.594471+0000","last_fullsized":"2024-09-16T15:46:58.594471+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:57.531046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:57.531046+0000","last_clean_scrub_stamp":"2024-09-16T15:46:57.531046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:55:01.184561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.1b","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-09-16T15:46:58.563604+0000","last_change":"2024-09-16T15:46:58.563604+0000","last_active":"2024-09-16T15:46:58.541934+0000","last_peered":"2024-09-16T15:46:58.541934+0000","last_clean":"2024-09-16T15:46:58.541934+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:46:58.563604+0000","last_undegraded":"2024-09-16T15:46:58.563604+0000","last_fullsized":"2024-09-16T15:46:58.563604+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_clean_scrub_stamp":"2024-09-16T15:46:58.541934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.1a","version":"0'0","reported_seq":10,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-16T15:46:58.604119+0000","last_change":"2024-09-16T15:46:58.600655+0000","last_active":"2024-09-16T15:46:58.604119+0000","last_peered":"2024-09-16T15:46:58.604119+0000","last_clean":"2024-09-16T15:46:58.604119+0000","last_became_active":"2024-09-16T15:46:58.599216+0000","last_became_peered":"2024-09-16T15:46:58.599216+0000","last_unstale":"2024-09-16T15:46:58.604119+0000","last_undegraded":"2024-09-16T15:46:58.604119+0000","last_fullsized":"2024-09-16T15:46:58.604119+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:57.531046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:57.531046+0000","last_clean_scrub_stamp":"2024-09-16T15:46:57.531046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:13:18.615772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.18","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-09-16T15:46:58.562911+0000","last_change":"2024-09-16T15:46:58.562911+0000","last_active":"2024-09-16T15:46:58.541934+0000","last_peered":"2024-09-16T15:46:58.541934+0000","last_clean":"2024-09-16T15:46:58.541934+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:46:58.562911+0000","last_undegraded":"2024-09-16T15:46:58.562911+0000","last_fullsized":"2024-09-16T15:46:58.562911+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_clean_scrub_stamp":"2024-09-16T15:46:58.541934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.19","version":"0'0","reported_seq":10,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-16T15:46:58.589251+0000","last_change":"2024-09-16T15:46:58.584364+0000","last_active":"2024-09-16T15:46:58.589251+0000","last_peered":"2024-09-16T15:46:58.589251+0000","last_clean":"2024-09-16T15:46:58.589251+0000","last_became_active":"2024-09-16T15:46:58.583975+0000","last_became_peered":"2024-09-16T15:46:58.583975+0000","last_unstale":"2024-09-16T15:46:58.589251+0000","last_undegraded":"2024-09-16T15:46:58.589251+0000","last_fullsized":"2024-09-16T15:46:58.589251+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:57.531046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:57.531046+0000","last_clean_scrub_stamp":"2024-09-16T15:46:57.531046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T00:25:29.249313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.19","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-09-16T15:46:58.566064+0000","last_change":"2024-09-16T15:46:58.566064+0000","last_active":"2024-09-16T15:46:58.541934+0000","last_peered":"2024-09-16T15:46:58.541934+0000","last_clean":"2024-09-16T15:46:58.541934+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:46:58.566064+0000","last_undegraded":"2024-09-16T15:46:58.566064+0000","last_fullsized":"2024-09-16T15:46:58.566064+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_clean_scrub_stamp":"2024-09-16T15:46:58.541934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.18","version":"0'0","reported_seq":10,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-16T15:46:58.574992+0000","last_change":"2024-09-16T15:46:58.570577+0000","last_active":"2024-09-16T15:46:58.574992+0000","last_peered":"2024-09-16T15:46:58.574992+0000","last_clean":"2024-09-16T15:46:58.574992+0000","last_became_active":"2024-09-16T15:46:58.570223+0000","last_became_peered":"2024-09-16T15:46:58.570223+0000","last_unstale":"2024-09-16T15:46:58.574992+0000","last_undegraded":"2024-09-16T15:46:58.574992+0000","last_fullsized":"2024-09-16T15:46:58.574992+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:57.531046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:57.531046+0000","last_clean_scrub_stamp":"2024-09-16T15:46:57.531046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:11:59.055640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.16","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-09-16T15:46:58.570594+0000","last_change":"2024-09-16T15:46:58.570594+0000","last_active":"2024-09-16T15:46:58.541934+0000","last_peered":"2024-09-16T15:46:58.541934+0000","last_clean":"2024-09-16T15:46:58.541934+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:46:58.570594+0000","last_undegraded":"2024-09-16T15:46:58.570594+0000","last_fullsized":"2024-09-16T15:46:58.570594+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_clean_scrub_stamp":"2024-09-16T15:46:58.541934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.17","version":"0'0","reported_seq":10,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-16T15:46:58.590618+0000","last_change":"2024-09-16T15:46:58.576463+0000","last_active":"2024-09-16T15:46:58.590618+0000","last_peered":"2024-09-16T15:46:58.590618+0000","last_clean":"2024-09-16T15:46:58.590618+0000","last_became_active":"2024-09-16T15:46:58.573574+0000","last_became_peered":"2024-09-16T15:46:58.573574+0000","last_unstale":"2024-09-16T15:46:58.590618+0000","last_undegraded":"2024-09-16T15:46:58.590618+0000","last_fullsized":"2024-09-16T15:46:58.590618+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:57.531046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:57.531046+0000","last_clean_scrub_stamp":"2024-09-16T15:46:57.531046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T01:24:15.136386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.17","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-09-16T15:46:58.579574+0000","last_change":"2024-09-16T15:46:58.579574+0000","last_active":"2024-09-16T15:46:58.541934+0000","last_peered":"2024-09-16T15:46:58.541934+0000","last_clean":"2024-09-16T15:46:58.541934+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:46:58.579574+0000","last_undegraded":"2024-09-16T15:46:58.579574+0000","last_fullsized":"2024-09-16T15:46:58.579574+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_clean_scrub_stamp":"2024-09-16T15:46:58.541934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.16","version":"0'0","reported_seq":10,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-16T15:46:58.602617+0000","last_change":"2024-09-16T15:46:58.597162+0000","last_active":"2024-09-16T15:46:58.602617+0000","last_peered":"2024-09-16T15:46:58.602617+0000","last_clean":"2024-09-16T15:46:58.602617+0000","last_became_active":"2024-09-16T15:46:58.596469+0000","last_became_peered":"2024-09-16T15:46:58.596469+0000","last_unstale":"2024-09-16T15:46:58.602617+0000","last_undegraded":"2024-09-16T15:46:58.602617+0000","last_fullsized":"2024-09-16T15:46:58.602617+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:57.531046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:57.531046+0000","last_clean_scrub_stamp":"2024-09-16T15:46:57.531046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:22:59.268994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.14","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-09-16T15:46:58.582525+0000","last_change":"2024-09-16T15:46:58.582525+0000","last_active":"2024-09-16T15:46:58.541934+0000","last_peered":"2024-09-16T15:46:58.541934+0000","last_clean":"2024-09-16T15:46:58.541934+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:46:58.582525+0000","last_undegraded":"2024-09-16T15:46:58.582525+0000","last_fullsized":"2024-09-16T15:46:58.582525+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_clean_scrub_stamp":"2024-09-16T15:46:58.541934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.15","version":"0'0","reported_seq":10,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-16T15:46:58.588947+0000","last_change":"2024-09-16T15:46:58.583907+0000","last_active":"2024-09-16T15:46:58.588947+0000","last_peered":"2024-09-16T15:46:58.588947+0000","last_clean":"2024-09-16T15:46:58.588947+0000","last_became_active":"2024-09-16T15:46:58.583530+0000","last_became_peered":"2024-09-16T15:46:58.583530+0000","last_unstale":"2024-09-16T15:46:58.588947+0000","last_undegraded":"2024-09-16T15:46:58.588947+0000","last_fullsized":"2024-09-16T15:46:58.588947+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:57.531046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:57.531046+0000","last_clean_scrub_stamp":"2024-09-16T15:46:57.531046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T01:06:36.276569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.15","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-09-16T15:46:58.563631+0000","last_change":"2024-09-16T15:46:58.563631+0000","last_active":"2024-09-16T15:46:58.541934+0000","last_peered":"2024-09-16T15:46:58.541934+0000","last_clean":"2024-09-16T15:46:58.541934+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:46:58.563631+0000","last_undegraded":"2024-09-16T15:46:58.563631+0000","last_fullsized":"2024-09-16T15:46:58.563631+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_clean_scrub_stamp":"2024-09-16T15:46:58.541934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.14","version":"0'0","reported_seq":10,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-16T15:46:58.575157+0000","last_change":"2024-09-16T15:46:58.570712+0000","last_active":"2024-09-16T15:46:58.575157+0000","last_peered":"2024-09-16T15:46:58.575157+0000","last_clean":"2024-09-16T15:46:58.575157+0000","last_became_active":"2024-09-16T15:46:58.570432+0000","last_became_peered":"2024-09-16T15:46:58.570432+0000","last_unstale":"2024-09-16T15:46:58.575157+0000","last_undegraded":"2024-09-16T15:46:58.575157+0000","last_fullsized":"2024-09-16T15:46:58.575157+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:57.531046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:57.531046+0000","last_clean_scrub_stamp":"2024-09-16T15:46:57.531046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:58:27.343595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.12","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-09-16T15:46:58.574297+0000","last_change":"2024-09-16T15:46:58.574297+0000","last_active":"2024-09-16T15:46:58.541934+0000","last_peered":"2024-09-16T15:46:58.541934+0000","last_clean":"2024-09-16T15:46:58.541934+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:46:58.574297+0000","last_undegraded":"2024-09-16T15:46:58.574297+0000","last_fullsized":"2024-09-16T15:46:58.574297+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_clean_scrub_stamp":"2024-09-16T15:46:58.541934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.13","version":"0'0","reported_seq":10,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-16T15:46:58.603021+0000","last_change":"2024-09-16T15:46:58.596492+0000","last_active":"2024-09-16T15:46:58.603021+0000","last_peered":"2024-09-16T15:46:58.603021+0000","last_clean":"2024-09-16T15:46:58.603021+0000","last_became_active":"2024-09-16T15:46:58.596172+0000","last_became_peered":"2024-09-16T15:46:58.596172+0000","last_unstale":"2024-09-16T15:46:58.603021+0000","last_undegraded":"2024-09-16T15:46:58.603021+0000","last_fullsized":"2024-09-16T15:46:58.603021+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:57.531046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:57.531046+0000","last_clean_scrub_stamp":"2024-09-16T15:46:57.531046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T02:09:19.272595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.13","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-09-16T15:46:58.569492+0000","last_change":"2024-09-16T15:46:58.569492+0000","last_active":"2024-09-16T15:46:58.541934+0000","last_peered":"2024-09-16T15:46:58.541934+0000","last_clean":"2024-09-16T15:46:58.541934+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:46:58.569492+0000","last_undegraded":"2024-09-16T15:46:58.569492+0000","last_fullsized":"2024-09-16T15:46:58.569492+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_clean_scrub_stamp":"2024-09-16T15:46:58.541934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.12","version":"0'0","reported_seq":10,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-16T15:46:58.602887+0000","last_change":"2024-09-16T15:46:58.595177+0000","last_active":"2024-09-16T15:46:58.602887+0000","last_peered":"2024-09-16T15:46:58.602887+0000","last_clean":"2024-09-16T15:46:58.602887+0000","last_became_active":"2024-09-16T15:46:58.593818+0000","last_became_peered":"2024-09-16T15:46:58.593818+0000","last_unstale":"2024-09-16T15:46:58.602887+0000","last_undegraded":"2024-09-16T15:46:58.602887+0000","last_fullsized":"2024-09-16T15:46:58.602887+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:57.531046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:57.531046+0000","last_clean_scrub_stamp":"2024-09-16T15:46:57.531046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T01:59:03.824917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.10","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-09-16T15:46:58.562563+0000","last_change":"2024-09-16T15:46:58.562563+0000","last_active":"2024-09-16T15:46:58.541934+0000","last_peered":"2024-09-16T15:46:58.541934+0000","last_clean":"2024-09-16T15:46:58.541934+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:46:58.562563+0000","last_undegraded":"2024-09-16T15:46:58.562563+0000","last_fullsized":"2024-09-16T15:46:58.562563+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_clean_scrub_stamp":"2024-09-16T15:46:58.541934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.11","version":"0'0","reported_seq":10,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-16T15:46:58.595218+0000","last_change":"2024-09-16T15:46:58.584495+0000","last_active":"2024-09-16T15:46:58.595218+0000","last_peered":"2024-09-16T15:46:58.595218+0000","last_clean":"2024-09-16T15:46:58.595218+0000","last_became_active":"2024-09-16T15:46:58.583741+0000","last_became_peered":"2024-09-16T15:46:58.583741+0000","last_unstale":"2024-09-16T15:46:58.595218+0000","last_undegraded":"2024-09-16T15:46:58.595218+0000","last_fullsized":"2024-09-16T15:46:58.595218+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:57.531046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:57.531046+0000","last_clean_scrub_stamp":"2024-09-16T15:46:57.531046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T01:58:08.477199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.11","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-09-16T15:46:58.560818+0000","last_change":"2024-09-16T15:46:58.560818+0000","last_active":"2024-09-16T15:46:58.541934+0000","last_peered":"2024-09-16T15:46:58.541934+0000","last_clean":"2024-09-16T15:46:58.541934+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:46:58.560818+0000","last_undegraded":"2024-09-16T15:46:58.560818+0000","last_fullsized":"2024-09-16T15:46:58.560818+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_clean_scrub_stamp":"2024-09-16T15:46:58.541934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.10","version":"0'0","reported_seq":10,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-16T15:46:58.604683+0000","last_change":"2024-09-16T15:46:58.597568+0000","last_active":"2024-09-16T15:46:58.604683+0000","last_peered":"2024-09-16T15:46:58.604683+0000","last_clean":"2024-09-16T15:46:58.604683+0000","last_became_active":"2024-09-16T15:46:58.596806+0000","last_became_peered":"2024-09-16T15:46:58.596806+0000","last_unstale":"2024-09-16T15:46:58.604683+0000","last_undegraded":"2024-09-16T15:46:58.604683+0000","last_fullsized":"2024-09-16T15:46:58.604683+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:57.531046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:57.531046+0000","last_clean_scrub_stamp":"2024-09-16T15:46:57.531046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:33:54.157169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.e","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-09-16T15:46:58.561429+0000","last_change":"2024-09-16T15:46:58.561429+0000","last_active":"2024-09-16T15:46:58.541934+0000","last_peered":"2024-09-16T15:46:58.541934+0000","last_clean":"2024-09-16T15:46:58.541934+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:46:58.561429+0000","last_undegraded":"2024-09-16T15:46:58.561429+0000","last_fullsized":"2024-09-16T15:46:58.561429+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_clean_scrub_stamp":"2024-09-16T15:46:58.541934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.f","version":"0'0","reported_seq":10,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-16T15:46:58.602564+0000","last_change":"2024-09-16T15:46:58.595767+0000","last_active":"2024-09-16T15:46:58.602564+0000","last_peered":"2024-09-16T15:46:58.602564+0000","last_clean":"2024-09-16T15:46:58.602564+0000","last_became_active":"2024-09-16T15:46:58.594655+0000","last_became_peered":"2024-09-16T15:46:58.594655+0000","last_unstale":"2024-09-16T15:46:58.602564+0000","last_undegraded":"2024-09-16T15:46:58.602564+0000","last_fullsized":"2024-09-16T15:46:58.602564+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:57.531046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:57.531046+0000","last_clean_scrub_stamp":"2024-09-16T15:46:57.531046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:56:28.530413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.f","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-09-16T15:46:58.578428+0000","last_change":"2024-09-16T15:46:58.578428+0000","last_active":"2024-09-16T15:46:58.541934+0000","last_peered":"2024-09-16T15:46:58.541934+0000","last_clean":"2024-09-16T15:46:58.541934+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:46:58.578428+0000","last_undegraded":"2024-09-16T15:46:58.578428+0000","last_fullsized":"2024-09-16T15:46:58.578428+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_clean_scrub_stamp":"2024-09-16T15:46:58.541934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.e","version":"0'0","reported_seq":10,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-16T15:46:58.604653+0000","last_change":"2024-09-16T15:46:58.597152+0000","last_active":"2024-09-16T15:46:58.604653+0000","last_peered":"2024-09-16T15:46:58.604653+0000","last_clean":"2024-09-16T15:46:58.604653+0000","last_became_active":"2024-09-16T15:46:58.596435+0000","last_became_peered":"2024-09-16T15:46:58.596435+0000","last_unstale":"2024-09-16T15:46:58.604653+0000","last_undegraded":"2024-09-16T15:46:58.604653+0000","last_fullsized":"2024-09-16T15:46:58.604653+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:57.531046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:57.531046+0000","last_clean_scrub_stamp":"2024-09-16T15:46:57.531046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:11:50.475616+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.c","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-09-16T15:46:58.559398+0000","last_change":"2024-09-16T15:46:58.559398+0000","last_active":"2024-09-16T15:46:58.541934+0000","last_peered":"2024-09-16T15:46:58.541934+0000","last_clean":"2024-09-16T15:46:58.541934+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:46:58.559398+0000","last_undegraded":"2024-09-16T15:46:58.559398+0000","last_fullsized":"2024-09-16T15:46:58.559398+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_clean_scrub_stamp":"2024-09-16T15:46:58.541934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.d","version":"0'0","reported_seq":10,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-16T15:46:58.602764+0000","last_change":"2024-09-16T15:46:58.598349+0000","last_active":"2024-09-16T15:46:58.602764+0000","last_peered":"2024-09-16T15:46:58.602764+0000","last_clean":"2024-09-16T15:46:58.602764+0000","last_became_active":"2024-09-16T15:46:58.595349+0000","last_became_peered":"2024-09-16T15:46:58.595349+0000","last_unstale":"2024-09-16T15:46:58.602764+0000","last_undegraded":"2024-09-16T15:46:58.602764+0000","last_fullsized":"2024-09-16T15:46:58.602764+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:57.531046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:57.531046+0000","last_clean_scrub_stamp":"2024-09-16T15:46:57.531046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T01:43:29.245355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.d","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-09-16T15:46:58.563246+0000","last_change":"2024-09-16T15:46:58.563246+0000","last_active":"2024-09-16T15:46:58.541934+0000","last_peered":"2024-09-16T15:46:58.541934+0000","last_clean":"2024-09-16T15:46:58.541934+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:46:58.563246+0000","last_undegraded":"2024-09-16T15:46:58.563246+0000","last_fullsized":"2024-09-16T15:46:58.563246+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_clean_scrub_stamp":"2024-09-16T15:46:58.541934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.c","version":"0'0","reported_seq":10,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-16T15:46:58.599912+0000","last_change":"2024-09-16T15:46:58.596863+0000","last_active":"2024-09-16T15:46:58.599912+0000","last_peered":"2024-09-16T15:46:58.599912+0000","last_clean":"2024-09-16T15:46:58.599912+0000","last_became_active":"2024-09-16T15:46:58.596143+0000","last_became_peered":"2024-09-16T15:46:58.596143+0000","last_unstale":"2024-09-16T15:46:58.599912+0000","last_undegraded":"2024-09-16T15:46:58.599912+0000","last_fullsized":"2024-09-16T15:46:58.599912+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:57.531046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:57.531046+0000","last_clean_scrub_stamp":"2024-09-16T15:46:57.531046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:50:55.009663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.b","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-09-16T15:46:58.567734+0000","last_change":"2024-09-16T15:46:58.567734+0000","last_active":"2024-09-16T15:46:58.541934+0000","last_peered":"2024-09-16T15:46:58.541934+0000","last_clean":"2024-09-16T15:46:58.541934+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:46:58.567734+0000","last_undegraded":"2024-09-16T15:46:58.567734+0000","last_fullsized":"2024-09-16T15:46:58.567734+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_clean_scrub_stamp":"2024-09-16T15:46:58.541934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.a","version":"0'0","reported_seq":10,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-16T15:46:58.597818+0000","last_change":"2024-09-16T15:46:58.585877+0000","last_active":"2024-09-16T15:46:58.597818+0000","last_peered":"2024-09-16T15:46:58.597818+0000","last_clean":"2024-09-16T15:46:58.597818+0000","last_became_active":"2024-09-16T15:46:58.585158+0000","last_became_peered":"2024-09-16T15:46:58.585158+0000","last_unstale":"2024-09-16T15:46:58.597818+0000","last_undegraded":"2024-09-16T15:46:58.597818+0000","last_fullsized":"2024-09-16T15:46:58.597818+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:57.531046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:57.531046+0000","last_clean_scrub_stamp":"2024-09-16T15:46:57.531046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T01:00:40.058766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.8","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-09-16T15:46:58.564210+0000","last_change":"2024-09-16T15:46:58.564210+0000","last_active":"2024-09-16T15:46:58.541934+0000","last_peered":"2024-09-16T15:46:58.541934+0000","last_clean":"2024-09-16T15:46:58.541934+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:46:58.564210+0000","last_undegraded":"2024-09-16T15:46:58.564210+0000","last_fullsized":"2024-09-16T15:46:58.564210+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_clean_scrub_stamp":"2024-09-16T15:46:58.541934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.9","version":"0'0","reported_seq":10,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-16T15:46:58.583297+0000","last_change":"2024-09-16T15:46:58.576781+0000","last_active":"2024-09-16T15:46:58.583297+0000","last_peered":"2024-09-16T15:46:58.583297+0000","last_clean":"2024-09-16T15:46:58.583297+0000","last_became_active":"2024-09-16T15:46:58.576081+0000","last_became_peered":"2024-09-16T15:46:58.576081+0000","last_unstale":"2024-09-16T15:46:58.583297+0000","last_undegraded":"2024-09-16T15:46:58.583297+0000","last_fullsized":"2024-09-16T15:46:58.583297+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:57.531046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:57.531046+0000","last_clean_scrub_stamp":"2024-09-16T15:46:57.531046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T22:41:00.960336+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.9","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-09-16T15:46:58.564335+0000","last_change":"2024-09-16T15:46:58.564335+0000","last_active":"2024-09-16T15:46:58.541934+0000","last_peered":"2024-09-16T15:46:58.541934+0000","last_clean":"2024-09-16T15:46:58.541934+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:46:58.564335+0000","last_undegraded":"2024-09-16T15:46:58.564335+0000","last_fullsized":"2024-09-16T15:46:58.564335+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_clean_scrub_stamp":"2024-09-16T15:46:58.541934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.8","version":"0'0","reported_seq":10,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-16T15:46:58.577818+0000","last_change":"2024-09-16T15:46:58.572736+0000","last_active":"2024-09-16T15:46:58.577818+0000","last_peered":"2024-09-16T15:46:58.577818+0000","last_clean":"2024-09-16T15:46:58.577818+0000","last_became_active":"2024-09-16T15:46:58.572311+0000","last_became_peered":"2024-09-16T15:46:58.572311+0000","last_unstale":"2024-09-16T15:46:58.577818+0000","last_undegraded":"2024-09-16T15:46:58.577818+0000","last_fullsized":"2024-09-16T15:46:58.577818+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:57.531046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:57.531046+0000","last_clean_scrub_stamp":"2024-09-16T15:46:57.531046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:48:10.471983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.6","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-09-16T15:46:58.562758+0000","last_change":"2024-09-16T15:46:58.562758+0000","last_active":"2024-09-16T15:46:58.541934+0000","last_peered":"2024-09-16T15:46:58.541934+0000","last_clean":"2024-09-16T15:46:58.541934+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:46:58.562758+0000","last_undegraded":"2024-09-16T15:46:58.562758+0000","last_fullsized":"2024-09-16T15:46:58.562758+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_clean_scrub_stamp":"2024-09-16T15:46:58.541934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.7","version":"0'0","reported_seq":10,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-16T15:46:58.594095+0000","last_change":"2024-09-16T15:46:58.579355+0000","last_active":"2024-09-16T15:46:58.594095+0000","last_peered":"2024-09-16T15:46:58.594095+0000","last_clean":"2024-09-16T15:46:58.594095+0000","last_became_active":"2024-09-16T15:46:58.578380+0000","last_became_peered":"2024-09-16T15:46:58.578380+0000","last_unstale":"2024-09-16T15:46:58.594095+0000","last_undegraded":"2024-09-16T15:46:58.594095+0000","last_fullsized":"2024-09-16T15:46:58.594095+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:57.531046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:57.531046+0000","last_clean_scrub_stamp":"2024-09-16T15:46:57.531046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T23:40:10.473815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.7","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-09-16T15:46:58.569698+0000","last_change":"2024-09-16T15:46:58.569698+0000","last_active":"2024-09-16T15:46:58.541934+0000","last_peered":"2024-09-16T15:46:58.541934+0000","last_clean":"2024-09-16T15:46:58.541934+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:46:58.569698+0000","last_undegraded":"2024-09-16T15:46:58.569698+0000","last_fullsized":"2024-09-16T15:46:58.569698+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_clean_scrub_stamp":"2024-09-16T15:46:58.541934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.6","version":"0'0","reported_seq":10,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-16T15:46:58.582810+0000","last_change":"2024-09-16T15:46:58.577011+0000","last_active":"2024-09-16T15:46:58.582810+0000","last_peered":"2024-09-16T15:46:58.582810+0000","last_clean":"2024-09-16T15:46:58.582810+0000","last_became_active":"2024-09-16T15:46:58.576294+0000","last_became_peered":"2024-09-16T15:46:58.576294+0000","last_unstale":"2024-09-16T15:46:58.582810+0000","last_undegraded":"2024-09-16T15:46:58.582810+0000","last_fullsized":"2024-09-16T15:46:58.582810+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:57.531046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:57.531046+0000","last_clean_scrub_stamp":"2024-09-16T15:46:57.531046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T03:31:45.091717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.4","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-09-16T15:46:58.574408+0000","last_change":"2024-09-16T15:46:58.574408+0000","last_active":"2024-09-16T15:46:58.541934+0000","last_peered":"2024-09-16T15:46:58.541934+0000","last_clean":"2024-09-16T15:46:58.541934+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:46:58.574408+0000","last_undegraded":"2024-09-16T15:46:58.574408+0000","last_fullsized":"2024-09-16T15:46:58.574408+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_clean_scrub_stamp":"2024-09-16T15:46:58.541934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.5","version":"0'0","reported_seq":10,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-16T15:46:58.578312+0000","last_change":"2024-09-16T15:46:58.575511+0000","last_active":"2024-09-16T15:46:58.578312+0000","last_peered":"2024-09-16T15:46:58.578312+0000","last_clean":"2024-09-16T15:46:58.578312+0000","last_became_active":"2024-09-16T15:46:58.575218+0000","last_became_peered":"2024-09-16T15:46:58.575218+0000","last_unstale":"2024-09-16T15:46:58.578312+0000","last_undegraded":"2024-09-16T15:46:58.578312+0000","last_fullsized":"2024-09-16T15:46:58.578312+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:57.531046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:57.531046+0000","last_clean_scrub_stamp":"2024-09-16T15:46:57.531046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:37:03.438301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.5","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-09-16T15:46:58.578126+0000","last_change":"2024-09-16T15:46:58.578126+0000","last_active":"2024-09-16T15:46:58.541934+0000","last_peered":"2024-09-16T15:46:58.541934+0000","last_clean":"2024-09-16T15:46:58.541934+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:46:58.578126+0000","last_undegraded":"2024-09-16T15:46:58.578126+0000","last_fullsized":"2024-09-16T15:46:58.578126+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_clean_scrub_stamp":"2024-09-16T15:46:58.541934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.4","version":"0'0","reported_seq":10,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-16T15:46:58.588834+0000","last_change":"2024-09-16T15:46:58.583703+0000","last_active":"2024-09-16T15:46:58.588834+0000","last_peered":"2024-09-16T15:46:58.588834+0000","last_clean":"2024-09-16T15:46:58.588834+0000","last_became_active":"2024-09-16T15:46:58.583384+0000","last_became_peered":"2024-09-16T15:46:58.583384+0000","last_unstale":"2024-09-16T15:46:58.588834+0000","last_undegraded":"2024-09-16T15:46:58.588834+0000","last_fullsized":"2024-09-16T15:46:58.588834+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:57.531046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:57.531046+0000","last_clean_scrub_stamp":"2024-09-16T15:46:57.531046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T23:06:05.405748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.2","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-09-16T15:46:58.560788+0000","last_change":"2024-09-16T15:46:58.560788+0000","last_active":"2024-09-16T15:46:58.541934+0000","last_peered":"2024-09-16T15:46:58.541934+0000","last_clean":"2024-09-16T15:46:58.541934+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:46:58.560788+0000","last_undegraded":"2024-09-16T15:46:58.560788+0000","last_fullsized":"2024-09-16T15:46:58.560788+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_clean_scrub_stamp":"2024-09-16T15:46:58.541934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.3","version":"0'0","reported_seq":10,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-16T15:46:58.577602+0000","last_change":"2024-09-16T15:46:58.573698+0000","last_active":"2024-09-16T15:46:58.577602+0000","last_peered":"2024-09-16T15:46:58.577602+0000","last_clean":"2024-09-16T15:46:58.577602+0000","last_became_active":"2024-09-16T15:46:58.573391+0000","last_became_peered":"2024-09-16T15:46:58.573391+0000","last_unstale":"2024-09-16T15:46:58.577602+0000","last_undegraded":"2024-09-16T15:46:58.577602+0000","last_fullsized":"2024-09-16T15:46:58.577602+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:57.531046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:57.531046+0000","last_clean_scrub_stamp":"2024-09-16T15:46:57.531046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T00:58:45.221067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.3","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-09-16T15:46:58.586430+0000","last_change":"2024-09-16T15:46:58.586430+0000","last_active":"2024-09-16T15:46:58.541934+0000","last_peered":"2024-09-16T15:46:58.541934+0000","last_clean":"2024-09-16T15:46:58.541934+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:46:58.586430+0000","last_undegraded":"2024-09-16T15:46:58.586430+0000","last_fullsized":"2024-09-16T15:46:58.586430+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_clean_scrub_stamp":"2024-09-16T15:46:58.541934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.2","version":"0'0","reported_seq":10,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-16T15:46:58.577483+0000","last_change":"2024-09-16T15:46:58.574227+0000","last_active":"2024-09-16T15:46:58.577483+0000","last_peered":"2024-09-16T15:46:58.577483+0000","last_clean":"2024-09-16T15:46:58.577483+0000","last_became_active":"2024-09-16T15:46:58.573898+0000","last_became_peered":"2024-09-16T15:46:58.573898+0000","last_unstale":"2024-09-16T15:46:58.577483+0000","last_undegraded":"2024-09-16T15:46:58.577483+0000","last_fullsized":"2024-09-16T15:46:58.577483+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:57.531046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:57.531046+0000","last_clean_scrub_stamp":"2024-09-16T15:46:57.531046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T18:41:18.272460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.0","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-09-16T15:46:58.559237+0000","last_change":"2024-09-16T15:46:58.559237+0000","last_active":"2024-09-16T15:46:58.541934+0000","last_peered":"2024-09-16T15:46:58.541934+0000","last_clean":"2024-09-16T15:46:58.541934+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:46:58.559237+0000","last_undegraded":"2024-09-16T15:46:58.559237+0000","last_fullsized":"2024-09-16T15:46:58.559237+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_clean_scrub_stamp":"2024-09-16T15:46:58.541934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.1","version":"0'0","reported_seq":10,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-16T15:46:58.604741+0000","last_change":"2024-09-16T15:46:58.598614+0000","last_active":"2024-09-16T15:46:58.604741+0000","last_peered":"2024-09-16T15:46:58.604741+0000","last_clean":"2024-09-16T15:46:58.604741+0000","last_became_active":"2024-09-16T15:46:58.597950+0000","last_became_peered":"2024-09-16T15:46:58.597950+0000","last_unstale":"2024-09-16T15:46:58.604741+0000","last_undegraded":"2024-09-16T15:46:58.604741+0000","last_fullsized":"2024-09-16T15:46:58.604741+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:57.531046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:57.531046+0000","last_clean_scrub_stamp":"2024-09-16T15:46:57.531046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T23:50:07.979345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-09-16T15:46:58.565520+0000","last_change":"2024-09-16T15:46:58.565520+0000","last_active":"2024-09-16T15:46:58.541934+0000","last_peered":"2024-09-16T15:46:58.541934+0000","last_clean":"2024-09-16T15:46:58.541934+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:46:58.565520+0000","last_undegraded":"2024-09-16T15:46:58.565520+0000","last_fullsized":"2024-09-16T15:46:58.565520+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_clean_scrub_stamp":"2024-09-16T15:46:58.541934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.0","version":"0'0","reported_seq":10,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-16T15:46:58.602740+0000","last_change":"2024-09-16T15:46:58.599504+0000","last_active":"2024-09-16T15:46:58.602740+0000","last_peered":"2024-09-16T15:46:58.602740+0000","last_clean":"2024-09-16T15:46:58.602740+0000","last_became_active":"2024-09-16T15:46:58.598743+0000","last_became_peered":"2024-09-16T15:46:58.598743+0000","last_unstale":"2024-09-16T15:46:58.602740+0000","last_undegraded":"2024-09-16T15:46:58.602740+0000","last_fullsized":"2024-09-16T15:46:58.602740+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:57.531046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:57.531046+0000","last_clean_scrub_stamp":"2024-09-16T15:46:57.531046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T18:45:22.195277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.a","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-09-16T15:46:58.564305+0000","last_change":"2024-09-16T15:46:58.564305+0000","last_active":"2024-09-16T15:46:58.541934+0000","last_peered":"2024-09-16T15:46:58.541934+0000","last_clean":"2024-09-16T15:46:58.541934+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:46:58.564305+0000","last_undegraded":"2024-09-16T15:46:58.564305+0000","last_fullsized":"2024-09-16T15:46:58.564305+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_clean_scrub_stamp":"2024-09-16T15:46:58.541934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.b","version":"0'0","reported_seq":10,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-16T15:46:58.604289+0000","last_change":"2024-09-16T15:46:58.601513+0000","last_active":"2024-09-16T15:46:58.604289+0000","last_peered":"2024-09-16T15:46:58.604289+0000","last_clean":"2024-09-16T15:46:58.604289+0000","last_became_active":"2024-09-16T15:46:58.600581+0000","last_became_peered":"2024-09-16T15:46:58.600581+0000","last_unstale":"2024-09-16T15:46:58.604289+0000","last_undegraded":"2024-09-16T15:46:58.604289+0000","last_fullsized":"2024-09-16T15:46:58.604289+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:57.531046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:57.531046+0000","last_clean_scrub_stamp":"2024-09-16T15:46:57.531046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T23:00:41.658258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"16'72","reported_seq":200,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-16T15:46:58.585260+0000","last_change":"2024-09-16T15:42:40.881820+0000","last_active":"2024-09-16T15:46:58.585260+0000","last_peered":"2024-09-16T15:46:58.585260+0000","last_clean":"2024-09-16T15:46:58.585260+0000","last_became_active":"2024-09-16T15:42:40.879965+0000","last_became_peered":"2024-09-16T15:42:40.879965+0000","last_unstale":"2024-09-16T15:46:58.585260+0000","last_undegraded":"2024-09-16T15:46:58.585260+0000","last_fullsized":"2024-09-16T15:46:58.585260+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:42:39.728762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:42:39.728762+0000","last_clean_scrub_stamp":"2024-09-16T15:42:39.728762+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:22:18.768932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":11,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":10,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542200,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28416,"kb_used_data":1384,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343424,"statfs":{"total":96636764160,"available":96607666176,"internally_reserved":0,"allocated":1417216,"data_stored":932261,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542199,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31996,"kb_used_data":804,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339844,"statfs":{"total":96636764160,"available":96604000256,"internally_reserved":0,"allocated":823296,"data_stored":341893,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":5,"up_from":14,"seq":60129542200,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27824,"kb_used_data":792,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344016,"statfs":{"total":96636764160,"available":96608272384,"internally_reserved":0,"allocated":811008,"data_stored":335618,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542198,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31980,"kb_used_data":792,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339860,"statfs":{"total":96636764160,"available":96604016640,"internally_reserved":0,"allocated":811008,"data_stored":335618,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":3,"up_from":14,"seq":60129542199,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31996,"kb_used_data":804,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339844,"statfs":{"total":96636764160,"available":96604000256,"internally_reserved":0,"allocated":823296,"data_stored":341893,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":14,"seq":60129542199,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27820,"kb_used_data":792,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344020,"statfs":{"total":96636764160,"available":96608276480,"internally_reserved":0,"allocated":811008,"data_stored":335618,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":14,"seq":60129542201,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27840,"kb_used_data":804,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344000,"statfs":{"total":96636764160,"available":96608256000,"internally_reserved":0,"allocated":823296,"data_stored":341893,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":14,"seq":60129542201,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32572,"kb_used_data":1384,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339268,"statfs":{"total":96636764160,"available":96603410432,"internally_reserved":0,"allocated":1417216,"data_stored":932261,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T15:47:08.093 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T15:47:08.229 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:08.224+0000 7fee403f9640 1 -- 172.21.15.123:0/1836237406 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fee20003a60 msgr2=0x7fee20003ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:47:08.229 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:08.224+0000 7fee403f9640 1 --2- 172.21.15.123:0/1836237406 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fee20003a60 0x7fee20003ea0 secure :-1 s=READY pgs=797 cs=0 l=1 rev1=1 crypto rx=0x7fee280016f0 tx=0x7fee2801c740 comp rx=0 tx=0).stop 2024-09-16T15:47:08.229 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:08.224+0000 7fee403f9640 1 -- 172.21.15.123:0/1836237406 shutdown_connections 2024-09-16T15:47:08.229 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:08.224+0000 7fee403f9640 1 --2- 172.21.15.123:0/1836237406 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fee20003a60 0x7fee20003ea0 unknown :-1 s=CLOSED pgs=797 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:08.230 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:08.224+0000 7fee403f9640 1 -- 172.21.15.123:0/1836237406 >> 172.21.15.123:0/1836237406 conn(0x7fee380cfb10 msgr2=0x7fee380cff20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:47:08.230 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:08.224+0000 7fee403f9640 1 -- 172.21.15.123:0/1836237406 shutdown_connections 2024-09-16T15:47:08.230 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:08.224+0000 7fee403f9640 1 -- 172.21.15.123:0/1836237406 wait complete. 2024-09-16T15:47:08.230 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:08.224+0000 7fee403f9640 1 Processor -- start 2024-09-16T15:47:08.230 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:08.224+0000 7fee403f9640 1 -- start start 2024-09-16T15:47:08.232 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:08.228+0000 7fee403f9640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fee20003a60 0x7fee381541c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:47:08.232 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:08.228+0000 7fee403f9640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fee380c7f60 0x7fee380b5250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:47:08.232 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:08.228+0000 7fee403f9640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fee380b5790 0x7fee380c8c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:47:08.232 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:08.228+0000 7fee403f9640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fee3815ec30 con 0x7fee380c7f60 2024-09-16T15:47:08.232 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:08.228+0000 7fee403f9640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fee380cf640 con 0x7fee380b5790 2024-09-16T15:47:08.232 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:08.228+0000 7fee403f9640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fee3815edb0 con 0x7fee20003a60 2024-09-16T15:47:08.232 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:08.228+0000 7fee3d994640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fee380c7f60 0x7fee380b5250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:47:08.232 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:08.228+0000 7fee3d994640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fee380c7f60 0x7fee380b5250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:46902/0 (socket says 172.21.15.123:46902) 2024-09-16T15:47:08.233 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:08.228+0000 7fee3d994640 1 -- 172.21.15.123:0/3168705534 learned_addr learned my addr 172.21.15.123:0/3168705534 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:47:08.233 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:08.228+0000 7fee3e195640 1 --2- 172.21.15.123:0/3168705534 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fee20003a60 0x7fee381541c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:47:08.233 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:08.228+0000 7fee3e996640 1 --2- 172.21.15.123:0/3168705534 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fee380b5790 0x7fee380c8c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:47:08.233 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:08.228+0000 7fee3d994640 1 -- 172.21.15.123:0/3168705534 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fee20003a60 msgr2=0x7fee381541c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:47:08.233 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:08.228+0000 7fee3d994640 1 --2- 172.21.15.123:0/3168705534 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fee20003a60 0x7fee381541c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:08.233 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:08.228+0000 7fee3d994640 1 -- 172.21.15.123:0/3168705534 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fee380b5790 msgr2=0x7fee380c8c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:47:08.233 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:08.228+0000 7fee3d994640 1 --2- 172.21.15.123:0/3168705534 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fee380b5790 0x7fee380c8c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:08.233 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:08.228+0000 7fee3d994640 1 -- 172.21.15.123:0/3168705534 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fee3814f570 con 0x7fee380c7f60 2024-09-16T15:47:08.233 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:08.228+0000 7fee3e195640 1 --2- 172.21.15.123:0/3168705534 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fee20003a60 0x7fee381541c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T15:47:08.233 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:08.228+0000 7fee3d994640 1 --2- 172.21.15.123:0/3168705534 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fee380c7f60 0x7fee380b5250 secure :-1 s=READY pgs=798 cs=0 l=1 rev1=1 crypto rx=0x7fee2c00e700 tx=0x7fee2c00ebc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:47:08.233 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:08.228+0000 7fee36ffd640 1 -- 172.21.15.123:0/3168705534 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee2c00ca50 con 0x7fee380c7f60 2024-09-16T15:47:08.233 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:08.228+0000 7fee36ffd640 1 -- 172.21.15.123:0/3168705534 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fee2c00cce0 con 0x7fee380c7f60 2024-09-16T15:47:08.233 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:08.228+0000 7fee403f9640 1 -- 172.21.15.123:0/3168705534 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fee380c6be0 con 0x7fee380c7f60 2024-09-16T15:47:08.233 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:08.228+0000 7fee36ffd640 1 -- 172.21.15.123:0/3168705534 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee2c0116d0 con 0x7fee380c7f60 2024-09-16T15:47:08.234 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:08.228+0000 7fee403f9640 1 -- 172.21.15.123:0/3168705534 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fee380c71a0 con 0x7fee380c7f60 2024-09-16T15:47:08.234 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:08.228+0000 7fee403f9640 1 -- 172.21.15.123:0/3168705534 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fee3815f310 con 0x7fee380c7f60 2024-09-16T15:47:08.234 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:08.228+0000 7fee36ffd640 1 -- 172.21.15.123:0/3168705534 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 11) ==== 97386+0+0 (secure 0 0 0) 0x7fee2c011870 con 0x7fee380c7f60 2024-09-16T15:47:08.234 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:08.228+0000 7fee36ffd640 1 --2- 172.21.15.123:0/3168705534 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fee24073220 0x7fee240756e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:47:08.235 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:08.228+0000 7fee36ffd640 1 -- 172.21.15.123:0/3168705534 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(70..70 src has 1..70) ==== 5709+0+0 (secure 0 0 0) 0x7fee2c094f00 con 0x7fee380c7f60 2024-09-16T15:47:08.235 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:08.228+0000 7fee3e195640 1 --2- 172.21.15.123:0/3168705534 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fee24073220 0x7fee240756e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:47:08.235 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:08.232+0000 7fee3e195640 1 --2- 172.21.15.123:0/3168705534 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fee24073220 0x7fee240756e0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7fee280016c0 tx=0x7fee280a7040 comp rx=0 tx=0).ready entity=mgr.4135 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:47:08.238 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:08.232+0000 7fee36ffd640 1 -- 172.21.15.123:0/3168705534 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7fee3814f570 con 0x7fee380c7f60 2024-09-16T15:47:08.411 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:08.408+0000 7fee403f9640 1 -- 172.21.15.123:0/3168705534 --> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fee380af7f0 con 0x7fee24073220 2024-09-16T15:47:08.422 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:08.416+0000 7fee36ffd640 1 -- 172.21.15.123:0/3168705534 <== mgr.4135 v2:172.21.15.184:6832/3114346423 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+252058 (secure 0 0 0) 0x7fee380af7f0 con 0x7fee24073220 2024-09-16T15:47:08.425 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-09-16T15:47:08.427 INFO:teuthology.orchestra.run.smithi123.stderr:dumped all 2024-09-16T15:47:08.427 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:08.420+0000 7fee403f9640 1 -- 172.21.15.123:0/3168705534 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fee24073220 msgr2=0x7fee240756e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:47:08.427 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:08.420+0000 7fee403f9640 1 --2- 172.21.15.123:0/3168705534 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fee24073220 0x7fee240756e0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7fee280016c0 tx=0x7fee280a7040 comp rx=0 tx=0).stop 2024-09-16T15:47:08.428 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:08.420+0000 7fee403f9640 1 -- 172.21.15.123:0/3168705534 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fee380c7f60 msgr2=0x7fee380b5250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:47:08.428 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:08.420+0000 7fee403f9640 1 --2- 172.21.15.123:0/3168705534 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fee380c7f60 0x7fee380b5250 secure :-1 s=READY pgs=798 cs=0 l=1 rev1=1 crypto rx=0x7fee2c00e700 tx=0x7fee2c00ebc0 comp rx=0 tx=0).stop 2024-09-16T15:47:08.428 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:08.424+0000 7fee403f9640 1 -- 172.21.15.123:0/3168705534 shutdown_connections 2024-09-16T15:47:08.428 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:08.424+0000 7fee403f9640 1 --2- 172.21.15.123:0/3168705534 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fee24073220 0x7fee240756e0 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:08.428 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:08.424+0000 7fee403f9640 1 --2- 172.21.15.123:0/3168705534 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fee380b5790 0x7fee380c8c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:08.428 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:08.424+0000 7fee403f9640 1 --2- 172.21.15.123:0/3168705534 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fee380c7f60 0x7fee380b5250 unknown :-1 s=CLOSED pgs=798 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:08.428 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:08.424+0000 7fee403f9640 1 --2- 172.21.15.123:0/3168705534 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fee20003a60 0x7fee381541c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:08.428 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:08.424+0000 7fee403f9640 1 -- 172.21.15.123:0/3168705534 >> 172.21.15.123:0/3168705534 conn(0x7fee380cfb10 msgr2=0x7fee380cc390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:47:08.428 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:08.424+0000 7fee403f9640 1 -- 172.21.15.123:0/3168705534 shutdown_connections 2024-09-16T15:47:08.428 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:08.424+0000 7fee403f9640 1 -- 172.21.15.123:0/3168705534 wait complete. 2024-09-16T15:47:08.444 INFO:teuthology.orchestra.run.smithi123.stdout:{"pg_ready":true,"pg_map":{"version":197,"stamp":"2024-09-16T15:47:07.332904+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":146,"num_write_kb":1642,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":111,"ondisk_log_size":111,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":240724,"kb_used_data":7848,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":754733996,"statfs":{"total":773094113280,"available":772847611904,"internally_reserved":0,"allocated":8036352,"data_stored":4025300,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":238276145},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":43,"apply_latency_ms":43,"commit_latency_ns":43000000,"apply_latency_ns":43000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.003231"},"pg_stats":[{"pgid":"11.3f","version":"0'0","reported_seq":13,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-09-16T15:47:01.271396+0000","last_change":"2024-09-16T15:46:59.595668+0000","last_active":"2024-09-16T15:47:01.271396+0000","last_peered":"2024-09-16T15:47:01.271396+0000","last_clean":"2024-09-16T15:47:01.271396+0000","last_became_active":"2024-09-16T15:46:59.595332+0000","last_became_peered":"2024-09-16T15:46:59.595332+0000","last_unstale":"2024-09-16T15:47:01.271396+0000","last_undegraded":"2024-09-16T15:47:01.271396+0000","last_fullsized":"2024-09-16T15:47:01.271396+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_clean_scrub_stamp":"2024-09-16T15:46:58.541934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T01:42:52.699759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.3e","version":"0'0","reported_seq":13,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-09-16T15:47:00.600146+0000","last_change":"2024-09-16T15:46:59.588019+0000","last_active":"2024-09-16T15:47:00.600146+0000","last_peered":"2024-09-16T15:47:00.600146+0000","last_clean":"2024-09-16T15:47:00.600146+0000","last_became_active":"2024-09-16T15:46:59.587686+0000","last_became_peered":"2024-09-16T15:46:59.587686+0000","last_unstale":"2024-09-16T15:47:00.600146+0000","last_undegraded":"2024-09-16T15:47:00.600146+0000","last_fullsized":"2024-09-16T15:47:00.600146+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_clean_scrub_stamp":"2024-09-16T15:46:58.541934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:43:11.396171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.3d","version":"0'0","reported_seq":13,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-09-16T15:47:02.581386+0000","last_change":"2024-09-16T15:46:59.589182+0000","last_active":"2024-09-16T15:47:02.581386+0000","last_peered":"2024-09-16T15:47:02.581386+0000","last_clean":"2024-09-16T15:47:02.581386+0000","last_became_active":"2024-09-16T15:46:59.588849+0000","last_became_peered":"2024-09-16T15:46:59.588849+0000","last_unstale":"2024-09-16T15:47:02.581386+0000","last_undegraded":"2024-09-16T15:47:02.581386+0000","last_fullsized":"2024-09-16T15:47:02.581386+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_clean_scrub_stamp":"2024-09-16T15:46:58.541934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:44:04.505297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.3c","version":"0'0","reported_seq":13,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-09-16T15:47:03.017731+0000","last_change":"2024-09-16T15:46:59.608887+0000","last_active":"2024-09-16T15:47:03.017731+0000","last_peered":"2024-09-16T15:47:03.017731+0000","last_clean":"2024-09-16T15:47:03.017731+0000","last_became_active":"2024-09-16T15:46:59.608597+0000","last_became_peered":"2024-09-16T15:46:59.608597+0000","last_unstale":"2024-09-16T15:47:03.017731+0000","last_undegraded":"2024-09-16T15:47:03.017731+0000","last_fullsized":"2024-09-16T15:47:03.017731+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_clean_scrub_stamp":"2024-09-16T15:46:58.541934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:43:45.935372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.3b","version":"0'0","reported_seq":13,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-09-16T15:47:00.595925+0000","last_change":"2024-09-16T15:46:59.594277+0000","last_active":"2024-09-16T15:47:00.595925+0000","last_peered":"2024-09-16T15:47:00.595925+0000","last_clean":"2024-09-16T15:47:00.595925+0000","last_became_active":"2024-09-16T15:46:59.593530+0000","last_became_peered":"2024-09-16T15:46:59.593530+0000","last_unstale":"2024-09-16T15:47:00.595925+0000","last_undegraded":"2024-09-16T15:47:00.595925+0000","last_fullsized":"2024-09-16T15:47:00.595925+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_clean_scrub_stamp":"2024-09-16T15:46:58.541934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T02:09:20.283484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.3a","version":"0'0","reported_seq":13,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-09-16T15:47:00.598931+0000","last_change":"2024-09-16T15:46:59.598279+0000","last_active":"2024-09-16T15:47:00.598931+0000","last_peered":"2024-09-16T15:47:00.598931+0000","last_clean":"2024-09-16T15:47:00.598931+0000","last_became_active":"2024-09-16T15:46:59.597118+0000","last_became_peered":"2024-09-16T15:46:59.597118+0000","last_unstale":"2024-09-16T15:47:00.598931+0000","last_undegraded":"2024-09-16T15:47:00.598931+0000","last_fullsized":"2024-09-16T15:47:00.598931+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_clean_scrub_stamp":"2024-09-16T15:46:58.541934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T18:01:47.484827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.39","version":"0'0","reported_seq":13,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-09-16T15:47:01.595545+0000","last_change":"2024-09-16T15:46:59.608991+0000","last_active":"2024-09-16T15:47:01.595545+0000","last_peered":"2024-09-16T15:47:01.595545+0000","last_clean":"2024-09-16T15:47:01.595545+0000","last_became_active":"2024-09-16T15:46:59.608701+0000","last_became_peered":"2024-09-16T15:46:59.608701+0000","last_unstale":"2024-09-16T15:47:01.595545+0000","last_undegraded":"2024-09-16T15:47:01.595545+0000","last_fullsized":"2024-09-16T15:47:01.595545+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_clean_scrub_stamp":"2024-09-16T15:46:58.541934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:59:00.964808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.38","version":"0'0","reported_seq":13,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-09-16T15:47:00.598374+0000","last_change":"2024-09-16T15:46:59.598978+0000","last_active":"2024-09-16T15:47:00.598374+0000","last_peered":"2024-09-16T15:47:00.598374+0000","last_clean":"2024-09-16T15:47:00.598374+0000","last_became_active":"2024-09-16T15:46:59.598683+0000","last_became_peered":"2024-09-16T15:46:59.598683+0000","last_unstale":"2024-09-16T15:47:00.598374+0000","last_undegraded":"2024-09-16T15:47:00.598374+0000","last_fullsized":"2024-09-16T15:47:00.598374+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_clean_scrub_stamp":"2024-09-16T15:46:58.541934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:50:36.734012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.37","version":"0'0","reported_seq":13,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-09-16T15:47:00.596834+0000","last_change":"2024-09-16T15:46:59.588040+0000","last_active":"2024-09-16T15:47:00.596834+0000","last_peered":"2024-09-16T15:47:00.596834+0000","last_clean":"2024-09-16T15:47:00.596834+0000","last_became_active":"2024-09-16T15:46:59.587424+0000","last_became_peered":"2024-09-16T15:46:59.587424+0000","last_unstale":"2024-09-16T15:47:00.596834+0000","last_undegraded":"2024-09-16T15:47:00.596834+0000","last_fullsized":"2024-09-16T15:47:00.596834+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_clean_scrub_stamp":"2024-09-16T15:46:58.541934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:20:33.007622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.36","version":"0'0","reported_seq":13,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-09-16T15:47:00.599901+0000","last_change":"2024-09-16T15:46:59.585701+0000","last_active":"2024-09-16T15:47:00.599901+0000","last_peered":"2024-09-16T15:47:00.599901+0000","last_clean":"2024-09-16T15:47:00.599901+0000","last_became_active":"2024-09-16T15:46:59.584949+0000","last_became_peered":"2024-09-16T15:46:59.584949+0000","last_unstale":"2024-09-16T15:47:00.599901+0000","last_undegraded":"2024-09-16T15:47:00.599901+0000","last_fullsized":"2024-09-16T15:47:00.599901+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_clean_scrub_stamp":"2024-09-16T15:46:58.541934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:19:30.848152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.35","version":"0'0","reported_seq":13,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-09-16T15:47:01.271145+0000","last_change":"2024-09-16T15:46:59.595162+0000","last_active":"2024-09-16T15:47:01.271145+0000","last_peered":"2024-09-16T15:47:01.271145+0000","last_clean":"2024-09-16T15:47:01.271145+0000","last_became_active":"2024-09-16T15:46:59.594591+0000","last_became_peered":"2024-09-16T15:46:59.594591+0000","last_unstale":"2024-09-16T15:47:01.271145+0000","last_undegraded":"2024-09-16T15:47:01.271145+0000","last_fullsized":"2024-09-16T15:47:01.271145+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_clean_scrub_stamp":"2024-09-16T15:46:58.541934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T18:41:59.567379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.34","version":"0'0","reported_seq":13,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-09-16T15:47:03.017602+0000","last_change":"2024-09-16T15:46:59.588609+0000","last_active":"2024-09-16T15:47:03.017602+0000","last_peered":"2024-09-16T15:47:03.017602+0000","last_clean":"2024-09-16T15:47:03.017602+0000","last_became_active":"2024-09-16T15:46:59.588355+0000","last_became_peered":"2024-09-16T15:46:59.588355+0000","last_unstale":"2024-09-16T15:47:03.017602+0000","last_undegraded":"2024-09-16T15:47:03.017602+0000","last_fullsized":"2024-09-16T15:47:03.017602+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_clean_scrub_stamp":"2024-09-16T15:46:58.541934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:42:33.375890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.33","version":"0'0","reported_seq":13,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-09-16T15:47:01.601859+0000","last_change":"2024-09-16T15:46:59.605745+0000","last_active":"2024-09-16T15:47:01.601859+0000","last_peered":"2024-09-16T15:47:01.601859+0000","last_clean":"2024-09-16T15:47:01.601859+0000","last_became_active":"2024-09-16T15:46:59.605022+0000","last_became_peered":"2024-09-16T15:46:59.605022+0000","last_unstale":"2024-09-16T15:47:01.601859+0000","last_undegraded":"2024-09-16T15:47:01.601859+0000","last_fullsized":"2024-09-16T15:47:01.601859+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_clean_scrub_stamp":"2024-09-16T15:46:58.541934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T01:17:26.460923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.32","version":"0'0","reported_seq":13,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-09-16T15:47:00.595946+0000","last_change":"2024-09-16T15:46:59.595493+0000","last_active":"2024-09-16T15:47:00.595946+0000","last_peered":"2024-09-16T15:47:00.595946+0000","last_clean":"2024-09-16T15:47:00.595946+0000","last_became_active":"2024-09-16T15:46:59.595174+0000","last_became_peered":"2024-09-16T15:46:59.595174+0000","last_unstale":"2024-09-16T15:47:00.595946+0000","last_undegraded":"2024-09-16T15:47:00.595946+0000","last_fullsized":"2024-09-16T15:47:00.595946+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_clean_scrub_stamp":"2024-09-16T15:46:58.541934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T02:34:49.782229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.31","version":"0'0","reported_seq":13,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-09-16T15:47:01.601537+0000","last_change":"2024-09-16T15:46:59.597495+0000","last_active":"2024-09-16T15:47:01.601537+0000","last_peered":"2024-09-16T15:47:01.601537+0000","last_clean":"2024-09-16T15:47:01.601537+0000","last_became_active":"2024-09-16T15:46:59.596296+0000","last_became_peered":"2024-09-16T15:46:59.596296+0000","last_unstale":"2024-09-16T15:47:01.601537+0000","last_undegraded":"2024-09-16T15:47:01.601537+0000","last_fullsized":"2024-09-16T15:47:01.601537+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_clean_scrub_stamp":"2024-09-16T15:46:58.541934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T18:32:04.675615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.30","version":"0'0","reported_seq":13,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-09-16T15:47:00.596750+0000","last_change":"2024-09-16T15:46:59.600736+0000","last_active":"2024-09-16T15:47:00.596750+0000","last_peered":"2024-09-16T15:47:00.596750+0000","last_clean":"2024-09-16T15:47:00.596750+0000","last_became_active":"2024-09-16T15:46:59.600165+0000","last_became_peered":"2024-09-16T15:46:59.600165+0000","last_unstale":"2024-09-16T15:47:00.596750+0000","last_undegraded":"2024-09-16T15:47:00.596750+0000","last_fullsized":"2024-09-16T15:47:00.596750+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_clean_scrub_stamp":"2024-09-16T15:46:58.541934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:35:54.053055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.2f","version":"0'0","reported_seq":13,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-09-16T15:47:00.596671+0000","last_change":"2024-09-16T15:46:59.587704+0000","last_active":"2024-09-16T15:47:00.596671+0000","last_peered":"2024-09-16T15:47:00.596671+0000","last_clean":"2024-09-16T15:47:00.596671+0000","last_became_active":"2024-09-16T15:46:59.587057+0000","last_became_peered":"2024-09-16T15:46:59.587057+0000","last_unstale":"2024-09-16T15:47:00.596671+0000","last_undegraded":"2024-09-16T15:47:00.596671+0000","last_fullsized":"2024-09-16T15:47:00.596671+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_clean_scrub_stamp":"2024-09-16T15:46:58.541934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T18:23:27.562527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.2e","version":"0'0","reported_seq":13,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-09-16T15:47:03.017622+0000","last_change":"2024-09-16T15:46:59.600335+0000","last_active":"2024-09-16T15:47:03.017622+0000","last_peered":"2024-09-16T15:47:03.017622+0000","last_clean":"2024-09-16T15:47:03.017622+0000","last_became_active":"2024-09-16T15:46:59.600046+0000","last_became_peered":"2024-09-16T15:46:59.600046+0000","last_unstale":"2024-09-16T15:47:03.017622+0000","last_undegraded":"2024-09-16T15:47:03.017622+0000","last_fullsized":"2024-09-16T15:47:03.017622+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_clean_scrub_stamp":"2024-09-16T15:46:58.541934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:02:46.535574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.2d","version":"0'0","reported_seq":13,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-09-16T15:47:00.598432+0000","last_change":"2024-09-16T15:46:59.598795+0000","last_active":"2024-09-16T15:47:00.598432+0000","last_peered":"2024-09-16T15:47:00.598432+0000","last_clean":"2024-09-16T15:47:00.598432+0000","last_became_active":"2024-09-16T15:46:59.598447+0000","last_became_peered":"2024-09-16T15:46:59.598447+0000","last_unstale":"2024-09-16T15:47:00.598432+0000","last_undegraded":"2024-09-16T15:47:00.598432+0000","last_fullsized":"2024-09-16T15:47:00.598432+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_clean_scrub_stamp":"2024-09-16T15:46:58.541934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:56:59.105062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.2c","version":"0'0","reported_seq":13,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-09-16T15:47:01.601958+0000","last_change":"2024-09-16T15:46:59.593941+0000","last_active":"2024-09-16T15:47:01.601958+0000","last_peered":"2024-09-16T15:47:01.601958+0000","last_clean":"2024-09-16T15:47:01.601958+0000","last_became_active":"2024-09-16T15:46:59.593630+0000","last_became_peered":"2024-09-16T15:46:59.593630+0000","last_unstale":"2024-09-16T15:47:01.601958+0000","last_undegraded":"2024-09-16T15:47:01.601958+0000","last_fullsized":"2024-09-16T15:47:01.601958+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_clean_scrub_stamp":"2024-09-16T15:46:58.541934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:36:15.427420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.2b","version":"0'0","reported_seq":13,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-09-16T15:47:00.598764+0000","last_change":"2024-09-16T15:46:59.589652+0000","last_active":"2024-09-16T15:47:00.598764+0000","last_peered":"2024-09-16T15:47:00.598764+0000","last_clean":"2024-09-16T15:47:00.598764+0000","last_became_active":"2024-09-16T15:46:59.588642+0000","last_became_peered":"2024-09-16T15:46:59.588642+0000","last_unstale":"2024-09-16T15:47:00.598764+0000","last_undegraded":"2024-09-16T15:47:00.598764+0000","last_fullsized":"2024-09-16T15:47:00.598764+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_clean_scrub_stamp":"2024-09-16T15:46:58.541934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T22:24:26.273749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.2a","version":"0'0","reported_seq":13,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-09-16T15:47:00.597543+0000","last_change":"2024-09-16T15:46:59.589013+0000","last_active":"2024-09-16T15:47:00.597543+0000","last_peered":"2024-09-16T15:47:00.597543+0000","last_clean":"2024-09-16T15:47:00.597543+0000","last_became_active":"2024-09-16T15:46:59.588602+0000","last_became_peered":"2024-09-16T15:46:59.588602+0000","last_unstale":"2024-09-16T15:47:00.597543+0000","last_undegraded":"2024-09-16T15:47:00.597543+0000","last_fullsized":"2024-09-16T15:47:00.597543+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_clean_scrub_stamp":"2024-09-16T15:46:58.541934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T00:33:06.498806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.29","version":"0'0","reported_seq":13,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-09-16T15:47:01.270746+0000","last_change":"2024-09-16T15:46:59.594695+0000","last_active":"2024-09-16T15:47:01.270746+0000","last_peered":"2024-09-16T15:47:01.270746+0000","last_clean":"2024-09-16T15:47:01.270746+0000","last_became_active":"2024-09-16T15:46:59.594365+0000","last_became_peered":"2024-09-16T15:46:59.594365+0000","last_unstale":"2024-09-16T15:47:01.270746+0000","last_undegraded":"2024-09-16T15:47:01.270746+0000","last_fullsized":"2024-09-16T15:47:01.270746+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_clean_scrub_stamp":"2024-09-16T15:46:58.541934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:01:16.802069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.28","version":"0'0","reported_seq":13,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-09-16T15:47:01.270471+0000","last_change":"2024-09-16T15:46:59.603345+0000","last_active":"2024-09-16T15:47:01.270471+0000","last_peered":"2024-09-16T15:47:01.270471+0000","last_clean":"2024-09-16T15:47:01.270471+0000","last_became_active":"2024-09-16T15:46:59.602810+0000","last_became_peered":"2024-09-16T15:46:59.602810+0000","last_unstale":"2024-09-16T15:47:01.270471+0000","last_undegraded":"2024-09-16T15:47:01.270471+0000","last_fullsized":"2024-09-16T15:47:01.270471+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_clean_scrub_stamp":"2024-09-16T15:46:58.541934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:59:29.369856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.27","version":"0'0","reported_seq":13,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-09-16T15:47:00.600102+0000","last_change":"2024-09-16T15:46:59.576462+0000","last_active":"2024-09-16T15:47:00.600102+0000","last_peered":"2024-09-16T15:47:00.600102+0000","last_clean":"2024-09-16T15:47:00.600102+0000","last_became_active":"2024-09-16T15:46:59.575729+0000","last_became_peered":"2024-09-16T15:46:59.575729+0000","last_unstale":"2024-09-16T15:47:00.600102+0000","last_undegraded":"2024-09-16T15:47:00.600102+0000","last_fullsized":"2024-09-16T15:47:00.600102+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_clean_scrub_stamp":"2024-09-16T15:46:58.541934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:33:31.572379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.26","version":"0'0","reported_seq":13,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-09-16T15:47:01.596283+0000","last_change":"2024-09-16T15:46:59.611496+0000","last_active":"2024-09-16T15:47:01.596283+0000","last_peered":"2024-09-16T15:47:01.596283+0000","last_clean":"2024-09-16T15:47:01.596283+0000","last_became_active":"2024-09-16T15:46:59.611061+0000","last_became_peered":"2024-09-16T15:46:59.611061+0000","last_unstale":"2024-09-16T15:47:01.596283+0000","last_undegraded":"2024-09-16T15:47:01.596283+0000","last_fullsized":"2024-09-16T15:47:01.596283+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_clean_scrub_stamp":"2024-09-16T15:46:58.541934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:22:23.433493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.25","version":"0'0","reported_seq":13,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-09-16T15:47:02.581586+0000","last_change":"2024-09-16T15:46:59.587897+0000","last_active":"2024-09-16T15:47:02.581586+0000","last_peered":"2024-09-16T15:47:02.581586+0000","last_clean":"2024-09-16T15:47:02.581586+0000","last_became_active":"2024-09-16T15:46:59.587572+0000","last_became_peered":"2024-09-16T15:46:59.587572+0000","last_unstale":"2024-09-16T15:47:02.581586+0000","last_undegraded":"2024-09-16T15:47:02.581586+0000","last_fullsized":"2024-09-16T15:47:02.581586+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_clean_scrub_stamp":"2024-09-16T15:46:58.541934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:13:20.200761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.24","version":"0'0","reported_seq":13,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-09-16T15:47:00.599686+0000","last_change":"2024-09-16T15:46:59.583799+0000","last_active":"2024-09-16T15:47:00.599686+0000","last_peered":"2024-09-16T15:47:00.599686+0000","last_clean":"2024-09-16T15:47:00.599686+0000","last_became_active":"2024-09-16T15:46:59.582990+0000","last_became_peered":"2024-09-16T15:46:59.582990+0000","last_unstale":"2024-09-16T15:47:00.599686+0000","last_undegraded":"2024-09-16T15:47:00.599686+0000","last_fullsized":"2024-09-16T15:47:00.599686+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_clean_scrub_stamp":"2024-09-16T15:46:58.541934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T22:06:54.359156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.23","version":"0'0","reported_seq":13,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-09-16T15:47:02.580817+0000","last_change":"2024-09-16T15:46:59.593975+0000","last_active":"2024-09-16T15:47:02.580817+0000","last_peered":"2024-09-16T15:47:02.580817+0000","last_clean":"2024-09-16T15:47:02.580817+0000","last_became_active":"2024-09-16T15:46:59.593684+0000","last_became_peered":"2024-09-16T15:46:59.593684+0000","last_unstale":"2024-09-16T15:47:02.580817+0000","last_undegraded":"2024-09-16T15:47:02.580817+0000","last_fullsized":"2024-09-16T15:47:02.580817+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_clean_scrub_stamp":"2024-09-16T15:46:58.541934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:56:30.584297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.22","version":"0'0","reported_seq":13,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-09-16T15:47:01.595775+0000","last_change":"2024-09-16T15:46:59.611445+0000","last_active":"2024-09-16T15:47:01.595775+0000","last_peered":"2024-09-16T15:47:01.595775+0000","last_clean":"2024-09-16T15:47:01.595775+0000","last_became_active":"2024-09-16T15:46:59.611139+0000","last_became_peered":"2024-09-16T15:46:59.611139+0000","last_unstale":"2024-09-16T15:47:01.595775+0000","last_undegraded":"2024-09-16T15:47:01.595775+0000","last_fullsized":"2024-09-16T15:47:01.595775+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_clean_scrub_stamp":"2024-09-16T15:46:58.541934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:45:41.180182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.21","version":"0'0","reported_seq":13,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-09-16T15:47:01.596481+0000","last_change":"2024-09-16T15:46:59.611304+0000","last_active":"2024-09-16T15:47:01.596481+0000","last_peered":"2024-09-16T15:47:01.596481+0000","last_clean":"2024-09-16T15:47:01.596481+0000","last_became_active":"2024-09-16T15:46:59.610992+0000","last_became_peered":"2024-09-16T15:46:59.610992+0000","last_unstale":"2024-09-16T15:47:01.596481+0000","last_undegraded":"2024-09-16T15:47:01.596481+0000","last_fullsized":"2024-09-16T15:47:01.596481+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_clean_scrub_stamp":"2024-09-16T15:46:58.541934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T18:46:53.267597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.20","version":"0'0","reported_seq":13,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-09-16T15:47:00.598227+0000","last_change":"2024-09-16T15:46:59.597400+0000","last_active":"2024-09-16T15:47:00.598227+0000","last_peered":"2024-09-16T15:47:00.598227+0000","last_clean":"2024-09-16T15:47:00.598227+0000","last_became_active":"2024-09-16T15:46:59.597080+0000","last_became_peered":"2024-09-16T15:46:59.597080+0000","last_unstale":"2024-09-16T15:47:00.598227+0000","last_undegraded":"2024-09-16T15:47:00.598227+0000","last_fullsized":"2024-09-16T15:47:00.598227+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_clean_scrub_stamp":"2024-09-16T15:46:58.541934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:04:21.844966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.1e","version":"0'0","reported_seq":13,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-09-16T15:47:01.271607+0000","last_change":"2024-09-16T15:46:59.604630+0000","last_active":"2024-09-16T15:47:01.271607+0000","last_peered":"2024-09-16T15:47:01.271607+0000","last_clean":"2024-09-16T15:47:01.271607+0000","last_became_active":"2024-09-16T15:46:59.604254+0000","last_became_peered":"2024-09-16T15:46:59.604254+0000","last_unstale":"2024-09-16T15:47:01.271607+0000","last_undegraded":"2024-09-16T15:47:01.271607+0000","last_fullsized":"2024-09-16T15:47:01.271607+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_clean_scrub_stamp":"2024-09-16T15:46:58.541934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:01:54.020966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.1f","version":"69'4","reported_seq":19,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-09-16T15:47:01.271935+0000","last_change":"2024-09-16T15:46:58.596071+0000","last_active":"2024-09-16T15:47:01.271935+0000","last_peered":"2024-09-16T15:47:01.271935+0000","last_clean":"2024-09-16T15:47:01.271935+0000","last_became_active":"2024-09-16T15:46:58.595309+0000","last_became_peered":"2024-09-16T15:46:58.595309+0000","last_unstale":"2024-09-16T15:47:01.271935+0000","last_undegraded":"2024-09-16T15:47:01.271935+0000","last_fullsized":"2024-09-16T15:47:01.271935+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:57.531046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:57.531046+0000","last_clean_scrub_stamp":"2024-09-16T15:46:57.531046+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:31:07.776398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.1f","version":"0'0","reported_seq":13,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-09-16T15:47:00.596775+0000","last_change":"2024-09-16T15:46:59.592887+0000","last_active":"2024-09-16T15:47:00.596775+0000","last_peered":"2024-09-16T15:47:00.596775+0000","last_clean":"2024-09-16T15:47:00.596775+0000","last_became_active":"2024-09-16T15:46:59.592495+0000","last_became_peered":"2024-09-16T15:46:59.592495+0000","last_unstale":"2024-09-16T15:47:00.596775+0000","last_undegraded":"2024-09-16T15:47:00.596775+0000","last_fullsized":"2024-09-16T15:47:00.596775+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_clean_scrub_stamp":"2024-09-16T15:46:58.541934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:37:07.444111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.1e","version":"69'1","reported_seq":16,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-09-16T15:47:01.602168+0000","last_change":"2024-09-16T15:46:58.572883+0000","last_active":"2024-09-16T15:47:01.602168+0000","last_peered":"2024-09-16T15:47:01.602168+0000","last_clean":"2024-09-16T15:47:01.602168+0000","last_became_active":"2024-09-16T15:46:58.572508+0000","last_became_peered":"2024-09-16T15:46:58.572508+0000","last_unstale":"2024-09-16T15:47:01.602168+0000","last_undegraded":"2024-09-16T15:47:01.602168+0000","last_fullsized":"2024-09-16T15:47:01.602168+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:57.531046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:57.531046+0000","last_clean_scrub_stamp":"2024-09-16T15:46:57.531046+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:37:47.205740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1c","version":"0'0","reported_seq":13,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-09-16T15:47:00.598551+0000","last_change":"2024-09-16T15:46:59.591976+0000","last_active":"2024-09-16T15:47:00.598551+0000","last_peered":"2024-09-16T15:47:00.598551+0000","last_clean":"2024-09-16T15:47:00.598551+0000","last_became_active":"2024-09-16T15:46:59.591124+0000","last_became_peered":"2024-09-16T15:46:59.591124+0000","last_unstale":"2024-09-16T15:47:00.598551+0000","last_undegraded":"2024-09-16T15:47:00.598551+0000","last_fullsized":"2024-09-16T15:47:00.598551+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_clean_scrub_stamp":"2024-09-16T15:46:58.541934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T23:32:43.604755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.1d","version":"69'2","reported_seq":17,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-09-16T15:47:00.599758+0000","last_change":"2024-09-16T15:46:58.598772+0000","last_active":"2024-09-16T15:47:00.599758+0000","last_peered":"2024-09-16T15:47:00.599758+0000","last_clean":"2024-09-16T15:47:00.599758+0000","last_became_active":"2024-09-16T15:46:58.597889+0000","last_became_peered":"2024-09-16T15:46:58.597889+0000","last_unstale":"2024-09-16T15:47:00.599758+0000","last_undegraded":"2024-09-16T15:47:00.599758+0000","last_fullsized":"2024-09-16T15:47:00.599758+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:57.531046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:57.531046+0000","last_clean_scrub_stamp":"2024-09-16T15:46:57.531046+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:56:59.397615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1d","version":"0'0","reported_seq":13,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-09-16T15:47:01.271379+0000","last_change":"2024-09-16T15:46:59.585287+0000","last_active":"2024-09-16T15:47:01.271379+0000","last_peered":"2024-09-16T15:47:01.271379+0000","last_clean":"2024-09-16T15:47:01.271379+0000","last_became_active":"2024-09-16T15:46:59.584504+0000","last_became_peered":"2024-09-16T15:46:59.584504+0000","last_unstale":"2024-09-16T15:47:01.271379+0000","last_undegraded":"2024-09-16T15:47:01.271379+0000","last_fullsized":"2024-09-16T15:47:01.271379+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_clean_scrub_stamp":"2024-09-16T15:46:58.541934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T01:35:56.639955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.1c","version":"69'2","reported_seq":17,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-09-16T15:47:00.598756+0000","last_change":"2024-09-16T15:46:58.588644+0000","last_active":"2024-09-16T15:47:00.598756+0000","last_peered":"2024-09-16T15:47:00.598756+0000","last_clean":"2024-09-16T15:47:00.598756+0000","last_became_active":"2024-09-16T15:46:58.587899+0000","last_became_peered":"2024-09-16T15:46:58.587899+0000","last_unstale":"2024-09-16T15:47:00.598756+0000","last_undegraded":"2024-09-16T15:47:00.598756+0000","last_fullsized":"2024-09-16T15:47:00.598756+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:57.531046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:57.531046+0000","last_clean_scrub_stamp":"2024-09-16T15:46:57.531046+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T18:12:19.424941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.1a","version":"0'0","reported_seq":13,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-09-16T15:47:01.595926+0000","last_change":"2024-09-16T15:46:59.610697+0000","last_active":"2024-09-16T15:47:01.595926+0000","last_peered":"2024-09-16T15:47:01.595926+0000","last_clean":"2024-09-16T15:47:01.595926+0000","last_became_active":"2024-09-16T15:46:59.609102+0000","last_became_peered":"2024-09-16T15:46:59.609102+0000","last_unstale":"2024-09-16T15:47:01.595926+0000","last_undegraded":"2024-09-16T15:47:01.595926+0000","last_fullsized":"2024-09-16T15:47:01.595926+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_clean_scrub_stamp":"2024-09-16T15:46:58.541934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T03:21:26.013131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.1b","version":"69'1","reported_seq":16,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-09-16T15:47:02.581282+0000","last_change":"2024-09-16T15:46:58.591621+0000","last_active":"2024-09-16T15:47:02.581282+0000","last_peered":"2024-09-16T15:47:02.581282+0000","last_clean":"2024-09-16T15:47:02.581282+0000","last_became_active":"2024-09-16T15:46:58.591311+0000","last_became_peered":"2024-09-16T15:46:58.591311+0000","last_unstale":"2024-09-16T15:47:02.581282+0000","last_undegraded":"2024-09-16T15:47:02.581282+0000","last_fullsized":"2024-09-16T15:47:02.581282+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:57.531046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:57.531046+0000","last_clean_scrub_stamp":"2024-09-16T15:46:57.531046+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:25:15.762791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.1b","version":"0'0","reported_seq":13,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-09-16T15:47:00.599531+0000","last_change":"2024-09-16T15:46:59.581989+0000","last_active":"2024-09-16T15:47:00.599531+0000","last_peered":"2024-09-16T15:47:00.599531+0000","last_clean":"2024-09-16T15:47:00.599531+0000","last_became_active":"2024-09-16T15:46:59.581259+0000","last_became_peered":"2024-09-16T15:46:59.581259+0000","last_unstale":"2024-09-16T15:47:00.599531+0000","last_undegraded":"2024-09-16T15:47:00.599531+0000","last_fullsized":"2024-09-16T15:47:00.599531+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_clean_scrub_stamp":"2024-09-16T15:46:58.541934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:16:49.824032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.1a","version":"69'1","reported_seq":16,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-09-16T15:47:00.598954+0000","last_change":"2024-09-16T15:46:58.600655+0000","last_active":"2024-09-16T15:47:00.598954+0000","last_peered":"2024-09-16T15:47:00.598954+0000","last_clean":"2024-09-16T15:47:00.598954+0000","last_became_active":"2024-09-16T15:46:58.599216+0000","last_became_peered":"2024-09-16T15:46:58.599216+0000","last_unstale":"2024-09-16T15:47:00.598954+0000","last_undegraded":"2024-09-16T15:47:00.598954+0000","last_fullsized":"2024-09-16T15:47:00.598954+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:57.531046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:57.531046+0000","last_clean_scrub_stamp":"2024-09-16T15:46:57.531046+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:11:50.475616+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.18","version":"0'0","reported_seq":13,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-09-16T15:47:00.595638+0000","last_change":"2024-09-16T15:46:59.572288+0000","last_active":"2024-09-16T15:47:00.595638+0000","last_peered":"2024-09-16T15:47:00.595638+0000","last_clean":"2024-09-16T15:47:00.595638+0000","last_became_active":"2024-09-16T15:46:59.571531+0000","last_became_peered":"2024-09-16T15:46:59.571531+0000","last_unstale":"2024-09-16T15:47:00.595638+0000","last_undegraded":"2024-09-16T15:47:00.595638+0000","last_fullsized":"2024-09-16T15:47:00.595638+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_clean_scrub_stamp":"2024-09-16T15:46:58.541934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T18:41:19.283349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.19","version":"0'0","reported_seq":15,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-09-16T15:47:03.017288+0000","last_change":"2024-09-16T15:46:58.584364+0000","last_active":"2024-09-16T15:47:03.017288+0000","last_peered":"2024-09-16T15:47:03.017288+0000","last_clean":"2024-09-16T15:47:03.017288+0000","last_became_active":"2024-09-16T15:46:58.583975+0000","last_became_peered":"2024-09-16T15:46:58.583975+0000","last_unstale":"2024-09-16T15:47:03.017288+0000","last_undegraded":"2024-09-16T15:47:03.017288+0000","last_fullsized":"2024-09-16T15:47:03.017288+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:57.531046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:57.531046+0000","last_clean_scrub_stamp":"2024-09-16T15:46:57.531046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T22:29:16.058306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.19","version":"0'0","reported_seq":13,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-09-16T15:47:01.601680+0000","last_change":"2024-09-16T15:46:59.598073+0000","last_active":"2024-09-16T15:47:01.601680+0000","last_peered":"2024-09-16T15:47:01.601680+0000","last_clean":"2024-09-16T15:47:01.601680+0000","last_became_active":"2024-09-16T15:46:59.595164+0000","last_became_peered":"2024-09-16T15:46:59.595164+0000","last_unstale":"2024-09-16T15:47:01.601680+0000","last_undegraded":"2024-09-16T15:47:01.601680+0000","last_fullsized":"2024-09-16T15:47:01.601680+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_clean_scrub_stamp":"2024-09-16T15:46:58.541934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T00:39:24.368070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.18","version":"0'0","reported_seq":15,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-09-16T15:47:00.595446+0000","last_change":"2024-09-16T15:46:58.570577+0000","last_active":"2024-09-16T15:47:00.595446+0000","last_peered":"2024-09-16T15:47:00.595446+0000","last_clean":"2024-09-16T15:47:00.595446+0000","last_became_active":"2024-09-16T15:46:58.570223+0000","last_became_peered":"2024-09-16T15:46:58.570223+0000","last_unstale":"2024-09-16T15:47:00.595446+0000","last_undegraded":"2024-09-16T15:47:00.595446+0000","last_fullsized":"2024-09-16T15:47:00.595446+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:57.531046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:57.531046+0000","last_clean_scrub_stamp":"2024-09-16T15:46:57.531046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:21:03.809931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.16","version":"0'0","reported_seq":13,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-09-16T15:47:01.596369+0000","last_change":"2024-09-16T15:46:59.607629+0000","last_active":"2024-09-16T15:47:01.596369+0000","last_peered":"2024-09-16T15:47:01.596369+0000","last_clean":"2024-09-16T15:47:01.596369+0000","last_became_active":"2024-09-16T15:46:59.594138+0000","last_became_peered":"2024-09-16T15:46:59.594138+0000","last_unstale":"2024-09-16T15:47:01.596369+0000","last_undegraded":"2024-09-16T15:47:01.596369+0000","last_fullsized":"2024-09-16T15:47:01.596369+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_clean_scrub_stamp":"2024-09-16T15:46:58.541934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T00:19:34.204760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.17","version":"0'0","reported_seq":15,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-09-16T15:47:00.599830+0000","last_change":"2024-09-16T15:46:58.576463+0000","last_active":"2024-09-16T15:47:00.599830+0000","last_peered":"2024-09-16T15:47:00.599830+0000","last_clean":"2024-09-16T15:47:00.599830+0000","last_became_active":"2024-09-16T15:46:58.573574+0000","last_became_peered":"2024-09-16T15:46:58.573574+0000","last_unstale":"2024-09-16T15:47:00.599830+0000","last_undegraded":"2024-09-16T15:47:00.599830+0000","last_fullsized":"2024-09-16T15:47:00.599830+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:57.531046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:57.531046+0000","last_clean_scrub_stamp":"2024-09-16T15:46:57.531046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T02:08:56.630171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.17","version":"0'0","reported_seq":13,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-09-16T15:47:01.271605+0000","last_change":"2024-09-16T15:46:59.602747+0000","last_active":"2024-09-16T15:47:01.271605+0000","last_peered":"2024-09-16T15:47:01.271605+0000","last_clean":"2024-09-16T15:47:01.271605+0000","last_became_active":"2024-09-16T15:46:59.602534+0000","last_became_peered":"2024-09-16T15:46:59.602534+0000","last_unstale":"2024-09-16T15:47:01.271605+0000","last_undegraded":"2024-09-16T15:47:01.271605+0000","last_fullsized":"2024-09-16T15:47:01.271605+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_clean_scrub_stamp":"2024-09-16T15:46:58.541934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T03:14:14.552144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.16","version":"0'0","reported_seq":15,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-09-16T15:47:00.600282+0000","last_change":"2024-09-16T15:46:58.597162+0000","last_active":"2024-09-16T15:47:00.600282+0000","last_peered":"2024-09-16T15:47:00.600282+0000","last_clean":"2024-09-16T15:47:00.600282+0000","last_became_active":"2024-09-16T15:46:58.596469+0000","last_became_peered":"2024-09-16T15:46:58.596469+0000","last_unstale":"2024-09-16T15:47:00.600282+0000","last_undegraded":"2024-09-16T15:47:00.600282+0000","last_fullsized":"2024-09-16T15:47:00.600282+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:57.531046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:57.531046+0000","last_clean_scrub_stamp":"2024-09-16T15:46:57.531046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:53:11.613889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.14","version":"0'0","reported_seq":13,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-09-16T15:47:00.598574+0000","last_change":"2024-09-16T15:46:59.591976+0000","last_active":"2024-09-16T15:47:00.598574+0000","last_peered":"2024-09-16T15:47:00.598574+0000","last_clean":"2024-09-16T15:47:00.598574+0000","last_became_active":"2024-09-16T15:46:59.591396+0000","last_became_peered":"2024-09-16T15:46:59.591396+0000","last_unstale":"2024-09-16T15:47:00.598574+0000","last_undegraded":"2024-09-16T15:47:00.598574+0000","last_fullsized":"2024-09-16T15:47:00.598574+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_clean_scrub_stamp":"2024-09-16T15:46:58.541934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:33.360136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.15","version":"69'3","reported_seq":18,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-09-16T15:47:02.581817+0000","last_change":"2024-09-16T15:46:58.583907+0000","last_active":"2024-09-16T15:47:02.581817+0000","last_peered":"2024-09-16T15:47:02.581817+0000","last_clean":"2024-09-16T15:47:02.581817+0000","last_became_active":"2024-09-16T15:46:58.583530+0000","last_became_peered":"2024-09-16T15:46:58.583530+0000","last_unstale":"2024-09-16T15:47:02.581817+0000","last_undegraded":"2024-09-16T15:47:02.581817+0000","last_fullsized":"2024-09-16T15:47:02.581817+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:57.531046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:57.531046+0000","last_clean_scrub_stamp":"2024-09-16T15:46:57.531046+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:48:54.056860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.15","version":"0'0","reported_seq":13,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-09-16T15:47:00.596203+0000","last_change":"2024-09-16T15:46:59.584744+0000","last_active":"2024-09-16T15:47:00.596203+0000","last_peered":"2024-09-16T15:47:00.596203+0000","last_clean":"2024-09-16T15:47:00.596203+0000","last_became_active":"2024-09-16T15:46:59.583999+0000","last_became_peered":"2024-09-16T15:46:59.583999+0000","last_unstale":"2024-09-16T15:47:00.596203+0000","last_undegraded":"2024-09-16T15:47:00.596203+0000","last_fullsized":"2024-09-16T15:47:00.596203+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_clean_scrub_stamp":"2024-09-16T15:46:58.541934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:31:15.719148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.14","version":"69'6","reported_seq":21,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-09-16T15:47:03.017921+0000","last_change":"2024-09-16T15:46:58.570712+0000","last_active":"2024-09-16T15:47:03.017921+0000","last_peered":"2024-09-16T15:47:03.017921+0000","last_clean":"2024-09-16T15:47:03.017921+0000","last_became_active":"2024-09-16T15:46:58.570432+0000","last_became_peered":"2024-09-16T15:46:58.570432+0000","last_unstale":"2024-09-16T15:47:03.017921+0000","last_undegraded":"2024-09-16T15:47:03.017921+0000","last_fullsized":"2024-09-16T15:47:03.017921+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:57.531046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:57.531046+0000","last_clean_scrub_stamp":"2024-09-16T15:46:57.531046+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T22:50:29.339028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.12","version":"0'0","reported_seq":13,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-09-16T15:47:01.270763+0000","last_change":"2024-09-16T15:46:59.592809+0000","last_active":"2024-09-16T15:47:01.270763+0000","last_peered":"2024-09-16T15:47:01.270763+0000","last_clean":"2024-09-16T15:47:01.270763+0000","last_became_active":"2024-09-16T15:46:59.592519+0000","last_became_peered":"2024-09-16T15:46:59.592519+0000","last_unstale":"2024-09-16T15:47:01.270763+0000","last_undegraded":"2024-09-16T15:47:01.270763+0000","last_fullsized":"2024-09-16T15:47:01.270763+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_clean_scrub_stamp":"2024-09-16T15:46:58.541934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:45:16.750796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.13","version":"69'2","reported_seq":17,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-09-16T15:47:03.017514+0000","last_change":"2024-09-16T15:46:58.596492+0000","last_active":"2024-09-16T15:47:03.017514+0000","last_peered":"2024-09-16T15:47:03.017514+0000","last_clean":"2024-09-16T15:47:03.017514+0000","last_became_active":"2024-09-16T15:46:58.596172+0000","last_became_peered":"2024-09-16T15:46:58.596172+0000","last_unstale":"2024-09-16T15:47:03.017514+0000","last_undegraded":"2024-09-16T15:47:03.017514+0000","last_fullsized":"2024-09-16T15:47:03.017514+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:57.531046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:57.531046+0000","last_clean_scrub_stamp":"2024-09-16T15:46:57.531046+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:08:03.416601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.13","version":"0'0","reported_seq":13,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-09-16T15:47:01.596900+0000","last_change":"2024-09-16T15:46:59.607756+0000","last_active":"2024-09-16T15:47:01.596900+0000","last_peered":"2024-09-16T15:47:01.596900+0000","last_clean":"2024-09-16T15:47:01.596900+0000","last_became_active":"2024-09-16T15:46:59.594115+0000","last_became_peered":"2024-09-16T15:46:59.594115+0000","last_unstale":"2024-09-16T15:47:01.596900+0000","last_undegraded":"2024-09-16T15:47:01.596900+0000","last_fullsized":"2024-09-16T15:47:01.596900+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_clean_scrub_stamp":"2024-09-16T15:46:58.541934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:55:23.302102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.12","version":"0'0","reported_seq":15,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-09-16T15:47:00.599033+0000","last_change":"2024-09-16T15:46:58.595177+0000","last_active":"2024-09-16T15:47:00.599033+0000","last_peered":"2024-09-16T15:47:00.599033+0000","last_clean":"2024-09-16T15:47:00.599033+0000","last_became_active":"2024-09-16T15:46:58.593818+0000","last_became_peered":"2024-09-16T15:46:58.593818+0000","last_unstale":"2024-09-16T15:47:00.599033+0000","last_undegraded":"2024-09-16T15:47:00.599033+0000","last_fullsized":"2024-09-16T15:47:00.599033+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:57.531046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:57.531046+0000","last_clean_scrub_stamp":"2024-09-16T15:46:57.531046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T02:00:05.454235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.10","version":"0'0","reported_seq":13,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-09-16T15:47:01.601483+0000","last_change":"2024-09-16T15:46:59.602885+0000","last_active":"2024-09-16T15:47:01.601483+0000","last_peered":"2024-09-16T15:47:01.601483+0000","last_clean":"2024-09-16T15:47:01.601483+0000","last_became_active":"2024-09-16T15:46:59.602187+0000","last_became_peered":"2024-09-16T15:46:59.602187+0000","last_unstale":"2024-09-16T15:47:01.601483+0000","last_undegraded":"2024-09-16T15:47:01.601483+0000","last_fullsized":"2024-09-16T15:47:01.601483+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_clean_scrub_stamp":"2024-09-16T15:46:58.541934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T00:45:25.077781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.11","version":"0'0","reported_seq":15,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-09-16T15:47:01.595515+0000","last_change":"2024-09-16T15:46:58.584495+0000","last_active":"2024-09-16T15:47:01.595515+0000","last_peered":"2024-09-16T15:47:01.595515+0000","last_clean":"2024-09-16T15:47:01.595515+0000","last_became_active":"2024-09-16T15:46:58.583741+0000","last_became_peered":"2024-09-16T15:46:58.583741+0000","last_unstale":"2024-09-16T15:47:01.595515+0000","last_undegraded":"2024-09-16T15:47:01.595515+0000","last_fullsized":"2024-09-16T15:47:01.595515+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:57.531046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:57.531046+0000","last_clean_scrub_stamp":"2024-09-16T15:46:57.531046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T03:21:06.658025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.11","version":"0'0","reported_seq":13,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-09-16T15:47:00.595623+0000","last_change":"2024-09-16T15:46:59.593192+0000","last_active":"2024-09-16T15:47:00.595623+0000","last_peered":"2024-09-16T15:47:00.595623+0000","last_clean":"2024-09-16T15:47:00.595623+0000","last_became_active":"2024-09-16T15:46:59.592659+0000","last_became_peered":"2024-09-16T15:46:59.592659+0000","last_unstale":"2024-09-16T15:47:00.595623+0000","last_undegraded":"2024-09-16T15:47:00.595623+0000","last_fullsized":"2024-09-16T15:47:00.595623+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_clean_scrub_stamp":"2024-09-16T15:46:58.541934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:58:28.354484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.10","version":"69'8","reported_seq":23,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-09-16T15:47:00.596785+0000","last_change":"2024-09-16T15:46:58.597568+0000","last_active":"2024-09-16T15:47:00.596785+0000","last_peered":"2024-09-16T15:47:00.596785+0000","last_clean":"2024-09-16T15:47:00.596785+0000","last_became_active":"2024-09-16T15:46:58.596806+0000","last_became_peered":"2024-09-16T15:46:58.596806+0000","last_unstale":"2024-09-16T15:47:00.596785+0000","last_undegraded":"2024-09-16T15:47:00.596785+0000","last_fullsized":"2024-09-16T15:47:00.596785+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:57.531046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:57.531046+0000","last_clean_scrub_stamp":"2024-09-16T15:46:57.531046+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T02:49:59.150430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.e","version":"0'0","reported_seq":13,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-09-16T15:47:00.599887+0000","last_change":"2024-09-16T15:46:59.581049+0000","last_active":"2024-09-16T15:47:00.599887+0000","last_peered":"2024-09-16T15:47:00.599887+0000","last_clean":"2024-09-16T15:47:00.599887+0000","last_became_active":"2024-09-16T15:46:59.580288+0000","last_became_peered":"2024-09-16T15:46:59.580288+0000","last_unstale":"2024-09-16T15:47:00.599887+0000","last_undegraded":"2024-09-16T15:47:00.599887+0000","last_fullsized":"2024-09-16T15:47:00.599887+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_clean_scrub_stamp":"2024-09-16T15:46:58.541934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:41:32.780496+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.f","version":"69'1","reported_seq":16,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-09-16T15:47:00.600199+0000","last_change":"2024-09-16T15:46:58.595767+0000","last_active":"2024-09-16T15:47:00.600199+0000","last_peered":"2024-09-16T15:47:00.600199+0000","last_clean":"2024-09-16T15:47:00.600199+0000","last_became_active":"2024-09-16T15:46:58.594655+0000","last_became_peered":"2024-09-16T15:46:58.594655+0000","last_unstale":"2024-09-16T15:47:00.600199+0000","last_undegraded":"2024-09-16T15:47:00.600199+0000","last_fullsized":"2024-09-16T15:47:00.600199+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:57.531046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:57.531046+0000","last_clean_scrub_stamp":"2024-09-16T15:46:57.531046+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:45:10.636370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.f","version":"0'0","reported_seq":13,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-09-16T15:47:01.272019+0000","last_change":"2024-09-16T15:46:59.592431+0000","last_active":"2024-09-16T15:47:01.272019+0000","last_peered":"2024-09-16T15:47:01.272019+0000","last_clean":"2024-09-16T15:47:01.272019+0000","last_became_active":"2024-09-16T15:46:59.592014+0000","last_became_peered":"2024-09-16T15:46:59.592014+0000","last_unstale":"2024-09-16T15:47:01.272019+0000","last_undegraded":"2024-09-16T15:47:01.272019+0000","last_fullsized":"2024-09-16T15:47:01.272019+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_clean_scrub_stamp":"2024-09-16T15:46:58.541934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:09:40.863816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.e","version":"69'2","reported_seq":17,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-09-16T15:47:00.600066+0000","last_change":"2024-09-16T15:46:58.597152+0000","last_active":"2024-09-16T15:47:00.600066+0000","last_peered":"2024-09-16T15:47:00.600066+0000","last_clean":"2024-09-16T15:47:00.600066+0000","last_became_active":"2024-09-16T15:46:58.596435+0000","last_became_peered":"2024-09-16T15:46:58.596435+0000","last_unstale":"2024-09-16T15:47:00.600066+0000","last_undegraded":"2024-09-16T15:47:00.600066+0000","last_fullsized":"2024-09-16T15:47:00.600066+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:57.531046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:57.531046+0000","last_clean_scrub_stamp":"2024-09-16T15:46:57.531046+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:34:40.101061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.c","version":"0'0","reported_seq":13,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-09-16T15:47:03.017902+0000","last_change":"2024-09-16T15:46:59.588876+0000","last_active":"2024-09-16T15:47:03.017902+0000","last_peered":"2024-09-16T15:47:03.017902+0000","last_clean":"2024-09-16T15:47:03.017902+0000","last_became_active":"2024-09-16T15:46:59.588308+0000","last_became_peered":"2024-09-16T15:46:59.588308+0000","last_unstale":"2024-09-16T15:47:03.017902+0000","last_undegraded":"2024-09-16T15:47:03.017902+0000","last_fullsized":"2024-09-16T15:47:03.017902+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_clean_scrub_stamp":"2024-09-16T15:46:58.541934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:45:43.693184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.d","version":"69'1","reported_seq":16,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-09-16T15:47:00.598557+0000","last_change":"2024-09-16T15:46:58.598349+0000","last_active":"2024-09-16T15:47:00.598557+0000","last_peered":"2024-09-16T15:47:00.598557+0000","last_clean":"2024-09-16T15:47:00.598557+0000","last_became_active":"2024-09-16T15:46:58.595349+0000","last_became_peered":"2024-09-16T15:46:58.595349+0000","last_unstale":"2024-09-16T15:47:00.598557+0000","last_undegraded":"2024-09-16T15:47:00.598557+0000","last_fullsized":"2024-09-16T15:47:00.598557+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:57.531046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:57.531046+0000","last_clean_scrub_stamp":"2024-09-16T15:46:57.531046+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:22:41.298787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.d","version":"0'0","reported_seq":13,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-09-16T15:47:00.596349+0000","last_change":"2024-09-16T15:46:59.575137+0000","last_active":"2024-09-16T15:47:00.596349+0000","last_peered":"2024-09-16T15:47:00.596349+0000","last_clean":"2024-09-16T15:47:00.596349+0000","last_became_active":"2024-09-16T15:46:59.574405+0000","last_became_peered":"2024-09-16T15:46:59.574405+0000","last_unstale":"2024-09-16T15:47:00.596349+0000","last_undegraded":"2024-09-16T15:47:00.596349+0000","last_fullsized":"2024-09-16T15:47:00.596349+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_clean_scrub_stamp":"2024-09-16T15:46:58.541934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T03:27:32.282539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.c","version":"0'0","reported_seq":15,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-09-16T15:47:01.271410+0000","last_change":"2024-09-16T15:46:58.596863+0000","last_active":"2024-09-16T15:47:01.271410+0000","last_peered":"2024-09-16T15:47:01.271410+0000","last_clean":"2024-09-16T15:47:01.271410+0000","last_became_active":"2024-09-16T15:46:58.596143+0000","last_became_peered":"2024-09-16T15:46:58.596143+0000","last_unstale":"2024-09-16T15:47:01.271410+0000","last_undegraded":"2024-09-16T15:47:01.271410+0000","last_fullsized":"2024-09-16T15:47:01.271410+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:57.531046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:57.531046+0000","last_clean_scrub_stamp":"2024-09-16T15:46:57.531046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T03:32:39.576676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.b","version":"0'0","reported_seq":13,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-09-16T15:47:02.580895+0000","last_change":"2024-09-16T15:46:59.586821+0000","last_active":"2024-09-16T15:47:02.580895+0000","last_peered":"2024-09-16T15:47:02.580895+0000","last_clean":"2024-09-16T15:47:02.580895+0000","last_became_active":"2024-09-16T15:46:59.586449+0000","last_became_peered":"2024-09-16T15:46:59.586449+0000","last_unstale":"2024-09-16T15:47:02.580895+0000","last_undegraded":"2024-09-16T15:47:02.580895+0000","last_fullsized":"2024-09-16T15:47:02.580895+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_clean_scrub_stamp":"2024-09-16T15:46:58.541934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:24:25.759774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.a","version":"0'0","reported_seq":15,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-09-16T15:47:00.599087+0000","last_change":"2024-09-16T15:46:58.585877+0000","last_active":"2024-09-16T15:47:00.599087+0000","last_peered":"2024-09-16T15:47:00.599087+0000","last_clean":"2024-09-16T15:47:00.599087+0000","last_became_active":"2024-09-16T15:46:58.585158+0000","last_became_peered":"2024-09-16T15:46:58.585158+0000","last_unstale":"2024-09-16T15:47:00.599087+0000","last_undegraded":"2024-09-16T15:47:00.599087+0000","last_fullsized":"2024-09-16T15:47:00.599087+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:57.531046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:57.531046+0000","last_clean_scrub_stamp":"2024-09-16T15:46:57.531046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T01:24:15.136386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.8","version":"0'0","reported_seq":13,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-09-16T15:47:02.581008+0000","last_change":"2024-09-16T15:46:59.607829+0000","last_active":"2024-09-16T15:47:02.581008+0000","last_peered":"2024-09-16T15:47:02.581008+0000","last_clean":"2024-09-16T15:47:02.581008+0000","last_became_active":"2024-09-16T15:46:59.607463+0000","last_became_peered":"2024-09-16T15:46:59.607463+0000","last_unstale":"2024-09-16T15:47:02.581008+0000","last_undegraded":"2024-09-16T15:47:02.581008+0000","last_fullsized":"2024-09-16T15:47:02.581008+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_clean_scrub_stamp":"2024-09-16T15:46:58.541934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T02:12:01.836641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.9","version":"0'0","reported_seq":15,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-09-16T15:47:01.596319+0000","last_change":"2024-09-16T15:46:58.576781+0000","last_active":"2024-09-16T15:47:01.596319+0000","last_peered":"2024-09-16T15:47:01.596319+0000","last_clean":"2024-09-16T15:47:01.596319+0000","last_became_active":"2024-09-16T15:46:58.576081+0000","last_became_peered":"2024-09-16T15:46:58.576081+0000","last_unstale":"2024-09-16T15:47:01.596319+0000","last_undegraded":"2024-09-16T15:47:01.596319+0000","last_fullsized":"2024-09-16T15:47:01.596319+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:57.531046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:57.531046+0000","last_clean_scrub_stamp":"2024-09-16T15:46:57.531046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:48:10.471983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.9","version":"0'0","reported_seq":13,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-09-16T15:47:02.580893+0000","last_change":"2024-09-16T15:46:59.605327+0000","last_active":"2024-09-16T15:47:02.580893+0000","last_peered":"2024-09-16T15:47:02.580893+0000","last_clean":"2024-09-16T15:47:02.580893+0000","last_became_active":"2024-09-16T15:46:59.605035+0000","last_became_peered":"2024-09-16T15:46:59.605035+0000","last_unstale":"2024-09-16T15:47:02.580893+0000","last_undegraded":"2024-09-16T15:47:02.580893+0000","last_fullsized":"2024-09-16T15:47:02.580893+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_clean_scrub_stamp":"2024-09-16T15:46:58.541934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:17:01.889391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.8","version":"0'0","reported_seq":15,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-09-16T15:47:01.601269+0000","last_change":"2024-09-16T15:46:58.572736+0000","last_active":"2024-09-16T15:47:01.601269+0000","last_peered":"2024-09-16T15:47:01.601269+0000","last_clean":"2024-09-16T15:47:01.601269+0000","last_became_active":"2024-09-16T15:46:58.572311+0000","last_became_peered":"2024-09-16T15:46:58.572311+0000","last_unstale":"2024-09-16T15:47:01.601269+0000","last_undegraded":"2024-09-16T15:47:01.601269+0000","last_fullsized":"2024-09-16T15:47:01.601269+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:57.531046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:57.531046+0000","last_clean_scrub_stamp":"2024-09-16T15:46:57.531046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T02:22:57.049894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.6","version":"0'0","reported_seq":13,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-09-16T15:47:01.602189+0000","last_change":"2024-09-16T15:46:59.593056+0000","last_active":"2024-09-16T15:47:01.602189+0000","last_peered":"2024-09-16T15:47:01.602189+0000","last_clean":"2024-09-16T15:47:01.602189+0000","last_became_active":"2024-09-16T15:46:59.592759+0000","last_became_peered":"2024-09-16T15:46:59.592759+0000","last_unstale":"2024-09-16T15:47:01.602189+0000","last_undegraded":"2024-09-16T15:47:01.602189+0000","last_fullsized":"2024-09-16T15:47:01.602189+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_clean_scrub_stamp":"2024-09-16T15:46:58.541934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T01:11:54.908266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.7","version":"69'2","reported_seq":17,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-09-16T15:47:00.600337+0000","last_change":"2024-09-16T15:46:58.579355+0000","last_active":"2024-09-16T15:47:00.600337+0000","last_peered":"2024-09-16T15:47:00.600337+0000","last_clean":"2024-09-16T15:47:00.600337+0000","last_became_active":"2024-09-16T15:46:58.578380+0000","last_became_peered":"2024-09-16T15:46:58.578380+0000","last_unstale":"2024-09-16T15:47:00.600337+0000","last_undegraded":"2024-09-16T15:47:00.600337+0000","last_fullsized":"2024-09-16T15:47:00.600337+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:57.531046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:57.531046+0000","last_clean_scrub_stamp":"2024-09-16T15:46:57.531046+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T00:08:23.102153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.7","version":"0'0","reported_seq":13,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-09-16T15:47:01.595958+0000","last_change":"2024-09-16T15:46:59.608798+0000","last_active":"2024-09-16T15:47:01.595958+0000","last_peered":"2024-09-16T15:47:01.595958+0000","last_clean":"2024-09-16T15:47:01.595958+0000","last_became_active":"2024-09-16T15:46:59.608405+0000","last_became_peered":"2024-09-16T15:46:59.608405+0000","last_unstale":"2024-09-16T15:47:01.595958+0000","last_undegraded":"2024-09-16T15:47:01.595958+0000","last_fullsized":"2024-09-16T15:47:01.595958+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_clean_scrub_stamp":"2024-09-16T15:46:58.541934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T18:32:04.675615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.6","version":"70'1","reported_seq":16,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-09-16T15:47:01.596697+0000","last_change":"2024-09-16T15:46:58.577011+0000","last_active":"2024-09-16T15:47:01.596697+0000","last_peered":"2024-09-16T15:47:01.596697+0000","last_clean":"2024-09-16T15:47:01.596697+0000","last_became_active":"2024-09-16T15:46:58.576294+0000","last_became_peered":"2024-09-16T15:46:58.576294+0000","last_unstale":"2024-09-16T15:47:01.596697+0000","last_undegraded":"2024-09-16T15:47:01.596697+0000","last_fullsized":"2024-09-16T15:47:01.596697+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:57.531046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:57.531046+0000","last_clean_scrub_stamp":"2024-09-16T15:46:57.531046+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:16:24.337739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.4","version":"0'0","reported_seq":13,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-09-16T15:47:01.271071+0000","last_change":"2024-09-16T15:46:59.605989+0000","last_active":"2024-09-16T15:47:01.271071+0000","last_peered":"2024-09-16T15:47:01.271071+0000","last_clean":"2024-09-16T15:47:01.271071+0000","last_became_active":"2024-09-16T15:46:59.605656+0000","last_became_peered":"2024-09-16T15:46:59.605656+0000","last_unstale":"2024-09-16T15:47:01.271071+0000","last_undegraded":"2024-09-16T15:47:01.271071+0000","last_fullsized":"2024-09-16T15:47:01.271071+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_clean_scrub_stamp":"2024-09-16T15:46:58.541934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:27:02.493721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.5","version":"0'0","reported_seq":15,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-09-16T15:47:00.596447+0000","last_change":"2024-09-16T15:46:58.575511+0000","last_active":"2024-09-16T15:47:00.596447+0000","last_peered":"2024-09-16T15:47:00.596447+0000","last_clean":"2024-09-16T15:47:00.596447+0000","last_became_active":"2024-09-16T15:46:58.575218+0000","last_became_peered":"2024-09-16T15:46:58.575218+0000","last_unstale":"2024-09-16T15:47:00.596447+0000","last_undegraded":"2024-09-16T15:47:00.596447+0000","last_fullsized":"2024-09-16T15:47:00.596447+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:57.531046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:57.531046+0000","last_clean_scrub_stamp":"2024-09-16T15:46:57.531046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T00:01:30.679666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.5","version":"0'0","reported_seq":13,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-09-16T15:47:01.271629+0000","last_change":"2024-09-16T15:46:59.593627+0000","last_active":"2024-09-16T15:47:01.271629+0000","last_peered":"2024-09-16T15:47:01.271629+0000","last_clean":"2024-09-16T15:47:01.271629+0000","last_became_active":"2024-09-16T15:46:59.593364+0000","last_became_peered":"2024-09-16T15:46:59.593364+0000","last_unstale":"2024-09-16T15:47:01.271629+0000","last_undegraded":"2024-09-16T15:47:01.271629+0000","last_fullsized":"2024-09-16T15:47:01.271629+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_clean_scrub_stamp":"2024-09-16T15:46:58.541934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T18:16:47.887286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.4","version":"0'0","reported_seq":15,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-09-16T15:47:01.601790+0000","last_change":"2024-09-16T15:46:58.583703+0000","last_active":"2024-09-16T15:47:01.601790+0000","last_peered":"2024-09-16T15:47:01.601790+0000","last_clean":"2024-09-16T15:47:01.601790+0000","last_became_active":"2024-09-16T15:46:58.583384+0000","last_became_peered":"2024-09-16T15:46:58.583384+0000","last_unstale":"2024-09-16T15:47:01.601790+0000","last_undegraded":"2024-09-16T15:47:01.601790+0000","last_fullsized":"2024-09-16T15:47:01.601790+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:57.531046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:57.531046+0000","last_clean_scrub_stamp":"2024-09-16T15:46:57.531046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T01:02:52.799185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.2","version":"0'0","reported_seq":13,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-09-16T15:47:00.595830+0000","last_change":"2024-09-16T15:46:59.583989+0000","last_active":"2024-09-16T15:47:00.595830+0000","last_peered":"2024-09-16T15:47:00.595830+0000","last_clean":"2024-09-16T15:47:00.595830+0000","last_became_active":"2024-09-16T15:46:59.583226+0000","last_became_peered":"2024-09-16T15:46:59.583226+0000","last_unstale":"2024-09-16T15:47:00.595830+0000","last_undegraded":"2024-09-16T15:47:00.595830+0000","last_fullsized":"2024-09-16T15:47:00.595830+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_clean_scrub_stamp":"2024-09-16T15:46:58.541934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T00:25:30.260202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.3","version":"0'0","reported_seq":15,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-09-16T15:47:00.595856+0000","last_change":"2024-09-16T15:46:58.573698+0000","last_active":"2024-09-16T15:47:00.595856+0000","last_peered":"2024-09-16T15:47:00.595856+0000","last_clean":"2024-09-16T15:47:00.595856+0000","last_became_active":"2024-09-16T15:46:58.573391+0000","last_became_peered":"2024-09-16T15:46:58.573391+0000","last_unstale":"2024-09-16T15:47:00.595856+0000","last_undegraded":"2024-09-16T15:47:00.595856+0000","last_fullsized":"2024-09-16T15:47:00.595856+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:57.531046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:57.531046+0000","last_clean_scrub_stamp":"2024-09-16T15:46:57.531046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:33:10.960302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.3","version":"0'0","reported_seq":13,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-09-16T15:47:00.598175+0000","last_change":"2024-09-16T15:46:59.599296+0000","last_active":"2024-09-16T15:47:00.598175+0000","last_peered":"2024-09-16T15:47:00.598175+0000","last_clean":"2024-09-16T15:47:00.598175+0000","last_became_active":"2024-09-16T15:46:59.597133+0000","last_became_peered":"2024-09-16T15:46:59.597133+0000","last_unstale":"2024-09-16T15:47:00.598175+0000","last_undegraded":"2024-09-16T15:47:00.598175+0000","last_fullsized":"2024-09-16T15:47:00.598175+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_clean_scrub_stamp":"2024-09-16T15:46:58.541934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T03:05:53.392138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.2","version":"0'0","reported_seq":15,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-09-16T15:47:03.017290+0000","last_change":"2024-09-16T15:46:58.574227+0000","last_active":"2024-09-16T15:47:03.017290+0000","last_peered":"2024-09-16T15:47:03.017290+0000","last_clean":"2024-09-16T15:47:03.017290+0000","last_became_active":"2024-09-16T15:46:58.573898+0000","last_became_peered":"2024-09-16T15:46:58.573898+0000","last_unstale":"2024-09-16T15:47:03.017290+0000","last_undegraded":"2024-09-16T15:47:03.017290+0000","last_fullsized":"2024-09-16T15:47:03.017290+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:57.531046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:57.531046+0000","last_clean_scrub_stamp":"2024-09-16T15:46:57.531046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:31:14.708259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.0","version":"0'0","reported_seq":13,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-09-16T15:47:03.017385+0000","last_change":"2024-09-16T15:46:59.591607+0000","last_active":"2024-09-16T15:47:03.017385+0000","last_peered":"2024-09-16T15:47:03.017385+0000","last_clean":"2024-09-16T15:47:03.017385+0000","last_became_active":"2024-09-16T15:46:59.591067+0000","last_became_peered":"2024-09-16T15:46:59.591067+0000","last_unstale":"2024-09-16T15:47:03.017385+0000","last_undegraded":"2024-09-16T15:47:03.017385+0000","last_fullsized":"2024-09-16T15:47:03.017385+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_clean_scrub_stamp":"2024-09-16T15:46:58.541934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T00:29:30.637635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.1","version":"0'0","reported_seq":15,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-09-16T15:47:00.596880+0000","last_change":"2024-09-16T15:46:58.598614+0000","last_active":"2024-09-16T15:47:00.596880+0000","last_peered":"2024-09-16T15:47:00.596880+0000","last_clean":"2024-09-16T15:47:00.596880+0000","last_became_active":"2024-09-16T15:46:58.597950+0000","last_became_peered":"2024-09-16T15:46:58.597950+0000","last_unstale":"2024-09-16T15:47:00.596880+0000","last_undegraded":"2024-09-16T15:47:00.596880+0000","last_fullsized":"2024-09-16T15:47:00.596880+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:57.531046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:57.531046+0000","last_clean_scrub_stamp":"2024-09-16T15:46:57.531046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T03:07:16.501292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1","version":"0'0","reported_seq":13,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-09-16T15:47:00.598169+0000","last_change":"2024-09-16T15:46:59.598419+0000","last_active":"2024-09-16T15:47:00.598169+0000","last_peered":"2024-09-16T15:47:00.598169+0000","last_clean":"2024-09-16T15:47:00.598169+0000","last_became_active":"2024-09-16T15:46:59.597174+0000","last_became_peered":"2024-09-16T15:46:59.597174+0000","last_unstale":"2024-09-16T15:47:00.598169+0000","last_undegraded":"2024-09-16T15:47:00.598169+0000","last_fullsized":"2024-09-16T15:47:00.598169+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_clean_scrub_stamp":"2024-09-16T15:46:58.541934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:26:33.922668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.0","version":"0'0","reported_seq":15,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-09-16T15:47:00.598423+0000","last_change":"2024-09-16T15:46:58.599504+0000","last_active":"2024-09-16T15:47:00.598423+0000","last_peered":"2024-09-16T15:47:00.598423+0000","last_clean":"2024-09-16T15:47:00.598423+0000","last_became_active":"2024-09-16T15:46:58.598743+0000","last_became_peered":"2024-09-16T15:46:58.598743+0000","last_unstale":"2024-09-16T15:47:00.598423+0000","last_undegraded":"2024-09-16T15:47:00.598423+0000","last_fullsized":"2024-09-16T15:47:00.598423+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:57.531046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:57.531046+0000","last_clean_scrub_stamp":"2024-09-16T15:46:57.531046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:33:54.157169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.a","version":"0'0","reported_seq":13,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-09-16T15:47:02.581033+0000","last_change":"2024-09-16T15:46:59.604881+0000","last_active":"2024-09-16T15:47:02.581033+0000","last_peered":"2024-09-16T15:47:02.581033+0000","last_clean":"2024-09-16T15:47:02.581033+0000","last_became_active":"2024-09-16T15:46:59.604614+0000","last_became_peered":"2024-09-16T15:46:59.604614+0000","last_unstale":"2024-09-16T15:47:02.581033+0000","last_undegraded":"2024-09-16T15:47:02.581033+0000","last_fullsized":"2024-09-16T15:47:02.581033+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:58.541934+0000","last_clean_scrub_stamp":"2024-09-16T15:46:58.541934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T23:49:51.207671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.b","version":"69'2","reported_seq":17,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-09-16T15:47:00.598661+0000","last_change":"2024-09-16T15:46:58.601513+0000","last_active":"2024-09-16T15:47:00.598661+0000","last_peered":"2024-09-16T15:47:00.598661+0000","last_clean":"2024-09-16T15:47:00.598661+0000","last_became_active":"2024-09-16T15:46:58.600581+0000","last_became_peered":"2024-09-16T15:46:58.600581+0000","last_unstale":"2024-09-16T15:47:00.598661+0000","last_undegraded":"2024-09-16T15:47:00.598661+0000","last_fullsized":"2024-09-16T15:47:00.598661+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:46:57.531046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:46:57.531046+0000","last_clean_scrub_stamp":"2024-09-16T15:46:57.531046+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T01:47:55.238762+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"16'72","reported_seq":204,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-09-16T15:47:01.270624+0000","last_change":"2024-09-16T15:42:40.881820+0000","last_active":"2024-09-16T15:47:01.270624+0000","last_peered":"2024-09-16T15:47:01.270624+0000","last_clean":"2024-09-16T15:47:01.270624+0000","last_became_active":"2024-09-16T15:42:40.879965+0000","last_became_peered":"2024-09-16T15:42:40.879965+0000","last_unstale":"2024-09-16T15:47:01.270624+0000","last_undegraded":"2024-09-16T15:47:01.270624+0000","last_fullsized":"2024-09-16T15:47:01.270624+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:42:39.728762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:42:39.728762+0000","last_clean_scrub_stamp":"2024-09-16T15:42:39.728762+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:22:18.768932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":11,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":10,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542201,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28444,"kb_used_data":1416,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343396,"statfs":{"total":96636764160,"available":96607637504,"internally_reserved":0,"allocated":1449984,"data_stored":945952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542200,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32028,"kb_used_data":836,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339812,"statfs":{"total":96636764160,"available":96603967488,"internally_reserved":0,"allocated":856064,"data_stored":355060,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":5,"up_from":14,"seq":60129542201,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27864,"kb_used_data":832,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343976,"statfs":{"total":96636764160,"available":96608231424,"internally_reserved":0,"allocated":851968,"data_stored":355058,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542199,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32020,"kb_used_data":832,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339820,"statfs":{"total":96636764160,"available":96603975680,"internally_reserved":0,"allocated":851968,"data_stored":355038,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":3,"up_from":14,"seq":60129542200,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32024,"kb_used_data":836,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339816,"statfs":{"total":96636764160,"available":96603971584,"internally_reserved":0,"allocated":856064,"data_stored":355080,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":14,"seq":60129542200,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27872,"kb_used_data":840,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343968,"statfs":{"total":96636764160,"available":96608223232,"internally_reserved":0,"allocated":860160,"data_stored":356580,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":1,"up_from":14,"seq":60129542202,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27876,"kb_used_data":848,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343964,"statfs":{"total":96636764160,"available":96608219136,"internally_reserved":0,"allocated":868352,"data_stored":357196,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":0,"up_from":14,"seq":60129542202,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32596,"kb_used_data":1408,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339244,"statfs":{"total":96636764160,"available":96603385856,"internally_reserved":0,"allocated":1441792,"data_stored":945336,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2228,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T15:47:08.448 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-09-16T15:47:08.448 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-09-16T15:47:08.623 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:08.620+0000 7f39e4c31640 1 -- 172.21.15.123:0/2728489758 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f39cc004d50 msgr2=0x7f39cc0051d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:47:08.623 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:08.620+0000 7f39e4c31640 1 --2- 172.21.15.123:0/2728489758 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f39cc004d50 0x7f39cc0051d0 secure :-1 s=READY pgs=800 cs=0 l=1 rev1=1 crypto rx=0x7f39d800f490 tx=0x7f39d801e410 comp rx=0 tx=0).stop 2024-09-16T15:47:08.623 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:08.620+0000 7f39ded9d640 1 -- 172.21.15.123:0/2728489758 reap_dead start 2024-09-16T15:47:08.623 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:08.620+0000 7f39e4c31640 1 -- 172.21.15.123:0/2728489758 shutdown_connections 2024-09-16T15:47:08.623 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:08.620+0000 7f39e4c31640 1 -- 172.21.15.123:0/2728489758 >> 172.21.15.123:0/2728489758 conn(0x7f39e00b1710 msgr2=0x7f39e00b1b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:47:08.623 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:08.620+0000 7f39e4c31640 1 -- 172.21.15.123:0/2728489758 shutdown_connections 2024-09-16T15:47:08.624 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:08.620+0000 7f39e4c31640 1 -- 172.21.15.123:0/2728489758 wait complete. 2024-09-16T15:47:08.624 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:08.620+0000 7f39e4c31640 1 Processor -- start 2024-09-16T15:47:08.624 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:08.620+0000 7f39e4c31640 1 -- start start 2024-09-16T15:47:08.626 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:08.620+0000 7f39e4c31640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f39cc004d50 0x7f39e00ae9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:47:08.626 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:08.620+0000 7f39e4c31640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f39e014fd10 0x7f39e0144c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:47:08.626 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:08.620+0000 7f39e4c31640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f39e00aef00 0x7f39e0145160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:47:08.626 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:08.620+0000 7f39e4c31640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f39e00cf730 con 0x7f39e014fd10 2024-09-16T15:47:08.626 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:08.620+0000 7f39e4c31640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f39e00cf5b0 con 0x7f39e00aef00 2024-09-16T15:47:08.626 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:08.620+0000 7f39e4c31640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f39e00cf8b0 con 0x7f39cc004d50 2024-09-16T15:47:08.627 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:08.620+0000 7f39ddd9b640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f39e014fd10 0x7f39e0144c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:47:08.627 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:08.620+0000 7f39ddd9b640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f39e014fd10 0x7f39e0144c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:46920/0 (socket says 172.21.15.123:46920) 2024-09-16T15:47:08.627 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:08.620+0000 7f39ddd9b640 1 -- 172.21.15.123:0/404215695 learned_addr learned my addr 172.21.15.123:0/404215695 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:47:08.627 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:08.620+0000 7f39ded9d640 1 --2- 172.21.15.123:0/404215695 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f39e00aef00 0x7f39e0145160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:47:08.627 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:08.620+0000 7f39de59c640 1 --2- 172.21.15.123:0/404215695 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f39cc004d50 0x7f39e00ae9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:47:08.627 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:08.620+0000 7f39ddd9b640 1 -- 172.21.15.123:0/404215695 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f39cc004d50 msgr2=0x7f39e00ae9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:47:08.627 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:08.620+0000 7f39ddd9b640 1 --2- 172.21.15.123:0/404215695 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f39cc004d50 0x7f39e00ae9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:08.627 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:08.620+0000 7f39ddd9b640 1 -- 172.21.15.123:0/404215695 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f39e00aef00 msgr2=0x7f39e0145160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:47:08.627 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:08.620+0000 7f39ddd9b640 1 --2- 172.21.15.123:0/404215695 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f39e00aef00 0x7f39e0145160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:08.627 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:08.620+0000 7f39ddd9b640 1 -- 172.21.15.123:0/404215695 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f39e00ad960 con 0x7f39e014fd10 2024-09-16T15:47:08.627 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:08.620+0000 7f39ddd9b640 1 --2- 172.21.15.123:0/404215695 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f39e014fd10 0x7f39e0144c20 secure :-1 s=READY pgs=801 cs=0 l=1 rev1=1 crypto rx=0x7f39d000d8f0 tx=0x7f39d000ddb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:47:08.627 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:08.620+0000 7f39d6ffd640 1 -- 172.21.15.123:0/404215695 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39d00064e0 con 0x7f39e014fd10 2024-09-16T15:47:08.627 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:08.620+0000 7f39d6ffd640 1 -- 172.21.15.123:0/404215695 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f39d000b920 con 0x7f39e014fd10 2024-09-16T15:47:08.628 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:08.620+0000 7f39e4c31640 1 -- 172.21.15.123:0/404215695 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f39e00adc00 con 0x7f39e014fd10 2024-09-16T15:47:08.628 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:08.620+0000 7f39d6ffd640 1 -- 172.21.15.123:0/404215695 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39d000bd50 con 0x7f39e014fd10 2024-09-16T15:47:08.628 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:08.620+0000 7f39e4c31640 1 -- 172.21.15.123:0/404215695 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f39e00ae1c0 con 0x7f39e014fd10 2024-09-16T15:47:08.628 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:08.620+0000 7f39e4c31640 1 -- 172.21.15.123:0/404215695 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f39e0150700 con 0x7f39e014fd10 2024-09-16T15:47:08.628 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:08.624+0000 7f39d6ffd640 1 -- 172.21.15.123:0/404215695 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 11) ==== 97386+0+0 (secure 0 0 0) 0x7f39d0003600 con 0x7f39e014fd10 2024-09-16T15:47:08.629 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:08.624+0000 7f39d6ffd640 1 --2- 172.21.15.123:0/404215695 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f39c0073220 0x7f39c00756e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:47:08.629 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:08.624+0000 7f39d6ffd640 1 -- 172.21.15.123:0/404215695 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(70..70 src has 1..70) ==== 5709+0+0 (secure 0 0 0) 0x7f39d0094ba0 con 0x7f39e014fd10 2024-09-16T15:47:08.629 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:08.624+0000 7f39de59c640 1 --2- 172.21.15.123:0/404215695 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f39c0073220 0x7f39c00756e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:47:08.629 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:08.624+0000 7f39de59c640 1 --2- 172.21.15.123:0/404215695 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f39c0073220 0x7f39c00756e0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f39c8005fd0 tx=0x7f39c800c620 comp rx=0 tx=0).ready entity=mgr.4135 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:47:08.632 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:08.628+0000 7f39d6ffd640 1 -- 172.21.15.123:0/404215695 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f39d005f860 con 0x7f39e014fd10 2024-09-16T15:47:08.869 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:08.864+0000 7f39e4c31640 1 -- 172.21.15.123:0/404215695 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) -- 0x7f39e00ce470 con 0x7f39e014fd10 2024-09-16T15:47:08.873 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:08.868+0000 7f39d6ffd640 1 -- 172.21.15.123:0/404215695 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v22) ==== 190+0+147 (secure 0 0 0) 0x7f39d005f160 con 0x7f39e014fd10 2024-09-16T15:47:08.873 INFO:teuthology.orchestra.run.smithi123.stdout:[client.0] 2024-09-16T15:47:08.873 INFO:teuthology.orchestra.run.smithi123.stdout: key = AQC1Uehm7GDTAhAAHWWVIyjVVnTeeozBR8DfFA== 2024-09-16T15:47:08.874 INFO:teuthology.orchestra.run.smithi123.stdout: caps mds = "allow" 2024-09-16T15:47:08.874 INFO:teuthology.orchestra.run.smithi123.stdout: caps mon = "allow r" 2024-09-16T15:47:08.874 INFO:teuthology.orchestra.run.smithi123.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-09-16T15:47:08.874 INFO:teuthology.orchestra.run.smithi123.stderr:updated caps for client.0 2024-09-16T15:47:08.876 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:08.872+0000 7f39e4c31640 1 -- 172.21.15.123:0/404215695 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f39c0073220 msgr2=0x7f39c00756e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:47:08.876 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:08.872+0000 7f39e4c31640 1 --2- 172.21.15.123:0/404215695 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f39c0073220 0x7f39c00756e0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f39c8005fd0 tx=0x7f39c800c620 comp rx=0 tx=0).stop 2024-09-16T15:47:08.876 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:08.872+0000 7f39e4c31640 1 -- 172.21.15.123:0/404215695 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f39e014fd10 msgr2=0x7f39e0144c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:47:08.876 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:08.872+0000 7f39e4c31640 1 --2- 172.21.15.123:0/404215695 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f39e014fd10 0x7f39e0144c20 secure :-1 s=READY pgs=801 cs=0 l=1 rev1=1 crypto rx=0x7f39d000d8f0 tx=0x7f39d000ddb0 comp rx=0 tx=0).stop 2024-09-16T15:47:08.876 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:08.872+0000 7f39e4c31640 1 -- 172.21.15.123:0/404215695 shutdown_connections 2024-09-16T15:47:08.877 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:08.872+0000 7f39e4c31640 1 --2- 172.21.15.123:0/404215695 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f39c0073220 0x7f39c00756e0 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:08.877 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:08.872+0000 7f39e4c31640 1 --2- 172.21.15.123:0/404215695 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f39e00aef00 0x7f39e0145160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:08.877 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:08.872+0000 7f39e4c31640 1 --2- 172.21.15.123:0/404215695 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f39e014fd10 0x7f39e0144c20 unknown :-1 s=CLOSED pgs=801 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:08.877 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:08.872+0000 7f39e4c31640 1 --2- 172.21.15.123:0/404215695 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f39cc004d50 0x7f39e00ae9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:08.877 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:08.872+0000 7f39e4c31640 1 -- 172.21.15.123:0/404215695 >> 172.21.15.123:0/404215695 conn(0x7f39e00b1710 msgr2=0x7f39e00b1b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:47:08.877 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:08.872+0000 7f39e4c31640 1 -- 172.21.15.123:0/404215695 shutdown_connections 2024-09-16T15:47:08.877 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:08.872+0000 7f39e4c31640 1 -- 172.21.15.123:0/404215695 wait complete. 2024-09-16T15:47:08.893 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-16T15:47:09.007 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:09.004+0000 7f8a3ddd0640 1 -- 172.21.15.123:0/2339605932 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8a20003a60 msgr2=0x7f8a20003ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:47:09.007 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:09.004+0000 7f8a3ddd0640 1 --2- 172.21.15.123:0/2339605932 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8a20003a60 0x7f8a20003ea0 secure :-1 s=READY pgs=803 cs=0 l=1 rev1=1 crypto rx=0x7f8a2800c2e0 tx=0x7f8a2801cf30 comp rx=0 tx=0).stop 2024-09-16T15:47:09.008 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:09.004+0000 7f8a3ddd0640 1 -- 172.21.15.123:0/2339605932 shutdown_connections 2024-09-16T15:47:09.008 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:09.004+0000 7f8a3ddd0640 1 --2- 172.21.15.123:0/2339605932 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8a20003a60 0x7f8a20003ea0 unknown :-1 s=CLOSED pgs=803 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:09.008 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:09.004+0000 7f8a3ddd0640 1 -- 172.21.15.123:0/2339605932 >> 172.21.15.123:0/2339605932 conn(0x7f8a380cfb10 msgr2=0x7f8a380cff20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:47:09.008 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:09.004+0000 7f8a3ddd0640 1 -- 172.21.15.123:0/2339605932 shutdown_connections 2024-09-16T15:47:09.008 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:09.004+0000 7f8a3ddd0640 1 -- 172.21.15.123:0/2339605932 wait complete. 2024-09-16T15:47:09.008 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:09.004+0000 7f8a3ddd0640 1 Processor -- start 2024-09-16T15:47:09.008 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:09.004+0000 7f8a3ddd0640 1 -- start start 2024-09-16T15:47:09.010 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:09.004+0000 7f8a3ddd0640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8a20003a60 0x7f8a381f9190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:47:09.010 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:09.004+0000 7f8a3ddd0640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f8a381f74c0 0x7f8a381f98f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:47:09.010 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:09.004+0000 7f8a3ddd0640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f8a381f9e30 0x7f8a380c8610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:47:09.010 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:09.004+0000 7f8a3ddd0640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f8a380cad00 con 0x7f8a20003a60 2024-09-16T15:47:09.011 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:09.004+0000 7f8a3ddd0640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f8a380cab80 con 0x7f8a381f9e30 2024-09-16T15:47:09.011 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:09.004+0000 7f8a3ddd0640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f8a380cae80 con 0x7f8a381f74c0 2024-09-16T15:47:09.011 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:09.004+0000 7f8a377fe640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8a20003a60 0x7f8a381f9190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:47:09.011 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:09.004+0000 7f8a377fe640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8a20003a60 0x7f8a381f9190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:46932/0 (socket says 172.21.15.123:46932) 2024-09-16T15:47:09.011 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:09.004+0000 7f8a377fe640 1 -- 172.21.15.123:0/1174803173 learned_addr learned my addr 172.21.15.123:0/1174803173 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:47:09.011 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:09.004+0000 7f8a36ffd640 1 --2- 172.21.15.123:0/1174803173 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f8a381f74c0 0x7f8a381f98f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:47:09.011 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:09.004+0000 7f8a37fff640 1 --2- 172.21.15.123:0/1174803173 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f8a381f9e30 0x7f8a380c8610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:47:09.011 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:09.004+0000 7f8a377fe640 1 -- 172.21.15.123:0/1174803173 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f8a381f74c0 msgr2=0x7f8a381f98f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:47:09.011 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:09.004+0000 7f8a377fe640 1 --2- 172.21.15.123:0/1174803173 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f8a381f74c0 0x7f8a381f98f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:09.011 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:09.004+0000 7f8a377fe640 1 -- 172.21.15.123:0/1174803173 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f8a381f9e30 msgr2=0x7f8a380c8610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:47:09.011 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:09.004+0000 7f8a377fe640 1 --2- 172.21.15.123:0/1174803173 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f8a381f9e30 0x7f8a380c8610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:09.011 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:09.004+0000 7f8a377fe640 1 -- 172.21.15.123:0/1174803173 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8a380c9220 con 0x7f8a20003a60 2024-09-16T15:47:09.012 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:09.004+0000 7f8a377fe640 1 --2- 172.21.15.123:0/1174803173 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8a20003a60 0x7f8a381f9190 secure :-1 s=READY pgs=804 cs=0 l=1 rev1=1 crypto rx=0x7f8a2800c2e0 tx=0x7f8a28006ca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:47:09.012 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:09.004+0000 7f8a1ffff640 1 -- 172.21.15.123:0/1174803173 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a280a5aa0 con 0x7f8a20003a60 2024-09-16T15:47:09.012 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:09.004+0000 7f8a1ffff640 1 -- 172.21.15.123:0/1174803173 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f8a280a5e50 con 0x7f8a20003a60 2024-09-16T15:47:09.012 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:09.004+0000 7f8a3ddd0640 1 -- 172.21.15.123:0/1174803173 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8a380c9500 con 0x7f8a20003a60 2024-09-16T15:47:09.012 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:09.004+0000 7f8a1ffff640 1 -- 172.21.15.123:0/1174803173 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a280060d0 con 0x7f8a20003a60 2024-09-16T15:47:09.012 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:09.004+0000 7f8a3ddd0640 1 -- 172.21.15.123:0/1174803173 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8a38158300 con 0x7f8a20003a60 2024-09-16T15:47:09.013 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:09.008+0000 7f8a3ddd0640 1 -- 172.21.15.123:0/1174803173 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8a3806f080 con 0x7f8a20003a60 2024-09-16T15:47:09.013 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:09.008+0000 7f8a1ffff640 1 -- 172.21.15.123:0/1174803173 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 11) ==== 97386+0+0 (secure 0 0 0) 0x7f8a28006270 con 0x7f8a20003a60 2024-09-16T15:47:09.013 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:09.008+0000 7f8a1ffff640 1 --2- 172.21.15.123:0/1174803173 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f8a24073220 0x7f8a240756e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:47:09.013 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:09.008+0000 7f8a1ffff640 1 -- 172.21.15.123:0/1174803173 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(70..70 src has 1..70) ==== 5709+0+0 (secure 0 0 0) 0x7f8a2812dd50 con 0x7f8a20003a60 2024-09-16T15:47:09.013 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:09.008+0000 7f8a36ffd640 1 --2- 172.21.15.123:0/1174803173 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f8a24073220 0x7f8a240756e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:47:09.013 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:09.008+0000 7f8a36ffd640 1 --2- 172.21.15.123:0/1174803173 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f8a24073220 0x7f8a240756e0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f8a380ca060 tx=0x7f8a2c009040 comp rx=0 tx=0).ready entity=mgr.4135 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:47:09.015 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:09.012+0000 7f8a1ffff640 1 -- 172.21.15.123:0/1174803173 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f8a28016d10 con 0x7f8a20003a60 2024-09-16T15:47:09.207 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:09.200+0000 7f8a3ddd0640 1 -- 172.21.15.123:0/1174803173 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f8a3815f630 con 0x7f8a20003a60 2024-09-16T15:47:09.208 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:09.204+0000 7f8a1ffff640 1 -- 172.21.15.123:0/1174803173 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 44 v44) ==== 94+0+5234 (secure 0 0 0) 0x7f8a280f8b10 con 0x7f8a20003a60 2024-09-16T15:47:09.208 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-09-16T15:47:09.208 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":44,"btime":"2024-09-16T15:47:02:606606+0000","default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5048,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6837/1498917352","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6835","nonce":1498917352},{"type":"v1","addr":"172.21.15.184:6837","nonce":1498917352}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":40},{"gid":6198,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6833/1715139909","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6832","nonce":1715139909},{"type":"v1","addr":"172.21.15.123:6833","nonce":1715139909}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":40},{"gid":6201,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6835/815923338","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6834","nonce":815923338},{"type":"v1","addr":"172.21.15.123:6835","nonce":815923338}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":40}],"filesystems":[{"mdsmap":{"epoch":44,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-16T15:46:58.916477+0000","modified":"2024-09-16T15:47:01.963874+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6216},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6216":{"gid":6216,"name":"d","rank":0,"incarnation":41,"state":"up:active","state_seq":3,"addr":"172.21.15.184:6836/3137795986","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":3137795986},{"type":"v1","addr":"172.21.15.184:6836","nonce":3137795986}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6216,"qdb_cluster":[6216]},"id":5}]} 2024-09-16T15:47:09.208 INFO:teuthology.orchestra.run.smithi123.stderr:dumped fsmap epoch 44 2024-09-16T15:47:09.210 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:09.204+0000 7f8a3ddd0640 1 -- 172.21.15.123:0/1174803173 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f8a24073220 msgr2=0x7f8a240756e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:47:09.210 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:09.204+0000 7f8a3ddd0640 1 --2- 172.21.15.123:0/1174803173 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f8a24073220 0x7f8a240756e0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f8a380ca060 tx=0x7f8a2c009040 comp rx=0 tx=0).stop 2024-09-16T15:47:09.210 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:09.204+0000 7f8a3ddd0640 1 -- 172.21.15.123:0/1174803173 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8a20003a60 msgr2=0x7f8a381f9190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:47:09.210 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:09.204+0000 7f8a3ddd0640 1 --2- 172.21.15.123:0/1174803173 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8a20003a60 0x7f8a381f9190 secure :-1 s=READY pgs=804 cs=0 l=1 rev1=1 crypto rx=0x7f8a2800c2e0 tx=0x7f8a28006ca0 comp rx=0 tx=0).stop 2024-09-16T15:47:09.210 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:09.204+0000 7f8a3ddd0640 1 -- 172.21.15.123:0/1174803173 shutdown_connections 2024-09-16T15:47:09.210 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:09.204+0000 7f8a3ddd0640 1 --2- 172.21.15.123:0/1174803173 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f8a24073220 0x7f8a240756e0 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:09.210 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:09.204+0000 7f8a3ddd0640 1 --2- 172.21.15.123:0/1174803173 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f8a381f9e30 0x7f8a380c8610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:09.210 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:09.204+0000 7f8a3ddd0640 1 --2- 172.21.15.123:0/1174803173 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f8a381f74c0 0x7f8a381f98f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:09.210 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:09.204+0000 7f8a3ddd0640 1 --2- 172.21.15.123:0/1174803173 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8a20003a60 0x7f8a381f9190 unknown :-1 s=CLOSED pgs=804 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:09.211 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:09.204+0000 7f8a3ddd0640 1 -- 172.21.15.123:0/1174803173 >> 172.21.15.123:0/1174803173 conn(0x7f8a380cfb10 msgr2=0x7f8a380c35e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:47:09.211 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:09.204+0000 7f8a3ddd0640 1 -- 172.21.15.123:0/1174803173 shutdown_connections 2024-09-16T15:47:09.211 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:09.204+0000 7f8a3ddd0640 1 -- 172.21.15.123:0/1174803173 wait complete. 2024-09-16T15:47:09.553 DEBUG:teuthology.orchestra.run.smithi123:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-16T15:47:09.556 DEBUG:teuthology.orchestra.run.smithi184:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-16T15:47:09.627 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 44, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-09-16T15:46:58.916477+0000', 'modified': '2024-09-16T15:47:01.963874+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 6216}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6216': {'gid': 6216, 'name': 'd', 'rank': 0, 'incarnation': 41, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.184:6836/3137795986', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.184:6834', 'nonce': 3137795986}, {'type': 'v1', 'addr': '172.21.15.184:6836', 'nonce': 3137795986}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540719139924082687, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [11], 'metadata_pool': 10, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 6216, 'qdb_cluster': [6216]} 2024-09-16T15:47:09.627 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-09-16T15:47:09.628 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.d status 2024-09-16T15:47:09.761 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:09.756+0000 7f4d4933a640 1 -- 172.21.15.123:0/967984180 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4d440cc8c0 msgr2=0x7f4d440cf650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:47:09.761 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:09.756+0000 7f4d4933a640 1 --2- 172.21.15.123:0/967984180 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4d440cc8c0 0x7f4d440cf650 secure :-1 s=READY pgs=806 cs=0 l=1 rev1=1 crypto rx=0x7f4d44051540 tx=0x7f4d3c01cb80 comp rx=0 tx=0).stop 2024-09-16T15:47:09.762 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:09.756+0000 7f4d4933a640 1 -- 172.21.15.123:0/967984180 shutdown_connections 2024-09-16T15:47:09.762 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:09.756+0000 7f4d4933a640 1 --2- 172.21.15.123:0/967984180 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4d440cc8c0 0x7f4d440cf650 unknown :-1 s=CLOSED pgs=806 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:09.762 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:09.756+0000 7f4d4933a640 1 --2- 172.21.15.123:0/967984180 >> v2:172.21.15.184:3300/0 conn(0x7f4d4415ac00 0x7f4d440702f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:09.762 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:09.756+0000 7f4d4933a640 1 -- 172.21.15.123:0/967984180 >> 172.21.15.123:0/967984180 conn(0x7f4d440b2610 msgr2=0x7f4d440b29f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:47:09.762 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:09.756+0000 7f4d4933a640 1 -- 172.21.15.123:0/967984180 shutdown_connections 2024-09-16T15:47:09.762 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:09.756+0000 7f4d4933a640 1 -- 172.21.15.123:0/967984180 wait complete. 2024-09-16T15:47:09.762 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:09.756+0000 7f4d4933a640 1 Processor -- start 2024-09-16T15:47:09.762 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:09.756+0000 7f4d4933a640 1 -- start start 2024-09-16T15:47:09.764 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:09.760+0000 7f4d4933a640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4d440b3100 0x7f4d44152e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:47:09.764 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:09.760+0000 7f4d4933a640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f4d440cc8c0 0x7f4d44156610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:47:09.764 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:09.760+0000 7f4d4933a640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4d4415ac00 0x7f4d440ca1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:47:09.765 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:09.760+0000 7f4d4933a640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f4d44168660 con 0x7f4d4415ac00 2024-09-16T15:47:09.765 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:09.760+0000 7f4d4933a640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f4d441684e0 con 0x7f4d440b3100 2024-09-16T15:47:09.765 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:09.760+0000 7f4d4933a640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f4d441687e0 con 0x7f4d440cc8c0 2024-09-16T15:47:09.765 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:09.760+0000 7f4d437fe640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4d4415ac00 0x7f4d440ca1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:47:09.765 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:09.760+0000 7f4d437fe640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4d4415ac00 0x7f4d440ca1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:46946/0 (socket says 172.21.15.123:46946) 2024-09-16T15:47:09.765 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:09.760+0000 7f4d437fe640 1 -- 172.21.15.123:0/1878007117 learned_addr learned my addr 172.21.15.123:0/1878007117 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:47:09.765 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:09.760+0000 7f4d437fe640 1 -- 172.21.15.123:0/1878007117 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f4d440cc8c0 msgr2=0x7f4d44156610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:47:09.765 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:09.760+0000 7f4d42ffd640 1 --2- 172.21.15.123:0/1878007117 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4d440b3100 0x7f4d44152e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:47:09.765 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:09.760+0000 7f4d3a7fc640 1 --2- 172.21.15.123:0/1878007117 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f4d440cc8c0 0x7f4d44156610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:47:09.765 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:09.760+0000 7f4d437fe640 1 --2- 172.21.15.123:0/1878007117 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f4d440cc8c0 0x7f4d44156610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:09.765 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:09.760+0000 7f4d437fe640 1 -- 172.21.15.123:0/1878007117 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4d440b3100 msgr2=0x7f4d44152e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:47:09.765 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:09.760+0000 7f4d437fe640 1 --2- 172.21.15.123:0/1878007117 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4d440b3100 0x7f4d44152e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:09.765 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:09.760+0000 7f4d437fe640 1 -- 172.21.15.123:0/1878007117 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4d4415be60 con 0x7f4d4415ac00 2024-09-16T15:47:09.765 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:09.760+0000 7f4d42ffd640 1 --2- 172.21.15.123:0/1878007117 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4d440b3100 0x7f4d44152e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T15:47:09.765 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:09.760+0000 7f4d437fe640 1 --2- 172.21.15.123:0/1878007117 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4d4415ac00 0x7f4d440ca1f0 secure :-1 s=READY pgs=807 cs=0 l=1 rev1=1 crypto rx=0x7f4d3c009b10 tx=0x7f4d3c0a5940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:47:09.766 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:09.760+0000 7f4d3bfff640 1 -- 172.21.15.123:0/1878007117 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d3c0bb070 con 0x7f4d4415ac00 2024-09-16T15:47:09.766 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:09.760+0000 7f4d3bfff640 1 -- 172.21.15.123:0/1878007117 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f4d3c004e00 con 0x7f4d4415ac00 2024-09-16T15:47:09.766 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:09.760+0000 7f4d4933a640 1 -- 172.21.15.123:0/1878007117 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4d4415e3a0 con 0x7f4d4415ac00 2024-09-16T15:47:09.766 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:09.760+0000 7f4d3bfff640 1 -- 172.21.15.123:0/1878007117 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d3c0b2040 con 0x7f4d4415ac00 2024-09-16T15:47:09.766 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:09.760+0000 7f4d4933a640 1 -- 172.21.15.123:0/1878007117 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4d4415c400 con 0x7f4d4415ac00 2024-09-16T15:47:09.767 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:09.764+0000 7f4d4c424000 1 Processor -- start 2024-09-16T15:47:09.767 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:09.764+0000 7f4d4c424000 1 -- start start 2024-09-16T15:47:09.769 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:09.764+0000 7f4d4c424000 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55dfa260b9d0 0x55dfa260bdb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:47:09.770 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:09.764+0000 7f4d4c424000 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x55dfa260c2f0 0x55dfa25bf8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:47:09.770 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:09.764+0000 7f4d4c424000 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55dfa25bfe10 0x55dfa25c01f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:47:09.770 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:09.764+0000 7f4d4c424000 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x55dfa24e03f0 con 0x55dfa25bfe10 2024-09-16T15:47:09.770 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:09.764+0000 7f4d4c424000 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x55dfa241db20 con 0x55dfa260b9d0 2024-09-16T15:47:09.770 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:09.764+0000 7f4d4c424000 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x55dfa240cf50 con 0x55dfa260c2f0 2024-09-16T15:47:09.770 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:09.764+0000 7f4d3a7fc640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55dfa25bfe10 0x55dfa25c01f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:47:09.770 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:09.764+0000 7f4d3a7fc640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55dfa25bfe10 0x55dfa25c01f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:46962/0 (socket says 172.21.15.123:46962) 2024-09-16T15:47:09.770 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:09.764+0000 7f4d3a7fc640 1 -- 172.21.15.123:0/1337969545 learned_addr learned my addr 172.21.15.123:0/1337969545 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:47:09.770 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:09.764+0000 7f4d42ffd640 1 --2- 172.21.15.123:0/1337969545 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x55dfa260c2f0 0x55dfa25bf8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:47:09.770 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:09.764+0000 7f4d3a7fc640 1 --2- 172.21.15.123:0/1337969545 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55dfa260b9d0 0x55dfa260bdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:47:09.770 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:09.764+0000 7f4d3a7fc640 1 -- 172.21.15.123:0/1337969545 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x55dfa260c2f0 msgr2=0x55dfa25bf8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T15:47:09.771 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:09.764+0000 7f4d3a7fc640 1 --2- 172.21.15.123:0/1337969545 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x55dfa260c2f0 0x55dfa25bf8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:09.771 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:09.764+0000 7f4d3a7fc640 1 -- 172.21.15.123:0/1337969545 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55dfa260b9d0 msgr2=0x55dfa260bdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T15:47:09.771 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:09.764+0000 7f4d3a7fc640 1 --2- 172.21.15.123:0/1337969545 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55dfa260b9d0 0x55dfa260bdb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:09.771 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:09.764+0000 7f4d3a7fc640 1 -- 172.21.15.123:0/1337969545 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55dfa23d4590 con 0x55dfa25bfe10 2024-09-16T15:47:09.771 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:09.764+0000 7f4d3bfff640 1 -- 172.21.15.123:0/1878007117 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 11) ==== 97386+0+0 (secure 0 0 0) 0x7f4d3c0be070 con 0x7f4d4415ac00 2024-09-16T15:47:09.771 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:09.764+0000 7f4d3a7fc640 1 --2- 172.21.15.123:0/1337969545 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55dfa25bfe10 0x55dfa25c01f0 secure :-1 s=READY pgs=808 cs=0 l=1 rev1=1 crypto rx=0x7f4d2c001c10 tx=0x7f4d2c007de0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e51ead3f6d895003 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:47:09.771 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:09.764+0000 7f4d38ff9640 1 -- 172.21.15.123:0/1337969545 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d2c01c070 con 0x55dfa25bfe10 2024-09-16T15:47:09.771 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:09.764+0000 7f4d38ff9640 1 -- 172.21.15.123:0/1337969545 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f4d2c0173f0 con 0x55dfa25bfe10 2024-09-16T15:47:09.771 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:09.764+0000 7f4d38ff9640 1 -- 172.21.15.123:0/1337969545 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d2c00d070 con 0x55dfa25bfe10 2024-09-16T15:47:09.771 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:09.764+0000 7f4d3bfff640 1 --2- 172.21.15.123:0/1878007117 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f4d24073250 0x7f4d24075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:47:09.771 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:09.764+0000 7f4d3bfff640 1 -- 172.21.15.123:0/1878007117 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(70..70 src has 1..70) ==== 5709+0+0 (secure 0 0 0) 0x7f4d3c12e2e0 con 0x7f4d4415ac00 2024-09-16T15:47:09.771 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:09.764+0000 7f4d4c424000 1 -- 172.21.15.123:0/1337969545 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55dfa25bfe10 msgr2=0x55dfa25c01f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:47:09.771 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:09.764+0000 7f4d4c424000 1 --2- 172.21.15.123:0/1337969545 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55dfa25bfe10 0x55dfa25c01f0 secure :-1 s=READY pgs=808 cs=0 l=1 rev1=1 crypto rx=0x7f4d2c001c10 tx=0x7f4d2c007de0 comp rx=0 tx=0).stop 2024-09-16T15:47:09.772 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:09.764+0000 7f4d4c424000 1 -- 172.21.15.123:0/1337969545 shutdown_connections 2024-09-16T15:47:09.772 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:09.764+0000 7f4d4c424000 1 --2- 172.21.15.123:0/1337969545 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55dfa25bfe10 0x55dfa25c01f0 unknown :-1 s=CLOSED pgs=808 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:09.772 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:09.764+0000 7f4d4c424000 1 --2- 172.21.15.123:0/1337969545 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x55dfa260c2f0 0x55dfa25bf8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:09.772 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:09.764+0000 7f4d42ffd640 1 --2- 172.21.15.123:0/1878007117 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f4d24073250 0x7f4d24075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:47:09.772 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:09.764+0000 7f4d4c424000 1 --2- 172.21.15.123:0/1337969545 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55dfa260b9d0 0x55dfa260bdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:09.772 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:09.764+0000 7f4d4c424000 1 -- 172.21.15.123:0/1337969545 >> 172.21.15.123:0/1337969545 conn(0x55dfa24a9130 msgr2=0x55dfa25eee80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:47:09.772 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:09.764+0000 7f4d4c424000 1 -- 172.21.15.123:0/1337969545 shutdown_connections 2024-09-16T15:47:09.772 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:09.764+0000 7f4d4c424000 1 -- 172.21.15.123:0/1337969545 wait complete. 2024-09-16T15:47:09.775 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:09.764+0000 7f4d4c424000 1 -- start start 2024-09-16T15:47:09.775 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:09.764+0000 7f4d4c424000 1 Processor -- start 2024-09-16T15:47:09.775 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:09.764+0000 7f4d42ffd640 1 --2- 172.21.15.123:0/1878007117 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f4d24073250 0x7f4d24075710 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f4d340058d0 tx=0x7f4d3400b370 comp rx=0 tx=0).ready entity=mgr.4135 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:47:09.775 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:09.768+0000 7f4d4c424000 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55dfa25bfe10 0x55dfa25ec6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:47:09.775 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:09.768+0000 7f4d4c424000 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x55dfa260b9d0 0x55dfa25ece70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:47:09.775 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:09.768+0000 7f4d4c424000 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55dfa260c2f0 0x55dfa25ed3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:47:09.775 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:09.768+0000 7f4d4c424000 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x55dfa23a75d0 con 0x55dfa25bfe10 2024-09-16T15:47:09.775 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:09.768+0000 7f4d4c424000 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x55dfa2410d60 con 0x55dfa260c2f0 2024-09-16T15:47:09.775 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:09.768+0000 7f4d4c424000 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x55dfa23b3500 con 0x55dfa260b9d0 2024-09-16T15:47:09.776 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:09.768+0000 7f4d4c424000 10 client.0 _ll_register_callbacks cb 0x55dfa23b7810 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T15:47:09.776 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:09.768+0000 7f4d42ffd640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55dfa25bfe10 0x55dfa25ec6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:47:09.776 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:09.768+0000 7f4d42ffd640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55dfa25bfe10 0x55dfa25ec6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:46966/0 (socket says 172.21.15.123:46966) 2024-09-16T15:47:09.776 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:09.768+0000 7f4d42ffd640 1 -- 172.21.15.123:0/3871455982 learned_addr learned my addr 172.21.15.123:0/3871455982 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:47:09.776 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:09.768+0000 7f4d3a7fc640 1 --2- 172.21.15.123:0/3871455982 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x55dfa260b9d0 0x55dfa25ece70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:47:09.776 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:09.768+0000 7f4d437fe640 1 --2- 172.21.15.123:0/3871455982 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55dfa260c2f0 0x55dfa25ed3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:47:09.776 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:09.768+0000 7f4d42ffd640 1 -- 172.21.15.123:0/3871455982 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x55dfa260b9d0 msgr2=0x55dfa25ece70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T15:47:09.776 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:09.768+0000 7f4d42ffd640 1 --2- 172.21.15.123:0/3871455982 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x55dfa260b9d0 0x55dfa25ece70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:09.776 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:09.768+0000 7f4d42ffd640 1 -- 172.21.15.123:0/3871455982 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55dfa260c2f0 msgr2=0x55dfa25ed3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T15:47:09.776 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:09.768+0000 7f4d42ffd640 1 --2- 172.21.15.123:0/3871455982 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55dfa260c2f0 0x55dfa25ed3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:09.776 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:09.768+0000 7f4d42ffd640 1 -- 172.21.15.123:0/3871455982 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55dfa23ab170 con 0x55dfa25bfe10 2024-09-16T15:47:09.776 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:09.768+0000 7f4d42ffd640 1 --2- 172.21.15.123:0/3871455982 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55dfa25bfe10 0x55dfa25ec6a0 secure :-1 s=READY pgs=809 cs=0 l=1 rev1=1 crypto rx=0x7f4d34005e10 tx=0x7f4d340046d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fd91125f885a0456 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:47:09.776 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:09.768+0000 7f4d21ffb640 10 client.0 ms_handle_connect on v2:172.21.15.123:3300/0 2024-09-16T15:47:09.777 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:09.768+0000 7f4d21ffb640 1 -- 172.21.15.123:0/3871455982 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d3400e980 con 0x55dfa25bfe10 2024-09-16T15:47:09.777 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:09.768+0000 7f4d21ffb640 1 -- 172.21.15.123:0/3871455982 <== mon.0 v2:172.21.15.123:3300/0 2 ==== osd_map(70..70 src has 1..70) ==== 5709+0+0 (secure 0 0 0) 0x7f4d3400eb20 con 0x55dfa25bfe10 2024-09-16T15:47:09.777 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:09.768+0000 7f4d4c424000 1 -- 172.21.15.123:0/3871455982 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55dfa23a50f0 con 0x55dfa25bfe10 2024-09-16T15:47:09.777 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:09.768+0000 7f4d21ffb640 1 -- 172.21.15.123:0/3871455982 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_get_version_reply(handle=1 version=44) ==== 24+0+0 (secure 0 0 0) 0x7f4d3400ed20 con 0x55dfa25bfe10 2024-09-16T15:47:09.777 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:09.768+0000 7f4d4c424000 10 client.6366 fetch_fsmap learned FSMap version 44 2024-09-16T15:47:09.777 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:09.768+0000 7f4d4c424000 1 -- 172.21.15.123:0/3871455982 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,fsmap=44,monmap=2+}) -- 0x55dfa2491ef0 con 0x55dfa25bfe10 2024-09-16T15:47:09.777 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:09.768+0000 7f4d21ffb640 1 -- 172.21.15.123:0/3871455982 <== mon.0 v2:172.21.15.123:3300/0 4 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x55dfa23a50f0 con 0x55dfa25bfe10 2024-09-16T15:47:09.777 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:09.768+0000 7f4d21ffb640 1 -- 172.21.15.123:0/3871455982 <== mon.0 v2:172.21.15.123:3300/0 5 ==== fsmap(e 44) ==== 3502+0+0 (secure 0 0 0) 0x7f4d34032080 con 0x55dfa25bfe10 2024-09-16T15:47:09.777 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:09.768+0000 7f4d4c424000 10 client.6366 fetch_fsmap finished waiting for FSMap version 44 2024-09-16T15:47:09.777 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:09.768+0000 7f4d4c424000 10 client.6366 resolve_mds: resolved name 'd' to daemon mds.d 2024-09-16T15:47:09.777 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:09.768+0000 7f4d4c424000 20 client.6366 populate_metadata read hostname 'smithi123' 2024-09-16T15:47:09.777 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:09.768+0000 7f4d4c424000 1 --2- 172.21.15.123:0/3871455982 >> [v2:172.21.15.184:6834/3137795986,v1:172.21.15.184:6836/3137795986] conn(0x55dfa25ee0f0 0x55dfa261dc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:47:09.777 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:09.768+0000 7f4d4c424000 4 client.6366 mds_command: new command op to 6216 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-16T15:47:09.777 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:09.768+0000 7f4d4c424000 1 -- 172.21.15.123:0/3871455982 --> [v2:172.21.15.184:6834/3137795986,v1:172.21.15.184:6836/3137795986] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55dfa241fa70 con 0x55dfa25ee0f0 2024-09-16T15:47:09.778 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:09.768+0000 7f4d3a7fc640 1 --2- 172.21.15.123:0/3871455982 >> [v2:172.21.15.184:6834/3137795986,v1:172.21.15.184:6836/3137795986] conn(0x55dfa25ee0f0 0x55dfa261dc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:47:09.778 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:09.768+0000 7f4d21ffb640 10 client.6366 ms_handle_connect on v2:172.21.15.184:6834/3137795986 2024-09-16T15:47:09.778 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:09.768+0000 7f4d3a7fc640 1 --2- 172.21.15.123:0/3871455982 >> [v2:172.21.15.184:6834/3137795986,v1:172.21.15.184:6836/3137795986] conn(0x55dfa25ee0f0 0x55dfa261dc90 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fcde04abb495333c server_cookie=9e3d63c727d05c07 in_seq=0 out_seq=0 2024-09-16T15:47:09.779 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:09.772+0000 7f4d21ffb640 1 -- 172.21.15.123:0/3871455982 <== mds.0 v2:172.21.15.184:6834/3137795986 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x55dfa241fa70 con 0x55dfa25ee0f0 2024-09-16T15:47:09.779 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:09.772+0000 7f4d21ffb640 10 client.6366 handle_command_reply: tid=0 2024-09-16T15:47:09.779 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:09.772+0000 7f4d4c424000 1 client.6366 shutdown 2024-09-16T15:47:09.779 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:09.772+0000 7f4d4c424000 1 -- 172.21.15.123:0/3871455982 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55dfa25bfe10 msgr2=0x55dfa25ec6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:47:09.779 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:09.772+0000 7f4d4c424000 1 --2- 172.21.15.123:0/3871455982 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55dfa25bfe10 0x55dfa25ec6a0 secure :-1 s=READY pgs=809 cs=0 l=1 rev1=1 crypto rx=0x7f4d34005e10 tx=0x7f4d340046d0 comp rx=0 tx=0).stop 2024-09-16T15:47:09.779 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:09.772+0000 7f4d4c424000 1 -- 172.21.15.123:0/3871455982 shutdown_connections 2024-09-16T15:47:09.779 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:09.772+0000 7f4d4c424000 1 --2- 172.21.15.123:0/3871455982 >> [v2:172.21.15.184:6834/3137795986,v1:172.21.15.184:6836/3137795986] conn(0x55dfa25ee0f0 0x55dfa261dc90 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:09.779 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:09.772+0000 7f4d4c424000 1 --2- 172.21.15.123:0/3871455982 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55dfa260c2f0 0x55dfa25ed3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:09.779 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:09.772+0000 7f4d4c424000 1 --2- 172.21.15.123:0/3871455982 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x55dfa260b9d0 0x55dfa25ece70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:09.779 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:09.772+0000 7f4d4c424000 1 --2- 172.21.15.123:0/3871455982 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55dfa25bfe10 0x55dfa25ec6a0 unknown :-1 s=CLOSED pgs=809 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:09.779 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:09.772+0000 7f4d21ffb640 0 client.6366 ms_handle_reset on v2:172.21.15.184:6834/3137795986 2024-09-16T15:47:09.779 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:09.772+0000 7f4d4c424000 1 -- 172.21.15.123:0/3871455982 >> 172.21.15.123:0/3871455982 conn(0x55dfa24a9130 msgr2=0x55dfa25befd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:47:09.780 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:09.772+0000 7f4d4c424000 1 -- 172.21.15.123:0/3871455982 shutdown_connections 2024-09-16T15:47:09.780 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:09.772+0000 7f4d4c424000 1 -- 172.21.15.123:0/3871455982 wait complete. 2024-09-16T15:47:09.780 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:09.772+0000 7f4d4c424000 20 client.6366 trim_cache size 0 max 16384 2024-09-16T15:47:09.794 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:09.788+0000 7f4d4c424000 1 Processor -- start 2024-09-16T15:47:09.795 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:09.792+0000 7f4d4c424000 1 -- start start 2024-09-16T15:47:09.796 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:09.792+0000 7f4d4c424000 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x55dfa25bfe10 0x55dfa26220a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:47:09.796 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:09.792+0000 7f4d4c424000 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55dfa260b9d0 0x55dfa26225e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:47:09.796 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:09.792+0000 7f4d4c424000 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55dfa260c2f0 0x55dfa25be0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:47:09.796 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:09.792+0000 7f4d4c424000 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f4d442023f0 con 0x55dfa260b9d0 2024-09-16T15:47:09.797 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:09.792+0000 7f4d4c424000 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f4d44201470 con 0x55dfa260c2f0 2024-09-16T15:47:09.797 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:09.792+0000 7f4d4c424000 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f4d4406d9b0 con 0x55dfa25bfe10 2024-09-16T15:47:09.797 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:09.792+0000 7f4d3a7fc640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55dfa260b9d0 0x55dfa26225e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:47:09.797 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:09.792+0000 7f4d3a7fc640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55dfa260b9d0 0x55dfa26225e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:46970/0 (socket says 172.21.15.123:46970) 2024-09-16T15:47:09.797 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:09.792+0000 7f4d3a7fc640 1 -- 172.21.15.123:0/3480163625 learned_addr learned my addr 172.21.15.123:0/3480163625 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:47:09.797 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:09.792+0000 7f4d3a7fc640 1 -- 172.21.15.123:0/3480163625 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x55dfa25bfe10 msgr2=0x55dfa26220a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T15:47:09.797 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:09.792+0000 7f4d3a7fc640 1 --2- 172.21.15.123:0/3480163625 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x55dfa25bfe10 0x55dfa26220a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:09.797 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:09.792+0000 7f4d3a7fc640 1 -- 172.21.15.123:0/3480163625 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55dfa260c2f0 msgr2=0x55dfa25be0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T15:47:09.797 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:09.792+0000 7f4d3a7fc640 1 --2- 172.21.15.123:0/3480163625 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55dfa260c2f0 0x55dfa25be0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:09.797 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:09.792+0000 7f4d3a7fc640 1 -- 172.21.15.123:0/3480163625 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55dfa2468d90 con 0x55dfa260b9d0 2024-09-16T15:47:09.797 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:09.792+0000 7f4d3a7fc640 1 --2- 172.21.15.123:0/3480163625 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55dfa260b9d0 0x55dfa26225e0 secure :-1 s=READY pgs=810 cs=0 l=1 rev1=1 crypto rx=0x7f4d2c016000 tx=0x7f4d2c000ee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=71e409b41ce3a719 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:47:09.797 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:09.792+0000 7f4d23fff640 1 -- 172.21.15.123:0/3480163625 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d2c00d040 con 0x55dfa260b9d0 2024-09-16T15:47:09.797 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:09.792+0000 7f4d23fff640 1 -- 172.21.15.123:0/3480163625 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f4d2c02e070 con 0x55dfa260b9d0 2024-09-16T15:47:09.797 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:09.792+0000 7f4d23fff640 1 -- 172.21.15.123:0/3480163625 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d2c01c040 con 0x55dfa260b9d0 2024-09-16T15:47:09.802 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:09.792+0000 7f4d4c424000 1 -- 172.21.15.123:0/3480163625 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55dfa260b9d0 msgr2=0x55dfa26225e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:47:09.802 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:09.792+0000 7f4d4c424000 1 --2- 172.21.15.123:0/3480163625 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55dfa260b9d0 0x55dfa26225e0 secure :-1 s=READY pgs=810 cs=0 l=1 rev1=1 crypto rx=0x7f4d2c016000 tx=0x7f4d2c000ee0 comp rx=0 tx=0).stop 2024-09-16T15:47:09.802 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:09.792+0000 7f4d4c424000 1 -- 172.21.15.123:0/3480163625 shutdown_connections 2024-09-16T15:47:09.802 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:09.792+0000 7f4d4c424000 1 --2- 172.21.15.123:0/3480163625 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55dfa260c2f0 0x55dfa25be0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:09.802 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:09.792+0000 7f4d4c424000 1 --2- 172.21.15.123:0/3480163625 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55dfa260b9d0 0x55dfa26225e0 unknown :-1 s=CLOSED pgs=810 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:09.802 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:09.792+0000 7f4d4c424000 1 --2- 172.21.15.123:0/3480163625 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x55dfa25bfe10 0x55dfa26220a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:09.802 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:09.792+0000 7f4d4c424000 1 -- 172.21.15.123:0/3480163625 >> 172.21.15.123:0/3480163625 conn(0x55dfa24a9130 msgr2=0x55dfa25c7b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:47:09.802 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:09.792+0000 7f4d4c424000 1 -- 172.21.15.123:0/3480163625 shutdown_connections 2024-09-16T15:47:09.802 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:09.792+0000 7f4d4c424000 1 -- 172.21.15.123:0/3480163625 wait complete. 2024-09-16T15:47:09.802 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:09.792+0000 7f4d4c424000 1 -- start start 2024-09-16T15:47:09.803 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:09.792+0000 7f4d4c424000 1 Processor -- start 2024-09-16T15:47:09.803 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:09.792+0000 7f4d4c424000 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55dfa25bfe10 0x55dfa2685fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:47:09.803 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:09.792+0000 7f4d4c424000 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55dfa260b9d0 0x55dfa2686520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:47:09.803 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:09.792+0000 7f4d4c424000 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x55dfa260c2f0 0x55dfa2686a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:47:09.803 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:09.792+0000 7f4d4c424000 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x55dfa2410070 con 0x55dfa25bfe10 2024-09-16T15:47:09.803 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:09.792+0000 7f4d4c424000 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x55dfa2415390 con 0x55dfa260b9d0 2024-09-16T15:47:09.803 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:09.792+0000 7f4d4c424000 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x55dfa241c8f0 con 0x55dfa260c2f0 2024-09-16T15:47:09.803 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:09.792+0000 7f4d4c424000 10 client.0 _ll_register_callbacks cb 0x7f4d3400b870 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T15:47:09.803 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:09.792+0000 7f4d42ffd640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55dfa25bfe10 0x55dfa2685fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:47:09.803 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:09.792+0000 7f4d42ffd640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55dfa25bfe10 0x55dfa2685fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:46976/0 (socket says 172.21.15.123:46976) 2024-09-16T15:47:09.803 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:09.792+0000 7f4d42ffd640 1 -- 172.21.15.123:0/2948776033 learned_addr learned my addr 172.21.15.123:0/2948776033 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:47:09.803 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:09.792+0000 7f4d42ffd640 1 -- 172.21.15.123:0/2948776033 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x55dfa260c2f0 msgr2=0x55dfa2686a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T15:47:09.803 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:09.796+0000 7f4d42ffd640 1 --2- 172.21.15.123:0/2948776033 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x55dfa260c2f0 0x55dfa2686a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:09.803 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:09.796+0000 7f4d42ffd640 1 -- 172.21.15.123:0/2948776033 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55dfa260b9d0 msgr2=0x55dfa2686520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T15:47:09.804 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:09.796+0000 7f4d42ffd640 1 --2- 172.21.15.123:0/2948776033 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55dfa260b9d0 0x55dfa2686520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:09.804 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:09.796+0000 7f4d42ffd640 1 -- 172.21.15.123:0/2948776033 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55dfa23d3a90 con 0x55dfa25bfe10 2024-09-16T15:47:09.804 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:09.796+0000 7f4d42ffd640 1 --2- 172.21.15.123:0/2948776033 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55dfa25bfe10 0x55dfa2685fe0 secure :-1 s=READY pgs=811 cs=0 l=1 rev1=1 crypto rx=0x7f4d34005f10 tx=0x7f4d3400ea30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=54e3b1200d75f0ec server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:47:09.804 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:09.796+0000 7f4d21ffb640 10 client.0 ms_handle_connect on v2:172.21.15.123:3300/0 2024-09-16T15:47:09.804 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:09.796+0000 7f4d21ffb640 1 -- 172.21.15.123:0/2948776033 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d34027a30 con 0x55dfa25bfe10 2024-09-16T15:47:09.804 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:09.796+0000 7f4d21ffb640 1 -- 172.21.15.123:0/2948776033 <== mon.0 v2:172.21.15.123:3300/0 2 ==== osd_map(70..70 src has 1..70) ==== 5709+0+0 (secure 0 0 0) 0x7f4d34027bd0 con 0x55dfa25bfe10 2024-09-16T15:47:09.804 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:09.796+0000 7f4d4c424000 1 -- 172.21.15.123:0/2948776033 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4d2c02e070 con 0x55dfa25bfe10 2024-09-16T15:47:09.804 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:09.796+0000 7f4d21ffb640 1 -- 172.21.15.123:0/2948776033 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_get_version_reply(handle=1 version=44) ==== 24+0+0 (secure 0 0 0) 0x7f4d34041050 con 0x55dfa25bfe10 2024-09-16T15:47:09.804 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:09.796+0000 7f4d4c424000 10 client.6372 fetch_fsmap learned FSMap version 44 2024-09-16T15:47:09.804 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:09.796+0000 7f4d4c424000 1 -- 172.21.15.123:0/2948776033 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,fsmap=44,monmap=2+}) -- 0x55dfa23b2050 con 0x55dfa25bfe10 2024-09-16T15:47:09.804 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:09.796+0000 7f4d21ffb640 1 -- 172.21.15.123:0/2948776033 <== mon.0 v2:172.21.15.123:3300/0 4 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f4d2c02e070 con 0x55dfa25bfe10 2024-09-16T15:47:09.804 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:09.796+0000 7f4d21ffb640 1 -- 172.21.15.123:0/2948776033 <== mon.0 v2:172.21.15.123:3300/0 5 ==== fsmap(e 44) ==== 3502+0+0 (secure 0 0 0) 0x7f4d34044020 con 0x55dfa25bfe10 2024-09-16T15:47:09.804 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:09.796+0000 7f4d4c424000 10 client.6372 fetch_fsmap finished waiting for FSMap version 44 2024-09-16T15:47:09.804 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:09.796+0000 7f4d4c424000 10 client.6372 resolve_mds: resolved name 'd' to daemon mds.d 2024-09-16T15:47:09.804 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:09.796+0000 7f4d4c424000 20 client.6372 populate_metadata read hostname 'smithi123' 2024-09-16T15:47:09.805 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:09.796+0000 7f4d4c424000 1 --2- 172.21.15.123:0/2948776033 >> [v2:172.21.15.184:6834/3137795986,v1:172.21.15.184:6836/3137795986] conn(0x55dfa2642630 0x55dfa268f240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:47:09.805 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:09.796+0000 7f4d4c424000 4 client.6372 mds_command: new command op to 6216 tid=0 multi_id=0 [{"prefix": "status"}] 2024-09-16T15:47:09.805 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:09.796+0000 7f4d4c424000 1 -- 172.21.15.123:0/2948776033 --> [v2:172.21.15.184:6834/3137795986,v1:172.21.15.184:6836/3137795986] -- command(tid 0: {"prefix": "status"}) -- 0x55dfa241b520 con 0x55dfa2642630 2024-09-16T15:47:09.805 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:09.796+0000 7f4d3a7fc640 1 --2- 172.21.15.123:0/2948776033 >> [v2:172.21.15.184:6834/3137795986,v1:172.21.15.184:6836/3137795986] conn(0x55dfa2642630 0x55dfa268f240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:47:09.805 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:09.796+0000 7f4d21ffb640 10 client.6372 ms_handle_connect on v2:172.21.15.184:6834/3137795986 2024-09-16T15:47:09.805 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:09.796+0000 7f4d3a7fc640 1 --2- 172.21.15.123:0/2948776033 >> [v2:172.21.15.184:6834/3137795986,v1:172.21.15.184:6836/3137795986] conn(0x55dfa2642630 0x55dfa268f240 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2b5a45e8b9dede79 server_cookie=4ec4175d1ca4af46 in_seq=0 out_seq=0 2024-09-16T15:47:09.805 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:09.796+0000 7f4d21ffb640 1 -- 172.21.15.123:0/2948776033 <== mds.0 v2:172.21.15.184:6834/3137795986 1 ==== command_reply(tid 0: 0 ) ==== 8+0+325 (crc 0 0 0) 0x55dfa241b520 con 0x55dfa2642630 2024-09-16T15:47:09.805 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:09.796+0000 7f4d21ffb640 10 client.6372 handle_command_reply: tid=0 2024-09-16T15:47:09.805 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:09.796+0000 7f4d4c424000 1 client.6372 shutdown 2024-09-16T15:47:09.805 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:09.796+0000 7f4d4c424000 1 -- 172.21.15.123:0/2948776033 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55dfa25bfe10 msgr2=0x55dfa2685fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:47:09.805 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:09.796+0000 7f4d4c424000 1 --2- 172.21.15.123:0/2948776033 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55dfa25bfe10 0x55dfa2685fe0 secure :-1 s=READY pgs=811 cs=0 l=1 rev1=1 crypto rx=0x7f4d34005f10 tx=0x7f4d3400ea30 comp rx=0 tx=0).stop 2024-09-16T15:47:09.805 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:09.796+0000 7f4d4c424000 1 -- 172.21.15.123:0/2948776033 shutdown_connections 2024-09-16T15:47:09.805 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:09.796+0000 7f4d4c424000 1 --2- 172.21.15.123:0/2948776033 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x55dfa260c2f0 0x55dfa2686a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:09.805 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:09.796+0000 7f4d4c424000 1 --2- 172.21.15.123:0/2948776033 >> [v2:172.21.15.184:6834/3137795986,v1:172.21.15.184:6836/3137795986] conn(0x55dfa2642630 0x55dfa268f240 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:09.806 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:09.796+0000 7f4d4c424000 1 --2- 172.21.15.123:0/2948776033 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55dfa260b9d0 0x55dfa2686520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:09.806 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:09.796+0000 7f4d4c424000 1 --2- 172.21.15.123:0/2948776033 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55dfa25bfe10 0x55dfa2685fe0 unknown :-1 s=CLOSED pgs=811 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:09.806 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:09.796+0000 7f4d4c424000 1 -- 172.21.15.123:0/2948776033 >> 172.21.15.123:0/2948776033 conn(0x55dfa24a9130 msgr2=0x55dfa25c7b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:47:09.806 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:09.796+0000 7f4d21ffb640 0 client.6372 ms_handle_reset on v2:172.21.15.184:6834/3137795986 2024-09-16T15:47:09.806 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:09.796+0000 7f4d4c424000 1 -- 172.21.15.123:0/2948776033 shutdown_connections 2024-09-16T15:47:09.806 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:09.796+0000 7f4d4c424000 1 -- 172.21.15.123:0/2948776033 wait complete. 2024-09-16T15:47:09.806 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:09.796+0000 7f4d4c424000 20 client.6372 trim_cache size 0 max 16384 2024-09-16T15:47:09.806 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2024-09-16T15:47:09.806 INFO:teuthology.orchestra.run.smithi123.stdout: "cluster_fsid": "56077f89-18c8-486f-ae7e-eecc0bb7822c", 2024-09-16T15:47:09.806 INFO:teuthology.orchestra.run.smithi123.stdout: "whoami": 0, 2024-09-16T15:47:09.806 INFO:teuthology.orchestra.run.smithi123.stdout: "id": 6216, 2024-09-16T15:47:09.807 INFO:teuthology.orchestra.run.smithi123.stdout: "want_state": "up:active", 2024-09-16T15:47:09.807 INFO:teuthology.orchestra.run.smithi123.stdout: "state": "up:active", 2024-09-16T15:47:09.807 INFO:teuthology.orchestra.run.smithi123.stdout: "fs_name": "cephfs", 2024-09-16T15:47:09.807 INFO:teuthology.orchestra.run.smithi123.stdout: "rank_uptime": 10.215451795, 2024-09-16T15:47:09.807 INFO:teuthology.orchestra.run.smithi123.stdout: "mdsmap_epoch": 44, 2024-09-16T15:47:09.807 INFO:teuthology.orchestra.run.smithi123.stdout: "osdmap_epoch": 70, 2024-09-16T15:47:09.807 INFO:teuthology.orchestra.run.smithi123.stdout: "osdmap_epoch_barrier": 70, 2024-09-16T15:47:09.807 INFO:teuthology.orchestra.run.smithi123.stdout: "uptime": 15.850601301999999 2024-09-16T15:47:09.807 INFO:teuthology.orchestra.run.smithi123.stdout:} 2024-09-16T15:47:09.808 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:09.796+0000 7f4d4933a640 1 -- 172.21.15.123:0/1878007117 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f4d24073250 msgr2=0x7f4d24075710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:47:09.808 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:09.796+0000 7f4d4933a640 1 --2- 172.21.15.123:0/1878007117 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f4d24073250 0x7f4d24075710 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f4d340058d0 tx=0x7f4d3400b370 comp rx=0 tx=0).stop 2024-09-16T15:47:09.808 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:09.796+0000 7f4d4933a640 1 -- 172.21.15.123:0/1878007117 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4d4415ac00 msgr2=0x7f4d440ca1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:47:09.808 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:09.796+0000 7f4d4933a640 1 --2- 172.21.15.123:0/1878007117 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4d4415ac00 0x7f4d440ca1f0 secure :-1 s=READY pgs=807 cs=0 l=1 rev1=1 crypto rx=0x7f4d3c009b10 tx=0x7f4d3c0a5940 comp rx=0 tx=0).stop 2024-09-16T15:47:09.808 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:09.796+0000 7f4d4933a640 1 -- 172.21.15.123:0/1878007117 shutdown_connections 2024-09-16T15:47:09.808 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:09.796+0000 7f4d4933a640 1 --2- 172.21.15.123:0/1878007117 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f4d24073250 0x7f4d24075710 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:09.808 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:09.796+0000 7f4d4933a640 1 --2- 172.21.15.123:0/1878007117 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4d4415ac00 0x7f4d440ca1f0 unknown :-1 s=CLOSED pgs=807 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:09.808 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:09.796+0000 7f4d4933a640 1 --2- 172.21.15.123:0/1878007117 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f4d440cc8c0 0x7f4d44156610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:09.809 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:09.796+0000 7f4d4933a640 1 --2- 172.21.15.123:0/1878007117 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4d440b3100 0x7f4d44152e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:09.809 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:09.796+0000 7f4d4933a640 1 -- 172.21.15.123:0/1878007117 >> 172.21.15.123:0/1878007117 conn(0x7f4d440b2610 msgr2=0x7f4d440cb960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:47:09.809 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:09.796+0000 7f4d4933a640 1 -- 172.21.15.123:0/1878007117 shutdown_connections 2024-09-16T15:47:09.809 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:09.796+0000 7f4d4933a640 1 -- 172.21.15.123:0/1878007117 wait complete. 2024-09-16T15:47:09.817 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-09-16T15:47:09.939 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:09.936+0000 7f6d088ed640 1 -- 172.21.15.123:0/2725760404 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6d04074600 msgr2=0x7f6d0406e750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:47:09.939 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:09.936+0000 7f6d088ed640 1 --2- 172.21.15.123:0/2725760404 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6d04074600 0x7f6d0406e750 secure :-1 s=READY pgs=813 cs=0 l=1 rev1=1 crypto rx=0x7f6d040cd650 tx=0x7f6cfc01c800 comp rx=0 tx=0).stop 2024-09-16T15:47:09.939 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:09.936+0000 7f6d088ed640 1 -- 172.21.15.123:0/2725760404 shutdown_connections 2024-09-16T15:47:09.939 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:09.936+0000 7f6d088ed640 1 --2- 172.21.15.123:0/2725760404 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6d04074600 0x7f6d0406e750 unknown :-1 s=CLOSED pgs=813 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:09.939 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:09.936+0000 7f6d088ed640 1 --2- 172.21.15.123:0/2725760404 >> v2:172.21.15.184:3300/0 conn(0x7f6d0415bca0 0x7f6d040d0630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:09.939 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:09.936+0000 7f6d088ed640 1 -- 172.21.15.123:0/2725760404 >> 172.21.15.123:0/2725760404 conn(0x7f6d040cfb10 msgr2=0x7f6d040cff20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:47:09.940 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:09.936+0000 7f6d088ed640 1 -- 172.21.15.123:0/2725760404 shutdown_connections 2024-09-16T15:47:09.940 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:09.936+0000 7f6d088ed640 1 -- 172.21.15.123:0/2725760404 wait complete. 2024-09-16T15:47:09.940 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:09.936+0000 7f6d088ed640 1 Processor -- start 2024-09-16T15:47:09.940 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:09.936+0000 7f6d088ed640 1 -- start start 2024-09-16T15:47:09.942 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:09.936+0000 7f6d088ed640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6d040726c0 0x7f6d041530f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:47:09.942 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:09.936+0000 7f6d088ed640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6d04074600 0x7f6d04159bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:47:09.942 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:09.936+0000 7f6d088ed640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f6d0415bca0 0x7f6d040c4810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:47:09.943 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:09.936+0000 7f6d088ed640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f6d04168a50 con 0x7f6d040726c0 2024-09-16T15:47:09.943 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:09.936+0000 7f6d088ed640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f6d0406f9e0 con 0x7f6d04074600 2024-09-16T15:47:09.943 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:09.936+0000 7f6d088ed640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f6d04168bd0 con 0x7f6d0415bca0 2024-09-16T15:47:09.943 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:09.936+0000 7f6d0259c640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6d040726c0 0x7f6d041530f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:47:09.943 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:09.936+0000 7f6d0259c640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6d040726c0 0x7f6d041530f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:46994/0 (socket says 172.21.15.123:46994) 2024-09-16T15:47:09.943 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:09.936+0000 7f6d0259c640 1 -- 172.21.15.123:0/3677371242 learned_addr learned my addr 172.21.15.123:0/3677371242 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:47:09.943 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:09.936+0000 7f6d0259c640 1 -- 172.21.15.123:0/3677371242 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f6d0415bca0 msgr2=0x7f6d040c4810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:47:09.943 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:09.936+0000 7f6d0259c640 1 --2- 172.21.15.123:0/3677371242 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f6d0415bca0 0x7f6d040c4810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:09.943 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:09.936+0000 7f6d0259c640 1 -- 172.21.15.123:0/3677371242 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6d04074600 msgr2=0x7f6d04159bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:47:09.943 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:09.936+0000 7f6d0259c640 1 --2- 172.21.15.123:0/3677371242 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6d04074600 0x7f6d04159bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:09.943 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:09.936+0000 7f6d0259c640 1 -- 172.21.15.123:0/3677371242 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6d0415a590 con 0x7f6d040726c0 2024-09-16T15:47:09.943 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:09.936+0000 7f6d0259c640 1 --2- 172.21.15.123:0/3677371242 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6d040726c0 0x7f6d041530f0 secure :-1 s=READY pgs=814 cs=0 l=1 rev1=1 crypto rx=0x7f6cf400ccb0 tx=0x7f6cf400a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:47:09.943 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:09.936+0000 7f6cf2ffd640 1 -- 172.21.15.123:0/3677371242 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6cf400f450 con 0x7f6d040726c0 2024-09-16T15:47:09.944 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:09.936+0000 7f6cf2ffd640 1 -- 172.21.15.123:0/3677371242 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f6cf400f6e0 con 0x7f6d040726c0 2024-09-16T15:47:09.944 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:09.936+0000 7f6d088ed640 1 -- 172.21.15.123:0/3677371242 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6d0415a870 con 0x7f6d040726c0 2024-09-16T15:47:09.944 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:09.936+0000 7f6cf2ffd640 1 -- 172.21.15.123:0/3677371242 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6cf400faf0 con 0x7f6d040726c0 2024-09-16T15:47:09.944 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:09.936+0000 7f6d088ed640 1 -- 172.21.15.123:0/3677371242 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6d040cd780 con 0x7f6d040726c0 2024-09-16T15:47:09.944 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:09.940+0000 7f6d088ed640 1 -- 172.21.15.123:0/3677371242 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6d0406e750 con 0x7f6d040726c0 2024-09-16T15:47:09.945 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:09.940+0000 7f6cf2ffd640 1 -- 172.21.15.123:0/3677371242 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 11) ==== 97386+0+0 (secure 0 0 0) 0x7f6cf400fc90 con 0x7f6d040726c0 2024-09-16T15:47:09.945 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:09.940+0000 7f6cf2ffd640 1 --2- 172.21.15.123:0/3677371242 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f6ce4073220 0x7f6ce40756e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:47:09.945 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:09.940+0000 7f6cf2ffd640 1 -- 172.21.15.123:0/3677371242 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(70..70 src has 1..70) ==== 5709+0+0 (secure 0 0 0) 0x7f6cf4094fd0 con 0x7f6d040726c0 2024-09-16T15:47:09.945 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:09.940+0000 7f6d01d9b640 1 --2- 172.21.15.123:0/3677371242 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f6ce4073220 0x7f6ce40756e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:47:09.945 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:09.940+0000 7f6d01d9b640 1 --2- 172.21.15.123:0/3677371242 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f6ce4073220 0x7f6ce40756e0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f6cf8007fd0 tx=0x7f6cf80078c0 comp rx=0 tx=0).ready entity=mgr.4135 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:47:09.948 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:09.944+0000 7f6cf2ffd640 1 -- 172.21.15.123:0/3677371242 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f6cf4014030 con 0x7f6d040726c0 2024-09-16T15:47:10.158 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:10.152+0000 7f6d088ed640 1 -- 172.21.15.123:0/3677371242 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f6d0415f630 con 0x7f6d040726c0 2024-09-16T15:47:10.159 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:10.156+0000 7f6cf2ffd640 1 -- 172.21.15.123:0/3677371242 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v44) ==== 53+0+73 (secure 0 0 0) 0x7f6cf405fc90 con 0x7f6d040726c0 2024-09-16T15:47:10.159 INFO:teuthology.orchestra.run.smithi123.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-09-16T15:47:10.161 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:10.156+0000 7f6d088ed640 1 -- 172.21.15.123:0/3677371242 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f6ce4073220 msgr2=0x7f6ce40756e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:47:10.161 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:10.156+0000 7f6d088ed640 1 --2- 172.21.15.123:0/3677371242 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f6ce4073220 0x7f6ce40756e0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f6cf8007fd0 tx=0x7f6cf80078c0 comp rx=0 tx=0).stop 2024-09-16T15:47:10.161 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:10.156+0000 7f6d088ed640 1 -- 172.21.15.123:0/3677371242 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6d040726c0 msgr2=0x7f6d041530f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:47:10.161 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:10.156+0000 7f6d088ed640 1 --2- 172.21.15.123:0/3677371242 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6d040726c0 0x7f6d041530f0 secure :-1 s=READY pgs=814 cs=0 l=1 rev1=1 crypto rx=0x7f6cf400ccb0 tx=0x7f6cf400a620 comp rx=0 tx=0).stop 2024-09-16T15:47:10.161 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:10.156+0000 7f6d088ed640 1 -- 172.21.15.123:0/3677371242 shutdown_connections 2024-09-16T15:47:10.161 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:10.156+0000 7f6d088ed640 1 --2- 172.21.15.123:0/3677371242 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f6ce4073220 0x7f6ce40756e0 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:10.162 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:10.156+0000 7f6d088ed640 1 --2- 172.21.15.123:0/3677371242 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f6d0415bca0 0x7f6d040c4810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:10.162 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:10.156+0000 7f6d088ed640 1 --2- 172.21.15.123:0/3677371242 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6d04074600 0x7f6d04159bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:10.162 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:10.156+0000 7f6d088ed640 1 --2- 172.21.15.123:0/3677371242 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6d040726c0 0x7f6d041530f0 unknown :-1 s=CLOSED pgs=814 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:10.162 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:10.156+0000 7f6d088ed640 1 -- 172.21.15.123:0/3677371242 >> 172.21.15.123:0/3677371242 conn(0x7f6d040cfb10 msgr2=0x7f6d040b43b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:47:10.162 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:10.156+0000 7f6d088ed640 1 -- 172.21.15.123:0/3677371242 shutdown_connections 2024-09-16T15:47:10.162 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:10.156+0000 7f6d088ed640 1 -- 172.21.15.123:0/3677371242 wait complete. 2024-09-16T15:47:10.176 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-09-16T15:47:10.176 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-09-16T15:47:10.176 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi123.front.sepia.ceph.com 2024-09-16T15:47:10.177 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-09-16T15:47:10.177 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-09-16T15:47:10.177 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-09-16T15:47:10.177 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-09-16T15:47:10.177 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-09-16T15:47:10.177 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-09-16T15:47:10.177 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:47:10.177 DEBUG:teuthology.orchestra.run.smithi123:> ip addr 2024-09-16T15:47:10.183 INFO:teuthology.orchestra.run.smithi123.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-09-16T15:47:10.183 INFO:teuthology.orchestra.run.smithi123.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-09-16T15:47:10.183 INFO:teuthology.orchestra.run.smithi123.stdout: inet 127.0.0.1/8 scope host lo 2024-09-16T15:47:10.183 INFO:teuthology.orchestra.run.smithi123.stdout: valid_lft forever preferred_lft forever 2024-09-16T15:47:10.183 INFO:teuthology.orchestra.run.smithi123.stdout: inet6 ::1/128 scope host 2024-09-16T15:47:10.183 INFO:teuthology.orchestra.run.smithi123.stdout: valid_lft forever preferred_lft forever 2024-09-16T15:47:10.183 INFO:teuthology.orchestra.run.smithi123.stdout:2: eno1: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-09-16T15:47:10.184 INFO:teuthology.orchestra.run.smithi123.stdout: link/ether 0c:c4:7a:d8:da:6c brd ff:ff:ff:ff:ff:ff 2024-09-16T15:47:10.184 INFO:teuthology.orchestra.run.smithi123.stdout: altname enp7s0f0 2024-09-16T15:47:10.184 INFO:teuthology.orchestra.run.smithi123.stdout:3: eno2: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-09-16T15:47:10.184 INFO:teuthology.orchestra.run.smithi123.stdout: link/ether 0c:c4:7a:d8:da:6d brd ff:ff:ff:ff:ff:ff 2024-09-16T15:47:10.184 INFO:teuthology.orchestra.run.smithi123.stdout: altname enp7s0f1 2024-09-16T15:47:10.184 INFO:teuthology.orchestra.run.smithi123.stdout:4: enp3s0f0: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-09-16T15:47:10.184 INFO:teuthology.orchestra.run.smithi123.stdout: link/ether 0c:c4:7a:8f:d3:f0 brd ff:ff:ff:ff:ff:ff 2024-09-16T15:47:10.184 INFO:teuthology.orchestra.run.smithi123.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-09-16T15:47:10.184 INFO:teuthology.orchestra.run.smithi123.stdout: link/ether 0c:c4:7a:8f:d3:f1 brd ff:ff:ff:ff:ff:ff 2024-09-16T15:47:10.184 INFO:teuthology.orchestra.run.smithi123.stdout: inet 172.21.15.123/20 metric 100 brd 172.21.15.255 scope global dynamic enp3s0f1 2024-09-16T15:47:10.184 INFO:teuthology.orchestra.run.smithi123.stdout: valid_lft 42238sec preferred_lft 42238sec 2024-09-16T15:47:10.184 INFO:teuthology.orchestra.run.smithi123.stdout: inet6 fe80::ec4:7aff:fe8f:d3f1/64 scope link 2024-09-16T15:47:10.184 INFO:teuthology.orchestra.run.smithi123.stdout: valid_lft forever preferred_lft forever 2024-09-16T15:47:10.184 INFO:teuthology.orchestra.run.smithi123.stdout:6: docker0: mtu 1500 qdisc noqueue state DOWN group default 2024-09-16T15:47:10.185 INFO:teuthology.orchestra.run.smithi123.stdout: link/ether 02:42:dd:82:a2:6e brd ff:ff:ff:ff:ff:ff 2024-09-16T15:47:10.185 INFO:teuthology.orchestra.run.smithi123.stdout: inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0 2024-09-16T15:47:10.185 INFO:teuthology.orchestra.run.smithi123.stdout: valid_lft forever preferred_lft forever 2024-09-16T15:47:10.185 INFO:teuthology.orchestra.run.smithi123.stdout:7: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-09-16T15:47:10.185 INFO:teuthology.orchestra.run.smithi123.stdout: link/ether 5e:1f:84:6d:79:ae brd ff:ff:ff:ff:ff:ff 2024-09-16T15:47:10.185 INFO:teuthology.orchestra.run.smithi123.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-09-16T15:47:10.185 INFO:teuthology.orchestra.run.smithi123.stdout: valid_lft forever preferred_lft forever 2024-09-16T15:47:10.185 INFO:teuthology.orchestra.run.smithi123.stdout: inet6 fe80::5c1f:84ff:fe6d:79ae/64 scope link 2024-09-16T15:47:10.185 INFO:teuthology.orchestra.run.smithi123.stdout: valid_lft forever preferred_lft forever 2024-09-16T15:47:10.185 INFO:teuthology.orchestra.run.smithi123.stdout:8: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-09-16T15:47:10.185 INFO:teuthology.orchestra.run.smithi123.stdout: link/ether 0e:5c:f6:1f:82:2f brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-09-16T15:47:10.185 INFO:teuthology.orchestra.run.smithi123.stdout: inet6 fe80::c5c:f6ff:fe1f:822f/64 scope link 2024-09-16T15:47:10.185 INFO:teuthology.orchestra.run.smithi123.stdout: valid_lft forever preferred_lft forever 2024-09-16T15:47:10.186 INFO:teuthology.orchestra.run.smithi123.stdout:9: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-09-16T15:47:10.186 INFO:teuthology.orchestra.run.smithi123.stdout: link/ether c6:a6:3d:69:d0:db brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-09-16T15:47:10.186 INFO:teuthology.orchestra.run.smithi123.stdout: inet6 fe80::c4a6:3dff:fe69:d0db/64 scope link 2024-09-16T15:47:10.186 INFO:teuthology.orchestra.run.smithi123.stdout: valid_lft forever preferred_lft forever 2024-09-16T15:47:10.186 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:47:10.186 DEBUG:teuthology.orchestra.run.smithi123:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-09-16T15:47:10.235 INFO:teuthology.orchestra.run.smithi123.stdout:1 2024-09-16T15:47:10.237 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:47:10.237 DEBUG:teuthology.orchestra.run.smithi123:> route 2024-09-16T15:47:10.287 INFO:teuthology.orchestra.run.smithi123.stdout:Kernel IP routing table 2024-09-16T15:47:10.287 INFO:teuthology.orchestra.run.smithi123.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-09-16T15:47:10.287 INFO:teuthology.orchestra.run.smithi123.stdout:default _gateway 0.0.0.0 UG 0 0 0 enp3s0f1 2024-09-16T15:47:10.287 INFO:teuthology.orchestra.run.smithi123.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-09-16T15:47:10.287 INFO:teuthology.orchestra.run.smithi123.stdout:158.69.67.47 _gateway 255.255.255.255 UGH 100 0 0 enp3s0f1 2024-09-16T15:47:10.287 INFO:teuthology.orchestra.run.smithi123.stdout:172.17.0.0 0.0.0.0 255.255.0.0 U 0 0 0 docker0 2024-09-16T15:47:10.287 INFO:teuthology.orchestra.run.smithi123.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-09-16T15:47:10.287 INFO:teuthology.orchestra.run.smithi123.stdout:ns1.front.sepia 0.0.0.0 255.255.255.255 UH 100 0 0 enp3s0f1 2024-09-16T15:47:10.287 INFO:teuthology.orchestra.run.smithi123.stdout:172.21.0.2 0.0.0.0 255.255.255.255 UH 100 0 0 enp3s0f1 2024-09-16T15:47:10.287 INFO:teuthology.orchestra.run.smithi123.stdout:_gateway 0.0.0.0 255.255.255.255 UH 100 0 0 enp3s0f1 2024-09-16T15:47:10.287 INFO:teuthology.orchestra.run.smithi123.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-09-16T15:47:10.288 DEBUG:teuthology.orchestra.run.smithi123:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-09-16T15:47:10.288 DEBUG:teuthology.orchestra.run.smithi123:> set -e 2024-09-16T15:47:10.288 DEBUG:teuthology.orchestra.run.smithi123:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-09-16T15:47:10.288 DEBUG:teuthology.orchestra.run.smithi123:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-09-16T15:47:10.288 DEBUG:teuthology.orchestra.run.smithi123:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-09-16T15:47:10.288 DEBUG:teuthology.orchestra.run.smithi123:> ') 2024-09-16T15:47:10.374 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:10 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-09-16T15:47:10.403 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:10 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-16T15:47:10.409 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:47:10.409 DEBUG:teuthology.orchestra.run.smithi123:> ip netns list 2024-09-16T15:47:10.456 INFO:teuthology.orchestra.run.smithi123.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-09-16T15:47:10.456 INFO:teuthology.orchestra.run.smithi123.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-09-16T15:47:10.456 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-09-16T15:47:10.456 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:47:10.456 DEBUG:teuthology.orchestra.run.smithi123:> sudo ip link set brx.0 up 2024-09-16T15:47:10.511 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2024-09-16T15:47:10.511 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-09-16T15:47:10.511 DEBUG:teuthology.orchestra.run.smithi123:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-09-16T15:47:10.560 INFO:teuthology.orchestra.run.smithi123.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-09-16T15:47:10.560 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-09-16T15:47:10.560 DEBUG:teuthology.orchestra.run.smithi123:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-09-16T15:47:10.607 DEBUG:teuthology.orchestra.run.smithi123:> sudo modprobe fuse 2024-09-16T15:47:10.664 DEBUG:teuthology.orchestra.run.smithi123:> cat /proc/self/mounts | awk '{print $2}' 2024-09-16T15:47:10.713 INFO:teuthology.orchestra.run.smithi123.stdout:/sys 2024-09-16T15:47:10.713 INFO:teuthology.orchestra.run.smithi123.stdout:/proc 2024-09-16T15:47:10.713 INFO:teuthology.orchestra.run.smithi123.stdout:/dev 2024-09-16T15:47:10.713 INFO:teuthology.orchestra.run.smithi123.stdout:/dev/pts 2024-09-16T15:47:10.713 INFO:teuthology.orchestra.run.smithi123.stdout:/run 2024-09-16T15:47:10.713 INFO:teuthology.orchestra.run.smithi123.stdout:/ 2024-09-16T15:47:10.713 INFO:teuthology.orchestra.run.smithi123.stdout:/sys/kernel/security 2024-09-16T15:47:10.713 INFO:teuthology.orchestra.run.smithi123.stdout:/dev/shm 2024-09-16T15:47:10.713 INFO:teuthology.orchestra.run.smithi123.stdout:/run/lock 2024-09-16T15:47:10.713 INFO:teuthology.orchestra.run.smithi123.stdout:/sys/fs/cgroup 2024-09-16T15:47:10.713 INFO:teuthology.orchestra.run.smithi123.stdout:/sys/fs/pstore 2024-09-16T15:47:10.713 INFO:teuthology.orchestra.run.smithi123.stdout:/sys/fs/bpf 2024-09-16T15:47:10.713 INFO:teuthology.orchestra.run.smithi123.stdout:/proc/sys/fs/binfmt_misc 2024-09-16T15:47:10.714 INFO:teuthology.orchestra.run.smithi123.stdout:/dev/hugepages 2024-09-16T15:47:10.714 INFO:teuthology.orchestra.run.smithi123.stdout:/dev/mqueue 2024-09-16T15:47:10.714 INFO:teuthology.orchestra.run.smithi123.stdout:/sys/kernel/debug 2024-09-16T15:47:10.714 INFO:teuthology.orchestra.run.smithi123.stdout:/sys/kernel/tracing 2024-09-16T15:47:10.714 INFO:teuthology.orchestra.run.smithi123.stdout:/proc/fs/nfsd 2024-09-16T15:47:10.714 INFO:teuthology.orchestra.run.smithi123.stdout:/sys/fs/fuse/connections 2024-09-16T15:47:10.714 INFO:teuthology.orchestra.run.smithi123.stdout:/sys/kernel/config 2024-09-16T15:47:10.714 INFO:teuthology.orchestra.run.smithi123.stdout:/run/credentials/systemd-sysusers.service 2024-09-16T15:47:10.714 INFO:teuthology.orchestra.run.smithi123.stdout:/run/rpc_pipefs 2024-09-16T15:47:10.714 INFO:teuthology.orchestra.run.smithi123.stdout:/run/user/1000 2024-09-16T15:47:10.714 INFO:teuthology.orchestra.run.smithi123.stdout:/var/lib/ceph 2024-09-16T15:47:10.714 INFO:teuthology.orchestra.run.smithi123.stdout:/var/lib/ceph/osd/ceph-0 2024-09-16T15:47:10.714 INFO:teuthology.orchestra.run.smithi123.stdout:/var/lib/ceph/osd/ceph-1 2024-09-16T15:47:10.714 INFO:teuthology.orchestra.run.smithi123.stdout:/var/lib/ceph/osd/ceph-2 2024-09-16T15:47:10.714 INFO:teuthology.orchestra.run.smithi123.stdout:/var/lib/ceph/osd/ceph-3 2024-09-16T15:47:10.715 INFO:teuthology.orchestra.run.smithi123.stdout:/run/netns 2024-09-16T15:47:10.715 INFO:teuthology.orchestra.run.smithi123.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-09-16T15:47:10.715 INFO:teuthology.orchestra.run.smithi123.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-09-16T15:47:10.715 INFO:teuthology.orchestra.run.smithi123.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-09-16T15:47:10.715 INFO:teuthology.orchestra.run.smithi123.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-09-16T15:47:10.716 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:47:10.716 DEBUG:teuthology.orchestra.run.smithi123:> ls /sys/fs/fuse/connections 2024-09-16T15:47:10.763 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-09-16T15:47:10.763 DEBUG:teuthology.orchestra.run.smithi123:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2024-09-16T15:47:10.808 DEBUG:teuthology.orchestra.run.smithi123:> sudo modprobe fuse 2024-09-16T15:47:10.824 DEBUG:teuthology.orchestra.run.smithi123:> cat /proc/self/mounts | awk '{print $2}' 2024-09-16T15:47:10.872 INFO:teuthology.orchestra.run.smithi123.stdout:/sys 2024-09-16T15:47:10.872 INFO:teuthology.orchestra.run.smithi123.stdout:/proc 2024-09-16T15:47:10.872 INFO:teuthology.orchestra.run.smithi123.stdout:/dev 2024-09-16T15:47:10.872 INFO:teuthology.orchestra.run.smithi123.stdout:/dev/pts 2024-09-16T15:47:10.872 INFO:teuthology.orchestra.run.smithi123.stdout:/run 2024-09-16T15:47:10.873 INFO:teuthology.orchestra.run.smithi123.stdout:/ 2024-09-16T15:47:10.873 INFO:teuthology.orchestra.run.smithi123.stdout:/sys/kernel/security 2024-09-16T15:47:10.873 INFO:teuthology.orchestra.run.smithi123.stdout:/dev/shm 2024-09-16T15:47:10.873 INFO:teuthology.orchestra.run.smithi123.stdout:/run/lock 2024-09-16T15:47:10.873 INFO:teuthology.orchestra.run.smithi123.stdout:/sys/fs/cgroup 2024-09-16T15:47:10.873 INFO:teuthology.orchestra.run.smithi123.stdout:/sys/fs/pstore 2024-09-16T15:47:10.873 INFO:teuthology.orchestra.run.smithi123.stdout:/sys/fs/bpf 2024-09-16T15:47:10.873 INFO:teuthology.orchestra.run.smithi123.stdout:/proc/sys/fs/binfmt_misc 2024-09-16T15:47:10.873 INFO:teuthology.orchestra.run.smithi123.stdout:/dev/hugepages 2024-09-16T15:47:10.873 INFO:teuthology.orchestra.run.smithi123.stdout:/dev/mqueue 2024-09-16T15:47:10.873 INFO:teuthology.orchestra.run.smithi123.stdout:/sys/kernel/debug 2024-09-16T15:47:10.873 INFO:teuthology.orchestra.run.smithi123.stdout:/sys/kernel/tracing 2024-09-16T15:47:10.873 INFO:teuthology.orchestra.run.smithi123.stdout:/proc/fs/nfsd 2024-09-16T15:47:10.873 INFO:teuthology.orchestra.run.smithi123.stdout:/sys/fs/fuse/connections 2024-09-16T15:47:10.874 INFO:teuthology.orchestra.run.smithi123.stdout:/sys/kernel/config 2024-09-16T15:47:10.874 INFO:teuthology.orchestra.run.smithi123.stdout:/run/credentials/systemd-sysusers.service 2024-09-16T15:47:10.874 INFO:teuthology.orchestra.run.smithi123.stdout:/run/rpc_pipefs 2024-09-16T15:47:10.874 INFO:teuthology.orchestra.run.smithi123.stdout:/run/user/1000 2024-09-16T15:47:10.874 INFO:teuthology.orchestra.run.smithi123.stdout:/var/lib/ceph 2024-09-16T15:47:10.874 INFO:teuthology.orchestra.run.smithi123.stdout:/var/lib/ceph/osd/ceph-0 2024-09-16T15:47:10.874 INFO:teuthology.orchestra.run.smithi123.stdout:/var/lib/ceph/osd/ceph-1 2024-09-16T15:47:10.874 INFO:teuthology.orchestra.run.smithi123.stdout:/var/lib/ceph/osd/ceph-2 2024-09-16T15:47:10.874 INFO:teuthology.orchestra.run.smithi123.stdout:/var/lib/ceph/osd/ceph-3 2024-09-16T15:47:10.874 INFO:teuthology.orchestra.run.smithi123.stdout:/run/netns 2024-09-16T15:47:10.874 INFO:teuthology.orchestra.run.smithi123.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-09-16T15:47:10.874 INFO:teuthology.orchestra.run.smithi123.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-09-16T15:47:10.874 INFO:teuthology.orchestra.run.smithi123.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-09-16T15:47:10.874 INFO:teuthology.orchestra.run.smithi123.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-09-16T15:47:10.875 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:47:10.875 DEBUG:teuthology.orchestra.run.smithi123:> ls /sys/fs/fuse/connections 2024-09-16T15:47:10.888 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi123.stderr:2024-09-16T15:47:10.884+0000 7f9678215580 -1 init, newargv = 0x560d28f60a50 newargc=15 2024-09-16T15:47:10.888 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi123.stderr:ceph-fuse[42891]: starting ceph client 2024-09-16T15:47:10.894 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi123.stderr:ceph-fuse[42891]: starting fuse 2024-09-16T15:47:11.993 DEBUG:teuthology.orchestra.run.smithi123:> sudo modprobe fuse 2024-09-16T15:47:12.009 DEBUG:teuthology.orchestra.run.smithi123:> cat /proc/self/mounts | awk '{print $2}' 2024-09-16T15:47:12.057 INFO:teuthology.orchestra.run.smithi123.stdout:/sys 2024-09-16T15:47:12.057 INFO:teuthology.orchestra.run.smithi123.stdout:/proc 2024-09-16T15:47:12.057 INFO:teuthology.orchestra.run.smithi123.stdout:/dev 2024-09-16T15:47:12.057 INFO:teuthology.orchestra.run.smithi123.stdout:/dev/pts 2024-09-16T15:47:12.057 INFO:teuthology.orchestra.run.smithi123.stdout:/run 2024-09-16T15:47:12.057 INFO:teuthology.orchestra.run.smithi123.stdout:/ 2024-09-16T15:47:12.057 INFO:teuthology.orchestra.run.smithi123.stdout:/sys/kernel/security 2024-09-16T15:47:12.057 INFO:teuthology.orchestra.run.smithi123.stdout:/dev/shm 2024-09-16T15:47:12.057 INFO:teuthology.orchestra.run.smithi123.stdout:/run/lock 2024-09-16T15:47:12.057 INFO:teuthology.orchestra.run.smithi123.stdout:/sys/fs/cgroup 2024-09-16T15:47:12.058 INFO:teuthology.orchestra.run.smithi123.stdout:/sys/fs/pstore 2024-09-16T15:47:12.058 INFO:teuthology.orchestra.run.smithi123.stdout:/sys/fs/bpf 2024-09-16T15:47:12.058 INFO:teuthology.orchestra.run.smithi123.stdout:/proc/sys/fs/binfmt_misc 2024-09-16T15:47:12.058 INFO:teuthology.orchestra.run.smithi123.stdout:/dev/hugepages 2024-09-16T15:47:12.058 INFO:teuthology.orchestra.run.smithi123.stdout:/dev/mqueue 2024-09-16T15:47:12.058 INFO:teuthology.orchestra.run.smithi123.stdout:/sys/kernel/debug 2024-09-16T15:47:12.058 INFO:teuthology.orchestra.run.smithi123.stdout:/sys/kernel/tracing 2024-09-16T15:47:12.058 INFO:teuthology.orchestra.run.smithi123.stdout:/proc/fs/nfsd 2024-09-16T15:47:12.058 INFO:teuthology.orchestra.run.smithi123.stdout:/sys/fs/fuse/connections 2024-09-16T15:47:12.058 INFO:teuthology.orchestra.run.smithi123.stdout:/sys/kernel/config 2024-09-16T15:47:12.058 INFO:teuthology.orchestra.run.smithi123.stdout:/run/credentials/systemd-sysusers.service 2024-09-16T15:47:12.058 INFO:teuthology.orchestra.run.smithi123.stdout:/run/rpc_pipefs 2024-09-16T15:47:12.058 INFO:teuthology.orchestra.run.smithi123.stdout:/run/user/1000 2024-09-16T15:47:12.058 INFO:teuthology.orchestra.run.smithi123.stdout:/var/lib/ceph 2024-09-16T15:47:12.058 INFO:teuthology.orchestra.run.smithi123.stdout:/var/lib/ceph/osd/ceph-0 2024-09-16T15:47:12.059 INFO:teuthology.orchestra.run.smithi123.stdout:/var/lib/ceph/osd/ceph-1 2024-09-16T15:47:12.059 INFO:teuthology.orchestra.run.smithi123.stdout:/var/lib/ceph/osd/ceph-2 2024-09-16T15:47:12.059 INFO:teuthology.orchestra.run.smithi123.stdout:/var/lib/ceph/osd/ceph-3 2024-09-16T15:47:12.059 INFO:teuthology.orchestra.run.smithi123.stdout:/run/netns 2024-09-16T15:47:12.059 INFO:teuthology.orchestra.run.smithi123.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-09-16T15:47:12.059 INFO:teuthology.orchestra.run.smithi123.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-09-16T15:47:12.059 INFO:teuthology.orchestra.run.smithi123.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-09-16T15:47:12.059 INFO:teuthology.orchestra.run.smithi123.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-09-16T15:47:12.059 INFO:teuthology.orchestra.run.smithi123.stdout:/home/ubuntu/cephtest/mnt.0 2024-09-16T15:47:12.060 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:47:12.060 DEBUG:teuthology.orchestra.run.smithi123:> ls /sys/fs/fuse/connections 2024-09-16T15:47:12.107 INFO:teuthology.orchestra.run.smithi123.stdout:45 2024-09-16T15:47:12.107 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [45] 2024-09-16T15:47:12.108 DEBUG:teuthology.orchestra.run.smithi123:> sudo stdin-killer -- python3 -c ' 2024-09-16T15:47:12.108 DEBUG:teuthology.orchestra.run.smithi123:> import glob 2024-09-16T15:47:12.108 DEBUG:teuthology.orchestra.run.smithi123:> import re 2024-09-16T15:47:12.108 DEBUG:teuthology.orchestra.run.smithi123:> import os 2024-09-16T15:47:12.108 DEBUG:teuthology.orchestra.run.smithi123:> import subprocess 2024-09-16T15:47:12.108 DEBUG:teuthology.orchestra.run.smithi123:> 2024-09-16T15:47:12.108 DEBUG:teuthology.orchestra.run.smithi123:> def _find_admin_socket(client_name): 2024-09-16T15:47:12.108 DEBUG:teuthology.orchestra.run.smithi123:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2024-09-16T15:47:12.108 DEBUG:teuthology.orchestra.run.smithi123:> files = glob.glob(asok_path) 2024-09-16T15:47:12.108 DEBUG:teuthology.orchestra.run.smithi123:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-09-16T15:47:12.108 DEBUG:teuthology.orchestra.run.smithi123:> 2024-09-16T15:47:12.108 DEBUG:teuthology.orchestra.run.smithi123:> # Given a non-glob path, it better be there 2024-09-16T15:47:12.109 DEBUG:teuthology.orchestra.run.smithi123:> if "*" not in asok_path: 2024-09-16T15:47:12.109 DEBUG:teuthology.orchestra.run.smithi123:> assert(len(files) == 1) 2024-09-16T15:47:12.109 DEBUG:teuthology.orchestra.run.smithi123:> return files[0] 2024-09-16T15:47:12.109 DEBUG:teuthology.orchestra.run.smithi123:> 2024-09-16T15:47:12.109 DEBUG:teuthology.orchestra.run.smithi123:> for f in files: 2024-09-16T15:47:12.109 DEBUG:teuthology.orchestra.run.smithi123:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-09-16T15:47:12.109 DEBUG:teuthology.orchestra.run.smithi123:> if os.path.exists("/proc/{0}".format(pid)): 2024-09-16T15:47:12.109 DEBUG:teuthology.orchestra.run.smithi123:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-09-16T15:47:12.109 DEBUG:teuthology.orchestra.run.smithi123:> contents = proc_f.read() 2024-09-16T15:47:12.109 DEBUG:teuthology.orchestra.run.smithi123:> if mountpoint in contents: 2024-09-16T15:47:12.109 DEBUG:teuthology.orchestra.run.smithi123:> return f 2024-09-16T15:47:12.109 DEBUG:teuthology.orchestra.run.smithi123:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-09-16T15:47:12.109 DEBUG:teuthology.orchestra.run.smithi123:> 2024-09-16T15:47:12.109 DEBUG:teuthology.orchestra.run.smithi123:> print(_find_admin_socket("client.0")) 2024-09-16T15:47:12.110 DEBUG:teuthology.orchestra.run.smithi123:> ' 2024-09-16T15:47:12.232 INFO:teuthology.orchestra.run.smithi123.stdout:/var/run/ceph/ceph-client.0.42891.asok 2024-09-16T15:47:12.236 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:12 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-16T15:47:12.243 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.42891.asok 2024-09-16T15:47:12.243 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:47:12.243 DEBUG:teuthology.orchestra.run.smithi123:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.42891.asok status 2024-09-16T15:47:12.402 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2024-09-16T15:47:12.402 INFO:teuthology.orchestra.run.smithi123.stdout: "metadata": { 2024-09-16T15:47:12.402 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph_sha1": "33f95d2ae1bcbe7eb04404303f179eeeaa6982df", 2024-09-16T15:47:12.402 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph_version": "ceph version 19.3.0-4938-g33f95d2a (33f95d2ae1bcbe7eb04404303f179eeeaa6982df) squid (dev)", 2024-09-16T15:47:12.402 INFO:teuthology.orchestra.run.smithi123.stdout: "entity_id": "0", 2024-09-16T15:47:12.402 INFO:teuthology.orchestra.run.smithi123.stdout: "hostname": "smithi123", 2024-09-16T15:47:12.402 INFO:teuthology.orchestra.run.smithi123.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2024-09-16T15:47:12.402 INFO:teuthology.orchestra.run.smithi123.stdout: "pid": "42891", 2024-09-16T15:47:12.402 INFO:teuthology.orchestra.run.smithi123.stdout: "root": "/" 2024-09-16T15:47:12.402 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:12.402 INFO:teuthology.orchestra.run.smithi123.stdout: "dentry_count": 0, 2024-09-16T15:47:12.403 INFO:teuthology.orchestra.run.smithi123.stdout: "dentry_pinned_count": 0, 2024-09-16T15:47:12.403 INFO:teuthology.orchestra.run.smithi123.stdout: "id": 6390, 2024-09-16T15:47:12.403 INFO:teuthology.orchestra.run.smithi123.stdout: "inst": { 2024-09-16T15:47:12.403 INFO:teuthology.orchestra.run.smithi123.stdout: "name": { 2024-09-16T15:47:12.403 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "client", 2024-09-16T15:47:12.403 INFO:teuthology.orchestra.run.smithi123.stdout: "num": 6390 2024-09-16T15:47:12.403 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:12.403 INFO:teuthology.orchestra.run.smithi123.stdout: "addr": { 2024-09-16T15:47:12.403 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "v1", 2024-09-16T15:47:12.403 INFO:teuthology.orchestra.run.smithi123.stdout: "addr": "192.168.0.1:0", 2024-09-16T15:47:12.403 INFO:teuthology.orchestra.run.smithi123.stdout: "nonce": 398300954 2024-09-16T15:47:12.403 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-16T15:47:12.403 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:12.403 INFO:teuthology.orchestra.run.smithi123.stdout: "addr": { 2024-09-16T15:47:12.404 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "v1", 2024-09-16T15:47:12.404 INFO:teuthology.orchestra.run.smithi123.stdout: "addr": "192.168.0.1:0", 2024-09-16T15:47:12.404 INFO:teuthology.orchestra.run.smithi123.stdout: "nonce": 398300954 2024-09-16T15:47:12.404 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:12.404 INFO:teuthology.orchestra.run.smithi123.stdout: "inst_str": "client.6390 192.168.0.1:0/398300954", 2024-09-16T15:47:12.404 INFO:teuthology.orchestra.run.smithi123.stdout: "addr_str": "192.168.0.1:0/398300954", 2024-09-16T15:47:12.404 INFO:teuthology.orchestra.run.smithi123.stdout: "inode_count": 1, 2024-09-16T15:47:12.404 INFO:teuthology.orchestra.run.smithi123.stdout: "mds_epoch": 44, 2024-09-16T15:47:12.404 INFO:teuthology.orchestra.run.smithi123.stdout: "osd_epoch": 70, 2024-09-16T15:47:12.404 INFO:teuthology.orchestra.run.smithi123.stdout: "osd_epoch_barrier": 0, 2024-09-16T15:47:12.404 INFO:teuthology.orchestra.run.smithi123.stdout: "blocklisted": false, 2024-09-16T15:47:12.404 INFO:teuthology.orchestra.run.smithi123.stdout: "fs_name": "cephfs" 2024-09-16T15:47:12.404 INFO:teuthology.orchestra.run.smithi123.stdout:} 2024-09-16T15:47:12.415 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:47:12.415 DEBUG:teuthology.orchestra.run.smithi123:> stat --file-system '--printf=%T 2024-09-16T15:47:12.415 DEBUG:teuthology.orchestra.run.smithi123:> ' -- /home/ubuntu/cephtest/mnt.0 2024-09-16T15:47:12.420 INFO:teuthology.orchestra.run.smithi123.stdout:fuseblk 2024-09-16T15:47:12.421 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2024-09-16T15:47:12.421 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:47:12.421 DEBUG:teuthology.orchestra.run.smithi123:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-09-16T15:47:12.475 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-09-16T15:47:12.644 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:12.640+0000 7fe966eb1640 1 -- 172.21.15.123:0/897495478 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe9600cd6e0 msgr2=0x7fe9600cdb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:47:12.644 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:12.640+0000 7fe966eb1640 1 --2- 172.21.15.123:0/897495478 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe9600cd6e0 0x7fe9600cdb50 secure :-1 s=READY pgs=818 cs=0 l=1 rev1=1 crypto rx=0x7fe954009900 tx=0x7fe95401c850 comp rx=0 tx=0).stop 2024-09-16T15:47:12.644 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:12.640+0000 7fe966eb1640 1 -- 172.21.15.123:0/897495478 shutdown_connections 2024-09-16T15:47:12.644 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:12.640+0000 7fe966eb1640 1 --2- 172.21.15.123:0/897495478 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fe96015e820 0x7fe96015ecb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:12.644 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:12.640+0000 7fe966eb1640 1 --2- 172.21.15.123:0/897495478 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe9600cd6e0 0x7fe9600cdb50 unknown :-1 s=CLOSED pgs=818 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:12.644 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:12.640+0000 7fe966eb1640 1 -- 172.21.15.123:0/897495478 >> 172.21.15.123:0/897495478 conn(0x7fe9600cfb10 msgr2=0x7fe9600cff20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:47:12.644 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:12.640+0000 7fe966eb1640 1 -- 172.21.15.123:0/897495478 shutdown_connections 2024-09-16T15:47:12.644 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:12.640+0000 7fe966eb1640 1 -- 172.21.15.123:0/897495478 wait complete. 2024-09-16T15:47:12.645 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:12.640+0000 7fe966eb1640 1 Processor -- start 2024-09-16T15:47:12.645 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:12.640+0000 7fe966eb1640 1 -- start start 2024-09-16T15:47:12.646 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:12.640+0000 7fe966eb1640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe9600cd6e0 0x7fe96014d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:47:12.647 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:12.640+0000 7fe966eb1640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fe96015e160 0x7fe9600c3640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:47:12.647 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:12.640+0000 7fe966eb1640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe96015e820 0x7fe9600c44b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:47:12.647 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:12.640+0000 7fe966eb1640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fe9600cf5b0 con 0x7fe9600cd6e0 2024-09-16T15:47:12.647 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:12.640+0000 7fe966eb1640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fe9600cf430 con 0x7fe96015e820 2024-09-16T15:47:12.647 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:12.640+0000 7fe966eb1640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fe9601683d0 con 0x7fe96015e160 2024-09-16T15:47:12.647 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:12.640+0000 7fe964c4d640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe9600cd6e0 0x7fe96014d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:47:12.647 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:12.640+0000 7fe964c4d640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe9600cd6e0 0x7fe96014d190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:37968/0 (socket says 172.21.15.123:37968) 2024-09-16T15:47:12.647 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:12.640+0000 7fe964c4d640 1 -- 172.21.15.123:0/47947859 learned_addr learned my addr 172.21.15.123:0/47947859 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:47:12.647 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:12.640+0000 7fe95ffff640 1 --2- 172.21.15.123:0/47947859 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fe96015e160 0x7fe9600c3640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:47:12.647 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:12.640+0000 7fe964c4d640 1 -- 172.21.15.123:0/47947859 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fe96015e160 msgr2=0x7fe9600c3640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:47:12.647 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:12.640+0000 7fe964c4d640 1 --2- 172.21.15.123:0/47947859 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fe96015e160 0x7fe9600c3640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:12.647 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:12.640+0000 7fe96544e640 1 --2- 172.21.15.123:0/47947859 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe96015e820 0x7fe9600c44b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:47:12.647 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:12.640+0000 7fe964c4d640 1 -- 172.21.15.123:0/47947859 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe96015e820 msgr2=0x7fe9600c44b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:47:12.648 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:12.640+0000 7fe964c4d640 1 --2- 172.21.15.123:0/47947859 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe96015e820 0x7fe9600c44b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:12.648 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:12.640+0000 7fe964c4d640 1 -- 172.21.15.123:0/47947859 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe960163e30 con 0x7fe9600cd6e0 2024-09-16T15:47:12.648 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:12.640+0000 7fe964c4d640 1 --2- 172.21.15.123:0/47947859 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe9600cd6e0 0x7fe96014d190 secure :-1 s=READY pgs=819 cs=0 l=1 rev1=1 crypto rx=0x7fe95000c7a0 tx=0x7fe95000cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:47:12.648 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:12.640+0000 7fe95d7fa640 1 -- 172.21.15.123:0/47947859 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe950017070 con 0x7fe9600cd6e0 2024-09-16T15:47:12.648 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:12.640+0000 7fe95d7fa640 1 -- 172.21.15.123:0/47947859 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fe950005070 con 0x7fe9600cd6e0 2024-09-16T15:47:12.648 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:12.640+0000 7fe966eb1640 1 -- 172.21.15.123:0/47947859 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe9601f86d0 con 0x7fe9600cd6e0 2024-09-16T15:47:12.648 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:12.640+0000 7fe95d7fa640 1 -- 172.21.15.123:0/47947859 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe950005500 con 0x7fe9600cd6e0 2024-09-16T15:47:12.648 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:12.640+0000 7fe966eb1640 1 -- 172.21.15.123:0/47947859 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe9600c8270 con 0x7fe9600cd6e0 2024-09-16T15:47:12.649 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:12.644+0000 7fe966eb1640 1 -- 172.21.15.123:0/47947859 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe9600cdb50 con 0x7fe9600cd6e0 2024-09-16T15:47:12.649 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:12.644+0000 7fe95d7fa640 1 -- 172.21.15.123:0/47947859 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 11) ==== 97386+0+0 (secure 0 0 0) 0x7fe95001a070 con 0x7fe9600cd6e0 2024-09-16T15:47:12.649 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:12.644+0000 7fe95d7fa640 1 --2- 172.21.15.123:0/47947859 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fe94c073220 0x7fe94c0756e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:47:12.649 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:12.644+0000 7fe95d7fa640 1 -- 172.21.15.123:0/47947859 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(70..70 src has 1..70) ==== 5709+0+0 (secure 0 0 0) 0x7fe950093ef0 con 0x7fe9600cd6e0 2024-09-16T15:47:12.649 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:12.644+0000 7fe95ffff640 1 --2- 172.21.15.123:0/47947859 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fe94c073220 0x7fe94c0756e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:47:12.650 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:12.644+0000 7fe95ffff640 1 --2- 172.21.15.123:0/47947859 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fe94c073220 0x7fe94c0756e0 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7fe9540af000 tx=0x7fe95401b090 comp rx=0 tx=0).ready entity=mgr.4135 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:47:12.653 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:12.648+0000 7fe95d7fa640 1 -- 172.21.15.123:0/47947859 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7fe950021020 con 0x7fe9600cd6e0 2024-09-16T15:47:12.864 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:12.860+0000 7fe966eb1640 1 -- 172.21.15.123:0/47947859 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7fe9600d06c0 con 0x7fe9600cd6e0 2024-09-16T15:47:12.864 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:12.860+0000 7fe95d7fa640 1 -- 172.21.15.123:0/47947859 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v44) ==== 53+0+73 (secure 0 0 0) 0x7fe95005ed20 con 0x7fe9600cd6e0 2024-09-16T15:47:12.864 INFO:teuthology.orchestra.run.smithi123.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-09-16T15:47:12.867 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:12.864+0000 7fe966eb1640 1 -- 172.21.15.123:0/47947859 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fe94c073220 msgr2=0x7fe94c0756e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:47:12.867 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:12.864+0000 7fe966eb1640 1 --2- 172.21.15.123:0/47947859 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fe94c073220 0x7fe94c0756e0 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7fe9540af000 tx=0x7fe95401b090 comp rx=0 tx=0).stop 2024-09-16T15:47:12.867 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:12.864+0000 7fe966eb1640 1 -- 172.21.15.123:0/47947859 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe9600cd6e0 msgr2=0x7fe96014d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:47:12.868 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:12.864+0000 7fe966eb1640 1 --2- 172.21.15.123:0/47947859 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe9600cd6e0 0x7fe96014d190 secure :-1 s=READY pgs=819 cs=0 l=1 rev1=1 crypto rx=0x7fe95000c7a0 tx=0x7fe95000cc60 comp rx=0 tx=0).stop 2024-09-16T15:47:12.868 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:12.864+0000 7fe966eb1640 1 -- 172.21.15.123:0/47947859 shutdown_connections 2024-09-16T15:47:12.868 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:12.864+0000 7fe966eb1640 1 --2- 172.21.15.123:0/47947859 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fe94c073220 0x7fe94c0756e0 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:12.868 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:12.864+0000 7fe966eb1640 1 --2- 172.21.15.123:0/47947859 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe96015e820 0x7fe9600c44b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:12.868 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:12.864+0000 7fe966eb1640 1 --2- 172.21.15.123:0/47947859 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fe96015e160 0x7fe9600c3640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:12.868 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:12.864+0000 7fe966eb1640 1 --2- 172.21.15.123:0/47947859 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe9600cd6e0 0x7fe96014d190 unknown :-1 s=CLOSED pgs=819 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:12.868 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:12.864+0000 7fe966eb1640 1 -- 172.21.15.123:0/47947859 >> 172.21.15.123:0/47947859 conn(0x7fe9600cfb10 msgr2=0x7fe960159cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:47:12.868 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:12.864+0000 7fe966eb1640 1 -- 172.21.15.123:0/47947859 shutdown_connections 2024-09-16T15:47:12.868 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:12.864+0000 7fe966eb1640 1 -- 172.21.15.123:0/47947859 wait complete. 2024-09-16T15:47:12.883 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-09-16T15:47:12.883 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-09-16T15:47:12.883 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi184.front.sepia.ceph.com 2024-09-16T15:47:12.883 INFO:tasks.cephfs.mount:self.client.name = client.1 2024-09-16T15:47:12.884 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2024-09-16T15:47:12.884 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-09-16T15:47:12.884 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-09-16T15:47:12.884 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-09-16T15:47:12.884 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.1' netns for '/home/ubuntu/cephtest/mnt.1' 2024-09-16T15:47:12.884 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:47:12.884 DEBUG:teuthology.orchestra.run.smithi184:> ip addr 2024-09-16T15:47:12.891 INFO:teuthology.orchestra.run.smithi184.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-09-16T15:47:12.892 INFO:teuthology.orchestra.run.smithi184.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-09-16T15:47:12.892 INFO:teuthology.orchestra.run.smithi184.stdout: inet 127.0.0.1/8 scope host lo 2024-09-16T15:47:12.892 INFO:teuthology.orchestra.run.smithi184.stdout: valid_lft forever preferred_lft forever 2024-09-16T15:47:12.892 INFO:teuthology.orchestra.run.smithi184.stdout: inet6 ::1/128 scope host 2024-09-16T15:47:12.892 INFO:teuthology.orchestra.run.smithi184.stdout: valid_lft forever preferred_lft forever 2024-09-16T15:47:12.892 INFO:teuthology.orchestra.run.smithi184.stdout:2: eno1: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-09-16T15:47:12.892 INFO:teuthology.orchestra.run.smithi184.stdout: link/ether 0c:c4:7a:d9:37:16 brd ff:ff:ff:ff:ff:ff 2024-09-16T15:47:12.892 INFO:teuthology.orchestra.run.smithi184.stdout: altname enp7s0f0 2024-09-16T15:47:12.892 INFO:teuthology.orchestra.run.smithi184.stdout:3: eno2: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-09-16T15:47:12.892 INFO:teuthology.orchestra.run.smithi184.stdout: link/ether 0c:c4:7a:d9:37:17 brd ff:ff:ff:ff:ff:ff 2024-09-16T15:47:12.892 INFO:teuthology.orchestra.run.smithi184.stdout: altname enp7s0f1 2024-09-16T15:47:12.892 INFO:teuthology.orchestra.run.smithi184.stdout:4: enp3s0f0: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-09-16T15:47:12.893 INFO:teuthology.orchestra.run.smithi184.stdout: link/ether 0c:c4:7a:88:7b:8e brd ff:ff:ff:ff:ff:ff 2024-09-16T15:47:12.893 INFO:teuthology.orchestra.run.smithi184.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-09-16T15:47:12.893 INFO:teuthology.orchestra.run.smithi184.stdout: link/ether 0c:c4:7a:88:7b:8f brd ff:ff:ff:ff:ff:ff 2024-09-16T15:47:12.893 INFO:teuthology.orchestra.run.smithi184.stdout: inet 172.21.15.184/20 metric 100 brd 172.21.15.255 scope global dynamic enp3s0f1 2024-09-16T15:47:12.893 INFO:teuthology.orchestra.run.smithi184.stdout: valid_lft 42224sec preferred_lft 42224sec 2024-09-16T15:47:12.893 INFO:teuthology.orchestra.run.smithi184.stdout: inet6 fe80::ec4:7aff:fe88:7b8f/64 scope link 2024-09-16T15:47:12.893 INFO:teuthology.orchestra.run.smithi184.stdout: valid_lft forever preferred_lft forever 2024-09-16T15:47:12.893 INFO:teuthology.orchestra.run.smithi184.stdout:6: docker0: mtu 1500 qdisc noqueue state DOWN group default 2024-09-16T15:47:12.893 INFO:teuthology.orchestra.run.smithi184.stdout: link/ether 02:42:5d:0a:44:b9 brd ff:ff:ff:ff:ff:ff 2024-09-16T15:47:12.893 INFO:teuthology.orchestra.run.smithi184.stdout: inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0 2024-09-16T15:47:12.893 INFO:teuthology.orchestra.run.smithi184.stdout: valid_lft forever preferred_lft forever 2024-09-16T15:47:12.893 INFO:teuthology.orchestra.run.smithi184.stdout:7: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-09-16T15:47:12.893 INFO:teuthology.orchestra.run.smithi184.stdout: link/ether 5e:1f:84:6d:79:ae brd ff:ff:ff:ff:ff:ff 2024-09-16T15:47:12.893 INFO:teuthology.orchestra.run.smithi184.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-09-16T15:47:12.894 INFO:teuthology.orchestra.run.smithi184.stdout: valid_lft forever preferred_lft forever 2024-09-16T15:47:12.894 INFO:teuthology.orchestra.run.smithi184.stdout: inet6 fe80::5c1f:84ff:fe6d:79ae/64 scope link 2024-09-16T15:47:12.894 INFO:teuthology.orchestra.run.smithi184.stdout: valid_lft forever preferred_lft forever 2024-09-16T15:47:12.894 INFO:teuthology.orchestra.run.smithi184.stdout:8: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-09-16T15:47:12.894 INFO:teuthology.orchestra.run.smithi184.stdout: link/ether 0e:5c:f6:1f:82:2f brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.1 2024-09-16T15:47:12.894 INFO:teuthology.orchestra.run.smithi184.stdout: inet6 fe80::c5c:f6ff:fe1f:822f/64 scope link 2024-09-16T15:47:12.894 INFO:teuthology.orchestra.run.smithi184.stdout: valid_lft forever preferred_lft forever 2024-09-16T15:47:12.894 INFO:teuthology.orchestra.run.smithi184.stdout:9: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-09-16T15:47:12.894 INFO:teuthology.orchestra.run.smithi184.stdout: link/ether c6:a6:3d:69:d0:db brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.3 2024-09-16T15:47:12.894 INFO:teuthology.orchestra.run.smithi184.stdout: inet6 fe80::c4a6:3dff:fe69:d0db/64 scope link 2024-09-16T15:47:12.894 INFO:teuthology.orchestra.run.smithi184.stdout: valid_lft forever preferred_lft forever 2024-09-16T15:47:12.894 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:47:12.894 DEBUG:teuthology.orchestra.run.smithi184:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-09-16T15:47:12.947 INFO:teuthology.orchestra.run.smithi184.stdout:1 2024-09-16T15:47:12.948 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:47:12.948 DEBUG:teuthology.orchestra.run.smithi184:> route 2024-09-16T15:47:12.999 INFO:teuthology.orchestra.run.smithi184.stdout:Kernel IP routing table 2024-09-16T15:47:12.999 INFO:teuthology.orchestra.run.smithi184.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-09-16T15:47:12.999 INFO:teuthology.orchestra.run.smithi184.stdout:default _gateway 0.0.0.0 UG 0 0 0 enp3s0f1 2024-09-16T15:47:12.999 INFO:teuthology.orchestra.run.smithi184.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-09-16T15:47:12.999 INFO:teuthology.orchestra.run.smithi184.stdout:158.69.67.47 _gateway 255.255.255.255 UGH 100 0 0 enp3s0f1 2024-09-16T15:47:12.999 INFO:teuthology.orchestra.run.smithi184.stdout:172.17.0.0 0.0.0.0 255.255.0.0 U 0 0 0 docker0 2024-09-16T15:47:13.000 INFO:teuthology.orchestra.run.smithi184.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-09-16T15:47:13.000 INFO:teuthology.orchestra.run.smithi184.stdout:ns1.front.sepia 0.0.0.0 255.255.255.255 UH 100 0 0 enp3s0f1 2024-09-16T15:47:13.000 INFO:teuthology.orchestra.run.smithi184.stdout:172.21.0.2 0.0.0.0 255.255.255.255 UH 100 0 0 enp3s0f1 2024-09-16T15:47:13.000 INFO:teuthology.orchestra.run.smithi184.stdout:_gateway 0.0.0.0 255.255.255.255 UH 100 0 0 enp3s0f1 2024-09-16T15:47:13.000 INFO:teuthology.orchestra.run.smithi184.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-09-16T15:47:13.000 DEBUG:teuthology.orchestra.run.smithi184:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-09-16T15:47:13.000 DEBUG:teuthology.orchestra.run.smithi184:> set -e 2024-09-16T15:47:13.000 DEBUG:teuthology.orchestra.run.smithi184:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-09-16T15:47:13.000 DEBUG:teuthology.orchestra.run.smithi184:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-09-16T15:47:13.000 DEBUG:teuthology.orchestra.run.smithi184:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-09-16T15:47:13.000 DEBUG:teuthology.orchestra.run.smithi184:> ') 2024-09-16T15:47:13.089 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:47:13 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-09-16T15:47:13.116 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:47:13 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-16T15:47:13.123 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:47:13.123 DEBUG:teuthology.orchestra.run.smithi184:> ip netns list 2024-09-16T15:47:13.129 INFO:teuthology.orchestra.run.smithi184.stdout:ceph-ns--home-ubuntu-cephtest-mnt.3 (id: 1) 2024-09-16T15:47:13.129 INFO:teuthology.orchestra.run.smithi184.stdout:ceph-ns--home-ubuntu-cephtest-mnt.1 (id: 0) 2024-09-16T15:47:13.130 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.1' netns for '/home/ubuntu/cephtest/mnt.1' 2024-09-16T15:47:13.130 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:47:13.130 DEBUG:teuthology.orchestra.run.smithi184:> sudo ip link set brx.0 up 2024-09-16T15:47:13.183 INFO:tasks.cephfs.fuse_mount:Client client.1 config is {} 2024-09-16T15:47:13.183 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-09-16T15:47:13.183 DEBUG:teuthology.orchestra.run.smithi184:> mkdir -p -v /home/ubuntu/cephtest/mnt.1 2024-09-16T15:47:13.231 INFO:teuthology.orchestra.run.smithi184.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.1' 2024-09-16T15:47:13.231 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-09-16T15:47:13.231 DEBUG:teuthology.orchestra.run.smithi184:> chmod 0000 /home/ubuntu/cephtest/mnt.1 2024-09-16T15:47:13.279 DEBUG:teuthology.orchestra.run.smithi184:> sudo modprobe fuse 2024-09-16T15:47:13.337 DEBUG:teuthology.orchestra.run.smithi184:> cat /proc/self/mounts | awk '{print $2}' 2024-09-16T15:47:13.384 INFO:teuthology.orchestra.run.smithi184.stdout:/sys 2024-09-16T15:47:13.384 INFO:teuthology.orchestra.run.smithi184.stdout:/proc 2024-09-16T15:47:13.384 INFO:teuthology.orchestra.run.smithi184.stdout:/dev 2024-09-16T15:47:13.384 INFO:teuthology.orchestra.run.smithi184.stdout:/dev/pts 2024-09-16T15:47:13.385 INFO:teuthology.orchestra.run.smithi184.stdout:/run 2024-09-16T15:47:13.385 INFO:teuthology.orchestra.run.smithi184.stdout:/ 2024-09-16T15:47:13.385 INFO:teuthology.orchestra.run.smithi184.stdout:/sys/kernel/security 2024-09-16T15:47:13.385 INFO:teuthology.orchestra.run.smithi184.stdout:/dev/shm 2024-09-16T15:47:13.385 INFO:teuthology.orchestra.run.smithi184.stdout:/run/lock 2024-09-16T15:47:13.385 INFO:teuthology.orchestra.run.smithi184.stdout:/sys/fs/cgroup 2024-09-16T15:47:13.385 INFO:teuthology.orchestra.run.smithi184.stdout:/sys/fs/pstore 2024-09-16T15:47:13.385 INFO:teuthology.orchestra.run.smithi184.stdout:/sys/fs/bpf 2024-09-16T15:47:13.385 INFO:teuthology.orchestra.run.smithi184.stdout:/proc/sys/fs/binfmt_misc 2024-09-16T15:47:13.385 INFO:teuthology.orchestra.run.smithi184.stdout:/dev/hugepages 2024-09-16T15:47:13.385 INFO:teuthology.orchestra.run.smithi184.stdout:/dev/mqueue 2024-09-16T15:47:13.385 INFO:teuthology.orchestra.run.smithi184.stdout:/sys/kernel/debug 2024-09-16T15:47:13.385 INFO:teuthology.orchestra.run.smithi184.stdout:/sys/kernel/tracing 2024-09-16T15:47:13.386 INFO:teuthology.orchestra.run.smithi184.stdout:/proc/fs/nfsd 2024-09-16T15:47:13.386 INFO:teuthology.orchestra.run.smithi184.stdout:/sys/fs/fuse/connections 2024-09-16T15:47:13.386 INFO:teuthology.orchestra.run.smithi184.stdout:/sys/kernel/config 2024-09-16T15:47:13.386 INFO:teuthology.orchestra.run.smithi184.stdout:/run/credentials/systemd-sysusers.service 2024-09-16T15:47:13.386 INFO:teuthology.orchestra.run.smithi184.stdout:/run/rpc_pipefs 2024-09-16T15:47:13.386 INFO:teuthology.orchestra.run.smithi184.stdout:/run/user/1000 2024-09-16T15:47:13.386 INFO:teuthology.orchestra.run.smithi184.stdout:/var/lib/ceph 2024-09-16T15:47:13.386 INFO:teuthology.orchestra.run.smithi184.stdout:/var/lib/ceph/osd/ceph-4 2024-09-16T15:47:13.386 INFO:teuthology.orchestra.run.smithi184.stdout:/var/lib/ceph/osd/ceph-5 2024-09-16T15:47:13.386 INFO:teuthology.orchestra.run.smithi184.stdout:/var/lib/ceph/osd/ceph-6 2024-09-16T15:47:13.386 INFO:teuthology.orchestra.run.smithi184.stdout:/var/lib/ceph/osd/ceph-7 2024-09-16T15:47:13.387 INFO:teuthology.orchestra.run.smithi184.stdout:/run/netns 2024-09-16T15:47:13.387 INFO:teuthology.orchestra.run.smithi184.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-09-16T15:47:13.387 INFO:teuthology.orchestra.run.smithi184.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-09-16T15:47:13.387 INFO:teuthology.orchestra.run.smithi184.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-09-16T15:47:13.387 INFO:teuthology.orchestra.run.smithi184.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-09-16T15:47:13.388 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:47:13.388 DEBUG:teuthology.orchestra.run.smithi184:> ls /sys/fs/fuse/connections 2024-09-16T15:47:13.439 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-09-16T15:47:13.440 DEBUG:teuthology.orchestra.run.smithi184:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.1 --id 1) 2024-09-16T15:47:13.485 DEBUG:teuthology.orchestra.run.smithi184:> sudo modprobe fuse 2024-09-16T15:47:13.500 DEBUG:teuthology.orchestra.run.smithi184:> cat /proc/self/mounts | awk '{print $2}' 2024-09-16T15:47:13.548 INFO:teuthology.orchestra.run.smithi184.stdout:/sys 2024-09-16T15:47:13.548 INFO:teuthology.orchestra.run.smithi184.stdout:/proc 2024-09-16T15:47:13.548 INFO:teuthology.orchestra.run.smithi184.stdout:/dev 2024-09-16T15:47:13.548 INFO:teuthology.orchestra.run.smithi184.stdout:/dev/pts 2024-09-16T15:47:13.548 INFO:teuthology.orchestra.run.smithi184.stdout:/run 2024-09-16T15:47:13.548 INFO:teuthology.orchestra.run.smithi184.stdout:/ 2024-09-16T15:47:13.548 INFO:teuthology.orchestra.run.smithi184.stdout:/sys/kernel/security 2024-09-16T15:47:13.549 INFO:teuthology.orchestra.run.smithi184.stdout:/dev/shm 2024-09-16T15:47:13.549 INFO:teuthology.orchestra.run.smithi184.stdout:/run/lock 2024-09-16T15:47:13.549 INFO:teuthology.orchestra.run.smithi184.stdout:/sys/fs/cgroup 2024-09-16T15:47:13.549 INFO:teuthology.orchestra.run.smithi184.stdout:/sys/fs/pstore 2024-09-16T15:47:13.549 INFO:teuthology.orchestra.run.smithi184.stdout:/sys/fs/bpf 2024-09-16T15:47:13.549 INFO:teuthology.orchestra.run.smithi184.stdout:/proc/sys/fs/binfmt_misc 2024-09-16T15:47:13.549 INFO:teuthology.orchestra.run.smithi184.stdout:/dev/hugepages 2024-09-16T15:47:13.549 INFO:teuthology.orchestra.run.smithi184.stdout:/dev/mqueue 2024-09-16T15:47:13.549 INFO:teuthology.orchestra.run.smithi184.stdout:/sys/kernel/debug 2024-09-16T15:47:13.549 INFO:teuthology.orchestra.run.smithi184.stdout:/sys/kernel/tracing 2024-09-16T15:47:13.549 INFO:teuthology.orchestra.run.smithi184.stdout:/proc/fs/nfsd 2024-09-16T15:47:13.549 INFO:teuthology.orchestra.run.smithi184.stdout:/sys/fs/fuse/connections 2024-09-16T15:47:13.549 INFO:teuthology.orchestra.run.smithi184.stdout:/sys/kernel/config 2024-09-16T15:47:13.549 INFO:teuthology.orchestra.run.smithi184.stdout:/run/credentials/systemd-sysusers.service 2024-09-16T15:47:13.550 INFO:teuthology.orchestra.run.smithi184.stdout:/run/rpc_pipefs 2024-09-16T15:47:13.550 INFO:teuthology.orchestra.run.smithi184.stdout:/run/user/1000 2024-09-16T15:47:13.550 INFO:teuthology.orchestra.run.smithi184.stdout:/var/lib/ceph 2024-09-16T15:47:13.550 INFO:teuthology.orchestra.run.smithi184.stdout:/var/lib/ceph/osd/ceph-4 2024-09-16T15:47:13.550 INFO:teuthology.orchestra.run.smithi184.stdout:/var/lib/ceph/osd/ceph-5 2024-09-16T15:47:13.550 INFO:teuthology.orchestra.run.smithi184.stdout:/var/lib/ceph/osd/ceph-6 2024-09-16T15:47:13.550 INFO:teuthology.orchestra.run.smithi184.stdout:/var/lib/ceph/osd/ceph-7 2024-09-16T15:47:13.550 INFO:teuthology.orchestra.run.smithi184.stdout:/run/netns 2024-09-16T15:47:13.550 INFO:teuthology.orchestra.run.smithi184.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-09-16T15:47:13.550 INFO:teuthology.orchestra.run.smithi184.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-09-16T15:47:13.550 INFO:teuthology.orchestra.run.smithi184.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-09-16T15:47:13.550 INFO:teuthology.orchestra.run.smithi184.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-09-16T15:47:13.550 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:47:13.551 DEBUG:teuthology.orchestra.run.smithi184:> ls /sys/fs/fuse/connections 2024-09-16T15:47:13.560 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi184.stderr:2024-09-16T15:47:13.557+0000 7f6616657580 -1 init, newargv = 0x5600ce17cb20 newargc=15 2024-09-16T15:47:13.560 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi184.stderr:ceph-fuse[35318]: starting ceph client 2024-09-16T15:47:13.569 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi184.stderr:ceph-fuse[35318]: starting fuse 2024-09-16T15:47:14.666 DEBUG:teuthology.orchestra.run.smithi184:> sudo modprobe fuse 2024-09-16T15:47:14.683 DEBUG:teuthology.orchestra.run.smithi184:> cat /proc/self/mounts | awk '{print $2}' 2024-09-16T15:47:14.732 INFO:teuthology.orchestra.run.smithi184.stdout:/sys 2024-09-16T15:47:14.733 INFO:teuthology.orchestra.run.smithi184.stdout:/proc 2024-09-16T15:47:14.733 INFO:teuthology.orchestra.run.smithi184.stdout:/dev 2024-09-16T15:47:14.733 INFO:teuthology.orchestra.run.smithi184.stdout:/dev/pts 2024-09-16T15:47:14.733 INFO:teuthology.orchestra.run.smithi184.stdout:/run 2024-09-16T15:47:14.733 INFO:teuthology.orchestra.run.smithi184.stdout:/ 2024-09-16T15:47:14.733 INFO:teuthology.orchestra.run.smithi184.stdout:/sys/kernel/security 2024-09-16T15:47:14.733 INFO:teuthology.orchestra.run.smithi184.stdout:/dev/shm 2024-09-16T15:47:14.733 INFO:teuthology.orchestra.run.smithi184.stdout:/run/lock 2024-09-16T15:47:14.733 INFO:teuthology.orchestra.run.smithi184.stdout:/sys/fs/cgroup 2024-09-16T15:47:14.733 INFO:teuthology.orchestra.run.smithi184.stdout:/sys/fs/pstore 2024-09-16T15:47:14.733 INFO:teuthology.orchestra.run.smithi184.stdout:/sys/fs/bpf 2024-09-16T15:47:14.734 INFO:teuthology.orchestra.run.smithi184.stdout:/proc/sys/fs/binfmt_misc 2024-09-16T15:47:14.734 INFO:teuthology.orchestra.run.smithi184.stdout:/dev/hugepages 2024-09-16T15:47:14.734 INFO:teuthology.orchestra.run.smithi184.stdout:/dev/mqueue 2024-09-16T15:47:14.734 INFO:teuthology.orchestra.run.smithi184.stdout:/sys/kernel/debug 2024-09-16T15:47:14.734 INFO:teuthology.orchestra.run.smithi184.stdout:/sys/kernel/tracing 2024-09-16T15:47:14.734 INFO:teuthology.orchestra.run.smithi184.stdout:/proc/fs/nfsd 2024-09-16T15:47:14.734 INFO:teuthology.orchestra.run.smithi184.stdout:/sys/fs/fuse/connections 2024-09-16T15:47:14.734 INFO:teuthology.orchestra.run.smithi184.stdout:/sys/kernel/config 2024-09-16T15:47:14.734 INFO:teuthology.orchestra.run.smithi184.stdout:/run/credentials/systemd-sysusers.service 2024-09-16T15:47:14.734 INFO:teuthology.orchestra.run.smithi184.stdout:/run/rpc_pipefs 2024-09-16T15:47:14.734 INFO:teuthology.orchestra.run.smithi184.stdout:/run/user/1000 2024-09-16T15:47:14.734 INFO:teuthology.orchestra.run.smithi184.stdout:/var/lib/ceph 2024-09-16T15:47:14.734 INFO:teuthology.orchestra.run.smithi184.stdout:/var/lib/ceph/osd/ceph-4 2024-09-16T15:47:14.734 INFO:teuthology.orchestra.run.smithi184.stdout:/var/lib/ceph/osd/ceph-5 2024-09-16T15:47:14.735 INFO:teuthology.orchestra.run.smithi184.stdout:/var/lib/ceph/osd/ceph-6 2024-09-16T15:47:14.735 INFO:teuthology.orchestra.run.smithi184.stdout:/var/lib/ceph/osd/ceph-7 2024-09-16T15:47:14.735 INFO:teuthology.orchestra.run.smithi184.stdout:/run/netns 2024-09-16T15:47:14.735 INFO:teuthology.orchestra.run.smithi184.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-09-16T15:47:14.735 INFO:teuthology.orchestra.run.smithi184.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-09-16T15:47:14.735 INFO:teuthology.orchestra.run.smithi184.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-09-16T15:47:14.735 INFO:teuthology.orchestra.run.smithi184.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-09-16T15:47:14.735 INFO:teuthology.orchestra.run.smithi184.stdout:/home/ubuntu/cephtest/mnt.1 2024-09-16T15:47:14.736 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:47:14.736 DEBUG:teuthology.orchestra.run.smithi184:> ls /sys/fs/fuse/connections 2024-09-16T15:47:14.783 INFO:teuthology.orchestra.run.smithi184.stdout:45 2024-09-16T15:47:14.783 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [45] 2024-09-16T15:47:14.783 DEBUG:teuthology.orchestra.run.smithi184:> sudo stdin-killer -- python3 -c ' 2024-09-16T15:47:14.783 DEBUG:teuthology.orchestra.run.smithi184:> import glob 2024-09-16T15:47:14.783 DEBUG:teuthology.orchestra.run.smithi184:> import re 2024-09-16T15:47:14.784 DEBUG:teuthology.orchestra.run.smithi184:> import os 2024-09-16T15:47:14.784 DEBUG:teuthology.orchestra.run.smithi184:> import subprocess 2024-09-16T15:47:14.784 DEBUG:teuthology.orchestra.run.smithi184:> 2024-09-16T15:47:14.784 DEBUG:teuthology.orchestra.run.smithi184:> def _find_admin_socket(client_name): 2024-09-16T15:47:14.784 DEBUG:teuthology.orchestra.run.smithi184:> asok_path = "/var/run/ceph/ceph-client.1.*.asok" 2024-09-16T15:47:14.784 DEBUG:teuthology.orchestra.run.smithi184:> files = glob.glob(asok_path) 2024-09-16T15:47:14.784 DEBUG:teuthology.orchestra.run.smithi184:> mountpoint = "/home/ubuntu/cephtest/mnt.1" 2024-09-16T15:47:14.784 DEBUG:teuthology.orchestra.run.smithi184:> 2024-09-16T15:47:14.784 DEBUG:teuthology.orchestra.run.smithi184:> # Given a non-glob path, it better be there 2024-09-16T15:47:14.784 DEBUG:teuthology.orchestra.run.smithi184:> if "*" not in asok_path: 2024-09-16T15:47:14.784 DEBUG:teuthology.orchestra.run.smithi184:> assert(len(files) == 1) 2024-09-16T15:47:14.784 DEBUG:teuthology.orchestra.run.smithi184:> return files[0] 2024-09-16T15:47:14.785 DEBUG:teuthology.orchestra.run.smithi184:> 2024-09-16T15:47:14.785 DEBUG:teuthology.orchestra.run.smithi184:> for f in files: 2024-09-16T15:47:14.785 DEBUG:teuthology.orchestra.run.smithi184:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-09-16T15:47:14.785 DEBUG:teuthology.orchestra.run.smithi184:> if os.path.exists("/proc/{0}".format(pid)): 2024-09-16T15:47:14.785 DEBUG:teuthology.orchestra.run.smithi184:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-09-16T15:47:14.785 DEBUG:teuthology.orchestra.run.smithi184:> contents = proc_f.read() 2024-09-16T15:47:14.785 DEBUG:teuthology.orchestra.run.smithi184:> if mountpoint in contents: 2024-09-16T15:47:14.785 DEBUG:teuthology.orchestra.run.smithi184:> return f 2024-09-16T15:47:14.785 DEBUG:teuthology.orchestra.run.smithi184:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-09-16T15:47:14.785 DEBUG:teuthology.orchestra.run.smithi184:> 2024-09-16T15:47:14.785 DEBUG:teuthology.orchestra.run.smithi184:> print(_find_admin_socket("client.1")) 2024-09-16T15:47:14.785 DEBUG:teuthology.orchestra.run.smithi184:> ' 2024-09-16T15:47:14.908 INFO:teuthology.orchestra.run.smithi184.stdout:/var/run/ceph/ceph-client.1.35318.asok 2024-09-16T15:47:14.912 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:47:14 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-16T15:47:14.920 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.1.35318.asok 2024-09-16T15:47:14.920 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:47:14.920 DEBUG:teuthology.orchestra.run.smithi184:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.1.35318.asok status 2024-09-16T15:47:15.076 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-16T15:47:15.076 INFO:teuthology.orchestra.run.smithi184.stdout: "metadata": { 2024-09-16T15:47:15.076 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph_sha1": "33f95d2ae1bcbe7eb04404303f179eeeaa6982df", 2024-09-16T15:47:15.076 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph_version": "ceph version 19.3.0-4938-g33f95d2a (33f95d2ae1bcbe7eb04404303f179eeeaa6982df) squid (dev)", 2024-09-16T15:47:15.076 INFO:teuthology.orchestra.run.smithi184.stdout: "entity_id": "1", 2024-09-16T15:47:15.076 INFO:teuthology.orchestra.run.smithi184.stdout: "hostname": "smithi184", 2024-09-16T15:47:15.076 INFO:teuthology.orchestra.run.smithi184.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.1", 2024-09-16T15:47:15.077 INFO:teuthology.orchestra.run.smithi184.stdout: "pid": "35318", 2024-09-16T15:47:15.077 INFO:teuthology.orchestra.run.smithi184.stdout: "root": "/" 2024-09-16T15:47:15.077 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-16T15:47:15.077 INFO:teuthology.orchestra.run.smithi184.stdout: "dentry_count": 0, 2024-09-16T15:47:15.077 INFO:teuthology.orchestra.run.smithi184.stdout: "dentry_pinned_count": 0, 2024-09-16T15:47:15.077 INFO:teuthology.orchestra.run.smithi184.stdout: "id": 5506, 2024-09-16T15:47:15.077 INFO:teuthology.orchestra.run.smithi184.stdout: "inst": { 2024-09-16T15:47:15.077 INFO:teuthology.orchestra.run.smithi184.stdout: "name": { 2024-09-16T15:47:15.077 INFO:teuthology.orchestra.run.smithi184.stdout: "type": "client", 2024-09-16T15:47:15.077 INFO:teuthology.orchestra.run.smithi184.stdout: "num": 5506 2024-09-16T15:47:15.077 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-16T15:47:15.077 INFO:teuthology.orchestra.run.smithi184.stdout: "addr": { 2024-09-16T15:47:15.078 INFO:teuthology.orchestra.run.smithi184.stdout: "type": "v1", 2024-09-16T15:47:15.078 INFO:teuthology.orchestra.run.smithi184.stdout: "addr": "192.168.0.1:0", 2024-09-16T15:47:15.078 INFO:teuthology.orchestra.run.smithi184.stdout: "nonce": 3003688428 2024-09-16T15:47:15.078 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-16T15:47:15.078 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-16T15:47:15.078 INFO:teuthology.orchestra.run.smithi184.stdout: "addr": { 2024-09-16T15:47:15.078 INFO:teuthology.orchestra.run.smithi184.stdout: "type": "v1", 2024-09-16T15:47:15.078 INFO:teuthology.orchestra.run.smithi184.stdout: "addr": "192.168.0.1:0", 2024-09-16T15:47:15.078 INFO:teuthology.orchestra.run.smithi184.stdout: "nonce": 3003688428 2024-09-16T15:47:15.078 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-16T15:47:15.078 INFO:teuthology.orchestra.run.smithi184.stdout: "inst_str": "client.5506 192.168.0.1:0/3003688428", 2024-09-16T15:47:15.078 INFO:teuthology.orchestra.run.smithi184.stdout: "addr_str": "192.168.0.1:0/3003688428", 2024-09-16T15:47:15.078 INFO:teuthology.orchestra.run.smithi184.stdout: "inode_count": 1, 2024-09-16T15:47:15.078 INFO:teuthology.orchestra.run.smithi184.stdout: "mds_epoch": 44, 2024-09-16T15:47:15.079 INFO:teuthology.orchestra.run.smithi184.stdout: "osd_epoch": 70, 2024-09-16T15:47:15.079 INFO:teuthology.orchestra.run.smithi184.stdout: "osd_epoch_barrier": 0, 2024-09-16T15:47:15.079 INFO:teuthology.orchestra.run.smithi184.stdout: "blocklisted": false, 2024-09-16T15:47:15.079 INFO:teuthology.orchestra.run.smithi184.stdout: "fs_name": "cephfs" 2024-09-16T15:47:15.079 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-16T15:47:15.090 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:47:15.090 DEBUG:teuthology.orchestra.run.smithi184:> stat --file-system '--printf=%T 2024-09-16T15:47:15.090 DEBUG:teuthology.orchestra.run.smithi184:> ' -- /home/ubuntu/cephtest/mnt.1 2024-09-16T15:47:15.096 INFO:teuthology.orchestra.run.smithi184.stdout:fuseblk 2024-09-16T15:47:15.096 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2024-09-16T15:47:15.096 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:47:15.096 DEBUG:teuthology.orchestra.run.smithi184:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.1 2024-09-16T15:47:15.151 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs flag set enable_multiple true --yes-i-really-mean-it 2024-09-16T15:47:15.296 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:15.292+0000 7fadb313d640 1 -- 172.21.15.123:0/1039544612 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fadac15bc40 msgr2=0x7fadac070470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:47:15.296 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:15.292+0000 7fadb313d640 1 --2- 172.21.15.123:0/1039544612 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fadac15bc40 0x7fadac070470 secure :-1 s=READY pgs=821 cs=0 l=1 rev1=1 crypto rx=0x7fad9c009900 tx=0x7fad9c01c820 comp rx=0 tx=0).stop 2024-09-16T15:47:15.296 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:15.292+0000 7fadb313d640 1 -- 172.21.15.123:0/1039544612 shutdown_connections 2024-09-16T15:47:15.296 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:15.292+0000 7fadb313d640 1 --2- 172.21.15.123:0/1039544612 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fadac160310 0x7fadac0725e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:15.296 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:15.292+0000 7fadb313d640 1 --2- 172.21.15.123:0/1039544612 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fadac15bc40 0x7fadac070470 unknown :-1 s=CLOSED pgs=821 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:15.296 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:15.292+0000 7fadb313d640 1 -- 172.21.15.123:0/1039544612 >> 172.21.15.123:0/1039544612 conn(0x7fadac0b1630 msgr2=0x7fadac0b1a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:47:15.296 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:15.292+0000 7fadb313d640 1 -- 172.21.15.123:0/1039544612 shutdown_connections 2024-09-16T15:47:15.296 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:15.292+0000 7fadb313d640 1 -- 172.21.15.123:0/1039544612 wait complete. 2024-09-16T15:47:15.297 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:15.292+0000 7fadb313d640 1 Processor -- start 2024-09-16T15:47:15.297 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:15.292+0000 7fadb313d640 1 -- start start 2024-09-16T15:47:15.299 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:15.292+0000 7fadb313d640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fadac15bc40 0x7fadac146070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:47:15.299 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:15.292+0000 7fadb313d640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fadac15f690 0x7fadac14aa10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:47:15.299 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:15.292+0000 7fadb313d640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fadac160310 0x7fadac14af50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:47:15.299 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:15.292+0000 7fadb313d640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fadac1693c0 con 0x7fadac160310 2024-09-16T15:47:15.299 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:15.292+0000 7fadb313d640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fadac169240 con 0x7fadac15f690 2024-09-16T15:47:15.299 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:15.292+0000 7fadb313d640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fadac169540 con 0x7fadac15bc40 2024-09-16T15:47:15.299 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:15.292+0000 7fadb16da640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fadac160310 0x7fadac14af50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:47:15.299 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:15.292+0000 7fadb16da640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fadac160310 0x7fadac14af50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:37984/0 (socket says 172.21.15.123:37984) 2024-09-16T15:47:15.300 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:15.292+0000 7fadb16da640 1 -- 172.21.15.123:0/3572342532 learned_addr learned my addr 172.21.15.123:0/3572342532 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:47:15.300 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:15.292+0000 7fadabfff640 1 --2- 172.21.15.123:0/3572342532 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fadac15f690 0x7fadac14aa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:47:15.300 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:15.292+0000 7fadb0ed9640 1 --2- 172.21.15.123:0/3572342532 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fadac15bc40 0x7fadac146070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:47:15.300 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:15.292+0000 7fadb16da640 1 -- 172.21.15.123:0/3572342532 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fadac15bc40 msgr2=0x7fadac146070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:47:15.300 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:15.292+0000 7fadb16da640 1 --2- 172.21.15.123:0/3572342532 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fadac15bc40 0x7fadac146070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:15.300 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:15.292+0000 7fadb16da640 1 -- 172.21.15.123:0/3572342532 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fadac15f690 msgr2=0x7fadac14aa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:47:15.300 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:15.292+0000 7fadb16da640 1 --2- 172.21.15.123:0/3572342532 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fadac15f690 0x7fadac14aa10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:15.300 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:15.292+0000 7fadb16da640 1 -- 172.21.15.123:0/3572342532 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fadac1f8bf0 con 0x7fadac160310 2024-09-16T15:47:15.300 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:15.292+0000 7fadabfff640 1 --2- 172.21.15.123:0/3572342532 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fadac15f690 0x7fadac14aa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T15:47:15.300 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:15.296+0000 7fadb16da640 1 --2- 172.21.15.123:0/3572342532 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fadac160310 0x7fadac14af50 secure :-1 s=READY pgs=822 cs=0 l=1 rev1=1 crypto rx=0x7fada400c7c0 tx=0x7fada400cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:47:15.301 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:15.296+0000 7fada97fa640 1 -- 172.21.15.123:0/3572342532 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fada4003b80 con 0x7fadac160310 2024-09-16T15:47:15.301 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:15.296+0000 7fada97fa640 1 -- 172.21.15.123:0/3572342532 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fada4003d20 con 0x7fadac160310 2024-09-16T15:47:15.301 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:15.296+0000 7fadb313d640 1 -- 172.21.15.123:0/3572342532 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fadac156180 con 0x7fadac160310 2024-09-16T15:47:15.301 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:15.296+0000 7fada97fa640 1 -- 172.21.15.123:0/3572342532 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fada400aab0 con 0x7fadac160310 2024-09-16T15:47:15.301 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:15.296+0000 7fadb313d640 1 -- 172.21.15.123:0/3572342532 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fadac1f88e0 con 0x7fadac160310 2024-09-16T15:47:15.302 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:15.296+0000 7fadb313d640 1 -- 172.21.15.123:0/3572342532 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fadac0766e0 con 0x7fadac160310 2024-09-16T15:47:15.302 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:15.296+0000 7fada97fa640 1 -- 172.21.15.123:0/3572342532 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 11) ==== 97386+0+0 (secure 0 0 0) 0x7fada400ac50 con 0x7fadac160310 2024-09-16T15:47:15.302 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:15.296+0000 7fada97fa640 1 --2- 172.21.15.123:0/3572342532 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fad8c073220 0x7fad8c0756e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:47:15.302 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:15.296+0000 7fada97fa640 1 -- 172.21.15.123:0/3572342532 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(70..70 src has 1..70) ==== 5709+0+0 (secure 0 0 0) 0x7fada4094d30 con 0x7fadac160310 2024-09-16T15:47:15.302 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:15.296+0000 7fadb0ed9640 1 --2- 172.21.15.123:0/3572342532 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fad8c073220 0x7fad8c0756e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:47:15.302 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:15.296+0000 7fadb0ed9640 1 --2- 172.21.15.123:0/3572342532 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fad8c073220 0x7fad8c0756e0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7fad9c0af000 tx=0x7fad9c01b060 comp rx=0 tx=0).ready entity=mgr.4135 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:47:15.304 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:15.300+0000 7fada97fa640 1 -- 172.21.15.123:0/3572342532 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7fada405faf0 con 0x7fadac160310 2024-09-16T15:47:15.518 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:15.512+0000 7fadb313d640 1 -- 172.21.15.123:0/3572342532 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "fs flag set", "flag_name": "enable_multiple", "val": "true", "yes_i_really_mean_it": true} v 0) -- 0x7fadac070470 con 0x7fadac160310 2024-09-16T15:47:16.412 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:16.408+0000 7fada97fa640 1 -- 172.21.15.123:0/3572342532 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "fs flag set", "flag_name": "enable_multiple", "val": "true", "yes_i_really_mean_it": true}]=0 v45) ==== 136+0+0 (secure 0 0 0) 0x7fada405f3f0 con 0x7fadac160310 2024-09-16T15:47:16.415 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:16.412+0000 7fadb313d640 1 -- 172.21.15.123:0/3572342532 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fad8c073220 msgr2=0x7fad8c0756e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:47:16.415 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:16.412+0000 7fadb313d640 1 --2- 172.21.15.123:0/3572342532 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fad8c073220 0x7fad8c0756e0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7fad9c0af000 tx=0x7fad9c01b060 comp rx=0 tx=0).stop 2024-09-16T15:47:16.415 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:16.412+0000 7fadb313d640 1 -- 172.21.15.123:0/3572342532 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fadac160310 msgr2=0x7fadac14af50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:47:16.415 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:16.412+0000 7fadb313d640 1 --2- 172.21.15.123:0/3572342532 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fadac160310 0x7fadac14af50 secure :-1 s=READY pgs=822 cs=0 l=1 rev1=1 crypto rx=0x7fada400c7c0 tx=0x7fada400cc80 comp rx=0 tx=0).stop 2024-09-16T15:47:16.415 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:16.412+0000 7fadb313d640 1 -- 172.21.15.123:0/3572342532 shutdown_connections 2024-09-16T15:47:16.416 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:16.412+0000 7fadb313d640 1 --2- 172.21.15.123:0/3572342532 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fad8c073220 0x7fad8c0756e0 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:16.416 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:16.412+0000 7fadb313d640 1 --2- 172.21.15.123:0/3572342532 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fadac160310 0x7fadac14af50 unknown :-1 s=CLOSED pgs=822 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:16.416 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:16.412+0000 7fadb313d640 1 --2- 172.21.15.123:0/3572342532 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fadac15f690 0x7fadac14aa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:16.416 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:16.412+0000 7fadb313d640 1 --2- 172.21.15.123:0/3572342532 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fadac15bc40 0x7fadac146070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:16.416 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:16.412+0000 7fadb313d640 1 -- 172.21.15.123:0/3572342532 >> 172.21.15.123:0/3572342532 conn(0x7fadac0b1630 msgr2=0x7fadac158110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:47:16.416 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:16.412+0000 7fadb313d640 1 -- 172.21.15.123:0/3572342532 shutdown_connections 2024-09-16T15:47:16.416 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:16.412+0000 7fadb313d640 1 -- 172.21.15.123:0/3572342532 wait complete. 2024-09-16T15:47:16.438 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2024-09-16T15:47:16.569 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:16.564+0000 7f28ff501640 1 -- 172.21.15.123:0/3331970618 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f28f80b23b0 msgr2=0x7f28f80d1b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:47:16.569 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:16.564+0000 7f28ff501640 1 --2- 172.21.15.123:0/3331970618 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f28f80b23b0 0x7f28f80d1b50 secure :-1 s=READY pgs=823 cs=0 l=1 rev1=1 crypto rx=0x7f28f80cc240 tx=0x7f28ec01c830 comp rx=0 tx=0).stop 2024-09-16T15:47:16.569 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:16.564+0000 7f28ff501640 1 -- 172.21.15.123:0/3331970618 shutdown_connections 2024-09-16T15:47:16.569 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:16.564+0000 7f28ff501640 1 --2- 172.21.15.123:0/3331970618 >> v2:172.21.15.184:3300/0 conn(0x7f28f80d2090 0x7f28f80aebe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:16.569 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:16.564+0000 7f28ff501640 1 --2- 172.21.15.123:0/3331970618 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f28f80b23b0 0x7f28f80d1b50 unknown :-1 s=CLOSED pgs=823 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:16.570 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:16.564+0000 7f28ff501640 1 -- 172.21.15.123:0/3331970618 >> 172.21.15.123:0/3331970618 conn(0x7f28f809aa40 msgr2=0x7f28f809ae50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:47:16.570 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:16.564+0000 7f28ff501640 1 -- 172.21.15.123:0/3331970618 shutdown_connections 2024-09-16T15:47:16.570 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:16.564+0000 7f28ff501640 1 -- 172.21.15.123:0/3331970618 wait complete. 2024-09-16T15:47:16.572 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:16.564+0000 7f28ff501640 1 Processor -- start 2024-09-16T15:47:16.572 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:16.564+0000 7f28ff501640 1 -- start start 2024-09-16T15:47:16.572 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:16.568+0000 7f28ff501640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f28f80b23b0 0x7f28f80c5540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:47:16.572 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:16.568+0000 7f28ff501640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f28f80d2090 0x7f28f81517f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:47:16.572 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:16.568+0000 7f28ff501640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f28f815bd20 0x7f28f8156170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:47:16.572 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:16.568+0000 7f28ff501640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f28f8168a60 con 0x7f28f815bd20 2024-09-16T15:47:16.573 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:16.568+0000 7f28ff501640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f28f80afed0 con 0x7f28f80b23b0 2024-09-16T15:47:16.573 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:16.568+0000 7f28ff501640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f28f8168be0 con 0x7f28f80d2090 2024-09-16T15:47:16.573 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:16.568+0000 7f28fda9e640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f28f815bd20 0x7f28f8156170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:47:16.573 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:16.568+0000 7f28fda9e640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f28f815bd20 0x7f28f8156170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:38010/0 (socket says 172.21.15.123:38010) 2024-09-16T15:47:16.573 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:16.568+0000 7f28fda9e640 1 -- 172.21.15.123:0/2724273763 learned_addr learned my addr 172.21.15.123:0/2724273763 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:47:16.573 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:16.568+0000 7f28fd29d640 1 --2- 172.21.15.123:0/2724273763 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f28f80b23b0 0x7f28f80c5540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:47:16.573 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:16.568+0000 7f28fca9c640 1 --2- 172.21.15.123:0/2724273763 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f28f80d2090 0x7f28f81517f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:47:16.573 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:16.568+0000 7f28fda9e640 1 -- 172.21.15.123:0/2724273763 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f28f80d2090 msgr2=0x7f28f81517f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:47:16.573 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:16.568+0000 7f28fda9e640 1 --2- 172.21.15.123:0/2724273763 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f28f80d2090 0x7f28f81517f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:16.573 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:16.568+0000 7f28fda9e640 1 -- 172.21.15.123:0/2724273763 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f28f80b23b0 msgr2=0x7f28f80c5540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:47:16.573 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:16.568+0000 7f28fda9e640 1 --2- 172.21.15.123:0/2724273763 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f28f80b23b0 0x7f28f80c5540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:16.573 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:16.568+0000 7f28fda9e640 1 -- 172.21.15.123:0/2724273763 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f28f80c7320 con 0x7f28f815bd20 2024-09-16T15:47:16.574 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:16.568+0000 7f28fca9c640 1 --2- 172.21.15.123:0/2724273763 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f28f80d2090 0x7f28f81517f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T15:47:16.574 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:16.568+0000 7f28fda9e640 1 --2- 172.21.15.123:0/2724273763 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f28f815bd20 0x7f28f8156170 secure :-1 s=READY pgs=824 cs=0 l=1 rev1=1 crypto rx=0x7f28e800d750 tx=0x7f28e800dc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:47:16.574 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:16.568+0000 7f28f5ffb640 1 -- 172.21.15.123:0/2724273763 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28e8017070 con 0x7f28f815bd20 2024-09-16T15:47:16.574 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:16.568+0000 7f28f5ffb640 1 -- 172.21.15.123:0/2724273763 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f28e8009d20 con 0x7f28f815bd20 2024-09-16T15:47:16.574 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:16.568+0000 7f28ff501640 1 -- 172.21.15.123:0/2724273763 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f28f80c75a0 con 0x7f28f815bd20 2024-09-16T15:47:16.574 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:16.568+0000 7f28f5ffb640 1 -- 172.21.15.123:0/2724273763 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28e800de60 con 0x7f28f815bd20 2024-09-16T15:47:16.574 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:16.568+0000 7f28ff501640 1 -- 172.21.15.123:0/2724273763 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f28f80c7b60 con 0x7f28f815bd20 2024-09-16T15:47:16.574 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:16.568+0000 7f28ff501640 1 -- 172.21.15.123:0/2724273763 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f28f806d6a0 con 0x7f28f815bd20 2024-09-16T15:47:16.575 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:16.568+0000 7f28f5ffb640 1 -- 172.21.15.123:0/2724273763 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 11) ==== 97386+0+0 (secure 0 0 0) 0x7f28e8005280 con 0x7f28f815bd20 2024-09-16T15:47:16.575 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:16.568+0000 7f28f5ffb640 1 --2- 172.21.15.123:0/2724273763 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f28e4073220 0x7f28e40756e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:47:16.575 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:16.568+0000 7f28f5ffb640 1 -- 172.21.15.123:0/2724273763 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(70..70 src has 1..70) ==== 5709+0+0 (secure 0 0 0) 0x7f28e80945b0 con 0x7f28f815bd20 2024-09-16T15:47:16.575 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:16.568+0000 7f28fd29d640 1 --2- 172.21.15.123:0/2724273763 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f28e4073220 0x7f28e40756e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:47:16.575 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:16.572+0000 7f28fd29d640 1 --2- 172.21.15.123:0/2724273763 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f28e4073220 0x7f28e40756e0 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f28f0009f20 tx=0x7f28f00094d0 comp rx=0 tx=0).ready entity=mgr.4135 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:47:16.578 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:16.572+0000 7f28f5ffb640 1 -- 172.21.15.123:0/2724273763 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f28f80c7320 con 0x7f28f815bd20 2024-09-16T15:47:16.775 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:16.772+0000 7f28ff501640 1 -- 172.21.15.123:0/2724273763 --> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) -- 0x7f28f80b3fb0 con 0x7f28e4073220 2024-09-16T15:47:16.779 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:16.776+0000 7f28f5ffb640 1 -- 172.21.15.123:0/2724273763 <== mgr.4135 v2:172.21.15.184:6832/3114346423 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+41 (secure 0 0 0) 0x7f28f80b3fb0 con 0x7f28e4073220 2024-09-16T15:47:16.779 INFO:teuthology.orchestra.run.smithi123.stdout:[ 2024-09-16T15:47:16.779 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-09-16T15:47:16.779 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "cephfs" 2024-09-16T15:47:16.779 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-16T15:47:16.779 INFO:teuthology.orchestra.run.smithi123.stdout:] 2024-09-16T15:47:16.781 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:16.776+0000 7f28ff501640 1 -- 172.21.15.123:0/2724273763 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f28e4073220 msgr2=0x7f28e40756e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:47:16.782 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:16.776+0000 7f28ff501640 1 --2- 172.21.15.123:0/2724273763 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f28e4073220 0x7f28e40756e0 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f28f0009f20 tx=0x7f28f00094d0 comp rx=0 tx=0).stop 2024-09-16T15:47:16.782 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:16.776+0000 7f28ff501640 1 -- 172.21.15.123:0/2724273763 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f28f815bd20 msgr2=0x7f28f8156170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:47:16.782 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:16.776+0000 7f28ff501640 1 --2- 172.21.15.123:0/2724273763 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f28f815bd20 0x7f28f8156170 secure :-1 s=READY pgs=824 cs=0 l=1 rev1=1 crypto rx=0x7f28e800d750 tx=0x7f28e800dc10 comp rx=0 tx=0).stop 2024-09-16T15:47:16.782 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:16.776+0000 7f28ff501640 1 -- 172.21.15.123:0/2724273763 shutdown_connections 2024-09-16T15:47:16.782 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:16.776+0000 7f28ff501640 1 --2- 172.21.15.123:0/2724273763 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f28e4073220 0x7f28e40756e0 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:16.782 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:16.776+0000 7f28ff501640 1 --2- 172.21.15.123:0/2724273763 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f28f815bd20 0x7f28f8156170 unknown :-1 s=CLOSED pgs=824 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:16.782 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:16.776+0000 7f28ff501640 1 --2- 172.21.15.123:0/2724273763 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f28f80d2090 0x7f28f81517f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:16.782 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:16.776+0000 7f28ff501640 1 --2- 172.21.15.123:0/2724273763 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f28f80b23b0 0x7f28f80c5540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:16.782 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:16.776+0000 7f28ff501640 1 -- 172.21.15.123:0/2724273763 >> 172.21.15.123:0/2724273763 conn(0x7f28f809aa40 msgr2=0x7f28f8071920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:47:16.782 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:16.776+0000 7f28ff501640 1 -- 172.21.15.123:0/2724273763 shutdown_connections 2024-09-16T15:47:16.782 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:16.776+0000 7f28ff501640 1 -- 172.21.15.123:0/2724273763 wait complete. 2024-09-16T15:47:16.797 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-09-16T15:47:16.916 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:16.912+0000 7f03a0da3640 1 -- 172.21.15.123:0/3492347744 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0388003a40 msgr2=0x7f0388003e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:47:16.916 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:16.912+0000 7f03a0da3640 1 --2- 172.21.15.123:0/3492347744 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0388003a40 0x7f0388003e60 secure :-1 s=READY pgs=826 cs=0 l=1 rev1=1 crypto rx=0x7f038c001690 tx=0x7f038c01c730 comp rx=0 tx=0).stop 2024-09-16T15:47:16.916 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:16.912+0000 7f039ad9d640 1 -- 172.21.15.123:0/3492347744 reap_dead start 2024-09-16T15:47:16.916 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:16.912+0000 7f03a0da3640 1 -- 172.21.15.123:0/3492347744 shutdown_connections 2024-09-16T15:47:16.916 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:16.912+0000 7f03a0da3640 1 -- 172.21.15.123:0/3492347744 >> 172.21.15.123:0/3492347744 conn(0x7f039c0b1710 msgr2=0x7f039c0b1b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:47:16.916 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:16.912+0000 7f03a0da3640 1 -- 172.21.15.123:0/3492347744 shutdown_connections 2024-09-16T15:47:16.916 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:16.912+0000 7f03a0da3640 1 -- 172.21.15.123:0/3492347744 wait complete. 2024-09-16T15:47:16.918 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:16.912+0000 7f03a0da3640 1 Processor -- start 2024-09-16T15:47:16.919 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:16.912+0000 7f03a0da3640 1 -- start start 2024-09-16T15:47:16.919 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:16.912+0000 7f03a0da3640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f0388003a40 0x7f039c1fb820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:47:16.919 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:16.912+0000 7f03a0da3640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f039c1f82b0 0x7f039c161310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:47:16.919 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:16.912+0000 7f03a0da3640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f039c0cbaa0 0x7f039c073460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:47:16.919 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:16.912+0000 7f03a0da3640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f039c0cf3b0 con 0x7f039c0cbaa0 2024-09-16T15:47:16.919 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:16.912+0000 7f03a0da3640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f039c15cbb0 con 0x7f039c1f82b0 2024-09-16T15:47:16.919 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:16.912+0000 7f03a0da3640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f039c0cf530 con 0x7f0388003a40 2024-09-16T15:47:16.919 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:16.912+0000 7f039ad9d640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f039c0cbaa0 0x7f039c073460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:47:16.919 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:16.912+0000 7f039ad9d640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f039c0cbaa0 0x7f039c073460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:38026/0 (socket says 172.21.15.123:38026) 2024-09-16T15:47:16.919 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:16.912+0000 7f039ad9d640 1 -- 172.21.15.123:0/1645325086 learned_addr learned my addr 172.21.15.123:0/1645325086 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:47:16.919 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:16.912+0000 7f0393fff640 1 --2- 172.21.15.123:0/1645325086 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f039c1f82b0 0x7f039c161310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:47:16.919 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:16.912+0000 7f039a59c640 1 --2- 172.21.15.123:0/1645325086 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f0388003a40 0x7f039c1fb820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:47:16.920 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:16.912+0000 7f039ad9d640 1 -- 172.21.15.123:0/1645325086 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f0388003a40 msgr2=0x7f039c1fb820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:47:16.920 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:16.912+0000 7f039ad9d640 1 --2- 172.21.15.123:0/1645325086 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f0388003a40 0x7f039c1fb820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:16.920 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:16.912+0000 7f039ad9d640 1 -- 172.21.15.123:0/1645325086 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f039c1f82b0 msgr2=0x7f039c161310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:47:16.920 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:16.912+0000 7f039ad9d640 1 --2- 172.21.15.123:0/1645325086 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f039c1f82b0 0x7f039c161310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:16.920 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:16.912+0000 7f039ad9d640 1 -- 172.21.15.123:0/1645325086 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f039c1f7840 con 0x7f039c0cbaa0 2024-09-16T15:47:16.920 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:16.912+0000 7f039ad9d640 1 --2- 172.21.15.123:0/1645325086 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f039c0cbaa0 0x7f039c073460 secure :-1 s=READY pgs=827 cs=0 l=1 rev1=1 crypto rx=0x7f0394001340 tx=0x7f039400dd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:47:16.920 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:16.912+0000 7f0392ffd640 1 -- 172.21.15.123:0/1645325086 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f039400b960 con 0x7f039c0cbaa0 2024-09-16T15:47:16.920 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:16.912+0000 7f0392ffd640 1 -- 172.21.15.123:0/1645325086 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f0394017070 con 0x7f039c0cbaa0 2024-09-16T15:47:16.920 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:16.912+0000 7f03a0da3640 1 -- 172.21.15.123:0/1645325086 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f039c161b40 con 0x7f039c0cbaa0 2024-09-16T15:47:16.920 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:16.912+0000 7f0392ffd640 1 -- 172.21.15.123:0/1645325086 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f039400bda0 con 0x7f039c0cbaa0 2024-09-16T15:47:16.920 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:16.912+0000 7f03a0da3640 1 -- 172.21.15.123:0/1645325086 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f039c1f7d90 con 0x7f039c0cbaa0 2024-09-16T15:47:16.921 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:16.916+0000 7f03a0da3640 1 -- 172.21.15.123:0/1645325086 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f039c0cc550 con 0x7f039c0cbaa0 2024-09-16T15:47:16.921 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:16.916+0000 7f0392ffd640 1 -- 172.21.15.123:0/1645325086 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 11) ==== 97386+0+0 (secure 0 0 0) 0x7f039401c460 con 0x7f039c0cbaa0 2024-09-16T15:47:16.921 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:16.916+0000 7f0392ffd640 1 --2- 172.21.15.123:0/1645325086 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f037c073250 0x7f037c075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:47:16.922 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:16.916+0000 7f0392ffd640 1 -- 172.21.15.123:0/1645325086 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(70..70 src has 1..70) ==== 5709+0+0 (secure 0 0 0) 0x7f0394094690 con 0x7f039c0cbaa0 2024-09-16T15:47:16.922 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:16.916+0000 7f039a59c640 1 --2- 172.21.15.123:0/1645325086 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f037c073250 0x7f037c075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:47:16.922 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:16.916+0000 7f039a59c640 1 --2- 172.21.15.123:0/1645325086 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f037c073250 0x7f037c075710 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f038c001660 tx=0x7f038c0194d0 comp rx=0 tx=0).ready entity=mgr.4135 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:47:16.924 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:16.920+0000 7f0392ffd640 1 -- 172.21.15.123:0/1645325086 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f039405f3c0 con 0x7f039c0cbaa0 2024-09-16T15:47:17.082 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:17.076+0000 7f03a0da3640 1 -- 172.21.15.123:0/1645325086 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) -- 0x7f039c1522c0 con 0x7f039c0cbaa0 2024-09-16T15:47:17.082 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:17.076+0000 7f0392ffd640 1 -- 172.21.15.123:0/1645325086 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v34)=0 v34) ==== 122+0+0 (secure 0 0 0) 0x7f039405ecc0 con 0x7f039c0cbaa0 2024-09-16T15:47:17.085 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:17.080+0000 7f03a0da3640 1 -- 172.21.15.123:0/1645325086 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f037c073250 msgr2=0x7f037c075710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:47:17.085 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:17.080+0000 7f03a0da3640 1 --2- 172.21.15.123:0/1645325086 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f037c073250 0x7f037c075710 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f038c001660 tx=0x7f038c0194d0 comp rx=0 tx=0).stop 2024-09-16T15:47:17.085 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:17.080+0000 7f03a0da3640 1 -- 172.21.15.123:0/1645325086 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f039c0cbaa0 msgr2=0x7f039c073460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:47:17.085 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:17.080+0000 7f03a0da3640 1 --2- 172.21.15.123:0/1645325086 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f039c0cbaa0 0x7f039c073460 secure :-1 s=READY pgs=827 cs=0 l=1 rev1=1 crypto rx=0x7f0394001340 tx=0x7f039400dd70 comp rx=0 tx=0).stop 2024-09-16T15:47:17.085 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:17.080+0000 7f03a0da3640 1 -- 172.21.15.123:0/1645325086 shutdown_connections 2024-09-16T15:47:17.086 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:17.080+0000 7f03a0da3640 1 --2- 172.21.15.123:0/1645325086 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f037c073250 0x7f037c075710 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:17.086 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:17.080+0000 7f03a0da3640 1 --2- 172.21.15.123:0/1645325086 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f039c0cbaa0 0x7f039c073460 unknown :-1 s=CLOSED pgs=827 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:17.086 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:17.080+0000 7f03a0da3640 1 --2- 172.21.15.123:0/1645325086 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f039c1f82b0 0x7f039c161310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:17.086 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:17.080+0000 7f03a0da3640 1 --2- 172.21.15.123:0/1645325086 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f0388003a40 0x7f039c1fb820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:17.086 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:17.080+0000 7f03a0da3640 1 -- 172.21.15.123:0/1645325086 >> 172.21.15.123:0/1645325086 conn(0x7f039c0b1710 msgr2=0x7f039c0af680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:47:17.086 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:17.080+0000 7f03a0da3640 1 -- 172.21.15.123:0/1645325086 shutdown_connections 2024-09-16T15:47:17.086 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:17.080+0000 7f03a0da3640 1 -- 172.21.15.123:0/1645325086 wait complete. 2024-09-16T15:47:17.100 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2024-09-16T15:47:17.100 DEBUG:teuthology.orchestra.run.smithi123:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-16T15:47:17.105 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-09-16T15:47:17.276 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:17.272+0000 7f9640f99640 1 -- 172.21.15.123:0/2064714403 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f963c06e470 msgr2=0x7f963c0cf7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:47:17.277 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:17.272+0000 7f9640f99640 1 --2- 172.21.15.123:0/2064714403 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f963c06e470 0x7f963c0cf7a0 secure :-1 s=READY pgs=828 cs=0 l=1 rev1=1 crypto rx=0x7f963c074440 tx=0x7f963401ccc0 comp rx=0 tx=0).stop 2024-09-16T15:47:17.277 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:17.272+0000 7f9640f99640 1 -- 172.21.15.123:0/2064714403 shutdown_connections 2024-09-16T15:47:17.277 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:17.272+0000 7f9640f99640 1 --2- 172.21.15.123:0/2064714403 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f963c06e470 0x7f963c0cf7a0 unknown :-1 s=CLOSED pgs=828 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:17.277 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:17.272+0000 7f9640f99640 1 --2- 172.21.15.123:0/2064714403 >> v2:172.21.15.184:3300/0 conn(0x7f963c15bfd0 0x7f963c06df30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:17.277 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:17.272+0000 7f9640f99640 1 -- 172.21.15.123:0/2064714403 >> 172.21.15.123:0/2064714403 conn(0x7f963c0b2c20 msgr2=0x7f963c0b3030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:47:17.277 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:17.272+0000 7f9640f99640 1 -- 172.21.15.123:0/2064714403 shutdown_connections 2024-09-16T15:47:17.277 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:17.272+0000 7f9640f99640 1 -- 172.21.15.123:0/2064714403 wait complete. 2024-09-16T15:47:17.278 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:17.272+0000 7f9640f99640 1 Processor -- start 2024-09-16T15:47:17.278 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:17.272+0000 7f9640f99640 1 -- start start 2024-09-16T15:47:17.280 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:17.272+0000 7f9640f99640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f963c06e470 0x7f963c141e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:47:17.280 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:17.272+0000 7f9640f99640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f963c15bbf0 0x7f963c141720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:47:17.280 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:17.272+0000 7f9640f99640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f963c15bfd0 0x7f963c147b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:47:17.280 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:17.272+0000 7f9640f99640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f963c168b60 con 0x7f963c15bbf0 2024-09-16T15:47:17.280 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:17.272+0000 7f9640f99640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f963c1689e0 con 0x7f963c06e470 2024-09-16T15:47:17.280 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:17.272+0000 7f9640f99640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f963c168ce0 con 0x7f963c15bfd0 2024-09-16T15:47:17.280 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:17.276+0000 7f9631d9b640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f963c15bbf0 0x7f963c141720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:47:17.280 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:17.276+0000 7f9631d9b640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f963c15bbf0 0x7f963c141720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:38054/0 (socket says 172.21.15.123:38054) 2024-09-16T15:47:17.280 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:17.276+0000 7f9631d9b640 1 -- 172.21.15.123:0/3071553217 learned_addr learned my addr 172.21.15.123:0/3071553217 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:47:17.280 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:17.276+0000 7f963ad9d640 1 --2- 172.21.15.123:0/3071553217 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f963c15bfd0 0x7f963c147b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:47:17.281 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:17.276+0000 7f963a59c640 1 --2- 172.21.15.123:0/3071553217 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f963c06e470 0x7f963c141e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:47:17.281 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:17.276+0000 7f9631d9b640 1 -- 172.21.15.123:0/3071553217 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f963c15bfd0 msgr2=0x7f963c147b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:47:17.281 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:17.276+0000 7f9631d9b640 1 --2- 172.21.15.123:0/3071553217 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f963c15bfd0 0x7f963c147b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:17.281 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:17.276+0000 7f9631d9b640 1 -- 172.21.15.123:0/3071553217 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f963c06e470 msgr2=0x7f963c141e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:47:17.281 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:17.276+0000 7f9631d9b640 1 --2- 172.21.15.123:0/3071553217 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f963c06e470 0x7f963c141e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:17.281 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:17.276+0000 7f9631d9b640 1 -- 172.21.15.123:0/3071553217 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f963c141c60 con 0x7f963c15bbf0 2024-09-16T15:47:17.281 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:17.276+0000 7f9631d9b640 1 --2- 172.21.15.123:0/3071553217 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f963c15bbf0 0x7f963c141720 secure :-1 s=READY pgs=829 cs=0 l=1 rev1=1 crypto rx=0x7f962c00c9f0 tx=0x7f962c00ceb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:47:17.281 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:17.276+0000 7f96337fe640 1 -- 172.21.15.123:0/3071553217 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f962c007b50 con 0x7f963c15bbf0 2024-09-16T15:47:17.281 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:17.276+0000 7f96337fe640 1 -- 172.21.15.123:0/3071553217 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f962c007de0 con 0x7f963c15bbf0 2024-09-16T15:47:17.281 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:17.276+0000 7f96337fe640 1 -- 172.21.15.123:0/3071553217 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f962c00f6b0 con 0x7f963c15bbf0 2024-09-16T15:47:17.281 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:17.276+0000 7f9640f99640 1 -- 172.21.15.123:0/3071553217 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f963c148ae0 con 0x7f963c15bbf0 2024-09-16T15:47:17.282 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:17.276+0000 7f9640f99640 1 -- 172.21.15.123:0/3071553217 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f963c1490a0 con 0x7f963c15bbf0 2024-09-16T15:47:17.282 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:17.276+0000 7f9640f99640 1 -- 172.21.15.123:0/3071553217 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f963c0cf7a0 con 0x7f963c15bbf0 2024-09-16T15:47:17.282 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:17.276+0000 7f96337fe640 1 -- 172.21.15.123:0/3071553217 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 11) ==== 97386+0+0 (secure 0 0 0) 0x7f962c00f850 con 0x7f963c15bbf0 2024-09-16T15:47:17.283 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:17.276+0000 7f96337fe640 1 --2- 172.21.15.123:0/3071553217 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f961c073250 0x7f961c075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:47:17.283 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:17.276+0000 7f96337fe640 1 -- 172.21.15.123:0/3071553217 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(70..70 src has 1..70) ==== 5709+0+0 (secure 0 0 0) 0x7f962c094ab0 con 0x7f963c15bbf0 2024-09-16T15:47:17.283 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:17.276+0000 7f963a59c640 1 --2- 172.21.15.123:0/3071553217 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f961c073250 0x7f961c075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:47:17.283 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:17.280+0000 7f963a59c640 1 --2- 172.21.15.123:0/3071553217 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f961c073250 0x7f961c075710 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f96240015f0 tx=0x7f9624009040 comp rx=0 tx=0).ready entity=mgr.4135 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:47:17.286 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:17.280+0000 7f96337fe640 1 -- 172.21.15.123:0/3071553217 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f962c014030 con 0x7f963c15bbf0 2024-09-16T15:47:17.473 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:17.468+0000 7f9640f99640 1 -- 172.21.15.123:0/3071553217 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f963c06ce90 con 0x7f963c15bbf0 2024-09-16T15:47:17.474 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:17.468+0000 7f96337fe640 1 -- 172.21.15.123:0/3071553217 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v70) ==== 80+0+0 (secure 0 0 0) 0x7f962c05f870 con 0x7f963c15bbf0 2024-09-16T15:47:17.474 INFO:teuthology.orchestra.run.smithi123.stderr:listed 0 entries 2024-09-16T15:47:17.477 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:17.472+0000 7f9640f99640 1 -- 172.21.15.123:0/3071553217 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f961c073250 msgr2=0x7f961c075710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:47:17.477 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:17.472+0000 7f9640f99640 1 --2- 172.21.15.123:0/3071553217 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f961c073250 0x7f961c075710 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f96240015f0 tx=0x7f9624009040 comp rx=0 tx=0).stop 2024-09-16T15:47:17.477 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:17.472+0000 7f9640f99640 1 -- 172.21.15.123:0/3071553217 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f963c15bbf0 msgr2=0x7f963c141720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:47:17.477 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:17.472+0000 7f9640f99640 1 --2- 172.21.15.123:0/3071553217 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f963c15bbf0 0x7f963c141720 secure :-1 s=READY pgs=829 cs=0 l=1 rev1=1 crypto rx=0x7f962c00c9f0 tx=0x7f962c00ceb0 comp rx=0 tx=0).stop 2024-09-16T15:47:17.477 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:17.472+0000 7f9640f99640 1 -- 172.21.15.123:0/3071553217 shutdown_connections 2024-09-16T15:47:17.477 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:17.472+0000 7f9640f99640 1 --2- 172.21.15.123:0/3071553217 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f961c073250 0x7f961c075710 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:17.478 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:17.472+0000 7f9640f99640 1 --2- 172.21.15.123:0/3071553217 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f963c15bfd0 0x7f963c147b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:17.478 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:17.472+0000 7f9640f99640 1 --2- 172.21.15.123:0/3071553217 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f963c15bbf0 0x7f963c141720 unknown :-1 s=CLOSED pgs=829 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:17.478 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:17.472+0000 7f9640f99640 1 --2- 172.21.15.123:0/3071553217 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f963c06e470 0x7f963c141e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:17.478 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:17.472+0000 7f9640f99640 1 -- 172.21.15.123:0/3071553217 >> 172.21.15.123:0/3071553217 conn(0x7f963c0b2c20 msgr2=0x7f963c15a970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:47:17.478 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:17.472+0000 7f9640f99640 1 -- 172.21.15.123:0/3071553217 shutdown_connections 2024-09-16T15:47:17.478 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:17.472+0000 7f9640f99640 1 -- 172.21.15.123:0/3071553217 wait complete. 2024-09-16T15:47:17.493 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2024-09-16T15:47:17.494 DEBUG:teuthology.orchestra.run.smithi123:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-16T15:47:17.500 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-09-16T15:47:17.669 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:17.664+0000 7f34adfcc640 1 -- 172.21.15.123:0/894614376 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f34a8074600 msgr2=0x7f34a806e750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:47:17.669 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:17.664+0000 7f34adfcc640 1 --2- 172.21.15.123:0/894614376 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f34a8074600 0x7f34a806e750 secure :-1 s=READY pgs=831 cs=0 l=1 rev1=1 crypto rx=0x7f34a0009bc0 tx=0x7f34a001cae0 comp rx=0 tx=0).stop 2024-09-16T15:47:17.670 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:17.664+0000 7f34adfcc640 1 -- 172.21.15.123:0/894614376 shutdown_connections 2024-09-16T15:47:17.670 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:17.664+0000 7f34adfcc640 1 --2- 172.21.15.123:0/894614376 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f34a8074600 0x7f34a806e750 unknown :-1 s=CLOSED pgs=831 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:17.670 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:17.664+0000 7f34adfcc640 1 --2- 172.21.15.123:0/894614376 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f34a815bca0 0x7f34a80d0630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:17.670 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:17.664+0000 7f34adfcc640 1 -- 172.21.15.123:0/894614376 >> 172.21.15.123:0/894614376 conn(0x7f34a80cfb10 msgr2=0x7f34a80cff20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:47:17.670 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:17.664+0000 7f34adfcc640 1 -- 172.21.15.123:0/894614376 shutdown_connections 2024-09-16T15:47:17.670 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:17.664+0000 7f34adfcc640 1 -- 172.21.15.123:0/894614376 wait complete. 2024-09-16T15:47:17.670 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:17.664+0000 7f34adfcc640 1 Processor -- start 2024-09-16T15:47:17.671 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:17.664+0000 7f34adfcc640 1 -- start start 2024-09-16T15:47:17.673 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:17.668+0000 7f34adfcc640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f34a80726c0 0x7f34a8153e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:47:17.673 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:17.668+0000 7f34adfcc640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f34a8074600 0x7f34a815eba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:47:17.673 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:17.668+0000 7f34adfcc640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f34a815bca0 0x7f34a8155c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:47:17.673 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:17.668+0000 7f34adfcc640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f34a8168a50 con 0x7f34a815bca0 2024-09-16T15:47:17.673 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:17.668+0000 7f34adfcc640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f34a806fa60 con 0x7f34a8074600 2024-09-16T15:47:17.673 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:17.668+0000 7f34adfcc640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f34a8168bd0 con 0x7f34a80726c0 2024-09-16T15:47:17.673 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:17.668+0000 7f34a7fff640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f34a815bca0 0x7f34a8155c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:47:17.673 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:17.668+0000 7f34a7fff640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f34a815bca0 0x7f34a8155c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:38070/0 (socket says 172.21.15.123:38070) 2024-09-16T15:47:17.673 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:17.668+0000 7f34a7fff640 1 -- 172.21.15.123:0/975232879 learned_addr learned my addr 172.21.15.123:0/975232879 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:47:17.673 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:17.668+0000 7f349ffff640 1 --2- 172.21.15.123:0/975232879 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f34a8074600 0x7f34a815eba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:47:17.674 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:17.668+0000 7f34a7fff640 1 -- 172.21.15.123:0/975232879 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f34a80726c0 msgr2=0x7f34a8153e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:47:17.674 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:17.668+0000 7f34a77fe640 1 --2- 172.21.15.123:0/975232879 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f34a80726c0 0x7f34a8153e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:47:17.674 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:17.668+0000 7f34a7fff640 1 --2- 172.21.15.123:0/975232879 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f34a80726c0 0x7f34a8153e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:17.674 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:17.668+0000 7f34a7fff640 1 -- 172.21.15.123:0/975232879 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f34a8074600 msgr2=0x7f34a815eba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:47:17.674 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:17.668+0000 7f34a7fff640 1 --2- 172.21.15.123:0/975232879 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f34a8074600 0x7f34a815eba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:17.674 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:17.668+0000 7f34a7fff640 1 -- 172.21.15.123:0/975232879 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f34a8154360 con 0x7f34a815bca0 2024-09-16T15:47:17.674 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:17.668+0000 7f349ffff640 1 --2- 172.21.15.123:0/975232879 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f34a8074600 0x7f34a815eba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T15:47:17.674 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:17.668+0000 7f34a7fff640 1 --2- 172.21.15.123:0/975232879 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f34a815bca0 0x7f34a8155c00 secure :-1 s=READY pgs=832 cs=0 l=1 rev1=1 crypto rx=0x7f34a001a710 tx=0x7f34a0009300 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:47:17.674 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:17.668+0000 7f34a4ff9640 1 -- 172.21.15.123:0/975232879 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34a00ba020 con 0x7f34a815bca0 2024-09-16T15:47:17.674 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:17.668+0000 7f34a4ff9640 1 -- 172.21.15.123:0/975232879 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f34a00a5e50 con 0x7f34a815bca0 2024-09-16T15:47:17.675 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:17.668+0000 7f34adfcc640 1 -- 172.21.15.123:0/975232879 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f34a81fb260 con 0x7f34a815bca0 2024-09-16T15:47:17.675 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:17.668+0000 7f34a4ff9640 1 -- 172.21.15.123:0/975232879 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34a00b2040 con 0x7f34a815bca0 2024-09-16T15:47:17.675 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:17.668+0000 7f34adfcc640 1 -- 172.21.15.123:0/975232879 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f34a81fb820 con 0x7f34a815bca0 2024-09-16T15:47:17.675 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:17.668+0000 7f34adfcc640 1 -- 172.21.15.123:0/975232879 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f34a806e750 con 0x7f34a815bca0 2024-09-16T15:47:17.676 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:17.668+0000 7f34a4ff9640 1 -- 172.21.15.123:0/975232879 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 11) ==== 97386+0+0 (secure 0 0 0) 0x7f34a00a5b30 con 0x7f34a815bca0 2024-09-16T15:47:17.676 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:17.668+0000 7f34a4ff9640 1 --2- 172.21.15.123:0/975232879 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f3488073220 0x7f34880756e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:47:17.676 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:17.668+0000 7f34a4ff9640 1 -- 172.21.15.123:0/975232879 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(70..70 src has 1..70) ==== 5709+0+0 (secure 0 0 0) 0x7f34a012e7f0 con 0x7f34a815bca0 2024-09-16T15:47:17.676 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:17.668+0000 7f34a77fe640 1 --2- 172.21.15.123:0/975232879 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f3488073220 0x7f34880756e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:47:17.676 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:17.672+0000 7f34a77fe640 1 --2- 172.21.15.123:0/975232879 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f3488073220 0x7f34880756e0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f34980058d0 tx=0x7f3498005bf0 comp rx=0 tx=0).ready entity=mgr.4135 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:47:17.678 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:17.676+0000 7f34a4ff9640 1 -- 172.21.15.123:0/975232879 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f34a00c4020 con 0x7f34a815bca0 2024-09-16T15:47:17.861 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:17.856+0000 7f34adfcc640 1 -- 172.21.15.123:0/975232879 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f34a815f630 con 0x7f34a815bca0 2024-09-16T15:47:17.862 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:17.856+0000 7f34a4ff9640 1 -- 172.21.15.123:0/975232879 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v70) ==== 80+0+0 (secure 0 0 0) 0x7f34a8154360 con 0x7f34a815bca0 2024-09-16T15:47:17.862 INFO:teuthology.orchestra.run.smithi123.stderr:listed 0 entries 2024-09-16T15:47:17.865 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:17.860+0000 7f34adfcc640 1 -- 172.21.15.123:0/975232879 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f3488073220 msgr2=0x7f34880756e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:47:17.865 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:17.860+0000 7f34adfcc640 1 --2- 172.21.15.123:0/975232879 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f3488073220 0x7f34880756e0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f34980058d0 tx=0x7f3498005bf0 comp rx=0 tx=0).stop 2024-09-16T15:47:17.865 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:17.860+0000 7f34adfcc640 1 -- 172.21.15.123:0/975232879 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f34a815bca0 msgr2=0x7f34a8155c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:47:17.865 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:17.860+0000 7f34adfcc640 1 --2- 172.21.15.123:0/975232879 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f34a815bca0 0x7f34a8155c00 secure :-1 s=READY pgs=832 cs=0 l=1 rev1=1 crypto rx=0x7f34a001a710 tx=0x7f34a0009300 comp rx=0 tx=0).stop 2024-09-16T15:47:17.865 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:17.860+0000 7f34adfcc640 1 -- 172.21.15.123:0/975232879 shutdown_connections 2024-09-16T15:47:17.865 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:17.860+0000 7f34adfcc640 1 --2- 172.21.15.123:0/975232879 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f3488073220 0x7f34880756e0 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:17.865 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:17.860+0000 7f34adfcc640 1 --2- 172.21.15.123:0/975232879 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f34a815bca0 0x7f34a8155c00 unknown :-1 s=CLOSED pgs=832 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:17.865 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:17.860+0000 7f34adfcc640 1 --2- 172.21.15.123:0/975232879 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f34a8074600 0x7f34a815eba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:17.865 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:17.860+0000 7f34adfcc640 1 --2- 172.21.15.123:0/975232879 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f34a80726c0 0x7f34a8153e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:17.865 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:17.860+0000 7f34adfcc640 1 -- 172.21.15.123:0/975232879 >> 172.21.15.123:0/975232879 conn(0x7f34a80cfb10 msgr2=0x7f34a80cd680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:47:17.866 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:17.860+0000 7f34adfcc640 1 -- 172.21.15.123:0/975232879 shutdown_connections 2024-09-16T15:47:17.866 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:17.860+0000 7f34adfcc640 1 -- 172.21.15.123:0/975232879 wait complete. 2024-09-16T15:47:17.881 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi123.front.sepia.ceph.com... 2024-09-16T15:47:17.882 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:47:17.882 DEBUG:teuthology.orchestra.run.smithi123:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2024-09-16T15:47:17.893 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi123.stderr:ceph-fuse[42891]: fuse finished with error 0 and tester_r 0 2024-09-16T15:47:17.893 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T15:47:23.995 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi123.front.sepia.ceph.com 2024-09-16T15:47:23.996 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:47:23.996 DEBUG:teuthology.orchestra.run.smithi123:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-09-16T15:47:24.003 DEBUG:teuthology.orchestra.run.smithi184:> set -ex 2024-09-16T15:47:24.003 DEBUG:teuthology.orchestra.run.smithi184:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-16T15:47:24.010 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-09-16T15:47:24.184 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:24.180+0000 7f28146c3640 1 -- 172.21.15.123:0/4203860823 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f280c158120 msgr2=0x7f280c0ca320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:47:24.184 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:24.180+0000 7f28146c3640 1 --2- 172.21.15.123:0/4203860823 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f280c158120 0x7f280c0ca320 secure :-1 s=READY pgs=834 cs=0 l=1 rev1=1 crypto rx=0x7f27fc009c80 tx=0x7f27fc01cc80 comp rx=0 tx=0).stop 2024-09-16T15:47:24.184 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:24.180+0000 7f28146c3640 1 -- 172.21.15.123:0/4203860823 shutdown_connections 2024-09-16T15:47:24.184 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:24.180+0000 7f28146c3640 1 --2- 172.21.15.123:0/4203860823 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f280c0cf5a0 0x7f280c06eee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:24.184 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:24.180+0000 7f28146c3640 1 --2- 172.21.15.123:0/4203860823 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f280c158120 0x7f280c0ca320 unknown :-1 s=CLOSED pgs=834 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:24.185 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:24.180+0000 7f28146c3640 1 -- 172.21.15.123:0/4203860823 >> 172.21.15.123:0/4203860823 conn(0x7f280c0cfb10 msgr2=0x7f280c0cff20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:47:24.185 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:24.180+0000 7f28146c3640 1 -- 172.21.15.123:0/4203860823 shutdown_connections 2024-09-16T15:47:24.185 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:24.180+0000 7f28146c3640 1 -- 172.21.15.123:0/4203860823 wait complete. 2024-09-16T15:47:24.185 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:24.180+0000 7f28146c3640 1 Processor -- start 2024-09-16T15:47:24.185 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:24.180+0000 7f28146c3640 1 -- start start 2024-09-16T15:47:24.187 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:24.180+0000 7f28146c3640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f280c0cf5a0 0x7f280c0b29a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:47:24.187 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:24.180+0000 7f28146c3640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f280c158120 0x7f280c0b3310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:47:24.187 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:24.180+0000 7f28146c3640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f280c15ec40 0x7f280c0c47c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:47:24.187 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:24.180+0000 7f28146c3640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f280c15b770 con 0x7f280c15ec40 2024-09-16T15:47:24.187 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:24.180+0000 7f28146c3640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f280c15b5f0 con 0x7f280c158120 2024-09-16T15:47:24.188 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:24.180+0000 7f28146c3640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f280c1696f0 con 0x7f280c0cf5a0 2024-09-16T15:47:24.188 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:24.180+0000 7f2812c60640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f280c15ec40 0x7f280c0c47c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:47:24.188 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:24.180+0000 7f2812c60640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f280c15ec40 0x7f280c0c47c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:44778/0 (socket says 172.21.15.123:44778) 2024-09-16T15:47:24.188 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:24.180+0000 7f2812c60640 1 -- 172.21.15.123:0/626877786 learned_addr learned my addr 172.21.15.123:0/626877786 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:47:24.188 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:24.180+0000 7f281245f640 1 --2- 172.21.15.123:0/626877786 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f280c0cf5a0 0x7f280c0b29a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:47:24.188 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:24.180+0000 7f2811c5e640 1 --2- 172.21.15.123:0/626877786 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f280c158120 0x7f280c0b3310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:47:24.188 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:24.180+0000 7f2812c60640 1 -- 172.21.15.123:0/626877786 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f280c0cf5a0 msgr2=0x7f280c0b29a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:47:24.188 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:24.180+0000 7f2812c60640 1 --2- 172.21.15.123:0/626877786 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f280c0cf5a0 0x7f280c0b29a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:24.188 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:24.180+0000 7f2812c60640 1 -- 172.21.15.123:0/626877786 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f280c158120 msgr2=0x7f280c0b3310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:47:24.188 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:24.180+0000 7f2812c60640 1 --2- 172.21.15.123:0/626877786 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f280c158120 0x7f280c0b3310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:24.188 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:24.180+0000 7f2812c60640 1 -- 172.21.15.123:0/626877786 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f280c1f5f70 con 0x7f280c15ec40 2024-09-16T15:47:24.188 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:24.180+0000 7f2811c5e640 1 --2- 172.21.15.123:0/626877786 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f280c158120 0x7f280c0b3310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T15:47:24.188 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:24.184+0000 7f2812c60640 1 --2- 172.21.15.123:0/626877786 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f280c15ec40 0x7f280c0c47c0 secure :-1 s=READY pgs=835 cs=0 l=1 rev1=1 crypto rx=0x7f280400c390 tx=0x7f280400c850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:47:24.189 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:24.184+0000 7f280affd640 1 -- 172.21.15.123:0/626877786 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2804017070 con 0x7f280c15ec40 2024-09-16T15:47:24.189 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:24.184+0000 7f280affd640 1 -- 172.21.15.123:0/626877786 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f28040053c0 con 0x7f280c15ec40 2024-09-16T15:47:24.189 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:24.184+0000 7f28146c3640 1 -- 172.21.15.123:0/626877786 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f280c1f6230 con 0x7f280c15ec40 2024-09-16T15:47:24.189 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:24.184+0000 7f280affd640 1 -- 172.21.15.123:0/626877786 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2804003860 con 0x7f280c15ec40 2024-09-16T15:47:24.189 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:24.184+0000 7f28146c3640 1 -- 172.21.15.123:0/626877786 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f280c1f67f0 con 0x7f280c15ec40 2024-09-16T15:47:24.189 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:24.184+0000 7f28146c3640 1 -- 172.21.15.123:0/626877786 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f280c06eee0 con 0x7f280c15ec40 2024-09-16T15:47:24.190 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:24.184+0000 7f280affd640 1 -- 172.21.15.123:0/626877786 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 11) ==== 97386+0+0 (secure 0 0 0) 0x7f280401a070 con 0x7f280c15ec40 2024-09-16T15:47:24.190 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:24.184+0000 7f280affd640 1 --2- 172.21.15.123:0/626877786 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f27f8073250 0x7f27f8075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:47:24.190 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:24.184+0000 7f280affd640 1 -- 172.21.15.123:0/626877786 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(70..70 src has 1..70) ==== 5709+0+0 (secure 0 0 0) 0x7f2804093a30 con 0x7f280c15ec40 2024-09-16T15:47:24.190 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:24.184+0000 7f281245f640 1 --2- 172.21.15.123:0/626877786 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f27f8073250 0x7f27f8075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:47:24.190 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:24.184+0000 7f281245f640 1 --2- 172.21.15.123:0/626877786 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f27f8073250 0x7f27f8075710 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f27fc0af000 tx=0x7f27fc01b4c0 comp rx=0 tx=0).ready entity=mgr.4135 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:47:24.193 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:24.188+0000 7f280affd640 1 -- 172.21.15.123:0/626877786 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f2804021020 con 0x7f280c15ec40 2024-09-16T15:47:24.376 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:24.372+0000 7f28146c3640 1 -- 172.21.15.123:0/626877786 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f280c0d06c0 con 0x7f280c15ec40 2024-09-16T15:47:24.377 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:24.372+0000 7f280affd640 1 -- 172.21.15.123:0/626877786 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v70) ==== 80+0+0 (secure 0 0 0) 0x7f280405e860 con 0x7f280c15ec40 2024-09-16T15:47:24.377 INFO:teuthology.orchestra.run.smithi123.stderr:listed 0 entries 2024-09-16T15:47:24.380 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:24.376+0000 7f28146c3640 1 -- 172.21.15.123:0/626877786 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f27f8073250 msgr2=0x7f27f8075710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:47:24.380 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:24.376+0000 7f28146c3640 1 --2- 172.21.15.123:0/626877786 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f27f8073250 0x7f27f8075710 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f27fc0af000 tx=0x7f27fc01b4c0 comp rx=0 tx=0).stop 2024-09-16T15:47:24.380 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:24.376+0000 7f28146c3640 1 -- 172.21.15.123:0/626877786 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f280c15ec40 msgr2=0x7f280c0c47c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:47:24.380 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:24.376+0000 7f28146c3640 1 --2- 172.21.15.123:0/626877786 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f280c15ec40 0x7f280c0c47c0 secure :-1 s=READY pgs=835 cs=0 l=1 rev1=1 crypto rx=0x7f280400c390 tx=0x7f280400c850 comp rx=0 tx=0).stop 2024-09-16T15:47:24.380 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:24.376+0000 7f28146c3640 1 -- 172.21.15.123:0/626877786 shutdown_connections 2024-09-16T15:47:24.380 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:24.376+0000 7f28146c3640 1 --2- 172.21.15.123:0/626877786 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f27f8073250 0x7f27f8075710 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:24.380 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:24.376+0000 7f28146c3640 1 --2- 172.21.15.123:0/626877786 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f280c15ec40 0x7f280c0c47c0 unknown :-1 s=CLOSED pgs=835 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:24.381 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:24.376+0000 7f28146c3640 1 --2- 172.21.15.123:0/626877786 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f280c158120 0x7f280c0b3310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:24.381 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:24.376+0000 7f28146c3640 1 --2- 172.21.15.123:0/626877786 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f280c0cf5a0 0x7f280c0b29a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:24.381 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:24.376+0000 7f28146c3640 1 -- 172.21.15.123:0/626877786 >> 172.21.15.123:0/626877786 conn(0x7f280c0cfb10 msgr2=0x7f280c0b0c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:47:24.381 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:24.376+0000 7f28146c3640 1 -- 172.21.15.123:0/626877786 shutdown_connections 2024-09-16T15:47:24.381 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:24.376+0000 7f28146c3640 1 -- 172.21.15.123:0/626877786 wait complete. 2024-09-16T15:47:24.396 DEBUG:teuthology.orchestra.run.smithi184:> set -ex 2024-09-16T15:47:24.396 DEBUG:teuthology.orchestra.run.smithi184:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-16T15:47:24.402 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-09-16T15:47:24.521 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:24.516+0000 7ff0ca7a4640 1 -- 172.21.15.123:0/2388370830 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff0c415e160 msgr2=0x7ff0c4073f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:47:24.521 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:24.516+0000 7ff0ca7a4640 1 --2- 172.21.15.123:0/2388370830 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff0c415e160 0x7ff0c4073f30 secure :-1 s=READY pgs=837 cs=0 l=1 rev1=1 crypto rx=0x7ff0c4073c20 tx=0x7ff0b401c700 comp rx=0 tx=0).stop 2024-09-16T15:47:24.521 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:24.516+0000 7ff0ca7a4640 1 -- 172.21.15.123:0/2388370830 shutdown_connections 2024-09-16T15:47:24.521 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:24.516+0000 7ff0ca7a4640 1 --2- 172.21.15.123:0/2388370830 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff0c415e160 0x7ff0c4073f30 unknown :-1 s=CLOSED pgs=837 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:24.521 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:24.516+0000 7ff0ca7a4640 1 -- 172.21.15.123:0/2388370830 >> 172.21.15.123:0/2388370830 conn(0x7ff0c40cfb10 msgr2=0x7ff0c40cff20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:47:24.521 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:24.516+0000 7ff0ca7a4640 1 -- 172.21.15.123:0/2388370830 shutdown_connections 2024-09-16T15:47:24.521 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:24.516+0000 7ff0ca7a4640 1 -- 172.21.15.123:0/2388370830 wait complete. 2024-09-16T15:47:24.522 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:24.516+0000 7ff0ca7a4640 1 Processor -- start 2024-09-16T15:47:24.522 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:24.516+0000 7ff0ca7a4640 1 -- start start 2024-09-16T15:47:24.524 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:24.520+0000 7ff0ca7a4640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff0c4074470 0x7ff0c41558c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:47:24.524 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:24.520+0000 7ff0ca7a4640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff0c415e160 0x7ff0c414e910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:47:24.524 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:24.520+0000 7ff0ca7a4640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7ff0c415e820 0x7ff0c40b3c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:47:24.524 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:24.520+0000 7ff0ca7a4640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7ff0c41697f0 con 0x7ff0c415e160 2024-09-16T15:47:24.524 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:24.520+0000 7ff0ca7a4640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7ff0c40cf640 con 0x7ff0c4074470 2024-09-16T15:47:24.524 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:24.520+0000 7ff0ca7a4640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7ff0c4169970 con 0x7ff0c415e820 2024-09-16T15:47:24.524 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:24.520+0000 7ff0c37fe640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff0c415e160 0x7ff0c414e910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:47:24.524 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:24.520+0000 7ff0c37fe640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff0c415e160 0x7ff0c414e910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:44796/0 (socket says 172.21.15.123:44796) 2024-09-16T15:47:24.524 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:24.520+0000 7ff0c37fe640 1 -- 172.21.15.123:0/4272927679 learned_addr learned my addr 172.21.15.123:0/4272927679 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:47:24.524 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:24.520+0000 7ff0c3fff640 1 --2- 172.21.15.123:0/4272927679 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff0c4074470 0x7ff0c41558c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:47:24.525 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:24.520+0000 7ff0c8d41640 1 --2- 172.21.15.123:0/4272927679 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7ff0c415e820 0x7ff0c40b3c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:47:24.525 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:24.520+0000 7ff0c37fe640 1 -- 172.21.15.123:0/4272927679 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7ff0c415e820 msgr2=0x7ff0c40b3c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:47:24.525 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:24.520+0000 7ff0c37fe640 1 --2- 172.21.15.123:0/4272927679 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7ff0c415e820 0x7ff0c40b3c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:24.525 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:24.520+0000 7ff0c37fe640 1 -- 172.21.15.123:0/4272927679 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff0c4074470 msgr2=0x7ff0c41558c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:47:24.525 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:24.520+0000 7ff0c37fe640 1 --2- 172.21.15.123:0/4272927679 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff0c4074470 0x7ff0c41558c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:24.525 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:24.520+0000 7ff0c37fe640 1 -- 172.21.15.123:0/4272927679 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff0c40b41b0 con 0x7ff0c415e160 2024-09-16T15:47:24.525 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:24.520+0000 7ff0c8d41640 1 --2- 172.21.15.123:0/4272927679 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7ff0c415e820 0x7ff0c40b3c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T15:47:24.525 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:24.520+0000 7ff0c37fe640 1 --2- 172.21.15.123:0/4272927679 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff0c415e160 0x7ff0c414e910 secure :-1 s=READY pgs=838 cs=0 l=1 rev1=1 crypto rx=0x7ff0b800c530 tx=0x7ff0b800c9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:47:24.525 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:24.520+0000 7ff0c0ff9640 1 -- 172.21.15.123:0/4272927679 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0b8017070 con 0x7ff0c415e160 2024-09-16T15:47:24.525 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:24.520+0000 7ff0c0ff9640 1 -- 172.21.15.123:0/4272927679 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7ff0b8003600 con 0x7ff0c415e160 2024-09-16T15:47:24.525 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:24.520+0000 7ff0c0ff9640 1 -- 172.21.15.123:0/4272927679 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0b8003a50 con 0x7ff0c415e160 2024-09-16T15:47:24.525 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:24.520+0000 7ff0ca7a4640 1 -- 172.21.15.123:0/4272927679 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff0c4165ea0 con 0x7ff0c415e160 2024-09-16T15:47:24.525 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:24.520+0000 7ff0ca7a4640 1 -- 172.21.15.123:0/4272927679 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff0c4166460 con 0x7ff0c415e160 2024-09-16T15:47:24.525 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:24.520+0000 7ff0ca7a4640 1 -- 172.21.15.123:0/4272927679 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff0c406eee0 con 0x7ff0c415e160 2024-09-16T15:47:24.526 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:24.520+0000 7ff0c0ff9640 1 -- 172.21.15.123:0/4272927679 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 11) ==== 97386+0+0 (secure 0 0 0) 0x7ff0b801a070 con 0x7ff0c415e160 2024-09-16T15:47:24.526 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:24.520+0000 7ff0c0ff9640 1 --2- 172.21.15.123:0/4272927679 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7ff0b0073250 0x7ff0b0075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:47:24.526 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:24.520+0000 7ff0c0ff9640 1 -- 172.21.15.123:0/4272927679 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(70..70 src has 1..70) ==== 5709+0+0 (secure 0 0 0) 0x7ff0b8093700 con 0x7ff0c415e160 2024-09-16T15:47:24.526 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:24.520+0000 7ff0c3fff640 1 --2- 172.21.15.123:0/4272927679 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7ff0b0073250 0x7ff0b0075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:47:24.527 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:24.524+0000 7ff0c3fff640 1 --2- 172.21.15.123:0/4272927679 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7ff0b0073250 0x7ff0b0075710 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7ff0b40ae000 tx=0x7ff0b401af40 comp rx=0 tx=0).ready entity=mgr.4135 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:47:24.529 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:24.524+0000 7ff0c0ff9640 1 -- 172.21.15.123:0/4272927679 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7ff0b805e5c0 con 0x7ff0c415e160 2024-09-16T15:47:24.727 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:24.724+0000 7ff0ca7a4640 1 -- 172.21.15.123:0/4272927679 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7ff0c40d06c0 con 0x7ff0c415e160 2024-09-16T15:47:24.728 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:24.724+0000 7ff0c0ff9640 1 -- 172.21.15.123:0/4272927679 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v70) ==== 80+0+0 (secure 0 0 0) 0x7ff0b805dec0 con 0x7ff0c415e160 2024-09-16T15:47:24.728 INFO:teuthology.orchestra.run.smithi123.stderr:listed 0 entries 2024-09-16T15:47:24.731 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:24.724+0000 7ff0ca7a4640 1 -- 172.21.15.123:0/4272927679 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7ff0b0073250 msgr2=0x7ff0b0075710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:47:24.731 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:24.724+0000 7ff0ca7a4640 1 --2- 172.21.15.123:0/4272927679 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7ff0b0073250 0x7ff0b0075710 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7ff0b40ae000 tx=0x7ff0b401af40 comp rx=0 tx=0).stop 2024-09-16T15:47:24.731 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:24.724+0000 7ff0ca7a4640 1 -- 172.21.15.123:0/4272927679 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff0c415e160 msgr2=0x7ff0c414e910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:47:24.731 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:24.724+0000 7ff0ca7a4640 1 --2- 172.21.15.123:0/4272927679 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff0c415e160 0x7ff0c414e910 secure :-1 s=READY pgs=838 cs=0 l=1 rev1=1 crypto rx=0x7ff0b800c530 tx=0x7ff0b800c9f0 comp rx=0 tx=0).stop 2024-09-16T15:47:24.731 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:24.724+0000 7ff0ca7a4640 1 -- 172.21.15.123:0/4272927679 shutdown_connections 2024-09-16T15:47:24.731 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:24.724+0000 7ff0ca7a4640 1 --2- 172.21.15.123:0/4272927679 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7ff0b0073250 0x7ff0b0075710 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:24.731 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:24.724+0000 7ff0ca7a4640 1 --2- 172.21.15.123:0/4272927679 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7ff0c415e820 0x7ff0c40b3c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:24.731 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:24.724+0000 7ff0ca7a4640 1 --2- 172.21.15.123:0/4272927679 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff0c415e160 0x7ff0c414e910 unknown :-1 s=CLOSED pgs=838 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:24.731 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:24.724+0000 7ff0ca7a4640 1 --2- 172.21.15.123:0/4272927679 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff0c4074470 0x7ff0c41558c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:24.731 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:24.724+0000 7ff0ca7a4640 1 -- 172.21.15.123:0/4272927679 >> 172.21.15.123:0/4272927679 conn(0x7ff0c40cfb10 msgr2=0x7ff0c4152b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:47:24.731 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:24.724+0000 7ff0ca7a4640 1 -- 172.21.15.123:0/4272927679 shutdown_connections 2024-09-16T15:47:24.731 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:24.728+0000 7ff0ca7a4640 1 -- 172.21.15.123:0/4272927679 wait complete. 2024-09-16T15:47:24.746 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi184.front.sepia.ceph.com... 2024-09-16T15:47:24.746 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:47:24.747 DEBUG:teuthology.orchestra.run.smithi184:> sudo fusermount -u /home/ubuntu/cephtest/mnt.1 2024-09-16T15:47:24.759 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi184.stderr:ceph-fuse[35318]: fuse finished with error 0 and tester_r 0 2024-09-16T15:47:24.761 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T15:47:30.862 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi184.front.sepia.ceph.com 2024-09-16T15:47:30.863 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:47:30.863 DEBUG:teuthology.orchestra.run.smithi184:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2024-09-16T15:47:30.870 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-16T15:47:31.004 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:31.000+0000 7fcbf7551640 1 -- 172.21.15.123:0/3513049485 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcbf00b2b00 msgr2=0x7fcbf00d24b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:47:31.005 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:31.000+0000 7fcbf7551640 1 --2- 172.21.15.123:0/3513049485 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcbf00b2b00 0x7fcbf00d24b0 secure :-1 s=READY pgs=840 cs=0 l=1 rev1=1 crypto rx=0x7fcbf008d220 tx=0x7fcbe001c5d0 comp rx=0 tx=0).stop 2024-09-16T15:47:31.005 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:31.000+0000 7fcbf7551640 1 -- 172.21.15.123:0/3513049485 shutdown_connections 2024-09-16T15:47:31.005 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:31.000+0000 7fcbf7551640 1 --2- 172.21.15.123:0/3513049485 >> v2:172.21.15.184:3300/0 conn(0x7fcbf00b0a50 0x7fcbf00b0ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:31.005 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:31.000+0000 7fcbf7551640 1 --2- 172.21.15.123:0/3513049485 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcbf00b2b00 0x7fcbf00d24b0 unknown :-1 s=CLOSED pgs=840 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:31.005 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:31.000+0000 7fcbf7551640 1 -- 172.21.15.123:0/3513049485 >> 172.21.15.123:0/3513049485 conn(0x7fcbf00d3ea0 msgr2=0x7fcbf00d42b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:47:31.005 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:31.000+0000 7fcbf7551640 1 -- 172.21.15.123:0/3513049485 shutdown_connections 2024-09-16T15:47:31.005 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:31.000+0000 7fcbf7551640 1 -- 172.21.15.123:0/3513049485 wait complete. 2024-09-16T15:47:31.005 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:31.000+0000 7fcbf7551640 1 Processor -- start 2024-09-16T15:47:31.006 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:31.000+0000 7fcbf7551640 1 -- start start 2024-09-16T15:47:31.007 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:31.000+0000 7fcbf7551640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fcbf00b0a50 0x7fcbf0154a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:47:31.008 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:31.000+0000 7fcbf7551640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fcbf00b2b00 0x7fcbf0155cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:47:31.008 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:31.004+0000 7fcbf7551640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcbf0161620 0x7fcbf00b6db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:47:31.008 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:31.004+0000 7fcbf7551640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fcbf016b0f0 con 0x7fcbf0161620 2024-09-16T15:47:31.008 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:31.004+0000 7fcbf7551640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fcbf016af70 con 0x7fcbf00b2b00 2024-09-16T15:47:31.008 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:31.004+0000 7fcbf7551640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fcbf016b270 con 0x7fcbf00b0a50 2024-09-16T15:47:31.008 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:31.004+0000 7fcbf5aee640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcbf0161620 0x7fcbf00b6db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:47:31.008 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:31.004+0000 7fcbf5aee640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcbf0161620 0x7fcbf00b6db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:44824/0 (socket says 172.21.15.123:44824) 2024-09-16T15:47:31.008 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:31.004+0000 7fcbf5aee640 1 -- 172.21.15.123:0/3674952899 learned_addr learned my addr 172.21.15.123:0/3674952899 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:47:31.008 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:31.004+0000 7fcbf52ed640 1 --2- 172.21.15.123:0/3674952899 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fcbf00b0a50 0x7fcbf0154a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:47:31.008 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:31.004+0000 7fcbf5aee640 1 -- 172.21.15.123:0/3674952899 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fcbf00b0a50 msgr2=0x7fcbf0154a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:47:31.008 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:31.004+0000 7fcbf5aee640 1 --2- 172.21.15.123:0/3674952899 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fcbf00b0a50 0x7fcbf0154a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:31.008 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:31.004+0000 7fcbf5aee640 1 -- 172.21.15.123:0/3674952899 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fcbf00b2b00 msgr2=0x7fcbf0155cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:47:31.009 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:31.004+0000 7fcbf5aee640 1 --2- 172.21.15.123:0/3674952899 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fcbf00b2b00 0x7fcbf0155cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:31.009 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:31.004+0000 7fcbf5aee640 1 -- 172.21.15.123:0/3674952899 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcbf0154fa0 con 0x7fcbf0161620 2024-09-16T15:47:31.009 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:31.004+0000 7fcbf5aee640 1 --2- 172.21.15.123:0/3674952899 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcbf0161620 0x7fcbf00b6db0 secure :-1 s=READY pgs=841 cs=0 l=1 rev1=1 crypto rx=0x7fcbe800e970 tx=0x7fcbe800ee30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:47:31.009 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:31.004+0000 7fcbe5ffb640 1 -- 172.21.15.123:0/3674952899 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbe8004d30 con 0x7fcbf0161620 2024-09-16T15:47:31.009 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:31.004+0000 7fcbe5ffb640 1 -- 172.21.15.123:0/3674952899 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fcbe800cb20 con 0x7fcbf0161620 2024-09-16T15:47:31.009 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:31.004+0000 7fcbe5ffb640 1 -- 172.21.15.123:0/3674952899 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbe8005020 con 0x7fcbf0161620 2024-09-16T15:47:31.009 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:31.004+0000 7fcbf7551640 1 -- 172.21.15.123:0/3674952899 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcbf00b9ae0 con 0x7fcbf0161620 2024-09-16T15:47:31.009 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:31.004+0000 7fcbf7551640 1 -- 172.21.15.123:0/3674952899 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcbf00b9520 con 0x7fcbf0161620 2024-09-16T15:47:31.010 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:31.004+0000 7fcbf7551640 1 -- 172.21.15.123:0/3674952899 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcbf00b0ea0 con 0x7fcbf0161620 2024-09-16T15:47:31.010 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:31.004+0000 7fcbe5ffb640 1 -- 172.21.15.123:0/3674952899 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 11) ==== 97386+0+0 (secure 0 0 0) 0x7fcbe80051c0 con 0x7fcbf0161620 2024-09-16T15:47:31.010 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:31.004+0000 7fcbe5ffb640 1 --2- 172.21.15.123:0/3674952899 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fcbd4073220 0x7fcbd40756e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:47:31.010 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:31.004+0000 7fcbe5ffb640 1 -- 172.21.15.123:0/3674952899 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(70..70 src has 1..70) ==== 5709+0+0 (secure 0 0 0) 0x7fcbe80951c0 con 0x7fcbf0161620 2024-09-16T15:47:31.010 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:31.004+0000 7fcbf52ed640 1 --2- 172.21.15.123:0/3674952899 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fcbd4073220 0x7fcbd40756e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:47:31.011 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:31.008+0000 7fcbf52ed640 1 --2- 172.21.15.123:0/3674952899 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fcbd4073220 0x7fcbd40756e0 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7fcbf015a600 tx=0x7fcbec009040 comp rx=0 tx=0).ready entity=mgr.4135 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:47:31.013 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:31.008+0000 7fcbe5ffb640 1 -- 172.21.15.123:0/3674952899 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7fcbe8016030 con 0x7fcbf0161620 2024-09-16T15:47:31.217 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:31.212+0000 7fcbf7551640 1 -- 172.21.15.123:0/3674952899 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fcbf00ce360 con 0x7fcbf0161620 2024-09-16T15:47:31.218 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:31.216+0000 7fcbe5ffb640 1 -- 172.21.15.123:0/3674952899 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 45 v45) ==== 94+0+5234 (secure 0 0 0) 0x7fcbe805ff80 con 0x7fcbf0161620 2024-09-16T15:47:31.219 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-09-16T15:47:31.219 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":45,"btime":"2024-09-16T15:47:16:409841+0000","default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5048,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6837/1498917352","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6835","nonce":1498917352},{"type":"v1","addr":"172.21.15.184:6837","nonce":1498917352}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":40},{"gid":6198,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6833/1715139909","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6832","nonce":1715139909},{"type":"v1","addr":"172.21.15.123:6833","nonce":1715139909}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":40},{"gid":6201,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6835/815923338","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6834","nonce":815923338},{"type":"v1","addr":"172.21.15.123:6835","nonce":815923338}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":40}],"filesystems":[{"mdsmap":{"epoch":44,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-16T15:46:58.916477+0000","modified":"2024-09-16T15:47:01.963874+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6216},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6216":{"gid":6216,"name":"d","rank":0,"incarnation":41,"state":"up:active","state_seq":3,"addr":"172.21.15.184:6836/3137795986","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":3137795986},{"type":"v1","addr":"172.21.15.184:6836","nonce":3137795986}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6216,"qdb_cluster":[6216]},"id":5}]} 2024-09-16T15:47:31.219 INFO:teuthology.orchestra.run.smithi123.stderr:dumped fsmap epoch 45 2024-09-16T15:47:31.221 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:31.216+0000 7fcbf7551640 1 -- 172.21.15.123:0/3674952899 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fcbd4073220 msgr2=0x7fcbd40756e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:47:31.221 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:31.216+0000 7fcbf7551640 1 --2- 172.21.15.123:0/3674952899 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fcbd4073220 0x7fcbd40756e0 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7fcbf015a600 tx=0x7fcbec009040 comp rx=0 tx=0).stop 2024-09-16T15:47:31.221 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:31.216+0000 7fcbf7551640 1 -- 172.21.15.123:0/3674952899 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcbf0161620 msgr2=0x7fcbf00b6db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:47:31.221 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:31.216+0000 7fcbf7551640 1 --2- 172.21.15.123:0/3674952899 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcbf0161620 0x7fcbf00b6db0 secure :-1 s=READY pgs=841 cs=0 l=1 rev1=1 crypto rx=0x7fcbe800e970 tx=0x7fcbe800ee30 comp rx=0 tx=0).stop 2024-09-16T15:47:31.221 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:31.216+0000 7fcbf7551640 1 -- 172.21.15.123:0/3674952899 shutdown_connections 2024-09-16T15:47:31.221 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:31.216+0000 7fcbf7551640 1 --2- 172.21.15.123:0/3674952899 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fcbd4073220 0x7fcbd40756e0 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:31.221 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:31.216+0000 7fcbf7551640 1 --2- 172.21.15.123:0/3674952899 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcbf0161620 0x7fcbf00b6db0 unknown :-1 s=CLOSED pgs=841 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:31.221 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:31.216+0000 7fcbf7551640 1 --2- 172.21.15.123:0/3674952899 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fcbf00b2b00 0x7fcbf0155cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:31.222 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:31.216+0000 7fcbf7551640 1 --2- 172.21.15.123:0/3674952899 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fcbf00b0a50 0x7fcbf0154a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:31.222 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:31.216+0000 7fcbf7551640 1 -- 172.21.15.123:0/3674952899 >> 172.21.15.123:0/3674952899 conn(0x7fcbf00d3ea0 msgr2=0x7fcbf0159cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:47:31.222 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:31.216+0000 7fcbf7551640 1 -- 172.21.15.123:0/3674952899 shutdown_connections 2024-09-16T15:47:31.222 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:31.216+0000 7fcbf7551640 1 -- 172.21.15.123:0/3674952899 wait complete. 2024-09-16T15:47:31.237 DEBUG:teuthology.orchestra.run.smithi184:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.d.asok --format=json session ls 2024-09-16T15:47:31.390 INFO:teuthology.orchestra.run.smithi184.stdout:[] 2024-09-16T15:47:31.390 DEBUG:tasks.cephfs.filesystem:_json_asok output [] 2024-09-16T15:47:31.391 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume create cephfs subvol49770 2024-09-16T15:47:31.528 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:31.524+0000 7fa4e19a9640 1 -- 172.21.15.123:0/2174038335 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa4dc15a100 msgr2=0x7fa4dc072f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:47:31.528 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:31.524+0000 7fa4e19a9640 1 --2- 172.21.15.123:0/2174038335 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa4dc15a100 0x7fa4dc072f20 secure :-1 s=READY pgs=842 cs=0 l=1 rev1=1 crypto rx=0x7fa4cc009900 tx=0x7fa4cc01c870 comp rx=0 tx=0).stop 2024-09-16T15:47:31.528 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:31.524+0000 7fa4e19a9640 1 -- 172.21.15.123:0/2174038335 shutdown_connections 2024-09-16T15:47:31.528 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:31.524+0000 7fa4e19a9640 1 --2- 172.21.15.123:0/2174038335 >> v2:172.21.15.184:3300/0 conn(0x7fa4dc0cbb40 0x7fa4dc0cbf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:31.528 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:31.524+0000 7fa4e19a9640 1 --2- 172.21.15.123:0/2174038335 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa4dc15a100 0x7fa4dc072f20 unknown :-1 s=CLOSED pgs=842 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:31.528 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:31.524+0000 7fa4e19a9640 1 -- 172.21.15.123:0/2174038335 >> 172.21.15.123:0/2174038335 conn(0x7fa4dc0b1710 msgr2=0x7fa4dc0b1b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:47:31.528 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:31.524+0000 7fa4e19a9640 1 -- 172.21.15.123:0/2174038335 shutdown_connections 2024-09-16T15:47:31.528 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:31.524+0000 7fa4e19a9640 1 -- 172.21.15.123:0/2174038335 wait complete. 2024-09-16T15:47:31.529 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:31.524+0000 7fa4e19a9640 1 Processor -- start 2024-09-16T15:47:31.529 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:31.524+0000 7fa4e19a9640 1 -- start start 2024-09-16T15:47:31.531 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:31.524+0000 7fa4e19a9640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa4dc0cbb40 0x7fa4dc151370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:47:31.531 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:31.524+0000 7fa4e19a9640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa4dc15a100 0x7fa4dc158cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:47:31.531 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:31.524+0000 7fa4e19a9640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fa4dc15ddb0 0x7fa4dc167e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:47:31.531 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:31.524+0000 7fa4e19a9640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fa4dc1678d0 con 0x7fa4dc0cbb40 2024-09-16T15:47:31.531 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:31.524+0000 7fa4e19a9640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fa4dc167750 con 0x7fa4dc15a100 2024-09-16T15:47:31.531 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:31.524+0000 7fa4e19a9640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fa4dc167a50 con 0x7fa4dc15ddb0 2024-09-16T15:47:31.531 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:31.524+0000 7fa4daffd640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa4dc0cbb40 0x7fa4dc151370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:47:31.531 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:31.524+0000 7fa4daffd640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa4dc0cbb40 0x7fa4dc151370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:44844/0 (socket says 172.21.15.123:44844) 2024-09-16T15:47:31.531 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:31.524+0000 7fa4daffd640 1 -- 172.21.15.123:0/2679949534 learned_addr learned my addr 172.21.15.123:0/2679949534 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:47:31.532 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:31.524+0000 7fa4daffd640 1 -- 172.21.15.123:0/2679949534 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fa4dc15ddb0 msgr2=0x7fa4dc167e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:47:31.532 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:31.524+0000 7fa4db7fe640 1 --2- 172.21.15.123:0/2679949534 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fa4dc15ddb0 0x7fa4dc167e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:47:31.532 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:31.524+0000 7fa4d3fff640 1 --2- 172.21.15.123:0/2679949534 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa4dc15a100 0x7fa4dc158cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:47:31.532 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:31.524+0000 7fa4daffd640 1 --2- 172.21.15.123:0/2679949534 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fa4dc15ddb0 0x7fa4dc167e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:31.532 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:31.524+0000 7fa4daffd640 1 -- 172.21.15.123:0/2679949534 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa4dc15a100 msgr2=0x7fa4dc158cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:47:31.532 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:31.524+0000 7fa4daffd640 1 --2- 172.21.15.123:0/2679949534 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa4dc15a100 0x7fa4dc158cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:31.532 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:31.524+0000 7fa4daffd640 1 -- 172.21.15.123:0/2679949534 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa4dc0b55c0 con 0x7fa4dc0cbb40 2024-09-16T15:47:31.532 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:31.524+0000 7fa4d3fff640 1 --2- 172.21.15.123:0/2679949534 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa4dc15a100 0x7fa4dc158cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T15:47:31.532 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:31.528+0000 7fa4daffd640 1 --2- 172.21.15.123:0/2679949534 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa4dc0cbb40 0x7fa4dc151370 secure :-1 s=READY pgs=843 cs=0 l=1 rev1=1 crypto rx=0x7fa4cc009850 tx=0x7fa4cc0040e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:47:31.532 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:31.528+0000 7fa4d8ff9640 1 -- 172.21.15.123:0/2679949534 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4cc0b2040 con 0x7fa4dc0cbb40 2024-09-16T15:47:31.532 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:31.528+0000 7fa4d8ff9640 1 -- 172.21.15.123:0/2679949534 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fa4cc005030 con 0x7fa4dc0cbb40 2024-09-16T15:47:31.532 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:31.528+0000 7fa4e19a9640 1 -- 172.21.15.123:0/2679949534 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa4dc0c8440 con 0x7fa4dc0cbb40 2024-09-16T15:47:31.533 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:31.528+0000 7fa4d8ff9640 1 -- 172.21.15.123:0/2679949534 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4cc005460 con 0x7fa4dc0cbb40 2024-09-16T15:47:31.533 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:31.528+0000 7fa4e19a9640 1 -- 172.21.15.123:0/2679949534 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa4dc0c8a00 con 0x7fa4dc0cbb40 2024-09-16T15:47:31.533 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:31.528+0000 7fa4e19a9640 1 -- 172.21.15.123:0/2679949534 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa4dc0cbf80 con 0x7fa4dc0cbb40 2024-09-16T15:47:31.533 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:31.528+0000 7fa4d8ff9640 1 -- 172.21.15.123:0/2679949534 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 11) ==== 97386+0+0 (secure 0 0 0) 0x7fa4cc0046d0 con 0x7fa4dc0cbb40 2024-09-16T15:47:31.534 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:31.528+0000 7fa4d8ff9640 1 --2- 172.21.15.123:0/2679949534 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fa4bc073220 0x7fa4bc0756e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:47:31.534 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:31.528+0000 7fa4d8ff9640 1 -- 172.21.15.123:0/2679949534 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(71..71 src has 1..71) ==== 5767+0+0 (secure 0 0 0) 0x7fa4cc12e0e0 con 0x7fa4dc0cbb40 2024-09-16T15:47:31.534 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:31.528+0000 7fa4d3fff640 1 --2- 172.21.15.123:0/2679949534 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fa4bc073220 0x7fa4bc0756e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:47:31.534 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:31.528+0000 7fa4d3fff640 1 --2- 172.21.15.123:0/2679949534 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fa4bc073220 0x7fa4bc0756e0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7fa4dc159b00 tx=0x7fa4c4008040 comp rx=0 tx=0).ready entity=mgr.4135 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:47:31.537 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:31.532+0000 7fa4d8ff9640 1 -- 172.21.15.123:0/2679949534 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7fa4cc016490 con 0x7fa4dc0cbb40 2024-09-16T15:47:31.739 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:31.736+0000 7fa4e19a9640 1 -- 172.21.15.123:0/2679949534 --> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "subvol49770", "target": ["mon-mgr", ""]}) -- 0x7fa4dc0008e0 con 0x7fa4bc073220 2024-09-16T15:47:31.751 INFO:tasks.ceph.mgr.x.smithi184.stderr:2024-09-16T15:47:31.749+0000 7fda6a8d4640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-16T15:47:31.751 INFO:tasks.ceph.mgr.x.smithi184.stderr:2024-09-16T15:47:31.749+0000 7fda6a8d4640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-16T15:47:31.751 INFO:tasks.ceph.mgr.x.smithi184.stderr:2024-09-16T15:47:31.749+0000 7fda6a8d4640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-16T15:47:31.751 INFO:tasks.ceph.mgr.x.smithi184.stderr:2024-09-16T15:47:31.749+0000 7fda6a8d4640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-16T15:47:31.751 INFO:tasks.ceph.mgr.x.smithi184.stderr:2024-09-16T15:47:31.749+0000 7fda6a8d4640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-16T15:47:31.800 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:31.796+0000 7fa4d8ff9640 1 -- 172.21.15.123:0/2679949534 <== mgr.4135 v2:172.21.15.184:6832/3114346423 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7fa4dc0008e0 con 0x7fa4bc073220 2024-09-16T15:47:31.803 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:31.800+0000 7fa4e19a9640 1 -- 172.21.15.123:0/2679949534 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fa4bc073220 msgr2=0x7fa4bc0756e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:47:31.804 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:31.800+0000 7fa4e19a9640 1 --2- 172.21.15.123:0/2679949534 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fa4bc073220 0x7fa4bc0756e0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7fa4dc159b00 tx=0x7fa4c4008040 comp rx=0 tx=0).stop 2024-09-16T15:47:31.804 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:31.800+0000 7fa4e19a9640 1 -- 172.21.15.123:0/2679949534 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa4dc0cbb40 msgr2=0x7fa4dc151370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:47:31.804 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:31.800+0000 7fa4e19a9640 1 --2- 172.21.15.123:0/2679949534 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa4dc0cbb40 0x7fa4dc151370 secure :-1 s=READY pgs=843 cs=0 l=1 rev1=1 crypto rx=0x7fa4cc009850 tx=0x7fa4cc0040e0 comp rx=0 tx=0).stop 2024-09-16T15:47:31.804 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:31.800+0000 7fa4e19a9640 1 -- 172.21.15.123:0/2679949534 shutdown_connections 2024-09-16T15:47:31.804 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:31.800+0000 7fa4e19a9640 1 --2- 172.21.15.123:0/2679949534 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7fa4bc073220 0x7fa4bc0756e0 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:31.804 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:31.800+0000 7fa4e19a9640 1 --2- 172.21.15.123:0/2679949534 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fa4dc15ddb0 0x7fa4dc167e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:31.804 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:31.800+0000 7fa4e19a9640 1 --2- 172.21.15.123:0/2679949534 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa4dc15a100 0x7fa4dc158cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:31.804 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:31.800+0000 7fa4e19a9640 1 --2- 172.21.15.123:0/2679949534 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa4dc0cbb40 0x7fa4dc151370 unknown :-1 s=CLOSED pgs=843 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:31.804 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:31.800+0000 7fa4e19a9640 1 -- 172.21.15.123:0/2679949534 >> 172.21.15.123:0/2679949534 conn(0x7fa4dc0b1710 msgr2=0x7fa4dc0b1af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:47:31.804 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:31.800+0000 7fa4e19a9640 1 -- 172.21.15.123:0/2679949534 shutdown_connections 2024-09-16T15:47:31.804 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:31.800+0000 7fa4e19a9640 1 -- 172.21.15.123:0/2679949534 wait complete. 2024-09-16T15:47:31.822 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-16T15:47:31.951 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:31.948+0000 7f2525653640 1 -- 172.21.15.123:0/3683864358 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2520161620 msgr2=0x7f25200d1da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:47:31.951 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:31.948+0000 7f2525653640 1 --2- 172.21.15.123:0/3683864358 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2520161620 0x7f25200d1da0 secure :-1 s=READY pgs=845 cs=0 l=1 rev1=1 crypto rx=0x7f2510005fd0 tx=0x7f251001c610 comp rx=0 tx=0).stop 2024-09-16T15:47:31.951 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:31.948+0000 7f2525653640 1 -- 172.21.15.123:0/3683864358 shutdown_connections 2024-09-16T15:47:31.951 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:31.948+0000 7f2525653640 1 --2- 172.21.15.123:0/3683864358 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2520161620 0x7f25200d1da0 unknown :-1 s=CLOSED pgs=845 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:31.951 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:31.948+0000 7f2525653640 1 -- 172.21.15.123:0/3683864358 >> 172.21.15.123:0/3683864358 conn(0x7f25200d3ea0 msgr2=0x7f25200d42b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:47:31.951 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:31.948+0000 7f2525653640 1 -- 172.21.15.123:0/3683864358 shutdown_connections 2024-09-16T15:47:31.951 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:31.948+0000 7f2525653640 1 -- 172.21.15.123:0/3683864358 wait complete. 2024-09-16T15:47:31.952 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:31.948+0000 7f2525653640 1 Processor -- start 2024-09-16T15:47:31.952 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:31.948+0000 7f2525653640 1 -- start start 2024-09-16T15:47:31.954 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:31.948+0000 7f2525653640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f25200b0a50 0x7f2520156dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:47:31.954 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:31.948+0000 7f2525653640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f25200b2b00 0x7f2520157310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:47:31.954 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:31.948+0000 7f2525653640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f2520161620 0x7f2520157850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:47:31.954 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:31.948+0000 7f2525653640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f252016c100 con 0x7f25200b2b00 2024-09-16T15:47:31.954 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:31.948+0000 7f2525653640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f252016bf80 con 0x7f25200b0a50 2024-09-16T15:47:31.954 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:31.948+0000 7f2525653640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f252016c280 con 0x7f2520161620 2024-09-16T15:47:31.955 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:31.948+0000 7f251e7fc640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f25200b2b00 0x7f2520157310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:47:31.955 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:31.948+0000 7f251e7fc640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f25200b2b00 0x7f2520157310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:44862/0 (socket says 172.21.15.123:44862) 2024-09-16T15:47:31.955 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:31.948+0000 7f251e7fc640 1 -- 172.21.15.123:0/1847162306 learned_addr learned my addr 172.21.15.123:0/1847162306 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:47:31.955 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:31.948+0000 7f251effd640 1 --2- 172.21.15.123:0/1847162306 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f25200b0a50 0x7f2520156dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:47:31.955 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:31.948+0000 7f251f7fe640 1 --2- 172.21.15.123:0/1847162306 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f2520161620 0x7f2520157850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:47:31.955 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:31.948+0000 7f251e7fc640 1 -- 172.21.15.123:0/1847162306 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f2520161620 msgr2=0x7f2520157850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:47:31.955 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:31.948+0000 7f251e7fc640 1 --2- 172.21.15.123:0/1847162306 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f2520161620 0x7f2520157850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:31.955 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:31.948+0000 7f251e7fc640 1 -- 172.21.15.123:0/1847162306 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f25200b0a50 msgr2=0x7f2520156dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:47:31.955 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:31.948+0000 7f251e7fc640 1 --2- 172.21.15.123:0/1847162306 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f25200b0a50 0x7f2520156dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:31.955 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:31.948+0000 7f251e7fc640 1 -- 172.21.15.123:0/1847162306 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f25200c7e40 con 0x7f25200b2b00 2024-09-16T15:47:31.955 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:31.948+0000 7f251e7fc640 1 --2- 172.21.15.123:0/1847162306 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f25200b2b00 0x7f2520157310 secure :-1 s=READY pgs=846 cs=0 l=1 rev1=1 crypto rx=0x7f251400c450 tx=0x7f251400c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:47:31.955 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:31.948+0000 7f25077fe640 1 -- 172.21.15.123:0/1847162306 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2514016020 con 0x7f25200b2b00 2024-09-16T15:47:31.955 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:31.948+0000 7f25077fe640 1 -- 172.21.15.123:0/1847162306 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f2514012920 con 0x7f25200b2b00 2024-09-16T15:47:31.956 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:31.948+0000 7f2525653640 1 -- 172.21.15.123:0/1847162306 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2520154630 con 0x7f25200b2b00 2024-09-16T15:47:31.956 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:31.948+0000 7f25077fe640 1 -- 172.21.15.123:0/1847162306 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2514012d70 con 0x7f25200b2b00 2024-09-16T15:47:31.956 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:31.948+0000 7f2525653640 1 -- 172.21.15.123:0/1847162306 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2520154bf0 con 0x7f25200b2b00 2024-09-16T15:47:31.956 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:31.952+0000 7f2525653640 1 -- 172.21.15.123:0/1847162306 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2520154df0 con 0x7f25200b2b00 2024-09-16T15:47:31.957 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:31.952+0000 7f25077fe640 1 -- 172.21.15.123:0/1847162306 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 11) ==== 97386+0+0 (secure 0 0 0) 0x7f2514019020 con 0x7f25200b2b00 2024-09-16T15:47:31.957 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:31.952+0000 7f25077fe640 1 --2- 172.21.15.123:0/1847162306 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f250c073220 0x7f250c0756e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:47:31.957 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:31.952+0000 7f25077fe640 1 -- 172.21.15.123:0/1847162306 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(71..71 src has 1..71) ==== 5767+0+0 (secure 0 0 0) 0x7f25140943a0 con 0x7f25200b2b00 2024-09-16T15:47:31.957 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:31.952+0000 7f251effd640 1 --2- 172.21.15.123:0/1847162306 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f250c073220 0x7f250c0756e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:47:31.957 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:31.952+0000 7f251effd640 1 --2- 172.21.15.123:0/1847162306 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f250c073220 0x7f250c0756e0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f25100ad000 tx=0x7f25100a9040 comp rx=0 tx=0).ready entity=mgr.4135 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:47:31.960 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:31.956+0000 7f25077fe640 1 -- 172.21.15.123:0/1847162306 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f2520154df0 con 0x7f25200b2b00 2024-09-16T15:47:32.172 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:32.168+0000 7f2525653640 1 -- 172.21.15.123:0/1847162306 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f25200b0ea0 con 0x7f25200b2b00 2024-09-16T15:47:32.173 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:32.168+0000 7f25077fe640 1 -- 172.21.15.123:0/1847162306 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 45 v45) ==== 94+0+5234 (secure 0 0 0) 0x7f2514020020 con 0x7f25200b2b00 2024-09-16T15:47:32.174 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-09-16T15:47:32.174 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":45,"btime":"2024-09-16T15:47:16:409841+0000","default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5048,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6837/1498917352","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6835","nonce":1498917352},{"type":"v1","addr":"172.21.15.184:6837","nonce":1498917352}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":40},{"gid":6198,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6833/1715139909","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6832","nonce":1715139909},{"type":"v1","addr":"172.21.15.123:6833","nonce":1715139909}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":40},{"gid":6201,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6835/815923338","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6834","nonce":815923338},{"type":"v1","addr":"172.21.15.123:6835","nonce":815923338}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":40}],"filesystems":[{"mdsmap":{"epoch":44,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-16T15:46:58.916477+0000","modified":"2024-09-16T15:47:01.963874+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6216},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6216":{"gid":6216,"name":"d","rank":0,"incarnation":41,"state":"up:active","state_seq":3,"addr":"172.21.15.184:6836/3137795986","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":3137795986},{"type":"v1","addr":"172.21.15.184:6836","nonce":3137795986}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6216,"qdb_cluster":[6216]},"id":5}]} 2024-09-16T15:47:32.174 INFO:teuthology.orchestra.run.smithi123.stderr:dumped fsmap epoch 45 2024-09-16T15:47:32.176 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:32.172+0000 7f2525653640 1 -- 172.21.15.123:0/1847162306 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f250c073220 msgr2=0x7f250c0756e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:47:32.176 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:32.172+0000 7f2525653640 1 --2- 172.21.15.123:0/1847162306 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f250c073220 0x7f250c0756e0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f25100ad000 tx=0x7f25100a9040 comp rx=0 tx=0).stop 2024-09-16T15:47:32.176 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:32.172+0000 7f2525653640 1 -- 172.21.15.123:0/1847162306 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f25200b2b00 msgr2=0x7f2520157310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:47:32.176 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:32.172+0000 7f2525653640 1 --2- 172.21.15.123:0/1847162306 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f25200b2b00 0x7f2520157310 secure :-1 s=READY pgs=846 cs=0 l=1 rev1=1 crypto rx=0x7f251400c450 tx=0x7f251400c910 comp rx=0 tx=0).stop 2024-09-16T15:47:32.177 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:32.172+0000 7f2525653640 1 -- 172.21.15.123:0/1847162306 shutdown_connections 2024-09-16T15:47:32.177 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:32.172+0000 7f2525653640 1 --2- 172.21.15.123:0/1847162306 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f250c073220 0x7f250c0756e0 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:32.177 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:32.172+0000 7f2525653640 1 --2- 172.21.15.123:0/1847162306 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f2520161620 0x7f2520157850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:32.177 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:32.172+0000 7f2525653640 1 --2- 172.21.15.123:0/1847162306 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f25200b2b00 0x7f2520157310 unknown :-1 s=CLOSED pgs=846 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:32.177 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:32.172+0000 7f2525653640 1 --2- 172.21.15.123:0/1847162306 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f25200b0a50 0x7f2520156dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:32.177 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:32.172+0000 7f2525653640 1 -- 172.21.15.123:0/1847162306 >> 172.21.15.123:0/1847162306 conn(0x7f25200d3ea0 msgr2=0x7f252006e630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:47:32.177 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:32.172+0000 7f2525653640 1 -- 172.21.15.123:0/1847162306 shutdown_connections 2024-09-16T15:47:32.177 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:32.172+0000 7f2525653640 1 -- 172.21.15.123:0/1847162306 wait complete. 2024-09-16T15:47:32.193 DEBUG:teuthology.orchestra.run.smithi184:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.d.asok --format=json session ls 2024-09-16T15:47:32.342 INFO:teuthology.orchestra.run.smithi184.stdout:[{"id":5530,"entity":{"name":{"type":"client","num":5530},"addr":{"type":"any","addr":"172.21.15.184:0","nonce":654280269}},"state":"open","num_leases":1,"num_caps":7,"request_load_avg":15,"uptime":0.575994856,"requests_in_flight":2,"num_completed_requests":0,"num_completed_flushes":1,"reconnecting":false,"recall_caps":{"value":0,"halflife":60},"release_caps":{"value":0,"halflife":60},"recall_caps_throttle":{"value":0,"halflife":1.5},"recall_caps_throttle2o":{"value":0,"halflife":0.5},"session_cache_liveness":{"value":21.971719565088829,"halflife":300},"cap_acquisition":{"value":0,"halflife":30},"last_trim_completed_requests_tid":15,"last_trim_completed_flushes_tid":2,"delegated_inos":[{"start":"0x10000000005","length":500}],"inst":"client.5530 172.21.15.184:0/654280269","completed_requests":[],"prealloc_inos":[{"start":"0x10000000005","length":996}],"client_metadata":{"client_features":{"feature_bits":"0x00000000003fffff"},"metric_spec":{"metric_flags":{"feature_bits":"0x000000000000ffff"}},"ceph_sha1":"33f95d2ae1bcbe7eb04404303f179eeeaa6982df","ceph_version":"ceph version 19.3.0-4938-g33f95d2a (33f95d2ae1bcbe7eb04404303f179eeeaa6982df) squid (dev)","entity_id":"x","hostname":"smithi184","pid":"28840","root":"/"}}] 2024-09-16T15:47:32.343 DEBUG:tasks.cephfs.filesystem:_json_asok output [ { "cap_acquisition": { "halflife": 30, "value": 0 }, "client_metadata": { "ceph_sha1": "33f95d2ae1bcbe7eb04404303f179eeeaa6982df", "ceph_version": "ceph version 19.3.0-4938-g33f95d2a (33f95d2ae1bcbe7eb04404303f179eeeaa6982df) squid (dev)", "client_features": { "feature_bits": "0x00000000003fffff" }, "entity_id": "x", "hostname": "smithi184", "metric_spec": { "metric_flags": { "feature_bits": "0x000000000000ffff" } }, "pid": "28840", "root": "/" }, "completed_requests": [], "delegated_inos": [ { "length": 500, "start": "0x10000000005" } ], "entity": { "addr": { "addr": "172.21.15.184:0", "nonce": 654280269, "type": "any" }, "name": { "num": 5530, "type": "client" } }, "id": 5530, "inst": "client.5530 172.21.15.184:0/654280269", "last_trim_completed_flushes_tid": 2, "last_trim_completed_requests_tid": 15, "num_caps": 7, "num_completed_flushes": 1, "num_completed_requests": 0, "num_leases": 1, "prealloc_inos": [ { "length": 996, "start": "0x10000000005" } ], "recall_caps": { "halflife": 60, "value": 0 }, "recall_caps_throttle": { "halflife": 1.5, "value": 0 }, "recall_caps_throttle2o": { "halflife": 0.5, "value": 0 }, "reconnecting": false, "release_caps": { "halflife": 60, "value": 0 }, "request_load_avg": 15, "requests_in_flight": 2, "session_cache_liveness": { "halflife": 300, "value": 21.97171956508883 }, "state": "open", "uptime": 0.575994856 } ] 2024-09-16T15:47:32.343 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json-pretty 2024-09-16T15:47:32.485 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:32.480+0000 7f0042c67640 1 -- 172.21.15.123:0/3530242090 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f003c161620 msgr2=0x7f003c0d1da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:47:32.485 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:32.480+0000 7f0042c67640 1 --2- 172.21.15.123:0/3530242090 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f003c161620 0x7f003c0d1da0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f0034005fd0 tx=0x7f003401c6d0 comp rx=0 tx=0).stop 2024-09-16T15:47:32.486 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:32.480+0000 7f0042c67640 1 -- 172.21.15.123:0/3530242090 shutdown_connections 2024-09-16T15:47:32.486 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:32.480+0000 7f0042c67640 1 --2- 172.21.15.123:0/3530242090 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f003c0b2b00 0x7f003c0d22e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:32.486 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:32.480+0000 7f0042c67640 1 --2- 172.21.15.123:0/3530242090 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f003c161620 0x7f003c0d1da0 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:32.486 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:32.480+0000 7f0042c67640 1 -- 172.21.15.123:0/3530242090 >> 172.21.15.123:0/3530242090 conn(0x7f003c0d3ea0 msgr2=0x7f003c0d42b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:47:32.486 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:32.480+0000 7f0042c67640 1 -- 172.21.15.123:0/3530242090 shutdown_connections 2024-09-16T15:47:32.486 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:32.480+0000 7f0042c67640 1 -- 172.21.15.123:0/3530242090 wait complete. 2024-09-16T15:47:32.487 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:32.484+0000 7f0042c67640 1 Processor -- start 2024-09-16T15:47:32.487 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:32.484+0000 7f0042c67640 1 -- start start 2024-09-16T15:47:32.489 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:32.484+0000 7f0042c67640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f003c0b2b00 0x7f003c0cc2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:47:32.489 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:32.484+0000 7f0042c67640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f003c0d2820 0x7f003c1fab10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:47:32.489 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:32.484+0000 7f0042c67640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f003c161620 0x7f003c156010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:47:32.489 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:32.484+0000 7f0042c67640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f003c16aee0 con 0x7f003c0d2820 2024-09-16T15:47:32.489 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:32.484+0000 7f0042c67640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f003c16ad60 con 0x7f003c161620 2024-09-16T15:47:32.489 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:32.484+0000 7f0042c67640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f003c16b060 con 0x7f003c0b2b00 2024-09-16T15:47:32.489 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:32.484+0000 7f0041204640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f003c161620 0x7f003c156010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:47:32.490 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:32.484+0000 7f0041204640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f003c161620 0x7f003c156010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.123:43142/0 (socket says 172.21.15.123:43142) 2024-09-16T15:47:32.490 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:32.484+0000 7f0041204640 1 -- 172.21.15.123:0/3919206734 learned_addr learned my addr 172.21.15.123:0/3919206734 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:47:32.490 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:32.484+0000 7f0040a03640 1 --2- 172.21.15.123:0/3919206734 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f003c0b2b00 0x7f003c0cc2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:47:32.490 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:32.484+0000 7f003bfff640 1 --2- 172.21.15.123:0/3919206734 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f003c0d2820 0x7f003c1fab10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:47:32.490 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:32.488+0000 7f0041204640 1 -- 172.21.15.123:0/3919206734 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f003c0b2b00 msgr2=0x7f003c0cc2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:47:32.491 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:32.488+0000 7f0041204640 1 --2- 172.21.15.123:0/3919206734 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f003c0b2b00 0x7f003c0cc2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:32.491 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:32.488+0000 7f0041204640 1 -- 172.21.15.123:0/3919206734 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f003c0d2820 msgr2=0x7f003c1fab10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:47:32.491 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:32.488+0000 7f0041204640 1 --2- 172.21.15.123:0/3919206734 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f003c0d2820 0x7f003c1fab10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:32.491 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:32.488+0000 7f0041204640 1 -- 172.21.15.123:0/3919206734 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f003c0cdf40 con 0x7f003c161620 2024-09-16T15:47:32.491 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:32.488+0000 7f0040a03640 1 --2- 172.21.15.123:0/3919206734 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f003c0b2b00 0x7f003c0cc2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T15:47:32.491 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:32.488+0000 7f0041204640 1 --2- 172.21.15.123:0/3919206734 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f003c161620 0x7f003c156010 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f002c00c470 tx=0x7f002c00c930 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:47:32.503 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:32.500+0000 7f00397fa640 1 -- 172.21.15.123:0/3919206734 <== mon.1 v2:172.21.15.184:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f002c016020 con 0x7f003c161620 2024-09-16T15:47:32.503 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:32.500+0000 7f00397fa640 1 -- 172.21.15.123:0/3919206734 <== mon.1 v2:172.21.15.184:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f002c012920 con 0x7f003c161620 2024-09-16T15:47:32.503 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:32.500+0000 7f0042c67640 1 -- 172.21.15.123:0/3919206734 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f003c1fb210 con 0x7f003c161620 2024-09-16T15:47:32.503 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:32.500+0000 7f00397fa640 1 -- 172.21.15.123:0/3919206734 <== mon.1 v2:172.21.15.184:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f002c012d30 con 0x7f003c161620 2024-09-16T15:47:32.503 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:32.500+0000 7f0042c67640 1 -- 172.21.15.123:0/3919206734 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f003c1fb7d0 con 0x7f003c161620 2024-09-16T15:47:32.503 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:32.500+0000 7f0042c67640 1 -- 172.21.15.123:0/3919206734 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f003c0cea00 con 0x7f003c161620 2024-09-16T15:47:32.505 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:32.500+0000 7f00397fa640 1 -- 172.21.15.123:0/3919206734 <== mon.1 v2:172.21.15.184:3300/0 4 ==== mgrmap(e 12) ==== 97437+0+0 (secure 0 0 0) 0x7f002c019020 con 0x7f003c161620 2024-09-16T15:47:32.505 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:32.500+0000 7f00397fa640 1 --2- 172.21.15.123:0/3919206734 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f00280732b0 0x7f0028075770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:47:32.505 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:32.500+0000 7f00397fa640 1 -- 172.21.15.123:0/3919206734 <== mon.1 v2:172.21.15.184:3300/0 5 ==== osd_map(72..72 src has 1..72) ==== 5767+0+0 (secure 0 0 0) 0x7f002c094710 con 0x7f003c161620 2024-09-16T15:47:32.505 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:32.500+0000 7f0040a03640 1 --2- 172.21.15.123:0/3919206734 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f00280732b0 0x7f0028075770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:47:32.506 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:32.504+0000 7f0040a03640 1 --2- 172.21.15.123:0/3919206734 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f00280732b0 0x7f0028075770 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f00340ae000 tx=0x7f003401af10 comp rx=0 tx=0).ready entity=mgr.4135 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:47:32.517 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:32.512+0000 7f00397fa640 1 -- 172.21.15.123:0/3919206734 <== mon.1 v2:172.21.15.184:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f002c020070 con 0x7f003c161620 2024-09-16T15:47:32.731 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:32.728+0000 7f0042c67640 1 -- 172.21.15.123:0/3919206734 --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json-pretty"} v 0) -- 0x7f003c0d0720 con 0x7f003c161620 2024-09-16T15:47:32.744 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:32.744+0000 7f00397fa640 1 -- 172.21.15.123:0/3919206734 <== mon.1 v2:172.21.15.184:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json-pretty"}]=0 v12) ==== 81+0+506007 (secure 0 0 0) 0x7f002c05ece0 con 0x7f003c161620 2024-09-16T15:47:32.751 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-09-16T15:47:32.752 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2024-09-16T15:47:32.752 INFO:teuthology.orchestra.run.smithi123.stdout: "epoch": 12, 2024-09-16T15:47:32.752 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-09-16T15:47:32.752 INFO:teuthology.orchestra.run.smithi123.stdout: "active_gid": 4135, 2024-09-16T15:47:32.752 INFO:teuthology.orchestra.run.smithi123.stdout: "active_name": "x", 2024-09-16T15:47:32.752 INFO:teuthology.orchestra.run.smithi123.stdout: "active_addrs": { 2024-09-16T15:47:32.752 INFO:teuthology.orchestra.run.smithi123.stdout: "addrvec": [ 2024-09-16T15:47:32.752 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-09-16T15:47:32.752 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "v2", 2024-09-16T15:47:32.752 INFO:teuthology.orchestra.run.smithi123.stdout: "addr": "172.21.15.184:6832", 2024-09-16T15:47:32.752 INFO:teuthology.orchestra.run.smithi123.stdout: "nonce": 3114346423 2024-09-16T15:47:32.753 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:32.753 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-09-16T15:47:32.753 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "v1", 2024-09-16T15:47:32.753 INFO:teuthology.orchestra.run.smithi123.stdout: "addr": "172.21.15.184:6833", 2024-09-16T15:47:32.753 INFO:teuthology.orchestra.run.smithi123.stdout: "nonce": 3114346423 2024-09-16T15:47:32.753 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-16T15:47:32.753 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-09-16T15:47:32.753 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:32.753 INFO:teuthology.orchestra.run.smithi123.stdout: "active_addr": "172.21.15.184:6833/3114346423", 2024-09-16T15:47:32.753 INFO:teuthology.orchestra.run.smithi123.stdout: "active_change": "2024-09-16T15:42:31.230175+0000", 2024-09-16T15:47:32.753 INFO:teuthology.orchestra.run.smithi123.stdout: "active_mgr_features": 4540719139924082687, 2024-09-16T15:47:32.753 INFO:teuthology.orchestra.run.smithi123.stdout: "available": true, 2024-09-16T15:47:32.753 INFO:teuthology.orchestra.run.smithi123.stdout: "standbys": [ 2024-09-16T15:47:32.754 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-09-16T15:47:32.754 INFO:teuthology.orchestra.run.smithi123.stdout: "gid": 4150, 2024-09-16T15:47:32.754 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "y", 2024-09-16T15:47:32.754 INFO:teuthology.orchestra.run.smithi123.stdout: "mgr_features": 4540719139924082687, 2024-09-16T15:47:32.754 INFO:teuthology.orchestra.run.smithi123.stdout: "available_modules": [ 2024-09-16T15:47:32.754 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-09-16T15:47:32.754 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "alerts", 2024-09-16T15:47:32.754 INFO:teuthology.orchestra.run.smithi123.stdout: "can_run": true, 2024-09-16T15:47:32.754 INFO:teuthology.orchestra.run.smithi123.stdout: "error_string": "", 2024-09-16T15:47:32.754 INFO:teuthology.orchestra.run.smithi123.stdout: "module_options": { 2024-09-16T15:47:32.754 INFO:teuthology.orchestra.run.smithi123.stdout: "interval": { 2024-09-16T15:47:32.754 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "interval", 2024-09-16T15:47:32.754 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "secs", 2024-09-16T15:47:32.754 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:32.755 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-09-16T15:47:32.755 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "60", 2024-09-16T15:47:32.755 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:32.755 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:32.755 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:32.755 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "How frequently to reexamine health status", 2024-09-16T15:47:32.755 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:32.755 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:32.755 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:32.755 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:32.755 INFO:teuthology.orchestra.run.smithi123.stdout: "log_level": { 2024-09-16T15:47:32.755 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_level", 2024-09-16T15:47:32.755 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-09-16T15:47:32.755 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:32.755 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-09-16T15:47:32.756 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "", 2024-09-16T15:47:32.756 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:32.756 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:32.756 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [ 2024-09-16T15:47:32.756 INFO:teuthology.orchestra.run.smithi123.stdout: "", 2024-09-16T15:47:32.756 INFO:teuthology.orchestra.run.smithi123.stdout: "critical", 2024-09-16T15:47:32.756 INFO:teuthology.orchestra.run.smithi123.stdout: "debug", 2024-09-16T15:47:32.756 INFO:teuthology.orchestra.run.smithi123.stdout: "error", 2024-09-16T15:47:32.756 INFO:teuthology.orchestra.run.smithi123.stdout: "info", 2024-09-16T15:47:32.756 INFO:teuthology.orchestra.run.smithi123.stdout: "warning" 2024-09-16T15:47:32.756 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-09-16T15:47:32.756 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:32.756 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:32.756 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:32.757 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:32.757 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:32.757 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_cluster": { 2024-09-16T15:47:32.757 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_cluster", 2024-09-16T15:47:32.757 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-09-16T15:47:32.757 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:32.757 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-09-16T15:47:32.757 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-09-16T15:47:32.757 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:32.757 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:32.757 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:32.757 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:32.757 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:32.757 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:32.758 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:32.758 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:32.758 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_cluster_level": { 2024-09-16T15:47:32.758 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_cluster_level", 2024-09-16T15:47:32.758 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-09-16T15:47:32.758 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:32.758 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-09-16T15:47:32.758 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "info", 2024-09-16T15:47:32.758 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:32.758 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:32.758 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [ 2024-09-16T15:47:32.758 INFO:teuthology.orchestra.run.smithi123.stdout: "", 2024-09-16T15:47:32.758 INFO:teuthology.orchestra.run.smithi123.stdout: "critical", 2024-09-16T15:47:32.758 INFO:teuthology.orchestra.run.smithi123.stdout: "debug", 2024-09-16T15:47:32.759 INFO:teuthology.orchestra.run.smithi123.stdout: "error", 2024-09-16T15:47:32.759 INFO:teuthology.orchestra.run.smithi123.stdout: "info", 2024-09-16T15:47:32.759 INFO:teuthology.orchestra.run.smithi123.stdout: "warning" 2024-09-16T15:47:32.759 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-09-16T15:47:32.759 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:32.759 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:32.759 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:32.759 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:32.759 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:32.759 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_file": { 2024-09-16T15:47:32.759 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_file", 2024-09-16T15:47:32.759 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-09-16T15:47:32.759 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:32.759 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-09-16T15:47:32.759 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-09-16T15:47:32.760 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:32.760 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:32.760 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:32.760 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:32.760 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:32.760 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:32.760 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:32.760 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:32.760 INFO:teuthology.orchestra.run.smithi123.stdout: "smtp_destination": { 2024-09-16T15:47:32.760 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "smtp_destination", 2024-09-16T15:47:32.760 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-09-16T15:47:32.760 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:32.760 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-09-16T15:47:32.760 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "", 2024-09-16T15:47:32.760 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:32.761 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:32.761 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:32.761 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "Email address to send alerts to, use commas to separate multiple", 2024-09-16T15:47:32.761 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:32.761 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:32.761 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:32.761 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:32.761 INFO:teuthology.orchestra.run.smithi123.stdout: "smtp_from_name": { 2024-09-16T15:47:32.761 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "smtp_from_name", 2024-09-16T15:47:32.761 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-09-16T15:47:32.761 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:32.761 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-09-16T15:47:32.761 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "Ceph", 2024-09-16T15:47:32.761 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:32.761 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:32.761 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:32.762 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "Email From: name", 2024-09-16T15:47:32.762 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:32.762 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:32.762 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:32.762 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:32.762 INFO:teuthology.orchestra.run.smithi123.stdout: "smtp_host": { 2024-09-16T15:47:32.762 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "smtp_host", 2024-09-16T15:47:32.762 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-09-16T15:47:32.762 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:32.762 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-09-16T15:47:32.762 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "", 2024-09-16T15:47:32.762 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:32.762 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:32.762 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:32.763 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "SMTP server", 2024-09-16T15:47:32.763 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:32.763 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:32.763 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:32.763 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:32.763 INFO:teuthology.orchestra.run.smithi123.stdout: "smtp_password": { 2024-09-16T15:47:32.763 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "smtp_password", 2024-09-16T15:47:32.763 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-09-16T15:47:32.763 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:32.763 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-09-16T15:47:32.763 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "", 2024-09-16T15:47:32.763 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:32.763 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:32.763 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:32.763 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "Password to authenticate with", 2024-09-16T15:47:32.764 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:32.764 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:32.764 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:32.764 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:32.764 INFO:teuthology.orchestra.run.smithi123.stdout: "smtp_port": { 2024-09-16T15:47:32.764 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "smtp_port", 2024-09-16T15:47:32.764 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "int", 2024-09-16T15:47:32.764 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:32.764 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-09-16T15:47:32.764 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "465", 2024-09-16T15:47:32.764 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:32.764 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:32.764 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:32.764 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "SMTP port", 2024-09-16T15:47:32.764 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:32.765 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:32.765 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:32.765 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:32.765 INFO:teuthology.orchestra.run.smithi123.stdout: "smtp_sender": { 2024-09-16T15:47:32.765 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "smtp_sender", 2024-09-16T15:47:32.765 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-09-16T15:47:32.765 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:32.765 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-09-16T15:47:32.765 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "", 2024-09-16T15:47:32.765 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:32.765 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:32.765 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:32.765 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "SMTP envelope sender", 2024-09-16T15:47:32.765 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:32.766 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:32.766 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:32.766 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:32.766 INFO:teuthology.orchestra.run.smithi123.stdout: "smtp_ssl": { 2024-09-16T15:47:32.766 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "smtp_ssl", 2024-09-16T15:47:32.766 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-09-16T15:47:32.766 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:32.766 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-09-16T15:47:32.766 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "True", 2024-09-16T15:47:32.766 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:32.766 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:32.766 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:32.766 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "Use SSL to connect to SMTP server", 2024-09-16T15:47:32.767 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:32.767 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:32.767 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:32.767 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:32.767 INFO:teuthology.orchestra.run.smithi123.stdout: "smtp_user": { 2024-09-16T15:47:32.767 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "smtp_user", 2024-09-16T15:47:32.767 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-09-16T15:47:32.767 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:32.767 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-09-16T15:47:32.767 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "", 2024-09-16T15:47:32.767 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:32.767 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:32.767 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:32.767 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "User to authenticate as", 2024-09-16T15:47:32.768 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:32.768 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:32.768 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:32.768 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:32.768 INFO:teuthology.orchestra.run.smithi123.stdout: "sqlite3_killpoint": { 2024-09-16T15:47:32.768 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "sqlite3_killpoint", 2024-09-16T15:47:32.768 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "int", 2024-09-16T15:47:32.768 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "dev", 2024-09-16T15:47:32.768 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-09-16T15:47:32.768 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "0", 2024-09-16T15:47:32.768 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:32.768 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:32.768 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:32.768 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:32.768 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:32.769 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:32.769 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:32.769 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-16T15:47:32.769 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-16T15:47:32.769 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:32.769 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-09-16T15:47:32.769 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "balancer", 2024-09-16T15:47:32.769 INFO:teuthology.orchestra.run.smithi123.stdout: "can_run": true, 2024-09-16T15:47:32.769 INFO:teuthology.orchestra.run.smithi123.stdout: "error_string": "", 2024-09-16T15:47:32.769 INFO:teuthology.orchestra.run.smithi123.stdout: "module_options": { 2024-09-16T15:47:32.769 INFO:teuthology.orchestra.run.smithi123.stdout: "active": { 2024-09-16T15:47:32.769 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "active", 2024-09-16T15:47:32.769 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-09-16T15:47:32.769 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:32.770 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-09-16T15:47:32.770 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "True", 2024-09-16T15:47:32.770 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:32.770 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:32.770 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:32.770 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "automatically balance PGs across cluster", 2024-09-16T15:47:32.770 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:32.770 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:32.770 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:32.770 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:32.770 INFO:teuthology.orchestra.run.smithi123.stdout: "begin_time": { 2024-09-16T15:47:32.770 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "begin_time", 2024-09-16T15:47:32.770 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-09-16T15:47:32.770 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:32.770 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-09-16T15:47:32.771 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "0000", 2024-09-16T15:47:32.771 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:32.771 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:32.771 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:32.771 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "beginning time of day to automatically balance", 2024-09-16T15:47:32.771 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-09-16T15:47:32.771 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:32.771 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:32.771 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:32.771 INFO:teuthology.orchestra.run.smithi123.stdout: "begin_weekday": { 2024-09-16T15:47:32.771 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "begin_weekday", 2024-09-16T15:47:32.771 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "uint", 2024-09-16T15:47:32.771 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:32.771 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-09-16T15:47:32.771 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "0", 2024-09-16T15:47:32.772 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "0", 2024-09-16T15:47:32.772 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "6", 2024-09-16T15:47:32.772 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:32.772 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "Restrict automatic balancing to this day of the week or later", 2024-09-16T15:47:32.772 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-09-16T15:47:32.772 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:32.772 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:32.772 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:32.772 INFO:teuthology.orchestra.run.smithi123.stdout: "crush_compat_max_iterations": { 2024-09-16T15:47:32.772 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "crush_compat_max_iterations", 2024-09-16T15:47:32.772 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "uint", 2024-09-16T15:47:32.772 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:32.772 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-09-16T15:47:32.773 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "25", 2024-09-16T15:47:32.773 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "1", 2024-09-16T15:47:32.773 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "250", 2024-09-16T15:47:32.773 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:32.773 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "maximum number of iterations to attempt optimization", 2024-09-16T15:47:32.773 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:32.773 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:32.773 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:32.773 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:32.773 INFO:teuthology.orchestra.run.smithi123.stdout: "crush_compat_metrics": { 2024-09-16T15:47:32.773 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "crush_compat_metrics", 2024-09-16T15:47:32.773 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-09-16T15:47:32.773 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:32.773 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-09-16T15:47:32.774 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "pgs,objects,bytes", 2024-09-16T15:47:32.774 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:32.774 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:32.774 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:32.774 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "metrics with which to calculate OSD utilization", 2024-09-16T15:47:32.774 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.", 2024-09-16T15:47:32.774 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:32.774 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:32.774 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:32.774 INFO:teuthology.orchestra.run.smithi123.stdout: "crush_compat_step": { 2024-09-16T15:47:32.774 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "crush_compat_step", 2024-09-16T15:47:32.774 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "float", 2024-09-16T15:47:32.775 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:32.775 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-09-16T15:47:32.775 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "0.5", 2024-09-16T15:47:32.775 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "0.001", 2024-09-16T15:47:32.775 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "0.999", 2024-09-16T15:47:32.775 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:32.775 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "aggressiveness of optimization", 2024-09-16T15:47:32.775 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": ".99 is very aggressive, .01 is less aggressive", 2024-09-16T15:47:32.775 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:32.775 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:32.775 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:32.775 INFO:teuthology.orchestra.run.smithi123.stdout: "end_time": { 2024-09-16T15:47:32.775 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "end_time", 2024-09-16T15:47:32.776 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-09-16T15:47:32.776 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:32.776 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-09-16T15:47:32.776 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "2359", 2024-09-16T15:47:32.776 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:32.776 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:32.776 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:32.776 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "ending time of day to automatically balance", 2024-09-16T15:47:32.776 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-09-16T15:47:32.776 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:32.776 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:32.777 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:32.777 INFO:teuthology.orchestra.run.smithi123.stdout: "end_weekday": { 2024-09-16T15:47:32.777 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "end_weekday", 2024-09-16T15:47:32.777 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "uint", 2024-09-16T15:47:32.777 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:32.777 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-09-16T15:47:32.777 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "0", 2024-09-16T15:47:32.777 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "0", 2024-09-16T15:47:32.777 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "6", 2024-09-16T15:47:32.777 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:32.777 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "Restrict automatic balancing to days of the week earlier than this", 2024-09-16T15:47:32.777 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-09-16T15:47:32.777 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:32.778 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:32.778 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:32.778 INFO:teuthology.orchestra.run.smithi123.stdout: "log_level": { 2024-09-16T15:47:32.778 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_level", 2024-09-16T15:47:32.778 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-09-16T15:47:32.778 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:32.778 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-09-16T15:47:32.778 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "", 2024-09-16T15:47:32.778 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:32.778 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:32.778 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [ 2024-09-16T15:47:32.778 INFO:teuthology.orchestra.run.smithi123.stdout: "", 2024-09-16T15:47:32.778 INFO:teuthology.orchestra.run.smithi123.stdout: "critical", 2024-09-16T15:47:32.778 INFO:teuthology.orchestra.run.smithi123.stdout: "debug", 2024-09-16T15:47:32.779 INFO:teuthology.orchestra.run.smithi123.stdout: "error", 2024-09-16T15:47:32.779 INFO:teuthology.orchestra.run.smithi123.stdout: "info", 2024-09-16T15:47:32.779 INFO:teuthology.orchestra.run.smithi123.stdout: "warning" 2024-09-16T15:47:32.779 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-09-16T15:47:32.779 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:32.779 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:32.779 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:32.779 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:32.779 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:32.779 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_cluster": { 2024-09-16T15:47:32.779 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_cluster", 2024-09-16T15:47:32.779 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-09-16T15:47:32.779 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:32.779 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-09-16T15:47:32.779 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-09-16T15:47:32.780 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:32.780 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:32.780 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:32.780 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:32.780 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:32.780 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:32.780 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:32.780 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:32.780 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_cluster_level": { 2024-09-16T15:47:32.780 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_cluster_level", 2024-09-16T15:47:32.780 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-09-16T15:47:32.780 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:32.780 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-09-16T15:47:32.780 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "info", 2024-09-16T15:47:32.781 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:32.781 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:32.781 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [ 2024-09-16T15:47:32.781 INFO:teuthology.orchestra.run.smithi123.stdout: "", 2024-09-16T15:47:32.781 INFO:teuthology.orchestra.run.smithi123.stdout: "critical", 2024-09-16T15:47:32.781 INFO:teuthology.orchestra.run.smithi123.stdout: "debug", 2024-09-16T15:47:32.781 INFO:teuthology.orchestra.run.smithi123.stdout: "error", 2024-09-16T15:47:32.781 INFO:teuthology.orchestra.run.smithi123.stdout: "info", 2024-09-16T15:47:32.781 INFO:teuthology.orchestra.run.smithi123.stdout: "warning" 2024-09-16T15:47:32.781 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-09-16T15:47:32.781 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:32.781 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:32.781 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:32.781 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:32.782 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:32.782 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_file": { 2024-09-16T15:47:32.782 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_file", 2024-09-16T15:47:32.782 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-09-16T15:47:32.782 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:32.782 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-09-16T15:47:32.782 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-09-16T15:47:32.782 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:32.782 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:32.782 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:32.782 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:32.782 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:32.782 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:32.782 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:32.782 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:32.782 INFO:teuthology.orchestra.run.smithi123.stdout: "min_score": { 2024-09-16T15:47:32.783 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "min_score", 2024-09-16T15:47:32.783 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "float", 2024-09-16T15:47:32.783 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:32.783 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-09-16T15:47:32.783 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "0", 2024-09-16T15:47:32.783 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:32.783 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:32.783 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:32.783 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "minimum score, below which no optimization is attempted", 2024-09-16T15:47:32.783 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:32.783 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:32.783 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:32.783 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:32.783 INFO:teuthology.orchestra.run.smithi123.stdout: "mode": { 2024-09-16T15:47:32.783 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "mode", 2024-09-16T15:47:32.784 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-09-16T15:47:32.784 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:32.784 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-09-16T15:47:32.784 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "upmap", 2024-09-16T15:47:32.784 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:32.784 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:32.784 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [ 2024-09-16T15:47:32.784 INFO:teuthology.orchestra.run.smithi123.stdout: "crush-compat", 2024-09-16T15:47:32.784 INFO:teuthology.orchestra.run.smithi123.stdout: "none", 2024-09-16T15:47:32.785 INFO:teuthology.orchestra.run.smithi123.stdout: "read", 2024-09-16T15:47:32.785 INFO:teuthology.orchestra.run.smithi123.stdout: "upmap", 2024-09-16T15:47:32.785 INFO:teuthology.orchestra.run.smithi123.stdout: "upmap-read" 2024-09-16T15:47:32.785 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-09-16T15:47:32.785 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "Balancer mode", 2024-09-16T15:47:32.785 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:32.785 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:32.785 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:32.785 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:32.785 INFO:teuthology.orchestra.run.smithi123.stdout: "pool_ids": { 2024-09-16T15:47:32.785 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "pool_ids", 2024-09-16T15:47:32.785 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-09-16T15:47:32.785 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:32.785 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-09-16T15:47:32.786 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "", 2024-09-16T15:47:32.786 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:32.786 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:32.786 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:32.786 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "pools which the automatic balancing will be limited to", 2024-09-16T15:47:32.786 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:32.786 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:32.786 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:32.786 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:32.786 INFO:teuthology.orchestra.run.smithi123.stdout: "sleep_interval": { 2024-09-16T15:47:32.786 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "sleep_interval", 2024-09-16T15:47:32.786 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "secs", 2024-09-16T15:47:32.786 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:32.786 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-09-16T15:47:32.786 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "60", 2024-09-16T15:47:32.786 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:32.787 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:32.787 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:32.787 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "how frequently to wake up and attempt optimization", 2024-09-16T15:47:32.787 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:32.787 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:32.787 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:32.787 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:32.787 INFO:teuthology.orchestra.run.smithi123.stdout: "sqlite3_killpoint": { 2024-09-16T15:47:32.787 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "sqlite3_killpoint", 2024-09-16T15:47:32.787 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "int", 2024-09-16T15:47:32.787 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "dev", 2024-09-16T15:47:32.787 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-09-16T15:47:32.788 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "0", 2024-09-16T15:47:32.788 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:32.788 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:32.788 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:32.788 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:32.788 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:32.788 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:32.788 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:32.788 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:32.788 INFO:teuthology.orchestra.run.smithi123.stdout: "upmap_max_deviation": { 2024-09-16T15:47:32.788 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "upmap_max_deviation", 2024-09-16T15:47:32.788 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "int", 2024-09-16T15:47:32.788 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:32.788 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-09-16T15:47:32.789 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "5", 2024-09-16T15:47:32.789 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "1", 2024-09-16T15:47:32.789 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:32.789 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:32.789 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "deviation below which no optimization is attempted", 2024-09-16T15:47:32.789 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "If the number of PGs are within this count then no optimization is attempted", 2024-09-16T15:47:32.789 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:32.789 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:32.789 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:32.789 INFO:teuthology.orchestra.run.smithi123.stdout: "upmap_max_optimizations": { 2024-09-16T15:47:32.789 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "upmap_max_optimizations", 2024-09-16T15:47:32.789 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "uint", 2024-09-16T15:47:32.789 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:32.789 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-09-16T15:47:32.789 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "10", 2024-09-16T15:47:32.790 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:32.790 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:32.790 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:32.790 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "maximum upmap optimizations to make per attempt", 2024-09-16T15:47:32.790 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:32.790 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:32.790 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:32.790 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-16T15:47:32.790 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-16T15:47:32.790 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:32.790 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-09-16T15:47:32.790 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "cephadm", 2024-09-16T15:47:32.790 INFO:teuthology.orchestra.run.smithi123.stdout: "can_run": true, 2024-09-16T15:47:32.790 INFO:teuthology.orchestra.run.smithi123.stdout: "error_string": "", 2024-09-16T15:47:32.790 INFO:teuthology.orchestra.run.smithi123.stdout: "module_options": { 2024-09-16T15:47:32.791 INFO:teuthology.orchestra.run.smithi123.stdout: "agent_down_multiplier": { 2024-09-16T15:47:32.791 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "agent_down_multiplier", 2024-09-16T15:47:32.791 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "float", 2024-09-16T15:47:32.791 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:32.791 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-09-16T15:47:32.791 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "3.0", 2024-09-16T15:47:32.791 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:32.791 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:32.791 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:32.791 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "Multiplied by agent refresh rate to calculate how long agent must not report before being marked down", 2024-09-16T15:47:32.791 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:32.791 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:32.791 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:32.791 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:32.791 INFO:teuthology.orchestra.run.smithi123.stdout: "agent_refresh_rate": { 2024-09-16T15:47:32.792 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "agent_refresh_rate", 2024-09-16T15:47:32.792 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "secs", 2024-09-16T15:47:32.792 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:32.792 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-09-16T15:47:32.792 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "20", 2024-09-16T15:47:32.792 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:32.792 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:32.792 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:32.792 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "How often agent on each host will try to gather and send metadata", 2024-09-16T15:47:32.792 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:32.792 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:32.792 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:32.792 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:32.792 INFO:teuthology.orchestra.run.smithi123.stdout: "agent_starting_port": { 2024-09-16T15:47:32.792 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "agent_starting_port", 2024-09-16T15:47:32.793 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "int", 2024-09-16T15:47:32.793 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:32.793 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-09-16T15:47:32.793 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "4721", 2024-09-16T15:47:32.793 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:32.793 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:32.793 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:32.793 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)", 2024-09-16T15:47:32.793 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:32.793 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:32.793 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:32.793 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:32.793 INFO:teuthology.orchestra.run.smithi123.stdout: "allow_ptrace": { 2024-09-16T15:47:32.793 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "allow_ptrace", 2024-09-16T15:47:32.794 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-09-16T15:47:32.794 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:32.794 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-09-16T15:47:32.794 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-09-16T15:47:32.794 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:32.794 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:32.794 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:32.794 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "allow SYS_PTRACE capability on ceph containers", 2024-09-16T15:47:32.794 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.", 2024-09-16T15:47:32.794 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:32.794 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:32.794 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:32.794 INFO:teuthology.orchestra.run.smithi123.stdout: "autotune_interval": { 2024-09-16T15:47:32.794 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "autotune_interval", 2024-09-16T15:47:32.794 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "secs", 2024-09-16T15:47:32.795 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:32.795 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-09-16T15:47:32.795 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "600", 2024-09-16T15:47:32.795 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:32.795 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:32.795 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:32.795 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "how frequently to autotune daemon memory", 2024-09-16T15:47:32.795 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:32.795 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:32.795 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:32.795 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:32.795 INFO:teuthology.orchestra.run.smithi123.stdout: "autotune_memory_target_ratio": { 2024-09-16T15:47:32.795 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "autotune_memory_target_ratio", 2024-09-16T15:47:32.795 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "float", 2024-09-16T15:47:32.796 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:32.796 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-09-16T15:47:32.796 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "0.7", 2024-09-16T15:47:32.796 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:32.796 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:32.796 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:32.796 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "ratio of total system memory to divide amongst autotuned daemons", 2024-09-16T15:47:32.796 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:32.796 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:32.796 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:32.796 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:32.796 INFO:teuthology.orchestra.run.smithi123.stdout: "cephadm_log_destination": { 2024-09-16T15:47:32.796 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "cephadm_log_destination", 2024-09-16T15:47:32.796 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-09-16T15:47:32.797 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:32.797 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-09-16T15:47:32.797 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "", 2024-09-16T15:47:32.797 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:32.797 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:32.797 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [ 2024-09-16T15:47:32.797 INFO:teuthology.orchestra.run.smithi123.stdout: "file", 2024-09-16T15:47:32.797 INFO:teuthology.orchestra.run.smithi123.stdout: "file,syslog", 2024-09-16T15:47:32.797 INFO:teuthology.orchestra.run.smithi123.stdout: "syslog" 2024-09-16T15:47:32.797 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-09-16T15:47:32.797 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "Destination for cephadm command's persistent logging", 2024-09-16T15:47:32.797 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:32.797 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:32.798 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:32.798 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:32.798 INFO:teuthology.orchestra.run.smithi123.stdout: "cgroups_split": { 2024-09-16T15:47:32.798 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "cgroups_split", 2024-09-16T15:47:32.798 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-09-16T15:47:32.798 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:32.798 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-09-16T15:47:32.798 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "True", 2024-09-16T15:47:32.798 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:32.798 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:32.798 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:32.798 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "Pass --cgroups=split when cephadm creates containers (currently podman only)", 2024-09-16T15:47:32.798 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:32.798 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:32.799 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:32.799 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:32.799 INFO:teuthology.orchestra.run.smithi123.stdout: "config_checks_enabled": { 2024-09-16T15:47:32.799 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "config_checks_enabled", 2024-09-16T15:47:32.799 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-09-16T15:47:32.799 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:32.799 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-09-16T15:47:32.799 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-09-16T15:47:32.799 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:32.799 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:32.799 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:32.799 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "Enable or disable the cephadm configuration analysis", 2024-09-16T15:47:32.799 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:32.800 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:32.800 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:32.800 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:32.800 INFO:teuthology.orchestra.run.smithi123.stdout: "config_dashboard": { 2024-09-16T15:47:32.800 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "config_dashboard", 2024-09-16T15:47:32.800 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-09-16T15:47:32.800 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:32.800 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-09-16T15:47:32.800 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "True", 2024-09-16T15:47:32.800 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:32.800 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:32.800 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:32.800 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "manage configs like API endpoints in Dashboard.", 2024-09-16T15:47:32.801 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:32.801 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:32.801 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:32.801 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:32.801 INFO:teuthology.orchestra.run.smithi123.stdout: "container_image_alertmanager": { 2024-09-16T15:47:32.801 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "container_image_alertmanager", 2024-09-16T15:47:32.801 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-09-16T15:47:32.801 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:32.801 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-09-16T15:47:32.801 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "quay.io/prometheus/alertmanager:v0.27.0", 2024-09-16T15:47:32.801 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:32.801 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:32.801 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:32.801 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "Prometheus container image", 2024-09-16T15:47:32.802 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:32.802 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:32.802 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:32.802 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:32.802 INFO:teuthology.orchestra.run.smithi123.stdout: "container_image_base": { 2024-09-16T15:47:32.802 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "container_image_base", 2024-09-16T15:47:32.802 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-09-16T15:47:32.802 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:32.802 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-09-16T15:47:32.802 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "quay.io/ceph/ceph", 2024-09-16T15:47:32.802 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:32.802 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:32.803 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:32.803 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "Container image name, without the tag", 2024-09-16T15:47:32.803 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:32.803 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:32.803 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:32.803 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:32.803 INFO:teuthology.orchestra.run.smithi123.stdout: "container_image_elasticsearch": { 2024-09-16T15:47:32.803 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "container_image_elasticsearch", 2024-09-16T15:47:32.803 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-09-16T15:47:32.803 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:32.803 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-09-16T15:47:32.803 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "quay.io/omrizeneva/elasticsearch:6.8.23", 2024-09-16T15:47:32.803 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:32.803 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:32.804 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:32.804 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "elasticsearch container image", 2024-09-16T15:47:32.804 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:32.804 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:32.804 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:32.804 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:32.804 INFO:teuthology.orchestra.run.smithi123.stdout: "container_image_grafana": { 2024-09-16T15:47:32.804 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "container_image_grafana", 2024-09-16T15:47:32.804 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-09-16T15:47:32.804 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:32.804 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-09-16T15:47:32.804 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "quay.io/ceph/grafana:10.4.0", 2024-09-16T15:47:32.804 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:32.804 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:32.805 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:32.805 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "Prometheus container image", 2024-09-16T15:47:32.805 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:32.805 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:32.805 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:32.805 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:32.805 INFO:teuthology.orchestra.run.smithi123.stdout: "container_image_haproxy": { 2024-09-16T15:47:32.805 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "container_image_haproxy", 2024-09-16T15:47:32.805 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-09-16T15:47:32.805 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:32.805 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-09-16T15:47:32.805 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "quay.io/ceph/haproxy:2.3", 2024-09-16T15:47:32.805 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:32.805 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:32.805 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:32.806 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "HAproxy container image", 2024-09-16T15:47:32.806 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:32.806 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:32.806 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:32.806 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:32.806 INFO:teuthology.orchestra.run.smithi123.stdout: "container_image_jaeger_agent": { 2024-09-16T15:47:32.806 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "container_image_jaeger_agent", 2024-09-16T15:47:32.806 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-09-16T15:47:32.806 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:32.806 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-09-16T15:47:32.806 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "quay.io/jaegertracing/jaeger-agent:1.29", 2024-09-16T15:47:32.806 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:32.806 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:32.807 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:32.807 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "Jaeger agent container image", 2024-09-16T15:47:32.807 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:32.807 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:32.807 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:32.807 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:32.807 INFO:teuthology.orchestra.run.smithi123.stdout: "container_image_jaeger_collector": { 2024-09-16T15:47:32.807 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "container_image_jaeger_collector", 2024-09-16T15:47:32.807 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-09-16T15:47:32.807 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:32.807 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-09-16T15:47:32.807 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "quay.io/jaegertracing/jaeger-collector:1.29", 2024-09-16T15:47:32.807 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:32.807 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:32.808 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:32.808 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "Jaeger collector container image", 2024-09-16T15:47:32.808 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:32.808 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:32.808 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:32.808 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:32.808 INFO:teuthology.orchestra.run.smithi123.stdout: "container_image_jaeger_query": { 2024-09-16T15:47:32.808 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "container_image_jaeger_query", 2024-09-16T15:47:32.808 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-09-16T15:47:32.808 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:32.808 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-09-16T15:47:32.808 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "quay.io/jaegertracing/jaeger-query:1.29", 2024-09-16T15:47:32.808 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:32.808 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:32.808 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:32.809 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "Jaeger query container image", 2024-09-16T15:47:32.809 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:32.809 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:32.809 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:32.809 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:32.809 INFO:teuthology.orchestra.run.smithi123.stdout: "container_image_keepalived": { 2024-09-16T15:47:32.809 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "container_image_keepalived", 2024-09-16T15:47:32.809 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-09-16T15:47:32.809 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:32.809 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-09-16T15:47:32.809 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "quay.io/ceph/keepalived:2.2.4", 2024-09-16T15:47:32.809 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:32.809 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:32.809 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:32.810 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "Keepalived container image", 2024-09-16T15:47:32.810 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:32.810 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:32.810 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:32.810 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:32.810 INFO:teuthology.orchestra.run.smithi123.stdout: "container_image_loki": { 2024-09-16T15:47:32.810 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "container_image_loki", 2024-09-16T15:47:32.810 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-09-16T15:47:32.810 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:32.810 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-09-16T15:47:32.810 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "docker.io/grafana/loki:3.0.0", 2024-09-16T15:47:32.810 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:32.810 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:32.810 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:32.810 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "Loki container image", 2024-09-16T15:47:32.811 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:32.811 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:32.811 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:32.811 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:32.811 INFO:teuthology.orchestra.run.smithi123.stdout: "container_image_nginx": { 2024-09-16T15:47:32.811 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "container_image_nginx", 2024-09-16T15:47:32.811 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-09-16T15:47:32.811 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:32.811 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-09-16T15:47:32.811 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "quay.io/ceph/nginx:sclorg-nginx-126", 2024-09-16T15:47:32.811 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:32.811 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:32.811 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:32.811 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "Nginx container image", 2024-09-16T15:47:32.811 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:32.812 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:32.812 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:32.812 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:32.812 INFO:teuthology.orchestra.run.smithi123.stdout: "container_image_node_exporter": { 2024-09-16T15:47:32.812 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "container_image_node_exporter", 2024-09-16T15:47:32.812 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-09-16T15:47:32.812 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:32.812 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-09-16T15:47:32.812 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "quay.io/prometheus/node-exporter:v1.7.0", 2024-09-16T15:47:32.812 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:32.812 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:32.812 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:32.812 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "Prometheus container image", 2024-09-16T15:47:32.812 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:32.812 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:32.813 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:32.813 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:32.813 INFO:teuthology.orchestra.run.smithi123.stdout: "container_image_nvmeof": { 2024-09-16T15:47:32.813 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "container_image_nvmeof", 2024-09-16T15:47:32.813 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-09-16T15:47:32.813 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:32.813 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-09-16T15:47:32.813 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "quay.io/ceph/nvmeof:1.2.17", 2024-09-16T15:47:32.813 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:32.813 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:32.813 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:32.813 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "Nvme-of container image", 2024-09-16T15:47:32.813 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:32.813 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:32.813 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:32.814 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:32.814 INFO:teuthology.orchestra.run.smithi123.stdout: "container_image_oauth2_proxy": { 2024-09-16T15:47:32.814 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "container_image_oauth2_proxy", 2024-09-16T15:47:32.814 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-09-16T15:47:32.814 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:32.814 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-09-16T15:47:32.814 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "quay.io/oauth2-proxy/oauth2-proxy:v7.6.0", 2024-09-16T15:47:32.814 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:32.814 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:32.814 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:32.814 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "oauth2-proxy container image", 2024-09-16T15:47:32.814 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:32.814 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:32.814 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:32.814 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:32.814 INFO:teuthology.orchestra.run.smithi123.stdout: "container_image_prometheus": { 2024-09-16T15:47:32.815 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "container_image_prometheus", 2024-09-16T15:47:32.815 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-09-16T15:47:32.815 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:32.815 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-09-16T15:47:32.815 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "quay.io/prometheus/prometheus:v2.51.0", 2024-09-16T15:47:32.815 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:32.815 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:32.815 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:32.815 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "Prometheus container image", 2024-09-16T15:47:32.815 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:32.815 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:32.815 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:32.815 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:32.815 INFO:teuthology.orchestra.run.smithi123.stdout: "container_image_promtail": { 2024-09-16T15:47:32.816 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "container_image_promtail", 2024-09-16T15:47:32.816 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-09-16T15:47:32.816 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:32.816 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-09-16T15:47:32.816 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "docker.io/grafana/promtail:3.0.0", 2024-09-16T15:47:32.816 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:32.816 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:32.816 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:32.816 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "Promtail container image", 2024-09-16T15:47:32.816 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:32.816 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:32.816 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:32.816 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:32.816 INFO:teuthology.orchestra.run.smithi123.stdout: "container_image_samba": { 2024-09-16T15:47:32.816 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "container_image_samba", 2024-09-16T15:47:32.817 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-09-16T15:47:32.817 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:32.817 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-09-16T15:47:32.817 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "quay.io/samba.org/samba-server:devbuilds-centos-amd64", 2024-09-16T15:47:32.817 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:32.817 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:32.817 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:32.817 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "Samba/SMB container image", 2024-09-16T15:47:32.817 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:32.817 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:32.817 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:32.817 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:32.817 INFO:teuthology.orchestra.run.smithi123.stdout: "container_image_samba_metrics": { 2024-09-16T15:47:32.817 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "container_image_samba_metrics", 2024-09-16T15:47:32.817 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-09-16T15:47:32.818 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:32.818 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-09-16T15:47:32.818 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "quay.io/samba.org/samba-metrics:latest", 2024-09-16T15:47:32.818 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:32.818 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:32.818 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:32.818 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "Samba/SMB metrics exporter container image", 2024-09-16T15:47:32.818 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:32.818 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:32.818 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:32.818 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:32.818 INFO:teuthology.orchestra.run.smithi123.stdout: "container_image_snmp_gateway": { 2024-09-16T15:47:32.818 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "container_image_snmp_gateway", 2024-09-16T15:47:32.818 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-09-16T15:47:32.818 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:32.819 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-09-16T15:47:32.819 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "docker.io/maxwo/snmp-notifier:v1.2.1", 2024-09-16T15:47:32.819 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:32.819 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:32.819 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:32.819 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "SNMP Gateway container image", 2024-09-16T15:47:32.819 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:32.819 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:32.819 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:32.819 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:32.819 INFO:teuthology.orchestra.run.smithi123.stdout: "container_init": { 2024-09-16T15:47:32.819 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "container_init", 2024-09-16T15:47:32.819 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-09-16T15:47:32.819 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:32.819 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-09-16T15:47:32.820 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "True", 2024-09-16T15:47:32.820 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:32.820 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:32.820 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:32.820 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "Run podman/docker with `--init`", 2024-09-16T15:47:32.820 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:32.820 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:32.820 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:32.820 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:32.820 INFO:teuthology.orchestra.run.smithi123.stdout: "daemon_cache_timeout": { 2024-09-16T15:47:32.820 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "daemon_cache_timeout", 2024-09-16T15:47:32.820 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "secs", 2024-09-16T15:47:32.820 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:32.820 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-09-16T15:47:32.820 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "600", 2024-09-16T15:47:32.821 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:32.821 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:32.821 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:32.821 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "seconds to cache service (daemon) inventory", 2024-09-16T15:47:32.821 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:32.821 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:32.821 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:32.821 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:32.821 INFO:teuthology.orchestra.run.smithi123.stdout: "default_cephadm_command_timeout": { 2024-09-16T15:47:32.821 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "default_cephadm_command_timeout", 2024-09-16T15:47:32.821 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "int", 2024-09-16T15:47:32.821 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:32.821 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-09-16T15:47:32.821 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "900", 2024-09-16T15:47:32.821 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:32.821 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:32.822 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:32.822 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "Default timeout applied to cephadm commands run directly on the host (in seconds)", 2024-09-16T15:47:32.822 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:32.822 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:32.822 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:32.822 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:32.822 INFO:teuthology.orchestra.run.smithi123.stdout: "default_registry": { 2024-09-16T15:47:32.822 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "default_registry", 2024-09-16T15:47:32.822 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-09-16T15:47:32.822 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:32.822 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-09-16T15:47:32.822 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "docker.io", 2024-09-16T15:47:32.822 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:32.822 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:32.822 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:32.823 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "Search-registry to which we should normalize unqualified image names. This is not the default registry", 2024-09-16T15:47:32.823 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:32.823 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:32.823 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:32.823 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:32.823 INFO:teuthology.orchestra.run.smithi123.stdout: "device_cache_timeout": { 2024-09-16T15:47:32.823 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "device_cache_timeout", 2024-09-16T15:47:32.823 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "secs", 2024-09-16T15:47:32.823 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:32.823 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-09-16T15:47:32.823 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "1800", 2024-09-16T15:47:32.823 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:32.823 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:32.823 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:32.823 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "seconds to cache device inventory", 2024-09-16T15:47:32.823 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:32.824 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:32.824 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:32.824 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:32.824 INFO:teuthology.orchestra.run.smithi123.stdout: "device_enhanced_scan": { 2024-09-16T15:47:32.824 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "device_enhanced_scan", 2024-09-16T15:47:32.824 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-09-16T15:47:32.824 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:32.824 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-09-16T15:47:32.824 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-09-16T15:47:32.824 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:32.824 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:32.824 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:32.824 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "Use libstoragemgmt during device scans", 2024-09-16T15:47:32.824 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:32.824 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:32.824 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:32.825 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:32.825 INFO:teuthology.orchestra.run.smithi123.stdout: "facts_cache_timeout": { 2024-09-16T15:47:32.825 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "facts_cache_timeout", 2024-09-16T15:47:32.825 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "secs", 2024-09-16T15:47:32.825 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:32.825 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-09-16T15:47:32.825 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "60", 2024-09-16T15:47:32.825 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:32.825 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:32.825 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:32.825 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "seconds to cache host facts data", 2024-09-16T15:47:32.825 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:32.825 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:32.825 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:32.825 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:32.826 INFO:teuthology.orchestra.run.smithi123.stdout: "grafana_dashboards_path": { 2024-09-16T15:47:32.826 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "grafana_dashboards_path", 2024-09-16T15:47:32.826 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-09-16T15:47:32.826 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:32.826 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-09-16T15:47:32.826 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "/etc/grafana/dashboards/ceph-dashboard/", 2024-09-16T15:47:32.826 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:32.826 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:32.826 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:32.826 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "location of dashboards to include in grafana deployments", 2024-09-16T15:47:32.826 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:32.826 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:32.826 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:32.826 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:32.826 INFO:teuthology.orchestra.run.smithi123.stdout: "host_check_interval": { 2024-09-16T15:47:32.827 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "host_check_interval", 2024-09-16T15:47:32.827 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "secs", 2024-09-16T15:47:32.827 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:32.827 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-09-16T15:47:32.827 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "600", 2024-09-16T15:47:32.827 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:32.827 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:32.827 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:32.827 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "how frequently to perform a host check", 2024-09-16T15:47:32.827 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:32.827 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:32.827 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:32.827 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:32.827 INFO:teuthology.orchestra.run.smithi123.stdout: "hw_monitoring": { 2024-09-16T15:47:32.827 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "hw_monitoring", 2024-09-16T15:47:32.828 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-09-16T15:47:32.828 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:32.828 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-09-16T15:47:32.828 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-09-16T15:47:32.828 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:32.828 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:32.828 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:32.828 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "Deploy hw monitoring daemon on every host.", 2024-09-16T15:47:32.828 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:32.828 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:32.828 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:32.828 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:32.828 INFO:teuthology.orchestra.run.smithi123.stdout: "inventory_list_all": { 2024-09-16T15:47:32.829 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "inventory_list_all", 2024-09-16T15:47:32.829 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-09-16T15:47:32.829 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:32.829 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-09-16T15:47:32.829 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-09-16T15:47:32.829 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:32.829 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:32.829 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:32.829 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)", 2024-09-16T15:47:32.829 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:32.829 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:32.829 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:32.829 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:32.829 INFO:teuthology.orchestra.run.smithi123.stdout: "log_level": { 2024-09-16T15:47:32.830 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_level", 2024-09-16T15:47:32.830 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-09-16T15:47:32.830 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:32.830 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-09-16T15:47:32.830 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "", 2024-09-16T15:47:32.830 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:32.830 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:32.830 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [ 2024-09-16T15:47:32.830 INFO:teuthology.orchestra.run.smithi123.stdout: "", 2024-09-16T15:47:32.830 INFO:teuthology.orchestra.run.smithi123.stdout: "critical", 2024-09-16T15:47:32.830 INFO:teuthology.orchestra.run.smithi123.stdout: "debug", 2024-09-16T15:47:32.830 INFO:teuthology.orchestra.run.smithi123.stdout: "error", 2024-09-16T15:47:32.830 INFO:teuthology.orchestra.run.smithi123.stdout: "info", 2024-09-16T15:47:32.830 INFO:teuthology.orchestra.run.smithi123.stdout: "warning" 2024-09-16T15:47:32.830 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-09-16T15:47:32.831 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:32.831 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:32.831 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:32.831 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:32.831 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:32.831 INFO:teuthology.orchestra.run.smithi123.stdout: "log_refresh_metadata": { 2024-09-16T15:47:32.831 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_refresh_metadata", 2024-09-16T15:47:32.831 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-09-16T15:47:32.831 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:32.831 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-09-16T15:47:32.831 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-09-16T15:47:32.831 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:32.831 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:32.831 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:32.832 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level", 2024-09-16T15:47:32.832 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:32.832 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:32.832 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:32.832 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:32.832 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_cluster": { 2024-09-16T15:47:32.832 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_cluster", 2024-09-16T15:47:32.832 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-09-16T15:47:32.832 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:32.832 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-09-16T15:47:32.832 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "True", 2024-09-16T15:47:32.832 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:32.833 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:32.833 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:32.833 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "log to the \"cephadm\" cluster log channel\"", 2024-09-16T15:47:32.833 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:32.833 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:32.833 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:32.833 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:32.833 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_cluster_level": { 2024-09-16T15:47:32.833 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_cluster_level", 2024-09-16T15:47:32.833 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-09-16T15:47:32.833 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:32.833 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-09-16T15:47:32.833 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "info", 2024-09-16T15:47:32.834 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:32.834 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:32.834 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [ 2024-09-16T15:47:32.834 INFO:teuthology.orchestra.run.smithi123.stdout: "", 2024-09-16T15:47:32.834 INFO:teuthology.orchestra.run.smithi123.stdout: "critical", 2024-09-16T15:47:32.834 INFO:teuthology.orchestra.run.smithi123.stdout: "debug", 2024-09-16T15:47:32.834 INFO:teuthology.orchestra.run.smithi123.stdout: "error", 2024-09-16T15:47:32.834 INFO:teuthology.orchestra.run.smithi123.stdout: "info", 2024-09-16T15:47:32.834 INFO:teuthology.orchestra.run.smithi123.stdout: "warning" 2024-09-16T15:47:32.834 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-09-16T15:47:32.834 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:32.834 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:32.834 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:32.834 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:32.835 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:32.835 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_file": { 2024-09-16T15:47:32.835 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_file", 2024-09-16T15:47:32.835 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-09-16T15:47:32.835 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:32.835 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-09-16T15:47:32.835 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-09-16T15:47:32.835 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:32.835 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:32.835 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:32.835 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:32.835 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:32.835 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:32.835 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:32.835 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:32.836 INFO:teuthology.orchestra.run.smithi123.stdout: "manage_etc_ceph_ceph_conf": { 2024-09-16T15:47:32.836 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "manage_etc_ceph_ceph_conf", 2024-09-16T15:47:32.836 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-09-16T15:47:32.836 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:32.836 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-09-16T15:47:32.836 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-09-16T15:47:32.836 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:32.836 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:32.836 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:32.836 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "Manage and own /etc/ceph/ceph.conf on the hosts.", 2024-09-16T15:47:32.836 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:32.836 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:32.836 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:32.836 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:32.837 INFO:teuthology.orchestra.run.smithi123.stdout: "manage_etc_ceph_ceph_conf_hosts": { 2024-09-16T15:47:32.837 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "manage_etc_ceph_ceph_conf_hosts", 2024-09-16T15:47:32.837 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-09-16T15:47:32.837 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:32.837 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-09-16T15:47:32.837 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "*", 2024-09-16T15:47:32.837 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:32.837 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:32.837 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:32.837 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf", 2024-09-16T15:47:32.837 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:32.837 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:32.837 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:32.837 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:32.837 INFO:teuthology.orchestra.run.smithi123.stdout: "max_count_per_host": { 2024-09-16T15:47:32.838 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "max_count_per_host", 2024-09-16T15:47:32.838 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "int", 2024-09-16T15:47:32.838 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:32.838 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-09-16T15:47:32.838 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "10", 2024-09-16T15:47:32.838 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:32.838 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:32.838 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:32.838 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "max number of daemons per service per host", 2024-09-16T15:47:32.838 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:32.838 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:32.838 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:32.838 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:32.838 INFO:teuthology.orchestra.run.smithi123.stdout: "max_osd_draining_count": { 2024-09-16T15:47:32.839 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "max_osd_draining_count", 2024-09-16T15:47:32.839 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "int", 2024-09-16T15:47:32.839 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:32.839 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-09-16T15:47:32.839 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "10", 2024-09-16T15:47:32.839 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:32.839 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:32.839 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:32.839 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "max number of osds that will be drained simultaneously when osds are removed", 2024-09-16T15:47:32.839 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:32.839 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:32.839 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:32.839 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:32.839 INFO:teuthology.orchestra.run.smithi123.stdout: "migration_current": { 2024-09-16T15:47:32.840 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "migration_current", 2024-09-16T15:47:32.840 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "int", 2024-09-16T15:47:32.840 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:32.840 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-09-16T15:47:32.840 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "", 2024-09-16T15:47:32.840 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:32.840 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:32.840 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:32.840 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "internal - do not modify", 2024-09-16T15:47:32.840 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:32.840 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:32.840 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:32.840 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:32.840 INFO:teuthology.orchestra.run.smithi123.stdout: "mode": { 2024-09-16T15:47:32.840 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "mode", 2024-09-16T15:47:32.840 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-09-16T15:47:32.841 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:32.841 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-09-16T15:47:32.841 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "root", 2024-09-16T15:47:32.841 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:32.841 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:32.841 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [ 2024-09-16T15:47:32.841 INFO:teuthology.orchestra.run.smithi123.stdout: "cephadm-package", 2024-09-16T15:47:32.841 INFO:teuthology.orchestra.run.smithi123.stdout: "root" 2024-09-16T15:47:32.841 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-09-16T15:47:32.841 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "mode for remote execution of cephadm", 2024-09-16T15:47:32.841 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:32.841 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:32.841 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:32.841 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:32.841 INFO:teuthology.orchestra.run.smithi123.stdout: "oob_default_addr": { 2024-09-16T15:47:32.842 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "oob_default_addr", 2024-09-16T15:47:32.842 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-09-16T15:47:32.842 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:32.842 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-09-16T15:47:32.842 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "169.254.1.1", 2024-09-16T15:47:32.842 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:32.842 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:32.842 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:32.842 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "Default address for RedFish API (oob management).", 2024-09-16T15:47:32.842 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:32.842 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:32.842 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:32.842 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:32.842 INFO:teuthology.orchestra.run.smithi123.stdout: "prometheus_alerts_path": { 2024-09-16T15:47:32.842 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "prometheus_alerts_path", 2024-09-16T15:47:32.843 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-09-16T15:47:32.843 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:32.843 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-09-16T15:47:32.843 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "/etc/prometheus/ceph/ceph_default_alerts.yml", 2024-09-16T15:47:32.843 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:32.843 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:32.843 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:32.843 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "location of alerts to include in prometheus deployments", 2024-09-16T15:47:32.843 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:32.843 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:32.843 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:32.843 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:32.843 INFO:teuthology.orchestra.run.smithi123.stdout: "registry_insecure": { 2024-09-16T15:47:32.843 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "registry_insecure", 2024-09-16T15:47:32.844 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-09-16T15:47:32.844 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:32.844 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-09-16T15:47:32.844 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-09-16T15:47:32.844 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:32.844 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:32.844 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:32.844 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "Registry is to be considered insecure (no TLS available). Only for development purposes.", 2024-09-16T15:47:32.844 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:32.844 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:32.844 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:32.844 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:32.844 INFO:teuthology.orchestra.run.smithi123.stdout: "registry_password": { 2024-09-16T15:47:32.844 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "registry_password", 2024-09-16T15:47:32.845 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-09-16T15:47:32.845 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:32.845 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-09-16T15:47:32.845 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "", 2024-09-16T15:47:32.845 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:32.845 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:32.845 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:32.845 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "Custom repository password. Only used for logging into a registry.", 2024-09-16T15:47:32.845 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:32.845 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:32.845 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:32.845 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:32.845 INFO:teuthology.orchestra.run.smithi123.stdout: "registry_url": { 2024-09-16T15:47:32.845 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "registry_url", 2024-09-16T15:47:32.846 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-09-16T15:47:32.846 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:32.846 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-09-16T15:47:32.846 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "", 2024-09-16T15:47:32.846 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:32.846 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:32.846 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:32.846 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "Registry url for login purposes. This is not the default registry", 2024-09-16T15:47:32.846 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:32.846 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:32.846 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:32.846 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:32.846 INFO:teuthology.orchestra.run.smithi123.stdout: "registry_username": { 2024-09-16T15:47:32.846 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "registry_username", 2024-09-16T15:47:32.846 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-09-16T15:47:32.846 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:32.847 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-09-16T15:47:32.847 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "", 2024-09-16T15:47:32.847 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:32.847 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:32.847 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:32.847 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "Custom repository username. Only used for logging into a registry.", 2024-09-16T15:47:32.847 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:32.847 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:32.847 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:32.847 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:32.847 INFO:teuthology.orchestra.run.smithi123.stdout: "secure_monitoring_stack": { 2024-09-16T15:47:32.847 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "secure_monitoring_stack", 2024-09-16T15:47:32.847 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-09-16T15:47:32.847 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:32.848 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-09-16T15:47:32.848 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-09-16T15:47:32.848 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:32.848 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:32.848 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:32.848 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "Enable TLS security for all the monitoring stack daemons", 2024-09-16T15:47:32.848 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:32.848 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:32.848 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:32.848 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:32.848 INFO:teuthology.orchestra.run.smithi123.stdout: "service_discovery_port": { 2024-09-16T15:47:32.848 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "service_discovery_port", 2024-09-16T15:47:32.848 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "int", 2024-09-16T15:47:32.848 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:32.848 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-09-16T15:47:32.849 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "8765", 2024-09-16T15:47:32.849 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:32.849 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:32.849 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:32.849 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "cephadm service discovery port", 2024-09-16T15:47:32.849 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:32.849 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:32.849 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:32.849 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:32.849 INFO:teuthology.orchestra.run.smithi123.stdout: "sqlite3_killpoint": { 2024-09-16T15:47:32.849 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "sqlite3_killpoint", 2024-09-16T15:47:32.849 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "int", 2024-09-16T15:47:32.849 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "dev", 2024-09-16T15:47:32.849 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-09-16T15:47:32.849 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "0", 2024-09-16T15:47:32.850 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:32.850 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:32.850 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:32.850 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:32.850 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:32.850 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:32.850 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:32.850 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:32.850 INFO:teuthology.orchestra.run.smithi123.stdout: "ssh_config_file": { 2024-09-16T15:47:32.850 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "ssh_config_file", 2024-09-16T15:47:32.850 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-09-16T15:47:32.850 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:32.850 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-09-16T15:47:32.850 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "", 2024-09-16T15:47:32.851 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:32.851 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:32.851 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:32.851 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "customized SSH config file to connect to managed hosts", 2024-09-16T15:47:32.851 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:32.851 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:32.851 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:32.851 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:32.851 INFO:teuthology.orchestra.run.smithi123.stdout: "ssh_keepalive_count_max": { 2024-09-16T15:47:32.851 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "ssh_keepalive_count_max", 2024-09-16T15:47:32.851 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "int", 2024-09-16T15:47:32.851 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:32.851 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-09-16T15:47:32.851 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "3", 2024-09-16T15:47:32.851 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:32.851 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:32.852 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:32.852 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "How many times ssh connections can fail liveness checks before the host is marked offline", 2024-09-16T15:47:32.852 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:32.852 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:32.852 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:32.852 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:32.852 INFO:teuthology.orchestra.run.smithi123.stdout: "ssh_keepalive_interval": { 2024-09-16T15:47:32.852 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "ssh_keepalive_interval", 2024-09-16T15:47:32.852 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "int", 2024-09-16T15:47:32.852 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:32.852 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-09-16T15:47:32.852 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "7", 2024-09-16T15:47:32.852 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:32.852 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:32.852 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:32.853 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "How often ssh connections are checked for liveness", 2024-09-16T15:47:32.853 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:32.853 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:32.853 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:32.853 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:32.853 INFO:teuthology.orchestra.run.smithi123.stdout: "use_agent": { 2024-09-16T15:47:32.853 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "use_agent", 2024-09-16T15:47:32.853 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-09-16T15:47:32.853 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:32.853 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-09-16T15:47:32.853 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-09-16T15:47:32.853 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:32.853 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:32.853 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:32.853 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "Use cephadm agent on each host to gather and send metadata", 2024-09-16T15:47:32.854 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:32.854 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:32.854 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:32.854 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:32.854 INFO:teuthology.orchestra.run.smithi123.stdout: "use_repo_digest": { 2024-09-16T15:47:32.854 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "use_repo_digest", 2024-09-16T15:47:32.854 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-09-16T15:47:32.854 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:32.854 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-09-16T15:47:32.854 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "True", 2024-09-16T15:47:32.854 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:32.854 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:32.854 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:32.854 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "Automatically convert image tags to image digest. Make sure all daemons use the same image", 2024-09-16T15:47:32.854 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:32.855 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:32.855 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:32.855 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:32.855 INFO:teuthology.orchestra.run.smithi123.stdout: "warn_on_failed_host_check": { 2024-09-16T15:47:32.855 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "warn_on_failed_host_check", 2024-09-16T15:47:32.855 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-09-16T15:47:32.855 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:32.855 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-09-16T15:47:32.855 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "True", 2024-09-16T15:47:32.855 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:32.855 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:32.855 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:32.855 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "raise a health warning if the host check fails", 2024-09-16T15:47:32.855 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:32.855 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:32.856 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:32.856 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:32.856 INFO:teuthology.orchestra.run.smithi123.stdout: "warn_on_stray_daemons": { 2024-09-16T15:47:32.856 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "warn_on_stray_daemons", 2024-09-16T15:47:32.856 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-09-16T15:47:32.856 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:32.856 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-09-16T15:47:32.856 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "True", 2024-09-16T15:47:32.856 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:32.856 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:32.856 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:32.856 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "raise a health warning if daemons are detected that are not managed by cephadm", 2024-09-16T15:47:32.856 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:32.856 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:32.856 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:32.857 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:32.857 INFO:teuthology.orchestra.run.smithi123.stdout: "warn_on_stray_hosts": { 2024-09-16T15:47:32.857 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "warn_on_stray_hosts", 2024-09-16T15:47:32.857 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-09-16T15:47:32.857 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:32.857 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-09-16T15:47:32.857 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "True", 2024-09-16T15:47:32.857 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:32.857 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:32.857 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:32.857 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "raise a health warning if daemons are detected on a host that is not managed by cephadm", 2024-09-16T15:47:32.857 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:32.857 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:32.857 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:32.857 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-16T15:47:32.858 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-16T15:47:32.858 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:32.858 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-09-16T15:47:32.858 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "crash", 2024-09-16T15:47:32.858 INFO:teuthology.orchestra.run.smithi123.stdout: "can_run": true, 2024-09-16T15:47:32.858 INFO:teuthology.orchestra.run.smithi123.stdout: "error_string": "", 2024-09-16T15:47:32.858 INFO:teuthology.orchestra.run.smithi123.stdout: "module_options": { 2024-09-16T15:47:32.858 INFO:teuthology.orchestra.run.smithi123.stdout: "log_level": { 2024-09-16T15:47:32.858 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_level", 2024-09-16T15:47:32.858 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-09-16T15:47:32.858 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:32.858 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-09-16T15:47:32.858 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "", 2024-09-16T15:47:32.858 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:32.858 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:32.859 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [ 2024-09-16T15:47:32.859 INFO:teuthology.orchestra.run.smithi123.stdout: "", 2024-09-16T15:47:32.859 INFO:teuthology.orchestra.run.smithi123.stdout: "critical", 2024-09-16T15:47:32.859 INFO:teuthology.orchestra.run.smithi123.stdout: "debug", 2024-09-16T15:47:32.859 INFO:teuthology.orchestra.run.smithi123.stdout: "error", 2024-09-16T15:47:32.859 INFO:teuthology.orchestra.run.smithi123.stdout: "info", 2024-09-16T15:47:32.859 INFO:teuthology.orchestra.run.smithi123.stdout: "warning" 2024-09-16T15:47:32.859 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-09-16T15:47:32.859 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:32.859 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:32.859 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:32.859 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:32.859 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:32.860 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_cluster": { 2024-09-16T15:47:32.860 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_cluster", 2024-09-16T15:47:32.860 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-09-16T15:47:32.860 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:32.860 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-09-16T15:47:32.860 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-09-16T15:47:32.860 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:32.860 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:32.860 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:32.860 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:32.860 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:32.860 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:32.860 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:32.860 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:32.861 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_cluster_level": { 2024-09-16T15:47:32.861 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_cluster_level", 2024-09-16T15:47:32.861 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-09-16T15:47:32.861 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:32.861 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-09-16T15:47:32.861 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "info", 2024-09-16T15:47:32.861 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:32.861 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:32.861 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [ 2024-09-16T15:47:32.861 INFO:teuthology.orchestra.run.smithi123.stdout: "", 2024-09-16T15:47:32.861 INFO:teuthology.orchestra.run.smithi123.stdout: "critical", 2024-09-16T15:47:32.861 INFO:teuthology.orchestra.run.smithi123.stdout: "debug", 2024-09-16T15:47:32.861 INFO:teuthology.orchestra.run.smithi123.stdout: "error", 2024-09-16T15:47:32.861 INFO:teuthology.orchestra.run.smithi123.stdout: "info", 2024-09-16T15:47:32.862 INFO:teuthology.orchestra.run.smithi123.stdout: "warning" 2024-09-16T15:47:32.862 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-09-16T15:47:32.862 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:32.862 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:32.862 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:32.862 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:32.862 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:32.862 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_file": { 2024-09-16T15:47:32.862 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_file", 2024-09-16T15:47:32.862 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-09-16T15:47:32.862 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:32.862 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-09-16T15:47:32.862 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-09-16T15:47:32.862 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:32.862 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:32.863 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:32.863 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:32.863 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:32.863 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:32.863 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:32.863 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:32.863 INFO:teuthology.orchestra.run.smithi123.stdout: "retain_interval": { 2024-09-16T15:47:32.863 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "retain_interval", 2024-09-16T15:47:32.863 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "secs", 2024-09-16T15:47:32.863 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:32.863 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-09-16T15:47:32.863 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "31536000", 2024-09-16T15:47:32.863 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:32.863 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:32.863 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:32.864 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "how long to retain crashes before pruning them", 2024-09-16T15:47:32.864 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:32.864 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:32.864 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:32.864 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:32.864 INFO:teuthology.orchestra.run.smithi123.stdout: "sqlite3_killpoint": { 2024-09-16T15:47:32.864 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "sqlite3_killpoint", 2024-09-16T15:47:32.864 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "int", 2024-09-16T15:47:32.864 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "dev", 2024-09-16T15:47:32.864 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-09-16T15:47:32.864 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "0", 2024-09-16T15:47:32.864 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:32.864 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:32.864 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:32.864 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:32.865 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:32.865 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:32.865 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:32.865 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:32.865 INFO:teuthology.orchestra.run.smithi123.stdout: "warn_recent_interval": { 2024-09-16T15:47:32.865 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "warn_recent_interval", 2024-09-16T15:47:32.865 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "secs", 2024-09-16T15:47:32.865 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:32.865 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-09-16T15:47:32.865 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "1209600", 2024-09-16T15:47:32.865 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:32.865 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:32.865 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:32.865 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "time interval in which to warn about recent crashes", 2024-09-16T15:47:32.865 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:32.866 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:32.866 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:32.866 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-16T15:47:32.866 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-16T15:47:32.866 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:32.866 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-09-16T15:47:32.866 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "dashboard", 2024-09-16T15:47:32.866 INFO:teuthology.orchestra.run.smithi123.stdout: "can_run": true, 2024-09-16T15:47:32.866 INFO:teuthology.orchestra.run.smithi123.stdout: "error_string": "", 2024-09-16T15:47:32.866 INFO:teuthology.orchestra.run.smithi123.stdout: "module_options": { 2024-09-16T15:47:32.866 INFO:teuthology.orchestra.run.smithi123.stdout: "ACCOUNT_LOCKOUT_ATTEMPTS": { 2024-09-16T15:47:32.866 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "ACCOUNT_LOCKOUT_ATTEMPTS", 2024-09-16T15:47:32.866 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "int", 2024-09-16T15:47:32.866 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:32.866 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-09-16T15:47:32.866 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "10", 2024-09-16T15:47:32.867 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:32.867 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:32.867 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:32.867 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:32.867 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:32.867 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:32.867 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:32.867 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:32.867 INFO:teuthology.orchestra.run.smithi123.stdout: "ALERTMANAGER_API_HOST": { 2024-09-16T15:47:32.867 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "ALERTMANAGER_API_HOST", 2024-09-16T15:47:32.867 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-09-16T15:47:32.867 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:32.867 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-09-16T15:47:32.867 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "", 2024-09-16T15:47:32.867 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:32.868 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:32.868 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:32.868 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:32.868 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:32.868 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:32.868 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:32.868 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:32.868 INFO:teuthology.orchestra.run.smithi123.stdout: "ALERTMANAGER_API_SSL_VERIFY": { 2024-09-16T15:47:32.868 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "ALERTMANAGER_API_SSL_VERIFY", 2024-09-16T15:47:32.868 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-09-16T15:47:32.868 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:32.868 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-09-16T15:47:32.868 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "True", 2024-09-16T15:47:32.868 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:32.868 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:32.868 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:32.869 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:32.869 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:32.869 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:32.869 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:32.869 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:32.869 INFO:teuthology.orchestra.run.smithi123.stdout: "AUDIT_API_ENABLED": { 2024-09-16T15:47:32.869 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "AUDIT_API_ENABLED", 2024-09-16T15:47:32.869 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-09-16T15:47:32.869 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:32.869 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-09-16T15:47:32.869 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-09-16T15:47:32.869 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:32.869 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:32.870 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:32.870 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:32.870 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:32.870 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:32.870 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:32.870 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:32.870 INFO:teuthology.orchestra.run.smithi123.stdout: "AUDIT_API_LOG_PAYLOAD": { 2024-09-16T15:47:32.870 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "AUDIT_API_LOG_PAYLOAD", 2024-09-16T15:47:32.870 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-09-16T15:47:32.870 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:32.870 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-09-16T15:47:32.870 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "True", 2024-09-16T15:47:32.871 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:32.871 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:32.871 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:32.871 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:32.871 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:32.871 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:32.871 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:32.871 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:32.871 INFO:teuthology.orchestra.run.smithi123.stdout: "ENABLE_BROWSABLE_API": { 2024-09-16T15:47:32.871 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "ENABLE_BROWSABLE_API", 2024-09-16T15:47:32.871 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-09-16T15:47:32.871 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:32.871 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-09-16T15:47:32.871 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "True", 2024-09-16T15:47:32.871 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:32.872 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:32.872 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:32.872 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:32.872 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:32.872 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:32.872 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:32.872 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:32.872 INFO:teuthology.orchestra.run.smithi123.stdout: "FEATURE_TOGGLE_CEPHFS": { 2024-09-16T15:47:32.872 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "FEATURE_TOGGLE_CEPHFS", 2024-09-16T15:47:32.872 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-09-16T15:47:32.872 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:32.872 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-09-16T15:47:32.873 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "True", 2024-09-16T15:47:32.873 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:32.873 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:32.873 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:32.873 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:32.873 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:32.873 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:32.873 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:32.873 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:32.873 INFO:teuthology.orchestra.run.smithi123.stdout: "FEATURE_TOGGLE_DASHBOARD": { 2024-09-16T15:47:32.873 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "FEATURE_TOGGLE_DASHBOARD", 2024-09-16T15:47:32.873 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-09-16T15:47:32.873 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:32.873 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-09-16T15:47:32.874 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "True", 2024-09-16T15:47:32.874 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:32.874 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:32.874 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:32.874 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:32.874 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:32.874 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:32.874 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:32.874 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:32.874 INFO:teuthology.orchestra.run.smithi123.stdout: "FEATURE_TOGGLE_ISCSI": { 2024-09-16T15:47:32.874 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "FEATURE_TOGGLE_ISCSI", 2024-09-16T15:47:32.874 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-09-16T15:47:32.874 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:32.875 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-09-16T15:47:32.875 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "True", 2024-09-16T15:47:32.875 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:32.875 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:32.875 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:32.875 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:32.875 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:32.875 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:32.875 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:32.875 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:32.875 INFO:teuthology.orchestra.run.smithi123.stdout: "FEATURE_TOGGLE_MIRRORING": { 2024-09-16T15:47:32.875 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "FEATURE_TOGGLE_MIRRORING", 2024-09-16T15:47:32.875 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-09-16T15:47:32.875 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:32.876 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-09-16T15:47:32.876 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "True", 2024-09-16T15:47:32.876 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:32.876 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:32.876 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:32.876 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:32.876 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:32.876 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:32.876 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:32.876 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:32.876 INFO:teuthology.orchestra.run.smithi123.stdout: "FEATURE_TOGGLE_NFS": { 2024-09-16T15:47:32.876 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "FEATURE_TOGGLE_NFS", 2024-09-16T15:47:32.876 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-09-16T15:47:32.877 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:32.877 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-09-16T15:47:32.877 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "True", 2024-09-16T15:47:32.877 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:32.877 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:32.877 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:32.877 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:32.877 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:32.877 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:32.877 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:32.877 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:32.877 INFO:teuthology.orchestra.run.smithi123.stdout: "FEATURE_TOGGLE_RBD": { 2024-09-16T15:47:32.877 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "FEATURE_TOGGLE_RBD", 2024-09-16T15:47:32.878 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-09-16T15:47:32.878 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:32.878 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-09-16T15:47:32.878 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "True", 2024-09-16T15:47:32.878 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:32.878 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:32.878 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:32.878 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:32.878 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:32.878 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:32.878 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:32.878 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:32.878 INFO:teuthology.orchestra.run.smithi123.stdout: "FEATURE_TOGGLE_RGW": { 2024-09-16T15:47:32.878 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "FEATURE_TOGGLE_RGW", 2024-09-16T15:47:32.878 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-09-16T15:47:32.879 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:32.879 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-09-16T15:47:32.879 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "True", 2024-09-16T15:47:32.879 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:32.879 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:32.879 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:32.879 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:32.879 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:32.879 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:32.879 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:32.879 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:32.879 INFO:teuthology.orchestra.run.smithi123.stdout: "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE": { 2024-09-16T15:47:32.879 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE", 2024-09-16T15:47:32.879 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-09-16T15:47:32.880 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:32.880 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-09-16T15:47:32.880 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "", 2024-09-16T15:47:32.880 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:32.880 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:32.880 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:32.880 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:32.880 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:32.880 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:32.880 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:32.880 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:32.880 INFO:teuthology.orchestra.run.smithi123.stdout: "GRAFANA_API_PASSWORD": { 2024-09-16T15:47:32.880 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "GRAFANA_API_PASSWORD", 2024-09-16T15:47:32.880 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-09-16T15:47:32.880 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:32.881 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-09-16T15:47:32.881 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "admin", 2024-09-16T15:47:32.881 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:32.881 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:32.881 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:32.881 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:32.881 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:32.881 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:32.881 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:32.881 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:32.881 INFO:teuthology.orchestra.run.smithi123.stdout: "GRAFANA_API_SSL_VERIFY": { 2024-09-16T15:47:32.881 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "GRAFANA_API_SSL_VERIFY", 2024-09-16T15:47:32.881 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-09-16T15:47:32.881 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:32.881 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-09-16T15:47:32.882 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "True", 2024-09-16T15:47:32.882 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:32.882 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:32.882 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:32.882 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:32.882 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:32.882 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:32.882 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:32.882 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:32.882 INFO:teuthology.orchestra.run.smithi123.stdout: "GRAFANA_API_URL": { 2024-09-16T15:47:32.882 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "GRAFANA_API_URL", 2024-09-16T15:47:32.882 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-09-16T15:47:32.882 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:32.882 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-09-16T15:47:32.882 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "", 2024-09-16T15:47:32.882 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:32.883 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:32.883 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:32.883 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:32.883 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:32.883 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:32.883 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:32.883 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:32.883 INFO:teuthology.orchestra.run.smithi123.stdout: "GRAFANA_API_USERNAME": { 2024-09-16T15:47:32.883 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "GRAFANA_API_USERNAME", 2024-09-16T15:47:32.883 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-09-16T15:47:32.883 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:32.883 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-09-16T15:47:32.883 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "admin", 2024-09-16T15:47:32.883 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:32.883 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:32.884 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:32.884 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:32.884 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:32.884 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:32.884 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:32.884 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:32.884 INFO:teuthology.orchestra.run.smithi123.stdout: "GRAFANA_FRONTEND_API_URL": { 2024-09-16T15:47:32.884 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "GRAFANA_FRONTEND_API_URL", 2024-09-16T15:47:32.884 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-09-16T15:47:32.884 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:32.884 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-09-16T15:47:32.884 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "", 2024-09-16T15:47:32.884 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:32.884 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:32.884 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:32.884 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:32.885 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:32.885 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:32.885 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:32.885 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:32.885 INFO:teuthology.orchestra.run.smithi123.stdout: "GRAFANA_UPDATE_DASHBOARDS": { 2024-09-16T15:47:32.885 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "GRAFANA_UPDATE_DASHBOARDS", 2024-09-16T15:47:32.885 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-09-16T15:47:32.885 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:32.885 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-09-16T15:47:32.885 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-09-16T15:47:32.885 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:32.885 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:32.885 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:32.885 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:32.885 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:32.886 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:32.886 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:32.886 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:32.886 INFO:teuthology.orchestra.run.smithi123.stdout: "ISCSI_API_SSL_VERIFICATION": { 2024-09-16T15:47:32.886 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "ISCSI_API_SSL_VERIFICATION", 2024-09-16T15:47:32.886 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-09-16T15:47:32.886 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:32.886 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-09-16T15:47:32.886 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "True", 2024-09-16T15:47:32.886 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:32.886 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:32.886 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:32.886 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:32.886 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:32.886 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:32.887 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:32.887 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:32.887 INFO:teuthology.orchestra.run.smithi123.stdout: "ISSUE_TRACKER_API_KEY": { 2024-09-16T15:47:32.887 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "ISSUE_TRACKER_API_KEY", 2024-09-16T15:47:32.887 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-09-16T15:47:32.887 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:32.887 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-09-16T15:47:32.887 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "", 2024-09-16T15:47:32.887 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:32.887 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:32.887 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:32.887 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:32.887 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:32.887 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:32.887 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:32.887 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:32.888 INFO:teuthology.orchestra.run.smithi123.stdout: "MANAGED_BY_CLUSTERS": { 2024-09-16T15:47:32.888 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "MANAGED_BY_CLUSTERS", 2024-09-16T15:47:32.888 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-09-16T15:47:32.888 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:32.888 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-09-16T15:47:32.888 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "[]", 2024-09-16T15:47:32.888 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:32.888 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:32.888 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:32.888 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:32.888 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:32.888 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:32.888 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:32.888 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:32.889 INFO:teuthology.orchestra.run.smithi123.stdout: "MULTICLUSTER_CONFIG": { 2024-09-16T15:47:32.889 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "MULTICLUSTER_CONFIG", 2024-09-16T15:47:32.889 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-09-16T15:47:32.889 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:32.889 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-09-16T15:47:32.889 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "{}", 2024-09-16T15:47:32.889 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:32.889 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:32.889 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:32.889 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:32.889 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:32.889 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:32.889 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:32.889 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:32.889 INFO:teuthology.orchestra.run.smithi123.stdout: "PROMETHEUS_API_HOST": { 2024-09-16T15:47:32.890 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "PROMETHEUS_API_HOST", 2024-09-16T15:47:32.890 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-09-16T15:47:32.890 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:32.890 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-09-16T15:47:32.890 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "", 2024-09-16T15:47:32.890 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:32.890 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:32.890 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:32.890 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:32.890 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:32.890 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:32.890 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:32.890 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:32.890 INFO:teuthology.orchestra.run.smithi123.stdout: "PROMETHEUS_API_SSL_VERIFY": { 2024-09-16T15:47:32.890 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "PROMETHEUS_API_SSL_VERIFY", 2024-09-16T15:47:32.890 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-09-16T15:47:32.891 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:32.891 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-09-16T15:47:32.891 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "True", 2024-09-16T15:47:32.891 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:32.891 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:32.891 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:32.891 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:32.891 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:32.891 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:32.891 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:32.891 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:32.891 INFO:teuthology.orchestra.run.smithi123.stdout: "PWD_POLICY_CHECK_COMPLEXITY_ENABLED": { 2024-09-16T15:47:32.891 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "PWD_POLICY_CHECK_COMPLEXITY_ENABLED", 2024-09-16T15:47:32.891 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-09-16T15:47:32.891 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:32.891 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-09-16T15:47:32.892 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-09-16T15:47:32.892 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:32.892 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:32.892 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:32.892 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:32.892 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:32.892 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:32.892 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:32.892 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:32.892 INFO:teuthology.orchestra.run.smithi123.stdout: "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED": { 2024-09-16T15:47:32.892 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED", 2024-09-16T15:47:32.892 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-09-16T15:47:32.892 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:32.892 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-09-16T15:47:32.892 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-09-16T15:47:32.893 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:32.893 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:32.893 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:32.893 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:32.893 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:32.893 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:32.893 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:32.893 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:32.893 INFO:teuthology.orchestra.run.smithi123.stdout: "PWD_POLICY_CHECK_LENGTH_ENABLED": { 2024-09-16T15:47:32.893 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "PWD_POLICY_CHECK_LENGTH_ENABLED", 2024-09-16T15:47:32.893 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-09-16T15:47:32.893 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:32.893 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-09-16T15:47:32.893 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "True", 2024-09-16T15:47:32.894 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:32.894 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:32.894 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:32.894 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:32.894 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:32.894 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:32.894 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:32.894 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:32.894 INFO:teuthology.orchestra.run.smithi123.stdout: "PWD_POLICY_CHECK_OLDPWD_ENABLED": { 2024-09-16T15:47:32.894 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "PWD_POLICY_CHECK_OLDPWD_ENABLED", 2024-09-16T15:47:32.894 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-09-16T15:47:32.894 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:32.894 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-09-16T15:47:32.894 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "True", 2024-09-16T15:47:32.895 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:32.895 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:32.895 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:32.895 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:32.895 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:32.895 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:32.895 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:32.895 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:32.895 INFO:teuthology.orchestra.run.smithi123.stdout: "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED": { 2024-09-16T15:47:32.895 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED", 2024-09-16T15:47:32.895 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-09-16T15:47:32.895 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:32.895 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-09-16T15:47:32.895 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-09-16T15:47:32.896 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:32.896 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:32.896 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:32.896 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:32.896 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:32.896 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:32.896 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:32.896 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:32.896 INFO:teuthology.orchestra.run.smithi123.stdout: "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED": { 2024-09-16T15:47:32.896 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED", 2024-09-16T15:47:32.896 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-09-16T15:47:32.896 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:32.896 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-09-16T15:47:32.897 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-09-16T15:47:32.897 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:32.897 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:32.897 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:32.897 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:32.897 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:32.897 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:32.897 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:32.897 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:32.897 INFO:teuthology.orchestra.run.smithi123.stdout: "PWD_POLICY_CHECK_USERNAME_ENABLED": { 2024-09-16T15:47:32.897 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "PWD_POLICY_CHECK_USERNAME_ENABLED", 2024-09-16T15:47:32.897 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-09-16T15:47:32.897 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:32.898 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-09-16T15:47:32.898 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-09-16T15:47:32.898 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:32.898 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:32.898 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:32.898 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:32.898 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:32.898 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:32.898 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:32.898 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:32.898 INFO:teuthology.orchestra.run.smithi123.stdout: "PWD_POLICY_ENABLED": { 2024-09-16T15:47:32.898 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "PWD_POLICY_ENABLED", 2024-09-16T15:47:32.898 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-09-16T15:47:32.898 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:32.899 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-09-16T15:47:32.899 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "True", 2024-09-16T15:47:32.899 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:32.899 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:32.899 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:32.899 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:32.899 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:32.899 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:32.899 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:32.899 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:32.899 INFO:teuthology.orchestra.run.smithi123.stdout: "PWD_POLICY_EXCLUSION_LIST": { 2024-09-16T15:47:32.899 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "PWD_POLICY_EXCLUSION_LIST", 2024-09-16T15:47:32.899 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-09-16T15:47:32.899 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:32.900 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-09-16T15:47:32.900 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps", 2024-09-16T15:47:32.900 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:32.900 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:32.900 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:32.900 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:32.900 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:32.900 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:32.900 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:32.900 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:32.900 INFO:teuthology.orchestra.run.smithi123.stdout: "PWD_POLICY_MIN_COMPLEXITY": { 2024-09-16T15:47:32.900 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "PWD_POLICY_MIN_COMPLEXITY", 2024-09-16T15:47:32.900 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "int", 2024-09-16T15:47:32.900 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:32.901 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-09-16T15:47:32.901 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "10", 2024-09-16T15:47:32.901 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:32.901 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:32.901 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:32.901 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:32.901 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:32.901 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:32.901 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:32.901 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:32.901 INFO:teuthology.orchestra.run.smithi123.stdout: "PWD_POLICY_MIN_LENGTH": { 2024-09-16T15:47:32.901 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "PWD_POLICY_MIN_LENGTH", 2024-09-16T15:47:32.902 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "int", 2024-09-16T15:47:32.902 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:32.902 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-09-16T15:47:32.902 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "8", 2024-09-16T15:47:32.902 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:32.902 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:32.902 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:32.902 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:32.902 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:32.902 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:32.902 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:32.902 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:32.902 INFO:teuthology.orchestra.run.smithi123.stdout: "REST_REQUESTS_TIMEOUT": { 2024-09-16T15:47:32.903 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "REST_REQUESTS_TIMEOUT", 2024-09-16T15:47:32.903 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "int", 2024-09-16T15:47:32.903 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:32.903 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-09-16T15:47:32.903 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "45", 2024-09-16T15:47:32.903 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:32.903 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:32.903 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:32.903 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:32.903 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:32.903 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:32.903 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:32.903 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:32.904 INFO:teuthology.orchestra.run.smithi123.stdout: "RGW_API_ACCESS_KEY": { 2024-09-16T15:47:32.904 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "RGW_API_ACCESS_KEY", 2024-09-16T15:47:32.904 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-09-16T15:47:32.904 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:32.904 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-09-16T15:47:32.904 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "", 2024-09-16T15:47:32.904 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:32.904 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:32.904 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:32.904 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:32.904 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:32.904 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:32.904 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:32.904 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:32.904 INFO:teuthology.orchestra.run.smithi123.stdout: "RGW_API_ADMIN_RESOURCE": { 2024-09-16T15:47:32.905 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "RGW_API_ADMIN_RESOURCE", 2024-09-16T15:47:32.905 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-09-16T15:47:32.905 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:32.905 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-09-16T15:47:32.905 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "admin", 2024-09-16T15:47:32.905 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:32.905 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:32.905 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:32.905 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:32.905 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:32.905 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:32.905 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:32.905 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:32.905 INFO:teuthology.orchestra.run.smithi123.stdout: "RGW_API_SECRET_KEY": { 2024-09-16T15:47:32.905 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "RGW_API_SECRET_KEY", 2024-09-16T15:47:32.905 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-09-16T15:47:32.906 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:32.906 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-09-16T15:47:32.906 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "", 2024-09-16T15:47:32.906 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:32.906 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:32.906 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:32.906 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:32.906 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:32.906 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:32.906 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:32.906 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:32.906 INFO:teuthology.orchestra.run.smithi123.stdout: "RGW_API_SSL_VERIFY": { 2024-09-16T15:47:32.906 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "RGW_API_SSL_VERIFY", 2024-09-16T15:47:32.906 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-09-16T15:47:32.906 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:32.907 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-09-16T15:47:32.907 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "True", 2024-09-16T15:47:32.907 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:32.907 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:32.907 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:32.907 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:32.907 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:32.907 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:32.907 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:32.907 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:32.907 INFO:teuthology.orchestra.run.smithi123.stdout: "UNSAFE_TLS_v1_2": { 2024-09-16T15:47:32.907 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "UNSAFE_TLS_v1_2", 2024-09-16T15:47:32.907 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-09-16T15:47:32.908 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:32.908 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-09-16T15:47:32.908 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-09-16T15:47:32.908 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:32.908 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:32.908 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:32.908 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:32.908 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:32.908 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:32.908 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:32.908 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:32.908 INFO:teuthology.orchestra.run.smithi123.stdout: "USER_PWD_EXPIRATION_SPAN": { 2024-09-16T15:47:32.908 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "USER_PWD_EXPIRATION_SPAN", 2024-09-16T15:47:32.908 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "int", 2024-09-16T15:47:32.908 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:32.909 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-09-16T15:47:32.909 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "0", 2024-09-16T15:47:32.909 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:32.909 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:32.909 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:32.909 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:32.909 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:32.909 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:32.909 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:32.909 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:32.909 INFO:teuthology.orchestra.run.smithi123.stdout: "USER_PWD_EXPIRATION_WARNING_1": { 2024-09-16T15:47:32.909 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "USER_PWD_EXPIRATION_WARNING_1", 2024-09-16T15:47:32.909 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "int", 2024-09-16T15:47:32.909 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:32.909 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-09-16T15:47:32.910 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "10", 2024-09-16T15:47:32.910 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:32.910 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:32.910 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:32.910 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:32.910 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:32.910 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:32.910 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:32.910 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:32.910 INFO:teuthology.orchestra.run.smithi123.stdout: "USER_PWD_EXPIRATION_WARNING_2": { 2024-09-16T15:47:32.910 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "USER_PWD_EXPIRATION_WARNING_2", 2024-09-16T15:47:32.910 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "int", 2024-09-16T15:47:32.910 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:32.910 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-09-16T15:47:32.910 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "5", 2024-09-16T15:47:32.910 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:32.911 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:32.911 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:32.911 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:32.911 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:32.911 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:32.911 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:32.911 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:32.911 INFO:teuthology.orchestra.run.smithi123.stdout: "cross_origin_url": { 2024-09-16T15:47:32.911 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "cross_origin_url", 2024-09-16T15:47:32.911 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-09-16T15:47:32.911 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:32.911 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-09-16T15:47:32.911 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "", 2024-09-16T15:47:32.911 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:32.911 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:32.912 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:32.912 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:32.912 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:32.912 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:32.912 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:32.912 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:32.912 INFO:teuthology.orchestra.run.smithi123.stdout: "crt_file": { 2024-09-16T15:47:32.912 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "crt_file", 2024-09-16T15:47:32.912 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-09-16T15:47:32.912 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:32.912 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-09-16T15:47:32.912 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "", 2024-09-16T15:47:32.912 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:32.912 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:32.912 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:32.912 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:32.913 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:32.913 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:32.913 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:32.913 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:32.913 INFO:teuthology.orchestra.run.smithi123.stdout: "debug": { 2024-09-16T15:47:32.913 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "debug", 2024-09-16T15:47:32.913 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-09-16T15:47:32.913 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:32.913 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-09-16T15:47:32.913 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-09-16T15:47:32.913 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:32.913 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:32.913 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:32.913 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "Enable/disable debug options", 2024-09-16T15:47:32.913 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:32.914 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:32.914 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:32.914 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:32.914 INFO:teuthology.orchestra.run.smithi123.stdout: "jwt_token_ttl": { 2024-09-16T15:47:32.914 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "jwt_token_ttl", 2024-09-16T15:47:32.914 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "int", 2024-09-16T15:47:32.914 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:32.914 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-09-16T15:47:32.914 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "28800", 2024-09-16T15:47:32.914 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:32.914 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:32.914 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:32.914 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:32.914 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:32.914 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:32.915 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:32.915 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:32.915 INFO:teuthology.orchestra.run.smithi123.stdout: "key_file": { 2024-09-16T15:47:32.915 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "key_file", 2024-09-16T15:47:32.915 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-09-16T15:47:32.915 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:32.915 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-09-16T15:47:32.915 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "", 2024-09-16T15:47:32.915 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:32.915 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:32.915 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:32.915 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:32.915 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:32.915 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:32.915 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:32.915 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:32.916 INFO:teuthology.orchestra.run.smithi123.stdout: "log_level": { 2024-09-16T15:47:32.916 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_level", 2024-09-16T15:47:32.916 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-09-16T15:47:32.916 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:32.916 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-09-16T15:47:32.916 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "", 2024-09-16T15:47:32.916 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:32.916 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:32.916 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [ 2024-09-16T15:47:32.916 INFO:teuthology.orchestra.run.smithi123.stdout: "", 2024-09-16T15:47:32.916 INFO:teuthology.orchestra.run.smithi123.stdout: "critical", 2024-09-16T15:47:32.916 INFO:teuthology.orchestra.run.smithi123.stdout: "debug", 2024-09-16T15:47:32.916 INFO:teuthology.orchestra.run.smithi123.stdout: "error", 2024-09-16T15:47:32.916 INFO:teuthology.orchestra.run.smithi123.stdout: "info", 2024-09-16T15:47:32.917 INFO:teuthology.orchestra.run.smithi123.stdout: "warning" 2024-09-16T15:47:32.917 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-09-16T15:47:32.917 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:32.917 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:32.917 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:32.917 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:32.917 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:32.917 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_cluster": { 2024-09-16T15:47:32.917 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_cluster", 2024-09-16T15:47:32.917 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-09-16T15:47:32.917 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:32.917 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-09-16T15:47:32.917 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-09-16T15:47:32.917 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:32.917 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:32.918 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:32.918 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:32.918 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:32.918 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:32.918 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:32.918 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:32.918 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_cluster_level": { 2024-09-16T15:47:32.918 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_cluster_level", 2024-09-16T15:47:32.918 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-09-16T15:47:32.918 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:32.918 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-09-16T15:47:32.918 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "info", 2024-09-16T15:47:32.918 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:32.918 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:32.918 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [ 2024-09-16T15:47:32.919 INFO:teuthology.orchestra.run.smithi123.stdout: "", 2024-09-16T15:47:32.919 INFO:teuthology.orchestra.run.smithi123.stdout: "critical", 2024-09-16T15:47:32.919 INFO:teuthology.orchestra.run.smithi123.stdout: "debug", 2024-09-16T15:47:32.919 INFO:teuthology.orchestra.run.smithi123.stdout: "error", 2024-09-16T15:47:32.919 INFO:teuthology.orchestra.run.smithi123.stdout: "info", 2024-09-16T15:47:32.919 INFO:teuthology.orchestra.run.smithi123.stdout: "warning" 2024-09-16T15:47:32.919 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-09-16T15:47:32.919 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:32.919 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:32.919 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:32.919 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:32.919 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:32.919 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_file": { 2024-09-16T15:47:32.919 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_file", 2024-09-16T15:47:32.919 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-09-16T15:47:32.919 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:32.920 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-09-16T15:47:32.920 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-09-16T15:47:32.920 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:32.920 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:32.920 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:32.920 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:32.920 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:32.920 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:32.920 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:32.920 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:32.920 INFO:teuthology.orchestra.run.smithi123.stdout: "motd": { 2024-09-16T15:47:32.920 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "motd", 2024-09-16T15:47:32.920 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-09-16T15:47:32.920 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:32.920 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-09-16T15:47:32.921 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "", 2024-09-16T15:47:32.921 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:32.921 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:32.921 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:32.921 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "The message of the day", 2024-09-16T15:47:32.921 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:32.921 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:32.921 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:32.921 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:32.921 INFO:teuthology.orchestra.run.smithi123.stdout: "redirect_resolve_ip_addr": { 2024-09-16T15:47:32.921 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "redirect_resolve_ip_addr", 2024-09-16T15:47:32.921 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-09-16T15:47:32.921 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:32.921 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-09-16T15:47:32.921 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-09-16T15:47:32.922 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:32.922 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:32.922 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:32.922 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:32.922 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:32.922 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:32.922 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:32.922 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:32.922 INFO:teuthology.orchestra.run.smithi123.stdout: "server_addr": { 2024-09-16T15:47:32.922 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "server_addr", 2024-09-16T15:47:32.922 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-09-16T15:47:32.922 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:32.922 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-09-16T15:47:32.922 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "::", 2024-09-16T15:47:32.922 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:32.922 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:32.923 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:32.923 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:32.923 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:32.923 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:32.923 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:32.923 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:32.923 INFO:teuthology.orchestra.run.smithi123.stdout: "server_port": { 2024-09-16T15:47:32.923 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "server_port", 2024-09-16T15:47:32.923 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "int", 2024-09-16T15:47:32.923 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:32.923 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-09-16T15:47:32.923 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "8080", 2024-09-16T15:47:32.923 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:32.923 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:32.923 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:32.924 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:32.924 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:32.924 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:32.924 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:32.924 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:32.924 INFO:teuthology.orchestra.run.smithi123.stdout: "sqlite3_killpoint": { 2024-09-16T15:47:32.924 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "sqlite3_killpoint", 2024-09-16T15:47:32.924 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "int", 2024-09-16T15:47:32.924 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "dev", 2024-09-16T15:47:32.924 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-09-16T15:47:32.924 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "0", 2024-09-16T15:47:32.924 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:32.924 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:32.924 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:32.924 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:32.924 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:32.925 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:32.925 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:32.925 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:32.925 INFO:teuthology.orchestra.run.smithi123.stdout: "ssl": { 2024-09-16T15:47:32.925 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "ssl", 2024-09-16T15:47:32.925 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-09-16T15:47:32.925 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:32.925 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-09-16T15:47:32.925 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "True", 2024-09-16T15:47:32.925 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:32.925 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:32.925 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:32.925 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:32.925 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:32.925 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:32.926 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:32.926 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:32.926 INFO:teuthology.orchestra.run.smithi123.stdout: "ssl_server_port": { 2024-09-16T15:47:32.926 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "ssl_server_port", 2024-09-16T15:47:32.926 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "int", 2024-09-16T15:47:32.926 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:32.926 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-09-16T15:47:32.926 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "8443", 2024-09-16T15:47:32.926 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:32.926 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:32.926 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:32.926 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:32.926 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:32.926 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:32.926 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:32.927 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:32.927 INFO:teuthology.orchestra.run.smithi123.stdout: "standby_behaviour": { 2024-09-16T15:47:32.927 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "standby_behaviour", 2024-09-16T15:47:32.927 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-09-16T15:47:32.927 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:32.927 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-09-16T15:47:32.927 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "redirect", 2024-09-16T15:47:32.927 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:32.927 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:32.927 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [ 2024-09-16T15:47:32.927 INFO:teuthology.orchestra.run.smithi123.stdout: "error", 2024-09-16T15:47:32.927 INFO:teuthology.orchestra.run.smithi123.stdout: "redirect" 2024-09-16T15:47:32.927 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-09-16T15:47:32.927 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:32.927 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:32.928 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:32.928 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:32.928 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:32.928 INFO:teuthology.orchestra.run.smithi123.stdout: "standby_error_status_code": { 2024-09-16T15:47:32.928 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "standby_error_status_code", 2024-09-16T15:47:32.928 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "int", 2024-09-16T15:47:32.928 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:32.928 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-09-16T15:47:32.928 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "500", 2024-09-16T15:47:32.928 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "400", 2024-09-16T15:47:32.928 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "599", 2024-09-16T15:47:32.928 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:32.928 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:32.928 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:32.928 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:32.928 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:32.929 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:32.929 INFO:teuthology.orchestra.run.smithi123.stdout: "url_prefix": { 2024-09-16T15:47:32.929 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "url_prefix", 2024-09-16T15:47:32.929 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-09-16T15:47:32.929 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:32.929 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-09-16T15:47:32.929 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "", 2024-09-16T15:47:32.929 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:32.929 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:32.929 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:32.929 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:32.929 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:32.929 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:32.929 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:32.929 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-16T15:47:32.930 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-16T15:47:32.930 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:32.930 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-09-16T15:47:32.930 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "devicehealth", 2024-09-16T15:47:32.930 INFO:teuthology.orchestra.run.smithi123.stdout: "can_run": true, 2024-09-16T15:47:32.930 INFO:teuthology.orchestra.run.smithi123.stdout: "error_string": "", 2024-09-16T15:47:32.930 INFO:teuthology.orchestra.run.smithi123.stdout: "module_options": { 2024-09-16T15:47:32.930 INFO:teuthology.orchestra.run.smithi123.stdout: "enable_monitoring": { 2024-09-16T15:47:32.930 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "enable_monitoring", 2024-09-16T15:47:32.930 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-09-16T15:47:32.930 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:32.930 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-09-16T15:47:32.930 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "True", 2024-09-16T15:47:32.930 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:32.930 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:32.930 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:32.931 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "monitor device health metrics", 2024-09-16T15:47:32.931 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:32.931 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:32.931 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:32.931 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:32.931 INFO:teuthology.orchestra.run.smithi123.stdout: "log_level": { 2024-09-16T15:47:32.931 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_level", 2024-09-16T15:47:32.931 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-09-16T15:47:32.931 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:32.931 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-09-16T15:47:32.931 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "", 2024-09-16T15:47:32.931 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:32.931 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:32.931 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [ 2024-09-16T15:47:32.931 INFO:teuthology.orchestra.run.smithi123.stdout: "", 2024-09-16T15:47:32.932 INFO:teuthology.orchestra.run.smithi123.stdout: "critical", 2024-09-16T15:47:32.932 INFO:teuthology.orchestra.run.smithi123.stdout: "debug", 2024-09-16T15:47:32.932 INFO:teuthology.orchestra.run.smithi123.stdout: "error", 2024-09-16T15:47:32.932 INFO:teuthology.orchestra.run.smithi123.stdout: "info", 2024-09-16T15:47:32.932 INFO:teuthology.orchestra.run.smithi123.stdout: "warning" 2024-09-16T15:47:32.932 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-09-16T15:47:32.932 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:32.932 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:32.932 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:32.932 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:32.932 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:32.932 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_cluster": { 2024-09-16T15:47:32.932 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_cluster", 2024-09-16T15:47:32.932 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-09-16T15:47:32.932 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:32.933 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-09-16T15:47:32.933 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-09-16T15:47:32.933 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:32.933 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:32.933 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:32.933 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:32.933 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:32.933 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:32.933 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:32.933 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:32.933 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_cluster_level": { 2024-09-16T15:47:32.933 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_cluster_level", 2024-09-16T15:47:32.933 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-09-16T15:47:32.933 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:32.933 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-09-16T15:47:32.933 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "info", 2024-09-16T15:47:32.934 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:32.934 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:32.934 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [ 2024-09-16T15:47:32.934 INFO:teuthology.orchestra.run.smithi123.stdout: "", 2024-09-16T15:47:32.934 INFO:teuthology.orchestra.run.smithi123.stdout: "critical", 2024-09-16T15:47:32.934 INFO:teuthology.orchestra.run.smithi123.stdout: "debug", 2024-09-16T15:47:32.934 INFO:teuthology.orchestra.run.smithi123.stdout: "error", 2024-09-16T15:47:32.934 INFO:teuthology.orchestra.run.smithi123.stdout: "info", 2024-09-16T15:47:32.934 INFO:teuthology.orchestra.run.smithi123.stdout: "warning" 2024-09-16T15:47:32.934 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-09-16T15:47:32.934 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:32.934 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:32.934 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:32.934 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:32.935 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:32.935 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_file": { 2024-09-16T15:47:32.935 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_file", 2024-09-16T15:47:32.935 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-09-16T15:47:32.935 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:32.935 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-09-16T15:47:32.935 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-09-16T15:47:32.935 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:32.935 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:32.935 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:32.935 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:32.935 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:32.935 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:32.935 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:32.935 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:32.936 INFO:teuthology.orchestra.run.smithi123.stdout: "mark_out_threshold": { 2024-09-16T15:47:32.936 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "mark_out_threshold", 2024-09-16T15:47:32.936 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "secs", 2024-09-16T15:47:32.936 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:32.936 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-09-16T15:47:32.936 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "2419200", 2024-09-16T15:47:32.936 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:32.936 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:32.936 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:32.936 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "automatically mark OSD if it may fail before this long", 2024-09-16T15:47:32.936 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:32.936 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:32.936 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:32.936 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:32.936 INFO:teuthology.orchestra.run.smithi123.stdout: "pool_name": { 2024-09-16T15:47:32.936 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "pool_name", 2024-09-16T15:47:32.937 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-09-16T15:47:32.937 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:32.937 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-09-16T15:47:32.937 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "device_health_metrics", 2024-09-16T15:47:32.937 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:32.937 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:32.937 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:32.937 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "name of pool in which to store device health metrics", 2024-09-16T15:47:32.937 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:32.937 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:32.937 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:32.937 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:32.937 INFO:teuthology.orchestra.run.smithi123.stdout: "retention_period": { 2024-09-16T15:47:32.937 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "retention_period", 2024-09-16T15:47:32.937 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "secs", 2024-09-16T15:47:32.938 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:32.938 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-09-16T15:47:32.938 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "15552000", 2024-09-16T15:47:32.938 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:32.938 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:32.938 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:32.938 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "how long to retain device health metrics", 2024-09-16T15:47:32.938 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:32.938 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:32.938 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:32.938 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:32.938 INFO:teuthology.orchestra.run.smithi123.stdout: "scrape_frequency": { 2024-09-16T15:47:32.938 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "scrape_frequency", 2024-09-16T15:47:32.938 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "secs", 2024-09-16T15:47:32.938 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:32.939 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-09-16T15:47:32.939 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "86400", 2024-09-16T15:47:32.939 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:32.939 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:32.939 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:32.939 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "how frequently to scrape device health metrics", 2024-09-16T15:47:32.939 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:32.939 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:32.939 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:32.939 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:32.939 INFO:teuthology.orchestra.run.smithi123.stdout: "self_heal": { 2024-09-16T15:47:32.939 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "self_heal", 2024-09-16T15:47:32.939 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-09-16T15:47:32.939 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:32.939 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-09-16T15:47:32.940 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "True", 2024-09-16T15:47:32.940 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:32.940 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:32.940 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:32.940 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "preemptively heal cluster around devices that may fail", 2024-09-16T15:47:32.940 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:32.940 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:32.940 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:32.940 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:32.940 INFO:teuthology.orchestra.run.smithi123.stdout: "sleep_interval": { 2024-09-16T15:47:32.940 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "sleep_interval", 2024-09-16T15:47:32.940 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "secs", 2024-09-16T15:47:32.940 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:32.940 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-09-16T15:47:32.940 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "600", 2024-09-16T15:47:32.941 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:32.941 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:32.941 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:32.941 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "how frequently to wake up and check device health", 2024-09-16T15:47:32.941 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:32.941 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:32.941 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:32.941 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:32.941 INFO:teuthology.orchestra.run.smithi123.stdout: "sqlite3_killpoint": { 2024-09-16T15:47:32.941 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "sqlite3_killpoint", 2024-09-16T15:47:32.941 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "int", 2024-09-16T15:47:32.941 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "dev", 2024-09-16T15:47:32.941 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-09-16T15:47:32.941 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "0", 2024-09-16T15:47:32.941 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:32.941 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:32.942 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:32.942 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:32.942 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:32.942 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:32.942 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:32.942 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:32.942 INFO:teuthology.orchestra.run.smithi123.stdout: "warn_threshold": { 2024-09-16T15:47:32.942 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "warn_threshold", 2024-09-16T15:47:32.942 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "secs", 2024-09-16T15:47:32.942 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:32.942 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-09-16T15:47:32.942 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "7257600", 2024-09-16T15:47:32.942 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:32.942 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:32.942 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:32.942 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "raise health warning if OSD may fail before this long", 2024-09-16T15:47:32.943 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:32.943 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:32.943 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:32.943 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-16T15:47:32.943 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-16T15:47:32.943 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:32.943 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-09-16T15:47:32.943 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "diskprediction_local", 2024-09-16T15:47:32.943 INFO:teuthology.orchestra.run.smithi123.stdout: "can_run": true, 2024-09-16T15:47:32.943 INFO:teuthology.orchestra.run.smithi123.stdout: "error_string": "", 2024-09-16T15:47:32.943 INFO:teuthology.orchestra.run.smithi123.stdout: "module_options": { 2024-09-16T15:47:32.943 INFO:teuthology.orchestra.run.smithi123.stdout: "log_level": { 2024-09-16T15:47:32.943 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_level", 2024-09-16T15:47:32.943 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-09-16T15:47:32.943 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:32.944 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-09-16T15:47:32.944 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "", 2024-09-16T15:47:32.944 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:32.944 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:32.944 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [ 2024-09-16T15:47:32.944 INFO:teuthology.orchestra.run.smithi123.stdout: "", 2024-09-16T15:47:32.944 INFO:teuthology.orchestra.run.smithi123.stdout: "critical", 2024-09-16T15:47:32.944 INFO:teuthology.orchestra.run.smithi123.stdout: "debug", 2024-09-16T15:47:32.944 INFO:teuthology.orchestra.run.smithi123.stdout: "error", 2024-09-16T15:47:32.944 INFO:teuthology.orchestra.run.smithi123.stdout: "info", 2024-09-16T15:47:32.944 INFO:teuthology.orchestra.run.smithi123.stdout: "warning" 2024-09-16T15:47:32.944 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-09-16T15:47:32.944 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:32.944 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:32.944 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:32.945 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:32.945 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:32.945 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_cluster": { 2024-09-16T15:47:32.945 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_cluster", 2024-09-16T15:47:32.945 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-09-16T15:47:32.945 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:32.945 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-09-16T15:47:32.945 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-09-16T15:47:32.945 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:32.945 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:32.945 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:32.945 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:32.945 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:32.945 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:32.945 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:32.945 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:32.946 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_cluster_level": { 2024-09-16T15:47:32.946 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_cluster_level", 2024-09-16T15:47:32.946 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-09-16T15:47:32.946 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:32.946 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-09-16T15:47:32.946 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "info", 2024-09-16T15:47:32.946 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:32.946 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:32.946 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [ 2024-09-16T15:47:32.946 INFO:teuthology.orchestra.run.smithi123.stdout: "", 2024-09-16T15:47:32.946 INFO:teuthology.orchestra.run.smithi123.stdout: "critical", 2024-09-16T15:47:32.946 INFO:teuthology.orchestra.run.smithi123.stdout: "debug", 2024-09-16T15:47:32.946 INFO:teuthology.orchestra.run.smithi123.stdout: "error", 2024-09-16T15:47:32.947 INFO:teuthology.orchestra.run.smithi123.stdout: "info", 2024-09-16T15:47:32.947 INFO:teuthology.orchestra.run.smithi123.stdout: "warning" 2024-09-16T15:47:32.947 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-09-16T15:47:32.947 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:32.947 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:32.947 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:32.947 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:32.947 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:32.947 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_file": { 2024-09-16T15:47:32.947 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_file", 2024-09-16T15:47:32.947 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-09-16T15:47:32.947 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:32.947 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-09-16T15:47:32.947 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-09-16T15:47:32.948 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:32.948 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:32.948 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:32.948 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:32.948 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:32.948 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:32.948 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:32.948 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:32.948 INFO:teuthology.orchestra.run.smithi123.stdout: "predict_interval": { 2024-09-16T15:47:32.948 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "predict_interval", 2024-09-16T15:47:32.948 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-09-16T15:47:32.948 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:32.949 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-09-16T15:47:32.949 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "86400", 2024-09-16T15:47:32.949 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:32.949 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:32.949 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:32.949 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:32.949 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:32.949 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:32.949 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:32.949 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:32.950 INFO:teuthology.orchestra.run.smithi123.stdout: "predictor_model": { 2024-09-16T15:47:32.950 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "predictor_model", 2024-09-16T15:47:32.950 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-09-16T15:47:32.950 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:32.950 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-09-16T15:47:32.950 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "prophetstor", 2024-09-16T15:47:32.950 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:32.950 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:32.950 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:32.950 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:32.950 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:32.951 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:32.951 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:32.951 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:32.951 INFO:teuthology.orchestra.run.smithi123.stdout: "sleep_interval": { 2024-09-16T15:47:32.951 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "sleep_interval", 2024-09-16T15:47:32.951 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-09-16T15:47:32.951 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:32.951 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-09-16T15:47:32.951 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "600", 2024-09-16T15:47:32.951 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:32.951 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:32.951 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:32.951 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:32.951 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:32.952 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:32.952 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:32.952 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:32.952 INFO:teuthology.orchestra.run.smithi123.stdout: "sqlite3_killpoint": { 2024-09-16T15:47:32.952 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "sqlite3_killpoint", 2024-09-16T15:47:32.952 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "int", 2024-09-16T15:47:32.952 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "dev", 2024-09-16T15:47:32.952 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-09-16T15:47:32.952 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "0", 2024-09-16T15:47:32.952 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:32.952 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:32.953 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:32.953 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:32.953 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:32.953 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:32.953 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:32.953 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-16T15:47:32.953 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-16T15:47:32.953 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:32.953 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-09-16T15:47:32.953 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "influx", 2024-09-16T15:47:32.953 INFO:teuthology.orchestra.run.smithi123.stdout: "can_run": false, 2024-09-16T15:47:32.953 INFO:teuthology.orchestra.run.smithi123.stdout: "error_string": "influxdb python module not found", 2024-09-16T15:47:32.953 INFO:teuthology.orchestra.run.smithi123.stdout: "module_options": { 2024-09-16T15:47:32.954 INFO:teuthology.orchestra.run.smithi123.stdout: "batch_size": { 2024-09-16T15:47:32.954 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "batch_size", 2024-09-16T15:47:32.954 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "int", 2024-09-16T15:47:32.954 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:32.954 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-09-16T15:47:32.954 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "5000", 2024-09-16T15:47:32.954 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:32.954 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:32.954 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:32.954 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "How big batches of data points should be when sending to InfluxDB.", 2024-09-16T15:47:32.954 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:32.954 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:32.955 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:32.955 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:32.955 INFO:teuthology.orchestra.run.smithi123.stdout: "database": { 2024-09-16T15:47:32.955 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "database", 2024-09-16T15:47:32.955 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-09-16T15:47:32.955 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:32.955 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-09-16T15:47:32.955 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "ceph", 2024-09-16T15:47:32.955 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:32.955 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:32.955 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:32.956 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.", 2024-09-16T15:47:32.956 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:32.956 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:32.956 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:32.956 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:32.956 INFO:teuthology.orchestra.run.smithi123.stdout: "hostname": { 2024-09-16T15:47:32.956 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "hostname", 2024-09-16T15:47:32.956 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-09-16T15:47:32.956 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:32.956 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-09-16T15:47:32.957 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "", 2024-09-16T15:47:32.957 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:32.957 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:32.957 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:32.957 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "InfluxDB server hostname", 2024-09-16T15:47:32.957 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:32.957 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:32.957 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:32.957 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:32.957 INFO:teuthology.orchestra.run.smithi123.stdout: "interval": { 2024-09-16T15:47:32.957 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "interval", 2024-09-16T15:47:32.958 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "secs", 2024-09-16T15:47:32.958 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:32.958 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-09-16T15:47:32.958 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "30", 2024-09-16T15:47:32.958 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "5", 2024-09-16T15:47:32.958 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:32.958 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:32.958 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "Time between reports to InfluxDB. Default 30 seconds.", 2024-09-16T15:47:32.958 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:32.958 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:32.958 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:32.958 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:32.958 INFO:teuthology.orchestra.run.smithi123.stdout: "log_level": { 2024-09-16T15:47:32.959 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_level", 2024-09-16T15:47:32.959 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-09-16T15:47:32.959 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:32.959 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-09-16T15:47:32.959 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "", 2024-09-16T15:47:32.959 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:32.959 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:32.959 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [ 2024-09-16T15:47:32.959 INFO:teuthology.orchestra.run.smithi123.stdout: "", 2024-09-16T15:47:32.959 INFO:teuthology.orchestra.run.smithi123.stdout: "critical", 2024-09-16T15:47:32.959 INFO:teuthology.orchestra.run.smithi123.stdout: "debug", 2024-09-16T15:47:32.959 INFO:teuthology.orchestra.run.smithi123.stdout: "error", 2024-09-16T15:47:32.960 INFO:teuthology.orchestra.run.smithi123.stdout: "info", 2024-09-16T15:47:32.960 INFO:teuthology.orchestra.run.smithi123.stdout: "warning" 2024-09-16T15:47:32.960 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-09-16T15:47:32.960 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:32.960 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:32.960 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:32.960 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:32.960 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:32.960 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_cluster": { 2024-09-16T15:47:32.960 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_cluster", 2024-09-16T15:47:32.960 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-09-16T15:47:32.961 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:32.961 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-09-16T15:47:32.961 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-09-16T15:47:32.961 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:32.961 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:32.961 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:32.961 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:32.961 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:32.961 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:32.961 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:32.961 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:32.962 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_cluster_level": { 2024-09-16T15:47:32.962 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_cluster_level", 2024-09-16T15:47:32.962 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-09-16T15:47:32.962 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:32.962 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-09-16T15:47:32.962 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "info", 2024-09-16T15:47:32.962 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:32.962 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:32.962 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [ 2024-09-16T15:47:32.962 INFO:teuthology.orchestra.run.smithi123.stdout: "", 2024-09-16T15:47:32.962 INFO:teuthology.orchestra.run.smithi123.stdout: "critical", 2024-09-16T15:47:32.962 INFO:teuthology.orchestra.run.smithi123.stdout: "debug", 2024-09-16T15:47:32.962 INFO:teuthology.orchestra.run.smithi123.stdout: "error", 2024-09-16T15:47:32.962 INFO:teuthology.orchestra.run.smithi123.stdout: "info", 2024-09-16T15:47:32.963 INFO:teuthology.orchestra.run.smithi123.stdout: "warning" 2024-09-16T15:47:32.963 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-09-16T15:47:32.963 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:32.963 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:32.963 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:32.963 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:32.963 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:32.963 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_file": { 2024-09-16T15:47:32.963 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_file", 2024-09-16T15:47:32.963 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-09-16T15:47:32.963 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:32.963 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-09-16T15:47:32.963 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-09-16T15:47:32.964 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:32.964 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:32.964 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:32.964 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:32.964 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:32.964 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:32.964 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:32.964 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:32.964 INFO:teuthology.orchestra.run.smithi123.stdout: "password": { 2024-09-16T15:47:32.964 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "password", 2024-09-16T15:47:32.964 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-09-16T15:47:32.964 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:32.964 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-09-16T15:47:32.964 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "", 2024-09-16T15:47:32.965 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:32.965 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:32.965 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:32.965 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "password of InfluxDB server user", 2024-09-16T15:47:32.965 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:32.965 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:32.965 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:32.965 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:32.965 INFO:teuthology.orchestra.run.smithi123.stdout: "port": { 2024-09-16T15:47:32.965 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "port", 2024-09-16T15:47:32.965 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "int", 2024-09-16T15:47:32.965 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:32.965 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-09-16T15:47:32.965 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "8086", 2024-09-16T15:47:32.966 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:32.966 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:32.966 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:32.966 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "InfluxDB server port", 2024-09-16T15:47:32.966 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:32.966 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:32.966 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:32.966 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:32.966 INFO:teuthology.orchestra.run.smithi123.stdout: "sqlite3_killpoint": { 2024-09-16T15:47:32.966 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "sqlite3_killpoint", 2024-09-16T15:47:32.966 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "int", 2024-09-16T15:47:32.966 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "dev", 2024-09-16T15:47:32.966 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-09-16T15:47:32.966 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "0", 2024-09-16T15:47:32.967 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:32.967 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:32.967 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:32.967 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:32.967 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:32.967 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:32.967 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:32.967 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:32.967 INFO:teuthology.orchestra.run.smithi123.stdout: "ssl": { 2024-09-16T15:47:32.967 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "ssl", 2024-09-16T15:47:32.967 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-09-16T15:47:32.967 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:32.967 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-09-16T15:47:32.968 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "false", 2024-09-16T15:47:32.968 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:32.968 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:32.968 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:32.968 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "Use https connection for InfluxDB server. Use \"true\" or \"false\".", 2024-09-16T15:47:32.968 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:32.968 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:32.968 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:32.968 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:32.968 INFO:teuthology.orchestra.run.smithi123.stdout: "threads": { 2024-09-16T15:47:32.968 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "threads", 2024-09-16T15:47:32.968 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "int", 2024-09-16T15:47:32.968 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:32.968 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-09-16T15:47:32.968 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "5", 2024-09-16T15:47:32.969 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "1", 2024-09-16T15:47:32.969 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "32", 2024-09-16T15:47:32.969 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:32.969 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "How many worker threads should be spawned for sending data to InfluxDB.", 2024-09-16T15:47:32.969 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:32.969 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:32.969 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:32.969 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:32.969 INFO:teuthology.orchestra.run.smithi123.stdout: "username": { 2024-09-16T15:47:32.969 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "username", 2024-09-16T15:47:32.969 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-09-16T15:47:32.969 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:32.969 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-09-16T15:47:32.969 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "", 2024-09-16T15:47:32.970 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:32.970 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:32.970 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:32.970 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "username of InfluxDB server user", 2024-09-16T15:47:32.970 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:32.970 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:32.970 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:32.970 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:32.970 INFO:teuthology.orchestra.run.smithi123.stdout: "verify_ssl": { 2024-09-16T15:47:32.970 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "verify_ssl", 2024-09-16T15:47:32.970 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-09-16T15:47:32.970 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:32.970 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-09-16T15:47:32.970 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "true", 2024-09-16T15:47:32.970 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:32.971 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:32.971 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:32.971 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "Verify https cert for InfluxDB server. Use \"true\" or \"false\".", 2024-09-16T15:47:32.971 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:32.971 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:32.971 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:32.971 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-16T15:47:32.971 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-16T15:47:32.971 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:32.971 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-09-16T15:47:32.971 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "insights", 2024-09-16T15:47:32.971 INFO:teuthology.orchestra.run.smithi123.stdout: "can_run": true, 2024-09-16T15:47:32.971 INFO:teuthology.orchestra.run.smithi123.stdout: "error_string": "", 2024-09-16T15:47:32.971 INFO:teuthology.orchestra.run.smithi123.stdout: "module_options": { 2024-09-16T15:47:32.971 INFO:teuthology.orchestra.run.smithi123.stdout: "log_level": { 2024-09-16T15:47:32.971 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_level", 2024-09-16T15:47:32.972 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-09-16T15:47:32.972 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:32.972 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-09-16T15:47:32.972 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "", 2024-09-16T15:47:32.972 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:32.972 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:32.972 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [ 2024-09-16T15:47:32.972 INFO:teuthology.orchestra.run.smithi123.stdout: "", 2024-09-16T15:47:32.972 INFO:teuthology.orchestra.run.smithi123.stdout: "critical", 2024-09-16T15:47:32.972 INFO:teuthology.orchestra.run.smithi123.stdout: "debug", 2024-09-16T15:47:32.972 INFO:teuthology.orchestra.run.smithi123.stdout: "error", 2024-09-16T15:47:32.972 INFO:teuthology.orchestra.run.smithi123.stdout: "info", 2024-09-16T15:47:32.972 INFO:teuthology.orchestra.run.smithi123.stdout: "warning" 2024-09-16T15:47:32.972 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-09-16T15:47:32.973 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:32.973 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:32.973 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:32.973 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:32.973 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:32.973 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_cluster": { 2024-09-16T15:47:32.973 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_cluster", 2024-09-16T15:47:32.973 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-09-16T15:47:32.973 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:32.973 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-09-16T15:47:32.973 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-09-16T15:47:32.973 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:32.973 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:32.973 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:32.974 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:32.974 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:32.974 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:32.974 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:32.974 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:32.974 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_cluster_level": { 2024-09-16T15:47:32.974 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_cluster_level", 2024-09-16T15:47:32.974 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-09-16T15:47:32.974 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:32.974 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-09-16T15:47:32.974 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "info", 2024-09-16T15:47:32.974 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:32.974 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:32.974 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [ 2024-09-16T15:47:32.974 INFO:teuthology.orchestra.run.smithi123.stdout: "", 2024-09-16T15:47:32.974 INFO:teuthology.orchestra.run.smithi123.stdout: "critical", 2024-09-16T15:47:32.975 INFO:teuthology.orchestra.run.smithi123.stdout: "debug", 2024-09-16T15:47:32.975 INFO:teuthology.orchestra.run.smithi123.stdout: "error", 2024-09-16T15:47:32.975 INFO:teuthology.orchestra.run.smithi123.stdout: "info", 2024-09-16T15:47:32.975 INFO:teuthology.orchestra.run.smithi123.stdout: "warning" 2024-09-16T15:47:32.975 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-09-16T15:47:32.975 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:32.975 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:32.975 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:32.975 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:32.975 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:32.975 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_file": { 2024-09-16T15:47:32.975 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_file", 2024-09-16T15:47:32.975 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-09-16T15:47:32.975 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:32.975 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-09-16T15:47:32.976 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-09-16T15:47:32.976 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:32.976 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:32.976 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:32.976 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:32.976 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:32.976 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:32.976 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:32.976 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:32.976 INFO:teuthology.orchestra.run.smithi123.stdout: "sqlite3_killpoint": { 2024-09-16T15:47:32.976 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "sqlite3_killpoint", 2024-09-16T15:47:32.976 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "int", 2024-09-16T15:47:32.976 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "dev", 2024-09-16T15:47:32.976 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-09-16T15:47:32.977 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "0", 2024-09-16T15:47:32.977 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:32.977 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:32.977 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:32.977 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:32.977 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:32.977 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:32.977 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:32.977 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-16T15:47:32.977 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-16T15:47:32.977 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:32.977 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-09-16T15:47:32.977 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "iostat", 2024-09-16T15:47:32.977 INFO:teuthology.orchestra.run.smithi123.stdout: "can_run": true, 2024-09-16T15:47:32.978 INFO:teuthology.orchestra.run.smithi123.stdout: "error_string": "", 2024-09-16T15:47:32.978 INFO:teuthology.orchestra.run.smithi123.stdout: "module_options": { 2024-09-16T15:47:32.978 INFO:teuthology.orchestra.run.smithi123.stdout: "log_level": { 2024-09-16T15:47:32.978 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_level", 2024-09-16T15:47:32.978 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-09-16T15:47:32.978 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:32.978 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-09-16T15:47:32.978 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "", 2024-09-16T15:47:32.978 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:32.978 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:32.978 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [ 2024-09-16T15:47:32.978 INFO:teuthology.orchestra.run.smithi123.stdout: "", 2024-09-16T15:47:32.979 INFO:teuthology.orchestra.run.smithi123.stdout: "critical", 2024-09-16T15:47:32.979 INFO:teuthology.orchestra.run.smithi123.stdout: "debug", 2024-09-16T15:47:32.979 INFO:teuthology.orchestra.run.smithi123.stdout: "error", 2024-09-16T15:47:32.979 INFO:teuthology.orchestra.run.smithi123.stdout: "info", 2024-09-16T15:47:32.979 INFO:teuthology.orchestra.run.smithi123.stdout: "warning" 2024-09-16T15:47:32.979 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-09-16T15:47:32.979 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:32.980 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:32.980 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:32.980 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:32.980 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:32.980 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_cluster": { 2024-09-16T15:47:32.980 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_cluster", 2024-09-16T15:47:32.980 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-09-16T15:47:32.980 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:32.980 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-09-16T15:47:32.980 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-09-16T15:47:32.980 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:32.981 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:32.981 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:32.981 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:32.981 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:32.981 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:32.981 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:32.981 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:32.981 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_cluster_level": { 2024-09-16T15:47:32.981 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_cluster_level", 2024-09-16T15:47:32.981 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-09-16T15:47:32.981 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:32.981 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-09-16T15:47:32.981 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "info", 2024-09-16T15:47:32.981 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:32.982 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:32.982 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [ 2024-09-16T15:47:32.982 INFO:teuthology.orchestra.run.smithi123.stdout: "", 2024-09-16T15:47:32.982 INFO:teuthology.orchestra.run.smithi123.stdout: "critical", 2024-09-16T15:47:32.982 INFO:teuthology.orchestra.run.smithi123.stdout: "debug", 2024-09-16T15:47:32.982 INFO:teuthology.orchestra.run.smithi123.stdout: "error", 2024-09-16T15:47:32.982 INFO:teuthology.orchestra.run.smithi123.stdout: "info", 2024-09-16T15:47:32.982 INFO:teuthology.orchestra.run.smithi123.stdout: "warning" 2024-09-16T15:47:32.982 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-09-16T15:47:32.982 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:32.982 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:32.982 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:32.982 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:32.982 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:32.982 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_file": { 2024-09-16T15:47:32.983 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_file", 2024-09-16T15:47:32.983 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-09-16T15:47:32.983 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:32.983 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-09-16T15:47:32.983 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-09-16T15:47:32.983 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:32.983 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:32.983 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:32.983 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:32.983 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:32.983 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:32.983 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:32.983 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:32.984 INFO:teuthology.orchestra.run.smithi123.stdout: "sqlite3_killpoint": { 2024-09-16T15:47:32.984 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "sqlite3_killpoint", 2024-09-16T15:47:32.984 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "int", 2024-09-16T15:47:32.984 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "dev", 2024-09-16T15:47:32.984 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-09-16T15:47:32.984 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "0", 2024-09-16T15:47:32.984 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:32.984 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:32.984 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:32.984 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:32.984 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:32.985 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:32.985 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:32.985 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-16T15:47:32.985 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-16T15:47:32.985 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:32.985 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-09-16T15:47:32.985 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "localpool", 2024-09-16T15:47:32.985 INFO:teuthology.orchestra.run.smithi123.stdout: "can_run": true, 2024-09-16T15:47:32.985 INFO:teuthology.orchestra.run.smithi123.stdout: "error_string": "", 2024-09-16T15:47:32.985 INFO:teuthology.orchestra.run.smithi123.stdout: "module_options": { 2024-09-16T15:47:32.985 INFO:teuthology.orchestra.run.smithi123.stdout: "failure_domain": { 2024-09-16T15:47:32.986 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "failure_domain", 2024-09-16T15:47:32.986 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-09-16T15:47:32.986 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:32.986 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-09-16T15:47:32.986 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "host", 2024-09-16T15:47:32.986 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:32.986 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:32.986 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:32.986 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "failure domain for any created local pool", 2024-09-16T15:47:32.986 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "what failure domain we should separate data replicas across.", 2024-09-16T15:47:32.986 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:32.986 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:32.986 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:32.987 INFO:teuthology.orchestra.run.smithi123.stdout: "log_level": { 2024-09-16T15:47:32.987 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_level", 2024-09-16T15:47:32.987 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-09-16T15:47:32.987 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:32.987 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-09-16T15:47:32.987 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "", 2024-09-16T15:47:32.987 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:32.987 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:32.987 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [ 2024-09-16T15:47:32.987 INFO:teuthology.orchestra.run.smithi123.stdout: "", 2024-09-16T15:47:32.987 INFO:teuthology.orchestra.run.smithi123.stdout: "critical", 2024-09-16T15:47:32.987 INFO:teuthology.orchestra.run.smithi123.stdout: "debug", 2024-09-16T15:47:32.987 INFO:teuthology.orchestra.run.smithi123.stdout: "error", 2024-09-16T15:47:32.988 INFO:teuthology.orchestra.run.smithi123.stdout: "info", 2024-09-16T15:47:32.988 INFO:teuthology.orchestra.run.smithi123.stdout: "warning" 2024-09-16T15:47:32.988 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-09-16T15:47:32.988 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:32.988 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:32.988 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:32.988 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:32.988 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:32.988 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_cluster": { 2024-09-16T15:47:32.988 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_cluster", 2024-09-16T15:47:32.988 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-09-16T15:47:32.989 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:32.989 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-09-16T15:47:32.989 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-09-16T15:47:32.989 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:32.989 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:32.989 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:32.989 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:32.989 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:32.989 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:32.989 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:32.989 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:32.989 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_cluster_level": { 2024-09-16T15:47:32.990 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_cluster_level", 2024-09-16T15:47:32.990 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-09-16T15:47:32.990 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:32.990 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-09-16T15:47:32.990 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "info", 2024-09-16T15:47:32.990 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:32.990 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:32.990 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [ 2024-09-16T15:47:32.990 INFO:teuthology.orchestra.run.smithi123.stdout: "", 2024-09-16T15:47:32.990 INFO:teuthology.orchestra.run.smithi123.stdout: "critical", 2024-09-16T15:47:32.990 INFO:teuthology.orchestra.run.smithi123.stdout: "debug", 2024-09-16T15:47:32.990 INFO:teuthology.orchestra.run.smithi123.stdout: "error", 2024-09-16T15:47:32.990 INFO:teuthology.orchestra.run.smithi123.stdout: "info", 2024-09-16T15:47:32.991 INFO:teuthology.orchestra.run.smithi123.stdout: "warning" 2024-09-16T15:47:32.991 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-09-16T15:47:32.991 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:32.991 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:32.991 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:32.991 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:32.991 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:32.991 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_file": { 2024-09-16T15:47:32.991 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_file", 2024-09-16T15:47:32.991 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-09-16T15:47:32.991 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:32.992 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-09-16T15:47:32.992 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-09-16T15:47:32.992 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:32.992 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:32.992 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:32.992 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:32.992 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:32.992 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:32.992 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:32.992 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:32.992 INFO:teuthology.orchestra.run.smithi123.stdout: "min_size": { 2024-09-16T15:47:32.993 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "min_size", 2024-09-16T15:47:32.993 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "int", 2024-09-16T15:47:32.993 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:32.993 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-09-16T15:47:32.993 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "", 2024-09-16T15:47:32.993 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:32.993 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:32.993 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:32.993 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "default min_size for any created local pool", 2024-09-16T15:47:32.993 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "value to set min_size to (unchanged from Ceph's default if this option is not set)", 2024-09-16T15:47:32.993 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:32.993 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:32.993 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:32.994 INFO:teuthology.orchestra.run.smithi123.stdout: "num_rep": { 2024-09-16T15:47:32.994 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "num_rep", 2024-09-16T15:47:32.994 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "int", 2024-09-16T15:47:32.994 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:32.994 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-09-16T15:47:32.994 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "3", 2024-09-16T15:47:32.994 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:32.994 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:32.994 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:32.994 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "default replica count for any created local pool", 2024-09-16T15:47:32.994 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:32.994 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:32.994 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:32.994 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:32.995 INFO:teuthology.orchestra.run.smithi123.stdout: "pg_num": { 2024-09-16T15:47:32.995 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "pg_num", 2024-09-16T15:47:32.995 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "int", 2024-09-16T15:47:32.995 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:32.995 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-09-16T15:47:32.995 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "128", 2024-09-16T15:47:32.995 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:32.995 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:32.995 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:32.995 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "default pg_num for any created local pool", 2024-09-16T15:47:32.995 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:32.995 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:32.995 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:32.995 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:32.995 INFO:teuthology.orchestra.run.smithi123.stdout: "prefix": { 2024-09-16T15:47:32.996 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "prefix", 2024-09-16T15:47:32.996 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-09-16T15:47:32.996 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:32.996 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-09-16T15:47:32.996 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "", 2024-09-16T15:47:32.996 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:32.996 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:32.996 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:32.996 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "name prefix for any created local pool", 2024-09-16T15:47:32.996 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:32.996 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:32.996 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:32.996 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:32.997 INFO:teuthology.orchestra.run.smithi123.stdout: "sqlite3_killpoint": { 2024-09-16T15:47:32.997 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "sqlite3_killpoint", 2024-09-16T15:47:32.997 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "int", 2024-09-16T15:47:32.997 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "dev", 2024-09-16T15:47:32.997 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-09-16T15:47:32.997 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "0", 2024-09-16T15:47:32.997 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:32.997 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:32.997 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:32.997 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:32.997 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:32.997 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:32.997 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:32.997 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:32.998 INFO:teuthology.orchestra.run.smithi123.stdout: "subtree": { 2024-09-16T15:47:32.998 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "subtree", 2024-09-16T15:47:32.998 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-09-16T15:47:32.998 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:32.998 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-09-16T15:47:32.998 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "rack", 2024-09-16T15:47:32.998 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:32.998 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:32.998 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:32.998 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "CRUSH level for which to create a local pool", 2024-09-16T15:47:32.998 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "which CRUSH subtree type the module should create a pool for.", 2024-09-16T15:47:32.998 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:32.998 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:32.999 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-16T15:47:32.999 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-16T15:47:32.999 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:32.999 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-09-16T15:47:32.999 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "mirroring", 2024-09-16T15:47:32.999 INFO:teuthology.orchestra.run.smithi123.stdout: "can_run": true, 2024-09-16T15:47:32.999 INFO:teuthology.orchestra.run.smithi123.stdout: "error_string": "", 2024-09-16T15:47:32.999 INFO:teuthology.orchestra.run.smithi123.stdout: "module_options": { 2024-09-16T15:47:32.999 INFO:teuthology.orchestra.run.smithi123.stdout: "log_level": { 2024-09-16T15:47:32.999 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_level", 2024-09-16T15:47:32.999 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-09-16T15:47:32.999 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:32.999 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-09-16T15:47:33.000 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "", 2024-09-16T15:47:33.000 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.000 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.000 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [ 2024-09-16T15:47:33.000 INFO:teuthology.orchestra.run.smithi123.stdout: "", 2024-09-16T15:47:33.000 INFO:teuthology.orchestra.run.smithi123.stdout: "critical", 2024-09-16T15:47:33.000 INFO:teuthology.orchestra.run.smithi123.stdout: "debug", 2024-09-16T15:47:33.000 INFO:teuthology.orchestra.run.smithi123.stdout: "error", 2024-09-16T15:47:33.000 INFO:teuthology.orchestra.run.smithi123.stdout: "info", 2024-09-16T15:47:33.000 INFO:teuthology.orchestra.run.smithi123.stdout: "warning" 2024-09-16T15:47:33.000 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-09-16T15:47:33.000 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:33.000 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.001 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.001 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.001 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.001 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_cluster": { 2024-09-16T15:47:33.001 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_cluster", 2024-09-16T15:47:33.001 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-09-16T15:47:33.001 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.001 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-09-16T15:47:33.001 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-09-16T15:47:33.001 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.001 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.001 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.002 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:33.002 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.002 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.002 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.002 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.002 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_cluster_level": { 2024-09-16T15:47:33.002 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_cluster_level", 2024-09-16T15:47:33.002 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-09-16T15:47:33.002 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.002 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-09-16T15:47:33.002 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "info", 2024-09-16T15:47:33.002 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.002 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.003 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [ 2024-09-16T15:47:33.003 INFO:teuthology.orchestra.run.smithi123.stdout: "", 2024-09-16T15:47:33.003 INFO:teuthology.orchestra.run.smithi123.stdout: "critical", 2024-09-16T15:47:33.003 INFO:teuthology.orchestra.run.smithi123.stdout: "debug", 2024-09-16T15:47:33.003 INFO:teuthology.orchestra.run.smithi123.stdout: "error", 2024-09-16T15:47:33.003 INFO:teuthology.orchestra.run.smithi123.stdout: "info", 2024-09-16T15:47:33.003 INFO:teuthology.orchestra.run.smithi123.stdout: "warning" 2024-09-16T15:47:33.003 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-09-16T15:47:33.003 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:33.003 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.003 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.003 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.003 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.004 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_file": { 2024-09-16T15:47:33.004 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_file", 2024-09-16T15:47:33.004 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-09-16T15:47:33.004 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.004 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-09-16T15:47:33.004 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-09-16T15:47:33.004 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.004 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.004 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.004 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:33.004 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.004 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.004 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.005 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.005 INFO:teuthology.orchestra.run.smithi123.stdout: "sqlite3_killpoint": { 2024-09-16T15:47:33.005 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "sqlite3_killpoint", 2024-09-16T15:47:33.005 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "int", 2024-09-16T15:47:33.005 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "dev", 2024-09-16T15:47:33.005 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-09-16T15:47:33.005 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "0", 2024-09-16T15:47:33.005 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.005 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.005 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.005 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:33.005 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.005 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.005 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.005 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-16T15:47:33.006 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-16T15:47:33.006 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.006 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-09-16T15:47:33.006 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "nfs", 2024-09-16T15:47:33.006 INFO:teuthology.orchestra.run.smithi123.stdout: "can_run": true, 2024-09-16T15:47:33.006 INFO:teuthology.orchestra.run.smithi123.stdout: "error_string": "", 2024-09-16T15:47:33.006 INFO:teuthology.orchestra.run.smithi123.stdout: "module_options": { 2024-09-16T15:47:33.006 INFO:teuthology.orchestra.run.smithi123.stdout: "log_level": { 2024-09-16T15:47:33.006 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_level", 2024-09-16T15:47:33.006 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-09-16T15:47:33.006 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.006 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-09-16T15:47:33.006 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "", 2024-09-16T15:47:33.006 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.007 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.007 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [ 2024-09-16T15:47:33.007 INFO:teuthology.orchestra.run.smithi123.stdout: "", 2024-09-16T15:47:33.007 INFO:teuthology.orchestra.run.smithi123.stdout: "critical", 2024-09-16T15:47:33.007 INFO:teuthology.orchestra.run.smithi123.stdout: "debug", 2024-09-16T15:47:33.007 INFO:teuthology.orchestra.run.smithi123.stdout: "error", 2024-09-16T15:47:33.007 INFO:teuthology.orchestra.run.smithi123.stdout: "info", 2024-09-16T15:47:33.007 INFO:teuthology.orchestra.run.smithi123.stdout: "warning" 2024-09-16T15:47:33.007 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-09-16T15:47:33.007 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:33.008 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.008 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.008 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.008 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.008 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_cluster": { 2024-09-16T15:47:33.008 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_cluster", 2024-09-16T15:47:33.008 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-09-16T15:47:33.008 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.008 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-09-16T15:47:33.008 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-09-16T15:47:33.008 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.008 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.009 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.009 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:33.009 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.009 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.009 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.009 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.009 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_cluster_level": { 2024-09-16T15:47:33.009 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_cluster_level", 2024-09-16T15:47:33.009 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-09-16T15:47:33.009 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.009 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-09-16T15:47:33.009 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "info", 2024-09-16T15:47:33.009 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.009 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.009 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [ 2024-09-16T15:47:33.010 INFO:teuthology.orchestra.run.smithi123.stdout: "", 2024-09-16T15:47:33.010 INFO:teuthology.orchestra.run.smithi123.stdout: "critical", 2024-09-16T15:47:33.010 INFO:teuthology.orchestra.run.smithi123.stdout: "debug", 2024-09-16T15:47:33.010 INFO:teuthology.orchestra.run.smithi123.stdout: "error", 2024-09-16T15:47:33.010 INFO:teuthology.orchestra.run.smithi123.stdout: "info", 2024-09-16T15:47:33.010 INFO:teuthology.orchestra.run.smithi123.stdout: "warning" 2024-09-16T15:47:33.010 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-09-16T15:47:33.010 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:33.010 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.010 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.010 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.010 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.010 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_file": { 2024-09-16T15:47:33.010 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_file", 2024-09-16T15:47:33.010 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-09-16T15:47:33.010 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.011 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-09-16T15:47:33.011 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-09-16T15:47:33.011 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.011 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.011 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.011 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:33.011 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.011 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.011 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.011 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.011 INFO:teuthology.orchestra.run.smithi123.stdout: "sqlite3_killpoint": { 2024-09-16T15:47:33.011 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "sqlite3_killpoint", 2024-09-16T15:47:33.012 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "int", 2024-09-16T15:47:33.012 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "dev", 2024-09-16T15:47:33.012 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-09-16T15:47:33.012 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "0", 2024-09-16T15:47:33.012 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.012 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.012 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.012 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:33.012 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.012 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.012 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.012 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-16T15:47:33.012 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-16T15:47:33.012 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.013 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-09-16T15:47:33.013 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "orchestrator", 2024-09-16T15:47:33.013 INFO:teuthology.orchestra.run.smithi123.stdout: "can_run": true, 2024-09-16T15:47:33.013 INFO:teuthology.orchestra.run.smithi123.stdout: "error_string": "", 2024-09-16T15:47:33.013 INFO:teuthology.orchestra.run.smithi123.stdout: "module_options": { 2024-09-16T15:47:33.013 INFO:teuthology.orchestra.run.smithi123.stdout: "fail_fs": { 2024-09-16T15:47:33.013 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "fail_fs", 2024-09-16T15:47:33.013 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-09-16T15:47:33.013 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.013 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-09-16T15:47:33.013 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-09-16T15:47:33.013 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.014 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.014 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.014 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "Fail filesystem for rapid multi-rank mds upgrade", 2024-09-16T15:47:33.014 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.014 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.014 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.014 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.014 INFO:teuthology.orchestra.run.smithi123.stdout: "log_level": { 2024-09-16T15:47:33.014 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_level", 2024-09-16T15:47:33.014 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-09-16T15:47:33.014 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.014 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-09-16T15:47:33.014 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "", 2024-09-16T15:47:33.014 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.015 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.015 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [ 2024-09-16T15:47:33.015 INFO:teuthology.orchestra.run.smithi123.stdout: "", 2024-09-16T15:47:33.015 INFO:teuthology.orchestra.run.smithi123.stdout: "critical", 2024-09-16T15:47:33.015 INFO:teuthology.orchestra.run.smithi123.stdout: "debug", 2024-09-16T15:47:33.015 INFO:teuthology.orchestra.run.smithi123.stdout: "error", 2024-09-16T15:47:33.015 INFO:teuthology.orchestra.run.smithi123.stdout: "info", 2024-09-16T15:47:33.015 INFO:teuthology.orchestra.run.smithi123.stdout: "warning" 2024-09-16T15:47:33.015 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-09-16T15:47:33.015 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:33.015 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.015 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.015 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.016 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.016 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_cluster": { 2024-09-16T15:47:33.016 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_cluster", 2024-09-16T15:47:33.016 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-09-16T15:47:33.016 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.016 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-09-16T15:47:33.016 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-09-16T15:47:33.016 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.016 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.016 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.016 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:33.016 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.017 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.017 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.017 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.017 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_cluster_level": { 2024-09-16T15:47:33.017 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_cluster_level", 2024-09-16T15:47:33.017 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-09-16T15:47:33.017 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.017 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-09-16T15:47:33.017 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "info", 2024-09-16T15:47:33.017 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.017 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.017 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [ 2024-09-16T15:47:33.017 INFO:teuthology.orchestra.run.smithi123.stdout: "", 2024-09-16T15:47:33.018 INFO:teuthology.orchestra.run.smithi123.stdout: "critical", 2024-09-16T15:47:33.018 INFO:teuthology.orchestra.run.smithi123.stdout: "debug", 2024-09-16T15:47:33.018 INFO:teuthology.orchestra.run.smithi123.stdout: "error", 2024-09-16T15:47:33.018 INFO:teuthology.orchestra.run.smithi123.stdout: "info", 2024-09-16T15:47:33.018 INFO:teuthology.orchestra.run.smithi123.stdout: "warning" 2024-09-16T15:47:33.018 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-09-16T15:47:33.018 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:33.018 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.018 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.018 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.018 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.018 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_file": { 2024-09-16T15:47:33.018 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_file", 2024-09-16T15:47:33.018 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-09-16T15:47:33.018 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.019 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-09-16T15:47:33.019 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-09-16T15:47:33.019 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.019 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.019 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.019 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:33.019 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.019 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.019 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.019 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.019 INFO:teuthology.orchestra.run.smithi123.stdout: "orchestrator": { 2024-09-16T15:47:33.019 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "orchestrator", 2024-09-16T15:47:33.019 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-09-16T15:47:33.019 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.019 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-09-16T15:47:33.020 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "", 2024-09-16T15:47:33.020 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.020 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.020 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [ 2024-09-16T15:47:33.020 INFO:teuthology.orchestra.run.smithi123.stdout: "cephadm", 2024-09-16T15:47:33.020 INFO:teuthology.orchestra.run.smithi123.stdout: "rook", 2024-09-16T15:47:33.020 INFO:teuthology.orchestra.run.smithi123.stdout: "test_orchestrator" 2024-09-16T15:47:33.020 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-09-16T15:47:33.020 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "Orchestrator backend", 2024-09-16T15:47:33.020 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.020 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.020 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.020 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.020 INFO:teuthology.orchestra.run.smithi123.stdout: "sqlite3_killpoint": { 2024-09-16T15:47:33.020 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "sqlite3_killpoint", 2024-09-16T15:47:33.021 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "int", 2024-09-16T15:47:33.021 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "dev", 2024-09-16T15:47:33.021 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-09-16T15:47:33.021 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "0", 2024-09-16T15:47:33.021 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.021 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.021 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.021 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:33.021 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.021 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.021 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.021 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-16T15:47:33.021 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-16T15:47:33.021 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.021 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-09-16T15:47:33.022 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "osd_perf_query", 2024-09-16T15:47:33.022 INFO:teuthology.orchestra.run.smithi123.stdout: "can_run": true, 2024-09-16T15:47:33.022 INFO:teuthology.orchestra.run.smithi123.stdout: "error_string": "", 2024-09-16T15:47:33.022 INFO:teuthology.orchestra.run.smithi123.stdout: "module_options": { 2024-09-16T15:47:33.022 INFO:teuthology.orchestra.run.smithi123.stdout: "log_level": { 2024-09-16T15:47:33.022 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_level", 2024-09-16T15:47:33.022 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-09-16T15:47:33.022 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.022 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-09-16T15:47:33.022 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "", 2024-09-16T15:47:33.022 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.022 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.022 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [ 2024-09-16T15:47:33.022 INFO:teuthology.orchestra.run.smithi123.stdout: "", 2024-09-16T15:47:33.022 INFO:teuthology.orchestra.run.smithi123.stdout: "critical", 2024-09-16T15:47:33.023 INFO:teuthology.orchestra.run.smithi123.stdout: "debug", 2024-09-16T15:47:33.023 INFO:teuthology.orchestra.run.smithi123.stdout: "error", 2024-09-16T15:47:33.023 INFO:teuthology.orchestra.run.smithi123.stdout: "info", 2024-09-16T15:47:33.023 INFO:teuthology.orchestra.run.smithi123.stdout: "warning" 2024-09-16T15:47:33.023 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-09-16T15:47:33.023 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:33.023 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.023 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.023 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.023 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.023 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_cluster": { 2024-09-16T15:47:33.023 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_cluster", 2024-09-16T15:47:33.023 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-09-16T15:47:33.023 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.023 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-09-16T15:47:33.023 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-09-16T15:47:33.024 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.024 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.024 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.024 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:33.024 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.024 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.024 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.024 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.024 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_cluster_level": { 2024-09-16T15:47:33.024 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_cluster_level", 2024-09-16T15:47:33.024 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-09-16T15:47:33.024 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.024 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-09-16T15:47:33.024 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "info", 2024-09-16T15:47:33.024 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.025 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.025 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [ 2024-09-16T15:47:33.025 INFO:teuthology.orchestra.run.smithi123.stdout: "", 2024-09-16T15:47:33.025 INFO:teuthology.orchestra.run.smithi123.stdout: "critical", 2024-09-16T15:47:33.025 INFO:teuthology.orchestra.run.smithi123.stdout: "debug", 2024-09-16T15:47:33.025 INFO:teuthology.orchestra.run.smithi123.stdout: "error", 2024-09-16T15:47:33.025 INFO:teuthology.orchestra.run.smithi123.stdout: "info", 2024-09-16T15:47:33.025 INFO:teuthology.orchestra.run.smithi123.stdout: "warning" 2024-09-16T15:47:33.025 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-09-16T15:47:33.025 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:33.025 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.025 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.025 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.025 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.025 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_file": { 2024-09-16T15:47:33.026 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_file", 2024-09-16T15:47:33.026 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-09-16T15:47:33.026 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.026 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-09-16T15:47:33.026 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-09-16T15:47:33.026 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.026 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.026 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.026 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:33.026 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.026 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.026 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.026 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.026 INFO:teuthology.orchestra.run.smithi123.stdout: "sqlite3_killpoint": { 2024-09-16T15:47:33.026 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "sqlite3_killpoint", 2024-09-16T15:47:33.027 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "int", 2024-09-16T15:47:33.027 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "dev", 2024-09-16T15:47:33.027 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-09-16T15:47:33.027 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "0", 2024-09-16T15:47:33.027 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.027 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.027 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.027 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:33.027 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.027 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.027 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.027 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-16T15:47:33.027 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-16T15:47:33.027 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.027 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-09-16T15:47:33.028 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "osd_support", 2024-09-16T15:47:33.028 INFO:teuthology.orchestra.run.smithi123.stdout: "can_run": true, 2024-09-16T15:47:33.028 INFO:teuthology.orchestra.run.smithi123.stdout: "error_string": "", 2024-09-16T15:47:33.028 INFO:teuthology.orchestra.run.smithi123.stdout: "module_options": { 2024-09-16T15:47:33.028 INFO:teuthology.orchestra.run.smithi123.stdout: "log_level": { 2024-09-16T15:47:33.028 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_level", 2024-09-16T15:47:33.028 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-09-16T15:47:33.028 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.028 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-09-16T15:47:33.028 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "", 2024-09-16T15:47:33.028 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.028 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.028 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [ 2024-09-16T15:47:33.028 INFO:teuthology.orchestra.run.smithi123.stdout: "", 2024-09-16T15:47:33.028 INFO:teuthology.orchestra.run.smithi123.stdout: "critical", 2024-09-16T15:47:33.029 INFO:teuthology.orchestra.run.smithi123.stdout: "debug", 2024-09-16T15:47:33.029 INFO:teuthology.orchestra.run.smithi123.stdout: "error", 2024-09-16T15:47:33.029 INFO:teuthology.orchestra.run.smithi123.stdout: "info", 2024-09-16T15:47:33.029 INFO:teuthology.orchestra.run.smithi123.stdout: "warning" 2024-09-16T15:47:33.029 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-09-16T15:47:33.029 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:33.029 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.029 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.029 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.029 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.029 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_cluster": { 2024-09-16T15:47:33.029 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_cluster", 2024-09-16T15:47:33.029 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-09-16T15:47:33.029 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.029 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-09-16T15:47:33.029 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-09-16T15:47:33.030 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.030 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.030 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.030 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:33.030 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.030 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.030 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.030 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.030 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_cluster_level": { 2024-09-16T15:47:33.030 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_cluster_level", 2024-09-16T15:47:33.030 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-09-16T15:47:33.030 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.030 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-09-16T15:47:33.030 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "info", 2024-09-16T15:47:33.030 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.031 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.031 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [ 2024-09-16T15:47:33.031 INFO:teuthology.orchestra.run.smithi123.stdout: "", 2024-09-16T15:47:33.031 INFO:teuthology.orchestra.run.smithi123.stdout: "critical", 2024-09-16T15:47:33.031 INFO:teuthology.orchestra.run.smithi123.stdout: "debug", 2024-09-16T15:47:33.031 INFO:teuthology.orchestra.run.smithi123.stdout: "error", 2024-09-16T15:47:33.031 INFO:teuthology.orchestra.run.smithi123.stdout: "info", 2024-09-16T15:47:33.031 INFO:teuthology.orchestra.run.smithi123.stdout: "warning" 2024-09-16T15:47:33.031 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-09-16T15:47:33.031 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:33.031 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.031 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.031 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.031 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.031 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_file": { 2024-09-16T15:47:33.032 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_file", 2024-09-16T15:47:33.032 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-09-16T15:47:33.032 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.032 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-09-16T15:47:33.032 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-09-16T15:47:33.032 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.032 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.032 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.032 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:33.032 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.032 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.032 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.032 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.032 INFO:teuthology.orchestra.run.smithi123.stdout: "sqlite3_killpoint": { 2024-09-16T15:47:33.032 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "sqlite3_killpoint", 2024-09-16T15:47:33.032 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "int", 2024-09-16T15:47:33.033 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "dev", 2024-09-16T15:47:33.033 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-09-16T15:47:33.033 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "0", 2024-09-16T15:47:33.033 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.033 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.033 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.033 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:33.033 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.033 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.033 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.033 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-16T15:47:33.033 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-16T15:47:33.033 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.033 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-09-16T15:47:33.033 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "pg_autoscaler", 2024-09-16T15:47:33.034 INFO:teuthology.orchestra.run.smithi123.stdout: "can_run": true, 2024-09-16T15:47:33.034 INFO:teuthology.orchestra.run.smithi123.stdout: "error_string": "", 2024-09-16T15:47:33.034 INFO:teuthology.orchestra.run.smithi123.stdout: "module_options": { 2024-09-16T15:47:33.034 INFO:teuthology.orchestra.run.smithi123.stdout: "log_level": { 2024-09-16T15:47:33.034 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_level", 2024-09-16T15:47:33.034 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-09-16T15:47:33.034 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.034 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-09-16T15:47:33.034 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "", 2024-09-16T15:47:33.034 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.034 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.034 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [ 2024-09-16T15:47:33.034 INFO:teuthology.orchestra.run.smithi123.stdout: "", 2024-09-16T15:47:33.035 INFO:teuthology.orchestra.run.smithi123.stdout: "critical", 2024-09-16T15:47:33.035 INFO:teuthology.orchestra.run.smithi123.stdout: "debug", 2024-09-16T15:47:33.035 INFO:teuthology.orchestra.run.smithi123.stdout: "error", 2024-09-16T15:47:33.035 INFO:teuthology.orchestra.run.smithi123.stdout: "info", 2024-09-16T15:47:33.035 INFO:teuthology.orchestra.run.smithi123.stdout: "warning" 2024-09-16T15:47:33.035 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-09-16T15:47:33.035 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:33.035 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.035 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.035 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.035 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.035 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_cluster": { 2024-09-16T15:47:33.035 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_cluster", 2024-09-16T15:47:33.035 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-09-16T15:47:33.035 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.035 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-09-16T15:47:33.036 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-09-16T15:47:33.036 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.036 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.036 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.036 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:33.036 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.036 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.036 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.036 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.036 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_cluster_level": { 2024-09-16T15:47:33.036 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_cluster_level", 2024-09-16T15:47:33.036 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-09-16T15:47:33.036 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.036 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-09-16T15:47:33.036 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "info", 2024-09-16T15:47:33.037 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.037 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.037 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [ 2024-09-16T15:47:33.037 INFO:teuthology.orchestra.run.smithi123.stdout: "", 2024-09-16T15:47:33.037 INFO:teuthology.orchestra.run.smithi123.stdout: "critical", 2024-09-16T15:47:33.037 INFO:teuthology.orchestra.run.smithi123.stdout: "debug", 2024-09-16T15:47:33.037 INFO:teuthology.orchestra.run.smithi123.stdout: "error", 2024-09-16T15:47:33.037 INFO:teuthology.orchestra.run.smithi123.stdout: "info", 2024-09-16T15:47:33.037 INFO:teuthology.orchestra.run.smithi123.stdout: "warning" 2024-09-16T15:47:33.037 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-09-16T15:47:33.037 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:33.037 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.037 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.037 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.037 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.038 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_file": { 2024-09-16T15:47:33.038 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_file", 2024-09-16T15:47:33.038 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-09-16T15:47:33.038 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.038 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-09-16T15:47:33.038 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-09-16T15:47:33.038 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.038 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.038 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.038 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:33.038 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.038 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.038 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.038 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.039 INFO:teuthology.orchestra.run.smithi123.stdout: "sleep_interval": { 2024-09-16T15:47:33.039 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "sleep_interval", 2024-09-16T15:47:33.039 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "secs", 2024-09-16T15:47:33.039 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.039 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-09-16T15:47:33.039 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "60", 2024-09-16T15:47:33.039 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.039 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.039 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.039 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:33.039 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.039 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.039 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.039 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.039 INFO:teuthology.orchestra.run.smithi123.stdout: "sqlite3_killpoint": { 2024-09-16T15:47:33.039 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "sqlite3_killpoint", 2024-09-16T15:47:33.040 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "int", 2024-09-16T15:47:33.040 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "dev", 2024-09-16T15:47:33.040 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-09-16T15:47:33.040 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "0", 2024-09-16T15:47:33.040 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.040 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.040 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.040 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:33.040 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.040 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.040 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.040 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.040 INFO:teuthology.orchestra.run.smithi123.stdout: "threshold": { 2024-09-16T15:47:33.040 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "threshold", 2024-09-16T15:47:33.040 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "float", 2024-09-16T15:47:33.041 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.041 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-09-16T15:47:33.041 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "3.0", 2024-09-16T15:47:33.041 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "1.0", 2024-09-16T15:47:33.041 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.041 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.041 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "scaling threshold", 2024-09-16T15:47:33.041 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0", 2024-09-16T15:47:33.041 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.041 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.041 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-16T15:47:33.041 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-16T15:47:33.041 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.041 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-09-16T15:47:33.041 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "progress", 2024-09-16T15:47:33.041 INFO:teuthology.orchestra.run.smithi123.stdout: "can_run": true, 2024-09-16T15:47:33.042 INFO:teuthology.orchestra.run.smithi123.stdout: "error_string": "", 2024-09-16T15:47:33.042 INFO:teuthology.orchestra.run.smithi123.stdout: "module_options": { 2024-09-16T15:47:33.042 INFO:teuthology.orchestra.run.smithi123.stdout: "allow_pg_recovery_event": { 2024-09-16T15:47:33.042 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "allow_pg_recovery_event", 2024-09-16T15:47:33.042 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-09-16T15:47:33.042 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.042 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-09-16T15:47:33.042 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-09-16T15:47:33.042 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.042 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.042 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.042 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "allow the module to show pg recovery progress", 2024-09-16T15:47:33.042 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.042 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.042 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.043 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.043 INFO:teuthology.orchestra.run.smithi123.stdout: "enabled": { 2024-09-16T15:47:33.043 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "enabled", 2024-09-16T15:47:33.043 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-09-16T15:47:33.043 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.043 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-09-16T15:47:33.043 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "True", 2024-09-16T15:47:33.043 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.043 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.043 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.043 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:33.043 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.043 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.043 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.043 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.043 INFO:teuthology.orchestra.run.smithi123.stdout: "log_level": { 2024-09-16T15:47:33.044 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_level", 2024-09-16T15:47:33.044 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-09-16T15:47:33.044 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.044 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-09-16T15:47:33.044 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "", 2024-09-16T15:47:33.044 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.044 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.044 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [ 2024-09-16T15:47:33.044 INFO:teuthology.orchestra.run.smithi123.stdout: "", 2024-09-16T15:47:33.044 INFO:teuthology.orchestra.run.smithi123.stdout: "critical", 2024-09-16T15:47:33.044 INFO:teuthology.orchestra.run.smithi123.stdout: "debug", 2024-09-16T15:47:33.044 INFO:teuthology.orchestra.run.smithi123.stdout: "error", 2024-09-16T15:47:33.044 INFO:teuthology.orchestra.run.smithi123.stdout: "info", 2024-09-16T15:47:33.044 INFO:teuthology.orchestra.run.smithi123.stdout: "warning" 2024-09-16T15:47:33.044 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-09-16T15:47:33.045 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:33.045 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.045 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.045 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.045 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.045 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_cluster": { 2024-09-16T15:47:33.045 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_cluster", 2024-09-16T15:47:33.045 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-09-16T15:47:33.045 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.045 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-09-16T15:47:33.045 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-09-16T15:47:33.045 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.045 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.045 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.045 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:33.045 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.046 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.046 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.046 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.046 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_cluster_level": { 2024-09-16T15:47:33.046 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_cluster_level", 2024-09-16T15:47:33.046 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-09-16T15:47:33.046 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.046 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-09-16T15:47:33.046 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "info", 2024-09-16T15:47:33.046 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.046 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.046 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [ 2024-09-16T15:47:33.046 INFO:teuthology.orchestra.run.smithi123.stdout: "", 2024-09-16T15:47:33.046 INFO:teuthology.orchestra.run.smithi123.stdout: "critical", 2024-09-16T15:47:33.046 INFO:teuthology.orchestra.run.smithi123.stdout: "debug", 2024-09-16T15:47:33.047 INFO:teuthology.orchestra.run.smithi123.stdout: "error", 2024-09-16T15:47:33.047 INFO:teuthology.orchestra.run.smithi123.stdout: "info", 2024-09-16T15:47:33.047 INFO:teuthology.orchestra.run.smithi123.stdout: "warning" 2024-09-16T15:47:33.047 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-09-16T15:47:33.047 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:33.047 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.047 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.047 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.047 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.047 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_file": { 2024-09-16T15:47:33.047 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_file", 2024-09-16T15:47:33.047 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-09-16T15:47:33.048 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.048 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-09-16T15:47:33.048 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-09-16T15:47:33.048 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.048 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.048 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.048 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:33.048 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.048 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.048 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.048 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.048 INFO:teuthology.orchestra.run.smithi123.stdout: "max_completed_events": { 2024-09-16T15:47:33.048 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "max_completed_events", 2024-09-16T15:47:33.049 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "int", 2024-09-16T15:47:33.049 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.049 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-09-16T15:47:33.049 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "50", 2024-09-16T15:47:33.049 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.049 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.049 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.049 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "number of past completed events to remember", 2024-09-16T15:47:33.049 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.049 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.049 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.049 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.049 INFO:teuthology.orchestra.run.smithi123.stdout: "sleep_interval": { 2024-09-16T15:47:33.049 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "sleep_interval", 2024-09-16T15:47:33.049 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "secs", 2024-09-16T15:47:33.050 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.050 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-09-16T15:47:33.050 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "5", 2024-09-16T15:47:33.050 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.050 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.050 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.050 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "how long the module is going to sleep", 2024-09-16T15:47:33.050 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.050 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.050 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.050 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.050 INFO:teuthology.orchestra.run.smithi123.stdout: "sqlite3_killpoint": { 2024-09-16T15:47:33.050 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "sqlite3_killpoint", 2024-09-16T15:47:33.050 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "int", 2024-09-16T15:47:33.051 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "dev", 2024-09-16T15:47:33.051 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-09-16T15:47:33.051 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "0", 2024-09-16T15:47:33.051 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.051 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.051 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.051 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:33.051 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.051 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.051 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.051 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-16T15:47:33.051 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-16T15:47:33.051 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.051 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-09-16T15:47:33.051 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "prometheus", 2024-09-16T15:47:33.052 INFO:teuthology.orchestra.run.smithi123.stdout: "can_run": true, 2024-09-16T15:47:33.052 INFO:teuthology.orchestra.run.smithi123.stdout: "error_string": "", 2024-09-16T15:47:33.052 INFO:teuthology.orchestra.run.smithi123.stdout: "module_options": { 2024-09-16T15:47:33.052 INFO:teuthology.orchestra.run.smithi123.stdout: "cache": { 2024-09-16T15:47:33.052 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "cache", 2024-09-16T15:47:33.052 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-09-16T15:47:33.052 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.052 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-09-16T15:47:33.052 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "True", 2024-09-16T15:47:33.052 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.052 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.052 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.052 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:33.052 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.053 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.053 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.053 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.053 INFO:teuthology.orchestra.run.smithi123.stdout: "exclude_perf_counters": { 2024-09-16T15:47:33.053 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "exclude_perf_counters", 2024-09-16T15:47:33.053 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-09-16T15:47:33.053 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.053 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-09-16T15:47:33.053 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "True", 2024-09-16T15:47:33.053 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.053 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.053 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.053 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "Do not include perf-counters in the metrics output", 2024-09-16T15:47:33.053 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.", 2024-09-16T15:47:33.054 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.054 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.054 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.054 INFO:teuthology.orchestra.run.smithi123.stdout: "log_level": { 2024-09-16T15:47:33.054 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_level", 2024-09-16T15:47:33.054 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-09-16T15:47:33.054 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.054 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-09-16T15:47:33.054 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "", 2024-09-16T15:47:33.054 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.054 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.054 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [ 2024-09-16T15:47:33.054 INFO:teuthology.orchestra.run.smithi123.stdout: "", 2024-09-16T15:47:33.054 INFO:teuthology.orchestra.run.smithi123.stdout: "critical", 2024-09-16T15:47:33.054 INFO:teuthology.orchestra.run.smithi123.stdout: "debug", 2024-09-16T15:47:33.055 INFO:teuthology.orchestra.run.smithi123.stdout: "error", 2024-09-16T15:47:33.055 INFO:teuthology.orchestra.run.smithi123.stdout: "info", 2024-09-16T15:47:33.055 INFO:teuthology.orchestra.run.smithi123.stdout: "warning" 2024-09-16T15:47:33.055 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-09-16T15:47:33.055 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:33.055 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.055 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.055 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.055 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.055 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_cluster": { 2024-09-16T15:47:33.055 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_cluster", 2024-09-16T15:47:33.055 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-09-16T15:47:33.055 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.055 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-09-16T15:47:33.056 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-09-16T15:47:33.056 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.056 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.056 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.056 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:33.056 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.056 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.056 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.056 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.056 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_cluster_level": { 2024-09-16T15:47:33.056 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_cluster_level", 2024-09-16T15:47:33.056 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-09-16T15:47:33.056 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.056 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-09-16T15:47:33.057 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "info", 2024-09-16T15:47:33.057 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.057 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.057 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [ 2024-09-16T15:47:33.057 INFO:teuthology.orchestra.run.smithi123.stdout: "", 2024-09-16T15:47:33.057 INFO:teuthology.orchestra.run.smithi123.stdout: "critical", 2024-09-16T15:47:33.057 INFO:teuthology.orchestra.run.smithi123.stdout: "debug", 2024-09-16T15:47:33.057 INFO:teuthology.orchestra.run.smithi123.stdout: "error", 2024-09-16T15:47:33.057 INFO:teuthology.orchestra.run.smithi123.stdout: "info", 2024-09-16T15:47:33.057 INFO:teuthology.orchestra.run.smithi123.stdout: "warning" 2024-09-16T15:47:33.057 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-09-16T15:47:33.057 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:33.057 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.057 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.057 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.058 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.058 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_file": { 2024-09-16T15:47:33.058 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_file", 2024-09-16T15:47:33.058 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-09-16T15:47:33.058 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.058 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-09-16T15:47:33.058 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-09-16T15:47:33.058 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.058 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.058 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.058 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:33.058 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.058 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.058 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.058 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.059 INFO:teuthology.orchestra.run.smithi123.stdout: "rbd_stats_pools": { 2024-09-16T15:47:33.059 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "rbd_stats_pools", 2024-09-16T15:47:33.059 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-09-16T15:47:33.059 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.059 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-09-16T15:47:33.059 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "", 2024-09-16T15:47:33.059 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.059 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.059 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.059 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:33.059 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.059 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.059 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.059 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.059 INFO:teuthology.orchestra.run.smithi123.stdout: "rbd_stats_pools_refresh_interval": { 2024-09-16T15:47:33.060 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "rbd_stats_pools_refresh_interval", 2024-09-16T15:47:33.060 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "int", 2024-09-16T15:47:33.060 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.060 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-09-16T15:47:33.060 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "300", 2024-09-16T15:47:33.060 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.060 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.060 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.060 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:33.060 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.060 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.060 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.060 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.060 INFO:teuthology.orchestra.run.smithi123.stdout: "scrape_interval": { 2024-09-16T15:47:33.060 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "scrape_interval", 2024-09-16T15:47:33.060 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "float", 2024-09-16T15:47:33.061 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.061 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-09-16T15:47:33.061 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "15.0", 2024-09-16T15:47:33.061 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.061 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.061 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.061 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:33.061 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.061 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.061 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.061 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.061 INFO:teuthology.orchestra.run.smithi123.stdout: "server_addr": { 2024-09-16T15:47:33.061 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "server_addr", 2024-09-16T15:47:33.061 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-09-16T15:47:33.061 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.062 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-09-16T15:47:33.062 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "::", 2024-09-16T15:47:33.062 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.062 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.062 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.062 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "the IPv4 or IPv6 address on which the module listens for HTTP requests", 2024-09-16T15:47:33.062 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.062 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.062 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.062 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.062 INFO:teuthology.orchestra.run.smithi123.stdout: "server_port": { 2024-09-16T15:47:33.062 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "server_port", 2024-09-16T15:47:33.062 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "int", 2024-09-16T15:47:33.062 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.062 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-09-16T15:47:33.062 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "9283", 2024-09-16T15:47:33.063 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.063 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.063 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.063 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "the port on which the module listens for HTTP requests", 2024-09-16T15:47:33.063 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.063 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.063 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.063 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.063 INFO:teuthology.orchestra.run.smithi123.stdout: "sqlite3_killpoint": { 2024-09-16T15:47:33.063 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "sqlite3_killpoint", 2024-09-16T15:47:33.063 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "int", 2024-09-16T15:47:33.064 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "dev", 2024-09-16T15:47:33.064 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-09-16T15:47:33.064 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "0", 2024-09-16T15:47:33.064 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.064 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.064 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.064 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:33.064 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.064 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.064 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.064 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.064 INFO:teuthology.orchestra.run.smithi123.stdout: "stale_cache_strategy": { 2024-09-16T15:47:33.064 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "stale_cache_strategy", 2024-09-16T15:47:33.064 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-09-16T15:47:33.065 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.065 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-09-16T15:47:33.065 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "log", 2024-09-16T15:47:33.065 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.065 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.065 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.065 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:33.065 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.065 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.065 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.065 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.065 INFO:teuthology.orchestra.run.smithi123.stdout: "standby_behaviour": { 2024-09-16T15:47:33.065 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "standby_behaviour", 2024-09-16T15:47:33.065 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-09-16T15:47:33.065 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.065 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-09-16T15:47:33.066 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "default", 2024-09-16T15:47:33.066 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.066 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.066 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [ 2024-09-16T15:47:33.066 INFO:teuthology.orchestra.run.smithi123.stdout: "default", 2024-09-16T15:47:33.066 INFO:teuthology.orchestra.run.smithi123.stdout: "error" 2024-09-16T15:47:33.066 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-09-16T15:47:33.066 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:33.066 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.066 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.066 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.066 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.066 INFO:teuthology.orchestra.run.smithi123.stdout: "standby_error_status_code": { 2024-09-16T15:47:33.066 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "standby_error_status_code", 2024-09-16T15:47:33.066 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "int", 2024-09-16T15:47:33.067 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.067 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-09-16T15:47:33.067 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "500", 2024-09-16T15:47:33.067 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "400", 2024-09-16T15:47:33.067 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "599", 2024-09-16T15:47:33.067 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.067 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:33.067 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.067 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.067 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.067 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-16T15:47:33.067 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-16T15:47:33.067 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.067 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-09-16T15:47:33.067 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "rbd_support", 2024-09-16T15:47:33.068 INFO:teuthology.orchestra.run.smithi123.stdout: "can_run": true, 2024-09-16T15:47:33.068 INFO:teuthology.orchestra.run.smithi123.stdout: "error_string": "", 2024-09-16T15:47:33.068 INFO:teuthology.orchestra.run.smithi123.stdout: "module_options": { 2024-09-16T15:47:33.068 INFO:teuthology.orchestra.run.smithi123.stdout: "log_level": { 2024-09-16T15:47:33.068 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_level", 2024-09-16T15:47:33.068 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-09-16T15:47:33.068 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.068 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-09-16T15:47:33.068 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "", 2024-09-16T15:47:33.068 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.068 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.068 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [ 2024-09-16T15:47:33.068 INFO:teuthology.orchestra.run.smithi123.stdout: "", 2024-09-16T15:47:33.068 INFO:teuthology.orchestra.run.smithi123.stdout: "critical", 2024-09-16T15:47:33.068 INFO:teuthology.orchestra.run.smithi123.stdout: "debug", 2024-09-16T15:47:33.068 INFO:teuthology.orchestra.run.smithi123.stdout: "error", 2024-09-16T15:47:33.069 INFO:teuthology.orchestra.run.smithi123.stdout: "info", 2024-09-16T15:47:33.069 INFO:teuthology.orchestra.run.smithi123.stdout: "warning" 2024-09-16T15:47:33.069 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-09-16T15:47:33.069 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:33.069 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.069 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.069 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.069 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.069 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_cluster": { 2024-09-16T15:47:33.069 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_cluster", 2024-09-16T15:47:33.069 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-09-16T15:47:33.069 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.069 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-09-16T15:47:33.069 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-09-16T15:47:33.069 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.070 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.070 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.070 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:33.070 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.070 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.070 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.070 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.070 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_cluster_level": { 2024-09-16T15:47:33.070 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_cluster_level", 2024-09-16T15:47:33.070 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-09-16T15:47:33.070 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.070 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-09-16T15:47:33.070 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "info", 2024-09-16T15:47:33.070 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.070 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.071 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [ 2024-09-16T15:47:33.071 INFO:teuthology.orchestra.run.smithi123.stdout: "", 2024-09-16T15:47:33.071 INFO:teuthology.orchestra.run.smithi123.stdout: "critical", 2024-09-16T15:47:33.071 INFO:teuthology.orchestra.run.smithi123.stdout: "debug", 2024-09-16T15:47:33.071 INFO:teuthology.orchestra.run.smithi123.stdout: "error", 2024-09-16T15:47:33.071 INFO:teuthology.orchestra.run.smithi123.stdout: "info", 2024-09-16T15:47:33.071 INFO:teuthology.orchestra.run.smithi123.stdout: "warning" 2024-09-16T15:47:33.071 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-09-16T15:47:33.071 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:33.071 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.071 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.071 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.071 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.071 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_file": { 2024-09-16T15:47:33.072 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_file", 2024-09-16T15:47:33.072 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-09-16T15:47:33.072 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.072 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-09-16T15:47:33.072 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-09-16T15:47:33.072 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.072 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.072 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.072 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:33.072 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.072 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.072 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.072 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.072 INFO:teuthology.orchestra.run.smithi123.stdout: "max_concurrent_snap_create": { 2024-09-16T15:47:33.073 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "max_concurrent_snap_create", 2024-09-16T15:47:33.073 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "int", 2024-09-16T15:47:33.073 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.073 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-09-16T15:47:33.073 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "10", 2024-09-16T15:47:33.073 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.073 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.073 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.073 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:33.073 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.073 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.073 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.073 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.073 INFO:teuthology.orchestra.run.smithi123.stdout: "mirror_snapshot_schedule": { 2024-09-16T15:47:33.073 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "mirror_snapshot_schedule", 2024-09-16T15:47:33.074 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-09-16T15:47:33.074 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.074 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-09-16T15:47:33.074 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "", 2024-09-16T15:47:33.074 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.074 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.074 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.074 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:33.074 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.074 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.074 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.074 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.074 INFO:teuthology.orchestra.run.smithi123.stdout: "sqlite3_killpoint": { 2024-09-16T15:47:33.074 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "sqlite3_killpoint", 2024-09-16T15:47:33.074 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "int", 2024-09-16T15:47:33.075 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "dev", 2024-09-16T15:47:33.075 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-09-16T15:47:33.075 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "0", 2024-09-16T15:47:33.075 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.075 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.075 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.075 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:33.075 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.075 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.075 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.075 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.075 INFO:teuthology.orchestra.run.smithi123.stdout: "trash_purge_schedule": { 2024-09-16T15:47:33.075 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "trash_purge_schedule", 2024-09-16T15:47:33.075 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-09-16T15:47:33.075 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.076 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-09-16T15:47:33.076 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "", 2024-09-16T15:47:33.076 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.076 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.076 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.076 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:33.076 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.076 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.076 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.076 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-16T15:47:33.076 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-16T15:47:33.076 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.076 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-09-16T15:47:33.076 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "restful", 2024-09-16T15:47:33.077 INFO:teuthology.orchestra.run.smithi123.stdout: "can_run": true, 2024-09-16T15:47:33.077 INFO:teuthology.orchestra.run.smithi123.stdout: "error_string": "", 2024-09-16T15:47:33.077 INFO:teuthology.orchestra.run.smithi123.stdout: "module_options": { 2024-09-16T15:47:33.077 INFO:teuthology.orchestra.run.smithi123.stdout: "enable_auth": { 2024-09-16T15:47:33.077 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "enable_auth", 2024-09-16T15:47:33.077 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-09-16T15:47:33.077 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.077 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-09-16T15:47:33.077 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "True", 2024-09-16T15:47:33.077 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.077 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.077 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.077 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:33.078 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.078 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.078 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.078 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.078 INFO:teuthology.orchestra.run.smithi123.stdout: "key_file": { 2024-09-16T15:47:33.078 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "key_file", 2024-09-16T15:47:33.078 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-09-16T15:47:33.078 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.078 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-09-16T15:47:33.078 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "", 2024-09-16T15:47:33.078 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.078 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.078 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.078 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:33.079 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.079 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.079 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.079 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.079 INFO:teuthology.orchestra.run.smithi123.stdout: "log_level": { 2024-09-16T15:47:33.079 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_level", 2024-09-16T15:47:33.079 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-09-16T15:47:33.079 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.079 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-09-16T15:47:33.079 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "", 2024-09-16T15:47:33.079 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.079 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.079 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [ 2024-09-16T15:47:33.079 INFO:teuthology.orchestra.run.smithi123.stdout: "", 2024-09-16T15:47:33.079 INFO:teuthology.orchestra.run.smithi123.stdout: "critical", 2024-09-16T15:47:33.079 INFO:teuthology.orchestra.run.smithi123.stdout: "debug", 2024-09-16T15:47:33.080 INFO:teuthology.orchestra.run.smithi123.stdout: "error", 2024-09-16T15:47:33.080 INFO:teuthology.orchestra.run.smithi123.stdout: "info", 2024-09-16T15:47:33.080 INFO:teuthology.orchestra.run.smithi123.stdout: "warning" 2024-09-16T15:47:33.080 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-09-16T15:47:33.080 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:33.080 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.080 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.080 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.080 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.080 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_cluster": { 2024-09-16T15:47:33.080 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_cluster", 2024-09-16T15:47:33.080 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-09-16T15:47:33.080 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.080 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-09-16T15:47:33.080 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-09-16T15:47:33.081 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.081 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.081 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.081 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:33.081 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.081 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.081 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.081 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.081 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_cluster_level": { 2024-09-16T15:47:33.081 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_cluster_level", 2024-09-16T15:47:33.081 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-09-16T15:47:33.081 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.081 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-09-16T15:47:33.081 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "info", 2024-09-16T15:47:33.081 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.082 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.082 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [ 2024-09-16T15:47:33.082 INFO:teuthology.orchestra.run.smithi123.stdout: "", 2024-09-16T15:47:33.082 INFO:teuthology.orchestra.run.smithi123.stdout: "critical", 2024-09-16T15:47:33.082 INFO:teuthology.orchestra.run.smithi123.stdout: "debug", 2024-09-16T15:47:33.082 INFO:teuthology.orchestra.run.smithi123.stdout: "error", 2024-09-16T15:47:33.082 INFO:teuthology.orchestra.run.smithi123.stdout: "info", 2024-09-16T15:47:33.082 INFO:teuthology.orchestra.run.smithi123.stdout: "warning" 2024-09-16T15:47:33.082 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-09-16T15:47:33.082 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:33.082 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.082 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.082 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.082 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.082 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_file": { 2024-09-16T15:47:33.083 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_file", 2024-09-16T15:47:33.083 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-09-16T15:47:33.083 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.083 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-09-16T15:47:33.083 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-09-16T15:47:33.083 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.083 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.083 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.083 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:33.083 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.083 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.083 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.083 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.083 INFO:teuthology.orchestra.run.smithi123.stdout: "max_requests": { 2024-09-16T15:47:33.083 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "max_requests", 2024-09-16T15:47:33.083 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "int", 2024-09-16T15:47:33.084 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.084 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-09-16T15:47:33.084 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "500", 2024-09-16T15:47:33.084 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.084 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.084 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.084 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "Maximum number of requests to keep in memory. When new request comes in, the oldest request will be removed if the number of requests exceeds the max request number.if un-finished request is removed, error message will be logged in the ceph-mgr log.", 2024-09-16T15:47:33.084 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.084 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.084 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.084 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.084 INFO:teuthology.orchestra.run.smithi123.stdout: "server_addr": { 2024-09-16T15:47:33.084 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "server_addr", 2024-09-16T15:47:33.084 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-09-16T15:47:33.084 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.085 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-09-16T15:47:33.085 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "", 2024-09-16T15:47:33.085 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.085 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.085 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.085 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:33.085 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.085 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.085 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.085 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.085 INFO:teuthology.orchestra.run.smithi123.stdout: "server_port": { 2024-09-16T15:47:33.085 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "server_port", 2024-09-16T15:47:33.085 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-09-16T15:47:33.085 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.085 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-09-16T15:47:33.085 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "", 2024-09-16T15:47:33.086 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.086 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.086 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.086 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:33.086 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.086 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.086 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.086 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.086 INFO:teuthology.orchestra.run.smithi123.stdout: "sqlite3_killpoint": { 2024-09-16T15:47:33.086 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "sqlite3_killpoint", 2024-09-16T15:47:33.086 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "int", 2024-09-16T15:47:33.086 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "dev", 2024-09-16T15:47:33.086 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-09-16T15:47:33.086 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "0", 2024-09-16T15:47:33.086 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.087 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.087 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.087 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:33.087 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.087 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.087 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.087 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-16T15:47:33.087 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-16T15:47:33.087 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.087 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-09-16T15:47:33.087 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "rgw", 2024-09-16T15:47:33.087 INFO:teuthology.orchestra.run.smithi123.stdout: "can_run": true, 2024-09-16T15:47:33.087 INFO:teuthology.orchestra.run.smithi123.stdout: "error_string": "", 2024-09-16T15:47:33.087 INFO:teuthology.orchestra.run.smithi123.stdout: "module_options": { 2024-09-16T15:47:33.087 INFO:teuthology.orchestra.run.smithi123.stdout: "log_level": { 2024-09-16T15:47:33.088 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_level", 2024-09-16T15:47:33.088 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-09-16T15:47:33.088 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.088 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-09-16T15:47:33.088 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "", 2024-09-16T15:47:33.088 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.088 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.088 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [ 2024-09-16T15:47:33.088 INFO:teuthology.orchestra.run.smithi123.stdout: "", 2024-09-16T15:47:33.088 INFO:teuthology.orchestra.run.smithi123.stdout: "critical", 2024-09-16T15:47:33.088 INFO:teuthology.orchestra.run.smithi123.stdout: "debug", 2024-09-16T15:47:33.088 INFO:teuthology.orchestra.run.smithi123.stdout: "error", 2024-09-16T15:47:33.088 INFO:teuthology.orchestra.run.smithi123.stdout: "info", 2024-09-16T15:47:33.088 INFO:teuthology.orchestra.run.smithi123.stdout: "warning" 2024-09-16T15:47:33.088 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-09-16T15:47:33.089 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:33.089 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.089 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.089 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.089 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.089 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_cluster": { 2024-09-16T15:47:33.089 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_cluster", 2024-09-16T15:47:33.089 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-09-16T15:47:33.089 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.089 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-09-16T15:47:33.089 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-09-16T15:47:33.089 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.089 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.089 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.089 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:33.090 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.090 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.090 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.090 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.090 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_cluster_level": { 2024-09-16T15:47:33.090 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_cluster_level", 2024-09-16T15:47:33.090 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-09-16T15:47:33.090 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.090 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-09-16T15:47:33.090 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "info", 2024-09-16T15:47:33.090 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.090 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.090 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [ 2024-09-16T15:47:33.090 INFO:teuthology.orchestra.run.smithi123.stdout: "", 2024-09-16T15:47:33.090 INFO:teuthology.orchestra.run.smithi123.stdout: "critical", 2024-09-16T15:47:33.090 INFO:teuthology.orchestra.run.smithi123.stdout: "debug", 2024-09-16T15:47:33.091 INFO:teuthology.orchestra.run.smithi123.stdout: "error", 2024-09-16T15:47:33.091 INFO:teuthology.orchestra.run.smithi123.stdout: "info", 2024-09-16T15:47:33.091 INFO:teuthology.orchestra.run.smithi123.stdout: "warning" 2024-09-16T15:47:33.091 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-09-16T15:47:33.091 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:33.091 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.091 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.091 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.091 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.091 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_file": { 2024-09-16T15:47:33.091 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_file", 2024-09-16T15:47:33.091 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-09-16T15:47:33.091 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.091 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-09-16T15:47:33.091 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-09-16T15:47:33.091 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.092 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.092 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.092 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:33.092 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.092 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.092 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.092 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.092 INFO:teuthology.orchestra.run.smithi123.stdout: "secondary_zone_period_retry_limit": { 2024-09-16T15:47:33.092 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "secondary_zone_period_retry_limit", 2024-09-16T15:47:33.092 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "int", 2024-09-16T15:47:33.092 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.092 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-09-16T15:47:33.092 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "5", 2024-09-16T15:47:33.092 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.092 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.093 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.093 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "RGW module period update retry limit for secondary site", 2024-09-16T15:47:33.093 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.093 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.093 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.093 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.093 INFO:teuthology.orchestra.run.smithi123.stdout: "sqlite3_killpoint": { 2024-09-16T15:47:33.093 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "sqlite3_killpoint", 2024-09-16T15:47:33.093 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "int", 2024-09-16T15:47:33.093 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "dev", 2024-09-16T15:47:33.093 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-09-16T15:47:33.093 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "0", 2024-09-16T15:47:33.093 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.093 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.093 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.093 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:33.094 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.094 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.094 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.094 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-16T15:47:33.094 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-16T15:47:33.094 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.094 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-09-16T15:47:33.094 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "selftest", 2024-09-16T15:47:33.094 INFO:teuthology.orchestra.run.smithi123.stdout: "can_run": true, 2024-09-16T15:47:33.094 INFO:teuthology.orchestra.run.smithi123.stdout: "error_string": "", 2024-09-16T15:47:33.094 INFO:teuthology.orchestra.run.smithi123.stdout: "module_options": { 2024-09-16T15:47:33.094 INFO:teuthology.orchestra.run.smithi123.stdout: "log_level": { 2024-09-16T15:47:33.094 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_level", 2024-09-16T15:47:33.094 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-09-16T15:47:33.094 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.095 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-09-16T15:47:33.095 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "", 2024-09-16T15:47:33.095 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.095 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.095 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [ 2024-09-16T15:47:33.095 INFO:teuthology.orchestra.run.smithi123.stdout: "", 2024-09-16T15:47:33.095 INFO:teuthology.orchestra.run.smithi123.stdout: "critical", 2024-09-16T15:47:33.095 INFO:teuthology.orchestra.run.smithi123.stdout: "debug", 2024-09-16T15:47:33.095 INFO:teuthology.orchestra.run.smithi123.stdout: "error", 2024-09-16T15:47:33.095 INFO:teuthology.orchestra.run.smithi123.stdout: "info", 2024-09-16T15:47:33.095 INFO:teuthology.orchestra.run.smithi123.stdout: "warning" 2024-09-16T15:47:33.095 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-09-16T15:47:33.095 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:33.095 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.096 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.096 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.096 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.096 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_cluster": { 2024-09-16T15:47:33.096 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_cluster", 2024-09-16T15:47:33.096 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-09-16T15:47:33.096 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.096 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-09-16T15:47:33.096 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-09-16T15:47:33.096 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.096 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.096 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.096 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:33.096 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.096 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.097 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.097 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.097 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_cluster_level": { 2024-09-16T15:47:33.097 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_cluster_level", 2024-09-16T15:47:33.097 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-09-16T15:47:33.097 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.097 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-09-16T15:47:33.097 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "info", 2024-09-16T15:47:33.097 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.097 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.097 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [ 2024-09-16T15:47:33.097 INFO:teuthology.orchestra.run.smithi123.stdout: "", 2024-09-16T15:47:33.097 INFO:teuthology.orchestra.run.smithi123.stdout: "critical", 2024-09-16T15:47:33.097 INFO:teuthology.orchestra.run.smithi123.stdout: "debug", 2024-09-16T15:47:33.097 INFO:teuthology.orchestra.run.smithi123.stdout: "error", 2024-09-16T15:47:33.098 INFO:teuthology.orchestra.run.smithi123.stdout: "info", 2024-09-16T15:47:33.098 INFO:teuthology.orchestra.run.smithi123.stdout: "warning" 2024-09-16T15:47:33.098 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-09-16T15:47:33.098 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:33.098 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.098 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.098 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.098 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.098 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_file": { 2024-09-16T15:47:33.098 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_file", 2024-09-16T15:47:33.098 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-09-16T15:47:33.098 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.099 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-09-16T15:47:33.099 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-09-16T15:47:33.099 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.099 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.099 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.099 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:33.099 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.099 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.099 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.099 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.099 INFO:teuthology.orchestra.run.smithi123.stdout: "roption1": { 2024-09-16T15:47:33.099 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "roption1", 2024-09-16T15:47:33.099 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-09-16T15:47:33.099 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.100 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-09-16T15:47:33.100 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "", 2024-09-16T15:47:33.100 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.100 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.100 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.100 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:33.100 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.100 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.100 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.100 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.100 INFO:teuthology.orchestra.run.smithi123.stdout: "roption2": { 2024-09-16T15:47:33.100 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "roption2", 2024-09-16T15:47:33.100 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-09-16T15:47:33.100 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.100 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-09-16T15:47:33.101 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "xyz", 2024-09-16T15:47:33.101 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.101 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.101 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.101 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:33.101 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.101 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.101 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.101 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.101 INFO:teuthology.orchestra.run.smithi123.stdout: "rwoption1": { 2024-09-16T15:47:33.101 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "rwoption1", 2024-09-16T15:47:33.101 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-09-16T15:47:33.101 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.101 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-09-16T15:47:33.101 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "", 2024-09-16T15:47:33.102 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.102 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.102 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.102 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:33.102 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.102 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.102 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.102 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.102 INFO:teuthology.orchestra.run.smithi123.stdout: "rwoption2": { 2024-09-16T15:47:33.102 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "rwoption2", 2024-09-16T15:47:33.102 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "int", 2024-09-16T15:47:33.102 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.102 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-09-16T15:47:33.103 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "", 2024-09-16T15:47:33.103 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.103 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.103 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.103 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:33.103 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.103 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.103 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.103 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.103 INFO:teuthology.orchestra.run.smithi123.stdout: "rwoption3": { 2024-09-16T15:47:33.103 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "rwoption3", 2024-09-16T15:47:33.103 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "float", 2024-09-16T15:47:33.103 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.103 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-09-16T15:47:33.104 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "", 2024-09-16T15:47:33.104 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.104 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.104 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.104 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:33.104 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.104 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.104 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.104 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.104 INFO:teuthology.orchestra.run.smithi123.stdout: "rwoption4": { 2024-09-16T15:47:33.104 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "rwoption4", 2024-09-16T15:47:33.104 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-09-16T15:47:33.104 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.104 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-09-16T15:47:33.105 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "", 2024-09-16T15:47:33.105 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.105 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.105 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.105 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:33.105 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.105 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.105 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.105 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.105 INFO:teuthology.orchestra.run.smithi123.stdout: "rwoption5": { 2024-09-16T15:47:33.105 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "rwoption5", 2024-09-16T15:47:33.105 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-09-16T15:47:33.105 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.105 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-09-16T15:47:33.105 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "", 2024-09-16T15:47:33.105 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.106 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.106 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.106 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:33.106 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.106 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.106 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.106 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.106 INFO:teuthology.orchestra.run.smithi123.stdout: "rwoption6": { 2024-09-16T15:47:33.106 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "rwoption6", 2024-09-16T15:47:33.106 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-09-16T15:47:33.106 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.106 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-09-16T15:47:33.106 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "True", 2024-09-16T15:47:33.106 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.106 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.107 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.107 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:33.107 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.107 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.107 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.107 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.107 INFO:teuthology.orchestra.run.smithi123.stdout: "rwoption7": { 2024-09-16T15:47:33.107 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "rwoption7", 2024-09-16T15:47:33.107 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "int", 2024-09-16T15:47:33.107 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.107 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-09-16T15:47:33.107 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "", 2024-09-16T15:47:33.107 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "1", 2024-09-16T15:47:33.107 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "42", 2024-09-16T15:47:33.107 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.108 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:33.108 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.108 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.108 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.108 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.108 INFO:teuthology.orchestra.run.smithi123.stdout: "sqlite3_killpoint": { 2024-09-16T15:47:33.108 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "sqlite3_killpoint", 2024-09-16T15:47:33.108 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "int", 2024-09-16T15:47:33.108 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "dev", 2024-09-16T15:47:33.108 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-09-16T15:47:33.108 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "0", 2024-09-16T15:47:33.108 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.108 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.108 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.108 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:33.109 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.109 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.109 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.109 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.109 INFO:teuthology.orchestra.run.smithi123.stdout: "testkey": { 2024-09-16T15:47:33.109 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "testkey", 2024-09-16T15:47:33.109 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-09-16T15:47:33.109 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.109 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-09-16T15:47:33.109 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "", 2024-09-16T15:47:33.109 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.109 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.109 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.109 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:33.109 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.109 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.110 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.110 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.110 INFO:teuthology.orchestra.run.smithi123.stdout: "testlkey": { 2024-09-16T15:47:33.110 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "testlkey", 2024-09-16T15:47:33.110 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-09-16T15:47:33.110 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.110 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-09-16T15:47:33.110 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "", 2024-09-16T15:47:33.110 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.110 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.110 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.110 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:33.110 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.110 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.110 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.111 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.111 INFO:teuthology.orchestra.run.smithi123.stdout: "testnewline": { 2024-09-16T15:47:33.111 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "testnewline", 2024-09-16T15:47:33.111 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-09-16T15:47:33.111 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.111 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-09-16T15:47:33.111 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "", 2024-09-16T15:47:33.111 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.111 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.111 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.111 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:33.111 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.111 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.111 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.111 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-16T15:47:33.111 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-16T15:47:33.112 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.112 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-09-16T15:47:33.112 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "snap_schedule", 2024-09-16T15:47:33.112 INFO:teuthology.orchestra.run.smithi123.stdout: "can_run": true, 2024-09-16T15:47:33.112 INFO:teuthology.orchestra.run.smithi123.stdout: "error_string": "", 2024-09-16T15:47:33.112 INFO:teuthology.orchestra.run.smithi123.stdout: "module_options": { 2024-09-16T15:47:33.112 INFO:teuthology.orchestra.run.smithi123.stdout: "allow_m_granularity": { 2024-09-16T15:47:33.112 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "allow_m_granularity", 2024-09-16T15:47:33.112 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-09-16T15:47:33.112 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.112 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-09-16T15:47:33.112 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-09-16T15:47:33.112 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.112 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.113 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.113 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "allow minute scheduled snapshots", 2024-09-16T15:47:33.113 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.113 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.113 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.113 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.113 INFO:teuthology.orchestra.run.smithi123.stdout: "dump_on_update": { 2024-09-16T15:47:33.113 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "dump_on_update", 2024-09-16T15:47:33.113 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-09-16T15:47:33.113 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.113 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-09-16T15:47:33.113 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-09-16T15:47:33.113 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.113 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.114 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.114 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "dump database to debug log on update", 2024-09-16T15:47:33.114 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.114 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.114 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.114 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.114 INFO:teuthology.orchestra.run.smithi123.stdout: "log_level": { 2024-09-16T15:47:33.114 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_level", 2024-09-16T15:47:33.114 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-09-16T15:47:33.114 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.114 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-09-16T15:47:33.114 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "", 2024-09-16T15:47:33.114 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.114 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.114 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [ 2024-09-16T15:47:33.114 INFO:teuthology.orchestra.run.smithi123.stdout: "", 2024-09-16T15:47:33.115 INFO:teuthology.orchestra.run.smithi123.stdout: "critical", 2024-09-16T15:47:33.115 INFO:teuthology.orchestra.run.smithi123.stdout: "debug", 2024-09-16T15:47:33.115 INFO:teuthology.orchestra.run.smithi123.stdout: "error", 2024-09-16T15:47:33.115 INFO:teuthology.orchestra.run.smithi123.stdout: "info", 2024-09-16T15:47:33.115 INFO:teuthology.orchestra.run.smithi123.stdout: "warning" 2024-09-16T15:47:33.115 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-09-16T15:47:33.115 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:33.115 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.115 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.115 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.115 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.115 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_cluster": { 2024-09-16T15:47:33.115 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_cluster", 2024-09-16T15:47:33.115 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-09-16T15:47:33.116 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.116 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-09-16T15:47:33.116 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-09-16T15:47:33.116 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.116 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.116 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.116 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:33.116 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.116 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.116 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.116 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.116 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_cluster_level": { 2024-09-16T15:47:33.116 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_cluster_level", 2024-09-16T15:47:33.116 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-09-16T15:47:33.116 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.117 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-09-16T15:47:33.117 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "info", 2024-09-16T15:47:33.117 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.117 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.117 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [ 2024-09-16T15:47:33.117 INFO:teuthology.orchestra.run.smithi123.stdout: "", 2024-09-16T15:47:33.117 INFO:teuthology.orchestra.run.smithi123.stdout: "critical", 2024-09-16T15:47:33.117 INFO:teuthology.orchestra.run.smithi123.stdout: "debug", 2024-09-16T15:47:33.117 INFO:teuthology.orchestra.run.smithi123.stdout: "error", 2024-09-16T15:47:33.117 INFO:teuthology.orchestra.run.smithi123.stdout: "info", 2024-09-16T15:47:33.117 INFO:teuthology.orchestra.run.smithi123.stdout: "warning" 2024-09-16T15:47:33.117 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-09-16T15:47:33.117 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:33.117 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.118 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.118 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.118 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.118 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_file": { 2024-09-16T15:47:33.118 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_file", 2024-09-16T15:47:33.118 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-09-16T15:47:33.118 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.118 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-09-16T15:47:33.118 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-09-16T15:47:33.118 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.118 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.118 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.118 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:33.118 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.118 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.118 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.119 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.119 INFO:teuthology.orchestra.run.smithi123.stdout: "sqlite3_killpoint": { 2024-09-16T15:47:33.119 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "sqlite3_killpoint", 2024-09-16T15:47:33.119 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "int", 2024-09-16T15:47:33.119 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "dev", 2024-09-16T15:47:33.119 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-09-16T15:47:33.119 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "0", 2024-09-16T15:47:33.119 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.119 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.119 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.119 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:33.119 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.119 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.119 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.119 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-16T15:47:33.119 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-16T15:47:33.120 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.120 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-09-16T15:47:33.120 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "stats", 2024-09-16T15:47:33.120 INFO:teuthology.orchestra.run.smithi123.stdout: "can_run": true, 2024-09-16T15:47:33.120 INFO:teuthology.orchestra.run.smithi123.stdout: "error_string": "", 2024-09-16T15:47:33.120 INFO:teuthology.orchestra.run.smithi123.stdout: "module_options": { 2024-09-16T15:47:33.120 INFO:teuthology.orchestra.run.smithi123.stdout: "log_level": { 2024-09-16T15:47:33.120 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_level", 2024-09-16T15:47:33.120 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-09-16T15:47:33.120 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.120 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-09-16T15:47:33.120 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "", 2024-09-16T15:47:33.120 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.120 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.121 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [ 2024-09-16T15:47:33.121 INFO:teuthology.orchestra.run.smithi123.stdout: "", 2024-09-16T15:47:33.121 INFO:teuthology.orchestra.run.smithi123.stdout: "critical", 2024-09-16T15:47:33.121 INFO:teuthology.orchestra.run.smithi123.stdout: "debug", 2024-09-16T15:47:33.121 INFO:teuthology.orchestra.run.smithi123.stdout: "error", 2024-09-16T15:47:33.121 INFO:teuthology.orchestra.run.smithi123.stdout: "info", 2024-09-16T15:47:33.121 INFO:teuthology.orchestra.run.smithi123.stdout: "warning" 2024-09-16T15:47:33.121 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-09-16T15:47:33.121 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:33.121 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.121 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.121 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.121 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.121 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_cluster": { 2024-09-16T15:47:33.122 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_cluster", 2024-09-16T15:47:33.122 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-09-16T15:47:33.122 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.122 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-09-16T15:47:33.122 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-09-16T15:47:33.122 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.122 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.122 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.122 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:33.122 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.122 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.122 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.122 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.122 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_cluster_level": { 2024-09-16T15:47:33.123 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_cluster_level", 2024-09-16T15:47:33.123 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-09-16T15:47:33.123 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.123 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-09-16T15:47:33.123 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "info", 2024-09-16T15:47:33.123 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.123 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.123 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [ 2024-09-16T15:47:33.123 INFO:teuthology.orchestra.run.smithi123.stdout: "", 2024-09-16T15:47:33.123 INFO:teuthology.orchestra.run.smithi123.stdout: "critical", 2024-09-16T15:47:33.123 INFO:teuthology.orchestra.run.smithi123.stdout: "debug", 2024-09-16T15:47:33.123 INFO:teuthology.orchestra.run.smithi123.stdout: "error", 2024-09-16T15:47:33.123 INFO:teuthology.orchestra.run.smithi123.stdout: "info", 2024-09-16T15:47:33.124 INFO:teuthology.orchestra.run.smithi123.stdout: "warning" 2024-09-16T15:47:33.124 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-09-16T15:47:33.124 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:33.124 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.124 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.124 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.124 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.124 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_file": { 2024-09-16T15:47:33.124 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_file", 2024-09-16T15:47:33.124 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-09-16T15:47:33.124 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.124 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-09-16T15:47:33.124 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-09-16T15:47:33.124 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.124 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.125 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.125 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:33.125 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.125 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.125 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.125 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.125 INFO:teuthology.orchestra.run.smithi123.stdout: "sqlite3_killpoint": { 2024-09-16T15:47:33.125 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "sqlite3_killpoint", 2024-09-16T15:47:33.125 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "int", 2024-09-16T15:47:33.125 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "dev", 2024-09-16T15:47:33.125 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-09-16T15:47:33.125 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "0", 2024-09-16T15:47:33.125 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.125 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.126 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.126 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:33.126 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.126 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.126 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.126 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-16T15:47:33.126 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-16T15:47:33.126 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.126 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-09-16T15:47:33.126 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "status", 2024-09-16T15:47:33.126 INFO:teuthology.orchestra.run.smithi123.stdout: "can_run": true, 2024-09-16T15:47:33.126 INFO:teuthology.orchestra.run.smithi123.stdout: "error_string": "", 2024-09-16T15:47:33.126 INFO:teuthology.orchestra.run.smithi123.stdout: "module_options": { 2024-09-16T15:47:33.127 INFO:teuthology.orchestra.run.smithi123.stdout: "log_level": { 2024-09-16T15:47:33.127 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_level", 2024-09-16T15:47:33.127 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-09-16T15:47:33.127 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.127 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-09-16T15:47:33.127 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "", 2024-09-16T15:47:33.127 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.127 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.127 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [ 2024-09-16T15:47:33.127 INFO:teuthology.orchestra.run.smithi123.stdout: "", 2024-09-16T15:47:33.127 INFO:teuthology.orchestra.run.smithi123.stdout: "critical", 2024-09-16T15:47:33.127 INFO:teuthology.orchestra.run.smithi123.stdout: "debug", 2024-09-16T15:47:33.127 INFO:teuthology.orchestra.run.smithi123.stdout: "error", 2024-09-16T15:47:33.127 INFO:teuthology.orchestra.run.smithi123.stdout: "info", 2024-09-16T15:47:33.128 INFO:teuthology.orchestra.run.smithi123.stdout: "warning" 2024-09-16T15:47:33.128 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-09-16T15:47:33.128 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:33.128 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.128 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.128 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.128 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.128 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_cluster": { 2024-09-16T15:47:33.128 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_cluster", 2024-09-16T15:47:33.128 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-09-16T15:47:33.128 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.128 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-09-16T15:47:33.128 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-09-16T15:47:33.128 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.128 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.129 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.129 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:33.129 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.129 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.129 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.129 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.129 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_cluster_level": { 2024-09-16T15:47:33.129 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_cluster_level", 2024-09-16T15:47:33.129 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-09-16T15:47:33.129 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.129 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-09-16T15:47:33.129 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "info", 2024-09-16T15:47:33.129 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.129 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.129 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [ 2024-09-16T15:47:33.130 INFO:teuthology.orchestra.run.smithi123.stdout: "", 2024-09-16T15:47:33.130 INFO:teuthology.orchestra.run.smithi123.stdout: "critical", 2024-09-16T15:47:33.130 INFO:teuthology.orchestra.run.smithi123.stdout: "debug", 2024-09-16T15:47:33.130 INFO:teuthology.orchestra.run.smithi123.stdout: "error", 2024-09-16T15:47:33.130 INFO:teuthology.orchestra.run.smithi123.stdout: "info", 2024-09-16T15:47:33.130 INFO:teuthology.orchestra.run.smithi123.stdout: "warning" 2024-09-16T15:47:33.130 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-09-16T15:47:33.130 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:33.130 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.130 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.130 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.130 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.130 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_file": { 2024-09-16T15:47:33.131 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_file", 2024-09-16T15:47:33.131 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-09-16T15:47:33.131 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.131 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-09-16T15:47:33.131 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-09-16T15:47:33.131 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.131 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.131 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.131 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:33.131 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.131 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.131 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.131 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.131 INFO:teuthology.orchestra.run.smithi123.stdout: "sqlite3_killpoint": { 2024-09-16T15:47:33.132 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "sqlite3_killpoint", 2024-09-16T15:47:33.132 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "int", 2024-09-16T15:47:33.132 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "dev", 2024-09-16T15:47:33.132 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-09-16T15:47:33.132 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "0", 2024-09-16T15:47:33.132 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.132 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.132 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.132 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:33.132 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.132 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.132 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.132 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-16T15:47:33.132 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-16T15:47:33.133 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.133 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-09-16T15:47:33.133 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "telegraf", 2024-09-16T15:47:33.133 INFO:teuthology.orchestra.run.smithi123.stdout: "can_run": true, 2024-09-16T15:47:33.133 INFO:teuthology.orchestra.run.smithi123.stdout: "error_string": "", 2024-09-16T15:47:33.133 INFO:teuthology.orchestra.run.smithi123.stdout: "module_options": { 2024-09-16T15:47:33.133 INFO:teuthology.orchestra.run.smithi123.stdout: "address": { 2024-09-16T15:47:33.133 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "address", 2024-09-16T15:47:33.133 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-09-16T15:47:33.133 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.133 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-09-16T15:47:33.133 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "unixgram:///tmp/telegraf.sock", 2024-09-16T15:47:33.133 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.133 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.133 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.134 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:33.134 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.134 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.134 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.134 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.134 INFO:teuthology.orchestra.run.smithi123.stdout: "interval": { 2024-09-16T15:47:33.134 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "interval", 2024-09-16T15:47:33.134 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "secs", 2024-09-16T15:47:33.134 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.134 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-09-16T15:47:33.134 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "15", 2024-09-16T15:47:33.134 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.134 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.134 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.135 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:33.135 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.135 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.135 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.135 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.135 INFO:teuthology.orchestra.run.smithi123.stdout: "log_level": { 2024-09-16T15:47:33.135 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_level", 2024-09-16T15:47:33.135 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-09-16T15:47:33.135 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.135 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-09-16T15:47:33.135 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "", 2024-09-16T15:47:33.135 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.135 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.136 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [ 2024-09-16T15:47:33.136 INFO:teuthology.orchestra.run.smithi123.stdout: "", 2024-09-16T15:47:33.136 INFO:teuthology.orchestra.run.smithi123.stdout: "critical", 2024-09-16T15:47:33.136 INFO:teuthology.orchestra.run.smithi123.stdout: "debug", 2024-09-16T15:47:33.136 INFO:teuthology.orchestra.run.smithi123.stdout: "error", 2024-09-16T15:47:33.136 INFO:teuthology.orchestra.run.smithi123.stdout: "info", 2024-09-16T15:47:33.136 INFO:teuthology.orchestra.run.smithi123.stdout: "warning" 2024-09-16T15:47:33.136 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-09-16T15:47:33.136 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:33.136 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.136 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.136 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.136 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.136 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_cluster": { 2024-09-16T15:47:33.137 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_cluster", 2024-09-16T15:47:33.137 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-09-16T15:47:33.137 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.137 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-09-16T15:47:33.137 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-09-16T15:47:33.137 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.137 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.137 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.137 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:33.137 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.137 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.137 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.137 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.137 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_cluster_level": { 2024-09-16T15:47:33.137 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_cluster_level", 2024-09-16T15:47:33.138 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-09-16T15:47:33.138 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.138 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-09-16T15:47:33.138 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "info", 2024-09-16T15:47:33.138 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.138 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.138 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [ 2024-09-16T15:47:33.138 INFO:teuthology.orchestra.run.smithi123.stdout: "", 2024-09-16T15:47:33.138 INFO:teuthology.orchestra.run.smithi123.stdout: "critical", 2024-09-16T15:47:33.138 INFO:teuthology.orchestra.run.smithi123.stdout: "debug", 2024-09-16T15:47:33.138 INFO:teuthology.orchestra.run.smithi123.stdout: "error", 2024-09-16T15:47:33.138 INFO:teuthology.orchestra.run.smithi123.stdout: "info", 2024-09-16T15:47:33.138 INFO:teuthology.orchestra.run.smithi123.stdout: "warning" 2024-09-16T15:47:33.139 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-09-16T15:47:33.139 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:33.139 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.139 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.139 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.139 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.139 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_file": { 2024-09-16T15:47:33.139 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_file", 2024-09-16T15:47:33.139 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-09-16T15:47:33.139 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.139 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-09-16T15:47:33.139 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-09-16T15:47:33.139 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.139 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.139 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.140 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:33.140 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.140 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.140 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.140 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.140 INFO:teuthology.orchestra.run.smithi123.stdout: "sqlite3_killpoint": { 2024-09-16T15:47:33.140 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "sqlite3_killpoint", 2024-09-16T15:47:33.140 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "int", 2024-09-16T15:47:33.140 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "dev", 2024-09-16T15:47:33.140 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-09-16T15:47:33.140 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "0", 2024-09-16T15:47:33.140 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.140 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.140 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.140 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:33.141 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.141 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.141 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.141 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-16T15:47:33.141 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-16T15:47:33.141 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.141 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-09-16T15:47:33.141 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "telemetry", 2024-09-16T15:47:33.141 INFO:teuthology.orchestra.run.smithi123.stdout: "can_run": true, 2024-09-16T15:47:33.141 INFO:teuthology.orchestra.run.smithi123.stdout: "error_string": "", 2024-09-16T15:47:33.141 INFO:teuthology.orchestra.run.smithi123.stdout: "module_options": { 2024-09-16T15:47:33.141 INFO:teuthology.orchestra.run.smithi123.stdout: "channel_basic": { 2024-09-16T15:47:33.141 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "channel_basic", 2024-09-16T15:47:33.141 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-09-16T15:47:33.141 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.142 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-09-16T15:47:33.142 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "True", 2024-09-16T15:47:33.142 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.142 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.142 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.142 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "Share basic cluster information (size, version)", 2024-09-16T15:47:33.142 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.142 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.142 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.142 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.142 INFO:teuthology.orchestra.run.smithi123.stdout: "channel_crash": { 2024-09-16T15:47:33.142 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "channel_crash", 2024-09-16T15:47:33.142 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-09-16T15:47:33.142 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.143 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-09-16T15:47:33.143 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "True", 2024-09-16T15:47:33.143 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.143 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.143 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.143 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "Share metadata about Ceph daemon crashes (version, stack straces, etc)", 2024-09-16T15:47:33.143 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.143 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.143 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.143 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.143 INFO:teuthology.orchestra.run.smithi123.stdout: "channel_device": { 2024-09-16T15:47:33.143 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "channel_device", 2024-09-16T15:47:33.143 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-09-16T15:47:33.143 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.143 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-09-16T15:47:33.144 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "True", 2024-09-16T15:47:33.144 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.144 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.144 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.144 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)", 2024-09-16T15:47:33.144 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.144 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.144 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.144 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.144 INFO:teuthology.orchestra.run.smithi123.stdout: "channel_ident": { 2024-09-16T15:47:33.144 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "channel_ident", 2024-09-16T15:47:33.144 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-09-16T15:47:33.144 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.144 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-09-16T15:47:33.144 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-09-16T15:47:33.145 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.145 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.145 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.145 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "Share a user-provided description and/or contact email for the cluster", 2024-09-16T15:47:33.145 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.145 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.145 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.145 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.145 INFO:teuthology.orchestra.run.smithi123.stdout: "channel_perf": { 2024-09-16T15:47:33.145 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "channel_perf", 2024-09-16T15:47:33.145 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-09-16T15:47:33.145 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.145 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-09-16T15:47:33.145 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-09-16T15:47:33.145 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.145 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.146 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.146 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "Share various performance metrics of a cluster", 2024-09-16T15:47:33.146 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.146 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.146 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.146 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.146 INFO:teuthology.orchestra.run.smithi123.stdout: "contact": { 2024-09-16T15:47:33.146 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "contact", 2024-09-16T15:47:33.146 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-09-16T15:47:33.146 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.146 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-09-16T15:47:33.146 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "", 2024-09-16T15:47:33.146 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.146 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.146 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.147 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:33.147 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.147 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.147 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.147 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.147 INFO:teuthology.orchestra.run.smithi123.stdout: "description": { 2024-09-16T15:47:33.147 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "description", 2024-09-16T15:47:33.147 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-09-16T15:47:33.147 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.147 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-09-16T15:47:33.147 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "", 2024-09-16T15:47:33.147 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.147 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.147 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.147 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:33.148 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.148 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.148 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.148 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.148 INFO:teuthology.orchestra.run.smithi123.stdout: "device_url": { 2024-09-16T15:47:33.148 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "device_url", 2024-09-16T15:47:33.148 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-09-16T15:47:33.148 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.148 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-09-16T15:47:33.148 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "https://telemetry.ceph.com/device", 2024-09-16T15:47:33.148 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.148 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.148 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.149 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:33.149 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.149 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.149 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.149 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.149 INFO:teuthology.orchestra.run.smithi123.stdout: "enabled": { 2024-09-16T15:47:33.149 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "enabled", 2024-09-16T15:47:33.149 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-09-16T15:47:33.149 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.149 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-09-16T15:47:33.149 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-09-16T15:47:33.149 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.149 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.149 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.150 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:33.150 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.150 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.150 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.150 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.150 INFO:teuthology.orchestra.run.smithi123.stdout: "interval": { 2024-09-16T15:47:33.150 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "interval", 2024-09-16T15:47:33.150 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "int", 2024-09-16T15:47:33.150 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.150 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-09-16T15:47:33.150 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "24", 2024-09-16T15:47:33.150 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "8", 2024-09-16T15:47:33.150 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.150 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.151 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:33.151 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.151 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.151 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.151 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.151 INFO:teuthology.orchestra.run.smithi123.stdout: "last_opt_revision": { 2024-09-16T15:47:33.151 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "last_opt_revision", 2024-09-16T15:47:33.151 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "int", 2024-09-16T15:47:33.151 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.151 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-09-16T15:47:33.151 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "1", 2024-09-16T15:47:33.151 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.151 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.151 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.151 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:33.151 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.152 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.152 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.152 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.152 INFO:teuthology.orchestra.run.smithi123.stdout: "leaderboard": { 2024-09-16T15:47:33.152 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "leaderboard", 2024-09-16T15:47:33.152 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-09-16T15:47:33.152 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.152 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-09-16T15:47:33.152 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-09-16T15:47:33.152 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.152 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.152 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.152 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:33.152 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.153 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.153 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.153 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.153 INFO:teuthology.orchestra.run.smithi123.stdout: "leaderboard_description": { 2024-09-16T15:47:33.153 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "leaderboard_description", 2024-09-16T15:47:33.153 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-09-16T15:47:33.153 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.153 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-09-16T15:47:33.153 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "", 2024-09-16T15:47:33.153 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.153 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.153 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.153 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:33.154 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.154 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.154 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.154 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.154 INFO:teuthology.orchestra.run.smithi123.stdout: "log_level": { 2024-09-16T15:47:33.154 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_level", 2024-09-16T15:47:33.154 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-09-16T15:47:33.154 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.154 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-09-16T15:47:33.154 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "", 2024-09-16T15:47:33.154 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.154 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.154 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [ 2024-09-16T15:47:33.154 INFO:teuthology.orchestra.run.smithi123.stdout: "", 2024-09-16T15:47:33.154 INFO:teuthology.orchestra.run.smithi123.stdout: "critical", 2024-09-16T15:47:33.155 INFO:teuthology.orchestra.run.smithi123.stdout: "debug", 2024-09-16T15:47:33.155 INFO:teuthology.orchestra.run.smithi123.stdout: "error", 2024-09-16T15:47:33.155 INFO:teuthology.orchestra.run.smithi123.stdout: "info", 2024-09-16T15:47:33.155 INFO:teuthology.orchestra.run.smithi123.stdout: "warning" 2024-09-16T15:47:33.155 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-09-16T15:47:33.155 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:33.155 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.155 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.155 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.155 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.155 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_cluster": { 2024-09-16T15:47:33.155 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_cluster", 2024-09-16T15:47:33.155 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-09-16T15:47:33.155 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.156 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-09-16T15:47:33.156 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-09-16T15:47:33.156 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.156 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.156 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.156 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:33.156 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.156 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.156 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.156 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.156 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_cluster_level": { 2024-09-16T15:47:33.156 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_cluster_level", 2024-09-16T15:47:33.156 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-09-16T15:47:33.156 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.156 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-09-16T15:47:33.157 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "info", 2024-09-16T15:47:33.157 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.157 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.157 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [ 2024-09-16T15:47:33.157 INFO:teuthology.orchestra.run.smithi123.stdout: "", 2024-09-16T15:47:33.157 INFO:teuthology.orchestra.run.smithi123.stdout: "critical", 2024-09-16T15:47:33.157 INFO:teuthology.orchestra.run.smithi123.stdout: "debug", 2024-09-16T15:47:33.157 INFO:teuthology.orchestra.run.smithi123.stdout: "error", 2024-09-16T15:47:33.157 INFO:teuthology.orchestra.run.smithi123.stdout: "info", 2024-09-16T15:47:33.157 INFO:teuthology.orchestra.run.smithi123.stdout: "warning" 2024-09-16T15:47:33.157 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-09-16T15:47:33.157 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:33.157 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.157 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.158 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.158 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.158 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_file": { 2024-09-16T15:47:33.158 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_file", 2024-09-16T15:47:33.158 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-09-16T15:47:33.158 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.158 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-09-16T15:47:33.158 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-09-16T15:47:33.158 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.158 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.158 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.158 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:33.158 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.158 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.159 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.159 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.159 INFO:teuthology.orchestra.run.smithi123.stdout: "organization": { 2024-09-16T15:47:33.159 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "organization", 2024-09-16T15:47:33.159 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-09-16T15:47:33.159 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.159 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-09-16T15:47:33.159 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "", 2024-09-16T15:47:33.159 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.159 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.159 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.159 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:33.159 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.159 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.159 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.160 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.160 INFO:teuthology.orchestra.run.smithi123.stdout: "proxy": { 2024-09-16T15:47:33.160 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "proxy", 2024-09-16T15:47:33.160 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-09-16T15:47:33.160 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.160 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-09-16T15:47:33.160 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "", 2024-09-16T15:47:33.160 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.160 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.160 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.160 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:33.160 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.160 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.160 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.160 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.161 INFO:teuthology.orchestra.run.smithi123.stdout: "sqlite3_killpoint": { 2024-09-16T15:47:33.161 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "sqlite3_killpoint", 2024-09-16T15:47:33.161 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "int", 2024-09-16T15:47:33.161 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "dev", 2024-09-16T15:47:33.161 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-09-16T15:47:33.161 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "0", 2024-09-16T15:47:33.161 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.161 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.161 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.161 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:33.161 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.161 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.161 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.161 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.161 INFO:teuthology.orchestra.run.smithi123.stdout: "url": { 2024-09-16T15:47:33.162 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "url", 2024-09-16T15:47:33.162 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-09-16T15:47:33.162 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.162 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-09-16T15:47:33.162 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "https://telemetry.ceph.com/report", 2024-09-16T15:47:33.162 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.162 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.162 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.162 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:33.162 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.162 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.162 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.162 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-16T15:47:33.162 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-16T15:47:33.162 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.162 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-09-16T15:47:33.163 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "test_orchestrator", 2024-09-16T15:47:33.163 INFO:teuthology.orchestra.run.smithi123.stdout: "can_run": true, 2024-09-16T15:47:33.163 INFO:teuthology.orchestra.run.smithi123.stdout: "error_string": "", 2024-09-16T15:47:33.163 INFO:teuthology.orchestra.run.smithi123.stdout: "module_options": { 2024-09-16T15:47:33.163 INFO:teuthology.orchestra.run.smithi123.stdout: "log_level": { 2024-09-16T15:47:33.163 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_level", 2024-09-16T15:47:33.163 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-09-16T15:47:33.163 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.163 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-09-16T15:47:33.163 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "", 2024-09-16T15:47:33.163 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.163 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.163 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [ 2024-09-16T15:47:33.163 INFO:teuthology.orchestra.run.smithi123.stdout: "", 2024-09-16T15:47:33.163 INFO:teuthology.orchestra.run.smithi123.stdout: "critical", 2024-09-16T15:47:33.163 INFO:teuthology.orchestra.run.smithi123.stdout: "debug", 2024-09-16T15:47:33.164 INFO:teuthology.orchestra.run.smithi123.stdout: "error", 2024-09-16T15:47:33.164 INFO:teuthology.orchestra.run.smithi123.stdout: "info", 2024-09-16T15:47:33.164 INFO:teuthology.orchestra.run.smithi123.stdout: "warning" 2024-09-16T15:47:33.164 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-09-16T15:47:33.164 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:33.164 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.164 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.164 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.164 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.164 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_cluster": { 2024-09-16T15:47:33.164 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_cluster", 2024-09-16T15:47:33.164 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-09-16T15:47:33.164 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.164 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-09-16T15:47:33.165 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-09-16T15:47:33.165 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.165 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.165 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.165 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:33.165 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.165 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.165 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.165 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.165 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_cluster_level": { 2024-09-16T15:47:33.165 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_cluster_level", 2024-09-16T15:47:33.165 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-09-16T15:47:33.165 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.165 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-09-16T15:47:33.165 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "info", 2024-09-16T15:47:33.166 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.166 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.166 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [ 2024-09-16T15:47:33.166 INFO:teuthology.orchestra.run.smithi123.stdout: "", 2024-09-16T15:47:33.166 INFO:teuthology.orchestra.run.smithi123.stdout: "critical", 2024-09-16T15:47:33.166 INFO:teuthology.orchestra.run.smithi123.stdout: "debug", 2024-09-16T15:47:33.166 INFO:teuthology.orchestra.run.smithi123.stdout: "error", 2024-09-16T15:47:33.166 INFO:teuthology.orchestra.run.smithi123.stdout: "info", 2024-09-16T15:47:33.166 INFO:teuthology.orchestra.run.smithi123.stdout: "warning" 2024-09-16T15:47:33.166 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-09-16T15:47:33.166 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:33.166 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.166 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.166 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.166 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.167 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_file": { 2024-09-16T15:47:33.167 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_file", 2024-09-16T15:47:33.167 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-09-16T15:47:33.167 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.167 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-09-16T15:47:33.167 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-09-16T15:47:33.167 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.167 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.167 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.167 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:33.167 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.167 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.167 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.167 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.168 INFO:teuthology.orchestra.run.smithi123.stdout: "sqlite3_killpoint": { 2024-09-16T15:47:33.168 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "sqlite3_killpoint", 2024-09-16T15:47:33.168 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "int", 2024-09-16T15:47:33.168 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "dev", 2024-09-16T15:47:33.168 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-09-16T15:47:33.168 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "0", 2024-09-16T15:47:33.168 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.168 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.168 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.168 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:33.168 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.168 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.168 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.169 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-16T15:47:33.169 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-16T15:47:33.169 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.169 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-09-16T15:47:33.169 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "volumes", 2024-09-16T15:47:33.169 INFO:teuthology.orchestra.run.smithi123.stdout: "can_run": true, 2024-09-16T15:47:33.169 INFO:teuthology.orchestra.run.smithi123.stdout: "error_string": "", 2024-09-16T15:47:33.169 INFO:teuthology.orchestra.run.smithi123.stdout: "module_options": { 2024-09-16T15:47:33.169 INFO:teuthology.orchestra.run.smithi123.stdout: "log_level": { 2024-09-16T15:47:33.169 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_level", 2024-09-16T15:47:33.169 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-09-16T15:47:33.169 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.170 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-09-16T15:47:33.170 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "", 2024-09-16T15:47:33.170 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.170 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.170 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [ 2024-09-16T15:47:33.170 INFO:teuthology.orchestra.run.smithi123.stdout: "", 2024-09-16T15:47:33.170 INFO:teuthology.orchestra.run.smithi123.stdout: "critical", 2024-09-16T15:47:33.170 INFO:teuthology.orchestra.run.smithi123.stdout: "debug", 2024-09-16T15:47:33.170 INFO:teuthology.orchestra.run.smithi123.stdout: "error", 2024-09-16T15:47:33.170 INFO:teuthology.orchestra.run.smithi123.stdout: "info", 2024-09-16T15:47:33.170 INFO:teuthology.orchestra.run.smithi123.stdout: "warning" 2024-09-16T15:47:33.170 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-09-16T15:47:33.170 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:33.171 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.171 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.171 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.171 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.171 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_cluster": { 2024-09-16T15:47:33.171 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_cluster", 2024-09-16T15:47:33.171 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-09-16T15:47:33.171 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.171 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-09-16T15:47:33.171 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-09-16T15:47:33.171 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.171 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.171 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.172 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:33.172 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.172 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.172 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.172 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.172 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_cluster_level": { 2024-09-16T15:47:33.172 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_cluster_level", 2024-09-16T15:47:33.172 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-09-16T15:47:33.172 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.172 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-09-16T15:47:33.172 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "info", 2024-09-16T15:47:33.173 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.173 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.173 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [ 2024-09-16T15:47:33.173 INFO:teuthology.orchestra.run.smithi123.stdout: "", 2024-09-16T15:47:33.173 INFO:teuthology.orchestra.run.smithi123.stdout: "critical", 2024-09-16T15:47:33.173 INFO:teuthology.orchestra.run.smithi123.stdout: "debug", 2024-09-16T15:47:33.173 INFO:teuthology.orchestra.run.smithi123.stdout: "error", 2024-09-16T15:47:33.173 INFO:teuthology.orchestra.run.smithi123.stdout: "info", 2024-09-16T15:47:33.173 INFO:teuthology.orchestra.run.smithi123.stdout: "warning" 2024-09-16T15:47:33.173 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-09-16T15:47:33.173 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:33.174 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.174 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.174 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.174 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.174 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_file": { 2024-09-16T15:47:33.174 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_file", 2024-09-16T15:47:33.174 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-09-16T15:47:33.174 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.174 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-09-16T15:47:33.174 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-09-16T15:47:33.174 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.174 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.174 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.175 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:33.175 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.175 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.175 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.175 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.175 INFO:teuthology.orchestra.run.smithi123.stdout: "max_concurrent_clones": { 2024-09-16T15:47:33.175 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "max_concurrent_clones", 2024-09-16T15:47:33.175 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "int", 2024-09-16T15:47:33.175 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.176 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-09-16T15:47:33.176 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "4", 2024-09-16T15:47:33.176 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.176 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.176 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.176 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "Number of asynchronous cloner threads", 2024-09-16T15:47:33.176 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.176 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.176 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.176 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.176 INFO:teuthology.orchestra.run.smithi123.stdout: "periodic_async_work": { 2024-09-16T15:47:33.176 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "periodic_async_work", 2024-09-16T15:47:33.176 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-09-16T15:47:33.177 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.177 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-09-16T15:47:33.177 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-09-16T15:47:33.177 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.177 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.177 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.177 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "Periodically check for async work", 2024-09-16T15:47:33.177 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.177 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.177 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.177 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.177 INFO:teuthology.orchestra.run.smithi123.stdout: "snapshot_clone_delay": { 2024-09-16T15:47:33.177 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "snapshot_clone_delay", 2024-09-16T15:47:33.178 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "int", 2024-09-16T15:47:33.178 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.178 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-09-16T15:47:33.178 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "0", 2024-09-16T15:47:33.178 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.178 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.178 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.178 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "Delay clone begin operation by snapshot_clone_delay seconds", 2024-09-16T15:47:33.178 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.178 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.178 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.178 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.178 INFO:teuthology.orchestra.run.smithi123.stdout: "snapshot_clone_no_wait": { 2024-09-16T15:47:33.178 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "snapshot_clone_no_wait", 2024-09-16T15:47:33.178 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-09-16T15:47:33.179 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.179 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-09-16T15:47:33.179 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "True", 2024-09-16T15:47:33.179 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.179 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.179 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.179 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "Reject subvolume clone request when cloner threads are busy", 2024-09-16T15:47:33.179 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.179 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.179 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.179 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.179 INFO:teuthology.orchestra.run.smithi123.stdout: "sqlite3_killpoint": { 2024-09-16T15:47:33.179 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "sqlite3_killpoint", 2024-09-16T15:47:33.179 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "int", 2024-09-16T15:47:33.179 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "dev", 2024-09-16T15:47:33.180 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-09-16T15:47:33.180 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "0", 2024-09-16T15:47:33.180 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.180 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.180 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.180 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:33.180 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.180 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.180 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.180 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-16T15:47:33.180 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-16T15:47:33.180 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.180 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-09-16T15:47:33.180 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "zabbix", 2024-09-16T15:47:33.180 INFO:teuthology.orchestra.run.smithi123.stdout: "can_run": true, 2024-09-16T15:47:33.181 INFO:teuthology.orchestra.run.smithi123.stdout: "error_string": "", 2024-09-16T15:47:33.181 INFO:teuthology.orchestra.run.smithi123.stdout: "module_options": { 2024-09-16T15:47:33.181 INFO:teuthology.orchestra.run.smithi123.stdout: "discovery_interval": { 2024-09-16T15:47:33.181 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "discovery_interval", 2024-09-16T15:47:33.181 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "uint", 2024-09-16T15:47:33.181 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.181 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-09-16T15:47:33.181 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "100", 2024-09-16T15:47:33.181 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.181 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.181 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.181 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:33.181 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.181 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.181 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.182 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.182 INFO:teuthology.orchestra.run.smithi123.stdout: "identifier": { 2024-09-16T15:47:33.182 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "identifier", 2024-09-16T15:47:33.182 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-09-16T15:47:33.182 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.182 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-09-16T15:47:33.182 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "", 2024-09-16T15:47:33.182 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.182 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.182 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.182 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:33.182 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.182 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.182 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.183 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.183 INFO:teuthology.orchestra.run.smithi123.stdout: "interval": { 2024-09-16T15:47:33.183 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "interval", 2024-09-16T15:47:33.183 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "secs", 2024-09-16T15:47:33.183 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.183 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-09-16T15:47:33.183 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "60", 2024-09-16T15:47:33.183 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.183 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.183 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.183 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:33.183 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.183 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.184 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.184 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.184 INFO:teuthology.orchestra.run.smithi123.stdout: "log_level": { 2024-09-16T15:47:33.184 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_level", 2024-09-16T15:47:33.184 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-09-16T15:47:33.184 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.184 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-09-16T15:47:33.184 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "", 2024-09-16T15:47:33.184 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.184 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.184 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [ 2024-09-16T15:47:33.184 INFO:teuthology.orchestra.run.smithi123.stdout: "", 2024-09-16T15:47:33.185 INFO:teuthology.orchestra.run.smithi123.stdout: "critical", 2024-09-16T15:47:33.185 INFO:teuthology.orchestra.run.smithi123.stdout: "debug", 2024-09-16T15:47:33.185 INFO:teuthology.orchestra.run.smithi123.stdout: "error", 2024-09-16T15:47:33.185 INFO:teuthology.orchestra.run.smithi123.stdout: "info", 2024-09-16T15:47:33.185 INFO:teuthology.orchestra.run.smithi123.stdout: "warning" 2024-09-16T15:47:33.185 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-09-16T15:47:33.185 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:33.185 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.185 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.185 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.185 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.185 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_cluster": { 2024-09-16T15:47:33.185 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_cluster", 2024-09-16T15:47:33.185 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-09-16T15:47:33.186 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.186 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-09-16T15:47:33.186 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-09-16T15:47:33.186 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.186 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.186 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.186 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:33.186 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.186 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.186 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.186 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.186 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_cluster_level": { 2024-09-16T15:47:33.186 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_cluster_level", 2024-09-16T15:47:33.186 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-09-16T15:47:33.186 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.187 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-09-16T15:47:33.187 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "info", 2024-09-16T15:47:33.187 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.187 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.187 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [ 2024-09-16T15:47:33.187 INFO:teuthology.orchestra.run.smithi123.stdout: "", 2024-09-16T15:47:33.187 INFO:teuthology.orchestra.run.smithi123.stdout: "critical", 2024-09-16T15:47:33.187 INFO:teuthology.orchestra.run.smithi123.stdout: "debug", 2024-09-16T15:47:33.187 INFO:teuthology.orchestra.run.smithi123.stdout: "error", 2024-09-16T15:47:33.187 INFO:teuthology.orchestra.run.smithi123.stdout: "info", 2024-09-16T15:47:33.187 INFO:teuthology.orchestra.run.smithi123.stdout: "warning" 2024-09-16T15:47:33.187 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-09-16T15:47:33.187 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:33.187 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.187 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.188 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.188 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.188 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_file": { 2024-09-16T15:47:33.188 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_file", 2024-09-16T15:47:33.188 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-09-16T15:47:33.188 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.188 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-09-16T15:47:33.188 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-09-16T15:47:33.188 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.188 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.188 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.188 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:33.188 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.188 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.189 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.189 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.189 INFO:teuthology.orchestra.run.smithi123.stdout: "sqlite3_killpoint": { 2024-09-16T15:47:33.189 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "sqlite3_killpoint", 2024-09-16T15:47:33.189 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "int", 2024-09-16T15:47:33.189 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "dev", 2024-09-16T15:47:33.189 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-09-16T15:47:33.189 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "0", 2024-09-16T15:47:33.189 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.189 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.189 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.189 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:33.189 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.189 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.190 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.190 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.190 INFO:teuthology.orchestra.run.smithi123.stdout: "zabbix_host": { 2024-09-16T15:47:33.190 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "zabbix_host", 2024-09-16T15:47:33.190 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-09-16T15:47:33.190 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.190 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-09-16T15:47:33.190 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "", 2024-09-16T15:47:33.190 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.190 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.190 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.190 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:33.190 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.190 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.190 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.191 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.191 INFO:teuthology.orchestra.run.smithi123.stdout: "zabbix_port": { 2024-09-16T15:47:33.191 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "zabbix_port", 2024-09-16T15:47:33.191 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "int", 2024-09-16T15:47:33.191 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.191 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-09-16T15:47:33.191 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "10051", 2024-09-16T15:47:33.191 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.191 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.191 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.191 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:33.191 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.191 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.191 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.191 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.192 INFO:teuthology.orchestra.run.smithi123.stdout: "zabbix_sender": { 2024-09-16T15:47:33.192 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "zabbix_sender", 2024-09-16T15:47:33.192 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-09-16T15:47:33.192 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.192 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-09-16T15:47:33.192 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "/usr/bin/zabbix_sender", 2024-09-16T15:47:33.192 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.192 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.192 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.192 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:33.192 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.192 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.192 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.192 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-16T15:47:33.192 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-16T15:47:33.193 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-16T15:47:33.193 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-09-16T15:47:33.193 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-16T15:47:33.193 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-09-16T15:47:33.193 INFO:teuthology.orchestra.run.smithi123.stdout: "modules": [ 2024-09-16T15:47:33.193 INFO:teuthology.orchestra.run.smithi123.stdout: "iostat", 2024-09-16T15:47:33.193 INFO:teuthology.orchestra.run.smithi123.stdout: "nfs", 2024-09-16T15:47:33.193 INFO:teuthology.orchestra.run.smithi123.stdout: "restful" 2024-09-16T15:47:33.193 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-09-16T15:47:33.193 INFO:teuthology.orchestra.run.smithi123.stdout: "available_modules": [ 2024-09-16T15:47:33.193 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-09-16T15:47:33.193 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "alerts", 2024-09-16T15:47:33.193 INFO:teuthology.orchestra.run.smithi123.stdout: "can_run": true, 2024-09-16T15:47:33.193 INFO:teuthology.orchestra.run.smithi123.stdout: "error_string": "", 2024-09-16T15:47:33.194 INFO:teuthology.orchestra.run.smithi123.stdout: "module_options": { 2024-09-16T15:47:33.194 INFO:teuthology.orchestra.run.smithi123.stdout: "interval": { 2024-09-16T15:47:33.194 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "interval", 2024-09-16T15:47:33.194 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "secs", 2024-09-16T15:47:33.194 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.194 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-09-16T15:47:33.194 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "60", 2024-09-16T15:47:33.194 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.194 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.194 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.194 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "How frequently to reexamine health status", 2024-09-16T15:47:33.194 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.194 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.194 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.194 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.194 INFO:teuthology.orchestra.run.smithi123.stdout: "log_level": { 2024-09-16T15:47:33.195 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_level", 2024-09-16T15:47:33.195 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-09-16T15:47:33.195 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.195 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-09-16T15:47:33.195 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "", 2024-09-16T15:47:33.195 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.195 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.195 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [ 2024-09-16T15:47:33.195 INFO:teuthology.orchestra.run.smithi123.stdout: "", 2024-09-16T15:47:33.195 INFO:teuthology.orchestra.run.smithi123.stdout: "critical", 2024-09-16T15:47:33.195 INFO:teuthology.orchestra.run.smithi123.stdout: "debug", 2024-09-16T15:47:33.195 INFO:teuthology.orchestra.run.smithi123.stdout: "error", 2024-09-16T15:47:33.195 INFO:teuthology.orchestra.run.smithi123.stdout: "info", 2024-09-16T15:47:33.195 INFO:teuthology.orchestra.run.smithi123.stdout: "warning" 2024-09-16T15:47:33.196 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-09-16T15:47:33.196 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:33.196 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.196 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.196 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.196 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.196 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_cluster": { 2024-09-16T15:47:33.196 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_cluster", 2024-09-16T15:47:33.196 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-09-16T15:47:33.196 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.196 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-09-16T15:47:33.196 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-09-16T15:47:33.196 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.196 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.196 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.197 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:33.197 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.197 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.197 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.197 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.197 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_cluster_level": { 2024-09-16T15:47:33.197 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_cluster_level", 2024-09-16T15:47:33.197 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-09-16T15:47:33.197 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.197 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-09-16T15:47:33.197 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "info", 2024-09-16T15:47:33.197 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.197 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.197 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [ 2024-09-16T15:47:33.198 INFO:teuthology.orchestra.run.smithi123.stdout: "", 2024-09-16T15:47:33.198 INFO:teuthology.orchestra.run.smithi123.stdout: "critical", 2024-09-16T15:47:33.198 INFO:teuthology.orchestra.run.smithi123.stdout: "debug", 2024-09-16T15:47:33.198 INFO:teuthology.orchestra.run.smithi123.stdout: "error", 2024-09-16T15:47:33.198 INFO:teuthology.orchestra.run.smithi123.stdout: "info", 2024-09-16T15:47:33.198 INFO:teuthology.orchestra.run.smithi123.stdout: "warning" 2024-09-16T15:47:33.198 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-09-16T15:47:33.198 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:33.198 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.198 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.198 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.198 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.198 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_file": { 2024-09-16T15:47:33.198 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_file", 2024-09-16T15:47:33.198 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-09-16T15:47:33.199 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.199 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-09-16T15:47:33.199 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-09-16T15:47:33.199 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.199 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.199 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.199 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:33.199 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.199 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.199 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.199 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.199 INFO:teuthology.orchestra.run.smithi123.stdout: "smtp_destination": { 2024-09-16T15:47:33.199 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "smtp_destination", 2024-09-16T15:47:33.199 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-09-16T15:47:33.199 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.200 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-09-16T15:47:33.200 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "", 2024-09-16T15:47:33.200 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.200 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.200 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.200 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "Email address to send alerts to, use commas to separate multiple", 2024-09-16T15:47:33.200 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.200 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.200 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.200 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.200 INFO:teuthology.orchestra.run.smithi123.stdout: "smtp_from_name": { 2024-09-16T15:47:33.200 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "smtp_from_name", 2024-09-16T15:47:33.200 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-09-16T15:47:33.200 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.200 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-09-16T15:47:33.201 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "Ceph", 2024-09-16T15:47:33.201 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.201 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.201 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.201 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "Email From: name", 2024-09-16T15:47:33.201 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.201 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.201 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.201 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.201 INFO:teuthology.orchestra.run.smithi123.stdout: "smtp_host": { 2024-09-16T15:47:33.201 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "smtp_host", 2024-09-16T15:47:33.201 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-09-16T15:47:33.201 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.201 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-09-16T15:47:33.201 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "", 2024-09-16T15:47:33.202 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.202 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.202 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.202 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "SMTP server", 2024-09-16T15:47:33.202 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.202 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.202 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.202 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.202 INFO:teuthology.orchestra.run.smithi123.stdout: "smtp_password": { 2024-09-16T15:47:33.202 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "smtp_password", 2024-09-16T15:47:33.202 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-09-16T15:47:33.202 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.202 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-09-16T15:47:33.202 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "", 2024-09-16T15:47:33.202 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.202 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.203 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.203 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "Password to authenticate with", 2024-09-16T15:47:33.203 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.203 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.203 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.203 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.203 INFO:teuthology.orchestra.run.smithi123.stdout: "smtp_port": { 2024-09-16T15:47:33.203 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "smtp_port", 2024-09-16T15:47:33.203 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "int", 2024-09-16T15:47:33.203 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.203 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-09-16T15:47:33.203 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "465", 2024-09-16T15:47:33.203 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.203 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.203 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.204 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "SMTP port", 2024-09-16T15:47:33.204 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.204 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.204 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.204 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.204 INFO:teuthology.orchestra.run.smithi123.stdout: "smtp_sender": { 2024-09-16T15:47:33.204 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "smtp_sender", 2024-09-16T15:47:33.204 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-09-16T15:47:33.204 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.204 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-09-16T15:47:33.204 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "", 2024-09-16T15:47:33.204 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.204 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.204 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.204 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "SMTP envelope sender", 2024-09-16T15:47:33.205 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.205 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.205 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.205 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.205 INFO:teuthology.orchestra.run.smithi123.stdout: "smtp_ssl": { 2024-09-16T15:47:33.205 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "smtp_ssl", 2024-09-16T15:47:33.205 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-09-16T15:47:33.205 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.205 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-09-16T15:47:33.205 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "True", 2024-09-16T15:47:33.205 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.205 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.205 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.205 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "Use SSL to connect to SMTP server", 2024-09-16T15:47:33.206 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.206 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.206 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.206 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.206 INFO:teuthology.orchestra.run.smithi123.stdout: "smtp_user": { 2024-09-16T15:47:33.206 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "smtp_user", 2024-09-16T15:47:33.206 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-09-16T15:47:33.206 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.206 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-09-16T15:47:33.206 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "", 2024-09-16T15:47:33.206 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.206 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.206 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.206 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "User to authenticate as", 2024-09-16T15:47:33.206 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.206 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.207 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.207 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.207 INFO:teuthology.orchestra.run.smithi123.stdout: "sqlite3_killpoint": { 2024-09-16T15:47:33.207 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "sqlite3_killpoint", 2024-09-16T15:47:33.207 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "int", 2024-09-16T15:47:33.207 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "dev", 2024-09-16T15:47:33.207 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-09-16T15:47:33.207 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "0", 2024-09-16T15:47:33.207 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.207 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.207 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.207 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:33.207 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.207 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.207 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.208 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-16T15:47:33.208 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-16T15:47:33.208 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.208 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-09-16T15:47:33.208 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "balancer", 2024-09-16T15:47:33.208 INFO:teuthology.orchestra.run.smithi123.stdout: "can_run": true, 2024-09-16T15:47:33.208 INFO:teuthology.orchestra.run.smithi123.stdout: "error_string": "", 2024-09-16T15:47:33.208 INFO:teuthology.orchestra.run.smithi123.stdout: "module_options": { 2024-09-16T15:47:33.208 INFO:teuthology.orchestra.run.smithi123.stdout: "active": { 2024-09-16T15:47:33.208 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "active", 2024-09-16T15:47:33.208 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-09-16T15:47:33.208 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.208 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-09-16T15:47:33.208 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "True", 2024-09-16T15:47:33.209 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.209 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.209 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.209 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "automatically balance PGs across cluster", 2024-09-16T15:47:33.209 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.209 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.209 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.209 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.209 INFO:teuthology.orchestra.run.smithi123.stdout: "begin_time": { 2024-09-16T15:47:33.209 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "begin_time", 2024-09-16T15:47:33.209 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-09-16T15:47:33.209 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.209 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-09-16T15:47:33.209 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "0000", 2024-09-16T15:47:33.210 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.210 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.210 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.210 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "beginning time of day to automatically balance", 2024-09-16T15:47:33.210 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-09-16T15:47:33.210 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.210 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.210 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.210 INFO:teuthology.orchestra.run.smithi123.stdout: "begin_weekday": { 2024-09-16T15:47:33.210 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "begin_weekday", 2024-09-16T15:47:33.210 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "uint", 2024-09-16T15:47:33.210 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.210 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-09-16T15:47:33.210 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "0", 2024-09-16T15:47:33.210 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "0", 2024-09-16T15:47:33.211 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "6", 2024-09-16T15:47:33.211 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.211 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "Restrict automatic balancing to this day of the week or later", 2024-09-16T15:47:33.211 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-09-16T15:47:33.211 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.211 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.211 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.211 INFO:teuthology.orchestra.run.smithi123.stdout: "crush_compat_max_iterations": { 2024-09-16T15:47:33.211 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "crush_compat_max_iterations", 2024-09-16T15:47:33.211 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "uint", 2024-09-16T15:47:33.211 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.211 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-09-16T15:47:33.211 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "25", 2024-09-16T15:47:33.211 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "1", 2024-09-16T15:47:33.211 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "250", 2024-09-16T15:47:33.212 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.212 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "maximum number of iterations to attempt optimization", 2024-09-16T15:47:33.212 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.212 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.212 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.212 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.212 INFO:teuthology.orchestra.run.smithi123.stdout: "crush_compat_metrics": { 2024-09-16T15:47:33.212 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "crush_compat_metrics", 2024-09-16T15:47:33.212 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-09-16T15:47:33.212 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.212 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-09-16T15:47:33.212 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "pgs,objects,bytes", 2024-09-16T15:47:33.212 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.212 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.212 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.213 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "metrics with which to calculate OSD utilization", 2024-09-16T15:47:33.213 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.", 2024-09-16T15:47:33.213 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.213 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.213 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.213 INFO:teuthology.orchestra.run.smithi123.stdout: "crush_compat_step": { 2024-09-16T15:47:33.213 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "crush_compat_step", 2024-09-16T15:47:33.213 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "float", 2024-09-16T15:47:33.213 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.213 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-09-16T15:47:33.213 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "0.5", 2024-09-16T15:47:33.213 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "0.001", 2024-09-16T15:47:33.213 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "0.999", 2024-09-16T15:47:33.214 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.214 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "aggressiveness of optimization", 2024-09-16T15:47:33.214 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": ".99 is very aggressive, .01 is less aggressive", 2024-09-16T15:47:33.214 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.214 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.214 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.214 INFO:teuthology.orchestra.run.smithi123.stdout: "end_time": { 2024-09-16T15:47:33.214 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "end_time", 2024-09-16T15:47:33.214 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-09-16T15:47:33.214 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.214 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-09-16T15:47:33.214 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "2359", 2024-09-16T15:47:33.214 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.214 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.214 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.215 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "ending time of day to automatically balance", 2024-09-16T15:47:33.215 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-09-16T15:47:33.215 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.215 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.215 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.215 INFO:teuthology.orchestra.run.smithi123.stdout: "end_weekday": { 2024-09-16T15:47:33.215 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "end_weekday", 2024-09-16T15:47:33.215 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "uint", 2024-09-16T15:47:33.215 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.215 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-09-16T15:47:33.215 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "0", 2024-09-16T15:47:33.215 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "0", 2024-09-16T15:47:33.215 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "6", 2024-09-16T15:47:33.215 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.216 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "Restrict automatic balancing to days of the week earlier than this", 2024-09-16T15:47:33.216 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-09-16T15:47:33.216 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.216 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.216 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.216 INFO:teuthology.orchestra.run.smithi123.stdout: "log_level": { 2024-09-16T15:47:33.216 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_level", 2024-09-16T15:47:33.216 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-09-16T15:47:33.216 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.216 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-09-16T15:47:33.216 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "", 2024-09-16T15:47:33.216 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.216 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.216 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [ 2024-09-16T15:47:33.217 INFO:teuthology.orchestra.run.smithi123.stdout: "", 2024-09-16T15:47:33.217 INFO:teuthology.orchestra.run.smithi123.stdout: "critical", 2024-09-16T15:47:33.217 INFO:teuthology.orchestra.run.smithi123.stdout: "debug", 2024-09-16T15:47:33.217 INFO:teuthology.orchestra.run.smithi123.stdout: "error", 2024-09-16T15:47:33.217 INFO:teuthology.orchestra.run.smithi123.stdout: "info", 2024-09-16T15:47:33.217 INFO:teuthology.orchestra.run.smithi123.stdout: "warning" 2024-09-16T15:47:33.217 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-09-16T15:47:33.217 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:33.217 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.217 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.217 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.218 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.218 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_cluster": { 2024-09-16T15:47:33.218 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_cluster", 2024-09-16T15:47:33.218 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-09-16T15:47:33.218 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.218 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-09-16T15:47:33.218 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-09-16T15:47:33.218 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.218 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.218 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.218 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:33.218 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.218 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.218 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.219 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.219 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_cluster_level": { 2024-09-16T15:47:33.219 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_cluster_level", 2024-09-16T15:47:33.219 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-09-16T15:47:33.219 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.219 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-09-16T15:47:33.219 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "info", 2024-09-16T15:47:33.219 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.219 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.219 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [ 2024-09-16T15:47:33.219 INFO:teuthology.orchestra.run.smithi123.stdout: "", 2024-09-16T15:47:33.219 INFO:teuthology.orchestra.run.smithi123.stdout: "critical", 2024-09-16T15:47:33.219 INFO:teuthology.orchestra.run.smithi123.stdout: "debug", 2024-09-16T15:47:33.219 INFO:teuthology.orchestra.run.smithi123.stdout: "error", 2024-09-16T15:47:33.220 INFO:teuthology.orchestra.run.smithi123.stdout: "info", 2024-09-16T15:47:33.220 INFO:teuthology.orchestra.run.smithi123.stdout: "warning" 2024-09-16T15:47:33.220 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-09-16T15:47:33.220 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:33.220 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.220 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.220 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.220 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.220 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_file": { 2024-09-16T15:47:33.220 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_file", 2024-09-16T15:47:33.220 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-09-16T15:47:33.220 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.220 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-09-16T15:47:33.220 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-09-16T15:47:33.220 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.221 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.221 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.221 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:33.221 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.221 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.221 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.221 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.221 INFO:teuthology.orchestra.run.smithi123.stdout: "min_score": { 2024-09-16T15:47:33.221 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "min_score", 2024-09-16T15:47:33.221 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "float", 2024-09-16T15:47:33.221 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.221 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-09-16T15:47:33.221 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "0", 2024-09-16T15:47:33.221 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.221 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.221 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.222 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "minimum score, below which no optimization is attempted", 2024-09-16T15:47:33.222 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.222 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.222 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.222 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.222 INFO:teuthology.orchestra.run.smithi123.stdout: "mode": { 2024-09-16T15:47:33.222 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "mode", 2024-09-16T15:47:33.222 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-09-16T15:47:33.222 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.222 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-09-16T15:47:33.222 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "upmap", 2024-09-16T15:47:33.222 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.222 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.222 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [ 2024-09-16T15:47:33.223 INFO:teuthology.orchestra.run.smithi123.stdout: "crush-compat", 2024-09-16T15:47:33.223 INFO:teuthology.orchestra.run.smithi123.stdout: "none", 2024-09-16T15:47:33.223 INFO:teuthology.orchestra.run.smithi123.stdout: "read", 2024-09-16T15:47:33.223 INFO:teuthology.orchestra.run.smithi123.stdout: "upmap", 2024-09-16T15:47:33.223 INFO:teuthology.orchestra.run.smithi123.stdout: "upmap-read" 2024-09-16T15:47:33.223 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-09-16T15:47:33.223 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "Balancer mode", 2024-09-16T15:47:33.223 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.223 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.223 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.223 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.223 INFO:teuthology.orchestra.run.smithi123.stdout: "pool_ids": { 2024-09-16T15:47:33.223 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "pool_ids", 2024-09-16T15:47:33.223 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-09-16T15:47:33.224 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.224 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-09-16T15:47:33.224 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "", 2024-09-16T15:47:33.224 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.224 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.224 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.224 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "pools which the automatic balancing will be limited to", 2024-09-16T15:47:33.224 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.224 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.224 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.224 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.224 INFO:teuthology.orchestra.run.smithi123.stdout: "sleep_interval": { 2024-09-16T15:47:33.224 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "sleep_interval", 2024-09-16T15:47:33.224 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "secs", 2024-09-16T15:47:33.225 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.225 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-09-16T15:47:33.225 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "60", 2024-09-16T15:47:33.225 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.225 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.225 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.225 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "how frequently to wake up and attempt optimization", 2024-09-16T15:47:33.225 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.225 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.225 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.225 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.225 INFO:teuthology.orchestra.run.smithi123.stdout: "sqlite3_killpoint": { 2024-09-16T15:47:33.225 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "sqlite3_killpoint", 2024-09-16T15:47:33.226 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "int", 2024-09-16T15:47:33.226 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "dev", 2024-09-16T15:47:33.226 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-09-16T15:47:33.226 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "0", 2024-09-16T15:47:33.226 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.226 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.226 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.226 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:33.226 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.226 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.226 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.226 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.226 INFO:teuthology.orchestra.run.smithi123.stdout: "upmap_max_deviation": { 2024-09-16T15:47:33.226 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "upmap_max_deviation", 2024-09-16T15:47:33.226 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "int", 2024-09-16T15:47:33.227 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.227 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-09-16T15:47:33.227 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "5", 2024-09-16T15:47:33.227 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "1", 2024-09-16T15:47:33.227 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.227 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.227 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "deviation below which no optimization is attempted", 2024-09-16T15:47:33.227 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "If the number of PGs are within this count then no optimization is attempted", 2024-09-16T15:47:33.227 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.227 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.227 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.227 INFO:teuthology.orchestra.run.smithi123.stdout: "upmap_max_optimizations": { 2024-09-16T15:47:33.227 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "upmap_max_optimizations", 2024-09-16T15:47:33.227 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "uint", 2024-09-16T15:47:33.227 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.227 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-09-16T15:47:33.228 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "10", 2024-09-16T15:47:33.228 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.228 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.228 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.228 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "maximum upmap optimizations to make per attempt", 2024-09-16T15:47:33.228 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.228 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.228 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.228 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-16T15:47:33.228 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-16T15:47:33.228 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.228 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-09-16T15:47:33.228 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "cephadm", 2024-09-16T15:47:33.228 INFO:teuthology.orchestra.run.smithi123.stdout: "can_run": true, 2024-09-16T15:47:33.229 INFO:teuthology.orchestra.run.smithi123.stdout: "error_string": "", 2024-09-16T15:47:33.229 INFO:teuthology.orchestra.run.smithi123.stdout: "module_options": { 2024-09-16T15:47:33.229 INFO:teuthology.orchestra.run.smithi123.stdout: "agent_down_multiplier": { 2024-09-16T15:47:33.229 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "agent_down_multiplier", 2024-09-16T15:47:33.229 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "float", 2024-09-16T15:47:33.229 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.229 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-09-16T15:47:33.229 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "3.0", 2024-09-16T15:47:33.229 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.229 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.229 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.229 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "Multiplied by agent refresh rate to calculate how long agent must not report before being marked down", 2024-09-16T15:47:33.229 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.229 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.230 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.230 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.230 INFO:teuthology.orchestra.run.smithi123.stdout: "agent_refresh_rate": { 2024-09-16T15:47:33.230 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "agent_refresh_rate", 2024-09-16T15:47:33.230 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "secs", 2024-09-16T15:47:33.230 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.230 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-09-16T15:47:33.230 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "20", 2024-09-16T15:47:33.230 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.230 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.230 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.230 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "How often agent on each host will try to gather and send metadata", 2024-09-16T15:47:33.230 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.230 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.230 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.231 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.231 INFO:teuthology.orchestra.run.smithi123.stdout: "agent_starting_port": { 2024-09-16T15:47:33.231 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "agent_starting_port", 2024-09-16T15:47:33.231 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "int", 2024-09-16T15:47:33.231 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.231 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-09-16T15:47:33.231 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "4721", 2024-09-16T15:47:33.231 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.231 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.231 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.231 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)", 2024-09-16T15:47:33.231 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.231 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.231 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.231 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.232 INFO:teuthology.orchestra.run.smithi123.stdout: "allow_ptrace": { 2024-09-16T15:47:33.232 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "allow_ptrace", 2024-09-16T15:47:33.232 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-09-16T15:47:33.232 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.232 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-09-16T15:47:33.232 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-09-16T15:47:33.232 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.232 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.232 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.232 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "allow SYS_PTRACE capability on ceph containers", 2024-09-16T15:47:33.232 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.", 2024-09-16T15:47:33.232 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.232 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.232 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.232 INFO:teuthology.orchestra.run.smithi123.stdout: "autotune_interval": { 2024-09-16T15:47:33.233 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "autotune_interval", 2024-09-16T15:47:33.233 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "secs", 2024-09-16T15:47:33.233 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.233 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-09-16T15:47:33.233 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "600", 2024-09-16T15:47:33.233 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.233 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.233 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.233 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "how frequently to autotune daemon memory", 2024-09-16T15:47:33.233 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.233 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.233 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.233 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.233 INFO:teuthology.orchestra.run.smithi123.stdout: "autotune_memory_target_ratio": { 2024-09-16T15:47:33.233 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "autotune_memory_target_ratio", 2024-09-16T15:47:33.233 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "float", 2024-09-16T15:47:33.234 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.234 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-09-16T15:47:33.234 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "0.7", 2024-09-16T15:47:33.234 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.234 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.234 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.234 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "ratio of total system memory to divide amongst autotuned daemons", 2024-09-16T15:47:33.234 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.234 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.234 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.234 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.234 INFO:teuthology.orchestra.run.smithi123.stdout: "cephadm_log_destination": { 2024-09-16T15:47:33.234 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "cephadm_log_destination", 2024-09-16T15:47:33.234 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-09-16T15:47:33.234 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.235 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-09-16T15:47:33.235 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "", 2024-09-16T15:47:33.235 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.235 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.235 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [ 2024-09-16T15:47:33.235 INFO:teuthology.orchestra.run.smithi123.stdout: "file", 2024-09-16T15:47:33.235 INFO:teuthology.orchestra.run.smithi123.stdout: "file,syslog", 2024-09-16T15:47:33.235 INFO:teuthology.orchestra.run.smithi123.stdout: "syslog" 2024-09-16T15:47:33.235 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-09-16T15:47:33.235 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "Destination for cephadm command's persistent logging", 2024-09-16T15:47:33.235 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.235 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.235 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.236 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.236 INFO:teuthology.orchestra.run.smithi123.stdout: "cgroups_split": { 2024-09-16T15:47:33.236 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "cgroups_split", 2024-09-16T15:47:33.236 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-09-16T15:47:33.236 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.236 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-09-16T15:47:33.236 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "True", 2024-09-16T15:47:33.236 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.236 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.236 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.236 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "Pass --cgroups=split when cephadm creates containers (currently podman only)", 2024-09-16T15:47:33.236 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.236 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.236 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.236 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.237 INFO:teuthology.orchestra.run.smithi123.stdout: "config_checks_enabled": { 2024-09-16T15:47:33.237 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "config_checks_enabled", 2024-09-16T15:47:33.237 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-09-16T15:47:33.237 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.237 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-09-16T15:47:33.237 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-09-16T15:47:33.237 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.237 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.237 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.237 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "Enable or disable the cephadm configuration analysis", 2024-09-16T15:47:33.237 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.237 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.237 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.237 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.237 INFO:teuthology.orchestra.run.smithi123.stdout: "config_dashboard": { 2024-09-16T15:47:33.237 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "config_dashboard", 2024-09-16T15:47:33.238 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-09-16T15:47:33.238 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.238 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-09-16T15:47:33.238 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "True", 2024-09-16T15:47:33.238 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.238 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.238 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.238 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "manage configs like API endpoints in Dashboard.", 2024-09-16T15:47:33.238 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.238 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.238 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.238 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.238 INFO:teuthology.orchestra.run.smithi123.stdout: "container_image_alertmanager": { 2024-09-16T15:47:33.238 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "container_image_alertmanager", 2024-09-16T15:47:33.239 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-09-16T15:47:33.239 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.239 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-09-16T15:47:33.239 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "quay.io/prometheus/alertmanager:v0.27.0", 2024-09-16T15:47:33.239 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.239 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.239 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.239 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "Prometheus container image", 2024-09-16T15:47:33.239 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.239 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.239 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.239 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.239 INFO:teuthology.orchestra.run.smithi123.stdout: "container_image_base": { 2024-09-16T15:47:33.239 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "container_image_base", 2024-09-16T15:47:33.239 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-09-16T15:47:33.239 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.240 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-09-16T15:47:33.240 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "quay.io/ceph/ceph", 2024-09-16T15:47:33.240 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.240 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.240 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.240 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "Container image name, without the tag", 2024-09-16T15:47:33.240 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.240 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.240 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.240 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.240 INFO:teuthology.orchestra.run.smithi123.stdout: "container_image_elasticsearch": { 2024-09-16T15:47:33.240 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "container_image_elasticsearch", 2024-09-16T15:47:33.240 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-09-16T15:47:33.240 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.240 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-09-16T15:47:33.241 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "quay.io/omrizeneva/elasticsearch:6.8.23", 2024-09-16T15:47:33.241 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.241 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.241 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.241 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "elasticsearch container image", 2024-09-16T15:47:33.241 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.241 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.241 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.241 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.241 INFO:teuthology.orchestra.run.smithi123.stdout: "container_image_grafana": { 2024-09-16T15:47:33.241 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "container_image_grafana", 2024-09-16T15:47:33.241 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-09-16T15:47:33.241 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.241 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-09-16T15:47:33.241 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "quay.io/ceph/grafana:10.4.0", 2024-09-16T15:47:33.241 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.242 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.242 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.242 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "Prometheus container image", 2024-09-16T15:47:33.242 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.242 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.242 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.242 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.242 INFO:teuthology.orchestra.run.smithi123.stdout: "container_image_haproxy": { 2024-09-16T15:47:33.242 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "container_image_haproxy", 2024-09-16T15:47:33.242 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-09-16T15:47:33.242 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.243 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-09-16T15:47:33.243 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "quay.io/ceph/haproxy:2.3", 2024-09-16T15:47:33.243 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.243 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.243 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.243 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "HAproxy container image", 2024-09-16T15:47:33.243 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.243 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.243 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.243 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.243 INFO:teuthology.orchestra.run.smithi123.stdout: "container_image_jaeger_agent": { 2024-09-16T15:47:33.243 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "container_image_jaeger_agent", 2024-09-16T15:47:33.243 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-09-16T15:47:33.243 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.243 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-09-16T15:47:33.244 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "quay.io/jaegertracing/jaeger-agent:1.29", 2024-09-16T15:47:33.244 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.244 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.244 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.244 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "Jaeger agent container image", 2024-09-16T15:47:33.244 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.244 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.244 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.244 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.244 INFO:teuthology.orchestra.run.smithi123.stdout: "container_image_jaeger_collector": { 2024-09-16T15:47:33.244 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "container_image_jaeger_collector", 2024-09-16T15:47:33.244 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-09-16T15:47:33.244 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.244 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-09-16T15:47:33.244 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "quay.io/jaegertracing/jaeger-collector:1.29", 2024-09-16T15:47:33.245 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.245 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.245 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.245 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "Jaeger collector container image", 2024-09-16T15:47:33.245 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.245 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.245 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.245 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.245 INFO:teuthology.orchestra.run.smithi123.stdout: "container_image_jaeger_query": { 2024-09-16T15:47:33.245 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "container_image_jaeger_query", 2024-09-16T15:47:33.245 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-09-16T15:47:33.245 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.245 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-09-16T15:47:33.246 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "quay.io/jaegertracing/jaeger-query:1.29", 2024-09-16T15:47:33.246 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.246 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.246 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.246 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "Jaeger query container image", 2024-09-16T15:47:33.246 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.246 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.246 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.246 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.246 INFO:teuthology.orchestra.run.smithi123.stdout: "container_image_keepalived": { 2024-09-16T15:47:33.246 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "container_image_keepalived", 2024-09-16T15:47:33.246 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-09-16T15:47:33.246 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.246 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-09-16T15:47:33.246 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "quay.io/ceph/keepalived:2.2.4", 2024-09-16T15:47:33.246 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.247 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.247 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.247 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "Keepalived container image", 2024-09-16T15:47:33.247 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.247 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.247 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.247 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.247 INFO:teuthology.orchestra.run.smithi123.stdout: "container_image_loki": { 2024-09-16T15:47:33.247 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "container_image_loki", 2024-09-16T15:47:33.247 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-09-16T15:47:33.247 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.247 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-09-16T15:47:33.247 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "docker.io/grafana/loki:3.0.0", 2024-09-16T15:47:33.247 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.248 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.248 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.248 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "Loki container image", 2024-09-16T15:47:33.248 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.248 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.248 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.248 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.248 INFO:teuthology.orchestra.run.smithi123.stdout: "container_image_nginx": { 2024-09-16T15:47:33.248 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "container_image_nginx", 2024-09-16T15:47:33.248 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-09-16T15:47:33.248 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.248 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-09-16T15:47:33.248 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "quay.io/ceph/nginx:sclorg-nginx-126", 2024-09-16T15:47:33.248 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.249 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.249 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.249 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "Nginx container image", 2024-09-16T15:47:33.249 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.249 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.249 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.249 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.249 INFO:teuthology.orchestra.run.smithi123.stdout: "container_image_node_exporter": { 2024-09-16T15:47:33.249 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "container_image_node_exporter", 2024-09-16T15:47:33.249 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-09-16T15:47:33.249 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.249 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-09-16T15:47:33.249 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "quay.io/prometheus/node-exporter:v1.7.0", 2024-09-16T15:47:33.250 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.250 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.250 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.250 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "Prometheus container image", 2024-09-16T15:47:33.250 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.250 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.250 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.250 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.250 INFO:teuthology.orchestra.run.smithi123.stdout: "container_image_nvmeof": { 2024-09-16T15:47:33.250 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "container_image_nvmeof", 2024-09-16T15:47:33.250 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-09-16T15:47:33.250 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.250 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-09-16T15:47:33.250 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "quay.io/ceph/nvmeof:1.2.17", 2024-09-16T15:47:33.250 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.251 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.251 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.251 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "Nvme-of container image", 2024-09-16T15:47:33.251 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.251 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.251 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.251 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.251 INFO:teuthology.orchestra.run.smithi123.stdout: "container_image_oauth2_proxy": { 2024-09-16T15:47:33.251 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "container_image_oauth2_proxy", 2024-09-16T15:47:33.251 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-09-16T15:47:33.251 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.251 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-09-16T15:47:33.251 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "quay.io/oauth2-proxy/oauth2-proxy:v7.6.0", 2024-09-16T15:47:33.251 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.251 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.251 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.252 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "oauth2-proxy container image", 2024-09-16T15:47:33.252 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.252 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.252 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.252 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.252 INFO:teuthology.orchestra.run.smithi123.stdout: "container_image_prometheus": { 2024-09-16T15:47:33.252 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "container_image_prometheus", 2024-09-16T15:47:33.252 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-09-16T15:47:33.252 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.252 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-09-16T15:47:33.252 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "quay.io/prometheus/prometheus:v2.51.0", 2024-09-16T15:47:33.252 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.252 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.252 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.252 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "Prometheus container image", 2024-09-16T15:47:33.253 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.253 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.253 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.253 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.253 INFO:teuthology.orchestra.run.smithi123.stdout: "container_image_promtail": { 2024-09-16T15:47:33.253 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "container_image_promtail", 2024-09-16T15:47:33.253 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-09-16T15:47:33.253 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.253 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-09-16T15:47:33.253 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "docker.io/grafana/promtail:3.0.0", 2024-09-16T15:47:33.253 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.253 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.253 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.253 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "Promtail container image", 2024-09-16T15:47:33.253 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.254 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.254 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.254 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.254 INFO:teuthology.orchestra.run.smithi123.stdout: "container_image_samba": { 2024-09-16T15:47:33.254 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "container_image_samba", 2024-09-16T15:47:33.254 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-09-16T15:47:33.254 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.254 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-09-16T15:47:33.254 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "quay.io/samba.org/samba-server:devbuilds-centos-amd64", 2024-09-16T15:47:33.254 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.254 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.254 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.254 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "Samba/SMB container image", 2024-09-16T15:47:33.254 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.254 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.255 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.255 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.255 INFO:teuthology.orchestra.run.smithi123.stdout: "container_image_samba_metrics": { 2024-09-16T15:47:33.255 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "container_image_samba_metrics", 2024-09-16T15:47:33.255 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-09-16T15:47:33.255 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.255 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-09-16T15:47:33.255 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "quay.io/samba.org/samba-metrics:latest", 2024-09-16T15:47:33.255 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.255 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.255 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.255 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "Samba/SMB metrics exporter container image", 2024-09-16T15:47:33.255 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.255 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.255 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.256 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.256 INFO:teuthology.orchestra.run.smithi123.stdout: "container_image_snmp_gateway": { 2024-09-16T15:47:33.256 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "container_image_snmp_gateway", 2024-09-16T15:47:33.256 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-09-16T15:47:33.256 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.256 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-09-16T15:47:33.256 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "docker.io/maxwo/snmp-notifier:v1.2.1", 2024-09-16T15:47:33.256 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.256 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.256 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.256 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "SNMP Gateway container image", 2024-09-16T15:47:33.256 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.256 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.256 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.256 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.256 INFO:teuthology.orchestra.run.smithi123.stdout: "container_init": { 2024-09-16T15:47:33.257 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "container_init", 2024-09-16T15:47:33.257 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-09-16T15:47:33.257 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.257 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-09-16T15:47:33.257 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "True", 2024-09-16T15:47:33.257 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.257 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.257 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.257 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "Run podman/docker with `--init`", 2024-09-16T15:47:33.257 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.257 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.257 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.257 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.257 INFO:teuthology.orchestra.run.smithi123.stdout: "daemon_cache_timeout": { 2024-09-16T15:47:33.257 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "daemon_cache_timeout", 2024-09-16T15:47:33.258 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "secs", 2024-09-16T15:47:33.258 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.258 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-09-16T15:47:33.258 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "600", 2024-09-16T15:47:33.258 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.258 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.258 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.258 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "seconds to cache service (daemon) inventory", 2024-09-16T15:47:33.258 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.258 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.258 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.259 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.259 INFO:teuthology.orchestra.run.smithi123.stdout: "default_cephadm_command_timeout": { 2024-09-16T15:47:33.259 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "default_cephadm_command_timeout", 2024-09-16T15:47:33.259 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "int", 2024-09-16T15:47:33.259 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.259 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-09-16T15:47:33.259 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "900", 2024-09-16T15:47:33.259 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.259 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.259 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.259 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "Default timeout applied to cephadm commands run directly on the host (in seconds)", 2024-09-16T15:47:33.259 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.259 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.259 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.259 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.259 INFO:teuthology.orchestra.run.smithi123.stdout: "default_registry": { 2024-09-16T15:47:33.260 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "default_registry", 2024-09-16T15:47:33.260 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-09-16T15:47:33.260 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.260 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-09-16T15:47:33.260 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "docker.io", 2024-09-16T15:47:33.260 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.260 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.260 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.260 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "Search-registry to which we should normalize unqualified image names. This is not the default registry", 2024-09-16T15:47:33.260 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.260 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.260 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.260 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.260 INFO:teuthology.orchestra.run.smithi123.stdout: "device_cache_timeout": { 2024-09-16T15:47:33.260 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "device_cache_timeout", 2024-09-16T15:47:33.261 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "secs", 2024-09-16T15:47:33.261 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.261 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-09-16T15:47:33.261 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "1800", 2024-09-16T15:47:33.261 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.261 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.261 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.261 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "seconds to cache device inventory", 2024-09-16T15:47:33.261 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.261 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.261 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.261 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.261 INFO:teuthology.orchestra.run.smithi123.stdout: "device_enhanced_scan": { 2024-09-16T15:47:33.261 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "device_enhanced_scan", 2024-09-16T15:47:33.261 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-09-16T15:47:33.261 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.262 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-09-16T15:47:33.262 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-09-16T15:47:33.262 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.262 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.262 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.262 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "Use libstoragemgmt during device scans", 2024-09-16T15:47:33.262 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.262 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.262 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.262 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.262 INFO:teuthology.orchestra.run.smithi123.stdout: "facts_cache_timeout": { 2024-09-16T15:47:33.262 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "facts_cache_timeout", 2024-09-16T15:47:33.262 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "secs", 2024-09-16T15:47:33.262 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.262 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-09-16T15:47:33.263 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "60", 2024-09-16T15:47:33.263 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.263 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.263 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.263 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "seconds to cache host facts data", 2024-09-16T15:47:33.263 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.263 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.263 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.263 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.263 INFO:teuthology.orchestra.run.smithi123.stdout: "grafana_dashboards_path": { 2024-09-16T15:47:33.263 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "grafana_dashboards_path", 2024-09-16T15:47:33.263 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-09-16T15:47:33.263 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.263 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-09-16T15:47:33.263 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "/etc/grafana/dashboards/ceph-dashboard/", 2024-09-16T15:47:33.264 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.264 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.264 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.264 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "location of dashboards to include in grafana deployments", 2024-09-16T15:47:33.264 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.264 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.264 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.264 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.264 INFO:teuthology.orchestra.run.smithi123.stdout: "host_check_interval": { 2024-09-16T15:47:33.264 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "host_check_interval", 2024-09-16T15:47:33.264 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "secs", 2024-09-16T15:47:33.264 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.264 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-09-16T15:47:33.264 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "600", 2024-09-16T15:47:33.264 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.264 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.265 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.265 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "how frequently to perform a host check", 2024-09-16T15:47:33.265 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.265 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.265 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.265 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.265 INFO:teuthology.orchestra.run.smithi123.stdout: "hw_monitoring": { 2024-09-16T15:47:33.265 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "hw_monitoring", 2024-09-16T15:47:33.265 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-09-16T15:47:33.265 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.265 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-09-16T15:47:33.265 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-09-16T15:47:33.265 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.265 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.265 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.266 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "Deploy hw monitoring daemon on every host.", 2024-09-16T15:47:33.266 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.266 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.266 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.266 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.266 INFO:teuthology.orchestra.run.smithi123.stdout: "inventory_list_all": { 2024-09-16T15:47:33.266 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "inventory_list_all", 2024-09-16T15:47:33.266 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-09-16T15:47:33.266 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.266 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-09-16T15:47:33.266 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-09-16T15:47:33.266 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.266 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.266 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.266 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)", 2024-09-16T15:47:33.267 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.267 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.267 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.267 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.267 INFO:teuthology.orchestra.run.smithi123.stdout: "log_level": { 2024-09-16T15:47:33.267 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_level", 2024-09-16T15:47:33.267 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-09-16T15:47:33.267 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.267 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-09-16T15:47:33.267 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "", 2024-09-16T15:47:33.267 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.267 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.267 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [ 2024-09-16T15:47:33.267 INFO:teuthology.orchestra.run.smithi123.stdout: "", 2024-09-16T15:47:33.267 INFO:teuthology.orchestra.run.smithi123.stdout: "critical", 2024-09-16T15:47:33.267 INFO:teuthology.orchestra.run.smithi123.stdout: "debug", 2024-09-16T15:47:33.268 INFO:teuthology.orchestra.run.smithi123.stdout: "error", 2024-09-16T15:47:33.268 INFO:teuthology.orchestra.run.smithi123.stdout: "info", 2024-09-16T15:47:33.268 INFO:teuthology.orchestra.run.smithi123.stdout: "warning" 2024-09-16T15:47:33.268 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-09-16T15:47:33.268 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:33.268 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.268 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.268 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.268 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.268 INFO:teuthology.orchestra.run.smithi123.stdout: "log_refresh_metadata": { 2024-09-16T15:47:33.268 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_refresh_metadata", 2024-09-16T15:47:33.268 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-09-16T15:47:33.268 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.268 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-09-16T15:47:33.268 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-09-16T15:47:33.269 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.269 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.269 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.269 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level", 2024-09-16T15:47:33.269 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.269 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.269 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.269 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.269 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_cluster": { 2024-09-16T15:47:33.269 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_cluster", 2024-09-16T15:47:33.269 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-09-16T15:47:33.269 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.269 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-09-16T15:47:33.270 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "True", 2024-09-16T15:47:33.270 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.270 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.270 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.270 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "log to the \"cephadm\" cluster log channel\"", 2024-09-16T15:47:33.270 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.270 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.270 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.270 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.270 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_cluster_level": { 2024-09-16T15:47:33.270 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_cluster_level", 2024-09-16T15:47:33.270 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-09-16T15:47:33.270 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.270 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-09-16T15:47:33.271 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "info", 2024-09-16T15:47:33.271 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.271 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.271 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [ 2024-09-16T15:47:33.271 INFO:teuthology.orchestra.run.smithi123.stdout: "", 2024-09-16T15:47:33.271 INFO:teuthology.orchestra.run.smithi123.stdout: "critical", 2024-09-16T15:47:33.271 INFO:teuthology.orchestra.run.smithi123.stdout: "debug", 2024-09-16T15:47:33.271 INFO:teuthology.orchestra.run.smithi123.stdout: "error", 2024-09-16T15:47:33.271 INFO:teuthology.orchestra.run.smithi123.stdout: "info", 2024-09-16T15:47:33.271 INFO:teuthology.orchestra.run.smithi123.stdout: "warning" 2024-09-16T15:47:33.271 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-09-16T15:47:33.272 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:33.272 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.272 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.272 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.272 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.272 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_file": { 2024-09-16T15:47:33.272 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_file", 2024-09-16T15:47:33.272 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-09-16T15:47:33.272 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.272 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-09-16T15:47:33.272 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-09-16T15:47:33.272 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.272 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.272 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.273 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:33.273 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.273 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.273 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.273 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.273 INFO:teuthology.orchestra.run.smithi123.stdout: "manage_etc_ceph_ceph_conf": { 2024-09-16T15:47:33.273 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "manage_etc_ceph_ceph_conf", 2024-09-16T15:47:33.273 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-09-16T15:47:33.273 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.273 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-09-16T15:47:33.273 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-09-16T15:47:33.273 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.273 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.273 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.273 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "Manage and own /etc/ceph/ceph.conf on the hosts.", 2024-09-16T15:47:33.274 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.274 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.274 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.274 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.274 INFO:teuthology.orchestra.run.smithi123.stdout: "manage_etc_ceph_ceph_conf_hosts": { 2024-09-16T15:47:33.274 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "manage_etc_ceph_ceph_conf_hosts", 2024-09-16T15:47:33.274 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-09-16T15:47:33.274 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.274 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-09-16T15:47:33.274 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "*", 2024-09-16T15:47:33.274 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.274 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.274 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.274 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf", 2024-09-16T15:47:33.275 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.275 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.275 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.275 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.275 INFO:teuthology.orchestra.run.smithi123.stdout: "max_count_per_host": { 2024-09-16T15:47:33.275 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "max_count_per_host", 2024-09-16T15:47:33.275 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "int", 2024-09-16T15:47:33.275 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.275 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-09-16T15:47:33.275 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "10", 2024-09-16T15:47:33.275 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.275 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.275 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.275 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "max number of daemons per service per host", 2024-09-16T15:47:33.275 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.276 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.276 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.276 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.276 INFO:teuthology.orchestra.run.smithi123.stdout: "max_osd_draining_count": { 2024-09-16T15:47:33.276 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "max_osd_draining_count", 2024-09-16T15:47:33.276 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "int", 2024-09-16T15:47:33.276 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.276 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-09-16T15:47:33.276 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "10", 2024-09-16T15:47:33.276 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.276 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.276 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.276 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "max number of osds that will be drained simultaneously when osds are removed", 2024-09-16T15:47:33.276 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.277 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.277 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.277 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.277 INFO:teuthology.orchestra.run.smithi123.stdout: "migration_current": { 2024-09-16T15:47:33.277 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "migration_current", 2024-09-16T15:47:33.277 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "int", 2024-09-16T15:47:33.277 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.277 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-09-16T15:47:33.277 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "", 2024-09-16T15:47:33.277 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.277 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.277 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.277 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "internal - do not modify", 2024-09-16T15:47:33.277 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.277 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.278 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.278 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.278 INFO:teuthology.orchestra.run.smithi123.stdout: "mode": { 2024-09-16T15:47:33.278 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "mode", 2024-09-16T15:47:33.278 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-09-16T15:47:33.278 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.278 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-09-16T15:47:33.278 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "root", 2024-09-16T15:47:33.278 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.278 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.278 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [ 2024-09-16T15:47:33.278 INFO:teuthology.orchestra.run.smithi123.stdout: "cephadm-package", 2024-09-16T15:47:33.278 INFO:teuthology.orchestra.run.smithi123.stdout: "root" 2024-09-16T15:47:33.278 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-09-16T15:47:33.278 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "mode for remote execution of cephadm", 2024-09-16T15:47:33.279 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.279 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.279 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.279 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.279 INFO:teuthology.orchestra.run.smithi123.stdout: "oob_default_addr": { 2024-09-16T15:47:33.279 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "oob_default_addr", 2024-09-16T15:47:33.279 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-09-16T15:47:33.279 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.279 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-09-16T15:47:33.279 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "169.254.1.1", 2024-09-16T15:47:33.279 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.279 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.279 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.279 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "Default address for RedFish API (oob management).", 2024-09-16T15:47:33.279 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.280 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.280 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.280 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.280 INFO:teuthology.orchestra.run.smithi123.stdout: "prometheus_alerts_path": { 2024-09-16T15:47:33.280 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "prometheus_alerts_path", 2024-09-16T15:47:33.280 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-09-16T15:47:33.280 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.280 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-09-16T15:47:33.280 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "/etc/prometheus/ceph/ceph_default_alerts.yml", 2024-09-16T15:47:33.280 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.280 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.280 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.280 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "location of alerts to include in prometheus deployments", 2024-09-16T15:47:33.280 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.280 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.280 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.281 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.281 INFO:teuthology.orchestra.run.smithi123.stdout: "registry_insecure": { 2024-09-16T15:47:33.281 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "registry_insecure", 2024-09-16T15:47:33.281 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-09-16T15:47:33.281 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.281 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-09-16T15:47:33.281 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-09-16T15:47:33.281 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.281 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.281 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.281 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "Registry is to be considered insecure (no TLS available). Only for development purposes.", 2024-09-16T15:47:33.281 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.281 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.281 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.281 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.282 INFO:teuthology.orchestra.run.smithi123.stdout: "registry_password": { 2024-09-16T15:47:33.282 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "registry_password", 2024-09-16T15:47:33.282 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-09-16T15:47:33.282 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.282 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-09-16T15:47:33.282 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "", 2024-09-16T15:47:33.282 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.282 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.282 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.282 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "Custom repository password. Only used for logging into a registry.", 2024-09-16T15:47:33.282 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.282 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.282 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.282 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.282 INFO:teuthology.orchestra.run.smithi123.stdout: "registry_url": { 2024-09-16T15:47:33.283 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "registry_url", 2024-09-16T15:47:33.283 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-09-16T15:47:33.283 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.283 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-09-16T15:47:33.283 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "", 2024-09-16T15:47:33.283 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.283 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.283 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.283 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "Registry url for login purposes. This is not the default registry", 2024-09-16T15:47:33.283 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.283 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.283 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.283 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.283 INFO:teuthology.orchestra.run.smithi123.stdout: "registry_username": { 2024-09-16T15:47:33.284 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "registry_username", 2024-09-16T15:47:33.284 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-09-16T15:47:33.284 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.284 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-09-16T15:47:33.284 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "", 2024-09-16T15:47:33.284 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.284 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.284 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.284 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "Custom repository username. Only used for logging into a registry.", 2024-09-16T15:47:33.284 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.284 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.284 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.284 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.284 INFO:teuthology.orchestra.run.smithi123.stdout: "secure_monitoring_stack": { 2024-09-16T15:47:33.285 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "secure_monitoring_stack", 2024-09-16T15:47:33.285 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-09-16T15:47:33.285 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.285 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-09-16T15:47:33.285 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-09-16T15:47:33.285 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.285 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.285 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.285 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "Enable TLS security for all the monitoring stack daemons", 2024-09-16T15:47:33.285 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.285 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.285 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.286 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.286 INFO:teuthology.orchestra.run.smithi123.stdout: "service_discovery_port": { 2024-09-16T15:47:33.286 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "service_discovery_port", 2024-09-16T15:47:33.286 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "int", 2024-09-16T15:47:33.286 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.286 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-09-16T15:47:33.286 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "8765", 2024-09-16T15:47:33.286 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.286 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.286 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.286 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "cephadm service discovery port", 2024-09-16T15:47:33.286 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.286 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.286 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.286 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.287 INFO:teuthology.orchestra.run.smithi123.stdout: "sqlite3_killpoint": { 2024-09-16T15:47:33.287 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "sqlite3_killpoint", 2024-09-16T15:47:33.287 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "int", 2024-09-16T15:47:33.287 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "dev", 2024-09-16T15:47:33.287 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-09-16T15:47:33.287 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "0", 2024-09-16T15:47:33.287 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.287 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.287 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.287 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:33.287 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.287 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.287 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.287 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.287 INFO:teuthology.orchestra.run.smithi123.stdout: "ssh_config_file": { 2024-09-16T15:47:33.288 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "ssh_config_file", 2024-09-16T15:47:33.288 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-09-16T15:47:33.288 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.288 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-09-16T15:47:33.288 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "", 2024-09-16T15:47:33.288 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.288 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.288 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.288 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "customized SSH config file to connect to managed hosts", 2024-09-16T15:47:33.288 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.288 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.288 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.289 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.289 INFO:teuthology.orchestra.run.smithi123.stdout: "ssh_keepalive_count_max": { 2024-09-16T15:47:33.289 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "ssh_keepalive_count_max", 2024-09-16T15:47:33.289 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "int", 2024-09-16T15:47:33.289 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.289 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-09-16T15:47:33.289 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "3", 2024-09-16T15:47:33.289 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.289 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.289 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.289 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "How many times ssh connections can fail liveness checks before the host is marked offline", 2024-09-16T15:47:33.289 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.289 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.289 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.290 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.290 INFO:teuthology.orchestra.run.smithi123.stdout: "ssh_keepalive_interval": { 2024-09-16T15:47:33.290 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "ssh_keepalive_interval", 2024-09-16T15:47:33.290 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "int", 2024-09-16T15:47:33.290 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.290 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-09-16T15:47:33.290 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "7", 2024-09-16T15:47:33.290 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.290 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.290 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.290 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "How often ssh connections are checked for liveness", 2024-09-16T15:47:33.290 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.290 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.290 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.291 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.291 INFO:teuthology.orchestra.run.smithi123.stdout: "use_agent": { 2024-09-16T15:47:33.291 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "use_agent", 2024-09-16T15:47:33.291 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-09-16T15:47:33.291 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.291 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-09-16T15:47:33.291 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-09-16T15:47:33.291 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.291 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.291 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.291 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "Use cephadm agent on each host to gather and send metadata", 2024-09-16T15:47:33.291 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.291 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.291 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.291 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.291 INFO:teuthology.orchestra.run.smithi123.stdout: "use_repo_digest": { 2024-09-16T15:47:33.292 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "use_repo_digest", 2024-09-16T15:47:33.292 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-09-16T15:47:33.292 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.292 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-09-16T15:47:33.292 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "True", 2024-09-16T15:47:33.292 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.292 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.292 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.292 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "Automatically convert image tags to image digest. Make sure all daemons use the same image", 2024-09-16T15:47:33.292 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.292 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.292 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.292 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.292 INFO:teuthology.orchestra.run.smithi123.stdout: "warn_on_failed_host_check": { 2024-09-16T15:47:33.293 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "warn_on_failed_host_check", 2024-09-16T15:47:33.293 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-09-16T15:47:33.293 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.293 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-09-16T15:47:33.293 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "True", 2024-09-16T15:47:33.293 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.293 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.293 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.293 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "raise a health warning if the host check fails", 2024-09-16T15:47:33.293 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.293 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.293 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.293 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.293 INFO:teuthology.orchestra.run.smithi123.stdout: "warn_on_stray_daemons": { 2024-09-16T15:47:33.293 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "warn_on_stray_daemons", 2024-09-16T15:47:33.294 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-09-16T15:47:33.294 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.294 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-09-16T15:47:33.294 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "True", 2024-09-16T15:47:33.294 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.294 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.294 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.294 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "raise a health warning if daemons are detected that are not managed by cephadm", 2024-09-16T15:47:33.294 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.294 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.294 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.294 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.294 INFO:teuthology.orchestra.run.smithi123.stdout: "warn_on_stray_hosts": { 2024-09-16T15:47:33.294 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "warn_on_stray_hosts", 2024-09-16T15:47:33.294 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-09-16T15:47:33.295 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.295 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-09-16T15:47:33.295 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "True", 2024-09-16T15:47:33.295 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.295 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.295 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.295 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "raise a health warning if daemons are detected on a host that is not managed by cephadm", 2024-09-16T15:47:33.295 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.295 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.295 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.295 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-16T15:47:33.295 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-16T15:47:33.295 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.295 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-09-16T15:47:33.295 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "crash", 2024-09-16T15:47:33.295 INFO:teuthology.orchestra.run.smithi123.stdout: "can_run": true, 2024-09-16T15:47:33.296 INFO:teuthology.orchestra.run.smithi123.stdout: "error_string": "", 2024-09-16T15:47:33.296 INFO:teuthology.orchestra.run.smithi123.stdout: "module_options": { 2024-09-16T15:47:33.296 INFO:teuthology.orchestra.run.smithi123.stdout: "log_level": { 2024-09-16T15:47:33.296 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_level", 2024-09-16T15:47:33.296 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-09-16T15:47:33.296 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.296 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-09-16T15:47:33.296 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "", 2024-09-16T15:47:33.296 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.296 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.296 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [ 2024-09-16T15:47:33.296 INFO:teuthology.orchestra.run.smithi123.stdout: "", 2024-09-16T15:47:33.296 INFO:teuthology.orchestra.run.smithi123.stdout: "critical", 2024-09-16T15:47:33.296 INFO:teuthology.orchestra.run.smithi123.stdout: "debug", 2024-09-16T15:47:33.297 INFO:teuthology.orchestra.run.smithi123.stdout: "error", 2024-09-16T15:47:33.297 INFO:teuthology.orchestra.run.smithi123.stdout: "info", 2024-09-16T15:47:33.297 INFO:teuthology.orchestra.run.smithi123.stdout: "warning" 2024-09-16T15:47:33.297 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-09-16T15:47:33.297 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:33.297 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.297 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.297 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.297 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.297 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_cluster": { 2024-09-16T15:47:33.297 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_cluster", 2024-09-16T15:47:33.297 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-09-16T15:47:33.297 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.297 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-09-16T15:47:33.297 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-09-16T15:47:33.297 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.298 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.298 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.298 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:33.298 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.298 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.298 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.298 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.298 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_cluster_level": { 2024-09-16T15:47:33.298 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_cluster_level", 2024-09-16T15:47:33.298 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-09-16T15:47:33.298 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.298 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-09-16T15:47:33.298 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "info", 2024-09-16T15:47:33.298 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.298 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.299 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [ 2024-09-16T15:47:33.299 INFO:teuthology.orchestra.run.smithi123.stdout: "", 2024-09-16T15:47:33.299 INFO:teuthology.orchestra.run.smithi123.stdout: "critical", 2024-09-16T15:47:33.299 INFO:teuthology.orchestra.run.smithi123.stdout: "debug", 2024-09-16T15:47:33.299 INFO:teuthology.orchestra.run.smithi123.stdout: "error", 2024-09-16T15:47:33.299 INFO:teuthology.orchestra.run.smithi123.stdout: "info", 2024-09-16T15:47:33.299 INFO:teuthology.orchestra.run.smithi123.stdout: "warning" 2024-09-16T15:47:33.299 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-09-16T15:47:33.299 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:33.299 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.299 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.299 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.299 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.299 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_file": { 2024-09-16T15:47:33.299 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_file", 2024-09-16T15:47:33.300 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-09-16T15:47:33.300 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.300 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-09-16T15:47:33.300 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-09-16T15:47:33.300 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.300 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.300 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.300 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:33.300 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.300 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.300 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.300 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.300 INFO:teuthology.orchestra.run.smithi123.stdout: "retain_interval": { 2024-09-16T15:47:33.300 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "retain_interval", 2024-09-16T15:47:33.300 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "secs", 2024-09-16T15:47:33.301 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.301 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-09-16T15:47:33.301 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "31536000", 2024-09-16T15:47:33.301 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.301 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.301 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.301 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "how long to retain crashes before pruning them", 2024-09-16T15:47:33.301 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.301 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.301 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.301 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.301 INFO:teuthology.orchestra.run.smithi123.stdout: "sqlite3_killpoint": { 2024-09-16T15:47:33.301 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "sqlite3_killpoint", 2024-09-16T15:47:33.302 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "int", 2024-09-16T15:47:33.302 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "dev", 2024-09-16T15:47:33.302 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-09-16T15:47:33.302 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "0", 2024-09-16T15:47:33.302 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.302 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.302 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.302 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:33.302 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.302 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.303 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.303 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.303 INFO:teuthology.orchestra.run.smithi123.stdout: "warn_recent_interval": { 2024-09-16T15:47:33.303 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "warn_recent_interval", 2024-09-16T15:47:33.303 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "secs", 2024-09-16T15:47:33.303 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.303 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-09-16T15:47:33.303 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "1209600", 2024-09-16T15:47:33.303 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.303 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.303 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.304 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "time interval in which to warn about recent crashes", 2024-09-16T15:47:33.304 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.304 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.304 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.304 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-16T15:47:33.304 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-16T15:47:33.304 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.304 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-09-16T15:47:33.304 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "dashboard", 2024-09-16T15:47:33.304 INFO:teuthology.orchestra.run.smithi123.stdout: "can_run": true, 2024-09-16T15:47:33.304 INFO:teuthology.orchestra.run.smithi123.stdout: "error_string": "", 2024-09-16T15:47:33.304 INFO:teuthology.orchestra.run.smithi123.stdout: "module_options": { 2024-09-16T15:47:33.304 INFO:teuthology.orchestra.run.smithi123.stdout: "ACCOUNT_LOCKOUT_ATTEMPTS": { 2024-09-16T15:47:33.304 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "ACCOUNT_LOCKOUT_ATTEMPTS", 2024-09-16T15:47:33.305 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "int", 2024-09-16T15:47:33.305 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.305 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-09-16T15:47:33.305 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "10", 2024-09-16T15:47:33.305 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.305 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.305 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.305 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:33.305 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.305 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.305 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.305 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.305 INFO:teuthology.orchestra.run.smithi123.stdout: "ALERTMANAGER_API_HOST": { 2024-09-16T15:47:33.305 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "ALERTMANAGER_API_HOST", 2024-09-16T15:47:33.305 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-09-16T15:47:33.306 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.306 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-09-16T15:47:33.306 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "", 2024-09-16T15:47:33.306 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.306 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.306 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.306 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:33.306 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.306 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.306 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.306 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.306 INFO:teuthology.orchestra.run.smithi123.stdout: "ALERTMANAGER_API_SSL_VERIFY": { 2024-09-16T15:47:33.306 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "ALERTMANAGER_API_SSL_VERIFY", 2024-09-16T15:47:33.306 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-09-16T15:47:33.307 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.307 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-09-16T15:47:33.307 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "True", 2024-09-16T15:47:33.307 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.307 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.307 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.307 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:33.307 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.307 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.307 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.307 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.307 INFO:teuthology.orchestra.run.smithi123.stdout: "AUDIT_API_ENABLED": { 2024-09-16T15:47:33.307 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "AUDIT_API_ENABLED", 2024-09-16T15:47:33.307 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-09-16T15:47:33.308 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.308 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-09-16T15:47:33.308 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-09-16T15:47:33.308 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.308 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.308 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.308 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:33.308 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.308 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.308 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.308 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.308 INFO:teuthology.orchestra.run.smithi123.stdout: "AUDIT_API_LOG_PAYLOAD": { 2024-09-16T15:47:33.308 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "AUDIT_API_LOG_PAYLOAD", 2024-09-16T15:47:33.308 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-09-16T15:47:33.308 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.308 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-09-16T15:47:33.309 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "True", 2024-09-16T15:47:33.309 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.309 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.309 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.309 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:33.309 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.309 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.309 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.309 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.309 INFO:teuthology.orchestra.run.smithi123.stdout: "ENABLE_BROWSABLE_API": { 2024-09-16T15:47:33.309 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "ENABLE_BROWSABLE_API", 2024-09-16T15:47:33.309 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-09-16T15:47:33.309 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.310 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-09-16T15:47:33.310 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "True", 2024-09-16T15:47:33.310 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.310 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.310 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.310 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:33.310 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.310 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.310 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.310 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.310 INFO:teuthology.orchestra.run.smithi123.stdout: "FEATURE_TOGGLE_CEPHFS": { 2024-09-16T15:47:33.310 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "FEATURE_TOGGLE_CEPHFS", 2024-09-16T15:47:33.310 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-09-16T15:47:33.310 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.310 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-09-16T15:47:33.310 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "True", 2024-09-16T15:47:33.311 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.311 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.311 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.311 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:33.311 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.311 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.311 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.311 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.311 INFO:teuthology.orchestra.run.smithi123.stdout: "FEATURE_TOGGLE_DASHBOARD": { 2024-09-16T15:47:33.311 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "FEATURE_TOGGLE_DASHBOARD", 2024-09-16T15:47:33.311 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-09-16T15:47:33.311 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.311 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-09-16T15:47:33.311 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "True", 2024-09-16T15:47:33.311 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.312 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.312 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.312 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:33.312 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.312 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.312 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.312 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.312 INFO:teuthology.orchestra.run.smithi123.stdout: "FEATURE_TOGGLE_ISCSI": { 2024-09-16T15:47:33.312 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "FEATURE_TOGGLE_ISCSI", 2024-09-16T15:47:33.312 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-09-16T15:47:33.312 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.312 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-09-16T15:47:33.312 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "True", 2024-09-16T15:47:33.312 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.312 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.313 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.313 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:33.313 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.313 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.313 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.313 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.313 INFO:teuthology.orchestra.run.smithi123.stdout: "FEATURE_TOGGLE_MIRRORING": { 2024-09-16T15:47:33.313 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "FEATURE_TOGGLE_MIRRORING", 2024-09-16T15:47:33.313 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-09-16T15:47:33.313 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.313 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-09-16T15:47:33.313 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "True", 2024-09-16T15:47:33.313 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.313 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.314 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.314 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:33.314 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.314 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.314 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.314 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.314 INFO:teuthology.orchestra.run.smithi123.stdout: "FEATURE_TOGGLE_NFS": { 2024-09-16T15:47:33.314 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "FEATURE_TOGGLE_NFS", 2024-09-16T15:47:33.314 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-09-16T15:47:33.314 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.314 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-09-16T15:47:33.314 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "True", 2024-09-16T15:47:33.314 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.314 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.314 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.315 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:33.315 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.315 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.315 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.315 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.315 INFO:teuthology.orchestra.run.smithi123.stdout: "FEATURE_TOGGLE_RBD": { 2024-09-16T15:47:33.315 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "FEATURE_TOGGLE_RBD", 2024-09-16T15:47:33.315 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-09-16T15:47:33.315 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.315 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-09-16T15:47:33.315 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "True", 2024-09-16T15:47:33.315 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.315 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.315 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.315 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:33.315 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.316 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.316 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.316 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.316 INFO:teuthology.orchestra.run.smithi123.stdout: "FEATURE_TOGGLE_RGW": { 2024-09-16T15:47:33.316 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "FEATURE_TOGGLE_RGW", 2024-09-16T15:47:33.316 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-09-16T15:47:33.316 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.316 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-09-16T15:47:33.316 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "True", 2024-09-16T15:47:33.316 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.316 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.316 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.316 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:33.316 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.316 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.317 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.317 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.317 INFO:teuthology.orchestra.run.smithi123.stdout: "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE": { 2024-09-16T15:47:33.317 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE", 2024-09-16T15:47:33.317 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-09-16T15:47:33.317 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.317 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-09-16T15:47:33.317 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "", 2024-09-16T15:47:33.317 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.317 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.317 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.317 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:33.317 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.317 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.317 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.318 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.318 INFO:teuthology.orchestra.run.smithi123.stdout: "GRAFANA_API_PASSWORD": { 2024-09-16T15:47:33.318 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "GRAFANA_API_PASSWORD", 2024-09-16T15:47:33.318 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-09-16T15:47:33.318 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.318 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-09-16T15:47:33.318 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "admin", 2024-09-16T15:47:33.318 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.318 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.318 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.318 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:33.318 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.318 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.319 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.319 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.319 INFO:teuthology.orchestra.run.smithi123.stdout: "GRAFANA_API_SSL_VERIFY": { 2024-09-16T15:47:33.319 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "GRAFANA_API_SSL_VERIFY", 2024-09-16T15:47:33.319 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-09-16T15:47:33.319 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.319 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-09-16T15:47:33.319 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "True", 2024-09-16T15:47:33.319 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.319 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.319 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.319 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:33.319 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.319 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.319 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.319 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.320 INFO:teuthology.orchestra.run.smithi123.stdout: "GRAFANA_API_URL": { 2024-09-16T15:47:33.320 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "GRAFANA_API_URL", 2024-09-16T15:47:33.320 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-09-16T15:47:33.320 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.320 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-09-16T15:47:33.320 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "", 2024-09-16T15:47:33.320 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.320 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.320 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.320 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:33.320 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.320 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.320 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.320 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.320 INFO:teuthology.orchestra.run.smithi123.stdout: "GRAFANA_API_USERNAME": { 2024-09-16T15:47:33.321 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "GRAFANA_API_USERNAME", 2024-09-16T15:47:33.321 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-09-16T15:47:33.321 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.321 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-09-16T15:47:33.321 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "admin", 2024-09-16T15:47:33.321 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.321 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.321 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.321 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:33.321 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.321 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.321 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.321 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.321 INFO:teuthology.orchestra.run.smithi123.stdout: "GRAFANA_FRONTEND_API_URL": { 2024-09-16T15:47:33.321 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "GRAFANA_FRONTEND_API_URL", 2024-09-16T15:47:33.322 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-09-16T15:47:33.322 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.322 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-09-16T15:47:33.322 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "", 2024-09-16T15:47:33.322 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.322 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.322 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.322 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:33.322 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.322 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.322 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.322 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.322 INFO:teuthology.orchestra.run.smithi123.stdout: "GRAFANA_UPDATE_DASHBOARDS": { 2024-09-16T15:47:33.322 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "GRAFANA_UPDATE_DASHBOARDS", 2024-09-16T15:47:33.322 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-09-16T15:47:33.322 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.323 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-09-16T15:47:33.323 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-09-16T15:47:33.323 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.323 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.323 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.323 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:33.323 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.323 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.323 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.323 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.323 INFO:teuthology.orchestra.run.smithi123.stdout: "ISCSI_API_SSL_VERIFICATION": { 2024-09-16T15:47:33.323 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "ISCSI_API_SSL_VERIFICATION", 2024-09-16T15:47:33.323 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-09-16T15:47:33.323 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.323 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-09-16T15:47:33.324 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "True", 2024-09-16T15:47:33.324 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.324 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.324 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.324 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:33.324 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.324 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.324 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.324 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.324 INFO:teuthology.orchestra.run.smithi123.stdout: "ISSUE_TRACKER_API_KEY": { 2024-09-16T15:47:33.324 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "ISSUE_TRACKER_API_KEY", 2024-09-16T15:47:33.324 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-09-16T15:47:33.324 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.324 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-09-16T15:47:33.324 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "", 2024-09-16T15:47:33.325 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.325 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.325 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.325 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:33.325 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.325 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.325 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.325 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.325 INFO:teuthology.orchestra.run.smithi123.stdout: "MANAGED_BY_CLUSTERS": { 2024-09-16T15:47:33.325 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "MANAGED_BY_CLUSTERS", 2024-09-16T15:47:33.325 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-09-16T15:47:33.325 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.325 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-09-16T15:47:33.325 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "[]", 2024-09-16T15:47:33.325 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.326 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.326 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.326 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:33.326 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.326 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.326 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.326 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.326 INFO:teuthology.orchestra.run.smithi123.stdout: "MULTICLUSTER_CONFIG": { 2024-09-16T15:47:33.326 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "MULTICLUSTER_CONFIG", 2024-09-16T15:47:33.326 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-09-16T15:47:33.326 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.326 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-09-16T15:47:33.326 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "{}", 2024-09-16T15:47:33.326 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.326 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.326 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.327 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:33.327 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.327 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.327 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.327 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.327 INFO:teuthology.orchestra.run.smithi123.stdout: "PROMETHEUS_API_HOST": { 2024-09-16T15:47:33.327 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "PROMETHEUS_API_HOST", 2024-09-16T15:47:33.327 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-09-16T15:47:33.327 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.327 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-09-16T15:47:33.327 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "", 2024-09-16T15:47:33.327 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.327 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.327 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.327 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:33.327 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.328 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.328 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.328 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.328 INFO:teuthology.orchestra.run.smithi123.stdout: "PROMETHEUS_API_SSL_VERIFY": { 2024-09-16T15:47:33.328 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "PROMETHEUS_API_SSL_VERIFY", 2024-09-16T15:47:33.328 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-09-16T15:47:33.328 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.328 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-09-16T15:47:33.328 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "True", 2024-09-16T15:47:33.328 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.328 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.328 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.328 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:33.328 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.328 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.329 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.329 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.329 INFO:teuthology.orchestra.run.smithi123.stdout: "PWD_POLICY_CHECK_COMPLEXITY_ENABLED": { 2024-09-16T15:47:33.329 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "PWD_POLICY_CHECK_COMPLEXITY_ENABLED", 2024-09-16T15:47:33.329 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-09-16T15:47:33.329 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.329 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-09-16T15:47:33.329 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-09-16T15:47:33.329 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.329 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.329 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.329 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:33.329 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.329 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.329 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.330 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.330 INFO:teuthology.orchestra.run.smithi123.stdout: "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED": { 2024-09-16T15:47:33.330 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED", 2024-09-16T15:47:33.330 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-09-16T15:47:33.330 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.330 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-09-16T15:47:33.330 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-09-16T15:47:33.330 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.330 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.330 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.330 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:33.330 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.330 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.330 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.330 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.331 INFO:teuthology.orchestra.run.smithi123.stdout: "PWD_POLICY_CHECK_LENGTH_ENABLED": { 2024-09-16T15:47:33.331 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "PWD_POLICY_CHECK_LENGTH_ENABLED", 2024-09-16T15:47:33.331 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-09-16T15:47:33.331 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.331 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-09-16T15:47:33.331 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "True", 2024-09-16T15:47:33.331 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.331 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.331 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.331 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:33.331 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.331 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.331 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.331 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.331 INFO:teuthology.orchestra.run.smithi123.stdout: "PWD_POLICY_CHECK_OLDPWD_ENABLED": { 2024-09-16T15:47:33.331 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "PWD_POLICY_CHECK_OLDPWD_ENABLED", 2024-09-16T15:47:33.332 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-09-16T15:47:33.332 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.332 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-09-16T15:47:33.332 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "True", 2024-09-16T15:47:33.332 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.332 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.332 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.332 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:33.332 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.332 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.332 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.332 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.332 INFO:teuthology.orchestra.run.smithi123.stdout: "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED": { 2024-09-16T15:47:33.332 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED", 2024-09-16T15:47:33.332 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-09-16T15:47:33.333 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.333 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-09-16T15:47:33.333 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-09-16T15:47:33.333 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.333 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.333 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.333 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:33.333 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.333 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.333 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.333 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.333 INFO:teuthology.orchestra.run.smithi123.stdout: "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED": { 2024-09-16T15:47:33.333 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED", 2024-09-16T15:47:33.333 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-09-16T15:47:33.333 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.333 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-09-16T15:47:33.334 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-09-16T15:47:33.334 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.334 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.334 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.334 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:33.334 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.334 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.334 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.334 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.334 INFO:teuthology.orchestra.run.smithi123.stdout: "PWD_POLICY_CHECK_USERNAME_ENABLED": { 2024-09-16T15:47:33.334 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "PWD_POLICY_CHECK_USERNAME_ENABLED", 2024-09-16T15:47:33.334 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-09-16T15:47:33.334 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.334 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-09-16T15:47:33.334 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-09-16T15:47:33.335 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.335 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.335 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.335 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:33.335 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.335 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.335 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.335 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.335 INFO:teuthology.orchestra.run.smithi123.stdout: "PWD_POLICY_ENABLED": { 2024-09-16T15:47:33.335 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "PWD_POLICY_ENABLED", 2024-09-16T15:47:33.335 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-09-16T15:47:33.335 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.335 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-09-16T15:47:33.336 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "True", 2024-09-16T15:47:33.336 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.336 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.336 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.336 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:33.336 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.336 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.336 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.336 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.336 INFO:teuthology.orchestra.run.smithi123.stdout: "PWD_POLICY_EXCLUSION_LIST": { 2024-09-16T15:47:33.336 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "PWD_POLICY_EXCLUSION_LIST", 2024-09-16T15:47:33.336 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-09-16T15:47:33.336 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.336 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-09-16T15:47:33.336 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps", 2024-09-16T15:47:33.337 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.337 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.337 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.337 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:33.337 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.337 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.337 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.337 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.337 INFO:teuthology.orchestra.run.smithi123.stdout: "PWD_POLICY_MIN_COMPLEXITY": { 2024-09-16T15:47:33.337 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "PWD_POLICY_MIN_COMPLEXITY", 2024-09-16T15:47:33.337 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "int", 2024-09-16T15:47:33.337 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.337 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-09-16T15:47:33.337 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "10", 2024-09-16T15:47:33.337 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.337 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.338 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.338 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:33.338 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.338 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.338 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.338 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.338 INFO:teuthology.orchestra.run.smithi123.stdout: "PWD_POLICY_MIN_LENGTH": { 2024-09-16T15:47:33.338 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "PWD_POLICY_MIN_LENGTH", 2024-09-16T15:47:33.338 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "int", 2024-09-16T15:47:33.338 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.338 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-09-16T15:47:33.338 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "8", 2024-09-16T15:47:33.338 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.338 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.339 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.339 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:33.339 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.339 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.339 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.339 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.339 INFO:teuthology.orchestra.run.smithi123.stdout: "REST_REQUESTS_TIMEOUT": { 2024-09-16T15:47:33.339 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "REST_REQUESTS_TIMEOUT", 2024-09-16T15:47:33.339 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "int", 2024-09-16T15:47:33.339 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.339 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-09-16T15:47:33.339 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "45", 2024-09-16T15:47:33.339 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.339 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.339 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.340 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:33.340 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.340 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.340 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.340 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.340 INFO:teuthology.orchestra.run.smithi123.stdout: "RGW_API_ACCESS_KEY": { 2024-09-16T15:47:33.340 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "RGW_API_ACCESS_KEY", 2024-09-16T15:47:33.340 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-09-16T15:47:33.340 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.340 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-09-16T15:47:33.340 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "", 2024-09-16T15:47:33.340 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.340 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.340 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.340 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:33.341 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.341 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.341 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.341 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.341 INFO:teuthology.orchestra.run.smithi123.stdout: "RGW_API_ADMIN_RESOURCE": { 2024-09-16T15:47:33.341 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "RGW_API_ADMIN_RESOURCE", 2024-09-16T15:47:33.341 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-09-16T15:47:33.341 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.341 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-09-16T15:47:33.341 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "admin", 2024-09-16T15:47:33.341 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.341 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.341 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.341 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:33.341 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.342 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.342 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.342 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.342 INFO:teuthology.orchestra.run.smithi123.stdout: "RGW_API_SECRET_KEY": { 2024-09-16T15:47:33.342 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "RGW_API_SECRET_KEY", 2024-09-16T15:47:33.342 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-09-16T15:47:33.342 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.342 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-09-16T15:47:33.342 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "", 2024-09-16T15:47:33.342 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.342 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.342 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.342 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:33.342 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.343 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.343 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.343 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.343 INFO:teuthology.orchestra.run.smithi123.stdout: "RGW_API_SSL_VERIFY": { 2024-09-16T15:47:33.343 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "RGW_API_SSL_VERIFY", 2024-09-16T15:47:33.343 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-09-16T15:47:33.343 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.343 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-09-16T15:47:33.343 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "True", 2024-09-16T15:47:33.343 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.343 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.343 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.344 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:33.344 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.344 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.344 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.344 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.344 INFO:teuthology.orchestra.run.smithi123.stdout: "UNSAFE_TLS_v1_2": { 2024-09-16T15:47:33.344 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "UNSAFE_TLS_v1_2", 2024-09-16T15:47:33.344 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-09-16T15:47:33.344 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.344 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-09-16T15:47:33.344 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-09-16T15:47:33.344 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.344 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.344 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.344 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:33.345 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.345 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.345 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.345 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.345 INFO:teuthology.orchestra.run.smithi123.stdout: "USER_PWD_EXPIRATION_SPAN": { 2024-09-16T15:47:33.345 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "USER_PWD_EXPIRATION_SPAN", 2024-09-16T15:47:33.345 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "int", 2024-09-16T15:47:33.345 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.345 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-09-16T15:47:33.345 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "0", 2024-09-16T15:47:33.345 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.345 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.346 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.346 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:33.346 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.346 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.346 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.346 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.346 INFO:teuthology.orchestra.run.smithi123.stdout: "USER_PWD_EXPIRATION_WARNING_1": { 2024-09-16T15:47:33.346 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "USER_PWD_EXPIRATION_WARNING_1", 2024-09-16T15:47:33.346 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "int", 2024-09-16T15:47:33.346 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.346 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-09-16T15:47:33.346 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "10", 2024-09-16T15:47:33.346 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.347 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.347 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.347 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:33.347 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.347 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.347 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.347 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.347 INFO:teuthology.orchestra.run.smithi123.stdout: "USER_PWD_EXPIRATION_WARNING_2": { 2024-09-16T15:47:33.347 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "USER_PWD_EXPIRATION_WARNING_2", 2024-09-16T15:47:33.347 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "int", 2024-09-16T15:47:33.347 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.347 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-09-16T15:47:33.347 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "5", 2024-09-16T15:47:33.347 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.348 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.348 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.348 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:33.348 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.348 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.348 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.348 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.348 INFO:teuthology.orchestra.run.smithi123.stdout: "cross_origin_url": { 2024-09-16T15:47:33.348 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "cross_origin_url", 2024-09-16T15:47:33.348 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-09-16T15:47:33.348 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.348 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-09-16T15:47:33.348 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "", 2024-09-16T15:47:33.348 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.348 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.349 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.349 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:33.349 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.349 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.349 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.349 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.349 INFO:teuthology.orchestra.run.smithi123.stdout: "crt_file": { 2024-09-16T15:47:33.349 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "crt_file", 2024-09-16T15:47:33.349 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-09-16T15:47:33.349 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.349 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-09-16T15:47:33.349 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "", 2024-09-16T15:47:33.350 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.350 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.350 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.350 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:33.350 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.350 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.350 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.350 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.350 INFO:teuthology.orchestra.run.smithi123.stdout: "debug": { 2024-09-16T15:47:33.350 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "debug", 2024-09-16T15:47:33.350 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-09-16T15:47:33.350 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.350 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-09-16T15:47:33.350 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-09-16T15:47:33.351 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.351 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.351 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.351 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "Enable/disable debug options", 2024-09-16T15:47:33.351 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.351 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.351 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.351 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.351 INFO:teuthology.orchestra.run.smithi123.stdout: "jwt_token_ttl": { 2024-09-16T15:47:33.351 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "jwt_token_ttl", 2024-09-16T15:47:33.351 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "int", 2024-09-16T15:47:33.351 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.351 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-09-16T15:47:33.351 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "28800", 2024-09-16T15:47:33.351 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.352 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.352 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.352 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:33.352 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.352 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.352 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.352 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.352 INFO:teuthology.orchestra.run.smithi123.stdout: "key_file": { 2024-09-16T15:47:33.352 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "key_file", 2024-09-16T15:47:33.352 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-09-16T15:47:33.352 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.352 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-09-16T15:47:33.352 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "", 2024-09-16T15:47:33.352 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.352 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.353 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.353 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:33.353 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.353 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.353 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.353 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.353 INFO:teuthology.orchestra.run.smithi123.stdout: "log_level": { 2024-09-16T15:47:33.353 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_level", 2024-09-16T15:47:33.353 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-09-16T15:47:33.353 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.353 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-09-16T15:47:33.353 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "", 2024-09-16T15:47:33.353 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.353 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.353 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [ 2024-09-16T15:47:33.353 INFO:teuthology.orchestra.run.smithi123.stdout: "", 2024-09-16T15:47:33.354 INFO:teuthology.orchestra.run.smithi123.stdout: "critical", 2024-09-16T15:47:33.354 INFO:teuthology.orchestra.run.smithi123.stdout: "debug", 2024-09-16T15:47:33.354 INFO:teuthology.orchestra.run.smithi123.stdout: "error", 2024-09-16T15:47:33.354 INFO:teuthology.orchestra.run.smithi123.stdout: "info", 2024-09-16T15:47:33.354 INFO:teuthology.orchestra.run.smithi123.stdout: "warning" 2024-09-16T15:47:33.354 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-09-16T15:47:33.354 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:33.354 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.354 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.354 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.354 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.354 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_cluster": { 2024-09-16T15:47:33.354 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_cluster", 2024-09-16T15:47:33.354 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-09-16T15:47:33.354 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.355 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-09-16T15:47:33.355 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-09-16T15:47:33.355 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.355 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.355 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.355 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:33.355 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.355 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.355 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.355 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.355 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_cluster_level": { 2024-09-16T15:47:33.355 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_cluster_level", 2024-09-16T15:47:33.355 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-09-16T15:47:33.355 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.355 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-09-16T15:47:33.356 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "info", 2024-09-16T15:47:33.356 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.356 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.356 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [ 2024-09-16T15:47:33.356 INFO:teuthology.orchestra.run.smithi123.stdout: "", 2024-09-16T15:47:33.356 INFO:teuthology.orchestra.run.smithi123.stdout: "critical", 2024-09-16T15:47:33.356 INFO:teuthology.orchestra.run.smithi123.stdout: "debug", 2024-09-16T15:47:33.356 INFO:teuthology.orchestra.run.smithi123.stdout: "error", 2024-09-16T15:47:33.356 INFO:teuthology.orchestra.run.smithi123.stdout: "info", 2024-09-16T15:47:33.356 INFO:teuthology.orchestra.run.smithi123.stdout: "warning" 2024-09-16T15:47:33.356 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-09-16T15:47:33.356 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:33.356 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.356 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.356 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.357 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.357 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_file": { 2024-09-16T15:47:33.357 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_file", 2024-09-16T15:47:33.357 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-09-16T15:47:33.357 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.357 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-09-16T15:47:33.357 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-09-16T15:47:33.357 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.357 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.357 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.357 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:33.357 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.357 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.357 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.357 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.358 INFO:teuthology.orchestra.run.smithi123.stdout: "motd": { 2024-09-16T15:47:33.358 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "motd", 2024-09-16T15:47:33.358 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-09-16T15:47:33.358 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.358 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-09-16T15:47:33.358 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "", 2024-09-16T15:47:33.358 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.358 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.358 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.358 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "The message of the day", 2024-09-16T15:47:33.358 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.359 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.359 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.359 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.359 INFO:teuthology.orchestra.run.smithi123.stdout: "redirect_resolve_ip_addr": { 2024-09-16T15:47:33.359 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "redirect_resolve_ip_addr", 2024-09-16T15:47:33.359 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-09-16T15:47:33.359 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.359 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-09-16T15:47:33.359 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-09-16T15:47:33.359 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.359 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.359 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.359 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:33.359 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.360 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.360 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.360 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.360 INFO:teuthology.orchestra.run.smithi123.stdout: "server_addr": { 2024-09-16T15:47:33.360 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "server_addr", 2024-09-16T15:47:33.360 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-09-16T15:47:33.360 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.360 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-09-16T15:47:33.360 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "::", 2024-09-16T15:47:33.360 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.360 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.360 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.360 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:33.360 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.361 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.361 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.361 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.361 INFO:teuthology.orchestra.run.smithi123.stdout: "server_port": { 2024-09-16T15:47:33.361 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "server_port", 2024-09-16T15:47:33.361 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "int", 2024-09-16T15:47:33.361 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.361 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-09-16T15:47:33.361 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "8080", 2024-09-16T15:47:33.361 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.361 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.361 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.361 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:33.361 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.361 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.362 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.362 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.362 INFO:teuthology.orchestra.run.smithi123.stdout: "sqlite3_killpoint": { 2024-09-16T15:47:33.362 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "sqlite3_killpoint", 2024-09-16T15:47:33.362 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "int", 2024-09-16T15:47:33.362 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "dev", 2024-09-16T15:47:33.362 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-09-16T15:47:33.362 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "0", 2024-09-16T15:47:33.362 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.362 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.362 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.362 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:33.362 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.362 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.362 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.362 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.363 INFO:teuthology.orchestra.run.smithi123.stdout: "ssl": { 2024-09-16T15:47:33.363 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "ssl", 2024-09-16T15:47:33.363 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-09-16T15:47:33.363 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.363 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-09-16T15:47:33.363 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "True", 2024-09-16T15:47:33.363 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.363 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.363 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.363 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:33.363 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.363 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.363 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.363 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.363 INFO:teuthology.orchestra.run.smithi123.stdout: "ssl_server_port": { 2024-09-16T15:47:33.364 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "ssl_server_port", 2024-09-16T15:47:33.364 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "int", 2024-09-16T15:47:33.364 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.364 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-09-16T15:47:33.364 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "8443", 2024-09-16T15:47:33.364 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.364 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.364 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.364 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:33.364 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.364 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.364 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.364 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.364 INFO:teuthology.orchestra.run.smithi123.stdout: "standby_behaviour": { 2024-09-16T15:47:33.364 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "standby_behaviour", 2024-09-16T15:47:33.365 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-09-16T15:47:33.365 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.365 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-09-16T15:47:33.365 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "redirect", 2024-09-16T15:47:33.365 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.365 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.365 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [ 2024-09-16T15:47:33.365 INFO:teuthology.orchestra.run.smithi123.stdout: "error", 2024-09-16T15:47:33.365 INFO:teuthology.orchestra.run.smithi123.stdout: "redirect" 2024-09-16T15:47:33.365 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-09-16T15:47:33.365 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:33.365 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.365 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.365 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.366 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.366 INFO:teuthology.orchestra.run.smithi123.stdout: "standby_error_status_code": { 2024-09-16T15:47:33.366 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "standby_error_status_code", 2024-09-16T15:47:33.366 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "int", 2024-09-16T15:47:33.366 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.366 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-09-16T15:47:33.366 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "500", 2024-09-16T15:47:33.366 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "400", 2024-09-16T15:47:33.366 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "599", 2024-09-16T15:47:33.366 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.366 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:33.366 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.366 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.366 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.366 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.367 INFO:teuthology.orchestra.run.smithi123.stdout: "url_prefix": { 2024-09-16T15:47:33.367 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "url_prefix", 2024-09-16T15:47:33.367 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-09-16T15:47:33.367 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.367 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-09-16T15:47:33.367 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "", 2024-09-16T15:47:33.367 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.367 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.367 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.367 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:33.367 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.367 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.367 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.367 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-16T15:47:33.367 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-16T15:47:33.368 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.368 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-09-16T15:47:33.368 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "devicehealth", 2024-09-16T15:47:33.368 INFO:teuthology.orchestra.run.smithi123.stdout: "can_run": true, 2024-09-16T15:47:33.368 INFO:teuthology.orchestra.run.smithi123.stdout: "error_string": "", 2024-09-16T15:47:33.368 INFO:teuthology.orchestra.run.smithi123.stdout: "module_options": { 2024-09-16T15:47:33.368 INFO:teuthology.orchestra.run.smithi123.stdout: "enable_monitoring": { 2024-09-16T15:47:33.368 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "enable_monitoring", 2024-09-16T15:47:33.368 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-09-16T15:47:33.368 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.368 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-09-16T15:47:33.368 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "True", 2024-09-16T15:47:33.368 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.368 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.368 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.369 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "monitor device health metrics", 2024-09-16T15:47:33.369 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.369 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.369 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.369 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.369 INFO:teuthology.orchestra.run.smithi123.stdout: "log_level": { 2024-09-16T15:47:33.369 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_level", 2024-09-16T15:47:33.369 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-09-16T15:47:33.369 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.369 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-09-16T15:47:33.369 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "", 2024-09-16T15:47:33.369 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.369 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.369 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [ 2024-09-16T15:47:33.369 INFO:teuthology.orchestra.run.smithi123.stdout: "", 2024-09-16T15:47:33.370 INFO:teuthology.orchestra.run.smithi123.stdout: "critical", 2024-09-16T15:47:33.370 INFO:teuthology.orchestra.run.smithi123.stdout: "debug", 2024-09-16T15:47:33.370 INFO:teuthology.orchestra.run.smithi123.stdout: "error", 2024-09-16T15:47:33.370 INFO:teuthology.orchestra.run.smithi123.stdout: "info", 2024-09-16T15:47:33.370 INFO:teuthology.orchestra.run.smithi123.stdout: "warning" 2024-09-16T15:47:33.370 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-09-16T15:47:33.370 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:33.370 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.370 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.370 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.370 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.370 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_cluster": { 2024-09-16T15:47:33.370 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_cluster", 2024-09-16T15:47:33.370 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-09-16T15:47:33.371 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.371 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-09-16T15:47:33.371 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-09-16T15:47:33.371 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.371 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.371 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.371 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:33.371 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.371 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.371 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.371 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.372 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_cluster_level": { 2024-09-16T15:47:33.372 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_cluster_level", 2024-09-16T15:47:33.372 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-09-16T15:47:33.372 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.372 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-09-16T15:47:33.372 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "info", 2024-09-16T15:47:33.372 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.372 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.372 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [ 2024-09-16T15:47:33.372 INFO:teuthology.orchestra.run.smithi123.stdout: "", 2024-09-16T15:47:33.372 INFO:teuthology.orchestra.run.smithi123.stdout: "critical", 2024-09-16T15:47:33.372 INFO:teuthology.orchestra.run.smithi123.stdout: "debug", 2024-09-16T15:47:33.372 INFO:teuthology.orchestra.run.smithi123.stdout: "error", 2024-09-16T15:47:33.372 INFO:teuthology.orchestra.run.smithi123.stdout: "info", 2024-09-16T15:47:33.373 INFO:teuthology.orchestra.run.smithi123.stdout: "warning" 2024-09-16T15:47:33.373 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-09-16T15:47:33.373 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:33.373 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.373 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.373 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.373 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.373 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_file": { 2024-09-16T15:47:33.373 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_file", 2024-09-16T15:47:33.373 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-09-16T15:47:33.373 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.373 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-09-16T15:47:33.373 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-09-16T15:47:33.373 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.373 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.373 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.374 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:33.374 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.374 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.374 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.374 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.374 INFO:teuthology.orchestra.run.smithi123.stdout: "mark_out_threshold": { 2024-09-16T15:47:33.374 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "mark_out_threshold", 2024-09-16T15:47:33.374 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "secs", 2024-09-16T15:47:33.374 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.374 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-09-16T15:47:33.374 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "2419200", 2024-09-16T15:47:33.374 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.374 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.374 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.374 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "automatically mark OSD if it may fail before this long", 2024-09-16T15:47:33.375 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.375 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.375 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.375 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.375 INFO:teuthology.orchestra.run.smithi123.stdout: "pool_name": { 2024-09-16T15:47:33.375 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "pool_name", 2024-09-16T15:47:33.375 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-09-16T15:47:33.375 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.375 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-09-16T15:47:33.375 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "device_health_metrics", 2024-09-16T15:47:33.375 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.375 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.375 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.375 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "name of pool in which to store device health metrics", 2024-09-16T15:47:33.376 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.376 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.376 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.376 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.376 INFO:teuthology.orchestra.run.smithi123.stdout: "retention_period": { 2024-09-16T15:47:33.376 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "retention_period", 2024-09-16T15:47:33.376 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "secs", 2024-09-16T15:47:33.376 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.376 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-09-16T15:47:33.376 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "15552000", 2024-09-16T15:47:33.376 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.376 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.376 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.376 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "how long to retain device health metrics", 2024-09-16T15:47:33.377 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.377 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.377 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.377 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.377 INFO:teuthology.orchestra.run.smithi123.stdout: "scrape_frequency": { 2024-09-16T15:47:33.377 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "scrape_frequency", 2024-09-16T15:47:33.377 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "secs", 2024-09-16T15:47:33.377 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.377 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-09-16T15:47:33.377 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "86400", 2024-09-16T15:47:33.377 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.377 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.377 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.377 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "how frequently to scrape device health metrics", 2024-09-16T15:47:33.377 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.378 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.378 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.378 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.378 INFO:teuthology.orchestra.run.smithi123.stdout: "self_heal": { 2024-09-16T15:47:33.378 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "self_heal", 2024-09-16T15:47:33.378 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-09-16T15:47:33.378 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.378 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-09-16T15:47:33.378 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "True", 2024-09-16T15:47:33.378 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.378 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.378 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.378 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "preemptively heal cluster around devices that may fail", 2024-09-16T15:47:33.378 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.378 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.379 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.379 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.379 INFO:teuthology.orchestra.run.smithi123.stdout: "sleep_interval": { 2024-09-16T15:47:33.379 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "sleep_interval", 2024-09-16T15:47:33.379 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "secs", 2024-09-16T15:47:33.379 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.379 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-09-16T15:47:33.379 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "600", 2024-09-16T15:47:33.379 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.379 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.379 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.379 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "how frequently to wake up and check device health", 2024-09-16T15:47:33.379 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.379 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.379 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.380 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.380 INFO:teuthology.orchestra.run.smithi123.stdout: "sqlite3_killpoint": { 2024-09-16T15:47:33.380 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "sqlite3_killpoint", 2024-09-16T15:47:33.380 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "int", 2024-09-16T15:47:33.380 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "dev", 2024-09-16T15:47:33.380 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-09-16T15:47:33.380 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "0", 2024-09-16T15:47:33.380 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.380 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.380 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.380 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:33.380 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.380 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.380 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.380 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.381 INFO:teuthology.orchestra.run.smithi123.stdout: "warn_threshold": { 2024-09-16T15:47:33.381 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "warn_threshold", 2024-09-16T15:47:33.381 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "secs", 2024-09-16T15:47:33.381 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.381 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-09-16T15:47:33.381 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "7257600", 2024-09-16T15:47:33.381 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.381 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.381 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.381 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "raise health warning if OSD may fail before this long", 2024-09-16T15:47:33.381 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.381 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.381 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.381 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-16T15:47:33.381 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-16T15:47:33.382 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.382 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-09-16T15:47:33.382 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "diskprediction_local", 2024-09-16T15:47:33.382 INFO:teuthology.orchestra.run.smithi123.stdout: "can_run": true, 2024-09-16T15:47:33.382 INFO:teuthology.orchestra.run.smithi123.stdout: "error_string": "", 2024-09-16T15:47:33.382 INFO:teuthology.orchestra.run.smithi123.stdout: "module_options": { 2024-09-16T15:47:33.382 INFO:teuthology.orchestra.run.smithi123.stdout: "log_level": { 2024-09-16T15:47:33.382 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_level", 2024-09-16T15:47:33.382 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-09-16T15:47:33.382 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.382 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-09-16T15:47:33.382 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "", 2024-09-16T15:47:33.382 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.382 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.382 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [ 2024-09-16T15:47:33.383 INFO:teuthology.orchestra.run.smithi123.stdout: "", 2024-09-16T15:47:33.383 INFO:teuthology.orchestra.run.smithi123.stdout: "critical", 2024-09-16T15:47:33.383 INFO:teuthology.orchestra.run.smithi123.stdout: "debug", 2024-09-16T15:47:33.383 INFO:teuthology.orchestra.run.smithi123.stdout: "error", 2024-09-16T15:47:33.383 INFO:teuthology.orchestra.run.smithi123.stdout: "info", 2024-09-16T15:47:33.383 INFO:teuthology.orchestra.run.smithi123.stdout: "warning" 2024-09-16T15:47:33.383 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-09-16T15:47:33.383 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:33.383 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.383 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.383 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.383 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.383 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_cluster": { 2024-09-16T15:47:33.383 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_cluster", 2024-09-16T15:47:33.384 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-09-16T15:47:33.384 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.384 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-09-16T15:47:33.384 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-09-16T15:47:33.384 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.384 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.384 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.384 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:33.384 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.384 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.384 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.384 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.384 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_cluster_level": { 2024-09-16T15:47:33.384 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_cluster_level", 2024-09-16T15:47:33.384 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-09-16T15:47:33.385 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.385 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-09-16T15:47:33.385 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "info", 2024-09-16T15:47:33.385 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.385 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.385 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [ 2024-09-16T15:47:33.385 INFO:teuthology.orchestra.run.smithi123.stdout: "", 2024-09-16T15:47:33.385 INFO:teuthology.orchestra.run.smithi123.stdout: "critical", 2024-09-16T15:47:33.385 INFO:teuthology.orchestra.run.smithi123.stdout: "debug", 2024-09-16T15:47:33.385 INFO:teuthology.orchestra.run.smithi123.stdout: "error", 2024-09-16T15:47:33.385 INFO:teuthology.orchestra.run.smithi123.stdout: "info", 2024-09-16T15:47:33.385 INFO:teuthology.orchestra.run.smithi123.stdout: "warning" 2024-09-16T15:47:33.386 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-09-16T15:47:33.386 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:33.386 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.386 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.386 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.386 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.386 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_file": { 2024-09-16T15:47:33.386 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_file", 2024-09-16T15:47:33.386 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-09-16T15:47:33.386 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.386 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-09-16T15:47:33.387 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-09-16T15:47:33.387 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.387 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.387 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.387 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:33.387 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.387 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.387 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.387 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.387 INFO:teuthology.orchestra.run.smithi123.stdout: "predict_interval": { 2024-09-16T15:47:33.387 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "predict_interval", 2024-09-16T15:47:33.387 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-09-16T15:47:33.387 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.387 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-09-16T15:47:33.387 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "86400", 2024-09-16T15:47:33.387 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.388 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.388 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.388 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:33.388 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.388 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.388 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.388 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.388 INFO:teuthology.orchestra.run.smithi123.stdout: "predictor_model": { 2024-09-16T15:47:33.388 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "predictor_model", 2024-09-16T15:47:33.388 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-09-16T15:47:33.388 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.388 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-09-16T15:47:33.388 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "prophetstor", 2024-09-16T15:47:33.389 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.389 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.389 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.389 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:33.389 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.389 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.389 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.389 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.389 INFO:teuthology.orchestra.run.smithi123.stdout: "sleep_interval": { 2024-09-16T15:47:33.389 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "sleep_interval", 2024-09-16T15:47:33.389 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-09-16T15:47:33.389 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.389 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-09-16T15:47:33.390 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "600", 2024-09-16T15:47:33.390 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.390 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.390 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.390 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:33.390 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.390 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.390 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.390 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.390 INFO:teuthology.orchestra.run.smithi123.stdout: "sqlite3_killpoint": { 2024-09-16T15:47:33.390 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "sqlite3_killpoint", 2024-09-16T15:47:33.390 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "int", 2024-09-16T15:47:33.390 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "dev", 2024-09-16T15:47:33.390 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-09-16T15:47:33.390 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "0", 2024-09-16T15:47:33.391 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.391 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.391 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.391 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:33.391 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.391 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.391 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.391 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-16T15:47:33.391 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-16T15:47:33.391 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.391 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-09-16T15:47:33.391 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "influx", 2024-09-16T15:47:33.391 INFO:teuthology.orchestra.run.smithi123.stdout: "can_run": false, 2024-09-16T15:47:33.391 INFO:teuthology.orchestra.run.smithi123.stdout: "error_string": "influxdb python module not found", 2024-09-16T15:47:33.391 INFO:teuthology.orchestra.run.smithi123.stdout: "module_options": { 2024-09-16T15:47:33.391 INFO:teuthology.orchestra.run.smithi123.stdout: "batch_size": { 2024-09-16T15:47:33.392 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "batch_size", 2024-09-16T15:47:33.392 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "int", 2024-09-16T15:47:33.392 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.392 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-09-16T15:47:33.392 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "5000", 2024-09-16T15:47:33.392 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.392 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.392 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.392 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "How big batches of data points should be when sending to InfluxDB.", 2024-09-16T15:47:33.392 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.392 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.392 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.392 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.392 INFO:teuthology.orchestra.run.smithi123.stdout: "database": { 2024-09-16T15:47:33.393 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "database", 2024-09-16T15:47:33.393 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-09-16T15:47:33.393 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.393 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-09-16T15:47:33.393 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "ceph", 2024-09-16T15:47:33.393 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.393 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.393 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.393 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.", 2024-09-16T15:47:33.393 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.393 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.393 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.393 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.393 INFO:teuthology.orchestra.run.smithi123.stdout: "hostname": { 2024-09-16T15:47:33.394 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "hostname", 2024-09-16T15:47:33.394 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-09-16T15:47:33.394 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.394 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-09-16T15:47:33.394 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "", 2024-09-16T15:47:33.394 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.394 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.394 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.394 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "InfluxDB server hostname", 2024-09-16T15:47:33.394 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.394 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.394 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.394 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.394 INFO:teuthology.orchestra.run.smithi123.stdout: "interval": { 2024-09-16T15:47:33.394 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "interval", 2024-09-16T15:47:33.394 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "secs", 2024-09-16T15:47:33.395 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.395 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-09-16T15:47:33.395 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "30", 2024-09-16T15:47:33.395 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "5", 2024-09-16T15:47:33.395 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.395 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.395 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "Time between reports to InfluxDB. Default 30 seconds.", 2024-09-16T15:47:33.395 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.395 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.395 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.395 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.395 INFO:teuthology.orchestra.run.smithi123.stdout: "log_level": { 2024-09-16T15:47:33.395 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_level", 2024-09-16T15:47:33.396 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-09-16T15:47:33.396 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.396 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-09-16T15:47:33.396 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "", 2024-09-16T15:47:33.396 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.396 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.396 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [ 2024-09-16T15:47:33.396 INFO:teuthology.orchestra.run.smithi123.stdout: "", 2024-09-16T15:47:33.396 INFO:teuthology.orchestra.run.smithi123.stdout: "critical", 2024-09-16T15:47:33.396 INFO:teuthology.orchestra.run.smithi123.stdout: "debug", 2024-09-16T15:47:33.396 INFO:teuthology.orchestra.run.smithi123.stdout: "error", 2024-09-16T15:47:33.396 INFO:teuthology.orchestra.run.smithi123.stdout: "info", 2024-09-16T15:47:33.396 INFO:teuthology.orchestra.run.smithi123.stdout: "warning" 2024-09-16T15:47:33.397 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-09-16T15:47:33.397 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:33.397 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.397 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.397 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.397 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.397 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_cluster": { 2024-09-16T15:47:33.397 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_cluster", 2024-09-16T15:47:33.397 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-09-16T15:47:33.397 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.397 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-09-16T15:47:33.397 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-09-16T15:47:33.397 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.397 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.398 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.398 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:33.398 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.398 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.398 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.398 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.398 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_cluster_level": { 2024-09-16T15:47:33.398 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_cluster_level", 2024-09-16T15:47:33.398 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-09-16T15:47:33.398 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.398 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-09-16T15:47:33.398 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "info", 2024-09-16T15:47:33.398 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.398 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.398 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [ 2024-09-16T15:47:33.399 INFO:teuthology.orchestra.run.smithi123.stdout: "", 2024-09-16T15:47:33.399 INFO:teuthology.orchestra.run.smithi123.stdout: "critical", 2024-09-16T15:47:33.399 INFO:teuthology.orchestra.run.smithi123.stdout: "debug", 2024-09-16T15:47:33.399 INFO:teuthology.orchestra.run.smithi123.stdout: "error", 2024-09-16T15:47:33.399 INFO:teuthology.orchestra.run.smithi123.stdout: "info", 2024-09-16T15:47:33.399 INFO:teuthology.orchestra.run.smithi123.stdout: "warning" 2024-09-16T15:47:33.399 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-09-16T15:47:33.399 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:33.399 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.399 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.399 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.399 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.399 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_file": { 2024-09-16T15:47:33.399 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_file", 2024-09-16T15:47:33.399 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-09-16T15:47:33.400 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.400 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-09-16T15:47:33.400 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-09-16T15:47:33.400 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.400 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.400 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.400 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:33.400 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.400 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.400 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.401 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.401 INFO:teuthology.orchestra.run.smithi123.stdout: "password": { 2024-09-16T15:47:33.401 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "password", 2024-09-16T15:47:33.401 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-09-16T15:47:33.401 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.401 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-09-16T15:47:33.401 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "", 2024-09-16T15:47:33.401 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.401 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.401 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.401 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "password of InfluxDB server user", 2024-09-16T15:47:33.401 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.401 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.402 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.402 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.402 INFO:teuthology.orchestra.run.smithi123.stdout: "port": { 2024-09-16T15:47:33.402 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "port", 2024-09-16T15:47:33.402 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "int", 2024-09-16T15:47:33.402 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.402 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-09-16T15:47:33.402 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "8086", 2024-09-16T15:47:33.402 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.402 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.402 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.402 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "InfluxDB server port", 2024-09-16T15:47:33.403 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.403 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.403 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.403 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.403 INFO:teuthology.orchestra.run.smithi123.stdout: "sqlite3_killpoint": { 2024-09-16T15:47:33.403 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "sqlite3_killpoint", 2024-09-16T15:47:33.403 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "int", 2024-09-16T15:47:33.403 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "dev", 2024-09-16T15:47:33.403 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-09-16T15:47:33.403 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "0", 2024-09-16T15:47:33.403 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.403 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.403 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.404 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:33.404 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.404 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.404 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.404 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.404 INFO:teuthology.orchestra.run.smithi123.stdout: "ssl": { 2024-09-16T15:47:33.404 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "ssl", 2024-09-16T15:47:33.404 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-09-16T15:47:33.404 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.404 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-09-16T15:47:33.404 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "false", 2024-09-16T15:47:33.404 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.404 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.404 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.404 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "Use https connection for InfluxDB server. Use \"true\" or \"false\".", 2024-09-16T15:47:33.405 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.405 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.405 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.405 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.405 INFO:teuthology.orchestra.run.smithi123.stdout: "threads": { 2024-09-16T15:47:33.405 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "threads", 2024-09-16T15:47:33.405 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "int", 2024-09-16T15:47:33.405 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.405 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-09-16T15:47:33.405 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "5", 2024-09-16T15:47:33.405 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "1", 2024-09-16T15:47:33.405 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "32", 2024-09-16T15:47:33.405 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.405 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "How many worker threads should be spawned for sending data to InfluxDB.", 2024-09-16T15:47:33.406 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.406 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.406 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.406 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.406 INFO:teuthology.orchestra.run.smithi123.stdout: "username": { 2024-09-16T15:47:33.406 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "username", 2024-09-16T15:47:33.406 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-09-16T15:47:33.406 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.406 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-09-16T15:47:33.406 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "", 2024-09-16T15:47:33.406 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.406 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.406 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.406 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "username of InfluxDB server user", 2024-09-16T15:47:33.406 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.406 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.407 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.407 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.407 INFO:teuthology.orchestra.run.smithi123.stdout: "verify_ssl": { 2024-09-16T15:47:33.407 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "verify_ssl", 2024-09-16T15:47:33.407 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-09-16T15:47:33.407 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.407 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-09-16T15:47:33.407 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "true", 2024-09-16T15:47:33.407 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.407 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.407 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.407 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "Verify https cert for InfluxDB server. Use \"true\" or \"false\".", 2024-09-16T15:47:33.407 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.407 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.407 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.407 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-16T15:47:33.408 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-16T15:47:33.408 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.408 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-09-16T15:47:33.408 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "insights", 2024-09-16T15:47:33.408 INFO:teuthology.orchestra.run.smithi123.stdout: "can_run": true, 2024-09-16T15:47:33.408 INFO:teuthology.orchestra.run.smithi123.stdout: "error_string": "", 2024-09-16T15:47:33.408 INFO:teuthology.orchestra.run.smithi123.stdout: "module_options": { 2024-09-16T15:47:33.408 INFO:teuthology.orchestra.run.smithi123.stdout: "log_level": { 2024-09-16T15:47:33.408 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_level", 2024-09-16T15:47:33.408 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-09-16T15:47:33.408 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.408 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-09-16T15:47:33.408 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "", 2024-09-16T15:47:33.408 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.409 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.409 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [ 2024-09-16T15:47:33.409 INFO:teuthology.orchestra.run.smithi123.stdout: "", 2024-09-16T15:47:33.409 INFO:teuthology.orchestra.run.smithi123.stdout: "critical", 2024-09-16T15:47:33.409 INFO:teuthology.orchestra.run.smithi123.stdout: "debug", 2024-09-16T15:47:33.409 INFO:teuthology.orchestra.run.smithi123.stdout: "error", 2024-09-16T15:47:33.409 INFO:teuthology.orchestra.run.smithi123.stdout: "info", 2024-09-16T15:47:33.409 INFO:teuthology.orchestra.run.smithi123.stdout: "warning" 2024-09-16T15:47:33.409 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-09-16T15:47:33.409 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:33.409 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.409 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.410 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.410 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.410 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_cluster": { 2024-09-16T15:47:33.410 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_cluster", 2024-09-16T15:47:33.410 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-09-16T15:47:33.410 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.410 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-09-16T15:47:33.410 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-09-16T15:47:33.410 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.410 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.410 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.410 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:33.410 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.411 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.411 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.411 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.411 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_cluster_level": { 2024-09-16T15:47:33.411 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_cluster_level", 2024-09-16T15:47:33.411 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-09-16T15:47:33.411 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.411 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-09-16T15:47:33.411 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "info", 2024-09-16T15:47:33.411 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.411 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.411 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [ 2024-09-16T15:47:33.411 INFO:teuthology.orchestra.run.smithi123.stdout: "", 2024-09-16T15:47:33.412 INFO:teuthology.orchestra.run.smithi123.stdout: "critical", 2024-09-16T15:47:33.412 INFO:teuthology.orchestra.run.smithi123.stdout: "debug", 2024-09-16T15:47:33.412 INFO:teuthology.orchestra.run.smithi123.stdout: "error", 2024-09-16T15:47:33.412 INFO:teuthology.orchestra.run.smithi123.stdout: "info", 2024-09-16T15:47:33.412 INFO:teuthology.orchestra.run.smithi123.stdout: "warning" 2024-09-16T15:47:33.412 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-09-16T15:47:33.412 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:33.412 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.412 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.412 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.412 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.412 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_file": { 2024-09-16T15:47:33.412 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_file", 2024-09-16T15:47:33.413 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-09-16T15:47:33.413 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.413 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-09-16T15:47:33.413 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-09-16T15:47:33.413 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.413 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.413 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.413 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:33.413 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.413 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.413 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.413 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.413 INFO:teuthology.orchestra.run.smithi123.stdout: "sqlite3_killpoint": { 2024-09-16T15:47:33.413 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "sqlite3_killpoint", 2024-09-16T15:47:33.414 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "int", 2024-09-16T15:47:33.414 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "dev", 2024-09-16T15:47:33.414 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-09-16T15:47:33.414 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "0", 2024-09-16T15:47:33.414 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.414 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.414 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.414 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:33.414 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.414 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.414 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.414 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-16T15:47:33.414 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-16T15:47:33.414 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.415 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-09-16T15:47:33.415 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "iostat", 2024-09-16T15:47:33.415 INFO:teuthology.orchestra.run.smithi123.stdout: "can_run": true, 2024-09-16T15:47:33.415 INFO:teuthology.orchestra.run.smithi123.stdout: "error_string": "", 2024-09-16T15:47:33.415 INFO:teuthology.orchestra.run.smithi123.stdout: "module_options": { 2024-09-16T15:47:33.415 INFO:teuthology.orchestra.run.smithi123.stdout: "log_level": { 2024-09-16T15:47:33.415 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_level", 2024-09-16T15:47:33.415 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-09-16T15:47:33.415 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.415 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-09-16T15:47:33.415 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "", 2024-09-16T15:47:33.415 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.415 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.415 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [ 2024-09-16T15:47:33.415 INFO:teuthology.orchestra.run.smithi123.stdout: "", 2024-09-16T15:47:33.416 INFO:teuthology.orchestra.run.smithi123.stdout: "critical", 2024-09-16T15:47:33.416 INFO:teuthology.orchestra.run.smithi123.stdout: "debug", 2024-09-16T15:47:33.416 INFO:teuthology.orchestra.run.smithi123.stdout: "error", 2024-09-16T15:47:33.416 INFO:teuthology.orchestra.run.smithi123.stdout: "info", 2024-09-16T15:47:33.416 INFO:teuthology.orchestra.run.smithi123.stdout: "warning" 2024-09-16T15:47:33.416 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-09-16T15:47:33.416 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:33.416 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.416 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.416 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.416 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.416 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_cluster": { 2024-09-16T15:47:33.416 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_cluster", 2024-09-16T15:47:33.417 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-09-16T15:47:33.417 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.417 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-09-16T15:47:33.417 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-09-16T15:47:33.417 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.417 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.417 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.417 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:33.417 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.417 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.417 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.417 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.417 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_cluster_level": { 2024-09-16T15:47:33.417 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_cluster_level", 2024-09-16T15:47:33.417 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-09-16T15:47:33.418 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.418 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-09-16T15:47:33.418 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "info", 2024-09-16T15:47:33.418 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.418 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.418 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [ 2024-09-16T15:47:33.418 INFO:teuthology.orchestra.run.smithi123.stdout: "", 2024-09-16T15:47:33.418 INFO:teuthology.orchestra.run.smithi123.stdout: "critical", 2024-09-16T15:47:33.418 INFO:teuthology.orchestra.run.smithi123.stdout: "debug", 2024-09-16T15:47:33.418 INFO:teuthology.orchestra.run.smithi123.stdout: "error", 2024-09-16T15:47:33.418 INFO:teuthology.orchestra.run.smithi123.stdout: "info", 2024-09-16T15:47:33.418 INFO:teuthology.orchestra.run.smithi123.stdout: "warning" 2024-09-16T15:47:33.419 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-09-16T15:47:33.419 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:33.419 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.419 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.419 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.419 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.419 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_file": { 2024-09-16T15:47:33.419 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_file", 2024-09-16T15:47:33.419 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-09-16T15:47:33.419 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.419 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-09-16T15:47:33.419 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-09-16T15:47:33.419 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.419 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.420 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.420 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:33.420 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.420 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.420 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.420 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.420 INFO:teuthology.orchestra.run.smithi123.stdout: "sqlite3_killpoint": { 2024-09-16T15:47:33.420 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "sqlite3_killpoint", 2024-09-16T15:47:33.420 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "int", 2024-09-16T15:47:33.420 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "dev", 2024-09-16T15:47:33.420 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-09-16T15:47:33.420 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "0", 2024-09-16T15:47:33.420 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.420 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.420 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.421 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:33.421 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.421 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.421 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.421 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-16T15:47:33.421 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-16T15:47:33.421 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.421 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-09-16T15:47:33.422 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "localpool", 2024-09-16T15:47:33.422 INFO:teuthology.orchestra.run.smithi123.stdout: "can_run": true, 2024-09-16T15:47:33.422 INFO:teuthology.orchestra.run.smithi123.stdout: "error_string": "", 2024-09-16T15:47:33.422 INFO:teuthology.orchestra.run.smithi123.stdout: "module_options": { 2024-09-16T15:47:33.422 INFO:teuthology.orchestra.run.smithi123.stdout: "failure_domain": { 2024-09-16T15:47:33.422 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "failure_domain", 2024-09-16T15:47:33.422 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-09-16T15:47:33.422 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.422 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-09-16T15:47:33.422 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "host", 2024-09-16T15:47:33.422 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.422 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.423 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.423 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "failure domain for any created local pool", 2024-09-16T15:47:33.423 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "what failure domain we should separate data replicas across.", 2024-09-16T15:47:33.423 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.423 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.423 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.423 INFO:teuthology.orchestra.run.smithi123.stdout: "log_level": { 2024-09-16T15:47:33.423 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_level", 2024-09-16T15:47:33.423 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-09-16T15:47:33.423 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.423 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-09-16T15:47:33.423 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "", 2024-09-16T15:47:33.424 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.424 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.424 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [ 2024-09-16T15:47:33.424 INFO:teuthology.orchestra.run.smithi123.stdout: "", 2024-09-16T15:47:33.424 INFO:teuthology.orchestra.run.smithi123.stdout: "critical", 2024-09-16T15:47:33.424 INFO:teuthology.orchestra.run.smithi123.stdout: "debug", 2024-09-16T15:47:33.424 INFO:teuthology.orchestra.run.smithi123.stdout: "error", 2024-09-16T15:47:33.424 INFO:teuthology.orchestra.run.smithi123.stdout: "info", 2024-09-16T15:47:33.424 INFO:teuthology.orchestra.run.smithi123.stdout: "warning" 2024-09-16T15:47:33.424 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-09-16T15:47:33.424 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:33.424 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.425 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.425 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.425 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.425 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_cluster": { 2024-09-16T15:47:33.425 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_cluster", 2024-09-16T15:47:33.425 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-09-16T15:47:33.425 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.425 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-09-16T15:47:33.425 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-09-16T15:47:33.425 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.425 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.426 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.426 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:33.426 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.426 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.426 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.426 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.426 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_cluster_level": { 2024-09-16T15:47:33.426 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_cluster_level", 2024-09-16T15:47:33.426 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-09-16T15:47:33.426 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.426 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-09-16T15:47:33.426 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "info", 2024-09-16T15:47:33.427 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.427 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.427 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [ 2024-09-16T15:47:33.427 INFO:teuthology.orchestra.run.smithi123.stdout: "", 2024-09-16T15:47:33.427 INFO:teuthology.orchestra.run.smithi123.stdout: "critical", 2024-09-16T15:47:33.427 INFO:teuthology.orchestra.run.smithi123.stdout: "debug", 2024-09-16T15:47:33.427 INFO:teuthology.orchestra.run.smithi123.stdout: "error", 2024-09-16T15:47:33.427 INFO:teuthology.orchestra.run.smithi123.stdout: "info", 2024-09-16T15:47:33.427 INFO:teuthology.orchestra.run.smithi123.stdout: "warning" 2024-09-16T15:47:33.427 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-09-16T15:47:33.427 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:33.427 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.427 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.428 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.428 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.428 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_file": { 2024-09-16T15:47:33.428 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_file", 2024-09-16T15:47:33.428 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-09-16T15:47:33.428 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.428 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-09-16T15:47:33.428 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-09-16T15:47:33.428 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.428 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.428 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.429 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:33.429 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.429 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.429 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.429 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.429 INFO:teuthology.orchestra.run.smithi123.stdout: "min_size": { 2024-09-16T15:47:33.429 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "min_size", 2024-09-16T15:47:33.429 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "int", 2024-09-16T15:47:33.429 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.429 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-09-16T15:47:33.429 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "", 2024-09-16T15:47:33.429 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.429 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.430 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.430 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "default min_size for any created local pool", 2024-09-16T15:47:33.430 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "value to set min_size to (unchanged from Ceph's default if this option is not set)", 2024-09-16T15:47:33.430 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.430 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.430 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.430 INFO:teuthology.orchestra.run.smithi123.stdout: "num_rep": { 2024-09-16T15:47:33.430 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "num_rep", 2024-09-16T15:47:33.430 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "int", 2024-09-16T15:47:33.430 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.430 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-09-16T15:47:33.430 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "3", 2024-09-16T15:47:33.431 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.431 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.431 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.431 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "default replica count for any created local pool", 2024-09-16T15:47:33.431 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.431 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.431 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.431 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.431 INFO:teuthology.orchestra.run.smithi123.stdout: "pg_num": { 2024-09-16T15:47:33.431 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "pg_num", 2024-09-16T15:47:33.431 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "int", 2024-09-16T15:47:33.431 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.431 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-09-16T15:47:33.432 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "128", 2024-09-16T15:47:33.432 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.432 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.432 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.432 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "default pg_num for any created local pool", 2024-09-16T15:47:33.432 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.432 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.432 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.432 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.432 INFO:teuthology.orchestra.run.smithi123.stdout: "prefix": { 2024-09-16T15:47:33.432 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "prefix", 2024-09-16T15:47:33.432 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-09-16T15:47:33.433 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.433 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-09-16T15:47:33.433 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "", 2024-09-16T15:47:33.433 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.433 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.433 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.433 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "name prefix for any created local pool", 2024-09-16T15:47:33.433 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.433 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.433 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.433 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.433 INFO:teuthology.orchestra.run.smithi123.stdout: "sqlite3_killpoint": { 2024-09-16T15:47:33.433 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "sqlite3_killpoint", 2024-09-16T15:47:33.434 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "int", 2024-09-16T15:47:33.434 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "dev", 2024-09-16T15:47:33.434 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-09-16T15:47:33.434 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "0", 2024-09-16T15:47:33.434 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.434 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.434 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.434 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:33.434 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.434 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.434 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.434 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.435 INFO:teuthology.orchestra.run.smithi123.stdout: "subtree": { 2024-09-16T15:47:33.435 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "subtree", 2024-09-16T15:47:33.435 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-09-16T15:47:33.435 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.435 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-09-16T15:47:33.435 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "rack", 2024-09-16T15:47:33.435 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.435 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.435 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.435 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "CRUSH level for which to create a local pool", 2024-09-16T15:47:33.436 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "which CRUSH subtree type the module should create a pool for.", 2024-09-16T15:47:33.436 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.436 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.436 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-16T15:47:33.436 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-16T15:47:33.436 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.436 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-09-16T15:47:33.436 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "mirroring", 2024-09-16T15:47:33.436 INFO:teuthology.orchestra.run.smithi123.stdout: "can_run": true, 2024-09-16T15:47:33.436 INFO:teuthology.orchestra.run.smithi123.stdout: "error_string": "", 2024-09-16T15:47:33.436 INFO:teuthology.orchestra.run.smithi123.stdout: "module_options": { 2024-09-16T15:47:33.437 INFO:teuthology.orchestra.run.smithi123.stdout: "log_level": { 2024-09-16T15:47:33.437 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_level", 2024-09-16T15:47:33.437 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-09-16T15:47:33.437 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.437 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-09-16T15:47:33.437 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "", 2024-09-16T15:47:33.437 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.437 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.437 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [ 2024-09-16T15:47:33.437 INFO:teuthology.orchestra.run.smithi123.stdout: "", 2024-09-16T15:47:33.437 INFO:teuthology.orchestra.run.smithi123.stdout: "critical", 2024-09-16T15:47:33.437 INFO:teuthology.orchestra.run.smithi123.stdout: "debug", 2024-09-16T15:47:33.438 INFO:teuthology.orchestra.run.smithi123.stdout: "error", 2024-09-16T15:47:33.438 INFO:teuthology.orchestra.run.smithi123.stdout: "info", 2024-09-16T15:47:33.438 INFO:teuthology.orchestra.run.smithi123.stdout: "warning" 2024-09-16T15:47:33.438 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-09-16T15:47:33.438 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:33.438 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.438 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.438 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.438 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.438 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_cluster": { 2024-09-16T15:47:33.439 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_cluster", 2024-09-16T15:47:33.439 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-09-16T15:47:33.439 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.439 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-09-16T15:47:33.439 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-09-16T15:47:33.439 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.439 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.439 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.439 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:33.439 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.439 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.440 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.440 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.440 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_cluster_level": { 2024-09-16T15:47:33.440 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_cluster_level", 2024-09-16T15:47:33.440 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-09-16T15:47:33.440 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.440 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-09-16T15:47:33.440 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "info", 2024-09-16T15:47:33.440 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.440 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.440 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [ 2024-09-16T15:47:33.441 INFO:teuthology.orchestra.run.smithi123.stdout: "", 2024-09-16T15:47:33.441 INFO:teuthology.orchestra.run.smithi123.stdout: "critical", 2024-09-16T15:47:33.441 INFO:teuthology.orchestra.run.smithi123.stdout: "debug", 2024-09-16T15:47:33.441 INFO:teuthology.orchestra.run.smithi123.stdout: "error", 2024-09-16T15:47:33.441 INFO:teuthology.orchestra.run.smithi123.stdout: "info", 2024-09-16T15:47:33.441 INFO:teuthology.orchestra.run.smithi123.stdout: "warning" 2024-09-16T15:47:33.441 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-09-16T15:47:33.441 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:33.441 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.441 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.441 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.441 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.441 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_file": { 2024-09-16T15:47:33.442 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_file", 2024-09-16T15:47:33.442 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-09-16T15:47:33.442 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.442 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-09-16T15:47:33.442 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-09-16T15:47:33.442 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.442 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.442 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.442 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:33.442 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.442 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.442 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.443 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.443 INFO:teuthology.orchestra.run.smithi123.stdout: "sqlite3_killpoint": { 2024-09-16T15:47:33.443 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "sqlite3_killpoint", 2024-09-16T15:47:33.443 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "int", 2024-09-16T15:47:33.443 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "dev", 2024-09-16T15:47:33.443 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-09-16T15:47:33.443 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "0", 2024-09-16T15:47:33.443 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.443 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.443 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.443 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:33.444 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.444 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.444 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.444 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-16T15:47:33.444 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-16T15:47:33.444 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.444 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-09-16T15:47:33.444 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "nfs", 2024-09-16T15:47:33.444 INFO:teuthology.orchestra.run.smithi123.stdout: "can_run": true, 2024-09-16T15:47:33.444 INFO:teuthology.orchestra.run.smithi123.stdout: "error_string": "", 2024-09-16T15:47:33.444 INFO:teuthology.orchestra.run.smithi123.stdout: "module_options": { 2024-09-16T15:47:33.444 INFO:teuthology.orchestra.run.smithi123.stdout: "log_level": { 2024-09-16T15:47:33.445 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_level", 2024-09-16T15:47:33.445 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-09-16T15:47:33.445 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.445 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-09-16T15:47:33.445 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "", 2024-09-16T15:47:33.445 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.445 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.445 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [ 2024-09-16T15:47:33.445 INFO:teuthology.orchestra.run.smithi123.stdout: "", 2024-09-16T15:47:33.445 INFO:teuthology.orchestra.run.smithi123.stdout: "critical", 2024-09-16T15:47:33.445 INFO:teuthology.orchestra.run.smithi123.stdout: "debug", 2024-09-16T15:47:33.445 INFO:teuthology.orchestra.run.smithi123.stdout: "error", 2024-09-16T15:47:33.445 INFO:teuthology.orchestra.run.smithi123.stdout: "info", 2024-09-16T15:47:33.446 INFO:teuthology.orchestra.run.smithi123.stdout: "warning" 2024-09-16T15:47:33.446 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-09-16T15:47:33.446 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:33.446 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.446 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.446 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.446 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.446 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_cluster": { 2024-09-16T15:47:33.446 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_cluster", 2024-09-16T15:47:33.446 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-09-16T15:47:33.446 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.446 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-09-16T15:47:33.446 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-09-16T15:47:33.447 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.447 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.447 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.447 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:33.447 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.447 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.447 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.447 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.447 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_cluster_level": { 2024-09-16T15:47:33.447 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_cluster_level", 2024-09-16T15:47:33.447 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-09-16T15:47:33.447 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.447 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-09-16T15:47:33.448 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "info", 2024-09-16T15:47:33.448 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.448 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.448 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [ 2024-09-16T15:47:33.448 INFO:teuthology.orchestra.run.smithi123.stdout: "", 2024-09-16T15:47:33.448 INFO:teuthology.orchestra.run.smithi123.stdout: "critical", 2024-09-16T15:47:33.448 INFO:teuthology.orchestra.run.smithi123.stdout: "debug", 2024-09-16T15:47:33.448 INFO:teuthology.orchestra.run.smithi123.stdout: "error", 2024-09-16T15:47:33.449 INFO:teuthology.orchestra.run.smithi123.stdout: "info", 2024-09-16T15:47:33.449 INFO:teuthology.orchestra.run.smithi123.stdout: "warning" 2024-09-16T15:47:33.449 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-09-16T15:47:33.449 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:33.449 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.449 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.449 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.449 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.449 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_file": { 2024-09-16T15:47:33.449 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_file", 2024-09-16T15:47:33.449 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-09-16T15:47:33.449 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.449 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-09-16T15:47:33.450 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-09-16T15:47:33.450 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.450 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.450 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.450 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:33.450 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.450 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.450 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.450 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.450 INFO:teuthology.orchestra.run.smithi123.stdout: "sqlite3_killpoint": { 2024-09-16T15:47:33.450 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "sqlite3_killpoint", 2024-09-16T15:47:33.450 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "int", 2024-09-16T15:47:33.450 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "dev", 2024-09-16T15:47:33.451 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-09-16T15:47:33.451 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "0", 2024-09-16T15:47:33.451 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.451 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.451 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.451 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:33.451 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.451 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.451 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.451 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-16T15:47:33.451 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-16T15:47:33.451 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.451 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-09-16T15:47:33.452 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "orchestrator", 2024-09-16T15:47:33.452 INFO:teuthology.orchestra.run.smithi123.stdout: "can_run": true, 2024-09-16T15:47:33.452 INFO:teuthology.orchestra.run.smithi123.stdout: "error_string": "", 2024-09-16T15:47:33.452 INFO:teuthology.orchestra.run.smithi123.stdout: "module_options": { 2024-09-16T15:47:33.452 INFO:teuthology.orchestra.run.smithi123.stdout: "fail_fs": { 2024-09-16T15:47:33.452 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "fail_fs", 2024-09-16T15:47:33.452 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-09-16T15:47:33.452 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.452 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-09-16T15:47:33.452 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-09-16T15:47:33.452 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.452 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.453 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.453 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "Fail filesystem for rapid multi-rank mds upgrade", 2024-09-16T15:47:33.453 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.453 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.453 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.453 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.453 INFO:teuthology.orchestra.run.smithi123.stdout: "log_level": { 2024-09-16T15:47:33.453 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_level", 2024-09-16T15:47:33.453 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-09-16T15:47:33.453 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.454 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-09-16T15:47:33.454 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "", 2024-09-16T15:47:33.454 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.454 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.454 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [ 2024-09-16T15:47:33.454 INFO:teuthology.orchestra.run.smithi123.stdout: "", 2024-09-16T15:47:33.454 INFO:teuthology.orchestra.run.smithi123.stdout: "critical", 2024-09-16T15:47:33.454 INFO:teuthology.orchestra.run.smithi123.stdout: "debug", 2024-09-16T15:47:33.454 INFO:teuthology.orchestra.run.smithi123.stdout: "error", 2024-09-16T15:47:33.454 INFO:teuthology.orchestra.run.smithi123.stdout: "info", 2024-09-16T15:47:33.454 INFO:teuthology.orchestra.run.smithi123.stdout: "warning" 2024-09-16T15:47:33.455 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-09-16T15:47:33.455 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:33.455 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.455 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.455 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.455 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.455 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_cluster": { 2024-09-16T15:47:33.455 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_cluster", 2024-09-16T15:47:33.455 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-09-16T15:47:33.455 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.455 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-09-16T15:47:33.456 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-09-16T15:47:33.456 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.456 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.456 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.456 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:33.456 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.456 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.456 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.456 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.456 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_cluster_level": { 2024-09-16T15:47:33.456 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_cluster_level", 2024-09-16T15:47:33.456 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-09-16T15:47:33.457 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.457 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-09-16T15:47:33.457 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "info", 2024-09-16T15:47:33.457 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.457 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.457 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [ 2024-09-16T15:47:33.457 INFO:teuthology.orchestra.run.smithi123.stdout: "", 2024-09-16T15:47:33.457 INFO:teuthology.orchestra.run.smithi123.stdout: "critical", 2024-09-16T15:47:33.457 INFO:teuthology.orchestra.run.smithi123.stdout: "debug", 2024-09-16T15:47:33.457 INFO:teuthology.orchestra.run.smithi123.stdout: "error", 2024-09-16T15:47:33.458 INFO:teuthology.orchestra.run.smithi123.stdout: "info", 2024-09-16T15:47:33.458 INFO:teuthology.orchestra.run.smithi123.stdout: "warning" 2024-09-16T15:47:33.458 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-09-16T15:47:33.458 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:33.458 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.458 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.458 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.458 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.458 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_file": { 2024-09-16T15:47:33.458 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_file", 2024-09-16T15:47:33.458 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-09-16T15:47:33.458 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.459 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-09-16T15:47:33.459 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-09-16T15:47:33.459 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.459 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.459 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.459 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:33.459 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.459 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.459 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.459 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.459 INFO:teuthology.orchestra.run.smithi123.stdout: "orchestrator": { 2024-09-16T15:47:33.460 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "orchestrator", 2024-09-16T15:47:33.460 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-09-16T15:47:33.460 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.460 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-09-16T15:47:33.460 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "", 2024-09-16T15:47:33.460 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.460 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.460 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [ 2024-09-16T15:47:33.460 INFO:teuthology.orchestra.run.smithi123.stdout: "cephadm", 2024-09-16T15:47:33.460 INFO:teuthology.orchestra.run.smithi123.stdout: "rook", 2024-09-16T15:47:33.460 INFO:teuthology.orchestra.run.smithi123.stdout: "test_orchestrator" 2024-09-16T15:47:33.460 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-09-16T15:47:33.461 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "Orchestrator backend", 2024-09-16T15:47:33.461 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.461 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.461 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.461 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.461 INFO:teuthology.orchestra.run.smithi123.stdout: "sqlite3_killpoint": { 2024-09-16T15:47:33.461 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "sqlite3_killpoint", 2024-09-16T15:47:33.461 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "int", 2024-09-16T15:47:33.461 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "dev", 2024-09-16T15:47:33.461 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-09-16T15:47:33.461 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "0", 2024-09-16T15:47:33.461 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.462 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.462 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.462 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:33.462 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.462 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.462 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.462 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-16T15:47:33.462 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-16T15:47:33.462 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.462 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-09-16T15:47:33.462 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "osd_perf_query", 2024-09-16T15:47:33.463 INFO:teuthology.orchestra.run.smithi123.stdout: "can_run": true, 2024-09-16T15:47:33.463 INFO:teuthology.orchestra.run.smithi123.stdout: "error_string": "", 2024-09-16T15:47:33.463 INFO:teuthology.orchestra.run.smithi123.stdout: "module_options": { 2024-09-16T15:47:33.463 INFO:teuthology.orchestra.run.smithi123.stdout: "log_level": { 2024-09-16T15:47:33.463 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_level", 2024-09-16T15:47:33.463 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-09-16T15:47:33.463 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.463 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-09-16T15:47:33.463 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "", 2024-09-16T15:47:33.463 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.464 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.464 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [ 2024-09-16T15:47:33.464 INFO:teuthology.orchestra.run.smithi123.stdout: "", 2024-09-16T15:47:33.464 INFO:teuthology.orchestra.run.smithi123.stdout: "critical", 2024-09-16T15:47:33.464 INFO:teuthology.orchestra.run.smithi123.stdout: "debug", 2024-09-16T15:47:33.464 INFO:teuthology.orchestra.run.smithi123.stdout: "error", 2024-09-16T15:47:33.464 INFO:teuthology.orchestra.run.smithi123.stdout: "info", 2024-09-16T15:47:33.464 INFO:teuthology.orchestra.run.smithi123.stdout: "warning" 2024-09-16T15:47:33.464 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-09-16T15:47:33.464 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:33.464 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.464 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.464 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.465 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.465 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_cluster": { 2024-09-16T15:47:33.465 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_cluster", 2024-09-16T15:47:33.465 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-09-16T15:47:33.465 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.465 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-09-16T15:47:33.465 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-09-16T15:47:33.465 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.465 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.465 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.465 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:33.465 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.465 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.466 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.466 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.466 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_cluster_level": { 2024-09-16T15:47:33.466 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_cluster_level", 2024-09-16T15:47:33.466 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-09-16T15:47:33.466 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.466 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-09-16T15:47:33.466 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "info", 2024-09-16T15:47:33.466 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.466 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.466 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [ 2024-09-16T15:47:33.467 INFO:teuthology.orchestra.run.smithi123.stdout: "", 2024-09-16T15:47:33.467 INFO:teuthology.orchestra.run.smithi123.stdout: "critical", 2024-09-16T15:47:33.467 INFO:teuthology.orchestra.run.smithi123.stdout: "debug", 2024-09-16T15:47:33.467 INFO:teuthology.orchestra.run.smithi123.stdout: "error", 2024-09-16T15:47:33.467 INFO:teuthology.orchestra.run.smithi123.stdout: "info", 2024-09-16T15:47:33.467 INFO:teuthology.orchestra.run.smithi123.stdout: "warning" 2024-09-16T15:47:33.467 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-09-16T15:47:33.467 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:33.467 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.467 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.467 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.468 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.468 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_file": { 2024-09-16T15:47:33.468 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_file", 2024-09-16T15:47:33.468 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-09-16T15:47:33.468 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.468 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-09-16T15:47:33.468 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-09-16T15:47:33.468 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.468 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.468 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.469 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:33.469 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.469 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.469 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.469 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.469 INFO:teuthology.orchestra.run.smithi123.stdout: "sqlite3_killpoint": { 2024-09-16T15:47:33.469 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "sqlite3_killpoint", 2024-09-16T15:47:33.469 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "int", 2024-09-16T15:47:33.469 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "dev", 2024-09-16T15:47:33.469 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-09-16T15:47:33.469 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "0", 2024-09-16T15:47:33.469 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.470 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.470 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.470 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:33.470 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.470 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.470 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.470 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-16T15:47:33.470 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-16T15:47:33.470 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.470 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-09-16T15:47:33.470 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "osd_support", 2024-09-16T15:47:33.471 INFO:teuthology.orchestra.run.smithi123.stdout: "can_run": true, 2024-09-16T15:47:33.471 INFO:teuthology.orchestra.run.smithi123.stdout: "error_string": "", 2024-09-16T15:47:33.471 INFO:teuthology.orchestra.run.smithi123.stdout: "module_options": { 2024-09-16T15:47:33.471 INFO:teuthology.orchestra.run.smithi123.stdout: "log_level": { 2024-09-16T15:47:33.471 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_level", 2024-09-16T15:47:33.471 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-09-16T15:47:33.471 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.471 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-09-16T15:47:33.471 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "", 2024-09-16T15:47:33.471 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.471 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.472 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [ 2024-09-16T15:47:33.472 INFO:teuthology.orchestra.run.smithi123.stdout: "", 2024-09-16T15:47:33.472 INFO:teuthology.orchestra.run.smithi123.stdout: "critical", 2024-09-16T15:47:33.472 INFO:teuthology.orchestra.run.smithi123.stdout: "debug", 2024-09-16T15:47:33.472 INFO:teuthology.orchestra.run.smithi123.stdout: "error", 2024-09-16T15:47:33.472 INFO:teuthology.orchestra.run.smithi123.stdout: "info", 2024-09-16T15:47:33.472 INFO:teuthology.orchestra.run.smithi123.stdout: "warning" 2024-09-16T15:47:33.472 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-09-16T15:47:33.472 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:33.472 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.472 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.472 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.473 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.473 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_cluster": { 2024-09-16T15:47:33.473 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_cluster", 2024-09-16T15:47:33.473 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-09-16T15:47:33.473 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.473 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-09-16T15:47:33.473 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-09-16T15:47:33.473 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.473 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.473 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.473 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:33.473 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.474 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.474 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.474 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.474 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_cluster_level": { 2024-09-16T15:47:33.474 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_cluster_level", 2024-09-16T15:47:33.474 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-09-16T15:47:33.474 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.474 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-09-16T15:47:33.474 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "info", 2024-09-16T15:47:33.474 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.474 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.475 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [ 2024-09-16T15:47:33.475 INFO:teuthology.orchestra.run.smithi123.stdout: "", 2024-09-16T15:47:33.475 INFO:teuthology.orchestra.run.smithi123.stdout: "critical", 2024-09-16T15:47:33.475 INFO:teuthology.orchestra.run.smithi123.stdout: "debug", 2024-09-16T15:47:33.475 INFO:teuthology.orchestra.run.smithi123.stdout: "error", 2024-09-16T15:47:33.475 INFO:teuthology.orchestra.run.smithi123.stdout: "info", 2024-09-16T15:47:33.475 INFO:teuthology.orchestra.run.smithi123.stdout: "warning" 2024-09-16T15:47:33.475 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-09-16T15:47:33.475 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:33.475 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.475 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.475 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.475 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.476 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_file": { 2024-09-16T15:47:33.476 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_file", 2024-09-16T15:47:33.476 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-09-16T15:47:33.476 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.476 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-09-16T15:47:33.476 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-09-16T15:47:33.476 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.476 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.476 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.476 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:33.476 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.476 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.477 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.477 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.477 INFO:teuthology.orchestra.run.smithi123.stdout: "sqlite3_killpoint": { 2024-09-16T15:47:33.477 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "sqlite3_killpoint", 2024-09-16T15:47:33.477 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "int", 2024-09-16T15:47:33.477 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "dev", 2024-09-16T15:47:33.477 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-09-16T15:47:33.477 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "0", 2024-09-16T15:47:33.477 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.477 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.477 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.478 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:33.478 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.478 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.478 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.478 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-16T15:47:33.478 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-16T15:47:33.478 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.478 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-09-16T15:47:33.478 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "pg_autoscaler", 2024-09-16T15:47:33.478 INFO:teuthology.orchestra.run.smithi123.stdout: "can_run": true, 2024-09-16T15:47:33.478 INFO:teuthology.orchestra.run.smithi123.stdout: "error_string": "", 2024-09-16T15:47:33.478 INFO:teuthology.orchestra.run.smithi123.stdout: "module_options": { 2024-09-16T15:47:33.478 INFO:teuthology.orchestra.run.smithi123.stdout: "log_level": { 2024-09-16T15:47:33.479 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_level", 2024-09-16T15:47:33.479 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-09-16T15:47:33.479 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.479 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-09-16T15:47:33.479 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "", 2024-09-16T15:47:33.479 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.479 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.479 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [ 2024-09-16T15:47:33.479 INFO:teuthology.orchestra.run.smithi123.stdout: "", 2024-09-16T15:47:33.479 INFO:teuthology.orchestra.run.smithi123.stdout: "critical", 2024-09-16T15:47:33.479 INFO:teuthology.orchestra.run.smithi123.stdout: "debug", 2024-09-16T15:47:33.479 INFO:teuthology.orchestra.run.smithi123.stdout: "error", 2024-09-16T15:47:33.480 INFO:teuthology.orchestra.run.smithi123.stdout: "info", 2024-09-16T15:47:33.480 INFO:teuthology.orchestra.run.smithi123.stdout: "warning" 2024-09-16T15:47:33.480 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-09-16T15:47:33.480 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:33.480 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.480 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.480 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.480 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.480 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_cluster": { 2024-09-16T15:47:33.480 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_cluster", 2024-09-16T15:47:33.480 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-09-16T15:47:33.481 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.481 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-09-16T15:47:33.481 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-09-16T15:47:33.481 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.481 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.481 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.481 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:33.481 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.481 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.481 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.481 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.482 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_cluster_level": { 2024-09-16T15:47:33.482 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_cluster_level", 2024-09-16T15:47:33.482 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-09-16T15:47:33.482 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.482 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-09-16T15:47:33.482 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "info", 2024-09-16T15:47:33.482 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.482 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.482 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [ 2024-09-16T15:47:33.482 INFO:teuthology.orchestra.run.smithi123.stdout: "", 2024-09-16T15:47:33.482 INFO:teuthology.orchestra.run.smithi123.stdout: "critical", 2024-09-16T15:47:33.483 INFO:teuthology.orchestra.run.smithi123.stdout: "debug", 2024-09-16T15:47:33.483 INFO:teuthology.orchestra.run.smithi123.stdout: "error", 2024-09-16T15:47:33.483 INFO:teuthology.orchestra.run.smithi123.stdout: "info", 2024-09-16T15:47:33.483 INFO:teuthology.orchestra.run.smithi123.stdout: "warning" 2024-09-16T15:47:33.483 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-09-16T15:47:33.483 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:33.483 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.483 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.483 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.483 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.483 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_file": { 2024-09-16T15:47:33.483 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_file", 2024-09-16T15:47:33.484 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-09-16T15:47:33.484 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.484 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-09-16T15:47:33.484 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-09-16T15:47:33.484 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.484 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.484 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.484 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:33.484 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.484 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.484 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.485 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.485 INFO:teuthology.orchestra.run.smithi123.stdout: "sleep_interval": { 2024-09-16T15:47:33.485 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "sleep_interval", 2024-09-16T15:47:33.485 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "secs", 2024-09-16T15:47:33.485 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.485 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-09-16T15:47:33.485 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "60", 2024-09-16T15:47:33.485 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.485 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.485 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.486 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:33.486 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.486 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.486 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.486 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.486 INFO:teuthology.orchestra.run.smithi123.stdout: "sqlite3_killpoint": { 2024-09-16T15:47:33.486 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "sqlite3_killpoint", 2024-09-16T15:47:33.486 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "int", 2024-09-16T15:47:33.486 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "dev", 2024-09-16T15:47:33.486 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-09-16T15:47:33.486 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "0", 2024-09-16T15:47:33.487 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.487 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.487 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.487 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:33.487 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.487 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.487 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.487 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.487 INFO:teuthology.orchestra.run.smithi123.stdout: "threshold": { 2024-09-16T15:47:33.487 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "threshold", 2024-09-16T15:47:33.487 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "float", 2024-09-16T15:47:33.487 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.488 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-09-16T15:47:33.488 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "3.0", 2024-09-16T15:47:33.488 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "1.0", 2024-09-16T15:47:33.488 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.488 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.488 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "scaling threshold", 2024-09-16T15:47:33.488 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0", 2024-09-16T15:47:33.488 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.488 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.488 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-16T15:47:33.488 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-16T15:47:33.489 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.489 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-09-16T15:47:33.489 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "progress", 2024-09-16T15:47:33.489 INFO:teuthology.orchestra.run.smithi123.stdout: "can_run": true, 2024-09-16T15:47:33.489 INFO:teuthology.orchestra.run.smithi123.stdout: "error_string": "", 2024-09-16T15:47:33.489 INFO:teuthology.orchestra.run.smithi123.stdout: "module_options": { 2024-09-16T15:47:33.489 INFO:teuthology.orchestra.run.smithi123.stdout: "allow_pg_recovery_event": { 2024-09-16T15:47:33.489 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "allow_pg_recovery_event", 2024-09-16T15:47:33.489 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-09-16T15:47:33.489 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.489 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-09-16T15:47:33.489 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-09-16T15:47:33.490 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.490 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.490 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.490 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "allow the module to show pg recovery progress", 2024-09-16T15:47:33.490 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.490 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.490 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.490 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.490 INFO:teuthology.orchestra.run.smithi123.stdout: "enabled": { 2024-09-16T15:47:33.490 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "enabled", 2024-09-16T15:47:33.490 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-09-16T15:47:33.491 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.491 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-09-16T15:47:33.491 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "True", 2024-09-16T15:47:33.491 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.491 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.491 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.491 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:33.491 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.491 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.491 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.491 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.491 INFO:teuthology.orchestra.run.smithi123.stdout: "log_level": { 2024-09-16T15:47:33.491 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_level", 2024-09-16T15:47:33.492 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-09-16T15:47:33.492 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.492 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-09-16T15:47:33.492 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "", 2024-09-16T15:47:33.492 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.492 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.492 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [ 2024-09-16T15:47:33.492 INFO:teuthology.orchestra.run.smithi123.stdout: "", 2024-09-16T15:47:33.492 INFO:teuthology.orchestra.run.smithi123.stdout: "critical", 2024-09-16T15:47:33.492 INFO:teuthology.orchestra.run.smithi123.stdout: "debug", 2024-09-16T15:47:33.492 INFO:teuthology.orchestra.run.smithi123.stdout: "error", 2024-09-16T15:47:33.492 INFO:teuthology.orchestra.run.smithi123.stdout: "info", 2024-09-16T15:47:33.493 INFO:teuthology.orchestra.run.smithi123.stdout: "warning" 2024-09-16T15:47:33.493 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-09-16T15:47:33.493 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:33.493 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.493 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.493 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.493 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.493 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_cluster": { 2024-09-16T15:47:33.493 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_cluster", 2024-09-16T15:47:33.493 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-09-16T15:47:33.493 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.493 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-09-16T15:47:33.494 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-09-16T15:47:33.494 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.494 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.494 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.494 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:33.494 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.494 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.494 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.494 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.494 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_cluster_level": { 2024-09-16T15:47:33.494 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_cluster_level", 2024-09-16T15:47:33.494 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-09-16T15:47:33.494 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.495 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-09-16T15:47:33.495 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "info", 2024-09-16T15:47:33.495 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.495 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.495 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [ 2024-09-16T15:47:33.495 INFO:teuthology.orchestra.run.smithi123.stdout: "", 2024-09-16T15:47:33.495 INFO:teuthology.orchestra.run.smithi123.stdout: "critical", 2024-09-16T15:47:33.495 INFO:teuthology.orchestra.run.smithi123.stdout: "debug", 2024-09-16T15:47:33.495 INFO:teuthology.orchestra.run.smithi123.stdout: "error", 2024-09-16T15:47:33.495 INFO:teuthology.orchestra.run.smithi123.stdout: "info", 2024-09-16T15:47:33.495 INFO:teuthology.orchestra.run.smithi123.stdout: "warning" 2024-09-16T15:47:33.495 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-09-16T15:47:33.495 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:33.496 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.496 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.496 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.496 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.496 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_file": { 2024-09-16T15:47:33.496 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_file", 2024-09-16T15:47:33.496 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-09-16T15:47:33.496 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.496 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-09-16T15:47:33.496 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-09-16T15:47:33.496 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.496 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.497 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.497 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:33.497 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.497 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.497 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.497 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.497 INFO:teuthology.orchestra.run.smithi123.stdout: "max_completed_events": { 2024-09-16T15:47:33.497 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "max_completed_events", 2024-09-16T15:47:33.497 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "int", 2024-09-16T15:47:33.497 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.497 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-09-16T15:47:33.498 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "50", 2024-09-16T15:47:33.498 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.498 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.498 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.498 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "number of past completed events to remember", 2024-09-16T15:47:33.498 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.498 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.498 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.498 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.498 INFO:teuthology.orchestra.run.smithi123.stdout: "sleep_interval": { 2024-09-16T15:47:33.498 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "sleep_interval", 2024-09-16T15:47:33.498 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "secs", 2024-09-16T15:47:33.499 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.499 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-09-16T15:47:33.499 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "5", 2024-09-16T15:47:33.499 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.499 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.499 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.499 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "how long the module is going to sleep", 2024-09-16T15:47:33.499 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.499 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.499 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.499 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.499 INFO:teuthology.orchestra.run.smithi123.stdout: "sqlite3_killpoint": { 2024-09-16T15:47:33.500 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "sqlite3_killpoint", 2024-09-16T15:47:33.500 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "int", 2024-09-16T15:47:33.500 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "dev", 2024-09-16T15:47:33.500 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-09-16T15:47:33.500 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "0", 2024-09-16T15:47:33.500 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.500 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.500 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.500 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:33.500 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.500 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.501 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.501 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-16T15:47:33.501 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-16T15:47:33.501 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.501 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-09-16T15:47:33.501 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "prometheus", 2024-09-16T15:47:33.501 INFO:teuthology.orchestra.run.smithi123.stdout: "can_run": true, 2024-09-16T15:47:33.501 INFO:teuthology.orchestra.run.smithi123.stdout: "error_string": "", 2024-09-16T15:47:33.501 INFO:teuthology.orchestra.run.smithi123.stdout: "module_options": { 2024-09-16T15:47:33.501 INFO:teuthology.orchestra.run.smithi123.stdout: "cache": { 2024-09-16T15:47:33.501 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "cache", 2024-09-16T15:47:33.501 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-09-16T15:47:33.501 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.502 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-09-16T15:47:33.502 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "True", 2024-09-16T15:47:33.502 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.502 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.502 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.502 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:33.502 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.502 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.502 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.502 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.502 INFO:teuthology.orchestra.run.smithi123.stdout: "exclude_perf_counters": { 2024-09-16T15:47:33.502 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "exclude_perf_counters", 2024-09-16T15:47:33.503 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-09-16T15:47:33.503 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.503 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-09-16T15:47:33.503 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "True", 2024-09-16T15:47:33.503 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.503 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.503 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.503 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "Do not include perf-counters in the metrics output", 2024-09-16T15:47:33.503 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.", 2024-09-16T15:47:33.503 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.503 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.503 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.504 INFO:teuthology.orchestra.run.smithi123.stdout: "log_level": { 2024-09-16T15:47:33.504 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_level", 2024-09-16T15:47:33.504 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-09-16T15:47:33.504 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.504 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-09-16T15:47:33.504 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "", 2024-09-16T15:47:33.504 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.504 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.504 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [ 2024-09-16T15:47:33.504 INFO:teuthology.orchestra.run.smithi123.stdout: "", 2024-09-16T15:47:33.504 INFO:teuthology.orchestra.run.smithi123.stdout: "critical", 2024-09-16T15:47:33.504 INFO:teuthology.orchestra.run.smithi123.stdout: "debug", 2024-09-16T15:47:33.505 INFO:teuthology.orchestra.run.smithi123.stdout: "error", 2024-09-16T15:47:33.505 INFO:teuthology.orchestra.run.smithi123.stdout: "info", 2024-09-16T15:47:33.505 INFO:teuthology.orchestra.run.smithi123.stdout: "warning" 2024-09-16T15:47:33.505 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-09-16T15:47:33.505 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:33.505 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.505 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.505 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.505 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.505 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_cluster": { 2024-09-16T15:47:33.505 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_cluster", 2024-09-16T15:47:33.505 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-09-16T15:47:33.506 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.506 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-09-16T15:47:33.506 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-09-16T15:47:33.506 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.506 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.506 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.506 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:33.506 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.506 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.506 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.506 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.506 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_cluster_level": { 2024-09-16T15:47:33.506 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_cluster_level", 2024-09-16T15:47:33.507 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-09-16T15:47:33.507 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.507 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-09-16T15:47:33.507 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "info", 2024-09-16T15:47:33.507 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.507 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.507 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [ 2024-09-16T15:47:33.507 INFO:teuthology.orchestra.run.smithi123.stdout: "", 2024-09-16T15:47:33.507 INFO:teuthology.orchestra.run.smithi123.stdout: "critical", 2024-09-16T15:47:33.507 INFO:teuthology.orchestra.run.smithi123.stdout: "debug", 2024-09-16T15:47:33.507 INFO:teuthology.orchestra.run.smithi123.stdout: "error", 2024-09-16T15:47:33.507 INFO:teuthology.orchestra.run.smithi123.stdout: "info", 2024-09-16T15:47:33.508 INFO:teuthology.orchestra.run.smithi123.stdout: "warning" 2024-09-16T15:47:33.508 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-09-16T15:47:33.508 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:33.508 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.508 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.508 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.508 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.508 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_file": { 2024-09-16T15:47:33.508 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_file", 2024-09-16T15:47:33.508 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-09-16T15:47:33.508 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.508 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-09-16T15:47:33.509 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-09-16T15:47:33.509 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.509 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.509 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.509 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:33.509 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.509 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.509 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.509 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.509 INFO:teuthology.orchestra.run.smithi123.stdout: "rbd_stats_pools": { 2024-09-16T15:47:33.509 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "rbd_stats_pools", 2024-09-16T15:47:33.509 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-09-16T15:47:33.510 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.510 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-09-16T15:47:33.510 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "", 2024-09-16T15:47:33.510 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.510 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.510 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.510 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:33.510 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.510 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.510 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.511 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.511 INFO:teuthology.orchestra.run.smithi123.stdout: "rbd_stats_pools_refresh_interval": { 2024-09-16T15:47:33.511 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "rbd_stats_pools_refresh_interval", 2024-09-16T15:47:33.511 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "int", 2024-09-16T15:47:33.511 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.511 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-09-16T15:47:33.511 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "300", 2024-09-16T15:47:33.511 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.511 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.511 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.511 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:33.512 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.512 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.512 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.512 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.512 INFO:teuthology.orchestra.run.smithi123.stdout: "scrape_interval": { 2024-09-16T15:47:33.512 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "scrape_interval", 2024-09-16T15:47:33.512 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "float", 2024-09-16T15:47:33.512 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.512 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-09-16T15:47:33.513 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "15.0", 2024-09-16T15:47:33.513 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.513 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.513 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.513 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:33.513 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.513 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.513 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.513 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.513 INFO:teuthology.orchestra.run.smithi123.stdout: "server_addr": { 2024-09-16T15:47:33.514 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "server_addr", 2024-09-16T15:47:33.514 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-09-16T15:47:33.514 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.514 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-09-16T15:47:33.514 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "::", 2024-09-16T15:47:33.514 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.514 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.514 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.514 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "the IPv4 or IPv6 address on which the module listens for HTTP requests", 2024-09-16T15:47:33.514 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.515 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.515 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.515 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.515 INFO:teuthology.orchestra.run.smithi123.stdout: "server_port": { 2024-09-16T15:47:33.515 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "server_port", 2024-09-16T15:47:33.515 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "int", 2024-09-16T15:47:33.515 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.515 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-09-16T15:47:33.515 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "9283", 2024-09-16T15:47:33.516 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.516 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.516 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.516 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "the port on which the module listens for HTTP requests", 2024-09-16T15:47:33.516 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.516 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.516 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.516 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.516 INFO:teuthology.orchestra.run.smithi123.stdout: "sqlite3_killpoint": { 2024-09-16T15:47:33.517 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "sqlite3_killpoint", 2024-09-16T15:47:33.517 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "int", 2024-09-16T15:47:33.517 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "dev", 2024-09-16T15:47:33.517 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-09-16T15:47:33.517 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "0", 2024-09-16T15:47:33.517 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.517 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.517 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.517 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:33.517 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.517 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.518 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.518 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.518 INFO:teuthology.orchestra.run.smithi123.stdout: "stale_cache_strategy": { 2024-09-16T15:47:33.518 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "stale_cache_strategy", 2024-09-16T15:47:33.518 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-09-16T15:47:33.518 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.518 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-09-16T15:47:33.518 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "log", 2024-09-16T15:47:33.518 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.518 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.519 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.519 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:33.519 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.519 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.519 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.519 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.519 INFO:teuthology.orchestra.run.smithi123.stdout: "standby_behaviour": { 2024-09-16T15:47:33.519 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "standby_behaviour", 2024-09-16T15:47:33.519 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-09-16T15:47:33.519 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.519 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-09-16T15:47:33.520 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "default", 2024-09-16T15:47:33.520 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.520 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.520 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [ 2024-09-16T15:47:33.520 INFO:teuthology.orchestra.run.smithi123.stdout: "default", 2024-09-16T15:47:33.520 INFO:teuthology.orchestra.run.smithi123.stdout: "error" 2024-09-16T15:47:33.520 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-09-16T15:47:33.520 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:33.520 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.520 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.520 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.521 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.521 INFO:teuthology.orchestra.run.smithi123.stdout: "standby_error_status_code": { 2024-09-16T15:47:33.521 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "standby_error_status_code", 2024-09-16T15:47:33.521 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "int", 2024-09-16T15:47:33.521 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.521 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-09-16T15:47:33.521 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "500", 2024-09-16T15:47:33.521 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "400", 2024-09-16T15:47:33.522 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "599", 2024-09-16T15:47:33.522 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.522 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:33.522 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.522 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.522 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.522 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-16T15:47:33.522 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-16T15:47:33.522 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.522 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-09-16T15:47:33.522 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "rbd_support", 2024-09-16T15:47:33.522 INFO:teuthology.orchestra.run.smithi123.stdout: "can_run": true, 2024-09-16T15:47:33.522 INFO:teuthology.orchestra.run.smithi123.stdout: "error_string": "", 2024-09-16T15:47:33.522 INFO:teuthology.orchestra.run.smithi123.stdout: "module_options": { 2024-09-16T15:47:33.523 INFO:teuthology.orchestra.run.smithi123.stdout: "log_level": { 2024-09-16T15:47:33.523 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_level", 2024-09-16T15:47:33.523 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-09-16T15:47:33.523 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.523 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-09-16T15:47:33.523 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "", 2024-09-16T15:47:33.523 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.523 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.523 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [ 2024-09-16T15:47:33.523 INFO:teuthology.orchestra.run.smithi123.stdout: "", 2024-09-16T15:47:33.523 INFO:teuthology.orchestra.run.smithi123.stdout: "critical", 2024-09-16T15:47:33.523 INFO:teuthology.orchestra.run.smithi123.stdout: "debug", 2024-09-16T15:47:33.523 INFO:teuthology.orchestra.run.smithi123.stdout: "error", 2024-09-16T15:47:33.524 INFO:teuthology.orchestra.run.smithi123.stdout: "info", 2024-09-16T15:47:33.524 INFO:teuthology.orchestra.run.smithi123.stdout: "warning" 2024-09-16T15:47:33.524 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-09-16T15:47:33.524 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:33.524 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.524 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.524 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.524 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.524 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_cluster": { 2024-09-16T15:47:33.524 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_cluster", 2024-09-16T15:47:33.524 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-09-16T15:47:33.524 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.524 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-09-16T15:47:33.524 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-09-16T15:47:33.525 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.525 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.525 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.525 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:33.525 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.525 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.525 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.525 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.525 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_cluster_level": { 2024-09-16T15:47:33.525 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_cluster_level", 2024-09-16T15:47:33.525 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-09-16T15:47:33.525 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.525 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-09-16T15:47:33.525 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "info", 2024-09-16T15:47:33.525 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.526 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.526 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [ 2024-09-16T15:47:33.526 INFO:teuthology.orchestra.run.smithi123.stdout: "", 2024-09-16T15:47:33.526 INFO:teuthology.orchestra.run.smithi123.stdout: "critical", 2024-09-16T15:47:33.526 INFO:teuthology.orchestra.run.smithi123.stdout: "debug", 2024-09-16T15:47:33.526 INFO:teuthology.orchestra.run.smithi123.stdout: "error", 2024-09-16T15:47:33.526 INFO:teuthology.orchestra.run.smithi123.stdout: "info", 2024-09-16T15:47:33.526 INFO:teuthology.orchestra.run.smithi123.stdout: "warning" 2024-09-16T15:47:33.526 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-09-16T15:47:33.526 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:33.526 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.526 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.526 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.526 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.526 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_file": { 2024-09-16T15:47:33.527 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_file", 2024-09-16T15:47:33.527 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-09-16T15:47:33.527 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.527 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-09-16T15:47:33.527 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-09-16T15:47:33.527 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.527 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.527 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.527 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:33.527 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.527 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.527 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.527 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.527 INFO:teuthology.orchestra.run.smithi123.stdout: "max_concurrent_snap_create": { 2024-09-16T15:47:33.527 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "max_concurrent_snap_create", 2024-09-16T15:47:33.528 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "int", 2024-09-16T15:47:33.528 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.528 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-09-16T15:47:33.528 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "10", 2024-09-16T15:47:33.528 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.528 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.528 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.528 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:33.528 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.528 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.528 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.528 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.528 INFO:teuthology.orchestra.run.smithi123.stdout: "mirror_snapshot_schedule": { 2024-09-16T15:47:33.528 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "mirror_snapshot_schedule", 2024-09-16T15:47:33.529 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-09-16T15:47:33.529 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.529 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-09-16T15:47:33.529 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "", 2024-09-16T15:47:33.529 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.529 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.529 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.529 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:33.529 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.529 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.529 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.529 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.529 INFO:teuthology.orchestra.run.smithi123.stdout: "sqlite3_killpoint": { 2024-09-16T15:47:33.529 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "sqlite3_killpoint", 2024-09-16T15:47:33.529 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "int", 2024-09-16T15:47:33.530 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "dev", 2024-09-16T15:47:33.530 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-09-16T15:47:33.530 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "0", 2024-09-16T15:47:33.530 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.530 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.530 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.530 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:33.530 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.530 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.530 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.530 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.530 INFO:teuthology.orchestra.run.smithi123.stdout: "trash_purge_schedule": { 2024-09-16T15:47:33.530 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "trash_purge_schedule", 2024-09-16T15:47:33.530 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-09-16T15:47:33.530 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.531 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-09-16T15:47:33.531 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "", 2024-09-16T15:47:33.531 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.531 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.531 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.531 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:33.531 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.531 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.531 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.531 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-16T15:47:33.531 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-16T15:47:33.531 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.531 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-09-16T15:47:33.531 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "restful", 2024-09-16T15:47:33.532 INFO:teuthology.orchestra.run.smithi123.stdout: "can_run": true, 2024-09-16T15:47:33.532 INFO:teuthology.orchestra.run.smithi123.stdout: "error_string": "", 2024-09-16T15:47:33.532 INFO:teuthology.orchestra.run.smithi123.stdout: "module_options": { 2024-09-16T15:47:33.532 INFO:teuthology.orchestra.run.smithi123.stdout: "enable_auth": { 2024-09-16T15:47:33.532 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "enable_auth", 2024-09-16T15:47:33.532 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-09-16T15:47:33.532 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.532 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-09-16T15:47:33.532 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "True", 2024-09-16T15:47:33.532 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.532 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.532 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.532 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:33.532 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.532 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.533 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.533 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.533 INFO:teuthology.orchestra.run.smithi123.stdout: "key_file": { 2024-09-16T15:47:33.533 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "key_file", 2024-09-16T15:47:33.533 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-09-16T15:47:33.533 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.533 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-09-16T15:47:33.533 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "", 2024-09-16T15:47:33.533 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.533 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.533 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.533 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:33.533 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.533 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.533 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.533 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.534 INFO:teuthology.orchestra.run.smithi123.stdout: "log_level": { 2024-09-16T15:47:33.534 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_level", 2024-09-16T15:47:33.534 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-09-16T15:47:33.534 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.534 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-09-16T15:47:33.534 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "", 2024-09-16T15:47:33.534 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.534 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.534 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [ 2024-09-16T15:47:33.534 INFO:teuthology.orchestra.run.smithi123.stdout: "", 2024-09-16T15:47:33.534 INFO:teuthology.orchestra.run.smithi123.stdout: "critical", 2024-09-16T15:47:33.534 INFO:teuthology.orchestra.run.smithi123.stdout: "debug", 2024-09-16T15:47:33.534 INFO:teuthology.orchestra.run.smithi123.stdout: "error", 2024-09-16T15:47:33.534 INFO:teuthology.orchestra.run.smithi123.stdout: "info", 2024-09-16T15:47:33.534 INFO:teuthology.orchestra.run.smithi123.stdout: "warning" 2024-09-16T15:47:33.535 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-09-16T15:47:33.535 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:33.535 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.535 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.535 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.535 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.535 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_cluster": { 2024-09-16T15:47:33.535 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_cluster", 2024-09-16T15:47:33.535 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-09-16T15:47:33.535 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.535 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-09-16T15:47:33.535 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-09-16T15:47:33.535 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.535 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.535 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.536 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:33.536 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.536 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.536 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.536 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.536 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_cluster_level": { 2024-09-16T15:47:33.536 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_cluster_level", 2024-09-16T15:47:33.536 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-09-16T15:47:33.536 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.536 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-09-16T15:47:33.536 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "info", 2024-09-16T15:47:33.536 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.536 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.536 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [ 2024-09-16T15:47:33.537 INFO:teuthology.orchestra.run.smithi123.stdout: "", 2024-09-16T15:47:33.537 INFO:teuthology.orchestra.run.smithi123.stdout: "critical", 2024-09-16T15:47:33.537 INFO:teuthology.orchestra.run.smithi123.stdout: "debug", 2024-09-16T15:47:33.537 INFO:teuthology.orchestra.run.smithi123.stdout: "error", 2024-09-16T15:47:33.537 INFO:teuthology.orchestra.run.smithi123.stdout: "info", 2024-09-16T15:47:33.537 INFO:teuthology.orchestra.run.smithi123.stdout: "warning" 2024-09-16T15:47:33.537 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-09-16T15:47:33.537 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:33.537 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.537 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.537 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.537 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.537 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_file": { 2024-09-16T15:47:33.537 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_file", 2024-09-16T15:47:33.537 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-09-16T15:47:33.537 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.538 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-09-16T15:47:33.538 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-09-16T15:47:33.538 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.538 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.538 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.538 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:33.538 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.538 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.538 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.538 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.538 INFO:teuthology.orchestra.run.smithi123.stdout: "max_requests": { 2024-09-16T15:47:33.538 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "max_requests", 2024-09-16T15:47:33.538 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "int", 2024-09-16T15:47:33.538 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.538 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-09-16T15:47:33.539 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "500", 2024-09-16T15:47:33.539 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.539 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.539 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.539 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "Maximum number of requests to keep in memory. When new request comes in, the oldest request will be removed if the number of requests exceeds the max request number.if un-finished request is removed, error message will be logged in the ceph-mgr log.", 2024-09-16T15:47:33.539 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.539 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.539 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.539 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.539 INFO:teuthology.orchestra.run.smithi123.stdout: "server_addr": { 2024-09-16T15:47:33.539 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "server_addr", 2024-09-16T15:47:33.539 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-09-16T15:47:33.539 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.539 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-09-16T15:47:33.539 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "", 2024-09-16T15:47:33.540 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.540 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.540 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.540 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:33.540 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.540 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.540 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.540 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.540 INFO:teuthology.orchestra.run.smithi123.stdout: "server_port": { 2024-09-16T15:47:33.540 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "server_port", 2024-09-16T15:47:33.540 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-09-16T15:47:33.540 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.540 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-09-16T15:47:33.540 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "", 2024-09-16T15:47:33.540 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.541 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.541 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.541 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:33.541 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.541 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.541 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.541 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.541 INFO:teuthology.orchestra.run.smithi123.stdout: "sqlite3_killpoint": { 2024-09-16T15:47:33.541 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "sqlite3_killpoint", 2024-09-16T15:47:33.541 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "int", 2024-09-16T15:47:33.541 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "dev", 2024-09-16T15:47:33.541 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-09-16T15:47:33.541 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "0", 2024-09-16T15:47:33.541 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.542 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.542 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.542 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:33.542 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.542 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.542 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.542 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-16T15:47:33.542 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-16T15:47:33.542 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.542 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-09-16T15:47:33.542 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "rgw", 2024-09-16T15:47:33.542 INFO:teuthology.orchestra.run.smithi123.stdout: "can_run": true, 2024-09-16T15:47:33.542 INFO:teuthology.orchestra.run.smithi123.stdout: "error_string": "", 2024-09-16T15:47:33.543 INFO:teuthology.orchestra.run.smithi123.stdout: "module_options": { 2024-09-16T15:47:33.543 INFO:teuthology.orchestra.run.smithi123.stdout: "log_level": { 2024-09-16T15:47:33.543 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_level", 2024-09-16T15:47:33.543 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-09-16T15:47:33.543 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.543 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-09-16T15:47:33.543 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "", 2024-09-16T15:47:33.543 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.543 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.543 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [ 2024-09-16T15:47:33.543 INFO:teuthology.orchestra.run.smithi123.stdout: "", 2024-09-16T15:47:33.543 INFO:teuthology.orchestra.run.smithi123.stdout: "critical", 2024-09-16T15:47:33.543 INFO:teuthology.orchestra.run.smithi123.stdout: "debug", 2024-09-16T15:47:33.543 INFO:teuthology.orchestra.run.smithi123.stdout: "error", 2024-09-16T15:47:33.543 INFO:teuthology.orchestra.run.smithi123.stdout: "info", 2024-09-16T15:47:33.544 INFO:teuthology.orchestra.run.smithi123.stdout: "warning" 2024-09-16T15:47:33.544 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-09-16T15:47:33.544 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:33.544 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.544 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.544 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.544 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.544 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_cluster": { 2024-09-16T15:47:33.544 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_cluster", 2024-09-16T15:47:33.544 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-09-16T15:47:33.544 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.544 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-09-16T15:47:33.544 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-09-16T15:47:33.544 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.545 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.545 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.545 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:33.545 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.545 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.545 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.545 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.545 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_cluster_level": { 2024-09-16T15:47:33.545 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_cluster_level", 2024-09-16T15:47:33.545 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-09-16T15:47:33.545 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.545 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-09-16T15:47:33.545 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "info", 2024-09-16T15:47:33.545 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.545 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.546 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [ 2024-09-16T15:47:33.546 INFO:teuthology.orchestra.run.smithi123.stdout: "", 2024-09-16T15:47:33.546 INFO:teuthology.orchestra.run.smithi123.stdout: "critical", 2024-09-16T15:47:33.546 INFO:teuthology.orchestra.run.smithi123.stdout: "debug", 2024-09-16T15:47:33.546 INFO:teuthology.orchestra.run.smithi123.stdout: "error", 2024-09-16T15:47:33.546 INFO:teuthology.orchestra.run.smithi123.stdout: "info", 2024-09-16T15:47:33.546 INFO:teuthology.orchestra.run.smithi123.stdout: "warning" 2024-09-16T15:47:33.546 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-09-16T15:47:33.546 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:33.546 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.546 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.546 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.546 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.546 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_file": { 2024-09-16T15:47:33.546 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_file", 2024-09-16T15:47:33.547 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-09-16T15:47:33.547 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.547 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-09-16T15:47:33.547 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-09-16T15:47:33.547 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.547 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.547 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.547 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:33.547 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.547 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.547 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.547 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.547 INFO:teuthology.orchestra.run.smithi123.stdout: "secondary_zone_period_retry_limit": { 2024-09-16T15:47:33.547 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "secondary_zone_period_retry_limit", 2024-09-16T15:47:33.547 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "int", 2024-09-16T15:47:33.548 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.548 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-09-16T15:47:33.548 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "5", 2024-09-16T15:47:33.548 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.548 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.548 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.548 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "RGW module period update retry limit for secondary site", 2024-09-16T15:47:33.548 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.548 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.548 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.548 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.548 INFO:teuthology.orchestra.run.smithi123.stdout: "sqlite3_killpoint": { 2024-09-16T15:47:33.548 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "sqlite3_killpoint", 2024-09-16T15:47:33.548 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "int", 2024-09-16T15:47:33.548 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "dev", 2024-09-16T15:47:33.548 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-09-16T15:47:33.549 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "0", 2024-09-16T15:47:33.549 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.549 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.549 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.549 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:33.549 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.549 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.549 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.549 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-16T15:47:33.549 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-16T15:47:33.549 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.549 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-09-16T15:47:33.549 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "selftest", 2024-09-16T15:47:33.549 INFO:teuthology.orchestra.run.smithi123.stdout: "can_run": true, 2024-09-16T15:47:33.549 INFO:teuthology.orchestra.run.smithi123.stdout: "error_string": "", 2024-09-16T15:47:33.550 INFO:teuthology.orchestra.run.smithi123.stdout: "module_options": { 2024-09-16T15:47:33.550 INFO:teuthology.orchestra.run.smithi123.stdout: "log_level": { 2024-09-16T15:47:33.550 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_level", 2024-09-16T15:47:33.550 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-09-16T15:47:33.550 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.550 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-09-16T15:47:33.550 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "", 2024-09-16T15:47:33.550 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.550 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.550 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [ 2024-09-16T15:47:33.550 INFO:teuthology.orchestra.run.smithi123.stdout: "", 2024-09-16T15:47:33.550 INFO:teuthology.orchestra.run.smithi123.stdout: "critical", 2024-09-16T15:47:33.550 INFO:teuthology.orchestra.run.smithi123.stdout: "debug", 2024-09-16T15:47:33.550 INFO:teuthology.orchestra.run.smithi123.stdout: "error", 2024-09-16T15:47:33.550 INFO:teuthology.orchestra.run.smithi123.stdout: "info", 2024-09-16T15:47:33.551 INFO:teuthology.orchestra.run.smithi123.stdout: "warning" 2024-09-16T15:47:33.551 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-09-16T15:47:33.551 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:33.551 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.551 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.551 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.551 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.551 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_cluster": { 2024-09-16T15:47:33.551 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_cluster", 2024-09-16T15:47:33.551 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-09-16T15:47:33.551 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.551 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-09-16T15:47:33.551 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-09-16T15:47:33.551 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.551 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.551 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.552 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:33.552 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.552 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.552 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.552 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.552 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_cluster_level": { 2024-09-16T15:47:33.552 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_cluster_level", 2024-09-16T15:47:33.552 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-09-16T15:47:33.552 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.552 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-09-16T15:47:33.552 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "info", 2024-09-16T15:47:33.552 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.552 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.552 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [ 2024-09-16T15:47:33.553 INFO:teuthology.orchestra.run.smithi123.stdout: "", 2024-09-16T15:47:33.553 INFO:teuthology.orchestra.run.smithi123.stdout: "critical", 2024-09-16T15:47:33.553 INFO:teuthology.orchestra.run.smithi123.stdout: "debug", 2024-09-16T15:47:33.553 INFO:teuthology.orchestra.run.smithi123.stdout: "error", 2024-09-16T15:47:33.553 INFO:teuthology.orchestra.run.smithi123.stdout: "info", 2024-09-16T15:47:33.553 INFO:teuthology.orchestra.run.smithi123.stdout: "warning" 2024-09-16T15:47:33.553 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-09-16T15:47:33.553 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:33.553 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.553 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.553 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.553 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.553 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_file": { 2024-09-16T15:47:33.553 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_file", 2024-09-16T15:47:33.553 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-09-16T15:47:33.553 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.554 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-09-16T15:47:33.554 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-09-16T15:47:33.554 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.554 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.554 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.554 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:33.554 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.554 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.554 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.554 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.554 INFO:teuthology.orchestra.run.smithi123.stdout: "roption1": { 2024-09-16T15:47:33.554 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "roption1", 2024-09-16T15:47:33.554 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-09-16T15:47:33.554 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.554 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-09-16T15:47:33.555 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "", 2024-09-16T15:47:33.555 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.555 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.555 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.555 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:33.555 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.555 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.555 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.555 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.555 INFO:teuthology.orchestra.run.smithi123.stdout: "roption2": { 2024-09-16T15:47:33.555 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "roption2", 2024-09-16T15:47:33.555 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-09-16T15:47:33.555 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.555 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-09-16T15:47:33.555 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "xyz", 2024-09-16T15:47:33.556 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.556 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.556 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.556 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:33.556 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.556 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.556 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.556 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.556 INFO:teuthology.orchestra.run.smithi123.stdout: "rwoption1": { 2024-09-16T15:47:33.556 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "rwoption1", 2024-09-16T15:47:33.556 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-09-16T15:47:33.556 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.556 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-09-16T15:47:33.556 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "", 2024-09-16T15:47:33.556 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.557 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.557 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.557 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:33.557 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.557 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.557 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.557 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.557 INFO:teuthology.orchestra.run.smithi123.stdout: "rwoption2": { 2024-09-16T15:47:33.557 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "rwoption2", 2024-09-16T15:47:33.557 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "int", 2024-09-16T15:47:33.557 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.557 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-09-16T15:47:33.557 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "", 2024-09-16T15:47:33.557 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.557 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.558 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.558 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:33.558 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.558 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.558 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.558 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.558 INFO:teuthology.orchestra.run.smithi123.stdout: "rwoption3": { 2024-09-16T15:47:33.558 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "rwoption3", 2024-09-16T15:47:33.558 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "float", 2024-09-16T15:47:33.558 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.558 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-09-16T15:47:33.558 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "", 2024-09-16T15:47:33.558 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.558 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.558 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.558 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:33.559 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.559 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.559 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.559 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.559 INFO:teuthology.orchestra.run.smithi123.stdout: "rwoption4": { 2024-09-16T15:47:33.559 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "rwoption4", 2024-09-16T15:47:33.559 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-09-16T15:47:33.559 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.559 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-09-16T15:47:33.559 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "", 2024-09-16T15:47:33.559 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.559 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.559 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.559 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:33.560 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.560 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.560 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.560 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.560 INFO:teuthology.orchestra.run.smithi123.stdout: "rwoption5": { 2024-09-16T15:47:33.560 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "rwoption5", 2024-09-16T15:47:33.560 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-09-16T15:47:33.560 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.560 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-09-16T15:47:33.560 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "", 2024-09-16T15:47:33.560 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.560 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.560 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.560 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:33.561 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.561 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.561 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.561 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.561 INFO:teuthology.orchestra.run.smithi123.stdout: "rwoption6": { 2024-09-16T15:47:33.561 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "rwoption6", 2024-09-16T15:47:33.561 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-09-16T15:47:33.561 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.561 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-09-16T15:47:33.561 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "True", 2024-09-16T15:47:33.561 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.561 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.561 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.561 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:33.561 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.561 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.562 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.562 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.562 INFO:teuthology.orchestra.run.smithi123.stdout: "rwoption7": { 2024-09-16T15:47:33.562 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "rwoption7", 2024-09-16T15:47:33.562 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "int", 2024-09-16T15:47:33.562 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.562 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-09-16T15:47:33.562 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "", 2024-09-16T15:47:33.562 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "1", 2024-09-16T15:47:33.562 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "42", 2024-09-16T15:47:33.562 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.562 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:33.562 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.562 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.562 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.563 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.563 INFO:teuthology.orchestra.run.smithi123.stdout: "sqlite3_killpoint": { 2024-09-16T15:47:33.563 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "sqlite3_killpoint", 2024-09-16T15:47:33.563 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "int", 2024-09-16T15:47:33.563 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "dev", 2024-09-16T15:47:33.563 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-09-16T15:47:33.563 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "0", 2024-09-16T15:47:33.563 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.563 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.563 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.563 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:33.563 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.563 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.563 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.563 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.564 INFO:teuthology.orchestra.run.smithi123.stdout: "testkey": { 2024-09-16T15:47:33.564 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "testkey", 2024-09-16T15:47:33.564 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-09-16T15:47:33.564 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.564 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-09-16T15:47:33.564 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "", 2024-09-16T15:47:33.564 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.564 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.564 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.564 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:33.564 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.564 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.564 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.564 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.564 INFO:teuthology.orchestra.run.smithi123.stdout: "testlkey": { 2024-09-16T15:47:33.564 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "testlkey", 2024-09-16T15:47:33.565 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-09-16T15:47:33.565 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.565 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-09-16T15:47:33.565 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "", 2024-09-16T15:47:33.565 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.565 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.565 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.565 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:33.565 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.565 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.565 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.565 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.565 INFO:teuthology.orchestra.run.smithi123.stdout: "testnewline": { 2024-09-16T15:47:33.565 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "testnewline", 2024-09-16T15:47:33.565 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-09-16T15:47:33.566 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.566 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-09-16T15:47:33.566 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "", 2024-09-16T15:47:33.566 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.566 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.566 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.566 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:33.566 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.566 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.566 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.566 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-16T15:47:33.566 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-16T15:47:33.566 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.566 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-09-16T15:47:33.566 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "snap_schedule", 2024-09-16T15:47:33.567 INFO:teuthology.orchestra.run.smithi123.stdout: "can_run": true, 2024-09-16T15:47:33.567 INFO:teuthology.orchestra.run.smithi123.stdout: "error_string": "", 2024-09-16T15:47:33.567 INFO:teuthology.orchestra.run.smithi123.stdout: "module_options": { 2024-09-16T15:47:33.567 INFO:teuthology.orchestra.run.smithi123.stdout: "allow_m_granularity": { 2024-09-16T15:47:33.567 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "allow_m_granularity", 2024-09-16T15:47:33.567 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-09-16T15:47:33.567 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.567 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-09-16T15:47:33.567 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-09-16T15:47:33.567 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.567 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.567 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.567 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "allow minute scheduled snapshots", 2024-09-16T15:47:33.567 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.567 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.567 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.568 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.568 INFO:teuthology.orchestra.run.smithi123.stdout: "dump_on_update": { 2024-09-16T15:47:33.568 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "dump_on_update", 2024-09-16T15:47:33.568 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-09-16T15:47:33.568 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.568 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-09-16T15:47:33.568 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-09-16T15:47:33.568 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.568 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.568 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.568 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "dump database to debug log on update", 2024-09-16T15:47:33.568 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.568 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.568 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.568 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.569 INFO:teuthology.orchestra.run.smithi123.stdout: "log_level": { 2024-09-16T15:47:33.569 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_level", 2024-09-16T15:47:33.569 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-09-16T15:47:33.569 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.569 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-09-16T15:47:33.569 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "", 2024-09-16T15:47:33.569 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.569 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.569 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [ 2024-09-16T15:47:33.569 INFO:teuthology.orchestra.run.smithi123.stdout: "", 2024-09-16T15:47:33.569 INFO:teuthology.orchestra.run.smithi123.stdout: "critical", 2024-09-16T15:47:33.569 INFO:teuthology.orchestra.run.smithi123.stdout: "debug", 2024-09-16T15:47:33.569 INFO:teuthology.orchestra.run.smithi123.stdout: "error", 2024-09-16T15:47:33.569 INFO:teuthology.orchestra.run.smithi123.stdout: "info", 2024-09-16T15:47:33.569 INFO:teuthology.orchestra.run.smithi123.stdout: "warning" 2024-09-16T15:47:33.570 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-09-16T15:47:33.570 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:33.570 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.570 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.570 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.570 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.570 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_cluster": { 2024-09-16T15:47:33.570 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_cluster", 2024-09-16T15:47:33.570 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-09-16T15:47:33.570 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.570 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-09-16T15:47:33.570 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-09-16T15:47:33.570 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.570 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.571 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.571 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:33.571 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.571 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.571 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.571 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.571 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_cluster_level": { 2024-09-16T15:47:33.571 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_cluster_level", 2024-09-16T15:47:33.571 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-09-16T15:47:33.571 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.571 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-09-16T15:47:33.571 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "info", 2024-09-16T15:47:33.571 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.571 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.572 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [ 2024-09-16T15:47:33.572 INFO:teuthology.orchestra.run.smithi123.stdout: "", 2024-09-16T15:47:33.572 INFO:teuthology.orchestra.run.smithi123.stdout: "critical", 2024-09-16T15:47:33.572 INFO:teuthology.orchestra.run.smithi123.stdout: "debug", 2024-09-16T15:47:33.572 INFO:teuthology.orchestra.run.smithi123.stdout: "error", 2024-09-16T15:47:33.572 INFO:teuthology.orchestra.run.smithi123.stdout: "info", 2024-09-16T15:47:33.572 INFO:teuthology.orchestra.run.smithi123.stdout: "warning" 2024-09-16T15:47:33.572 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-09-16T15:47:33.572 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:33.572 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.572 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.572 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.572 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.573 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_file": { 2024-09-16T15:47:33.573 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_file", 2024-09-16T15:47:33.573 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-09-16T15:47:33.573 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.573 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-09-16T15:47:33.573 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-09-16T15:47:33.573 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.573 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.573 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.573 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:33.573 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.573 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.573 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.573 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.573 INFO:teuthology.orchestra.run.smithi123.stdout: "sqlite3_killpoint": { 2024-09-16T15:47:33.574 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "sqlite3_killpoint", 2024-09-16T15:47:33.574 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "int", 2024-09-16T15:47:33.574 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "dev", 2024-09-16T15:47:33.574 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-09-16T15:47:33.574 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "0", 2024-09-16T15:47:33.574 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.574 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.574 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.574 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:33.574 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.574 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.574 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.574 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-16T15:47:33.574 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-16T15:47:33.574 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.575 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-09-16T15:47:33.575 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "stats", 2024-09-16T15:47:33.575 INFO:teuthology.orchestra.run.smithi123.stdout: "can_run": true, 2024-09-16T15:47:33.575 INFO:teuthology.orchestra.run.smithi123.stdout: "error_string": "", 2024-09-16T15:47:33.575 INFO:teuthology.orchestra.run.smithi123.stdout: "module_options": { 2024-09-16T15:47:33.575 INFO:teuthology.orchestra.run.smithi123.stdout: "log_level": { 2024-09-16T15:47:33.575 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_level", 2024-09-16T15:47:33.575 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-09-16T15:47:33.575 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.575 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-09-16T15:47:33.575 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "", 2024-09-16T15:47:33.575 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.575 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.575 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [ 2024-09-16T15:47:33.575 INFO:teuthology.orchestra.run.smithi123.stdout: "", 2024-09-16T15:47:33.576 INFO:teuthology.orchestra.run.smithi123.stdout: "critical", 2024-09-16T15:47:33.576 INFO:teuthology.orchestra.run.smithi123.stdout: "debug", 2024-09-16T15:47:33.576 INFO:teuthology.orchestra.run.smithi123.stdout: "error", 2024-09-16T15:47:33.576 INFO:teuthology.orchestra.run.smithi123.stdout: "info", 2024-09-16T15:47:33.576 INFO:teuthology.orchestra.run.smithi123.stdout: "warning" 2024-09-16T15:47:33.576 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-09-16T15:47:33.576 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:33.576 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.576 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.576 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.576 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.576 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_cluster": { 2024-09-16T15:47:33.576 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_cluster", 2024-09-16T15:47:33.576 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-09-16T15:47:33.577 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.577 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-09-16T15:47:33.577 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-09-16T15:47:33.577 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.577 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.577 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.577 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:33.577 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.577 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.577 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.577 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.577 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_cluster_level": { 2024-09-16T15:47:33.577 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_cluster_level", 2024-09-16T15:47:33.577 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-09-16T15:47:33.578 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.578 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-09-16T15:47:33.578 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "info", 2024-09-16T15:47:33.578 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.578 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.578 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [ 2024-09-16T15:47:33.578 INFO:teuthology.orchestra.run.smithi123.stdout: "", 2024-09-16T15:47:33.578 INFO:teuthology.orchestra.run.smithi123.stdout: "critical", 2024-09-16T15:47:33.578 INFO:teuthology.orchestra.run.smithi123.stdout: "debug", 2024-09-16T15:47:33.578 INFO:teuthology.orchestra.run.smithi123.stdout: "error", 2024-09-16T15:47:33.578 INFO:teuthology.orchestra.run.smithi123.stdout: "info", 2024-09-16T15:47:33.578 INFO:teuthology.orchestra.run.smithi123.stdout: "warning" 2024-09-16T15:47:33.578 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-09-16T15:47:33.578 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:33.578 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.579 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.579 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.579 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.579 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_file": { 2024-09-16T15:47:33.579 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_file", 2024-09-16T15:47:33.579 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-09-16T15:47:33.579 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.579 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-09-16T15:47:33.579 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-09-16T15:47:33.579 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.579 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.579 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.579 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:33.579 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.579 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.579 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.580 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.580 INFO:teuthology.orchestra.run.smithi123.stdout: "sqlite3_killpoint": { 2024-09-16T15:47:33.580 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "sqlite3_killpoint", 2024-09-16T15:47:33.580 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "int", 2024-09-16T15:47:33.580 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "dev", 2024-09-16T15:47:33.580 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-09-16T15:47:33.580 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "0", 2024-09-16T15:47:33.580 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.580 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.580 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.580 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:33.580 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.580 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.580 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.580 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-16T15:47:33.581 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-16T15:47:33.581 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.581 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-09-16T15:47:33.581 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "status", 2024-09-16T15:47:33.581 INFO:teuthology.orchestra.run.smithi123.stdout: "can_run": true, 2024-09-16T15:47:33.581 INFO:teuthology.orchestra.run.smithi123.stdout: "error_string": "", 2024-09-16T15:47:33.581 INFO:teuthology.orchestra.run.smithi123.stdout: "module_options": { 2024-09-16T15:47:33.581 INFO:teuthology.orchestra.run.smithi123.stdout: "log_level": { 2024-09-16T15:47:33.581 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_level", 2024-09-16T15:47:33.581 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-09-16T15:47:33.581 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.581 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-09-16T15:47:33.581 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "", 2024-09-16T15:47:33.581 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.581 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.582 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [ 2024-09-16T15:47:33.582 INFO:teuthology.orchestra.run.smithi123.stdout: "", 2024-09-16T15:47:33.582 INFO:teuthology.orchestra.run.smithi123.stdout: "critical", 2024-09-16T15:47:33.582 INFO:teuthology.orchestra.run.smithi123.stdout: "debug", 2024-09-16T15:47:33.582 INFO:teuthology.orchestra.run.smithi123.stdout: "error", 2024-09-16T15:47:33.582 INFO:teuthology.orchestra.run.smithi123.stdout: "info", 2024-09-16T15:47:33.582 INFO:teuthology.orchestra.run.smithi123.stdout: "warning" 2024-09-16T15:47:33.582 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-09-16T15:47:33.582 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:33.582 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.582 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.582 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.582 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.582 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_cluster": { 2024-09-16T15:47:33.582 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_cluster", 2024-09-16T15:47:33.583 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-09-16T15:47:33.583 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.583 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-09-16T15:47:33.583 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-09-16T15:47:33.583 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.583 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.583 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.583 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:33.583 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.583 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.583 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.583 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.583 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_cluster_level": { 2024-09-16T15:47:33.583 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_cluster_level", 2024-09-16T15:47:33.583 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-09-16T15:47:33.583 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.584 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-09-16T15:47:33.584 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "info", 2024-09-16T15:47:33.584 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.584 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.584 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [ 2024-09-16T15:47:33.584 INFO:teuthology.orchestra.run.smithi123.stdout: "", 2024-09-16T15:47:33.584 INFO:teuthology.orchestra.run.smithi123.stdout: "critical", 2024-09-16T15:47:33.584 INFO:teuthology.orchestra.run.smithi123.stdout: "debug", 2024-09-16T15:47:33.584 INFO:teuthology.orchestra.run.smithi123.stdout: "error", 2024-09-16T15:47:33.584 INFO:teuthology.orchestra.run.smithi123.stdout: "info", 2024-09-16T15:47:33.584 INFO:teuthology.orchestra.run.smithi123.stdout: "warning" 2024-09-16T15:47:33.584 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-09-16T15:47:33.584 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:33.584 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.584 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.585 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.585 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.585 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_file": { 2024-09-16T15:47:33.585 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_file", 2024-09-16T15:47:33.585 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-09-16T15:47:33.585 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.585 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-09-16T15:47:33.585 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-09-16T15:47:33.585 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.585 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.585 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.585 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:33.585 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.585 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.585 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.586 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.586 INFO:teuthology.orchestra.run.smithi123.stdout: "sqlite3_killpoint": { 2024-09-16T15:47:33.586 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "sqlite3_killpoint", 2024-09-16T15:47:33.586 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "int", 2024-09-16T15:47:33.586 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "dev", 2024-09-16T15:47:33.586 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-09-16T15:47:33.586 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "0", 2024-09-16T15:47:33.586 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.586 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.586 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.586 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:33.586 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.586 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.586 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.586 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-16T15:47:33.587 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-16T15:47:33.587 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.587 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-09-16T15:47:33.587 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "telegraf", 2024-09-16T15:47:33.587 INFO:teuthology.orchestra.run.smithi123.stdout: "can_run": true, 2024-09-16T15:47:33.587 INFO:teuthology.orchestra.run.smithi123.stdout: "error_string": "", 2024-09-16T15:47:33.587 INFO:teuthology.orchestra.run.smithi123.stdout: "module_options": { 2024-09-16T15:47:33.587 INFO:teuthology.orchestra.run.smithi123.stdout: "address": { 2024-09-16T15:47:33.587 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "address", 2024-09-16T15:47:33.587 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-09-16T15:47:33.587 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.587 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-09-16T15:47:33.587 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "unixgram:///tmp/telegraf.sock", 2024-09-16T15:47:33.587 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.587 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.588 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.588 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:33.588 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.588 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.588 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.588 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.588 INFO:teuthology.orchestra.run.smithi123.stdout: "interval": { 2024-09-16T15:47:33.588 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "interval", 2024-09-16T15:47:33.588 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "secs", 2024-09-16T15:47:33.588 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.588 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-09-16T15:47:33.588 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "15", 2024-09-16T15:47:33.588 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.588 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.588 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.589 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:33.589 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.589 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.589 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.589 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.589 INFO:teuthology.orchestra.run.smithi123.stdout: "log_level": { 2024-09-16T15:47:33.589 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_level", 2024-09-16T15:47:33.589 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-09-16T15:47:33.589 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.589 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-09-16T15:47:33.589 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "", 2024-09-16T15:47:33.589 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.589 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.589 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [ 2024-09-16T15:47:33.589 INFO:teuthology.orchestra.run.smithi123.stdout: "", 2024-09-16T15:47:33.590 INFO:teuthology.orchestra.run.smithi123.stdout: "critical", 2024-09-16T15:47:33.590 INFO:teuthology.orchestra.run.smithi123.stdout: "debug", 2024-09-16T15:47:33.590 INFO:teuthology.orchestra.run.smithi123.stdout: "error", 2024-09-16T15:47:33.590 INFO:teuthology.orchestra.run.smithi123.stdout: "info", 2024-09-16T15:47:33.590 INFO:teuthology.orchestra.run.smithi123.stdout: "warning" 2024-09-16T15:47:33.590 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-09-16T15:47:33.590 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:33.590 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.590 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.590 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.590 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.590 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_cluster": { 2024-09-16T15:47:33.590 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_cluster", 2024-09-16T15:47:33.590 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-09-16T15:47:33.590 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.590 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-09-16T15:47:33.591 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-09-16T15:47:33.591 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.591 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.591 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.591 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:33.591 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.591 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.591 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.591 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.591 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_cluster_level": { 2024-09-16T15:47:33.591 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_cluster_level", 2024-09-16T15:47:33.591 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-09-16T15:47:33.591 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.591 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-09-16T15:47:33.591 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "info", 2024-09-16T15:47:33.592 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.592 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.592 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [ 2024-09-16T15:47:33.592 INFO:teuthology.orchestra.run.smithi123.stdout: "", 2024-09-16T15:47:33.592 INFO:teuthology.orchestra.run.smithi123.stdout: "critical", 2024-09-16T15:47:33.592 INFO:teuthology.orchestra.run.smithi123.stdout: "debug", 2024-09-16T15:47:33.592 INFO:teuthology.orchestra.run.smithi123.stdout: "error", 2024-09-16T15:47:33.592 INFO:teuthology.orchestra.run.smithi123.stdout: "info", 2024-09-16T15:47:33.592 INFO:teuthology.orchestra.run.smithi123.stdout: "warning" 2024-09-16T15:47:33.592 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-09-16T15:47:33.592 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:33.592 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.592 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.592 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.592 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.593 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_file": { 2024-09-16T15:47:33.593 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_file", 2024-09-16T15:47:33.593 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-09-16T15:47:33.593 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.593 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-09-16T15:47:33.593 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-09-16T15:47:33.593 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.593 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.593 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.593 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:33.593 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.593 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.593 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.593 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.593 INFO:teuthology.orchestra.run.smithi123.stdout: "sqlite3_killpoint": { 2024-09-16T15:47:33.594 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "sqlite3_killpoint", 2024-09-16T15:47:33.594 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "int", 2024-09-16T15:47:33.594 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "dev", 2024-09-16T15:47:33.594 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-09-16T15:47:33.594 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "0", 2024-09-16T15:47:33.594 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.594 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.594 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.594 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:33.594 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.594 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.594 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.594 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-16T15:47:33.594 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-16T15:47:33.594 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.595 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-09-16T15:47:33.595 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "telemetry", 2024-09-16T15:47:33.595 INFO:teuthology.orchestra.run.smithi123.stdout: "can_run": true, 2024-09-16T15:47:33.595 INFO:teuthology.orchestra.run.smithi123.stdout: "error_string": "", 2024-09-16T15:47:33.595 INFO:teuthology.orchestra.run.smithi123.stdout: "module_options": { 2024-09-16T15:47:33.595 INFO:teuthology.orchestra.run.smithi123.stdout: "channel_basic": { 2024-09-16T15:47:33.595 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "channel_basic", 2024-09-16T15:47:33.595 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-09-16T15:47:33.595 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.595 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-09-16T15:47:33.595 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "True", 2024-09-16T15:47:33.595 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.595 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.595 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.595 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "Share basic cluster information (size, version)", 2024-09-16T15:47:33.595 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.596 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.596 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.596 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.596 INFO:teuthology.orchestra.run.smithi123.stdout: "channel_crash": { 2024-09-16T15:47:33.596 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "channel_crash", 2024-09-16T15:47:33.596 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-09-16T15:47:33.596 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.596 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-09-16T15:47:33.596 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "True", 2024-09-16T15:47:33.596 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.596 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.596 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.596 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "Share metadata about Ceph daemon crashes (version, stack straces, etc)", 2024-09-16T15:47:33.596 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.597 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.597 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.597 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.597 INFO:teuthology.orchestra.run.smithi123.stdout: "channel_device": { 2024-09-16T15:47:33.597 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "channel_device", 2024-09-16T15:47:33.597 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-09-16T15:47:33.597 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.597 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-09-16T15:47:33.597 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "True", 2024-09-16T15:47:33.597 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.597 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.597 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.597 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)", 2024-09-16T15:47:33.597 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.597 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.598 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.598 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.598 INFO:teuthology.orchestra.run.smithi123.stdout: "channel_ident": { 2024-09-16T15:47:33.598 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "channel_ident", 2024-09-16T15:47:33.598 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-09-16T15:47:33.598 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.598 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-09-16T15:47:33.598 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-09-16T15:47:33.598 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.598 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.598 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.598 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "Share a user-provided description and/or contact email for the cluster", 2024-09-16T15:47:33.598 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.598 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.598 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.599 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.599 INFO:teuthology.orchestra.run.smithi123.stdout: "channel_perf": { 2024-09-16T15:47:33.599 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "channel_perf", 2024-09-16T15:47:33.599 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-09-16T15:47:33.599 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.599 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-09-16T15:47:33.599 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-09-16T15:47:33.599 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.599 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.599 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.599 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "Share various performance metrics of a cluster", 2024-09-16T15:47:33.599 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.599 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.599 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.599 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.599 INFO:teuthology.orchestra.run.smithi123.stdout: "contact": { 2024-09-16T15:47:33.600 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "contact", 2024-09-16T15:47:33.600 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-09-16T15:47:33.600 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.600 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-09-16T15:47:33.600 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "", 2024-09-16T15:47:33.600 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.600 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.600 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.600 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:33.600 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.600 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.600 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.600 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.600 INFO:teuthology.orchestra.run.smithi123.stdout: "description": { 2024-09-16T15:47:33.601 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "description", 2024-09-16T15:47:33.601 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-09-16T15:47:33.601 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.601 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-09-16T15:47:33.601 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "", 2024-09-16T15:47:33.601 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.601 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.601 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.601 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:33.601 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.601 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.601 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.601 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.601 INFO:teuthology.orchestra.run.smithi123.stdout: "device_url": { 2024-09-16T15:47:33.601 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "device_url", 2024-09-16T15:47:33.602 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-09-16T15:47:33.602 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.602 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-09-16T15:47:33.602 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "https://telemetry.ceph.com/device", 2024-09-16T15:47:33.602 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.602 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.602 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.602 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:33.602 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.602 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.602 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.602 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.602 INFO:teuthology.orchestra.run.smithi123.stdout: "enabled": { 2024-09-16T15:47:33.602 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "enabled", 2024-09-16T15:47:33.602 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-09-16T15:47:33.602 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.603 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-09-16T15:47:33.603 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-09-16T15:47:33.603 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.603 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.603 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.603 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:33.603 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.603 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.603 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.603 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.603 INFO:teuthology.orchestra.run.smithi123.stdout: "interval": { 2024-09-16T15:47:33.603 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "interval", 2024-09-16T15:47:33.603 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "int", 2024-09-16T15:47:33.603 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.603 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-09-16T15:47:33.604 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "24", 2024-09-16T15:47:33.604 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "8", 2024-09-16T15:47:33.604 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.604 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.604 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:33.604 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.604 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.604 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.604 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.604 INFO:teuthology.orchestra.run.smithi123.stdout: "last_opt_revision": { 2024-09-16T15:47:33.604 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "last_opt_revision", 2024-09-16T15:47:33.604 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "int", 2024-09-16T15:47:33.604 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.604 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-09-16T15:47:33.605 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "1", 2024-09-16T15:47:33.605 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.605 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.605 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.605 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:33.605 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.605 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.605 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.605 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.605 INFO:teuthology.orchestra.run.smithi123.stdout: "leaderboard": { 2024-09-16T15:47:33.605 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "leaderboard", 2024-09-16T15:47:33.605 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-09-16T15:47:33.605 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.605 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-09-16T15:47:33.605 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-09-16T15:47:33.606 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.606 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.606 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.606 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:33.606 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.606 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.606 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.606 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.606 INFO:teuthology.orchestra.run.smithi123.stdout: "leaderboard_description": { 2024-09-16T15:47:33.606 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "leaderboard_description", 2024-09-16T15:47:33.606 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-09-16T15:47:33.606 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.606 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-09-16T15:47:33.606 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "", 2024-09-16T15:47:33.606 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.607 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.607 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.607 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:33.607 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.607 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.607 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.607 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.607 INFO:teuthology.orchestra.run.smithi123.stdout: "log_level": { 2024-09-16T15:47:33.607 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_level", 2024-09-16T15:47:33.607 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-09-16T15:47:33.607 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.607 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-09-16T15:47:33.607 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "", 2024-09-16T15:47:33.607 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.607 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.608 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [ 2024-09-16T15:47:33.608 INFO:teuthology.orchestra.run.smithi123.stdout: "", 2024-09-16T15:47:33.608 INFO:teuthology.orchestra.run.smithi123.stdout: "critical", 2024-09-16T15:47:33.608 INFO:teuthology.orchestra.run.smithi123.stdout: "debug", 2024-09-16T15:47:33.608 INFO:teuthology.orchestra.run.smithi123.stdout: "error", 2024-09-16T15:47:33.608 INFO:teuthology.orchestra.run.smithi123.stdout: "info", 2024-09-16T15:47:33.608 INFO:teuthology.orchestra.run.smithi123.stdout: "warning" 2024-09-16T15:47:33.608 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-09-16T15:47:33.608 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:33.608 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.608 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.608 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.608 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.608 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_cluster": { 2024-09-16T15:47:33.608 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_cluster", 2024-09-16T15:47:33.609 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-09-16T15:47:33.609 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.609 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-09-16T15:47:33.609 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-09-16T15:47:33.609 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.609 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.609 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.609 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:33.609 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.609 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.609 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.609 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.609 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_cluster_level": { 2024-09-16T15:47:33.609 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_cluster_level", 2024-09-16T15:47:33.610 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-09-16T15:47:33.610 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.610 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-09-16T15:47:33.610 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "info", 2024-09-16T15:47:33.610 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.610 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.610 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [ 2024-09-16T15:47:33.610 INFO:teuthology.orchestra.run.smithi123.stdout: "", 2024-09-16T15:47:33.610 INFO:teuthology.orchestra.run.smithi123.stdout: "critical", 2024-09-16T15:47:33.610 INFO:teuthology.orchestra.run.smithi123.stdout: "debug", 2024-09-16T15:47:33.610 INFO:teuthology.orchestra.run.smithi123.stdout: "error", 2024-09-16T15:47:33.610 INFO:teuthology.orchestra.run.smithi123.stdout: "info", 2024-09-16T15:47:33.610 INFO:teuthology.orchestra.run.smithi123.stdout: "warning" 2024-09-16T15:47:33.611 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-09-16T15:47:33.611 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:33.611 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.611 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.611 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.611 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.611 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_file": { 2024-09-16T15:47:33.611 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_file", 2024-09-16T15:47:33.611 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-09-16T15:47:33.611 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.611 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-09-16T15:47:33.611 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-09-16T15:47:33.611 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.612 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.612 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.612 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:33.612 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.612 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.612 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.612 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.612 INFO:teuthology.orchestra.run.smithi123.stdout: "organization": { 2024-09-16T15:47:33.612 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "organization", 2024-09-16T15:47:33.612 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-09-16T15:47:33.612 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.612 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-09-16T15:47:33.612 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "", 2024-09-16T15:47:33.612 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.613 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.613 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.613 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:33.613 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.613 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.613 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.613 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.613 INFO:teuthology.orchestra.run.smithi123.stdout: "proxy": { 2024-09-16T15:47:33.613 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "proxy", 2024-09-16T15:47:33.613 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-09-16T15:47:33.613 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.613 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-09-16T15:47:33.614 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "", 2024-09-16T15:47:33.614 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.614 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.614 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.614 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:33.614 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.614 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.614 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.614 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.614 INFO:teuthology.orchestra.run.smithi123.stdout: "sqlite3_killpoint": { 2024-09-16T15:47:33.614 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "sqlite3_killpoint", 2024-09-16T15:47:33.614 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "int", 2024-09-16T15:47:33.614 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "dev", 2024-09-16T15:47:33.615 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-09-16T15:47:33.615 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "0", 2024-09-16T15:47:33.615 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.615 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.615 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.615 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:33.615 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.615 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.615 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.615 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.615 INFO:teuthology.orchestra.run.smithi123.stdout: "url": { 2024-09-16T15:47:33.615 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "url", 2024-09-16T15:47:33.615 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-09-16T15:47:33.615 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.615 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-09-16T15:47:33.616 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "https://telemetry.ceph.com/report", 2024-09-16T15:47:33.616 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.616 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.616 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.616 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:33.616 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.616 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.616 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.616 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-16T15:47:33.616 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-16T15:47:33.616 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.616 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-09-16T15:47:33.616 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "test_orchestrator", 2024-09-16T15:47:33.616 INFO:teuthology.orchestra.run.smithi123.stdout: "can_run": true, 2024-09-16T15:47:33.617 INFO:teuthology.orchestra.run.smithi123.stdout: "error_string": "", 2024-09-16T15:47:33.617 INFO:teuthology.orchestra.run.smithi123.stdout: "module_options": { 2024-09-16T15:47:33.617 INFO:teuthology.orchestra.run.smithi123.stdout: "log_level": { 2024-09-16T15:47:33.617 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_level", 2024-09-16T15:47:33.617 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-09-16T15:47:33.617 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.617 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-09-16T15:47:33.617 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "", 2024-09-16T15:47:33.617 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.617 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.617 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [ 2024-09-16T15:47:33.617 INFO:teuthology.orchestra.run.smithi123.stdout: "", 2024-09-16T15:47:33.617 INFO:teuthology.orchestra.run.smithi123.stdout: "critical", 2024-09-16T15:47:33.617 INFO:teuthology.orchestra.run.smithi123.stdout: "debug", 2024-09-16T15:47:33.617 INFO:teuthology.orchestra.run.smithi123.stdout: "error", 2024-09-16T15:47:33.618 INFO:teuthology.orchestra.run.smithi123.stdout: "info", 2024-09-16T15:47:33.618 INFO:teuthology.orchestra.run.smithi123.stdout: "warning" 2024-09-16T15:47:33.618 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-09-16T15:47:33.618 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:33.618 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.618 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.618 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.618 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.618 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_cluster": { 2024-09-16T15:47:33.618 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_cluster", 2024-09-16T15:47:33.618 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-09-16T15:47:33.618 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.618 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-09-16T15:47:33.618 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-09-16T15:47:33.618 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.619 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.619 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.619 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:33.619 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.619 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.619 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.619 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.619 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_cluster_level": { 2024-09-16T15:47:33.619 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_cluster_level", 2024-09-16T15:47:33.619 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-09-16T15:47:33.619 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.619 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-09-16T15:47:33.619 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "info", 2024-09-16T15:47:33.619 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.620 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.620 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [ 2024-09-16T15:47:33.620 INFO:teuthology.orchestra.run.smithi123.stdout: "", 2024-09-16T15:47:33.620 INFO:teuthology.orchestra.run.smithi123.stdout: "critical", 2024-09-16T15:47:33.620 INFO:teuthology.orchestra.run.smithi123.stdout: "debug", 2024-09-16T15:47:33.620 INFO:teuthology.orchestra.run.smithi123.stdout: "error", 2024-09-16T15:47:33.620 INFO:teuthology.orchestra.run.smithi123.stdout: "info", 2024-09-16T15:47:33.620 INFO:teuthology.orchestra.run.smithi123.stdout: "warning" 2024-09-16T15:47:33.620 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-09-16T15:47:33.620 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:33.620 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.620 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.620 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.620 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.620 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_file": { 2024-09-16T15:47:33.621 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_file", 2024-09-16T15:47:33.621 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-09-16T15:47:33.621 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.621 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-09-16T15:47:33.621 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-09-16T15:47:33.621 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.621 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.621 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.621 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:33.621 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.621 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.621 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.621 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.621 INFO:teuthology.orchestra.run.smithi123.stdout: "sqlite3_killpoint": { 2024-09-16T15:47:33.621 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "sqlite3_killpoint", 2024-09-16T15:47:33.621 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "int", 2024-09-16T15:47:33.622 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "dev", 2024-09-16T15:47:33.622 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-09-16T15:47:33.622 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "0", 2024-09-16T15:47:33.622 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.622 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.622 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.622 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:33.622 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.622 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.622 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.622 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-16T15:47:33.622 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-16T15:47:33.622 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.622 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-09-16T15:47:33.622 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "volumes", 2024-09-16T15:47:33.623 INFO:teuthology.orchestra.run.smithi123.stdout: "can_run": true, 2024-09-16T15:47:33.623 INFO:teuthology.orchestra.run.smithi123.stdout: "error_string": "", 2024-09-16T15:47:33.623 INFO:teuthology.orchestra.run.smithi123.stdout: "module_options": { 2024-09-16T15:47:33.623 INFO:teuthology.orchestra.run.smithi123.stdout: "log_level": { 2024-09-16T15:47:33.623 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_level", 2024-09-16T15:47:33.623 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-09-16T15:47:33.623 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.623 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-09-16T15:47:33.623 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "", 2024-09-16T15:47:33.623 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.623 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.623 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [ 2024-09-16T15:47:33.623 INFO:teuthology.orchestra.run.smithi123.stdout: "", 2024-09-16T15:47:33.623 INFO:teuthology.orchestra.run.smithi123.stdout: "critical", 2024-09-16T15:47:33.623 INFO:teuthology.orchestra.run.smithi123.stdout: "debug", 2024-09-16T15:47:33.624 INFO:teuthology.orchestra.run.smithi123.stdout: "error", 2024-09-16T15:47:33.624 INFO:teuthology.orchestra.run.smithi123.stdout: "info", 2024-09-16T15:47:33.624 INFO:teuthology.orchestra.run.smithi123.stdout: "warning" 2024-09-16T15:47:33.624 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-09-16T15:47:33.624 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:33.624 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.624 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.624 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.624 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.624 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_cluster": { 2024-09-16T15:47:33.624 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_cluster", 2024-09-16T15:47:33.624 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-09-16T15:47:33.624 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.624 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-09-16T15:47:33.624 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-09-16T15:47:33.625 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.625 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.625 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.625 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:33.625 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.625 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.625 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.625 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.625 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_cluster_level": { 2024-09-16T15:47:33.625 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_cluster_level", 2024-09-16T15:47:33.625 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-09-16T15:47:33.626 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.626 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-09-16T15:47:33.626 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "info", 2024-09-16T15:47:33.626 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.626 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.626 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [ 2024-09-16T15:47:33.626 INFO:teuthology.orchestra.run.smithi123.stdout: "", 2024-09-16T15:47:33.626 INFO:teuthology.orchestra.run.smithi123.stdout: "critical", 2024-09-16T15:47:33.626 INFO:teuthology.orchestra.run.smithi123.stdout: "debug", 2024-09-16T15:47:33.626 INFO:teuthology.orchestra.run.smithi123.stdout: "error", 2024-09-16T15:47:33.626 INFO:teuthology.orchestra.run.smithi123.stdout: "info", 2024-09-16T15:47:33.626 INFO:teuthology.orchestra.run.smithi123.stdout: "warning" 2024-09-16T15:47:33.626 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-09-16T15:47:33.627 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:33.627 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.627 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.627 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.627 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.627 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_file": { 2024-09-16T15:47:33.627 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_file", 2024-09-16T15:47:33.627 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-09-16T15:47:33.627 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.627 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-09-16T15:47:33.627 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-09-16T15:47:33.627 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.627 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.627 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.627 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:33.628 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.628 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.628 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.628 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.628 INFO:teuthology.orchestra.run.smithi123.stdout: "max_concurrent_clones": { 2024-09-16T15:47:33.628 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "max_concurrent_clones", 2024-09-16T15:47:33.628 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "int", 2024-09-16T15:47:33.628 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.628 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-09-16T15:47:33.628 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "4", 2024-09-16T15:47:33.628 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.628 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.628 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.628 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "Number of asynchronous cloner threads", 2024-09-16T15:47:33.628 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.629 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.629 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.629 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.629 INFO:teuthology.orchestra.run.smithi123.stdout: "periodic_async_work": { 2024-09-16T15:47:33.629 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "periodic_async_work", 2024-09-16T15:47:33.629 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-09-16T15:47:33.629 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.629 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-09-16T15:47:33.629 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-09-16T15:47:33.629 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.629 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.629 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.629 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "Periodically check for async work", 2024-09-16T15:47:33.629 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.629 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.630 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.630 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.630 INFO:teuthology.orchestra.run.smithi123.stdout: "snapshot_clone_delay": { 2024-09-16T15:47:33.630 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "snapshot_clone_delay", 2024-09-16T15:47:33.630 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "int", 2024-09-16T15:47:33.630 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.630 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-09-16T15:47:33.630 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "0", 2024-09-16T15:47:33.630 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.630 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.630 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.630 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "Delay clone begin operation by snapshot_clone_delay seconds", 2024-09-16T15:47:33.630 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.630 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.630 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.630 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.631 INFO:teuthology.orchestra.run.smithi123.stdout: "snapshot_clone_no_wait": { 2024-09-16T15:47:33.631 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "snapshot_clone_no_wait", 2024-09-16T15:47:33.631 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-09-16T15:47:33.631 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.631 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-09-16T15:47:33.631 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "True", 2024-09-16T15:47:33.631 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.631 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.631 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.631 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "Reject subvolume clone request when cloner threads are busy", 2024-09-16T15:47:33.631 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.631 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.631 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.631 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.632 INFO:teuthology.orchestra.run.smithi123.stdout: "sqlite3_killpoint": { 2024-09-16T15:47:33.632 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "sqlite3_killpoint", 2024-09-16T15:47:33.632 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "int", 2024-09-16T15:47:33.632 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "dev", 2024-09-16T15:47:33.632 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-09-16T15:47:33.632 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "0", 2024-09-16T15:47:33.632 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.632 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.632 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.632 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:33.632 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.632 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.632 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.632 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-16T15:47:33.633 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-16T15:47:33.633 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.633 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-09-16T15:47:33.633 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "zabbix", 2024-09-16T15:47:33.633 INFO:teuthology.orchestra.run.smithi123.stdout: "can_run": true, 2024-09-16T15:47:33.633 INFO:teuthology.orchestra.run.smithi123.stdout: "error_string": "", 2024-09-16T15:47:33.633 INFO:teuthology.orchestra.run.smithi123.stdout: "module_options": { 2024-09-16T15:47:33.633 INFO:teuthology.orchestra.run.smithi123.stdout: "discovery_interval": { 2024-09-16T15:47:33.633 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "discovery_interval", 2024-09-16T15:47:33.633 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "uint", 2024-09-16T15:47:33.633 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.633 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-09-16T15:47:33.633 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "100", 2024-09-16T15:47:33.633 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.633 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.633 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.634 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:33.634 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.634 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.634 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.634 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.634 INFO:teuthology.orchestra.run.smithi123.stdout: "identifier": { 2024-09-16T15:47:33.634 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "identifier", 2024-09-16T15:47:33.634 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-09-16T15:47:33.634 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.634 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-09-16T15:47:33.634 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "", 2024-09-16T15:47:33.634 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.634 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.634 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.634 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:33.635 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.635 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.635 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.635 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.635 INFO:teuthology.orchestra.run.smithi123.stdout: "interval": { 2024-09-16T15:47:33.635 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "interval", 2024-09-16T15:47:33.635 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "secs", 2024-09-16T15:47:33.635 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.635 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-09-16T15:47:33.635 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "60", 2024-09-16T15:47:33.635 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.635 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.635 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.635 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:33.635 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.635 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.636 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.636 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.636 INFO:teuthology.orchestra.run.smithi123.stdout: "log_level": { 2024-09-16T15:47:33.636 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_level", 2024-09-16T15:47:33.636 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-09-16T15:47:33.636 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.636 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-09-16T15:47:33.636 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "", 2024-09-16T15:47:33.636 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.636 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.636 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [ 2024-09-16T15:47:33.636 INFO:teuthology.orchestra.run.smithi123.stdout: "", 2024-09-16T15:47:33.636 INFO:teuthology.orchestra.run.smithi123.stdout: "critical", 2024-09-16T15:47:33.636 INFO:teuthology.orchestra.run.smithi123.stdout: "debug", 2024-09-16T15:47:33.637 INFO:teuthology.orchestra.run.smithi123.stdout: "error", 2024-09-16T15:47:33.637 INFO:teuthology.orchestra.run.smithi123.stdout: "info", 2024-09-16T15:47:33.637 INFO:teuthology.orchestra.run.smithi123.stdout: "warning" 2024-09-16T15:47:33.637 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-09-16T15:47:33.637 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:33.637 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.637 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.637 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.637 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.637 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_cluster": { 2024-09-16T15:47:33.637 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_cluster", 2024-09-16T15:47:33.637 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-09-16T15:47:33.637 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.637 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-09-16T15:47:33.637 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-09-16T15:47:33.638 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.638 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.638 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.638 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:33.638 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.638 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.638 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.638 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.638 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_cluster_level": { 2024-09-16T15:47:33.638 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_cluster_level", 2024-09-16T15:47:33.638 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-09-16T15:47:33.638 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.638 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-09-16T15:47:33.638 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "info", 2024-09-16T15:47:33.638 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.638 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.639 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [ 2024-09-16T15:47:33.639 INFO:teuthology.orchestra.run.smithi123.stdout: "", 2024-09-16T15:47:33.639 INFO:teuthology.orchestra.run.smithi123.stdout: "critical", 2024-09-16T15:47:33.639 INFO:teuthology.orchestra.run.smithi123.stdout: "debug", 2024-09-16T15:47:33.639 INFO:teuthology.orchestra.run.smithi123.stdout: "error", 2024-09-16T15:47:33.639 INFO:teuthology.orchestra.run.smithi123.stdout: "info", 2024-09-16T15:47:33.639 INFO:teuthology.orchestra.run.smithi123.stdout: "warning" 2024-09-16T15:47:33.639 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-09-16T15:47:33.639 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:33.639 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.639 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.639 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.639 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.639 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_file": { 2024-09-16T15:47:33.639 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "log_to_file", 2024-09-16T15:47:33.640 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "bool", 2024-09-16T15:47:33.640 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.640 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-09-16T15:47:33.640 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "False", 2024-09-16T15:47:33.640 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.640 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.640 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.640 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:33.640 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.640 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.640 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.640 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.640 INFO:teuthology.orchestra.run.smithi123.stdout: "sqlite3_killpoint": { 2024-09-16T15:47:33.640 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "sqlite3_killpoint", 2024-09-16T15:47:33.640 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "int", 2024-09-16T15:47:33.641 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "dev", 2024-09-16T15:47:33.641 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2024-09-16T15:47:33.641 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "0", 2024-09-16T15:47:33.641 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.641 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.641 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.641 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:33.641 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.641 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.641 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.641 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.641 INFO:teuthology.orchestra.run.smithi123.stdout: "zabbix_host": { 2024-09-16T15:47:33.641 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "zabbix_host", 2024-09-16T15:47:33.641 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-09-16T15:47:33.641 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.642 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-09-16T15:47:33.642 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "", 2024-09-16T15:47:33.642 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.642 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.642 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.642 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:33.642 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.642 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.642 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.642 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.642 INFO:teuthology.orchestra.run.smithi123.stdout: "zabbix_port": { 2024-09-16T15:47:33.642 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "zabbix_port", 2024-09-16T15:47:33.642 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "int", 2024-09-16T15:47:33.642 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.642 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-09-16T15:47:33.642 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "10051", 2024-09-16T15:47:33.643 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.643 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.643 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.643 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:33.643 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.643 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.643 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.643 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.643 INFO:teuthology.orchestra.run.smithi123.stdout: "zabbix_sender": { 2024-09-16T15:47:33.643 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "zabbix_sender", 2024-09-16T15:47:33.643 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "str", 2024-09-16T15:47:33.643 INFO:teuthology.orchestra.run.smithi123.stdout: "level": "advanced", 2024-09-16T15:47:33.643 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 0, 2024-09-16T15:47:33.643 INFO:teuthology.orchestra.run.smithi123.stdout: "default_value": "/usr/bin/zabbix_sender", 2024-09-16T15:47:33.643 INFO:teuthology.orchestra.run.smithi123.stdout: "min": "", 2024-09-16T15:47:33.644 INFO:teuthology.orchestra.run.smithi123.stdout: "max": "", 2024-09-16T15:47:33.644 INFO:teuthology.orchestra.run.smithi123.stdout: "enum_allowed": [], 2024-09-16T15:47:33.644 INFO:teuthology.orchestra.run.smithi123.stdout: "desc": "", 2024-09-16T15:47:33.644 INFO:teuthology.orchestra.run.smithi123.stdout: "long_desc": "", 2024-09-16T15:47:33.644 INFO:teuthology.orchestra.run.smithi123.stdout: "tags": [], 2024-09-16T15:47:33.644 INFO:teuthology.orchestra.run.smithi123.stdout: "see_also": [] 2024-09-16T15:47:33.644 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-16T15:47:33.644 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-16T15:47:33.644 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-16T15:47:33.644 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-09-16T15:47:33.644 INFO:teuthology.orchestra.run.smithi123.stdout: "services": {}, 2024-09-16T15:47:33.644 INFO:teuthology.orchestra.run.smithi123.stdout: "always_on_modules": { 2024-09-16T15:47:33.644 INFO:teuthology.orchestra.run.smithi123.stdout: "octopus": [ 2024-09-16T15:47:33.644 INFO:teuthology.orchestra.run.smithi123.stdout: "balancer", 2024-09-16T15:47:33.644 INFO:teuthology.orchestra.run.smithi123.stdout: "crash", 2024-09-16T15:47:33.645 INFO:teuthology.orchestra.run.smithi123.stdout: "devicehealth", 2024-09-16T15:47:33.645 INFO:teuthology.orchestra.run.smithi123.stdout: "orchestrator", 2024-09-16T15:47:33.645 INFO:teuthology.orchestra.run.smithi123.stdout: "pg_autoscaler", 2024-09-16T15:47:33.645 INFO:teuthology.orchestra.run.smithi123.stdout: "progress", 2024-09-16T15:47:33.645 INFO:teuthology.orchestra.run.smithi123.stdout: "rbd_support", 2024-09-16T15:47:33.645 INFO:teuthology.orchestra.run.smithi123.stdout: "status", 2024-09-16T15:47:33.645 INFO:teuthology.orchestra.run.smithi123.stdout: "telemetry", 2024-09-16T15:47:33.645 INFO:teuthology.orchestra.run.smithi123.stdout: "volumes" 2024-09-16T15:47:33.645 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-09-16T15:47:33.645 INFO:teuthology.orchestra.run.smithi123.stdout: "pacific": [ 2024-09-16T15:47:33.645 INFO:teuthology.orchestra.run.smithi123.stdout: "balancer", 2024-09-16T15:47:33.645 INFO:teuthology.orchestra.run.smithi123.stdout: "crash", 2024-09-16T15:47:33.645 INFO:teuthology.orchestra.run.smithi123.stdout: "devicehealth", 2024-09-16T15:47:33.645 INFO:teuthology.orchestra.run.smithi123.stdout: "orchestrator", 2024-09-16T15:47:33.645 INFO:teuthology.orchestra.run.smithi123.stdout: "pg_autoscaler", 2024-09-16T15:47:33.646 INFO:teuthology.orchestra.run.smithi123.stdout: "progress", 2024-09-16T15:47:33.646 INFO:teuthology.orchestra.run.smithi123.stdout: "rbd_support", 2024-09-16T15:47:33.646 INFO:teuthology.orchestra.run.smithi123.stdout: "status", 2024-09-16T15:47:33.646 INFO:teuthology.orchestra.run.smithi123.stdout: "telemetry", 2024-09-16T15:47:33.646 INFO:teuthology.orchestra.run.smithi123.stdout: "volumes" 2024-09-16T15:47:33.646 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-09-16T15:47:33.646 INFO:teuthology.orchestra.run.smithi123.stdout: "quincy": [ 2024-09-16T15:47:33.646 INFO:teuthology.orchestra.run.smithi123.stdout: "balancer", 2024-09-16T15:47:33.646 INFO:teuthology.orchestra.run.smithi123.stdout: "crash", 2024-09-16T15:47:33.646 INFO:teuthology.orchestra.run.smithi123.stdout: "devicehealth", 2024-09-16T15:47:33.646 INFO:teuthology.orchestra.run.smithi123.stdout: "orchestrator", 2024-09-16T15:47:33.646 INFO:teuthology.orchestra.run.smithi123.stdout: "pg_autoscaler", 2024-09-16T15:47:33.646 INFO:teuthology.orchestra.run.smithi123.stdout: "progress", 2024-09-16T15:47:33.646 INFO:teuthology.orchestra.run.smithi123.stdout: "rbd_support", 2024-09-16T15:47:33.646 INFO:teuthology.orchestra.run.smithi123.stdout: "status", 2024-09-16T15:47:33.647 INFO:teuthology.orchestra.run.smithi123.stdout: "telemetry", 2024-09-16T15:47:33.647 INFO:teuthology.orchestra.run.smithi123.stdout: "volumes" 2024-09-16T15:47:33.647 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-09-16T15:47:33.647 INFO:teuthology.orchestra.run.smithi123.stdout: "reef": [ 2024-09-16T15:47:33.647 INFO:teuthology.orchestra.run.smithi123.stdout: "balancer", 2024-09-16T15:47:33.647 INFO:teuthology.orchestra.run.smithi123.stdout: "crash", 2024-09-16T15:47:33.647 INFO:teuthology.orchestra.run.smithi123.stdout: "devicehealth", 2024-09-16T15:47:33.647 INFO:teuthology.orchestra.run.smithi123.stdout: "orchestrator", 2024-09-16T15:47:33.647 INFO:teuthology.orchestra.run.smithi123.stdout: "pg_autoscaler", 2024-09-16T15:47:33.647 INFO:teuthology.orchestra.run.smithi123.stdout: "progress", 2024-09-16T15:47:33.647 INFO:teuthology.orchestra.run.smithi123.stdout: "rbd_support", 2024-09-16T15:47:33.647 INFO:teuthology.orchestra.run.smithi123.stdout: "status", 2024-09-16T15:47:33.647 INFO:teuthology.orchestra.run.smithi123.stdout: "telemetry", 2024-09-16T15:47:33.647 INFO:teuthology.orchestra.run.smithi123.stdout: "volumes" 2024-09-16T15:47:33.647 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-09-16T15:47:33.647 INFO:teuthology.orchestra.run.smithi123.stdout: "squid": [ 2024-09-16T15:47:33.648 INFO:teuthology.orchestra.run.smithi123.stdout: "balancer", 2024-09-16T15:47:33.648 INFO:teuthology.orchestra.run.smithi123.stdout: "crash", 2024-09-16T15:47:33.648 INFO:teuthology.orchestra.run.smithi123.stdout: "devicehealth", 2024-09-16T15:47:33.648 INFO:teuthology.orchestra.run.smithi123.stdout: "orchestrator", 2024-09-16T15:47:33.648 INFO:teuthology.orchestra.run.smithi123.stdout: "pg_autoscaler", 2024-09-16T15:47:33.648 INFO:teuthology.orchestra.run.smithi123.stdout: "progress", 2024-09-16T15:47:33.648 INFO:teuthology.orchestra.run.smithi123.stdout: "rbd_support", 2024-09-16T15:47:33.648 INFO:teuthology.orchestra.run.smithi123.stdout: "status", 2024-09-16T15:47:33.648 INFO:teuthology.orchestra.run.smithi123.stdout: "telemetry", 2024-09-16T15:47:33.648 INFO:teuthology.orchestra.run.smithi123.stdout: "volumes" 2024-09-16T15:47:33.648 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-09-16T15:47:33.648 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.648 INFO:teuthology.orchestra.run.smithi123.stdout: "last_failure_osd_epoch": 0, 2024-09-16T15:47:33.648 INFO:teuthology.orchestra.run.smithi123.stdout: "active_clients": [ 2024-09-16T15:47:33.648 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-09-16T15:47:33.649 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "devicehealth", 2024-09-16T15:47:33.649 INFO:teuthology.orchestra.run.smithi123.stdout: "addrvec": [ 2024-09-16T15:47:33.649 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-09-16T15:47:33.649 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "v2", 2024-09-16T15:47:33.649 INFO:teuthology.orchestra.run.smithi123.stdout: "addr": "172.21.15.184:0", 2024-09-16T15:47:33.649 INFO:teuthology.orchestra.run.smithi123.stdout: "nonce": 1715342118 2024-09-16T15:47:33.649 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-16T15:47:33.649 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-09-16T15:47:33.649 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.649 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-09-16T15:47:33.649 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "libcephsqlite", 2024-09-16T15:47:33.649 INFO:teuthology.orchestra.run.smithi123.stdout: "addrvec": [ 2024-09-16T15:47:33.649 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-09-16T15:47:33.649 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "v2", 2024-09-16T15:47:33.649 INFO:teuthology.orchestra.run.smithi123.stdout: "addr": "172.21.15.184:0", 2024-09-16T15:47:33.650 INFO:teuthology.orchestra.run.smithi123.stdout: "nonce": 2291367472 2024-09-16T15:47:33.650 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-16T15:47:33.650 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-09-16T15:47:33.650 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.650 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-09-16T15:47:33.650 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "rbd_support", 2024-09-16T15:47:33.650 INFO:teuthology.orchestra.run.smithi123.stdout: "addrvec": [ 2024-09-16T15:47:33.650 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-09-16T15:47:33.650 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "v2", 2024-09-16T15:47:33.650 INFO:teuthology.orchestra.run.smithi123.stdout: "addr": "172.21.15.184:0", 2024-09-16T15:47:33.650 INFO:teuthology.orchestra.run.smithi123.stdout: "nonce": 1402401038 2024-09-16T15:47:33.650 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-16T15:47:33.650 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-09-16T15:47:33.650 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.650 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-09-16T15:47:33.651 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "volumes", 2024-09-16T15:47:33.651 INFO:teuthology.orchestra.run.smithi123.stdout: "addrvec": [ 2024-09-16T15:47:33.651 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-09-16T15:47:33.651 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "v2", 2024-09-16T15:47:33.651 INFO:teuthology.orchestra.run.smithi123.stdout: "addr": "172.21.15.184:0", 2024-09-16T15:47:33.651 INFO:teuthology.orchestra.run.smithi123.stdout: "nonce": 2406729396 2024-09-16T15:47:33.651 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-16T15:47:33.651 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-09-16T15:47:33.651 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.651 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-09-16T15:47:33.651 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "volumes", 2024-09-16T15:47:33.651 INFO:teuthology.orchestra.run.smithi123.stdout: "addrvec": [ 2024-09-16T15:47:33.651 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-09-16T15:47:33.651 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "v2", 2024-09-16T15:47:33.651 INFO:teuthology.orchestra.run.smithi123.stdout: "addr": "172.21.15.184:0", 2024-09-16T15:47:33.652 INFO:teuthology.orchestra.run.smithi123.stdout: "nonce": 1023177165 2024-09-16T15:47:33.652 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-16T15:47:33.652 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-09-16T15:47:33.652 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.652 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-09-16T15:47:33.652 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "volumes", 2024-09-16T15:47:33.652 INFO:teuthology.orchestra.run.smithi123.stdout: "addrvec": [ 2024-09-16T15:47:33.652 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-09-16T15:47:33.652 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "v2", 2024-09-16T15:47:33.652 INFO:teuthology.orchestra.run.smithi123.stdout: "addr": "172.21.15.184:0", 2024-09-16T15:47:33.652 INFO:teuthology.orchestra.run.smithi123.stdout: "nonce": 2916845314 2024-09-16T15:47:33.652 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-16T15:47:33.652 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-09-16T15:47:33.652 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.652 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-09-16T15:47:33.653 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "volumes", 2024-09-16T15:47:33.653 INFO:teuthology.orchestra.run.smithi123.stdout: "addrvec": [ 2024-09-16T15:47:33.653 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-09-16T15:47:33.653 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "v2", 2024-09-16T15:47:33.653 INFO:teuthology.orchestra.run.smithi123.stdout: "addr": "172.21.15.184:0", 2024-09-16T15:47:33.653 INFO:teuthology.orchestra.run.smithi123.stdout: "nonce": 78639736 2024-09-16T15:47:33.653 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-16T15:47:33.653 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-09-16T15:47:33.653 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.653 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-09-16T15:47:33.653 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "volumes", 2024-09-16T15:47:33.653 INFO:teuthology.orchestra.run.smithi123.stdout: "addrvec": [ 2024-09-16T15:47:33.653 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-09-16T15:47:33.653 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "v2", 2024-09-16T15:47:33.653 INFO:teuthology.orchestra.run.smithi123.stdout: "addr": "172.21.15.184:0", 2024-09-16T15:47:33.653 INFO:teuthology.orchestra.run.smithi123.stdout: "nonce": 3070886766 2024-09-16T15:47:33.654 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-16T15:47:33.654 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-09-16T15:47:33.654 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.654 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-09-16T15:47:33.654 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "volumes", 2024-09-16T15:47:33.654 INFO:teuthology.orchestra.run.smithi123.stdout: "addrvec": [ 2024-09-16T15:47:33.654 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-09-16T15:47:33.654 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "v2", 2024-09-16T15:47:33.654 INFO:teuthology.orchestra.run.smithi123.stdout: "addr": "172.21.15.184:0", 2024-09-16T15:47:33.654 INFO:teuthology.orchestra.run.smithi123.stdout: "nonce": 3204392673 2024-09-16T15:47:33.654 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-16T15:47:33.654 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-09-16T15:47:33.654 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.654 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-09-16T15:47:33.654 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "volumes", 2024-09-16T15:47:33.655 INFO:teuthology.orchestra.run.smithi123.stdout: "addrvec": [ 2024-09-16T15:47:33.655 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-09-16T15:47:33.655 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "v2", 2024-09-16T15:47:33.655 INFO:teuthology.orchestra.run.smithi123.stdout: "addr": "172.21.15.184:0", 2024-09-16T15:47:33.655 INFO:teuthology.orchestra.run.smithi123.stdout: "nonce": 140936988 2024-09-16T15:47:33.655 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-16T15:47:33.655 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-09-16T15:47:33.655 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:33.655 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-09-16T15:47:33.655 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "volumes", 2024-09-16T15:47:33.655 INFO:teuthology.orchestra.run.smithi123.stdout: "addrvec": [ 2024-09-16T15:47:33.655 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-09-16T15:47:33.655 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "v2", 2024-09-16T15:47:33.655 INFO:teuthology.orchestra.run.smithi123.stdout: "addr": "172.21.15.184:0", 2024-09-16T15:47:33.655 INFO:teuthology.orchestra.run.smithi123.stdout: "nonce": 654280269 2024-09-16T15:47:33.656 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-16T15:47:33.656 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-09-16T15:47:33.656 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-16T15:47:33.656 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-09-16T15:47:33.656 INFO:teuthology.orchestra.run.smithi123.stdout:} 2024-09-16T15:47:33.657 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:32.748+0000 7f0042c67640 1 -- 172.21.15.123:0/3919206734 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f00280732b0 msgr2=0x7f0028075770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:47:33.657 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:32.748+0000 7f0042c67640 1 --2- 172.21.15.123:0/3919206734 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f00280732b0 0x7f0028075770 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f00340ae000 tx=0x7f003401af10 comp rx=0 tx=0).stop 2024-09-16T15:47:33.657 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:32.748+0000 7f0042c67640 1 -- 172.21.15.123:0/3919206734 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f003c161620 msgr2=0x7f003c156010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:47:33.657 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:32.748+0000 7f0042c67640 1 --2- 172.21.15.123:0/3919206734 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f003c161620 0x7f003c156010 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f002c00c470 tx=0x7f002c00c930 comp rx=0 tx=0).stop 2024-09-16T15:47:33.657 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:32.748+0000 7f0042c67640 1 -- 172.21.15.123:0/3919206734 shutdown_connections 2024-09-16T15:47:33.657 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:32.748+0000 7f0042c67640 1 --2- 172.21.15.123:0/3919206734 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f00280732b0 0x7f0028075770 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:33.657 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:32.748+0000 7f0042c67640 1 --2- 172.21.15.123:0/3919206734 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f003c161620 0x7f003c156010 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:33.657 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:32.748+0000 7f0042c67640 1 --2- 172.21.15.123:0/3919206734 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f003c0d2820 0x7f003c1fab10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:33.657 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:32.748+0000 7f0042c67640 1 --2- 172.21.15.123:0/3919206734 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f003c0b2b00 0x7f003c0cc2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:33.658 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:32.748+0000 7f0042c67640 1 -- 172.21.15.123:0/3919206734 >> 172.21.15.123:0/3919206734 conn(0x7f003c0d3ea0 msgr2=0x7f003c0b5920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:47:33.658 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:32.748+0000 7f0042c67640 1 -- 172.21.15.123:0/3919206734 shutdown_connections 2024-09-16T15:47:33.658 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:32.748+0000 7f0042c67640 1 -- 172.21.15.123:0/3919206734 wait complete. 2024-09-16T15:47:33.662 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr fail x 2024-09-16T15:47:33.797 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:33.792+0000 7f9babd02640 1 -- 172.21.15.123:0/789577248 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9ba415f720 msgr2=0x7f9ba40cd7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:47:33.797 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:33.792+0000 7f9babd02640 1 --2- 172.21.15.123:0/789577248 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9ba415f720 0x7f9ba40cd7e0 secure :-1 s=READY pgs=849 cs=0 l=1 rev1=1 crypto rx=0x7f9ba4069940 tx=0x7f9b9c01c800 comp rx=0 tx=0).stop 2024-09-16T15:47:33.797 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:33.792+0000 7f9babd02640 1 -- 172.21.15.123:0/789577248 shutdown_connections 2024-09-16T15:47:33.797 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:33.792+0000 7f9babd02640 1 --2- 172.21.15.123:0/789577248 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9ba415f720 0x7f9ba40cd7e0 unknown :-1 s=CLOSED pgs=849 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:33.797 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:33.792+0000 7f9babd02640 1 --2- 172.21.15.123:0/789577248 >> v2:172.21.15.184:3300/0 conn(0x7f9ba415f340 0x7f9ba40b1490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:33.797 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:33.792+0000 7f9babd02640 1 -- 172.21.15.123:0/789577248 >> 172.21.15.123:0/789577248 conn(0x7f9ba40d1c70 msgr2=0x7f9ba40d2050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:47:33.797 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:33.792+0000 7f9babd02640 1 -- 172.21.15.123:0/789577248 shutdown_connections 2024-09-16T15:47:33.797 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:33.792+0000 7f9babd02640 1 -- 172.21.15.123:0/789577248 wait complete. 2024-09-16T15:47:33.798 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:33.792+0000 7f9babd02640 1 Processor -- start 2024-09-16T15:47:33.798 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:33.792+0000 7f9babd02640 1 -- start start 2024-09-16T15:47:33.799 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:33.796+0000 7f9babd02640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f9ba40b0f40 0x7f9ba414d7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:47:33.800 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:33.796+0000 7f9babd02640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f9ba415f340 0x7f9ba4153f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:47:33.800 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:33.796+0000 7f9babd02640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9ba415f720 0x7f9ba4152fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:47:33.800 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:33.796+0000 7f9babd02640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f9ba41690e0 con 0x7f9ba415f720 2024-09-16T15:47:33.800 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:33.796+0000 7f9babd02640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f9ba4168f60 con 0x7f9ba40b0f40 2024-09-16T15:47:33.800 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:33.796+0000 7f9babd02640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f9ba4169260 con 0x7f9ba415f340 2024-09-16T15:47:33.800 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:33.796+0000 7f9baa29f640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9ba415f720 0x7f9ba4152fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:47:33.800 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:33.796+0000 7f9baa29f640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9ba415f720 0x7f9ba4152fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:52794/0 (socket says 172.21.15.123:52794) 2024-09-16T15:47:33.800 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:33.796+0000 7f9baa29f640 1 -- 172.21.15.123:0/632290373 learned_addr learned my addr 172.21.15.123:0/632290373 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:47:33.800 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:33.796+0000 7f9ba9a9e640 1 --2- 172.21.15.123:0/632290373 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f9ba40b0f40 0x7f9ba414d7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:47:33.800 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:33.796+0000 7f9baa29f640 1 -- 172.21.15.123:0/632290373 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f9ba415f340 msgr2=0x7f9ba4153f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:47:33.800 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:33.796+0000 7f9baa29f640 1 --2- 172.21.15.123:0/632290373 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f9ba415f340 0x7f9ba4153f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:33.800 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:33.796+0000 7f9baa29f640 1 -- 172.21.15.123:0/632290373 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f9ba40b0f40 msgr2=0x7f9ba414d7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:47:33.801 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:33.796+0000 7f9baa29f640 1 --2- 172.21.15.123:0/632290373 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f9ba40b0f40 0x7f9ba414d7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:33.801 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:33.796+0000 7f9baa29f640 1 -- 172.21.15.123:0/632290373 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9ba41f8ad0 con 0x7f9ba415f720 2024-09-16T15:47:33.801 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:33.796+0000 7f9ba9a9e640 1 --2- 172.21.15.123:0/632290373 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f9ba40b0f40 0x7f9ba414d7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T15:47:33.801 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:33.796+0000 7f9baa29f640 1 --2- 172.21.15.123:0/632290373 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9ba415f720 0x7f9ba4152fa0 secure :-1 s=READY pgs=850 cs=0 l=1 rev1=1 crypto rx=0x7f9b9c009790 tx=0x7f9b9c0a6550 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:47:33.801 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:33.796+0000 7f9ba27fc640 1 -- 172.21.15.123:0/632290373 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b9c004040 con 0x7f9ba415f720 2024-09-16T15:47:33.801 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:33.796+0000 7f9ba27fc640 1 -- 172.21.15.123:0/632290373 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f9b9c0042d0 con 0x7f9ba415f720 2024-09-16T15:47:33.801 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:33.796+0000 7f9babd02640 1 -- 172.21.15.123:0/632290373 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9ba41f7f50 con 0x7f9ba415f720 2024-09-16T15:47:33.801 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:33.796+0000 7f9ba27fc640 1 -- 172.21.15.123:0/632290373 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b9c0044c0 con 0x7f9ba415f720 2024-09-16T15:47:33.801 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:33.796+0000 7f9babd02640 1 -- 172.21.15.123:0/632290373 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9ba41f8510 con 0x7f9ba415f720 2024-09-16T15:47:33.802 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:33.796+0000 7f9babd02640 1 -- 172.21.15.123:0/632290373 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9ba40b1490 con 0x7f9ba415f720 2024-09-16T15:47:33.802 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:33.796+0000 7f9ba27fc640 1 -- 172.21.15.123:0/632290373 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 12) ==== 97437+0+0 (secure 0 0 0) 0x7f9b9c0c1030 con 0x7f9ba415f720 2024-09-16T15:47:33.802 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:33.796+0000 7f9ba27fc640 1 --2- 172.21.15.123:0/632290373 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f9b900732b0 0x7f9b90075770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:47:33.802 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:33.796+0000 7f9ba27fc640 1 -- 172.21.15.123:0/632290373 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(72..72 src has 1..72) ==== 5767+0+0 (secure 0 0 0) 0x7f9b9c12ec90 con 0x7f9ba415f720 2024-09-16T15:47:33.802 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:33.796+0000 7f9ba9a9e640 1 --2- 172.21.15.123:0/632290373 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f9b900732b0 0x7f9b90075770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:47:33.803 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:33.800+0000 7f9ba9a9e640 1 --2- 172.21.15.123:0/632290373 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f9b900732b0 0x7f9b90075770 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f9b94009e30 tx=0x7f9b940093c0 comp rx=0 tx=0).ready entity=mgr.4135 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:47:33.806 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:33.804+0000 7f9ba27fc640 1 -- 172.21.15.123:0/632290373 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f9b9c0c8020 con 0x7f9ba415f720 2024-09-16T15:47:34.023 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:34.020+0000 7f9babd02640 1 -- 172.21.15.123:0/632290373 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "mgr fail", "who": "x"} v 0) -- 0x7f9ba40cd8c0 con 0x7f9ba415f720 2024-09-16T15:47:34.027 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:34.024+0000 7f9ba27fc640 1 -- 172.21.15.123:0/632290373 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "mgr fail", "who": "x"}]=0 v13) ==== 68+0+0 (secure 0 0 0) 0x7f9b9c0163f0 con 0x7f9ba415f720 2024-09-16T15:47:34.031 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:34.028+0000 7f9babd02640 1 -- 172.21.15.123:0/632290373 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f9b900732b0 msgr2=0x7f9b90075770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:47:34.031 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:34.028+0000 7f9babd02640 1 --2- 172.21.15.123:0/632290373 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f9b900732b0 0x7f9b90075770 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f9b94009e30 tx=0x7f9b940093c0 comp rx=0 tx=0).stop 2024-09-16T15:47:34.031 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:34.028+0000 7f9babd02640 1 -- 172.21.15.123:0/632290373 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9ba415f720 msgr2=0x7f9ba4152fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:47:34.031 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:34.028+0000 7f9babd02640 1 --2- 172.21.15.123:0/632290373 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9ba415f720 0x7f9ba4152fa0 secure :-1 s=READY pgs=850 cs=0 l=1 rev1=1 crypto rx=0x7f9b9c009790 tx=0x7f9b9c0a6550 comp rx=0 tx=0).stop 2024-09-16T15:47:34.032 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:34.028+0000 7f9babd02640 1 -- 172.21.15.123:0/632290373 shutdown_connections 2024-09-16T15:47:34.032 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:34.028+0000 7f9babd02640 1 --2- 172.21.15.123:0/632290373 >> [v2:172.21.15.184:6832/3114346423,v1:172.21.15.184:6833/3114346423] conn(0x7f9b900732b0 0x7f9b90075770 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:34.033 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:34.028+0000 7f9babd02640 1 --2- 172.21.15.123:0/632290373 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9ba415f720 0x7f9ba4152fa0 unknown :-1 s=CLOSED pgs=850 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:34.033 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:34.028+0000 7f9babd02640 1 --2- 172.21.15.123:0/632290373 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f9ba415f340 0x7f9ba4153f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:34.033 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:34.028+0000 7f9babd02640 1 --2- 172.21.15.123:0/632290373 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f9ba40b0f40 0x7f9ba414d7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:34.033 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:34.028+0000 7f9babd02640 1 -- 172.21.15.123:0/632290373 >> 172.21.15.123:0/632290373 conn(0x7f9ba40d1c70 msgr2=0x7f9ba4071f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:47:34.033 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:34.028+0000 7f9babd02640 1 -- 172.21.15.123:0/632290373 shutdown_connections 2024-09-16T15:47:34.033 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:34.028+0000 7f9babd02640 1 -- 172.21.15.123:0/632290373 wait complete. 2024-09-16T15:47:34.041 INFO:tasks.ceph.mgr.x.smithi184.stderr:2024-09-16T15:47:34.037+0000 7fdae394f640 -1 mgr handle_mgr_map I was active but no longer am 2024-09-16T15:47:34.062 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-16T15:47:34.147 INFO:tasks.ceph.mgr.y.smithi123.stderr:2024-09-16T15:47:34.144+0000 7fa9b66fa640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-16T15:47:34.148 INFO:tasks.ceph.mgr.y.smithi123.stderr:2024-09-16T15:47:34.144+0000 7fa9b66fa640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-16T15:47:34.148 INFO:tasks.ceph.mgr.y.smithi123.stderr:2024-09-16T15:47:34.144+0000 7fa9b66fa640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-16T15:47:34.148 INFO:tasks.ceph.mgr.y.smithi123.stderr:2024-09-16T15:47:34.144+0000 7fa9b66fa640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-16T15:47:34.148 INFO:tasks.ceph.mgr.y.smithi123.stderr:2024-09-16T15:47:34.144+0000 7fa9b66fa640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-16T15:47:34.149 INFO:tasks.ceph.mgr.y.smithi123.stderr:2024-09-16T15:47:34.144+0000 7fa9b9700640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-16T15:47:34.149 INFO:tasks.ceph.mgr.y.smithi123.stderr:2024-09-16T15:47:34.144+0000 7fa9b9700640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-16T15:47:34.149 INFO:tasks.ceph.mgr.y.smithi123.stderr:2024-09-16T15:47:34.144+0000 7fa9b9700640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-16T15:47:34.149 INFO:tasks.ceph.mgr.y.smithi123.stderr:2024-09-16T15:47:34.144+0000 7fa9b9700640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-16T15:47:34.150 INFO:tasks.ceph.mgr.y.smithi123.stderr:2024-09-16T15:47:34.144+0000 7fa9b9700640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-16T15:47:34.186 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:34.180+0000 7f915db50640 1 -- 172.21.15.123:0/3313086179 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f91580b2b00 msgr2=0x7f91580d24b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:47:34.186 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:34.180+0000 7f915db50640 1 --2- 172.21.15.123:0/3313086179 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f91580b2b00 0x7f91580d24b0 secure :-1 s=READY pgs=857 cs=0 l=1 rev1=1 crypto rx=0x7f915808d220 tx=0x7f914801c770 comp rx=0 tx=0).stop 2024-09-16T15:47:34.186 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:34.180+0000 7f915db50640 1 -- 172.21.15.123:0/3313086179 shutdown_connections 2024-09-16T15:47:34.186 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:34.180+0000 7f915db50640 1 --2- 172.21.15.123:0/3313086179 >> v2:172.21.15.184:3300/0 conn(0x7f91580b0a50 0x7f91580b0ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:34.186 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:34.180+0000 7f915db50640 1 --2- 172.21.15.123:0/3313086179 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f91580b2b00 0x7f91580d24b0 unknown :-1 s=CLOSED pgs=857 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:34.186 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:34.180+0000 7f915db50640 1 -- 172.21.15.123:0/3313086179 >> 172.21.15.123:0/3313086179 conn(0x7f91580d3ea0 msgr2=0x7f91580d42b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:47:34.186 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:34.180+0000 7f915db50640 1 -- 172.21.15.123:0/3313086179 shutdown_connections 2024-09-16T15:47:34.187 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:34.180+0000 7f915db50640 1 -- 172.21.15.123:0/3313086179 wait complete. 2024-09-16T15:47:34.187 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:34.184+0000 7f915db50640 1 Processor -- start 2024-09-16T15:47:34.187 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:34.184+0000 7f915db50640 1 -- start start 2024-09-16T15:47:34.189 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:34.184+0000 7f915db50640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f91580b0a50 0x7f91580c9180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:47:34.190 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:34.184+0000 7f915db50640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f91580b2b00 0x7f915815bd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:47:34.190 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:34.184+0000 7f915db50640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f9158161620 0x7f915815c2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:47:34.190 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:34.184+0000 7f915db50640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f915816b0f0 con 0x7f91580b2b00 2024-09-16T15:47:34.190 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:34.184+0000 7f915db50640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f915816af70 con 0x7f9158161620 2024-09-16T15:47:34.190 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:34.184+0000 7f915db50640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f915816b270 con 0x7f91580b0a50 2024-09-16T15:47:34.190 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:34.184+0000 7f9156ffd640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f91580b2b00 0x7f915815bd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:47:34.190 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:34.184+0000 7f9156ffd640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f91580b2b00 0x7f915815bd80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:52922/0 (socket says 172.21.15.123:52922) 2024-09-16T15:47:34.190 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:34.184+0000 7f9156ffd640 1 -- 172.21.15.123:0/3170524592 learned_addr learned my addr 172.21.15.123:0/3170524592 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:47:34.190 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:34.184+0000 7f9157fff640 1 --2- 172.21.15.123:0/3170524592 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f9158161620 0x7f915815c2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:47:34.190 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:34.184+0000 7f91577fe640 1 --2- 172.21.15.123:0/3170524592 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f91580b0a50 0x7f91580c9180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:47:34.190 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:34.184+0000 7f9156ffd640 1 -- 172.21.15.123:0/3170524592 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f91580b0a50 msgr2=0x7f91580c9180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:47:34.190 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:34.184+0000 7f9156ffd640 1 --2- 172.21.15.123:0/3170524592 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f91580b0a50 0x7f91580c9180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:34.190 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:34.184+0000 7f9156ffd640 1 -- 172.21.15.123:0/3170524592 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f9158161620 msgr2=0x7f915815c2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:47:34.191 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:34.184+0000 7f9156ffd640 1 --2- 172.21.15.123:0/3170524592 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f9158161620 0x7f915815c2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:34.191 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:34.184+0000 7f9156ffd640 1 -- 172.21.15.123:0/3170524592 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f915815a5a0 con 0x7f91580b2b00 2024-09-16T15:47:34.191 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:34.184+0000 7f9156ffd640 1 --2- 172.21.15.123:0/3170524592 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f91580b2b00 0x7f915815bd80 secure :-1 s=READY pgs=858 cs=0 l=1 rev1=1 crypto rx=0x7f9148000c10 tx=0x7f91480a5690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:47:34.191 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:34.184+0000 7f913ffff640 1 -- 172.21.15.123:0/3170524592 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91480b2040 con 0x7f91580b2b00 2024-09-16T15:47:34.191 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:34.184+0000 7f913ffff640 1 -- 172.21.15.123:0/3170524592 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f9148004430 con 0x7f91580b2b00 2024-09-16T15:47:34.191 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:34.184+0000 7f915db50640 1 -- 172.21.15.123:0/3170524592 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9158147710 con 0x7f91580b2b00 2024-09-16T15:47:34.191 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:34.184+0000 7f913ffff640 1 -- 172.21.15.123:0/3170524592 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9148003860 con 0x7f91580b2b00 2024-09-16T15:47:34.191 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:34.184+0000 7f915db50640 1 -- 172.21.15.123:0/3170524592 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9158147cd0 con 0x7f91580b2b00 2024-09-16T15:47:34.191 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:34.184+0000 7f915db50640 1 -- 172.21.15.123:0/3170524592 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f91580b0ea0 con 0x7f91580b2b00 2024-09-16T15:47:34.191 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:34.184+0000 7f913ffff640 1 -- 172.21.15.123:0/3170524592 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 13) ==== 48578+0+0 (secure 0 0 0) 0x7f91480bf060 con 0x7f91580b2b00 2024-09-16T15:47:34.191 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:34.184+0000 7f913ffff640 1 -- 172.21.15.123:0/3170524592 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(73..73 src has 1..73) ==== 6326+0+0 (secure 0 0 0) 0x7f91480e9c10 con 0x7f91580b2b00 2024-09-16T15:47:34.194 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:34.188+0000 7f913ffff640 1 -- 172.21.15.123:0/3170524592 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f91480164b0 con 0x7f91580b2b00 2024-09-16T15:47:34.385 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:34.380+0000 7f915db50640 1 -- 172.21.15.123:0/3170524592 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f91580b27a0 con 0x7f91580b2b00 2024-09-16T15:47:34.386 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:34.384+0000 7f913ffff640 1 -- 172.21.15.123:0/3170524592 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 45 v45) ==== 94+0+5234 (secure 0 0 0) 0x7f91480c03b0 con 0x7f91580b2b00 2024-09-16T15:47:34.387 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-09-16T15:47:34.387 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":45,"btime":"2024-09-16T15:47:16:409841+0000","default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5048,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6837/1498917352","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6835","nonce":1498917352},{"type":"v1","addr":"172.21.15.184:6837","nonce":1498917352}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":40},{"gid":6198,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6833/1715139909","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6832","nonce":1715139909},{"type":"v1","addr":"172.21.15.123:6833","nonce":1715139909}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":40},{"gid":6201,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6835/815923338","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6834","nonce":815923338},{"type":"v1","addr":"172.21.15.123:6835","nonce":815923338}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":40}],"filesystems":[{"mdsmap":{"epoch":44,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-16T15:46:58.916477+0000","modified":"2024-09-16T15:47:01.963874+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6216},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6216":{"gid":6216,"name":"d","rank":0,"incarnation":41,"state":"up:active","state_seq":3,"addr":"172.21.15.184:6836/3137795986","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":3137795986},{"type":"v1","addr":"172.21.15.184:6836","nonce":3137795986}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6216,"qdb_cluster":[6216]},"id":5}]} 2024-09-16T15:47:34.387 INFO:teuthology.orchestra.run.smithi123.stderr:dumped fsmap epoch 45 2024-09-16T15:47:34.389 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:34.384+0000 7f915db50640 1 -- 172.21.15.123:0/3170524592 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f91580b2b00 msgr2=0x7f915815bd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:47:34.389 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:34.384+0000 7f915db50640 1 --2- 172.21.15.123:0/3170524592 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f91580b2b00 0x7f915815bd80 secure :-1 s=READY pgs=858 cs=0 l=1 rev1=1 crypto rx=0x7f9148000c10 tx=0x7f91480a5690 comp rx=0 tx=0).stop 2024-09-16T15:47:34.389 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:34.384+0000 7f915db50640 1 -- 172.21.15.123:0/3170524592 shutdown_connections 2024-09-16T15:47:34.389 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:34.384+0000 7f915db50640 1 --2- 172.21.15.123:0/3170524592 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f9158161620 0x7f915815c2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:34.389 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:34.384+0000 7f915db50640 1 --2- 172.21.15.123:0/3170524592 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f91580b2b00 0x7f915815bd80 unknown :-1 s=CLOSED pgs=858 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:34.389 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:34.384+0000 7f915db50640 1 --2- 172.21.15.123:0/3170524592 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f91580b0a50 0x7f91580c9180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:34.389 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:34.384+0000 7f915db50640 1 -- 172.21.15.123:0/3170524592 >> 172.21.15.123:0/3170524592 conn(0x7f91580d3ea0 msgr2=0x7f91580d4280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:47:34.389 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:34.384+0000 7f915db50640 1 -- 172.21.15.123:0/3170524592 shutdown_connections 2024-09-16T15:47:34.389 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:34.384+0000 7f915db50640 1 -- 172.21.15.123:0/3170524592 wait complete. 2024-09-16T15:47:34.403 DEBUG:teuthology.orchestra.run.smithi184:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.d.asok --format=json session ls 2024-09-16T15:47:34.541 INFO:teuthology.orchestra.run.smithi184.stdout:[{"id":5186,"entity":{"name":{"type":"client","num":5186},"addr":{"type":"any","addr":"172.21.15.123:0","nonce":1139710172}},"state":"open","num_leases":0,"num_caps":2,"request_load_avg":2,"uptime":0.37599664100000002,"requests_in_flight":0,"num_completed_requests":0,"num_completed_flushes":0,"reconnecting":false,"recall_caps":{"value":0,"halflife":60},"release_caps":{"value":0,"halflife":60},"recall_caps_throttle":{"value":0,"halflife":1.5},"recall_caps_throttle2o":{"value":0,"halflife":0.5},"session_cache_liveness":{"value":1.9982817488721052,"halflife":300},"cap_acquisition":{"value":0,"halflife":30},"last_trim_completed_requests_tid":2,"last_trim_completed_flushes_tid":0,"delegated_inos":[],"inst":"client.5186 172.21.15.123:0/1139710172","completed_requests":[],"prealloc_inos":[],"client_metadata":{"client_features":{"feature_bits":"0x00000000003fffff"},"metric_spec":{"metric_flags":{"feature_bits":"0x000000000000ffff"}},"ceph_sha1":"33f95d2ae1bcbe7eb04404303f179eeeaa6982df","ceph_version":"ceph version 19.3.0-4938-g33f95d2a (33f95d2ae1bcbe7eb04404303f179eeeaa6982df) squid (dev)","entity_id":"y","hostname":"smithi123","pid":"28877","root":"/"}},{"id":5183,"entity":{"name":{"type":"client","num":5183},"addr":{"type":"any","addr":"172.21.15.123:0","nonce":50140765}},"state":"open","num_leases":0,"num_caps":2,"request_load_avg":2,"uptime":0.37999660499999999,"requests_in_flight":0,"num_completed_requests":0,"num_completed_flushes":0,"reconnecting":false,"recall_caps":{"value":0,"halflife":60},"release_caps":{"value":0,"halflife":60},"recall_caps_throttle":{"value":0,"halflife":1.5},"recall_caps_throttle2o":{"value":0,"halflife":0.5},"session_cache_liveness":{"value":1.9982817488721052,"halflife":300},"cap_acquisition":{"value":0,"halflife":30},"last_trim_completed_requests_tid":2,"last_trim_completed_flushes_tid":0,"delegated_inos":[],"inst":"client.5183 172.21.15.123:0/50140765","completed_requests":[],"prealloc_inos":[],"client_metadata":{"client_features":{"feature_bits":"0x00000000003fffff"},"metric_spec":{"metric_flags":{"feature_bits":"0x000000000000ffff"}},"ceph_sha1":"33f95d2ae1bcbe7eb04404303f179eeeaa6982df","ceph_version":"ceph version 19.3.0-4938-g33f95d2a (33f95d2ae1bcbe7eb04404303f179eeeaa6982df) squid (dev)","entity_id":"y","hostname":"smithi123","pid":"28877","root":"/"}}] 2024-09-16T15:47:34.542 DEBUG:tasks.cephfs.filesystem:_json_asok output [ { "cap_acquisition": { "halflife": 30, "value": 0 }, "client_metadata": { "ceph_sha1": "33f95d2ae1bcbe7eb04404303f179eeeaa6982df", "ceph_version": "ceph version 19.3.0-4938-g33f95d2a (33f95d2ae1bcbe7eb04404303f179eeeaa6982df) squid (dev)", "client_features": { "feature_bits": "0x00000000003fffff" }, "entity_id": "y", "hostname": "smithi123", "metric_spec": { "metric_flags": { "feature_bits": "0x000000000000ffff" } }, "pid": "28877", "root": "/" }, "completed_requests": [], "delegated_inos": [], "entity": { "addr": { "addr": "172.21.15.123:0", "nonce": 1139710172, "type": "any" }, "name": { "num": 5186, "type": "client" } }, "id": 5186, "inst": "client.5186 172.21.15.123:0/1139710172", "last_trim_completed_flushes_tid": 0, "last_trim_completed_requests_tid": 2, "num_caps": 2, "num_completed_flushes": 0, "num_completed_requests": 0, "num_leases": 0, "prealloc_inos": [], "recall_caps": { "halflife": 60, "value": 0 }, "recall_caps_throttle": { "halflife": 1.5, "value": 0 }, "recall_caps_throttle2o": { "halflife": 0.5, "value": 0 }, "reconnecting": false, "release_caps": { "halflife": 60, "value": 0 }, "request_load_avg": 2, "requests_in_flight": 0, "session_cache_liveness": { "halflife": 300, "value": 1.9982817488721052 }, "state": "open", "uptime": 0.375996641 }, { "cap_acquisition": { "halflife": 30, "value": 0 }, "client_metadata": { "ceph_sha1": "33f95d2ae1bcbe7eb04404303f179eeeaa6982df", "ceph_version": "ceph version 19.3.0-4938-g33f95d2a (33f95d2ae1bcbe7eb04404303f179eeeaa6982df) squid (dev)", "client_features": { "feature_bits": "0x00000000003fffff" }, "entity_id": "y", "hostname": "smithi123", "metric_spec": { "metric_flags": { "feature_bits": "0x000000000000ffff" } }, "pid": "28877", "root": "/" }, "completed_requests": [], "delegated_inos": [], "entity": { "addr": { "addr": "172.21.15.123:0", "nonce": 50140765, "type": "any" }, "name": { "num": 5183, "type": "client" } }, "id": 5183, "inst": "client.5183 172.21.15.123:0/50140765", "last_trim_completed_flushes_tid": 0, "last_trim_completed_requests_tid": 2, "num_caps": 2, "num_completed_flushes": 0, "num_completed_requests": 0, "num_leases": 0, "prealloc_inos": [], "recall_caps": { "halflife": 60, "value": 0 }, "recall_caps_throttle": { "halflife": 1.5, "value": 0 }, "recall_caps_throttle2o": { "halflife": 0.5, "value": 0 }, "reconnecting": false, "release_caps": { "halflife": 60, "value": 0 }, "request_load_avg": 2, "requests_in_flight": 0, "session_cache_liveness": { "halflife": 300, "value": 1.9982817488721052 }, "state": "open", "uptime": 0.379996605 } ] 2024-09-16T15:47:34.542 DEBUG:tasks.ceph_test_case:wait_until_true: success in 0s and 0 retries 2024-09-16T15:47:34.542 INFO:teuthology.nuke:Clearing teuthology firewall rules... 2024-09-16T15:47:34.542 DEBUG:teuthology.orchestra.run.smithi123:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-09-16T15:47:34.620 DEBUG:teuthology.orchestra.run.smithi184:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-09-16T15:47:34.692 INFO:teuthology.nuke:Cleared teuthology firewall rules. 2024-09-16T15:47:34.692 INFO:tasks.cephfs.mount:Terminating background process 2024-09-16T15:47:34.692 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2024-09-16T15:47:34.692 DEBUG:teuthology.orchestra.run.smithi123:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-16T15:47:34.698 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16336468k,nr_inodes=4084117,mode=755,inode64 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3274684k,mode=755,inode64 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev,inode64 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=17082 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3274680k,nr_inodes=818670,mode=700,uid=1000,gid=1271,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3274684k,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-09-16T15:47:34.698 INFO:tasks.cephfs.mount:Terminating background process 2024-09-16T15:47:34.698 DEBUG:teuthology.orchestra.run.smithi184:> set -ex 2024-09-16T15:47:34.698 DEBUG:teuthology.orchestra.run.smithi184:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-16T15:47:34.704 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16336488k,nr_inodes=4084122,mode=755,inode64 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3274688k,mode=755,inode64 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev,inode64 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=17834 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3274684k,nr_inodes=818671,mode=700,uid=1000,gid=1271,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3274688k,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-09-16T15:47:34.704 INFO:tasks.cephfs.mount:Terminating background process 2024-09-16T15:47:34.704 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2024-09-16T15:47:34.704 DEBUG:teuthology.orchestra.run.smithi123:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-16T15:47:34.747 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16336468k,nr_inodes=4084117,mode=755,inode64 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3274684k,mode=755,inode64 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev,inode64 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=17082 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3274680k,nr_inodes=818670,mode=700,uid=1000,gid=1271,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3274684k,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-09-16T15:47:34.748 INFO:tasks.cephfs.mount:Terminating background process 2024-09-16T15:47:34.748 DEBUG:teuthology.orchestra.run.smithi184:> set -ex 2024-09-16T15:47:34.748 DEBUG:teuthology.orchestra.run.smithi184:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-16T15:47:34.753 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16336488k,nr_inodes=4084122,mode=755,inode64 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3274688k,mode=755,inode64 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev,inode64 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=17834 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3274684k,nr_inodes=818671,mode=700,uid=1000,gid=1271,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3274688k,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-09-16T15:47:34.753 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-16T15:47:34.925 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:34.920+0000 7f1104c0a640 1 -- 172.21.15.123:0/804367551 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f110015ac00 msgr2=0x7f11000702f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:47:34.925 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:34.920+0000 7f1104c0a640 1 --2- 172.21.15.123:0/804367551 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f110015ac00 0x7f11000702f0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f10f8009840 tx=0x7f10f801c820 comp rx=0 tx=0).stop 2024-09-16T15:47:34.925 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:34.920+0000 7f1104c0a640 1 -- 172.21.15.123:0/804367551 shutdown_connections 2024-09-16T15:47:34.926 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:34.920+0000 7f1104c0a640 1 --2- 172.21.15.123:0/804367551 >> v2:172.21.15.184:3300/0 conn(0x7f11000b3100 0x7f11000cc1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:34.926 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:34.920+0000 7f1104c0a640 1 --2- 172.21.15.123:0/804367551 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f110015ac00 0x7f11000702f0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:34.926 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:34.920+0000 7f1104c0a640 1 -- 172.21.15.123:0/804367551 >> 172.21.15.123:0/804367551 conn(0x7f11000b2610 msgr2=0x7f11000b29f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:47:34.926 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:34.920+0000 7f1104c0a640 1 -- 172.21.15.123:0/804367551 shutdown_connections 2024-09-16T15:47:34.926 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:34.920+0000 7f1104c0a640 1 -- 172.21.15.123:0/804367551 wait complete. 2024-09-16T15:47:34.926 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:34.924+0000 7f1104c0a640 1 Processor -- start 2024-09-16T15:47:34.926 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:34.924+0000 7f1104c0a640 1 -- start start 2024-09-16T15:47:34.928 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:34.924+0000 7f1104c0a640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1100070830 0x7f1100153960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:47:34.928 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:34.924+0000 7f1104c0a640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f11000b3100 0x7f1100157510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:47:34.929 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:34.924+0000 7f1104c0a640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f110015ac00 0x7f1100157a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:47:34.929 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:34.924+0000 7f1104c0a640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f1100168660 con 0x7f110015ac00 2024-09-16T15:47:34.929 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:34.924+0000 7f1104c0a640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f11001684e0 con 0x7f1100070830 2024-09-16T15:47:34.929 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:34.924+0000 7f1104c0a640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f11001687e0 con 0x7f11000b3100 2024-09-16T15:47:34.929 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:34.924+0000 7f10fed9d640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f110015ac00 0x7f1100157a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:47:34.929 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:34.924+0000 7f10fed9d640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f110015ac00 0x7f1100157a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:52936/0 (socket says 172.21.15.123:52936) 2024-09-16T15:47:34.929 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:34.924+0000 7f10fed9d640 1 -- 172.21.15.123:0/2678658503 learned_addr learned my addr 172.21.15.123:0/2678658503 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:47:34.929 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:34.924+0000 7f10fe59c640 1 --2- 172.21.15.123:0/2678658503 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1100070830 0x7f1100153960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:47:34.929 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:34.924+0000 7f10fdd9b640 1 --2- 172.21.15.123:0/2678658503 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f11000b3100 0x7f1100157510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:47:34.929 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:34.924+0000 7f10fed9d640 1 -- 172.21.15.123:0/2678658503 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f11000b3100 msgr2=0x7f1100157510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:47:34.929 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:34.924+0000 7f10fed9d640 1 --2- 172.21.15.123:0/2678658503 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f11000b3100 0x7f1100157510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:34.929 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:34.924+0000 7f10fed9d640 1 -- 172.21.15.123:0/2678658503 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1100070830 msgr2=0x7f1100153960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:47:34.929 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:34.924+0000 7f10fed9d640 1 --2- 172.21.15.123:0/2678658503 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1100070830 0x7f1100153960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:34.929 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:34.924+0000 7f10fed9d640 1 -- 172.21.15.123:0/2678658503 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f11001f5e30 con 0x7f110015ac00 2024-09-16T15:47:34.929 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:34.924+0000 7f10fe59c640 1 --2- 172.21.15.123:0/2678658503 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1100070830 0x7f1100153960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T15:47:34.930 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:34.924+0000 7f10fed9d640 1 --2- 172.21.15.123:0/2678658503 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f110015ac00 0x7f1100157a50 secure :-1 s=READY pgs=859 cs=0 l=1 rev1=1 crypto rx=0x7f10f000c730 tx=0x7f10f000cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:47:34.930 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:34.924+0000 7f10f6ffd640 1 -- 172.21.15.123:0/2678658503 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10f0003b80 con 0x7f110015ac00 2024-09-16T15:47:34.930 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:34.924+0000 7f10f6ffd640 1 -- 172.21.15.123:0/2678658503 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f10f0003d20 con 0x7f110015ac00 2024-09-16T15:47:34.930 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:34.924+0000 7f1104c0a640 1 -- 172.21.15.123:0/2678658503 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f11001f60b0 con 0x7f110015ac00 2024-09-16T15:47:34.930 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:34.924+0000 7f10f6ffd640 1 -- 172.21.15.123:0/2678658503 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10f000aab0 con 0x7f110015ac00 2024-09-16T15:47:34.930 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:34.924+0000 7f1104c0a640 1 -- 172.21.15.123:0/2678658503 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f11001f6670 con 0x7f110015ac00 2024-09-16T15:47:34.930 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:34.924+0000 7f10f6ffd640 1 -- 172.21.15.123:0/2678658503 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 13) ==== 48578+0+0 (secure 0 0 0) 0x7f10f000ac50 con 0x7f110015ac00 2024-09-16T15:47:34.930 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:34.924+0000 7f1104c0a640 1 -- 172.21.15.123:0/2678658503 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f11000cc1b0 con 0x7f110015ac00 2024-09-16T15:47:34.931 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:34.924+0000 7f10f6ffd640 1 -- 172.21.15.123:0/2678658503 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(73..73 src has 1..73) ==== 6326+0+0 (secure 0 0 0) 0x7f10f004fba0 con 0x7f110015ac00 2024-09-16T15:47:34.933 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:34.928+0000 7f10f6ffd640 1 -- 172.21.15.123:0/2678658503 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f11001f5e30 con 0x7f110015ac00 2024-09-16T15:47:35.038 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:35.032+0000 7f10f6ffd640 1 -- 172.21.15.123:0/2678658503 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mgrmap(e 14) ==== 48913+0+0 (secure 0 0 0) 0x7f10f0014d40 con 0x7f110015ac00 2024-09-16T15:47:35.038 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:35.032+0000 7f10f6ffd640 1 --2- 172.21.15.123:0/2678658503 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f10e00416d0 0x7f10e0043ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:47:35.040 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:35.036+0000 7f10fe59c640 1 --2- 172.21.15.123:0/2678658503 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f10e00416d0 0x7f10e0043ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:47:35.045 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:35.040+0000 7f10fe59c640 1 --2- 172.21.15.123:0/2678658503 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f10e00416d0 0x7f10e0043ac0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f10f80af000 tx=0x7f10f801b060 comp rx=0 tx=0).ready entity=mgr.4150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:47:35.148 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:35.144+0000 7f1104c0a640 1 -- 172.21.15.123:0/2678658503 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f11000b6aa0 con 0x7f110015ac00 2024-09-16T15:47:35.149 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:35.144+0000 7f10f6ffd640 1 -- 172.21.15.123:0/2678658503 <== mon.0 v2:172.21.15.123:3300/0 8 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 45 v45) ==== 94+0+5234 (secure 0 0 0) 0x7f10f00161c0 con 0x7f110015ac00 2024-09-16T15:47:35.150 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-09-16T15:47:35.150 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":45,"btime":"2024-09-16T15:47:16:409841+0000","default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5048,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6837/1498917352","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6835","nonce":1498917352},{"type":"v1","addr":"172.21.15.184:6837","nonce":1498917352}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":40},{"gid":6198,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6833/1715139909","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6832","nonce":1715139909},{"type":"v1","addr":"172.21.15.123:6833","nonce":1715139909}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":40},{"gid":6201,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6835/815923338","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6834","nonce":815923338},{"type":"v1","addr":"172.21.15.123:6835","nonce":815923338}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":40}],"filesystems":[{"mdsmap":{"epoch":44,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-16T15:46:58.916477+0000","modified":"2024-09-16T15:47:01.963874+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6216},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6216":{"gid":6216,"name":"d","rank":0,"incarnation":41,"state":"up:active","state_seq":3,"addr":"172.21.15.184:6836/3137795986","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":3137795986},{"type":"v1","addr":"172.21.15.184:6836","nonce":3137795986}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6216,"qdb_cluster":[6216]},"id":5}]} 2024-09-16T15:47:35.150 INFO:teuthology.orchestra.run.smithi123.stderr:dumped fsmap epoch 45 2024-09-16T15:47:35.152 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:35.148+0000 7f1104c0a640 1 -- 172.21.15.123:0/2678658503 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f10e00416d0 msgr2=0x7f10e0043ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:47:35.152 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:35.148+0000 7f1104c0a640 1 --2- 172.21.15.123:0/2678658503 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f10e00416d0 0x7f10e0043ac0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f10f80af000 tx=0x7f10f801b060 comp rx=0 tx=0).stop 2024-09-16T15:47:35.152 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:35.148+0000 7f1104c0a640 1 -- 172.21.15.123:0/2678658503 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f110015ac00 msgr2=0x7f1100157a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:47:35.152 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:35.148+0000 7f1104c0a640 1 --2- 172.21.15.123:0/2678658503 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f110015ac00 0x7f1100157a50 secure :-1 s=READY pgs=859 cs=0 l=1 rev1=1 crypto rx=0x7f10f000c730 tx=0x7f10f000cbf0 comp rx=0 tx=0).stop 2024-09-16T15:47:35.152 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:35.148+0000 7f1104c0a640 1 -- 172.21.15.123:0/2678658503 shutdown_connections 2024-09-16T15:47:35.153 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:35.148+0000 7f1104c0a640 1 --2- 172.21.15.123:0/2678658503 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f10e00416d0 0x7f10e0043ac0 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:35.153 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:35.148+0000 7f1104c0a640 1 --2- 172.21.15.123:0/2678658503 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f110015ac00 0x7f1100157a50 unknown :-1 s=CLOSED pgs=859 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:35.153 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:35.148+0000 7f1104c0a640 1 --2- 172.21.15.123:0/2678658503 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f11000b3100 0x7f1100157510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:35.153 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:35.148+0000 7f1104c0a640 1 --2- 172.21.15.123:0/2678658503 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1100070830 0x7f1100153960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:35.153 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:35.148+0000 7f1104c0a640 1 -- 172.21.15.123:0/2678658503 >> 172.21.15.123:0/2678658503 conn(0x7f11000b2610 msgr2=0x7f1100071830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:47:35.153 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:35.148+0000 7f1104c0a640 1 -- 172.21.15.123:0/2678658503 shutdown_connections 2024-09-16T15:47:35.153 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:35.148+0000 7f1104c0a640 1 -- 172.21.15.123:0/2678658503 wait complete. 2024-09-16T15:47:35.167 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-16T15:47:35.280 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:35.276+0000 7f72d9f6a640 1 -- 172.21.15.123:0/1757145229 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f72bc004810 msgr2=0x7f72bc004c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:47:35.280 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:35.276+0000 7f72d9f6a640 1 --2- 172.21.15.123:0/1757145229 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f72bc004810 0x7f72bc004c90 secure :-1 s=READY pgs=861 cs=0 l=1 rev1=1 crypto rx=0x7f72c800c2e0 tx=0x7f72c801f6e0 comp rx=0 tx=0).stop 2024-09-16T15:47:35.280 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:35.276+0000 7f72d9f6a640 1 -- 172.21.15.123:0/1757145229 shutdown_connections 2024-09-16T15:47:35.280 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:35.276+0000 7f72d9f6a640 1 --2- 172.21.15.123:0/1757145229 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f72bc004810 0x7f72bc004c90 unknown :-1 s=CLOSED pgs=861 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:35.280 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:35.276+0000 7f72d9f6a640 1 -- 172.21.15.123:0/1757145229 >> 172.21.15.123:0/1757145229 conn(0x7f72d40b1df0 msgr2=0x7f72d40b2200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:47:35.281 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:35.276+0000 7f72d9f6a640 1 -- 172.21.15.123:0/1757145229 shutdown_connections 2024-09-16T15:47:35.281 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:35.276+0000 7f72d9f6a640 1 -- 172.21.15.123:0/1757145229 wait complete. 2024-09-16T15:47:35.281 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:35.276+0000 7f72d9f6a640 1 Processor -- start 2024-09-16T15:47:35.281 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:35.276+0000 7f72d9f6a640 1 -- start start 2024-09-16T15:47:35.283 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:35.276+0000 7f72d9f6a640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f72bc004810 0x7f72d40c6260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:47:35.283 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:35.276+0000 7f72d9f6a640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f72d40b5330 0x7f72d4155ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:47:35.283 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:35.276+0000 7f72d9f6a640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f72d4154c60 0x7f72d4156220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:47:35.284 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:35.276+0000 7f72d9f6a640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f72d4068e00 con 0x7f72bc004810 2024-09-16T15:47:35.284 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:35.276+0000 7f72d9f6a640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f72d4068c80 con 0x7f72d4154c60 2024-09-16T15:47:35.284 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:35.276+0000 7f72d9f6a640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f72d40741c0 con 0x7f72d40b5330 2024-09-16T15:47:35.284 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:35.280+0000 7f72d37fe640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f72bc004810 0x7f72d40c6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:47:35.284 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:35.280+0000 7f72d37fe640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f72bc004810 0x7f72d40c6260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:52956/0 (socket says 172.21.15.123:52956) 2024-09-16T15:47:35.284 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:35.280+0000 7f72d37fe640 1 -- 172.21.15.123:0/3137418165 learned_addr learned my addr 172.21.15.123:0/3137418165 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:47:35.284 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:35.280+0000 7f72d3fff640 1 --2- 172.21.15.123:0/3137418165 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f72d4154c60 0x7f72d4156220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:47:35.284 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:35.280+0000 7f72d2ffd640 1 --2- 172.21.15.123:0/3137418165 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f72d40b5330 0x7f72d4155ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:47:35.284 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:35.280+0000 7f72d37fe640 1 -- 172.21.15.123:0/3137418165 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f72d40b5330 msgr2=0x7f72d4155ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:47:35.284 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:35.280+0000 7f72d37fe640 1 --2- 172.21.15.123:0/3137418165 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f72d40b5330 0x7f72d4155ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:35.284 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:35.280+0000 7f72d37fe640 1 -- 172.21.15.123:0/3137418165 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f72d4154c60 msgr2=0x7f72d4156220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:47:35.284 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:35.280+0000 7f72d37fe640 1 --2- 172.21.15.123:0/3137418165 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f72d4154c60 0x7f72d4156220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:35.284 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:35.280+0000 7f72d37fe640 1 -- 172.21.15.123:0/3137418165 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f72d41f4910 con 0x7f72bc004810 2024-09-16T15:47:35.285 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:35.280+0000 7f72d3fff640 1 --2- 172.21.15.123:0/3137418165 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f72d4154c60 0x7f72d4156220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T15:47:35.285 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:35.280+0000 7f72d37fe640 1 --2- 172.21.15.123:0/3137418165 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f72bc004810 0x7f72d40c6260 secure :-1 s=READY pgs=862 cs=0 l=1 rev1=1 crypto rx=0x7f72c400d4e0 tx=0x7f72c400d9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:47:35.285 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:35.280+0000 7f72bbfff640 1 -- 172.21.15.123:0/3137418165 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72c4016020 con 0x7f72bc004810 2024-09-16T15:47:35.285 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:35.280+0000 7f72bbfff640 1 -- 172.21.15.123:0/3137418165 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f72c4012630 con 0x7f72bc004810 2024-09-16T15:47:35.285 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:35.280+0000 7f72d9f6a640 1 -- 172.21.15.123:0/3137418165 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f72d41f4bf0 con 0x7f72bc004810 2024-09-16T15:47:35.285 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:35.280+0000 7f72bbfff640 1 -- 172.21.15.123:0/3137418165 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72c4012a80 con 0x7f72bc004810 2024-09-16T15:47:35.285 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:35.280+0000 7f72d9f6a640 1 -- 172.21.15.123:0/3137418165 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f72d41f51b0 con 0x7f72bc004810 2024-09-16T15:47:35.286 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:35.280+0000 7f72bbfff640 1 -- 172.21.15.123:0/3137418165 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 14) ==== 48913+0+0 (secure 0 0 0) 0x7f72c4019020 con 0x7f72bc004810 2024-09-16T15:47:35.286 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:35.280+0000 7f72d9f6a640 1 -- 172.21.15.123:0/3137418165 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f72d40b5710 con 0x7f72bc004810 2024-09-16T15:47:35.286 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:35.280+0000 7f72bbfff640 1 --2- 172.21.15.123:0/3137418165 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f72c003ba50 0x7f72c003df10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:47:35.286 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:35.280+0000 7f72bbfff640 1 -- 172.21.15.123:0/3137418165 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(73..73 src has 1..73) ==== 6326+0+0 (secure 0 0 0) 0x7f72c4050340 con 0x7f72bc004810 2024-09-16T15:47:35.286 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:35.280+0000 7f72d2ffd640 1 --2- 172.21.15.123:0/3137418165 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f72c003ba50 0x7f72c003df10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:47:35.286 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:35.280+0000 7f72d2ffd640 1 --2- 172.21.15.123:0/3137418165 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f72c003ba50 0x7f72c003df10 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f72c80b0000 tx=0x7f72c801c520 comp rx=0 tx=0).ready entity=mgr.4150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:47:35.289 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:35.284+0000 7f72bbfff640 1 -- 172.21.15.123:0/3137418165 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f72c4020020 con 0x7f72bc004810 2024-09-16T15:47:35.500 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:35.496+0000 7f72d9f6a640 1 -- 172.21.15.123:0/3137418165 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f72d4074690 con 0x7f72bc004810 2024-09-16T15:47:35.501 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:35.496+0000 7f72bbfff640 1 -- 172.21.15.123:0/3137418165 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 45 v45) ==== 94+0+5234 (secure 0 0 0) 0x7f72c4023350 con 0x7f72bc004810 2024-09-16T15:47:35.501 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-09-16T15:47:35.502 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":45,"btime":"2024-09-16T15:47:16:409841+0000","default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5048,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6837/1498917352","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6835","nonce":1498917352},{"type":"v1","addr":"172.21.15.184:6837","nonce":1498917352}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":40},{"gid":6198,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6833/1715139909","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6832","nonce":1715139909},{"type":"v1","addr":"172.21.15.123:6833","nonce":1715139909}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":40},{"gid":6201,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6835/815923338","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6834","nonce":815923338},{"type":"v1","addr":"172.21.15.123:6835","nonce":815923338}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":40}],"filesystems":[{"mdsmap":{"epoch":44,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-16T15:46:58.916477+0000","modified":"2024-09-16T15:47:01.963874+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6216},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6216":{"gid":6216,"name":"d","rank":0,"incarnation":41,"state":"up:active","state_seq":3,"addr":"172.21.15.184:6836/3137795986","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":3137795986},{"type":"v1","addr":"172.21.15.184:6836","nonce":3137795986}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6216,"qdb_cluster":[6216]},"id":5}]} 2024-09-16T15:47:35.502 INFO:teuthology.orchestra.run.smithi123.stderr:dumped fsmap epoch 45 2024-09-16T15:47:35.504 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:35.500+0000 7f72d9f6a640 1 -- 172.21.15.123:0/3137418165 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f72c003ba50 msgr2=0x7f72c003df10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:47:35.504 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:35.500+0000 7f72d9f6a640 1 --2- 172.21.15.123:0/3137418165 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f72c003ba50 0x7f72c003df10 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f72c80b0000 tx=0x7f72c801c520 comp rx=0 tx=0).stop 2024-09-16T15:47:35.504 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:35.500+0000 7f72d9f6a640 1 -- 172.21.15.123:0/3137418165 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f72bc004810 msgr2=0x7f72d40c6260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:47:35.504 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:35.500+0000 7f72d9f6a640 1 --2- 172.21.15.123:0/3137418165 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f72bc004810 0x7f72d40c6260 secure :-1 s=READY pgs=862 cs=0 l=1 rev1=1 crypto rx=0x7f72c400d4e0 tx=0x7f72c400d9a0 comp rx=0 tx=0).stop 2024-09-16T15:47:35.504 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:35.500+0000 7f72d9f6a640 1 -- 172.21.15.123:0/3137418165 shutdown_connections 2024-09-16T15:47:35.505 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:35.500+0000 7f72d9f6a640 1 --2- 172.21.15.123:0/3137418165 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f72c003ba50 0x7f72c003df10 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:35.505 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:35.500+0000 7f72d9f6a640 1 --2- 172.21.15.123:0/3137418165 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f72d4154c60 0x7f72d4156220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:35.505 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:35.500+0000 7f72d9f6a640 1 --2- 172.21.15.123:0/3137418165 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f72d40b5330 0x7f72d4155ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:35.505 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:35.500+0000 7f72d9f6a640 1 --2- 172.21.15.123:0/3137418165 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f72bc004810 0x7f72d40c6260 unknown :-1 s=CLOSED pgs=862 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:35.505 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:35.500+0000 7f72d9f6a640 1 -- 172.21.15.123:0/3137418165 >> 172.21.15.123:0/3137418165 conn(0x7f72d40b1df0 msgr2=0x7f72d40b4b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:47:35.505 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:35.500+0000 7f72d9f6a640 1 -- 172.21.15.123:0/3137418165 shutdown_connections 2024-09-16T15:47:35.505 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:35.500+0000 7f72d9f6a640 1 -- 172.21.15.123:0/3137418165 wait complete. 2024-09-16T15:47:35.520 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-16T15:47:35.634 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:35.628+0000 7f7adb7d4640 1 -- 172.21.15.123:0/2303076491 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7ad4074470 msgr2=0x7f7ad40cd6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:47:35.634 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:35.628+0000 7f7adb7d4640 1 --2- 172.21.15.123:0/2303076491 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7ad4074470 0x7f7ad40cd6e0 secure :-1 s=READY pgs=864 cs=0 l=1 rev1=1 crypto rx=0x7f7ad4070730 tx=0x7f7ac401c6e0 comp rx=0 tx=0).stop 2024-09-16T15:47:35.634 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:35.628+0000 7f7adb7d4640 1 -- 172.21.15.123:0/2303076491 shutdown_connections 2024-09-16T15:47:35.635 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:35.628+0000 7f7adb7d4640 1 --2- 172.21.15.123:0/2303076491 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7ad4074470 0x7f7ad40cd6e0 unknown :-1 s=CLOSED pgs=864 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:35.635 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:35.628+0000 7f7adb7d4640 1 --2- 172.21.15.123:0/2303076491 >> v2:172.21.15.184:3300/0 conn(0x7f7ad415e160 0x7f7ad4073f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:35.635 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:35.628+0000 7f7adb7d4640 1 -- 172.21.15.123:0/2303076491 >> 172.21.15.123:0/2303076491 conn(0x7f7ad40cfb10 msgr2=0x7f7ad40cff20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:47:35.635 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:35.628+0000 7f7adb7d4640 1 -- 172.21.15.123:0/2303076491 shutdown_connections 2024-09-16T15:47:35.635 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:35.628+0000 7f7adb7d4640 1 -- 172.21.15.123:0/2303076491 wait complete. 2024-09-16T15:47:35.635 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:35.632+0000 7f7adb7d4640 1 Processor -- start 2024-09-16T15:47:35.635 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:35.632+0000 7f7adb7d4640 1 -- start start 2024-09-16T15:47:35.638 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:35.632+0000 7f7adb7d4640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f7ad4074470 0x7f7ad4152160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:47:35.638 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:35.632+0000 7f7adb7d4640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7ad40cdc20 0x7f7ad40b4400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:47:35.638 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:35.632+0000 7f7adb7d4640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7ad415e160 0x7f7ad41516c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:47:35.638 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:35.632+0000 7f7adb7d4640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f7ad4167dd0 con 0x7f7ad40cdc20 2024-09-16T15:47:35.638 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:35.632+0000 7f7adb7d4640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f7ad40cf620 con 0x7f7ad415e160 2024-09-16T15:47:35.638 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:35.632+0000 7f7adb7d4640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f7ad4167f50 con 0x7f7ad4074470 2024-09-16T15:47:35.638 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:35.632+0000 7f7ad8d6f640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7ad40cdc20 0x7f7ad40b4400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:47:35.639 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:35.632+0000 7f7ad8d6f640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7ad40cdc20 0x7f7ad40b4400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:52964/0 (socket says 172.21.15.123:52964) 2024-09-16T15:47:35.639 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:35.632+0000 7f7ad8d6f640 1 -- 172.21.15.123:0/952166405 learned_addr learned my addr 172.21.15.123:0/952166405 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:47:35.639 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:35.632+0000 7f7ad9d71640 1 --2- 172.21.15.123:0/952166405 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7ad415e160 0x7f7ad41516c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:47:35.639 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:35.632+0000 7f7ad9570640 1 --2- 172.21.15.123:0/952166405 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f7ad4074470 0x7f7ad4152160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:47:35.639 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:35.632+0000 7f7ad8d6f640 1 -- 172.21.15.123:0/952166405 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f7ad4074470 msgr2=0x7f7ad4152160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:47:35.639 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:35.632+0000 7f7ad8d6f640 1 --2- 172.21.15.123:0/952166405 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f7ad4074470 0x7f7ad4152160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:35.639 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:35.632+0000 7f7ad8d6f640 1 -- 172.21.15.123:0/952166405 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7ad415e160 msgr2=0x7f7ad41516c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:47:35.639 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:35.632+0000 7f7ad8d6f640 1 --2- 172.21.15.123:0/952166405 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7ad415e160 0x7f7ad41516c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:35.639 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:35.632+0000 7f7ad8d6f640 1 -- 172.21.15.123:0/952166405 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7ad4150900 con 0x7f7ad40cdc20 2024-09-16T15:47:35.639 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:35.632+0000 7f7ad9570640 1 --2- 172.21.15.123:0/952166405 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f7ad4074470 0x7f7ad4152160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T15:47:35.639 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:35.632+0000 7f7ad8d6f640 1 --2- 172.21.15.123:0/952166405 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7ad40cdc20 0x7f7ad40b4400 secure :-1 s=READY pgs=865 cs=0 l=1 rev1=1 crypto rx=0x7f7ac401cbc0 tx=0x7f7ac40a67b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:47:35.639 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:35.632+0000 7f7ac9ffb640 1 -- 172.21.15.123:0/952166405 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ac40be020 con 0x7f7ad40cdc20 2024-09-16T15:47:35.639 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:35.632+0000 7f7ac9ffb640 1 -- 172.21.15.123:0/952166405 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f7ac4004430 con 0x7f7ad40cdc20 2024-09-16T15:47:35.639 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:35.632+0000 7f7adb7d4640 1 -- 172.21.15.123:0/952166405 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7ad4150be0 con 0x7f7ad40cdc20 2024-09-16T15:47:35.639 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:35.632+0000 7f7ac9ffb640 1 -- 172.21.15.123:0/952166405 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ac4003840 con 0x7f7ad40cdc20 2024-09-16T15:47:35.640 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:35.632+0000 7f7adb7d4640 1 -- 172.21.15.123:0/952166405 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7ad41511a0 con 0x7f7ad40cdc20 2024-09-16T15:47:35.640 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:35.632+0000 7f7adb7d4640 1 -- 172.21.15.123:0/952166405 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7ad40cd6e0 con 0x7f7ad40cdc20 2024-09-16T15:47:35.640 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:35.632+0000 7f7ac9ffb640 1 -- 172.21.15.123:0/952166405 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 14) ==== 48913+0+0 (secure 0 0 0) 0x7f7ac40c1020 con 0x7f7ad40cdc20 2024-09-16T15:47:35.640 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:35.632+0000 7f7ac9ffb640 1 --2- 172.21.15.123:0/952166405 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f7ab803ba80 0x7f7ab803df40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:47:35.640 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:35.632+0000 7f7ac9ffb640 1 -- 172.21.15.123:0/952166405 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(73..73 src has 1..73) ==== 6326+0+0 (secure 0 0 0) 0x7f7ac40eb860 con 0x7f7ad40cdc20 2024-09-16T15:47:35.640 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:35.632+0000 7f7ad9570640 1 --2- 172.21.15.123:0/952166405 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f7ab803ba80 0x7f7ab803df40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:47:35.640 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:35.636+0000 7f7ad9570640 1 --2- 172.21.15.123:0/952166405 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f7ab803ba80 0x7f7ab803df40 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f7ad000a870 tx=0x7f7ad0008040 comp rx=0 tx=0).ready entity=mgr.4150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:47:35.642 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:35.640+0000 7f7ac9ffb640 1 -- 172.21.15.123:0/952166405 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f7ac40c8030 con 0x7f7ad40cdc20 2024-09-16T15:47:35.800 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:35.796+0000 7f7adb7d4640 1 -- 172.21.15.123:0/952166405 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f7ad4151370 con 0x7f7ad40cdc20 2024-09-16T15:47:35.803 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:35.800+0000 7f7ac9ffb640 1 -- 172.21.15.123:0/952166405 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v73) ==== 74+0+17705 (secure 0 0 0) 0x7f7ac40163f0 con 0x7f7ad40cdc20 2024-09-16T15:47:35.803 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-09-16T15:47:35.803 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":73,"fsid":"56077f89-18c8-486f-ae7e-eecc0bb7822c","created":"2024-09-16T15:42:26.364892+0000","modified":"2024-09-16T15:47:34.024919+0000","last_up_change":"2024-09-16T15:42:38.707234+0000","last_in_change":"2024-09-16T15:42:29.423662+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-16T15:42:39.287764+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_metadata","create_time":"2024-09-16T15:46:56.639820+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"69","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_data","create_time":"2024-09-16T15:46:57.893968+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"70","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9e7c1955-a607-4671-9be7-f2ef760ceeaa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":71,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6800","nonce":3882044969},{"type":"v1","addr":"172.21.15.123:6801","nonce":3882044969}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6802","nonce":3882044969},{"type":"v1","addr":"172.21.15.123:6803","nonce":3882044969}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6806","nonce":3882044969},{"type":"v1","addr":"172.21.15.123:6807","nonce":3882044969}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6804","nonce":3882044969},{"type":"v1","addr":"172.21.15.123:6805","nonce":3882044969}]},"public_addr":"172.21.15.123:6801/3882044969","cluster_addr":"172.21.15.123:6803/3882044969","heartbeat_back_addr":"172.21.15.123:6807/3882044969","heartbeat_front_addr":"172.21.15.123:6805/3882044969","state":["exists","up"]},{"osd":1,"uuid":"c540cfde-0977-4742-9b17-fd634668e010","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6816","nonce":4037001850},{"type":"v1","addr":"172.21.15.123:6817","nonce":4037001850}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6818","nonce":4037001850},{"type":"v1","addr":"172.21.15.123:6819","nonce":4037001850}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6822","nonce":4037001850},{"type":"v1","addr":"172.21.15.123:6823","nonce":4037001850}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6820","nonce":4037001850},{"type":"v1","addr":"172.21.15.123:6821","nonce":4037001850}]},"public_addr":"172.21.15.123:6817/4037001850","cluster_addr":"172.21.15.123:6819/4037001850","heartbeat_back_addr":"172.21.15.123:6823/4037001850","heartbeat_front_addr":"172.21.15.123:6821/4037001850","state":["exists","up"]},{"osd":2,"uuid":"9864a496-f169-486b-95c4-1d59162793a7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6824","nonce":1332143088},{"type":"v1","addr":"172.21.15.123:6825","nonce":1332143088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6826","nonce":1332143088},{"type":"v1","addr":"172.21.15.123:6827","nonce":1332143088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6830","nonce":1332143088},{"type":"v1","addr":"172.21.15.123:6831","nonce":1332143088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6828","nonce":1332143088},{"type":"v1","addr":"172.21.15.123:6829","nonce":1332143088}]},"public_addr":"172.21.15.123:6825/1332143088","cluster_addr":"172.21.15.123:6827/1332143088","heartbeat_back_addr":"172.21.15.123:6831/1332143088","heartbeat_front_addr":"172.21.15.123:6829/1332143088","state":["exists","up"]},{"osd":3,"uuid":"c6d1f817-f074-4325-95e1-578eba13edda","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6808","nonce":988586998},{"type":"v1","addr":"172.21.15.123:6809","nonce":988586998}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6810","nonce":988586998},{"type":"v1","addr":"172.21.15.123:6811","nonce":988586998}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6814","nonce":988586998},{"type":"v1","addr":"172.21.15.123:6815","nonce":988586998}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6812","nonce":988586998},{"type":"v1","addr":"172.21.15.123:6813","nonce":988586998}]},"public_addr":"172.21.15.123:6809/988586998","cluster_addr":"172.21.15.123:6811/988586998","heartbeat_back_addr":"172.21.15.123:6815/988586998","heartbeat_front_addr":"172.21.15.123:6813/988586998","state":["exists","up"]},{"osd":4,"uuid":"200a8df4-50ef-40a5-b48a-58981269c086","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6816","nonce":3387813890},{"type":"v1","addr":"172.21.15.184:6817","nonce":3387813890}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6818","nonce":3387813890},{"type":"v1","addr":"172.21.15.184:6819","nonce":3387813890}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6822","nonce":3387813890},{"type":"v1","addr":"172.21.15.184:6823","nonce":3387813890}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6820","nonce":3387813890},{"type":"v1","addr":"172.21.15.184:6821","nonce":3387813890}]},"public_addr":"172.21.15.184:6817/3387813890","cluster_addr":"172.21.15.184:6819/3387813890","heartbeat_back_addr":"172.21.15.184:6823/3387813890","heartbeat_front_addr":"172.21.15.184:6821/3387813890","state":["exists","up"]},{"osd":5,"uuid":"32d5a900-da3d-412c-bc19-13861a2368d8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6808","nonce":2680916017},{"type":"v1","addr":"172.21.15.184:6809","nonce":2680916017}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6810","nonce":2680916017},{"type":"v1","addr":"172.21.15.184:6811","nonce":2680916017}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6814","nonce":2680916017},{"type":"v1","addr":"172.21.15.184:6815","nonce":2680916017}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6812","nonce":2680916017},{"type":"v1","addr":"172.21.15.184:6813","nonce":2680916017}]},"public_addr":"172.21.15.184:6809/2680916017","cluster_addr":"172.21.15.184:6811/2680916017","heartbeat_back_addr":"172.21.15.184:6815/2680916017","heartbeat_front_addr":"172.21.15.184:6813/2680916017","state":["exists","up"]},{"osd":6,"uuid":"cde6f39e-aa1b-4260-ac19-774ee6aaca72","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6824","nonce":2183120457},{"type":"v1","addr":"172.21.15.184:6825","nonce":2183120457}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6826","nonce":2183120457},{"type":"v1","addr":"172.21.15.184:6827","nonce":2183120457}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6830","nonce":2183120457},{"type":"v1","addr":"172.21.15.184:6831","nonce":2183120457}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6828","nonce":2183120457},{"type":"v1","addr":"172.21.15.184:6829","nonce":2183120457}]},"public_addr":"172.21.15.184:6825/2183120457","cluster_addr":"172.21.15.184:6827/2183120457","heartbeat_back_addr":"172.21.15.184:6831/2183120457","heartbeat_front_addr":"172.21.15.184:6829/2183120457","state":["exists","up"]},{"osd":7,"uuid":"4f381778-eee1-47c6-804f-df0e6b33e313","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":71,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6800","nonce":4102957285},{"type":"v1","addr":"172.21.15.184:6801","nonce":4102957285}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6802","nonce":4102957285},{"type":"v1","addr":"172.21.15.184:6803","nonce":4102957285}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6806","nonce":4102957285},{"type":"v1","addr":"172.21.15.184:6807","nonce":4102957285}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6804","nonce":4102957285},{"type":"v1","addr":"172.21.15.184:6805","nonce":4102957285}]},"public_addr":"172.21.15.184:6801/4102957285","cluster_addr":"172.21.15.184:6803/4102957285","heartbeat_back_addr":"172.21.15.184:6807/4102957285","heartbeat_front_addr":"172.21.15.184:6805/4102957285","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:35.972210+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.222007+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.198617+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.385663+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.358935+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.156066+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.752973+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:35.682846+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"11.3","mappings":[{"from":6,"to":0}]},{"pgid":"11.28","mappings":[{"from":6,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.184:0/1023177165":"2024-09-17T15:47:34.024879+0000","172.21.15.184:6833/3114346423":"2024-09-17T15:47:34.024879+0000","172.21.15.184:6832/3114346423":"2024-09-17T15:47:34.024879+0000","172.21.15.184:0/2406729396":"2024-09-17T15:47:34.024879+0000","172.21.15.184:0/78639736":"2024-09-17T15:47:34.024879+0000","172.21.15.184:0/3204392673":"2024-09-17T15:47:34.024879+0000","172.21.15.184:0/654280269":"2024-09-17T15:47:34.024879+0000","172.21.15.184:0/2291367472":"2024-09-17T15:47:34.024879+0000","172.21.15.184:0/1715342118":"2024-09-17T15:47:34.024879+0000","172.21.15.184:0/140936988":"2024-09-17T15:47:34.024879+0000","172.21.15.184:0/1402401038":"2024-09-17T15:47:34.024879+0000","172.21.15.184:0/3070886766":"2024-09-17T15:47:34.024879+0000","172.21.15.184:0/2916845314":"2024-09-17T15:47:34.024879+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T15:47:35.805 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:35.800+0000 7f7adb7d4640 1 -- 172.21.15.123:0/952166405 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f7ab803ba80 msgr2=0x7f7ab803df40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:47:35.805 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:35.800+0000 7f7adb7d4640 1 --2- 172.21.15.123:0/952166405 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f7ab803ba80 0x7f7ab803df40 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f7ad000a870 tx=0x7f7ad0008040 comp rx=0 tx=0).stop 2024-09-16T15:47:35.805 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:35.800+0000 7f7adb7d4640 1 -- 172.21.15.123:0/952166405 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7ad40cdc20 msgr2=0x7f7ad40b4400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:47:35.805 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:35.800+0000 7f7adb7d4640 1 --2- 172.21.15.123:0/952166405 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7ad40cdc20 0x7f7ad40b4400 secure :-1 s=READY pgs=865 cs=0 l=1 rev1=1 crypto rx=0x7f7ac401cbc0 tx=0x7f7ac40a67b0 comp rx=0 tx=0).stop 2024-09-16T15:47:35.806 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:35.800+0000 7f7adb7d4640 1 -- 172.21.15.123:0/952166405 shutdown_connections 2024-09-16T15:47:35.806 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:35.800+0000 7f7adb7d4640 1 --2- 172.21.15.123:0/952166405 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f7ab803ba80 0x7f7ab803df40 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:35.806 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:35.800+0000 7f7adb7d4640 1 --2- 172.21.15.123:0/952166405 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7ad415e160 0x7f7ad41516c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:35.806 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:35.800+0000 7f7adb7d4640 1 --2- 172.21.15.123:0/952166405 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7ad40cdc20 0x7f7ad40b4400 unknown :-1 s=CLOSED pgs=865 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:35.806 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:35.800+0000 7f7adb7d4640 1 --2- 172.21.15.123:0/952166405 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f7ad4074470 0x7f7ad4152160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:35.806 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:35.800+0000 7f7adb7d4640 1 -- 172.21.15.123:0/952166405 >> 172.21.15.123:0/952166405 conn(0x7f7ad40cfb10 msgr2=0x7f7ad40758d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:47:35.806 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:35.800+0000 7f7adb7d4640 1 -- 172.21.15.123:0/952166405 shutdown_connections 2024-09-16T15:47:35.806 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:35.800+0000 7f7adb7d4640 1 -- 172.21.15.123:0/952166405 wait complete. 2024-09-16T15:47:35.820 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-09-16T15:47:35.820 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-16T15:47:35.933 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:35.928+0000 7f3f701cb640 1 -- 172.21.15.123:0/626456615 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3f54003a10 msgr2=0x7f3f54003e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:47:35.933 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:35.928+0000 7f3f701cb640 1 --2- 172.21.15.123:0/626456615 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3f54003a10 0x7f3f54003e50 secure :-1 s=READY pgs=867 cs=0 l=1 rev1=1 crypto rx=0x7f3f6400c340 tx=0x7f3f6401cfa0 comp rx=0 tx=0).stop 2024-09-16T15:47:35.933 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:35.928+0000 7f3f6e768640 1 -- 172.21.15.123:0/626456615 reap_dead start 2024-09-16T15:47:35.933 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:35.928+0000 7f3f701cb640 1 -- 172.21.15.123:0/626456615 shutdown_connections 2024-09-16T15:47:35.933 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:35.928+0000 7f3f701cb640 1 -- 172.21.15.123:0/626456615 >> 172.21.15.123:0/626456615 conn(0x7f3f680b2610 msgr2=0x7f3f680b29f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:47:35.933 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:35.928+0000 7f3f701cb640 1 -- 172.21.15.123:0/626456615 shutdown_connections 2024-09-16T15:47:35.933 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:35.928+0000 7f3f701cb640 1 -- 172.21.15.123:0/626456615 wait complete. 2024-09-16T15:47:35.934 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:35.928+0000 7f3f701cb640 1 Processor -- start 2024-09-16T15:47:35.934 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:35.928+0000 7f3f701cb640 1 -- start start 2024-09-16T15:47:35.936 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:35.932+0000 7f3f701cb640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3f54003a10 0x7f3f680cf8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:47:35.937 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:35.932+0000 7f3f701cb640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f3f680d0260 0x7f3f68157d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:47:35.937 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:35.932+0000 7f3f701cb640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3f68160b10 0x7f3f68160f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:47:35.937 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:35.932+0000 7f3f701cb640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f3f68166d80 con 0x7f3f54003a10 2024-09-16T15:47:35.937 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:35.932+0000 7f3f701cb640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f3f6806db40 con 0x7f3f68160b10 2024-09-16T15:47:35.937 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:35.932+0000 7f3f701cb640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f3f68166f00 con 0x7f3f680d0260 2024-09-16T15:47:35.937 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:35.932+0000 7f3f6df67640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3f54003a10 0x7f3f680cf8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:47:35.937 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:35.932+0000 7f3f6df67640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3f54003a10 0x7f3f680cf8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:52978/0 (socket says 172.21.15.123:52978) 2024-09-16T15:47:35.937 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:35.932+0000 7f3f6df67640 1 -- 172.21.15.123:0/1899777754 learned_addr learned my addr 172.21.15.123:0/1899777754 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:47:35.937 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:35.932+0000 7f3f6e768640 1 --2- 172.21.15.123:0/1899777754 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3f68160b10 0x7f3f68160f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:47:35.937 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:35.932+0000 7f3f6d766640 1 --2- 172.21.15.123:0/1899777754 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f3f680d0260 0x7f3f68157d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:47:35.937 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:35.932+0000 7f3f6df67640 1 -- 172.21.15.123:0/1899777754 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f3f680d0260 msgr2=0x7f3f68157d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:47:35.938 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:35.932+0000 7f3f6df67640 1 --2- 172.21.15.123:0/1899777754 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f3f680d0260 0x7f3f68157d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:35.938 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:35.932+0000 7f3f6df67640 1 -- 172.21.15.123:0/1899777754 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3f68160b10 msgr2=0x7f3f68160f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:47:35.938 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:35.932+0000 7f3f6df67640 1 --2- 172.21.15.123:0/1899777754 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3f68160b10 0x7f3f68160f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:35.938 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:35.932+0000 7f3f6df67640 1 -- 172.21.15.123:0/1899777754 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3f680cd350 con 0x7f3f54003a10 2024-09-16T15:47:35.938 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:35.932+0000 7f3f6df67640 1 --2- 172.21.15.123:0/1899777754 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3f54003a10 0x7f3f680cf8c0 secure :-1 s=READY pgs=868 cs=0 l=1 rev1=1 crypto rx=0x7f3f6401b6a0 tx=0x7f3f64006c60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:47:35.938 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:35.932+0000 7f3f5e7fc640 1 -- 172.21.15.123:0/1899777754 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f64006180 con 0x7f3f54003a10 2024-09-16T15:47:35.938 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:35.932+0000 7f3f5e7fc640 1 -- 172.21.15.123:0/1899777754 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f3f64006320 con 0x7f3f54003a10 2024-09-16T15:47:35.938 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:35.932+0000 7f3f701cb640 1 -- 172.21.15.123:0/1899777754 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3f681588d0 con 0x7f3f54003a10 2024-09-16T15:47:35.938 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:35.932+0000 7f3f5e7fc640 1 -- 172.21.15.123:0/1899777754 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f64003600 con 0x7f3f54003a10 2024-09-16T15:47:35.938 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:35.932+0000 7f3f701cb640 1 -- 172.21.15.123:0/1899777754 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3f680addf0 con 0x7f3f54003a10 2024-09-16T15:47:35.938 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:35.932+0000 7f3f701cb640 1 -- 172.21.15.123:0/1899777754 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3f6806cbf0 con 0x7f3f54003a10 2024-09-16T15:47:35.938 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:35.932+0000 7f3f5e7fc640 1 -- 172.21.15.123:0/1899777754 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 14) ==== 48913+0+0 (secure 0 0 0) 0x7f3f640037a0 con 0x7f3f54003a10 2024-09-16T15:47:35.938 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:35.932+0000 7f3f5e7fc640 1 --2- 172.21.15.123:0/1899777754 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f3f4c03ba50 0x7f3f4c03df10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:47:35.938 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:35.932+0000 7f3f5e7fc640 1 -- 172.21.15.123:0/1899777754 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(73..73 src has 1..73) ==== 6326+0+0 (secure 0 0 0) 0x7f3f640ea740 con 0x7f3f54003a10 2024-09-16T15:47:35.939 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:35.932+0000 7f3f6d766640 1 --2- 172.21.15.123:0/1899777754 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f3f4c03ba50 0x7f3f4c03df10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:47:35.939 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:35.932+0000 7f3f6d766640 1 --2- 172.21.15.123:0/1899777754 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f3f4c03ba50 0x7f3f4c03df10 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f3f580058d0 tx=0x7f3f5800b2c0 comp rx=0 tx=0).ready entity=mgr.4150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:47:35.941 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:35.936+0000 7f3f5e7fc640 1 -- 172.21.15.123:0/1899777754 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f3f64016d50 con 0x7f3f54003a10 2024-09-16T15:47:36.047 INFO:tasks.ceph.mgr.x.smithi184.stderr:/usr/lib/python3/dist-packages/scipy/__init__.py:67: UserWarning: NumPy was imported from a Python sub-interpreter but NumPy does not properly support sub-interpreters. This will likely work for most users but might cause hard to track down issues or subtle bugs. A common user of the rare sub-interpreter feature is wsgi which also allows single-interpreter mode. 2024-09-16T15:47:36.047 INFO:tasks.ceph.mgr.x.smithi184.stderr:Improvements in the case of bugs are welcome, but is not on the NumPy roadmap, and full support may require significant effort to achieve. 2024-09-16T15:47:36.047 INFO:tasks.ceph.mgr.x.smithi184.stderr: from numpy import show_config as show_numpy_config 2024-09-16T15:47:36.137 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:36.132+0000 7f3f701cb640 1 -- 172.21.15.123:0/1899777754 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f3f680b2d70 con 0x7f3f54003a10 2024-09-16T15:47:36.138 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:36.132+0000 7f3f5e7fc640 1 -- 172.21.15.123:0/1899777754 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 45 v45) ==== 94+0+5234 (secure 0 0 0) 0x7f3f640ad3d0 con 0x7f3f54003a10 2024-09-16T15:47:36.138 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-09-16T15:47:36.138 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":45,"btime":"2024-09-16T15:47:16:409841+0000","default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5048,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6837/1498917352","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6835","nonce":1498917352},{"type":"v1","addr":"172.21.15.184:6837","nonce":1498917352}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":40},{"gid":6198,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6833/1715139909","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6832","nonce":1715139909},{"type":"v1","addr":"172.21.15.123:6833","nonce":1715139909}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":40},{"gid":6201,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6835/815923338","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6834","nonce":815923338},{"type":"v1","addr":"172.21.15.123:6835","nonce":815923338}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":40}],"filesystems":[{"mdsmap":{"epoch":44,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-16T15:46:58.916477+0000","modified":"2024-09-16T15:47:01.963874+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6216},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6216":{"gid":6216,"name":"d","rank":0,"incarnation":41,"state":"up:active","state_seq":3,"addr":"172.21.15.184:6836/3137795986","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":3137795986},{"type":"v1","addr":"172.21.15.184:6836","nonce":3137795986}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6216,"qdb_cluster":[6216]},"id":5}]} 2024-09-16T15:47:36.138 INFO:teuthology.orchestra.run.smithi123.stderr:dumped fsmap epoch 45 2024-09-16T15:47:36.141 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:36.136+0000 7f3f701cb640 1 -- 172.21.15.123:0/1899777754 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f3f4c03ba50 msgr2=0x7f3f4c03df10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:47:36.141 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:36.136+0000 7f3f701cb640 1 --2- 172.21.15.123:0/1899777754 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f3f4c03ba50 0x7f3f4c03df10 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f3f580058d0 tx=0x7f3f5800b2c0 comp rx=0 tx=0).stop 2024-09-16T15:47:36.141 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:36.136+0000 7f3f701cb640 1 -- 172.21.15.123:0/1899777754 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3f54003a10 msgr2=0x7f3f680cf8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:47:36.141 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:36.136+0000 7f3f701cb640 1 --2- 172.21.15.123:0/1899777754 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3f54003a10 0x7f3f680cf8c0 secure :-1 s=READY pgs=868 cs=0 l=1 rev1=1 crypto rx=0x7f3f6401b6a0 tx=0x7f3f64006c60 comp rx=0 tx=0).stop 2024-09-16T15:47:36.141 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:36.136+0000 7f3f701cb640 1 -- 172.21.15.123:0/1899777754 shutdown_connections 2024-09-16T15:47:36.141 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:36.136+0000 7f3f701cb640 1 --2- 172.21.15.123:0/1899777754 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f3f4c03ba50 0x7f3f4c03df10 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:36.141 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:36.136+0000 7f3f701cb640 1 --2- 172.21.15.123:0/1899777754 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3f68160b10 0x7f3f68160f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:36.141 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:36.136+0000 7f3f701cb640 1 --2- 172.21.15.123:0/1899777754 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f3f680d0260 0x7f3f68157d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:36.141 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:36.136+0000 7f3f701cb640 1 --2- 172.21.15.123:0/1899777754 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3f54003a10 0x7f3f680cf8c0 unknown :-1 s=CLOSED pgs=868 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:36.142 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:36.136+0000 7f3f701cb640 1 -- 172.21.15.123:0/1899777754 >> 172.21.15.123:0/1899777754 conn(0x7f3f680b2610 msgr2=0x7f3f680b5960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:47:36.142 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:36.136+0000 7f3f701cb640 1 -- 172.21.15.123:0/1899777754 shutdown_connections 2024-09-16T15:47:36.142 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:36.136+0000 7f3f701cb640 1 -- 172.21.15.123:0/1899777754 wait complete. 2024-09-16T15:47:36.157 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-16T15:47:36.266 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:36.260+0000 7f23248ee640 1 -- 172.21.15.123:0/775940191 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f230c004090 msgr2=0x7f230c004810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:47:36.267 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:36.260+0000 7f23248ee640 1 --2- 172.21.15.123:0/775940191 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f230c004090 0x7f230c004810 secure :-1 s=READY pgs=872 cs=0 l=1 rev1=1 crypto rx=0x7f2310006fd0 tx=0x7f231001cfb0 comp rx=0 tx=0).stop 2024-09-16T15:47:36.267 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:36.260+0000 7f231ed9d640 1 -- 172.21.15.123:0/775940191 reap_dead start 2024-09-16T15:47:36.267 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:36.260+0000 7f23248ee640 1 -- 172.21.15.123:0/775940191 shutdown_connections 2024-09-16T15:47:36.267 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:36.260+0000 7f23248ee640 1 -- 172.21.15.123:0/775940191 >> 172.21.15.123:0/775940191 conn(0x7f2320074bd0 msgr2=0x7f232009aa40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:47:36.267 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:36.260+0000 7f23248ee640 1 -- 172.21.15.123:0/775940191 shutdown_connections 2024-09-16T15:47:36.267 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:36.260+0000 7f23248ee640 1 -- 172.21.15.123:0/775940191 wait complete. 2024-09-16T15:47:36.267 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:36.264+0000 7f23248ee640 1 Processor -- start 2024-09-16T15:47:36.268 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:36.264+0000 7f23248ee640 1 -- start start 2024-09-16T15:47:36.270 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:36.264+0000 7f23248ee640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f230c004090 0x7f23200cb700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:47:36.270 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:36.264+0000 7f23248ee640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f23200ca590 0x7f23200c8460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:47:36.270 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:36.264+0000 7f23248ee640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2320159c90 0x7f232015a110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:47:36.270 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:36.264+0000 7f23248ee640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f23200cdcc0 con 0x7f230c004090 2024-09-16T15:47:36.270 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:36.264+0000 7f23248ee640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f23200cdb40 con 0x7f2320159c90 2024-09-16T15:47:36.270 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:36.264+0000 7f23248ee640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f2320166860 con 0x7f23200ca590 2024-09-16T15:47:36.270 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:36.264+0000 7f231e59c640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f230c004090 0x7f23200cb700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:47:36.270 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:36.264+0000 7f231dd9b640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f23200ca590 0x7f23200c8460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:47:36.270 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:36.264+0000 7f231e59c640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f230c004090 0x7f23200cb700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:53006/0 (socket says 172.21.15.123:53006) 2024-09-16T15:47:36.270 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:36.264+0000 7f231e59c640 1 -- 172.21.15.123:0/3153745965 learned_addr learned my addr 172.21.15.123:0/3153745965 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:47:36.270 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:36.264+0000 7f231ed9d640 1 --2- 172.21.15.123:0/3153745965 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2320159c90 0x7f232015a110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:47:36.271 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:36.264+0000 7f231e59c640 1 -- 172.21.15.123:0/3153745965 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f23200ca590 msgr2=0x7f23200c8460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:47:36.271 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:36.264+0000 7f231e59c640 1 --2- 172.21.15.123:0/3153745965 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f23200ca590 0x7f23200c8460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:36.271 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:36.264+0000 7f231e59c640 1 -- 172.21.15.123:0/3153745965 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2320159c90 msgr2=0x7f232015a110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:47:36.271 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:36.264+0000 7f231e59c640 1 --2- 172.21.15.123:0/3153745965 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2320159c90 0x7f232015a110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:36.271 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:36.264+0000 7f231e59c640 1 -- 172.21.15.123:0/3153745965 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f23200c8f30 con 0x7f230c004090 2024-09-16T15:47:36.271 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:36.264+0000 7f231ed9d640 1 --2- 172.21.15.123:0/3153745965 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2320159c90 0x7f232015a110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T15:47:36.271 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:36.264+0000 7f231e59c640 1 --2- 172.21.15.123:0/3153745965 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f230c004090 0x7f23200cb700 secure :-1 s=READY pgs=873 cs=0 l=1 rev1=1 crypto rx=0x7f230800fa60 tx=0x7f230800ff20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:47:36.271 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:36.264+0000 7f2316ffd640 1 -- 172.21.15.123:0/3153745965 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f230800bd10 con 0x7f230c004090 2024-09-16T15:47:36.271 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:36.264+0000 7f2316ffd640 1 -- 172.21.15.123:0/3153745965 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f2308004030 con 0x7f230c004090 2024-09-16T15:47:36.271 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:36.264+0000 7f2316ffd640 1 -- 172.21.15.123:0/3153745965 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2308004460 con 0x7f230c004090 2024-09-16T15:47:36.271 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:36.264+0000 7f23248ee640 1 -- 172.21.15.123:0/3153745965 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f23200c9210 con 0x7f230c004090 2024-09-16T15:47:36.271 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:36.264+0000 7f23248ee640 1 -- 172.21.15.123:0/3153745965 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f23200c97d0 con 0x7f230c004090 2024-09-16T15:47:36.272 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:36.264+0000 7f23248ee640 1 -- 172.21.15.123:0/3153745965 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f23200d0310 con 0x7f230c004090 2024-09-16T15:47:36.272 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:36.268+0000 7f2316ffd640 1 -- 172.21.15.123:0/3153745965 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 14) ==== 48913+0+0 (secure 0 0 0) 0x7f2308003600 con 0x7f230c004090 2024-09-16T15:47:36.273 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:36.268+0000 7f2316ffd640 1 --2- 172.21.15.123:0/3153745965 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f230003ba80 0x7f230003df40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:47:36.273 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:36.268+0000 7f2316ffd640 1 -- 172.21.15.123:0/3153745965 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(73..73 src has 1..73) ==== 6326+0+0 (secure 0 0 0) 0x7f2308050f90 con 0x7f230c004090 2024-09-16T15:47:36.273 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:36.268+0000 7f231dd9b640 1 --2- 172.21.15.123:0/3153745965 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f230003ba80 0x7f230003df40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:47:36.273 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:36.268+0000 7f231dd9b640 1 --2- 172.21.15.123:0/3153745965 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f230003ba80 0x7f230003df40 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f23100ac000 tx=0x7f2310019da0 comp rx=0 tx=0).ready entity=mgr.4150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:47:36.275 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:36.272+0000 7f2316ffd640 1 -- 172.21.15.123:0/3153745965 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f23080038f0 con 0x7f230c004090 2024-09-16T15:47:36.468 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:36.464+0000 7f23248ee640 1 -- 172.21.15.123:0/3153745965 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f23200cebc0 con 0x7f230c004090 2024-09-16T15:47:36.469 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:36.464+0000 7f2316ffd640 1 -- 172.21.15.123:0/3153745965 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 45 v45) ==== 94+0+5234 (secure 0 0 0) 0x7f2308011440 con 0x7f230c004090 2024-09-16T15:47:36.469 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-09-16T15:47:36.469 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":45,"btime":"2024-09-16T15:47:16:409841+0000","default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5048,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6837/1498917352","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6835","nonce":1498917352},{"type":"v1","addr":"172.21.15.184:6837","nonce":1498917352}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":40},{"gid":6198,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6833/1715139909","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6832","nonce":1715139909},{"type":"v1","addr":"172.21.15.123:6833","nonce":1715139909}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":40},{"gid":6201,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6835/815923338","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6834","nonce":815923338},{"type":"v1","addr":"172.21.15.123:6835","nonce":815923338}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":40}],"filesystems":[{"mdsmap":{"epoch":44,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-16T15:46:58.916477+0000","modified":"2024-09-16T15:47:01.963874+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6216},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6216":{"gid":6216,"name":"d","rank":0,"incarnation":41,"state":"up:active","state_seq":3,"addr":"172.21.15.184:6836/3137795986","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":3137795986},{"type":"v1","addr":"172.21.15.184:6836","nonce":3137795986}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6216,"qdb_cluster":[6216]},"id":5}]} 2024-09-16T15:47:36.469 INFO:teuthology.orchestra.run.smithi123.stderr:dumped fsmap epoch 45 2024-09-16T15:47:36.471 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:36.468+0000 7f23248ee640 1 -- 172.21.15.123:0/3153745965 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f230003ba80 msgr2=0x7f230003df40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:47:36.471 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:36.468+0000 7f23248ee640 1 --2- 172.21.15.123:0/3153745965 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f230003ba80 0x7f230003df40 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f23100ac000 tx=0x7f2310019da0 comp rx=0 tx=0).stop 2024-09-16T15:47:36.471 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:36.468+0000 7f23248ee640 1 -- 172.21.15.123:0/3153745965 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f230c004090 msgr2=0x7f23200cb700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:47:36.471 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:36.468+0000 7f23248ee640 1 --2- 172.21.15.123:0/3153745965 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f230c004090 0x7f23200cb700 secure :-1 s=READY pgs=873 cs=0 l=1 rev1=1 crypto rx=0x7f230800fa60 tx=0x7f230800ff20 comp rx=0 tx=0).stop 2024-09-16T15:47:36.471 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:36.468+0000 7f23248ee640 1 -- 172.21.15.123:0/3153745965 shutdown_connections 2024-09-16T15:47:36.471 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:36.468+0000 7f23248ee640 1 --2- 172.21.15.123:0/3153745965 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f230003ba80 0x7f230003df40 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:36.472 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:36.468+0000 7f23248ee640 1 --2- 172.21.15.123:0/3153745965 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2320159c90 0x7f232015a110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:36.472 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:36.468+0000 7f23248ee640 1 --2- 172.21.15.123:0/3153745965 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f23200ca590 0x7f23200c8460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:36.472 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:36.468+0000 7f23248ee640 1 --2- 172.21.15.123:0/3153745965 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f230c004090 0x7f23200cb700 unknown :-1 s=CLOSED pgs=873 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:36.472 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:36.468+0000 7f23248ee640 1 -- 172.21.15.123:0/3153745965 >> 172.21.15.123:0/3153745965 conn(0x7f2320074bd0 msgr2=0x7f232015af50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:47:36.472 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:36.468+0000 7f23248ee640 1 -- 172.21.15.123:0/3153745965 shutdown_connections 2024-09-16T15:47:36.472 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:36.468+0000 7f23248ee640 1 -- 172.21.15.123:0/3153745965 wait complete. 2024-09-16T15:47:36.486 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-16T15:47:36.608 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:36.604+0000 7f9bf86eb640 1 -- 172.21.15.123:0/1110668644 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9bf006f5f0 msgr2=0x7f9bf015aa60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:47:36.608 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:36.604+0000 7f9bf86eb640 1 --2- 172.21.15.123:0/1110668644 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9bf006f5f0 0x7f9bf015aa60 secure :-1 s=READY pgs=874 cs=0 l=1 rev1=1 crypto rx=0x7f9be8009c80 tx=0x7f9be801cd10 comp rx=0 tx=0).stop 2024-09-16T15:47:36.608 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:36.604+0000 7f9bf86eb640 1 -- 172.21.15.123:0/1110668644 shutdown_connections 2024-09-16T15:47:36.608 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:36.604+0000 7f9bf86eb640 1 --2- 172.21.15.123:0/1110668644 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9bf006f5f0 0x7f9bf015aa60 unknown :-1 s=CLOSED pgs=874 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:36.609 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:36.604+0000 7f9bf86eb640 1 --2- 172.21.15.123:0/1110668644 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f9bf0158120 0x7f9bf00ca320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:36.609 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:36.604+0000 7f9bf86eb640 1 -- 172.21.15.123:0/1110668644 >> 172.21.15.123:0/1110668644 conn(0x7f9bf00cfb10 msgr2=0x7f9bf00cff20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:47:36.609 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:36.604+0000 7f9bf86eb640 1 -- 172.21.15.123:0/1110668644 shutdown_connections 2024-09-16T15:47:36.609 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:36.604+0000 7f9bf86eb640 1 -- 172.21.15.123:0/1110668644 wait complete. 2024-09-16T15:47:36.609 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:36.604+0000 7f9bf86eb640 1 Processor -- start 2024-09-16T15:47:36.609 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:36.604+0000 7f9bf86eb640 1 -- start start 2024-09-16T15:47:36.611 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:36.604+0000 7f9bf86eb640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f9bf006f5f0 0x7f9bf00aefc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:47:36.611 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:36.604+0000 7f9bf86eb640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9bf00cf5a0 0x7f9bf01f89b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:47:36.611 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:36.604+0000 7f9bf86eb640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f9bf0158120 0x7f9bf01663d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:47:36.611 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:36.604+0000 7f9bf86eb640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f9bf015b840 con 0x7f9bf00cf5a0 2024-09-16T15:47:36.611 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:36.604+0000 7f9bf86eb640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f9bf015b6c0 con 0x7f9bf006f5f0 2024-09-16T15:47:36.611 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:36.604+0000 7f9bf86eb640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f9bf01653c0 con 0x7f9bf0158120 2024-09-16T15:47:36.612 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:36.604+0000 7f9bf5c86640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9bf00cf5a0 0x7f9bf01f89b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:47:36.612 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:36.604+0000 7f9bf5c86640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9bf00cf5a0 0x7f9bf01f89b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:53030/0 (socket says 172.21.15.123:53030) 2024-09-16T15:47:36.612 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:36.604+0000 7f9bf5c86640 1 -- 172.21.15.123:0/2818150104 learned_addr learned my addr 172.21.15.123:0/2818150104 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:47:36.612 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:36.608+0000 7f9bf6487640 1 --2- 172.21.15.123:0/2818150104 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f9bf006f5f0 0x7f9bf00aefc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:47:36.612 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:36.608+0000 7f9bf6c88640 1 --2- 172.21.15.123:0/2818150104 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f9bf0158120 0x7f9bf01663d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:47:36.612 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:36.608+0000 7f9bf5c86640 1 -- 172.21.15.123:0/2818150104 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f9bf0158120 msgr2=0x7f9bf01663d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:47:36.612 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:36.608+0000 7f9bf5c86640 1 --2- 172.21.15.123:0/2818150104 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f9bf0158120 0x7f9bf01663d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:36.612 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:36.608+0000 7f9bf5c86640 1 -- 172.21.15.123:0/2818150104 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f9bf006f5f0 msgr2=0x7f9bf00aefc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:47:36.612 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:36.608+0000 7f9bf5c86640 1 --2- 172.21.15.123:0/2818150104 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f9bf006f5f0 0x7f9bf00aefc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:36.612 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:36.608+0000 7f9bf5c86640 1 -- 172.21.15.123:0/2818150104 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9bf01594c0 con 0x7f9bf00cf5a0 2024-09-16T15:47:36.613 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:36.608+0000 7f9bf5c86640 1 --2- 172.21.15.123:0/2818150104 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9bf00cf5a0 0x7f9bf01f89b0 secure :-1 s=READY pgs=875 cs=0 l=1 rev1=1 crypto rx=0x7f9be000c400 tx=0x7f9be000c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:47:36.613 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:36.608+0000 7f9be6ffd640 1 -- 172.21.15.123:0/2818150104 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9be0017070 con 0x7f9bf00cf5a0 2024-09-16T15:47:36.613 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:36.608+0000 7f9be6ffd640 1 -- 172.21.15.123:0/2818150104 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f9be00053e0 con 0x7f9bf00cf5a0 2024-09-16T15:47:36.613 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:36.608+0000 7f9bf86eb640 1 -- 172.21.15.123:0/2818150104 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9bf015d4c0 con 0x7f9bf00cf5a0 2024-09-16T15:47:36.613 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:36.608+0000 7f9be6ffd640 1 -- 172.21.15.123:0/2818150104 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9be0003860 con 0x7f9bf00cf5a0 2024-09-16T15:47:36.613 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:36.608+0000 7f9bf86eb640 1 -- 172.21.15.123:0/2818150104 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9bf01589e0 con 0x7f9bf00cf5a0 2024-09-16T15:47:36.614 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:36.608+0000 7f9be6ffd640 1 -- 172.21.15.123:0/2818150104 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 14) ==== 48913+0+0 (secure 0 0 0) 0x7f9be001a070 con 0x7f9bf00cf5a0 2024-09-16T15:47:36.614 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:36.608+0000 7f9bf86eb640 1 -- 172.21.15.123:0/2818150104 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9bf015aa60 con 0x7f9bf00cf5a0 2024-09-16T15:47:36.614 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:36.608+0000 7f9be6ffd640 1 --2- 172.21.15.123:0/2818150104 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f9bd403ba50 0x7f9bd403df10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:47:36.614 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:36.608+0000 7f9be6ffd640 1 -- 172.21.15.123:0/2818150104 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(73..73 src has 1..73) ==== 6326+0+0 (secure 0 0 0) 0x7f9be004f980 con 0x7f9bf00cf5a0 2024-09-16T15:47:36.614 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:36.608+0000 7f9bf6487640 1 --2- 172.21.15.123:0/2818150104 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f9bd403ba50 0x7f9bd403df10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:47:36.614 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:36.608+0000 7f9bf6487640 1 --2- 172.21.15.123:0/2818150104 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f9bd403ba50 0x7f9bd403df10 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f9bec0058d0 tx=0x7f9bec005860 comp rx=0 tx=0).ready entity=mgr.4150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:47:36.616 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:36.612+0000 7f9be6ffd640 1 -- 172.21.15.123:0/2818150104 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f9be0011720 con 0x7f9bf00cf5a0 2024-09-16T15:47:36.789 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:36.784+0000 7f9bf86eb640 1 -- 172.21.15.123:0/2818150104 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f9bf0070980 con 0x7f9bf00cf5a0 2024-09-16T15:47:36.790 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:36.788+0000 7f9be6ffd640 1 -- 172.21.15.123:0/2818150104 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v73) ==== 74+0+17705 (secure 0 0 0) 0x7f9be003bbd0 con 0x7f9bf00cf5a0 2024-09-16T15:47:36.791 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-09-16T15:47:36.791 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":73,"fsid":"56077f89-18c8-486f-ae7e-eecc0bb7822c","created":"2024-09-16T15:42:26.364892+0000","modified":"2024-09-16T15:47:34.024919+0000","last_up_change":"2024-09-16T15:42:38.707234+0000","last_in_change":"2024-09-16T15:42:29.423662+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-16T15:42:39.287764+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_metadata","create_time":"2024-09-16T15:46:56.639820+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"69","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_data","create_time":"2024-09-16T15:46:57.893968+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"70","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9e7c1955-a607-4671-9be7-f2ef760ceeaa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":71,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6800","nonce":3882044969},{"type":"v1","addr":"172.21.15.123:6801","nonce":3882044969}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6802","nonce":3882044969},{"type":"v1","addr":"172.21.15.123:6803","nonce":3882044969}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6806","nonce":3882044969},{"type":"v1","addr":"172.21.15.123:6807","nonce":3882044969}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6804","nonce":3882044969},{"type":"v1","addr":"172.21.15.123:6805","nonce":3882044969}]},"public_addr":"172.21.15.123:6801/3882044969","cluster_addr":"172.21.15.123:6803/3882044969","heartbeat_back_addr":"172.21.15.123:6807/3882044969","heartbeat_front_addr":"172.21.15.123:6805/3882044969","state":["exists","up"]},{"osd":1,"uuid":"c540cfde-0977-4742-9b17-fd634668e010","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6816","nonce":4037001850},{"type":"v1","addr":"172.21.15.123:6817","nonce":4037001850}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6818","nonce":4037001850},{"type":"v1","addr":"172.21.15.123:6819","nonce":4037001850}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6822","nonce":4037001850},{"type":"v1","addr":"172.21.15.123:6823","nonce":4037001850}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6820","nonce":4037001850},{"type":"v1","addr":"172.21.15.123:6821","nonce":4037001850}]},"public_addr":"172.21.15.123:6817/4037001850","cluster_addr":"172.21.15.123:6819/4037001850","heartbeat_back_addr":"172.21.15.123:6823/4037001850","heartbeat_front_addr":"172.21.15.123:6821/4037001850","state":["exists","up"]},{"osd":2,"uuid":"9864a496-f169-486b-95c4-1d59162793a7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6824","nonce":1332143088},{"type":"v1","addr":"172.21.15.123:6825","nonce":1332143088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6826","nonce":1332143088},{"type":"v1","addr":"172.21.15.123:6827","nonce":1332143088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6830","nonce":1332143088},{"type":"v1","addr":"172.21.15.123:6831","nonce":1332143088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6828","nonce":1332143088},{"type":"v1","addr":"172.21.15.123:6829","nonce":1332143088}]},"public_addr":"172.21.15.123:6825/1332143088","cluster_addr":"172.21.15.123:6827/1332143088","heartbeat_back_addr":"172.21.15.123:6831/1332143088","heartbeat_front_addr":"172.21.15.123:6829/1332143088","state":["exists","up"]},{"osd":3,"uuid":"c6d1f817-f074-4325-95e1-578eba13edda","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6808","nonce":988586998},{"type":"v1","addr":"172.21.15.123:6809","nonce":988586998}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6810","nonce":988586998},{"type":"v1","addr":"172.21.15.123:6811","nonce":988586998}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6814","nonce":988586998},{"type":"v1","addr":"172.21.15.123:6815","nonce":988586998}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6812","nonce":988586998},{"type":"v1","addr":"172.21.15.123:6813","nonce":988586998}]},"public_addr":"172.21.15.123:6809/988586998","cluster_addr":"172.21.15.123:6811/988586998","heartbeat_back_addr":"172.21.15.123:6815/988586998","heartbeat_front_addr":"172.21.15.123:6813/988586998","state":["exists","up"]},{"osd":4,"uuid":"200a8df4-50ef-40a5-b48a-58981269c086","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6816","nonce":3387813890},{"type":"v1","addr":"172.21.15.184:6817","nonce":3387813890}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6818","nonce":3387813890},{"type":"v1","addr":"172.21.15.184:6819","nonce":3387813890}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6822","nonce":3387813890},{"type":"v1","addr":"172.21.15.184:6823","nonce":3387813890}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6820","nonce":3387813890},{"type":"v1","addr":"172.21.15.184:6821","nonce":3387813890}]},"public_addr":"172.21.15.184:6817/3387813890","cluster_addr":"172.21.15.184:6819/3387813890","heartbeat_back_addr":"172.21.15.184:6823/3387813890","heartbeat_front_addr":"172.21.15.184:6821/3387813890","state":["exists","up"]},{"osd":5,"uuid":"32d5a900-da3d-412c-bc19-13861a2368d8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6808","nonce":2680916017},{"type":"v1","addr":"172.21.15.184:6809","nonce":2680916017}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6810","nonce":2680916017},{"type":"v1","addr":"172.21.15.184:6811","nonce":2680916017}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6814","nonce":2680916017},{"type":"v1","addr":"172.21.15.184:6815","nonce":2680916017}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6812","nonce":2680916017},{"type":"v1","addr":"172.21.15.184:6813","nonce":2680916017}]},"public_addr":"172.21.15.184:6809/2680916017","cluster_addr":"172.21.15.184:6811/2680916017","heartbeat_back_addr":"172.21.15.184:6815/2680916017","heartbeat_front_addr":"172.21.15.184:6813/2680916017","state":["exists","up"]},{"osd":6,"uuid":"cde6f39e-aa1b-4260-ac19-774ee6aaca72","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6824","nonce":2183120457},{"type":"v1","addr":"172.21.15.184:6825","nonce":2183120457}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6826","nonce":2183120457},{"type":"v1","addr":"172.21.15.184:6827","nonce":2183120457}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6830","nonce":2183120457},{"type":"v1","addr":"172.21.15.184:6831","nonce":2183120457}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6828","nonce":2183120457},{"type":"v1","addr":"172.21.15.184:6829","nonce":2183120457}]},"public_addr":"172.21.15.184:6825/2183120457","cluster_addr":"172.21.15.184:6827/2183120457","heartbeat_back_addr":"172.21.15.184:6831/2183120457","heartbeat_front_addr":"172.21.15.184:6829/2183120457","state":["exists","up"]},{"osd":7,"uuid":"4f381778-eee1-47c6-804f-df0e6b33e313","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":71,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6800","nonce":4102957285},{"type":"v1","addr":"172.21.15.184:6801","nonce":4102957285}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6802","nonce":4102957285},{"type":"v1","addr":"172.21.15.184:6803","nonce":4102957285}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6806","nonce":4102957285},{"type":"v1","addr":"172.21.15.184:6807","nonce":4102957285}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6804","nonce":4102957285},{"type":"v1","addr":"172.21.15.184:6805","nonce":4102957285}]},"public_addr":"172.21.15.184:6801/4102957285","cluster_addr":"172.21.15.184:6803/4102957285","heartbeat_back_addr":"172.21.15.184:6807/4102957285","heartbeat_front_addr":"172.21.15.184:6805/4102957285","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:35.972210+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.222007+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.198617+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.385663+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.358935+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.156066+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.752973+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:35.682846+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"11.3","mappings":[{"from":6,"to":0}]},{"pgid":"11.28","mappings":[{"from":6,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.184:0/1023177165":"2024-09-17T15:47:34.024879+0000","172.21.15.184:6833/3114346423":"2024-09-17T15:47:34.024879+0000","172.21.15.184:6832/3114346423":"2024-09-17T15:47:34.024879+0000","172.21.15.184:0/2406729396":"2024-09-17T15:47:34.024879+0000","172.21.15.184:0/78639736":"2024-09-17T15:47:34.024879+0000","172.21.15.184:0/3204392673":"2024-09-17T15:47:34.024879+0000","172.21.15.184:0/654280269":"2024-09-17T15:47:34.024879+0000","172.21.15.184:0/2291367472":"2024-09-17T15:47:34.024879+0000","172.21.15.184:0/1715342118":"2024-09-17T15:47:34.024879+0000","172.21.15.184:0/140936988":"2024-09-17T15:47:34.024879+0000","172.21.15.184:0/1402401038":"2024-09-17T15:47:34.024879+0000","172.21.15.184:0/3070886766":"2024-09-17T15:47:34.024879+0000","172.21.15.184:0/2916845314":"2024-09-17T15:47:34.024879+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T15:47:36.793 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:36.788+0000 7f9bf86eb640 1 -- 172.21.15.123:0/2818150104 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f9bd403ba50 msgr2=0x7f9bd403df10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:47:36.793 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:36.788+0000 7f9bf86eb640 1 --2- 172.21.15.123:0/2818150104 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f9bd403ba50 0x7f9bd403df10 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f9bec0058d0 tx=0x7f9bec005860 comp rx=0 tx=0).stop 2024-09-16T15:47:36.793 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:36.788+0000 7f9bf86eb640 1 -- 172.21.15.123:0/2818150104 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9bf00cf5a0 msgr2=0x7f9bf01f89b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:47:36.793 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:36.788+0000 7f9bf86eb640 1 --2- 172.21.15.123:0/2818150104 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9bf00cf5a0 0x7f9bf01f89b0 secure :-1 s=READY pgs=875 cs=0 l=1 rev1=1 crypto rx=0x7f9be000c400 tx=0x7f9be000c8c0 comp rx=0 tx=0).stop 2024-09-16T15:47:36.794 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:36.788+0000 7f9bf86eb640 1 -- 172.21.15.123:0/2818150104 shutdown_connections 2024-09-16T15:47:36.794 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:36.788+0000 7f9bf86eb640 1 --2- 172.21.15.123:0/2818150104 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f9bd403ba50 0x7f9bd403df10 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:36.794 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:36.788+0000 7f9bf86eb640 1 --2- 172.21.15.123:0/2818150104 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f9bf0158120 0x7f9bf01663d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:36.794 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:36.788+0000 7f9bf86eb640 1 --2- 172.21.15.123:0/2818150104 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9bf00cf5a0 0x7f9bf01f89b0 unknown :-1 s=CLOSED pgs=875 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:36.794 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:36.788+0000 7f9bf86eb640 1 --2- 172.21.15.123:0/2818150104 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f9bf006f5f0 0x7f9bf00aefc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:36.794 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:36.788+0000 7f9bf86eb640 1 -- 172.21.15.123:0/2818150104 >> 172.21.15.123:0/2818150104 conn(0x7f9bf00cfb10 msgr2=0x7f9bf00aff90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:47:36.794 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:36.788+0000 7f9bf86eb640 1 -- 172.21.15.123:0/2818150104 shutdown_connections 2024-09-16T15:47:36.794 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:36.788+0000 7f9bf86eb640 1 -- 172.21.15.123:0/2818150104 wait complete. 2024-09-16T15:47:36.810 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-09-16T15:47:36.939 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:36.936+0000 7f6aa33d7640 1 -- 172.21.15.123:0/1658240832 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6a9c0d05b0 msgr2=0x7f6a9c0d0990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:47:36.939 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:36.936+0000 7f6aa33d7640 1 --2- 172.21.15.123:0/1658240832 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6a9c0d05b0 0x7f6a9c0d0990 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f6a98009900 tx=0x7f6a9801c850 comp rx=0 tx=0).stop 2024-09-16T15:47:36.939 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:36.936+0000 7f6aa33d7640 1 -- 172.21.15.123:0/1658240832 shutdown_connections 2024-09-16T15:47:36.939 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:36.936+0000 7f6aa33d7640 1 --2- 172.21.15.123:0/1658240832 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f6a9c0af690 0x7f6a9c0cdf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:36.939 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:36.936+0000 7f6aa33d7640 1 --2- 172.21.15.123:0/1658240832 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6a9c0d05b0 0x7f6a9c0d0990 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:36.939 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:36.936+0000 7f6aa33d7640 1 -- 172.21.15.123:0/1658240832 >> 172.21.15.123:0/1658240832 conn(0x7f6a9c0732a0 msgr2=0x7f6a9c074150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:47:36.939 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:36.936+0000 7f6aa33d7640 1 -- 172.21.15.123:0/1658240832 shutdown_connections 2024-09-16T15:47:36.939 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:36.936+0000 7f6aa33d7640 1 -- 172.21.15.123:0/1658240832 wait complete. 2024-09-16T15:47:36.940 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:36.936+0000 7f6aa33d7640 1 Processor -- start 2024-09-16T15:47:36.940 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:36.936+0000 7f6aa33d7640 1 -- start start 2024-09-16T15:47:36.942 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:36.936+0000 7f6aa33d7640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6a9c0af690 0x7f6a9c0b37a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:47:36.942 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:36.936+0000 7f6aa33d7640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6a9c0ce4c0 0x7f6a9c1fa470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:47:36.942 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:36.936+0000 7f6aa33d7640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f6a9c0d05b0 0x7f6a9c0c4130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:47:36.942 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:36.936+0000 7f6aa33d7640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f6a9c0ad990 con 0x7f6a9c0af690 2024-09-16T15:47:36.942 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:36.936+0000 7f6aa33d7640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f6a9c0ad810 con 0x7f6a9c0ce4c0 2024-09-16T15:47:36.942 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:36.936+0000 7f6aa33d7640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f6a9c0adb10 con 0x7f6a9c0d05b0 2024-09-16T15:47:36.942 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:36.936+0000 7f6aa1173640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6a9c0af690 0x7f6a9c0b37a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:47:36.942 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:36.936+0000 7f6aa1173640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6a9c0af690 0x7f6a9c0b37a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:53032/0 (socket says 172.21.15.123:53032) 2024-09-16T15:47:36.943 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:36.936+0000 7f6aa1173640 1 -- 172.21.15.123:0/1254703285 learned_addr learned my addr 172.21.15.123:0/1254703285 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:47:36.943 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:36.936+0000 7f6aa1974640 1 --2- 172.21.15.123:0/1254703285 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f6a9c0d05b0 0x7f6a9c0c4130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:47:36.943 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:36.936+0000 7f6aa0972640 1 --2- 172.21.15.123:0/1254703285 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6a9c0ce4c0 0x7f6a9c1fa470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:47:36.943 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:36.936+0000 7f6aa1173640 1 -- 172.21.15.123:0/1254703285 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f6a9c0d05b0 msgr2=0x7f6a9c0c4130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:47:36.943 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:36.936+0000 7f6aa1173640 1 --2- 172.21.15.123:0/1254703285 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f6a9c0d05b0 0x7f6a9c0c4130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:36.943 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:36.936+0000 7f6aa1173640 1 -- 172.21.15.123:0/1254703285 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6a9c0ce4c0 msgr2=0x7f6a9c1fa470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:47:36.943 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:36.936+0000 7f6aa1173640 1 --2- 172.21.15.123:0/1254703285 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6a9c0ce4c0 0x7f6a9c1fa470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:36.943 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:36.936+0000 7f6aa1173640 1 -- 172.21.15.123:0/1254703285 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6a9c0b22d0 con 0x7f6a9c0af690 2024-09-16T15:47:36.943 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:36.936+0000 7f6aa1974640 1 --2- 172.21.15.123:0/1254703285 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f6a9c0d05b0 0x7f6a9c0c4130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T15:47:36.943 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:36.936+0000 7f6aa1173640 1 --2- 172.21.15.123:0/1254703285 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6a9c0af690 0x7f6a9c0b37a0 secure :-1 s=READY pgs=876 cs=0 l=1 rev1=1 crypto rx=0x7f6a980b0040 tx=0x7f6a98005760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:47:36.943 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:36.936+0000 7f6a91ffb640 1 -- 172.21.15.123:0/1254703285 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a980b2040 con 0x7f6a9c0af690 2024-09-16T15:47:36.943 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:36.936+0000 7f6aa33d7640 1 -- 172.21.15.123:0/1254703285 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6a9c0c6270 con 0x7f6a9c0af690 2024-09-16T15:47:36.944 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:36.936+0000 7f6a91ffb640 1 -- 172.21.15.123:0/1254703285 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f6a98004040 con 0x7f6a9c0af690 2024-09-16T15:47:36.944 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:36.936+0000 7f6a91ffb640 1 -- 172.21.15.123:0/1254703285 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a98004410 con 0x7f6a9c0af690 2024-09-16T15:47:36.944 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:36.936+0000 7f6aa33d7640 1 -- 172.21.15.123:0/1254703285 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6a9c0c6830 con 0x7f6a9c0af690 2024-09-16T15:47:36.944 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:36.940+0000 7f6aa33d7640 1 -- 172.21.15.123:0/1254703285 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6a9c0b6330 con 0x7f6a9c0af690 2024-09-16T15:47:36.945 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:36.940+0000 7f6a91ffb640 1 -- 172.21.15.123:0/1254703285 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 14) ==== 48913+0+0 (secure 0 0 0) 0x7f6a980bf020 con 0x7f6a9c0af690 2024-09-16T15:47:36.945 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:36.940+0000 7f6a91ffb640 1 --2- 172.21.15.123:0/1254703285 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f6a8003ba50 0x7f6a8003df10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:47:36.945 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:36.940+0000 7f6a91ffb640 1 -- 172.21.15.123:0/1254703285 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(73..73 src has 1..73) ==== 6326+0+0 (secure 0 0 0) 0x7f6a980ea4f0 con 0x7f6a9c0af690 2024-09-16T15:47:36.945 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:36.940+0000 7f6aa0972640 1 --2- 172.21.15.123:0/1254703285 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f6a8003ba50 0x7f6a8003df10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:47:36.945 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:36.940+0000 7f6aa0972640 1 --2- 172.21.15.123:0/1254703285 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f6a8003ba50 0x7f6a8003df10 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f6a9c06a200 tx=0x7f6a8c0091e0 comp rx=0 tx=0).ready entity=mgr.4150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:47:36.947 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:36.944+0000 7f6a91ffb640 1 -- 172.21.15.123:0/1254703285 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f6a980c6020 con 0x7f6a9c0af690 2024-09-16T15:47:37.057 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:37.052+0000 7f6a91ffb640 1 -- 172.21.15.123:0/1254703285 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mgrmap(e 15) ==== 48969+0+0 (secure 0 0 0) 0x7f6a980ad440 con 0x7f6a9c0af690 2024-09-16T15:47:37.154 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:37.148+0000 7f6aa33d7640 1 -- 172.21.15.123:0/1254703285 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7f6a9c0c6040 con 0x7f6a9c0af690 2024-09-16T15:47:37.155 INFO:tasks.ceph.mon.a.smithi123.stderr:2024-09-16T15:47:37.152+0000 7f268d076640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-09-16T15:47:37.157 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:37.152+0000 7f6a91ffb640 1 -- 172.21.15.123:0/1254703285 <== mon.0 v2:172.21.15.123:3300/0 8 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v46) ==== 161+0+0 (secure 0 0 0) 0x7f6a980164d0 con 0x7f6a9c0af690 2024-09-16T15:47:37.158 INFO:teuthology.orchestra.run.smithi123.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-09-16T15:47:37.161 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:37.156+0000 7f6aa33d7640 1 -- 172.21.15.123:0/1254703285 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f6a8003ba50 msgr2=0x7f6a8003df10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:47:37.161 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:37.156+0000 7f6aa33d7640 1 --2- 172.21.15.123:0/1254703285 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f6a8003ba50 0x7f6a8003df10 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f6a9c06a200 tx=0x7f6a8c0091e0 comp rx=0 tx=0).stop 2024-09-16T15:47:37.161 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:37.156+0000 7f6aa33d7640 1 -- 172.21.15.123:0/1254703285 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6a9c0af690 msgr2=0x7f6a9c0b37a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:47:37.161 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:37.156+0000 7f6aa33d7640 1 --2- 172.21.15.123:0/1254703285 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6a9c0af690 0x7f6a9c0b37a0 secure :-1 s=READY pgs=876 cs=0 l=1 rev1=1 crypto rx=0x7f6a980b0040 tx=0x7f6a98005760 comp rx=0 tx=0).stop 2024-09-16T15:47:37.161 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:37.156+0000 7f6aa33d7640 1 -- 172.21.15.123:0/1254703285 shutdown_connections 2024-09-16T15:47:37.161 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:37.156+0000 7f6aa33d7640 1 --2- 172.21.15.123:0/1254703285 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f6a8003ba50 0x7f6a8003df10 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:37.161 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:37.156+0000 7f6aa33d7640 1 --2- 172.21.15.123:0/1254703285 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f6a9c0d05b0 0x7f6a9c0c4130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:37.161 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:37.156+0000 7f6aa33d7640 1 --2- 172.21.15.123:0/1254703285 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6a9c0ce4c0 0x7f6a9c1fa470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:37.162 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:37.156+0000 7f6aa33d7640 1 --2- 172.21.15.123:0/1254703285 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6a9c0af690 0x7f6a9c0b37a0 unknown :-1 s=CLOSED pgs=876 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:37.162 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:37.156+0000 7f6aa33d7640 1 -- 172.21.15.123:0/1254703285 >> 172.21.15.123:0/1254703285 conn(0x7f6a9c0732a0 msgr2=0x7f6a9c0b4380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:47:37.162 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:37.156+0000 7f6aa33d7640 1 -- 172.21.15.123:0/1254703285 shutdown_connections 2024-09-16T15:47:37.162 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:37.156+0000 7f6aa33d7640 1 -- 172.21.15.123:0/1254703285 wait complete. 2024-09-16T15:47:37.176 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-09-16T15:47:37.177 INFO:tasks.ceph.mds.d.smithi184.stderr: -7750> 2024-09-16T15:47:04.525+0000 7fcc85957640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-09-16T15:47:37.247 INFO:tasks.ceph.mds.d.smithi184.stdout:starting mds.d at 2024-09-16T15:47:37.288 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:37.284+0000 7f80c9ed4640 1 -- 172.21.15.123:0/3699320434 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f80c4072720 msgr2=0x7f80c415e5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:47:37.288 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:37.284+0000 7f80c9ed4640 1 --2- 172.21.15.123:0/3699320434 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f80c4072720 0x7f80c415e5d0 secure :-1 s=READY pgs=880 cs=0 l=1 rev1=1 crypto rx=0x7f80b8009840 tx=0x7f80b801c820 comp rx=0 tx=0).stop 2024-09-16T15:47:37.288 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:37.284+0000 7f80c9ed4640 1 -- 172.21.15.123:0/3699320434 shutdown_connections 2024-09-16T15:47:37.288 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:37.284+0000 7f80c9ed4640 1 --2- 172.21.15.123:0/3699320434 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f80c40cc3c0 0x7f80c40cc850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:37.288 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:37.284+0000 7f80c9ed4640 1 --2- 172.21.15.123:0/3699320434 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f80c4072720 0x7f80c415e5d0 unknown :-1 s=CLOSED pgs=880 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:37.288 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:37.284+0000 7f80c9ed4640 1 -- 172.21.15.123:0/3699320434 >> 172.21.15.123:0/3699320434 conn(0x7f80c40b1710 msgr2=0x7f80c40b1b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:47:37.288 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:37.284+0000 7f80c9ed4640 1 -- 172.21.15.123:0/3699320434 shutdown_connections 2024-09-16T15:47:37.288 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:37.284+0000 7f80c9ed4640 1 -- 172.21.15.123:0/3699320434 wait complete. 2024-09-16T15:47:37.290 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:37.284+0000 7f80c9ed4640 1 Processor -- start 2024-09-16T15:47:37.290 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:37.284+0000 7f80c9ed4640 1 -- start start 2024-09-16T15:47:37.291 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:37.284+0000 7f80c9ed4640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f80c4072720 0x7f80c40c60e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:47:37.291 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:37.284+0000 7f80c9ed4640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f80c40cc3c0 0x7f80c40cfa20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:47:37.291 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:37.284+0000 7f80c9ed4640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f80c415aa00 0x7f80c40cff60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:47:37.291 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:37.284+0000 7f80c9ed4640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f80c4167af0 con 0x7f80c4072720 2024-09-16T15:47:37.291 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:37.284+0000 7f80c9ed4640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f80c4167970 con 0x7f80c415aa00 2024-09-16T15:47:37.291 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:37.284+0000 7f80c9ed4640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f80c4167c70 con 0x7f80c40cc3c0 2024-09-16T15:47:37.291 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:37.284+0000 7f80c37fe640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f80c4072720 0x7f80c40c60e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:47:37.291 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:37.284+0000 7f80c37fe640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f80c4072720 0x7f80c40c60e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:53056/0 (socket says 172.21.15.123:53056) 2024-09-16T15:47:37.291 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:37.284+0000 7f80c37fe640 1 -- 172.21.15.123:0/2461022480 learned_addr learned my addr 172.21.15.123:0/2461022480 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:47:37.291 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:37.284+0000 7f80c2ffd640 1 --2- 172.21.15.123:0/2461022480 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f80c40cc3c0 0x7f80c40cfa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:47:37.291 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:37.284+0000 7f80c3fff640 1 --2- 172.21.15.123:0/2461022480 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f80c415aa00 0x7f80c40cff60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:47:37.291 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:37.284+0000 7f80c37fe640 1 -- 172.21.15.123:0/2461022480 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f80c40cc3c0 msgr2=0x7f80c40cfa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:47:37.292 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:37.284+0000 7f80c37fe640 1 --2- 172.21.15.123:0/2461022480 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f80c40cc3c0 0x7f80c40cfa20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:37.292 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:37.284+0000 7f80c37fe640 1 -- 172.21.15.123:0/2461022480 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f80c415aa00 msgr2=0x7f80c40cff60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:47:37.292 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:37.284+0000 7f80c37fe640 1 --2- 172.21.15.123:0/2461022480 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f80c415aa00 0x7f80c40cff60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:37.292 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:37.284+0000 7f80c37fe640 1 -- 172.21.15.123:0/2461022480 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f80c4141ab0 con 0x7f80c4072720 2024-09-16T15:47:37.292 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:37.284+0000 7f80c3fff640 1 --2- 172.21.15.123:0/2461022480 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f80c415aa00 0x7f80c40cff60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T15:47:37.292 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:37.284+0000 7f80c37fe640 1 --2- 172.21.15.123:0/2461022480 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f80c4072720 0x7f80c40c60e0 secure :-1 s=READY pgs=881 cs=0 l=1 rev1=1 crypto rx=0x7f80b400c500 tx=0x7f80b400c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:47:37.292 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:37.284+0000 7f80abfff640 1 -- 172.21.15.123:0/2461022480 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80b4017070 con 0x7f80c4072720 2024-09-16T15:47:37.292 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:37.284+0000 7f80abfff640 1 -- 172.21.15.123:0/2461022480 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f80b4003600 con 0x7f80c4072720 2024-09-16T15:47:37.292 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:37.284+0000 7f80c9ed4640 1 -- 172.21.15.123:0/2461022480 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f80c40c6d40 con 0x7f80c4072720 2024-09-16T15:47:37.292 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:37.284+0000 7f80abfff640 1 -- 172.21.15.123:0/2461022480 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80b4003a90 con 0x7f80c4072720 2024-09-16T15:47:37.292 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:37.284+0000 7f80c9ed4640 1 -- 172.21.15.123:0/2461022480 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f80c41417d0 con 0x7f80c4072720 2024-09-16T15:47:37.293 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:37.288+0000 7f80c9ed4640 1 -- 172.21.15.123:0/2461022480 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f80c415e5d0 con 0x7f80c4072720 2024-09-16T15:47:37.293 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:37.288+0000 7f80abfff640 1 -- 172.21.15.123:0/2461022480 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 15) ==== 48969+0+0 (secure 0 0 0) 0x7f80b401a070 con 0x7f80c4072720 2024-09-16T15:47:37.293 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:37.288+0000 7f80abfff640 1 --2- 172.21.15.123:0/2461022480 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f80b003bae0 0x7f80b003dfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:47:37.293 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:37.288+0000 7f80abfff640 1 -- 172.21.15.123:0/2461022480 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(74..74 src has 1..74) ==== 6412+0+0 (secure 0 0 0) 0x7f80b404fb50 con 0x7f80c4072720 2024-09-16T15:47:37.293 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:37.288+0000 7f80c2ffd640 1 --2- 172.21.15.123:0/2461022480 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f80b003bae0 0x7f80b003dfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:47:37.294 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:37.288+0000 7f80c2ffd640 1 --2- 172.21.15.123:0/2461022480 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f80b003bae0 0x7f80b003dfa0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f80b80af000 tx=0x7f80b801b060 comp rx=0 tx=0).ready entity=mgr.4150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:47:37.296 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:37.292+0000 7f80abfff640 1 -- 172.21.15.123:0/2461022480 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f80b40192c0 con 0x7f80c4072720 2024-09-16T15:47:37.488 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:37.484+0000 7f80c9ed4640 1 -- 172.21.15.123:0/2461022480 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) -- 0x7f80c40cd440 con 0x7f80c4072720 2024-09-16T15:47:38.174 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:38.172+0000 7f80abfff640 1 -- 172.21.15.123:0/2461022480 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mgrmap(e 16) ==== 97182+0+0 (secure 0 0 0) 0x7f80b401b8d0 con 0x7f80c4072720 2024-09-16T15:47:38.178 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:38.172+0000 7f80abfff640 1 -- 172.21.15.123:0/2461022480 <== mon.0 v2:172.21.15.123:3300/0 8 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v47) ==== 104+0+0 (secure 0 0 0) 0x7f80b40669a0 con 0x7f80c4072720 2024-09-16T15:47:38.183 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:38.180+0000 7f80c9ed4640 1 -- 172.21.15.123:0/2461022480 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f80b003bae0 msgr2=0x7f80b003dfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:47:38.184 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:38.180+0000 7f80c9ed4640 1 --2- 172.21.15.123:0/2461022480 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f80b003bae0 0x7f80b003dfa0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f80b80af000 tx=0x7f80b801b060 comp rx=0 tx=0).stop 2024-09-16T15:47:38.184 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:38.180+0000 7f80c9ed4640 1 -- 172.21.15.123:0/2461022480 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f80c4072720 msgr2=0x7f80c40c60e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:47:38.184 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:38.180+0000 7f80c9ed4640 1 --2- 172.21.15.123:0/2461022480 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f80c4072720 0x7f80c40c60e0 secure :-1 s=READY pgs=881 cs=0 l=1 rev1=1 crypto rx=0x7f80b400c500 tx=0x7f80b400c9c0 comp rx=0 tx=0).stop 2024-09-16T15:47:38.184 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:38.180+0000 7f80c9ed4640 1 -- 172.21.15.123:0/2461022480 shutdown_connections 2024-09-16T15:47:38.184 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:38.180+0000 7f80c9ed4640 1 --2- 172.21.15.123:0/2461022480 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f80b003bae0 0x7f80b003dfa0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:38.184 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:38.180+0000 7f80c9ed4640 1 --2- 172.21.15.123:0/2461022480 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f80c415aa00 0x7f80c40cff60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:38.184 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:38.180+0000 7f80c9ed4640 1 --2- 172.21.15.123:0/2461022480 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f80c40cc3c0 0x7f80c40cfa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:38.184 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:38.180+0000 7f80c9ed4640 1 --2- 172.21.15.123:0/2461022480 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f80c4072720 0x7f80c40c60e0 unknown :-1 s=CLOSED pgs=881 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:38.184 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:38.180+0000 7f80c9ed4640 1 -- 172.21.15.123:0/2461022480 >> 172.21.15.123:0/2461022480 conn(0x7f80c40b1710 msgr2=0x7f80c4159270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:47:38.184 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:38.180+0000 7f80c9ed4640 1 -- 172.21.15.123:0/2461022480 shutdown_connections 2024-09-16T15:47:38.184 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:38.180+0000 7f80c9ed4640 1 -- 172.21.15.123:0/2461022480 wait complete. 2024-09-16T15:47:38.204 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-09-16T15:47:38.337 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:38.332+0000 7f4f44b17640 1 -- 172.21.15.123:0/2614551702 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4f4015a1b0 msgr2=0x7f4f40072fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:47:38.337 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:38.332+0000 7f4f44b17640 1 --2- 172.21.15.123:0/2614551702 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4f4015a1b0 0x7f4f40072fb0 secure :-1 s=READY pgs=882 cs=0 l=1 rev1=1 crypto rx=0x7f4f40076300 tx=0x7f4f3801f0b0 comp rx=0 tx=0).stop 2024-09-16T15:47:38.337 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:38.332+0000 7f4f44b17640 1 -- 172.21.15.123:0/2614551702 shutdown_connections 2024-09-16T15:47:38.337 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:38.332+0000 7f4f44b17640 1 --2- 172.21.15.123:0/2614551702 >> v2:172.21.15.184:3300/0 conn(0x7f4f4015a590 0x7f4f4015a9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:38.337 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:38.332+0000 7f4f44b17640 1 --2- 172.21.15.123:0/2614551702 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f4f4015dc40 0x7f4f4015e090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:38.337 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:38.332+0000 7f4f44b17640 1 --2- 172.21.15.123:0/2614551702 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4f4015a1b0 0x7f4f40072fb0 unknown :-1 s=CLOSED pgs=882 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:38.337 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:38.332+0000 7f4f44b17640 1 -- 172.21.15.123:0/2614551702 >> 172.21.15.123:0/2614551702 conn(0x7f4f400b1630 msgr2=0x7f4f400b1a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:47:38.337 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:38.332+0000 7f4f44b17640 1 -- 172.21.15.123:0/2614551702 shutdown_connections 2024-09-16T15:47:38.337 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:38.332+0000 7f4f44b17640 1 -- 172.21.15.123:0/2614551702 wait complete. 2024-09-16T15:47:38.338 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:38.332+0000 7f4f44b17640 1 Processor -- start 2024-09-16T15:47:38.338 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:38.332+0000 7f4f44b17640 1 -- start start 2024-09-16T15:47:38.340 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:38.336+0000 7f4f44b17640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4f4015a1b0 0x7f4f401f2ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:47:38.340 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:38.336+0000 7f4f44b17640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4f4015a590 0x7f4f4013c390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:47:38.340 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:38.336+0000 7f4f44b17640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f4f4015dc40 0x7f4f400cda70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:47:38.340 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:38.336+0000 7f4f44b17640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f4f4015b0a0 con 0x7f4f4015a590 2024-09-16T15:47:38.340 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:38.336+0000 7f4f44b17640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f4f4015af20 con 0x7f4f4015a1b0 2024-09-16T15:47:38.340 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:38.336+0000 7f4f44b17640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f4f4015b220 con 0x7f4f4015dc40 2024-09-16T15:47:38.340 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:38.336+0000 7f4f3dd9b640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4f4015a590 0x7f4f4013c390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:47:38.340 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:38.336+0000 7f4f3dd9b640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4f4015a590 0x7f4f4013c390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:53076/0 (socket says 172.21.15.123:53076) 2024-09-16T15:47:38.340 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:38.336+0000 7f4f3dd9b640 1 -- 172.21.15.123:0/38982708 learned_addr learned my addr 172.21.15.123:0/38982708 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:47:38.340 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:38.336+0000 7f4f3e59c640 1 --2- 172.21.15.123:0/38982708 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4f4015a1b0 0x7f4f401f2ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:47:38.340 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:38.336+0000 7f4f3ed9d640 1 --2- 172.21.15.123:0/38982708 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f4f4015dc40 0x7f4f400cda70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:47:38.340 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:38.336+0000 7f4f3dd9b640 1 -- 172.21.15.123:0/38982708 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f4f4015dc40 msgr2=0x7f4f400cda70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:47:38.341 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:38.336+0000 7f4f3dd9b640 1 --2- 172.21.15.123:0/38982708 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f4f4015dc40 0x7f4f400cda70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:38.341 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:38.336+0000 7f4f3dd9b640 1 -- 172.21.15.123:0/38982708 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4f4015a1b0 msgr2=0x7f4f401f2ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:47:38.341 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:38.336+0000 7f4f3dd9b640 1 --2- 172.21.15.123:0/38982708 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4f4015a1b0 0x7f4f401f2ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:38.341 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:38.336+0000 7f4f3dd9b640 1 -- 172.21.15.123:0/38982708 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4f400cc9d0 con 0x7f4f4015a590 2024-09-16T15:47:38.341 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:38.336+0000 7f4f3dd9b640 1 --2- 172.21.15.123:0/38982708 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4f4015a590 0x7f4f4013c390 secure :-1 s=READY pgs=883 cs=0 l=1 rev1=1 crypto rx=0x7f4f2800d720 tx=0x7f4f2800dbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:47:38.341 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:38.336+0000 7f4f36ffd640 1 -- 172.21.15.123:0/38982708 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f28017070 con 0x7f4f4015a590 2024-09-16T15:47:38.341 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:38.336+0000 7f4f36ffd640 1 -- 172.21.15.123:0/38982708 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f4f28009d20 con 0x7f4f4015a590 2024-09-16T15:47:38.341 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:38.336+0000 7f4f44b17640 1 -- 172.21.15.123:0/38982708 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4f400cccb0 con 0x7f4f4015a590 2024-09-16T15:47:38.341 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:38.336+0000 7f4f36ffd640 1 -- 172.21.15.123:0/38982708 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f28005280 con 0x7f4f4015a590 2024-09-16T15:47:38.341 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:38.336+0000 7f4f44b17640 1 -- 172.21.15.123:0/38982708 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4f400cd270 con 0x7f4f4015a590 2024-09-16T15:47:38.342 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:38.336+0000 7f4f44b17640 1 -- 172.21.15.123:0/38982708 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4f4015a9e0 con 0x7f4f4015a590 2024-09-16T15:47:38.342 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:38.336+0000 7f4f36ffd640 1 -- 172.21.15.123:0/38982708 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 16) ==== 97182+0+0 (secure 0 0 0) 0x7f4f28005420 con 0x7f4f4015a590 2024-09-16T15:47:38.342 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:38.336+0000 7f4f36ffd640 1 --2- 172.21.15.123:0/38982708 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f4f20072fe0 0x7f4f200754a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:47:38.342 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:38.336+0000 7f4f36ffd640 1 -- 172.21.15.123:0/38982708 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(74..74 src has 1..74) ==== 6412+0+0 (secure 0 0 0) 0x7f4f280943c0 con 0x7f4f4015a590 2024-09-16T15:47:38.342 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:38.336+0000 7f4f3e59c640 1 --2- 172.21.15.123:0/38982708 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f4f20072fe0 0x7f4f200754a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:47:38.342 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:38.336+0000 7f4f3e59c640 1 --2- 172.21.15.123:0/38982708 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f4f20072fe0 0x7f4f200754a0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f4f380b2000 tx=0x7f4f3801d8f0 comp rx=0 tx=0).ready entity=mgr.4150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:47:38.345 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:38.340+0000 7f4f36ffd640 1 -- 172.21.15.123:0/38982708 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f4f2805f130 con 0x7f4f4015a590 2024-09-16T15:47:38.520 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:38.516+0000 7f4f44b17640 1 -- 172.21.15.123:0/38982708 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) -- 0x7f4f400703c0 con 0x7f4f4015a590 2024-09-16T15:47:39.181 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:39.176+0000 7f4f36ffd640 1 -- 172.21.15.123:0/38982708 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v75) ==== 181+0+0 (secure 0 0 0) 0x7f4f2805ea30 con 0x7f4f4015a590 2024-09-16T15:47:39.181 INFO:teuthology.orchestra.run.smithi123.stderr:pool 'cephfs_metadata' removed 2024-09-16T15:47:39.185 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:39.180+0000 7f4f44b17640 1 -- 172.21.15.123:0/38982708 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f4f20072fe0 msgr2=0x7f4f200754a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:47:39.185 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:39.180+0000 7f4f44b17640 1 --2- 172.21.15.123:0/38982708 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f4f20072fe0 0x7f4f200754a0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f4f380b2000 tx=0x7f4f3801d8f0 comp rx=0 tx=0).stop 2024-09-16T15:47:39.185 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:39.180+0000 7f4f44b17640 1 -- 172.21.15.123:0/38982708 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4f4015a590 msgr2=0x7f4f4013c390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:47:39.185 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:39.180+0000 7f4f44b17640 1 --2- 172.21.15.123:0/38982708 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4f4015a590 0x7f4f4013c390 secure :-1 s=READY pgs=883 cs=0 l=1 rev1=1 crypto rx=0x7f4f2800d720 tx=0x7f4f2800dbe0 comp rx=0 tx=0).stop 2024-09-16T15:47:39.186 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:39.180+0000 7f4f44b17640 1 -- 172.21.15.123:0/38982708 shutdown_connections 2024-09-16T15:47:39.186 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:39.180+0000 7f4f44b17640 1 --2- 172.21.15.123:0/38982708 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f4f20072fe0 0x7f4f200754a0 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:39.186 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:39.180+0000 7f4f44b17640 1 --2- 172.21.15.123:0/38982708 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f4f4015dc40 0x7f4f400cda70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:39.187 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:39.180+0000 7f4f44b17640 1 --2- 172.21.15.123:0/38982708 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4f4015a590 0x7f4f4013c390 unknown :-1 s=CLOSED pgs=883 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:39.187 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:39.180+0000 7f4f44b17640 1 --2- 172.21.15.123:0/38982708 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4f4015a1b0 0x7f4f401f2ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:39.187 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:39.180+0000 7f4f44b17640 1 -- 172.21.15.123:0/38982708 >> 172.21.15.123:0/38982708 conn(0x7f4f400b1630 msgr2=0x7f4f4006f630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:47:39.187 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:39.184+0000 7f4f44b17640 1 -- 172.21.15.123:0/38982708 shutdown_connections 2024-09-16T15:47:39.187 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:39.184+0000 7f4f44b17640 1 -- 172.21.15.123:0/38982708 wait complete. 2024-09-16T15:47:39.208 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-09-16T15:47:39.335 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:39.332+0000 7fb9e1a29640 1 -- 172.21.15.123:0/2739667812 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb9dc0cf3a0 msgr2=0x7fb9dc0b4120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:47:39.335 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:39.332+0000 7fb9e1a29640 1 --2- 172.21.15.123:0/2739667812 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb9dc0cf3a0 0x7fb9dc0b4120 secure :-1 s=READY pgs=884 cs=0 l=1 rev1=1 crypto rx=0x7fb9dc0722d0 tx=0x7fb9d401ccf0 comp rx=0 tx=0).stop 2024-09-16T15:47:39.335 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:39.332+0000 7fb9e1a29640 1 -- 172.21.15.123:0/2739667812 shutdown_connections 2024-09-16T15:47:39.335 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:39.332+0000 7fb9e1a29640 1 --2- 172.21.15.123:0/2739667812 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb9dc0cf3a0 0x7fb9dc0b4120 unknown :-1 s=CLOSED pgs=884 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:39.335 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:39.332+0000 7fb9e1a29640 1 --2- 172.21.15.123:0/2739667812 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fb9dc072ff0 0x7fb9dc0cee60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:39.336 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:39.332+0000 7fb9e1a29640 1 --2- 172.21.15.123:0/2739667812 >> v2:172.21.15.184:3300/0 conn(0x7fb9dc0ae680 0x7fb9dc15d6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:39.336 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:39.332+0000 7fb9e1a29640 1 -- 172.21.15.123:0/2739667812 >> 172.21.15.123:0/2739667812 conn(0x7fb9dc0b1630 msgr2=0x7fb9dc0b1a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:47:39.336 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:39.332+0000 7fb9e1a29640 1 -- 172.21.15.123:0/2739667812 shutdown_connections 2024-09-16T15:47:39.336 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:39.332+0000 7fb9e1a29640 1 -- 172.21.15.123:0/2739667812 wait complete. 2024-09-16T15:47:39.336 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:39.332+0000 7fb9e1a29640 1 Processor -- start 2024-09-16T15:47:39.336 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:39.332+0000 7fb9e1a29640 1 -- start start 2024-09-16T15:47:39.338 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:39.332+0000 7fb9e1a29640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fb9dc072ff0 0x7fb9dc1f9c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:47:39.338 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:39.332+0000 7fb9e1a29640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb9dc0ae680 0x7fb9dc158b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:47:39.338 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:39.332+0000 7fb9e1a29640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb9dc0cf3a0 0x7fb9dc1f8450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:47:39.338 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:39.332+0000 7fb9e1a29640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fb9dc15e9f0 con 0x7fb9dc0ae680 2024-09-16T15:47:39.339 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:39.332+0000 7fb9e1a29640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fb9dc15e870 con 0x7fb9dc0cf3a0 2024-09-16T15:47:39.339 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:39.332+0000 7fb9e1a29640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fb9dc15eb70 con 0x7fb9dc072ff0 2024-09-16T15:47:39.339 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:39.332+0000 7fb9d3fff640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb9dc0ae680 0x7fb9dc158b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:47:39.339 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:39.332+0000 7fb9d3fff640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb9dc0ae680 0x7fb9dc158b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:53106/0 (socket says 172.21.15.123:53106) 2024-09-16T15:47:39.339 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:39.332+0000 7fb9d3fff640 1 -- 172.21.15.123:0/3803623961 learned_addr learned my addr 172.21.15.123:0/3803623961 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:47:39.339 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:39.332+0000 7fb9daffd640 1 --2- 172.21.15.123:0/3803623961 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fb9dc072ff0 0x7fb9dc1f9c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:47:39.339 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:39.332+0000 7fb9d3fff640 1 -- 172.21.15.123:0/3803623961 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fb9dc072ff0 msgr2=0x7fb9dc1f9c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:47:39.339 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:39.332+0000 7fb9db7fe640 1 --2- 172.21.15.123:0/3803623961 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb9dc0cf3a0 0x7fb9dc1f8450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:47:39.339 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:39.332+0000 7fb9d3fff640 1 --2- 172.21.15.123:0/3803623961 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fb9dc072ff0 0x7fb9dc1f9c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:39.339 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:39.332+0000 7fb9d3fff640 1 -- 172.21.15.123:0/3803623961 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb9dc0cf3a0 msgr2=0x7fb9dc1f8450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:47:39.339 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:39.332+0000 7fb9d3fff640 1 --2- 172.21.15.123:0/3803623961 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb9dc0cf3a0 0x7fb9dc1f8450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:39.340 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:39.332+0000 7fb9d3fff640 1 -- 172.21.15.123:0/3803623961 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb9dc14e2e0 con 0x7fb9dc0ae680 2024-09-16T15:47:39.340 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:39.332+0000 7fb9d3fff640 1 --2- 172.21.15.123:0/3803623961 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb9dc0ae680 0x7fb9dc158b00 secure :-1 s=READY pgs=885 cs=0 l=1 rev1=1 crypto rx=0x7fb9c400b580 tx=0x7fb9c400ba40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:47:39.340 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:39.332+0000 7fb9d8ff9640 1 -- 172.21.15.123:0/3803623961 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9c4014070 con 0x7fb9dc0ae680 2024-09-16T15:47:39.340 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:39.332+0000 7fb9d8ff9640 1 -- 172.21.15.123:0/3803623961 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fb9c4010980 con 0x7fb9dc0ae680 2024-09-16T15:47:39.340 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:39.332+0000 7fb9e1a29640 1 -- 172.21.15.123:0/3803623961 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb9dc14ce50 con 0x7fb9dc0ae680 2024-09-16T15:47:39.340 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:39.332+0000 7fb9d8ff9640 1 -- 172.21.15.123:0/3803623961 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9c4010dd0 con 0x7fb9dc0ae680 2024-09-16T15:47:39.340 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:39.332+0000 7fb9e1a29640 1 -- 172.21.15.123:0/3803623961 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb9dc14d410 con 0x7fb9dc0ae680 2024-09-16T15:47:39.341 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:39.336+0000 7fb9e1a29640 1 -- 172.21.15.123:0/3803623961 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb9dc15d6a0 con 0x7fb9dc0ae680 2024-09-16T15:47:39.341 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:39.336+0000 7fb9d8ff9640 1 -- 172.21.15.123:0/3803623961 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 16) ==== 97182+0+0 (secure 0 0 0) 0x7fb9c4017070 con 0x7fb9dc0ae680 2024-09-16T15:47:39.341 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:39.336+0000 7fb9d8ff9640 1 --2- 172.21.15.123:0/3803623961 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fb9bc072fe0 0x7fb9bc0754a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:47:39.341 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:39.336+0000 7fb9d8ff9640 1 -- 172.21.15.123:0/3803623961 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(75..75 src has 1..75) ==== 5974+0+0 (secure 0 0 0) 0x7fb9c4093f60 con 0x7fb9dc0ae680 2024-09-16T15:47:39.341 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:39.336+0000 7fb9daffd640 1 --2- 172.21.15.123:0/3803623961 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fb9bc072fe0 0x7fb9bc0754a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:47:39.341 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:39.336+0000 7fb9daffd640 1 --2- 172.21.15.123:0/3803623961 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fb9bc072fe0 0x7fb9bc0754a0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fb9cc00a830 tx=0x7fb9cc008040 comp rx=0 tx=0).ready entity=mgr.4150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:47:39.344 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:39.340+0000 7fb9d8ff9640 1 -- 172.21.15.123:0/3803623961 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7fb9c405ee90 con 0x7fb9dc0ae680 2024-09-16T15:47:39.517 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:39.512+0000 7fb9e1a29640 1 -- 172.21.15.123:0/3803623961 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) -- 0x7fb9dc0b2b00 con 0x7fb9dc0ae680 2024-09-16T15:47:39.685 DEBUG:teuthology.orchestra.run.smithi123:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-16T15:47:39.688 DEBUG:teuthology.orchestra.run.smithi184:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-16T15:47:40.190 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:40.184+0000 7fb9d8ff9640 1 -- 172.21.15.123:0/3803623961 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v76) ==== 169+0+0 (secure 0 0 0) 0x7fb9c405e790 con 0x7fb9dc0ae680 2024-09-16T15:47:40.190 INFO:teuthology.orchestra.run.smithi123.stderr:pool 'cephfs_data' removed 2024-09-16T15:47:40.193 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:40.188+0000 7fb9e1a29640 1 -- 172.21.15.123:0/3803623961 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fb9bc072fe0 msgr2=0x7fb9bc0754a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:47:40.193 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:40.188+0000 7fb9e1a29640 1 --2- 172.21.15.123:0/3803623961 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fb9bc072fe0 0x7fb9bc0754a0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fb9cc00a830 tx=0x7fb9cc008040 comp rx=0 tx=0).stop 2024-09-16T15:47:40.193 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:40.188+0000 7fb9e1a29640 1 -- 172.21.15.123:0/3803623961 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb9dc0ae680 msgr2=0x7fb9dc158b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:47:40.193 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:40.188+0000 7fb9e1a29640 1 --2- 172.21.15.123:0/3803623961 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb9dc0ae680 0x7fb9dc158b00 secure :-1 s=READY pgs=885 cs=0 l=1 rev1=1 crypto rx=0x7fb9c400b580 tx=0x7fb9c400ba40 comp rx=0 tx=0).stop 2024-09-16T15:47:40.193 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:40.188+0000 7fb9e1a29640 1 -- 172.21.15.123:0/3803623961 shutdown_connections 2024-09-16T15:47:40.193 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:40.188+0000 7fb9e1a29640 1 --2- 172.21.15.123:0/3803623961 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fb9bc072fe0 0x7fb9bc0754a0 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:40.193 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:40.188+0000 7fb9e1a29640 1 --2- 172.21.15.123:0/3803623961 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb9dc0cf3a0 0x7fb9dc1f8450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:40.193 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:40.188+0000 7fb9e1a29640 1 --2- 172.21.15.123:0/3803623961 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb9dc0ae680 0x7fb9dc158b00 secure :-1 s=CLOSED pgs=885 cs=0 l=1 rev1=1 crypto rx=0x7fb9c400b580 tx=0x7fb9c400ba40 comp rx=0 tx=0).stop 2024-09-16T15:47:40.193 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:40.188+0000 7fb9e1a29640 1 --2- 172.21.15.123:0/3803623961 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fb9dc072ff0 0x7fb9dc1f9c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:40.193 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:40.188+0000 7fb9e1a29640 1 -- 172.21.15.123:0/3803623961 >> 172.21.15.123:0/3803623961 conn(0x7fb9dc0b1630 msgr2=0x7fb9dc0b1a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:47:40.195 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:40.192+0000 7fb9e1a29640 1 -- 172.21.15.123:0/3803623961 shutdown_connections 2024-09-16T15:47:40.196 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:40.192+0000 7fb9e1a29640 1 -- 172.21.15.123:0/3803623961 wait complete. 2024-09-16T15:47:40.219 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config reset 27 2024-09-16T15:47:40.349 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:40.344+0000 7f9370129640 1 -- 172.21.15.123:0/1920703453 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9354004810 msgr2=0x7f9354004c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:47:40.349 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:40.344+0000 7f9370129640 1 --2- 172.21.15.123:0/1920703453 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9354004810 0x7f9354004c90 secure :-1 s=READY pgs=887 cs=0 l=1 rev1=1 crypto rx=0x7f9358005fd0 tx=0x7f935801cd20 comp rx=0 tx=0).stop 2024-09-16T15:47:40.349 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:40.344+0000 7f936e6c6640 1 -- 172.21.15.123:0/1920703453 reap_dead start 2024-09-16T15:47:40.349 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:40.344+0000 7f9370129640 1 -- 172.21.15.123:0/1920703453 shutdown_connections 2024-09-16T15:47:40.349 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:40.344+0000 7f9370129640 1 -- 172.21.15.123:0/1920703453 >> 172.21.15.123:0/1920703453 conn(0x7f93680b1df0 msgr2=0x7f93680b2200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:47:40.349 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:40.344+0000 7f9370129640 1 -- 172.21.15.123:0/1920703453 shutdown_connections 2024-09-16T15:47:40.349 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:40.344+0000 7f9370129640 1 -- 172.21.15.123:0/1920703453 wait complete. 2024-09-16T15:47:40.350 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:40.344+0000 7f9370129640 1 Processor -- start 2024-09-16T15:47:40.350 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:40.344+0000 7f9370129640 1 -- start start 2024-09-16T15:47:40.352 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:40.348+0000 7f9370129640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9354004810 0x7f93681e7280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:47:40.352 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:40.348+0000 7f9370129640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f93681f5a30 0x7f936809d0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:47:40.352 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:40.348+0000 7f9370129640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f93681f4e90 0x7f93681f5310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:47:40.352 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:40.348+0000 7f9370129640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f93680b0890 con 0x7f9354004810 2024-09-16T15:47:40.352 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:40.348+0000 7f9370129640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f93680b0710 con 0x7f93681f5a30 2024-09-16T15:47:40.352 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:40.348+0000 7f9370129640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f93680ce5e0 con 0x7f93681f4e90 2024-09-16T15:47:40.352 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:40.348+0000 7f936dec5640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9354004810 0x7f93681e7280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:47:40.352 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:40.348+0000 7f936dec5640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9354004810 0x7f93681e7280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:53118/0 (socket says 172.21.15.123:53118) 2024-09-16T15:47:40.352 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:40.348+0000 7f936dec5640 1 -- 172.21.15.123:0/2005679331 learned_addr learned my addr 172.21.15.123:0/2005679331 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:47:40.353 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:40.348+0000 7f936d6c4640 1 --2- 172.21.15.123:0/2005679331 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f93681f5a30 0x7f936809d0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:47:40.353 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:40.348+0000 7f936e6c6640 1 --2- 172.21.15.123:0/2005679331 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f93681f4e90 0x7f93681f5310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:47:40.353 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:40.348+0000 7f936dec5640 1 -- 172.21.15.123:0/2005679331 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f93681f4e90 msgr2=0x7f93681f5310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:47:40.353 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:40.348+0000 7f936dec5640 1 --2- 172.21.15.123:0/2005679331 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f93681f4e90 0x7f93681f5310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:40.353 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:40.348+0000 7f936dec5640 1 -- 172.21.15.123:0/2005679331 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f93681f5a30 msgr2=0x7f936809d0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:47:40.353 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:40.348+0000 7f936dec5640 1 --2- 172.21.15.123:0/2005679331 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f93681f5a30 0x7f936809d0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:40.353 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:40.348+0000 7f936dec5640 1 -- 172.21.15.123:0/2005679331 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f93681e4ca0 con 0x7f9354004810 2024-09-16T15:47:40.353 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:40.348+0000 7f936dec5640 1 --2- 172.21.15.123:0/2005679331 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9354004810 0x7f93681e7280 secure :-1 s=READY pgs=888 cs=0 l=1 rev1=1 crypto rx=0x7f936400fa00 tx=0x7f936400fec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:47:40.353 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:40.348+0000 7f935e7fc640 1 -- 172.21.15.123:0/2005679331 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f936400bc50 con 0x7f9354004810 2024-09-16T15:47:40.353 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:40.348+0000 7f935e7fc640 1 -- 172.21.15.123:0/2005679331 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f9364011440 con 0x7f9354004810 2024-09-16T15:47:40.353 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:40.348+0000 7f9370129640 1 -- 172.21.15.123:0/2005679331 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f93681e4f60 con 0x7f9354004810 2024-09-16T15:47:40.353 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:40.348+0000 7f935e7fc640 1 -- 172.21.15.123:0/2005679331 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9364011850 con 0x7f9354004810 2024-09-16T15:47:40.353 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:40.348+0000 7f9370129640 1 -- 172.21.15.123:0/2005679331 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f93681e5240 con 0x7f9354004810 2024-09-16T15:47:40.354 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:40.348+0000 7f9370129640 1 -- 172.21.15.123:0/2005679331 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f93680b1200 con 0x7f9354004810 2024-09-16T15:47:40.354 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:40.348+0000 7f935e7fc640 1 -- 172.21.15.123:0/2005679331 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 16) ==== 97182+0+0 (secure 0 0 0) 0x7f9364019020 con 0x7f9354004810 2024-09-16T15:47:40.354 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:40.348+0000 7f935e7fc640 1 --2- 172.21.15.123:0/2005679331 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f934c073040 0x7f934c075500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:47:40.354 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:40.348+0000 7f935e7fc640 1 -- 172.21.15.123:0/2005679331 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(76..76 src has 1..76) ==== 5502+0+0 (secure 0 0 0) 0x7f93640946f0 con 0x7f9354004810 2024-09-16T15:47:40.355 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:40.348+0000 7f936d6c4640 1 --2- 172.21.15.123:0/2005679331 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f934c073040 0x7f934c075500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:47:40.355 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:40.352+0000 7f936d6c4640 1 --2- 172.21.15.123:0/2005679331 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f934c073040 0x7f934c075500 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f93580b0000 tx=0x7f93580a7040 comp rx=0 tx=0).ready entity=mgr.4150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:47:40.358 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:40.356+0000 7f935e7fc640 1 -- 172.21.15.123:0/2005679331 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f936405f880 con 0x7f9354004810 2024-09-16T15:47:40.537 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:40.532+0000 7f9370129640 1 -- 172.21.15.123:0/2005679331 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "config reset", "num": 27} v 0) -- 0x7f93680c3bb0 con 0x7f9354004810 2024-09-16T15:47:40.543 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:40.540+0000 7f935e7fc640 1 -- 172.21.15.123:0/2005679331 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "config reset", "num": 27}]=0 v35) ==== 71+0+0 (secure 0 0 0) 0x7f936405f180 con 0x7f9354004810 2024-09-16T15:47:40.545 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:40.540+0000 7f9370129640 1 -- 172.21.15.123:0/2005679331 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f934c073040 msgr2=0x7f934c075500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:47:40.545 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:40.540+0000 7f9370129640 1 --2- 172.21.15.123:0/2005679331 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f934c073040 0x7f934c075500 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f93580b0000 tx=0x7f93580a7040 comp rx=0 tx=0).stop 2024-09-16T15:47:40.546 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:40.540+0000 7f9370129640 1 -- 172.21.15.123:0/2005679331 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9354004810 msgr2=0x7f93681e7280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:47:40.546 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:40.540+0000 7f9370129640 1 --2- 172.21.15.123:0/2005679331 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9354004810 0x7f93681e7280 secure :-1 s=READY pgs=888 cs=0 l=1 rev1=1 crypto rx=0x7f936400fa00 tx=0x7f936400fec0 comp rx=0 tx=0).stop 2024-09-16T15:47:40.546 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:40.540+0000 7f9370129640 1 -- 172.21.15.123:0/2005679331 shutdown_connections 2024-09-16T15:47:40.546 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:40.540+0000 7f9370129640 1 --2- 172.21.15.123:0/2005679331 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f934c073040 0x7f934c075500 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:40.546 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:40.540+0000 7f9370129640 1 --2- 172.21.15.123:0/2005679331 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f93681f4e90 0x7f93681f5310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:40.546 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:40.540+0000 7f9370129640 1 --2- 172.21.15.123:0/2005679331 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f93681f5a30 0x7f936809d0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:40.546 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:40.540+0000 7f9370129640 1 --2- 172.21.15.123:0/2005679331 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9354004810 0x7f93681e7280 unknown :-1 s=CLOSED pgs=888 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:40.546 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:40.540+0000 7f9370129640 1 -- 172.21.15.123:0/2005679331 >> 172.21.15.123:0/2005679331 conn(0x7f93680b1df0 msgr2=0x7f93680b2200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:47:40.546 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:40.540+0000 7f9370129640 1 -- 172.21.15.123:0/2005679331 shutdown_connections 2024-09-16T15:47:40.546 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:40.540+0000 7f9370129640 1 -- 172.21.15.123:0/2005679331 wait complete. 2024-09-16T15:47:40.561 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_volumes.TestMisc.test_mgr_eviction' 2024-09-16T15:47:40.677 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:40.672+0000 7f38e5ede640 1 -- 172.21.15.123:0/174349628 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f38e015ec40 msgr2=0x7f38e006f420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:47:40.677 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:40.672+0000 7f38e5ede640 1 --2- 172.21.15.123:0/174349628 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f38e015ec40 0x7f38e006f420 secure :-1 s=READY pgs=889 cs=0 l=1 rev1=1 crypto rx=0x7f38d8009c80 tx=0x7f38d801cc80 comp rx=0 tx=0).stop 2024-09-16T15:47:40.677 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:40.672+0000 7f38e5ede640 1 -- 172.21.15.123:0/174349628 shutdown_connections 2024-09-16T15:47:40.677 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:40.672+0000 7f38e5ede640 1 --2- 172.21.15.123:0/174349628 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f38e015ec40 0x7f38e006f420 unknown :-1 s=CLOSED pgs=889 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:40.677 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:40.672+0000 7f38e5ede640 1 --2- 172.21.15.123:0/174349628 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f38e00cf5a0 0x7f38e006eee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:40.677 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:40.672+0000 7f38e5ede640 1 -- 172.21.15.123:0/174349628 >> 172.21.15.123:0/174349628 conn(0x7f38e00cfb10 msgr2=0x7f38e00cff20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:47:40.677 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:40.672+0000 7f38e5ede640 1 -- 172.21.15.123:0/174349628 shutdown_connections 2024-09-16T15:47:40.677 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:40.672+0000 7f38e5ede640 1 -- 172.21.15.123:0/174349628 wait complete. 2024-09-16T15:47:40.678 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:40.672+0000 7f38e5ede640 1 Processor -- start 2024-09-16T15:47:40.678 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:40.672+0000 7f38e5ede640 1 -- start start 2024-09-16T15:47:40.679 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:40.676+0000 7f38e5ede640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f38e00cf5a0 0x7f38e00c6f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:47:40.680 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:40.676+0000 7f38e5ede640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f38e0158120 0x7f38e01f99a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:47:40.680 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:40.676+0000 7f38e5ede640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f38e015ec40 0x7f38e0148020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:47:40.680 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:40.676+0000 7f38e5ede640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f38e015b300 con 0x7f38e015ec40 2024-09-16T15:47:40.680 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:40.676+0000 7f38e5ede640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f38e015b180 con 0x7f38e0158120 2024-09-16T15:47:40.680 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:40.676+0000 7f38e5ede640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f38e015b480 con 0x7f38e00cf5a0 2024-09-16T15:47:40.680 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:40.676+0000 7f38dffff640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f38e015ec40 0x7f38e0148020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:47:40.680 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:40.676+0000 7f38dffff640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f38e015ec40 0x7f38e0148020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:53134/0 (socket says 172.21.15.123:53134) 2024-09-16T15:47:40.680 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:40.676+0000 7f38dffff640 1 -- 172.21.15.123:0/560599915 learned_addr learned my addr 172.21.15.123:0/560599915 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:47:40.680 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:40.676+0000 7f38df7fe640 1 --2- 172.21.15.123:0/560599915 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f38e00cf5a0 0x7f38e00c6f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:47:40.681 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:40.676+0000 7f38deffd640 1 --2- 172.21.15.123:0/560599915 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f38e0158120 0x7f38e01f99a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:47:40.681 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:40.676+0000 7f38dffff640 1 -- 172.21.15.123:0/560599915 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f38e00cf5a0 msgr2=0x7f38e00c6f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:47:40.681 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:40.676+0000 7f38dffff640 1 --2- 172.21.15.123:0/560599915 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f38e00cf5a0 0x7f38e00c6f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:40.681 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:40.676+0000 7f38dffff640 1 -- 172.21.15.123:0/560599915 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f38e0158120 msgr2=0x7f38e01f99a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:47:40.681 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:40.676+0000 7f38dffff640 1 --2- 172.21.15.123:0/560599915 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f38e0158120 0x7f38e01f99a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:40.681 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:40.676+0000 7f38dffff640 1 -- 172.21.15.123:0/560599915 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f38e0148560 con 0x7f38e015ec40 2024-09-16T15:47:40.681 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:40.676+0000 7f38deffd640 1 --2- 172.21.15.123:0/560599915 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f38e0158120 0x7f38e01f99a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T15:47:40.681 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:40.676+0000 7f38df7fe640 1 --2- 172.21.15.123:0/560599915 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f38e00cf5a0 0x7f38e00c6f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T15:47:40.681 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:40.676+0000 7f38dffff640 1 --2- 172.21.15.123:0/560599915 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f38e015ec40 0x7f38e0148020 secure :-1 s=READY pgs=890 cs=0 l=1 rev1=1 crypto rx=0x7f38d801d520 tx=0x7f38d8009350 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:47:40.681 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:40.676+0000 7f38c7fff640 1 -- 172.21.15.123:0/560599915 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38d80bb070 con 0x7f38e015ec40 2024-09-16T15:47:40.681 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:40.676+0000 7f38c7fff640 1 -- 172.21.15.123:0/560599915 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f38d8004d40 con 0x7f38e015ec40 2024-09-16T15:47:40.681 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:40.676+0000 7f38c7fff640 1 -- 172.21.15.123:0/560599915 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38d80b2040 con 0x7f38e015ec40 2024-09-16T15:47:40.682 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:40.676+0000 7f38e5ede640 1 -- 172.21.15.123:0/560599915 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f38e01f5bb0 con 0x7f38e015ec40 2024-09-16T15:47:40.682 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:40.676+0000 7f38e5ede640 1 -- 172.21.15.123:0/560599915 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f38e01f6150 con 0x7f38e015ec40 2024-09-16T15:47:40.682 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:40.676+0000 7f38e5ede640 1 -- 172.21.15.123:0/560599915 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f38e00ca320 con 0x7f38e015ec40 2024-09-16T15:47:40.682 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:40.676+0000 7f38c7fff640 1 -- 172.21.15.123:0/560599915 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 16) ==== 97182+0+0 (secure 0 0 0) 0x7f38d80a5a40 con 0x7f38e015ec40 2024-09-16T15:47:40.682 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:40.676+0000 7f38c7fff640 1 --2- 172.21.15.123:0/560599915 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f38cc072fe0 0x7f38cc0754a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:47:40.683 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:40.676+0000 7f38c7fff640 1 -- 172.21.15.123:0/560599915 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(76..76 src has 1..76) ==== 5502+0+0 (secure 0 0 0) 0x7f38d812dd90 con 0x7f38e015ec40 2024-09-16T15:47:40.683 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:40.676+0000 7f38df7fe640 1 --2- 172.21.15.123:0/560599915 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f38cc072fe0 0x7f38cc0754a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:47:40.683 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:40.676+0000 7f38df7fe640 1 --2- 172.21.15.123:0/560599915 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f38cc072fe0 0x7f38cc0754a0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f38e01517a0 tx=0x7f38d0009040 comp rx=0 tx=0).ready entity=mgr.4150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:47:40.685 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:40.680+0000 7f38c7fff640 1 -- 172.21.15.123:0/560599915 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f38d8016990 con 0x7f38e015ec40 2024-09-16T15:47:40.920 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:40.916+0000 7f38e5ede640 1 -- 172.21.15.123:0/560599915 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_volumes.TestMisc.test_mgr_eviction"]} v 0) -- 0x7f38e00d06c0 con 0x7f38e015ec40 2024-09-16T15:47:41.192 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:41.188+0000 7f38c7fff640 1 -- 172.21.15.123:0/560599915 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_volumes.TestMisc.test_mgr_eviction"]}]=0 v248) ==== 131+0+0 (secure 0 0 0) 0x7f38d80f8f20 con 0x7f38e015ec40 2024-09-16T15:47:41.196 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:41.192+0000 7f38e5ede640 1 -- 172.21.15.123:0/560599915 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f38cc072fe0 msgr2=0x7f38cc0754a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:47:41.196 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:41.192+0000 7f38e5ede640 1 --2- 172.21.15.123:0/560599915 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f38cc072fe0 0x7f38cc0754a0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f38e01517a0 tx=0x7f38d0009040 comp rx=0 tx=0).stop 2024-09-16T15:47:41.196 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:41.192+0000 7f38e5ede640 1 -- 172.21.15.123:0/560599915 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f38e015ec40 msgr2=0x7f38e0148020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:47:41.196 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:41.192+0000 7f38e5ede640 1 --2- 172.21.15.123:0/560599915 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f38e015ec40 0x7f38e0148020 secure :-1 s=READY pgs=890 cs=0 l=1 rev1=1 crypto rx=0x7f38d801d520 tx=0x7f38d8009350 comp rx=0 tx=0).stop 2024-09-16T15:47:41.196 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:41.192+0000 7f38e5ede640 1 -- 172.21.15.123:0/560599915 shutdown_connections 2024-09-16T15:47:41.197 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:41.192+0000 7f38e5ede640 1 --2- 172.21.15.123:0/560599915 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f38cc072fe0 0x7f38cc0754a0 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:41.197 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:41.192+0000 7f38e5ede640 1 --2- 172.21.15.123:0/560599915 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f38e015ec40 0x7f38e0148020 unknown :-1 s=CLOSED pgs=890 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:41.197 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:41.192+0000 7f38e5ede640 1 --2- 172.21.15.123:0/560599915 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f38e0158120 0x7f38e01f99a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:41.197 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:41.192+0000 7f38e5ede640 1 --2- 172.21.15.123:0/560599915 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f38e00cf5a0 0x7f38e00c6f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:41.197 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:41.192+0000 7f38e5ede640 1 -- 172.21.15.123:0/560599915 >> 172.21.15.123:0/560599915 conn(0x7f38e00cfb10 msgr2=0x7f38e0072730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:47:41.197 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:41.192+0000 7f38e5ede640 1 -- 172.21.15.123:0/560599915 shutdown_connections 2024-09-16T15:47:41.197 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:41.192+0000 7f38e5ede640 1 -- 172.21.15.123:0/560599915 wait complete. 2024-09-16T15:47:41.216 INFO:tasks.cephfs_test_runner:test_mgr_eviction (tasks.cephfs.test_volumes.TestMisc) ... ok 2024-09-16T15:47:41.216 INFO:tasks.cephfs_test_runner:Starting test: test_names_can_only_be_goodchars (tasks.cephfs.test_volumes.TestMisc) Test the creating vols, subvols subvolgroups fails when their names uses 2024-09-16T15:47:41.216 INFO:tasks.cephfs_test_runner:test_names_can_only_be_goodchars (tasks.cephfs.test_volumes.TestMisc) 2024-09-16T15:47:41.216 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-16T15:47:41.344 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:41.340+0000 7f3bde96b640 1 -- 172.21.15.123:0/3410319734 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3bd8073d10 msgr2=0x7f3bd80cfcb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:47:41.344 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:41.340+0000 7f3bde96b640 1 --2- 172.21.15.123:0/3410319734 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3bd8073d10 0x7f3bd80cfcb0 secure :-1 s=READY pgs=891 cs=0 l=1 rev1=1 crypto rx=0x7f3bd8051870 tx=0x7f3bc801cd50 comp rx=0 tx=0).stop 2024-09-16T15:47:41.344 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:41.340+0000 7f3bde96b640 1 -- 172.21.15.123:0/3410319734 shutdown_connections 2024-09-16T15:47:41.344 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:41.340+0000 7f3bde96b640 1 --2- 172.21.15.123:0/3410319734 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3bd8073d10 0x7f3bd80cfcb0 unknown :-1 s=CLOSED pgs=891 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:41.344 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:41.340+0000 7f3bde96b640 1 --2- 172.21.15.123:0/3410319734 >> v2:172.21.15.184:3300/0 conn(0x7f3bd8151b10 0x7f3bd815b120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:41.344 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:41.340+0000 7f3bde96b640 1 -- 172.21.15.123:0/3410319734 >> 172.21.15.123:0/3410319734 conn(0x7f3bd80cd5d0 msgr2=0x7f3bd80cd9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:47:41.344 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:41.340+0000 7f3bde96b640 1 -- 172.21.15.123:0/3410319734 shutdown_connections 2024-09-16T15:47:41.344 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:41.340+0000 7f3bde96b640 1 -- 172.21.15.123:0/3410319734 wait complete. 2024-09-16T15:47:41.345 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:41.340+0000 7f3bde96b640 1 Processor -- start 2024-09-16T15:47:41.345 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:41.340+0000 7f3bde96b640 1 -- start start 2024-09-16T15:47:41.347 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:41.340+0000 7f3bde96b640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f3bd8073d10 0x7f3bd8154620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:47:41.347 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:41.340+0000 7f3bde96b640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3bd80cb760 0x7f3bd80c2300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:47:41.347 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:41.340+0000 7f3bde96b640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3bd8151b10 0x7f3bd80c2840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:47:41.347 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:41.340+0000 7f3bde96b640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f3bd806fb70 con 0x7f3bd8151b10 2024-09-16T15:47:41.347 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:41.340+0000 7f3bde96b640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f3bd806f9f0 con 0x7f3bd80cb760 2024-09-16T15:47:41.347 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:41.340+0000 7f3bde96b640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f3bd806fcf0 con 0x7f3bd8073d10 2024-09-16T15:47:41.347 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:41.340+0000 7f3bdcf08640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3bd8151b10 0x7f3bd80c2840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:47:41.347 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:41.340+0000 7f3bdcf08640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3bd8151b10 0x7f3bd80c2840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:53160/0 (socket says 172.21.15.123:53160) 2024-09-16T15:47:41.348 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:41.340+0000 7f3bdcf08640 1 -- 172.21.15.123:0/2379224540 learned_addr learned my addr 172.21.15.123:0/2379224540 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:47:41.348 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:41.340+0000 7f3bd77fe640 1 --2- 172.21.15.123:0/2379224540 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3bd80cb760 0x7f3bd80c2300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:47:41.348 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:41.340+0000 7f3bd7fff640 1 --2- 172.21.15.123:0/2379224540 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f3bd8073d10 0x7f3bd8154620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:47:41.348 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:41.340+0000 7f3bdcf08640 1 -- 172.21.15.123:0/2379224540 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f3bd8073d10 msgr2=0x7f3bd8154620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:47:41.348 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:41.340+0000 7f3bdcf08640 1 --2- 172.21.15.123:0/2379224540 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f3bd8073d10 0x7f3bd8154620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:41.348 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:41.340+0000 7f3bdcf08640 1 -- 172.21.15.123:0/2379224540 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3bd80cb760 msgr2=0x7f3bd80c2300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:47:41.348 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:41.344+0000 7f3bdcf08640 1 --2- 172.21.15.123:0/2379224540 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3bd80cb760 0x7f3bd80c2300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:41.348 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:41.344+0000 7f3bdcf08640 1 -- 172.21.15.123:0/2379224540 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3bd80b67f0 con 0x7f3bd8151b10 2024-09-16T15:47:41.348 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:41.344+0000 7f3bd7fff640 1 --2- 172.21.15.123:0/2379224540 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f3bd8073d10 0x7f3bd8154620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T15:47:41.348 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:41.344+0000 7f3bdcf08640 1 --2- 172.21.15.123:0/2379224540 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3bd8151b10 0x7f3bd80c2840 secure :-1 s=READY pgs=892 cs=0 l=1 rev1=1 crypto rx=0x7f3bc8009bd0 tx=0x7f3bc8003f60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:47:41.348 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:41.344+0000 7f3bd4ff9640 1 -- 172.21.15.123:0/2379224540 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3bc80bb070 con 0x7f3bd8151b10 2024-09-16T15:47:41.348 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:41.344+0000 7f3bd4ff9640 1 -- 172.21.15.123:0/2379224540 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f3bc8004e00 con 0x7f3bd8151b10 2024-09-16T15:47:41.348 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:41.344+0000 7f3bd4ff9640 1 -- 172.21.15.123:0/2379224540 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3bc80b2040 con 0x7f3bd8151b10 2024-09-16T15:47:41.349 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:41.344+0000 7f3bde96b640 1 -- 172.21.15.123:0/2379224540 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3bd814e780 con 0x7f3bd8151b10 2024-09-16T15:47:41.349 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:41.344+0000 7f3bde96b640 1 -- 172.21.15.123:0/2379224540 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3bd80b62f0 con 0x7f3bd8151b10 2024-09-16T15:47:41.349 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:41.344+0000 7f3bde96b640 1 -- 172.21.15.123:0/2379224540 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3bd80cfcb0 con 0x7f3bd8151b10 2024-09-16T15:47:41.350 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:41.344+0000 7f3bd4ff9640 1 -- 172.21.15.123:0/2379224540 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 16) ==== 97182+0+0 (secure 0 0 0) 0x7f3bc80be070 con 0x7f3bd8151b10 2024-09-16T15:47:41.350 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:41.344+0000 7f3bd4ff9640 1 --2- 172.21.15.123:0/2379224540 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f3bb8072fe0 0x7f3bb80754a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:47:41.350 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:41.344+0000 7f3bd4ff9640 1 -- 172.21.15.123:0/2379224540 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(76..76 src has 1..76) ==== 5502+0+0 (secure 0 0 0) 0x7f3bc812df60 con 0x7f3bd8151b10 2024-09-16T15:47:41.350 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:41.344+0000 7f3bd7fff640 1 --2- 172.21.15.123:0/2379224540 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f3bb8072fe0 0x7f3bb80754a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:47:41.350 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:41.344+0000 7f3bd7fff640 1 --2- 172.21.15.123:0/2379224540 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f3bb8072fe0 0x7f3bb80754a0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f3bd0005870 tx=0x7f3bd0005db0 comp rx=0 tx=0).ready entity=mgr.4150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:47:41.353 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:41.348+0000 7f3bd4ff9640 1 -- 172.21.15.123:0/2379224540 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f3bc8016970 con 0x7f3bd8151b10 2024-09-16T15:47:41.521 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:41.516+0000 7f3bde96b640 1 -- 172.21.15.123:0/2379224540 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f3bd80ce4e0 con 0x7f3bd8151b10 2024-09-16T15:47:41.522 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:41.516+0000 7f3bd4ff9640 1 -- 172.21.15.123:0/2379224540 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v76) ==== 74+0+13916 (secure 0 0 0) 0x7f3bc80f91d0 con 0x7f3bd8151b10 2024-09-16T15:47:41.522 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-09-16T15:47:41.522 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":76,"fsid":"56077f89-18c8-486f-ae7e-eecc0bb7822c","created":"2024-09-16T15:42:26.364892+0000","modified":"2024-09-16T15:47:40.182121+0000","last_up_change":"2024-09-16T15:42:38.707234+0000","last_in_change":"2024-09-16T15:42:29.423662+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-16T15:42:39.287764+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9e7c1955-a607-4671-9be7-f2ef760ceeaa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":71,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6800","nonce":3882044969},{"type":"v1","addr":"172.21.15.123:6801","nonce":3882044969}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6802","nonce":3882044969},{"type":"v1","addr":"172.21.15.123:6803","nonce":3882044969}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6806","nonce":3882044969},{"type":"v1","addr":"172.21.15.123:6807","nonce":3882044969}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6804","nonce":3882044969},{"type":"v1","addr":"172.21.15.123:6805","nonce":3882044969}]},"public_addr":"172.21.15.123:6801/3882044969","cluster_addr":"172.21.15.123:6803/3882044969","heartbeat_back_addr":"172.21.15.123:6807/3882044969","heartbeat_front_addr":"172.21.15.123:6805/3882044969","state":["exists","up"]},{"osd":1,"uuid":"c540cfde-0977-4742-9b17-fd634668e010","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6816","nonce":4037001850},{"type":"v1","addr":"172.21.15.123:6817","nonce":4037001850}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6818","nonce":4037001850},{"type":"v1","addr":"172.21.15.123:6819","nonce":4037001850}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6822","nonce":4037001850},{"type":"v1","addr":"172.21.15.123:6823","nonce":4037001850}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6820","nonce":4037001850},{"type":"v1","addr":"172.21.15.123:6821","nonce":4037001850}]},"public_addr":"172.21.15.123:6817/4037001850","cluster_addr":"172.21.15.123:6819/4037001850","heartbeat_back_addr":"172.21.15.123:6823/4037001850","heartbeat_front_addr":"172.21.15.123:6821/4037001850","state":["exists","up"]},{"osd":2,"uuid":"9864a496-f169-486b-95c4-1d59162793a7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6824","nonce":1332143088},{"type":"v1","addr":"172.21.15.123:6825","nonce":1332143088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6826","nonce":1332143088},{"type":"v1","addr":"172.21.15.123:6827","nonce":1332143088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6830","nonce":1332143088},{"type":"v1","addr":"172.21.15.123:6831","nonce":1332143088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6828","nonce":1332143088},{"type":"v1","addr":"172.21.15.123:6829","nonce":1332143088}]},"public_addr":"172.21.15.123:6825/1332143088","cluster_addr":"172.21.15.123:6827/1332143088","heartbeat_back_addr":"172.21.15.123:6831/1332143088","heartbeat_front_addr":"172.21.15.123:6829/1332143088","state":["exists","up"]},{"osd":3,"uuid":"c6d1f817-f074-4325-95e1-578eba13edda","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6808","nonce":988586998},{"type":"v1","addr":"172.21.15.123:6809","nonce":988586998}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6810","nonce":988586998},{"type":"v1","addr":"172.21.15.123:6811","nonce":988586998}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6814","nonce":988586998},{"type":"v1","addr":"172.21.15.123:6815","nonce":988586998}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6812","nonce":988586998},{"type":"v1","addr":"172.21.15.123:6813","nonce":988586998}]},"public_addr":"172.21.15.123:6809/988586998","cluster_addr":"172.21.15.123:6811/988586998","heartbeat_back_addr":"172.21.15.123:6815/988586998","heartbeat_front_addr":"172.21.15.123:6813/988586998","state":["exists","up"]},{"osd":4,"uuid":"200a8df4-50ef-40a5-b48a-58981269c086","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6816","nonce":3387813890},{"type":"v1","addr":"172.21.15.184:6817","nonce":3387813890}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6818","nonce":3387813890},{"type":"v1","addr":"172.21.15.184:6819","nonce":3387813890}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6822","nonce":3387813890},{"type":"v1","addr":"172.21.15.184:6823","nonce":3387813890}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6820","nonce":3387813890},{"type":"v1","addr":"172.21.15.184:6821","nonce":3387813890}]},"public_addr":"172.21.15.184:6817/3387813890","cluster_addr":"172.21.15.184:6819/3387813890","heartbeat_back_addr":"172.21.15.184:6823/3387813890","heartbeat_front_addr":"172.21.15.184:6821/3387813890","state":["exists","up"]},{"osd":5,"uuid":"32d5a900-da3d-412c-bc19-13861a2368d8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6808","nonce":2680916017},{"type":"v1","addr":"172.21.15.184:6809","nonce":2680916017}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6810","nonce":2680916017},{"type":"v1","addr":"172.21.15.184:6811","nonce":2680916017}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6814","nonce":2680916017},{"type":"v1","addr":"172.21.15.184:6815","nonce":2680916017}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6812","nonce":2680916017},{"type":"v1","addr":"172.21.15.184:6813","nonce":2680916017}]},"public_addr":"172.21.15.184:6809/2680916017","cluster_addr":"172.21.15.184:6811/2680916017","heartbeat_back_addr":"172.21.15.184:6815/2680916017","heartbeat_front_addr":"172.21.15.184:6813/2680916017","state":["exists","up"]},{"osd":6,"uuid":"cde6f39e-aa1b-4260-ac19-774ee6aaca72","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6824","nonce":2183120457},{"type":"v1","addr":"172.21.15.184:6825","nonce":2183120457}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6826","nonce":2183120457},{"type":"v1","addr":"172.21.15.184:6827","nonce":2183120457}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6830","nonce":2183120457},{"type":"v1","addr":"172.21.15.184:6831","nonce":2183120457}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6828","nonce":2183120457},{"type":"v1","addr":"172.21.15.184:6829","nonce":2183120457}]},"public_addr":"172.21.15.184:6825/2183120457","cluster_addr":"172.21.15.184:6827/2183120457","heartbeat_back_addr":"172.21.15.184:6831/2183120457","heartbeat_front_addr":"172.21.15.184:6829/2183120457","state":["exists","up"]},{"osd":7,"uuid":"4f381778-eee1-47c6-804f-df0e6b33e313","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":71,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6800","nonce":4102957285},{"type":"v1","addr":"172.21.15.184:6801","nonce":4102957285}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6802","nonce":4102957285},{"type":"v1","addr":"172.21.15.184:6803","nonce":4102957285}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6806","nonce":4102957285},{"type":"v1","addr":"172.21.15.184:6807","nonce":4102957285}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6804","nonce":4102957285},{"type":"v1","addr":"172.21.15.184:6805","nonce":4102957285}]},"public_addr":"172.21.15.184:6801/4102957285","cluster_addr":"172.21.15.184:6803/4102957285","heartbeat_back_addr":"172.21.15.184:6807/4102957285","heartbeat_front_addr":"172.21.15.184:6805/4102957285","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:35.972210+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.222007+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.198617+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.385663+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.358935+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.156066+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.752973+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:35.682846+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.184:6836/3137795986":"2024-09-17T15:47:37.155867+0000","172.21.15.184:6834/3137795986":"2024-09-17T15:47:37.155867+0000","172.21.15.184:0/3070886766":"2024-09-17T15:47:34.024879+0000","172.21.15.184:0/1402401038":"2024-09-17T15:47:34.024879+0000","172.21.15.184:0/1715342118":"2024-09-17T15:47:34.024879+0000","172.21.15.184:0/2291367472":"2024-09-17T15:47:34.024879+0000","172.21.15.184:0/654280269":"2024-09-17T15:47:34.024879+0000","172.21.15.184:0/3204392673":"2024-09-17T15:47:34.024879+0000","172.21.15.184:0/78639736":"2024-09-17T15:47:34.024879+0000","172.21.15.184:0/2406729396":"2024-09-17T15:47:34.024879+0000","172.21.15.184:6832/3114346423":"2024-09-17T15:47:34.024879+0000","172.21.15.184:6833/3114346423":"2024-09-17T15:47:34.024879+0000","172.21.15.184:0/2916845314":"2024-09-17T15:47:34.024879+0000","172.21.15.184:0/140936988":"2024-09-17T15:47:34.024879+0000","172.21.15.184:0/1023177165":"2024-09-17T15:47:34.024879+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T15:47:41.524 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:41.520+0000 7f3bde96b640 1 -- 172.21.15.123:0/2379224540 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f3bb8072fe0 msgr2=0x7f3bb80754a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:47:41.524 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:41.520+0000 7f3bde96b640 1 --2- 172.21.15.123:0/2379224540 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f3bb8072fe0 0x7f3bb80754a0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f3bd0005870 tx=0x7f3bd0005db0 comp rx=0 tx=0).stop 2024-09-16T15:47:41.524 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:41.520+0000 7f3bde96b640 1 -- 172.21.15.123:0/2379224540 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3bd8151b10 msgr2=0x7f3bd80c2840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:47:41.525 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:41.520+0000 7f3bde96b640 1 --2- 172.21.15.123:0/2379224540 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3bd8151b10 0x7f3bd80c2840 secure :-1 s=READY pgs=892 cs=0 l=1 rev1=1 crypto rx=0x7f3bc8009bd0 tx=0x7f3bc8003f60 comp rx=0 tx=0).stop 2024-09-16T15:47:41.525 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:41.520+0000 7f3bde96b640 1 -- 172.21.15.123:0/2379224540 shutdown_connections 2024-09-16T15:47:41.525 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:41.520+0000 7f3bde96b640 1 --2- 172.21.15.123:0/2379224540 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f3bb8072fe0 0x7f3bb80754a0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:41.525 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:41.520+0000 7f3bde96b640 1 --2- 172.21.15.123:0/2379224540 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3bd8151b10 0x7f3bd80c2840 unknown :-1 s=CLOSED pgs=892 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:41.525 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:41.520+0000 7f3bde96b640 1 --2- 172.21.15.123:0/2379224540 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3bd80cb760 0x7f3bd80c2300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:41.525 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:41.520+0000 7f3bde96b640 1 --2- 172.21.15.123:0/2379224540 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f3bd8073d10 0x7f3bd8154620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:41.525 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:41.520+0000 7f3bde96b640 1 -- 172.21.15.123:0/2379224540 >> 172.21.15.123:0/2379224540 conn(0x7f3bd80cd5d0 msgr2=0x7f3bd8150870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:47:41.525 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:41.520+0000 7f3bde96b640 1 -- 172.21.15.123:0/2379224540 shutdown_connections 2024-09-16T15:47:41.525 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:41.520+0000 7f3bde96b640 1 -- 172.21.15.123:0/2379224540 wait complete. 2024-09-16T15:47:41.539 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-16T15:42:39.287764+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-09-16T15:47:41.540 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-09-16T15:47:41.656 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:41.652+0000 7fa310192640 1 -- 172.21.15.123:0/3024211509 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa30815aa00 msgr2=0x7fa30815e5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:47:41.656 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:41.652+0000 7fa310192640 1 --2- 172.21.15.123:0/3024211509 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa30815aa00 0x7fa30815e5d0 secure :-1 s=READY pgs=894 cs=0 l=1 rev1=1 crypto rx=0x7fa3080cf850 tx=0x7fa2f801c8c0 comp rx=0 tx=0).stop 2024-09-16T15:47:41.656 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:41.652+0000 7fa310192640 1 -- 172.21.15.123:0/3024211509 shutdown_connections 2024-09-16T15:47:41.656 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:41.652+0000 7fa310192640 1 --2- 172.21.15.123:0/3024211509 >> v2:172.21.15.184:3300/0 conn(0x7fa308072720 0x7fa3080cabc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:41.656 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:41.652+0000 7fa310192640 1 --2- 172.21.15.123:0/3024211509 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa30815aa00 0x7fa30815e5d0 unknown :-1 s=CLOSED pgs=894 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:41.657 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:41.652+0000 7fa310192640 1 -- 172.21.15.123:0/3024211509 >> 172.21.15.123:0/3024211509 conn(0x7fa3080b1710 msgr2=0x7fa3080b1b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:47:41.657 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:41.652+0000 7fa310192640 1 -- 172.21.15.123:0/3024211509 shutdown_connections 2024-09-16T15:47:41.657 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:41.652+0000 7fa310192640 1 -- 172.21.15.123:0/3024211509 wait complete. 2024-09-16T15:47:41.657 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:41.652+0000 7fa310192640 1 Processor -- start 2024-09-16T15:47:41.657 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:41.652+0000 7fa310192640 1 -- start start 2024-09-16T15:47:41.659 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:41.656+0000 7fa310192640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa308072720 0x7fa3081f6420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:47:41.659 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:41.656+0000 7fa310192640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa3080cc3c0 0x7fa3081f6da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:47:41.659 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:41.656+0000 7fa310192640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fa30815aa00 0x7fa308154df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:47:41.660 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:41.656+0000 7fa310192640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fa308168300 con 0x7fa308072720 2024-09-16T15:47:41.660 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:41.656+0000 7fa310192640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fa308168180 con 0x7fa3080cc3c0 2024-09-16T15:47:41.660 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:41.656+0000 7fa310192640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fa308168480 con 0x7fa30815aa00 2024-09-16T15:47:41.660 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:41.656+0000 7fa30df2e640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa308072720 0x7fa3081f6420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:47:41.660 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:41.656+0000 7fa30df2e640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa308072720 0x7fa3081f6420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:53182/0 (socket says 172.21.15.123:53182) 2024-09-16T15:47:41.660 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:41.656+0000 7fa30df2e640 1 -- 172.21.15.123:0/809723017 learned_addr learned my addr 172.21.15.123:0/809723017 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:47:41.660 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:41.656+0000 7fa30d72d640 1 --2- 172.21.15.123:0/809723017 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa3080cc3c0 0x7fa3081f6da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:47:41.660 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:41.656+0000 7fa30e72f640 1 --2- 172.21.15.123:0/809723017 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fa30815aa00 0x7fa308154df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:47:41.660 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:41.656+0000 7fa30df2e640 1 -- 172.21.15.123:0/809723017 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fa30815aa00 msgr2=0x7fa308154df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:47:41.660 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:41.656+0000 7fa30df2e640 1 --2- 172.21.15.123:0/809723017 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fa30815aa00 0x7fa308154df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:41.660 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:41.656+0000 7fa30df2e640 1 -- 172.21.15.123:0/809723017 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa3080cc3c0 msgr2=0x7fa3081f6da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:47:41.660 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:41.656+0000 7fa30df2e640 1 --2- 172.21.15.123:0/809723017 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa3080cc3c0 0x7fa3081f6da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:41.660 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:41.656+0000 7fa30df2e640 1 -- 172.21.15.123:0/809723017 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa308155fc0 con 0x7fa308072720 2024-09-16T15:47:41.661 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:41.656+0000 7fa30d72d640 1 --2- 172.21.15.123:0/809723017 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa3080cc3c0 0x7fa3081f6da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T15:47:41.661 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:41.656+0000 7fa30df2e640 1 --2- 172.21.15.123:0/809723017 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa308072720 0x7fa3081f6420 secure :-1 s=READY pgs=895 cs=0 l=1 rev1=1 crypto rx=0x7fa2f8000c10 tx=0x7fa2f80040c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:47:41.661 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:41.656+0000 7fa2fe7fc640 1 -- 172.21.15.123:0/809723017 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2f80b2040 con 0x7fa308072720 2024-09-16T15:47:41.661 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:41.656+0000 7fa2fe7fc640 1 -- 172.21.15.123:0/809723017 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fa2f8005020 con 0x7fa308072720 2024-09-16T15:47:41.661 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:41.656+0000 7fa2fe7fc640 1 -- 172.21.15.123:0/809723017 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2f8005470 con 0x7fa308072720 2024-09-16T15:47:41.661 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:41.656+0000 7fa310192640 1 -- 172.21.15.123:0/809723017 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa308155720 con 0x7fa308072720 2024-09-16T15:47:41.661 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:41.656+0000 7fa310192640 1 -- 172.21.15.123:0/809723017 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa308155ce0 con 0x7fa308072720 2024-09-16T15:47:41.662 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:41.656+0000 7fa310192640 1 -- 172.21.15.123:0/809723017 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa3080cd440 con 0x7fa308072720 2024-09-16T15:47:41.662 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:41.656+0000 7fa2fe7fc640 1 -- 172.21.15.123:0/809723017 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 16) ==== 97182+0+0 (secure 0 0 0) 0x7fa2f80046d0 con 0x7fa308072720 2024-09-16T15:47:41.662 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:41.656+0000 7fa2fe7fc640 1 --2- 172.21.15.123:0/809723017 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fa2ec072fe0 0x7fa2ec0754a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:47:41.662 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:41.656+0000 7fa2fe7fc640 1 -- 172.21.15.123:0/809723017 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(76..76 src has 1..76) ==== 5502+0+0 (secure 0 0 0) 0x7fa2f812e1c0 con 0x7fa308072720 2024-09-16T15:47:41.662 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:41.656+0000 7fa30d72d640 1 --2- 172.21.15.123:0/809723017 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fa2ec072fe0 0x7fa2ec0754a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:47:41.662 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:41.660+0000 7fa30d72d640 1 --2- 172.21.15.123:0/809723017 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fa2ec072fe0 0x7fa2ec0754a0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fa308165930 tx=0x7fa304008040 comp rx=0 tx=0).ready entity=mgr.4150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:47:41.665 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:41.660+0000 7fa2fe7fc640 1 -- 172.21.15.123:0/809723017 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7fa2f80164b0 con 0x7fa308072720 2024-09-16T15:47:41.835 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:41.832+0000 7fa310192640 1 -- 172.21.15.123:0/809723017 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fa308070430 con 0x7fa308072720 2024-09-16T15:47:41.836 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:41.832+0000 7fa2fe7fc640 1 -- 172.21.15.123:0/809723017 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v76) ==== 93+0+10 (secure 0 0 0) 0x7fa2f80f9350 con 0x7fa308072720 2024-09-16T15:47:41.836 INFO:teuthology.orchestra.run.smithi123.stdout:pg_num: 1 2024-09-16T15:47:41.838 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:41.832+0000 7fa310192640 1 -- 172.21.15.123:0/809723017 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fa2ec072fe0 msgr2=0x7fa2ec0754a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:47:41.839 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:41.832+0000 7fa310192640 1 --2- 172.21.15.123:0/809723017 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fa2ec072fe0 0x7fa2ec0754a0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fa308165930 tx=0x7fa304008040 comp rx=0 tx=0).stop 2024-09-16T15:47:41.839 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:41.832+0000 7fa310192640 1 -- 172.21.15.123:0/809723017 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa308072720 msgr2=0x7fa3081f6420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:47:41.839 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:41.832+0000 7fa310192640 1 --2- 172.21.15.123:0/809723017 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa308072720 0x7fa3081f6420 secure :-1 s=READY pgs=895 cs=0 l=1 rev1=1 crypto rx=0x7fa2f8000c10 tx=0x7fa2f80040c0 comp rx=0 tx=0).stop 2024-09-16T15:47:41.839 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:41.836+0000 7fa310192640 1 -- 172.21.15.123:0/809723017 shutdown_connections 2024-09-16T15:47:41.839 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:41.836+0000 7fa310192640 1 --2- 172.21.15.123:0/809723017 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fa2ec072fe0 0x7fa2ec0754a0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:41.839 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:41.836+0000 7fa310192640 1 --2- 172.21.15.123:0/809723017 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fa30815aa00 0x7fa308154df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:41.839 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:41.836+0000 7fa310192640 1 --2- 172.21.15.123:0/809723017 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa3080cc3c0 0x7fa3081f6da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:41.839 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:41.836+0000 7fa310192640 1 --2- 172.21.15.123:0/809723017 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa308072720 0x7fa3081f6420 unknown :-1 s=CLOSED pgs=895 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:41.839 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:41.836+0000 7fa310192640 1 -- 172.21.15.123:0/809723017 >> 172.21.15.123:0/809723017 conn(0x7fa3080b1710 msgr2=0x7fa3080af760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:47:41.839 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:41.836+0000 7fa310192640 1 -- 172.21.15.123:0/809723017 shutdown_connections 2024-09-16T15:47:41.839 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:41.836+0000 7fa310192640 1 -- 172.21.15.123:0/809723017 wait complete. 2024-09-16T15:47:41.854 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_volumes.TestMisc.test_names_can_only_be_goodchars' 2024-09-16T15:47:41.968 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:41.964+0000 7fb61f4b8640 1 -- 172.21.15.123:0/811700151 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb6040063e0 msgr2=0x7fb60400cae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:47:41.968 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:41.964+0000 7fb61f4b8640 1 --2- 172.21.15.123:0/811700151 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb6040063e0 0x7fb60400cae0 secure :-1 s=READY pgs=897 cs=0 l=1 rev1=1 crypto rx=0x7fb61000c640 tx=0x7fb61001faa0 comp rx=0 tx=0).stop 2024-09-16T15:47:41.968 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:41.964+0000 7fb61da55640 1 -- 172.21.15.123:0/811700151 reap_dead start 2024-09-16T15:47:41.968 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:41.964+0000 7fb61f4b8640 1 -- 172.21.15.123:0/811700151 shutdown_connections 2024-09-16T15:47:41.968 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:41.964+0000 7fb61f4b8640 1 -- 172.21.15.123:0/811700151 >> 172.21.15.123:0/811700151 conn(0x7fb6180cfb10 msgr2=0x7fb6180cff20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:47:41.968 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:41.964+0000 7fb61f4b8640 1 -- 172.21.15.123:0/811700151 shutdown_connections 2024-09-16T15:47:41.968 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:41.964+0000 7fb61f4b8640 1 -- 172.21.15.123:0/811700151 wait complete. 2024-09-16T15:47:41.969 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:41.964+0000 7fb61f4b8640 1 Processor -- start 2024-09-16T15:47:41.969 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:41.964+0000 7fb61f4b8640 1 -- start start 2024-09-16T15:47:41.971 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:41.964+0000 7fb61f4b8640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb6040063e0 0x7fb6180c9a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:47:41.971 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:41.964+0000 7fb61f4b8640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fb61814e450 0x7fb6181f4250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:47:41.971 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:41.964+0000 7fb61f4b8640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb6180c6be0 0x7fb6180c5790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:47:41.971 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:41.964+0000 7fb61f4b8640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fb61806e7d0 con 0x7fb6040063e0 2024-09-16T15:47:41.971 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:41.964+0000 7fb61f4b8640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fb61806e650 con 0x7fb6180c6be0 2024-09-16T15:47:41.971 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:41.964+0000 7fb61f4b8640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fb61806e950 con 0x7fb61814e450 2024-09-16T15:47:41.971 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:41.964+0000 7fb61d254640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb6040063e0 0x7fb6180c9a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:47:41.971 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:41.964+0000 7fb61d254640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb6040063e0 0x7fb6180c9a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:53204/0 (socket says 172.21.15.123:53204) 2024-09-16T15:47:41.971 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:41.964+0000 7fb61d254640 1 -- 172.21.15.123:0/2790102964 learned_addr learned my addr 172.21.15.123:0/2790102964 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:47:41.971 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:41.964+0000 7fb61da55640 1 --2- 172.21.15.123:0/2790102964 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb6180c6be0 0x7fb6180c5790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:47:41.971 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:41.964+0000 7fb61ca53640 1 --2- 172.21.15.123:0/2790102964 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fb61814e450 0x7fb6181f4250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:47:41.972 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:41.964+0000 7fb61d254640 1 -- 172.21.15.123:0/2790102964 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fb61814e450 msgr2=0x7fb6181f4250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:47:41.972 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:41.964+0000 7fb61d254640 1 --2- 172.21.15.123:0/2790102964 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fb61814e450 0x7fb6181f4250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:41.972 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:41.964+0000 7fb61d254640 1 -- 172.21.15.123:0/2790102964 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb6180c6be0 msgr2=0x7fb6180c5790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:47:41.972 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:41.964+0000 7fb61d254640 1 --2- 172.21.15.123:0/2790102964 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb6180c6be0 0x7fb6180c5790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:41.972 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:41.964+0000 7fb61d254640 1 -- 172.21.15.123:0/2790102964 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb6180c89f0 con 0x7fb6040063e0 2024-09-16T15:47:41.972 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:41.964+0000 7fb61da55640 1 --2- 172.21.15.123:0/2790102964 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb6180c6be0 0x7fb6180c5790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T15:47:41.972 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:41.964+0000 7fb61ca53640 1 --2- 172.21.15.123:0/2790102964 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fb61814e450 0x7fb6181f4250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T15:47:41.972 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:41.964+0000 7fb61d254640 1 --2- 172.21.15.123:0/2790102964 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb6040063e0 0x7fb6180c9a90 secure :-1 s=READY pgs=898 cs=0 l=1 rev1=1 crypto rx=0x7fb614005bd0 tx=0x7fb61400da90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:47:41.972 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:41.968+0000 7fb60dffb640 1 -- 172.21.15.123:0/2790102964 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb614004230 con 0x7fb6040063e0 2024-09-16T15:47:41.972 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:41.968+0000 7fb60dffb640 1 -- 172.21.15.123:0/2790102964 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fb6140044e0 con 0x7fb6040063e0 2024-09-16T15:47:41.972 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:41.968+0000 7fb61f4b8640 1 -- 172.21.15.123:0/2790102964 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb6180c8cd0 con 0x7fb6040063e0 2024-09-16T15:47:41.972 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:41.968+0000 7fb60dffb640 1 -- 172.21.15.123:0/2790102964 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb614009950 con 0x7fb6040063e0 2024-09-16T15:47:41.972 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:41.968+0000 7fb61f4b8640 1 -- 172.21.15.123:0/2790102964 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb6180c9290 con 0x7fb6040063e0 2024-09-16T15:47:41.973 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:41.968+0000 7fb61f4b8640 1 -- 172.21.15.123:0/2790102964 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb61806eee0 con 0x7fb6040063e0 2024-09-16T15:47:41.973 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:41.968+0000 7fb60dffb640 1 -- 172.21.15.123:0/2790102964 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 16) ==== 97182+0+0 (secure 0 0 0) 0x7fb614009af0 con 0x7fb6040063e0 2024-09-16T15:47:41.973 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:41.968+0000 7fb60dffb640 1 --2- 172.21.15.123:0/2790102964 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fb5fc072fe0 0x7fb5fc0754a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:47:41.973 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:41.968+0000 7fb60dffb640 1 -- 172.21.15.123:0/2790102964 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(76..76 src has 1..76) ==== 5502+0+0 (secure 0 0 0) 0x7fb614094450 con 0x7fb6040063e0 2024-09-16T15:47:41.973 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:41.968+0000 7fb61ca53640 1 --2- 172.21.15.123:0/2790102964 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fb5fc072fe0 0x7fb5fc0754a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:47:41.974 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:41.968+0000 7fb61ca53640 1 --2- 172.21.15.123:0/2790102964 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fb5fc072fe0 0x7fb5fc0754a0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fb6181e6820 tx=0x7fb608006ba0 comp rx=0 tx=0).ready entity=mgr.4150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:47:41.976 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:41.972+0000 7fb60dffb640 1 -- 172.21.15.123:0/2790102964 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7fb614017020 con 0x7fb6040063e0 2024-09-16T15:47:42.195 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:42.192+0000 7fb61f4b8640 1 -- 172.21.15.123:0/2790102964 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_volumes.TestMisc.test_names_can_only_be_goodchars"]} v 0) -- 0x7fb6180b57b0 con 0x7fb6040063e0 2024-09-16T15:47:43.203 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:43.200+0000 7fb60dffb640 1 -- 172.21.15.123:0/2790102964 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_volumes.TestMisc.test_names_can_only_be_goodchars"]}]=0 v250) ==== 149+0+0 (secure 0 0 0) 0x7fb61405f6c0 con 0x7fb6040063e0 2024-09-16T15:47:43.206 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:43.200+0000 7fb61f4b8640 1 -- 172.21.15.123:0/2790102964 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fb5fc072fe0 msgr2=0x7fb5fc0754a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:47:43.206 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:43.200+0000 7fb61f4b8640 1 --2- 172.21.15.123:0/2790102964 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fb5fc072fe0 0x7fb5fc0754a0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fb6181e6820 tx=0x7fb608006ba0 comp rx=0 tx=0).stop 2024-09-16T15:47:43.206 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:43.200+0000 7fb61f4b8640 1 -- 172.21.15.123:0/2790102964 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb6040063e0 msgr2=0x7fb6180c9a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:47:43.206 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:43.200+0000 7fb61f4b8640 1 --2- 172.21.15.123:0/2790102964 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb6040063e0 0x7fb6180c9a90 secure :-1 s=READY pgs=898 cs=0 l=1 rev1=1 crypto rx=0x7fb614005bd0 tx=0x7fb61400da90 comp rx=0 tx=0).stop 2024-09-16T15:47:43.206 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:43.200+0000 7fb61f4b8640 1 -- 172.21.15.123:0/2790102964 shutdown_connections 2024-09-16T15:47:43.206 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:43.200+0000 7fb61f4b8640 1 --2- 172.21.15.123:0/2790102964 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fb5fc072fe0 0x7fb5fc0754a0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:43.206 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:43.200+0000 7fb61f4b8640 1 --2- 172.21.15.123:0/2790102964 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb6180c6be0 0x7fb6180c5790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:43.206 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:43.200+0000 7fb61f4b8640 1 --2- 172.21.15.123:0/2790102964 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fb61814e450 0x7fb6181f4250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:43.206 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:43.200+0000 7fb61f4b8640 1 --2- 172.21.15.123:0/2790102964 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb6040063e0 0x7fb6180c9a90 unknown :-1 s=CLOSED pgs=898 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:43.206 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:43.200+0000 7fb61f4b8640 1 -- 172.21.15.123:0/2790102964 >> 172.21.15.123:0/2790102964 conn(0x7fb6180cfb10 msgr2=0x7fb618072980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:47:43.206 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:43.200+0000 7fb61f4b8640 1 -- 172.21.15.123:0/2790102964 shutdown_connections 2024-09-16T15:47:43.206 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:43.200+0000 7fb61f4b8640 1 -- 172.21.15.123:0/2790102964 wait complete. 2024-09-16T15:47:43.225 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config dump 2024-09-16T15:47:43.351 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:43.348+0000 7f5c88555640 1 -- 172.21.15.123:0/2992730566 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5c8015bca0 msgr2=0x7f5c800d0630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:47:43.351 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:43.348+0000 7f5c88555640 1 --2- 172.21.15.123:0/2992730566 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5c8015bca0 0x7f5c800d0630 secure :-1 s=READY pgs=900 cs=0 l=1 rev1=1 crypto rx=0x7f5c8006c970 tx=0x7f5c7c01c740 comp rx=0 tx=0).stop 2024-09-16T15:47:43.351 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:43.348+0000 7f5c88555640 1 -- 172.21.15.123:0/2992730566 shutdown_connections 2024-09-16T15:47:43.351 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:43.348+0000 7f5c88555640 1 --2- 172.21.15.123:0/2992730566 >> v2:172.21.15.184:3300/0 conn(0x7f5c80074600 0x7f5c8006e750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:43.352 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:43.348+0000 7f5c88555640 1 --2- 172.21.15.123:0/2992730566 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5c8015bca0 0x7f5c800d0630 unknown :-1 s=CLOSED pgs=900 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:43.352 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:43.348+0000 7f5c88555640 1 -- 172.21.15.123:0/2992730566 >> 172.21.15.123:0/2992730566 conn(0x7f5c800cfb10 msgr2=0x7f5c800cff20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:47:43.352 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:43.348+0000 7f5c88555640 1 -- 172.21.15.123:0/2992730566 shutdown_connections 2024-09-16T15:47:43.352 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:43.348+0000 7f5c88555640 1 -- 172.21.15.123:0/2992730566 wait complete. 2024-09-16T15:47:43.352 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:43.348+0000 7f5c88555640 1 Processor -- start 2024-09-16T15:47:43.352 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:43.348+0000 7f5c88555640 1 -- start start 2024-09-16T15:47:43.354 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:43.348+0000 7f5c88555640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f5c800726c0 0x7f5c801fa610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:47:43.354 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:43.348+0000 7f5c88555640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5c80074600 0x7f5c800cec40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:47:43.355 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:43.348+0000 7f5c88555640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5c8015bca0 0x7f5c800cf180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:47:43.355 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:43.348+0000 7f5c88555640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f5c80168bd0 con 0x7f5c8015bca0 2024-09-16T15:47:43.355 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:43.348+0000 7f5c88555640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f5c80168a50 con 0x7f5c80074600 2024-09-16T15:47:43.355 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:43.348+0000 7f5c88555640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f5c80168d50 con 0x7f5c800726c0 2024-09-16T15:47:43.355 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:43.348+0000 7f5c86af2640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5c8015bca0 0x7f5c800cf180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:47:43.355 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:43.348+0000 7f5c86af2640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5c8015bca0 0x7f5c800cf180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:48480/0 (socket says 172.21.15.123:48480) 2024-09-16T15:47:43.355 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:43.348+0000 7f5c86af2640 1 -- 172.21.15.123:0/2167457249 learned_addr learned my addr 172.21.15.123:0/2167457249 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:47:43.355 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:43.348+0000 7f5c862f1640 1 --2- 172.21.15.123:0/2167457249 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f5c800726c0 0x7f5c801fa610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:47:43.355 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:43.348+0000 7f5c85af0640 1 --2- 172.21.15.123:0/2167457249 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5c80074600 0x7f5c800cec40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:47:43.356 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:43.348+0000 7f5c86af2640 1 -- 172.21.15.123:0/2167457249 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f5c800726c0 msgr2=0x7f5c801fa610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:47:43.356 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:43.348+0000 7f5c86af2640 1 --2- 172.21.15.123:0/2167457249 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f5c800726c0 0x7f5c801fa610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:43.356 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:43.348+0000 7f5c86af2640 1 -- 172.21.15.123:0/2167457249 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5c80074600 msgr2=0x7f5c800cec40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:47:43.356 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:43.348+0000 7f5c86af2640 1 --2- 172.21.15.123:0/2167457249 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5c80074600 0x7f5c800cec40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:43.356 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:43.348+0000 7f5c86af2640 1 -- 172.21.15.123:0/2167457249 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5c801fab50 con 0x7f5c8015bca0 2024-09-16T15:47:43.356 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:43.348+0000 7f5c86af2640 1 --2- 172.21.15.123:0/2167457249 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5c8015bca0 0x7f5c800cf180 secure :-1 s=READY pgs=901 cs=0 l=1 rev1=1 crypto rx=0x7f5c7800cce0 tx=0x7f5c7800a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:47:43.356 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:43.348+0000 7f5c76ffd640 1 -- 172.21.15.123:0/2167457249 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c78004b30 con 0x7f5c8015bca0 2024-09-16T15:47:43.356 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:43.348+0000 7f5c76ffd640 1 -- 172.21.15.123:0/2167457249 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f5c78004dc0 con 0x7f5c8015bca0 2024-09-16T15:47:43.356 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:43.348+0000 7f5c88555640 1 -- 172.21.15.123:0/2167457249 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5c801545e0 con 0x7f5c8015bca0 2024-09-16T15:47:43.356 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:43.348+0000 7f5c76ffd640 1 -- 172.21.15.123:0/2167457249 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c78005280 con 0x7f5c8015bca0 2024-09-16T15:47:43.356 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:43.348+0000 7f5c88555640 1 -- 172.21.15.123:0/2167457249 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5c80154ba0 con 0x7f5c8015bca0 2024-09-16T15:47:43.357 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:43.352+0000 7f5c88555640 1 -- 172.21.15.123:0/2167457249 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5c8006e750 con 0x7f5c8015bca0 2024-09-16T15:47:43.357 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:43.352+0000 7f5c76ffd640 1 -- 172.21.15.123:0/2167457249 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 16) ==== 97182+0+0 (secure 0 0 0) 0x7f5c78005420 con 0x7f5c8015bca0 2024-09-16T15:47:43.357 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:43.352+0000 7f5c76ffd640 1 --2- 172.21.15.123:0/2167457249 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f5c64072fe0 0x7f5c640754a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:47:43.357 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:43.352+0000 7f5c862f1640 1 --2- 172.21.15.123:0/2167457249 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f5c64072fe0 0x7f5c640754a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:47:43.358 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:43.352+0000 7f5c76ffd640 1 -- 172.21.15.123:0/2167457249 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(76..76 src has 1..76) ==== 5502+0+0 (secure 0 0 0) 0x7f5c78094720 con 0x7f5c8015bca0 2024-09-16T15:47:43.358 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:43.352+0000 7f5c862f1640 1 --2- 172.21.15.123:0/2167457249 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f5c64072fe0 0x7f5c640754a0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f5c7c0af000 tx=0x7f5c7c01af80 comp rx=0 tx=0).ready entity=mgr.4150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:47:43.360 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:43.356+0000 7f5c76ffd640 1 -- 172.21.15.123:0/2167457249 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f5c801fab50 con 0x7f5c8015bca0 2024-09-16T15:47:43.532 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:43.528+0000 7f5c88555640 1 -- 172.21.15.123:0/2167457249 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "config dump"} v 0) -- 0x7f5c8015f630 con 0x7f5c8015bca0 2024-09-16T15:47:43.532 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:43.528+0000 7f5c76ffd640 1 -- 172.21.15.123:0/2167457249 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "config dump"}]=0 v35) ==== 59+0+1998 (secure 0 0 0) 0x7f5c78064e20 con 0x7f5c8015bca0 2024-09-16T15:47:43.533 INFO:teuthology.orchestra.run.smithi123.stdout:WHO MASK LEVEL OPTION VALUE RO 2024-09-16T15:47:43.533 INFO:teuthology.orchestra.run.smithi123.stdout:mon advanced mon_op_complaint_time 120 2024-09-16T15:47:43.533 INFO:teuthology.orchestra.run.smithi123.stdout:mon dev mon_warn_on_pool_no_app false 2024-09-16T15:47:43.533 INFO:teuthology.orchestra.run.smithi123.stdout:osd advanced osd_op_complaint_time 180.000000 2024-09-16T15:47:43.533 INFO:teuthology.orchestra.run.smithi123.stdout:osd.0 basic osd_mclock_max_capacity_iops_ssd 4805.300677 2024-09-16T15:47:43.533 INFO:teuthology.orchestra.run.smithi123.stdout:osd.1 basic osd_mclock_max_capacity_iops_ssd 4806.240607 2024-09-16T15:47:43.533 INFO:teuthology.orchestra.run.smithi123.stdout:osd.2 basic osd_mclock_max_capacity_iops_ssd 4046.441686 2024-09-16T15:47:43.533 INFO:teuthology.orchestra.run.smithi123.stdout:osd.3 basic osd_mclock_max_capacity_iops_ssd 6882.576123 2024-09-16T15:47:43.533 INFO:teuthology.orchestra.run.smithi123.stdout:osd.4 basic osd_mclock_max_capacity_iops_ssd 6108.444971 2024-09-16T15:47:43.533 INFO:teuthology.orchestra.run.smithi123.stdout:osd.5 basic osd_mclock_max_capacity_iops_ssd 4995.096306 2024-09-16T15:47:43.533 INFO:teuthology.orchestra.run.smithi123.stdout:osd.6 basic osd_mclock_max_capacity_iops_ssd 6237.822095 2024-09-16T15:47:43.533 INFO:teuthology.orchestra.run.smithi123.stdout:osd.7 basic osd_mclock_max_capacity_iops_ssd 5202.385138 2024-09-16T15:47:43.533 INFO:teuthology.orchestra.run.smithi123.stdout:mds advanced debug_mds 20/20 2024-09-16T15:47:43.534 INFO:teuthology.orchestra.run.smithi123.stdout:mds advanced debug_mds_balancer 20/20 2024-09-16T15:47:43.534 INFO:teuthology.orchestra.run.smithi123.stdout:mds advanced debug_ms 2/2 2024-09-16T15:47:43.534 INFO:teuthology.orchestra.run.smithi123.stdout:mds dev mds_debug_frag true 2024-09-16T15:47:43.534 INFO:teuthology.orchestra.run.smithi123.stdout:mds dev mds_debug_scatterstat true 2024-09-16T15:47:43.534 INFO:teuthology.orchestra.run.smithi123.stdout:mds advanced mds_op_complaint_time 180.000000 2024-09-16T15:47:43.534 INFO:teuthology.orchestra.run.smithi123.stdout:mds dev mds_verify_scatter true 2024-09-16T15:47:43.534 INFO:teuthology.orchestra.run.smithi123.stdout:mds advanced osd_op_complaint_time 180.000000 2024-09-16T15:47:43.534 INFO:teuthology.orchestra.run.smithi123.stdout:mds advanced rados_mon_op_timeout 900 2024-09-16T15:47:43.534 INFO:teuthology.orchestra.run.smithi123.stdout:mds advanced rados_osd_op_timeout 900 2024-09-16T15:47:43.534 INFO:teuthology.orchestra.run.smithi123.stdout:client advanced client_mount_timeout 600 2024-09-16T15:47:43.534 INFO:teuthology.orchestra.run.smithi123.stdout:client advanced debug_client 20/20 2024-09-16T15:47:43.534 INFO:teuthology.orchestra.run.smithi123.stdout:client advanced debug_ms 1/1 2024-09-16T15:47:43.534 INFO:teuthology.orchestra.run.smithi123.stdout:client advanced rados_mon_op_timeout 900 2024-09-16T15:47:43.534 INFO:teuthology.orchestra.run.smithi123.stdout:client advanced rados_osd_op_timeout 900 2024-09-16T15:47:43.535 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:43.532+0000 7f5c88555640 1 -- 172.21.15.123:0/2167457249 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f5c64072fe0 msgr2=0x7f5c640754a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:47:43.535 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:43.532+0000 7f5c88555640 1 --2- 172.21.15.123:0/2167457249 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f5c64072fe0 0x7f5c640754a0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f5c7c0af000 tx=0x7f5c7c01af80 comp rx=0 tx=0).stop 2024-09-16T15:47:43.535 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:43.532+0000 7f5c88555640 1 -- 172.21.15.123:0/2167457249 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5c8015bca0 msgr2=0x7f5c800cf180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:47:43.535 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:43.532+0000 7f5c88555640 1 --2- 172.21.15.123:0/2167457249 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5c8015bca0 0x7f5c800cf180 secure :-1 s=READY pgs=901 cs=0 l=1 rev1=1 crypto rx=0x7f5c7800cce0 tx=0x7f5c7800a620 comp rx=0 tx=0).stop 2024-09-16T15:47:43.536 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:43.532+0000 7f5c88555640 1 -- 172.21.15.123:0/2167457249 shutdown_connections 2024-09-16T15:47:43.536 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:43.532+0000 7f5c88555640 1 --2- 172.21.15.123:0/2167457249 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f5c64072fe0 0x7f5c640754a0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:43.536 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:43.532+0000 7f5c88555640 1 --2- 172.21.15.123:0/2167457249 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5c8015bca0 0x7f5c800cf180 unknown :-1 s=CLOSED pgs=901 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:43.536 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:43.532+0000 7f5c88555640 1 --2- 172.21.15.123:0/2167457249 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5c80074600 0x7f5c800cec40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:43.536 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:43.532+0000 7f5c88555640 1 --2- 172.21.15.123:0/2167457249 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f5c800726c0 0x7f5c801fa610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:43.536 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:43.532+0000 7f5c88555640 1 -- 172.21.15.123:0/2167457249 >> 172.21.15.123:0/2167457249 conn(0x7f5c800cfb10 msgr2=0x7f5c800758d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:47:43.536 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:43.532+0000 7f5c88555640 1 -- 172.21.15.123:0/2167457249 shutdown_connections 2024-09-16T15:47:43.536 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:43.532+0000 7f5c88555640 1 -- 172.21.15.123:0/2167457249 wait complete. 2024-09-16T15:47:43.552 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-09-16T15:47:43.679 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:43.676+0000 7f8fabf79640 1 -- 172.21.15.123:0/4090483311 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8fa40cc3c0 msgr2=0x7f8fa40cc870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:47:43.679 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:43.676+0000 7f8fabf79640 1 --2- 172.21.15.123:0/4090483311 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8fa40cc3c0 0x7f8fa40cc870 secure :-1 s=READY pgs=903 cs=0 l=1 rev1=1 crypto rx=0x7f8f9c009c80 tx=0x7f8f9c01cda0 comp rx=0 tx=0).stop 2024-09-16T15:47:43.679 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:43.676+0000 7f8fabf79640 1 -- 172.21.15.123:0/4090483311 shutdown_connections 2024-09-16T15:47:43.679 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:43.676+0000 7f8fabf79640 1 --2- 172.21.15.123:0/4090483311 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8fa40cc3c0 0x7f8fa40cc870 unknown :-1 s=CLOSED pgs=903 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:43.679 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:43.676+0000 7f8fabf79640 1 --2- 172.21.15.123:0/4090483311 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f8fa415aa00 0x7f8fa415e5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:43.679 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:43.676+0000 7f8fabf79640 1 -- 172.21.15.123:0/4090483311 >> 172.21.15.123:0/4090483311 conn(0x7f8fa40b1710 msgr2=0x7f8fa40b1b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:47:43.680 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:43.676+0000 7f8fabf79640 1 -- 172.21.15.123:0/4090483311 shutdown_connections 2024-09-16T15:47:43.680 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:43.676+0000 7f8fabf79640 1 -- 172.21.15.123:0/4090483311 wait complete. 2024-09-16T15:47:43.680 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:43.676+0000 7f8fabf79640 1 Processor -- start 2024-09-16T15:47:43.680 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:43.676+0000 7f8fabf79640 1 -- start start 2024-09-16T15:47:43.682 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:43.676+0000 7f8fabf79640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8fa4072720 0x7f8fa40c8de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:47:43.682 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:43.676+0000 7f8fabf79640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f8fa40cc3c0 0x7f8fa40c9320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:47:43.682 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:43.676+0000 7f8fabf79640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f8fa415aa00 0x7f8fa41fb780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:47:43.682 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:43.676+0000 7f8fabf79640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f8fa4167b80 con 0x7f8fa4072720 2024-09-16T15:47:43.682 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:43.676+0000 7f8fabf79640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f8fa406ffc0 con 0x7f8fa415aa00 2024-09-16T15:47:43.683 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:43.676+0000 7f8fabf79640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f8fa4167d00 con 0x7f8fa40cc3c0 2024-09-16T15:47:43.683 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:43.676+0000 7f8fa9d15640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8fa4072720 0x7f8fa40c8de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:47:43.683 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:43.676+0000 7f8fa9d15640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8fa4072720 0x7f8fa40c8de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:48498/0 (socket says 172.21.15.123:48498) 2024-09-16T15:47:43.683 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:43.676+0000 7f8fa9d15640 1 -- 172.21.15.123:0/2403334885 learned_addr learned my addr 172.21.15.123:0/2403334885 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:47:43.683 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:43.676+0000 7f8fa9514640 1 --2- 172.21.15.123:0/2403334885 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f8fa40cc3c0 0x7f8fa40c9320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:47:43.683 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:43.676+0000 7f8fa9d15640 1 -- 172.21.15.123:0/2403334885 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f8fa40cc3c0 msgr2=0x7f8fa40c9320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:47:43.683 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:43.676+0000 7f8fa9d15640 1 --2- 172.21.15.123:0/2403334885 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f8fa40cc3c0 0x7f8fa40c9320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:43.683 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:43.676+0000 7f8fa9d15640 1 -- 172.21.15.123:0/2403334885 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f8fa415aa00 msgr2=0x7f8fa41fb780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:47:43.683 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:43.676+0000 7f8fa9d15640 1 --2- 172.21.15.123:0/2403334885 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f8fa415aa00 0x7f8fa41fb780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:43.683 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:43.676+0000 7f8fa9d15640 1 -- 172.21.15.123:0/2403334885 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8fa40ca2d0 con 0x7f8fa4072720 2024-09-16T15:47:43.683 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:43.676+0000 7f8fa9d15640 1 --2- 172.21.15.123:0/2403334885 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8fa4072720 0x7f8fa40c8de0 secure :-1 s=READY pgs=904 cs=0 l=1 rev1=1 crypto rx=0x7f8fa000d980 tx=0x7f8fa000de40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:47:43.683 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:43.676+0000 7f8f9a7fc640 1 -- 172.21.15.123:0/2403334885 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8fa0017070 con 0x7f8fa4072720 2024-09-16T15:47:43.683 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:43.676+0000 7f8f9a7fc640 1 -- 172.21.15.123:0/2403334885 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f8fa0009d20 con 0x7f8fa4072720 2024-09-16T15:47:43.684 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:43.676+0000 7f8f9a7fc640 1 -- 172.21.15.123:0/2403334885 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8fa0004280 con 0x7f8fa4072720 2024-09-16T15:47:43.684 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:43.676+0000 7f8fabf79640 1 -- 172.21.15.123:0/2403334885 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8fa4158e50 con 0x7f8fa4072720 2024-09-16T15:47:43.684 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:43.676+0000 7f8fabf79640 1 -- 172.21.15.123:0/2403334885 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8fa40ae370 con 0x7f8fa4072720 2024-09-16T15:47:43.684 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:43.680+0000 7f8fabf79640 1 -- 172.21.15.123:0/2403334885 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8fa40cc870 con 0x7f8fa4072720 2024-09-16T15:47:43.684 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:43.680+0000 7f8f9a7fc640 1 -- 172.21.15.123:0/2403334885 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 16) ==== 97182+0+0 (secure 0 0 0) 0x7f8fa0005020 con 0x7f8fa4072720 2024-09-16T15:47:43.685 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:43.680+0000 7f8f9a7fc640 1 --2- 172.21.15.123:0/2403334885 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f8f88073010 0x7f8f880754d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:47:43.685 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:43.680+0000 7f8f9a7fc640 1 -- 172.21.15.123:0/2403334885 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(76..76 src has 1..76) ==== 5502+0+0 (secure 0 0 0) 0x7f8fa0094440 con 0x7f8fa4072720 2024-09-16T15:47:43.685 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:43.680+0000 7f8fa9514640 1 --2- 172.21.15.123:0/2403334885 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f8f88073010 0x7f8f880754d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:47:43.685 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:43.680+0000 7f8fa9514640 1 --2- 172.21.15.123:0/2403334885 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f8f88073010 0x7f8f880754d0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f8fa40c99a0 tx=0x7f8f94009040 comp rx=0 tx=0).ready entity=mgr.4150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:47:43.687 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:43.684+0000 7f8f9a7fc640 1 -- 172.21.15.123:0/2403334885 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f8fa0020030 con 0x7f8fa4072720 2024-09-16T15:47:43.851 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:43.847+0000 7f8fabf79640 1 -- 172.21.15.123:0/2403334885 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) -- 0x7f8fa40cd440 con 0x7f8fa4072720 2024-09-16T15:47:43.858 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:43.851+0000 7f8f9a7fc640 1 -- 172.21.15.123:0/2403334885 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v36)=0 v36) ==== 122+0+0 (secure 0 0 0) 0x7f8fa005f620 con 0x7f8fa4072720 2024-09-16T15:47:43.860 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:43.855+0000 7f8fabf79640 1 -- 172.21.15.123:0/2403334885 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f8f88073010 msgr2=0x7f8f880754d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:47:43.860 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:43.855+0000 7f8fabf79640 1 --2- 172.21.15.123:0/2403334885 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f8f88073010 0x7f8f880754d0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f8fa40c99a0 tx=0x7f8f94009040 comp rx=0 tx=0).stop 2024-09-16T15:47:43.860 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:43.855+0000 7f8fabf79640 1 -- 172.21.15.123:0/2403334885 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8fa4072720 msgr2=0x7f8fa40c8de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:47:43.860 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:43.855+0000 7f8fabf79640 1 --2- 172.21.15.123:0/2403334885 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8fa4072720 0x7f8fa40c8de0 secure :-1 s=READY pgs=904 cs=0 l=1 rev1=1 crypto rx=0x7f8fa000d980 tx=0x7f8fa000de40 comp rx=0 tx=0).stop 2024-09-16T15:47:43.861 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:43.855+0000 7f8fabf79640 1 -- 172.21.15.123:0/2403334885 shutdown_connections 2024-09-16T15:47:43.861 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:43.855+0000 7f8fabf79640 1 --2- 172.21.15.123:0/2403334885 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f8f88073010 0x7f8f880754d0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:43.861 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:43.855+0000 7f8fabf79640 1 --2- 172.21.15.123:0/2403334885 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f8fa415aa00 0x7f8fa41fb780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:43.861 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:43.855+0000 7f8fabf79640 1 --2- 172.21.15.123:0/2403334885 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f8fa40cc3c0 0x7f8fa40c9320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:43.861 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:43.855+0000 7f8fabf79640 1 --2- 172.21.15.123:0/2403334885 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8fa4072720 0x7f8fa40c8de0 unknown :-1 s=CLOSED pgs=904 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:43.861 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:43.855+0000 7f8fabf79640 1 -- 172.21.15.123:0/2403334885 >> 172.21.15.123:0/2403334885 conn(0x7f8fa40b1710 msgr2=0x7f8fa40b01a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:47:43.861 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:43.855+0000 7f8fabf79640 1 -- 172.21.15.123:0/2403334885 shutdown_connections 2024-09-16T15:47:43.861 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:43.855+0000 7f8fabf79640 1 -- 172.21.15.123:0/2403334885 wait complete. 2024-09-16T15:47:43.876 INFO:teuthology.nuke:Clearing teuthology firewall rules... 2024-09-16T15:47:43.876 DEBUG:teuthology.orchestra.run.smithi123:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-09-16T15:47:43.951 DEBUG:teuthology.orchestra.run.smithi184:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-09-16T15:47:44.024 INFO:teuthology.nuke:Cleared teuthology firewall rules. 2024-09-16T15:47:44.024 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2024-09-16T15:47:44.024 DEBUG:teuthology.orchestra.run.smithi123:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-16T15:47:44.030 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16336468k,nr_inodes=4084117,mode=755,inode64 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3274684k,mode=755,inode64 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev,inode64 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=17082 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3274680k,nr_inodes=818670,mode=700,uid=1000,gid=1271,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3274684k,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-09-16T15:47:44.030 DEBUG:teuthology.orchestra.run.smithi184:> set -ex 2024-09-16T15:47:44.030 DEBUG:teuthology.orchestra.run.smithi184:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-16T15:47:44.036 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16336488k,nr_inodes=4084122,mode=755,inode64 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3274688k,mode=755,inode64 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev,inode64 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=17834 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3274684k,nr_inodes=818671,mode=700,uid=1000,gid=1271,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3274688k,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-09-16T15:47:44.036 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2024-09-16T15:47:44.036 DEBUG:teuthology.orchestra.run.smithi123:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-16T15:47:44.079 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16336468k,nr_inodes=4084117,mode=755,inode64 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3274684k,mode=755,inode64 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev,inode64 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=17082 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3274680k,nr_inodes=818670,mode=700,uid=1000,gid=1271,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3274684k,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-09-16T15:47:44.079 DEBUG:teuthology.orchestra.run.smithi184:> set -ex 2024-09-16T15:47:44.079 DEBUG:teuthology.orchestra.run.smithi184:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-16T15:47:44.085 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16336488k,nr_inodes=4084122,mode=755,inode64 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3274688k,mode=755,inode64 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev,inode64 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=17834 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3274684k,nr_inodes=818671,mode=700,uid=1000,gid=1271,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3274688k,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-09-16T15:47:44.085 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-09-16T15:47:44.085 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-16T15:47:44.255 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:44.251+0000 7f436c9bb640 1 -- 172.21.15.123:0/2402065495 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f43680726c0 msgr2=0x7f43680d0630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:47:44.255 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:44.251+0000 7f436c9bb640 1 --2- 172.21.15.123:0/2402065495 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f43680726c0 0x7f43680d0630 secure :-1 s=READY pgs=905 cs=0 l=1 rev1=1 crypto rx=0x7f4350009900 tx=0x7f435001c820 comp rx=0 tx=0).stop 2024-09-16T15:47:44.255 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:44.251+0000 7f436c9bb640 1 -- 172.21.15.123:0/2402065495 shutdown_connections 2024-09-16T15:47:44.255 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:44.251+0000 7f436c9bb640 1 --2- 172.21.15.123:0/2402065495 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f4368074600 0x7f436806e750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:44.255 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:44.251+0000 7f436c9bb640 1 --2- 172.21.15.123:0/2402065495 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f43680726c0 0x7f43680d0630 unknown :-1 s=CLOSED pgs=905 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:44.255 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:44.251+0000 7f436c9bb640 1 -- 172.21.15.123:0/2402065495 >> 172.21.15.123:0/2402065495 conn(0x7f43680cfb10 msgr2=0x7f43680cff20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:47:44.255 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:44.251+0000 7f436c9bb640 1 -- 172.21.15.123:0/2402065495 shutdown_connections 2024-09-16T15:47:44.255 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:44.251+0000 7f436c9bb640 1 -- 172.21.15.123:0/2402065495 wait complete. 2024-09-16T15:47:44.256 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:44.251+0000 7f436c9bb640 1 Processor -- start 2024-09-16T15:47:44.256 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:44.251+0000 7f436c9bb640 1 -- start start 2024-09-16T15:47:44.258 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:44.251+0000 7f436c9bb640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f43680726c0 0x7f4368156040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:47:44.258 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:44.251+0000 7f436c9bb640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4368074600 0x7f43681594e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:47:44.258 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:44.251+0000 7f436c9bb640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f436815bca0 0x7f4368157120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:47:44.258 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:44.251+0000 7f436c9bb640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f4368168bd0 con 0x7f436815bca0 2024-09-16T15:47:44.258 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:44.251+0000 7f436c9bb640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f4368168a50 con 0x7f4368074600 2024-09-16T15:47:44.258 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:44.251+0000 7f436c9bb640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f4368168d50 con 0x7f43680726c0 2024-09-16T15:47:44.259 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:44.251+0000 7f4366d9d640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f436815bca0 0x7f4368157120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:47:44.259 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:44.251+0000 7f4366d9d640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f436815bca0 0x7f4368157120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:48506/0 (socket says 172.21.15.123:48506) 2024-09-16T15:47:44.259 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:44.251+0000 7f4366d9d640 1 -- 172.21.15.123:0/3756840199 learned_addr learned my addr 172.21.15.123:0/3756840199 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:47:44.259 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:44.251+0000 7f435dd9b640 1 --2- 172.21.15.123:0/3756840199 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4368074600 0x7f43681594e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:47:44.259 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:44.251+0000 7f436659c640 1 --2- 172.21.15.123:0/3756840199 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f43680726c0 0x7f4368156040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:47:44.259 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:44.251+0000 7f4366d9d640 1 -- 172.21.15.123:0/3756840199 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f43680726c0 msgr2=0x7f4368156040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:47:44.259 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:44.251+0000 7f4366d9d640 1 --2- 172.21.15.123:0/3756840199 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f43680726c0 0x7f4368156040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:44.259 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:44.251+0000 7f4366d9d640 1 -- 172.21.15.123:0/3756840199 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4368074600 msgr2=0x7f43681594e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:47:44.259 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:44.251+0000 7f4366d9d640 1 --2- 172.21.15.123:0/3756840199 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4368074600 0x7f43681594e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:44.259 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:44.251+0000 7f4366d9d640 1 -- 172.21.15.123:0/3756840199 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4368155280 con 0x7f436815bca0 2024-09-16T15:47:44.259 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:44.251+0000 7f4366d9d640 1 --2- 172.21.15.123:0/3756840199 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f436815bca0 0x7f4368157120 secure :-1 s=READY pgs=906 cs=0 l=1 rev1=1 crypto rx=0x7f436000ea20 tx=0x7f436000eee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:47:44.259 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:44.251+0000 7f435f7fe640 1 -- 172.21.15.123:0/3756840199 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f436000cde0 con 0x7f436815bca0 2024-09-16T15:47:44.259 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:44.251+0000 7f435f7fe640 1 -- 172.21.15.123:0/3756840199 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f4360004c40 con 0x7f436815bca0 2024-09-16T15:47:44.260 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:44.251+0000 7f436c9bb640 1 -- 172.21.15.123:0/3756840199 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4368155560 con 0x7f436815bca0 2024-09-16T15:47:44.260 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:44.251+0000 7f435f7fe640 1 -- 172.21.15.123:0/3756840199 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4360005280 con 0x7f436815bca0 2024-09-16T15:47:44.260 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:44.251+0000 7f436c9bb640 1 -- 172.21.15.123:0/3756840199 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4368155b20 con 0x7f436815bca0 2024-09-16T15:47:44.260 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:44.255+0000 7f436c9bb640 1 -- 172.21.15.123:0/3756840199 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f43680d0630 con 0x7f436815bca0 2024-09-16T15:47:44.261 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:44.255+0000 7f435f7fe640 1 -- 172.21.15.123:0/3756840199 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 16) ==== 97182+0+0 (secure 0 0 0) 0x7f4360005420 con 0x7f436815bca0 2024-09-16T15:47:44.261 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:44.255+0000 7f435f7fe640 1 --2- 172.21.15.123:0/3756840199 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f4348072fe0 0x7f43480754a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:47:44.261 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:44.255+0000 7f436659c640 1 --2- 172.21.15.123:0/3756840199 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f4348072fe0 0x7f43480754a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:47:44.261 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:44.255+0000 7f435f7fe640 1 -- 172.21.15.123:0/3756840199 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(76..76 src has 1..76) ==== 5502+0+0 (secure 0 0 0) 0x7f43600949c0 con 0x7f436815bca0 2024-09-16T15:47:44.261 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:44.255+0000 7f436659c640 1 --2- 172.21.15.123:0/3756840199 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f4348072fe0 0x7f43480754a0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f43680c7e70 tx=0x7f4358008040 comp rx=0 tx=0).ready entity=mgr.4150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:47:44.264 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:44.259+0000 7f435f7fe640 1 -- 172.21.15.123:0/3756840199 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f4360096e20 con 0x7f436815bca0 2024-09-16T15:47:44.467 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:44.463+0000 7f436c9bb640 1 -- 172.21.15.123:0/3756840199 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f43680700e0 con 0x7f436815bca0 2024-09-16T15:47:44.467 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:44.463+0000 7f435f7fe640 1 -- 172.21.15.123:0/3756840199 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 47 v47) ==== 94+0+3920 (secure 0 0 0) 0x7f4360016030 con 0x7f436815bca0 2024-09-16T15:47:44.468 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-09-16T15:47:44.468 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":47,"btime":"2024-09-16T15:47:38:160543+0000","default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5048,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6837/1498917352","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6835","nonce":1498917352},{"type":"v1","addr":"172.21.15.184:6837","nonce":1498917352}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":40},{"gid":5225,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6833/1069315256","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":1069315256},{"type":"v1","addr":"172.21.15.184:6833","nonce":1069315256}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":47},{"gid":6198,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6833/1715139909","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6832","nonce":1715139909},{"type":"v1","addr":"172.21.15.123:6833","nonce":1715139909}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":40},{"gid":6201,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6835/815923338","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6834","nonce":815923338},{"type":"v1","addr":"172.21.15.123:6835","nonce":815923338}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":40}],"filesystems":[]} 2024-09-16T15:47:44.468 INFO:teuthology.orchestra.run.smithi123.stderr:dumped fsmap epoch 47 2024-09-16T15:47:44.470 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:44.463+0000 7f436c9bb640 1 -- 172.21.15.123:0/3756840199 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f4348072fe0 msgr2=0x7f43480754a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:47:44.470 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:44.463+0000 7f436c9bb640 1 --2- 172.21.15.123:0/3756840199 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f4348072fe0 0x7f43480754a0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f43680c7e70 tx=0x7f4358008040 comp rx=0 tx=0).stop 2024-09-16T15:47:44.470 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:44.463+0000 7f436c9bb640 1 -- 172.21.15.123:0/3756840199 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f436815bca0 msgr2=0x7f4368157120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:47:44.470 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:44.463+0000 7f436c9bb640 1 --2- 172.21.15.123:0/3756840199 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f436815bca0 0x7f4368157120 secure :-1 s=READY pgs=906 cs=0 l=1 rev1=1 crypto rx=0x7f436000ea20 tx=0x7f436000eee0 comp rx=0 tx=0).stop 2024-09-16T15:47:44.470 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:44.463+0000 7f436c9bb640 1 -- 172.21.15.123:0/3756840199 shutdown_connections 2024-09-16T15:47:44.470 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:44.463+0000 7f436c9bb640 1 --2- 172.21.15.123:0/3756840199 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f4348072fe0 0x7f43480754a0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:44.470 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:44.463+0000 7f436c9bb640 1 --2- 172.21.15.123:0/3756840199 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f436815bca0 0x7f4368157120 unknown :-1 s=CLOSED pgs=906 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:44.471 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:44.463+0000 7f436c9bb640 1 --2- 172.21.15.123:0/3756840199 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4368074600 0x7f43681594e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:44.471 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:44.463+0000 7f436c9bb640 1 --2- 172.21.15.123:0/3756840199 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f43680726c0 0x7f4368156040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:44.471 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:44.463+0000 7f436c9bb640 1 -- 172.21.15.123:0/3756840199 >> 172.21.15.123:0/3756840199 conn(0x7f43680cfb10 msgr2=0x7f436815af50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:47:44.471 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:44.463+0000 7f436c9bb640 1 -- 172.21.15.123:0/3756840199 shutdown_connections 2024-09-16T15:47:44.471 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:44.463+0000 7f436c9bb640 1 -- 172.21.15.123:0/3756840199 wait complete. 2024-09-16T15:47:44.485 INFO:tasks.ceph.mds.a:Restarting daemon 2024-09-16T15:47:44.485 INFO:tasks.ceph.mds.a:Stopping old one... 2024-09-16T15:47:44.485 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-09-16T15:47:44.486 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T15:47:44.486 INFO:tasks.ceph.mds.c:Restarting daemon 2024-09-16T15:47:44.486 INFO:tasks.ceph.mds.c:Stopping old one... 2024-09-16T15:47:44.486 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-09-16T15:47:44.486 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T15:47:44.486 INFO:tasks.ceph.mds.b:Restarting daemon 2024-09-16T15:47:44.486 INFO:tasks.ceph.mds.b:Stopping old one... 2024-09-16T15:47:44.487 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-09-16T15:47:44.487 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T15:47:44.487 INFO:tasks.ceph.mds.d:Restarting daemon 2024-09-16T15:47:44.487 INFO:tasks.ceph.mds.d:Stopping old one... 2024-09-16T15:47:44.487 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-09-16T15:47:44.487 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T15:47:44.497 INFO:tasks.ceph.mds.a:Stopped 2024-09-16T15:47:44.497 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-09-16T15:47:44.498 INFO:tasks.ceph.mds.c:Stopped 2024-09-16T15:47:44.498 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-09-16T15:47:44.501 INFO:tasks.ceph.mds.b:Stopped 2024-09-16T15:47:44.502 DEBUG:teuthology.orchestra.run.smithi184:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-09-16T15:47:44.504 INFO:tasks.ceph.mds.d:Stopped 2024-09-16T15:47:44.504 DEBUG:teuthology.orchestra.run.smithi184:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-09-16T15:47:44.540 INFO:tasks.ceph.mds.a:Started 2024-09-16T15:47:44.540 DEBUG:teuthology.parallel:result is None 2024-09-16T15:47:44.540 INFO:tasks.ceph.mds.c:Started 2024-09-16T15:47:44.541 DEBUG:teuthology.parallel:result is None 2024-09-16T15:47:44.547 INFO:tasks.ceph.mds.b:Started 2024-09-16T15:47:44.548 DEBUG:teuthology.parallel:result is None 2024-09-16T15:47:44.549 INFO:tasks.ceph.mds.d:Started 2024-09-16T15:47:44.549 DEBUG:teuthology.parallel:result is None 2024-09-16T15:47:44.550 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-09-16T15:47:44.603 INFO:tasks.ceph.mds.a.smithi123.stdout:starting mds.a at 2024-09-16T15:47:44.604 INFO:tasks.ceph.mds.c.smithi123.stdout:starting mds.c at 2024-09-16T15:47:44.619 INFO:tasks.ceph.mds.d.smithi184.stdout:starting mds.d at 2024-09-16T15:47:44.619 INFO:tasks.ceph.mds.b.smithi184.stdout:starting mds.b at 2024-09-16T15:47:44.661 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:44.655+0000 7f1ade304640 1 -- 172.21.15.123:0/3326518702 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1ad80b11a0 msgr2=0x7f1ad80d1b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:47:44.661 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:44.655+0000 7f1ade304640 1 --2- 172.21.15.123:0/3326518702 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1ad80b11a0 0x7f1ad80d1b90 secure :-1 s=READY pgs=915 cs=0 l=1 rev1=1 crypto rx=0x7f1ad8072ef0 tx=0x7f1acc01c890 comp rx=0 tx=0).stop 2024-09-16T15:47:44.661 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:44.655+0000 7f1ade304640 1 -- 172.21.15.123:0/3326518702 shutdown_connections 2024-09-16T15:47:44.661 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:44.655+0000 7f1ade304640 1 --2- 172.21.15.123:0/3326518702 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1ad80b11a0 0x7f1ad80d1b90 unknown :-1 s=CLOSED pgs=915 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:44.661 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:44.655+0000 7f1ade304640 1 --2- 172.21.15.123:0/3326518702 >> v2:172.21.15.184:3300/0 conn(0x7f1ad815bbc0 0x7f1ad80b2d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:44.661 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:44.655+0000 7f1ade304640 1 -- 172.21.15.123:0/3326518702 >> 172.21.15.123:0/3326518702 conn(0x7f1ad809aa40 msgr2=0x7f1ad809ae50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:47:44.661 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:44.655+0000 7f1ade304640 1 -- 172.21.15.123:0/3326518702 shutdown_connections 2024-09-16T15:47:44.661 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:44.655+0000 7f1ade304640 1 -- 172.21.15.123:0/3326518702 wait complete. 2024-09-16T15:47:44.662 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:44.655+0000 7f1ade304640 1 Processor -- start 2024-09-16T15:47:44.662 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:44.655+0000 7f1ade304640 1 -- start start 2024-09-16T15:47:44.664 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:44.659+0000 7f1ade304640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f1ad80b32c0 0x7f1ad81f6cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:47:44.664 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:44.659+0000 7f1ade304640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1ad815bbc0 0x7f1ad814bf30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:47:44.664 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:44.659+0000 7f1ade304640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1ad8153ea0 0x7f1ad8154330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:47:44.664 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:44.659+0000 7f1ade304640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f1ad81693f0 con 0x7f1ad815bbc0 2024-09-16T15:47:44.664 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:44.659+0000 7f1ade304640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f1ad8169270 con 0x7f1ad8153ea0 2024-09-16T15:47:44.664 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:44.659+0000 7f1ade304640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f1ad8169570 con 0x7f1ad80b32c0 2024-09-16T15:47:44.664 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:44.659+0000 7f1ad77fe640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1ad815bbc0 0x7f1ad814bf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:47:44.664 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:44.659+0000 7f1ad77fe640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1ad815bbc0 0x7f1ad814bf30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:48570/0 (socket says 172.21.15.123:48570) 2024-09-16T15:47:44.664 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:44.659+0000 7f1ad77fe640 1 -- 172.21.15.123:0/1227955448 learned_addr learned my addr 172.21.15.123:0/1227955448 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:47:44.664 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:44.659+0000 7f1ad7fff640 1 --2- 172.21.15.123:0/1227955448 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f1ad80b32c0 0x7f1ad81f6cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:47:44.664 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:44.659+0000 7f1adc8a1640 1 --2- 172.21.15.123:0/1227955448 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1ad8153ea0 0x7f1ad8154330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:47:44.664 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:44.659+0000 7f1ad77fe640 1 -- 172.21.15.123:0/1227955448 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f1ad80b32c0 msgr2=0x7f1ad81f6cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:47:44.665 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:44.659+0000 7f1ad77fe640 1 --2- 172.21.15.123:0/1227955448 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f1ad80b32c0 0x7f1ad81f6cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:44.665 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:44.659+0000 7f1ad77fe640 1 -- 172.21.15.123:0/1227955448 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1ad8153ea0 msgr2=0x7f1ad8154330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:47:44.665 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:44.659+0000 7f1ad77fe640 1 --2- 172.21.15.123:0/1227955448 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1ad8153ea0 0x7f1ad8154330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:44.665 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:44.659+0000 7f1ad77fe640 1 -- 172.21.15.123:0/1227955448 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1ad80c8220 con 0x7f1ad815bbc0 2024-09-16T15:47:44.665 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:44.659+0000 7f1ad7fff640 1 --2- 172.21.15.123:0/1227955448 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f1ad80b32c0 0x7f1ad81f6cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T15:47:44.665 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:44.659+0000 7f1adc8a1640 1 --2- 172.21.15.123:0/1227955448 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1ad8153ea0 0x7f1ad8154330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T15:47:44.665 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:44.659+0000 7f1ad77fe640 1 --2- 172.21.15.123:0/1227955448 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1ad815bbc0 0x7f1ad814bf30 secure :-1 s=READY pgs=916 cs=0 l=1 rev1=1 crypto rx=0x7f1acc0194a0 tx=0x7f1acc005760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:47:44.665 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:44.659+0000 7f1ad4ff9640 1 -- 172.21.15.123:0/1227955448 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1acc0b2040 con 0x7f1ad815bbc0 2024-09-16T15:47:44.665 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:44.659+0000 7f1ad4ff9640 1 -- 172.21.15.123:0/1227955448 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f1acc005250 con 0x7f1ad815bbc0 2024-09-16T15:47:44.665 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:44.659+0000 7f1ade304640 1 -- 172.21.15.123:0/1227955448 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1ad80b56a0 con 0x7f1ad815bbc0 2024-09-16T15:47:44.665 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:44.659+0000 7f1ad4ff9640 1 -- 172.21.15.123:0/1227955448 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1acc0046d0 con 0x7f1ad815bbc0 2024-09-16T15:47:44.665 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:44.659+0000 7f1ade304640 1 -- 172.21.15.123:0/1227955448 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1ad80b5c60 con 0x7f1ad815bbc0 2024-09-16T15:47:44.665 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:44.659+0000 7f1ade304640 1 -- 172.21.15.123:0/1227955448 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1ad80d1b90 con 0x7f1ad815bbc0 2024-09-16T15:47:44.666 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:44.659+0000 7f1ad4ff9640 1 -- 172.21.15.123:0/1227955448 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 16) ==== 97182+0+0 (secure 0 0 0) 0x7f1acc004870 con 0x7f1ad815bbc0 2024-09-16T15:47:44.666 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:44.659+0000 7f1ad4ff9640 1 --2- 172.21.15.123:0/1227955448 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f1ac4072fe0 0x7f1ac40754a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:47:44.666 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:44.659+0000 7f1ad4ff9640 1 -- 172.21.15.123:0/1227955448 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(76..76 src has 1..76) ==== 5502+0+0 (secure 0 0 0) 0x7f1acc12e500 con 0x7f1ad815bbc0 2024-09-16T15:47:44.666 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:44.659+0000 7f1ad7fff640 1 --2- 172.21.15.123:0/1227955448 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f1ac4072fe0 0x7f1ac40754a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:47:44.666 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:44.659+0000 7f1ad7fff640 1 --2- 172.21.15.123:0/1227955448 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f1ac4072fe0 0x7f1ac40754a0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f1ac8007eb0 tx=0x7f1ac80073c0 comp rx=0 tx=0).ready entity=mgr.4150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:47:44.669 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:44.663+0000 7f1ad4ff9640 1 -- 172.21.15.123:0/1227955448 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f1acc0c4060 con 0x7f1ad815bbc0 2024-09-16T15:47:44.837 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:44.831+0000 7f1ade304640 1 -- 172.21.15.123:0/1227955448 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) -- 0x7f1ad80c50a0 con 0x7f1ad815bbc0 2024-09-16T15:47:45.893 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:45.887+0000 7f1ad4ff9640 1 -- 172.21.15.123:0/1227955448 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v81) ==== 97+0+0 (secure 0 0 0) 0x7f1acc0164b0 con 0x7f1ad815bbc0 2024-09-16T15:47:45.893 INFO:teuthology.orchestra.run.smithi123.stderr: removed all blocklist entries 2024-09-16T15:47:45.896 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:45.891+0000 7f1ade304640 1 -- 172.21.15.123:0/1227955448 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f1ac4072fe0 msgr2=0x7f1ac40754a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:47:45.896 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:45.891+0000 7f1ade304640 1 --2- 172.21.15.123:0/1227955448 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f1ac4072fe0 0x7f1ac40754a0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f1ac8007eb0 tx=0x7f1ac80073c0 comp rx=0 tx=0).stop 2024-09-16T15:47:45.896 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:45.891+0000 7f1ade304640 1 -- 172.21.15.123:0/1227955448 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1ad815bbc0 msgr2=0x7f1ad814bf30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:47:45.896 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:45.891+0000 7f1ade304640 1 --2- 172.21.15.123:0/1227955448 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1ad815bbc0 0x7f1ad814bf30 secure :-1 s=READY pgs=916 cs=0 l=1 rev1=1 crypto rx=0x7f1acc0194a0 tx=0x7f1acc005760 comp rx=0 tx=0).stop 2024-09-16T15:47:45.896 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:45.891+0000 7f1ade304640 1 -- 172.21.15.123:0/1227955448 shutdown_connections 2024-09-16T15:47:45.897 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:45.891+0000 7f1ade304640 1 --2- 172.21.15.123:0/1227955448 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f1ac4072fe0 0x7f1ac40754a0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:45.897 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:45.891+0000 7f1ade304640 1 --2- 172.21.15.123:0/1227955448 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1ad8153ea0 0x7f1ad8154330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:45.897 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:45.891+0000 7f1ade304640 1 --2- 172.21.15.123:0/1227955448 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1ad815bbc0 0x7f1ad814bf30 unknown :-1 s=CLOSED pgs=916 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:45.897 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:45.891+0000 7f1ade304640 1 --2- 172.21.15.123:0/1227955448 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f1ad80b32c0 0x7f1ad81f6cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:45.897 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:45.891+0000 7f1ade304640 1 -- 172.21.15.123:0/1227955448 >> 172.21.15.123:0/1227955448 conn(0x7f1ad809aa40 msgr2=0x7f1ad8152460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:47:45.897 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:45.891+0000 7f1ade304640 1 -- 172.21.15.123:0/1227955448 shutdown_connections 2024-09-16T15:47:45.897 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:45.891+0000 7f1ade304640 1 -- 172.21.15.123:0/1227955448 wait complete. 2024-09-16T15:47:45.918 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-09-16T15:47:46.045 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:46.039+0000 7f4b06327640 1 -- 172.21.15.123:0/1820309877 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f4b000b3100 msgr2=0x7f4b000cc1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:47:46.045 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:46.039+0000 7f4b06327640 1 --2- 172.21.15.123:0/1820309877 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f4b000b3100 0x7f4b000cc1b0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f4af4009900 tx=0x7f4af401c8e0 comp rx=0 tx=0).stop 2024-09-16T15:47:46.046 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:46.039+0000 7f4b06327640 1 -- 172.21.15.123:0/1820309877 shutdown_connections 2024-09-16T15:47:46.046 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:46.039+0000 7f4b06327640 1 --2- 172.21.15.123:0/1820309877 >> v2:172.21.15.184:3300/0 conn(0x7f4b000cc6f0 0x7f4b000cf650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:46.046 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:46.039+0000 7f4b06327640 1 --2- 172.21.15.123:0/1820309877 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f4b000b3100 0x7f4b000cc1b0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:46.046 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:46.039+0000 7f4b06327640 1 -- 172.21.15.123:0/1820309877 >> 172.21.15.123:0/1820309877 conn(0x7f4b000b2610 msgr2=0x7f4b000b29f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:47:46.046 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:46.039+0000 7f4b06327640 1 -- 172.21.15.123:0/1820309877 shutdown_connections 2024-09-16T15:47:46.046 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:46.039+0000 7f4b06327640 1 -- 172.21.15.123:0/1820309877 wait complete. 2024-09-16T15:47:46.046 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:46.043+0000 7f4b06327640 1 Processor -- start 2024-09-16T15:47:46.047 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:46.043+0000 7f4b06327640 1 -- start start 2024-09-16T15:47:46.049 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:46.043+0000 7f4b06327640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4b000b3100 0x7f4b0014f810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:47:46.049 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:46.043+0000 7f4b06327640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4b000cc6f0 0x7f4b0015d120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:47:46.049 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:46.043+0000 7f4b06327640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f4b0015ac00 0x7f4b00071340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:47:46.049 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:46.043+0000 7f4b06327640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f4b001682d0 con 0x7f4b000cc6f0 2024-09-16T15:47:46.049 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:46.043+0000 7f4b06327640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f4b00070a00 con 0x7f4b000b3100 2024-09-16T15:47:46.049 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:46.043+0000 7f4b06327640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f4b00168450 con 0x7f4b0015ac00 2024-09-16T15:47:46.049 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:46.043+0000 7f4aff7fe640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4b000cc6f0 0x7f4b0015d120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:47:46.049 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:46.043+0000 7f4aff7fe640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4b000cc6f0 0x7f4b0015d120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:48586/0 (socket says 172.21.15.123:48586) 2024-09-16T15:47:46.049 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:46.043+0000 7f4aff7fe640 1 -- 172.21.15.123:0/1752666737 learned_addr learned my addr 172.21.15.123:0/1752666737 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:47:46.049 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:46.043+0000 7f4b048c4640 1 --2- 172.21.15.123:0/1752666737 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f4b0015ac00 0x7f4b00071340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:47:46.049 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:46.043+0000 7f4afffff640 1 --2- 172.21.15.123:0/1752666737 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4b000b3100 0x7f4b0014f810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:47:46.049 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:46.043+0000 7f4aff7fe640 1 -- 172.21.15.123:0/1752666737 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f4b0015ac00 msgr2=0x7f4b00071340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:47:46.050 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:46.043+0000 7f4aff7fe640 1 --2- 172.21.15.123:0/1752666737 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f4b0015ac00 0x7f4b00071340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:46.050 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:46.043+0000 7f4aff7fe640 1 -- 172.21.15.123:0/1752666737 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4b000b3100 msgr2=0x7f4b0014f810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:47:46.050 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:46.043+0000 7f4aff7fe640 1 --2- 172.21.15.123:0/1752666737 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4b000b3100 0x7f4b0014f810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:46.050 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:46.043+0000 7f4aff7fe640 1 -- 172.21.15.123:0/1752666737 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4b0015d970 con 0x7f4b000cc6f0 2024-09-16T15:47:46.050 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:46.043+0000 7f4aff7fe640 1 --2- 172.21.15.123:0/1752666737 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4b000cc6f0 0x7f4b0015d120 secure :-1 s=READY pgs=917 cs=0 l=1 rev1=1 crypto rx=0x7f4af4009850 tx=0x7f4af40a55a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:47:46.050 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:46.043+0000 7f4afcff9640 1 -- 172.21.15.123:0/1752666737 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4af40b2040 con 0x7f4b000cc6f0 2024-09-16T15:47:46.050 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:46.043+0000 7f4afcff9640 1 -- 172.21.15.123:0/1752666737 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f4af40043b0 con 0x7f4b000cc6f0 2024-09-16T15:47:46.050 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:46.043+0000 7f4afcff9640 1 -- 172.21.15.123:0/1752666737 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4af4005290 con 0x7f4b000cc6f0 2024-09-16T15:47:46.050 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:46.043+0000 7f4b06327640 1 -- 172.21.15.123:0/1752666737 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4b0015dc50 con 0x7f4b000cc6f0 2024-09-16T15:47:46.050 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:46.043+0000 7f4b06327640 1 -- 172.21.15.123:0/1752666737 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4b000c7d10 con 0x7f4b000cc6f0 2024-09-16T15:47:46.051 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:46.043+0000 7f4b06327640 1 -- 172.21.15.123:0/1752666737 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4b000cc1b0 con 0x7f4b000cc6f0 2024-09-16T15:47:46.051 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:46.047+0000 7f4afcff9640 1 -- 172.21.15.123:0/1752666737 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 16) ==== 97182+0+0 (secure 0 0 0) 0x7f4af4005430 con 0x7f4b000cc6f0 2024-09-16T15:47:46.051 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:46.047+0000 7f4afcff9640 1 --2- 172.21.15.123:0/1752666737 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f4aec072fe0 0x7f4aec0754a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:47:46.051 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:46.047+0000 7f4afcff9640 1 -- 172.21.15.123:0/1752666737 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 4857+0+0 (secure 0 0 0) 0x7f4af412d1e0 con 0x7f4b000cc6f0 2024-09-16T15:47:46.051 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:46.047+0000 7f4afffff640 1 --2- 172.21.15.123:0/1752666737 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f4aec072fe0 0x7f4aec0754a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:47:46.052 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:46.047+0000 7f4afffff640 1 --2- 172.21.15.123:0/1752666737 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f4aec072fe0 0x7f4aec0754a0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f4af0007fd0 tx=0x7f4af0009040 comp rx=0 tx=0).ready entity=mgr.4150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:47:46.055 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:46.051+0000 7f4afcff9640 1 -- 172.21.15.123:0/1752666737 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f4af40bc060 con 0x7f4b000cc6f0 2024-09-16T15:47:46.290 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:46.287+0000 7f4b06327640 1 -- 172.21.15.123:0/1752666737 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) -- 0x7f4b000b6aa0 con 0x7f4b000cc6f0 2024-09-16T15:47:46.291 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:46.287+0000 7f4afcff9640 1 -- 172.21.15.123:0/1752666737 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v22) ==== 80+0+5304 (secure 0 0 0) 0x7f4af40164d0 con 0x7f4b000cc6f0 2024-09-16T15:47:46.291 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-09-16T15:47:46.292 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2024-09-16T15:47:46.292 INFO:teuthology.orchestra.run.smithi123.stdout: "auth_dump": [ 2024-09-16T15:47:46.292 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-09-16T15:47:46.292 INFO:teuthology.orchestra.run.smithi123.stdout: "entity": "mds.a", 2024-09-16T15:47:46.292 INFO:teuthology.orchestra.run.smithi123.stdout: "key": "AQC0UehmbZpOJBAAaYGyDcWrt4k0FS5P/bb+4A==", 2024-09-16T15:47:46.292 INFO:teuthology.orchestra.run.smithi123.stdout: "caps": { 2024-09-16T15:47:46.292 INFO:teuthology.orchestra.run.smithi123.stdout: "mds": "allow", 2024-09-16T15:47:46.292 INFO:teuthology.orchestra.run.smithi123.stdout: "mgr": "allow *", 2024-09-16T15:47:46.292 INFO:teuthology.orchestra.run.smithi123.stdout: "mon": "allow *", 2024-09-16T15:47:46.292 INFO:teuthology.orchestra.run.smithi123.stdout: "osd": "allow *" 2024-09-16T15:47:46.292 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-16T15:47:46.293 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:46.293 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-09-16T15:47:46.293 INFO:teuthology.orchestra.run.smithi123.stdout: "entity": "mds.b", 2024-09-16T15:47:46.293 INFO:teuthology.orchestra.run.smithi123.stdout: "key": "AQC0UehmF1HFMhAADcvqXNKdHZMPAbLc+se4Aw==", 2024-09-16T15:47:46.293 INFO:teuthology.orchestra.run.smithi123.stdout: "caps": { 2024-09-16T15:47:46.293 INFO:teuthology.orchestra.run.smithi123.stdout: "mds": "allow", 2024-09-16T15:47:46.293 INFO:teuthology.orchestra.run.smithi123.stdout: "mgr": "allow *", 2024-09-16T15:47:46.293 INFO:teuthology.orchestra.run.smithi123.stdout: "mon": "allow *", 2024-09-16T15:47:46.293 INFO:teuthology.orchestra.run.smithi123.stdout: "osd": "allow *" 2024-09-16T15:47:46.293 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-16T15:47:46.293 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:46.293 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-09-16T15:47:46.294 INFO:teuthology.orchestra.run.smithi123.stdout: "entity": "mds.c", 2024-09-16T15:47:46.294 INFO:teuthology.orchestra.run.smithi123.stdout: "key": "AQC0UehmCBjCLBAAXd85J25r7cgrXeDCG7pahg==", 2024-09-16T15:47:46.294 INFO:teuthology.orchestra.run.smithi123.stdout: "caps": { 2024-09-16T15:47:46.294 INFO:teuthology.orchestra.run.smithi123.stdout: "mds": "allow", 2024-09-16T15:47:46.294 INFO:teuthology.orchestra.run.smithi123.stdout: "mgr": "allow *", 2024-09-16T15:47:46.294 INFO:teuthology.orchestra.run.smithi123.stdout: "mon": "allow *", 2024-09-16T15:47:46.294 INFO:teuthology.orchestra.run.smithi123.stdout: "osd": "allow *" 2024-09-16T15:47:46.294 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-16T15:47:46.294 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:46.294 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-09-16T15:47:46.294 INFO:teuthology.orchestra.run.smithi123.stdout: "entity": "mds.d", 2024-09-16T15:47:46.294 INFO:teuthology.orchestra.run.smithi123.stdout: "key": "AQC0UehmMh3ZOBAAx0RCa3R9RMT93/icZhPGyg==", 2024-09-16T15:47:46.295 INFO:teuthology.orchestra.run.smithi123.stdout: "caps": { 2024-09-16T15:47:46.295 INFO:teuthology.orchestra.run.smithi123.stdout: "mds": "allow", 2024-09-16T15:47:46.295 INFO:teuthology.orchestra.run.smithi123.stdout: "mgr": "allow *", 2024-09-16T15:47:46.295 INFO:teuthology.orchestra.run.smithi123.stdout: "mon": "allow *", 2024-09-16T15:47:46.295 INFO:teuthology.orchestra.run.smithi123.stdout: "osd": "allow *" 2024-09-16T15:47:46.295 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-16T15:47:46.295 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:46.295 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-09-16T15:47:46.295 INFO:teuthology.orchestra.run.smithi123.stdout: "entity": "osd.0", 2024-09-16T15:47:46.295 INFO:teuthology.orchestra.run.smithi123.stdout: "key": "AQC4UehmT633CRAAIhpkeIBOzwJO0kiXLu0x9Q==", 2024-09-16T15:47:46.295 INFO:teuthology.orchestra.run.smithi123.stdout: "caps": { 2024-09-16T15:47:46.295 INFO:teuthology.orchestra.run.smithi123.stdout: "mgr": "allow profile osd", 2024-09-16T15:47:46.296 INFO:teuthology.orchestra.run.smithi123.stdout: "mon": "allow profile osd", 2024-09-16T15:47:46.296 INFO:teuthology.orchestra.run.smithi123.stdout: "osd": "allow *" 2024-09-16T15:47:46.296 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-16T15:47:46.296 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:46.296 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-09-16T15:47:46.296 INFO:teuthology.orchestra.run.smithi123.stdout: "entity": "osd.1", 2024-09-16T15:47:46.296 INFO:teuthology.orchestra.run.smithi123.stdout: "key": "AQC7UehmFwF7GBAAvOUG+rWjANKT8wm/ai4xGg==", 2024-09-16T15:47:46.296 INFO:teuthology.orchestra.run.smithi123.stdout: "caps": { 2024-09-16T15:47:46.296 INFO:teuthology.orchestra.run.smithi123.stdout: "mgr": "allow profile osd", 2024-09-16T15:47:46.296 INFO:teuthology.orchestra.run.smithi123.stdout: "mon": "allow profile osd", 2024-09-16T15:47:46.296 INFO:teuthology.orchestra.run.smithi123.stdout: "osd": "allow *" 2024-09-16T15:47:46.296 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-16T15:47:46.297 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:46.297 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-09-16T15:47:46.297 INFO:teuthology.orchestra.run.smithi123.stdout: "entity": "osd.2", 2024-09-16T15:47:46.297 INFO:teuthology.orchestra.run.smithi123.stdout: "key": "AQC+UehmlCwWKxAAl3iO7ZALWyaorQH2urHKCw==", 2024-09-16T15:47:46.297 INFO:teuthology.orchestra.run.smithi123.stdout: "caps": { 2024-09-16T15:47:46.297 INFO:teuthology.orchestra.run.smithi123.stdout: "mgr": "allow profile osd", 2024-09-16T15:47:46.297 INFO:teuthology.orchestra.run.smithi123.stdout: "mon": "allow profile osd", 2024-09-16T15:47:46.297 INFO:teuthology.orchestra.run.smithi123.stdout: "osd": "allow *" 2024-09-16T15:47:46.297 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-16T15:47:46.297 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:46.298 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-09-16T15:47:46.298 INFO:teuthology.orchestra.run.smithi123.stdout: "entity": "osd.3", 2024-09-16T15:47:46.298 INFO:teuthology.orchestra.run.smithi123.stdout: "key": "AQDCUehmD8VgAhAAneoIeGr3MflSmi+nyavj2A==", 2024-09-16T15:47:46.298 INFO:teuthology.orchestra.run.smithi123.stdout: "caps": { 2024-09-16T15:47:46.298 INFO:teuthology.orchestra.run.smithi123.stdout: "mgr": "allow profile osd", 2024-09-16T15:47:46.298 INFO:teuthology.orchestra.run.smithi123.stdout: "mon": "allow profile osd", 2024-09-16T15:47:46.298 INFO:teuthology.orchestra.run.smithi123.stdout: "osd": "allow *" 2024-09-16T15:47:46.298 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-16T15:47:46.298 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:46.298 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-09-16T15:47:46.298 INFO:teuthology.orchestra.run.smithi123.stdout: "entity": "osd.4", 2024-09-16T15:47:46.298 INFO:teuthology.orchestra.run.smithi123.stdout: "key": "AQDIUehmA9B9EBAAsu0X5LPx1TmatZeHzjnVig==", 2024-09-16T15:47:46.299 INFO:teuthology.orchestra.run.smithi123.stdout: "caps": { 2024-09-16T15:47:46.299 INFO:teuthology.orchestra.run.smithi123.stdout: "mgr": "allow profile osd", 2024-09-16T15:47:46.299 INFO:teuthology.orchestra.run.smithi123.stdout: "mon": "allow profile osd", 2024-09-16T15:47:46.299 INFO:teuthology.orchestra.run.smithi123.stdout: "osd": "allow *" 2024-09-16T15:47:46.299 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-16T15:47:46.299 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:46.299 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-09-16T15:47:46.299 INFO:teuthology.orchestra.run.smithi123.stdout: "entity": "osd.5", 2024-09-16T15:47:46.299 INFO:teuthology.orchestra.run.smithi123.stdout: "key": "AQDLUehmcugdHxAAQMSS1ZH71FUzCx5A+yNnZg==", 2024-09-16T15:47:46.299 INFO:teuthology.orchestra.run.smithi123.stdout: "caps": { 2024-09-16T15:47:46.299 INFO:teuthology.orchestra.run.smithi123.stdout: "mgr": "allow profile osd", 2024-09-16T15:47:46.299 INFO:teuthology.orchestra.run.smithi123.stdout: "mon": "allow profile osd", 2024-09-16T15:47:46.300 INFO:teuthology.orchestra.run.smithi123.stdout: "osd": "allow *" 2024-09-16T15:47:46.300 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-16T15:47:46.300 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:46.300 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-09-16T15:47:46.300 INFO:teuthology.orchestra.run.smithi123.stdout: "entity": "osd.6", 2024-09-16T15:47:46.300 INFO:teuthology.orchestra.run.smithi123.stdout: "key": "AQDOUehmWLwYLhAAhOlQLepcWElsVPMnHlZm4w==", 2024-09-16T15:47:46.300 INFO:teuthology.orchestra.run.smithi123.stdout: "caps": { 2024-09-16T15:47:46.300 INFO:teuthology.orchestra.run.smithi123.stdout: "mgr": "allow profile osd", 2024-09-16T15:47:46.300 INFO:teuthology.orchestra.run.smithi123.stdout: "mon": "allow profile osd", 2024-09-16T15:47:46.300 INFO:teuthology.orchestra.run.smithi123.stdout: "osd": "allow *" 2024-09-16T15:47:46.300 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-16T15:47:46.301 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:46.301 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-09-16T15:47:46.301 INFO:teuthology.orchestra.run.smithi123.stdout: "entity": "osd.7", 2024-09-16T15:47:46.301 INFO:teuthology.orchestra.run.smithi123.stdout: "key": "AQDSUehmipkbABAAQuL/2O7w0tsM1OKeOil97w==", 2024-09-16T15:47:46.301 INFO:teuthology.orchestra.run.smithi123.stdout: "caps": { 2024-09-16T15:47:46.301 INFO:teuthology.orchestra.run.smithi123.stdout: "mgr": "allow profile osd", 2024-09-16T15:47:46.301 INFO:teuthology.orchestra.run.smithi123.stdout: "mon": "allow profile osd", 2024-09-16T15:47:46.301 INFO:teuthology.orchestra.run.smithi123.stdout: "osd": "allow *" 2024-09-16T15:47:46.301 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-16T15:47:46.301 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:46.302 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-09-16T15:47:46.302 INFO:teuthology.orchestra.run.smithi123.stdout: "entity": "client.0", 2024-09-16T15:47:46.302 INFO:teuthology.orchestra.run.smithi123.stdout: "key": "AQC1Uehm7GDTAhAAHWWVIyjVVnTeeozBR8DfFA==", 2024-09-16T15:47:46.302 INFO:teuthology.orchestra.run.smithi123.stdout: "caps": { 2024-09-16T15:47:46.302 INFO:teuthology.orchestra.run.smithi123.stdout: "mds": "allow", 2024-09-16T15:47:46.302 INFO:teuthology.orchestra.run.smithi123.stdout: "mon": "allow r", 2024-09-16T15:47:46.302 INFO:teuthology.orchestra.run.smithi123.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-09-16T15:47:46.302 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-16T15:47:46.302 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:46.302 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-09-16T15:47:46.302 INFO:teuthology.orchestra.run.smithi123.stdout: "entity": "client.1", 2024-09-16T15:47:46.302 INFO:teuthology.orchestra.run.smithi123.stdout: "key": "AQC1UehmFbaWChAAFtRmKucm+pbKtCNvRCTNqA==", 2024-09-16T15:47:46.303 INFO:teuthology.orchestra.run.smithi123.stdout: "caps": { 2024-09-16T15:47:46.303 INFO:teuthology.orchestra.run.smithi123.stdout: "mds": "allow", 2024-09-16T15:47:46.303 INFO:teuthology.orchestra.run.smithi123.stdout: "mgr": "allow r", 2024-09-16T15:47:46.303 INFO:teuthology.orchestra.run.smithi123.stdout: "mon": "allow rw", 2024-09-16T15:47:46.303 INFO:teuthology.orchestra.run.smithi123.stdout: "osd": "allow rwx" 2024-09-16T15:47:46.303 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-16T15:47:46.303 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:46.303 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-09-16T15:47:46.303 INFO:teuthology.orchestra.run.smithi123.stdout: "entity": "client.2", 2024-09-16T15:47:46.303 INFO:teuthology.orchestra.run.smithi123.stdout: "key": "AQC1Uehm1ADrBxAAW/8ayNHI6Wq/YlI4ivFZiw==", 2024-09-16T15:47:46.303 INFO:teuthology.orchestra.run.smithi123.stdout: "caps": { 2024-09-16T15:47:46.303 INFO:teuthology.orchestra.run.smithi123.stdout: "mds": "allow", 2024-09-16T15:47:46.304 INFO:teuthology.orchestra.run.smithi123.stdout: "mgr": "allow r", 2024-09-16T15:47:46.304 INFO:teuthology.orchestra.run.smithi123.stdout: "mon": "allow rw", 2024-09-16T15:47:46.304 INFO:teuthology.orchestra.run.smithi123.stdout: "osd": "allow rwx" 2024-09-16T15:47:46.304 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-16T15:47:46.304 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:46.304 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-09-16T15:47:46.304 INFO:teuthology.orchestra.run.smithi123.stdout: "entity": "client.3", 2024-09-16T15:47:46.304 INFO:teuthology.orchestra.run.smithi123.stdout: "key": "AQC1UehmLv/3DxAAuGloBxSuza95uajpfiZ9MQ==", 2024-09-16T15:47:46.304 INFO:teuthology.orchestra.run.smithi123.stdout: "caps": { 2024-09-16T15:47:46.304 INFO:teuthology.orchestra.run.smithi123.stdout: "mds": "allow", 2024-09-16T15:47:46.304 INFO:teuthology.orchestra.run.smithi123.stdout: "mgr": "allow r", 2024-09-16T15:47:46.305 INFO:teuthology.orchestra.run.smithi123.stdout: "mon": "allow rw", 2024-09-16T15:47:46.305 INFO:teuthology.orchestra.run.smithi123.stdout: "osd": "allow rwx" 2024-09-16T15:47:46.305 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-16T15:47:46.305 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:46.305 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-09-16T15:47:46.305 INFO:teuthology.orchestra.run.smithi123.stdout: "entity": "client.admin", 2024-09-16T15:47:46.305 INFO:teuthology.orchestra.run.smithi123.stdout: "key": "AQC0Uehm1ZDRDBAAhyQ1BmJg1Mw6wQ04I2CbFA==", 2024-09-16T15:47:46.305 INFO:teuthology.orchestra.run.smithi123.stdout: "caps": { 2024-09-16T15:47:46.305 INFO:teuthology.orchestra.run.smithi123.stdout: "mds": "allow *", 2024-09-16T15:47:46.305 INFO:teuthology.orchestra.run.smithi123.stdout: "mgr": "allow *", 2024-09-16T15:47:46.305 INFO:teuthology.orchestra.run.smithi123.stdout: "mon": "allow *", 2024-09-16T15:47:46.305 INFO:teuthology.orchestra.run.smithi123.stdout: "osd": "allow *" 2024-09-16T15:47:46.306 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-16T15:47:46.306 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:46.306 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-09-16T15:47:46.306 INFO:teuthology.orchestra.run.smithi123.stdout: "entity": "mgr.x", 2024-09-16T15:47:46.306 INFO:teuthology.orchestra.run.smithi123.stdout: "key": "AQC0UehmfPilIBAAERlBbduOgAU9ba0n/1tl3g==", 2024-09-16T15:47:46.306 INFO:teuthology.orchestra.run.smithi123.stdout: "caps": { 2024-09-16T15:47:46.306 INFO:teuthology.orchestra.run.smithi123.stdout: "mds": "allow *", 2024-09-16T15:47:46.306 INFO:teuthology.orchestra.run.smithi123.stdout: "mon": "allow profile mgr", 2024-09-16T15:47:46.306 INFO:teuthology.orchestra.run.smithi123.stdout: "osd": "allow *" 2024-09-16T15:47:46.306 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-16T15:47:46.306 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:46.306 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-09-16T15:47:46.307 INFO:teuthology.orchestra.run.smithi123.stdout: "entity": "mgr.y", 2024-09-16T15:47:46.307 INFO:teuthology.orchestra.run.smithi123.stdout: "key": "AQC0UehmkjrlHRAA+3xbQC+SPWGWW1gND7xcAg==", 2024-09-16T15:47:46.307 INFO:teuthology.orchestra.run.smithi123.stdout: "caps": { 2024-09-16T15:47:46.307 INFO:teuthology.orchestra.run.smithi123.stdout: "mds": "allow *", 2024-09-16T15:47:46.307 INFO:teuthology.orchestra.run.smithi123.stdout: "mon": "allow profile mgr", 2024-09-16T15:47:46.307 INFO:teuthology.orchestra.run.smithi123.stdout: "osd": "allow *" 2024-09-16T15:47:46.307 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-16T15:47:46.307 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-16T15:47:46.307 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-09-16T15:47:46.307 INFO:teuthology.orchestra.run.smithi123.stdout:} 2024-09-16T15:47:46.308 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:46.287+0000 7f4b06327640 1 -- 172.21.15.123:0/1752666737 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f4aec072fe0 msgr2=0x7f4aec0754a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:47:46.308 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:46.287+0000 7f4b06327640 1 --2- 172.21.15.123:0/1752666737 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f4aec072fe0 0x7f4aec0754a0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f4af0007fd0 tx=0x7f4af0009040 comp rx=0 tx=0).stop 2024-09-16T15:47:46.308 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:46.287+0000 7f4b06327640 1 -- 172.21.15.123:0/1752666737 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4b000cc6f0 msgr2=0x7f4b0015d120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:47:46.309 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:46.287+0000 7f4b06327640 1 --2- 172.21.15.123:0/1752666737 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4b000cc6f0 0x7f4b0015d120 secure :-1 s=READY pgs=917 cs=0 l=1 rev1=1 crypto rx=0x7f4af4009850 tx=0x7f4af40a55a0 comp rx=0 tx=0).stop 2024-09-16T15:47:46.309 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:46.287+0000 7f4b06327640 1 -- 172.21.15.123:0/1752666737 shutdown_connections 2024-09-16T15:47:46.309 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:46.287+0000 7f4b06327640 1 --2- 172.21.15.123:0/1752666737 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f4aec072fe0 0x7f4aec0754a0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:46.309 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:46.287+0000 7f4b06327640 1 --2- 172.21.15.123:0/1752666737 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f4b0015ac00 0x7f4b00071340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:46.309 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:46.287+0000 7f4b06327640 1 --2- 172.21.15.123:0/1752666737 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4b000cc6f0 0x7f4b0015d120 unknown :-1 s=CLOSED pgs=917 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:46.309 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:46.287+0000 7f4b06327640 1 --2- 172.21.15.123:0/1752666737 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4b000b3100 0x7f4b0014f810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:46.309 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:46.287+0000 7f4b06327640 1 -- 172.21.15.123:0/1752666737 >> 172.21.15.123:0/1752666737 conn(0x7f4b000b2610 msgr2=0x7f4b000b0fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:47:46.309 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:46.287+0000 7f4b06327640 1 -- 172.21.15.123:0/1752666737 shutdown_connections 2024-09-16T15:47:46.309 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:46.287+0000 7f4b06327640 1 -- 172.21.15.123:0/1752666737 wait complete. 2024-09-16T15:47:46.310 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-09-16T15:47:46.435 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:46.427+0000 7fed2212c640 1 -- 172.21.15.123:0/2643656467 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fed1c0cd6e0 msgr2=0x7fed1c0cdb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:47:46.435 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:46.427+0000 7fed2212c640 1 --2- 172.21.15.123:0/2643656467 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fed1c0cd6e0 0x7fed1c0cdb50 secure :-1 s=READY pgs=919 cs=0 l=1 rev1=1 crypto rx=0x7fed04009900 tx=0x7fed0401c850 comp rx=0 tx=0).stop 2024-09-16T15:47:46.435 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:46.431+0000 7fed2212c640 1 -- 172.21.15.123:0/2643656467 shutdown_connections 2024-09-16T15:47:46.435 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:46.431+0000 7fed2212c640 1 --2- 172.21.15.123:0/2643656467 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fed1c15e820 0x7fed1c15ecb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:46.435 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:46.431+0000 7fed2212c640 1 --2- 172.21.15.123:0/2643656467 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fed1c0cd6e0 0x7fed1c0cdb50 unknown :-1 s=CLOSED pgs=919 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:46.435 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:46.431+0000 7fed2212c640 1 -- 172.21.15.123:0/2643656467 >> 172.21.15.123:0/2643656467 conn(0x7fed1c0cfb10 msgr2=0x7fed1c0cff20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:47:46.435 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:46.431+0000 7fed2212c640 1 -- 172.21.15.123:0/2643656467 shutdown_connections 2024-09-16T15:47:46.435 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:46.431+0000 7fed2212c640 1 -- 172.21.15.123:0/2643656467 wait complete. 2024-09-16T15:47:46.437 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:46.431+0000 7fed2212c640 1 Processor -- start 2024-09-16T15:47:46.437 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:46.431+0000 7fed2212c640 1 -- start start 2024-09-16T15:47:46.438 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:46.431+0000 7fed2212c640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fed1c0cd6e0 0x7fed1c156970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:47:46.438 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:46.431+0000 7fed2212c640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fed1c15e160 0x7fed1c158730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:47:46.438 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:46.431+0000 7fed2212c640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fed1c15e820 0x7fed1c1579f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:47:46.438 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:46.431+0000 7fed2212c640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fed1c0cf5b0 con 0x7fed1c0cd6e0 2024-09-16T15:47:46.438 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:46.431+0000 7fed2212c640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fed1c0cf430 con 0x7fed1c15e820 2024-09-16T15:47:46.438 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:46.431+0000 7fed2212c640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fed1c1683d0 con 0x7fed1c15e160 2024-09-16T15:47:46.438 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:46.431+0000 7fed1b7fe640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fed1c0cd6e0 0x7fed1c156970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:47:46.438 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:46.431+0000 7fed1b7fe640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fed1c0cd6e0 0x7fed1c156970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:48616/0 (socket says 172.21.15.123:48616) 2024-09-16T15:47:46.438 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:46.431+0000 7fed1b7fe640 1 -- 172.21.15.123:0/19987881 learned_addr learned my addr 172.21.15.123:0/19987881 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:47:46.438 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:46.431+0000 7fed1b7fe640 1 -- 172.21.15.123:0/19987881 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fed1c15e160 msgr2=0x7fed1c158730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:47:46.439 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:46.431+0000 7fed1b7fe640 1 --2- 172.21.15.123:0/19987881 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fed1c15e160 0x7fed1c158730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:46.439 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:46.431+0000 7fed1b7fe640 1 -- 172.21.15.123:0/19987881 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fed1c15e820 msgr2=0x7fed1c1579f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:47:46.439 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:46.431+0000 7fed1b7fe640 1 --2- 172.21.15.123:0/19987881 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fed1c15e820 0x7fed1c1579f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:46.439 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:46.431+0000 7fed1b7fe640 1 -- 172.21.15.123:0/19987881 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fed1c156eb0 con 0x7fed1c0cd6e0 2024-09-16T15:47:46.439 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:46.431+0000 7fed1b7fe640 1 --2- 172.21.15.123:0/19987881 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fed1c0cd6e0 0x7fed1c156970 secure :-1 s=READY pgs=920 cs=0 l=1 rev1=1 crypto rx=0x7fed0c00c530 tx=0x7fed0c00c9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:47:46.439 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:46.431+0000 7fed18ff9640 1 -- 172.21.15.123:0/19987881 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed0c017070 con 0x7fed1c0cd6e0 2024-09-16T15:47:46.439 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:46.431+0000 7fed18ff9640 1 -- 172.21.15.123:0/19987881 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fed0c0053c0 con 0x7fed1c0cd6e0 2024-09-16T15:47:46.439 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:46.431+0000 7fed2212c640 1 -- 172.21.15.123:0/19987881 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fed1c159030 con 0x7fed1c0cd6e0 2024-09-16T15:47:46.439 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:46.431+0000 7fed18ff9640 1 -- 172.21.15.123:0/19987881 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed0c012890 con 0x7fed1c0cd6e0 2024-09-16T15:47:46.439 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:46.431+0000 7fed2212c640 1 -- 172.21.15.123:0/19987881 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fed1c0c4620 con 0x7fed1c0cd6e0 2024-09-16T15:47:46.440 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:46.435+0000 7fed2212c640 1 -- 172.21.15.123:0/19987881 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fed1c0cdb50 con 0x7fed1c0cd6e0 2024-09-16T15:47:46.440 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:46.435+0000 7fed18ff9640 1 -- 172.21.15.123:0/19987881 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 16) ==== 97182+0+0 (secure 0 0 0) 0x7fed0c01a070 con 0x7fed1c0cd6e0 2024-09-16T15:47:46.441 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:46.435+0000 7fed18ff9640 1 --2- 172.21.15.123:0/19987881 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fecfc072fe0 0x7fecfc0754a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:47:46.441 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:46.435+0000 7fed18ff9640 1 -- 172.21.15.123:0/19987881 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 4857+0+0 (secure 0 0 0) 0x7fed0c092fb0 con 0x7fed1c0cd6e0 2024-09-16T15:47:46.441 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:46.435+0000 7fed12ffd640 1 --2- 172.21.15.123:0/19987881 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fecfc072fe0 0x7fecfc0754a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:47:46.441 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:46.435+0000 7fed12ffd640 1 --2- 172.21.15.123:0/19987881 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fecfc072fe0 0x7fecfc0754a0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fed040af000 tx=0x7fed040195f0 comp rx=0 tx=0).ready entity=mgr.4150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:47:46.443 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:46.439+0000 7fed18ff9640 1 -- 172.21.15.123:0/19987881 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7fed0c05e3c0 con 0x7fed1c0cd6e0 2024-09-16T15:47:46.623 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:46.619+0000 7fed2212c640 1 -- 172.21.15.123:0/19987881 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) -- 0x7fed1c0c3350 con 0x7fed1c0cd6e0 2024-09-16T15:47:46.624 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:46.619+0000 7fed18ff9640 1 -- 172.21.15.123:0/19987881 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v81) ==== 84+0+65 (secure 0 0 0) 0x7fed0c05dcc0 con 0x7fed1c0cd6e0 2024-09-16T15:47:46.624 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-09-16T15:47:46.624 INFO:teuthology.orchestra.run.smithi123.stdout:[ 2024-09-16T15:47:46.624 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-09-16T15:47:46.624 INFO:teuthology.orchestra.run.smithi123.stdout: "poolnum": 1, 2024-09-16T15:47:46.624 INFO:teuthology.orchestra.run.smithi123.stdout: "poolname": ".mgr" 2024-09-16T15:47:46.625 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-16T15:47:46.625 INFO:teuthology.orchestra.run.smithi123.stdout:] 2024-09-16T15:47:46.627 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:46.623+0000 7fed2212c640 1 -- 172.21.15.123:0/19987881 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fecfc072fe0 msgr2=0x7fecfc0754a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:47:46.627 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:46.623+0000 7fed2212c640 1 --2- 172.21.15.123:0/19987881 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fecfc072fe0 0x7fecfc0754a0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fed040af000 tx=0x7fed040195f0 comp rx=0 tx=0).stop 2024-09-16T15:47:46.627 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:46.623+0000 7fed2212c640 1 -- 172.21.15.123:0/19987881 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fed1c0cd6e0 msgr2=0x7fed1c156970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:47:46.627 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:46.623+0000 7fed2212c640 1 --2- 172.21.15.123:0/19987881 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fed1c0cd6e0 0x7fed1c156970 secure :-1 s=READY pgs=920 cs=0 l=1 rev1=1 crypto rx=0x7fed0c00c530 tx=0x7fed0c00c9f0 comp rx=0 tx=0).stop 2024-09-16T15:47:46.627 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:46.623+0000 7fed2212c640 1 -- 172.21.15.123:0/19987881 shutdown_connections 2024-09-16T15:47:46.627 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:46.623+0000 7fed2212c640 1 --2- 172.21.15.123:0/19987881 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fecfc072fe0 0x7fecfc0754a0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:46.627 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:46.623+0000 7fed2212c640 1 --2- 172.21.15.123:0/19987881 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fed1c15e820 0x7fed1c1579f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:46.627 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:46.623+0000 7fed2212c640 1 --2- 172.21.15.123:0/19987881 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fed1c15e160 0x7fed1c158730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:46.627 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:46.623+0000 7fed2212c640 1 --2- 172.21.15.123:0/19987881 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fed1c0cd6e0 0x7fed1c156970 unknown :-1 s=CLOSED pgs=920 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:46.628 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:46.623+0000 7fed2212c640 1 -- 172.21.15.123:0/19987881 >> 172.21.15.123:0/19987881 conn(0x7fed1c0cfb10 msgr2=0x7fed1c06eee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:47:46.628 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:46.623+0000 7fed2212c640 1 -- 172.21.15.123:0/19987881 shutdown_connections 2024-09-16T15:47:46.628 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:46.623+0000 7fed2212c640 1 -- 172.21.15.123:0/19987881 wait complete. 2024-09-16T15:47:46.643 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-09-16T15:47:46.643 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-09-16T15:47:46.773 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:46.767+0000 7f8bedb38640 1 -- 172.21.15.123:0/426097844 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8be8161390 msgr2=0x7f8be80d0630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:47:46.773 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:46.767+0000 7f8bedb38640 1 --2- 172.21.15.123:0/426097844 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8be8161390 0x7f8be80d0630 secure :-1 s=READY pgs=922 cs=0 l=1 rev1=1 crypto rx=0x7f8bd8009930 tx=0x7f8bd801c8b0 comp rx=0 tx=0).stop 2024-09-16T15:47:46.773 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:46.767+0000 7f8bedb38640 1 -- 172.21.15.123:0/426097844 shutdown_connections 2024-09-16T15:47:46.773 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:46.767+0000 7f8bedb38640 1 --2- 172.21.15.123:0/426097844 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8be8161390 0x7f8be80d0630 unknown :-1 s=CLOSED pgs=922 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:46.774 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:46.767+0000 7f8bedb38640 1 -- 172.21.15.123:0/426097844 >> 172.21.15.123:0/426097844 conn(0x7f8be80d3e30 msgr2=0x7f8be80d4240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:47:46.774 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:46.767+0000 7f8bedb38640 1 -- 172.21.15.123:0/426097844 shutdown_connections 2024-09-16T15:47:46.774 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:46.767+0000 7f8bedb38640 1 -- 172.21.15.123:0/426097844 wait complete. 2024-09-16T15:47:46.776 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:46.767+0000 7f8bedb38640 1 Processor -- start 2024-09-16T15:47:46.776 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:46.767+0000 7f8bedb38640 1 -- start start 2024-09-16T15:47:46.776 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:46.771+0000 7f8bedb38640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8be80d0b70 0x7f8be815ffa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:47:46.776 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:46.771+0000 7f8bedb38640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f8be80d2a10 0x7f8be81604e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:47:46.776 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:46.771+0000 7f8bedb38640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f8be8161390 0x7f8be814f120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:47:46.776 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:46.771+0000 7f8bedb38640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f8be816c700 con 0x7f8be80d0b70 2024-09-16T15:47:46.776 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:46.771+0000 7f8bedb38640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f8be816c580 con 0x7f8be80d2a10 2024-09-16T15:47:46.776 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:46.771+0000 7f8bedb38640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f8be816c880 con 0x7f8be8161390 2024-09-16T15:47:46.776 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:46.771+0000 7f8be77fe640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8be80d0b70 0x7f8be815ffa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:47:46.776 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:46.771+0000 7f8be77fe640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8be80d0b70 0x7f8be815ffa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:48646/0 (socket says 172.21.15.123:48646) 2024-09-16T15:47:46.777 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:46.771+0000 7f8be77fe640 1 -- 172.21.15.123:0/2768508784 learned_addr learned my addr 172.21.15.123:0/2768508784 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:47:46.777 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:46.771+0000 7f8be77fe640 1 -- 172.21.15.123:0/2768508784 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f8be8161390 msgr2=0x7f8be814f120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:47:46.777 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:46.771+0000 7f8be77fe640 1 --2- 172.21.15.123:0/2768508784 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f8be8161390 0x7f8be814f120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:46.777 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:46.771+0000 7f8be77fe640 1 -- 172.21.15.123:0/2768508784 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f8be80d2a10 msgr2=0x7f8be81604e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:47:46.777 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:46.771+0000 7f8be77fe640 1 --2- 172.21.15.123:0/2768508784 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f8be80d2a10 0x7f8be81604e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:46.777 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:46.771+0000 7f8be77fe640 1 -- 172.21.15.123:0/2768508784 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8be814e080 con 0x7f8be80d0b70 2024-09-16T15:47:46.777 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:46.771+0000 7f8be77fe640 1 --2- 172.21.15.123:0/2768508784 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8be80d0b70 0x7f8be815ffa0 secure :-1 s=READY pgs=923 cs=0 l=1 rev1=1 crypto rx=0x7f8bd8009850 tx=0x7f8bd8004150 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:47:46.777 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:46.771+0000 7f8bcffff640 1 -- 172.21.15.123:0/2768508784 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8bd80bc020 con 0x7f8be80d0b70 2024-09-16T15:47:46.777 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:46.771+0000 7f8bcffff640 1 -- 172.21.15.123:0/2768508784 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f8bd8003600 con 0x7f8be80d0b70 2024-09-16T15:47:46.777 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:46.771+0000 7f8bedb38640 1 -- 172.21.15.123:0/2768508784 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8be814e360 con 0x7f8be80d0b70 2024-09-16T15:47:46.777 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:46.771+0000 7f8bcffff640 1 -- 172.21.15.123:0/2768508784 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8bd80b2040 con 0x7f8be80d0b70 2024-09-16T15:47:46.778 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:46.771+0000 7f8bedb38640 1 -- 172.21.15.123:0/2768508784 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8be814e920 con 0x7f8be80d0b70 2024-09-16T15:47:46.778 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:46.771+0000 7f8bedb38640 1 -- 172.21.15.123:0/2768508784 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8be80d0630 con 0x7f8be80d0b70 2024-09-16T15:47:46.779 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:46.771+0000 7f8bcffff640 1 -- 172.21.15.123:0/2768508784 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 16) ==== 97182+0+0 (secure 0 0 0) 0x7f8bd80bf070 con 0x7f8be80d0b70 2024-09-16T15:47:46.779 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:46.771+0000 7f8bcffff640 1 --2- 172.21.15.123:0/2768508784 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f8bd4072fe0 0x7f8bd40754a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:47:46.779 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:46.771+0000 7f8bcffff640 1 -- 172.21.15.123:0/2768508784 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 4857+0+0 (secure 0 0 0) 0x7f8bd812dc60 con 0x7f8be80d0b70 2024-09-16T15:47:46.779 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:46.771+0000 7f8be6ffd640 1 --2- 172.21.15.123:0/2768508784 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f8bd4072fe0 0x7f8bd40754a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:47:46.779 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:46.771+0000 7f8be6ffd640 1 --2- 172.21.15.123:0/2768508784 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f8bd4072fe0 0x7f8bd40754a0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f8bdc000f70 tx=0x7f8bdc0079e0 comp rx=0 tx=0).ready entity=mgr.4150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:47:46.782 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:46.775+0000 7f8bcffff640 1 -- 172.21.15.123:0/2768508784 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f8bd80164d0 con 0x7f8be80d0b70 2024-09-16T15:47:46.952 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:46.947+0000 7f8bedb38640 1 -- 172.21.15.123:0/2768508784 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) -- 0x7f8be80b54f0 con 0x7f8be80d0b70 2024-09-16T15:47:47.899 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:47.895+0000 7f8bcffff640 1 -- 172.21.15.123:0/2768508784 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v82) ==== 138+0+0 (secure 0 0 0) 0x7f8bd80f8fe0 con 0x7f8be80d0b70 2024-09-16T15:47:47.899 INFO:teuthology.orchestra.run.smithi123.stderr:pool 'cephfs_metadata' created 2024-09-16T15:47:47.904 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:47.899+0000 7f8bedb38640 1 -- 172.21.15.123:0/2768508784 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f8bd4072fe0 msgr2=0x7f8bd40754a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:47:47.904 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:47.899+0000 7f8bedb38640 1 --2- 172.21.15.123:0/2768508784 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f8bd4072fe0 0x7f8bd40754a0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f8bdc000f70 tx=0x7f8bdc0079e0 comp rx=0 tx=0).stop 2024-09-16T15:47:47.904 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:47.899+0000 7f8bedb38640 1 -- 172.21.15.123:0/2768508784 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8be80d0b70 msgr2=0x7f8be815ffa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:47:47.904 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:47.899+0000 7f8bedb38640 1 --2- 172.21.15.123:0/2768508784 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8be80d0b70 0x7f8be815ffa0 secure :-1 s=READY pgs=923 cs=0 l=1 rev1=1 crypto rx=0x7f8bd8009850 tx=0x7f8bd8004150 comp rx=0 tx=0).stop 2024-09-16T15:47:47.904 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:47.899+0000 7f8bedb38640 1 -- 172.21.15.123:0/2768508784 shutdown_connections 2024-09-16T15:47:47.904 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:47.899+0000 7f8bedb38640 1 --2- 172.21.15.123:0/2768508784 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f8bd4072fe0 0x7f8bd40754a0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:47.904 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:47.899+0000 7f8bedb38640 1 --2- 172.21.15.123:0/2768508784 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f8be8161390 0x7f8be814f120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:47.904 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:47.899+0000 7f8bedb38640 1 --2- 172.21.15.123:0/2768508784 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f8be80d2a10 0x7f8be81604e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:47.904 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:47.899+0000 7f8bedb38640 1 --2- 172.21.15.123:0/2768508784 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8be80d0b70 0x7f8be815ffa0 unknown :-1 s=CLOSED pgs=923 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:47.905 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:47.899+0000 7f8bedb38640 1 -- 172.21.15.123:0/2768508784 >> 172.21.15.123:0/2768508784 conn(0x7f8be80d3e30 msgr2=0x7f8be80d4210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:47:47.905 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:47.899+0000 7f8bedb38640 1 -- 172.21.15.123:0/2768508784 shutdown_connections 2024-09-16T15:47:47.905 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:47.899+0000 7f8bedb38640 1 -- 172.21.15.123:0/2768508784 wait complete. 2024-09-16T15:47:47.930 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-09-16T15:47:48.070 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:48.063+0000 7fac335dd640 1 -- 172.21.15.123:0/232811058 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fac18004810 msgr2=0x7fac18004c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:47:48.070 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:48.063+0000 7fac335dd640 1 --2- 172.21.15.123:0/232811058 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fac18004810 0x7fac18004c90 secure :-1 s=READY pgs=925 cs=0 l=1 rev1=1 crypto rx=0x7fac1c001ec0 tx=0x7fac1c01f6e0 comp rx=0 tx=0).stop 2024-09-16T15:47:48.070 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:48.063+0000 7fac335dd640 1 -- 172.21.15.123:0/232811058 shutdown_connections 2024-09-16T15:47:48.070 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:48.063+0000 7fac335dd640 1 --2- 172.21.15.123:0/232811058 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fac18004810 0x7fac18004c90 unknown :-1 s=CLOSED pgs=925 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:48.070 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:48.063+0000 7fac335dd640 1 -- 172.21.15.123:0/232811058 >> 172.21.15.123:0/232811058 conn(0x7fac2c0b1710 msgr2=0x7fac2c0b1b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:47:48.070 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:48.063+0000 7fac335dd640 1 -- 172.21.15.123:0/232811058 shutdown_connections 2024-09-16T15:47:48.071 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:48.063+0000 7fac335dd640 1 -- 172.21.15.123:0/232811058 wait complete. 2024-09-16T15:47:48.073 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:48.067+0000 7fac335dd640 1 Processor -- start 2024-09-16T15:47:48.073 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:48.067+0000 7fac335dd640 1 -- start start 2024-09-16T15:47:48.073 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:48.067+0000 7fac335dd640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fac18004810 0x7fac2c0c95e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:47:48.073 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:48.067+0000 7fac335dd640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fac2c150660 0x7fac2c0c8090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:47:48.073 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:48.067+0000 7fac335dd640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fac2c1f5620 0x7fac2c0c4e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:47:48.073 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:48.067+0000 7fac335dd640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fac2c0cd890 con 0x7fac18004810 2024-09-16T15:47:48.074 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:48.067+0000 7fac335dd640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fac2c0cd710 con 0x7fac2c1f5620 2024-09-16T15:47:48.074 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:48.067+0000 7fac335dd640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fac2c0cda10 con 0x7fac2c150660 2024-09-16T15:47:48.074 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:48.067+0000 7fac31379640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fac18004810 0x7fac2c0c95e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:47:48.074 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:48.067+0000 7fac31379640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fac18004810 0x7fac2c0c95e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:48674/0 (socket says 172.21.15.123:48674) 2024-09-16T15:47:48.074 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:48.067+0000 7fac31379640 1 -- 172.21.15.123:0/2007045404 learned_addr learned my addr 172.21.15.123:0/2007045404 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:47:48.074 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:48.067+0000 7fac30b78640 1 --2- 172.21.15.123:0/2007045404 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fac2c150660 0x7fac2c0c8090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:47:48.074 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:48.067+0000 7fac31b7a640 1 --2- 172.21.15.123:0/2007045404 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fac2c1f5620 0x7fac2c0c4e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:47:48.074 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:48.067+0000 7fac31379640 1 -- 172.21.15.123:0/2007045404 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fac2c150660 msgr2=0x7fac2c0c8090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:47:48.074 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:48.067+0000 7fac31379640 1 --2- 172.21.15.123:0/2007045404 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fac2c150660 0x7fac2c0c8090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:48.074 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:48.067+0000 7fac31379640 1 -- 172.21.15.123:0/2007045404 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fac2c1f5620 msgr2=0x7fac2c0c4e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:47:48.074 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:48.067+0000 7fac31379640 1 --2- 172.21.15.123:0/2007045404 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fac2c1f5620 0x7fac2c0c4e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:48.075 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:48.067+0000 7fac31379640 1 -- 172.21.15.123:0/2007045404 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fac2c0c6ff0 con 0x7fac18004810 2024-09-16T15:47:48.075 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:48.067+0000 7fac31379640 1 --2- 172.21.15.123:0/2007045404 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fac18004810 0x7fac2c0c95e0 secure :-1 s=READY pgs=926 cs=0 l=1 rev1=1 crypto rx=0x7fac2800b530 tx=0x7fac2800df90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:47:48.075 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:48.067+0000 7fac21ffb640 1 -- 172.21.15.123:0/2007045404 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac28015030 con 0x7fac18004810 2024-09-16T15:47:48.075 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:48.067+0000 7fac21ffb640 1 -- 172.21.15.123:0/2007045404 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fac280063f0 con 0x7fac18004810 2024-09-16T15:47:48.075 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:48.067+0000 7fac21ffb640 1 -- 172.21.15.123:0/2007045404 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac28003b20 con 0x7fac18004810 2024-09-16T15:47:48.075 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:48.067+0000 7fac335dd640 1 -- 172.21.15.123:0/2007045404 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fac2c0c72d0 con 0x7fac18004810 2024-09-16T15:47:48.075 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:48.067+0000 7fac335dd640 1 -- 172.21.15.123:0/2007045404 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fac2c0c7890 con 0x7fac18004810 2024-09-16T15:47:48.076 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:48.067+0000 7fac335dd640 1 -- 172.21.15.123:0/2007045404 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fac2c0cc550 con 0x7fac18004810 2024-09-16T15:47:48.076 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:48.071+0000 7fac21ffb640 1 -- 172.21.15.123:0/2007045404 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 16) ==== 97182+0+0 (secure 0 0 0) 0x7fac28018070 con 0x7fac18004810 2024-09-16T15:47:48.076 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:48.071+0000 7fac21ffb640 1 --2- 172.21.15.123:0/2007045404 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fac10072fe0 0x7fac100754a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:47:48.076 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:48.071+0000 7fac30b78640 1 --2- 172.21.15.123:0/2007045404 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fac10072fe0 0x7fac100754a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:47:48.076 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:48.071+0000 7fac21ffb640 1 -- 172.21.15.123:0/2007045404 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 5227+0+0 (secure 0 0 0) 0x7fac280941a0 con 0x7fac18004810 2024-09-16T15:47:48.076 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:48.071+0000 7fac30b78640 1 --2- 172.21.15.123:0/2007045404 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fac10072fe0 0x7fac100754a0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fac1c0b0000 tx=0x7fac1c01c520 comp rx=0 tx=0).ready entity=mgr.4150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:47:48.079 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:48.075+0000 7fac21ffb640 1 -- 172.21.15.123:0/2007045404 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7fac2805f3b0 con 0x7fac18004810 2024-09-16T15:47:48.239 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:48.235+0000 7fac335dd640 1 -- 172.21.15.123:0/2007045404 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) -- 0x7fac2c15cbd0 con 0x7fac18004810 2024-09-16T15:47:48.910 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:48.903+0000 7fac21ffb640 1 -- 172.21.15.123:0/2007045404 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v83) ==== 170+0+0 (secure 0 0 0) 0x7fac2805ecb0 con 0x7fac18004810 2024-09-16T15:47:48.911 INFO:teuthology.orchestra.run.smithi123.stderr:pool 'cephfs_data' created 2024-09-16T15:47:48.915 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:48.911+0000 7fac335dd640 1 -- 172.21.15.123:0/2007045404 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fac10072fe0 msgr2=0x7fac100754a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:47:48.916 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:48.911+0000 7fac335dd640 1 --2- 172.21.15.123:0/2007045404 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fac10072fe0 0x7fac100754a0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fac1c0b0000 tx=0x7fac1c01c520 comp rx=0 tx=0).stop 2024-09-16T15:47:48.916 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:48.911+0000 7fac335dd640 1 -- 172.21.15.123:0/2007045404 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fac18004810 msgr2=0x7fac2c0c95e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:47:48.916 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:48.911+0000 7fac335dd640 1 --2- 172.21.15.123:0/2007045404 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fac18004810 0x7fac2c0c95e0 secure :-1 s=READY pgs=926 cs=0 l=1 rev1=1 crypto rx=0x7fac2800b530 tx=0x7fac2800df90 comp rx=0 tx=0).stop 2024-09-16T15:47:48.916 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:48.911+0000 7fac335dd640 1 -- 172.21.15.123:0/2007045404 shutdown_connections 2024-09-16T15:47:48.916 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:48.911+0000 7fac335dd640 1 --2- 172.21.15.123:0/2007045404 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fac10072fe0 0x7fac100754a0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:48.916 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:48.911+0000 7fac335dd640 1 --2- 172.21.15.123:0/2007045404 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fac2c1f5620 0x7fac2c0c4e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:48.916 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:48.911+0000 7fac335dd640 1 --2- 172.21.15.123:0/2007045404 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fac2c150660 0x7fac2c0c8090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:48.916 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:48.911+0000 7fac335dd640 1 --2- 172.21.15.123:0/2007045404 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fac18004810 0x7fac2c0c95e0 unknown :-1 s=CLOSED pgs=926 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:48.916 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:48.911+0000 7fac335dd640 1 -- 172.21.15.123:0/2007045404 >> 172.21.15.123:0/2007045404 conn(0x7fac2c0b1710 msgr2=0x7fac2c0b1af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:47:48.916 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:48.911+0000 7fac335dd640 1 -- 172.21.15.123:0/2007045404 shutdown_connections 2024-09-16T15:47:48.916 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:48.911+0000 7fac335dd640 1 -- 172.21.15.123:0/2007045404 wait complete. 2024-09-16T15:47:48.950 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-09-16T15:47:49.075 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:49.071+0000 7f9c366dd640 1 -- 172.21.15.123:0/1787171077 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9c18005220 msgr2=0x7f9c1800b9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:47:49.075 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:49.071+0000 7f9c366dd640 1 --2- 172.21.15.123:0/1787171077 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9c18005220 0x7f9c1800b9d0 secure :-1 s=READY pgs=928 cs=0 l=1 rev1=1 crypto rx=0x7f9c2800c2e0 tx=0x7f9c2801cf60 comp rx=0 tx=0).stop 2024-09-16T15:47:49.076 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:49.071+0000 7f9c366dd640 1 -- 172.21.15.123:0/1787171077 shutdown_connections 2024-09-16T15:47:49.076 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:49.071+0000 7f9c366dd640 1 --2- 172.21.15.123:0/1787171077 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9c18005220 0x7f9c1800b9d0 unknown :-1 s=CLOSED pgs=928 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:49.076 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:49.071+0000 7f9c366dd640 1 -- 172.21.15.123:0/1787171077 >> 172.21.15.123:0/1787171077 conn(0x7f9c300b1710 msgr2=0x7f9c300b1b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:47:49.076 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:49.071+0000 7f9c366dd640 1 -- 172.21.15.123:0/1787171077 shutdown_connections 2024-09-16T15:47:49.076 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:49.071+0000 7f9c366dd640 1 -- 172.21.15.123:0/1787171077 wait complete. 2024-09-16T15:47:49.076 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:49.071+0000 7f9c366dd640 1 Processor -- start 2024-09-16T15:47:49.076 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:49.071+0000 7f9c366dd640 1 -- start start 2024-09-16T15:47:49.078 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:49.071+0000 7f9c366dd640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f9c18005220 0x7f9c3014df40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:47:49.078 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:49.071+0000 7f9c366dd640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f9c3015aa30 0x7f9c30152d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:47:49.078 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:49.071+0000 7f9c366dd640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9c30154a50 0x7f9c30154ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:47:49.078 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:49.071+0000 7f9c366dd640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f9c300cef50 con 0x7f9c30154a50 2024-09-16T15:47:49.079 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:49.071+0000 7f9c366dd640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f9c300cedd0 con 0x7f9c18005220 2024-09-16T15:47:49.079 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:49.071+0000 7f9c366dd640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f9c300cf0d0 con 0x7f9c3015aa30 2024-09-16T15:47:49.079 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:49.071+0000 7f9c34c7a640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9c30154a50 0x7f9c30154ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:47:49.079 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:49.071+0000 7f9c34c7a640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9c30154a50 0x7f9c30154ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:48698/0 (socket says 172.21.15.123:48698) 2024-09-16T15:47:49.079 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:49.071+0000 7f9c34c7a640 1 -- 172.21.15.123:0/306168889 learned_addr learned my addr 172.21.15.123:0/306168889 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:47:49.079 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:49.071+0000 7f9c2ffff640 1 --2- 172.21.15.123:0/306168889 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f9c18005220 0x7f9c3014df40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:47:49.079 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:49.071+0000 7f9c2f7fe640 1 --2- 172.21.15.123:0/306168889 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f9c3015aa30 0x7f9c30152d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:47:49.079 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:49.071+0000 7f9c34c7a640 1 -- 172.21.15.123:0/306168889 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f9c3015aa30 msgr2=0x7f9c30152d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:47:49.079 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:49.071+0000 7f9c34c7a640 1 --2- 172.21.15.123:0/306168889 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f9c3015aa30 0x7f9c30152d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:49.079 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:49.071+0000 7f9c34c7a640 1 -- 172.21.15.123:0/306168889 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f9c18005220 msgr2=0x7f9c3014df40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:47:49.079 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:49.071+0000 7f9c34c7a640 1 --2- 172.21.15.123:0/306168889 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f9c18005220 0x7f9c3014df40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:49.079 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:49.071+0000 7f9c34c7a640 1 -- 172.21.15.123:0/306168889 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9c301535c0 con 0x7f9c30154a50 2024-09-16T15:47:49.080 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:49.075+0000 7f9c34c7a640 1 --2- 172.21.15.123:0/306168889 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9c30154a50 0x7f9c30154ed0 secure :-1 s=READY pgs=929 cs=0 l=1 rev1=1 crypto rx=0x7f9c2801b660 tx=0x7f9c28006c60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:47:49.080 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:49.075+0000 7f9c2cff9640 1 -- 172.21.15.123:0/306168889 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c28006420 con 0x7f9c30154a50 2024-09-16T15:47:49.080 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:49.075+0000 7f9c2cff9640 1 -- 172.21.15.123:0/306168889 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f9c28001500 con 0x7f9c30154a50 2024-09-16T15:47:49.080 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:49.075+0000 7f9c366dd640 1 -- 172.21.15.123:0/306168889 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9c3014e480 con 0x7f9c30154a50 2024-09-16T15:47:49.080 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:49.075+0000 7f9c2cff9640 1 -- 172.21.15.123:0/306168889 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c28003860 con 0x7f9c30154a50 2024-09-16T15:47:49.080 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:49.075+0000 7f9c366dd640 1 -- 172.21.15.123:0/306168889 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9c300c8dd0 con 0x7f9c30154a50 2024-09-16T15:47:49.081 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:49.075+0000 7f9c366dd640 1 -- 172.21.15.123:0/306168889 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9c3006fe10 con 0x7f9c30154a50 2024-09-16T15:47:49.081 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:49.075+0000 7f9c2cff9640 1 -- 172.21.15.123:0/306168889 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 16) ==== 97182+0+0 (secure 0 0 0) 0x7f9c28003a00 con 0x7f9c30154a50 2024-09-16T15:47:49.081 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:49.075+0000 7f9c2cff9640 1 --2- 172.21.15.123:0/306168889 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f9c1c072fe0 0x7f9c1c0754a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:47:49.081 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:49.075+0000 7f9c2cff9640 1 -- 172.21.15.123:0/306168889 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 5609+0+0 (secure 0 0 0) 0x7f9c2812e000 con 0x7f9c30154a50 2024-09-16T15:47:49.081 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:49.075+0000 7f9c2ffff640 1 --2- 172.21.15.123:0/306168889 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f9c1c072fe0 0x7f9c1c0754a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:47:49.081 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:49.075+0000 7f9c2ffff640 1 --2- 172.21.15.123:0/306168889 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f9c1c072fe0 0x7f9c1c0754a0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f9c24009cb0 tx=0x7f9c240093d0 comp rx=0 tx=0).ready entity=mgr.4150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:47:49.084 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:49.079+0000 7f9c2cff9640 1 -- 172.21.15.123:0/306168889 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f9c28016d10 con 0x7f9c30154a50 2024-09-16T15:47:49.293 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:49.287+0000 7f9c366dd640 1 -- 172.21.15.123:0/306168889 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) -- 0x7f9c30156650 con 0x7f9c30154a50 2024-09-16T15:47:49.940 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:49.935+0000 7f9c2cff9640 1 -- 172.21.15.123:0/306168889 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 12 and data pool 13 v49) ==== 174+0+0 (secure 0 0 0) 0x7f9c280f8bb0 con 0x7f9c30154a50 2024-09-16T15:47:49.948 INFO:teuthology.orchestra.run.smithi123.stderr:new fs with metadata pool 12 and data pool 13 2024-09-16T15:47:49.968 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:49.963+0000 7f9c366dd640 1 -- 172.21.15.123:0/306168889 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f9c1c072fe0 msgr2=0x7f9c1c0754a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:47:49.969 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:49.963+0000 7f9c366dd640 1 --2- 172.21.15.123:0/306168889 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f9c1c072fe0 0x7f9c1c0754a0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f9c24009cb0 tx=0x7f9c240093d0 comp rx=0 tx=0).stop 2024-09-16T15:47:49.969 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:49.963+0000 7f9c366dd640 1 -- 172.21.15.123:0/306168889 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9c30154a50 msgr2=0x7f9c30154ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:47:49.969 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:49.963+0000 7f9c366dd640 1 --2- 172.21.15.123:0/306168889 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9c30154a50 0x7f9c30154ed0 secure :-1 s=READY pgs=929 cs=0 l=1 rev1=1 crypto rx=0x7f9c2801b660 tx=0x7f9c28006c60 comp rx=0 tx=0).stop 2024-09-16T15:47:49.969 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:49.963+0000 7f9c366dd640 1 -- 172.21.15.123:0/306168889 shutdown_connections 2024-09-16T15:47:49.969 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:49.963+0000 7f9c366dd640 1 --2- 172.21.15.123:0/306168889 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f9c1c072fe0 0x7f9c1c0754a0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:49.969 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:49.963+0000 7f9c366dd640 1 --2- 172.21.15.123:0/306168889 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9c30154a50 0x7f9c30154ed0 unknown :-1 s=CLOSED pgs=929 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:49.969 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:49.963+0000 7f9c366dd640 1 --2- 172.21.15.123:0/306168889 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f9c3015aa30 0x7f9c30152d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:49.969 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:49.963+0000 7f9c366dd640 1 --2- 172.21.15.123:0/306168889 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f9c18005220 0x7f9c3014df40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:49.969 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:49.963+0000 7f9c366dd640 1 -- 172.21.15.123:0/306168889 >> 172.21.15.123:0/306168889 conn(0x7f9c300b1710 msgr2=0x7f9c300cc510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:47:49.969 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:49.963+0000 7f9c366dd640 1 -- 172.21.15.123:0/306168889 shutdown_connections 2024-09-16T15:47:49.969 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:49.963+0000 7f9c366dd640 1 -- 172.21.15.123:0/306168889 wait complete. 2024-09-16T15:47:49.988 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-16T15:47:50.115 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:50.111+0000 7f16f853a640 1 -- 172.21.15.123:0/3421329347 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f16f0073600 msgr2=0x7f16f0152e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:47:50.116 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:50.111+0000 7f16f853a640 1 --2- 172.21.15.123:0/3421329347 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f16f0073600 0x7f16f0152e30 secure :-1 s=READY pgs=930 cs=0 l=1 rev1=1 crypto rx=0x7f16e0009900 tx=0x7f16e001c8b0 comp rx=0 tx=0).stop 2024-09-16T15:47:50.116 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:50.111+0000 7f16f853a640 1 -- 172.21.15.123:0/3421329347 shutdown_connections 2024-09-16T15:47:50.116 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:50.111+0000 7f16f853a640 1 --2- 172.21.15.123:0/3421329347 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f16f0073600 0x7f16f0152e30 unknown :-1 s=CLOSED pgs=930 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:50.116 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:50.111+0000 7f16f853a640 1 --2- 172.21.15.123:0/3421329347 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f16f006ce50 0x7f16f00d00e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:50.116 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:50.111+0000 7f16f853a640 1 -- 172.21.15.123:0/3421329347 >> 172.21.15.123:0/3421329347 conn(0x7f16f00cd5d0 msgr2=0x7f16f00cd9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:47:50.116 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:50.111+0000 7f16f853a640 1 -- 172.21.15.123:0/3421329347 shutdown_connections 2024-09-16T15:47:50.116 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:50.111+0000 7f16f853a640 1 -- 172.21.15.123:0/3421329347 wait complete. 2024-09-16T15:47:50.118 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:50.111+0000 7f16f853a640 1 Processor -- start 2024-09-16T15:47:50.118 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:50.111+0000 7f16f853a640 1 -- start start 2024-09-16T15:47:50.118 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:50.111+0000 7f16f853a640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f16f006ce50 0x7f16f01545d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:47:50.118 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:50.111+0000 7f16f853a640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f16f0073600 0x7f16f0154c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:47:50.119 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:50.111+0000 7f16f853a640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f16f00d0620 0x7f16f01551c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:47:50.119 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:50.111+0000 7f16f853a640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f16f015cac0 con 0x7f16f006ce50 2024-09-16T15:47:50.119 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:50.111+0000 7f16f853a640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f16f015c940 con 0x7f16f0073600 2024-09-16T15:47:50.119 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:50.111+0000 7f16f853a640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f16f015cc40 con 0x7f16f00d0620 2024-09-16T15:47:50.119 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:50.111+0000 7f16f62d6640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f16f006ce50 0x7f16f01545d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:47:50.119 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:50.111+0000 7f16f62d6640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f16f006ce50 0x7f16f01545d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:48720/0 (socket says 172.21.15.123:48720) 2024-09-16T15:47:50.119 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:50.111+0000 7f16f62d6640 1 -- 172.21.15.123:0/2713558393 learned_addr learned my addr 172.21.15.123:0/2713558393 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:47:50.119 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:50.111+0000 7f16f6ad7640 1 --2- 172.21.15.123:0/2713558393 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f16f00d0620 0x7f16f01551c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:47:50.119 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:50.111+0000 7f16f5ad5640 1 --2- 172.21.15.123:0/2713558393 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f16f0073600 0x7f16f0154c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:47:50.119 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:50.111+0000 7f16f62d6640 1 -- 172.21.15.123:0/2713558393 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f16f00d0620 msgr2=0x7f16f01551c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:47:50.119 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:50.111+0000 7f16f62d6640 1 --2- 172.21.15.123:0/2713558393 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f16f00d0620 0x7f16f01551c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:50.119 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:50.111+0000 7f16f62d6640 1 -- 172.21.15.123:0/2713558393 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f16f0073600 msgr2=0x7f16f0154c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:47:50.119 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:50.111+0000 7f16f62d6640 1 --2- 172.21.15.123:0/2713558393 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f16f0073600 0x7f16f0154c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:50.120 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:50.111+0000 7f16f62d6640 1 -- 172.21.15.123:0/2713558393 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f16f00b4e00 con 0x7f16f006ce50 2024-09-16T15:47:50.120 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:50.111+0000 7f16f6ad7640 1 --2- 172.21.15.123:0/2713558393 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f16f00d0620 0x7f16f01551c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T15:47:50.120 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:50.111+0000 7f16f62d6640 1 --2- 172.21.15.123:0/2713558393 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f16f006ce50 0x7f16f01545d0 secure :-1 s=READY pgs=931 cs=0 l=1 rev1=1 crypto rx=0x7f16e800c890 tx=0x7f16e800cd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:47:50.120 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:50.111+0000 7f16eeffd640 1 -- 172.21.15.123:0/2713558393 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16e8004220 con 0x7f16f006ce50 2024-09-16T15:47:50.120 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:50.115+0000 7f16eeffd640 1 -- 172.21.15.123:0/2713558393 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f16e80079b0 con 0x7f16f006ce50 2024-09-16T15:47:50.120 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:50.115+0000 7f16f853a640 1 -- 172.21.15.123:0/2713558393 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f16f00b50e0 con 0x7f16f006ce50 2024-09-16T15:47:50.120 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:50.115+0000 7f16eeffd640 1 -- 172.21.15.123:0/2713558393 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16e80044b0 con 0x7f16f006ce50 2024-09-16T15:47:50.120 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:50.115+0000 7f16f853a640 1 -- 172.21.15.123:0/2713558393 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f16f00b56a0 con 0x7f16f006ce50 2024-09-16T15:47:50.121 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:50.115+0000 7f16f853a640 1 -- 172.21.15.123:0/2713558393 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f16f015d0f0 con 0x7f16f006ce50 2024-09-16T15:47:50.121 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:50.115+0000 7f16eeffd640 1 -- 172.21.15.123:0/2713558393 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 16) ==== 97182+0+0 (secure 0 0 0) 0x7f16e8005030 con 0x7f16f006ce50 2024-09-16T15:47:50.121 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:50.115+0000 7f16eeffd640 1 --2- 172.21.15.123:0/2713558393 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f16dc072fe0 0x7f16dc0754a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:47:50.121 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:50.115+0000 7f16f5ad5640 1 --2- 172.21.15.123:0/2713558393 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f16dc072fe0 0x7f16dc0754a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:47:50.121 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:50.115+0000 7f16eeffd640 1 -- 172.21.15.123:0/2713558393 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 5709+0+0 (secure 0 0 0) 0x7f16e8094630 con 0x7f16f006ce50 2024-09-16T15:47:50.121 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:50.115+0000 7f16f5ad5640 1 --2- 172.21.15.123:0/2713558393 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f16dc072fe0 0x7f16dc0754a0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f16f00cb130 tx=0x7f16e40094d0 comp rx=0 tx=0).ready entity=mgr.4150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:47:50.125 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:50.119+0000 7f16eeffd640 1 -- 172.21.15.123:0/2713558393 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f16e8014070 con 0x7f16f006ce50 2024-09-16T15:47:50.299 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:50.295+0000 7f16f853a640 1 -- 172.21.15.123:0/2713558393 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f16f015f270 con 0x7f16f006ce50 2024-09-16T15:47:50.301 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:50.295+0000 7f16eeffd640 1 -- 172.21.15.123:0/2713558393 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v84) ==== 74+0+16793 (secure 0 0 0) 0x7f16e805f660 con 0x7f16f006ce50 2024-09-16T15:47:50.301 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-09-16T15:47:50.301 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":84,"fsid":"56077f89-18c8-486f-ae7e-eecc0bb7822c","created":"2024-09-16T15:42:26.364892+0000","modified":"2024-09-16T15:47:49.295869+0000","last_up_change":"2024-09-16T15:42:38.707234+0000","last_in_change":"2024-09-16T15:42:29.423662+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":13,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-16T15:42:39.287764+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_metadata","create_time":"2024-09-16T15:47:46.961480+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs_data","create_time":"2024-09-16T15:47:48.246604+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9e7c1955-a607-4671-9be7-f2ef760ceeaa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6800","nonce":3882044969},{"type":"v1","addr":"172.21.15.123:6801","nonce":3882044969}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6802","nonce":3882044969},{"type":"v1","addr":"172.21.15.123:6803","nonce":3882044969}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6806","nonce":3882044969},{"type":"v1","addr":"172.21.15.123:6807","nonce":3882044969}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6804","nonce":3882044969},{"type":"v1","addr":"172.21.15.123:6805","nonce":3882044969}]},"public_addr":"172.21.15.123:6801/3882044969","cluster_addr":"172.21.15.123:6803/3882044969","heartbeat_back_addr":"172.21.15.123:6807/3882044969","heartbeat_front_addr":"172.21.15.123:6805/3882044969","state":["exists","up"]},{"osd":1,"uuid":"c540cfde-0977-4742-9b17-fd634668e010","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6816","nonce":4037001850},{"type":"v1","addr":"172.21.15.123:6817","nonce":4037001850}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6818","nonce":4037001850},{"type":"v1","addr":"172.21.15.123:6819","nonce":4037001850}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6822","nonce":4037001850},{"type":"v1","addr":"172.21.15.123:6823","nonce":4037001850}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6820","nonce":4037001850},{"type":"v1","addr":"172.21.15.123:6821","nonce":4037001850}]},"public_addr":"172.21.15.123:6817/4037001850","cluster_addr":"172.21.15.123:6819/4037001850","heartbeat_back_addr":"172.21.15.123:6823/4037001850","heartbeat_front_addr":"172.21.15.123:6821/4037001850","state":["exists","up"]},{"osd":2,"uuid":"9864a496-f169-486b-95c4-1d59162793a7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6824","nonce":1332143088},{"type":"v1","addr":"172.21.15.123:6825","nonce":1332143088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6826","nonce":1332143088},{"type":"v1","addr":"172.21.15.123:6827","nonce":1332143088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6830","nonce":1332143088},{"type":"v1","addr":"172.21.15.123:6831","nonce":1332143088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6828","nonce":1332143088},{"type":"v1","addr":"172.21.15.123:6829","nonce":1332143088}]},"public_addr":"172.21.15.123:6825/1332143088","cluster_addr":"172.21.15.123:6827/1332143088","heartbeat_back_addr":"172.21.15.123:6831/1332143088","heartbeat_front_addr":"172.21.15.123:6829/1332143088","state":["exists","up"]},{"osd":3,"uuid":"c6d1f817-f074-4325-95e1-578eba13edda","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6808","nonce":988586998},{"type":"v1","addr":"172.21.15.123:6809","nonce":988586998}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6810","nonce":988586998},{"type":"v1","addr":"172.21.15.123:6811","nonce":988586998}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6814","nonce":988586998},{"type":"v1","addr":"172.21.15.123:6815","nonce":988586998}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6812","nonce":988586998},{"type":"v1","addr":"172.21.15.123:6813","nonce":988586998}]},"public_addr":"172.21.15.123:6809/988586998","cluster_addr":"172.21.15.123:6811/988586998","heartbeat_back_addr":"172.21.15.123:6815/988586998","heartbeat_front_addr":"172.21.15.123:6813/988586998","state":["exists","up"]},{"osd":4,"uuid":"200a8df4-50ef-40a5-b48a-58981269c086","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6816","nonce":3387813890},{"type":"v1","addr":"172.21.15.184:6817","nonce":3387813890}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6818","nonce":3387813890},{"type":"v1","addr":"172.21.15.184:6819","nonce":3387813890}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6822","nonce":3387813890},{"type":"v1","addr":"172.21.15.184:6823","nonce":3387813890}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6820","nonce":3387813890},{"type":"v1","addr":"172.21.15.184:6821","nonce":3387813890}]},"public_addr":"172.21.15.184:6817/3387813890","cluster_addr":"172.21.15.184:6819/3387813890","heartbeat_back_addr":"172.21.15.184:6823/3387813890","heartbeat_front_addr":"172.21.15.184:6821/3387813890","state":["exists","up"]},{"osd":5,"uuid":"32d5a900-da3d-412c-bc19-13861a2368d8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6808","nonce":2680916017},{"type":"v1","addr":"172.21.15.184:6809","nonce":2680916017}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6810","nonce":2680916017},{"type":"v1","addr":"172.21.15.184:6811","nonce":2680916017}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6814","nonce":2680916017},{"type":"v1","addr":"172.21.15.184:6815","nonce":2680916017}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6812","nonce":2680916017},{"type":"v1","addr":"172.21.15.184:6813","nonce":2680916017}]},"public_addr":"172.21.15.184:6809/2680916017","cluster_addr":"172.21.15.184:6811/2680916017","heartbeat_back_addr":"172.21.15.184:6815/2680916017","heartbeat_front_addr":"172.21.15.184:6813/2680916017","state":["exists","up"]},{"osd":6,"uuid":"cde6f39e-aa1b-4260-ac19-774ee6aaca72","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6824","nonce":2183120457},{"type":"v1","addr":"172.21.15.184:6825","nonce":2183120457}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6826","nonce":2183120457},{"type":"v1","addr":"172.21.15.184:6827","nonce":2183120457}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6830","nonce":2183120457},{"type":"v1","addr":"172.21.15.184:6831","nonce":2183120457}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6828","nonce":2183120457},{"type":"v1","addr":"172.21.15.184:6829","nonce":2183120457}]},"public_addr":"172.21.15.184:6825/2183120457","cluster_addr":"172.21.15.184:6827/2183120457","heartbeat_back_addr":"172.21.15.184:6831/2183120457","heartbeat_front_addr":"172.21.15.184:6829/2183120457","state":["exists","up"]},{"osd":7,"uuid":"4f381778-eee1-47c6-804f-df0e6b33e313","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6800","nonce":4102957285},{"type":"v1","addr":"172.21.15.184:6801","nonce":4102957285}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6802","nonce":4102957285},{"type":"v1","addr":"172.21.15.184:6803","nonce":4102957285}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6806","nonce":4102957285},{"type":"v1","addr":"172.21.15.184:6807","nonce":4102957285}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6804","nonce":4102957285},{"type":"v1","addr":"172.21.15.184:6805","nonce":4102957285}]},"public_addr":"172.21.15.184:6801/4102957285","cluster_addr":"172.21.15.184:6803/4102957285","heartbeat_back_addr":"172.21.15.184:6807/4102957285","heartbeat_front_addr":"172.21.15.184:6805/4102957285","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:35.972210+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.222007+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.198617+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.385663+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.358935+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.156066+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.752973+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:35.682846+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T15:47:50.304 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:50.299+0000 7f16f853a640 1 -- 172.21.15.123:0/2713558393 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f16dc072fe0 msgr2=0x7f16dc0754a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:47:50.304 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:50.299+0000 7f16f853a640 1 --2- 172.21.15.123:0/2713558393 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f16dc072fe0 0x7f16dc0754a0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f16f00cb130 tx=0x7f16e40094d0 comp rx=0 tx=0).stop 2024-09-16T15:47:50.304 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:50.299+0000 7f16f853a640 1 -- 172.21.15.123:0/2713558393 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f16f006ce50 msgr2=0x7f16f01545d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:47:50.304 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:50.299+0000 7f16f853a640 1 --2- 172.21.15.123:0/2713558393 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f16f006ce50 0x7f16f01545d0 secure :-1 s=READY pgs=931 cs=0 l=1 rev1=1 crypto rx=0x7f16e800c890 tx=0x7f16e800cd50 comp rx=0 tx=0).stop 2024-09-16T15:47:50.304 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:50.299+0000 7f16f853a640 1 -- 172.21.15.123:0/2713558393 shutdown_connections 2024-09-16T15:47:50.304 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:50.299+0000 7f16f853a640 1 --2- 172.21.15.123:0/2713558393 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f16dc072fe0 0x7f16dc0754a0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:50.304 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:50.299+0000 7f16f853a640 1 --2- 172.21.15.123:0/2713558393 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f16f00d0620 0x7f16f01551c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:50.304 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:50.299+0000 7f16f853a640 1 --2- 172.21.15.123:0/2713558393 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f16f0073600 0x7f16f0154c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:50.304 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:50.299+0000 7f16f853a640 1 --2- 172.21.15.123:0/2713558393 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f16f006ce50 0x7f16f01545d0 unknown :-1 s=CLOSED pgs=931 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:50.305 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:50.299+0000 7f16f853a640 1 -- 172.21.15.123:0/2713558393 >> 172.21.15.123:0/2713558393 conn(0x7f16f00cd5d0 msgr2=0x7f16f015e5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:47:50.305 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:50.299+0000 7f16f853a640 1 -- 172.21.15.123:0/2713558393 shutdown_connections 2024-09-16T15:47:50.305 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:50.299+0000 7f16f853a640 1 -- 172.21.15.123:0/2713558393 wait complete. 2024-09-16T15:47:50.321 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-16T15:47:50.448 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:50.443+0000 7fa2ed040640 1 -- 172.21.15.123:0/2910260146 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa2e80726c0 msgr2=0x7fa2e8073ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:47:50.448 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:50.443+0000 7fa2ed040640 1 --2- 172.21.15.123:0/2910260146 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa2e80726c0 0x7fa2e8073ef0 secure :-1 s=READY pgs=932 cs=0 l=1 rev1=1 crypto rx=0x7fa2d8009900 tx=0x7fa2d801c820 comp rx=0 tx=0).stop 2024-09-16T15:47:50.449 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:50.443+0000 7fa2ed040640 1 -- 172.21.15.123:0/2910260146 shutdown_connections 2024-09-16T15:47:50.449 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:50.443+0000 7fa2ed040640 1 --2- 172.21.15.123:0/2910260146 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa2e80726c0 0x7fa2e8073ef0 unknown :-1 s=CLOSED pgs=932 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:50.449 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:50.443+0000 7fa2ed040640 1 --2- 172.21.15.123:0/2910260146 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fa2e815bca0 0x7fa2e80d0630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:50.449 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:50.443+0000 7fa2ed040640 1 -- 172.21.15.123:0/2910260146 >> 172.21.15.123:0/2910260146 conn(0x7fa2e80cfb10 msgr2=0x7fa2e80cff20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:47:50.449 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:50.443+0000 7fa2ed040640 1 -- 172.21.15.123:0/2910260146 shutdown_connections 2024-09-16T15:47:50.449 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:50.443+0000 7fa2ed040640 1 -- 172.21.15.123:0/2910260146 wait complete. 2024-09-16T15:47:50.449 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:50.443+0000 7fa2ed040640 1 Processor -- start 2024-09-16T15:47:50.450 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:50.443+0000 7fa2ed040640 1 -- start start 2024-09-16T15:47:50.451 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:50.447+0000 7fa2ed040640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fa2e80726c0 0x7fa2e81fbad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:47:50.452 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:50.447+0000 7fa2ed040640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa2e8074430 0x7fa2e8157b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:47:50.452 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:50.447+0000 7fa2ed040640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa2e815bca0 0x7fa2e8152ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:47:50.452 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:50.447+0000 7fa2ed040640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fa2e8168e90 con 0x7fa2e815bca0 2024-09-16T15:47:50.452 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:50.447+0000 7fa2ed040640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fa2e8168d10 con 0x7fa2e8074430 2024-09-16T15:47:50.452 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:50.447+0000 7fa2ed040640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fa2e8169010 con 0x7fa2e80726c0 2024-09-16T15:47:50.452 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:50.447+0000 7fa2e759e640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa2e815bca0 0x7fa2e8152ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:47:50.452 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:50.447+0000 7fa2e759e640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa2e815bca0 0x7fa2e8152ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:48730/0 (socket says 172.21.15.123:48730) 2024-09-16T15:47:50.452 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:50.447+0000 7fa2e759e640 1 -- 172.21.15.123:0/49131149 learned_addr learned my addr 172.21.15.123:0/49131149 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:47:50.452 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:50.447+0000 7fa2e6d9d640 1 --2- 172.21.15.123:0/49131149 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fa2e80726c0 0x7fa2e81fbad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:47:50.452 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:50.447+0000 7fa2e659c640 1 --2- 172.21.15.123:0/49131149 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa2e8074430 0x7fa2e8157b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:47:50.452 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:50.447+0000 7fa2e759e640 1 -- 172.21.15.123:0/49131149 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fa2e80726c0 msgr2=0x7fa2e81fbad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:47:50.453 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:50.447+0000 7fa2e759e640 1 --2- 172.21.15.123:0/49131149 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fa2e80726c0 0x7fa2e81fbad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:50.453 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:50.447+0000 7fa2e759e640 1 -- 172.21.15.123:0/49131149 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa2e8074430 msgr2=0x7fa2e8157b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:47:50.453 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:50.447+0000 7fa2e759e640 1 --2- 172.21.15.123:0/49131149 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa2e8074430 0x7fa2e8157b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:50.453 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:50.447+0000 7fa2e759e640 1 -- 172.21.15.123:0/49131149 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa2e8165330 con 0x7fa2e815bca0 2024-09-16T15:47:50.453 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:50.447+0000 7fa2e6d9d640 1 --2- 172.21.15.123:0/49131149 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fa2e80726c0 0x7fa2e81fbad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T15:47:50.453 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:50.447+0000 7fa2e759e640 1 --2- 172.21.15.123:0/49131149 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa2e815bca0 0x7fa2e8152ce0 secure :-1 s=READY pgs=933 cs=0 l=1 rev1=1 crypto rx=0x7fa2e000c7b0 tx=0x7fa2e000cc70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:47:50.453 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:50.447+0000 7fa2cf7fe640 1 -- 172.21.15.123:0/49131149 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2e0004b70 con 0x7fa2e815bca0 2024-09-16T15:47:50.453 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:50.447+0000 7fa2cf7fe640 1 -- 172.21.15.123:0/49131149 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fa2e0004e00 con 0x7fa2e815bca0 2024-09-16T15:47:50.453 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:50.447+0000 7fa2ed040640 1 -- 172.21.15.123:0/49131149 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa2e81fc010 con 0x7fa2e815bca0 2024-09-16T15:47:50.453 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:50.447+0000 7fa2cf7fe640 1 -- 172.21.15.123:0/49131149 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2e000aab0 con 0x7fa2e815bca0 2024-09-16T15:47:50.453 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:50.447+0000 7fa2ed040640 1 -- 172.21.15.123:0/49131149 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa2e80cdc10 con 0x7fa2e815bca0 2024-09-16T15:47:50.454 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:50.447+0000 7fa2ed040640 1 -- 172.21.15.123:0/49131149 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa2e8073ef0 con 0x7fa2e815bca0 2024-09-16T15:47:50.454 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:50.447+0000 7fa2cf7fe640 1 -- 172.21.15.123:0/49131149 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 16) ==== 97182+0+0 (secure 0 0 0) 0x7fa2e000ac50 con 0x7fa2e815bca0 2024-09-16T15:47:50.454 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:50.447+0000 7fa2cf7fe640 1 --2- 172.21.15.123:0/49131149 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fa2d4072fe0 0x7fa2d40754a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:47:50.454 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:50.447+0000 7fa2e6d9d640 1 --2- 172.21.15.123:0/49131149 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fa2d4072fe0 0x7fa2d40754a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:47:50.454 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:50.447+0000 7fa2cf7fe640 1 -- 172.21.15.123:0/49131149 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 5709+0+0 (secure 0 0 0) 0x7fa2e0094320 con 0x7fa2e815bca0 2024-09-16T15:47:50.455 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:50.447+0000 7fa2e6d9d640 1 --2- 172.21.15.123:0/49131149 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fa2d4072fe0 0x7fa2d40754a0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fa2dc0058d0 tx=0x7fa2dc005ca0 comp rx=0 tx=0).ready entity=mgr.4150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:47:50.458 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:50.455+0000 7fa2cf7fe640 1 -- 172.21.15.123:0/49131149 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7fa2e0020070 con 0x7fa2e815bca0 2024-09-16T15:47:50.633 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:50.627+0000 7fa2ed040640 1 -- 172.21.15.123:0/49131149 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fa2e815f630 con 0x7fa2e815bca0 2024-09-16T15:47:50.635 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:50.631+0000 7fa2cf7fe640 1 -- 172.21.15.123:0/49131149 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v84) ==== 74+0+16793 (secure 0 0 0) 0x7fa2e81fc010 con 0x7fa2e815bca0 2024-09-16T15:47:50.635 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-09-16T15:47:50.635 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":84,"fsid":"56077f89-18c8-486f-ae7e-eecc0bb7822c","created":"2024-09-16T15:42:26.364892+0000","modified":"2024-09-16T15:47:49.295869+0000","last_up_change":"2024-09-16T15:42:38.707234+0000","last_in_change":"2024-09-16T15:42:29.423662+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":13,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-16T15:42:39.287764+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_metadata","create_time":"2024-09-16T15:47:46.961480+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs_data","create_time":"2024-09-16T15:47:48.246604+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9e7c1955-a607-4671-9be7-f2ef760ceeaa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6800","nonce":3882044969},{"type":"v1","addr":"172.21.15.123:6801","nonce":3882044969}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6802","nonce":3882044969},{"type":"v1","addr":"172.21.15.123:6803","nonce":3882044969}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6806","nonce":3882044969},{"type":"v1","addr":"172.21.15.123:6807","nonce":3882044969}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6804","nonce":3882044969},{"type":"v1","addr":"172.21.15.123:6805","nonce":3882044969}]},"public_addr":"172.21.15.123:6801/3882044969","cluster_addr":"172.21.15.123:6803/3882044969","heartbeat_back_addr":"172.21.15.123:6807/3882044969","heartbeat_front_addr":"172.21.15.123:6805/3882044969","state":["exists","up"]},{"osd":1,"uuid":"c540cfde-0977-4742-9b17-fd634668e010","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6816","nonce":4037001850},{"type":"v1","addr":"172.21.15.123:6817","nonce":4037001850}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6818","nonce":4037001850},{"type":"v1","addr":"172.21.15.123:6819","nonce":4037001850}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6822","nonce":4037001850},{"type":"v1","addr":"172.21.15.123:6823","nonce":4037001850}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6820","nonce":4037001850},{"type":"v1","addr":"172.21.15.123:6821","nonce":4037001850}]},"public_addr":"172.21.15.123:6817/4037001850","cluster_addr":"172.21.15.123:6819/4037001850","heartbeat_back_addr":"172.21.15.123:6823/4037001850","heartbeat_front_addr":"172.21.15.123:6821/4037001850","state":["exists","up"]},{"osd":2,"uuid":"9864a496-f169-486b-95c4-1d59162793a7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6824","nonce":1332143088},{"type":"v1","addr":"172.21.15.123:6825","nonce":1332143088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6826","nonce":1332143088},{"type":"v1","addr":"172.21.15.123:6827","nonce":1332143088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6830","nonce":1332143088},{"type":"v1","addr":"172.21.15.123:6831","nonce":1332143088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6828","nonce":1332143088},{"type":"v1","addr":"172.21.15.123:6829","nonce":1332143088}]},"public_addr":"172.21.15.123:6825/1332143088","cluster_addr":"172.21.15.123:6827/1332143088","heartbeat_back_addr":"172.21.15.123:6831/1332143088","heartbeat_front_addr":"172.21.15.123:6829/1332143088","state":["exists","up"]},{"osd":3,"uuid":"c6d1f817-f074-4325-95e1-578eba13edda","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6808","nonce":988586998},{"type":"v1","addr":"172.21.15.123:6809","nonce":988586998}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6810","nonce":988586998},{"type":"v1","addr":"172.21.15.123:6811","nonce":988586998}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6814","nonce":988586998},{"type":"v1","addr":"172.21.15.123:6815","nonce":988586998}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6812","nonce":988586998},{"type":"v1","addr":"172.21.15.123:6813","nonce":988586998}]},"public_addr":"172.21.15.123:6809/988586998","cluster_addr":"172.21.15.123:6811/988586998","heartbeat_back_addr":"172.21.15.123:6815/988586998","heartbeat_front_addr":"172.21.15.123:6813/988586998","state":["exists","up"]},{"osd":4,"uuid":"200a8df4-50ef-40a5-b48a-58981269c086","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6816","nonce":3387813890},{"type":"v1","addr":"172.21.15.184:6817","nonce":3387813890}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6818","nonce":3387813890},{"type":"v1","addr":"172.21.15.184:6819","nonce":3387813890}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6822","nonce":3387813890},{"type":"v1","addr":"172.21.15.184:6823","nonce":3387813890}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6820","nonce":3387813890},{"type":"v1","addr":"172.21.15.184:6821","nonce":3387813890}]},"public_addr":"172.21.15.184:6817/3387813890","cluster_addr":"172.21.15.184:6819/3387813890","heartbeat_back_addr":"172.21.15.184:6823/3387813890","heartbeat_front_addr":"172.21.15.184:6821/3387813890","state":["exists","up"]},{"osd":5,"uuid":"32d5a900-da3d-412c-bc19-13861a2368d8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6808","nonce":2680916017},{"type":"v1","addr":"172.21.15.184:6809","nonce":2680916017}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6810","nonce":2680916017},{"type":"v1","addr":"172.21.15.184:6811","nonce":2680916017}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6814","nonce":2680916017},{"type":"v1","addr":"172.21.15.184:6815","nonce":2680916017}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6812","nonce":2680916017},{"type":"v1","addr":"172.21.15.184:6813","nonce":2680916017}]},"public_addr":"172.21.15.184:6809/2680916017","cluster_addr":"172.21.15.184:6811/2680916017","heartbeat_back_addr":"172.21.15.184:6815/2680916017","heartbeat_front_addr":"172.21.15.184:6813/2680916017","state":["exists","up"]},{"osd":6,"uuid":"cde6f39e-aa1b-4260-ac19-774ee6aaca72","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6824","nonce":2183120457},{"type":"v1","addr":"172.21.15.184:6825","nonce":2183120457}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6826","nonce":2183120457},{"type":"v1","addr":"172.21.15.184:6827","nonce":2183120457}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6830","nonce":2183120457},{"type":"v1","addr":"172.21.15.184:6831","nonce":2183120457}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6828","nonce":2183120457},{"type":"v1","addr":"172.21.15.184:6829","nonce":2183120457}]},"public_addr":"172.21.15.184:6825/2183120457","cluster_addr":"172.21.15.184:6827/2183120457","heartbeat_back_addr":"172.21.15.184:6831/2183120457","heartbeat_front_addr":"172.21.15.184:6829/2183120457","state":["exists","up"]},{"osd":7,"uuid":"4f381778-eee1-47c6-804f-df0e6b33e313","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6800","nonce":4102957285},{"type":"v1","addr":"172.21.15.184:6801","nonce":4102957285}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6802","nonce":4102957285},{"type":"v1","addr":"172.21.15.184:6803","nonce":4102957285}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6806","nonce":4102957285},{"type":"v1","addr":"172.21.15.184:6807","nonce":4102957285}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6804","nonce":4102957285},{"type":"v1","addr":"172.21.15.184:6805","nonce":4102957285}]},"public_addr":"172.21.15.184:6801/4102957285","cluster_addr":"172.21.15.184:6803/4102957285","heartbeat_back_addr":"172.21.15.184:6807/4102957285","heartbeat_front_addr":"172.21.15.184:6805/4102957285","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:35.972210+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.222007+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.198617+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.385663+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.358935+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.156066+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.752973+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:35.682846+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T15:47:50.638 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:50.631+0000 7fa2ed040640 1 -- 172.21.15.123:0/49131149 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fa2d4072fe0 msgr2=0x7fa2d40754a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:47:50.638 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:50.631+0000 7fa2ed040640 1 --2- 172.21.15.123:0/49131149 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fa2d4072fe0 0x7fa2d40754a0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fa2dc0058d0 tx=0x7fa2dc005ca0 comp rx=0 tx=0).stop 2024-09-16T15:47:50.638 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:50.631+0000 7fa2ed040640 1 -- 172.21.15.123:0/49131149 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa2e815bca0 msgr2=0x7fa2e8152ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:47:50.638 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:50.631+0000 7fa2ed040640 1 --2- 172.21.15.123:0/49131149 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa2e815bca0 0x7fa2e8152ce0 secure :-1 s=READY pgs=933 cs=0 l=1 rev1=1 crypto rx=0x7fa2e000c7b0 tx=0x7fa2e000cc70 comp rx=0 tx=0).stop 2024-09-16T15:47:50.638 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:50.631+0000 7fa2ed040640 1 -- 172.21.15.123:0/49131149 shutdown_connections 2024-09-16T15:47:50.638 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:50.631+0000 7fa2ed040640 1 --2- 172.21.15.123:0/49131149 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fa2d4072fe0 0x7fa2d40754a0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:50.638 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:50.631+0000 7fa2ed040640 1 --2- 172.21.15.123:0/49131149 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa2e815bca0 0x7fa2e8152ce0 unknown :-1 s=CLOSED pgs=933 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:50.638 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:50.631+0000 7fa2ed040640 1 --2- 172.21.15.123:0/49131149 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa2e8074430 0x7fa2e8157b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:50.638 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:50.631+0000 7fa2ed040640 1 --2- 172.21.15.123:0/49131149 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fa2e80726c0 0x7fa2e81fbad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:50.638 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:50.631+0000 7fa2ed040640 1 -- 172.21.15.123:0/49131149 >> 172.21.15.123:0/49131149 conn(0x7fa2e80cfb10 msgr2=0x7fa2e80b43c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:47:50.638 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:50.631+0000 7fa2ed040640 1 -- 172.21.15.123:0/49131149 shutdown_connections 2024-09-16T15:47:50.638 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:50.631+0000 7fa2ed040640 1 -- 172.21.15.123:0/49131149 wait complete. 2024-09-16T15:47:50.655 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-09-16T15:47:50.788 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:50.783+0000 7f12fdf18640 1 -- 172.21.15.123:0/940874957 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f12e4004810 msgr2=0x7f12e4004c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:47:50.788 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:50.783+0000 7f12fdf18640 1 --2- 172.21.15.123:0/940874957 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f12e4004810 0x7f12e4004c90 secure :-1 s=READY pgs=935 cs=0 l=1 rev1=1 crypto rx=0x7f12e00016f0 tx=0x7f12e001c790 comp rx=0 tx=0).stop 2024-09-16T15:47:50.789 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:50.783+0000 7f12fdf18640 1 -- 172.21.15.123:0/940874957 shutdown_connections 2024-09-16T15:47:50.789 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:50.783+0000 7f12fdf18640 1 --2- 172.21.15.123:0/940874957 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f12e4004810 0x7f12e4004c90 unknown :-1 s=CLOSED pgs=935 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:50.789 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:50.783+0000 7f12fdf18640 1 -- 172.21.15.123:0/940874957 >> 172.21.15.123:0/940874957 conn(0x7f12f80b1710 msgr2=0x7f12f80b1b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:47:50.789 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:50.783+0000 7f12fdf18640 1 -- 172.21.15.123:0/940874957 shutdown_connections 2024-09-16T15:47:50.789 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:50.783+0000 7f12fdf18640 1 -- 172.21.15.123:0/940874957 wait complete. 2024-09-16T15:47:50.789 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:50.783+0000 7f12fdf18640 1 Processor -- start 2024-09-16T15:47:50.789 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:50.783+0000 7f12fdf18640 1 -- start start 2024-09-16T15:47:50.791 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:50.783+0000 7f12fdf18640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f12e4004810 0x7f12f80c8810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:47:50.791 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:50.783+0000 7f12fdf18640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f12f80b4c20 0x7f12f80b50a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:47:50.791 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:50.783+0000 7f12fdf18640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f12f81f34f0 0x7f12f81f3970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:47:50.791 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:50.783+0000 7f12fdf18640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f12f815e750 con 0x7f12f81f34f0 2024-09-16T15:47:50.791 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:50.783+0000 7f12fdf18640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f12f815e5d0 con 0x7f12e4004810 2024-09-16T15:47:50.791 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:50.783+0000 7f12fdf18640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f12f815e8d0 con 0x7f12f80b4c20 2024-09-16T15:47:50.791 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:50.787+0000 7f12f7fff640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f12f81f34f0 0x7f12f81f3970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:47:50.792 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:50.787+0000 7f12f7fff640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f12f81f34f0 0x7f12f81f3970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:48756/0 (socket says 172.21.15.123:48756) 2024-09-16T15:47:50.792 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:50.787+0000 7f12f7fff640 1 -- 172.21.15.123:0/3115458730 learned_addr learned my addr 172.21.15.123:0/3115458730 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:47:50.792 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:50.787+0000 7f12f77fe640 1 --2- 172.21.15.123:0/3115458730 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f12e4004810 0x7f12f80c8810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:47:50.792 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:50.787+0000 7f12effff640 1 --2- 172.21.15.123:0/3115458730 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f12f80b4c20 0x7f12f80b50a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:47:50.792 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:50.787+0000 7f12f7fff640 1 -- 172.21.15.123:0/3115458730 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f12f80b4c20 msgr2=0x7f12f80b50a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:47:50.792 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:50.787+0000 7f12f7fff640 1 --2- 172.21.15.123:0/3115458730 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f12f80b4c20 0x7f12f80b50a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:50.792 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:50.787+0000 7f12f7fff640 1 -- 172.21.15.123:0/3115458730 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f12e4004810 msgr2=0x7f12f80c8810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:47:50.792 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:50.787+0000 7f12f7fff640 1 --2- 172.21.15.123:0/3115458730 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f12e4004810 0x7f12f80c8810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:50.792 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:50.787+0000 7f12f7fff640 1 -- 172.21.15.123:0/3115458730 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f12f81f7730 con 0x7f12f81f34f0 2024-09-16T15:47:50.792 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:50.787+0000 7f12f7fff640 1 --2- 172.21.15.123:0/3115458730 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f12f81f34f0 0x7f12f81f3970 secure :-1 s=READY pgs=936 cs=0 l=1 rev1=1 crypto rx=0x7f12f80af630 tx=0x7f12f0011950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:47:50.792 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:50.787+0000 7f12f4ff9640 1 -- 172.21.15.123:0/3115458730 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12f0013860 con 0x7f12f81f34f0 2024-09-16T15:47:50.793 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:50.787+0000 7f12f4ff9640 1 -- 172.21.15.123:0/3115458730 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f12f0013af0 con 0x7f12f81f34f0 2024-09-16T15:47:50.793 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:50.787+0000 7f12f4ff9640 1 -- 172.21.15.123:0/3115458730 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12f00123e0 con 0x7f12f81f34f0 2024-09-16T15:47:50.793 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:50.787+0000 7f12fdf18640 1 -- 172.21.15.123:0/3115458730 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f12f80c8d50 con 0x7f12f81f34f0 2024-09-16T15:47:50.793 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:50.787+0000 7f12fdf18640 1 -- 172.21.15.123:0/3115458730 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f12f81f6e00 con 0x7f12f81f34f0 2024-09-16T15:47:50.794 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:50.787+0000 7f12fdf18640 1 -- 172.21.15.123:0/3115458730 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f12f80ccb00 con 0x7f12f81f34f0 2024-09-16T15:47:50.794 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:50.787+0000 7f12f4ff9640 1 -- 172.21.15.123:0/3115458730 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 16) ==== 97182+0+0 (secure 0 0 0) 0x7f12f0012580 con 0x7f12f81f34f0 2024-09-16T15:47:50.794 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:50.787+0000 7f12f4ff9640 1 --2- 172.21.15.123:0/3115458730 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f12d8072fe0 0x7f12d80754a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:47:50.794 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:50.787+0000 7f12f77fe640 1 --2- 172.21.15.123:0/3115458730 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f12d8072fe0 0x7f12d80754a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:47:50.794 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:50.787+0000 7f12f4ff9640 1 -- 172.21.15.123:0/3115458730 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 5709+0+0 (secure 0 0 0) 0x7f12f0097d20 con 0x7f12f81f34f0 2024-09-16T15:47:50.794 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:50.787+0000 7f12f77fe640 1 --2- 172.21.15.123:0/3115458730 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f12d8072fe0 0x7f12d80754a0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f12e8009ce0 tx=0x7f12e80091e0 comp rx=0 tx=0).ready entity=mgr.4150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:47:50.797 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:50.791+0000 7f12f4ff9640 1 -- 172.21.15.123:0/3115458730 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f12f0062d50 con 0x7f12f81f34f0 2024-09-16T15:47:51.009 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:51.003+0000 7f12fdf18640 1 -- 172.21.15.123:0/3115458730 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) -- 0x7f12f81f3250 con 0x7f12f81f34f0 2024-09-16T15:47:51.955 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:51.951+0000 7f12f4ff9640 1 -- 172.21.15.123:0/3115458730 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v52) ==== 118+0+0 (secure 0 0 0) 0x7f12f0062650 con 0x7f12f81f34f0 2024-09-16T15:47:51.959 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:51.955+0000 7f12fdf18640 1 -- 172.21.15.123:0/3115458730 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f12d8072fe0 msgr2=0x7f12d80754a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:47:51.959 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:51.955+0000 7f12fdf18640 1 --2- 172.21.15.123:0/3115458730 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f12d8072fe0 0x7f12d80754a0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f12e8009ce0 tx=0x7f12e80091e0 comp rx=0 tx=0).stop 2024-09-16T15:47:51.959 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:51.955+0000 7f12fdf18640 1 -- 172.21.15.123:0/3115458730 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f12f81f34f0 msgr2=0x7f12f81f3970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:47:51.959 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:51.955+0000 7f12fdf18640 1 --2- 172.21.15.123:0/3115458730 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f12f81f34f0 0x7f12f81f3970 secure :-1 s=READY pgs=936 cs=0 l=1 rev1=1 crypto rx=0x7f12f80af630 tx=0x7f12f0011950 comp rx=0 tx=0).stop 2024-09-16T15:47:51.959 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:51.955+0000 7f12fdf18640 1 -- 172.21.15.123:0/3115458730 shutdown_connections 2024-09-16T15:47:51.959 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:51.955+0000 7f12fdf18640 1 --2- 172.21.15.123:0/3115458730 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f12d8072fe0 0x7f12d80754a0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:51.959 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:51.955+0000 7f12fdf18640 1 --2- 172.21.15.123:0/3115458730 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f12f81f34f0 0x7f12f81f3970 unknown :-1 s=CLOSED pgs=936 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:51.959 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:51.955+0000 7f12fdf18640 1 --2- 172.21.15.123:0/3115458730 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f12f80b4c20 0x7f12f80b50a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:51.959 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:51.955+0000 7f12fdf18640 1 --2- 172.21.15.123:0/3115458730 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f12e4004810 0x7f12f80c8810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:51.959 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:51.955+0000 7f12fdf18640 1 -- 172.21.15.123:0/3115458730 >> 172.21.15.123:0/3115458730 conn(0x7f12f80b1710 msgr2=0x7f12f80af680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:47:51.959 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:51.955+0000 7f12fdf18640 1 -- 172.21.15.123:0/3115458730 shutdown_connections 2024-09-16T15:47:51.959 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:51.955+0000 7f12fdf18640 1 -- 172.21.15.123:0/3115458730 wait complete. 2024-09-16T15:47:51.977 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-09-16T15:47:51.978 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-09-16T15:47:52.107 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:52.103+0000 7f55d698d640 1 -- 172.21.15.123:0/1129499700 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f55bc004d50 msgr2=0x7f55bc0051d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:47:52.107 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:52.103+0000 7f55d698d640 1 --2- 172.21.15.123:0/1129499700 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f55bc004d50 0x7f55bc0051d0 secure :-1 s=READY pgs=938 cs=0 l=1 rev1=1 crypto rx=0x7f55c8009e30 tx=0x7f55c801d170 comp rx=0 tx=0).stop 2024-09-16T15:47:52.107 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:52.103+0000 7f55d4f2a640 1 -- 172.21.15.123:0/1129499700 reap_dead start 2024-09-16T15:47:52.108 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:52.103+0000 7f55d698d640 1 -- 172.21.15.123:0/1129499700 shutdown_connections 2024-09-16T15:47:52.108 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:52.103+0000 7f55d698d640 1 -- 172.21.15.123:0/1129499700 >> 172.21.15.123:0/1129499700 conn(0x7f55d00b1710 msgr2=0x7f55d00b1b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:47:52.108 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:52.103+0000 7f55d698d640 1 -- 172.21.15.123:0/1129499700 shutdown_connections 2024-09-16T15:47:52.108 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:52.103+0000 7f55d698d640 1 -- 172.21.15.123:0/1129499700 wait complete. 2024-09-16T15:47:52.108 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:52.103+0000 7f55d698d640 1 Processor -- start 2024-09-16T15:47:52.109 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:52.103+0000 7f55d698d640 1 -- start start 2024-09-16T15:47:52.110 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:52.103+0000 7f55d698d640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f55d015d800 0x7f55d00c94f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:47:52.110 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:52.103+0000 7f55d698d640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f55d015dbe0 0x7f55d00afeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:47:52.111 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:52.103+0000 7f55d698d640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f55d0157f40 0x7f55d00b03f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:47:52.111 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:52.103+0000 7f55d698d640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f55d00cca80 con 0x7f55d0157f40 2024-09-16T15:47:52.111 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:52.103+0000 7f55d698d640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f55d00cc900 con 0x7f55d015dbe0 2024-09-16T15:47:52.111 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:52.103+0000 7f55d698d640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f55d00ccc00 con 0x7f55d015d800 2024-09-16T15:47:52.111 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:52.103+0000 7f55d4f2a640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f55d0157f40 0x7f55d00b03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:47:52.111 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:52.103+0000 7f55d4f2a640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f55d0157f40 0x7f55d00b03f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:48780/0 (socket says 172.21.15.123:48780) 2024-09-16T15:47:52.111 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:52.103+0000 7f55d4f2a640 1 -- 172.21.15.123:0/866753488 learned_addr learned my addr 172.21.15.123:0/866753488 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:47:52.111 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:52.103+0000 7f55cffff640 1 --2- 172.21.15.123:0/866753488 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f55d015d800 0x7f55d00c94f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:47:52.111 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:52.103+0000 7f55cf7fe640 1 --2- 172.21.15.123:0/866753488 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f55d015dbe0 0x7f55d00afeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:47:52.111 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:52.103+0000 7f55d4f2a640 1 -- 172.21.15.123:0/866753488 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f55d015d800 msgr2=0x7f55d00c94f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:47:52.111 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:52.103+0000 7f55d4f2a640 1 --2- 172.21.15.123:0/866753488 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f55d015d800 0x7f55d00c94f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:52.111 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:52.103+0000 7f55d4f2a640 1 -- 172.21.15.123:0/866753488 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f55d015dbe0 msgr2=0x7f55d00afeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:47:52.111 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:52.103+0000 7f55d4f2a640 1 --2- 172.21.15.123:0/866753488 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f55d015dbe0 0x7f55d00afeb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:52.112 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:52.103+0000 7f55d4f2a640 1 -- 172.21.15.123:0/866753488 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f55d01f4690 con 0x7f55d0157f40 2024-09-16T15:47:52.112 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:52.103+0000 7f55d4f2a640 1 --2- 172.21.15.123:0/866753488 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f55d0157f40 0x7f55d00b03f0 secure :-1 s=READY pgs=939 cs=0 l=1 rev1=1 crypto rx=0x7f55c8002240 tx=0x7f55c80addc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:47:52.112 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:52.103+0000 7f55ccff9640 1 -- 172.21.15.123:0/866753488 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55c80b1040 con 0x7f55d0157f40 2024-09-16T15:47:52.112 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:52.103+0000 7f55ccff9640 1 -- 172.21.15.123:0/866753488 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f55c80bc070 con 0x7f55d0157f40 2024-09-16T15:47:52.112 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:52.103+0000 7f55ccff9640 1 -- 172.21.15.123:0/866753488 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55c8009a00 con 0x7f55d0157f40 2024-09-16T15:47:52.112 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:52.103+0000 7f55d698d640 1 -- 172.21.15.123:0/866753488 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f55d01f4970 con 0x7f55d0157f40 2024-09-16T15:47:52.112 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:52.107+0000 7f55d698d640 1 -- 172.21.15.123:0/866753488 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f55d01f4f30 con 0x7f55d0157f40 2024-09-16T15:47:52.113 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:52.107+0000 7f55d698d640 1 -- 172.21.15.123:0/866753488 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f55d00cb380 con 0x7f55d0157f40 2024-09-16T15:47:52.113 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:52.107+0000 7f55ccff9640 1 -- 172.21.15.123:0/866753488 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 16) ==== 97182+0+0 (secure 0 0 0) 0x7f55c8004a50 con 0x7f55d0157f40 2024-09-16T15:47:52.113 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:52.107+0000 7f55ccff9640 1 --2- 172.21.15.123:0/866753488 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f55b0072fe0 0x7f55b00754a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:47:52.113 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:52.107+0000 7f55cffff640 1 --2- 172.21.15.123:0/866753488 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f55b0072fe0 0x7f55b00754a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:47:52.113 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:52.107+0000 7f55ccff9640 1 -- 172.21.15.123:0/866753488 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 5709+0+0 (secure 0 0 0) 0x7f55c812eba0 con 0x7f55d0157f40 2024-09-16T15:47:52.113 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:52.107+0000 7f55cffff640 1 --2- 172.21.15.123:0/866753488 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f55b0072fe0 0x7f55b00754a0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f55b80116f0 tx=0x7f55b8001610 comp rx=0 tx=0).ready entity=mgr.4150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:47:52.117 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:52.111+0000 7f55ccff9640 1 -- 172.21.15.123:0/866753488 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f55c8016e80 con 0x7f55d0157f40 2024-09-16T15:47:52.332 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:52.327+0000 7f55d698d640 1 -- 172.21.15.123:0/866753488 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) -- 0x7f55d00697b0 con 0x7f55d0157f40 2024-09-16T15:47:52.964 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:52.959+0000 7f55ccff9640 1 -- 172.21.15.123:0/866753488 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v53) ==== 122+0+0 (secure 0 0 0) 0x7f55c80f9c60 con 0x7f55d0157f40 2024-09-16T15:47:52.968 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:52.963+0000 7f55d698d640 1 -- 172.21.15.123:0/866753488 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f55b0072fe0 msgr2=0x7f55b00754a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:47:52.968 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:52.963+0000 7f55d698d640 1 --2- 172.21.15.123:0/866753488 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f55b0072fe0 0x7f55b00754a0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f55b80116f0 tx=0x7f55b8001610 comp rx=0 tx=0).stop 2024-09-16T15:47:52.969 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:52.963+0000 7f55d698d640 1 -- 172.21.15.123:0/866753488 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f55d0157f40 msgr2=0x7f55d00b03f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:47:52.969 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:52.963+0000 7f55d698d640 1 --2- 172.21.15.123:0/866753488 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f55d0157f40 0x7f55d00b03f0 secure :-1 s=READY pgs=939 cs=0 l=1 rev1=1 crypto rx=0x7f55c8002240 tx=0x7f55c80addc0 comp rx=0 tx=0).stop 2024-09-16T15:47:52.969 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:52.963+0000 7f55d698d640 1 -- 172.21.15.123:0/866753488 shutdown_connections 2024-09-16T15:47:52.969 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:52.963+0000 7f55d698d640 1 --2- 172.21.15.123:0/866753488 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f55b0072fe0 0x7f55b00754a0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:52.969 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:52.963+0000 7f55d698d640 1 --2- 172.21.15.123:0/866753488 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f55d0157f40 0x7f55d00b03f0 unknown :-1 s=CLOSED pgs=939 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:52.969 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:52.963+0000 7f55d698d640 1 --2- 172.21.15.123:0/866753488 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f55d015dbe0 0x7f55d00afeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:52.969 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:52.963+0000 7f55d698d640 1 --2- 172.21.15.123:0/866753488 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f55d015d800 0x7f55d00c94f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:52.969 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:52.963+0000 7f55d698d640 1 -- 172.21.15.123:0/866753488 >> 172.21.15.123:0/866753488 conn(0x7f55d00b1710 msgr2=0x7f55d00d02f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:47:52.969 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:52.963+0000 7f55d698d640 1 -- 172.21.15.123:0/866753488 shutdown_connections 2024-09-16T15:47:52.969 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:52.963+0000 7f55d698d640 1 -- 172.21.15.123:0/866753488 wait complete. 2024-09-16T15:47:52.990 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-09-16T15:47:52.990 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-16T15:47:53.120 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:53.115+0000 7f54657c3640 1 -- 172.21.15.123:0/3213863451 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f546015f9a0 msgr2=0x7f546015fd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:47:53.120 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:53.115+0000 7f54657c3640 1 --2- 172.21.15.123:0/3213863451 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f546015f9a0 0x7f546015fd80 secure :-1 s=READY pgs=941 cs=0 l=1 rev1=1 crypto rx=0x7f54600c3e80 tx=0x7f545801c890 comp rx=0 tx=0).stop 2024-09-16T15:47:53.120 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:53.115+0000 7f54657c3640 1 -- 172.21.15.123:0/3213863451 shutdown_connections 2024-09-16T15:47:53.120 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:53.115+0000 7f54657c3640 1 --2- 172.21.15.123:0/3213863451 >> v2:172.21.15.184:3300/0 conn(0x7f54600cc430 0x7f54600cffa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:53.120 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:53.115+0000 7f54657c3640 1 --2- 172.21.15.123:0/3213863451 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f546015f9a0 0x7f546015fd80 unknown :-1 s=CLOSED pgs=941 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:53.120 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:53.115+0000 7f54657c3640 1 -- 172.21.15.123:0/3213863451 >> 172.21.15.123:0/3213863451 conn(0x7f54600732a0 msgr2=0x7f5460074150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:47:53.120 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:53.115+0000 7f54657c3640 1 -- 172.21.15.123:0/3213863451 shutdown_connections 2024-09-16T15:47:53.120 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:53.115+0000 7f54657c3640 1 -- 172.21.15.123:0/3213863451 wait complete. 2024-09-16T15:47:53.122 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:53.115+0000 7f54657c3640 1 Processor -- start 2024-09-16T15:47:53.122 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:53.115+0000 7f54657c3640 1 -- start start 2024-09-16T15:47:53.122 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:53.115+0000 7f54657c3640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5460074860 0x7f54600c31e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:47:53.123 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:53.115+0000 7f54657c3640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f54600cc430 0x7f54600c7480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:47:53.123 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:53.115+0000 7f54657c3640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f546015f9a0 0x7f54600af940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:47:53.123 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:53.115+0000 7f54657c3640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f546015d860 con 0x7f54600cc430 2024-09-16T15:47:53.123 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:53.115+0000 7f54657c3640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f546015d6e0 con 0x7f5460074860 2024-09-16T15:47:53.123 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:53.115+0000 7f54657c3640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f546015d9e0 con 0x7f546015f9a0 2024-09-16T15:47:53.123 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:53.115+0000 7f545e7fc640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f54600cc430 0x7f54600c7480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:47:53.123 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:53.115+0000 7f545e7fc640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f54600cc430 0x7f54600c7480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:43802/0 (socket says 172.21.15.123:43802) 2024-09-16T15:47:53.123 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:53.115+0000 7f545e7fc640 1 -- 172.21.15.123:0/250998792 learned_addr learned my addr 172.21.15.123:0/250998792 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:47:53.123 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:53.115+0000 7f545e7fc640 1 -- 172.21.15.123:0/250998792 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f546015f9a0 msgr2=0x7f54600af940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:47:53.123 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:53.115+0000 7f545e7fc640 1 --2- 172.21.15.123:0/250998792 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f546015f9a0 0x7f54600af940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:53.123 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:53.115+0000 7f545e7fc640 1 -- 172.21.15.123:0/250998792 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5460074860 msgr2=0x7f54600c31e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:47:53.123 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:53.115+0000 7f545e7fc640 1 --2- 172.21.15.123:0/250998792 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5460074860 0x7f54600c31e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:53.124 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:53.115+0000 7f545e7fc640 1 -- 172.21.15.123:0/250998792 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f54600ca030 con 0x7f54600cc430 2024-09-16T15:47:53.124 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:53.119+0000 7f545e7fc640 1 --2- 172.21.15.123:0/250998792 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f54600cc430 0x7f54600c7480 secure :-1 s=READY pgs=942 cs=0 l=1 rev1=1 crypto rx=0x7f544800c420 tx=0x7f544800c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:47:53.124 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:53.119+0000 7f54577fe640 1 -- 172.21.15.123:0/250998792 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5448017070 con 0x7f54600cc430 2024-09-16T15:47:53.124 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:53.119+0000 7f54577fe640 1 -- 172.21.15.123:0/250998792 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f5448003600 con 0x7f54600cc430 2024-09-16T15:47:53.124 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:53.119+0000 7f54657c3640 1 -- 172.21.15.123:0/250998792 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f54600b0f20 con 0x7f54600cc430 2024-09-16T15:47:53.124 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:53.119+0000 7f54577fe640 1 -- 172.21.15.123:0/250998792 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5448003a90 con 0x7f54600cc430 2024-09-16T15:47:53.124 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:53.119+0000 7f54657c3640 1 -- 172.21.15.123:0/250998792 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f54600c9d30 con 0x7f54600cc430 2024-09-16T15:47:53.125 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:53.119+0000 7f54657c3640 1 -- 172.21.15.123:0/250998792 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f546015e6c0 con 0x7f54600cc430 2024-09-16T15:47:53.125 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:53.119+0000 7f54577fe640 1 -- 172.21.15.123:0/250998792 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 16) ==== 97182+0+0 (secure 0 0 0) 0x7f544801a070 con 0x7f54600cc430 2024-09-16T15:47:53.125 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:53.119+0000 7f54577fe640 1 --2- 172.21.15.123:0/250998792 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f5440072fe0 0x7f54400754a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:47:53.125 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:53.119+0000 7f54577fe640 1 -- 172.21.15.123:0/250998792 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 5709+0+0 (secure 0 0 0) 0x7f54480932f0 con 0x7f54600cc430 2024-09-16T15:47:53.125 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:53.119+0000 7f545effd640 1 --2- 172.21.15.123:0/250998792 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f5440072fe0 0x7f54400754a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:47:53.125 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:53.119+0000 7f545effd640 1 --2- 172.21.15.123:0/250998792 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f5440072fe0 0x7f54400754a0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f54580af000 tx=0x7f5458019630 comp rx=0 tx=0).ready entity=mgr.4150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:47:53.129 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:53.123+0000 7f54577fe640 1 -- 172.21.15.123:0/250998792 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f544805e3b0 con 0x7f54600cc430 2024-09-16T15:47:53.336 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:53.331+0000 7f54657c3640 1 -- 172.21.15.123:0/250998792 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f54600945e0 con 0x7f54600cc430 2024-09-16T15:47:53.337 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:53.331+0000 7f54577fe640 1 -- 172.21.15.123:0/250998792 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 53 v53) ==== 94+0+5237 (secure 0 0 0) 0x7f544805dcb0 con 0x7f54600cc430 2024-09-16T15:47:53.337 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-09-16T15:47:53.337 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":53,"btime":"2024-09-16T15:47:52:957004+0000","default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5677,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6835/1609511477","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6833","nonce":1609511477},{"type":"v1","addr":"172.21.15.184:6835","nonce":1609511477}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":48},{"gid":6672,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6834/3658721718","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6832","nonce":3658721718},{"type":"v1","addr":"172.21.15.123:6834","nonce":3658721718}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":48},{"gid":6675,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6835/2536398045","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6833","nonce":2536398045},{"type":"v1","addr":"172.21.15.123:6835","nonce":2536398045}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":48}],"filesystems":[{"mdsmap":{"epoch":53,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-16T15:47:49.295797+0000","modified":"2024-09-16T15:47:52.333498+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6684},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6684":{"gid":6684,"name":"b","rank":0,"incarnation":49,"state":"up:active","state_seq":3,"addr":"172.21.15.184:6834/1546532515","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":1546532515},{"type":"v1","addr":"172.21.15.184:6834","nonce":1546532515}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6684,"qdb_cluster":[6684]},"id":6}]} 2024-09-16T15:47:53.337 INFO:teuthology.orchestra.run.smithi123.stderr:dumped fsmap epoch 53 2024-09-16T15:47:53.339 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:53.335+0000 7f54657c3640 1 -- 172.21.15.123:0/250998792 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f5440072fe0 msgr2=0x7f54400754a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:47:53.339 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:53.335+0000 7f54657c3640 1 --2- 172.21.15.123:0/250998792 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f5440072fe0 0x7f54400754a0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f54580af000 tx=0x7f5458019630 comp rx=0 tx=0).stop 2024-09-16T15:47:53.340 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:53.335+0000 7f54657c3640 1 -- 172.21.15.123:0/250998792 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f54600cc430 msgr2=0x7f54600c7480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:47:53.340 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:53.335+0000 7f54657c3640 1 --2- 172.21.15.123:0/250998792 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f54600cc430 0x7f54600c7480 secure :-1 s=READY pgs=942 cs=0 l=1 rev1=1 crypto rx=0x7f544800c420 tx=0x7f544800c8e0 comp rx=0 tx=0).stop 2024-09-16T15:47:53.340 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:53.335+0000 7f54657c3640 1 -- 172.21.15.123:0/250998792 shutdown_connections 2024-09-16T15:47:53.340 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:53.335+0000 7f54657c3640 1 --2- 172.21.15.123:0/250998792 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f5440072fe0 0x7f54400754a0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:53.340 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:53.335+0000 7f54657c3640 1 --2- 172.21.15.123:0/250998792 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f546015f9a0 0x7f54600af940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:53.340 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:53.335+0000 7f54657c3640 1 --2- 172.21.15.123:0/250998792 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f54600cc430 0x7f54600c7480 unknown :-1 s=CLOSED pgs=942 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:53.340 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:53.335+0000 7f54657c3640 1 --2- 172.21.15.123:0/250998792 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5460074860 0x7f54600c31e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:53.340 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:53.335+0000 7f54657c3640 1 -- 172.21.15.123:0/250998792 >> 172.21.15.123:0/250998792 conn(0x7f54600732a0 msgr2=0x7f546015afe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:47:53.340 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:53.335+0000 7f54657c3640 1 -- 172.21.15.123:0/250998792 shutdown_connections 2024-09-16T15:47:53.340 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:53.335+0000 7f54657c3640 1 -- 172.21.15.123:0/250998792 wait complete. 2024-09-16T15:47:53.356 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-16T15:47:53.472 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:53.467+0000 7f632412e640 1 -- 172.21.15.123:0/2738606743 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f631c161620 msgr2=0x7f631c0d1da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:47:53.473 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:53.467+0000 7f632412e640 1 --2- 172.21.15.123:0/2738606743 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f631c161620 0x7f631c0d1da0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f63180098f0 tx=0x7f631801c790 comp rx=0 tx=0).stop 2024-09-16T15:47:53.473 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:53.467+0000 7f632412e640 1 -- 172.21.15.123:0/2738606743 shutdown_connections 2024-09-16T15:47:53.473 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:53.467+0000 7f632412e640 1 --2- 172.21.15.123:0/2738606743 >> v2:172.21.15.184:3300/0 conn(0x7f631c0b0a50 0x7f631c0b0f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:53.473 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:53.467+0000 7f632412e640 1 --2- 172.21.15.123:0/2738606743 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f631c161620 0x7f631c0d1da0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:53.473 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:53.467+0000 7f632412e640 1 -- 172.21.15.123:0/2738606743 >> 172.21.15.123:0/2738606743 conn(0x7f631c0d3ea0 msgr2=0x7f631c0d42b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:47:53.473 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:53.467+0000 7f632412e640 1 -- 172.21.15.123:0/2738606743 shutdown_connections 2024-09-16T15:47:53.473 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:53.467+0000 7f632412e640 1 -- 172.21.15.123:0/2738606743 wait complete. 2024-09-16T15:47:53.474 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:53.467+0000 7f632412e640 1 Processor -- start 2024-09-16T15:47:53.474 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:53.467+0000 7f632412e640 1 -- start start 2024-09-16T15:47:53.476 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:53.471+0000 7f632412e640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f631c0b0a50 0x7f631c0cde30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:47:53.476 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:53.471+0000 7f632412e640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f631c0b2b00 0x7f631c1fb080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:47:53.476 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:53.471+0000 7f632412e640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f631c161620 0x7f631c0ccde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:47:53.476 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:53.471+0000 7f632412e640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f631c16b0f0 con 0x7f631c0b0a50 2024-09-16T15:47:53.476 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:53.471+0000 7f632412e640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f631c16af70 con 0x7f631c0b2b00 2024-09-16T15:47:53.476 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:53.471+0000 7f632412e640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f631c16b270 con 0x7f631c161620 2024-09-16T15:47:53.476 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:53.471+0000 7f6321eca640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f631c0b0a50 0x7f631c0cde30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:47:53.476 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:53.471+0000 7f6321eca640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f631c0b0a50 0x7f631c0cde30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:43804/0 (socket says 172.21.15.123:43804) 2024-09-16T15:47:53.477 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:53.471+0000 7f6321eca640 1 -- 172.21.15.123:0/930379371 learned_addr learned my addr 172.21.15.123:0/930379371 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:47:53.477 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:53.471+0000 7f6321eca640 1 -- 172.21.15.123:0/930379371 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f631c161620 msgr2=0x7f631c0ccde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:47:53.477 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:53.471+0000 7f6321eca640 1 --2- 172.21.15.123:0/930379371 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f631c161620 0x7f631c0ccde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:53.477 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:53.471+0000 7f6321eca640 1 -- 172.21.15.123:0/930379371 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f631c0b2b00 msgr2=0x7f631c1fb080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:47:53.477 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:53.471+0000 7f6321eca640 1 --2- 172.21.15.123:0/930379371 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f631c0b2b00 0x7f631c1fb080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:53.477 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:53.471+0000 7f6321eca640 1 -- 172.21.15.123:0/930379371 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f631c1fba60 con 0x7f631c0b0a50 2024-09-16T15:47:53.477 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:53.471+0000 7f6321eca640 1 --2- 172.21.15.123:0/930379371 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f631c0b0a50 0x7f631c0cde30 secure :-1 s=READY pgs=943 cs=0 l=1 rev1=1 crypto rx=0x7f6318000c10 tx=0x7f63180057d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:47:53.477 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:53.471+0000 7f63127fc640 1 -- 172.21.15.123:0/930379371 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63180b2040 con 0x7f631c0b0a50 2024-09-16T15:47:53.477 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:53.471+0000 7f63127fc640 1 -- 172.21.15.123:0/930379371 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f6318004040 con 0x7f631c0b0a50 2024-09-16T15:47:53.477 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:53.471+0000 7f632412e640 1 -- 172.21.15.123:0/930379371 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f631c1fbd40 con 0x7f631c0b0a50 2024-09-16T15:47:53.477 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:53.471+0000 7f63127fc640 1 -- 172.21.15.123:0/930379371 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6318004450 con 0x7f631c0b0a50 2024-09-16T15:47:53.477 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:53.471+0000 7f632412e640 1 -- 172.21.15.123:0/930379371 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f631c06ec70 con 0x7f631c0b0a50 2024-09-16T15:47:53.478 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:53.471+0000 7f632412e640 1 -- 172.21.15.123:0/930379371 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f631c1fbf80 con 0x7f631c0b0a50 2024-09-16T15:47:53.478 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:53.471+0000 7f63127fc640 1 -- 172.21.15.123:0/930379371 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 16) ==== 97182+0+0 (secure 0 0 0) 0x7f63180bd030 con 0x7f631c0b0a50 2024-09-16T15:47:53.478 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:53.471+0000 7f63127fc640 1 --2- 172.21.15.123:0/930379371 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f6300072fe0 0x7f63000754a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:47:53.478 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:53.471+0000 7f63127fc640 1 -- 172.21.15.123:0/930379371 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 5709+0+0 (secure 0 0 0) 0x7f631812d9b0 con 0x7f631c0b0a50 2024-09-16T15:47:53.478 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:53.471+0000 7f63216c9640 1 --2- 172.21.15.123:0/930379371 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f6300072fe0 0x7f63000754a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:47:53.479 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:53.475+0000 7f63216c9640 1 --2- 172.21.15.123:0/930379371 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f6300072fe0 0x7f63000754a0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f630c007fd0 tx=0x7f630c007980 comp rx=0 tx=0).ready entity=mgr.4150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:47:53.482 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:53.479+0000 7f63127fc640 1 -- 172.21.15.123:0/930379371 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f631c1fbf80 con 0x7f631c0b0a50 2024-09-16T15:47:53.651 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:53.647+0000 7f632412e640 1 -- 172.21.15.123:0/930379371 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f631c0b0f00 con 0x7f631c0b0a50 2024-09-16T15:47:53.653 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:53.647+0000 7f63127fc640 1 -- 172.21.15.123:0/930379371 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v85) ==== 74+0+16780 (secure 0 0 0) 0x7f63180c4060 con 0x7f631c0b0a50 2024-09-16T15:47:53.653 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-09-16T15:47:53.653 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":85,"fsid":"56077f89-18c8-486f-ae7e-eecc0bb7822c","created":"2024-09-16T15:42:26.364892+0000","modified":"2024-09-16T15:47:50.936444+0000","last_up_change":"2024-09-16T15:42:38.707234+0000","last_in_change":"2024-09-16T15:42:29.423662+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":13,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-16T15:42:39.287764+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_metadata","create_time":"2024-09-16T15:47:46.961480+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs_data","create_time":"2024-09-16T15:47:48.246604+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9e7c1955-a607-4671-9be7-f2ef760ceeaa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6800","nonce":3882044969},{"type":"v1","addr":"172.21.15.123:6801","nonce":3882044969}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6802","nonce":3882044969},{"type":"v1","addr":"172.21.15.123:6803","nonce":3882044969}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6806","nonce":3882044969},{"type":"v1","addr":"172.21.15.123:6807","nonce":3882044969}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6804","nonce":3882044969},{"type":"v1","addr":"172.21.15.123:6805","nonce":3882044969}]},"public_addr":"172.21.15.123:6801/3882044969","cluster_addr":"172.21.15.123:6803/3882044969","heartbeat_back_addr":"172.21.15.123:6807/3882044969","heartbeat_front_addr":"172.21.15.123:6805/3882044969","state":["exists","up"]},{"osd":1,"uuid":"c540cfde-0977-4742-9b17-fd634668e010","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6816","nonce":4037001850},{"type":"v1","addr":"172.21.15.123:6817","nonce":4037001850}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6818","nonce":4037001850},{"type":"v1","addr":"172.21.15.123:6819","nonce":4037001850}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6822","nonce":4037001850},{"type":"v1","addr":"172.21.15.123:6823","nonce":4037001850}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6820","nonce":4037001850},{"type":"v1","addr":"172.21.15.123:6821","nonce":4037001850}]},"public_addr":"172.21.15.123:6817/4037001850","cluster_addr":"172.21.15.123:6819/4037001850","heartbeat_back_addr":"172.21.15.123:6823/4037001850","heartbeat_front_addr":"172.21.15.123:6821/4037001850","state":["exists","up"]},{"osd":2,"uuid":"9864a496-f169-486b-95c4-1d59162793a7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6824","nonce":1332143088},{"type":"v1","addr":"172.21.15.123:6825","nonce":1332143088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6826","nonce":1332143088},{"type":"v1","addr":"172.21.15.123:6827","nonce":1332143088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6830","nonce":1332143088},{"type":"v1","addr":"172.21.15.123:6831","nonce":1332143088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6828","nonce":1332143088},{"type":"v1","addr":"172.21.15.123:6829","nonce":1332143088}]},"public_addr":"172.21.15.123:6825/1332143088","cluster_addr":"172.21.15.123:6827/1332143088","heartbeat_back_addr":"172.21.15.123:6831/1332143088","heartbeat_front_addr":"172.21.15.123:6829/1332143088","state":["exists","up"]},{"osd":3,"uuid":"c6d1f817-f074-4325-95e1-578eba13edda","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6808","nonce":988586998},{"type":"v1","addr":"172.21.15.123:6809","nonce":988586998}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6810","nonce":988586998},{"type":"v1","addr":"172.21.15.123:6811","nonce":988586998}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6814","nonce":988586998},{"type":"v1","addr":"172.21.15.123:6815","nonce":988586998}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6812","nonce":988586998},{"type":"v1","addr":"172.21.15.123:6813","nonce":988586998}]},"public_addr":"172.21.15.123:6809/988586998","cluster_addr":"172.21.15.123:6811/988586998","heartbeat_back_addr":"172.21.15.123:6815/988586998","heartbeat_front_addr":"172.21.15.123:6813/988586998","state":["exists","up"]},{"osd":4,"uuid":"200a8df4-50ef-40a5-b48a-58981269c086","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6816","nonce":3387813890},{"type":"v1","addr":"172.21.15.184:6817","nonce":3387813890}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6818","nonce":3387813890},{"type":"v1","addr":"172.21.15.184:6819","nonce":3387813890}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6822","nonce":3387813890},{"type":"v1","addr":"172.21.15.184:6823","nonce":3387813890}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6820","nonce":3387813890},{"type":"v1","addr":"172.21.15.184:6821","nonce":3387813890}]},"public_addr":"172.21.15.184:6817/3387813890","cluster_addr":"172.21.15.184:6819/3387813890","heartbeat_back_addr":"172.21.15.184:6823/3387813890","heartbeat_front_addr":"172.21.15.184:6821/3387813890","state":["exists","up"]},{"osd":5,"uuid":"32d5a900-da3d-412c-bc19-13861a2368d8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6808","nonce":2680916017},{"type":"v1","addr":"172.21.15.184:6809","nonce":2680916017}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6810","nonce":2680916017},{"type":"v1","addr":"172.21.15.184:6811","nonce":2680916017}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6814","nonce":2680916017},{"type":"v1","addr":"172.21.15.184:6815","nonce":2680916017}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6812","nonce":2680916017},{"type":"v1","addr":"172.21.15.184:6813","nonce":2680916017}]},"public_addr":"172.21.15.184:6809/2680916017","cluster_addr":"172.21.15.184:6811/2680916017","heartbeat_back_addr":"172.21.15.184:6815/2680916017","heartbeat_front_addr":"172.21.15.184:6813/2680916017","state":["exists","up"]},{"osd":6,"uuid":"cde6f39e-aa1b-4260-ac19-774ee6aaca72","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6824","nonce":2183120457},{"type":"v1","addr":"172.21.15.184:6825","nonce":2183120457}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6826","nonce":2183120457},{"type":"v1","addr":"172.21.15.184:6827","nonce":2183120457}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6830","nonce":2183120457},{"type":"v1","addr":"172.21.15.184:6831","nonce":2183120457}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6828","nonce":2183120457},{"type":"v1","addr":"172.21.15.184:6829","nonce":2183120457}]},"public_addr":"172.21.15.184:6825/2183120457","cluster_addr":"172.21.15.184:6827/2183120457","heartbeat_back_addr":"172.21.15.184:6831/2183120457","heartbeat_front_addr":"172.21.15.184:6829/2183120457","state":["exists","up"]},{"osd":7,"uuid":"4f381778-eee1-47c6-804f-df0e6b33e313","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6800","nonce":4102957285},{"type":"v1","addr":"172.21.15.184:6801","nonce":4102957285}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6802","nonce":4102957285},{"type":"v1","addr":"172.21.15.184:6803","nonce":4102957285}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6806","nonce":4102957285},{"type":"v1","addr":"172.21.15.184:6807","nonce":4102957285}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6804","nonce":4102957285},{"type":"v1","addr":"172.21.15.184:6805","nonce":4102957285}]},"public_addr":"172.21.15.184:6801/4102957285","cluster_addr":"172.21.15.184:6803/4102957285","heartbeat_back_addr":"172.21.15.184:6807/4102957285","heartbeat_front_addr":"172.21.15.184:6805/4102957285","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:35.972210+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.222007+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.198617+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.385663+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.358935+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.156066+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.752973+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:35.682846+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T15:47:53.655 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:53.651+0000 7f632412e640 1 -- 172.21.15.123:0/930379371 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f6300072fe0 msgr2=0x7f63000754a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:47:53.655 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:53.651+0000 7f632412e640 1 --2- 172.21.15.123:0/930379371 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f6300072fe0 0x7f63000754a0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f630c007fd0 tx=0x7f630c007980 comp rx=0 tx=0).stop 2024-09-16T15:47:53.655 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:53.651+0000 7f632412e640 1 -- 172.21.15.123:0/930379371 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f631c0b0a50 msgr2=0x7f631c0cde30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:47:53.655 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:53.651+0000 7f632412e640 1 --2- 172.21.15.123:0/930379371 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f631c0b0a50 0x7f631c0cde30 secure :-1 s=READY pgs=943 cs=0 l=1 rev1=1 crypto rx=0x7f6318000c10 tx=0x7f63180057d0 comp rx=0 tx=0).stop 2024-09-16T15:47:53.655 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:53.651+0000 7f632412e640 1 -- 172.21.15.123:0/930379371 shutdown_connections 2024-09-16T15:47:53.656 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:53.651+0000 7f632412e640 1 --2- 172.21.15.123:0/930379371 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f6300072fe0 0x7f63000754a0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:53.656 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:53.651+0000 7f632412e640 1 --2- 172.21.15.123:0/930379371 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f631c161620 0x7f631c0ccde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:53.656 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:53.651+0000 7f632412e640 1 --2- 172.21.15.123:0/930379371 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f631c0b2b00 0x7f631c1fb080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:53.656 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:53.651+0000 7f632412e640 1 --2- 172.21.15.123:0/930379371 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f631c0b0a50 0x7f631c0cde30 unknown :-1 s=CLOSED pgs=943 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:53.656 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:53.651+0000 7f632412e640 1 -- 172.21.15.123:0/930379371 >> 172.21.15.123:0/930379371 conn(0x7f631c0d3ea0 msgr2=0x7f631c0d1da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:47:53.656 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:53.651+0000 7f632412e640 1 -- 172.21.15.123:0/930379371 shutdown_connections 2024-09-16T15:47:53.656 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:53.651+0000 7f632412e640 1 -- 172.21.15.123:0/930379371 wait complete. 2024-09-16T15:47:53.671 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-09-16T15:47:53.671 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T15:47:53.792 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:53.787+0000 7f6108c02640 1 -- 172.21.15.123:0/4131949606 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f60ec004d50 msgr2=0x7f60ec0051d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:47:53.792 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:53.787+0000 7f6108c02640 1 --2- 172.21.15.123:0/4131949606 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f60ec004d50 0x7f60ec0051d0 secure :-1 s=READY pgs=945 cs=0 l=1 rev1=1 crypto rx=0x7f60fc0043e0 tx=0x7f60fc01ca60 comp rx=0 tx=0).stop 2024-09-16T15:47:53.793 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:53.787+0000 7f6102d9d640 1 -- 172.21.15.123:0/4131949606 reap_dead start 2024-09-16T15:47:53.793 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:53.787+0000 7f6108c02640 1 -- 172.21.15.123:0/4131949606 shutdown_connections 2024-09-16T15:47:53.793 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:53.787+0000 7f6108c02640 1 -- 172.21.15.123:0/4131949606 >> 172.21.15.123:0/4131949606 conn(0x7f61040cfb10 msgr2=0x7f61040cff20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:47:53.793 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:53.787+0000 7f6108c02640 1 -- 172.21.15.123:0/4131949606 shutdown_connections 2024-09-16T15:47:53.793 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:53.787+0000 7f6108c02640 1 -- 172.21.15.123:0/4131949606 wait complete. 2024-09-16T15:47:53.793 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:53.787+0000 7f6108c02640 1 Processor -- start 2024-09-16T15:47:53.793 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:53.787+0000 7f6108c02640 1 -- start start 2024-09-16T15:47:53.796 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:53.791+0000 7f6108c02640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f60ec004d50 0x7f61040b1120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:47:53.796 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:53.791+0000 7f6108c02640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f61041f3300 0x7f61040c75f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:47:53.796 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:53.791+0000 7f6108c02640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f61041f4d00 0x7f61040c4770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:47:53.796 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:53.791+0000 7f6108c02640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f61040d09c0 con 0x7f60ec004d50 2024-09-16T15:47:53.796 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:53.791+0000 7f6108c02640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f61040d0840 con 0x7f61041f3300 2024-09-16T15:47:53.796 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:53.791+0000 7f6108c02640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f610406ee60 con 0x7f61041f4d00 2024-09-16T15:47:53.796 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:53.791+0000 7f610259c640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f60ec004d50 0x7f61040b1120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:47:53.796 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:53.791+0000 7f610259c640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f60ec004d50 0x7f61040b1120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:43820/0 (socket says 172.21.15.123:43820) 2024-09-16T15:47:53.797 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:53.791+0000 7f610259c640 1 -- 172.21.15.123:0/1553516535 learned_addr learned my addr 172.21.15.123:0/1553516535 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:47:53.797 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:53.791+0000 7f6101d9b640 1 --2- 172.21.15.123:0/1553516535 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f61041f3300 0x7f61040c75f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:47:53.797 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:53.791+0000 7f6102d9d640 1 --2- 172.21.15.123:0/1553516535 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f61041f4d00 0x7f61040c4770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:47:53.797 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:53.791+0000 7f610259c640 1 -- 172.21.15.123:0/1553516535 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f61041f4d00 msgr2=0x7f61040c4770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:47:53.797 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:53.791+0000 7f610259c640 1 --2- 172.21.15.123:0/1553516535 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f61041f4d00 0x7f61040c4770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:53.797 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:53.791+0000 7f610259c640 1 -- 172.21.15.123:0/1553516535 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f61041f3300 msgr2=0x7f61040c75f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:47:53.797 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:53.791+0000 7f610259c640 1 --2- 172.21.15.123:0/1553516535 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f61041f3300 0x7f61040c75f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:53.797 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:53.791+0000 7f610259c640 1 -- 172.21.15.123:0/1553516535 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f61040c90a0 con 0x7f60ec004d50 2024-09-16T15:47:53.797 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:53.791+0000 7f6101d9b640 1 --2- 172.21.15.123:0/1553516535 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f61041f3300 0x7f61040c75f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T15:47:53.797 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:53.791+0000 7f610259c640 1 --2- 172.21.15.123:0/1553516535 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f60ec004d50 0x7f61040b1120 secure :-1 s=READY pgs=946 cs=0 l=1 rev1=1 crypto rx=0x7f60f400f9d0 tx=0x7f60f400fe90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:47:53.797 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:53.791+0000 7f60f2ffd640 1 -- 172.21.15.123:0/1553516535 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60f400bcb0 con 0x7f60ec004d50 2024-09-16T15:47:53.798 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:53.791+0000 7f60f2ffd640 1 -- 172.21.15.123:0/1553516535 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f60f4011440 con 0x7f60ec004d50 2024-09-16T15:47:53.798 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:53.791+0000 7f60f2ffd640 1 -- 172.21.15.123:0/1553516535 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60f4011850 con 0x7f60ec004d50 2024-09-16T15:47:53.798 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:53.791+0000 7f6108c02640 1 -- 172.21.15.123:0/1553516535 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f61040c9340 con 0x7f60ec004d50 2024-09-16T15:47:53.798 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:53.791+0000 7f6108c02640 1 -- 172.21.15.123:0/1553516535 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f61040c9900 con 0x7f60ec004d50 2024-09-16T15:47:53.798 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:53.791+0000 7f6108c02640 1 -- 172.21.15.123:0/1553516535 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f610415f630 con 0x7f60ec004d50 2024-09-16T15:47:53.798 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:53.791+0000 7f60f2ffd640 1 -- 172.21.15.123:0/1553516535 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 16) ==== 97182+0+0 (secure 0 0 0) 0x7f60f4019020 con 0x7f60ec004d50 2024-09-16T15:47:53.799 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:53.791+0000 7f60f2ffd640 1 --2- 172.21.15.123:0/1553516535 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f60e4072fe0 0x7f60e40754a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:47:53.799 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:53.791+0000 7f60f2ffd640 1 -- 172.21.15.123:0/1553516535 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 5709+0+0 (secure 0 0 0) 0x7f60f4094a00 con 0x7f60ec004d50 2024-09-16T15:47:53.799 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:53.791+0000 7f6101d9b640 1 --2- 172.21.15.123:0/1553516535 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f60e4072fe0 0x7f60e40754a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:47:53.799 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:53.791+0000 7f6101d9b640 1 --2- 172.21.15.123:0/1553516535 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f60e4072fe0 0x7f60e40754a0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f60f80046e0 tx=0x7f60f8001120 comp rx=0 tx=0).ready entity=mgr.4150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:47:53.801 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:53.795+0000 7f60f2ffd640 1 -- 172.21.15.123:0/1553516535 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f60f405fa30 con 0x7f60ec004d50 2024-09-16T15:47:53.968 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:53.963+0000 7f6108c02640 1 -- 172.21.15.123:0/1553516535 --> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f6104073eb0 con 0x7f60e4072fe0 2024-09-16T15:47:53.975 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:53.971+0000 7f60f2ffd640 1 -- 172.21.15.123:0/1553516535 <== mgr.4150 v2:172.21.15.123:6836/2464759676 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+252068 (secure 0 0 0) 0x7f6104073eb0 con 0x7f60e4072fe0 2024-09-16T15:47:53.977 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-09-16T15:47:53.979 INFO:teuthology.orchestra.run.smithi123.stderr:dumped all 2024-09-16T15:47:53.980 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:53.975+0000 7f6108c02640 1 -- 172.21.15.123:0/1553516535 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f60e4072fe0 msgr2=0x7f60e40754a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:47:53.980 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:53.975+0000 7f6108c02640 1 --2- 172.21.15.123:0/1553516535 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f60e4072fe0 0x7f60e40754a0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f60f80046e0 tx=0x7f60f8001120 comp rx=0 tx=0).stop 2024-09-16T15:47:53.980 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:53.975+0000 7f6108c02640 1 -- 172.21.15.123:0/1553516535 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f60ec004d50 msgr2=0x7f61040b1120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:47:53.980 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:53.975+0000 7f6108c02640 1 --2- 172.21.15.123:0/1553516535 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f60ec004d50 0x7f61040b1120 secure :-1 s=READY pgs=946 cs=0 l=1 rev1=1 crypto rx=0x7f60f400f9d0 tx=0x7f60f400fe90 comp rx=0 tx=0).stop 2024-09-16T15:47:53.981 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:53.975+0000 7f6108c02640 1 -- 172.21.15.123:0/1553516535 shutdown_connections 2024-09-16T15:47:53.981 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:53.975+0000 7f6108c02640 1 --2- 172.21.15.123:0/1553516535 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f60e4072fe0 0x7f60e40754a0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:53.981 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:53.975+0000 7f6108c02640 1 --2- 172.21.15.123:0/1553516535 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f61041f4d00 0x7f61040c4770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:53.981 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:53.975+0000 7f6108c02640 1 --2- 172.21.15.123:0/1553516535 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f61041f3300 0x7f61040c75f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:53.981 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:53.975+0000 7f6108c02640 1 --2- 172.21.15.123:0/1553516535 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f60ec004d50 0x7f61040b1120 unknown :-1 s=CLOSED pgs=946 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:53.981 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:53.975+0000 7f6108c02640 1 -- 172.21.15.123:0/1553516535 >> 172.21.15.123:0/1553516535 conn(0x7f61040cfb10 msgr2=0x7f610406f320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:47:53.981 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:53.975+0000 7f6108c02640 1 -- 172.21.15.123:0/1553516535 shutdown_connections 2024-09-16T15:47:53.981 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:53.975+0000 7f6108c02640 1 -- 172.21.15.123:0/1553516535 wait complete. 2024-09-16T15:47:53.994 INFO:teuthology.orchestra.run.smithi123.stdout:{"pg_ready":true,"pg_map":{"version":25,"stamp":"2024-09-16T15:47:52.071470+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":157,"num_write_kb":1642,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":72,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":250296,"kb_used_data":9096,"kb_used_omap":12,"kb_used_meta":241011,"kb_avail":754724424,"statfs":{"total":773094113280,"available":772837810176,"internally_reserved":0,"allocated":9314304,"data_stored":4652888,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":246795825},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":38,"apply_latency_ms":38,"commit_latency_ns":38000000,"apply_latency_ns":38000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.170082"},"pg_stats":[{"pgid":"13.3f","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-16T15:47:49.965606+0000","last_change":"2024-09-16T15:47:49.959302+0000","last_active":"2024-09-16T15:47:49.965606+0000","last_peered":"2024-09-16T15:47:49.965606+0000","last_clean":"2024-09-16T15:47:49.965606+0000","last_became_active":"2024-09-16T15:47:49.954739+0000","last_became_peered":"2024-09-16T15:47:49.954739+0000","last_unstale":"2024-09-16T15:47:49.965606+0000","last_undegraded":"2024-09-16T15:47:49.965606+0000","last_fullsized":"2024-09-16T15:47:49.965606+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:47:48.900296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:47:48.900296+0000","last_clean_scrub_stamp":"2024-09-16T15:47:48.900296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T23:26:28.735670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.3e","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-16T15:47:49.963868+0000","last_change":"2024-09-16T15:47:49.957730+0000","last_active":"2024-09-16T15:47:49.963868+0000","last_peered":"2024-09-16T15:47:49.963868+0000","last_clean":"2024-09-16T15:47:49.963868+0000","last_became_active":"2024-09-16T15:47:49.957226+0000","last_became_peered":"2024-09-16T15:47:49.957226+0000","last_unstale":"2024-09-16T15:47:49.963868+0000","last_undegraded":"2024-09-16T15:47:49.963868+0000","last_fullsized":"2024-09-16T15:47:49.963868+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:47:48.900296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:47:48.900296+0000","last_clean_scrub_stamp":"2024-09-16T15:47:48.900296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:51:36.683460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.3d","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-16T15:47:49.952793+0000","last_change":"2024-09-16T15:47:49.950700+0000","last_active":"2024-09-16T15:47:49.952793+0000","last_peered":"2024-09-16T15:47:49.952793+0000","last_clean":"2024-09-16T15:47:49.952793+0000","last_became_active":"2024-09-16T15:47:49.950374+0000","last_became_peered":"2024-09-16T15:47:49.950374+0000","last_unstale":"2024-09-16T15:47:49.952793+0000","last_undegraded":"2024-09-16T15:47:49.952793+0000","last_fullsized":"2024-09-16T15:47:49.952793+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:47:48.900296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:47:48.900296+0000","last_clean_scrub_stamp":"2024-09-16T15:47:48.900296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:18:30.188685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.3c","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-16T15:47:49.961221+0000","last_change":"2024-09-16T15:47:49.957584+0000","last_active":"2024-09-16T15:47:49.961221+0000","last_peered":"2024-09-16T15:47:49.961221+0000","last_clean":"2024-09-16T15:47:49.961221+0000","last_became_active":"2024-09-16T15:47:49.957263+0000","last_became_peered":"2024-09-16T15:47:49.957263+0000","last_unstale":"2024-09-16T15:47:49.961221+0000","last_undegraded":"2024-09-16T15:47:49.961221+0000","last_fullsized":"2024-09-16T15:47:49.961221+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:47:48.900296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:47:48.900296+0000","last_clean_scrub_stamp":"2024-09-16T15:47:48.900296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T18:09:41.664969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.3b","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-16T15:47:49.947011+0000","last_change":"2024-09-16T15:47:49.945731+0000","last_active":"2024-09-16T15:47:49.947011+0000","last_peered":"2024-09-16T15:47:49.947011+0000","last_clean":"2024-09-16T15:47:49.947011+0000","last_became_active":"2024-09-16T15:47:49.945418+0000","last_became_peered":"2024-09-16T15:47:49.945418+0000","last_unstale":"2024-09-16T15:47:49.947011+0000","last_undegraded":"2024-09-16T15:47:49.947011+0000","last_fullsized":"2024-09-16T15:47:49.947011+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:47:48.900296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:47:48.900296+0000","last_clean_scrub_stamp":"2024-09-16T15:47:48.900296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:25:16.118135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.3a","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-16T15:47:49.967679+0000","last_change":"2024-09-16T15:47:49.965103+0000","last_active":"2024-09-16T15:47:49.967679+0000","last_peered":"2024-09-16T15:47:49.967679+0000","last_clean":"2024-09-16T15:47:49.967679+0000","last_became_active":"2024-09-16T15:47:49.964784+0000","last_became_peered":"2024-09-16T15:47:49.964784+0000","last_unstale":"2024-09-16T15:47:49.967679+0000","last_undegraded":"2024-09-16T15:47:49.967679+0000","last_fullsized":"2024-09-16T15:47:49.967679+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:47:48.900296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:47:48.900296+0000","last_clean_scrub_stamp":"2024-09-16T15:47:48.900296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:19:44.543517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.39","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-16T15:47:49.968650+0000","last_change":"2024-09-16T15:47:49.960684+0000","last_active":"2024-09-16T15:47:49.968650+0000","last_peered":"2024-09-16T15:47:49.968650+0000","last_clean":"2024-09-16T15:47:49.968650+0000","last_became_active":"2024-09-16T15:47:49.960294+0000","last_became_peered":"2024-09-16T15:47:49.960294+0000","last_unstale":"2024-09-16T15:47:49.968650+0000","last_undegraded":"2024-09-16T15:47:49.968650+0000","last_fullsized":"2024-09-16T15:47:49.968650+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:47:48.900296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:47:48.900296+0000","last_clean_scrub_stamp":"2024-09-16T15:47:48.900296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T03:47:33.379740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.38","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-16T15:47:49.964424+0000","last_change":"2024-09-16T15:47:49.961962+0000","last_active":"2024-09-16T15:47:49.964424+0000","last_peered":"2024-09-16T15:47:49.964424+0000","last_clean":"2024-09-16T15:47:49.964424+0000","last_became_active":"2024-09-16T15:47:49.961632+0000","last_became_peered":"2024-09-16T15:47:49.961632+0000","last_unstale":"2024-09-16T15:47:49.964424+0000","last_undegraded":"2024-09-16T15:47:49.964424+0000","last_fullsized":"2024-09-16T15:47:49.964424+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:47:48.900296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:47:48.900296+0000","last_clean_scrub_stamp":"2024-09-16T15:47:48.900296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:32:37.242027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.37","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-16T15:47:49.961797+0000","last_change":"2024-09-16T15:47:49.957050+0000","last_active":"2024-09-16T15:47:49.961797+0000","last_peered":"2024-09-16T15:47:49.961797+0000","last_clean":"2024-09-16T15:47:49.961797+0000","last_became_active":"2024-09-16T15:47:49.956699+0000","last_became_peered":"2024-09-16T15:47:49.956699+0000","last_unstale":"2024-09-16T15:47:49.961797+0000","last_undegraded":"2024-09-16T15:47:49.961797+0000","last_fullsized":"2024-09-16T15:47:49.961797+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:47:48.900296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:47:48.900296+0000","last_clean_scrub_stamp":"2024-09-16T15:47:48.900296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:33:07.023978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.36","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-16T15:47:49.965630+0000","last_change":"2024-09-16T15:47:49.959730+0000","last_active":"2024-09-16T15:47:49.965630+0000","last_peered":"2024-09-16T15:47:49.965630+0000","last_clean":"2024-09-16T15:47:49.965630+0000","last_became_active":"2024-09-16T15:47:49.959395+0000","last_became_peered":"2024-09-16T15:47:49.959395+0000","last_unstale":"2024-09-16T15:47:49.965630+0000","last_undegraded":"2024-09-16T15:47:49.965630+0000","last_fullsized":"2024-09-16T15:47:49.965630+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:47:48.900296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:47:48.900296+0000","last_clean_scrub_stamp":"2024-09-16T15:47:48.900296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T02:14:30.434884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.35","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-16T15:47:49.953682+0000","last_change":"2024-09-16T15:47:49.950477+0000","last_active":"2024-09-16T15:47:49.953682+0000","last_peered":"2024-09-16T15:47:49.953682+0000","last_clean":"2024-09-16T15:47:49.953682+0000","last_became_active":"2024-09-16T15:47:49.950140+0000","last_became_peered":"2024-09-16T15:47:49.950140+0000","last_unstale":"2024-09-16T15:47:49.953682+0000","last_undegraded":"2024-09-16T15:47:49.953682+0000","last_fullsized":"2024-09-16T15:47:49.953682+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:47:48.900296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:47:48.900296+0000","last_clean_scrub_stamp":"2024-09-16T15:47:48.900296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T00:36:31.093928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.34","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-16T15:47:49.965514+0000","last_change":"2024-09-16T15:47:49.955834+0000","last_active":"2024-09-16T15:47:49.965514+0000","last_peered":"2024-09-16T15:47:49.965514+0000","last_clean":"2024-09-16T15:47:49.965514+0000","last_became_active":"2024-09-16T15:47:49.955446+0000","last_became_peered":"2024-09-16T15:47:49.955446+0000","last_unstale":"2024-09-16T15:47:49.965514+0000","last_undegraded":"2024-09-16T15:47:49.965514+0000","last_fullsized":"2024-09-16T15:47:49.965514+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:47:48.900296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:47:48.900296+0000","last_clean_scrub_stamp":"2024-09-16T15:47:48.900296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T18:59:41.944452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.33","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-16T15:47:49.949211+0000","last_change":"2024-09-16T15:47:49.946749+0000","last_active":"2024-09-16T15:47:49.949211+0000","last_peered":"2024-09-16T15:47:49.949211+0000","last_clean":"2024-09-16T15:47:49.949211+0000","last_became_active":"2024-09-16T15:47:49.946411+0000","last_became_peered":"2024-09-16T15:47:49.946411+0000","last_unstale":"2024-09-16T15:47:49.949211+0000","last_undegraded":"2024-09-16T15:47:49.949211+0000","last_fullsized":"2024-09-16T15:47:49.949211+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:47:48.900296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:47:48.900296+0000","last_clean_scrub_stamp":"2024-09-16T15:47:48.900296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:37:13.529669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.32","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-16T15:47:49.968708+0000","last_change":"2024-09-16T15:47:49.962199+0000","last_active":"2024-09-16T15:47:49.968708+0000","last_peered":"2024-09-16T15:47:49.968708+0000","last_clean":"2024-09-16T15:47:49.968708+0000","last_became_active":"2024-09-16T15:47:49.959206+0000","last_became_peered":"2024-09-16T15:47:49.959206+0000","last_unstale":"2024-09-16T15:47:49.968708+0000","last_undegraded":"2024-09-16T15:47:49.968708+0000","last_fullsized":"2024-09-16T15:47:49.968708+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:47:48.900296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:47:48.900296+0000","last_clean_scrub_stamp":"2024-09-16T15:47:48.900296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T22:33:10.685584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.31","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-16T15:47:49.954403+0000","last_change":"2024-09-16T15:47:49.943665+0000","last_active":"2024-09-16T15:47:49.954403+0000","last_peered":"2024-09-16T15:47:49.954403+0000","last_clean":"2024-09-16T15:47:49.954403+0000","last_became_active":"2024-09-16T15:47:49.942996+0000","last_became_peered":"2024-09-16T15:47:49.942996+0000","last_unstale":"2024-09-16T15:47:49.954403+0000","last_undegraded":"2024-09-16T15:47:49.954403+0000","last_fullsized":"2024-09-16T15:47:49.954403+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:47:48.900296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:47:48.900296+0000","last_clean_scrub_stamp":"2024-09-16T15:47:48.900296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:21:09.352580+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.30","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-16T15:47:49.953750+0000","last_change":"2024-09-16T15:47:49.950406+0000","last_active":"2024-09-16T15:47:49.953750+0000","last_peered":"2024-09-16T15:47:49.953750+0000","last_clean":"2024-09-16T15:47:49.953750+0000","last_became_active":"2024-09-16T15:47:49.950033+0000","last_became_peered":"2024-09-16T15:47:49.950033+0000","last_unstale":"2024-09-16T15:47:49.953750+0000","last_undegraded":"2024-09-16T15:47:49.953750+0000","last_fullsized":"2024-09-16T15:47:49.953750+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:47:48.900296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:47:48.900296+0000","last_clean_scrub_stamp":"2024-09-16T15:47:48.900296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T23:26:28.735670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.2f","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-16T15:47:49.942331+0000","last_change":"2024-09-16T15:47:49.940861+0000","last_active":"2024-09-16T15:47:49.942331+0000","last_peered":"2024-09-16T15:47:49.942331+0000","last_clean":"2024-09-16T15:47:49.942331+0000","last_became_active":"2024-09-16T15:47:49.940455+0000","last_became_peered":"2024-09-16T15:47:49.940455+0000","last_unstale":"2024-09-16T15:47:49.942331+0000","last_undegraded":"2024-09-16T15:47:49.942331+0000","last_fullsized":"2024-09-16T15:47:49.942331+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:47:48.900296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:47:48.900296+0000","last_clean_scrub_stamp":"2024-09-16T15:47:48.900296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T03:44:37.047484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.2e","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-16T15:47:49.968762+0000","last_change":"2024-09-16T15:47:49.963895+0000","last_active":"2024-09-16T15:47:49.968762+0000","last_peered":"2024-09-16T15:47:49.968762+0000","last_clean":"2024-09-16T15:47:49.968762+0000","last_became_active":"2024-09-16T15:47:49.963241+0000","last_became_peered":"2024-09-16T15:47:49.963241+0000","last_unstale":"2024-09-16T15:47:49.968762+0000","last_undegraded":"2024-09-16T15:47:49.968762+0000","last_fullsized":"2024-09-16T15:47:49.968762+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:47:48.900296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:47:48.900296+0000","last_clean_scrub_stamp":"2024-09-16T15:47:48.900296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:04:49.834873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.2d","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-16T15:47:49.964268+0000","last_change":"2024-09-16T15:47:49.958744+0000","last_active":"2024-09-16T15:47:49.964268+0000","last_peered":"2024-09-16T15:47:49.964268+0000","last_clean":"2024-09-16T15:47:49.964268+0000","last_became_active":"2024-09-16T15:47:49.956668+0000","last_became_peered":"2024-09-16T15:47:49.956668+0000","last_unstale":"2024-09-16T15:47:49.964268+0000","last_undegraded":"2024-09-16T15:47:49.964268+0000","last_fullsized":"2024-09-16T15:47:49.964268+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:47:48.900296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:47:48.900296+0000","last_clean_scrub_stamp":"2024-09-16T15:47:48.900296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:57:25.380284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.2c","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-16T15:47:49.961194+0000","last_change":"2024-09-16T15:47:49.948275+0000","last_active":"2024-09-16T15:47:49.961194+0000","last_peered":"2024-09-16T15:47:49.961194+0000","last_clean":"2024-09-16T15:47:49.961194+0000","last_became_active":"2024-09-16T15:47:49.947417+0000","last_became_peered":"2024-09-16T15:47:49.947417+0000","last_unstale":"2024-09-16T15:47:49.961194+0000","last_undegraded":"2024-09-16T15:47:49.961194+0000","last_fullsized":"2024-09-16T15:47:49.961194+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:47:48.900296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:47:48.900296+0000","last_clean_scrub_stamp":"2024-09-16T15:47:48.900296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:08:17.863899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.2b","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-16T15:47:49.967584+0000","last_change":"2024-09-16T15:47:49.948581+0000","last_active":"2024-09-16T15:47:49.967584+0000","last_peered":"2024-09-16T15:47:49.967584+0000","last_clean":"2024-09-16T15:47:49.967584+0000","last_became_active":"2024-09-16T15:47:49.947689+0000","last_became_peered":"2024-09-16T15:47:49.947689+0000","last_unstale":"2024-09-16T15:47:49.967584+0000","last_undegraded":"2024-09-16T15:47:49.967584+0000","last_fullsized":"2024-09-16T15:47:49.967584+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:47:48.900296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:47:48.900296+0000","last_clean_scrub_stamp":"2024-09-16T15:47:48.900296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:43:31.738244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.2a","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-16T15:47:49.964508+0000","last_change":"2024-09-16T15:47:49.958769+0000","last_active":"2024-09-16T15:47:49.964508+0000","last_peered":"2024-09-16T15:47:49.964508+0000","last_clean":"2024-09-16T15:47:49.964508+0000","last_became_active":"2024-09-16T15:47:49.956923+0000","last_became_peered":"2024-09-16T15:47:49.956923+0000","last_unstale":"2024-09-16T15:47:49.964508+0000","last_undegraded":"2024-09-16T15:47:49.964508+0000","last_fullsized":"2024-09-16T15:47:49.964508+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:47:48.900296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:47:48.900296+0000","last_clean_scrub_stamp":"2024-09-16T15:47:48.900296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T03:37:28.810840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.29","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-16T15:47:49.965349+0000","last_change":"2024-09-16T15:47:49.953710+0000","last_active":"2024-09-16T15:47:49.965349+0000","last_peered":"2024-09-16T15:47:49.965349+0000","last_clean":"2024-09-16T15:47:49.965349+0000","last_became_active":"2024-09-16T15:47:49.950771+0000","last_became_peered":"2024-09-16T15:47:49.950771+0000","last_unstale":"2024-09-16T15:47:49.965349+0000","last_undegraded":"2024-09-16T15:47:49.965349+0000","last_fullsized":"2024-09-16T15:47:49.965349+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:47:48.900296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:47:48.900296+0000","last_clean_scrub_stamp":"2024-09-16T15:47:48.900296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:06:06.905606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.28","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-16T15:47:49.968835+0000","last_change":"2024-09-16T15:47:49.963059+0000","last_active":"2024-09-16T15:47:49.968835+0000","last_peered":"2024-09-16T15:47:49.968835+0000","last_clean":"2024-09-16T15:47:49.968835+0000","last_became_active":"2024-09-16T15:47:49.962723+0000","last_became_peered":"2024-09-16T15:47:49.962723+0000","last_unstale":"2024-09-16T15:47:49.968835+0000","last_undegraded":"2024-09-16T15:47:49.968835+0000","last_fullsized":"2024-09-16T15:47:49.968835+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:47:48.900296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:47:48.900296+0000","last_clean_scrub_stamp":"2024-09-16T15:47:48.900296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T00:04:30.122975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.27","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-16T15:47:49.963291+0000","last_change":"2024-09-16T15:47:49.958253+0000","last_active":"2024-09-16T15:47:49.963291+0000","last_peered":"2024-09-16T15:47:49.963291+0000","last_clean":"2024-09-16T15:47:49.963291+0000","last_became_active":"2024-09-16T15:47:49.957662+0000","last_became_peered":"2024-09-16T15:47:49.957662+0000","last_unstale":"2024-09-16T15:47:49.963291+0000","last_undegraded":"2024-09-16T15:47:49.963291+0000","last_fullsized":"2024-09-16T15:47:49.963291+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:47:48.900296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:47:48.900296+0000","last_clean_scrub_stamp":"2024-09-16T15:47:48.900296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:52:28.109252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.26","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-16T15:47:49.953458+0000","last_change":"2024-09-16T15:47:49.952381+0000","last_active":"2024-09-16T15:47:49.953458+0000","last_peered":"2024-09-16T15:47:49.953458+0000","last_clean":"2024-09-16T15:47:49.953458+0000","last_became_active":"2024-09-16T15:47:49.951442+0000","last_became_peered":"2024-09-16T15:47:49.951442+0000","last_unstale":"2024-09-16T15:47:49.953458+0000","last_undegraded":"2024-09-16T15:47:49.953458+0000","last_fullsized":"2024-09-16T15:47:49.953458+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:47:48.900296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:47:48.900296+0000","last_clean_scrub_stamp":"2024-09-16T15:47:48.900296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T02:07:55.051351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.25","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-16T15:47:49.946643+0000","last_change":"2024-09-16T15:47:49.942464+0000","last_active":"2024-09-16T15:47:49.946643+0000","last_peered":"2024-09-16T15:47:49.946643+0000","last_clean":"2024-09-16T15:47:49.946643+0000","last_became_active":"2024-09-16T15:47:49.942105+0000","last_became_peered":"2024-09-16T15:47:49.942105+0000","last_unstale":"2024-09-16T15:47:49.946643+0000","last_undegraded":"2024-09-16T15:47:49.946643+0000","last_fullsized":"2024-09-16T15:47:49.946643+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:47:48.900296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:47:48.900296+0000","last_clean_scrub_stamp":"2024-09-16T15:47:48.900296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T23:45:21.885067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.24","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-16T15:47:49.964220+0000","last_change":"2024-09-16T15:47:49.951311+0000","last_active":"2024-09-16T15:47:49.964220+0000","last_peered":"2024-09-16T15:47:49.964220+0000","last_clean":"2024-09-16T15:47:49.964220+0000","last_became_active":"2024-09-16T15:47:49.950928+0000","last_became_peered":"2024-09-16T15:47:49.950928+0000","last_unstale":"2024-09-16T15:47:49.964220+0000","last_undegraded":"2024-09-16T15:47:49.964220+0000","last_fullsized":"2024-09-16T15:47:49.964220+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:47:48.900296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:47:48.900296+0000","last_clean_scrub_stamp":"2024-09-16T15:47:48.900296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T00:04:45.746226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.23","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-16T15:47:49.956659+0000","last_change":"2024-09-16T15:47:49.955480+0000","last_active":"2024-09-16T15:47:49.956659+0000","last_peered":"2024-09-16T15:47:49.956659+0000","last_clean":"2024-09-16T15:47:49.956659+0000","last_became_active":"2024-09-16T15:47:49.955164+0000","last_became_peered":"2024-09-16T15:47:49.955164+0000","last_unstale":"2024-09-16T15:47:49.956659+0000","last_undegraded":"2024-09-16T15:47:49.956659+0000","last_fullsized":"2024-09-16T15:47:49.956659+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:47:48.900296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:47:48.900296+0000","last_clean_scrub_stamp":"2024-09-16T15:47:48.900296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T02:28:31.665606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.22","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-16T15:47:49.971251+0000","last_change":"2024-09-16T15:47:49.969647+0000","last_active":"2024-09-16T15:47:49.971251+0000","last_peered":"2024-09-16T15:47:49.971251+0000","last_clean":"2024-09-16T15:47:49.971251+0000","last_became_active":"2024-09-16T15:47:49.969364+0000","last_became_peered":"2024-09-16T15:47:49.969364+0000","last_unstale":"2024-09-16T15:47:49.971251+0000","last_undegraded":"2024-09-16T15:47:49.971251+0000","last_fullsized":"2024-09-16T15:47:49.971251+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:47:48.900296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:47:48.900296+0000","last_clean_scrub_stamp":"2024-09-16T15:47:48.900296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T00:46:15.436142+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.21","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-16T15:47:49.946887+0000","last_change":"2024-09-16T15:47:49.944362+0000","last_active":"2024-09-16T15:47:49.946887+0000","last_peered":"2024-09-16T15:47:49.946887+0000","last_clean":"2024-09-16T15:47:49.946887+0000","last_became_active":"2024-09-16T15:47:49.943769+0000","last_became_peered":"2024-09-16T15:47:49.943769+0000","last_unstale":"2024-09-16T15:47:49.946887+0000","last_undegraded":"2024-09-16T15:47:49.946887+0000","last_fullsized":"2024-09-16T15:47:49.946887+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:47:48.900296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:47:48.900296+0000","last_clean_scrub_stamp":"2024-09-16T15:47:48.900296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T01:10:12.302580+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.20","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-16T15:47:49.965233+0000","last_change":"2024-09-16T15:47:49.954464+0000","last_active":"2024-09-16T15:47:49.965233+0000","last_peered":"2024-09-16T15:47:49.965233+0000","last_clean":"2024-09-16T15:47:49.965233+0000","last_became_active":"2024-09-16T15:47:49.951610+0000","last_became_peered":"2024-09-16T15:47:49.951610+0000","last_unstale":"2024-09-16T15:47:49.965233+0000","last_undegraded":"2024-09-16T15:47:49.965233+0000","last_fullsized":"2024-09-16T15:47:49.965233+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:47:48.900296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:47:48.900296+0000","last_clean_scrub_stamp":"2024-09-16T15:47:48.900296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T01:40:00.990103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1e","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-16T15:47:49.960907+0000","last_change":"2024-09-16T15:47:49.954470+0000","last_active":"2024-09-16T15:47:49.960907+0000","last_peered":"2024-09-16T15:47:49.960907+0000","last_clean":"2024-09-16T15:47:49.960907+0000","last_became_active":"2024-09-16T15:47:49.954122+0000","last_became_peered":"2024-09-16T15:47:49.954122+0000","last_unstale":"2024-09-16T15:47:49.960907+0000","last_undegraded":"2024-09-16T15:47:49.960907+0000","last_fullsized":"2024-09-16T15:47:49.960907+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:47:48.900296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:47:48.900296+0000","last_clean_scrub_stamp":"2024-09-16T15:47:48.900296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T02:14:30.434884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.1f","version":"84'4","reported_seq":17,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-16T15:47:49.969379+0000","last_change":"2024-09-16T15:47:48.955316+0000","last_active":"2024-09-16T15:47:49.969379+0000","last_peered":"2024-09-16T15:47:49.969379+0000","last_clean":"2024-09-16T15:47:49.969379+0000","last_became_active":"2024-09-16T15:47:48.953993+0000","last_became_peered":"2024-09-16T15:47:48.953993+0000","last_unstale":"2024-09-16T15:47:49.969379+0000","last_undegraded":"2024-09-16T15:47:49.969379+0000","last_fullsized":"2024-09-16T15:47:49.969379+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:47:47.896376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:47:47.896376+0000","last_clean_scrub_stamp":"2024-09-16T15:47:47.896376+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T02:04:51.201894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1f","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-16T15:47:49.965713+0000","last_change":"2024-09-16T15:47:49.960313+0000","last_active":"2024-09-16T15:47:49.965713+0000","last_peered":"2024-09-16T15:47:49.965713+0000","last_clean":"2024-09-16T15:47:49.965713+0000","last_became_active":"2024-09-16T15:47:49.959756+0000","last_became_peered":"2024-09-16T15:47:49.959756+0000","last_unstale":"2024-09-16T15:47:49.965713+0000","last_undegraded":"2024-09-16T15:47:49.965713+0000","last_fullsized":"2024-09-16T15:47:49.965713+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:47:48.900296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:47:48.900296+0000","last_clean_scrub_stamp":"2024-09-16T15:47:48.900296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:20:52.357755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.1e","version":"84'1","reported_seq":14,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-16T15:47:49.968016+0000","last_change":"2024-09-16T15:47:48.947418+0000","last_active":"2024-09-16T15:47:49.968016+0000","last_peered":"2024-09-16T15:47:49.968016+0000","last_clean":"2024-09-16T15:47:49.968016+0000","last_became_active":"2024-09-16T15:47:48.946721+0000","last_became_peered":"2024-09-16T15:47:48.946721+0000","last_unstale":"2024-09-16T15:47:49.968016+0000","last_undegraded":"2024-09-16T15:47:49.968016+0000","last_fullsized":"2024-09-16T15:47:49.968016+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:47:47.896376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:47:47.896376+0000","last_clean_scrub_stamp":"2024-09-16T15:47:47.896376+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T02:04:51.201894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.1c","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-16T15:47:49.963788+0000","last_change":"2024-09-16T15:47:49.950226+0000","last_active":"2024-09-16T15:47:49.963788+0000","last_peered":"2024-09-16T15:47:49.963788+0000","last_clean":"2024-09-16T15:47:49.963788+0000","last_became_active":"2024-09-16T15:47:49.949883+0000","last_became_peered":"2024-09-16T15:47:49.949883+0000","last_unstale":"2024-09-16T15:47:49.963788+0000","last_undegraded":"2024-09-16T15:47:49.963788+0000","last_fullsized":"2024-09-16T15:47:49.963788+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:47:48.900296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:47:48.900296+0000","last_clean_scrub_stamp":"2024-09-16T15:47:48.900296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T01:10:12.302580+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.1d","version":"84'2","reported_seq":15,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-16T15:47:49.969800+0000","last_change":"2024-09-16T15:47:48.936871+0000","last_active":"2024-09-16T15:47:49.969800+0000","last_peered":"2024-09-16T15:47:49.969800+0000","last_clean":"2024-09-16T15:47:49.969800+0000","last_became_active":"2024-09-16T15:47:48.936597+0000","last_became_peered":"2024-09-16T15:47:48.936597+0000","last_unstale":"2024-09-16T15:47:49.969800+0000","last_undegraded":"2024-09-16T15:47:49.969800+0000","last_fullsized":"2024-09-16T15:47:49.969800+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:47:47.896376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:47:47.896376+0000","last_clean_scrub_stamp":"2024-09-16T15:47:47.896376+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T02:32:37.856846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1d","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-16T15:47:49.952288+0000","last_change":"2024-09-16T15:47:49.949921+0000","last_active":"2024-09-16T15:47:49.952288+0000","last_peered":"2024-09-16T15:47:49.952288+0000","last_clean":"2024-09-16T15:47:49.952288+0000","last_became_active":"2024-09-16T15:47:49.949454+0000","last_became_peered":"2024-09-16T15:47:49.949454+0000","last_unstale":"2024-09-16T15:47:49.952288+0000","last_undegraded":"2024-09-16T15:47:49.952288+0000","last_fullsized":"2024-09-16T15:47:49.952288+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:47:48.900296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:47:48.900296+0000","last_clean_scrub_stamp":"2024-09-16T15:47:48.900296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T00:55:43.589146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1c","version":"84'2","reported_seq":15,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-16T15:47:49.971603+0000","last_change":"2024-09-16T15:47:48.938690+0000","last_active":"2024-09-16T15:47:49.971603+0000","last_peered":"2024-09-16T15:47:49.971603+0000","last_clean":"2024-09-16T15:47:49.971603+0000","last_became_active":"2024-09-16T15:47:48.938155+0000","last_became_peered":"2024-09-16T15:47:48.938155+0000","last_unstale":"2024-09-16T15:47:49.971603+0000","last_undegraded":"2024-09-16T15:47:49.971603+0000","last_fullsized":"2024-09-16T15:47:49.971603+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:47:47.896376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:47:47.896376+0000","last_clean_scrub_stamp":"2024-09-16T15:47:47.896376+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:11:58.285172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1a","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-16T15:47:49.971217+0000","last_change":"2024-09-16T15:47:49.968094+0000","last_active":"2024-09-16T15:47:49.971217+0000","last_peered":"2024-09-16T15:47:49.971217+0000","last_clean":"2024-09-16T15:47:49.971217+0000","last_became_active":"2024-09-16T15:47:49.965959+0000","last_became_peered":"2024-09-16T15:47:49.965959+0000","last_unstale":"2024-09-16T15:47:49.971217+0000","last_undegraded":"2024-09-16T15:47:49.971217+0000","last_fullsized":"2024-09-16T15:47:49.971217+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:47:48.900296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:47:48.900296+0000","last_clean_scrub_stamp":"2024-09-16T15:47:48.900296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:21:44.087843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.1b","version":"84'1","reported_seq":14,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-16T15:47:49.971203+0000","last_change":"2024-09-16T15:47:48.939448+0000","last_active":"2024-09-16T15:47:49.971203+0000","last_peered":"2024-09-16T15:47:49.971203+0000","last_clean":"2024-09-16T15:47:49.971203+0000","last_became_active":"2024-09-16T15:47:48.938758+0000","last_became_peered":"2024-09-16T15:47:48.938758+0000","last_unstale":"2024-09-16T15:47:49.971203+0000","last_undegraded":"2024-09-16T15:47:49.971203+0000","last_fullsized":"2024-09-16T15:47:49.971203+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:47:47.896376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:47:47.896376+0000","last_clean_scrub_stamp":"2024-09-16T15:47:47.896376+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:54:59.899958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.1b","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-16T15:47:49.946979+0000","last_change":"2024-09-16T15:47:49.944951+0000","last_active":"2024-09-16T15:47:49.946979+0000","last_peered":"2024-09-16T15:47:49.946979+0000","last_clean":"2024-09-16T15:47:49.946979+0000","last_became_active":"2024-09-16T15:47:49.944068+0000","last_became_peered":"2024-09-16T15:47:49.944068+0000","last_unstale":"2024-09-16T15:47:49.946979+0000","last_undegraded":"2024-09-16T15:47:49.946979+0000","last_fullsized":"2024-09-16T15:47:49.946979+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:47:48.900296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:47:48.900296+0000","last_clean_scrub_stamp":"2024-09-16T15:47:48.900296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T00:04:45.746226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.1a","version":"84'1","reported_seq":14,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-16T15:47:49.968408+0000","last_change":"2024-09-16T15:47:48.947975+0000","last_active":"2024-09-16T15:47:49.968408+0000","last_peered":"2024-09-16T15:47:49.968408+0000","last_clean":"2024-09-16T15:47:49.968408+0000","last_became_active":"2024-09-16T15:47:48.945655+0000","last_became_peered":"2024-09-16T15:47:48.945655+0000","last_unstale":"2024-09-16T15:47:49.968408+0000","last_undegraded":"2024-09-16T15:47:49.968408+0000","last_fullsized":"2024-09-16T15:47:49.968408+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:47:47.896376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:47:47.896376+0000","last_clean_scrub_stamp":"2024-09-16T15:47:47.896376+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T01:13:31.262332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.18","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-16T15:47:49.958863+0000","last_change":"2024-09-16T15:47:49.956715+0000","last_active":"2024-09-16T15:47:49.958863+0000","last_peered":"2024-09-16T15:47:49.958863+0000","last_clean":"2024-09-16T15:47:49.958863+0000","last_became_active":"2024-09-16T15:47:49.956383+0000","last_became_peered":"2024-09-16T15:47:49.956383+0000","last_unstale":"2024-09-16T15:47:49.958863+0000","last_undegraded":"2024-09-16T15:47:49.958863+0000","last_fullsized":"2024-09-16T15:47:49.958863+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:47:48.900296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:47:48.900296+0000","last_clean_scrub_stamp":"2024-09-16T15:47:48.900296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:18:26.678453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.19","version":"0'0","reported_seq":13,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-16T15:47:49.932543+0000","last_change":"2024-09-16T15:47:48.932898+0000","last_active":"2024-09-16T15:47:49.932543+0000","last_peered":"2024-09-16T15:47:49.932543+0000","last_clean":"2024-09-16T15:47:49.932543+0000","last_became_active":"2024-09-16T15:47:48.932438+0000","last_became_peered":"2024-09-16T15:47:48.932438+0000","last_unstale":"2024-09-16T15:47:49.932543+0000","last_undegraded":"2024-09-16T15:47:49.932543+0000","last_fullsized":"2024-09-16T15:47:49.932543+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:47:47.896376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:47:47.896376+0000","last_clean_scrub_stamp":"2024-09-16T15:47:47.896376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T23:56:33.543562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.19","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-16T15:47:49.967023+0000","last_change":"2024-09-16T15:47:49.949215+0000","last_active":"2024-09-16T15:47:49.967023+0000","last_peered":"2024-09-16T15:47:49.967023+0000","last_clean":"2024-09-16T15:47:49.967023+0000","last_became_active":"2024-09-16T15:47:49.947130+0000","last_became_peered":"2024-09-16T15:47:49.947130+0000","last_unstale":"2024-09-16T15:47:49.967023+0000","last_undegraded":"2024-09-16T15:47:49.967023+0000","last_fullsized":"2024-09-16T15:47:49.967023+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:47:48.900296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:47:48.900296+0000","last_clean_scrub_stamp":"2024-09-16T15:47:48.900296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T02:53:54.255699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.18","version":"0'0","reported_seq":13,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-16T15:47:49.926748+0000","last_change":"2024-09-16T15:47:48.942427+0000","last_active":"2024-09-16T15:47:49.926748+0000","last_peered":"2024-09-16T15:47:49.926748+0000","last_clean":"2024-09-16T15:47:49.926748+0000","last_became_active":"2024-09-16T15:47:48.941828+0000","last_became_peered":"2024-09-16T15:47:48.941828+0000","last_unstale":"2024-09-16T15:47:49.926748+0000","last_undegraded":"2024-09-16T15:47:49.926748+0000","last_fullsized":"2024-09-16T15:47:49.926748+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:47:47.896376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:47:47.896376+0000","last_clean_scrub_stamp":"2024-09-16T15:47:47.896376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:49:00.837314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.16","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-16T15:47:49.967017+0000","last_change":"2024-09-16T15:47:49.951327+0000","last_active":"2024-09-16T15:47:49.967017+0000","last_peered":"2024-09-16T15:47:49.967017+0000","last_clean":"2024-09-16T15:47:49.967017+0000","last_became_active":"2024-09-16T15:47:49.951047+0000","last_became_peered":"2024-09-16T15:47:49.951047+0000","last_unstale":"2024-09-16T15:47:49.967017+0000","last_undegraded":"2024-09-16T15:47:49.967017+0000","last_fullsized":"2024-09-16T15:47:49.967017+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:47:48.900296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:47:48.900296+0000","last_clean_scrub_stamp":"2024-09-16T15:47:48.900296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:42:34.333211+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.17","version":"0'0","reported_seq":13,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-16T15:47:49.944184+0000","last_change":"2024-09-16T15:47:48.944783+0000","last_active":"2024-09-16T15:47:49.944184+0000","last_peered":"2024-09-16T15:47:49.944184+0000","last_clean":"2024-09-16T15:47:49.944184+0000","last_became_active":"2024-09-16T15:47:48.944048+0000","last_became_peered":"2024-09-16T15:47:48.944048+0000","last_unstale":"2024-09-16T15:47:49.944184+0000","last_undegraded":"2024-09-16T15:47:49.944184+0000","last_fullsized":"2024-09-16T15:47:49.944184+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:47:47.896376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:47:47.896376+0000","last_clean_scrub_stamp":"2024-09-16T15:47:47.896376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T03:44:36.043565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.17","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-16T15:47:49.952340+0000","last_change":"2024-09-16T15:47:49.947717+0000","last_active":"2024-09-16T15:47:49.952340+0000","last_peered":"2024-09-16T15:47:49.952340+0000","last_clean":"2024-09-16T15:47:49.952340+0000","last_became_active":"2024-09-16T15:47:49.947382+0000","last_became_peered":"2024-09-16T15:47:49.947382+0000","last_unstale":"2024-09-16T15:47:49.952340+0000","last_undegraded":"2024-09-16T15:47:49.952340+0000","last_fullsized":"2024-09-16T15:47:49.952340+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:47:48.900296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:47:48.900296+0000","last_clean_scrub_stamp":"2024-09-16T15:47:48.900296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:46:07.109158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.16","version":"0'0","reported_seq":13,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-16T15:47:49.933651+0000","last_change":"2024-09-16T15:47:48.952563+0000","last_active":"2024-09-16T15:47:49.933651+0000","last_peered":"2024-09-16T15:47:49.933651+0000","last_clean":"2024-09-16T15:47:49.933651+0000","last_became_active":"2024-09-16T15:47:48.951819+0000","last_became_peered":"2024-09-16T15:47:48.951819+0000","last_unstale":"2024-09-16T15:47:49.933651+0000","last_undegraded":"2024-09-16T15:47:49.933651+0000","last_fullsized":"2024-09-16T15:47:49.933651+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:47:47.896376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:47:47.896376+0000","last_clean_scrub_stamp":"2024-09-16T15:47:47.896376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T18:51:35.155740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.14","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-16T15:47:49.959697+0000","last_change":"2024-09-16T15:47:49.955291+0000","last_active":"2024-09-16T15:47:49.959697+0000","last_peered":"2024-09-16T15:47:49.959697+0000","last_clean":"2024-09-16T15:47:49.959697+0000","last_became_active":"2024-09-16T15:47:49.954525+0000","last_became_peered":"2024-09-16T15:47:49.954525+0000","last_unstale":"2024-09-16T15:47:49.959697+0000","last_undegraded":"2024-09-16T15:47:49.959697+0000","last_fullsized":"2024-09-16T15:47:49.959697+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:47:48.900296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:47:48.900296+0000","last_clean_scrub_stamp":"2024-09-16T15:47:48.900296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T00:22:18.050637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.15","version":"84'3","reported_seq":16,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-16T15:47:49.971701+0000","last_change":"2024-09-16T15:47:48.937825+0000","last_active":"2024-09-16T15:47:49.971701+0000","last_peered":"2024-09-16T15:47:49.971701+0000","last_clean":"2024-09-16T15:47:49.971701+0000","last_became_active":"2024-09-16T15:47:48.937020+0000","last_became_peered":"2024-09-16T15:47:48.937020+0000","last_unstale":"2024-09-16T15:47:49.971701+0000","last_undegraded":"2024-09-16T15:47:49.971701+0000","last_fullsized":"2024-09-16T15:47:49.971701+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:47:47.896376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:47:47.896376+0000","last_clean_scrub_stamp":"2024-09-16T15:47:47.896376+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:31:05.983747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.15","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-16T15:47:49.952691+0000","last_change":"2024-09-16T15:47:49.948684+0000","last_active":"2024-09-16T15:47:49.952691+0000","last_peered":"2024-09-16T15:47:49.952691+0000","last_clean":"2024-09-16T15:47:49.952691+0000","last_became_active":"2024-09-16T15:47:49.948316+0000","last_became_peered":"2024-09-16T15:47:49.948316+0000","last_unstale":"2024-09-16T15:47:49.952691+0000","last_undegraded":"2024-09-16T15:47:49.952691+0000","last_fullsized":"2024-09-16T15:47:49.952691+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:47:48.900296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:47:48.900296+0000","last_clean_scrub_stamp":"2024-09-16T15:47:48.900296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T18:59:41.944452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.14","version":"84'6","reported_seq":19,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-16T15:47:49.975591+0000","last_change":"2024-09-16T15:47:48.937794+0000","last_active":"2024-09-16T15:47:49.975591+0000","last_peered":"2024-09-16T15:47:49.975591+0000","last_clean":"2024-09-16T15:47:49.975591+0000","last_became_active":"2024-09-16T15:47:48.937048+0000","last_became_peered":"2024-09-16T15:47:48.937048+0000","last_unstale":"2024-09-16T15:47:49.975591+0000","last_undegraded":"2024-09-16T15:47:49.975591+0000","last_fullsized":"2024-09-16T15:47:49.975591+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:47:47.896376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:47:47.896376+0000","last_clean_scrub_stamp":"2024-09-16T15:47:47.896376+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T01:00:53.376119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.12","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-16T15:47:49.955792+0000","last_change":"2024-09-16T15:47:49.945512+0000","last_active":"2024-09-16T15:47:49.955792+0000","last_peered":"2024-09-16T15:47:49.955792+0000","last_clean":"2024-09-16T15:47:49.955792+0000","last_became_active":"2024-09-16T15:47:49.944649+0000","last_became_peered":"2024-09-16T15:47:49.944649+0000","last_unstale":"2024-09-16T15:47:49.955792+0000","last_undegraded":"2024-09-16T15:47:49.955792+0000","last_fullsized":"2024-09-16T15:47:49.955792+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:47:48.900296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:47:48.900296+0000","last_clean_scrub_stamp":"2024-09-16T15:47:48.900296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T00:04:26.824059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.13","version":"84'2","reported_seq":15,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-16T15:47:49.969134+0000","last_change":"2024-09-16T15:47:48.935163+0000","last_active":"2024-09-16T15:47:49.969134+0000","last_peered":"2024-09-16T15:47:49.969134+0000","last_clean":"2024-09-16T15:47:49.969134+0000","last_became_active":"2024-09-16T15:47:48.934838+0000","last_became_peered":"2024-09-16T15:47:48.934838+0000","last_unstale":"2024-09-16T15:47:49.969134+0000","last_undegraded":"2024-09-16T15:47:49.969134+0000","last_fullsized":"2024-09-16T15:47:49.969134+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:47:47.896376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:47:47.896376+0000","last_clean_scrub_stamp":"2024-09-16T15:47:47.896376+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:17:14.703070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.13","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-16T15:47:49.949074+0000","last_change":"2024-09-16T15:47:49.945243+0000","last_active":"2024-09-16T15:47:49.949074+0000","last_peered":"2024-09-16T15:47:49.949074+0000","last_clean":"2024-09-16T15:47:49.949074+0000","last_became_active":"2024-09-16T15:47:49.944910+0000","last_became_peered":"2024-09-16T15:47:49.944910+0000","last_unstale":"2024-09-16T15:47:49.949074+0000","last_undegraded":"2024-09-16T15:47:49.949074+0000","last_fullsized":"2024-09-16T15:47:49.949074+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:47:48.900296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:47:48.900296+0000","last_clean_scrub_stamp":"2024-09-16T15:47:48.900296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T22:08:30.436461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.12","version":"0'0","reported_seq":13,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-16T15:47:49.918876+0000","last_change":"2024-09-16T15:47:48.933762+0000","last_active":"2024-09-16T15:47:49.918876+0000","last_peered":"2024-09-16T15:47:49.918876+0000","last_clean":"2024-09-16T15:47:49.918876+0000","last_became_active":"2024-09-16T15:47:48.933454+0000","last_became_peered":"2024-09-16T15:47:48.933454+0000","last_unstale":"2024-09-16T15:47:49.918876+0000","last_undegraded":"2024-09-16T15:47:49.918876+0000","last_fullsized":"2024-09-16T15:47:49.918876+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:47:47.896376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:47:47.896376+0000","last_clean_scrub_stamp":"2024-09-16T15:47:47.896376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:34:51.472023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.10","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-16T15:47:49.958604+0000","last_change":"2024-09-16T15:47:49.949896+0000","last_active":"2024-09-16T15:47:49.958604+0000","last_peered":"2024-09-16T15:47:49.958604+0000","last_clean":"2024-09-16T15:47:49.958604+0000","last_became_active":"2024-09-16T15:47:49.945795+0000","last_became_peered":"2024-09-16T15:47:49.945795+0000","last_unstale":"2024-09-16T15:47:49.958604+0000","last_undegraded":"2024-09-16T15:47:49.958604+0000","last_fullsized":"2024-09-16T15:47:49.958604+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:47:48.900296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:47:48.900296+0000","last_clean_scrub_stamp":"2024-09-16T15:47:48.900296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:33:49.653526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.11","version":"0'0","reported_seq":13,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-16T15:47:49.938870+0000","last_change":"2024-09-16T15:47:48.950081+0000","last_active":"2024-09-16T15:47:49.938870+0000","last_peered":"2024-09-16T15:47:49.938870+0000","last_clean":"2024-09-16T15:47:49.938870+0000","last_became_active":"2024-09-16T15:47:48.949230+0000","last_became_peered":"2024-09-16T15:47:48.949230+0000","last_unstale":"2024-09-16T15:47:49.938870+0000","last_undegraded":"2024-09-16T15:47:49.938870+0000","last_fullsized":"2024-09-16T15:47:49.938870+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:47:47.896376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:47:47.896376+0000","last_clean_scrub_stamp":"2024-09-16T15:47:47.896376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:02:43.375408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.11","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-16T15:47:49.954196+0000","last_change":"2024-09-16T15:47:49.952975+0000","last_active":"2024-09-16T15:47:49.954196+0000","last_peered":"2024-09-16T15:47:49.954196+0000","last_clean":"2024-09-16T15:47:49.954196+0000","last_became_active":"2024-09-16T15:47:49.952671+0000","last_became_peered":"2024-09-16T15:47:49.952671+0000","last_unstale":"2024-09-16T15:47:49.954196+0000","last_undegraded":"2024-09-16T15:47:49.954196+0000","last_fullsized":"2024-09-16T15:47:49.954196+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:47:48.900296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:47:48.900296+0000","last_clean_scrub_stamp":"2024-09-16T15:47:48.900296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T18:15:01.990027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.10","version":"84'8","reported_seq":21,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-16T15:47:49.970650+0000","last_change":"2024-09-16T15:47:48.950051+0000","last_active":"2024-09-16T15:47:49.970650+0000","last_peered":"2024-09-16T15:47:49.970650+0000","last_clean":"2024-09-16T15:47:49.970650+0000","last_became_active":"2024-09-16T15:47:48.949613+0000","last_became_peered":"2024-09-16T15:47:48.949613+0000","last_unstale":"2024-09-16T15:47:49.970650+0000","last_undegraded":"2024-09-16T15:47:49.970650+0000","last_fullsized":"2024-09-16T15:47:49.970650+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:47:47.896376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:47:47.896376+0000","last_clean_scrub_stamp":"2024-09-16T15:47:47.896376+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:32:36.238108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.e","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-16T15:47:49.949029+0000","last_change":"2024-09-16T15:47:49.943986+0000","last_active":"2024-09-16T15:47:49.949029+0000","last_peered":"2024-09-16T15:47:49.949029+0000","last_clean":"2024-09-16T15:47:49.949029+0000","last_became_active":"2024-09-16T15:47:49.943648+0000","last_became_peered":"2024-09-16T15:47:49.943648+0000","last_unstale":"2024-09-16T15:47:49.949029+0000","last_undegraded":"2024-09-16T15:47:49.949029+0000","last_fullsized":"2024-09-16T15:47:49.949029+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:47:48.900296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:47:48.900296+0000","last_clean_scrub_stamp":"2024-09-16T15:47:48.900296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T23:31:42.834248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.f","version":"84'1","reported_seq":14,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-16T15:47:49.967813+0000","last_change":"2024-09-16T15:47:48.949583+0000","last_active":"2024-09-16T15:47:49.967813+0000","last_peered":"2024-09-16T15:47:49.967813+0000","last_clean":"2024-09-16T15:47:49.967813+0000","last_became_active":"2024-09-16T15:47:48.948872+0000","last_became_peered":"2024-09-16T15:47:48.948872+0000","last_unstale":"2024-09-16T15:47:49.967813+0000","last_undegraded":"2024-09-16T15:47:49.967813+0000","last_fullsized":"2024-09-16T15:47:49.967813+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:47:47.896376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:47:47.896376+0000","last_clean_scrub_stamp":"2024-09-16T15:47:47.896376+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T23:45:20.881147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.f","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-16T15:47:49.965508+0000","last_change":"2024-09-16T15:47:49.954670+0000","last_active":"2024-09-16T15:47:49.965508+0000","last_peered":"2024-09-16T15:47:49.965508+0000","last_clean":"2024-09-16T15:47:49.965508+0000","last_became_active":"2024-09-16T15:47:49.954335+0000","last_became_peered":"2024-09-16T15:47:49.954335+0000","last_unstale":"2024-09-16T15:47:49.965508+0000","last_undegraded":"2024-09-16T15:47:49.965508+0000","last_fullsized":"2024-09-16T15:47:49.965508+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:47:48.900296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:47:48.900296+0000","last_clean_scrub_stamp":"2024-09-16T15:47:48.900296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T23:50:41.566032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.e","version":"84'2","reported_seq":15,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-16T15:47:49.967357+0000","last_change":"2024-09-16T15:47:48.948673+0000","last_active":"2024-09-16T15:47:49.967357+0000","last_peered":"2024-09-16T15:47:49.967357+0000","last_clean":"2024-09-16T15:47:49.967357+0000","last_became_active":"2024-09-16T15:47:48.948305+0000","last_became_peered":"2024-09-16T15:47:48.948305+0000","last_unstale":"2024-09-16T15:47:49.967357+0000","last_undegraded":"2024-09-16T15:47:49.967357+0000","last_fullsized":"2024-09-16T15:47:49.967357+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:47:47.896376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:47:47.896376+0000","last_clean_scrub_stamp":"2024-09-16T15:47:47.896376+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:19:43.539598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.d","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-16T15:47:49.949814+0000","last_change":"2024-09-16T15:47:49.947062+0000","last_active":"2024-09-16T15:47:49.949814+0000","last_peered":"2024-09-16T15:47:49.949814+0000","last_clean":"2024-09-16T15:47:49.949814+0000","last_became_active":"2024-09-16T15:47:49.946466+0000","last_became_peered":"2024-09-16T15:47:49.946466+0000","last_unstale":"2024-09-16T15:47:49.949814+0000","last_undegraded":"2024-09-16T15:47:49.949814+0000","last_fullsized":"2024-09-16T15:47:49.949814+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:47:48.900296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:47:48.900296+0000","last_clean_scrub_stamp":"2024-09-16T15:47:48.900296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T23:50:41.566032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.c","version":"0'0","reported_seq":13,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-16T15:47:49.930396+0000","last_change":"2024-09-16T15:47:48.944636+0000","last_active":"2024-09-16T15:47:49.930396+0000","last_peered":"2024-09-16T15:47:49.930396+0000","last_clean":"2024-09-16T15:47:49.930396+0000","last_became_active":"2024-09-16T15:47:48.944233+0000","last_became_peered":"2024-09-16T15:47:48.944233+0000","last_unstale":"2024-09-16T15:47:49.930396+0000","last_undegraded":"2024-09-16T15:47:49.930396+0000","last_fullsized":"2024-09-16T15:47:49.930396+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:47:47.896376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:47:47.896376+0000","last_clean_scrub_stamp":"2024-09-16T15:47:47.896376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:27:47.846253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.a","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-16T15:47:49.968675+0000","last_change":"2024-09-16T15:47:49.959515+0000","last_active":"2024-09-16T15:47:49.968675+0000","last_peered":"2024-09-16T15:47:49.968675+0000","last_clean":"2024-09-16T15:47:49.968675+0000","last_became_active":"2024-09-16T15:47:49.958726+0000","last_became_peered":"2024-09-16T15:47:49.958726+0000","last_unstale":"2024-09-16T15:47:49.968675+0000","last_undegraded":"2024-09-16T15:47:49.968675+0000","last_fullsized":"2024-09-16T15:47:49.968675+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:47:48.900296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:47:48.900296+0000","last_clean_scrub_stamp":"2024-09-16T15:47:48.900296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T23:47:32.797895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.b","version":"84'2","reported_seq":15,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-16T15:47:49.969318+0000","last_change":"2024-09-16T15:47:48.935075+0000","last_active":"2024-09-16T15:47:49.969318+0000","last_peered":"2024-09-16T15:47:49.969318+0000","last_clean":"2024-09-16T15:47:49.969318+0000","last_became_active":"2024-09-16T15:47:48.933928+0000","last_became_peered":"2024-09-16T15:47:48.933928+0000","last_unstale":"2024-09-16T15:47:49.969318+0000","last_undegraded":"2024-09-16T15:47:49.969318+0000","last_fullsized":"2024-09-16T15:47:49.969318+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:47:47.896376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:47:47.896376+0000","last_clean_scrub_stamp":"2024-09-16T15:47:47.896376+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T02:31:10.840460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.b","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-16T15:47:49.960071+0000","last_change":"2024-09-16T15:47:49.955540+0000","last_active":"2024-09-16T15:47:49.960071+0000","last_peered":"2024-09-16T15:47:49.960071+0000","last_clean":"2024-09-16T15:47:49.960071+0000","last_became_active":"2024-09-16T15:47:49.955159+0000","last_became_peered":"2024-09-16T15:47:49.955159+0000","last_unstale":"2024-09-16T15:47:49.960071+0000","last_undegraded":"2024-09-16T15:47:49.960071+0000","last_fullsized":"2024-09-16T15:47:49.960071+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:47:48.900296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:47:48.900296+0000","last_clean_scrub_stamp":"2024-09-16T15:47:48.900296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:10:31.222177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.a","version":"0'0","reported_seq":13,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-16T15:47:49.930610+0000","last_change":"2024-09-16T15:47:48.937958+0000","last_active":"2024-09-16T15:47:49.930610+0000","last_peered":"2024-09-16T15:47:49.930610+0000","last_clean":"2024-09-16T15:47:49.930610+0000","last_became_active":"2024-09-16T15:47:48.937371+0000","last_became_peered":"2024-09-16T15:47:48.937371+0000","last_unstale":"2024-09-16T15:47:49.930610+0000","last_undegraded":"2024-09-16T15:47:49.930610+0000","last_fullsized":"2024-09-16T15:47:49.930610+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:47:47.896376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:47:47.896376+0000","last_clean_scrub_stamp":"2024-09-16T15:47:47.896376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:33.183070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.8","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-16T15:47:49.961264+0000","last_change":"2024-09-16T15:47:49.947927+0000","last_active":"2024-09-16T15:47:49.961264+0000","last_peered":"2024-09-16T15:47:49.961264+0000","last_clean":"2024-09-16T15:47:49.961264+0000","last_became_active":"2024-09-16T15:47:49.947313+0000","last_became_peered":"2024-09-16T15:47:49.947313+0000","last_unstale":"2024-09-16T15:47:49.961264+0000","last_undegraded":"2024-09-16T15:47:49.961264+0000","last_fullsized":"2024-09-16T15:47:49.961264+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:47:48.900296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:47:48.900296+0000","last_clean_scrub_stamp":"2024-09-16T15:47:48.900296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T22:05:54.637849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.9","version":"0'0","reported_seq":13,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-16T15:47:49.926283+0000","last_change":"2024-09-16T15:47:48.950021+0000","last_active":"2024-09-16T15:47:49.926283+0000","last_peered":"2024-09-16T15:47:49.926283+0000","last_clean":"2024-09-16T15:47:49.926283+0000","last_became_active":"2024-09-16T15:47:48.949573+0000","last_became_peered":"2024-09-16T15:47:48.949573+0000","last_unstale":"2024-09-16T15:47:49.926283+0000","last_undegraded":"2024-09-16T15:47:49.926283+0000","last_fullsized":"2024-09-16T15:47:49.926283+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:47:47.896376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:47:47.896376+0000","last_clean_scrub_stamp":"2024-09-16T15:47:47.896376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T02:24:11.311434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.9","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-16T15:47:49.961757+0000","last_change":"2024-09-16T15:47:49.957145+0000","last_active":"2024-09-16T15:47:49.961757+0000","last_peered":"2024-09-16T15:47:49.961757+0000","last_clean":"2024-09-16T15:47:49.961757+0000","last_became_active":"2024-09-16T15:47:49.956767+0000","last_became_peered":"2024-09-16T15:47:49.956767+0000","last_unstale":"2024-09-16T15:47:49.961757+0000","last_undegraded":"2024-09-16T15:47:49.961757+0000","last_fullsized":"2024-09-16T15:47:49.961757+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:47:48.900296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:47:48.900296+0000","last_clean_scrub_stamp":"2024-09-16T15:47:48.900296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T01:43:48.754810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.8","version":"0'0","reported_seq":13,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-16T15:47:49.918403+0000","last_change":"2024-09-16T15:47:48.934536+0000","last_active":"2024-09-16T15:47:49.918403+0000","last_peered":"2024-09-16T15:47:49.918403+0000","last_clean":"2024-09-16T15:47:49.918403+0000","last_became_active":"2024-09-16T15:47:48.933921+0000","last_became_peered":"2024-09-16T15:47:48.933921+0000","last_unstale":"2024-09-16T15:47:49.918403+0000","last_undegraded":"2024-09-16T15:47:49.918403+0000","last_fullsized":"2024-09-16T15:47:49.918403+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:47:47.896376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:47:47.896376+0000","last_clean_scrub_stamp":"2024-09-16T15:47:47.896376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T00:36:28.684545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.6","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-16T15:47:49.951537+0000","last_change":"2024-09-16T15:47:49.946212+0000","last_active":"2024-09-16T15:47:49.951537+0000","last_peered":"2024-09-16T15:47:49.951537+0000","last_clean":"2024-09-16T15:47:49.951537+0000","last_became_active":"2024-09-16T15:47:49.945916+0000","last_became_peered":"2024-09-16T15:47:49.945916+0000","last_unstale":"2024-09-16T15:47:49.951537+0000","last_undegraded":"2024-09-16T15:47:49.951537+0000","last_fullsized":"2024-09-16T15:47:49.951537+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:47:48.900296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:47:48.900296+0000","last_clean_scrub_stamp":"2024-09-16T15:47:48.900296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:06:06.905606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.7","version":"84'2","reported_seq":15,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-16T15:47:49.970300+0000","last_change":"2024-09-16T15:47:48.948961+0000","last_active":"2024-09-16T15:47:49.970300+0000","last_peered":"2024-09-16T15:47:49.970300+0000","last_clean":"2024-09-16T15:47:49.970300+0000","last_became_active":"2024-09-16T15:47:48.948573+0000","last_became_peered":"2024-09-16T15:47:48.948573+0000","last_unstale":"2024-09-16T15:47:49.970300+0000","last_undegraded":"2024-09-16T15:47:49.970300+0000","last_fullsized":"2024-09-16T15:47:49.970300+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:47:47.896376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:47:47.896376+0000","last_clean_scrub_stamp":"2024-09-16T15:47:47.896376+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T23:06:55.771079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.7","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-16T15:47:49.958721+0000","last_change":"2024-09-16T15:47:49.956120+0000","last_active":"2024-09-16T15:47:49.958721+0000","last_peered":"2024-09-16T15:47:49.958721+0000","last_clean":"2024-09-16T15:47:49.958721+0000","last_became_active":"2024-09-16T15:47:49.955651+0000","last_became_peered":"2024-09-16T15:47:49.955651+0000","last_unstale":"2024-09-16T15:47:49.958721+0000","last_undegraded":"2024-09-16T15:47:49.958721+0000","last_fullsized":"2024-09-16T15:47:49.958721+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:47:48.900296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:47:48.900296+0000","last_clean_scrub_stamp":"2024-09-16T15:47:48.900296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:39:18.537968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.6","version":"0'0","reported_seq":13,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-16T15:47:49.936049+0000","last_change":"2024-09-16T15:47:48.933160+0000","last_active":"2024-09-16T15:47:49.936049+0000","last_peered":"2024-09-16T15:47:49.936049+0000","last_clean":"2024-09-16T15:47:49.936049+0000","last_became_active":"2024-09-16T15:47:48.932792+0000","last_became_peered":"2024-09-16T15:47:48.932792+0000","last_unstale":"2024-09-16T15:47:49.936049+0000","last_undegraded":"2024-09-16T15:47:49.936049+0000","last_fullsized":"2024-09-16T15:47:49.936049+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:47:47.896376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:47:47.896376+0000","last_clean_scrub_stamp":"2024-09-16T15:47:47.896376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T01:25:02.263975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.4","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-16T15:47:49.951163+0000","last_change":"2024-09-16T15:47:49.942492+0000","last_active":"2024-09-16T15:47:49.951163+0000","last_peered":"2024-09-16T15:47:49.951163+0000","last_clean":"2024-09-16T15:47:49.951163+0000","last_became_active":"2024-09-16T15:47:49.941976+0000","last_became_peered":"2024-09-16T15:47:49.941976+0000","last_unstale":"2024-09-16T15:47:49.951163+0000","last_undegraded":"2024-09-16T15:47:49.951163+0000","last_fullsized":"2024-09-16T15:47:49.951163+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:47:48.900296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:47:48.900296+0000","last_clean_scrub_stamp":"2024-09-16T15:47:48.900296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:55:00.903877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.5","version":"0'0","reported_seq":13,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-16T15:47:49.930559+0000","last_change":"2024-09-16T15:47:48.949355+0000","last_active":"2024-09-16T15:47:49.930559+0000","last_peered":"2024-09-16T15:47:49.930559+0000","last_clean":"2024-09-16T15:47:49.930559+0000","last_became_active":"2024-09-16T15:47:48.948585+0000","last_became_peered":"2024-09-16T15:47:48.948585+0000","last_unstale":"2024-09-16T15:47:49.930559+0000","last_undegraded":"2024-09-16T15:47:49.930559+0000","last_fullsized":"2024-09-16T15:47:49.930559+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:47:47.896376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:47:47.896376+0000","last_clean_scrub_stamp":"2024-09-16T15:47:47.896376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:49:44.422191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.5","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-16T15:47:49.949784+0000","last_change":"2024-09-16T15:47:49.946789+0000","last_active":"2024-09-16T15:47:49.949784+0000","last_peered":"2024-09-16T15:47:49.949784+0000","last_clean":"2024-09-16T15:47:49.949784+0000","last_became_active":"2024-09-16T15:47:49.945994+0000","last_became_peered":"2024-09-16T15:47:49.945994+0000","last_unstale":"2024-09-16T15:47:49.949784+0000","last_undegraded":"2024-09-16T15:47:49.949784+0000","last_fullsized":"2024-09-16T15:47:49.949784+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:47:48.900296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:47:48.900296+0000","last_clean_scrub_stamp":"2024-09-16T15:47:48.900296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T01:40:00.990103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.4","version":"0'0","reported_seq":13,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-16T15:47:49.940738+0000","last_change":"2024-09-16T15:47:48.942233+0000","last_active":"2024-09-16T15:47:49.940738+0000","last_peered":"2024-09-16T15:47:49.940738+0000","last_clean":"2024-09-16T15:47:49.940738+0000","last_became_active":"2024-09-16T15:47:48.941519+0000","last_became_peered":"2024-09-16T15:47:48.941519+0000","last_unstale":"2024-09-16T15:47:49.940738+0000","last_undegraded":"2024-09-16T15:47:49.940738+0000","last_fullsized":"2024-09-16T15:47:49.940738+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:47:47.896376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:47:47.896376+0000","last_clean_scrub_stamp":"2024-09-16T15:47:47.896376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:23:16.080216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.2","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-16T15:47:49.972948+0000","last_change":"2024-09-16T15:47:49.967401+0000","last_active":"2024-09-16T15:47:49.972948+0000","last_peered":"2024-09-16T15:47:49.972948+0000","last_clean":"2024-09-16T15:47:49.972948+0000","last_became_active":"2024-09-16T15:47:49.967107+0000","last_became_peered":"2024-09-16T15:47:49.967107+0000","last_unstale":"2024-09-16T15:47:49.972948+0000","last_undegraded":"2024-09-16T15:47:49.972948+0000","last_fullsized":"2024-09-16T15:47:49.972948+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:47:48.900296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:47:48.900296+0000","last_clean_scrub_stamp":"2024-09-16T15:47:48.900296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T22:18:29.564614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.3","version":"0'0","reported_seq":13,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-16T15:47:49.928689+0000","last_change":"2024-09-16T15:47:48.942460+0000","last_active":"2024-09-16T15:47:49.928689+0000","last_peered":"2024-09-16T15:47:49.928689+0000","last_clean":"2024-09-16T15:47:49.928689+0000","last_became_active":"2024-09-16T15:47:48.942168+0000","last_became_peered":"2024-09-16T15:47:48.942168+0000","last_unstale":"2024-09-16T15:47:49.928689+0000","last_undegraded":"2024-09-16T15:47:49.928689+0000","last_fullsized":"2024-09-16T15:47:49.928689+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:47:47.896376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:47:47.896376+0000","last_clean_scrub_stamp":"2024-09-16T15:47:47.896376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T22:18:28.560694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.3","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-16T15:47:49.955493+0000","last_change":"2024-09-16T15:47:49.945626+0000","last_active":"2024-09-16T15:47:49.955493+0000","last_peered":"2024-09-16T15:47:49.955493+0000","last_clean":"2024-09-16T15:47:49.955493+0000","last_became_active":"2024-09-16T15:47:49.944910+0000","last_became_peered":"2024-09-16T15:47:49.944910+0000","last_unstale":"2024-09-16T15:47:49.955493+0000","last_undegraded":"2024-09-16T15:47:49.955493+0000","last_fullsized":"2024-09-16T15:47:49.955493+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:47:48.900296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:47:48.900296+0000","last_clean_scrub_stamp":"2024-09-16T15:47:48.900296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T00:55:24.956739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.2","version":"0'0","reported_seq":13,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-16T15:47:49.923417+0000","last_change":"2024-09-16T15:47:48.939060+0000","last_active":"2024-09-16T15:47:49.923417+0000","last_peered":"2024-09-16T15:47:49.923417+0000","last_clean":"2024-09-16T15:47:49.923417+0000","last_became_active":"2024-09-16T15:47:48.938258+0000","last_became_peered":"2024-09-16T15:47:48.938258+0000","last_unstale":"2024-09-16T15:47:49.923417+0000","last_undegraded":"2024-09-16T15:47:49.923417+0000","last_fullsized":"2024-09-16T15:47:49.923417+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:47:47.896376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:47:47.896376+0000","last_clean_scrub_stamp":"2024-09-16T15:47:47.896376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T18:51:35.155740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.0","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-16T15:47:49.964094+0000","last_change":"2024-09-16T15:47:49.958222+0000","last_active":"2024-09-16T15:47:49.964094+0000","last_peered":"2024-09-16T15:47:49.964094+0000","last_clean":"2024-09-16T15:47:49.964094+0000","last_became_active":"2024-09-16T15:47:49.957921+0000","last_became_peered":"2024-09-16T15:47:49.957921+0000","last_unstale":"2024-09-16T15:47:49.964094+0000","last_undegraded":"2024-09-16T15:47:49.964094+0000","last_fullsized":"2024-09-16T15:47:49.964094+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:47:48.900296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:47:48.900296+0000","last_clean_scrub_stamp":"2024-09-16T15:47:48.900296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:50:05.184566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1","version":"0'0","reported_seq":13,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-16T15:47:49.932546+0000","last_change":"2024-09-16T15:47:48.944931+0000","last_active":"2024-09-16T15:47:49.932546+0000","last_peered":"2024-09-16T15:47:49.932546+0000","last_clean":"2024-09-16T15:47:49.932546+0000","last_became_active":"2024-09-16T15:47:48.944168+0000","last_became_peered":"2024-09-16T15:47:48.944168+0000","last_unstale":"2024-09-16T15:47:49.932546+0000","last_undegraded":"2024-09-16T15:47:49.932546+0000","last_fullsized":"2024-09-16T15:47:49.932546+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:47:47.896376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:47:47.896376+0000","last_clean_scrub_stamp":"2024-09-16T15:47:47.896376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T01:13:31.262332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-16T15:47:49.964256+0000","last_change":"2024-09-16T15:47:49.951521+0000","last_active":"2024-09-16T15:47:49.964256+0000","last_peered":"2024-09-16T15:47:49.964256+0000","last_clean":"2024-09-16T15:47:49.964256+0000","last_became_active":"2024-09-16T15:47:49.950245+0000","last_became_peered":"2024-09-16T15:47:49.950245+0000","last_unstale":"2024-09-16T15:47:49.964256+0000","last_undegraded":"2024-09-16T15:47:49.964256+0000","last_fullsized":"2024-09-16T15:47:49.964256+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:47:48.900296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:47:48.900296+0000","last_clean_scrub_stamp":"2024-09-16T15:47:48.900296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:25:16.118135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.0","version":"0'0","reported_seq":13,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-16T15:47:49.924289+0000","last_change":"2024-09-16T15:47:48.951057+0000","last_active":"2024-09-16T15:47:49.924289+0000","last_peered":"2024-09-16T15:47:49.924289+0000","last_clean":"2024-09-16T15:47:49.924289+0000","last_became_active":"2024-09-16T15:47:48.950243+0000","last_became_peered":"2024-09-16T15:47:48.950243+0000","last_unstale":"2024-09-16T15:47:49.924289+0000","last_undegraded":"2024-09-16T15:47:49.924289+0000","last_fullsized":"2024-09-16T15:47:49.924289+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:47:47.896376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:47:47.896376+0000","last_clean_scrub_stamp":"2024-09-16T15:47:47.896376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:23:16.080216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.c","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-16T15:47:49.948898+0000","last_change":"2024-09-16T15:47:49.946744+0000","last_active":"2024-09-16T15:47:49.948898+0000","last_peered":"2024-09-16T15:47:49.948898+0000","last_clean":"2024-09-16T15:47:49.948898+0000","last_became_active":"2024-09-16T15:47:49.946341+0000","last_became_peered":"2024-09-16T15:47:49.946341+0000","last_unstale":"2024-09-16T15:47:49.948898+0000","last_undegraded":"2024-09-16T15:47:49.948898+0000","last_fullsized":"2024-09-16T15:47:49.948898+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:47:48.900296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:47:48.900296+0000","last_clean_scrub_stamp":"2024-09-16T15:47:48.900296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T03:44:33.687212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.d","version":"84'1","reported_seq":14,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-16T15:47:49.969893+0000","last_change":"2024-09-16T15:47:48.955307+0000","last_active":"2024-09-16T15:47:49.969893+0000","last_peered":"2024-09-16T15:47:49.969893+0000","last_clean":"2024-09-16T15:47:49.969893+0000","last_became_active":"2024-09-16T15:47:48.954591+0000","last_became_peered":"2024-09-16T15:47:48.954591+0000","last_unstale":"2024-09-16T15:47:49.969893+0000","last_undegraded":"2024-09-16T15:47:49.969893+0000","last_fullsized":"2024-09-16T15:47:49.969893+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:47:47.896376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:47:47.896376+0000","last_clean_scrub_stamp":"2024-09-16T15:47:47.896376+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:52:27.701367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"73'79","reported_seq":256,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-16T15:47:49.934224+0000","last_change":"2024-09-16T15:42:40.881820+0000","last_active":"2024-09-16T15:47:49.934224+0000","last_peered":"2024-09-16T15:47:49.934224+0000","last_clean":"2024-09-16T15:47:49.934224+0000","last_became_active":"2024-09-16T15:42:40.879965+0000","last_became_peered":"2024-09-16T15:42:40.879965+0000","last_unstale":"2024-09-16T15:47:49.934224+0000","last_undegraded":"2024-09-16T15:47:49.934224+0000","last_fullsized":"2024-09-16T15:47:49.934224+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:42:39.728762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:42:39.728762+0000","last_clean_scrub_stamp":"2024-09-16T15:42:39.728762+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:22:18.768932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":13,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":12,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79,"ondisk_log_size":79,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542210,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32760,"kb_used_data":1572,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339080,"statfs":{"total":96636764160,"available":96603217920,"internally_reserved":0,"allocated":1609728,"data_stored":1024387,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542209,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32184,"kb_used_data":992,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339656,"statfs":{"total":96636764160,"available":96603807744,"internally_reserved":0,"allocated":1015808,"data_stored":434019,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":5,"up_from":14,"seq":60129542210,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28024,"kb_used_data":992,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343816,"statfs":{"total":96636764160,"available":96608067584,"internally_reserved":0,"allocated":1015808,"data_stored":434019,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542208,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32180,"kb_used_data":992,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339660,"statfs":{"total":96636764160,"available":96603811840,"internally_reserved":0,"allocated":1015808,"data_stored":434019,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":3,"up_from":14,"seq":60129542209,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32180,"kb_used_data":992,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339660,"statfs":{"total":96636764160,"available":96603811840,"internally_reserved":0,"allocated":1015808,"data_stored":434019,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":2,"up_from":14,"seq":60129542209,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32184,"kb_used_data":992,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339656,"statfs":{"total":96636764160,"available":96603807744,"internally_reserved":0,"allocated":1015808,"data_stored":434019,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":1,"up_from":14,"seq":60129542211,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28024,"kb_used_data":992,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343816,"statfs":{"total":96636764160,"available":96608067584,"internally_reserved":0,"allocated":1015808,"data_stored":434019,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":0,"up_from":14,"seq":60129542211,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32760,"kb_used_data":1572,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339080,"statfs":{"total":96636764160,"available":96603217920,"internally_reserved":0,"allocated":1609728,"data_stored":1024387,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":194,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T15:47:53.998 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T15:47:54.159 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:54.155+0000 7efc1dbbb640 1 -- 172.21.15.123:0/3454731026 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7efc1815e820 msgr2=0x7efc1815ec90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:47:54.159 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:54.155+0000 7efc1dbbb640 1 --2- 172.21.15.123:0/3454731026 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7efc1815e820 0x7efc1815ec90 secure :-1 s=READY pgs=947 cs=0 l=1 rev1=1 crypto rx=0x7efc18070730 tx=0x7efc1001c7a0 comp rx=0 tx=0).stop 2024-09-16T15:47:54.159 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:54.155+0000 7efc1dbbb640 1 -- 172.21.15.123:0/3454731026 shutdown_connections 2024-09-16T15:47:54.159 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:54.155+0000 7efc1dbbb640 1 --2- 172.21.15.123:0/3454731026 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7efc1815e820 0x7efc1815ec90 unknown :-1 s=CLOSED pgs=947 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:54.159 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:54.155+0000 7efc1dbbb640 1 --2- 172.21.15.123:0/3454731026 >> v2:172.21.15.184:3300/0 conn(0x7efc1815e160 0x7efc18073f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:54.159 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:54.155+0000 7efc1dbbb640 1 -- 172.21.15.123:0/3454731026 >> 172.21.15.123:0/3454731026 conn(0x7efc180cfb10 msgr2=0x7efc180cff20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:47:54.159 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:54.155+0000 7efc1dbbb640 1 -- 172.21.15.123:0/3454731026 shutdown_connections 2024-09-16T15:47:54.159 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:54.155+0000 7efc1dbbb640 1 -- 172.21.15.123:0/3454731026 wait complete. 2024-09-16T15:47:54.160 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:54.155+0000 7efc1dbbb640 1 Processor -- start 2024-09-16T15:47:54.160 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:54.155+0000 7efc1dbbb640 1 -- start start 2024-09-16T15:47:54.162 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:54.155+0000 7efc1dbbb640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7efc18074470 0x7efc1806f420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:47:54.162 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:54.155+0000 7efc1dbbb640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7efc1815e160 0x7efc1815a830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:47:54.162 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:54.155+0000 7efc1dbbb640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7efc1815e820 0x7efc18156890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:47:54.162 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:54.155+0000 7efc1dbbb640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7efc180cf5b0 con 0x7efc1815e820 2024-09-16T15:47:54.162 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:54.155+0000 7efc1dbbb640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7efc180cf430 con 0x7efc18074470 2024-09-16T15:47:54.162 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:54.155+0000 7efc1dbbb640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7efc181683d0 con 0x7efc1815e160 2024-09-16T15:47:54.162 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:54.155+0000 7efc17fff640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7efc1815e820 0x7efc18156890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:47:54.162 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:54.155+0000 7efc17fff640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7efc1815e820 0x7efc18156890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:43838/0 (socket says 172.21.15.123:43838) 2024-09-16T15:47:54.162 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:54.155+0000 7efc17fff640 1 -- 172.21.15.123:0/116542713 learned_addr learned my addr 172.21.15.123:0/116542713 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:47:54.163 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:54.155+0000 7efc177fe640 1 --2- 172.21.15.123:0/116542713 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7efc18074470 0x7efc1806f420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:47:54.163 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:54.155+0000 7efc17fff640 1 -- 172.21.15.123:0/116542713 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7efc1815e160 msgr2=0x7efc1815a830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:47:54.163 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:54.155+0000 7efc17fff640 1 --2- 172.21.15.123:0/116542713 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7efc1815e160 0x7efc1815a830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:54.163 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:54.155+0000 7efc17fff640 1 -- 172.21.15.123:0/116542713 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7efc18074470 msgr2=0x7efc1806f420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:47:54.163 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:54.155+0000 7efc17fff640 1 --2- 172.21.15.123:0/116542713 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7efc18074470 0x7efc1806f420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:54.163 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:54.155+0000 7efc17fff640 1 -- 172.21.15.123:0/116542713 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efc181f9ab0 con 0x7efc1815e820 2024-09-16T15:47:54.163 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:54.155+0000 7efc177fe640 1 --2- 172.21.15.123:0/116542713 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7efc18074470 0x7efc1806f420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T15:47:54.163 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:54.155+0000 7efc17fff640 1 --2- 172.21.15.123:0/116542713 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7efc1815e820 0x7efc18156890 secure :-1 s=READY pgs=948 cs=0 l=1 rev1=1 crypto rx=0x7efc10009850 tx=0x7efc100a5690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:47:54.163 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:54.155+0000 7efbfffff640 1 -- 172.21.15.123:0/116542713 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc100b2040 con 0x7efc1815e820 2024-09-16T15:47:54.163 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:54.155+0000 7efbfffff640 1 -- 172.21.15.123:0/116542713 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7efc10004230 con 0x7efc1815e820 2024-09-16T15:47:54.163 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:54.155+0000 7efc1dbbb640 1 -- 172.21.15.123:0/116542713 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efc18155db0 con 0x7efc1815e820 2024-09-16T15:47:54.163 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:54.155+0000 7efbfffff640 1 -- 172.21.15.123:0/116542713 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc10003600 con 0x7efc1815e820 2024-09-16T15:47:54.163 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:54.155+0000 7efc1dbbb640 1 -- 172.21.15.123:0/116542713 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efc1806f960 con 0x7efc1815e820 2024-09-16T15:47:54.164 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:54.159+0000 7efc1dbbb640 1 -- 172.21.15.123:0/116542713 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efc18073f30 con 0x7efc1815e820 2024-09-16T15:47:54.164 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:54.159+0000 7efbfffff640 1 -- 172.21.15.123:0/116542713 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 16) ==== 97182+0+0 (secure 0 0 0) 0x7efc100bd020 con 0x7efc1815e820 2024-09-16T15:47:54.164 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:54.159+0000 7efbfffff640 1 --2- 172.21.15.123:0/116542713 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7efc04072fe0 0x7efc040754a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:47:54.164 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:54.159+0000 7efc177fe640 1 --2- 172.21.15.123:0/116542713 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7efc04072fe0 0x7efc040754a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:47:54.165 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:54.159+0000 7efbfffff640 1 -- 172.21.15.123:0/116542713 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 5709+0+0 (secure 0 0 0) 0x7efc1012dd10 con 0x7efc1815e820 2024-09-16T15:47:54.165 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:54.159+0000 7efc177fe640 1 --2- 172.21.15.123:0/116542713 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7efc04072fe0 0x7efc040754a0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7efc0c00a8b0 tx=0x7efc0c008040 comp rx=0 tx=0).ready entity=mgr.4150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:47:54.167 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:54.163+0000 7efbfffff640 1 -- 172.21.15.123:0/116542713 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7efc100c4020 con 0x7efc1815e820 2024-09-16T15:47:54.321 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:54.315+0000 7efc1dbbb640 1 -- 172.21.15.123:0/116542713 --> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7efc180d06c0 con 0x7efc04072fe0 2024-09-16T15:47:54.332 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:54.327+0000 7efbfffff640 1 -- 172.21.15.123:0/116542713 <== mgr.4150 v2:172.21.15.123:6836/2464759676 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+252068 (secure 0 0 0) 0x7efc180d06c0 con 0x7efc04072fe0 2024-09-16T15:47:54.335 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-09-16T15:47:54.336 INFO:teuthology.orchestra.run.smithi123.stderr:dumped all 2024-09-16T15:47:54.337 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:54.331+0000 7efc1dbbb640 1 -- 172.21.15.123:0/116542713 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7efc04072fe0 msgr2=0x7efc040754a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:47:54.338 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:54.331+0000 7efc1dbbb640 1 --2- 172.21.15.123:0/116542713 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7efc04072fe0 0x7efc040754a0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7efc0c00a8b0 tx=0x7efc0c008040 comp rx=0 tx=0).stop 2024-09-16T15:47:54.338 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:54.331+0000 7efc1dbbb640 1 -- 172.21.15.123:0/116542713 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7efc1815e820 msgr2=0x7efc18156890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:47:54.338 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:54.331+0000 7efc1dbbb640 1 --2- 172.21.15.123:0/116542713 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7efc1815e820 0x7efc18156890 secure :-1 s=READY pgs=948 cs=0 l=1 rev1=1 crypto rx=0x7efc10009850 tx=0x7efc100a5690 comp rx=0 tx=0).stop 2024-09-16T15:47:54.338 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:54.331+0000 7efc1dbbb640 1 -- 172.21.15.123:0/116542713 shutdown_connections 2024-09-16T15:47:54.338 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:54.331+0000 7efc1dbbb640 1 --2- 172.21.15.123:0/116542713 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7efc04072fe0 0x7efc040754a0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:54.338 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:54.331+0000 7efc1dbbb640 1 --2- 172.21.15.123:0/116542713 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7efc1815e820 0x7efc18156890 unknown :-1 s=CLOSED pgs=948 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:54.338 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:54.331+0000 7efc1dbbb640 1 --2- 172.21.15.123:0/116542713 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7efc1815e160 0x7efc1815a830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:54.338 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:54.331+0000 7efc1dbbb640 1 --2- 172.21.15.123:0/116542713 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7efc18074470 0x7efc1806f420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:54.338 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:54.331+0000 7efc1dbbb640 1 -- 172.21.15.123:0/116542713 >> 172.21.15.123:0/116542713 conn(0x7efc180cfb10 msgr2=0x7efc180af8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:47:54.338 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:54.331+0000 7efc1dbbb640 1 -- 172.21.15.123:0/116542713 shutdown_connections 2024-09-16T15:47:54.338 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:54.331+0000 7efc1dbbb640 1 -- 172.21.15.123:0/116542713 wait complete. 2024-09-16T15:47:54.351 INFO:teuthology.orchestra.run.smithi123.stdout:{"pg_ready":true,"pg_map":{"version":26,"stamp":"2024-09-16T15:47:54.071892+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":157,"num_write_kb":1642,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":72,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":250296,"kb_used_data":9096,"kb_used_omap":12,"kb_used_meta":241011,"kb_avail":754724424,"statfs":{"total":773094113280,"available":772837810176,"internally_reserved":0,"allocated":9314304,"data_stored":4652888,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":246795825},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":38,"apply_latency_ms":38,"commit_latency_ns":38000000,"apply_latency_ns":38000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001288"},"pg_stats":[{"pgid":"13.3f","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-16T15:47:49.965606+0000","last_change":"2024-09-16T15:47:49.959302+0000","last_active":"2024-09-16T15:47:49.965606+0000","last_peered":"2024-09-16T15:47:49.965606+0000","last_clean":"2024-09-16T15:47:49.965606+0000","last_became_active":"2024-09-16T15:47:49.954739+0000","last_became_peered":"2024-09-16T15:47:49.954739+0000","last_unstale":"2024-09-16T15:47:49.965606+0000","last_undegraded":"2024-09-16T15:47:49.965606+0000","last_fullsized":"2024-09-16T15:47:49.965606+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:47:48.900296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:47:48.900296+0000","last_clean_scrub_stamp":"2024-09-16T15:47:48.900296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T23:26:28.735670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.3e","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-16T15:47:49.963868+0000","last_change":"2024-09-16T15:47:49.957730+0000","last_active":"2024-09-16T15:47:49.963868+0000","last_peered":"2024-09-16T15:47:49.963868+0000","last_clean":"2024-09-16T15:47:49.963868+0000","last_became_active":"2024-09-16T15:47:49.957226+0000","last_became_peered":"2024-09-16T15:47:49.957226+0000","last_unstale":"2024-09-16T15:47:49.963868+0000","last_undegraded":"2024-09-16T15:47:49.963868+0000","last_fullsized":"2024-09-16T15:47:49.963868+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:47:48.900296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:47:48.900296+0000","last_clean_scrub_stamp":"2024-09-16T15:47:48.900296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:51:36.683460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.3d","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-16T15:47:49.952793+0000","last_change":"2024-09-16T15:47:49.950700+0000","last_active":"2024-09-16T15:47:49.952793+0000","last_peered":"2024-09-16T15:47:49.952793+0000","last_clean":"2024-09-16T15:47:49.952793+0000","last_became_active":"2024-09-16T15:47:49.950374+0000","last_became_peered":"2024-09-16T15:47:49.950374+0000","last_unstale":"2024-09-16T15:47:49.952793+0000","last_undegraded":"2024-09-16T15:47:49.952793+0000","last_fullsized":"2024-09-16T15:47:49.952793+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:47:48.900296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:47:48.900296+0000","last_clean_scrub_stamp":"2024-09-16T15:47:48.900296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:18:30.188685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.3c","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-16T15:47:49.961221+0000","last_change":"2024-09-16T15:47:49.957584+0000","last_active":"2024-09-16T15:47:49.961221+0000","last_peered":"2024-09-16T15:47:49.961221+0000","last_clean":"2024-09-16T15:47:49.961221+0000","last_became_active":"2024-09-16T15:47:49.957263+0000","last_became_peered":"2024-09-16T15:47:49.957263+0000","last_unstale":"2024-09-16T15:47:49.961221+0000","last_undegraded":"2024-09-16T15:47:49.961221+0000","last_fullsized":"2024-09-16T15:47:49.961221+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:47:48.900296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:47:48.900296+0000","last_clean_scrub_stamp":"2024-09-16T15:47:48.900296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T18:09:41.664969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.3b","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-16T15:47:49.947011+0000","last_change":"2024-09-16T15:47:49.945731+0000","last_active":"2024-09-16T15:47:49.947011+0000","last_peered":"2024-09-16T15:47:49.947011+0000","last_clean":"2024-09-16T15:47:49.947011+0000","last_became_active":"2024-09-16T15:47:49.945418+0000","last_became_peered":"2024-09-16T15:47:49.945418+0000","last_unstale":"2024-09-16T15:47:49.947011+0000","last_undegraded":"2024-09-16T15:47:49.947011+0000","last_fullsized":"2024-09-16T15:47:49.947011+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:47:48.900296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:47:48.900296+0000","last_clean_scrub_stamp":"2024-09-16T15:47:48.900296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:25:16.118135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.3a","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-16T15:47:49.967679+0000","last_change":"2024-09-16T15:47:49.965103+0000","last_active":"2024-09-16T15:47:49.967679+0000","last_peered":"2024-09-16T15:47:49.967679+0000","last_clean":"2024-09-16T15:47:49.967679+0000","last_became_active":"2024-09-16T15:47:49.964784+0000","last_became_peered":"2024-09-16T15:47:49.964784+0000","last_unstale":"2024-09-16T15:47:49.967679+0000","last_undegraded":"2024-09-16T15:47:49.967679+0000","last_fullsized":"2024-09-16T15:47:49.967679+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:47:48.900296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:47:48.900296+0000","last_clean_scrub_stamp":"2024-09-16T15:47:48.900296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:19:44.543517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.39","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-16T15:47:49.968650+0000","last_change":"2024-09-16T15:47:49.960684+0000","last_active":"2024-09-16T15:47:49.968650+0000","last_peered":"2024-09-16T15:47:49.968650+0000","last_clean":"2024-09-16T15:47:49.968650+0000","last_became_active":"2024-09-16T15:47:49.960294+0000","last_became_peered":"2024-09-16T15:47:49.960294+0000","last_unstale":"2024-09-16T15:47:49.968650+0000","last_undegraded":"2024-09-16T15:47:49.968650+0000","last_fullsized":"2024-09-16T15:47:49.968650+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:47:48.900296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:47:48.900296+0000","last_clean_scrub_stamp":"2024-09-16T15:47:48.900296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T03:47:33.379740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.38","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-16T15:47:49.964424+0000","last_change":"2024-09-16T15:47:49.961962+0000","last_active":"2024-09-16T15:47:49.964424+0000","last_peered":"2024-09-16T15:47:49.964424+0000","last_clean":"2024-09-16T15:47:49.964424+0000","last_became_active":"2024-09-16T15:47:49.961632+0000","last_became_peered":"2024-09-16T15:47:49.961632+0000","last_unstale":"2024-09-16T15:47:49.964424+0000","last_undegraded":"2024-09-16T15:47:49.964424+0000","last_fullsized":"2024-09-16T15:47:49.964424+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:47:48.900296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:47:48.900296+0000","last_clean_scrub_stamp":"2024-09-16T15:47:48.900296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:32:37.242027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.37","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-16T15:47:49.961797+0000","last_change":"2024-09-16T15:47:49.957050+0000","last_active":"2024-09-16T15:47:49.961797+0000","last_peered":"2024-09-16T15:47:49.961797+0000","last_clean":"2024-09-16T15:47:49.961797+0000","last_became_active":"2024-09-16T15:47:49.956699+0000","last_became_peered":"2024-09-16T15:47:49.956699+0000","last_unstale":"2024-09-16T15:47:49.961797+0000","last_undegraded":"2024-09-16T15:47:49.961797+0000","last_fullsized":"2024-09-16T15:47:49.961797+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:47:48.900296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:47:48.900296+0000","last_clean_scrub_stamp":"2024-09-16T15:47:48.900296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:33:07.023978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.36","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-16T15:47:49.965630+0000","last_change":"2024-09-16T15:47:49.959730+0000","last_active":"2024-09-16T15:47:49.965630+0000","last_peered":"2024-09-16T15:47:49.965630+0000","last_clean":"2024-09-16T15:47:49.965630+0000","last_became_active":"2024-09-16T15:47:49.959395+0000","last_became_peered":"2024-09-16T15:47:49.959395+0000","last_unstale":"2024-09-16T15:47:49.965630+0000","last_undegraded":"2024-09-16T15:47:49.965630+0000","last_fullsized":"2024-09-16T15:47:49.965630+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:47:48.900296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:47:48.900296+0000","last_clean_scrub_stamp":"2024-09-16T15:47:48.900296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T02:14:30.434884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.35","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-16T15:47:49.953682+0000","last_change":"2024-09-16T15:47:49.950477+0000","last_active":"2024-09-16T15:47:49.953682+0000","last_peered":"2024-09-16T15:47:49.953682+0000","last_clean":"2024-09-16T15:47:49.953682+0000","last_became_active":"2024-09-16T15:47:49.950140+0000","last_became_peered":"2024-09-16T15:47:49.950140+0000","last_unstale":"2024-09-16T15:47:49.953682+0000","last_undegraded":"2024-09-16T15:47:49.953682+0000","last_fullsized":"2024-09-16T15:47:49.953682+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:47:48.900296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:47:48.900296+0000","last_clean_scrub_stamp":"2024-09-16T15:47:48.900296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T00:36:31.093928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.34","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-16T15:47:49.965514+0000","last_change":"2024-09-16T15:47:49.955834+0000","last_active":"2024-09-16T15:47:49.965514+0000","last_peered":"2024-09-16T15:47:49.965514+0000","last_clean":"2024-09-16T15:47:49.965514+0000","last_became_active":"2024-09-16T15:47:49.955446+0000","last_became_peered":"2024-09-16T15:47:49.955446+0000","last_unstale":"2024-09-16T15:47:49.965514+0000","last_undegraded":"2024-09-16T15:47:49.965514+0000","last_fullsized":"2024-09-16T15:47:49.965514+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:47:48.900296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:47:48.900296+0000","last_clean_scrub_stamp":"2024-09-16T15:47:48.900296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T18:59:41.944452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.33","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-16T15:47:49.949211+0000","last_change":"2024-09-16T15:47:49.946749+0000","last_active":"2024-09-16T15:47:49.949211+0000","last_peered":"2024-09-16T15:47:49.949211+0000","last_clean":"2024-09-16T15:47:49.949211+0000","last_became_active":"2024-09-16T15:47:49.946411+0000","last_became_peered":"2024-09-16T15:47:49.946411+0000","last_unstale":"2024-09-16T15:47:49.949211+0000","last_undegraded":"2024-09-16T15:47:49.949211+0000","last_fullsized":"2024-09-16T15:47:49.949211+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:47:48.900296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:47:48.900296+0000","last_clean_scrub_stamp":"2024-09-16T15:47:48.900296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:37:13.529669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.32","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-16T15:47:49.968708+0000","last_change":"2024-09-16T15:47:49.962199+0000","last_active":"2024-09-16T15:47:49.968708+0000","last_peered":"2024-09-16T15:47:49.968708+0000","last_clean":"2024-09-16T15:47:49.968708+0000","last_became_active":"2024-09-16T15:47:49.959206+0000","last_became_peered":"2024-09-16T15:47:49.959206+0000","last_unstale":"2024-09-16T15:47:49.968708+0000","last_undegraded":"2024-09-16T15:47:49.968708+0000","last_fullsized":"2024-09-16T15:47:49.968708+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:47:48.900296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:47:48.900296+0000","last_clean_scrub_stamp":"2024-09-16T15:47:48.900296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T22:33:10.685584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.31","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-16T15:47:49.954403+0000","last_change":"2024-09-16T15:47:49.943665+0000","last_active":"2024-09-16T15:47:49.954403+0000","last_peered":"2024-09-16T15:47:49.954403+0000","last_clean":"2024-09-16T15:47:49.954403+0000","last_became_active":"2024-09-16T15:47:49.942996+0000","last_became_peered":"2024-09-16T15:47:49.942996+0000","last_unstale":"2024-09-16T15:47:49.954403+0000","last_undegraded":"2024-09-16T15:47:49.954403+0000","last_fullsized":"2024-09-16T15:47:49.954403+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:47:48.900296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:47:48.900296+0000","last_clean_scrub_stamp":"2024-09-16T15:47:48.900296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:21:09.352580+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.30","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-16T15:47:49.953750+0000","last_change":"2024-09-16T15:47:49.950406+0000","last_active":"2024-09-16T15:47:49.953750+0000","last_peered":"2024-09-16T15:47:49.953750+0000","last_clean":"2024-09-16T15:47:49.953750+0000","last_became_active":"2024-09-16T15:47:49.950033+0000","last_became_peered":"2024-09-16T15:47:49.950033+0000","last_unstale":"2024-09-16T15:47:49.953750+0000","last_undegraded":"2024-09-16T15:47:49.953750+0000","last_fullsized":"2024-09-16T15:47:49.953750+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:47:48.900296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:47:48.900296+0000","last_clean_scrub_stamp":"2024-09-16T15:47:48.900296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T23:26:28.735670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.2f","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-16T15:47:49.942331+0000","last_change":"2024-09-16T15:47:49.940861+0000","last_active":"2024-09-16T15:47:49.942331+0000","last_peered":"2024-09-16T15:47:49.942331+0000","last_clean":"2024-09-16T15:47:49.942331+0000","last_became_active":"2024-09-16T15:47:49.940455+0000","last_became_peered":"2024-09-16T15:47:49.940455+0000","last_unstale":"2024-09-16T15:47:49.942331+0000","last_undegraded":"2024-09-16T15:47:49.942331+0000","last_fullsized":"2024-09-16T15:47:49.942331+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:47:48.900296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:47:48.900296+0000","last_clean_scrub_stamp":"2024-09-16T15:47:48.900296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T03:44:37.047484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.2e","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-16T15:47:49.968762+0000","last_change":"2024-09-16T15:47:49.963895+0000","last_active":"2024-09-16T15:47:49.968762+0000","last_peered":"2024-09-16T15:47:49.968762+0000","last_clean":"2024-09-16T15:47:49.968762+0000","last_became_active":"2024-09-16T15:47:49.963241+0000","last_became_peered":"2024-09-16T15:47:49.963241+0000","last_unstale":"2024-09-16T15:47:49.968762+0000","last_undegraded":"2024-09-16T15:47:49.968762+0000","last_fullsized":"2024-09-16T15:47:49.968762+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:47:48.900296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:47:48.900296+0000","last_clean_scrub_stamp":"2024-09-16T15:47:48.900296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:04:49.834873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.2d","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-16T15:47:49.964268+0000","last_change":"2024-09-16T15:47:49.958744+0000","last_active":"2024-09-16T15:47:49.964268+0000","last_peered":"2024-09-16T15:47:49.964268+0000","last_clean":"2024-09-16T15:47:49.964268+0000","last_became_active":"2024-09-16T15:47:49.956668+0000","last_became_peered":"2024-09-16T15:47:49.956668+0000","last_unstale":"2024-09-16T15:47:49.964268+0000","last_undegraded":"2024-09-16T15:47:49.964268+0000","last_fullsized":"2024-09-16T15:47:49.964268+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:47:48.900296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:47:48.900296+0000","last_clean_scrub_stamp":"2024-09-16T15:47:48.900296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:57:25.380284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.2c","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-16T15:47:49.961194+0000","last_change":"2024-09-16T15:47:49.948275+0000","last_active":"2024-09-16T15:47:49.961194+0000","last_peered":"2024-09-16T15:47:49.961194+0000","last_clean":"2024-09-16T15:47:49.961194+0000","last_became_active":"2024-09-16T15:47:49.947417+0000","last_became_peered":"2024-09-16T15:47:49.947417+0000","last_unstale":"2024-09-16T15:47:49.961194+0000","last_undegraded":"2024-09-16T15:47:49.961194+0000","last_fullsized":"2024-09-16T15:47:49.961194+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:47:48.900296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:47:48.900296+0000","last_clean_scrub_stamp":"2024-09-16T15:47:48.900296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:08:17.863899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.2b","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-16T15:47:49.967584+0000","last_change":"2024-09-16T15:47:49.948581+0000","last_active":"2024-09-16T15:47:49.967584+0000","last_peered":"2024-09-16T15:47:49.967584+0000","last_clean":"2024-09-16T15:47:49.967584+0000","last_became_active":"2024-09-16T15:47:49.947689+0000","last_became_peered":"2024-09-16T15:47:49.947689+0000","last_unstale":"2024-09-16T15:47:49.967584+0000","last_undegraded":"2024-09-16T15:47:49.967584+0000","last_fullsized":"2024-09-16T15:47:49.967584+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:47:48.900296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:47:48.900296+0000","last_clean_scrub_stamp":"2024-09-16T15:47:48.900296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:43:31.738244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.2a","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-16T15:47:49.964508+0000","last_change":"2024-09-16T15:47:49.958769+0000","last_active":"2024-09-16T15:47:49.964508+0000","last_peered":"2024-09-16T15:47:49.964508+0000","last_clean":"2024-09-16T15:47:49.964508+0000","last_became_active":"2024-09-16T15:47:49.956923+0000","last_became_peered":"2024-09-16T15:47:49.956923+0000","last_unstale":"2024-09-16T15:47:49.964508+0000","last_undegraded":"2024-09-16T15:47:49.964508+0000","last_fullsized":"2024-09-16T15:47:49.964508+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:47:48.900296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:47:48.900296+0000","last_clean_scrub_stamp":"2024-09-16T15:47:48.900296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T03:37:28.810840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.29","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-16T15:47:49.965349+0000","last_change":"2024-09-16T15:47:49.953710+0000","last_active":"2024-09-16T15:47:49.965349+0000","last_peered":"2024-09-16T15:47:49.965349+0000","last_clean":"2024-09-16T15:47:49.965349+0000","last_became_active":"2024-09-16T15:47:49.950771+0000","last_became_peered":"2024-09-16T15:47:49.950771+0000","last_unstale":"2024-09-16T15:47:49.965349+0000","last_undegraded":"2024-09-16T15:47:49.965349+0000","last_fullsized":"2024-09-16T15:47:49.965349+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:47:48.900296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:47:48.900296+0000","last_clean_scrub_stamp":"2024-09-16T15:47:48.900296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:06:06.905606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.28","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-16T15:47:49.968835+0000","last_change":"2024-09-16T15:47:49.963059+0000","last_active":"2024-09-16T15:47:49.968835+0000","last_peered":"2024-09-16T15:47:49.968835+0000","last_clean":"2024-09-16T15:47:49.968835+0000","last_became_active":"2024-09-16T15:47:49.962723+0000","last_became_peered":"2024-09-16T15:47:49.962723+0000","last_unstale":"2024-09-16T15:47:49.968835+0000","last_undegraded":"2024-09-16T15:47:49.968835+0000","last_fullsized":"2024-09-16T15:47:49.968835+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:47:48.900296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:47:48.900296+0000","last_clean_scrub_stamp":"2024-09-16T15:47:48.900296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T00:04:30.122975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.27","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-16T15:47:49.963291+0000","last_change":"2024-09-16T15:47:49.958253+0000","last_active":"2024-09-16T15:47:49.963291+0000","last_peered":"2024-09-16T15:47:49.963291+0000","last_clean":"2024-09-16T15:47:49.963291+0000","last_became_active":"2024-09-16T15:47:49.957662+0000","last_became_peered":"2024-09-16T15:47:49.957662+0000","last_unstale":"2024-09-16T15:47:49.963291+0000","last_undegraded":"2024-09-16T15:47:49.963291+0000","last_fullsized":"2024-09-16T15:47:49.963291+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:47:48.900296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:47:48.900296+0000","last_clean_scrub_stamp":"2024-09-16T15:47:48.900296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:52:28.109252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.26","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-16T15:47:49.953458+0000","last_change":"2024-09-16T15:47:49.952381+0000","last_active":"2024-09-16T15:47:49.953458+0000","last_peered":"2024-09-16T15:47:49.953458+0000","last_clean":"2024-09-16T15:47:49.953458+0000","last_became_active":"2024-09-16T15:47:49.951442+0000","last_became_peered":"2024-09-16T15:47:49.951442+0000","last_unstale":"2024-09-16T15:47:49.953458+0000","last_undegraded":"2024-09-16T15:47:49.953458+0000","last_fullsized":"2024-09-16T15:47:49.953458+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:47:48.900296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:47:48.900296+0000","last_clean_scrub_stamp":"2024-09-16T15:47:48.900296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T02:07:55.051351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.25","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-16T15:47:49.946643+0000","last_change":"2024-09-16T15:47:49.942464+0000","last_active":"2024-09-16T15:47:49.946643+0000","last_peered":"2024-09-16T15:47:49.946643+0000","last_clean":"2024-09-16T15:47:49.946643+0000","last_became_active":"2024-09-16T15:47:49.942105+0000","last_became_peered":"2024-09-16T15:47:49.942105+0000","last_unstale":"2024-09-16T15:47:49.946643+0000","last_undegraded":"2024-09-16T15:47:49.946643+0000","last_fullsized":"2024-09-16T15:47:49.946643+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:47:48.900296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:47:48.900296+0000","last_clean_scrub_stamp":"2024-09-16T15:47:48.900296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T23:45:21.885067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.24","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-16T15:47:49.964220+0000","last_change":"2024-09-16T15:47:49.951311+0000","last_active":"2024-09-16T15:47:49.964220+0000","last_peered":"2024-09-16T15:47:49.964220+0000","last_clean":"2024-09-16T15:47:49.964220+0000","last_became_active":"2024-09-16T15:47:49.950928+0000","last_became_peered":"2024-09-16T15:47:49.950928+0000","last_unstale":"2024-09-16T15:47:49.964220+0000","last_undegraded":"2024-09-16T15:47:49.964220+0000","last_fullsized":"2024-09-16T15:47:49.964220+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:47:48.900296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:47:48.900296+0000","last_clean_scrub_stamp":"2024-09-16T15:47:48.900296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T00:04:45.746226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.23","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-16T15:47:49.956659+0000","last_change":"2024-09-16T15:47:49.955480+0000","last_active":"2024-09-16T15:47:49.956659+0000","last_peered":"2024-09-16T15:47:49.956659+0000","last_clean":"2024-09-16T15:47:49.956659+0000","last_became_active":"2024-09-16T15:47:49.955164+0000","last_became_peered":"2024-09-16T15:47:49.955164+0000","last_unstale":"2024-09-16T15:47:49.956659+0000","last_undegraded":"2024-09-16T15:47:49.956659+0000","last_fullsized":"2024-09-16T15:47:49.956659+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:47:48.900296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:47:48.900296+0000","last_clean_scrub_stamp":"2024-09-16T15:47:48.900296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T02:28:31.665606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.22","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-16T15:47:49.971251+0000","last_change":"2024-09-16T15:47:49.969647+0000","last_active":"2024-09-16T15:47:49.971251+0000","last_peered":"2024-09-16T15:47:49.971251+0000","last_clean":"2024-09-16T15:47:49.971251+0000","last_became_active":"2024-09-16T15:47:49.969364+0000","last_became_peered":"2024-09-16T15:47:49.969364+0000","last_unstale":"2024-09-16T15:47:49.971251+0000","last_undegraded":"2024-09-16T15:47:49.971251+0000","last_fullsized":"2024-09-16T15:47:49.971251+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:47:48.900296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:47:48.900296+0000","last_clean_scrub_stamp":"2024-09-16T15:47:48.900296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T00:46:15.436142+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.21","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-16T15:47:49.946887+0000","last_change":"2024-09-16T15:47:49.944362+0000","last_active":"2024-09-16T15:47:49.946887+0000","last_peered":"2024-09-16T15:47:49.946887+0000","last_clean":"2024-09-16T15:47:49.946887+0000","last_became_active":"2024-09-16T15:47:49.943769+0000","last_became_peered":"2024-09-16T15:47:49.943769+0000","last_unstale":"2024-09-16T15:47:49.946887+0000","last_undegraded":"2024-09-16T15:47:49.946887+0000","last_fullsized":"2024-09-16T15:47:49.946887+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:47:48.900296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:47:48.900296+0000","last_clean_scrub_stamp":"2024-09-16T15:47:48.900296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T01:10:12.302580+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.20","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-16T15:47:49.965233+0000","last_change":"2024-09-16T15:47:49.954464+0000","last_active":"2024-09-16T15:47:49.965233+0000","last_peered":"2024-09-16T15:47:49.965233+0000","last_clean":"2024-09-16T15:47:49.965233+0000","last_became_active":"2024-09-16T15:47:49.951610+0000","last_became_peered":"2024-09-16T15:47:49.951610+0000","last_unstale":"2024-09-16T15:47:49.965233+0000","last_undegraded":"2024-09-16T15:47:49.965233+0000","last_fullsized":"2024-09-16T15:47:49.965233+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:47:48.900296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:47:48.900296+0000","last_clean_scrub_stamp":"2024-09-16T15:47:48.900296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T01:40:00.990103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1e","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-16T15:47:49.960907+0000","last_change":"2024-09-16T15:47:49.954470+0000","last_active":"2024-09-16T15:47:49.960907+0000","last_peered":"2024-09-16T15:47:49.960907+0000","last_clean":"2024-09-16T15:47:49.960907+0000","last_became_active":"2024-09-16T15:47:49.954122+0000","last_became_peered":"2024-09-16T15:47:49.954122+0000","last_unstale":"2024-09-16T15:47:49.960907+0000","last_undegraded":"2024-09-16T15:47:49.960907+0000","last_fullsized":"2024-09-16T15:47:49.960907+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:47:48.900296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:47:48.900296+0000","last_clean_scrub_stamp":"2024-09-16T15:47:48.900296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T02:14:30.434884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.1f","version":"84'4","reported_seq":17,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-16T15:47:49.969379+0000","last_change":"2024-09-16T15:47:48.955316+0000","last_active":"2024-09-16T15:47:49.969379+0000","last_peered":"2024-09-16T15:47:49.969379+0000","last_clean":"2024-09-16T15:47:49.969379+0000","last_became_active":"2024-09-16T15:47:48.953993+0000","last_became_peered":"2024-09-16T15:47:48.953993+0000","last_unstale":"2024-09-16T15:47:49.969379+0000","last_undegraded":"2024-09-16T15:47:49.969379+0000","last_fullsized":"2024-09-16T15:47:49.969379+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:47:47.896376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:47:47.896376+0000","last_clean_scrub_stamp":"2024-09-16T15:47:47.896376+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T02:04:51.201894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1f","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-16T15:47:49.965713+0000","last_change":"2024-09-16T15:47:49.960313+0000","last_active":"2024-09-16T15:47:49.965713+0000","last_peered":"2024-09-16T15:47:49.965713+0000","last_clean":"2024-09-16T15:47:49.965713+0000","last_became_active":"2024-09-16T15:47:49.959756+0000","last_became_peered":"2024-09-16T15:47:49.959756+0000","last_unstale":"2024-09-16T15:47:49.965713+0000","last_undegraded":"2024-09-16T15:47:49.965713+0000","last_fullsized":"2024-09-16T15:47:49.965713+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:47:48.900296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:47:48.900296+0000","last_clean_scrub_stamp":"2024-09-16T15:47:48.900296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:20:52.357755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.1e","version":"84'1","reported_seq":14,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-16T15:47:49.968016+0000","last_change":"2024-09-16T15:47:48.947418+0000","last_active":"2024-09-16T15:47:49.968016+0000","last_peered":"2024-09-16T15:47:49.968016+0000","last_clean":"2024-09-16T15:47:49.968016+0000","last_became_active":"2024-09-16T15:47:48.946721+0000","last_became_peered":"2024-09-16T15:47:48.946721+0000","last_unstale":"2024-09-16T15:47:49.968016+0000","last_undegraded":"2024-09-16T15:47:49.968016+0000","last_fullsized":"2024-09-16T15:47:49.968016+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:47:47.896376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:47:47.896376+0000","last_clean_scrub_stamp":"2024-09-16T15:47:47.896376+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T02:04:51.201894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.1c","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-16T15:47:49.963788+0000","last_change":"2024-09-16T15:47:49.950226+0000","last_active":"2024-09-16T15:47:49.963788+0000","last_peered":"2024-09-16T15:47:49.963788+0000","last_clean":"2024-09-16T15:47:49.963788+0000","last_became_active":"2024-09-16T15:47:49.949883+0000","last_became_peered":"2024-09-16T15:47:49.949883+0000","last_unstale":"2024-09-16T15:47:49.963788+0000","last_undegraded":"2024-09-16T15:47:49.963788+0000","last_fullsized":"2024-09-16T15:47:49.963788+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:47:48.900296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:47:48.900296+0000","last_clean_scrub_stamp":"2024-09-16T15:47:48.900296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T01:10:12.302580+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.1d","version":"84'2","reported_seq":15,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-16T15:47:49.969800+0000","last_change":"2024-09-16T15:47:48.936871+0000","last_active":"2024-09-16T15:47:49.969800+0000","last_peered":"2024-09-16T15:47:49.969800+0000","last_clean":"2024-09-16T15:47:49.969800+0000","last_became_active":"2024-09-16T15:47:48.936597+0000","last_became_peered":"2024-09-16T15:47:48.936597+0000","last_unstale":"2024-09-16T15:47:49.969800+0000","last_undegraded":"2024-09-16T15:47:49.969800+0000","last_fullsized":"2024-09-16T15:47:49.969800+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:47:47.896376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:47:47.896376+0000","last_clean_scrub_stamp":"2024-09-16T15:47:47.896376+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T02:32:37.856846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1d","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-16T15:47:49.952288+0000","last_change":"2024-09-16T15:47:49.949921+0000","last_active":"2024-09-16T15:47:49.952288+0000","last_peered":"2024-09-16T15:47:49.952288+0000","last_clean":"2024-09-16T15:47:49.952288+0000","last_became_active":"2024-09-16T15:47:49.949454+0000","last_became_peered":"2024-09-16T15:47:49.949454+0000","last_unstale":"2024-09-16T15:47:49.952288+0000","last_undegraded":"2024-09-16T15:47:49.952288+0000","last_fullsized":"2024-09-16T15:47:49.952288+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:47:48.900296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:47:48.900296+0000","last_clean_scrub_stamp":"2024-09-16T15:47:48.900296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T00:55:43.589146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1c","version":"84'2","reported_seq":15,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-16T15:47:49.971603+0000","last_change":"2024-09-16T15:47:48.938690+0000","last_active":"2024-09-16T15:47:49.971603+0000","last_peered":"2024-09-16T15:47:49.971603+0000","last_clean":"2024-09-16T15:47:49.971603+0000","last_became_active":"2024-09-16T15:47:48.938155+0000","last_became_peered":"2024-09-16T15:47:48.938155+0000","last_unstale":"2024-09-16T15:47:49.971603+0000","last_undegraded":"2024-09-16T15:47:49.971603+0000","last_fullsized":"2024-09-16T15:47:49.971603+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:47:47.896376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:47:47.896376+0000","last_clean_scrub_stamp":"2024-09-16T15:47:47.896376+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:11:58.285172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1a","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-16T15:47:49.971217+0000","last_change":"2024-09-16T15:47:49.968094+0000","last_active":"2024-09-16T15:47:49.971217+0000","last_peered":"2024-09-16T15:47:49.971217+0000","last_clean":"2024-09-16T15:47:49.971217+0000","last_became_active":"2024-09-16T15:47:49.965959+0000","last_became_peered":"2024-09-16T15:47:49.965959+0000","last_unstale":"2024-09-16T15:47:49.971217+0000","last_undegraded":"2024-09-16T15:47:49.971217+0000","last_fullsized":"2024-09-16T15:47:49.971217+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:47:48.900296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:47:48.900296+0000","last_clean_scrub_stamp":"2024-09-16T15:47:48.900296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:21:44.087843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.1b","version":"84'1","reported_seq":14,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-16T15:47:49.971203+0000","last_change":"2024-09-16T15:47:48.939448+0000","last_active":"2024-09-16T15:47:49.971203+0000","last_peered":"2024-09-16T15:47:49.971203+0000","last_clean":"2024-09-16T15:47:49.971203+0000","last_became_active":"2024-09-16T15:47:48.938758+0000","last_became_peered":"2024-09-16T15:47:48.938758+0000","last_unstale":"2024-09-16T15:47:49.971203+0000","last_undegraded":"2024-09-16T15:47:49.971203+0000","last_fullsized":"2024-09-16T15:47:49.971203+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:47:47.896376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:47:47.896376+0000","last_clean_scrub_stamp":"2024-09-16T15:47:47.896376+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:54:59.899958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.1b","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-16T15:47:49.946979+0000","last_change":"2024-09-16T15:47:49.944951+0000","last_active":"2024-09-16T15:47:49.946979+0000","last_peered":"2024-09-16T15:47:49.946979+0000","last_clean":"2024-09-16T15:47:49.946979+0000","last_became_active":"2024-09-16T15:47:49.944068+0000","last_became_peered":"2024-09-16T15:47:49.944068+0000","last_unstale":"2024-09-16T15:47:49.946979+0000","last_undegraded":"2024-09-16T15:47:49.946979+0000","last_fullsized":"2024-09-16T15:47:49.946979+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:47:48.900296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:47:48.900296+0000","last_clean_scrub_stamp":"2024-09-16T15:47:48.900296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T00:04:45.746226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.1a","version":"84'1","reported_seq":14,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-16T15:47:49.968408+0000","last_change":"2024-09-16T15:47:48.947975+0000","last_active":"2024-09-16T15:47:49.968408+0000","last_peered":"2024-09-16T15:47:49.968408+0000","last_clean":"2024-09-16T15:47:49.968408+0000","last_became_active":"2024-09-16T15:47:48.945655+0000","last_became_peered":"2024-09-16T15:47:48.945655+0000","last_unstale":"2024-09-16T15:47:49.968408+0000","last_undegraded":"2024-09-16T15:47:49.968408+0000","last_fullsized":"2024-09-16T15:47:49.968408+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:47:47.896376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:47:47.896376+0000","last_clean_scrub_stamp":"2024-09-16T15:47:47.896376+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T01:13:31.262332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.18","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-16T15:47:49.958863+0000","last_change":"2024-09-16T15:47:49.956715+0000","last_active":"2024-09-16T15:47:49.958863+0000","last_peered":"2024-09-16T15:47:49.958863+0000","last_clean":"2024-09-16T15:47:49.958863+0000","last_became_active":"2024-09-16T15:47:49.956383+0000","last_became_peered":"2024-09-16T15:47:49.956383+0000","last_unstale":"2024-09-16T15:47:49.958863+0000","last_undegraded":"2024-09-16T15:47:49.958863+0000","last_fullsized":"2024-09-16T15:47:49.958863+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:47:48.900296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:47:48.900296+0000","last_clean_scrub_stamp":"2024-09-16T15:47:48.900296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:18:26.678453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.19","version":"0'0","reported_seq":13,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-16T15:47:49.932543+0000","last_change":"2024-09-16T15:47:48.932898+0000","last_active":"2024-09-16T15:47:49.932543+0000","last_peered":"2024-09-16T15:47:49.932543+0000","last_clean":"2024-09-16T15:47:49.932543+0000","last_became_active":"2024-09-16T15:47:48.932438+0000","last_became_peered":"2024-09-16T15:47:48.932438+0000","last_unstale":"2024-09-16T15:47:49.932543+0000","last_undegraded":"2024-09-16T15:47:49.932543+0000","last_fullsized":"2024-09-16T15:47:49.932543+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:47:47.896376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:47:47.896376+0000","last_clean_scrub_stamp":"2024-09-16T15:47:47.896376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T23:56:33.543562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.19","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-16T15:47:49.967023+0000","last_change":"2024-09-16T15:47:49.949215+0000","last_active":"2024-09-16T15:47:49.967023+0000","last_peered":"2024-09-16T15:47:49.967023+0000","last_clean":"2024-09-16T15:47:49.967023+0000","last_became_active":"2024-09-16T15:47:49.947130+0000","last_became_peered":"2024-09-16T15:47:49.947130+0000","last_unstale":"2024-09-16T15:47:49.967023+0000","last_undegraded":"2024-09-16T15:47:49.967023+0000","last_fullsized":"2024-09-16T15:47:49.967023+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:47:48.900296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:47:48.900296+0000","last_clean_scrub_stamp":"2024-09-16T15:47:48.900296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T02:53:54.255699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.18","version":"0'0","reported_seq":13,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-16T15:47:49.926748+0000","last_change":"2024-09-16T15:47:48.942427+0000","last_active":"2024-09-16T15:47:49.926748+0000","last_peered":"2024-09-16T15:47:49.926748+0000","last_clean":"2024-09-16T15:47:49.926748+0000","last_became_active":"2024-09-16T15:47:48.941828+0000","last_became_peered":"2024-09-16T15:47:48.941828+0000","last_unstale":"2024-09-16T15:47:49.926748+0000","last_undegraded":"2024-09-16T15:47:49.926748+0000","last_fullsized":"2024-09-16T15:47:49.926748+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:47:47.896376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:47:47.896376+0000","last_clean_scrub_stamp":"2024-09-16T15:47:47.896376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:49:00.837314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.16","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-16T15:47:49.967017+0000","last_change":"2024-09-16T15:47:49.951327+0000","last_active":"2024-09-16T15:47:49.967017+0000","last_peered":"2024-09-16T15:47:49.967017+0000","last_clean":"2024-09-16T15:47:49.967017+0000","last_became_active":"2024-09-16T15:47:49.951047+0000","last_became_peered":"2024-09-16T15:47:49.951047+0000","last_unstale":"2024-09-16T15:47:49.967017+0000","last_undegraded":"2024-09-16T15:47:49.967017+0000","last_fullsized":"2024-09-16T15:47:49.967017+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:47:48.900296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:47:48.900296+0000","last_clean_scrub_stamp":"2024-09-16T15:47:48.900296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:42:34.333211+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.17","version":"0'0","reported_seq":13,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-16T15:47:49.944184+0000","last_change":"2024-09-16T15:47:48.944783+0000","last_active":"2024-09-16T15:47:49.944184+0000","last_peered":"2024-09-16T15:47:49.944184+0000","last_clean":"2024-09-16T15:47:49.944184+0000","last_became_active":"2024-09-16T15:47:48.944048+0000","last_became_peered":"2024-09-16T15:47:48.944048+0000","last_unstale":"2024-09-16T15:47:49.944184+0000","last_undegraded":"2024-09-16T15:47:49.944184+0000","last_fullsized":"2024-09-16T15:47:49.944184+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:47:47.896376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:47:47.896376+0000","last_clean_scrub_stamp":"2024-09-16T15:47:47.896376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T03:44:36.043565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.17","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-16T15:47:49.952340+0000","last_change":"2024-09-16T15:47:49.947717+0000","last_active":"2024-09-16T15:47:49.952340+0000","last_peered":"2024-09-16T15:47:49.952340+0000","last_clean":"2024-09-16T15:47:49.952340+0000","last_became_active":"2024-09-16T15:47:49.947382+0000","last_became_peered":"2024-09-16T15:47:49.947382+0000","last_unstale":"2024-09-16T15:47:49.952340+0000","last_undegraded":"2024-09-16T15:47:49.952340+0000","last_fullsized":"2024-09-16T15:47:49.952340+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:47:48.900296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:47:48.900296+0000","last_clean_scrub_stamp":"2024-09-16T15:47:48.900296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:46:07.109158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.16","version":"0'0","reported_seq":13,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-16T15:47:49.933651+0000","last_change":"2024-09-16T15:47:48.952563+0000","last_active":"2024-09-16T15:47:49.933651+0000","last_peered":"2024-09-16T15:47:49.933651+0000","last_clean":"2024-09-16T15:47:49.933651+0000","last_became_active":"2024-09-16T15:47:48.951819+0000","last_became_peered":"2024-09-16T15:47:48.951819+0000","last_unstale":"2024-09-16T15:47:49.933651+0000","last_undegraded":"2024-09-16T15:47:49.933651+0000","last_fullsized":"2024-09-16T15:47:49.933651+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:47:47.896376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:47:47.896376+0000","last_clean_scrub_stamp":"2024-09-16T15:47:47.896376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T18:51:35.155740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.14","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-16T15:47:49.959697+0000","last_change":"2024-09-16T15:47:49.955291+0000","last_active":"2024-09-16T15:47:49.959697+0000","last_peered":"2024-09-16T15:47:49.959697+0000","last_clean":"2024-09-16T15:47:49.959697+0000","last_became_active":"2024-09-16T15:47:49.954525+0000","last_became_peered":"2024-09-16T15:47:49.954525+0000","last_unstale":"2024-09-16T15:47:49.959697+0000","last_undegraded":"2024-09-16T15:47:49.959697+0000","last_fullsized":"2024-09-16T15:47:49.959697+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:47:48.900296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:47:48.900296+0000","last_clean_scrub_stamp":"2024-09-16T15:47:48.900296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T00:22:18.050637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.15","version":"84'3","reported_seq":16,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-16T15:47:49.971701+0000","last_change":"2024-09-16T15:47:48.937825+0000","last_active":"2024-09-16T15:47:49.971701+0000","last_peered":"2024-09-16T15:47:49.971701+0000","last_clean":"2024-09-16T15:47:49.971701+0000","last_became_active":"2024-09-16T15:47:48.937020+0000","last_became_peered":"2024-09-16T15:47:48.937020+0000","last_unstale":"2024-09-16T15:47:49.971701+0000","last_undegraded":"2024-09-16T15:47:49.971701+0000","last_fullsized":"2024-09-16T15:47:49.971701+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:47:47.896376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:47:47.896376+0000","last_clean_scrub_stamp":"2024-09-16T15:47:47.896376+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:31:05.983747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.15","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-16T15:47:49.952691+0000","last_change":"2024-09-16T15:47:49.948684+0000","last_active":"2024-09-16T15:47:49.952691+0000","last_peered":"2024-09-16T15:47:49.952691+0000","last_clean":"2024-09-16T15:47:49.952691+0000","last_became_active":"2024-09-16T15:47:49.948316+0000","last_became_peered":"2024-09-16T15:47:49.948316+0000","last_unstale":"2024-09-16T15:47:49.952691+0000","last_undegraded":"2024-09-16T15:47:49.952691+0000","last_fullsized":"2024-09-16T15:47:49.952691+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:47:48.900296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:47:48.900296+0000","last_clean_scrub_stamp":"2024-09-16T15:47:48.900296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T18:59:41.944452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.14","version":"84'6","reported_seq":19,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-16T15:47:49.975591+0000","last_change":"2024-09-16T15:47:48.937794+0000","last_active":"2024-09-16T15:47:49.975591+0000","last_peered":"2024-09-16T15:47:49.975591+0000","last_clean":"2024-09-16T15:47:49.975591+0000","last_became_active":"2024-09-16T15:47:48.937048+0000","last_became_peered":"2024-09-16T15:47:48.937048+0000","last_unstale":"2024-09-16T15:47:49.975591+0000","last_undegraded":"2024-09-16T15:47:49.975591+0000","last_fullsized":"2024-09-16T15:47:49.975591+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:47:47.896376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:47:47.896376+0000","last_clean_scrub_stamp":"2024-09-16T15:47:47.896376+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T01:00:53.376119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.12","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-16T15:47:49.955792+0000","last_change":"2024-09-16T15:47:49.945512+0000","last_active":"2024-09-16T15:47:49.955792+0000","last_peered":"2024-09-16T15:47:49.955792+0000","last_clean":"2024-09-16T15:47:49.955792+0000","last_became_active":"2024-09-16T15:47:49.944649+0000","last_became_peered":"2024-09-16T15:47:49.944649+0000","last_unstale":"2024-09-16T15:47:49.955792+0000","last_undegraded":"2024-09-16T15:47:49.955792+0000","last_fullsized":"2024-09-16T15:47:49.955792+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:47:48.900296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:47:48.900296+0000","last_clean_scrub_stamp":"2024-09-16T15:47:48.900296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T00:04:26.824059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.13","version":"84'2","reported_seq":15,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-16T15:47:49.969134+0000","last_change":"2024-09-16T15:47:48.935163+0000","last_active":"2024-09-16T15:47:49.969134+0000","last_peered":"2024-09-16T15:47:49.969134+0000","last_clean":"2024-09-16T15:47:49.969134+0000","last_became_active":"2024-09-16T15:47:48.934838+0000","last_became_peered":"2024-09-16T15:47:48.934838+0000","last_unstale":"2024-09-16T15:47:49.969134+0000","last_undegraded":"2024-09-16T15:47:49.969134+0000","last_fullsized":"2024-09-16T15:47:49.969134+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:47:47.896376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:47:47.896376+0000","last_clean_scrub_stamp":"2024-09-16T15:47:47.896376+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:17:14.703070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.13","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-16T15:47:49.949074+0000","last_change":"2024-09-16T15:47:49.945243+0000","last_active":"2024-09-16T15:47:49.949074+0000","last_peered":"2024-09-16T15:47:49.949074+0000","last_clean":"2024-09-16T15:47:49.949074+0000","last_became_active":"2024-09-16T15:47:49.944910+0000","last_became_peered":"2024-09-16T15:47:49.944910+0000","last_unstale":"2024-09-16T15:47:49.949074+0000","last_undegraded":"2024-09-16T15:47:49.949074+0000","last_fullsized":"2024-09-16T15:47:49.949074+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:47:48.900296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:47:48.900296+0000","last_clean_scrub_stamp":"2024-09-16T15:47:48.900296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T22:08:30.436461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.12","version":"0'0","reported_seq":13,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-16T15:47:49.918876+0000","last_change":"2024-09-16T15:47:48.933762+0000","last_active":"2024-09-16T15:47:49.918876+0000","last_peered":"2024-09-16T15:47:49.918876+0000","last_clean":"2024-09-16T15:47:49.918876+0000","last_became_active":"2024-09-16T15:47:48.933454+0000","last_became_peered":"2024-09-16T15:47:48.933454+0000","last_unstale":"2024-09-16T15:47:49.918876+0000","last_undegraded":"2024-09-16T15:47:49.918876+0000","last_fullsized":"2024-09-16T15:47:49.918876+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:47:47.896376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:47:47.896376+0000","last_clean_scrub_stamp":"2024-09-16T15:47:47.896376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:34:51.472023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.10","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-16T15:47:49.958604+0000","last_change":"2024-09-16T15:47:49.949896+0000","last_active":"2024-09-16T15:47:49.958604+0000","last_peered":"2024-09-16T15:47:49.958604+0000","last_clean":"2024-09-16T15:47:49.958604+0000","last_became_active":"2024-09-16T15:47:49.945795+0000","last_became_peered":"2024-09-16T15:47:49.945795+0000","last_unstale":"2024-09-16T15:47:49.958604+0000","last_undegraded":"2024-09-16T15:47:49.958604+0000","last_fullsized":"2024-09-16T15:47:49.958604+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:47:48.900296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:47:48.900296+0000","last_clean_scrub_stamp":"2024-09-16T15:47:48.900296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:33:49.653526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.11","version":"0'0","reported_seq":13,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-16T15:47:49.938870+0000","last_change":"2024-09-16T15:47:48.950081+0000","last_active":"2024-09-16T15:47:49.938870+0000","last_peered":"2024-09-16T15:47:49.938870+0000","last_clean":"2024-09-16T15:47:49.938870+0000","last_became_active":"2024-09-16T15:47:48.949230+0000","last_became_peered":"2024-09-16T15:47:48.949230+0000","last_unstale":"2024-09-16T15:47:49.938870+0000","last_undegraded":"2024-09-16T15:47:49.938870+0000","last_fullsized":"2024-09-16T15:47:49.938870+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:47:47.896376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:47:47.896376+0000","last_clean_scrub_stamp":"2024-09-16T15:47:47.896376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:02:43.375408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.11","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-16T15:47:49.954196+0000","last_change":"2024-09-16T15:47:49.952975+0000","last_active":"2024-09-16T15:47:49.954196+0000","last_peered":"2024-09-16T15:47:49.954196+0000","last_clean":"2024-09-16T15:47:49.954196+0000","last_became_active":"2024-09-16T15:47:49.952671+0000","last_became_peered":"2024-09-16T15:47:49.952671+0000","last_unstale":"2024-09-16T15:47:49.954196+0000","last_undegraded":"2024-09-16T15:47:49.954196+0000","last_fullsized":"2024-09-16T15:47:49.954196+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:47:48.900296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:47:48.900296+0000","last_clean_scrub_stamp":"2024-09-16T15:47:48.900296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T18:15:01.990027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.10","version":"84'8","reported_seq":21,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-16T15:47:49.970650+0000","last_change":"2024-09-16T15:47:48.950051+0000","last_active":"2024-09-16T15:47:49.970650+0000","last_peered":"2024-09-16T15:47:49.970650+0000","last_clean":"2024-09-16T15:47:49.970650+0000","last_became_active":"2024-09-16T15:47:48.949613+0000","last_became_peered":"2024-09-16T15:47:48.949613+0000","last_unstale":"2024-09-16T15:47:49.970650+0000","last_undegraded":"2024-09-16T15:47:49.970650+0000","last_fullsized":"2024-09-16T15:47:49.970650+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:47:47.896376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:47:47.896376+0000","last_clean_scrub_stamp":"2024-09-16T15:47:47.896376+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:32:36.238108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.e","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-16T15:47:49.949029+0000","last_change":"2024-09-16T15:47:49.943986+0000","last_active":"2024-09-16T15:47:49.949029+0000","last_peered":"2024-09-16T15:47:49.949029+0000","last_clean":"2024-09-16T15:47:49.949029+0000","last_became_active":"2024-09-16T15:47:49.943648+0000","last_became_peered":"2024-09-16T15:47:49.943648+0000","last_unstale":"2024-09-16T15:47:49.949029+0000","last_undegraded":"2024-09-16T15:47:49.949029+0000","last_fullsized":"2024-09-16T15:47:49.949029+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:47:48.900296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:47:48.900296+0000","last_clean_scrub_stamp":"2024-09-16T15:47:48.900296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T23:31:42.834248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.f","version":"84'1","reported_seq":14,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-16T15:47:49.967813+0000","last_change":"2024-09-16T15:47:48.949583+0000","last_active":"2024-09-16T15:47:49.967813+0000","last_peered":"2024-09-16T15:47:49.967813+0000","last_clean":"2024-09-16T15:47:49.967813+0000","last_became_active":"2024-09-16T15:47:48.948872+0000","last_became_peered":"2024-09-16T15:47:48.948872+0000","last_unstale":"2024-09-16T15:47:49.967813+0000","last_undegraded":"2024-09-16T15:47:49.967813+0000","last_fullsized":"2024-09-16T15:47:49.967813+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:47:47.896376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:47:47.896376+0000","last_clean_scrub_stamp":"2024-09-16T15:47:47.896376+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T23:45:20.881147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.f","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-16T15:47:49.965508+0000","last_change":"2024-09-16T15:47:49.954670+0000","last_active":"2024-09-16T15:47:49.965508+0000","last_peered":"2024-09-16T15:47:49.965508+0000","last_clean":"2024-09-16T15:47:49.965508+0000","last_became_active":"2024-09-16T15:47:49.954335+0000","last_became_peered":"2024-09-16T15:47:49.954335+0000","last_unstale":"2024-09-16T15:47:49.965508+0000","last_undegraded":"2024-09-16T15:47:49.965508+0000","last_fullsized":"2024-09-16T15:47:49.965508+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:47:48.900296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:47:48.900296+0000","last_clean_scrub_stamp":"2024-09-16T15:47:48.900296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T23:50:41.566032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.e","version":"84'2","reported_seq":15,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-16T15:47:49.967357+0000","last_change":"2024-09-16T15:47:48.948673+0000","last_active":"2024-09-16T15:47:49.967357+0000","last_peered":"2024-09-16T15:47:49.967357+0000","last_clean":"2024-09-16T15:47:49.967357+0000","last_became_active":"2024-09-16T15:47:48.948305+0000","last_became_peered":"2024-09-16T15:47:48.948305+0000","last_unstale":"2024-09-16T15:47:49.967357+0000","last_undegraded":"2024-09-16T15:47:49.967357+0000","last_fullsized":"2024-09-16T15:47:49.967357+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:47:47.896376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:47:47.896376+0000","last_clean_scrub_stamp":"2024-09-16T15:47:47.896376+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:19:43.539598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.d","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-16T15:47:49.949814+0000","last_change":"2024-09-16T15:47:49.947062+0000","last_active":"2024-09-16T15:47:49.949814+0000","last_peered":"2024-09-16T15:47:49.949814+0000","last_clean":"2024-09-16T15:47:49.949814+0000","last_became_active":"2024-09-16T15:47:49.946466+0000","last_became_peered":"2024-09-16T15:47:49.946466+0000","last_unstale":"2024-09-16T15:47:49.949814+0000","last_undegraded":"2024-09-16T15:47:49.949814+0000","last_fullsized":"2024-09-16T15:47:49.949814+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:47:48.900296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:47:48.900296+0000","last_clean_scrub_stamp":"2024-09-16T15:47:48.900296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T23:50:41.566032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.c","version":"0'0","reported_seq":13,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-16T15:47:49.930396+0000","last_change":"2024-09-16T15:47:48.944636+0000","last_active":"2024-09-16T15:47:49.930396+0000","last_peered":"2024-09-16T15:47:49.930396+0000","last_clean":"2024-09-16T15:47:49.930396+0000","last_became_active":"2024-09-16T15:47:48.944233+0000","last_became_peered":"2024-09-16T15:47:48.944233+0000","last_unstale":"2024-09-16T15:47:49.930396+0000","last_undegraded":"2024-09-16T15:47:49.930396+0000","last_fullsized":"2024-09-16T15:47:49.930396+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:47:47.896376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:47:47.896376+0000","last_clean_scrub_stamp":"2024-09-16T15:47:47.896376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:27:47.846253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.a","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-16T15:47:49.968675+0000","last_change":"2024-09-16T15:47:49.959515+0000","last_active":"2024-09-16T15:47:49.968675+0000","last_peered":"2024-09-16T15:47:49.968675+0000","last_clean":"2024-09-16T15:47:49.968675+0000","last_became_active":"2024-09-16T15:47:49.958726+0000","last_became_peered":"2024-09-16T15:47:49.958726+0000","last_unstale":"2024-09-16T15:47:49.968675+0000","last_undegraded":"2024-09-16T15:47:49.968675+0000","last_fullsized":"2024-09-16T15:47:49.968675+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:47:48.900296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:47:48.900296+0000","last_clean_scrub_stamp":"2024-09-16T15:47:48.900296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T23:47:32.797895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.b","version":"84'2","reported_seq":15,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-16T15:47:49.969318+0000","last_change":"2024-09-16T15:47:48.935075+0000","last_active":"2024-09-16T15:47:49.969318+0000","last_peered":"2024-09-16T15:47:49.969318+0000","last_clean":"2024-09-16T15:47:49.969318+0000","last_became_active":"2024-09-16T15:47:48.933928+0000","last_became_peered":"2024-09-16T15:47:48.933928+0000","last_unstale":"2024-09-16T15:47:49.969318+0000","last_undegraded":"2024-09-16T15:47:49.969318+0000","last_fullsized":"2024-09-16T15:47:49.969318+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:47:47.896376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:47:47.896376+0000","last_clean_scrub_stamp":"2024-09-16T15:47:47.896376+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T02:31:10.840460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.b","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-16T15:47:49.960071+0000","last_change":"2024-09-16T15:47:49.955540+0000","last_active":"2024-09-16T15:47:49.960071+0000","last_peered":"2024-09-16T15:47:49.960071+0000","last_clean":"2024-09-16T15:47:49.960071+0000","last_became_active":"2024-09-16T15:47:49.955159+0000","last_became_peered":"2024-09-16T15:47:49.955159+0000","last_unstale":"2024-09-16T15:47:49.960071+0000","last_undegraded":"2024-09-16T15:47:49.960071+0000","last_fullsized":"2024-09-16T15:47:49.960071+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:47:48.900296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:47:48.900296+0000","last_clean_scrub_stamp":"2024-09-16T15:47:48.900296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:10:31.222177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.a","version":"0'0","reported_seq":13,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-16T15:47:49.930610+0000","last_change":"2024-09-16T15:47:48.937958+0000","last_active":"2024-09-16T15:47:49.930610+0000","last_peered":"2024-09-16T15:47:49.930610+0000","last_clean":"2024-09-16T15:47:49.930610+0000","last_became_active":"2024-09-16T15:47:48.937371+0000","last_became_peered":"2024-09-16T15:47:48.937371+0000","last_unstale":"2024-09-16T15:47:49.930610+0000","last_undegraded":"2024-09-16T15:47:49.930610+0000","last_fullsized":"2024-09-16T15:47:49.930610+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:47:47.896376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:47:47.896376+0000","last_clean_scrub_stamp":"2024-09-16T15:47:47.896376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:33.183070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.8","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-16T15:47:49.961264+0000","last_change":"2024-09-16T15:47:49.947927+0000","last_active":"2024-09-16T15:47:49.961264+0000","last_peered":"2024-09-16T15:47:49.961264+0000","last_clean":"2024-09-16T15:47:49.961264+0000","last_became_active":"2024-09-16T15:47:49.947313+0000","last_became_peered":"2024-09-16T15:47:49.947313+0000","last_unstale":"2024-09-16T15:47:49.961264+0000","last_undegraded":"2024-09-16T15:47:49.961264+0000","last_fullsized":"2024-09-16T15:47:49.961264+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:47:48.900296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:47:48.900296+0000","last_clean_scrub_stamp":"2024-09-16T15:47:48.900296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T22:05:54.637849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.9","version":"0'0","reported_seq":13,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-16T15:47:49.926283+0000","last_change":"2024-09-16T15:47:48.950021+0000","last_active":"2024-09-16T15:47:49.926283+0000","last_peered":"2024-09-16T15:47:49.926283+0000","last_clean":"2024-09-16T15:47:49.926283+0000","last_became_active":"2024-09-16T15:47:48.949573+0000","last_became_peered":"2024-09-16T15:47:48.949573+0000","last_unstale":"2024-09-16T15:47:49.926283+0000","last_undegraded":"2024-09-16T15:47:49.926283+0000","last_fullsized":"2024-09-16T15:47:49.926283+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:47:47.896376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:47:47.896376+0000","last_clean_scrub_stamp":"2024-09-16T15:47:47.896376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T02:24:11.311434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.9","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-16T15:47:49.961757+0000","last_change":"2024-09-16T15:47:49.957145+0000","last_active":"2024-09-16T15:47:49.961757+0000","last_peered":"2024-09-16T15:47:49.961757+0000","last_clean":"2024-09-16T15:47:49.961757+0000","last_became_active":"2024-09-16T15:47:49.956767+0000","last_became_peered":"2024-09-16T15:47:49.956767+0000","last_unstale":"2024-09-16T15:47:49.961757+0000","last_undegraded":"2024-09-16T15:47:49.961757+0000","last_fullsized":"2024-09-16T15:47:49.961757+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:47:48.900296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:47:48.900296+0000","last_clean_scrub_stamp":"2024-09-16T15:47:48.900296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T01:43:48.754810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.8","version":"0'0","reported_seq":13,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-16T15:47:49.918403+0000","last_change":"2024-09-16T15:47:48.934536+0000","last_active":"2024-09-16T15:47:49.918403+0000","last_peered":"2024-09-16T15:47:49.918403+0000","last_clean":"2024-09-16T15:47:49.918403+0000","last_became_active":"2024-09-16T15:47:48.933921+0000","last_became_peered":"2024-09-16T15:47:48.933921+0000","last_unstale":"2024-09-16T15:47:49.918403+0000","last_undegraded":"2024-09-16T15:47:49.918403+0000","last_fullsized":"2024-09-16T15:47:49.918403+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:47:47.896376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:47:47.896376+0000","last_clean_scrub_stamp":"2024-09-16T15:47:47.896376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T00:36:28.684545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.6","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-16T15:47:49.951537+0000","last_change":"2024-09-16T15:47:49.946212+0000","last_active":"2024-09-16T15:47:49.951537+0000","last_peered":"2024-09-16T15:47:49.951537+0000","last_clean":"2024-09-16T15:47:49.951537+0000","last_became_active":"2024-09-16T15:47:49.945916+0000","last_became_peered":"2024-09-16T15:47:49.945916+0000","last_unstale":"2024-09-16T15:47:49.951537+0000","last_undegraded":"2024-09-16T15:47:49.951537+0000","last_fullsized":"2024-09-16T15:47:49.951537+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:47:48.900296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:47:48.900296+0000","last_clean_scrub_stamp":"2024-09-16T15:47:48.900296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:06:06.905606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.7","version":"84'2","reported_seq":15,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-16T15:47:49.970300+0000","last_change":"2024-09-16T15:47:48.948961+0000","last_active":"2024-09-16T15:47:49.970300+0000","last_peered":"2024-09-16T15:47:49.970300+0000","last_clean":"2024-09-16T15:47:49.970300+0000","last_became_active":"2024-09-16T15:47:48.948573+0000","last_became_peered":"2024-09-16T15:47:48.948573+0000","last_unstale":"2024-09-16T15:47:49.970300+0000","last_undegraded":"2024-09-16T15:47:49.970300+0000","last_fullsized":"2024-09-16T15:47:49.970300+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:47:47.896376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:47:47.896376+0000","last_clean_scrub_stamp":"2024-09-16T15:47:47.896376+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T23:06:55.771079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.7","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-16T15:47:49.958721+0000","last_change":"2024-09-16T15:47:49.956120+0000","last_active":"2024-09-16T15:47:49.958721+0000","last_peered":"2024-09-16T15:47:49.958721+0000","last_clean":"2024-09-16T15:47:49.958721+0000","last_became_active":"2024-09-16T15:47:49.955651+0000","last_became_peered":"2024-09-16T15:47:49.955651+0000","last_unstale":"2024-09-16T15:47:49.958721+0000","last_undegraded":"2024-09-16T15:47:49.958721+0000","last_fullsized":"2024-09-16T15:47:49.958721+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:47:48.900296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:47:48.900296+0000","last_clean_scrub_stamp":"2024-09-16T15:47:48.900296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:39:18.537968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.6","version":"0'0","reported_seq":13,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-16T15:47:49.936049+0000","last_change":"2024-09-16T15:47:48.933160+0000","last_active":"2024-09-16T15:47:49.936049+0000","last_peered":"2024-09-16T15:47:49.936049+0000","last_clean":"2024-09-16T15:47:49.936049+0000","last_became_active":"2024-09-16T15:47:48.932792+0000","last_became_peered":"2024-09-16T15:47:48.932792+0000","last_unstale":"2024-09-16T15:47:49.936049+0000","last_undegraded":"2024-09-16T15:47:49.936049+0000","last_fullsized":"2024-09-16T15:47:49.936049+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:47:47.896376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:47:47.896376+0000","last_clean_scrub_stamp":"2024-09-16T15:47:47.896376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T01:25:02.263975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.4","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-16T15:47:49.951163+0000","last_change":"2024-09-16T15:47:49.942492+0000","last_active":"2024-09-16T15:47:49.951163+0000","last_peered":"2024-09-16T15:47:49.951163+0000","last_clean":"2024-09-16T15:47:49.951163+0000","last_became_active":"2024-09-16T15:47:49.941976+0000","last_became_peered":"2024-09-16T15:47:49.941976+0000","last_unstale":"2024-09-16T15:47:49.951163+0000","last_undegraded":"2024-09-16T15:47:49.951163+0000","last_fullsized":"2024-09-16T15:47:49.951163+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:47:48.900296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:47:48.900296+0000","last_clean_scrub_stamp":"2024-09-16T15:47:48.900296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:55:00.903877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.5","version":"0'0","reported_seq":13,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-16T15:47:49.930559+0000","last_change":"2024-09-16T15:47:48.949355+0000","last_active":"2024-09-16T15:47:49.930559+0000","last_peered":"2024-09-16T15:47:49.930559+0000","last_clean":"2024-09-16T15:47:49.930559+0000","last_became_active":"2024-09-16T15:47:48.948585+0000","last_became_peered":"2024-09-16T15:47:48.948585+0000","last_unstale":"2024-09-16T15:47:49.930559+0000","last_undegraded":"2024-09-16T15:47:49.930559+0000","last_fullsized":"2024-09-16T15:47:49.930559+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:47:47.896376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:47:47.896376+0000","last_clean_scrub_stamp":"2024-09-16T15:47:47.896376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:49:44.422191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.5","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-16T15:47:49.949784+0000","last_change":"2024-09-16T15:47:49.946789+0000","last_active":"2024-09-16T15:47:49.949784+0000","last_peered":"2024-09-16T15:47:49.949784+0000","last_clean":"2024-09-16T15:47:49.949784+0000","last_became_active":"2024-09-16T15:47:49.945994+0000","last_became_peered":"2024-09-16T15:47:49.945994+0000","last_unstale":"2024-09-16T15:47:49.949784+0000","last_undegraded":"2024-09-16T15:47:49.949784+0000","last_fullsized":"2024-09-16T15:47:49.949784+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:47:48.900296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:47:48.900296+0000","last_clean_scrub_stamp":"2024-09-16T15:47:48.900296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T01:40:00.990103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.4","version":"0'0","reported_seq":13,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-16T15:47:49.940738+0000","last_change":"2024-09-16T15:47:48.942233+0000","last_active":"2024-09-16T15:47:49.940738+0000","last_peered":"2024-09-16T15:47:49.940738+0000","last_clean":"2024-09-16T15:47:49.940738+0000","last_became_active":"2024-09-16T15:47:48.941519+0000","last_became_peered":"2024-09-16T15:47:48.941519+0000","last_unstale":"2024-09-16T15:47:49.940738+0000","last_undegraded":"2024-09-16T15:47:49.940738+0000","last_fullsized":"2024-09-16T15:47:49.940738+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:47:47.896376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:47:47.896376+0000","last_clean_scrub_stamp":"2024-09-16T15:47:47.896376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:23:16.080216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.2","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-16T15:47:49.972948+0000","last_change":"2024-09-16T15:47:49.967401+0000","last_active":"2024-09-16T15:47:49.972948+0000","last_peered":"2024-09-16T15:47:49.972948+0000","last_clean":"2024-09-16T15:47:49.972948+0000","last_became_active":"2024-09-16T15:47:49.967107+0000","last_became_peered":"2024-09-16T15:47:49.967107+0000","last_unstale":"2024-09-16T15:47:49.972948+0000","last_undegraded":"2024-09-16T15:47:49.972948+0000","last_fullsized":"2024-09-16T15:47:49.972948+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:47:48.900296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:47:48.900296+0000","last_clean_scrub_stamp":"2024-09-16T15:47:48.900296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T22:18:29.564614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.3","version":"0'0","reported_seq":13,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-16T15:47:49.928689+0000","last_change":"2024-09-16T15:47:48.942460+0000","last_active":"2024-09-16T15:47:49.928689+0000","last_peered":"2024-09-16T15:47:49.928689+0000","last_clean":"2024-09-16T15:47:49.928689+0000","last_became_active":"2024-09-16T15:47:48.942168+0000","last_became_peered":"2024-09-16T15:47:48.942168+0000","last_unstale":"2024-09-16T15:47:49.928689+0000","last_undegraded":"2024-09-16T15:47:49.928689+0000","last_fullsized":"2024-09-16T15:47:49.928689+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:47:47.896376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:47:47.896376+0000","last_clean_scrub_stamp":"2024-09-16T15:47:47.896376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T22:18:28.560694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.3","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-16T15:47:49.955493+0000","last_change":"2024-09-16T15:47:49.945626+0000","last_active":"2024-09-16T15:47:49.955493+0000","last_peered":"2024-09-16T15:47:49.955493+0000","last_clean":"2024-09-16T15:47:49.955493+0000","last_became_active":"2024-09-16T15:47:49.944910+0000","last_became_peered":"2024-09-16T15:47:49.944910+0000","last_unstale":"2024-09-16T15:47:49.955493+0000","last_undegraded":"2024-09-16T15:47:49.955493+0000","last_fullsized":"2024-09-16T15:47:49.955493+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:47:48.900296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:47:48.900296+0000","last_clean_scrub_stamp":"2024-09-16T15:47:48.900296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T00:55:24.956739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.2","version":"0'0","reported_seq":13,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-16T15:47:49.923417+0000","last_change":"2024-09-16T15:47:48.939060+0000","last_active":"2024-09-16T15:47:49.923417+0000","last_peered":"2024-09-16T15:47:49.923417+0000","last_clean":"2024-09-16T15:47:49.923417+0000","last_became_active":"2024-09-16T15:47:48.938258+0000","last_became_peered":"2024-09-16T15:47:48.938258+0000","last_unstale":"2024-09-16T15:47:49.923417+0000","last_undegraded":"2024-09-16T15:47:49.923417+0000","last_fullsized":"2024-09-16T15:47:49.923417+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:47:47.896376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:47:47.896376+0000","last_clean_scrub_stamp":"2024-09-16T15:47:47.896376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T18:51:35.155740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.0","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-16T15:47:49.964094+0000","last_change":"2024-09-16T15:47:49.958222+0000","last_active":"2024-09-16T15:47:49.964094+0000","last_peered":"2024-09-16T15:47:49.964094+0000","last_clean":"2024-09-16T15:47:49.964094+0000","last_became_active":"2024-09-16T15:47:49.957921+0000","last_became_peered":"2024-09-16T15:47:49.957921+0000","last_unstale":"2024-09-16T15:47:49.964094+0000","last_undegraded":"2024-09-16T15:47:49.964094+0000","last_fullsized":"2024-09-16T15:47:49.964094+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:47:48.900296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:47:48.900296+0000","last_clean_scrub_stamp":"2024-09-16T15:47:48.900296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:50:05.184566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1","version":"0'0","reported_seq":13,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-16T15:47:49.932546+0000","last_change":"2024-09-16T15:47:48.944931+0000","last_active":"2024-09-16T15:47:49.932546+0000","last_peered":"2024-09-16T15:47:49.932546+0000","last_clean":"2024-09-16T15:47:49.932546+0000","last_became_active":"2024-09-16T15:47:48.944168+0000","last_became_peered":"2024-09-16T15:47:48.944168+0000","last_unstale":"2024-09-16T15:47:49.932546+0000","last_undegraded":"2024-09-16T15:47:49.932546+0000","last_fullsized":"2024-09-16T15:47:49.932546+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:47:47.896376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:47:47.896376+0000","last_clean_scrub_stamp":"2024-09-16T15:47:47.896376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T01:13:31.262332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-16T15:47:49.964256+0000","last_change":"2024-09-16T15:47:49.951521+0000","last_active":"2024-09-16T15:47:49.964256+0000","last_peered":"2024-09-16T15:47:49.964256+0000","last_clean":"2024-09-16T15:47:49.964256+0000","last_became_active":"2024-09-16T15:47:49.950245+0000","last_became_peered":"2024-09-16T15:47:49.950245+0000","last_unstale":"2024-09-16T15:47:49.964256+0000","last_undegraded":"2024-09-16T15:47:49.964256+0000","last_fullsized":"2024-09-16T15:47:49.964256+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:47:48.900296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:47:48.900296+0000","last_clean_scrub_stamp":"2024-09-16T15:47:48.900296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:25:16.118135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.0","version":"0'0","reported_seq":13,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-16T15:47:49.924289+0000","last_change":"2024-09-16T15:47:48.951057+0000","last_active":"2024-09-16T15:47:49.924289+0000","last_peered":"2024-09-16T15:47:49.924289+0000","last_clean":"2024-09-16T15:47:49.924289+0000","last_became_active":"2024-09-16T15:47:48.950243+0000","last_became_peered":"2024-09-16T15:47:48.950243+0000","last_unstale":"2024-09-16T15:47:49.924289+0000","last_undegraded":"2024-09-16T15:47:49.924289+0000","last_fullsized":"2024-09-16T15:47:49.924289+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:47:47.896376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:47:47.896376+0000","last_clean_scrub_stamp":"2024-09-16T15:47:47.896376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:23:16.080216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.c","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-16T15:47:49.948898+0000","last_change":"2024-09-16T15:47:49.946744+0000","last_active":"2024-09-16T15:47:49.948898+0000","last_peered":"2024-09-16T15:47:49.948898+0000","last_clean":"2024-09-16T15:47:49.948898+0000","last_became_active":"2024-09-16T15:47:49.946341+0000","last_became_peered":"2024-09-16T15:47:49.946341+0000","last_unstale":"2024-09-16T15:47:49.948898+0000","last_undegraded":"2024-09-16T15:47:49.948898+0000","last_fullsized":"2024-09-16T15:47:49.948898+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:47:48.900296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:47:48.900296+0000","last_clean_scrub_stamp":"2024-09-16T15:47:48.900296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T03:44:33.687212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.d","version":"84'1","reported_seq":14,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-16T15:47:49.969893+0000","last_change":"2024-09-16T15:47:48.955307+0000","last_active":"2024-09-16T15:47:49.969893+0000","last_peered":"2024-09-16T15:47:49.969893+0000","last_clean":"2024-09-16T15:47:49.969893+0000","last_became_active":"2024-09-16T15:47:48.954591+0000","last_became_peered":"2024-09-16T15:47:48.954591+0000","last_unstale":"2024-09-16T15:47:49.969893+0000","last_undegraded":"2024-09-16T15:47:49.969893+0000","last_fullsized":"2024-09-16T15:47:49.969893+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:47:47.896376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:47:47.896376+0000","last_clean_scrub_stamp":"2024-09-16T15:47:47.896376+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:52:27.701367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"73'79","reported_seq":256,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-16T15:47:49.934224+0000","last_change":"2024-09-16T15:42:40.881820+0000","last_active":"2024-09-16T15:47:49.934224+0000","last_peered":"2024-09-16T15:47:49.934224+0000","last_clean":"2024-09-16T15:47:49.934224+0000","last_became_active":"2024-09-16T15:42:40.879965+0000","last_became_peered":"2024-09-16T15:42:40.879965+0000","last_unstale":"2024-09-16T15:47:49.934224+0000","last_undegraded":"2024-09-16T15:47:49.934224+0000","last_fullsized":"2024-09-16T15:47:49.934224+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:42:39.728762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:42:39.728762+0000","last_clean_scrub_stamp":"2024-09-16T15:42:39.728762+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:22:18.768932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":13,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":12,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79,"ondisk_log_size":79,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542210,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32760,"kb_used_data":1572,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339080,"statfs":{"total":96636764160,"available":96603217920,"internally_reserved":0,"allocated":1609728,"data_stored":1024387,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542209,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32184,"kb_used_data":992,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339656,"statfs":{"total":96636764160,"available":96603807744,"internally_reserved":0,"allocated":1015808,"data_stored":434019,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":5,"up_from":14,"seq":60129542210,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28024,"kb_used_data":992,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343816,"statfs":{"total":96636764160,"available":96608067584,"internally_reserved":0,"allocated":1015808,"data_stored":434019,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542208,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32180,"kb_used_data":992,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339660,"statfs":{"total":96636764160,"available":96603811840,"internally_reserved":0,"allocated":1015808,"data_stored":434019,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":3,"up_from":14,"seq":60129542209,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32180,"kb_used_data":992,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339660,"statfs":{"total":96636764160,"available":96603811840,"internally_reserved":0,"allocated":1015808,"data_stored":434019,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":2,"up_from":14,"seq":60129542209,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32184,"kb_used_data":992,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339656,"statfs":{"total":96636764160,"available":96603807744,"internally_reserved":0,"allocated":1015808,"data_stored":434019,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":1,"up_from":14,"seq":60129542211,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28024,"kb_used_data":992,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343816,"statfs":{"total":96636764160,"available":96608067584,"internally_reserved":0,"allocated":1015808,"data_stored":434019,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":0,"up_from":14,"seq":60129542211,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32760,"kb_used_data":1572,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339080,"statfs":{"total":96636764160,"available":96603217920,"internally_reserved":0,"allocated":1609728,"data_stored":1024387,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":194,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T15:47:54.355 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-09-16T15:47:54.355 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-09-16T15:47:54.521 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:54.515+0000 7f105925c640 1 -- 172.21.15.123:0/2326359225 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1054072c70 msgr2=0x7f10540cc3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:47:54.522 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:54.515+0000 7f105925c640 1 --2- 172.21.15.123:0/2326359225 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1054072c70 0x7f10540cc3c0 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f1048009900 tx=0x7f104801c820 comp rx=0 tx=0).stop 2024-09-16T15:47:54.522 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:54.515+0000 7f105925c640 1 -- 172.21.15.123:0/2326359225 shutdown_connections 2024-09-16T15:47:54.522 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:54.515+0000 7f105925c640 1 --2- 172.21.15.123:0/2326359225 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1054072c70 0x7f10540cc3c0 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:54.522 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:54.515+0000 7f105925c640 1 --2- 172.21.15.123:0/2326359225 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f105415a9d0 0x7f105406ebe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:54.522 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:54.515+0000 7f105925c640 1 -- 172.21.15.123:0/2326359225 >> 172.21.15.123:0/2326359225 conn(0x7f10540b1710 msgr2=0x7f10540b1b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:47:54.522 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:54.515+0000 7f105925c640 1 -- 172.21.15.123:0/2326359225 shutdown_connections 2024-09-16T15:47:54.522 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:54.515+0000 7f105925c640 1 -- 172.21.15.123:0/2326359225 wait complete. 2024-09-16T15:47:54.523 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:54.519+0000 7f105925c640 1 Processor -- start 2024-09-16T15:47:54.523 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:54.519+0000 7f105925c640 1 -- start start 2024-09-16T15:47:54.525 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:54.519+0000 7f105925c640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1054072720 0x7f105414c920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:47:54.525 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:54.519+0000 7f105925c640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f1054072c70 0x7f10540b5270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:47:54.525 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:54.519+0000 7f105925c640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f105415a9d0 0x7f1054152dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:47:54.525 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:54.519+0000 7f105925c640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f10540ce0a0 con 0x7f105415a9d0 2024-09-16T15:47:54.525 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:54.519+0000 7f105925c640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f10540cdf20 con 0x7f1054072720 2024-09-16T15:47:54.525 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:54.519+0000 7f105925c640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f10540ce220 con 0x7f1054072c70 2024-09-16T15:47:54.525 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:54.519+0000 7f105359e640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f105415a9d0 0x7f1054152dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:47:54.526 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:54.519+0000 7f105359e640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f105415a9d0 0x7f1054152dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:43840/0 (socket says 172.21.15.123:43840) 2024-09-16T15:47:54.526 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:54.519+0000 7f105359e640 1 -- 172.21.15.123:0/3179772241 learned_addr learned my addr 172.21.15.123:0/3179772241 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:47:54.526 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:54.519+0000 7f105259c640 1 --2- 172.21.15.123:0/3179772241 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f1054072c70 0x7f10540b5270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:47:54.526 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:54.519+0000 7f1052d9d640 1 --2- 172.21.15.123:0/3179772241 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1054072720 0x7f105414c920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:47:54.526 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:54.519+0000 7f105359e640 1 -- 172.21.15.123:0/3179772241 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f1054072c70 msgr2=0x7f10540b5270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:47:54.526 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:54.519+0000 7f105359e640 1 --2- 172.21.15.123:0/3179772241 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f1054072c70 0x7f10540b5270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:54.526 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:54.519+0000 7f105359e640 1 -- 172.21.15.123:0/3179772241 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1054072720 msgr2=0x7f105414c920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:47:54.526 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:54.519+0000 7f105359e640 1 --2- 172.21.15.123:0/3179772241 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1054072720 0x7f105414c920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:54.526 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:54.519+0000 7f105359e640 1 -- 172.21.15.123:0/3179772241 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f10540b5ac0 con 0x7f105415a9d0 2024-09-16T15:47:54.526 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:54.519+0000 7f105359e640 1 --2- 172.21.15.123:0/3179772241 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f105415a9d0 0x7f1054152dd0 secure :-1 s=READY pgs=949 cs=0 l=1 rev1=1 crypto rx=0x7f104c00c730 tx=0x7f104c00cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:47:54.526 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:54.519+0000 7f103b7fe640 1 -- 172.21.15.123:0/3179772241 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f104c003b80 con 0x7f105415a9d0 2024-09-16T15:47:54.526 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:54.519+0000 7f103b7fe640 1 -- 172.21.15.123:0/3179772241 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f104c003d20 con 0x7f105415a9d0 2024-09-16T15:47:54.527 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:54.519+0000 7f105925c640 1 -- 172.21.15.123:0/3179772241 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f105414d9e0 con 0x7f105415a9d0 2024-09-16T15:47:54.527 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:54.519+0000 7f103b7fe640 1 -- 172.21.15.123:0/3179772241 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f104c00aab0 con 0x7f105415a9d0 2024-09-16T15:47:54.527 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:54.519+0000 7f105925c640 1 -- 172.21.15.123:0/3179772241 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f105414dfa0 con 0x7f105415a9d0 2024-09-16T15:47:54.527 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:54.519+0000 7f105925c640 1 -- 172.21.15.123:0/3179772241 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f10540cc3c0 con 0x7f105415a9d0 2024-09-16T15:47:54.527 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:54.523+0000 7f103b7fe640 1 -- 172.21.15.123:0/3179772241 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 16) ==== 97182+0+0 (secure 0 0 0) 0x7f104c00ac50 con 0x7f105415a9d0 2024-09-16T15:47:54.528 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:54.523+0000 7f103b7fe640 1 --2- 172.21.15.123:0/3179772241 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f1040073010 0x7f10400754d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:47:54.528 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:54.523+0000 7f103b7fe640 1 -- 172.21.15.123:0/3179772241 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 5709+0+0 (secure 0 0 0) 0x7f104c093d40 con 0x7f105415a9d0 2024-09-16T15:47:54.528 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:54.523+0000 7f1052d9d640 1 --2- 172.21.15.123:0/3179772241 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f1040073010 0x7f10400754d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:47:54.528 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:54.523+0000 7f1052d9d640 1 --2- 172.21.15.123:0/3179772241 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f1040073010 0x7f10400754d0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f104400b810 tx=0x7f104400d040 comp rx=0 tx=0).ready entity=mgr.4150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:47:54.531 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:54.527+0000 7f103b7fe640 1 -- 172.21.15.123:0/3179772241 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f104c05ee00 con 0x7f105415a9d0 2024-09-16T15:47:54.746 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:54.743+0000 7f105925c640 1 -- 172.21.15.123:0/3179772241 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) -- 0x7f10540c6ca0 con 0x7f105415a9d0 2024-09-16T15:47:54.750 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:54.747+0000 7f103b7fe640 1 -- 172.21.15.123:0/3179772241 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v23) ==== 190+0+147 (secure 0 0 0) 0x7f104c05e700 con 0x7f105415a9d0 2024-09-16T15:47:54.751 INFO:teuthology.orchestra.run.smithi123.stdout:[client.0] 2024-09-16T15:47:54.751 INFO:teuthology.orchestra.run.smithi123.stdout: key = AQC1Uehm7GDTAhAAHWWVIyjVVnTeeozBR8DfFA== 2024-09-16T15:47:54.751 INFO:teuthology.orchestra.run.smithi123.stdout: caps mds = "allow" 2024-09-16T15:47:54.751 INFO:teuthology.orchestra.run.smithi123.stdout: caps mon = "allow r" 2024-09-16T15:47:54.751 INFO:teuthology.orchestra.run.smithi123.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-09-16T15:47:54.751 INFO:teuthology.orchestra.run.smithi123.stderr:updated caps for client.0 2024-09-16T15:47:54.753 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:54.747+0000 7f105925c640 1 -- 172.21.15.123:0/3179772241 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f1040073010 msgr2=0x7f10400754d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:47:54.754 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:54.747+0000 7f105925c640 1 --2- 172.21.15.123:0/3179772241 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f1040073010 0x7f10400754d0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f104400b810 tx=0x7f104400d040 comp rx=0 tx=0).stop 2024-09-16T15:47:54.754 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:54.747+0000 7f105925c640 1 -- 172.21.15.123:0/3179772241 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f105415a9d0 msgr2=0x7f1054152dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:47:54.754 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:54.747+0000 7f105925c640 1 --2- 172.21.15.123:0/3179772241 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f105415a9d0 0x7f1054152dd0 secure :-1 s=READY pgs=949 cs=0 l=1 rev1=1 crypto rx=0x7f104c00c730 tx=0x7f104c00cbf0 comp rx=0 tx=0).stop 2024-09-16T15:47:54.754 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:54.747+0000 7f105925c640 1 -- 172.21.15.123:0/3179772241 shutdown_connections 2024-09-16T15:47:54.754 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:54.747+0000 7f105925c640 1 --2- 172.21.15.123:0/3179772241 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f1040073010 0x7f10400754d0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:54.754 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:54.747+0000 7f105925c640 1 --2- 172.21.15.123:0/3179772241 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f105415a9d0 0x7f1054152dd0 unknown :-1 s=CLOSED pgs=949 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:54.754 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:54.747+0000 7f105925c640 1 --2- 172.21.15.123:0/3179772241 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f1054072c70 0x7f10540b5270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:54.754 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:54.747+0000 7f105925c640 1 --2- 172.21.15.123:0/3179772241 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1054072720 0x7f105414c920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:54.754 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:54.747+0000 7f105925c640 1 -- 172.21.15.123:0/3179772241 >> 172.21.15.123:0/3179772241 conn(0x7f10540b1710 msgr2=0x7f1054154a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:47:54.754 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:54.747+0000 7f105925c640 1 -- 172.21.15.123:0/3179772241 shutdown_connections 2024-09-16T15:47:54.754 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:54.747+0000 7f105925c640 1 -- 172.21.15.123:0/3179772241 wait complete. 2024-09-16T15:47:54.769 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-16T15:47:54.894 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:54.887+0000 7f7b93015640 1 -- 172.21.15.123:0/1705078479 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7b8c158120 msgr2=0x7f7b8c0ca320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:47:54.894 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:54.887+0000 7f7b93015640 1 --2- 172.21.15.123:0/1705078479 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7b8c158120 0x7f7b8c0ca320 secure :-1 s=READY pgs=950 cs=0 l=1 rev1=1 crypto rx=0x7f7b8c06df40 tx=0x7f7b7c01cd10 comp rx=0 tx=0).stop 2024-09-16T15:47:54.894 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:54.887+0000 7f7b93015640 1 -- 172.21.15.123:0/1705078479 shutdown_connections 2024-09-16T15:47:54.894 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:54.887+0000 7f7b93015640 1 --2- 172.21.15.123:0/1705078479 >> v2:172.21.15.184:3300/0 conn(0x7f7b8c06f5f0 0x7f7b8c15aa60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:54.894 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:54.887+0000 7f7b93015640 1 --2- 172.21.15.123:0/1705078479 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7b8c158120 0x7f7b8c0ca320 unknown :-1 s=CLOSED pgs=950 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:54.894 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:54.887+0000 7f7b93015640 1 -- 172.21.15.123:0/1705078479 >> 172.21.15.123:0/1705078479 conn(0x7f7b8c0cfb10 msgr2=0x7f7b8c0cff20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:47:54.894 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:54.887+0000 7f7b93015640 1 -- 172.21.15.123:0/1705078479 shutdown_connections 2024-09-16T15:47:54.895 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:54.887+0000 7f7b93015640 1 -- 172.21.15.123:0/1705078479 wait complete. 2024-09-16T15:47:54.897 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:54.891+0000 7f7b93015640 1 Processor -- start 2024-09-16T15:47:54.897 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:54.891+0000 7f7b93015640 1 -- start start 2024-09-16T15:47:54.897 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:54.891+0000 7f7b93015640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f7b8c06f5f0 0x7f7b8c151580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:47:54.897 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:54.891+0000 7f7b93015640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7b8c0cf5a0 0x7f7b8c1565d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:47:54.897 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:54.891+0000 7f7b93015640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7b8c158120 0x7f7b8c14f090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:47:54.897 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:54.891+0000 7f7b93015640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f7b8c15b710 con 0x7f7b8c0cf5a0 2024-09-16T15:47:54.897 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:54.891+0000 7f7b93015640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f7b8c15b590 con 0x7f7b8c158120 2024-09-16T15:47:54.897 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:54.891+0000 7f7b93015640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f7b8c0b4950 con 0x7f7b8c06f5f0 2024-09-16T15:47:54.897 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:54.891+0000 7f7b8bfff640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7b8c0cf5a0 0x7f7b8c1565d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:47:54.897 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:54.891+0000 7f7b8bfff640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7b8c0cf5a0 0x7f7b8c1565d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:43858/0 (socket says 172.21.15.123:43858) 2024-09-16T15:47:54.898 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:54.891+0000 7f7b8bfff640 1 -- 172.21.15.123:0/2059394435 learned_addr learned my addr 172.21.15.123:0/2059394435 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:47:54.898 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:54.891+0000 7f7b915b2640 1 --2- 172.21.15.123:0/2059394435 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7b8c158120 0x7f7b8c14f090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:47:54.898 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:54.891+0000 7f7b90db1640 1 --2- 172.21.15.123:0/2059394435 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f7b8c06f5f0 0x7f7b8c151580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:47:54.898 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:54.891+0000 7f7b8bfff640 1 -- 172.21.15.123:0/2059394435 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f7b8c06f5f0 msgr2=0x7f7b8c151580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:47:54.898 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:54.891+0000 7f7b8bfff640 1 --2- 172.21.15.123:0/2059394435 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f7b8c06f5f0 0x7f7b8c151580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:54.898 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:54.891+0000 7f7b8bfff640 1 -- 172.21.15.123:0/2059394435 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7b8c158120 msgr2=0x7f7b8c14f090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:47:54.898 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:54.891+0000 7f7b8bfff640 1 --2- 172.21.15.123:0/2059394435 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7b8c158120 0x7f7b8c14f090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:54.898 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:54.891+0000 7f7b8bfff640 1 -- 172.21.15.123:0/2059394435 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7b8c0c3620 con 0x7f7b8c0cf5a0 2024-09-16T15:47:54.898 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:54.891+0000 7f7b915b2640 1 --2- 172.21.15.123:0/2059394435 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7b8c158120 0x7f7b8c14f090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T15:47:54.898 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:54.891+0000 7f7b8bfff640 1 --2- 172.21.15.123:0/2059394435 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7b8c0cf5a0 0x7f7b8c1565d0 secure :-1 s=READY pgs=951 cs=0 l=1 rev1=1 crypto rx=0x7f7b7400c490 tx=0x7f7b74001e80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:47:54.898 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:54.891+0000 7f7b897fa640 1 -- 172.21.15.123:0/2059394435 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b74017070 con 0x7f7b8c0cf5a0 2024-09-16T15:47:54.898 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:54.891+0000 7f7b897fa640 1 -- 172.21.15.123:0/2059394435 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f7b74003600 con 0x7f7b8c0cf5a0 2024-09-16T15:47:54.899 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:54.891+0000 7f7b93015640 1 -- 172.21.15.123:0/2059394435 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7b8c0b4b80 con 0x7f7b8c0cf5a0 2024-09-16T15:47:54.899 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:54.891+0000 7f7b897fa640 1 -- 172.21.15.123:0/2059394435 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b74003a50 con 0x7f7b8c0cf5a0 2024-09-16T15:47:54.899 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:54.891+0000 7f7b93015640 1 -- 172.21.15.123:0/2059394435 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7b8c0b5140 con 0x7f7b8c0cf5a0 2024-09-16T15:47:54.900 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:54.891+0000 7f7b93015640 1 -- 172.21.15.123:0/2059394435 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7b8c15aa60 con 0x7f7b8c0cf5a0 2024-09-16T15:47:54.900 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:54.895+0000 7f7b897fa640 1 -- 172.21.15.123:0/2059394435 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 16) ==== 97182+0+0 (secure 0 0 0) 0x7f7b7401a070 con 0x7f7b8c0cf5a0 2024-09-16T15:47:54.900 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:54.895+0000 7f7b897fa640 1 --2- 172.21.15.123:0/2059394435 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f7b6c072fe0 0x7f7b6c0754a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:47:54.900 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:54.895+0000 7f7b897fa640 1 -- 172.21.15.123:0/2059394435 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 5709+0+0 (secure 0 0 0) 0x7f7b740933d0 con 0x7f7b8c0cf5a0 2024-09-16T15:47:54.900 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:54.895+0000 7f7b90db1640 1 --2- 172.21.15.123:0/2059394435 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f7b6c072fe0 0x7f7b6c0754a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:47:54.900 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:54.895+0000 7f7b90db1640 1 --2- 172.21.15.123:0/2059394435 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f7b6c072fe0 0x7f7b6c0754a0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f7b7c0af000 tx=0x7f7b7c01b550 comp rx=0 tx=0).ready entity=mgr.4150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:47:54.903 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:54.899+0000 7f7b897fa640 1 -- 172.21.15.123:0/2059394435 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f7b7405e400 con 0x7f7b8c0cf5a0 2024-09-16T15:47:55.100 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:55.095+0000 7f7b93015640 1 -- 172.21.15.123:0/2059394435 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f7b8c0d06c0 con 0x7f7b8c0cf5a0 2024-09-16T15:47:55.100 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:55.095+0000 7f7b897fa640 1 -- 172.21.15.123:0/2059394435 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 53 v53) ==== 94+0+5237 (secure 0 0 0) 0x7f7b7405dd00 con 0x7f7b8c0cf5a0 2024-09-16T15:47:55.101 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-09-16T15:47:55.101 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":53,"btime":"2024-09-16T15:47:52:957004+0000","default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5677,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6835/1609511477","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6833","nonce":1609511477},{"type":"v1","addr":"172.21.15.184:6835","nonce":1609511477}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":48},{"gid":6672,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6834/3658721718","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6832","nonce":3658721718},{"type":"v1","addr":"172.21.15.123:6834","nonce":3658721718}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":48},{"gid":6675,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6835/2536398045","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6833","nonce":2536398045},{"type":"v1","addr":"172.21.15.123:6835","nonce":2536398045}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":48}],"filesystems":[{"mdsmap":{"epoch":53,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-16T15:47:49.295797+0000","modified":"2024-09-16T15:47:52.333498+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6684},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6684":{"gid":6684,"name":"b","rank":0,"incarnation":49,"state":"up:active","state_seq":3,"addr":"172.21.15.184:6834/1546532515","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":1546532515},{"type":"v1","addr":"172.21.15.184:6834","nonce":1546532515}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6684,"qdb_cluster":[6684]},"id":6}]} 2024-09-16T15:47:55.101 INFO:teuthology.orchestra.run.smithi123.stderr:dumped fsmap epoch 53 2024-09-16T15:47:55.103 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:55.099+0000 7f7b93015640 1 -- 172.21.15.123:0/2059394435 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f7b6c072fe0 msgr2=0x7f7b6c0754a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:47:55.103 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:55.099+0000 7f7b93015640 1 --2- 172.21.15.123:0/2059394435 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f7b6c072fe0 0x7f7b6c0754a0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f7b7c0af000 tx=0x7f7b7c01b550 comp rx=0 tx=0).stop 2024-09-16T15:47:55.104 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:55.099+0000 7f7b93015640 1 -- 172.21.15.123:0/2059394435 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7b8c0cf5a0 msgr2=0x7f7b8c1565d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:47:55.104 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:55.099+0000 7f7b93015640 1 --2- 172.21.15.123:0/2059394435 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7b8c0cf5a0 0x7f7b8c1565d0 secure :-1 s=READY pgs=951 cs=0 l=1 rev1=1 crypto rx=0x7f7b7400c490 tx=0x7f7b74001e80 comp rx=0 tx=0).stop 2024-09-16T15:47:55.104 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:55.099+0000 7f7b93015640 1 -- 172.21.15.123:0/2059394435 shutdown_connections 2024-09-16T15:47:55.104 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:55.099+0000 7f7b93015640 1 --2- 172.21.15.123:0/2059394435 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f7b6c072fe0 0x7f7b6c0754a0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:55.104 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:55.099+0000 7f7b93015640 1 --2- 172.21.15.123:0/2059394435 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7b8c158120 0x7f7b8c14f090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:55.104 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:55.099+0000 7f7b93015640 1 --2- 172.21.15.123:0/2059394435 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7b8c0cf5a0 0x7f7b8c1565d0 unknown :-1 s=CLOSED pgs=951 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:55.104 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:55.099+0000 7f7b93015640 1 --2- 172.21.15.123:0/2059394435 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f7b8c06f5f0 0x7f7b8c151580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:55.104 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:55.099+0000 7f7b93015640 1 -- 172.21.15.123:0/2059394435 >> 172.21.15.123:0/2059394435 conn(0x7f7b8c0cfb10 msgr2=0x7f7b8c15c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:47:55.104 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:55.099+0000 7f7b93015640 1 -- 172.21.15.123:0/2059394435 shutdown_connections 2024-09-16T15:47:55.104 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:55.099+0000 7f7b93015640 1 -- 172.21.15.123:0/2059394435 wait complete. 2024-09-16T15:47:55.222 INFO:tasks.ceph.mds.b.smithi184.stderr:2024-09-16T15:47:55.221+0000 7f2276feb640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-09-16T15:47:55.521 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 53, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-09-16T15:47:49.295797+0000', 'modified': '2024-09-16T15:47:52.333498+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 6684}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6684': {'gid': 6684, 'name': 'b', 'rank': 0, 'incarnation': 49, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.184:6834/1546532515', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.184:6832', 'nonce': 1546532515}, {'type': 'v1', 'addr': '172.21.15.184:6834', 'nonce': 1546532515}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540719139924082687, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [13], 'metadata_pool': 12, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 6684, 'qdb_cluster': [6684]} 2024-09-16T15:47:55.522 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-09-16T15:47:55.522 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-09-16T15:47:55.656 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:55.651+0000 7fcddec9b640 1 -- 172.21.15.123:0/1431605322 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcdd80cddf0 msgr2=0x7fcdd8162470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:47:55.656 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:55.651+0000 7fcddec9b640 1 --2- 172.21.15.123:0/1431605322 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcdd80cddf0 0x7fcdd8162470 secure :-1 s=READY pgs=953 cs=0 l=1 rev1=1 crypto rx=0x7fcdcc009900 tx=0x7fcdcc01c8e0 comp rx=0 tx=0).stop 2024-09-16T15:47:55.656 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:55.651+0000 7fcddec9b640 1 -- 172.21.15.123:0/1431605322 shutdown_connections 2024-09-16T15:47:55.656 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:55.651+0000 7fcddec9b640 1 --2- 172.21.15.123:0/1431605322 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcdd80cddf0 0x7fcdd8162470 unknown :-1 s=CLOSED pgs=953 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:55.656 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:55.651+0000 7fcddec9b640 1 -- 172.21.15.123:0/1431605322 >> 172.21.15.123:0/1431605322 conn(0x7fcdd80cfb10 msgr2=0x7fcdd80cff20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:47:55.657 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:55.651+0000 7fcddec9b640 1 -- 172.21.15.123:0/1431605322 shutdown_connections 2024-09-16T15:47:55.657 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:55.651+0000 7fcddec9b640 1 -- 172.21.15.123:0/1431605322 wait complete. 2024-09-16T15:47:55.657 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:55.651+0000 7fcddec9b640 1 Processor -- start 2024-09-16T15:47:55.657 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:55.651+0000 7fcddec9b640 1 -- start start 2024-09-16T15:47:55.659 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:55.651+0000 7fcddec9b640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcdd80cddf0 0x7fcdd814ff80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:47:55.659 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:55.651+0000 7fcddec9b640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fcdd815de50 0x7fcdd80c3e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:47:55.659 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:55.651+0000 7fcddec9b640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fcdd815e230 0x7fcdd80c43a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:47:55.659 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:55.651+0000 7fcddec9b640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fcdd816c120 con 0x7fcdd80cddf0 2024-09-16T15:47:55.660 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:55.651+0000 7fcddec9b640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fcdd816bfa0 con 0x7fcdd815de50 2024-09-16T15:47:55.660 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:55.651+0000 7fcddec9b640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fcdd816c2a0 con 0x7fcdd815e230 2024-09-16T15:47:55.660 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:55.655+0000 7fcddca37640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcdd80cddf0 0x7fcdd814ff80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:47:55.660 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:55.655+0000 7fcddca37640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcdd80cddf0 0x7fcdd814ff80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:43866/0 (socket says 172.21.15.123:43866) 2024-09-16T15:47:55.660 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:55.655+0000 7fcddca37640 1 -- 172.21.15.123:0/2736486336 learned_addr learned my addr 172.21.15.123:0/2736486336 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:47:55.660 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:55.655+0000 7fcdd7fff640 1 --2- 172.21.15.123:0/2736486336 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fcdd815de50 0x7fcdd80c3e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:47:55.660 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:55.655+0000 7fcddca37640 1 -- 172.21.15.123:0/2736486336 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fcdd815e230 msgr2=0x7fcdd80c43a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:47:55.660 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:55.655+0000 7fcddd238640 1 --2- 172.21.15.123:0/2736486336 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fcdd815e230 0x7fcdd80c43a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:47:55.660 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:55.655+0000 7fcddca37640 1 --2- 172.21.15.123:0/2736486336 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fcdd815e230 0x7fcdd80c43a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:55.660 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:55.655+0000 7fcddca37640 1 -- 172.21.15.123:0/2736486336 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fcdd815de50 msgr2=0x7fcdd80c3e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:47:55.660 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:55.655+0000 7fcddca37640 1 --2- 172.21.15.123:0/2736486336 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fcdd815de50 0x7fcdd80c3e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:55.660 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:55.655+0000 7fcddca37640 1 -- 172.21.15.123:0/2736486336 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcdd8150840 con 0x7fcdd80cddf0 2024-09-16T15:47:55.661 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:55.655+0000 7fcdd7fff640 1 --2- 172.21.15.123:0/2736486336 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fcdd815de50 0x7fcdd80c3e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T15:47:55.661 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:55.655+0000 7fcddca37640 1 --2- 172.21.15.123:0/2736486336 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcdd80cddf0 0x7fcdd814ff80 secure :-1 s=READY pgs=954 cs=0 l=1 rev1=1 crypto rx=0x7fcdc800c530 tx=0x7fcdc800c9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:47:55.661 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:55.655+0000 7fcdd57fa640 1 -- 172.21.15.123:0/2736486336 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcdc8017070 con 0x7fcdd80cddf0 2024-09-16T15:47:55.661 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:55.655+0000 7fcdd57fa640 1 -- 172.21.15.123:0/2736486336 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fcdc8003600 con 0x7fcdd80cddf0 2024-09-16T15:47:55.661 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:55.655+0000 7fcddec9b640 1 -- 172.21.15.123:0/2736486336 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcdd8160a30 con 0x7fcdd80cddf0 2024-09-16T15:47:55.661 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:55.655+0000 7fcdd57fa640 1 -- 172.21.15.123:0/2736486336 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcdc8003a50 con 0x7fcdd80cddf0 2024-09-16T15:47:55.661 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:55.655+0000 7fcddec9b640 1 -- 172.21.15.123:0/2736486336 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcdd8160ff0 con 0x7fcdd80cddf0 2024-09-16T15:47:55.662 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:55.655+0000 7fcdd57fa640 1 -- 172.21.15.123:0/2736486336 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 16) ==== 97182+0+0 (secure 0 0 0) 0x7fcdc801a070 con 0x7fcdd80cddf0 2024-09-16T15:47:55.662 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:55.655+0000 7fcdd57fa640 1 --2- 172.21.15.123:0/2736486336 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fcdc4072fe0 0x7fcdc40754a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:47:55.662 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:55.655+0000 7fcdd57fa640 1 -- 172.21.15.123:0/2736486336 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 5709+0+0 (secure 0 0 0) 0x7fcdc80933b0 con 0x7fcdd80cddf0 2024-09-16T15:47:55.662 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:55.655+0000 7fcdd7fff640 1 --2- 172.21.15.123:0/2736486336 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fcdc4072fe0 0x7fcdc40754a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:47:55.662 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:55.655+0000 7fcde1d85000 1 Processor -- start 2024-09-16T15:47:55.662 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:55.659+0000 7fcde1d85000 1 -- start start 2024-09-16T15:47:55.663 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:55.659+0000 7fcdd7fff640 1 --2- 172.21.15.123:0/2736486336 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fcdc4072fe0 0x7fcdc40754a0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fcdcc0af000 tx=0x7fcdcc01b120 comp rx=0 tx=0).ready entity=mgr.4150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:47:55.665 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:55.659+0000 7fcde1d85000 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5617dd8fe9d0 0x5617dd8fedb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:47:55.665 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:55.659+0000 7fcde1d85000 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5617dd8ff2f0 0x5617dd8b28d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:47:55.665 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:55.659+0000 7fcde1d85000 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x5617dd8b2e10 0x5617dd8b31f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:47:55.665 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:55.659+0000 7fcde1d85000 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x5617dd7d33f0 con 0x5617dd8ff2f0 2024-09-16T15:47:55.665 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:55.659+0000 7fcde1d85000 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x5617dd710b20 con 0x5617dd8fe9d0 2024-09-16T15:47:55.665 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:55.659+0000 7fcde1d85000 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x5617dd6fff50 con 0x5617dd8b2e10 2024-09-16T15:47:55.665 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:55.659+0000 7fcdd7fff640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5617dd8ff2f0 0x5617dd8b28d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:47:55.665 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:55.659+0000 7fcdd7fff640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5617dd8ff2f0 0x5617dd8b28d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:43870/0 (socket says 172.21.15.123:43870) 2024-09-16T15:47:55.666 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:55.659+0000 7fcdd7fff640 1 -- 172.21.15.123:0/1694040968 learned_addr learned my addr 172.21.15.123:0/1694040968 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:47:55.666 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:55.659+0000 7fcddca37640 1 --2- 172.21.15.123:0/1694040968 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5617dd8fe9d0 0x5617dd8fedb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:47:55.666 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:55.659+0000 7fcddd238640 1 --2- 172.21.15.123:0/1694040968 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x5617dd8b2e10 0x5617dd8b31f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:47:55.666 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:55.659+0000 7fcdd7fff640 1 -- 172.21.15.123:0/1694040968 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x5617dd8b2e10 msgr2=0x5617dd8b31f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T15:47:55.666 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:55.659+0000 7fcdd7fff640 1 --2- 172.21.15.123:0/1694040968 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x5617dd8b2e10 0x5617dd8b31f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:55.666 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:55.659+0000 7fcdd7fff640 1 -- 172.21.15.123:0/1694040968 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5617dd8fe9d0 msgr2=0x5617dd8fedb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T15:47:55.666 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:55.659+0000 7fcdd7fff640 1 --2- 172.21.15.123:0/1694040968 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5617dd8fe9d0 0x5617dd8fedb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:55.666 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:55.659+0000 7fcdd7fff640 1 -- 172.21.15.123:0/1694040968 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5617dd6c7590 con 0x5617dd8ff2f0 2024-09-16T15:47:55.666 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:55.659+0000 7fcdd7fff640 1 --2- 172.21.15.123:0/1694040968 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5617dd8ff2f0 0x5617dd8b28d0 secure :-1 s=READY pgs=955 cs=0 l=1 rev1=1 crypto rx=0x7fcdcc009850 tx=0x7fcdcc0b4000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b16e052916743c0a server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:47:55.666 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:55.659+0000 7fcdbe7fc640 1 -- 172.21.15.123:0/1694040968 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcdcc0c8070 con 0x5617dd8ff2f0 2024-09-16T15:47:55.666 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:55.659+0000 7fcdbe7fc640 1 -- 172.21.15.123:0/1694040968 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fcdcc0051c0 con 0x5617dd8ff2f0 2024-09-16T15:47:55.666 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:55.659+0000 7fcdbe7fc640 1 -- 172.21.15.123:0/1694040968 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcdcc0b2040 con 0x5617dd8ff2f0 2024-09-16T15:47:55.666 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:55.659+0000 7fcde1d85000 1 -- 172.21.15.123:0/1694040968 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5617dd8ff2f0 msgr2=0x5617dd8b28d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:47:55.667 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:55.659+0000 7fcde1d85000 1 --2- 172.21.15.123:0/1694040968 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5617dd8ff2f0 0x5617dd8b28d0 secure :-1 s=READY pgs=955 cs=0 l=1 rev1=1 crypto rx=0x7fcdcc009850 tx=0x7fcdcc0b4000 comp rx=0 tx=0).stop 2024-09-16T15:47:55.667 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:55.659+0000 7fcde1d85000 1 -- 172.21.15.123:0/1694040968 shutdown_connections 2024-09-16T15:47:55.667 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:55.659+0000 7fcde1d85000 1 --2- 172.21.15.123:0/1694040968 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x5617dd8b2e10 0x5617dd8b31f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:55.667 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:55.659+0000 7fcde1d85000 1 --2- 172.21.15.123:0/1694040968 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5617dd8ff2f0 0x5617dd8b28d0 unknown :-1 s=CLOSED pgs=955 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:55.667 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:55.659+0000 7fcde1d85000 1 --2- 172.21.15.123:0/1694040968 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5617dd8fe9d0 0x5617dd8fedb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:55.667 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:55.659+0000 7fcde1d85000 1 -- 172.21.15.123:0/1694040968 >> 172.21.15.123:0/1694040968 conn(0x5617dd79c130 msgr2=0x5617dd8e1e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:47:55.667 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:55.659+0000 7fcde1d85000 1 -- 172.21.15.123:0/1694040968 shutdown_connections 2024-09-16T15:47:55.667 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:55.659+0000 7fcde1d85000 1 -- 172.21.15.123:0/1694040968 wait complete. 2024-09-16T15:47:55.667 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:55.659+0000 7fcde1d85000 1 -- start start 2024-09-16T15:47:55.667 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:55.659+0000 7fcde1d85000 1 Processor -- start 2024-09-16T15:47:55.670 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:55.659+0000 7fcde1d85000 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x5617dd8b2e10 0x5617dd8b4240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:47:55.671 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:55.659+0000 7fcde1d85000 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5617dd8fe9d0 0x5617dd8b4780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:47:55.671 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:55.659+0000 7fcde1d85000 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5617dd8ff2f0 0x5617dd8b4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:47:55.671 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:55.659+0000 7fcde1d85000 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x5617dd69a5d0 con 0x5617dd8fe9d0 2024-09-16T15:47:55.671 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:55.659+0000 7fcde1d85000 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x5617dd703d60 con 0x5617dd8ff2f0 2024-09-16T15:47:55.671 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:55.659+0000 7fcde1d85000 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x5617dd6a6500 con 0x5617dd8b2e10 2024-09-16T15:47:55.671 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:55.659+0000 7fcde1d85000 10 client.0 _ll_register_callbacks cb 0x5617dd6aa810 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T15:47:55.671 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:55.659+0000 7fcdd7fff640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5617dd8fe9d0 0x5617dd8b4780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:47:55.671 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:55.659+0000 7fcdd7fff640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5617dd8fe9d0 0x5617dd8b4780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:43878/0 (socket says 172.21.15.123:43878) 2024-09-16T15:47:55.671 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:55.659+0000 7fcdd7fff640 1 -- 172.21.15.123:0/4136086434 learned_addr learned my addr 172.21.15.123:0/4136086434 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:47:55.671 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:55.663+0000 7fcdd7fff640 1 -- 172.21.15.123:0/4136086434 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x5617dd8b2e10 msgr2=0x5617dd8b4240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T15:47:55.671 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:55.663+0000 7fcdd7fff640 1 --2- 172.21.15.123:0/4136086434 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x5617dd8b2e10 0x5617dd8b4240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:55.671 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:55.663+0000 7fcddd238640 1 --2- 172.21.15.123:0/4136086434 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5617dd8ff2f0 0x5617dd8b4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:47:55.672 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:55.663+0000 7fcdd7fff640 1 -- 172.21.15.123:0/4136086434 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5617dd8ff2f0 msgr2=0x5617dd8b4cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T15:47:55.672 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:55.663+0000 7fcdd7fff640 1 --2- 172.21.15.123:0/4136086434 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5617dd8ff2f0 0x5617dd8b4cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:55.672 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:55.663+0000 7fcdd7fff640 1 -- 172.21.15.123:0/4136086434 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5617dd69e170 con 0x5617dd8fe9d0 2024-09-16T15:47:55.672 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:55.663+0000 7fcdd7fff640 1 --2- 172.21.15.123:0/4136086434 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5617dd8fe9d0 0x5617dd8b4780 secure :-1 s=READY pgs=956 cs=0 l=1 rev1=1 crypto rx=0x7fcdcc009850 tx=0x7fcdcc0056f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4043eeb4d6caa631 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:47:55.672 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:55.663+0000 7fcdb77fe640 10 client.0 ms_handle_connect on v2:172.21.15.123:3300/0 2024-09-16T15:47:55.672 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:55.663+0000 7fcdb77fe640 1 -- 172.21.15.123:0/4136086434 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcdcc0c8070 con 0x5617dd8fe9d0 2024-09-16T15:47:55.672 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:55.663+0000 7fcdb77fe640 1 -- 172.21.15.123:0/4136086434 <== mon.0 v2:172.21.15.123:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 5709+0+0 (secure 0 0 0) 0x7fcdcc0cdab0 con 0x5617dd8fe9d0 2024-09-16T15:47:55.672 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:55.663+0000 7fcde1d85000 1 -- 172.21.15.123:0/4136086434 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x5617dd6980f0 con 0x5617dd8fe9d0 2024-09-16T15:47:55.672 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:55.663+0000 7fcdb77fe640 1 -- 172.21.15.123:0/4136086434 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_get_version_reply(handle=1 version=53) ==== 24+0+0 (secure 0 0 0) 0x7fcdcc0b2050 con 0x5617dd8fe9d0 2024-09-16T15:47:55.672 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:55.663+0000 7fcde1d85000 10 client.6801 fetch_fsmap learned FSMap version 53 2024-09-16T15:47:55.672 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:55.663+0000 7fcde1d85000 1 -- 172.21.15.123:0/4136086434 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,fsmap=53,monmap=2+}) -- 0x5617dd784ef0 con 0x5617dd8fe9d0 2024-09-16T15:47:55.672 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:55.663+0000 7fcdb77fe640 1 -- 172.21.15.123:0/4136086434 <== mon.0 v2:172.21.15.123:3300/0 4 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x5617dd6980f0 con 0x5617dd8fe9d0 2024-09-16T15:47:55.673 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:55.663+0000 7fcdb77fe640 1 -- 172.21.15.123:0/4136086434 <== mon.0 v2:172.21.15.123:3300/0 5 ==== fsmap(e 53) ==== 3502+0+0 (secure 0 0 0) 0x7fcdcc0b5030 con 0x5617dd8fe9d0 2024-09-16T15:47:55.673 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:55.663+0000 7fcde1d85000 10 client.6801 fetch_fsmap finished waiting for FSMap version 53 2024-09-16T15:47:55.673 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:55.663+0000 7fcde1d85000 10 client.6801 resolve_mds: resolved name 'b' to daemon mds.b 2024-09-16T15:47:55.673 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:55.663+0000 7fcde1d85000 20 client.6801 populate_metadata read hostname 'smithi123' 2024-09-16T15:47:55.673 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:55.663+0000 7fcde1d85000 1 --2- 172.21.15.123:0/4136086434 >> [v2:172.21.15.184:6832/1546532515,v1:172.21.15.184:6834/1546532515] conn(0x5617dd90dc70 0x5617dd910060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:47:55.673 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:55.663+0000 7fcde1d85000 4 client.6801 mds_command: new command op to 6684 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-16T15:47:55.673 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:55.663+0000 7fcde1d85000 1 -- 172.21.15.123:0/4136086434 --> [v2:172.21.15.184:6832/1546532515,v1:172.21.15.184:6834/1546532515] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5617dd712a70 con 0x5617dd90dc70 2024-09-16T15:47:55.673 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:55.663+0000 7fcddca37640 1 --2- 172.21.15.123:0/4136086434 >> [v2:172.21.15.184:6832/1546532515,v1:172.21.15.184:6834/1546532515] conn(0x5617dd90dc70 0x5617dd910060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:47:55.673 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:55.663+0000 7fcdb77fe640 10 client.6801 ms_handle_connect on v2:172.21.15.184:6832/1546532515 2024-09-16T15:47:55.673 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:55.663+0000 7fcddca37640 1 --2- 172.21.15.123:0/4136086434 >> [v2:172.21.15.184:6832/1546532515,v1:172.21.15.184:6834/1546532515] conn(0x5617dd90dc70 0x5617dd910060 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9418c1957a39d92b server_cookie=1c4de6fd8bb8dc76 in_seq=0 out_seq=0 2024-09-16T15:47:55.673 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:55.663+0000 7fcdb77fe640 1 -- 172.21.15.123:0/4136086434 <== mds.0 v2:172.21.15.184:6832/1546532515 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x5617dd712a70 con 0x5617dd90dc70 2024-09-16T15:47:55.673 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:55.663+0000 7fcdb77fe640 10 client.6801 handle_command_reply: tid=0 2024-09-16T15:47:55.673 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:55.663+0000 7fcde1d85000 1 client.6801 shutdown 2024-09-16T15:47:55.673 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:55.663+0000 7fcde1d85000 1 -- 172.21.15.123:0/4136086434 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5617dd8fe9d0 msgr2=0x5617dd8b4780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:47:55.674 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:55.663+0000 7fcde1d85000 1 --2- 172.21.15.123:0/4136086434 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5617dd8fe9d0 0x5617dd8b4780 secure :-1 s=READY pgs=956 cs=0 l=1 rev1=1 crypto rx=0x7fcdcc009850 tx=0x7fcdcc0056f0 comp rx=0 tx=0).stop 2024-09-16T15:47:55.674 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:55.663+0000 7fcde1d85000 1 -- 172.21.15.123:0/4136086434 shutdown_connections 2024-09-16T15:47:55.674 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:55.663+0000 7fcde1d85000 1 --2- 172.21.15.123:0/4136086434 >> [v2:172.21.15.184:6832/1546532515,v1:172.21.15.184:6834/1546532515] conn(0x5617dd90dc70 0x5617dd910060 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:55.674 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:55.663+0000 7fcde1d85000 1 --2- 172.21.15.123:0/4136086434 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5617dd8ff2f0 0x5617dd8b4cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:55.674 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:55.663+0000 7fcdb77fe640 0 client.6801 ms_handle_reset on v2:172.21.15.184:6832/1546532515 2024-09-16T15:47:55.674 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:55.663+0000 7fcde1d85000 1 --2- 172.21.15.123:0/4136086434 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5617dd8fe9d0 0x5617dd8b4780 unknown :-1 s=CLOSED pgs=956 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:55.674 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:55.663+0000 7fcde1d85000 1 --2- 172.21.15.123:0/4136086434 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x5617dd8b2e10 0x5617dd8b4240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:55.674 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:55.663+0000 7fcde1d85000 1 -- 172.21.15.123:0/4136086434 >> 172.21.15.123:0/4136086434 conn(0x5617dd79c130 msgr2=0x5617dd8b1fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:47:55.674 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:55.663+0000 7fcde1d85000 1 -- 172.21.15.123:0/4136086434 shutdown_connections 2024-09-16T15:47:55.674 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:55.663+0000 7fcde1d85000 1 -- 172.21.15.123:0/4136086434 wait complete. 2024-09-16T15:47:55.674 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:55.667+0000 7fcde1d85000 20 client.6801 trim_cache size 0 max 16384 2024-09-16T15:47:55.689 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:55.683+0000 7fcde1d85000 1 Processor -- start 2024-09-16T15:47:55.689 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:55.683+0000 7fcde1d85000 1 -- start start 2024-09-16T15:47:55.691 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:55.683+0000 7fcde1d85000 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x5617dd8b2e10 0x5617dd970c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:47:55.691 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:55.683+0000 7fcde1d85000 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5617dd8fe9d0 0x5617dd971140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:47:55.691 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:55.683+0000 7fcde1d85000 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5617dd8ff2f0 0x5617dd971680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:47:55.691 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:55.683+0000 7fcde1d85000 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fcdd8152940 con 0x5617dd8fe9d0 2024-09-16T15:47:55.691 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:55.683+0000 7fcde1d85000 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fcdd8152f90 con 0x5617dd8ff2f0 2024-09-16T15:47:55.692 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:55.683+0000 7fcde1d85000 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x5617dd708390 con 0x5617dd8b2e10 2024-09-16T15:47:55.692 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:55.683+0000 7fcdd7fff640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5617dd8fe9d0 0x5617dd971140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:47:55.692 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:55.683+0000 7fcdd7fff640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5617dd8fe9d0 0x5617dd971140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:43880/0 (socket says 172.21.15.123:43880) 2024-09-16T15:47:55.692 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:55.683+0000 7fcdd7fff640 1 -- 172.21.15.123:0/1120583102 learned_addr learned my addr 172.21.15.123:0/1120583102 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:47:55.692 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:55.683+0000 7fcdd7fff640 1 -- 172.21.15.123:0/1120583102 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x5617dd8b2e10 msgr2=0x5617dd970c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T15:47:55.692 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:55.683+0000 7fcdd7fff640 1 --2- 172.21.15.123:0/1120583102 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x5617dd8b2e10 0x5617dd970c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:55.692 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:55.683+0000 7fcdd7fff640 1 -- 172.21.15.123:0/1120583102 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5617dd8ff2f0 msgr2=0x5617dd971680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T15:47:55.692 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:55.683+0000 7fcdd7fff640 1 --2- 172.21.15.123:0/1120583102 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5617dd8ff2f0 0x5617dd971680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:55.692 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:55.683+0000 7fcdd7fff640 1 -- 172.21.15.123:0/1120583102 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5617dd75bd90 con 0x5617dd8fe9d0 2024-09-16T15:47:55.692 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:55.687+0000 7fcdd7fff640 1 --2- 172.21.15.123:0/1120583102 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5617dd8fe9d0 0x5617dd971140 secure :-1 s=READY pgs=957 cs=0 l=1 rev1=1 crypto rx=0x7fcdcc01c710 tx=0x7fcdcc004320 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aaac667e0e4ad6ae server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:47:55.692 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:55.687+0000 7fcdbdffb640 1 -- 172.21.15.123:0/1120583102 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcdcc0c8070 con 0x5617dd8fe9d0 2024-09-16T15:47:55.693 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:55.687+0000 7fcdbdffb640 1 -- 172.21.15.123:0/1120583102 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fcdcc0cdd50 con 0x5617dd8fe9d0 2024-09-16T15:47:55.693 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:55.687+0000 7fcdbdffb640 1 -- 172.21.15.123:0/1120583102 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcdcc0a59e0 con 0x5617dd8fe9d0 2024-09-16T15:47:55.693 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:55.687+0000 7fcde1d85000 1 -- 172.21.15.123:0/1120583102 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5617dd8fe9d0 msgr2=0x5617dd971140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:47:55.693 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:55.687+0000 7fcde1d85000 1 --2- 172.21.15.123:0/1120583102 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5617dd8fe9d0 0x5617dd971140 secure :-1 s=READY pgs=957 cs=0 l=1 rev1=1 crypto rx=0x7fcdcc01c710 tx=0x7fcdcc004320 comp rx=0 tx=0).stop 2024-09-16T15:47:55.693 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:55.687+0000 7fcde1d85000 1 -- 172.21.15.123:0/1120583102 shutdown_connections 2024-09-16T15:47:55.693 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:55.687+0000 7fcde1d85000 1 --2- 172.21.15.123:0/1120583102 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5617dd8ff2f0 0x5617dd971680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:55.693 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:55.687+0000 7fcde1d85000 1 --2- 172.21.15.123:0/1120583102 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5617dd8fe9d0 0x5617dd971140 unknown :-1 s=CLOSED pgs=957 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:55.693 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:55.687+0000 7fcde1d85000 1 --2- 172.21.15.123:0/1120583102 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x5617dd8b2e10 0x5617dd970c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:55.693 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:55.687+0000 7fcde1d85000 1 -- 172.21.15.123:0/1120583102 >> 172.21.15.123:0/1120583102 conn(0x5617dd79c130 msgr2=0x5617dd915e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:47:55.693 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:55.687+0000 7fcde1d85000 1 -- 172.21.15.123:0/1120583102 shutdown_connections 2024-09-16T15:47:55.693 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:55.687+0000 7fcde1d85000 1 -- 172.21.15.123:0/1120583102 wait complete. 2024-09-16T15:47:55.693 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:55.687+0000 7fcde1d85000 1 -- start start 2024-09-16T15:47:55.694 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:55.687+0000 7fcde1d85000 1 Processor -- start 2024-09-16T15:47:55.697 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:55.687+0000 7fcde1d85000 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x5617dd8b2e10 0x5617dd959430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:47:55.697 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:55.687+0000 7fcde1d85000 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5617dd8fe9d0 0x5617dd914f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:47:55.697 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:55.687+0000 7fcde1d85000 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5617dd8ff2f0 0x5617dd9154c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:47:55.697 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:55.687+0000 7fcde1d85000 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x5617dd703070 con 0x5617dd8fe9d0 2024-09-16T15:47:55.698 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:55.687+0000 7fcde1d85000 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x5617dd784a60 con 0x5617dd8ff2f0 2024-09-16T15:47:55.698 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:55.687+0000 7fcde1d85000 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x5617dd70f8f0 con 0x5617dd8b2e10 2024-09-16T15:47:55.698 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:55.687+0000 7fcde1d85000 10 client.0 _ll_register_callbacks cb 0x7fcdcc0c6070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T15:47:55.698 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:55.687+0000 7fcdd7fff640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5617dd8fe9d0 0x5617dd914f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:47:55.698 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:55.687+0000 7fcdd7fff640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5617dd8fe9d0 0x5617dd914f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:43886/0 (socket says 172.21.15.123:43886) 2024-09-16T15:47:55.698 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:55.687+0000 7fcdd7fff640 1 -- 172.21.15.123:0/3315247736 learned_addr learned my addr 172.21.15.123:0/3315247736 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:47:55.698 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:55.687+0000 7fcddca37640 1 --2- 172.21.15.123:0/3315247736 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x5617dd8b2e10 0x5617dd959430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:47:55.698 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:55.687+0000 7fcdd7fff640 1 -- 172.21.15.123:0/3315247736 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x5617dd8b2e10 msgr2=0x5617dd959430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T15:47:55.698 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:55.687+0000 7fcdd7fff640 1 --2- 172.21.15.123:0/3315247736 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x5617dd8b2e10 0x5617dd959430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:55.698 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:55.687+0000 7fcdd7fff640 1 -- 172.21.15.123:0/3315247736 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5617dd8ff2f0 msgr2=0x5617dd9154c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T15:47:55.698 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:55.687+0000 7fcdd7fff640 1 --2- 172.21.15.123:0/3315247736 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5617dd8ff2f0 0x5617dd9154c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:55.698 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:55.687+0000 7fcdd7fff640 1 -- 172.21.15.123:0/3315247736 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5617dd6c6a90 con 0x5617dd8fe9d0 2024-09-16T15:47:55.698 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:55.687+0000 7fcdd7fff640 1 --2- 172.21.15.123:0/3315247736 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5617dd8fe9d0 0x5617dd914f80 secure :-1 s=READY pgs=958 cs=0 l=1 rev1=1 crypto rx=0x7fcdcc01a510 tx=0x7fcdcc0a5cc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ea65a3c2c431a04a server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:47:55.699 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:55.687+0000 7fcdb77fe640 10 client.0 ms_handle_connect on v2:172.21.15.123:3300/0 2024-09-16T15:47:55.699 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:55.687+0000 7fcdb77fe640 1 -- 172.21.15.123:0/3315247736 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcdcc0c8070 con 0x5617dd8fe9d0 2024-09-16T15:47:55.699 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:55.687+0000 7fcdb77fe640 1 -- 172.21.15.123:0/3315247736 <== mon.0 v2:172.21.15.123:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 5709+0+0 (secure 0 0 0) 0x7fcdcc0046d0 con 0x5617dd8fe9d0 2024-09-16T15:47:55.699 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:55.687+0000 7fcde1d85000 1 -- 172.21.15.123:0/3315247736 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcdcc0cdd50 con 0x5617dd8fe9d0 2024-09-16T15:47:55.699 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:55.687+0000 7fcdb77fe640 1 -- 172.21.15.123:0/3315247736 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_get_version_reply(handle=1 version=53) ==== 24+0+0 (secure 0 0 0) 0x7fcdcc0e5050 con 0x5617dd8fe9d0 2024-09-16T15:47:55.699 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:55.687+0000 7fcde1d85000 10 client.6807 fetch_fsmap learned FSMap version 53 2024-09-16T15:47:55.699 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:55.687+0000 7fcde1d85000 1 -- 172.21.15.123:0/3315247736 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,fsmap=53,monmap=2+}) -- 0x5617dd6a5050 con 0x5617dd8fe9d0 2024-09-16T15:47:55.699 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:55.687+0000 7fcdb77fe640 1 -- 172.21.15.123:0/3315247736 <== mon.0 v2:172.21.15.123:3300/0 4 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fcdcc0cdd50 con 0x5617dd8fe9d0 2024-09-16T15:47:55.699 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:55.687+0000 7fcdb77fe640 1 -- 172.21.15.123:0/3315247736 <== mon.0 v2:172.21.15.123:3300/0 5 ==== fsmap(e 53) ==== 3502+0+0 (secure 0 0 0) 0x7fcdcc0e8020 con 0x5617dd8fe9d0 2024-09-16T15:47:55.699 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:55.687+0000 7fcde1d85000 10 client.6807 fetch_fsmap finished waiting for FSMap version 53 2024-09-16T15:47:55.699 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:55.687+0000 7fcde1d85000 10 client.6807 resolve_mds: resolved name 'b' to daemon mds.b 2024-09-16T15:47:55.699 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:55.687+0000 7fcde1d85000 20 client.6807 populate_metadata read hostname 'smithi123' 2024-09-16T15:47:55.699 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:55.687+0000 7fcde1d85000 1 --2- 172.21.15.123:0/3315247736 >> [v2:172.21.15.184:6832/1546532515,v1:172.21.15.184:6834/1546532515] conn(0x5617dd95a050 0x5617dd95a430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:47:55.699 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:55.687+0000 7fcde1d85000 4 client.6807 mds_command: new command op to 6684 tid=0 multi_id=0 [{"prefix": "status"}] 2024-09-16T15:47:55.700 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:55.687+0000 7fcde1d85000 1 -- 172.21.15.123:0/3315247736 --> [v2:172.21.15.184:6832/1546532515,v1:172.21.15.184:6834/1546532515] -- command(tid 0: {"prefix": "status"}) -- 0x5617dd70e520 con 0x5617dd95a050 2024-09-16T15:47:55.700 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:55.687+0000 7fcddca37640 1 --2- 172.21.15.123:0/3315247736 >> [v2:172.21.15.184:6832/1546532515,v1:172.21.15.184:6834/1546532515] conn(0x5617dd95a050 0x5617dd95a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:47:55.700 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:55.691+0000 7fcdb77fe640 10 client.6807 ms_handle_connect on v2:172.21.15.184:6832/1546532515 2024-09-16T15:47:55.700 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:55.691+0000 7fcddca37640 1 --2- 172.21.15.123:0/3315247736 >> [v2:172.21.15.184:6832/1546532515,v1:172.21.15.184:6834/1546532515] conn(0x5617dd95a050 0x5617dd95a430 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1f54fb448b2552bd server_cookie=a308025439845cc in_seq=0 out_seq=0 2024-09-16T15:47:55.700 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:55.691+0000 7fcdb77fe640 1 -- 172.21.15.123:0/3315247736 <== mds.0 v2:172.21.15.184:6832/1546532515 1 ==== command_reply(tid 0: 0 ) ==== 8+0+325 (crc 0 0 0) 0x5617dd70e520 con 0x5617dd95a050 2024-09-16T15:47:55.700 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:55.691+0000 7fcdb77fe640 10 client.6807 handle_command_reply: tid=0 2024-09-16T15:47:55.700 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:55.691+0000 7fcde1d85000 1 client.6807 shutdown 2024-09-16T15:47:55.700 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:55.691+0000 7fcde1d85000 1 -- 172.21.15.123:0/3315247736 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5617dd8fe9d0 msgr2=0x5617dd914f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:47:55.700 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:55.691+0000 7fcde1d85000 1 --2- 172.21.15.123:0/3315247736 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5617dd8fe9d0 0x5617dd914f80 secure :-1 s=READY pgs=958 cs=0 l=1 rev1=1 crypto rx=0x7fcdcc01a510 tx=0x7fcdcc0a5cc0 comp rx=0 tx=0).stop 2024-09-16T15:47:55.700 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:55.691+0000 7fcde1d85000 1 -- 172.21.15.123:0/3315247736 shutdown_connections 2024-09-16T15:47:55.700 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:55.691+0000 7fcde1d85000 1 --2- 172.21.15.123:0/3315247736 >> [v2:172.21.15.184:6832/1546532515,v1:172.21.15.184:6834/1546532515] conn(0x5617dd95a050 0x5617dd95a430 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:55.700 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:55.691+0000 7fcde1d85000 1 --2- 172.21.15.123:0/3315247736 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x5617dd8ff2f0 0x5617dd9154c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:55.700 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:55.691+0000 7fcde1d85000 1 --2- 172.21.15.123:0/3315247736 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x5617dd8fe9d0 0x5617dd914f80 unknown :-1 s=CLOSED pgs=958 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:55.701 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:55.691+0000 7fcde1d85000 1 --2- 172.21.15.123:0/3315247736 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x5617dd8b2e10 0x5617dd959430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:55.701 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:55.691+0000 7fcdb77fe640 0 client.6807 ms_handle_reset on v2:172.21.15.184:6832/1546532515 2024-09-16T15:47:55.701 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:55.691+0000 7fcde1d85000 1 -- 172.21.15.123:0/3315247736 >> 172.21.15.123:0/3315247736 conn(0x5617dd79c130 msgr2=0x5617dd910060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:47:55.701 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:55.691+0000 7fcde1d85000 1 -- 172.21.15.123:0/3315247736 shutdown_connections 2024-09-16T15:47:55.701 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:55.691+0000 7fcde1d85000 1 -- 172.21.15.123:0/3315247736 wait complete. 2024-09-16T15:47:55.701 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:55.691+0000 7fcde1d85000 20 client.6807 trim_cache size 0 max 16384 2024-09-16T15:47:55.701 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2024-09-16T15:47:55.701 INFO:teuthology.orchestra.run.smithi123.stdout: "cluster_fsid": "56077f89-18c8-486f-ae7e-eecc0bb7822c", 2024-09-16T15:47:55.701 INFO:teuthology.orchestra.run.smithi123.stdout: "whoami": 0, 2024-09-16T15:47:55.701 INFO:teuthology.orchestra.run.smithi123.stdout: "id": 6684, 2024-09-16T15:47:55.701 INFO:teuthology.orchestra.run.smithi123.stdout: "want_state": "up:active", 2024-09-16T15:47:55.702 INFO:teuthology.orchestra.run.smithi123.stdout: "state": "up:active", 2024-09-16T15:47:55.702 INFO:teuthology.orchestra.run.smithi123.stdout: "fs_name": "cephfs", 2024-09-16T15:47:55.702 INFO:teuthology.orchestra.run.smithi123.stdout: "rank_uptime": 5.7581937889999999, 2024-09-16T15:47:55.702 INFO:teuthology.orchestra.run.smithi123.stdout: "mdsmap_epoch": 53, 2024-09-16T15:47:55.702 INFO:teuthology.orchestra.run.smithi123.stdout: "osdmap_epoch": 85, 2024-09-16T15:47:55.702 INFO:teuthology.orchestra.run.smithi123.stdout: "osdmap_epoch_barrier": 85, 2024-09-16T15:47:55.702 INFO:teuthology.orchestra.run.smithi123.stdout: "uptime": 11.074843304 2024-09-16T15:47:55.702 INFO:teuthology.orchestra.run.smithi123.stdout:} 2024-09-16T15:47:55.703 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:55.691+0000 7fcddec9b640 1 -- 172.21.15.123:0/2736486336 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fcdc4072fe0 msgr2=0x7fcdc40754a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:47:55.703 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:55.691+0000 7fcddec9b640 1 --2- 172.21.15.123:0/2736486336 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fcdc4072fe0 0x7fcdc40754a0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fcdcc0af000 tx=0x7fcdcc01b120 comp rx=0 tx=0).stop 2024-09-16T15:47:55.703 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:55.691+0000 7fcddec9b640 1 -- 172.21.15.123:0/2736486336 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcdd80cddf0 msgr2=0x7fcdd814ff80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:47:55.703 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:55.691+0000 7fcddec9b640 1 --2- 172.21.15.123:0/2736486336 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcdd80cddf0 0x7fcdd814ff80 secure :-1 s=READY pgs=954 cs=0 l=1 rev1=1 crypto rx=0x7fcdc800c530 tx=0x7fcdc800c9f0 comp rx=0 tx=0).stop 2024-09-16T15:47:55.703 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:55.691+0000 7fcddec9b640 1 -- 172.21.15.123:0/2736486336 shutdown_connections 2024-09-16T15:47:55.703 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:55.691+0000 7fcddec9b640 1 --2- 172.21.15.123:0/2736486336 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fcdc4072fe0 0x7fcdc40754a0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:55.703 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:55.691+0000 7fcddec9b640 1 --2- 172.21.15.123:0/2736486336 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fcdd815e230 0x7fcdd80c43a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:55.703 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:55.691+0000 7fcddec9b640 1 --2- 172.21.15.123:0/2736486336 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fcdd815de50 0x7fcdd80c3e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:55.703 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:55.691+0000 7fcddec9b640 1 --2- 172.21.15.123:0/2736486336 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcdd80cddf0 0x7fcdd814ff80 unknown :-1 s=CLOSED pgs=954 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:55.703 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:55.691+0000 7fcddec9b640 1 -- 172.21.15.123:0/2736486336 >> 172.21.15.123:0/2736486336 conn(0x7fcdd80cfb10 msgr2=0x7fcdd80cd6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:47:55.703 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:55.691+0000 7fcddec9b640 1 -- 172.21.15.123:0/2736486336 shutdown_connections 2024-09-16T15:47:55.703 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:55.691+0000 7fcddec9b640 1 -- 172.21.15.123:0/2736486336 wait complete. 2024-09-16T15:47:55.711 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-09-16T15:47:55.821 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:55.815+0000 7f236b2db640 1 -- 172.21.15.123:0/1699740724 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2364074050 msgr2=0x7f236406e150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:47:55.821 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:55.815+0000 7f236b2db640 1 --2- 172.21.15.123:0/1699740724 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2364074050 0x7f236406e150 secure :-1 s=READY pgs=960 cs=0 l=1 rev1=1 crypto rx=0x7f236415bfe0 tx=0x7f235401c800 comp rx=0 tx=0).stop 2024-09-16T15:47:55.821 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:55.815+0000 7f236b2db640 1 -- 172.21.15.123:0/1699740724 shutdown_connections 2024-09-16T15:47:55.822 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:55.815+0000 7f236b2db640 1 --2- 172.21.15.123:0/1699740724 >> v2:172.21.15.184:3300/0 conn(0x7f236406e690 0x7f23640b4530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:55.822 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:55.815+0000 7f236b2db640 1 --2- 172.21.15.123:0/1699740724 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2364074050 0x7f236406e150 unknown :-1 s=CLOSED pgs=960 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:55.822 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:55.815+0000 7f236b2db640 1 -- 172.21.15.123:0/1699740724 >> 172.21.15.123:0/1699740724 conn(0x7f23640b2c20 msgr2=0x7f23640b3030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:47:55.822 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:55.815+0000 7f236b2db640 1 -- 172.21.15.123:0/1699740724 shutdown_connections 2024-09-16T15:47:55.822 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:55.815+0000 7f236b2db640 1 -- 172.21.15.123:0/1699740724 wait complete. 2024-09-16T15:47:55.822 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:55.819+0000 7f236b2db640 1 Processor -- start 2024-09-16T15:47:55.822 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:55.819+0000 7f236b2db640 1 -- start start 2024-09-16T15:47:55.824 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:55.819+0000 7f236b2db640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f236406e690 0x7f236414ed80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:47:55.824 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:55.819+0000 7f236b2db640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2364074050 0x7f23640b15c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:47:55.824 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:55.819+0000 7f236b2db640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f23640d05e0 0x7f2364140610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:47:55.824 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:55.819+0000 7f236b2db640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f2364167d00 con 0x7f2364074050 2024-09-16T15:47:55.825 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:55.819+0000 7f236b2db640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f2364167b80 con 0x7f23640d05e0 2024-09-16T15:47:55.825 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:55.819+0000 7f236b2db640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f2364167e80 con 0x7f236406e690 2024-09-16T15:47:55.825 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:55.819+0000 7f2368876640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2364074050 0x7f23640b15c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:47:55.825 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:55.819+0000 7f2368876640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2364074050 0x7f23640b15c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:43900/0 (socket says 172.21.15.123:43900) 2024-09-16T15:47:55.825 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:55.819+0000 7f2368876640 1 -- 172.21.15.123:0/3534407989 learned_addr learned my addr 172.21.15.123:0/3534407989 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:47:55.825 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:55.819+0000 7f2368876640 1 -- 172.21.15.123:0/3534407989 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f236406e690 msgr2=0x7f236414ed80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:47:55.825 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:55.819+0000 7f2368876640 1 --2- 172.21.15.123:0/3534407989 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f236406e690 0x7f236414ed80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:55.825 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:55.819+0000 7f2368876640 1 -- 172.21.15.123:0/3534407989 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f23640d05e0 msgr2=0x7f2364140610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:47:55.825 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:55.819+0000 7f2368876640 1 --2- 172.21.15.123:0/3534407989 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f23640d05e0 0x7f2364140610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:55.825 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:55.819+0000 7f2368876640 1 -- 172.21.15.123:0/3534407989 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f236406f680 con 0x7f2364074050 2024-09-16T15:47:55.825 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:55.819+0000 7f2368876640 1 --2- 172.21.15.123:0/3534407989 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2364074050 0x7f23640b15c0 secure :-1 s=READY pgs=961 cs=0 l=1 rev1=1 crypto rx=0x7f23540b0040 tx=0x7f23540a6790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:47:55.826 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:55.819+0000 7f2359ffb640 1 -- 172.21.15.123:0/3534407989 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f23540041c0 con 0x7f2364074050 2024-09-16T15:47:55.826 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:55.819+0000 7f2359ffb640 1 -- 172.21.15.123:0/3534407989 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f2354004470 con 0x7f2364074050 2024-09-16T15:47:55.826 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:55.819+0000 7f236b2db640 1 -- 172.21.15.123:0/3534407989 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f23640b1c50 con 0x7f2364074050 2024-09-16T15:47:55.826 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:55.819+0000 7f2359ffb640 1 -- 172.21.15.123:0/3534407989 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2354005280 con 0x7f2364074050 2024-09-16T15:47:55.827 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:55.819+0000 7f236b2db640 1 -- 172.21.15.123:0/3534407989 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f23640b2210 con 0x7f2364074050 2024-09-16T15:47:55.827 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:55.819+0000 7f236b2db640 1 -- 172.21.15.123:0/3534407989 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f23640b4530 con 0x7f2364074050 2024-09-16T15:47:55.827 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:55.819+0000 7f2359ffb640 1 -- 172.21.15.123:0/3534407989 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 16) ==== 97182+0+0 (secure 0 0 0) 0x7f23540a6800 con 0x7f2364074050 2024-09-16T15:47:55.827 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:55.823+0000 7f2359ffb640 1 --2- 172.21.15.123:0/3534407989 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f2348073010 0x7f23480754d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:47:55.827 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:55.823+0000 7f2359ffb640 1 -- 172.21.15.123:0/3534407989 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 5709+0+0 (secure 0 0 0) 0x7f235412e000 con 0x7f2364074050 2024-09-16T15:47:55.827 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:55.823+0000 7f2369077640 1 --2- 172.21.15.123:0/3534407989 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f2348073010 0x7f23480754d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:47:55.827 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:55.823+0000 7f2369077640 1 --2- 172.21.15.123:0/3534407989 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f2348073010 0x7f23480754d0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f2360007fd0 tx=0x7f23600079a0 comp rx=0 tx=0).ready entity=mgr.4150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:47:55.830 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:55.827+0000 7f2359ffb640 1 -- 172.21.15.123:0/3534407989 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f23540164b0 con 0x7f2364074050 2024-09-16T15:47:56.024 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:56.019+0000 7f236b2db640 1 -- 172.21.15.123:0/3534407989 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f236406fbf0 con 0x7f2364074050 2024-09-16T15:47:56.024 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:56.019+0000 7f2359ffb640 1 -- 172.21.15.123:0/3534407989 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v53) ==== 53+0+73 (secure 0 0 0) 0x7f23540f90c0 con 0x7f2364074050 2024-09-16T15:47:56.024 INFO:teuthology.orchestra.run.smithi123.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-09-16T15:47:56.026 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:56.023+0000 7f236b2db640 1 -- 172.21.15.123:0/3534407989 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f2348073010 msgr2=0x7f23480754d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:47:56.026 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:56.023+0000 7f236b2db640 1 --2- 172.21.15.123:0/3534407989 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f2348073010 0x7f23480754d0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f2360007fd0 tx=0x7f23600079a0 comp rx=0 tx=0).stop 2024-09-16T15:47:56.027 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:56.023+0000 7f236b2db640 1 -- 172.21.15.123:0/3534407989 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2364074050 msgr2=0x7f23640b15c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:47:56.027 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:56.023+0000 7f236b2db640 1 --2- 172.21.15.123:0/3534407989 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2364074050 0x7f23640b15c0 secure :-1 s=READY pgs=961 cs=0 l=1 rev1=1 crypto rx=0x7f23540b0040 tx=0x7f23540a6790 comp rx=0 tx=0).stop 2024-09-16T15:47:56.027 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:56.023+0000 7f236b2db640 1 -- 172.21.15.123:0/3534407989 shutdown_connections 2024-09-16T15:47:56.027 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:56.023+0000 7f236b2db640 1 --2- 172.21.15.123:0/3534407989 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f2348073010 0x7f23480754d0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:56.027 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:56.023+0000 7f236b2db640 1 --2- 172.21.15.123:0/3534407989 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f23640d05e0 0x7f2364140610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:56.027 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:56.023+0000 7f236b2db640 1 --2- 172.21.15.123:0/3534407989 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2364074050 0x7f23640b15c0 unknown :-1 s=CLOSED pgs=961 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:56.027 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:56.023+0000 7f236b2db640 1 --2- 172.21.15.123:0/3534407989 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f236406e690 0x7f236414ed80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:56.027 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:56.023+0000 7f236b2db640 1 -- 172.21.15.123:0/3534407989 >> 172.21.15.123:0/3534407989 conn(0x7f23640b2c20 msgr2=0x7f236415d420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:47:56.027 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:56.023+0000 7f236b2db640 1 -- 172.21.15.123:0/3534407989 shutdown_connections 2024-09-16T15:47:56.027 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:56.023+0000 7f236b2db640 1 -- 172.21.15.123:0/3534407989 wait complete. 2024-09-16T15:47:56.041 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-09-16T15:47:56.041 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-09-16T15:47:56.041 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi123.front.sepia.ceph.com 2024-09-16T15:47:56.042 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-09-16T15:47:56.042 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-09-16T15:47:56.042 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-09-16T15:47:56.042 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-09-16T15:47:56.042 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-09-16T15:47:56.042 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-09-16T15:47:56.042 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:47:56.042 DEBUG:teuthology.orchestra.run.smithi123:> ip addr 2024-09-16T15:47:56.048 INFO:teuthology.orchestra.run.smithi123.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-09-16T15:47:56.048 INFO:teuthology.orchestra.run.smithi123.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-09-16T15:47:56.048 INFO:teuthology.orchestra.run.smithi123.stdout: inet 127.0.0.1/8 scope host lo 2024-09-16T15:47:56.048 INFO:teuthology.orchestra.run.smithi123.stdout: valid_lft forever preferred_lft forever 2024-09-16T15:47:56.048 INFO:teuthology.orchestra.run.smithi123.stdout: inet6 ::1/128 scope host 2024-09-16T15:47:56.049 INFO:teuthology.orchestra.run.smithi123.stdout: valid_lft forever preferred_lft forever 2024-09-16T15:47:56.049 INFO:teuthology.orchestra.run.smithi123.stdout:2: eno1: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-09-16T15:47:56.049 INFO:teuthology.orchestra.run.smithi123.stdout: link/ether 0c:c4:7a:d8:da:6c brd ff:ff:ff:ff:ff:ff 2024-09-16T15:47:56.049 INFO:teuthology.orchestra.run.smithi123.stdout: altname enp7s0f0 2024-09-16T15:47:56.049 INFO:teuthology.orchestra.run.smithi123.stdout:3: eno2: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-09-16T15:47:56.049 INFO:teuthology.orchestra.run.smithi123.stdout: link/ether 0c:c4:7a:d8:da:6d brd ff:ff:ff:ff:ff:ff 2024-09-16T15:47:56.049 INFO:teuthology.orchestra.run.smithi123.stdout: altname enp7s0f1 2024-09-16T15:47:56.049 INFO:teuthology.orchestra.run.smithi123.stdout:4: enp3s0f0: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-09-16T15:47:56.049 INFO:teuthology.orchestra.run.smithi123.stdout: link/ether 0c:c4:7a:8f:d3:f0 brd ff:ff:ff:ff:ff:ff 2024-09-16T15:47:56.049 INFO:teuthology.orchestra.run.smithi123.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-09-16T15:47:56.049 INFO:teuthology.orchestra.run.smithi123.stdout: link/ether 0c:c4:7a:8f:d3:f1 brd ff:ff:ff:ff:ff:ff 2024-09-16T15:47:56.049 INFO:teuthology.orchestra.run.smithi123.stdout: inet 172.21.15.123/20 metric 100 brd 172.21.15.255 scope global dynamic enp3s0f1 2024-09-16T15:47:56.050 INFO:teuthology.orchestra.run.smithi123.stdout: valid_lft 42192sec preferred_lft 42192sec 2024-09-16T15:47:56.050 INFO:teuthology.orchestra.run.smithi123.stdout: inet6 fe80::ec4:7aff:fe8f:d3f1/64 scope link 2024-09-16T15:47:56.050 INFO:teuthology.orchestra.run.smithi123.stdout: valid_lft forever preferred_lft forever 2024-09-16T15:47:56.050 INFO:teuthology.orchestra.run.smithi123.stdout:6: docker0: mtu 1500 qdisc noqueue state DOWN group default 2024-09-16T15:47:56.050 INFO:teuthology.orchestra.run.smithi123.stdout: link/ether 02:42:dd:82:a2:6e brd ff:ff:ff:ff:ff:ff 2024-09-16T15:47:56.050 INFO:teuthology.orchestra.run.smithi123.stdout: inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0 2024-09-16T15:47:56.050 INFO:teuthology.orchestra.run.smithi123.stdout: valid_lft forever preferred_lft forever 2024-09-16T15:47:56.050 INFO:teuthology.orchestra.run.smithi123.stdout:7: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-09-16T15:47:56.050 INFO:teuthology.orchestra.run.smithi123.stdout: link/ether 5e:1f:84:6d:79:ae brd ff:ff:ff:ff:ff:ff 2024-09-16T15:47:56.050 INFO:teuthology.orchestra.run.smithi123.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-09-16T15:47:56.050 INFO:teuthology.orchestra.run.smithi123.stdout: valid_lft forever preferred_lft forever 2024-09-16T15:47:56.050 INFO:teuthology.orchestra.run.smithi123.stdout: inet6 fe80::5c1f:84ff:fe6d:79ae/64 scope link 2024-09-16T15:47:56.050 INFO:teuthology.orchestra.run.smithi123.stdout: valid_lft forever preferred_lft forever 2024-09-16T15:47:56.050 INFO:teuthology.orchestra.run.smithi123.stdout:8: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-09-16T15:47:56.051 INFO:teuthology.orchestra.run.smithi123.stdout: link/ether 0e:5c:f6:1f:82:2f brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-09-16T15:47:56.051 INFO:teuthology.orchestra.run.smithi123.stdout: inet6 fe80::c5c:f6ff:fe1f:822f/64 scope link 2024-09-16T15:47:56.051 INFO:teuthology.orchestra.run.smithi123.stdout: valid_lft forever preferred_lft forever 2024-09-16T15:47:56.051 INFO:teuthology.orchestra.run.smithi123.stdout:9: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-09-16T15:47:56.051 INFO:teuthology.orchestra.run.smithi123.stdout: link/ether c6:a6:3d:69:d0:db brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-09-16T15:47:56.051 INFO:teuthology.orchestra.run.smithi123.stdout: inet6 fe80::c4a6:3dff:fe69:d0db/64 scope link 2024-09-16T15:47:56.051 INFO:teuthology.orchestra.run.smithi123.stdout: valid_lft forever preferred_lft forever 2024-09-16T15:47:56.051 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:47:56.051 DEBUG:teuthology.orchestra.run.smithi123:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-09-16T15:47:56.098 INFO:teuthology.orchestra.run.smithi123.stdout:1 2024-09-16T15:47:56.100 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:47:56.100 DEBUG:teuthology.orchestra.run.smithi123:> route 2024-09-16T15:47:56.151 INFO:teuthology.orchestra.run.smithi123.stdout:Kernel IP routing table 2024-09-16T15:47:56.151 INFO:teuthology.orchestra.run.smithi123.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-09-16T15:47:56.151 INFO:teuthology.orchestra.run.smithi123.stdout:default _gateway 0.0.0.0 UG 0 0 0 enp3s0f1 2024-09-16T15:47:56.151 INFO:teuthology.orchestra.run.smithi123.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-09-16T15:47:56.151 INFO:teuthology.orchestra.run.smithi123.stdout:158.69.67.47 _gateway 255.255.255.255 UGH 100 0 0 enp3s0f1 2024-09-16T15:47:56.152 INFO:teuthology.orchestra.run.smithi123.stdout:172.17.0.0 0.0.0.0 255.255.0.0 U 0 0 0 docker0 2024-09-16T15:47:56.152 INFO:teuthology.orchestra.run.smithi123.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-09-16T15:47:56.152 INFO:teuthology.orchestra.run.smithi123.stdout:ns1.front.sepia 0.0.0.0 255.255.255.255 UH 100 0 0 enp3s0f1 2024-09-16T15:47:56.152 INFO:teuthology.orchestra.run.smithi123.stdout:172.21.0.2 0.0.0.0 255.255.255.255 UH 100 0 0 enp3s0f1 2024-09-16T15:47:56.152 INFO:teuthology.orchestra.run.smithi123.stdout:_gateway 0.0.0.0 255.255.255.255 UH 100 0 0 enp3s0f1 2024-09-16T15:47:56.152 INFO:teuthology.orchestra.run.smithi123.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-09-16T15:47:56.152 DEBUG:teuthology.orchestra.run.smithi123:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-09-16T15:47:56.152 DEBUG:teuthology.orchestra.run.smithi123:> set -e 2024-09-16T15:47:56.152 DEBUG:teuthology.orchestra.run.smithi123:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-09-16T15:47:56.152 DEBUG:teuthology.orchestra.run.smithi123:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-09-16T15:47:56.153 DEBUG:teuthology.orchestra.run.smithi123:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-09-16T15:47:56.153 DEBUG:teuthology.orchestra.run.smithi123:> ') 2024-09-16T15:47:56.236 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:56 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-09-16T15:47:56.263 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:56 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-16T15:47:56.270 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:47:56.270 DEBUG:teuthology.orchestra.run.smithi123:> ip netns list 2024-09-16T15:47:56.323 INFO:teuthology.orchestra.run.smithi123.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-09-16T15:47:56.323 INFO:teuthology.orchestra.run.smithi123.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-09-16T15:47:56.323 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-09-16T15:47:56.323 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:47:56.323 DEBUG:teuthology.orchestra.run.smithi123:> sudo ip link set brx.0 up 2024-09-16T15:47:56.378 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2024-09-16T15:47:56.378 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-09-16T15:47:56.378 DEBUG:teuthology.orchestra.run.smithi123:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-09-16T15:47:56.426 INFO:teuthology.orchestra.run.smithi123.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-09-16T15:47:56.427 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-09-16T15:47:56.427 DEBUG:teuthology.orchestra.run.smithi123:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-09-16T15:47:56.478 DEBUG:teuthology.orchestra.run.smithi123:> sudo modprobe fuse 2024-09-16T15:47:56.536 DEBUG:teuthology.orchestra.run.smithi123:> cat /proc/self/mounts | awk '{print $2}' 2024-09-16T15:47:56.583 INFO:teuthology.orchestra.run.smithi123.stdout:/sys 2024-09-16T15:47:56.583 INFO:teuthology.orchestra.run.smithi123.stdout:/proc 2024-09-16T15:47:56.583 INFO:teuthology.orchestra.run.smithi123.stdout:/dev 2024-09-16T15:47:56.583 INFO:teuthology.orchestra.run.smithi123.stdout:/dev/pts 2024-09-16T15:47:56.583 INFO:teuthology.orchestra.run.smithi123.stdout:/run 2024-09-16T15:47:56.583 INFO:teuthology.orchestra.run.smithi123.stdout:/ 2024-09-16T15:47:56.583 INFO:teuthology.orchestra.run.smithi123.stdout:/sys/kernel/security 2024-09-16T15:47:56.583 INFO:teuthology.orchestra.run.smithi123.stdout:/dev/shm 2024-09-16T15:47:56.583 INFO:teuthology.orchestra.run.smithi123.stdout:/run/lock 2024-09-16T15:47:56.584 INFO:teuthology.orchestra.run.smithi123.stdout:/sys/fs/cgroup 2024-09-16T15:47:56.584 INFO:teuthology.orchestra.run.smithi123.stdout:/sys/fs/pstore 2024-09-16T15:47:56.584 INFO:teuthology.orchestra.run.smithi123.stdout:/sys/fs/bpf 2024-09-16T15:47:56.584 INFO:teuthology.orchestra.run.smithi123.stdout:/proc/sys/fs/binfmt_misc 2024-09-16T15:47:56.584 INFO:teuthology.orchestra.run.smithi123.stdout:/dev/hugepages 2024-09-16T15:47:56.584 INFO:teuthology.orchestra.run.smithi123.stdout:/dev/mqueue 2024-09-16T15:47:56.584 INFO:teuthology.orchestra.run.smithi123.stdout:/sys/kernel/debug 2024-09-16T15:47:56.584 INFO:teuthology.orchestra.run.smithi123.stdout:/sys/kernel/tracing 2024-09-16T15:47:56.584 INFO:teuthology.orchestra.run.smithi123.stdout:/proc/fs/nfsd 2024-09-16T15:47:56.584 INFO:teuthology.orchestra.run.smithi123.stdout:/sys/fs/fuse/connections 2024-09-16T15:47:56.584 INFO:teuthology.orchestra.run.smithi123.stdout:/sys/kernel/config 2024-09-16T15:47:56.584 INFO:teuthology.orchestra.run.smithi123.stdout:/run/credentials/systemd-sysusers.service 2024-09-16T15:47:56.584 INFO:teuthology.orchestra.run.smithi123.stdout:/run/rpc_pipefs 2024-09-16T15:47:56.585 INFO:teuthology.orchestra.run.smithi123.stdout:/run/user/1000 2024-09-16T15:47:56.585 INFO:teuthology.orchestra.run.smithi123.stdout:/var/lib/ceph 2024-09-16T15:47:56.585 INFO:teuthology.orchestra.run.smithi123.stdout:/var/lib/ceph/osd/ceph-0 2024-09-16T15:47:56.585 INFO:teuthology.orchestra.run.smithi123.stdout:/var/lib/ceph/osd/ceph-1 2024-09-16T15:47:56.585 INFO:teuthology.orchestra.run.smithi123.stdout:/var/lib/ceph/osd/ceph-2 2024-09-16T15:47:56.585 INFO:teuthology.orchestra.run.smithi123.stdout:/var/lib/ceph/osd/ceph-3 2024-09-16T15:47:56.585 INFO:teuthology.orchestra.run.smithi123.stdout:/run/netns 2024-09-16T15:47:56.585 INFO:teuthology.orchestra.run.smithi123.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-09-16T15:47:56.585 INFO:teuthology.orchestra.run.smithi123.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-09-16T15:47:56.585 INFO:teuthology.orchestra.run.smithi123.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-09-16T15:47:56.585 INFO:teuthology.orchestra.run.smithi123.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-09-16T15:47:56.586 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:47:56.586 DEBUG:teuthology.orchestra.run.smithi123:> ls /sys/fs/fuse/connections 2024-09-16T15:47:56.634 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-09-16T15:47:56.635 DEBUG:teuthology.orchestra.run.smithi123:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2024-09-16T15:47:56.679 DEBUG:teuthology.orchestra.run.smithi123:> sudo modprobe fuse 2024-09-16T15:47:56.693 DEBUG:teuthology.orchestra.run.smithi123:> cat /proc/self/mounts | awk '{print $2}' 2024-09-16T15:47:56.743 INFO:teuthology.orchestra.run.smithi123.stdout:/sys 2024-09-16T15:47:56.744 INFO:teuthology.orchestra.run.smithi123.stdout:/proc 2024-09-16T15:47:56.744 INFO:teuthology.orchestra.run.smithi123.stdout:/dev 2024-09-16T15:47:56.744 INFO:teuthology.orchestra.run.smithi123.stdout:/dev/pts 2024-09-16T15:47:56.744 INFO:teuthology.orchestra.run.smithi123.stdout:/run 2024-09-16T15:47:56.744 INFO:teuthology.orchestra.run.smithi123.stdout:/ 2024-09-16T15:47:56.744 INFO:teuthology.orchestra.run.smithi123.stdout:/sys/kernel/security 2024-09-16T15:47:56.744 INFO:teuthology.orchestra.run.smithi123.stdout:/dev/shm 2024-09-16T15:47:56.744 INFO:teuthology.orchestra.run.smithi123.stdout:/run/lock 2024-09-16T15:47:56.744 INFO:teuthology.orchestra.run.smithi123.stdout:/sys/fs/cgroup 2024-09-16T15:47:56.744 INFO:teuthology.orchestra.run.smithi123.stdout:/sys/fs/pstore 2024-09-16T15:47:56.744 INFO:teuthology.orchestra.run.smithi123.stdout:/sys/fs/bpf 2024-09-16T15:47:56.744 INFO:teuthology.orchestra.run.smithi123.stdout:/proc/sys/fs/binfmt_misc 2024-09-16T15:47:56.744 INFO:teuthology.orchestra.run.smithi123.stdout:/dev/hugepages 2024-09-16T15:47:56.745 INFO:teuthology.orchestra.run.smithi123.stdout:/dev/mqueue 2024-09-16T15:47:56.745 INFO:teuthology.orchestra.run.smithi123.stdout:/sys/kernel/debug 2024-09-16T15:47:56.745 INFO:teuthology.orchestra.run.smithi123.stdout:/sys/kernel/tracing 2024-09-16T15:47:56.745 INFO:teuthology.orchestra.run.smithi123.stdout:/proc/fs/nfsd 2024-09-16T15:47:56.745 INFO:teuthology.orchestra.run.smithi123.stdout:/sys/fs/fuse/connections 2024-09-16T15:47:56.745 INFO:teuthology.orchestra.run.smithi123.stdout:/sys/kernel/config 2024-09-16T15:47:56.745 INFO:teuthology.orchestra.run.smithi123.stdout:/run/credentials/systemd-sysusers.service 2024-09-16T15:47:56.745 INFO:teuthology.orchestra.run.smithi123.stdout:/run/rpc_pipefs 2024-09-16T15:47:56.745 INFO:teuthology.orchestra.run.smithi123.stdout:/run/user/1000 2024-09-16T15:47:56.745 INFO:teuthology.orchestra.run.smithi123.stdout:/var/lib/ceph 2024-09-16T15:47:56.745 INFO:teuthology.orchestra.run.smithi123.stdout:/var/lib/ceph/osd/ceph-0 2024-09-16T15:47:56.745 INFO:teuthology.orchestra.run.smithi123.stdout:/var/lib/ceph/osd/ceph-1 2024-09-16T15:47:56.745 INFO:teuthology.orchestra.run.smithi123.stdout:/var/lib/ceph/osd/ceph-2 2024-09-16T15:47:56.745 INFO:teuthology.orchestra.run.smithi123.stdout:/var/lib/ceph/osd/ceph-3 2024-09-16T15:47:56.746 INFO:teuthology.orchestra.run.smithi123.stdout:/run/netns 2024-09-16T15:47:56.746 INFO:teuthology.orchestra.run.smithi123.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-09-16T15:47:56.746 INFO:teuthology.orchestra.run.smithi123.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-09-16T15:47:56.746 INFO:teuthology.orchestra.run.smithi123.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-09-16T15:47:56.746 INFO:teuthology.orchestra.run.smithi123.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-09-16T15:47:56.746 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:47:56.746 DEBUG:teuthology.orchestra.run.smithi123:> ls /sys/fs/fuse/connections 2024-09-16T15:47:56.757 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi123.stderr:2024-09-16T15:47:56.751+0000 7fd148696580 -1 init, newargv = 0x564818ae1ee0 newargc=15 2024-09-16T15:47:56.757 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi123.stderr:ceph-fuse[44490]: starting ceph client 2024-09-16T15:47:56.765 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi123.stderr:ceph-fuse[44490]: starting fuse 2024-09-16T15:47:57.863 DEBUG:teuthology.orchestra.run.smithi123:> sudo modprobe fuse 2024-09-16T15:47:57.879 DEBUG:teuthology.orchestra.run.smithi123:> cat /proc/self/mounts | awk '{print $2}' 2024-09-16T15:47:57.933 INFO:teuthology.orchestra.run.smithi123.stdout:/sys 2024-09-16T15:47:57.933 INFO:teuthology.orchestra.run.smithi123.stdout:/proc 2024-09-16T15:47:57.933 INFO:teuthology.orchestra.run.smithi123.stdout:/dev 2024-09-16T15:47:57.933 INFO:teuthology.orchestra.run.smithi123.stdout:/dev/pts 2024-09-16T15:47:57.933 INFO:teuthology.orchestra.run.smithi123.stdout:/run 2024-09-16T15:47:57.933 INFO:teuthology.orchestra.run.smithi123.stdout:/ 2024-09-16T15:47:57.934 INFO:teuthology.orchestra.run.smithi123.stdout:/sys/kernel/security 2024-09-16T15:47:57.934 INFO:teuthology.orchestra.run.smithi123.stdout:/dev/shm 2024-09-16T15:47:57.934 INFO:teuthology.orchestra.run.smithi123.stdout:/run/lock 2024-09-16T15:47:57.934 INFO:teuthology.orchestra.run.smithi123.stdout:/sys/fs/cgroup 2024-09-16T15:47:57.934 INFO:teuthology.orchestra.run.smithi123.stdout:/sys/fs/pstore 2024-09-16T15:47:57.934 INFO:teuthology.orchestra.run.smithi123.stdout:/sys/fs/bpf 2024-09-16T15:47:57.934 INFO:teuthology.orchestra.run.smithi123.stdout:/proc/sys/fs/binfmt_misc 2024-09-16T15:47:57.934 INFO:teuthology.orchestra.run.smithi123.stdout:/dev/hugepages 2024-09-16T15:47:57.934 INFO:teuthology.orchestra.run.smithi123.stdout:/dev/mqueue 2024-09-16T15:47:57.934 INFO:teuthology.orchestra.run.smithi123.stdout:/sys/kernel/debug 2024-09-16T15:47:57.934 INFO:teuthology.orchestra.run.smithi123.stdout:/sys/kernel/tracing 2024-09-16T15:47:57.934 INFO:teuthology.orchestra.run.smithi123.stdout:/proc/fs/nfsd 2024-09-16T15:47:57.934 INFO:teuthology.orchestra.run.smithi123.stdout:/sys/fs/fuse/connections 2024-09-16T15:47:57.935 INFO:teuthology.orchestra.run.smithi123.stdout:/sys/kernel/config 2024-09-16T15:47:57.935 INFO:teuthology.orchestra.run.smithi123.stdout:/run/credentials/systemd-sysusers.service 2024-09-16T15:47:57.935 INFO:teuthology.orchestra.run.smithi123.stdout:/run/rpc_pipefs 2024-09-16T15:47:57.935 INFO:teuthology.orchestra.run.smithi123.stdout:/run/user/1000 2024-09-16T15:47:57.935 INFO:teuthology.orchestra.run.smithi123.stdout:/var/lib/ceph 2024-09-16T15:47:57.935 INFO:teuthology.orchestra.run.smithi123.stdout:/var/lib/ceph/osd/ceph-0 2024-09-16T15:47:57.935 INFO:teuthology.orchestra.run.smithi123.stdout:/var/lib/ceph/osd/ceph-1 2024-09-16T15:47:57.935 INFO:teuthology.orchestra.run.smithi123.stdout:/var/lib/ceph/osd/ceph-2 2024-09-16T15:47:57.935 INFO:teuthology.orchestra.run.smithi123.stdout:/var/lib/ceph/osd/ceph-3 2024-09-16T15:47:57.935 INFO:teuthology.orchestra.run.smithi123.stdout:/run/netns 2024-09-16T15:47:57.935 INFO:teuthology.orchestra.run.smithi123.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-09-16T15:47:57.935 INFO:teuthology.orchestra.run.smithi123.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-09-16T15:47:57.935 INFO:teuthology.orchestra.run.smithi123.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-09-16T15:47:57.935 INFO:teuthology.orchestra.run.smithi123.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-09-16T15:47:57.936 INFO:teuthology.orchestra.run.smithi123.stdout:/home/ubuntu/cephtest/mnt.0 2024-09-16T15:47:57.936 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:47:57.936 DEBUG:teuthology.orchestra.run.smithi123:> ls /sys/fs/fuse/connections 2024-09-16T15:47:57.983 INFO:teuthology.orchestra.run.smithi123.stdout:45 2024-09-16T15:47:57.983 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [45] 2024-09-16T15:47:57.983 DEBUG:teuthology.orchestra.run.smithi123:> sudo stdin-killer -- python3 -c ' 2024-09-16T15:47:57.983 DEBUG:teuthology.orchestra.run.smithi123:> import glob 2024-09-16T15:47:57.984 DEBUG:teuthology.orchestra.run.smithi123:> import re 2024-09-16T15:47:57.984 DEBUG:teuthology.orchestra.run.smithi123:> import os 2024-09-16T15:47:57.984 DEBUG:teuthology.orchestra.run.smithi123:> import subprocess 2024-09-16T15:47:57.984 DEBUG:teuthology.orchestra.run.smithi123:> 2024-09-16T15:47:57.984 DEBUG:teuthology.orchestra.run.smithi123:> def _find_admin_socket(client_name): 2024-09-16T15:47:57.984 DEBUG:teuthology.orchestra.run.smithi123:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2024-09-16T15:47:57.984 DEBUG:teuthology.orchestra.run.smithi123:> files = glob.glob(asok_path) 2024-09-16T15:47:57.984 DEBUG:teuthology.orchestra.run.smithi123:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-09-16T15:47:57.984 DEBUG:teuthology.orchestra.run.smithi123:> 2024-09-16T15:47:57.984 DEBUG:teuthology.orchestra.run.smithi123:> # Given a non-glob path, it better be there 2024-09-16T15:47:57.984 DEBUG:teuthology.orchestra.run.smithi123:> if "*" not in asok_path: 2024-09-16T15:47:57.984 DEBUG:teuthology.orchestra.run.smithi123:> assert(len(files) == 1) 2024-09-16T15:47:57.984 DEBUG:teuthology.orchestra.run.smithi123:> return files[0] 2024-09-16T15:47:57.984 DEBUG:teuthology.orchestra.run.smithi123:> 2024-09-16T15:47:57.985 DEBUG:teuthology.orchestra.run.smithi123:> for f in files: 2024-09-16T15:47:57.985 DEBUG:teuthology.orchestra.run.smithi123:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-09-16T15:47:57.985 DEBUG:teuthology.orchestra.run.smithi123:> if os.path.exists("/proc/{0}".format(pid)): 2024-09-16T15:47:57.985 DEBUG:teuthology.orchestra.run.smithi123:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-09-16T15:47:57.985 DEBUG:teuthology.orchestra.run.smithi123:> contents = proc_f.read() 2024-09-16T15:47:57.985 DEBUG:teuthology.orchestra.run.smithi123:> if mountpoint in contents: 2024-09-16T15:47:57.985 DEBUG:teuthology.orchestra.run.smithi123:> return f 2024-09-16T15:47:57.985 DEBUG:teuthology.orchestra.run.smithi123:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-09-16T15:47:57.985 DEBUG:teuthology.orchestra.run.smithi123:> 2024-09-16T15:47:57.985 DEBUG:teuthology.orchestra.run.smithi123:> print(_find_admin_socket("client.0")) 2024-09-16T15:47:57.985 DEBUG:teuthology.orchestra.run.smithi123:> ' 2024-09-16T15:47:58.107 INFO:teuthology.orchestra.run.smithi123.stdout:/var/run/ceph/ceph-client.0.44490.asok 2024-09-16T15:47:58.111 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:58 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-16T15:47:58.119 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.44490.asok 2024-09-16T15:47:58.119 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:47:58.119 DEBUG:teuthology.orchestra.run.smithi123:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.44490.asok status 2024-09-16T15:47:58.273 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2024-09-16T15:47:58.273 INFO:teuthology.orchestra.run.smithi123.stdout: "metadata": { 2024-09-16T15:47:58.273 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph_sha1": "33f95d2ae1bcbe7eb04404303f179eeeaa6982df", 2024-09-16T15:47:58.273 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph_version": "ceph version 19.3.0-4938-g33f95d2a (33f95d2ae1bcbe7eb04404303f179eeeaa6982df) squid (dev)", 2024-09-16T15:47:58.274 INFO:teuthology.orchestra.run.smithi123.stdout: "entity_id": "0", 2024-09-16T15:47:58.274 INFO:teuthology.orchestra.run.smithi123.stdout: "hostname": "smithi123", 2024-09-16T15:47:58.274 INFO:teuthology.orchestra.run.smithi123.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2024-09-16T15:47:58.274 INFO:teuthology.orchestra.run.smithi123.stdout: "pid": "44490", 2024-09-16T15:47:58.274 INFO:teuthology.orchestra.run.smithi123.stdout: "root": "/" 2024-09-16T15:47:58.274 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:58.274 INFO:teuthology.orchestra.run.smithi123.stdout: "dentry_count": 0, 2024-09-16T15:47:58.274 INFO:teuthology.orchestra.run.smithi123.stdout: "dentry_pinned_count": 0, 2024-09-16T15:47:58.274 INFO:teuthology.orchestra.run.smithi123.stdout: "id": 6825, 2024-09-16T15:47:58.274 INFO:teuthology.orchestra.run.smithi123.stdout: "inst": { 2024-09-16T15:47:58.274 INFO:teuthology.orchestra.run.smithi123.stdout: "name": { 2024-09-16T15:47:58.274 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "client", 2024-09-16T15:47:58.275 INFO:teuthology.orchestra.run.smithi123.stdout: "num": 6825 2024-09-16T15:47:58.275 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:58.275 INFO:teuthology.orchestra.run.smithi123.stdout: "addr": { 2024-09-16T15:47:58.275 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "v1", 2024-09-16T15:47:58.275 INFO:teuthology.orchestra.run.smithi123.stdout: "addr": "192.168.0.1:0", 2024-09-16T15:47:58.275 INFO:teuthology.orchestra.run.smithi123.stdout: "nonce": 4143761041 2024-09-16T15:47:58.275 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-16T15:47:58.275 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:58.275 INFO:teuthology.orchestra.run.smithi123.stdout: "addr": { 2024-09-16T15:47:58.275 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "v1", 2024-09-16T15:47:58.275 INFO:teuthology.orchestra.run.smithi123.stdout: "addr": "192.168.0.1:0", 2024-09-16T15:47:58.275 INFO:teuthology.orchestra.run.smithi123.stdout: "nonce": 4143761041 2024-09-16T15:47:58.276 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:47:58.276 INFO:teuthology.orchestra.run.smithi123.stdout: "inst_str": "client.6825 192.168.0.1:0/4143761041", 2024-09-16T15:47:58.276 INFO:teuthology.orchestra.run.smithi123.stdout: "addr_str": "192.168.0.1:0/4143761041", 2024-09-16T15:47:58.276 INFO:teuthology.orchestra.run.smithi123.stdout: "inode_count": 1, 2024-09-16T15:47:58.276 INFO:teuthology.orchestra.run.smithi123.stdout: "mds_epoch": 53, 2024-09-16T15:47:58.276 INFO:teuthology.orchestra.run.smithi123.stdout: "osd_epoch": 85, 2024-09-16T15:47:58.276 INFO:teuthology.orchestra.run.smithi123.stdout: "osd_epoch_barrier": 0, 2024-09-16T15:47:58.276 INFO:teuthology.orchestra.run.smithi123.stdout: "blocklisted": false, 2024-09-16T15:47:58.276 INFO:teuthology.orchestra.run.smithi123.stdout: "fs_name": "cephfs" 2024-09-16T15:47:58.276 INFO:teuthology.orchestra.run.smithi123.stdout:} 2024-09-16T15:47:58.287 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:47:58.287 DEBUG:teuthology.orchestra.run.smithi123:> stat --file-system '--printf=%T 2024-09-16T15:47:58.287 DEBUG:teuthology.orchestra.run.smithi123:> ' -- /home/ubuntu/cephtest/mnt.0 2024-09-16T15:47:58.293 INFO:teuthology.orchestra.run.smithi123.stdout:fuseblk 2024-09-16T15:47:58.293 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2024-09-16T15:47:58.293 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:47:58.293 DEBUG:teuthology.orchestra.run.smithi123:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-09-16T15:47:58.347 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-09-16T15:47:58.523 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:58.519+0000 7f7289b15640 1 -- 172.21.15.123:0/4023371522 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f728415e160 msgr2=0x7f7284073f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:47:58.523 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:58.519+0000 7f7289b15640 1 --2- 172.21.15.123:0/4023371522 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f728415e160 0x7f7284073f30 secure :-1 s=READY pgs=965 cs=0 l=1 rev1=1 crypto rx=0x7f72840cf1f0 tx=0x7f727c01cc60 comp rx=0 tx=0).stop 2024-09-16T15:47:58.523 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:58.519+0000 7f7289b15640 1 -- 172.21.15.123:0/4023371522 shutdown_connections 2024-09-16T15:47:58.523 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:58.519+0000 7f7289b15640 1 --2- 172.21.15.123:0/4023371522 >> v2:172.21.15.184:3300/0 conn(0x7f7284074470 0x7f72840cd6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:58.523 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:58.519+0000 7f7289b15640 1 --2- 172.21.15.123:0/4023371522 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f728415e160 0x7f7284073f30 unknown :-1 s=CLOSED pgs=965 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:58.523 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:58.519+0000 7f7289b15640 1 -- 172.21.15.123:0/4023371522 >> 172.21.15.123:0/4023371522 conn(0x7f72840cfb10 msgr2=0x7f72840cff20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:47:58.523 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:58.519+0000 7f7289b15640 1 -- 172.21.15.123:0/4023371522 shutdown_connections 2024-09-16T15:47:58.524 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:58.519+0000 7f7289b15640 1 -- 172.21.15.123:0/4023371522 wait complete. 2024-09-16T15:47:58.524 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:58.519+0000 7f7289b15640 1 Processor -- start 2024-09-16T15:47:58.524 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:58.519+0000 7f7289b15640 1 -- start start 2024-09-16T15:47:58.526 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:58.519+0000 7f7289b15640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7284074470 0x7f7284164770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:47:58.526 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:58.519+0000 7f7289b15640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f72840cdc20 0x7f7284153f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:47:58.526 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:58.519+0000 7f7289b15640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f728415e160 0x7f7284154440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:47:58.526 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:58.519+0000 7f7289b15640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f7284167d40 con 0x7f7284074470 2024-09-16T15:47:58.526 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:58.519+0000 7f7289b15640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f7284167bc0 con 0x7f728415e160 2024-09-16T15:47:58.526 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:58.519+0000 7f7289b15640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f7284167ec0 con 0x7f72840cdc20 2024-09-16T15:47:58.526 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:58.519+0000 7f72837fe640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7284074470 0x7f7284164770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:47:58.526 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:58.519+0000 7f72837fe640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7284074470 0x7f7284164770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:43924/0 (socket says 172.21.15.123:43924) 2024-09-16T15:47:58.526 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:58.519+0000 7f72837fe640 1 -- 172.21.15.123:0/999713282 learned_addr learned my addr 172.21.15.123:0/999713282 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:47:58.526 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:58.519+0000 7f72837fe640 1 -- 172.21.15.123:0/999713282 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f72840cdc20 msgr2=0x7f7284153f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:47:58.527 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:58.519+0000 7f72837fe640 1 --2- 172.21.15.123:0/999713282 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f72840cdc20 0x7f7284153f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:58.527 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:58.519+0000 7f72837fe640 1 -- 172.21.15.123:0/999713282 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f728415e160 msgr2=0x7f7284154440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:47:58.527 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:58.519+0000 7f72837fe640 1 --2- 172.21.15.123:0/999713282 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f728415e160 0x7f7284154440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:58.527 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:58.519+0000 7f72837fe640 1 -- 172.21.15.123:0/999713282 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7284165130 con 0x7f7284074470 2024-09-16T15:47:58.527 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:58.519+0000 7f72837fe640 1 --2- 172.21.15.123:0/999713282 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7284074470 0x7f7284164770 secure :-1 s=READY pgs=966 cs=0 l=1 rev1=1 crypto rx=0x7f727c000f90 tx=0x7f727c003f60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:47:58.527 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:58.519+0000 7f726bfff640 1 -- 172.21.15.123:0/999713282 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f727c0b2040 con 0x7f7284074470 2024-09-16T15:47:58.527 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:58.519+0000 7f726bfff640 1 -- 172.21.15.123:0/999713282 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f727c004340 con 0x7f7284074470 2024-09-16T15:47:58.527 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:58.519+0000 7f726bfff640 1 -- 172.21.15.123:0/999713282 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f727c004710 con 0x7f7284074470 2024-09-16T15:47:58.527 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:58.519+0000 7f7289b15640 1 -- 172.21.15.123:0/999713282 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7284165410 con 0x7f7284074470 2024-09-16T15:47:58.527 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:58.519+0000 7f7289b15640 1 -- 172.21.15.123:0/999713282 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f72840c3e50 con 0x7f7284074470 2024-09-16T15:47:58.528 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:58.523+0000 7f7289b15640 1 -- 172.21.15.123:0/999713282 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f72840cd6e0 con 0x7f7284074470 2024-09-16T15:47:58.528 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:58.523+0000 7f726bfff640 1 -- 172.21.15.123:0/999713282 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 16) ==== 97182+0+0 (secure 0 0 0) 0x7f727c0ad440 con 0x7f7284074470 2024-09-16T15:47:58.528 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:58.523+0000 7f726bfff640 1 --2- 172.21.15.123:0/999713282 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f7270072fe0 0x7f72700754a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:47:58.528 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:58.523+0000 7f726bfff640 1 -- 172.21.15.123:0/999713282 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 5709+0+0 (secure 0 0 0) 0x7f727c12e880 con 0x7f7284074470 2024-09-16T15:47:58.528 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:58.523+0000 7f7282ffd640 1 --2- 172.21.15.123:0/999713282 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f7270072fe0 0x7f72700754a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:47:58.529 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:58.523+0000 7f7282ffd640 1 --2- 172.21.15.123:0/999713282 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f7270072fe0 0x7f72700754a0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f72840ca0d0 tx=0x7f7278005c00 comp rx=0 tx=0).ready entity=mgr.4150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:47:58.532 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:58.527+0000 7f726bfff640 1 -- 172.21.15.123:0/999713282 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f727c016970 con 0x7f7284074470 2024-09-16T15:47:58.744 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:58.739+0000 7f7289b15640 1 -- 172.21.15.123:0/999713282 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f72840d06c0 con 0x7f7284074470 2024-09-16T15:47:58.745 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:58.739+0000 7f726bfff640 1 -- 172.21.15.123:0/999713282 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v53) ==== 53+0+73 (secure 0 0 0) 0x7f727c0f9940 con 0x7f7284074470 2024-09-16T15:47:58.745 INFO:teuthology.orchestra.run.smithi123.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-09-16T15:47:58.747 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:58.743+0000 7f7289b15640 1 -- 172.21.15.123:0/999713282 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f7270072fe0 msgr2=0x7f72700754a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:47:58.747 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:58.743+0000 7f7289b15640 1 --2- 172.21.15.123:0/999713282 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f7270072fe0 0x7f72700754a0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f72840ca0d0 tx=0x7f7278005c00 comp rx=0 tx=0).stop 2024-09-16T15:47:58.747 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:58.743+0000 7f7289b15640 1 -- 172.21.15.123:0/999713282 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7284074470 msgr2=0x7f7284164770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:47:58.747 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:58.743+0000 7f7289b15640 1 --2- 172.21.15.123:0/999713282 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7284074470 0x7f7284164770 secure :-1 s=READY pgs=966 cs=0 l=1 rev1=1 crypto rx=0x7f727c000f90 tx=0x7f727c003f60 comp rx=0 tx=0).stop 2024-09-16T15:47:58.747 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:58.743+0000 7f7289b15640 1 -- 172.21.15.123:0/999713282 shutdown_connections 2024-09-16T15:47:58.747 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:58.743+0000 7f7289b15640 1 --2- 172.21.15.123:0/999713282 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f7270072fe0 0x7f72700754a0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:58.748 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:58.743+0000 7f7289b15640 1 --2- 172.21.15.123:0/999713282 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f728415e160 0x7f7284154440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:58.748 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:58.743+0000 7f7289b15640 1 --2- 172.21.15.123:0/999713282 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f72840cdc20 0x7f7284153f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:58.748 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:58.743+0000 7f7289b15640 1 --2- 172.21.15.123:0/999713282 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7284074470 0x7f7284164770 unknown :-1 s=CLOSED pgs=966 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:47:58.748 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:58.743+0000 7f7289b15640 1 -- 172.21.15.123:0/999713282 >> 172.21.15.123:0/999713282 conn(0x7f72840cfb10 msgr2=0x7f728406eee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:47:58.748 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:58.743+0000 7f7289b15640 1 -- 172.21.15.123:0/999713282 shutdown_connections 2024-09-16T15:47:58.748 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:47:58.743+0000 7f7289b15640 1 -- 172.21.15.123:0/999713282 wait complete. 2024-09-16T15:47:58.763 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-09-16T15:47:58.763 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-09-16T15:47:58.763 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi184.front.sepia.ceph.com 2024-09-16T15:47:58.763 INFO:tasks.cephfs.mount:self.client.name = client.1 2024-09-16T15:47:58.763 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2024-09-16T15:47:58.763 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-09-16T15:47:58.763 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-09-16T15:47:58.764 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-09-16T15:47:58.764 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.1' netns for '/home/ubuntu/cephtest/mnt.1' 2024-09-16T15:47:58.764 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:47:58.764 DEBUG:teuthology.orchestra.run.smithi184:> ip addr 2024-09-16T15:47:58.771 INFO:teuthology.orchestra.run.smithi184.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-09-16T15:47:58.771 INFO:teuthology.orchestra.run.smithi184.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-09-16T15:47:58.771 INFO:teuthology.orchestra.run.smithi184.stdout: inet 127.0.0.1/8 scope host lo 2024-09-16T15:47:58.771 INFO:teuthology.orchestra.run.smithi184.stdout: valid_lft forever preferred_lft forever 2024-09-16T15:47:58.771 INFO:teuthology.orchestra.run.smithi184.stdout: inet6 ::1/128 scope host 2024-09-16T15:47:58.771 INFO:teuthology.orchestra.run.smithi184.stdout: valid_lft forever preferred_lft forever 2024-09-16T15:47:58.772 INFO:teuthology.orchestra.run.smithi184.stdout:2: eno1: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-09-16T15:47:58.772 INFO:teuthology.orchestra.run.smithi184.stdout: link/ether 0c:c4:7a:d9:37:16 brd ff:ff:ff:ff:ff:ff 2024-09-16T15:47:58.772 INFO:teuthology.orchestra.run.smithi184.stdout: altname enp7s0f0 2024-09-16T15:47:58.772 INFO:teuthology.orchestra.run.smithi184.stdout:3: eno2: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-09-16T15:47:58.772 INFO:teuthology.orchestra.run.smithi184.stdout: link/ether 0c:c4:7a:d9:37:17 brd ff:ff:ff:ff:ff:ff 2024-09-16T15:47:58.772 INFO:teuthology.orchestra.run.smithi184.stdout: altname enp7s0f1 2024-09-16T15:47:58.772 INFO:teuthology.orchestra.run.smithi184.stdout:4: enp3s0f0: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-09-16T15:47:58.772 INFO:teuthology.orchestra.run.smithi184.stdout: link/ether 0c:c4:7a:88:7b:8e brd ff:ff:ff:ff:ff:ff 2024-09-16T15:47:58.772 INFO:teuthology.orchestra.run.smithi184.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-09-16T15:47:58.772 INFO:teuthology.orchestra.run.smithi184.stdout: link/ether 0c:c4:7a:88:7b:8f brd ff:ff:ff:ff:ff:ff 2024-09-16T15:47:58.772 INFO:teuthology.orchestra.run.smithi184.stdout: inet 172.21.15.184/20 metric 100 brd 172.21.15.255 scope global dynamic enp3s0f1 2024-09-16T15:47:58.772 INFO:teuthology.orchestra.run.smithi184.stdout: valid_lft 42178sec preferred_lft 42178sec 2024-09-16T15:47:58.772 INFO:teuthology.orchestra.run.smithi184.stdout: inet6 fe80::ec4:7aff:fe88:7b8f/64 scope link 2024-09-16T15:47:58.773 INFO:teuthology.orchestra.run.smithi184.stdout: valid_lft forever preferred_lft forever 2024-09-16T15:47:58.773 INFO:teuthology.orchestra.run.smithi184.stdout:6: docker0: mtu 1500 qdisc noqueue state DOWN group default 2024-09-16T15:47:58.773 INFO:teuthology.orchestra.run.smithi184.stdout: link/ether 02:42:5d:0a:44:b9 brd ff:ff:ff:ff:ff:ff 2024-09-16T15:47:58.773 INFO:teuthology.orchestra.run.smithi184.stdout: inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0 2024-09-16T15:47:58.773 INFO:teuthology.orchestra.run.smithi184.stdout: valid_lft forever preferred_lft forever 2024-09-16T15:47:58.773 INFO:teuthology.orchestra.run.smithi184.stdout:7: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-09-16T15:47:58.773 INFO:teuthology.orchestra.run.smithi184.stdout: link/ether 5e:1f:84:6d:79:ae brd ff:ff:ff:ff:ff:ff 2024-09-16T15:47:58.773 INFO:teuthology.orchestra.run.smithi184.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-09-16T15:47:58.773 INFO:teuthology.orchestra.run.smithi184.stdout: valid_lft forever preferred_lft forever 2024-09-16T15:47:58.773 INFO:teuthology.orchestra.run.smithi184.stdout: inet6 fe80::5c1f:84ff:fe6d:79ae/64 scope link 2024-09-16T15:47:58.773 INFO:teuthology.orchestra.run.smithi184.stdout: valid_lft forever preferred_lft forever 2024-09-16T15:47:58.773 INFO:teuthology.orchestra.run.smithi184.stdout:8: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-09-16T15:47:58.773 INFO:teuthology.orchestra.run.smithi184.stdout: link/ether 0e:5c:f6:1f:82:2f brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.1 2024-09-16T15:47:58.773 INFO:teuthology.orchestra.run.smithi184.stdout: inet6 fe80::c5c:f6ff:fe1f:822f/64 scope link 2024-09-16T15:47:58.774 INFO:teuthology.orchestra.run.smithi184.stdout: valid_lft forever preferred_lft forever 2024-09-16T15:47:58.774 INFO:teuthology.orchestra.run.smithi184.stdout:9: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-09-16T15:47:58.774 INFO:teuthology.orchestra.run.smithi184.stdout: link/ether c6:a6:3d:69:d0:db brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.3 2024-09-16T15:47:58.774 INFO:teuthology.orchestra.run.smithi184.stdout: inet6 fe80::c4a6:3dff:fe69:d0db/64 scope link 2024-09-16T15:47:58.774 INFO:teuthology.orchestra.run.smithi184.stdout: valid_lft forever preferred_lft forever 2024-09-16T15:47:58.774 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:47:58.774 DEBUG:teuthology.orchestra.run.smithi184:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-09-16T15:47:58.826 INFO:teuthology.orchestra.run.smithi184.stdout:1 2024-09-16T15:47:58.828 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:47:58.828 DEBUG:teuthology.orchestra.run.smithi184:> route 2024-09-16T15:47:58.879 INFO:teuthology.orchestra.run.smithi184.stdout:Kernel IP routing table 2024-09-16T15:47:58.879 INFO:teuthology.orchestra.run.smithi184.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-09-16T15:47:58.879 INFO:teuthology.orchestra.run.smithi184.stdout:default _gateway 0.0.0.0 UG 0 0 0 enp3s0f1 2024-09-16T15:47:58.879 INFO:teuthology.orchestra.run.smithi184.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-09-16T15:47:58.879 INFO:teuthology.orchestra.run.smithi184.stdout:158.69.67.47 _gateway 255.255.255.255 UGH 100 0 0 enp3s0f1 2024-09-16T15:47:58.879 INFO:teuthology.orchestra.run.smithi184.stdout:172.17.0.0 0.0.0.0 255.255.0.0 U 0 0 0 docker0 2024-09-16T15:47:58.879 INFO:teuthology.orchestra.run.smithi184.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-09-16T15:47:58.880 INFO:teuthology.orchestra.run.smithi184.stdout:ns1.front.sepia 0.0.0.0 255.255.255.255 UH 100 0 0 enp3s0f1 2024-09-16T15:47:58.880 INFO:teuthology.orchestra.run.smithi184.stdout:172.21.0.2 0.0.0.0 255.255.255.255 UH 100 0 0 enp3s0f1 2024-09-16T15:47:58.880 INFO:teuthology.orchestra.run.smithi184.stdout:_gateway 0.0.0.0 255.255.255.255 UH 100 0 0 enp3s0f1 2024-09-16T15:47:58.880 INFO:teuthology.orchestra.run.smithi184.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-09-16T15:47:58.880 DEBUG:teuthology.orchestra.run.smithi184:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-09-16T15:47:58.880 DEBUG:teuthology.orchestra.run.smithi184:> set -e 2024-09-16T15:47:58.880 DEBUG:teuthology.orchestra.run.smithi184:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-09-16T15:47:58.880 DEBUG:teuthology.orchestra.run.smithi184:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-09-16T15:47:58.880 DEBUG:teuthology.orchestra.run.smithi184:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-09-16T15:47:58.880 DEBUG:teuthology.orchestra.run.smithi184:> ') 2024-09-16T15:47:58.968 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:47:58 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-09-16T15:47:58.996 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:47:58 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-16T15:47:59.004 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:47:59.004 DEBUG:teuthology.orchestra.run.smithi184:> ip netns list 2024-09-16T15:47:59.009 INFO:teuthology.orchestra.run.smithi184.stdout:ceph-ns--home-ubuntu-cephtest-mnt.3 (id: 1) 2024-09-16T15:47:59.010 INFO:teuthology.orchestra.run.smithi184.stdout:ceph-ns--home-ubuntu-cephtest-mnt.1 (id: 0) 2024-09-16T15:47:59.010 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.1' netns for '/home/ubuntu/cephtest/mnt.1' 2024-09-16T15:47:59.010 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:47:59.010 DEBUG:teuthology.orchestra.run.smithi184:> sudo ip link set brx.0 up 2024-09-16T15:47:59.066 INFO:tasks.cephfs.fuse_mount:Client client.1 config is {} 2024-09-16T15:47:59.066 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-09-16T15:47:59.066 DEBUG:teuthology.orchestra.run.smithi184:> mkdir -p -v /home/ubuntu/cephtest/mnt.1 2024-09-16T15:47:59.115 INFO:teuthology.orchestra.run.smithi184.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.1' 2024-09-16T15:47:59.115 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-09-16T15:47:59.115 DEBUG:teuthology.orchestra.run.smithi184:> chmod 0000 /home/ubuntu/cephtest/mnt.1 2024-09-16T15:47:59.164 DEBUG:teuthology.orchestra.run.smithi184:> sudo modprobe fuse 2024-09-16T15:47:59.220 DEBUG:teuthology.orchestra.run.smithi184:> cat /proc/self/mounts | awk '{print $2}' 2024-09-16T15:47:59.268 INFO:teuthology.orchestra.run.smithi184.stdout:/sys 2024-09-16T15:47:59.268 INFO:teuthology.orchestra.run.smithi184.stdout:/proc 2024-09-16T15:47:59.268 INFO:teuthology.orchestra.run.smithi184.stdout:/dev 2024-09-16T15:47:59.268 INFO:teuthology.orchestra.run.smithi184.stdout:/dev/pts 2024-09-16T15:47:59.268 INFO:teuthology.orchestra.run.smithi184.stdout:/run 2024-09-16T15:47:59.268 INFO:teuthology.orchestra.run.smithi184.stdout:/ 2024-09-16T15:47:59.268 INFO:teuthology.orchestra.run.smithi184.stdout:/sys/kernel/security 2024-09-16T15:47:59.268 INFO:teuthology.orchestra.run.smithi184.stdout:/dev/shm 2024-09-16T15:47:59.269 INFO:teuthology.orchestra.run.smithi184.stdout:/run/lock 2024-09-16T15:47:59.269 INFO:teuthology.orchestra.run.smithi184.stdout:/sys/fs/cgroup 2024-09-16T15:47:59.269 INFO:teuthology.orchestra.run.smithi184.stdout:/sys/fs/pstore 2024-09-16T15:47:59.269 INFO:teuthology.orchestra.run.smithi184.stdout:/sys/fs/bpf 2024-09-16T15:47:59.269 INFO:teuthology.orchestra.run.smithi184.stdout:/proc/sys/fs/binfmt_misc 2024-09-16T15:47:59.269 INFO:teuthology.orchestra.run.smithi184.stdout:/dev/hugepages 2024-09-16T15:47:59.269 INFO:teuthology.orchestra.run.smithi184.stdout:/dev/mqueue 2024-09-16T15:47:59.269 INFO:teuthology.orchestra.run.smithi184.stdout:/sys/kernel/debug 2024-09-16T15:47:59.269 INFO:teuthology.orchestra.run.smithi184.stdout:/sys/kernel/tracing 2024-09-16T15:47:59.269 INFO:teuthology.orchestra.run.smithi184.stdout:/proc/fs/nfsd 2024-09-16T15:47:59.269 INFO:teuthology.orchestra.run.smithi184.stdout:/sys/fs/fuse/connections 2024-09-16T15:47:59.269 INFO:teuthology.orchestra.run.smithi184.stdout:/sys/kernel/config 2024-09-16T15:47:59.270 INFO:teuthology.orchestra.run.smithi184.stdout:/run/credentials/systemd-sysusers.service 2024-09-16T15:47:59.270 INFO:teuthology.orchestra.run.smithi184.stdout:/run/rpc_pipefs 2024-09-16T15:47:59.270 INFO:teuthology.orchestra.run.smithi184.stdout:/run/user/1000 2024-09-16T15:47:59.270 INFO:teuthology.orchestra.run.smithi184.stdout:/var/lib/ceph 2024-09-16T15:47:59.270 INFO:teuthology.orchestra.run.smithi184.stdout:/var/lib/ceph/osd/ceph-4 2024-09-16T15:47:59.270 INFO:teuthology.orchestra.run.smithi184.stdout:/var/lib/ceph/osd/ceph-5 2024-09-16T15:47:59.270 INFO:teuthology.orchestra.run.smithi184.stdout:/var/lib/ceph/osd/ceph-6 2024-09-16T15:47:59.270 INFO:teuthology.orchestra.run.smithi184.stdout:/var/lib/ceph/osd/ceph-7 2024-09-16T15:47:59.270 INFO:teuthology.orchestra.run.smithi184.stdout:/run/netns 2024-09-16T15:47:59.270 INFO:teuthology.orchestra.run.smithi184.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-09-16T15:47:59.270 INFO:teuthology.orchestra.run.smithi184.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-09-16T15:47:59.270 INFO:teuthology.orchestra.run.smithi184.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-09-16T15:47:59.270 INFO:teuthology.orchestra.run.smithi184.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-09-16T15:47:59.271 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:47:59.271 DEBUG:teuthology.orchestra.run.smithi184:> ls /sys/fs/fuse/connections 2024-09-16T15:47:59.319 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-09-16T15:47:59.319 DEBUG:teuthology.orchestra.run.smithi184:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.1 --id 1) 2024-09-16T15:47:59.364 DEBUG:teuthology.orchestra.run.smithi184:> sudo modprobe fuse 2024-09-16T15:47:59.379 DEBUG:teuthology.orchestra.run.smithi184:> cat /proc/self/mounts | awk '{print $2}' 2024-09-16T15:47:59.427 INFO:teuthology.orchestra.run.smithi184.stdout:/sys 2024-09-16T15:47:59.427 INFO:teuthology.orchestra.run.smithi184.stdout:/proc 2024-09-16T15:47:59.428 INFO:teuthology.orchestra.run.smithi184.stdout:/dev 2024-09-16T15:47:59.428 INFO:teuthology.orchestra.run.smithi184.stdout:/dev/pts 2024-09-16T15:47:59.428 INFO:teuthology.orchestra.run.smithi184.stdout:/run 2024-09-16T15:47:59.428 INFO:teuthology.orchestra.run.smithi184.stdout:/ 2024-09-16T15:47:59.428 INFO:teuthology.orchestra.run.smithi184.stdout:/sys/kernel/security 2024-09-16T15:47:59.428 INFO:teuthology.orchestra.run.smithi184.stdout:/dev/shm 2024-09-16T15:47:59.428 INFO:teuthology.orchestra.run.smithi184.stdout:/run/lock 2024-09-16T15:47:59.428 INFO:teuthology.orchestra.run.smithi184.stdout:/sys/fs/cgroup 2024-09-16T15:47:59.428 INFO:teuthology.orchestra.run.smithi184.stdout:/sys/fs/pstore 2024-09-16T15:47:59.428 INFO:teuthology.orchestra.run.smithi184.stdout:/sys/fs/bpf 2024-09-16T15:47:59.428 INFO:teuthology.orchestra.run.smithi184.stdout:/proc/sys/fs/binfmt_misc 2024-09-16T15:47:59.428 INFO:teuthology.orchestra.run.smithi184.stdout:/dev/hugepages 2024-09-16T15:47:59.428 INFO:teuthology.orchestra.run.smithi184.stdout:/dev/mqueue 2024-09-16T15:47:59.428 INFO:teuthology.orchestra.run.smithi184.stdout:/sys/kernel/debug 2024-09-16T15:47:59.429 INFO:teuthology.orchestra.run.smithi184.stdout:/sys/kernel/tracing 2024-09-16T15:47:59.429 INFO:teuthology.orchestra.run.smithi184.stdout:/proc/fs/nfsd 2024-09-16T15:47:59.429 INFO:teuthology.orchestra.run.smithi184.stdout:/sys/fs/fuse/connections 2024-09-16T15:47:59.429 INFO:teuthology.orchestra.run.smithi184.stdout:/sys/kernel/config 2024-09-16T15:47:59.429 INFO:teuthology.orchestra.run.smithi184.stdout:/run/credentials/systemd-sysusers.service 2024-09-16T15:47:59.429 INFO:teuthology.orchestra.run.smithi184.stdout:/run/rpc_pipefs 2024-09-16T15:47:59.429 INFO:teuthology.orchestra.run.smithi184.stdout:/run/user/1000 2024-09-16T15:47:59.429 INFO:teuthology.orchestra.run.smithi184.stdout:/var/lib/ceph 2024-09-16T15:47:59.429 INFO:teuthology.orchestra.run.smithi184.stdout:/var/lib/ceph/osd/ceph-4 2024-09-16T15:47:59.429 INFO:teuthology.orchestra.run.smithi184.stdout:/var/lib/ceph/osd/ceph-5 2024-09-16T15:47:59.429 INFO:teuthology.orchestra.run.smithi184.stdout:/var/lib/ceph/osd/ceph-6 2024-09-16T15:47:59.429 INFO:teuthology.orchestra.run.smithi184.stdout:/var/lib/ceph/osd/ceph-7 2024-09-16T15:47:59.429 INFO:teuthology.orchestra.run.smithi184.stdout:/run/netns 2024-09-16T15:47:59.430 INFO:teuthology.orchestra.run.smithi184.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-09-16T15:47:59.430 INFO:teuthology.orchestra.run.smithi184.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-09-16T15:47:59.430 INFO:teuthology.orchestra.run.smithi184.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-09-16T15:47:59.430 INFO:teuthology.orchestra.run.smithi184.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-09-16T15:47:59.430 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:47:59.430 DEBUG:teuthology.orchestra.run.smithi184:> ls /sys/fs/fuse/connections 2024-09-16T15:47:59.445 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi184.stderr:2024-09-16T15:47:59.441+0000 7fdcad762580 -1 init, newargv = 0x563d1ceb4ab0 newargc=15 2024-09-16T15:47:59.445 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi184.stderr:ceph-fuse[35560]: starting ceph client 2024-09-16T15:47:59.451 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi184.stderr:ceph-fuse[35560]: starting fuse 2024-09-16T15:48:00.551 DEBUG:teuthology.orchestra.run.smithi184:> sudo modprobe fuse 2024-09-16T15:48:00.567 DEBUG:teuthology.orchestra.run.smithi184:> cat /proc/self/mounts | awk '{print $2}' 2024-09-16T15:48:00.616 INFO:teuthology.orchestra.run.smithi184.stdout:/sys 2024-09-16T15:48:00.616 INFO:teuthology.orchestra.run.smithi184.stdout:/proc 2024-09-16T15:48:00.616 INFO:teuthology.orchestra.run.smithi184.stdout:/dev 2024-09-16T15:48:00.616 INFO:teuthology.orchestra.run.smithi184.stdout:/dev/pts 2024-09-16T15:48:00.616 INFO:teuthology.orchestra.run.smithi184.stdout:/run 2024-09-16T15:48:00.616 INFO:teuthology.orchestra.run.smithi184.stdout:/ 2024-09-16T15:48:00.616 INFO:teuthology.orchestra.run.smithi184.stdout:/sys/kernel/security 2024-09-16T15:48:00.617 INFO:teuthology.orchestra.run.smithi184.stdout:/dev/shm 2024-09-16T15:48:00.617 INFO:teuthology.orchestra.run.smithi184.stdout:/run/lock 2024-09-16T15:48:00.617 INFO:teuthology.orchestra.run.smithi184.stdout:/sys/fs/cgroup 2024-09-16T15:48:00.617 INFO:teuthology.orchestra.run.smithi184.stdout:/sys/fs/pstore 2024-09-16T15:48:00.617 INFO:teuthology.orchestra.run.smithi184.stdout:/sys/fs/bpf 2024-09-16T15:48:00.617 INFO:teuthology.orchestra.run.smithi184.stdout:/proc/sys/fs/binfmt_misc 2024-09-16T15:48:00.617 INFO:teuthology.orchestra.run.smithi184.stdout:/dev/hugepages 2024-09-16T15:48:00.617 INFO:teuthology.orchestra.run.smithi184.stdout:/dev/mqueue 2024-09-16T15:48:00.617 INFO:teuthology.orchestra.run.smithi184.stdout:/sys/kernel/debug 2024-09-16T15:48:00.617 INFO:teuthology.orchestra.run.smithi184.stdout:/sys/kernel/tracing 2024-09-16T15:48:00.617 INFO:teuthology.orchestra.run.smithi184.stdout:/proc/fs/nfsd 2024-09-16T15:48:00.617 INFO:teuthology.orchestra.run.smithi184.stdout:/sys/fs/fuse/connections 2024-09-16T15:48:00.617 INFO:teuthology.orchestra.run.smithi184.stdout:/sys/kernel/config 2024-09-16T15:48:00.617 INFO:teuthology.orchestra.run.smithi184.stdout:/run/credentials/systemd-sysusers.service 2024-09-16T15:48:00.618 INFO:teuthology.orchestra.run.smithi184.stdout:/run/rpc_pipefs 2024-09-16T15:48:00.618 INFO:teuthology.orchestra.run.smithi184.stdout:/run/user/1000 2024-09-16T15:48:00.618 INFO:teuthology.orchestra.run.smithi184.stdout:/var/lib/ceph 2024-09-16T15:48:00.618 INFO:teuthology.orchestra.run.smithi184.stdout:/var/lib/ceph/osd/ceph-4 2024-09-16T15:48:00.618 INFO:teuthology.orchestra.run.smithi184.stdout:/var/lib/ceph/osd/ceph-5 2024-09-16T15:48:00.618 INFO:teuthology.orchestra.run.smithi184.stdout:/var/lib/ceph/osd/ceph-6 2024-09-16T15:48:00.618 INFO:teuthology.orchestra.run.smithi184.stdout:/var/lib/ceph/osd/ceph-7 2024-09-16T15:48:00.618 INFO:teuthology.orchestra.run.smithi184.stdout:/run/netns 2024-09-16T15:48:00.618 INFO:teuthology.orchestra.run.smithi184.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-09-16T15:48:00.618 INFO:teuthology.orchestra.run.smithi184.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-09-16T15:48:00.618 INFO:teuthology.orchestra.run.smithi184.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-09-16T15:48:00.618 INFO:teuthology.orchestra.run.smithi184.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-09-16T15:48:00.618 INFO:teuthology.orchestra.run.smithi184.stdout:/home/ubuntu/cephtest/mnt.1 2024-09-16T15:48:00.619 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:48:00.619 DEBUG:teuthology.orchestra.run.smithi184:> ls /sys/fs/fuse/connections 2024-09-16T15:48:00.667 INFO:teuthology.orchestra.run.smithi184.stdout:45 2024-09-16T15:48:00.667 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [45] 2024-09-16T15:48:00.667 DEBUG:teuthology.orchestra.run.smithi184:> sudo stdin-killer -- python3 -c ' 2024-09-16T15:48:00.667 DEBUG:teuthology.orchestra.run.smithi184:> import glob 2024-09-16T15:48:00.667 DEBUG:teuthology.orchestra.run.smithi184:> import re 2024-09-16T15:48:00.667 DEBUG:teuthology.orchestra.run.smithi184:> import os 2024-09-16T15:48:00.668 DEBUG:teuthology.orchestra.run.smithi184:> import subprocess 2024-09-16T15:48:00.668 DEBUG:teuthology.orchestra.run.smithi184:> 2024-09-16T15:48:00.668 DEBUG:teuthology.orchestra.run.smithi184:> def _find_admin_socket(client_name): 2024-09-16T15:48:00.668 DEBUG:teuthology.orchestra.run.smithi184:> asok_path = "/var/run/ceph/ceph-client.1.*.asok" 2024-09-16T15:48:00.668 DEBUG:teuthology.orchestra.run.smithi184:> files = glob.glob(asok_path) 2024-09-16T15:48:00.668 DEBUG:teuthology.orchestra.run.smithi184:> mountpoint = "/home/ubuntu/cephtest/mnt.1" 2024-09-16T15:48:00.668 DEBUG:teuthology.orchestra.run.smithi184:> 2024-09-16T15:48:00.668 DEBUG:teuthology.orchestra.run.smithi184:> # Given a non-glob path, it better be there 2024-09-16T15:48:00.668 DEBUG:teuthology.orchestra.run.smithi184:> if "*" not in asok_path: 2024-09-16T15:48:00.668 DEBUG:teuthology.orchestra.run.smithi184:> assert(len(files) == 1) 2024-09-16T15:48:00.668 DEBUG:teuthology.orchestra.run.smithi184:> return files[0] 2024-09-16T15:48:00.668 DEBUG:teuthology.orchestra.run.smithi184:> 2024-09-16T15:48:00.668 DEBUG:teuthology.orchestra.run.smithi184:> for f in files: 2024-09-16T15:48:00.668 DEBUG:teuthology.orchestra.run.smithi184:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-09-16T15:48:00.669 DEBUG:teuthology.orchestra.run.smithi184:> if os.path.exists("/proc/{0}".format(pid)): 2024-09-16T15:48:00.669 DEBUG:teuthology.orchestra.run.smithi184:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-09-16T15:48:00.669 DEBUG:teuthology.orchestra.run.smithi184:> contents = proc_f.read() 2024-09-16T15:48:00.669 DEBUG:teuthology.orchestra.run.smithi184:> if mountpoint in contents: 2024-09-16T15:48:00.669 DEBUG:teuthology.orchestra.run.smithi184:> return f 2024-09-16T15:48:00.669 DEBUG:teuthology.orchestra.run.smithi184:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-09-16T15:48:00.669 DEBUG:teuthology.orchestra.run.smithi184:> 2024-09-16T15:48:00.669 DEBUG:teuthology.orchestra.run.smithi184:> print(_find_admin_socket("client.1")) 2024-09-16T15:48:00.669 DEBUG:teuthology.orchestra.run.smithi184:> ' 2024-09-16T15:48:00.792 INFO:teuthology.orchestra.run.smithi184.stdout:/var/run/ceph/ceph-client.1.35560.asok 2024-09-16T15:48:00.796 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:48:00 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-16T15:48:00.803 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.1.35560.asok 2024-09-16T15:48:00.803 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:48:00.804 DEBUG:teuthology.orchestra.run.smithi184:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.1.35560.asok status 2024-09-16T15:48:00.955 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-16T15:48:00.955 INFO:teuthology.orchestra.run.smithi184.stdout: "metadata": { 2024-09-16T15:48:00.955 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph_sha1": "33f95d2ae1bcbe7eb04404303f179eeeaa6982df", 2024-09-16T15:48:00.955 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph_version": "ceph version 19.3.0-4938-g33f95d2a (33f95d2ae1bcbe7eb04404303f179eeeaa6982df) squid (dev)", 2024-09-16T15:48:00.955 INFO:teuthology.orchestra.run.smithi184.stdout: "entity_id": "1", 2024-09-16T15:48:00.956 INFO:teuthology.orchestra.run.smithi184.stdout: "hostname": "smithi184", 2024-09-16T15:48:00.956 INFO:teuthology.orchestra.run.smithi184.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.1", 2024-09-16T15:48:00.956 INFO:teuthology.orchestra.run.smithi184.stdout: "pid": "35560", 2024-09-16T15:48:00.956 INFO:teuthology.orchestra.run.smithi184.stdout: "root": "/" 2024-09-16T15:48:00.956 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-16T15:48:00.956 INFO:teuthology.orchestra.run.smithi184.stdout: "dentry_count": 0, 2024-09-16T15:48:00.956 INFO:teuthology.orchestra.run.smithi184.stdout: "dentry_pinned_count": 0, 2024-09-16T15:48:00.956 INFO:teuthology.orchestra.run.smithi184.stdout: "id": 5728, 2024-09-16T15:48:00.956 INFO:teuthology.orchestra.run.smithi184.stdout: "inst": { 2024-09-16T15:48:00.956 INFO:teuthology.orchestra.run.smithi184.stdout: "name": { 2024-09-16T15:48:00.956 INFO:teuthology.orchestra.run.smithi184.stdout: "type": "client", 2024-09-16T15:48:00.956 INFO:teuthology.orchestra.run.smithi184.stdout: "num": 5728 2024-09-16T15:48:00.956 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-16T15:48:00.956 INFO:teuthology.orchestra.run.smithi184.stdout: "addr": { 2024-09-16T15:48:00.957 INFO:teuthology.orchestra.run.smithi184.stdout: "type": "v1", 2024-09-16T15:48:00.957 INFO:teuthology.orchestra.run.smithi184.stdout: "addr": "192.168.0.1:0", 2024-09-16T15:48:00.957 INFO:teuthology.orchestra.run.smithi184.stdout: "nonce": 2933060184 2024-09-16T15:48:00.957 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-16T15:48:00.957 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-16T15:48:00.957 INFO:teuthology.orchestra.run.smithi184.stdout: "addr": { 2024-09-16T15:48:00.957 INFO:teuthology.orchestra.run.smithi184.stdout: "type": "v1", 2024-09-16T15:48:00.957 INFO:teuthology.orchestra.run.smithi184.stdout: "addr": "192.168.0.1:0", 2024-09-16T15:48:00.957 INFO:teuthology.orchestra.run.smithi184.stdout: "nonce": 2933060184 2024-09-16T15:48:00.957 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-16T15:48:00.957 INFO:teuthology.orchestra.run.smithi184.stdout: "inst_str": "client.5728 192.168.0.1:0/2933060184", 2024-09-16T15:48:00.957 INFO:teuthology.orchestra.run.smithi184.stdout: "addr_str": "192.168.0.1:0/2933060184", 2024-09-16T15:48:00.957 INFO:teuthology.orchestra.run.smithi184.stdout: "inode_count": 1, 2024-09-16T15:48:00.957 INFO:teuthology.orchestra.run.smithi184.stdout: "mds_epoch": 53, 2024-09-16T15:48:00.958 INFO:teuthology.orchestra.run.smithi184.stdout: "osd_epoch": 85, 2024-09-16T15:48:00.958 INFO:teuthology.orchestra.run.smithi184.stdout: "osd_epoch_barrier": 0, 2024-09-16T15:48:00.958 INFO:teuthology.orchestra.run.smithi184.stdout: "blocklisted": false, 2024-09-16T15:48:00.958 INFO:teuthology.orchestra.run.smithi184.stdout: "fs_name": "cephfs" 2024-09-16T15:48:00.958 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-16T15:48:00.971 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:48:00.971 DEBUG:teuthology.orchestra.run.smithi184:> stat --file-system '--printf=%T 2024-09-16T15:48:00.972 DEBUG:teuthology.orchestra.run.smithi184:> ' -- /home/ubuntu/cephtest/mnt.1 2024-09-16T15:48:00.978 INFO:teuthology.orchestra.run.smithi184.stdout:fuseblk 2024-09-16T15:48:00.978 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2024-09-16T15:48:00.978 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:48:00.978 DEBUG:teuthology.orchestra.run.smithi184:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.1 2024-09-16T15:48:01.032 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs flag set enable_multiple true --yes-i-really-mean-it 2024-09-16T15:48:01.177 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:01.167+0000 7f29d7e34640 1 -- 172.21.15.123:0/874865598 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f29bc004da0 msgr2=0x7f29bc005220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:48:01.178 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:01.167+0000 7f29d7e34640 1 --2- 172.21.15.123:0/874865598 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f29bc004da0 0x7f29bc005220 secure :-1 s=READY pgs=968 cs=0 l=1 rev1=1 crypto rx=0x7f29cc005fd0 tx=0x7f29cc01cd60 comp rx=0 tx=0).stop 2024-09-16T15:48:01.178 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:01.167+0000 7f29d63d1640 1 -- 172.21.15.123:0/874865598 reap_dead start 2024-09-16T15:48:01.178 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:01.167+0000 7f29d7e34640 1 -- 172.21.15.123:0/874865598 shutdown_connections 2024-09-16T15:48:01.178 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:01.167+0000 7f29d7e34640 1 -- 172.21.15.123:0/874865598 >> 172.21.15.123:0/874865598 conn(0x7f29d00b1630 msgr2=0x7f29d00b1a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:48:01.178 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:01.167+0000 7f29d7e34640 1 -- 172.21.15.123:0/874865598 shutdown_connections 2024-09-16T15:48:01.178 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:01.167+0000 7f29d7e34640 1 -- 172.21.15.123:0/874865598 wait complete. 2024-09-16T15:48:01.178 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:01.171+0000 7f29d7e34640 1 Processor -- start 2024-09-16T15:48:01.178 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:01.171+0000 7f29d7e34640 1 -- start start 2024-09-16T15:48:01.178 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:01.171+0000 7f29d7e34640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f29d013edd0 0x7f29d014fde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:48:01.178 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:01.171+0000 7f29d7e34640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f29d0150320 0x7f29d01f86b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:48:01.178 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:01.171+0000 7f29d7e34640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f29d01f8bf0 0x7f29d0148520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:48:01.178 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:01.171+0000 7f29d7e34640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f29d00706d0 con 0x7f29d0150320 2024-09-16T15:48:01.179 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:01.171+0000 7f29d7e34640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f29d006ff40 con 0x7f29d01f8bf0 2024-09-16T15:48:01.179 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:01.171+0000 7f29d7e34640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f29d0070850 con 0x7f29d013edd0 2024-09-16T15:48:01.179 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:01.171+0000 7f29d53cf640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f29d0150320 0x7f29d01f86b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:48:01.179 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:01.171+0000 7f29d53cf640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f29d0150320 0x7f29d01f86b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:43930/0 (socket says 172.21.15.123:43930) 2024-09-16T15:48:01.179 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:01.171+0000 7f29d53cf640 1 -- 172.21.15.123:0/3168968414 learned_addr learned my addr 172.21.15.123:0/3168968414 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:48:01.179 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:01.171+0000 7f29d63d1640 1 --2- 172.21.15.123:0/3168968414 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f29d01f8bf0 0x7f29d0148520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:48:01.179 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:01.171+0000 7f29d5bd0640 1 --2- 172.21.15.123:0/3168968414 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f29d013edd0 0x7f29d014fde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:48:01.179 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:01.171+0000 7f29d53cf640 1 -- 172.21.15.123:0/3168968414 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f29d013edd0 msgr2=0x7f29d014fde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:48:01.179 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:01.171+0000 7f29d53cf640 1 --2- 172.21.15.123:0/3168968414 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f29d013edd0 0x7f29d014fde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:01.179 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:01.171+0000 7f29d53cf640 1 -- 172.21.15.123:0/3168968414 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f29d01f8bf0 msgr2=0x7f29d0148520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:48:01.179 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:01.171+0000 7f29d53cf640 1 --2- 172.21.15.123:0/3168968414 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f29d01f8bf0 0x7f29d0148520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:01.179 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:01.171+0000 7f29d53cf640 1 -- 172.21.15.123:0/3168968414 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f29d014ed40 con 0x7f29d0150320 2024-09-16T15:48:01.180 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:01.171+0000 7f29d53cf640 1 --2- 172.21.15.123:0/3168968414 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f29d0150320 0x7f29d01f86b0 secure :-1 s=READY pgs=969 cs=0 l=1 rev1=1 crypto rx=0x7f29c400e720 tx=0x7f29c400ebe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:48:01.180 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:01.171+0000 7f29c27fc640 1 -- 172.21.15.123:0/3168968414 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29c4015070 con 0x7f29d0150320 2024-09-16T15:48:01.180 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:01.171+0000 7f29c27fc640 1 -- 172.21.15.123:0/3168968414 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f29c4004330 con 0x7f29d0150320 2024-09-16T15:48:01.180 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:01.171+0000 7f29d7e34640 1 -- 172.21.15.123:0/3168968414 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f29d014f020 con 0x7f29d0150320 2024-09-16T15:48:01.180 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:01.171+0000 7f29c27fc640 1 -- 172.21.15.123:0/3168968414 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29c400c840 con 0x7f29d0150320 2024-09-16T15:48:01.180 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:01.171+0000 7f29d7e34640 1 -- 172.21.15.123:0/3168968414 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f29d014f5e0 con 0x7f29d0150320 2024-09-16T15:48:01.180 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:01.171+0000 7f29d7e34640 1 -- 172.21.15.123:0/3168968414 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f29d01f9000 con 0x7f29d0150320 2024-09-16T15:48:01.181 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:01.175+0000 7f29c27fc640 1 -- 172.21.15.123:0/3168968414 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 16) ==== 97182+0+0 (secure 0 0 0) 0x7f29c400c9e0 con 0x7f29d0150320 2024-09-16T15:48:01.181 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:01.175+0000 7f29c27fc640 1 --2- 172.21.15.123:0/3168968414 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f29b4073010 0x7f29b40754d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:48:01.181 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:01.175+0000 7f29c27fc640 1 -- 172.21.15.123:0/3168968414 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 5709+0+0 (secure 0 0 0) 0x7f29c4094900 con 0x7f29d0150320 2024-09-16T15:48:01.181 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:01.175+0000 7f29d5bd0640 1 --2- 172.21.15.123:0/3168968414 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f29b4073010 0x7f29b40754d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:48:01.181 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:01.175+0000 7f29d5bd0640 1 --2- 172.21.15.123:0/3168968414 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f29b4073010 0x7f29b40754d0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f29c8001e60 tx=0x7f29c8006e50 comp rx=0 tx=0).ready entity=mgr.4150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:48:01.182 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:01.179+0000 7f29c27fc640 1 -- 172.21.15.123:0/3168968414 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f29d01f9000 con 0x7f29d0150320 2024-09-16T15:48:01.388 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:01.383+0000 7f29d7e34640 1 -- 172.21.15.123:0/3168968414 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "fs flag set", "flag_name": "enable_multiple", "val": "true", "yes_i_really_mean_it": true} v 0) -- 0x7f29d006e880 con 0x7f29d0150320 2024-09-16T15:48:02.141 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:02.135+0000 7f29c27fc640 1 -- 172.21.15.123:0/3168968414 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "fs flag set", "flag_name": "enable_multiple", "val": "true", "yes_i_really_mean_it": true}]=0 v54) ==== 136+0+0 (secure 0 0 0) 0x7f29c405fa10 con 0x7f29d0150320 2024-09-16T15:48:02.144 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:02.139+0000 7f29d7e34640 1 -- 172.21.15.123:0/3168968414 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f29b4073010 msgr2=0x7f29b40754d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:48:02.145 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:02.139+0000 7f29d7e34640 1 --2- 172.21.15.123:0/3168968414 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f29b4073010 0x7f29b40754d0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f29c8001e60 tx=0x7f29c8006e50 comp rx=0 tx=0).stop 2024-09-16T15:48:02.145 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:02.139+0000 7f29d7e34640 1 -- 172.21.15.123:0/3168968414 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f29d0150320 msgr2=0x7f29d01f86b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:48:02.145 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:02.139+0000 7f29d7e34640 1 --2- 172.21.15.123:0/3168968414 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f29d0150320 0x7f29d01f86b0 secure :-1 s=READY pgs=969 cs=0 l=1 rev1=1 crypto rx=0x7f29c400e720 tx=0x7f29c400ebe0 comp rx=0 tx=0).stop 2024-09-16T15:48:02.145 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:02.139+0000 7f29d7e34640 1 -- 172.21.15.123:0/3168968414 shutdown_connections 2024-09-16T15:48:02.145 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:02.139+0000 7f29d7e34640 1 --2- 172.21.15.123:0/3168968414 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f29b4073010 0x7f29b40754d0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:02.145 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:02.139+0000 7f29d7e34640 1 --2- 172.21.15.123:0/3168968414 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f29d01f8bf0 0x7f29d0148520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:02.145 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:02.139+0000 7f29d7e34640 1 --2- 172.21.15.123:0/3168968414 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f29d0150320 0x7f29d01f86b0 unknown :-1 s=CLOSED pgs=969 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:02.145 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:02.139+0000 7f29d7e34640 1 --2- 172.21.15.123:0/3168968414 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f29d013edd0 0x7f29d014fde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:02.145 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:02.139+0000 7f29d7e34640 1 -- 172.21.15.123:0/3168968414 >> 172.21.15.123:0/3168968414 conn(0x7f29d00b1630 msgr2=0x7f29d00ae160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:48:02.145 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:02.139+0000 7f29d7e34640 1 -- 172.21.15.123:0/3168968414 shutdown_connections 2024-09-16T15:48:02.145 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:02.139+0000 7f29d7e34640 1 -- 172.21.15.123:0/3168968414 wait complete. 2024-09-16T15:48:02.163 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2024-09-16T15:48:02.294 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:02.287+0000 7f3362eb9640 1 -- 172.21.15.123:0/552044592 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f335c074600 msgr2=0x7f335c06e750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:48:02.294 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:02.287+0000 7f3362eb9640 1 --2- 172.21.15.123:0/552044592 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f335c074600 0x7f335c06e750 secure :-1 s=READY pgs=971 cs=0 l=1 rev1=1 crypto rx=0x7f335c0cd650 tx=0x7f335401cac0 comp rx=0 tx=0).stop 2024-09-16T15:48:02.294 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:02.287+0000 7f3362eb9640 1 -- 172.21.15.123:0/552044592 shutdown_connections 2024-09-16T15:48:02.294 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:02.287+0000 7f3362eb9640 1 --2- 172.21.15.123:0/552044592 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f335c074600 0x7f335c06e750 unknown :-1 s=CLOSED pgs=971 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:02.294 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:02.287+0000 7f3362eb9640 1 --2- 172.21.15.123:0/552044592 >> v2:172.21.15.184:3300/0 conn(0x7f335c15bca0 0x7f335c0d0630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:02.294 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:02.287+0000 7f3362eb9640 1 -- 172.21.15.123:0/552044592 >> 172.21.15.123:0/552044592 conn(0x7f335c0cfb10 msgr2=0x7f335c0cff20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:48:02.294 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:02.287+0000 7f3362eb9640 1 -- 172.21.15.123:0/552044592 shutdown_connections 2024-09-16T15:48:02.294 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:02.287+0000 7f3362eb9640 1 -- 172.21.15.123:0/552044592 wait complete. 2024-09-16T15:48:02.295 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:02.291+0000 7f3362eb9640 1 Processor -- start 2024-09-16T15:48:02.295 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:02.291+0000 7f3362eb9640 1 -- start start 2024-09-16T15:48:02.296 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:02.291+0000 7f3362eb9640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f335c0726c0 0x7f335c157d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:48:02.296 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:02.291+0000 7f3362eb9640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f335c074600 0x7f335c0c7780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:48:02.297 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:02.291+0000 7f3362eb9640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f335c15bca0 0x7f335c0c7cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:48:02.297 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:02.291+0000 7f3362eb9640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f335c168a50 con 0x7f335c0726c0 2024-09-16T15:48:02.297 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:02.291+0000 7f3362eb9640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f335c06fa10 con 0x7f335c074600 2024-09-16T15:48:02.297 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:02.291+0000 7f3362eb9640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f335c168bd0 con 0x7f335c15bca0 2024-09-16T15:48:02.297 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:02.291+0000 7f3360c55640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f335c0726c0 0x7f335c157d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:48:02.297 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:02.291+0000 7f3360c55640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f335c0726c0 0x7f335c157d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:43944/0 (socket says 172.21.15.123:43944) 2024-09-16T15:48:02.297 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:02.291+0000 7f3360c55640 1 -- 172.21.15.123:0/306303376 learned_addr learned my addr 172.21.15.123:0/306303376 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:48:02.297 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:02.291+0000 7f3360c55640 1 -- 172.21.15.123:0/306303376 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f335c15bca0 msgr2=0x7f335c0c7cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:48:02.297 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:02.291+0000 7f3360c55640 1 --2- 172.21.15.123:0/306303376 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f335c15bca0 0x7f335c0c7cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:02.297 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:02.291+0000 7f3360c55640 1 -- 172.21.15.123:0/306303376 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f335c074600 msgr2=0x7f335c0c7780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:48:02.297 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:02.291+0000 7f3360c55640 1 --2- 172.21.15.123:0/306303376 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f335c074600 0x7f335c0c7780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:02.297 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:02.291+0000 7f3360c55640 1 -- 172.21.15.123:0/306303376 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f335c0c4ff0 con 0x7f335c0726c0 2024-09-16T15:48:02.297 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:02.291+0000 7f3360c55640 1 --2- 172.21.15.123:0/306303376 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f335c0726c0 0x7f335c157d20 secure :-1 s=READY pgs=972 cs=0 l=1 rev1=1 crypto rx=0x7f334c00c9f0 tx=0x7f334c00ceb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:48:02.298 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:02.291+0000 7f33597fa640 1 -- 172.21.15.123:0/306303376 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f334c007bb0 con 0x7f335c0726c0 2024-09-16T15:48:02.298 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:02.291+0000 7f33597fa640 1 -- 172.21.15.123:0/306303376 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f334c007e40 con 0x7f335c0726c0 2024-09-16T15:48:02.298 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:02.291+0000 7f3362eb9640 1 -- 172.21.15.123:0/306303376 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f335c0c5f30 con 0x7f335c0726c0 2024-09-16T15:48:02.298 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:02.291+0000 7f33597fa640 1 -- 172.21.15.123:0/306303376 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f334c00f690 con 0x7f335c0726c0 2024-09-16T15:48:02.298 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:02.291+0000 7f3362eb9640 1 -- 172.21.15.123:0/306303376 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f335c0c8400 con 0x7f335c0726c0 2024-09-16T15:48:02.299 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:02.291+0000 7f3362eb9640 1 -- 172.21.15.123:0/306303376 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f335c06e750 con 0x7f335c0726c0 2024-09-16T15:48:02.299 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:02.295+0000 7f33597fa640 1 -- 172.21.15.123:0/306303376 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 16) ==== 97182+0+0 (secure 0 0 0) 0x7f334c00f830 con 0x7f335c0726c0 2024-09-16T15:48:02.299 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:02.295+0000 7f33597fa640 1 --2- 172.21.15.123:0/306303376 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f3348072fe0 0x7f33480754a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:48:02.299 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:02.295+0000 7f335bfff640 1 --2- 172.21.15.123:0/306303376 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f3348072fe0 0x7f33480754a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:48:02.299 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:02.295+0000 7f33597fa640 1 -- 172.21.15.123:0/306303376 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 5709+0+0 (secure 0 0 0) 0x7f334c0948d0 con 0x7f335c0726c0 2024-09-16T15:48:02.299 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:02.295+0000 7f335bfff640 1 --2- 172.21.15.123:0/306303376 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f3348072fe0 0x7f33480754a0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f33500046c0 tx=0x7f3350007910 comp rx=0 tx=0).ready entity=mgr.4150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:48:02.303 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:02.299+0000 7f33597fa640 1 -- 172.21.15.123:0/306303376 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f334c014030 con 0x7f335c0726c0 2024-09-16T15:48:02.481 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:02.475+0000 7f3362eb9640 1 -- 172.21.15.123:0/306303376 --> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) -- 0x7f335c0c5260 con 0x7f3348072fe0 2024-09-16T15:48:02.484 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:02.479+0000 7f33597fa640 1 -- 172.21.15.123:0/306303376 <== mgr.4150 v2:172.21.15.123:6836/2464759676 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+41 (secure 0 0 0) 0x7f335c0c5260 con 0x7f3348072fe0 2024-09-16T15:48:02.484 INFO:teuthology.orchestra.run.smithi123.stdout:[ 2024-09-16T15:48:02.485 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-09-16T15:48:02.485 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "cephfs" 2024-09-16T15:48:02.485 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-16T15:48:02.485 INFO:teuthology.orchestra.run.smithi123.stdout:] 2024-09-16T15:48:02.487 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:02.483+0000 7f3362eb9640 1 -- 172.21.15.123:0/306303376 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f3348072fe0 msgr2=0x7f33480754a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:48:02.487 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:02.483+0000 7f3362eb9640 1 --2- 172.21.15.123:0/306303376 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f3348072fe0 0x7f33480754a0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f33500046c0 tx=0x7f3350007910 comp rx=0 tx=0).stop 2024-09-16T15:48:02.487 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:02.483+0000 7f3362eb9640 1 -- 172.21.15.123:0/306303376 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f335c0726c0 msgr2=0x7f335c157d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:48:02.487 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:02.483+0000 7f3362eb9640 1 --2- 172.21.15.123:0/306303376 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f335c0726c0 0x7f335c157d20 secure :-1 s=READY pgs=972 cs=0 l=1 rev1=1 crypto rx=0x7f334c00c9f0 tx=0x7f334c00ceb0 comp rx=0 tx=0).stop 2024-09-16T15:48:02.487 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:02.483+0000 7f3362eb9640 1 -- 172.21.15.123:0/306303376 shutdown_connections 2024-09-16T15:48:02.487 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:02.483+0000 7f3362eb9640 1 --2- 172.21.15.123:0/306303376 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f3348072fe0 0x7f33480754a0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:02.487 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:02.483+0000 7f3362eb9640 1 --2- 172.21.15.123:0/306303376 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f335c15bca0 0x7f335c0c7cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:02.487 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:02.483+0000 7f3362eb9640 1 --2- 172.21.15.123:0/306303376 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f335c074600 0x7f335c0c7780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:02.487 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:02.483+0000 7f3362eb9640 1 --2- 172.21.15.123:0/306303376 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f335c0726c0 0x7f335c157d20 unknown :-1 s=CLOSED pgs=972 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:02.487 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:02.483+0000 7f3362eb9640 1 -- 172.21.15.123:0/306303376 >> 172.21.15.123:0/306303376 conn(0x7f335c0cfb10 msgr2=0x7f335c0b45a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:48:02.487 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:02.483+0000 7f3362eb9640 1 -- 172.21.15.123:0/306303376 shutdown_connections 2024-09-16T15:48:02.487 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:02.483+0000 7f3362eb9640 1 -- 172.21.15.123:0/306303376 wait complete. 2024-09-16T15:48:02.502 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-09-16T15:48:02.683 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:02.679+0000 7f61d99d5640 1 -- 172.21.15.123:0/1412289623 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f61d415bc80 msgr2=0x7f61d415f6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:48:02.683 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:02.679+0000 7f61d99d5640 1 --2- 172.21.15.123:0/1412289623 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f61d415bc80 0x7f61d415f6b0 secure :-1 s=READY pgs=973 cs=0 l=1 rev1=1 crypto rx=0x7f61cc009c80 tx=0x7f61cc01cd10 comp rx=0 tx=0).stop 2024-09-16T15:48:02.683 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:02.679+0000 7f61d99d5640 1 -- 172.21.15.123:0/1412289623 shutdown_connections 2024-09-16T15:48:02.683 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:02.679+0000 7f61d99d5640 1 --2- 172.21.15.123:0/1412289623 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f61d415bc80 0x7f61d415f6b0 unknown :-1 s=CLOSED pgs=973 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:02.684 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:02.679+0000 7f61d99d5640 1 --2- 172.21.15.123:0/1412289623 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f61d40cf030 0x7f61d406fca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:02.684 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:02.679+0000 7f61d99d5640 1 -- 172.21.15.123:0/1412289623 >> 172.21.15.123:0/1412289623 conn(0x7f61d40b1710 msgr2=0x7f61d40b1b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:48:02.684 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:02.679+0000 7f61d99d5640 1 -- 172.21.15.123:0/1412289623 shutdown_connections 2024-09-16T15:48:02.684 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:02.679+0000 7f61d99d5640 1 -- 172.21.15.123:0/1412289623 wait complete. 2024-09-16T15:48:02.684 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:02.679+0000 7f61d99d5640 1 Processor -- start 2024-09-16T15:48:02.684 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:02.679+0000 7f61d99d5640 1 -- start start 2024-09-16T15:48:02.686 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:02.679+0000 7f61d99d5640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f61d40701e0 0x7f61d4147630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:48:02.686 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:02.679+0000 7f61d99d5640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f61d40cf030 0x7f61d41469b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:48:02.687 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:02.679+0000 7f61d99d5640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f61d415bc80 0x7f61d4146ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:48:02.687 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:02.679+0000 7f61d99d5640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f61d4169310 con 0x7f61d40cf030 2024-09-16T15:48:02.687 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:02.679+0000 7f61d99d5640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f61d4169190 con 0x7f61d40701e0 2024-09-16T15:48:02.687 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:02.679+0000 7f61d99d5640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f61d4169490 con 0x7f61d415bc80 2024-09-16T15:48:02.687 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:02.679+0000 7f61ca7fc640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f61d40cf030 0x7f61d41469b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:48:02.687 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:02.679+0000 7f61ca7fc640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f61d40cf030 0x7f61d41469b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:57754/0 (socket says 172.21.15.123:57754) 2024-09-16T15:48:02.687 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:02.679+0000 7f61ca7fc640 1 -- 172.21.15.123:0/2030681751 learned_addr learned my addr 172.21.15.123:0/2030681751 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:48:02.687 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:02.679+0000 7f61d37fe640 1 --2- 172.21.15.123:0/2030681751 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f61d415bc80 0x7f61d4146ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:48:02.687 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:02.679+0000 7f61d2ffd640 1 --2- 172.21.15.123:0/2030681751 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f61d40701e0 0x7f61d4147630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:48:02.687 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:02.679+0000 7f61ca7fc640 1 -- 172.21.15.123:0/2030681751 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f61d415bc80 msgr2=0x7f61d4146ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:48:02.687 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:02.679+0000 7f61ca7fc640 1 --2- 172.21.15.123:0/2030681751 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f61d415bc80 0x7f61d4146ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:02.687 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:02.679+0000 7f61ca7fc640 1 -- 172.21.15.123:0/2030681751 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f61d40701e0 msgr2=0x7f61d4147630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:48:02.688 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:02.679+0000 7f61ca7fc640 1 --2- 172.21.15.123:0/2030681751 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f61d40701e0 0x7f61d4147630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:02.688 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:02.679+0000 7f61ca7fc640 1 -- 172.21.15.123:0/2030681751 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f61d4073590 con 0x7f61d40cf030 2024-09-16T15:48:02.688 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:02.679+0000 7f61ca7fc640 1 --2- 172.21.15.123:0/2030681751 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f61d40cf030 0x7f61d41469b0 secure :-1 s=READY pgs=974 cs=0 l=1 rev1=1 crypto rx=0x7f61bc00c450 tx=0x7f61bc00c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:48:02.688 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:02.679+0000 7f61d0ff9640 1 -- 172.21.15.123:0/2030681751 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61bc012830 con 0x7f61d40cf030 2024-09-16T15:48:02.688 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:02.683+0000 7f61d0ff9640 1 -- 172.21.15.123:0/2030681751 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f61bc017070 con 0x7f61d40cf030 2024-09-16T15:48:02.688 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:02.683+0000 7f61d99d5640 1 -- 172.21.15.123:0/2030681751 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f61d4154810 con 0x7f61d40cf030 2024-09-16T15:48:02.688 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:02.683+0000 7f61d0ff9640 1 -- 172.21.15.123:0/2030681751 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61bc012d80 con 0x7f61d40cf030 2024-09-16T15:48:02.688 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:02.683+0000 7f61d99d5640 1 -- 172.21.15.123:0/2030681751 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f61d40c62e0 con 0x7f61d40cf030 2024-09-16T15:48:02.688 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:02.683+0000 7f61d99d5640 1 -- 172.21.15.123:0/2030681751 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f61d406fca0 con 0x7f61d40cf030 2024-09-16T15:48:02.689 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:02.683+0000 7f61d0ff9640 1 -- 172.21.15.123:0/2030681751 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 16) ==== 97182+0+0 (secure 0 0 0) 0x7f61bc007020 con 0x7f61d40cf030 2024-09-16T15:48:02.689 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:02.683+0000 7f61d0ff9640 1 --2- 172.21.15.123:0/2030681751 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f61b4072fe0 0x7f61b40754a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:48:02.689 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:02.683+0000 7f61d0ff9640 1 -- 172.21.15.123:0/2030681751 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 5709+0+0 (secure 0 0 0) 0x7f61bc094450 con 0x7f61d40cf030 2024-09-16T15:48:02.689 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:02.683+0000 7f61d2ffd640 1 --2- 172.21.15.123:0/2030681751 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f61b4072fe0 0x7f61b40754a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:48:02.689 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:02.683+0000 7f61d2ffd640 1 --2- 172.21.15.123:0/2030681751 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f61b4072fe0 0x7f61b40754a0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f61c40058d0 tx=0x7f61c4005860 comp rx=0 tx=0).ready entity=mgr.4150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:48:02.692 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:02.687+0000 7f61d0ff9640 1 -- 172.21.15.123:0/2030681751 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f61bc05f5f0 con 0x7f61d40cf030 2024-09-16T15:48:02.862 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:02.859+0000 7f61d99d5640 1 -- 172.21.15.123:0/2030681751 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) -- 0x7f61d41522c0 con 0x7f61d40cf030 2024-09-16T15:48:02.862 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:02.859+0000 7f61d0ff9640 1 -- 172.21.15.123:0/2030681751 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v36)=0 v36) ==== 122+0+0 (secure 0 0 0) 0x7f61bc05eef0 con 0x7f61d40cf030 2024-09-16T15:48:02.865 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:02.859+0000 7f61d99d5640 1 -- 172.21.15.123:0/2030681751 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f61b4072fe0 msgr2=0x7f61b40754a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:48:02.865 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:02.859+0000 7f61d99d5640 1 --2- 172.21.15.123:0/2030681751 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f61b4072fe0 0x7f61b40754a0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f61c40058d0 tx=0x7f61c4005860 comp rx=0 tx=0).stop 2024-09-16T15:48:02.865 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:02.859+0000 7f61d99d5640 1 -- 172.21.15.123:0/2030681751 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f61d40cf030 msgr2=0x7f61d41469b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:48:02.865 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:02.859+0000 7f61d99d5640 1 --2- 172.21.15.123:0/2030681751 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f61d40cf030 0x7f61d41469b0 secure :-1 s=READY pgs=974 cs=0 l=1 rev1=1 crypto rx=0x7f61bc00c450 tx=0x7f61bc00c910 comp rx=0 tx=0).stop 2024-09-16T15:48:02.865 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:02.859+0000 7f61d99d5640 1 -- 172.21.15.123:0/2030681751 shutdown_connections 2024-09-16T15:48:02.866 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:02.859+0000 7f61d99d5640 1 --2- 172.21.15.123:0/2030681751 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f61b4072fe0 0x7f61b40754a0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:02.866 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:02.859+0000 7f61d99d5640 1 --2- 172.21.15.123:0/2030681751 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f61d415bc80 0x7f61d4146ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:02.866 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:02.859+0000 7f61d99d5640 1 --2- 172.21.15.123:0/2030681751 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f61d40cf030 0x7f61d41469b0 unknown :-1 s=CLOSED pgs=974 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:02.866 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:02.859+0000 7f61d99d5640 1 --2- 172.21.15.123:0/2030681751 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f61d40701e0 0x7f61d4147630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:02.866 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:02.859+0000 7f61d99d5640 1 -- 172.21.15.123:0/2030681751 >> 172.21.15.123:0/2030681751 conn(0x7f61d40b1710 msgr2=0x7f61d415c130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:48:02.866 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:02.859+0000 7f61d99d5640 1 -- 172.21.15.123:0/2030681751 shutdown_connections 2024-09-16T15:48:02.866 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:02.859+0000 7f61d99d5640 1 -- 172.21.15.123:0/2030681751 wait complete. 2024-09-16T15:48:02.881 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume create 'abcd@#' 2024-09-16T15:48:02.995 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:02.991+0000 7f8a525a7640 1 -- 172.21.15.123:0/2339457524 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8a4c06e3b0 msgr2=0x7f8a4c06e860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:48:02.995 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:02.991+0000 7f8a525a7640 1 --2- 172.21.15.123:0/2339457524 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8a4c06e3b0 0x7f8a4c06e860 secure :-1 s=READY pgs=975 cs=0 l=1 rev1=1 crypto rx=0x7f8a44009c80 tx=0x7f8a4401cb30 comp rx=0 tx=0).stop 2024-09-16T15:48:02.995 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:02.991+0000 7f8a525a7640 1 -- 172.21.15.123:0/2339457524 shutdown_connections 2024-09-16T15:48:02.995 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:02.991+0000 7f8a525a7640 1 --2- 172.21.15.123:0/2339457524 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8a4c06e3b0 0x7f8a4c06e860 unknown :-1 s=CLOSED pgs=975 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:02.995 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:02.991+0000 7f8a525a7640 1 --2- 172.21.15.123:0/2339457524 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f8a4c0cbaf0 0x7f8a4c0cbf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:02.995 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:02.991+0000 7f8a525a7640 1 -- 172.21.15.123:0/2339457524 >> 172.21.15.123:0/2339457524 conn(0x7f8a4c0b1710 msgr2=0x7f8a4c0b1b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:48:02.995 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:02.991+0000 7f8a525a7640 1 -- 172.21.15.123:0/2339457524 shutdown_connections 2024-09-16T15:48:02.995 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:02.991+0000 7f8a525a7640 1 -- 172.21.15.123:0/2339457524 wait complete. 2024-09-16T15:48:02.996 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:02.991+0000 7f8a525a7640 1 Processor -- start 2024-09-16T15:48:02.996 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:02.991+0000 7f8a525a7640 1 -- start start 2024-09-16T15:48:02.998 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:02.991+0000 7f8a525a7640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8a4c06e3b0 0x7f8a4c155fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:48:02.999 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:02.991+0000 7f8a525a7640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f8a4c0cbaf0 0x7f8a4c158920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:48:02.999 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:02.991+0000 7f8a525a7640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f8a4c15d780 0x7f8a4c0c2840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:48:02.999 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:02.991+0000 7f8a525a7640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f8a4c167600 con 0x7f8a4c06e3b0 2024-09-16T15:48:02.999 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:02.991+0000 7f8a525a7640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f8a4c167480 con 0x7f8a4c15d780 2024-09-16T15:48:02.999 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:02.991+0000 7f8a525a7640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f8a4c167780 con 0x7f8a4c0cbaf0 2024-09-16T15:48:02.999 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:02.991+0000 7f8a4bfff640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8a4c06e3b0 0x7f8a4c155fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:48:02.999 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:02.991+0000 7f8a4bfff640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8a4c06e3b0 0x7f8a4c155fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:57768/0 (socket says 172.21.15.123:57768) 2024-09-16T15:48:02.999 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:02.991+0000 7f8a4bfff640 1 -- 172.21.15.123:0/3543310967 learned_addr learned my addr 172.21.15.123:0/3543310967 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:48:02.999 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:02.991+0000 7f8a50b44640 1 --2- 172.21.15.123:0/3543310967 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f8a4c15d780 0x7f8a4c0c2840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:48:02.999 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:02.991+0000 7f8a437fe640 1 --2- 172.21.15.123:0/3543310967 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f8a4c0cbaf0 0x7f8a4c158920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:48:02.999 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:02.991+0000 7f8a4bfff640 1 -- 172.21.15.123:0/3543310967 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f8a4c0cbaf0 msgr2=0x7f8a4c158920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:48:03.000 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:02.991+0000 7f8a4bfff640 1 --2- 172.21.15.123:0/3543310967 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f8a4c0cbaf0 0x7f8a4c158920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:03.000 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:02.991+0000 7f8a4bfff640 1 -- 172.21.15.123:0/3543310967 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f8a4c15d780 msgr2=0x7f8a4c0c2840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:48:03.000 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:02.991+0000 7f8a4bfff640 1 --2- 172.21.15.123:0/3543310967 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f8a4c15d780 0x7f8a4c0c2840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:03.000 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:02.991+0000 7f8a4bfff640 1 -- 172.21.15.123:0/3543310967 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8a4c15d000 con 0x7f8a4c06e3b0 2024-09-16T15:48:03.000 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:02.991+0000 7f8a4bfff640 1 --2- 172.21.15.123:0/3543310967 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8a4c06e3b0 0x7f8a4c155fc0 secure :-1 s=READY pgs=976 cs=0 l=1 rev1=1 crypto rx=0x7f8a3c00b3e0 tx=0x7f8a3c00b8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:48:03.000 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:02.991+0000 7f8a497fa640 1 -- 172.21.15.123:0/3543310967 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a3c015020 con 0x7f8a4c06e3b0 2024-09-16T15:48:03.000 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:02.995+0000 7f8a497fa640 1 -- 172.21.15.123:0/3543310967 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f8a3c011940 con 0x7f8a4c06e3b0 2024-09-16T15:48:03.000 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:02.995+0000 7f8a525a7640 1 -- 172.21.15.123:0/3543310967 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8a4c158e60 con 0x7f8a4c06e3b0 2024-09-16T15:48:03.000 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:02.995+0000 7f8a497fa640 1 -- 172.21.15.123:0/3543310967 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a3c011d90 con 0x7f8a4c06e3b0 2024-09-16T15:48:03.000 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:02.995+0000 7f8a525a7640 1 -- 172.21.15.123:0/3543310967 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8a4c0c3570 con 0x7f8a4c06e3b0 2024-09-16T15:48:03.000 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:02.995+0000 7f8a525a7640 1 -- 172.21.15.123:0/3543310967 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8a4c0c3770 con 0x7f8a4c06e3b0 2024-09-16T15:48:03.001 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:02.995+0000 7f8a497fa640 1 -- 172.21.15.123:0/3543310967 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 16) ==== 97182+0+0 (secure 0 0 0) 0x7f8a3c018020 con 0x7f8a4c06e3b0 2024-09-16T15:48:03.001 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:02.995+0000 7f8a497fa640 1 --2- 172.21.15.123:0/3543310967 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f8a2c072fe0 0x7f8a2c0754a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:48:03.001 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:02.995+0000 7f8a437fe640 1 --2- 172.21.15.123:0/3543310967 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f8a2c072fe0 0x7f8a2c0754a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:48:03.001 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:02.995+0000 7f8a497fa640 1 -- 172.21.15.123:0/3543310967 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 5709+0+0 (secure 0 0 0) 0x7f8a3c093fa0 con 0x7f8a4c06e3b0 2024-09-16T15:48:03.002 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:02.995+0000 7f8a437fe640 1 --2- 172.21.15.123:0/3543310967 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f8a2c072fe0 0x7f8a2c0754a0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f8a34005fd0 tx=0x7f8a3400a4e0 comp rx=0 tx=0).ready entity=mgr.4150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:48:03.004 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:02.999+0000 7f8a497fa640 1 -- 172.21.15.123:0/3543310967 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f8a4c0c3770 con 0x7f8a4c06e3b0 2024-09-16T15:48:03.188 INFO:teuthology.orchestra.run.smithi123.stderr:Invalid command: invalid chars @# in abcd@# 2024-09-16T15:48:03.188 INFO:teuthology.orchestra.run.smithi123.stderr:fs volume create [] : Create a CephFS volume 2024-09-16T15:48:03.188 INFO:teuthology.orchestra.run.smithi123.stderr:Error EINVAL: invalid command 2024-09-16T15:48:03.193 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-16T15:48:03.193 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume create testvol 2024-09-16T15:48:03.317 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:03.311+0000 7f3b0ec5a640 1 -- 172.21.15.123:0/3377137127 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3b0815a170 msgr2=0x7f3b08072f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:48:03.317 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:03.311+0000 7f3b0ec5a640 1 --2- 172.21.15.123:0/3377137127 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3b0815a170 0x7f3b08072f30 secure :-1 s=READY pgs=977 cs=0 l=1 rev1=1 crypto rx=0x7f3af8009900 tx=0x7f3af801c7e0 comp rx=0 tx=0).stop 2024-09-16T15:48:03.317 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:03.311+0000 7f3b0ec5a640 1 -- 172.21.15.123:0/3377137127 shutdown_connections 2024-09-16T15:48:03.317 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:03.311+0000 7f3b0ec5a640 1 --2- 172.21.15.123:0/3377137127 >> v2:172.21.15.184:3300/0 conn(0x7f3b080ca650 0x7f3b080cbc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:03.317 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:03.311+0000 7f3b0ec5a640 1 --2- 172.21.15.123:0/3377137127 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3b0815a170 0x7f3b08072f30 unknown :-1 s=CLOSED pgs=977 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:03.318 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:03.311+0000 7f3b0ec5a640 1 -- 172.21.15.123:0/3377137127 >> 172.21.15.123:0/3377137127 conn(0x7f3b080b1710 msgr2=0x7f3b080b1b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:48:03.318 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:03.311+0000 7f3b0ec5a640 1 -- 172.21.15.123:0/3377137127 shutdown_connections 2024-09-16T15:48:03.318 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:03.311+0000 7f3b0ec5a640 1 -- 172.21.15.123:0/3377137127 wait complete. 2024-09-16T15:48:03.319 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:03.311+0000 7f3b0ec5a640 1 Processor -- start 2024-09-16T15:48:03.319 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:03.315+0000 7f3b0ec5a640 1 -- start start 2024-09-16T15:48:03.319 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:03.315+0000 7f3b0ec5a640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3b080ca650 0x7f3b08148e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:48:03.320 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:03.315+0000 7f3b0ec5a640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f3b0815a170 0x7f3b08149440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:48:03.320 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:03.315+0000 7f3b0ec5a640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3b08149980 0x7f3b0806ee70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:48:03.320 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:03.315+0000 7f3b0ec5a640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f3b0806e830 con 0x7f3b080ca650 2024-09-16T15:48:03.320 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:03.315+0000 7f3b0ec5a640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f3b0806e6b0 con 0x7f3b08149980 2024-09-16T15:48:03.320 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:03.315+0000 7f3b0ec5a640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f3b0806e9b0 con 0x7f3b0815a170 2024-09-16T15:48:03.320 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:03.315+0000 7f3b0c9f6640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3b080ca650 0x7f3b08148e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:48:03.320 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:03.315+0000 7f3b0c9f6640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3b080ca650 0x7f3b08148e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:57776/0 (socket says 172.21.15.123:57776) 2024-09-16T15:48:03.320 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:03.315+0000 7f3b0c9f6640 1 -- 172.21.15.123:0/3552811107 learned_addr learned my addr 172.21.15.123:0/3552811107 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:48:03.320 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:03.315+0000 7f3b0c9f6640 1 -- 172.21.15.123:0/3552811107 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f3b0815a170 msgr2=0x7f3b08149440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:48:03.320 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:03.315+0000 7f3b0c9f6640 1 --2- 172.21.15.123:0/3552811107 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f3b0815a170 0x7f3b08149440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:03.320 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:03.315+0000 7f3b0c9f6640 1 -- 172.21.15.123:0/3552811107 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3b08149980 msgr2=0x7f3b0806ee70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:48:03.320 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:03.315+0000 7f3b0c9f6640 1 --2- 172.21.15.123:0/3552811107 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3b08149980 0x7f3b0806ee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:03.321 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:03.315+0000 7f3b0c9f6640 1 -- 172.21.15.123:0/3552811107 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3b080b5690 con 0x7f3b080ca650 2024-09-16T15:48:03.321 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:03.315+0000 7f3b0c9f6640 1 --2- 172.21.15.123:0/3552811107 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3b080ca650 0x7f3b08148e10 secure :-1 s=READY pgs=978 cs=0 l=1 rev1=1 crypto rx=0x7f3af8009850 tx=0x7f3af80a57e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:48:03.321 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:03.315+0000 7f3b057fa640 1 -- 172.21.15.123:0/3552811107 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3af80b2040 con 0x7f3b080ca650 2024-09-16T15:48:03.321 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:03.315+0000 7f3b057fa640 1 -- 172.21.15.123:0/3552811107 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f3af80044b0 con 0x7f3b080ca650 2024-09-16T15:48:03.321 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:03.315+0000 7f3b0ec5a640 1 -- 172.21.15.123:0/3552811107 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3b080b53b0 con 0x7f3b080ca650 2024-09-16T15:48:03.321 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:03.315+0000 7f3b057fa640 1 -- 172.21.15.123:0/3552811107 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3af8003840 con 0x7f3b080ca650 2024-09-16T15:48:03.321 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:03.315+0000 7f3b0ec5a640 1 -- 172.21.15.123:0/3552811107 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3b0815df20 con 0x7f3b080ca650 2024-09-16T15:48:03.322 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:03.315+0000 7f3b0ec5a640 1 -- 172.21.15.123:0/3552811107 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3b080cbc60 con 0x7f3b080ca650 2024-09-16T15:48:03.322 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:03.315+0000 7f3b057fa640 1 -- 172.21.15.123:0/3552811107 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 16) ==== 97182+0+0 (secure 0 0 0) 0x7f3af80bd060 con 0x7f3b080ca650 2024-09-16T15:48:03.322 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:03.315+0000 7f3b057fa640 1 --2- 172.21.15.123:0/3552811107 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f3af4072fe0 0x7f3af40754a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:48:03.322 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:03.315+0000 7f3b057fa640 1 -- 172.21.15.123:0/3552811107 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 5709+0+0 (secure 0 0 0) 0x7f3af812db60 con 0x7f3b080ca650 2024-09-16T15:48:03.322 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:03.315+0000 7f3b07fff640 1 --2- 172.21.15.123:0/3552811107 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f3af4072fe0 0x7f3af40754a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:48:03.322 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:03.319+0000 7f3b07fff640 1 --2- 172.21.15.123:0/3552811107 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f3af4072fe0 0x7f3af40754a0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f3b081537f0 tx=0x7f3afc005c00 comp rx=0 tx=0).ready entity=mgr.4150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:48:03.325 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:03.319+0000 7f3b057fa640 1 -- 172.21.15.123:0/3552811107 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f3af8016490 con 0x7f3b080ca650 2024-09-16T15:48:03.510 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:03.507+0000 7f3b0ec5a640 1 -- 172.21.15.123:0/3552811107 --> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] -- mgr_command(tid 0: {"prefix": "fs volume create", "name": "testvol", "target": ["mon-mgr", ""]}) -- 0x7f3b08153820 con 0x7f3af4072fe0 2024-09-16T15:48:06.212 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:06.207+0000 7f3b057fa640 1 -- 172.21.15.123:0/3552811107 <== mgr.4150 v2:172.21.15.123:6836/2464759676 1 ==== mgr_command_reply(tid 0: 0 Volume created successfully (no MDS daemons created)) ==== 60+0+0 (secure 0 0 0) 0x7f3b08153820 con 0x7f3af4072fe0 2024-09-16T15:48:06.216 INFO:teuthology.orchestra.run.smithi123.stderr:Volume created successfully (no MDS daemons created) 2024-09-16T15:48:06.217 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:06.211+0000 7f3b0ec5a640 1 -- 172.21.15.123:0/3552811107 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f3af4072fe0 msgr2=0x7f3af40754a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:48:06.217 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:06.211+0000 7f3b0ec5a640 1 --2- 172.21.15.123:0/3552811107 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f3af4072fe0 0x7f3af40754a0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f3b081537f0 tx=0x7f3afc005c00 comp rx=0 tx=0).stop 2024-09-16T15:48:06.217 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:06.211+0000 7f3b0ec5a640 1 -- 172.21.15.123:0/3552811107 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3b080ca650 msgr2=0x7f3b08148e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:48:06.217 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:06.211+0000 7f3b0ec5a640 1 --2- 172.21.15.123:0/3552811107 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3b080ca650 0x7f3b08148e10 secure :-1 s=READY pgs=978 cs=0 l=1 rev1=1 crypto rx=0x7f3af8009850 tx=0x7f3af80a57e0 comp rx=0 tx=0).stop 2024-09-16T15:48:06.217 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:06.211+0000 7f3b0ec5a640 1 -- 172.21.15.123:0/3552811107 shutdown_connections 2024-09-16T15:48:06.217 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:06.211+0000 7f3b0ec5a640 1 --2- 172.21.15.123:0/3552811107 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f3af4072fe0 0x7f3af40754a0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:06.217 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:06.211+0000 7f3b0ec5a640 1 --2- 172.21.15.123:0/3552811107 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3b08149980 0x7f3b0806ee70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:06.217 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:06.211+0000 7f3b0ec5a640 1 --2- 172.21.15.123:0/3552811107 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f3b0815a170 0x7f3b08149440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:06.217 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:06.211+0000 7f3b0ec5a640 1 --2- 172.21.15.123:0/3552811107 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3b080ca650 0x7f3b08148e10 unknown :-1 s=CLOSED pgs=978 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:06.217 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:06.211+0000 7f3b0ec5a640 1 -- 172.21.15.123:0/3552811107 >> 172.21.15.123:0/3552811107 conn(0x7f3b080b1710 msgr2=0x7f3b08156b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:48:06.218 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:06.211+0000 7f3b0ec5a640 1 -- 172.21.15.123:0/3552811107 shutdown_connections 2024-09-16T15:48:06.218 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:06.211+0000 7f3b0ec5a640 1 -- 172.21.15.123:0/3552811107 wait complete. 2024-09-16T15:48:06.239 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolumegroup create testvol 'abcd@#' 2024-09-16T15:48:06.365 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:06.359+0000 7f00d3e1f640 1 -- 172.21.15.123:0/2138430908 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f00b4003a10 msgr2=0x7f00b4003e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:48:06.365 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:06.359+0000 7f00d3e1f640 1 --2- 172.21.15.123:0/2138430908 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f00b4003a10 0x7f00b4003e30 secure :-1 s=READY pgs=980 cs=0 l=1 rev1=1 crypto rx=0x7f00bc009ee0 tx=0x7f00bc01d5f0 comp rx=0 tx=0).stop 2024-09-16T15:48:06.365 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:06.359+0000 7f00d23bc640 1 -- 172.21.15.123:0/2138430908 reap_dead start 2024-09-16T15:48:06.365 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:06.359+0000 7f00d3e1f640 1 -- 172.21.15.123:0/2138430908 shutdown_connections 2024-09-16T15:48:06.365 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:06.359+0000 7f00d3e1f640 1 -- 172.21.15.123:0/2138430908 >> 172.21.15.123:0/2138430908 conn(0x7f00cc0d0bc0 msgr2=0x7f00cc0d0fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:48:06.365 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:06.359+0000 7f00d3e1f640 1 -- 172.21.15.123:0/2138430908 shutdown_connections 2024-09-16T15:48:06.366 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:06.359+0000 7f00d3e1f640 1 -- 172.21.15.123:0/2138430908 wait complete. 2024-09-16T15:48:06.366 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:06.363+0000 7f00d3e1f640 1 Processor -- start 2024-09-16T15:48:06.366 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:06.363+0000 7f00d3e1f640 1 -- start start 2024-09-16T15:48:06.368 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:06.363+0000 7f00d3e1f640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f00b4003a10 0x7f00cc1f89f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:48:06.368 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:06.363+0000 7f00d3e1f640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f00cc0c63b0 0x7f00cc0703f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:48:06.368 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:06.363+0000 7f00d3e1f640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f00cc0c6830 0x7f00cc070930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:48:06.368 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:06.363+0000 7f00d3e1f640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f00cc166d30 con 0x7f00cc0c63b0 2024-09-16T15:48:06.368 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:06.363+0000 7f00d3e1f640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f00cc166bb0 con 0x7f00cc0c6830 2024-09-16T15:48:06.368 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:06.363+0000 7f00d3e1f640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f00cc166eb0 con 0x7f00b4003a10 2024-09-16T15:48:06.368 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:06.363+0000 7f00d13ba640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f00cc0c63b0 0x7f00cc0703f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:48:06.369 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:06.363+0000 7f00d13ba640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f00cc0c63b0 0x7f00cc0703f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:57788/0 (socket says 172.21.15.123:57788) 2024-09-16T15:48:06.369 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:06.363+0000 7f00d13ba640 1 -- 172.21.15.123:0/2547822115 learned_addr learned my addr 172.21.15.123:0/2547822115 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:48:06.369 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:06.363+0000 7f00d1bbb640 1 --2- 172.21.15.123:0/2547822115 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f00b4003a10 0x7f00cc1f89f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:48:06.369 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:06.363+0000 7f00d23bc640 1 --2- 172.21.15.123:0/2547822115 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f00cc0c6830 0x7f00cc070930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:48:06.369 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:06.363+0000 7f00d13ba640 1 -- 172.21.15.123:0/2547822115 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f00b4003a10 msgr2=0x7f00cc1f89f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:48:06.369 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:06.363+0000 7f00d13ba640 1 --2- 172.21.15.123:0/2547822115 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f00b4003a10 0x7f00cc1f89f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:06.369 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:06.363+0000 7f00d13ba640 1 -- 172.21.15.123:0/2547822115 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f00cc0c6830 msgr2=0x7f00cc070930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:48:06.369 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:06.363+0000 7f00d13ba640 1 --2- 172.21.15.123:0/2547822115 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f00cc0c6830 0x7f00cc070930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:06.369 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:06.363+0000 7f00d13ba640 1 -- 172.21.15.123:0/2547822115 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f00cc0c5d60 con 0x7f00cc0c63b0 2024-09-16T15:48:06.369 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:06.363+0000 7f00d23bc640 1 --2- 172.21.15.123:0/2547822115 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f00cc0c6830 0x7f00cc070930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T15:48:06.369 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:06.363+0000 7f00d13ba640 1 --2- 172.21.15.123:0/2547822115 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f00cc0c63b0 0x7f00cc0703f0 secure :-1 s=READY pgs=981 cs=0 l=1 rev1=1 crypto rx=0x7f00c000d7b0 tx=0x7f00c000dc70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:48:06.369 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:06.363+0000 7f00ca7fc640 1 -- 172.21.15.123:0/2547822115 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00c0018070 con 0x7f00cc0c63b0 2024-09-16T15:48:06.369 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:06.363+0000 7f00ca7fc640 1 -- 172.21.15.123:0/2547822115 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f00c00050d0 con 0x7f00cc0c63b0 2024-09-16T15:48:06.370 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:06.363+0000 7f00d3e1f640 1 -- 172.21.15.123:0/2547822115 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f00cc0c4a40 con 0x7f00cc0c63b0 2024-09-16T15:48:06.370 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:06.363+0000 7f00ca7fc640 1 -- 172.21.15.123:0/2547822115 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00c000b6c0 con 0x7f00cc0c63b0 2024-09-16T15:48:06.370 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:06.363+0000 7f00d3e1f640 1 -- 172.21.15.123:0/2547822115 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f00cc0c5000 con 0x7f00cc0c63b0 2024-09-16T15:48:06.371 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:06.363+0000 7f00d3e1f640 1 -- 172.21.15.123:0/2547822115 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f00cc15f6c0 con 0x7f00cc0c63b0 2024-09-16T15:48:06.371 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:06.363+0000 7f00ca7fc640 1 -- 172.21.15.123:0/2547822115 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 16) ==== 97182+0+0 (secure 0 0 0) 0x7f00c001b070 con 0x7f00cc0c63b0 2024-09-16T15:48:06.371 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:06.367+0000 7f00ca7fc640 1 --2- 172.21.15.123:0/2547822115 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f00b8073010 0x7f00b80754d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:48:06.371 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:06.367+0000 7f00ca7fc640 1 -- 172.21.15.123:0/2547822115 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 6543+0+0 (secure 0 0 0) 0x7f00c0094160 con 0x7f00cc0c63b0 2024-09-16T15:48:06.371 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:06.367+0000 7f00d1bbb640 1 --2- 172.21.15.123:0/2547822115 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f00b8073010 0x7f00b80754d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:48:06.371 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:06.367+0000 7f00d1bbb640 1 --2- 172.21.15.123:0/2547822115 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f00b8073010 0x7f00b80754d0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f00bc0b0000 tx=0x7f00bc01a390 comp rx=0 tx=0).ready entity=mgr.4150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:48:06.374 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:06.371+0000 7f00ca7fc640 1 -- 172.21.15.123:0/2547822115 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f00c005e800 con 0x7f00cc0c63b0 2024-09-16T15:48:06.573 INFO:teuthology.orchestra.run.smithi123.stderr:Invalid command: invalid chars #@ in abcd@# 2024-09-16T15:48:06.573 INFO:teuthology.orchestra.run.smithi123.stderr:fs subvolumegroup create [] [] [] [] [] : Create a CephFS subvolume group in a volume, and optionally, with a specific data pool layout, and a specific numeric mode 2024-09-16T15:48:06.573 INFO:teuthology.orchestra.run.smithi123.stderr:Error EINVAL: invalid command 2024-09-16T15:48:06.578 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-16T15:48:06.578 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume create testvol 'abcd@#' 2024-09-16T15:48:06.711 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:06.707+0000 7f184d3b2640 1 -- 172.21.15.123:0/562192651 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f18480701e0 msgr2=0x7f184815b570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:48:06.711 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:06.707+0000 7f184d3b2640 1 --2- 172.21.15.123:0/562192651 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f18480701e0 0x7f184815b570 secure :-1 s=READY pgs=982 cs=0 l=1 rev1=1 crypto rx=0x7f184806db30 tx=0x7f183001c830 comp rx=0 tx=0).stop 2024-09-16T15:48:06.711 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:06.707+0000 7f184d3b2640 1 -- 172.21.15.123:0/562192651 shutdown_connections 2024-09-16T15:48:06.711 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:06.707+0000 7f184d3b2640 1 --2- 172.21.15.123:0/562192651 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f18480c31c0 0x7f184815bab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:06.711 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:06.707+0000 7f184d3b2640 1 --2- 172.21.15.123:0/562192651 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f18480701e0 0x7f184815b570 unknown :-1 s=CLOSED pgs=982 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:06.712 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:06.707+0000 7f184d3b2640 1 --2- 172.21.15.123:0/562192651 >> v2:172.21.15.184:3300/0 conn(0x7f18480cf030 0x7f184806fca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:06.712 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:06.707+0000 7f184d3b2640 1 -- 172.21.15.123:0/562192651 >> 172.21.15.123:0/562192651 conn(0x7f18480b1710 msgr2=0x7f18480b1b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:48:06.712 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:06.707+0000 7f184d3b2640 1 -- 172.21.15.123:0/562192651 shutdown_connections 2024-09-16T15:48:06.712 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:06.707+0000 7f184d3b2640 1 -- 172.21.15.123:0/562192651 wait complete. 2024-09-16T15:48:06.712 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:06.707+0000 7f184d3b2640 1 Processor -- start 2024-09-16T15:48:06.712 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:06.707+0000 7f184d3b2640 1 -- start start 2024-09-16T15:48:06.714 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:06.707+0000 7f184d3b2640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f18480701e0 0x7f184814fdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:48:06.715 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:06.707+0000 7f184d3b2640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f18480c31c0 0x7f184814f230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:48:06.715 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:06.707+0000 7f184d3b2640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f18480cf030 0x7f1848150d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:48:06.715 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:06.707+0000 7f184d3b2640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f184815f830 con 0x7f18480c31c0 2024-09-16T15:48:06.715 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:06.707+0000 7f184d3b2640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f184815f6b0 con 0x7f18480cf030 2024-09-16T15:48:06.715 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:06.707+0000 7f184d3b2640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f184815f9b0 con 0x7f18480701e0 2024-09-16T15:48:06.715 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:06.707+0000 7f18467fc640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f18480c31c0 0x7f184814f230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:48:06.715 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:06.707+0000 7f18467fc640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f18480c31c0 0x7f184814f230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:57808/0 (socket says 172.21.15.123:57808) 2024-09-16T15:48:06.715 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:06.707+0000 7f18467fc640 1 -- 172.21.15.123:0/3614911252 learned_addr learned my addr 172.21.15.123:0/3614911252 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:48:06.715 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:06.707+0000 7f18477fe640 1 --2- 172.21.15.123:0/3614911252 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f18480cf030 0x7f1848150d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:48:06.715 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:06.707+0000 7f1846ffd640 1 --2- 172.21.15.123:0/3614911252 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f18480701e0 0x7f184814fdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:48:06.715 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:06.711+0000 7f18467fc640 1 -- 172.21.15.123:0/3614911252 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f18480701e0 msgr2=0x7f184814fdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:48:06.716 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:06.711+0000 7f18467fc640 1 --2- 172.21.15.123:0/3614911252 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f18480701e0 0x7f184814fdd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:06.716 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:06.711+0000 7f18467fc640 1 -- 172.21.15.123:0/3614911252 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f18480cf030 msgr2=0x7f1848150d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:48:06.716 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:06.711+0000 7f18467fc640 1 --2- 172.21.15.123:0/3614911252 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f18480cf030 0x7f1848150d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:06.716 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:06.711+0000 7f18467fc640 1 -- 172.21.15.123:0/3614911252 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f18481fbbc0 con 0x7f18480c31c0 2024-09-16T15:48:06.716 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:06.711+0000 7f18467fc640 1 --2- 172.21.15.123:0/3614911252 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f18480c31c0 0x7f184814f230 secure :-1 s=READY pgs=983 cs=0 l=1 rev1=1 crypto rx=0x7f18300b0040 tx=0x7f18300a5640 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:48:06.716 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:06.711+0000 7f183f7fe640 1 -- 172.21.15.123:0/3614911252 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18300b2040 con 0x7f18480c31c0 2024-09-16T15:48:06.716 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:06.711+0000 7f184d3b2640 1 -- 172.21.15.123:0/3614911252 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f18481fae00 con 0x7f18480c31c0 2024-09-16T15:48:06.716 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:06.711+0000 7f183f7fe640 1 -- 172.21.15.123:0/3614911252 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f1830004490 con 0x7f18480c31c0 2024-09-16T15:48:06.716 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:06.711+0000 7f183f7fe640 1 -- 172.21.15.123:0/3614911252 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1830005290 con 0x7f18480c31c0 2024-09-16T15:48:06.716 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:06.711+0000 7f184d3b2640 1 -- 172.21.15.123:0/3614911252 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f18481fb3c0 con 0x7f18480c31c0 2024-09-16T15:48:06.717 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:06.711+0000 7f184d3b2640 1 -- 172.21.15.123:0/3614911252 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f184806fca0 con 0x7f18480c31c0 2024-09-16T15:48:06.717 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:06.711+0000 7f183f7fe640 1 -- 172.21.15.123:0/3614911252 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 16) ==== 97182+0+0 (secure 0 0 0) 0x7f1830005430 con 0x7f18480c31c0 2024-09-16T15:48:06.717 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:06.711+0000 7f183f7fe640 1 --2- 172.21.15.123:0/3614911252 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f1828072fe0 0x7f18280754a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:48:06.717 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:06.711+0000 7f183f7fe640 1 -- 172.21.15.123:0/3614911252 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 6543+0+0 (secure 0 0 0) 0x7f183012e400 con 0x7f18480c31c0 2024-09-16T15:48:06.717 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:06.711+0000 7f1846ffd640 1 --2- 172.21.15.123:0/3614911252 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f1828072fe0 0x7f18280754a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:48:06.718 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:06.711+0000 7f1846ffd640 1 --2- 172.21.15.123:0/3614911252 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f1828072fe0 0x7f18280754a0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f1848074890 tx=0x7f183800d4b0 comp rx=0 tx=0).ready entity=mgr.4150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:48:06.721 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:06.715+0000 7f183f7fe640 1 -- 172.21.15.123:0/3614911252 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f18300164b0 con 0x7f18480c31c0 2024-09-16T15:48:06.920 INFO:teuthology.orchestra.run.smithi123.stderr:Invalid command: invalid chars @# in abcd@# 2024-09-16T15:48:06.920 INFO:teuthology.orchestra.run.smithi123.stderr:fs subvolume create [] [] [] [] [] [] [--namespace-isolated] [] : Create a CephFS subvolume in a volume, and optionally, with a specific size (in bytes), a specific data pool layout, a specific mode, in a specific subvolume group and in separate RADOS namespace 2024-09-16T15:48:06.920 INFO:teuthology.orchestra.run.smithi123.stderr:Error EINVAL: invalid command 2024-09-16T15:48:06.926 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-16T15:48:06.926 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume rm testvol --yes-i-really-mean-it 2024-09-16T15:48:07.060 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:07.055+0000 7f9650fb6640 1 -- 172.21.15.123:0/1015270373 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9634004010 msgr2=0x7f9634004810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:48:07.060 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:07.055+0000 7f9650fb6640 1 --2- 172.21.15.123:0/1015270373 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9634004010 0x7f9634004810 secure :-1 s=READY pgs=985 cs=0 l=1 rev1=1 crypto rx=0x7f9640001060 tx=0x7f964001cf60 comp rx=0 tx=0).stop 2024-09-16T15:48:07.060 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:07.055+0000 7f964ad9d640 1 -- 172.21.15.123:0/1015270373 reap_dead start 2024-09-16T15:48:07.060 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:07.055+0000 7f9650fb6640 1 -- 172.21.15.123:0/1015270373 shutdown_connections 2024-09-16T15:48:07.060 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:07.055+0000 7f9650fb6640 1 -- 172.21.15.123:0/1015270373 >> 172.21.15.123:0/1015270373 conn(0x7f964c0d3e40 msgr2=0x7f964c0d4220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:48:07.060 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:07.055+0000 7f9650fb6640 1 -- 172.21.15.123:0/1015270373 shutdown_connections 2024-09-16T15:48:07.060 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:07.055+0000 7f9650fb6640 1 -- 172.21.15.123:0/1015270373 wait complete. 2024-09-16T15:48:07.061 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:07.055+0000 7f9650fb6640 1 Processor -- start 2024-09-16T15:48:07.061 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:07.055+0000 7f9650fb6640 1 -- start start 2024-09-16T15:48:07.063 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:07.059+0000 7f9650fb6640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f9634004010 0x7f964c0b7df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:48:07.063 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:07.059+0000 7f9650fb6640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f964c15b2d0 0x7f964c15bae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:48:07.063 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:07.059+0000 7f9650fb6640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f964c1f7940 0x7f964c15c020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:48:07.063 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:07.059+0000 7f9650fb6640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f964c169f90 con 0x7f964c15b2d0 2024-09-16T15:48:07.063 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:07.059+0000 7f9650fb6640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f964c169e10 con 0x7f964c1f7940 2024-09-16T15:48:07.063 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:07.059+0000 7f9650fb6640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f964c16a110 con 0x7f9634004010 2024-09-16T15:48:07.064 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:07.059+0000 7f9649d9b640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f964c15b2d0 0x7f964c15bae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:48:07.064 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:07.059+0000 7f9649d9b640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f964c15b2d0 0x7f964c15bae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:57814/0 (socket says 172.21.15.123:57814) 2024-09-16T15:48:07.064 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:07.059+0000 7f9649d9b640 1 -- 172.21.15.123:0/117526333 learned_addr learned my addr 172.21.15.123:0/117526333 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:48:07.064 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:07.059+0000 7f964a59c640 1 --2- 172.21.15.123:0/117526333 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f9634004010 0x7f964c0b7df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:48:07.064 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:07.059+0000 7f964ad9d640 1 --2- 172.21.15.123:0/117526333 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f964c1f7940 0x7f964c15c020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:48:07.064 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:07.059+0000 7f9649d9b640 1 -- 172.21.15.123:0/117526333 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f9634004010 msgr2=0x7f964c0b7df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:48:07.064 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:07.059+0000 7f9649d9b640 1 --2- 172.21.15.123:0/117526333 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f9634004010 0x7f964c0b7df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:07.064 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:07.059+0000 7f9649d9b640 1 -- 172.21.15.123:0/117526333 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f964c1f7940 msgr2=0x7f964c15c020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:48:07.064 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:07.059+0000 7f9649d9b640 1 --2- 172.21.15.123:0/117526333 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f964c1f7940 0x7f964c15c020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:07.064 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:07.059+0000 7f9649d9b640 1 -- 172.21.15.123:0/117526333 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f964c1f7f00 con 0x7f964c15b2d0 2024-09-16T15:48:07.064 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:07.059+0000 7f964a59c640 1 --2- 172.21.15.123:0/117526333 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f9634004010 0x7f964c0b7df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T15:48:07.064 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:07.059+0000 7f9649d9b640 1 --2- 172.21.15.123:0/117526333 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f964c15b2d0 0x7f964c15bae0 secure :-1 s=READY pgs=986 cs=0 l=1 rev1=1 crypto rx=0x7f9640000c10 tx=0x7f96400a5c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:48:07.064 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:07.059+0000 7f963affd640 1 -- 172.21.15.123:0/117526333 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9640000ee0 con 0x7f964c15b2d0 2024-09-16T15:48:07.064 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:07.059+0000 7f963affd640 1 -- 172.21.15.123:0/117526333 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f9640001080 con 0x7f964c15b2d0 2024-09-16T15:48:07.065 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:07.059+0000 7f963affd640 1 -- 172.21.15.123:0/117526333 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9640001720 con 0x7f964c15b2d0 2024-09-16T15:48:07.065 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:07.059+0000 7f9650fb6640 1 -- 172.21.15.123:0/117526333 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f964c1f81e0 con 0x7f964c15b2d0 2024-09-16T15:48:07.065 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:07.059+0000 7f9650fb6640 1 -- 172.21.15.123:0/117526333 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f964c1f87a0 con 0x7f964c15b2d0 2024-09-16T15:48:07.065 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:07.059+0000 7f9650fb6640 1 -- 172.21.15.123:0/117526333 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f964c0ce9c0 con 0x7f964c15b2d0 2024-09-16T15:48:07.066 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:07.059+0000 7f963affd640 1 -- 172.21.15.123:0/117526333 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 16) ==== 97182+0+0 (secure 0 0 0) 0x7f96400b7030 con 0x7f964c15b2d0 2024-09-16T15:48:07.066 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:07.059+0000 7f963affd640 1 --2- 172.21.15.123:0/117526333 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f962c073010 0x7f962c0754d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:48:07.066 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:07.059+0000 7f964a59c640 1 --2- 172.21.15.123:0/117526333 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f962c073010 0x7f962c0754d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:48:07.066 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:07.059+0000 7f963affd640 1 -- 172.21.15.123:0/117526333 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 6543+0+0 (secure 0 0 0) 0x7f964012d0a0 con 0x7f964c15b2d0 2024-09-16T15:48:07.066 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:07.063+0000 7f964a59c640 1 --2- 172.21.15.123:0/117526333 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f962c073010 0x7f962c0754d0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f963c000c10 tx=0x7f963c006c10 comp rx=0 tx=0).ready entity=mgr.4150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:48:07.070 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:07.067+0000 7f963affd640 1 -- 172.21.15.123:0/117526333 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f96400be070 con 0x7f964c15b2d0 2024-09-16T15:48:07.274 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:07.271+0000 7f9650fb6640 1 -- 172.21.15.123:0/117526333 --> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] -- mgr_command(tid 0: {"prefix": "fs volume rm", "vol_name": "testvol", "yes-i-really-mean-it": "--yes-i-really-mean-it", "target": ["mon-mgr", ""]}) -- 0x7f964c0b90d0 con 0x7f962c073010 2024-09-16T15:48:07.283 INFO:tasks.ceph.mon.a.smithi123.stderr:2024-09-16T15:48:07.279+0000 7f268d076640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-09-16T15:48:07.313 INFO:tasks.ceph.mds.c.smithi123.stdout:starting mds.c at 2024-09-16T15:48:09.733 DEBUG:teuthology.orchestra.run.smithi123:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-16T15:48:09.736 DEBUG:teuthology.orchestra.run.smithi184:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-16T15:48:10.323 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:10.319+0000 7f963affd640 1 -- 172.21.15.123:0/117526333 <== mgr.4150 v2:172.21.15.123:6836/2464759676 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+78 (secure 0 0 0) 0x7f964c0b90d0 con 0x7f962c073010 2024-09-16T15:48:10.323 INFO:teuthology.orchestra.run.smithi123.stdout:metadata pool: cephfs.testvol.meta data pool: ['cephfs.testvol.data'] removed 2024-09-16T15:48:10.327 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:10.323+0000 7f9650fb6640 1 -- 172.21.15.123:0/117526333 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f962c073010 msgr2=0x7f962c0754d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:48:10.327 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:10.323+0000 7f9650fb6640 1 --2- 172.21.15.123:0/117526333 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f962c073010 0x7f962c0754d0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f963c000c10 tx=0x7f963c006c10 comp rx=0 tx=0).stop 2024-09-16T15:48:10.327 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:10.323+0000 7f9650fb6640 1 -- 172.21.15.123:0/117526333 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f964c15b2d0 msgr2=0x7f964c15bae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:48:10.327 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:10.323+0000 7f9650fb6640 1 --2- 172.21.15.123:0/117526333 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f964c15b2d0 0x7f964c15bae0 secure :-1 s=READY pgs=986 cs=0 l=1 rev1=1 crypto rx=0x7f9640000c10 tx=0x7f96400a5c00 comp rx=0 tx=0).stop 2024-09-16T15:48:10.327 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:10.323+0000 7f9650fb6640 1 -- 172.21.15.123:0/117526333 shutdown_connections 2024-09-16T15:48:10.327 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:10.323+0000 7f9650fb6640 1 --2- 172.21.15.123:0/117526333 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f962c073010 0x7f962c0754d0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:10.327 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:10.323+0000 7f9650fb6640 1 --2- 172.21.15.123:0/117526333 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f964c1f7940 0x7f964c15c020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:10.327 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:10.323+0000 7f9650fb6640 1 --2- 172.21.15.123:0/117526333 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f964c15b2d0 0x7f964c15bae0 unknown :-1 s=CLOSED pgs=986 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:10.327 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:10.323+0000 7f9650fb6640 1 --2- 172.21.15.123:0/117526333 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f9634004010 0x7f964c0b7df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:10.328 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:10.323+0000 7f9650fb6640 1 -- 172.21.15.123:0/117526333 >> 172.21.15.123:0/117526333 conn(0x7f964c0d3e40 msgr2=0x7f964c0b1170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:48:10.328 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:10.323+0000 7f9650fb6640 1 -- 172.21.15.123:0/117526333 shutdown_connections 2024-09-16T15:48:10.328 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:10.323+0000 7f9650fb6640 1 -- 172.21.15.123:0/117526333 wait complete. 2024-09-16T15:48:10.347 INFO:teuthology.nuke:Clearing teuthology firewall rules... 2024-09-16T15:48:10.347 DEBUG:teuthology.orchestra.run.smithi123:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-09-16T15:48:10.432 DEBUG:teuthology.orchestra.run.smithi184:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-09-16T15:48:10.508 INFO:teuthology.nuke:Cleared teuthology firewall rules. 2024-09-16T15:48:10.508 INFO:tasks.cephfs.mount:Terminating background process 2024-09-16T15:48:10.508 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2024-09-16T15:48:10.508 DEBUG:teuthology.orchestra.run.smithi123:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-16T15:48:10.514 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2024-09-16T15:48:10.514 DEBUG:teuthology.orchestra.run.smithi123:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-16T15:48:10.563 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-09-16T15:48:10.741 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:10.735+0000 7f92f8c13640 1 -- 172.21.15.123:0/1814333719 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f92dc0051d0 msgr2=0x7f92dc00b9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:48:10.741 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:10.735+0000 7f92f8c13640 1 --2- 172.21.15.123:0/1814333719 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f92dc0051d0 0x7f92dc00b9d0 secure :-1 s=READY pgs=991 cs=0 l=1 rev1=1 crypto rx=0x7f92e400c470 tx=0x7f92e401cf40 comp rx=0 tx=0).stop 2024-09-16T15:48:10.741 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:10.735+0000 7f92f8c13640 1 -- 172.21.15.123:0/1814333719 shutdown_connections 2024-09-16T15:48:10.741 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:10.735+0000 7f92f8c13640 1 --2- 172.21.15.123:0/1814333719 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f92dc0051d0 0x7f92dc00b9d0 unknown :-1 s=CLOSED pgs=991 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:10.741 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:10.735+0000 7f92f8c13640 1 -- 172.21.15.123:0/1814333719 >> 172.21.15.123:0/1814333719 conn(0x7f92f4074230 msgr2=0x7f92f4074640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:48:10.741 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:10.735+0000 7f92f8c13640 1 -- 172.21.15.123:0/1814333719 shutdown_connections 2024-09-16T15:48:10.741 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:10.735+0000 7f92f8c13640 1 -- 172.21.15.123:0/1814333719 wait complete. 2024-09-16T15:48:10.742 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:10.739+0000 7f92f8c13640 1 Processor -- start 2024-09-16T15:48:10.742 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:10.739+0000 7f92f8c13640 1 -- start start 2024-09-16T15:48:10.744 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:10.739+0000 7f92f8c13640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f92dc0051d0 0x7f92f40cf4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:48:10.744 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:10.739+0000 7f92f8c13640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f92f41f50f0 0x7f92f4099900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:48:10.744 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:10.739+0000 7f92f8c13640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f92f40c8ac0 0x7f92f40c8f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:48:10.745 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:10.739+0000 7f92f8c13640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f92f40b1100 con 0x7f92dc0051d0 2024-09-16T15:48:10.745 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:10.739+0000 7f92f8c13640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f92f40b0f80 con 0x7f92f41f50f0 2024-09-16T15:48:10.745 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:10.739+0000 7f92f8c13640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f92f40b1280 con 0x7f92f40c8ac0 2024-09-16T15:48:10.745 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:10.739+0000 7f92f259c640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f92dc0051d0 0x7f92f40cf4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:48:10.745 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:10.739+0000 7f92f259c640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f92dc0051d0 0x7f92f40cf4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:57848/0 (socket says 172.21.15.123:57848) 2024-09-16T15:48:10.745 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:10.739+0000 7f92f259c640 1 -- 172.21.15.123:0/3775823137 learned_addr learned my addr 172.21.15.123:0/3775823137 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:48:10.745 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:10.739+0000 7f92f1d9b640 1 --2- 172.21.15.123:0/3775823137 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f92f41f50f0 0x7f92f4099900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:48:10.745 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:10.739+0000 7f92f2d9d640 1 --2- 172.21.15.123:0/3775823137 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f92f40c8ac0 0x7f92f40c8f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:48:10.745 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:10.739+0000 7f92f259c640 1 -- 172.21.15.123:0/3775823137 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f92f40c8ac0 msgr2=0x7f92f40c8f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:48:10.745 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:10.739+0000 7f92f259c640 1 --2- 172.21.15.123:0/3775823137 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f92f40c8ac0 0x7f92f40c8f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:10.745 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:10.739+0000 7f92f259c640 1 -- 172.21.15.123:0/3775823137 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f92f41f50f0 msgr2=0x7f92f4099900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:48:10.745 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:10.739+0000 7f92f259c640 1 --2- 172.21.15.123:0/3775823137 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f92f41f50f0 0x7f92f4099900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:10.745 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:10.739+0000 7f92f259c640 1 -- 172.21.15.123:0/3775823137 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f92f4099e40 con 0x7f92dc0051d0 2024-09-16T15:48:10.746 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:10.739+0000 7f92f259c640 1 --2- 172.21.15.123:0/3775823137 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f92dc0051d0 0x7f92f40cf4c0 secure :-1 s=READY pgs=992 cs=0 l=1 rev1=1 crypto rx=0x7f92ec0018d0 tx=0x7f92ec001d90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:48:10.746 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:10.739+0000 7f92e2ffd640 1 -- 172.21.15.123:0/3775823137 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92ec015070 con 0x7f92dc0051d0 2024-09-16T15:48:10.746 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:10.739+0000 7f92e2ffd640 1 -- 172.21.15.123:0/3775823137 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f92ec00eb50 con 0x7f92dc0051d0 2024-09-16T15:48:10.746 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:10.739+0000 7f92f8c13640 1 -- 172.21.15.123:0/3775823137 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f92f40ce420 con 0x7f92dc0051d0 2024-09-16T15:48:10.746 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:10.739+0000 7f92e2ffd640 1 -- 172.21.15.123:0/3775823137 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92ec011460 con 0x7f92dc0051d0 2024-09-16T15:48:10.746 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:10.739+0000 7f92f8c13640 1 -- 172.21.15.123:0/3775823137 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f92f40ce9e0 con 0x7f92dc0051d0 2024-09-16T15:48:10.747 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:10.739+0000 7f92f8c13640 1 -- 172.21.15.123:0/3775823137 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f92f4068a40 con 0x7f92dc0051d0 2024-09-16T15:48:10.747 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:10.743+0000 7f92e2ffd640 1 -- 172.21.15.123:0/3775823137 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 16) ==== 97182+0+0 (secure 0 0 0) 0x7f92ec018070 con 0x7f92dc0051d0 2024-09-16T15:48:10.747 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:10.743+0000 7f92e2ffd640 1 --2- 172.21.15.123:0/3775823137 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f92d4073010 0x7f92d40754d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:48:10.747 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:10.743+0000 7f92f1d9b640 1 --2- 172.21.15.123:0/3775823137 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f92d4073010 0x7f92d40754d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:48:10.747 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:10.743+0000 7f92e2ffd640 1 -- 172.21.15.123:0/3775823137 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 5795+0+0 (secure 0 0 0) 0x7f92ec094970 con 0x7f92dc0051d0 2024-09-16T15:48:10.747 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:10.743+0000 7f92f1d9b640 1 --2- 172.21.15.123:0/3775823137 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f92d4073010 0x7f92d40754d0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f92f40c7480 tx=0x7f92e80091e0 comp rx=0 tx=0).ready entity=mgr.4150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:48:10.750 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:10.747+0000 7f92e2ffd640 1 -- 172.21.15.123:0/3775823137 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f92ec05f940 con 0x7f92dc0051d0 2024-09-16T15:48:10.933 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:10.927+0000 7f92f8c13640 1 -- 172.21.15.123:0/3775823137 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f92f406e720 con 0x7f92dc0051d0 2024-09-16T15:48:10.934 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:10.931+0000 7f92e2ffd640 1 -- 172.21.15.123:0/3775823137 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 2 entries v92) ==== 80+0+124 (secure 0 0 0) 0x7f92ec05f240 con 0x7f92dc0051d0 2024-09-16T15:48:10.934 INFO:teuthology.orchestra.run.smithi123.stdout:172.21.15.123:6835/2536398045 2024-09-17T15:48:07.284775+0000 2024-09-16T15:48:10.934 INFO:teuthology.orchestra.run.smithi123.stdout:172.21.15.123:6833/2536398045 2024-09-17T15:48:07.284775+0000 2024-09-16T15:48:10.934 INFO:teuthology.orchestra.run.smithi123.stderr:listed 2 entries 2024-09-16T15:48:10.936 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:10.931+0000 7f92f8c13640 1 -- 172.21.15.123:0/3775823137 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f92d4073010 msgr2=0x7f92d40754d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:48:10.937 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:10.931+0000 7f92f8c13640 1 --2- 172.21.15.123:0/3775823137 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f92d4073010 0x7f92d40754d0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f92f40c7480 tx=0x7f92e80091e0 comp rx=0 tx=0).stop 2024-09-16T15:48:10.937 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:10.931+0000 7f92f8c13640 1 -- 172.21.15.123:0/3775823137 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f92dc0051d0 msgr2=0x7f92f40cf4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:48:10.937 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:10.931+0000 7f92f8c13640 1 --2- 172.21.15.123:0/3775823137 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f92dc0051d0 0x7f92f40cf4c0 secure :-1 s=READY pgs=992 cs=0 l=1 rev1=1 crypto rx=0x7f92ec0018d0 tx=0x7f92ec001d90 comp rx=0 tx=0).stop 2024-09-16T15:48:10.937 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:10.931+0000 7f92f8c13640 1 -- 172.21.15.123:0/3775823137 shutdown_connections 2024-09-16T15:48:10.937 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:10.931+0000 7f92f8c13640 1 --2- 172.21.15.123:0/3775823137 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f92d4073010 0x7f92d40754d0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:10.937 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:10.931+0000 7f92f8c13640 1 --2- 172.21.15.123:0/3775823137 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f92f40c8ac0 0x7f92f40c8f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:10.937 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:10.931+0000 7f92f8c13640 1 --2- 172.21.15.123:0/3775823137 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f92f41f50f0 0x7f92f4099900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:10.937 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:10.931+0000 7f92f8c13640 1 --2- 172.21.15.123:0/3775823137 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f92dc0051d0 0x7f92f40cf4c0 unknown :-1 s=CLOSED pgs=992 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:10.937 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:10.931+0000 7f92f8c13640 1 -- 172.21.15.123:0/3775823137 >> 172.21.15.123:0/3775823137 conn(0x7f92f4074230 msgr2=0x7f92f406f300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:48:10.937 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:10.931+0000 7f92f8c13640 1 -- 172.21.15.123:0/3775823137 shutdown_connections 2024-09-16T15:48:10.937 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:10.931+0000 7f92f8c13640 1 -- 172.21.15.123:0/3775823137 wait complete. 2024-09-16T15:48:10.953 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi123.front.sepia.ceph.com... 2024-09-16T15:48:10.954 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:48:10.954 DEBUG:teuthology.orchestra.run.smithi123:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2024-09-16T15:48:10.965 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi123.stderr:ceph-fuse[44490]: fuse finished with error 0 and tester_r 0 2024-09-16T15:48:10.967 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi123.front.sepia.ceph.com 2024-09-16T15:48:10.967 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:48:10.967 DEBUG:teuthology.orchestra.run.smithi123:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-09-16T15:48:11.132 INFO:tasks.cephfs.mount:Terminating background process 2024-09-16T15:48:11.133 DEBUG:teuthology.orchestra.run.smithi184:> set -ex 2024-09-16T15:48:11.133 DEBUG:teuthology.orchestra.run.smithi184:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-16T15:48:11.139 DEBUG:teuthology.orchestra.run.smithi184:> set -ex 2024-09-16T15:48:11.139 DEBUG:teuthology.orchestra.run.smithi184:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-16T15:48:11.187 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-09-16T15:48:11.325 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:11.319+0000 7fd03deab640 1 -- 172.21.15.123:0/3227095880 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd0380d0200 msgr2=0x7fd0380b3f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:48:11.325 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:11.319+0000 7fd03deab640 1 --2- 172.21.15.123:0/3227095880 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd0380d0200 0x7fd0380b3f80 secure :-1 s=READY pgs=993 cs=0 l=1 rev1=1 crypto rx=0x7fd0380b6a00 tx=0x7fd03001f580 comp rx=0 tx=0).stop 2024-09-16T15:48:11.326 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:11.319+0000 7fd03deab640 1 -- 172.21.15.123:0/3227095880 shutdown_connections 2024-09-16T15:48:11.326 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:11.319+0000 7fd03deab640 1 --2- 172.21.15.123:0/3227095880 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd0380d0200 0x7fd0380b3f80 unknown :-1 s=CLOSED pgs=993 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:11.326 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:11.319+0000 7fd03deab640 1 --2- 172.21.15.123:0/3227095880 >> v2:172.21.15.184:3300/0 conn(0x7fd0380d0aa0 0x7fd0380d1d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:11.326 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:11.319+0000 7fd03deab640 1 -- 172.21.15.123:0/3227095880 >> 172.21.15.123:0/3227095880 conn(0x7fd0380d3e30 msgr2=0x7fd0380d4240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:48:11.326 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:11.319+0000 7fd03deab640 1 -- 172.21.15.123:0/3227095880 shutdown_connections 2024-09-16T15:48:11.326 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:11.319+0000 7fd03deab640 1 -- 172.21.15.123:0/3227095880 wait complete. 2024-09-16T15:48:11.326 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:11.323+0000 7fd03deab640 1 Processor -- start 2024-09-16T15:48:11.327 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:11.323+0000 7fd03deab640 1 -- start start 2024-09-16T15:48:11.329 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:11.323+0000 7fd03deab640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd0380cf530 0x7fd0380cd4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:48:11.329 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:11.323+0000 7fd03deab640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fd0380d0200 0x7fd0380c9fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:48:11.329 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:11.323+0000 7fd03deab640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd0380d0aa0 0x7fd038158060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:48:11.329 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:11.323+0000 7fd03deab640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fd0380b44c0 con 0x7fd0380d0aa0 2024-09-16T15:48:11.329 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:11.323+0000 7fd03deab640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fd0380d1090 con 0x7fd0380cf530 2024-09-16T15:48:11.329 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:11.323+0000 7fd03deab640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fd0380b4640 con 0x7fd0380d0200 2024-09-16T15:48:11.330 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:11.323+0000 7fd037fff640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd0380d0aa0 0x7fd038158060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:48:11.330 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:11.323+0000 7fd037fff640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd0380d0aa0 0x7fd038158060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:57862/0 (socket says 172.21.15.123:57862) 2024-09-16T15:48:11.330 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:11.323+0000 7fd037fff640 1 -- 172.21.15.123:0/1348689213 learned_addr learned my addr 172.21.15.123:0/1348689213 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:48:11.330 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:11.323+0000 7fd0377fe640 1 --2- 172.21.15.123:0/1348689213 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd0380cf530 0x7fd0380cd4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:48:11.330 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:11.323+0000 7fd037fff640 1 -- 172.21.15.123:0/1348689213 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fd0380d0200 msgr2=0x7fd0380c9fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:48:11.330 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:11.323+0000 7fd037fff640 1 --2- 172.21.15.123:0/1348689213 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fd0380d0200 0x7fd0380c9fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:11.330 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:11.323+0000 7fd037fff640 1 -- 172.21.15.123:0/1348689213 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd0380cf530 msgr2=0x7fd0380cd4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:48:11.330 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:11.323+0000 7fd037fff640 1 --2- 172.21.15.123:0/1348689213 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd0380cf530 0x7fd0380cd4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:11.330 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:11.323+0000 7fd037fff640 1 -- 172.21.15.123:0/1348689213 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd0380cd9f0 con 0x7fd0380d0aa0 2024-09-16T15:48:11.330 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:11.323+0000 7fd0377fe640 1 --2- 172.21.15.123:0/1348689213 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd0380cf530 0x7fd0380cd4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T15:48:11.330 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:11.323+0000 7fd037fff640 1 --2- 172.21.15.123:0/1348689213 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd0380d0aa0 0x7fd038158060 secure :-1 s=READY pgs=994 cs=0 l=1 rev1=1 crypto rx=0x7fd0300b3040 tx=0x7fd0300093b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:48:11.330 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:11.323+0000 7fd01ffff640 1 -- 172.21.15.123:0/1348689213 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0300b5040 con 0x7fd0380d0aa0 2024-09-16T15:48:11.331 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:11.323+0000 7fd01ffff640 1 -- 172.21.15.123:0/1348689213 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fd030004b40 con 0x7fd0380d0aa0 2024-09-16T15:48:11.331 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:11.323+0000 7fd01ffff640 1 -- 172.21.15.123:0/1348689213 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd030004d90 con 0x7fd0380d0aa0 2024-09-16T15:48:11.331 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:11.323+0000 7fd03deab640 1 -- 172.21.15.123:0/1348689213 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd0380ca810 con 0x7fd0380d0aa0 2024-09-16T15:48:11.331 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:11.323+0000 7fd03deab640 1 -- 172.21.15.123:0/1348689213 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd0380cc9d0 con 0x7fd0380d0aa0 2024-09-16T15:48:11.331 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:11.327+0000 7fd03deab640 1 -- 172.21.15.123:0/1348689213 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd0380b3f80 con 0x7fd0380d0aa0 2024-09-16T15:48:11.332 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:11.327+0000 7fd01ffff640 1 -- 172.21.15.123:0/1348689213 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 16) ==== 97182+0+0 (secure 0 0 0) 0x7fd0300c0020 con 0x7fd0380d0aa0 2024-09-16T15:48:11.332 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:11.327+0000 7fd01ffff640 1 --2- 172.21.15.123:0/1348689213 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fd024072fe0 0x7fd0240754a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:48:11.332 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:11.327+0000 7fd01ffff640 1 -- 172.21.15.123:0/1348689213 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 5795+0+0 (secure 0 0 0) 0x7fd030131910 con 0x7fd0380d0aa0 2024-09-16T15:48:11.332 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:11.327+0000 7fd0377fe640 1 --2- 172.21.15.123:0/1348689213 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fd024072fe0 0x7fd0240754a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:48:11.332 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:11.327+0000 7fd0377fe640 1 --2- 172.21.15.123:0/1348689213 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fd024072fe0 0x7fd0240754a0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fd028005fd0 tx=0x7fd028005d00 comp rx=0 tx=0).ready entity=mgr.4150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:48:11.336 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:11.331+0000 7fd01ffff640 1 -- 172.21.15.123:0/1348689213 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7fd0300190e0 con 0x7fd0380d0aa0 2024-09-16T15:48:11.519 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:11.515+0000 7fd03deab640 1 -- 172.21.15.123:0/1348689213 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7fd0380b2f00 con 0x7fd0380d0aa0 2024-09-16T15:48:11.520 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:11.515+0000 7fd01ffff640 1 -- 172.21.15.123:0/1348689213 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 2 entries v92) ==== 80+0+124 (secure 0 0 0) 0x7fd0300fc9a0 con 0x7fd0380d0aa0 2024-09-16T15:48:11.520 INFO:teuthology.orchestra.run.smithi123.stdout:172.21.15.123:6835/2536398045 2024-09-17T15:48:07.284775+0000 2024-09-16T15:48:11.520 INFO:teuthology.orchestra.run.smithi123.stdout:172.21.15.123:6833/2536398045 2024-09-17T15:48:07.284775+0000 2024-09-16T15:48:11.521 INFO:teuthology.orchestra.run.smithi123.stderr:listed 2 entries 2024-09-16T15:48:11.522 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:11.519+0000 7fd03deab640 1 -- 172.21.15.123:0/1348689213 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fd024072fe0 msgr2=0x7fd0240754a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:48:11.522 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:11.519+0000 7fd03deab640 1 --2- 172.21.15.123:0/1348689213 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fd024072fe0 0x7fd0240754a0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fd028005fd0 tx=0x7fd028005d00 comp rx=0 tx=0).stop 2024-09-16T15:48:11.523 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:11.519+0000 7fd03deab640 1 -- 172.21.15.123:0/1348689213 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd0380d0aa0 msgr2=0x7fd038158060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:48:11.523 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:11.519+0000 7fd03deab640 1 --2- 172.21.15.123:0/1348689213 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd0380d0aa0 0x7fd038158060 secure :-1 s=READY pgs=994 cs=0 l=1 rev1=1 crypto rx=0x7fd0300b3040 tx=0x7fd0300093b0 comp rx=0 tx=0).stop 2024-09-16T15:48:11.523 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:11.519+0000 7fd03deab640 1 -- 172.21.15.123:0/1348689213 shutdown_connections 2024-09-16T15:48:11.523 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:11.519+0000 7fd03deab640 1 --2- 172.21.15.123:0/1348689213 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fd024072fe0 0x7fd0240754a0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:11.523 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:11.519+0000 7fd03deab640 1 --2- 172.21.15.123:0/1348689213 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd0380d0aa0 0x7fd038158060 unknown :-1 s=CLOSED pgs=994 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:11.523 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:11.519+0000 7fd03deab640 1 --2- 172.21.15.123:0/1348689213 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fd0380d0200 0x7fd0380c9fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:11.523 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:11.519+0000 7fd03deab640 1 --2- 172.21.15.123:0/1348689213 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd0380cf530 0x7fd0380cd4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:11.523 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:11.519+0000 7fd03deab640 1 -- 172.21.15.123:0/1348689213 >> 172.21.15.123:0/1348689213 conn(0x7fd0380d3e30 msgr2=0x7fd0380b4a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:48:11.523 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:11.519+0000 7fd03deab640 1 -- 172.21.15.123:0/1348689213 shutdown_connections 2024-09-16T15:48:11.523 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:11.519+0000 7fd03deab640 1 -- 172.21.15.123:0/1348689213 wait complete. 2024-09-16T15:48:11.540 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi184.front.sepia.ceph.com... 2024-09-16T15:48:11.540 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:48:11.540 DEBUG:teuthology.orchestra.run.smithi184:> sudo fusermount -u /home/ubuntu/cephtest/mnt.1 2024-09-16T15:48:11.553 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi184.stderr:ceph-fuse[35560]: fuse finished with error 0 and tester_r 0 2024-09-16T15:48:11.555 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi184.front.sepia.ceph.com 2024-09-16T15:48:11.555 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:48:11.555 DEBUG:teuthology.orchestra.run.smithi184:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2024-09-16T15:48:11.654 INFO:tasks.cephfs.mount:Terminating background process 2024-09-16T15:48:11.654 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2024-09-16T15:48:11.654 DEBUG:teuthology.orchestra.run.smithi123:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-16T15:48:11.660 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16336468k,nr_inodes=4084117,mode=755,inode64 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3274684k,mode=755,inode64 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev,inode64 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=17082 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3274680k,nr_inodes=818670,mode=700,uid=1000,gid=1271,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3274684k,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-09-16T15:48:11.660 INFO:tasks.cephfs.mount:Terminating background process 2024-09-16T15:48:11.660 DEBUG:teuthology.orchestra.run.smithi184:> set -ex 2024-09-16T15:48:11.661 DEBUG:teuthology.orchestra.run.smithi184:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-16T15:48:11.666 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16336488k,nr_inodes=4084122,mode=755,inode64 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3274688k,mode=755,inode64 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev,inode64 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=17834 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3274684k,nr_inodes=818671,mode=700,uid=1000,gid=1271,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3274688k,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-09-16T15:48:11.666 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-16T15:48:11.832 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:11.827+0000 7f1469aff640 1 -- 172.21.15.123:0/1947191327 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f144c004810 msgr2=0x7f144c004c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:48:11.832 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:11.827+0000 7f1469aff640 1 --2- 172.21.15.123:0/1947191327 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f144c004810 0x7f144c004c90 secure :-1 s=READY pgs=996 cs=0 l=1 rev1=1 crypto rx=0x7f145800c260 tx=0x7f145801cf90 comp rx=0 tx=0).stop 2024-09-16T15:48:11.832 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:11.827+0000 7f1463fff640 1 -- 172.21.15.123:0/1947191327 reap_dead start 2024-09-16T15:48:11.832 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:11.827+0000 7f1469aff640 1 -- 172.21.15.123:0/1947191327 shutdown_connections 2024-09-16T15:48:11.832 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:11.827+0000 7f1469aff640 1 -- 172.21.15.123:0/1947191327 >> 172.21.15.123:0/1947191327 conn(0x7f1464074bd0 msgr2=0x7f146409aa40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:48:11.832 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:11.827+0000 7f1469aff640 1 -- 172.21.15.123:0/1947191327 shutdown_connections 2024-09-16T15:48:11.832 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:11.827+0000 7f1469aff640 1 -- 172.21.15.123:0/1947191327 wait complete. 2024-09-16T15:48:11.833 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:11.827+0000 7f1469aff640 1 Processor -- start 2024-09-16T15:48:11.833 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:11.827+0000 7f1469aff640 1 -- start start 2024-09-16T15:48:11.835 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:11.831+0000 7f1469aff640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f146414de70 0x7f1464159930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:48:11.835 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:11.831+0000 7f1469aff640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f14641f68c0 0x7f1464159e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:48:11.835 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:11.831+0000 7f1469aff640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f146414e250 0x7f14640c90c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:48:11.835 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:11.831+0000 7f1469aff640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f14640cdce0 con 0x7f146414e250 2024-09-16T15:48:11.835 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:11.831+0000 7f1469aff640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f14640cdb60 con 0x7f146414de70 2024-09-16T15:48:11.835 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:11.831+0000 7f1469aff640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f1464166860 con 0x7f14641f68c0 2024-09-16T15:48:11.836 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:11.831+0000 7f1463fff640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f146414e250 0x7f14640c90c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:48:11.836 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:11.831+0000 7f1463fff640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f146414e250 0x7f14640c90c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:57882/0 (socket says 172.21.15.123:57882) 2024-09-16T15:48:11.836 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:11.831+0000 7f1463fff640 1 -- 172.21.15.123:0/1078033923 learned_addr learned my addr 172.21.15.123:0/1078033923 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:48:11.836 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:11.831+0000 7f1462ffd640 1 --2- 172.21.15.123:0/1078033923 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f14641f68c0 0x7f1464159e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:48:11.836 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:11.831+0000 7f14637fe640 1 --2- 172.21.15.123:0/1078033923 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f146414de70 0x7f1464159930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:48:11.836 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:11.831+0000 7f1463fff640 1 -- 172.21.15.123:0/1078033923 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f14641f68c0 msgr2=0x7f1464159e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:48:11.836 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:11.831+0000 7f1463fff640 1 --2- 172.21.15.123:0/1078033923 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f14641f68c0 0x7f1464159e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:11.836 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:11.831+0000 7f1463fff640 1 -- 172.21.15.123:0/1078033923 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f146414de70 msgr2=0x7f1464159930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:48:11.836 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:11.831+0000 7f1463fff640 1 --2- 172.21.15.123:0/1078033923 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f146414de70 0x7f1464159930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:11.836 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:11.831+0000 7f1463fff640 1 -- 172.21.15.123:0/1078033923 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f14640c9ca0 con 0x7f146414e250 2024-09-16T15:48:11.836 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:11.831+0000 7f14637fe640 1 --2- 172.21.15.123:0/1078033923 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f146414de70 0x7f1464159930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T15:48:11.836 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:11.831+0000 7f1463fff640 1 --2- 172.21.15.123:0/1078033923 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f146414e250 0x7f14640c90c0 secure :-1 s=READY pgs=997 cs=0 l=1 rev1=1 crypto rx=0x7f14640d2d40 tx=0x7f145c00dd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:48:11.836 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:11.831+0000 7f144bfff640 1 -- 172.21.15.123:0/1078033923 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f145c019070 con 0x7f146414e250 2024-09-16T15:48:11.836 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:11.831+0000 7f144bfff640 1 -- 172.21.15.123:0/1078033923 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f145c008d60 con 0x7f146414e250 2024-09-16T15:48:11.837 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:11.831+0000 7f1469aff640 1 -- 172.21.15.123:0/1078033923 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f14640c9f80 con 0x7f146414e250 2024-09-16T15:48:11.837 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:11.831+0000 7f144bfff640 1 -- 172.21.15.123:0/1078033923 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f145c003be0 con 0x7f146414e250 2024-09-16T15:48:11.837 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:11.831+0000 7f1469aff640 1 -- 172.21.15.123:0/1078033923 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f14640b5e70 con 0x7f146414e250 2024-09-16T15:48:11.837 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:11.831+0000 7f1469aff640 1 -- 172.21.15.123:0/1078033923 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f14640d0310 con 0x7f146414e250 2024-09-16T15:48:11.838 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:11.831+0000 7f144bfff640 1 -- 172.21.15.123:0/1078033923 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 16) ==== 97182+0+0 (secure 0 0 0) 0x7f145c01c070 con 0x7f146414e250 2024-09-16T15:48:11.838 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:11.831+0000 7f144bfff640 1 --2- 172.21.15.123:0/1078033923 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f1450073010 0x7f14500754d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:48:11.838 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:11.831+0000 7f144bfff640 1 -- 172.21.15.123:0/1078033923 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 5795+0+0 (secure 0 0 0) 0x7f145c0936f0 con 0x7f146414e250 2024-09-16T15:48:11.838 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:11.831+0000 7f14637fe640 1 --2- 172.21.15.123:0/1078033923 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f1450073010 0x7f14500754d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:48:11.838 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:11.831+0000 7f14637fe640 1 --2- 172.21.15.123:0/1078033923 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f1450073010 0x7f14500754d0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f14540058d0 tx=0x7f1454007040 comp rx=0 tx=0).ready entity=mgr.4150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:48:11.841 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:11.835+0000 7f144bfff640 1 -- 172.21.15.123:0/1078033923 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f145c05e750 con 0x7f146414e250 2024-09-16T15:48:12.048 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:12.043+0000 7f1469aff640 1 -- 172.21.15.123:0/1078033923 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f14640b7ab0 con 0x7f146414e250 2024-09-16T15:48:12.049 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:12.043+0000 7f144bfff640 1 -- 172.21.15.123:0/1078033923 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 59 v59) ==== 94+0+5231 (secure 0 0 0) 0x7f145c065e20 con 0x7f146414e250 2024-09-16T15:48:12.050 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-09-16T15:48:12.050 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":59,"btime":"2024-09-16T15:48:08:287830+0000","default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5677,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6835/1609511477","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6833","nonce":1609511477},{"type":"v1","addr":"172.21.15.184:6835","nonce":1609511477}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":48},{"gid":6672,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6834/3658721718","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6832","nonce":3658721718},{"type":"v1","addr":"172.21.15.123:6834","nonce":3658721718}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":48},{"gid":6891,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6835/95665084","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6833","nonce":95665084},{"type":"v1","addr":"172.21.15.123:6835","nonce":95665084}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":59}],"filesystems":[{"mdsmap":{"epoch":53,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-16T15:47:49.295797+0000","modified":"2024-09-16T15:47:52.333498+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6684},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6684":{"gid":6684,"name":"b","rank":0,"incarnation":49,"state":"up:active","state_seq":3,"addr":"172.21.15.184:6834/1546532515","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":1546532515},{"type":"v1","addr":"172.21.15.184:6834","nonce":1546532515}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6684,"qdb_cluster":[6684]},"id":6}]} 2024-09-16T15:48:12.050 INFO:teuthology.orchestra.run.smithi123.stderr:dumped fsmap epoch 59 2024-09-16T15:48:12.052 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:12.047+0000 7f1469aff640 1 -- 172.21.15.123:0/1078033923 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f1450073010 msgr2=0x7f14500754d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:48:12.052 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:12.047+0000 7f1469aff640 1 --2- 172.21.15.123:0/1078033923 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f1450073010 0x7f14500754d0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f14540058d0 tx=0x7f1454007040 comp rx=0 tx=0).stop 2024-09-16T15:48:12.052 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:12.047+0000 7f1469aff640 1 -- 172.21.15.123:0/1078033923 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f146414e250 msgr2=0x7f14640c90c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:48:12.052 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:12.047+0000 7f1469aff640 1 --2- 172.21.15.123:0/1078033923 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f146414e250 0x7f14640c90c0 secure :-1 s=READY pgs=997 cs=0 l=1 rev1=1 crypto rx=0x7f14640d2d40 tx=0x7f145c00dd70 comp rx=0 tx=0).stop 2024-09-16T15:48:12.053 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:12.047+0000 7f1469aff640 1 -- 172.21.15.123:0/1078033923 shutdown_connections 2024-09-16T15:48:12.053 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:12.047+0000 7f1469aff640 1 --2- 172.21.15.123:0/1078033923 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f1450073010 0x7f14500754d0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:12.053 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:12.047+0000 7f1469aff640 1 --2- 172.21.15.123:0/1078033923 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f146414e250 0x7f14640c90c0 unknown :-1 s=CLOSED pgs=997 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:12.053 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:12.047+0000 7f1469aff640 1 --2- 172.21.15.123:0/1078033923 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f14641f68c0 0x7f1464159e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:12.053 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:12.047+0000 7f1469aff640 1 --2- 172.21.15.123:0/1078033923 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f146414de70 0x7f1464159930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:12.053 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:12.047+0000 7f1469aff640 1 -- 172.21.15.123:0/1078033923 >> 172.21.15.123:0/1078033923 conn(0x7f1464074bd0 msgr2=0x7f14640ce230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:48:12.053 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:12.047+0000 7f1469aff640 1 -- 172.21.15.123:0/1078033923 shutdown_connections 2024-09-16T15:48:12.053 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:12.047+0000 7f1469aff640 1 -- 172.21.15.123:0/1078033923 wait complete. 2024-09-16T15:48:12.068 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-16T15:48:12.188 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:12.183+0000 7fac0d95a640 1 -- 172.21.15.123:0/218835975 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fac080726c0 msgr2=0x7fac08073ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:48:12.188 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:12.183+0000 7fac0d95a640 1 --2- 172.21.15.123:0/218835975 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fac080726c0 0x7fac08073ef0 secure :-1 s=READY pgs=999 cs=0 l=1 rev1=1 crypto rx=0x7fac080cd650 tx=0x7fabf801c800 comp rx=0 tx=0).stop 2024-09-16T15:48:12.188 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:12.183+0000 7fac0d95a640 1 -- 172.21.15.123:0/218835975 shutdown_connections 2024-09-16T15:48:12.188 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:12.183+0000 7fac0d95a640 1 --2- 172.21.15.123:0/218835975 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fac080726c0 0x7fac08073ef0 unknown :-1 s=CLOSED pgs=999 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:12.188 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:12.183+0000 7fac0d95a640 1 --2- 172.21.15.123:0/218835975 >> v2:172.21.15.184:3300/0 conn(0x7fac0815bca0 0x7fac080d0630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:12.188 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:12.183+0000 7fac0d95a640 1 -- 172.21.15.123:0/218835975 >> 172.21.15.123:0/218835975 conn(0x7fac080cfb10 msgr2=0x7fac080cff20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:48:12.188 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:12.183+0000 7fac0d95a640 1 -- 172.21.15.123:0/218835975 shutdown_connections 2024-09-16T15:48:12.188 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:12.183+0000 7fac0d95a640 1 -- 172.21.15.123:0/218835975 wait complete. 2024-09-16T15:48:12.189 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:12.183+0000 7fac0d95a640 1 Processor -- start 2024-09-16T15:48:12.189 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:12.183+0000 7fac0d95a640 1 -- start start 2024-09-16T15:48:12.190 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:12.183+0000 7fac0d95a640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fac080726c0 0x7fac080b3460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:48:12.191 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:12.183+0000 7fac0d95a640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fac08074430 0x7fac08164e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:48:12.191 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:12.183+0000 7fac0d95a640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fac0815bca0 0x7fac08165340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:48:12.191 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:12.183+0000 7fac0d95a640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fac08168e30 con 0x7fac080726c0 2024-09-16T15:48:12.191 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:12.183+0000 7fac0d95a640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fac08168cb0 con 0x7fac0815bca0 2024-09-16T15:48:12.191 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:12.183+0000 7fac0d95a640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fac08168fb0 con 0x7fac08074430 2024-09-16T15:48:12.191 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:12.187+0000 7fac06ffd640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fac080726c0 0x7fac080b3460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:48:12.191 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:12.187+0000 7fac06ffd640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fac080726c0 0x7fac080b3460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:57900/0 (socket says 172.21.15.123:57900) 2024-09-16T15:48:12.191 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:12.187+0000 7fac06ffd640 1 -- 172.21.15.123:0/1378972082 learned_addr learned my addr 172.21.15.123:0/1378972082 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:48:12.191 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:12.187+0000 7fac067fc640 1 --2- 172.21.15.123:0/1378972082 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fac08074430 0x7fac08164e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:48:12.191 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:12.187+0000 7fac077fe640 1 --2- 172.21.15.123:0/1378972082 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fac0815bca0 0x7fac08165340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:48:12.191 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:12.187+0000 7fac06ffd640 1 -- 172.21.15.123:0/1378972082 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fac08074430 msgr2=0x7fac08164e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:48:12.191 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:12.187+0000 7fac06ffd640 1 --2- 172.21.15.123:0/1378972082 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fac08074430 0x7fac08164e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:12.191 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:12.187+0000 7fac06ffd640 1 -- 172.21.15.123:0/1378972082 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fac0815bca0 msgr2=0x7fac08165340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:48:12.191 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:12.187+0000 7fac06ffd640 1 --2- 172.21.15.123:0/1378972082 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fac0815bca0 0x7fac08165340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:12.192 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:12.187+0000 7fac06ffd640 1 -- 172.21.15.123:0/1378972082 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fac080b29a0 con 0x7fac080726c0 2024-09-16T15:48:12.192 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:12.187+0000 7fac06ffd640 1 --2- 172.21.15.123:0/1378972082 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fac080726c0 0x7fac080b3460 secure :-1 s=READY pgs=1000 cs=0 l=1 rev1=1 crypto rx=0x7fabfc0058d0 tx=0x7fabfc00dde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:48:12.192 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:12.187+0000 7fabeffff640 1 -- 172.21.15.123:0/1378972082 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fabfc017070 con 0x7fac080726c0 2024-09-16T15:48:12.192 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:12.187+0000 7fabeffff640 1 -- 172.21.15.123:0/1378972082 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fabfc009d20 con 0x7fac080726c0 2024-09-16T15:48:12.192 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:12.187+0000 7fac0d95a640 1 -- 172.21.15.123:0/1378972082 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fac081f8970 con 0x7fac080726c0 2024-09-16T15:48:12.192 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:12.187+0000 7fabeffff640 1 -- 172.21.15.123:0/1378972082 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fabfc0106c0 con 0x7fac080726c0 2024-09-16T15:48:12.192 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:12.187+0000 7fac0d95a640 1 -- 172.21.15.123:0/1378972082 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fac080b2c60 con 0x7fac080726c0 2024-09-16T15:48:12.193 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:12.187+0000 7fac0d95a640 1 -- 172.21.15.123:0/1378972082 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fac08073ef0 con 0x7fac080726c0 2024-09-16T15:48:12.193 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:12.187+0000 7fabeffff640 1 -- 172.21.15.123:0/1378972082 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 16) ==== 97182+0+0 (secure 0 0 0) 0x7fabfc01a070 con 0x7fac080726c0 2024-09-16T15:48:12.193 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:12.187+0000 7fabeffff640 1 --2- 172.21.15.123:0/1378972082 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fabf4072fe0 0x7fabf40754a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:48:12.193 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:12.187+0000 7fabeffff640 1 -- 172.21.15.123:0/1378972082 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 5795+0+0 (secure 0 0 0) 0x7fabfc094260 con 0x7fac080726c0 2024-09-16T15:48:12.193 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:12.187+0000 7fac067fc640 1 --2- 172.21.15.123:0/1378972082 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fabf4072fe0 0x7fabf40754a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:48:12.193 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:12.187+0000 7fac067fc640 1 --2- 172.21.15.123:0/1378972082 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fabf4072fe0 0x7fabf40754a0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fabf80af000 tx=0x7fabf801b040 comp rx=0 tx=0).ready entity=mgr.4150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:48:12.197 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:12.191+0000 7fabeffff640 1 -- 172.21.15.123:0/1378972082 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7fabfc05f230 con 0x7fac080726c0 2024-09-16T15:48:12.406 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:12.399+0000 7fac0d95a640 1 -- 172.21.15.123:0/1378972082 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fac0815f630 con 0x7fac080726c0 2024-09-16T15:48:12.406 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:12.403+0000 7fabeffff640 1 -- 172.21.15.123:0/1378972082 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 59 v59) ==== 94+0+5231 (secure 0 0 0) 0x7fabfc05eb30 con 0x7fac080726c0 2024-09-16T15:48:12.407 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-09-16T15:48:12.407 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":59,"btime":"2024-09-16T15:48:08:287830+0000","default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5677,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6835/1609511477","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6833","nonce":1609511477},{"type":"v1","addr":"172.21.15.184:6835","nonce":1609511477}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":48},{"gid":6672,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6834/3658721718","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6832","nonce":3658721718},{"type":"v1","addr":"172.21.15.123:6834","nonce":3658721718}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":48},{"gid":6891,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6835/95665084","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6833","nonce":95665084},{"type":"v1","addr":"172.21.15.123:6835","nonce":95665084}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":59}],"filesystems":[{"mdsmap":{"epoch":53,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-16T15:47:49.295797+0000","modified":"2024-09-16T15:47:52.333498+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6684},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6684":{"gid":6684,"name":"b","rank":0,"incarnation":49,"state":"up:active","state_seq":3,"addr":"172.21.15.184:6834/1546532515","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":1546532515},{"type":"v1","addr":"172.21.15.184:6834","nonce":1546532515}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6684,"qdb_cluster":[6684]},"id":6}]} 2024-09-16T15:48:12.407 INFO:teuthology.orchestra.run.smithi123.stderr:dumped fsmap epoch 59 2024-09-16T15:48:12.409 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:12.403+0000 7fac0d95a640 1 -- 172.21.15.123:0/1378972082 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fabf4072fe0 msgr2=0x7fabf40754a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:48:12.409 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:12.403+0000 7fac0d95a640 1 --2- 172.21.15.123:0/1378972082 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fabf4072fe0 0x7fabf40754a0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fabf80af000 tx=0x7fabf801b040 comp rx=0 tx=0).stop 2024-09-16T15:48:12.410 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:12.403+0000 7fac0d95a640 1 -- 172.21.15.123:0/1378972082 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fac080726c0 msgr2=0x7fac080b3460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:48:12.410 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:12.403+0000 7fac0d95a640 1 --2- 172.21.15.123:0/1378972082 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fac080726c0 0x7fac080b3460 secure :-1 s=READY pgs=1000 cs=0 l=1 rev1=1 crypto rx=0x7fabfc0058d0 tx=0x7fabfc00dde0 comp rx=0 tx=0).stop 2024-09-16T15:48:12.410 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:12.403+0000 7fac0d95a640 1 -- 172.21.15.123:0/1378972082 shutdown_connections 2024-09-16T15:48:12.410 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:12.403+0000 7fac0d95a640 1 --2- 172.21.15.123:0/1378972082 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fabf4072fe0 0x7fabf40754a0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:12.410 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:12.403+0000 7fac0d95a640 1 --2- 172.21.15.123:0/1378972082 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fac0815bca0 0x7fac08165340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:12.410 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:12.403+0000 7fac0d95a640 1 --2- 172.21.15.123:0/1378972082 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fac08074430 0x7fac08164e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:12.410 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:12.403+0000 7fac0d95a640 1 --2- 172.21.15.123:0/1378972082 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fac080726c0 0x7fac080b3460 unknown :-1 s=CLOSED pgs=1000 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:12.410 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:12.403+0000 7fac0d95a640 1 -- 172.21.15.123:0/1378972082 >> 172.21.15.123:0/1378972082 conn(0x7fac080cfb10 msgr2=0x7fac0806eee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:48:12.410 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:12.403+0000 7fac0d95a640 1 -- 172.21.15.123:0/1378972082 shutdown_connections 2024-09-16T15:48:12.410 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:12.403+0000 7fac0d95a640 1 -- 172.21.15.123:0/1378972082 wait complete. 2024-09-16T15:48:12.426 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-16T15:48:12.540 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:12.535+0000 7fc7ca3ea640 1 -- 172.21.15.123:0/3424479213 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc7c415bca0 msgr2=0x7fc7c40d0630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:48:12.540 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:12.535+0000 7fc7ca3ea640 1 --2- 172.21.15.123:0/3424479213 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc7c415bca0 0x7fc7c40d0630 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7fc7b4009840 tx=0x7fc7b401c760 comp rx=0 tx=0).stop 2024-09-16T15:48:12.540 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:12.535+0000 7fc7ca3ea640 1 -- 172.21.15.123:0/3424479213 shutdown_connections 2024-09-16T15:48:12.540 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:12.535+0000 7fc7ca3ea640 1 --2- 172.21.15.123:0/3424479213 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fc7c40726c0 0x7fc7c4073ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:12.540 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:12.535+0000 7fc7ca3ea640 1 --2- 172.21.15.123:0/3424479213 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc7c415bca0 0x7fc7c40d0630 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:12.540 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:12.535+0000 7fc7ca3ea640 1 -- 172.21.15.123:0/3424479213 >> 172.21.15.123:0/3424479213 conn(0x7fc7c40cfb10 msgr2=0x7fc7c40cff20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:48:12.540 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:12.535+0000 7fc7ca3ea640 1 -- 172.21.15.123:0/3424479213 shutdown_connections 2024-09-16T15:48:12.540 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:12.535+0000 7fc7ca3ea640 1 -- 172.21.15.123:0/3424479213 wait complete. 2024-09-16T15:48:12.541 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:12.535+0000 7fc7ca3ea640 1 Processor -- start 2024-09-16T15:48:12.541 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:12.535+0000 7fc7ca3ea640 1 -- start start 2024-09-16T15:48:12.543 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:12.535+0000 7fc7ca3ea640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc7c40726c0 0x7fc7c41f6580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:48:12.543 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:12.535+0000 7fc7ca3ea640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc7c4074430 0x7fc7c40c5340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:48:12.543 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:12.535+0000 7fc7ca3ea640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fc7c415bca0 0x7fc7c40c8cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:48:12.543 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:12.535+0000 7fc7ca3ea640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fc7c4168f30 con 0x7fc7c40726c0 2024-09-16T15:48:12.543 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:12.535+0000 7fc7ca3ea640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fc7c4168db0 con 0x7fc7c4074430 2024-09-16T15:48:12.543 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:12.535+0000 7fc7ca3ea640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fc7c41690b0 con 0x7fc7c415bca0 2024-09-16T15:48:12.543 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:12.539+0000 7fc7c3fff640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc7c40726c0 0x7fc7c41f6580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:48:12.543 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:12.539+0000 7fc7c3fff640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc7c40726c0 0x7fc7c41f6580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:41418/0 (socket says 172.21.15.123:41418) 2024-09-16T15:48:12.543 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:12.539+0000 7fc7c3fff640 1 -- 172.21.15.123:0/2290586655 learned_addr learned my addr 172.21.15.123:0/2290586655 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:48:12.543 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:12.539+0000 7fc7c37fe640 1 --2- 172.21.15.123:0/2290586655 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc7c4074430 0x7fc7c40c5340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:48:12.543 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:12.539+0000 7fc7c8987640 1 --2- 172.21.15.123:0/2290586655 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fc7c415bca0 0x7fc7c40c8cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:48:12.543 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:12.539+0000 7fc7c3fff640 1 -- 172.21.15.123:0/2290586655 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fc7c415bca0 msgr2=0x7fc7c40c8cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:48:12.543 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:12.539+0000 7fc7c3fff640 1 --2- 172.21.15.123:0/2290586655 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fc7c415bca0 0x7fc7c40c8cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:12.544 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:12.539+0000 7fc7c3fff640 1 -- 172.21.15.123:0/2290586655 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc7c4074430 msgr2=0x7fc7c40c5340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:48:12.544 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:12.539+0000 7fc7c3fff640 1 --2- 172.21.15.123:0/2290586655 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc7c4074430 0x7fc7c40c5340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:12.544 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:12.539+0000 7fc7c3fff640 1 -- 172.21.15.123:0/2290586655 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc7c41f9300 con 0x7fc7c40726c0 2024-09-16T15:48:12.544 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:12.539+0000 7fc7c37fe640 1 --2- 172.21.15.123:0/2290586655 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc7c4074430 0x7fc7c40c5340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T15:48:12.544 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:12.539+0000 7fc7c8987640 1 --2- 172.21.15.123:0/2290586655 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fc7c415bca0 0x7fc7c40c8cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T15:48:12.544 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:12.539+0000 7fc7c3fff640 1 --2- 172.21.15.123:0/2290586655 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc7c40726c0 0x7fc7c41f6580 secure :-1 s=READY pgs=1001 cs=0 l=1 rev1=1 crypto rx=0x7fc7b4009790 tx=0x7fc7b4005db0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:48:12.544 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:12.539+0000 7fc7c0ff9640 1 -- 172.21.15.123:0/2290586655 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7b40b2040 con 0x7fc7c40726c0 2024-09-16T15:48:12.544 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:12.539+0000 7fc7c0ff9640 1 -- 172.21.15.123:0/2290586655 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fc7b40043e0 con 0x7fc7c40726c0 2024-09-16T15:48:12.544 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:12.539+0000 7fc7ca3ea640 1 -- 172.21.15.123:0/2290586655 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc7c41f95e0 con 0x7fc7c40726c0 2024-09-16T15:48:12.544 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:12.539+0000 7fc7c0ff9640 1 -- 172.21.15.123:0/2290586655 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7b40052d0 con 0x7fc7c40726c0 2024-09-16T15:48:12.544 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:12.539+0000 7fc7ca3ea640 1 -- 172.21.15.123:0/2290586655 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc7c41f9ba0 con 0x7fc7c40726c0 2024-09-16T15:48:12.545 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:12.539+0000 7fc7ca3ea640 1 -- 172.21.15.123:0/2290586655 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc7c41f9f80 con 0x7fc7c40726c0 2024-09-16T15:48:12.545 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:12.539+0000 7fc7c0ff9640 1 -- 172.21.15.123:0/2290586655 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 16) ==== 97182+0+0 (secure 0 0 0) 0x7fc7b4005470 con 0x7fc7c40726c0 2024-09-16T15:48:12.545 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:12.539+0000 7fc7c0ff9640 1 --2- 172.21.15.123:0/2290586655 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fc7a8072fe0 0x7fc7a80754a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:48:12.545 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:12.539+0000 7fc7c0ff9640 1 -- 172.21.15.123:0/2290586655 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 5795+0+0 (secure 0 0 0) 0x7fc7b412de90 con 0x7fc7c40726c0 2024-09-16T15:48:12.545 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:12.539+0000 7fc7c37fe640 1 --2- 172.21.15.123:0/2290586655 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fc7a8072fe0 0x7fc7a80754a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:48:12.546 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:12.539+0000 7fc7c37fe640 1 --2- 172.21.15.123:0/2290586655 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fc7a8072fe0 0x7fc7a80754a0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fc7c40c35b0 tx=0x7fc7ac009260 comp rx=0 tx=0).ready entity=mgr.4150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:48:12.549 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:12.543+0000 7fc7c0ff9640 1 -- 172.21.15.123:0/2290586655 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7fc7c41f9f80 con 0x7fc7c40726c0 2024-09-16T15:48:12.719 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:12.715+0000 7fc7ca3ea640 1 -- 172.21.15.123:0/2290586655 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fc7c4073ef0 con 0x7fc7c40726c0 2024-09-16T15:48:12.722 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:12.719+0000 7fc7c0ff9640 1 -- 172.21.15.123:0/2290586655 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v92) ==== 74+0+16911 (secure 0 0 0) 0x7fc7b40bd060 con 0x7fc7c40726c0 2024-09-16T15:48:12.722 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-09-16T15:48:12.722 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":92,"fsid":"56077f89-18c8-486f-ae7e-eecc0bb7822c","created":"2024-09-16T15:42:26.364892+0000","modified":"2024-09-16T15:48:10.302093+0000","last_up_change":"2024-09-16T15:42:38.707234+0000","last_in_change":"2024-09-16T15:42:29.423662+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":15,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-16T15:42:39.287764+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_metadata","create_time":"2024-09-16T15:47:46.961480+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs_data","create_time":"2024-09-16T15:47:48.246604+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9e7c1955-a607-4671-9be7-f2ef760ceeaa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6800","nonce":3882044969},{"type":"v1","addr":"172.21.15.123:6801","nonce":3882044969}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6802","nonce":3882044969},{"type":"v1","addr":"172.21.15.123:6803","nonce":3882044969}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6806","nonce":3882044969},{"type":"v1","addr":"172.21.15.123:6807","nonce":3882044969}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6804","nonce":3882044969},{"type":"v1","addr":"172.21.15.123:6805","nonce":3882044969}]},"public_addr":"172.21.15.123:6801/3882044969","cluster_addr":"172.21.15.123:6803/3882044969","heartbeat_back_addr":"172.21.15.123:6807/3882044969","heartbeat_front_addr":"172.21.15.123:6805/3882044969","state":["exists","up"]},{"osd":1,"uuid":"c540cfde-0977-4742-9b17-fd634668e010","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6816","nonce":4037001850},{"type":"v1","addr":"172.21.15.123:6817","nonce":4037001850}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6818","nonce":4037001850},{"type":"v1","addr":"172.21.15.123:6819","nonce":4037001850}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6822","nonce":4037001850},{"type":"v1","addr":"172.21.15.123:6823","nonce":4037001850}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6820","nonce":4037001850},{"type":"v1","addr":"172.21.15.123:6821","nonce":4037001850}]},"public_addr":"172.21.15.123:6817/4037001850","cluster_addr":"172.21.15.123:6819/4037001850","heartbeat_back_addr":"172.21.15.123:6823/4037001850","heartbeat_front_addr":"172.21.15.123:6821/4037001850","state":["exists","up"]},{"osd":2,"uuid":"9864a496-f169-486b-95c4-1d59162793a7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6824","nonce":1332143088},{"type":"v1","addr":"172.21.15.123:6825","nonce":1332143088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6826","nonce":1332143088},{"type":"v1","addr":"172.21.15.123:6827","nonce":1332143088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6830","nonce":1332143088},{"type":"v1","addr":"172.21.15.123:6831","nonce":1332143088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6828","nonce":1332143088},{"type":"v1","addr":"172.21.15.123:6829","nonce":1332143088}]},"public_addr":"172.21.15.123:6825/1332143088","cluster_addr":"172.21.15.123:6827/1332143088","heartbeat_back_addr":"172.21.15.123:6831/1332143088","heartbeat_front_addr":"172.21.15.123:6829/1332143088","state":["exists","up"]},{"osd":3,"uuid":"c6d1f817-f074-4325-95e1-578eba13edda","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6808","nonce":988586998},{"type":"v1","addr":"172.21.15.123:6809","nonce":988586998}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6810","nonce":988586998},{"type":"v1","addr":"172.21.15.123:6811","nonce":988586998}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6814","nonce":988586998},{"type":"v1","addr":"172.21.15.123:6815","nonce":988586998}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6812","nonce":988586998},{"type":"v1","addr":"172.21.15.123:6813","nonce":988586998}]},"public_addr":"172.21.15.123:6809/988586998","cluster_addr":"172.21.15.123:6811/988586998","heartbeat_back_addr":"172.21.15.123:6815/988586998","heartbeat_front_addr":"172.21.15.123:6813/988586998","state":["exists","up"]},{"osd":4,"uuid":"200a8df4-50ef-40a5-b48a-58981269c086","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6816","nonce":3387813890},{"type":"v1","addr":"172.21.15.184:6817","nonce":3387813890}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6818","nonce":3387813890},{"type":"v1","addr":"172.21.15.184:6819","nonce":3387813890}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6822","nonce":3387813890},{"type":"v1","addr":"172.21.15.184:6823","nonce":3387813890}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6820","nonce":3387813890},{"type":"v1","addr":"172.21.15.184:6821","nonce":3387813890}]},"public_addr":"172.21.15.184:6817/3387813890","cluster_addr":"172.21.15.184:6819/3387813890","heartbeat_back_addr":"172.21.15.184:6823/3387813890","heartbeat_front_addr":"172.21.15.184:6821/3387813890","state":["exists","up"]},{"osd":5,"uuid":"32d5a900-da3d-412c-bc19-13861a2368d8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6808","nonce":2680916017},{"type":"v1","addr":"172.21.15.184:6809","nonce":2680916017}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6810","nonce":2680916017},{"type":"v1","addr":"172.21.15.184:6811","nonce":2680916017}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6814","nonce":2680916017},{"type":"v1","addr":"172.21.15.184:6815","nonce":2680916017}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6812","nonce":2680916017},{"type":"v1","addr":"172.21.15.184:6813","nonce":2680916017}]},"public_addr":"172.21.15.184:6809/2680916017","cluster_addr":"172.21.15.184:6811/2680916017","heartbeat_back_addr":"172.21.15.184:6815/2680916017","heartbeat_front_addr":"172.21.15.184:6813/2680916017","state":["exists","up"]},{"osd":6,"uuid":"cde6f39e-aa1b-4260-ac19-774ee6aaca72","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6824","nonce":2183120457},{"type":"v1","addr":"172.21.15.184:6825","nonce":2183120457}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6826","nonce":2183120457},{"type":"v1","addr":"172.21.15.184:6827","nonce":2183120457}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6830","nonce":2183120457},{"type":"v1","addr":"172.21.15.184:6831","nonce":2183120457}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6828","nonce":2183120457},{"type":"v1","addr":"172.21.15.184:6829","nonce":2183120457}]},"public_addr":"172.21.15.184:6825/2183120457","cluster_addr":"172.21.15.184:6827/2183120457","heartbeat_back_addr":"172.21.15.184:6831/2183120457","heartbeat_front_addr":"172.21.15.184:6829/2183120457","state":["exists","up"]},{"osd":7,"uuid":"4f381778-eee1-47c6-804f-df0e6b33e313","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6800","nonce":4102957285},{"type":"v1","addr":"172.21.15.184:6801","nonce":4102957285}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6802","nonce":4102957285},{"type":"v1","addr":"172.21.15.184:6803","nonce":4102957285}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6806","nonce":4102957285},{"type":"v1","addr":"172.21.15.184:6807","nonce":4102957285}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6804","nonce":4102957285},{"type":"v1","addr":"172.21.15.184:6805","nonce":4102957285}]},"public_addr":"172.21.15.184:6801/4102957285","cluster_addr":"172.21.15.184:6803/4102957285","heartbeat_back_addr":"172.21.15.184:6807/4102957285","heartbeat_front_addr":"172.21.15.184:6805/4102957285","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:35.972210+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.222007+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.198617+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.385663+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.358935+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.156066+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.752973+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:35.682846+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.123:6835/2536398045":"2024-09-17T15:48:07.284775+0000","172.21.15.123:6833/2536398045":"2024-09-17T15:48:07.284775+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T15:48:12.724 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:12.719+0000 7fc7ca3ea640 1 -- 172.21.15.123:0/2290586655 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fc7a8072fe0 msgr2=0x7fc7a80754a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:48:12.724 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:12.719+0000 7fc7ca3ea640 1 --2- 172.21.15.123:0/2290586655 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fc7a8072fe0 0x7fc7a80754a0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fc7c40c35b0 tx=0x7fc7ac009260 comp rx=0 tx=0).stop 2024-09-16T15:48:12.724 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:12.719+0000 7fc7ca3ea640 1 -- 172.21.15.123:0/2290586655 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc7c40726c0 msgr2=0x7fc7c41f6580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:48:12.725 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:12.719+0000 7fc7ca3ea640 1 --2- 172.21.15.123:0/2290586655 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc7c40726c0 0x7fc7c41f6580 secure :-1 s=READY pgs=1001 cs=0 l=1 rev1=1 crypto rx=0x7fc7b4009790 tx=0x7fc7b4005db0 comp rx=0 tx=0).stop 2024-09-16T15:48:12.725 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:12.719+0000 7fc7ca3ea640 1 -- 172.21.15.123:0/2290586655 shutdown_connections 2024-09-16T15:48:12.725 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:12.719+0000 7fc7ca3ea640 1 --2- 172.21.15.123:0/2290586655 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fc7a8072fe0 0x7fc7a80754a0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:12.725 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:12.719+0000 7fc7ca3ea640 1 --2- 172.21.15.123:0/2290586655 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fc7c415bca0 0x7fc7c40c8cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:12.725 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:12.719+0000 7fc7ca3ea640 1 --2- 172.21.15.123:0/2290586655 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc7c4074430 0x7fc7c40c5340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:12.725 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:12.719+0000 7fc7ca3ea640 1 --2- 172.21.15.123:0/2290586655 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc7c40726c0 0x7fc7c41f6580 unknown :-1 s=CLOSED pgs=1001 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:12.725 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:12.719+0000 7fc7ca3ea640 1 -- 172.21.15.123:0/2290586655 >> 172.21.15.123:0/2290586655 conn(0x7fc7c40cfb10 msgr2=0x7fc7c40ca140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:48:12.725 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:12.719+0000 7fc7ca3ea640 1 -- 172.21.15.123:0/2290586655 shutdown_connections 2024-09-16T15:48:12.725 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:12.719+0000 7fc7ca3ea640 1 -- 172.21.15.123:0/2290586655 wait complete. 2024-09-16T15:48:12.740 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-09-16T15:48:12.740 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-16T15:48:12.859 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:12.855+0000 7f923321a640 1 -- 172.21.15.123:0/3112705940 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f922c161620 msgr2=0x7f922c0d1da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:48:12.859 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:12.855+0000 7f923321a640 1 --2- 172.21.15.123:0/3112705940 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f922c161620 0x7f922c0d1da0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f92140098f0 tx=0x7f921401c790 comp rx=0 tx=0).stop 2024-09-16T15:48:12.860 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:12.855+0000 7f923321a640 1 -- 172.21.15.123:0/3112705940 shutdown_connections 2024-09-16T15:48:12.860 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:12.855+0000 7f923321a640 1 --2- 172.21.15.123:0/3112705940 >> v2:172.21.15.184:3300/0 conn(0x7f922c0b0a50 0x7f922c0b0f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:12.860 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:12.855+0000 7f923321a640 1 --2- 172.21.15.123:0/3112705940 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f922c161620 0x7f922c0d1da0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:12.860 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:12.855+0000 7f923321a640 1 -- 172.21.15.123:0/3112705940 >> 172.21.15.123:0/3112705940 conn(0x7f922c0d3ea0 msgr2=0x7f922c0d42b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:48:12.860 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:12.855+0000 7f923321a640 1 -- 172.21.15.123:0/3112705940 shutdown_connections 2024-09-16T15:48:12.860 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:12.855+0000 7f923321a640 1 -- 172.21.15.123:0/3112705940 wait complete. 2024-09-16T15:48:12.860 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:12.855+0000 7f923321a640 1 Processor -- start 2024-09-16T15:48:12.861 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:12.855+0000 7f923321a640 1 -- start start 2024-09-16T15:48:12.863 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:12.855+0000 7f923321a640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f922c0b0a50 0x7f922c155cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:48:12.863 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:12.855+0000 7f923321a640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f922c0b2b00 0x7f922c15c1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:48:12.863 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:12.859+0000 7f923321a640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f922c161620 0x7f922c15c6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:48:12.863 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:12.859+0000 7f923321a640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f922c16b0f0 con 0x7f922c161620 2024-09-16T15:48:12.863 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:12.859+0000 7f923321a640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f922c16af70 con 0x7f922c0b2b00 2024-09-16T15:48:12.863 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:12.859+0000 7f923321a640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f922c16b270 con 0x7f922c0b0a50 2024-09-16T15:48:12.863 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:12.859+0000 7f92317b7640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f922c161620 0x7f922c15c6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:48:12.863 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:12.859+0000 7f92317b7640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f922c161620 0x7f922c15c6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:41420/0 (socket says 172.21.15.123:41420) 2024-09-16T15:48:12.863 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:12.859+0000 7f92317b7640 1 -- 172.21.15.123:0/3060760771 learned_addr learned my addr 172.21.15.123:0/3060760771 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:48:12.863 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:12.859+0000 7f9230fb6640 1 --2- 172.21.15.123:0/3060760771 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f922c0b0a50 0x7f922c155cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:48:12.863 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:12.859+0000 7f922bfff640 1 --2- 172.21.15.123:0/3060760771 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f922c0b2b00 0x7f922c15c1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:48:12.864 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:12.859+0000 7f92317b7640 1 -- 172.21.15.123:0/3060760771 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f922c0b0a50 msgr2=0x7f922c155cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:48:12.864 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:12.859+0000 7f92317b7640 1 --2- 172.21.15.123:0/3060760771 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f922c0b0a50 0x7f922c155cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:12.864 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:12.859+0000 7f92317b7640 1 -- 172.21.15.123:0/3060760771 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f922c0b2b00 msgr2=0x7f922c15c1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:48:12.864 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:12.859+0000 7f92317b7640 1 --2- 172.21.15.123:0/3060760771 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f922c0b2b00 0x7f922c15c1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:12.864 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:12.859+0000 7f92317b7640 1 -- 172.21.15.123:0/3060760771 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f922c1fb190 con 0x7f922c161620 2024-09-16T15:48:12.864 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:12.859+0000 7f922bfff640 1 --2- 172.21.15.123:0/3060760771 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f922c0b2b00 0x7f922c15c1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T15:48:12.864 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:12.859+0000 7f92317b7640 1 --2- 172.21.15.123:0/3060760771 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f922c161620 0x7f922c15c6f0 secure :-1 s=READY pgs=1002 cs=0 l=1 rev1=1 crypto rx=0x7f921c00e8c0 tx=0x7f921c00ed80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:48:12.864 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:12.859+0000 7f92297fa640 1 -- 172.21.15.123:0/3060760771 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f921c00ccc0 con 0x7f922c161620 2024-09-16T15:48:12.864 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:12.859+0000 7f92297fa640 1 -- 172.21.15.123:0/3060760771 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f921c00ce60 con 0x7f922c161620 2024-09-16T15:48:12.864 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:12.859+0000 7f923321a640 1 -- 172.21.15.123:0/3060760771 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f922c1fb470 con 0x7f922c161620 2024-09-16T15:48:12.864 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:12.859+0000 7f92297fa640 1 -- 172.21.15.123:0/3060760771 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f921c005280 con 0x7f922c161620 2024-09-16T15:48:12.864 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:12.859+0000 7f923321a640 1 -- 172.21.15.123:0/3060760771 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f922c1fba30 con 0x7f922c161620 2024-09-16T15:48:12.865 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:12.859+0000 7f923321a640 1 -- 172.21.15.123:0/3060760771 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f922c0b0f00 con 0x7f922c161620 2024-09-16T15:48:12.865 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:12.859+0000 7f92297fa640 1 -- 172.21.15.123:0/3060760771 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 16) ==== 97182+0+0 (secure 0 0 0) 0x7f921c005420 con 0x7f922c161620 2024-09-16T15:48:12.865 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:12.859+0000 7f92297fa640 1 --2- 172.21.15.123:0/3060760771 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f920c072fe0 0x7f920c0754a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:48:12.865 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:12.859+0000 7f9230fb6640 1 --2- 172.21.15.123:0/3060760771 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f920c072fe0 0x7f920c0754a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:48:12.865 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:12.859+0000 7f92297fa640 1 -- 172.21.15.123:0/3060760771 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 5795+0+0 (secure 0 0 0) 0x7f921c094be0 con 0x7f922c161620 2024-09-16T15:48:12.865 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:12.859+0000 7f9230fb6640 1 --2- 172.21.15.123:0/3060760771 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f920c072fe0 0x7f920c0754a0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f92140af000 tx=0x7f921401afd0 comp rx=0 tx=0).ready entity=mgr.4150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:48:12.869 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:12.863+0000 7f92297fa640 1 -- 172.21.15.123:0/3060760771 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f921c05fbc0 con 0x7f922c161620 2024-09-16T15:48:13.063 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:13.059+0000 7f923321a640 1 -- 172.21.15.123:0/3060760771 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f922c15d180 con 0x7f922c161620 2024-09-16T15:48:13.064 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:13.059+0000 7f92297fa640 1 -- 172.21.15.123:0/3060760771 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 59 v59) ==== 94+0+5231 (secure 0 0 0) 0x7f921c05f4c0 con 0x7f922c161620 2024-09-16T15:48:13.065 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-09-16T15:48:13.065 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":59,"btime":"2024-09-16T15:48:08:287830+0000","default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5677,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6835/1609511477","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6833","nonce":1609511477},{"type":"v1","addr":"172.21.15.184:6835","nonce":1609511477}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":48},{"gid":6672,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6834/3658721718","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6832","nonce":3658721718},{"type":"v1","addr":"172.21.15.123:6834","nonce":3658721718}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":48},{"gid":6891,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6835/95665084","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6833","nonce":95665084},{"type":"v1","addr":"172.21.15.123:6835","nonce":95665084}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":59}],"filesystems":[{"mdsmap":{"epoch":53,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-16T15:47:49.295797+0000","modified":"2024-09-16T15:47:52.333498+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6684},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6684":{"gid":6684,"name":"b","rank":0,"incarnation":49,"state":"up:active","state_seq":3,"addr":"172.21.15.184:6834/1546532515","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":1546532515},{"type":"v1","addr":"172.21.15.184:6834","nonce":1546532515}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6684,"qdb_cluster":[6684]},"id":6}]} 2024-09-16T15:48:13.065 INFO:teuthology.orchestra.run.smithi123.stderr:dumped fsmap epoch 59 2024-09-16T15:48:13.067 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:13.063+0000 7f923321a640 1 -- 172.21.15.123:0/3060760771 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f920c072fe0 msgr2=0x7f920c0754a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:48:13.067 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:13.063+0000 7f923321a640 1 --2- 172.21.15.123:0/3060760771 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f920c072fe0 0x7f920c0754a0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f92140af000 tx=0x7f921401afd0 comp rx=0 tx=0).stop 2024-09-16T15:48:13.067 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:13.063+0000 7f923321a640 1 -- 172.21.15.123:0/3060760771 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f922c161620 msgr2=0x7f922c15c6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:48:13.067 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:13.063+0000 7f923321a640 1 --2- 172.21.15.123:0/3060760771 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f922c161620 0x7f922c15c6f0 secure :-1 s=READY pgs=1002 cs=0 l=1 rev1=1 crypto rx=0x7f921c00e8c0 tx=0x7f921c00ed80 comp rx=0 tx=0).stop 2024-09-16T15:48:13.067 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:13.063+0000 7f923321a640 1 -- 172.21.15.123:0/3060760771 shutdown_connections 2024-09-16T15:48:13.068 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:13.063+0000 7f923321a640 1 --2- 172.21.15.123:0/3060760771 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f920c072fe0 0x7f920c0754a0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:13.068 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:13.063+0000 7f923321a640 1 --2- 172.21.15.123:0/3060760771 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f922c161620 0x7f922c15c6f0 unknown :-1 s=CLOSED pgs=1002 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:13.068 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:13.063+0000 7f923321a640 1 --2- 172.21.15.123:0/3060760771 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f922c0b2b00 0x7f922c15c1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:13.068 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:13.063+0000 7f923321a640 1 --2- 172.21.15.123:0/3060760771 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f922c0b0a50 0x7f922c155cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:13.068 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:13.063+0000 7f923321a640 1 -- 172.21.15.123:0/3060760771 >> 172.21.15.123:0/3060760771 conn(0x7f922c0d3ea0 msgr2=0x7f922c06dbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:48:13.068 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:13.063+0000 7f923321a640 1 -- 172.21.15.123:0/3060760771 shutdown_connections 2024-09-16T15:48:13.068 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:13.063+0000 7f923321a640 1 -- 172.21.15.123:0/3060760771 wait complete. 2024-09-16T15:48:13.082 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-16T15:48:13.193 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:13.187+0000 7f0baa1c8640 1 -- 172.21.15.123:0/1629654174 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f0ba4158120 msgr2=0x7f0ba40ca320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:48:13.193 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:13.187+0000 7f0baa1c8640 1 --2- 172.21.15.123:0/1629654174 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f0ba4158120 0x7f0ba40ca320 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f0b98009900 tx=0x7f0b9801c850 comp rx=0 tx=0).stop 2024-09-16T15:48:13.193 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:13.187+0000 7f0baa1c8640 1 -- 172.21.15.123:0/1629654174 shutdown_connections 2024-09-16T15:48:13.194 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:13.187+0000 7f0baa1c8640 1 --2- 172.21.15.123:0/1629654174 >> v2:172.21.15.184:3300/0 conn(0x7f0ba406f5f0 0x7f0ba415aa60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:13.194 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:13.187+0000 7f0baa1c8640 1 --2- 172.21.15.123:0/1629654174 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f0ba4158120 0x7f0ba40ca320 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:13.194 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:13.187+0000 7f0baa1c8640 1 -- 172.21.15.123:0/1629654174 >> 172.21.15.123:0/1629654174 conn(0x7f0ba40cfb10 msgr2=0x7f0ba40cff20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:48:13.194 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:13.187+0000 7f0baa1c8640 1 -- 172.21.15.123:0/1629654174 shutdown_connections 2024-09-16T15:48:13.194 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:13.187+0000 7f0baa1c8640 1 -- 172.21.15.123:0/1629654174 wait complete. 2024-09-16T15:48:13.194 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:13.191+0000 7f0baa1c8640 1 Processor -- start 2024-09-16T15:48:13.194 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:13.191+0000 7f0baa1c8640 1 -- start start 2024-09-16T15:48:13.196 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:13.191+0000 7f0baa1c8640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f0ba406f5f0 0x7f0ba4149e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:48:13.196 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:13.191+0000 7f0baa1c8640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0ba40cf5a0 0x7f0ba41f66a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:48:13.196 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:13.191+0000 7f0baa1c8640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f0ba4158120 0x7f0ba40b3fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:48:13.196 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:13.191+0000 7f0baa1c8640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f0ba41673d0 con 0x7f0ba40cf5a0 2024-09-16T15:48:13.197 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:13.191+0000 7f0baa1c8640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f0ba406ecd0 con 0x7f0ba4158120 2024-09-16T15:48:13.197 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:13.191+0000 7f0baa1c8640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f0ba4167550 con 0x7f0ba406f5f0 2024-09-16T15:48:13.197 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:13.191+0000 7f0ba2ffd640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0ba40cf5a0 0x7f0ba41f66a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:48:13.197 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:13.191+0000 7f0ba2ffd640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0ba40cf5a0 0x7f0ba41f66a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:41432/0 (socket says 172.21.15.123:41432) 2024-09-16T15:48:13.197 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:13.191+0000 7f0ba2ffd640 1 -- 172.21.15.123:0/3030749775 learned_addr learned my addr 172.21.15.123:0/3030749775 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:48:13.197 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:13.191+0000 7f0ba37fe640 1 --2- 172.21.15.123:0/3030749775 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f0ba406f5f0 0x7f0ba4149e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:48:13.197 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:13.191+0000 7f0ba2ffd640 1 -- 172.21.15.123:0/3030749775 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f0ba406f5f0 msgr2=0x7f0ba4149e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:48:13.197 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:13.191+0000 7f0ba3fff640 1 --2- 172.21.15.123:0/3030749775 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f0ba4158120 0x7f0ba40b3fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:48:13.197 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:13.191+0000 7f0ba2ffd640 1 --2- 172.21.15.123:0/3030749775 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f0ba406f5f0 0x7f0ba4149e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:13.197 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:13.191+0000 7f0ba2ffd640 1 -- 172.21.15.123:0/3030749775 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f0ba4158120 msgr2=0x7f0ba40b3fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:48:13.197 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:13.191+0000 7f0ba2ffd640 1 --2- 172.21.15.123:0/3030749775 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f0ba4158120 0x7f0ba40b3fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:13.197 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:13.191+0000 7f0ba2ffd640 1 -- 172.21.15.123:0/3030749775 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0ba41f93e0 con 0x7f0ba40cf5a0 2024-09-16T15:48:13.197 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:13.191+0000 7f0ba2ffd640 1 --2- 172.21.15.123:0/3030749775 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0ba40cf5a0 0x7f0ba41f66a0 secure :-1 s=READY pgs=1003 cs=0 l=1 rev1=1 crypto rx=0x7f0b9400c430 tx=0x7f0b9400c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:48:13.197 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:13.191+0000 7f0b8bfff640 1 -- 172.21.15.123:0/3030749775 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b94017070 con 0x7f0ba40cf5a0 2024-09-16T15:48:13.198 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:13.191+0000 7f0baa1c8640 1 -- 172.21.15.123:0/3030749775 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0ba41f96c0 con 0x7f0ba40cf5a0 2024-09-16T15:48:13.198 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:13.191+0000 7f0b8bfff640 1 -- 172.21.15.123:0/3030749775 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f0b940053e0 con 0x7f0ba40cf5a0 2024-09-16T15:48:13.198 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:13.191+0000 7f0baa1c8640 1 -- 172.21.15.123:0/3030749775 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0ba41f9c80 con 0x7f0ba40cf5a0 2024-09-16T15:48:13.198 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:13.191+0000 7f0b8bfff640 1 -- 172.21.15.123:0/3030749775 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b94003860 con 0x7f0ba40cf5a0 2024-09-16T15:48:13.199 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:13.191+0000 7f0baa1c8640 1 -- 172.21.15.123:0/3030749775 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0b840052a0 con 0x7f0ba40cf5a0 2024-09-16T15:48:13.199 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:13.195+0000 7f0b8bfff640 1 -- 172.21.15.123:0/3030749775 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 16) ==== 97182+0+0 (secure 0 0 0) 0x7f0b9401a070 con 0x7f0ba40cf5a0 2024-09-16T15:48:13.199 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:13.195+0000 7f0b8bfff640 1 --2- 172.21.15.123:0/3030749775 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f0b90072fe0 0x7f0b900754a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:48:13.199 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:13.195+0000 7f0b8bfff640 1 -- 172.21.15.123:0/3030749775 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 5795+0+0 (secure 0 0 0) 0x7f0b94093400 con 0x7f0ba40cf5a0 2024-09-16T15:48:13.199 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:13.195+0000 7f0ba37fe640 1 --2- 172.21.15.123:0/3030749775 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f0b90072fe0 0x7f0b900754a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:48:13.199 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:13.195+0000 7f0ba37fe640 1 --2- 172.21.15.123:0/3030749775 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f0b90072fe0 0x7f0b900754a0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f0b980af000 tx=0x7f0b9801b090 comp rx=0 tx=0).ready entity=mgr.4150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:48:13.202 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:13.199+0000 7f0b8bfff640 1 -- 172.21.15.123:0/3030749775 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f0b94001130 con 0x7f0ba40cf5a0 2024-09-16T15:48:13.396 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:13.391+0000 7f0baa1c8640 1 -- 172.21.15.123:0/3030749775 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f0b84005860 con 0x7f0ba40cf5a0 2024-09-16T15:48:13.397 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:13.391+0000 7f0b8bfff640 1 -- 172.21.15.123:0/3030749775 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 59 v59) ==== 94+0+5231 (secure 0 0 0) 0x7f0b9405e2c0 con 0x7f0ba40cf5a0 2024-09-16T15:48:13.397 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-09-16T15:48:13.397 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":59,"btime":"2024-09-16T15:48:08:287830+0000","default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5677,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6835/1609511477","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6833","nonce":1609511477},{"type":"v1","addr":"172.21.15.184:6835","nonce":1609511477}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":48},{"gid":6672,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6834/3658721718","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6832","nonce":3658721718},{"type":"v1","addr":"172.21.15.123:6834","nonce":3658721718}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":48},{"gid":6891,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6835/95665084","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6833","nonce":95665084},{"type":"v1","addr":"172.21.15.123:6835","nonce":95665084}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":59}],"filesystems":[{"mdsmap":{"epoch":53,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-16T15:47:49.295797+0000","modified":"2024-09-16T15:47:52.333498+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6684},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6684":{"gid":6684,"name":"b","rank":0,"incarnation":49,"state":"up:active","state_seq":3,"addr":"172.21.15.184:6834/1546532515","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":1546532515},{"type":"v1","addr":"172.21.15.184:6834","nonce":1546532515}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6684,"qdb_cluster":[6684]},"id":6}]} 2024-09-16T15:48:13.397 INFO:teuthology.orchestra.run.smithi123.stderr:dumped fsmap epoch 59 2024-09-16T15:48:13.400 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:13.395+0000 7f0baa1c8640 1 -- 172.21.15.123:0/3030749775 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f0b90072fe0 msgr2=0x7f0b900754a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:48:13.400 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:13.395+0000 7f0baa1c8640 1 --2- 172.21.15.123:0/3030749775 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f0b90072fe0 0x7f0b900754a0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f0b980af000 tx=0x7f0b9801b090 comp rx=0 tx=0).stop 2024-09-16T15:48:13.400 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:13.395+0000 7f0baa1c8640 1 -- 172.21.15.123:0/3030749775 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0ba40cf5a0 msgr2=0x7f0ba41f66a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:48:13.400 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:13.395+0000 7f0baa1c8640 1 --2- 172.21.15.123:0/3030749775 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0ba40cf5a0 0x7f0ba41f66a0 secure :-1 s=READY pgs=1003 cs=0 l=1 rev1=1 crypto rx=0x7f0b9400c430 tx=0x7f0b9400c8f0 comp rx=0 tx=0).stop 2024-09-16T15:48:13.400 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:13.395+0000 7f0baa1c8640 1 -- 172.21.15.123:0/3030749775 shutdown_connections 2024-09-16T15:48:13.400 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:13.395+0000 7f0baa1c8640 1 --2- 172.21.15.123:0/3030749775 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f0b90072fe0 0x7f0b900754a0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:13.400 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:13.395+0000 7f0baa1c8640 1 --2- 172.21.15.123:0/3030749775 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f0ba4158120 0x7f0ba40b3fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:13.400 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:13.395+0000 7f0baa1c8640 1 --2- 172.21.15.123:0/3030749775 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0ba40cf5a0 0x7f0ba41f66a0 unknown :-1 s=CLOSED pgs=1003 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:13.400 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:13.395+0000 7f0baa1c8640 1 --2- 172.21.15.123:0/3030749775 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f0ba406f5f0 0x7f0ba4149e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:13.400 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:13.395+0000 7f0baa1c8640 1 -- 172.21.15.123:0/3030749775 >> 172.21.15.123:0/3030749775 conn(0x7f0ba40cfb10 msgr2=0x7f0ba406eee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:48:13.400 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:13.395+0000 7f0baa1c8640 1 -- 172.21.15.123:0/3030749775 shutdown_connections 2024-09-16T15:48:13.400 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:13.395+0000 7f0baa1c8640 1 -- 172.21.15.123:0/3030749775 wait complete. 2024-09-16T15:48:13.414 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-16T15:48:13.536 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:13.531+0000 7f3aadbf0640 1 -- 172.21.15.123:0/2599342922 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3aa80b8630 msgr2=0x7f3aa80d0d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:48:13.536 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:13.531+0000 7f3aadbf0640 1 --2- 172.21.15.123:0/2599342922 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3aa80b8630 0x7f3aa80d0d80 secure :-1 s=READY pgs=1005 cs=0 l=1 rev1=1 crypto rx=0x7f3aa806a110 tx=0x7f3aa001cd80 comp rx=0 tx=0).stop 2024-09-16T15:48:13.536 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:13.531+0000 7f3aadbf0640 1 -- 172.21.15.123:0/2599342922 shutdown_connections 2024-09-16T15:48:13.536 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:13.531+0000 7f3aadbf0640 1 --2- 172.21.15.123:0/2599342922 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3aa80b8630 0x7f3aa80d0d80 unknown :-1 s=CLOSED pgs=1005 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:13.536 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:13.531+0000 7f3aadbf0640 1 --2- 172.21.15.123:0/2599342922 >> v2:172.21.15.184:3300/0 conn(0x7f3aa8161520 0x7f3aa80d0840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:13.536 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:13.531+0000 7f3aadbf0640 1 -- 172.21.15.123:0/2599342922 >> 172.21.15.123:0/2599342922 conn(0x7f3aa80d3e30 msgr2=0x7f3aa80d4240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:48:13.536 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:13.531+0000 7f3aadbf0640 1 -- 172.21.15.123:0/2599342922 shutdown_connections 2024-09-16T15:48:13.536 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:13.531+0000 7f3aadbf0640 1 -- 172.21.15.123:0/2599342922 wait complete. 2024-09-16T15:48:13.537 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:13.531+0000 7f3aadbf0640 1 Processor -- start 2024-09-16T15:48:13.537 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:13.531+0000 7f3aadbf0640 1 -- start start 2024-09-16T15:48:13.538 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:13.531+0000 7f3aadbf0640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3aa80b8630 0x7f3aa81565a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:48:13.538 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:13.531+0000 7f3aadbf0640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3aa80d23c0 0x7f3aa8142c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:48:13.539 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:13.531+0000 7f3aadbf0640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f3aa8161520 0x7f3aa8157770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:48:13.539 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:13.531+0000 7f3aadbf0640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f3aa816b310 con 0x7f3aa80b8630 2024-09-16T15:48:13.539 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:13.531+0000 7f3aadbf0640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f3aa816b190 con 0x7f3aa80d23c0 2024-09-16T15:48:13.539 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:13.531+0000 7f3aadbf0640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f3aa816b490 con 0x7f3aa8161520 2024-09-16T15:48:13.539 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:13.531+0000 7f3aa77fe640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3aa80b8630 0x7f3aa81565a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:48:13.539 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:13.531+0000 7f3aa77fe640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3aa80b8630 0x7f3aa81565a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:41446/0 (socket says 172.21.15.123:41446) 2024-09-16T15:48:13.539 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:13.531+0000 7f3aa77fe640 1 -- 172.21.15.123:0/3831007469 learned_addr learned my addr 172.21.15.123:0/3831007469 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:48:13.539 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:13.535+0000 7f3aa77fe640 1 -- 172.21.15.123:0/3831007469 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f3aa8161520 msgr2=0x7f3aa8157770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:48:13.539 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:13.535+0000 7f3aa77fe640 1 --2- 172.21.15.123:0/3831007469 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f3aa8161520 0x7f3aa8157770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:13.539 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:13.535+0000 7f3aa77fe640 1 -- 172.21.15.123:0/3831007469 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3aa80d23c0 msgr2=0x7f3aa8142c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:48:13.539 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:13.535+0000 7f3aa77fe640 1 --2- 172.21.15.123:0/3831007469 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3aa80d23c0 0x7f3aa8142c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:13.539 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:13.535+0000 7f3aa77fe640 1 -- 172.21.15.123:0/3831007469 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3aa80c9110 con 0x7f3aa80b8630 2024-09-16T15:48:13.539 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:13.535+0000 7f3aa77fe640 1 --2- 172.21.15.123:0/3831007469 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3aa80b8630 0x7f3aa81565a0 secure :-1 s=READY pgs=1006 cs=0 l=1 rev1=1 crypto rx=0x7f3a9800b410 tx=0x7f3a9800b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:48:13.540 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:13.535+0000 7f3a8ffff640 1 -- 172.21.15.123:0/3831007469 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a98015020 con 0x7f3aa80b8630 2024-09-16T15:48:13.540 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:13.535+0000 7f3a8ffff640 1 -- 172.21.15.123:0/3831007469 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f3a98011920 con 0x7f3aa80b8630 2024-09-16T15:48:13.540 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:13.535+0000 7f3a8ffff640 1 -- 172.21.15.123:0/3831007469 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a98011d70 con 0x7f3aa80b8630 2024-09-16T15:48:13.540 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:13.535+0000 7f3aadbf0640 1 -- 172.21.15.123:0/3831007469 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3aa80c93f0 con 0x7f3aa80b8630 2024-09-16T15:48:13.540 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:13.535+0000 7f3aadbf0640 1 -- 172.21.15.123:0/3831007469 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3aa8156ae0 con 0x7f3aa80b8630 2024-09-16T15:48:13.541 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:13.535+0000 7f3aadbf0640 1 -- 172.21.15.123:0/3831007469 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3aa80cea40 con 0x7f3aa80b8630 2024-09-16T15:48:13.541 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:13.535+0000 7f3a8ffff640 1 -- 172.21.15.123:0/3831007469 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 16) ==== 97182+0+0 (secure 0 0 0) 0x7f3a98018020 con 0x7f3aa80b8630 2024-09-16T15:48:13.541 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:13.535+0000 7f3a8ffff640 1 --2- 172.21.15.123:0/3831007469 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f3a94072fe0 0x7f3a940754a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:48:13.541 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:13.535+0000 7f3aa6ffd640 1 --2- 172.21.15.123:0/3831007469 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f3a94072fe0 0x7f3a940754a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:48:13.541 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:13.535+0000 7f3a8ffff640 1 -- 172.21.15.123:0/3831007469 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 5795+0+0 (secure 0 0 0) 0x7f3a98093f90 con 0x7f3aa80b8630 2024-09-16T15:48:13.541 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:13.535+0000 7f3aa6ffd640 1 --2- 172.21.15.123:0/3831007469 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f3a94072fe0 0x7f3a940754a0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f3a9c007fd0 tx=0x7f3a9c0078e0 comp rx=0 tx=0).ready entity=mgr.4150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:48:13.544 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:13.539+0000 7f3a8ffff640 1 -- 172.21.15.123:0/3831007469 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f3a9801f020 con 0x7f3aa80b8630 2024-09-16T15:48:13.706 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:13.703+0000 7f3aadbf0640 1 -- 172.21.15.123:0/3831007469 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f3aa80c9ab0 con 0x7f3aa80b8630 2024-09-16T15:48:13.709 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:13.703+0000 7f3a8ffff640 1 -- 172.21.15.123:0/3831007469 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v92) ==== 74+0+16911 (secure 0 0 0) 0x7f3a9805eff0 con 0x7f3aa80b8630 2024-09-16T15:48:13.709 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-09-16T15:48:13.709 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":92,"fsid":"56077f89-18c8-486f-ae7e-eecc0bb7822c","created":"2024-09-16T15:42:26.364892+0000","modified":"2024-09-16T15:48:10.302093+0000","last_up_change":"2024-09-16T15:42:38.707234+0000","last_in_change":"2024-09-16T15:42:29.423662+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":15,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-16T15:42:39.287764+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_metadata","create_time":"2024-09-16T15:47:46.961480+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs_data","create_time":"2024-09-16T15:47:48.246604+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9e7c1955-a607-4671-9be7-f2ef760ceeaa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6800","nonce":3882044969},{"type":"v1","addr":"172.21.15.123:6801","nonce":3882044969}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6802","nonce":3882044969},{"type":"v1","addr":"172.21.15.123:6803","nonce":3882044969}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6806","nonce":3882044969},{"type":"v1","addr":"172.21.15.123:6807","nonce":3882044969}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6804","nonce":3882044969},{"type":"v1","addr":"172.21.15.123:6805","nonce":3882044969}]},"public_addr":"172.21.15.123:6801/3882044969","cluster_addr":"172.21.15.123:6803/3882044969","heartbeat_back_addr":"172.21.15.123:6807/3882044969","heartbeat_front_addr":"172.21.15.123:6805/3882044969","state":["exists","up"]},{"osd":1,"uuid":"c540cfde-0977-4742-9b17-fd634668e010","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6816","nonce":4037001850},{"type":"v1","addr":"172.21.15.123:6817","nonce":4037001850}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6818","nonce":4037001850},{"type":"v1","addr":"172.21.15.123:6819","nonce":4037001850}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6822","nonce":4037001850},{"type":"v1","addr":"172.21.15.123:6823","nonce":4037001850}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6820","nonce":4037001850},{"type":"v1","addr":"172.21.15.123:6821","nonce":4037001850}]},"public_addr":"172.21.15.123:6817/4037001850","cluster_addr":"172.21.15.123:6819/4037001850","heartbeat_back_addr":"172.21.15.123:6823/4037001850","heartbeat_front_addr":"172.21.15.123:6821/4037001850","state":["exists","up"]},{"osd":2,"uuid":"9864a496-f169-486b-95c4-1d59162793a7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6824","nonce":1332143088},{"type":"v1","addr":"172.21.15.123:6825","nonce":1332143088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6826","nonce":1332143088},{"type":"v1","addr":"172.21.15.123:6827","nonce":1332143088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6830","nonce":1332143088},{"type":"v1","addr":"172.21.15.123:6831","nonce":1332143088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6828","nonce":1332143088},{"type":"v1","addr":"172.21.15.123:6829","nonce":1332143088}]},"public_addr":"172.21.15.123:6825/1332143088","cluster_addr":"172.21.15.123:6827/1332143088","heartbeat_back_addr":"172.21.15.123:6831/1332143088","heartbeat_front_addr":"172.21.15.123:6829/1332143088","state":["exists","up"]},{"osd":3,"uuid":"c6d1f817-f074-4325-95e1-578eba13edda","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6808","nonce":988586998},{"type":"v1","addr":"172.21.15.123:6809","nonce":988586998}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6810","nonce":988586998},{"type":"v1","addr":"172.21.15.123:6811","nonce":988586998}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6814","nonce":988586998},{"type":"v1","addr":"172.21.15.123:6815","nonce":988586998}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6812","nonce":988586998},{"type":"v1","addr":"172.21.15.123:6813","nonce":988586998}]},"public_addr":"172.21.15.123:6809/988586998","cluster_addr":"172.21.15.123:6811/988586998","heartbeat_back_addr":"172.21.15.123:6815/988586998","heartbeat_front_addr":"172.21.15.123:6813/988586998","state":["exists","up"]},{"osd":4,"uuid":"200a8df4-50ef-40a5-b48a-58981269c086","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6816","nonce":3387813890},{"type":"v1","addr":"172.21.15.184:6817","nonce":3387813890}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6818","nonce":3387813890},{"type":"v1","addr":"172.21.15.184:6819","nonce":3387813890}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6822","nonce":3387813890},{"type":"v1","addr":"172.21.15.184:6823","nonce":3387813890}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6820","nonce":3387813890},{"type":"v1","addr":"172.21.15.184:6821","nonce":3387813890}]},"public_addr":"172.21.15.184:6817/3387813890","cluster_addr":"172.21.15.184:6819/3387813890","heartbeat_back_addr":"172.21.15.184:6823/3387813890","heartbeat_front_addr":"172.21.15.184:6821/3387813890","state":["exists","up"]},{"osd":5,"uuid":"32d5a900-da3d-412c-bc19-13861a2368d8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6808","nonce":2680916017},{"type":"v1","addr":"172.21.15.184:6809","nonce":2680916017}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6810","nonce":2680916017},{"type":"v1","addr":"172.21.15.184:6811","nonce":2680916017}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6814","nonce":2680916017},{"type":"v1","addr":"172.21.15.184:6815","nonce":2680916017}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6812","nonce":2680916017},{"type":"v1","addr":"172.21.15.184:6813","nonce":2680916017}]},"public_addr":"172.21.15.184:6809/2680916017","cluster_addr":"172.21.15.184:6811/2680916017","heartbeat_back_addr":"172.21.15.184:6815/2680916017","heartbeat_front_addr":"172.21.15.184:6813/2680916017","state":["exists","up"]},{"osd":6,"uuid":"cde6f39e-aa1b-4260-ac19-774ee6aaca72","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6824","nonce":2183120457},{"type":"v1","addr":"172.21.15.184:6825","nonce":2183120457}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6826","nonce":2183120457},{"type":"v1","addr":"172.21.15.184:6827","nonce":2183120457}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6830","nonce":2183120457},{"type":"v1","addr":"172.21.15.184:6831","nonce":2183120457}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6828","nonce":2183120457},{"type":"v1","addr":"172.21.15.184:6829","nonce":2183120457}]},"public_addr":"172.21.15.184:6825/2183120457","cluster_addr":"172.21.15.184:6827/2183120457","heartbeat_back_addr":"172.21.15.184:6831/2183120457","heartbeat_front_addr":"172.21.15.184:6829/2183120457","state":["exists","up"]},{"osd":7,"uuid":"4f381778-eee1-47c6-804f-df0e6b33e313","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6800","nonce":4102957285},{"type":"v1","addr":"172.21.15.184:6801","nonce":4102957285}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6802","nonce":4102957285},{"type":"v1","addr":"172.21.15.184:6803","nonce":4102957285}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6806","nonce":4102957285},{"type":"v1","addr":"172.21.15.184:6807","nonce":4102957285}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6804","nonce":4102957285},{"type":"v1","addr":"172.21.15.184:6805","nonce":4102957285}]},"public_addr":"172.21.15.184:6801/4102957285","cluster_addr":"172.21.15.184:6803/4102957285","heartbeat_back_addr":"172.21.15.184:6807/4102957285","heartbeat_front_addr":"172.21.15.184:6805/4102957285","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:35.972210+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.222007+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.198617+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.385663+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.358935+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.156066+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.752973+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:35.682846+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.123:6835/2536398045":"2024-09-17T15:48:07.284775+0000","172.21.15.123:6833/2536398045":"2024-09-17T15:48:07.284775+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T15:48:13.711 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:13.707+0000 7f3aadbf0640 1 -- 172.21.15.123:0/3831007469 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f3a94072fe0 msgr2=0x7f3a940754a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:48:13.711 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:13.707+0000 7f3aadbf0640 1 --2- 172.21.15.123:0/3831007469 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f3a94072fe0 0x7f3a940754a0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f3a9c007fd0 tx=0x7f3a9c0078e0 comp rx=0 tx=0).stop 2024-09-16T15:48:13.711 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:13.707+0000 7f3aadbf0640 1 -- 172.21.15.123:0/3831007469 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3aa80b8630 msgr2=0x7f3aa81565a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:48:13.711 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:13.707+0000 7f3aadbf0640 1 --2- 172.21.15.123:0/3831007469 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3aa80b8630 0x7f3aa81565a0 secure :-1 s=READY pgs=1006 cs=0 l=1 rev1=1 crypto rx=0x7f3a9800b410 tx=0x7f3a9800b8d0 comp rx=0 tx=0).stop 2024-09-16T15:48:13.711 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:13.707+0000 7f3aadbf0640 1 -- 172.21.15.123:0/3831007469 shutdown_connections 2024-09-16T15:48:13.712 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:13.707+0000 7f3aadbf0640 1 --2- 172.21.15.123:0/3831007469 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f3a94072fe0 0x7f3a940754a0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:13.712 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:13.707+0000 7f3aadbf0640 1 --2- 172.21.15.123:0/3831007469 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f3aa8161520 0x7f3aa8157770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:13.712 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:13.707+0000 7f3aadbf0640 1 --2- 172.21.15.123:0/3831007469 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3aa80d23c0 0x7f3aa8142c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:13.712 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:13.707+0000 7f3aadbf0640 1 --2- 172.21.15.123:0/3831007469 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3aa80b8630 0x7f3aa81565a0 unknown :-1 s=CLOSED pgs=1006 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:13.712 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:13.707+0000 7f3aadbf0640 1 -- 172.21.15.123:0/3831007469 >> 172.21.15.123:0/3831007469 conn(0x7f3aa80d3e30 msgr2=0x7f3aa806ec90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:48:13.712 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:13.707+0000 7f3aadbf0640 1 -- 172.21.15.123:0/3831007469 shutdown_connections 2024-09-16T15:48:13.712 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:13.707+0000 7f3aadbf0640 1 -- 172.21.15.123:0/3831007469 wait complete. 2024-09-16T15:48:13.727 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-09-16T15:48:13.844 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:13.839+0000 7faf8c5fe640 1 -- 172.21.15.123:0/2505757918 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7faf840d23c0 msgr2=0x7faf840d0d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:48:13.844 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:13.839+0000 7faf8c5fe640 1 --2- 172.21.15.123:0/2505757918 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7faf840d23c0 0x7faf840d0d80 secure :-1 s=READY pgs=1007 cs=0 l=1 rev1=1 crypto rx=0x7faf8406a110 tx=0x7faf7801cd80 comp rx=0 tx=0).stop 2024-09-16T15:48:13.844 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:13.839+0000 7faf8c5fe640 1 -- 172.21.15.123:0/2505757918 shutdown_connections 2024-09-16T15:48:13.844 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:13.839+0000 7faf8c5fe640 1 --2- 172.21.15.123:0/2505757918 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7faf840d23c0 0x7faf840d0d80 unknown :-1 s=CLOSED pgs=1007 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:13.844 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:13.839+0000 7faf8c5fe640 1 --2- 172.21.15.123:0/2505757918 >> v2:172.21.15.184:3300/0 conn(0x7faf84161520 0x7faf840d0840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:13.844 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:13.839+0000 7faf8c5fe640 1 -- 172.21.15.123:0/2505757918 >> 172.21.15.123:0/2505757918 conn(0x7faf840d3e30 msgr2=0x7faf840d4240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:48:13.844 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:13.839+0000 7faf8c5fe640 1 -- 172.21.15.123:0/2505757918 shutdown_connections 2024-09-16T15:48:13.845 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:13.839+0000 7faf8c5fe640 1 -- 172.21.15.123:0/2505757918 wait complete. 2024-09-16T15:48:13.846 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:13.839+0000 7faf8c5fe640 1 Processor -- start 2024-09-16T15:48:13.846 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:13.839+0000 7faf8c5fe640 1 -- start start 2024-09-16T15:48:13.846 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:13.839+0000 7faf8c5fe640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7faf840d23c0 0x7faf84154330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:48:13.847 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:13.839+0000 7faf8c5fe640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7faf840d4950 0x7faf841587d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:48:13.847 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:13.839+0000 7faf8c5fe640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7faf84161520 0x7faf84155800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:48:13.847 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:13.839+0000 7faf8c5fe640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7faf8416b3a0 con 0x7faf840d4950 2024-09-16T15:48:13.847 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:13.839+0000 7faf8c5fe640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7faf8416b220 con 0x7faf840d23c0 2024-09-16T15:48:13.847 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:13.839+0000 7faf8c5fe640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7faf8416b520 con 0x7faf84161520 2024-09-16T15:48:13.847 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:13.839+0000 7faf89b99640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7faf840d4950 0x7faf841587d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:48:13.847 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:13.839+0000 7faf89b99640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7faf840d4950 0x7faf841587d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:41454/0 (socket says 172.21.15.123:41454) 2024-09-16T15:48:13.847 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:13.839+0000 7faf89b99640 1 -- 172.21.15.123:0/3091332181 learned_addr learned my addr 172.21.15.123:0/3091332181 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:48:13.847 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:13.843+0000 7faf89b99640 1 -- 172.21.15.123:0/3091332181 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7faf84161520 msgr2=0x7faf84155800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:48:13.847 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:13.843+0000 7faf89b99640 1 --2- 172.21.15.123:0/3091332181 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7faf84161520 0x7faf84155800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:13.847 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:13.843+0000 7faf89b99640 1 -- 172.21.15.123:0/3091332181 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7faf840d23c0 msgr2=0x7faf84154330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:48:13.847 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:13.843+0000 7faf8a39a640 1 --2- 172.21.15.123:0/3091332181 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7faf840d23c0 0x7faf84154330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:48:13.847 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:13.843+0000 7faf89b99640 1 --2- 172.21.15.123:0/3091332181 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7faf840d23c0 0x7faf84154330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:13.847 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:13.843+0000 7faf89b99640 1 -- 172.21.15.123:0/3091332181 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faf84159260 con 0x7faf840d4950 2024-09-16T15:48:13.848 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:13.843+0000 7faf89b99640 1 --2- 172.21.15.123:0/3091332181 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7faf840d4950 0x7faf841587d0 secure :-1 s=READY pgs=1008 cs=0 l=1 rev1=1 crypto rx=0x7faf78000f90 tx=0x7faf780a5a90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:48:13.848 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:13.843+0000 7faf82ffd640 1 -- 172.21.15.123:0/3091332181 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf780bb070 con 0x7faf840d4950 2024-09-16T15:48:13.848 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:13.843+0000 7faf82ffd640 1 -- 172.21.15.123:0/3091332181 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7faf78004d40 con 0x7faf840d4950 2024-09-16T15:48:13.848 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:13.843+0000 7faf8c5fe640 1 -- 172.21.15.123:0/3091332181 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faf84159540 con 0x7faf840d4950 2024-09-16T15:48:13.848 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:13.843+0000 7faf82ffd640 1 -- 172.21.15.123:0/3091332181 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf780b2040 con 0x7faf840d4950 2024-09-16T15:48:13.848 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:13.843+0000 7faf8c5fe640 1 -- 172.21.15.123:0/3091332181 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faf841552e0 con 0x7faf840d4950 2024-09-16T15:48:13.849 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:13.843+0000 7faf8c5fe640 1 -- 172.21.15.123:0/3091332181 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faf840cea40 con 0x7faf840d4950 2024-09-16T15:48:13.849 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:13.843+0000 7faf82ffd640 1 -- 172.21.15.123:0/3091332181 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 16) ==== 97182+0+0 (secure 0 0 0) 0x7faf780be070 con 0x7faf840d4950 2024-09-16T15:48:13.849 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:13.843+0000 7faf82ffd640 1 --2- 172.21.15.123:0/3091332181 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7faf70072fe0 0x7faf700754a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:48:13.849 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:13.843+0000 7faf8a39a640 1 --2- 172.21.15.123:0/3091332181 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7faf70072fe0 0x7faf700754a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:48:13.849 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:13.843+0000 7faf82ffd640 1 -- 172.21.15.123:0/3091332181 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 5795+0+0 (secure 0 0 0) 0x7faf7812e150 con 0x7faf840d4950 2024-09-16T15:48:13.849 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:13.843+0000 7faf8a39a640 1 --2- 172.21.15.123:0/3091332181 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7faf70072fe0 0x7faf700754a0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7faf740058d0 tx=0x7faf74005bd0 comp rx=0 tx=0).ready entity=mgr.4150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:48:13.852 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:13.847+0000 7faf82ffd640 1 -- 172.21.15.123:0/3091332181 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7faf78016970 con 0x7faf840d4950 2024-09-16T15:48:14.054 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:14.051+0000 7faf8c5fe640 1 -- 172.21.15.123:0/3091332181 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7faf841554b0 con 0x7faf840d4950 2024-09-16T15:48:14.055 INFO:tasks.ceph.mon.a.smithi123.stderr:2024-09-16T15:48:14.051+0000 7f268d076640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-09-16T15:48:14.057 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:14.051+0000 7faf82ffd640 1 -- 172.21.15.123:0/3091332181 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v60) ==== 161+0+0 (secure 0 0 0) 0x7faf780f9130 con 0x7faf840d4950 2024-09-16T15:48:14.057 INFO:teuthology.orchestra.run.smithi123.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-09-16T15:48:14.062 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:14.059+0000 7faf8c5fe640 1 -- 172.21.15.123:0/3091332181 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7faf70072fe0 msgr2=0x7faf700754a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:48:14.063 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:14.059+0000 7faf8c5fe640 1 --2- 172.21.15.123:0/3091332181 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7faf70072fe0 0x7faf700754a0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7faf740058d0 tx=0x7faf74005bd0 comp rx=0 tx=0).stop 2024-09-16T15:48:14.063 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:14.059+0000 7faf8c5fe640 1 -- 172.21.15.123:0/3091332181 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7faf840d4950 msgr2=0x7faf841587d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:48:14.063 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:14.059+0000 7faf8c5fe640 1 --2- 172.21.15.123:0/3091332181 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7faf840d4950 0x7faf841587d0 secure :-1 s=READY pgs=1008 cs=0 l=1 rev1=1 crypto rx=0x7faf78000f90 tx=0x7faf780a5a90 comp rx=0 tx=0).stop 2024-09-16T15:48:14.063 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:14.059+0000 7faf8c5fe640 1 -- 172.21.15.123:0/3091332181 shutdown_connections 2024-09-16T15:48:14.063 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:14.059+0000 7faf8c5fe640 1 --2- 172.21.15.123:0/3091332181 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7faf70072fe0 0x7faf700754a0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:14.063 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:14.059+0000 7faf8c5fe640 1 --2- 172.21.15.123:0/3091332181 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7faf84161520 0x7faf84155800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:14.063 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:14.059+0000 7faf8c5fe640 1 --2- 172.21.15.123:0/3091332181 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7faf840d4950 0x7faf841587d0 unknown :-1 s=CLOSED pgs=1008 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:14.063 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:14.059+0000 7faf8c5fe640 1 --2- 172.21.15.123:0/3091332181 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7faf840d23c0 0x7faf84154330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:14.063 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:14.059+0000 7faf8c5fe640 1 -- 172.21.15.123:0/3091332181 >> 172.21.15.123:0/3091332181 conn(0x7faf840d3e30 msgr2=0x7faf8406eb00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:48:14.063 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:14.059+0000 7faf8c5fe640 1 -- 172.21.15.123:0/3091332181 shutdown_connections 2024-09-16T15:48:14.063 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:14.059+0000 7faf8c5fe640 1 -- 172.21.15.123:0/3091332181 wait complete. 2024-09-16T15:48:14.070 INFO:tasks.ceph.mds.b.smithi184.stderr: -1438> 2024-09-16T15:47:55.221+0000 7f2276feb640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-09-16T15:48:14.078 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-09-16T15:48:14.125 INFO:tasks.ceph.mds.b.smithi184.stdout:starting mds.b at 2024-09-16T15:48:14.199 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:14.195+0000 7f9cf5696640 1 -- 172.21.15.123:0/3988666374 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9cd4004d50 msgr2=0x7f9cd40051d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:48:14.199 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:14.195+0000 7f9cf5696640 1 --2- 172.21.15.123:0/3988666374 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9cd4004d50 0x7f9cd40051d0 secure :-1 s=READY pgs=1011 cs=0 l=1 rev1=1 crypto rx=0x7f9cf00cf700 tx=0x7f9ce801d5d0 comp rx=0 tx=0).stop 2024-09-16T15:48:14.199 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:14.195+0000 7f9cef7fe640 1 -- 172.21.15.123:0/3988666374 reap_dead start 2024-09-16T15:48:14.199 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:14.195+0000 7f9cf5696640 1 -- 172.21.15.123:0/3988666374 shutdown_connections 2024-09-16T15:48:14.199 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:14.195+0000 7f9cf5696640 1 -- 172.21.15.123:0/3988666374 >> 172.21.15.123:0/3988666374 conn(0x7f9cf00b1710 msgr2=0x7f9cf00b1b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:48:14.199 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:14.195+0000 7f9cf5696640 1 -- 172.21.15.123:0/3988666374 shutdown_connections 2024-09-16T15:48:14.199 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:14.195+0000 7f9cf5696640 1 -- 172.21.15.123:0/3988666374 wait complete. 2024-09-16T15:48:14.200 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:14.195+0000 7f9cf5696640 1 Processor -- start 2024-09-16T15:48:14.200 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:14.195+0000 7f9cf5696640 1 -- start start 2024-09-16T15:48:14.202 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:14.195+0000 7f9cf5696640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f9cd4004d50 0x7f9cf014dfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:48:14.202 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:14.195+0000 7f9cf5696640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9cf014e520 0x7f9cf0158ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:48:14.202 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:14.195+0000 7f9cf5696640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f9cf00c6c10 0x7f9cf0159430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:48:14.202 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:14.195+0000 7f9cf5696640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f9cf00732b0 con 0x7f9cf014e520 2024-09-16T15:48:14.202 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:14.195+0000 7f9cf5696640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f9cf0073130 con 0x7f9cf00c6c10 2024-09-16T15:48:14.202 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:14.195+0000 7f9cf5696640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f9cf0073430 con 0x7f9cd4004d50 2024-09-16T15:48:14.202 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:14.195+0000 7f9cee7fc640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9cf014e520 0x7f9cf0158ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:48:14.203 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:14.195+0000 7f9cee7fc640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9cf014e520 0x7f9cf0158ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:41480/0 (socket says 172.21.15.123:41480) 2024-09-16T15:48:14.203 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:14.195+0000 7f9cee7fc640 1 -- 172.21.15.123:0/850616240 learned_addr learned my addr 172.21.15.123:0/850616240 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:48:14.203 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:14.195+0000 7f9ceeffd640 1 --2- 172.21.15.123:0/850616240 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f9cd4004d50 0x7f9cf014dfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:48:14.203 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:14.195+0000 7f9cef7fe640 1 --2- 172.21.15.123:0/850616240 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f9cf00c6c10 0x7f9cf0159430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:48:14.203 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:14.195+0000 7f9cee7fc640 1 -- 172.21.15.123:0/850616240 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f9cd4004d50 msgr2=0x7f9cf014dfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:48:14.203 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:14.195+0000 7f9cee7fc640 1 --2- 172.21.15.123:0/850616240 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f9cd4004d50 0x7f9cf014dfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:14.203 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:14.195+0000 7f9cee7fc640 1 -- 172.21.15.123:0/850616240 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f9cf00c6c10 msgr2=0x7f9cf0159430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:48:14.203 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:14.195+0000 7f9cee7fc640 1 --2- 172.21.15.123:0/850616240 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f9cf00c6c10 0x7f9cf0159430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:14.203 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:14.195+0000 7f9cee7fc640 1 -- 172.21.15.123:0/850616240 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9cf0168990 con 0x7f9cf014e520 2024-09-16T15:48:14.203 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:14.195+0000 7f9cee7fc640 1 --2- 172.21.15.123:0/850616240 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9cf014e520 0x7f9cf0158ef0 secure :-1 s=READY pgs=1012 cs=0 l=1 rev1=1 crypto rx=0x7f9cdc000fc0 tx=0x7f9cdc00fa30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:48:14.204 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:14.195+0000 7f9ce77fe640 1 -- 172.21.15.123:0/850616240 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9cdc0118a0 con 0x7f9cf014e520 2024-09-16T15:48:14.204 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:14.195+0000 7f9ce77fe640 1 -- 172.21.15.123:0/850616240 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f9cdc011b30 con 0x7f9cf014e520 2024-09-16T15:48:14.204 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:14.195+0000 7f9cf5696640 1 -- 172.21.15.123:0/850616240 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9cf014fc30 con 0x7f9cf014e520 2024-09-16T15:48:14.204 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:14.195+0000 7f9ce77fe640 1 -- 172.21.15.123:0/850616240 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9cdc012470 con 0x7f9cf014e520 2024-09-16T15:48:14.204 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:14.195+0000 7f9cf5696640 1 -- 172.21.15.123:0/850616240 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9cf01685b0 con 0x7f9cf014e520 2024-09-16T15:48:14.204 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:14.199+0000 7f9cf5696640 1 -- 172.21.15.123:0/850616240 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9cf014fa20 con 0x7f9cf014e520 2024-09-16T15:48:14.205 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:14.199+0000 7f9ce77fe640 1 -- 172.21.15.123:0/850616240 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 16) ==== 97182+0+0 (secure 0 0 0) 0x7f9cdc004020 con 0x7f9cf014e520 2024-09-16T15:48:14.205 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:14.199+0000 7f9ce77fe640 1 --2- 172.21.15.123:0/850616240 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f9cd8072fe0 0x7f9cd80754a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:48:14.205 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:14.199+0000 7f9ce77fe640 1 -- 172.21.15.123:0/850616240 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 5881+0+0 (secure 0 0 0) 0x7f9cdc097890 con 0x7f9cf014e520 2024-09-16T15:48:14.205 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:14.199+0000 7f9ceeffd640 1 --2- 172.21.15.123:0/850616240 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f9cd8072fe0 0x7f9cd80754a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:48:14.205 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:14.199+0000 7f9ceeffd640 1 --2- 172.21.15.123:0/850616240 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f9cd8072fe0 0x7f9cd80754a0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f9cf014f9d0 tx=0x7f9ce00091e0 comp rx=0 tx=0).ready entity=mgr.4150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:48:14.207 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:14.203+0000 7f9ce77fe640 1 -- 172.21.15.123:0/850616240 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f9cdc017030 con 0x7f9cf014e520 2024-09-16T15:48:14.403 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:14.399+0000 7f9cf5696640 1 -- 172.21.15.123:0/850616240 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) -- 0x7f9cf00cbf30 con 0x7f9cf014e520 2024-09-16T15:48:15.068 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:15.063+0000 7f9ce77fe640 1 -- 172.21.15.123:0/850616240 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v61) ==== 104+0+0 (secure 0 0 0) 0x7f9cdc062980 con 0x7f9cf014e520 2024-09-16T15:48:15.072 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:15.067+0000 7f9cf5696640 1 -- 172.21.15.123:0/850616240 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f9cd8072fe0 msgr2=0x7f9cd80754a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:48:15.072 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:15.067+0000 7f9cf5696640 1 --2- 172.21.15.123:0/850616240 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f9cd8072fe0 0x7f9cd80754a0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f9cf014f9d0 tx=0x7f9ce00091e0 comp rx=0 tx=0).stop 2024-09-16T15:48:15.072 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:15.067+0000 7f9cf5696640 1 -- 172.21.15.123:0/850616240 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9cf014e520 msgr2=0x7f9cf0158ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:48:15.072 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:15.067+0000 7f9cf5696640 1 --2- 172.21.15.123:0/850616240 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9cf014e520 0x7f9cf0158ef0 secure :-1 s=READY pgs=1012 cs=0 l=1 rev1=1 crypto rx=0x7f9cdc000fc0 tx=0x7f9cdc00fa30 comp rx=0 tx=0).stop 2024-09-16T15:48:15.072 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:15.067+0000 7f9cf5696640 1 -- 172.21.15.123:0/850616240 shutdown_connections 2024-09-16T15:48:15.072 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:15.067+0000 7f9cf5696640 1 --2- 172.21.15.123:0/850616240 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f9cd8072fe0 0x7f9cd80754a0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:15.072 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:15.067+0000 7f9cf5696640 1 --2- 172.21.15.123:0/850616240 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f9cf00c6c10 0x7f9cf0159430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:15.072 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:15.067+0000 7f9cf5696640 1 --2- 172.21.15.123:0/850616240 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9cf014e520 0x7f9cf0158ef0 unknown :-1 s=CLOSED pgs=1012 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:15.072 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:15.067+0000 7f9cf5696640 1 --2- 172.21.15.123:0/850616240 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f9cd4004d50 0x7f9cf014dfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:15.072 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:15.067+0000 7f9cf5696640 1 -- 172.21.15.123:0/850616240 >> 172.21.15.123:0/850616240 conn(0x7f9cf00b1710 msgr2=0x7f9cf00b1b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:48:15.073 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:15.067+0000 7f9cf5696640 1 -- 172.21.15.123:0/850616240 shutdown_connections 2024-09-16T15:48:15.073 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:15.067+0000 7f9cf5696640 1 -- 172.21.15.123:0/850616240 wait complete. 2024-09-16T15:48:15.091 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-09-16T15:48:15.225 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:15.219+0000 7f2231046640 1 -- 172.21.15.123:0/1311229915 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2214003a10 msgr2=0x7f2214003e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:48:15.226 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:15.219+0000 7f2231046640 1 --2- 172.21.15.123:0/1311229915 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2214003a10 0x7f2214003e30 secure :-1 s=READY pgs=1014 cs=0 l=1 rev1=1 crypto rx=0x7f221c009b60 tx=0x7f221c01d270 comp rx=0 tx=0).stop 2024-09-16T15:48:15.226 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:15.219+0000 7f222b59e640 1 -- 172.21.15.123:0/1311229915 reap_dead start 2024-09-16T15:48:15.226 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:15.219+0000 7f2231046640 1 -- 172.21.15.123:0/1311229915 shutdown_connections 2024-09-16T15:48:15.226 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:15.219+0000 7f2231046640 1 -- 172.21.15.123:0/1311229915 >> 172.21.15.123:0/1311229915 conn(0x7f222c09aa40 msgr2=0x7f222c09ae50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:48:15.226 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:15.219+0000 7f2231046640 1 -- 172.21.15.123:0/1311229915 shutdown_connections 2024-09-16T15:48:15.226 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:15.219+0000 7f2231046640 1 -- 172.21.15.123:0/1311229915 wait complete. 2024-09-16T15:48:15.226 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:15.223+0000 7f2231046640 1 Processor -- start 2024-09-16T15:48:15.227 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:15.223+0000 7f2231046640 1 -- start start 2024-09-16T15:48:15.229 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:15.223+0000 7f2231046640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f2214003a10 0x7f222c0c8520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:48:15.229 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:15.223+0000 7f2231046640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f222c1f5890 0x7f222c0c5fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:48:15.229 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:15.223+0000 7f2231046640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f222c0c64f0 0x7f222c0c69a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:48:15.229 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:15.223+0000 7f2231046640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f222c166be0 con 0x7f222c1f5890 2024-09-16T15:48:15.229 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:15.223+0000 7f2231046640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f222c166a60 con 0x7f222c0c64f0 2024-09-16T15:48:15.229 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:15.223+0000 7f2231046640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f222c166d60 con 0x7f2214003a10 2024-09-16T15:48:15.229 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:15.223+0000 7f222a59c640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f222c1f5890 0x7f222c0c5fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:48:15.229 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:15.223+0000 7f222a59c640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f222c1f5890 0x7f222c0c5fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:41500/0 (socket says 172.21.15.123:41500) 2024-09-16T15:48:15.229 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:15.223+0000 7f222a59c640 1 -- 172.21.15.123:0/4226252427 learned_addr learned my addr 172.21.15.123:0/4226252427 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:48:15.230 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:15.223+0000 7f222b59e640 1 --2- 172.21.15.123:0/4226252427 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f222c0c64f0 0x7f222c0c69a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:48:15.230 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:15.223+0000 7f222ad9d640 1 --2- 172.21.15.123:0/4226252427 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f2214003a10 0x7f222c0c8520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:48:15.230 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:15.223+0000 7f222a59c640 1 -- 172.21.15.123:0/4226252427 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f2214003a10 msgr2=0x7f222c0c8520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:48:15.230 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:15.223+0000 7f222a59c640 1 --2- 172.21.15.123:0/4226252427 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f2214003a10 0x7f222c0c8520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:15.230 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:15.223+0000 7f222a59c640 1 -- 172.21.15.123:0/4226252427 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f222c0c64f0 msgr2=0x7f222c0c69a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:48:15.230 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:15.223+0000 7f222a59c640 1 --2- 172.21.15.123:0/4226252427 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f222c0c64f0 0x7f222c0c69a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:15.230 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:15.223+0000 7f222a59c640 1 -- 172.21.15.123:0/4226252427 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f222c1f5d10 con 0x7f222c1f5890 2024-09-16T15:48:15.230 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:15.223+0000 7f222b59e640 1 --2- 172.21.15.123:0/4226252427 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f222c0c64f0 0x7f222c0c69a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T15:48:15.230 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:15.223+0000 7f222a59c640 1 --2- 172.21.15.123:0/4226252427 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f222c1f5890 0x7f222c0c5fb0 secure :-1 s=READY pgs=1015 cs=0 l=1 rev1=1 crypto rx=0x7f222000d7f0 tx=0x7f222000dcb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:48:15.230 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:15.223+0000 7f22137fe640 1 -- 172.21.15.123:0/4226252427 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2220019070 con 0x7f222c1f5890 2024-09-16T15:48:15.230 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:15.223+0000 7f22137fe640 1 -- 172.21.15.123:0/4226252427 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f22200052e0 con 0x7f222c1f5890 2024-09-16T15:48:15.230 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:15.223+0000 7f2231046640 1 -- 172.21.15.123:0/4226252427 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f222c1634f0 con 0x7f222c1f5890 2024-09-16T15:48:15.231 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:15.223+0000 7f22137fe640 1 -- 172.21.15.123:0/4226252427 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f222000b8c0 con 0x7f222c1f5890 2024-09-16T15:48:15.231 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:15.223+0000 7f2231046640 1 -- 172.21.15.123:0/4226252427 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f222c163ab0 con 0x7f222c1f5890 2024-09-16T15:48:15.231 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:15.223+0000 7f2231046640 1 -- 172.21.15.123:0/4226252427 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f222c15c680 con 0x7f222c1f5890 2024-09-16T15:48:15.231 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:15.223+0000 7f22137fe640 1 -- 172.21.15.123:0/4226252427 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 16) ==== 97182+0+0 (secure 0 0 0) 0x7f222001c070 con 0x7f222c1f5890 2024-09-16T15:48:15.232 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:15.227+0000 7f22137fe640 1 --2- 172.21.15.123:0/4226252427 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f2218073010 0x7f22180754d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:48:15.232 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:15.227+0000 7f222ad9d640 1 --2- 172.21.15.123:0/4226252427 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f2218073010 0x7f22180754d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:48:15.232 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:15.227+0000 7f22137fe640 1 -- 172.21.15.123:0/4226252427 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 5881+0+0 (secure 0 0 0) 0x7f22200938e0 con 0x7f222c1f5890 2024-09-16T15:48:15.232 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:15.227+0000 7f222ad9d640 1 --2- 172.21.15.123:0/4226252427 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f2218073010 0x7f22180754d0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f221c0ae000 tx=0x7f221c01a010 comp rx=0 tx=0).ready entity=mgr.4150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:48:15.234 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:15.231+0000 7f22137fe640 1 -- 172.21.15.123:0/4226252427 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f222005e920 con 0x7f222c1f5890 2024-09-16T15:48:15.411 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:15.407+0000 7f2231046640 1 -- 172.21.15.123:0/4226252427 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) -- 0x7f222c074a00 con 0x7f222c1f5890 2024-09-16T15:48:16.338 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:16.331+0000 7f22137fe640 1 -- 172.21.15.123:0/4226252427 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v94) ==== 181+0+0 (secure 0 0 0) 0x7f222005e220 con 0x7f222c1f5890 2024-09-16T15:48:16.338 INFO:teuthology.orchestra.run.smithi123.stderr:pool 'cephfs_metadata' removed 2024-09-16T15:48:16.342 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:16.335+0000 7f2231046640 1 -- 172.21.15.123:0/4226252427 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f2218073010 msgr2=0x7f22180754d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:48:16.342 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:16.335+0000 7f2231046640 1 --2- 172.21.15.123:0/4226252427 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f2218073010 0x7f22180754d0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f221c0ae000 tx=0x7f221c01a010 comp rx=0 tx=0).stop 2024-09-16T15:48:16.342 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:16.335+0000 7f2231046640 1 -- 172.21.15.123:0/4226252427 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f222c1f5890 msgr2=0x7f222c0c5fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:48:16.342 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:16.335+0000 7f2231046640 1 --2- 172.21.15.123:0/4226252427 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f222c1f5890 0x7f222c0c5fb0 secure :-1 s=READY pgs=1015 cs=0 l=1 rev1=1 crypto rx=0x7f222000d7f0 tx=0x7f222000dcb0 comp rx=0 tx=0).stop 2024-09-16T15:48:16.343 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:16.335+0000 7f2231046640 1 -- 172.21.15.123:0/4226252427 shutdown_connections 2024-09-16T15:48:16.343 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:16.335+0000 7f2231046640 1 --2- 172.21.15.123:0/4226252427 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f2218073010 0x7f22180754d0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:16.343 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:16.335+0000 7f2231046640 1 --2- 172.21.15.123:0/4226252427 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f222c0c64f0 0x7f222c0c69a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:16.343 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:16.335+0000 7f2231046640 1 --2- 172.21.15.123:0/4226252427 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f222c1f5890 0x7f222c0c5fb0 secure :-1 s=CLOSED pgs=1015 cs=0 l=1 rev1=1 crypto rx=0x7f222000d7f0 tx=0x7f222000dcb0 comp rx=0 tx=0).stop 2024-09-16T15:48:16.343 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:16.335+0000 7f2231046640 1 --2- 172.21.15.123:0/4226252427 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f2214003a10 0x7f222c0c8520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:16.343 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:16.335+0000 7f2231046640 1 -- 172.21.15.123:0/4226252427 >> 172.21.15.123:0/4226252427 conn(0x7f222c09aa40 msgr2=0x7f222c0707c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:48:16.343 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:16.339+0000 7f2231046640 1 -- 172.21.15.123:0/4226252427 shutdown_connections 2024-09-16T15:48:16.343 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:16.339+0000 7f2231046640 1 -- 172.21.15.123:0/4226252427 wait complete. 2024-09-16T15:48:16.363 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-09-16T15:48:16.492 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:16.487+0000 7f16bb00d640 1 -- 172.21.15.123:0/4146773901 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f16b4159680 msgr2=0x7f16b40704e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:48:16.492 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:16.487+0000 7f16bb00d640 1 --2- 172.21.15.123:0/4146773901 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f16b4159680 0x7f16b40704e0 secure :-1 s=READY pgs=1017 cs=0 l=1 rev1=1 crypto rx=0x7f16a4009bc0 tx=0x7f16a401cba0 comp rx=0 tx=0).stop 2024-09-16T15:48:16.492 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:16.487+0000 7f16bb00d640 1 -- 172.21.15.123:0/4146773901 shutdown_connections 2024-09-16T15:48:16.492 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:16.487+0000 7f16bb00d640 1 --2- 172.21.15.123:0/4146773901 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f16b4159680 0x7f16b40704e0 unknown :-1 s=CLOSED pgs=1017 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:16.493 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:16.487+0000 7f16bb00d640 1 -- 172.21.15.123:0/4146773901 >> 172.21.15.123:0/4146773901 conn(0x7f16b40b1630 msgr2=0x7f16b40b1a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:48:16.493 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:16.487+0000 7f16bb00d640 1 -- 172.21.15.123:0/4146773901 shutdown_connections 2024-09-16T15:48:16.493 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:16.487+0000 7f16bb00d640 1 -- 172.21.15.123:0/4146773901 wait complete. 2024-09-16T15:48:16.493 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:16.487+0000 7f16bb00d640 1 Processor -- start 2024-09-16T15:48:16.493 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:16.487+0000 7f16bb00d640 1 -- start start 2024-09-16T15:48:16.495 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:16.491+0000 7f16bb00d640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f16b4074440 0x7f16b414f8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:48:16.495 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:16.491+0000 7f16bb00d640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f16b4159680 0x7f16b41ffd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:48:16.495 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:16.491+0000 7f16bb00d640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f16b415fd80 0x7f16b4148390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:48:16.495 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:16.491+0000 7f16bb00d640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f16b4169f10 con 0x7f16b4074440 2024-09-16T15:48:16.495 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:16.491+0000 7f16bb00d640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f16b4169d90 con 0x7f16b415fd80 2024-09-16T15:48:16.495 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:16.491+0000 7f16bb00d640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f16b416a090 con 0x7f16b4159680 2024-09-16T15:48:16.495 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:16.491+0000 7f16b8da9640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f16b4074440 0x7f16b414f8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:48:16.496 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:16.491+0000 7f16b8da9640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f16b4074440 0x7f16b414f8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:41532/0 (socket says 172.21.15.123:41532) 2024-09-16T15:48:16.496 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:16.491+0000 7f16b8da9640 1 -- 172.21.15.123:0/2579084883 learned_addr learned my addr 172.21.15.123:0/2579084883 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:48:16.496 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:16.491+0000 7f16b8da9640 1 -- 172.21.15.123:0/2579084883 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f16b4159680 msgr2=0x7f16b41ffd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:48:16.496 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:16.491+0000 7f16b8da9640 1 --2- 172.21.15.123:0/2579084883 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f16b4159680 0x7f16b41ffd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:16.496 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:16.491+0000 7f16b8da9640 1 -- 172.21.15.123:0/2579084883 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f16b415fd80 msgr2=0x7f16b4148390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:48:16.496 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:16.491+0000 7f16b95aa640 1 --2- 172.21.15.123:0/2579084883 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f16b415fd80 0x7f16b4148390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:48:16.496 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:16.491+0000 7f16b8da9640 1 --2- 172.21.15.123:0/2579084883 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f16b415fd80 0x7f16b4148390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:16.496 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:16.491+0000 7f16b8da9640 1 -- 172.21.15.123:0/2579084883 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f16b414ff40 con 0x7f16b4074440 2024-09-16T15:48:16.496 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:16.491+0000 7f16b8da9640 1 --2- 172.21.15.123:0/2579084883 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f16b4074440 0x7f16b414f8b0 secure :-1 s=READY pgs=1018 cs=0 l=1 rev1=1 crypto rx=0x7f16a401b250 tx=0x7f16a40a59a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:48:16.496 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:16.491+0000 7f16b17fa640 1 -- 172.21.15.123:0/2579084883 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16a40bb070 con 0x7f16b4074440 2024-09-16T15:48:16.496 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:16.491+0000 7f16b17fa640 1 -- 172.21.15.123:0/2579084883 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f16a4004e00 con 0x7f16b4074440 2024-09-16T15:48:16.496 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:16.491+0000 7f16bb00d640 1 -- 172.21.15.123:0/2579084883 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f16b4150220 con 0x7f16b4074440 2024-09-16T15:48:16.496 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:16.491+0000 7f16b17fa640 1 -- 172.21.15.123:0/2579084883 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16a40b2040 con 0x7f16b4074440 2024-09-16T15:48:16.497 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:16.491+0000 7f16bb00d640 1 -- 172.21.15.123:0/2579084883 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f16b41507e0 con 0x7f16b4074440 2024-09-16T15:48:16.497 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:16.491+0000 7f16bb00d640 1 -- 172.21.15.123:0/2579084883 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f16b40704e0 con 0x7f16b4074440 2024-09-16T15:48:16.497 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:16.491+0000 7f16b17fa640 1 -- 172.21.15.123:0/2579084883 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 16) ==== 97182+0+0 (secure 0 0 0) 0x7f16a40be070 con 0x7f16b4074440 2024-09-16T15:48:16.497 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:16.491+0000 7f16b17fa640 1 --2- 172.21.15.123:0/2579084883 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f16a0072fe0 0x7f16a00754a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:48:16.498 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:16.495+0000 7f16b17fa640 1 -- 172.21.15.123:0/2579084883 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 5443+0+0 (secure 0 0 0) 0x7f16a412df60 con 0x7f16b4074440 2024-09-16T15:48:16.498 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:16.495+0000 7f16b3fff640 1 --2- 172.21.15.123:0/2579084883 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f16a0072fe0 0x7f16a00754a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:48:16.498 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:16.495+0000 7f16b3fff640 1 --2- 172.21.15.123:0/2579084883 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f16a0072fe0 0x7f16a00754a0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f16a80046c0 tx=0x7f16a8007910 comp rx=0 tx=0).ready entity=mgr.4150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:48:16.500 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:16.495+0000 7f16b17fa640 1 -- 172.21.15.123:0/2579084883 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f16a4016790 con 0x7f16b4074440 2024-09-16T15:48:16.666 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:16.663+0000 7f16bb00d640 1 -- 172.21.15.123:0/2579084883 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) -- 0x7f16b4074220 con 0x7f16b4074440 2024-09-16T15:48:17.340 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:17.335+0000 7f16b17fa640 1 -- 172.21.15.123:0/2579084883 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v95) ==== 169+0+0 (secure 0 0 0) 0x7f16a40f9200 con 0x7f16b4074440 2024-09-16T15:48:17.340 INFO:teuthology.orchestra.run.smithi123.stderr:pool 'cephfs_data' removed 2024-09-16T15:48:17.344 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:17.339+0000 7f16bb00d640 1 -- 172.21.15.123:0/2579084883 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f16a0072fe0 msgr2=0x7f16a00754a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:48:17.344 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:17.339+0000 7f16bb00d640 1 --2- 172.21.15.123:0/2579084883 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f16a0072fe0 0x7f16a00754a0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f16a80046c0 tx=0x7f16a8007910 comp rx=0 tx=0).stop 2024-09-16T15:48:17.344 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:17.339+0000 7f16bb00d640 1 -- 172.21.15.123:0/2579084883 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f16b4074440 msgr2=0x7f16b414f8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:48:17.344 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:17.339+0000 7f16bb00d640 1 --2- 172.21.15.123:0/2579084883 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f16b4074440 0x7f16b414f8b0 secure :-1 s=READY pgs=1018 cs=0 l=1 rev1=1 crypto rx=0x7f16a401b250 tx=0x7f16a40a59a0 comp rx=0 tx=0).stop 2024-09-16T15:48:17.344 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:17.339+0000 7f16bb00d640 1 -- 172.21.15.123:0/2579084883 shutdown_connections 2024-09-16T15:48:17.344 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:17.339+0000 7f16bb00d640 1 --2- 172.21.15.123:0/2579084883 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f16a0072fe0 0x7f16a00754a0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:17.344 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:17.339+0000 7f16bb00d640 1 --2- 172.21.15.123:0/2579084883 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f16b415fd80 0x7f16b4148390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:17.344 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:17.339+0000 7f16bb00d640 1 --2- 172.21.15.123:0/2579084883 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f16b4159680 0x7f16b41ffd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:17.344 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:17.339+0000 7f16bb00d640 1 --2- 172.21.15.123:0/2579084883 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f16b4074440 0x7f16b414f8b0 unknown :-1 s=CLOSED pgs=1018 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:17.344 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:17.339+0000 7f16bb00d640 1 -- 172.21.15.123:0/2579084883 >> 172.21.15.123:0/2579084883 conn(0x7f16b40b1630 msgr2=0x7f16b415d1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:48:17.344 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:17.339+0000 7f16bb00d640 1 -- 172.21.15.123:0/2579084883 shutdown_connections 2024-09-16T15:48:17.344 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:17.339+0000 7f16bb00d640 1 -- 172.21.15.123:0/2579084883 wait complete. 2024-09-16T15:48:17.367 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config reset 27 2024-09-16T15:48:17.497 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:17.491+0000 7f8b68aa9640 1 -- 172.21.15.123:0/3058412075 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8b6415f1b0 msgr2=0x7f8b640ce220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:48:17.497 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:17.491+0000 7f8b68aa9640 1 --2- 172.21.15.123:0/3058412075 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8b6415f1b0 0x7f8b640ce220 secure :-1 s=READY pgs=1020 cs=0 l=1 rev1=1 crypto rx=0x7f8b54009840 tx=0x7f8b5401c700 comp rx=0 tx=0).stop 2024-09-16T15:48:17.497 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:17.491+0000 7f8b68aa9640 1 -- 172.21.15.123:0/3058412075 shutdown_connections 2024-09-16T15:48:17.497 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:17.491+0000 7f8b68aa9640 1 --2- 172.21.15.123:0/3058412075 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8b6415f1b0 0x7f8b640ce220 unknown :-1 s=CLOSED pgs=1020 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:17.498 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:17.491+0000 7f8b68aa9640 1 -- 172.21.15.123:0/3058412075 >> 172.21.15.123:0/3058412075 conn(0x7f8b640732a0 msgr2=0x7f8b64074150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:48:17.498 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:17.491+0000 7f8b68aa9640 1 -- 172.21.15.123:0/3058412075 shutdown_connections 2024-09-16T15:48:17.498 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:17.491+0000 7f8b68aa9640 1 -- 172.21.15.123:0/3058412075 wait complete. 2024-09-16T15:48:17.499 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:17.495+0000 7f8b68aa9640 1 Processor -- start 2024-09-16T15:48:17.499 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:17.495+0000 7f8b68aa9640 1 -- start start 2024-09-16T15:48:17.501 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:17.495+0000 7f8b68aa9640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f8b640adf40 0x7f8b640c6cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:48:17.501 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:17.495+0000 7f8b68aa9640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8b6415f1b0 0x7f8b6406f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:48:17.501 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:17.495+0000 7f8b68aa9640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f8b6415f590 0x7f8b6406f620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:48:17.501 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:17.495+0000 7f8b68aa9640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f8b64169080 con 0x7f8b6415f1b0 2024-09-16T15:48:17.502 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:17.495+0000 7f8b68aa9640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f8b64168f00 con 0x7f8b640adf40 2024-09-16T15:48:17.502 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:17.495+0000 7f8b68aa9640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f8b64169200 con 0x7f8b6415f590 2024-09-16T15:48:17.502 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:17.495+0000 7f8b61d9b640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8b6415f1b0 0x7f8b6406f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:48:17.502 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:17.495+0000 7f8b61d9b640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8b6415f1b0 0x7f8b6406f0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:41560/0 (socket says 172.21.15.123:41560) 2024-09-16T15:48:17.502 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:17.495+0000 7f8b61d9b640 1 -- 172.21.15.123:0/2723634752 learned_addr learned my addr 172.21.15.123:0/2723634752 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:48:17.502 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:17.495+0000 7f8b6259c640 1 --2- 172.21.15.123:0/2723634752 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f8b640adf40 0x7f8b640c6cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:48:17.502 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:17.495+0000 7f8b62d9d640 1 --2- 172.21.15.123:0/2723634752 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f8b6415f590 0x7f8b6406f620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:48:17.502 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:17.495+0000 7f8b61d9b640 1 -- 172.21.15.123:0/2723634752 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f8b6415f590 msgr2=0x7f8b6406f620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:48:17.502 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:17.495+0000 7f8b61d9b640 1 --2- 172.21.15.123:0/2723634752 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f8b6415f590 0x7f8b6406f620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:17.502 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:17.495+0000 7f8b61d9b640 1 -- 172.21.15.123:0/2723634752 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f8b640adf40 msgr2=0x7f8b640c6cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:48:17.502 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:17.495+0000 7f8b61d9b640 1 --2- 172.21.15.123:0/2723634752 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f8b640adf40 0x7f8b640c6cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:17.502 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:17.495+0000 7f8b61d9b640 1 -- 172.21.15.123:0/2723634752 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8b641594a0 con 0x7f8b6415f1b0 2024-09-16T15:48:17.502 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:17.495+0000 7f8b61d9b640 1 --2- 172.21.15.123:0/2723634752 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8b6415f1b0 0x7f8b6406f0e0 secure :-1 s=READY pgs=1021 cs=0 l=1 rev1=1 crypto rx=0x7f8b5800c430 tx=0x7f8b5800c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:48:17.503 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:17.495+0000 7f8b52ffd640 1 -- 172.21.15.123:0/2723634752 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b58011790 con 0x7f8b6415f1b0 2024-09-16T15:48:17.503 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:17.495+0000 7f8b52ffd640 1 -- 172.21.15.123:0/2723634752 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f8b58017070 con 0x7f8b6415f1b0 2024-09-16T15:48:17.503 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:17.495+0000 7f8b52ffd640 1 -- 172.21.15.123:0/2723634752 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b58011ce0 con 0x7f8b6415f1b0 2024-09-16T15:48:17.503 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:17.495+0000 7f8b68aa9640 1 -- 172.21.15.123:0/2723634752 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8b64159780 con 0x7f8b6415f1b0 2024-09-16T15:48:17.503 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:17.495+0000 7f8b68aa9640 1 -- 172.21.15.123:0/2723634752 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8b64159d40 con 0x7f8b6415f1b0 2024-09-16T15:48:17.503 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:17.499+0000 7f8b68aa9640 1 -- 172.21.15.123:0/2723634752 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8b640ae3b0 con 0x7f8b6415f1b0 2024-09-16T15:48:17.504 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:17.499+0000 7f8b52ffd640 1 -- 172.21.15.123:0/2723634752 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 16) ==== 97182+0+0 (secure 0 0 0) 0x7f8b5801c460 con 0x7f8b6415f1b0 2024-09-16T15:48:17.504 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:17.499+0000 7f8b52ffd640 1 --2- 172.21.15.123:0/2723634752 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f8b44072fe0 0x7f8b440754a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:48:17.504 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:17.499+0000 7f8b52ffd640 1 -- 172.21.15.123:0/2723634752 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 5029+0+0 (secure 0 0 0) 0x7f8b58093b30 con 0x7f8b6415f1b0 2024-09-16T15:48:17.504 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:17.499+0000 7f8b6259c640 1 --2- 172.21.15.123:0/2723634752 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f8b44072fe0 0x7f8b440754a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:48:17.504 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:17.499+0000 7f8b6259c640 1 --2- 172.21.15.123:0/2723634752 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f8b44072fe0 0x7f8b440754a0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f8b540af000 tx=0x7f8b5401af40 comp rx=0 tx=0).ready entity=mgr.4150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:48:17.507 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:17.503+0000 7f8b52ffd640 1 -- 172.21.15.123:0/2723634752 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f8b5805eec0 con 0x7f8b6415f1b0 2024-09-16T15:48:17.679 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:17.675+0000 7f8b68aa9640 1 -- 172.21.15.123:0/2723634752 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "config reset", "num": 27} v 0) -- 0x7f8b640b34e0 con 0x7f8b6415f1b0 2024-09-16T15:48:17.685 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:17.679+0000 7f8b52ffd640 1 -- 172.21.15.123:0/2723634752 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "config reset", "num": 27}]=0 v37) ==== 71+0+0 (secure 0 0 0) 0x7f8b5805e7c0 con 0x7f8b6415f1b0 2024-09-16T15:48:17.687 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:17.683+0000 7f8b68aa9640 1 -- 172.21.15.123:0/2723634752 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f8b44072fe0 msgr2=0x7f8b440754a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:48:17.687 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:17.683+0000 7f8b68aa9640 1 --2- 172.21.15.123:0/2723634752 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f8b44072fe0 0x7f8b440754a0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f8b540af000 tx=0x7f8b5401af40 comp rx=0 tx=0).stop 2024-09-16T15:48:17.687 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:17.683+0000 7f8b68aa9640 1 -- 172.21.15.123:0/2723634752 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8b6415f1b0 msgr2=0x7f8b6406f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:48:17.688 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:17.683+0000 7f8b68aa9640 1 --2- 172.21.15.123:0/2723634752 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8b6415f1b0 0x7f8b6406f0e0 secure :-1 s=READY pgs=1021 cs=0 l=1 rev1=1 crypto rx=0x7f8b5800c430 tx=0x7f8b5800c8f0 comp rx=0 tx=0).stop 2024-09-16T15:48:17.688 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:17.683+0000 7f8b68aa9640 1 -- 172.21.15.123:0/2723634752 shutdown_connections 2024-09-16T15:48:17.688 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:17.683+0000 7f8b68aa9640 1 --2- 172.21.15.123:0/2723634752 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f8b44072fe0 0x7f8b440754a0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:17.688 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:17.683+0000 7f8b68aa9640 1 --2- 172.21.15.123:0/2723634752 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f8b6415f590 0x7f8b6406f620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:17.688 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:17.683+0000 7f8b68aa9640 1 --2- 172.21.15.123:0/2723634752 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8b6415f1b0 0x7f8b6406f0e0 unknown :-1 s=CLOSED pgs=1021 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:17.688 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:17.683+0000 7f8b68aa9640 1 --2- 172.21.15.123:0/2723634752 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f8b640adf40 0x7f8b640c6cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:17.688 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:17.683+0000 7f8b68aa9640 1 -- 172.21.15.123:0/2723634752 >> 172.21.15.123:0/2723634752 conn(0x7f8b640732a0 msgr2=0x7f8b640b4150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:48:17.688 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:17.683+0000 7f8b68aa9640 1 -- 172.21.15.123:0/2723634752 shutdown_connections 2024-09-16T15:48:17.688 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:17.683+0000 7f8b68aa9640 1 -- 172.21.15.123:0/2723634752 wait complete. 2024-09-16T15:48:17.704 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_volumes.TestMisc.test_names_can_only_be_goodchars' 2024-09-16T15:48:17.823 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:17.819+0000 7f5285330640 1 -- 172.21.15.123:0/43844015 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5280074470 msgr2=0x7f52800cd6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:48:17.823 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:17.819+0000 7f5285330640 1 --2- 172.21.15.123:0/43844015 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5280074470 0x7f52800cd6e0 secure :-1 s=READY pgs=1023 cs=0 l=1 rev1=1 crypto rx=0x7f5280070730 tx=0x7f527401c6e0 comp rx=0 tx=0).stop 2024-09-16T15:48:17.823 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:17.819+0000 7f5285330640 1 -- 172.21.15.123:0/43844015 shutdown_connections 2024-09-16T15:48:17.823 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:17.819+0000 7f5285330640 1 --2- 172.21.15.123:0/43844015 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5280074470 0x7f52800cd6e0 unknown :-1 s=CLOSED pgs=1023 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:17.823 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:17.819+0000 7f5285330640 1 --2- 172.21.15.123:0/43844015 >> v2:172.21.15.184:3300/0 conn(0x7f528015e160 0x7f5280073f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:17.823 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:17.819+0000 7f5285330640 1 -- 172.21.15.123:0/43844015 >> 172.21.15.123:0/43844015 conn(0x7f52800cfb10 msgr2=0x7f52800cff20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:48:17.824 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:17.819+0000 7f5285330640 1 -- 172.21.15.123:0/43844015 shutdown_connections 2024-09-16T15:48:17.824 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:17.819+0000 7f5285330640 1 -- 172.21.15.123:0/43844015 wait complete. 2024-09-16T15:48:17.824 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:17.819+0000 7f5285330640 1 Processor -- start 2024-09-16T15:48:17.824 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:17.819+0000 7f5285330640 1 -- start start 2024-09-16T15:48:17.826 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:17.819+0000 7f5285330640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5280074470 0x7f52801f9080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:48:17.826 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:17.819+0000 7f5285330640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f52800cdc20 0x7f528015cae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:48:17.826 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:17.819+0000 7f5285330640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f528015e160 0x7f528014fc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:48:17.826 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:17.819+0000 7f5285330640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f5280167d40 con 0x7f5280074470 2024-09-16T15:48:17.826 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:17.819+0000 7f5285330640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f5280167bc0 con 0x7f528015e160 2024-09-16T15:48:17.826 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:17.819+0000 7f5285330640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f5280167ec0 con 0x7f52800cdc20 2024-09-16T15:48:17.827 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:17.819+0000 7f527effd640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5280074470 0x7f52801f9080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:48:17.827 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:17.819+0000 7f527effd640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5280074470 0x7f52801f9080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:41582/0 (socket says 172.21.15.123:41582) 2024-09-16T15:48:17.827 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:17.819+0000 7f527effd640 1 -- 172.21.15.123:0/1174173069 learned_addr learned my addr 172.21.15.123:0/1174173069 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:48:17.827 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:17.819+0000 7f527f7fe640 1 --2- 172.21.15.123:0/1174173069 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f528015e160 0x7f528014fc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:48:17.827 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:17.819+0000 7f527e7fc640 1 --2- 172.21.15.123:0/1174173069 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f52800cdc20 0x7f528015cae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:48:17.827 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:17.819+0000 7f527effd640 1 -- 172.21.15.123:0/1174173069 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f52800cdc20 msgr2=0x7f528015cae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:48:17.827 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:17.819+0000 7f527effd640 1 --2- 172.21.15.123:0/1174173069 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f52800cdc20 0x7f528015cae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:17.827 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:17.819+0000 7f527effd640 1 -- 172.21.15.123:0/1174173069 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f528015e160 msgr2=0x7f528014fc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:48:17.827 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:17.819+0000 7f527effd640 1 --2- 172.21.15.123:0/1174173069 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f528015e160 0x7f528014fc10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:17.827 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:17.819+0000 7f527effd640 1 -- 172.21.15.123:0/1174173069 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f52801f9f00 con 0x7f5280074470 2024-09-16T15:48:17.827 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:17.819+0000 7f527effd640 1 --2- 172.21.15.123:0/1174173069 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5280074470 0x7f52801f9080 secure :-1 s=READY pgs=1024 cs=0 l=1 rev1=1 crypto rx=0x7f527000e7d0 tx=0x7f527000ec90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:48:17.827 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:17.823+0000 7f52677fe640 1 -- 172.21.15.123:0/1174173069 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f527000ccc0 con 0x7f5280074470 2024-09-16T15:48:17.828 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:17.823+0000 7f52677fe640 1 -- 172.21.15.123:0/1174173069 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f5270010430 con 0x7f5280074470 2024-09-16T15:48:17.828 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:17.823+0000 7f5285330640 1 -- 172.21.15.123:0/1174173069 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f52801f9940 con 0x7f5280074470 2024-09-16T15:48:17.828 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:17.823+0000 7f52677fe640 1 -- 172.21.15.123:0/1174173069 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5270010860 con 0x7f5280074470 2024-09-16T15:48:17.828 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:17.823+0000 7f5285330640 1 -- 172.21.15.123:0/1174173069 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f52801f5ba0 con 0x7f5280074470 2024-09-16T15:48:17.829 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:17.823+0000 7f5285330640 1 -- 172.21.15.123:0/1174173069 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f52800cd6e0 con 0x7f5280074470 2024-09-16T15:48:17.829 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:17.823+0000 7f52677fe640 1 -- 172.21.15.123:0/1174173069 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 16) ==== 97182+0+0 (secure 0 0 0) 0x7f5270010a00 con 0x7f5280074470 2024-09-16T15:48:17.829 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:17.823+0000 7f52677fe640 1 --2- 172.21.15.123:0/1174173069 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f526c072fe0 0x7f526c0754a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:48:17.829 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:17.823+0000 7f52677fe640 1 -- 172.21.15.123:0/1174173069 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 5029+0+0 (secure 0 0 0) 0x7f5270094810 con 0x7f5280074470 2024-09-16T15:48:17.829 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:17.823+0000 7f527e7fc640 1 --2- 172.21.15.123:0/1174173069 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f526c072fe0 0x7f526c0754a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:48:17.829 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:17.823+0000 7f527e7fc640 1 --2- 172.21.15.123:0/1174173069 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f526c072fe0 0x7f526c0754a0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f52740af000 tx=0x7f527401af20 comp rx=0 tx=0).ready entity=mgr.4150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:48:17.832 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:17.827+0000 7f52677fe640 1 -- 172.21.15.123:0/1174173069 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f527005fae0 con 0x7f5280074470 2024-09-16T15:48:18.051 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:18.047+0000 7f5285330640 1 -- 172.21.15.123:0/1174173069 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_volumes.TestMisc.test_names_can_only_be_goodchars"]} v 0) -- 0x7f52800cd120 con 0x7f5280074470 2024-09-16T15:48:18.361 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:18.359+0000 7f52677fe640 1 -- 172.21.15.123:0/1174173069 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_volumes.TestMisc.test_names_can_only_be_goodchars"]}]=0 v282) ==== 146+0+0 (secure 0 0 0) 0x7f5270068020 con 0x7f5280074470 2024-09-16T15:48:18.364 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:18.359+0000 7f5285330640 1 -- 172.21.15.123:0/1174173069 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f526c072fe0 msgr2=0x7f526c0754a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:48:18.365 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:18.359+0000 7f5285330640 1 --2- 172.21.15.123:0/1174173069 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f526c072fe0 0x7f526c0754a0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f52740af000 tx=0x7f527401af20 comp rx=0 tx=0).stop 2024-09-16T15:48:18.365 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:18.359+0000 7f5285330640 1 -- 172.21.15.123:0/1174173069 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5280074470 msgr2=0x7f52801f9080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:48:18.365 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:18.359+0000 7f5285330640 1 --2- 172.21.15.123:0/1174173069 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5280074470 0x7f52801f9080 secure :-1 s=READY pgs=1024 cs=0 l=1 rev1=1 crypto rx=0x7f527000e7d0 tx=0x7f527000ec90 comp rx=0 tx=0).stop 2024-09-16T15:48:18.365 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:18.359+0000 7f5285330640 1 -- 172.21.15.123:0/1174173069 shutdown_connections 2024-09-16T15:48:18.365 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:18.359+0000 7f5285330640 1 --2- 172.21.15.123:0/1174173069 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f526c072fe0 0x7f526c0754a0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:18.365 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:18.359+0000 7f5285330640 1 --2- 172.21.15.123:0/1174173069 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f528015e160 0x7f528014fc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:18.365 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:18.359+0000 7f5285330640 1 --2- 172.21.15.123:0/1174173069 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f52800cdc20 0x7f528015cae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:18.365 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:18.359+0000 7f5285330640 1 --2- 172.21.15.123:0/1174173069 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5280074470 0x7f52801f9080 unknown :-1 s=CLOSED pgs=1024 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:18.365 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:18.359+0000 7f5285330640 1 -- 172.21.15.123:0/1174173069 >> 172.21.15.123:0/1174173069 conn(0x7f52800cfb10 msgr2=0x7f5280154580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:48:18.365 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:18.359+0000 7f5285330640 1 -- 172.21.15.123:0/1174173069 shutdown_connections 2024-09-16T15:48:18.365 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:18.359+0000 7f5285330640 1 -- 172.21.15.123:0/1174173069 wait complete. 2024-09-16T15:48:18.383 INFO:tasks.cephfs_test_runner:Test the creating vols, subvols subvolgroups fails when their names uses ... ok 2024-09-16T15:48:18.383 INFO:tasks.cephfs_test_runner:Starting test: test_subvolume_no_upgrade_v1_sanity (tasks.cephfs.test_volumes.TestMisc) poor man's upgrade test -- theme continues... 2024-09-16T15:48:18.383 INFO:tasks.cephfs_test_runner:test_subvolume_no_upgrade_v1_sanity (tasks.cephfs.test_volumes.TestMisc) 2024-09-16T15:48:18.384 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-16T15:48:18.510 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:18.507+0000 7ff1435a8640 1 -- 172.21.15.123:0/2409500788 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff13c0ce560 msgr2=0x7ff13c0d1ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:48:18.510 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:18.507+0000 7ff1435a8640 1 --2- 172.21.15.123:0/2409500788 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff13c0ce560 0x7ff13c0d1ec0 secure :-1 s=READY pgs=1026 cs=0 l=1 rev1=1 crypto rx=0x7ff134009bc0 tx=0x7ff13401cb70 comp rx=0 tx=0).stop 2024-09-16T15:48:18.510 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:18.507+0000 7ff1435a8640 1 -- 172.21.15.123:0/2409500788 shutdown_connections 2024-09-16T15:48:18.510 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:18.507+0000 7ff1435a8640 1 --2- 172.21.15.123:0/2409500788 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff13c0ce560 0x7ff13c0d1ec0 unknown :-1 s=CLOSED pgs=1026 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:18.511 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:18.507+0000 7ff1435a8640 1 --2- 172.21.15.123:0/2409500788 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7ff13c15ddc0 0x7ff13c0d1980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:18.511 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:18.507+0000 7ff1435a8640 1 -- 172.21.15.123:0/2409500788 >> 172.21.15.123:0/2409500788 conn(0x7ff13c074bd0 msgr2=0x7ff13c09aa40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:48:18.511 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:18.507+0000 7ff1435a8640 1 -- 172.21.15.123:0/2409500788 shutdown_connections 2024-09-16T15:48:18.511 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:18.507+0000 7ff1435a8640 1 -- 172.21.15.123:0/2409500788 wait complete. 2024-09-16T15:48:18.511 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:18.507+0000 7ff1435a8640 1 Processor -- start 2024-09-16T15:48:18.511 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:18.507+0000 7ff1435a8640 1 -- start start 2024-09-16T15:48:18.513 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:18.507+0000 7ff1435a8640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff13c0b32f0 0x7ff13c14fab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:48:18.514 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:18.507+0000 7ff1435a8640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff13c0ce560 0x7ff13c160ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:48:18.514 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:18.507+0000 7ff1435a8640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7ff13c15ddc0 0x7ff13c161010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:48:18.514 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:18.507+0000 7ff1435a8640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7ff13c16ace0 con 0x7ff13c0b32f0 2024-09-16T15:48:18.514 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:18.507+0000 7ff1435a8640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7ff13c16ab60 con 0x7ff13c0ce560 2024-09-16T15:48:18.514 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:18.507+0000 7ff1435a8640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7ff13c16ae60 con 0x7ff13c15ddc0 2024-09-16T15:48:18.514 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:18.507+0000 7ff141344640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff13c0b32f0 0x7ff13c14fab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:48:18.514 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:18.507+0000 7ff141344640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff13c0b32f0 0x7ff13c14fab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:41600/0 (socket says 172.21.15.123:41600) 2024-09-16T15:48:18.514 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:18.507+0000 7ff141344640 1 -- 172.21.15.123:0/1500721814 learned_addr learned my addr 172.21.15.123:0/1500721814 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:48:18.514 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:18.507+0000 7ff141b45640 1 --2- 172.21.15.123:0/1500721814 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7ff13c15ddc0 0x7ff13c161010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:48:18.514 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:18.507+0000 7ff141344640 1 -- 172.21.15.123:0/1500721814 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7ff13c15ddc0 msgr2=0x7ff13c161010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:48:18.514 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:18.507+0000 7ff141344640 1 --2- 172.21.15.123:0/1500721814 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7ff13c15ddc0 0x7ff13c161010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:18.514 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:18.507+0000 7ff141344640 1 -- 172.21.15.123:0/1500721814 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff13c0ce560 msgr2=0x7ff13c160ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:48:18.515 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:18.507+0000 7ff140b43640 1 --2- 172.21.15.123:0/1500721814 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff13c0ce560 0x7ff13c160ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:48:18.515 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:18.507+0000 7ff141344640 1 --2- 172.21.15.123:0/1500721814 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff13c0ce560 0x7ff13c160ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:18.515 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:18.507+0000 7ff141344640 1 -- 172.21.15.123:0/1500721814 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff13c0b4cb0 con 0x7ff13c0b32f0 2024-09-16T15:48:18.515 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:18.507+0000 7ff141344640 1 --2- 172.21.15.123:0/1500721814 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff13c0b32f0 0x7ff13c14fab0 secure :-1 s=READY pgs=1027 cs=0 l=1 rev1=1 crypto rx=0x7ff13800b440 tx=0x7ff13800b900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:48:18.515 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:18.507+0000 7ff131ffb640 1 -- 172.21.15.123:0/1500721814 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff138015020 con 0x7ff13c0b32f0 2024-09-16T15:48:18.515 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:18.507+0000 7ff131ffb640 1 -- 172.21.15.123:0/1500721814 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7ff1380119a0 con 0x7ff13c0b32f0 2024-09-16T15:48:18.515 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:18.507+0000 7ff1435a8640 1 -- 172.21.15.123:0/1500721814 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff13c0b4f90 con 0x7ff13c0b32f0 2024-09-16T15:48:18.515 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:18.507+0000 7ff131ffb640 1 -- 172.21.15.123:0/1500721814 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff138011df0 con 0x7ff13c0b32f0 2024-09-16T15:48:18.515 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:18.507+0000 7ff1435a8640 1 -- 172.21.15.123:0/1500721814 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff13c0b5550 con 0x7ff13c0b32f0 2024-09-16T15:48:18.515 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:18.511+0000 7ff1435a8640 1 -- 172.21.15.123:0/1500721814 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff13c1619d0 con 0x7ff13c0b32f0 2024-09-16T15:48:18.516 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:18.511+0000 7ff131ffb640 1 -- 172.21.15.123:0/1500721814 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 16) ==== 97182+0+0 (secure 0 0 0) 0x7ff138018020 con 0x7ff13c0b32f0 2024-09-16T15:48:18.516 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:18.511+0000 7ff131ffb640 1 --2- 172.21.15.123:0/1500721814 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7ff120072fe0 0x7ff1200754a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:48:18.516 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:18.511+0000 7ff140b43640 1 --2- 172.21.15.123:0/1500721814 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7ff120072fe0 0x7ff1200754a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:48:18.516 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:18.511+0000 7ff131ffb640 1 -- 172.21.15.123:0/1500721814 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 5029+0+0 (secure 0 0 0) 0x7ff138093f80 con 0x7ff13c0b32f0 2024-09-16T15:48:18.516 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:18.511+0000 7ff140b43640 1 --2- 172.21.15.123:0/1500721814 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7ff120072fe0 0x7ff1200754a0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7ff12c007fd0 tx=0x7ff12c009040 comp rx=0 tx=0).ready entity=mgr.4150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:48:18.519 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:18.515+0000 7ff131ffb640 1 -- 172.21.15.123:0/1500721814 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7ff13801f020 con 0x7ff13c0b32f0 2024-09-16T15:48:18.693 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:18.687+0000 7ff1435a8640 1 -- 172.21.15.123:0/1500721814 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7ff13c071200 con 0x7ff13c0b32f0 2024-09-16T15:48:18.695 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-09-16T15:48:18.695 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":95,"fsid":"56077f89-18c8-486f-ae7e-eecc0bb7822c","created":"2024-09-16T15:42:26.364892+0000","modified":"2024-09-16T15:48:17.339173+0000","last_up_change":"2024-09-16T15:42:38.707234+0000","last_in_change":"2024-09-16T15:42:29.423662+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":15,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-16T15:42:39.287764+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9e7c1955-a607-4671-9be7-f2ef760ceeaa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6800","nonce":3882044969},{"type":"v1","addr":"172.21.15.123:6801","nonce":3882044969}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6802","nonce":3882044969},{"type":"v1","addr":"172.21.15.123:6803","nonce":3882044969}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6806","nonce":3882044969},{"type":"v1","addr":"172.21.15.123:6807","nonce":3882044969}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6804","nonce":3882044969},{"type":"v1","addr":"172.21.15.123:6805","nonce":3882044969}]},"public_addr":"172.21.15.123:6801/3882044969","cluster_addr":"172.21.15.123:6803/3882044969","heartbeat_back_addr":"172.21.15.123:6807/3882044969","heartbeat_front_addr":"172.21.15.123:6805/3882044969","state":["exists","up"]},{"osd":1,"uuid":"c540cfde-0977-4742-9b17-fd634668e010","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6816","nonce":4037001850},{"type":"v1","addr":"172.21.15.123:6817","nonce":4037001850}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6818","nonce":4037001850},{"type":"v1","addr":"172.21.15.123:6819","nonce":4037001850}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6822","nonce":4037001850},{"type":"v1","addr":"172.21.15.123:6823","nonce":4037001850}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6820","nonce":4037001850},{"type":"v1","addr":"172.21.15.123:6821","nonce":4037001850}]},"public_addr":"172.21.15.123:6817/4037001850","cluster_addr":"172.21.15.123:6819/4037001850","heartbeat_back_addr":"172.21.15.123:6823/4037001850","heartbeat_front_addr":"172.21.15.123:6821/4037001850","state":["exists","up"]},{"osd":2,"uuid":"9864a496-f169-486b-95c4-1d59162793a7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6824","nonce":1332143088},{"type":"v1","addr":"172.21.15.123:6825","nonce":1332143088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6826","nonce":1332143088},{"type":"v1","addr":"172.21.15.123:6827","nonce":1332143088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6830","nonce":1332143088},{"type":"v1","addr":"172.21.15.123:6831","nonce":1332143088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6828","nonce":1332143088},{"type":"v1","addr":"172.21.15.123:6829","nonce":1332143088}]},"public_addr":"172.21.15.123:6825/1332143088","cluster_addr":"172.21.15.123:6827/1332143088","heartbeat_back_addr":"172.21.15.123:6831/1332143088","heartbeat_front_addr":"172.21.15.123:6829/1332143088","state":["exists","up"]},{"osd":3,"uuid":"c6d1f817-f074-4325-95e1-578eba13edda","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6808","nonce":988586998},{"type":"v1","addr":"172.21.15.123:6809","nonce":988586998}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6810","nonce":988586998},{"type":"v1","addr":"172.21.15.123:6811","nonce":988586998}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6814","nonce":988586998},{"type":"v1","addr":"172.21.15.123:6815","nonce":988586998}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6812","nonce":988586998},{"type":"v1","addr":"172.21.15.123:6813","nonce":988586998}]},"public_addr":"172.21.15.123:6809/988586998","cluster_addr":"172.21.15.123:6811/988586998","heartbeat_back_addr":"172.21.15.123:6815/988586998","heartbeat_front_addr":"172.21.15.123:6813/988586998","state":["exists","up"]},{"osd":4,"uuid":"200a8df4-50ef-40a5-b48a-58981269c086","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6816","nonce":3387813890},{"type":"v1","addr":"172.21.15.184:6817","nonce":3387813890}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6818","nonce":3387813890},{"type":"v1","addr":"172.21.15.184:6819","nonce":3387813890}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6822","nonce":3387813890},{"type":"v1","addr":"172.21.15.184:6823","nonce":3387813890}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6820","nonce":3387813890},{"type":"v1","addr":"172.21.15.184:6821","nonce":3387813890}]},"public_addr":"172.21.15.184:6817/3387813890","cluster_addr":"172.21.15.184:6819/3387813890","heartbeat_back_addr":"172.21.15.184:6823/3387813890","heartbeat_front_addr":"172.21.15.184:6821/3387813890","state":["exists","up"]},{"osd":5,"uuid":"32d5a900-da3d-412c-bc19-13861a2368d8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6808","nonce":2680916017},{"type":"v1","addr":"172.21.15.184:6809","nonce":2680916017}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6810","nonce":2680916017},{"type":"v1","addr":"172.21.15.184:6811","nonce":2680916017}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6814","nonce":2680916017},{"type":"v1","addr":"172.21.15.184:6815","nonce":2680916017}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6812","nonce":2680916017},{"type":"v1","addr":"172.21.15.184:6813","nonce":2680916017}]},"public_addr":"172.21.15.184:6809/2680916017","cluster_addr":"172.21.15.184:6811/2680916017","heartbeat_back_addr":"172.21.15.184:6815/2680916017","heartbeat_front_addr":"172.21.15.184:6813/2680916017","state":["exists","up"]},{"osd":6,"uuid":"cde6f39e-aa1b-4260-ac19-774ee6aaca72","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6824","nonce":2183120457},{"type":"v1","addr":"172.21.15.184:6825","nonce":2183120457}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6826","nonce":2183120457},{"type":"v1","addr":"172.21.15.184:6827","nonce":2183120457}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6830","nonce":2183120457},{"type":"v1","addr":"172.21.15.184:6831","nonce":2183120457}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6828","nonce":2183120457},{"type":"v1","addr":"172.21.15.184:6829","nonce":2183120457}]},"public_addr":"172.21.15.184:6825/2183120457","cluster_addr":"172.21.15.184:6827/2183120457","heartbeat_back_addr":"172.21.15.184:6831/2183120457","heartbeat_front_addr":"172.21.15.184:6829/2183120457","state":["exists","up"]},{"osd":7,"uuid":"4f381778-eee1-47c6-804f-df0e6b33e313","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6800","nonce":4102957285},{"type":"v1","addr":"172.21.15.184:6801","nonce":4102957285}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6802","nonce":4102957285},{"type":"v1","addr":"172.21.15.184:6803","nonce":4102957285}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6806","nonce":4102957285},{"type":"v1","addr":"172.21.15.184:6807","nonce":4102957285}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6804","nonce":4102957285},{"type":"v1","addr":"172.21.15.184:6805","nonce":4102957285}]},"public_addr":"172.21.15.184:6801/4102957285","cluster_addr":"172.21.15.184:6803/4102957285","heartbeat_back_addr":"172.21.15.184:6807/4102957285","heartbeat_front_addr":"172.21.15.184:6805/4102957285","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:35.972210+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.222007+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.198617+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.385663+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.358935+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.156066+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.752973+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:35.682846+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.184:6834/1546532515":"2024-09-17T15:48:14.056433+0000","172.21.15.184:6832/1546532515":"2024-09-17T15:48:14.056433+0000","172.21.15.123:6835/2536398045":"2024-09-17T15:48:07.284775+0000","172.21.15.123:6833/2536398045":"2024-09-17T15:48:07.284775+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T15:48:18.695 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:18.691+0000 7ff131ffb640 1 -- 172.21.15.123:0/1500721814 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v95) ==== 74+0+13227 (secure 0 0 0) 0x7ff13800d040 con 0x7ff13c0b32f0 2024-09-16T15:48:18.700 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:18.695+0000 7ff1435a8640 1 -- 172.21.15.123:0/1500721814 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7ff120072fe0 msgr2=0x7ff1200754a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:48:18.701 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:18.695+0000 7ff1435a8640 1 --2- 172.21.15.123:0/1500721814 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7ff120072fe0 0x7ff1200754a0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7ff12c007fd0 tx=0x7ff12c009040 comp rx=0 tx=0).stop 2024-09-16T15:48:18.701 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:18.695+0000 7ff1435a8640 1 -- 172.21.15.123:0/1500721814 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff13c0b32f0 msgr2=0x7ff13c14fab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:48:18.701 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:18.695+0000 7ff1435a8640 1 --2- 172.21.15.123:0/1500721814 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff13c0b32f0 0x7ff13c14fab0 secure :-1 s=READY pgs=1027 cs=0 l=1 rev1=1 crypto rx=0x7ff13800b440 tx=0x7ff13800b900 comp rx=0 tx=0).stop 2024-09-16T15:48:18.701 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:18.695+0000 7ff1435a8640 1 -- 172.21.15.123:0/1500721814 shutdown_connections 2024-09-16T15:48:18.701 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:18.695+0000 7ff1435a8640 1 --2- 172.21.15.123:0/1500721814 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7ff120072fe0 0x7ff1200754a0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:18.702 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:18.695+0000 7ff1435a8640 1 --2- 172.21.15.123:0/1500721814 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7ff13c15ddc0 0x7ff13c161010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:18.702 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:18.695+0000 7ff1435a8640 1 --2- 172.21.15.123:0/1500721814 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff13c0ce560 0x7ff13c160ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:18.702 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:18.695+0000 7ff1435a8640 1 --2- 172.21.15.123:0/1500721814 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff13c0b32f0 0x7ff13c14fab0 unknown :-1 s=CLOSED pgs=1027 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:18.702 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:18.695+0000 7ff1435a8640 1 -- 172.21.15.123:0/1500721814 >> 172.21.15.123:0/1500721814 conn(0x7ff13c074bd0 msgr2=0x7ff13c15b380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:48:18.702 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:18.695+0000 7ff1435a8640 1 -- 172.21.15.123:0/1500721814 shutdown_connections 2024-09-16T15:48:18.702 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:18.695+0000 7ff1435a8640 1 -- 172.21.15.123:0/1500721814 wait complete. 2024-09-16T15:48:18.719 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-16T15:42:39.287764+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-09-16T15:48:18.720 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-09-16T15:48:18.848 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:18.843+0000 7f39ab55e640 1 -- 172.21.15.123:0/2422786170 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f39a415aa90 msgr2=0x7f39a40d00f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:48:18.849 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:18.843+0000 7f39ab55e640 1 --2- 172.21.15.123:0/2422786170 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f39a415aa90 0x7f39a40d00f0 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f39a0009900 tx=0x7f39a001c820 comp rx=0 tx=0).stop 2024-09-16T15:48:18.849 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:18.843+0000 7f39ab55e640 1 -- 172.21.15.123:0/2422786170 shutdown_connections 2024-09-16T15:48:18.849 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:18.843+0000 7f39ab55e640 1 --2- 172.21.15.123:0/2422786170 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f39a40cdb00 0x7f39a415f0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:18.849 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:18.843+0000 7f39ab55e640 1 --2- 172.21.15.123:0/2422786170 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f39a415aa90 0x7f39a40d00f0 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:18.849 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:18.843+0000 7f39ab55e640 1 -- 172.21.15.123:0/2422786170 >> 172.21.15.123:0/2422786170 conn(0x7f39a40d0b70 msgr2=0x7f39a40d0f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:48:18.849 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:18.843+0000 7f39ab55e640 1 -- 172.21.15.123:0/2422786170 shutdown_connections 2024-09-16T15:48:18.849 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:18.843+0000 7f39ab55e640 1 -- 172.21.15.123:0/2422786170 wait complete. 2024-09-16T15:48:18.850 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:18.847+0000 7f39ab55e640 1 Processor -- start 2024-09-16T15:48:18.850 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:18.847+0000 7f39ab55e640 1 -- start start 2024-09-16T15:48:18.852 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:18.847+0000 7f39ab55e640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f39a40cdb00 0x7f39a41452f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:48:18.852 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:18.847+0000 7f39ab55e640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f39a40cfca0 0x7f39a4159620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:48:18.852 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:18.847+0000 7f39ab55e640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f39a415aa90 0x7f39a40b1830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:48:18.852 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:18.847+0000 7f39ab55e640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f39a406cd70 con 0x7f39a415aa90 2024-09-16T15:48:18.852 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:18.847+0000 7f39ab55e640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f39a406cbf0 con 0x7f39a40cdb00 2024-09-16T15:48:18.852 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:18.847+0000 7f39ab55e640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f39a406cef0 con 0x7f39a40cfca0 2024-09-16T15:48:18.853 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:18.847+0000 7f39a9afb640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f39a415aa90 0x7f39a40b1830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:48:18.853 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:18.847+0000 7f39a9afb640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f39a415aa90 0x7f39a40b1830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:41616/0 (socket says 172.21.15.123:41616) 2024-09-16T15:48:18.853 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:18.847+0000 7f39a9afb640 1 -- 172.21.15.123:0/30163632 learned_addr learned my addr 172.21.15.123:0/30163632 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:48:18.853 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:18.847+0000 7f39a92fa640 1 --2- 172.21.15.123:0/30163632 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f39a40cdb00 0x7f39a41452f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:48:18.853 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:18.847+0000 7f39a8af9640 1 --2- 172.21.15.123:0/30163632 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f39a40cfca0 0x7f39a4159620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:48:18.853 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:18.847+0000 7f39a9afb640 1 -- 172.21.15.123:0/30163632 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f39a40cfca0 msgr2=0x7f39a4159620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:48:18.853 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:18.847+0000 7f39a9afb640 1 --2- 172.21.15.123:0/30163632 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f39a40cfca0 0x7f39a4159620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:18.853 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:18.847+0000 7f39a9afb640 1 -- 172.21.15.123:0/30163632 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f39a40cdb00 msgr2=0x7f39a41452f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:48:18.853 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:18.847+0000 7f39a9afb640 1 --2- 172.21.15.123:0/30163632 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f39a40cdb00 0x7f39a41452f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:18.853 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:18.847+0000 7f39a9afb640 1 -- 172.21.15.123:0/30163632 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f39a4145ab0 con 0x7f39a415aa90 2024-09-16T15:48:18.853 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:18.847+0000 7f39a9afb640 1 --2- 172.21.15.123:0/30163632 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f39a415aa90 0x7f39a40b1830 secure :-1 s=READY pgs=1028 cs=0 l=1 rev1=1 crypto rx=0x7f399c00eaf0 tx=0x7f399c00c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:48:18.854 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:18.847+0000 7f3999ffb640 1 -- 172.21.15.123:0/30163632 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f399c004b30 con 0x7f39a415aa90 2024-09-16T15:48:18.854 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:18.847+0000 7f3999ffb640 1 -- 172.21.15.123:0/30163632 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f399c004cf0 con 0x7f39a415aa90 2024-09-16T15:48:18.854 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:18.847+0000 7f3999ffb640 1 -- 172.21.15.123:0/30163632 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f399c005280 con 0x7f39a415aa90 2024-09-16T15:48:18.854 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:18.847+0000 7f39ab55e640 1 -- 172.21.15.123:0/30163632 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f39a4145d90 con 0x7f39a415aa90 2024-09-16T15:48:18.854 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:18.847+0000 7f39ab55e640 1 -- 172.21.15.123:0/30163632 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f39a4146350 con 0x7f39a415aa90 2024-09-16T15:48:18.855 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:18.847+0000 7f39ab55e640 1 -- 172.21.15.123:0/30163632 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f39a415f0f0 con 0x7f39a415aa90 2024-09-16T15:48:18.855 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:18.851+0000 7f3999ffb640 1 -- 172.21.15.123:0/30163632 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 16) ==== 97182+0+0 (secure 0 0 0) 0x7f399c005420 con 0x7f39a415aa90 2024-09-16T15:48:18.855 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:18.851+0000 7f3999ffb640 1 --2- 172.21.15.123:0/30163632 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f3988072fe0 0x7f39880754a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:48:18.855 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:18.851+0000 7f3999ffb640 1 -- 172.21.15.123:0/30163632 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 5029+0+0 (secure 0 0 0) 0x7f399c094b40 con 0x7f39a415aa90 2024-09-16T15:48:18.855 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:18.851+0000 7f39a92fa640 1 --2- 172.21.15.123:0/30163632 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f3988072fe0 0x7f39880754a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:48:18.855 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:18.851+0000 7f39a92fa640 1 --2- 172.21.15.123:0/30163632 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f3988072fe0 0x7f39880754a0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f39a00af000 tx=0x7f39a001b060 comp rx=0 tx=0).ready entity=mgr.4150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:48:18.859 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:18.855+0000 7f3999ffb640 1 -- 172.21.15.123:0/30163632 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f399c05fe10 con 0x7f39a415aa90 2024-09-16T15:48:19.029 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:19.023+0000 7f39ab55e640 1 -- 172.21.15.123:0/30163632 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f39a406f890 con 0x7f39a415aa90 2024-09-16T15:48:19.030 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:19.027+0000 7f3999ffb640 1 -- 172.21.15.123:0/30163632 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v95) ==== 93+0+10 (secure 0 0 0) 0x7f399c05f710 con 0x7f39a415aa90 2024-09-16T15:48:19.030 INFO:teuthology.orchestra.run.smithi123.stdout:pg_num: 1 2024-09-16T15:48:19.033 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:19.027+0000 7f39ab55e640 1 -- 172.21.15.123:0/30163632 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f3988072fe0 msgr2=0x7f39880754a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:48:19.033 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:19.027+0000 7f39ab55e640 1 --2- 172.21.15.123:0/30163632 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f3988072fe0 0x7f39880754a0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f39a00af000 tx=0x7f39a001b060 comp rx=0 tx=0).stop 2024-09-16T15:48:19.033 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:19.027+0000 7f39ab55e640 1 -- 172.21.15.123:0/30163632 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f39a415aa90 msgr2=0x7f39a40b1830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:48:19.033 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:19.027+0000 7f39ab55e640 1 --2- 172.21.15.123:0/30163632 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f39a415aa90 0x7f39a40b1830 secure :-1 s=READY pgs=1028 cs=0 l=1 rev1=1 crypto rx=0x7f399c00eaf0 tx=0x7f399c00c2d0 comp rx=0 tx=0).stop 2024-09-16T15:48:19.033 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:19.027+0000 7f39ab55e640 1 -- 172.21.15.123:0/30163632 shutdown_connections 2024-09-16T15:48:19.033 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:19.027+0000 7f39ab55e640 1 --2- 172.21.15.123:0/30163632 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f3988072fe0 0x7f39880754a0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:19.033 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:19.027+0000 7f39ab55e640 1 --2- 172.21.15.123:0/30163632 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f39a415aa90 0x7f39a40b1830 unknown :-1 s=CLOSED pgs=1028 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:19.034 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:19.027+0000 7f39ab55e640 1 --2- 172.21.15.123:0/30163632 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f39a40cfca0 0x7f39a4159620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:19.034 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:19.027+0000 7f39ab55e640 1 --2- 172.21.15.123:0/30163632 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f39a40cdb00 0x7f39a41452f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:19.034 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:19.027+0000 7f39ab55e640 1 -- 172.21.15.123:0/30163632 >> 172.21.15.123:0/30163632 conn(0x7f39a40d0b70 msgr2=0x7f39a414a420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:48:19.034 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:19.027+0000 7f39ab55e640 1 -- 172.21.15.123:0/30163632 shutdown_connections 2024-09-16T15:48:19.034 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:19.027+0000 7f39ab55e640 1 -- 172.21.15.123:0/30163632 wait complete. 2024-09-16T15:48:19.049 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_volumes.TestMisc.test_subvolume_no_upgrade_v1_sanity' 2024-09-16T15:48:19.166 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:19.163+0000 7f3effd7f640 1 -- 172.21.15.123:0/1420780827 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3ef80726c0 msgr2=0x7f3ef8073ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:48:19.167 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:19.163+0000 7f3effd7f640 1 --2- 172.21.15.123:0/1420780827 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3ef80726c0 0x7f3ef8073ef0 secure :-1 s=READY pgs=1029 cs=0 l=1 rev1=1 crypto rx=0x7f3ef80cd650 tx=0x7f3ee801c800 comp rx=0 tx=0).stop 2024-09-16T15:48:19.167 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:19.163+0000 7f3effd7f640 1 -- 172.21.15.123:0/1420780827 shutdown_connections 2024-09-16T15:48:19.167 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:19.163+0000 7f3effd7f640 1 --2- 172.21.15.123:0/1420780827 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3ef80726c0 0x7f3ef8073ef0 unknown :-1 s=CLOSED pgs=1029 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:19.167 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:19.163+0000 7f3effd7f640 1 --2- 172.21.15.123:0/1420780827 >> v2:172.21.15.184:3300/0 conn(0x7f3ef815bca0 0x7f3ef80d0630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:19.167 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:19.163+0000 7f3effd7f640 1 -- 172.21.15.123:0/1420780827 >> 172.21.15.123:0/1420780827 conn(0x7f3ef80cfb10 msgr2=0x7f3ef80cff20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:48:19.167 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:19.163+0000 7f3effd7f640 1 -- 172.21.15.123:0/1420780827 shutdown_connections 2024-09-16T15:48:19.167 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:19.163+0000 7f3effd7f640 1 -- 172.21.15.123:0/1420780827 wait complete. 2024-09-16T15:48:19.168 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:19.163+0000 7f3effd7f640 1 Processor -- start 2024-09-16T15:48:19.168 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:19.163+0000 7f3effd7f640 1 -- start start 2024-09-16T15:48:19.170 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:19.163+0000 7f3effd7f640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f3ef80726c0 0x7f3ef815ee90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:48:19.170 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:19.163+0000 7f3effd7f640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3ef8074430 0x7f3ef81f8c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:48:19.170 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:19.163+0000 7f3effd7f640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3ef815bca0 0x7f3ef8152f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:48:19.170 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:19.163+0000 7f3effd7f640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f3ef8168ef0 con 0x7f3ef8074430 2024-09-16T15:48:19.170 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:19.163+0000 7f3effd7f640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f3ef8168d70 con 0x7f3ef815bca0 2024-09-16T15:48:19.170 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:19.163+0000 7f3effd7f640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f3ef8169070 con 0x7f3ef80726c0 2024-09-16T15:48:19.170 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:19.163+0000 7f3efd31a640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3ef8074430 0x7f3ef81f8c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:48:19.170 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:19.163+0000 7f3efd31a640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3ef8074430 0x7f3ef81f8c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:41640/0 (socket says 172.21.15.123:41640) 2024-09-16T15:48:19.170 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:19.163+0000 7f3efd31a640 1 -- 172.21.15.123:0/2528573018 learned_addr learned my addr 172.21.15.123:0/2528573018 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:48:19.170 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:19.163+0000 7f3efd31a640 1 -- 172.21.15.123:0/2528573018 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f3ef80726c0 msgr2=0x7f3ef815ee90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:48:19.170 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:19.163+0000 7f3efe31c640 1 --2- 172.21.15.123:0/2528573018 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3ef815bca0 0x7f3ef8152f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:48:19.171 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:19.163+0000 7f3efd31a640 1 --2- 172.21.15.123:0/2528573018 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f3ef80726c0 0x7f3ef815ee90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:19.171 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:19.163+0000 7f3efd31a640 1 -- 172.21.15.123:0/2528573018 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3ef815bca0 msgr2=0x7f3ef8152f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:48:19.171 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:19.163+0000 7f3efd31a640 1 --2- 172.21.15.123:0/2528573018 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3ef815bca0 0x7f3ef8152f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:19.171 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:19.163+0000 7f3efd31a640 1 -- 172.21.15.123:0/2528573018 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3ef80b2bf0 con 0x7f3ef8074430 2024-09-16T15:48:19.171 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:19.163+0000 7f3efe31c640 1 --2- 172.21.15.123:0/2528573018 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3ef815bca0 0x7f3ef8152f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T15:48:19.171 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:19.163+0000 7f3efd31a640 1 --2- 172.21.15.123:0/2528573018 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3ef8074430 0x7f3ef81f8c00 secure :-1 s=READY pgs=1030 cs=0 l=1 rev1=1 crypto rx=0x7f3ee80b0040 tx=0x7f3ee80a5700 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:48:19.171 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:19.163+0000 7f3eee7fc640 1 -- 172.21.15.123:0/2528573018 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ee80b2040 con 0x7f3ef8074430 2024-09-16T15:48:19.171 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:19.163+0000 7f3eee7fc640 1 -- 172.21.15.123:0/2528573018 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f3ee8004380 con 0x7f3ef8074430 2024-09-16T15:48:19.171 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:19.163+0000 7f3eee7fc640 1 -- 172.21.15.123:0/2528573018 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ee8005290 con 0x7f3ef8074430 2024-09-16T15:48:19.171 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:19.163+0000 7f3effd7f640 1 -- 172.21.15.123:0/2528573018 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3ef80b2ed0 con 0x7f3ef8074430 2024-09-16T15:48:19.171 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:19.163+0000 7f3effd7f640 1 -- 172.21.15.123:0/2528573018 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3ef8153480 con 0x7f3ef8074430 2024-09-16T15:48:19.171 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:19.167+0000 7f3effd7f640 1 -- 172.21.15.123:0/2528573018 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3ef8073ef0 con 0x7f3ef8074430 2024-09-16T15:48:19.172 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:19.167+0000 7f3eee7fc640 1 -- 172.21.15.123:0/2528573018 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 16) ==== 97182+0+0 (secure 0 0 0) 0x7f3ee8005430 con 0x7f3ef8074430 2024-09-16T15:48:19.172 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:19.167+0000 7f3eee7fc640 1 --2- 172.21.15.123:0/2528573018 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f3edc072fe0 0x7f3edc0754a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:48:19.172 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:19.167+0000 7f3eee7fc640 1 -- 172.21.15.123:0/2528573018 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 5029+0+0 (secure 0 0 0) 0x7f3ee812db00 con 0x7f3ef8074430 2024-09-16T15:48:19.172 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:19.167+0000 7f3efdb1b640 1 --2- 172.21.15.123:0/2528573018 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f3edc072fe0 0x7f3edc0754a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:48:19.172 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:19.167+0000 7f3efdb1b640 1 --2- 172.21.15.123:0/2528573018 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f3edc072fe0 0x7f3edc0754a0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f3ef40058d0 tx=0x7f3ef4005860 comp rx=0 tx=0).ready entity=mgr.4150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:48:19.175 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:19.171+0000 7f3eee7fc640 1 -- 172.21.15.123:0/2528573018 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f3ee80164b0 con 0x7f3ef8074430 2024-09-16T15:48:19.406 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:19.403+0000 7f3effd7f640 1 -- 172.21.15.123:0/2528573018 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_volumes.TestMisc.test_subvolume_no_upgrade_v1_sanity"]} v 0) -- 0x7f3ef8165f00 con 0x7f3ef8074430 2024-09-16T15:48:20.377 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:20.371+0000 7f3eee7fc640 1 -- 172.21.15.123:0/2528573018 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_volumes.TestMisc.test_subvolume_no_upgrade_v1_sanity"]}]=0 v284) ==== 152+0+0 (secure 0 0 0) 0x7f3ee80f8dc0 con 0x7f3ef8074430 2024-09-16T15:48:20.380 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:20.375+0000 7f3effd7f640 1 -- 172.21.15.123:0/2528573018 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f3edc072fe0 msgr2=0x7f3edc0754a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:48:20.380 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:20.375+0000 7f3effd7f640 1 --2- 172.21.15.123:0/2528573018 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f3edc072fe0 0x7f3edc0754a0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f3ef40058d0 tx=0x7f3ef4005860 comp rx=0 tx=0).stop 2024-09-16T15:48:20.381 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:20.375+0000 7f3effd7f640 1 -- 172.21.15.123:0/2528573018 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3ef8074430 msgr2=0x7f3ef81f8c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:48:20.381 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:20.375+0000 7f3effd7f640 1 --2- 172.21.15.123:0/2528573018 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3ef8074430 0x7f3ef81f8c00 secure :-1 s=READY pgs=1030 cs=0 l=1 rev1=1 crypto rx=0x7f3ee80b0040 tx=0x7f3ee80a5700 comp rx=0 tx=0).stop 2024-09-16T15:48:20.381 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:20.375+0000 7f3effd7f640 1 -- 172.21.15.123:0/2528573018 shutdown_connections 2024-09-16T15:48:20.381 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:20.375+0000 7f3effd7f640 1 --2- 172.21.15.123:0/2528573018 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f3edc072fe0 0x7f3edc0754a0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:20.381 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:20.375+0000 7f3effd7f640 1 --2- 172.21.15.123:0/2528573018 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3ef815bca0 0x7f3ef8152f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:20.381 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:20.375+0000 7f3effd7f640 1 --2- 172.21.15.123:0/2528573018 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3ef8074430 0x7f3ef81f8c00 unknown :-1 s=CLOSED pgs=1030 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:20.381 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:20.375+0000 7f3effd7f640 1 --2- 172.21.15.123:0/2528573018 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f3ef80726c0 0x7f3ef815ee90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:20.381 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:20.375+0000 7f3effd7f640 1 -- 172.21.15.123:0/2528573018 >> 172.21.15.123:0/2528573018 conn(0x7f3ef80cfb10 msgr2=0x7f3ef80cd6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:48:20.381 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:20.375+0000 7f3effd7f640 1 -- 172.21.15.123:0/2528573018 shutdown_connections 2024-09-16T15:48:20.381 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:20.375+0000 7f3effd7f640 1 -- 172.21.15.123:0/2528573018 wait complete. 2024-09-16T15:48:20.400 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config dump 2024-09-16T15:48:20.531 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:20.527+0000 7efe629ad640 1 -- 172.21.15.123:0/1175029604 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7efe44004d50 msgr2=0x7efe440051d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:48:20.532 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:20.527+0000 7efe629ad640 1 --2- 172.21.15.123:0/1175029604 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7efe44004d50 0x7efe440051d0 secure :-1 s=READY pgs=1032 cs=0 l=1 rev1=1 crypto rx=0x7efe54009e00 tx=0x7efe5401d170 comp rx=0 tx=0).stop 2024-09-16T15:48:20.532 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:20.527+0000 7efe60f4a640 1 -- 172.21.15.123:0/1175029604 reap_dead start 2024-09-16T15:48:20.532 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:20.527+0000 7efe629ad640 1 -- 172.21.15.123:0/1175029604 shutdown_connections 2024-09-16T15:48:20.532 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:20.527+0000 7efe629ad640 1 -- 172.21.15.123:0/1175029604 >> 172.21.15.123:0/1175029604 conn(0x7efe5c0b2610 msgr2=0x7efe5c0b29f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:48:20.532 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:20.527+0000 7efe629ad640 1 -- 172.21.15.123:0/1175029604 shutdown_connections 2024-09-16T15:48:20.532 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:20.527+0000 7efe629ad640 1 -- 172.21.15.123:0/1175029604 wait complete. 2024-09-16T15:48:20.533 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:20.527+0000 7efe629ad640 1 Processor -- start 2024-09-16T15:48:20.533 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:20.527+0000 7efe629ad640 1 -- start start 2024-09-16T15:48:20.535 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:20.531+0000 7efe629ad640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7efe44004d50 0x7efe5c1f2d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:48:20.535 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:20.531+0000 7efe629ad640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7efe5c1f6230 0x7efe5c155950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:48:20.535 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:20.531+0000 7efe629ad640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7efe5c156380 0x7efe5c156800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:48:20.535 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:20.531+0000 7efe629ad640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7efe5c06dab0 con 0x7efe5c156380 2024-09-16T15:48:20.536 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:20.531+0000 7efe629ad640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7efe5c06d930 con 0x7efe5c1f6230 2024-09-16T15:48:20.536 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:20.531+0000 7efe629ad640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7efe5c06cbf0 con 0x7efe44004d50 2024-09-16T15:48:20.536 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:20.531+0000 7efe60f4a640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7efe5c156380 0x7efe5c156800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:48:20.536 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:20.531+0000 7efe60f4a640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7efe5c156380 0x7efe5c156800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:41662/0 (socket says 172.21.15.123:41662) 2024-09-16T15:48:20.536 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:20.531+0000 7efe60f4a640 1 -- 172.21.15.123:0/2164473813 learned_addr learned my addr 172.21.15.123:0/2164473813 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:48:20.536 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:20.531+0000 7efe5bfff640 1 --2- 172.21.15.123:0/2164473813 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7efe44004d50 0x7efe5c1f2d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:48:20.536 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:20.531+0000 7efe5b7fe640 1 --2- 172.21.15.123:0/2164473813 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7efe5c1f6230 0x7efe5c155950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:48:20.536 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:20.531+0000 7efe60f4a640 1 -- 172.21.15.123:0/2164473813 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7efe44004d50 msgr2=0x7efe5c1f2d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:48:20.536 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:20.531+0000 7efe60f4a640 1 --2- 172.21.15.123:0/2164473813 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7efe44004d50 0x7efe5c1f2d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:20.536 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:20.531+0000 7efe60f4a640 1 -- 172.21.15.123:0/2164473813 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7efe5c1f6230 msgr2=0x7efe5c155950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:48:20.536 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:20.531+0000 7efe60f4a640 1 --2- 172.21.15.123:0/2164473813 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7efe5c1f6230 0x7efe5c155950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:20.536 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:20.531+0000 7efe60f4a640 1 -- 172.21.15.123:0/2164473813 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efe5c1f2270 con 0x7efe5c156380 2024-09-16T15:48:20.537 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:20.531+0000 7efe5b7fe640 1 --2- 172.21.15.123:0/2164473813 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7efe5c1f6230 0x7efe5c155950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T15:48:20.537 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:20.531+0000 7efe60f4a640 1 --2- 172.21.15.123:0/2164473813 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7efe5c156380 0x7efe5c156800 secure :-1 s=READY pgs=1033 cs=0 l=1 rev1=1 crypto rx=0x7efe540af040 tx=0x7efe540add50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:48:20.537 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:20.531+0000 7efe58ff9640 1 -- 172.21.15.123:0/2164473813 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe540b1040 con 0x7efe5c156380 2024-09-16T15:48:20.537 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:20.531+0000 7efe58ff9640 1 -- 172.21.15.123:0/2164473813 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7efe540bc070 con 0x7efe5c156380 2024-09-16T15:48:20.537 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:20.531+0000 7efe629ad640 1 -- 172.21.15.123:0/2164473813 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efe5c156f50 con 0x7efe5c156380 2024-09-16T15:48:20.537 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:20.531+0000 7efe58ff9640 1 -- 172.21.15.123:0/2164473813 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe540a5e60 con 0x7efe5c156380 2024-09-16T15:48:20.537 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:20.531+0000 7efe629ad640 1 -- 172.21.15.123:0/2164473813 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efe5c15eef0 con 0x7efe5c156380 2024-09-16T15:48:20.538 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:20.531+0000 7efe629ad640 1 -- 172.21.15.123:0/2164473813 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efe5c06d110 con 0x7efe5c156380 2024-09-16T15:48:20.538 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:20.531+0000 7efe58ff9640 1 -- 172.21.15.123:0/2164473813 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 16) ==== 97182+0+0 (secure 0 0 0) 0x7efe54004a50 con 0x7efe5c156380 2024-09-16T15:48:20.538 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:20.531+0000 7efe58ff9640 1 --2- 172.21.15.123:0/2164473813 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7efe48072fe0 0x7efe480754a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:48:20.538 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:20.531+0000 7efe58ff9640 1 -- 172.21.15.123:0/2164473813 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 5029+0+0 (secure 0 0 0) 0x7efe5412e7b0 con 0x7efe5c156380 2024-09-16T15:48:20.538 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:20.531+0000 7efe5bfff640 1 --2- 172.21.15.123:0/2164473813 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7efe48072fe0 0x7efe480754a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:48:20.538 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:20.531+0000 7efe5bfff640 1 --2- 172.21.15.123:0/2164473813 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7efe48072fe0 0x7efe480754a0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7efe4c00cc30 tx=0x7efe4c005f30 comp rx=0 tx=0).ready entity=mgr.4150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:48:20.541 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:20.535+0000 7efe58ff9640 1 -- 172.21.15.123:0/2164473813 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7efe54016e80 con 0x7efe5c156380 2024-09-16T15:48:20.704 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:20.699+0000 7efe629ad640 1 -- 172.21.15.123:0/2164473813 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "config dump"} v 0) -- 0x7efe5c0b59c0 con 0x7efe5c156380 2024-09-16T15:48:20.705 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:20.699+0000 7efe58ff9640 1 -- 172.21.15.123:0/2164473813 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "config dump"}]=0 v37) ==== 59+0+1998 (secure 0 0 0) 0x7efe540f94c0 con 0x7efe5c156380 2024-09-16T15:48:20.705 INFO:teuthology.orchestra.run.smithi123.stdout:WHO MASK LEVEL OPTION VALUE RO 2024-09-16T15:48:20.705 INFO:teuthology.orchestra.run.smithi123.stdout:mon advanced mon_op_complaint_time 120 2024-09-16T15:48:20.705 INFO:teuthology.orchestra.run.smithi123.stdout:mon dev mon_warn_on_pool_no_app false 2024-09-16T15:48:20.705 INFO:teuthology.orchestra.run.smithi123.stdout:osd advanced osd_op_complaint_time 180.000000 2024-09-16T15:48:20.705 INFO:teuthology.orchestra.run.smithi123.stdout:osd.0 basic osd_mclock_max_capacity_iops_ssd 4805.300677 2024-09-16T15:48:20.705 INFO:teuthology.orchestra.run.smithi123.stdout:osd.1 basic osd_mclock_max_capacity_iops_ssd 4806.240607 2024-09-16T15:48:20.705 INFO:teuthology.orchestra.run.smithi123.stdout:osd.2 basic osd_mclock_max_capacity_iops_ssd 4046.441686 2024-09-16T15:48:20.705 INFO:teuthology.orchestra.run.smithi123.stdout:osd.3 basic osd_mclock_max_capacity_iops_ssd 6882.576123 2024-09-16T15:48:20.705 INFO:teuthology.orchestra.run.smithi123.stdout:osd.4 basic osd_mclock_max_capacity_iops_ssd 6108.444971 2024-09-16T15:48:20.706 INFO:teuthology.orchestra.run.smithi123.stdout:osd.5 basic osd_mclock_max_capacity_iops_ssd 4995.096306 2024-09-16T15:48:20.706 INFO:teuthology.orchestra.run.smithi123.stdout:osd.6 basic osd_mclock_max_capacity_iops_ssd 6237.822095 2024-09-16T15:48:20.706 INFO:teuthology.orchestra.run.smithi123.stdout:osd.7 basic osd_mclock_max_capacity_iops_ssd 5202.385138 2024-09-16T15:48:20.706 INFO:teuthology.orchestra.run.smithi123.stdout:mds advanced debug_mds 20/20 2024-09-16T15:48:20.706 INFO:teuthology.orchestra.run.smithi123.stdout:mds advanced debug_mds_balancer 20/20 2024-09-16T15:48:20.706 INFO:teuthology.orchestra.run.smithi123.stdout:mds advanced debug_ms 2/2 2024-09-16T15:48:20.706 INFO:teuthology.orchestra.run.smithi123.stdout:mds dev mds_debug_frag true 2024-09-16T15:48:20.706 INFO:teuthology.orchestra.run.smithi123.stdout:mds dev mds_debug_scatterstat true 2024-09-16T15:48:20.706 INFO:teuthology.orchestra.run.smithi123.stdout:mds advanced mds_op_complaint_time 180.000000 2024-09-16T15:48:20.706 INFO:teuthology.orchestra.run.smithi123.stdout:mds dev mds_verify_scatter true 2024-09-16T15:48:20.706 INFO:teuthology.orchestra.run.smithi123.stdout:mds advanced osd_op_complaint_time 180.000000 2024-09-16T15:48:20.706 INFO:teuthology.orchestra.run.smithi123.stdout:mds advanced rados_mon_op_timeout 900 2024-09-16T15:48:20.706 INFO:teuthology.orchestra.run.smithi123.stdout:mds advanced rados_osd_op_timeout 900 2024-09-16T15:48:20.706 INFO:teuthology.orchestra.run.smithi123.stdout:client advanced client_mount_timeout 600 2024-09-16T15:48:20.707 INFO:teuthology.orchestra.run.smithi123.stdout:client advanced debug_client 20/20 2024-09-16T15:48:20.707 INFO:teuthology.orchestra.run.smithi123.stdout:client advanced debug_ms 1/1 2024-09-16T15:48:20.707 INFO:teuthology.orchestra.run.smithi123.stdout:client advanced rados_mon_op_timeout 900 2024-09-16T15:48:20.707 INFO:teuthology.orchestra.run.smithi123.stdout:client advanced rados_osd_op_timeout 900 2024-09-16T15:48:20.707 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:20.707+0000 7efe629ad640 1 -- 172.21.15.123:0/2164473813 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7efe48072fe0 msgr2=0x7efe480754a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:48:20.708 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:20.707+0000 7efe629ad640 1 --2- 172.21.15.123:0/2164473813 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7efe48072fe0 0x7efe480754a0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7efe4c00cc30 tx=0x7efe4c005f30 comp rx=0 tx=0).stop 2024-09-16T15:48:20.708 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:20.707+0000 7efe629ad640 1 -- 172.21.15.123:0/2164473813 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7efe5c156380 msgr2=0x7efe5c156800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:48:20.708 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:20.707+0000 7efe629ad640 1 --2- 172.21.15.123:0/2164473813 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7efe5c156380 0x7efe5c156800 secure :-1 s=READY pgs=1033 cs=0 l=1 rev1=1 crypto rx=0x7efe540af040 tx=0x7efe540add50 comp rx=0 tx=0).stop 2024-09-16T15:48:20.708 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:20.707+0000 7efe629ad640 1 -- 172.21.15.123:0/2164473813 shutdown_connections 2024-09-16T15:48:20.708 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:20.707+0000 7efe629ad640 1 --2- 172.21.15.123:0/2164473813 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7efe48072fe0 0x7efe480754a0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:20.708 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:20.707+0000 7efe629ad640 1 --2- 172.21.15.123:0/2164473813 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7efe5c156380 0x7efe5c156800 unknown :-1 s=CLOSED pgs=1033 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:20.708 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:20.707+0000 7efe629ad640 1 --2- 172.21.15.123:0/2164473813 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7efe5c1f6230 0x7efe5c155950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:20.708 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:20.707+0000 7efe629ad640 1 --2- 172.21.15.123:0/2164473813 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7efe44004d50 0x7efe5c1f2d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:20.708 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:20.707+0000 7efe629ad640 1 -- 172.21.15.123:0/2164473813 >> 172.21.15.123:0/2164473813 conn(0x7efe5c0b2610 msgr2=0x7efe5c0cc2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:48:20.708 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:20.707+0000 7efe629ad640 1 -- 172.21.15.123:0/2164473813 shutdown_connections 2024-09-16T15:48:20.708 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:20.707+0000 7efe629ad640 1 -- 172.21.15.123:0/2164473813 wait complete. 2024-09-16T15:48:20.722 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-09-16T15:48:20.851 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:20.851+0000 7f8739093640 1 -- 172.21.15.123:0/777234759 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f87340cc3c0 msgr2=0x7f87340cc810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:48:20.851 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:20.851+0000 7f8739093640 1 --2- 172.21.15.123:0/777234759 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f87340cc3c0 0x7f87340cc810 secure :-1 s=READY pgs=1035 cs=0 l=1 rev1=1 crypto rx=0x7f873406db90 tx=0x7f872c01cd80 comp rx=0 tx=0).stop 2024-09-16T15:48:20.851 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:20.851+0000 7f8739093640 1 -- 172.21.15.123:0/777234759 shutdown_connections 2024-09-16T15:48:20.851 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:20.851+0000 7f8739093640 1 --2- 172.21.15.123:0/777234759 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f87340cc3c0 0x7f87340cc810 unknown :-1 s=CLOSED pgs=1035 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:20.851 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:20.851+0000 7f8739093640 1 --2- 172.21.15.123:0/777234759 >> v2:172.21.15.184:3300/0 conn(0x7f873415aa00 0x7f873415e5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:20.851 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:20.851+0000 7f8739093640 1 -- 172.21.15.123:0/777234759 >> 172.21.15.123:0/777234759 conn(0x7f87340b1710 msgr2=0x7f87340b1b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:48:20.851 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:20.851+0000 7f8739093640 1 -- 172.21.15.123:0/777234759 shutdown_connections 2024-09-16T15:48:20.852 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:20.851+0000 7f8739093640 1 -- 172.21.15.123:0/777234759 wait complete. 2024-09-16T15:48:20.852 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:20.851+0000 7f8739093640 1 Processor -- start 2024-09-16T15:48:20.852 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:20.851+0000 7f8739093640 1 -- start start 2024-09-16T15:48:20.854 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:20.851+0000 7f8739093640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f8734072720 0x7f87340c5b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:48:20.854 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:20.851+0000 7f8739093640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f87340cc3c0 0x7f87340c6c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:48:20.854 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:20.851+0000 7f8739093640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f873415aa00 0x7f87341f9b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:48:20.854 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:20.851+0000 7f8739093640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f8734167af0 con 0x7f87340cc3c0 2024-09-16T15:48:20.854 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:20.851+0000 7f8739093640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f8734167970 con 0x7f873415aa00 2024-09-16T15:48:20.854 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:20.851+0000 7f8739093640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f8734167c70 con 0x7f8734072720 2024-09-16T15:48:20.854 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:20.851+0000 7f873259c640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f87340cc3c0 0x7f87340c6c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:48:20.854 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:20.851+0000 7f873259c640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f87340cc3c0 0x7f87340c6c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:41670/0 (socket says 172.21.15.123:41670) 2024-09-16T15:48:20.854 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:20.851+0000 7f873259c640 1 -- 172.21.15.123:0/2245131349 learned_addr learned my addr 172.21.15.123:0/2245131349 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:48:20.855 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:20.851+0000 7f873259c640 1 -- 172.21.15.123:0/2245131349 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f8734072720 msgr2=0x7f87340c5b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:48:20.855 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:20.851+0000 7f873259c640 1 --2- 172.21.15.123:0/2245131349 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f8734072720 0x7f87340c5b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:20.855 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:20.851+0000 7f873259c640 1 -- 172.21.15.123:0/2245131349 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f873415aa00 msgr2=0x7f87341f9b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:48:20.855 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:20.851+0000 7f873259c640 1 --2- 172.21.15.123:0/2245131349 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f873415aa00 0x7f87341f9b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:20.855 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:20.851+0000 7f873259c640 1 -- 172.21.15.123:0/2245131349 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f87340c7150 con 0x7f87340cc3c0 2024-09-16T15:48:20.855 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:20.851+0000 7f873259c640 1 --2- 172.21.15.123:0/2245131349 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f87340cc3c0 0x7f87340c6c10 secure :-1 s=READY pgs=1036 cs=0 l=1 rev1=1 crypto rx=0x7f872800c500 tx=0x7f872800c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:48:20.855 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:20.851+0000 7f871b7fe640 1 -- 172.21.15.123:0/2245131349 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8728017070 con 0x7f87340cc3c0 2024-09-16T15:48:20.855 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:20.851+0000 7f871b7fe640 1 -- 172.21.15.123:0/2245131349 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f87280053c0 con 0x7f87340cc3c0 2024-09-16T15:48:20.855 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:20.851+0000 7f871b7fe640 1 -- 172.21.15.123:0/2245131349 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8728012890 con 0x7f87340cc3c0 2024-09-16T15:48:20.855 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:20.851+0000 7f8739093640 1 -- 172.21.15.123:0/2245131349 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8734158e80 con 0x7f87340cc3c0 2024-09-16T15:48:20.855 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:20.855+0000 7f8739093640 1 -- 172.21.15.123:0/2245131349 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8734159440 con 0x7f87340cc3c0 2024-09-16T15:48:20.856 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:20.855+0000 7f8739093640 1 -- 172.21.15.123:0/2245131349 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f87340cc810 con 0x7f87340cc3c0 2024-09-16T15:48:20.856 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:20.855+0000 7f871b7fe640 1 -- 172.21.15.123:0/2245131349 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 16) ==== 97182+0+0 (secure 0 0 0) 0x7f872801a070 con 0x7f87340cc3c0 2024-09-16T15:48:20.856 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:20.855+0000 7f871b7fe640 1 --2- 172.21.15.123:0/2245131349 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f8720072fe0 0x7f87200754a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:48:20.856 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:20.855+0000 7f8732d9d640 1 --2- 172.21.15.123:0/2245131349 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f8720072fe0 0x7f87200754a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:48:20.857 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:20.855+0000 7f871b7fe640 1 -- 172.21.15.123:0/2245131349 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 5029+0+0 (secure 0 0 0) 0x7f87280930b0 con 0x7f87340cc3c0 2024-09-16T15:48:20.857 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:20.855+0000 7f8732d9d640 1 --2- 172.21.15.123:0/2245131349 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f8720072fe0 0x7f87200754a0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f872400a860 tx=0x7f8724005b60 comp rx=0 tx=0).ready entity=mgr.4150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:48:20.860 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:20.859+0000 7f871b7fe640 1 -- 172.21.15.123:0/2245131349 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f872805e390 con 0x7f87340cc3c0 2024-09-16T15:48:21.020 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:21.019+0000 7f8739093640 1 -- 172.21.15.123:0/2245131349 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) -- 0x7f87340cd440 con 0x7f87340cc3c0 2024-09-16T15:48:21.024 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:21.023+0000 7f871b7fe640 1 -- 172.21.15.123:0/2245131349 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v38)=0 v38) ==== 122+0+0 (secure 0 0 0) 0x7f872805dc90 con 0x7f87340cc3c0 2024-09-16T15:48:21.027 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:21.027+0000 7f8739093640 1 -- 172.21.15.123:0/2245131349 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f8720072fe0 msgr2=0x7f87200754a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:48:21.027 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:21.027+0000 7f8739093640 1 --2- 172.21.15.123:0/2245131349 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f8720072fe0 0x7f87200754a0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f872400a860 tx=0x7f8724005b60 comp rx=0 tx=0).stop 2024-09-16T15:48:21.027 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:21.027+0000 7f8739093640 1 -- 172.21.15.123:0/2245131349 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f87340cc3c0 msgr2=0x7f87340c6c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:48:21.027 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:21.027+0000 7f8739093640 1 --2- 172.21.15.123:0/2245131349 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f87340cc3c0 0x7f87340c6c10 secure :-1 s=READY pgs=1036 cs=0 l=1 rev1=1 crypto rx=0x7f872800c500 tx=0x7f872800c9c0 comp rx=0 tx=0).stop 2024-09-16T15:48:21.027 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:21.027+0000 7f8739093640 1 -- 172.21.15.123:0/2245131349 shutdown_connections 2024-09-16T15:48:21.027 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:21.027+0000 7f8739093640 1 --2- 172.21.15.123:0/2245131349 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f8720072fe0 0x7f87200754a0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:21.027 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:21.027+0000 7f8739093640 1 --2- 172.21.15.123:0/2245131349 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f873415aa00 0x7f87341f9b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:21.027 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:21.027+0000 7f8739093640 1 --2- 172.21.15.123:0/2245131349 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f87340cc3c0 0x7f87340c6c10 unknown :-1 s=CLOSED pgs=1036 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:21.027 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:21.027+0000 7f8739093640 1 --2- 172.21.15.123:0/2245131349 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f8734072720 0x7f87340c5b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:21.027 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:21.027+0000 7f8739093640 1 -- 172.21.15.123:0/2245131349 >> 172.21.15.123:0/2245131349 conn(0x7f87340b1710 msgr2=0x7f87340b0130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:48:21.027 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:21.027+0000 7f8739093640 1 -- 172.21.15.123:0/2245131349 shutdown_connections 2024-09-16T15:48:21.028 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:21.027+0000 7f8739093640 1 -- 172.21.15.123:0/2245131349 wait complete. 2024-09-16T15:48:21.042 INFO:teuthology.nuke:Clearing teuthology firewall rules... 2024-09-16T15:48:21.042 DEBUG:teuthology.orchestra.run.smithi123:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-09-16T15:48:21.119 DEBUG:teuthology.orchestra.run.smithi184:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-09-16T15:48:21.195 INFO:teuthology.nuke:Cleared teuthology firewall rules. 2024-09-16T15:48:21.196 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2024-09-16T15:48:21.196 DEBUG:teuthology.orchestra.run.smithi123:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-16T15:48:21.202 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16336468k,nr_inodes=4084117,mode=755,inode64 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3274684k,mode=755,inode64 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev,inode64 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=17082 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3274680k,nr_inodes=818670,mode=700,uid=1000,gid=1271,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3274684k,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-09-16T15:48:21.202 DEBUG:teuthology.orchestra.run.smithi184:> set -ex 2024-09-16T15:48:21.202 DEBUG:teuthology.orchestra.run.smithi184:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-16T15:48:21.207 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16336488k,nr_inodes=4084122,mode=755,inode64 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3274688k,mode=755,inode64 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev,inode64 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=17834 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3274684k,nr_inodes=818671,mode=700,uid=1000,gid=1271,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3274688k,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-09-16T15:48:21.207 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2024-09-16T15:48:21.207 DEBUG:teuthology.orchestra.run.smithi123:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-16T15:48:21.250 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16336468k,nr_inodes=4084117,mode=755,inode64 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3274684k,mode=755,inode64 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev,inode64 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=17082 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3274680k,nr_inodes=818670,mode=700,uid=1000,gid=1271,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3274684k,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-09-16T15:48:21.250 DEBUG:teuthology.orchestra.run.smithi184:> set -ex 2024-09-16T15:48:21.250 DEBUG:teuthology.orchestra.run.smithi184:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-16T15:48:21.256 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16336488k,nr_inodes=4084122,mode=755,inode64 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3274688k,mode=755,inode64 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev,inode64 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=17834 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3274684k,nr_inodes=818671,mode=700,uid=1000,gid=1271,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3274688k,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-09-16T15:48:21.256 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-09-16T15:48:21.256 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-16T15:48:21.423 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:21.423+0000 7f5f2dcbe640 1 -- 172.21.15.123:0/1525301517 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5f2815bca0 msgr2=0x7f5f280d0630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:48:21.424 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:21.423+0000 7f5f2dcbe640 1 --2- 172.21.15.123:0/1525301517 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5f2815bca0 0x7f5f280d0630 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f5f18009900 tx=0x7f5f1801c820 comp rx=0 tx=0).stop 2024-09-16T15:48:21.424 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:21.423+0000 7f5f2dcbe640 1 -- 172.21.15.123:0/1525301517 shutdown_connections 2024-09-16T15:48:21.424 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:21.423+0000 7f5f2dcbe640 1 --2- 172.21.15.123:0/1525301517 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f5f280726c0 0x7f5f28073ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:21.424 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:21.423+0000 7f5f2dcbe640 1 --2- 172.21.15.123:0/1525301517 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5f2815bca0 0x7f5f280d0630 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:21.424 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:21.423+0000 7f5f2dcbe640 1 -- 172.21.15.123:0/1525301517 >> 172.21.15.123:0/1525301517 conn(0x7f5f280cfb10 msgr2=0x7f5f280cff20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:48:21.424 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:21.423+0000 7f5f2dcbe640 1 -- 172.21.15.123:0/1525301517 shutdown_connections 2024-09-16T15:48:21.424 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:21.423+0000 7f5f2dcbe640 1 -- 172.21.15.123:0/1525301517 wait complete. 2024-09-16T15:48:21.425 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:21.423+0000 7f5f2dcbe640 1 Processor -- start 2024-09-16T15:48:21.425 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:21.423+0000 7f5f2dcbe640 1 -- start start 2024-09-16T15:48:21.427 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:21.427+0000 7f5f2dcbe640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5f280726c0 0x7f5f280c9090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:48:21.427 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:21.427+0000 7f5f2dcbe640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5f28074430 0x7f5f280c50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:48:21.427 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:21.427+0000 7f5f2dcbe640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f5f2815bca0 0x7f5f280c5620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:48:21.427 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:21.427+0000 7f5f2dcbe640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f5f28168e20 con 0x7f5f28074430 2024-09-16T15:48:21.427 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:21.427+0000 7f5f2dcbe640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f5f28168ca0 con 0x7f5f280726c0 2024-09-16T15:48:21.427 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:21.427+0000 7f5f2dcbe640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f5f28168fa0 con 0x7f5f2815bca0 2024-09-16T15:48:21.427 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:21.427+0000 7f5f26ffd640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5f28074430 0x7f5f280c50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:48:21.427 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:21.427+0000 7f5f26ffd640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5f28074430 0x7f5f280c50e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:41672/0 (socket says 172.21.15.123:41672) 2024-09-16T15:48:21.427 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:21.427+0000 7f5f26ffd640 1 -- 172.21.15.123:0/804208500 learned_addr learned my addr 172.21.15.123:0/804208500 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:48:21.428 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:21.427+0000 7f5f277fe640 1 --2- 172.21.15.123:0/804208500 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5f280726c0 0x7f5f280c9090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:48:21.428 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:21.427+0000 7f5f27fff640 1 --2- 172.21.15.123:0/804208500 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f5f2815bca0 0x7f5f280c5620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:48:21.428 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:21.427+0000 7f5f26ffd640 1 -- 172.21.15.123:0/804208500 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f5f2815bca0 msgr2=0x7f5f280c5620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:48:21.428 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:21.427+0000 7f5f26ffd640 1 --2- 172.21.15.123:0/804208500 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f5f2815bca0 0x7f5f280c5620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:21.428 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:21.427+0000 7f5f26ffd640 1 -- 172.21.15.123:0/804208500 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5f280726c0 msgr2=0x7f5f280c9090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:48:21.428 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:21.427+0000 7f5f26ffd640 1 --2- 172.21.15.123:0/804208500 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5f280726c0 0x7f5f280c9090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:21.428 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:21.427+0000 7f5f26ffd640 1 -- 172.21.15.123:0/804208500 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5f28159360 con 0x7f5f28074430 2024-09-16T15:48:21.428 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:21.427+0000 7f5f26ffd640 1 --2- 172.21.15.123:0/804208500 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5f28074430 0x7f5f280c50e0 secure :-1 s=READY pgs=1037 cs=0 l=1 rev1=1 crypto rx=0x7f5f1c00e9f0 tx=0x7f5f1c00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:48:21.428 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:21.427+0000 7f5f0ffff640 1 -- 172.21.15.123:0/804208500 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f1c00cd80 con 0x7f5f28074430 2024-09-16T15:48:21.428 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:21.427+0000 7f5f0ffff640 1 -- 172.21.15.123:0/804208500 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f5f1c010430 con 0x7f5f28074430 2024-09-16T15:48:21.428 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:21.427+0000 7f5f0ffff640 1 -- 172.21.15.123:0/804208500 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f1c010860 con 0x7f5f28074430 2024-09-16T15:48:21.428 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:21.427+0000 7f5f2dcbe640 1 -- 172.21.15.123:0/804208500 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5f2815d0a0 con 0x7f5f28074430 2024-09-16T15:48:21.429 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:21.427+0000 7f5f2dcbe640 1 -- 172.21.15.123:0/804208500 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5f281598e0 con 0x7f5f28074430 2024-09-16T15:48:21.429 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:21.427+0000 7f5f2dcbe640 1 -- 172.21.15.123:0/804208500 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5f28073ef0 con 0x7f5f28074430 2024-09-16T15:48:21.429 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:21.427+0000 7f5f0ffff640 1 -- 172.21.15.123:0/804208500 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 16) ==== 97182+0+0 (secure 0 0 0) 0x7f5f1c010a00 con 0x7f5f28074430 2024-09-16T15:48:21.429 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:21.427+0000 7f5f0ffff640 1 --2- 172.21.15.123:0/804208500 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f5f14072fe0 0x7f5f140754a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:48:21.430 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:21.427+0000 7f5f0ffff640 1 -- 172.21.15.123:0/804208500 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 5029+0+0 (secure 0 0 0) 0x7f5f1c095130 con 0x7f5f28074430 2024-09-16T15:48:21.430 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:21.427+0000 7f5f277fe640 1 --2- 172.21.15.123:0/804208500 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f5f14072fe0 0x7f5f140754a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:48:21.430 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:21.431+0000 7f5f277fe640 1 --2- 172.21.15.123:0/804208500 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f5f14072fe0 0x7f5f140754a0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f5f180af000 tx=0x7f5f1801b060 comp rx=0 tx=0).ready entity=mgr.4150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:48:21.432 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:21.431+0000 7f5f0ffff640 1 -- 172.21.15.123:0/804208500 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f5f1c0605a0 con 0x7f5f28074430 2024-09-16T15:48:21.640 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:21.639+0000 7f5f2dcbe640 1 -- 172.21.15.123:0/804208500 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f5f280d08a0 con 0x7f5f28074430 2024-09-16T15:48:21.641 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:21.639+0000 7f5f0ffff640 1 -- 172.21.15.123:0/804208500 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 61 v61) ==== 94+0+3914 (secure 0 0 0) 0x7f5f1c05fea0 con 0x7f5f28074430 2024-09-16T15:48:21.641 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-09-16T15:48:21.641 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":61,"btime":"2024-09-16T15:48:15:059900+0000","default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5369,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6834/355272346","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":355272346},{"type":"v1","addr":"172.21.15.184:6834","nonce":355272346}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":61},{"gid":5677,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6835/1609511477","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6833","nonce":1609511477},{"type":"v1","addr":"172.21.15.184:6835","nonce":1609511477}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":48},{"gid":6672,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6834/3658721718","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6832","nonce":3658721718},{"type":"v1","addr":"172.21.15.123:6834","nonce":3658721718}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":48},{"gid":6891,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6835/95665084","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6833","nonce":95665084},{"type":"v1","addr":"172.21.15.123:6835","nonce":95665084}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":59}],"filesystems":[]} 2024-09-16T15:48:21.642 INFO:teuthology.orchestra.run.smithi123.stderr:dumped fsmap epoch 61 2024-09-16T15:48:21.644 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:21.643+0000 7f5f2dcbe640 1 -- 172.21.15.123:0/804208500 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f5f14072fe0 msgr2=0x7f5f140754a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:48:21.644 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:21.643+0000 7f5f2dcbe640 1 --2- 172.21.15.123:0/804208500 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f5f14072fe0 0x7f5f140754a0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f5f180af000 tx=0x7f5f1801b060 comp rx=0 tx=0).stop 2024-09-16T15:48:21.644 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:21.643+0000 7f5f2dcbe640 1 -- 172.21.15.123:0/804208500 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5f28074430 msgr2=0x7f5f280c50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:48:21.644 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:21.643+0000 7f5f2dcbe640 1 --2- 172.21.15.123:0/804208500 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5f28074430 0x7f5f280c50e0 secure :-1 s=READY pgs=1037 cs=0 l=1 rev1=1 crypto rx=0x7f5f1c00e9f0 tx=0x7f5f1c00eeb0 comp rx=0 tx=0).stop 2024-09-16T15:48:21.644 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:21.643+0000 7f5f2dcbe640 1 -- 172.21.15.123:0/804208500 shutdown_connections 2024-09-16T15:48:21.644 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:21.643+0000 7f5f2dcbe640 1 --2- 172.21.15.123:0/804208500 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f5f14072fe0 0x7f5f140754a0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:21.645 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:21.643+0000 7f5f2dcbe640 1 --2- 172.21.15.123:0/804208500 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f5f2815bca0 0x7f5f280c5620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:21.645 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:21.643+0000 7f5f2dcbe640 1 --2- 172.21.15.123:0/804208500 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5f28074430 0x7f5f280c50e0 unknown :-1 s=CLOSED pgs=1037 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:21.645 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:21.643+0000 7f5f2dcbe640 1 --2- 172.21.15.123:0/804208500 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5f280726c0 0x7f5f280c9090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:21.645 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:21.643+0000 7f5f2dcbe640 1 -- 172.21.15.123:0/804208500 >> 172.21.15.123:0/804208500 conn(0x7f5f280cfb10 msgr2=0x7f5f280ca140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:48:21.645 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:21.643+0000 7f5f2dcbe640 1 -- 172.21.15.123:0/804208500 shutdown_connections 2024-09-16T15:48:21.645 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:21.643+0000 7f5f2dcbe640 1 -- 172.21.15.123:0/804208500 wait complete. 2024-09-16T15:48:21.660 INFO:tasks.ceph.mds.a:Restarting daemon 2024-09-16T15:48:21.660 INFO:tasks.ceph.mds.a:Stopping old one... 2024-09-16T15:48:21.660 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-09-16T15:48:21.660 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T15:48:21.660 INFO:tasks.ceph.mds.c:Restarting daemon 2024-09-16T15:48:21.661 INFO:tasks.ceph.mds.c:Stopping old one... 2024-09-16T15:48:21.661 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-09-16T15:48:21.661 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T15:48:21.661 INFO:tasks.ceph.mds.b:Restarting daemon 2024-09-16T15:48:21.661 INFO:tasks.ceph.mds.b:Stopping old one... 2024-09-16T15:48:21.661 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-09-16T15:48:21.661 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T15:48:21.662 INFO:tasks.ceph.mds.d:Restarting daemon 2024-09-16T15:48:21.662 INFO:tasks.ceph.mds.d:Stopping old one... 2024-09-16T15:48:21.662 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-09-16T15:48:21.662 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T15:48:21.670 INFO:tasks.ceph.mds.a:Stopped 2024-09-16T15:48:21.670 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-09-16T15:48:21.671 INFO:tasks.ceph.mds.c:Stopped 2024-09-16T15:48:21.671 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-09-16T15:48:21.676 INFO:tasks.ceph.mds.d:Stopped 2024-09-16T15:48:21.676 DEBUG:teuthology.orchestra.run.smithi184:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-09-16T15:48:21.677 INFO:tasks.ceph.mds.b:Stopped 2024-09-16T15:48:21.677 DEBUG:teuthology.orchestra.run.smithi184:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-09-16T15:48:21.719 INFO:tasks.ceph.mds.a:Started 2024-09-16T15:48:21.719 INFO:tasks.ceph.mds.d:Started 2024-09-16T15:48:21.720 DEBUG:teuthology.parallel:result is None 2024-09-16T15:48:21.720 DEBUG:teuthology.parallel:result is None 2024-09-16T15:48:21.720 INFO:tasks.ceph.mds.c:Started 2024-09-16T15:48:21.720 DEBUG:teuthology.parallel:result is None 2024-09-16T15:48:21.721 INFO:tasks.ceph.mds.b:Started 2024-09-16T15:48:21.721 DEBUG:teuthology.parallel:result is None 2024-09-16T15:48:21.721 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-09-16T15:48:21.778 INFO:tasks.ceph.mds.a.smithi123.stdout:starting mds.a at 2024-09-16T15:48:21.782 INFO:tasks.ceph.mds.c.smithi123.stdout:starting mds.c at 2024-09-16T15:48:21.788 INFO:tasks.ceph.mds.d.smithi184.stdout:starting mds.d at 2024-09-16T15:48:21.795 INFO:tasks.ceph.mds.b.smithi184.stdout:starting mds.b at 2024-09-16T15:48:21.837 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:21.835+0000 7fb84745a640 1 -- 172.21.15.123:0/4154960650 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb82c004d50 msgr2=0x7fb82c0051d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:48:21.837 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:21.835+0000 7fb84745a640 1 --2- 172.21.15.123:0/4154960650 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb82c004d50 0x7fb82c0051d0 secure :-1 s=READY pgs=1044 cs=0 l=1 rev1=1 crypto rx=0x7fb83800c640 tx=0x7fb83801d200 comp rx=0 tx=0).stop 2024-09-16T15:48:21.837 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:21.835+0000 7fb8459f7640 1 -- 172.21.15.123:0/4154960650 reap_dead start 2024-09-16T15:48:21.838 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:21.835+0000 7fb84745a640 1 -- 172.21.15.123:0/4154960650 shutdown_connections 2024-09-16T15:48:21.838 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:21.835+0000 7fb84745a640 1 -- 172.21.15.123:0/4154960650 >> 172.21.15.123:0/4154960650 conn(0x7fb8400d0b40 msgr2=0x7fb8400d0f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:48:21.838 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:21.835+0000 7fb84745a640 1 -- 172.21.15.123:0/4154960650 shutdown_connections 2024-09-16T15:48:21.838 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:21.835+0000 7fb84745a640 1 -- 172.21.15.123:0/4154960650 wait complete. 2024-09-16T15:48:21.838 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:21.839+0000 7fb84745a640 1 Processor -- start 2024-09-16T15:48:21.838 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:21.839+0000 7fb84745a640 1 -- start start 2024-09-16T15:48:21.840 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:21.839+0000 7fb84745a640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb82c004d50 0x7fb840165110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:48:21.841 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:21.839+0000 7fb84745a640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fb840151550 0x7fb840143430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:48:21.841 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:21.839+0000 7fb84745a640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb84015f940 0x7fb84015fdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:48:21.841 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:21.839+0000 7fb84745a640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fb840168330 con 0x7fb84015f940 2024-09-16T15:48:21.841 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:21.839+0000 7fb84745a640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fb8400d08c0 con 0x7fb82c004d50 2024-09-16T15:48:21.841 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:21.839+0000 7fb84745a640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fb8401684b0 con 0x7fb840151550 2024-09-16T15:48:21.841 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:21.839+0000 7fb8459f7640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb84015f940 0x7fb84015fdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:48:21.841 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:21.839+0000 7fb8459f7640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb84015f940 0x7fb84015fdc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:41708/0 (socket says 172.21.15.123:41708) 2024-09-16T15:48:21.841 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:21.839+0000 7fb8459f7640 1 -- 172.21.15.123:0/3132038832 learned_addr learned my addr 172.21.15.123:0/3132038832 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:48:21.841 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:21.839+0000 7fb8449f5640 1 --2- 172.21.15.123:0/3132038832 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fb840151550 0x7fb840143430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:48:21.841 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:21.839+0000 7fb8451f6640 1 --2- 172.21.15.123:0/3132038832 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb82c004d50 0x7fb840165110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:48:21.841 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:21.839+0000 7fb8459f7640 1 -- 172.21.15.123:0/3132038832 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fb840151550 msgr2=0x7fb840143430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:48:21.842 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:21.839+0000 7fb8459f7640 1 --2- 172.21.15.123:0/3132038832 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fb840151550 0x7fb840143430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:21.842 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:21.839+0000 7fb8459f7640 1 -- 172.21.15.123:0/3132038832 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb82c004d50 msgr2=0x7fb840165110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:48:21.842 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:21.839+0000 7fb8459f7640 1 --2- 172.21.15.123:0/3132038832 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb82c004d50 0x7fb840165110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:21.842 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:21.839+0000 7fb8459f7640 1 -- 172.21.15.123:0/3132038832 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb840165650 con 0x7fb84015f940 2024-09-16T15:48:21.842 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:21.839+0000 7fb8451f6640 1 --2- 172.21.15.123:0/3132038832 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb82c004d50 0x7fb840165110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T15:48:21.842 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:21.839+0000 7fb8459f7640 1 --2- 172.21.15.123:0/3132038832 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb84015f940 0x7fb84015fdc0 secure :-1 s=READY pgs=1045 cs=0 l=1 rev1=1 crypto rx=0x7fb83801b900 tx=0x7fb838006620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:48:21.842 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:21.839+0000 7fb835ffb640 1 -- 172.21.15.123:0/3132038832 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb838004e60 con 0x7fb84015f940 2024-09-16T15:48:21.842 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:21.839+0000 7fb835ffb640 1 -- 172.21.15.123:0/3132038832 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fb8380018a0 con 0x7fb84015f940 2024-09-16T15:48:21.842 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:21.839+0000 7fb84745a640 1 -- 172.21.15.123:0/3132038832 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb840164070 con 0x7fb84015f940 2024-09-16T15:48:21.842 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:21.839+0000 7fb835ffb640 1 -- 172.21.15.123:0/3132038832 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb838001fc0 con 0x7fb84015f940 2024-09-16T15:48:21.842 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:21.839+0000 7fb84745a640 1 -- 172.21.15.123:0/3132038832 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb840164630 con 0x7fb84015f940 2024-09-16T15:48:21.842 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:21.839+0000 7fb84745a640 1 -- 172.21.15.123:0/3132038832 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb8400cfd60 con 0x7fb84015f940 2024-09-16T15:48:21.843 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:21.843+0000 7fb835ffb640 1 -- 172.21.15.123:0/3132038832 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 16) ==== 97182+0+0 (secure 0 0 0) 0x7fb8380c0060 con 0x7fb84015f940 2024-09-16T15:48:21.843 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:21.843+0000 7fb835ffb640 1 --2- 172.21.15.123:0/3132038832 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fb824073010 0x7fb8240754d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:48:21.843 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:21.843+0000 7fb8451f6640 1 --2- 172.21.15.123:0/3132038832 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fb824073010 0x7fb8240754d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:48:21.843 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:21.843+0000 7fb835ffb640 1 -- 172.21.15.123:0/3132038832 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 5029+0+0 (secure 0 0 0) 0x7fb83812ea40 con 0x7fb84015f940 2024-09-16T15:48:21.843 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:21.843+0000 7fb8451f6640 1 --2- 172.21.15.123:0/3132038832 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fb824073010 0x7fb8240754d0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fb83c009c60 tx=0x7fb83c0091e0 comp rx=0 tx=0).ready entity=mgr.4150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:48:21.845 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:21.843+0000 7fb835ffb640 1 -- 172.21.15.123:0/3132038832 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7fb8380c7020 con 0x7fb84015f940 2024-09-16T15:48:22.015 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:22.015+0000 7fb84745a640 1 -- 172.21.15.123:0/3132038832 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) -- 0x7fb84015bf70 con 0x7fb84015f940 2024-09-16T15:48:22.402 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:22.403+0000 7fb835ffb640 1 -- 172.21.15.123:0/3132038832 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v97) ==== 97+0+0 (secure 0 0 0) 0x7fb838016fb0 con 0x7fb84015f940 2024-09-16T15:48:22.402 INFO:teuthology.orchestra.run.smithi123.stderr: removed all blocklist entries 2024-09-16T15:48:22.405 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:22.403+0000 7fb84745a640 1 -- 172.21.15.123:0/3132038832 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fb824073010 msgr2=0x7fb8240754d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:48:22.405 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:22.403+0000 7fb84745a640 1 --2- 172.21.15.123:0/3132038832 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fb824073010 0x7fb8240754d0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fb83c009c60 tx=0x7fb83c0091e0 comp rx=0 tx=0).stop 2024-09-16T15:48:22.405 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:22.403+0000 7fb84745a640 1 -- 172.21.15.123:0/3132038832 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb84015f940 msgr2=0x7fb84015fdc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:48:22.405 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:22.403+0000 7fb84745a640 1 --2- 172.21.15.123:0/3132038832 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb84015f940 0x7fb84015fdc0 secure :-1 s=READY pgs=1045 cs=0 l=1 rev1=1 crypto rx=0x7fb83801b900 tx=0x7fb838006620 comp rx=0 tx=0).stop 2024-09-16T15:48:22.405 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:22.403+0000 7fb84745a640 1 -- 172.21.15.123:0/3132038832 shutdown_connections 2024-09-16T15:48:22.406 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:22.403+0000 7fb84745a640 1 --2- 172.21.15.123:0/3132038832 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fb824073010 0x7fb8240754d0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:22.406 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:22.403+0000 7fb84745a640 1 --2- 172.21.15.123:0/3132038832 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb84015f940 0x7fb84015fdc0 unknown :-1 s=CLOSED pgs=1045 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:22.406 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:22.403+0000 7fb84745a640 1 --2- 172.21.15.123:0/3132038832 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fb840151550 0x7fb840143430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:22.406 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:22.403+0000 7fb84745a640 1 --2- 172.21.15.123:0/3132038832 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb82c004d50 0x7fb840165110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:22.406 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:22.403+0000 7fb84745a640 1 -- 172.21.15.123:0/3132038832 >> 172.21.15.123:0/3132038832 conn(0x7fb8400d0b40 msgr2=0x7fb84015d9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:48:22.406 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:22.403+0000 7fb84745a640 1 -- 172.21.15.123:0/3132038832 shutdown_connections 2024-09-16T15:48:22.406 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:22.403+0000 7fb84745a640 1 -- 172.21.15.123:0/3132038832 wait complete. 2024-09-16T15:48:22.429 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-09-16T15:48:22.556 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:22.555+0000 7f62354e3640 1 -- 172.21.15.123:0/3956589617 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f62180053d0 msgr2=0x7f621800b9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:48:22.557 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:22.555+0000 7f62354e3640 1 --2- 172.21.15.123:0/3956589617 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f62180053d0 0x7f621800b9d0 secure :-1 s=READY pgs=1047 cs=0 l=1 rev1=1 crypto rx=0x7f6228009eb0 tx=0x7f622801d5c0 comp rx=0 tx=0).stop 2024-09-16T15:48:22.557 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:22.555+0000 7f62354e3640 1 -- 172.21.15.123:0/3956589617 shutdown_connections 2024-09-16T15:48:22.557 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:22.555+0000 7f62354e3640 1 --2- 172.21.15.123:0/3956589617 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f62180053d0 0x7f621800b9d0 unknown :-1 s=CLOSED pgs=1047 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:22.557 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:22.555+0000 7f62354e3640 1 -- 172.21.15.123:0/3956589617 >> 172.21.15.123:0/3956589617 conn(0x7f62300d3ea0 msgr2=0x7f62300d42b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:48:22.557 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:22.555+0000 7f62354e3640 1 -- 172.21.15.123:0/3956589617 shutdown_connections 2024-09-16T15:48:22.557 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:22.555+0000 7f62354e3640 1 -- 172.21.15.123:0/3956589617 wait complete. 2024-09-16T15:48:22.559 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:22.555+0000 7f62354e3640 1 Processor -- start 2024-09-16T15:48:22.559 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:22.555+0000 7f62354e3640 1 -- start start 2024-09-16T15:48:22.559 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:22.559+0000 7f62354e3640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f62180053d0 0x7f62300c9140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:48:22.559 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:22.559+0000 7f62354e3640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f623015d820 0x7f6230156430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:48:22.560 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:22.559+0000 7f62354e3640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f62301581d0 0x7f62300b3540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:48:22.560 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:22.559+0000 7f62354e3640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f62300d2130 con 0x7f62301581d0 2024-09-16T15:48:22.560 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:22.559+0000 7f62354e3640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f62300d1fb0 con 0x7f62180053d0 2024-09-16T15:48:22.560 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:22.559+0000 7f62354e3640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f62300d22b0 con 0x7f623015d820 2024-09-16T15:48:22.560 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:22.559+0000 7f622f7fe640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f62301581d0 0x7f62300b3540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:48:22.560 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:22.559+0000 7f622f7fe640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f62301581d0 0x7f62300b3540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:47294/0 (socket says 172.21.15.123:47294) 2024-09-16T15:48:22.560 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:22.559+0000 7f622f7fe640 1 -- 172.21.15.123:0/123632491 learned_addr learned my addr 172.21.15.123:0/123632491 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:48:22.560 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:22.559+0000 7f622effd640 1 --2- 172.21.15.123:0/123632491 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f62180053d0 0x7f62300c9140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:48:22.560 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:22.559+0000 7f622e7fc640 1 --2- 172.21.15.123:0/123632491 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f623015d820 0x7f6230156430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:48:22.560 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:22.559+0000 7f622f7fe640 1 -- 172.21.15.123:0/123632491 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f623015d820 msgr2=0x7f6230156430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:48:22.560 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:22.559+0000 7f622f7fe640 1 --2- 172.21.15.123:0/123632491 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f623015d820 0x7f6230156430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:22.560 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:22.559+0000 7f622f7fe640 1 -- 172.21.15.123:0/123632491 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f62180053d0 msgr2=0x7f62300c9140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:48:22.561 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:22.559+0000 7f622f7fe640 1 --2- 172.21.15.123:0/123632491 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f62180053d0 0x7f62300c9140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:22.561 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:22.559+0000 7f622f7fe640 1 -- 172.21.15.123:0/123632491 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f623015dda0 con 0x7f62301581d0 2024-09-16T15:48:22.561 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:22.559+0000 7f622effd640 1 --2- 172.21.15.123:0/123632491 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f62180053d0 0x7f62300c9140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T15:48:22.561 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:22.559+0000 7f622f7fe640 1 --2- 172.21.15.123:0/123632491 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f62301581d0 0x7f62300b3540 secure :-1 s=READY pgs=1048 cs=0 l=1 rev1=1 crypto rx=0x7f6228009eb0 tx=0x7f6228009280 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:48:22.561 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:22.559+0000 7f62177fe640 1 -- 172.21.15.123:0/123632491 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62280ae030 con 0x7f62301581d0 2024-09-16T15:48:22.561 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:22.559+0000 7f62177fe640 1 -- 172.21.15.123:0/123632491 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f62280b9070 con 0x7f62301581d0 2024-09-16T15:48:22.561 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:22.559+0000 7f62354e3640 1 -- 172.21.15.123:0/123632491 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f623015e080 con 0x7f62301581d0 2024-09-16T15:48:22.561 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:22.559+0000 7f62177fe640 1 -- 172.21.15.123:0/123632491 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6228004c90 con 0x7f62301581d0 2024-09-16T15:48:22.561 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:22.559+0000 7f62354e3640 1 -- 172.21.15.123:0/123632491 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f623015e640 con 0x7f62301581d0 2024-09-16T15:48:22.562 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:22.559+0000 7f62354e3640 1 -- 172.21.15.123:0/123632491 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f62300d27a0 con 0x7f62301581d0 2024-09-16T15:48:22.562 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:22.559+0000 7f62177fe640 1 -- 172.21.15.123:0/123632491 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 16) ==== 97182+0+0 (secure 0 0 0) 0x7f6228004020 con 0x7f62301581d0 2024-09-16T15:48:22.562 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:22.559+0000 7f62177fe640 1 --2- 172.21.15.123:0/123632491 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f621c072fe0 0x7f621c0754a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:48:22.562 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:22.559+0000 7f622effd640 1 --2- 172.21.15.123:0/123632491 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f621c072fe0 0x7f621c0754a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:48:22.562 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:22.559+0000 7f62177fe640 1 -- 172.21.15.123:0/123632491 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 4857+0+0 (secure 0 0 0) 0x7f622812f360 con 0x7f62301581d0 2024-09-16T15:48:22.562 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:22.563+0000 7f622effd640 1 --2- 172.21.15.123:0/123632491 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f621c072fe0 0x7f621c0754a0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f6220001ec0 tx=0x7f6220009040 comp rx=0 tx=0).ready entity=mgr.4150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:48:22.565 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:22.563+0000 7f62177fe640 1 -- 172.21.15.123:0/123632491 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f6228017300 con 0x7f62301581d0 2024-09-16T15:48:22.781 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:22.779+0000 7f62354e3640 1 -- 172.21.15.123:0/123632491 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) -- 0x7f62300b59b0 con 0x7f62301581d0 2024-09-16T15:48:22.781 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:22.779+0000 7f62177fe640 1 -- 172.21.15.123:0/123632491 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v23) ==== 80+0+5304 (secure 0 0 0) 0x7f62280fa7c0 con 0x7f62301581d0 2024-09-16T15:48:22.781 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-09-16T15:48:22.782 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2024-09-16T15:48:22.782 INFO:teuthology.orchestra.run.smithi123.stdout: "auth_dump": [ 2024-09-16T15:48:22.782 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-09-16T15:48:22.782 INFO:teuthology.orchestra.run.smithi123.stdout: "entity": "mds.a", 2024-09-16T15:48:22.782 INFO:teuthology.orchestra.run.smithi123.stdout: "key": "AQC0UehmbZpOJBAAaYGyDcWrt4k0FS5P/bb+4A==", 2024-09-16T15:48:22.782 INFO:teuthology.orchestra.run.smithi123.stdout: "caps": { 2024-09-16T15:48:22.782 INFO:teuthology.orchestra.run.smithi123.stdout: "mds": "allow", 2024-09-16T15:48:22.782 INFO:teuthology.orchestra.run.smithi123.stdout: "mgr": "allow *", 2024-09-16T15:48:22.782 INFO:teuthology.orchestra.run.smithi123.stdout: "mon": "allow *", 2024-09-16T15:48:22.782 INFO:teuthology.orchestra.run.smithi123.stdout: "osd": "allow *" 2024-09-16T15:48:22.782 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-16T15:48:22.782 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:48:22.782 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-09-16T15:48:22.782 INFO:teuthology.orchestra.run.smithi123.stdout: "entity": "mds.b", 2024-09-16T15:48:22.783 INFO:teuthology.orchestra.run.smithi123.stdout: "key": "AQC0UehmF1HFMhAADcvqXNKdHZMPAbLc+se4Aw==", 2024-09-16T15:48:22.783 INFO:teuthology.orchestra.run.smithi123.stdout: "caps": { 2024-09-16T15:48:22.783 INFO:teuthology.orchestra.run.smithi123.stdout: "mds": "allow", 2024-09-16T15:48:22.783 INFO:teuthology.orchestra.run.smithi123.stdout: "mgr": "allow *", 2024-09-16T15:48:22.783 INFO:teuthology.orchestra.run.smithi123.stdout: "mon": "allow *", 2024-09-16T15:48:22.783 INFO:teuthology.orchestra.run.smithi123.stdout: "osd": "allow *" 2024-09-16T15:48:22.783 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-16T15:48:22.783 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:48:22.783 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-09-16T15:48:22.783 INFO:teuthology.orchestra.run.smithi123.stdout: "entity": "mds.c", 2024-09-16T15:48:22.783 INFO:teuthology.orchestra.run.smithi123.stdout: "key": "AQC0UehmCBjCLBAAXd85J25r7cgrXeDCG7pahg==", 2024-09-16T15:48:22.783 INFO:teuthology.orchestra.run.smithi123.stdout: "caps": { 2024-09-16T15:48:22.783 INFO:teuthology.orchestra.run.smithi123.stdout: "mds": "allow", 2024-09-16T15:48:22.783 INFO:teuthology.orchestra.run.smithi123.stdout: "mgr": "allow *", 2024-09-16T15:48:22.783 INFO:teuthology.orchestra.run.smithi123.stdout: "mon": "allow *", 2024-09-16T15:48:22.784 INFO:teuthology.orchestra.run.smithi123.stdout: "osd": "allow *" 2024-09-16T15:48:22.784 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-16T15:48:22.784 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:48:22.784 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-09-16T15:48:22.784 INFO:teuthology.orchestra.run.smithi123.stdout: "entity": "mds.d", 2024-09-16T15:48:22.784 INFO:teuthology.orchestra.run.smithi123.stdout: "key": "AQC0UehmMh3ZOBAAx0RCa3R9RMT93/icZhPGyg==", 2024-09-16T15:48:22.784 INFO:teuthology.orchestra.run.smithi123.stdout: "caps": { 2024-09-16T15:48:22.784 INFO:teuthology.orchestra.run.smithi123.stdout: "mds": "allow", 2024-09-16T15:48:22.784 INFO:teuthology.orchestra.run.smithi123.stdout: "mgr": "allow *", 2024-09-16T15:48:22.784 INFO:teuthology.orchestra.run.smithi123.stdout: "mon": "allow *", 2024-09-16T15:48:22.784 INFO:teuthology.orchestra.run.smithi123.stdout: "osd": "allow *" 2024-09-16T15:48:22.784 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-16T15:48:22.784 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:48:22.784 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-09-16T15:48:22.785 INFO:teuthology.orchestra.run.smithi123.stdout: "entity": "osd.0", 2024-09-16T15:48:22.785 INFO:teuthology.orchestra.run.smithi123.stdout: "key": "AQC4UehmT633CRAAIhpkeIBOzwJO0kiXLu0x9Q==", 2024-09-16T15:48:22.785 INFO:teuthology.orchestra.run.smithi123.stdout: "caps": { 2024-09-16T15:48:22.785 INFO:teuthology.orchestra.run.smithi123.stdout: "mgr": "allow profile osd", 2024-09-16T15:48:22.785 INFO:teuthology.orchestra.run.smithi123.stdout: "mon": "allow profile osd", 2024-09-16T15:48:22.785 INFO:teuthology.orchestra.run.smithi123.stdout: "osd": "allow *" 2024-09-16T15:48:22.785 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-16T15:48:22.785 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:48:22.785 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-09-16T15:48:22.785 INFO:teuthology.orchestra.run.smithi123.stdout: "entity": "osd.1", 2024-09-16T15:48:22.785 INFO:teuthology.orchestra.run.smithi123.stdout: "key": "AQC7UehmFwF7GBAAvOUG+rWjANKT8wm/ai4xGg==", 2024-09-16T15:48:22.785 INFO:teuthology.orchestra.run.smithi123.stdout: "caps": { 2024-09-16T15:48:22.785 INFO:teuthology.orchestra.run.smithi123.stdout: "mgr": "allow profile osd", 2024-09-16T15:48:22.786 INFO:teuthology.orchestra.run.smithi123.stdout: "mon": "allow profile osd", 2024-09-16T15:48:22.786 INFO:teuthology.orchestra.run.smithi123.stdout: "osd": "allow *" 2024-09-16T15:48:22.786 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-16T15:48:22.786 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:48:22.786 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-09-16T15:48:22.786 INFO:teuthology.orchestra.run.smithi123.stdout: "entity": "osd.2", 2024-09-16T15:48:22.786 INFO:teuthology.orchestra.run.smithi123.stdout: "key": "AQC+UehmlCwWKxAAl3iO7ZALWyaorQH2urHKCw==", 2024-09-16T15:48:22.786 INFO:teuthology.orchestra.run.smithi123.stdout: "caps": { 2024-09-16T15:48:22.786 INFO:teuthology.orchestra.run.smithi123.stdout: "mgr": "allow profile osd", 2024-09-16T15:48:22.786 INFO:teuthology.orchestra.run.smithi123.stdout: "mon": "allow profile osd", 2024-09-16T15:48:22.786 INFO:teuthology.orchestra.run.smithi123.stdout: "osd": "allow *" 2024-09-16T15:48:22.786 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-16T15:48:22.786 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:48:22.786 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-09-16T15:48:22.786 INFO:teuthology.orchestra.run.smithi123.stdout: "entity": "osd.3", 2024-09-16T15:48:22.787 INFO:teuthology.orchestra.run.smithi123.stdout: "key": "AQDCUehmD8VgAhAAneoIeGr3MflSmi+nyavj2A==", 2024-09-16T15:48:22.787 INFO:teuthology.orchestra.run.smithi123.stdout: "caps": { 2024-09-16T15:48:22.787 INFO:teuthology.orchestra.run.smithi123.stdout: "mgr": "allow profile osd", 2024-09-16T15:48:22.787 INFO:teuthology.orchestra.run.smithi123.stdout: "mon": "allow profile osd", 2024-09-16T15:48:22.787 INFO:teuthology.orchestra.run.smithi123.stdout: "osd": "allow *" 2024-09-16T15:48:22.787 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-16T15:48:22.787 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:48:22.787 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-09-16T15:48:22.787 INFO:teuthology.orchestra.run.smithi123.stdout: "entity": "osd.4", 2024-09-16T15:48:22.787 INFO:teuthology.orchestra.run.smithi123.stdout: "key": "AQDIUehmA9B9EBAAsu0X5LPx1TmatZeHzjnVig==", 2024-09-16T15:48:22.787 INFO:teuthology.orchestra.run.smithi123.stdout: "caps": { 2024-09-16T15:48:22.787 INFO:teuthology.orchestra.run.smithi123.stdout: "mgr": "allow profile osd", 2024-09-16T15:48:22.787 INFO:teuthology.orchestra.run.smithi123.stdout: "mon": "allow profile osd", 2024-09-16T15:48:22.788 INFO:teuthology.orchestra.run.smithi123.stdout: "osd": "allow *" 2024-09-16T15:48:22.788 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-16T15:48:22.788 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:48:22.788 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-09-16T15:48:22.788 INFO:teuthology.orchestra.run.smithi123.stdout: "entity": "osd.5", 2024-09-16T15:48:22.788 INFO:teuthology.orchestra.run.smithi123.stdout: "key": "AQDLUehmcugdHxAAQMSS1ZH71FUzCx5A+yNnZg==", 2024-09-16T15:48:22.788 INFO:teuthology.orchestra.run.smithi123.stdout: "caps": { 2024-09-16T15:48:22.788 INFO:teuthology.orchestra.run.smithi123.stdout: "mgr": "allow profile osd", 2024-09-16T15:48:22.788 INFO:teuthology.orchestra.run.smithi123.stdout: "mon": "allow profile osd", 2024-09-16T15:48:22.788 INFO:teuthology.orchestra.run.smithi123.stdout: "osd": "allow *" 2024-09-16T15:48:22.788 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-16T15:48:22.788 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:48:22.788 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-09-16T15:48:22.788 INFO:teuthology.orchestra.run.smithi123.stdout: "entity": "osd.6", 2024-09-16T15:48:22.788 INFO:teuthology.orchestra.run.smithi123.stdout: "key": "AQDOUehmWLwYLhAAhOlQLepcWElsVPMnHlZm4w==", 2024-09-16T15:48:22.789 INFO:teuthology.orchestra.run.smithi123.stdout: "caps": { 2024-09-16T15:48:22.789 INFO:teuthology.orchestra.run.smithi123.stdout: "mgr": "allow profile osd", 2024-09-16T15:48:22.789 INFO:teuthology.orchestra.run.smithi123.stdout: "mon": "allow profile osd", 2024-09-16T15:48:22.789 INFO:teuthology.orchestra.run.smithi123.stdout: "osd": "allow *" 2024-09-16T15:48:22.789 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-16T15:48:22.789 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:48:22.789 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-09-16T15:48:22.789 INFO:teuthology.orchestra.run.smithi123.stdout: "entity": "osd.7", 2024-09-16T15:48:22.789 INFO:teuthology.orchestra.run.smithi123.stdout: "key": "AQDSUehmipkbABAAQuL/2O7w0tsM1OKeOil97w==", 2024-09-16T15:48:22.789 INFO:teuthology.orchestra.run.smithi123.stdout: "caps": { 2024-09-16T15:48:22.789 INFO:teuthology.orchestra.run.smithi123.stdout: "mgr": "allow profile osd", 2024-09-16T15:48:22.789 INFO:teuthology.orchestra.run.smithi123.stdout: "mon": "allow profile osd", 2024-09-16T15:48:22.789 INFO:teuthology.orchestra.run.smithi123.stdout: "osd": "allow *" 2024-09-16T15:48:22.789 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-16T15:48:22.790 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:48:22.790 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-09-16T15:48:22.790 INFO:teuthology.orchestra.run.smithi123.stdout: "entity": "client.0", 2024-09-16T15:48:22.790 INFO:teuthology.orchestra.run.smithi123.stdout: "key": "AQC1Uehm7GDTAhAAHWWVIyjVVnTeeozBR8DfFA==", 2024-09-16T15:48:22.790 INFO:teuthology.orchestra.run.smithi123.stdout: "caps": { 2024-09-16T15:48:22.790 INFO:teuthology.orchestra.run.smithi123.stdout: "mds": "allow", 2024-09-16T15:48:22.790 INFO:teuthology.orchestra.run.smithi123.stdout: "mon": "allow r", 2024-09-16T15:48:22.790 INFO:teuthology.orchestra.run.smithi123.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-09-16T15:48:22.790 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-16T15:48:22.790 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:48:22.790 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-09-16T15:48:22.790 INFO:teuthology.orchestra.run.smithi123.stdout: "entity": "client.1", 2024-09-16T15:48:22.790 INFO:teuthology.orchestra.run.smithi123.stdout: "key": "AQC1UehmFbaWChAAFtRmKucm+pbKtCNvRCTNqA==", 2024-09-16T15:48:22.790 INFO:teuthology.orchestra.run.smithi123.stdout: "caps": { 2024-09-16T15:48:22.791 INFO:teuthology.orchestra.run.smithi123.stdout: "mds": "allow", 2024-09-16T15:48:22.791 INFO:teuthology.orchestra.run.smithi123.stdout: "mgr": "allow r", 2024-09-16T15:48:22.791 INFO:teuthology.orchestra.run.smithi123.stdout: "mon": "allow rw", 2024-09-16T15:48:22.791 INFO:teuthology.orchestra.run.smithi123.stdout: "osd": "allow rwx" 2024-09-16T15:48:22.791 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-16T15:48:22.791 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:48:22.791 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-09-16T15:48:22.791 INFO:teuthology.orchestra.run.smithi123.stdout: "entity": "client.2", 2024-09-16T15:48:22.791 INFO:teuthology.orchestra.run.smithi123.stdout: "key": "AQC1Uehm1ADrBxAAW/8ayNHI6Wq/YlI4ivFZiw==", 2024-09-16T15:48:22.791 INFO:teuthology.orchestra.run.smithi123.stdout: "caps": { 2024-09-16T15:48:22.791 INFO:teuthology.orchestra.run.smithi123.stdout: "mds": "allow", 2024-09-16T15:48:22.791 INFO:teuthology.orchestra.run.smithi123.stdout: "mgr": "allow r", 2024-09-16T15:48:22.791 INFO:teuthology.orchestra.run.smithi123.stdout: "mon": "allow rw", 2024-09-16T15:48:22.791 INFO:teuthology.orchestra.run.smithi123.stdout: "osd": "allow rwx" 2024-09-16T15:48:22.791 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-16T15:48:22.792 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:48:22.792 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-09-16T15:48:22.792 INFO:teuthology.orchestra.run.smithi123.stdout: "entity": "client.3", 2024-09-16T15:48:22.792 INFO:teuthology.orchestra.run.smithi123.stdout: "key": "AQC1UehmLv/3DxAAuGloBxSuza95uajpfiZ9MQ==", 2024-09-16T15:48:22.792 INFO:teuthology.orchestra.run.smithi123.stdout: "caps": { 2024-09-16T15:48:22.792 INFO:teuthology.orchestra.run.smithi123.stdout: "mds": "allow", 2024-09-16T15:48:22.792 INFO:teuthology.orchestra.run.smithi123.stdout: "mgr": "allow r", 2024-09-16T15:48:22.792 INFO:teuthology.orchestra.run.smithi123.stdout: "mon": "allow rw", 2024-09-16T15:48:22.792 INFO:teuthology.orchestra.run.smithi123.stdout: "osd": "allow rwx" 2024-09-16T15:48:22.792 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-16T15:48:22.792 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:48:22.792 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-09-16T15:48:22.792 INFO:teuthology.orchestra.run.smithi123.stdout: "entity": "client.admin", 2024-09-16T15:48:22.793 INFO:teuthology.orchestra.run.smithi123.stdout: "key": "AQC0Uehm1ZDRDBAAhyQ1BmJg1Mw6wQ04I2CbFA==", 2024-09-16T15:48:22.793 INFO:teuthology.orchestra.run.smithi123.stdout: "caps": { 2024-09-16T15:48:22.793 INFO:teuthology.orchestra.run.smithi123.stdout: "mds": "allow *", 2024-09-16T15:48:22.793 INFO:teuthology.orchestra.run.smithi123.stdout: "mgr": "allow *", 2024-09-16T15:48:22.793 INFO:teuthology.orchestra.run.smithi123.stdout: "mon": "allow *", 2024-09-16T15:48:22.793 INFO:teuthology.orchestra.run.smithi123.stdout: "osd": "allow *" 2024-09-16T15:48:22.793 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-16T15:48:22.793 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:48:22.793 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-09-16T15:48:22.793 INFO:teuthology.orchestra.run.smithi123.stdout: "entity": "mgr.x", 2024-09-16T15:48:22.793 INFO:teuthology.orchestra.run.smithi123.stdout: "key": "AQC0UehmfPilIBAAERlBbduOgAU9ba0n/1tl3g==", 2024-09-16T15:48:22.793 INFO:teuthology.orchestra.run.smithi123.stdout: "caps": { 2024-09-16T15:48:22.793 INFO:teuthology.orchestra.run.smithi123.stdout: "mds": "allow *", 2024-09-16T15:48:22.793 INFO:teuthology.orchestra.run.smithi123.stdout: "mon": "allow profile mgr", 2024-09-16T15:48:22.794 INFO:teuthology.orchestra.run.smithi123.stdout: "osd": "allow *" 2024-09-16T15:48:22.794 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-16T15:48:22.794 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:48:22.794 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-09-16T15:48:22.794 INFO:teuthology.orchestra.run.smithi123.stdout: "entity": "mgr.y", 2024-09-16T15:48:22.794 INFO:teuthology.orchestra.run.smithi123.stdout: "key": "AQC0UehmkjrlHRAA+3xbQC+SPWGWW1gND7xcAg==", 2024-09-16T15:48:22.794 INFO:teuthology.orchestra.run.smithi123.stdout: "caps": { 2024-09-16T15:48:22.794 INFO:teuthology.orchestra.run.smithi123.stdout: "mds": "allow *", 2024-09-16T15:48:22.794 INFO:teuthology.orchestra.run.smithi123.stdout: "mon": "allow profile mgr", 2024-09-16T15:48:22.794 INFO:teuthology.orchestra.run.smithi123.stdout: "osd": "allow *" 2024-09-16T15:48:22.794 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-16T15:48:22.794 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-16T15:48:22.794 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-09-16T15:48:22.794 INFO:teuthology.orchestra.run.smithi123.stdout:} 2024-09-16T15:48:22.795 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:22.783+0000 7f62354e3640 1 -- 172.21.15.123:0/123632491 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f621c072fe0 msgr2=0x7f621c0754a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:48:22.795 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:22.783+0000 7f62354e3640 1 --2- 172.21.15.123:0/123632491 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f621c072fe0 0x7f621c0754a0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f6220001ec0 tx=0x7f6220009040 comp rx=0 tx=0).stop 2024-09-16T15:48:22.795 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:22.783+0000 7f62354e3640 1 -- 172.21.15.123:0/123632491 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f62301581d0 msgr2=0x7f62300b3540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:48:22.795 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:22.783+0000 7f62354e3640 1 --2- 172.21.15.123:0/123632491 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f62301581d0 0x7f62300b3540 secure :-1 s=READY pgs=1048 cs=0 l=1 rev1=1 crypto rx=0x7f6228009eb0 tx=0x7f6228009280 comp rx=0 tx=0).stop 2024-09-16T15:48:22.795 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:22.783+0000 7f62354e3640 1 -- 172.21.15.123:0/123632491 shutdown_connections 2024-09-16T15:48:22.796 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:22.783+0000 7f62354e3640 1 --2- 172.21.15.123:0/123632491 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f621c072fe0 0x7f621c0754a0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:22.796 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:22.783+0000 7f62354e3640 1 --2- 172.21.15.123:0/123632491 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f62301581d0 0x7f62300b3540 unknown :-1 s=CLOSED pgs=1048 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:22.796 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:22.783+0000 7f62354e3640 1 --2- 172.21.15.123:0/123632491 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f623015d820 0x7f6230156430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:22.796 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:22.783+0000 7f62354e3640 1 --2- 172.21.15.123:0/123632491 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f62180053d0 0x7f62300c9140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:22.796 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:22.783+0000 7f62354e3640 1 -- 172.21.15.123:0/123632491 >> 172.21.15.123:0/123632491 conn(0x7f62300d3ea0 msgr2=0x7f623006e460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:48:22.796 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:22.783+0000 7f62354e3640 1 -- 172.21.15.123:0/123632491 shutdown_connections 2024-09-16T15:48:22.796 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:22.783+0000 7f62354e3640 1 -- 172.21.15.123:0/123632491 wait complete. 2024-09-16T15:48:22.800 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-09-16T15:48:22.910 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:22.911+0000 7f2e153be640 1 -- 172.21.15.123:0/3732448695 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2e100b2b00 msgr2=0x7f2e100d22e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:48:22.910 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:22.911+0000 7f2e153be640 1 --2- 172.21.15.123:0/3732448695 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2e100b2b00 0x7f2e100d22e0 secure :-1 s=READY pgs=1050 cs=0 l=1 rev1=1 crypto rx=0x7f2e1006f930 tx=0x7f2e0401c7a0 comp rx=0 tx=0).stop 2024-09-16T15:48:22.911 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:22.911+0000 7f2e153be640 1 -- 172.21.15.123:0/3732448695 shutdown_connections 2024-09-16T15:48:22.911 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:22.911+0000 7f2e153be640 1 --2- 172.21.15.123:0/3732448695 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2e100b2b00 0x7f2e100d22e0 unknown :-1 s=CLOSED pgs=1050 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:22.911 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:22.911+0000 7f2e153be640 1 --2- 172.21.15.123:0/3732448695 >> v2:172.21.15.184:3300/0 conn(0x7f2e10161620 0x7f2e100d1da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:22.911 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:22.911+0000 7f2e153be640 1 -- 172.21.15.123:0/3732448695 >> 172.21.15.123:0/3732448695 conn(0x7f2e100d3ea0 msgr2=0x7f2e100d42b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:48:22.911 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:22.911+0000 7f2e153be640 1 -- 172.21.15.123:0/3732448695 shutdown_connections 2024-09-16T15:48:22.911 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:22.911+0000 7f2e153be640 1 -- 172.21.15.123:0/3732448695 wait complete. 2024-09-16T15:48:22.911 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:22.911+0000 7f2e153be640 1 Processor -- start 2024-09-16T15:48:22.911 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:22.911+0000 7f2e153be640 1 -- start start 2024-09-16T15:48:22.913 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:22.911+0000 7f2e153be640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2e100b2b00 0x7f2e100cdef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:48:22.913 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:22.911+0000 7f2e153be640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f2e100d2820 0x7f2e100cb210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:48:22.913 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:22.911+0000 7f2e153be640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2e10161620 0x7f2e101f7f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:48:22.913 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:22.911+0000 7f2e153be640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f2e1016aee0 con 0x7f2e100b2b00 2024-09-16T15:48:22.913 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:22.911+0000 7f2e153be640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f2e1016ad60 con 0x7f2e10161620 2024-09-16T15:48:22.913 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:22.911+0000 7f2e153be640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f2e1016b060 con 0x7f2e100d2820 2024-09-16T15:48:22.913 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:22.911+0000 7f2e0effd640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2e100b2b00 0x7f2e100cdef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:48:22.913 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:22.911+0000 7f2e0effd640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2e100b2b00 0x7f2e100cdef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:47322/0 (socket says 172.21.15.123:47322) 2024-09-16T15:48:22.914 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:22.911+0000 7f2e0effd640 1 -- 172.21.15.123:0/3687366280 learned_addr learned my addr 172.21.15.123:0/3687366280 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:48:22.914 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:22.911+0000 7f2e0effd640 1 -- 172.21.15.123:0/3687366280 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f2e100d2820 msgr2=0x7f2e100cb210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:48:22.914 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:22.911+0000 7f2e0effd640 1 --2- 172.21.15.123:0/3687366280 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f2e100d2820 0x7f2e100cb210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:22.914 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:22.911+0000 7f2e0effd640 1 -- 172.21.15.123:0/3687366280 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2e10161620 msgr2=0x7f2e101f7f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:48:22.914 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:22.911+0000 7f2e0f7fe640 1 --2- 172.21.15.123:0/3687366280 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2e10161620 0x7f2e101f7f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:48:22.914 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:22.911+0000 7f2e0effd640 1 --2- 172.21.15.123:0/3687366280 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2e10161620 0x7f2e101f7f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:22.914 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:22.911+0000 7f2e0effd640 1 -- 172.21.15.123:0/3687366280 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2e100cb750 con 0x7f2e100b2b00 2024-09-16T15:48:22.914 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:22.911+0000 7f2e0effd640 1 --2- 172.21.15.123:0/3687366280 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2e100b2b00 0x7f2e100cdef0 secure :-1 s=READY pgs=1051 cs=0 l=1 rev1=1 crypto rx=0x7f2e0000c9a0 tx=0x7f2e0000ce60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:48:22.914 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:22.911+0000 7f2df77fe640 1 -- 172.21.15.123:0/3687366280 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e00007bb0 con 0x7f2e100b2b00 2024-09-16T15:48:22.914 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:22.911+0000 7f2df77fe640 1 -- 172.21.15.123:0/3687366280 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f2e00007e40 con 0x7f2e100b2b00 2024-09-16T15:48:22.914 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:22.911+0000 7f2e153be640 1 -- 172.21.15.123:0/3687366280 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2e100cba20 con 0x7f2e100b2b00 2024-09-16T15:48:22.914 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:22.911+0000 7f2df77fe640 1 -- 172.21.15.123:0/3687366280 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e00005270 con 0x7f2e100b2b00 2024-09-16T15:48:22.914 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:22.911+0000 7f2e153be640 1 -- 172.21.15.123:0/3687366280 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2e100cd710 con 0x7f2e100b2b00 2024-09-16T15:48:22.915 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:22.915+0000 7f2e153be640 1 -- 172.21.15.123:0/3687366280 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2e100cea00 con 0x7f2e100b2b00 2024-09-16T15:48:22.915 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:22.915+0000 7f2df77fe640 1 -- 172.21.15.123:0/3687366280 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 16) ==== 97182+0+0 (secure 0 0 0) 0x7f2e00005410 con 0x7f2e100b2b00 2024-09-16T15:48:22.915 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:22.915+0000 7f2df77fe640 1 --2- 172.21.15.123:0/3687366280 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f2dfc072fe0 0x7f2dfc0754a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:48:22.915 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:22.915+0000 7f2df77fe640 1 -- 172.21.15.123:0/3687366280 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 4857+0+0 (secure 0 0 0) 0x7f2e00094150 con 0x7f2e100b2b00 2024-09-16T15:48:22.915 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:22.915+0000 7f2e0e7fc640 1 --2- 172.21.15.123:0/3687366280 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f2dfc072fe0 0x7f2dfc0754a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:48:22.915 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:22.915+0000 7f2e0e7fc640 1 --2- 172.21.15.123:0/3687366280 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f2dfc072fe0 0x7f2dfc0754a0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f2e040af000 tx=0x7f2e04019540 comp rx=0 tx=0).ready entity=mgr.4150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:48:22.919 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:22.919+0000 7f2df77fe640 1 -- 172.21.15.123:0/3687366280 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f2e00014070 con 0x7f2e100b2b00 2024-09-16T15:48:23.080 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:23.079+0000 7f2e153be640 1 -- 172.21.15.123:0/3687366280 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) -- 0x7f2e100c8ef0 con 0x7f2e100b2b00 2024-09-16T15:48:23.080 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:23.079+0000 7f2df77fe640 1 -- 172.21.15.123:0/3687366280 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v99) ==== 84+0+65 (secure 0 0 0) 0x7f2e0005f620 con 0x7f2e100b2b00 2024-09-16T15:48:23.081 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-09-16T15:48:23.081 INFO:teuthology.orchestra.run.smithi123.stdout:[ 2024-09-16T15:48:23.081 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-09-16T15:48:23.081 INFO:teuthology.orchestra.run.smithi123.stdout: "poolnum": 1, 2024-09-16T15:48:23.081 INFO:teuthology.orchestra.run.smithi123.stdout: "poolname": ".mgr" 2024-09-16T15:48:23.081 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-16T15:48:23.081 INFO:teuthology.orchestra.run.smithi123.stdout:] 2024-09-16T15:48:23.083 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:23.083+0000 7f2e153be640 1 -- 172.21.15.123:0/3687366280 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f2dfc072fe0 msgr2=0x7f2dfc0754a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:48:23.083 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:23.083+0000 7f2e153be640 1 --2- 172.21.15.123:0/3687366280 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f2dfc072fe0 0x7f2dfc0754a0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f2e040af000 tx=0x7f2e04019540 comp rx=0 tx=0).stop 2024-09-16T15:48:23.083 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:23.083+0000 7f2e153be640 1 -- 172.21.15.123:0/3687366280 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2e100b2b00 msgr2=0x7f2e100cdef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:48:23.083 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:23.083+0000 7f2e153be640 1 --2- 172.21.15.123:0/3687366280 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2e100b2b00 0x7f2e100cdef0 secure :-1 s=READY pgs=1051 cs=0 l=1 rev1=1 crypto rx=0x7f2e0000c9a0 tx=0x7f2e0000ce60 comp rx=0 tx=0).stop 2024-09-16T15:48:23.083 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:23.083+0000 7f2e153be640 1 -- 172.21.15.123:0/3687366280 shutdown_connections 2024-09-16T15:48:23.083 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:23.083+0000 7f2e153be640 1 --2- 172.21.15.123:0/3687366280 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f2dfc072fe0 0x7f2dfc0754a0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:23.083 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:23.083+0000 7f2e153be640 1 --2- 172.21.15.123:0/3687366280 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2e10161620 0x7f2e101f7f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:23.083 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:23.083+0000 7f2e153be640 1 --2- 172.21.15.123:0/3687366280 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f2e100d2820 0x7f2e100cb210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:23.083 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:23.083+0000 7f2e153be640 1 --2- 172.21.15.123:0/3687366280 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2e100b2b00 0x7f2e100cdef0 unknown :-1 s=CLOSED pgs=1051 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:23.083 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:23.083+0000 7f2e153be640 1 -- 172.21.15.123:0/3687366280 >> 172.21.15.123:0/3687366280 conn(0x7f2e100d3ea0 msgr2=0x7f2e100d4280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:48:23.083 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:23.083+0000 7f2e153be640 1 -- 172.21.15.123:0/3687366280 shutdown_connections 2024-09-16T15:48:23.083 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:23.083+0000 7f2e153be640 1 -- 172.21.15.123:0/3687366280 wait complete. 2024-09-16T15:48:23.098 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-09-16T15:48:23.098 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-09-16T15:48:23.228 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:23.227+0000 7f5b93a85640 1 -- 172.21.15.123:0/1881942752 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5b8c073c20 msgr2=0x7f5b8c15b000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:48:23.229 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:23.227+0000 7f5b93a85640 1 --2- 172.21.15.123:0/1881942752 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5b8c073c20 0x7f5b8c15b000 secure :-1 s=READY pgs=1053 cs=0 l=1 rev1=1 crypto rx=0x7f5b8c1556a0 tx=0x7f5b8801c600 comp rx=0 tx=0).stop 2024-09-16T15:48:23.229 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:23.227+0000 7f5b93a85640 1 -- 172.21.15.123:0/1881942752 shutdown_connections 2024-09-16T15:48:23.229 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:23.227+0000 7f5b93a85640 1 --2- 172.21.15.123:0/1881942752 >> v2:172.21.15.184:3300/0 conn(0x7f5b8c070640 0x7f5b8c0b3260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:23.229 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:23.227+0000 7f5b93a85640 1 --2- 172.21.15.123:0/1881942752 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5b8c073c20 0x7f5b8c15b000 unknown :-1 s=CLOSED pgs=1053 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:23.229 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:23.227+0000 7f5b93a85640 1 -- 172.21.15.123:0/1881942752 >> 172.21.15.123:0/1881942752 conn(0x7f5b8c0b1630 msgr2=0x7f5b8c0b1a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:48:23.229 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:23.227+0000 7f5b93a85640 1 -- 172.21.15.123:0/1881942752 shutdown_connections 2024-09-16T15:48:23.229 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:23.227+0000 7f5b93a85640 1 -- 172.21.15.123:0/1881942752 wait complete. 2024-09-16T15:48:23.229 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:23.227+0000 7f5b93a85640 1 Processor -- start 2024-09-16T15:48:23.230 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:23.231+0000 7f5b93a85640 1 -- start start 2024-09-16T15:48:23.231 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:23.231+0000 7f5b93a85640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5b8c070640 0x7f5b8c06fed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:48:23.231 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:23.231+0000 7f5b93a85640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f5b8c073c20 0x7f5b8c13d3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:48:23.231 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:23.231+0000 7f5b93a85640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5b8c15b540 0x7f5b8c0ce020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:48:23.231 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:23.231+0000 7f5b93a85640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f5b8c167d70 con 0x7f5b8c15b540 2024-09-16T15:48:23.232 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:23.231+0000 7f5b93a85640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f5b8c167bf0 con 0x7f5b8c070640 2024-09-16T15:48:23.232 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:23.231+0000 7f5b93a85640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f5b8c167ef0 con 0x7f5b8c073c20 2024-09-16T15:48:23.232 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:23.231+0000 7f5b92022640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5b8c15b540 0x7f5b8c0ce020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:48:23.232 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:23.231+0000 7f5b92022640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5b8c15b540 0x7f5b8c0ce020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:47344/0 (socket says 172.21.15.123:47344) 2024-09-16T15:48:23.232 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:23.231+0000 7f5b92022640 1 -- 172.21.15.123:0/2551919804 learned_addr learned my addr 172.21.15.123:0/2551919804 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:48:23.232 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:23.231+0000 7f5b91020640 1 --2- 172.21.15.123:0/2551919804 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f5b8c073c20 0x7f5b8c13d3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:48:23.232 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:23.231+0000 7f5b91821640 1 --2- 172.21.15.123:0/2551919804 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5b8c070640 0x7f5b8c06fed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:48:23.232 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:23.231+0000 7f5b92022640 1 -- 172.21.15.123:0/2551919804 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f5b8c073c20 msgr2=0x7f5b8c13d3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:48:23.232 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:23.231+0000 7f5b92022640 1 --2- 172.21.15.123:0/2551919804 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f5b8c073c20 0x7f5b8c13d3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:23.232 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:23.231+0000 7f5b92022640 1 -- 172.21.15.123:0/2551919804 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5b8c070640 msgr2=0x7f5b8c06fed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:48:23.232 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:23.231+0000 7f5b92022640 1 --2- 172.21.15.123:0/2551919804 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5b8c070640 0x7f5b8c06fed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:23.232 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:23.231+0000 7f5b92022640 1 -- 172.21.15.123:0/2551919804 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5b8c070410 con 0x7f5b8c15b540 2024-09-16T15:48:23.232 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:23.231+0000 7f5b91020640 1 --2- 172.21.15.123:0/2551919804 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f5b8c073c20 0x7f5b8c13d3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T15:48:23.233 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:23.231+0000 7f5b91821640 1 --2- 172.21.15.123:0/2551919804 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5b8c070640 0x7f5b8c06fed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T15:48:23.233 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:23.231+0000 7f5b92022640 1 --2- 172.21.15.123:0/2551919804 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5b8c15b540 0x7f5b8c0ce020 secure :-1 s=READY pgs=1054 cs=0 l=1 rev1=1 crypto rx=0x7f5b8400cae0 tx=0x7f5b8400a250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:48:23.233 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:23.231+0000 7f5b827fc640 1 -- 172.21.15.123:0/2551919804 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b8400ae30 con 0x7f5b8c15b540 2024-09-16T15:48:23.233 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:23.231+0000 7f5b827fc640 1 -- 172.21.15.123:0/2551919804 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f5b84004b30 con 0x7f5b8c15b540 2024-09-16T15:48:23.233 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:23.231+0000 7f5b93a85640 1 -- 172.21.15.123:0/2551919804 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5b8c0c9fc0 con 0x7f5b8c15b540 2024-09-16T15:48:23.233 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:23.231+0000 7f5b827fc640 1 -- 172.21.15.123:0/2551919804 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b84005020 con 0x7f5b8c15b540 2024-09-16T15:48:23.233 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:23.231+0000 7f5b93a85640 1 -- 172.21.15.123:0/2551919804 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5b8c0ca580 con 0x7f5b8c15b540 2024-09-16T15:48:23.234 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:23.231+0000 7f5b93a85640 1 -- 172.21.15.123:0/2551919804 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5b8c0b3260 con 0x7f5b8c15b540 2024-09-16T15:48:23.234 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:23.231+0000 7f5b827fc640 1 -- 172.21.15.123:0/2551919804 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 16) ==== 97182+0+0 (secure 0 0 0) 0x7f5b840051c0 con 0x7f5b8c15b540 2024-09-16T15:48:23.234 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:23.231+0000 7f5b827fc640 1 --2- 172.21.15.123:0/2551919804 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f5b70072fe0 0x7f5b700754a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:48:23.234 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:23.231+0000 7f5b827fc640 1 -- 172.21.15.123:0/2551919804 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 4857+0+0 (secure 0 0 0) 0x7f5b84094810 con 0x7f5b8c15b540 2024-09-16T15:48:23.234 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:23.231+0000 7f5b91821640 1 --2- 172.21.15.123:0/2551919804 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f5b70072fe0 0x7f5b700754a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:48:23.234 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:23.235+0000 7f5b91821640 1 --2- 172.21.15.123:0/2551919804 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f5b70072fe0 0x7f5b700754a0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f5b880ae000 tx=0x7f5b8801ae40 comp rx=0 tx=0).ready entity=mgr.4150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:48:23.237 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:23.239+0000 7f5b827fc640 1 -- 172.21.15.123:0/2551919804 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f5b84014030 con 0x7f5b8c15b540 2024-09-16T15:48:23.409 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:23.407+0000 7f5b93a85640 1 -- 172.21.15.123:0/2551919804 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) -- 0x7f5b8c06ef50 con 0x7f5b8c15b540 2024-09-16T15:48:24.419 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:24.419+0000 7f5b827fc640 1 -- 172.21.15.123:0/2551919804 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v100) ==== 138+0+0 (secure 0 0 0) 0x7f5b8405fc20 con 0x7f5b8c15b540 2024-09-16T15:48:24.419 INFO:teuthology.orchestra.run.smithi123.stderr:pool 'cephfs_metadata' created 2024-09-16T15:48:24.423 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:24.423+0000 7f5b93a85640 1 -- 172.21.15.123:0/2551919804 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f5b70072fe0 msgr2=0x7f5b700754a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:48:24.423 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:24.423+0000 7f5b93a85640 1 --2- 172.21.15.123:0/2551919804 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f5b70072fe0 0x7f5b700754a0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f5b880ae000 tx=0x7f5b8801ae40 comp rx=0 tx=0).stop 2024-09-16T15:48:24.423 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:24.423+0000 7f5b93a85640 1 -- 172.21.15.123:0/2551919804 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5b8c15b540 msgr2=0x7f5b8c0ce020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:48:24.423 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:24.423+0000 7f5b93a85640 1 --2- 172.21.15.123:0/2551919804 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5b8c15b540 0x7f5b8c0ce020 secure :-1 s=READY pgs=1054 cs=0 l=1 rev1=1 crypto rx=0x7f5b8400cae0 tx=0x7f5b8400a250 comp rx=0 tx=0).stop 2024-09-16T15:48:24.423 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:24.423+0000 7f5b93a85640 1 -- 172.21.15.123:0/2551919804 shutdown_connections 2024-09-16T15:48:24.423 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:24.423+0000 7f5b93a85640 1 --2- 172.21.15.123:0/2551919804 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f5b70072fe0 0x7f5b700754a0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:24.423 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:24.423+0000 7f5b93a85640 1 --2- 172.21.15.123:0/2551919804 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5b8c15b540 0x7f5b8c0ce020 unknown :-1 s=CLOSED pgs=1054 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:24.423 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:24.423+0000 7f5b93a85640 1 --2- 172.21.15.123:0/2551919804 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f5b8c073c20 0x7f5b8c13d3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:24.423 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:24.423+0000 7f5b93a85640 1 --2- 172.21.15.123:0/2551919804 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5b8c070640 0x7f5b8c06fed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:24.423 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:24.423+0000 7f5b93a85640 1 -- 172.21.15.123:0/2551919804 >> 172.21.15.123:0/2551919804 conn(0x7f5b8c0b1630 msgr2=0x7f5b8c1507b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:48:24.424 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:24.423+0000 7f5b93a85640 1 -- 172.21.15.123:0/2551919804 shutdown_connections 2024-09-16T15:48:24.424 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:24.423+0000 7f5b93a85640 1 -- 172.21.15.123:0/2551919804 wait complete. 2024-09-16T15:48:24.450 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-09-16T15:48:24.584 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:24.583+0000 7f30f5a21640 1 -- 172.21.15.123:0/3555020115 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f30f00ca650 msgr2=0x7f30f00cbc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:48:24.584 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:24.583+0000 7f30f5a21640 1 --2- 172.21.15.123:0/3555020115 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f30f00ca650 0x7f30f00cbc60 secure :-1 s=READY pgs=1056 cs=0 l=1 rev1=1 crypto rx=0x7f30e4009900 tx=0x7f30e401c820 comp rx=0 tx=0).stop 2024-09-16T15:48:24.584 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:24.583+0000 7f30f5a21640 1 -- 172.21.15.123:0/3555020115 shutdown_connections 2024-09-16T15:48:24.584 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:24.583+0000 7f30f5a21640 1 --2- 172.21.15.123:0/3555020115 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f30f00ca650 0x7f30f00cbc60 unknown :-1 s=CLOSED pgs=1056 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:24.584 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:24.583+0000 7f30f5a21640 1 --2- 172.21.15.123:0/3555020115 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f30f015a170 0x7f30f0072f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:24.584 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:24.583+0000 7f30f5a21640 1 -- 172.21.15.123:0/3555020115 >> 172.21.15.123:0/3555020115 conn(0x7f30f00b1710 msgr2=0x7f30f00b1b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:48:24.585 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:24.583+0000 7f30f5a21640 1 -- 172.21.15.123:0/3555020115 shutdown_connections 2024-09-16T15:48:24.585 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:24.583+0000 7f30f5a21640 1 -- 172.21.15.123:0/3555020115 wait complete. 2024-09-16T15:48:24.585 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:24.583+0000 7f30f5a21640 1 Processor -- start 2024-09-16T15:48:24.585 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:24.583+0000 7f30f5a21640 1 -- start start 2024-09-16T15:48:24.587 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:24.587+0000 7f30f5a21640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f30f00ca650 0x7f30f01f59e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:48:24.587 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:24.587+0000 7f30f5a21640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f30f015a170 0x7f30f00c8690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:48:24.587 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:24.587+0000 7f30f5a21640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f30f01f8550 0x7f30f01f89e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:48:24.587 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:24.587+0000 7f30f5a21640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f30f01670d0 con 0x7f30f00ca650 2024-09-16T15:48:24.587 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:24.587+0000 7f30f5a21640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f30f0166f50 con 0x7f30f01f8550 2024-09-16T15:48:24.587 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:24.587+0000 7f30f5a21640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f30f0167250 con 0x7f30f015a170 2024-09-16T15:48:24.587 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:24.587+0000 7f30eeffd640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f30f00ca650 0x7f30f01f59e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:48:24.587 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:24.587+0000 7f30eeffd640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f30f00ca650 0x7f30f01f59e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:47372/0 (socket says 172.21.15.123:47372) 2024-09-16T15:48:24.587 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:24.587+0000 7f30eeffd640 1 -- 172.21.15.123:0/723992358 learned_addr learned my addr 172.21.15.123:0/723992358 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:48:24.588 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:24.587+0000 7f30eeffd640 1 -- 172.21.15.123:0/723992358 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f30f015a170 msgr2=0x7f30f00c8690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:48:24.588 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:24.587+0000 7f30eeffd640 1 --2- 172.21.15.123:0/723992358 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f30f015a170 0x7f30f00c8690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:24.588 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:24.587+0000 7f30eeffd640 1 -- 172.21.15.123:0/723992358 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f30f01f8550 msgr2=0x7f30f01f89e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:48:24.588 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:24.587+0000 7f30eeffd640 1 --2- 172.21.15.123:0/723992358 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f30f01f8550 0x7f30f01f89e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:24.588 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:24.587+0000 7f30eeffd640 1 -- 172.21.15.123:0/723992358 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f30f01f9010 con 0x7f30f00ca650 2024-09-16T15:48:24.588 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:24.587+0000 7f30eeffd640 1 --2- 172.21.15.123:0/723992358 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f30f00ca650 0x7f30f01f59e0 secure :-1 s=READY pgs=1057 cs=0 l=1 rev1=1 crypto rx=0x7f30e000c940 tx=0x7f30e000ce00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:48:24.588 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:24.587+0000 7f30d7fff640 1 -- 172.21.15.123:0/723992358 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30e0004220 con 0x7f30f00ca650 2024-09-16T15:48:24.588 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:24.587+0000 7f30d7fff640 1 -- 172.21.15.123:0/723992358 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f30e00079b0 con 0x7f30f00ca650 2024-09-16T15:48:24.588 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:24.587+0000 7f30f5a21640 1 -- 172.21.15.123:0/723992358 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f30f01f92f0 con 0x7f30f00ca650 2024-09-16T15:48:24.589 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:24.587+0000 7f30d7fff640 1 -- 172.21.15.123:0/723992358 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30e00044b0 con 0x7f30f00ca650 2024-09-16T15:48:24.589 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:24.587+0000 7f30f5a21640 1 -- 172.21.15.123:0/723992358 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f30f01f54c0 con 0x7f30f00ca650 2024-09-16T15:48:24.589 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:24.587+0000 7f30f5a21640 1 -- 172.21.15.123:0/723992358 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f30f0072f30 con 0x7f30f00ca650 2024-09-16T15:48:24.590 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:24.587+0000 7f30d7fff640 1 -- 172.21.15.123:0/723992358 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 16) ==== 97182+0+0 (secure 0 0 0) 0x7f30e0003600 con 0x7f30f00ca650 2024-09-16T15:48:24.590 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:24.587+0000 7f30d7fff640 1 --2- 172.21.15.123:0/723992358 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f30dc072fe0 0x7f30dc0754a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:48:24.590 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:24.587+0000 7f30ee7fc640 1 --2- 172.21.15.123:0/723992358 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f30dc072fe0 0x7f30dc0754a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:48:24.590 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:24.587+0000 7f30d7fff640 1 -- 172.21.15.123:0/723992358 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 5227+0+0 (secure 0 0 0) 0x7f30e0094040 con 0x7f30f00ca650 2024-09-16T15:48:24.590 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:24.587+0000 7f30ee7fc640 1 --2- 172.21.15.123:0/723992358 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f30dc072fe0 0x7f30dc0754a0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f30e40af000 tx=0x7f30e40195c0 comp rx=0 tx=0).ready entity=mgr.4150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:48:24.593 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:24.591+0000 7f30d7fff640 1 -- 172.21.15.123:0/723992358 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f30e0014030 con 0x7f30f00ca650 2024-09-16T15:48:24.767 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:24.767+0000 7f30f5a21640 1 -- 172.21.15.123:0/723992358 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) -- 0x7f30f015dfa0 con 0x7f30f00ca650 2024-09-16T15:48:25.424 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:25.423+0000 7f30d7fff640 1 -- 172.21.15.123:0/723992358 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v101) ==== 170+0+0 (secure 0 0 0) 0x7f30e005f250 con 0x7f30f00ca650 2024-09-16T15:48:25.424 INFO:teuthology.orchestra.run.smithi123.stderr:pool 'cephfs_data' created 2024-09-16T15:48:25.428 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:25.427+0000 7f30f5a21640 1 -- 172.21.15.123:0/723992358 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f30dc072fe0 msgr2=0x7f30dc0754a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:48:25.428 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:25.427+0000 7f30f5a21640 1 --2- 172.21.15.123:0/723992358 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f30dc072fe0 0x7f30dc0754a0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f30e40af000 tx=0x7f30e40195c0 comp rx=0 tx=0).stop 2024-09-16T15:48:25.428 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:25.427+0000 7f30f5a21640 1 -- 172.21.15.123:0/723992358 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f30f00ca650 msgr2=0x7f30f01f59e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:48:25.428 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:25.427+0000 7f30f5a21640 1 --2- 172.21.15.123:0/723992358 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f30f00ca650 0x7f30f01f59e0 secure :-1 s=READY pgs=1057 cs=0 l=1 rev1=1 crypto rx=0x7f30e000c940 tx=0x7f30e000ce00 comp rx=0 tx=0).stop 2024-09-16T15:48:25.428 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:25.427+0000 7f30f5a21640 1 -- 172.21.15.123:0/723992358 shutdown_connections 2024-09-16T15:48:25.428 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:25.427+0000 7f30f5a21640 1 --2- 172.21.15.123:0/723992358 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f30dc072fe0 0x7f30dc0754a0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:25.428 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:25.427+0000 7f30f5a21640 1 --2- 172.21.15.123:0/723992358 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f30f01f8550 0x7f30f01f89e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:25.429 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:25.427+0000 7f30f5a21640 1 --2- 172.21.15.123:0/723992358 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f30f015a170 0x7f30f00c8690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:25.429 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:25.427+0000 7f30f5a21640 1 --2- 172.21.15.123:0/723992358 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f30f00ca650 0x7f30f01f59e0 unknown :-1 s=CLOSED pgs=1057 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:25.429 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:25.427+0000 7f30f5a21640 1 -- 172.21.15.123:0/723992358 >> 172.21.15.123:0/723992358 conn(0x7f30f00b1710 msgr2=0x7f30f006f020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:48:25.429 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:25.427+0000 7f30f5a21640 1 -- 172.21.15.123:0/723992358 shutdown_connections 2024-09-16T15:48:25.429 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:25.427+0000 7f30f5a21640 1 -- 172.21.15.123:0/723992358 wait complete. 2024-09-16T15:48:25.464 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-09-16T15:48:25.591 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:25.591+0000 7f9374435640 1 -- 172.21.15.123:0/3850333894 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f936c0cc1a0 msgr2=0x7f936c15ddc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:48:25.591 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:25.591+0000 7f9374435640 1 --2- 172.21.15.123:0/3850333894 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f936c0cc1a0 0x7f936c15ddc0 secure :-1 s=READY pgs=1059 cs=0 l=1 rev1=1 crypto rx=0x7f9364009bc0 tx=0x7f936401cae0 comp rx=0 tx=0).stop 2024-09-16T15:48:25.591 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:25.591+0000 7f9374435640 1 -- 172.21.15.123:0/3850333894 shutdown_connections 2024-09-16T15:48:25.591 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:25.591+0000 7f9374435640 1 --2- 172.21.15.123:0/3850333894 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f936c0cc1a0 0x7f936c15ddc0 unknown :-1 s=CLOSED pgs=1059 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:25.591 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:25.591+0000 7f9374435640 1 --2- 172.21.15.123:0/3850333894 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f936c0ca650 0x7f936c0cbc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:25.591 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:25.591+0000 7f9374435640 1 -- 172.21.15.123:0/3850333894 >> 172.21.15.123:0/3850333894 conn(0x7f936c0b1710 msgr2=0x7f936c0b1b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:48:25.591 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:25.591+0000 7f9374435640 1 -- 172.21.15.123:0/3850333894 shutdown_connections 2024-09-16T15:48:25.591 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:25.591+0000 7f9374435640 1 -- 172.21.15.123:0/3850333894 wait complete. 2024-09-16T15:48:25.592 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:25.591+0000 7f9374435640 1 Processor -- start 2024-09-16T15:48:25.592 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:25.591+0000 7f9374435640 1 -- start start 2024-09-16T15:48:25.594 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:25.591+0000 7f9374435640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f936c0ca650 0x7f936c156ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:48:25.594 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:25.591+0000 7f9374435640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f936c15a170 0x7f936c0c5710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:48:25.594 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:25.591+0000 7f9374435640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f936c0c6c10 0x7f936c0c5c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:48:25.594 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:25.591+0000 7f9374435640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f936c06eca0 con 0x7f936c0ca650 2024-09-16T15:48:25.594 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:25.591+0000 7f9374435640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f936c06eb20 con 0x7f936c0c6c10 2024-09-16T15:48:25.594 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:25.591+0000 7f9374435640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f936c06ee20 con 0x7f936c15a170 2024-09-16T15:48:25.594 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:25.591+0000 7f93721d1640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f936c0ca650 0x7f936c156ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:48:25.594 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:25.591+0000 7f93721d1640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f936c0ca650 0x7f936c156ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:47386/0 (socket says 172.21.15.123:47386) 2024-09-16T15:48:25.595 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:25.591+0000 7f93721d1640 1 -- 172.21.15.123:0/954860913 learned_addr learned my addr 172.21.15.123:0/954860913 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:48:25.595 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:25.591+0000 7f93719d0640 1 --2- 172.21.15.123:0/954860913 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f936c15a170 0x7f936c0c5710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:48:25.595 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:25.591+0000 7f93729d2640 1 --2- 172.21.15.123:0/954860913 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f936c0c6c10 0x7f936c0c5c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:48:25.595 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:25.591+0000 7f93721d1640 1 -- 172.21.15.123:0/954860913 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f936c15a170 msgr2=0x7f936c0c5710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:48:25.595 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:25.591+0000 7f93721d1640 1 --2- 172.21.15.123:0/954860913 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f936c15a170 0x7f936c0c5710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:25.595 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:25.591+0000 7f93721d1640 1 -- 172.21.15.123:0/954860913 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f936c0c6c10 msgr2=0x7f936c0c5c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:48:25.595 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:25.591+0000 7f93721d1640 1 --2- 172.21.15.123:0/954860913 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f936c0c6c10 0x7f936c0c5c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:25.595 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:25.591+0000 7f93721d1640 1 -- 172.21.15.123:0/954860913 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f936c1f98c0 con 0x7f936c0ca650 2024-09-16T15:48:25.595 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:25.591+0000 7f93721d1640 1 --2- 172.21.15.123:0/954860913 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f936c0ca650 0x7f936c156ce0 secure :-1 s=READY pgs=1060 cs=0 l=1 rev1=1 crypto rx=0x7f936800c400 tx=0x7f936800c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:48:25.595 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:25.591+0000 7f9362ffd640 1 -- 172.21.15.123:0/954860913 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9368017070 con 0x7f936c0ca650 2024-09-16T15:48:25.595 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:25.591+0000 7f9362ffd640 1 -- 172.21.15.123:0/954860913 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f9368003600 con 0x7f936c0ca650 2024-09-16T15:48:25.595 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:25.591+0000 7f9374435640 1 -- 172.21.15.123:0/954860913 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f936c1f8b00 con 0x7f936c0ca650 2024-09-16T15:48:25.595 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:25.591+0000 7f9362ffd640 1 -- 172.21.15.123:0/954860913 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9368003a50 con 0x7f936c0ca650 2024-09-16T15:48:25.595 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:25.591+0000 7f9374435640 1 -- 172.21.15.123:0/954860913 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f936c1f90c0 con 0x7f936c0ca650 2024-09-16T15:48:25.596 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:25.595+0000 7f9374435640 1 -- 172.21.15.123:0/954860913 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f936c0cbc60 con 0x7f936c0ca650 2024-09-16T15:48:25.596 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:25.595+0000 7f9362ffd640 1 -- 172.21.15.123:0/954860913 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 16) ==== 97182+0+0 (secure 0 0 0) 0x7f936801a070 con 0x7f936c0ca650 2024-09-16T15:48:25.596 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:25.595+0000 7f9362ffd640 1 --2- 172.21.15.123:0/954860913 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f9350072fe0 0x7f93500754a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:48:25.596 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:25.595+0000 7f93719d0640 1 --2- 172.21.15.123:0/954860913 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f9350072fe0 0x7f93500754a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:48:25.596 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:25.595+0000 7f9362ffd640 1 -- 172.21.15.123:0/954860913 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(101..101 src has 1..101) ==== 5609+0+0 (secure 0 0 0) 0x7f9368093770 con 0x7f936c0ca650 2024-09-16T15:48:25.596 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:25.595+0000 7f93719d0640 1 --2- 172.21.15.123:0/954860913 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f9350072fe0 0x7f93500754a0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f936c15c400 tx=0x7f935c005cd0 comp rx=0 tx=0).ready entity=mgr.4150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:48:25.599 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:25.599+0000 7f9362ffd640 1 -- 172.21.15.123:0/954860913 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f9368021020 con 0x7f936c0ca650 2024-09-16T15:48:25.810 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:25.811+0000 7f9374435640 1 -- 172.21.15.123:0/954860913 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) -- 0x7f936c1645a0 con 0x7f936c0ca650 2024-09-16T15:48:26.471 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:26.471+0000 7f9362ffd640 1 -- 172.21.15.123:0/954860913 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 16 and data pool 17 v64) ==== 174+0+0 (secure 0 0 0) 0x7f936805e890 con 0x7f936c0ca650 2024-09-16T15:48:26.475 INFO:teuthology.orchestra.run.smithi123.stderr:new fs with metadata pool 16 and data pool 17 2024-09-16T15:48:26.480 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:26.479+0000 7f9374435640 1 -- 172.21.15.123:0/954860913 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f9350072fe0 msgr2=0x7f93500754a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:48:26.480 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:26.479+0000 7f9374435640 1 --2- 172.21.15.123:0/954860913 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f9350072fe0 0x7f93500754a0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f936c15c400 tx=0x7f935c005cd0 comp rx=0 tx=0).stop 2024-09-16T15:48:26.480 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:26.479+0000 7f9374435640 1 -- 172.21.15.123:0/954860913 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f936c0ca650 msgr2=0x7f936c156ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:48:26.480 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:26.479+0000 7f9374435640 1 --2- 172.21.15.123:0/954860913 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f936c0ca650 0x7f936c156ce0 secure :-1 s=READY pgs=1060 cs=0 l=1 rev1=1 crypto rx=0x7f936800c400 tx=0x7f936800c8c0 comp rx=0 tx=0).stop 2024-09-16T15:48:26.480 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:26.479+0000 7f9374435640 1 -- 172.21.15.123:0/954860913 shutdown_connections 2024-09-16T15:48:26.480 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:26.479+0000 7f9374435640 1 --2- 172.21.15.123:0/954860913 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f9350072fe0 0x7f93500754a0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:26.480 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:26.479+0000 7f9374435640 1 --2- 172.21.15.123:0/954860913 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f936c0c6c10 0x7f936c0c5c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:26.480 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:26.479+0000 7f9374435640 1 --2- 172.21.15.123:0/954860913 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f936c15a170 0x7f936c0c5710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:26.480 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:26.479+0000 7f9374435640 1 --2- 172.21.15.123:0/954860913 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f936c0ca650 0x7f936c156ce0 unknown :-1 s=CLOSED pgs=1060 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:26.480 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:26.479+0000 7f9374435640 1 -- 172.21.15.123:0/954860913 >> 172.21.15.123:0/954860913 conn(0x7f936c0b1710 msgr2=0x7f936c165290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:48:26.481 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:26.479+0000 7f9374435640 1 -- 172.21.15.123:0/954860913 shutdown_connections 2024-09-16T15:48:26.481 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:26.479+0000 7f9374435640 1 -- 172.21.15.123:0/954860913 wait complete. 2024-09-16T15:48:26.516 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-16T15:48:26.635 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:26.635+0000 7fcd8ae69640 1 -- 172.21.15.123:0/245873272 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcd8415e160 msgr2=0x7fcd84073f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:48:26.636 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:26.635+0000 7fcd8ae69640 1 --2- 172.21.15.123:0/245873272 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcd8415e160 0x7fcd84073f30 secure :-1 s=READY pgs=1061 cs=0 l=1 rev1=1 crypto rx=0x7fcd840c3570 tx=0x7fcd7401c570 comp rx=0 tx=0).stop 2024-09-16T15:48:26.636 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:26.635+0000 7fcd8ae69640 1 -- 172.21.15.123:0/245873272 shutdown_connections 2024-09-16T15:48:26.636 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:26.635+0000 7fcd8ae69640 1 --2- 172.21.15.123:0/245873272 >> v2:172.21.15.184:3300/0 conn(0x7fcd840cdc20 0x7fcd840ca860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:26.636 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:26.635+0000 7fcd8ae69640 1 --2- 172.21.15.123:0/245873272 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fcd84074470 0x7fcd840cd6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:26.636 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:26.635+0000 7fcd8ae69640 1 --2- 172.21.15.123:0/245873272 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcd8415e160 0x7fcd84073f30 unknown :-1 s=CLOSED pgs=1061 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:26.636 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:26.635+0000 7fcd8ae69640 1 -- 172.21.15.123:0/245873272 >> 172.21.15.123:0/245873272 conn(0x7fcd840cfb10 msgr2=0x7fcd840cff20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:48:26.636 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:26.635+0000 7fcd8ae69640 1 -- 172.21.15.123:0/245873272 shutdown_connections 2024-09-16T15:48:26.636 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:26.635+0000 7fcd8ae69640 1 -- 172.21.15.123:0/245873272 wait complete. 2024-09-16T15:48:26.637 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:26.635+0000 7fcd8ae69640 1 Processor -- start 2024-09-16T15:48:26.637 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:26.635+0000 7fcd8ae69640 1 -- start start 2024-09-16T15:48:26.638 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:26.635+0000 7fcd8ae69640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fcd84074470 0x7fcd840c29f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:48:26.639 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:26.635+0000 7fcd8ae69640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcd840cdc20 0x7fcd8415a630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:48:26.639 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:26.635+0000 7fcd8ae69640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fcd8415e160 0x7fcd840c92b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:48:26.639 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:26.635+0000 7fcd8ae69640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fcd8415ebb0 con 0x7fcd840cdc20 2024-09-16T15:48:26.639 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:26.635+0000 7fcd8ae69640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fcd8415ea30 con 0x7fcd84074470 2024-09-16T15:48:26.639 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:26.635+0000 7fcd8ae69640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fcd8415ed30 con 0x7fcd8415e160 2024-09-16T15:48:26.639 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:26.635+0000 7fcd83fff640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcd840cdc20 0x7fcd8415a630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:48:26.639 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:26.639+0000 7fcd83fff640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcd840cdc20 0x7fcd8415a630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:47408/0 (socket says 172.21.15.123:47408) 2024-09-16T15:48:26.639 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:26.639+0000 7fcd83fff640 1 -- 172.21.15.123:0/853617022 learned_addr learned my addr 172.21.15.123:0/853617022 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:48:26.639 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:26.639+0000 7fcd89406640 1 --2- 172.21.15.123:0/853617022 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fcd8415e160 0x7fcd840c92b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:48:26.639 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:26.639+0000 7fcd83fff640 1 -- 172.21.15.123:0/853617022 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fcd8415e160 msgr2=0x7fcd840c92b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:48:26.639 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:26.639+0000 7fcd83fff640 1 --2- 172.21.15.123:0/853617022 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fcd8415e160 0x7fcd840c92b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:26.639 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:26.639+0000 7fcd88c05640 1 --2- 172.21.15.123:0/853617022 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fcd84074470 0x7fcd840c29f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:48:26.640 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:26.639+0000 7fcd83fff640 1 -- 172.21.15.123:0/853617022 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fcd84074470 msgr2=0x7fcd840c29f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:48:26.640 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:26.639+0000 7fcd83fff640 1 --2- 172.21.15.123:0/853617022 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fcd84074470 0x7fcd840c29f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:26.640 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:26.639+0000 7fcd83fff640 1 -- 172.21.15.123:0/853617022 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcd840c8810 con 0x7fcd840cdc20 2024-09-16T15:48:26.640 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:26.639+0000 7fcd89406640 1 --2- 172.21.15.123:0/853617022 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fcd8415e160 0x7fcd840c92b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T15:48:26.640 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:26.639+0000 7fcd83fff640 1 --2- 172.21.15.123:0/853617022 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcd840cdc20 0x7fcd8415a630 secure :-1 s=READY pgs=1062 cs=0 l=1 rev1=1 crypto rx=0x7fcd7800b480 tx=0x7fcd7800b940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:48:26.640 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:26.639+0000 7fcd817fa640 1 -- 172.21.15.123:0/853617022 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd78014070 con 0x7fcd840cdc20 2024-09-16T15:48:26.640 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:26.639+0000 7fcd817fa640 1 -- 172.21.15.123:0/853617022 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fcd78010980 con 0x7fcd840cdc20 2024-09-16T15:48:26.640 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:26.639+0000 7fcd8ae69640 1 -- 172.21.15.123:0/853617022 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcd8415d650 con 0x7fcd840cdc20 2024-09-16T15:48:26.640 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:26.639+0000 7fcd817fa640 1 -- 172.21.15.123:0/853617022 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd78010dd0 con 0x7fcd840cdc20 2024-09-16T15:48:26.640 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:26.639+0000 7fcd8ae69640 1 -- 172.21.15.123:0/853617022 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcd840c8d90 con 0x7fcd840cdc20 2024-09-16T15:48:26.641 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:26.639+0000 7fcd8ae69640 1 -- 172.21.15.123:0/853617022 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcd840ca860 con 0x7fcd840cdc20 2024-09-16T15:48:26.641 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:26.639+0000 7fcd817fa640 1 -- 172.21.15.123:0/853617022 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 16) ==== 97182+0+0 (secure 0 0 0) 0x7fcd78017070 con 0x7fcd840cdc20 2024-09-16T15:48:26.641 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:26.639+0000 7fcd817fa640 1 --2- 172.21.15.123:0/853617022 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fcd70072fe0 0x7fcd700754a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:48:26.641 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:26.639+0000 7fcd88c05640 1 --2- 172.21.15.123:0/853617022 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fcd70072fe0 0x7fcd700754a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:48:26.641 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:26.639+0000 7fcd817fa640 1 -- 172.21.15.123:0/853617022 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(102..102 src has 1..102) ==== 5709+0+0 (secure 0 0 0) 0x7fcd78093e50 con 0x7fcd840cdc20 2024-09-16T15:48:26.642 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:26.639+0000 7fcd88c05640 1 --2- 172.21.15.123:0/853617022 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fcd70072fe0 0x7fcd700754a0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fcd740af000 tx=0x7fcd7401adb0 comp rx=0 tx=0).ready entity=mgr.4150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:48:26.645 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:26.643+0000 7fcd817fa640 1 -- 172.21.15.123:0/853617022 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7fcd7805ef10 con 0x7fcd840cdc20 2024-09-16T15:48:26.817 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:26.815+0000 7fcd8ae69640 1 -- 172.21.15.123:0/853617022 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fcd840d06c0 con 0x7fcd840cdc20 2024-09-16T15:48:26.820 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:26.819+0000 7fcd817fa640 1 -- 172.21.15.123:0/853617022 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v102) ==== 74+0+16760 (secure 0 0 0) 0x7fcd7805e810 con 0x7fcd840cdc20 2024-09-16T15:48:26.820 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-09-16T15:48:26.820 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":102,"fsid":"56077f89-18c8-486f-ae7e-eecc0bb7822c","created":"2024-09-16T15:42:26.364892+0000","modified":"2024-09-16T15:48:25.813401+0000","last_up_change":"2024-09-16T15:42:38.707234+0000","last_in_change":"2024-09-16T15:42:29.423662+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":17,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-16T15:42:39.287764+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_metadata","create_time":"2024-09-16T15:48:23.417905+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"102","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":17,"pool_name":"cephfs_data","create_time":"2024-09-16T15:48:24.776060+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"102","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9e7c1955-a607-4671-9be7-f2ef760ceeaa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6800","nonce":3882044969},{"type":"v1","addr":"172.21.15.123:6801","nonce":3882044969}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6802","nonce":3882044969},{"type":"v1","addr":"172.21.15.123:6803","nonce":3882044969}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6806","nonce":3882044969},{"type":"v1","addr":"172.21.15.123:6807","nonce":3882044969}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6804","nonce":3882044969},{"type":"v1","addr":"172.21.15.123:6805","nonce":3882044969}]},"public_addr":"172.21.15.123:6801/3882044969","cluster_addr":"172.21.15.123:6803/3882044969","heartbeat_back_addr":"172.21.15.123:6807/3882044969","heartbeat_front_addr":"172.21.15.123:6805/3882044969","state":["exists","up"]},{"osd":1,"uuid":"c540cfde-0977-4742-9b17-fd634668e010","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6816","nonce":4037001850},{"type":"v1","addr":"172.21.15.123:6817","nonce":4037001850}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6818","nonce":4037001850},{"type":"v1","addr":"172.21.15.123:6819","nonce":4037001850}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6822","nonce":4037001850},{"type":"v1","addr":"172.21.15.123:6823","nonce":4037001850}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6820","nonce":4037001850},{"type":"v1","addr":"172.21.15.123:6821","nonce":4037001850}]},"public_addr":"172.21.15.123:6817/4037001850","cluster_addr":"172.21.15.123:6819/4037001850","heartbeat_back_addr":"172.21.15.123:6823/4037001850","heartbeat_front_addr":"172.21.15.123:6821/4037001850","state":["exists","up"]},{"osd":2,"uuid":"9864a496-f169-486b-95c4-1d59162793a7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6824","nonce":1332143088},{"type":"v1","addr":"172.21.15.123:6825","nonce":1332143088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6826","nonce":1332143088},{"type":"v1","addr":"172.21.15.123:6827","nonce":1332143088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6830","nonce":1332143088},{"type":"v1","addr":"172.21.15.123:6831","nonce":1332143088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6828","nonce":1332143088},{"type":"v1","addr":"172.21.15.123:6829","nonce":1332143088}]},"public_addr":"172.21.15.123:6825/1332143088","cluster_addr":"172.21.15.123:6827/1332143088","heartbeat_back_addr":"172.21.15.123:6831/1332143088","heartbeat_front_addr":"172.21.15.123:6829/1332143088","state":["exists","up"]},{"osd":3,"uuid":"c6d1f817-f074-4325-95e1-578eba13edda","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6808","nonce":988586998},{"type":"v1","addr":"172.21.15.123:6809","nonce":988586998}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6810","nonce":988586998},{"type":"v1","addr":"172.21.15.123:6811","nonce":988586998}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6814","nonce":988586998},{"type":"v1","addr":"172.21.15.123:6815","nonce":988586998}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6812","nonce":988586998},{"type":"v1","addr":"172.21.15.123:6813","nonce":988586998}]},"public_addr":"172.21.15.123:6809/988586998","cluster_addr":"172.21.15.123:6811/988586998","heartbeat_back_addr":"172.21.15.123:6815/988586998","heartbeat_front_addr":"172.21.15.123:6813/988586998","state":["exists","up"]},{"osd":4,"uuid":"200a8df4-50ef-40a5-b48a-58981269c086","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6816","nonce":3387813890},{"type":"v1","addr":"172.21.15.184:6817","nonce":3387813890}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6818","nonce":3387813890},{"type":"v1","addr":"172.21.15.184:6819","nonce":3387813890}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6822","nonce":3387813890},{"type":"v1","addr":"172.21.15.184:6823","nonce":3387813890}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6820","nonce":3387813890},{"type":"v1","addr":"172.21.15.184:6821","nonce":3387813890}]},"public_addr":"172.21.15.184:6817/3387813890","cluster_addr":"172.21.15.184:6819/3387813890","heartbeat_back_addr":"172.21.15.184:6823/3387813890","heartbeat_front_addr":"172.21.15.184:6821/3387813890","state":["exists","up"]},{"osd":5,"uuid":"32d5a900-da3d-412c-bc19-13861a2368d8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6808","nonce":2680916017},{"type":"v1","addr":"172.21.15.184:6809","nonce":2680916017}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6810","nonce":2680916017},{"type":"v1","addr":"172.21.15.184:6811","nonce":2680916017}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6814","nonce":2680916017},{"type":"v1","addr":"172.21.15.184:6815","nonce":2680916017}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6812","nonce":2680916017},{"type":"v1","addr":"172.21.15.184:6813","nonce":2680916017}]},"public_addr":"172.21.15.184:6809/2680916017","cluster_addr":"172.21.15.184:6811/2680916017","heartbeat_back_addr":"172.21.15.184:6815/2680916017","heartbeat_front_addr":"172.21.15.184:6813/2680916017","state":["exists","up"]},{"osd":6,"uuid":"cde6f39e-aa1b-4260-ac19-774ee6aaca72","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6824","nonce":2183120457},{"type":"v1","addr":"172.21.15.184:6825","nonce":2183120457}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6826","nonce":2183120457},{"type":"v1","addr":"172.21.15.184:6827","nonce":2183120457}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6830","nonce":2183120457},{"type":"v1","addr":"172.21.15.184:6831","nonce":2183120457}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6828","nonce":2183120457},{"type":"v1","addr":"172.21.15.184:6829","nonce":2183120457}]},"public_addr":"172.21.15.184:6825/2183120457","cluster_addr":"172.21.15.184:6827/2183120457","heartbeat_back_addr":"172.21.15.184:6831/2183120457","heartbeat_front_addr":"172.21.15.184:6829/2183120457","state":["exists","up"]},{"osd":7,"uuid":"4f381778-eee1-47c6-804f-df0e6b33e313","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6800","nonce":4102957285},{"type":"v1","addr":"172.21.15.184:6801","nonce":4102957285}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6802","nonce":4102957285},{"type":"v1","addr":"172.21.15.184:6803","nonce":4102957285}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6806","nonce":4102957285},{"type":"v1","addr":"172.21.15.184:6807","nonce":4102957285}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6804","nonce":4102957285},{"type":"v1","addr":"172.21.15.184:6805","nonce":4102957285}]},"public_addr":"172.21.15.184:6801/4102957285","cluster_addr":"172.21.15.184:6803/4102957285","heartbeat_back_addr":"172.21.15.184:6807/4102957285","heartbeat_front_addr":"172.21.15.184:6805/4102957285","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:35.972210+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.222007+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.198617+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.385663+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.358935+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.156066+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.752973+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:35.682846+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T15:48:26.823 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:26.823+0000 7fcd8ae69640 1 -- 172.21.15.123:0/853617022 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fcd70072fe0 msgr2=0x7fcd700754a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:48:26.823 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:26.823+0000 7fcd8ae69640 1 --2- 172.21.15.123:0/853617022 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fcd70072fe0 0x7fcd700754a0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fcd740af000 tx=0x7fcd7401adb0 comp rx=0 tx=0).stop 2024-09-16T15:48:26.823 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:26.823+0000 7fcd8ae69640 1 -- 172.21.15.123:0/853617022 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcd840cdc20 msgr2=0x7fcd8415a630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:48:26.823 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:26.823+0000 7fcd8ae69640 1 --2- 172.21.15.123:0/853617022 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcd840cdc20 0x7fcd8415a630 secure :-1 s=READY pgs=1062 cs=0 l=1 rev1=1 crypto rx=0x7fcd7800b480 tx=0x7fcd7800b940 comp rx=0 tx=0).stop 2024-09-16T15:48:26.823 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:26.823+0000 7fcd8ae69640 1 -- 172.21.15.123:0/853617022 shutdown_connections 2024-09-16T15:48:26.823 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:26.823+0000 7fcd8ae69640 1 --2- 172.21.15.123:0/853617022 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fcd70072fe0 0x7fcd700754a0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:26.823 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:26.823+0000 7fcd8ae69640 1 --2- 172.21.15.123:0/853617022 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fcd8415e160 0x7fcd840c92b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:26.823 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:26.823+0000 7fcd8ae69640 1 --2- 172.21.15.123:0/853617022 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcd840cdc20 0x7fcd8415a630 unknown :-1 s=CLOSED pgs=1062 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:26.823 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:26.823+0000 7fcd8ae69640 1 --2- 172.21.15.123:0/853617022 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fcd84074470 0x7fcd840c29f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:26.823 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:26.823+0000 7fcd8ae69640 1 -- 172.21.15.123:0/853617022 >> 172.21.15.123:0/853617022 conn(0x7fcd840cfb10 msgr2=0x7fcd840c5ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:48:26.823 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:26.823+0000 7fcd8ae69640 1 -- 172.21.15.123:0/853617022 shutdown_connections 2024-09-16T15:48:26.823 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:26.823+0000 7fcd8ae69640 1 -- 172.21.15.123:0/853617022 wait complete. 2024-09-16T15:48:26.840 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-16T15:48:26.967 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:26.967+0000 7f20ff502640 1 -- 172.21.15.123:0/3786877533 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f20f815bbc0 msgr2=0x7f20f80b2d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:48:26.967 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:26.967+0000 7f20ff502640 1 --2- 172.21.15.123:0/3786877533 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f20f815bbc0 0x7f20f80b2d80 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f20f4009840 tx=0x7f20f401c7f0 comp rx=0 tx=0).stop 2024-09-16T15:48:26.967 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:26.967+0000 7f20ff502640 1 -- 172.21.15.123:0/3786877533 shutdown_connections 2024-09-16T15:48:26.967 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:26.967+0000 7f20ff502640 1 --2- 172.21.15.123:0/3786877533 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f20f80d22a0 0x7f20f80ae860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:26.967 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:26.967+0000 7f20ff502640 1 --2- 172.21.15.123:0/3786877533 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f20f815bbc0 0x7f20f80b2d80 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:26.967 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:26.967+0000 7f20ff502640 1 -- 172.21.15.123:0/3786877533 >> 172.21.15.123:0/3786877533 conn(0x7f20f809aa40 msgr2=0x7f20f809ae50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:48:26.967 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:26.967+0000 7f20ff502640 1 -- 172.21.15.123:0/3786877533 shutdown_connections 2024-09-16T15:48:26.968 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:26.967+0000 7f20ff502640 1 -- 172.21.15.123:0/3786877533 wait complete. 2024-09-16T15:48:26.968 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:26.967+0000 7f20ff502640 1 Processor -- start 2024-09-16T15:48:26.968 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:26.967+0000 7f20ff502640 1 -- start start 2024-09-16T15:48:26.970 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:26.967+0000 7f20ff502640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f20f80d22a0 0x7f20f8158090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:48:26.970 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:26.967+0000 7f20ff502640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f20f815bbc0 0x7f20f8071810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:48:26.970 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:26.967+0000 7f20ff502640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f20f80afd30 0x7f20f8071d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:48:26.970 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:26.967+0000 7f20ff502640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f20f8169020 con 0x7f20f80afd30 2024-09-16T15:48:26.970 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:26.967+0000 7f20ff502640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f20f8168ea0 con 0x7f20f80d22a0 2024-09-16T15:48:26.970 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:26.967+0000 7f20ff502640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f20f81691a0 con 0x7f20f815bbc0 2024-09-16T15:48:26.970 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:26.967+0000 7f20fda9f640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f20f80afd30 0x7f20f8071d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:48:26.971 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:26.967+0000 7f20fda9f640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f20f80afd30 0x7f20f8071d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:47416/0 (socket says 172.21.15.123:47416) 2024-09-16T15:48:26.971 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:26.967+0000 7f20fda9f640 1 -- 172.21.15.123:0/742841272 learned_addr learned my addr 172.21.15.123:0/742841272 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:48:26.971 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:26.967+0000 7f20fca9d640 1 --2- 172.21.15.123:0/742841272 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f20f815bbc0 0x7f20f8071810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:48:26.971 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:26.967+0000 7f20fd29e640 1 --2- 172.21.15.123:0/742841272 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f20f80d22a0 0x7f20f8158090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:48:26.971 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:26.967+0000 7f20fda9f640 1 -- 172.21.15.123:0/742841272 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f20f815bbc0 msgr2=0x7f20f8071810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:48:26.971 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:26.971+0000 7f20fda9f640 1 --2- 172.21.15.123:0/742841272 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f20f815bbc0 0x7f20f8071810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:26.971 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:26.971+0000 7f20fda9f640 1 -- 172.21.15.123:0/742841272 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f20f80d22a0 msgr2=0x7f20f8158090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:48:26.971 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:26.971+0000 7f20fda9f640 1 --2- 172.21.15.123:0/742841272 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f20f80d22a0 0x7f20f8158090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:26.971 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:26.971+0000 7f20fda9f640 1 -- 172.21.15.123:0/742841272 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f20f80cdb40 con 0x7f20f80afd30 2024-09-16T15:48:26.971 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:26.971+0000 7f20fd29e640 1 --2- 172.21.15.123:0/742841272 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f20f80d22a0 0x7f20f8158090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T15:48:26.971 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:26.971+0000 7f20fda9f640 1 --2- 172.21.15.123:0/742841272 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f20f80afd30 0x7f20f8071d50 secure :-1 s=READY pgs=1063 cs=0 l=1 rev1=1 crypto rx=0x7f20f000ef60 tx=0x7f20f000ef90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:48:26.971 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:26.971+0000 7f20edffb640 1 -- 172.21.15.123:0/742841272 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20f000ce80 con 0x7f20f80afd30 2024-09-16T15:48:26.972 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:26.971+0000 7f20edffb640 1 -- 172.21.15.123:0/742841272 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f20f0004c20 con 0x7f20f80afd30 2024-09-16T15:48:26.972 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:26.971+0000 7f20ff502640 1 -- 172.21.15.123:0/742841272 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f20f8153870 con 0x7f20f80afd30 2024-09-16T15:48:26.972 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:26.971+0000 7f20edffb640 1 -- 172.21.15.123:0/742841272 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20f000b570 con 0x7f20f80afd30 2024-09-16T15:48:26.972 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:26.971+0000 7f20ff502640 1 -- 172.21.15.123:0/742841272 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f20f80cde20 con 0x7f20f80afd30 2024-09-16T15:48:26.973 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:26.971+0000 7f20ff502640 1 -- 172.21.15.123:0/742841272 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f20f80d2ad0 con 0x7f20f80afd30 2024-09-16T15:48:26.973 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:26.971+0000 7f20edffb640 1 -- 172.21.15.123:0/742841272 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 16) ==== 97182+0+0 (secure 0 0 0) 0x7f20f000b7e0 con 0x7f20f80afd30 2024-09-16T15:48:26.973 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:26.971+0000 7f20edffb640 1 --2- 172.21.15.123:0/742841272 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f20dc073010 0x7f20dc0754d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:48:26.973 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:26.971+0000 7f20fd29e640 1 --2- 172.21.15.123:0/742841272 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f20dc073010 0x7f20dc0754d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:48:26.973 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:26.971+0000 7f20edffb640 1 -- 172.21.15.123:0/742841272 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(102..102 src has 1..102) ==== 5709+0+0 (secure 0 0 0) 0x7f20f0094800 con 0x7f20f80afd30 2024-09-16T15:48:26.973 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:26.971+0000 7f20fd29e640 1 --2- 172.21.15.123:0/742841272 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f20dc073010 0x7f20dc0754d0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f20f40af000 tx=0x7f20f401b030 comp rx=0 tx=0).ready entity=mgr.4150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:48:26.977 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:26.975+0000 7f20edffb640 1 -- 172.21.15.123:0/742841272 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f20f005f1f0 con 0x7f20f80afd30 2024-09-16T15:48:27.147 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:27.147+0000 7f20ff502640 1 -- 172.21.15.123:0/742841272 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f20f80b0b00 con 0x7f20f80afd30 2024-09-16T15:48:27.149 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:27.147+0000 7f20edffb640 1 -- 172.21.15.123:0/742841272 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v102) ==== 74+0+16760 (secure 0 0 0) 0x7f20f0063c20 con 0x7f20f80afd30 2024-09-16T15:48:27.149 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-09-16T15:48:27.149 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":102,"fsid":"56077f89-18c8-486f-ae7e-eecc0bb7822c","created":"2024-09-16T15:42:26.364892+0000","modified":"2024-09-16T15:48:25.813401+0000","last_up_change":"2024-09-16T15:42:38.707234+0000","last_in_change":"2024-09-16T15:42:29.423662+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":17,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-16T15:42:39.287764+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_metadata","create_time":"2024-09-16T15:48:23.417905+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"102","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":17,"pool_name":"cephfs_data","create_time":"2024-09-16T15:48:24.776060+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"102","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9e7c1955-a607-4671-9be7-f2ef760ceeaa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6800","nonce":3882044969},{"type":"v1","addr":"172.21.15.123:6801","nonce":3882044969}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6802","nonce":3882044969},{"type":"v1","addr":"172.21.15.123:6803","nonce":3882044969}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6806","nonce":3882044969},{"type":"v1","addr":"172.21.15.123:6807","nonce":3882044969}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6804","nonce":3882044969},{"type":"v1","addr":"172.21.15.123:6805","nonce":3882044969}]},"public_addr":"172.21.15.123:6801/3882044969","cluster_addr":"172.21.15.123:6803/3882044969","heartbeat_back_addr":"172.21.15.123:6807/3882044969","heartbeat_front_addr":"172.21.15.123:6805/3882044969","state":["exists","up"]},{"osd":1,"uuid":"c540cfde-0977-4742-9b17-fd634668e010","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6816","nonce":4037001850},{"type":"v1","addr":"172.21.15.123:6817","nonce":4037001850}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6818","nonce":4037001850},{"type":"v1","addr":"172.21.15.123:6819","nonce":4037001850}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6822","nonce":4037001850},{"type":"v1","addr":"172.21.15.123:6823","nonce":4037001850}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6820","nonce":4037001850},{"type":"v1","addr":"172.21.15.123:6821","nonce":4037001850}]},"public_addr":"172.21.15.123:6817/4037001850","cluster_addr":"172.21.15.123:6819/4037001850","heartbeat_back_addr":"172.21.15.123:6823/4037001850","heartbeat_front_addr":"172.21.15.123:6821/4037001850","state":["exists","up"]},{"osd":2,"uuid":"9864a496-f169-486b-95c4-1d59162793a7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6824","nonce":1332143088},{"type":"v1","addr":"172.21.15.123:6825","nonce":1332143088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6826","nonce":1332143088},{"type":"v1","addr":"172.21.15.123:6827","nonce":1332143088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6830","nonce":1332143088},{"type":"v1","addr":"172.21.15.123:6831","nonce":1332143088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6828","nonce":1332143088},{"type":"v1","addr":"172.21.15.123:6829","nonce":1332143088}]},"public_addr":"172.21.15.123:6825/1332143088","cluster_addr":"172.21.15.123:6827/1332143088","heartbeat_back_addr":"172.21.15.123:6831/1332143088","heartbeat_front_addr":"172.21.15.123:6829/1332143088","state":["exists","up"]},{"osd":3,"uuid":"c6d1f817-f074-4325-95e1-578eba13edda","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6808","nonce":988586998},{"type":"v1","addr":"172.21.15.123:6809","nonce":988586998}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6810","nonce":988586998},{"type":"v1","addr":"172.21.15.123:6811","nonce":988586998}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6814","nonce":988586998},{"type":"v1","addr":"172.21.15.123:6815","nonce":988586998}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6812","nonce":988586998},{"type":"v1","addr":"172.21.15.123:6813","nonce":988586998}]},"public_addr":"172.21.15.123:6809/988586998","cluster_addr":"172.21.15.123:6811/988586998","heartbeat_back_addr":"172.21.15.123:6815/988586998","heartbeat_front_addr":"172.21.15.123:6813/988586998","state":["exists","up"]},{"osd":4,"uuid":"200a8df4-50ef-40a5-b48a-58981269c086","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6816","nonce":3387813890},{"type":"v1","addr":"172.21.15.184:6817","nonce":3387813890}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6818","nonce":3387813890},{"type":"v1","addr":"172.21.15.184:6819","nonce":3387813890}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6822","nonce":3387813890},{"type":"v1","addr":"172.21.15.184:6823","nonce":3387813890}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6820","nonce":3387813890},{"type":"v1","addr":"172.21.15.184:6821","nonce":3387813890}]},"public_addr":"172.21.15.184:6817/3387813890","cluster_addr":"172.21.15.184:6819/3387813890","heartbeat_back_addr":"172.21.15.184:6823/3387813890","heartbeat_front_addr":"172.21.15.184:6821/3387813890","state":["exists","up"]},{"osd":5,"uuid":"32d5a900-da3d-412c-bc19-13861a2368d8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6808","nonce":2680916017},{"type":"v1","addr":"172.21.15.184:6809","nonce":2680916017}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6810","nonce":2680916017},{"type":"v1","addr":"172.21.15.184:6811","nonce":2680916017}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6814","nonce":2680916017},{"type":"v1","addr":"172.21.15.184:6815","nonce":2680916017}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6812","nonce":2680916017},{"type":"v1","addr":"172.21.15.184:6813","nonce":2680916017}]},"public_addr":"172.21.15.184:6809/2680916017","cluster_addr":"172.21.15.184:6811/2680916017","heartbeat_back_addr":"172.21.15.184:6815/2680916017","heartbeat_front_addr":"172.21.15.184:6813/2680916017","state":["exists","up"]},{"osd":6,"uuid":"cde6f39e-aa1b-4260-ac19-774ee6aaca72","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6824","nonce":2183120457},{"type":"v1","addr":"172.21.15.184:6825","nonce":2183120457}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6826","nonce":2183120457},{"type":"v1","addr":"172.21.15.184:6827","nonce":2183120457}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6830","nonce":2183120457},{"type":"v1","addr":"172.21.15.184:6831","nonce":2183120457}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6828","nonce":2183120457},{"type":"v1","addr":"172.21.15.184:6829","nonce":2183120457}]},"public_addr":"172.21.15.184:6825/2183120457","cluster_addr":"172.21.15.184:6827/2183120457","heartbeat_back_addr":"172.21.15.184:6831/2183120457","heartbeat_front_addr":"172.21.15.184:6829/2183120457","state":["exists","up"]},{"osd":7,"uuid":"4f381778-eee1-47c6-804f-df0e6b33e313","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6800","nonce":4102957285},{"type":"v1","addr":"172.21.15.184:6801","nonce":4102957285}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6802","nonce":4102957285},{"type":"v1","addr":"172.21.15.184:6803","nonce":4102957285}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6806","nonce":4102957285},{"type":"v1","addr":"172.21.15.184:6807","nonce":4102957285}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6804","nonce":4102957285},{"type":"v1","addr":"172.21.15.184:6805","nonce":4102957285}]},"public_addr":"172.21.15.184:6801/4102957285","cluster_addr":"172.21.15.184:6803/4102957285","heartbeat_back_addr":"172.21.15.184:6807/4102957285","heartbeat_front_addr":"172.21.15.184:6805/4102957285","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:35.972210+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.222007+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.198617+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.385663+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.358935+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.156066+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.752973+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:35.682846+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T15:48:27.152 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:27.151+0000 7f20ff502640 1 -- 172.21.15.123:0/742841272 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f20dc073010 msgr2=0x7f20dc0754d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:48:27.152 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:27.151+0000 7f20ff502640 1 --2- 172.21.15.123:0/742841272 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f20dc073010 0x7f20dc0754d0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f20f40af000 tx=0x7f20f401b030 comp rx=0 tx=0).stop 2024-09-16T15:48:27.152 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:27.151+0000 7f20ff502640 1 -- 172.21.15.123:0/742841272 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f20f80afd30 msgr2=0x7f20f8071d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:48:27.152 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:27.151+0000 7f20ff502640 1 --2- 172.21.15.123:0/742841272 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f20f80afd30 0x7f20f8071d50 secure :-1 s=READY pgs=1063 cs=0 l=1 rev1=1 crypto rx=0x7f20f000ef60 tx=0x7f20f000ef90 comp rx=0 tx=0).stop 2024-09-16T15:48:27.152 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:27.151+0000 7f20ff502640 1 -- 172.21.15.123:0/742841272 shutdown_connections 2024-09-16T15:48:27.152 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:27.151+0000 7f20ff502640 1 --2- 172.21.15.123:0/742841272 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f20dc073010 0x7f20dc0754d0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:27.152 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:27.151+0000 7f20ff502640 1 --2- 172.21.15.123:0/742841272 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f20f80afd30 0x7f20f8071d50 unknown :-1 s=CLOSED pgs=1063 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:27.152 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:27.151+0000 7f20ff502640 1 --2- 172.21.15.123:0/742841272 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f20f815bbc0 0x7f20f8071810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:27.152 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:27.151+0000 7f20ff502640 1 --2- 172.21.15.123:0/742841272 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f20f80d22a0 0x7f20f8158090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:27.153 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:27.151+0000 7f20ff502640 1 -- 172.21.15.123:0/742841272 >> 172.21.15.123:0/742841272 conn(0x7f20f809aa40 msgr2=0x7f20f8151a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:48:27.153 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:27.151+0000 7f20ff502640 1 -- 172.21.15.123:0/742841272 shutdown_connections 2024-09-16T15:48:27.153 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:27.151+0000 7f20ff502640 1 -- 172.21.15.123:0/742841272 wait complete. 2024-09-16T15:48:27.168 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-09-16T15:48:27.294 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:27.291+0000 7f6faf7de640 1 -- 172.21.15.123:0/458380125 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6fa8072720 msgr2=0x7f6fa80cc900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:48:27.294 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:27.291+0000 7f6faf7de640 1 --2- 172.21.15.123:0/458380125 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6fa8072720 0x7f6fa80cc900 secure :-1 s=READY pgs=1065 cs=0 l=1 rev1=1 crypto rx=0x7f6fa0009bc0 tx=0x7f6fa001cae0 comp rx=0 tx=0).stop 2024-09-16T15:48:27.294 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:27.291+0000 7f6faf7de640 1 -- 172.21.15.123:0/458380125 shutdown_connections 2024-09-16T15:48:27.294 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:27.291+0000 7f6faf7de640 1 --2- 172.21.15.123:0/458380125 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6fa8072720 0x7f6fa80cc900 unknown :-1 s=CLOSED pgs=1065 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:27.294 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:27.291+0000 7f6faf7de640 1 --2- 172.21.15.123:0/458380125 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f6fa8072c70 0x7f6fa80cc3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:27.294 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:27.295+0000 7f6faf7de640 1 -- 172.21.15.123:0/458380125 >> 172.21.15.123:0/458380125 conn(0x7f6fa80b1710 msgr2=0x7f6fa80b1b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:48:27.294 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:27.295+0000 7f6faf7de640 1 -- 172.21.15.123:0/458380125 shutdown_connections 2024-09-16T15:48:27.294 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:27.295+0000 7f6faf7de640 1 -- 172.21.15.123:0/458380125 wait complete. 2024-09-16T15:48:27.295 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:27.295+0000 7f6faf7de640 1 Processor -- start 2024-09-16T15:48:27.295 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:27.295+0000 7f6faf7de640 1 -- start start 2024-09-16T15:48:27.297 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:27.295+0000 7f6faf7de640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f6fa8072720 0x7f6fa80c6c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:48:27.297 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:27.295+0000 7f6faf7de640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6fa8072c70 0x7f6fa8158150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:48:27.297 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:27.295+0000 7f6faf7de640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6fa815a9d0 0x7f6fa8159da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:48:27.297 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:27.295+0000 7f6faf7de640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f6fa8167b70 con 0x7f6fa8072c70 2024-09-16T15:48:27.297 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:27.295+0000 7f6faf7de640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f6fa81679f0 con 0x7f6fa815a9d0 2024-09-16T15:48:27.297 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:27.295+0000 7f6faf7de640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f6fa8167cf0 con 0x7f6fa8072720 2024-09-16T15:48:27.297 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:27.295+0000 7f6facd79640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6fa8072c70 0x7f6fa8158150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:48:27.297 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:27.295+0000 7f6facd79640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6fa8072c70 0x7f6fa8158150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:47436/0 (socket says 172.21.15.123:47436) 2024-09-16T15:48:27.297 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:27.295+0000 7f6facd79640 1 -- 172.21.15.123:0/2706684088 learned_addr learned my addr 172.21.15.123:0/2706684088 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:48:27.298 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:27.295+0000 7f6facd79640 1 -- 172.21.15.123:0/2706684088 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f6fa8072720 msgr2=0x7f6fa80c6c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:48:27.298 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:27.295+0000 7f6facd79640 1 --2- 172.21.15.123:0/2706684088 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f6fa8072720 0x7f6fa80c6c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:27.298 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:27.295+0000 7f6facd79640 1 -- 172.21.15.123:0/2706684088 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6fa815a9d0 msgr2=0x7f6fa8159da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:48:27.298 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:27.295+0000 7f6facd79640 1 --2- 172.21.15.123:0/2706684088 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6fa815a9d0 0x7f6fa8159da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:27.298 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:27.295+0000 7f6facd79640 1 -- 172.21.15.123:0/2706684088 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6fa81fab70 con 0x7f6fa8072c70 2024-09-16T15:48:27.298 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:27.295+0000 7f6facd79640 1 --2- 172.21.15.123:0/2706684088 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6fa8072c70 0x7f6fa8158150 secure :-1 s=READY pgs=1066 cs=0 l=1 rev1=1 crypto rx=0x7f6f9800ca20 tx=0x7f6f9800cee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:48:27.298 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:27.295+0000 7f6f9dffb640 1 -- 172.21.15.123:0/2706684088 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f98007c70 con 0x7f6fa8072c70 2024-09-16T15:48:27.298 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:27.295+0000 7f6f9dffb640 1 -- 172.21.15.123:0/2706684088 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f6f9800f450 con 0x7f6fa8072c70 2024-09-16T15:48:27.298 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:27.295+0000 7f6faf7de640 1 -- 172.21.15.123:0/2706684088 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6fa81fae50 con 0x7f6fa8072c70 2024-09-16T15:48:27.298 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:27.295+0000 7f6f9dffb640 1 -- 172.21.15.123:0/2706684088 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f9800f8a0 con 0x7f6fa8072c70 2024-09-16T15:48:27.298 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:27.295+0000 7f6faf7de640 1 -- 172.21.15.123:0/2706684088 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6fa81fb410 con 0x7f6fa8072c70 2024-09-16T15:48:27.299 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:27.295+0000 7f6faf7de640 1 -- 172.21.15.123:0/2706684088 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6fa806ebe0 con 0x7f6fa8072c70 2024-09-16T15:48:27.299 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:27.299+0000 7f6f9dffb640 1 -- 172.21.15.123:0/2706684088 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 16) ==== 97182+0+0 (secure 0 0 0) 0x7f6f9800fa40 con 0x7f6fa8072c70 2024-09-16T15:48:27.300 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:27.299+0000 7f6f9dffb640 1 --2- 172.21.15.123:0/2706684088 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f6f8c072fe0 0x7f6f8c0754a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:48:27.300 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:27.299+0000 7f6f9dffb640 1 -- 172.21.15.123:0/2706684088 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(102..102 src has 1..102) ==== 5709+0+0 (secure 0 0 0) 0x7f6f980947f0 con 0x7f6fa8072c70 2024-09-16T15:48:27.300 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:27.299+0000 7f6fad57a640 1 --2- 172.21.15.123:0/2706684088 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f6f8c072fe0 0x7f6f8c0754a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:48:27.300 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:27.299+0000 7f6fad57a640 1 --2- 172.21.15.123:0/2706684088 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f6f8c072fe0 0x7f6f8c0754a0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f6fa4007fd0 tx=0x7f6fa40078e0 comp rx=0 tx=0).ready entity=mgr.4150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:48:27.303 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:27.303+0000 7f6f9dffb640 1 -- 172.21.15.123:0/2706684088 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f6f98014030 con 0x7f6fa8072c70 2024-09-16T15:48:27.506 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:27.507+0000 7f6faf7de640 1 -- 172.21.15.123:0/2706684088 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) -- 0x7f6fa80ca9a0 con 0x7f6fa8072c70 2024-09-16T15:48:28.480 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:28.479+0000 7f6f9dffb640 1 -- 172.21.15.123:0/2706684088 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v67) ==== 118+0+0 (secure 0 0 0) 0x7f6f98064e20 con 0x7f6fa8072c70 2024-09-16T15:48:28.485 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:28.483+0000 7f6faf7de640 1 -- 172.21.15.123:0/2706684088 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f6f8c072fe0 msgr2=0x7f6f8c0754a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:48:28.485 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:28.483+0000 7f6faf7de640 1 --2- 172.21.15.123:0/2706684088 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f6f8c072fe0 0x7f6f8c0754a0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f6fa4007fd0 tx=0x7f6fa40078e0 comp rx=0 tx=0).stop 2024-09-16T15:48:28.485 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:28.483+0000 7f6faf7de640 1 -- 172.21.15.123:0/2706684088 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6fa8072c70 msgr2=0x7f6fa8158150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:48:28.485 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:28.483+0000 7f6faf7de640 1 --2- 172.21.15.123:0/2706684088 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6fa8072c70 0x7f6fa8158150 secure :-1 s=READY pgs=1066 cs=0 l=1 rev1=1 crypto rx=0x7f6f9800ca20 tx=0x7f6f9800cee0 comp rx=0 tx=0).stop 2024-09-16T15:48:28.486 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:28.483+0000 7f6faf7de640 1 -- 172.21.15.123:0/2706684088 shutdown_connections 2024-09-16T15:48:28.486 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:28.483+0000 7f6faf7de640 1 --2- 172.21.15.123:0/2706684088 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f6f8c072fe0 0x7f6f8c0754a0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:28.486 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:28.483+0000 7f6faf7de640 1 --2- 172.21.15.123:0/2706684088 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6fa815a9d0 0x7f6fa8159da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:28.486 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:28.483+0000 7f6faf7de640 1 --2- 172.21.15.123:0/2706684088 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6fa8072c70 0x7f6fa8158150 unknown :-1 s=CLOSED pgs=1066 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:28.486 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:28.483+0000 7f6faf7de640 1 --2- 172.21.15.123:0/2706684088 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f6fa8072720 0x7f6fa80c6c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:28.486 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:28.483+0000 7f6faf7de640 1 -- 172.21.15.123:0/2706684088 >> 172.21.15.123:0/2706684088 conn(0x7f6fa80b1710 msgr2=0x7f6fa80d03e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:48:28.486 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:28.483+0000 7f6faf7de640 1 -- 172.21.15.123:0/2706684088 shutdown_connections 2024-09-16T15:48:28.486 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:28.483+0000 7f6faf7de640 1 -- 172.21.15.123:0/2706684088 wait complete. 2024-09-16T15:48:28.505 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-09-16T15:48:28.505 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-09-16T15:48:28.627 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:28.627+0000 7f090beeb640 1 -- 172.21.15.123:0/159124133 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f090406df30 msgr2=0x7f090406e3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:48:28.627 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:28.627+0000 7f090beeb640 1 --2- 172.21.15.123:0/159124133 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f090406df30 0x7f090406e3c0 secure :-1 s=READY pgs=1068 cs=0 l=1 rev1=1 crypto rx=0x7f08fc009840 tx=0x7f08fc01c790 comp rx=0 tx=0).stop 2024-09-16T15:48:28.627 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:28.627+0000 7f090beeb640 1 -- 172.21.15.123:0/159124133 shutdown_connections 2024-09-16T15:48:28.627 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:28.627+0000 7f090beeb640 1 --2- 172.21.15.123:0/159124133 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f090406df30 0x7f090406e3c0 unknown :-1 s=CLOSED pgs=1068 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:28.628 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:28.627+0000 7f090beeb640 1 --2- 172.21.15.123:0/159124133 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f09040cd7c0 0x7f090415c8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:28.628 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:28.627+0000 7f090beeb640 1 -- 172.21.15.123:0/159124133 >> 172.21.15.123:0/159124133 conn(0x7f09040b29f0 msgr2=0x7f09040b2e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:48:28.628 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:28.627+0000 7f090beeb640 1 -- 172.21.15.123:0/159124133 shutdown_connections 2024-09-16T15:48:28.628 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:28.627+0000 7f090beeb640 1 -- 172.21.15.123:0/159124133 wait complete. 2024-09-16T15:48:28.628 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:28.627+0000 7f090beeb640 1 Processor -- start 2024-09-16T15:48:28.628 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:28.627+0000 7f090beeb640 1 -- start start 2024-09-16T15:48:28.630 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:28.627+0000 7f090beeb640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f090406df30 0x7f090414bd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:48:28.630 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:28.627+0000 7f090beeb640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f09040cd7c0 0x7f0904149790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:48:28.630 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:28.627+0000 7f090beeb640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f09040d0610 0x7f090413f310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:48:28.631 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:28.627+0000 7f090beeb640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f0904168aa0 con 0x7f09040d0610 2024-09-16T15:48:28.631 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:28.627+0000 7f090beeb640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f0904168920 con 0x7f09040cd7c0 2024-09-16T15:48:28.631 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:28.627+0000 7f090beeb640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f0904168c20 con 0x7f090406df30 2024-09-16T15:48:28.631 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:28.627+0000 7f090a488640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f09040d0610 0x7f090413f310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:48:28.631 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:28.627+0000 7f090a488640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f09040d0610 0x7f090413f310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:47458/0 (socket says 172.21.15.123:47458) 2024-09-16T15:48:28.631 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:28.627+0000 7f090a488640 1 -- 172.21.15.123:0/3323044472 learned_addr learned my addr 172.21.15.123:0/3323044472 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:48:28.631 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:28.631+0000 7f0909486640 1 --2- 172.21.15.123:0/3323044472 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f09040cd7c0 0x7f0904149790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:48:28.631 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:28.631+0000 7f090a488640 1 -- 172.21.15.123:0/3323044472 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f090406df30 msgr2=0x7f090414bd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:48:28.631 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:28.631+0000 7f0909c87640 1 --2- 172.21.15.123:0/3323044472 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f090406df30 0x7f090414bd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:48:28.631 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:28.631+0000 7f090a488640 1 --2- 172.21.15.123:0/3323044472 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f090406df30 0x7f090414bd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:28.631 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:28.631+0000 7f090a488640 1 -- 172.21.15.123:0/3323044472 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f09040cd7c0 msgr2=0x7f0904149790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:48:28.631 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:28.631+0000 7f090a488640 1 --2- 172.21.15.123:0/3323044472 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f09040cd7c0 0x7f0904149790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:28.631 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:28.631+0000 7f090a488640 1 -- 172.21.15.123:0/3323044472 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0904143440 con 0x7f09040d0610 2024-09-16T15:48:28.631 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:28.631+0000 7f0909486640 1 --2- 172.21.15.123:0/3323044472 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f09040cd7c0 0x7f0904149790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T15:48:28.632 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:28.631+0000 7f0909c87640 1 --2- 172.21.15.123:0/3323044472 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f090406df30 0x7f090414bd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T15:48:28.632 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:28.631+0000 7f090a488640 1 --2- 172.21.15.123:0/3323044472 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f09040d0610 0x7f090413f310 secure :-1 s=READY pgs=1069 cs=0 l=1 rev1=1 crypto rx=0x7f08fc009790 tx=0x7f08fc0a64f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:48:28.632 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:28.631+0000 7f08fa7fc640 1 -- 172.21.15.123:0/3323044472 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08fc0ac070 con 0x7f09040d0610 2024-09-16T15:48:28.632 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:28.631+0000 7f08fa7fc640 1 -- 172.21.15.123:0/3323044472 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f08fc004320 con 0x7f09040d0610 2024-09-16T15:48:28.632 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:28.631+0000 7f090beeb640 1 -- 172.21.15.123:0/3323044472 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f090414d1c0 con 0x7f09040d0610 2024-09-16T15:48:28.632 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:28.631+0000 7f08fa7fc640 1 -- 172.21.15.123:0/3323044472 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08fc0044f0 con 0x7f09040d0610 2024-09-16T15:48:28.632 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:28.631+0000 7f090beeb640 1 -- 172.21.15.123:0/3323044472 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f090415e8e0 con 0x7f09040d0610 2024-09-16T15:48:28.633 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:28.631+0000 7f090beeb640 1 -- 172.21.15.123:0/3323044472 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f090406e3c0 con 0x7f09040d0610 2024-09-16T15:48:28.633 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:28.631+0000 7f08fa7fc640 1 -- 172.21.15.123:0/3323044472 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 16) ==== 97182+0+0 (secure 0 0 0) 0x7f08fc0af070 con 0x7f09040d0610 2024-09-16T15:48:28.633 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:28.631+0000 7f08fa7fc640 1 --2- 172.21.15.123:0/3323044472 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f08e8072fe0 0x7f08e80754a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:48:28.633 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:28.631+0000 7f0909c87640 1 --2- 172.21.15.123:0/3323044472 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f08e8072fe0 0x7f08e80754a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:48:28.633 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:28.631+0000 7f08fa7fc640 1 -- 172.21.15.123:0/3323044472 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(103..103 src has 1..103) ==== 5709+0+0 (secure 0 0 0) 0x7f08fc12e210 con 0x7f09040d0610 2024-09-16T15:48:28.633 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:28.631+0000 7f0909c87640 1 --2- 172.21.15.123:0/3323044472 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f08e8072fe0 0x7f08e80754a0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f0900007fd0 tx=0x7f0900009040 comp rx=0 tx=0).ready entity=mgr.4150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:48:28.637 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:28.635+0000 7f08fa7fc640 1 -- 172.21.15.123:0/3323044472 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f08fc016410 con 0x7f09040d0610 2024-09-16T15:48:28.832 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:28.831+0000 7f090beeb640 1 -- 172.21.15.123:0/3323044472 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) -- 0x7f09041509b0 con 0x7f09040d0610 2024-09-16T15:48:29.491 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:29.491+0000 7f08fa7fc640 1 -- 172.21.15.123:0/3323044472 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v68) ==== 122+0+0 (secure 0 0 0) 0x7f08fc0f9240 con 0x7f09040d0610 2024-09-16T15:48:29.495 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:29.495+0000 7f090beeb640 1 -- 172.21.15.123:0/3323044472 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f08e8072fe0 msgr2=0x7f08e80754a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:48:29.495 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:29.495+0000 7f090beeb640 1 --2- 172.21.15.123:0/3323044472 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f08e8072fe0 0x7f08e80754a0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f0900007fd0 tx=0x7f0900009040 comp rx=0 tx=0).stop 2024-09-16T15:48:29.495 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:29.495+0000 7f090beeb640 1 -- 172.21.15.123:0/3323044472 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f09040d0610 msgr2=0x7f090413f310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:48:29.495 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:29.495+0000 7f090beeb640 1 --2- 172.21.15.123:0/3323044472 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f09040d0610 0x7f090413f310 secure :-1 s=READY pgs=1069 cs=0 l=1 rev1=1 crypto rx=0x7f08fc009790 tx=0x7f08fc0a64f0 comp rx=0 tx=0).stop 2024-09-16T15:48:29.495 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:29.495+0000 7f090beeb640 1 -- 172.21.15.123:0/3323044472 shutdown_connections 2024-09-16T15:48:29.495 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:29.495+0000 7f090beeb640 1 --2- 172.21.15.123:0/3323044472 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f08e8072fe0 0x7f08e80754a0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:29.495 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:29.495+0000 7f090beeb640 1 --2- 172.21.15.123:0/3323044472 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f09040d0610 0x7f090413f310 unknown :-1 s=CLOSED pgs=1069 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:29.496 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:29.495+0000 7f090beeb640 1 --2- 172.21.15.123:0/3323044472 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f09040cd7c0 0x7f0904149790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:29.496 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:29.495+0000 7f090beeb640 1 --2- 172.21.15.123:0/3323044472 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f090406df30 0x7f090414bd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:29.496 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:29.495+0000 7f090beeb640 1 -- 172.21.15.123:0/3323044472 >> 172.21.15.123:0/3323044472 conn(0x7f09040b29f0 msgr2=0x7f09040b2dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:48:29.496 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:29.495+0000 7f090beeb640 1 -- 172.21.15.123:0/3323044472 shutdown_connections 2024-09-16T15:48:29.496 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:29.495+0000 7f090beeb640 1 -- 172.21.15.123:0/3323044472 wait complete. 2024-09-16T15:48:29.519 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-09-16T15:48:29.519 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-16T15:48:29.654 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:29.651+0000 7f3bdf14e640 1 -- 172.21.15.123:0/1589446039 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3bd80b3100 msgr2=0x7f3bd80cc1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:48:29.654 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:29.651+0000 7f3bdf14e640 1 --2- 172.21.15.123:0/1589446039 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3bd80b3100 0x7f3bd80cc1b0 secure :-1 s=READY pgs=1070 cs=0 l=1 rev1=1 crypto rx=0x7f3bcc009900 tx=0x7f3bcc01c8e0 comp rx=0 tx=0).stop 2024-09-16T15:48:29.654 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:29.651+0000 7f3bdf14e640 1 -- 172.21.15.123:0/1589446039 shutdown_connections 2024-09-16T15:48:29.654 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:29.651+0000 7f3bdf14e640 1 --2- 172.21.15.123:0/1589446039 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3bd80b3100 0x7f3bd80cc1b0 unknown :-1 s=CLOSED pgs=1070 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:29.654 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:29.651+0000 7f3bdf14e640 1 --2- 172.21.15.123:0/1589446039 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f3bd815ac00 0x7f3bd80702f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:29.654 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:29.655+0000 7f3bdf14e640 1 -- 172.21.15.123:0/1589446039 >> 172.21.15.123:0/1589446039 conn(0x7f3bd80b2610 msgr2=0x7f3bd80b29f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:48:29.654 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:29.655+0000 7f3bdf14e640 1 -- 172.21.15.123:0/1589446039 shutdown_connections 2024-09-16T15:48:29.654 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:29.655+0000 7f3bdf14e640 1 -- 172.21.15.123:0/1589446039 wait complete. 2024-09-16T15:48:29.655 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:29.655+0000 7f3bdf14e640 1 Processor -- start 2024-09-16T15:48:29.655 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:29.655+0000 7f3bdf14e640 1 -- start start 2024-09-16T15:48:29.657 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:29.655+0000 7f3bdf14e640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3bd8070830 0x7f3bd8154e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:48:29.657 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:29.655+0000 7f3bdf14e640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f3bd80b3100 0x7f3bd80c6650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:48:29.657 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:29.655+0000 7f3bdf14e640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3bd815ac00 0x7f3bd80b5b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:48:29.657 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:29.655+0000 7f3bdf14e640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f3bd8168660 con 0x7f3bd815ac00 2024-09-16T15:48:29.657 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:29.655+0000 7f3bdf14e640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f3bd81684e0 con 0x7f3bd8070830 2024-09-16T15:48:29.657 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:29.655+0000 7f3bdf14e640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f3bd81687e0 con 0x7f3bd80b3100 2024-09-16T15:48:29.657 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:29.655+0000 7f3bdd6eb640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3bd815ac00 0x7f3bd80b5b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:48:29.657 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:29.655+0000 7f3bdd6eb640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3bd815ac00 0x7f3bd80b5b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:47474/0 (socket says 172.21.15.123:47474) 2024-09-16T15:48:29.657 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:29.655+0000 7f3bdd6eb640 1 -- 172.21.15.123:0/211646979 learned_addr learned my addr 172.21.15.123:0/211646979 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:48:29.657 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:29.655+0000 7f3bdceea640 1 --2- 172.21.15.123:0/211646979 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3bd8070830 0x7f3bd8154e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:48:29.658 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:29.655+0000 7f3bd7fff640 1 --2- 172.21.15.123:0/211646979 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f3bd80b3100 0x7f3bd80c6650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:48:29.658 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:29.655+0000 7f3bdd6eb640 1 -- 172.21.15.123:0/211646979 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f3bd80b3100 msgr2=0x7f3bd80c6650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:48:29.658 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:29.655+0000 7f3bdd6eb640 1 --2- 172.21.15.123:0/211646979 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f3bd80b3100 0x7f3bd80c6650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:29.658 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:29.655+0000 7f3bdd6eb640 1 -- 172.21.15.123:0/211646979 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3bd8070830 msgr2=0x7f3bd8154e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:48:29.658 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:29.655+0000 7f3bdd6eb640 1 --2- 172.21.15.123:0/211646979 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3bd8070830 0x7f3bd8154e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:29.658 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:29.655+0000 7f3bdd6eb640 1 -- 172.21.15.123:0/211646979 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3bd81f9a50 con 0x7f3bd815ac00 2024-09-16T15:48:29.658 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:29.655+0000 7f3bdd6eb640 1 --2- 172.21.15.123:0/211646979 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3bd815ac00 0x7f3bd80b5b60 secure :-1 s=READY pgs=1071 cs=0 l=1 rev1=1 crypto rx=0x7f3bd000c780 tx=0x7f3bd000cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:48:29.658 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:29.655+0000 7f3bd57fa640 1 -- 172.21.15.123:0/211646979 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3bd0016020 con 0x7f3bd815ac00 2024-09-16T15:48:29.658 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:29.655+0000 7f3bd57fa640 1 -- 172.21.15.123:0/211646979 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f3bd000aae0 con 0x7f3bd815ac00 2024-09-16T15:48:29.658 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:29.655+0000 7f3bdf14e640 1 -- 172.21.15.123:0/211646979 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3bd81f9cf0 con 0x7f3bd815ac00 2024-09-16T15:48:29.658 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:29.655+0000 7f3bd57fa640 1 -- 172.21.15.123:0/211646979 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3bd0012420 con 0x7f3bd815ac00 2024-09-16T15:48:29.658 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:29.655+0000 7f3bdf14e640 1 -- 172.21.15.123:0/211646979 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3bd81fa590 con 0x7f3bd815ac00 2024-09-16T15:48:29.659 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:29.659+0000 7f3bdf14e640 1 -- 172.21.15.123:0/211646979 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3bd80cc1b0 con 0x7f3bd815ac00 2024-09-16T15:48:29.659 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:29.659+0000 7f3bd57fa640 1 -- 172.21.15.123:0/211646979 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 16) ==== 97182+0+0 (secure 0 0 0) 0x7f3bd0019020 con 0x7f3bd815ac00 2024-09-16T15:48:29.659 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:29.659+0000 7f3bd57fa640 1 --2- 172.21.15.123:0/211646979 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f3bc4072fe0 0x7f3bc40754a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:48:29.659 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:29.659+0000 7f3bdceea640 1 --2- 172.21.15.123:0/211646979 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f3bc4072fe0 0x7f3bc40754a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:48:29.659 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:29.659+0000 7f3bd57fa640 1 -- 172.21.15.123:0/211646979 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(103..103 src has 1..103) ==== 5709+0+0 (secure 0 0 0) 0x7f3bd0093c30 con 0x7f3bd815ac00 2024-09-16T15:48:29.659 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:29.659+0000 7f3bdceea640 1 --2- 172.21.15.123:0/211646979 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f3bc4072fe0 0x7f3bc40754a0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f3bc8005fd0 tx=0x7f3bc800c460 comp rx=0 tx=0).ready entity=mgr.4150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:48:29.663 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:29.663+0000 7f3bd57fa640 1 -- 172.21.15.123:0/211646979 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f3bd0020070 con 0x7f3bd815ac00 2024-09-16T15:48:29.867 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:29.867+0000 7f3bdf14e640 1 -- 172.21.15.123:0/211646979 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f3bd81f9ec0 con 0x7f3bd815ac00 2024-09-16T15:48:29.868 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:29.867+0000 7f3bd57fa640 1 -- 172.21.15.123:0/211646979 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 68 v68) ==== 94+0+5237 (secure 0 0 0) 0x7f3bd005ecf0 con 0x7f3bd815ac00 2024-09-16T15:48:29.868 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-09-16T15:48:29.868 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":68,"btime":"2024-09-16T15:48:29:489616+0000","default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5417,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6835/3411324594","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":3411324594},{"type":"v1","addr":"172.21.15.184:6835","nonce":3411324594}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":63},{"gid":5848,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6833/1488440148","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":1488440148},{"type":"v1","addr":"172.21.15.184:6833","nonce":1488440148}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":63},{"gid":7032,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6833/1841102587","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6832","nonce":1841102587},{"type":"v1","addr":"172.21.15.123:6833","nonce":1841102587}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":63}],"filesystems":[{"mdsmap":{"epoch":68,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-16T15:48:25.813334+0000","modified":"2024-09-16T15:48:28.834708+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7035},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7035":{"gid":7035,"name":"c","rank":0,"incarnation":64,"state":"up:active","state_seq":3,"addr":"172.21.15.123:6835/2302124587","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6834","nonce":2302124587},{"type":"v1","addr":"172.21.15.123:6835","nonce":2302124587}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7035,"qdb_cluster":[7035]},"id":8}]} 2024-09-16T15:48:29.869 INFO:teuthology.orchestra.run.smithi123.stderr:dumped fsmap epoch 68 2024-09-16T15:48:29.870 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:29.871+0000 7f3bdf14e640 1 -- 172.21.15.123:0/211646979 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f3bc4072fe0 msgr2=0x7f3bc40754a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:48:29.870 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:29.871+0000 7f3bdf14e640 1 --2- 172.21.15.123:0/211646979 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f3bc4072fe0 0x7f3bc40754a0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f3bc8005fd0 tx=0x7f3bc800c460 comp rx=0 tx=0).stop 2024-09-16T15:48:29.871 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:29.871+0000 7f3bdf14e640 1 -- 172.21.15.123:0/211646979 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3bd815ac00 msgr2=0x7f3bd80b5b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:48:29.871 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:29.871+0000 7f3bdf14e640 1 --2- 172.21.15.123:0/211646979 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3bd815ac00 0x7f3bd80b5b60 secure :-1 s=READY pgs=1071 cs=0 l=1 rev1=1 crypto rx=0x7f3bd000c780 tx=0x7f3bd000cc40 comp rx=0 tx=0).stop 2024-09-16T15:48:29.871 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:29.871+0000 7f3bdf14e640 1 -- 172.21.15.123:0/211646979 shutdown_connections 2024-09-16T15:48:29.871 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:29.871+0000 7f3bdf14e640 1 --2- 172.21.15.123:0/211646979 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f3bc4072fe0 0x7f3bc40754a0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:29.871 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:29.871+0000 7f3bdf14e640 1 --2- 172.21.15.123:0/211646979 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3bd815ac00 0x7f3bd80b5b60 unknown :-1 s=CLOSED pgs=1071 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:29.871 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:29.871+0000 7f3bdf14e640 1 --2- 172.21.15.123:0/211646979 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f3bd80b3100 0x7f3bd80c6650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:29.871 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:29.871+0000 7f3bdf14e640 1 --2- 172.21.15.123:0/211646979 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3bd8070830 0x7f3bd8154e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:29.871 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:29.871+0000 7f3bdf14e640 1 -- 172.21.15.123:0/211646979 >> 172.21.15.123:0/211646979 conn(0x7f3bd80b2610 msgr2=0x7f3bd8071480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:48:29.871 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:29.871+0000 7f3bdf14e640 1 -- 172.21.15.123:0/211646979 shutdown_connections 2024-09-16T15:48:29.871 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:29.871+0000 7f3bdf14e640 1 -- 172.21.15.123:0/211646979 wait complete. 2024-09-16T15:48:29.886 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-16T15:48:29.998 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:29.999+0000 7ff81fbfa640 1 -- 172.21.15.123:0/1183223079 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff81815e820 msgr2=0x7ff81815ecd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:48:29.998 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:29.999+0000 7ff81fbfa640 1 --2- 172.21.15.123:0/1183223079 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff81815e820 0x7ff81815ecd0 secure :-1 s=READY pgs=1073 cs=0 l=1 rev1=1 crypto rx=0x7ff810009bc0 tx=0x7ff81001cb10 comp rx=0 tx=0).stop 2024-09-16T15:48:29.998 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:29.999+0000 7ff81fbfa640 1 -- 172.21.15.123:0/1183223079 shutdown_connections 2024-09-16T15:48:29.999 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:29.999+0000 7ff81fbfa640 1 --2- 172.21.15.123:0/1183223079 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff81815e820 0x7ff81815ecd0 unknown :-1 s=CLOSED pgs=1073 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:29.999 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:29.999+0000 7ff81fbfa640 1 --2- 172.21.15.123:0/1183223079 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7ff81815e160 0x7ff818073f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:29.999 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:29.999+0000 7ff81fbfa640 1 -- 172.21.15.123:0/1183223079 >> 172.21.15.123:0/1183223079 conn(0x7ff8180cfb10 msgr2=0x7ff8180cff20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:48:29.999 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:29.999+0000 7ff81fbfa640 1 -- 172.21.15.123:0/1183223079 shutdown_connections 2024-09-16T15:48:29.999 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:29.999+0000 7ff81fbfa640 1 -- 172.21.15.123:0/1183223079 wait complete. 2024-09-16T15:48:29.999 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:29.999+0000 7ff81fbfa640 1 Processor -- start 2024-09-16T15:48:29.999 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:29.999+0000 7ff81fbfa640 1 -- start start 2024-09-16T15:48:30.001 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:29.999+0000 7ff81fbfa640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff818074470 0x7ff81814d200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:48:30.001 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:29.999+0000 7ff81fbfa640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7ff81815e160 0x7ff8181fb9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:48:30.001 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:29.999+0000 7ff81fbfa640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff81815e820 0x7ff81814f0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:48:30.001 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:29.999+0000 7ff81fbfa640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7ff8180cf5b0 con 0x7ff81815e820 2024-09-16T15:48:30.002 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:29.999+0000 7ff81fbfa640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7ff8180cf430 con 0x7ff818074470 2024-09-16T15:48:30.002 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:29.999+0000 7ff81fbfa640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7ff8181663c0 con 0x7ff81815e160 2024-09-16T15:48:30.002 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:29.999+0000 7ff81e197640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff81815e820 0x7ff81814f0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:48:30.002 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:29.999+0000 7ff81e197640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff81815e820 0x7ff81814f0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:47498/0 (socket says 172.21.15.123:47498) 2024-09-16T15:48:30.002 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:29.999+0000 7ff81e197640 1 -- 172.21.15.123:0/2749511439 learned_addr learned my addr 172.21.15.123:0/2749511439 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:48:30.002 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:29.999+0000 7ff81d996640 1 --2- 172.21.15.123:0/2749511439 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff818074470 0x7ff81814d200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:48:30.002 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:29.999+0000 7ff81d195640 1 --2- 172.21.15.123:0/2749511439 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7ff81815e160 0x7ff8181fb9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:48:30.002 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:29.999+0000 7ff81e197640 1 -- 172.21.15.123:0/2749511439 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7ff81815e160 msgr2=0x7ff8181fb9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:48:30.002 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:29.999+0000 7ff81e197640 1 --2- 172.21.15.123:0/2749511439 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7ff81815e160 0x7ff8181fb9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:30.002 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:29.999+0000 7ff81e197640 1 -- 172.21.15.123:0/2749511439 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff818074470 msgr2=0x7ff81814d200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:48:30.002 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:29.999+0000 7ff81e197640 1 --2- 172.21.15.123:0/2749511439 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff818074470 0x7ff81814d200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:30.002 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:29.999+0000 7ff81e197640 1 -- 172.21.15.123:0/2749511439 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff8180d08e0 con 0x7ff81815e820 2024-09-16T15:48:30.002 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:29.999+0000 7ff81e197640 1 --2- 172.21.15.123:0/2749511439 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff81815e820 0x7ff81814f0c0 secure :-1 s=READY pgs=1074 cs=0 l=1 rev1=1 crypto rx=0x7ff8100b0040 tx=0x7ff810005ae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:48:30.002 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:29.999+0000 7ff80e7fc640 1 -- 172.21.15.123:0/2749511439 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8100b2040 con 0x7ff81815e820 2024-09-16T15:48:30.003 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:29.999+0000 7ff80e7fc640 1 -- 172.21.15.123:0/2749511439 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7ff8100041f0 con 0x7ff81815e820 2024-09-16T15:48:30.003 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:29.999+0000 7ff81fbfa640 1 -- 172.21.15.123:0/2749511439 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff81814f600 con 0x7ff81815e820 2024-09-16T15:48:30.003 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:29.999+0000 7ff80e7fc640 1 -- 172.21.15.123:0/2749511439 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff810004400 con 0x7ff81815e820 2024-09-16T15:48:30.003 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:29.999+0000 7ff81fbfa640 1 -- 172.21.15.123:0/2749511439 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff8181671d0 con 0x7ff81815e820 2024-09-16T15:48:30.003 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:30.003+0000 7ff81fbfa640 1 -- 172.21.15.123:0/2749511439 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff818166870 con 0x7ff81815e820 2024-09-16T15:48:30.004 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:30.003+0000 7ff80e7fc640 1 -- 172.21.15.123:0/2749511439 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 16) ==== 97182+0+0 (secure 0 0 0) 0x7ff8100bd030 con 0x7ff81815e820 2024-09-16T15:48:30.004 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:30.003+0000 7ff80e7fc640 1 --2- 172.21.15.123:0/2749511439 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7ff7fc072fe0 0x7ff7fc0754a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:48:30.004 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:30.003+0000 7ff80e7fc640 1 -- 172.21.15.123:0/2749511439 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(103..103 src has 1..103) ==== 5709+0+0 (secure 0 0 0) 0x7ff81012dff0 con 0x7ff81815e820 2024-09-16T15:48:30.004 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:30.003+0000 7ff81d996640 1 --2- 172.21.15.123:0/2749511439 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7ff7fc072fe0 0x7ff7fc0754a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:48:30.004 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:30.003+0000 7ff81d996640 1 --2- 172.21.15.123:0/2749511439 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7ff7fc072fe0 0x7ff7fc0754a0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7ff8140058d0 tx=0x7ff814005e30 comp rx=0 tx=0).ready entity=mgr.4150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:48:30.007 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:30.007+0000 7ff80e7fc640 1 -- 172.21.15.123:0/2749511439 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7ff8100c4060 con 0x7ff81815e820 2024-09-16T15:48:30.176 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:30.175+0000 7ff81fbfa640 1 -- 172.21.15.123:0/2749511439 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7ff818168bb0 con 0x7ff81815e820 2024-09-16T15:48:30.178 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:30.179+0000 7ff80e7fc640 1 -- 172.21.15.123:0/2749511439 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v103) ==== 74+0+16747 (secure 0 0 0) 0x7ff810016790 con 0x7ff81815e820 2024-09-16T15:48:30.179 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-09-16T15:48:30.179 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":103,"fsid":"56077f89-18c8-486f-ae7e-eecc0bb7822c","created":"2024-09-16T15:42:26.364892+0000","modified":"2024-09-16T15:48:27.447327+0000","last_up_change":"2024-09-16T15:42:38.707234+0000","last_in_change":"2024-09-16T15:42:29.423662+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":17,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-16T15:42:39.287764+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_metadata","create_time":"2024-09-16T15:48:23.417905+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"102","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":17,"pool_name":"cephfs_data","create_time":"2024-09-16T15:48:24.776060+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"103","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9e7c1955-a607-4671-9be7-f2ef760ceeaa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6800","nonce":3882044969},{"type":"v1","addr":"172.21.15.123:6801","nonce":3882044969}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6802","nonce":3882044969},{"type":"v1","addr":"172.21.15.123:6803","nonce":3882044969}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6806","nonce":3882044969},{"type":"v1","addr":"172.21.15.123:6807","nonce":3882044969}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6804","nonce":3882044969},{"type":"v1","addr":"172.21.15.123:6805","nonce":3882044969}]},"public_addr":"172.21.15.123:6801/3882044969","cluster_addr":"172.21.15.123:6803/3882044969","heartbeat_back_addr":"172.21.15.123:6807/3882044969","heartbeat_front_addr":"172.21.15.123:6805/3882044969","state":["exists","up"]},{"osd":1,"uuid":"c540cfde-0977-4742-9b17-fd634668e010","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6816","nonce":4037001850},{"type":"v1","addr":"172.21.15.123:6817","nonce":4037001850}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6818","nonce":4037001850},{"type":"v1","addr":"172.21.15.123:6819","nonce":4037001850}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6822","nonce":4037001850},{"type":"v1","addr":"172.21.15.123:6823","nonce":4037001850}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6820","nonce":4037001850},{"type":"v1","addr":"172.21.15.123:6821","nonce":4037001850}]},"public_addr":"172.21.15.123:6817/4037001850","cluster_addr":"172.21.15.123:6819/4037001850","heartbeat_back_addr":"172.21.15.123:6823/4037001850","heartbeat_front_addr":"172.21.15.123:6821/4037001850","state":["exists","up"]},{"osd":2,"uuid":"9864a496-f169-486b-95c4-1d59162793a7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6824","nonce":1332143088},{"type":"v1","addr":"172.21.15.123:6825","nonce":1332143088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6826","nonce":1332143088},{"type":"v1","addr":"172.21.15.123:6827","nonce":1332143088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6830","nonce":1332143088},{"type":"v1","addr":"172.21.15.123:6831","nonce":1332143088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6828","nonce":1332143088},{"type":"v1","addr":"172.21.15.123:6829","nonce":1332143088}]},"public_addr":"172.21.15.123:6825/1332143088","cluster_addr":"172.21.15.123:6827/1332143088","heartbeat_back_addr":"172.21.15.123:6831/1332143088","heartbeat_front_addr":"172.21.15.123:6829/1332143088","state":["exists","up"]},{"osd":3,"uuid":"c6d1f817-f074-4325-95e1-578eba13edda","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6808","nonce":988586998},{"type":"v1","addr":"172.21.15.123:6809","nonce":988586998}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6810","nonce":988586998},{"type":"v1","addr":"172.21.15.123:6811","nonce":988586998}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6814","nonce":988586998},{"type":"v1","addr":"172.21.15.123:6815","nonce":988586998}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6812","nonce":988586998},{"type":"v1","addr":"172.21.15.123:6813","nonce":988586998}]},"public_addr":"172.21.15.123:6809/988586998","cluster_addr":"172.21.15.123:6811/988586998","heartbeat_back_addr":"172.21.15.123:6815/988586998","heartbeat_front_addr":"172.21.15.123:6813/988586998","state":["exists","up"]},{"osd":4,"uuid":"200a8df4-50ef-40a5-b48a-58981269c086","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6816","nonce":3387813890},{"type":"v1","addr":"172.21.15.184:6817","nonce":3387813890}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6818","nonce":3387813890},{"type":"v1","addr":"172.21.15.184:6819","nonce":3387813890}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6822","nonce":3387813890},{"type":"v1","addr":"172.21.15.184:6823","nonce":3387813890}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6820","nonce":3387813890},{"type":"v1","addr":"172.21.15.184:6821","nonce":3387813890}]},"public_addr":"172.21.15.184:6817/3387813890","cluster_addr":"172.21.15.184:6819/3387813890","heartbeat_back_addr":"172.21.15.184:6823/3387813890","heartbeat_front_addr":"172.21.15.184:6821/3387813890","state":["exists","up"]},{"osd":5,"uuid":"32d5a900-da3d-412c-bc19-13861a2368d8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6808","nonce":2680916017},{"type":"v1","addr":"172.21.15.184:6809","nonce":2680916017}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6810","nonce":2680916017},{"type":"v1","addr":"172.21.15.184:6811","nonce":2680916017}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6814","nonce":2680916017},{"type":"v1","addr":"172.21.15.184:6815","nonce":2680916017}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6812","nonce":2680916017},{"type":"v1","addr":"172.21.15.184:6813","nonce":2680916017}]},"public_addr":"172.21.15.184:6809/2680916017","cluster_addr":"172.21.15.184:6811/2680916017","heartbeat_back_addr":"172.21.15.184:6815/2680916017","heartbeat_front_addr":"172.21.15.184:6813/2680916017","state":["exists","up"]},{"osd":6,"uuid":"cde6f39e-aa1b-4260-ac19-774ee6aaca72","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6824","nonce":2183120457},{"type":"v1","addr":"172.21.15.184:6825","nonce":2183120457}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6826","nonce":2183120457},{"type":"v1","addr":"172.21.15.184:6827","nonce":2183120457}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6830","nonce":2183120457},{"type":"v1","addr":"172.21.15.184:6831","nonce":2183120457}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6828","nonce":2183120457},{"type":"v1","addr":"172.21.15.184:6829","nonce":2183120457}]},"public_addr":"172.21.15.184:6825/2183120457","cluster_addr":"172.21.15.184:6827/2183120457","heartbeat_back_addr":"172.21.15.184:6831/2183120457","heartbeat_front_addr":"172.21.15.184:6829/2183120457","state":["exists","up"]},{"osd":7,"uuid":"4f381778-eee1-47c6-804f-df0e6b33e313","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6800","nonce":4102957285},{"type":"v1","addr":"172.21.15.184:6801","nonce":4102957285}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6802","nonce":4102957285},{"type":"v1","addr":"172.21.15.184:6803","nonce":4102957285}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6806","nonce":4102957285},{"type":"v1","addr":"172.21.15.184:6807","nonce":4102957285}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6804","nonce":4102957285},{"type":"v1","addr":"172.21.15.184:6805","nonce":4102957285}]},"public_addr":"172.21.15.184:6801/4102957285","cluster_addr":"172.21.15.184:6803/4102957285","heartbeat_back_addr":"172.21.15.184:6807/4102957285","heartbeat_front_addr":"172.21.15.184:6805/4102957285","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:35.972210+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.222007+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.198617+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.385663+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.358935+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.156066+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.752973+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:35.682846+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T15:48:30.181 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:30.179+0000 7ff81fbfa640 1 -- 172.21.15.123:0/2749511439 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7ff7fc072fe0 msgr2=0x7ff7fc0754a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:48:30.181 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:30.179+0000 7ff81fbfa640 1 --2- 172.21.15.123:0/2749511439 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7ff7fc072fe0 0x7ff7fc0754a0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7ff8140058d0 tx=0x7ff814005e30 comp rx=0 tx=0).stop 2024-09-16T15:48:30.182 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:30.179+0000 7ff81fbfa640 1 -- 172.21.15.123:0/2749511439 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff81815e820 msgr2=0x7ff81814f0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:48:30.182 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:30.179+0000 7ff81fbfa640 1 --2- 172.21.15.123:0/2749511439 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff81815e820 0x7ff81814f0c0 secure :-1 s=READY pgs=1074 cs=0 l=1 rev1=1 crypto rx=0x7ff8100b0040 tx=0x7ff810005ae0 comp rx=0 tx=0).stop 2024-09-16T15:48:30.182 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:30.179+0000 7ff81fbfa640 1 -- 172.21.15.123:0/2749511439 shutdown_connections 2024-09-16T15:48:30.182 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:30.179+0000 7ff81fbfa640 1 --2- 172.21.15.123:0/2749511439 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7ff7fc072fe0 0x7ff7fc0754a0 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:30.182 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:30.179+0000 7ff81fbfa640 1 --2- 172.21.15.123:0/2749511439 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff81815e820 0x7ff81814f0c0 unknown :-1 s=CLOSED pgs=1074 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:30.182 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:30.179+0000 7ff81fbfa640 1 --2- 172.21.15.123:0/2749511439 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7ff81815e160 0x7ff8181fb9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:30.182 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:30.179+0000 7ff81fbfa640 1 --2- 172.21.15.123:0/2749511439 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff818074470 0x7ff81814d200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:30.182 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:30.179+0000 7ff81fbfa640 1 -- 172.21.15.123:0/2749511439 >> 172.21.15.123:0/2749511439 conn(0x7ff8180cfb10 msgr2=0x7ff818159840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:48:30.182 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:30.179+0000 7ff81fbfa640 1 -- 172.21.15.123:0/2749511439 shutdown_connections 2024-09-16T15:48:30.182 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:30.179+0000 7ff81fbfa640 1 -- 172.21.15.123:0/2749511439 wait complete. 2024-09-16T15:48:30.197 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-09-16T15:48:30.197 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T15:48:30.313 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:30.311+0000 7f2401926640 1 -- 172.21.15.123:0/2100586594 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f23fc15e820 msgr2=0x7f23fc15ecd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:48:30.314 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:30.311+0000 7f2401926640 1 --2- 172.21.15.123:0/2100586594 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f23fc15e820 0x7f23fc15ecd0 secure :-1 s=READY pgs=1075 cs=0 l=1 rev1=1 crypto rx=0x7f23f4009c80 tx=0x7f23f401cd10 comp rx=0 tx=0).stop 2024-09-16T15:48:30.314 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:30.311+0000 7f2401926640 1 -- 172.21.15.123:0/2100586594 shutdown_connections 2024-09-16T15:48:30.314 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:30.311+0000 7f2401926640 1 --2- 172.21.15.123:0/2100586594 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f23fc15e820 0x7f23fc15ecd0 unknown :-1 s=CLOSED pgs=1075 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:30.314 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:30.311+0000 7f2401926640 1 --2- 172.21.15.123:0/2100586594 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f23fc15e160 0x7f23fc073f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:30.314 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:30.311+0000 7f2401926640 1 -- 172.21.15.123:0/2100586594 >> 172.21.15.123:0/2100586594 conn(0x7f23fc0cfb10 msgr2=0x7f23fc0cff20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:48:30.314 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:30.315+0000 7f2401926640 1 -- 172.21.15.123:0/2100586594 shutdown_connections 2024-09-16T15:48:30.314 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:30.315+0000 7f2401926640 1 -- 172.21.15.123:0/2100586594 wait complete. 2024-09-16T15:48:30.315 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:30.315+0000 7f2401926640 1 Processor -- start 2024-09-16T15:48:30.315 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:30.315+0000 7f2401926640 1 -- start start 2024-09-16T15:48:30.317 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:30.315+0000 7f2401926640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f23fc074470 0x7f23fc1f8240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:48:30.317 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:30.315+0000 7f2401926640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f23fc15e160 0x7f23fc0b4ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:48:30.317 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:30.315+0000 7f2401926640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f23fc15e820 0x7f23fc0aff40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:48:30.317 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:30.315+0000 7f2401926640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f23fc0cf5b0 con 0x7f23fc074470 2024-09-16T15:48:30.317 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:30.315+0000 7f2401926640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f23fc0cf430 con 0x7f23fc15e820 2024-09-16T15:48:30.317 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:30.315+0000 7f2401926640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f23fc1683d0 con 0x7f23fc15e160 2024-09-16T15:48:30.317 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:30.315+0000 7f23faffd640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f23fc074470 0x7f23fc1f8240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:48:30.317 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:30.315+0000 7f23faffd640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f23fc074470 0x7f23fc1f8240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:47508/0 (socket says 172.21.15.123:47508) 2024-09-16T15:48:30.318 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:30.315+0000 7f23faffd640 1 -- 172.21.15.123:0/931609539 learned_addr learned my addr 172.21.15.123:0/931609539 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:48:30.318 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:30.315+0000 7f23fb7fe640 1 --2- 172.21.15.123:0/931609539 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f23fc15e820 0x7f23fc0aff40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:48:30.318 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:30.315+0000 7f23faffd640 1 -- 172.21.15.123:0/931609539 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f23fc15e160 msgr2=0x7f23fc0b4ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:48:30.318 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:30.315+0000 7f23faffd640 1 --2- 172.21.15.123:0/931609539 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f23fc15e160 0x7f23fc0b4ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:30.318 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:30.315+0000 7f23faffd640 1 -- 172.21.15.123:0/931609539 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f23fc15e820 msgr2=0x7f23fc0aff40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:48:30.318 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:30.315+0000 7f23faffd640 1 --2- 172.21.15.123:0/931609539 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f23fc15e820 0x7f23fc0aff40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:30.318 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:30.315+0000 7f23faffd640 1 -- 172.21.15.123:0/931609539 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f23fc0b5440 con 0x7f23fc074470 2024-09-16T15:48:30.318 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:30.315+0000 7f23faffd640 1 --2- 172.21.15.123:0/931609539 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f23fc074470 0x7f23fc1f8240 secure :-1 s=READY pgs=1076 cs=0 l=1 rev1=1 crypto rx=0x7f23ec00c9a0 tx=0x7f23ec00ce60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:48:30.318 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:30.315+0000 7f23e3fff640 1 -- 172.21.15.123:0/931609539 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f23ec007c10 con 0x7f23fc074470 2024-09-16T15:48:30.318 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:30.315+0000 7f23e3fff640 1 -- 172.21.15.123:0/931609539 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f23ec005020 con 0x7f23fc074470 2024-09-16T15:48:30.318 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:30.315+0000 7f23e3fff640 1 -- 172.21.15.123:0/931609539 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f23ec005450 con 0x7f23fc074470 2024-09-16T15:48:30.318 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:30.315+0000 7f2401926640 1 -- 172.21.15.123:0/931609539 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f23fc0b5720 con 0x7f23fc074470 2024-09-16T15:48:30.318 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:30.315+0000 7f2401926640 1 -- 172.21.15.123:0/931609539 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f23fc1f8780 con 0x7f23fc074470 2024-09-16T15:48:30.319 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:30.319+0000 7f2401926640 1 -- 172.21.15.123:0/931609539 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f23fc073f30 con 0x7f23fc074470 2024-09-16T15:48:30.319 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:30.319+0000 7f23e3fff640 1 -- 172.21.15.123:0/931609539 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 16) ==== 97182+0+0 (secure 0 0 0) 0x7f23ec0047b0 con 0x7f23fc074470 2024-09-16T15:48:30.319 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:30.319+0000 7f23e3fff640 1 --2- 172.21.15.123:0/931609539 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f23e8072fe0 0x7f23e80754a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:48:30.319 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:30.319+0000 7f23e3fff640 1 -- 172.21.15.123:0/931609539 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(103..103 src has 1..103) ==== 5709+0+0 (secure 0 0 0) 0x7f23ec0945d0 con 0x7f23fc074470 2024-09-16T15:48:30.319 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:30.319+0000 7f23fa7fc640 1 --2- 172.21.15.123:0/931609539 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f23e8072fe0 0x7f23e80754a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:48:30.319 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:30.319+0000 7f23fa7fc640 1 --2- 172.21.15.123:0/931609539 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f23e8072fe0 0x7f23e80754a0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f23fc0c5770 tx=0x7f23f00079c0 comp rx=0 tx=0).ready entity=mgr.4150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:48:30.323 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:30.323+0000 7f23e3fff640 1 -- 172.21.15.123:0/931609539 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f23ec014030 con 0x7f23fc074470 2024-09-16T15:48:30.484 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:30.483+0000 7f2401926640 1 -- 172.21.15.123:0/931609539 --> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f23fc0765d0 con 0x7f23e8072fe0 2024-09-16T15:48:30.491 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:30.491+0000 7f23e3fff640 1 -- 172.21.15.123:0/931609539 <== mgr.4150 v2:172.21.15.123:6836/2464759676 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+252481 (secure 0 0 0) 0x7f23fc0765d0 con 0x7f23e8072fe0 2024-09-16T15:48:30.494 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-09-16T15:48:30.495 INFO:teuthology.orchestra.run.smithi123.stderr:dumped all 2024-09-16T15:48:30.497 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:30.495+0000 7f2401926640 1 -- 172.21.15.123:0/931609539 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f23e8072fe0 msgr2=0x7f23e80754a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:48:30.497 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:30.495+0000 7f2401926640 1 --2- 172.21.15.123:0/931609539 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f23e8072fe0 0x7f23e80754a0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f23fc0c5770 tx=0x7f23f00079c0 comp rx=0 tx=0).stop 2024-09-16T15:48:30.497 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:30.495+0000 7f2401926640 1 -- 172.21.15.123:0/931609539 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f23fc074470 msgr2=0x7f23fc1f8240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:48:30.497 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:30.495+0000 7f2401926640 1 --2- 172.21.15.123:0/931609539 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f23fc074470 0x7f23fc1f8240 secure :-1 s=READY pgs=1076 cs=0 l=1 rev1=1 crypto rx=0x7f23ec00c9a0 tx=0x7f23ec00ce60 comp rx=0 tx=0).stop 2024-09-16T15:48:30.497 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:30.495+0000 7f2401926640 1 -- 172.21.15.123:0/931609539 shutdown_connections 2024-09-16T15:48:30.497 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:30.495+0000 7f2401926640 1 --2- 172.21.15.123:0/931609539 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f23e8072fe0 0x7f23e80754a0 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:30.497 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:30.495+0000 7f2401926640 1 --2- 172.21.15.123:0/931609539 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f23fc15e820 0x7f23fc0aff40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:30.497 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:30.495+0000 7f2401926640 1 --2- 172.21.15.123:0/931609539 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f23fc15e160 0x7f23fc0b4ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:30.497 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:30.495+0000 7f2401926640 1 --2- 172.21.15.123:0/931609539 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f23fc074470 0x7f23fc1f8240 unknown :-1 s=CLOSED pgs=1076 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:30.497 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:30.495+0000 7f2401926640 1 -- 172.21.15.123:0/931609539 >> 172.21.15.123:0/931609539 conn(0x7f23fc0cfb10 msgr2=0x7f23fc0cd6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:48:30.497 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:30.495+0000 7f2401926640 1 -- 172.21.15.123:0/931609539 shutdown_connections 2024-09-16T15:48:30.497 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:30.495+0000 7f2401926640 1 -- 172.21.15.123:0/931609539 wait complete. 2024-09-16T15:48:30.511 INFO:teuthology.orchestra.run.smithi123.stdout:{"pg_ready":true,"pg_map":{"version":62,"stamp":"2024-09-16T15:48:30.083454+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":158,"num_write_kb":1642,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":118,"ondisk_log_size":118,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":260732,"kb_used_data":11208,"kb_used_omap":12,"kb_used_meta":249331,"kb_avail":754713988,"statfs":{"total":773094113280,"available":772827123712,"internally_reserved":0,"allocated":11476992,"data_stored":5662836,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":255315505},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":44,"apply_latency_ms":44,"commit_latency_ns":44000000,"apply_latency_ns":44000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.660544"},"pg_stats":[{"pgid":"17.3f","version":"0'0","reported_seq":13,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-09-16T15:48:27.461697+0000","last_change":"2024-09-16T15:48:26.492027+0000","last_active":"2024-09-16T15:48:27.461697+0000","last_peered":"2024-09-16T15:48:27.461697+0000","last_clean":"2024-09-16T15:48:27.461697+0000","last_became_active":"2024-09-16T15:48:26.490634+0000","last_became_peered":"2024-09-16T15:48:26.490634+0000","last_unstale":"2024-09-16T15:48:27.461697+0000","last_undegraded":"2024-09-16T15:48:27.461697+0000","last_fullsized":"2024-09-16T15:48:27.461697+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:48:25.420844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:48:25.420844+0000","last_clean_scrub_stamp":"2024-09-16T15:48:25.420844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:26:46.532167+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.3e","version":"0'0","reported_seq":13,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-09-16T15:48:27.468269+0000","last_change":"2024-09-16T15:48:26.488569+0000","last_active":"2024-09-16T15:48:27.468269+0000","last_peered":"2024-09-16T15:48:27.468269+0000","last_clean":"2024-09-16T15:48:27.468269+0000","last_became_active":"2024-09-16T15:48:26.487794+0000","last_became_peered":"2024-09-16T15:48:26.487794+0000","last_unstale":"2024-09-16T15:48:27.468269+0000","last_undegraded":"2024-09-16T15:48:27.468269+0000","last_fullsized":"2024-09-16T15:48:27.468269+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:48:25.420844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:48:25.420844+0000","last_clean_scrub_stamp":"2024-09-16T15:48:25.420844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:22:45.568480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.3d","version":"0'0","reported_seq":13,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-09-16T15:48:27.887548+0000","last_change":"2024-09-16T15:48:26.510041+0000","last_active":"2024-09-16T15:48:27.887548+0000","last_peered":"2024-09-16T15:48:27.887548+0000","last_clean":"2024-09-16T15:48:27.887548+0000","last_became_active":"2024-09-16T15:48:26.509528+0000","last_became_peered":"2024-09-16T15:48:26.509528+0000","last_unstale":"2024-09-16T15:48:27.887548+0000","last_undegraded":"2024-09-16T15:48:27.887548+0000","last_fullsized":"2024-09-16T15:48:27.887548+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:48:25.420844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:48:25.420844+0000","last_clean_scrub_stamp":"2024-09-16T15:48:25.420844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T01:28:35.015978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.3c","version":"0'0","reported_seq":13,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-09-16T15:48:27.462135+0000","last_change":"2024-09-16T15:48:26.509093+0000","last_active":"2024-09-16T15:48:27.462135+0000","last_peered":"2024-09-16T15:48:27.462135+0000","last_clean":"2024-09-16T15:48:27.462135+0000","last_became_active":"2024-09-16T15:48:26.508843+0000","last_became_peered":"2024-09-16T15:48:26.508843+0000","last_unstale":"2024-09-16T15:48:27.462135+0000","last_undegraded":"2024-09-16T15:48:27.462135+0000","last_fullsized":"2024-09-16T15:48:27.462135+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:48:25.420844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:48:25.420844+0000","last_clean_scrub_stamp":"2024-09-16T15:48:25.420844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T01:40:10.109130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.3b","version":"0'0","reported_seq":13,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-09-16T15:48:27.496431+0000","last_change":"2024-09-16T15:48:26.481161+0000","last_active":"2024-09-16T15:48:27.496431+0000","last_peered":"2024-09-16T15:48:27.496431+0000","last_clean":"2024-09-16T15:48:27.496431+0000","last_became_active":"2024-09-16T15:48:26.480831+0000","last_became_peered":"2024-09-16T15:48:26.480831+0000","last_unstale":"2024-09-16T15:48:27.496431+0000","last_undegraded":"2024-09-16T15:48:27.496431+0000","last_fullsized":"2024-09-16T15:48:27.496431+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:48:25.420844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:48:25.420844+0000","last_clean_scrub_stamp":"2024-09-16T15:48:25.420844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T03:07:09.728518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.3a","version":"0'0","reported_seq":13,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-09-16T15:48:27.999012+0000","last_change":"2024-09-16T15:48:26.476018+0000","last_active":"2024-09-16T15:48:27.999012+0000","last_peered":"2024-09-16T15:48:27.999012+0000","last_clean":"2024-09-16T15:48:27.999012+0000","last_became_active":"2024-09-16T15:48:26.475640+0000","last_became_peered":"2024-09-16T15:48:26.475640+0000","last_unstale":"2024-09-16T15:48:27.999012+0000","last_undegraded":"2024-09-16T15:48:27.999012+0000","last_fullsized":"2024-09-16T15:48:27.999012+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:48:25.420844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:48:25.420844+0000","last_clean_scrub_stamp":"2024-09-16T15:48:25.420844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T23:24:05.461619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.39","version":"0'0","reported_seq":13,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-09-16T15:48:27.484014+0000","last_change":"2024-09-16T15:48:26.497742+0000","last_active":"2024-09-16T15:48:27.484014+0000","last_peered":"2024-09-16T15:48:27.484014+0000","last_clean":"2024-09-16T15:48:27.484014+0000","last_became_active":"2024-09-16T15:48:26.495135+0000","last_became_peered":"2024-09-16T15:48:26.495135+0000","last_unstale":"2024-09-16T15:48:27.484014+0000","last_undegraded":"2024-09-16T15:48:27.484014+0000","last_fullsized":"2024-09-16T15:48:27.484014+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:48:25.420844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:48:25.420844+0000","last_clean_scrub_stamp":"2024-09-16T15:48:25.420844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T18:56:16.511052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.38","version":"0'0","reported_seq":13,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-09-16T15:48:27.887213+0000","last_change":"2024-09-16T15:48:26.495526+0000","last_active":"2024-09-16T15:48:27.887213+0000","last_peered":"2024-09-16T15:48:27.887213+0000","last_clean":"2024-09-16T15:48:27.887213+0000","last_became_active":"2024-09-16T15:48:26.490886+0000","last_became_peered":"2024-09-16T15:48:26.490886+0000","last_unstale":"2024-09-16T15:48:27.887213+0000","last_undegraded":"2024-09-16T15:48:27.887213+0000","last_fullsized":"2024-09-16T15:48:27.887213+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:48:25.420844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:48:25.420844+0000","last_clean_scrub_stamp":"2024-09-16T15:48:25.420844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:40:39.688162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.37","version":"0'0","reported_seq":13,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-09-16T15:48:27.461918+0000","last_change":"2024-09-16T15:48:26.490579+0000","last_active":"2024-09-16T15:48:27.461918+0000","last_peered":"2024-09-16T15:48:27.461918+0000","last_clean":"2024-09-16T15:48:27.461918+0000","last_became_active":"2024-09-16T15:48:26.486770+0000","last_became_peered":"2024-09-16T15:48:26.486770+0000","last_unstale":"2024-09-16T15:48:27.461918+0000","last_undegraded":"2024-09-16T15:48:27.461918+0000","last_fullsized":"2024-09-16T15:48:27.461918+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:48:25.420844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:48:25.420844+0000","last_clean_scrub_stamp":"2024-09-16T15:48:25.420844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T23:29:27.871860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.36","version":"0'0","reported_seq":13,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-09-16T15:48:27.463194+0000","last_change":"2024-09-16T15:48:26.493063+0000","last_active":"2024-09-16T15:48:27.463194+0000","last_peered":"2024-09-16T15:48:27.463194+0000","last_clean":"2024-09-16T15:48:27.463194+0000","last_became_active":"2024-09-16T15:48:26.492024+0000","last_became_peered":"2024-09-16T15:48:26.492024+0000","last_unstale":"2024-09-16T15:48:27.463194+0000","last_undegraded":"2024-09-16T15:48:27.463194+0000","last_fullsized":"2024-09-16T15:48:27.463194+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:48:25.420844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:48:25.420844+0000","last_clean_scrub_stamp":"2024-09-16T15:48:25.420844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T18:40:16.437764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.35","version":"0'0","reported_seq":13,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-09-16T15:48:27.467862+0000","last_change":"2024-09-16T15:48:26.471375+0000","last_active":"2024-09-16T15:48:27.467862+0000","last_peered":"2024-09-16T15:48:27.467862+0000","last_clean":"2024-09-16T15:48:27.467862+0000","last_became_active":"2024-09-16T15:48:26.470597+0000","last_became_peered":"2024-09-16T15:48:26.470597+0000","last_unstale":"2024-09-16T15:48:27.467862+0000","last_undegraded":"2024-09-16T15:48:27.467862+0000","last_fullsized":"2024-09-16T15:48:27.467862+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:48:25.420844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:48:25.420844+0000","last_clean_scrub_stamp":"2024-09-16T15:48:25.420844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:00:26.169424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.34","version":"0'0","reported_seq":13,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-09-16T15:48:27.463848+0000","last_change":"2024-09-16T15:48:26.499834+0000","last_active":"2024-09-16T15:48:27.463848+0000","last_peered":"2024-09-16T15:48:27.463848+0000","last_clean":"2024-09-16T15:48:27.463848+0000","last_became_active":"2024-09-16T15:48:26.493251+0000","last_became_peered":"2024-09-16T15:48:26.493251+0000","last_unstale":"2024-09-16T15:48:27.463848+0000","last_undegraded":"2024-09-16T15:48:27.463848+0000","last_fullsized":"2024-09-16T15:48:27.463848+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:48:25.420844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:48:25.420844+0000","last_clean_scrub_stamp":"2024-09-16T15:48:25.420844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T18:48:08.106660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.33","version":"0'0","reported_seq":13,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-09-16T15:48:27.887776+0000","last_change":"2024-09-16T15:48:26.489082+0000","last_active":"2024-09-16T15:48:27.887776+0000","last_peered":"2024-09-16T15:48:27.887776+0000","last_clean":"2024-09-16T15:48:27.887776+0000","last_became_active":"2024-09-16T15:48:26.488251+0000","last_became_peered":"2024-09-16T15:48:26.488251+0000","last_unstale":"2024-09-16T15:48:27.887776+0000","last_undegraded":"2024-09-16T15:48:27.887776+0000","last_fullsized":"2024-09-16T15:48:27.887776+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:48:25.420844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:48:25.420844+0000","last_clean_scrub_stamp":"2024-09-16T15:48:25.420844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:07:44.883589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.32","version":"0'0","reported_seq":13,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-09-16T15:48:27.887539+0000","last_change":"2024-09-16T15:48:26.487177+0000","last_active":"2024-09-16T15:48:27.887539+0000","last_peered":"2024-09-16T15:48:27.887539+0000","last_clean":"2024-09-16T15:48:27.887539+0000","last_became_active":"2024-09-16T15:48:26.486331+0000","last_became_peered":"2024-09-16T15:48:26.486331+0000","last_unstale":"2024-09-16T15:48:27.887539+0000","last_undegraded":"2024-09-16T15:48:27.887539+0000","last_fullsized":"2024-09-16T15:48:27.887539+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:48:25.420844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:48:25.420844+0000","last_clean_scrub_stamp":"2024-09-16T15:48:25.420844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T22:10:31.295216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.31","version":"0'0","reported_seq":13,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-09-16T15:48:27.998911+0000","last_change":"2024-09-16T15:48:26.478731+0000","last_active":"2024-09-16T15:48:27.998911+0000","last_peered":"2024-09-16T15:48:27.998911+0000","last_clean":"2024-09-16T15:48:27.998911+0000","last_became_active":"2024-09-16T15:48:26.478366+0000","last_became_peered":"2024-09-16T15:48:26.478366+0000","last_unstale":"2024-09-16T15:48:27.998911+0000","last_undegraded":"2024-09-16T15:48:27.998911+0000","last_fullsized":"2024-09-16T15:48:27.998911+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:48:25.420844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:48:25.420844+0000","last_clean_scrub_stamp":"2024-09-16T15:48:25.420844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:21:49.177349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.30","version":"0'0","reported_seq":13,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-09-16T15:48:27.461362+0000","last_change":"2024-09-16T15:48:26.502708+0000","last_active":"2024-09-16T15:48:27.461362+0000","last_peered":"2024-09-16T15:48:27.461362+0000","last_clean":"2024-09-16T15:48:27.461362+0000","last_became_active":"2024-09-16T15:48:26.502397+0000","last_became_peered":"2024-09-16T15:48:26.502397+0000","last_unstale":"2024-09-16T15:48:27.461362+0000","last_undegraded":"2024-09-16T15:48:27.461362+0000","last_fullsized":"2024-09-16T15:48:27.461362+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:48:25.420844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:48:25.420844+0000","last_clean_scrub_stamp":"2024-09-16T15:48:25.420844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:40:23.507844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.2f","version":"0'0","reported_seq":13,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-09-16T15:48:27.468222+0000","last_change":"2024-09-16T15:48:26.474831+0000","last_active":"2024-09-16T15:48:27.468222+0000","last_peered":"2024-09-16T15:48:27.468222+0000","last_clean":"2024-09-16T15:48:27.468222+0000","last_became_active":"2024-09-16T15:48:26.473342+0000","last_became_peered":"2024-09-16T15:48:26.473342+0000","last_unstale":"2024-09-16T15:48:27.468222+0000","last_undegraded":"2024-09-16T15:48:27.468222+0000","last_fullsized":"2024-09-16T15:48:27.468222+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:48:25.420844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:48:25.420844+0000","last_clean_scrub_stamp":"2024-09-16T15:48:25.420844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T02:55:20.157975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.2e","version":"0'0","reported_seq":13,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-09-16T15:48:27.466862+0000","last_change":"2024-09-16T15:48:26.468006+0000","last_active":"2024-09-16T15:48:27.466862+0000","last_peered":"2024-09-16T15:48:27.466862+0000","last_clean":"2024-09-16T15:48:27.466862+0000","last_became_active":"2024-09-16T15:48:26.467250+0000","last_became_peered":"2024-09-16T15:48:26.467250+0000","last_unstale":"2024-09-16T15:48:27.466862+0000","last_undegraded":"2024-09-16T15:48:27.466862+0000","last_fullsized":"2024-09-16T15:48:27.466862+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:48:25.420844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:48:25.420844+0000","last_clean_scrub_stamp":"2024-09-16T15:48:25.420844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:29:34.405446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.2d","version":"0'0","reported_seq":13,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-09-16T15:48:27.887849+0000","last_change":"2024-09-16T15:48:26.489641+0000","last_active":"2024-09-16T15:48:27.887849+0000","last_peered":"2024-09-16T15:48:27.887849+0000","last_clean":"2024-09-16T15:48:27.887849+0000","last_became_active":"2024-09-16T15:48:26.488057+0000","last_became_peered":"2024-09-16T15:48:26.488057+0000","last_unstale":"2024-09-16T15:48:27.887849+0000","last_undegraded":"2024-09-16T15:48:27.887849+0000","last_fullsized":"2024-09-16T15:48:27.887849+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:48:25.420844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:48:25.420844+0000","last_clean_scrub_stamp":"2024-09-16T15:48:25.420844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:19:58.858823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.2c","version":"0'0","reported_seq":13,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-09-16T15:48:27.467701+0000","last_change":"2024-09-16T15:48:26.465853+0000","last_active":"2024-09-16T15:48:27.467701+0000","last_peered":"2024-09-16T15:48:27.467701+0000","last_clean":"2024-09-16T15:48:27.467701+0000","last_became_active":"2024-09-16T15:48:26.465105+0000","last_became_peered":"2024-09-16T15:48:26.465105+0000","last_unstale":"2024-09-16T15:48:27.467701+0000","last_undegraded":"2024-09-16T15:48:27.467701+0000","last_fullsized":"2024-09-16T15:48:27.467701+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:48:25.420844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:48:25.420844+0000","last_clean_scrub_stamp":"2024-09-16T15:48:25.420844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T23:20:00.312557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.2b","version":"0'0","reported_seq":13,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-09-16T15:48:27.466957+0000","last_change":"2024-09-16T15:48:26.486192+0000","last_active":"2024-09-16T15:48:27.466957+0000","last_peered":"2024-09-16T15:48:27.466957+0000","last_clean":"2024-09-16T15:48:27.466957+0000","last_became_active":"2024-09-16T15:48:26.485417+0000","last_became_peered":"2024-09-16T15:48:26.485417+0000","last_unstale":"2024-09-16T15:48:27.466957+0000","last_undegraded":"2024-09-16T15:48:27.466957+0000","last_fullsized":"2024-09-16T15:48:27.466957+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:48:25.420844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:48:25.420844+0000","last_clean_scrub_stamp":"2024-09-16T15:48:25.420844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T02:31:29.163804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.2a","version":"0'0","reported_seq":13,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-09-16T15:48:27.997142+0000","last_change":"2024-09-16T15:48:26.459479+0000","last_active":"2024-09-16T15:48:27.997142+0000","last_peered":"2024-09-16T15:48:27.997142+0000","last_clean":"2024-09-16T15:48:27.997142+0000","last_became_active":"2024-09-16T15:48:26.458168+0000","last_became_peered":"2024-09-16T15:48:26.458168+0000","last_unstale":"2024-09-16T15:48:27.997142+0000","last_undegraded":"2024-09-16T15:48:27.997142+0000","last_fullsized":"2024-09-16T15:48:27.997142+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:48:25.420844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:48:25.420844+0000","last_clean_scrub_stamp":"2024-09-16T15:48:25.420844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:44:15.690048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.29","version":"0'0","reported_seq":13,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-09-16T15:48:27.465052+0000","last_change":"2024-09-16T15:48:26.467978+0000","last_active":"2024-09-16T15:48:27.465052+0000","last_peered":"2024-09-16T15:48:27.465052+0000","last_clean":"2024-09-16T15:48:27.465052+0000","last_became_active":"2024-09-16T15:48:26.467166+0000","last_became_peered":"2024-09-16T15:48:26.467166+0000","last_unstale":"2024-09-16T15:48:27.465052+0000","last_undegraded":"2024-09-16T15:48:27.465052+0000","last_fullsized":"2024-09-16T15:48:27.465052+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:48:25.420844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:48:25.420844+0000","last_clean_scrub_stamp":"2024-09-16T15:48:25.420844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:18:22.614416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.28","version":"0'0","reported_seq":13,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-09-16T15:48:27.997749+0000","last_change":"2024-09-16T15:48:26.474394+0000","last_active":"2024-09-16T15:48:27.997749+0000","last_peered":"2024-09-16T15:48:27.997749+0000","last_clean":"2024-09-16T15:48:27.997749+0000","last_became_active":"2024-09-16T15:48:26.474072+0000","last_became_peered":"2024-09-16T15:48:26.474072+0000","last_unstale":"2024-09-16T15:48:27.997749+0000","last_undegraded":"2024-09-16T15:48:27.997749+0000","last_fullsized":"2024-09-16T15:48:27.997749+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:48:25.420844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:48:25.420844+0000","last_clean_scrub_stamp":"2024-09-16T15:48:25.420844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:02:13.807139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.27","version":"0'0","reported_seq":13,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-09-16T15:48:27.468519+0000","last_change":"2024-09-16T15:48:26.474043+0000","last_active":"2024-09-16T15:48:27.468519+0000","last_peered":"2024-09-16T15:48:27.468519+0000","last_clean":"2024-09-16T15:48:27.468519+0000","last_became_active":"2024-09-16T15:48:26.472415+0000","last_became_peered":"2024-09-16T15:48:26.472415+0000","last_unstale":"2024-09-16T15:48:27.468519+0000","last_undegraded":"2024-09-16T15:48:27.468519+0000","last_fullsized":"2024-09-16T15:48:27.468519+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:48:25.420844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:48:25.420844+0000","last_clean_scrub_stamp":"2024-09-16T15:48:25.420844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T18:03:51.621047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.26","version":"0'0","reported_seq":13,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-09-16T15:48:27.463260+0000","last_change":"2024-09-16T15:48:26.492754+0000","last_active":"2024-09-16T15:48:27.463260+0000","last_peered":"2024-09-16T15:48:27.463260+0000","last_clean":"2024-09-16T15:48:27.463260+0000","last_became_active":"2024-09-16T15:48:26.487615+0000","last_became_peered":"2024-09-16T15:48:26.487615+0000","last_unstale":"2024-09-16T15:48:27.463260+0000","last_undegraded":"2024-09-16T15:48:27.463260+0000","last_fullsized":"2024-09-16T15:48:27.463260+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:48:25.420844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:48:25.420844+0000","last_clean_scrub_stamp":"2024-09-16T15:48:25.420844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:00:04.834017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.25","version":"0'0","reported_seq":13,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-09-16T15:48:27.999811+0000","last_change":"2024-09-16T15:48:26.490676+0000","last_active":"2024-09-16T15:48:27.999811+0000","last_peered":"2024-09-16T15:48:27.999811+0000","last_clean":"2024-09-16T15:48:27.999811+0000","last_became_active":"2024-09-16T15:48:26.490387+0000","last_became_peered":"2024-09-16T15:48:26.490387+0000","last_unstale":"2024-09-16T15:48:27.999811+0000","last_undegraded":"2024-09-16T15:48:27.999811+0000","last_fullsized":"2024-09-16T15:48:27.999811+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:48:25.420844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:48:25.420844+0000","last_clean_scrub_stamp":"2024-09-16T15:48:25.420844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:14:14.378687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.24","version":"0'0","reported_seq":13,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-09-16T15:48:27.997318+0000","last_change":"2024-09-16T15:48:26.496746+0000","last_active":"2024-09-16T15:48:27.997318+0000","last_peered":"2024-09-16T15:48:27.997318+0000","last_clean":"2024-09-16T15:48:27.997318+0000","last_became_active":"2024-09-16T15:48:26.496472+0000","last_became_peered":"2024-09-16T15:48:26.496472+0000","last_unstale":"2024-09-16T15:48:27.997318+0000","last_undegraded":"2024-09-16T15:48:27.997318+0000","last_fullsized":"2024-09-16T15:48:27.997318+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:48:25.420844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:48:25.420844+0000","last_clean_scrub_stamp":"2024-09-16T15:48:25.420844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:58:33.486166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.23","version":"0'0","reported_seq":13,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-09-16T15:48:27.465956+0000","last_change":"2024-09-16T15:48:26.488919+0000","last_active":"2024-09-16T15:48:27.465956+0000","last_peered":"2024-09-16T15:48:27.465956+0000","last_clean":"2024-09-16T15:48:27.465956+0000","last_became_active":"2024-09-16T15:48:26.487265+0000","last_became_peered":"2024-09-16T15:48:26.487265+0000","last_unstale":"2024-09-16T15:48:27.465956+0000","last_undegraded":"2024-09-16T15:48:27.465956+0000","last_fullsized":"2024-09-16T15:48:27.465956+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:48:25.420844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:48:25.420844+0000","last_clean_scrub_stamp":"2024-09-16T15:48:25.420844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:17:26.625596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.22","version":"0'0","reported_seq":13,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-09-16T15:48:27.483405+0000","last_change":"2024-09-16T15:48:26.495110+0000","last_active":"2024-09-16T15:48:27.483405+0000","last_peered":"2024-09-16T15:48:27.483405+0000","last_clean":"2024-09-16T15:48:27.483405+0000","last_became_active":"2024-09-16T15:48:26.494036+0000","last_became_peered":"2024-09-16T15:48:26.494036+0000","last_unstale":"2024-09-16T15:48:27.483405+0000","last_undegraded":"2024-09-16T15:48:27.483405+0000","last_fullsized":"2024-09-16T15:48:27.483405+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:48:25.420844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:48:25.420844+0000","last_clean_scrub_stamp":"2024-09-16T15:48:25.420844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:14:21.492784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.21","version":"0'0","reported_seq":13,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-09-16T15:48:27.461691+0000","last_change":"2024-09-16T15:48:26.500768+0000","last_active":"2024-09-16T15:48:27.461691+0000","last_peered":"2024-09-16T15:48:27.461691+0000","last_clean":"2024-09-16T15:48:27.461691+0000","last_became_active":"2024-09-16T15:48:26.500434+0000","last_became_peered":"2024-09-16T15:48:26.500434+0000","last_unstale":"2024-09-16T15:48:27.461691+0000","last_undegraded":"2024-09-16T15:48:27.461691+0000","last_fullsized":"2024-09-16T15:48:27.461691+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:48:25.420844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:48:25.420844+0000","last_clean_scrub_stamp":"2024-09-16T15:48:25.420844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:14.364288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.20","version":"0'0","reported_seq":13,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-09-16T15:48:27.998891+0000","last_change":"2024-09-16T15:48:26.474517+0000","last_active":"2024-09-16T15:48:27.998891+0000","last_peered":"2024-09-16T15:48:27.998891+0000","last_clean":"2024-09-16T15:48:27.998891+0000","last_became_active":"2024-09-16T15:48:26.473941+0000","last_became_peered":"2024-09-16T15:48:26.473941+0000","last_unstale":"2024-09-16T15:48:27.998891+0000","last_undegraded":"2024-09-16T15:48:27.998891+0000","last_fullsized":"2024-09-16T15:48:27.998891+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:48:25.420844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:48:25.420844+0000","last_clean_scrub_stamp":"2024-09-16T15:48:25.420844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:52:25.564241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.1e","version":"0'0","reported_seq":13,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-09-16T15:48:27.999924+0000","last_change":"2024-09-16T15:48:26.475804+0000","last_active":"2024-09-16T15:48:27.999924+0000","last_peered":"2024-09-16T15:48:27.999924+0000","last_clean":"2024-09-16T15:48:27.999924+0000","last_became_active":"2024-09-16T15:48:26.475322+0000","last_became_peered":"2024-09-16T15:48:26.475322+0000","last_unstale":"2024-09-16T15:48:27.999924+0000","last_undegraded":"2024-09-16T15:48:27.999924+0000","last_fullsized":"2024-09-16T15:48:27.999924+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:48:25.420844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:48:25.420844+0000","last_clean_scrub_stamp":"2024-09-16T15:48:25.420844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T23:21:43.437807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.1f","version":"102'4","reported_seq":19,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-09-16T15:48:27.483202+0000","last_change":"2024-09-16T15:48:25.465785+0000","last_active":"2024-09-16T15:48:27.483202+0000","last_peered":"2024-09-16T15:48:27.483202+0000","last_clean":"2024-09-16T15:48:27.483202+0000","last_became_active":"2024-09-16T15:48:25.464855+0000","last_became_peered":"2024-09-16T15:48:25.464855+0000","last_unstale":"2024-09-16T15:48:27.483202+0000","last_undegraded":"2024-09-16T15:48:27.483202+0000","last_fullsized":"2024-09-16T15:48:27.483202+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:48:24.407732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:48:24.407732+0000","last_clean_scrub_stamp":"2024-09-16T15:48:24.407732+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T23:21:42.424695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.1f","version":"0'0","reported_seq":13,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-09-16T15:48:27.467691+0000","last_change":"2024-09-16T15:48:26.470107+0000","last_active":"2024-09-16T15:48:27.467691+0000","last_peered":"2024-09-16T15:48:27.467691+0000","last_clean":"2024-09-16T15:48:27.467691+0000","last_became_active":"2024-09-16T15:48:26.469290+0000","last_became_peered":"2024-09-16T15:48:26.469290+0000","last_unstale":"2024-09-16T15:48:27.467691+0000","last_undegraded":"2024-09-16T15:48:27.467691+0000","last_fullsized":"2024-09-16T15:48:27.467691+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:48:25.420844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:48:25.420844+0000","last_clean_scrub_stamp":"2024-09-16T15:48:25.420844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:29:35.762056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.1e","version":"102'1","reported_seq":16,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-09-16T15:48:27.887676+0000","last_change":"2024-09-16T15:48:25.469533+0000","last_active":"2024-09-16T15:48:27.887676+0000","last_peered":"2024-09-16T15:48:27.887676+0000","last_clean":"2024-09-16T15:48:27.887676+0000","last_became_active":"2024-09-16T15:48:25.468882+0000","last_became_peered":"2024-09-16T15:48:25.468882+0000","last_unstale":"2024-09-16T15:48:27.887676+0000","last_undegraded":"2024-09-16T15:48:27.887676+0000","last_fullsized":"2024-09-16T15:48:27.887676+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:48:24.407732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:48:24.407732+0000","last_clean_scrub_stamp":"2024-09-16T15:48:24.407732+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T03:32:30.135599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.1c","version":"0'0","reported_seq":13,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-09-16T15:48:27.999784+0000","last_change":"2024-09-16T15:48:26.474260+0000","last_active":"2024-09-16T15:48:27.999784+0000","last_peered":"2024-09-16T15:48:27.999784+0000","last_clean":"2024-09-16T15:48:27.999784+0000","last_became_active":"2024-09-16T15:48:26.473902+0000","last_became_peered":"2024-09-16T15:48:26.473902+0000","last_unstale":"2024-09-16T15:48:27.999784+0000","last_undegraded":"2024-09-16T15:48:27.999784+0000","last_fullsized":"2024-09-16T15:48:27.999784+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:48:25.420844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:48:25.420844+0000","last_clean_scrub_stamp":"2024-09-16T15:48:25.420844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T03:15:19.215261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.1d","version":"102'2","reported_seq":17,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-09-16T15:48:27.495904+0000","last_change":"2024-09-16T15:48:25.449694+0000","last_active":"2024-09-16T15:48:27.495904+0000","last_peered":"2024-09-16T15:48:27.495904+0000","last_clean":"2024-09-16T15:48:27.495904+0000","last_became_active":"2024-09-16T15:48:25.449121+0000","last_became_peered":"2024-09-16T15:48:25.449121+0000","last_unstale":"2024-09-16T15:48:27.495904+0000","last_undegraded":"2024-09-16T15:48:27.495904+0000","last_fullsized":"2024-09-16T15:48:27.495904+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:48:24.407732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:48:24.407732+0000","last_clean_scrub_stamp":"2024-09-16T15:48:24.407732+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:19:14.112868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1d","version":"0'0","reported_seq":13,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-09-16T15:48:27.888159+0000","last_change":"2024-09-16T15:48:26.490466+0000","last_active":"2024-09-16T15:48:27.888159+0000","last_peered":"2024-09-16T15:48:27.888159+0000","last_clean":"2024-09-16T15:48:27.888159+0000","last_became_active":"2024-09-16T15:48:26.488882+0000","last_became_peered":"2024-09-16T15:48:26.488882+0000","last_unstale":"2024-09-16T15:48:27.888159+0000","last_undegraded":"2024-09-16T15:48:27.888159+0000","last_fullsized":"2024-09-16T15:48:27.888159+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:48:25.420844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:48:25.420844+0000","last_clean_scrub_stamp":"2024-09-16T15:48:25.420844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T23:18:21.147916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.1c","version":"102'2","reported_seq":17,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-09-16T15:48:27.495679+0000","last_change":"2024-09-16T15:48:25.446615+0000","last_active":"2024-09-16T15:48:27.495679+0000","last_peered":"2024-09-16T15:48:27.495679+0000","last_clean":"2024-09-16T15:48:27.495679+0000","last_became_active":"2024-09-16T15:48:25.446259+0000","last_became_peered":"2024-09-16T15:48:25.446259+0000","last_unstale":"2024-09-16T15:48:27.495679+0000","last_undegraded":"2024-09-16T15:48:27.495679+0000","last_fullsized":"2024-09-16T15:48:27.495679+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:48:24.407732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:48:24.407732+0000","last_clean_scrub_stamp":"2024-09-16T15:48:24.407732+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:35:07.163165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1a","version":"0'0","reported_seq":13,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-09-16T15:48:27.461696+0000","last_change":"2024-09-16T15:48:26.487828+0000","last_active":"2024-09-16T15:48:27.461696+0000","last_peered":"2024-09-16T15:48:27.461696+0000","last_clean":"2024-09-16T15:48:27.461696+0000","last_became_active":"2024-09-16T15:48:26.486834+0000","last_became_peered":"2024-09-16T15:48:26.486834+0000","last_unstale":"2024-09-16T15:48:27.461696+0000","last_undegraded":"2024-09-16T15:48:27.461696+0000","last_fullsized":"2024-09-16T15:48:27.461696+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:48:25.420844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:48:25.420844+0000","last_clean_scrub_stamp":"2024-09-16T15:48:25.420844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T01:10:19.234916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.1b","version":"102'1","reported_seq":16,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-09-16T15:48:27.482907+0000","last_change":"2024-09-16T15:48:25.458440+0000","last_active":"2024-09-16T15:48:27.482907+0000","last_peered":"2024-09-16T15:48:27.482907+0000","last_clean":"2024-09-16T15:48:27.482907+0000","last_became_active":"2024-09-16T15:48:25.457689+0000","last_became_peered":"2024-09-16T15:48:25.457689+0000","last_unstale":"2024-09-16T15:48:27.482907+0000","last_undegraded":"2024-09-16T15:48:27.482907+0000","last_fullsized":"2024-09-16T15:48:27.482907+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:48:24.407732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:48:24.407732+0000","last_clean_scrub_stamp":"2024-09-16T15:48:24.407732+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T03:15:18.202149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.1b","version":"0'0","reported_seq":13,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-09-16T15:48:27.465971+0000","last_change":"2024-09-16T15:48:26.489699+0000","last_active":"2024-09-16T15:48:27.465971+0000","last_peered":"2024-09-16T15:48:27.465971+0000","last_clean":"2024-09-16T15:48:27.465971+0000","last_became_active":"2024-09-16T15:48:26.488220+0000","last_became_peered":"2024-09-16T15:48:26.488220+0000","last_unstale":"2024-09-16T15:48:27.465971+0000","last_undegraded":"2024-09-16T15:48:27.465971+0000","last_fullsized":"2024-09-16T15:48:27.465971+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:48:25.420844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:48:25.420844+0000","last_clean_scrub_stamp":"2024-09-16T15:48:25.420844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:47:19.716595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.1a","version":"102'1","reported_seq":16,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-09-16T15:48:27.997380+0000","last_change":"2024-09-16T15:48:25.445022+0000","last_active":"2024-09-16T15:48:27.997380+0000","last_peered":"2024-09-16T15:48:27.997380+0000","last_clean":"2024-09-16T15:48:27.997380+0000","last_became_active":"2024-09-16T15:48:25.444656+0000","last_became_peered":"2024-09-16T15:48:25.444656+0000","last_unstale":"2024-09-16T15:48:27.997380+0000","last_undegraded":"2024-09-16T15:48:27.997380+0000","last_fullsized":"2024-09-16T15:48:27.997380+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:48:24.407732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:48:24.407732+0000","last_clean_scrub_stamp":"2024-09-16T15:48:24.407732+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T22:19:51.476505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.18","version":"0'0","reported_seq":13,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-09-16T15:48:27.463522+0000","last_change":"2024-09-16T15:48:26.488718+0000","last_active":"2024-09-16T15:48:27.463522+0000","last_peered":"2024-09-16T15:48:27.463522+0000","last_clean":"2024-09-16T15:48:27.463522+0000","last_became_active":"2024-09-16T15:48:26.487539+0000","last_became_peered":"2024-09-16T15:48:26.487539+0000","last_unstale":"2024-09-16T15:48:27.463522+0000","last_undegraded":"2024-09-16T15:48:27.463522+0000","last_fullsized":"2024-09-16T15:48:27.463522+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:48:25.420844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:48:25.420844+0000","last_clean_scrub_stamp":"2024-09-16T15:48:25.420844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T22:48:59.876278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.19","version":"0'0","reported_seq":15,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-09-16T15:48:27.464844+0000","last_change":"2024-09-16T15:48:25.446394+0000","last_active":"2024-09-16T15:48:27.464844+0000","last_peered":"2024-09-16T15:48:27.464844+0000","last_clean":"2024-09-16T15:48:27.464844+0000","last_became_active":"2024-09-16T15:48:25.445446+0000","last_became_peered":"2024-09-16T15:48:25.445446+0000","last_unstale":"2024-09-16T15:48:27.464844+0000","last_undegraded":"2024-09-16T15:48:27.464844+0000","last_fullsized":"2024-09-16T15:48:27.464844+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:48:24.407732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:48:24.407732+0000","last_clean_scrub_stamp":"2024-09-16T15:48:24.407732+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T22:19:51.476505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.19","version":"0'0","reported_seq":13,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-09-16T15:48:27.483026+0000","last_change":"2024-09-16T15:48:26.498541+0000","last_active":"2024-09-16T15:48:27.483026+0000","last_peered":"2024-09-16T15:48:27.483026+0000","last_clean":"2024-09-16T15:48:27.483026+0000","last_became_active":"2024-09-16T15:48:26.495967+0000","last_became_peered":"2024-09-16T15:48:26.495967+0000","last_unstale":"2024-09-16T15:48:27.483026+0000","last_undegraded":"2024-09-16T15:48:27.483026+0000","last_fullsized":"2024-09-16T15:48:27.483026+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:48:25.420844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:48:25.420844+0000","last_clean_scrub_stamp":"2024-09-16T15:48:25.420844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:19:49.639111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.18","version":"0'0","reported_seq":15,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-09-16T15:48:27.461686+0000","last_change":"2024-09-16T15:48:25.461569+0000","last_active":"2024-09-16T15:48:27.461686+0000","last_peered":"2024-09-16T15:48:27.461686+0000","last_clean":"2024-09-16T15:48:27.461686+0000","last_became_active":"2024-09-16T15:48:25.461207+0000","last_became_peered":"2024-09-16T15:48:25.461207+0000","last_unstale":"2024-09-16T15:48:27.461686+0000","last_undegraded":"2024-09-16T15:48:27.461686+0000","last_fullsized":"2024-09-16T15:48:27.461686+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:48:24.407732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:48:24.407732+0000","last_clean_scrub_stamp":"2024-09-16T15:48:24.407732+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:27:25.935391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.16","version":"0'0","reported_seq":13,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-09-16T15:48:27.484145+0000","last_change":"2024-09-16T15:48:26.482968+0000","last_active":"2024-09-16T15:48:27.484145+0000","last_peered":"2024-09-16T15:48:27.484145+0000","last_clean":"2024-09-16T15:48:27.484145+0000","last_became_active":"2024-09-16T15:48:26.482005+0000","last_became_peered":"2024-09-16T15:48:26.482005+0000","last_unstale":"2024-09-16T15:48:27.484145+0000","last_undegraded":"2024-09-16T15:48:27.484145+0000","last_fullsized":"2024-09-16T15:48:27.484145+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:48:25.420844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:48:25.420844+0000","last_clean_scrub_stamp":"2024-09-16T15:48:25.420844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:34:11.300050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.17","version":"0'0","reported_seq":15,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-09-16T15:48:27.495999+0000","last_change":"2024-09-16T15:48:25.452524+0000","last_active":"2024-09-16T15:48:27.495999+0000","last_peered":"2024-09-16T15:48:27.495999+0000","last_clean":"2024-09-16T15:48:27.495999+0000","last_became_active":"2024-09-16T15:48:25.452224+0000","last_became_peered":"2024-09-16T15:48:25.452224+0000","last_unstale":"2024-09-16T15:48:27.495999+0000","last_undegraded":"2024-09-16T15:48:27.495999+0000","last_fullsized":"2024-09-16T15:48:27.495999+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:48:24.407732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:48:24.407732+0000","last_clean_scrub_stamp":"2024-09-16T15:48:24.407732+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:09:30.240163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.17","version":"0'0","reported_seq":13,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-09-16T15:48:27.467970+0000","last_change":"2024-09-16T15:48:26.498755+0000","last_active":"2024-09-16T15:48:27.467970+0000","last_peered":"2024-09-16T15:48:27.467970+0000","last_clean":"2024-09-16T15:48:27.467970+0000","last_became_active":"2024-09-16T15:48:26.497874+0000","last_became_peered":"2024-09-16T15:48:26.497874+0000","last_unstale":"2024-09-16T15:48:27.467970+0000","last_undegraded":"2024-09-16T15:48:27.467970+0000","last_fullsized":"2024-09-16T15:48:27.467970+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:48:25.420844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:48:25.420844+0000","last_clean_scrub_stamp":"2024-09-16T15:48:25.420844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T18:36:32.286233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.16","version":"0'0","reported_seq":15,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-09-16T15:48:27.996678+0000","last_change":"2024-09-16T15:48:25.444804+0000","last_active":"2024-09-16T15:48:27.996678+0000","last_peered":"2024-09-16T15:48:27.996678+0000","last_clean":"2024-09-16T15:48:27.996678+0000","last_became_active":"2024-09-16T15:48:25.444511+0000","last_became_peered":"2024-09-16T15:48:25.444511+0000","last_unstale":"2024-09-16T15:48:27.996678+0000","last_undegraded":"2024-09-16T15:48:27.996678+0000","last_fullsized":"2024-09-16T15:48:27.996678+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:48:24.407732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:48:24.407732+0000","last_clean_scrub_stamp":"2024-09-16T15:48:24.407732+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T22:15:28.730113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.14","version":"0'0","reported_seq":13,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-09-16T15:48:27.997616+0000","last_change":"2024-09-16T15:48:26.469903+0000","last_active":"2024-09-16T15:48:27.997616+0000","last_peered":"2024-09-16T15:48:27.997616+0000","last_clean":"2024-09-16T15:48:27.997616+0000","last_became_active":"2024-09-16T15:48:26.469227+0000","last_became_peered":"2024-09-16T15:48:26.469227+0000","last_unstale":"2024-09-16T15:48:27.997616+0000","last_undegraded":"2024-09-16T15:48:27.997616+0000","last_fullsized":"2024-09-16T15:48:27.997616+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:48:25.420844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:48:25.420844+0000","last_clean_scrub_stamp":"2024-09-16T15:48:25.420844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T23:19:49.157256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.15","version":"102'3","reported_seq":18,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-09-16T15:48:27.462567+0000","last_change":"2024-09-16T15:48:25.457438+0000","last_active":"2024-09-16T15:48:27.462567+0000","last_peered":"2024-09-16T15:48:27.462567+0000","last_clean":"2024-09-16T15:48:27.462567+0000","last_became_active":"2024-09-16T15:48:25.456752+0000","last_became_peered":"2024-09-16T15:48:25.456752+0000","last_unstale":"2024-09-16T15:48:27.462567+0000","last_undegraded":"2024-09-16T15:48:27.462567+0000","last_fullsized":"2024-09-16T15:48:27.462567+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:48:24.407732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:48:24.407732+0000","last_clean_scrub_stamp":"2024-09-16T15:48:24.407732+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:23:52.067881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.15","version":"0'0","reported_seq":13,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-09-16T15:48:27.483758+0000","last_change":"2024-09-16T15:48:26.506294+0000","last_active":"2024-09-16T15:48:27.483758+0000","last_peered":"2024-09-16T15:48:27.483758+0000","last_clean":"2024-09-16T15:48:27.483758+0000","last_became_active":"2024-09-16T15:48:26.505894+0000","last_became_peered":"2024-09-16T15:48:26.505894+0000","last_unstale":"2024-09-16T15:48:27.483758+0000","last_undegraded":"2024-09-16T15:48:27.483758+0000","last_fullsized":"2024-09-16T15:48:27.483758+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:48:25.420844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:48:25.420844+0000","last_clean_scrub_stamp":"2024-09-16T15:48:25.420844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:12:21.419337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.14","version":"102'6","reported_seq":21,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-09-16T15:48:27.462843+0000","last_change":"2024-09-16T15:48:25.461989+0000","last_active":"2024-09-16T15:48:27.462843+0000","last_peered":"2024-09-16T15:48:27.462843+0000","last_clean":"2024-09-16T15:48:27.462843+0000","last_became_active":"2024-09-16T15:48:25.461384+0000","last_became_peered":"2024-09-16T15:48:25.461384+0000","last_unstale":"2024-09-16T15:48:27.462843+0000","last_undegraded":"2024-09-16T15:48:27.462843+0000","last_fullsized":"2024-09-16T15:48:27.462843+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:48:24.407732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:48:24.407732+0000","last_clean_scrub_stamp":"2024-09-16T15:48:24.407732+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:19:57.845711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.12","version":"0'0","reported_seq":13,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-09-16T15:48:27.997394+0000","last_change":"2024-09-16T15:48:26.508270+0000","last_active":"2024-09-16T15:48:27.997394+0000","last_peered":"2024-09-16T15:48:27.997394+0000","last_clean":"2024-09-16T15:48:27.997394+0000","last_became_active":"2024-09-16T15:48:26.507897+0000","last_became_peered":"2024-09-16T15:48:26.507897+0000","last_unstale":"2024-09-16T15:48:27.997394+0000","last_undegraded":"2024-09-16T15:48:27.997394+0000","last_fullsized":"2024-09-16T15:48:27.997394+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:48:25.420844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:48:25.420844+0000","last_clean_scrub_stamp":"2024-09-16T15:48:25.420844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T23:32:17.736399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.13","version":"102'2","reported_seq":17,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-09-16T15:48:27.999066+0000","last_change":"2024-09-16T15:48:25.450490+0000","last_active":"2024-09-16T15:48:27.999066+0000","last_peered":"2024-09-16T15:48:27.999066+0000","last_clean":"2024-09-16T15:48:27.999066+0000","last_became_active":"2024-09-16T15:48:25.450170+0000","last_became_peered":"2024-09-16T15:48:25.450170+0000","last_unstale":"2024-09-16T15:48:27.999066+0000","last_undegraded":"2024-09-16T15:48:27.999066+0000","last_fullsized":"2024-09-16T15:48:27.999066+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:48:24.407732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:48:24.407732+0000","last_clean_scrub_stamp":"2024-09-16T15:48:24.407732+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T00:59:17.551961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.13","version":"0'0","reported_seq":13,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-09-16T15:48:27.997045+0000","last_change":"2024-09-16T15:48:26.468069+0000","last_active":"2024-09-16T15:48:27.997045+0000","last_peered":"2024-09-16T15:48:27.997045+0000","last_clean":"2024-09-16T15:48:27.997045+0000","last_became_active":"2024-09-16T15:48:26.466980+0000","last_became_peered":"2024-09-16T15:48:26.466980+0000","last_unstale":"2024-09-16T15:48:27.997045+0000","last_undegraded":"2024-09-16T15:48:27.997045+0000","last_fullsized":"2024-09-16T15:48:27.997045+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:48:25.420844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:48:25.420844+0000","last_clean_scrub_stamp":"2024-09-16T15:48:25.420844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T22:01:08.106237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.12","version":"0'0","reported_seq":15,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-09-16T15:48:27.466423+0000","last_change":"2024-09-16T15:48:25.459083+0000","last_active":"2024-09-16T15:48:27.466423+0000","last_peered":"2024-09-16T15:48:27.466423+0000","last_clean":"2024-09-16T15:48:27.466423+0000","last_became_active":"2024-09-16T15:48:25.457843+0000","last_became_peered":"2024-09-16T15:48:25.457843+0000","last_unstale":"2024-09-16T15:48:27.466423+0000","last_undegraded":"2024-09-16T15:48:27.466423+0000","last_fullsized":"2024-09-16T15:48:27.466423+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:48:24.407732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:48:24.407732+0000","last_clean_scrub_stamp":"2024-09-16T15:48:24.407732+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:00:41.831554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.11","version":"0'0","reported_seq":13,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-09-16T15:48:27.996755+0000","last_change":"2024-09-16T15:48:26.467575+0000","last_active":"2024-09-16T15:48:27.996755+0000","last_peered":"2024-09-16T15:48:27.996755+0000","last_clean":"2024-09-16T15:48:27.996755+0000","last_became_active":"2024-09-16T15:48:26.465430+0000","last_became_peered":"2024-09-16T15:48:26.465430+0000","last_unstale":"2024-09-16T15:48:27.996755+0000","last_undegraded":"2024-09-16T15:48:27.996755+0000","last_fullsized":"2024-09-16T15:48:27.996755+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:48:25.420844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:48:25.420844+0000","last_clean_scrub_stamp":"2024-09-16T15:48:25.420844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T22:09:48.246143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.10","version":"102'8","reported_seq":23,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-09-16T15:48:27.495629+0000","last_change":"2024-09-16T15:48:25.462164+0000","last_active":"2024-09-16T15:48:27.495629+0000","last_peered":"2024-09-16T15:48:27.495629+0000","last_clean":"2024-09-16T15:48:27.495629+0000","last_became_active":"2024-09-16T15:48:25.461886+0000","last_became_peered":"2024-09-16T15:48:25.461886+0000","last_unstale":"2024-09-16T15:48:27.495629+0000","last_undegraded":"2024-09-16T15:48:27.495629+0000","last_fullsized":"2024-09-16T15:48:27.495629+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:48:24.407732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:48:24.407732+0000","last_clean_scrub_stamp":"2024-09-16T15:48:24.407732+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:34:13.844138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.e","version":"0'0","reported_seq":13,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-09-16T15:48:27.496063+0000","last_change":"2024-09-16T15:48:26.501077+0000","last_active":"2024-09-16T15:48:27.496063+0000","last_peered":"2024-09-16T15:48:27.496063+0000","last_clean":"2024-09-16T15:48:27.496063+0000","last_became_active":"2024-09-16T15:48:26.500794+0000","last_became_peered":"2024-09-16T15:48:26.500794+0000","last_unstale":"2024-09-16T15:48:27.496063+0000","last_undegraded":"2024-09-16T15:48:27.496063+0000","last_fullsized":"2024-09-16T15:48:27.496063+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:48:25.420844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:48:25.420844+0000","last_clean_scrub_stamp":"2024-09-16T15:48:25.420844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:32:28.584284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.f","version":"102'1","reported_seq":16,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-09-16T15:48:27.462272+0000","last_change":"2024-09-16T15:48:25.461746+0000","last_active":"2024-09-16T15:48:27.462272+0000","last_peered":"2024-09-16T15:48:27.462272+0000","last_clean":"2024-09-16T15:48:27.462272+0000","last_became_active":"2024-09-16T15:48:25.461035+0000","last_became_peered":"2024-09-16T15:48:25.461035+0000","last_unstale":"2024-09-16T15:48:27.462272+0000","last_undegraded":"2024-09-16T15:48:27.462272+0000","last_fullsized":"2024-09-16T15:48:27.462272+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:48:24.407732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:48:24.407732+0000","last_clean_scrub_stamp":"2024-09-16T15:48:24.407732+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:07:43.870477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.f","version":"0'0","reported_seq":13,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-09-16T15:48:27.999658+0000","last_change":"2024-09-16T15:48:26.493307+0000","last_active":"2024-09-16T15:48:27.999658+0000","last_peered":"2024-09-16T15:48:27.999658+0000","last_clean":"2024-09-16T15:48:27.999658+0000","last_became_active":"2024-09-16T15:48:26.493028+0000","last_became_peered":"2024-09-16T15:48:26.493028+0000","last_unstale":"2024-09-16T15:48:27.999658+0000","last_undegraded":"2024-09-16T15:48:27.999658+0000","last_fullsized":"2024-09-16T15:48:27.999658+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:48:25.420844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:48:25.420844+0000","last_clean_scrub_stamp":"2024-09-16T15:48:25.420844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T22:46:45.269484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.e","version":"102'2","reported_seq":17,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-09-16T15:48:27.496162+0000","last_change":"2024-09-16T15:48:25.449185+0000","last_active":"2024-09-16T15:48:27.496162+0000","last_peered":"2024-09-16T15:48:27.496162+0000","last_clean":"2024-09-16T15:48:27.496162+0000","last_became_active":"2024-09-16T15:48:25.448865+0000","last_became_peered":"2024-09-16T15:48:25.448865+0000","last_unstale":"2024-09-16T15:48:27.496162+0000","last_undegraded":"2024-09-16T15:48:27.496162+0000","last_fullsized":"2024-09-16T15:48:27.496162+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:48:24.407732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:48:24.407732+0000","last_clean_scrub_stamp":"2024-09-16T15:48:24.407732+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:15:53.516955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.c","version":"0'0","reported_seq":13,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-09-16T15:48:27.887691+0000","last_change":"2024-09-16T15:48:26.502464+0000","last_active":"2024-09-16T15:48:27.887691+0000","last_peered":"2024-09-16T15:48:27.887691+0000","last_clean":"2024-09-16T15:48:27.887691+0000","last_became_active":"2024-09-16T15:48:26.501748+0000","last_became_peered":"2024-09-16T15:48:26.501748+0000","last_unstale":"2024-09-16T15:48:27.887691+0000","last_undegraded":"2024-09-16T15:48:27.887691+0000","last_fullsized":"2024-09-16T15:48:27.887691+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:48:25.420844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:48:25.420844+0000","last_clean_scrub_stamp":"2024-09-16T15:48:25.420844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T03:22:41.088805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.d","version":"102'1","reported_seq":16,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-09-16T15:48:27.466883+0000","last_change":"2024-09-16T15:48:25.452923+0000","last_active":"2024-09-16T15:48:27.466883+0000","last_peered":"2024-09-16T15:48:27.466883+0000","last_clean":"2024-09-16T15:48:27.466883+0000","last_became_active":"2024-09-16T15:48:25.452160+0000","last_became_peered":"2024-09-16T15:48:25.452160+0000","last_unstale":"2024-09-16T15:48:27.466883+0000","last_undegraded":"2024-09-16T15:48:27.466883+0000","last_fullsized":"2024-09-16T15:48:27.466883+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:48:24.407732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:48:24.407732+0000","last_clean_scrub_stamp":"2024-09-16T15:48:24.407732+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:55:18.056917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.d","version":"0'0","reported_seq":13,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-09-16T15:48:27.888231+0000","last_change":"2024-09-16T15:48:26.503986+0000","last_active":"2024-09-16T15:48:27.888231+0000","last_peered":"2024-09-16T15:48:27.888231+0000","last_clean":"2024-09-16T15:48:27.888231+0000","last_became_active":"2024-09-16T15:48:26.492181+0000","last_became_peered":"2024-09-16T15:48:26.492181+0000","last_unstale":"2024-09-16T15:48:27.888231+0000","last_undegraded":"2024-09-16T15:48:27.888231+0000","last_fullsized":"2024-09-16T15:48:27.888231+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:48:25.420844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:48:25.420844+0000","last_clean_scrub_stamp":"2024-09-16T15:48:25.420844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:43:22.624149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.c","version":"0'0","reported_seq":15,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-09-16T15:48:27.483316+0000","last_change":"2024-09-16T15:48:25.465602+0000","last_active":"2024-09-16T15:48:27.483316+0000","last_peered":"2024-09-16T15:48:27.483316+0000","last_clean":"2024-09-16T15:48:27.483316+0000","last_became_active":"2024-09-16T15:48:25.464705+0000","last_became_peered":"2024-09-16T15:48:25.464705+0000","last_unstale":"2024-09-16T15:48:27.483316+0000","last_undegraded":"2024-09-16T15:48:27.483316+0000","last_fullsized":"2024-09-16T15:48:27.483316+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:48:24.407732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:48:24.407732+0000","last_clean_scrub_stamp":"2024-09-16T15:48:24.407732+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T22:46:44.256372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.a","version":"0'0","reported_seq":13,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-09-16T15:48:27.887377+0000","last_change":"2024-09-16T15:48:26.509669+0000","last_active":"2024-09-16T15:48:27.887377+0000","last_peered":"2024-09-16T15:48:27.887377+0000","last_clean":"2024-09-16T15:48:27.887377+0000","last_became_active":"2024-09-16T15:48:26.509072+0000","last_became_peered":"2024-09-16T15:48:26.509072+0000","last_unstale":"2024-09-16T15:48:27.887377+0000","last_undegraded":"2024-09-16T15:48:27.887377+0000","last_fullsized":"2024-09-16T15:48:27.887377+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:48:25.420844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:48:25.420844+0000","last_clean_scrub_stamp":"2024-09-16T15:48:25.420844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:27:26.948503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.b","version":"102'2","reported_seq":17,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-09-16T15:48:27.999293+0000","last_change":"2024-09-16T15:48:25.446745+0000","last_active":"2024-09-16T15:48:27.999293+0000","last_peered":"2024-09-16T15:48:27.999293+0000","last_clean":"2024-09-16T15:48:27.999293+0000","last_became_active":"2024-09-16T15:48:25.446466+0000","last_became_peered":"2024-09-16T15:48:25.446466+0000","last_unstale":"2024-09-16T15:48:27.999293+0000","last_undegraded":"2024-09-16T15:48:27.999293+0000","last_fullsized":"2024-09-16T15:48:27.999293+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:48:24.407732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:48:24.407732+0000","last_clean_scrub_stamp":"2024-09-16T15:48:24.407732+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T22:55:53.466832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.b","version":"0'0","reported_seq":13,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-09-16T15:48:27.463002+0000","last_change":"2024-09-16T15:48:26.502896+0000","last_active":"2024-09-16T15:48:27.463002+0000","last_peered":"2024-09-16T15:48:27.463002+0000","last_clean":"2024-09-16T15:48:27.463002+0000","last_became_active":"2024-09-16T15:48:26.502324+0000","last_became_peered":"2024-09-16T15:48:26.502324+0000","last_unstale":"2024-09-16T15:48:27.463002+0000","last_undegraded":"2024-09-16T15:48:27.463002+0000","last_fullsized":"2024-09-16T15:48:27.463002+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:48:25.420844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:48:25.420844+0000","last_clean_scrub_stamp":"2024-09-16T15:48:25.420844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T02:50:17.688000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.a","version":"0'0","reported_seq":15,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-09-16T15:48:27.495549+0000","last_change":"2024-09-16T15:48:25.449113+0000","last_active":"2024-09-16T15:48:27.495549+0000","last_peered":"2024-09-16T15:48:27.495549+0000","last_clean":"2024-09-16T15:48:27.495549+0000","last_became_active":"2024-09-16T15:48:25.448377+0000","last_became_peered":"2024-09-16T15:48:25.448377+0000","last_unstale":"2024-09-16T15:48:27.495549+0000","last_undegraded":"2024-09-16T15:48:27.495549+0000","last_fullsized":"2024-09-16T15:48:27.495549+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:48:24.407732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:48:24.407732+0000","last_clean_scrub_stamp":"2024-09-16T15:48:24.407732+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:00:13.928185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.8","version":"0'0","reported_seq":13,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-09-16T15:48:27.887817+0000","last_change":"2024-09-16T15:48:26.500174+0000","last_active":"2024-09-16T15:48:27.887817+0000","last_peered":"2024-09-16T15:48:27.887817+0000","last_clean":"2024-09-16T15:48:27.887817+0000","last_became_active":"2024-09-16T15:48:26.492446+0000","last_became_peered":"2024-09-16T15:48:26.492446+0000","last_unstale":"2024-09-16T15:48:27.887817+0000","last_undegraded":"2024-09-16T15:48:27.887817+0000","last_fullsized":"2024-09-16T15:48:27.887817+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:48:25.420844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:48:25.420844+0000","last_clean_scrub_stamp":"2024-09-16T15:48:25.420844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T18:03:38.683560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.9","version":"0'0","reported_seq":15,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-09-16T15:48:27.998833+0000","last_change":"2024-09-16T15:48:25.447068+0000","last_active":"2024-09-16T15:48:27.998833+0000","last_peered":"2024-09-16T15:48:27.998833+0000","last_clean":"2024-09-16T15:48:27.998833+0000","last_became_active":"2024-09-16T15:48:25.446737+0000","last_became_peered":"2024-09-16T15:48:25.446737+0000","last_unstale":"2024-09-16T15:48:27.998833+0000","last_undegraded":"2024-09-16T15:48:27.998833+0000","last_fullsized":"2024-09-16T15:48:27.998833+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:48:24.407732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:48:24.407732+0000","last_clean_scrub_stamp":"2024-09-16T15:48:24.407732+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T00:27:59.760074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.9","version":"0'0","reported_seq":13,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-09-16T15:48:27.996655+0000","last_change":"2024-09-16T15:48:26.488945+0000","last_active":"2024-09-16T15:48:27.996655+0000","last_peered":"2024-09-16T15:48:27.996655+0000","last_clean":"2024-09-16T15:48:27.996655+0000","last_became_active":"2024-09-16T15:48:26.488681+0000","last_became_peered":"2024-09-16T15:48:26.488681+0000","last_unstale":"2024-09-16T15:48:27.996655+0000","last_undegraded":"2024-09-16T15:48:27.996655+0000","last_fullsized":"2024-09-16T15:48:27.996655+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:48:25.420844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:48:25.420844+0000","last_clean_scrub_stamp":"2024-09-16T15:48:25.420844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T02:48:53.694369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.8","version":"0'0","reported_seq":15,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-09-16T15:48:27.461938+0000","last_change":"2024-09-16T15:48:25.464185+0000","last_active":"2024-09-16T15:48:27.461938+0000","last_peered":"2024-09-16T15:48:27.461938+0000","last_clean":"2024-09-16T15:48:27.461938+0000","last_became_active":"2024-09-16T15:48:25.463336+0000","last_became_peered":"2024-09-16T15:48:25.463336+0000","last_unstale":"2024-09-16T15:48:27.461938+0000","last_undegraded":"2024-09-16T15:48:27.461938+0000","last_fullsized":"2024-09-16T15:48:27.461938+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:48:24.407732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:48:24.407732+0000","last_clean_scrub_stamp":"2024-09-16T15:48:24.407732+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T03:22:40.075693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.6","version":"0'0","reported_seq":13,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-09-16T15:48:27.467448+0000","last_change":"2024-09-16T15:48:26.501860+0000","last_active":"2024-09-16T15:48:27.467448+0000","last_peered":"2024-09-16T15:48:27.467448+0000","last_clean":"2024-09-16T15:48:27.467448+0000","last_became_active":"2024-09-16T15:48:26.500898+0000","last_became_peered":"2024-09-16T15:48:26.500898+0000","last_unstale":"2024-09-16T15:48:27.467448+0000","last_undegraded":"2024-09-16T15:48:27.467448+0000","last_fullsized":"2024-09-16T15:48:27.467448+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:48:25.420844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:48:25.420844+0000","last_clean_scrub_stamp":"2024-09-16T15:48:25.420844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:17:26.625596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.7","version":"102'2","reported_seq":17,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-09-16T15:48:27.997022+0000","last_change":"2024-09-16T15:48:25.463111+0000","last_active":"2024-09-16T15:48:27.997022+0000","last_peered":"2024-09-16T15:48:27.997022+0000","last_clean":"2024-09-16T15:48:27.997022+0000","last_became_active":"2024-09-16T15:48:25.462771+0000","last_became_peered":"2024-09-16T15:48:25.462771+0000","last_unstale":"2024-09-16T15:48:27.997022+0000","last_undegraded":"2024-09-16T15:48:27.997022+0000","last_fullsized":"2024-09-16T15:48:27.997022+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:48:24.407732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:48:24.407732+0000","last_clean_scrub_stamp":"2024-09-16T15:48:24.407732+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:19:56.751913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.7","version":"0'0","reported_seq":13,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-09-16T15:48:27.462125+0000","last_change":"2024-09-16T15:48:26.492559+0000","last_active":"2024-09-16T15:48:27.462125+0000","last_peered":"2024-09-16T15:48:27.462125+0000","last_clean":"2024-09-16T15:48:27.462125+0000","last_became_active":"2024-09-16T15:48:26.492232+0000","last_became_peered":"2024-09-16T15:48:26.492232+0000","last_unstale":"2024-09-16T15:48:27.462125+0000","last_undegraded":"2024-09-16T15:48:27.462125+0000","last_fullsized":"2024-09-16T15:48:27.462125+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:48:25.420844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:48:25.420844+0000","last_clean_scrub_stamp":"2024-09-16T15:48:25.420844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:21:55.934422+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.6","version":"103'1","reported_seq":16,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-09-16T15:48:27.467783+0000","last_change":"2024-09-16T15:48:25.458600+0000","last_active":"2024-09-16T15:48:27.467783+0000","last_peered":"2024-09-16T15:48:27.467783+0000","last_clean":"2024-09-16T15:48:27.467783+0000","last_became_active":"2024-09-16T15:48:25.457924+0000","last_became_peered":"2024-09-16T15:48:25.457924+0000","last_unstale":"2024-09-16T15:48:27.467783+0000","last_undegraded":"2024-09-16T15:48:27.467783+0000","last_fullsized":"2024-09-16T15:48:27.467783+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:48:24.407732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:48:24.407732+0000","last_clean_scrub_stamp":"2024-09-16T15:48:24.407732+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:07:47.395564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.4","version":"0'0","reported_seq":13,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-09-16T15:48:27.462681+0000","last_change":"2024-09-16T15:48:26.496183+0000","last_active":"2024-09-16T15:48:27.462681+0000","last_peered":"2024-09-16T15:48:27.462681+0000","last_clean":"2024-09-16T15:48:27.462681+0000","last_became_active":"2024-09-16T15:48:26.493124+0000","last_became_peered":"2024-09-16T15:48:26.493124+0000","last_unstale":"2024-09-16T15:48:27.462681+0000","last_undegraded":"2024-09-16T15:48:27.462681+0000","last_fullsized":"2024-09-16T15:48:27.462681+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:48:25.420844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:48:25.420844+0000","last_clean_scrub_stamp":"2024-09-16T15:48:25.420844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:50:48.377270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.5","version":"0'0","reported_seq":15,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-09-16T15:48:27.462847+0000","last_change":"2024-09-16T15:48:25.463122+0000","last_active":"2024-09-16T15:48:27.462847+0000","last_peered":"2024-09-16T15:48:27.462847+0000","last_clean":"2024-09-16T15:48:27.462847+0000","last_became_active":"2024-09-16T15:48:25.462802+0000","last_became_peered":"2024-09-16T15:48:25.462802+0000","last_unstale":"2024-09-16T15:48:27.462847+0000","last_undegraded":"2024-09-16T15:48:27.462847+0000","last_fullsized":"2024-09-16T15:48:27.462847+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:48:24.407732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:48:24.407732+0000","last_clean_scrub_stamp":"2024-09-16T15:48:24.407732+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T18:03:37.670448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.5","version":"0'0","reported_seq":13,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-09-16T15:48:27.466706+0000","last_change":"2024-09-16T15:48:26.488782+0000","last_active":"2024-09-16T15:48:27.466706+0000","last_peered":"2024-09-16T15:48:27.466706+0000","last_clean":"2024-09-16T15:48:27.466706+0000","last_became_active":"2024-09-16T15:48:26.487907+0000","last_became_peered":"2024-09-16T15:48:26.487907+0000","last_unstale":"2024-09-16T15:48:27.466706+0000","last_undegraded":"2024-09-16T15:48:27.466706+0000","last_fullsized":"2024-09-16T15:48:27.466706+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:48:25.420844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:48:25.420844+0000","last_clean_scrub_stamp":"2024-09-16T15:48:25.420844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:18:22.614416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.4","version":"0'0","reported_seq":15,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-09-16T15:48:27.997404+0000","last_change":"2024-09-16T15:48:25.449183+0000","last_active":"2024-09-16T15:48:27.997404+0000","last_peered":"2024-09-16T15:48:27.997404+0000","last_clean":"2024-09-16T15:48:27.997404+0000","last_became_active":"2024-09-16T15:48:25.448828+0000","last_became_peered":"2024-09-16T15:48:25.448828+0000","last_unstale":"2024-09-16T15:48:27.997404+0000","last_undegraded":"2024-09-16T15:48:27.997404+0000","last_fullsized":"2024-09-16T15:48:27.997404+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:48:24.407732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:48:24.407732+0000","last_clean_scrub_stamp":"2024-09-16T15:48:24.407732+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T02:56:07.811567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.2","version":"0'0","reported_seq":13,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-09-16T15:48:27.465265+0000","last_change":"2024-09-16T15:48:26.493658+0000","last_active":"2024-09-16T15:48:27.465265+0000","last_peered":"2024-09-16T15:48:27.465265+0000","last_clean":"2024-09-16T15:48:27.465265+0000","last_became_active":"2024-09-16T15:48:26.492904+0000","last_became_peered":"2024-09-16T15:48:26.492904+0000","last_unstale":"2024-09-16T15:48:27.465265+0000","last_undegraded":"2024-09-16T15:48:27.465265+0000","last_fullsized":"2024-09-16T15:48:27.465265+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:48:25.420844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:48:25.420844+0000","last_clean_scrub_stamp":"2024-09-16T15:48:25.420844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:29:34.405446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.3","version":"0'0","reported_seq":15,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-09-16T15:48:27.483021+0000","last_change":"2024-09-16T15:48:25.454707+0000","last_active":"2024-09-16T15:48:27.483021+0000","last_peered":"2024-09-16T15:48:27.483021+0000","last_clean":"2024-09-16T15:48:27.483021+0000","last_became_active":"2024-09-16T15:48:25.453977+0000","last_became_peered":"2024-09-16T15:48:25.453977+0000","last_unstale":"2024-09-16T15:48:27.483021+0000","last_undegraded":"2024-09-16T15:48:27.483021+0000","last_fullsized":"2024-09-16T15:48:27.483021+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:48:24.407732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:48:24.407732+0000","last_clean_scrub_stamp":"2024-09-16T15:48:24.407732+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T00:21:40.348171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.3","version":"0'0","reported_seq":13,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-09-16T15:48:27.466330+0000","last_change":"2024-09-16T15:48:26.471300+0000","last_active":"2024-09-16T15:48:27.466330+0000","last_peered":"2024-09-16T15:48:27.466330+0000","last_clean":"2024-09-16T15:48:27.466330+0000","last_became_active":"2024-09-16T15:48:26.470479+0000","last_became_peered":"2024-09-16T15:48:26.470479+0000","last_unstale":"2024-09-16T15:48:27.466330+0000","last_undegraded":"2024-09-16T15:48:27.466330+0000","last_fullsized":"2024-09-16T15:48:27.466330+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:48:25.420844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:48:25.420844+0000","last_clean_scrub_stamp":"2024-09-16T15:48:25.420844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T18:36:32.286233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.2","version":"0'0","reported_seq":15,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-09-16T15:48:27.465455+0000","last_change":"2024-09-16T15:48:25.449899+0000","last_active":"2024-09-16T15:48:27.465455+0000","last_peered":"2024-09-16T15:48:27.465455+0000","last_clean":"2024-09-16T15:48:27.465455+0000","last_became_active":"2024-09-16T15:48:25.449158+0000","last_became_peered":"2024-09-16T15:48:25.449158+0000","last_unstale":"2024-09-16T15:48:27.465455+0000","last_undegraded":"2024-09-16T15:48:27.465455+0000","last_fullsized":"2024-09-16T15:48:27.465455+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:48:24.407732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:48:24.407732+0000","last_clean_scrub_stamp":"2024-09-16T15:48:24.407732+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:06:09.718860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.0","version":"0'0","reported_seq":13,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-09-16T15:48:27.888643+0000","last_change":"2024-09-16T15:48:26.499317+0000","last_active":"2024-09-16T15:48:27.888643+0000","last_peered":"2024-09-16T15:48:27.888643+0000","last_clean":"2024-09-16T15:48:27.888643+0000","last_became_active":"2024-09-16T15:48:26.493152+0000","last_became_peered":"2024-09-16T15:48:26.493152+0000","last_unstale":"2024-09-16T15:48:27.888643+0000","last_undegraded":"2024-09-16T15:48:27.888643+0000","last_fullsized":"2024-09-16T15:48:27.888643+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:48:25.420844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:48:25.420844+0000","last_clean_scrub_stamp":"2024-09-16T15:48:25.420844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:16:49.823764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.1","version":"0'0","reported_seq":15,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-09-16T15:48:27.495694+0000","last_change":"2024-09-16T15:48:25.448607+0000","last_active":"2024-09-16T15:48:27.495694+0000","last_peered":"2024-09-16T15:48:27.495694+0000","last_clean":"2024-09-16T15:48:27.495694+0000","last_became_active":"2024-09-16T15:48:25.448153+0000","last_became_peered":"2024-09-16T15:48:25.448153+0000","last_unstale":"2024-09-16T15:48:27.495694+0000","last_undegraded":"2024-09-16T15:48:27.495694+0000","last_fullsized":"2024-09-16T15:48:27.495694+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:48:24.407732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:48:24.407732+0000","last_clean_scrub_stamp":"2024-09-16T15:48:24.407732+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:18:45.048781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1","version":"0'0","reported_seq":13,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-09-16T15:48:27.462262+0000","last_change":"2024-09-16T15:48:26.488351+0000","last_active":"2024-09-16T15:48:27.462262+0000","last_peered":"2024-09-16T15:48:27.462262+0000","last_clean":"2024-09-16T15:48:27.462262+0000","last_became_active":"2024-09-16T15:48:26.487799+0000","last_became_peered":"2024-09-16T15:48:26.487799+0000","last_unstale":"2024-09-16T15:48:27.462262+0000","last_undegraded":"2024-09-16T15:48:27.462262+0000","last_fullsized":"2024-09-16T15:48:27.462262+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:48:25.420844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:48:25.420844+0000","last_clean_scrub_stamp":"2024-09-16T15:48:25.420844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T23:34:29.350068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.0","version":"0'0","reported_seq":15,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-09-16T15:48:27.462196+0000","last_change":"2024-09-16T15:48:25.463812+0000","last_active":"2024-09-16T15:48:27.462196+0000","last_peered":"2024-09-16T15:48:27.462196+0000","last_clean":"2024-09-16T15:48:27.462196+0000","last_became_active":"2024-09-16T15:48:25.462975+0000","last_became_peered":"2024-09-16T15:48:25.462975+0000","last_unstale":"2024-09-16T15:48:27.462196+0000","last_undegraded":"2024-09-16T15:48:27.462196+0000","last_fullsized":"2024-09-16T15:48:27.462196+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:48:24.407732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:48:24.407732+0000","last_clean_scrub_stamp":"2024-09-16T15:48:24.407732+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T22:10:30.282104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.10","version":"0'0","reported_seq":13,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-09-16T15:48:27.996502+0000","last_change":"2024-09-16T15:48:26.471378+0000","last_active":"2024-09-16T15:48:27.996502+0000","last_peered":"2024-09-16T15:48:27.996502+0000","last_clean":"2024-09-16T15:48:27.996502+0000","last_became_active":"2024-09-16T15:48:26.470643+0000","last_became_peered":"2024-09-16T15:48:26.470643+0000","last_unstale":"2024-09-16T15:48:27.996502+0000","last_undegraded":"2024-09-16T15:48:27.996502+0000","last_fullsized":"2024-09-16T15:48:27.996502+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:48:25.420844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:48:25.420844+0000","last_clean_scrub_stamp":"2024-09-16T15:48:25.420844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:17:26.625596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.11","version":"0'0","reported_seq":15,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-09-16T15:48:27.483436+0000","last_change":"2024-09-16T15:48:25.463108+0000","last_active":"2024-09-16T15:48:27.483436+0000","last_peered":"2024-09-16T15:48:27.483436+0000","last_clean":"2024-09-16T15:48:27.483436+0000","last_became_active":"2024-09-16T15:48:25.462386+0000","last_became_peered":"2024-09-16T15:48:25.462386+0000","last_unstale":"2024-09-16T15:48:27.483436+0000","last_undegraded":"2024-09-16T15:48:27.483436+0000","last_fullsized":"2024-09-16T15:48:27.483436+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:48:24.407732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:48:24.407732+0000","last_clean_scrub_stamp":"2024-09-16T15:48:24.407732+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T00:25:43.259607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"73'79","reported_seq":291,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-09-16T15:48:27.888137+0000","last_change":"2024-09-16T15:42:40.881820+0000","last_active":"2024-09-16T15:48:27.888137+0000","last_peered":"2024-09-16T15:48:27.888137+0000","last_clean":"2024-09-16T15:48:27.888137+0000","last_became_active":"2024-09-16T15:42:40.879965+0000","last_became_peered":"2024-09-16T15:42:40.879965+0000","last_unstale":"2024-09-16T15:48:27.888137+0000","last_undegraded":"2024-09-16T15:48:27.888137+0000","last_fullsized":"2024-09-16T15:48:27.888137+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:42:39.728762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:42:39.728762+0000","last_clean_scrub_stamp":"2024-09-16T15:42:39.728762+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:22:18.768932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":17,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":16,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79,"ondisk_log_size":79,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542218,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33024,"kb_used_data":1836,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338816,"statfs":{"total":96636764160,"available":96602947584,"internally_reserved":0,"allocated":1880064,"data_stored":1150644,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542217,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32440,"kb_used_data":1252,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339400,"statfs":{"total":96636764160,"available":96603545600,"internally_reserved":0,"allocated":1282048,"data_stored":559730,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":5,"up_from":14,"seq":60129542218,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32448,"kb_used_data":1256,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339392,"statfs":{"total":96636764160,"available":96603537408,"internally_reserved":0,"allocated":1286144,"data_stored":560264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542216,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32436,"kb_used_data":1248,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339404,"statfs":{"total":96636764160,"available":96603549696,"internally_reserved":0,"allocated":1277952,"data_stored":559660,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[]},{"osd":3,"up_from":14,"seq":60129542217,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32452,"kb_used_data":1264,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339388,"statfs":{"total":96636764160,"available":96603533312,"internally_reserved":0,"allocated":1294336,"data_stored":561342,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":2,"up_from":14,"seq":60129542217,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32440,"kb_used_data":1248,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339400,"statfs":{"total":96636764160,"available":96603545600,"internally_reserved":0,"allocated":1277952,"data_stored":559660,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":14,"seq":60129542219,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32448,"kb_used_data":1256,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339392,"statfs":{"total":96636764160,"available":96603537408,"internally_reserved":0,"allocated":1286144,"data_stored":559784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":14,"seq":60129542219,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33044,"kb_used_data":1848,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338796,"statfs":{"total":96636764160,"available":96602927104,"internally_reserved":0,"allocated":1892352,"data_stored":1151752,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T15:48:30.515 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T15:48:30.674 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:30.675+0000 7fa2bc11f640 1 -- 172.21.15.123:0/3214718273 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa2b40afd60 msgr2=0x7fa2b41638d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:48:30.674 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:30.675+0000 7fa2bc11f640 1 --2- 172.21.15.123:0/3214718273 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa2b40afd60 0x7fa2b41638d0 secure :-1 s=READY pgs=1078 cs=0 l=1 rev1=1 crypto rx=0x7fa2b40b66b0 tx=0x7fa2b001cd10 comp rx=0 tx=0).stop 2024-09-16T15:48:30.674 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:30.675+0000 7fa2bc11f640 1 -- 172.21.15.123:0/3214718273 shutdown_connections 2024-09-16T15:48:30.674 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:30.675+0000 7fa2bc11f640 1 --2- 172.21.15.123:0/3214718273 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fa2b406d300 0x7fa2b4165e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:30.675 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:30.675+0000 7fa2bc11f640 1 --2- 172.21.15.123:0/3214718273 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa2b40afd60 0x7fa2b41638d0 unknown :-1 s=CLOSED pgs=1078 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:30.675 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:30.675+0000 7fa2bc11f640 1 -- 172.21.15.123:0/3214718273 >> 172.21.15.123:0/3214718273 conn(0x7fa2b40b2610 msgr2=0x7fa2b40b29f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:48:30.675 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:30.675+0000 7fa2bc11f640 1 -- 172.21.15.123:0/3214718273 shutdown_connections 2024-09-16T15:48:30.675 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:30.675+0000 7fa2bc11f640 1 -- 172.21.15.123:0/3214718273 wait complete. 2024-09-16T15:48:30.675 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:30.675+0000 7fa2bc11f640 1 Processor -- start 2024-09-16T15:48:30.675 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:30.675+0000 7fa2bc11f640 1 -- start start 2024-09-16T15:48:30.677 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:30.675+0000 7fa2bc11f640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa2b406d300 0x7fa2b41f7d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:48:30.677 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:30.675+0000 7fa2bc11f640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fa2b40afd60 0x7fa2b40cfd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:48:30.677 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:30.675+0000 7fa2bc11f640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa2b415bd30 0x7fa2b40d02a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:48:30.677 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:30.675+0000 7fa2bc11f640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fa2b4168f60 con 0x7fa2b406d300 2024-09-16T15:48:30.677 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:30.675+0000 7fa2bc11f640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fa2b4168de0 con 0x7fa2b415bd30 2024-09-16T15:48:30.677 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:30.675+0000 7fa2bc11f640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fa2b41690e0 con 0x7fa2b40afd60 2024-09-16T15:48:30.677 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:30.675+0000 7fa2b9ebb640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa2b406d300 0x7fa2b41f7d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:48:30.677 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:30.675+0000 7fa2b9ebb640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa2b406d300 0x7fa2b41f7d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:47530/0 (socket says 172.21.15.123:47530) 2024-09-16T15:48:30.677 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:30.675+0000 7fa2b9ebb640 1 -- 172.21.15.123:0/2988402192 learned_addr learned my addr 172.21.15.123:0/2988402192 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:48:30.677 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:30.675+0000 7fa2b9ebb640 1 -- 172.21.15.123:0/2988402192 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fa2b40afd60 msgr2=0x7fa2b40cfd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:48:30.678 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:30.675+0000 7fa2b9ebb640 1 --2- 172.21.15.123:0/2988402192 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fa2b40afd60 0x7fa2b40cfd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:30.678 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:30.675+0000 7fa2b9ebb640 1 -- 172.21.15.123:0/2988402192 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa2b415bd30 msgr2=0x7fa2b40d02a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:48:30.678 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:30.675+0000 7fa2b9ebb640 1 --2- 172.21.15.123:0/2988402192 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa2b415bd30 0x7fa2b40d02a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:30.678 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:30.675+0000 7fa2b9ebb640 1 -- 172.21.15.123:0/2988402192 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa2b41fb3e0 con 0x7fa2b406d300 2024-09-16T15:48:30.678 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:30.675+0000 7fa2b9ebb640 1 --2- 172.21.15.123:0/2988402192 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa2b406d300 0x7fa2b41f7d50 secure :-1 s=READY pgs=1079 cs=0 l=1 rev1=1 crypto rx=0x7fa2a400c3c0 tx=0x7fa2a400c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:48:30.678 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:30.675+0000 7fa2aa7fc640 1 -- 172.21.15.123:0/2988402192 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2a4017070 con 0x7fa2b406d300 2024-09-16T15:48:30.678 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:30.675+0000 7fa2aa7fc640 1 -- 172.21.15.123:0/2988402192 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fa2a40053c0 con 0x7fa2b406d300 2024-09-16T15:48:30.678 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:30.675+0000 7fa2aa7fc640 1 -- 172.21.15.123:0/2988402192 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2a4003860 con 0x7fa2b406d300 2024-09-16T15:48:30.678 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:30.675+0000 7fa2bc11f640 1 -- 172.21.15.123:0/2988402192 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa2b4154970 con 0x7fa2b406d300 2024-09-16T15:48:30.678 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:30.675+0000 7fa2bc11f640 1 -- 172.21.15.123:0/2988402192 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa2b4154f30 con 0x7fa2b406d300 2024-09-16T15:48:30.679 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:30.679+0000 7fa2bc11f640 1 -- 172.21.15.123:0/2988402192 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa2b406cbf0 con 0x7fa2b406d300 2024-09-16T15:48:30.679 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:30.679+0000 7fa2aa7fc640 1 -- 172.21.15.123:0/2988402192 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 16) ==== 97182+0+0 (secure 0 0 0) 0x7fa2a401a070 con 0x7fa2b406d300 2024-09-16T15:48:30.679 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:30.679+0000 7fa2aa7fc640 1 --2- 172.21.15.123:0/2988402192 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fa2a0072fe0 0x7fa2a00754a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:48:30.679 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:30.679+0000 7fa2aa7fc640 1 -- 172.21.15.123:0/2988402192 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(103..103 src has 1..103) ==== 5709+0+0 (secure 0 0 0) 0x7fa2a4093300 con 0x7fa2b406d300 2024-09-16T15:48:30.679 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:30.679+0000 7fa2b96ba640 1 --2- 172.21.15.123:0/2988402192 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fa2a0072fe0 0x7fa2a00754a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:48:30.680 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:30.679+0000 7fa2b96ba640 1 --2- 172.21.15.123:0/2988402192 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fa2a0072fe0 0x7fa2a00754a0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fa2b00af000 tx=0x7fa2b0019ab0 comp rx=0 tx=0).ready entity=mgr.4150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:48:30.683 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:30.683+0000 7fa2aa7fc640 1 -- 172.21.15.123:0/2988402192 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7fa2a405e3c0 con 0x7fa2b406d300 2024-09-16T15:48:30.838 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:30.839+0000 7fa2bc11f640 1 -- 172.21.15.123:0/2988402192 --> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fa2b40b42e0 con 0x7fa2a0072fe0 2024-09-16T15:48:30.845 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:30.843+0000 7fa2aa7fc640 1 -- 172.21.15.123:0/2988402192 <== mgr.4150 v2:172.21.15.123:6836/2464759676 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+252481 (secure 0 0 0) 0x7fa2b40b42e0 con 0x7fa2a0072fe0 2024-09-16T15:48:30.847 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-09-16T15:48:30.848 INFO:teuthology.orchestra.run.smithi123.stderr:dumped all 2024-09-16T15:48:30.849 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:30.847+0000 7fa2bc11f640 1 -- 172.21.15.123:0/2988402192 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fa2a0072fe0 msgr2=0x7fa2a00754a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:48:30.850 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:30.847+0000 7fa2bc11f640 1 --2- 172.21.15.123:0/2988402192 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fa2a0072fe0 0x7fa2a00754a0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fa2b00af000 tx=0x7fa2b0019ab0 comp rx=0 tx=0).stop 2024-09-16T15:48:30.850 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:30.847+0000 7fa2bc11f640 1 -- 172.21.15.123:0/2988402192 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa2b406d300 msgr2=0x7fa2b41f7d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:48:30.850 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:30.847+0000 7fa2bc11f640 1 --2- 172.21.15.123:0/2988402192 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa2b406d300 0x7fa2b41f7d50 secure :-1 s=READY pgs=1079 cs=0 l=1 rev1=1 crypto rx=0x7fa2a400c3c0 tx=0x7fa2a400c880 comp rx=0 tx=0).stop 2024-09-16T15:48:30.850 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:30.847+0000 7fa2bc11f640 1 -- 172.21.15.123:0/2988402192 shutdown_connections 2024-09-16T15:48:30.850 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:30.847+0000 7fa2bc11f640 1 --2- 172.21.15.123:0/2988402192 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fa2a0072fe0 0x7fa2a00754a0 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:30.850 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:30.847+0000 7fa2bc11f640 1 --2- 172.21.15.123:0/2988402192 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa2b415bd30 0x7fa2b40d02a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:30.850 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:30.847+0000 7fa2bc11f640 1 --2- 172.21.15.123:0/2988402192 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fa2b40afd60 0x7fa2b40cfd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:30.850 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:30.847+0000 7fa2bc11f640 1 --2- 172.21.15.123:0/2988402192 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa2b406d300 0x7fa2b41f7d50 unknown :-1 s=CLOSED pgs=1079 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:30.850 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:30.847+0000 7fa2bc11f640 1 -- 172.21.15.123:0/2988402192 >> 172.21.15.123:0/2988402192 conn(0x7fa2b40b2610 msgr2=0x7fa2b415ac30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:48:30.850 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:30.847+0000 7fa2bc11f640 1 -- 172.21.15.123:0/2988402192 shutdown_connections 2024-09-16T15:48:30.850 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:30.847+0000 7fa2bc11f640 1 -- 172.21.15.123:0/2988402192 wait complete. 2024-09-16T15:48:30.864 INFO:teuthology.orchestra.run.smithi123.stdout:{"pg_ready":true,"pg_map":{"version":62,"stamp":"2024-09-16T15:48:30.083454+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":158,"num_write_kb":1642,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":118,"ondisk_log_size":118,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":260732,"kb_used_data":11208,"kb_used_omap":12,"kb_used_meta":249331,"kb_avail":754713988,"statfs":{"total":773094113280,"available":772827123712,"internally_reserved":0,"allocated":11476992,"data_stored":5662836,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":255315505},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":44,"apply_latency_ms":44,"commit_latency_ns":44000000,"apply_latency_ns":44000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.660544"},"pg_stats":[{"pgid":"17.3f","version":"0'0","reported_seq":13,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-09-16T15:48:27.461697+0000","last_change":"2024-09-16T15:48:26.492027+0000","last_active":"2024-09-16T15:48:27.461697+0000","last_peered":"2024-09-16T15:48:27.461697+0000","last_clean":"2024-09-16T15:48:27.461697+0000","last_became_active":"2024-09-16T15:48:26.490634+0000","last_became_peered":"2024-09-16T15:48:26.490634+0000","last_unstale":"2024-09-16T15:48:27.461697+0000","last_undegraded":"2024-09-16T15:48:27.461697+0000","last_fullsized":"2024-09-16T15:48:27.461697+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:48:25.420844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:48:25.420844+0000","last_clean_scrub_stamp":"2024-09-16T15:48:25.420844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:26:46.532167+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.3e","version":"0'0","reported_seq":13,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-09-16T15:48:27.468269+0000","last_change":"2024-09-16T15:48:26.488569+0000","last_active":"2024-09-16T15:48:27.468269+0000","last_peered":"2024-09-16T15:48:27.468269+0000","last_clean":"2024-09-16T15:48:27.468269+0000","last_became_active":"2024-09-16T15:48:26.487794+0000","last_became_peered":"2024-09-16T15:48:26.487794+0000","last_unstale":"2024-09-16T15:48:27.468269+0000","last_undegraded":"2024-09-16T15:48:27.468269+0000","last_fullsized":"2024-09-16T15:48:27.468269+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:48:25.420844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:48:25.420844+0000","last_clean_scrub_stamp":"2024-09-16T15:48:25.420844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:22:45.568480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.3d","version":"0'0","reported_seq":13,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-09-16T15:48:27.887548+0000","last_change":"2024-09-16T15:48:26.510041+0000","last_active":"2024-09-16T15:48:27.887548+0000","last_peered":"2024-09-16T15:48:27.887548+0000","last_clean":"2024-09-16T15:48:27.887548+0000","last_became_active":"2024-09-16T15:48:26.509528+0000","last_became_peered":"2024-09-16T15:48:26.509528+0000","last_unstale":"2024-09-16T15:48:27.887548+0000","last_undegraded":"2024-09-16T15:48:27.887548+0000","last_fullsized":"2024-09-16T15:48:27.887548+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:48:25.420844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:48:25.420844+0000","last_clean_scrub_stamp":"2024-09-16T15:48:25.420844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T01:28:35.015978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.3c","version":"0'0","reported_seq":13,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-09-16T15:48:27.462135+0000","last_change":"2024-09-16T15:48:26.509093+0000","last_active":"2024-09-16T15:48:27.462135+0000","last_peered":"2024-09-16T15:48:27.462135+0000","last_clean":"2024-09-16T15:48:27.462135+0000","last_became_active":"2024-09-16T15:48:26.508843+0000","last_became_peered":"2024-09-16T15:48:26.508843+0000","last_unstale":"2024-09-16T15:48:27.462135+0000","last_undegraded":"2024-09-16T15:48:27.462135+0000","last_fullsized":"2024-09-16T15:48:27.462135+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:48:25.420844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:48:25.420844+0000","last_clean_scrub_stamp":"2024-09-16T15:48:25.420844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T01:40:10.109130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.3b","version":"0'0","reported_seq":13,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-09-16T15:48:27.496431+0000","last_change":"2024-09-16T15:48:26.481161+0000","last_active":"2024-09-16T15:48:27.496431+0000","last_peered":"2024-09-16T15:48:27.496431+0000","last_clean":"2024-09-16T15:48:27.496431+0000","last_became_active":"2024-09-16T15:48:26.480831+0000","last_became_peered":"2024-09-16T15:48:26.480831+0000","last_unstale":"2024-09-16T15:48:27.496431+0000","last_undegraded":"2024-09-16T15:48:27.496431+0000","last_fullsized":"2024-09-16T15:48:27.496431+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:48:25.420844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:48:25.420844+0000","last_clean_scrub_stamp":"2024-09-16T15:48:25.420844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T03:07:09.728518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.3a","version":"0'0","reported_seq":13,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-09-16T15:48:27.999012+0000","last_change":"2024-09-16T15:48:26.476018+0000","last_active":"2024-09-16T15:48:27.999012+0000","last_peered":"2024-09-16T15:48:27.999012+0000","last_clean":"2024-09-16T15:48:27.999012+0000","last_became_active":"2024-09-16T15:48:26.475640+0000","last_became_peered":"2024-09-16T15:48:26.475640+0000","last_unstale":"2024-09-16T15:48:27.999012+0000","last_undegraded":"2024-09-16T15:48:27.999012+0000","last_fullsized":"2024-09-16T15:48:27.999012+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:48:25.420844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:48:25.420844+0000","last_clean_scrub_stamp":"2024-09-16T15:48:25.420844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T23:24:05.461619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.39","version":"0'0","reported_seq":13,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-09-16T15:48:27.484014+0000","last_change":"2024-09-16T15:48:26.497742+0000","last_active":"2024-09-16T15:48:27.484014+0000","last_peered":"2024-09-16T15:48:27.484014+0000","last_clean":"2024-09-16T15:48:27.484014+0000","last_became_active":"2024-09-16T15:48:26.495135+0000","last_became_peered":"2024-09-16T15:48:26.495135+0000","last_unstale":"2024-09-16T15:48:27.484014+0000","last_undegraded":"2024-09-16T15:48:27.484014+0000","last_fullsized":"2024-09-16T15:48:27.484014+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:48:25.420844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:48:25.420844+0000","last_clean_scrub_stamp":"2024-09-16T15:48:25.420844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T18:56:16.511052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.38","version":"0'0","reported_seq":13,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-09-16T15:48:27.887213+0000","last_change":"2024-09-16T15:48:26.495526+0000","last_active":"2024-09-16T15:48:27.887213+0000","last_peered":"2024-09-16T15:48:27.887213+0000","last_clean":"2024-09-16T15:48:27.887213+0000","last_became_active":"2024-09-16T15:48:26.490886+0000","last_became_peered":"2024-09-16T15:48:26.490886+0000","last_unstale":"2024-09-16T15:48:27.887213+0000","last_undegraded":"2024-09-16T15:48:27.887213+0000","last_fullsized":"2024-09-16T15:48:27.887213+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:48:25.420844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:48:25.420844+0000","last_clean_scrub_stamp":"2024-09-16T15:48:25.420844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:40:39.688162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.37","version":"0'0","reported_seq":13,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-09-16T15:48:27.461918+0000","last_change":"2024-09-16T15:48:26.490579+0000","last_active":"2024-09-16T15:48:27.461918+0000","last_peered":"2024-09-16T15:48:27.461918+0000","last_clean":"2024-09-16T15:48:27.461918+0000","last_became_active":"2024-09-16T15:48:26.486770+0000","last_became_peered":"2024-09-16T15:48:26.486770+0000","last_unstale":"2024-09-16T15:48:27.461918+0000","last_undegraded":"2024-09-16T15:48:27.461918+0000","last_fullsized":"2024-09-16T15:48:27.461918+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:48:25.420844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:48:25.420844+0000","last_clean_scrub_stamp":"2024-09-16T15:48:25.420844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T23:29:27.871860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.36","version":"0'0","reported_seq":13,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-09-16T15:48:27.463194+0000","last_change":"2024-09-16T15:48:26.493063+0000","last_active":"2024-09-16T15:48:27.463194+0000","last_peered":"2024-09-16T15:48:27.463194+0000","last_clean":"2024-09-16T15:48:27.463194+0000","last_became_active":"2024-09-16T15:48:26.492024+0000","last_became_peered":"2024-09-16T15:48:26.492024+0000","last_unstale":"2024-09-16T15:48:27.463194+0000","last_undegraded":"2024-09-16T15:48:27.463194+0000","last_fullsized":"2024-09-16T15:48:27.463194+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:48:25.420844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:48:25.420844+0000","last_clean_scrub_stamp":"2024-09-16T15:48:25.420844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T18:40:16.437764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.35","version":"0'0","reported_seq":13,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-09-16T15:48:27.467862+0000","last_change":"2024-09-16T15:48:26.471375+0000","last_active":"2024-09-16T15:48:27.467862+0000","last_peered":"2024-09-16T15:48:27.467862+0000","last_clean":"2024-09-16T15:48:27.467862+0000","last_became_active":"2024-09-16T15:48:26.470597+0000","last_became_peered":"2024-09-16T15:48:26.470597+0000","last_unstale":"2024-09-16T15:48:27.467862+0000","last_undegraded":"2024-09-16T15:48:27.467862+0000","last_fullsized":"2024-09-16T15:48:27.467862+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:48:25.420844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:48:25.420844+0000","last_clean_scrub_stamp":"2024-09-16T15:48:25.420844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:00:26.169424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.34","version":"0'0","reported_seq":13,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-09-16T15:48:27.463848+0000","last_change":"2024-09-16T15:48:26.499834+0000","last_active":"2024-09-16T15:48:27.463848+0000","last_peered":"2024-09-16T15:48:27.463848+0000","last_clean":"2024-09-16T15:48:27.463848+0000","last_became_active":"2024-09-16T15:48:26.493251+0000","last_became_peered":"2024-09-16T15:48:26.493251+0000","last_unstale":"2024-09-16T15:48:27.463848+0000","last_undegraded":"2024-09-16T15:48:27.463848+0000","last_fullsized":"2024-09-16T15:48:27.463848+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:48:25.420844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:48:25.420844+0000","last_clean_scrub_stamp":"2024-09-16T15:48:25.420844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T18:48:08.106660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.33","version":"0'0","reported_seq":13,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-09-16T15:48:27.887776+0000","last_change":"2024-09-16T15:48:26.489082+0000","last_active":"2024-09-16T15:48:27.887776+0000","last_peered":"2024-09-16T15:48:27.887776+0000","last_clean":"2024-09-16T15:48:27.887776+0000","last_became_active":"2024-09-16T15:48:26.488251+0000","last_became_peered":"2024-09-16T15:48:26.488251+0000","last_unstale":"2024-09-16T15:48:27.887776+0000","last_undegraded":"2024-09-16T15:48:27.887776+0000","last_fullsized":"2024-09-16T15:48:27.887776+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:48:25.420844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:48:25.420844+0000","last_clean_scrub_stamp":"2024-09-16T15:48:25.420844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:07:44.883589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.32","version":"0'0","reported_seq":13,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-09-16T15:48:27.887539+0000","last_change":"2024-09-16T15:48:26.487177+0000","last_active":"2024-09-16T15:48:27.887539+0000","last_peered":"2024-09-16T15:48:27.887539+0000","last_clean":"2024-09-16T15:48:27.887539+0000","last_became_active":"2024-09-16T15:48:26.486331+0000","last_became_peered":"2024-09-16T15:48:26.486331+0000","last_unstale":"2024-09-16T15:48:27.887539+0000","last_undegraded":"2024-09-16T15:48:27.887539+0000","last_fullsized":"2024-09-16T15:48:27.887539+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:48:25.420844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:48:25.420844+0000","last_clean_scrub_stamp":"2024-09-16T15:48:25.420844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T22:10:31.295216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.31","version":"0'0","reported_seq":13,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-09-16T15:48:27.998911+0000","last_change":"2024-09-16T15:48:26.478731+0000","last_active":"2024-09-16T15:48:27.998911+0000","last_peered":"2024-09-16T15:48:27.998911+0000","last_clean":"2024-09-16T15:48:27.998911+0000","last_became_active":"2024-09-16T15:48:26.478366+0000","last_became_peered":"2024-09-16T15:48:26.478366+0000","last_unstale":"2024-09-16T15:48:27.998911+0000","last_undegraded":"2024-09-16T15:48:27.998911+0000","last_fullsized":"2024-09-16T15:48:27.998911+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:48:25.420844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:48:25.420844+0000","last_clean_scrub_stamp":"2024-09-16T15:48:25.420844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:21:49.177349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.30","version":"0'0","reported_seq":13,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-09-16T15:48:27.461362+0000","last_change":"2024-09-16T15:48:26.502708+0000","last_active":"2024-09-16T15:48:27.461362+0000","last_peered":"2024-09-16T15:48:27.461362+0000","last_clean":"2024-09-16T15:48:27.461362+0000","last_became_active":"2024-09-16T15:48:26.502397+0000","last_became_peered":"2024-09-16T15:48:26.502397+0000","last_unstale":"2024-09-16T15:48:27.461362+0000","last_undegraded":"2024-09-16T15:48:27.461362+0000","last_fullsized":"2024-09-16T15:48:27.461362+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:48:25.420844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:48:25.420844+0000","last_clean_scrub_stamp":"2024-09-16T15:48:25.420844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:40:23.507844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.2f","version":"0'0","reported_seq":13,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-09-16T15:48:27.468222+0000","last_change":"2024-09-16T15:48:26.474831+0000","last_active":"2024-09-16T15:48:27.468222+0000","last_peered":"2024-09-16T15:48:27.468222+0000","last_clean":"2024-09-16T15:48:27.468222+0000","last_became_active":"2024-09-16T15:48:26.473342+0000","last_became_peered":"2024-09-16T15:48:26.473342+0000","last_unstale":"2024-09-16T15:48:27.468222+0000","last_undegraded":"2024-09-16T15:48:27.468222+0000","last_fullsized":"2024-09-16T15:48:27.468222+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:48:25.420844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:48:25.420844+0000","last_clean_scrub_stamp":"2024-09-16T15:48:25.420844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T02:55:20.157975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.2e","version":"0'0","reported_seq":13,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-09-16T15:48:27.466862+0000","last_change":"2024-09-16T15:48:26.468006+0000","last_active":"2024-09-16T15:48:27.466862+0000","last_peered":"2024-09-16T15:48:27.466862+0000","last_clean":"2024-09-16T15:48:27.466862+0000","last_became_active":"2024-09-16T15:48:26.467250+0000","last_became_peered":"2024-09-16T15:48:26.467250+0000","last_unstale":"2024-09-16T15:48:27.466862+0000","last_undegraded":"2024-09-16T15:48:27.466862+0000","last_fullsized":"2024-09-16T15:48:27.466862+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:48:25.420844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:48:25.420844+0000","last_clean_scrub_stamp":"2024-09-16T15:48:25.420844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:29:34.405446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.2d","version":"0'0","reported_seq":13,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-09-16T15:48:27.887849+0000","last_change":"2024-09-16T15:48:26.489641+0000","last_active":"2024-09-16T15:48:27.887849+0000","last_peered":"2024-09-16T15:48:27.887849+0000","last_clean":"2024-09-16T15:48:27.887849+0000","last_became_active":"2024-09-16T15:48:26.488057+0000","last_became_peered":"2024-09-16T15:48:26.488057+0000","last_unstale":"2024-09-16T15:48:27.887849+0000","last_undegraded":"2024-09-16T15:48:27.887849+0000","last_fullsized":"2024-09-16T15:48:27.887849+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:48:25.420844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:48:25.420844+0000","last_clean_scrub_stamp":"2024-09-16T15:48:25.420844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:19:58.858823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.2c","version":"0'0","reported_seq":13,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-09-16T15:48:27.467701+0000","last_change":"2024-09-16T15:48:26.465853+0000","last_active":"2024-09-16T15:48:27.467701+0000","last_peered":"2024-09-16T15:48:27.467701+0000","last_clean":"2024-09-16T15:48:27.467701+0000","last_became_active":"2024-09-16T15:48:26.465105+0000","last_became_peered":"2024-09-16T15:48:26.465105+0000","last_unstale":"2024-09-16T15:48:27.467701+0000","last_undegraded":"2024-09-16T15:48:27.467701+0000","last_fullsized":"2024-09-16T15:48:27.467701+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:48:25.420844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:48:25.420844+0000","last_clean_scrub_stamp":"2024-09-16T15:48:25.420844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T23:20:00.312557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.2b","version":"0'0","reported_seq":13,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-09-16T15:48:27.466957+0000","last_change":"2024-09-16T15:48:26.486192+0000","last_active":"2024-09-16T15:48:27.466957+0000","last_peered":"2024-09-16T15:48:27.466957+0000","last_clean":"2024-09-16T15:48:27.466957+0000","last_became_active":"2024-09-16T15:48:26.485417+0000","last_became_peered":"2024-09-16T15:48:26.485417+0000","last_unstale":"2024-09-16T15:48:27.466957+0000","last_undegraded":"2024-09-16T15:48:27.466957+0000","last_fullsized":"2024-09-16T15:48:27.466957+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:48:25.420844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:48:25.420844+0000","last_clean_scrub_stamp":"2024-09-16T15:48:25.420844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T02:31:29.163804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.2a","version":"0'0","reported_seq":13,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-09-16T15:48:27.997142+0000","last_change":"2024-09-16T15:48:26.459479+0000","last_active":"2024-09-16T15:48:27.997142+0000","last_peered":"2024-09-16T15:48:27.997142+0000","last_clean":"2024-09-16T15:48:27.997142+0000","last_became_active":"2024-09-16T15:48:26.458168+0000","last_became_peered":"2024-09-16T15:48:26.458168+0000","last_unstale":"2024-09-16T15:48:27.997142+0000","last_undegraded":"2024-09-16T15:48:27.997142+0000","last_fullsized":"2024-09-16T15:48:27.997142+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:48:25.420844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:48:25.420844+0000","last_clean_scrub_stamp":"2024-09-16T15:48:25.420844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:44:15.690048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.29","version":"0'0","reported_seq":13,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-09-16T15:48:27.465052+0000","last_change":"2024-09-16T15:48:26.467978+0000","last_active":"2024-09-16T15:48:27.465052+0000","last_peered":"2024-09-16T15:48:27.465052+0000","last_clean":"2024-09-16T15:48:27.465052+0000","last_became_active":"2024-09-16T15:48:26.467166+0000","last_became_peered":"2024-09-16T15:48:26.467166+0000","last_unstale":"2024-09-16T15:48:27.465052+0000","last_undegraded":"2024-09-16T15:48:27.465052+0000","last_fullsized":"2024-09-16T15:48:27.465052+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:48:25.420844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:48:25.420844+0000","last_clean_scrub_stamp":"2024-09-16T15:48:25.420844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:18:22.614416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.28","version":"0'0","reported_seq":13,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-09-16T15:48:27.997749+0000","last_change":"2024-09-16T15:48:26.474394+0000","last_active":"2024-09-16T15:48:27.997749+0000","last_peered":"2024-09-16T15:48:27.997749+0000","last_clean":"2024-09-16T15:48:27.997749+0000","last_became_active":"2024-09-16T15:48:26.474072+0000","last_became_peered":"2024-09-16T15:48:26.474072+0000","last_unstale":"2024-09-16T15:48:27.997749+0000","last_undegraded":"2024-09-16T15:48:27.997749+0000","last_fullsized":"2024-09-16T15:48:27.997749+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:48:25.420844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:48:25.420844+0000","last_clean_scrub_stamp":"2024-09-16T15:48:25.420844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:02:13.807139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.27","version":"0'0","reported_seq":13,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-09-16T15:48:27.468519+0000","last_change":"2024-09-16T15:48:26.474043+0000","last_active":"2024-09-16T15:48:27.468519+0000","last_peered":"2024-09-16T15:48:27.468519+0000","last_clean":"2024-09-16T15:48:27.468519+0000","last_became_active":"2024-09-16T15:48:26.472415+0000","last_became_peered":"2024-09-16T15:48:26.472415+0000","last_unstale":"2024-09-16T15:48:27.468519+0000","last_undegraded":"2024-09-16T15:48:27.468519+0000","last_fullsized":"2024-09-16T15:48:27.468519+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:48:25.420844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:48:25.420844+0000","last_clean_scrub_stamp":"2024-09-16T15:48:25.420844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T18:03:51.621047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.26","version":"0'0","reported_seq":13,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-09-16T15:48:27.463260+0000","last_change":"2024-09-16T15:48:26.492754+0000","last_active":"2024-09-16T15:48:27.463260+0000","last_peered":"2024-09-16T15:48:27.463260+0000","last_clean":"2024-09-16T15:48:27.463260+0000","last_became_active":"2024-09-16T15:48:26.487615+0000","last_became_peered":"2024-09-16T15:48:26.487615+0000","last_unstale":"2024-09-16T15:48:27.463260+0000","last_undegraded":"2024-09-16T15:48:27.463260+0000","last_fullsized":"2024-09-16T15:48:27.463260+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:48:25.420844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:48:25.420844+0000","last_clean_scrub_stamp":"2024-09-16T15:48:25.420844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:00:04.834017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.25","version":"0'0","reported_seq":13,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-09-16T15:48:27.999811+0000","last_change":"2024-09-16T15:48:26.490676+0000","last_active":"2024-09-16T15:48:27.999811+0000","last_peered":"2024-09-16T15:48:27.999811+0000","last_clean":"2024-09-16T15:48:27.999811+0000","last_became_active":"2024-09-16T15:48:26.490387+0000","last_became_peered":"2024-09-16T15:48:26.490387+0000","last_unstale":"2024-09-16T15:48:27.999811+0000","last_undegraded":"2024-09-16T15:48:27.999811+0000","last_fullsized":"2024-09-16T15:48:27.999811+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:48:25.420844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:48:25.420844+0000","last_clean_scrub_stamp":"2024-09-16T15:48:25.420844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:14:14.378687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.24","version":"0'0","reported_seq":13,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-09-16T15:48:27.997318+0000","last_change":"2024-09-16T15:48:26.496746+0000","last_active":"2024-09-16T15:48:27.997318+0000","last_peered":"2024-09-16T15:48:27.997318+0000","last_clean":"2024-09-16T15:48:27.997318+0000","last_became_active":"2024-09-16T15:48:26.496472+0000","last_became_peered":"2024-09-16T15:48:26.496472+0000","last_unstale":"2024-09-16T15:48:27.997318+0000","last_undegraded":"2024-09-16T15:48:27.997318+0000","last_fullsized":"2024-09-16T15:48:27.997318+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:48:25.420844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:48:25.420844+0000","last_clean_scrub_stamp":"2024-09-16T15:48:25.420844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:58:33.486166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.23","version":"0'0","reported_seq":13,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-09-16T15:48:27.465956+0000","last_change":"2024-09-16T15:48:26.488919+0000","last_active":"2024-09-16T15:48:27.465956+0000","last_peered":"2024-09-16T15:48:27.465956+0000","last_clean":"2024-09-16T15:48:27.465956+0000","last_became_active":"2024-09-16T15:48:26.487265+0000","last_became_peered":"2024-09-16T15:48:26.487265+0000","last_unstale":"2024-09-16T15:48:27.465956+0000","last_undegraded":"2024-09-16T15:48:27.465956+0000","last_fullsized":"2024-09-16T15:48:27.465956+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:48:25.420844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:48:25.420844+0000","last_clean_scrub_stamp":"2024-09-16T15:48:25.420844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:17:26.625596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.22","version":"0'0","reported_seq":13,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-09-16T15:48:27.483405+0000","last_change":"2024-09-16T15:48:26.495110+0000","last_active":"2024-09-16T15:48:27.483405+0000","last_peered":"2024-09-16T15:48:27.483405+0000","last_clean":"2024-09-16T15:48:27.483405+0000","last_became_active":"2024-09-16T15:48:26.494036+0000","last_became_peered":"2024-09-16T15:48:26.494036+0000","last_unstale":"2024-09-16T15:48:27.483405+0000","last_undegraded":"2024-09-16T15:48:27.483405+0000","last_fullsized":"2024-09-16T15:48:27.483405+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:48:25.420844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:48:25.420844+0000","last_clean_scrub_stamp":"2024-09-16T15:48:25.420844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:14:21.492784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.21","version":"0'0","reported_seq":13,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-09-16T15:48:27.461691+0000","last_change":"2024-09-16T15:48:26.500768+0000","last_active":"2024-09-16T15:48:27.461691+0000","last_peered":"2024-09-16T15:48:27.461691+0000","last_clean":"2024-09-16T15:48:27.461691+0000","last_became_active":"2024-09-16T15:48:26.500434+0000","last_became_peered":"2024-09-16T15:48:26.500434+0000","last_unstale":"2024-09-16T15:48:27.461691+0000","last_undegraded":"2024-09-16T15:48:27.461691+0000","last_fullsized":"2024-09-16T15:48:27.461691+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:48:25.420844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:48:25.420844+0000","last_clean_scrub_stamp":"2024-09-16T15:48:25.420844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:14.364288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.20","version":"0'0","reported_seq":13,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-09-16T15:48:27.998891+0000","last_change":"2024-09-16T15:48:26.474517+0000","last_active":"2024-09-16T15:48:27.998891+0000","last_peered":"2024-09-16T15:48:27.998891+0000","last_clean":"2024-09-16T15:48:27.998891+0000","last_became_active":"2024-09-16T15:48:26.473941+0000","last_became_peered":"2024-09-16T15:48:26.473941+0000","last_unstale":"2024-09-16T15:48:27.998891+0000","last_undegraded":"2024-09-16T15:48:27.998891+0000","last_fullsized":"2024-09-16T15:48:27.998891+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:48:25.420844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:48:25.420844+0000","last_clean_scrub_stamp":"2024-09-16T15:48:25.420844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:52:25.564241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.1e","version":"0'0","reported_seq":13,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-09-16T15:48:27.999924+0000","last_change":"2024-09-16T15:48:26.475804+0000","last_active":"2024-09-16T15:48:27.999924+0000","last_peered":"2024-09-16T15:48:27.999924+0000","last_clean":"2024-09-16T15:48:27.999924+0000","last_became_active":"2024-09-16T15:48:26.475322+0000","last_became_peered":"2024-09-16T15:48:26.475322+0000","last_unstale":"2024-09-16T15:48:27.999924+0000","last_undegraded":"2024-09-16T15:48:27.999924+0000","last_fullsized":"2024-09-16T15:48:27.999924+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:48:25.420844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:48:25.420844+0000","last_clean_scrub_stamp":"2024-09-16T15:48:25.420844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T23:21:43.437807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.1f","version":"102'4","reported_seq":19,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-09-16T15:48:27.483202+0000","last_change":"2024-09-16T15:48:25.465785+0000","last_active":"2024-09-16T15:48:27.483202+0000","last_peered":"2024-09-16T15:48:27.483202+0000","last_clean":"2024-09-16T15:48:27.483202+0000","last_became_active":"2024-09-16T15:48:25.464855+0000","last_became_peered":"2024-09-16T15:48:25.464855+0000","last_unstale":"2024-09-16T15:48:27.483202+0000","last_undegraded":"2024-09-16T15:48:27.483202+0000","last_fullsized":"2024-09-16T15:48:27.483202+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:48:24.407732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:48:24.407732+0000","last_clean_scrub_stamp":"2024-09-16T15:48:24.407732+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T23:21:42.424695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.1f","version":"0'0","reported_seq":13,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-09-16T15:48:27.467691+0000","last_change":"2024-09-16T15:48:26.470107+0000","last_active":"2024-09-16T15:48:27.467691+0000","last_peered":"2024-09-16T15:48:27.467691+0000","last_clean":"2024-09-16T15:48:27.467691+0000","last_became_active":"2024-09-16T15:48:26.469290+0000","last_became_peered":"2024-09-16T15:48:26.469290+0000","last_unstale":"2024-09-16T15:48:27.467691+0000","last_undegraded":"2024-09-16T15:48:27.467691+0000","last_fullsized":"2024-09-16T15:48:27.467691+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:48:25.420844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:48:25.420844+0000","last_clean_scrub_stamp":"2024-09-16T15:48:25.420844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:29:35.762056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.1e","version":"102'1","reported_seq":16,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-09-16T15:48:27.887676+0000","last_change":"2024-09-16T15:48:25.469533+0000","last_active":"2024-09-16T15:48:27.887676+0000","last_peered":"2024-09-16T15:48:27.887676+0000","last_clean":"2024-09-16T15:48:27.887676+0000","last_became_active":"2024-09-16T15:48:25.468882+0000","last_became_peered":"2024-09-16T15:48:25.468882+0000","last_unstale":"2024-09-16T15:48:27.887676+0000","last_undegraded":"2024-09-16T15:48:27.887676+0000","last_fullsized":"2024-09-16T15:48:27.887676+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:48:24.407732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:48:24.407732+0000","last_clean_scrub_stamp":"2024-09-16T15:48:24.407732+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T03:32:30.135599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.1c","version":"0'0","reported_seq":13,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-09-16T15:48:27.999784+0000","last_change":"2024-09-16T15:48:26.474260+0000","last_active":"2024-09-16T15:48:27.999784+0000","last_peered":"2024-09-16T15:48:27.999784+0000","last_clean":"2024-09-16T15:48:27.999784+0000","last_became_active":"2024-09-16T15:48:26.473902+0000","last_became_peered":"2024-09-16T15:48:26.473902+0000","last_unstale":"2024-09-16T15:48:27.999784+0000","last_undegraded":"2024-09-16T15:48:27.999784+0000","last_fullsized":"2024-09-16T15:48:27.999784+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:48:25.420844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:48:25.420844+0000","last_clean_scrub_stamp":"2024-09-16T15:48:25.420844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T03:15:19.215261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.1d","version":"102'2","reported_seq":17,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-09-16T15:48:27.495904+0000","last_change":"2024-09-16T15:48:25.449694+0000","last_active":"2024-09-16T15:48:27.495904+0000","last_peered":"2024-09-16T15:48:27.495904+0000","last_clean":"2024-09-16T15:48:27.495904+0000","last_became_active":"2024-09-16T15:48:25.449121+0000","last_became_peered":"2024-09-16T15:48:25.449121+0000","last_unstale":"2024-09-16T15:48:27.495904+0000","last_undegraded":"2024-09-16T15:48:27.495904+0000","last_fullsized":"2024-09-16T15:48:27.495904+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:48:24.407732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:48:24.407732+0000","last_clean_scrub_stamp":"2024-09-16T15:48:24.407732+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:19:14.112868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1d","version":"0'0","reported_seq":13,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-09-16T15:48:27.888159+0000","last_change":"2024-09-16T15:48:26.490466+0000","last_active":"2024-09-16T15:48:27.888159+0000","last_peered":"2024-09-16T15:48:27.888159+0000","last_clean":"2024-09-16T15:48:27.888159+0000","last_became_active":"2024-09-16T15:48:26.488882+0000","last_became_peered":"2024-09-16T15:48:26.488882+0000","last_unstale":"2024-09-16T15:48:27.888159+0000","last_undegraded":"2024-09-16T15:48:27.888159+0000","last_fullsized":"2024-09-16T15:48:27.888159+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:48:25.420844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:48:25.420844+0000","last_clean_scrub_stamp":"2024-09-16T15:48:25.420844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T23:18:21.147916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.1c","version":"102'2","reported_seq":17,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-09-16T15:48:27.495679+0000","last_change":"2024-09-16T15:48:25.446615+0000","last_active":"2024-09-16T15:48:27.495679+0000","last_peered":"2024-09-16T15:48:27.495679+0000","last_clean":"2024-09-16T15:48:27.495679+0000","last_became_active":"2024-09-16T15:48:25.446259+0000","last_became_peered":"2024-09-16T15:48:25.446259+0000","last_unstale":"2024-09-16T15:48:27.495679+0000","last_undegraded":"2024-09-16T15:48:27.495679+0000","last_fullsized":"2024-09-16T15:48:27.495679+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:48:24.407732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:48:24.407732+0000","last_clean_scrub_stamp":"2024-09-16T15:48:24.407732+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:35:07.163165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1a","version":"0'0","reported_seq":13,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-09-16T15:48:27.461696+0000","last_change":"2024-09-16T15:48:26.487828+0000","last_active":"2024-09-16T15:48:27.461696+0000","last_peered":"2024-09-16T15:48:27.461696+0000","last_clean":"2024-09-16T15:48:27.461696+0000","last_became_active":"2024-09-16T15:48:26.486834+0000","last_became_peered":"2024-09-16T15:48:26.486834+0000","last_unstale":"2024-09-16T15:48:27.461696+0000","last_undegraded":"2024-09-16T15:48:27.461696+0000","last_fullsized":"2024-09-16T15:48:27.461696+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:48:25.420844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:48:25.420844+0000","last_clean_scrub_stamp":"2024-09-16T15:48:25.420844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T01:10:19.234916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.1b","version":"102'1","reported_seq":16,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-09-16T15:48:27.482907+0000","last_change":"2024-09-16T15:48:25.458440+0000","last_active":"2024-09-16T15:48:27.482907+0000","last_peered":"2024-09-16T15:48:27.482907+0000","last_clean":"2024-09-16T15:48:27.482907+0000","last_became_active":"2024-09-16T15:48:25.457689+0000","last_became_peered":"2024-09-16T15:48:25.457689+0000","last_unstale":"2024-09-16T15:48:27.482907+0000","last_undegraded":"2024-09-16T15:48:27.482907+0000","last_fullsized":"2024-09-16T15:48:27.482907+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:48:24.407732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:48:24.407732+0000","last_clean_scrub_stamp":"2024-09-16T15:48:24.407732+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T03:15:18.202149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.1b","version":"0'0","reported_seq":13,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-09-16T15:48:27.465971+0000","last_change":"2024-09-16T15:48:26.489699+0000","last_active":"2024-09-16T15:48:27.465971+0000","last_peered":"2024-09-16T15:48:27.465971+0000","last_clean":"2024-09-16T15:48:27.465971+0000","last_became_active":"2024-09-16T15:48:26.488220+0000","last_became_peered":"2024-09-16T15:48:26.488220+0000","last_unstale":"2024-09-16T15:48:27.465971+0000","last_undegraded":"2024-09-16T15:48:27.465971+0000","last_fullsized":"2024-09-16T15:48:27.465971+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:48:25.420844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:48:25.420844+0000","last_clean_scrub_stamp":"2024-09-16T15:48:25.420844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:47:19.716595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.1a","version":"102'1","reported_seq":16,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-09-16T15:48:27.997380+0000","last_change":"2024-09-16T15:48:25.445022+0000","last_active":"2024-09-16T15:48:27.997380+0000","last_peered":"2024-09-16T15:48:27.997380+0000","last_clean":"2024-09-16T15:48:27.997380+0000","last_became_active":"2024-09-16T15:48:25.444656+0000","last_became_peered":"2024-09-16T15:48:25.444656+0000","last_unstale":"2024-09-16T15:48:27.997380+0000","last_undegraded":"2024-09-16T15:48:27.997380+0000","last_fullsized":"2024-09-16T15:48:27.997380+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:48:24.407732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:48:24.407732+0000","last_clean_scrub_stamp":"2024-09-16T15:48:24.407732+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T22:19:51.476505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.18","version":"0'0","reported_seq":13,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-09-16T15:48:27.463522+0000","last_change":"2024-09-16T15:48:26.488718+0000","last_active":"2024-09-16T15:48:27.463522+0000","last_peered":"2024-09-16T15:48:27.463522+0000","last_clean":"2024-09-16T15:48:27.463522+0000","last_became_active":"2024-09-16T15:48:26.487539+0000","last_became_peered":"2024-09-16T15:48:26.487539+0000","last_unstale":"2024-09-16T15:48:27.463522+0000","last_undegraded":"2024-09-16T15:48:27.463522+0000","last_fullsized":"2024-09-16T15:48:27.463522+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:48:25.420844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:48:25.420844+0000","last_clean_scrub_stamp":"2024-09-16T15:48:25.420844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T22:48:59.876278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.19","version":"0'0","reported_seq":15,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-09-16T15:48:27.464844+0000","last_change":"2024-09-16T15:48:25.446394+0000","last_active":"2024-09-16T15:48:27.464844+0000","last_peered":"2024-09-16T15:48:27.464844+0000","last_clean":"2024-09-16T15:48:27.464844+0000","last_became_active":"2024-09-16T15:48:25.445446+0000","last_became_peered":"2024-09-16T15:48:25.445446+0000","last_unstale":"2024-09-16T15:48:27.464844+0000","last_undegraded":"2024-09-16T15:48:27.464844+0000","last_fullsized":"2024-09-16T15:48:27.464844+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:48:24.407732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:48:24.407732+0000","last_clean_scrub_stamp":"2024-09-16T15:48:24.407732+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T22:19:51.476505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.19","version":"0'0","reported_seq":13,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-09-16T15:48:27.483026+0000","last_change":"2024-09-16T15:48:26.498541+0000","last_active":"2024-09-16T15:48:27.483026+0000","last_peered":"2024-09-16T15:48:27.483026+0000","last_clean":"2024-09-16T15:48:27.483026+0000","last_became_active":"2024-09-16T15:48:26.495967+0000","last_became_peered":"2024-09-16T15:48:26.495967+0000","last_unstale":"2024-09-16T15:48:27.483026+0000","last_undegraded":"2024-09-16T15:48:27.483026+0000","last_fullsized":"2024-09-16T15:48:27.483026+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:48:25.420844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:48:25.420844+0000","last_clean_scrub_stamp":"2024-09-16T15:48:25.420844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:19:49.639111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.18","version":"0'0","reported_seq":15,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-09-16T15:48:27.461686+0000","last_change":"2024-09-16T15:48:25.461569+0000","last_active":"2024-09-16T15:48:27.461686+0000","last_peered":"2024-09-16T15:48:27.461686+0000","last_clean":"2024-09-16T15:48:27.461686+0000","last_became_active":"2024-09-16T15:48:25.461207+0000","last_became_peered":"2024-09-16T15:48:25.461207+0000","last_unstale":"2024-09-16T15:48:27.461686+0000","last_undegraded":"2024-09-16T15:48:27.461686+0000","last_fullsized":"2024-09-16T15:48:27.461686+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:48:24.407732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:48:24.407732+0000","last_clean_scrub_stamp":"2024-09-16T15:48:24.407732+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:27:25.935391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.16","version":"0'0","reported_seq":13,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-09-16T15:48:27.484145+0000","last_change":"2024-09-16T15:48:26.482968+0000","last_active":"2024-09-16T15:48:27.484145+0000","last_peered":"2024-09-16T15:48:27.484145+0000","last_clean":"2024-09-16T15:48:27.484145+0000","last_became_active":"2024-09-16T15:48:26.482005+0000","last_became_peered":"2024-09-16T15:48:26.482005+0000","last_unstale":"2024-09-16T15:48:27.484145+0000","last_undegraded":"2024-09-16T15:48:27.484145+0000","last_fullsized":"2024-09-16T15:48:27.484145+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:48:25.420844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:48:25.420844+0000","last_clean_scrub_stamp":"2024-09-16T15:48:25.420844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:34:11.300050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.17","version":"0'0","reported_seq":15,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-09-16T15:48:27.495999+0000","last_change":"2024-09-16T15:48:25.452524+0000","last_active":"2024-09-16T15:48:27.495999+0000","last_peered":"2024-09-16T15:48:27.495999+0000","last_clean":"2024-09-16T15:48:27.495999+0000","last_became_active":"2024-09-16T15:48:25.452224+0000","last_became_peered":"2024-09-16T15:48:25.452224+0000","last_unstale":"2024-09-16T15:48:27.495999+0000","last_undegraded":"2024-09-16T15:48:27.495999+0000","last_fullsized":"2024-09-16T15:48:27.495999+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:48:24.407732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:48:24.407732+0000","last_clean_scrub_stamp":"2024-09-16T15:48:24.407732+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:09:30.240163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.17","version":"0'0","reported_seq":13,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-09-16T15:48:27.467970+0000","last_change":"2024-09-16T15:48:26.498755+0000","last_active":"2024-09-16T15:48:27.467970+0000","last_peered":"2024-09-16T15:48:27.467970+0000","last_clean":"2024-09-16T15:48:27.467970+0000","last_became_active":"2024-09-16T15:48:26.497874+0000","last_became_peered":"2024-09-16T15:48:26.497874+0000","last_unstale":"2024-09-16T15:48:27.467970+0000","last_undegraded":"2024-09-16T15:48:27.467970+0000","last_fullsized":"2024-09-16T15:48:27.467970+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:48:25.420844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:48:25.420844+0000","last_clean_scrub_stamp":"2024-09-16T15:48:25.420844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T18:36:32.286233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.16","version":"0'0","reported_seq":15,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-09-16T15:48:27.996678+0000","last_change":"2024-09-16T15:48:25.444804+0000","last_active":"2024-09-16T15:48:27.996678+0000","last_peered":"2024-09-16T15:48:27.996678+0000","last_clean":"2024-09-16T15:48:27.996678+0000","last_became_active":"2024-09-16T15:48:25.444511+0000","last_became_peered":"2024-09-16T15:48:25.444511+0000","last_unstale":"2024-09-16T15:48:27.996678+0000","last_undegraded":"2024-09-16T15:48:27.996678+0000","last_fullsized":"2024-09-16T15:48:27.996678+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:48:24.407732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:48:24.407732+0000","last_clean_scrub_stamp":"2024-09-16T15:48:24.407732+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T22:15:28.730113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.14","version":"0'0","reported_seq":13,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-09-16T15:48:27.997616+0000","last_change":"2024-09-16T15:48:26.469903+0000","last_active":"2024-09-16T15:48:27.997616+0000","last_peered":"2024-09-16T15:48:27.997616+0000","last_clean":"2024-09-16T15:48:27.997616+0000","last_became_active":"2024-09-16T15:48:26.469227+0000","last_became_peered":"2024-09-16T15:48:26.469227+0000","last_unstale":"2024-09-16T15:48:27.997616+0000","last_undegraded":"2024-09-16T15:48:27.997616+0000","last_fullsized":"2024-09-16T15:48:27.997616+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:48:25.420844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:48:25.420844+0000","last_clean_scrub_stamp":"2024-09-16T15:48:25.420844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T23:19:49.157256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.15","version":"102'3","reported_seq":18,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-09-16T15:48:27.462567+0000","last_change":"2024-09-16T15:48:25.457438+0000","last_active":"2024-09-16T15:48:27.462567+0000","last_peered":"2024-09-16T15:48:27.462567+0000","last_clean":"2024-09-16T15:48:27.462567+0000","last_became_active":"2024-09-16T15:48:25.456752+0000","last_became_peered":"2024-09-16T15:48:25.456752+0000","last_unstale":"2024-09-16T15:48:27.462567+0000","last_undegraded":"2024-09-16T15:48:27.462567+0000","last_fullsized":"2024-09-16T15:48:27.462567+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:48:24.407732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:48:24.407732+0000","last_clean_scrub_stamp":"2024-09-16T15:48:24.407732+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:23:52.067881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.15","version":"0'0","reported_seq":13,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-09-16T15:48:27.483758+0000","last_change":"2024-09-16T15:48:26.506294+0000","last_active":"2024-09-16T15:48:27.483758+0000","last_peered":"2024-09-16T15:48:27.483758+0000","last_clean":"2024-09-16T15:48:27.483758+0000","last_became_active":"2024-09-16T15:48:26.505894+0000","last_became_peered":"2024-09-16T15:48:26.505894+0000","last_unstale":"2024-09-16T15:48:27.483758+0000","last_undegraded":"2024-09-16T15:48:27.483758+0000","last_fullsized":"2024-09-16T15:48:27.483758+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:48:25.420844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:48:25.420844+0000","last_clean_scrub_stamp":"2024-09-16T15:48:25.420844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:12:21.419337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.14","version":"102'6","reported_seq":21,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-09-16T15:48:27.462843+0000","last_change":"2024-09-16T15:48:25.461989+0000","last_active":"2024-09-16T15:48:27.462843+0000","last_peered":"2024-09-16T15:48:27.462843+0000","last_clean":"2024-09-16T15:48:27.462843+0000","last_became_active":"2024-09-16T15:48:25.461384+0000","last_became_peered":"2024-09-16T15:48:25.461384+0000","last_unstale":"2024-09-16T15:48:27.462843+0000","last_undegraded":"2024-09-16T15:48:27.462843+0000","last_fullsized":"2024-09-16T15:48:27.462843+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:48:24.407732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:48:24.407732+0000","last_clean_scrub_stamp":"2024-09-16T15:48:24.407732+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:19:57.845711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.12","version":"0'0","reported_seq":13,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-09-16T15:48:27.997394+0000","last_change":"2024-09-16T15:48:26.508270+0000","last_active":"2024-09-16T15:48:27.997394+0000","last_peered":"2024-09-16T15:48:27.997394+0000","last_clean":"2024-09-16T15:48:27.997394+0000","last_became_active":"2024-09-16T15:48:26.507897+0000","last_became_peered":"2024-09-16T15:48:26.507897+0000","last_unstale":"2024-09-16T15:48:27.997394+0000","last_undegraded":"2024-09-16T15:48:27.997394+0000","last_fullsized":"2024-09-16T15:48:27.997394+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:48:25.420844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:48:25.420844+0000","last_clean_scrub_stamp":"2024-09-16T15:48:25.420844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T23:32:17.736399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.13","version":"102'2","reported_seq":17,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-09-16T15:48:27.999066+0000","last_change":"2024-09-16T15:48:25.450490+0000","last_active":"2024-09-16T15:48:27.999066+0000","last_peered":"2024-09-16T15:48:27.999066+0000","last_clean":"2024-09-16T15:48:27.999066+0000","last_became_active":"2024-09-16T15:48:25.450170+0000","last_became_peered":"2024-09-16T15:48:25.450170+0000","last_unstale":"2024-09-16T15:48:27.999066+0000","last_undegraded":"2024-09-16T15:48:27.999066+0000","last_fullsized":"2024-09-16T15:48:27.999066+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:48:24.407732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:48:24.407732+0000","last_clean_scrub_stamp":"2024-09-16T15:48:24.407732+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T00:59:17.551961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.13","version":"0'0","reported_seq":13,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-09-16T15:48:27.997045+0000","last_change":"2024-09-16T15:48:26.468069+0000","last_active":"2024-09-16T15:48:27.997045+0000","last_peered":"2024-09-16T15:48:27.997045+0000","last_clean":"2024-09-16T15:48:27.997045+0000","last_became_active":"2024-09-16T15:48:26.466980+0000","last_became_peered":"2024-09-16T15:48:26.466980+0000","last_unstale":"2024-09-16T15:48:27.997045+0000","last_undegraded":"2024-09-16T15:48:27.997045+0000","last_fullsized":"2024-09-16T15:48:27.997045+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:48:25.420844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:48:25.420844+0000","last_clean_scrub_stamp":"2024-09-16T15:48:25.420844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T22:01:08.106237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.12","version":"0'0","reported_seq":15,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-09-16T15:48:27.466423+0000","last_change":"2024-09-16T15:48:25.459083+0000","last_active":"2024-09-16T15:48:27.466423+0000","last_peered":"2024-09-16T15:48:27.466423+0000","last_clean":"2024-09-16T15:48:27.466423+0000","last_became_active":"2024-09-16T15:48:25.457843+0000","last_became_peered":"2024-09-16T15:48:25.457843+0000","last_unstale":"2024-09-16T15:48:27.466423+0000","last_undegraded":"2024-09-16T15:48:27.466423+0000","last_fullsized":"2024-09-16T15:48:27.466423+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:48:24.407732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:48:24.407732+0000","last_clean_scrub_stamp":"2024-09-16T15:48:24.407732+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:00:41.831554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.11","version":"0'0","reported_seq":13,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-09-16T15:48:27.996755+0000","last_change":"2024-09-16T15:48:26.467575+0000","last_active":"2024-09-16T15:48:27.996755+0000","last_peered":"2024-09-16T15:48:27.996755+0000","last_clean":"2024-09-16T15:48:27.996755+0000","last_became_active":"2024-09-16T15:48:26.465430+0000","last_became_peered":"2024-09-16T15:48:26.465430+0000","last_unstale":"2024-09-16T15:48:27.996755+0000","last_undegraded":"2024-09-16T15:48:27.996755+0000","last_fullsized":"2024-09-16T15:48:27.996755+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:48:25.420844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:48:25.420844+0000","last_clean_scrub_stamp":"2024-09-16T15:48:25.420844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T22:09:48.246143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.10","version":"102'8","reported_seq":23,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-09-16T15:48:27.495629+0000","last_change":"2024-09-16T15:48:25.462164+0000","last_active":"2024-09-16T15:48:27.495629+0000","last_peered":"2024-09-16T15:48:27.495629+0000","last_clean":"2024-09-16T15:48:27.495629+0000","last_became_active":"2024-09-16T15:48:25.461886+0000","last_became_peered":"2024-09-16T15:48:25.461886+0000","last_unstale":"2024-09-16T15:48:27.495629+0000","last_undegraded":"2024-09-16T15:48:27.495629+0000","last_fullsized":"2024-09-16T15:48:27.495629+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:48:24.407732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:48:24.407732+0000","last_clean_scrub_stamp":"2024-09-16T15:48:24.407732+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:34:13.844138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.e","version":"0'0","reported_seq":13,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-09-16T15:48:27.496063+0000","last_change":"2024-09-16T15:48:26.501077+0000","last_active":"2024-09-16T15:48:27.496063+0000","last_peered":"2024-09-16T15:48:27.496063+0000","last_clean":"2024-09-16T15:48:27.496063+0000","last_became_active":"2024-09-16T15:48:26.500794+0000","last_became_peered":"2024-09-16T15:48:26.500794+0000","last_unstale":"2024-09-16T15:48:27.496063+0000","last_undegraded":"2024-09-16T15:48:27.496063+0000","last_fullsized":"2024-09-16T15:48:27.496063+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:48:25.420844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:48:25.420844+0000","last_clean_scrub_stamp":"2024-09-16T15:48:25.420844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:32:28.584284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.f","version":"102'1","reported_seq":16,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-09-16T15:48:27.462272+0000","last_change":"2024-09-16T15:48:25.461746+0000","last_active":"2024-09-16T15:48:27.462272+0000","last_peered":"2024-09-16T15:48:27.462272+0000","last_clean":"2024-09-16T15:48:27.462272+0000","last_became_active":"2024-09-16T15:48:25.461035+0000","last_became_peered":"2024-09-16T15:48:25.461035+0000","last_unstale":"2024-09-16T15:48:27.462272+0000","last_undegraded":"2024-09-16T15:48:27.462272+0000","last_fullsized":"2024-09-16T15:48:27.462272+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:48:24.407732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:48:24.407732+0000","last_clean_scrub_stamp":"2024-09-16T15:48:24.407732+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:07:43.870477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.f","version":"0'0","reported_seq":13,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-09-16T15:48:27.999658+0000","last_change":"2024-09-16T15:48:26.493307+0000","last_active":"2024-09-16T15:48:27.999658+0000","last_peered":"2024-09-16T15:48:27.999658+0000","last_clean":"2024-09-16T15:48:27.999658+0000","last_became_active":"2024-09-16T15:48:26.493028+0000","last_became_peered":"2024-09-16T15:48:26.493028+0000","last_unstale":"2024-09-16T15:48:27.999658+0000","last_undegraded":"2024-09-16T15:48:27.999658+0000","last_fullsized":"2024-09-16T15:48:27.999658+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:48:25.420844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:48:25.420844+0000","last_clean_scrub_stamp":"2024-09-16T15:48:25.420844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T22:46:45.269484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.e","version":"102'2","reported_seq":17,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-09-16T15:48:27.496162+0000","last_change":"2024-09-16T15:48:25.449185+0000","last_active":"2024-09-16T15:48:27.496162+0000","last_peered":"2024-09-16T15:48:27.496162+0000","last_clean":"2024-09-16T15:48:27.496162+0000","last_became_active":"2024-09-16T15:48:25.448865+0000","last_became_peered":"2024-09-16T15:48:25.448865+0000","last_unstale":"2024-09-16T15:48:27.496162+0000","last_undegraded":"2024-09-16T15:48:27.496162+0000","last_fullsized":"2024-09-16T15:48:27.496162+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:48:24.407732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:48:24.407732+0000","last_clean_scrub_stamp":"2024-09-16T15:48:24.407732+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:15:53.516955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.c","version":"0'0","reported_seq":13,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-09-16T15:48:27.887691+0000","last_change":"2024-09-16T15:48:26.502464+0000","last_active":"2024-09-16T15:48:27.887691+0000","last_peered":"2024-09-16T15:48:27.887691+0000","last_clean":"2024-09-16T15:48:27.887691+0000","last_became_active":"2024-09-16T15:48:26.501748+0000","last_became_peered":"2024-09-16T15:48:26.501748+0000","last_unstale":"2024-09-16T15:48:27.887691+0000","last_undegraded":"2024-09-16T15:48:27.887691+0000","last_fullsized":"2024-09-16T15:48:27.887691+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:48:25.420844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:48:25.420844+0000","last_clean_scrub_stamp":"2024-09-16T15:48:25.420844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T03:22:41.088805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.d","version":"102'1","reported_seq":16,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-09-16T15:48:27.466883+0000","last_change":"2024-09-16T15:48:25.452923+0000","last_active":"2024-09-16T15:48:27.466883+0000","last_peered":"2024-09-16T15:48:27.466883+0000","last_clean":"2024-09-16T15:48:27.466883+0000","last_became_active":"2024-09-16T15:48:25.452160+0000","last_became_peered":"2024-09-16T15:48:25.452160+0000","last_unstale":"2024-09-16T15:48:27.466883+0000","last_undegraded":"2024-09-16T15:48:27.466883+0000","last_fullsized":"2024-09-16T15:48:27.466883+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:48:24.407732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:48:24.407732+0000","last_clean_scrub_stamp":"2024-09-16T15:48:24.407732+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:55:18.056917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.d","version":"0'0","reported_seq":13,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-09-16T15:48:27.888231+0000","last_change":"2024-09-16T15:48:26.503986+0000","last_active":"2024-09-16T15:48:27.888231+0000","last_peered":"2024-09-16T15:48:27.888231+0000","last_clean":"2024-09-16T15:48:27.888231+0000","last_became_active":"2024-09-16T15:48:26.492181+0000","last_became_peered":"2024-09-16T15:48:26.492181+0000","last_unstale":"2024-09-16T15:48:27.888231+0000","last_undegraded":"2024-09-16T15:48:27.888231+0000","last_fullsized":"2024-09-16T15:48:27.888231+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:48:25.420844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:48:25.420844+0000","last_clean_scrub_stamp":"2024-09-16T15:48:25.420844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:43:22.624149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.c","version":"0'0","reported_seq":15,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-09-16T15:48:27.483316+0000","last_change":"2024-09-16T15:48:25.465602+0000","last_active":"2024-09-16T15:48:27.483316+0000","last_peered":"2024-09-16T15:48:27.483316+0000","last_clean":"2024-09-16T15:48:27.483316+0000","last_became_active":"2024-09-16T15:48:25.464705+0000","last_became_peered":"2024-09-16T15:48:25.464705+0000","last_unstale":"2024-09-16T15:48:27.483316+0000","last_undegraded":"2024-09-16T15:48:27.483316+0000","last_fullsized":"2024-09-16T15:48:27.483316+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:48:24.407732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:48:24.407732+0000","last_clean_scrub_stamp":"2024-09-16T15:48:24.407732+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T22:46:44.256372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.a","version":"0'0","reported_seq":13,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-09-16T15:48:27.887377+0000","last_change":"2024-09-16T15:48:26.509669+0000","last_active":"2024-09-16T15:48:27.887377+0000","last_peered":"2024-09-16T15:48:27.887377+0000","last_clean":"2024-09-16T15:48:27.887377+0000","last_became_active":"2024-09-16T15:48:26.509072+0000","last_became_peered":"2024-09-16T15:48:26.509072+0000","last_unstale":"2024-09-16T15:48:27.887377+0000","last_undegraded":"2024-09-16T15:48:27.887377+0000","last_fullsized":"2024-09-16T15:48:27.887377+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:48:25.420844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:48:25.420844+0000","last_clean_scrub_stamp":"2024-09-16T15:48:25.420844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:27:26.948503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.b","version":"102'2","reported_seq":17,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-09-16T15:48:27.999293+0000","last_change":"2024-09-16T15:48:25.446745+0000","last_active":"2024-09-16T15:48:27.999293+0000","last_peered":"2024-09-16T15:48:27.999293+0000","last_clean":"2024-09-16T15:48:27.999293+0000","last_became_active":"2024-09-16T15:48:25.446466+0000","last_became_peered":"2024-09-16T15:48:25.446466+0000","last_unstale":"2024-09-16T15:48:27.999293+0000","last_undegraded":"2024-09-16T15:48:27.999293+0000","last_fullsized":"2024-09-16T15:48:27.999293+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:48:24.407732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:48:24.407732+0000","last_clean_scrub_stamp":"2024-09-16T15:48:24.407732+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T22:55:53.466832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.b","version":"0'0","reported_seq":13,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-09-16T15:48:27.463002+0000","last_change":"2024-09-16T15:48:26.502896+0000","last_active":"2024-09-16T15:48:27.463002+0000","last_peered":"2024-09-16T15:48:27.463002+0000","last_clean":"2024-09-16T15:48:27.463002+0000","last_became_active":"2024-09-16T15:48:26.502324+0000","last_became_peered":"2024-09-16T15:48:26.502324+0000","last_unstale":"2024-09-16T15:48:27.463002+0000","last_undegraded":"2024-09-16T15:48:27.463002+0000","last_fullsized":"2024-09-16T15:48:27.463002+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:48:25.420844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:48:25.420844+0000","last_clean_scrub_stamp":"2024-09-16T15:48:25.420844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T02:50:17.688000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.a","version":"0'0","reported_seq":15,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-09-16T15:48:27.495549+0000","last_change":"2024-09-16T15:48:25.449113+0000","last_active":"2024-09-16T15:48:27.495549+0000","last_peered":"2024-09-16T15:48:27.495549+0000","last_clean":"2024-09-16T15:48:27.495549+0000","last_became_active":"2024-09-16T15:48:25.448377+0000","last_became_peered":"2024-09-16T15:48:25.448377+0000","last_unstale":"2024-09-16T15:48:27.495549+0000","last_undegraded":"2024-09-16T15:48:27.495549+0000","last_fullsized":"2024-09-16T15:48:27.495549+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:48:24.407732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:48:24.407732+0000","last_clean_scrub_stamp":"2024-09-16T15:48:24.407732+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:00:13.928185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.8","version":"0'0","reported_seq":13,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-09-16T15:48:27.887817+0000","last_change":"2024-09-16T15:48:26.500174+0000","last_active":"2024-09-16T15:48:27.887817+0000","last_peered":"2024-09-16T15:48:27.887817+0000","last_clean":"2024-09-16T15:48:27.887817+0000","last_became_active":"2024-09-16T15:48:26.492446+0000","last_became_peered":"2024-09-16T15:48:26.492446+0000","last_unstale":"2024-09-16T15:48:27.887817+0000","last_undegraded":"2024-09-16T15:48:27.887817+0000","last_fullsized":"2024-09-16T15:48:27.887817+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:48:25.420844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:48:25.420844+0000","last_clean_scrub_stamp":"2024-09-16T15:48:25.420844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T18:03:38.683560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.9","version":"0'0","reported_seq":15,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-09-16T15:48:27.998833+0000","last_change":"2024-09-16T15:48:25.447068+0000","last_active":"2024-09-16T15:48:27.998833+0000","last_peered":"2024-09-16T15:48:27.998833+0000","last_clean":"2024-09-16T15:48:27.998833+0000","last_became_active":"2024-09-16T15:48:25.446737+0000","last_became_peered":"2024-09-16T15:48:25.446737+0000","last_unstale":"2024-09-16T15:48:27.998833+0000","last_undegraded":"2024-09-16T15:48:27.998833+0000","last_fullsized":"2024-09-16T15:48:27.998833+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:48:24.407732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:48:24.407732+0000","last_clean_scrub_stamp":"2024-09-16T15:48:24.407732+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T00:27:59.760074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.9","version":"0'0","reported_seq":13,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-09-16T15:48:27.996655+0000","last_change":"2024-09-16T15:48:26.488945+0000","last_active":"2024-09-16T15:48:27.996655+0000","last_peered":"2024-09-16T15:48:27.996655+0000","last_clean":"2024-09-16T15:48:27.996655+0000","last_became_active":"2024-09-16T15:48:26.488681+0000","last_became_peered":"2024-09-16T15:48:26.488681+0000","last_unstale":"2024-09-16T15:48:27.996655+0000","last_undegraded":"2024-09-16T15:48:27.996655+0000","last_fullsized":"2024-09-16T15:48:27.996655+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:48:25.420844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:48:25.420844+0000","last_clean_scrub_stamp":"2024-09-16T15:48:25.420844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T02:48:53.694369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.8","version":"0'0","reported_seq":15,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-09-16T15:48:27.461938+0000","last_change":"2024-09-16T15:48:25.464185+0000","last_active":"2024-09-16T15:48:27.461938+0000","last_peered":"2024-09-16T15:48:27.461938+0000","last_clean":"2024-09-16T15:48:27.461938+0000","last_became_active":"2024-09-16T15:48:25.463336+0000","last_became_peered":"2024-09-16T15:48:25.463336+0000","last_unstale":"2024-09-16T15:48:27.461938+0000","last_undegraded":"2024-09-16T15:48:27.461938+0000","last_fullsized":"2024-09-16T15:48:27.461938+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:48:24.407732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:48:24.407732+0000","last_clean_scrub_stamp":"2024-09-16T15:48:24.407732+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T03:22:40.075693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.6","version":"0'0","reported_seq":13,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-09-16T15:48:27.467448+0000","last_change":"2024-09-16T15:48:26.501860+0000","last_active":"2024-09-16T15:48:27.467448+0000","last_peered":"2024-09-16T15:48:27.467448+0000","last_clean":"2024-09-16T15:48:27.467448+0000","last_became_active":"2024-09-16T15:48:26.500898+0000","last_became_peered":"2024-09-16T15:48:26.500898+0000","last_unstale":"2024-09-16T15:48:27.467448+0000","last_undegraded":"2024-09-16T15:48:27.467448+0000","last_fullsized":"2024-09-16T15:48:27.467448+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:48:25.420844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:48:25.420844+0000","last_clean_scrub_stamp":"2024-09-16T15:48:25.420844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:17:26.625596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.7","version":"102'2","reported_seq":17,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-09-16T15:48:27.997022+0000","last_change":"2024-09-16T15:48:25.463111+0000","last_active":"2024-09-16T15:48:27.997022+0000","last_peered":"2024-09-16T15:48:27.997022+0000","last_clean":"2024-09-16T15:48:27.997022+0000","last_became_active":"2024-09-16T15:48:25.462771+0000","last_became_peered":"2024-09-16T15:48:25.462771+0000","last_unstale":"2024-09-16T15:48:27.997022+0000","last_undegraded":"2024-09-16T15:48:27.997022+0000","last_fullsized":"2024-09-16T15:48:27.997022+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:48:24.407732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:48:24.407732+0000","last_clean_scrub_stamp":"2024-09-16T15:48:24.407732+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:19:56.751913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.7","version":"0'0","reported_seq":13,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-09-16T15:48:27.462125+0000","last_change":"2024-09-16T15:48:26.492559+0000","last_active":"2024-09-16T15:48:27.462125+0000","last_peered":"2024-09-16T15:48:27.462125+0000","last_clean":"2024-09-16T15:48:27.462125+0000","last_became_active":"2024-09-16T15:48:26.492232+0000","last_became_peered":"2024-09-16T15:48:26.492232+0000","last_unstale":"2024-09-16T15:48:27.462125+0000","last_undegraded":"2024-09-16T15:48:27.462125+0000","last_fullsized":"2024-09-16T15:48:27.462125+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:48:25.420844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:48:25.420844+0000","last_clean_scrub_stamp":"2024-09-16T15:48:25.420844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:21:55.934422+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.6","version":"103'1","reported_seq":16,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-09-16T15:48:27.467783+0000","last_change":"2024-09-16T15:48:25.458600+0000","last_active":"2024-09-16T15:48:27.467783+0000","last_peered":"2024-09-16T15:48:27.467783+0000","last_clean":"2024-09-16T15:48:27.467783+0000","last_became_active":"2024-09-16T15:48:25.457924+0000","last_became_peered":"2024-09-16T15:48:25.457924+0000","last_unstale":"2024-09-16T15:48:27.467783+0000","last_undegraded":"2024-09-16T15:48:27.467783+0000","last_fullsized":"2024-09-16T15:48:27.467783+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:48:24.407732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:48:24.407732+0000","last_clean_scrub_stamp":"2024-09-16T15:48:24.407732+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:07:47.395564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.4","version":"0'0","reported_seq":13,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-09-16T15:48:27.462681+0000","last_change":"2024-09-16T15:48:26.496183+0000","last_active":"2024-09-16T15:48:27.462681+0000","last_peered":"2024-09-16T15:48:27.462681+0000","last_clean":"2024-09-16T15:48:27.462681+0000","last_became_active":"2024-09-16T15:48:26.493124+0000","last_became_peered":"2024-09-16T15:48:26.493124+0000","last_unstale":"2024-09-16T15:48:27.462681+0000","last_undegraded":"2024-09-16T15:48:27.462681+0000","last_fullsized":"2024-09-16T15:48:27.462681+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:48:25.420844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:48:25.420844+0000","last_clean_scrub_stamp":"2024-09-16T15:48:25.420844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:50:48.377270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.5","version":"0'0","reported_seq":15,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-09-16T15:48:27.462847+0000","last_change":"2024-09-16T15:48:25.463122+0000","last_active":"2024-09-16T15:48:27.462847+0000","last_peered":"2024-09-16T15:48:27.462847+0000","last_clean":"2024-09-16T15:48:27.462847+0000","last_became_active":"2024-09-16T15:48:25.462802+0000","last_became_peered":"2024-09-16T15:48:25.462802+0000","last_unstale":"2024-09-16T15:48:27.462847+0000","last_undegraded":"2024-09-16T15:48:27.462847+0000","last_fullsized":"2024-09-16T15:48:27.462847+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:48:24.407732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:48:24.407732+0000","last_clean_scrub_stamp":"2024-09-16T15:48:24.407732+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T18:03:37.670448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.5","version":"0'0","reported_seq":13,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-09-16T15:48:27.466706+0000","last_change":"2024-09-16T15:48:26.488782+0000","last_active":"2024-09-16T15:48:27.466706+0000","last_peered":"2024-09-16T15:48:27.466706+0000","last_clean":"2024-09-16T15:48:27.466706+0000","last_became_active":"2024-09-16T15:48:26.487907+0000","last_became_peered":"2024-09-16T15:48:26.487907+0000","last_unstale":"2024-09-16T15:48:27.466706+0000","last_undegraded":"2024-09-16T15:48:27.466706+0000","last_fullsized":"2024-09-16T15:48:27.466706+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:48:25.420844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:48:25.420844+0000","last_clean_scrub_stamp":"2024-09-16T15:48:25.420844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:18:22.614416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.4","version":"0'0","reported_seq":15,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-09-16T15:48:27.997404+0000","last_change":"2024-09-16T15:48:25.449183+0000","last_active":"2024-09-16T15:48:27.997404+0000","last_peered":"2024-09-16T15:48:27.997404+0000","last_clean":"2024-09-16T15:48:27.997404+0000","last_became_active":"2024-09-16T15:48:25.448828+0000","last_became_peered":"2024-09-16T15:48:25.448828+0000","last_unstale":"2024-09-16T15:48:27.997404+0000","last_undegraded":"2024-09-16T15:48:27.997404+0000","last_fullsized":"2024-09-16T15:48:27.997404+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:48:24.407732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:48:24.407732+0000","last_clean_scrub_stamp":"2024-09-16T15:48:24.407732+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T02:56:07.811567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.2","version":"0'0","reported_seq":13,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-09-16T15:48:27.465265+0000","last_change":"2024-09-16T15:48:26.493658+0000","last_active":"2024-09-16T15:48:27.465265+0000","last_peered":"2024-09-16T15:48:27.465265+0000","last_clean":"2024-09-16T15:48:27.465265+0000","last_became_active":"2024-09-16T15:48:26.492904+0000","last_became_peered":"2024-09-16T15:48:26.492904+0000","last_unstale":"2024-09-16T15:48:27.465265+0000","last_undegraded":"2024-09-16T15:48:27.465265+0000","last_fullsized":"2024-09-16T15:48:27.465265+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:48:25.420844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:48:25.420844+0000","last_clean_scrub_stamp":"2024-09-16T15:48:25.420844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:29:34.405446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.3","version":"0'0","reported_seq":15,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-09-16T15:48:27.483021+0000","last_change":"2024-09-16T15:48:25.454707+0000","last_active":"2024-09-16T15:48:27.483021+0000","last_peered":"2024-09-16T15:48:27.483021+0000","last_clean":"2024-09-16T15:48:27.483021+0000","last_became_active":"2024-09-16T15:48:25.453977+0000","last_became_peered":"2024-09-16T15:48:25.453977+0000","last_unstale":"2024-09-16T15:48:27.483021+0000","last_undegraded":"2024-09-16T15:48:27.483021+0000","last_fullsized":"2024-09-16T15:48:27.483021+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:48:24.407732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:48:24.407732+0000","last_clean_scrub_stamp":"2024-09-16T15:48:24.407732+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T00:21:40.348171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.3","version":"0'0","reported_seq":13,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-09-16T15:48:27.466330+0000","last_change":"2024-09-16T15:48:26.471300+0000","last_active":"2024-09-16T15:48:27.466330+0000","last_peered":"2024-09-16T15:48:27.466330+0000","last_clean":"2024-09-16T15:48:27.466330+0000","last_became_active":"2024-09-16T15:48:26.470479+0000","last_became_peered":"2024-09-16T15:48:26.470479+0000","last_unstale":"2024-09-16T15:48:27.466330+0000","last_undegraded":"2024-09-16T15:48:27.466330+0000","last_fullsized":"2024-09-16T15:48:27.466330+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:48:25.420844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:48:25.420844+0000","last_clean_scrub_stamp":"2024-09-16T15:48:25.420844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T18:36:32.286233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.2","version":"0'0","reported_seq":15,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-09-16T15:48:27.465455+0000","last_change":"2024-09-16T15:48:25.449899+0000","last_active":"2024-09-16T15:48:27.465455+0000","last_peered":"2024-09-16T15:48:27.465455+0000","last_clean":"2024-09-16T15:48:27.465455+0000","last_became_active":"2024-09-16T15:48:25.449158+0000","last_became_peered":"2024-09-16T15:48:25.449158+0000","last_unstale":"2024-09-16T15:48:27.465455+0000","last_undegraded":"2024-09-16T15:48:27.465455+0000","last_fullsized":"2024-09-16T15:48:27.465455+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:48:24.407732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:48:24.407732+0000","last_clean_scrub_stamp":"2024-09-16T15:48:24.407732+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:06:09.718860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.0","version":"0'0","reported_seq":13,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-09-16T15:48:27.888643+0000","last_change":"2024-09-16T15:48:26.499317+0000","last_active":"2024-09-16T15:48:27.888643+0000","last_peered":"2024-09-16T15:48:27.888643+0000","last_clean":"2024-09-16T15:48:27.888643+0000","last_became_active":"2024-09-16T15:48:26.493152+0000","last_became_peered":"2024-09-16T15:48:26.493152+0000","last_unstale":"2024-09-16T15:48:27.888643+0000","last_undegraded":"2024-09-16T15:48:27.888643+0000","last_fullsized":"2024-09-16T15:48:27.888643+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:48:25.420844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:48:25.420844+0000","last_clean_scrub_stamp":"2024-09-16T15:48:25.420844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:16:49.823764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.1","version":"0'0","reported_seq":15,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-09-16T15:48:27.495694+0000","last_change":"2024-09-16T15:48:25.448607+0000","last_active":"2024-09-16T15:48:27.495694+0000","last_peered":"2024-09-16T15:48:27.495694+0000","last_clean":"2024-09-16T15:48:27.495694+0000","last_became_active":"2024-09-16T15:48:25.448153+0000","last_became_peered":"2024-09-16T15:48:25.448153+0000","last_unstale":"2024-09-16T15:48:27.495694+0000","last_undegraded":"2024-09-16T15:48:27.495694+0000","last_fullsized":"2024-09-16T15:48:27.495694+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:48:24.407732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:48:24.407732+0000","last_clean_scrub_stamp":"2024-09-16T15:48:24.407732+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:18:45.048781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1","version":"0'0","reported_seq":13,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-09-16T15:48:27.462262+0000","last_change":"2024-09-16T15:48:26.488351+0000","last_active":"2024-09-16T15:48:27.462262+0000","last_peered":"2024-09-16T15:48:27.462262+0000","last_clean":"2024-09-16T15:48:27.462262+0000","last_became_active":"2024-09-16T15:48:26.487799+0000","last_became_peered":"2024-09-16T15:48:26.487799+0000","last_unstale":"2024-09-16T15:48:27.462262+0000","last_undegraded":"2024-09-16T15:48:27.462262+0000","last_fullsized":"2024-09-16T15:48:27.462262+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:48:25.420844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:48:25.420844+0000","last_clean_scrub_stamp":"2024-09-16T15:48:25.420844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T23:34:29.350068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.0","version":"0'0","reported_seq":15,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-09-16T15:48:27.462196+0000","last_change":"2024-09-16T15:48:25.463812+0000","last_active":"2024-09-16T15:48:27.462196+0000","last_peered":"2024-09-16T15:48:27.462196+0000","last_clean":"2024-09-16T15:48:27.462196+0000","last_became_active":"2024-09-16T15:48:25.462975+0000","last_became_peered":"2024-09-16T15:48:25.462975+0000","last_unstale":"2024-09-16T15:48:27.462196+0000","last_undegraded":"2024-09-16T15:48:27.462196+0000","last_fullsized":"2024-09-16T15:48:27.462196+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:48:24.407732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:48:24.407732+0000","last_clean_scrub_stamp":"2024-09-16T15:48:24.407732+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T22:10:30.282104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.10","version":"0'0","reported_seq":13,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-09-16T15:48:27.996502+0000","last_change":"2024-09-16T15:48:26.471378+0000","last_active":"2024-09-16T15:48:27.996502+0000","last_peered":"2024-09-16T15:48:27.996502+0000","last_clean":"2024-09-16T15:48:27.996502+0000","last_became_active":"2024-09-16T15:48:26.470643+0000","last_became_peered":"2024-09-16T15:48:26.470643+0000","last_unstale":"2024-09-16T15:48:27.996502+0000","last_undegraded":"2024-09-16T15:48:27.996502+0000","last_fullsized":"2024-09-16T15:48:27.996502+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:48:25.420844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:48:25.420844+0000","last_clean_scrub_stamp":"2024-09-16T15:48:25.420844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:17:26.625596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.11","version":"0'0","reported_seq":15,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-09-16T15:48:27.483436+0000","last_change":"2024-09-16T15:48:25.463108+0000","last_active":"2024-09-16T15:48:27.483436+0000","last_peered":"2024-09-16T15:48:27.483436+0000","last_clean":"2024-09-16T15:48:27.483436+0000","last_became_active":"2024-09-16T15:48:25.462386+0000","last_became_peered":"2024-09-16T15:48:25.462386+0000","last_unstale":"2024-09-16T15:48:27.483436+0000","last_undegraded":"2024-09-16T15:48:27.483436+0000","last_fullsized":"2024-09-16T15:48:27.483436+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:48:24.407732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:48:24.407732+0000","last_clean_scrub_stamp":"2024-09-16T15:48:24.407732+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T00:25:43.259607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"73'79","reported_seq":291,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-09-16T15:48:27.888137+0000","last_change":"2024-09-16T15:42:40.881820+0000","last_active":"2024-09-16T15:48:27.888137+0000","last_peered":"2024-09-16T15:48:27.888137+0000","last_clean":"2024-09-16T15:48:27.888137+0000","last_became_active":"2024-09-16T15:42:40.879965+0000","last_became_peered":"2024-09-16T15:42:40.879965+0000","last_unstale":"2024-09-16T15:48:27.888137+0000","last_undegraded":"2024-09-16T15:48:27.888137+0000","last_fullsized":"2024-09-16T15:48:27.888137+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:42:39.728762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:42:39.728762+0000","last_clean_scrub_stamp":"2024-09-16T15:42:39.728762+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:22:18.768932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":17,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":16,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79,"ondisk_log_size":79,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542218,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33024,"kb_used_data":1836,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338816,"statfs":{"total":96636764160,"available":96602947584,"internally_reserved":0,"allocated":1880064,"data_stored":1150644,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542217,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32440,"kb_used_data":1252,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339400,"statfs":{"total":96636764160,"available":96603545600,"internally_reserved":0,"allocated":1282048,"data_stored":559730,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":5,"up_from":14,"seq":60129542218,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32448,"kb_used_data":1256,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339392,"statfs":{"total":96636764160,"available":96603537408,"internally_reserved":0,"allocated":1286144,"data_stored":560264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542216,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32436,"kb_used_data":1248,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339404,"statfs":{"total":96636764160,"available":96603549696,"internally_reserved":0,"allocated":1277952,"data_stored":559660,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[]},{"osd":3,"up_from":14,"seq":60129542217,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32452,"kb_used_data":1264,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339388,"statfs":{"total":96636764160,"available":96603533312,"internally_reserved":0,"allocated":1294336,"data_stored":561342,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":2,"up_from":14,"seq":60129542217,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32440,"kb_used_data":1248,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339400,"statfs":{"total":96636764160,"available":96603545600,"internally_reserved":0,"allocated":1277952,"data_stored":559660,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":14,"seq":60129542219,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32448,"kb_used_data":1256,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339392,"statfs":{"total":96636764160,"available":96603537408,"internally_reserved":0,"allocated":1286144,"data_stored":559784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":14,"seq":60129542219,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33044,"kb_used_data":1848,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338796,"statfs":{"total":96636764160,"available":96602927104,"internally_reserved":0,"allocated":1892352,"data_stored":1151752,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T15:48:30.867 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-09-16T15:48:30.867 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-09-16T15:48:31.046 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:31.047+0000 7f7a6992f640 1 -- 172.21.15.123:0/1364528803 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7a4c004010 msgr2=0x7f7a4c004810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:48:31.046 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:31.047+0000 7f7a6992f640 1 --2- 172.21.15.123:0/1364528803 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7a4c004010 0x7f7a4c004810 secure :-1 s=READY pgs=1081 cs=0 l=1 rev1=1 crypto rx=0x7f7a58009b30 tx=0x7f7a5801d230 comp rx=0 tx=0).stop 2024-09-16T15:48:31.046 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:31.047+0000 7f7a637fe640 1 -- 172.21.15.123:0/1364528803 reap_dead start 2024-09-16T15:48:31.046 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:31.047+0000 7f7a6992f640 1 -- 172.21.15.123:0/1364528803 shutdown_connections 2024-09-16T15:48:31.046 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:31.047+0000 7f7a6992f640 1 -- 172.21.15.123:0/1364528803 >> 172.21.15.123:0/1364528803 conn(0x7f7a6409aa40 msgr2=0x7f7a6409ae50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:48:31.046 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:31.047+0000 7f7a6992f640 1 -- 172.21.15.123:0/1364528803 shutdown_connections 2024-09-16T15:48:31.046 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:31.047+0000 7f7a6992f640 1 -- 172.21.15.123:0/1364528803 wait complete. 2024-09-16T15:48:31.047 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:31.047+0000 7f7a6992f640 1 Processor -- start 2024-09-16T15:48:31.047 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:31.047+0000 7f7a6992f640 1 -- start start 2024-09-16T15:48:31.049 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:31.047+0000 7f7a6992f640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7a4c004010 0x7f7a6406f350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:48:31.049 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:31.047+0000 7f7a6992f640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7a641582e0 0x7f7a640b3ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:48:31.049 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:31.047+0000 7f7a6992f640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f7a641f7ed0 0x7f7a641f8350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:48:31.049 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:31.047+0000 7f7a6992f640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f7a64071260 con 0x7f7a641582e0 2024-09-16T15:48:31.050 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:31.047+0000 7f7a6992f640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f7a640710e0 con 0x7f7a4c004010 2024-09-16T15:48:31.050 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:31.047+0000 7f7a6992f640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f7a640ccf50 con 0x7f7a641f7ed0 2024-09-16T15:48:31.050 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:31.047+0000 7f7a627fc640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7a641582e0 0x7f7a640b3ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:48:31.050 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:31.047+0000 7f7a627fc640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7a641582e0 0x7f7a640b3ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:47540/0 (socket says 172.21.15.123:47540) 2024-09-16T15:48:31.050 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:31.047+0000 7f7a627fc640 1 -- 172.21.15.123:0/3319968417 learned_addr learned my addr 172.21.15.123:0/3319968417 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:48:31.050 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:31.047+0000 7f7a62ffd640 1 --2- 172.21.15.123:0/3319968417 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7a4c004010 0x7f7a6406f350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:48:31.050 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:31.047+0000 7f7a637fe640 1 --2- 172.21.15.123:0/3319968417 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f7a641f7ed0 0x7f7a641f8350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:48:31.050 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:31.047+0000 7f7a627fc640 1 -- 172.21.15.123:0/3319968417 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f7a641f7ed0 msgr2=0x7f7a641f8350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:48:31.050 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:31.047+0000 7f7a627fc640 1 --2- 172.21.15.123:0/3319968417 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f7a641f7ed0 0x7f7a641f8350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:31.050 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:31.047+0000 7f7a627fc640 1 -- 172.21.15.123:0/3319968417 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7a4c004010 msgr2=0x7f7a6406f350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:48:31.050 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:31.047+0000 7f7a627fc640 1 --2- 172.21.15.123:0/3319968417 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7a4c004010 0x7f7a6406f350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:31.050 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:31.047+0000 7f7a627fc640 1 -- 172.21.15.123:0/3319968417 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7a640b4000 con 0x7f7a641582e0 2024-09-16T15:48:31.050 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:31.047+0000 7f7a627fc640 1 --2- 172.21.15.123:0/3319968417 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7a641582e0 0x7f7a640b3ac0 secure :-1 s=READY pgs=1082 cs=0 l=1 rev1=1 crypto rx=0x7f7a5801d060 tx=0x7f7a58005620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:48:31.050 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:31.047+0000 7f7a4bfff640 1 -- 172.21.15.123:0/3319968417 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a580b1040 con 0x7f7a641582e0 2024-09-16T15:48:31.051 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:31.047+0000 7f7a4bfff640 1 -- 172.21.15.123:0/3319968417 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f7a580bc070 con 0x7f7a641582e0 2024-09-16T15:48:31.051 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:31.047+0000 7f7a4bfff640 1 -- 172.21.15.123:0/3319968417 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a580acb00 con 0x7f7a641582e0 2024-09-16T15:48:31.051 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:31.047+0000 7f7a6992f640 1 -- 172.21.15.123:0/3319968417 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7a6406dfd0 con 0x7f7a641582e0 2024-09-16T15:48:31.051 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:31.047+0000 7f7a6992f640 1 -- 172.21.15.123:0/3319968417 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7a6406e590 con 0x7f7a641582e0 2024-09-16T15:48:31.051 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:31.051+0000 7f7a6992f640 1 -- 172.21.15.123:0/3319968417 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7a6415f880 con 0x7f7a641582e0 2024-09-16T15:48:31.051 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:31.051+0000 7f7a4bfff640 1 -- 172.21.15.123:0/3319968417 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 16) ==== 97182+0+0 (secure 0 0 0) 0x7f7a580acca0 con 0x7f7a641582e0 2024-09-16T15:48:31.052 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:31.051+0000 7f7a4bfff640 1 --2- 172.21.15.123:0/3319968417 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f7a50073010 0x7f7a500754d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:48:31.052 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:31.051+0000 7f7a4bfff640 1 -- 172.21.15.123:0/3319968417 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(103..103 src has 1..103) ==== 5709+0+0 (secure 0 0 0) 0x7f7a5812faf0 con 0x7f7a641582e0 2024-09-16T15:48:31.052 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:31.051+0000 7f7a62ffd640 1 --2- 172.21.15.123:0/3319968417 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f7a50073010 0x7f7a500754d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:48:31.052 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:31.051+0000 7f7a62ffd640 1 --2- 172.21.15.123:0/3319968417 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f7a50073010 0x7f7a500754d0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f7a54006fd0 tx=0x7f7a54006b30 comp rx=0 tx=0).ready entity=mgr.4150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:48:31.055 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:31.055+0000 7f7a4bfff640 1 -- 172.21.15.123:0/3319968417 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f7a58016f40 con 0x7f7a641582e0 2024-09-16T15:48:31.294 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:31.295+0000 7f7a6992f640 1 -- 172.21.15.123:0/3319968417 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) -- 0x7f7a64158740 con 0x7f7a641582e0 2024-09-16T15:48:31.297 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:31.295+0000 7f7a4bfff640 1 -- 172.21.15.123:0/3319968417 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v24) ==== 190+0+147 (secure 0 0 0) 0x7f7a580fac90 con 0x7f7a641582e0 2024-09-16T15:48:31.298 INFO:teuthology.orchestra.run.smithi123.stdout:[client.0] 2024-09-16T15:48:31.298 INFO:teuthology.orchestra.run.smithi123.stdout: key = AQC1Uehm7GDTAhAAHWWVIyjVVnTeeozBR8DfFA== 2024-09-16T15:48:31.298 INFO:teuthology.orchestra.run.smithi123.stdout: caps mds = "allow" 2024-09-16T15:48:31.298 INFO:teuthology.orchestra.run.smithi123.stdout: caps mon = "allow r" 2024-09-16T15:48:31.298 INFO:teuthology.orchestra.run.smithi123.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-09-16T15:48:31.298 INFO:teuthology.orchestra.run.smithi123.stderr:updated caps for client.0 2024-09-16T15:48:31.300 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:31.299+0000 7f7a6992f640 1 -- 172.21.15.123:0/3319968417 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f7a50073010 msgr2=0x7f7a500754d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:48:31.300 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:31.299+0000 7f7a6992f640 1 --2- 172.21.15.123:0/3319968417 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f7a50073010 0x7f7a500754d0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f7a54006fd0 tx=0x7f7a54006b30 comp rx=0 tx=0).stop 2024-09-16T15:48:31.300 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:31.299+0000 7f7a6992f640 1 -- 172.21.15.123:0/3319968417 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7a641582e0 msgr2=0x7f7a640b3ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:48:31.300 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:31.299+0000 7f7a6992f640 1 --2- 172.21.15.123:0/3319968417 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7a641582e0 0x7f7a640b3ac0 secure :-1 s=READY pgs=1082 cs=0 l=1 rev1=1 crypto rx=0x7f7a5801d060 tx=0x7f7a58005620 comp rx=0 tx=0).stop 2024-09-16T15:48:31.301 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:31.299+0000 7f7a6992f640 1 -- 172.21.15.123:0/3319968417 shutdown_connections 2024-09-16T15:48:31.301 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:31.299+0000 7f7a6992f640 1 --2- 172.21.15.123:0/3319968417 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f7a50073010 0x7f7a500754d0 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:31.301 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:31.299+0000 7f7a6992f640 1 --2- 172.21.15.123:0/3319968417 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f7a641f7ed0 0x7f7a641f8350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:31.301 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:31.299+0000 7f7a6992f640 1 --2- 172.21.15.123:0/3319968417 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7a641582e0 0x7f7a640b3ac0 unknown :-1 s=CLOSED pgs=1082 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:31.301 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:31.299+0000 7f7a6992f640 1 --2- 172.21.15.123:0/3319968417 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7a4c004010 0x7f7a6406f350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:31.301 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:31.299+0000 7f7a6992f640 1 -- 172.21.15.123:0/3319968417 >> 172.21.15.123:0/3319968417 conn(0x7f7a6409aa40 msgr2=0x7f7a6415b370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:48:31.301 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:31.299+0000 7f7a6992f640 1 -- 172.21.15.123:0/3319968417 shutdown_connections 2024-09-16T15:48:31.301 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:31.299+0000 7f7a6992f640 1 -- 172.21.15.123:0/3319968417 wait complete. 2024-09-16T15:48:31.316 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-16T15:48:31.438 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:31.439+0000 7f78eea61640 1 -- 172.21.15.123:0/1612177701 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f78e80d0200 msgr2=0x7f78e80b3f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:48:31.438 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:31.439+0000 7f78eea61640 1 --2- 172.21.15.123:0/1612177701 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f78e80d0200 0x7f78e80b3f80 secure :-1 s=READY pgs=1084 cs=0 l=1 rev1=1 crypto rx=0x7f78d8009840 tx=0x7f78d801f010 comp rx=0 tx=0).stop 2024-09-16T15:48:31.438 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:31.439+0000 7f78eea61640 1 -- 172.21.15.123:0/1612177701 shutdown_connections 2024-09-16T15:48:31.438 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:31.439+0000 7f78eea61640 1 --2- 172.21.15.123:0/1612177701 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f78e80d0200 0x7f78e80b3f80 unknown :-1 s=CLOSED pgs=1084 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:31.439 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:31.439+0000 7f78eea61640 1 --2- 172.21.15.123:0/1612177701 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f78e80d0aa0 0x7f78e80d1d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:31.439 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:31.439+0000 7f78eea61640 1 -- 172.21.15.123:0/1612177701 >> 172.21.15.123:0/1612177701 conn(0x7f78e80d3e30 msgr2=0x7f78e80d4240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:48:31.439 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:31.439+0000 7f78eea61640 1 -- 172.21.15.123:0/1612177701 shutdown_connections 2024-09-16T15:48:31.439 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:31.439+0000 7f78eea61640 1 -- 172.21.15.123:0/1612177701 wait complete. 2024-09-16T15:48:31.439 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:31.439+0000 7f78eea61640 1 Processor -- start 2024-09-16T15:48:31.439 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:31.439+0000 7f78eea61640 1 -- start start 2024-09-16T15:48:31.441 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:31.439+0000 7f78eea61640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f78e80cf530 0x7f78e80ca310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:48:31.441 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:31.439+0000 7f78eea61640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f78e80d0200 0x7f78e80b0a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:48:31.442 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:31.439+0000 7f78eea61640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f78e80d0aa0 0x7f78e80b0f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:48:31.442 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:31.439+0000 7f78eea61640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f78e80b44c0 con 0x7f78e80cf530 2024-09-16T15:48:31.442 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:31.439+0000 7f78eea61640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f78e80d1090 con 0x7f78e80d0aa0 2024-09-16T15:48:31.442 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:31.439+0000 7f78eea61640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f78e80b4640 con 0x7f78e80d0200 2024-09-16T15:48:31.442 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:31.439+0000 7f78e7fff640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f78e80cf530 0x7f78e80ca310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:48:31.442 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:31.439+0000 7f78e7fff640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f78e80cf530 0x7f78e80ca310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:47556/0 (socket says 172.21.15.123:47556) 2024-09-16T15:48:31.442 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:31.439+0000 7f78e7fff640 1 -- 172.21.15.123:0/1907168199 learned_addr learned my addr 172.21.15.123:0/1907168199 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:48:31.442 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:31.439+0000 7f78e77fe640 1 --2- 172.21.15.123:0/1907168199 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f78e80d0200 0x7f78e80b0a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:48:31.442 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:31.439+0000 7f78e7fff640 1 -- 172.21.15.123:0/1907168199 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f78e80d0200 msgr2=0x7f78e80b0a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:48:31.442 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:31.439+0000 7f78e7fff640 1 --2- 172.21.15.123:0/1907168199 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f78e80d0200 0x7f78e80b0a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:31.442 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:31.439+0000 7f78ecffe640 1 --2- 172.21.15.123:0/1907168199 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f78e80d0aa0 0x7f78e80b0f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:48:31.442 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:31.439+0000 7f78e7fff640 1 -- 172.21.15.123:0/1907168199 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f78e80d0aa0 msgr2=0x7f78e80b0f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:48:31.442 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:31.439+0000 7f78e7fff640 1 --2- 172.21.15.123:0/1907168199 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f78e80d0aa0 0x7f78e80b0f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:31.443 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:31.439+0000 7f78e7fff640 1 -- 172.21.15.123:0/1907168199 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f78e80ca850 con 0x7f78e80cf530 2024-09-16T15:48:31.443 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:31.439+0000 7f78e7fff640 1 --2- 172.21.15.123:0/1907168199 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f78e80cf530 0x7f78e80ca310 secure :-1 s=READY pgs=1085 cs=0 l=1 rev1=1 crypto rx=0x7f78e000ba50 tx=0x7f78e000bf10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:48:31.443 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:31.439+0000 7f78e4ff9640 1 -- 172.21.15.123:0/1907168199 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78e0019070 con 0x7f78e80cf530 2024-09-16T15:48:31.443 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:31.439+0000 7f78e4ff9640 1 -- 172.21.15.123:0/1907168199 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f78e000ccf0 con 0x7f78e80cf530 2024-09-16T15:48:31.443 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:31.439+0000 7f78e4ff9640 1 -- 172.21.15.123:0/1907168199 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78e00146a0 con 0x7f78e80cf530 2024-09-16T15:48:31.443 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:31.439+0000 7f78eea61640 1 -- 172.21.15.123:0/1907168199 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f78e80c8a10 con 0x7f78e80cf530 2024-09-16T15:48:31.443 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:31.443+0000 7f78eea61640 1 -- 172.21.15.123:0/1907168199 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f78e80c8fd0 con 0x7f78e80cf530 2024-09-16T15:48:31.444 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:31.443+0000 7f78eea61640 1 -- 172.21.15.123:0/1907168199 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f78e80a1db0 con 0x7f78e80cf530 2024-09-16T15:48:31.444 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:31.443+0000 7f78e4ff9640 1 -- 172.21.15.123:0/1907168199 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 16) ==== 97182+0+0 (secure 0 0 0) 0x7f78e001c070 con 0x7f78e80cf530 2024-09-16T15:48:31.444 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:31.443+0000 7f78e4ff9640 1 --2- 172.21.15.123:0/1907168199 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f78cc072fe0 0x7f78cc0754a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:48:31.444 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:31.443+0000 7f78e4ff9640 1 -- 172.21.15.123:0/1907168199 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(103..103 src has 1..103) ==== 5709+0+0 (secure 0 0 0) 0x7f78e0093e50 con 0x7f78e80cf530 2024-09-16T15:48:31.444 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:31.443+0000 7f78e77fe640 1 --2- 172.21.15.123:0/1907168199 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f78cc072fe0 0x7f78cc0754a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:48:31.444 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:31.443+0000 7f78e77fe640 1 --2- 172.21.15.123:0/1907168199 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f78cc072fe0 0x7f78cc0754a0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f78e80ce0b0 tx=0x7f78d0009040 comp rx=0 tx=0).ready entity=mgr.4150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:48:31.447 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:31.447+0000 7f78e4ff9640 1 -- 172.21.15.123:0/1907168199 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f78e005ee80 con 0x7f78e80cf530 2024-09-16T15:48:31.642 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:31.643+0000 7f78eea61640 1 -- 172.21.15.123:0/1907168199 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f78e80b2150 con 0x7f78e80cf530 2024-09-16T15:48:31.643 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:31.643+0000 7f78e4ff9640 1 -- 172.21.15.123:0/1907168199 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 68 v68) ==== 94+0+5237 (secure 0 0 0) 0x7f78e005e780 con 0x7f78e80cf530 2024-09-16T15:48:31.643 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-09-16T15:48:31.643 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":68,"btime":"2024-09-16T15:48:29:489616+0000","default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5417,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6835/3411324594","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":3411324594},{"type":"v1","addr":"172.21.15.184:6835","nonce":3411324594}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":63},{"gid":5848,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6833/1488440148","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":1488440148},{"type":"v1","addr":"172.21.15.184:6833","nonce":1488440148}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":63},{"gid":7032,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6833/1841102587","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6832","nonce":1841102587},{"type":"v1","addr":"172.21.15.123:6833","nonce":1841102587}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":63}],"filesystems":[{"mdsmap":{"epoch":68,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-16T15:48:25.813334+0000","modified":"2024-09-16T15:48:28.834708+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7035},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7035":{"gid":7035,"name":"c","rank":0,"incarnation":64,"state":"up:active","state_seq":3,"addr":"172.21.15.123:6835/2302124587","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6834","nonce":2302124587},{"type":"v1","addr":"172.21.15.123:6835","nonce":2302124587}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7035,"qdb_cluster":[7035]},"id":8}]} 2024-09-16T15:48:31.644 INFO:teuthology.orchestra.run.smithi123.stderr:dumped fsmap epoch 68 2024-09-16T15:48:31.646 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:31.647+0000 7f78eea61640 1 -- 172.21.15.123:0/1907168199 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f78cc072fe0 msgr2=0x7f78cc0754a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:48:31.646 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:31.647+0000 7f78eea61640 1 --2- 172.21.15.123:0/1907168199 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f78cc072fe0 0x7f78cc0754a0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f78e80ce0b0 tx=0x7f78d0009040 comp rx=0 tx=0).stop 2024-09-16T15:48:31.646 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:31.647+0000 7f78eea61640 1 -- 172.21.15.123:0/1907168199 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f78e80cf530 msgr2=0x7f78e80ca310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:48:31.646 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:31.647+0000 7f78eea61640 1 --2- 172.21.15.123:0/1907168199 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f78e80cf530 0x7f78e80ca310 secure :-1 s=READY pgs=1085 cs=0 l=1 rev1=1 crypto rx=0x7f78e000ba50 tx=0x7f78e000bf10 comp rx=0 tx=0).stop 2024-09-16T15:48:31.646 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:31.647+0000 7f78eea61640 1 -- 172.21.15.123:0/1907168199 shutdown_connections 2024-09-16T15:48:31.647 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:31.647+0000 7f78eea61640 1 --2- 172.21.15.123:0/1907168199 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f78cc072fe0 0x7f78cc0754a0 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:31.647 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:31.647+0000 7f78eea61640 1 --2- 172.21.15.123:0/1907168199 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f78e80d0aa0 0x7f78e80b0f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:31.647 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:31.647+0000 7f78eea61640 1 --2- 172.21.15.123:0/1907168199 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f78e80d0200 0x7f78e80b0a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:31.647 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:31.647+0000 7f78eea61640 1 --2- 172.21.15.123:0/1907168199 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f78e80cf530 0x7f78e80ca310 unknown :-1 s=CLOSED pgs=1085 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:31.647 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:31.647+0000 7f78eea61640 1 -- 172.21.15.123:0/1907168199 >> 172.21.15.123:0/1907168199 conn(0x7f78e80d3e30 msgr2=0x7f78e80b1a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:48:31.647 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:31.647+0000 7f78eea61640 1 -- 172.21.15.123:0/1907168199 shutdown_connections 2024-09-16T15:48:31.647 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:31.647+0000 7f78eea61640 1 -- 172.21.15.123:0/1907168199 wait complete. 2024-09-16T15:48:32.065 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 68, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-09-16T15:48:25.813334+0000', 'modified': '2024-09-16T15:48:28.834708+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 7035}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7035': {'gid': 7035, 'name': 'c', 'rank': 0, 'incarnation': 64, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.123:6835/2302124587', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.123:6834', 'nonce': 2302124587}, {'type': 'v1', 'addr': '172.21.15.123:6835', 'nonce': 2302124587}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540719139924082687, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [17], 'metadata_pool': 16, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 7035, 'qdb_cluster': [7035]} 2024-09-16T15:48:32.065 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-09-16T15:48:32.065 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-09-16T15:48:32.202 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:32.203+0000 7f090cfaa640 1 -- 172.21.15.123:0/4277971968 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f09080cf5a0 msgr2=0x7f090806eee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:48:32.202 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:32.203+0000 7f090cfaa640 1 --2- 172.21.15.123:0/4277971968 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f09080cf5a0 0x7f090806eee0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f08f000b9d0 tx=0x7f08f001e490 comp rx=0 tx=0).stop 2024-09-16T15:48:32.202 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:32.203+0000 7f090cfaa640 1 -- 172.21.15.123:0/4277971968 shutdown_connections 2024-09-16T15:48:32.202 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:32.203+0000 7f090cfaa640 1 --2- 172.21.15.123:0/4277971968 >> v2:172.21.15.184:3300/0 conn(0x7f090815ec40 0x7f090806f420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:32.202 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:32.203+0000 7f090cfaa640 1 --2- 172.21.15.123:0/4277971968 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f09080cf5a0 0x7f090806eee0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:32.202 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:32.203+0000 7f090cfaa640 1 -- 172.21.15.123:0/4277971968 >> 172.21.15.123:0/4277971968 conn(0x7f09080cfb10 msgr2=0x7f09080cff20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:48:32.203 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:32.203+0000 7f090cfaa640 1 -- 172.21.15.123:0/4277971968 shutdown_connections 2024-09-16T15:48:32.203 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:32.203+0000 7f090cfaa640 1 -- 172.21.15.123:0/4277971968 wait complete. 2024-09-16T15:48:32.203 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:32.203+0000 7f090cfaa640 1 Processor -- start 2024-09-16T15:48:32.203 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:32.203+0000 7f090cfaa640 1 -- start start 2024-09-16T15:48:32.205 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:32.203+0000 7f090cfaa640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f09080cf5a0 0x7f090815a200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:48:32.205 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:32.203+0000 7f090cfaa640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f0908158120 0x7f0908072710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:48:32.205 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:32.203+0000 7f090cfaa640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f090815ec40 0x7f09080734b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:48:32.205 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:32.203+0000 7f090cfaa640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f090815b230 con 0x7f09080cf5a0 2024-09-16T15:48:32.205 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:32.203+0000 7f090cfaa640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f090815b0b0 con 0x7f0908158120 2024-09-16T15:48:32.206 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:32.203+0000 7f090cfaa640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f090815b3b0 con 0x7f090815ec40 2024-09-16T15:48:32.206 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:32.203+0000 7f090659c640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f09080cf5a0 0x7f090815a200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:48:32.206 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:32.203+0000 7f090659c640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f09080cf5a0 0x7f090815a200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:47560/0 (socket says 172.21.15.123:47560) 2024-09-16T15:48:32.206 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:32.203+0000 7f090659c640 1 -- 172.21.15.123:0/1275143563 learned_addr learned my addr 172.21.15.123:0/1275143563 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:48:32.206 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:32.203+0000 7f08fffff640 1 --2- 172.21.15.123:0/1275143563 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f0908158120 0x7f0908072710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:48:32.206 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:32.203+0000 7f0906d9d640 1 --2- 172.21.15.123:0/1275143563 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f090815ec40 0x7f09080734b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:48:32.206 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:32.203+0000 7f090659c640 1 -- 172.21.15.123:0/1275143563 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f090815ec40 msgr2=0x7f09080734b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:48:32.206 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:32.203+0000 7f090659c640 1 --2- 172.21.15.123:0/1275143563 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f090815ec40 0x7f09080734b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:32.206 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:32.203+0000 7f090659c640 1 -- 172.21.15.123:0/1275143563 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f0908158120 msgr2=0x7f0908072710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:48:32.206 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:32.203+0000 7f090659c640 1 --2- 172.21.15.123:0/1275143563 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f0908158120 0x7f0908072710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:32.206 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:32.203+0000 7f090659c640 1 -- 172.21.15.123:0/1275143563 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0908153290 con 0x7f09080cf5a0 2024-09-16T15:48:32.206 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:32.203+0000 7f090659c640 1 --2- 172.21.15.123:0/1275143563 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f09080cf5a0 0x7f090815a200 secure :-1 s=READY pgs=1086 cs=0 l=1 rev1=1 crypto rx=0x7f08f800c530 tx=0x7f08f800c9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:48:32.207 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:32.203+0000 7f08feffd640 1 -- 172.21.15.123:0/1275143563 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08f8017070 con 0x7f09080cf5a0 2024-09-16T15:48:32.207 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:32.203+0000 7f08feffd640 1 -- 172.21.15.123:0/1275143563 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f08f80053c0 con 0x7f09080cf5a0 2024-09-16T15:48:32.207 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:32.203+0000 7f08feffd640 1 -- 172.21.15.123:0/1275143563 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08f8012890 con 0x7f09080cf5a0 2024-09-16T15:48:32.207 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:32.203+0000 7f090cfaa640 1 -- 172.21.15.123:0/1275143563 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0908153510 con 0x7f09080cf5a0 2024-09-16T15:48:32.207 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:32.203+0000 7f090cfaa640 1 -- 172.21.15.123:0/1275143563 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0908153ad0 con 0x7f09080cf5a0 2024-09-16T15:48:32.208 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:32.207+0000 7f08feffd640 1 -- 172.21.15.123:0/1275143563 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 16) ==== 97182+0+0 (secure 0 0 0) 0x7f08f801a070 con 0x7f09080cf5a0 2024-09-16T15:48:32.208 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:32.207+0000 7f08feffd640 1 --2- 172.21.15.123:0/1275143563 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f08e8072fe0 0x7f08e80754a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:48:32.208 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:32.207+0000 7f08feffd640 1 -- 172.21.15.123:0/1275143563 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(103..103 src has 1..103) ==== 5709+0+0 (secure 0 0 0) 0x7f08f8093300 con 0x7f09080cf5a0 2024-09-16T15:48:32.208 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:32.207+0000 7f08fffff640 1 --2- 172.21.15.123:0/1275143563 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f08e8072fe0 0x7f08e80754a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:48:32.208 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:32.207+0000 7f0910094000 1 Processor -- start 2024-09-16T15:48:32.208 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:32.207+0000 7f08fffff640 1 --2- 172.21.15.123:0/1275143563 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f08e8072fe0 0x7f08e80754a0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f08f00b1000 tx=0x7f08f001ccd0 comp rx=0 tx=0).ready entity=mgr.4150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:48:32.208 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:32.207+0000 7f0910094000 1 -- start start 2024-09-16T15:48:32.210 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:32.207+0000 7f0910094000 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55a9fb4979d0 0x55a9fb497db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:48:32.211 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:32.207+0000 7f0910094000 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x55a9fb4982f0 0x55a9fb44b8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:48:32.211 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:32.207+0000 7f0910094000 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a9fb44be10 0x55a9fb44c1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:48:32.211 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:32.207+0000 7f0910094000 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x55a9fb36c3f0 con 0x55a9fb4979d0 2024-09-16T15:48:32.211 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:32.207+0000 7f0910094000 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x55a9fb2a9b20 con 0x55a9fb44be10 2024-09-16T15:48:32.211 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:32.207+0000 7f0910094000 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x55a9fb298f50 con 0x55a9fb4982f0 2024-09-16T15:48:32.211 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:32.207+0000 7f090659c640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55a9fb4979d0 0x55a9fb497db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:48:32.211 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:32.207+0000 7f090659c640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55a9fb4979d0 0x55a9fb497db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:47562/0 (socket says 172.21.15.123:47562) 2024-09-16T15:48:32.211 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:32.207+0000 7f090659c640 1 -- 172.21.15.123:0/2587713216 learned_addr learned my addr 172.21.15.123:0/2587713216 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:48:32.211 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:32.207+0000 7f090659c640 1 -- 172.21.15.123:0/2587713216 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x55a9fb4982f0 msgr2=0x55a9fb44b8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T15:48:32.211 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:32.207+0000 7f090659c640 1 --2- 172.21.15.123:0/2587713216 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x55a9fb4982f0 0x55a9fb44b8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:32.211 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:32.207+0000 7f090659c640 1 -- 172.21.15.123:0/2587713216 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a9fb44be10 msgr2=0x55a9fb44c1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T15:48:32.211 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:32.207+0000 7f090659c640 1 --2- 172.21.15.123:0/2587713216 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a9fb44be10 0x55a9fb44c1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:32.211 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:32.207+0000 7f090659c640 1 -- 172.21.15.123:0/2587713216 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a9fb260590 con 0x55a9fb4979d0 2024-09-16T15:48:32.212 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:32.207+0000 7f090659c640 1 --2- 172.21.15.123:0/2587713216 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55a9fb4979d0 0x55a9fb497db0 secure :-1 s=READY pgs=1087 cs=0 l=1 rev1=1 crypto rx=0x7f08f800c500 tx=0x7f08f805d7d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f77995554093bb33 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:48:32.212 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:32.207+0000 7f08e7fff640 1 -- 172.21.15.123:0/2587713216 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08f805dcc0 con 0x55a9fb4979d0 2024-09-16T15:48:32.212 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:32.207+0000 7f08e7fff640 1 -- 172.21.15.123:0/2587713216 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f08f80627a0 con 0x55a9fb4979d0 2024-09-16T15:48:32.212 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:32.211+0000 7f08e7fff640 1 -- 172.21.15.123:0/2587713216 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08f8067650 con 0x55a9fb4979d0 2024-09-16T15:48:32.212 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:32.211+0000 7f0910094000 1 -- 172.21.15.123:0/2587713216 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55a9fb4979d0 msgr2=0x55a9fb497db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:48:32.212 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:32.211+0000 7f0910094000 1 --2- 172.21.15.123:0/2587713216 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55a9fb4979d0 0x55a9fb497db0 secure :-1 s=READY pgs=1087 cs=0 l=1 rev1=1 crypto rx=0x7f08f800c500 tx=0x7f08f805d7d0 comp rx=0 tx=0).stop 2024-09-16T15:48:32.212 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:32.211+0000 7f0910094000 1 -- 172.21.15.123:0/2587713216 shutdown_connections 2024-09-16T15:48:32.212 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:32.211+0000 7f0910094000 1 --2- 172.21.15.123:0/2587713216 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a9fb44be10 0x55a9fb44c1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:32.212 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:32.211+0000 7f0910094000 1 --2- 172.21.15.123:0/2587713216 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x55a9fb4982f0 0x55a9fb44b8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:32.212 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:32.211+0000 7f0910094000 1 --2- 172.21.15.123:0/2587713216 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55a9fb4979d0 0x55a9fb497db0 unknown :-1 s=CLOSED pgs=1087 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:32.212 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:32.211+0000 7f0910094000 1 -- 172.21.15.123:0/2587713216 >> 172.21.15.123:0/2587713216 conn(0x55a9fb335130 msgr2=0x55a9fb47ae80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:48:32.212 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:32.211+0000 7f0910094000 1 -- 172.21.15.123:0/2587713216 shutdown_connections 2024-09-16T15:48:32.212 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:32.211+0000 7f0910094000 1 -- 172.21.15.123:0/2587713216 wait complete. 2024-09-16T15:48:32.212 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:32.211+0000 7f0910094000 1 -- start start 2024-09-16T15:48:32.213 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:32.211+0000 7f0910094000 1 Processor -- start 2024-09-16T15:48:32.215 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:32.211+0000 7f0910094000 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a9fb44be10 0x55a9fb4786a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:48:32.216 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:32.211+0000 7f0910094000 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x55a9fb4979d0 0x55a9fb478e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:48:32.216 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:32.211+0000 7f0910094000 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55a9fb4982f0 0x55a9fb4793b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:48:32.216 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:32.211+0000 7f0910094000 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x55a9fb2335d0 con 0x55a9fb4982f0 2024-09-16T15:48:32.216 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:32.211+0000 7f0910094000 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x55a9fb29cd60 con 0x55a9fb44be10 2024-09-16T15:48:32.216 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:32.211+0000 7f0910094000 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x55a9fb23f500 con 0x55a9fb4979d0 2024-09-16T15:48:32.216 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:32.211+0000 7f0910094000 10 client.0 _ll_register_callbacks cb 0x55a9fb243810 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T15:48:32.216 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:32.211+0000 7f0906d9d640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55a9fb4982f0 0x55a9fb4793b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:48:32.216 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:32.211+0000 7f0906d9d640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55a9fb4982f0 0x55a9fb4793b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:47568/0 (socket says 172.21.15.123:47568) 2024-09-16T15:48:32.216 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:32.211+0000 7f0906d9d640 1 -- 172.21.15.123:0/122856791 learned_addr learned my addr 172.21.15.123:0/122856791 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:48:32.216 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:32.211+0000 7f0906d9d640 1 -- 172.21.15.123:0/122856791 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x55a9fb4979d0 msgr2=0x55a9fb478e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T15:48:32.216 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:32.211+0000 7f0906d9d640 1 --2- 172.21.15.123:0/122856791 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x55a9fb4979d0 0x55a9fb478e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:32.216 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:32.211+0000 7f0906d9d640 1 -- 172.21.15.123:0/122856791 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a9fb44be10 msgr2=0x55a9fb4786a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T15:48:32.216 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:32.211+0000 7f0906d9d640 1 --2- 172.21.15.123:0/122856791 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a9fb44be10 0x55a9fb4786a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:32.217 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:32.211+0000 7f0906d9d640 1 -- 172.21.15.123:0/122856791 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a9fb237170 con 0x55a9fb4982f0 2024-09-16T15:48:32.217 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:32.211+0000 7f0906d9d640 1 --2- 172.21.15.123:0/122856791 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55a9fb4982f0 0x55a9fb4793b0 secure :-1 s=READY pgs=1088 cs=0 l=1 rev1=1 crypto rx=0x7f090000bad0 tx=0x7f090000bf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e7487e5b73a11853 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:48:32.217 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:32.211+0000 7f08e57fa640 10 client.0 ms_handle_connect on v2:172.21.15.123:3300/0 2024-09-16T15:48:32.217 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:32.211+0000 7f08e57fa640 1 -- 172.21.15.123:0/122856791 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0900008ba0 con 0x55a9fb4982f0 2024-09-16T15:48:32.217 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:32.211+0000 7f08e57fa640 1 -- 172.21.15.123:0/122856791 <== mon.0 v2:172.21.15.123:3300/0 2 ==== osd_map(103..103 src has 1..103) ==== 5709+0+0 (secure 0 0 0) 0x7f0900008d40 con 0x55a9fb4982f0 2024-09-16T15:48:32.217 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:32.211+0000 7f0910094000 1 -- 172.21.15.123:0/122856791 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55a9fb2310f0 con 0x55a9fb4982f0 2024-09-16T15:48:32.217 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:32.211+0000 7f08e57fa640 1 -- 172.21.15.123:0/122856791 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_get_version_reply(handle=1 version=68) ==== 24+0+0 (secure 0 0 0) 0x7f0900004b30 con 0x55a9fb4982f0 2024-09-16T15:48:32.217 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:32.211+0000 7f0910094000 10 client.7176 fetch_fsmap learned FSMap version 68 2024-09-16T15:48:32.217 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:32.211+0000 7f0910094000 1 -- 172.21.15.123:0/122856791 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,fsmap=68,monmap=2+}) -- 0x55a9fb31def0 con 0x55a9fb4982f0 2024-09-16T15:48:32.217 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:32.211+0000 7f08e57fa640 1 -- 172.21.15.123:0/122856791 <== mon.0 v2:172.21.15.123:3300/0 4 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x55a9fb2310f0 con 0x55a9fb4982f0 2024-09-16T15:48:32.217 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:32.211+0000 7f08e57fa640 1 -- 172.21.15.123:0/122856791 <== mon.0 v2:172.21.15.123:3300/0 5 ==== fsmap(e 68) ==== 3502+0+0 (secure 0 0 0) 0x7f0900026080 con 0x55a9fb4982f0 2024-09-16T15:48:32.217 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:32.211+0000 7f0910094000 10 client.7176 fetch_fsmap finished waiting for FSMap version 68 2024-09-16T15:48:32.217 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:32.211+0000 7f0910094000 10 client.7176 resolve_mds: resolved name 'c' to daemon mds.c 2024-09-16T15:48:32.217 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:32.211+0000 7f0910094000 20 client.7176 populate_metadata read hostname 'smithi123' 2024-09-16T15:48:32.218 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:32.211+0000 7f0910094000 1 --2- 172.21.15.123:0/122856791 >> [v2:172.21.15.123:6834/2302124587,v1:172.21.15.123:6835/2302124587] conn(0x55a9fb47a0f0 0x55a9fb4a9c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:48:32.218 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:32.211+0000 7f0910094000 4 client.7176 mds_command: new command op to 7035 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-16T15:48:32.218 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:32.211+0000 7f0910094000 1 -- 172.21.15.123:0/122856791 --> [v2:172.21.15.123:6834/2302124587,v1:172.21.15.123:6835/2302124587] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a9fb2aba70 con 0x55a9fb47a0f0 2024-09-16T15:48:32.218 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:32.211+0000 7f090659c640 1 --2- 172.21.15.123:0/122856791 >> [v2:172.21.15.123:6834/2302124587,v1:172.21.15.123:6835/2302124587] conn(0x55a9fb47a0f0 0x55a9fb4a9c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:48:32.218 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:32.211+0000 7f090659c640 1 --2- 172.21.15.123:0/122856791 >> [v2:172.21.15.123:6834/2302124587,v1:172.21.15.123:6835/2302124587] conn(0x55a9fb47a0f0 0x55a9fb4a9c90 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d5740c2495c59b7d server_cookie=7260e70c263fab09 in_seq=0 out_seq=0 2024-09-16T15:48:32.218 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:32.211+0000 7f08e57fa640 10 client.7176 ms_handle_connect on v2:172.21.15.123:6834/2302124587 2024-09-16T15:48:32.218 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:32.215+0000 7f08e57fa640 1 -- 172.21.15.123:0/122856791 <== mds.0 v2:172.21.15.123:6834/2302124587 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x55a9fb2aba70 con 0x55a9fb47a0f0 2024-09-16T15:48:32.218 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:32.215+0000 7f08e57fa640 10 client.7176 handle_command_reply: tid=0 2024-09-16T15:48:32.218 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:32.215+0000 7f0910094000 1 client.7176 shutdown 2024-09-16T15:48:32.218 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:32.215+0000 7f0910094000 1 -- 172.21.15.123:0/122856791 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55a9fb4982f0 msgr2=0x55a9fb4793b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:48:32.218 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:32.215+0000 7f0910094000 1 --2- 172.21.15.123:0/122856791 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55a9fb4982f0 0x55a9fb4793b0 secure :-1 s=READY pgs=1088 cs=0 l=1 rev1=1 crypto rx=0x7f090000bad0 tx=0x7f090000bf90 comp rx=0 tx=0).stop 2024-09-16T15:48:32.218 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:32.215+0000 7f0910094000 1 -- 172.21.15.123:0/122856791 shutdown_connections 2024-09-16T15:48:32.218 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:32.215+0000 7f0910094000 1 --2- 172.21.15.123:0/122856791 >> [v2:172.21.15.123:6834/2302124587,v1:172.21.15.123:6835/2302124587] conn(0x55a9fb47a0f0 0x55a9fb4a9c90 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:32.218 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:32.215+0000 7f0910094000 1 --2- 172.21.15.123:0/122856791 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55a9fb4982f0 0x55a9fb4793b0 unknown :-1 s=CLOSED pgs=1088 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:32.219 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:32.215+0000 7f0910094000 1 --2- 172.21.15.123:0/122856791 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x55a9fb4979d0 0x55a9fb478e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:32.219 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:32.215+0000 7f0910094000 1 --2- 172.21.15.123:0/122856791 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a9fb44be10 0x55a9fb4786a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:32.219 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:32.215+0000 7f08e57fa640 0 client.7176 ms_handle_reset on v2:172.21.15.123:6834/2302124587 2024-09-16T15:48:32.219 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:32.215+0000 7f0910094000 1 -- 172.21.15.123:0/122856791 >> 172.21.15.123:0/122856791 conn(0x55a9fb335130 msgr2=0x55a9fb44afd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:48:32.219 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:32.215+0000 7f0910094000 1 -- 172.21.15.123:0/122856791 shutdown_connections 2024-09-16T15:48:32.219 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:32.215+0000 7f0910094000 1 -- 172.21.15.123:0/122856791 wait complete. 2024-09-16T15:48:32.219 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:32.215+0000 7f0910094000 20 client.7176 trim_cache size 0 max 16384 2024-09-16T15:48:32.232 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:32.231+0000 7f0910094000 1 Processor -- start 2024-09-16T15:48:32.232 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:32.231+0000 7f0910094000 1 -- start start 2024-09-16T15:48:32.234 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:32.231+0000 7f0910094000 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x55a9fb44be10 0x55a9fb509bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:48:32.234 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:32.231+0000 7f0910094000 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a9fb4979d0 0x55a9fb50a100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:48:32.234 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:32.231+0000 7f0910094000 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55a9fb4982f0 0x55a9fb50a640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:48:32.234 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:32.231+0000 7f0910094000 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f0908203ea0 con 0x55a9fb4982f0 2024-09-16T15:48:32.234 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:32.231+0000 7f0910094000 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f0908202ec0 con 0x55a9fb4979d0 2024-09-16T15:48:32.234 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:32.231+0000 7f0910094000 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f09082022f0 con 0x55a9fb44be10 2024-09-16T15:48:32.234 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:32.231+0000 7f0906d9d640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55a9fb4982f0 0x55a9fb50a640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:48:32.234 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:32.231+0000 7f0906d9d640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55a9fb4982f0 0x55a9fb50a640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:47576/0 (socket says 172.21.15.123:47576) 2024-09-16T15:48:32.234 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:32.231+0000 7f0906d9d640 1 -- 172.21.15.123:0/1221764329 learned_addr learned my addr 172.21.15.123:0/1221764329 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:48:32.234 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:32.231+0000 7f0906d9d640 1 -- 172.21.15.123:0/1221764329 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x55a9fb44be10 msgr2=0x55a9fb509bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T15:48:32.234 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:32.231+0000 7f0906d9d640 1 --2- 172.21.15.123:0/1221764329 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x55a9fb44be10 0x55a9fb509bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:32.234 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:32.231+0000 7f0906d9d640 1 -- 172.21.15.123:0/1221764329 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a9fb4979d0 msgr2=0x55a9fb50a100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T15:48:32.235 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:32.231+0000 7f0906d9d640 1 --2- 172.21.15.123:0/1221764329 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a9fb4979d0 0x55a9fb50a100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:32.235 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:32.231+0000 7f0906d9d640 1 -- 172.21.15.123:0/1221764329 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a9fb2f4d90 con 0x55a9fb4982f0 2024-09-16T15:48:32.235 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:32.235+0000 7f0906d9d640 1 --2- 172.21.15.123:0/1221764329 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55a9fb4982f0 0x55a9fb50a640 secure :-1 s=READY pgs=1089 cs=0 l=1 rev1=1 crypto rx=0x7f09000082a0 tx=0x7f0900008d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=86265be3e81e910e server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:48:32.235 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:32.235+0000 7f08e77fe640 1 -- 172.21.15.123:0/1221764329 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f090000a620 con 0x55a9fb4982f0 2024-09-16T15:48:32.235 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:32.235+0000 7f08e77fe640 1 -- 172.21.15.123:0/1221764329 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f0900017470 con 0x55a9fb4982f0 2024-09-16T15:48:32.235 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:32.235+0000 7f08e77fe640 1 -- 172.21.15.123:0/1221764329 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09000178a0 con 0x55a9fb4982f0 2024-09-16T15:48:32.239 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:32.235+0000 7f0910094000 1 -- 172.21.15.123:0/1221764329 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55a9fb4982f0 msgr2=0x55a9fb50a640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:48:32.239 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:32.235+0000 7f0910094000 1 --2- 172.21.15.123:0/1221764329 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55a9fb4982f0 0x55a9fb50a640 secure :-1 s=READY pgs=1089 cs=0 l=1 rev1=1 crypto rx=0x7f09000082a0 tx=0x7f0900008d50 comp rx=0 tx=0).stop 2024-09-16T15:48:32.240 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:32.235+0000 7f0910094000 1 -- 172.21.15.123:0/1221764329 shutdown_connections 2024-09-16T15:48:32.240 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:32.235+0000 7f0910094000 1 --2- 172.21.15.123:0/1221764329 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55a9fb4982f0 0x55a9fb50a640 unknown :-1 s=CLOSED pgs=1089 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:32.240 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:32.235+0000 7f0910094000 1 --2- 172.21.15.123:0/1221764329 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a9fb4979d0 0x55a9fb50a100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:32.240 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:32.235+0000 7f0910094000 1 --2- 172.21.15.123:0/1221764329 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x55a9fb44be10 0x55a9fb509bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:32.240 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:32.235+0000 7f0910094000 1 -- 172.21.15.123:0/1221764329 >> 172.21.15.123:0/1221764329 conn(0x55a9fb335130 msgr2=0x55a9fb4a4fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:48:32.240 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:32.235+0000 7f0910094000 1 -- 172.21.15.123:0/1221764329 shutdown_connections 2024-09-16T15:48:32.240 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:32.235+0000 7f0910094000 1 -- 172.21.15.123:0/1221764329 wait complete. 2024-09-16T15:48:32.240 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:32.235+0000 7f0910094000 1 -- start start 2024-09-16T15:48:32.240 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:32.235+0000 7f0910094000 1 Processor -- start 2024-09-16T15:48:32.240 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:32.235+0000 7f0910094000 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55a9fb44be10 0x55a9fb50de10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:48:32.240 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:32.235+0000 7f0910094000 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x55a9fb4979d0 0x55a9fb50e350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:48:32.240 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:32.235+0000 7f0910094000 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a9fb4982f0 0x55a9fb50e890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:48:32.240 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:32.235+0000 7f0910094000 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f0908202870 con 0x55a9fb44be10 2024-09-16T15:48:32.241 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:32.235+0000 7f0910094000 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x55a9fb4aece0 con 0x55a9fb4982f0 2024-09-16T15:48:32.241 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:32.235+0000 7f0910094000 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x55a9fb358730 con 0x55a9fb4979d0 2024-09-16T15:48:32.241 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:32.235+0000 7f0910094000 10 client.0 _ll_register_callbacks cb 0x7f090000a9e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T15:48:32.241 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:32.235+0000 7f090659c640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55a9fb44be10 0x55a9fb50de10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:48:32.241 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:32.235+0000 7f090659c640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55a9fb44be10 0x55a9fb50de10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:47586/0 (socket says 172.21.15.123:47586) 2024-09-16T15:48:32.241 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:32.235+0000 7f090659c640 1 -- 172.21.15.123:0/1990312854 learned_addr learned my addr 172.21.15.123:0/1990312854 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:48:32.241 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:32.235+0000 7f090659c640 1 -- 172.21.15.123:0/1990312854 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x55a9fb4979d0 msgr2=0x55a9fb50e350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T15:48:32.241 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:32.235+0000 7f090659c640 1 --2- 172.21.15.123:0/1990312854 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x55a9fb4979d0 0x55a9fb50e350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:32.241 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:32.235+0000 7f090659c640 1 -- 172.21.15.123:0/1990312854 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a9fb4982f0 msgr2=0x55a9fb50e890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T15:48:32.241 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:32.235+0000 7f090659c640 1 --2- 172.21.15.123:0/1990312854 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a9fb4982f0 0x55a9fb50e890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:32.241 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:32.235+0000 7f090659c640 1 -- 172.21.15.123:0/1990312854 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a9fb25fa90 con 0x55a9fb44be10 2024-09-16T15:48:32.241 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:32.235+0000 7f090659c640 1 --2- 172.21.15.123:0/1990312854 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55a9fb44be10 0x55a9fb50de10 secure :-1 s=READY pgs=1090 cs=0 l=1 rev1=1 crypto rx=0x7f08f80916e0 tx=0x7f08f805d7d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=96baa858961c593c server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:48:32.241 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:32.235+0000 7f08e57fa640 10 client.0 ms_handle_connect on v2:172.21.15.123:3300/0 2024-09-16T15:48:32.241 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:32.235+0000 7f08e57fa640 1 -- 172.21.15.123:0/1990312854 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08f8001190 con 0x55a9fb44be10 2024-09-16T15:48:32.242 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:32.235+0000 7f08e57fa640 1 -- 172.21.15.123:0/1990312854 <== mon.0 v2:172.21.15.123:3300/0 2 ==== osd_map(103..103 src has 1..103) ==== 5709+0+0 (secure 0 0 0) 0x7f08f8059f30 con 0x55a9fb44be10 2024-09-16T15:48:32.242 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:32.235+0000 7f0910094000 1 -- 172.21.15.123:0/1990312854 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0900017470 con 0x55a9fb44be10 2024-09-16T15:48:32.242 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:32.235+0000 7f08e57fa640 1 -- 172.21.15.123:0/1990312854 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_get_version_reply(handle=1 version=68) ==== 24+0+0 (secure 0 0 0) 0x7f08f805dcc0 con 0x55a9fb44be10 2024-09-16T15:48:32.242 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:32.235+0000 7f0910094000 10 client.7182 fetch_fsmap learned FSMap version 68 2024-09-16T15:48:32.242 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:32.235+0000 7f0910094000 1 -- 172.21.15.123:0/1990312854 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,fsmap=68,monmap=2+}) -- 0x55a9fb23e050 con 0x55a9fb44be10 2024-09-16T15:48:32.242 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:32.235+0000 7f08e57fa640 1 -- 172.21.15.123:0/1990312854 <== mon.0 v2:172.21.15.123:3300/0 4 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f0900017470 con 0x55a9fb44be10 2024-09-16T15:48:32.242 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:32.235+0000 7f08e57fa640 1 -- 172.21.15.123:0/1990312854 <== mon.0 v2:172.21.15.123:3300/0 5 ==== fsmap(e 68) ==== 3502+0+0 (secure 0 0 0) 0x7f08f807f0b0 con 0x55a9fb44be10 2024-09-16T15:48:32.242 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:32.235+0000 7f0910094000 10 client.7182 fetch_fsmap finished waiting for FSMap version 68 2024-09-16T15:48:32.242 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:32.235+0000 7f0910094000 10 client.7182 resolve_mds: resolved name 'c' to daemon mds.c 2024-09-16T15:48:32.242 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:32.235+0000 7f0910094000 20 client.7182 populate_metadata read hostname 'smithi123' 2024-09-16T15:48:32.242 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:32.235+0000 7f0910094000 1 --2- 172.21.15.123:0/1990312854 >> [v2:172.21.15.123:6834/2302124587,v1:172.21.15.123:6835/2302124587] conn(0x55a9fb4abbb0 0x55a9fb4abf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:48:32.242 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:32.235+0000 7f0910094000 4 client.7182 mds_command: new command op to 7035 tid=0 multi_id=0 [{"prefix": "status"}] 2024-09-16T15:48:32.242 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:32.235+0000 7f0910094000 1 -- 172.21.15.123:0/1990312854 --> [v2:172.21.15.123:6834/2302124587,v1:172.21.15.123:6835/2302124587] -- command(tid 0: {"prefix": "status"}) -- 0x55a9fb2a7520 con 0x55a9fb4abbb0 2024-09-16T15:48:32.242 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:32.235+0000 7f08fffff640 1 --2- 172.21.15.123:0/1990312854 >> [v2:172.21.15.123:6834/2302124587,v1:172.21.15.123:6835/2302124587] conn(0x55a9fb4abbb0 0x55a9fb4abf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:48:32.242 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:32.235+0000 7f08fffff640 1 --2- 172.21.15.123:0/1990312854 >> [v2:172.21.15.123:6834/2302124587,v1:172.21.15.123:6835/2302124587] conn(0x55a9fb4abbb0 0x55a9fb4abf90 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ff32147664f665a8 server_cookie=eb171aa94755f321 in_seq=0 out_seq=0 2024-09-16T15:48:32.243 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:32.235+0000 7f08e57fa640 10 client.7182 ms_handle_connect on v2:172.21.15.123:6834/2302124587 2024-09-16T15:48:32.243 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:32.239+0000 7f08e57fa640 1 -- 172.21.15.123:0/1990312854 <== mds.0 v2:172.21.15.123:6834/2302124587 1 ==== command_reply(tid 0: 0 ) ==== 8+0+326 (crc 0 0 0) 0x55a9fb2a7520 con 0x55a9fb4abbb0 2024-09-16T15:48:32.243 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:32.239+0000 7f08e57fa640 10 client.7182 handle_command_reply: tid=0 2024-09-16T15:48:32.243 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:32.239+0000 7f0910094000 1 client.7182 shutdown 2024-09-16T15:48:32.243 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:32.239+0000 7f0910094000 1 -- 172.21.15.123:0/1990312854 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55a9fb44be10 msgr2=0x55a9fb50de10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:48:32.243 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:32.239+0000 7f0910094000 1 --2- 172.21.15.123:0/1990312854 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55a9fb44be10 0x55a9fb50de10 secure :-1 s=READY pgs=1090 cs=0 l=1 rev1=1 crypto rx=0x7f08f80916e0 tx=0x7f08f805d7d0 comp rx=0 tx=0).stop 2024-09-16T15:48:32.243 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:32.239+0000 7f0910094000 1 -- 172.21.15.123:0/1990312854 shutdown_connections 2024-09-16T15:48:32.243 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:32.239+0000 7f0910094000 1 --2- 172.21.15.123:0/1990312854 >> [v2:172.21.15.123:6834/2302124587,v1:172.21.15.123:6835/2302124587] conn(0x55a9fb4abbb0 0x55a9fb4abf90 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:32.243 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:32.239+0000 7f0910094000 1 --2- 172.21.15.123:0/1990312854 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55a9fb4982f0 0x55a9fb50e890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:32.243 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:32.239+0000 7f08e57fa640 0 client.7182 ms_handle_reset on v2:172.21.15.123:6834/2302124587 2024-09-16T15:48:32.243 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:32.239+0000 7f0910094000 1 --2- 172.21.15.123:0/1990312854 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x55a9fb4979d0 0x55a9fb50e350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:32.243 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:32.239+0000 7f0910094000 1 --2- 172.21.15.123:0/1990312854 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55a9fb44be10 0x55a9fb50de10 unknown :-1 s=CLOSED pgs=1090 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:32.243 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:32.239+0000 7f0910094000 1 -- 172.21.15.123:0/1990312854 >> 172.21.15.123:0/1990312854 conn(0x55a9fb335130 msgr2=0x55a9fb453b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:48:32.243 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:32.239+0000 7f0910094000 1 -- 172.21.15.123:0/1990312854 shutdown_connections 2024-09-16T15:48:32.243 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:32.239+0000 7f0910094000 1 -- 172.21.15.123:0/1990312854 wait complete. 2024-09-16T15:48:32.244 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:32.239+0000 7f0910094000 20 client.7182 trim_cache size 0 max 16384 2024-09-16T15:48:32.244 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2024-09-16T15:48:32.244 INFO:teuthology.orchestra.run.smithi123.stdout: "cluster_fsid": "56077f89-18c8-486f-ae7e-eecc0bb7822c", 2024-09-16T15:48:32.244 INFO:teuthology.orchestra.run.smithi123.stdout: "whoami": 0, 2024-09-16T15:48:32.244 INFO:teuthology.orchestra.run.smithi123.stdout: "id": 7035, 2024-09-16T15:48:32.244 INFO:teuthology.orchestra.run.smithi123.stdout: "want_state": "up:active", 2024-09-16T15:48:32.244 INFO:teuthology.orchestra.run.smithi123.stdout: "state": "up:active", 2024-09-16T15:48:32.244 INFO:teuthology.orchestra.run.smithi123.stdout: "fs_name": "cephfs", 2024-09-16T15:48:32.244 INFO:teuthology.orchestra.run.smithi123.stdout: "rank_uptime": 5.7658535300000002, 2024-09-16T15:48:32.244 INFO:teuthology.orchestra.run.smithi123.stdout: "mdsmap_epoch": 68, 2024-09-16T15:48:32.245 INFO:teuthology.orchestra.run.smithi123.stdout: "osdmap_epoch": 103, 2024-09-16T15:48:32.245 INFO:teuthology.orchestra.run.smithi123.stdout: "osdmap_epoch_barrier": 103, 2024-09-16T15:48:32.245 INFO:teuthology.orchestra.run.smithi123.stdout: "uptime": 10.45412992 2024-09-16T15:48:32.245 INFO:teuthology.orchestra.run.smithi123.stdout:} 2024-09-16T15:48:32.245 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:32.239+0000 7f090cfaa640 1 -- 172.21.15.123:0/1275143563 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f08e8072fe0 msgr2=0x7f08e80754a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:48:32.246 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:32.239+0000 7f090cfaa640 1 --2- 172.21.15.123:0/1275143563 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f08e8072fe0 0x7f08e80754a0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f08f00b1000 tx=0x7f08f001ccd0 comp rx=0 tx=0).stop 2024-09-16T15:48:32.246 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:32.239+0000 7f090cfaa640 1 -- 172.21.15.123:0/1275143563 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f09080cf5a0 msgr2=0x7f090815a200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:48:32.246 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:32.239+0000 7f090cfaa640 1 --2- 172.21.15.123:0/1275143563 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f09080cf5a0 0x7f090815a200 secure :-1 s=READY pgs=1086 cs=0 l=1 rev1=1 crypto rx=0x7f08f800c530 tx=0x7f08f800c9f0 comp rx=0 tx=0).stop 2024-09-16T15:48:32.246 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:32.239+0000 7f090cfaa640 1 -- 172.21.15.123:0/1275143563 shutdown_connections 2024-09-16T15:48:32.246 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:32.239+0000 7f090cfaa640 1 --2- 172.21.15.123:0/1275143563 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f08e8072fe0 0x7f08e80754a0 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:32.246 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:32.239+0000 7f090cfaa640 1 --2- 172.21.15.123:0/1275143563 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f090815ec40 0x7f09080734b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:32.246 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:32.239+0000 7f090cfaa640 1 --2- 172.21.15.123:0/1275143563 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f0908158120 0x7f0908072710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:32.246 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:32.239+0000 7f090cfaa640 1 --2- 172.21.15.123:0/1275143563 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f09080cf5a0 0x7f090815a200 unknown :-1 s=CLOSED pgs=1086 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:32.246 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:32.239+0000 7f090cfaa640 1 -- 172.21.15.123:0/1275143563 >> 172.21.15.123:0/1275143563 conn(0x7f09080cfb10 msgr2=0x7f09080cc520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:48:32.246 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:32.239+0000 7f090cfaa640 1 -- 172.21.15.123:0/1275143563 shutdown_connections 2024-09-16T15:48:32.246 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:32.239+0000 7f090cfaa640 1 -- 172.21.15.123:0/1275143563 wait complete. 2024-09-16T15:48:32.254 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-09-16T15:48:32.358 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:32.359+0000 7f77ec035640 1 -- 172.21.15.123:0/1992925764 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f77d0004810 msgr2=0x7f77d0004c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:48:32.359 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:32.359+0000 7f77ec035640 1 --2- 172.21.15.123:0/1992925764 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f77d0004810 0x7f77d0004c90 secure :-1 s=READY pgs=1092 cs=0 l=1 rev1=1 crypto rx=0x7f77d400bb10 tx=0x7f77d401cd30 comp rx=0 tx=0).stop 2024-09-16T15:48:32.359 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:32.359+0000 7f77ea5d2640 1 -- 172.21.15.123:0/1992925764 reap_dead start 2024-09-16T15:48:32.359 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:32.359+0000 7f77ec035640 1 -- 172.21.15.123:0/1992925764 shutdown_connections 2024-09-16T15:48:32.359 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:32.359+0000 7f77ec035640 1 -- 172.21.15.123:0/1992925764 >> 172.21.15.123:0/1992925764 conn(0x7f77e40d3ea0 msgr2=0x7f77e40d42b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:48:32.359 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:32.359+0000 7f77ec035640 1 -- 172.21.15.123:0/1992925764 shutdown_connections 2024-09-16T15:48:32.359 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:32.359+0000 7f77ec035640 1 -- 172.21.15.123:0/1992925764 wait complete. 2024-09-16T15:48:32.359 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:32.359+0000 7f77ec035640 1 Processor -- start 2024-09-16T15:48:32.360 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:32.359+0000 7f77ec035640 1 -- start start 2024-09-16T15:48:32.361 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:32.359+0000 7f77ec035640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f77d0004810 0x7f77e4162ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:48:32.362 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:32.359+0000 7f77ec035640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f77e415b270 0x7f77e40ca6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:48:32.362 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:32.359+0000 7f77ec035640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f77e415e220 0x7f77e415e6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:48:32.362 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:32.359+0000 7f77ec035640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f77e40b19d0 con 0x7f77e415e220 2024-09-16T15:48:32.362 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:32.359+0000 7f77ec035640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f77e40b1850 con 0x7f77e415b270 2024-09-16T15:48:32.362 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:32.359+0000 7f77ec035640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f77e40b1b50 con 0x7f77d0004810 2024-09-16T15:48:32.362 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:32.359+0000 7f77ea5d2640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f77e415e220 0x7f77e415e6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:48:32.362 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:32.359+0000 7f77ea5d2640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f77e415e220 0x7f77e415e6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:47600/0 (socket says 172.21.15.123:47600) 2024-09-16T15:48:32.362 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:32.359+0000 7f77ea5d2640 1 -- 172.21.15.123:0/3664346285 learned_addr learned my addr 172.21.15.123:0/3664346285 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:48:32.362 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:32.359+0000 7f77e95d0640 1 --2- 172.21.15.123:0/3664346285 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f77e415b270 0x7f77e40ca6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:48:32.362 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:32.359+0000 7f77e9dd1640 1 --2- 172.21.15.123:0/3664346285 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f77d0004810 0x7f77e4162ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:48:32.362 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:32.359+0000 7f77ea5d2640 1 -- 172.21.15.123:0/3664346285 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f77d0004810 msgr2=0x7f77e4162ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:48:32.362 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:32.359+0000 7f77ea5d2640 1 --2- 172.21.15.123:0/3664346285 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f77d0004810 0x7f77e4162ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:32.362 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:32.359+0000 7f77ea5d2640 1 -- 172.21.15.123:0/3664346285 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f77e415b270 msgr2=0x7f77e40ca6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:48:32.363 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:32.359+0000 7f77ea5d2640 1 --2- 172.21.15.123:0/3664346285 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f77e415b270 0x7f77e40ca6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:32.363 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:32.359+0000 7f77ea5d2640 1 -- 172.21.15.123:0/3664346285 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f77e41faea0 con 0x7f77e415e220 2024-09-16T15:48:32.363 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:32.359+0000 7f77ea5d2640 1 --2- 172.21.15.123:0/3664346285 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f77e415e220 0x7f77e415e6d0 secure :-1 s=READY pgs=1093 cs=0 l=1 rev1=1 crypto rx=0x7f77e40b5870 tx=0x7f77dc00cd80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:48:32.363 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:32.359+0000 7f77da7fc640 1 -- 172.21.15.123:0/3664346285 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77dc007b80 con 0x7f77e415e220 2024-09-16T15:48:32.363 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:32.359+0000 7f77da7fc640 1 -- 172.21.15.123:0/3664346285 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f77dc017070 con 0x7f77e415e220 2024-09-16T15:48:32.363 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:32.359+0000 7f77da7fc640 1 -- 172.21.15.123:0/3664346285 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77dc00f6b0 con 0x7f77e415e220 2024-09-16T15:48:32.363 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:32.359+0000 7f77ec035640 1 -- 172.21.15.123:0/3664346285 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f77e41fb180 con 0x7f77e415e220 2024-09-16T15:48:32.363 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:32.359+0000 7f77ec035640 1 -- 172.21.15.123:0/3664346285 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f77e41fb740 con 0x7f77e415e220 2024-09-16T15:48:32.364 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:32.363+0000 7f77ec035640 1 -- 172.21.15.123:0/3664346285 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f77e40b0ee0 con 0x7f77e415e220 2024-09-16T15:48:32.364 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:32.363+0000 7f77da7fc640 1 -- 172.21.15.123:0/3664346285 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 16) ==== 97182+0+0 (secure 0 0 0) 0x7f77dc00f850 con 0x7f77e415e220 2024-09-16T15:48:32.364 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:32.363+0000 7f77da7fc640 1 --2- 172.21.15.123:0/3664346285 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f77c8072fe0 0x7f77c80754a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:48:32.364 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:32.363+0000 7f77da7fc640 1 -- 172.21.15.123:0/3664346285 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(103..103 src has 1..103) ==== 5709+0+0 (secure 0 0 0) 0x7f77dc094cc0 con 0x7f77e415e220 2024-09-16T15:48:32.364 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:32.363+0000 7f77e9dd1640 1 --2- 172.21.15.123:0/3664346285 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f77c8072fe0 0x7f77c80754a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:48:32.364 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:32.363+0000 7f77e9dd1640 1 --2- 172.21.15.123:0/3664346285 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f77c8072fe0 0x7f77c80754a0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f77e0001ec0 tx=0x7f77e0000fd0 comp rx=0 tx=0).ready entity=mgr.4150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:48:32.367 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:32.367+0000 7f77da7fc640 1 -- 172.21.15.123:0/3664346285 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f77dc05fdd0 con 0x7f77e415e220 2024-09-16T15:48:32.398 INFO:tasks.ceph.mds.c.smithi123.stderr:2024-09-16T15:48:32.399+0000 7fdd124eb640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-09-16T15:48:32.573 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:32.571+0000 7f77ec035640 1 -- 172.21.15.123:0/3664346285 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f77e415b9b0 con 0x7f77e415e220 2024-09-16T15:48:32.573 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:32.575+0000 7f77da7fc640 1 -- 172.21.15.123:0/3664346285 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v68) ==== 53+0+73 (secure 0 0 0) 0x7f77dc05f6d0 con 0x7f77e415e220 2024-09-16T15:48:32.574 INFO:teuthology.orchestra.run.smithi123.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-09-16T15:48:32.576 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:32.575+0000 7f77ec035640 1 -- 172.21.15.123:0/3664346285 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f77c8072fe0 msgr2=0x7f77c80754a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:48:32.576 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:32.575+0000 7f77ec035640 1 --2- 172.21.15.123:0/3664346285 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f77c8072fe0 0x7f77c80754a0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f77e0001ec0 tx=0x7f77e0000fd0 comp rx=0 tx=0).stop 2024-09-16T15:48:32.576 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:32.575+0000 7f77ec035640 1 -- 172.21.15.123:0/3664346285 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f77e415e220 msgr2=0x7f77e415e6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:48:32.576 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:32.575+0000 7f77ec035640 1 --2- 172.21.15.123:0/3664346285 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f77e415e220 0x7f77e415e6d0 secure :-1 s=READY pgs=1093 cs=0 l=1 rev1=1 crypto rx=0x7f77e40b5870 tx=0x7f77dc00cd80 comp rx=0 tx=0).stop 2024-09-16T15:48:32.576 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:32.575+0000 7f77ec035640 1 -- 172.21.15.123:0/3664346285 shutdown_connections 2024-09-16T15:48:32.576 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:32.575+0000 7f77ec035640 1 --2- 172.21.15.123:0/3664346285 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f77c8072fe0 0x7f77c80754a0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:32.576 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:32.575+0000 7f77ec035640 1 --2- 172.21.15.123:0/3664346285 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f77e415e220 0x7f77e415e6d0 unknown :-1 s=CLOSED pgs=1093 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:32.577 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:32.575+0000 7f77ec035640 1 --2- 172.21.15.123:0/3664346285 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f77e415b270 0x7f77e40ca6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:32.577 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:32.575+0000 7f77ec035640 1 --2- 172.21.15.123:0/3664346285 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f77d0004810 0x7f77e4162ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:32.577 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:32.575+0000 7f77ec035640 1 -- 172.21.15.123:0/3664346285 >> 172.21.15.123:0/3664346285 conn(0x7f77e40d3ea0 msgr2=0x7f77e406d2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:48:32.577 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:32.575+0000 7f77ec035640 1 -- 172.21.15.123:0/3664346285 shutdown_connections 2024-09-16T15:48:32.577 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:32.575+0000 7f77ec035640 1 -- 172.21.15.123:0/3664346285 wait complete. 2024-09-16T15:48:32.591 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-09-16T15:48:32.591 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-09-16T15:48:32.591 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi123.front.sepia.ceph.com 2024-09-16T15:48:32.591 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-09-16T15:48:32.592 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-09-16T15:48:32.592 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-09-16T15:48:32.592 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-09-16T15:48:32.592 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-09-16T15:48:32.592 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-09-16T15:48:32.592 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:48:32.592 DEBUG:teuthology.orchestra.run.smithi123:> ip addr 2024-09-16T15:48:32.598 INFO:teuthology.orchestra.run.smithi123.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-09-16T15:48:32.598 INFO:teuthology.orchestra.run.smithi123.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-09-16T15:48:32.598 INFO:teuthology.orchestra.run.smithi123.stdout: inet 127.0.0.1/8 scope host lo 2024-09-16T15:48:32.598 INFO:teuthology.orchestra.run.smithi123.stdout: valid_lft forever preferred_lft forever 2024-09-16T15:48:32.598 INFO:teuthology.orchestra.run.smithi123.stdout: inet6 ::1/128 scope host 2024-09-16T15:48:32.598 INFO:teuthology.orchestra.run.smithi123.stdout: valid_lft forever preferred_lft forever 2024-09-16T15:48:32.599 INFO:teuthology.orchestra.run.smithi123.stdout:2: eno1: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-09-16T15:48:32.599 INFO:teuthology.orchestra.run.smithi123.stdout: link/ether 0c:c4:7a:d8:da:6c brd ff:ff:ff:ff:ff:ff 2024-09-16T15:48:32.599 INFO:teuthology.orchestra.run.smithi123.stdout: altname enp7s0f0 2024-09-16T15:48:32.599 INFO:teuthology.orchestra.run.smithi123.stdout:3: eno2: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-09-16T15:48:32.599 INFO:teuthology.orchestra.run.smithi123.stdout: link/ether 0c:c4:7a:d8:da:6d brd ff:ff:ff:ff:ff:ff 2024-09-16T15:48:32.599 INFO:teuthology.orchestra.run.smithi123.stdout: altname enp7s0f1 2024-09-16T15:48:32.599 INFO:teuthology.orchestra.run.smithi123.stdout:4: enp3s0f0: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-09-16T15:48:32.599 INFO:teuthology.orchestra.run.smithi123.stdout: link/ether 0c:c4:7a:8f:d3:f0 brd ff:ff:ff:ff:ff:ff 2024-09-16T15:48:32.599 INFO:teuthology.orchestra.run.smithi123.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-09-16T15:48:32.599 INFO:teuthology.orchestra.run.smithi123.stdout: link/ether 0c:c4:7a:8f:d3:f1 brd ff:ff:ff:ff:ff:ff 2024-09-16T15:48:32.599 INFO:teuthology.orchestra.run.smithi123.stdout: inet 172.21.15.123/20 metric 100 brd 172.21.15.255 scope global dynamic enp3s0f1 2024-09-16T15:48:32.599 INFO:teuthology.orchestra.run.smithi123.stdout: valid_lft 42156sec preferred_lft 42156sec 2024-09-16T15:48:32.599 INFO:teuthology.orchestra.run.smithi123.stdout: inet6 fe80::ec4:7aff:fe8f:d3f1/64 scope link 2024-09-16T15:48:32.600 INFO:teuthology.orchestra.run.smithi123.stdout: valid_lft forever preferred_lft forever 2024-09-16T15:48:32.600 INFO:teuthology.orchestra.run.smithi123.stdout:6: docker0: mtu 1500 qdisc noqueue state DOWN group default 2024-09-16T15:48:32.600 INFO:teuthology.orchestra.run.smithi123.stdout: link/ether 02:42:dd:82:a2:6e brd ff:ff:ff:ff:ff:ff 2024-09-16T15:48:32.600 INFO:teuthology.orchestra.run.smithi123.stdout: inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0 2024-09-16T15:48:32.600 INFO:teuthology.orchestra.run.smithi123.stdout: valid_lft forever preferred_lft forever 2024-09-16T15:48:32.600 INFO:teuthology.orchestra.run.smithi123.stdout:7: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-09-16T15:48:32.600 INFO:teuthology.orchestra.run.smithi123.stdout: link/ether 5e:1f:84:6d:79:ae brd ff:ff:ff:ff:ff:ff 2024-09-16T15:48:32.600 INFO:teuthology.orchestra.run.smithi123.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-09-16T15:48:32.600 INFO:teuthology.orchestra.run.smithi123.stdout: valid_lft forever preferred_lft forever 2024-09-16T15:48:32.600 INFO:teuthology.orchestra.run.smithi123.stdout: inet6 fe80::5c1f:84ff:fe6d:79ae/64 scope link 2024-09-16T15:48:32.600 INFO:teuthology.orchestra.run.smithi123.stdout: valid_lft forever preferred_lft forever 2024-09-16T15:48:32.600 INFO:teuthology.orchestra.run.smithi123.stdout:8: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-09-16T15:48:32.601 INFO:teuthology.orchestra.run.smithi123.stdout: link/ether 0e:5c:f6:1f:82:2f brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-09-16T15:48:32.601 INFO:teuthology.orchestra.run.smithi123.stdout: inet6 fe80::c5c:f6ff:fe1f:822f/64 scope link 2024-09-16T15:48:32.601 INFO:teuthology.orchestra.run.smithi123.stdout: valid_lft forever preferred_lft forever 2024-09-16T15:48:32.601 INFO:teuthology.orchestra.run.smithi123.stdout:9: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-09-16T15:48:32.601 INFO:teuthology.orchestra.run.smithi123.stdout: link/ether c6:a6:3d:69:d0:db brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-09-16T15:48:32.601 INFO:teuthology.orchestra.run.smithi123.stdout: inet6 fe80::c4a6:3dff:fe69:d0db/64 scope link 2024-09-16T15:48:32.601 INFO:teuthology.orchestra.run.smithi123.stdout: valid_lft forever preferred_lft forever 2024-09-16T15:48:32.601 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:48:32.601 DEBUG:teuthology.orchestra.run.smithi123:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-09-16T15:48:32.652 INFO:teuthology.orchestra.run.smithi123.stdout:1 2024-09-16T15:48:32.653 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:48:32.653 DEBUG:teuthology.orchestra.run.smithi123:> route 2024-09-16T15:48:32.705 INFO:teuthology.orchestra.run.smithi123.stdout:Kernel IP routing table 2024-09-16T15:48:32.706 INFO:teuthology.orchestra.run.smithi123.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-09-16T15:48:32.706 INFO:teuthology.orchestra.run.smithi123.stdout:default _gateway 0.0.0.0 UG 0 0 0 enp3s0f1 2024-09-16T15:48:32.706 INFO:teuthology.orchestra.run.smithi123.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-09-16T15:48:32.706 INFO:teuthology.orchestra.run.smithi123.stdout:158.69.67.47 _gateway 255.255.255.255 UGH 100 0 0 enp3s0f1 2024-09-16T15:48:32.706 INFO:teuthology.orchestra.run.smithi123.stdout:172.17.0.0 0.0.0.0 255.255.0.0 U 0 0 0 docker0 2024-09-16T15:48:32.706 INFO:teuthology.orchestra.run.smithi123.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-09-16T15:48:32.706 INFO:teuthology.orchestra.run.smithi123.stdout:ns1.front.sepia 0.0.0.0 255.255.255.255 UH 100 0 0 enp3s0f1 2024-09-16T15:48:32.706 INFO:teuthology.orchestra.run.smithi123.stdout:172.21.0.2 0.0.0.0 255.255.255.255 UH 100 0 0 enp3s0f1 2024-09-16T15:48:32.706 INFO:teuthology.orchestra.run.smithi123.stdout:_gateway 0.0.0.0 255.255.255.255 UH 100 0 0 enp3s0f1 2024-09-16T15:48:32.706 INFO:teuthology.orchestra.run.smithi123.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-09-16T15:48:32.707 DEBUG:teuthology.orchestra.run.smithi123:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-09-16T15:48:32.707 DEBUG:teuthology.orchestra.run.smithi123:> set -e 2024-09-16T15:48:32.707 DEBUG:teuthology.orchestra.run.smithi123:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-09-16T15:48:32.707 DEBUG:teuthology.orchestra.run.smithi123:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-09-16T15:48:32.707 DEBUG:teuthology.orchestra.run.smithi123:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-09-16T15:48:32.707 DEBUG:teuthology.orchestra.run.smithi123:> ') 2024-09-16T15:48:32.793 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:32 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-09-16T15:48:32.820 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:32 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-16T15:48:32.827 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:48:32.827 DEBUG:teuthology.orchestra.run.smithi123:> ip netns list 2024-09-16T15:48:32.874 INFO:teuthology.orchestra.run.smithi123.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-09-16T15:48:32.874 INFO:teuthology.orchestra.run.smithi123.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-09-16T15:48:32.875 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-09-16T15:48:32.875 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:48:32.875 DEBUG:teuthology.orchestra.run.smithi123:> sudo ip link set brx.0 up 2024-09-16T15:48:32.930 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2024-09-16T15:48:32.930 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-09-16T15:48:32.930 DEBUG:teuthology.orchestra.run.smithi123:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-09-16T15:48:32.978 INFO:teuthology.orchestra.run.smithi123.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-09-16T15:48:32.979 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-09-16T15:48:32.979 DEBUG:teuthology.orchestra.run.smithi123:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-09-16T15:48:33.026 DEBUG:teuthology.orchestra.run.smithi123:> sudo modprobe fuse 2024-09-16T15:48:33.083 DEBUG:teuthology.orchestra.run.smithi123:> cat /proc/self/mounts | awk '{print $2}' 2024-09-16T15:48:33.131 INFO:teuthology.orchestra.run.smithi123.stdout:/sys 2024-09-16T15:48:33.131 INFO:teuthology.orchestra.run.smithi123.stdout:/proc 2024-09-16T15:48:33.131 INFO:teuthology.orchestra.run.smithi123.stdout:/dev 2024-09-16T15:48:33.131 INFO:teuthology.orchestra.run.smithi123.stdout:/dev/pts 2024-09-16T15:48:33.131 INFO:teuthology.orchestra.run.smithi123.stdout:/run 2024-09-16T15:48:33.131 INFO:teuthology.orchestra.run.smithi123.stdout:/ 2024-09-16T15:48:33.131 INFO:teuthology.orchestra.run.smithi123.stdout:/sys/kernel/security 2024-09-16T15:48:33.132 INFO:teuthology.orchestra.run.smithi123.stdout:/dev/shm 2024-09-16T15:48:33.132 INFO:teuthology.orchestra.run.smithi123.stdout:/run/lock 2024-09-16T15:48:33.132 INFO:teuthology.orchestra.run.smithi123.stdout:/sys/fs/cgroup 2024-09-16T15:48:33.132 INFO:teuthology.orchestra.run.smithi123.stdout:/sys/fs/pstore 2024-09-16T15:48:33.132 INFO:teuthology.orchestra.run.smithi123.stdout:/sys/fs/bpf 2024-09-16T15:48:33.132 INFO:teuthology.orchestra.run.smithi123.stdout:/proc/sys/fs/binfmt_misc 2024-09-16T15:48:33.132 INFO:teuthology.orchestra.run.smithi123.stdout:/dev/hugepages 2024-09-16T15:48:33.132 INFO:teuthology.orchestra.run.smithi123.stdout:/dev/mqueue 2024-09-16T15:48:33.132 INFO:teuthology.orchestra.run.smithi123.stdout:/sys/kernel/debug 2024-09-16T15:48:33.132 INFO:teuthology.orchestra.run.smithi123.stdout:/sys/kernel/tracing 2024-09-16T15:48:33.132 INFO:teuthology.orchestra.run.smithi123.stdout:/proc/fs/nfsd 2024-09-16T15:48:33.132 INFO:teuthology.orchestra.run.smithi123.stdout:/sys/fs/fuse/connections 2024-09-16T15:48:33.132 INFO:teuthology.orchestra.run.smithi123.stdout:/sys/kernel/config 2024-09-16T15:48:33.132 INFO:teuthology.orchestra.run.smithi123.stdout:/run/credentials/systemd-sysusers.service 2024-09-16T15:48:33.133 INFO:teuthology.orchestra.run.smithi123.stdout:/run/rpc_pipefs 2024-09-16T15:48:33.133 INFO:teuthology.orchestra.run.smithi123.stdout:/run/user/1000 2024-09-16T15:48:33.133 INFO:teuthology.orchestra.run.smithi123.stdout:/var/lib/ceph 2024-09-16T15:48:33.133 INFO:teuthology.orchestra.run.smithi123.stdout:/var/lib/ceph/osd/ceph-0 2024-09-16T15:48:33.133 INFO:teuthology.orchestra.run.smithi123.stdout:/var/lib/ceph/osd/ceph-1 2024-09-16T15:48:33.133 INFO:teuthology.orchestra.run.smithi123.stdout:/var/lib/ceph/osd/ceph-2 2024-09-16T15:48:33.133 INFO:teuthology.orchestra.run.smithi123.stdout:/var/lib/ceph/osd/ceph-3 2024-09-16T15:48:33.133 INFO:teuthology.orchestra.run.smithi123.stdout:/run/netns 2024-09-16T15:48:33.133 INFO:teuthology.orchestra.run.smithi123.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-09-16T15:48:33.133 INFO:teuthology.orchestra.run.smithi123.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-09-16T15:48:33.133 INFO:teuthology.orchestra.run.smithi123.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-09-16T15:48:33.133 INFO:teuthology.orchestra.run.smithi123.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-09-16T15:48:33.134 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:48:33.134 DEBUG:teuthology.orchestra.run.smithi123:> ls /sys/fs/fuse/connections 2024-09-16T15:48:33.182 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-09-16T15:48:33.182 DEBUG:teuthology.orchestra.run.smithi123:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2024-09-16T15:48:33.227 DEBUG:teuthology.orchestra.run.smithi123:> sudo modprobe fuse 2024-09-16T15:48:33.280 DEBUG:teuthology.orchestra.run.smithi123:> cat /proc/self/mounts | awk '{print $2}' 2024-09-16T15:48:33.308 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi123.stderr:2024-09-16T15:48:33.307+0000 7f4a47c62580 -1 init, newargv = 0x5595cfdabe20 newargc=15 2024-09-16T15:48:33.308 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi123.stderr:ceph-fuse[45941]: starting ceph client 2024-09-16T15:48:33.312 INFO:teuthology.orchestra.run.smithi123.stdout:/sys 2024-09-16T15:48:33.313 INFO:teuthology.orchestra.run.smithi123.stdout:/proc 2024-09-16T15:48:33.313 INFO:teuthology.orchestra.run.smithi123.stdout:/dev 2024-09-16T15:48:33.313 INFO:teuthology.orchestra.run.smithi123.stdout:/dev/pts 2024-09-16T15:48:33.313 INFO:teuthology.orchestra.run.smithi123.stdout:/run 2024-09-16T15:48:33.313 INFO:teuthology.orchestra.run.smithi123.stdout:/ 2024-09-16T15:48:33.313 INFO:teuthology.orchestra.run.smithi123.stdout:/sys/kernel/security 2024-09-16T15:48:33.313 INFO:teuthology.orchestra.run.smithi123.stdout:/dev/shm 2024-09-16T15:48:33.313 INFO:teuthology.orchestra.run.smithi123.stdout:/run/lock 2024-09-16T15:48:33.313 INFO:teuthology.orchestra.run.smithi123.stdout:/sys/fs/cgroup 2024-09-16T15:48:33.313 INFO:teuthology.orchestra.run.smithi123.stdout:/sys/fs/pstore 2024-09-16T15:48:33.313 INFO:teuthology.orchestra.run.smithi123.stdout:/sys/fs/bpf 2024-09-16T15:48:33.314 INFO:teuthology.orchestra.run.smithi123.stdout:/proc/sys/fs/binfmt_misc 2024-09-16T15:48:33.314 INFO:teuthology.orchestra.run.smithi123.stdout:/dev/hugepages 2024-09-16T15:48:33.314 INFO:teuthology.orchestra.run.smithi123.stdout:/dev/mqueue 2024-09-16T15:48:33.314 INFO:teuthology.orchestra.run.smithi123.stdout:/sys/kernel/debug 2024-09-16T15:48:33.314 INFO:teuthology.orchestra.run.smithi123.stdout:/sys/kernel/tracing 2024-09-16T15:48:33.314 INFO:teuthology.orchestra.run.smithi123.stdout:/proc/fs/nfsd 2024-09-16T15:48:33.314 INFO:teuthology.orchestra.run.smithi123.stdout:/sys/fs/fuse/connections 2024-09-16T15:48:33.314 INFO:teuthology.orchestra.run.smithi123.stdout:/sys/kernel/config 2024-09-16T15:48:33.314 INFO:teuthology.orchestra.run.smithi123.stdout:/run/credentials/systemd-sysusers.service 2024-09-16T15:48:33.314 INFO:teuthology.orchestra.run.smithi123.stdout:/run/rpc_pipefs 2024-09-16T15:48:33.314 INFO:teuthology.orchestra.run.smithi123.stdout:/run/user/1000 2024-09-16T15:48:33.314 INFO:teuthology.orchestra.run.smithi123.stdout:/var/lib/ceph 2024-09-16T15:48:33.314 INFO:teuthology.orchestra.run.smithi123.stdout:/var/lib/ceph/osd/ceph-0 2024-09-16T15:48:33.315 INFO:teuthology.orchestra.run.smithi123.stdout:/var/lib/ceph/osd/ceph-1 2024-09-16T15:48:33.315 INFO:teuthology.orchestra.run.smithi123.stdout:/var/lib/ceph/osd/ceph-2 2024-09-16T15:48:33.315 INFO:teuthology.orchestra.run.smithi123.stdout:/var/lib/ceph/osd/ceph-3 2024-09-16T15:48:33.315 INFO:teuthology.orchestra.run.smithi123.stdout:/run/netns 2024-09-16T15:48:33.315 INFO:teuthology.orchestra.run.smithi123.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-09-16T15:48:33.315 INFO:teuthology.orchestra.run.smithi123.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-09-16T15:48:33.315 INFO:teuthology.orchestra.run.smithi123.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-09-16T15:48:33.315 INFO:teuthology.orchestra.run.smithi123.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-09-16T15:48:33.315 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi123.stderr:ceph-fuse[45941]: starting fuse 2024-09-16T15:48:33.316 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:48:33.316 DEBUG:teuthology.orchestra.run.smithi123:> ls /sys/fs/fuse/connections 2024-09-16T15:48:33.361 INFO:teuthology.orchestra.run.smithi123.stdout:45 2024-09-16T15:48:33.361 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [45] 2024-09-16T15:48:33.361 DEBUG:teuthology.orchestra.run.smithi123:> sudo stdin-killer -- python3 -c ' 2024-09-16T15:48:33.362 DEBUG:teuthology.orchestra.run.smithi123:> import glob 2024-09-16T15:48:33.362 DEBUG:teuthology.orchestra.run.smithi123:> import re 2024-09-16T15:48:33.362 DEBUG:teuthology.orchestra.run.smithi123:> import os 2024-09-16T15:48:33.362 DEBUG:teuthology.orchestra.run.smithi123:> import subprocess 2024-09-16T15:48:33.362 DEBUG:teuthology.orchestra.run.smithi123:> 2024-09-16T15:48:33.362 DEBUG:teuthology.orchestra.run.smithi123:> def _find_admin_socket(client_name): 2024-09-16T15:48:33.362 DEBUG:teuthology.orchestra.run.smithi123:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2024-09-16T15:48:33.362 DEBUG:teuthology.orchestra.run.smithi123:> files = glob.glob(asok_path) 2024-09-16T15:48:33.362 DEBUG:teuthology.orchestra.run.smithi123:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-09-16T15:48:33.362 DEBUG:teuthology.orchestra.run.smithi123:> 2024-09-16T15:48:33.362 DEBUG:teuthology.orchestra.run.smithi123:> # Given a non-glob path, it better be there 2024-09-16T15:48:33.362 DEBUG:teuthology.orchestra.run.smithi123:> if "*" not in asok_path: 2024-09-16T15:48:33.362 DEBUG:teuthology.orchestra.run.smithi123:> assert(len(files) == 1) 2024-09-16T15:48:33.362 DEBUG:teuthology.orchestra.run.smithi123:> return files[0] 2024-09-16T15:48:33.363 DEBUG:teuthology.orchestra.run.smithi123:> 2024-09-16T15:48:33.363 DEBUG:teuthology.orchestra.run.smithi123:> for f in files: 2024-09-16T15:48:33.363 DEBUG:teuthology.orchestra.run.smithi123:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-09-16T15:48:33.363 DEBUG:teuthology.orchestra.run.smithi123:> if os.path.exists("/proc/{0}".format(pid)): 2024-09-16T15:48:33.363 DEBUG:teuthology.orchestra.run.smithi123:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-09-16T15:48:33.363 DEBUG:teuthology.orchestra.run.smithi123:> contents = proc_f.read() 2024-09-16T15:48:33.363 DEBUG:teuthology.orchestra.run.smithi123:> if mountpoint in contents: 2024-09-16T15:48:33.363 DEBUG:teuthology.orchestra.run.smithi123:> return f 2024-09-16T15:48:33.363 DEBUG:teuthology.orchestra.run.smithi123:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-09-16T15:48:33.363 DEBUG:teuthology.orchestra.run.smithi123:> 2024-09-16T15:48:33.363 DEBUG:teuthology.orchestra.run.smithi123:> print(_find_admin_socket("client.0")) 2024-09-16T15:48:33.363 DEBUG:teuthology.orchestra.run.smithi123:> ' 2024-09-16T15:48:33.472 INFO:teuthology.orchestra.run.smithi123.stdout:/var/run/ceph/ceph-client.0.45941.asok 2024-09-16T15:48:33.476 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:33 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-16T15:48:33.483 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.45941.asok 2024-09-16T15:48:33.483 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:48:33.483 DEBUG:teuthology.orchestra.run.smithi123:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.45941.asok status 2024-09-16T15:48:33.632 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2024-09-16T15:48:33.632 INFO:teuthology.orchestra.run.smithi123.stdout: "metadata": { 2024-09-16T15:48:33.632 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph_sha1": "33f95d2ae1bcbe7eb04404303f179eeeaa6982df", 2024-09-16T15:48:33.632 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph_version": "ceph version 19.3.0-4938-g33f95d2a (33f95d2ae1bcbe7eb04404303f179eeeaa6982df) squid (dev)", 2024-09-16T15:48:33.632 INFO:teuthology.orchestra.run.smithi123.stdout: "entity_id": "0", 2024-09-16T15:48:33.632 INFO:teuthology.orchestra.run.smithi123.stdout: "hostname": "smithi123", 2024-09-16T15:48:33.632 INFO:teuthology.orchestra.run.smithi123.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2024-09-16T15:48:33.632 INFO:teuthology.orchestra.run.smithi123.stdout: "pid": "45941", 2024-09-16T15:48:33.632 INFO:teuthology.orchestra.run.smithi123.stdout: "root": "/" 2024-09-16T15:48:33.633 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:48:33.633 INFO:teuthology.orchestra.run.smithi123.stdout: "dentry_count": 0, 2024-09-16T15:48:33.633 INFO:teuthology.orchestra.run.smithi123.stdout: "dentry_pinned_count": 0, 2024-09-16T15:48:33.633 INFO:teuthology.orchestra.run.smithi123.stdout: "id": 7194, 2024-09-16T15:48:33.633 INFO:teuthology.orchestra.run.smithi123.stdout: "inst": { 2024-09-16T15:48:33.633 INFO:teuthology.orchestra.run.smithi123.stdout: "name": { 2024-09-16T15:48:33.633 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "client", 2024-09-16T15:48:33.633 INFO:teuthology.orchestra.run.smithi123.stdout: "num": 7194 2024-09-16T15:48:33.633 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:48:33.633 INFO:teuthology.orchestra.run.smithi123.stdout: "addr": { 2024-09-16T15:48:33.633 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "v1", 2024-09-16T15:48:33.633 INFO:teuthology.orchestra.run.smithi123.stdout: "addr": "192.168.0.1:0", 2024-09-16T15:48:33.633 INFO:teuthology.orchestra.run.smithi123.stdout: "nonce": 2119003805 2024-09-16T15:48:33.633 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-16T15:48:33.633 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:48:33.634 INFO:teuthology.orchestra.run.smithi123.stdout: "addr": { 2024-09-16T15:48:33.634 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "v1", 2024-09-16T15:48:33.634 INFO:teuthology.orchestra.run.smithi123.stdout: "addr": "192.168.0.1:0", 2024-09-16T15:48:33.634 INFO:teuthology.orchestra.run.smithi123.stdout: "nonce": 2119003805 2024-09-16T15:48:33.634 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:48:33.634 INFO:teuthology.orchestra.run.smithi123.stdout: "inst_str": "client.7194 192.168.0.1:0/2119003805", 2024-09-16T15:48:33.634 INFO:teuthology.orchestra.run.smithi123.stdout: "addr_str": "192.168.0.1:0/2119003805", 2024-09-16T15:48:33.634 INFO:teuthology.orchestra.run.smithi123.stdout: "inode_count": 1, 2024-09-16T15:48:33.634 INFO:teuthology.orchestra.run.smithi123.stdout: "mds_epoch": 68, 2024-09-16T15:48:33.634 INFO:teuthology.orchestra.run.smithi123.stdout: "osd_epoch": 103, 2024-09-16T15:48:33.634 INFO:teuthology.orchestra.run.smithi123.stdout: "osd_epoch_barrier": 0, 2024-09-16T15:48:33.634 INFO:teuthology.orchestra.run.smithi123.stdout: "blocklisted": false, 2024-09-16T15:48:33.634 INFO:teuthology.orchestra.run.smithi123.stdout: "fs_name": "cephfs" 2024-09-16T15:48:33.634 INFO:teuthology.orchestra.run.smithi123.stdout:} 2024-09-16T15:48:33.645 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:48:33.645 DEBUG:teuthology.orchestra.run.smithi123:> stat --file-system '--printf=%T 2024-09-16T15:48:33.645 DEBUG:teuthology.orchestra.run.smithi123:> ' -- /home/ubuntu/cephtest/mnt.0 2024-09-16T15:48:33.651 INFO:teuthology.orchestra.run.smithi123.stdout:fuseblk 2024-09-16T15:48:33.651 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2024-09-16T15:48:33.651 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:48:33.652 DEBUG:teuthology.orchestra.run.smithi123:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-09-16T15:48:33.705 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-09-16T15:48:33.874 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:33.875+0000 7f041f3d9640 1 -- 172.21.15.123:0/2907385541 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f041815e160 msgr2=0x7f0418073f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:48:33.874 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:33.875+0000 7f041f3d9640 1 --2- 172.21.15.123:0/2907385541 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f041815e160 0x7f0418073f30 secure :-1 s=READY pgs=1097 cs=0 l=1 rev1=1 crypto rx=0x7f0418073c20 tx=0x7f041401c700 comp rx=0 tx=0).stop 2024-09-16T15:48:33.874 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:33.875+0000 7f041f3d9640 1 -- 172.21.15.123:0/2907385541 shutdown_connections 2024-09-16T15:48:33.874 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:33.875+0000 7f041f3d9640 1 --2- 172.21.15.123:0/2907385541 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f041815e160 0x7f0418073f30 unknown :-1 s=CLOSED pgs=1097 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:33.874 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:33.875+0000 7f041f3d9640 1 -- 172.21.15.123:0/2907385541 >> 172.21.15.123:0/2907385541 conn(0x7f04180cfb10 msgr2=0x7f04180cff20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:48:33.874 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:33.875+0000 7f041f3d9640 1 -- 172.21.15.123:0/2907385541 shutdown_connections 2024-09-16T15:48:33.875 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:33.875+0000 7f041f3d9640 1 -- 172.21.15.123:0/2907385541 wait complete. 2024-09-16T15:48:33.875 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:33.875+0000 7f041f3d9640 1 Processor -- start 2024-09-16T15:48:33.875 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:33.875+0000 7f041f3d9640 1 -- start start 2024-09-16T15:48:33.877 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:33.875+0000 7f041f3d9640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f0418074470 0x7f04180c4120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:48:33.877 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:33.875+0000 7f041f3d9640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f041815e160 0x7f0418159610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:48:33.877 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:33.875+0000 7f041f3d9640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f041815e820 0x7f0418159b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:48:33.877 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:33.875+0000 7f041f3d9640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f04181697f0 con 0x7f041815e820 2024-09-16T15:48:33.877 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:33.875+0000 7f041f3d9640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f04180cf640 con 0x7f0418074470 2024-09-16T15:48:33.877 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:33.875+0000 7f041f3d9640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f0418169970 con 0x7f041815e160 2024-09-16T15:48:33.877 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:33.875+0000 7f041d976640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f041815e820 0x7f0418159b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:48:33.877 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:33.875+0000 7f041d976640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f041815e820 0x7f0418159b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:55122/0 (socket says 172.21.15.123:55122) 2024-09-16T15:48:33.877 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:33.875+0000 7f041d976640 1 -- 172.21.15.123:0/923556919 learned_addr learned my addr 172.21.15.123:0/923556919 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:48:33.877 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:33.875+0000 7f041d976640 1 -- 172.21.15.123:0/923556919 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f041815e160 msgr2=0x7f0418159610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:48:33.878 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:33.875+0000 7f041d976640 1 --2- 172.21.15.123:0/923556919 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f041815e160 0x7f0418159610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:33.878 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:33.875+0000 7f041d976640 1 -- 172.21.15.123:0/923556919 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f0418074470 msgr2=0x7f04180c4120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:48:33.878 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:33.875+0000 7f041d976640 1 --2- 172.21.15.123:0/923556919 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f0418074470 0x7f04180c4120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:33.878 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:33.875+0000 7f041d976640 1 -- 172.21.15.123:0/923556919 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0418156bd0 con 0x7f041815e820 2024-09-16T15:48:33.878 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:33.875+0000 7f041d976640 1 --2- 172.21.15.123:0/923556919 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f041815e820 0x7f0418159b50 secure :-1 s=READY pgs=1098 cs=0 l=1 rev1=1 crypto rx=0x7f041000c730 tx=0x7f041000cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:48:33.878 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:33.875+0000 7f040dffb640 1 -- 172.21.15.123:0/923556919 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0410003b80 con 0x7f041815e820 2024-09-16T15:48:33.878 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:33.875+0000 7f040dffb640 1 -- 172.21.15.123:0/923556919 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f0410003d40 con 0x7f041815e820 2024-09-16T15:48:33.878 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:33.875+0000 7f040dffb640 1 -- 172.21.15.123:0/923556919 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04100052d0 con 0x7f041815e820 2024-09-16T15:48:33.878 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:33.875+0000 7f041f3d9640 1 -- 172.21.15.123:0/923556919 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0418156e50 con 0x7f041815e820 2024-09-16T15:48:33.878 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:33.875+0000 7f041f3d9640 1 -- 172.21.15.123:0/923556919 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f04180c3640 con 0x7f041815e820 2024-09-16T15:48:33.879 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:33.879+0000 7f041f3d9640 1 -- 172.21.15.123:0/923556919 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f04180d06c0 con 0x7f041815e820 2024-09-16T15:48:33.879 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:33.879+0000 7f040dffb640 1 -- 172.21.15.123:0/923556919 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 16) ==== 97182+0+0 (secure 0 0 0) 0x7f0410005680 con 0x7f041815e820 2024-09-16T15:48:33.879 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:33.879+0000 7f040dffb640 1 --2- 172.21.15.123:0/923556919 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f03fc072fe0 0x7f03fc0754a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:48:33.879 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:33.879+0000 7f040dffb640 1 -- 172.21.15.123:0/923556919 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(103..103 src has 1..103) ==== 5709+0+0 (secure 0 0 0) 0x7f0410094000 con 0x7f041815e820 2024-09-16T15:48:33.879 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:33.879+0000 7f041d175640 1 --2- 172.21.15.123:0/923556919 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f03fc072fe0 0x7f03fc0754a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:48:33.879 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:33.879+0000 7f041d175640 1 --2- 172.21.15.123:0/923556919 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f03fc072fe0 0x7f03fc0754a0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f04140af000 tx=0x7f04140194a0 comp rx=0 tx=0).ready entity=mgr.4150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:48:33.883 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:33.883+0000 7f040dffb640 1 -- 172.21.15.123:0/923556919 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f0418156bd0 con 0x7f041815e820 2024-09-16T15:48:34.093 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:34.091+0000 7f041f3d9640 1 -- 172.21.15.123:0/923556919 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f0418073f10 con 0x7f041815e820 2024-09-16T15:48:34.093 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:34.095+0000 7f040dffb640 1 -- 172.21.15.123:0/923556919 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v68) ==== 53+0+73 (secure 0 0 0) 0x7f041005f030 con 0x7f041815e820 2024-09-16T15:48:34.093 INFO:teuthology.orchestra.run.smithi123.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-09-16T15:48:34.097 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:34.095+0000 7f041f3d9640 1 -- 172.21.15.123:0/923556919 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f03fc072fe0 msgr2=0x7f03fc0754a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:48:34.097 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:34.095+0000 7f041f3d9640 1 --2- 172.21.15.123:0/923556919 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f03fc072fe0 0x7f03fc0754a0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f04140af000 tx=0x7f04140194a0 comp rx=0 tx=0).stop 2024-09-16T15:48:34.097 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:34.095+0000 7f041f3d9640 1 -- 172.21.15.123:0/923556919 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f041815e820 msgr2=0x7f0418159b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:48:34.097 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:34.095+0000 7f041f3d9640 1 --2- 172.21.15.123:0/923556919 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f041815e820 0x7f0418159b50 secure :-1 s=READY pgs=1098 cs=0 l=1 rev1=1 crypto rx=0x7f041000c730 tx=0x7f041000cbf0 comp rx=0 tx=0).stop 2024-09-16T15:48:34.097 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:34.095+0000 7f041f3d9640 1 -- 172.21.15.123:0/923556919 shutdown_connections 2024-09-16T15:48:34.097 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:34.095+0000 7f041f3d9640 1 --2- 172.21.15.123:0/923556919 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f03fc072fe0 0x7f03fc0754a0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:34.097 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:34.095+0000 7f041f3d9640 1 --2- 172.21.15.123:0/923556919 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f041815e820 0x7f0418159b50 unknown :-1 s=CLOSED pgs=1098 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:34.097 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:34.095+0000 7f041f3d9640 1 --2- 172.21.15.123:0/923556919 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f041815e160 0x7f0418159610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:34.098 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:34.095+0000 7f041f3d9640 1 --2- 172.21.15.123:0/923556919 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f0418074470 0x7f04180c4120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:34.098 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:34.095+0000 7f041f3d9640 1 -- 172.21.15.123:0/923556919 >> 172.21.15.123:0/923556919 conn(0x7f04180cfb10 msgr2=0x7f041806fa40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:48:34.098 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:34.095+0000 7f041f3d9640 1 -- 172.21.15.123:0/923556919 shutdown_connections 2024-09-16T15:48:34.098 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:34.095+0000 7f041f3d9640 1 -- 172.21.15.123:0/923556919 wait complete. 2024-09-16T15:48:34.113 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-09-16T15:48:34.113 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-09-16T15:48:34.113 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi184.front.sepia.ceph.com 2024-09-16T15:48:34.113 INFO:tasks.cephfs.mount:self.client.name = client.1 2024-09-16T15:48:34.113 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2024-09-16T15:48:34.113 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-09-16T15:48:34.113 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-09-16T15:48:34.113 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-09-16T15:48:34.114 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.1' netns for '/home/ubuntu/cephtest/mnt.1' 2024-09-16T15:48:34.114 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:48:34.114 DEBUG:teuthology.orchestra.run.smithi184:> ip addr 2024-09-16T15:48:34.121 INFO:teuthology.orchestra.run.smithi184.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-09-16T15:48:34.121 INFO:teuthology.orchestra.run.smithi184.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-09-16T15:48:34.121 INFO:teuthology.orchestra.run.smithi184.stdout: inet 127.0.0.1/8 scope host lo 2024-09-16T15:48:34.121 INFO:teuthology.orchestra.run.smithi184.stdout: valid_lft forever preferred_lft forever 2024-09-16T15:48:34.121 INFO:teuthology.orchestra.run.smithi184.stdout: inet6 ::1/128 scope host 2024-09-16T15:48:34.121 INFO:teuthology.orchestra.run.smithi184.stdout: valid_lft forever preferred_lft forever 2024-09-16T15:48:34.121 INFO:teuthology.orchestra.run.smithi184.stdout:2: eno1: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-09-16T15:48:34.122 INFO:teuthology.orchestra.run.smithi184.stdout: link/ether 0c:c4:7a:d9:37:16 brd ff:ff:ff:ff:ff:ff 2024-09-16T15:48:34.122 INFO:teuthology.orchestra.run.smithi184.stdout: altname enp7s0f0 2024-09-16T15:48:34.122 INFO:teuthology.orchestra.run.smithi184.stdout:3: eno2: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-09-16T15:48:34.122 INFO:teuthology.orchestra.run.smithi184.stdout: link/ether 0c:c4:7a:d9:37:17 brd ff:ff:ff:ff:ff:ff 2024-09-16T15:48:34.122 INFO:teuthology.orchestra.run.smithi184.stdout: altname enp7s0f1 2024-09-16T15:48:34.122 INFO:teuthology.orchestra.run.smithi184.stdout:4: enp3s0f0: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-09-16T15:48:34.122 INFO:teuthology.orchestra.run.smithi184.stdout: link/ether 0c:c4:7a:88:7b:8e brd ff:ff:ff:ff:ff:ff 2024-09-16T15:48:34.122 INFO:teuthology.orchestra.run.smithi184.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-09-16T15:48:34.122 INFO:teuthology.orchestra.run.smithi184.stdout: link/ether 0c:c4:7a:88:7b:8f brd ff:ff:ff:ff:ff:ff 2024-09-16T15:48:34.122 INFO:teuthology.orchestra.run.smithi184.stdout: inet 172.21.15.184/20 metric 100 brd 172.21.15.255 scope global dynamic enp3s0f1 2024-09-16T15:48:34.122 INFO:teuthology.orchestra.run.smithi184.stdout: valid_lft 42143sec preferred_lft 42143sec 2024-09-16T15:48:34.122 INFO:teuthology.orchestra.run.smithi184.stdout: inet6 fe80::ec4:7aff:fe88:7b8f/64 scope link 2024-09-16T15:48:34.122 INFO:teuthology.orchestra.run.smithi184.stdout: valid_lft forever preferred_lft forever 2024-09-16T15:48:34.122 INFO:teuthology.orchestra.run.smithi184.stdout:6: docker0: mtu 1500 qdisc noqueue state DOWN group default 2024-09-16T15:48:34.123 INFO:teuthology.orchestra.run.smithi184.stdout: link/ether 02:42:5d:0a:44:b9 brd ff:ff:ff:ff:ff:ff 2024-09-16T15:48:34.123 INFO:teuthology.orchestra.run.smithi184.stdout: inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0 2024-09-16T15:48:34.123 INFO:teuthology.orchestra.run.smithi184.stdout: valid_lft forever preferred_lft forever 2024-09-16T15:48:34.123 INFO:teuthology.orchestra.run.smithi184.stdout:7: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-09-16T15:48:34.123 INFO:teuthology.orchestra.run.smithi184.stdout: link/ether 5e:1f:84:6d:79:ae brd ff:ff:ff:ff:ff:ff 2024-09-16T15:48:34.123 INFO:teuthology.orchestra.run.smithi184.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-09-16T15:48:34.123 INFO:teuthology.orchestra.run.smithi184.stdout: valid_lft forever preferred_lft forever 2024-09-16T15:48:34.123 INFO:teuthology.orchestra.run.smithi184.stdout: inet6 fe80::5c1f:84ff:fe6d:79ae/64 scope link 2024-09-16T15:48:34.123 INFO:teuthology.orchestra.run.smithi184.stdout: valid_lft forever preferred_lft forever 2024-09-16T15:48:34.123 INFO:teuthology.orchestra.run.smithi184.stdout:8: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-09-16T15:48:34.123 INFO:teuthology.orchestra.run.smithi184.stdout: link/ether 0e:5c:f6:1f:82:2f brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.1 2024-09-16T15:48:34.123 INFO:teuthology.orchestra.run.smithi184.stdout: inet6 fe80::c5c:f6ff:fe1f:822f/64 scope link 2024-09-16T15:48:34.123 INFO:teuthology.orchestra.run.smithi184.stdout: valid_lft forever preferred_lft forever 2024-09-16T15:48:34.123 INFO:teuthology.orchestra.run.smithi184.stdout:9: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-09-16T15:48:34.124 INFO:teuthology.orchestra.run.smithi184.stdout: link/ether c6:a6:3d:69:d0:db brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.3 2024-09-16T15:48:34.124 INFO:teuthology.orchestra.run.smithi184.stdout: inet6 fe80::c4a6:3dff:fe69:d0db/64 scope link 2024-09-16T15:48:34.124 INFO:teuthology.orchestra.run.smithi184.stdout: valid_lft forever preferred_lft forever 2024-09-16T15:48:34.124 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:48:34.124 DEBUG:teuthology.orchestra.run.smithi184:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-09-16T15:48:34.182 INFO:teuthology.orchestra.run.smithi184.stdout:1 2024-09-16T15:48:34.183 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:48:34.183 DEBUG:teuthology.orchestra.run.smithi184:> route 2024-09-16T15:48:34.238 INFO:teuthology.orchestra.run.smithi184.stdout:Kernel IP routing table 2024-09-16T15:48:34.238 INFO:teuthology.orchestra.run.smithi184.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-09-16T15:48:34.238 INFO:teuthology.orchestra.run.smithi184.stdout:default _gateway 0.0.0.0 UG 0 0 0 enp3s0f1 2024-09-16T15:48:34.238 INFO:teuthology.orchestra.run.smithi184.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-09-16T15:48:34.238 INFO:teuthology.orchestra.run.smithi184.stdout:158.69.67.47 _gateway 255.255.255.255 UGH 100 0 0 enp3s0f1 2024-09-16T15:48:34.238 INFO:teuthology.orchestra.run.smithi184.stdout:172.17.0.0 0.0.0.0 255.255.0.0 U 0 0 0 docker0 2024-09-16T15:48:34.238 INFO:teuthology.orchestra.run.smithi184.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-09-16T15:48:34.239 INFO:teuthology.orchestra.run.smithi184.stdout:ns1.front.sepia 0.0.0.0 255.255.255.255 UH 100 0 0 enp3s0f1 2024-09-16T15:48:34.239 INFO:teuthology.orchestra.run.smithi184.stdout:172.21.0.2 0.0.0.0 255.255.255.255 UH 100 0 0 enp3s0f1 2024-09-16T15:48:34.239 INFO:teuthology.orchestra.run.smithi184.stdout:_gateway 0.0.0.0 255.255.255.255 UH 100 0 0 enp3s0f1 2024-09-16T15:48:34.239 INFO:teuthology.orchestra.run.smithi184.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-09-16T15:48:34.239 DEBUG:teuthology.orchestra.run.smithi184:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-09-16T15:48:34.239 DEBUG:teuthology.orchestra.run.smithi184:> set -e 2024-09-16T15:48:34.239 DEBUG:teuthology.orchestra.run.smithi184:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-09-16T15:48:34.239 DEBUG:teuthology.orchestra.run.smithi184:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-09-16T15:48:34.239 DEBUG:teuthology.orchestra.run.smithi184:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-09-16T15:48:34.239 DEBUG:teuthology.orchestra.run.smithi184:> ') 2024-09-16T15:48:34.327 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:48:34 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-09-16T15:48:34.354 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:48:34 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-16T15:48:34.361 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:48:34.361 DEBUG:teuthology.orchestra.run.smithi184:> ip netns list 2024-09-16T15:48:34.366 INFO:teuthology.orchestra.run.smithi184.stdout:ceph-ns--home-ubuntu-cephtest-mnt.3 (id: 1) 2024-09-16T15:48:34.367 INFO:teuthology.orchestra.run.smithi184.stdout:ceph-ns--home-ubuntu-cephtest-mnt.1 (id: 0) 2024-09-16T15:48:34.367 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.1' netns for '/home/ubuntu/cephtest/mnt.1' 2024-09-16T15:48:34.367 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:48:34.367 DEBUG:teuthology.orchestra.run.smithi184:> sudo ip link set brx.0 up 2024-09-16T15:48:34.421 INFO:tasks.cephfs.fuse_mount:Client client.1 config is {} 2024-09-16T15:48:34.422 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-09-16T15:48:34.422 DEBUG:teuthology.orchestra.run.smithi184:> mkdir -p -v /home/ubuntu/cephtest/mnt.1 2024-09-16T15:48:34.470 INFO:teuthology.orchestra.run.smithi184.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.1' 2024-09-16T15:48:34.471 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-09-16T15:48:34.471 DEBUG:teuthology.orchestra.run.smithi184:> chmod 0000 /home/ubuntu/cephtest/mnt.1 2024-09-16T15:48:34.519 DEBUG:teuthology.orchestra.run.smithi184:> sudo modprobe fuse 2024-09-16T15:48:34.579 DEBUG:teuthology.orchestra.run.smithi184:> cat /proc/self/mounts | awk '{print $2}' 2024-09-16T15:48:34.627 INFO:teuthology.orchestra.run.smithi184.stdout:/sys 2024-09-16T15:48:34.627 INFO:teuthology.orchestra.run.smithi184.stdout:/proc 2024-09-16T15:48:34.627 INFO:teuthology.orchestra.run.smithi184.stdout:/dev 2024-09-16T15:48:34.627 INFO:teuthology.orchestra.run.smithi184.stdout:/dev/pts 2024-09-16T15:48:34.627 INFO:teuthology.orchestra.run.smithi184.stdout:/run 2024-09-16T15:48:34.628 INFO:teuthology.orchestra.run.smithi184.stdout:/ 2024-09-16T15:48:34.628 INFO:teuthology.orchestra.run.smithi184.stdout:/sys/kernel/security 2024-09-16T15:48:34.628 INFO:teuthology.orchestra.run.smithi184.stdout:/dev/shm 2024-09-16T15:48:34.628 INFO:teuthology.orchestra.run.smithi184.stdout:/run/lock 2024-09-16T15:48:34.628 INFO:teuthology.orchestra.run.smithi184.stdout:/sys/fs/cgroup 2024-09-16T15:48:34.628 INFO:teuthology.orchestra.run.smithi184.stdout:/sys/fs/pstore 2024-09-16T15:48:34.628 INFO:teuthology.orchestra.run.smithi184.stdout:/sys/fs/bpf 2024-09-16T15:48:34.628 INFO:teuthology.orchestra.run.smithi184.stdout:/proc/sys/fs/binfmt_misc 2024-09-16T15:48:34.628 INFO:teuthology.orchestra.run.smithi184.stdout:/dev/hugepages 2024-09-16T15:48:34.628 INFO:teuthology.orchestra.run.smithi184.stdout:/dev/mqueue 2024-09-16T15:48:34.628 INFO:teuthology.orchestra.run.smithi184.stdout:/sys/kernel/debug 2024-09-16T15:48:34.628 INFO:teuthology.orchestra.run.smithi184.stdout:/sys/kernel/tracing 2024-09-16T15:48:34.629 INFO:teuthology.orchestra.run.smithi184.stdout:/proc/fs/nfsd 2024-09-16T15:48:34.629 INFO:teuthology.orchestra.run.smithi184.stdout:/sys/fs/fuse/connections 2024-09-16T15:48:34.629 INFO:teuthology.orchestra.run.smithi184.stdout:/sys/kernel/config 2024-09-16T15:48:34.629 INFO:teuthology.orchestra.run.smithi184.stdout:/run/credentials/systemd-sysusers.service 2024-09-16T15:48:34.629 INFO:teuthology.orchestra.run.smithi184.stdout:/run/rpc_pipefs 2024-09-16T15:48:34.629 INFO:teuthology.orchestra.run.smithi184.stdout:/run/user/1000 2024-09-16T15:48:34.629 INFO:teuthology.orchestra.run.smithi184.stdout:/var/lib/ceph 2024-09-16T15:48:34.629 INFO:teuthology.orchestra.run.smithi184.stdout:/var/lib/ceph/osd/ceph-4 2024-09-16T15:48:34.629 INFO:teuthology.orchestra.run.smithi184.stdout:/var/lib/ceph/osd/ceph-5 2024-09-16T15:48:34.629 INFO:teuthology.orchestra.run.smithi184.stdout:/var/lib/ceph/osd/ceph-6 2024-09-16T15:48:34.629 INFO:teuthology.orchestra.run.smithi184.stdout:/var/lib/ceph/osd/ceph-7 2024-09-16T15:48:34.629 INFO:teuthology.orchestra.run.smithi184.stdout:/run/netns 2024-09-16T15:48:34.629 INFO:teuthology.orchestra.run.smithi184.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-09-16T15:48:34.630 INFO:teuthology.orchestra.run.smithi184.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-09-16T15:48:34.630 INFO:teuthology.orchestra.run.smithi184.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-09-16T15:48:34.630 INFO:teuthology.orchestra.run.smithi184.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-09-16T15:48:34.630 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:48:34.631 DEBUG:teuthology.orchestra.run.smithi184:> ls /sys/fs/fuse/connections 2024-09-16T15:48:34.678 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-09-16T15:48:34.678 DEBUG:teuthology.orchestra.run.smithi184:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.1 --id 1) 2024-09-16T15:48:34.723 DEBUG:teuthology.orchestra.run.smithi184:> sudo modprobe fuse 2024-09-16T15:48:34.738 DEBUG:teuthology.orchestra.run.smithi184:> cat /proc/self/mounts | awk '{print $2}' 2024-09-16T15:48:34.787 INFO:teuthology.orchestra.run.smithi184.stdout:/sys 2024-09-16T15:48:34.787 INFO:teuthology.orchestra.run.smithi184.stdout:/proc 2024-09-16T15:48:34.787 INFO:teuthology.orchestra.run.smithi184.stdout:/dev 2024-09-16T15:48:34.787 INFO:teuthology.orchestra.run.smithi184.stdout:/dev/pts 2024-09-16T15:48:34.787 INFO:teuthology.orchestra.run.smithi184.stdout:/run 2024-09-16T15:48:34.787 INFO:teuthology.orchestra.run.smithi184.stdout:/ 2024-09-16T15:48:34.787 INFO:teuthology.orchestra.run.smithi184.stdout:/sys/kernel/security 2024-09-16T15:48:34.787 INFO:teuthology.orchestra.run.smithi184.stdout:/dev/shm 2024-09-16T15:48:34.788 INFO:teuthology.orchestra.run.smithi184.stdout:/run/lock 2024-09-16T15:48:34.788 INFO:teuthology.orchestra.run.smithi184.stdout:/sys/fs/cgroup 2024-09-16T15:48:34.788 INFO:teuthology.orchestra.run.smithi184.stdout:/sys/fs/pstore 2024-09-16T15:48:34.788 INFO:teuthology.orchestra.run.smithi184.stdout:/sys/fs/bpf 2024-09-16T15:48:34.788 INFO:teuthology.orchestra.run.smithi184.stdout:/proc/sys/fs/binfmt_misc 2024-09-16T15:48:34.788 INFO:teuthology.orchestra.run.smithi184.stdout:/dev/hugepages 2024-09-16T15:48:34.788 INFO:teuthology.orchestra.run.smithi184.stdout:/dev/mqueue 2024-09-16T15:48:34.788 INFO:teuthology.orchestra.run.smithi184.stdout:/sys/kernel/debug 2024-09-16T15:48:34.788 INFO:teuthology.orchestra.run.smithi184.stdout:/sys/kernel/tracing 2024-09-16T15:48:34.788 INFO:teuthology.orchestra.run.smithi184.stdout:/proc/fs/nfsd 2024-09-16T15:48:34.788 INFO:teuthology.orchestra.run.smithi184.stdout:/sys/fs/fuse/connections 2024-09-16T15:48:34.788 INFO:teuthology.orchestra.run.smithi184.stdout:/sys/kernel/config 2024-09-16T15:48:34.788 INFO:teuthology.orchestra.run.smithi184.stdout:/run/credentials/systemd-sysusers.service 2024-09-16T15:48:34.789 INFO:teuthology.orchestra.run.smithi184.stdout:/run/rpc_pipefs 2024-09-16T15:48:34.789 INFO:teuthology.orchestra.run.smithi184.stdout:/run/user/1000 2024-09-16T15:48:34.789 INFO:teuthology.orchestra.run.smithi184.stdout:/var/lib/ceph 2024-09-16T15:48:34.789 INFO:teuthology.orchestra.run.smithi184.stdout:/var/lib/ceph/osd/ceph-4 2024-09-16T15:48:34.789 INFO:teuthology.orchestra.run.smithi184.stdout:/var/lib/ceph/osd/ceph-5 2024-09-16T15:48:34.789 INFO:teuthology.orchestra.run.smithi184.stdout:/var/lib/ceph/osd/ceph-6 2024-09-16T15:48:34.789 INFO:teuthology.orchestra.run.smithi184.stdout:/var/lib/ceph/osd/ceph-7 2024-09-16T15:48:34.789 INFO:teuthology.orchestra.run.smithi184.stdout:/run/netns 2024-09-16T15:48:34.789 INFO:teuthology.orchestra.run.smithi184.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-09-16T15:48:34.789 INFO:teuthology.orchestra.run.smithi184.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-09-16T15:48:34.789 INFO:teuthology.orchestra.run.smithi184.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-09-16T15:48:34.789 INFO:teuthology.orchestra.run.smithi184.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-09-16T15:48:34.790 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:48:34.790 DEBUG:teuthology.orchestra.run.smithi184:> ls /sys/fs/fuse/connections 2024-09-16T15:48:34.803 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi184.stderr:2024-09-16T15:48:34.800+0000 7fe7dadce580 -1 init, newargv = 0x55a9d9e49d10 newargc=15 2024-09-16T15:48:34.803 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi184.stderr:ceph-fuse[35733]: starting ceph client 2024-09-16T15:48:34.812 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi184.stderr:ceph-fuse[35733]: starting fuse 2024-09-16T15:48:35.908 DEBUG:teuthology.orchestra.run.smithi184:> sudo modprobe fuse 2024-09-16T15:48:35.924 DEBUG:teuthology.orchestra.run.smithi184:> cat /proc/self/mounts | awk '{print $2}' 2024-09-16T15:48:35.971 INFO:teuthology.orchestra.run.smithi184.stdout:/sys 2024-09-16T15:48:35.971 INFO:teuthology.orchestra.run.smithi184.stdout:/proc 2024-09-16T15:48:35.971 INFO:teuthology.orchestra.run.smithi184.stdout:/dev 2024-09-16T15:48:35.972 INFO:teuthology.orchestra.run.smithi184.stdout:/dev/pts 2024-09-16T15:48:35.972 INFO:teuthology.orchestra.run.smithi184.stdout:/run 2024-09-16T15:48:35.972 INFO:teuthology.orchestra.run.smithi184.stdout:/ 2024-09-16T15:48:35.972 INFO:teuthology.orchestra.run.smithi184.stdout:/sys/kernel/security 2024-09-16T15:48:35.972 INFO:teuthology.orchestra.run.smithi184.stdout:/dev/shm 2024-09-16T15:48:35.972 INFO:teuthology.orchestra.run.smithi184.stdout:/run/lock 2024-09-16T15:48:35.972 INFO:teuthology.orchestra.run.smithi184.stdout:/sys/fs/cgroup 2024-09-16T15:48:35.972 INFO:teuthology.orchestra.run.smithi184.stdout:/sys/fs/pstore 2024-09-16T15:48:35.972 INFO:teuthology.orchestra.run.smithi184.stdout:/sys/fs/bpf 2024-09-16T15:48:35.972 INFO:teuthology.orchestra.run.smithi184.stdout:/proc/sys/fs/binfmt_misc 2024-09-16T15:48:35.972 INFO:teuthology.orchestra.run.smithi184.stdout:/dev/hugepages 2024-09-16T15:48:35.972 INFO:teuthology.orchestra.run.smithi184.stdout:/dev/mqueue 2024-09-16T15:48:35.972 INFO:teuthology.orchestra.run.smithi184.stdout:/sys/kernel/debug 2024-09-16T15:48:35.972 INFO:teuthology.orchestra.run.smithi184.stdout:/sys/kernel/tracing 2024-09-16T15:48:35.973 INFO:teuthology.orchestra.run.smithi184.stdout:/proc/fs/nfsd 2024-09-16T15:48:35.973 INFO:teuthology.orchestra.run.smithi184.stdout:/sys/fs/fuse/connections 2024-09-16T15:48:35.973 INFO:teuthology.orchestra.run.smithi184.stdout:/sys/kernel/config 2024-09-16T15:48:35.973 INFO:teuthology.orchestra.run.smithi184.stdout:/run/credentials/systemd-sysusers.service 2024-09-16T15:48:35.973 INFO:teuthology.orchestra.run.smithi184.stdout:/run/rpc_pipefs 2024-09-16T15:48:35.973 INFO:teuthology.orchestra.run.smithi184.stdout:/run/user/1000 2024-09-16T15:48:35.973 INFO:teuthology.orchestra.run.smithi184.stdout:/var/lib/ceph 2024-09-16T15:48:35.973 INFO:teuthology.orchestra.run.smithi184.stdout:/var/lib/ceph/osd/ceph-4 2024-09-16T15:48:35.973 INFO:teuthology.orchestra.run.smithi184.stdout:/var/lib/ceph/osd/ceph-5 2024-09-16T15:48:35.973 INFO:teuthology.orchestra.run.smithi184.stdout:/var/lib/ceph/osd/ceph-6 2024-09-16T15:48:35.973 INFO:teuthology.orchestra.run.smithi184.stdout:/var/lib/ceph/osd/ceph-7 2024-09-16T15:48:35.973 INFO:teuthology.orchestra.run.smithi184.stdout:/run/netns 2024-09-16T15:48:35.973 INFO:teuthology.orchestra.run.smithi184.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-09-16T15:48:35.973 INFO:teuthology.orchestra.run.smithi184.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-09-16T15:48:35.974 INFO:teuthology.orchestra.run.smithi184.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-09-16T15:48:35.974 INFO:teuthology.orchestra.run.smithi184.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-09-16T15:48:35.974 INFO:teuthology.orchestra.run.smithi184.stdout:/home/ubuntu/cephtest/mnt.1 2024-09-16T15:48:35.974 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:48:35.974 DEBUG:teuthology.orchestra.run.smithi184:> ls /sys/fs/fuse/connections 2024-09-16T15:48:36.022 INFO:teuthology.orchestra.run.smithi184.stdout:45 2024-09-16T15:48:36.023 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [45] 2024-09-16T15:48:36.023 DEBUG:teuthology.orchestra.run.smithi184:> sudo stdin-killer -- python3 -c ' 2024-09-16T15:48:36.023 DEBUG:teuthology.orchestra.run.smithi184:> import glob 2024-09-16T15:48:36.023 DEBUG:teuthology.orchestra.run.smithi184:> import re 2024-09-16T15:48:36.023 DEBUG:teuthology.orchestra.run.smithi184:> import os 2024-09-16T15:48:36.023 DEBUG:teuthology.orchestra.run.smithi184:> import subprocess 2024-09-16T15:48:36.023 DEBUG:teuthology.orchestra.run.smithi184:> 2024-09-16T15:48:36.023 DEBUG:teuthology.orchestra.run.smithi184:> def _find_admin_socket(client_name): 2024-09-16T15:48:36.023 DEBUG:teuthology.orchestra.run.smithi184:> asok_path = "/var/run/ceph/ceph-client.1.*.asok" 2024-09-16T15:48:36.023 DEBUG:teuthology.orchestra.run.smithi184:> files = glob.glob(asok_path) 2024-09-16T15:48:36.023 DEBUG:teuthology.orchestra.run.smithi184:> mountpoint = "/home/ubuntu/cephtest/mnt.1" 2024-09-16T15:48:36.024 DEBUG:teuthology.orchestra.run.smithi184:> 2024-09-16T15:48:36.024 DEBUG:teuthology.orchestra.run.smithi184:> # Given a non-glob path, it better be there 2024-09-16T15:48:36.024 DEBUG:teuthology.orchestra.run.smithi184:> if "*" not in asok_path: 2024-09-16T15:48:36.024 DEBUG:teuthology.orchestra.run.smithi184:> assert(len(files) == 1) 2024-09-16T15:48:36.024 DEBUG:teuthology.orchestra.run.smithi184:> return files[0] 2024-09-16T15:48:36.024 DEBUG:teuthology.orchestra.run.smithi184:> 2024-09-16T15:48:36.024 DEBUG:teuthology.orchestra.run.smithi184:> for f in files: 2024-09-16T15:48:36.024 DEBUG:teuthology.orchestra.run.smithi184:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-09-16T15:48:36.024 DEBUG:teuthology.orchestra.run.smithi184:> if os.path.exists("/proc/{0}".format(pid)): 2024-09-16T15:48:36.024 DEBUG:teuthology.orchestra.run.smithi184:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-09-16T15:48:36.024 DEBUG:teuthology.orchestra.run.smithi184:> contents = proc_f.read() 2024-09-16T15:48:36.024 DEBUG:teuthology.orchestra.run.smithi184:> if mountpoint in contents: 2024-09-16T15:48:36.024 DEBUG:teuthology.orchestra.run.smithi184:> return f 2024-09-16T15:48:36.024 DEBUG:teuthology.orchestra.run.smithi184:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-09-16T15:48:36.024 DEBUG:teuthology.orchestra.run.smithi184:> 2024-09-16T15:48:36.025 DEBUG:teuthology.orchestra.run.smithi184:> print(_find_admin_socket("client.1")) 2024-09-16T15:48:36.025 DEBUG:teuthology.orchestra.run.smithi184:> ' 2024-09-16T15:48:36.148 INFO:teuthology.orchestra.run.smithi184.stdout:/var/run/ceph/ceph-client.1.35733.asok 2024-09-16T15:48:36.151 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:48:36 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-16T15:48:36.160 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.1.35733.asok 2024-09-16T15:48:36.160 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:48:36.161 DEBUG:teuthology.orchestra.run.smithi184:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.1.35733.asok status 2024-09-16T15:48:36.313 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-16T15:48:36.313 INFO:teuthology.orchestra.run.smithi184.stdout: "metadata": { 2024-09-16T15:48:36.313 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph_sha1": "33f95d2ae1bcbe7eb04404303f179eeeaa6982df", 2024-09-16T15:48:36.313 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph_version": "ceph version 19.3.0-4938-g33f95d2a (33f95d2ae1bcbe7eb04404303f179eeeaa6982df) squid (dev)", 2024-09-16T15:48:36.313 INFO:teuthology.orchestra.run.smithi184.stdout: "entity_id": "1", 2024-09-16T15:48:36.314 INFO:teuthology.orchestra.run.smithi184.stdout: "hostname": "smithi184", 2024-09-16T15:48:36.314 INFO:teuthology.orchestra.run.smithi184.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.1", 2024-09-16T15:48:36.314 INFO:teuthology.orchestra.run.smithi184.stdout: "pid": "35733", 2024-09-16T15:48:36.314 INFO:teuthology.orchestra.run.smithi184.stdout: "root": "/" 2024-09-16T15:48:36.314 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-16T15:48:36.314 INFO:teuthology.orchestra.run.smithi184.stdout: "dentry_count": 0, 2024-09-16T15:48:36.314 INFO:teuthology.orchestra.run.smithi184.stdout: "dentry_pinned_count": 0, 2024-09-16T15:48:36.314 INFO:teuthology.orchestra.run.smithi184.stdout: "id": 5462, 2024-09-16T15:48:36.314 INFO:teuthology.orchestra.run.smithi184.stdout: "inst": { 2024-09-16T15:48:36.314 INFO:teuthology.orchestra.run.smithi184.stdout: "name": { 2024-09-16T15:48:36.314 INFO:teuthology.orchestra.run.smithi184.stdout: "type": "client", 2024-09-16T15:48:36.314 INFO:teuthology.orchestra.run.smithi184.stdout: "num": 5462 2024-09-16T15:48:36.314 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-16T15:48:36.315 INFO:teuthology.orchestra.run.smithi184.stdout: "addr": { 2024-09-16T15:48:36.315 INFO:teuthology.orchestra.run.smithi184.stdout: "type": "v1", 2024-09-16T15:48:36.315 INFO:teuthology.orchestra.run.smithi184.stdout: "addr": "192.168.0.1:0", 2024-09-16T15:48:36.315 INFO:teuthology.orchestra.run.smithi184.stdout: "nonce": 3887985097 2024-09-16T15:48:36.315 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-16T15:48:36.315 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-16T15:48:36.315 INFO:teuthology.orchestra.run.smithi184.stdout: "addr": { 2024-09-16T15:48:36.315 INFO:teuthology.orchestra.run.smithi184.stdout: "type": "v1", 2024-09-16T15:48:36.315 INFO:teuthology.orchestra.run.smithi184.stdout: "addr": "192.168.0.1:0", 2024-09-16T15:48:36.315 INFO:teuthology.orchestra.run.smithi184.stdout: "nonce": 3887985097 2024-09-16T15:48:36.315 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-16T15:48:36.315 INFO:teuthology.orchestra.run.smithi184.stdout: "inst_str": "client.5462 192.168.0.1:0/3887985097", 2024-09-16T15:48:36.315 INFO:teuthology.orchestra.run.smithi184.stdout: "addr_str": "192.168.0.1:0/3887985097", 2024-09-16T15:48:36.316 INFO:teuthology.orchestra.run.smithi184.stdout: "inode_count": 1, 2024-09-16T15:48:36.316 INFO:teuthology.orchestra.run.smithi184.stdout: "mds_epoch": 68, 2024-09-16T15:48:36.316 INFO:teuthology.orchestra.run.smithi184.stdout: "osd_epoch": 104, 2024-09-16T15:48:36.316 INFO:teuthology.orchestra.run.smithi184.stdout: "osd_epoch_barrier": 0, 2024-09-16T15:48:36.316 INFO:teuthology.orchestra.run.smithi184.stdout: "blocklisted": false, 2024-09-16T15:48:36.316 INFO:teuthology.orchestra.run.smithi184.stdout: "fs_name": "cephfs" 2024-09-16T15:48:36.316 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-16T15:48:36.327 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:48:36.327 DEBUG:teuthology.orchestra.run.smithi184:> stat --file-system '--printf=%T 2024-09-16T15:48:36.327 DEBUG:teuthology.orchestra.run.smithi184:> ' -- /home/ubuntu/cephtest/mnt.1 2024-09-16T15:48:36.332 INFO:teuthology.orchestra.run.smithi184.stdout:fuseblk 2024-09-16T15:48:36.332 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2024-09-16T15:48:36.333 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:48:36.333 DEBUG:teuthology.orchestra.run.smithi184:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.1 2024-09-16T15:48:36.384 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs flag set enable_multiple true --yes-i-really-mean-it 2024-09-16T15:48:36.528 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:36.527+0000 7f1384f33640 1 -- 172.21.15.123:0/2782760396 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1368003fe0 msgr2=0x7f1368004810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:48:36.528 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:36.527+0000 7f1384f33640 1 --2- 172.21.15.123:0/2782760396 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1368003fe0 0x7f1368004810 secure :-1 s=READY pgs=1100 cs=0 l=1 rev1=1 crypto rx=0x7f137400c310 tx=0x7f137401cfc0 comp rx=0 tx=0).stop 2024-09-16T15:48:36.528 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:36.527+0000 7f137ed9d640 1 -- 172.21.15.123:0/2782760396 reap_dead start 2024-09-16T15:48:36.528 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:36.527+0000 7f1384f33640 1 -- 172.21.15.123:0/2782760396 shutdown_connections 2024-09-16T15:48:36.528 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:36.527+0000 7f1384f33640 1 -- 172.21.15.123:0/2782760396 >> 172.21.15.123:0/2782760396 conn(0x7f13800b1630 msgr2=0x7f13800b1a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:48:36.528 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:36.527+0000 7f1384f33640 1 -- 172.21.15.123:0/2782760396 shutdown_connections 2024-09-16T15:48:36.528 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:36.527+0000 7f1384f33640 1 -- 172.21.15.123:0/2782760396 wait complete. 2024-09-16T15:48:36.529 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:36.527+0000 7f1384f33640 1 Processor -- start 2024-09-16T15:48:36.529 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:36.527+0000 7f1384f33640 1 -- start start 2024-09-16T15:48:36.530 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:36.531+0000 7f1384f33640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1368003fe0 0x7f13801f5dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:48:36.530 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:36.531+0000 7f1384f33640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f13801f5440 0x7f1380155e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:48:36.530 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:36.531+0000 7f1384f33640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f138014e5b0 0x7f138014ea60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:48:36.530 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:36.531+0000 7f1384f33640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f13800cf0b0 con 0x7f138014e5b0 2024-09-16T15:48:36.531 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:36.531+0000 7f1384f33640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f13800cef30 con 0x7f1368003fe0 2024-09-16T15:48:36.531 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:36.531+0000 7f1384f33640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f13800cf230 con 0x7f13801f5440 2024-09-16T15:48:36.531 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:36.531+0000 7f137ed9d640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f138014e5b0 0x7f138014ea60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:48:36.531 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:36.531+0000 7f137ed9d640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f138014e5b0 0x7f138014ea60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:55136/0 (socket says 172.21.15.123:55136) 2024-09-16T15:48:36.531 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:36.531+0000 7f137ed9d640 1 -- 172.21.15.123:0/1616200921 learned_addr learned my addr 172.21.15.123:0/1616200921 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:48:36.531 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:36.531+0000 7f137dd9b640 1 --2- 172.21.15.123:0/1616200921 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f13801f5440 0x7f1380155e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:48:36.531 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:36.531+0000 7f137e59c640 1 --2- 172.21.15.123:0/1616200921 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1368003fe0 0x7f13801f5dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:48:36.531 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:36.531+0000 7f137ed9d640 1 -- 172.21.15.123:0/1616200921 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f13801f5440 msgr2=0x7f1380155e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:48:36.531 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:36.531+0000 7f137ed9d640 1 --2- 172.21.15.123:0/1616200921 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f13801f5440 0x7f1380155e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:36.531 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:36.531+0000 7f137ed9d640 1 -- 172.21.15.123:0/1616200921 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1368003fe0 msgr2=0x7f13801f5dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:48:36.531 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:36.531+0000 7f137ed9d640 1 --2- 172.21.15.123:0/1616200921 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1368003fe0 0x7f13801f5dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:36.531 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:36.531+0000 7f137ed9d640 1 -- 172.21.15.123:0/1616200921 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f138014efa0 con 0x7f138014e5b0 2024-09-16T15:48:36.531 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:36.531+0000 7f137ed9d640 1 --2- 172.21.15.123:0/1616200921 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f138014e5b0 0x7f138014ea60 secure :-1 s=READY pgs=1101 cs=0 l=1 rev1=1 crypto rx=0x7f13800b4d60 tx=0x7f137800fe20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:48:36.533 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:36.531+0000 7f136effd640 1 -- 172.21.15.123:0/1616200921 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1378004d30 con 0x7f138014e5b0 2024-09-16T15:48:36.533 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:36.531+0000 7f136effd640 1 -- 172.21.15.123:0/1616200921 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f137800db40 con 0x7f138014e5b0 2024-09-16T15:48:36.533 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:36.531+0000 7f1384f33640 1 -- 172.21.15.123:0/1616200921 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f13801577f0 con 0x7f138014e5b0 2024-09-16T15:48:36.533 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:36.531+0000 7f136effd640 1 -- 172.21.15.123:0/1616200921 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1378005020 con 0x7f138014e5b0 2024-09-16T15:48:36.533 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:36.531+0000 7f1384f33640 1 -- 172.21.15.123:0/1616200921 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1380156a30 con 0x7f138014e5b0 2024-09-16T15:48:36.534 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:36.535+0000 7f1384f33640 1 -- 172.21.15.123:0/1616200921 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f13800d0750 con 0x7f138014e5b0 2024-09-16T15:48:36.534 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:36.535+0000 7f136effd640 1 -- 172.21.15.123:0/1616200921 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 16) ==== 97182+0+0 (secure 0 0 0) 0x7f13780051c0 con 0x7f138014e5b0 2024-09-16T15:48:36.535 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:36.535+0000 7f136effd640 1 --2- 172.21.15.123:0/1616200921 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f1360073010 0x7f13600754d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:48:36.535 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:36.535+0000 7f136effd640 1 -- 172.21.15.123:0/1616200921 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(105..105 src has 1..105) ==== 5767+0+0 (secure 0 0 0) 0x7f13780947e0 con 0x7f138014e5b0 2024-09-16T15:48:36.535 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:36.535+0000 7f137e59c640 1 --2- 172.21.15.123:0/1616200921 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f1360073010 0x7f13600754d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:48:36.535 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:36.535+0000 7f137e59c640 1 --2- 172.21.15.123:0/1616200921 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f1360073010 0x7f13600754d0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f1370009a90 tx=0x7f1370005750 comp rx=0 tx=0).ready entity=mgr.4150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:48:36.539 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:36.539+0000 7f136effd640 1 -- 172.21.15.123:0/1616200921 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f1378020030 con 0x7f138014e5b0 2024-09-16T15:48:36.750 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:36.751+0000 7f1384f33640 1 -- 172.21.15.123:0/1616200921 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "fs flag set", "flag_name": "enable_multiple", "val": "true", "yes_i_really_mean_it": true} v 0) -- 0x7f13800d0470 con 0x7f138014e5b0 2024-09-16T15:48:37.540 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:37.539+0000 7f136effd640 1 -- 172.21.15.123:0/1616200921 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "fs flag set", "flag_name": "enable_multiple", "val": "true", "yes_i_really_mean_it": true}]=0 v69) ==== 136+0+0 (secure 0 0 0) 0x7f137805f7d0 con 0x7f138014e5b0 2024-09-16T15:48:37.544 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:37.543+0000 7f1384f33640 1 -- 172.21.15.123:0/1616200921 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f1360073010 msgr2=0x7f13600754d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:48:37.544 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:37.543+0000 7f1384f33640 1 --2- 172.21.15.123:0/1616200921 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f1360073010 0x7f13600754d0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f1370009a90 tx=0x7f1370005750 comp rx=0 tx=0).stop 2024-09-16T15:48:37.544 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:37.543+0000 7f1384f33640 1 -- 172.21.15.123:0/1616200921 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f138014e5b0 msgr2=0x7f138014ea60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:48:37.544 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:37.543+0000 7f1384f33640 1 --2- 172.21.15.123:0/1616200921 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f138014e5b0 0x7f138014ea60 secure :-1 s=READY pgs=1101 cs=0 l=1 rev1=1 crypto rx=0x7f13800b4d60 tx=0x7f137800fe20 comp rx=0 tx=0).stop 2024-09-16T15:48:37.544 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:37.543+0000 7f1384f33640 1 -- 172.21.15.123:0/1616200921 shutdown_connections 2024-09-16T15:48:37.544 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:37.543+0000 7f1384f33640 1 --2- 172.21.15.123:0/1616200921 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f1360073010 0x7f13600754d0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:37.545 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:37.543+0000 7f1384f33640 1 --2- 172.21.15.123:0/1616200921 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f138014e5b0 0x7f138014ea60 unknown :-1 s=CLOSED pgs=1101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:37.545 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:37.543+0000 7f1384f33640 1 --2- 172.21.15.123:0/1616200921 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f13801f5440 0x7f1380155e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:37.545 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:37.543+0000 7f1384f33640 1 --2- 172.21.15.123:0/1616200921 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1368003fe0 0x7f13801f5dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:37.545 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:37.543+0000 7f1384f33640 1 -- 172.21.15.123:0/1616200921 >> 172.21.15.123:0/1616200921 conn(0x7f13800b1630 msgr2=0x7f1380158ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:48:37.545 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:37.543+0000 7f1384f33640 1 -- 172.21.15.123:0/1616200921 shutdown_connections 2024-09-16T15:48:37.545 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:37.543+0000 7f1384f33640 1 -- 172.21.15.123:0/1616200921 wait complete. 2024-09-16T15:48:37.563 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2024-09-16T15:48:37.692 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:37.691+0000 7f6ccfcca640 1 -- 172.21.15.123:0/1919567915 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6cc80749a0 msgr2=0x7f6cc806edf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:48:37.693 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:37.691+0000 7f6ccfcca640 1 --2- 172.21.15.123:0/1919567915 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6cc80749a0 0x7f6cc806edf0 secure :-1 s=READY pgs=1102 cs=0 l=1 rev1=1 crypto rx=0x7f6cc4009900 tx=0x7f6cc401c7c0 comp rx=0 tx=0).stop 2024-09-16T15:48:37.693 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:37.691+0000 7f6ccfcca640 1 -- 172.21.15.123:0/1919567915 shutdown_connections 2024-09-16T15:48:37.693 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:37.691+0000 7f6ccfcca640 1 --2- 172.21.15.123:0/1919567915 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f6cc80af3f0 0x7f6cc806f330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:37.693 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:37.691+0000 7f6ccfcca640 1 --2- 172.21.15.123:0/1919567915 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6cc80749a0 0x7f6cc806edf0 unknown :-1 s=CLOSED pgs=1102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:37.693 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:37.691+0000 7f6ccfcca640 1 -- 172.21.15.123:0/1919567915 >> 172.21.15.123:0/1919567915 conn(0x7f6cc80cc980 msgr2=0x7f6cc80ccd90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:48:37.693 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:37.691+0000 7f6ccfcca640 1 -- 172.21.15.123:0/1919567915 shutdown_connections 2024-09-16T15:48:37.693 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:37.691+0000 7f6ccfcca640 1 -- 172.21.15.123:0/1919567915 wait complete. 2024-09-16T15:48:37.693 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:37.695+0000 7f6ccfcca640 1 Processor -- start 2024-09-16T15:48:37.694 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:37.695+0000 7f6ccfcca640 1 -- start start 2024-09-16T15:48:37.696 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:37.695+0000 7f6ccfcca640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6cc80749a0 0x7f6cc81584a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:48:37.696 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:37.695+0000 7f6ccfcca640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6cc80af3f0 0x7f6cc8159b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:48:37.696 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:37.695+0000 7f6ccfcca640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f6cc80cbd10 0x7f6cc80c73d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:48:37.696 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:37.695+0000 7f6ccfcca640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f6cc816b160 con 0x7f6cc80af3f0 2024-09-16T15:48:37.696 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:37.695+0000 7f6ccfcca640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f6cc816afe0 con 0x7f6cc80749a0 2024-09-16T15:48:37.696 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:37.695+0000 7f6ccfcca640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f6cc816b2e0 con 0x7f6cc80cbd10 2024-09-16T15:48:37.696 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:37.695+0000 7f6ccd265640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6cc80af3f0 0x7f6cc8159b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:48:37.696 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:37.695+0000 7f6ccd265640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6cc80af3f0 0x7f6cc8159b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:55152/0 (socket says 172.21.15.123:55152) 2024-09-16T15:48:37.696 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:37.695+0000 7f6ccd265640 1 -- 172.21.15.123:0/3435084576 learned_addr learned my addr 172.21.15.123:0/3435084576 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:48:37.696 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:37.695+0000 7f6cce267640 1 --2- 172.21.15.123:0/3435084576 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f6cc80cbd10 0x7f6cc80c73d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:48:37.696 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:37.695+0000 7f6ccda66640 1 --2- 172.21.15.123:0/3435084576 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6cc80749a0 0x7f6cc81584a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:48:37.696 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:37.695+0000 7f6ccd265640 1 -- 172.21.15.123:0/3435084576 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f6cc80cbd10 msgr2=0x7f6cc80c73d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:48:37.696 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:37.695+0000 7f6ccd265640 1 --2- 172.21.15.123:0/3435084576 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f6cc80cbd10 0x7f6cc80c73d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:37.697 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:37.695+0000 7f6ccd265640 1 -- 172.21.15.123:0/3435084576 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6cc80749a0 msgr2=0x7f6cc81584a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:48:37.697 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:37.695+0000 7f6ccd265640 1 --2- 172.21.15.123:0/3435084576 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6cc80749a0 0x7f6cc81584a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:37.697 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:37.695+0000 7f6ccd265640 1 -- 172.21.15.123:0/3435084576 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6cc8158a80 con 0x7f6cc80af3f0 2024-09-16T15:48:37.697 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:37.695+0000 7f6ccda66640 1 --2- 172.21.15.123:0/3435084576 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6cc80749a0 0x7f6cc81584a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T15:48:37.697 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:37.695+0000 7f6ccd265640 1 --2- 172.21.15.123:0/3435084576 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6cc80af3f0 0x7f6cc8159b20 secure :-1 s=READY pgs=1103 cs=0 l=1 rev1=1 crypto rx=0x7f6cc401d060 tx=0x7f6cc4004040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:48:37.697 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:37.695+0000 7f6cbe7fc640 1 -- 172.21.15.123:0/3435084576 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6cc40b2040 con 0x7f6cc80af3f0 2024-09-16T15:48:37.697 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:37.695+0000 7f6cbe7fc640 1 -- 172.21.15.123:0/3435084576 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f6cc4005030 con 0x7f6cc80af3f0 2024-09-16T15:48:37.697 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:37.695+0000 7f6ccfcca640 1 -- 172.21.15.123:0/3435084576 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6cc8158d60 con 0x7f6cc80af3f0 2024-09-16T15:48:37.697 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:37.695+0000 7f6cbe7fc640 1 -- 172.21.15.123:0/3435084576 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6cc4005440 con 0x7f6cc80af3f0 2024-09-16T15:48:37.697 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:37.695+0000 7f6ccfcca640 1 -- 172.21.15.123:0/3435084576 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6cc8159320 con 0x7f6cc80af3f0 2024-09-16T15:48:37.698 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:37.695+0000 7f6ccfcca640 1 -- 172.21.15.123:0/3435084576 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6cc80cc0f0 con 0x7f6cc80af3f0 2024-09-16T15:48:37.698 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:37.695+0000 7f6cbe7fc640 1 -- 172.21.15.123:0/3435084576 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 16) ==== 97182+0+0 (secure 0 0 0) 0x7f6cc40046d0 con 0x7f6cc80af3f0 2024-09-16T15:48:37.698 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:37.699+0000 7f6cbe7fc640 1 --2- 172.21.15.123:0/3435084576 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f6cac073010 0x7f6cac0754d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:48:37.698 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:37.699+0000 7f6cbe7fc640 1 -- 172.21.15.123:0/3435084576 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(105..105 src has 1..105) ==== 5767+0+0 (secure 0 0 0) 0x7f6cc412dfc0 con 0x7f6cc80af3f0 2024-09-16T15:48:37.698 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:37.699+0000 7f6ccda66640 1 --2- 172.21.15.123:0/3435084576 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f6cac073010 0x7f6cac0754d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:48:37.698 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:37.699+0000 7f6ccda66640 1 --2- 172.21.15.123:0/3435084576 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f6cac073010 0x7f6cac0754d0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f6cb800a830 tx=0x7f6cb8008040 comp rx=0 tx=0).ready entity=mgr.4150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:48:37.701 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:37.699+0000 7f6cbe7fc640 1 -- 172.21.15.123:0/3435084576 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f6cc40164d0 con 0x7f6cc80af3f0 2024-09-16T15:48:37.897 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:37.895+0000 7f6ccfcca640 1 -- 172.21.15.123:0/3435084576 --> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) -- 0x7f6cc80cc520 con 0x7f6cac073010 2024-09-16T15:48:37.900 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:37.899+0000 7f6cbe7fc640 1 -- 172.21.15.123:0/3435084576 <== mgr.4150 v2:172.21.15.123:6836/2464759676 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+41 (secure 0 0 0) 0x7f6cc80cc520 con 0x7f6cac073010 2024-09-16T15:48:37.900 INFO:teuthology.orchestra.run.smithi123.stdout:[ 2024-09-16T15:48:37.900 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-09-16T15:48:37.901 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "cephfs" 2024-09-16T15:48:37.901 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-16T15:48:37.901 INFO:teuthology.orchestra.run.smithi123.stdout:] 2024-09-16T15:48:37.903 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:37.903+0000 7f6ccfcca640 1 -- 172.21.15.123:0/3435084576 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f6cac073010 msgr2=0x7f6cac0754d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:48:37.903 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:37.903+0000 7f6ccfcca640 1 --2- 172.21.15.123:0/3435084576 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f6cac073010 0x7f6cac0754d0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f6cb800a830 tx=0x7f6cb8008040 comp rx=0 tx=0).stop 2024-09-16T15:48:37.903 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:37.903+0000 7f6ccfcca640 1 -- 172.21.15.123:0/3435084576 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6cc80af3f0 msgr2=0x7f6cc8159b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:48:37.903 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:37.903+0000 7f6ccfcca640 1 --2- 172.21.15.123:0/3435084576 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6cc80af3f0 0x7f6cc8159b20 secure :-1 s=READY pgs=1103 cs=0 l=1 rev1=1 crypto rx=0x7f6cc401d060 tx=0x7f6cc4004040 comp rx=0 tx=0).stop 2024-09-16T15:48:37.903 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:37.903+0000 7f6ccfcca640 1 -- 172.21.15.123:0/3435084576 shutdown_connections 2024-09-16T15:48:37.903 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:37.903+0000 7f6ccfcca640 1 --2- 172.21.15.123:0/3435084576 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f6cac073010 0x7f6cac0754d0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:37.903 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:37.903+0000 7f6ccfcca640 1 --2- 172.21.15.123:0/3435084576 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f6cc80cbd10 0x7f6cc80c73d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:37.903 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:37.903+0000 7f6ccfcca640 1 --2- 172.21.15.123:0/3435084576 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6cc80af3f0 0x7f6cc8159b20 unknown :-1 s=CLOSED pgs=1103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:37.903 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:37.903+0000 7f6ccfcca640 1 --2- 172.21.15.123:0/3435084576 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6cc80749a0 0x7f6cc81584a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:37.903 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:37.903+0000 7f6ccfcca640 1 -- 172.21.15.123:0/3435084576 >> 172.21.15.123:0/3435084576 conn(0x7f6cc80cc980 msgr2=0x7f6cc815a500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:48:37.903 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:37.903+0000 7f6ccfcca640 1 -- 172.21.15.123:0/3435084576 shutdown_connections 2024-09-16T15:48:37.904 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:37.903+0000 7f6ccfcca640 1 -- 172.21.15.123:0/3435084576 wait complete. 2024-09-16T15:48:37.919 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-09-16T15:48:38.050 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:38.051+0000 7f4158cb4640 1 -- 172.21.15.123:0/813956209 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f41540b2d00 msgr2=0x7f41540aeae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:48:38.050 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:38.051+0000 7f4158cb4640 1 --2- 172.21.15.123:0/813956209 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f41540b2d00 0x7f41540aeae0 secure :-1 s=READY pgs=1104 cs=0 l=1 rev1=1 crypto rx=0x7f41540d2920 tx=0x7f414c01d7e0 comp rx=0 tx=0).stop 2024-09-16T15:48:38.050 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:38.051+0000 7f4158cb4640 1 -- 172.21.15.123:0/813956209 shutdown_connections 2024-09-16T15:48:38.050 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:38.051+0000 7f4158cb4640 1 --2- 172.21.15.123:0/813956209 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f41540b2d00 0x7f41540aeae0 unknown :-1 s=CLOSED pgs=1104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:38.051 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:38.051+0000 7f4158cb4640 1 --2- 172.21.15.123:0/813956209 >> v2:172.21.15.184:3300/0 conn(0x7f415415bae0 0x7f41540d1bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:38.051 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:38.051+0000 7f4158cb4640 1 -- 172.21.15.123:0/813956209 >> 172.21.15.123:0/813956209 conn(0x7f415409aa40 msgr2=0x7f415409ae50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:48:38.051 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:38.051+0000 7f4158cb4640 1 -- 172.21.15.123:0/813956209 shutdown_connections 2024-09-16T15:48:38.051 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:38.051+0000 7f4158cb4640 1 -- 172.21.15.123:0/813956209 wait complete. 2024-09-16T15:48:38.051 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:38.051+0000 7f4158cb4640 1 Processor -- start 2024-09-16T15:48:38.051 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:38.051+0000 7f4158cb4640 1 -- start start 2024-09-16T15:48:38.053 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:38.051+0000 7f4158cb4640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f41540b2d00 0x7f4154156230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:48:38.053 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:38.051+0000 7f4158cb4640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f41540d2100 0x7f415406f580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:48:38.053 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:38.051+0000 7f4158cb4640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f415415bae0 0x7f415414cdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:48:38.054 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:38.051+0000 7f4158cb4640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f41541689a0 con 0x7f41540d2100 2024-09-16T15:48:38.054 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:38.051+0000 7f4158cb4640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f41540afde0 con 0x7f41540b2d00 2024-09-16T15:48:38.054 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:38.051+0000 7f4158cb4640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f4154168b20 con 0x7f415415bae0 2024-09-16T15:48:38.054 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:38.051+0000 7f414bfff640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f41540d2100 0x7f415406f580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:48:38.054 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:38.051+0000 7f414bfff640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f41540d2100 0x7f415406f580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:55162/0 (socket says 172.21.15.123:55162) 2024-09-16T15:48:38.054 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:38.051+0000 7f414bfff640 1 -- 172.21.15.123:0/2191423036 learned_addr learned my addr 172.21.15.123:0/2191423036 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:48:38.054 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:38.051+0000 7f415259c640 1 --2- 172.21.15.123:0/2191423036 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f41540b2d00 0x7f4154156230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:48:38.054 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:38.051+0000 7f4152d9d640 1 --2- 172.21.15.123:0/2191423036 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f415415bae0 0x7f415414cdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:48:38.054 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:38.051+0000 7f414bfff640 1 -- 172.21.15.123:0/2191423036 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f415415bae0 msgr2=0x7f415414cdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:48:38.054 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:38.051+0000 7f414bfff640 1 --2- 172.21.15.123:0/2191423036 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f415415bae0 0x7f415414cdd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:38.054 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:38.051+0000 7f414bfff640 1 -- 172.21.15.123:0/2191423036 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f41540b2d00 msgr2=0x7f4154156230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:48:38.054 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:38.051+0000 7f414bfff640 1 --2- 172.21.15.123:0/2191423036 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f41540b2d00 0x7f4154156230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:38.054 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:38.051+0000 7f414bfff640 1 -- 172.21.15.123:0/2191423036 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f41541fbb00 con 0x7f41540d2100 2024-09-16T15:48:38.054 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:38.051+0000 7f415259c640 1 --2- 172.21.15.123:0/2191423036 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f41540b2d00 0x7f4154156230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T15:48:38.054 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:38.051+0000 7f414bfff640 1 --2- 172.21.15.123:0/2191423036 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f41540d2100 0x7f415406f580 secure :-1 s=READY pgs=1105 cs=0 l=1 rev1=1 crypto rx=0x7f413c00c490 tx=0x7f413c001e80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:48:38.055 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:38.051+0000 7f414affd640 1 -- 172.21.15.123:0/2191423036 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f413c017070 con 0x7f41540d2100 2024-09-16T15:48:38.055 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:38.051+0000 7f414affd640 1 -- 172.21.15.123:0/2191423036 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f413c003600 con 0x7f41540d2100 2024-09-16T15:48:38.055 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:38.051+0000 7f4158cb4640 1 -- 172.21.15.123:0/2191423036 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f41541fad40 con 0x7f41540d2100 2024-09-16T15:48:38.055 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:38.051+0000 7f414affd640 1 -- 172.21.15.123:0/2191423036 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f413c003a90 con 0x7f41540d2100 2024-09-16T15:48:38.055 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:38.051+0000 7f4158cb4640 1 -- 172.21.15.123:0/2191423036 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f41541fb300 con 0x7f41540d2100 2024-09-16T15:48:38.056 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:38.055+0000 7f4158cb4640 1 -- 172.21.15.123:0/2191423036 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f41540aeae0 con 0x7f41540d2100 2024-09-16T15:48:38.056 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:38.055+0000 7f414affd640 1 -- 172.21.15.123:0/2191423036 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 16) ==== 97182+0+0 (secure 0 0 0) 0x7f413c01a070 con 0x7f41540d2100 2024-09-16T15:48:38.056 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:38.055+0000 7f414affd640 1 --2- 172.21.15.123:0/2191423036 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f4138072fe0 0x7f41380754a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:48:38.056 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:38.055+0000 7f415259c640 1 --2- 172.21.15.123:0/2191423036 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f4138072fe0 0x7f41380754a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:48:38.056 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:38.055+0000 7f414affd640 1 -- 172.21.15.123:0/2191423036 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(105..105 src has 1..105) ==== 5767+0+0 (secure 0 0 0) 0x7f413c093740 con 0x7f41540d2100 2024-09-16T15:48:38.056 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:38.055+0000 7f415259c640 1 --2- 172.21.15.123:0/2191423036 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f4138072fe0 0x7f41380754a0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f41440058d0 tx=0x7f4144005cc0 comp rx=0 tx=0).ready entity=mgr.4150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:48:38.059 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:38.059+0000 7f414affd640 1 -- 172.21.15.123:0/2191423036 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f413c021020 con 0x7f41540d2100 2024-09-16T15:48:38.224 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:38.223+0000 7f4158cb4640 1 -- 172.21.15.123:0/2191423036 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) -- 0x7f41540c5470 con 0x7f41540d2100 2024-09-16T15:48:38.225 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:38.223+0000 7f414affd640 1 -- 172.21.15.123:0/2191423036 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v38)=0 v38) ==== 122+0+0 (secure 0 0 0) 0x7f413c05e8a0 con 0x7f41540d2100 2024-09-16T15:48:38.227 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:38.227+0000 7f4158cb4640 1 -- 172.21.15.123:0/2191423036 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f4138072fe0 msgr2=0x7f41380754a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:48:38.227 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:38.227+0000 7f4158cb4640 1 --2- 172.21.15.123:0/2191423036 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f4138072fe0 0x7f41380754a0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f41440058d0 tx=0x7f4144005cc0 comp rx=0 tx=0).stop 2024-09-16T15:48:38.228 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:38.227+0000 7f4158cb4640 1 -- 172.21.15.123:0/2191423036 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f41540d2100 msgr2=0x7f415406f580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:48:38.228 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:38.227+0000 7f4158cb4640 1 --2- 172.21.15.123:0/2191423036 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f41540d2100 0x7f415406f580 secure :-1 s=READY pgs=1105 cs=0 l=1 rev1=1 crypto rx=0x7f413c00c490 tx=0x7f413c001e80 comp rx=0 tx=0).stop 2024-09-16T15:48:38.228 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:38.227+0000 7f4158cb4640 1 -- 172.21.15.123:0/2191423036 shutdown_connections 2024-09-16T15:48:38.228 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:38.227+0000 7f4158cb4640 1 --2- 172.21.15.123:0/2191423036 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f4138072fe0 0x7f41380754a0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:38.228 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:38.227+0000 7f4158cb4640 1 --2- 172.21.15.123:0/2191423036 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f415415bae0 0x7f415414cdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:38.228 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:38.227+0000 7f4158cb4640 1 --2- 172.21.15.123:0/2191423036 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f41540d2100 0x7f415406f580 unknown :-1 s=CLOSED pgs=1105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:38.228 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:38.227+0000 7f4158cb4640 1 --2- 172.21.15.123:0/2191423036 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f41540b2d00 0x7f4154156230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:38.228 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:38.227+0000 7f4158cb4640 1 -- 172.21.15.123:0/2191423036 >> 172.21.15.123:0/2191423036 conn(0x7f415409aa40 msgr2=0x7f41540b3f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:48:38.228 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:38.227+0000 7f4158cb4640 1 -- 172.21.15.123:0/2191423036 shutdown_connections 2024-09-16T15:48:38.228 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:38.227+0000 7f4158cb4640 1 -- 172.21.15.123:0/2191423036 wait complete. 2024-09-16T15:48:38.243 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:48:38.243 DEBUG:teuthology.orchestra.run.smithi123:> (cd /home/ubuntu/cephtest/mnt.0 && exec sudo mkdir -p volumes/_nogroup/subvol37810/b00a5d9c-1a58-42eb-be9a-5f5c5fe23179) 2024-09-16T15:48:38.262 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:48:38.263 DEBUG:teuthology.orchestra.run.smithi123:> (cd /home/ubuntu/cephtest/mnt.0 && exec sudo mkdir -p volumes/_nogroup/subvol37810/b00a5d9c-1a58-42eb-be9a-5f5c5fe23179/.snap/fake) 2024-09-16T15:48:38.324 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:48:38.324 DEBUG:teuthology.orchestra.run.smithi123:> (cd /home/ubuntu/cephtest/mnt.0 && exec getfattr --only-values -n ceph.dir.layout.pool .) 2024-09-16T15:48:38.374 INFO:teuthology.orchestra.run.smithi123.stdout:cephfs_data 2024-09-16T15:48:38.374 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:48:38.374 DEBUG:teuthology.orchestra.run.smithi123:> (cd /home/ubuntu/cephtest/mnt.0 && exec sudo setfattr -n ceph.dir.layout.pool -v cephfs_data volumes/_nogroup/subvol37810/b00a5d9c-1a58-42eb-be9a-5f5c5fe23179) 2024-09-16T15:48:38.435 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2024-09-16T15:48:38.435 DEBUG:teuthology.orchestra.run.smithi123:> sudo dd of=/home/ubuntu/cephtest/mnt.0/volumes/_nogroup/subvol37810/.meta 2024-09-16T15:48:38.495 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs subvol37810 2024-09-16T15:48:38.667 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:38.667+0000 7fa417477640 1 -- 172.21.15.123:0/1991916523 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa410072720 msgr2=0x7fa4100cabc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:48:38.667 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:38.667+0000 7fa417477640 1 --2- 172.21.15.123:0/1991916523 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa410072720 0x7fa4100cabc0 secure :-1 s=READY pgs=1106 cs=0 l=1 rev1=1 crypto rx=0x7fa41006e9b0 tx=0x7fa40c01c8c0 comp rx=0 tx=0).stop 2024-09-16T15:48:38.667 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:38.667+0000 7fa417477640 1 -- 172.21.15.123:0/1991916523 shutdown_connections 2024-09-16T15:48:38.667 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:38.667+0000 7fa417477640 1 --2- 172.21.15.123:0/1991916523 >> v2:172.21.15.184:3300/0 conn(0x7fa4100cc3c0 0x7fa4100cc870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:38.667 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:38.667+0000 7fa417477640 1 --2- 172.21.15.123:0/1991916523 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa410072720 0x7fa4100cabc0 unknown :-1 s=CLOSED pgs=1106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:38.667 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:38.667+0000 7fa417477640 1 --2- 172.21.15.123:0/1991916523 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fa41015aa00 0x7fa41015e5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:38.667 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:38.667+0000 7fa417477640 1 -- 172.21.15.123:0/1991916523 >> 172.21.15.123:0/1991916523 conn(0x7fa4100b1710 msgr2=0x7fa4100b1b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:48:38.667 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:38.667+0000 7fa417477640 1 -- 172.21.15.123:0/1991916523 shutdown_connections 2024-09-16T15:48:38.667 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:38.667+0000 7fa417477640 1 -- 172.21.15.123:0/1991916523 wait complete. 2024-09-16T15:48:38.668 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:38.667+0000 7fa417477640 1 Processor -- start 2024-09-16T15:48:38.668 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:38.667+0000 7fa417477640 1 -- start start 2024-09-16T15:48:38.670 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:38.667+0000 7fa417477640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fa410072720 0x7fa4100c6c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:48:38.670 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:38.667+0000 7fa417477640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa4100cc3c0 0x7fa4100c6520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:48:38.671 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:38.667+0000 7fa417477640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa41015aa00 0x7fa41014dbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:48:38.671 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:38.667+0000 7fa417477640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fa4100cd710 con 0x7fa41015aa00 2024-09-16T15:48:38.671 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:38.667+0000 7fa417477640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fa41006ff20 con 0x7fa4100cc3c0 2024-09-16T15:48:38.671 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:38.667+0000 7fa417477640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fa4100cd890 con 0x7fa410072720 2024-09-16T15:48:38.671 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:38.667+0000 7fa415a14640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa41015aa00 0x7fa41014dbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:48:38.671 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:38.667+0000 7fa415a14640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa41015aa00 0x7fa41014dbb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:55178/0 (socket says 172.21.15.123:55178) 2024-09-16T15:48:38.671 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:38.667+0000 7fa415a14640 1 -- 172.21.15.123:0/3368437203 learned_addr learned my addr 172.21.15.123:0/3368437203 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:48:38.671 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:38.671+0000 7fa415213640 1 --2- 172.21.15.123:0/3368437203 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fa410072720 0x7fa4100c6c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:48:38.671 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:38.671+0000 7fa414a12640 1 --2- 172.21.15.123:0/3368437203 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa4100cc3c0 0x7fa4100c6520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:48:38.671 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:38.671+0000 7fa415a14640 1 -- 172.21.15.123:0/3368437203 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fa410072720 msgr2=0x7fa4100c6c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:48:38.672 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:38.671+0000 7fa415a14640 1 --2- 172.21.15.123:0/3368437203 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fa410072720 0x7fa4100c6c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:38.672 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:38.671+0000 7fa415a14640 1 -- 172.21.15.123:0/3368437203 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa4100cc3c0 msgr2=0x7fa4100c6520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:48:38.672 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:38.671+0000 7fa415a14640 1 --2- 172.21.15.123:0/3368437203 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa4100cc3c0 0x7fa4100c6520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:38.672 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:38.671+0000 7fa415a14640 1 -- 172.21.15.123:0/3368437203 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa4101f7a80 con 0x7fa41015aa00 2024-09-16T15:48:38.672 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:38.671+0000 7fa415213640 1 --2- 172.21.15.123:0/3368437203 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fa410072720 0x7fa4100c6c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T15:48:38.672 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:38.671+0000 7fa415a14640 1 --2- 172.21.15.123:0/3368437203 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa41015aa00 0x7fa41014dbb0 secure :-1 s=READY pgs=1107 cs=0 l=1 rev1=1 crypto rx=0x7fa40800b970 tx=0x7fa40800be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:48:38.672 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:38.671+0000 7fa405ffb640 1 -- 172.21.15.123:0/3368437203 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa408019070 con 0x7fa41015aa00 2024-09-16T15:48:38.672 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:38.671+0000 7fa405ffb640 1 -- 172.21.15.123:0/3368437203 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fa40800cc50 con 0x7fa41015aa00 2024-09-16T15:48:38.672 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:38.671+0000 7fa417477640 1 -- 172.21.15.123:0/3368437203 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa4100c97e0 con 0x7fa41015aa00 2024-09-16T15:48:38.672 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:38.671+0000 7fa405ffb640 1 -- 172.21.15.123:0/3368437203 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4080146a0 con 0x7fa41015aa00 2024-09-16T15:48:38.673 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:38.671+0000 7fa417477640 1 -- 172.21.15.123:0/3368437203 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa41014e0f0 con 0x7fa41015aa00 2024-09-16T15:48:38.673 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:38.671+0000 7fa417477640 1 -- 172.21.15.123:0/3368437203 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa4100c95d0 con 0x7fa41015aa00 2024-09-16T15:48:38.673 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:38.671+0000 7fa405ffb640 1 -- 172.21.15.123:0/3368437203 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 16) ==== 97182+0+0 (secure 0 0 0) 0x7fa40801c070 con 0x7fa41015aa00 2024-09-16T15:48:38.673 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:38.671+0000 7fa405ffb640 1 --2- 172.21.15.123:0/3368437203 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fa3f4072fe0 0x7fa3f40754a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:48:38.674 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:38.671+0000 7fa405ffb640 1 -- 172.21.15.123:0/3368437203 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(105..105 src has 1..105) ==== 5767+0+0 (secure 0 0 0) 0x7fa408093da0 con 0x7fa41015aa00 2024-09-16T15:48:38.674 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:38.671+0000 7fa415213640 1 --2- 172.21.15.123:0/3368437203 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fa3f4072fe0 0x7fa3f40754a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:48:38.674 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:38.671+0000 7fa415213640 1 --2- 172.21.15.123:0/3368437203 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fa3f4072fe0 0x7fa3f40754a0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fa40000a870 tx=0x7fa400008040 comp rx=0 tx=0).ready entity=mgr.4150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:48:38.676 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:38.675+0000 7fa405ffb640 1 -- 172.21.15.123:0/3368437203 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7fa40805ee20 con 0x7fa41015aa00 2024-09-16T15:48:38.871 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:38.871+0000 7fa417477640 1 -- 172.21.15.123:0/3368437203 --> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "subvol37810", "target": ["mon-mgr", ""]}) -- 0x7fa4100cc870 con 0x7fa3f4072fe0 2024-09-16T15:48:38.878 INFO:tasks.ceph.mgr.y.smithi123.stderr:2024-09-16T15:48:38.879+0000 7fa9b36f4640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-16T15:48:38.878 INFO:tasks.ceph.mgr.y.smithi123.stderr:2024-09-16T15:48:38.879+0000 7fa9b36f4640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-16T15:48:38.878 INFO:tasks.ceph.mgr.y.smithi123.stderr:2024-09-16T15:48:38.879+0000 7fa9b36f4640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-16T15:48:38.878 INFO:tasks.ceph.mgr.y.smithi123.stderr:2024-09-16T15:48:38.879+0000 7fa9b36f4640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-16T15:48:38.879 INFO:tasks.ceph.mgr.y.smithi123.stderr:2024-09-16T15:48:38.879+0000 7fa9b36f4640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-16T15:48:39.055 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:39.055+0000 7fa405ffb640 1 -- 172.21.15.123:0/3368437203 <== mgr.4150 v2:172.21.15.123:6836/2464759676 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+67 (secure 0 0 0) 0x7fa4100cc870 con 0x7fa3f4072fe0 2024-09-16T15:48:39.055 INFO:teuthology.orchestra.run.smithi123.stdout:/volumes/_nogroup/subvol37810/b00a5d9c-1a58-42eb-be9a-5f5c5fe23179 2024-09-16T15:48:39.059 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:39.059+0000 7fa417477640 1 -- 172.21.15.123:0/3368437203 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fa3f4072fe0 msgr2=0x7fa3f40754a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:48:39.059 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:39.059+0000 7fa417477640 1 --2- 172.21.15.123:0/3368437203 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fa3f4072fe0 0x7fa3f40754a0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fa40000a870 tx=0x7fa400008040 comp rx=0 tx=0).stop 2024-09-16T15:48:39.059 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:39.059+0000 7fa417477640 1 -- 172.21.15.123:0/3368437203 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa41015aa00 msgr2=0x7fa41014dbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:48:39.059 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:39.059+0000 7fa417477640 1 --2- 172.21.15.123:0/3368437203 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa41015aa00 0x7fa41014dbb0 secure :-1 s=READY pgs=1107 cs=0 l=1 rev1=1 crypto rx=0x7fa40800b970 tx=0x7fa40800be30 comp rx=0 tx=0).stop 2024-09-16T15:48:39.059 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:39.059+0000 7fa417477640 1 -- 172.21.15.123:0/3368437203 shutdown_connections 2024-09-16T15:48:39.059 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:39.059+0000 7fa417477640 1 --2- 172.21.15.123:0/3368437203 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fa3f4072fe0 0x7fa3f40754a0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:39.059 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:39.059+0000 7fa417477640 1 --2- 172.21.15.123:0/3368437203 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa41015aa00 0x7fa41014dbb0 unknown :-1 s=CLOSED pgs=1107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:39.059 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:39.059+0000 7fa417477640 1 --2- 172.21.15.123:0/3368437203 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa4100cc3c0 0x7fa4100c6520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:39.059 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:39.059+0000 7fa417477640 1 --2- 172.21.15.123:0/3368437203 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fa410072720 0x7fa4100c6c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:39.060 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:39.059+0000 7fa417477640 1 -- 172.21.15.123:0/3368437203 >> 172.21.15.123:0/3368437203 conn(0x7fa4100b1710 msgr2=0x7fa41015b050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:48:39.060 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:39.059+0000 7fa417477640 1 -- 172.21.15.123:0/3368437203 shutdown_connections 2024-09-16T15:48:39.060 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:39.059+0000 7fa417477640 1 -- 172.21.15.123:0/3368437203 wait complete. 2024-09-16T15:48:39.077 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume ls cephfs 2024-09-16T15:48:39.202 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:39.203+0000 7f6d18300640 1 -- 172.21.15.123:0/3720303484 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6cfc0052a0 msgr2=0x7f6cfc00b9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:48:39.202 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:39.203+0000 7f6d18300640 1 --2- 172.21.15.123:0/3720303484 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6cfc0052a0 0x7f6cfc00b9d0 secure :-1 s=READY pgs=1111 cs=0 l=1 rev1=1 crypto rx=0x7f6d080043e0 tx=0x7f6d0801ca60 comp rx=0 tx=0).stop 2024-09-16T15:48:39.203 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:39.203+0000 7f6d18300640 1 -- 172.21.15.123:0/3720303484 shutdown_connections 2024-09-16T15:48:39.203 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:39.203+0000 7f6d18300640 1 --2- 172.21.15.123:0/3720303484 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6cfc0052a0 0x7f6cfc00b9d0 unknown :-1 s=CLOSED pgs=1111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:39.203 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:39.203+0000 7f6d18300640 1 -- 172.21.15.123:0/3720303484 >> 172.21.15.123:0/3720303484 conn(0x7f6d100741a0 msgr2=0x7f6d100745b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:48:39.203 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:39.203+0000 7f6d18300640 1 -- 172.21.15.123:0/3720303484 shutdown_connections 2024-09-16T15:48:39.203 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:39.203+0000 7f6d18300640 1 -- 172.21.15.123:0/3720303484 wait complete. 2024-09-16T15:48:39.203 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:39.203+0000 7f6d18300640 1 Processor -- start 2024-09-16T15:48:39.204 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:39.203+0000 7f6d18300640 1 -- start start 2024-09-16T15:48:39.206 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:39.203+0000 7f6d18300640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6cfc0052a0 0x7f6d1015b920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:48:39.206 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:39.203+0000 7f6d18300640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f6d1015d320 0x7f6d10156090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:48:39.206 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:39.203+0000 7f6d18300640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6d10158eb0 0x7f6d100b41d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:48:39.206 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:39.203+0000 7f6d18300640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f6d100b0ff0 con 0x7f6d10158eb0 2024-09-16T15:48:39.206 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:39.203+0000 7f6d18300640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f6d100cdc10 con 0x7f6cfc0052a0 2024-09-16T15:48:39.206 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:39.203+0000 7f6d18300640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f6d100b1170 con 0x7f6d1015d320 2024-09-16T15:48:39.207 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:39.203+0000 7f6d1689d640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6d10158eb0 0x7f6d100b41d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:48:39.207 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:39.203+0000 7f6d1689d640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6d10158eb0 0x7f6d100b41d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:55208/0 (socket says 172.21.15.123:55208) 2024-09-16T15:48:39.207 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:39.203+0000 7f6d1689d640 1 -- 172.21.15.123:0/1436029531 learned_addr learned my addr 172.21.15.123:0/1436029531 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:48:39.207 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:39.203+0000 7f6d1609c640 1 --2- 172.21.15.123:0/1436029531 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6cfc0052a0 0x7f6d1015b920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:48:39.207 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:39.203+0000 7f6d1589b640 1 --2- 172.21.15.123:0/1436029531 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f6d1015d320 0x7f6d10156090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:48:39.207 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:39.203+0000 7f6d1689d640 1 -- 172.21.15.123:0/1436029531 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f6d1015d320 msgr2=0x7f6d10156090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:48:39.207 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:39.203+0000 7f6d1689d640 1 --2- 172.21.15.123:0/1436029531 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f6d1015d320 0x7f6d10156090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:39.207 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:39.203+0000 7f6d1689d640 1 -- 172.21.15.123:0/1436029531 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6cfc0052a0 msgr2=0x7f6d1015b920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:48:39.207 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:39.203+0000 7f6d1689d640 1 --2- 172.21.15.123:0/1436029531 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6cfc0052a0 0x7f6d1015b920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:39.208 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:39.203+0000 7f6d1689d640 1 -- 172.21.15.123:0/1436029531 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6d1015ce40 con 0x7f6d10158eb0 2024-09-16T15:48:39.208 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:39.203+0000 7f6d1589b640 1 --2- 172.21.15.123:0/1436029531 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f6d1015d320 0x7f6d10156090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T15:48:39.208 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:39.203+0000 7f6d1689d640 1 --2- 172.21.15.123:0/1436029531 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6d10158eb0 0x7f6d100b41d0 secure :-1 s=READY pgs=1112 cs=0 l=1 rev1=1 crypto rx=0x7f6d08006040 tx=0x7f6d08006d30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:48:39.208 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:39.203+0000 7f6d06ffd640 1 -- 172.21.15.123:0/1436029531 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d080a5a60 con 0x7f6d10158eb0 2024-09-16T15:48:39.208 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:39.203+0000 7f6d06ffd640 1 -- 172.21.15.123:0/1436029531 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f6d080b3030 con 0x7f6d10158eb0 2024-09-16T15:48:39.208 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:39.203+0000 7f6d18300640 1 -- 172.21.15.123:0/1436029531 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6d1009b3c0 con 0x7f6d10158eb0 2024-09-16T15:48:39.208 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:39.203+0000 7f6d06ffd640 1 -- 172.21.15.123:0/1436029531 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d080060d0 con 0x7f6d10158eb0 2024-09-16T15:48:39.208 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:39.203+0000 7f6d18300640 1 -- 172.21.15.123:0/1436029531 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6d1009b0e0 con 0x7f6d10158eb0 2024-09-16T15:48:39.208 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:39.207+0000 7f6d18300640 1 -- 172.21.15.123:0/1436029531 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6d1015d010 con 0x7f6d10158eb0 2024-09-16T15:48:39.209 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:39.207+0000 7f6d06ffd640 1 -- 172.21.15.123:0/1436029531 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 16) ==== 97182+0+0 (secure 0 0 0) 0x7f6d08006270 con 0x7f6d10158eb0 2024-09-16T15:48:39.209 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:39.207+0000 7f6d06ffd640 1 --2- 172.21.15.123:0/1436029531 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f6cf4073010 0x7f6cf40754d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:48:39.209 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:39.207+0000 7f6d1609c640 1 --2- 172.21.15.123:0/1436029531 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f6cf4073010 0x7f6cf40754d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:48:39.209 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:39.207+0000 7f6d06ffd640 1 -- 172.21.15.123:0/1436029531 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(105..105 src has 1..105) ==== 5767+0+0 (secure 0 0 0) 0x7f6d0812de60 con 0x7f6d10158eb0 2024-09-16T15:48:39.209 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:39.207+0000 7f6d1609c640 1 --2- 172.21.15.123:0/1436029531 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f6cf4073010 0x7f6cf40754d0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f6d0c000fc0 tx=0x7f6d0c00c040 comp rx=0 tx=0).ready entity=mgr.4150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:48:39.211 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:39.211+0000 7f6d06ffd640 1 -- 172.21.15.123:0/1436029531 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f6d08016770 con 0x7f6d10158eb0 2024-09-16T15:48:39.408 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:39.407+0000 7f6d18300640 1 -- 172.21.15.123:0/1436029531 --> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] -- mgr_command(tid 0: {"prefix": "fs subvolume ls", "vol_name": "cephfs", "target": ["mon-mgr", ""]}) -- 0x7f6d1009abb0 con 0x7f6cf4073010 2024-09-16T15:48:39.415 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:39.415+0000 7f6d06ffd640 1 -- 172.21.15.123:0/1436029531 <== mgr.4150 v2:172.21.15.123:6836/2464759676 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+46 (secure 0 0 0) 0x7f6d1009abb0 con 0x7f6cf4073010 2024-09-16T15:48:39.415 INFO:teuthology.orchestra.run.smithi123.stdout:[ 2024-09-16T15:48:39.415 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-09-16T15:48:39.415 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "subvol37810" 2024-09-16T15:48:39.415 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-16T15:48:39.415 INFO:teuthology.orchestra.run.smithi123.stdout:] 2024-09-16T15:48:39.418 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:39.415+0000 7f6d18300640 1 -- 172.21.15.123:0/1436029531 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f6cf4073010 msgr2=0x7f6cf40754d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:48:39.418 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:39.415+0000 7f6d18300640 1 --2- 172.21.15.123:0/1436029531 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f6cf4073010 0x7f6cf40754d0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f6d0c000fc0 tx=0x7f6d0c00c040 comp rx=0 tx=0).stop 2024-09-16T15:48:39.418 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:39.415+0000 7f6d18300640 1 -- 172.21.15.123:0/1436029531 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6d10158eb0 msgr2=0x7f6d100b41d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:48:39.418 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:39.415+0000 7f6d18300640 1 --2- 172.21.15.123:0/1436029531 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6d10158eb0 0x7f6d100b41d0 secure :-1 s=READY pgs=1112 cs=0 l=1 rev1=1 crypto rx=0x7f6d08006040 tx=0x7f6d08006d30 comp rx=0 tx=0).stop 2024-09-16T15:48:39.418 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:39.415+0000 7f6d18300640 1 -- 172.21.15.123:0/1436029531 shutdown_connections 2024-09-16T15:48:39.418 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:39.415+0000 7f6d18300640 1 --2- 172.21.15.123:0/1436029531 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f6cf4073010 0x7f6cf40754d0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:39.418 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:39.415+0000 7f6d18300640 1 --2- 172.21.15.123:0/1436029531 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6d10158eb0 0x7f6d100b41d0 unknown :-1 s=CLOSED pgs=1112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:39.418 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:39.415+0000 7f6d18300640 1 --2- 172.21.15.123:0/1436029531 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f6d1015d320 0x7f6d10156090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:39.418 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:39.419+0000 7f6d18300640 1 --2- 172.21.15.123:0/1436029531 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6cfc0052a0 0x7f6d1015b920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:39.419 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:39.419+0000 7f6d18300640 1 -- 172.21.15.123:0/1436029531 >> 172.21.15.123:0/1436029531 conn(0x7f6d100741a0 msgr2=0x7f6d100ade90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:48:39.419 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:39.419+0000 7f6d18300640 1 -- 172.21.15.123:0/1436029531 shutdown_connections 2024-09-16T15:48:39.419 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:39.419+0000 7f6d18300640 1 -- 172.21.15.123:0/1436029531 wait complete. 2024-09-16T15:48:39.434 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume info cephfs subvol37810 2024-09-16T15:48:39.561 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:39.559+0000 7f1f4647e640 1 -- 172.21.15.123:0/2372812595 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f1f40071c60 msgr2=0x7f1f401667c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:48:39.562 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:39.559+0000 7f1f4647e640 1 --2- 172.21.15.123:0/2372812595 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f1f40071c60 0x7f1f401667c0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f1f38009bf0 tx=0x7f1f3801cbb0 comp rx=0 tx=0).stop 2024-09-16T15:48:39.562 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:39.563+0000 7f1f4647e640 1 -- 172.21.15.123:0/2372812595 shutdown_connections 2024-09-16T15:48:39.562 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:39.563+0000 7f1f4647e640 1 --2- 172.21.15.123:0/2372812595 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f1f40071c60 0x7f1f401667c0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:39.562 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:39.563+0000 7f1f4647e640 1 -- 172.21.15.123:0/2372812595 >> 172.21.15.123:0/2372812595 conn(0x7f1f400d0bc0 msgr2=0x7f1f400d0fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:48:39.562 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:39.563+0000 7f1f4647e640 1 -- 172.21.15.123:0/2372812595 shutdown_connections 2024-09-16T15:48:39.562 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:39.563+0000 7f1f4647e640 1 -- 172.21.15.123:0/2372812595 wait complete. 2024-09-16T15:48:39.563 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:39.563+0000 7f1f4647e640 1 Processor -- start 2024-09-16T15:48:39.563 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:39.563+0000 7f1f4647e640 1 -- start start 2024-09-16T15:48:39.565 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:39.563+0000 7f1f4647e640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f1f40071c60 0x7f1f400c46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:48:39.565 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:39.563+0000 7f1f4647e640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1f4015ed20 0x7f1f4015f590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:48:39.565 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:39.563+0000 7f1f4647e640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1f4015f100 0x7f1f4015fad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:48:39.565 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:39.563+0000 7f1f4647e640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f1f40169e60 con 0x7f1f4015ed20 2024-09-16T15:48:39.565 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:39.563+0000 7f1f4647e640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f1f40169ce0 con 0x7f1f4015f100 2024-09-16T15:48:39.565 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:39.563+0000 7f1f4647e640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f1f40169fe0 con 0x7f1f40071c60 2024-09-16T15:48:39.566 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:39.563+0000 7f1f44a1b640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1f4015f100 0x7f1f4015fad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:48:39.566 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:39.563+0000 7f1f44a1b640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1f4015f100 0x7f1f4015fad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.184:3300/0 says I am v2:172.21.15.123:58422/0 (socket says 172.21.15.123:58422) 2024-09-16T15:48:39.566 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:39.563+0000 7f1f44a1b640 1 -- 172.21.15.123:0/589507300 learned_addr learned my addr 172.21.15.123:0/589507300 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:48:39.566 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:39.563+0000 7f1f3ffff640 1 --2- 172.21.15.123:0/589507300 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f1f40071c60 0x7f1f400c46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:48:39.566 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:39.567+0000 7f1f3ffff640 1 -- 172.21.15.123:0/589507300 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1f4015f100 msgr2=0x7f1f4015fad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:48:39.566 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:39.567+0000 7f1f3ffff640 1 --2- 172.21.15.123:0/589507300 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1f4015f100 0x7f1f4015fad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:39.566 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:39.567+0000 7f1f3ffff640 1 -- 172.21.15.123:0/589507300 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1f4015ed20 msgr2=0x7f1f4015f590 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T15:48:39.567 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:39.567+0000 7f1f3ffff640 1 --2- 172.21.15.123:0/589507300 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1f4015ed20 0x7f1f4015f590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:39.567 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:39.567+0000 7f1f3ffff640 1 -- 172.21.15.123:0/589507300 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1f400c5140 con 0x7f1f40071c60 2024-09-16T15:48:39.567 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:39.567+0000 7f1f3ffff640 1 --2- 172.21.15.123:0/589507300 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f1f40071c60 0x7f1f400c46f0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f1f3000c400 tx=0x7f1f3000c8c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:48:39.575 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:39.575+0000 7f1f3d7fa640 1 -- 172.21.15.123:0/589507300 <== mon.2 v2:172.21.15.184:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f30011790 con 0x7f1f40071c60 2024-09-16T15:48:39.575 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:39.575+0000 7f1f3d7fa640 1 -- 172.21.15.123:0/589507300 <== mon.2 v2:172.21.15.184:3301/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f1f30017070 con 0x7f1f40071c60 2024-09-16T15:48:39.575 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:39.575+0000 7f1f4647e640 1 -- 172.21.15.123:0/589507300 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1f400c5420 con 0x7f1f40071c60 2024-09-16T15:48:39.575 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:39.575+0000 7f1f3d7fa640 1 -- 172.21.15.123:0/589507300 <== mon.2 v2:172.21.15.184:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f30011c80 con 0x7f1f40071c60 2024-09-16T15:48:39.575 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:39.575+0000 7f1f4647e640 1 -- 172.21.15.123:0/589507300 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f1f400c59e0 con 0x7f1f40071c60 2024-09-16T15:48:39.575 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:39.575+0000 7f1f4647e640 1 -- 172.21.15.123:0/589507300 --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1f4006ff80 con 0x7f1f40071c60 2024-09-16T15:48:39.578 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:39.579+0000 7f1f3d7fa640 1 -- 172.21.15.123:0/589507300 <== mon.2 v2:172.21.15.184:3301/0 4 ==== mgrmap(e 17) ==== 97233+0+0 (secure 0 0 0) 0x7f1f3001c460 con 0x7f1f40071c60 2024-09-16T15:48:39.578 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:39.579+0000 7f1f3d7fa640 1 --2- 172.21.15.123:0/589507300 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f1f20073070 0x7f1f20075530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:48:39.578 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:39.579+0000 7f1f3d7fa640 1 -- 172.21.15.123:0/589507300 <== mon.2 v2:172.21.15.184:3301/0 5 ==== osd_map(105..105 src has 1..105) ==== 5767+0+0 (secure 0 0 0) 0x7f1f30094950 con 0x7f1f40071c60 2024-09-16T15:48:39.578 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:39.579+0000 7f1f37fff640 1 --2- 172.21.15.123:0/589507300 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f1f20073070 0x7f1f20075530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:48:39.578 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:39.579+0000 7f1f37fff640 1 --2- 172.21.15.123:0/589507300 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f1f20073070 0x7f1f20075530 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f1f40160500 tx=0x7f1f280079c0 comp rx=0 tx=0).ready entity=mgr.4150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:48:39.589 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:39.591+0000 7f1f3d7fa640 1 -- 172.21.15.123:0/589507300 <== mon.2 v2:172.21.15.184:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f1f3005f8b0 con 0x7f1f40071c60 2024-09-16T15:48:39.773 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:39.771+0000 7f1f4647e640 1 -- 172.21.15.123:0/589507300 --> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] -- mgr_command(tid 0: {"prefix": "fs subvolume info", "vol_name": "cephfs", "sub_name": "subvol37810", "target": ["mon-mgr", ""]}) -- 0x7f1f4015b880 con 0x7f1f20073070 2024-09-16T15:48:39.791 DEBUG:teuthology.orchestra.run.smithi123:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-16T15:48:39.793 DEBUG:teuthology.orchestra.run.smithi184:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-16T15:48:39.797 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:39.795+0000 7f1f3d7fa640 1 -- 172.21.15.123:0/589507300 <== mgr.4150 v2:172.21.15.123:6836/2464759676 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+706 (secure 0 0 0) 0x7f1f4015b880 con 0x7f1f20073070 2024-09-16T15:48:39.798 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2024-09-16T15:48:39.798 INFO:teuthology.orchestra.run.smithi123.stdout: "atime": "2024-09-16 15:48:38", 2024-09-16T15:48:39.798 INFO:teuthology.orchestra.run.smithi123.stdout: "bytes_pcent": "undefined", 2024-09-16T15:48:39.798 INFO:teuthology.orchestra.run.smithi123.stdout: "bytes_quota": "infinite", 2024-09-16T15:48:39.798 INFO:teuthology.orchestra.run.smithi123.stdout: "bytes_used": 0, 2024-09-16T15:48:39.798 INFO:teuthology.orchestra.run.smithi123.stdout: "created_at": "2024-09-16 15:48:38", 2024-09-16T15:48:39.798 INFO:teuthology.orchestra.run.smithi123.stdout: "ctime": "2024-09-16 15:48:39", 2024-09-16T15:48:39.798 INFO:teuthology.orchestra.run.smithi123.stdout: "data_pool": "cephfs_data", 2024-09-16T15:48:39.798 INFO:teuthology.orchestra.run.smithi123.stdout: "earmark": "", 2024-09-16T15:48:39.799 INFO:teuthology.orchestra.run.smithi123.stdout: "features": [ 2024-09-16T15:48:39.799 INFO:teuthology.orchestra.run.smithi123.stdout: "snapshot-clone", 2024-09-16T15:48:39.799 INFO:teuthology.orchestra.run.smithi123.stdout: "snapshot-autoprotect" 2024-09-16T15:48:39.799 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-09-16T15:48:39.799 INFO:teuthology.orchestra.run.smithi123.stdout: "flavor": 1, 2024-09-16T15:48:39.799 INFO:teuthology.orchestra.run.smithi123.stdout: "gid": 0, 2024-09-16T15:48:39.799 INFO:teuthology.orchestra.run.smithi123.stdout: "mode": 16877, 2024-09-16T15:48:39.799 INFO:teuthology.orchestra.run.smithi123.stdout: "mon_addrs": [ 2024-09-16T15:48:39.799 INFO:teuthology.orchestra.run.smithi123.stdout: "172.21.15.123:6789", 2024-09-16T15:48:39.800 INFO:teuthology.orchestra.run.smithi123.stdout: "172.21.15.184:6789", 2024-09-16T15:48:39.800 INFO:teuthology.orchestra.run.smithi123.stdout: "172.21.15.184:6790" 2024-09-16T15:48:39.800 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-09-16T15:48:39.800 INFO:teuthology.orchestra.run.smithi123.stdout: "mtime": "2024-09-16 15:48:38", 2024-09-16T15:48:39.800 INFO:teuthology.orchestra.run.smithi123.stdout: "path": "/volumes/_nogroup/subvol37810/b00a5d9c-1a58-42eb-be9a-5f5c5fe23179", 2024-09-16T15:48:39.800 INFO:teuthology.orchestra.run.smithi123.stdout: "pool_namespace": "", 2024-09-16T15:48:39.800 INFO:teuthology.orchestra.run.smithi123.stdout: "state": "complete", 2024-09-16T15:48:39.800 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "subvolume", 2024-09-16T15:48:39.800 INFO:teuthology.orchestra.run.smithi123.stdout: "uid": 0 2024-09-16T15:48:39.800 INFO:teuthology.orchestra.run.smithi123.stdout:} 2024-09-16T15:48:39.802 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:39.799+0000 7f1f4647e640 1 -- 172.21.15.123:0/589507300 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f1f20073070 msgr2=0x7f1f20075530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:48:39.802 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:39.799+0000 7f1f4647e640 1 --2- 172.21.15.123:0/589507300 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f1f20073070 0x7f1f20075530 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f1f40160500 tx=0x7f1f280079c0 comp rx=0 tx=0).stop 2024-09-16T15:48:39.802 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:39.799+0000 7f1f4647e640 1 -- 172.21.15.123:0/589507300 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f1f40071c60 msgr2=0x7f1f400c46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:48:39.802 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:39.799+0000 7f1f4647e640 1 --2- 172.21.15.123:0/589507300 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f1f40071c60 0x7f1f400c46f0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f1f3000c400 tx=0x7f1f3000c8c0 comp rx=0 tx=0).stop 2024-09-16T15:48:39.802 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:39.799+0000 7f1f4647e640 1 -- 172.21.15.123:0/589507300 shutdown_connections 2024-09-16T15:48:39.802 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:39.799+0000 7f1f4647e640 1 --2- 172.21.15.123:0/589507300 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f1f20073070 0x7f1f20075530 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:39.803 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:39.799+0000 7f1f4647e640 1 --2- 172.21.15.123:0/589507300 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1f4015f100 0x7f1f4015fad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:39.803 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:39.799+0000 7f1f4647e640 1 --2- 172.21.15.123:0/589507300 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1f4015ed20 0x7f1f4015f590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:39.803 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:39.799+0000 7f1f4647e640 1 --2- 172.21.15.123:0/589507300 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f1f40071c60 0x7f1f400c46f0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:39.803 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:39.799+0000 7f1f4647e640 1 -- 172.21.15.123:0/589507300 >> 172.21.15.123:0/589507300 conn(0x7f1f400d0bc0 msgr2=0x7f1f40157d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:48:39.803 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:39.799+0000 7f1f4647e640 1 -- 172.21.15.123:0/589507300 shutdown_connections 2024-09-16T15:48:39.803 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:39.799+0000 7f1f4647e640 1 -- 172.21.15.123:0/589507300 wait complete. 2024-09-16T15:48:39.816 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume resize cephfs subvol37810 10485760 2024-09-16T15:48:40.004 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:40.003+0000 7f745f806640 1 -- 172.21.15.123:0/3850345613 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7458070470 msgr2=0x7f7458157d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:48:40.004 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:40.003+0000 7f745f806640 1 --2- 172.21.15.123:0/3850345613 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7458070470 0x7f7458157d00 secure :-1 s=READY pgs=1114 cs=0 l=1 rev1=1 crypto rx=0x7f7450009c80 tx=0x7f745001cda0 comp rx=0 tx=0).stop 2024-09-16T15:48:40.005 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:40.003+0000 7f745f806640 1 -- 172.21.15.123:0/3850345613 shutdown_connections 2024-09-16T15:48:40.005 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:40.003+0000 7f745f806640 1 --2- 172.21.15.123:0/3850345613 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7458070470 0x7f7458157d00 unknown :-1 s=CLOSED pgs=1114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:40.005 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:40.003+0000 7f745f806640 1 --2- 172.21.15.123:0/3850345613 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f7458073390 0x7f745806ff30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:40.005 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:40.003+0000 7f745f806640 1 -- 172.21.15.123:0/3850345613 >> 172.21.15.123:0/3850345613 conn(0x7f74580b1630 msgr2=0x7f74580b1a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:48:40.005 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:40.003+0000 7f745f806640 1 -- 172.21.15.123:0/3850345613 shutdown_connections 2024-09-16T15:48:40.005 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:40.003+0000 7f745f806640 1 -- 172.21.15.123:0/3850345613 wait complete. 2024-09-16T15:48:40.005 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:40.007+0000 7f745f806640 1 Processor -- start 2024-09-16T15:48:40.005 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:40.007+0000 7f745f806640 1 -- start start 2024-09-16T15:48:40.007 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:40.007+0000 7f745f806640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7458070470 0x7f74581523a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:48:40.007 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:40.007+0000 7f745f806640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f7458073390 0x7f745813acd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:48:40.007 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:40.007+0000 7f745f806640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f74580aee40 0x7f745814fc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:48:40.007 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:40.007+0000 7f745f806640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f74580b3a50 con 0x7f74580aee40 2024-09-16T15:48:40.008 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:40.007+0000 7f745f806640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f74580b38d0 con 0x7f7458070470 2024-09-16T15:48:40.008 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:40.007+0000 7f745f806640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f74580b3bd0 con 0x7f7458073390 2024-09-16T15:48:40.008 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:40.007+0000 7f745dda3640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f74580aee40 0x7f745814fc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:48:40.008 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:40.007+0000 7f745dda3640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f74580aee40 0x7f745814fc70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:55232/0 (socket says 172.21.15.123:55232) 2024-09-16T15:48:40.008 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:40.007+0000 7f745dda3640 1 -- 172.21.15.123:0/823372211 learned_addr learned my addr 172.21.15.123:0/823372211 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:48:40.008 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:40.007+0000 7f745cda1640 1 --2- 172.21.15.123:0/823372211 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f7458073390 0x7f745813acd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:48:40.008 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:40.007+0000 7f745d5a2640 1 --2- 172.21.15.123:0/823372211 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7458070470 0x7f74581523a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:48:40.008 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:40.007+0000 7f745dda3640 1 -- 172.21.15.123:0/823372211 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f7458073390 msgr2=0x7f745813acd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:48:40.008 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:40.007+0000 7f745dda3640 1 --2- 172.21.15.123:0/823372211 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f7458073390 0x7f745813acd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:40.008 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:40.007+0000 7f745dda3640 1 -- 172.21.15.123:0/823372211 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7458070470 msgr2=0x7f74581523a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:48:40.008 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:40.007+0000 7f745dda3640 1 --2- 172.21.15.123:0/823372211 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7458070470 0x7f74581523a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:40.008 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:40.007+0000 7f745dda3640 1 -- 172.21.15.123:0/823372211 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f745813b210 con 0x7f74580aee40 2024-09-16T15:48:40.009 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:40.007+0000 7f745dda3640 1 --2- 172.21.15.123:0/823372211 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f74580aee40 0x7f745814fc70 secure :-1 s=READY pgs=1115 cs=0 l=1 rev1=1 crypto rx=0x7f7450009bd0 tx=0x7f7450009390 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:48:40.009 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:40.007+0000 7f744dffb640 1 -- 172.21.15.123:0/823372211 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74500bb070 con 0x7f74580aee40 2024-09-16T15:48:40.009 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:40.007+0000 7f744dffb640 1 -- 172.21.15.123:0/823372211 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f7450004e00 con 0x7f74580aee40 2024-09-16T15:48:40.009 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:40.007+0000 7f745f806640 1 -- 172.21.15.123:0/823372211 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f74581501b0 con 0x7f74580aee40 2024-09-16T15:48:40.009 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:40.007+0000 7f744dffb640 1 -- 172.21.15.123:0/823372211 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74500b2040 con 0x7f74580aee40 2024-09-16T15:48:40.009 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:40.007+0000 7f745f806640 1 -- 172.21.15.123:0/823372211 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7458152cb0 con 0x7f74580aee40 2024-09-16T15:48:40.010 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:40.007+0000 7f745f806640 1 -- 172.21.15.123:0/823372211 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f74580b40d0 con 0x7f74580aee40 2024-09-16T15:48:40.010 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:40.007+0000 7f744dffb640 1 -- 172.21.15.123:0/823372211 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 17) ==== 97233+0+0 (secure 0 0 0) 0x7f74500be070 con 0x7f74580aee40 2024-09-16T15:48:40.010 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:40.011+0000 7f744dffb640 1 --2- 172.21.15.123:0/823372211 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f743c073070 0x7f743c075530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:48:40.010 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:40.011+0000 7f744dffb640 1 -- 172.21.15.123:0/823372211 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(105..105 src has 1..105) ==== 5767+0+0 (secure 0 0 0) 0x7f745012e090 con 0x7f74580aee40 2024-09-16T15:48:40.010 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:40.011+0000 7f745d5a2640 1 --2- 172.21.15.123:0/823372211 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f743c073070 0x7f743c075530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:48:40.010 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:40.011+0000 7f745d5a2640 1 --2- 172.21.15.123:0/823372211 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f743c073070 0x7f743c075530 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f7454001570 tx=0x7f7454009040 comp rx=0 tx=0).ready entity=mgr.4150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:48:40.013 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:40.011+0000 7f744dffb640 1 -- 172.21.15.123:0/823372211 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f745813b210 con 0x7f74580aee40 2024-09-16T15:48:40.223 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:40.223+0000 7f745f806640 1 -- 172.21.15.123:0/823372211 --> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] -- mgr_command(tid 0: {"prefix": "fs subvolume resize", "vol_name": "cephfs", "sub_name": "subvol37810", "new_size": "10485760", "target": ["mon-mgr", ""]}) -- 0x7f745814c660 con 0x7f743c073070 2024-09-16T15:48:40.258 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:40.259+0000 7f744dffb640 1 -- 172.21.15.123:0/823372211 <== mgr.4150 v2:172.21.15.123:6836/2464759676 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+128 (secure 0 0 0) 0x7f7454001210 con 0x7f743c073070 2024-09-16T15:48:40.259 INFO:teuthology.orchestra.run.smithi123.stdout:[ 2024-09-16T15:48:40.259 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-09-16T15:48:40.259 INFO:teuthology.orchestra.run.smithi123.stdout: "bytes_used": 0 2024-09-16T15:48:40.259 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:48:40.259 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-09-16T15:48:40.259 INFO:teuthology.orchestra.run.smithi123.stdout: "bytes_quota": 10485760 2024-09-16T15:48:40.259 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:48:40.259 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-09-16T15:48:40.259 INFO:teuthology.orchestra.run.smithi123.stdout: "bytes_pcent": "0.00" 2024-09-16T15:48:40.259 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-16T15:48:40.259 INFO:teuthology.orchestra.run.smithi123.stdout:] 2024-09-16T15:48:40.261 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:40.259+0000 7f745f806640 1 -- 172.21.15.123:0/823372211 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f743c073070 msgr2=0x7f743c075530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:48:40.261 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:40.259+0000 7f745f806640 1 --2- 172.21.15.123:0/823372211 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f743c073070 0x7f743c075530 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f7454001570 tx=0x7f7454009040 comp rx=0 tx=0).stop 2024-09-16T15:48:40.262 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:40.259+0000 7f745f806640 1 -- 172.21.15.123:0/823372211 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f74580aee40 msgr2=0x7f745814fc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:48:40.262 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:40.259+0000 7f745f806640 1 --2- 172.21.15.123:0/823372211 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f74580aee40 0x7f745814fc70 secure :-1 s=READY pgs=1115 cs=0 l=1 rev1=1 crypto rx=0x7f7450009bd0 tx=0x7f7450009390 comp rx=0 tx=0).stop 2024-09-16T15:48:40.262 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:40.259+0000 7f745f806640 1 -- 172.21.15.123:0/823372211 shutdown_connections 2024-09-16T15:48:40.262 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:40.259+0000 7f745f806640 1 --2- 172.21.15.123:0/823372211 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f743c073070 0x7f743c075530 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:40.262 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:40.259+0000 7f745f806640 1 --2- 172.21.15.123:0/823372211 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f74580aee40 0x7f745814fc70 unknown :-1 s=CLOSED pgs=1115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:40.262 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:40.259+0000 7f745f806640 1 --2- 172.21.15.123:0/823372211 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f7458073390 0x7f745813acd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:40.262 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:40.263+0000 7f745f806640 1 --2- 172.21.15.123:0/823372211 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7458070470 0x7f74581523a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:40.262 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:40.263+0000 7f745f806640 1 -- 172.21.15.123:0/823372211 >> 172.21.15.123:0/823372211 conn(0x7f74580b1630 msgr2=0x7f745815c450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:48:40.262 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:40.263+0000 7f745f806640 1 -- 172.21.15.123:0/823372211 shutdown_connections 2024-09-16T15:48:40.262 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:40.263+0000 7f745f806640 1 -- 172.21.15.123:0/823372211 wait complete. 2024-09-16T15:48:40.278 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume info cephfs subvol37810 2024-09-16T15:48:40.399 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:40.399+0000 7fe2e0189640 1 -- 172.21.15.123:0/1050376490 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe2d815a100 msgr2=0x7fe2d8072f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:48:40.399 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:40.399+0000 7fe2e0189640 1 --2- 172.21.15.123:0/1050376490 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe2d815a100 0x7fe2d8072f20 secure :-1 s=READY pgs=1116 cs=0 l=1 rev1=1 crypto rx=0x7fe2d80cf700 tx=0x7fe2c801c8c0 comp rx=0 tx=0).stop 2024-09-16T15:48:40.399 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:40.399+0000 7fe2e0189640 1 -- 172.21.15.123:0/1050376490 shutdown_connections 2024-09-16T15:48:40.399 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:40.399+0000 7fe2e0189640 1 --2- 172.21.15.123:0/1050376490 >> v2:172.21.15.184:3300/0 conn(0x7fe2d815ddb0 0x7fe2d815e240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:40.399 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:40.399+0000 7fe2e0189640 1 --2- 172.21.15.123:0/1050376490 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fe2d80cbb40 0x7fe2d80cbf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:40.399 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:40.399+0000 7fe2e0189640 1 --2- 172.21.15.123:0/1050376490 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe2d815a100 0x7fe2d8072f20 unknown :-1 s=CLOSED pgs=1116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:40.400 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:40.399+0000 7fe2e0189640 1 -- 172.21.15.123:0/1050376490 >> 172.21.15.123:0/1050376490 conn(0x7fe2d80b1710 msgr2=0x7fe2d80b1b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:48:40.400 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:40.399+0000 7fe2e0189640 1 -- 172.21.15.123:0/1050376490 shutdown_connections 2024-09-16T15:48:40.400 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:40.399+0000 7fe2e0189640 1 -- 172.21.15.123:0/1050376490 wait complete. 2024-09-16T15:48:40.400 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:40.399+0000 7fe2e0189640 1 Processor -- start 2024-09-16T15:48:40.400 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:40.399+0000 7fe2e0189640 1 -- start start 2024-09-16T15:48:40.402 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:40.399+0000 7fe2e0189640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe2d80cbb40 0x7fe2d8163550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:48:40.402 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:40.403+0000 7fe2e0189640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe2d815a100 0x7fe2d80c6dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:48:40.402 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:40.403+0000 7fe2e0189640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fe2d815ddb0 0x7fe2d80c7310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:48:40.402 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:40.403+0000 7fe2e0189640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fe2d806e740 con 0x7fe2d815a100 2024-09-16T15:48:40.403 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:40.403+0000 7fe2e0189640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fe2d806e5c0 con 0x7fe2d80cbb40 2024-09-16T15:48:40.403 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:40.403+0000 7fe2e0189640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fe2d806e8c0 con 0x7fe2d815ddb0 2024-09-16T15:48:40.403 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:40.403+0000 7fe2dd724640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe2d815a100 0x7fe2d80c6dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:48:40.403 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:40.403+0000 7fe2dd724640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe2d815a100 0x7fe2d80c6dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:55240/0 (socket says 172.21.15.123:55240) 2024-09-16T15:48:40.403 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:40.403+0000 7fe2dd724640 1 -- 172.21.15.123:0/1329090875 learned_addr learned my addr 172.21.15.123:0/1329090875 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:48:40.403 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:40.403+0000 7fe2de726640 1 --2- 172.21.15.123:0/1329090875 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fe2d815ddb0 0x7fe2d80c7310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:48:40.403 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:40.403+0000 7fe2ddf25640 1 --2- 172.21.15.123:0/1329090875 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe2d80cbb40 0x7fe2d8163550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:48:40.403 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:40.403+0000 7fe2dd724640 1 -- 172.21.15.123:0/1329090875 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fe2d815ddb0 msgr2=0x7fe2d80c7310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:48:40.403 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:40.403+0000 7fe2dd724640 1 --2- 172.21.15.123:0/1329090875 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fe2d815ddb0 0x7fe2d80c7310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:40.403 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:40.403+0000 7fe2dd724640 1 -- 172.21.15.123:0/1329090875 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe2d80cbb40 msgr2=0x7fe2d8163550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:48:40.403 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:40.403+0000 7fe2dd724640 1 --2- 172.21.15.123:0/1329090875 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe2d80cbb40 0x7fe2d8163550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:40.403 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:40.403+0000 7fe2dd724640 1 -- 172.21.15.123:0/1329090875 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe2d80c8f50 con 0x7fe2d815a100 2024-09-16T15:48:40.404 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:40.403+0000 7fe2dd724640 1 --2- 172.21.15.123:0/1329090875 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe2d815a100 0x7fe2d80c6dd0 secure :-1 s=READY pgs=1117 cs=0 l=1 rev1=1 crypto rx=0x7fe2d400dbf0 tx=0x7fe2d400a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:48:40.404 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:40.403+0000 7fe2ce7fc640 1 -- 172.21.15.123:0/1329090875 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2d4005020 con 0x7fe2d815a100 2024-09-16T15:48:40.404 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:40.403+0000 7fe2ce7fc640 1 -- 172.21.15.123:0/1329090875 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fe2d40052b0 con 0x7fe2d815a100 2024-09-16T15:48:40.404 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:40.403+0000 7fe2e0189640 1 -- 172.21.15.123:0/1329090875 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe2d80c9230 con 0x7fe2d815a100 2024-09-16T15:48:40.404 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:40.403+0000 7fe2ce7fc640 1 -- 172.21.15.123:0/1329090875 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2d40047b0 con 0x7fe2d815a100 2024-09-16T15:48:40.404 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:40.403+0000 7fe2e0189640 1 -- 172.21.15.123:0/1329090875 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe2d80c97f0 con 0x7fe2d815a100 2024-09-16T15:48:40.405 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:40.403+0000 7fe2e0189640 1 -- 172.21.15.123:0/1329090875 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe2d80ce6f0 con 0x7fe2d815a100 2024-09-16T15:48:40.405 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:40.403+0000 7fe2ce7fc640 1 -- 172.21.15.123:0/1329090875 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 17) ==== 97233+0+0 (secure 0 0 0) 0x7fe2d4004950 con 0x7fe2d815a100 2024-09-16T15:48:40.405 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:40.403+0000 7fe2ce7fc640 1 --2- 172.21.15.123:0/1329090875 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fe2bc073070 0x7fe2bc075530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:48:40.405 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:40.403+0000 7fe2ce7fc640 1 -- 172.21.15.123:0/1329090875 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(105..105 src has 1..105) ==== 5767+0+0 (secure 0 0 0) 0x7fe2d4094800 con 0x7fe2d815a100 2024-09-16T15:48:40.405 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:40.403+0000 7fe2ddf25640 1 --2- 172.21.15.123:0/1329090875 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fe2bc073070 0x7fe2bc075530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:48:40.405 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:40.403+0000 7fe2ddf25640 1 --2- 172.21.15.123:0/1329090875 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fe2bc073070 0x7fe2bc075530 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fe2c80af000 tx=0x7fe2c801b100 comp rx=0 tx=0).ready entity=mgr.4150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:48:40.408 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:40.407+0000 7fe2ce7fc640 1 -- 172.21.15.123:0/1329090875 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7fe2d405f710 con 0x7fe2d815a100 2024-09-16T15:48:40.608 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:40.607+0000 7fe2e0189640 1 -- 172.21.15.123:0/1329090875 --> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] -- mgr_command(tid 0: {"prefix": "fs subvolume info", "vol_name": "cephfs", "sub_name": "subvol37810", "target": ["mon-mgr", ""]}) -- 0x7fe2d80697b0 con 0x7fe2bc073070 2024-09-16T15:48:40.630 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:40.631+0000 7fe2ce7fc640 1 -- 172.21.15.123:0/1329090875 <== mgr.4150 v2:172.21.15.123:6836/2464759676 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+699 (secure 0 0 0) 0x7fe2d80697b0 con 0x7fe2bc073070 2024-09-16T15:48:40.630 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2024-09-16T15:48:40.630 INFO:teuthology.orchestra.run.smithi123.stdout: "atime": "2024-09-16 15:48:38", 2024-09-16T15:48:40.630 INFO:teuthology.orchestra.run.smithi123.stdout: "bytes_pcent": "0.00", 2024-09-16T15:48:40.630 INFO:teuthology.orchestra.run.smithi123.stdout: "bytes_quota": 10485760, 2024-09-16T15:48:40.630 INFO:teuthology.orchestra.run.smithi123.stdout: "bytes_used": 0, 2024-09-16T15:48:40.631 INFO:teuthology.orchestra.run.smithi123.stdout: "created_at": "2024-09-16 15:48:38", 2024-09-16T15:48:40.631 INFO:teuthology.orchestra.run.smithi123.stdout: "ctime": "2024-09-16 15:48:40", 2024-09-16T15:48:40.631 INFO:teuthology.orchestra.run.smithi123.stdout: "data_pool": "cephfs_data", 2024-09-16T15:48:40.631 INFO:teuthology.orchestra.run.smithi123.stdout: "earmark": "", 2024-09-16T15:48:40.631 INFO:teuthology.orchestra.run.smithi123.stdout: "features": [ 2024-09-16T15:48:40.631 INFO:teuthology.orchestra.run.smithi123.stdout: "snapshot-clone", 2024-09-16T15:48:40.631 INFO:teuthology.orchestra.run.smithi123.stdout: "snapshot-autoprotect" 2024-09-16T15:48:40.631 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-09-16T15:48:40.631 INFO:teuthology.orchestra.run.smithi123.stdout: "flavor": 1, 2024-09-16T15:48:40.631 INFO:teuthology.orchestra.run.smithi123.stdout: "gid": 0, 2024-09-16T15:48:40.631 INFO:teuthology.orchestra.run.smithi123.stdout: "mode": 16877, 2024-09-16T15:48:40.631 INFO:teuthology.orchestra.run.smithi123.stdout: "mon_addrs": [ 2024-09-16T15:48:40.631 INFO:teuthology.orchestra.run.smithi123.stdout: "172.21.15.123:6789", 2024-09-16T15:48:40.631 INFO:teuthology.orchestra.run.smithi123.stdout: "172.21.15.184:6789", 2024-09-16T15:48:40.632 INFO:teuthology.orchestra.run.smithi123.stdout: "172.21.15.184:6790" 2024-09-16T15:48:40.632 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-09-16T15:48:40.632 INFO:teuthology.orchestra.run.smithi123.stdout: "mtime": "2024-09-16 15:48:38", 2024-09-16T15:48:40.632 INFO:teuthology.orchestra.run.smithi123.stdout: "path": "/volumes/_nogroup/subvol37810/b00a5d9c-1a58-42eb-be9a-5f5c5fe23179", 2024-09-16T15:48:40.632 INFO:teuthology.orchestra.run.smithi123.stdout: "pool_namespace": "", 2024-09-16T15:48:40.632 INFO:teuthology.orchestra.run.smithi123.stdout: "state": "complete", 2024-09-16T15:48:40.632 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "subvolume", 2024-09-16T15:48:40.632 INFO:teuthology.orchestra.run.smithi123.stdout: "uid": 0 2024-09-16T15:48:40.632 INFO:teuthology.orchestra.run.smithi123.stdout:} 2024-09-16T15:48:40.633 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:40.631+0000 7fe2e0189640 1 -- 172.21.15.123:0/1329090875 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fe2bc073070 msgr2=0x7fe2bc075530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:48:40.633 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:40.631+0000 7fe2e0189640 1 --2- 172.21.15.123:0/1329090875 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fe2bc073070 0x7fe2bc075530 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fe2c80af000 tx=0x7fe2c801b100 comp rx=0 tx=0).stop 2024-09-16T15:48:40.633 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:40.631+0000 7fe2e0189640 1 -- 172.21.15.123:0/1329090875 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe2d815a100 msgr2=0x7fe2d80c6dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:48:40.633 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:40.631+0000 7fe2e0189640 1 --2- 172.21.15.123:0/1329090875 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe2d815a100 0x7fe2d80c6dd0 secure :-1 s=READY pgs=1117 cs=0 l=1 rev1=1 crypto rx=0x7fe2d400dbf0 tx=0x7fe2d400a620 comp rx=0 tx=0).stop 2024-09-16T15:48:40.633 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:40.631+0000 7fe2e0189640 1 -- 172.21.15.123:0/1329090875 shutdown_connections 2024-09-16T15:48:40.633 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:40.631+0000 7fe2e0189640 1 --2- 172.21.15.123:0/1329090875 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fe2bc073070 0x7fe2bc075530 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:40.633 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:40.631+0000 7fe2e0189640 1 --2- 172.21.15.123:0/1329090875 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fe2d815ddb0 0x7fe2d80c7310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:40.634 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:40.631+0000 7fe2e0189640 1 --2- 172.21.15.123:0/1329090875 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe2d815a100 0x7fe2d80c6dd0 unknown :-1 s=CLOSED pgs=1117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:40.634 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:40.631+0000 7fe2e0189640 1 --2- 172.21.15.123:0/1329090875 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe2d80cbb40 0x7fe2d8163550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:40.634 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:40.631+0000 7fe2e0189640 1 -- 172.21.15.123:0/1329090875 >> 172.21.15.123:0/1329090875 conn(0x7fe2d80b1710 msgr2=0x7fe2d81646e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:48:40.634 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:40.631+0000 7fe2e0189640 1 -- 172.21.15.123:0/1329090875 shutdown_connections 2024-09-16T15:48:40.634 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:40.631+0000 7fe2e0189640 1 -- 172.21.15.123:0/1329090875 wait complete. 2024-09-16T15:48:40.649 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume create cephfs subvol37810 --mode 777 --uid 1000 --gid 1000 2024-09-16T15:48:40.783 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:40.783+0000 7f9ddc88c640 1 -- 172.21.15.123:0/3650873631 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9dc0004d50 msgr2=0x7f9dc00051d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:48:40.784 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:40.783+0000 7f9ddc88c640 1 --2- 172.21.15.123:0/3650873631 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9dc0004d50 0x7f9dc00051d0 secure :-1 s=READY pgs=1119 cs=0 l=1 rev1=1 crypto rx=0x7f9dd808d6f0 tx=0x7f9dd001d150 comp rx=0 tx=0).stop 2024-09-16T15:48:40.784 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:40.783+0000 7f9dd6d9d640 1 -- 172.21.15.123:0/3650873631 reap_dead start 2024-09-16T15:48:40.784 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:40.783+0000 7f9ddc88c640 1 -- 172.21.15.123:0/3650873631 shutdown_connections 2024-09-16T15:48:40.784 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:40.783+0000 7f9ddc88c640 1 -- 172.21.15.123:0/3650873631 >> 172.21.15.123:0/3650873631 conn(0x7f9dd80d3eb0 msgr2=0x7f9dd80d42c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:48:40.784 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:40.783+0000 7f9ddc88c640 1 -- 172.21.15.123:0/3650873631 shutdown_connections 2024-09-16T15:48:40.784 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:40.783+0000 7f9ddc88c640 1 -- 172.21.15.123:0/3650873631 wait complete. 2024-09-16T15:48:40.784 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:40.783+0000 7f9ddc88c640 1 Processor -- start 2024-09-16T15:48:40.785 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:40.783+0000 7f9ddc88c640 1 -- start start 2024-09-16T15:48:40.789 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:40.783+0000 7f9ddc88c640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f9dc0004d50 0x7f9dd80caf80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:48:40.789 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:40.787+0000 7f9ddc88c640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f9dd8150970 0x7f9dd80ca280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:48:40.789 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:40.787+0000 7f9ddc88c640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9dd814ced0 0x7f9dd814d350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:48:40.789 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:40.787+0000 7f9ddc88c640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f9dd81628b0 con 0x7f9dd814ced0 2024-09-16T15:48:40.789 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:40.787+0000 7f9ddc88c640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f9dd8162730 con 0x7f9dc0004d50 2024-09-16T15:48:40.789 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:40.787+0000 7f9ddc88c640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f9dd8162a30 con 0x7f9dd8150970 2024-09-16T15:48:40.789 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:40.787+0000 7f9dd6d9d640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9dd814ced0 0x7f9dd814d350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:48:40.789 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:40.787+0000 7f9dd6d9d640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9dd814ced0 0x7f9dd814d350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:55272/0 (socket says 172.21.15.123:55272) 2024-09-16T15:48:40.789 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:40.787+0000 7f9dd6d9d640 1 -- 172.21.15.123:0/4248345182 learned_addr learned my addr 172.21.15.123:0/4248345182 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:48:40.789 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:40.787+0000 7f9dd659c640 1 --2- 172.21.15.123:0/4248345182 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f9dc0004d50 0x7f9dd80caf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:48:40.789 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:40.787+0000 7f9dd5d9b640 1 --2- 172.21.15.123:0/4248345182 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f9dd8150970 0x7f9dd80ca280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:48:40.789 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:40.787+0000 7f9dd6d9d640 1 -- 172.21.15.123:0/4248345182 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f9dd8150970 msgr2=0x7f9dd80ca280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:48:40.789 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:40.787+0000 7f9dd6d9d640 1 --2- 172.21.15.123:0/4248345182 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f9dd8150970 0x7f9dd80ca280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:40.790 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:40.787+0000 7f9dd6d9d640 1 -- 172.21.15.123:0/4248345182 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f9dc0004d50 msgr2=0x7f9dd80caf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:48:40.790 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:40.787+0000 7f9dd6d9d640 1 --2- 172.21.15.123:0/4248345182 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f9dc0004d50 0x7f9dd80caf80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:40.790 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:40.787+0000 7f9dd6d9d640 1 -- 172.21.15.123:0/4248345182 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9dd814d890 con 0x7f9dd814ced0 2024-09-16T15:48:40.790 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:40.787+0000 7f9dd6d9d640 1 --2- 172.21.15.123:0/4248345182 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9dd814ced0 0x7f9dd814d350 secure :-1 s=READY pgs=1120 cs=0 l=1 rev1=1 crypto rx=0x7f9dd00af040 tx=0x7f9dd00addc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:48:40.790 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:40.787+0000 7f9dc6ffd640 1 -- 172.21.15.123:0/4248345182 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9dd00b1040 con 0x7f9dd814ced0 2024-09-16T15:48:40.790 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:40.787+0000 7f9dc6ffd640 1 -- 172.21.15.123:0/4248345182 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f9dd00bc070 con 0x7f9dd814ced0 2024-09-16T15:48:40.790 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:40.787+0000 7f9ddc88c640 1 -- 172.21.15.123:0/4248345182 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9dd80cc850 con 0x7f9dd814ced0 2024-09-16T15:48:40.790 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:40.787+0000 7f9dc6ffd640 1 -- 172.21.15.123:0/4248345182 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9dd00ade30 con 0x7f9dd814ced0 2024-09-16T15:48:40.790 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:40.787+0000 7f9ddc88c640 1 -- 172.21.15.123:0/4248345182 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9dd80cba90 con 0x7f9dd814ced0 2024-09-16T15:48:40.790 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:40.787+0000 7f9ddc88c640 1 -- 172.21.15.123:0/4248345182 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9dd80b8fb0 con 0x7f9dd814ced0 2024-09-16T15:48:40.790 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:40.787+0000 7f9dc6ffd640 1 -- 172.21.15.123:0/4248345182 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 17) ==== 97233+0+0 (secure 0 0 0) 0x7f9dd0004a50 con 0x7f9dd814ced0 2024-09-16T15:48:40.790 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:40.787+0000 7f9dc6ffd640 1 --2- 172.21.15.123:0/4248345182 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f9db80730a0 0x7f9db8075560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:48:40.790 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:40.787+0000 7f9dc6ffd640 1 -- 172.21.15.123:0/4248345182 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(105..105 src has 1..105) ==== 5767+0+0 (secure 0 0 0) 0x7f9dd012f000 con 0x7f9dd814ced0 2024-09-16T15:48:40.790 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:40.787+0000 7f9dd659c640 1 --2- 172.21.15.123:0/4248345182 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f9db80730a0 0x7f9db8075560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:48:40.791 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:40.787+0000 7f9dd659c640 1 --2- 172.21.15.123:0/4248345182 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f9db80730a0 0x7f9db8075560 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f9dc8005fd0 tx=0x7f9dc8005e60 comp rx=0 tx=0).ready entity=mgr.4150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:48:40.792 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:40.791+0000 7f9dc6ffd640 1 -- 172.21.15.123:0/4248345182 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f9dd0016e60 con 0x7f9dd814ced0 2024-09-16T15:48:40.992 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:40.991+0000 7f9ddc88c640 1 -- 172.21.15.123:0/4248345182 --> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "subvol37810", "mode": "777", "uid": 1000, "gid": 1000, "target": ["mon-mgr", ""]}) -- 0x7f9dd80d0650 con 0x7f9db80730a0 2024-09-16T15:48:41.021 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:41.023+0000 7f9dc6ffd640 1 -- 172.21.15.123:0/4248345182 <== mgr.4150 v2:172.21.15.123:6836/2464759676 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f9dd80d0650 con 0x7f9db80730a0 2024-09-16T15:48:41.024 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:41.023+0000 7f9ddc88c640 1 -- 172.21.15.123:0/4248345182 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f9db80730a0 msgr2=0x7f9db8075560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:48:41.024 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:41.023+0000 7f9ddc88c640 1 --2- 172.21.15.123:0/4248345182 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f9db80730a0 0x7f9db8075560 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f9dc8005fd0 tx=0x7f9dc8005e60 comp rx=0 tx=0).stop 2024-09-16T15:48:41.024 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:41.023+0000 7f9ddc88c640 1 -- 172.21.15.123:0/4248345182 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9dd814ced0 msgr2=0x7f9dd814d350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:48:41.025 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:41.023+0000 7f9ddc88c640 1 --2- 172.21.15.123:0/4248345182 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9dd814ced0 0x7f9dd814d350 secure :-1 s=READY pgs=1120 cs=0 l=1 rev1=1 crypto rx=0x7f9dd00af040 tx=0x7f9dd00addc0 comp rx=0 tx=0).stop 2024-09-16T15:48:41.025 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:41.023+0000 7f9ddc88c640 1 -- 172.21.15.123:0/4248345182 shutdown_connections 2024-09-16T15:48:41.025 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:41.023+0000 7f9ddc88c640 1 --2- 172.21.15.123:0/4248345182 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f9db80730a0 0x7f9db8075560 secure :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f9dc8005fd0 tx=0x7f9dc8005e60 comp rx=0 tx=0).stop 2024-09-16T15:48:41.025 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:41.023+0000 7f9ddc88c640 1 --2- 172.21.15.123:0/4248345182 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9dd814ced0 0x7f9dd814d350 unknown :-1 s=CLOSED pgs=1120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:41.025 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:41.023+0000 7f9ddc88c640 1 --2- 172.21.15.123:0/4248345182 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f9dd8150970 0x7f9dd80ca280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:41.025 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:41.023+0000 7f9ddc88c640 1 --2- 172.21.15.123:0/4248345182 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f9dc0004d50 0x7f9dd80caf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:41.025 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:41.023+0000 7f9ddc88c640 1 -- 172.21.15.123:0/4248345182 >> 172.21.15.123:0/4248345182 conn(0x7f9dd80d3eb0 msgr2=0x7f9dd80758d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:48:41.025 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:41.023+0000 7f9ddc88c640 1 -- 172.21.15.123:0/4248345182 shutdown_connections 2024-09-16T15:48:41.025 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:41.023+0000 7f9ddc88c640 1 -- 172.21.15.123:0/4248345182 wait complete. 2024-09-16T15:48:41.041 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs subvol37810 2024-09-16T15:48:41.171 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:41.171+0000 7f0481ab5640 1 -- 172.21.15.123:0/1089690603 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0464003a90 msgr2=0x7f0464003eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:48:41.171 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:41.171+0000 7f0481ab5640 1 --2- 172.21.15.123:0/1089690603 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0464003a90 0x7f0464003eb0 secure :-1 s=READY pgs=1122 cs=0 l=1 rev1=1 crypto rx=0x7f046c00c2e0 tx=0x7f046c01cf60 comp rx=0 tx=0).stop 2024-09-16T15:48:41.171 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:41.171+0000 7f047bfff640 1 -- 172.21.15.123:0/1089690603 reap_dead start 2024-09-16T15:48:41.171 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:41.171+0000 7f0481ab5640 1 -- 172.21.15.123:0/1089690603 shutdown_connections 2024-09-16T15:48:41.171 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:41.171+0000 7f0481ab5640 1 -- 172.21.15.123:0/1089690603 >> 172.21.15.123:0/1089690603 conn(0x7f047c0b1710 msgr2=0x7f047c0b1b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:48:41.171 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:41.171+0000 7f0481ab5640 1 -- 172.21.15.123:0/1089690603 shutdown_connections 2024-09-16T15:48:41.171 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:41.171+0000 7f0481ab5640 1 -- 172.21.15.123:0/1089690603 wait complete. 2024-09-16T15:48:41.172 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:41.171+0000 7f0481ab5640 1 Processor -- start 2024-09-16T15:48:41.172 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:41.171+0000 7f0481ab5640 1 -- start start 2024-09-16T15:48:41.174 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:41.171+0000 7f0481ab5640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f0464003a90 0x7f047c0b52d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:48:41.174 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:41.171+0000 7f0481ab5640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f047c0b5810 0x7f047c1f4fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:48:41.174 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:41.171+0000 7f0481ab5640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f047c1f54e0 0x7f047c0c88d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:48:41.174 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:41.171+0000 7f0481ab5640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f047c0ccca0 con 0x7f047c1f54e0 2024-09-16T15:48:41.174 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:41.171+0000 7f0481ab5640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f047c0ccb20 con 0x7f047c0b5810 2024-09-16T15:48:41.174 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:41.171+0000 7f0481ab5640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f047c0cdf20 con 0x7f0464003a90 2024-09-16T15:48:41.174 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:41.171+0000 7f047bfff640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f047c1f54e0 0x7f047c0c88d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:48:41.175 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:41.171+0000 7f047bfff640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f047c1f54e0 0x7f047c0c88d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:55294/0 (socket says 172.21.15.123:55294) 2024-09-16T15:48:41.175 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:41.171+0000 7f047bfff640 1 -- 172.21.15.123:0/3471754825 learned_addr learned my addr 172.21.15.123:0/3471754825 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:48:41.175 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:41.171+0000 7f047affd640 1 --2- 172.21.15.123:0/3471754825 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f047c0b5810 0x7f047c1f4fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:48:41.175 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:41.171+0000 7f047b7fe640 1 --2- 172.21.15.123:0/3471754825 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f0464003a90 0x7f047c0b52d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:48:41.175 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:41.171+0000 7f047bfff640 1 -- 172.21.15.123:0/3471754825 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f0464003a90 msgr2=0x7f047c0b52d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:48:41.175 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:41.171+0000 7f047bfff640 1 --2- 172.21.15.123:0/3471754825 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f0464003a90 0x7f047c0b52d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:41.175 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:41.171+0000 7f047bfff640 1 -- 172.21.15.123:0/3471754825 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f047c0b5810 msgr2=0x7f047c1f4fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:48:41.175 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:41.171+0000 7f047bfff640 1 --2- 172.21.15.123:0/3471754825 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f047c0b5810 0x7f047c1f4fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:41.175 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:41.171+0000 7f047bfff640 1 -- 172.21.15.123:0/3471754825 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f047c0c6db0 con 0x7f047c1f54e0 2024-09-16T15:48:41.175 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:41.175+0000 7f047bfff640 1 --2- 172.21.15.123:0/3471754825 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f047c1f54e0 0x7f047c0c88d0 secure :-1 s=READY pgs=1123 cs=0 l=1 rev1=1 crypto rx=0x7f047c0cd0c0 tx=0x7f047400cb90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:48:41.175 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:41.175+0000 7f0463fff640 1 -- 172.21.15.123:0/3471754825 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0474017070 con 0x7f047c1f54e0 2024-09-16T15:48:41.175 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:41.175+0000 7f0463fff640 1 -- 172.21.15.123:0/3471754825 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f047400a610 con 0x7f047c1f54e0 2024-09-16T15:48:41.175 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:41.175+0000 7f0481ab5640 1 -- 172.21.15.123:0/3471754825 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f047c0c7090 con 0x7f047c1f54e0 2024-09-16T15:48:41.176 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:41.175+0000 7f0463fff640 1 -- 172.21.15.123:0/3471754825 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0474005e60 con 0x7f047c1f54e0 2024-09-16T15:48:41.176 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:41.175+0000 7f0481ab5640 1 -- 172.21.15.123:0/3471754825 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f047c0c7650 con 0x7f047c1f54e0 2024-09-16T15:48:41.176 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:41.175+0000 7f0481ab5640 1 -- 172.21.15.123:0/3471754825 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f047c0ce4b0 con 0x7f047c1f54e0 2024-09-16T15:48:41.176 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:41.175+0000 7f0463fff640 1 -- 172.21.15.123:0/3471754825 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 17) ==== 97233+0+0 (secure 0 0 0) 0x7f047401a070 con 0x7f047c1f54e0 2024-09-16T15:48:41.177 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:41.175+0000 7f0463fff640 1 --2- 172.21.15.123:0/3471754825 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f04680730a0 0x7f0468075560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:48:41.177 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:41.175+0000 7f0463fff640 1 -- 172.21.15.123:0/3471754825 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(105..105 src has 1..105) ==== 5767+0+0 (secure 0 0 0) 0x7f0474093680 con 0x7f047c1f54e0 2024-09-16T15:48:41.177 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:41.175+0000 7f047b7fe640 1 --2- 172.21.15.123:0/3471754825 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f04680730a0 0x7f0468075560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:48:41.177 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:41.175+0000 7f047b7fe640 1 --2- 172.21.15.123:0/3471754825 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f04680730a0 0x7f0468075560 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f046c0ab000 tx=0x7f046c0a7040 comp rx=0 tx=0).ready entity=mgr.4150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:48:41.180 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:41.179+0000 7f0463fff640 1 -- 172.21.15.123:0/3471754825 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f047405e460 con 0x7f047c1f54e0 2024-09-16T15:48:41.379 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:41.379+0000 7f0481ab5640 1 -- 172.21.15.123:0/3471754825 --> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "subvol37810", "target": ["mon-mgr", ""]}) -- 0x7f047c0cbf90 con 0x7f04680730a0 2024-09-16T15:48:41.393 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:41.395+0000 7f0463fff640 1 -- 172.21.15.123:0/3471754825 <== mgr.4150 v2:172.21.15.123:6836/2464759676 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+67 (secure 0 0 0) 0x7f047c0cbf90 con 0x7f04680730a0 2024-09-16T15:48:41.394 INFO:teuthology.orchestra.run.smithi123.stdout:/volumes/_nogroup/subvol37810/b00a5d9c-1a58-42eb-be9a-5f5c5fe23179 2024-09-16T15:48:41.396 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:41.395+0000 7f0481ab5640 1 -- 172.21.15.123:0/3471754825 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f04680730a0 msgr2=0x7f0468075560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:48:41.396 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:41.395+0000 7f0481ab5640 1 --2- 172.21.15.123:0/3471754825 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f04680730a0 0x7f0468075560 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f046c0ab000 tx=0x7f046c0a7040 comp rx=0 tx=0).stop 2024-09-16T15:48:41.396 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:41.395+0000 7f0481ab5640 1 -- 172.21.15.123:0/3471754825 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f047c1f54e0 msgr2=0x7f047c0c88d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:48:41.396 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:41.395+0000 7f0481ab5640 1 --2- 172.21.15.123:0/3471754825 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f047c1f54e0 0x7f047c0c88d0 secure :-1 s=READY pgs=1123 cs=0 l=1 rev1=1 crypto rx=0x7f047c0cd0c0 tx=0x7f047400cb90 comp rx=0 tx=0).stop 2024-09-16T15:48:41.396 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:41.395+0000 7f0481ab5640 1 -- 172.21.15.123:0/3471754825 shutdown_connections 2024-09-16T15:48:41.396 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:41.395+0000 7f0481ab5640 1 --2- 172.21.15.123:0/3471754825 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f04680730a0 0x7f0468075560 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:41.397 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:41.395+0000 7f0481ab5640 1 --2- 172.21.15.123:0/3471754825 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f047c1f54e0 0x7f047c0c88d0 unknown :-1 s=CLOSED pgs=1123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:41.397 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:41.395+0000 7f0481ab5640 1 --2- 172.21.15.123:0/3471754825 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f047c0b5810 0x7f047c1f4fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:41.397 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:41.395+0000 7f0481ab5640 1 --2- 172.21.15.123:0/3471754825 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f0464003a90 0x7f047c0b52d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:41.397 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:41.395+0000 7f0481ab5640 1 -- 172.21.15.123:0/3471754825 >> 172.21.15.123:0/3471754825 conn(0x7f047c0b1710 msgr2=0x7f047c157ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:48:41.397 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:41.395+0000 7f0481ab5640 1 -- 172.21.15.123:0/3471754825 shutdown_connections 2024-09-16T15:48:41.397 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:41.395+0000 7f0481ab5640 1 -- 172.21.15.123:0/3471754825 wait complete. 2024-09-16T15:48:41.413 DEBUG:tasks.cephfs.test_volumes:filling subvolume subvol37810 with 8 files each 1MB size under directory volumes/_nogroup/subvol37810/b00a5d9c-1a58-42eb-be9a-5f5c5fe23179 2024-09-16T15:48:41.413 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2024-09-16T15:48:41.413 DEBUG:teuthology.orchestra.run.smithi123:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-16T15:48:41.419 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:48:41.419 DEBUG:teuthology.orchestra.run.smithi123:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol37810/b00a5d9c-1a58-42eb-be9a-5f5c5fe23179/subvolume_file.0 bs=1M conv=fdatasync count=1 seek=0) 2024-09-16T15:48:41.481 INFO:teuthology.orchestra.run.smithi123.stderr:1+0 records in 2024-09-16T15:48:41.481 INFO:teuthology.orchestra.run.smithi123.stderr:1+0 records out 2024-09-16T15:48:41.481 INFO:teuthology.orchestra.run.smithi123.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0124724 s, 84.1 MB/s 2024-09-16T15:48:41.481 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2024-09-16T15:48:41.482 DEBUG:teuthology.orchestra.run.smithi123:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-16T15:48:41.530 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:48:41.531 DEBUG:teuthology.orchestra.run.smithi123:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol37810/b00a5d9c-1a58-42eb-be9a-5f5c5fe23179/subvolume_file.1 bs=1M conv=fdatasync count=1 seek=0) 2024-09-16T15:48:41.605 INFO:teuthology.orchestra.run.smithi123.stderr:1+0 records in 2024-09-16T15:48:41.605 INFO:teuthology.orchestra.run.smithi123.stderr:1+0 records out 2024-09-16T15:48:41.605 INFO:teuthology.orchestra.run.smithi123.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0173788 s, 60.3 MB/s 2024-09-16T15:48:41.606 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2024-09-16T15:48:41.606 DEBUG:teuthology.orchestra.run.smithi123:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-16T15:48:41.658 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:48:41.659 DEBUG:teuthology.orchestra.run.smithi123:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol37810/b00a5d9c-1a58-42eb-be9a-5f5c5fe23179/subvolume_file.2 bs=1M conv=fdatasync count=1 seek=0) 2024-09-16T15:48:41.734 INFO:teuthology.orchestra.run.smithi123.stderr:1+0 records in 2024-09-16T15:48:41.735 INFO:teuthology.orchestra.run.smithi123.stderr:1+0 records out 2024-09-16T15:48:41.735 INFO:teuthology.orchestra.run.smithi123.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0186378 s, 56.3 MB/s 2024-09-16T15:48:41.736 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2024-09-16T15:48:41.736 DEBUG:teuthology.orchestra.run.smithi123:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-16T15:48:41.782 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:48:41.783 DEBUG:teuthology.orchestra.run.smithi123:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol37810/b00a5d9c-1a58-42eb-be9a-5f5c5fe23179/subvolume_file.3 bs=1M conv=fdatasync count=1 seek=0) 2024-09-16T15:48:41.859 INFO:teuthology.orchestra.run.smithi123.stderr:1+0 records in 2024-09-16T15:48:41.859 INFO:teuthology.orchestra.run.smithi123.stderr:1+0 records out 2024-09-16T15:48:41.860 INFO:teuthology.orchestra.run.smithi123.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0201138 s, 52.1 MB/s 2024-09-16T15:48:41.860 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2024-09-16T15:48:41.861 DEBUG:teuthology.orchestra.run.smithi123:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-16T15:48:41.907 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:48:41.907 DEBUG:teuthology.orchestra.run.smithi123:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol37810/b00a5d9c-1a58-42eb-be9a-5f5c5fe23179/subvolume_file.4 bs=1M conv=fdatasync count=1 seek=0) 2024-09-16T15:48:41.982 INFO:teuthology.orchestra.run.smithi123.stderr:1+0 records in 2024-09-16T15:48:41.982 INFO:teuthology.orchestra.run.smithi123.stderr:1+0 records out 2024-09-16T15:48:41.982 INFO:teuthology.orchestra.run.smithi123.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.015938 s, 65.8 MB/s 2024-09-16T15:48:41.983 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2024-09-16T15:48:41.983 DEBUG:teuthology.orchestra.run.smithi123:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-16T15:48:42.031 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:48:42.031 DEBUG:teuthology.orchestra.run.smithi123:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol37810/b00a5d9c-1a58-42eb-be9a-5f5c5fe23179/subvolume_file.5 bs=1M conv=fdatasync count=1 seek=0) 2024-09-16T15:48:42.102 INFO:teuthology.orchestra.run.smithi123.stderr:1+0 records in 2024-09-16T15:48:42.102 INFO:teuthology.orchestra.run.smithi123.stderr:1+0 records out 2024-09-16T15:48:42.102 INFO:teuthology.orchestra.run.smithi123.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0140192 s, 74.8 MB/s 2024-09-16T15:48:42.104 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2024-09-16T15:48:42.104 DEBUG:teuthology.orchestra.run.smithi123:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-16T15:48:42.151 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:48:42.151 DEBUG:teuthology.orchestra.run.smithi123:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol37810/b00a5d9c-1a58-42eb-be9a-5f5c5fe23179/subvolume_file.6 bs=1M conv=fdatasync count=1 seek=0) 2024-09-16T15:48:42.230 INFO:teuthology.orchestra.run.smithi123.stderr:1+0 records in 2024-09-16T15:48:42.230 INFO:teuthology.orchestra.run.smithi123.stderr:1+0 records out 2024-09-16T15:48:42.230 INFO:teuthology.orchestra.run.smithi123.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.020353 s, 51.5 MB/s 2024-09-16T15:48:42.231 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2024-09-16T15:48:42.231 DEBUG:teuthology.orchestra.run.smithi123:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-16T15:48:42.279 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:48:42.280 DEBUG:teuthology.orchestra.run.smithi123:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvol37810/b00a5d9c-1a58-42eb-be9a-5f5c5fe23179/subvolume_file.7 bs=1M conv=fdatasync count=1 seek=0) 2024-09-16T15:48:42.356 INFO:teuthology.orchestra.run.smithi123.stderr:1+0 records in 2024-09-16T15:48:42.356 INFO:teuthology.orchestra.run.smithi123.stderr:1+0 records out 2024-09-16T15:48:42.356 INFO:teuthology.orchestra.run.smithi123.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0201113 s, 52.1 MB/s 2024-09-16T15:48:42.356 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot create cephfs subvol37810 subvol_snap6960 2024-09-16T15:48:42.542 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:42.543+0000 7ff9f22bd640 1 -- 172.21.15.123:0/1203328376 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff9d4003a60 msgr2=0x7ff9d4003e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:48:42.542 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:42.543+0000 7ff9f22bd640 1 --2- 172.21.15.123:0/1203328376 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff9d4003a60 0x7ff9d4003e80 secure :-1 s=READY pgs=1125 cs=0 l=1 rev1=1 crypto rx=0x7ff9dc0016c0 tx=0x7ff9dc01c770 comp rx=0 tx=0).stop 2024-09-16T15:48:42.543 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:42.543+0000 7ff9f22bd640 1 -- 172.21.15.123:0/1203328376 shutdown_connections 2024-09-16T15:48:42.543 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:42.543+0000 7ff9f22bd640 1 --2- 172.21.15.123:0/1203328376 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff9d4003a60 0x7ff9d4003e80 unknown :-1 s=CLOSED pgs=1125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:42.543 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:42.543+0000 7ff9f22bd640 1 -- 172.21.15.123:0/1203328376 >> 172.21.15.123:0/1203328376 conn(0x7ff9ec0b1630 msgr2=0x7ff9ec0b1a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:48:42.543 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:42.543+0000 7ff9f22bd640 1 -- 172.21.15.123:0/1203328376 shutdown_connections 2024-09-16T15:48:42.543 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:42.543+0000 7ff9f22bd640 1 -- 172.21.15.123:0/1203328376 wait complete. 2024-09-16T15:48:42.543 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:42.543+0000 7ff9f22bd640 1 Processor -- start 2024-09-16T15:48:42.544 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:42.543+0000 7ff9f22bd640 1 -- start start 2024-09-16T15:48:42.546 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:42.543+0000 7ff9f22bd640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7ff9d4003a60 0x7ff9ec1582d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:48:42.546 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:42.543+0000 7ff9f22bd640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff9ec14a8a0 0x7ff9ec0c8cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:48:42.546 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:42.543+0000 7ff9f22bd640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff9ec159d20 0x7ff9ec15a1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:48:42.546 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:42.543+0000 7ff9f22bd640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7ff9ec15ed80 con 0x7ff9ec14a8a0 2024-09-16T15:48:42.546 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:42.543+0000 7ff9f22bd640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7ff9ec15ec00 con 0x7ff9ec159d20 2024-09-16T15:48:42.546 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:42.543+0000 7ff9f22bd640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7ff9ec15ef00 con 0x7ff9d4003a60 2024-09-16T15:48:42.546 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:42.543+0000 7ff9eb7fe640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff9ec14a8a0 0x7ff9ec0c8cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:48:42.546 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:42.543+0000 7ff9eb7fe640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff9ec14a8a0 0x7ff9ec0c8cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:45988/0 (socket says 172.21.15.123:45988) 2024-09-16T15:48:42.547 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:42.543+0000 7ff9eb7fe640 1 -- 172.21.15.123:0/2110740897 learned_addr learned my addr 172.21.15.123:0/2110740897 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:48:42.547 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:42.543+0000 7ff9f085a640 1 --2- 172.21.15.123:0/2110740897 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff9ec159d20 0x7ff9ec15a1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:48:42.547 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:42.543+0000 7ff9ebfff640 1 --2- 172.21.15.123:0/2110740897 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7ff9d4003a60 0x7ff9ec1582d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:48:42.547 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:42.543+0000 7ff9eb7fe640 1 -- 172.21.15.123:0/2110740897 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7ff9d4003a60 msgr2=0x7ff9ec1582d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:48:42.547 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:42.543+0000 7ff9eb7fe640 1 --2- 172.21.15.123:0/2110740897 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7ff9d4003a60 0x7ff9ec1582d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:42.547 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:42.543+0000 7ff9eb7fe640 1 -- 172.21.15.123:0/2110740897 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff9ec159d20 msgr2=0x7ff9ec15a1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:48:42.547 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:42.543+0000 7ff9eb7fe640 1 --2- 172.21.15.123:0/2110740897 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff9ec159d20 0x7ff9ec15a1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:42.547 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:42.543+0000 7ff9eb7fe640 1 -- 172.21.15.123:0/2110740897 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff9ec1f7d70 con 0x7ff9ec14a8a0 2024-09-16T15:48:42.547 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:42.543+0000 7ff9eb7fe640 1 --2- 172.21.15.123:0/2110740897 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff9ec14a8a0 0x7ff9ec0c8cb0 secure :-1 s=READY pgs=1126 cs=0 l=1 rev1=1 crypto rx=0x7ff9e0010990 tx=0x7ff9e0010e50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:48:42.547 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:42.543+0000 7ff9e8ff9640 1 -- 172.21.15.123:0/2110740897 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9e000ed20 con 0x7ff9ec14a8a0 2024-09-16T15:48:42.547 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:42.543+0000 7ff9e8ff9640 1 -- 172.21.15.123:0/2110740897 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7ff9e0005020 con 0x7ff9ec14a8a0 2024-09-16T15:48:42.548 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:42.543+0000 7ff9f22bd640 1 -- 172.21.15.123:0/2110740897 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff9ec14ad20 con 0x7ff9ec14a8a0 2024-09-16T15:48:42.548 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:42.543+0000 7ff9e8ff9640 1 -- 172.21.15.123:0/2110740897 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9e0005450 con 0x7ff9ec14a8a0 2024-09-16T15:48:42.548 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:42.543+0000 7ff9f22bd640 1 -- 172.21.15.123:0/2110740897 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff9ec1fbae0 con 0x7ff9ec14a8a0 2024-09-16T15:48:42.548 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:42.547+0000 7ff9f22bd640 1 -- 172.21.15.123:0/2110740897 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff9ec0b3670 con 0x7ff9ec14a8a0 2024-09-16T15:48:42.549 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:42.547+0000 7ff9e8ff9640 1 -- 172.21.15.123:0/2110740897 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 17) ==== 97233+0+0 (secure 0 0 0) 0x7ff9e00047b0 con 0x7ff9ec14a8a0 2024-09-16T15:48:42.549 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:42.547+0000 7ff9e8ff9640 1 --2- 172.21.15.123:0/2110740897 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7ff9d80730a0 0x7ff9d8075560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:48:42.549 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:42.547+0000 7ff9e8ff9640 1 -- 172.21.15.123:0/2110740897 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(105..105 src has 1..105) ==== 5767+0+0 (secure 0 0 0) 0x7ff9e0094ea0 con 0x7ff9ec14a8a0 2024-09-16T15:48:42.549 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:42.547+0000 7ff9ebfff640 1 --2- 172.21.15.123:0/2110740897 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7ff9d80730a0 0x7ff9d8075560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:48:42.549 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:42.547+0000 7ff9ebfff640 1 --2- 172.21.15.123:0/2110740897 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7ff9d80730a0 0x7ff9d8075560 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7ff9d40057f0 tx=0x7ff9dc019560 comp rx=0 tx=0).ready entity=mgr.4150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:48:42.551 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:42.551+0000 7ff9e8ff9640 1 -- 172.21.15.123:0/2110740897 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7ff9e005fdb0 con 0x7ff9ec14a8a0 2024-09-16T15:48:42.757 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:42.755+0000 7ff9f22bd640 1 -- 172.21.15.123:0/2110740897 --> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot create", "vol_name": "cephfs", "sub_name": "subvol37810", "snap_name": "subvol_snap6960", "target": ["mon-mgr", ""]}) -- 0x7ff9ec1f7f40 con 0x7ff9d80730a0 2024-09-16T15:48:42.790 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:42.791+0000 7ff9e8ff9640 1 -- 172.21.15.123:0/2110740897 <== mgr.4150 v2:172.21.15.123:6836/2464759676 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7ff9dc006130 con 0x7ff9d80730a0 2024-09-16T15:48:42.794 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:42.795+0000 7ff9f22bd640 1 -- 172.21.15.123:0/2110740897 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7ff9d80730a0 msgr2=0x7ff9d8075560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:48:42.794 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:42.795+0000 7ff9f22bd640 1 --2- 172.21.15.123:0/2110740897 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7ff9d80730a0 0x7ff9d8075560 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7ff9d40057f0 tx=0x7ff9dc019560 comp rx=0 tx=0).stop 2024-09-16T15:48:42.794 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:42.795+0000 7ff9f22bd640 1 -- 172.21.15.123:0/2110740897 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff9ec14a8a0 msgr2=0x7ff9ec0c8cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:48:42.794 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:42.795+0000 7ff9f22bd640 1 --2- 172.21.15.123:0/2110740897 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff9ec14a8a0 0x7ff9ec0c8cb0 secure :-1 s=READY pgs=1126 cs=0 l=1 rev1=1 crypto rx=0x7ff9e0010990 tx=0x7ff9e0010e50 comp rx=0 tx=0).stop 2024-09-16T15:48:42.794 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:42.795+0000 7ff9f22bd640 1 -- 172.21.15.123:0/2110740897 shutdown_connections 2024-09-16T15:48:42.794 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:42.795+0000 7ff9f22bd640 1 --2- 172.21.15.123:0/2110740897 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7ff9d80730a0 0x7ff9d8075560 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:42.794 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:42.795+0000 7ff9f22bd640 1 --2- 172.21.15.123:0/2110740897 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff9ec159d20 0x7ff9ec15a1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:42.794 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:42.795+0000 7ff9f22bd640 1 --2- 172.21.15.123:0/2110740897 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff9ec14a8a0 0x7ff9ec0c8cb0 unknown :-1 s=CLOSED pgs=1126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:42.794 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:42.795+0000 7ff9f22bd640 1 --2- 172.21.15.123:0/2110740897 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7ff9d4003a60 0x7ff9ec1582d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:42.795 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:42.795+0000 7ff9f22bd640 1 -- 172.21.15.123:0/2110740897 >> 172.21.15.123:0/2110740897 conn(0x7ff9ec0b1630 msgr2=0x7ff9ec154bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:48:42.795 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:42.795+0000 7ff9f22bd640 1 -- 172.21.15.123:0/2110740897 shutdown_connections 2024-09-16T15:48:42.795 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:42.795+0000 7ff9f22bd640 1 -- 172.21.15.123:0/2110740897 wait complete. 2024-09-16T15:48:42.811 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot clone cephfs subvol37810 subvol_snap6960 subvol_clone4560 2024-09-16T15:48:42.943 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:42.943+0000 7f6709074640 1 -- 172.21.15.123:0/3603059917 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f67040ccad0 msgr2=0x7f670415e5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:48:42.943 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:42.943+0000 7f6709074640 1 --2- 172.21.15.123:0/3603059917 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f67040ccad0 0x7f670415e5d0 secure :-1 s=READY pgs=1128 cs=0 l=1 rev1=1 crypto rx=0x7f66fc009c80 tx=0x7f66fc01cda0 comp rx=0 tx=0).stop 2024-09-16T15:48:42.943 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:42.943+0000 7f6709074640 1 -- 172.21.15.123:0/3603059917 shutdown_connections 2024-09-16T15:48:42.943 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:42.943+0000 7f6709074640 1 --2- 172.21.15.123:0/3603059917 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f67040ccad0 0x7f670415e5d0 unknown :-1 s=CLOSED pgs=1128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:42.943 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:42.943+0000 7f6709074640 1 -- 172.21.15.123:0/3603059917 >> 172.21.15.123:0/3603059917 conn(0x7f67040b1710 msgr2=0x7f67040b1b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:48:42.943 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:42.943+0000 7f6709074640 1 -- 172.21.15.123:0/3603059917 shutdown_connections 2024-09-16T15:48:42.943 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:42.943+0000 7f6709074640 1 -- 172.21.15.123:0/3603059917 wait complete. 2024-09-16T15:48:42.944 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:42.943+0000 7f6709074640 1 Processor -- start 2024-09-16T15:48:42.944 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:42.943+0000 7f6709074640 1 -- start start 2024-09-16T15:48:42.946 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:42.943+0000 7f6709074640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6704072720 0x7f670415cb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:48:42.946 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:42.943+0000 7f6709074640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f67040ccad0 0x7f67040aec40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:48:42.946 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:42.943+0000 7f6709074640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f670415aa00 0x7f67040cfd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:48:42.946 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:42.943+0000 7f6709074640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f6704168d10 con 0x7f6704072720 2024-09-16T15:48:42.946 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:42.943+0000 7f6709074640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f6704168b90 con 0x7f67040ccad0 2024-09-16T15:48:42.946 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:42.943+0000 7f6709074640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f6704168e90 con 0x7f670415aa00 2024-09-16T15:48:42.946 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:42.943+0000 7f6702d9d640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6704072720 0x7f670415cb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:48:42.946 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:42.943+0000 7f6702d9d640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6704072720 0x7f670415cb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:46004/0 (socket says 172.21.15.123:46004) 2024-09-16T15:48:42.946 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:42.943+0000 7f6702d9d640 1 -- 172.21.15.123:0/442335923 learned_addr learned my addr 172.21.15.123:0/442335923 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:48:42.947 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:42.943+0000 7f6702d9d640 1 -- 172.21.15.123:0/442335923 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f670415aa00 msgr2=0x7f67040cfd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:48:42.947 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:42.943+0000 7f670359e640 1 --2- 172.21.15.123:0/442335923 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f670415aa00 0x7f67040cfd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:48:42.947 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:42.943+0000 7f670259c640 1 --2- 172.21.15.123:0/442335923 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f67040ccad0 0x7f67040aec40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:48:42.947 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:42.943+0000 7f6702d9d640 1 --2- 172.21.15.123:0/442335923 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f670415aa00 0x7f67040cfd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:42.947 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:42.943+0000 7f6702d9d640 1 -- 172.21.15.123:0/442335923 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f67040ccad0 msgr2=0x7f67040aec40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:48:42.947 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:42.943+0000 7f6702d9d640 1 --2- 172.21.15.123:0/442335923 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f67040ccad0 0x7f67040aec40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:42.947 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:42.947+0000 7f6702d9d640 1 -- 172.21.15.123:0/442335923 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f670414ee40 con 0x7f6704072720 2024-09-16T15:48:42.947 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:42.947+0000 7f670259c640 1 --2- 172.21.15.123:0/442335923 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f67040ccad0 0x7f67040aec40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T15:48:42.947 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:42.947+0000 7f6702d9d640 1 --2- 172.21.15.123:0/442335923 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6704072720 0x7f670415cb90 secure :-1 s=READY pgs=1129 cs=0 l=1 rev1=1 crypto rx=0x7f66f400c450 tx=0x7f66f400c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:48:42.947 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:42.947+0000 7f66eb7fe640 1 -- 172.21.15.123:0/442335923 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66f40052d0 con 0x7f6704072720 2024-09-16T15:48:42.947 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:42.947+0000 7f66eb7fe640 1 -- 172.21.15.123:0/442335923 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f66f4005470 con 0x7f6704072720 2024-09-16T15:48:42.947 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:42.947+0000 7f6709074640 1 -- 172.21.15.123:0/442335923 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f670414f120 con 0x7f6704072720 2024-09-16T15:48:42.947 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:42.947+0000 7f66eb7fe640 1 -- 172.21.15.123:0/442335923 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66f4012870 con 0x7f6704072720 2024-09-16T15:48:42.948 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:42.947+0000 7f6709074640 1 -- 172.21.15.123:0/442335923 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f670414f6e0 con 0x7f6704072720 2024-09-16T15:48:42.948 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:42.947+0000 7f6709074640 1 -- 172.21.15.123:0/442335923 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f670415e5d0 con 0x7f6704072720 2024-09-16T15:48:42.948 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:42.947+0000 7f66eb7fe640 1 -- 172.21.15.123:0/442335923 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 17) ==== 97233+0+0 (secure 0 0 0) 0x7f66f4003600 con 0x7f6704072720 2024-09-16T15:48:42.949 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:42.947+0000 7f66eb7fe640 1 --2- 172.21.15.123:0/442335923 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f66f0073070 0x7f66f0075530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:48:42.949 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:42.947+0000 7f670259c640 1 --2- 172.21.15.123:0/442335923 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f66f0073070 0x7f66f0075530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:48:42.949 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:42.947+0000 7f66eb7fe640 1 -- 172.21.15.123:0/442335923 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(105..105 src has 1..105) ==== 5767+0+0 (secure 0 0 0) 0x7f66f40946b0 con 0x7f6704072720 2024-09-16T15:48:42.949 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:42.947+0000 7f670259c640 1 --2- 172.21.15.123:0/442335923 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f66f0073070 0x7f66f0075530 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f67041fbb00 tx=0x7f66f8009040 comp rx=0 tx=0).ready entity=mgr.4150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:48:42.951 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:42.951+0000 7f66eb7fe640 1 -- 172.21.15.123:0/442335923 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f66f405f6a0 con 0x7f6704072720 2024-09-16T15:48:43.157 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:43.159+0000 7f6709074640 1 -- 172.21.15.123:0/442335923 --> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot clone", "vol_name": "cephfs", "sub_name": "subvol37810", "snap_name": "subvol_snap6960", "target_sub_name": "subvol_clone4560", "target": ["mon-mgr", ""]}) -- 0x7f67040cd440 con 0x7f66f0073070 2024-09-16T15:48:43.310 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:43.311+0000 7f66eb7fe640 1 -- 172.21.15.123:0/442335923 <== mgr.4150 v2:172.21.15.123:6836/2464759676 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f67040cd440 con 0x7f66f0073070 2024-09-16T15:48:43.313 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:43.311+0000 7f6709074640 1 -- 172.21.15.123:0/442335923 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f66f0073070 msgr2=0x7f66f0075530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:48:43.313 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:43.311+0000 7f6709074640 1 --2- 172.21.15.123:0/442335923 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f66f0073070 0x7f66f0075530 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f67041fbb00 tx=0x7f66f8009040 comp rx=0 tx=0).stop 2024-09-16T15:48:43.313 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:43.311+0000 7f6709074640 1 -- 172.21.15.123:0/442335923 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6704072720 msgr2=0x7f670415cb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:48:43.313 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:43.311+0000 7f6709074640 1 --2- 172.21.15.123:0/442335923 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6704072720 0x7f670415cb90 secure :-1 s=READY pgs=1129 cs=0 l=1 rev1=1 crypto rx=0x7f66f400c450 tx=0x7f66f400c910 comp rx=0 tx=0).stop 2024-09-16T15:48:43.313 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:43.311+0000 7f6709074640 1 -- 172.21.15.123:0/442335923 shutdown_connections 2024-09-16T15:48:43.313 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:43.311+0000 7f6709074640 1 --2- 172.21.15.123:0/442335923 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f66f0073070 0x7f66f0075530 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:43.314 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:43.311+0000 7f6709074640 1 --2- 172.21.15.123:0/442335923 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f670415aa00 0x7f67040cfd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:43.314 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:43.311+0000 7f6709074640 1 --2- 172.21.15.123:0/442335923 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f67040ccad0 0x7f67040aec40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:43.314 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:43.311+0000 7f6709074640 1 --2- 172.21.15.123:0/442335923 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6704072720 0x7f670415cb90 unknown :-1 s=CLOSED pgs=1129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:43.314 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:43.311+0000 7f6709074640 1 -- 172.21.15.123:0/442335923 >> 172.21.15.123:0/442335923 conn(0x7f67040b1710 msgr2=0x7f6704073130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:48:43.314 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:43.311+0000 7f6709074640 1 -- 172.21.15.123:0/442335923 shutdown_connections 2024-09-16T15:48:43.314 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:43.311+0000 7f6709074640 1 -- 172.21.15.123:0/442335923 wait complete. 2024-09-16T15:48:43.316 INFO:tasks.ceph.mgr.y.smithi123.stderr:2024-09-16T15:48:43.315+0000 7fa9b9700640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-16T15:48:43.317 INFO:tasks.ceph.mgr.y.smithi123.stderr:2024-09-16T15:48:43.315+0000 7fa9b9700640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-16T15:48:43.317 INFO:tasks.ceph.mgr.y.smithi123.stderr:2024-09-16T15:48:43.315+0000 7fa9b9700640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-16T15:48:43.317 INFO:tasks.ceph.mgr.y.smithi123.stderr:2024-09-16T15:48:43.315+0000 7fa9b9700640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-16T15:48:43.317 INFO:tasks.ceph.mgr.y.smithi123.stderr:2024-09-16T15:48:43.315+0000 7fa9b9700640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-16T15:48:43.334 INFO:tasks.ceph.mgr.y.smithi123.stderr:2024-09-16T15:48:43.335+0000 7fa9b7efd640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-16T15:48:43.334 INFO:tasks.ceph.mgr.y.smithi123.stderr:2024-09-16T15:48:43.335+0000 7fa9b7efd640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-16T15:48:43.335 INFO:tasks.ceph.mgr.y.smithi123.stderr:2024-09-16T15:48:43.335+0000 7fa9b7efd640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-16T15:48:43.335 INFO:tasks.ceph.mgr.y.smithi123.stderr:2024-09-16T15:48:43.335+0000 7fa9b7efd640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-16T15:48:43.335 INFO:tasks.ceph.mgr.y.smithi123.stderr:2024-09-16T15:48:43.335+0000 7fa9b7efd640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-16T15:48:43.336 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol_clone4560 2024-09-16T15:48:43.481 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:43.479+0000 7f0a01ee9640 1 -- 172.21.15.123:0/1518310095 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f09fc15b8b0 msgr2=0x7f09fc06e510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:48:43.481 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:43.479+0000 7f0a01ee9640 1 --2- 172.21.15.123:0/1518310095 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f09fc15b8b0 0x7f09fc06e510 secure :-1 s=READY pgs=1133 cs=0 l=1 rev1=1 crypto rx=0x7f09fc06edc0 tx=0x7f09ec01c870 comp rx=0 tx=0).stop 2024-09-16T15:48:43.482 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:43.479+0000 7f0a01ee9640 1 -- 172.21.15.123:0/1518310095 shutdown_connections 2024-09-16T15:48:43.482 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:43.479+0000 7f0a01ee9640 1 --2- 172.21.15.123:0/1518310095 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f09fc070fd0 0x7f09fc073b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:43.482 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:43.479+0000 7f0a01ee9640 1 --2- 172.21.15.123:0/1518310095 >> v2:172.21.15.184:3300/0 conn(0x7f09fc15bc90 0x7f09fc073600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:43.482 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:43.479+0000 7f0a01ee9640 1 --2- 172.21.15.123:0/1518310095 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f09fc15b8b0 0x7f09fc06e510 unknown :-1 s=CLOSED pgs=1133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:43.482 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:43.479+0000 7f0a01ee9640 1 -- 172.21.15.123:0/1518310095 >> 172.21.15.123:0/1518310095 conn(0x7f09fc0d0bf0 msgr2=0x7f09fc0d1000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:48:43.482 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:43.479+0000 7f0a01ee9640 1 -- 172.21.15.123:0/1518310095 shutdown_connections 2024-09-16T15:48:43.482 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:43.479+0000 7f0a01ee9640 1 -- 172.21.15.123:0/1518310095 wait complete. 2024-09-16T15:48:43.483 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:43.483+0000 7f0a01ee9640 1 Processor -- start 2024-09-16T15:48:43.483 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:43.483+0000 7f0a01ee9640 1 -- start start 2024-09-16T15:48:43.485 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:43.483+0000 7f0a01ee9640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f09fc070fd0 0x7f09fc147a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:48:43.485 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:43.483+0000 7f0a01ee9640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f09fc15b8b0 0x7f09fc14c5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:48:43.485 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:43.483+0000 7f0a01ee9640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f09fc15bc90 0x7f09fc0b3bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:48:43.485 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:43.483+0000 7f0a01ee9640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f09fc06cd70 con 0x7f09fc15bc90 2024-09-16T15:48:43.485 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:43.483+0000 7f0a01ee9640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f09fc06cbf0 con 0x7f09fc070fd0 2024-09-16T15:48:43.485 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:43.483+0000 7f0a01ee9640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f09fc06cef0 con 0x7f09fc15b8b0 2024-09-16T15:48:43.485 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:43.483+0000 7f09fbfff640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f09fc15bc90 0x7f09fc0b3bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:48:43.485 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:43.483+0000 7f09fbfff640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f09fc15bc90 0x7f09fc0b3bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:46070/0 (socket says 172.21.15.123:46070) 2024-09-16T15:48:43.485 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:43.483+0000 7f09fbfff640 1 -- 172.21.15.123:0/4178943044 learned_addr learned my addr 172.21.15.123:0/4178943044 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:48:43.485 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:43.483+0000 7f09fb7fe640 1 --2- 172.21.15.123:0/4178943044 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f09fc070fd0 0x7f09fc147a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:48:43.485 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:43.483+0000 7f09f2ffd640 1 --2- 172.21.15.123:0/4178943044 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f09fc15b8b0 0x7f09fc14c5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:48:43.486 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:43.483+0000 7f09fbfff640 1 -- 172.21.15.123:0/4178943044 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f09fc15b8b0 msgr2=0x7f09fc14c5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:48:43.486 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:43.483+0000 7f09fbfff640 1 --2- 172.21.15.123:0/4178943044 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f09fc15b8b0 0x7f09fc14c5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:43.486 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:43.483+0000 7f09fbfff640 1 -- 172.21.15.123:0/4178943044 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f09fc070fd0 msgr2=0x7f09fc147a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:48:43.486 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:43.483+0000 7f09fbfff640 1 --2- 172.21.15.123:0/4178943044 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f09fc070fd0 0x7f09fc147a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:43.486 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:43.483+0000 7f09fbfff640 1 -- 172.21.15.123:0/4178943044 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f09fc0b4110 con 0x7f09fc15bc90 2024-09-16T15:48:43.486 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:43.483+0000 7f09fbfff640 1 --2- 172.21.15.123:0/4178943044 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f09fc15bc90 0x7f09fc0b3bd0 secure :-1 s=READY pgs=1134 cs=0 l=1 rev1=1 crypto rx=0x7f09f400b970 tx=0x7f09f400be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:48:43.486 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:43.483+0000 7f09f8ff9640 1 -- 172.21.15.123:0/4178943044 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09f4019070 con 0x7f09fc15bc90 2024-09-16T15:48:43.486 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:43.483+0000 7f0a01ee9640 1 -- 172.21.15.123:0/4178943044 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f09fc148860 con 0x7f09fc15bc90 2024-09-16T15:48:43.486 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:43.483+0000 7f09f8ff9640 1 -- 172.21.15.123:0/4178943044 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f09f400cc50 con 0x7f09fc15bc90 2024-09-16T15:48:43.486 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:43.483+0000 7f09f8ff9640 1 -- 172.21.15.123:0/4178943044 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09f40146a0 con 0x7f09fc15bc90 2024-09-16T15:48:43.486 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:43.483+0000 7f0a01ee9640 1 -- 172.21.15.123:0/4178943044 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f09fc14ce70 con 0x7f09fc15bc90 2024-09-16T15:48:43.487 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:43.487+0000 7f0a01ee9640 1 -- 172.21.15.123:0/4178943044 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f09fc06e510 con 0x7f09fc15bc90 2024-09-16T15:48:43.487 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:43.487+0000 7f09f8ff9640 1 -- 172.21.15.123:0/4178943044 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 17) ==== 97233+0+0 (secure 0 0 0) 0x7f09f401c070 con 0x7f09fc15bc90 2024-09-16T15:48:43.487 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:43.487+0000 7f09f8ff9640 1 --2- 172.21.15.123:0/4178943044 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f09dc0730a0 0x7f09dc075560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:48:43.487 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:43.487+0000 7f09fb7fe640 1 --2- 172.21.15.123:0/4178943044 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f09dc0730a0 0x7f09dc075560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:48:43.487 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:43.487+0000 7f09f8ff9640 1 -- 172.21.15.123:0/4178943044 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(105..105 src has 1..105) ==== 5767+0+0 (secure 0 0 0) 0x7f09f4094040 con 0x7f09fc15bc90 2024-09-16T15:48:43.488 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:43.487+0000 7f09fb7fe640 1 --2- 172.21.15.123:0/4178943044 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f09dc0730a0 0x7f09dc075560 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f09ec0af000 tx=0x7f09ec01b0b0 comp rx=0 tx=0).ready entity=mgr.4150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:48:43.491 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:43.491+0000 7f09f8ff9640 1 -- 172.21.15.123:0/4178943044 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f09fc0b4110 con 0x7f09fc15bc90 2024-09-16T15:48:43.722 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:43.723+0000 7f0a01ee9640 1 -- 172.21.15.123:0/4178943044 --> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvol_clone4560", "target": ["mon-mgr", ""]}) -- 0x7f09fc152ec0 con 0x7f09dc0730a0 2024-09-16T15:48:44.315 INFO:tasks.ceph.mgr.y.smithi123.stderr:2024-09-16T15:48:44.315+0000 7fa9a05ce640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-16T15:48:44.316 INFO:tasks.ceph.mgr.y.smithi123.stderr:2024-09-16T15:48:44.315+0000 7fa9a05ce640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-16T15:48:44.316 INFO:tasks.ceph.mgr.y.smithi123.stderr:2024-09-16T15:48:44.315+0000 7fa9a05ce640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-16T15:48:44.316 INFO:tasks.ceph.mgr.y.smithi123.stderr:2024-09-16T15:48:44.315+0000 7fa9a05ce640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-16T15:48:44.316 INFO:tasks.ceph.mgr.y.smithi123.stderr:2024-09-16T15:48:44.315+0000 7fa9a05ce640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-16T15:48:44.329 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:44.327+0000 7f09f8ff9640 1 -- 172.21.15.123:0/4178943044 <== mgr.4150 v2:172.21.15.123:6836/2464759676 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+46 (secure 0 0 0) 0x7f09fc152ec0 con 0x7f09dc0730a0 2024-09-16T15:48:44.329 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2024-09-16T15:48:44.329 INFO:teuthology.orchestra.run.smithi123.stdout: "status": { 2024-09-16T15:48:44.329 INFO:teuthology.orchestra.run.smithi123.stdout: "state": "complete" 2024-09-16T15:48:44.329 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-16T15:48:44.329 INFO:teuthology.orchestra.run.smithi123.stdout:} 2024-09-16T15:48:44.332 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:44.331+0000 7f0a01ee9640 1 -- 172.21.15.123:0/4178943044 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f09dc0730a0 msgr2=0x7f09dc075560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:48:44.332 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:44.331+0000 7f0a01ee9640 1 --2- 172.21.15.123:0/4178943044 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f09dc0730a0 0x7f09dc075560 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f09ec0af000 tx=0x7f09ec01b0b0 comp rx=0 tx=0).stop 2024-09-16T15:48:44.332 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:44.331+0000 7f0a01ee9640 1 -- 172.21.15.123:0/4178943044 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f09fc15bc90 msgr2=0x7f09fc0b3bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:48:44.332 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:44.331+0000 7f0a01ee9640 1 --2- 172.21.15.123:0/4178943044 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f09fc15bc90 0x7f09fc0b3bd0 secure :-1 s=READY pgs=1134 cs=0 l=1 rev1=1 crypto rx=0x7f09f400b970 tx=0x7f09f400be30 comp rx=0 tx=0).stop 2024-09-16T15:48:44.333 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:44.331+0000 7f0a01ee9640 1 -- 172.21.15.123:0/4178943044 shutdown_connections 2024-09-16T15:48:44.333 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:44.331+0000 7f0a01ee9640 1 --2- 172.21.15.123:0/4178943044 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f09dc0730a0 0x7f09dc075560 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:44.333 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:44.331+0000 7f0a01ee9640 1 --2- 172.21.15.123:0/4178943044 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f09fc15bc90 0x7f09fc0b3bd0 unknown :-1 s=CLOSED pgs=1134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:44.333 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:44.331+0000 7f0a01ee9640 1 --2- 172.21.15.123:0/4178943044 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f09fc15b8b0 0x7f09fc14c5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:44.333 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:44.331+0000 7f0a01ee9640 1 --2- 172.21.15.123:0/4178943044 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f09fc070fd0 0x7f09fc147a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:44.333 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:44.331+0000 7f0a01ee9640 1 -- 172.21.15.123:0/4178943044 >> 172.21.15.123:0/4178943044 conn(0x7f09fc0d0bf0 msgr2=0x7f09fc073600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:48:44.333 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:44.331+0000 7f0a01ee9640 1 -- 172.21.15.123:0/4178943044 shutdown_connections 2024-09-16T15:48:44.333 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:44.331+0000 7f0a01ee9640 1 -- 172.21.15.123:0/4178943044 wait complete. 2024-09-16T15:48:44.351 DEBUG:tasks.cephfs.test_volumes:current clone state = complete 2024-09-16T15:48:44.351 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:48:44.351 DEBUG:teuthology.orchestra.run.smithi123:> (cd /home/ubuntu/cephtest/mnt.0 && exec sudo cat ./volumes/_nogroup/subvol_clone4560/.meta) 2024-09-16T15:48:44.367 INFO:teuthology.orchestra.run.smithi123.stdout:[GLOBAL] 2024-09-16T15:48:44.367 INFO:teuthology.orchestra.run.smithi123.stdout:version = 2 2024-09-16T15:48:44.367 INFO:teuthology.orchestra.run.smithi123.stdout:type = clone 2024-09-16T15:48:44.367 INFO:teuthology.orchestra.run.smithi123.stdout:path = /volumes/_nogroup/subvol_clone4560/dfb063e9-21ee-464c-9273-de9270d8f8a2 2024-09-16T15:48:44.367 INFO:teuthology.orchestra.run.smithi123.stdout:state = complete 2024-09-16T15:48:44.367 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-09-16T15:48:44.368 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs subvol37810 2024-09-16T15:48:44.544 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:44.543+0000 7fd8c324f640 1 -- 172.21.15.123:0/1080618656 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd8bc0cc3c0 msgr2=0x7fd8bc0cc870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:48:44.544 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:44.543+0000 7fd8c324f640 1 --2- 172.21.15.123:0/1080618656 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd8bc0cc3c0 0x7fd8bc0cc870 secure :-1 s=READY pgs=1138 cs=0 l=1 rev1=1 crypto rx=0x7fd8b4009bc0 tx=0x7fd8b401cba0 comp rx=0 tx=0).stop 2024-09-16T15:48:44.545 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:44.543+0000 7fd8c324f640 1 -- 172.21.15.123:0/1080618656 shutdown_connections 2024-09-16T15:48:44.545 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:44.543+0000 7fd8c324f640 1 --2- 172.21.15.123:0/1080618656 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd8bc0cc3c0 0x7fd8bc0cc870 unknown :-1 s=CLOSED pgs=1138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:44.545 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:44.543+0000 7fd8c324f640 1 --2- 172.21.15.123:0/1080618656 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fd8bc15aa00 0x7fd8bc15e5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:44.545 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:44.543+0000 7fd8c324f640 1 -- 172.21.15.123:0/1080618656 >> 172.21.15.123:0/1080618656 conn(0x7fd8bc0b1710 msgr2=0x7fd8bc0b1b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:48:44.545 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:44.543+0000 7fd8c324f640 1 -- 172.21.15.123:0/1080618656 shutdown_connections 2024-09-16T15:48:44.545 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:44.543+0000 7fd8c324f640 1 -- 172.21.15.123:0/1080618656 wait complete. 2024-09-16T15:48:44.545 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:44.547+0000 7fd8c324f640 1 Processor -- start 2024-09-16T15:48:44.545 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:44.547+0000 7fd8c324f640 1 -- start start 2024-09-16T15:48:44.547 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:44.547+0000 7fd8c324f640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd8bc072720 0x7fd8bc0b3850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:48:44.547 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:44.547+0000 7fd8c324f640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fd8bc0cc3c0 0x7fd8bc1f7660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:48:44.547 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:44.547+0000 7fd8c324f640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd8bc15aa00 0x7fd8bc157d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:48:44.547 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:44.547+0000 7fd8c324f640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fd8bc167d00 con 0x7fd8bc15aa00 2024-09-16T15:48:44.548 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:44.547+0000 7fd8c324f640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fd8bc167b80 con 0x7fd8bc072720 2024-09-16T15:48:44.548 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:44.547+0000 7fd8c324f640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fd8bc167e80 con 0x7fd8bc0cc3c0 2024-09-16T15:48:44.548 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:44.547+0000 7fd8c17ec640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd8bc15aa00 0x7fd8bc157d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:48:44.548 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:44.547+0000 7fd8c17ec640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd8bc15aa00 0x7fd8bc157d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:46096/0 (socket says 172.21.15.123:46096) 2024-09-16T15:48:44.548 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:44.547+0000 7fd8c17ec640 1 -- 172.21.15.123:0/273153768 learned_addr learned my addr 172.21.15.123:0/273153768 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:48:44.548 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:44.547+0000 7fd8bbfff640 1 --2- 172.21.15.123:0/273153768 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fd8bc0cc3c0 0x7fd8bc1f7660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:48:44.548 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:44.547+0000 7fd8c0feb640 1 --2- 172.21.15.123:0/273153768 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd8bc072720 0x7fd8bc0b3850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:48:44.548 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:44.547+0000 7fd8c17ec640 1 -- 172.21.15.123:0/273153768 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fd8bc0cc3c0 msgr2=0x7fd8bc1f7660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:48:44.548 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:44.547+0000 7fd8c17ec640 1 --2- 172.21.15.123:0/273153768 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fd8bc0cc3c0 0x7fd8bc1f7660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:44.548 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:44.547+0000 7fd8c17ec640 1 -- 172.21.15.123:0/273153768 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd8bc072720 msgr2=0x7fd8bc0b3850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:48:44.548 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:44.547+0000 7fd8c17ec640 1 --2- 172.21.15.123:0/273153768 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd8bc072720 0x7fd8bc0b3850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:44.548 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:44.547+0000 7fd8c17ec640 1 -- 172.21.15.123:0/273153768 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd8bc0b3d90 con 0x7fd8bc15aa00 2024-09-16T15:48:44.548 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:44.547+0000 7fd8bbfff640 1 --2- 172.21.15.123:0/273153768 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fd8bc0cc3c0 0x7fd8bc1f7660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T15:48:44.549 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:44.547+0000 7fd8c17ec640 1 --2- 172.21.15.123:0/273153768 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd8bc15aa00 0x7fd8bc157d60 secure :-1 s=READY pgs=1139 cs=0 l=1 rev1=1 crypto rx=0x7fd8b401a7d0 tx=0x7fd8b40a59a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:48:44.549 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:44.547+0000 7fd8b97fa640 1 -- 172.21.15.123:0/273153768 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8b40b2040 con 0x7fd8bc15aa00 2024-09-16T15:48:44.549 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:44.547+0000 7fd8b97fa640 1 -- 172.21.15.123:0/273153768 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fd8b4004d40 con 0x7fd8bc15aa00 2024-09-16T15:48:44.549 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:44.547+0000 7fd8c324f640 1 -- 172.21.15.123:0/273153768 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd8bc1fb9d0 con 0x7fd8bc15aa00 2024-09-16T15:48:44.549 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:44.547+0000 7fd8b97fa640 1 -- 172.21.15.123:0/273153768 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8b4004300 con 0x7fd8bc15aa00 2024-09-16T15:48:44.549 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:44.547+0000 7fd8c324f640 1 -- 172.21.15.123:0/273153768 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd8bc1fbf90 con 0x7fd8bc15aa00 2024-09-16T15:48:44.550 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:44.547+0000 7fd8c324f640 1 -- 172.21.15.123:0/273153768 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd8bc0cc870 con 0x7fd8bc15aa00 2024-09-16T15:48:44.550 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:44.547+0000 7fd8b97fa640 1 -- 172.21.15.123:0/273153768 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 17) ==== 97233+0+0 (secure 0 0 0) 0x7fd8b40be060 con 0x7fd8bc15aa00 2024-09-16T15:48:44.550 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:44.551+0000 7fd8b97fa640 1 --2- 172.21.15.123:0/273153768 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fd8a8073070 0x7fd8a8075530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:48:44.550 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:44.551+0000 7fd8c0feb640 1 --2- 172.21.15.123:0/273153768 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fd8a8073070 0x7fd8a8075530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:48:44.550 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:44.551+0000 7fd8b97fa640 1 -- 172.21.15.123:0/273153768 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(105..105 src has 1..105) ==== 5767+0+0 (secure 0 0 0) 0x7fd8b412e030 con 0x7fd8bc15aa00 2024-09-16T15:48:44.550 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:44.551+0000 7fd8c0feb640 1 --2- 172.21.15.123:0/273153768 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fd8a8073070 0x7fd8a8075530 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fd8ac0058d0 tx=0x7fd8ac005860 comp rx=0 tx=0).ready entity=mgr.4150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:48:44.554 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:44.555+0000 7fd8b97fa640 1 -- 172.21.15.123:0/273153768 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7fd8b40a5e20 con 0x7fd8bc15aa00 2024-09-16T15:48:44.755 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:44.755+0000 7fd8c324f640 1 -- 172.21.15.123:0/273153768 --> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "subvol37810", "target": ["mon-mgr", ""]}) -- 0x7fd8bc0cd440 con 0x7fd8a8073070 2024-09-16T15:48:44.778 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:44.779+0000 7fd8b97fa640 1 -- 172.21.15.123:0/273153768 <== mgr.4150 v2:172.21.15.123:6836/2464759676 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+67 (secure 0 0 0) 0x7fd8bc0cd440 con 0x7fd8a8073070 2024-09-16T15:48:44.778 INFO:teuthology.orchestra.run.smithi123.stdout:/volumes/_nogroup/subvol37810/b00a5d9c-1a58-42eb-be9a-5f5c5fe23179 2024-09-16T15:48:44.781 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:44.779+0000 7fd8c324f640 1 -- 172.21.15.123:0/273153768 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fd8a8073070 msgr2=0x7fd8a8075530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:48:44.781 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:44.779+0000 7fd8c324f640 1 --2- 172.21.15.123:0/273153768 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fd8a8073070 0x7fd8a8075530 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fd8ac0058d0 tx=0x7fd8ac005860 comp rx=0 tx=0).stop 2024-09-16T15:48:44.781 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:44.779+0000 7fd8c324f640 1 -- 172.21.15.123:0/273153768 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd8bc15aa00 msgr2=0x7fd8bc157d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:48:44.781 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:44.779+0000 7fd8c324f640 1 --2- 172.21.15.123:0/273153768 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd8bc15aa00 0x7fd8bc157d60 secure :-1 s=READY pgs=1139 cs=0 l=1 rev1=1 crypto rx=0x7fd8b401a7d0 tx=0x7fd8b40a59a0 comp rx=0 tx=0).stop 2024-09-16T15:48:44.781 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:44.779+0000 7fd8c324f640 1 -- 172.21.15.123:0/273153768 shutdown_connections 2024-09-16T15:48:44.781 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:44.779+0000 7fd8c324f640 1 --2- 172.21.15.123:0/273153768 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fd8a8073070 0x7fd8a8075530 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:44.781 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:44.779+0000 7fd8c324f640 1 --2- 172.21.15.123:0/273153768 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd8bc15aa00 0x7fd8bc157d60 unknown :-1 s=CLOSED pgs=1139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:44.781 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:44.779+0000 7fd8c324f640 1 --2- 172.21.15.123:0/273153768 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fd8bc0cc3c0 0x7fd8bc1f7660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:44.781 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:44.779+0000 7fd8c324f640 1 --2- 172.21.15.123:0/273153768 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd8bc072720 0x7fd8bc0b3850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:44.781 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:44.779+0000 7fd8c324f640 1 -- 172.21.15.123:0/273153768 >> 172.21.15.123:0/273153768 conn(0x7fd8bc0b1710 msgr2=0x7fd8bc0cabc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:48:44.781 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:44.779+0000 7fd8c324f640 1 -- 172.21.15.123:0/273153768 shutdown_connections 2024-09-16T15:48:44.781 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:44.779+0000 7fd8c324f640 1 -- 172.21.15.123:0/273153768 wait complete. 2024-09-16T15:48:44.798 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs subvol_clone4560 2024-09-16T15:48:44.930 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:44.931+0000 7fe4e1d7d640 1 -- 172.21.15.123:0/2523183047 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe4dc15a170 msgr2=0x7fe4dc072f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:48:44.930 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:44.931+0000 7fe4e1d7d640 1 --2- 172.21.15.123:0/2523183047 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe4dc15a170 0x7fe4dc072f30 secure :-1 s=READY pgs=1140 cs=0 l=1 rev1=1 crypto rx=0x7fe4dc0c6ba0 tx=0x7fe4cc01ccf0 comp rx=0 tx=0).stop 2024-09-16T15:48:44.931 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:44.931+0000 7fe4e1d7d640 1 -- 172.21.15.123:0/2523183047 shutdown_connections 2024-09-16T15:48:44.931 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:44.931+0000 7fe4e1d7d640 1 --2- 172.21.15.123:0/2523183047 >> v2:172.21.15.184:3300/0 conn(0x7fe4dc15ddc0 0x7fe4dc15e250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:44.931 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:44.931+0000 7fe4e1d7d640 1 --2- 172.21.15.123:0/2523183047 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe4dc15a170 0x7fe4dc072f30 unknown :-1 s=CLOSED pgs=1140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:44.931 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:44.931+0000 7fe4e1d7d640 1 -- 172.21.15.123:0/2523183047 >> 172.21.15.123:0/2523183047 conn(0x7fe4dc0b1710 msgr2=0x7fe4dc0b1b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:48:44.931 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:44.931+0000 7fe4e1d7d640 1 -- 172.21.15.123:0/2523183047 shutdown_connections 2024-09-16T15:48:44.931 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:44.931+0000 7fe4e1d7d640 1 -- 172.21.15.123:0/2523183047 wait complete. 2024-09-16T15:48:44.931 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:44.931+0000 7fe4e1d7d640 1 Processor -- start 2024-09-16T15:48:44.932 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:44.931+0000 7fe4e1d7d640 1 -- start start 2024-09-16T15:48:44.933 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:44.931+0000 7fe4e1d7d640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe4dc0ca650 0x7fe4dc0c5a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:48:44.933 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:44.931+0000 7fe4e1d7d640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fe4dc15a170 0x7fe4dc15f900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:48:44.934 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:44.931+0000 7fe4e1d7d640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe4dc15ddc0 0x7fe4dc148fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:48:44.934 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:44.931+0000 7fe4e1d7d640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fe4dc1670d0 con 0x7fe4dc15ddc0 2024-09-16T15:48:44.934 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:44.931+0000 7fe4e1d7d640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fe4dc166f50 con 0x7fe4dc0ca650 2024-09-16T15:48:44.934 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:44.931+0000 7fe4e1d7d640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fe4dc167250 con 0x7fe4dc15a170 2024-09-16T15:48:44.934 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:44.931+0000 7fe4dbfff640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe4dc15ddc0 0x7fe4dc148fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:48:44.934 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:44.931+0000 7fe4dbfff640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe4dc15ddc0 0x7fe4dc148fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:46106/0 (socket says 172.21.15.123:46106) 2024-09-16T15:48:44.934 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:44.931+0000 7fe4dbfff640 1 -- 172.21.15.123:0/1876518721 learned_addr learned my addr 172.21.15.123:0/1876518721 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:48:44.934 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:44.931+0000 7fe4db7fe640 1 --2- 172.21.15.123:0/1876518721 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe4dc0ca650 0x7fe4dc0c5a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:48:44.934 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:44.931+0000 7fe4daffd640 1 --2- 172.21.15.123:0/1876518721 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fe4dc15a170 0x7fe4dc15f900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:48:44.934 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:44.931+0000 7fe4dbfff640 1 -- 172.21.15.123:0/1876518721 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fe4dc15a170 msgr2=0x7fe4dc15f900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:48:44.934 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:44.931+0000 7fe4dbfff640 1 --2- 172.21.15.123:0/1876518721 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fe4dc15a170 0x7fe4dc15f900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:44.934 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:44.931+0000 7fe4dbfff640 1 -- 172.21.15.123:0/1876518721 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe4dc0ca650 msgr2=0x7fe4dc0c5a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:48:44.934 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:44.931+0000 7fe4dbfff640 1 --2- 172.21.15.123:0/1876518721 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe4dc0ca650 0x7fe4dc0c5a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:44.935 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:44.931+0000 7fe4dbfff640 1 -- 172.21.15.123:0/1876518721 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe4dc145cf0 con 0x7fe4dc15ddc0 2024-09-16T15:48:44.935 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:44.931+0000 7fe4db7fe640 1 --2- 172.21.15.123:0/1876518721 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe4dc0ca650 0x7fe4dc0c5a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T15:48:44.935 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:44.931+0000 7fe4dbfff640 1 --2- 172.21.15.123:0/1876518721 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe4dc15ddc0 0x7fe4dc148fb0 secure :-1 s=READY pgs=1141 cs=0 l=1 rev1=1 crypto rx=0x7fe4d400d690 tx=0x7fe4d400db50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:48:44.935 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:44.931+0000 7fe4d3fff640 1 -- 172.21.15.123:0/1876518721 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4d4017070 con 0x7fe4dc15ddc0 2024-09-16T15:48:44.935 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:44.931+0000 7fe4d3fff640 1 -- 172.21.15.123:0/1876518721 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fe4d4005310 con 0x7fe4dc15ddc0 2024-09-16T15:48:44.935 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:44.931+0000 7fe4d3fff640 1 -- 172.21.15.123:0/1876518721 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4d4009820 con 0x7fe4dc15ddc0 2024-09-16T15:48:44.935 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:44.931+0000 7fe4e1d7d640 1 -- 172.21.15.123:0/1876518721 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe4dc145fd0 con 0x7fe4dc15ddc0 2024-09-16T15:48:44.935 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:44.931+0000 7fe4e1d7d640 1 -- 172.21.15.123:0/1876518721 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe4dc146590 con 0x7fe4dc15ddc0 2024-09-16T15:48:44.936 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:44.935+0000 7fe4e1d7d640 1 -- 172.21.15.123:0/1876518721 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe4dc072f30 con 0x7fe4dc15ddc0 2024-09-16T15:48:44.936 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:44.935+0000 7fe4d3fff640 1 -- 172.21.15.123:0/1876518721 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 17) ==== 97233+0+0 (secure 0 0 0) 0x7fe4d40099c0 con 0x7fe4dc15ddc0 2024-09-16T15:48:44.936 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:44.935+0000 7fe4d3fff640 1 --2- 172.21.15.123:0/1876518721 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fe4bc073070 0x7fe4bc075530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:48:44.936 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:44.935+0000 7fe4d3fff640 1 -- 172.21.15.123:0/1876518721 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(105..105 src has 1..105) ==== 5767+0+0 (secure 0 0 0) 0x7fe4d4094660 con 0x7fe4dc15ddc0 2024-09-16T15:48:44.936 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:44.935+0000 7fe4db7fe640 1 --2- 172.21.15.123:0/1876518721 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fe4bc073070 0x7fe4bc075530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:48:44.936 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:44.935+0000 7fe4db7fe640 1 --2- 172.21.15.123:0/1876518721 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fe4bc073070 0x7fe4bc075530 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fe4cc0af000 tx=0x7fe4cc01b530 comp rx=0 tx=0).ready entity=mgr.4150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:48:44.939 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:44.939+0000 7fe4d3fff640 1 -- 172.21.15.123:0/1876518721 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7fe4d405f5c0 con 0x7fe4dc15ddc0 2024-09-16T15:48:45.144 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:45.143+0000 7fe4e1d7d640 1 -- 172.21.15.123:0/1876518721 --> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "subvol_clone4560", "target": ["mon-mgr", ""]}) -- 0x7fe4dc0b57c0 con 0x7fe4bc073070 2024-09-16T15:48:45.155 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:45.155+0000 7fe4d3fff640 1 -- 172.21.15.123:0/1876518721 <== mgr.4150 v2:172.21.15.123:6836/2464759676 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+72 (secure 0 0 0) 0x7fe4dc0b57c0 con 0x7fe4bc073070 2024-09-16T15:48:45.155 INFO:teuthology.orchestra.run.smithi123.stdout:/volumes/_nogroup/subvol_clone4560/dfb063e9-21ee-464c-9273-de9270d8f8a2 2024-09-16T15:48:45.158 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:45.159+0000 7fe4e1d7d640 1 -- 172.21.15.123:0/1876518721 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fe4bc073070 msgr2=0x7fe4bc075530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:48:45.158 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:45.159+0000 7fe4e1d7d640 1 --2- 172.21.15.123:0/1876518721 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fe4bc073070 0x7fe4bc075530 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fe4cc0af000 tx=0x7fe4cc01b530 comp rx=0 tx=0).stop 2024-09-16T15:48:45.158 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:45.159+0000 7fe4e1d7d640 1 -- 172.21.15.123:0/1876518721 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe4dc15ddc0 msgr2=0x7fe4dc148fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:48:45.158 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:45.159+0000 7fe4e1d7d640 1 --2- 172.21.15.123:0/1876518721 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe4dc15ddc0 0x7fe4dc148fb0 secure :-1 s=READY pgs=1141 cs=0 l=1 rev1=1 crypto rx=0x7fe4d400d690 tx=0x7fe4d400db50 comp rx=0 tx=0).stop 2024-09-16T15:48:45.158 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:45.159+0000 7fe4e1d7d640 1 -- 172.21.15.123:0/1876518721 shutdown_connections 2024-09-16T15:48:45.158 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:45.159+0000 7fe4e1d7d640 1 --2- 172.21.15.123:0/1876518721 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fe4bc073070 0x7fe4bc075530 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:45.158 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:45.159+0000 7fe4e1d7d640 1 --2- 172.21.15.123:0/1876518721 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe4dc15ddc0 0x7fe4dc148fb0 unknown :-1 s=CLOSED pgs=1141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:45.158 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:45.159+0000 7fe4e1d7d640 1 --2- 172.21.15.123:0/1876518721 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fe4dc15a170 0x7fe4dc15f900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:45.158 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:45.159+0000 7fe4e1d7d640 1 --2- 172.21.15.123:0/1876518721 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe4dc0ca650 0x7fe4dc0c5a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:45.159 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:45.159+0000 7fe4e1d7d640 1 -- 172.21.15.123:0/1876518721 >> 172.21.15.123:0/1876518721 conn(0x7fe4dc0b1710 msgr2=0x7fe4dc0b1af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:48:45.159 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:45.159+0000 7fe4e1d7d640 1 -- 172.21.15.123:0/1876518721 shutdown_connections 2024-09-16T15:48:45.159 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:45.159+0000 7fe4e1d7d640 1 -- 172.21.15.123:0/1876518721 wait complete. 2024-09-16T15:48:45.174 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:48:45.174 DEBUG:teuthology.orchestra.run.smithi123:> (cd /home/ubuntu/cephtest/mnt.0 && exec getfattr --only-values -n ceph.dir.rentries volumes/_nogroup/subvol37810/b00a5d9c-1a58-42eb-be9a-5f5c5fe23179/.snap/subvol_snap6960) 2024-09-16T15:48:45.188 INFO:teuthology.orchestra.run.smithi123.stdout:9 2024-09-16T15:48:45.189 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:48:45.189 DEBUG:teuthology.orchestra.run.smithi123:> (cd /home/ubuntu/cephtest/mnt.0 && exec getfattr --only-values -n ceph.dir.rentries volumes/_nogroup/subvol_clone4560/dfb063e9-21ee-464c-9273-de9270d8f8a2) 2024-09-16T15:48:45.239 INFO:teuthology.orchestra.run.smithi123.stdout:9 2024-09-16T15:48:45.239 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume info cephfs subvol_clone4560 2024-09-16T15:48:45.416 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:45.415+0000 7f7fdae37640 1 -- 172.21.15.123:0/1057522078 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7fd40cc3c0 msgr2=0x7f7fd40cc870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:48:45.416 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:45.415+0000 7f7fdae37640 1 --2- 172.21.15.123:0/1057522078 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7fd40cc3c0 0x7f7fd40cc870 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f7fcc009c80 tx=0x7f7fcc01cda0 comp rx=0 tx=0).stop 2024-09-16T15:48:45.416 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:45.415+0000 7f7fdae37640 1 -- 172.21.15.123:0/1057522078 shutdown_connections 2024-09-16T15:48:45.416 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:45.415+0000 7f7fdae37640 1 --2- 172.21.15.123:0/1057522078 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7fd40cc3c0 0x7f7fd40cc870 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:45.416 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:45.415+0000 7f7fdae37640 1 --2- 172.21.15.123:0/1057522078 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f7fd415aa00 0x7f7fd415e5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:45.417 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:45.415+0000 7f7fdae37640 1 -- 172.21.15.123:0/1057522078 >> 172.21.15.123:0/1057522078 conn(0x7f7fd40b1710 msgr2=0x7f7fd40b1b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:48:45.417 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:45.415+0000 7f7fdae37640 1 -- 172.21.15.123:0/1057522078 shutdown_connections 2024-09-16T15:48:45.417 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:45.415+0000 7f7fdae37640 1 -- 172.21.15.123:0/1057522078 wait complete. 2024-09-16T15:48:45.417 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:45.415+0000 7f7fdae37640 1 Processor -- start 2024-09-16T15:48:45.417 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:45.415+0000 7f7fdae37640 1 -- start start 2024-09-16T15:48:45.419 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:45.419+0000 7f7fdae37640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f7fd4072720 0x7f7fd414d4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:48:45.419 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:45.419+0000 7f7fdae37640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7fd40cc3c0 0x7f7fd415c590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:48:45.420 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:45.419+0000 7f7fdae37640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7fd415aa00 0x7f7fd40b33b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:48:45.420 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:45.419+0000 7f7fdae37640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f7fd4167af0 con 0x7f7fd40cc3c0 2024-09-16T15:48:45.420 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:45.419+0000 7f7fdae37640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f7fd4167970 con 0x7f7fd415aa00 2024-09-16T15:48:45.420 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:45.419+0000 7f7fdae37640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f7fd4167c70 con 0x7f7fd4072720 2024-09-16T15:48:45.420 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:45.419+0000 7f7fd3fff640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7fd40cc3c0 0x7f7fd415c590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:48:45.420 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:45.419+0000 7f7fd3fff640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7fd40cc3c0 0x7f7fd415c590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:46120/0 (socket says 172.21.15.123:46120) 2024-09-16T15:48:45.420 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:45.419+0000 7f7fd3fff640 1 -- 172.21.15.123:0/1721923499 learned_addr learned my addr 172.21.15.123:0/1721923499 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:48:45.420 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:45.419+0000 7f7fd8bd3640 1 --2- 172.21.15.123:0/1721923499 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f7fd4072720 0x7f7fd414d4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:48:45.420 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:45.419+0000 7f7fd93d4640 1 --2- 172.21.15.123:0/1721923499 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7fd415aa00 0x7f7fd40b33b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:48:45.420 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:45.419+0000 7f7fd3fff640 1 -- 172.21.15.123:0/1721923499 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f7fd4072720 msgr2=0x7f7fd414d4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:48:45.420 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:45.419+0000 7f7fd3fff640 1 --2- 172.21.15.123:0/1721923499 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f7fd4072720 0x7f7fd414d4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:45.420 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:45.419+0000 7f7fd3fff640 1 -- 172.21.15.123:0/1721923499 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7fd415aa00 msgr2=0x7f7fd40b33b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:48:45.420 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:45.419+0000 7f7fd3fff640 1 --2- 172.21.15.123:0/1721923499 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7fd415aa00 0x7f7fd40b33b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:45.421 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:45.419+0000 7f7fd3fff640 1 -- 172.21.15.123:0/1721923499 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7fd415cf50 con 0x7f7fd40cc3c0 2024-09-16T15:48:45.421 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:45.419+0000 7f7fd3fff640 1 --2- 172.21.15.123:0/1721923499 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7fd40cc3c0 0x7f7fd415c590 secure :-1 s=READY pgs=1142 cs=0 l=1 rev1=1 crypto rx=0x7f7fc800c430 tx=0x7f7fc800c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:48:45.421 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:45.419+0000 7f7fd17fa640 1 -- 172.21.15.123:0/1721923499 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7fc8017070 con 0x7f7fd40cc3c0 2024-09-16T15:48:45.421 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:45.419+0000 7f7fdae37640 1 -- 172.21.15.123:0/1721923499 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7fd415d230 con 0x7f7fd40cc3c0 2024-09-16T15:48:45.421 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:45.419+0000 7f7fd17fa640 1 -- 172.21.15.123:0/1721923499 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f7fc80053e0 con 0x7f7fd40cc3c0 2024-09-16T15:48:45.421 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:45.419+0000 7f7fd17fa640 1 -- 172.21.15.123:0/1721923499 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7fc8003860 con 0x7f7fd40cc3c0 2024-09-16T15:48:45.421 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:45.419+0000 7f7fdae37640 1 -- 172.21.15.123:0/1721923499 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7fd40b2e90 con 0x7f7fd40cc3c0 2024-09-16T15:48:45.422 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:45.419+0000 7f7fdae37640 1 -- 172.21.15.123:0/1721923499 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7fd40b3090 con 0x7f7fd40cc3c0 2024-09-16T15:48:45.422 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:45.419+0000 7f7fd17fa640 1 -- 172.21.15.123:0/1721923499 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 17) ==== 97233+0+0 (secure 0 0 0) 0x7f7fc801a070 con 0x7f7fd40cc3c0 2024-09-16T15:48:45.422 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:45.423+0000 7f7fd17fa640 1 --2- 172.21.15.123:0/1721923499 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f7fc0073070 0x7f7fc0075530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:48:45.422 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:45.423+0000 7f7fd8bd3640 1 --2- 172.21.15.123:0/1721923499 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f7fc0073070 0x7f7fc0075530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:48:45.422 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:45.423+0000 7f7fd17fa640 1 -- 172.21.15.123:0/1721923499 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(105..105 src has 1..105) ==== 5767+0+0 (secure 0 0 0) 0x7f7fc8093ab0 con 0x7f7fd40cc3c0 2024-09-16T15:48:45.422 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:45.423+0000 7f7fd8bd3640 1 --2- 172.21.15.123:0/1721923499 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f7fc0073070 0x7f7fc0075530 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f7fc4009ad0 tx=0x7f7fc40091e0 comp rx=0 tx=0).ready entity=mgr.4150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:48:45.425 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:45.427+0000 7f7fd17fa640 1 -- 172.21.15.123:0/1721923499 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f7fd40b3090 con 0x7f7fd40cc3c0 2024-09-16T15:48:45.606 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:45.607+0000 7f7fd17fa640 1 -- 172.21.15.123:0/1721923499 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mgrmap(e 18) ==== 97386+0+0 (secure 0 0 0) 0x7f7fc805a4a0 con 0x7f7fd40cc3c0 2024-09-16T15:48:45.635 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:45.635+0000 7f7fdae37640 1 -- 172.21.15.123:0/1721923499 --> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] -- mgr_command(tid 0: {"prefix": "fs subvolume info", "vol_name": "cephfs", "sub_name": "subvol_clone4560", "target": ["mon-mgr", ""]}) -- 0x7f7fd415d400 con 0x7f7fc0073070 2024-09-16T15:48:45.653 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:45.651+0000 7f7fd17fa640 1 -- 172.21.15.123:0/1721923499 <== mgr.4150 v2:172.21.15.123:6836/2464759676 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+743 (secure 0 0 0) 0x7f7fd415d400 con 0x7f7fc0073070 2024-09-16T15:48:45.653 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2024-09-16T15:48:45.653 INFO:teuthology.orchestra.run.smithi123.stdout: "atime": "2024-09-16 15:48:38", 2024-09-16T15:48:45.653 INFO:teuthology.orchestra.run.smithi123.stdout: "bytes_pcent": "80.00", 2024-09-16T15:48:45.653 INFO:teuthology.orchestra.run.smithi123.stdout: "bytes_quota": 10485760, 2024-09-16T15:48:45.653 INFO:teuthology.orchestra.run.smithi123.stdout: "bytes_used": 8388608, 2024-09-16T15:48:45.653 INFO:teuthology.orchestra.run.smithi123.stdout: "created_at": "2024-09-16 15:48:43", 2024-09-16T15:48:45.653 INFO:teuthology.orchestra.run.smithi123.stdout: "ctime": "2024-09-16 15:48:43", 2024-09-16T15:48:45.654 INFO:teuthology.orchestra.run.smithi123.stdout: "data_pool": "cephfs_data", 2024-09-16T15:48:45.654 INFO:teuthology.orchestra.run.smithi123.stdout: "earmark": "", 2024-09-16T15:48:45.654 INFO:teuthology.orchestra.run.smithi123.stdout: "features": [ 2024-09-16T15:48:45.654 INFO:teuthology.orchestra.run.smithi123.stdout: "snapshot-clone", 2024-09-16T15:48:45.654 INFO:teuthology.orchestra.run.smithi123.stdout: "snapshot-autoprotect", 2024-09-16T15:48:45.654 INFO:teuthology.orchestra.run.smithi123.stdout: "snapshot-retention" 2024-09-16T15:48:45.654 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-09-16T15:48:45.654 INFO:teuthology.orchestra.run.smithi123.stdout: "flavor": 2, 2024-09-16T15:48:45.654 INFO:teuthology.orchestra.run.smithi123.stdout: "gid": 1000, 2024-09-16T15:48:45.654 INFO:teuthology.orchestra.run.smithi123.stdout: "mode": 16895, 2024-09-16T15:48:45.654 INFO:teuthology.orchestra.run.smithi123.stdout: "mon_addrs": [ 2024-09-16T15:48:45.654 INFO:teuthology.orchestra.run.smithi123.stdout: "172.21.15.123:6789", 2024-09-16T15:48:45.654 INFO:teuthology.orchestra.run.smithi123.stdout: "172.21.15.184:6789", 2024-09-16T15:48:45.654 INFO:teuthology.orchestra.run.smithi123.stdout: "172.21.15.184:6790" 2024-09-16T15:48:45.655 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2024-09-16T15:48:45.655 INFO:teuthology.orchestra.run.smithi123.stdout: "mtime": "2024-09-16 15:48:42", 2024-09-16T15:48:45.655 INFO:teuthology.orchestra.run.smithi123.stdout: "path": "/volumes/_nogroup/subvol_clone4560/dfb063e9-21ee-464c-9273-de9270d8f8a2", 2024-09-16T15:48:45.655 INFO:teuthology.orchestra.run.smithi123.stdout: "pool_namespace": "", 2024-09-16T15:48:45.655 INFO:teuthology.orchestra.run.smithi123.stdout: "state": "complete", 2024-09-16T15:48:45.655 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "clone", 2024-09-16T15:48:45.655 INFO:teuthology.orchestra.run.smithi123.stdout: "uid": 1000 2024-09-16T15:48:45.655 INFO:teuthology.orchestra.run.smithi123.stdout:} 2024-09-16T15:48:45.656 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:45.655+0000 7f7fdae37640 1 -- 172.21.15.123:0/1721923499 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f7fc0073070 msgr2=0x7f7fc0075530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:48:45.656 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:45.655+0000 7f7fdae37640 1 --2- 172.21.15.123:0/1721923499 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f7fc0073070 0x7f7fc0075530 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f7fc4009ad0 tx=0x7f7fc40091e0 comp rx=0 tx=0).stop 2024-09-16T15:48:45.656 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:45.655+0000 7f7fdae37640 1 -- 172.21.15.123:0/1721923499 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7fd40cc3c0 msgr2=0x7f7fd415c590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:48:45.656 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:45.655+0000 7f7fdae37640 1 --2- 172.21.15.123:0/1721923499 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7fd40cc3c0 0x7f7fd415c590 secure :-1 s=READY pgs=1142 cs=0 l=1 rev1=1 crypto rx=0x7f7fc800c430 tx=0x7f7fc800c8f0 comp rx=0 tx=0).stop 2024-09-16T15:48:45.656 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:45.655+0000 7f7fdae37640 1 -- 172.21.15.123:0/1721923499 shutdown_connections 2024-09-16T15:48:45.656 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:45.655+0000 7f7fdae37640 1 --2- 172.21.15.123:0/1721923499 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f7fc0073070 0x7f7fc0075530 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:45.656 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:45.655+0000 7f7fdae37640 1 --2- 172.21.15.123:0/1721923499 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7fd415aa00 0x7f7fd40b33b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:45.656 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:45.655+0000 7f7fdae37640 1 --2- 172.21.15.123:0/1721923499 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7fd40cc3c0 0x7f7fd415c590 unknown :-1 s=CLOSED pgs=1142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:45.657 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:45.655+0000 7f7fdae37640 1 --2- 172.21.15.123:0/1721923499 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f7fd4072720 0x7f7fd414d4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:45.657 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:45.655+0000 7f7fdae37640 1 -- 172.21.15.123:0/1721923499 >> 172.21.15.123:0/1721923499 conn(0x7f7fd40b1710 msgr2=0x7f7fd415b8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:48:45.657 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:45.655+0000 7f7fdae37640 1 -- 172.21.15.123:0/1721923499 shutdown_connections 2024-09-16T15:48:45.657 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:45.655+0000 7f7fdae37640 1 -- 172.21.15.123:0/1721923499 wait complete. 2024-09-16T15:48:45.670 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:48:45.670 DEBUG:teuthology.orchestra.run.smithi123:> (cd /home/ubuntu/cephtest/mnt.0 && exec getfattr --only-values -n ceph.quota.max_bytes volumes/_nogroup/subvol37810/b00a5d9c-1a58-42eb-be9a-5f5c5fe23179/.snap/subvol_snap6960) 2024-09-16T15:48:45.686 INFO:teuthology.orchestra.run.smithi123.stdout:10485760 2024-09-16T15:48:45.686 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:48:45.686 DEBUG:teuthology.orchestra.run.smithi123:> (cd /home/ubuntu/cephtest/mnt.0 && exec getfattr --only-values -n ceph.dir.layout.pool volumes/_nogroup/subvol37810/b00a5d9c-1a58-42eb-be9a-5f5c5fe23179/.snap/subvol_snap6960) 2024-09-16T15:48:45.741 INFO:teuthology.orchestra.run.smithi123.stdout:cephfs_data 2024-09-16T15:48:45.742 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:48:45.742 DEBUG:teuthology.orchestra.run.smithi123:> (cd /home/ubuntu/cephtest/mnt.0 && exec getfattr --only-values -n ceph.dir.layout.pool_namespace volumes/_nogroup/subvol37810/b00a5d9c-1a58-42eb-be9a-5f5c5fe23179/.snap/subvol_snap6960) 2024-09-16T15:48:45.799 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:48:45.800 DEBUG:teuthology.orchestra.run.smithi123:> (cd /home/ubuntu/cephtest/mnt.0 && exec find volumes/_nogroup/subvol37810/b00a5d9c-1a58-42eb-be9a-5f5c5fe23179/.snap/subvol_snap6960) 2024-09-16T15:48:45.859 INFO:teuthology.orchestra.run.smithi123.stdout:volumes/_nogroup/subvol37810/b00a5d9c-1a58-42eb-be9a-5f5c5fe23179/.snap/subvol_snap6960 2024-09-16T15:48:45.859 INFO:teuthology.orchestra.run.smithi123.stdout:volumes/_nogroup/subvol37810/b00a5d9c-1a58-42eb-be9a-5f5c5fe23179/.snap/subvol_snap6960/subvolume_file.2 2024-09-16T15:48:45.859 INFO:teuthology.orchestra.run.smithi123.stdout:volumes/_nogroup/subvol37810/b00a5d9c-1a58-42eb-be9a-5f5c5fe23179/.snap/subvol_snap6960/subvolume_file.5 2024-09-16T15:48:45.859 INFO:teuthology.orchestra.run.smithi123.stdout:volumes/_nogroup/subvol37810/b00a5d9c-1a58-42eb-be9a-5f5c5fe23179/.snap/subvol_snap6960/subvolume_file.4 2024-09-16T15:48:45.859 INFO:teuthology.orchestra.run.smithi123.stdout:volumes/_nogroup/subvol37810/b00a5d9c-1a58-42eb-be9a-5f5c5fe23179/.snap/subvol_snap6960/subvolume_file.7 2024-09-16T15:48:45.860 INFO:teuthology.orchestra.run.smithi123.stdout:volumes/_nogroup/subvol37810/b00a5d9c-1a58-42eb-be9a-5f5c5fe23179/.snap/subvol_snap6960/subvolume_file.6 2024-09-16T15:48:45.860 INFO:teuthology.orchestra.run.smithi123.stdout:volumes/_nogroup/subvol37810/b00a5d9c-1a58-42eb-be9a-5f5c5fe23179/.snap/subvol_snap6960/subvolume_file.0 2024-09-16T15:48:45.860 INFO:teuthology.orchestra.run.smithi123.stdout:volumes/_nogroup/subvol37810/b00a5d9c-1a58-42eb-be9a-5f5c5fe23179/.snap/subvol_snap6960/subvolume_file.3 2024-09-16T15:48:45.860 INFO:teuthology.orchestra.run.smithi123.stdout:volumes/_nogroup/subvol37810/b00a5d9c-1a58-42eb-be9a-5f5c5fe23179/.snap/subvol_snap6960/subvolume_file.1 2024-09-16T15:48:45.861 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:48:45.861 DEBUG:teuthology.orchestra.run.smithi123:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol37810/b00a5d9c-1a58-42eb-be9a-5f5c5fe23179/.snap/subvol_snap6960) 2024-09-16T15:48:45.913 INFO:teuthology.orchestra.run.smithi123.stdout:41ff 2024-09-16T15:48:45.914 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:48:45.914 DEBUG:teuthology.orchestra.run.smithi123:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol_clone4560/dfb063e9-21ee-464c-9273-de9270d8f8a2/) 2024-09-16T15:48:45.964 INFO:teuthology.orchestra.run.smithi123.stdout:41ff 2024-09-16T15:48:45.964 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:48:45.964 DEBUG:teuthology.orchestra.run.smithi123:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol37810/b00a5d9c-1a58-42eb-be9a-5f5c5fe23179/.snap/subvol_snap6960) 2024-09-16T15:48:46.013 INFO:teuthology.orchestra.run.smithi123.stdout:1000 2024-09-16T15:48:46.014 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:48:46.014 DEBUG:teuthology.orchestra.run.smithi123:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol_clone4560/dfb063e9-21ee-464c-9273-de9270d8f8a2/) 2024-09-16T15:48:46.063 INFO:teuthology.orchestra.run.smithi123.stdout:1000 2024-09-16T15:48:46.064 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:48:46.064 DEBUG:teuthology.orchestra.run.smithi123:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol37810/b00a5d9c-1a58-42eb-be9a-5f5c5fe23179/.snap/subvol_snap6960) 2024-09-16T15:48:46.113 INFO:teuthology.orchestra.run.smithi123.stdout:1000 2024-09-16T15:48:46.114 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:48:46.114 DEBUG:teuthology.orchestra.run.smithi123:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol_clone4560/dfb063e9-21ee-464c-9273-de9270d8f8a2/) 2024-09-16T15:48:46.163 INFO:teuthology.orchestra.run.smithi123.stdout:1000 2024-09-16T15:48:46.163 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:48:46.163 DEBUG:teuthology.orchestra.run.smithi123:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol37810/b00a5d9c-1a58-42eb-be9a-5f5c5fe23179/.snap/subvol_snap6960) 2024-09-16T15:48:46.213 INFO:teuthology.orchestra.run.smithi123.stdout:1726501722 2024-09-16T15:48:46.214 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:48:46.214 DEBUG:teuthology.orchestra.run.smithi123:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol_clone4560/dfb063e9-21ee-464c-9273-de9270d8f8a2/) 2024-09-16T15:48:46.263 INFO:teuthology.orchestra.run.smithi123.stdout:1726501722 2024-09-16T15:48:46.264 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:48:46.264 DEBUG:teuthology.orchestra.run.smithi123:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol37810/b00a5d9c-1a58-42eb-be9a-5f5c5fe23179/.snap/subvol_snap6960/subvolume_file.2) 2024-09-16T15:48:46.315 INFO:teuthology.orchestra.run.smithi123.stdout:81b4 2024-09-16T15:48:46.315 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:48:46.316 DEBUG:teuthology.orchestra.run.smithi123:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol_clone4560/dfb063e9-21ee-464c-9273-de9270d8f8a2/subvolume_file.2) 2024-09-16T15:48:46.377 INFO:teuthology.orchestra.run.smithi123.stdout:81b4 2024-09-16T15:48:46.378 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:48:46.378 DEBUG:teuthology.orchestra.run.smithi123:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol37810/b00a5d9c-1a58-42eb-be9a-5f5c5fe23179/.snap/subvol_snap6960/subvolume_file.2) 2024-09-16T15:48:46.430 INFO:teuthology.orchestra.run.smithi123.stdout:1000 2024-09-16T15:48:46.430 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:48:46.431 DEBUG:teuthology.orchestra.run.smithi123:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol_clone4560/dfb063e9-21ee-464c-9273-de9270d8f8a2/subvolume_file.2) 2024-09-16T15:48:46.483 INFO:teuthology.orchestra.run.smithi123.stdout:1000 2024-09-16T15:48:46.484 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:48:46.484 DEBUG:teuthology.orchestra.run.smithi123:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol37810/b00a5d9c-1a58-42eb-be9a-5f5c5fe23179/.snap/subvol_snap6960/subvolume_file.2) 2024-09-16T15:48:46.534 INFO:teuthology.orchestra.run.smithi123.stdout:1271 2024-09-16T15:48:46.535 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:48:46.535 DEBUG:teuthology.orchestra.run.smithi123:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol_clone4560/dfb063e9-21ee-464c-9273-de9270d8f8a2/subvolume_file.2) 2024-09-16T15:48:46.583 INFO:teuthology.orchestra.run.smithi123.stdout:1271 2024-09-16T15:48:46.584 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:48:46.584 DEBUG:teuthology.orchestra.run.smithi123:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol37810/b00a5d9c-1a58-42eb-be9a-5f5c5fe23179/.snap/subvol_snap6960/subvolume_file.2) 2024-09-16T15:48:46.635 INFO:teuthology.orchestra.run.smithi123.stdout:1726501721 2024-09-16T15:48:46.635 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:48:46.635 DEBUG:teuthology.orchestra.run.smithi123:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol_clone4560/dfb063e9-21ee-464c-9273-de9270d8f8a2/subvolume_file.2) 2024-09-16T15:48:46.683 INFO:teuthology.orchestra.run.smithi123.stdout:1726501721 2024-09-16T15:48:46.684 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:48:46.684 DEBUG:teuthology.orchestra.run.smithi123:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol37810/b00a5d9c-1a58-42eb-be9a-5f5c5fe23179/.snap/subvol_snap6960/subvolume_file.5) 2024-09-16T15:48:46.735 INFO:teuthology.orchestra.run.smithi123.stdout:81b4 2024-09-16T15:48:46.735 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:48:46.735 DEBUG:teuthology.orchestra.run.smithi123:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol_clone4560/dfb063e9-21ee-464c-9273-de9270d8f8a2/subvolume_file.5) 2024-09-16T15:48:46.803 INFO:teuthology.orchestra.run.smithi123.stdout:81b4 2024-09-16T15:48:46.804 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:48:46.804 DEBUG:teuthology.orchestra.run.smithi123:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol37810/b00a5d9c-1a58-42eb-be9a-5f5c5fe23179/.snap/subvol_snap6960/subvolume_file.5) 2024-09-16T15:48:46.854 INFO:teuthology.orchestra.run.smithi123.stdout:1000 2024-09-16T15:48:46.855 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:48:46.855 DEBUG:teuthology.orchestra.run.smithi123:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol_clone4560/dfb063e9-21ee-464c-9273-de9270d8f8a2/subvolume_file.5) 2024-09-16T15:48:46.903 INFO:teuthology.orchestra.run.smithi123.stdout:1000 2024-09-16T15:48:46.904 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:48:46.904 DEBUG:teuthology.orchestra.run.smithi123:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol37810/b00a5d9c-1a58-42eb-be9a-5f5c5fe23179/.snap/subvol_snap6960/subvolume_file.5) 2024-09-16T15:48:46.954 INFO:teuthology.orchestra.run.smithi123.stdout:1271 2024-09-16T15:48:46.954 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:48:46.955 DEBUG:teuthology.orchestra.run.smithi123:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol_clone4560/dfb063e9-21ee-464c-9273-de9270d8f8a2/subvolume_file.5) 2024-09-16T15:48:47.003 INFO:teuthology.orchestra.run.smithi123.stdout:1271 2024-09-16T15:48:47.004 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:48:47.004 DEBUG:teuthology.orchestra.run.smithi123:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol37810/b00a5d9c-1a58-42eb-be9a-5f5c5fe23179/.snap/subvol_snap6960/subvolume_file.5) 2024-09-16T15:48:47.054 INFO:teuthology.orchestra.run.smithi123.stdout:1726501722 2024-09-16T15:48:47.054 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:48:47.055 DEBUG:teuthology.orchestra.run.smithi123:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol_clone4560/dfb063e9-21ee-464c-9273-de9270d8f8a2/subvolume_file.5) 2024-09-16T15:48:47.104 INFO:teuthology.orchestra.run.smithi123.stdout:1726501722 2024-09-16T15:48:47.104 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:48:47.105 DEBUG:teuthology.orchestra.run.smithi123:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol37810/b00a5d9c-1a58-42eb-be9a-5f5c5fe23179/.snap/subvol_snap6960/subvolume_file.4) 2024-09-16T15:48:47.154 INFO:teuthology.orchestra.run.smithi123.stdout:81b4 2024-09-16T15:48:47.155 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:48:47.155 DEBUG:teuthology.orchestra.run.smithi123:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol_clone4560/dfb063e9-21ee-464c-9273-de9270d8f8a2/subvolume_file.4) 2024-09-16T15:48:47.211 INFO:teuthology.orchestra.run.smithi123.stdout:81b4 2024-09-16T15:48:47.212 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:48:47.213 DEBUG:teuthology.orchestra.run.smithi123:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol37810/b00a5d9c-1a58-42eb-be9a-5f5c5fe23179/.snap/subvol_snap6960/subvolume_file.4) 2024-09-16T15:48:47.261 INFO:teuthology.orchestra.run.smithi123.stdout:1000 2024-09-16T15:48:47.262 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:48:47.262 DEBUG:teuthology.orchestra.run.smithi123:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol_clone4560/dfb063e9-21ee-464c-9273-de9270d8f8a2/subvolume_file.4) 2024-09-16T15:48:47.310 INFO:teuthology.orchestra.run.smithi123.stdout:1000 2024-09-16T15:48:47.311 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:48:47.311 DEBUG:teuthology.orchestra.run.smithi123:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol37810/b00a5d9c-1a58-42eb-be9a-5f5c5fe23179/.snap/subvol_snap6960/subvolume_file.4) 2024-09-16T15:48:47.362 INFO:teuthology.orchestra.run.smithi123.stdout:1271 2024-09-16T15:48:47.362 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:48:47.362 DEBUG:teuthology.orchestra.run.smithi123:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol_clone4560/dfb063e9-21ee-464c-9273-de9270d8f8a2/subvolume_file.4) 2024-09-16T15:48:47.411 INFO:teuthology.orchestra.run.smithi123.stdout:1271 2024-09-16T15:48:47.412 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:48:47.412 DEBUG:teuthology.orchestra.run.smithi123:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol37810/b00a5d9c-1a58-42eb-be9a-5f5c5fe23179/.snap/subvol_snap6960/subvolume_file.4) 2024-09-16T15:48:47.461 INFO:teuthology.orchestra.run.smithi123.stdout:1726501721 2024-09-16T15:48:47.462 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:48:47.462 DEBUG:teuthology.orchestra.run.smithi123:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol_clone4560/dfb063e9-21ee-464c-9273-de9270d8f8a2/subvolume_file.4) 2024-09-16T15:48:47.511 INFO:teuthology.orchestra.run.smithi123.stdout:1726501721 2024-09-16T15:48:47.512 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:48:47.512 DEBUG:teuthology.orchestra.run.smithi123:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol37810/b00a5d9c-1a58-42eb-be9a-5f5c5fe23179/.snap/subvol_snap6960/subvolume_file.7) 2024-09-16T15:48:47.562 INFO:teuthology.orchestra.run.smithi123.stdout:81b4 2024-09-16T15:48:47.562 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:48:47.563 DEBUG:teuthology.orchestra.run.smithi123:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol_clone4560/dfb063e9-21ee-464c-9273-de9270d8f8a2/subvolume_file.7) 2024-09-16T15:48:47.622 INFO:teuthology.orchestra.run.smithi123.stdout:81b4 2024-09-16T15:48:47.622 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:48:47.622 DEBUG:teuthology.orchestra.run.smithi123:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol37810/b00a5d9c-1a58-42eb-be9a-5f5c5fe23179/.snap/subvol_snap6960/subvolume_file.7) 2024-09-16T15:48:47.678 INFO:teuthology.orchestra.run.smithi123.stdout:1000 2024-09-16T15:48:47.678 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:48:47.679 DEBUG:teuthology.orchestra.run.smithi123:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol_clone4560/dfb063e9-21ee-464c-9273-de9270d8f8a2/subvolume_file.7) 2024-09-16T15:48:47.731 INFO:teuthology.orchestra.run.smithi123.stdout:1000 2024-09-16T15:48:47.732 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:48:47.732 DEBUG:teuthology.orchestra.run.smithi123:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol37810/b00a5d9c-1a58-42eb-be9a-5f5c5fe23179/.snap/subvol_snap6960/subvolume_file.7) 2024-09-16T15:48:47.782 INFO:teuthology.orchestra.run.smithi123.stdout:1271 2024-09-16T15:48:47.782 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:48:47.783 DEBUG:teuthology.orchestra.run.smithi123:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol_clone4560/dfb063e9-21ee-464c-9273-de9270d8f8a2/subvolume_file.7) 2024-09-16T15:48:47.831 INFO:teuthology.orchestra.run.smithi123.stdout:1271 2024-09-16T15:48:47.832 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:48:47.832 DEBUG:teuthology.orchestra.run.smithi123:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol37810/b00a5d9c-1a58-42eb-be9a-5f5c5fe23179/.snap/subvol_snap6960/subvolume_file.7) 2024-09-16T15:48:47.882 INFO:teuthology.orchestra.run.smithi123.stdout:1726501722 2024-09-16T15:48:47.883 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:48:47.883 DEBUG:teuthology.orchestra.run.smithi123:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol_clone4560/dfb063e9-21ee-464c-9273-de9270d8f8a2/subvolume_file.7) 2024-09-16T15:48:47.931 INFO:teuthology.orchestra.run.smithi123.stdout:1726501722 2024-09-16T15:48:47.932 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:48:47.932 DEBUG:teuthology.orchestra.run.smithi123:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol37810/b00a5d9c-1a58-42eb-be9a-5f5c5fe23179/.snap/subvol_snap6960/subvolume_file.6) 2024-09-16T15:48:47.982 INFO:teuthology.orchestra.run.smithi123.stdout:81b4 2024-09-16T15:48:47.983 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:48:47.983 DEBUG:teuthology.orchestra.run.smithi123:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol_clone4560/dfb063e9-21ee-464c-9273-de9270d8f8a2/subvolume_file.6) 2024-09-16T15:48:48.042 INFO:teuthology.orchestra.run.smithi123.stdout:81b4 2024-09-16T15:48:48.043 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:48:48.043 DEBUG:teuthology.orchestra.run.smithi123:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol37810/b00a5d9c-1a58-42eb-be9a-5f5c5fe23179/.snap/subvol_snap6960/subvolume_file.6) 2024-09-16T15:48:48.094 INFO:teuthology.orchestra.run.smithi123.stdout:1000 2024-09-16T15:48:48.094 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:48:48.094 DEBUG:teuthology.orchestra.run.smithi123:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol_clone4560/dfb063e9-21ee-464c-9273-de9270d8f8a2/subvolume_file.6) 2024-09-16T15:48:48.143 INFO:teuthology.orchestra.run.smithi123.stdout:1000 2024-09-16T15:48:48.144 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:48:48.145 DEBUG:teuthology.orchestra.run.smithi123:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol37810/b00a5d9c-1a58-42eb-be9a-5f5c5fe23179/.snap/subvol_snap6960/subvolume_file.6) 2024-09-16T15:48:48.195 INFO:teuthology.orchestra.run.smithi123.stdout:1271 2024-09-16T15:48:48.195 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:48:48.195 DEBUG:teuthology.orchestra.run.smithi123:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol_clone4560/dfb063e9-21ee-464c-9273-de9270d8f8a2/subvolume_file.6) 2024-09-16T15:48:48.243 INFO:teuthology.orchestra.run.smithi123.stdout:1271 2024-09-16T15:48:48.244 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:48:48.244 DEBUG:teuthology.orchestra.run.smithi123:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol37810/b00a5d9c-1a58-42eb-be9a-5f5c5fe23179/.snap/subvol_snap6960/subvolume_file.6) 2024-09-16T15:48:48.293 INFO:teuthology.orchestra.run.smithi123.stdout:1726501722 2024-09-16T15:48:48.293 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:48:48.294 DEBUG:teuthology.orchestra.run.smithi123:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol_clone4560/dfb063e9-21ee-464c-9273-de9270d8f8a2/subvolume_file.6) 2024-09-16T15:48:48.343 INFO:teuthology.orchestra.run.smithi123.stdout:1726501722 2024-09-16T15:48:48.344 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:48:48.344 DEBUG:teuthology.orchestra.run.smithi123:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol37810/b00a5d9c-1a58-42eb-be9a-5f5c5fe23179/.snap/subvol_snap6960/subvolume_file.0) 2024-09-16T15:48:48.393 INFO:teuthology.orchestra.run.smithi123.stdout:81b4 2024-09-16T15:48:48.394 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:48:48.394 DEBUG:teuthology.orchestra.run.smithi123:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol_clone4560/dfb063e9-21ee-464c-9273-de9270d8f8a2/subvolume_file.0) 2024-09-16T15:48:48.452 INFO:teuthology.orchestra.run.smithi123.stdout:81b4 2024-09-16T15:48:48.453 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:48:48.453 DEBUG:teuthology.orchestra.run.smithi123:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol37810/b00a5d9c-1a58-42eb-be9a-5f5c5fe23179/.snap/subvol_snap6960/subvolume_file.0) 2024-09-16T15:48:48.506 INFO:teuthology.orchestra.run.smithi123.stdout:1000 2024-09-16T15:48:48.506 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:48:48.506 DEBUG:teuthology.orchestra.run.smithi123:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol_clone4560/dfb063e9-21ee-464c-9273-de9270d8f8a2/subvolume_file.0) 2024-09-16T15:48:48.555 INFO:teuthology.orchestra.run.smithi123.stdout:1000 2024-09-16T15:48:48.556 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:48:48.556 DEBUG:teuthology.orchestra.run.smithi123:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol37810/b00a5d9c-1a58-42eb-be9a-5f5c5fe23179/.snap/subvol_snap6960/subvolume_file.0) 2024-09-16T15:48:48.606 INFO:teuthology.orchestra.run.smithi123.stdout:1271 2024-09-16T15:48:48.606 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:48:48.606 DEBUG:teuthology.orchestra.run.smithi123:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol_clone4560/dfb063e9-21ee-464c-9273-de9270d8f8a2/subvolume_file.0) 2024-09-16T15:48:48.655 INFO:teuthology.orchestra.run.smithi123.stdout:1271 2024-09-16T15:48:48.656 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:48:48.656 DEBUG:teuthology.orchestra.run.smithi123:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol37810/b00a5d9c-1a58-42eb-be9a-5f5c5fe23179/.snap/subvol_snap6960/subvolume_file.0) 2024-09-16T15:48:48.705 INFO:teuthology.orchestra.run.smithi123.stdout:1726501721 2024-09-16T15:48:48.706 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:48:48.706 DEBUG:teuthology.orchestra.run.smithi123:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol_clone4560/dfb063e9-21ee-464c-9273-de9270d8f8a2/subvolume_file.0) 2024-09-16T15:48:48.756 INFO:teuthology.orchestra.run.smithi123.stdout:1726501721 2024-09-16T15:48:48.757 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:48:48.757 DEBUG:teuthology.orchestra.run.smithi123:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol37810/b00a5d9c-1a58-42eb-be9a-5f5c5fe23179/.snap/subvol_snap6960/subvolume_file.3) 2024-09-16T15:48:48.805 INFO:teuthology.orchestra.run.smithi123.stdout:81b4 2024-09-16T15:48:48.806 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:48:48.806 DEBUG:teuthology.orchestra.run.smithi123:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol_clone4560/dfb063e9-21ee-464c-9273-de9270d8f8a2/subvolume_file.3) 2024-09-16T15:48:48.864 INFO:teuthology.orchestra.run.smithi123.stdout:81b4 2024-09-16T15:48:48.865 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:48:48.865 DEBUG:teuthology.orchestra.run.smithi123:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol37810/b00a5d9c-1a58-42eb-be9a-5f5c5fe23179/.snap/subvol_snap6960/subvolume_file.3) 2024-09-16T15:48:48.918 INFO:teuthology.orchestra.run.smithi123.stdout:1000 2024-09-16T15:48:48.919 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:48:48.919 DEBUG:teuthology.orchestra.run.smithi123:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol_clone4560/dfb063e9-21ee-464c-9273-de9270d8f8a2/subvolume_file.3) 2024-09-16T15:48:48.967 INFO:teuthology.orchestra.run.smithi123.stdout:1000 2024-09-16T15:48:48.968 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:48:48.968 DEBUG:teuthology.orchestra.run.smithi123:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol37810/b00a5d9c-1a58-42eb-be9a-5f5c5fe23179/.snap/subvol_snap6960/subvolume_file.3) 2024-09-16T15:48:49.017 INFO:teuthology.orchestra.run.smithi123.stdout:1271 2024-09-16T15:48:49.018 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:48:49.018 DEBUG:teuthology.orchestra.run.smithi123:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol_clone4560/dfb063e9-21ee-464c-9273-de9270d8f8a2/subvolume_file.3) 2024-09-16T15:48:49.067 INFO:teuthology.orchestra.run.smithi123.stdout:1271 2024-09-16T15:48:49.068 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:48:49.068 DEBUG:teuthology.orchestra.run.smithi123:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol37810/b00a5d9c-1a58-42eb-be9a-5f5c5fe23179/.snap/subvol_snap6960/subvolume_file.3) 2024-09-16T15:48:49.117 INFO:teuthology.orchestra.run.smithi123.stdout:1726501721 2024-09-16T15:48:49.118 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:48:49.118 DEBUG:teuthology.orchestra.run.smithi123:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol_clone4560/dfb063e9-21ee-464c-9273-de9270d8f8a2/subvolume_file.3) 2024-09-16T15:48:49.167 INFO:teuthology.orchestra.run.smithi123.stdout:1726501721 2024-09-16T15:48:49.168 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:48:49.168 DEBUG:teuthology.orchestra.run.smithi123:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol37810/b00a5d9c-1a58-42eb-be9a-5f5c5fe23179/.snap/subvol_snap6960/subvolume_file.1) 2024-09-16T15:48:49.218 INFO:teuthology.orchestra.run.smithi123.stdout:81b4 2024-09-16T15:48:49.219 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:48:49.219 DEBUG:teuthology.orchestra.run.smithi123:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%f volumes/_nogroup/subvol_clone4560/dfb063e9-21ee-464c-9273-de9270d8f8a2/subvolume_file.1) 2024-09-16T15:48:49.276 INFO:teuthology.orchestra.run.smithi123.stdout:81b4 2024-09-16T15:48:49.277 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:48:49.277 DEBUG:teuthology.orchestra.run.smithi123:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol37810/b00a5d9c-1a58-42eb-be9a-5f5c5fe23179/.snap/subvol_snap6960/subvolume_file.1) 2024-09-16T15:48:49.330 INFO:teuthology.orchestra.run.smithi123.stdout:1000 2024-09-16T15:48:49.331 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:48:49.331 DEBUG:teuthology.orchestra.run.smithi123:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%u volumes/_nogroup/subvol_clone4560/dfb063e9-21ee-464c-9273-de9270d8f8a2/subvolume_file.1) 2024-09-16T15:48:49.384 INFO:teuthology.orchestra.run.smithi123.stdout:1000 2024-09-16T15:48:49.385 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:48:49.385 DEBUG:teuthology.orchestra.run.smithi123:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol37810/b00a5d9c-1a58-42eb-be9a-5f5c5fe23179/.snap/subvol_snap6960/subvolume_file.1) 2024-09-16T15:48:49.434 INFO:teuthology.orchestra.run.smithi123.stdout:1271 2024-09-16T15:48:49.435 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:48:49.435 DEBUG:teuthology.orchestra.run.smithi123:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%g volumes/_nogroup/subvol_clone4560/dfb063e9-21ee-464c-9273-de9270d8f8a2/subvolume_file.1) 2024-09-16T15:48:49.488 INFO:teuthology.orchestra.run.smithi123.stdout:1271 2024-09-16T15:48:49.489 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:48:49.489 DEBUG:teuthology.orchestra.run.smithi123:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol37810/b00a5d9c-1a58-42eb-be9a-5f5c5fe23179/.snap/subvol_snap6960/subvolume_file.1) 2024-09-16T15:48:49.542 INFO:teuthology.orchestra.run.smithi123.stdout:1726501721 2024-09-16T15:48:49.542 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:48:49.543 DEBUG:teuthology.orchestra.run.smithi123:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat -c%Y volumes/_nogroup/subvol_clone4560/dfb063e9-21ee-464c-9273-de9270d8f8a2/subvolume_file.1) 2024-09-16T15:48:49.591 INFO:teuthology.orchestra.run.smithi123.stdout:1726501721 2024-09-16T15:48:49.592 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot clone cephfs subvol37810 fake subvol_clone62021 2024-09-16T15:48:49.778 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:49.779+0000 7f7fbd764640 1 -- 172.21.15.123:0/3088490789 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7fb80c3230 msgr2=0x7f7fb80cedd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:48:49.778 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:49.779+0000 7f7fbd764640 1 --2- 172.21.15.123:0/3088490789 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7fb80c3230 0x7f7fb80cedd0 secure :-1 s=READY pgs=1143 cs=0 l=1 rev1=1 crypto rx=0x7f7fb806db90 tx=0x7f7fa001c890 comp rx=0 tx=0).stop 2024-09-16T15:48:49.778 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:49.779+0000 7f7fbd764640 1 -- 172.21.15.123:0/3088490789 shutdown_connections 2024-09-16T15:48:49.778 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:49.779+0000 7f7fbd764640 1 --2- 172.21.15.123:0/3088490789 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f7fb80701e0 0x7f7fb80cf310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:49.778 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:49.779+0000 7f7fbd764640 1 --2- 172.21.15.123:0/3088490789 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7fb80c3230 0x7f7fb80cedd0 unknown :-1 s=CLOSED pgs=1143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:49.778 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:49.779+0000 7f7fbd764640 1 --2- 172.21.15.123:0/3088490789 >> v2:172.21.15.184:3300/0 conn(0x7f7fb815b9f0 0x7f7fb806fca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:49.778 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:49.779+0000 7f7fbd764640 1 -- 172.21.15.123:0/3088490789 >> 172.21.15.123:0/3088490789 conn(0x7f7fb80b1710 msgr2=0x7f7fb80b1b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:48:49.779 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:49.779+0000 7f7fbd764640 1 -- 172.21.15.123:0/3088490789 shutdown_connections 2024-09-16T15:48:49.779 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:49.779+0000 7f7fbd764640 1 -- 172.21.15.123:0/3088490789 wait complete. 2024-09-16T15:48:49.779 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:49.779+0000 7f7fbd764640 1 Processor -- start 2024-09-16T15:48:49.779 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:49.779+0000 7f7fbd764640 1 -- start start 2024-09-16T15:48:49.781 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:49.779+0000 7f7fbd764640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7fb80701e0 0x7f7fb814d170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:48:49.781 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:49.779+0000 7f7fbd764640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7fb80c3230 0x7f7fb814c450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:48:49.781 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:49.779+0000 7f7fbd764640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f7fb815b9f0 0x7f7fb80cb380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:48:49.782 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:49.779+0000 7f7fbd764640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f7fb815f6b0 con 0x7f7fb80701e0 2024-09-16T15:48:49.782 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:49.779+0000 7f7fbd764640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f7fb8073290 con 0x7f7fb80c3230 2024-09-16T15:48:49.782 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:49.779+0000 7f7fbd764640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f7fb815f830 con 0x7f7fb815b9f0 2024-09-16T15:48:49.782 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:49.779+0000 7f7fb6ffd640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7fb80701e0 0x7f7fb814d170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:48:49.782 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:49.779+0000 7f7fb6ffd640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7fb80701e0 0x7f7fb814d170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:46146/0 (socket says 172.21.15.123:46146) 2024-09-16T15:48:49.782 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:49.779+0000 7f7fb6ffd640 1 -- 172.21.15.123:0/2593143928 learned_addr learned my addr 172.21.15.123:0/2593143928 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:48:49.782 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:49.779+0000 7f7fb77fe640 1 --2- 172.21.15.123:0/2593143928 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f7fb815b9f0 0x7f7fb80cb380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:48:49.782 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:49.779+0000 7f7fb67fc640 1 --2- 172.21.15.123:0/2593143928 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7fb80c3230 0x7f7fb814c450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:48:49.782 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:49.779+0000 7f7fb6ffd640 1 -- 172.21.15.123:0/2593143928 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f7fb815b9f0 msgr2=0x7f7fb80cb380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:48:49.782 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:49.779+0000 7f7fb6ffd640 1 --2- 172.21.15.123:0/2593143928 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f7fb815b9f0 0x7f7fb80cb380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:49.782 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:49.779+0000 7f7fb6ffd640 1 -- 172.21.15.123:0/2593143928 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7fb80c3230 msgr2=0x7f7fb814c450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:48:49.782 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:49.779+0000 7f7fb6ffd640 1 --2- 172.21.15.123:0/2593143928 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7fb80c3230 0x7f7fb814c450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:49.782 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:49.779+0000 7f7fb6ffd640 1 -- 172.21.15.123:0/2593143928 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7fb81faf80 con 0x7f7fb80701e0 2024-09-16T15:48:49.782 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:49.779+0000 7f7fb77fe640 1 --2- 172.21.15.123:0/2593143928 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f7fb815b9f0 0x7f7fb80cb380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T15:48:49.783 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:49.779+0000 7f7fb67fc640 1 --2- 172.21.15.123:0/2593143928 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7fb80c3230 0x7f7fb814c450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T15:48:49.783 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:49.779+0000 7f7fb6ffd640 1 --2- 172.21.15.123:0/2593143928 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7fb80701e0 0x7f7fb814d170 secure :-1 s=READY pgs=1144 cs=0 l=1 rev1=1 crypto rx=0x7f7fb000ccb0 tx=0x7f7fb000a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:48:49.783 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:49.779+0000 7f7faf7fe640 1 -- 172.21.15.123:0/2593143928 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7fb000f450 con 0x7f7fb80701e0 2024-09-16T15:48:49.783 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:49.779+0000 7f7fbd764640 1 -- 172.21.15.123:0/2593143928 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7fb81fb260 con 0x7f7fb80701e0 2024-09-16T15:48:49.783 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:49.779+0000 7f7fbd764640 1 -- 172.21.15.123:0/2593143928 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7fb814cd90 con 0x7f7fb80701e0 2024-09-16T15:48:49.783 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:49.779+0000 7f7faf7fe640 1 -- 172.21.15.123:0/2593143928 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f7fb0004b30 con 0x7f7fb80701e0 2024-09-16T15:48:49.783 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:49.783+0000 7f7faf7fe640 1 -- 172.21.15.123:0/2593143928 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7fb000f5f0 con 0x7f7fb80701e0 2024-09-16T15:48:49.784 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:49.783+0000 7f7fbd764640 1 -- 172.21.15.123:0/2593143928 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7f9c0052a0 con 0x7f7fb80701e0 2024-09-16T15:48:49.784 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:49.783+0000 7f7faf7fe640 1 -- 172.21.15.123:0/2593143928 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 18) ==== 97386+0+0 (secure 0 0 0) 0x7f7fb0018400 con 0x7f7fb80701e0 2024-09-16T15:48:49.784 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:49.783+0000 7f7faf7fe640 1 --2- 172.21.15.123:0/2593143928 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f7f98073220 0x7f7f980756e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:48:49.784 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:49.783+0000 7f7faf7fe640 1 -- 172.21.15.123:0/2593143928 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(105..105 src has 1..105) ==== 5767+0+0 (secure 0 0 0) 0x7f7fb00951f0 con 0x7f7fb80701e0 2024-09-16T15:48:49.784 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:49.783+0000 7f7fb67fc640 1 --2- 172.21.15.123:0/2593143928 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f7f98073220 0x7f7f980756e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:48:49.784 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:49.783+0000 7f7fb67fc640 1 --2- 172.21.15.123:0/2593143928 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f7f98073220 0x7f7f980756e0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f7fa00af000 tx=0x7f7fa001b0d0 comp rx=0 tx=0).ready entity=mgr.4150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:48:49.787 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:49.787+0000 7f7faf7fe640 1 -- 172.21.15.123:0/2593143928 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f7fb0014030 con 0x7f7fb80701e0 2024-09-16T15:48:49.996 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:49.995+0000 7f7fbd764640 1 -- 172.21.15.123:0/2593143928 --> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot clone", "vol_name": "cephfs", "sub_name": "subvol37810", "snap_name": "fake", "target_sub_name": "subvol_clone62021", "target": ["mon-mgr", ""]}) -- 0x7f7f9c000c90 con 0x7f7f98073220 2024-09-16T15:48:50.144 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:50.143+0000 7f7faf7fe640 1 -- 172.21.15.123:0/2593143928 <== mgr.4150 v2:172.21.15.123:6836/2464759676 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f7f9c000c90 con 0x7f7f98073220 2024-09-16T15:48:50.148 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:50.147+0000 7f7fbd764640 1 -- 172.21.15.123:0/2593143928 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f7f98073220 msgr2=0x7f7f980756e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:48:50.148 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:50.147+0000 7f7fbd764640 1 --2- 172.21.15.123:0/2593143928 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f7f98073220 0x7f7f980756e0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f7fa00af000 tx=0x7f7fa001b0d0 comp rx=0 tx=0).stop 2024-09-16T15:48:50.148 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:50.147+0000 7f7fbd764640 1 -- 172.21.15.123:0/2593143928 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7fb80701e0 msgr2=0x7f7fb814d170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:48:50.148 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:50.147+0000 7f7fbd764640 1 --2- 172.21.15.123:0/2593143928 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7fb80701e0 0x7f7fb814d170 secure :-1 s=READY pgs=1144 cs=0 l=1 rev1=1 crypto rx=0x7f7fb000ccb0 tx=0x7f7fb000a620 comp rx=0 tx=0).stop 2024-09-16T15:48:50.149 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:50.147+0000 7f7fbd764640 1 -- 172.21.15.123:0/2593143928 shutdown_connections 2024-09-16T15:48:50.149 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:50.147+0000 7f7fbd764640 1 --2- 172.21.15.123:0/2593143928 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f7f98073220 0x7f7f980756e0 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:50.149 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:50.147+0000 7f7fbd764640 1 --2- 172.21.15.123:0/2593143928 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f7fb815b9f0 0x7f7fb80cb380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:50.149 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:50.147+0000 7f7fbd764640 1 --2- 172.21.15.123:0/2593143928 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7fb80c3230 0x7f7fb814c450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:50.149 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:50.147+0000 7f7fbd764640 1 --2- 172.21.15.123:0/2593143928 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7fb80701e0 0x7f7fb814d170 unknown :-1 s=CLOSED pgs=1144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:50.149 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:50.147+0000 7f7fbd764640 1 -- 172.21.15.123:0/2593143928 >> 172.21.15.123:0/2593143928 conn(0x7f7fb80b1710 msgr2=0x7f7fb80b1af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:48:50.149 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:50.147+0000 7f7fbd764640 1 -- 172.21.15.123:0/2593143928 shutdown_connections 2024-09-16T15:48:50.149 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:50.147+0000 7f7fbd764640 1 -- 172.21.15.123:0/2593143928 wait complete. 2024-09-16T15:48:50.171 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol_clone62021 2024-09-16T15:48:50.299 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:50.299+0000 7fcce60ec640 1 -- 172.21.15.123:0/1690617301 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcce00d0d90 msgr2=0x7fcce00b3eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:48:50.299 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:50.299+0000 7fcce60ec640 1 --2- 172.21.15.123:0/1690617301 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcce00d0d90 0x7fcce00b3eb0 secure :-1 s=READY pgs=1146 cs=0 l=1 rev1=1 crypto rx=0x7fcce00b69d0 tx=0x7fccd801caf0 comp rx=0 tx=0).stop 2024-09-16T15:48:50.299 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:50.299+0000 7fcce60ec640 1 -- 172.21.15.123:0/1690617301 shutdown_connections 2024-09-16T15:48:50.299 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:50.299+0000 7fcce60ec640 1 --2- 172.21.15.123:0/1690617301 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcce00d0d90 0x7fcce00b3eb0 unknown :-1 s=CLOSED pgs=1146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:50.299 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:50.299+0000 7fcce60ec640 1 --2- 172.21.15.123:0/1690617301 >> v2:172.21.15.184:3300/0 conn(0x7fcce01615c0 0x7fcce00d0850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:50.299 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:50.299+0000 7fcce60ec640 1 -- 172.21.15.123:0/1690617301 >> 172.21.15.123:0/1690617301 conn(0x7fcce00d3e40 msgr2=0x7fcce00d4220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:48:50.299 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:50.299+0000 7fcce60ec640 1 -- 172.21.15.123:0/1690617301 shutdown_connections 2024-09-16T15:48:50.299 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:50.299+0000 7fcce60ec640 1 -- 172.21.15.123:0/1690617301 wait complete. 2024-09-16T15:48:50.300 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:50.299+0000 7fcce60ec640 1 Processor -- start 2024-09-16T15:48:50.300 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:50.299+0000 7fcce60ec640 1 -- start start 2024-09-16T15:48:50.302 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:50.299+0000 7fcce60ec640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fcce00d0d90 0x7fcce0160560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:48:50.302 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:50.299+0000 7fcce60ec640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fcce00d2120 0x7fcce00b1e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:48:50.302 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:50.299+0000 7fcce60ec640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcce01615c0 0x7fcce01f7c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:48:50.302 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:50.299+0000 7fcce60ec640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fcce00b43f0 con 0x7fcce01615c0 2024-09-16T15:48:50.302 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:50.299+0000 7fcce60ec640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fcce00b3120 con 0x7fcce00d2120 2024-09-16T15:48:50.302 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:50.299+0000 7fcce60ec640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fcce00b4570 con 0x7fcce00d0d90 2024-09-16T15:48:50.302 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:50.299+0000 7fccdffff640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcce01615c0 0x7fcce01f7c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:48:50.302 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:50.299+0000 7fccdffff640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcce01615c0 0x7fcce01f7c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:46162/0 (socket says 172.21.15.123:46162) 2024-09-16T15:48:50.302 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:50.299+0000 7fccdffff640 1 -- 172.21.15.123:0/3597028992 learned_addr learned my addr 172.21.15.123:0/3597028992 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:48:50.302 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:50.299+0000 7fccdeffd640 1 --2- 172.21.15.123:0/3597028992 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fcce00d2120 0x7fcce00b1e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:48:50.302 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:50.299+0000 7fccdf7fe640 1 --2- 172.21.15.123:0/3597028992 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fcce00d0d90 0x7fcce0160560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:48:50.303 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:50.299+0000 7fccdffff640 1 -- 172.21.15.123:0/3597028992 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fcce00d0d90 msgr2=0x7fcce0160560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:48:50.303 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:50.299+0000 7fccdffff640 1 --2- 172.21.15.123:0/3597028992 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fcce00d0d90 0x7fcce0160560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:50.303 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:50.299+0000 7fccdffff640 1 -- 172.21.15.123:0/3597028992 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fcce00d2120 msgr2=0x7fcce00b1e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:48:50.303 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:50.299+0000 7fccdffff640 1 --2- 172.21.15.123:0/3597028992 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fcce00d2120 0x7fcce00b1e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:50.303 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:50.299+0000 7fccdffff640 1 -- 172.21.15.123:0/3597028992 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcce00b8d10 con 0x7fcce01615c0 2024-09-16T15:48:50.303 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:50.299+0000 7fccdeffd640 1 --2- 172.21.15.123:0/3597028992 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fcce00d2120 0x7fcce00b1e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T15:48:50.303 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:50.303+0000 7fccdffff640 1 --2- 172.21.15.123:0/3597028992 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcce01615c0 0x7fcce01f7c00 secure :-1 s=READY pgs=1147 cs=0 l=1 rev1=1 crypto rx=0x7fccd8009b10 tx=0x7fccd80a57e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:48:50.303 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:50.303+0000 7fccc7fff640 1 -- 172.21.15.123:0/3597028992 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fccd80bb070 con 0x7fcce01615c0 2024-09-16T15:48:50.303 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:50.303+0000 7fccc7fff640 1 -- 172.21.15.123:0/3597028992 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fccd8004d40 con 0x7fcce01615c0 2024-09-16T15:48:50.303 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:50.303+0000 7fcce60ec640 1 -- 172.21.15.123:0/3597028992 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcce01fb5d0 con 0x7fcce01615c0 2024-09-16T15:48:50.303 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:50.303+0000 7fccc7fff640 1 -- 172.21.15.123:0/3597028992 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fccd80b2040 con 0x7fcce01615c0 2024-09-16T15:48:50.303 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:50.303+0000 7fcce60ec640 1 -- 172.21.15.123:0/3597028992 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcce01fbb90 con 0x7fcce01615c0 2024-09-16T15:48:50.304 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:50.303+0000 7fcce60ec640 1 -- 172.21.15.123:0/3597028992 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcce00d0850 con 0x7fcce01615c0 2024-09-16T15:48:50.304 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:50.303+0000 7fccc7fff640 1 -- 172.21.15.123:0/3597028992 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 18) ==== 97386+0+0 (secure 0 0 0) 0x7fccd80be070 con 0x7fcce01615c0 2024-09-16T15:48:50.304 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:50.303+0000 7fccc7fff640 1 --2- 172.21.15.123:0/3597028992 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fcccc073220 0x7fcccc0756e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:48:50.304 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:50.303+0000 7fccdf7fe640 1 --2- 172.21.15.123:0/3597028992 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fcccc073220 0x7fcccc0756e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:48:50.304 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:50.303+0000 7fccc7fff640 1 -- 172.21.15.123:0/3597028992 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(105..105 src has 1..105) ==== 5767+0+0 (secure 0 0 0) 0x7fccd812e4c0 con 0x7fcce01615c0 2024-09-16T15:48:50.305 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:50.303+0000 7fccdf7fe640 1 --2- 172.21.15.123:0/3597028992 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fcccc073220 0x7fcccc0756e0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fccd00058d0 tx=0x7fccd0005860 comp rx=0 tx=0).ready entity=mgr.4150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:48:50.308 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:50.307+0000 7fccc7fff640 1 -- 172.21.15.123:0/3597028992 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7fcce00b8d10 con 0x7fcce01615c0 2024-09-16T15:48:50.520 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:50.519+0000 7fcce60ec640 1 -- 172.21.15.123:0/3597028992 --> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvol_clone62021", "target": ["mon-mgr", ""]}) -- 0x7fcce006d040 con 0x7fcccc073220 2024-09-16T15:48:51.831 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:51.831+0000 7fccc7fff640 1 -- 172.21.15.123:0/3597028992 <== mgr.4150 v2:172.21.15.123:6836/2464759676 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+153 (secure 0 0 0) 0x7fcce006d040 con 0x7fcccc073220 2024-09-16T15:48:51.831 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2024-09-16T15:48:51.831 INFO:teuthology.orchestra.run.smithi123.stdout: "status": { 2024-09-16T15:48:51.831 INFO:teuthology.orchestra.run.smithi123.stdout: "state": "pending", 2024-09-16T15:48:51.831 INFO:teuthology.orchestra.run.smithi123.stdout: "source": { 2024-09-16T15:48:51.831 INFO:teuthology.orchestra.run.smithi123.stdout: "volume": "cephfs", 2024-09-16T15:48:51.831 INFO:teuthology.orchestra.run.smithi123.stdout: "subvolume": "subvol37810", 2024-09-16T15:48:51.832 INFO:teuthology.orchestra.run.smithi123.stdout: "snapshot": "fake" 2024-09-16T15:48:51.832 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-16T15:48:51.832 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-16T15:48:51.832 INFO:teuthology.orchestra.run.smithi123.stdout:} 2024-09-16T15:48:51.834 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:51.835+0000 7fcce60ec640 1 -- 172.21.15.123:0/3597028992 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fcccc073220 msgr2=0x7fcccc0756e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:48:51.834 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:51.835+0000 7fcce60ec640 1 --2- 172.21.15.123:0/3597028992 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fcccc073220 0x7fcccc0756e0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fccd00058d0 tx=0x7fccd0005860 comp rx=0 tx=0).stop 2024-09-16T15:48:51.834 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:51.835+0000 7fcce60ec640 1 -- 172.21.15.123:0/3597028992 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcce01615c0 msgr2=0x7fcce01f7c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:48:51.834 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:51.835+0000 7fcce60ec640 1 --2- 172.21.15.123:0/3597028992 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcce01615c0 0x7fcce01f7c00 secure :-1 s=READY pgs=1147 cs=0 l=1 rev1=1 crypto rx=0x7fccd8009b10 tx=0x7fccd80a57e0 comp rx=0 tx=0).stop 2024-09-16T15:48:51.834 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:51.835+0000 7fcce60ec640 1 -- 172.21.15.123:0/3597028992 shutdown_connections 2024-09-16T15:48:51.834 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:51.835+0000 7fcce60ec640 1 --2- 172.21.15.123:0/3597028992 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fcccc073220 0x7fcccc0756e0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:51.834 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:51.835+0000 7fcce60ec640 1 --2- 172.21.15.123:0/3597028992 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcce01615c0 0x7fcce01f7c00 unknown :-1 s=CLOSED pgs=1147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:51.834 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:51.835+0000 7fcce60ec640 1 --2- 172.21.15.123:0/3597028992 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fcce00d2120 0x7fcce00b1e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:51.835 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:51.835+0000 7fcce60ec640 1 --2- 172.21.15.123:0/3597028992 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fcce00d0d90 0x7fcce0160560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:51.835 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:51.835+0000 7fcce60ec640 1 -- 172.21.15.123:0/3597028992 >> 172.21.15.123:0/3597028992 conn(0x7fcce00d3e40 msgr2=0x7fcce01599f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:48:51.835 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:51.835+0000 7fcce60ec640 1 -- 172.21.15.123:0/3597028992 shutdown_connections 2024-09-16T15:48:51.835 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:51.835+0000 7fcce60ec640 1 -- 172.21.15.123:0/3597028992 wait complete. 2024-09-16T15:48:51.857 DEBUG:tasks.cephfs.test_volumes:current clone state = pending 2024-09-16T15:48:52.858 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol_clone62021 2024-09-16T15:48:53.000 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:52.999+0000 7f2ef57c0640 1 -- 172.21.15.123:0/2966871081 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2ef00cf430 msgr2=0x7f2ef00cf8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:48:53.000 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:52.999+0000 7f2ef57c0640 1 --2- 172.21.15.123:0/2966871081 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2ef00cf430 0x7f2ef00cf8a0 secure :-1 s=READY pgs=1148 cs=0 l=1 rev1=1 crypto rx=0x7f2ef0072830 tx=0x7f2ee801cd80 comp rx=0 tx=0).stop 2024-09-16T15:48:53.000 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:52.999+0000 7f2ef57c0640 1 -- 172.21.15.123:0/2966871081 shutdown_connections 2024-09-16T15:48:53.000 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:52.999+0000 7f2ef57c0640 1 --2- 172.21.15.123:0/2966871081 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2ef00cf430 0x7f2ef00cf8a0 unknown :-1 s=CLOSED pgs=1148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:53.000 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:52.999+0000 7f2ef57c0640 1 --2- 172.21.15.123:0/2966871081 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f2ef00af170 0x7f2ef00cb9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:53.000 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:52.999+0000 7f2ef57c0640 1 --2- 172.21.15.123:0/2966871081 >> v2:172.21.15.184:3300/0 conn(0x7f2ef0161510 0x7f2ef0162570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:53.000 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:52.999+0000 7f2ef57c0640 1 -- 172.21.15.123:0/2966871081 >> 172.21.15.123:0/2966871081 conn(0x7f2ef0074210 msgr2=0x7f2ef0074620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:48:53.000 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:52.999+0000 7f2ef57c0640 1 -- 172.21.15.123:0/2966871081 shutdown_connections 2024-09-16T15:48:53.001 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:52.999+0000 7f2ef57c0640 1 -- 172.21.15.123:0/2966871081 wait complete. 2024-09-16T15:48:53.001 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:52.999+0000 7f2ef57c0640 1 Processor -- start 2024-09-16T15:48:53.001 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:52.999+0000 7f2ef57c0640 1 -- start start 2024-09-16T15:48:53.003 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:53.003+0000 7f2ef57c0640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2ef00af170 0x7f2ef0099980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:48:53.003 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:53.003+0000 7f2ef57c0640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f2ef00cf430 0x7f2ef0156a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:48:53.003 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:53.003+0000 7f2ef57c0640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2ef0161510 0x7f2ef01600f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:48:53.003 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:53.003+0000 7f2ef57c0640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f2ef00d02d0 con 0x7f2ef0161510 2024-09-16T15:48:53.003 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:53.003+0000 7f2ef57c0640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f2ef00ccf10 con 0x7f2ef00af170 2024-09-16T15:48:53.004 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:53.003+0000 7f2ef57c0640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f2ef00d0450 con 0x7f2ef00cf430 2024-09-16T15:48:53.004 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:53.003+0000 7f2eef7fe640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2ef0161510 0x7f2ef01600f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:48:53.004 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:53.003+0000 7f2eef7fe640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2ef0161510 0x7f2ef01600f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:43774/0 (socket says 172.21.15.123:43774) 2024-09-16T15:48:53.004 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:53.003+0000 7f2eef7fe640 1 -- 172.21.15.123:0/1152124103 learned_addr learned my addr 172.21.15.123:0/1152124103 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:48:53.004 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:53.003+0000 7f2eeeffd640 1 --2- 172.21.15.123:0/1152124103 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2ef00af170 0x7f2ef0099980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:48:53.004 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:53.003+0000 7f2eee7fc640 1 --2- 172.21.15.123:0/1152124103 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f2ef00cf430 0x7f2ef0156a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:48:53.004 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:53.003+0000 7f2eef7fe640 1 -- 172.21.15.123:0/1152124103 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f2ef00cf430 msgr2=0x7f2ef0156a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:48:53.004 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:53.003+0000 7f2eef7fe640 1 --2- 172.21.15.123:0/1152124103 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f2ef00cf430 0x7f2ef0156a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:53.004 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:53.003+0000 7f2eef7fe640 1 -- 172.21.15.123:0/1152124103 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2ef00af170 msgr2=0x7f2ef0099980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:48:53.004 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:53.003+0000 7f2eef7fe640 1 --2- 172.21.15.123:0/1152124103 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2ef00af170 0x7f2ef0099980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:53.004 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:53.003+0000 7f2eef7fe640 1 -- 172.21.15.123:0/1152124103 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2ef0099ec0 con 0x7f2ef0161510 2024-09-16T15:48:53.004 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:53.003+0000 7f2eef7fe640 1 --2- 172.21.15.123:0/1152124103 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2ef0161510 0x7f2ef01600f0 secure :-1 s=READY pgs=1149 cs=0 l=1 rev1=1 crypto rx=0x7f2ee8009bd0 tx=0x7f2ee80a5b70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:48:53.005 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:53.003+0000 7f2ed77fe640 1 -- 172.21.15.123:0/1152124103 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ee80bb070 con 0x7f2ef0161510 2024-09-16T15:48:53.005 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:53.003+0000 7f2ed77fe640 1 -- 172.21.15.123:0/1152124103 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f2ee8004d40 con 0x7f2ef0161510 2024-09-16T15:48:53.005 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:53.003+0000 7f2ef57c0640 1 -- 172.21.15.123:0/1152124103 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2ef0151bc0 con 0x7f2ef0161510 2024-09-16T15:48:53.005 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:53.003+0000 7f2ed77fe640 1 -- 172.21.15.123:0/1152124103 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ee80b2040 con 0x7f2ef0161510 2024-09-16T15:48:53.005 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:53.003+0000 7f2ef57c0640 1 -- 172.21.15.123:0/1152124103 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2ef0152180 con 0x7f2ef0161510 2024-09-16T15:48:53.005 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:53.003+0000 7f2ef57c0640 1 -- 172.21.15.123:0/1152124103 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2ef00cf8a0 con 0x7f2ef0161510 2024-09-16T15:48:53.006 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:53.003+0000 7f2ed77fe640 1 -- 172.21.15.123:0/1152124103 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 18) ==== 97386+0+0 (secure 0 0 0) 0x7f2ee80be070 con 0x7f2ef0161510 2024-09-16T15:48:53.006 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:53.003+0000 7f2ed77fe640 1 --2- 172.21.15.123:0/1152124103 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f2edc073250 0x7f2edc075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:48:53.006 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:53.003+0000 7f2eeeffd640 1 --2- 172.21.15.123:0/1152124103 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f2edc073250 0x7f2edc075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:48:53.006 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:53.003+0000 7f2ed77fe640 1 -- 172.21.15.123:0/1152124103 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(105..105 src has 1..105) ==== 5767+0+0 (secure 0 0 0) 0x7f2ee812e350 con 0x7f2ef0161510 2024-09-16T15:48:53.006 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:53.007+0000 7f2eeeffd640 1 --2- 172.21.15.123:0/1152124103 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f2edc073250 0x7f2edc075710 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f2ee0006fd0 tx=0x7f2ee0008040 comp rx=0 tx=0).ready entity=mgr.4150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:48:53.009 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:53.011+0000 7f2ed77fe640 1 -- 172.21.15.123:0/1152124103 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f2ee8016970 con 0x7f2ef0161510 2024-09-16T15:48:53.217 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:53.219+0000 7f2ef57c0640 1 -- 172.21.15.123:0/1152124103 --> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvol_clone62021", "target": ["mon-mgr", ""]}) -- 0x7f2ef00ce490 con 0x7f2edc073250 2024-09-16T15:48:56.809 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:56.811+0000 7f2ed77fe640 1 -- 172.21.15.123:0/1152124103 <== mgr.4150 v2:172.21.15.123:6836/2464759676 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+46 (secure 0 0 0) 0x7f2ef00ce490 con 0x7f2edc073250 2024-09-16T15:48:56.810 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2024-09-16T15:48:56.810 INFO:teuthology.orchestra.run.smithi123.stdout: "status": { 2024-09-16T15:48:56.810 INFO:teuthology.orchestra.run.smithi123.stdout: "state": "complete" 2024-09-16T15:48:56.810 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-16T15:48:56.810 INFO:teuthology.orchestra.run.smithi123.stdout:} 2024-09-16T15:48:56.814 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:56.815+0000 7f2ef57c0640 1 -- 172.21.15.123:0/1152124103 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f2edc073250 msgr2=0x7f2edc075710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:48:56.815 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:56.815+0000 7f2ef57c0640 1 --2- 172.21.15.123:0/1152124103 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f2edc073250 0x7f2edc075710 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f2ee0006fd0 tx=0x7f2ee0008040 comp rx=0 tx=0).stop 2024-09-16T15:48:56.815 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:56.815+0000 7f2ef57c0640 1 -- 172.21.15.123:0/1152124103 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2ef0161510 msgr2=0x7f2ef01600f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:48:56.815 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:56.815+0000 7f2ef57c0640 1 --2- 172.21.15.123:0/1152124103 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2ef0161510 0x7f2ef01600f0 secure :-1 s=READY pgs=1149 cs=0 l=1 rev1=1 crypto rx=0x7f2ee8009bd0 tx=0x7f2ee80a5b70 comp rx=0 tx=0).stop 2024-09-16T15:48:56.815 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:56.815+0000 7f2ef57c0640 1 -- 172.21.15.123:0/1152124103 shutdown_connections 2024-09-16T15:48:56.815 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:56.815+0000 7f2ef57c0640 1 --2- 172.21.15.123:0/1152124103 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f2edc073250 0x7f2edc075710 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:56.815 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:56.815+0000 7f2ef57c0640 1 --2- 172.21.15.123:0/1152124103 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2ef0161510 0x7f2ef01600f0 unknown :-1 s=CLOSED pgs=1149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:56.815 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:56.815+0000 7f2ef57c0640 1 --2- 172.21.15.123:0/1152124103 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f2ef00cf430 0x7f2ef0156a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:56.815 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:56.815+0000 7f2ef57c0640 1 --2- 172.21.15.123:0/1152124103 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2ef00af170 0x7f2ef0099980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:56.815 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:56.815+0000 7f2ef57c0640 1 -- 172.21.15.123:0/1152124103 >> 172.21.15.123:0/1152124103 conn(0x7f2ef0074210 msgr2=0x7f2ef01575a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:48:56.815 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:56.815+0000 7f2ef57c0640 1 -- 172.21.15.123:0/1152124103 shutdown_connections 2024-09-16T15:48:56.815 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:56.815+0000 7f2ef57c0640 1 -- 172.21.15.123:0/1152124103 wait complete. 2024-09-16T15:48:56.837 DEBUG:tasks.cephfs.test_volumes:current clone state = complete 2024-09-16T15:48:56.837 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:48:56.837 DEBUG:teuthology.orchestra.run.smithi123:> (cd /home/ubuntu/cephtest/mnt.0 && exec sudo cat ./volumes/_nogroup/subvol_clone62021/.meta) 2024-09-16T15:48:56.856 INFO:teuthology.orchestra.run.smithi123.stdout:[GLOBAL] 2024-09-16T15:48:56.856 INFO:teuthology.orchestra.run.smithi123.stdout:version = 2 2024-09-16T15:48:56.856 INFO:teuthology.orchestra.run.smithi123.stdout:type = clone 2024-09-16T15:48:56.856 INFO:teuthology.orchestra.run.smithi123.stdout:path = /volumes/_nogroup/subvol_clone62021/e37efe41-7fe9-41ad-9d40-b93487d2deff 2024-09-16T15:48:56.856 INFO:teuthology.orchestra.run.smithi123.stdout:state = complete 2024-09-16T15:48:56.856 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-09-16T15:48:56.857 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot info cephfs subvol37810 subvol_snap6960 2024-09-16T15:48:57.039 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:57.039+0000 7fd792673640 1 -- 172.21.15.123:0/2215934028 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd78c160310 msgr2=0x7fd78c0725e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:48:57.039 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:57.039+0000 7fd792673640 1 --2- 172.21.15.123:0/2215934028 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd78c160310 0x7fd78c0725e0 secure :-1 s=READY pgs=1150 cs=0 l=1 rev1=1 crypto rx=0x7fd78c0cd410 tx=0x7fd78001c830 comp rx=0 tx=0).stop 2024-09-16T15:48:57.039 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:57.039+0000 7fd792673640 1 -- 172.21.15.123:0/2215934028 shutdown_connections 2024-09-16T15:48:57.039 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:57.039+0000 7fd792673640 1 --2- 172.21.15.123:0/2215934028 >> v2:172.21.15.184:3300/0 conn(0x7fd78c15f690 0x7fd78c15fb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:57.039 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:57.039+0000 7fd792673640 1 --2- 172.21.15.123:0/2215934028 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd78c160310 0x7fd78c0725e0 unknown :-1 s=CLOSED pgs=1150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:57.040 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:57.039+0000 7fd792673640 1 --2- 172.21.15.123:0/2215934028 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fd78c15bc40 0x7fd78c070470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:57.040 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:57.039+0000 7fd792673640 1 -- 172.21.15.123:0/2215934028 >> 172.21.15.123:0/2215934028 conn(0x7fd78c0b1630 msgr2=0x7fd78c0b1a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:48:57.040 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:57.039+0000 7fd792673640 1 -- 172.21.15.123:0/2215934028 shutdown_connections 2024-09-16T15:48:57.040 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:57.039+0000 7fd792673640 1 -- 172.21.15.123:0/2215934028 wait complete. 2024-09-16T15:48:57.040 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:57.039+0000 7fd792673640 1 Processor -- start 2024-09-16T15:48:57.040 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:57.039+0000 7fd792673640 1 -- start start 2024-09-16T15:48:57.042 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:57.043+0000 7fd792673640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd78c15bc40 0x7fd78c15e780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:48:57.042 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:57.043+0000 7fd792673640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fd78c15f690 0x7fd78c1f7040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:48:57.042 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:57.043+0000 7fd792673640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd78c160310 0x7fd78c151c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:48:57.042 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:57.043+0000 7fd792673640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fd78c0cb9a0 con 0x7fd78c15bc40 2024-09-16T15:48:57.042 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:57.043+0000 7fd792673640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fd78c0cf420 con 0x7fd78c160310 2024-09-16T15:48:57.043 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:57.043+0000 7fd792673640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fd78c0cbb20 con 0x7fd78c15f690 2024-09-16T15:48:57.043 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:57.043+0000 7fd78bfff640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd78c15bc40 0x7fd78c15e780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:48:57.043 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:57.043+0000 7fd78bfff640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd78c15bc40 0x7fd78c15e780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:43782/0 (socket says 172.21.15.123:43782) 2024-09-16T15:48:57.043 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:57.043+0000 7fd78bfff640 1 -- 172.21.15.123:0/3556760875 learned_addr learned my addr 172.21.15.123:0/3556760875 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:48:57.043 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:57.043+0000 7fd78b7fe640 1 --2- 172.21.15.123:0/3556760875 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fd78c15f690 0x7fd78c1f7040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:48:57.043 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:57.043+0000 7fd790c10640 1 --2- 172.21.15.123:0/3556760875 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd78c160310 0x7fd78c151c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:48:57.043 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:57.043+0000 7fd78bfff640 1 -- 172.21.15.123:0/3556760875 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fd78c15f690 msgr2=0x7fd78c1f7040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:48:57.043 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:57.043+0000 7fd78bfff640 1 --2- 172.21.15.123:0/3556760875 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fd78c15f690 0x7fd78c1f7040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:57.043 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:57.043+0000 7fd78bfff640 1 -- 172.21.15.123:0/3556760875 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd78c160310 msgr2=0x7fd78c151c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:48:57.043 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:57.043+0000 7fd78bfff640 1 --2- 172.21.15.123:0/3556760875 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd78c160310 0x7fd78c151c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:57.043 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:57.043+0000 7fd78bfff640 1 -- 172.21.15.123:0/3556760875 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd78c1521b0 con 0x7fd78c15bc40 2024-09-16T15:48:57.044 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:57.043+0000 7fd790c10640 1 --2- 172.21.15.123:0/3556760875 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd78c160310 0x7fd78c151c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T15:48:57.044 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:57.043+0000 7fd78bfff640 1 --2- 172.21.15.123:0/3556760875 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd78c15bc40 0x7fd78c15e780 secure :-1 s=READY pgs=1151 cs=0 l=1 rev1=1 crypto rx=0x7fd77c005950 tx=0x7fd77c00cdd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:48:57.044 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:57.043+0000 7fd788ff9640 1 -- 172.21.15.123:0/3556760875 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd77c004220 con 0x7fd78c15bc40 2024-09-16T15:48:57.044 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:57.043+0000 7fd788ff9640 1 -- 172.21.15.123:0/3556760875 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fd77c007950 con 0x7fd78c15bc40 2024-09-16T15:48:57.044 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:57.043+0000 7fd792673640 1 -- 172.21.15.123:0/3556760875 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd78c1fabc0 con 0x7fd78c15bc40 2024-09-16T15:48:57.044 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:57.043+0000 7fd788ff9640 1 -- 172.21.15.123:0/3556760875 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd77c007da0 con 0x7fd78c15bc40 2024-09-16T15:48:57.044 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:57.043+0000 7fd792673640 1 -- 172.21.15.123:0/3556760875 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd78c1fb180 con 0x7fd78c15bc40 2024-09-16T15:48:57.045 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:57.043+0000 7fd792673640 1 -- 172.21.15.123:0/3556760875 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd78c1f7580 con 0x7fd78c15bc40 2024-09-16T15:48:57.045 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:57.043+0000 7fd788ff9640 1 -- 172.21.15.123:0/3556760875 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 18) ==== 97386+0+0 (secure 0 0 0) 0x7fd77c003600 con 0x7fd78c15bc40 2024-09-16T15:48:57.045 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:57.043+0000 7fd788ff9640 1 --2- 172.21.15.123:0/3556760875 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fd778073220 0x7fd7780756e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:48:57.045 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:57.043+0000 7fd78b7fe640 1 --2- 172.21.15.123:0/3556760875 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fd778073220 0x7fd7780756e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:48:57.045 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:57.043+0000 7fd788ff9640 1 -- 172.21.15.123:0/3556760875 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(105..105 src has 1..105) ==== 5767+0+0 (secure 0 0 0) 0x7fd77c0949c0 con 0x7fd78c15bc40 2024-09-16T15:48:57.045 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:57.043+0000 7fd78b7fe640 1 --2- 172.21.15.123:0/3556760875 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fd778073220 0x7fd7780756e0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fd7800af000 tx=0x7fd78001b070 comp rx=0 tx=0).ready entity=mgr.4150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:48:57.049 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:57.047+0000 7fd788ff9640 1 -- 172.21.15.123:0/3556760875 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7fd77c014030 con 0x7fd78c15bc40 2024-09-16T15:48:57.252 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:57.251+0000 7fd792673640 1 -- 172.21.15.123:0/3556760875 --> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot info", "vol_name": "cephfs", "sub_name": "subvol37810", "snap_name": "subvol_snap6960", "target": ["mon-mgr", ""]}) -- 0x7fd78c0766e0 con 0x7fd778073220 2024-09-16T15:48:57.283 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:57.283+0000 7fd788ff9640 1 -- 172.21.15.123:0/3556760875 <== mgr.4150 v2:172.21.15.123:6836/2464759676 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+115 (secure 0 0 0) 0x7fd78c0766e0 con 0x7fd778073220 2024-09-16T15:48:57.284 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2024-09-16T15:48:57.284 INFO:teuthology.orchestra.run.smithi123.stdout: "created_at": "2024-09-16 15:48:42.785188", 2024-09-16T15:48:57.284 INFO:teuthology.orchestra.run.smithi123.stdout: "data_pool": "cephfs_data", 2024-09-16T15:48:57.284 INFO:teuthology.orchestra.run.smithi123.stdout: "has_pending_clones": "no" 2024-09-16T15:48:57.284 INFO:teuthology.orchestra.run.smithi123.stdout:} 2024-09-16T15:48:57.286 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:57.287+0000 7fd792673640 1 -- 172.21.15.123:0/3556760875 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fd778073220 msgr2=0x7fd7780756e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:48:57.286 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:57.287+0000 7fd792673640 1 --2- 172.21.15.123:0/3556760875 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fd778073220 0x7fd7780756e0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fd7800af000 tx=0x7fd78001b070 comp rx=0 tx=0).stop 2024-09-16T15:48:57.286 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:57.287+0000 7fd792673640 1 -- 172.21.15.123:0/3556760875 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd78c15bc40 msgr2=0x7fd78c15e780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:48:57.286 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:57.287+0000 7fd792673640 1 --2- 172.21.15.123:0/3556760875 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd78c15bc40 0x7fd78c15e780 secure :-1 s=READY pgs=1151 cs=0 l=1 rev1=1 crypto rx=0x7fd77c005950 tx=0x7fd77c00cdd0 comp rx=0 tx=0).stop 2024-09-16T15:48:57.287 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:57.287+0000 7fd792673640 1 -- 172.21.15.123:0/3556760875 shutdown_connections 2024-09-16T15:48:57.287 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:57.287+0000 7fd792673640 1 --2- 172.21.15.123:0/3556760875 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fd778073220 0x7fd7780756e0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:57.287 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:57.287+0000 7fd792673640 1 --2- 172.21.15.123:0/3556760875 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd78c160310 0x7fd78c151c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:57.287 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:57.287+0000 7fd792673640 1 --2- 172.21.15.123:0/3556760875 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fd78c15f690 0x7fd78c1f7040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:57.287 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:57.287+0000 7fd792673640 1 --2- 172.21.15.123:0/3556760875 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd78c15bc40 0x7fd78c15e780 unknown :-1 s=CLOSED pgs=1151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:57.287 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:57.287+0000 7fd792673640 1 -- 172.21.15.123:0/3556760875 >> 172.21.15.123:0/3556760875 conn(0x7fd78c0b1630 msgr2=0x7fd78c0cc180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:48:57.287 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:57.287+0000 7fd792673640 1 -- 172.21.15.123:0/3556760875 shutdown_connections 2024-09-16T15:48:57.287 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:57.287+0000 7fd792673640 1 -- 172.21.15.123:0/3556760875 wait complete. 2024-09-16T15:48:57.304 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot ls cephfs subvol37810 2024-09-16T15:48:57.431 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:57.431+0000 7f908dd67640 1 -- 172.21.15.123:0/2402029043 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9088160310 msgr2=0x7f9088070470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:48:57.432 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:57.431+0000 7f908dd67640 1 --2- 172.21.15.123:0/2402029043 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9088160310 0x7f9088070470 secure :-1 s=READY pgs=1153 cs=0 l=1 rev1=1 crypto rx=0x7f9070009840 tx=0x7f907001c760 comp rx=0 tx=0).stop 2024-09-16T15:48:57.432 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:57.431+0000 7f908dd67640 1 -- 172.21.15.123:0/2402029043 shutdown_connections 2024-09-16T15:48:57.432 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:57.431+0000 7f908dd67640 1 --2- 172.21.15.123:0/2402029043 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f908815f690 0x7f908815faf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:57.432 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:57.431+0000 7f908dd67640 1 --2- 172.21.15.123:0/2402029043 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9088160310 0x7f9088070470 unknown :-1 s=CLOSED pgs=1153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:57.432 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:57.431+0000 7f908dd67640 1 -- 172.21.15.123:0/2402029043 >> 172.21.15.123:0/2402029043 conn(0x7f90880b1630 msgr2=0x7f90880b1a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:48:57.432 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:57.431+0000 7f908dd67640 1 -- 172.21.15.123:0/2402029043 shutdown_connections 2024-09-16T15:48:57.432 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:57.431+0000 7f908dd67640 1 -- 172.21.15.123:0/2402029043 wait complete. 2024-09-16T15:48:57.433 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:57.431+0000 7f908dd67640 1 Processor -- start 2024-09-16T15:48:57.433 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:57.431+0000 7f908dd67640 1 -- start start 2024-09-16T15:48:57.434 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:57.435+0000 7f908dd67640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f908815bc40 0x7f908815c2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:48:57.434 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:57.435+0000 7f908dd67640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f908815f690 0x7f908815c810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:48:57.434 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:57.435+0000 7f908dd67640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f9088160310 0x7f90881fa9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:48:57.435 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:57.435+0000 7f908dd67640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f9088168bb0 con 0x7f908815bc40 2024-09-16T15:48:57.435 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:57.435+0000 7f908dd67640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f9088168a30 con 0x7f9088160310 2024-09-16T15:48:57.435 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:57.435+0000 7f908dd67640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f9088168d30 con 0x7f908815f690 2024-09-16T15:48:57.435 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:57.435+0000 7f90877fe640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f908815bc40 0x7f908815c2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:48:57.435 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:57.435+0000 7f90877fe640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f908815bc40 0x7f908815c2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:43806/0 (socket says 172.21.15.123:43806) 2024-09-16T15:48:57.435 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:57.435+0000 7f90877fe640 1 -- 172.21.15.123:0/2008801378 learned_addr learned my addr 172.21.15.123:0/2008801378 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:48:57.435 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:57.435+0000 7f90877fe640 1 -- 172.21.15.123:0/2008801378 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f908815f690 msgr2=0x7f908815c810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:48:57.435 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:57.435+0000 7f90877fe640 1 --2- 172.21.15.123:0/2008801378 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f908815f690 0x7f908815c810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:57.435 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:57.435+0000 7f90877fe640 1 -- 172.21.15.123:0/2008801378 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f9088160310 msgr2=0x7f90881fa9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:48:57.435 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:57.435+0000 7f90877fe640 1 --2- 172.21.15.123:0/2008801378 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f9088160310 0x7f90881fa9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:57.435 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:57.435+0000 7f90877fe640 1 -- 172.21.15.123:0/2008801378 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f908815e300 con 0x7f908815bc40 2024-09-16T15:48:57.435 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:57.435+0000 7f90877fe640 1 --2- 172.21.15.123:0/2008801378 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f908815bc40 0x7f908815c2d0 secure :-1 s=READY pgs=1154 cs=0 l=1 rev1=1 crypto rx=0x7f907800c500 tx=0x7f907800c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:48:57.435 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:57.435+0000 7f9084ff9640 1 -- 172.21.15.123:0/2008801378 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9078017070 con 0x7f908815bc40 2024-09-16T15:48:57.436 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:57.435+0000 7f9084ff9640 1 -- 172.21.15.123:0/2008801378 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f9078003600 con 0x7f908815bc40 2024-09-16T15:48:57.436 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:57.435+0000 7f908dd67640 1 -- 172.21.15.123:0/2008801378 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f908815e5e0 con 0x7f908815bc40 2024-09-16T15:48:57.436 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:57.435+0000 7f9084ff9640 1 -- 172.21.15.123:0/2008801378 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9078003a90 con 0x7f908815bc40 2024-09-16T15:48:57.436 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:57.435+0000 7f908dd67640 1 -- 172.21.15.123:0/2008801378 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f908815ee80 con 0x7f908815bc40 2024-09-16T15:48:57.437 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:57.435+0000 7f908dd67640 1 -- 172.21.15.123:0/2008801378 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f90880766e0 con 0x7f908815bc40 2024-09-16T15:48:57.437 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:57.435+0000 7f9084ff9640 1 -- 172.21.15.123:0/2008801378 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 18) ==== 97386+0+0 (secure 0 0 0) 0x7f907801a070 con 0x7f908815bc40 2024-09-16T15:48:57.437 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:57.435+0000 7f9084ff9640 1 --2- 172.21.15.123:0/2008801378 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f9068073220 0x7f90680756e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:48:57.437 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:57.435+0000 7f907effd640 1 --2- 172.21.15.123:0/2008801378 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f9068073220 0x7f90680756e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:48:57.437 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:57.435+0000 7f9084ff9640 1 -- 172.21.15.123:0/2008801378 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(105..105 src has 1..105) ==== 5767+0+0 (secure 0 0 0) 0x7f90780938e0 con 0x7f908815bc40 2024-09-16T15:48:57.437 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:57.435+0000 7f907effd640 1 --2- 172.21.15.123:0/2008801378 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f9068073220 0x7f90680756e0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f90700af000 tx=0x7f9070019500 comp rx=0 tx=0).ready entity=mgr.4150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:48:57.440 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:57.439+0000 7f9084ff9640 1 -- 172.21.15.123:0/2008801378 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f907805e720 con 0x7f908815bc40 2024-09-16T15:48:57.645 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:57.643+0000 7f908dd67640 1 -- 172.21.15.123:0/2008801378 --> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot ls", "vol_name": "cephfs", "sub_name": "subvol37810", "target": ["mon-mgr", ""]}) -- 0x7f9088074220 con 0x7f9068073220 2024-09-16T15:48:57.667 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:57.667+0000 7f9084ff9640 1 -- 172.21.15.123:0/2008801378 <== mgr.4150 v2:172.21.15.123:6836/2464759676 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+86 (secure 0 0 0) 0x7f9088074220 con 0x7f9068073220 2024-09-16T15:48:57.667 INFO:teuthology.orchestra.run.smithi123.stdout:[ 2024-09-16T15:48:57.667 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-09-16T15:48:57.667 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "fake" 2024-09-16T15:48:57.667 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:48:57.667 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-09-16T15:48:57.667 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "subvol_snap6960" 2024-09-16T15:48:57.667 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-16T15:48:57.667 INFO:teuthology.orchestra.run.smithi123.stdout:] 2024-09-16T15:48:57.669 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:57.671+0000 7f908dd67640 1 -- 172.21.15.123:0/2008801378 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f9068073220 msgr2=0x7f90680756e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:48:57.670 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:57.671+0000 7f908dd67640 1 --2- 172.21.15.123:0/2008801378 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f9068073220 0x7f90680756e0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f90700af000 tx=0x7f9070019500 comp rx=0 tx=0).stop 2024-09-16T15:48:57.670 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:57.671+0000 7f908dd67640 1 -- 172.21.15.123:0/2008801378 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f908815bc40 msgr2=0x7f908815c2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:48:57.670 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:57.671+0000 7f908dd67640 1 --2- 172.21.15.123:0/2008801378 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f908815bc40 0x7f908815c2d0 secure :-1 s=READY pgs=1154 cs=0 l=1 rev1=1 crypto rx=0x7f907800c500 tx=0x7f907800c9c0 comp rx=0 tx=0).stop 2024-09-16T15:48:57.670 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:57.671+0000 7f908dd67640 1 -- 172.21.15.123:0/2008801378 shutdown_connections 2024-09-16T15:48:57.670 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:57.671+0000 7f908dd67640 1 --2- 172.21.15.123:0/2008801378 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f9068073220 0x7f90680756e0 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:57.670 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:57.671+0000 7f908dd67640 1 --2- 172.21.15.123:0/2008801378 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f9088160310 0x7f90881fa9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:57.670 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:57.671+0000 7f908dd67640 1 --2- 172.21.15.123:0/2008801378 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f908815f690 0x7f908815c810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:57.670 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:57.671+0000 7f908dd67640 1 --2- 172.21.15.123:0/2008801378 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f908815bc40 0x7f908815c2d0 unknown :-1 s=CLOSED pgs=1154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:57.670 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:57.671+0000 7f908dd67640 1 -- 172.21.15.123:0/2008801378 >> 172.21.15.123:0/2008801378 conn(0x7f90880b1630 msgr2=0x7f9088157710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:48:57.670 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:57.671+0000 7f908dd67640 1 -- 172.21.15.123:0/2008801378 shutdown_connections 2024-09-16T15:48:57.670 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:57.671+0000 7f908dd67640 1 -- 172.21.15.123:0/2008801378 wait complete. 2024-09-16T15:48:57.685 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot rm cephfs subvol37810 subvol_snap6960 2024-09-16T15:48:57.804 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:57.803+0000 7f7a87273640 1 -- 172.21.15.123:0/3534986515 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7a8015b540 msgr2=0x7f7a8006ff30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:48:57.805 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:57.803+0000 7f7a87273640 1 --2- 172.21.15.123:0/3534986515 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7a8015b540 0x7f7a8006ff30 secure :-1 s=READY pgs=1155 cs=0 l=1 rev1=1 crypto rx=0x7f7a800b2600 tx=0x7f7a7001c570 comp rx=0 tx=0).stop 2024-09-16T15:48:57.805 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:57.803+0000 7f7a87273640 1 -- 172.21.15.123:0/3534986515 shutdown_connections 2024-09-16T15:48:57.805 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:57.803+0000 7f7a87273640 1 --2- 172.21.15.123:0/3534986515 >> v2:172.21.15.184:3300/0 conn(0x7f7a80070470 0x7f7a8015e640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:57.805 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:57.803+0000 7f7a87273640 1 --2- 172.21.15.123:0/3534986515 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7a8015b540 0x7f7a8006ff30 unknown :-1 s=CLOSED pgs=1155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:57.805 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:57.803+0000 7f7a87273640 1 --2- 172.21.15.123:0/3534986515 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f7a80073c20 0x7f7a8015b000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:57.805 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:57.803+0000 7f7a87273640 1 -- 172.21.15.123:0/3534986515 >> 172.21.15.123:0/3534986515 conn(0x7f7a800b1630 msgr2=0x7f7a800b1a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:48:57.805 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:57.803+0000 7f7a87273640 1 -- 172.21.15.123:0/3534986515 shutdown_connections 2024-09-16T15:48:57.805 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:57.803+0000 7f7a87273640 1 -- 172.21.15.123:0/3534986515 wait complete. 2024-09-16T15:48:57.806 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:57.807+0000 7f7a87273640 1 Processor -- start 2024-09-16T15:48:57.806 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:57.807+0000 7f7a87273640 1 -- start start 2024-09-16T15:48:57.807 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:57.807+0000 7f7a87273640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7a80070470 0x7f7a8014abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:48:57.807 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:57.807+0000 7f7a87273640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f7a80073c20 0x7f7a80146aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:48:57.807 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:57.807+0000 7f7a87273640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7a8015b540 0x7f7a80149ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:48:57.808 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:57.807+0000 7f7a87273640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f7a800b3ba0 con 0x7f7a8015b540 2024-09-16T15:48:57.808 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:57.807+0000 7f7a87273640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f7a800b3a20 con 0x7f7a80070470 2024-09-16T15:48:57.808 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:57.807+0000 7f7a87273640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f7a800b3d20 con 0x7f7a80073c20 2024-09-16T15:48:57.808 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:57.807+0000 7f7a85810640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7a8015b540 0x7f7a80149ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:48:57.808 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:57.807+0000 7f7a85810640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7a8015b540 0x7f7a80149ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:43824/0 (socket says 172.21.15.123:43824) 2024-09-16T15:48:57.808 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:57.807+0000 7f7a85810640 1 -- 172.21.15.123:0/2909532600 learned_addr learned my addr 172.21.15.123:0/2909532600 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:48:57.808 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:57.807+0000 7f7a8500f640 1 --2- 172.21.15.123:0/2909532600 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7a80070470 0x7f7a8014abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:48:57.808 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:57.807+0000 7f7a8480e640 1 --2- 172.21.15.123:0/2909532600 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f7a80073c20 0x7f7a80146aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:48:57.808 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:57.807+0000 7f7a85810640 1 -- 172.21.15.123:0/2909532600 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f7a80073c20 msgr2=0x7f7a80146aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:48:57.808 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:57.807+0000 7f7a85810640 1 --2- 172.21.15.123:0/2909532600 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f7a80073c20 0x7f7a80146aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:57.808 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:57.807+0000 7f7a85810640 1 -- 172.21.15.123:0/2909532600 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7a80070470 msgr2=0x7f7a8014abd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:48:57.808 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:57.807+0000 7f7a85810640 1 --2- 172.21.15.123:0/2909532600 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7a80070470 0x7f7a8014abd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:57.808 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:57.807+0000 7f7a85810640 1 -- 172.21.15.123:0/2909532600 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7a80146fe0 con 0x7f7a8015b540 2024-09-16T15:48:57.809 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:57.807+0000 7f7a85810640 1 --2- 172.21.15.123:0/2909532600 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7a8015b540 0x7f7a80149ab0 secure :-1 s=READY pgs=1156 cs=0 l=1 rev1=1 crypto rx=0x7f7a7800ccb0 tx=0x7f7a7800a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:48:57.809 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:57.807+0000 7f7a75ffb640 1 -- 172.21.15.123:0/2909532600 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a78004b30 con 0x7f7a8015b540 2024-09-16T15:48:57.809 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:57.807+0000 7f7a75ffb640 1 -- 172.21.15.123:0/2909532600 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f7a78004dc0 con 0x7f7a8015b540 2024-09-16T15:48:57.809 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:57.807+0000 7f7a87273640 1 -- 172.21.15.123:0/2909532600 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7a8015fae0 con 0x7f7a8015b540 2024-09-16T15:48:57.809 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:57.807+0000 7f7a75ffb640 1 -- 172.21.15.123:0/2909532600 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a78005280 con 0x7f7a8015b540 2024-09-16T15:48:57.809 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:57.807+0000 7f7a87273640 1 -- 172.21.15.123:0/2909532600 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7a8015ae30 con 0x7f7a8015b540 2024-09-16T15:48:57.810 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:57.807+0000 7f7a87273640 1 -- 172.21.15.123:0/2909532600 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7a800b4230 con 0x7f7a8015b540 2024-09-16T15:48:57.810 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:57.807+0000 7f7a75ffb640 1 -- 172.21.15.123:0/2909532600 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 18) ==== 97386+0+0 (secure 0 0 0) 0x7f7a780054e0 con 0x7f7a8015b540 2024-09-16T15:48:57.810 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:57.811+0000 7f7a75ffb640 1 --2- 172.21.15.123:0/2909532600 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f7a64073220 0x7f7a640756e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:48:57.810 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:57.811+0000 7f7a8500f640 1 --2- 172.21.15.123:0/2909532600 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f7a64073220 0x7f7a640756e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:48:57.810 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:57.811+0000 7f7a75ffb640 1 -- 172.21.15.123:0/2909532600 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(105..105 src has 1..105) ==== 5767+0+0 (secure 0 0 0) 0x7f7a78094fa0 con 0x7f7a8015b540 2024-09-16T15:48:57.810 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:57.811+0000 7f7a8500f640 1 --2- 172.21.15.123:0/2909532600 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f7a64073220 0x7f7a640756e0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f7a7c00a830 tx=0x7f7a7c008040 comp rx=0 tx=0).ready entity=mgr.4150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:48:57.813 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:57.815+0000 7f7a75ffb640 1 -- 172.21.15.123:0/2909532600 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f7a78014030 con 0x7f7a8015b540 2024-09-16T15:48:58.005 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:58.003+0000 7f7a87273640 1 -- 172.21.15.123:0/2909532600 --> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot rm", "vol_name": "cephfs", "sub_name": "subvol37810", "snap_name": "subvol_snap6960", "target": ["mon-mgr", ""]}) -- 0x7f7a800b4850 con 0x7f7a64073220 2024-09-16T15:48:58.057 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:58.059+0000 7f7a75ffb640 1 -- 172.21.15.123:0/2909532600 <== mgr.4150 v2:172.21.15.123:6836/2464759676 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f7a800b4850 con 0x7f7a64073220 2024-09-16T15:48:58.060 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:58.059+0000 7f7a87273640 1 -- 172.21.15.123:0/2909532600 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f7a64073220 msgr2=0x7f7a640756e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:48:58.060 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:58.059+0000 7f7a87273640 1 --2- 172.21.15.123:0/2909532600 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f7a64073220 0x7f7a640756e0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f7a7c00a830 tx=0x7f7a7c008040 comp rx=0 tx=0).stop 2024-09-16T15:48:58.060 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:58.059+0000 7f7a87273640 1 -- 172.21.15.123:0/2909532600 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7a8015b540 msgr2=0x7f7a80149ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:48:58.060 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:58.059+0000 7f7a87273640 1 --2- 172.21.15.123:0/2909532600 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7a8015b540 0x7f7a80149ab0 secure :-1 s=READY pgs=1156 cs=0 l=1 rev1=1 crypto rx=0x7f7a7800ccb0 tx=0x7f7a7800a620 comp rx=0 tx=0).stop 2024-09-16T15:48:58.060 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:58.059+0000 7f7a87273640 1 -- 172.21.15.123:0/2909532600 shutdown_connections 2024-09-16T15:48:58.060 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:58.059+0000 7f7a87273640 1 --2- 172.21.15.123:0/2909532600 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f7a64073220 0x7f7a640756e0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:58.061 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:58.059+0000 7f7a87273640 1 --2- 172.21.15.123:0/2909532600 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7a8015b540 0x7f7a80149ab0 unknown :-1 s=CLOSED pgs=1156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:58.061 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:58.059+0000 7f7a87273640 1 --2- 172.21.15.123:0/2909532600 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f7a80073c20 0x7f7a80146aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:58.061 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:58.059+0000 7f7a87273640 1 --2- 172.21.15.123:0/2909532600 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7a80070470 0x7f7a8014abd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:58.061 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:58.059+0000 7f7a87273640 1 -- 172.21.15.123:0/2909532600 >> 172.21.15.123:0/2909532600 conn(0x7f7a800b1630 msgr2=0x7f7a8014d6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:48:58.061 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:58.059+0000 7f7a87273640 1 -- 172.21.15.123:0/2909532600 shutdown_connections 2024-09-16T15:48:58.061 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:58.059+0000 7f7a87273640 1 -- 172.21.15.123:0/2909532600 wait complete. 2024-09-16T15:48:58.075 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot rm cephfs subvol37810 fake 2024-09-16T15:48:58.194 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:58.195+0000 7fe9e6527640 1 -- 172.21.15.123:0/3679992826 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe9c80051b0 msgr2=0x7fe9c8005920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:48:58.195 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:58.195+0000 7fe9e6527640 1 --2- 172.21.15.123:0/3679992826 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe9c80051b0 0x7fe9c8005920 secure :-1 s=READY pgs=1158 cs=0 l=1 rev1=1 crypto rx=0x7fe9d4006fd0 tx=0x7fe9d401f6c0 comp rx=0 tx=0).stop 2024-09-16T15:48:58.195 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:58.195+0000 7fe9e4ac4640 1 -- 172.21.15.123:0/3679992826 reap_dead start 2024-09-16T15:48:58.195 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:58.195+0000 7fe9e6527640 1 -- 172.21.15.123:0/3679992826 shutdown_connections 2024-09-16T15:48:58.195 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:58.195+0000 7fe9e6527640 1 -- 172.21.15.123:0/3679992826 >> 172.21.15.123:0/3679992826 conn(0x7fe9e00b1630 msgr2=0x7fe9e00b1a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:48:58.195 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:58.195+0000 7fe9e6527640 1 -- 172.21.15.123:0/3679992826 shutdown_connections 2024-09-16T15:48:58.195 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:58.195+0000 7fe9e6527640 1 -- 172.21.15.123:0/3679992826 wait complete. 2024-09-16T15:48:58.196 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:58.195+0000 7fe9e6527640 1 Processor -- start 2024-09-16T15:48:58.196 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:58.195+0000 7fe9e6527640 1 -- start start 2024-09-16T15:48:58.197 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:58.195+0000 7fe9e6527640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe9c80051b0 0x7fe9e015a4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:48:58.197 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:58.195+0000 7fe9e6527640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe9e014de30 0x7fe9e01542a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:48:58.198 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:58.195+0000 7fe9e6527640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fe9e014e2b0 0x7fe9e0152ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:48:58.198 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:58.195+0000 7fe9e6527640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fe9e00b4af0 con 0x7fe9c80051b0 2024-09-16T15:48:58.198 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:58.195+0000 7fe9e6527640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fe9e0072ff0 con 0x7fe9e014de30 2024-09-16T15:48:58.198 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:58.195+0000 7fe9e6527640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fe9e01552b0 con 0x7fe9e014e2b0 2024-09-16T15:48:58.198 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:58.195+0000 7fe9dffff640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe9c80051b0 0x7fe9e015a4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:48:58.198 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:58.195+0000 7fe9dffff640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe9c80051b0 0x7fe9e015a4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:43842/0 (socket says 172.21.15.123:43842) 2024-09-16T15:48:58.198 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:58.195+0000 7fe9dffff640 1 -- 172.21.15.123:0/3647124928 learned_addr learned my addr 172.21.15.123:0/3647124928 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:48:58.198 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:58.195+0000 7fe9dffff640 1 -- 172.21.15.123:0/3647124928 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fe9e014e2b0 msgr2=0x7fe9e0152ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:48:58.198 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:58.195+0000 7fe9dffff640 1 --2- 172.21.15.123:0/3647124928 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fe9e014e2b0 0x7fe9e0152ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:58.198 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:58.195+0000 7fe9dffff640 1 -- 172.21.15.123:0/3647124928 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe9e014de30 msgr2=0x7fe9e01542a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:48:58.198 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:58.195+0000 7fe9dffff640 1 --2- 172.21.15.123:0/3647124928 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe9e014de30 0x7fe9e01542a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:58.198 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:58.195+0000 7fe9dffff640 1 -- 172.21.15.123:0/3647124928 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe9e0154cf0 con 0x7fe9c80051b0 2024-09-16T15:48:58.199 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:58.195+0000 7fe9dffff640 1 --2- 172.21.15.123:0/3647124928 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe9c80051b0 0x7fe9e015a4a0 secure :-1 s=READY pgs=1159 cs=0 l=1 rev1=1 crypto rx=0x7fe9d000e9c0 tx=0x7fe9d000ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:48:58.199 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:58.199+0000 7fe9dcff9640 1 -- 172.21.15.123:0/3647124928 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9d000cc60 con 0x7fe9c80051b0 2024-09-16T15:48:58.199 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:58.199+0000 7fe9dcff9640 1 -- 172.21.15.123:0/3647124928 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fe9d0005020 con 0x7fe9c80051b0 2024-09-16T15:48:58.199 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:58.199+0000 7fe9e6527640 1 -- 172.21.15.123:0/3647124928 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe9e0154fd0 con 0x7fe9c80051b0 2024-09-16T15:48:58.199 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:58.199+0000 7fe9dcff9640 1 -- 172.21.15.123:0/3647124928 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9d0005450 con 0x7fe9c80051b0 2024-09-16T15:48:58.199 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:58.199+0000 7fe9e6527640 1 -- 172.21.15.123:0/3647124928 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe9e015a9e0 con 0x7fe9c80051b0 2024-09-16T15:48:58.200 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:58.199+0000 7fe9e6527640 1 -- 172.21.15.123:0/3647124928 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe9e00cf310 con 0x7fe9c80051b0 2024-09-16T15:48:58.200 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:58.199+0000 7fe9dcff9640 1 -- 172.21.15.123:0/3647124928 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 18) ==== 97386+0+0 (secure 0 0 0) 0x7fe9d00047b0 con 0x7fe9c80051b0 2024-09-16T15:48:58.200 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:58.199+0000 7fe9dcff9640 1 --2- 172.21.15.123:0/3647124928 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fe9cc073220 0x7fe9cc0756e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:48:58.200 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:58.199+0000 7fe9dcff9640 1 -- 172.21.15.123:0/3647124928 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(105..105 src has 1..105) ==== 5767+0+0 (secure 0 0 0) 0x7fe9d0094e40 con 0x7fe9c80051b0 2024-09-16T15:48:58.200 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:58.199+0000 7fe9df7fe640 1 --2- 172.21.15.123:0/3647124928 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fe9cc073220 0x7fe9cc0756e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:48:58.200 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:58.199+0000 7fe9df7fe640 1 --2- 172.21.15.123:0/3647124928 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fe9cc073220 0x7fe9cc0756e0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fe9d40b0000 tx=0x7fe9c8006950 comp rx=0 tx=0).ready entity=mgr.4150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:48:58.203 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:58.203+0000 7fe9dcff9640 1 -- 172.21.15.123:0/3647124928 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7fe9d0016030 con 0x7fe9c80051b0 2024-09-16T15:48:58.402 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:58.403+0000 7fe9e6527640 1 -- 172.21.15.123:0/3647124928 --> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot rm", "vol_name": "cephfs", "sub_name": "subvol37810", "snap_name": "fake", "target": ["mon-mgr", ""]}) -- 0x7fe9e0072d40 con 0x7fe9cc073220 2024-09-16T15:48:58.452 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:58.451+0000 7fe9dcff9640 1 -- 172.21.15.123:0/3647124928 <== mgr.4150 v2:172.21.15.123:6836/2464759676 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7fe9e0072d40 con 0x7fe9cc073220 2024-09-16T15:48:58.455 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:58.455+0000 7fe9e6527640 1 -- 172.21.15.123:0/3647124928 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fe9cc073220 msgr2=0x7fe9cc0756e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:48:58.455 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:58.455+0000 7fe9e6527640 1 --2- 172.21.15.123:0/3647124928 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fe9cc073220 0x7fe9cc0756e0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fe9d40b0000 tx=0x7fe9c8006950 comp rx=0 tx=0).stop 2024-09-16T15:48:58.455 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:58.455+0000 7fe9e6527640 1 -- 172.21.15.123:0/3647124928 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe9c80051b0 msgr2=0x7fe9e015a4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:48:58.455 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:58.455+0000 7fe9e6527640 1 --2- 172.21.15.123:0/3647124928 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe9c80051b0 0x7fe9e015a4a0 secure :-1 s=READY pgs=1159 cs=0 l=1 rev1=1 crypto rx=0x7fe9d000e9c0 tx=0x7fe9d000ee80 comp rx=0 tx=0).stop 2024-09-16T15:48:58.455 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:58.455+0000 7fe9e6527640 1 -- 172.21.15.123:0/3647124928 shutdown_connections 2024-09-16T15:48:58.455 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:58.455+0000 7fe9e6527640 1 --2- 172.21.15.123:0/3647124928 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fe9cc073220 0x7fe9cc0756e0 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:58.455 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:58.455+0000 7fe9e6527640 1 --2- 172.21.15.123:0/3647124928 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fe9e014e2b0 0x7fe9e0152ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:58.455 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:58.455+0000 7fe9e6527640 1 --2- 172.21.15.123:0/3647124928 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe9e014de30 0x7fe9e01542a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:58.455 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:58.455+0000 7fe9e6527640 1 --2- 172.21.15.123:0/3647124928 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe9c80051b0 0x7fe9e015a4a0 unknown :-1 s=CLOSED pgs=1159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:48:58.456 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:58.455+0000 7fe9e6527640 1 -- 172.21.15.123:0/3647124928 >> 172.21.15.123:0/3647124928 conn(0x7fe9e00b1630 msgr2=0x7fe9e00cfa40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:48:58.456 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:58.455+0000 7fe9e6527640 1 -- 172.21.15.123:0/3647124928 shutdown_connections 2024-09-16T15:48:58.456 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:48:58.455+0000 7fe9e6527640 1 -- 172.21.15.123:0/3647124928 wait complete. 2024-09-16T15:48:58.470 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:48:58.470 DEBUG:teuthology.orchestra.run.smithi123:> (cd /home/ubuntu/cephtest/mnt.0 && exec sudo cat ./volumes/_nogroup/subvol37810/.meta) 2024-09-16T15:49:01.797 INFO:teuthology.orchestra.run.smithi123.stdout:[GLOBAL] 2024-09-16T15:49:01.797 INFO:teuthology.orchestra.run.smithi123.stdout:version = 1 2024-09-16T15:49:01.797 INFO:teuthology.orchestra.run.smithi123.stdout:type = subvolume 2024-09-16T15:49:01.797 INFO:teuthology.orchestra.run.smithi123.stdout:path = /volumes/_nogroup/subvol37810/b00a5d9c-1a58-42eb-be9a-5f5c5fe23179 2024-09-16T15:49:01.797 INFO:teuthology.orchestra.run.smithi123.stdout:state = complete 2024-09-16T15:49:01.797 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-09-16T15:49:01.800 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume rm cephfs subvol37810 2024-09-16T15:49:01.938 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:01.939+0000 7f177df4d640 1 -- 172.21.15.123:0/1260743545 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f17780d45e0 msgr2=0x7f17780d2380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:01.939 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:01.939+0000 7f177df4d640 1 --2- 172.21.15.123:0/1260743545 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f17780d45e0 0x7f17780d2380 secure :-1 s=READY pgs=1161 cs=0 l=1 rev1=1 crypto rx=0x7f176c009840 tx=0x7f176c01c6b0 comp rx=0 tx=0).stop 2024-09-16T15:49:01.939 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:01.939+0000 7f177df4d640 1 -- 172.21.15.123:0/1260743545 shutdown_connections 2024-09-16T15:49:01.939 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:01.939+0000 7f177df4d640 1 --2- 172.21.15.123:0/1260743545 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f17780d45e0 0x7f17780d2380 unknown :-1 s=CLOSED pgs=1161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:01.939 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:01.939+0000 7f177df4d640 1 --2- 172.21.15.123:0/1260743545 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f17781625d0 0x7f17780d1e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:01.939 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:01.939+0000 7f177df4d640 1 -- 172.21.15.123:0/1260743545 >> 172.21.15.123:0/1260743545 conn(0x7f177809d1d0 msgr2=0x7f17780d3ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:49:01.939 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:01.939+0000 7f177df4d640 1 -- 172.21.15.123:0/1260743545 shutdown_connections 2024-09-16T15:49:01.939 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:01.939+0000 7f177df4d640 1 -- 172.21.15.123:0/1260743545 wait complete. 2024-09-16T15:49:01.940 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:01.939+0000 7f177df4d640 1 Processor -- start 2024-09-16T15:49:01.940 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:01.939+0000 7f177df4d640 1 -- start start 2024-09-16T15:49:01.942 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:01.939+0000 7f177df4d640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f17780d28c0 0x7f177815be50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:49:01.942 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:01.939+0000 7f177df4d640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f17780d45e0 0x7f17781f84b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:49:01.942 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:01.939+0000 7f177df4d640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f17781625d0 0x7f17781514c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:49:01.942 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:01.939+0000 7f177df4d640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f177816b3e0 con 0x7f17780d45e0 2024-09-16T15:49:01.942 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:01.939+0000 7f177df4d640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f177816b260 con 0x7f17781625d0 2024-09-16T15:49:01.942 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:01.939+0000 7f177df4d640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f177816b560 con 0x7f17780d28c0 2024-09-16T15:49:01.942 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:01.939+0000 7f1776ffd640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f17780d45e0 0x7f17781f84b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:49:01.942 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:01.939+0000 7f1776ffd640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f17780d45e0 0x7f17781f84b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:43858/0 (socket says 172.21.15.123:43858) 2024-09-16T15:49:01.942 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:01.939+0000 7f1776ffd640 1 -- 172.21.15.123:0/3498037767 learned_addr learned my addr 172.21.15.123:0/3498037767 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:49:01.942 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:01.939+0000 7f1777fff640 1 --2- 172.21.15.123:0/3498037767 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f17781625d0 0x7f17781514c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:49:01.943 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:01.939+0000 7f1776ffd640 1 -- 172.21.15.123:0/3498037767 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f17780d28c0 msgr2=0x7f177815be50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:01.943 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:01.939+0000 7f17777fe640 1 --2- 172.21.15.123:0/3498037767 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f17780d28c0 0x7f177815be50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:49:01.943 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:01.939+0000 7f1776ffd640 1 --2- 172.21.15.123:0/3498037767 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f17780d28c0 0x7f177815be50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:01.943 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:01.939+0000 7f1776ffd640 1 -- 172.21.15.123:0/3498037767 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f17781625d0 msgr2=0x7f17781514c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:01.943 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:01.939+0000 7f1776ffd640 1 --2- 172.21.15.123:0/3498037767 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f17781625d0 0x7f17781514c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:01.943 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:01.939+0000 7f1776ffd640 1 -- 172.21.15.123:0/3498037767 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f17781f89f0 con 0x7f17780d45e0 2024-09-16T15:49:01.943 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:01.939+0000 7f1777fff640 1 --2- 172.21.15.123:0/3498037767 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f17781625d0 0x7f17781514c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T15:49:01.943 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:01.943+0000 7f1776ffd640 1 --2- 172.21.15.123:0/3498037767 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f17780d45e0 0x7f17781f84b0 secure :-1 s=READY pgs=1162 cs=0 l=1 rev1=1 crypto rx=0x7f176c009790 tx=0x7f176c0037e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:49:01.943 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:01.943+0000 7f175ffff640 1 -- 172.21.15.123:0/3498037767 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f176c003de0 con 0x7f17780d45e0 2024-09-16T15:49:01.943 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:01.943+0000 7f175ffff640 1 -- 172.21.15.123:0/3498037767 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f176c0b9070 con 0x7f17780d45e0 2024-09-16T15:49:01.943 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:01.943+0000 7f177df4d640 1 -- 172.21.15.123:0/3498037767 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1778151a00 con 0x7f17780d45e0 2024-09-16T15:49:01.944 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:01.943+0000 7f175ffff640 1 -- 172.21.15.123:0/3498037767 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f176c0053a0 con 0x7f17780d45e0 2024-09-16T15:49:01.944 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:01.943+0000 7f177df4d640 1 -- 172.21.15.123:0/3498037767 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f177815c450 con 0x7f17780d45e0 2024-09-16T15:49:01.944 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:01.943+0000 7f177df4d640 1 -- 172.21.15.123:0/3498037767 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f177806d3a0 con 0x7f17780d45e0 2024-09-16T15:49:01.944 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:01.943+0000 7f175ffff640 1 -- 172.21.15.123:0/3498037767 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 18) ==== 97386+0+0 (secure 0 0 0) 0x7f176c0046d0 con 0x7f17780d45e0 2024-09-16T15:49:01.945 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:01.943+0000 7f175ffff640 1 --2- 172.21.15.123:0/3498037767 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f1764073220 0x7f17640756e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:49:01.945 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:01.943+0000 7f175ffff640 1 -- 172.21.15.123:0/3498037767 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(105..105 src has 1..105) ==== 5767+0+0 (secure 0 0 0) 0x7f176c12f8b0 con 0x7f17780d45e0 2024-09-16T15:49:01.945 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:01.943+0000 7f17777fe640 1 --2- 172.21.15.123:0/3498037767 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f1764073220 0x7f17640756e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:49:01.945 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:01.943+0000 7f17777fe640 1 --2- 172.21.15.123:0/3498037767 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f1764073220 0x7f17640756e0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f176800a790 tx=0x7f1768008040 comp rx=0 tx=0).ready entity=mgr.4150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:49:01.947 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:01.947+0000 7f175ffff640 1 -- 172.21.15.123:0/3498037767 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f17781f89f0 con 0x7f17780d45e0 2024-09-16T15:49:02.144 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:02.143+0000 7f177df4d640 1 -- 172.21.15.123:0/3498037767 --> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] -- mgr_command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "subvol37810", "target": ["mon-mgr", ""]}) -- 0x7f177808df80 con 0x7f1764073220 2024-09-16T15:49:02.211 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:02.211+0000 7f175ffff640 1 -- 172.21.15.123:0/3498037767 <== mgr.4150 v2:172.21.15.123:6836/2464759676 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f177808df80 con 0x7f1764073220 2024-09-16T15:49:02.214 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:02.215+0000 7f177df4d640 1 -- 172.21.15.123:0/3498037767 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f1764073220 msgr2=0x7f17640756e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:02.214 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:02.215+0000 7f177df4d640 1 --2- 172.21.15.123:0/3498037767 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f1764073220 0x7f17640756e0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f176800a790 tx=0x7f1768008040 comp rx=0 tx=0).stop 2024-09-16T15:49:02.214 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:02.215+0000 7f177df4d640 1 -- 172.21.15.123:0/3498037767 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f17780d45e0 msgr2=0x7f17781f84b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:02.214 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:02.215+0000 7f177df4d640 1 --2- 172.21.15.123:0/3498037767 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f17780d45e0 0x7f17781f84b0 secure :-1 s=READY pgs=1162 cs=0 l=1 rev1=1 crypto rx=0x7f176c009790 tx=0x7f176c0037e0 comp rx=0 tx=0).stop 2024-09-16T15:49:02.214 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:02.215+0000 7f177df4d640 1 -- 172.21.15.123:0/3498037767 shutdown_connections 2024-09-16T15:49:02.214 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:02.215+0000 7f177df4d640 1 --2- 172.21.15.123:0/3498037767 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f1764073220 0x7f17640756e0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:02.214 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:02.215+0000 7f177df4d640 1 --2- 172.21.15.123:0/3498037767 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f17781625d0 0x7f17781514c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:02.214 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:02.215+0000 7f177df4d640 1 --2- 172.21.15.123:0/3498037767 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f17780d45e0 0x7f17781f84b0 unknown :-1 s=CLOSED pgs=1162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:02.215 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:02.215+0000 7f177df4d640 1 --2- 172.21.15.123:0/3498037767 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f17780d28c0 0x7f177815be50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:02.215 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:02.215+0000 7f177df4d640 1 -- 172.21.15.123:0/3498037767 >> 172.21.15.123:0/3498037767 conn(0x7f177809d1d0 msgr2=0x7f17781607e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:49:02.215 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:02.215+0000 7f177df4d640 1 -- 172.21.15.123:0/3498037767 shutdown_connections 2024-09-16T15:49:02.215 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:02.215+0000 7f177df4d640 1 -- 172.21.15.123:0/3498037767 wait complete. 2024-09-16T15:49:02.218 INFO:tasks.ceph.mgr.y.smithi123.stderr:2024-09-16T15:49:02.219+0000 7fa9b4ef7640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-16T15:49:02.218 INFO:tasks.ceph.mgr.y.smithi123.stderr:2024-09-16T15:49:02.219+0000 7fa9b4ef7640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-16T15:49:02.218 INFO:tasks.ceph.mgr.y.smithi123.stderr:2024-09-16T15:49:02.219+0000 7fa9b4ef7640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-16T15:49:02.218 INFO:tasks.ceph.mgr.y.smithi123.stderr:2024-09-16T15:49:02.219+0000 7fa9b4ef7640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-16T15:49:02.219 INFO:tasks.ceph.mgr.y.smithi123.stderr:2024-09-16T15:49:02.219+0000 7fa9b4ef7640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-16T15:49:02.235 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume rm cephfs subvol_clone4560 2024-09-16T15:49:02.236 INFO:tasks.ceph.mgr.y.smithi123.stderr:2024-09-16T15:49:02.235+0000 7fa9b56f8640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-16T15:49:02.236 INFO:tasks.ceph.mgr.y.smithi123.stderr:2024-09-16T15:49:02.235+0000 7fa9b56f8640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-16T15:49:02.236 INFO:tasks.ceph.mgr.y.smithi123.stderr:2024-09-16T15:49:02.235+0000 7fa9b56f8640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-16T15:49:02.237 INFO:tasks.ceph.mgr.y.smithi123.stderr:2024-09-16T15:49:02.235+0000 7fa9b56f8640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-16T15:49:02.237 INFO:tasks.ceph.mgr.y.smithi123.stderr:2024-09-16T15:49:02.235+0000 7fa9b56f8640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-16T15:49:02.360 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:02.359+0000 7effae08c640 1 -- 172.21.15.123:0/3085873898 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7effa8072720 msgr2=0x7effa80cabc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:02.360 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:02.359+0000 7effae08c640 1 --2- 172.21.15.123:0/3085873898 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7effa8072720 0x7effa80cabc0 secure :-1 s=READY pgs=1167 cs=0 l=1 rev1=1 crypto rx=0x7eff9c009900 tx=0x7eff9c01c8e0 comp rx=0 tx=0).stop 2024-09-16T15:49:02.360 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:02.359+0000 7effae08c640 1 -- 172.21.15.123:0/3085873898 shutdown_connections 2024-09-16T15:49:02.360 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:02.359+0000 7effae08c640 1 --2- 172.21.15.123:0/3085873898 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7effa8072720 0x7effa80cabc0 unknown :-1 s=CLOSED pgs=1167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:02.360 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:02.359+0000 7effae08c640 1 --2- 172.21.15.123:0/3085873898 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7effa815aa00 0x7effa815e5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:02.360 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:02.359+0000 7effae08c640 1 -- 172.21.15.123:0/3085873898 >> 172.21.15.123:0/3085873898 conn(0x7effa80b1710 msgr2=0x7effa80b1b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:49:02.360 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:02.359+0000 7effae08c640 1 -- 172.21.15.123:0/3085873898 shutdown_connections 2024-09-16T15:49:02.360 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:02.359+0000 7effae08c640 1 -- 172.21.15.123:0/3085873898 wait complete. 2024-09-16T15:49:02.361 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:02.359+0000 7effae08c640 1 Processor -- start 2024-09-16T15:49:02.361 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:02.363+0000 7effae08c640 1 -- start start 2024-09-16T15:49:02.363 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:02.363+0000 7effae08c640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7effa8072720 0x7effa814a930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:49:02.363 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:02.363+0000 7effae08c640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7effa80cc3c0 0x7effa8155ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:49:02.363 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:02.363+0000 7effae08c640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7effa815aa00 0x7effa80ae260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:49:02.363 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:02.363+0000 7effae08c640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7effa8168510 con 0x7effa815aa00 2024-09-16T15:49:02.363 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:02.363+0000 7effae08c640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7effa8168390 con 0x7effa80cc3c0 2024-09-16T15:49:02.363 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:02.363+0000 7effae08c640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7effa8168690 con 0x7effa8072720 2024-09-16T15:49:02.363 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:02.363+0000 7effa7fff640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7effa815aa00 0x7effa80ae260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:49:02.364 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:02.363+0000 7effa7fff640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7effa815aa00 0x7effa80ae260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:43924/0 (socket says 172.21.15.123:43924) 2024-09-16T15:49:02.364 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:02.363+0000 7effa7fff640 1 -- 172.21.15.123:0/2996770226 learned_addr learned my addr 172.21.15.123:0/2996770226 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:49:02.364 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:02.363+0000 7effa77fe640 1 --2- 172.21.15.123:0/2996770226 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7effa8072720 0x7effa814a930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:49:02.364 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:02.363+0000 7effa6ffd640 1 --2- 172.21.15.123:0/2996770226 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7effa80cc3c0 0x7effa8155ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:49:02.364 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:02.363+0000 7effa7fff640 1 -- 172.21.15.123:0/2996770226 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7effa8072720 msgr2=0x7effa814a930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:02.364 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:02.363+0000 7effa7fff640 1 --2- 172.21.15.123:0/2996770226 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7effa8072720 0x7effa814a930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:02.364 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:02.363+0000 7effa7fff640 1 -- 172.21.15.123:0/2996770226 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7effa80cc3c0 msgr2=0x7effa8155ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:02.364 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:02.363+0000 7effa7fff640 1 --2- 172.21.15.123:0/2996770226 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7effa80cc3c0 0x7effa8155ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:02.364 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:02.363+0000 7effa7fff640 1 -- 172.21.15.123:0/2996770226 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7effa80b45f0 con 0x7effa815aa00 2024-09-16T15:49:02.364 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:02.363+0000 7effa6ffd640 1 --2- 172.21.15.123:0/2996770226 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7effa80cc3c0 0x7effa8155ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T15:49:02.364 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:02.363+0000 7effa7fff640 1 --2- 172.21.15.123:0/2996770226 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7effa815aa00 0x7effa80ae260 secure :-1 s=READY pgs=1168 cs=0 l=1 rev1=1 crypto rx=0x7effa000c780 tx=0x7effa000cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:49:02.364 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:02.363+0000 7eff8ffff640 1 -- 172.21.15.123:0/2996770226 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7effa0004b70 con 0x7effa815aa00 2024-09-16T15:49:02.365 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:02.363+0000 7eff8ffff640 1 -- 172.21.15.123:0/2996770226 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7effa0004e00 con 0x7effa815aa00 2024-09-16T15:49:02.365 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:02.363+0000 7effae08c640 1 -- 172.21.15.123:0/2996770226 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7effa80b48d0 con 0x7effa815aa00 2024-09-16T15:49:02.365 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:02.363+0000 7eff8ffff640 1 -- 172.21.15.123:0/2996770226 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7effa000aab0 con 0x7effa815aa00 2024-09-16T15:49:02.365 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:02.363+0000 7effae08c640 1 -- 172.21.15.123:0/2996770226 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7effa80add40 con 0x7effa815aa00 2024-09-16T15:49:02.366 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:02.363+0000 7effae08c640 1 -- 172.21.15.123:0/2996770226 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7effa80adf40 con 0x7effa815aa00 2024-09-16T15:49:02.366 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:02.363+0000 7eff8ffff640 1 -- 172.21.15.123:0/2996770226 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 18) ==== 97386+0+0 (secure 0 0 0) 0x7effa000ac50 con 0x7effa815aa00 2024-09-16T15:49:02.366 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:02.363+0000 7eff8ffff640 1 --2- 172.21.15.123:0/2996770226 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7eff94073220 0x7eff940756e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:49:02.366 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:02.363+0000 7eff8ffff640 1 -- 172.21.15.123:0/2996770226 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(106..106 src has 1..106) ==== 5823+0+0 (secure 0 0 0) 0x7effa0094790 con 0x7effa815aa00 2024-09-16T15:49:02.366 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:02.363+0000 7effa77fe640 1 --2- 172.21.15.123:0/2996770226 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7eff94073220 0x7eff940756e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:49:02.366 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:02.367+0000 7effa77fe640 1 --2- 172.21.15.123:0/2996770226 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7eff94073220 0x7eff940756e0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7eff98005fd0 tx=0x7eff98005d10 comp rx=0 tx=0).ready entity=mgr.4150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:49:02.369 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:02.367+0000 7eff8ffff640 1 -- 172.21.15.123:0/2996770226 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7effa80adf40 con 0x7effa815aa00 2024-09-16T15:49:02.571 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:02.571+0000 7effae08c640 1 -- 172.21.15.123:0/2996770226 --> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] -- mgr_command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "subvol_clone4560", "target": ["mon-mgr", ""]}) -- 0x7effa80b5b40 con 0x7eff94073220 2024-09-16T15:49:02.598 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:02.599+0000 7eff8ffff640 1 -- 172.21.15.123:0/2996770226 <== mgr.4150 v2:172.21.15.123:6836/2464759676 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7eff9800ea40 con 0x7eff94073220 2024-09-16T15:49:02.601 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:02.599+0000 7effae08c640 1 -- 172.21.15.123:0/2996770226 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7eff94073220 msgr2=0x7eff940756e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:02.601 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:02.599+0000 7effae08c640 1 --2- 172.21.15.123:0/2996770226 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7eff94073220 0x7eff940756e0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7eff98005fd0 tx=0x7eff98005d10 comp rx=0 tx=0).stop 2024-09-16T15:49:02.601 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:02.599+0000 7effae08c640 1 -- 172.21.15.123:0/2996770226 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7effa815aa00 msgr2=0x7effa80ae260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:02.601 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:02.599+0000 7effae08c640 1 --2- 172.21.15.123:0/2996770226 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7effa815aa00 0x7effa80ae260 secure :-1 s=READY pgs=1168 cs=0 l=1 rev1=1 crypto rx=0x7effa000c780 tx=0x7effa000cc40 comp rx=0 tx=0).stop 2024-09-16T15:49:02.601 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:02.603+0000 7effae08c640 1 -- 172.21.15.123:0/2996770226 shutdown_connections 2024-09-16T15:49:02.601 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:02.603+0000 7effae08c640 1 --2- 172.21.15.123:0/2996770226 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7eff94073220 0x7eff940756e0 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:02.601 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:02.603+0000 7effae08c640 1 --2- 172.21.15.123:0/2996770226 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7effa815aa00 0x7effa80ae260 unknown :-1 s=CLOSED pgs=1168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:02.602 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:02.603+0000 7effae08c640 1 --2- 172.21.15.123:0/2996770226 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7effa80cc3c0 0x7effa8155ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:02.602 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:02.603+0000 7effae08c640 1 --2- 172.21.15.123:0/2996770226 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7effa8072720 0x7effa814a930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:02.602 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:02.603+0000 7effae08c640 1 -- 172.21.15.123:0/2996770226 >> 172.21.15.123:0/2996770226 conn(0x7effa80b1710 msgr2=0x7effa80758d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:49:02.602 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:02.603+0000 7effae08c640 1 -- 172.21.15.123:0/2996770226 shutdown_connections 2024-09-16T15:49:02.602 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:02.603+0000 7effae08c640 1 -- 172.21.15.123:0/2996770226 wait complete. 2024-09-16T15:49:02.621 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume rm cephfs subvol_clone62021 2024-09-16T15:49:02.749 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:02.751+0000 7fe62ef6a640 1 -- 172.21.15.123:0/3499670043 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe628159f70 msgr2=0x7fe628072f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:02.749 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:02.751+0000 7fe62ef6a640 1 --2- 172.21.15.123:0/3499670043 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe628159f70 0x7fe628072f20 secure :-1 s=READY pgs=1169 cs=0 l=1 rev1=1 crypto rx=0x7fe628074030 tx=0x7fe61c01c8c0 comp rx=0 tx=0).stop 2024-09-16T15:49:02.750 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:02.751+0000 7fe62ef6a640 1 -- 172.21.15.123:0/3499670043 shutdown_connections 2024-09-16T15:49:02.750 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:02.751+0000 7fe62ef6a640 1 --2- 172.21.15.123:0/3499670043 >> v2:172.21.15.184:3300/0 conn(0x7fe62815ddb0 0x7fe62815e220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:02.750 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:02.751+0000 7fe62ef6a640 1 --2- 172.21.15.123:0/3499670043 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe628159f70 0x7fe628072f20 unknown :-1 s=CLOSED pgs=1169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:02.750 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:02.751+0000 7fe62ef6a640 1 -- 172.21.15.123:0/3499670043 >> 172.21.15.123:0/3499670043 conn(0x7fe6280b1710 msgr2=0x7fe6280b1b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:49:02.750 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:02.751+0000 7fe62ef6a640 1 -- 172.21.15.123:0/3499670043 shutdown_connections 2024-09-16T15:49:02.750 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:02.751+0000 7fe62ef6a640 1 -- 172.21.15.123:0/3499670043 wait complete. 2024-09-16T15:49:02.750 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:02.751+0000 7fe62ef6a640 1 Processor -- start 2024-09-16T15:49:02.751 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:02.751+0000 7fe62ef6a640 1 -- start start 2024-09-16T15:49:02.752 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:02.751+0000 7fe62ef6a640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fe628159f70 0x7fe6281fb930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:49:02.752 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:02.751+0000 7fe62ef6a640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe62815a350 0x7fe628164360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:49:02.752 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:02.751+0000 7fe62ef6a640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe62815ddb0 0x7fe6281648a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:49:02.752 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:02.751+0000 7fe62ef6a640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fe6280cc6d0 con 0x7fe62815a350 2024-09-16T15:49:02.752 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:02.751+0000 7fe62ef6a640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fe6280cc550 con 0x7fe62815ddb0 2024-09-16T15:49:02.753 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:02.751+0000 7fe62ef6a640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fe6280cc850 con 0x7fe628159f70 2024-09-16T15:49:02.753 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:02.751+0000 7fe627fff640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe62815a350 0x7fe628164360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:49:02.753 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:02.751+0000 7fe627fff640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe62815a350 0x7fe628164360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:55348/0 (socket says 172.21.15.123:55348) 2024-09-16T15:49:02.753 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:02.751+0000 7fe627fff640 1 -- 172.21.15.123:0/226907956 learned_addr learned my addr 172.21.15.123:0/226907956 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:49:02.753 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:02.751+0000 7fe62cd06640 1 --2- 172.21.15.123:0/226907956 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fe628159f70 0x7fe6281fb930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:49:02.753 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:02.751+0000 7fe627fff640 1 -- 172.21.15.123:0/226907956 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fe628159f70 msgr2=0x7fe6281fb930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:02.753 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:02.751+0000 7fe627fff640 1 --2- 172.21.15.123:0/226907956 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fe628159f70 0x7fe6281fb930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:02.753 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:02.751+0000 7fe627fff640 1 -- 172.21.15.123:0/226907956 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe62815ddb0 msgr2=0x7fe6281648a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:02.753 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:02.751+0000 7fe62d507640 1 --2- 172.21.15.123:0/226907956 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe62815ddb0 0x7fe6281648a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:49:02.753 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:02.751+0000 7fe627fff640 1 --2- 172.21.15.123:0/226907956 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe62815ddb0 0x7fe6281648a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:02.753 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:02.751+0000 7fe627fff640 1 -- 172.21.15.123:0/226907956 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe6281fbe70 con 0x7fe62815a350 2024-09-16T15:49:02.753 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:02.751+0000 7fe627fff640 1 --2- 172.21.15.123:0/226907956 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe62815a350 0x7fe628164360 secure :-1 s=READY pgs=1170 cs=0 l=1 rev1=1 crypto rx=0x7fe61800c490 tx=0x7fe618001e80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:49:02.753 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:02.751+0000 7fe6257fa640 1 -- 172.21.15.123:0/226907956 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe618017070 con 0x7fe62815a350 2024-09-16T15:49:02.754 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:02.751+0000 7fe6257fa640 1 -- 172.21.15.123:0/226907956 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fe618003600 con 0x7fe62815a350 2024-09-16T15:49:02.754 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:02.751+0000 7fe6257fa640 1 -- 172.21.15.123:0/226907956 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe618003a50 con 0x7fe62815a350 2024-09-16T15:49:02.754 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:02.751+0000 7fe62ef6a640 1 -- 172.21.15.123:0/226907956 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe6280b38c0 con 0x7fe62815a350 2024-09-16T15:49:02.754 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:02.751+0000 7fe62ef6a640 1 -- 172.21.15.123:0/226907956 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe6280b3e80 con 0x7fe62815a350 2024-09-16T15:49:02.754 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:02.755+0000 7fe62ef6a640 1 -- 172.21.15.123:0/226907956 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe628070220 con 0x7fe62815a350 2024-09-16T15:49:02.754 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:02.755+0000 7fe6257fa640 1 -- 172.21.15.123:0/226907956 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 18) ==== 97386+0+0 (secure 0 0 0) 0x7fe618003bf0 con 0x7fe62815a350 2024-09-16T15:49:02.755 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:02.755+0000 7fe6257fa640 1 --2- 172.21.15.123:0/226907956 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fe614073250 0x7fe614075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:49:02.755 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:02.755+0000 7fe6257fa640 1 -- 172.21.15.123:0/226907956 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(106..106 src has 1..106) ==== 5823+0+0 (secure 0 0 0) 0x7fe618093660 con 0x7fe62815a350 2024-09-16T15:49:02.755 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:02.755+0000 7fe62cd06640 1 --2- 172.21.15.123:0/226907956 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fe614073250 0x7fe614075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:49:02.756 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:02.755+0000 7fe62cd06640 1 --2- 172.21.15.123:0/226907956 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fe614073250 0x7fe614075710 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fe61c0af000 tx=0x7fe61c01b100 comp rx=0 tx=0).ready entity=mgr.4150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:49:02.759 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:02.759+0000 7fe6257fa640 1 -- 172.21.15.123:0/226907956 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7fe6281fbe70 con 0x7fe62815a350 2024-09-16T15:49:02.959 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:02.959+0000 7fe62ef6a640 1 -- 172.21.15.123:0/226907956 --> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] -- mgr_command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "subvol_clone62021", "target": ["mon-mgr", ""]}) -- 0x7fe628072ea0 con 0x7fe614073250 2024-09-16T15:49:02.983 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:02.983+0000 7fe6257fa640 1 -- 172.21.15.123:0/226907956 <== mgr.4150 v2:172.21.15.123:6836/2464759676 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7fe628072ea0 con 0x7fe614073250 2024-09-16T15:49:02.986 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:02.987+0000 7fe62ef6a640 1 -- 172.21.15.123:0/226907956 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fe614073250 msgr2=0x7fe614075710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:02.987 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:02.987+0000 7fe62ef6a640 1 --2- 172.21.15.123:0/226907956 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fe614073250 0x7fe614075710 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fe61c0af000 tx=0x7fe61c01b100 comp rx=0 tx=0).stop 2024-09-16T15:49:02.987 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:02.987+0000 7fe62ef6a640 1 -- 172.21.15.123:0/226907956 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe62815a350 msgr2=0x7fe628164360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:02.987 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:02.987+0000 7fe62ef6a640 1 --2- 172.21.15.123:0/226907956 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe62815a350 0x7fe628164360 secure :-1 s=READY pgs=1170 cs=0 l=1 rev1=1 crypto rx=0x7fe61800c490 tx=0x7fe618001e80 comp rx=0 tx=0).stop 2024-09-16T15:49:02.987 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:02.987+0000 7fe62ef6a640 1 -- 172.21.15.123:0/226907956 shutdown_connections 2024-09-16T15:49:02.987 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:02.987+0000 7fe62ef6a640 1 --2- 172.21.15.123:0/226907956 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fe614073250 0x7fe614075710 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:02.987 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:02.987+0000 7fe62ef6a640 1 --2- 172.21.15.123:0/226907956 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe62815ddb0 0x7fe6281648a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:02.987 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:02.987+0000 7fe62ef6a640 1 --2- 172.21.15.123:0/226907956 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe62815a350 0x7fe628164360 unknown :-1 s=CLOSED pgs=1170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:02.987 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:02.987+0000 7fe62ef6a640 1 --2- 172.21.15.123:0/226907956 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fe628159f70 0x7fe6281fb930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:02.987 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:02.987+0000 7fe62ef6a640 1 -- 172.21.15.123:0/226907956 >> 172.21.15.123:0/226907956 conn(0x7fe6280b1710 msgr2=0x7fe62806e3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:49:02.987 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:02.987+0000 7fe62ef6a640 1 -- 172.21.15.123:0/226907956 shutdown_connections 2024-09-16T15:49:02.987 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:02.987+0000 7fe62ef6a640 1 -- 172.21.15.123:0/226907956 wait complete. 2024-09-16T15:49:03.007 DEBUG:teuthology.orchestra.run.smithi123:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'stat -c %h /home/ubuntu/cephtest/mnt.0/./volumes/_deleting') 2024-09-16T15:49:03.049 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:03 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-09-16T15:49:03.054 INFO:teuthology.orchestra.run.smithi123.stdout:2 2024-09-16T15:49:03.054 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:03 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-16T15:49:03.061 INFO:teuthology.nuke:Clearing teuthology firewall rules... 2024-09-16T15:49:03.061 DEBUG:teuthology.orchestra.run.smithi123:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-09-16T15:49:03.199 DEBUG:teuthology.orchestra.run.smithi184:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-09-16T15:49:03.291 INFO:teuthology.nuke:Cleared teuthology firewall rules. 2024-09-16T15:49:03.291 INFO:tasks.cephfs.mount:Terminating background process 2024-09-16T15:49:03.291 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2024-09-16T15:49:03.291 DEBUG:teuthology.orchestra.run.smithi123:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-16T15:49:03.298 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2024-09-16T15:49:03.298 DEBUG:teuthology.orchestra.run.smithi123:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-16T15:49:03.346 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-09-16T15:49:03.525 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:03.523+0000 7f7f0f444640 1 -- 172.21.15.123:0/2958650504 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7ef40039f0 msgr2=0x7f7ef4003df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:03.525 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:03.523+0000 7f7f0f444640 1 --2- 172.21.15.123:0/2958650504 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7ef40039f0 0x7f7ef4003df0 secure :-1 s=READY pgs=1172 cs=0 l=1 rev1=1 crypto rx=0x7f7f040043e0 tx=0x7f7f0401ca60 comp rx=0 tx=0).stop 2024-09-16T15:49:03.525 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:03.523+0000 7f7f0d9e1640 1 -- 172.21.15.123:0/2958650504 reap_dead start 2024-09-16T15:49:03.525 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:03.523+0000 7f7f0f444640 1 -- 172.21.15.123:0/2958650504 shutdown_connections 2024-09-16T15:49:03.525 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:03.523+0000 7f7f0f444640 1 -- 172.21.15.123:0/2958650504 >> 172.21.15.123:0/2958650504 conn(0x7f7f080cfb10 msgr2=0x7f7f080cff20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:49:03.525 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:03.523+0000 7f7f0f444640 1 -- 172.21.15.123:0/2958650504 shutdown_connections 2024-09-16T15:49:03.526 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:03.523+0000 7f7f0f444640 1 -- 172.21.15.123:0/2958650504 wait complete. 2024-09-16T15:49:03.526 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:03.527+0000 7f7f0f444640 1 Processor -- start 2024-09-16T15:49:03.526 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:03.527+0000 7f7f0f444640 1 -- start start 2024-09-16T15:49:03.528 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:03.527+0000 7f7f0f444640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f7ef40039f0 0x7f7f0814b8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:49:03.528 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:03.527+0000 7f7f0f444640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7f080c4f90 0x7f7f080c3e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:49:03.529 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:03.527+0000 7f7f0f444640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7f0814bed0 0x7f7f08163d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:49:03.529 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:03.527+0000 7f7f0f444640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f7f080747d0 con 0x7f7f080c4f90 2024-09-16T15:49:03.529 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:03.527+0000 7f7f0f444640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f7f080cf640 con 0x7f7f0814bed0 2024-09-16T15:49:03.529 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:03.527+0000 7f7f0f444640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f7f08166bc0 con 0x7f7ef40039f0 2024-09-16T15:49:03.529 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:03.527+0000 7f7f0c9df640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7f080c4f90 0x7f7f080c3e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:49:03.529 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:03.527+0000 7f7f0c9df640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7f080c4f90 0x7f7f080c3e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:55366/0 (socket says 172.21.15.123:55366) 2024-09-16T15:49:03.529 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:03.527+0000 7f7f0c9df640 1 -- 172.21.15.123:0/2202800477 learned_addr learned my addr 172.21.15.123:0/2202800477 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:49:03.529 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:03.527+0000 7f7f0d9e1640 1 --2- 172.21.15.123:0/2202800477 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7f0814bed0 0x7f7f08163d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:49:03.529 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:03.527+0000 7f7f0d1e0640 1 --2- 172.21.15.123:0/2202800477 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f7ef40039f0 0x7f7f0814b8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:49:03.529 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:03.527+0000 7f7f0c9df640 1 -- 172.21.15.123:0/2202800477 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f7ef40039f0 msgr2=0x7f7f0814b8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:03.530 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:03.527+0000 7f7f0c9df640 1 --2- 172.21.15.123:0/2202800477 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f7ef40039f0 0x7f7f0814b8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:03.530 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:03.527+0000 7f7f0c9df640 1 -- 172.21.15.123:0/2202800477 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7f0814bed0 msgr2=0x7f7f08163d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:03.530 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:03.527+0000 7f7f0c9df640 1 --2- 172.21.15.123:0/2202800477 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7f0814bed0 0x7f7f08163d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:03.530 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:03.527+0000 7f7f0c9df640 1 -- 172.21.15.123:0/2202800477 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7f081556b0 con 0x7f7f080c4f90 2024-09-16T15:49:03.530 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:03.527+0000 7f7f0c9df640 1 --2- 172.21.15.123:0/2202800477 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7f080c4f90 0x7f7f080c3e40 secure :-1 s=READY pgs=1173 cs=0 l=1 rev1=1 crypto rx=0x7f7ef800d7f0 tx=0x7f7ef800dcb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:49:03.530 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:03.527+0000 7f7efdffb640 1 -- 172.21.15.123:0/2202800477 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ef8018070 con 0x7f7f080c4f90 2024-09-16T15:49:03.530 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:03.527+0000 7f7efdffb640 1 -- 172.21.15.123:0/2202800477 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f7ef80052e0 con 0x7f7f080c4f90 2024-09-16T15:49:03.530 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:03.527+0000 7f7f0f444640 1 -- 172.21.15.123:0/2202800477 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7f08155990 con 0x7f7f080c4f90 2024-09-16T15:49:03.530 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:03.527+0000 7f7efdffb640 1 -- 172.21.15.123:0/2202800477 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ef800b920 con 0x7f7f080c4f90 2024-09-16T15:49:03.530 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:03.527+0000 7f7f0f444640 1 -- 172.21.15.123:0/2202800477 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7f080c3640 con 0x7f7f080c4f90 2024-09-16T15:49:03.530 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:03.527+0000 7f7f0f444640 1 -- 172.21.15.123:0/2202800477 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7f0815e540 con 0x7f7f080c4f90 2024-09-16T15:49:03.531 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:03.531+0000 7f7efdffb640 1 -- 172.21.15.123:0/2202800477 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 18) ==== 97386+0+0 (secure 0 0 0) 0x7f7ef801b070 con 0x7f7f080c4f90 2024-09-16T15:49:03.531 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:03.531+0000 7f7efdffb640 1 --2- 172.21.15.123:0/2202800477 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f7eec073220 0x7f7eec0756e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:49:03.531 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:03.531+0000 7f7efdffb640 1 -- 172.21.15.123:0/2202800477 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(107..107 src has 1..107) ==== 5839+0+0 (secure 0 0 0) 0x7f7ef8093c70 con 0x7f7f080c4f90 2024-09-16T15:49:03.531 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:03.531+0000 7f7f0d1e0640 1 --2- 172.21.15.123:0/2202800477 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f7eec073220 0x7f7eec0756e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:49:03.532 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:03.531+0000 7f7f0d1e0640 1 --2- 172.21.15.123:0/2202800477 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f7eec073220 0x7f7eec0756e0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f7f040ab000 tx=0x7f7f040a7040 comp rx=0 tx=0).ready entity=mgr.4150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:49:03.534 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:03.535+0000 7f7efdffb640 1 -- 172.21.15.123:0/2202800477 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f7ef805e260 con 0x7f7f080c4f90 2024-09-16T15:49:03.722 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:03.723+0000 7f7f0f444640 1 -- 172.21.15.123:0/2202800477 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f7f080cc430 con 0x7f7f080c4f90 2024-09-16T15:49:03.723 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:03.723+0000 7f7efdffb640 1 -- 172.21.15.123:0/2202800477 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v107) ==== 80+0+0 (secure 0 0 0) 0x7f7ef8063e20 con 0x7f7f080c4f90 2024-09-16T15:49:03.723 INFO:teuthology.orchestra.run.smithi123.stderr:listed 0 entries 2024-09-16T15:49:03.726 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:03.727+0000 7f7f0f444640 1 -- 172.21.15.123:0/2202800477 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f7eec073220 msgr2=0x7f7eec0756e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:03.726 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:03.727+0000 7f7f0f444640 1 --2- 172.21.15.123:0/2202800477 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f7eec073220 0x7f7eec0756e0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f7f040ab000 tx=0x7f7f040a7040 comp rx=0 tx=0).stop 2024-09-16T15:49:03.726 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:03.727+0000 7f7f0f444640 1 -- 172.21.15.123:0/2202800477 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7f080c4f90 msgr2=0x7f7f080c3e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:03.726 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:03.727+0000 7f7f0f444640 1 --2- 172.21.15.123:0/2202800477 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7f080c4f90 0x7f7f080c3e40 secure :-1 s=READY pgs=1173 cs=0 l=1 rev1=1 crypto rx=0x7f7ef800d7f0 tx=0x7f7ef800dcb0 comp rx=0 tx=0).stop 2024-09-16T15:49:03.726 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:03.727+0000 7f7f0f444640 1 -- 172.21.15.123:0/2202800477 shutdown_connections 2024-09-16T15:49:03.726 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:03.727+0000 7f7f0f444640 1 --2- 172.21.15.123:0/2202800477 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f7eec073220 0x7f7eec0756e0 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:03.726 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:03.727+0000 7f7f0f444640 1 --2- 172.21.15.123:0/2202800477 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7f0814bed0 0x7f7f08163d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:03.726 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:03.727+0000 7f7f0f444640 1 --2- 172.21.15.123:0/2202800477 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7f080c4f90 0x7f7f080c3e40 unknown :-1 s=CLOSED pgs=1173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:03.726 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:03.727+0000 7f7f0f444640 1 --2- 172.21.15.123:0/2202800477 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f7ef40039f0 0x7f7f0814b8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:03.726 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:03.727+0000 7f7f0f444640 1 -- 172.21.15.123:0/2202800477 >> 172.21.15.123:0/2202800477 conn(0x7f7f080cfb10 msgr2=0x7f7f08075a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:49:03.726 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:03.727+0000 7f7f0f444640 1 -- 172.21.15.123:0/2202800477 shutdown_connections 2024-09-16T15:49:03.726 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:03.727+0000 7f7f0f444640 1 -- 172.21.15.123:0/2202800477 wait complete. 2024-09-16T15:49:03.742 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi123.front.sepia.ceph.com... 2024-09-16T15:49:03.742 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:49:03.742 DEBUG:teuthology.orchestra.run.smithi123:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2024-09-16T15:49:03.753 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi123.stderr:ceph-fuse[45941]: fuse finished with error 0 and tester_r 0 2024-09-16T15:49:03.755 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi123.front.sepia.ceph.com 2024-09-16T15:49:03.755 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:49:03.755 DEBUG:teuthology.orchestra.run.smithi123:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-09-16T15:49:03.921 INFO:tasks.cephfs.mount:Terminating background process 2024-09-16T15:49:03.921 DEBUG:teuthology.orchestra.run.smithi184:> set -ex 2024-09-16T15:49:03.921 DEBUG:teuthology.orchestra.run.smithi184:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-16T15:49:03.928 DEBUG:teuthology.orchestra.run.smithi184:> set -ex 2024-09-16T15:49:03.928 DEBUG:teuthology.orchestra.run.smithi184:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-16T15:49:03.974 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-09-16T15:49:04.109 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:04.107+0000 7fdecf082640 1 -- 172.21.15.123:0/3939935026 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdec80cf5a0 msgr2=0x7fdec806eee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:04.109 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:04.107+0000 7fdecf082640 1 --2- 172.21.15.123:0/3939935026 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdec80cf5a0 0x7fdec806eee0 secure :-1 s=READY pgs=1175 cs=0 l=1 rev1=1 crypto rx=0x7fdec8076280 tx=0x7fdeb801da10 comp rx=0 tx=0).stop 2024-09-16T15:49:04.109 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:04.107+0000 7fdecf082640 1 -- 172.21.15.123:0/3939935026 shutdown_connections 2024-09-16T15:49:04.109 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:04.107+0000 7fdecf082640 1 --2- 172.21.15.123:0/3939935026 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdec80cf5a0 0x7fdec806eee0 unknown :-1 s=CLOSED pgs=1175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:04.109 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:04.107+0000 7fdecf082640 1 --2- 172.21.15.123:0/3939935026 >> v2:172.21.15.184:3300/0 conn(0x7fdec8158120 0x7fdec80ca320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:04.109 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:04.107+0000 7fdecf082640 1 -- 172.21.15.123:0/3939935026 >> 172.21.15.123:0/3939935026 conn(0x7fdec80cfb10 msgr2=0x7fdec80cff20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:49:04.109 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:04.107+0000 7fdecf082640 1 -- 172.21.15.123:0/3939935026 shutdown_connections 2024-09-16T15:49:04.109 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:04.107+0000 7fdecf082640 1 -- 172.21.15.123:0/3939935026 wait complete. 2024-09-16T15:49:04.110 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:04.111+0000 7fdecf082640 1 Processor -- start 2024-09-16T15:49:04.110 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:04.111+0000 7fdecf082640 1 -- start start 2024-09-16T15:49:04.112 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:04.111+0000 7fdecf082640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fdec80cf5a0 0x7fdec80ad8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:49:04.112 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:04.111+0000 7fdecf082640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdec8158120 0x7fdec8159990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:49:04.112 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:04.111+0000 7fdecf082640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fdec815ec40 0x7fdec8072d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:49:04.112 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:04.111+0000 7fdecf082640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fdec815b740 con 0x7fdec8158120 2024-09-16T15:49:04.112 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:04.111+0000 7fdecf082640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fdec815b5c0 con 0x7fdec815ec40 2024-09-16T15:49:04.112 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:04.111+0000 7fdecf082640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fdec81673d0 con 0x7fdec80cf5a0 2024-09-16T15:49:04.112 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:04.111+0000 7fdebffff640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdec8158120 0x7fdec8159990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:49:04.112 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:04.111+0000 7fdebffff640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdec8158120 0x7fdec8159990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:55378/0 (socket says 172.21.15.123:55378) 2024-09-16T15:49:04.112 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:04.111+0000 7fdebffff640 1 -- 172.21.15.123:0/1033821502 learned_addr learned my addr 172.21.15.123:0/1033821502 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:49:04.112 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:04.111+0000 7fdebffff640 1 -- 172.21.15.123:0/1033821502 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fdec80cf5a0 msgr2=0x7fdec80ad8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:04.113 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:04.111+0000 7fdebffff640 1 --2- 172.21.15.123:0/1033821502 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fdec80cf5a0 0x7fdec80ad8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:04.113 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:04.111+0000 7fdebffff640 1 -- 172.21.15.123:0/1033821502 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fdec815ec40 msgr2=0x7fdec8072d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:04.113 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:04.111+0000 7fdecd61f640 1 --2- 172.21.15.123:0/1033821502 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fdec815ec40 0x7fdec8072d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:49:04.113 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:04.111+0000 7fdebffff640 1 --2- 172.21.15.123:0/1033821502 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fdec815ec40 0x7fdec8072d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:04.113 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:04.111+0000 7fdebffff640 1 -- 172.21.15.123:0/1033821502 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdec8159ed0 con 0x7fdec8158120 2024-09-16T15:49:04.113 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:04.111+0000 7fdecd61f640 1 --2- 172.21.15.123:0/1033821502 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fdec815ec40 0x7fdec8072d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T15:49:04.113 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:04.111+0000 7fdebffff640 1 --2- 172.21.15.123:0/1033821502 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdec8158120 0x7fdec8159990 secure :-1 s=READY pgs=1176 cs=0 l=1 rev1=1 crypto rx=0x7fdeb8000c10 tx=0x7fdeb8004220 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:49:04.113 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:04.111+0000 7fdebd7fa640 1 -- 172.21.15.123:0/1033821502 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdeb80bc070 con 0x7fdec8158120 2024-09-16T15:49:04.113 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:04.111+0000 7fdebd7fa640 1 -- 172.21.15.123:0/1033821502 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fdeb8005360 con 0x7fdec8158120 2024-09-16T15:49:04.113 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:04.111+0000 7fdebd7fa640 1 -- 172.21.15.123:0/1033821502 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdeb80b3040 con 0x7fdec8158120 2024-09-16T15:49:04.113 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:04.111+0000 7fdecf082640 1 -- 172.21.15.123:0/1033821502 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdec81f8360 con 0x7fdec8158120 2024-09-16T15:49:04.113 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:04.111+0000 7fdecf082640 1 -- 172.21.15.123:0/1033821502 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdec81f8920 con 0x7fdec8158120 2024-09-16T15:49:04.114 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:04.111+0000 7fdecf082640 1 -- 172.21.15.123:0/1033821502 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdec806eee0 con 0x7fdec8158120 2024-09-16T15:49:04.114 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:04.115+0000 7fdebd7fa640 1 -- 172.21.15.123:0/1033821502 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 18) ==== 97386+0+0 (secure 0 0 0) 0x7fdeb80bf070 con 0x7fdec8158120 2024-09-16T15:49:04.114 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:04.115+0000 7fdebd7fa640 1 --2- 172.21.15.123:0/1033821502 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fdeb4073220 0x7fdeb40756e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:49:04.115 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:04.115+0000 7fdebd7fa640 1 -- 172.21.15.123:0/1033821502 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(107..107 src has 1..107) ==== 5839+0+0 (secure 0 0 0) 0x7fdeb812e3b0 con 0x7fdec8158120 2024-09-16T15:49:04.115 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:04.115+0000 7fdecce1e640 1 --2- 172.21.15.123:0/1033821502 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fdeb4073220 0x7fdeb40756e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:49:04.115 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:04.115+0000 7fdecce1e640 1 --2- 172.21.15.123:0/1033821502 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fdeb4073220 0x7fdeb40756e0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fdec4009730 tx=0x7fdec4006ca0 comp rx=0 tx=0).ready entity=mgr.4150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:49:04.118 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:04.119+0000 7fdebd7fa640 1 -- 172.21.15.123:0/1033821502 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7fdeb8017720 con 0x7fdec8158120 2024-09-16T15:49:04.230 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:04.231+0000 7fdebd7fa640 1 -- 172.21.15.123:0/1033821502 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mgrmap(e 19) ==== 97488+0+0 (secure 0 0 0) 0x7fdeb80f4b60 con 0x7fdec8158120 2024-09-16T15:49:04.304 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:04.303+0000 7fdecf082640 1 -- 172.21.15.123:0/1033821502 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7fdec80d06c0 con 0x7fdec8158120 2024-09-16T15:49:04.304 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:04.303+0000 7fdebd7fa640 1 -- 172.21.15.123:0/1033821502 <== mon.0 v2:172.21.15.123:3300/0 8 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v107) ==== 80+0+0 (secure 0 0 0) 0x7fdeb80f8ff0 con 0x7fdec8158120 2024-09-16T15:49:04.304 INFO:teuthology.orchestra.run.smithi123.stderr:listed 0 entries 2024-09-16T15:49:04.307 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:04.307+0000 7fdecf082640 1 -- 172.21.15.123:0/1033821502 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fdeb4073220 msgr2=0x7fdeb40756e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:04.307 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:04.307+0000 7fdecf082640 1 --2- 172.21.15.123:0/1033821502 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fdeb4073220 0x7fdeb40756e0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fdec4009730 tx=0x7fdec4006ca0 comp rx=0 tx=0).stop 2024-09-16T15:49:04.307 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:04.307+0000 7fdecf082640 1 -- 172.21.15.123:0/1033821502 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdec8158120 msgr2=0x7fdec8159990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:04.307 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:04.307+0000 7fdecf082640 1 --2- 172.21.15.123:0/1033821502 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdec8158120 0x7fdec8159990 secure :-1 s=READY pgs=1176 cs=0 l=1 rev1=1 crypto rx=0x7fdeb8000c10 tx=0x7fdeb8004220 comp rx=0 tx=0).stop 2024-09-16T15:49:04.307 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:04.307+0000 7fdecf082640 1 -- 172.21.15.123:0/1033821502 shutdown_connections 2024-09-16T15:49:04.307 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:04.307+0000 7fdecf082640 1 --2- 172.21.15.123:0/1033821502 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fdeb4073220 0x7fdeb40756e0 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:04.307 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:04.307+0000 7fdecf082640 1 --2- 172.21.15.123:0/1033821502 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fdec815ec40 0x7fdec8072d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:04.308 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:04.307+0000 7fdecf082640 1 --2- 172.21.15.123:0/1033821502 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdec8158120 0x7fdec8159990 unknown :-1 s=CLOSED pgs=1176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:04.308 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:04.307+0000 7fdecf082640 1 --2- 172.21.15.123:0/1033821502 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fdec80cf5a0 0x7fdec80ad8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:04.308 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:04.307+0000 7fdecf082640 1 -- 172.21.15.123:0/1033821502 >> 172.21.15.123:0/1033821502 conn(0x7fdec80cfb10 msgr2=0x7fdec8156df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:49:04.308 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:04.307+0000 7fdecf082640 1 -- 172.21.15.123:0/1033821502 shutdown_connections 2024-09-16T15:49:04.308 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:04.307+0000 7fdecf082640 1 -- 172.21.15.123:0/1033821502 wait complete. 2024-09-16T15:49:04.323 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi184.front.sepia.ceph.com... 2024-09-16T15:49:04.323 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:49:04.323 DEBUG:teuthology.orchestra.run.smithi184:> sudo fusermount -u /home/ubuntu/cephtest/mnt.1 2024-09-16T15:49:04.336 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi184.stderr:ceph-fuse[35733]: fuse finished with error 0 and tester_r 0 2024-09-16T15:49:04.338 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi184.front.sepia.ceph.com 2024-09-16T15:49:04.338 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:49:04.338 DEBUG:teuthology.orchestra.run.smithi184:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2024-09-16T15:49:04.431 INFO:tasks.cephfs.mount:Terminating background process 2024-09-16T15:49:04.432 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2024-09-16T15:49:04.432 DEBUG:teuthology.orchestra.run.smithi123:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-16T15:49:04.438 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16336468k,nr_inodes=4084117,mode=755,inode64 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3274684k,mode=755,inode64 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev,inode64 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=17082 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3274680k,nr_inodes=818670,mode=700,uid=1000,gid=1271,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3274684k,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-09-16T15:49:04.438 INFO:tasks.cephfs.mount:Terminating background process 2024-09-16T15:49:04.438 DEBUG:teuthology.orchestra.run.smithi184:> set -ex 2024-09-16T15:49:04.438 DEBUG:teuthology.orchestra.run.smithi184:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-16T15:49:04.479 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16336488k,nr_inodes=4084122,mode=755,inode64 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3274688k,mode=755,inode64 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev,inode64 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=17834 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3274684k,nr_inodes=818671,mode=700,uid=1000,gid=1271,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3274688k,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-09-16T15:49:04.479 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-16T15:49:04.613 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:04.615+0000 7fd78d491640 1 -- 172.21.15.123:0/1425760204 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd7880cd6e0 msgr2=0x7fd7880cdb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:04.614 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:04.615+0000 7fd78d491640 1 --2- 172.21.15.123:0/1425760204 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd7880cd6e0 0x7fd7880cdb50 secure :-1 s=READY pgs=1177 cs=0 l=1 rev1=1 crypto rx=0x7fd7880708f0 tx=0x7fd77001c830 comp rx=0 tx=0).stop 2024-09-16T15:49:04.614 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:04.615+0000 7fd78d491640 1 -- 172.21.15.123:0/1425760204 shutdown_connections 2024-09-16T15:49:04.614 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:04.615+0000 7fd78d491640 1 --2- 172.21.15.123:0/1425760204 >> v2:172.21.15.184:3300/0 conn(0x7fd78815e820 0x7fd78815ec90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:04.614 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:04.615+0000 7fd78d491640 1 --2- 172.21.15.123:0/1425760204 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd7880cd6e0 0x7fd7880cdb50 unknown :-1 s=CLOSED pgs=1177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:04.614 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:04.615+0000 7fd78d491640 1 -- 172.21.15.123:0/1425760204 >> 172.21.15.123:0/1425760204 conn(0x7fd7880cfb10 msgr2=0x7fd7880cff20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:49:04.614 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:04.615+0000 7fd78d491640 1 -- 172.21.15.123:0/1425760204 shutdown_connections 2024-09-16T15:49:04.614 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:04.615+0000 7fd78d491640 1 -- 172.21.15.123:0/1425760204 wait complete. 2024-09-16T15:49:04.616 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:04.615+0000 7fd78d491640 1 Processor -- start 2024-09-16T15:49:04.616 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:04.615+0000 7fd78d491640 1 -- start start 2024-09-16T15:49:04.616 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:04.615+0000 7fd78d491640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd7880cd6e0 0x7fd78814b370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:49:04.616 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:04.615+0000 7fd78d491640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd78815e160 0x7fd7880b4ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:49:04.616 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:04.615+0000 7fd78d491640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fd78815e820 0x7fd7880b5400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:49:04.617 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:04.615+0000 7fd78d491640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fd7880cf5b0 con 0x7fd7880cd6e0 2024-09-16T15:49:04.617 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:04.615+0000 7fd78d491640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fd7880cf430 con 0x7fd78815e160 2024-09-16T15:49:04.617 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:04.615+0000 7fd78d491640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fd7881683d0 con 0x7fd78815e820 2024-09-16T15:49:04.617 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:04.615+0000 7fd786ffd640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd7880cd6e0 0x7fd78814b370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:49:04.617 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:04.615+0000 7fd786ffd640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd7880cd6e0 0x7fd78814b370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:55384/0 (socket says 172.21.15.123:55384) 2024-09-16T15:49:04.617 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:04.615+0000 7fd786ffd640 1 -- 172.21.15.123:0/68196233 learned_addr learned my addr 172.21.15.123:0/68196233 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:49:04.617 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:04.615+0000 7fd786ffd640 1 -- 172.21.15.123:0/68196233 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fd78815e820 msgr2=0x7fd7880b5400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:04.617 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:04.615+0000 7fd7867fc640 1 --2- 172.21.15.123:0/68196233 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd78815e160 0x7fd7880b4ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:49:04.617 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:04.615+0000 7fd786ffd640 1 --2- 172.21.15.123:0/68196233 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fd78815e820 0x7fd7880b5400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:04.617 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:04.615+0000 7fd786ffd640 1 -- 172.21.15.123:0/68196233 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd78815e160 msgr2=0x7fd7880b4ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:04.617 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:04.615+0000 7fd786ffd640 1 --2- 172.21.15.123:0/68196233 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd78815e160 0x7fd7880b4ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:04.617 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:04.615+0000 7fd786ffd640 1 -- 172.21.15.123:0/68196233 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd78815c280 con 0x7fd7880cd6e0 2024-09-16T15:49:04.618 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:04.615+0000 7fd786ffd640 1 --2- 172.21.15.123:0/68196233 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd7880cd6e0 0x7fd78814b370 secure :-1 s=READY pgs=1178 cs=0 l=1 rev1=1 crypto rx=0x7fd78000c7a0 tx=0x7fd78000cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:49:04.618 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:04.615+0000 7fd77f7fe640 1 -- 172.21.15.123:0/68196233 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd780017070 con 0x7fd7880cd6e0 2024-09-16T15:49:04.618 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:04.615+0000 7fd78d491640 1 -- 172.21.15.123:0/68196233 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd78815c560 con 0x7fd7880cd6e0 2024-09-16T15:49:04.618 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:04.615+0000 7fd77f7fe640 1 -- 172.21.15.123:0/68196233 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fd780005070 con 0x7fd7880cd6e0 2024-09-16T15:49:04.618 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:04.615+0000 7fd77f7fe640 1 -- 172.21.15.123:0/68196233 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd780005500 con 0x7fd7880cd6e0 2024-09-16T15:49:04.618 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:04.615+0000 7fd78d491640 1 -- 172.21.15.123:0/68196233 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd78815cb20 con 0x7fd7880cd6e0 2024-09-16T15:49:04.619 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:04.619+0000 7fd78d491640 1 -- 172.21.15.123:0/68196233 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd7880cdb50 con 0x7fd7880cd6e0 2024-09-16T15:49:04.619 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:04.619+0000 7fd77f7fe640 1 -- 172.21.15.123:0/68196233 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 19) ==== 97488+0+0 (secure 0 0 0) 0x7fd78001a070 con 0x7fd7880cd6e0 2024-09-16T15:49:04.619 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:04.619+0000 7fd77f7fe640 1 --2- 172.21.15.123:0/68196233 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fd76c073370 0x7fd76c075830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:49:04.619 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:04.619+0000 7fd7867fc640 1 --2- 172.21.15.123:0/68196233 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fd76c073370 0x7fd76c075830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:49:04.619 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:04.619+0000 7fd77f7fe640 1 -- 172.21.15.123:0/68196233 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(107..107 src has 1..107) ==== 5839+0+0 (secure 0 0 0) 0x7fd780093e30 con 0x7fd7880cd6e0 2024-09-16T15:49:04.619 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:04.619+0000 7fd7867fc640 1 --2- 172.21.15.123:0/68196233 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fd76c073370 0x7fd76c075830 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fd7700af000 tx=0x7fd77001b070 comp rx=0 tx=0).ready entity=mgr.4150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:49:04.622 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:04.623+0000 7fd77f7fe640 1 -- 172.21.15.123:0/68196233 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7fd78005e8c0 con 0x7fd7880cd6e0 2024-09-16T15:49:04.831 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:04.831+0000 7fd78d491640 1 -- 172.21.15.123:0/68196233 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd7880d06c0 con 0x7fd7880cd6e0 2024-09-16T15:49:04.832 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:04.831+0000 7fd77f7fe640 1 -- 172.21.15.123:0/68196233 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 69 v69) ==== 94+0+5237 (secure 0 0 0) 0x7fd78005e1c0 con 0x7fd7880cd6e0 2024-09-16T15:49:04.832 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-09-16T15:49:04.832 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":69,"btime":"2024-09-16T15:48:37:532986+0000","default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5417,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6835/3411324594","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":3411324594},{"type":"v1","addr":"172.21.15.184:6835","nonce":3411324594}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":63},{"gid":5848,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6833/1488440148","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":1488440148},{"type":"v1","addr":"172.21.15.184:6833","nonce":1488440148}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":63},{"gid":7032,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6833/1841102587","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6832","nonce":1841102587},{"type":"v1","addr":"172.21.15.123:6833","nonce":1841102587}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":63}],"filesystems":[{"mdsmap":{"epoch":68,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-16T15:48:25.813334+0000","modified":"2024-09-16T15:48:28.834708+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7035},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7035":{"gid":7035,"name":"c","rank":0,"incarnation":64,"state":"up:active","state_seq":3,"addr":"172.21.15.123:6835/2302124587","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6834","nonce":2302124587},{"type":"v1","addr":"172.21.15.123:6835","nonce":2302124587}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7035,"qdb_cluster":[7035]},"id":8}]} 2024-09-16T15:49:04.833 INFO:teuthology.orchestra.run.smithi123.stderr:dumped fsmap epoch 69 2024-09-16T15:49:04.835 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:04.835+0000 7fd78d491640 1 -- 172.21.15.123:0/68196233 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fd76c073370 msgr2=0x7fd76c075830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:04.835 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:04.835+0000 7fd78d491640 1 --2- 172.21.15.123:0/68196233 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fd76c073370 0x7fd76c075830 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fd7700af000 tx=0x7fd77001b070 comp rx=0 tx=0).stop 2024-09-16T15:49:04.835 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:04.835+0000 7fd78d491640 1 -- 172.21.15.123:0/68196233 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd7880cd6e0 msgr2=0x7fd78814b370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:04.835 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:04.835+0000 7fd78d491640 1 --2- 172.21.15.123:0/68196233 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd7880cd6e0 0x7fd78814b370 secure :-1 s=READY pgs=1178 cs=0 l=1 rev1=1 crypto rx=0x7fd78000c7a0 tx=0x7fd78000cc60 comp rx=0 tx=0).stop 2024-09-16T15:49:04.835 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:04.835+0000 7fd78d491640 1 -- 172.21.15.123:0/68196233 shutdown_connections 2024-09-16T15:49:04.835 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:04.835+0000 7fd78d491640 1 --2- 172.21.15.123:0/68196233 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fd76c073370 0x7fd76c075830 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:04.835 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:04.835+0000 7fd78d491640 1 --2- 172.21.15.123:0/68196233 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fd78815e820 0x7fd7880b5400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:04.835 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:04.835+0000 7fd78d491640 1 --2- 172.21.15.123:0/68196233 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd78815e160 0x7fd7880b4ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:04.835 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:04.835+0000 7fd78d491640 1 --2- 172.21.15.123:0/68196233 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd7880cd6e0 0x7fd78814b370 unknown :-1 s=CLOSED pgs=1178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:04.835 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:04.835+0000 7fd78d491640 1 -- 172.21.15.123:0/68196233 >> 172.21.15.123:0/68196233 conn(0x7fd7880cfb10 msgr2=0x7fd7880ca800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:49:04.835 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:04.835+0000 7fd78d491640 1 -- 172.21.15.123:0/68196233 shutdown_connections 2024-09-16T15:49:04.835 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:04.835+0000 7fd78d491640 1 -- 172.21.15.123:0/68196233 wait complete. 2024-09-16T15:49:04.852 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-16T15:49:04.976 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:04.975+0000 7fc02bf66640 1 -- 172.21.15.123:0/3910132942 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc02415ac00 msgr2=0x7fc0240702f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:04.976 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:04.975+0000 7fc02bf66640 1 --2- 172.21.15.123:0/3910132942 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc02415ac00 0x7fc0240702f0 secure :-1 s=READY pgs=1179 cs=0 l=1 rev1=1 crypto rx=0x7fc020009c80 tx=0x7fc02001cda0 comp rx=0 tx=0).stop 2024-09-16T15:49:04.977 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:04.975+0000 7fc02bf66640 1 -- 172.21.15.123:0/3910132942 shutdown_connections 2024-09-16T15:49:04.977 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:04.975+0000 7fc02bf66640 1 --2- 172.21.15.123:0/3910132942 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fc0240b3100 0x7fc0240cc1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:04.977 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:04.975+0000 7fc02bf66640 1 --2- 172.21.15.123:0/3910132942 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc02415ac00 0x7fc0240702f0 unknown :-1 s=CLOSED pgs=1179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:04.977 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:04.975+0000 7fc02bf66640 1 -- 172.21.15.123:0/3910132942 >> 172.21.15.123:0/3910132942 conn(0x7fc0240b2610 msgr2=0x7fc0240b29f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:49:04.977 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:04.975+0000 7fc02bf66640 1 -- 172.21.15.123:0/3910132942 shutdown_connections 2024-09-16T15:49:04.977 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:04.975+0000 7fc02bf66640 1 -- 172.21.15.123:0/3910132942 wait complete. 2024-09-16T15:49:04.977 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:04.979+0000 7fc02bf66640 1 Processor -- start 2024-09-16T15:49:04.978 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:04.979+0000 7fc02bf66640 1 -- start start 2024-09-16T15:49:04.979 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:04.979+0000 7fc02bf66640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc024070830 0x7fc024155150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:49:04.979 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:04.979+0000 7fc02bf66640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fc0240b3100 0x7fc024155690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:49:04.980 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:04.979+0000 7fc02bf66640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc02415ac00 0x7fc02415c7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:49:04.980 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:04.979+0000 7fc02bf66640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fc024166650 con 0x7fc02415ac00 2024-09-16T15:49:04.980 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:04.979+0000 7fc02bf66640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fc0241664d0 con 0x7fc024070830 2024-09-16T15:49:04.980 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:04.979+0000 7fc02bf66640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fc0241667d0 con 0x7fc0240b3100 2024-09-16T15:49:04.980 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:04.979+0000 7fc02a503640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc02415ac00 0x7fc02415c7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:49:04.980 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:04.979+0000 7fc02a503640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc02415ac00 0x7fc02415c7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:55392/0 (socket says 172.21.15.123:55392) 2024-09-16T15:49:04.980 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:04.979+0000 7fc02a503640 1 -- 172.21.15.123:0/2525625274 learned_addr learned my addr 172.21.15.123:0/2525625274 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:49:04.980 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:04.979+0000 7fc029501640 1 --2- 172.21.15.123:0/2525625274 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fc0240b3100 0x7fc024155690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:49:04.980 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:04.979+0000 7fc029d02640 1 --2- 172.21.15.123:0/2525625274 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc024070830 0x7fc024155150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:49:04.980 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:04.979+0000 7fc02a503640 1 -- 172.21.15.123:0/2525625274 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fc0240b3100 msgr2=0x7fc024155690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:04.980 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:04.979+0000 7fc02a503640 1 --2- 172.21.15.123:0/2525625274 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fc0240b3100 0x7fc024155690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:04.980 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:04.979+0000 7fc02a503640 1 -- 172.21.15.123:0/2525625274 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc024070830 msgr2=0x7fc024155150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:04.981 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:04.979+0000 7fc02a503640 1 --2- 172.21.15.123:0/2525625274 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc024070830 0x7fc024155150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:04.981 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:04.979+0000 7fc02a503640 1 -- 172.21.15.123:0/2525625274 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc0241f9660 con 0x7fc02415ac00 2024-09-16T15:49:04.981 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:04.979+0000 7fc029d02640 1 --2- 172.21.15.123:0/2525625274 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc024070830 0x7fc024155150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T15:49:04.981 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:04.979+0000 7fc02a503640 1 --2- 172.21.15.123:0/2525625274 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc02415ac00 0x7fc02415c7d0 secure :-1 s=READY pgs=1180 cs=0 l=1 rev1=1 crypto rx=0x7fc01c00c390 tx=0x7fc01c00c850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:49:04.981 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:04.979+0000 7fc0167fc640 1 -- 172.21.15.123:0/2525625274 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc01c011790 con 0x7fc02415ac00 2024-09-16T15:49:04.981 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:04.979+0000 7fc0167fc640 1 -- 172.21.15.123:0/2525625274 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fc01c017070 con 0x7fc02415ac00 2024-09-16T15:49:04.981 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:04.979+0000 7fc02bf66640 1 -- 172.21.15.123:0/2525625274 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc0241f9940 con 0x7fc02415ac00 2024-09-16T15:49:04.981 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:04.979+0000 7fc0167fc640 1 -- 172.21.15.123:0/2525625274 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc01c011cc0 con 0x7fc02415ac00 2024-09-16T15:49:04.981 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:04.979+0000 7fc02bf66640 1 -- 172.21.15.123:0/2525625274 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc0241f9f00 con 0x7fc02415ac00 2024-09-16T15:49:04.982 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:04.979+0000 7fc02bf66640 1 -- 172.21.15.123:0/2525625274 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc0240cc1b0 con 0x7fc02415ac00 2024-09-16T15:49:04.982 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:04.979+0000 7fc0167fc640 1 -- 172.21.15.123:0/2525625274 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 19) ==== 97488+0+0 (secure 0 0 0) 0x7fc01c01c460 con 0x7fc02415ac00 2024-09-16T15:49:04.982 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:04.983+0000 7fc0167fc640 1 --2- 172.21.15.123:0/2525625274 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fc008073340 0x7fc008075800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:49:04.982 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:04.983+0000 7fc029d02640 1 --2- 172.21.15.123:0/2525625274 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fc008073340 0x7fc008075800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:49:04.982 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:04.983+0000 7fc0167fc640 1 -- 172.21.15.123:0/2525625274 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(107..107 src has 1..107) ==== 5839+0+0 (secure 0 0 0) 0x7fc01c094520 con 0x7fc02415ac00 2024-09-16T15:49:04.982 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:04.983+0000 7fc029d02640 1 --2- 172.21.15.123:0/2525625274 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fc008073340 0x7fc008075800 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fc0200af000 tx=0x7fc02001b5e0 comp rx=0 tx=0).ready entity=mgr.4150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:49:04.985 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:04.987+0000 7fc0167fc640 1 -- 172.21.15.123:0/2525625274 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7fc01c05efd0 con 0x7fc02415ac00 2024-09-16T15:49:05.180 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:05.179+0000 7fc02bf66640 1 -- 172.21.15.123:0/2525625274 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc0240b6aa0 con 0x7fc02415ac00 2024-09-16T15:49:05.180 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:05.179+0000 7fc0167fc640 1 -- 172.21.15.123:0/2525625274 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 69 v69) ==== 94+0+5237 (secure 0 0 0) 0x7fc01c05e8d0 con 0x7fc02415ac00 2024-09-16T15:49:05.181 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-09-16T15:49:05.181 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":69,"btime":"2024-09-16T15:48:37:532986+0000","default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5417,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6835/3411324594","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":3411324594},{"type":"v1","addr":"172.21.15.184:6835","nonce":3411324594}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":63},{"gid":5848,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6833/1488440148","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":1488440148},{"type":"v1","addr":"172.21.15.184:6833","nonce":1488440148}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":63},{"gid":7032,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6833/1841102587","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6832","nonce":1841102587},{"type":"v1","addr":"172.21.15.123:6833","nonce":1841102587}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":63}],"filesystems":[{"mdsmap":{"epoch":68,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-16T15:48:25.813334+0000","modified":"2024-09-16T15:48:28.834708+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7035},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7035":{"gid":7035,"name":"c","rank":0,"incarnation":64,"state":"up:active","state_seq":3,"addr":"172.21.15.123:6835/2302124587","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6834","nonce":2302124587},{"type":"v1","addr":"172.21.15.123:6835","nonce":2302124587}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7035,"qdb_cluster":[7035]},"id":8}]} 2024-09-16T15:49:05.181 INFO:teuthology.orchestra.run.smithi123.stderr:dumped fsmap epoch 69 2024-09-16T15:49:05.183 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:05.183+0000 7fc02bf66640 1 -- 172.21.15.123:0/2525625274 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fc008073340 msgr2=0x7fc008075800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:05.184 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:05.183+0000 7fc02bf66640 1 --2- 172.21.15.123:0/2525625274 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fc008073340 0x7fc008075800 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fc0200af000 tx=0x7fc02001b5e0 comp rx=0 tx=0).stop 2024-09-16T15:49:05.184 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:05.183+0000 7fc02bf66640 1 -- 172.21.15.123:0/2525625274 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc02415ac00 msgr2=0x7fc02415c7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:05.184 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:05.183+0000 7fc02bf66640 1 --2- 172.21.15.123:0/2525625274 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc02415ac00 0x7fc02415c7d0 secure :-1 s=READY pgs=1180 cs=0 l=1 rev1=1 crypto rx=0x7fc01c00c390 tx=0x7fc01c00c850 comp rx=0 tx=0).stop 2024-09-16T15:49:05.184 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:05.183+0000 7fc02bf66640 1 -- 172.21.15.123:0/2525625274 shutdown_connections 2024-09-16T15:49:05.184 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:05.183+0000 7fc02bf66640 1 --2- 172.21.15.123:0/2525625274 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fc008073340 0x7fc008075800 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:05.184 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:05.183+0000 7fc02bf66640 1 --2- 172.21.15.123:0/2525625274 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc02415ac00 0x7fc02415c7d0 unknown :-1 s=CLOSED pgs=1180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:05.184 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:05.183+0000 7fc02bf66640 1 --2- 172.21.15.123:0/2525625274 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fc0240b3100 0x7fc024155690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:05.184 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:05.183+0000 7fc02bf66640 1 --2- 172.21.15.123:0/2525625274 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc024070830 0x7fc024155150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:05.184 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:05.183+0000 7fc02bf66640 1 -- 172.21.15.123:0/2525625274 >> 172.21.15.123:0/2525625274 conn(0x7fc0240b2610 msgr2=0x7fc0240b06f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:49:05.184 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:05.183+0000 7fc02bf66640 1 -- 172.21.15.123:0/2525625274 shutdown_connections 2024-09-16T15:49:05.184 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:05.183+0000 7fc02bf66640 1 -- 172.21.15.123:0/2525625274 wait complete. 2024-09-16T15:49:05.199 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-16T15:49:05.319 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:05.319+0000 7fcf49d01640 1 -- 172.21.15.123:0/3857872954 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fcf440cd6e0 msgr2=0x7fcf440cdb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:05.319 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:05.319+0000 7fcf49d01640 1 --2- 172.21.15.123:0/3857872954 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fcf440cd6e0 0x7fcf440cdb50 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7fcf38009840 tx=0x7fcf3801c620 comp rx=0 tx=0).stop 2024-09-16T15:49:05.319 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:05.319+0000 7fcf49d01640 1 -- 172.21.15.123:0/3857872954 shutdown_connections 2024-09-16T15:49:05.319 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:05.319+0000 7fcf49d01640 1 --2- 172.21.15.123:0/3857872954 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fcf4415e820 0x7fcf4415ecb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:05.319 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:05.319+0000 7fcf49d01640 1 --2- 172.21.15.123:0/3857872954 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fcf440cd6e0 0x7fcf440cdb50 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:05.319 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:05.319+0000 7fcf49d01640 1 -- 172.21.15.123:0/3857872954 >> 172.21.15.123:0/3857872954 conn(0x7fcf440cfb10 msgr2=0x7fcf440cff20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:49:05.320 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:05.319+0000 7fcf49d01640 1 -- 172.21.15.123:0/3857872954 shutdown_connections 2024-09-16T15:49:05.320 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:05.319+0000 7fcf49d01640 1 -- 172.21.15.123:0/3857872954 wait complete. 2024-09-16T15:49:05.322 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:05.319+0000 7fcf49d01640 1 Processor -- start 2024-09-16T15:49:05.322 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:05.319+0000 7fcf49d01640 1 -- start start 2024-09-16T15:49:05.322 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:05.319+0000 7fcf49d01640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fcf440cd6e0 0x7fcf4415d260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:49:05.322 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:05.319+0000 7fcf49d01640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fcf4415e160 0x7fcf44156d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:49:05.322 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:05.319+0000 7fcf49d01640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcf4415e820 0x7fcf441555d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:49:05.322 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:05.319+0000 7fcf49d01640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fcf440cf5b0 con 0x7fcf4415e820 2024-09-16T15:49:05.322 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:05.319+0000 7fcf49d01640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fcf440cf430 con 0x7fcf440cd6e0 2024-09-16T15:49:05.322 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:05.319+0000 7fcf49d01640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fcf441683d0 con 0x7fcf4415e160 2024-09-16T15:49:05.322 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:05.319+0000 7fcf43fff640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcf4415e820 0x7fcf441555d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:49:05.322 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:05.319+0000 7fcf43fff640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcf4415e820 0x7fcf441555d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:55406/0 (socket says 172.21.15.123:55406) 2024-09-16T15:49:05.322 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:05.319+0000 7fcf43fff640 1 -- 172.21.15.123:0/1984640217 learned_addr learned my addr 172.21.15.123:0/1984640217 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:49:05.323 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:05.319+0000 7fcf437fe640 1 --2- 172.21.15.123:0/1984640217 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fcf440cd6e0 0x7fcf4415d260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:49:05.323 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:05.319+0000 7fcf42ffd640 1 --2- 172.21.15.123:0/1984640217 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fcf4415e160 0x7fcf44156d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:49:05.323 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:05.323+0000 7fcf43fff640 1 -- 172.21.15.123:0/1984640217 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fcf4415e160 msgr2=0x7fcf44156d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:05.323 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:05.323+0000 7fcf43fff640 1 --2- 172.21.15.123:0/1984640217 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fcf4415e160 0x7fcf44156d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:05.323 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:05.323+0000 7fcf43fff640 1 -- 172.21.15.123:0/1984640217 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fcf440cd6e0 msgr2=0x7fcf4415d260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:05.323 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:05.323+0000 7fcf43fff640 1 --2- 172.21.15.123:0/1984640217 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fcf440cd6e0 0x7fcf4415d260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:05.323 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:05.323+0000 7fcf43fff640 1 -- 172.21.15.123:0/1984640217 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcf44155cb0 con 0x7fcf4415e820 2024-09-16T15:49:05.323 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:05.323+0000 7fcf42ffd640 1 --2- 172.21.15.123:0/1984640217 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fcf4415e160 0x7fcf44156d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T15:49:05.323 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:05.323+0000 7fcf437fe640 1 --2- 172.21.15.123:0/1984640217 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fcf440cd6e0 0x7fcf4415d260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T15:49:05.323 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:05.323+0000 7fcf43fff640 1 --2- 172.21.15.123:0/1984640217 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcf4415e820 0x7fcf441555d0 secure :-1 s=READY pgs=1181 cs=0 l=1 rev1=1 crypto rx=0x7fcf3c00ec70 tx=0x7fcf3c00c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:49:05.323 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:05.323+0000 7fcf2bfff640 1 -- 172.21.15.123:0/1984640217 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf3c003b80 con 0x7fcf4415e820 2024-09-16T15:49:05.323 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:05.323+0000 7fcf49d01640 1 -- 172.21.15.123:0/1984640217 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcf44155f90 con 0x7fcf4415e820 2024-09-16T15:49:05.324 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:05.323+0000 7fcf2bfff640 1 -- 172.21.15.123:0/1984640217 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fcf3c003d20 con 0x7fcf4415e820 2024-09-16T15:49:05.324 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:05.323+0000 7fcf2bfff640 1 -- 172.21.15.123:0/1984640217 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf3c004280 con 0x7fcf4415e820 2024-09-16T15:49:05.324 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:05.323+0000 7fcf49d01640 1 -- 172.21.15.123:0/1984640217 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcf44156550 con 0x7fcf4415e820 2024-09-16T15:49:05.324 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:05.323+0000 7fcf49d01640 1 -- 172.21.15.123:0/1984640217 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcf440cdb50 con 0x7fcf4415e820 2024-09-16T15:49:05.325 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:05.323+0000 7fcf2bfff640 1 -- 172.21.15.123:0/1984640217 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 19) ==== 97488+0+0 (secure 0 0 0) 0x7fcf3c004420 con 0x7fcf4415e820 2024-09-16T15:49:05.325 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:05.323+0000 7fcf2bfff640 1 --2- 172.21.15.123:0/1984640217 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fcf30073340 0x7fcf30075800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:49:05.325 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:05.323+0000 7fcf437fe640 1 --2- 172.21.15.123:0/1984640217 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fcf30073340 0x7fcf30075800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:49:05.325 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:05.323+0000 7fcf2bfff640 1 -- 172.21.15.123:0/1984640217 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(107..107 src has 1..107) ==== 5839+0+0 (secure 0 0 0) 0x7fcf3c095630 con 0x7fcf4415e820 2024-09-16T15:49:05.325 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:05.323+0000 7fcf437fe640 1 --2- 172.21.15.123:0/1984640217 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fcf30073340 0x7fcf30075800 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fcf44158d70 tx=0x7fcf34009040 comp rx=0 tx=0).ready entity=mgr.4150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:49:05.327 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:05.327+0000 7fcf2bfff640 1 -- 172.21.15.123:0/1984640217 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7fcf3c060160 con 0x7fcf4415e820 2024-09-16T15:49:05.500 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:05.499+0000 7fcf49d01640 1 -- 172.21.15.123:0/1984640217 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fcf440c32c0 con 0x7fcf4415e820 2024-09-16T15:49:05.502 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:05.503+0000 7fcf2bfff640 1 -- 172.21.15.123:0/1984640217 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v107) ==== 74+0+17033 (secure 0 0 0) 0x7fcf3c05fa60 con 0x7fcf4415e820 2024-09-16T15:49:05.503 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-09-16T15:49:05.503 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":107,"fsid":"56077f89-18c8-486f-ae7e-eecc0bb7822c","created":"2024-09-16T15:42:26.364892+0000","modified":"2024-09-16T15:49:03.198103+0000","last_up_change":"2024-09-16T15:42:38.707234+0000","last_in_change":"2024-09-16T15:42:29.423662+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":17,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-16T15:42:39.287764+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_metadata","create_time":"2024-09-16T15:48:23.417905+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"102","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":17,"pool_name":"cephfs_data","create_time":"2024-09-16T15:48:24.776060+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"107","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":5,"snap_epoch":107,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9e7c1955-a607-4671-9be7-f2ef760ceeaa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6800","nonce":3882044969},{"type":"v1","addr":"172.21.15.123:6801","nonce":3882044969}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6802","nonce":3882044969},{"type":"v1","addr":"172.21.15.123:6803","nonce":3882044969}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6806","nonce":3882044969},{"type":"v1","addr":"172.21.15.123:6807","nonce":3882044969}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6804","nonce":3882044969},{"type":"v1","addr":"172.21.15.123:6805","nonce":3882044969}]},"public_addr":"172.21.15.123:6801/3882044969","cluster_addr":"172.21.15.123:6803/3882044969","heartbeat_back_addr":"172.21.15.123:6807/3882044969","heartbeat_front_addr":"172.21.15.123:6805/3882044969","state":["exists","up"]},{"osd":1,"uuid":"c540cfde-0977-4742-9b17-fd634668e010","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6816","nonce":4037001850},{"type":"v1","addr":"172.21.15.123:6817","nonce":4037001850}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6818","nonce":4037001850},{"type":"v1","addr":"172.21.15.123:6819","nonce":4037001850}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6822","nonce":4037001850},{"type":"v1","addr":"172.21.15.123:6823","nonce":4037001850}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6820","nonce":4037001850},{"type":"v1","addr":"172.21.15.123:6821","nonce":4037001850}]},"public_addr":"172.21.15.123:6817/4037001850","cluster_addr":"172.21.15.123:6819/4037001850","heartbeat_back_addr":"172.21.15.123:6823/4037001850","heartbeat_front_addr":"172.21.15.123:6821/4037001850","state":["exists","up"]},{"osd":2,"uuid":"9864a496-f169-486b-95c4-1d59162793a7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6824","nonce":1332143088},{"type":"v1","addr":"172.21.15.123:6825","nonce":1332143088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6826","nonce":1332143088},{"type":"v1","addr":"172.21.15.123:6827","nonce":1332143088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6830","nonce":1332143088},{"type":"v1","addr":"172.21.15.123:6831","nonce":1332143088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6828","nonce":1332143088},{"type":"v1","addr":"172.21.15.123:6829","nonce":1332143088}]},"public_addr":"172.21.15.123:6825/1332143088","cluster_addr":"172.21.15.123:6827/1332143088","heartbeat_back_addr":"172.21.15.123:6831/1332143088","heartbeat_front_addr":"172.21.15.123:6829/1332143088","state":["exists","up"]},{"osd":3,"uuid":"c6d1f817-f074-4325-95e1-578eba13edda","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6808","nonce":988586998},{"type":"v1","addr":"172.21.15.123:6809","nonce":988586998}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6810","nonce":988586998},{"type":"v1","addr":"172.21.15.123:6811","nonce":988586998}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6814","nonce":988586998},{"type":"v1","addr":"172.21.15.123:6815","nonce":988586998}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6812","nonce":988586998},{"type":"v1","addr":"172.21.15.123:6813","nonce":988586998}]},"public_addr":"172.21.15.123:6809/988586998","cluster_addr":"172.21.15.123:6811/988586998","heartbeat_back_addr":"172.21.15.123:6815/988586998","heartbeat_front_addr":"172.21.15.123:6813/988586998","state":["exists","up"]},{"osd":4,"uuid":"200a8df4-50ef-40a5-b48a-58981269c086","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6816","nonce":3387813890},{"type":"v1","addr":"172.21.15.184:6817","nonce":3387813890}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6818","nonce":3387813890},{"type":"v1","addr":"172.21.15.184:6819","nonce":3387813890}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6822","nonce":3387813890},{"type":"v1","addr":"172.21.15.184:6823","nonce":3387813890}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6820","nonce":3387813890},{"type":"v1","addr":"172.21.15.184:6821","nonce":3387813890}]},"public_addr":"172.21.15.184:6817/3387813890","cluster_addr":"172.21.15.184:6819/3387813890","heartbeat_back_addr":"172.21.15.184:6823/3387813890","heartbeat_front_addr":"172.21.15.184:6821/3387813890","state":["exists","up"]},{"osd":5,"uuid":"32d5a900-da3d-412c-bc19-13861a2368d8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":104,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6808","nonce":2680916017},{"type":"v1","addr":"172.21.15.184:6809","nonce":2680916017}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6810","nonce":2680916017},{"type":"v1","addr":"172.21.15.184:6811","nonce":2680916017}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6814","nonce":2680916017},{"type":"v1","addr":"172.21.15.184:6815","nonce":2680916017}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6812","nonce":2680916017},{"type":"v1","addr":"172.21.15.184:6813","nonce":2680916017}]},"public_addr":"172.21.15.184:6809/2680916017","cluster_addr":"172.21.15.184:6811/2680916017","heartbeat_back_addr":"172.21.15.184:6815/2680916017","heartbeat_front_addr":"172.21.15.184:6813/2680916017","state":["exists","up"]},{"osd":6,"uuid":"cde6f39e-aa1b-4260-ac19-774ee6aaca72","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":104,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6824","nonce":2183120457},{"type":"v1","addr":"172.21.15.184:6825","nonce":2183120457}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6826","nonce":2183120457},{"type":"v1","addr":"172.21.15.184:6827","nonce":2183120457}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6830","nonce":2183120457},{"type":"v1","addr":"172.21.15.184:6831","nonce":2183120457}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6828","nonce":2183120457},{"type":"v1","addr":"172.21.15.184:6829","nonce":2183120457}]},"public_addr":"172.21.15.184:6825/2183120457","cluster_addr":"172.21.15.184:6827/2183120457","heartbeat_back_addr":"172.21.15.184:6831/2183120457","heartbeat_front_addr":"172.21.15.184:6829/2183120457","state":["exists","up"]},{"osd":7,"uuid":"4f381778-eee1-47c6-804f-df0e6b33e313","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6800","nonce":4102957285},{"type":"v1","addr":"172.21.15.184:6801","nonce":4102957285}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6802","nonce":4102957285},{"type":"v1","addr":"172.21.15.184:6803","nonce":4102957285}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6806","nonce":4102957285},{"type":"v1","addr":"172.21.15.184:6807","nonce":4102957285}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6804","nonce":4102957285},{"type":"v1","addr":"172.21.15.184:6805","nonce":4102957285}]},"public_addr":"172.21.15.184:6801/4102957285","cluster_addr":"172.21.15.184:6803/4102957285","heartbeat_back_addr":"172.21.15.184:6807/4102957285","heartbeat_front_addr":"172.21.15.184:6805/4102957285","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:35.972210+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.222007+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.198617+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.385663+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.358935+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.156066+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.752973+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:35.682846+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"17.5","mappings":[{"from":0,"to":2}]},{"pgid":"17.b","mappings":[{"from":0,"to":5}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":17,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[{"pool":17,"snaps":[{"begin":2,"length":1},{"begin":5,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T15:49:05.505 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:05.503+0000 7fcf49d01640 1 -- 172.21.15.123:0/1984640217 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fcf30073340 msgr2=0x7fcf30075800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:05.505 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:05.503+0000 7fcf49d01640 1 --2- 172.21.15.123:0/1984640217 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fcf30073340 0x7fcf30075800 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fcf44158d70 tx=0x7fcf34009040 comp rx=0 tx=0).stop 2024-09-16T15:49:05.505 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:05.503+0000 7fcf49d01640 1 -- 172.21.15.123:0/1984640217 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcf4415e820 msgr2=0x7fcf441555d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:05.505 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:05.503+0000 7fcf49d01640 1 --2- 172.21.15.123:0/1984640217 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcf4415e820 0x7fcf441555d0 secure :-1 s=READY pgs=1181 cs=0 l=1 rev1=1 crypto rx=0x7fcf3c00ec70 tx=0x7fcf3c00c6a0 comp rx=0 tx=0).stop 2024-09-16T15:49:05.505 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:05.503+0000 7fcf49d01640 1 -- 172.21.15.123:0/1984640217 shutdown_connections 2024-09-16T15:49:05.506 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:05.503+0000 7fcf49d01640 1 --2- 172.21.15.123:0/1984640217 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fcf30073340 0x7fcf30075800 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:05.506 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:05.503+0000 7fcf49d01640 1 --2- 172.21.15.123:0/1984640217 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcf4415e820 0x7fcf441555d0 unknown :-1 s=CLOSED pgs=1181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:05.506 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:05.503+0000 7fcf49d01640 1 --2- 172.21.15.123:0/1984640217 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fcf4415e160 0x7fcf44156d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:05.506 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:05.503+0000 7fcf49d01640 1 --2- 172.21.15.123:0/1984640217 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fcf440cd6e0 0x7fcf4415d260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:05.506 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:05.503+0000 7fcf49d01640 1 -- 172.21.15.123:0/1984640217 >> 172.21.15.123:0/1984640217 conn(0x7fcf440cfb10 msgr2=0x7fcf4406eee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:49:05.506 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:05.503+0000 7fcf49d01640 1 -- 172.21.15.123:0/1984640217 shutdown_connections 2024-09-16T15:49:05.506 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:05.507+0000 7fcf49d01640 1 -- 172.21.15.123:0/1984640217 wait complete. 2024-09-16T15:49:05.520 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-09-16T15:49:05.520 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-16T15:49:05.634 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:05.635+0000 7f2262afe640 1 -- 172.21.15.123:0/1386065222 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2248003a10 msgr2=0x7f2248003e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:05.634 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:05.635+0000 7f2262afe640 1 --2- 172.21.15.123:0/1386065222 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2248003a10 0x7f2248003e30 secure :-1 s=READY pgs=1183 cs=0 l=1 rev1=1 crypto rx=0x7f225400c310 tx=0x7f225401cf90 comp rx=0 tx=0).stop 2024-09-16T15:49:05.634 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:05.635+0000 7f226109b640 1 -- 172.21.15.123:0/1386065222 reap_dead start 2024-09-16T15:49:05.634 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:05.635+0000 7f2262afe640 1 -- 172.21.15.123:0/1386065222 shutdown_connections 2024-09-16T15:49:05.634 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:05.635+0000 7f2262afe640 1 -- 172.21.15.123:0/1386065222 >> 172.21.15.123:0/1386065222 conn(0x7f225c0cfb10 msgr2=0x7f225c0cff20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:49:05.634 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:05.635+0000 7f2262afe640 1 -- 172.21.15.123:0/1386065222 shutdown_connections 2024-09-16T15:49:05.634 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:05.635+0000 7f2262afe640 1 -- 172.21.15.123:0/1386065222 wait complete. 2024-09-16T15:49:05.635 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:05.635+0000 7f2262afe640 1 Processor -- start 2024-09-16T15:49:05.635 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:05.635+0000 7f2262afe640 1 -- start start 2024-09-16T15:49:05.637 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:05.635+0000 7f2262afe640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f2248003a10 0x7f225c0b4a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:49:05.637 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:05.635+0000 7f2262afe640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f225c0b4280 0x7f225c147eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:49:05.637 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:05.635+0000 7f2262afe640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f225c152950 0x7f225c1561c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:49:05.637 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:05.635+0000 7f2262afe640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f225c06faa0 con 0x7f225c152950 2024-09-16T15:49:05.637 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:05.635+0000 7f2262afe640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f225c06f920 con 0x7f225c0b4280 2024-09-16T15:49:05.637 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:05.635+0000 7f2262afe640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f225c166a40 con 0x7f2248003a10 2024-09-16T15:49:05.637 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:05.635+0000 7f226109b640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f225c152950 0x7f225c1561c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:49:05.637 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:05.635+0000 7f226109b640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f225c152950 0x7f225c1561c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:55416/0 (socket says 172.21.15.123:55416) 2024-09-16T15:49:05.637 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:05.635+0000 7f226109b640 1 -- 172.21.15.123:0/2432004963 learned_addr learned my addr 172.21.15.123:0/2432004963 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:49:05.637 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:05.635+0000 7f226109b640 1 -- 172.21.15.123:0/2432004963 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f2248003a10 msgr2=0x7f225c0b4a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:05.638 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:05.635+0000 7f226109b640 1 --2- 172.21.15.123:0/2432004963 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f2248003a10 0x7f225c0b4a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:05.638 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:05.635+0000 7f226109b640 1 -- 172.21.15.123:0/2432004963 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f225c0b4280 msgr2=0x7f225c147eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:05.638 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:05.635+0000 7f226109b640 1 --2- 172.21.15.123:0/2432004963 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f225c0b4280 0x7f225c147eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:05.638 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:05.635+0000 7f226109b640 1 -- 172.21.15.123:0/2432004963 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f225c1f9050 con 0x7f225c152950 2024-09-16T15:49:05.638 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:05.635+0000 7f226109b640 1 --2- 172.21.15.123:0/2432004963 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f225c152950 0x7f225c1561c0 secure :-1 s=READY pgs=1184 cs=0 l=1 rev1=1 crypto rx=0x7f225c1602a0 tx=0x7f224c00da20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:49:05.638 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:05.635+0000 7f22597fa640 1 -- 172.21.15.123:0/2432004963 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f224c005020 con 0x7f225c152950 2024-09-16T15:49:05.638 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:05.635+0000 7f22597fa640 1 -- 172.21.15.123:0/2432004963 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f224c0052b0 con 0x7f225c152950 2024-09-16T15:49:05.638 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:05.635+0000 7f2262afe640 1 -- 172.21.15.123:0/2432004963 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f225c0b4f60 con 0x7f225c152950 2024-09-16T15:49:05.638 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:05.635+0000 7f22597fa640 1 -- 172.21.15.123:0/2432004963 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f224c0047b0 con 0x7f225c152950 2024-09-16T15:49:05.638 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:05.635+0000 7f2262afe640 1 -- 172.21.15.123:0/2432004963 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f225c1f95d0 con 0x7f225c152950 2024-09-16T15:49:05.639 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:05.639+0000 7f2262afe640 1 -- 172.21.15.123:0/2432004963 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f225c06e750 con 0x7f225c152950 2024-09-16T15:49:05.639 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:05.639+0000 7f22597fa640 1 -- 172.21.15.123:0/2432004963 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 19) ==== 97488+0+0 (secure 0 0 0) 0x7f224c01b020 con 0x7f225c152950 2024-09-16T15:49:05.639 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:05.639+0000 7f22597fa640 1 --2- 172.21.15.123:0/2432004963 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f223c073340 0x7f223c075800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:49:05.639 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:05.639+0000 7f22597fa640 1 -- 172.21.15.123:0/2432004963 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(107..107 src has 1..107) ==== 5839+0+0 (secure 0 0 0) 0x7f224c093fc0 con 0x7f225c152950 2024-09-16T15:49:05.639 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:05.639+0000 7f226089a640 1 --2- 172.21.15.123:0/2432004963 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f223c073340 0x7f223c075800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:49:05.640 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:05.639+0000 7f226089a640 1 --2- 172.21.15.123:0/2432004963 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f223c073340 0x7f223c075800 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f22540ac000 tx=0x7f2248005840 comp rx=0 tx=0).ready entity=mgr.4150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:49:05.642 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:05.643+0000 7f22597fa640 1 -- 172.21.15.123:0/2432004963 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f224c05eaf0 con 0x7f225c152950 2024-09-16T15:49:05.844 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:05.843+0000 7f2262afe640 1 -- 172.21.15.123:0/2432004963 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f225c15f630 con 0x7f225c152950 2024-09-16T15:49:05.845 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:05.847+0000 7f22597fa640 1 -- 172.21.15.123:0/2432004963 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 69 v69) ==== 94+0+5237 (secure 0 0 0) 0x7f224c05e3f0 con 0x7f225c152950 2024-09-16T15:49:05.845 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-09-16T15:49:05.846 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":69,"btime":"2024-09-16T15:48:37:532986+0000","default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5417,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6835/3411324594","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":3411324594},{"type":"v1","addr":"172.21.15.184:6835","nonce":3411324594}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":63},{"gid":5848,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6833/1488440148","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":1488440148},{"type":"v1","addr":"172.21.15.184:6833","nonce":1488440148}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":63},{"gid":7032,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6833/1841102587","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6832","nonce":1841102587},{"type":"v1","addr":"172.21.15.123:6833","nonce":1841102587}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":63}],"filesystems":[{"mdsmap":{"epoch":68,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-16T15:48:25.813334+0000","modified":"2024-09-16T15:48:28.834708+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7035},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7035":{"gid":7035,"name":"c","rank":0,"incarnation":64,"state":"up:active","state_seq":3,"addr":"172.21.15.123:6835/2302124587","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6834","nonce":2302124587},{"type":"v1","addr":"172.21.15.123:6835","nonce":2302124587}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7035,"qdb_cluster":[7035]},"id":8}]} 2024-09-16T15:49:05.846 INFO:teuthology.orchestra.run.smithi123.stderr:dumped fsmap epoch 69 2024-09-16T15:49:05.847 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:05.847+0000 7f2262afe640 1 -- 172.21.15.123:0/2432004963 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f223c073340 msgr2=0x7f223c075800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:05.848 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:05.847+0000 7f2262afe640 1 --2- 172.21.15.123:0/2432004963 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f223c073340 0x7f223c075800 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f22540ac000 tx=0x7f2248005840 comp rx=0 tx=0).stop 2024-09-16T15:49:05.848 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:05.847+0000 7f2262afe640 1 -- 172.21.15.123:0/2432004963 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f225c152950 msgr2=0x7f225c1561c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:05.848 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:05.847+0000 7f2262afe640 1 --2- 172.21.15.123:0/2432004963 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f225c152950 0x7f225c1561c0 secure :-1 s=READY pgs=1184 cs=0 l=1 rev1=1 crypto rx=0x7f225c1602a0 tx=0x7f224c00da20 comp rx=0 tx=0).stop 2024-09-16T15:49:05.848 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:05.847+0000 7f2262afe640 1 -- 172.21.15.123:0/2432004963 shutdown_connections 2024-09-16T15:49:05.848 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:05.847+0000 7f2262afe640 1 --2- 172.21.15.123:0/2432004963 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f223c073340 0x7f223c075800 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:05.848 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:05.847+0000 7f2262afe640 1 --2- 172.21.15.123:0/2432004963 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f225c152950 0x7f225c1561c0 unknown :-1 s=CLOSED pgs=1184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:05.848 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:05.847+0000 7f2262afe640 1 --2- 172.21.15.123:0/2432004963 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f225c0b4280 0x7f225c147eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:05.848 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:05.847+0000 7f2262afe640 1 --2- 172.21.15.123:0/2432004963 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f2248003a10 0x7f225c0b4a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:05.848 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:05.847+0000 7f2262afe640 1 -- 172.21.15.123:0/2432004963 >> 172.21.15.123:0/2432004963 conn(0x7f225c0cfb10 msgr2=0x7f225c1579e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:49:05.848 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:05.847+0000 7f2262afe640 1 -- 172.21.15.123:0/2432004963 shutdown_connections 2024-09-16T15:49:05.848 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:05.847+0000 7f2262afe640 1 -- 172.21.15.123:0/2432004963 wait complete. 2024-09-16T15:49:05.863 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-16T15:49:05.990 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:05.991+0000 7f60621b0640 1 -- 172.21.15.123:0/3781240885 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f605c0b25a0 msgr2=0x7f605c0d1c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:05.990 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:05.991+0000 7f60621b0640 1 --2- 172.21.15.123:0/3781240885 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f605c0b25a0 0x7f605c0d1c10 secure :-1 s=READY pgs=1185 cs=0 l=1 rev1=1 crypto rx=0x7f6044009930 tx=0x7f604401c7f0 comp rx=0 tx=0).stop 2024-09-16T15:49:05.990 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:05.991+0000 7f60621b0640 1 -- 172.21.15.123:0/3781240885 shutdown_connections 2024-09-16T15:49:05.990 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:05.991+0000 7f60621b0640 1 --2- 172.21.15.123:0/3781240885 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f605c0d2150 0x7f605c0ccab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:05.991 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:05.991+0000 7f60621b0640 1 --2- 172.21.15.123:0/3781240885 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f605c0b25a0 0x7f605c0d1c10 unknown :-1 s=CLOSED pgs=1185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:05.991 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:05.991+0000 7f60621b0640 1 -- 172.21.15.123:0/3781240885 >> 172.21.15.123:0/3781240885 conn(0x7f605c074bd0 msgr2=0x7f605c09aa40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:49:05.991 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:05.991+0000 7f60621b0640 1 -- 172.21.15.123:0/3781240885 shutdown_connections 2024-09-16T15:49:05.991 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:05.991+0000 7f60621b0640 1 -- 172.21.15.123:0/3781240885 wait complete. 2024-09-16T15:49:05.991 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:05.991+0000 7f60621b0640 1 Processor -- start 2024-09-16T15:49:05.991 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:05.991+0000 7f60621b0640 1 -- start start 2024-09-16T15:49:05.994 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:05.991+0000 7f60621b0640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f605c0b25a0 0x7f605c154bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:49:05.994 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:05.991+0000 7f60621b0640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f605c0d2150 0x7f605c153240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:49:05.994 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:05.991+0000 7f60621b0640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f605c15bda0 0x7f605c0ca790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:49:05.994 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:05.991+0000 7f60621b0640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f605c168a80 con 0x7f605c0d2150 2024-09-16T15:49:05.994 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:05.991+0000 7f60621b0640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f605c0cddd0 con 0x7f605c0b25a0 2024-09-16T15:49:05.994 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:05.991+0000 7f60621b0640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f605c168c00 con 0x7f605c15bda0 2024-09-16T15:49:05.994 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:05.991+0000 7f605affd640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f605c0d2150 0x7f605c153240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:49:05.994 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:05.991+0000 7f605affd640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f605c0d2150 0x7f605c153240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:55426/0 (socket says 172.21.15.123:55426) 2024-09-16T15:49:05.994 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:05.991+0000 7f605affd640 1 -- 172.21.15.123:0/349213070 learned_addr learned my addr 172.21.15.123:0/349213070 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:49:05.994 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:05.991+0000 7f605b7fe640 1 --2- 172.21.15.123:0/349213070 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f605c0b25a0 0x7f605c154bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:49:05.994 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:05.991+0000 7f605bfff640 1 --2- 172.21.15.123:0/349213070 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f605c15bda0 0x7f605c0ca790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:49:05.994 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:05.991+0000 7f605affd640 1 -- 172.21.15.123:0/349213070 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f605c15bda0 msgr2=0x7f605c0ca790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:05.994 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:05.991+0000 7f605affd640 1 --2- 172.21.15.123:0/349213070 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f605c15bda0 0x7f605c0ca790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:05.995 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:05.991+0000 7f605affd640 1 -- 172.21.15.123:0/349213070 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f605c0b25a0 msgr2=0x7f605c154bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:05.995 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:05.991+0000 7f605affd640 1 --2- 172.21.15.123:0/349213070 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f605c0b25a0 0x7f605c154bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:05.995 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:05.991+0000 7f605affd640 1 -- 172.21.15.123:0/349213070 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f605c0cb090 con 0x7f605c0d2150 2024-09-16T15:49:05.995 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:05.991+0000 7f605b7fe640 1 --2- 172.21.15.123:0/349213070 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f605c0b25a0 0x7f605c154bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T15:49:05.995 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:05.991+0000 7f605affd640 1 --2- 172.21.15.123:0/349213070 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f605c0d2150 0x7f605c153240 secure :-1 s=READY pgs=1186 cs=0 l=1 rev1=1 crypto rx=0x7f604401af00 tx=0x7f60440a6590 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:49:05.995 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:05.991+0000 7f6053fff640 1 -- 172.21.15.123:0/349213070 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60440ac070 con 0x7f605c0d2150 2024-09-16T15:49:05.995 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:05.991+0000 7f6053fff640 1 -- 172.21.15.123:0/349213070 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f6044004340 con 0x7f605c0d2150 2024-09-16T15:49:05.995 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:05.991+0000 7f60621b0640 1 -- 172.21.15.123:0/349213070 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f605c1582e0 con 0x7f605c0d2150 2024-09-16T15:49:05.995 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:05.991+0000 7f6053fff640 1 -- 172.21.15.123:0/349213070 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60440053a0 con 0x7f605c0d2150 2024-09-16T15:49:05.995 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:05.991+0000 7f60621b0640 1 -- 172.21.15.123:0/349213070 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f605c1588a0 con 0x7f605c0d2150 2024-09-16T15:49:05.995 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:05.995+0000 7f60621b0640 1 -- 172.21.15.123:0/349213070 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f605c09fbd0 con 0x7f605c0d2150 2024-09-16T15:49:05.996 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:05.995+0000 7f6053fff640 1 -- 172.21.15.123:0/349213070 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 19) ==== 97488+0+0 (secure 0 0 0) 0x7f60440af070 con 0x7f605c0d2150 2024-09-16T15:49:05.996 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:05.995+0000 7f6053fff640 1 --2- 172.21.15.123:0/349213070 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f6040073340 0x7f6040075800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:49:05.996 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:05.995+0000 7f6053fff640 1 -- 172.21.15.123:0/349213070 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(107..107 src has 1..107) ==== 5839+0+0 (secure 0 0 0) 0x7f604412ecf0 con 0x7f605c0d2150 2024-09-16T15:49:05.996 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:05.995+0000 7f605b7fe640 1 --2- 172.21.15.123:0/349213070 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f6040073340 0x7f6040075800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:49:05.996 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:05.995+0000 7f605b7fe640 1 --2- 172.21.15.123:0/349213070 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f6040073340 0x7f6040075800 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f604c007eb0 tx=0x7f604c0073c0 comp rx=0 tx=0).ready entity=mgr.4150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:49:05.999 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:05.999+0000 7f6053fff640 1 -- 172.21.15.123:0/349213070 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f60440164d0 con 0x7f605c0d2150 2024-09-16T15:49:06.198 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:06.199+0000 7f60621b0640 1 -- 172.21.15.123:0/349213070 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f605c0cee20 con 0x7f605c0d2150 2024-09-16T15:49:06.199 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:06.199+0000 7f6053fff640 1 -- 172.21.15.123:0/349213070 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 69 v69) ==== 94+0+5237 (secure 0 0 0) 0x7f60440f9940 con 0x7f605c0d2150 2024-09-16T15:49:06.200 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-09-16T15:49:06.200 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":69,"btime":"2024-09-16T15:48:37:532986+0000","default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5417,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6835/3411324594","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":3411324594},{"type":"v1","addr":"172.21.15.184:6835","nonce":3411324594}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":63},{"gid":5848,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6833/1488440148","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":1488440148},{"type":"v1","addr":"172.21.15.184:6833","nonce":1488440148}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":63},{"gid":7032,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6833/1841102587","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6832","nonce":1841102587},{"type":"v1","addr":"172.21.15.123:6833","nonce":1841102587}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":63}],"filesystems":[{"mdsmap":{"epoch":68,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-16T15:48:25.813334+0000","modified":"2024-09-16T15:48:28.834708+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7035},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7035":{"gid":7035,"name":"c","rank":0,"incarnation":64,"state":"up:active","state_seq":3,"addr":"172.21.15.123:6835/2302124587","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6834","nonce":2302124587},{"type":"v1","addr":"172.21.15.123:6835","nonce":2302124587}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7035,"qdb_cluster":[7035]},"id":8}]} 2024-09-16T15:49:06.200 INFO:teuthology.orchestra.run.smithi123.stderr:dumped fsmap epoch 69 2024-09-16T15:49:06.202 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:06.203+0000 7f60621b0640 1 -- 172.21.15.123:0/349213070 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f6040073340 msgr2=0x7f6040075800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:06.202 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:06.203+0000 7f60621b0640 1 --2- 172.21.15.123:0/349213070 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f6040073340 0x7f6040075800 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f604c007eb0 tx=0x7f604c0073c0 comp rx=0 tx=0).stop 2024-09-16T15:49:06.202 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:06.203+0000 7f60621b0640 1 -- 172.21.15.123:0/349213070 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f605c0d2150 msgr2=0x7f605c153240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:06.202 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:06.203+0000 7f60621b0640 1 --2- 172.21.15.123:0/349213070 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f605c0d2150 0x7f605c153240 secure :-1 s=READY pgs=1186 cs=0 l=1 rev1=1 crypto rx=0x7f604401af00 tx=0x7f60440a6590 comp rx=0 tx=0).stop 2024-09-16T15:49:06.202 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:06.203+0000 7f60621b0640 1 -- 172.21.15.123:0/349213070 shutdown_connections 2024-09-16T15:49:06.202 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:06.203+0000 7f60621b0640 1 --2- 172.21.15.123:0/349213070 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f6040073340 0x7f6040075800 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:06.202 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:06.203+0000 7f60621b0640 1 --2- 172.21.15.123:0/349213070 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f605c15bda0 0x7f605c0ca790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:06.203 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:06.203+0000 7f60621b0640 1 --2- 172.21.15.123:0/349213070 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f605c0d2150 0x7f605c153240 unknown :-1 s=CLOSED pgs=1186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:06.203 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:06.203+0000 7f60621b0640 1 --2- 172.21.15.123:0/349213070 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f605c0b25a0 0x7f605c154bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:06.203 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:06.203+0000 7f60621b0640 1 -- 172.21.15.123:0/349213070 >> 172.21.15.123:0/349213070 conn(0x7f605c074bd0 msgr2=0x7f605c0b02d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:49:06.203 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:06.203+0000 7f60621b0640 1 -- 172.21.15.123:0/349213070 shutdown_connections 2024-09-16T15:49:06.203 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:06.203+0000 7f60621b0640 1 -- 172.21.15.123:0/349213070 wait complete. 2024-09-16T15:49:06.217 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-16T15:49:06.330 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:06.331+0000 7f574f503640 1 -- 172.21.15.123:0/2784641569 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5734004d50 msgr2=0x7f57340051d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:06.331 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:06.331+0000 7f574f503640 1 --2- 172.21.15.123:0/2784641569 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5734004d50 0x7f57340051d0 secure :-1 s=READY pgs=1188 cs=0 l=1 rev1=1 crypto rx=0x7f57480cc420 tx=0x7f574401cd80 comp rx=0 tx=0).stop 2024-09-16T15:49:06.331 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:06.331+0000 7f574daa0640 1 -- 172.21.15.123:0/2784641569 reap_dead start 2024-09-16T15:49:06.331 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:06.331+0000 7f574f503640 1 -- 172.21.15.123:0/2784641569 shutdown_connections 2024-09-16T15:49:06.331 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:06.331+0000 7f574f503640 1 -- 172.21.15.123:0/2784641569 >> 172.21.15.123:0/2784641569 conn(0x7f57480d0d10 msgr2=0x7f57480d1120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:49:06.331 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:06.331+0000 7f574f503640 1 -- 172.21.15.123:0/2784641569 shutdown_connections 2024-09-16T15:49:06.331 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:06.331+0000 7f574f503640 1 -- 172.21.15.123:0/2784641569 wait complete. 2024-09-16T15:49:06.331 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:06.331+0000 7f574f503640 1 Processor -- start 2024-09-16T15:49:06.332 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:06.331+0000 7f574f503640 1 -- start start 2024-09-16T15:49:06.333 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:06.331+0000 7f574f503640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f57481f28b0 0x7f57481f2c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:49:06.333 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:06.331+0000 7f574f503640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f57481f31d0 0x7f57480c68d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:49:06.333 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:06.331+0000 7f574f503640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f57480b6630 0x7f57480c6e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:49:06.333 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:06.331+0000 7f574f503640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f57480cfa30 con 0x7f57481f31d0 2024-09-16T15:49:06.333 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:06.331+0000 7f574f503640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f57480cf8b0 con 0x7f57480b6630 2024-09-16T15:49:06.334 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:06.331+0000 7f574f503640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f57480cfbb0 con 0x7f57481f28b0 2024-09-16T15:49:06.334 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:06.331+0000 7f574ca9e640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f57481f31d0 0x7f57480c68d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:49:06.334 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:06.331+0000 7f574ca9e640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f57481f31d0 0x7f57480c68d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:55446/0 (socket says 172.21.15.123:55446) 2024-09-16T15:49:06.334 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:06.331+0000 7f574ca9e640 1 -- 172.21.15.123:0/3374869391 learned_addr learned my addr 172.21.15.123:0/3374869391 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:49:06.334 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:06.331+0000 7f574d29f640 1 --2- 172.21.15.123:0/3374869391 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f57481f28b0 0x7f57481f2c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:49:06.334 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:06.331+0000 7f574daa0640 1 --2- 172.21.15.123:0/3374869391 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f57480b6630 0x7f57480c6e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:49:06.334 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:06.331+0000 7f574ca9e640 1 -- 172.21.15.123:0/3374869391 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f57481f28b0 msgr2=0x7f57481f2c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:06.334 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:06.331+0000 7f574ca9e640 1 --2- 172.21.15.123:0/3374869391 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f57481f28b0 0x7f57481f2c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:06.334 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:06.331+0000 7f574ca9e640 1 -- 172.21.15.123:0/3374869391 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f57480b6630 msgr2=0x7f57480c6e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:06.334 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:06.331+0000 7f574ca9e640 1 --2- 172.21.15.123:0/3374869391 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f57480b6630 0x7f57480c6e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:06.334 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:06.331+0000 7f574ca9e640 1 -- 172.21.15.123:0/3374869391 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5748159910 con 0x7f57481f31d0 2024-09-16T15:49:06.334 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:06.331+0000 7f574ca9e640 1 --2- 172.21.15.123:0/3374869391 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f57481f31d0 0x7f57480c68d0 secure :-1 s=READY pgs=1189 cs=0 l=1 rev1=1 crypto rx=0x7f57400119c0 tx=0x7f5740011e80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:49:06.335 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:06.331+0000 7f573dffb640 1 -- 172.21.15.123:0/3374869391 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5740017070 con 0x7f57481f31d0 2024-09-16T15:49:06.335 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:06.331+0000 7f573dffb640 1 -- 172.21.15.123:0/3374869391 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f574000fdf0 con 0x7f57481f31d0 2024-09-16T15:49:06.335 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:06.331+0000 7f574f503640 1 -- 172.21.15.123:0/3374869391 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f57480c4a40 con 0x7f57481f31d0 2024-09-16T15:49:06.335 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:06.331+0000 7f573dffb640 1 -- 172.21.15.123:0/3374869391 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5740013690 con 0x7f57481f31d0 2024-09-16T15:49:06.335 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:06.335+0000 7f574f503640 1 -- 172.21.15.123:0/3374869391 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f57480c5000 con 0x7f57481f31d0 2024-09-16T15:49:06.336 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:06.335+0000 7f574f503640 1 -- 172.21.15.123:0/3374869391 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f57480d0140 con 0x7f57481f31d0 2024-09-16T15:49:06.336 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:06.335+0000 7f573dffb640 1 -- 172.21.15.123:0/3374869391 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 19) ==== 97488+0+0 (secure 0 0 0) 0x7f5740013830 con 0x7f57481f31d0 2024-09-16T15:49:06.336 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:06.335+0000 7f573dffb640 1 --2- 172.21.15.123:0/3374869391 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f572c073340 0x7f572c075800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:49:06.336 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:06.335+0000 7f573dffb640 1 -- 172.21.15.123:0/3374869391 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(107..107 src has 1..107) ==== 5839+0+0 (secure 0 0 0) 0x7f57400989a0 con 0x7f57481f31d0 2024-09-16T15:49:06.336 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:06.335+0000 7f574d29f640 1 --2- 172.21.15.123:0/3374869391 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f572c073340 0x7f572c075800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:49:06.336 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:06.335+0000 7f574d29f640 1 --2- 172.21.15.123:0/3374869391 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f572c073340 0x7f572c075800 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f57380058d0 tx=0x7f5738005860 comp rx=0 tx=0).ready entity=mgr.4150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:49:06.339 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:06.339+0000 7f573dffb640 1 -- 172.21.15.123:0/3374869391 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f57400633a0 con 0x7f57481f31d0 2024-09-16T15:49:06.504 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:06.503+0000 7f574f503640 1 -- 172.21.15.123:0/3374869391 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f57480b6c90 con 0x7f57481f31d0 2024-09-16T15:49:06.505 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:06.507+0000 7f573dffb640 1 -- 172.21.15.123:0/3374869391 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v107) ==== 74+0+17033 (secure 0 0 0) 0x7f5740062ca0 con 0x7f57481f31d0 2024-09-16T15:49:06.506 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-09-16T15:49:06.506 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":107,"fsid":"56077f89-18c8-486f-ae7e-eecc0bb7822c","created":"2024-09-16T15:42:26.364892+0000","modified":"2024-09-16T15:49:03.198103+0000","last_up_change":"2024-09-16T15:42:38.707234+0000","last_in_change":"2024-09-16T15:42:29.423662+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":17,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-16T15:42:39.287764+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_metadata","create_time":"2024-09-16T15:48:23.417905+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"102","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":17,"pool_name":"cephfs_data","create_time":"2024-09-16T15:48:24.776060+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"107","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":5,"snap_epoch":107,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9e7c1955-a607-4671-9be7-f2ef760ceeaa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6800","nonce":3882044969},{"type":"v1","addr":"172.21.15.123:6801","nonce":3882044969}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6802","nonce":3882044969},{"type":"v1","addr":"172.21.15.123:6803","nonce":3882044969}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6806","nonce":3882044969},{"type":"v1","addr":"172.21.15.123:6807","nonce":3882044969}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6804","nonce":3882044969},{"type":"v1","addr":"172.21.15.123:6805","nonce":3882044969}]},"public_addr":"172.21.15.123:6801/3882044969","cluster_addr":"172.21.15.123:6803/3882044969","heartbeat_back_addr":"172.21.15.123:6807/3882044969","heartbeat_front_addr":"172.21.15.123:6805/3882044969","state":["exists","up"]},{"osd":1,"uuid":"c540cfde-0977-4742-9b17-fd634668e010","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6816","nonce":4037001850},{"type":"v1","addr":"172.21.15.123:6817","nonce":4037001850}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6818","nonce":4037001850},{"type":"v1","addr":"172.21.15.123:6819","nonce":4037001850}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6822","nonce":4037001850},{"type":"v1","addr":"172.21.15.123:6823","nonce":4037001850}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6820","nonce":4037001850},{"type":"v1","addr":"172.21.15.123:6821","nonce":4037001850}]},"public_addr":"172.21.15.123:6817/4037001850","cluster_addr":"172.21.15.123:6819/4037001850","heartbeat_back_addr":"172.21.15.123:6823/4037001850","heartbeat_front_addr":"172.21.15.123:6821/4037001850","state":["exists","up"]},{"osd":2,"uuid":"9864a496-f169-486b-95c4-1d59162793a7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6824","nonce":1332143088},{"type":"v1","addr":"172.21.15.123:6825","nonce":1332143088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6826","nonce":1332143088},{"type":"v1","addr":"172.21.15.123:6827","nonce":1332143088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6830","nonce":1332143088},{"type":"v1","addr":"172.21.15.123:6831","nonce":1332143088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6828","nonce":1332143088},{"type":"v1","addr":"172.21.15.123:6829","nonce":1332143088}]},"public_addr":"172.21.15.123:6825/1332143088","cluster_addr":"172.21.15.123:6827/1332143088","heartbeat_back_addr":"172.21.15.123:6831/1332143088","heartbeat_front_addr":"172.21.15.123:6829/1332143088","state":["exists","up"]},{"osd":3,"uuid":"c6d1f817-f074-4325-95e1-578eba13edda","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6808","nonce":988586998},{"type":"v1","addr":"172.21.15.123:6809","nonce":988586998}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6810","nonce":988586998},{"type":"v1","addr":"172.21.15.123:6811","nonce":988586998}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6814","nonce":988586998},{"type":"v1","addr":"172.21.15.123:6815","nonce":988586998}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6812","nonce":988586998},{"type":"v1","addr":"172.21.15.123:6813","nonce":988586998}]},"public_addr":"172.21.15.123:6809/988586998","cluster_addr":"172.21.15.123:6811/988586998","heartbeat_back_addr":"172.21.15.123:6815/988586998","heartbeat_front_addr":"172.21.15.123:6813/988586998","state":["exists","up"]},{"osd":4,"uuid":"200a8df4-50ef-40a5-b48a-58981269c086","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6816","nonce":3387813890},{"type":"v1","addr":"172.21.15.184:6817","nonce":3387813890}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6818","nonce":3387813890},{"type":"v1","addr":"172.21.15.184:6819","nonce":3387813890}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6822","nonce":3387813890},{"type":"v1","addr":"172.21.15.184:6823","nonce":3387813890}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6820","nonce":3387813890},{"type":"v1","addr":"172.21.15.184:6821","nonce":3387813890}]},"public_addr":"172.21.15.184:6817/3387813890","cluster_addr":"172.21.15.184:6819/3387813890","heartbeat_back_addr":"172.21.15.184:6823/3387813890","heartbeat_front_addr":"172.21.15.184:6821/3387813890","state":["exists","up"]},{"osd":5,"uuid":"32d5a900-da3d-412c-bc19-13861a2368d8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":104,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6808","nonce":2680916017},{"type":"v1","addr":"172.21.15.184:6809","nonce":2680916017}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6810","nonce":2680916017},{"type":"v1","addr":"172.21.15.184:6811","nonce":2680916017}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6814","nonce":2680916017},{"type":"v1","addr":"172.21.15.184:6815","nonce":2680916017}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6812","nonce":2680916017},{"type":"v1","addr":"172.21.15.184:6813","nonce":2680916017}]},"public_addr":"172.21.15.184:6809/2680916017","cluster_addr":"172.21.15.184:6811/2680916017","heartbeat_back_addr":"172.21.15.184:6815/2680916017","heartbeat_front_addr":"172.21.15.184:6813/2680916017","state":["exists","up"]},{"osd":6,"uuid":"cde6f39e-aa1b-4260-ac19-774ee6aaca72","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":104,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6824","nonce":2183120457},{"type":"v1","addr":"172.21.15.184:6825","nonce":2183120457}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6826","nonce":2183120457},{"type":"v1","addr":"172.21.15.184:6827","nonce":2183120457}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6830","nonce":2183120457},{"type":"v1","addr":"172.21.15.184:6831","nonce":2183120457}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6828","nonce":2183120457},{"type":"v1","addr":"172.21.15.184:6829","nonce":2183120457}]},"public_addr":"172.21.15.184:6825/2183120457","cluster_addr":"172.21.15.184:6827/2183120457","heartbeat_back_addr":"172.21.15.184:6831/2183120457","heartbeat_front_addr":"172.21.15.184:6829/2183120457","state":["exists","up"]},{"osd":7,"uuid":"4f381778-eee1-47c6-804f-df0e6b33e313","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6800","nonce":4102957285},{"type":"v1","addr":"172.21.15.184:6801","nonce":4102957285}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6802","nonce":4102957285},{"type":"v1","addr":"172.21.15.184:6803","nonce":4102957285}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6806","nonce":4102957285},{"type":"v1","addr":"172.21.15.184:6807","nonce":4102957285}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6804","nonce":4102957285},{"type":"v1","addr":"172.21.15.184:6805","nonce":4102957285}]},"public_addr":"172.21.15.184:6801/4102957285","cluster_addr":"172.21.15.184:6803/4102957285","heartbeat_back_addr":"172.21.15.184:6807/4102957285","heartbeat_front_addr":"172.21.15.184:6805/4102957285","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:35.972210+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.222007+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.198617+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.385663+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.358935+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.156066+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.752973+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:35.682846+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"17.5","mappings":[{"from":0,"to":2}]},{"pgid":"17.b","mappings":[{"from":0,"to":5}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":17,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[{"pool":17,"snaps":[{"begin":2,"length":1},{"begin":5,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T15:49:06.508 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:06.507+0000 7f574f503640 1 -- 172.21.15.123:0/3374869391 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f572c073340 msgr2=0x7f572c075800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:06.508 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:06.507+0000 7f574f503640 1 --2- 172.21.15.123:0/3374869391 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f572c073340 0x7f572c075800 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f57380058d0 tx=0x7f5738005860 comp rx=0 tx=0).stop 2024-09-16T15:49:06.508 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:06.507+0000 7f574f503640 1 -- 172.21.15.123:0/3374869391 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f57481f31d0 msgr2=0x7f57480c68d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:06.508 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:06.507+0000 7f574f503640 1 --2- 172.21.15.123:0/3374869391 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f57481f31d0 0x7f57480c68d0 secure :-1 s=READY pgs=1189 cs=0 l=1 rev1=1 crypto rx=0x7f57400119c0 tx=0x7f5740011e80 comp rx=0 tx=0).stop 2024-09-16T15:49:06.508 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:06.507+0000 7f574f503640 1 -- 172.21.15.123:0/3374869391 shutdown_connections 2024-09-16T15:49:06.509 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:06.507+0000 7f574f503640 1 --2- 172.21.15.123:0/3374869391 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f572c073340 0x7f572c075800 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:06.509 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:06.507+0000 7f574f503640 1 --2- 172.21.15.123:0/3374869391 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f57480b6630 0x7f57480c6e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:06.509 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:06.507+0000 7f574f503640 1 --2- 172.21.15.123:0/3374869391 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f57481f31d0 0x7f57480c68d0 unknown :-1 s=CLOSED pgs=1189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:06.509 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:06.507+0000 7f574f503640 1 --2- 172.21.15.123:0/3374869391 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f57481f28b0 0x7f57481f2c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:06.509 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:06.507+0000 7f574f503640 1 -- 172.21.15.123:0/3374869391 >> 172.21.15.123:0/3374869391 conn(0x7f57480d0d10 msgr2=0x7f57480c9340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:49:06.509 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:06.507+0000 7f574f503640 1 -- 172.21.15.123:0/3374869391 shutdown_connections 2024-09-16T15:49:06.509 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:06.507+0000 7f574f503640 1 -- 172.21.15.123:0/3374869391 wait complete. 2024-09-16T15:49:06.524 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-09-16T15:49:06.649 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:06.647+0000 7f47a9a97640 1 -- 172.21.15.123:0/3180794277 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f47a40cd6e0 msgr2=0x7f47a40cdb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:06.649 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:06.647+0000 7f47a9a97640 1 --2- 172.21.15.123:0/3180794277 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f47a40cd6e0 0x7f47a40cdb20 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f4798009840 tx=0x7f479801c790 comp rx=0 tx=0).stop 2024-09-16T15:49:06.649 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:06.647+0000 7f47a9a97640 1 -- 172.21.15.123:0/3180794277 shutdown_connections 2024-09-16T15:49:06.649 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:06.647+0000 7f47a9a97640 1 --2- 172.21.15.123:0/3180794277 >> v2:172.21.15.184:3300/0 conn(0x7f47a415e820 0x7f47a415ecd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:06.649 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:06.647+0000 7f47a9a97640 1 --2- 172.21.15.123:0/3180794277 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f47a40cd6e0 0x7f47a40cdb20 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:06.649 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:06.647+0000 7f47a9a97640 1 -- 172.21.15.123:0/3180794277 >> 172.21.15.123:0/3180794277 conn(0x7f47a40cfb10 msgr2=0x7f47a40cff20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:49:06.649 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:06.647+0000 7f47a9a97640 1 -- 172.21.15.123:0/3180794277 shutdown_connections 2024-09-16T15:49:06.649 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:06.647+0000 7f47a9a97640 1 -- 172.21.15.123:0/3180794277 wait complete. 2024-09-16T15:49:06.651 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:06.651+0000 7f47a9a97640 1 Processor -- start 2024-09-16T15:49:06.651 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:06.651+0000 7f47a9a97640 1 -- start start 2024-09-16T15:49:06.651 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:06.651+0000 7f47a9a97640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f47a40cd6e0 0x7f47a414d100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:49:06.651 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:06.651+0000 7f47a9a97640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f47a415e160 0x7f47a40b33b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:49:06.651 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:06.651+0000 7f47a9a97640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f47a415e820 0x7f47a415cb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:49:06.651 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:06.651+0000 7f47a9a97640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f47a40cf5b0 con 0x7f47a415e160 2024-09-16T15:49:06.651 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:06.651+0000 7f47a9a97640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f47a40cf430 con 0x7f47a40cd6e0 2024-09-16T15:49:06.651 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:06.651+0000 7f47a9a97640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f47a41683d0 con 0x7f47a415e820 2024-09-16T15:49:06.651 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:06.651+0000 7f47a2ffd640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f47a415e160 0x7f47a40b33b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:49:06.651 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:06.651+0000 7f47a2ffd640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f47a415e160 0x7f47a40b33b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:55458/0 (socket says 172.21.15.123:55458) 2024-09-16T15:49:06.651 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:06.651+0000 7f47a2ffd640 1 -- 172.21.15.123:0/2666931601 learned_addr learned my addr 172.21.15.123:0/2666931601 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:49:06.652 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:06.651+0000 7f47a37fe640 1 --2- 172.21.15.123:0/2666931601 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f47a40cd6e0 0x7f47a414d100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:49:06.652 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:06.651+0000 7f47a3fff640 1 --2- 172.21.15.123:0/2666931601 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f47a415e820 0x7f47a415cb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:49:06.652 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:06.651+0000 7f47a2ffd640 1 -- 172.21.15.123:0/2666931601 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f47a415e820 msgr2=0x7f47a415cb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:06.652 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:06.651+0000 7f47a2ffd640 1 --2- 172.21.15.123:0/2666931601 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f47a415e820 0x7f47a415cb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:06.652 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:06.651+0000 7f47a2ffd640 1 -- 172.21.15.123:0/2666931601 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f47a40cd6e0 msgr2=0x7f47a414d100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:06.652 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:06.651+0000 7f47a2ffd640 1 --2- 172.21.15.123:0/2666931601 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f47a40cd6e0 0x7f47a414d100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:06.652 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:06.651+0000 7f47a2ffd640 1 -- 172.21.15.123:0/2666931601 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f47a40b3d00 con 0x7f47a415e160 2024-09-16T15:49:06.653 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:06.651+0000 7f47a2ffd640 1 --2- 172.21.15.123:0/2666931601 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f47a415e160 0x7f47a40b33b0 secure :-1 s=READY pgs=1190 cs=0 l=1 rev1=1 crypto rx=0x7f4798009790 tx=0x7f47980a66f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:49:06.653 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:06.651+0000 7f478bfff640 1 -- 172.21.15.123:0/2666931601 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47980bf020 con 0x7f47a415e160 2024-09-16T15:49:06.654 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:06.651+0000 7f478bfff640 1 -- 172.21.15.123:0/2666931601 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f47980043b0 con 0x7f47a415e160 2024-09-16T15:49:06.654 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:06.651+0000 7f47a9a97640 1 -- 172.21.15.123:0/2666931601 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f47a40b3fe0 con 0x7f47a415e160 2024-09-16T15:49:06.654 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:06.651+0000 7f478bfff640 1 -- 172.21.15.123:0/2666931601 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4798005280 con 0x7f47a415e160 2024-09-16T15:49:06.654 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:06.651+0000 7f47a9a97640 1 -- 172.21.15.123:0/2666931601 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f47a40c7e60 con 0x7f47a415e160 2024-09-16T15:49:06.654 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:06.651+0000 7f47a9a97640 1 -- 172.21.15.123:0/2666931601 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f47a40cdb20 con 0x7f47a415e160 2024-09-16T15:49:06.654 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:06.655+0000 7f478bfff640 1 -- 172.21.15.123:0/2666931601 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 19) ==== 97488+0+0 (secure 0 0 0) 0x7f47980c2070 con 0x7f47a415e160 2024-09-16T15:49:06.654 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:06.655+0000 7f478bfff640 1 --2- 172.21.15.123:0/2666931601 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f4790073340 0x7f4790075800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:49:06.654 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:06.655+0000 7f478bfff640 1 -- 172.21.15.123:0/2666931601 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(107..107 src has 1..107) ==== 5839+0+0 (secure 0 0 0) 0x7f479812ee40 con 0x7f47a415e160 2024-09-16T15:49:06.654 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:06.655+0000 7f47a37fe640 1 --2- 172.21.15.123:0/2666931601 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f4790073340 0x7f4790075800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:49:06.655 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:06.655+0000 7f47a37fe640 1 --2- 172.21.15.123:0/2666931601 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f4790073340 0x7f4790075800 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f4794001690 tx=0x7f4794009040 comp rx=0 tx=0).ready entity=mgr.4150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:49:06.657 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:06.659+0000 7f478bfff640 1 -- 172.21.15.123:0/2666931601 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f4798016410 con 0x7f47a415e160 2024-09-16T15:49:06.868 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:06.867+0000 7f47a9a97640 1 -- 172.21.15.123:0/2666931601 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7f47a40d06c0 con 0x7f47a415e160 2024-09-16T15:49:06.868 INFO:tasks.ceph.mon.a.smithi123.stderr:2024-09-16T15:49:06.867+0000 7f268d076640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-09-16T15:49:06.871 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:06.871+0000 7f478bfff640 1 -- 172.21.15.123:0/2666931601 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v70) ==== 161+0+0 (secure 0 0 0) 0x7f47980f9840 con 0x7f47a415e160 2024-09-16T15:49:06.871 INFO:teuthology.orchestra.run.smithi123.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-09-16T15:49:06.875 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:06.875+0000 7f47a9a97640 1 -- 172.21.15.123:0/2666931601 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f4790073340 msgr2=0x7f4790075800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:06.875 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:06.875+0000 7f47a9a97640 1 --2- 172.21.15.123:0/2666931601 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f4790073340 0x7f4790075800 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f4794001690 tx=0x7f4794009040 comp rx=0 tx=0).stop 2024-09-16T15:49:06.875 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:06.875+0000 7f47a9a97640 1 -- 172.21.15.123:0/2666931601 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f47a415e160 msgr2=0x7f47a40b33b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:06.875 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:06.875+0000 7f47a9a97640 1 --2- 172.21.15.123:0/2666931601 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f47a415e160 0x7f47a40b33b0 secure :-1 s=READY pgs=1190 cs=0 l=1 rev1=1 crypto rx=0x7f4798009790 tx=0x7f47980a66f0 comp rx=0 tx=0).stop 2024-09-16T15:49:06.875 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:06.875+0000 7f47a9a97640 1 -- 172.21.15.123:0/2666931601 shutdown_connections 2024-09-16T15:49:06.875 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:06.875+0000 7f47a9a97640 1 --2- 172.21.15.123:0/2666931601 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f4790073340 0x7f4790075800 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:06.875 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:06.875+0000 7f47a9a97640 1 --2- 172.21.15.123:0/2666931601 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f47a415e820 0x7f47a415cb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:06.876 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:06.875+0000 7f47a9a97640 1 --2- 172.21.15.123:0/2666931601 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f47a415e160 0x7f47a40b33b0 unknown :-1 s=CLOSED pgs=1190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:06.876 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:06.875+0000 7f47a9a97640 1 --2- 172.21.15.123:0/2666931601 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f47a40cd6e0 0x7f47a414d100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:06.876 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:06.875+0000 7f47a9a97640 1 -- 172.21.15.123:0/2666931601 >> 172.21.15.123:0/2666931601 conn(0x7f47a40cfb10 msgr2=0x7f47a40b0e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:49:06.876 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:06.875+0000 7f47a9a97640 1 -- 172.21.15.123:0/2666931601 shutdown_connections 2024-09-16T15:49:06.876 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:06.875+0000 7f47a9a97640 1 -- 172.21.15.123:0/2666931601 wait complete. 2024-09-16T15:49:06.897 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-09-16T15:49:06.920 INFO:tasks.ceph.mds.c.smithi123.stdout:starting mds.c at 2024-09-16T15:49:07.029 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:07.027+0000 7f1f08bca640 1 -- 172.21.15.123:0/1430936416 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1eec004b50 msgr2=0x7f1eec004f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:07.029 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:07.027+0000 7f1f08bca640 1 --2- 172.21.15.123:0/1430936416 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1eec004b50 0x7f1eec004f70 secure :-1 s=READY pgs=1194 cs=0 l=1 rev1=1 crypto rx=0x7f1ef400c2e0 tx=0x7f1ef401cf40 comp rx=0 tx=0).stop 2024-09-16T15:49:07.029 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:07.027+0000 7f1f02d9d640 1 -- 172.21.15.123:0/1430936416 reap_dead start 2024-09-16T15:49:07.029 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:07.027+0000 7f1f08bca640 1 -- 172.21.15.123:0/1430936416 shutdown_connections 2024-09-16T15:49:07.029 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:07.027+0000 7f1f08bca640 1 -- 172.21.15.123:0/1430936416 >> 172.21.15.123:0/1430936416 conn(0x7f1f040b1710 msgr2=0x7f1f040b1b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:49:07.029 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:07.027+0000 7f1f08bca640 1 -- 172.21.15.123:0/1430936416 shutdown_connections 2024-09-16T15:49:07.030 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:07.027+0000 7f1f08bca640 1 -- 172.21.15.123:0/1430936416 wait complete. 2024-09-16T15:49:07.030 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:07.031+0000 7f1f08bca640 1 Processor -- start 2024-09-16T15:49:07.030 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:07.031+0000 7f1f08bca640 1 -- start start 2024-09-16T15:49:07.033 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:07.031+0000 7f1f08bca640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1eec004b50 0x7f1f040738b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:49:07.033 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:07.031+0000 7f1f08bca640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1f04159650 0x7f1f0414a340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:49:07.033 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:07.031+0000 7f1f08bca640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f1f040c6c10 0x7f1f0414a880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:49:07.033 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:07.031+0000 7f1f08bca640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f1f04165d90 con 0x7f1eec004b50 2024-09-16T15:49:07.033 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:07.031+0000 7f1f08bca640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f1f04165c10 con 0x7f1f04159650 2024-09-16T15:49:07.033 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:07.031+0000 7f1f08bca640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f1f04165f10 con 0x7f1f040c6c10 2024-09-16T15:49:07.033 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:07.031+0000 7f1f0259c640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1eec004b50 0x7f1f040738b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:49:07.034 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:07.031+0000 7f1f0259c640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1eec004b50 0x7f1f040738b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:55496/0 (socket says 172.21.15.123:55496) 2024-09-16T15:49:07.034 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:07.031+0000 7f1f0259c640 1 -- 172.21.15.123:0/2402291543 learned_addr learned my addr 172.21.15.123:0/2402291543 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:49:07.034 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:07.031+0000 7f1f02d9d640 1 --2- 172.21.15.123:0/2402291543 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f1f040c6c10 0x7f1f0414a880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:49:07.034 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:07.031+0000 7f1f01d9b640 1 --2- 172.21.15.123:0/2402291543 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1f04159650 0x7f1f0414a340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:49:07.034 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:07.031+0000 7f1f0259c640 1 -- 172.21.15.123:0/2402291543 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f1f040c6c10 msgr2=0x7f1f0414a880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:07.034 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:07.031+0000 7f1f0259c640 1 --2- 172.21.15.123:0/2402291543 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f1f040c6c10 0x7f1f0414a880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:07.034 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:07.031+0000 7f1f0259c640 1 -- 172.21.15.123:0/2402291543 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1f04159650 msgr2=0x7f1f0414a340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:07.034 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:07.031+0000 7f1f0259c640 1 --2- 172.21.15.123:0/2402291543 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1f04159650 0x7f1f0414a340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:07.034 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:07.031+0000 7f1f0259c640 1 -- 172.21.15.123:0/2402291543 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1f041f3c60 con 0x7f1eec004b50 2024-09-16T15:49:07.034 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:07.031+0000 7f1f01d9b640 1 --2- 172.21.15.123:0/2402291543 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1f04159650 0x7f1f0414a340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T15:49:07.034 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:07.031+0000 7f1f0259c640 1 --2- 172.21.15.123:0/2402291543 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1eec004b50 0x7f1f040738b0 secure :-1 s=READY pgs=1195 cs=0 l=1 rev1=1 crypto rx=0x7f1ef401d420 tx=0x7f1ef4001d00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:49:07.035 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:07.031+0000 7f1ef2ffd640 1 -- 172.21.15.123:0/2402291543 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ef40014e0 con 0x7f1eec004b50 2024-09-16T15:49:07.035 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:07.031+0000 7f1ef2ffd640 1 -- 172.21.15.123:0/2402291543 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f1ef4001680 con 0x7f1eec004b50 2024-09-16T15:49:07.035 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:07.031+0000 7f1f08bca640 1 -- 172.21.15.123:0/2402291543 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1f041f8000 con 0x7f1eec004b50 2024-09-16T15:49:07.035 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:07.031+0000 7f1ef2ffd640 1 -- 172.21.15.123:0/2402291543 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ef4006310 con 0x7f1eec004b50 2024-09-16T15:49:07.035 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:07.031+0000 7f1f08bca640 1 -- 172.21.15.123:0/2402291543 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1f041f4200 con 0x7f1eec004b50 2024-09-16T15:49:07.035 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:07.031+0000 7f1f08bca640 1 -- 172.21.15.123:0/2402291543 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1f040cc930 con 0x7f1eec004b50 2024-09-16T15:49:07.036 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:07.035+0000 7f1ef2ffd640 1 -- 172.21.15.123:0/2402291543 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 19) ==== 97488+0+0 (secure 0 0 0) 0x7f1ef40064b0 con 0x7f1eec004b50 2024-09-16T15:49:07.036 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:07.035+0000 7f1ef2ffd640 1 --2- 172.21.15.123:0/2402291543 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f1ee4073370 0x7f1ee4075830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:49:07.036 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:07.035+0000 7f1ef2ffd640 1 -- 172.21.15.123:0/2402291543 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(108..108 src has 1..108) ==== 5881+0+0 (secure 0 0 0) 0x7f1ef412e5c0 con 0x7f1eec004b50 2024-09-16T15:49:07.036 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:07.035+0000 7f1f01d9b640 1 --2- 172.21.15.123:0/2402291543 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f1ee4073370 0x7f1ee4075830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:49:07.036 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:07.035+0000 7f1f01d9b640 1 --2- 172.21.15.123:0/2402291543 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f1ee4073370 0x7f1ee4075830 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f1f040cdb40 tx=0x7f1ef8006ac0 comp rx=0 tx=0).ready entity=mgr.4150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:49:07.038 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:07.039+0000 7f1ef2ffd640 1 -- 172.21.15.123:0/2402291543 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f1ef4016cf0 con 0x7f1eec004b50 2024-09-16T15:49:07.232 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:07.231+0000 7f1f08bca640 1 -- 172.21.15.123:0/2402291543 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) -- 0x7f1f040cdb70 con 0x7f1eec004b50 2024-09-16T15:49:07.874 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:07.875+0000 7f1ef2ffd640 1 -- 172.21.15.123:0/2402291543 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v71) ==== 104+0+0 (secure 0 0 0) 0x7f1ef40f8f90 con 0x7f1eec004b50 2024-09-16T15:49:07.878 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:07.879+0000 7f1f08bca640 1 -- 172.21.15.123:0/2402291543 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f1ee4073370 msgr2=0x7f1ee4075830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:07.878 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:07.879+0000 7f1f08bca640 1 --2- 172.21.15.123:0/2402291543 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f1ee4073370 0x7f1ee4075830 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f1f040cdb40 tx=0x7f1ef8006ac0 comp rx=0 tx=0).stop 2024-09-16T15:49:07.878 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:07.879+0000 7f1f08bca640 1 -- 172.21.15.123:0/2402291543 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1eec004b50 msgr2=0x7f1f040738b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:07.878 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:07.879+0000 7f1f08bca640 1 --2- 172.21.15.123:0/2402291543 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1eec004b50 0x7f1f040738b0 secure :-1 s=READY pgs=1195 cs=0 l=1 rev1=1 crypto rx=0x7f1ef401d420 tx=0x7f1ef4001d00 comp rx=0 tx=0).stop 2024-09-16T15:49:07.878 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:07.879+0000 7f1f08bca640 1 -- 172.21.15.123:0/2402291543 shutdown_connections 2024-09-16T15:49:07.878 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:07.879+0000 7f1f08bca640 1 --2- 172.21.15.123:0/2402291543 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f1ee4073370 0x7f1ee4075830 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:07.878 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:07.879+0000 7f1f08bca640 1 --2- 172.21.15.123:0/2402291543 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f1f040c6c10 0x7f1f0414a880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:07.878 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:07.879+0000 7f1f08bca640 1 --2- 172.21.15.123:0/2402291543 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1f04159650 0x7f1f0414a340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:07.878 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:07.879+0000 7f1f08bca640 1 --2- 172.21.15.123:0/2402291543 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1eec004b50 0x7f1f040738b0 unknown :-1 s=CLOSED pgs=1195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:07.879 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:07.879+0000 7f1f08bca640 1 -- 172.21.15.123:0/2402291543 >> 172.21.15.123:0/2402291543 conn(0x7f1f040b1710 msgr2=0x7f1f040b1b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:49:07.879 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:07.879+0000 7f1f08bca640 1 -- 172.21.15.123:0/2402291543 shutdown_connections 2024-09-16T15:49:07.879 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:07.879+0000 7f1f08bca640 1 -- 172.21.15.123:0/2402291543 wait complete. 2024-09-16T15:49:07.900 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-09-16T15:49:08.073 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:08.071+0000 7f91531a6640 1 -- 172.21.15.123:0/3628129570 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f914c0b4bb0 msgr2=0x7f914c0d0d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:08.073 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:08.071+0000 7f91531a6640 1 --2- 172.21.15.123:0/3628129570 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f914c0b4bb0 0x7f914c0d0d40 secure :-1 s=READY pgs=1197 cs=0 l=1 rev1=1 crypto rx=0x7f9138009870 tx=0x7f913801c7f0 comp rx=0 tx=0).stop 2024-09-16T15:49:08.073 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:08.071+0000 7f91531a6640 1 -- 172.21.15.123:0/3628129570 shutdown_connections 2024-09-16T15:49:08.073 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:08.071+0000 7f91531a6640 1 --2- 172.21.15.123:0/3628129570 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f914c0d1a00 0x7f914c0d1e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:08.073 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:08.071+0000 7f91531a6640 1 --2- 172.21.15.123:0/3628129570 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f914c0b4bb0 0x7f914c0d0d40 unknown :-1 s=CLOSED pgs=1197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:08.073 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:08.071+0000 7f91531a6640 1 -- 172.21.15.123:0/3628129570 >> 172.21.15.123:0/3628129570 conn(0x7f914c0d3e30 msgr2=0x7f914c0d4240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:49:08.073 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:08.071+0000 7f91531a6640 1 -- 172.21.15.123:0/3628129570 shutdown_connections 2024-09-16T15:49:08.073 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:08.071+0000 7f91531a6640 1 -- 172.21.15.123:0/3628129570 wait complete. 2024-09-16T15:49:08.074 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:08.075+0000 7f91531a6640 1 Processor -- start 2024-09-16T15:49:08.074 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:08.075+0000 7f91531a6640 1 -- start start 2024-09-16T15:49:08.076 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:08.075+0000 7f91531a6640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f914c0b4bb0 0x7f914c14f300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:49:08.076 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:08.075+0000 7f91531a6640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f914c0d1a00 0x7f914c0cb160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:49:08.076 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:08.075+0000 7f91531a6640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f914c161390 0x7f914c0cce80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:49:08.076 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:08.075+0000 7f91531a6640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f914c16af30 con 0x7f914c161390 2024-09-16T15:49:08.076 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:08.075+0000 7f91531a6640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f914c16adb0 con 0x7f914c0b4bb0 2024-09-16T15:49:08.076 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:08.075+0000 7f91531a6640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f914c16b0b0 con 0x7f914c0d1a00 2024-09-16T15:49:08.076 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:08.075+0000 7f9151743640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f914c161390 0x7f914c0cce80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:49:08.077 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:08.075+0000 7f9151743640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f914c161390 0x7f914c0cce80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:55524/0 (socket says 172.21.15.123:55524) 2024-09-16T15:49:08.077 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:08.075+0000 7f9151743640 1 -- 172.21.15.123:0/2127711491 learned_addr learned my addr 172.21.15.123:0/2127711491 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:49:08.077 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:08.075+0000 7f9151743640 1 -- 172.21.15.123:0/2127711491 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f914c0d1a00 msgr2=0x7f914c0cb160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:08.077 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:08.075+0000 7f9151743640 1 --2- 172.21.15.123:0/2127711491 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f914c0d1a00 0x7f914c0cb160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:08.077 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:08.075+0000 7f9151743640 1 -- 172.21.15.123:0/2127711491 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f914c0b4bb0 msgr2=0x7f914c14f300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:08.077 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:08.075+0000 7f9151743640 1 --2- 172.21.15.123:0/2127711491 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f914c0b4bb0 0x7f914c14f300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:08.077 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:08.075+0000 7f9151743640 1 -- 172.21.15.123:0/2127711491 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f914c14e860 con 0x7f914c161390 2024-09-16T15:49:08.077 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:08.075+0000 7f9151743640 1 --2- 172.21.15.123:0/2127711491 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f914c161390 0x7f914c0cce80 secure :-1 s=READY pgs=1198 cs=0 l=1 rev1=1 crypto rx=0x7f914400b9a0 tx=0x7f914400be60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:49:08.077 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:08.075+0000 7f91497fa640 1 -- 172.21.15.123:0/2127711491 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9144019070 con 0x7f914c161390 2024-09-16T15:49:08.077 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:08.075+0000 7f91497fa640 1 -- 172.21.15.123:0/2127711491 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f914400ccb0 con 0x7f914c161390 2024-09-16T15:49:08.077 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:08.075+0000 7f91497fa640 1 -- 172.21.15.123:0/2127711491 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91440146a0 con 0x7f914c161390 2024-09-16T15:49:08.078 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:08.075+0000 7f91531a6640 1 -- 172.21.15.123:0/2127711491 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f914c0cdb60 con 0x7f914c161390 2024-09-16T15:49:08.078 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:08.075+0000 7f91531a6640 1 -- 172.21.15.123:0/2127711491 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f914c14eb00 con 0x7f914c161390 2024-09-16T15:49:08.078 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:08.075+0000 7f91531a6640 1 -- 172.21.15.123:0/2127711491 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f914c14ef30 con 0x7f914c161390 2024-09-16T15:49:08.078 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:08.079+0000 7f91497fa640 1 -- 172.21.15.123:0/2127711491 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 19) ==== 97488+0+0 (secure 0 0 0) 0x7f914401c070 con 0x7f914c161390 2024-09-16T15:49:08.078 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:08.079+0000 7f91497fa640 1 --2- 172.21.15.123:0/2127711491 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f9130073340 0x7f9130075800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:49:08.079 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:08.079+0000 7f9150f42640 1 --2- 172.21.15.123:0/2127711491 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f9130073340 0x7f9130075800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:49:08.079 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:08.079+0000 7f91497fa640 1 -- 172.21.15.123:0/2127711491 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(108..108 src has 1..108) ==== 5881+0+0 (secure 0 0 0) 0x7f9144094550 con 0x7f914c161390 2024-09-16T15:49:08.079 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:08.079+0000 7f9150f42640 1 --2- 172.21.15.123:0/2127711491 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f9130073340 0x7f9130075800 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f9140007fd0 tx=0x7f9140007940 comp rx=0 tx=0).ready entity=mgr.4150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:49:08.081 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:08.083+0000 7f91497fa640 1 -- 172.21.15.123:0/2127711491 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f914405ef20 con 0x7f914c161390 2024-09-16T15:49:08.244 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:08.243+0000 7f91531a6640 1 -- 172.21.15.123:0/2127711491 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) -- 0x7f914c0d1e90 con 0x7f914c161390 2024-09-16T15:49:09.251 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:09.251+0000 7f91497fa640 1 -- 172.21.15.123:0/2127711491 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v109) ==== 181+0+0 (secure 0 0 0) 0x7f914405e820 con 0x7f914c161390 2024-09-16T15:49:09.251 INFO:teuthology.orchestra.run.smithi123.stderr:pool 'cephfs_metadata' removed 2024-09-16T15:49:09.256 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:09.255+0000 7f91531a6640 1 -- 172.21.15.123:0/2127711491 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f9130073340 msgr2=0x7f9130075800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:09.256 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:09.255+0000 7f91531a6640 1 --2- 172.21.15.123:0/2127711491 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f9130073340 0x7f9130075800 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f9140007fd0 tx=0x7f9140007940 comp rx=0 tx=0).stop 2024-09-16T15:49:09.256 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:09.255+0000 7f91531a6640 1 -- 172.21.15.123:0/2127711491 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f914c161390 msgr2=0x7f914c0cce80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:09.257 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:09.255+0000 7f91531a6640 1 --2- 172.21.15.123:0/2127711491 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f914c161390 0x7f914c0cce80 secure :-1 s=READY pgs=1198 cs=0 l=1 rev1=1 crypto rx=0x7f914400b9a0 tx=0x7f914400be60 comp rx=0 tx=0).stop 2024-09-16T15:49:09.257 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:09.255+0000 7f91531a6640 1 -- 172.21.15.123:0/2127711491 shutdown_connections 2024-09-16T15:49:09.257 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:09.255+0000 7f91531a6640 1 --2- 172.21.15.123:0/2127711491 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f9130073340 0x7f9130075800 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:09.257 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:09.255+0000 7f91531a6640 1 --2- 172.21.15.123:0/2127711491 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f914c161390 0x7f914c0cce80 unknown :-1 s=CLOSED pgs=1198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:09.257 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:09.255+0000 7f91531a6640 1 --2- 172.21.15.123:0/2127711491 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f914c0d1a00 0x7f914c0cb160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:09.257 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:09.255+0000 7f91531a6640 1 --2- 172.21.15.123:0/2127711491 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f914c0b4bb0 0x7f914c14f300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:09.257 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:09.255+0000 7f91531a6640 1 -- 172.21.15.123:0/2127711491 >> 172.21.15.123:0/2127711491 conn(0x7f914c0d3e30 msgr2=0x7f914c150cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:49:09.258 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:09.255+0000 7f91531a6640 1 -- 172.21.15.123:0/2127711491 shutdown_connections 2024-09-16T15:49:09.258 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:09.259+0000 7f91531a6640 1 -- 172.21.15.123:0/2127711491 wait complete. 2024-09-16T15:49:09.278 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-09-16T15:49:09.402 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:09.403+0000 7f23d63ef640 1 -- 172.21.15.123:0/227085981 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f23d00725e0 msgr2=0x7f23d015f670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:09.403 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:09.403+0000 7f23d63ef640 1 --2- 172.21.15.123:0/227085981 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f23d00725e0 0x7f23d015f670 secure :-1 s=READY pgs=1200 cs=0 l=1 rev1=1 crypto rx=0x7f23d0069b40 tx=0x7f23c401c800 comp rx=0 tx=0).stop 2024-09-16T15:49:09.403 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:09.403+0000 7f23d63ef640 1 -- 172.21.15.123:0/227085981 shutdown_connections 2024-09-16T15:49:09.403 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:09.403+0000 7f23d63ef640 1 --2- 172.21.15.123:0/227085981 >> v2:172.21.15.184:3300/0 conn(0x7f23d015fbb0 0x7f23d00cebd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:09.403 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:09.403+0000 7f23d63ef640 1 --2- 172.21.15.123:0/227085981 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f23d00725e0 0x7f23d015f670 unknown :-1 s=CLOSED pgs=1200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:09.403 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:09.403+0000 7f23d63ef640 1 -- 172.21.15.123:0/227085981 >> 172.21.15.123:0/227085981 conn(0x7f23d00b1630 msgr2=0x7f23d00b1a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:49:09.403 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:09.403+0000 7f23d63ef640 1 -- 172.21.15.123:0/227085981 shutdown_connections 2024-09-16T15:49:09.403 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:09.403+0000 7f23d63ef640 1 -- 172.21.15.123:0/227085981 wait complete. 2024-09-16T15:49:09.404 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:09.403+0000 7f23d63ef640 1 Processor -- start 2024-09-16T15:49:09.404 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:09.403+0000 7f23d63ef640 1 -- start start 2024-09-16T15:49:09.405 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:09.403+0000 7f23d63ef640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f23d00725e0 0x7f23d01f9080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:49:09.406 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:09.403+0000 7f23d63ef640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f23d0152cf0 0x7f23d014bec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:49:09.406 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:09.403+0000 7f23d63ef640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f23d015fbb0 0x7f23d015d900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:49:09.406 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:09.403+0000 7f23d63ef640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f23d0168da0 con 0x7f23d00725e0 2024-09-16T15:49:09.406 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:09.403+0000 7f23d63ef640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f23d0168c20 con 0x7f23d015fbb0 2024-09-16T15:49:09.406 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:09.403+0000 7f23d63ef640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f23d0168f20 con 0x7f23d0152cf0 2024-09-16T15:49:09.406 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:09.403+0000 7f23cffff640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f23d00725e0 0x7f23d01f9080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:49:09.406 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:09.403+0000 7f23cffff640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f23d00725e0 0x7f23d01f9080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:55542/0 (socket says 172.21.15.123:55542) 2024-09-16T15:49:09.406 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:09.403+0000 7f23cffff640 1 -- 172.21.15.123:0/2327603565 learned_addr learned my addr 172.21.15.123:0/2327603565 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:49:09.406 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:09.403+0000 7f23cffff640 1 -- 172.21.15.123:0/2327603565 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f23d0152cf0 msgr2=0x7f23d014bec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:09.406 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:09.403+0000 7f23cf7fe640 1 --2- 172.21.15.123:0/2327603565 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f23d0152cf0 0x7f23d014bec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:49:09.406 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:09.403+0000 7f23d498c640 1 --2- 172.21.15.123:0/2327603565 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f23d015fbb0 0x7f23d015d900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:49:09.406 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:09.403+0000 7f23cffff640 1 --2- 172.21.15.123:0/2327603565 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f23d0152cf0 0x7f23d014bec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:09.407 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:09.403+0000 7f23cffff640 1 -- 172.21.15.123:0/2327603565 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f23d015fbb0 msgr2=0x7f23d015d900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:09.407 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:09.403+0000 7f23cffff640 1 --2- 172.21.15.123:0/2327603565 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f23d015fbb0 0x7f23d015d900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:09.407 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:09.403+0000 7f23cffff640 1 -- 172.21.15.123:0/2327603565 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f23d01f9fb0 con 0x7f23d00725e0 2024-09-16T15:49:09.407 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:09.403+0000 7f23cffff640 1 --2- 172.21.15.123:0/2327603565 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f23d00725e0 0x7f23d01f9080 secure :-1 s=READY pgs=1201 cs=0 l=1 rev1=1 crypto rx=0x7f23c000c730 tx=0x7f23c000cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:49:09.407 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:09.407+0000 7f23ccff9640 1 -- 172.21.15.123:0/2327603565 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f23c0017070 con 0x7f23d00725e0 2024-09-16T15:49:09.407 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:09.407+0000 7f23ccff9640 1 -- 172.21.15.123:0/2327603565 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f23c00051a0 con 0x7f23d00725e0 2024-09-16T15:49:09.407 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:09.407+0000 7f23ccff9640 1 -- 172.21.15.123:0/2327603565 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f23c000a640 con 0x7f23d00725e0 2024-09-16T15:49:09.407 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:09.407+0000 7f23d63ef640 1 -- 172.21.15.123:0/2327603565 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f23d01f99f0 con 0x7f23d00725e0 2024-09-16T15:49:09.407 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:09.407+0000 7f23d63ef640 1 -- 172.21.15.123:0/2327603565 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f23d014b6c0 con 0x7f23d00725e0 2024-09-16T15:49:09.408 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:09.407+0000 7f23d63ef640 1 -- 172.21.15.123:0/2327603565 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f23d015f670 con 0x7f23d00725e0 2024-09-16T15:49:09.408 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:09.407+0000 7f23ccff9640 1 -- 172.21.15.123:0/2327603565 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 19) ==== 97488+0+0 (secure 0 0 0) 0x7f23c001a070 con 0x7f23d00725e0 2024-09-16T15:49:09.408 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:09.407+0000 7f23ccff9640 1 --2- 172.21.15.123:0/2327603565 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f23bc073340 0x7f23bc075800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:49:09.408 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:09.407+0000 7f23ccff9640 1 -- 172.21.15.123:0/2327603565 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(109..109 src has 1..109) ==== 5443+0+0 (secure 0 0 0) 0x7f23c0093850 con 0x7f23d00725e0 2024-09-16T15:49:09.408 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:09.407+0000 7f23cf7fe640 1 --2- 172.21.15.123:0/2327603565 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f23bc073340 0x7f23bc075800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:49:09.409 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:09.407+0000 7f23cf7fe640 1 --2- 172.21.15.123:0/2327603565 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f23bc073340 0x7f23bc075800 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f23c40af000 tx=0x7f23c401b040 comp rx=0 tx=0).ready entity=mgr.4150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:49:09.411 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:09.411+0000 7f23ccff9640 1 -- 172.21.15.123:0/2327603565 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f23d00cde70 con 0x7f23d00725e0 2024-09-16T15:49:09.583 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:09.583+0000 7f23d63ef640 1 -- 172.21.15.123:0/2327603565 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) -- 0x7f23d015ef50 con 0x7f23d00725e0 2024-09-16T15:49:09.840 DEBUG:teuthology.orchestra.run.smithi123:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-16T15:49:09.843 DEBUG:teuthology.orchestra.run.smithi184:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-16T15:49:10.262 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:10.263+0000 7f23ccff9640 1 -- 172.21.15.123:0/2327603565 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v110) ==== 169+0+0 (secure 0 0 0) 0x7f23c005de10 con 0x7f23d00725e0 2024-09-16T15:49:10.262 INFO:teuthology.orchestra.run.smithi123.stderr:pool 'cephfs_data' removed 2024-09-16T15:49:10.267 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:10.267+0000 7f23d63ef640 1 -- 172.21.15.123:0/2327603565 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f23bc073340 msgr2=0x7f23bc075800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:10.268 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:10.267+0000 7f23d63ef640 1 --2- 172.21.15.123:0/2327603565 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f23bc073340 0x7f23bc075800 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f23c40af000 tx=0x7f23c401b040 comp rx=0 tx=0).stop 2024-09-16T15:49:10.268 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:10.267+0000 7f23d63ef640 1 -- 172.21.15.123:0/2327603565 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f23d00725e0 msgr2=0x7f23d01f9080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:10.268 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:10.267+0000 7f23d63ef640 1 --2- 172.21.15.123:0/2327603565 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f23d00725e0 0x7f23d01f9080 secure :-1 s=READY pgs=1201 cs=0 l=1 rev1=1 crypto rx=0x7f23c000c730 tx=0x7f23c000cbf0 comp rx=0 tx=0).stop 2024-09-16T15:49:10.268 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:10.267+0000 7f23d63ef640 1 -- 172.21.15.123:0/2327603565 shutdown_connections 2024-09-16T15:49:10.268 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:10.267+0000 7f23d63ef640 1 --2- 172.21.15.123:0/2327603565 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f23bc073340 0x7f23bc075800 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:10.268 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:10.267+0000 7f23d63ef640 1 --2- 172.21.15.123:0/2327603565 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f23d015fbb0 0x7f23d015d900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:10.268 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:10.267+0000 7f23d63ef640 1 --2- 172.21.15.123:0/2327603565 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f23d0152cf0 0x7f23d014bec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:10.268 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:10.267+0000 7f23d63ef640 1 --2- 172.21.15.123:0/2327603565 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f23d00725e0 0x7f23d01f9080 unknown :-1 s=CLOSED pgs=1201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:10.268 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:10.267+0000 7f23d63ef640 1 -- 172.21.15.123:0/2327603565 >> 172.21.15.123:0/2327603565 conn(0x7f23d00b1630 msgr2=0x7f23d014f1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:49:10.269 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:10.267+0000 7f23d63ef640 1 -- 172.21.15.123:0/2327603565 shutdown_connections 2024-09-16T15:49:10.269 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:10.267+0000 7f23d63ef640 1 -- 172.21.15.123:0/2327603565 wait complete. 2024-09-16T15:49:10.289 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config reset 27 2024-09-16T15:49:10.411 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:10.411+0000 7fac631f1640 1 -- 172.21.15.123:0/2978685141 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fac5c0b3100 msgr2=0x7fac5c0cc1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:10.411 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:10.411+0000 7fac631f1640 1 --2- 172.21.15.123:0/2978685141 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fac5c0b3100 0x7fac5c0cc1b0 secure :-1 s=READY pgs=1202 cs=0 l=1 rev1=1 crypto rx=0x7fac5c051540 tx=0x7fac5001c8c0 comp rx=0 tx=0).stop 2024-09-16T15:49:10.412 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:10.411+0000 7fac631f1640 1 -- 172.21.15.123:0/2978685141 shutdown_connections 2024-09-16T15:49:10.412 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:10.411+0000 7fac631f1640 1 --2- 172.21.15.123:0/2978685141 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fac5c0b3100 0x7fac5c0cc1b0 unknown :-1 s=CLOSED pgs=1202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:10.412 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:10.411+0000 7fac631f1640 1 --2- 172.21.15.123:0/2978685141 >> v2:172.21.15.184:3300/0 conn(0x7fac5c15ac00 0x7fac5c0702f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:10.412 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:10.411+0000 7fac631f1640 1 -- 172.21.15.123:0/2978685141 >> 172.21.15.123:0/2978685141 conn(0x7fac5c0b2610 msgr2=0x7fac5c0b29f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:49:10.412 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:10.411+0000 7fac631f1640 1 -- 172.21.15.123:0/2978685141 shutdown_connections 2024-09-16T15:49:10.412 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:10.411+0000 7fac631f1640 1 -- 172.21.15.123:0/2978685141 wait complete. 2024-09-16T15:49:10.412 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:10.411+0000 7fac631f1640 1 Processor -- start 2024-09-16T15:49:10.413 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:10.411+0000 7fac631f1640 1 -- start start 2024-09-16T15:49:10.414 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:10.415+0000 7fac631f1640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fac5c070830 0x7fac5c165460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:49:10.415 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:10.415+0000 7fac631f1640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fac5c0b3100 0x7fac5c151de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:49:10.415 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:10.415+0000 7fac631f1640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fac5c15ac00 0x7fac5c0b4f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:49:10.415 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:10.415+0000 7fac631f1640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fac5c168660 con 0x7fac5c15ac00 2024-09-16T15:49:10.415 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:10.415+0000 7fac631f1640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fac5c1684e0 con 0x7fac5c070830 2024-09-16T15:49:10.415 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:10.415+0000 7fac631f1640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fac5c1687e0 con 0x7fac5c0b3100 2024-09-16T15:49:10.415 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:10.415+0000 7fac6178e640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fac5c15ac00 0x7fac5c0b4f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:49:10.415 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:10.415+0000 7fac6178e640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fac5c15ac00 0x7fac5c0b4f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:55568/0 (socket says 172.21.15.123:55568) 2024-09-16T15:49:10.415 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:10.415+0000 7fac6178e640 1 -- 172.21.15.123:0/1784146897 learned_addr learned my addr 172.21.15.123:0/1784146897 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:49:10.415 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:10.415+0000 7fac6178e640 1 -- 172.21.15.123:0/1784146897 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fac5c0b3100 msgr2=0x7fac5c151de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:10.415 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:10.415+0000 7fac6178e640 1 --2- 172.21.15.123:0/1784146897 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fac5c0b3100 0x7fac5c151de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:10.415 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:10.415+0000 7fac6178e640 1 -- 172.21.15.123:0/1784146897 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fac5c070830 msgr2=0x7fac5c165460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:10.415 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:10.415+0000 7fac6178e640 1 --2- 172.21.15.123:0/1784146897 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fac5c070830 0x7fac5c165460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:10.415 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:10.415+0000 7fac6178e640 1 -- 172.21.15.123:0/1784146897 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fac5c154120 con 0x7fac5c15ac00 2024-09-16T15:49:10.416 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:10.415+0000 7fac6178e640 1 --2- 172.21.15.123:0/1784146897 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fac5c15ac00 0x7fac5c0b4f40 secure :-1 s=READY pgs=1203 cs=0 l=1 rev1=1 crypto rx=0x7fac5400c780 tx=0x7fac5400cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:49:10.416 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:10.415+0000 7fac597fa640 1 -- 172.21.15.123:0/1784146897 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac54017070 con 0x7fac5c15ac00 2024-09-16T15:49:10.416 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:10.415+0000 7fac597fa640 1 -- 172.21.15.123:0/1784146897 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fac54005070 con 0x7fac5c15ac00 2024-09-16T15:49:10.416 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:10.415+0000 7fac597fa640 1 -- 172.21.15.123:0/1784146897 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac540054c0 con 0x7fac5c15ac00 2024-09-16T15:49:10.416 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:10.415+0000 7fac631f1640 1 -- 172.21.15.123:0/1784146897 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fac5c154400 con 0x7fac5c15ac00 2024-09-16T15:49:10.416 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:10.415+0000 7fac631f1640 1 -- 172.21.15.123:0/1784146897 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fac5c15b280 con 0x7fac5c15ac00 2024-09-16T15:49:10.417 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:10.415+0000 7fac631f1640 1 -- 172.21.15.123:0/1784146897 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fac5c0cc1b0 con 0x7fac5c15ac00 2024-09-16T15:49:10.417 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:10.415+0000 7fac597fa640 1 -- 172.21.15.123:0/1784146897 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 19) ==== 97488+0+0 (secure 0 0 0) 0x7fac5401a070 con 0x7fac5c15ac00 2024-09-16T15:49:10.417 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:10.415+0000 7fac597fa640 1 --2- 172.21.15.123:0/1784146897 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fac48073340 0x7fac48075800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:49:10.417 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:10.415+0000 7fac597fa640 1 -- 172.21.15.123:0/1784146897 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(110..110 src has 1..110) ==== 4971+0+0 (secure 0 0 0) 0x7fac54093e30 con 0x7fac5c15ac00 2024-09-16T15:49:10.417 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:10.415+0000 7fac60f8d640 1 --2- 172.21.15.123:0/1784146897 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fac48073340 0x7fac48075800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:49:10.417 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:10.419+0000 7fac60f8d640 1 --2- 172.21.15.123:0/1784146897 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fac48073340 0x7fac48075800 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fac4c00a860 tx=0x7fac4c005b60 comp rx=0 tx=0).ready entity=mgr.4150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:49:10.420 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:10.419+0000 7fac597fa640 1 -- 172.21.15.123:0/1784146897 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7fac54021020 con 0x7fac5c15ac00 2024-09-16T15:49:10.593 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:10.595+0000 7fac631f1640 1 -- 172.21.15.123:0/1784146897 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "config reset", "num": 27} v 0) -- 0x7fac5c0b6a50 con 0x7fac5c15ac00 2024-09-16T15:49:10.599 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:10.599+0000 7fac597fa640 1 -- 172.21.15.123:0/1784146897 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "config reset", "num": 27}]=0 v39) ==== 71+0+0 (secure 0 0 0) 0x7fac5405ed20 con 0x7fac5c15ac00 2024-09-16T15:49:10.602 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:10.603+0000 7fac631f1640 1 -- 172.21.15.123:0/1784146897 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fac48073340 msgr2=0x7fac48075800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:10.602 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:10.603+0000 7fac631f1640 1 --2- 172.21.15.123:0/1784146897 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fac48073340 0x7fac48075800 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fac4c00a860 tx=0x7fac4c005b60 comp rx=0 tx=0).stop 2024-09-16T15:49:10.602 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:10.603+0000 7fac631f1640 1 -- 172.21.15.123:0/1784146897 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fac5c15ac00 msgr2=0x7fac5c0b4f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:10.602 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:10.603+0000 7fac631f1640 1 --2- 172.21.15.123:0/1784146897 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fac5c15ac00 0x7fac5c0b4f40 secure :-1 s=READY pgs=1203 cs=0 l=1 rev1=1 crypto rx=0x7fac5400c780 tx=0x7fac5400cc40 comp rx=0 tx=0).stop 2024-09-16T15:49:10.602 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:10.603+0000 7fac631f1640 1 -- 172.21.15.123:0/1784146897 shutdown_connections 2024-09-16T15:49:10.602 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:10.603+0000 7fac631f1640 1 --2- 172.21.15.123:0/1784146897 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fac48073340 0x7fac48075800 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:10.602 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:10.603+0000 7fac631f1640 1 --2- 172.21.15.123:0/1784146897 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fac5c15ac00 0x7fac5c0b4f40 unknown :-1 s=CLOSED pgs=1203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:10.602 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:10.603+0000 7fac631f1640 1 --2- 172.21.15.123:0/1784146897 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fac5c0b3100 0x7fac5c151de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:10.603 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:10.603+0000 7fac631f1640 1 --2- 172.21.15.123:0/1784146897 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fac5c070830 0x7fac5c165460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:10.603 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:10.603+0000 7fac631f1640 1 -- 172.21.15.123:0/1784146897 >> 172.21.15.123:0/1784146897 conn(0x7fac5c0b2610 msgr2=0x7fac5c071350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:49:10.603 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:10.603+0000 7fac631f1640 1 -- 172.21.15.123:0/1784146897 shutdown_connections 2024-09-16T15:49:10.603 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:10.603+0000 7fac631f1640 1 -- 172.21.15.123:0/1784146897 wait complete. 2024-09-16T15:49:10.619 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_volumes.TestMisc.test_subvolume_no_upgrade_v1_sanity' 2024-09-16T15:49:10.739 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:10.739+0000 7f57ffe83640 1 -- 172.21.15.123:0/2404135864 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f57e40053b0 msgr2=0x7f57e400ba10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:10.739 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:10.739+0000 7f57ffe83640 1 --2- 172.21.15.123:0/2404135864 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f57e40053b0 0x7f57e400ba10 secure :-1 s=READY pgs=1205 cs=0 l=1 rev1=1 crypto rx=0x7f57f0009e00 tx=0x7f57f001d140 comp rx=0 tx=0).stop 2024-09-16T15:49:10.740 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:10.739+0000 7f57ffe83640 1 -- 172.21.15.123:0/2404135864 shutdown_connections 2024-09-16T15:49:10.740 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:10.739+0000 7f57ffe83640 1 --2- 172.21.15.123:0/2404135864 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f57e40053b0 0x7f57e400ba10 unknown :-1 s=CLOSED pgs=1205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:10.740 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:10.739+0000 7f57ffe83640 1 -- 172.21.15.123:0/2404135864 >> 172.21.15.123:0/2404135864 conn(0x7f57f80d3ea0 msgr2=0x7f57f80d42b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:49:10.740 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:10.739+0000 7f57ffe83640 1 -- 172.21.15.123:0/2404135864 shutdown_connections 2024-09-16T15:49:10.740 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:10.739+0000 7f57ffe83640 1 -- 172.21.15.123:0/2404135864 wait complete. 2024-09-16T15:49:10.740 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:10.739+0000 7f57ffe83640 1 Processor -- start 2024-09-16T15:49:10.740 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:10.739+0000 7f57ffe83640 1 -- start start 2024-09-16T15:49:10.742 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:10.743+0000 7f57ffe83640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f57e40053b0 0x7f57f80cdd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:49:10.742 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:10.743+0000 7f57ffe83640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f57f8156220 0x7f57f80caf70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:49:10.742 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:10.743+0000 7f57ffe83640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f57f8156680 0x7f57f80cb4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:49:10.743 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:10.743+0000 7f57ffe83640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f57f80d2130 con 0x7f57f8156220 2024-09-16T15:49:10.743 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:10.743+0000 7f57ffe83640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f57f80d1fb0 con 0x7f57e40053b0 2024-09-16T15:49:10.743 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:10.743+0000 7f57ffe83640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f57f80d22b0 con 0x7f57f8156680 2024-09-16T15:49:10.743 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:10.743+0000 7f57fd41e640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f57f8156220 0x7f57f80caf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:49:10.743 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:10.743+0000 7f57fd41e640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f57f8156220 0x7f57f80caf70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:55586/0 (socket says 172.21.15.123:55586) 2024-09-16T15:49:10.743 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:10.743+0000 7f57fd41e640 1 -- 172.21.15.123:0/3879850204 learned_addr learned my addr 172.21.15.123:0/3879850204 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:49:10.743 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:10.743+0000 7f57fdc1f640 1 --2- 172.21.15.123:0/3879850204 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f57e40053b0 0x7f57f80cdd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:49:10.743 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:10.743+0000 7f57fe420640 1 --2- 172.21.15.123:0/3879850204 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f57f8156680 0x7f57f80cb4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:49:10.743 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:10.743+0000 7f57fd41e640 1 -- 172.21.15.123:0/3879850204 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f57f8156680 msgr2=0x7f57f80cb4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:10.743 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:10.743+0000 7f57fd41e640 1 --2- 172.21.15.123:0/3879850204 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f57f8156680 0x7f57f80cb4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:10.743 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:10.743+0000 7f57fd41e640 1 -- 172.21.15.123:0/3879850204 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f57e40053b0 msgr2=0x7f57f80cdd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:10.743 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:10.743+0000 7f57fd41e640 1 --2- 172.21.15.123:0/3879850204 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f57e40053b0 0x7f57f80cdd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:10.743 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:10.743+0000 7f57fd41e640 1 -- 172.21.15.123:0/3879850204 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f57f81f1610 con 0x7f57f8156220 2024-09-16T15:49:10.744 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:10.743+0000 7f57fdc1f640 1 --2- 172.21.15.123:0/3879850204 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f57e40053b0 0x7f57f80cdd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T15:49:10.744 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:10.743+0000 7f57fd41e640 1 --2- 172.21.15.123:0/3879850204 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f57f8156220 0x7f57f80caf70 secure :-1 s=READY pgs=1206 cs=0 l=1 rev1=1 crypto rx=0x7f57f400d560 tx=0x7f57f400da20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:49:10.744 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:10.743+0000 7f57ee7fc640 1 -- 172.21.15.123:0/3879850204 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57f4012630 con 0x7f57f8156220 2024-09-16T15:49:10.744 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:10.743+0000 7f57ee7fc640 1 -- 172.21.15.123:0/3879850204 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f57f4017070 con 0x7f57f8156220 2024-09-16T15:49:10.744 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:10.743+0000 7f57ffe83640 1 -- 172.21.15.123:0/3879850204 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f57f80ccfc0 con 0x7f57f8156220 2024-09-16T15:49:10.744 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:10.743+0000 7f57ee7fc640 1 -- 172.21.15.123:0/3879850204 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57f4012a50 con 0x7f57f8156220 2024-09-16T15:49:10.744 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:10.743+0000 7f57ffe83640 1 -- 172.21.15.123:0/3879850204 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f57f80cd580 con 0x7f57f8156220 2024-09-16T15:49:10.745 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:10.743+0000 7f57ffe83640 1 -- 172.21.15.123:0/3879850204 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f57f80d2810 con 0x7f57f8156220 2024-09-16T15:49:10.745 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:10.743+0000 7f57ee7fc640 1 -- 172.21.15.123:0/3879850204 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 19) ==== 97488+0+0 (secure 0 0 0) 0x7f57f4012bf0 con 0x7f57f8156220 2024-09-16T15:49:10.745 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:10.743+0000 7f57ee7fc640 1 --2- 172.21.15.123:0/3879850204 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f57dc073340 0x7f57dc075800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:49:10.745 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:10.743+0000 7f57ee7fc640 1 -- 172.21.15.123:0/3879850204 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(110..110 src has 1..110) ==== 4971+0+0 (secure 0 0 0) 0x7f57f4093f60 con 0x7f57f8156220 2024-09-16T15:49:10.745 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:10.743+0000 7f57fdc1f640 1 --2- 172.21.15.123:0/3879850204 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f57dc073340 0x7f57dc075800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:49:10.745 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:10.743+0000 7f57fdc1f640 1 --2- 172.21.15.123:0/3879850204 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f57dc073340 0x7f57dc075800 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f57e8006860 tx=0x7f57e8001140 comp rx=0 tx=0).ready entity=mgr.4150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:49:10.748 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:10.747+0000 7f57ee7fc640 1 -- 172.21.15.123:0/3879850204 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f57f405edf0 con 0x7f57f8156220 2024-09-16T15:49:10.976 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:10.975+0000 7f57ffe83640 1 -- 172.21.15.123:0/3879850204 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_volumes.TestMisc.test_subvolume_no_upgrade_v1_sanity"]} v 0) -- 0x7f57f815c450 con 0x7f57f8156220 2024-09-16T15:49:11.264 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:11.263+0000 7f57ee7fc640 1 -- 172.21.15.123:0/3879850204 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_volumes.TestMisc.test_subvolume_no_upgrade_v1_sanity"]}]=0 v329) ==== 149+0+0 (secure 0 0 0) 0x7f57f405e6f0 con 0x7f57f8156220 2024-09-16T15:49:11.268 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:11.267+0000 7f57ffe83640 1 -- 172.21.15.123:0/3879850204 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f57dc073340 msgr2=0x7f57dc075800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:11.268 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:11.267+0000 7f57ffe83640 1 --2- 172.21.15.123:0/3879850204 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f57dc073340 0x7f57dc075800 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f57e8006860 tx=0x7f57e8001140 comp rx=0 tx=0).stop 2024-09-16T15:49:11.268 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:11.267+0000 7f57ffe83640 1 -- 172.21.15.123:0/3879850204 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f57f8156220 msgr2=0x7f57f80caf70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:11.268 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:11.267+0000 7f57ffe83640 1 --2- 172.21.15.123:0/3879850204 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f57f8156220 0x7f57f80caf70 secure :-1 s=READY pgs=1206 cs=0 l=1 rev1=1 crypto rx=0x7f57f400d560 tx=0x7f57f400da20 comp rx=0 tx=0).stop 2024-09-16T15:49:11.268 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:11.267+0000 7f57ffe83640 1 -- 172.21.15.123:0/3879850204 shutdown_connections 2024-09-16T15:49:11.268 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:11.267+0000 7f57ffe83640 1 --2- 172.21.15.123:0/3879850204 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f57dc073340 0x7f57dc075800 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:11.268 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:11.267+0000 7f57ffe83640 1 --2- 172.21.15.123:0/3879850204 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f57f8156680 0x7f57f80cb4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:11.268 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:11.267+0000 7f57ffe83640 1 --2- 172.21.15.123:0/3879850204 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f57f8156220 0x7f57f80caf70 unknown :-1 s=CLOSED pgs=1206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:11.268 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:11.267+0000 7f57ffe83640 1 --2- 172.21.15.123:0/3879850204 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f57e40053b0 0x7f57f80cdd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:11.268 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:11.267+0000 7f57ffe83640 1 -- 172.21.15.123:0/3879850204 >> 172.21.15.123:0/3879850204 conn(0x7f57f80d3ea0 msgr2=0x7f57f80cf130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:49:11.268 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:11.267+0000 7f57ffe83640 1 -- 172.21.15.123:0/3879850204 shutdown_connections 2024-09-16T15:49:11.268 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:11.267+0000 7f57ffe83640 1 -- 172.21.15.123:0/3879850204 wait complete. 2024-09-16T15:49:11.289 INFO:tasks.cephfs_test_runner:poor man's upgrade test -- theme continues... ... ok 2024-09-16T15:49:11.290 INFO:tasks.cephfs_test_runner:Starting test: test_subvolume_no_upgrade_v1_to_v2 (tasks.cephfs.test_volumes.TestMisc) poor man's upgrade test -- theme continues... 2024-09-16T15:49:11.290 INFO:tasks.cephfs_test_runner:test_subvolume_no_upgrade_v1_to_v2 (tasks.cephfs.test_volumes.TestMisc) 2024-09-16T15:49:11.290 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-16T15:49:11.419 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:11.419+0000 7f26f5aa6640 1 -- 172.21.15.123:0/1685913688 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f26f015e160 msgr2=0x7f26f0073f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:11.419 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:11.419+0000 7f26f5aa6640 1 --2- 172.21.15.123:0/1685913688 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f26f015e160 0x7f26f0073f30 secure :-1 s=READY pgs=1207 cs=0 l=1 rev1=1 crypto rx=0x7f26f00c3570 tx=0x7f26e001c510 comp rx=0 tx=0).stop 2024-09-16T15:49:11.419 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:11.419+0000 7f26f5aa6640 1 -- 172.21.15.123:0/1685913688 shutdown_connections 2024-09-16T15:49:11.419 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:11.419+0000 7f26f5aa6640 1 --2- 172.21.15.123:0/1685913688 >> v2:172.21.15.184:3300/0 conn(0x7f26f00cdc20 0x7f26f00ca860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:11.419 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:11.419+0000 7f26f5aa6640 1 --2- 172.21.15.123:0/1685913688 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f26f0074470 0x7f26f00cd6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:11.419 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:11.419+0000 7f26f5aa6640 1 --2- 172.21.15.123:0/1685913688 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f26f015e160 0x7f26f0073f30 unknown :-1 s=CLOSED pgs=1207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:11.419 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:11.419+0000 7f26f5aa6640 1 -- 172.21.15.123:0/1685913688 >> 172.21.15.123:0/1685913688 conn(0x7f26f00cfb10 msgr2=0x7f26f00cff20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:49:11.420 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:11.419+0000 7f26f5aa6640 1 -- 172.21.15.123:0/1685913688 shutdown_connections 2024-09-16T15:49:11.420 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:11.419+0000 7f26f5aa6640 1 -- 172.21.15.123:0/1685913688 wait complete. 2024-09-16T15:49:11.420 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:11.419+0000 7f26f5aa6640 1 Processor -- start 2024-09-16T15:49:11.420 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:11.419+0000 7f26f5aa6640 1 -- start start 2024-09-16T15:49:11.423 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:11.423+0000 7f26f5aa6640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f26f0074470 0x7f26f0151fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:49:11.423 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:11.423+0000 7f26f5aa6640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f26f00cdc20 0x7f26f00c8e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:49:11.423 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:11.423+0000 7f26f5aa6640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f26f015e160 0x7f26f0157d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:49:11.423 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:11.423+0000 7f26f5aa6640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f26f015ecf0 con 0x7f26f00cdc20 2024-09-16T15:49:11.423 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:11.423+0000 7f26f5aa6640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f26f015eb70 con 0x7f26f0074470 2024-09-16T15:49:11.423 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:11.423+0000 7f26f5aa6640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f26f015ee70 con 0x7f26f015e160 2024-09-16T15:49:11.423 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:11.423+0000 7f26eeffd640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f26f00cdc20 0x7f26f00c8e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:49:11.423 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:11.423+0000 7f26eeffd640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f26f00cdc20 0x7f26f00c8e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:55592/0 (socket says 172.21.15.123:55592) 2024-09-16T15:49:11.423 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:11.423+0000 7f26eeffd640 1 -- 172.21.15.123:0/220327926 learned_addr learned my addr 172.21.15.123:0/220327926 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:49:11.424 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:11.423+0000 7f26ef7fe640 1 --2- 172.21.15.123:0/220327926 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f26f0074470 0x7f26f0151fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:49:11.424 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:11.423+0000 7f26effff640 1 --2- 172.21.15.123:0/220327926 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f26f015e160 0x7f26f0157d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:49:11.424 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:11.423+0000 7f26eeffd640 1 -- 172.21.15.123:0/220327926 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f26f015e160 msgr2=0x7f26f0157d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:11.424 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:11.423+0000 7f26eeffd640 1 --2- 172.21.15.123:0/220327926 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f26f015e160 0x7f26f0157d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:11.424 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:11.423+0000 7f26eeffd640 1 -- 172.21.15.123:0/220327926 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f26f0074470 msgr2=0x7f26f0151fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:11.424 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:11.423+0000 7f26eeffd640 1 --2- 172.21.15.123:0/220327926 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f26f0074470 0x7f26f0151fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:11.424 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:11.423+0000 7f26eeffd640 1 -- 172.21.15.123:0/220327926 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f26f0158780 con 0x7f26f00cdc20 2024-09-16T15:49:11.424 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:11.423+0000 7f26eeffd640 1 --2- 172.21.15.123:0/220327926 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f26f00cdc20 0x7f26f00c8e30 secure :-1 s=READY pgs=1208 cs=0 l=1 rev1=1 crypto rx=0x7f26e400d730 tx=0x7f26e400dbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:49:11.424 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:11.423+0000 7f26d7fff640 1 -- 172.21.15.123:0/220327926 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26e4017070 con 0x7f26f00cdc20 2024-09-16T15:49:11.424 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:11.423+0000 7f26d7fff640 1 -- 172.21.15.123:0/220327926 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f26e4009a30 con 0x7f26f00cdc20 2024-09-16T15:49:11.424 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:11.423+0000 7f26f5aa6640 1 -- 172.21.15.123:0/220327926 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f26f0158a60 con 0x7f26f00cdc20 2024-09-16T15:49:11.424 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:11.423+0000 7f26d7fff640 1 -- 172.21.15.123:0/220327926 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26e4009e60 con 0x7f26f00cdc20 2024-09-16T15:49:11.425 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:11.423+0000 7f26f5aa6640 1 -- 172.21.15.123:0/220327926 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f26f00c8910 con 0x7f26f00cdc20 2024-09-16T15:49:11.425 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:11.423+0000 7f26f5aa6640 1 -- 172.21.15.123:0/220327926 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f26f00ca860 con 0x7f26f00cdc20 2024-09-16T15:49:11.425 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:11.423+0000 7f26d7fff640 1 -- 172.21.15.123:0/220327926 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 19) ==== 97488+0+0 (secure 0 0 0) 0x7f26e4004020 con 0x7f26f00cdc20 2024-09-16T15:49:11.425 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:11.423+0000 7f26d7fff640 1 --2- 172.21.15.123:0/220327926 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f26dc073340 0x7f26dc075800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:49:11.426 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:11.423+0000 7f26ef7fe640 1 --2- 172.21.15.123:0/220327926 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f26dc073340 0x7f26dc075800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:49:11.426 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:11.423+0000 7f26d7fff640 1 -- 172.21.15.123:0/220327926 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(110..110 src has 1..110) ==== 4971+0+0 (secure 0 0 0) 0x7f26e4094510 con 0x7f26f00cdc20 2024-09-16T15:49:11.426 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:11.423+0000 7f26ef7fe640 1 --2- 172.21.15.123:0/220327926 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f26dc073340 0x7f26dc075800 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f26e00af000 tx=0x7f26e001adb0 comp rx=0 tx=0).ready entity=mgr.4150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:49:11.428 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:11.427+0000 7f26d7fff640 1 -- 172.21.15.123:0/220327926 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f26e405f270 con 0x7f26f00cdc20 2024-09-16T15:49:11.602 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:11.603+0000 7f26f5aa6640 1 -- 172.21.15.123:0/220327926 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f26f01545d0 con 0x7f26f00cdc20 2024-09-16T15:49:11.604 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:11.603+0000 7f26d7fff640 1 -- 172.21.15.123:0/220327926 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v110) ==== 74+0+13148 (secure 0 0 0) 0x7f26e405eb70 con 0x7f26f00cdc20 2024-09-16T15:49:11.604 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-09-16T15:49:11.604 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":110,"fsid":"56077f89-18c8-486f-ae7e-eecc0bb7822c","created":"2024-09-16T15:42:26.364892+0000","modified":"2024-09-16T15:49:10.253748+0000","last_up_change":"2024-09-16T15:42:38.707234+0000","last_in_change":"2024-09-16T15:42:29.423662+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":17,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-16T15:42:39.287764+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9e7c1955-a607-4671-9be7-f2ef760ceeaa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6800","nonce":3882044969},{"type":"v1","addr":"172.21.15.123:6801","nonce":3882044969}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6802","nonce":3882044969},{"type":"v1","addr":"172.21.15.123:6803","nonce":3882044969}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6806","nonce":3882044969},{"type":"v1","addr":"172.21.15.123:6807","nonce":3882044969}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6804","nonce":3882044969},{"type":"v1","addr":"172.21.15.123:6805","nonce":3882044969}]},"public_addr":"172.21.15.123:6801/3882044969","cluster_addr":"172.21.15.123:6803/3882044969","heartbeat_back_addr":"172.21.15.123:6807/3882044969","heartbeat_front_addr":"172.21.15.123:6805/3882044969","state":["exists","up"]},{"osd":1,"uuid":"c540cfde-0977-4742-9b17-fd634668e010","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6816","nonce":4037001850},{"type":"v1","addr":"172.21.15.123:6817","nonce":4037001850}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6818","nonce":4037001850},{"type":"v1","addr":"172.21.15.123:6819","nonce":4037001850}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6822","nonce":4037001850},{"type":"v1","addr":"172.21.15.123:6823","nonce":4037001850}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6820","nonce":4037001850},{"type":"v1","addr":"172.21.15.123:6821","nonce":4037001850}]},"public_addr":"172.21.15.123:6817/4037001850","cluster_addr":"172.21.15.123:6819/4037001850","heartbeat_back_addr":"172.21.15.123:6823/4037001850","heartbeat_front_addr":"172.21.15.123:6821/4037001850","state":["exists","up"]},{"osd":2,"uuid":"9864a496-f169-486b-95c4-1d59162793a7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6824","nonce":1332143088},{"type":"v1","addr":"172.21.15.123:6825","nonce":1332143088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6826","nonce":1332143088},{"type":"v1","addr":"172.21.15.123:6827","nonce":1332143088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6830","nonce":1332143088},{"type":"v1","addr":"172.21.15.123:6831","nonce":1332143088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6828","nonce":1332143088},{"type":"v1","addr":"172.21.15.123:6829","nonce":1332143088}]},"public_addr":"172.21.15.123:6825/1332143088","cluster_addr":"172.21.15.123:6827/1332143088","heartbeat_back_addr":"172.21.15.123:6831/1332143088","heartbeat_front_addr":"172.21.15.123:6829/1332143088","state":["exists","up"]},{"osd":3,"uuid":"c6d1f817-f074-4325-95e1-578eba13edda","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6808","nonce":988586998},{"type":"v1","addr":"172.21.15.123:6809","nonce":988586998}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6810","nonce":988586998},{"type":"v1","addr":"172.21.15.123:6811","nonce":988586998}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6814","nonce":988586998},{"type":"v1","addr":"172.21.15.123:6815","nonce":988586998}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6812","nonce":988586998},{"type":"v1","addr":"172.21.15.123:6813","nonce":988586998}]},"public_addr":"172.21.15.123:6809/988586998","cluster_addr":"172.21.15.123:6811/988586998","heartbeat_back_addr":"172.21.15.123:6815/988586998","heartbeat_front_addr":"172.21.15.123:6813/988586998","state":["exists","up"]},{"osd":4,"uuid":"200a8df4-50ef-40a5-b48a-58981269c086","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6816","nonce":3387813890},{"type":"v1","addr":"172.21.15.184:6817","nonce":3387813890}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6818","nonce":3387813890},{"type":"v1","addr":"172.21.15.184:6819","nonce":3387813890}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6822","nonce":3387813890},{"type":"v1","addr":"172.21.15.184:6823","nonce":3387813890}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6820","nonce":3387813890},{"type":"v1","addr":"172.21.15.184:6821","nonce":3387813890}]},"public_addr":"172.21.15.184:6817/3387813890","cluster_addr":"172.21.15.184:6819/3387813890","heartbeat_back_addr":"172.21.15.184:6823/3387813890","heartbeat_front_addr":"172.21.15.184:6821/3387813890","state":["exists","up"]},{"osd":5,"uuid":"32d5a900-da3d-412c-bc19-13861a2368d8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":104,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6808","nonce":2680916017},{"type":"v1","addr":"172.21.15.184:6809","nonce":2680916017}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6810","nonce":2680916017},{"type":"v1","addr":"172.21.15.184:6811","nonce":2680916017}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6814","nonce":2680916017},{"type":"v1","addr":"172.21.15.184:6815","nonce":2680916017}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6812","nonce":2680916017},{"type":"v1","addr":"172.21.15.184:6813","nonce":2680916017}]},"public_addr":"172.21.15.184:6809/2680916017","cluster_addr":"172.21.15.184:6811/2680916017","heartbeat_back_addr":"172.21.15.184:6815/2680916017","heartbeat_front_addr":"172.21.15.184:6813/2680916017","state":["exists","up"]},{"osd":6,"uuid":"cde6f39e-aa1b-4260-ac19-774ee6aaca72","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":104,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6824","nonce":2183120457},{"type":"v1","addr":"172.21.15.184:6825","nonce":2183120457}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6826","nonce":2183120457},{"type":"v1","addr":"172.21.15.184:6827","nonce":2183120457}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6830","nonce":2183120457},{"type":"v1","addr":"172.21.15.184:6831","nonce":2183120457}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6828","nonce":2183120457},{"type":"v1","addr":"172.21.15.184:6829","nonce":2183120457}]},"public_addr":"172.21.15.184:6825/2183120457","cluster_addr":"172.21.15.184:6827/2183120457","heartbeat_back_addr":"172.21.15.184:6831/2183120457","heartbeat_front_addr":"172.21.15.184:6829/2183120457","state":["exists","up"]},{"osd":7,"uuid":"4f381778-eee1-47c6-804f-df0e6b33e313","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6800","nonce":4102957285},{"type":"v1","addr":"172.21.15.184:6801","nonce":4102957285}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6802","nonce":4102957285},{"type":"v1","addr":"172.21.15.184:6803","nonce":4102957285}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6806","nonce":4102957285},{"type":"v1","addr":"172.21.15.184:6807","nonce":4102957285}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6804","nonce":4102957285},{"type":"v1","addr":"172.21.15.184:6805","nonce":4102957285}]},"public_addr":"172.21.15.184:6801/4102957285","cluster_addr":"172.21.15.184:6803/4102957285","heartbeat_back_addr":"172.21.15.184:6807/4102957285","heartbeat_front_addr":"172.21.15.184:6805/4102957285","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:35.972210+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.222007+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.198617+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.385663+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.358935+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.156066+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.752973+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:35.682846+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.123:6835/2302124587":"2024-09-17T15:49:06.870924+0000","172.21.15.123:6834/2302124587":"2024-09-17T15:49:06.870924+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":17,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T15:49:11.606 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:11.607+0000 7f26f5aa6640 1 -- 172.21.15.123:0/220327926 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f26dc073340 msgr2=0x7f26dc075800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:11.606 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:11.607+0000 7f26f5aa6640 1 --2- 172.21.15.123:0/220327926 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f26dc073340 0x7f26dc075800 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f26e00af000 tx=0x7f26e001adb0 comp rx=0 tx=0).stop 2024-09-16T15:49:11.606 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:11.607+0000 7f26f5aa6640 1 -- 172.21.15.123:0/220327926 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f26f00cdc20 msgr2=0x7f26f00c8e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:11.606 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:11.607+0000 7f26f5aa6640 1 --2- 172.21.15.123:0/220327926 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f26f00cdc20 0x7f26f00c8e30 secure :-1 s=READY pgs=1208 cs=0 l=1 rev1=1 crypto rx=0x7f26e400d730 tx=0x7f26e400dbf0 comp rx=0 tx=0).stop 2024-09-16T15:49:11.606 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:11.607+0000 7f26f5aa6640 1 -- 172.21.15.123:0/220327926 shutdown_connections 2024-09-16T15:49:11.607 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:11.607+0000 7f26f5aa6640 1 --2- 172.21.15.123:0/220327926 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f26dc073340 0x7f26dc075800 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:11.607 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:11.607+0000 7f26f5aa6640 1 --2- 172.21.15.123:0/220327926 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f26f015e160 0x7f26f0157d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:11.607 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:11.607+0000 7f26f5aa6640 1 --2- 172.21.15.123:0/220327926 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f26f00cdc20 0x7f26f00c8e30 unknown :-1 s=CLOSED pgs=1208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:11.607 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:11.607+0000 7f26f5aa6640 1 --2- 172.21.15.123:0/220327926 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f26f0074470 0x7f26f0151fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:11.607 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:11.607+0000 7f26f5aa6640 1 -- 172.21.15.123:0/220327926 >> 172.21.15.123:0/220327926 conn(0x7f26f00cfb10 msgr2=0x7f26f0157570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:49:11.607 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:11.607+0000 7f26f5aa6640 1 -- 172.21.15.123:0/220327926 shutdown_connections 2024-09-16T15:49:11.607 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:11.607+0000 7f26f5aa6640 1 -- 172.21.15.123:0/220327926 wait complete. 2024-09-16T15:49:11.624 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-16T15:42:39.287764+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-09-16T15:49:11.624 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-09-16T15:49:11.750 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:11.751+0000 7f738c278640 1 -- 172.21.15.123:0/2036799673 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7370004f50 msgr2=0x7f737000b9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:11.751 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:11.751+0000 7f738c278640 1 --2- 172.21.15.123:0/2036799673 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7370004f50 0x7f737000b9d0 secure :-1 s=READY pgs=1210 cs=0 l=1 rev1=1 crypto rx=0x7f737c00c640 tx=0x7f737c01d260 comp rx=0 tx=0).stop 2024-09-16T15:49:11.751 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:11.751+0000 7f738a815640 1 -- 172.21.15.123:0/2036799673 reap_dead start 2024-09-16T15:49:11.751 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:11.751+0000 7f738c278640 1 -- 172.21.15.123:0/2036799673 shutdown_connections 2024-09-16T15:49:11.751 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:11.751+0000 7f738c278640 1 -- 172.21.15.123:0/2036799673 >> 172.21.15.123:0/2036799673 conn(0x7f73840d3e40 msgr2=0x7f73840d4220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:49:11.751 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:11.751+0000 7f738c278640 1 -- 172.21.15.123:0/2036799673 shutdown_connections 2024-09-16T15:49:11.751 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:11.751+0000 7f738c278640 1 -- 172.21.15.123:0/2036799673 wait complete. 2024-09-16T15:49:11.751 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:11.751+0000 7f738c278640 1 Processor -- start 2024-09-16T15:49:11.752 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:11.751+0000 7f738c278640 1 -- start start 2024-09-16T15:49:11.754 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:11.751+0000 7f738c278640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7370004f50 0x7f7384154e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:49:11.754 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:11.751+0000 7f738c278640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7384161b40 0x7f738415bb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:49:11.754 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:11.751+0000 7f738c278640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f7384161fc0 0x7f738415c0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:49:11.754 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:11.751+0000 7f738c278640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f738416a7a0 con 0x7f7384161b40 2024-09-16T15:49:11.754 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:11.751+0000 7f738c278640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f738416a620 con 0x7f7370004f50 2024-09-16T15:49:11.754 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:11.751+0000 7f738c278640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f738416a920 con 0x7f7384161fc0 2024-09-16T15:49:11.754 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:11.751+0000 7f7389813640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7384161b40 0x7f738415bb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:49:11.755 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:11.751+0000 7f7389813640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7384161b40 0x7f738415bb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:55614/0 (socket says 172.21.15.123:55614) 2024-09-16T15:49:11.755 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:11.751+0000 7f7389813640 1 -- 172.21.15.123:0/4073964073 learned_addr learned my addr 172.21.15.123:0/4073964073 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:49:11.755 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:11.751+0000 7f738a014640 1 --2- 172.21.15.123:0/4073964073 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7370004f50 0x7f7384154e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:49:11.755 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:11.751+0000 7f738a815640 1 --2- 172.21.15.123:0/4073964073 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f7384161fc0 0x7f738415c0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:49:11.755 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:11.751+0000 7f7389813640 1 -- 172.21.15.123:0/4073964073 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f7384161fc0 msgr2=0x7f738415c0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:11.755 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:11.751+0000 7f7389813640 1 --2- 172.21.15.123:0/4073964073 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f7384161fc0 0x7f738415c0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:11.755 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:11.751+0000 7f7389813640 1 -- 172.21.15.123:0/4073964073 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7370004f50 msgr2=0x7f7384154e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:11.755 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:11.751+0000 7f7389813640 1 --2- 172.21.15.123:0/4073964073 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7370004f50 0x7f7384154e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:11.755 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:11.751+0000 7f7389813640 1 -- 172.21.15.123:0/4073964073 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f73841639b0 con 0x7f7384161b40 2024-09-16T15:49:11.755 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:11.755+0000 7f738a815640 1 --2- 172.21.15.123:0/4073964073 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f7384161fc0 0x7f738415c0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T15:49:11.755 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:11.755+0000 7f7389813640 1 --2- 172.21.15.123:0/4073964073 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7384161b40 0x7f738415bb70 secure :-1 s=READY pgs=1211 cs=0 l=1 rev1=1 crypto rx=0x7f737400e7a0 tx=0x7f737400ec60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:49:11.755 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:11.755+0000 7f737affd640 1 -- 172.21.15.123:0/4073964073 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7374016070 con 0x7f7384161b40 2024-09-16T15:49:11.756 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:11.755+0000 7f737affd640 1 -- 172.21.15.123:0/4073964073 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f737400c7f0 con 0x7f7384161b40 2024-09-16T15:49:11.756 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:11.755+0000 7f737affd640 1 -- 172.21.15.123:0/4073964073 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f737400cc20 con 0x7f7384161b40 2024-09-16T15:49:11.756 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:11.755+0000 7f738c278640 1 -- 172.21.15.123:0/4073964073 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7384167980 con 0x7f7384161b40 2024-09-16T15:49:11.756 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:11.755+0000 7f738c278640 1 -- 172.21.15.123:0/4073964073 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7384163f10 con 0x7f7384161b40 2024-09-16T15:49:11.756 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:11.755+0000 7f738c278640 1 -- 172.21.15.123:0/4073964073 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f73840d0850 con 0x7f7384161b40 2024-09-16T15:49:11.757 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:11.755+0000 7f737affd640 1 -- 172.21.15.123:0/4073964073 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 19) ==== 97488+0+0 (secure 0 0 0) 0x7f7374019070 con 0x7f7384161b40 2024-09-16T15:49:11.757 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:11.755+0000 7f737affd640 1 --2- 172.21.15.123:0/4073964073 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f7368073340 0x7f7368075800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:49:11.757 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:11.755+0000 7f738a014640 1 --2- 172.21.15.123:0/4073964073 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f7368073340 0x7f7368075800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:49:11.757 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:11.755+0000 7f737affd640 1 -- 172.21.15.123:0/4073964073 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(110..110 src has 1..110) ==== 4971+0+0 (secure 0 0 0) 0x7f7374094a60 con 0x7f7384161b40 2024-09-16T15:49:11.757 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:11.755+0000 7f738a014640 1 --2- 172.21.15.123:0/4073964073 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f7368073340 0x7f7368075800 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f7380001810 tx=0x7f7380006b70 comp rx=0 tx=0).ready entity=mgr.4150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:49:11.759 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:11.759+0000 7f737affd640 1 -- 172.21.15.123:0/4073964073 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f73841639b0 con 0x7f7384161b40 2024-09-16T15:49:11.931 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:11.931+0000 7f738c278640 1 -- 172.21.15.123:0/4073964073 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f738415fc80 con 0x7f7384161b40 2024-09-16T15:49:11.932 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:11.931+0000 7f737affd640 1 -- 172.21.15.123:0/4073964073 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v110) ==== 93+0+10 (secure 0 0 0) 0x7f737405f7c0 con 0x7f7384161b40 2024-09-16T15:49:11.932 INFO:teuthology.orchestra.run.smithi123.stdout:pg_num: 1 2024-09-16T15:49:11.934 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:11.935+0000 7f738c278640 1 -- 172.21.15.123:0/4073964073 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f7368073340 msgr2=0x7f7368075800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:11.935 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:11.935+0000 7f738c278640 1 --2- 172.21.15.123:0/4073964073 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f7368073340 0x7f7368075800 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f7380001810 tx=0x7f7380006b70 comp rx=0 tx=0).stop 2024-09-16T15:49:11.935 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:11.935+0000 7f738c278640 1 -- 172.21.15.123:0/4073964073 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7384161b40 msgr2=0x7f738415bb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:11.935 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:11.935+0000 7f738c278640 1 --2- 172.21.15.123:0/4073964073 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7384161b40 0x7f738415bb70 secure :-1 s=READY pgs=1211 cs=0 l=1 rev1=1 crypto rx=0x7f737400e7a0 tx=0x7f737400ec60 comp rx=0 tx=0).stop 2024-09-16T15:49:11.935 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:11.935+0000 7f738c278640 1 -- 172.21.15.123:0/4073964073 shutdown_connections 2024-09-16T15:49:11.935 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:11.935+0000 7f738c278640 1 --2- 172.21.15.123:0/4073964073 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f7368073340 0x7f7368075800 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:11.935 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:11.935+0000 7f738c278640 1 --2- 172.21.15.123:0/4073964073 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f7384161fc0 0x7f738415c0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:11.935 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:11.935+0000 7f738c278640 1 --2- 172.21.15.123:0/4073964073 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7384161b40 0x7f738415bb70 unknown :-1 s=CLOSED pgs=1211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:11.935 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:11.935+0000 7f738c278640 1 --2- 172.21.15.123:0/4073964073 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7370004f50 0x7f7384154e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:11.935 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:11.935+0000 7f738c278640 1 -- 172.21.15.123:0/4073964073 >> 172.21.15.123:0/4073964073 conn(0x7f73840d3e40 msgr2=0x7f73840b27b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:49:11.935 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:11.935+0000 7f738c278640 1 -- 172.21.15.123:0/4073964073 shutdown_connections 2024-09-16T15:49:11.935 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:11.935+0000 7f738c278640 1 -- 172.21.15.123:0/4073964073 wait complete. 2024-09-16T15:49:11.952 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_volumes.TestMisc.test_subvolume_no_upgrade_v1_to_v2' 2024-09-16T15:49:12.077 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:12.079+0000 7f727682b640 1 -- 172.21.15.123:0/303144581 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f725c004d50 msgr2=0x7f725c0051d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:12.077 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:12.079+0000 7f727682b640 1 --2- 172.21.15.123:0/303144581 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f725c004d50 0x7f725c0051d0 secure :-1 s=READY pgs=1213 cs=0 l=1 rev1=1 crypto rx=0x7f7268009e30 tx=0x7f726801d1b0 comp rx=0 tx=0).stop 2024-09-16T15:49:12.077 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:12.079+0000 7f7274dc8640 1 -- 172.21.15.123:0/303144581 reap_dead start 2024-09-16T15:49:12.078 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:12.079+0000 7f727682b640 1 -- 172.21.15.123:0/303144581 shutdown_connections 2024-09-16T15:49:12.078 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:12.079+0000 7f727682b640 1 -- 172.21.15.123:0/303144581 >> 172.21.15.123:0/303144581 conn(0x7f72700cfb10 msgr2=0x7f72700cff20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:49:12.078 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:12.079+0000 7f727682b640 1 -- 172.21.15.123:0/303144581 shutdown_connections 2024-09-16T15:49:12.078 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:12.079+0000 7f727682b640 1 -- 172.21.15.123:0/303144581 wait complete. 2024-09-16T15:49:12.078 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:12.079+0000 7f727682b640 1 Processor -- start 2024-09-16T15:49:12.078 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:12.079+0000 7f727682b640 1 -- start start 2024-09-16T15:49:12.080 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:12.079+0000 7f727682b640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f725c004d50 0x7f7270157ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:49:12.080 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:12.079+0000 7f727682b640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f72700cd6e0 0x7f72700cdb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:49:12.080 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:12.079+0000 7f727682b640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f727015c240 0x7f727015c6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:49:12.080 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:12.079+0000 7f727682b640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f72700d09c0 con 0x7f72700cd6e0 2024-09-16T15:49:12.080 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:12.079+0000 7f727682b640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f72700d0840 con 0x7f727015c240 2024-09-16T15:49:12.081 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:12.079+0000 7f727682b640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f727006ec90 con 0x7f725c004d50 2024-09-16T15:49:12.081 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:12.079+0000 7f726f7fe640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f72700cd6e0 0x7f72700cdb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:49:12.081 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:12.079+0000 7f726f7fe640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f72700cd6e0 0x7f72700cdb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:55640/0 (socket says 172.21.15.123:55640) 2024-09-16T15:49:12.081 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:12.079+0000 7f726f7fe640 1 -- 172.21.15.123:0/4290104210 learned_addr learned my addr 172.21.15.123:0/4290104210 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:49:12.081 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:12.079+0000 7f726ffff640 1 --2- 172.21.15.123:0/4290104210 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f725c004d50 0x7f7270157ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:49:12.081 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:12.079+0000 7f7274dc8640 1 --2- 172.21.15.123:0/4290104210 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f727015c240 0x7f727015c6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:49:12.081 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:12.079+0000 7f726f7fe640 1 -- 172.21.15.123:0/4290104210 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f725c004d50 msgr2=0x7f7270157ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:12.081 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:12.079+0000 7f726f7fe640 1 --2- 172.21.15.123:0/4290104210 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f725c004d50 0x7f7270157ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:12.081 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:12.079+0000 7f726f7fe640 1 -- 172.21.15.123:0/4290104210 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f727015c240 msgr2=0x7f727015c6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:12.081 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:12.079+0000 7f726f7fe640 1 --2- 172.21.15.123:0/4290104210 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f727015c240 0x7f727015c6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:12.081 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:12.079+0000 7f726f7fe640 1 -- 172.21.15.123:0/4290104210 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f72700ce160 con 0x7f72700cd6e0 2024-09-16T15:49:12.081 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:12.079+0000 7f726f7fe640 1 --2- 172.21.15.123:0/4290104210 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f72700cd6e0 0x7f72700cdb60 secure :-1 s=READY pgs=1214 cs=0 l=1 rev1=1 crypto rx=0x7f725800f640 tx=0x7f725800fb00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:49:12.082 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:12.079+0000 7f726cff9640 1 -- 172.21.15.123:0/4290104210 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7258017070 con 0x7f72700cd6e0 2024-09-16T15:49:12.082 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:12.079+0000 7f726cff9640 1 -- 172.21.15.123:0/4290104210 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f725800aa10 con 0x7f72700cd6e0 2024-09-16T15:49:12.082 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:12.079+0000 7f727682b640 1 -- 172.21.15.123:0/4290104210 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f72700ce440 con 0x7f72700cd6e0 2024-09-16T15:49:12.082 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:12.079+0000 7f726cff9640 1 -- 172.21.15.123:0/4290104210 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f725800ae60 con 0x7f72700cd6e0 2024-09-16T15:49:12.082 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:12.079+0000 7f727682b640 1 -- 172.21.15.123:0/4290104210 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f72700cea00 con 0x7f72700cd6e0 2024-09-16T15:49:12.083 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:12.079+0000 7f727682b640 1 -- 172.21.15.123:0/4290104210 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f727006e750 con 0x7f72700cd6e0 2024-09-16T15:49:12.083 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:12.083+0000 7f726cff9640 1 -- 172.21.15.123:0/4290104210 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 19) ==== 97488+0+0 (secure 0 0 0) 0x7f725801a070 con 0x7f72700cd6e0 2024-09-16T15:49:12.083 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:12.083+0000 7f726cff9640 1 --2- 172.21.15.123:0/4290104210 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f7250073340 0x7f7250075800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:49:12.083 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:12.083+0000 7f726cff9640 1 -- 172.21.15.123:0/4290104210 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(110..110 src has 1..110) ==== 4971+0+0 (secure 0 0 0) 0x7f72580974f0 con 0x7f72700cd6e0 2024-09-16T15:49:12.083 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:12.083+0000 7f726ffff640 1 --2- 172.21.15.123:0/4290104210 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f7250073340 0x7f7250075800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:49:12.083 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:12.083+0000 7f726ffff640 1 --2- 172.21.15.123:0/4290104210 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f7250073340 0x7f7250075800 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f7260005fd0 tx=0x7f7260005f40 comp rx=0 tx=0).ready entity=mgr.4150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:49:12.086 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:12.087+0000 7f726cff9640 1 -- 172.21.15.123:0/4290104210 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f7258062250 con 0x7f72700cd6e0 2024-09-16T15:49:12.306 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:12.307+0000 7f727682b640 1 -- 172.21.15.123:0/4290104210 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_volumes.TestMisc.test_subvolume_no_upgrade_v1_to_v2"]} v 0) -- 0x7f727015f630 con 0x7f72700cd6e0 2024-09-16T15:49:13.275 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:13.275+0000 7f726cff9640 1 -- 172.21.15.123:0/4290104210 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_volumes.TestMisc.test_subvolume_no_upgrade_v1_to_v2"]}]=0 v331) ==== 151+0+0 (secure 0 0 0) 0x7f7258061b50 con 0x7f72700cd6e0 2024-09-16T15:49:13.280 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:13.279+0000 7f727682b640 1 -- 172.21.15.123:0/4290104210 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f7250073340 msgr2=0x7f7250075800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:13.280 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:13.279+0000 7f727682b640 1 --2- 172.21.15.123:0/4290104210 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f7250073340 0x7f7250075800 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f7260005fd0 tx=0x7f7260005f40 comp rx=0 tx=0).stop 2024-09-16T15:49:13.280 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:13.279+0000 7f727682b640 1 -- 172.21.15.123:0/4290104210 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f72700cd6e0 msgr2=0x7f72700cdb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:13.280 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:13.279+0000 7f727682b640 1 --2- 172.21.15.123:0/4290104210 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f72700cd6e0 0x7f72700cdb60 secure :-1 s=READY pgs=1214 cs=0 l=1 rev1=1 crypto rx=0x7f725800f640 tx=0x7f725800fb00 comp rx=0 tx=0).stop 2024-09-16T15:49:13.280 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:13.279+0000 7f727682b640 1 -- 172.21.15.123:0/4290104210 shutdown_connections 2024-09-16T15:49:13.280 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:13.279+0000 7f727682b640 1 --2- 172.21.15.123:0/4290104210 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f7250073340 0x7f7250075800 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:13.281 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:13.279+0000 7f727682b640 1 --2- 172.21.15.123:0/4290104210 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f727015c240 0x7f727015c6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:13.281 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:13.279+0000 7f727682b640 1 --2- 172.21.15.123:0/4290104210 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f72700cd6e0 0x7f72700cdb60 unknown :-1 s=CLOSED pgs=1214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:13.281 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:13.279+0000 7f727682b640 1 --2- 172.21.15.123:0/4290104210 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f725c004d50 0x7f7270157ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:13.281 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:13.279+0000 7f727682b640 1 -- 172.21.15.123:0/4290104210 >> 172.21.15.123:0/4290104210 conn(0x7f72700cfb10 msgr2=0x7f72700cc590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:49:13.281 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:13.279+0000 7f727682b640 1 -- 172.21.15.123:0/4290104210 shutdown_connections 2024-09-16T15:49:13.281 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:13.279+0000 7f727682b640 1 -- 172.21.15.123:0/4290104210 wait complete. 2024-09-16T15:49:13.304 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config dump 2024-09-16T15:49:13.434 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:13.435+0000 7f2549b01640 1 -- 172.21.15.123:0/3677305994 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2544074600 msgr2=0x7f254406e750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:13.434 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:13.435+0000 7f2549b01640 1 --2- 172.21.15.123:0/3677305994 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2544074600 0x7f254406e750 secure :-1 s=READY pgs=1216 cs=0 l=1 rev1=1 crypto rx=0x7f25440cd650 tx=0x7f253c01cac0 comp rx=0 tx=0).stop 2024-09-16T15:49:13.434 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:13.435+0000 7f2549b01640 1 -- 172.21.15.123:0/3677305994 shutdown_connections 2024-09-16T15:49:13.434 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:13.435+0000 7f2549b01640 1 --2- 172.21.15.123:0/3677305994 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2544074600 0x7f254406e750 unknown :-1 s=CLOSED pgs=1216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:13.434 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:13.435+0000 7f2549b01640 1 --2- 172.21.15.123:0/3677305994 >> v2:172.21.15.184:3300/0 conn(0x7f254415bca0 0x7f25440d0630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:13.434 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:13.435+0000 7f2549b01640 1 -- 172.21.15.123:0/3677305994 >> 172.21.15.123:0/3677305994 conn(0x7f25440cfb10 msgr2=0x7f25440cff20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:49:13.434 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:13.435+0000 7f2549b01640 1 -- 172.21.15.123:0/3677305994 shutdown_connections 2024-09-16T15:49:13.435 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:13.435+0000 7f2549b01640 1 -- 172.21.15.123:0/3677305994 wait complete. 2024-09-16T15:49:13.435 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:13.435+0000 7f2549b01640 1 Processor -- start 2024-09-16T15:49:13.435 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:13.435+0000 7f2549b01640 1 -- start start 2024-09-16T15:49:13.437 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:13.435+0000 7f2549b01640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f25440726c0 0x7f254414e910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:49:13.437 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:13.435+0000 7f2549b01640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f2544074600 0x7f2544159ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:49:13.437 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:13.435+0000 7f2549b01640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f254415bca0 0x7f254415a420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:49:13.437 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:13.435+0000 7f2549b01640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f2544168a50 con 0x7f25440726c0 2024-09-16T15:49:13.437 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:13.435+0000 7f2549b01640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f254406fa10 con 0x7f254415bca0 2024-09-16T15:49:13.437 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:13.435+0000 7f2549b01640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f2544168bd0 con 0x7f2544074600 2024-09-16T15:49:13.437 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:13.435+0000 7f25437fe640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f25440726c0 0x7f254414e910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:49:13.437 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:13.435+0000 7f25437fe640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f25440726c0 0x7f254414e910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:52148/0 (socket says 172.21.15.123:52148) 2024-09-16T15:49:13.438 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:13.435+0000 7f25437fe640 1 -- 172.21.15.123:0/1024470927 learned_addr learned my addr 172.21.15.123:0/1024470927 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:49:13.438 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:13.435+0000 7f25437fe640 1 -- 172.21.15.123:0/1024470927 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f2544074600 msgr2=0x7f2544159ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:13.438 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:13.435+0000 7f25437fe640 1 --2- 172.21.15.123:0/1024470927 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f2544074600 0x7f2544159ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:13.438 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:13.435+0000 7f25437fe640 1 -- 172.21.15.123:0/1024470927 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f254415bca0 msgr2=0x7f254415a420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:13.438 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:13.435+0000 7f25437fe640 1 --2- 172.21.15.123:0/1024470927 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f254415bca0 0x7f254415a420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:13.438 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:13.435+0000 7f25437fe640 1 -- 172.21.15.123:0/1024470927 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f254415d4e0 con 0x7f25440726c0 2024-09-16T15:49:13.438 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:13.435+0000 7f25437fe640 1 --2- 172.21.15.123:0/1024470927 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f25440726c0 0x7f254414e910 secure :-1 s=READY pgs=1217 cs=0 l=1 rev1=1 crypto rx=0x7f253800c9f0 tx=0x7f253800ceb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:49:13.438 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:13.435+0000 7f252bfff640 1 -- 172.21.15.123:0/1024470927 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2538007bb0 con 0x7f25440726c0 2024-09-16T15:49:13.438 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:13.435+0000 7f252bfff640 1 -- 172.21.15.123:0/1024470927 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f2538007e40 con 0x7f25440726c0 2024-09-16T15:49:13.438 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:13.435+0000 7f2549b01640 1 -- 172.21.15.123:0/1024470927 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f254415d7c0 con 0x7f25440726c0 2024-09-16T15:49:13.438 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:13.435+0000 7f252bfff640 1 -- 172.21.15.123:0/1024470927 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f253800f690 con 0x7f25440726c0 2024-09-16T15:49:13.438 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:13.435+0000 7f2549b01640 1 -- 172.21.15.123:0/1024470927 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f254415dd80 con 0x7f25440726c0 2024-09-16T15:49:13.439 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:13.439+0000 7f2549b01640 1 -- 172.21.15.123:0/1024470927 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f254406e750 con 0x7f25440726c0 2024-09-16T15:49:13.439 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:13.439+0000 7f252bfff640 1 -- 172.21.15.123:0/1024470927 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 19) ==== 97488+0+0 (secure 0 0 0) 0x7f253800f830 con 0x7f25440726c0 2024-09-16T15:49:13.439 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:13.439+0000 7f252bfff640 1 --2- 172.21.15.123:0/1024470927 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f2530073370 0x7f2530075830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:49:13.439 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:13.439+0000 7f252bfff640 1 -- 172.21.15.123:0/1024470927 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(110..110 src has 1..110) ==== 4971+0+0 (secure 0 0 0) 0x7f2538094d00 con 0x7f25440726c0 2024-09-16T15:49:13.440 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:13.439+0000 7f2542ffd640 1 --2- 172.21.15.123:0/1024470927 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f2530073370 0x7f2530075830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:49:13.440 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:13.439+0000 7f2542ffd640 1 --2- 172.21.15.123:0/1024470927 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f2530073370 0x7f2530075830 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f2544165ed0 tx=0x7f2534007a40 comp rx=0 tx=0).ready entity=mgr.4150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:49:13.443 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:13.443+0000 7f252bfff640 1 -- 172.21.15.123:0/1024470927 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f2538014030 con 0x7f25440726c0 2024-09-16T15:49:13.611 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:13.611+0000 7f2549b01640 1 -- 172.21.15.123:0/1024470927 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "config dump"} v 0) -- 0x7f25440700e0 con 0x7f25440726c0 2024-09-16T15:49:13.612 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:13.611+0000 7f252bfff640 1 -- 172.21.15.123:0/1024470927 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "config dump"}]=0 v39) ==== 59+0+1998 (secure 0 0 0) 0x7f253805f9d0 con 0x7f25440726c0 2024-09-16T15:49:13.612 INFO:teuthology.orchestra.run.smithi123.stdout:WHO MASK LEVEL OPTION VALUE RO 2024-09-16T15:49:13.612 INFO:teuthology.orchestra.run.smithi123.stdout:mon advanced mon_op_complaint_time 120 2024-09-16T15:49:13.613 INFO:teuthology.orchestra.run.smithi123.stdout:mon dev mon_warn_on_pool_no_app false 2024-09-16T15:49:13.613 INFO:teuthology.orchestra.run.smithi123.stdout:osd advanced osd_op_complaint_time 180.000000 2024-09-16T15:49:13.613 INFO:teuthology.orchestra.run.smithi123.stdout:osd.0 basic osd_mclock_max_capacity_iops_ssd 4805.300677 2024-09-16T15:49:13.613 INFO:teuthology.orchestra.run.smithi123.stdout:osd.1 basic osd_mclock_max_capacity_iops_ssd 4806.240607 2024-09-16T15:49:13.613 INFO:teuthology.orchestra.run.smithi123.stdout:osd.2 basic osd_mclock_max_capacity_iops_ssd 4046.441686 2024-09-16T15:49:13.613 INFO:teuthology.orchestra.run.smithi123.stdout:osd.3 basic osd_mclock_max_capacity_iops_ssd 6882.576123 2024-09-16T15:49:13.613 INFO:teuthology.orchestra.run.smithi123.stdout:osd.4 basic osd_mclock_max_capacity_iops_ssd 6108.444971 2024-09-16T15:49:13.613 INFO:teuthology.orchestra.run.smithi123.stdout:osd.5 basic osd_mclock_max_capacity_iops_ssd 4995.096306 2024-09-16T15:49:13.613 INFO:teuthology.orchestra.run.smithi123.stdout:osd.6 basic osd_mclock_max_capacity_iops_ssd 6237.822095 2024-09-16T15:49:13.613 INFO:teuthology.orchestra.run.smithi123.stdout:osd.7 basic osd_mclock_max_capacity_iops_ssd 5202.385138 2024-09-16T15:49:13.613 INFO:teuthology.orchestra.run.smithi123.stdout:mds advanced debug_mds 20/20 2024-09-16T15:49:13.613 INFO:teuthology.orchestra.run.smithi123.stdout:mds advanced debug_mds_balancer 20/20 2024-09-16T15:49:13.613 INFO:teuthology.orchestra.run.smithi123.stdout:mds advanced debug_ms 2/2 2024-09-16T15:49:13.614 INFO:teuthology.orchestra.run.smithi123.stdout:mds dev mds_debug_frag true 2024-09-16T15:49:13.614 INFO:teuthology.orchestra.run.smithi123.stdout:mds dev mds_debug_scatterstat true 2024-09-16T15:49:13.614 INFO:teuthology.orchestra.run.smithi123.stdout:mds advanced mds_op_complaint_time 180.000000 2024-09-16T15:49:13.614 INFO:teuthology.orchestra.run.smithi123.stdout:mds dev mds_verify_scatter true 2024-09-16T15:49:13.614 INFO:teuthology.orchestra.run.smithi123.stdout:mds advanced osd_op_complaint_time 180.000000 2024-09-16T15:49:13.614 INFO:teuthology.orchestra.run.smithi123.stdout:mds advanced rados_mon_op_timeout 900 2024-09-16T15:49:13.614 INFO:teuthology.orchestra.run.smithi123.stdout:mds advanced rados_osd_op_timeout 900 2024-09-16T15:49:13.614 INFO:teuthology.orchestra.run.smithi123.stdout:client advanced client_mount_timeout 600 2024-09-16T15:49:13.614 INFO:teuthology.orchestra.run.smithi123.stdout:client advanced debug_client 20/20 2024-09-16T15:49:13.614 INFO:teuthology.orchestra.run.smithi123.stdout:client advanced debug_ms 1/1 2024-09-16T15:49:13.614 INFO:teuthology.orchestra.run.smithi123.stdout:client advanced rados_mon_op_timeout 900 2024-09-16T15:49:13.614 INFO:teuthology.orchestra.run.smithi123.stdout:client advanced rados_osd_op_timeout 900 2024-09-16T15:49:13.615 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:13.615+0000 7f2549b01640 1 -- 172.21.15.123:0/1024470927 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f2530073370 msgr2=0x7f2530075830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:13.615 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:13.615+0000 7f2549b01640 1 --2- 172.21.15.123:0/1024470927 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f2530073370 0x7f2530075830 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f2544165ed0 tx=0x7f2534007a40 comp rx=0 tx=0).stop 2024-09-16T15:49:13.615 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:13.615+0000 7f2549b01640 1 -- 172.21.15.123:0/1024470927 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f25440726c0 msgr2=0x7f254414e910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:13.615 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:13.615+0000 7f2549b01640 1 --2- 172.21.15.123:0/1024470927 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f25440726c0 0x7f254414e910 secure :-1 s=READY pgs=1217 cs=0 l=1 rev1=1 crypto rx=0x7f253800c9f0 tx=0x7f253800ceb0 comp rx=0 tx=0).stop 2024-09-16T15:49:13.615 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:13.615+0000 7f2549b01640 1 -- 172.21.15.123:0/1024470927 shutdown_connections 2024-09-16T15:49:13.615 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:13.615+0000 7f2549b01640 1 --2- 172.21.15.123:0/1024470927 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f2530073370 0x7f2530075830 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:13.616 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:13.615+0000 7f2549b01640 1 --2- 172.21.15.123:0/1024470927 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f254415bca0 0x7f254415a420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:13.616 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:13.615+0000 7f2549b01640 1 --2- 172.21.15.123:0/1024470927 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f2544074600 0x7f2544159ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:13.616 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:13.615+0000 7f2549b01640 1 --2- 172.21.15.123:0/1024470927 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f25440726c0 0x7f254414e910 unknown :-1 s=CLOSED pgs=1217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:13.616 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:13.615+0000 7f2549b01640 1 -- 172.21.15.123:0/1024470927 >> 172.21.15.123:0/1024470927 conn(0x7f25440cfb10 msgr2=0x7f25440b3790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:49:13.616 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:13.615+0000 7f2549b01640 1 -- 172.21.15.123:0/1024470927 shutdown_connections 2024-09-16T15:49:13.616 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:13.615+0000 7f2549b01640 1 -- 172.21.15.123:0/1024470927 wait complete. 2024-09-16T15:49:13.630 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-09-16T15:49:13.750 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:13.751+0000 7f7ed45d3640 1 -- 172.21.15.123:0/3833728385 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7eb8003a10 msgr2=0x7f7eb8003e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:13.750 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:13.751+0000 7f7ed45d3640 1 --2- 172.21.15.123:0/3833728385 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7eb8003a10 0x7f7eb8003e30 secure :-1 s=READY pgs=1219 cs=0 l=1 rev1=1 crypto rx=0x7f7ec8005fd0 tx=0x7f7ec801ccc0 comp rx=0 tx=0).stop 2024-09-16T15:49:13.750 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:13.751+0000 7f7ed2b70640 1 -- 172.21.15.123:0/3833728385 reap_dead start 2024-09-16T15:49:13.750 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:13.751+0000 7f7ed45d3640 1 -- 172.21.15.123:0/3833728385 shutdown_connections 2024-09-16T15:49:13.750 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:13.751+0000 7f7ed45d3640 1 -- 172.21.15.123:0/3833728385 >> 172.21.15.123:0/3833728385 conn(0x7f7ecc0b1710 msgr2=0x7f7ecc0b1b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:49:13.750 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:13.751+0000 7f7ed45d3640 1 -- 172.21.15.123:0/3833728385 shutdown_connections 2024-09-16T15:49:13.750 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:13.751+0000 7f7ed45d3640 1 -- 172.21.15.123:0/3833728385 wait complete. 2024-09-16T15:49:13.751 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:13.751+0000 7f7ed45d3640 1 Processor -- start 2024-09-16T15:49:13.751 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:13.751+0000 7f7ed45d3640 1 -- start start 2024-09-16T15:49:13.753 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:13.751+0000 7f7ed45d3640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7eb8003a10 0x7f7ecc163b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:49:13.753 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:13.751+0000 7f7ed45d3640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f7ecc1644a0 0x7f7ecc156010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:49:13.753 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:13.751+0000 7f7ed45d3640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7ecc15c880 0x7f7ecc15cd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:49:13.753 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:13.751+0000 7f7ed45d3640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f7ecc0ccba0 con 0x7f7eb8003a10 2024-09-16T15:49:13.753 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:13.751+0000 7f7ed45d3640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f7ecc0cca20 con 0x7f7ecc15c880 2024-09-16T15:49:13.753 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:13.751+0000 7f7ed45d3640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f7ecc0ccd20 con 0x7f7ecc1644a0 2024-09-16T15:49:13.753 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:13.751+0000 7f7ed236f640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7eb8003a10 0x7f7ecc163b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:49:13.753 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:13.751+0000 7f7ed236f640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7eb8003a10 0x7f7ecc163b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:52168/0 (socket says 172.21.15.123:52168) 2024-09-16T15:49:13.754 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:13.751+0000 7f7ed236f640 1 -- 172.21.15.123:0/3245549624 learned_addr learned my addr 172.21.15.123:0/3245549624 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:49:13.754 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:13.751+0000 7f7ed236f640 1 -- 172.21.15.123:0/3245549624 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f7ecc1644a0 msgr2=0x7f7ecc156010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:13.754 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:13.751+0000 7f7ed236f640 1 --2- 172.21.15.123:0/3245549624 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f7ecc1644a0 0x7f7ecc156010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:13.754 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:13.751+0000 7f7ed236f640 1 -- 172.21.15.123:0/3245549624 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7ecc15c880 msgr2=0x7f7ecc15cd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:13.754 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:13.751+0000 7f7ed2b70640 1 --2- 172.21.15.123:0/3245549624 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7ecc15c880 0x7f7ecc15cd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:49:13.754 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:13.751+0000 7f7ed236f640 1 --2- 172.21.15.123:0/3245549624 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7ecc15c880 0x7f7ecc15cd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:13.754 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:13.751+0000 7f7ed236f640 1 -- 172.21.15.123:0/3245549624 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7ecc0ad9f0 con 0x7f7eb8003a10 2024-09-16T15:49:13.754 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:13.751+0000 7f7ed236f640 1 --2- 172.21.15.123:0/3245549624 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7eb8003a10 0x7f7ecc163b10 secure :-1 s=READY pgs=1220 cs=0 l=1 rev1=1 crypto rx=0x7f7ec80b1040 tx=0x7f7ec80056f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:49:13.754 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:13.751+0000 7f7ebeffd640 1 -- 172.21.15.123:0/3245549624 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ec80ab040 con 0x7f7eb8003a10 2024-09-16T15:49:13.754 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:13.751+0000 7f7ebeffd640 1 -- 172.21.15.123:0/3245549624 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f7ec8004020 con 0x7f7eb8003a10 2024-09-16T15:49:13.754 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:13.751+0000 7f7ed45d3640 1 -- 172.21.15.123:0/3245549624 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7ecc0adcd0 con 0x7f7eb8003a10 2024-09-16T15:49:13.754 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:13.751+0000 7f7ebeffd640 1 -- 172.21.15.123:0/3245549624 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ec8004470 con 0x7f7eb8003a10 2024-09-16T15:49:13.754 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:13.751+0000 7f7ed45d3640 1 -- 172.21.15.123:0/3245549624 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7ecc0ae290 con 0x7f7eb8003a10 2024-09-16T15:49:13.754 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:13.755+0000 7f7ed45d3640 1 -- 172.21.15.123:0/3245549624 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7ecc15d240 con 0x7f7eb8003a10 2024-09-16T15:49:13.755 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:13.755+0000 7f7ebeffd640 1 -- 172.21.15.123:0/3245549624 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 19) ==== 97488+0+0 (secure 0 0 0) 0x7f7ec80c0020 con 0x7f7eb8003a10 2024-09-16T15:49:13.755 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:13.755+0000 7f7ebeffd640 1 --2- 172.21.15.123:0/3245549624 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f7eb0073340 0x7f7eb0075800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:49:13.755 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:13.755+0000 7f7ebeffd640 1 -- 172.21.15.123:0/3245549624 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(110..110 src has 1..110) ==== 4971+0+0 (secure 0 0 0) 0x7f7ec812e9b0 con 0x7f7eb8003a10 2024-09-16T15:49:13.755 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:13.755+0000 7f7ed1b6e640 1 --2- 172.21.15.123:0/3245549624 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f7eb0073340 0x7f7eb0075800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:49:13.755 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:13.755+0000 7f7ed1b6e640 1 --2- 172.21.15.123:0/3245549624 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f7eb0073340 0x7f7eb0075800 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f7ecc0aed10 tx=0x7f7ec0009540 comp rx=0 tx=0).ready entity=mgr.4150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:49:13.758 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:13.759+0000 7f7ebeffd640 1 -- 172.21.15.123:0/3245549624 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f7ec80169d0 con 0x7f7eb8003a10 2024-09-16T15:49:13.923 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:13.923+0000 7f7ed45d3640 1 -- 172.21.15.123:0/3245549624 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) -- 0x7f7ecc0cd440 con 0x7f7eb8003a10 2024-09-16T15:49:13.928 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:13.927+0000 7f7ebeffd640 1 -- 172.21.15.123:0/3245549624 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v40)=0 v40) ==== 122+0+0 (secure 0 0 0) 0x7f7ec80f9960 con 0x7f7eb8003a10 2024-09-16T15:49:13.930 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:13.931+0000 7f7ed45d3640 1 -- 172.21.15.123:0/3245549624 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f7eb0073340 msgr2=0x7f7eb0075800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:13.930 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:13.931+0000 7f7ed45d3640 1 --2- 172.21.15.123:0/3245549624 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f7eb0073340 0x7f7eb0075800 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f7ecc0aed10 tx=0x7f7ec0009540 comp rx=0 tx=0).stop 2024-09-16T15:49:13.930 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:13.931+0000 7f7ed45d3640 1 -- 172.21.15.123:0/3245549624 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7eb8003a10 msgr2=0x7f7ecc163b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:13.931 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:13.931+0000 7f7ed45d3640 1 --2- 172.21.15.123:0/3245549624 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7eb8003a10 0x7f7ecc163b10 secure :-1 s=READY pgs=1220 cs=0 l=1 rev1=1 crypto rx=0x7f7ec80b1040 tx=0x7f7ec80056f0 comp rx=0 tx=0).stop 2024-09-16T15:49:13.931 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:13.931+0000 7f7ed45d3640 1 -- 172.21.15.123:0/3245549624 shutdown_connections 2024-09-16T15:49:13.931 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:13.931+0000 7f7ed45d3640 1 --2- 172.21.15.123:0/3245549624 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f7eb0073340 0x7f7eb0075800 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:13.931 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:13.931+0000 7f7ed45d3640 1 --2- 172.21.15.123:0/3245549624 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7ecc15c880 0x7f7ecc15cd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:13.931 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:13.931+0000 7f7ed45d3640 1 --2- 172.21.15.123:0/3245549624 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f7ecc1644a0 0x7f7ecc156010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:13.931 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:13.931+0000 7f7ed45d3640 1 --2- 172.21.15.123:0/3245549624 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7eb8003a10 0x7f7ecc163b10 unknown :-1 s=CLOSED pgs=1220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:13.931 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:13.931+0000 7f7ed45d3640 1 -- 172.21.15.123:0/3245549624 >> 172.21.15.123:0/3245549624 conn(0x7f7ecc0b1710 msgr2=0x7f7ecc156a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:49:13.931 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:13.931+0000 7f7ed45d3640 1 -- 172.21.15.123:0/3245549624 shutdown_connections 2024-09-16T15:49:13.931 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:13.931+0000 7f7ed45d3640 1 -- 172.21.15.123:0/3245549624 wait complete. 2024-09-16T15:49:13.946 INFO:teuthology.nuke:Clearing teuthology firewall rules... 2024-09-16T15:49:13.946 DEBUG:teuthology.orchestra.run.smithi123:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-09-16T15:49:14.026 DEBUG:teuthology.orchestra.run.smithi184:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-09-16T15:49:14.090 INFO:teuthology.nuke:Cleared teuthology firewall rules. 2024-09-16T15:49:14.091 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2024-09-16T15:49:14.091 DEBUG:teuthology.orchestra.run.smithi123:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-16T15:49:14.098 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16336468k,nr_inodes=4084117,mode=755,inode64 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3274684k,mode=755,inode64 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev,inode64 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=17082 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3274680k,nr_inodes=818670,mode=700,uid=1000,gid=1271,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3274684k,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-09-16T15:49:14.098 DEBUG:teuthology.orchestra.run.smithi184:> set -ex 2024-09-16T15:49:14.098 DEBUG:teuthology.orchestra.run.smithi184:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-16T15:49:14.104 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16336488k,nr_inodes=4084122,mode=755,inode64 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3274688k,mode=755,inode64 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev,inode64 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=17834 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3274684k,nr_inodes=818671,mode=700,uid=1000,gid=1271,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3274688k,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-09-16T15:49:14.104 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2024-09-16T15:49:14.104 DEBUG:teuthology.orchestra.run.smithi123:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-16T15:49:14.145 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16336468k,nr_inodes=4084117,mode=755,inode64 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3274684k,mode=755,inode64 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev,inode64 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=17082 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3274680k,nr_inodes=818670,mode=700,uid=1000,gid=1271,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3274684k,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-09-16T15:49:14.146 DEBUG:teuthology.orchestra.run.smithi184:> set -ex 2024-09-16T15:49:14.146 DEBUG:teuthology.orchestra.run.smithi184:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-16T15:49:14.151 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16336488k,nr_inodes=4084122,mode=755,inode64 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3274688k,mode=755,inode64 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev,inode64 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=17834 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3274684k,nr_inodes=818671,mode=700,uid=1000,gid=1271,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3274688k,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-09-16T15:49:14.152 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-09-16T15:49:14.152 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-16T15:49:14.314 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:14.315+0000 7fd89cc3b640 1 -- 172.21.15.123:0/815428144 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd8980cd6e0 msgr2=0x7fd8980cdb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:14.314 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:14.315+0000 7fd89cc3b640 1 --2- 172.21.15.123:0/815428144 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd8980cd6e0 0x7fd8980cdb50 secure :-1 s=READY pgs=1222 cs=0 l=1 rev1=1 crypto rx=0x7fd898073c20 tx=0x7fd88001c790 comp rx=0 tx=0).stop 2024-09-16T15:49:14.314 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:14.315+0000 7fd89cc3b640 1 -- 172.21.15.123:0/815428144 shutdown_connections 2024-09-16T15:49:14.314 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:14.315+0000 7fd89cc3b640 1 --2- 172.21.15.123:0/815428144 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd8980cd6e0 0x7fd8980cdb50 unknown :-1 s=CLOSED pgs=1222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:14.314 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:14.315+0000 7fd89cc3b640 1 -- 172.21.15.123:0/815428144 >> 172.21.15.123:0/815428144 conn(0x7fd8980cfb10 msgr2=0x7fd8980cff20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:49:14.315 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:14.315+0000 7fd89cc3b640 1 -- 172.21.15.123:0/815428144 shutdown_connections 2024-09-16T15:49:14.315 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:14.315+0000 7fd89cc3b640 1 -- 172.21.15.123:0/815428144 wait complete. 2024-09-16T15:49:14.320 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:14.315+0000 7fd89cc3b640 1 Processor -- start 2024-09-16T15:49:14.320 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:14.315+0000 7fd89cc3b640 1 -- start start 2024-09-16T15:49:14.320 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:14.315+0000 7fd89cc3b640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd8980cd6e0 0x7fd8981f8670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:49:14.320 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:14.315+0000 7fd89cc3b640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd89815e160 0x7fd898144ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:49:14.321 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:14.315+0000 7fd89cc3b640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fd89815e820 0x7fd8980b4290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:49:14.321 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:14.315+0000 7fd89cc3b640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fd8981697f0 con 0x7fd89815e160 2024-09-16T15:49:14.321 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:14.315+0000 7fd89cc3b640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fd8980cf640 con 0x7fd8980cd6e0 2024-09-16T15:49:14.321 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:14.315+0000 7fd89cc3b640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fd898169970 con 0x7fd89815e820 2024-09-16T15:49:14.321 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:14.315+0000 7fd895d9b640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd89815e160 0x7fd898144ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:49:14.321 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:14.315+0000 7fd895d9b640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd89815e160 0x7fd898144ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:52190/0 (socket says 172.21.15.123:52190) 2024-09-16T15:49:14.321 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:14.315+0000 7fd895d9b640 1 -- 172.21.15.123:0/4118797906 learned_addr learned my addr 172.21.15.123:0/4118797906 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:49:14.322 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:14.315+0000 7fd895d9b640 1 -- 172.21.15.123:0/4118797906 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fd89815e820 msgr2=0x7fd8980b4290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:14.322 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:14.315+0000 7fd896d9d640 1 --2- 172.21.15.123:0/4118797906 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fd89815e820 0x7fd8980b4290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:49:14.322 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:14.315+0000 7fd89659c640 1 --2- 172.21.15.123:0/4118797906 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd8980cd6e0 0x7fd8981f8670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:49:14.322 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:14.315+0000 7fd895d9b640 1 --2- 172.21.15.123:0/4118797906 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fd89815e820 0x7fd8980b4290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:14.322 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:14.315+0000 7fd895d9b640 1 -- 172.21.15.123:0/4118797906 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd8980cd6e0 msgr2=0x7fd8981f8670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:14.322 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:14.315+0000 7fd895d9b640 1 --2- 172.21.15.123:0/4118797906 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd8980cd6e0 0x7fd8981f8670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:14.322 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:14.315+0000 7fd895d9b640 1 -- 172.21.15.123:0/4118797906 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd8980b47d0 con 0x7fd89815e160 2024-09-16T15:49:14.322 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:14.315+0000 7fd89659c640 1 --2- 172.21.15.123:0/4118797906 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd8980cd6e0 0x7fd8981f8670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T15:49:14.322 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:14.315+0000 7fd895d9b640 1 --2- 172.21.15.123:0/4118797906 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd89815e160 0x7fd898144ff0 secure :-1 s=READY pgs=1223 cs=0 l=1 rev1=1 crypto rx=0x7fd880098240 tx=0x7fd880005d20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:49:14.322 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:14.315+0000 7fd88effd640 1 -- 172.21.15.123:0/4118797906 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8800bb070 con 0x7fd89815e160 2024-09-16T15:49:14.322 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:14.315+0000 7fd88effd640 1 -- 172.21.15.123:0/4118797906 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fd880004330 con 0x7fd89815e160 2024-09-16T15:49:14.322 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:14.315+0000 7fd89cc3b640 1 -- 172.21.15.123:0/4118797906 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd8981fb1c0 con 0x7fd89815e160 2024-09-16T15:49:14.322 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:14.315+0000 7fd88effd640 1 -- 172.21.15.123:0/4118797906 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8800b2040 con 0x7fd89815e160 2024-09-16T15:49:14.322 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:14.315+0000 7fd89cc3b640 1 -- 172.21.15.123:0/4118797906 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd8981fb780 con 0x7fd89815e160 2024-09-16T15:49:14.323 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:14.315+0000 7fd89cc3b640 1 -- 172.21.15.123:0/4118797906 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd8980cdb50 con 0x7fd89815e160 2024-09-16T15:49:14.323 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:14.319+0000 7fd88effd640 1 -- 172.21.15.123:0/4118797906 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 19) ==== 97488+0+0 (secure 0 0 0) 0x7fd8800a5620 con 0x7fd89815e160 2024-09-16T15:49:14.323 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:14.319+0000 7fd88effd640 1 --2- 172.21.15.123:0/4118797906 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fd878073340 0x7fd878075800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:49:14.323 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:14.319+0000 7fd88effd640 1 -- 172.21.15.123:0/4118797906 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(110..110 src has 1..110) ==== 4971+0+0 (secure 0 0 0) 0x7fd88012dcb0 con 0x7fd89815e160 2024-09-16T15:49:14.323 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:14.319+0000 7fd89659c640 1 --2- 172.21.15.123:0/4118797906 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fd878073340 0x7fd878075800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:49:14.323 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:14.319+0000 7fd89659c640 1 --2- 172.21.15.123:0/4118797906 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fd878073340 0x7fd878075800 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fd898157410 tx=0x7fd888009040 comp rx=0 tx=0).ready entity=mgr.4150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:49:14.323 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:14.323+0000 7fd88effd640 1 -- 172.21.15.123:0/4118797906 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7fd8980b47d0 con 0x7fd89815e160 2024-09-16T15:49:14.519 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:14.519+0000 7fd89cc3b640 1 -- 172.21.15.123:0/4118797906 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd8980d06c0 con 0x7fd89815e160 2024-09-16T15:49:14.520 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:14.519+0000 7fd88effd640 1 -- 172.21.15.123:0/4118797906 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 71 v71) ==== 94+0+3923 (secure 0 0 0) 0x7fd880016410 con 0x7fd89815e160 2024-09-16T15:49:14.520 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-09-16T15:49:14.520 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":71,"btime":"2024-09-16T15:49:07:873364+0000","default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5417,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6835/3411324594","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":3411324594},{"type":"v1","addr":"172.21.15.184:6835","nonce":3411324594}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":63},{"gid":5848,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6833/1488440148","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":1488440148},{"type":"v1","addr":"172.21.15.184:6833","nonce":1488440148}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":63},{"gid":7032,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6833/1841102587","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6832","nonce":1841102587},{"type":"v1","addr":"172.21.15.123:6833","nonce":1841102587}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":63},{"gid":7464,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6835/3696113722","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6834","nonce":3696113722},{"type":"v1","addr":"172.21.15.123:6835","nonce":3696113722}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":71}],"filesystems":[]} 2024-09-16T15:49:14.520 INFO:teuthology.orchestra.run.smithi123.stderr:dumped fsmap epoch 71 2024-09-16T15:49:14.522 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:14.523+0000 7fd89cc3b640 1 -- 172.21.15.123:0/4118797906 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fd878073340 msgr2=0x7fd878075800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:14.523 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:14.523+0000 7fd89cc3b640 1 --2- 172.21.15.123:0/4118797906 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fd878073340 0x7fd878075800 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fd898157410 tx=0x7fd888009040 comp rx=0 tx=0).stop 2024-09-16T15:49:14.523 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:14.523+0000 7fd89cc3b640 1 -- 172.21.15.123:0/4118797906 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd89815e160 msgr2=0x7fd898144ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:14.523 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:14.523+0000 7fd89cc3b640 1 --2- 172.21.15.123:0/4118797906 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd89815e160 0x7fd898144ff0 secure :-1 s=READY pgs=1223 cs=0 l=1 rev1=1 crypto rx=0x7fd880098240 tx=0x7fd880005d20 comp rx=0 tx=0).stop 2024-09-16T15:49:14.523 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:14.523+0000 7fd89cc3b640 1 -- 172.21.15.123:0/4118797906 shutdown_connections 2024-09-16T15:49:14.523 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:14.523+0000 7fd89cc3b640 1 --2- 172.21.15.123:0/4118797906 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fd878073340 0x7fd878075800 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:14.523 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:14.523+0000 7fd89cc3b640 1 --2- 172.21.15.123:0/4118797906 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fd89815e820 0x7fd8980b4290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:14.523 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:14.523+0000 7fd89cc3b640 1 --2- 172.21.15.123:0/4118797906 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd89815e160 0x7fd898144ff0 unknown :-1 s=CLOSED pgs=1223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:14.523 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:14.523+0000 7fd89cc3b640 1 --2- 172.21.15.123:0/4118797906 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd8980cd6e0 0x7fd8981f8670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:14.523 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:14.523+0000 7fd89cc3b640 1 -- 172.21.15.123:0/4118797906 >> 172.21.15.123:0/4118797906 conn(0x7fd8980cfb10 msgr2=0x7fd898159b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:49:14.523 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:14.523+0000 7fd89cc3b640 1 -- 172.21.15.123:0/4118797906 shutdown_connections 2024-09-16T15:49:14.523 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:14.523+0000 7fd89cc3b640 1 -- 172.21.15.123:0/4118797906 wait complete. 2024-09-16T15:49:14.538 INFO:tasks.ceph.mds.a:Restarting daemon 2024-09-16T15:49:14.538 INFO:tasks.ceph.mds.a:Stopping old one... 2024-09-16T15:49:14.538 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-09-16T15:49:14.539 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T15:49:14.539 INFO:tasks.ceph.mds.c:Restarting daemon 2024-09-16T15:49:14.539 INFO:tasks.ceph.mds.c:Stopping old one... 2024-09-16T15:49:14.539 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-09-16T15:49:14.539 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T15:49:14.539 INFO:tasks.ceph.mds.b:Restarting daemon 2024-09-16T15:49:14.539 INFO:tasks.ceph.mds.b:Stopping old one... 2024-09-16T15:49:14.540 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-09-16T15:49:14.540 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T15:49:14.540 INFO:tasks.ceph.mds.d:Restarting daemon 2024-09-16T15:49:14.540 INFO:tasks.ceph.mds.d:Stopping old one... 2024-09-16T15:49:14.540 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-09-16T15:49:14.540 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T15:49:14.548 INFO:tasks.ceph.mds.c:Stopped 2024-09-16T15:49:14.548 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-09-16T15:49:14.549 INFO:tasks.ceph.mds.a:Stopped 2024-09-16T15:49:14.549 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-09-16T15:49:14.558 INFO:tasks.ceph.mds.b:Stopped 2024-09-16T15:49:14.559 DEBUG:teuthology.orchestra.run.smithi184:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-09-16T15:49:14.559 INFO:tasks.ceph.mds.d:Stopped 2024-09-16T15:49:14.559 DEBUG:teuthology.orchestra.run.smithi184:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-09-16T15:49:14.593 INFO:tasks.ceph.mds.c:Started 2024-09-16T15:49:14.594 DEBUG:teuthology.parallel:result is None 2024-09-16T15:49:14.595 INFO:tasks.ceph.mds.a:Started 2024-09-16T15:49:14.595 DEBUG:teuthology.parallel:result is None 2024-09-16T15:49:14.607 INFO:tasks.ceph.mds.b:Started 2024-09-16T15:49:14.607 DEBUG:teuthology.parallel:result is None 2024-09-16T15:49:14.607 INFO:tasks.ceph.mds.d:Started 2024-09-16T15:49:14.608 DEBUG:teuthology.parallel:result is None 2024-09-16T15:49:14.608 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-09-16T15:49:14.652 INFO:tasks.ceph.mds.c.smithi123.stdout:starting mds.c at 2024-09-16T15:49:14.652 INFO:tasks.ceph.mds.a.smithi123.stdout:starting mds.a at 2024-09-16T15:49:14.676 INFO:tasks.ceph.mds.b.smithi184.stdout:starting mds.b at 2024-09-16T15:49:14.679 INFO:tasks.ceph.mds.d.smithi184.stdout:starting mds.d at 2024-09-16T15:49:14.727 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:14.727+0000 7f7c17a0c640 1 -- 172.21.15.123:0/2092592830 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7c100b2b00 msgr2=0x7f7c100d24b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:14.727 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:14.727+0000 7f7c17a0c640 1 --2- 172.21.15.123:0/2092592830 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7c100b2b00 0x7f7c100d24b0 secure :-1 s=READY pgs=1232 cs=0 l=1 rev1=1 crypto rx=0x7f7c00009900 tx=0x7f7c0001c790 comp rx=0 tx=0).stop 2024-09-16T15:49:14.728 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:14.727+0000 7f7c17a0c640 1 -- 172.21.15.123:0/2092592830 shutdown_connections 2024-09-16T15:49:14.728 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:14.727+0000 7f7c17a0c640 1 --2- 172.21.15.123:0/2092592830 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f7c100b0a50 0x7f7c100b0ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:14.728 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:14.727+0000 7f7c17a0c640 1 --2- 172.21.15.123:0/2092592830 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7c100b2b00 0x7f7c100d24b0 unknown :-1 s=CLOSED pgs=1232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:14.728 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:14.727+0000 7f7c17a0c640 1 -- 172.21.15.123:0/2092592830 >> 172.21.15.123:0/2092592830 conn(0x7f7c100d3ea0 msgr2=0x7f7c100d42b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:49:14.728 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:14.727+0000 7f7c17a0c640 1 -- 172.21.15.123:0/2092592830 shutdown_connections 2024-09-16T15:49:14.728 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:14.727+0000 7f7c17a0c640 1 -- 172.21.15.123:0/2092592830 wait complete. 2024-09-16T15:49:14.728 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:14.727+0000 7f7c17a0c640 1 Processor -- start 2024-09-16T15:49:14.729 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:14.727+0000 7f7c17a0c640 1 -- start start 2024-09-16T15:49:14.731 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:14.731+0000 7f7c17a0c640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7c100b0a50 0x7f7c100c79d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:49:14.731 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:14.731+0000 7f7c17a0c640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f7c100b2b00 0x7f7c100b4280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:49:14.731 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:14.731+0000 7f7c17a0c640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7c10161620 0x7f7c100c8800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:49:14.731 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:14.731+0000 7f7c17a0c640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f7c1016b0f0 con 0x7f7c100b0a50 2024-09-16T15:49:14.731 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:14.731+0000 7f7c17a0c640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f7c1016af70 con 0x7f7c10161620 2024-09-16T15:49:14.731 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:14.731+0000 7f7c17a0c640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f7c1016b270 con 0x7f7c100b2b00 2024-09-16T15:49:14.731 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:14.731+0000 7f7c157a8640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7c100b0a50 0x7f7c100c79d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:49:14.731 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:14.731+0000 7f7c157a8640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7c100b0a50 0x7f7c100c79d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:52234/0 (socket says 172.21.15.123:52234) 2024-09-16T15:49:14.731 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:14.731+0000 7f7c157a8640 1 -- 172.21.15.123:0/4250113173 learned_addr learned my addr 172.21.15.123:0/4250113173 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:49:14.731 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:14.731+0000 7f7c15fa9640 1 --2- 172.21.15.123:0/4250113173 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7c10161620 0x7f7c100c8800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:49:14.731 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:14.731+0000 7f7c14fa7640 1 --2- 172.21.15.123:0/4250113173 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f7c100b2b00 0x7f7c100b4280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:49:14.731 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:14.731+0000 7f7c157a8640 1 -- 172.21.15.123:0/4250113173 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f7c100b2b00 msgr2=0x7f7c100b4280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:14.732 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:14.731+0000 7f7c157a8640 1 --2- 172.21.15.123:0/4250113173 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f7c100b2b00 0x7f7c100b4280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:14.732 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:14.731+0000 7f7c157a8640 1 -- 172.21.15.123:0/4250113173 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7c10161620 msgr2=0x7f7c100c8800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:14.732 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:14.731+0000 7f7c157a8640 1 --2- 172.21.15.123:0/4250113173 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7c10161620 0x7f7c100c8800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:14.732 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:14.731+0000 7f7c157a8640 1 -- 172.21.15.123:0/4250113173 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7c100b4a00 con 0x7f7c100b0a50 2024-09-16T15:49:14.732 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:14.731+0000 7f7c15fa9640 1 --2- 172.21.15.123:0/4250113173 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7c10161620 0x7f7c100c8800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T15:49:14.732 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:14.731+0000 7f7c14fa7640 1 --2- 172.21.15.123:0/4250113173 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f7c100b2b00 0x7f7c100b4280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T15:49:14.732 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:14.731+0000 7f7c157a8640 1 --2- 172.21.15.123:0/4250113173 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7c100b0a50 0x7f7c100c79d0 secure :-1 s=READY pgs=1233 cs=0 l=1 rev1=1 crypto rx=0x7f7c0c00b410 tx=0x7f7c0c00b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:49:14.732 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:14.731+0000 7f7c05ffb640 1 -- 172.21.15.123:0/4250113173 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c0c015030 con 0x7f7c100b0a50 2024-09-16T15:49:14.732 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:14.731+0000 7f7c05ffb640 1 -- 172.21.15.123:0/4250113173 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f7c0c005930 con 0x7f7c100b0a50 2024-09-16T15:49:14.732 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:14.731+0000 7f7c17a0c640 1 -- 172.21.15.123:0/4250113173 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7c100b4ce0 con 0x7f7c100b0a50 2024-09-16T15:49:14.732 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:14.731+0000 7f7c05ffb640 1 -- 172.21.15.123:0/4250113173 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c0c005d80 con 0x7f7c100b0a50 2024-09-16T15:49:14.733 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:14.731+0000 7f7c17a0c640 1 -- 172.21.15.123:0/4250113173 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7c1006d6b0 con 0x7f7c100b0a50 2024-09-16T15:49:14.733 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:14.731+0000 7f7c17a0c640 1 -- 172.21.15.123:0/4250113173 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7c100b0ee0 con 0x7f7c100b0a50 2024-09-16T15:49:14.733 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:14.731+0000 7f7c05ffb640 1 -- 172.21.15.123:0/4250113173 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 19) ==== 97488+0+0 (secure 0 0 0) 0x7f7c0c018070 con 0x7f7c100b0a50 2024-09-16T15:49:14.733 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:14.731+0000 7f7c05ffb640 1 --2- 172.21.15.123:0/4250113173 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f7bf4073340 0x7f7bf4075800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:49:14.734 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:14.731+0000 7f7c05ffb640 1 -- 172.21.15.123:0/4250113173 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(110..110 src has 1..110) ==== 4971+0+0 (secure 0 0 0) 0x7f7c0c094410 con 0x7f7c100b0a50 2024-09-16T15:49:14.734 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:14.731+0000 7f7c14fa7640 1 --2- 172.21.15.123:0/4250113173 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f7bf4073340 0x7f7bf4075800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:49:14.734 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:14.731+0000 7f7c14fa7640 1 --2- 172.21.15.123:0/4250113173 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f7bf4073340 0x7f7bf4075800 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f7c000af000 tx=0x7f7c0001afd0 comp rx=0 tx=0).ready entity=mgr.4150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:49:14.735 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:14.735+0000 7f7c05ffb640 1 -- 172.21.15.123:0/4250113173 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f7c0c05f3a0 con 0x7f7c100b0a50 2024-09-16T15:49:14.913 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:14.915+0000 7f7c17a0c640 1 -- 172.21.15.123:0/4250113173 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) -- 0x7f7c100d4400 con 0x7f7c100b0a50 2024-09-16T15:49:15.961 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:15.959+0000 7f7c05ffb640 1 -- 172.21.15.123:0/4250113173 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v115) ==== 97+0+0 (secure 0 0 0) 0x7f7c0c05eca0 con 0x7f7c100b0a50 2024-09-16T15:49:15.961 INFO:teuthology.orchestra.run.smithi123.stderr: removed all blocklist entries 2024-09-16T15:49:15.964 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:15.963+0000 7f7c17a0c640 1 -- 172.21.15.123:0/4250113173 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f7bf4073340 msgr2=0x7f7bf4075800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:15.964 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:15.963+0000 7f7c17a0c640 1 --2- 172.21.15.123:0/4250113173 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f7bf4073340 0x7f7bf4075800 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f7c000af000 tx=0x7f7c0001afd0 comp rx=0 tx=0).stop 2024-09-16T15:49:15.964 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:15.963+0000 7f7c17a0c640 1 -- 172.21.15.123:0/4250113173 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7c100b0a50 msgr2=0x7f7c100c79d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:15.964 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:15.963+0000 7f7c17a0c640 1 --2- 172.21.15.123:0/4250113173 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7c100b0a50 0x7f7c100c79d0 secure :-1 s=READY pgs=1233 cs=0 l=1 rev1=1 crypto rx=0x7f7c0c00b410 tx=0x7f7c0c00b8d0 comp rx=0 tx=0).stop 2024-09-16T15:49:15.964 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:15.963+0000 7f7c17a0c640 1 -- 172.21.15.123:0/4250113173 shutdown_connections 2024-09-16T15:49:15.964 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:15.963+0000 7f7c17a0c640 1 --2- 172.21.15.123:0/4250113173 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f7bf4073340 0x7f7bf4075800 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:15.964 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:15.963+0000 7f7c17a0c640 1 --2- 172.21.15.123:0/4250113173 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7c10161620 0x7f7c100c8800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:15.964 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:15.963+0000 7f7c17a0c640 1 --2- 172.21.15.123:0/4250113173 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f7c100b2b00 0x7f7c100b4280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:15.964 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:15.963+0000 7f7c17a0c640 1 --2- 172.21.15.123:0/4250113173 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7c100b0a50 0x7f7c100c79d0 unknown :-1 s=CLOSED pgs=1233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:15.964 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:15.963+0000 7f7c17a0c640 1 -- 172.21.15.123:0/4250113173 >> 172.21.15.123:0/4250113173 conn(0x7f7c100d3ea0 msgr2=0x7f7c100b62c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:49:15.964 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:15.963+0000 7f7c17a0c640 1 -- 172.21.15.123:0/4250113173 shutdown_connections 2024-09-16T15:49:15.965 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:15.963+0000 7f7c17a0c640 1 -- 172.21.15.123:0/4250113173 wait complete. 2024-09-16T15:49:15.985 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-09-16T15:49:16.160 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:16.159+0000 7fe2b7774640 1 -- 172.21.15.123:0/1456843587 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe29c003a60 msgr2=0x7fe29c003e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:16.160 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:16.159+0000 7fe2b7774640 1 --2- 172.21.15.123:0/1456843587 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe29c003a60 0x7fe29c003e80 secure :-1 s=READY pgs=1235 cs=0 l=1 rev1=1 crypto rx=0x7fe2ac00bb10 tx=0x7fe2ac01cda0 comp rx=0 tx=0).stop 2024-09-16T15:49:16.160 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:16.159+0000 7fe2b7774640 1 -- 172.21.15.123:0/1456843587 shutdown_connections 2024-09-16T15:49:16.160 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:16.159+0000 7fe2b7774640 1 --2- 172.21.15.123:0/1456843587 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe29c003a60 0x7fe29c003e80 unknown :-1 s=CLOSED pgs=1235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:16.160 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:16.159+0000 7fe2b7774640 1 -- 172.21.15.123:0/1456843587 >> 172.21.15.123:0/1456843587 conn(0x7fe2b0069650 msgr2=0x7fe2b009d170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:49:16.160 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:16.159+0000 7fe2b7774640 1 -- 172.21.15.123:0/1456843587 shutdown_connections 2024-09-16T15:49:16.160 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:16.159+0000 7fe2b7774640 1 -- 172.21.15.123:0/1456843587 wait complete. 2024-09-16T15:49:16.161 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:16.163+0000 7fe2b7774640 1 Processor -- start 2024-09-16T15:49:16.161 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:16.163+0000 7fe2b7774640 1 -- start start 2024-09-16T15:49:16.162 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:16.163+0000 7fe2b7774640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe29c003a60 0x7fe2b009ed60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:49:16.163 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:16.163+0000 7fe2b7774640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fe2b00cbde0 0x7fe2b0160590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:49:16.163 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:16.163+0000 7fe2b7774640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe2b0160ad0 0x7fe2b01f83a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:49:16.163 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:16.163+0000 7fe2b7774640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fe2b00d1490 con 0x7fe29c003a60 2024-09-16T15:49:16.163 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:16.163+0000 7fe2b7774640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fe2b00d1310 con 0x7fe2b0160ad0 2024-09-16T15:49:16.163 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:16.163+0000 7fe2b7774640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fe2b00d1610 con 0x7fe2b00cbde0 2024-09-16T15:49:16.163 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:16.163+0000 7fe2b5510640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe29c003a60 0x7fe2b009ed60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:49:16.163 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:16.163+0000 7fe2b5510640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe29c003a60 0x7fe2b009ed60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:52260/0 (socket says 172.21.15.123:52260) 2024-09-16T15:49:16.163 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:16.163+0000 7fe2b5510640 1 -- 172.21.15.123:0/2930087708 learned_addr learned my addr 172.21.15.123:0/2930087708 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:49:16.163 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:16.163+0000 7fe2b5d11640 1 --2- 172.21.15.123:0/2930087708 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe2b0160ad0 0x7fe2b01f83a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:49:16.163 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:16.163+0000 7fe2b5510640 1 -- 172.21.15.123:0/2930087708 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fe2b00cbde0 msgr2=0x7fe2b0160590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:16.163 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:16.163+0000 7fe2b5510640 1 --2- 172.21.15.123:0/2930087708 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fe2b00cbde0 0x7fe2b0160590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:16.163 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:16.163+0000 7fe2b5510640 1 -- 172.21.15.123:0/2930087708 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe2b0160ad0 msgr2=0x7fe2b01f83a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:16.164 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:16.163+0000 7fe2b5510640 1 --2- 172.21.15.123:0/2930087708 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe2b0160ad0 0x7fe2b01f83a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:16.164 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:16.163+0000 7fe2b5510640 1 -- 172.21.15.123:0/2930087708 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe2b01f9110 con 0x7fe29c003a60 2024-09-16T15:49:16.164 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:16.163+0000 7fe2b5510640 1 --2- 172.21.15.123:0/2930087708 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe29c003a60 0x7fe2b009ed60 secure :-1 s=READY pgs=1236 cs=0 l=1 rev1=1 crypto rx=0x7fe2ac0b1040 tx=0x7fe2ac004620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:49:16.164 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:16.163+0000 7fe2a1ffb640 1 -- 172.21.15.123:0/2930087708 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2ac0be070 con 0x7fe29c003a60 2024-09-16T15:49:16.164 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:16.163+0000 7fe2a1ffb640 1 -- 172.21.15.123:0/2930087708 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fe2ac005330 con 0x7fe29c003a60 2024-09-16T15:49:16.164 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:16.163+0000 7fe2b7774640 1 -- 172.21.15.123:0/2930087708 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe2b00cb020 con 0x7fe29c003a60 2024-09-16T15:49:16.164 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:16.163+0000 7fe2a1ffb640 1 -- 172.21.15.123:0/2930087708 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2ac0ab050 con 0x7fe29c003a60 2024-09-16T15:49:16.164 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:16.163+0000 7fe2b7774640 1 -- 172.21.15.123:0/2930087708 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe2b00cb5e0 con 0x7fe29c003a60 2024-09-16T15:49:16.165 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:16.163+0000 7fe2b7774640 1 -- 172.21.15.123:0/2930087708 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe2b00d1e50 con 0x7fe29c003a60 2024-09-16T15:49:16.165 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:16.163+0000 7fe2a1ffb640 1 -- 172.21.15.123:0/2930087708 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 19) ==== 97488+0+0 (secure 0 0 0) 0x7fe2ac0b6070 con 0x7fe29c003a60 2024-09-16T15:49:16.165 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:16.163+0000 7fe2a1ffb640 1 --2- 172.21.15.123:0/2930087708 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fe294073340 0x7fe294075800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:49:16.165 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:16.163+0000 7fe2a1ffb640 1 -- 172.21.15.123:0/2930087708 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(115..115 src has 1..115) ==== 4885+0+0 (secure 0 0 0) 0x7fe2ac12e2c0 con 0x7fe29c003a60 2024-09-16T15:49:16.165 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:16.163+0000 7fe2b4d0f640 1 --2- 172.21.15.123:0/2930087708 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fe294073340 0x7fe294075800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:49:16.165 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:16.167+0000 7fe2b4d0f640 1 --2- 172.21.15.123:0/2930087708 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fe294073340 0x7fe294075800 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fe2b00ca8f0 tx=0x7fe2a4006a70 comp rx=0 tx=0).ready entity=mgr.4150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:49:16.169 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:16.171+0000 7fe2a1ffb640 1 -- 172.21.15.123:0/2930087708 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7fe2ac016ab0 con 0x7fe29c003a60 2024-09-16T15:49:16.389 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:16.391+0000 7fe2b7774640 1 -- 172.21.15.123:0/2930087708 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) -- 0x7fe2b00b29b0 con 0x7fe29c003a60 2024-09-16T15:49:16.390 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:16.391+0000 7fe2a1ffb640 1 -- 172.21.15.123:0/2930087708 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v24) ==== 80+0+5304 (secure 0 0 0) 0x7fe2ac0f8ff0 con 0x7fe29c003a60 2024-09-16T15:49:16.390 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-09-16T15:49:16.390 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2024-09-16T15:49:16.390 INFO:teuthology.orchestra.run.smithi123.stdout: "auth_dump": [ 2024-09-16T15:49:16.390 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-09-16T15:49:16.390 INFO:teuthology.orchestra.run.smithi123.stdout: "entity": "mds.a", 2024-09-16T15:49:16.390 INFO:teuthology.orchestra.run.smithi123.stdout: "key": "AQC0UehmbZpOJBAAaYGyDcWrt4k0FS5P/bb+4A==", 2024-09-16T15:49:16.390 INFO:teuthology.orchestra.run.smithi123.stdout: "caps": { 2024-09-16T15:49:16.390 INFO:teuthology.orchestra.run.smithi123.stdout: "mds": "allow", 2024-09-16T15:49:16.391 INFO:teuthology.orchestra.run.smithi123.stdout: "mgr": "allow *", 2024-09-16T15:49:16.391 INFO:teuthology.orchestra.run.smithi123.stdout: "mon": "allow *", 2024-09-16T15:49:16.391 INFO:teuthology.orchestra.run.smithi123.stdout: "osd": "allow *" 2024-09-16T15:49:16.391 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-16T15:49:16.391 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:49:16.391 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-09-16T15:49:16.391 INFO:teuthology.orchestra.run.smithi123.stdout: "entity": "mds.b", 2024-09-16T15:49:16.391 INFO:teuthology.orchestra.run.smithi123.stdout: "key": "AQC0UehmF1HFMhAADcvqXNKdHZMPAbLc+se4Aw==", 2024-09-16T15:49:16.391 INFO:teuthology.orchestra.run.smithi123.stdout: "caps": { 2024-09-16T15:49:16.391 INFO:teuthology.orchestra.run.smithi123.stdout: "mds": "allow", 2024-09-16T15:49:16.391 INFO:teuthology.orchestra.run.smithi123.stdout: "mgr": "allow *", 2024-09-16T15:49:16.391 INFO:teuthology.orchestra.run.smithi123.stdout: "mon": "allow *", 2024-09-16T15:49:16.392 INFO:teuthology.orchestra.run.smithi123.stdout: "osd": "allow *" 2024-09-16T15:49:16.392 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-16T15:49:16.392 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:49:16.392 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-09-16T15:49:16.392 INFO:teuthology.orchestra.run.smithi123.stdout: "entity": "mds.c", 2024-09-16T15:49:16.392 INFO:teuthology.orchestra.run.smithi123.stdout: "key": "AQC0UehmCBjCLBAAXd85J25r7cgrXeDCG7pahg==", 2024-09-16T15:49:16.392 INFO:teuthology.orchestra.run.smithi123.stdout: "caps": { 2024-09-16T15:49:16.392 INFO:teuthology.orchestra.run.smithi123.stdout: "mds": "allow", 2024-09-16T15:49:16.392 INFO:teuthology.orchestra.run.smithi123.stdout: "mgr": "allow *", 2024-09-16T15:49:16.392 INFO:teuthology.orchestra.run.smithi123.stdout: "mon": "allow *", 2024-09-16T15:49:16.392 INFO:teuthology.orchestra.run.smithi123.stdout: "osd": "allow *" 2024-09-16T15:49:16.392 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-16T15:49:16.392 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:49:16.393 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-09-16T15:49:16.393 INFO:teuthology.orchestra.run.smithi123.stdout: "entity": "mds.d", 2024-09-16T15:49:16.393 INFO:teuthology.orchestra.run.smithi123.stdout: "key": "AQC0UehmMh3ZOBAAx0RCa3R9RMT93/icZhPGyg==", 2024-09-16T15:49:16.393 INFO:teuthology.orchestra.run.smithi123.stdout: "caps": { 2024-09-16T15:49:16.393 INFO:teuthology.orchestra.run.smithi123.stdout: "mds": "allow", 2024-09-16T15:49:16.393 INFO:teuthology.orchestra.run.smithi123.stdout: "mgr": "allow *", 2024-09-16T15:49:16.393 INFO:teuthology.orchestra.run.smithi123.stdout: "mon": "allow *", 2024-09-16T15:49:16.393 INFO:teuthology.orchestra.run.smithi123.stdout: "osd": "allow *" 2024-09-16T15:49:16.393 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-16T15:49:16.393 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:49:16.393 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-09-16T15:49:16.393 INFO:teuthology.orchestra.run.smithi123.stdout: "entity": "osd.0", 2024-09-16T15:49:16.394 INFO:teuthology.orchestra.run.smithi123.stdout: "key": "AQC4UehmT633CRAAIhpkeIBOzwJO0kiXLu0x9Q==", 2024-09-16T15:49:16.394 INFO:teuthology.orchestra.run.smithi123.stdout: "caps": { 2024-09-16T15:49:16.394 INFO:teuthology.orchestra.run.smithi123.stdout: "mgr": "allow profile osd", 2024-09-16T15:49:16.394 INFO:teuthology.orchestra.run.smithi123.stdout: "mon": "allow profile osd", 2024-09-16T15:49:16.394 INFO:teuthology.orchestra.run.smithi123.stdout: "osd": "allow *" 2024-09-16T15:49:16.394 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-16T15:49:16.394 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:49:16.394 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-09-16T15:49:16.394 INFO:teuthology.orchestra.run.smithi123.stdout: "entity": "osd.1", 2024-09-16T15:49:16.394 INFO:teuthology.orchestra.run.smithi123.stdout: "key": "AQC7UehmFwF7GBAAvOUG+rWjANKT8wm/ai4xGg==", 2024-09-16T15:49:16.394 INFO:teuthology.orchestra.run.smithi123.stdout: "caps": { 2024-09-16T15:49:16.394 INFO:teuthology.orchestra.run.smithi123.stdout: "mgr": "allow profile osd", 2024-09-16T15:49:16.395 INFO:teuthology.orchestra.run.smithi123.stdout: "mon": "allow profile osd", 2024-09-16T15:49:16.395 INFO:teuthology.orchestra.run.smithi123.stdout: "osd": "allow *" 2024-09-16T15:49:16.395 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-16T15:49:16.395 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:49:16.395 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-09-16T15:49:16.395 INFO:teuthology.orchestra.run.smithi123.stdout: "entity": "osd.2", 2024-09-16T15:49:16.395 INFO:teuthology.orchestra.run.smithi123.stdout: "key": "AQC+UehmlCwWKxAAl3iO7ZALWyaorQH2urHKCw==", 2024-09-16T15:49:16.395 INFO:teuthology.orchestra.run.smithi123.stdout: "caps": { 2024-09-16T15:49:16.395 INFO:teuthology.orchestra.run.smithi123.stdout: "mgr": "allow profile osd", 2024-09-16T15:49:16.395 INFO:teuthology.orchestra.run.smithi123.stdout: "mon": "allow profile osd", 2024-09-16T15:49:16.395 INFO:teuthology.orchestra.run.smithi123.stdout: "osd": "allow *" 2024-09-16T15:49:16.395 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-16T15:49:16.396 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:49:16.396 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-09-16T15:49:16.396 INFO:teuthology.orchestra.run.smithi123.stdout: "entity": "osd.3", 2024-09-16T15:49:16.396 INFO:teuthology.orchestra.run.smithi123.stdout: "key": "AQDCUehmD8VgAhAAneoIeGr3MflSmi+nyavj2A==", 2024-09-16T15:49:16.396 INFO:teuthology.orchestra.run.smithi123.stdout: "caps": { 2024-09-16T15:49:16.396 INFO:teuthology.orchestra.run.smithi123.stdout: "mgr": "allow profile osd", 2024-09-16T15:49:16.396 INFO:teuthology.orchestra.run.smithi123.stdout: "mon": "allow profile osd", 2024-09-16T15:49:16.396 INFO:teuthology.orchestra.run.smithi123.stdout: "osd": "allow *" 2024-09-16T15:49:16.396 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-16T15:49:16.396 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:49:16.396 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-09-16T15:49:16.396 INFO:teuthology.orchestra.run.smithi123.stdout: "entity": "osd.4", 2024-09-16T15:49:16.397 INFO:teuthology.orchestra.run.smithi123.stdout: "key": "AQDIUehmA9B9EBAAsu0X5LPx1TmatZeHzjnVig==", 2024-09-16T15:49:16.397 INFO:teuthology.orchestra.run.smithi123.stdout: "caps": { 2024-09-16T15:49:16.397 INFO:teuthology.orchestra.run.smithi123.stdout: "mgr": "allow profile osd", 2024-09-16T15:49:16.397 INFO:teuthology.orchestra.run.smithi123.stdout: "mon": "allow profile osd", 2024-09-16T15:49:16.397 INFO:teuthology.orchestra.run.smithi123.stdout: "osd": "allow *" 2024-09-16T15:49:16.397 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-16T15:49:16.397 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:49:16.397 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-09-16T15:49:16.397 INFO:teuthology.orchestra.run.smithi123.stdout: "entity": "osd.5", 2024-09-16T15:49:16.397 INFO:teuthology.orchestra.run.smithi123.stdout: "key": "AQDLUehmcugdHxAAQMSS1ZH71FUzCx5A+yNnZg==", 2024-09-16T15:49:16.397 INFO:teuthology.orchestra.run.smithi123.stdout: "caps": { 2024-09-16T15:49:16.397 INFO:teuthology.orchestra.run.smithi123.stdout: "mgr": "allow profile osd", 2024-09-16T15:49:16.398 INFO:teuthology.orchestra.run.smithi123.stdout: "mon": "allow profile osd", 2024-09-16T15:49:16.398 INFO:teuthology.orchestra.run.smithi123.stdout: "osd": "allow *" 2024-09-16T15:49:16.398 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-16T15:49:16.398 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:49:16.398 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-09-16T15:49:16.398 INFO:teuthology.orchestra.run.smithi123.stdout: "entity": "osd.6", 2024-09-16T15:49:16.398 INFO:teuthology.orchestra.run.smithi123.stdout: "key": "AQDOUehmWLwYLhAAhOlQLepcWElsVPMnHlZm4w==", 2024-09-16T15:49:16.398 INFO:teuthology.orchestra.run.smithi123.stdout: "caps": { 2024-09-16T15:49:16.398 INFO:teuthology.orchestra.run.smithi123.stdout: "mgr": "allow profile osd", 2024-09-16T15:49:16.398 INFO:teuthology.orchestra.run.smithi123.stdout: "mon": "allow profile osd", 2024-09-16T15:49:16.398 INFO:teuthology.orchestra.run.smithi123.stdout: "osd": "allow *" 2024-09-16T15:49:16.398 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-16T15:49:16.398 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:49:16.399 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-09-16T15:49:16.399 INFO:teuthology.orchestra.run.smithi123.stdout: "entity": "osd.7", 2024-09-16T15:49:16.399 INFO:teuthology.orchestra.run.smithi123.stdout: "key": "AQDSUehmipkbABAAQuL/2O7w0tsM1OKeOil97w==", 2024-09-16T15:49:16.399 INFO:teuthology.orchestra.run.smithi123.stdout: "caps": { 2024-09-16T15:49:16.399 INFO:teuthology.orchestra.run.smithi123.stdout: "mgr": "allow profile osd", 2024-09-16T15:49:16.399 INFO:teuthology.orchestra.run.smithi123.stdout: "mon": "allow profile osd", 2024-09-16T15:49:16.399 INFO:teuthology.orchestra.run.smithi123.stdout: "osd": "allow *" 2024-09-16T15:49:16.399 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-16T15:49:16.399 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:49:16.399 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-09-16T15:49:16.399 INFO:teuthology.orchestra.run.smithi123.stdout: "entity": "client.0", 2024-09-16T15:49:16.399 INFO:teuthology.orchestra.run.smithi123.stdout: "key": "AQC1Uehm7GDTAhAAHWWVIyjVVnTeeozBR8DfFA==", 2024-09-16T15:49:16.399 INFO:teuthology.orchestra.run.smithi123.stdout: "caps": { 2024-09-16T15:49:16.400 INFO:teuthology.orchestra.run.smithi123.stdout: "mds": "allow", 2024-09-16T15:49:16.400 INFO:teuthology.orchestra.run.smithi123.stdout: "mon": "allow r", 2024-09-16T15:49:16.400 INFO:teuthology.orchestra.run.smithi123.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-09-16T15:49:16.400 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-16T15:49:16.400 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:49:16.400 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-09-16T15:49:16.400 INFO:teuthology.orchestra.run.smithi123.stdout: "entity": "client.1", 2024-09-16T15:49:16.400 INFO:teuthology.orchestra.run.smithi123.stdout: "key": "AQC1UehmFbaWChAAFtRmKucm+pbKtCNvRCTNqA==", 2024-09-16T15:49:16.400 INFO:teuthology.orchestra.run.smithi123.stdout: "caps": { 2024-09-16T15:49:16.400 INFO:teuthology.orchestra.run.smithi123.stdout: "mds": "allow", 2024-09-16T15:49:16.400 INFO:teuthology.orchestra.run.smithi123.stdout: "mgr": "allow r", 2024-09-16T15:49:16.400 INFO:teuthology.orchestra.run.smithi123.stdout: "mon": "allow rw", 2024-09-16T15:49:16.400 INFO:teuthology.orchestra.run.smithi123.stdout: "osd": "allow rwx" 2024-09-16T15:49:16.401 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-16T15:49:16.401 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:49:16.401 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-09-16T15:49:16.401 INFO:teuthology.orchestra.run.smithi123.stdout: "entity": "client.2", 2024-09-16T15:49:16.401 INFO:teuthology.orchestra.run.smithi123.stdout: "key": "AQC1Uehm1ADrBxAAW/8ayNHI6Wq/YlI4ivFZiw==", 2024-09-16T15:49:16.401 INFO:teuthology.orchestra.run.smithi123.stdout: "caps": { 2024-09-16T15:49:16.401 INFO:teuthology.orchestra.run.smithi123.stdout: "mds": "allow", 2024-09-16T15:49:16.401 INFO:teuthology.orchestra.run.smithi123.stdout: "mgr": "allow r", 2024-09-16T15:49:16.401 INFO:teuthology.orchestra.run.smithi123.stdout: "mon": "allow rw", 2024-09-16T15:49:16.401 INFO:teuthology.orchestra.run.smithi123.stdout: "osd": "allow rwx" 2024-09-16T15:49:16.401 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-16T15:49:16.401 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:49:16.401 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-09-16T15:49:16.402 INFO:teuthology.orchestra.run.smithi123.stdout: "entity": "client.3", 2024-09-16T15:49:16.402 INFO:teuthology.orchestra.run.smithi123.stdout: "key": "AQC1UehmLv/3DxAAuGloBxSuza95uajpfiZ9MQ==", 2024-09-16T15:49:16.402 INFO:teuthology.orchestra.run.smithi123.stdout: "caps": { 2024-09-16T15:49:16.402 INFO:teuthology.orchestra.run.smithi123.stdout: "mds": "allow", 2024-09-16T15:49:16.402 INFO:teuthology.orchestra.run.smithi123.stdout: "mgr": "allow r", 2024-09-16T15:49:16.402 INFO:teuthology.orchestra.run.smithi123.stdout: "mon": "allow rw", 2024-09-16T15:49:16.402 INFO:teuthology.orchestra.run.smithi123.stdout: "osd": "allow rwx" 2024-09-16T15:49:16.402 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-16T15:49:16.402 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:49:16.402 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-09-16T15:49:16.402 INFO:teuthology.orchestra.run.smithi123.stdout: "entity": "client.admin", 2024-09-16T15:49:16.402 INFO:teuthology.orchestra.run.smithi123.stdout: "key": "AQC0Uehm1ZDRDBAAhyQ1BmJg1Mw6wQ04I2CbFA==", 2024-09-16T15:49:16.402 INFO:teuthology.orchestra.run.smithi123.stdout: "caps": { 2024-09-16T15:49:16.402 INFO:teuthology.orchestra.run.smithi123.stdout: "mds": "allow *", 2024-09-16T15:49:16.403 INFO:teuthology.orchestra.run.smithi123.stdout: "mgr": "allow *", 2024-09-16T15:49:16.403 INFO:teuthology.orchestra.run.smithi123.stdout: "mon": "allow *", 2024-09-16T15:49:16.403 INFO:teuthology.orchestra.run.smithi123.stdout: "osd": "allow *" 2024-09-16T15:49:16.403 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-16T15:49:16.403 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:49:16.403 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-09-16T15:49:16.403 INFO:teuthology.orchestra.run.smithi123.stdout: "entity": "mgr.x", 2024-09-16T15:49:16.403 INFO:teuthology.orchestra.run.smithi123.stdout: "key": "AQC0UehmfPilIBAAERlBbduOgAU9ba0n/1tl3g==", 2024-09-16T15:49:16.403 INFO:teuthology.orchestra.run.smithi123.stdout: "caps": { 2024-09-16T15:49:16.403 INFO:teuthology.orchestra.run.smithi123.stdout: "mds": "allow *", 2024-09-16T15:49:16.403 INFO:teuthology.orchestra.run.smithi123.stdout: "mon": "allow profile mgr", 2024-09-16T15:49:16.403 INFO:teuthology.orchestra.run.smithi123.stdout: "osd": "allow *" 2024-09-16T15:49:16.403 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-16T15:49:16.403 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:49:16.404 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-09-16T15:49:16.404 INFO:teuthology.orchestra.run.smithi123.stdout: "entity": "mgr.y", 2024-09-16T15:49:16.404 INFO:teuthology.orchestra.run.smithi123.stdout: "key": "AQC0UehmkjrlHRAA+3xbQC+SPWGWW1gND7xcAg==", 2024-09-16T15:49:16.404 INFO:teuthology.orchestra.run.smithi123.stdout: "caps": { 2024-09-16T15:49:16.404 INFO:teuthology.orchestra.run.smithi123.stdout: "mds": "allow *", 2024-09-16T15:49:16.404 INFO:teuthology.orchestra.run.smithi123.stdout: "mon": "allow profile mgr", 2024-09-16T15:49:16.404 INFO:teuthology.orchestra.run.smithi123.stdout: "osd": "allow *" 2024-09-16T15:49:16.404 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-16T15:49:16.404 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-16T15:49:16.404 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-09-16T15:49:16.404 INFO:teuthology.orchestra.run.smithi123.stdout:} 2024-09-16T15:49:16.405 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:16.391+0000 7fe2b7774640 1 -- 172.21.15.123:0/2930087708 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fe294073340 msgr2=0x7fe294075800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:16.405 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:16.391+0000 7fe2b7774640 1 --2- 172.21.15.123:0/2930087708 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fe294073340 0x7fe294075800 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fe2b00ca8f0 tx=0x7fe2a4006a70 comp rx=0 tx=0).stop 2024-09-16T15:49:16.405 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:16.391+0000 7fe2b7774640 1 -- 172.21.15.123:0/2930087708 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe29c003a60 msgr2=0x7fe2b009ed60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:16.405 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:16.391+0000 7fe2b7774640 1 --2- 172.21.15.123:0/2930087708 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe29c003a60 0x7fe2b009ed60 secure :-1 s=READY pgs=1236 cs=0 l=1 rev1=1 crypto rx=0x7fe2ac0b1040 tx=0x7fe2ac004620 comp rx=0 tx=0).stop 2024-09-16T15:49:16.406 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:16.391+0000 7fe2b7774640 1 -- 172.21.15.123:0/2930087708 shutdown_connections 2024-09-16T15:49:16.406 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:16.391+0000 7fe2b7774640 1 --2- 172.21.15.123:0/2930087708 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fe294073340 0x7fe294075800 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:16.406 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:16.391+0000 7fe2b7774640 1 --2- 172.21.15.123:0/2930087708 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe2b0160ad0 0x7fe2b01f83a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:16.406 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:16.391+0000 7fe2b7774640 1 --2- 172.21.15.123:0/2930087708 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fe2b00cbde0 0x7fe2b0160590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:16.406 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:16.391+0000 7fe2b7774640 1 --2- 172.21.15.123:0/2930087708 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe29c003a60 0x7fe2b009ed60 unknown :-1 s=CLOSED pgs=1236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:16.406 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:16.391+0000 7fe2b7774640 1 -- 172.21.15.123:0/2930087708 >> 172.21.15.123:0/2930087708 conn(0x7fe2b0069650 msgr2=0x7fe2b00b8cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:49:16.406 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:16.391+0000 7fe2b7774640 1 -- 172.21.15.123:0/2930087708 shutdown_connections 2024-09-16T15:49:16.406 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:16.391+0000 7fe2b7774640 1 -- 172.21.15.123:0/2930087708 wait complete. 2024-09-16T15:49:16.409 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-09-16T15:49:16.519 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:16.519+0000 7f2db7378640 1 -- 172.21.15.123:0/134590969 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2db00b2910 msgr2=0x7f2db0074700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:16.520 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:16.519+0000 7f2db7378640 1 --2- 172.21.15.123:0/134590969 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2db00b2910 0x7f2db0074700 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f2da8009c80 tx=0x7f2da801cc80 comp rx=0 tx=0).stop 2024-09-16T15:49:16.520 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:16.519+0000 7f2db7378640 1 -- 172.21.15.123:0/134590969 shutdown_connections 2024-09-16T15:49:16.520 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:16.519+0000 7f2db7378640 1 --2- 172.21.15.123:0/134590969 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2db00b2910 0x7f2db0074700 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:16.520 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:16.519+0000 7f2db7378640 1 --2- 172.21.15.123:0/134590969 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f2db00cc830 0x7f2db00741c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:16.520 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:16.519+0000 7f2db7378640 1 -- 172.21.15.123:0/134590969 >> 172.21.15.123:0/134590969 conn(0x7f2db00b1df0 msgr2=0x7f2db00b2200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:49:16.520 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:16.519+0000 7f2db7378640 1 -- 172.21.15.123:0/134590969 shutdown_connections 2024-09-16T15:49:16.520 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:16.519+0000 7f2db7378640 1 -- 172.21.15.123:0/134590969 wait complete. 2024-09-16T15:49:16.520 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:16.519+0000 7f2db7378640 1 Processor -- start 2024-09-16T15:49:16.521 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:16.519+0000 7f2db7378640 1 -- start start 2024-09-16T15:49:16.523 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:16.523+0000 7f2db7378640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2db00b2910 0x7f2db0097060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:49:16.523 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:16.523+0000 7f2db7378640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2db00cc830 0x7f2db00c8050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:49:16.523 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:16.523+0000 7f2db7378640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f2db0161520 0x7f2db0154630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:49:16.524 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:16.523+0000 7f2db7378640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f2db016b190 con 0x7f2db00cc830 2024-09-16T15:49:16.524 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:16.523+0000 7f2db7378640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f2db016b010 con 0x7f2db00b2910 2024-09-16T15:49:16.524 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:16.523+0000 7f2db7378640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f2db016b310 con 0x7f2db0161520 2024-09-16T15:49:16.524 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:16.523+0000 7f2db4913640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2db00cc830 0x7f2db00c8050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:49:16.524 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:16.523+0000 7f2db4913640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2db00cc830 0x7f2db00c8050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:52274/0 (socket says 172.21.15.123:52274) 2024-09-16T15:49:16.524 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:16.523+0000 7f2db4913640 1 -- 172.21.15.123:0/4213695351 learned_addr learned my addr 172.21.15.123:0/4213695351 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:49:16.524 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:16.523+0000 7f2db5915640 1 --2- 172.21.15.123:0/4213695351 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f2db0161520 0x7f2db0154630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:49:16.524 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:16.523+0000 7f2db5114640 1 --2- 172.21.15.123:0/4213695351 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2db00b2910 0x7f2db0097060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:49:16.524 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:16.523+0000 7f2db4913640 1 -- 172.21.15.123:0/4213695351 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f2db0161520 msgr2=0x7f2db0154630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:16.524 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:16.523+0000 7f2db4913640 1 --2- 172.21.15.123:0/4213695351 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f2db0161520 0x7f2db0154630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:16.524 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:16.523+0000 7f2db4913640 1 -- 172.21.15.123:0/4213695351 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2db00b2910 msgr2=0x7f2db0097060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:16.524 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:16.523+0000 7f2db4913640 1 --2- 172.21.15.123:0/4213695351 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2db00b2910 0x7f2db0097060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:16.525 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:16.523+0000 7f2db4913640 1 -- 172.21.15.123:0/4213695351 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2db0154b70 con 0x7f2db00cc830 2024-09-16T15:49:16.525 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:16.523+0000 7f2db4913640 1 --2- 172.21.15.123:0/4213695351 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2db00cc830 0x7f2db00c8050 secure :-1 s=READY pgs=1237 cs=0 l=1 rev1=1 crypto rx=0x7f2da000e9f0 tx=0x7f2da000eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:49:16.525 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:16.523+0000 7f2da5ffb640 1 -- 172.21.15.123:0/4213695351 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2da000cd80 con 0x7f2db00cc830 2024-09-16T15:49:16.525 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:16.523+0000 7f2da5ffb640 1 -- 172.21.15.123:0/4213695351 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f2da0005020 con 0x7f2db00cc830 2024-09-16T15:49:16.525 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:16.523+0000 7f2db7378640 1 -- 172.21.15.123:0/4213695351 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2db00b3720 con 0x7f2db00cc830 2024-09-16T15:49:16.525 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:16.523+0000 7f2da5ffb640 1 -- 172.21.15.123:0/4213695351 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2da0005470 con 0x7f2db00cc830 2024-09-16T15:49:16.525 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:16.523+0000 7f2db7378640 1 -- 172.21.15.123:0/4213695351 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2db0154e30 con 0x7f2db00cc830 2024-09-16T15:49:16.526 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:16.523+0000 7f2db7378640 1 -- 172.21.15.123:0/4213695351 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2db00cba60 con 0x7f2db00cc830 2024-09-16T15:49:16.526 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:16.523+0000 7f2da5ffb640 1 -- 172.21.15.123:0/4213695351 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 19) ==== 97488+0+0 (secure 0 0 0) 0x7f2da00047b0 con 0x7f2db00cc830 2024-09-16T15:49:16.526 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:16.523+0000 7f2da5ffb640 1 --2- 172.21.15.123:0/4213695351 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f2d94073340 0x7f2d94075800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:49:16.526 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:16.527+0000 7f2db5114640 1 --2- 172.21.15.123:0/4213695351 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f2d94073340 0x7f2d94075800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:49:16.526 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:16.527+0000 7f2da5ffb640 1 -- 172.21.15.123:0/4213695351 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(115..115 src has 1..115) ==== 4885+0+0 (secure 0 0 0) 0x7f2da0094ed0 con 0x7f2db00cc830 2024-09-16T15:49:16.526 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:16.527+0000 7f2db5114640 1 --2- 172.21.15.123:0/4213695351 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f2d94073340 0x7f2d94075800 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f2dac001650 tx=0x7f2dac009040 comp rx=0 tx=0).ready entity=mgr.4150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:49:16.529 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:16.531+0000 7f2da5ffb640 1 -- 172.21.15.123:0/4213695351 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f2da005fd50 con 0x7f2db00cc830 2024-09-16T15:49:16.701 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:16.703+0000 7f2db7378640 1 -- 172.21.15.123:0/4213695351 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) -- 0x7f2db006dd90 con 0x7f2db00cc830 2024-09-16T15:49:16.702 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:16.703+0000 7f2da5ffb640 1 -- 172.21.15.123:0/4213695351 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v115) ==== 84+0+65 (secure 0 0 0) 0x7f2da005f650 con 0x7f2db00cc830 2024-09-16T15:49:16.702 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-09-16T15:49:16.702 INFO:teuthology.orchestra.run.smithi123.stdout:[ 2024-09-16T15:49:16.702 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-09-16T15:49:16.702 INFO:teuthology.orchestra.run.smithi123.stdout: "poolnum": 1, 2024-09-16T15:49:16.702 INFO:teuthology.orchestra.run.smithi123.stdout: "poolname": ".mgr" 2024-09-16T15:49:16.702 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-16T15:49:16.702 INFO:teuthology.orchestra.run.smithi123.stdout:] 2024-09-16T15:49:16.704 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:16.703+0000 7f2db7378640 1 -- 172.21.15.123:0/4213695351 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f2d94073340 msgr2=0x7f2d94075800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:16.704 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:16.703+0000 7f2db7378640 1 --2- 172.21.15.123:0/4213695351 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f2d94073340 0x7f2d94075800 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f2dac001650 tx=0x7f2dac009040 comp rx=0 tx=0).stop 2024-09-16T15:49:16.704 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:16.703+0000 7f2db7378640 1 -- 172.21.15.123:0/4213695351 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2db00cc830 msgr2=0x7f2db00c8050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:16.704 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:16.703+0000 7f2db7378640 1 --2- 172.21.15.123:0/4213695351 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2db00cc830 0x7f2db00c8050 secure :-1 s=READY pgs=1237 cs=0 l=1 rev1=1 crypto rx=0x7f2da000e9f0 tx=0x7f2da000eeb0 comp rx=0 tx=0).stop 2024-09-16T15:49:16.705 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:16.703+0000 7f2db7378640 1 -- 172.21.15.123:0/4213695351 shutdown_connections 2024-09-16T15:49:16.705 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:16.703+0000 7f2db7378640 1 --2- 172.21.15.123:0/4213695351 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f2d94073340 0x7f2d94075800 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:16.705 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:16.703+0000 7f2db7378640 1 --2- 172.21.15.123:0/4213695351 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f2db0161520 0x7f2db0154630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:16.705 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:16.703+0000 7f2db7378640 1 --2- 172.21.15.123:0/4213695351 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2db00cc830 0x7f2db00c8050 unknown :-1 s=CLOSED pgs=1237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:16.705 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:16.703+0000 7f2db7378640 1 --2- 172.21.15.123:0/4213695351 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2db00b2910 0x7f2db0097060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:16.705 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:16.703+0000 7f2db7378640 1 -- 172.21.15.123:0/4213695351 >> 172.21.15.123:0/4213695351 conn(0x7f2db00b1df0 msgr2=0x7f2db00706c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:49:16.705 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:16.703+0000 7f2db7378640 1 -- 172.21.15.123:0/4213695351 shutdown_connections 2024-09-16T15:49:16.705 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:16.703+0000 7f2db7378640 1 -- 172.21.15.123:0/4213695351 wait complete. 2024-09-16T15:49:16.720 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-09-16T15:49:16.720 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-09-16T15:49:16.837 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:16.835+0000 7f2ff1ca5640 1 -- 172.21.15.123:0/1174522032 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2fd4004010 msgr2=0x7f2fd4004810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:16.837 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:16.835+0000 7f2ff1ca5640 1 --2- 172.21.15.123:0/1174522032 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2fd4004010 0x7f2fd4004810 secure :-1 s=READY pgs=1239 cs=0 l=1 rev1=1 crypto rx=0x7f2fe0009b10 tx=0x7f2fe001d230 comp rx=0 tx=0).stop 2024-09-16T15:49:16.837 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:16.835+0000 7f2febfff640 1 -- 172.21.15.123:0/1174522032 reap_dead start 2024-09-16T15:49:16.837 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:16.835+0000 7f2ff1ca5640 1 -- 172.21.15.123:0/1174522032 shutdown_connections 2024-09-16T15:49:16.837 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:16.835+0000 7f2ff1ca5640 1 -- 172.21.15.123:0/1174522032 >> 172.21.15.123:0/1174522032 conn(0x7f2fec0b2c20 msgr2=0x7f2fec0b3030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:49:16.837 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:16.835+0000 7f2ff1ca5640 1 -- 172.21.15.123:0/1174522032 shutdown_connections 2024-09-16T15:49:16.837 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:16.835+0000 7f2ff1ca5640 1 -- 172.21.15.123:0/1174522032 wait complete. 2024-09-16T15:49:16.840 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:16.839+0000 7f2ff1ca5640 1 Processor -- start 2024-09-16T15:49:16.840 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:16.839+0000 7f2ff1ca5640 1 -- start start 2024-09-16T15:49:16.840 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:16.839+0000 7f2ff1ca5640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2fd4004010 0x7f2fec1e5d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:49:16.840 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:16.839+0000 7f2ff1ca5640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2fec0ccc90 0x7f2fec1e7810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:49:16.840 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:16.839+0000 7f2ff1ca5640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f2fec14f250 0x7f2fec14f6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:49:16.840 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:16.839+0000 7f2ff1ca5640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f2fec06e150 con 0x7f2fec0ccc90 2024-09-16T15:49:16.840 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:16.839+0000 7f2ff1ca5640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f2fec15b280 con 0x7f2fd4004010 2024-09-16T15:49:16.841 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:16.839+0000 7f2ff1ca5640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f2fec06e2d0 con 0x7f2fec14f250 2024-09-16T15:49:16.841 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:16.839+0000 7f2feaffd640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2fec0ccc90 0x7f2fec1e7810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:49:16.841 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:16.839+0000 7f2feaffd640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2fec0ccc90 0x7f2fec1e7810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:52304/0 (socket says 172.21.15.123:52304) 2024-09-16T15:49:16.841 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:16.839+0000 7f2feaffd640 1 -- 172.21.15.123:0/2416537316 learned_addr learned my addr 172.21.15.123:0/2416537316 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:49:16.841 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:16.839+0000 7f2feb7fe640 1 --2- 172.21.15.123:0/2416537316 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2fd4004010 0x7f2fec1e5d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:49:16.841 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:16.839+0000 7f2febfff640 1 --2- 172.21.15.123:0/2416537316 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f2fec14f250 0x7f2fec14f6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:49:16.841 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:16.839+0000 7f2feaffd640 1 -- 172.21.15.123:0/2416537316 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f2fec14f250 msgr2=0x7f2fec14f6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:16.841 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:16.839+0000 7f2feaffd640 1 --2- 172.21.15.123:0/2416537316 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f2fec14f250 0x7f2fec14f6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:16.841 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:16.839+0000 7f2feaffd640 1 -- 172.21.15.123:0/2416537316 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2fd4004010 msgr2=0x7f2fec1e5d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:16.841 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:16.839+0000 7f2feaffd640 1 --2- 172.21.15.123:0/2416537316 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2fd4004010 0x7f2fec1e5d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:16.841 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:16.839+0000 7f2feaffd640 1 -- 172.21.15.123:0/2416537316 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2fec0cd0f0 con 0x7f2fec0ccc90 2024-09-16T15:49:16.841 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:16.839+0000 7f2feaffd640 1 --2- 172.21.15.123:0/2416537316 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2fec0ccc90 0x7f2fec1e7810 secure :-1 s=READY pgs=1240 cs=0 l=1 rev1=1 crypto rx=0x7f2fe0009a60 tx=0x7f2fe0005620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:49:16.842 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:16.839+0000 7f2fd3fff640 1 -- 172.21.15.123:0/2416537316 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2fe00b1070 con 0x7f2fec0ccc90 2024-09-16T15:49:16.842 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:16.839+0000 7f2fd3fff640 1 -- 172.21.15.123:0/2416537316 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f2fe00b2040 con 0x7f2fec0ccc90 2024-09-16T15:49:16.842 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:16.839+0000 7f2ff1ca5640 1 -- 172.21.15.123:0/2416537316 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2fec14fc10 con 0x7f2fec0ccc90 2024-09-16T15:49:16.842 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:16.839+0000 7f2fd3fff640 1 -- 172.21.15.123:0/2416537316 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2fe00acc20 con 0x7f2fec0ccc90 2024-09-16T15:49:16.842 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:16.839+0000 7f2ff1ca5640 1 -- 172.21.15.123:0/2416537316 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2fec1e6680 con 0x7f2fec0ccc90 2024-09-16T15:49:16.842 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:16.839+0000 7f2ff1ca5640 1 -- 172.21.15.123:0/2416537316 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2fec0b4cd0 con 0x7f2fec0ccc90 2024-09-16T15:49:16.843 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:16.843+0000 7f2fd3fff640 1 -- 172.21.15.123:0/2416537316 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 19) ==== 97488+0+0 (secure 0 0 0) 0x7f2fe00c0020 con 0x7f2fec0ccc90 2024-09-16T15:49:16.843 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:16.843+0000 7f2fd3fff640 1 --2- 172.21.15.123:0/2416537316 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f2fd8073340 0x7f2fd8075800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:49:16.843 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:16.843+0000 7f2fd3fff640 1 -- 172.21.15.123:0/2416537316 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(115..115 src has 1..115) ==== 4885+0+0 (secure 0 0 0) 0x7f2fe012f1c0 con 0x7f2fec0ccc90 2024-09-16T15:49:16.843 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:16.843+0000 7f2feb7fe640 1 --2- 172.21.15.123:0/2416537316 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f2fd8073340 0x7f2fd8075800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:49:16.843 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:16.843+0000 7f2feb7fe640 1 --2- 172.21.15.123:0/2416537316 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f2fd8073340 0x7f2fd8075800 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f2fdc00ace0 tx=0x7f2fdc001740 comp rx=0 tx=0).ready entity=mgr.4150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:49:16.845 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:16.847+0000 7f2fd3fff640 1 -- 172.21.15.123:0/2416537316 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f2fe00c7020 con 0x7f2fec0ccc90 2024-09-16T15:49:17.005 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:17.007+0000 7f2ff1ca5640 1 -- 172.21.15.123:0/2416537316 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) -- 0x7f2fec0b4660 con 0x7f2fec0ccc90 2024-09-16T15:49:17.965 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:17.967+0000 7f2fd3fff640 1 -- 172.21.15.123:0/2416537316 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v116) ==== 138+0+0 (secure 0 0 0) 0x7f2fe0016f40 con 0x7f2fec0ccc90 2024-09-16T15:49:17.966 INFO:teuthology.orchestra.run.smithi123.stderr:pool 'cephfs_metadata' created 2024-09-16T15:49:17.969 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:17.967+0000 7f2ff1ca5640 1 -- 172.21.15.123:0/2416537316 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f2fd8073340 msgr2=0x7f2fd8075800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:17.969 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:17.967+0000 7f2ff1ca5640 1 --2- 172.21.15.123:0/2416537316 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f2fd8073340 0x7f2fd8075800 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f2fdc00ace0 tx=0x7f2fdc001740 comp rx=0 tx=0).stop 2024-09-16T15:49:17.969 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:17.967+0000 7f2ff1ca5640 1 -- 172.21.15.123:0/2416537316 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2fec0ccc90 msgr2=0x7f2fec1e7810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:17.969 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:17.967+0000 7f2ff1ca5640 1 --2- 172.21.15.123:0/2416537316 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2fec0ccc90 0x7f2fec1e7810 secure :-1 s=READY pgs=1240 cs=0 l=1 rev1=1 crypto rx=0x7f2fe0009a60 tx=0x7f2fe0005620 comp rx=0 tx=0).stop 2024-09-16T15:49:17.969 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:17.967+0000 7f2ff1ca5640 1 -- 172.21.15.123:0/2416537316 shutdown_connections 2024-09-16T15:49:17.969 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:17.967+0000 7f2ff1ca5640 1 --2- 172.21.15.123:0/2416537316 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f2fd8073340 0x7f2fd8075800 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:17.969 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:17.967+0000 7f2ff1ca5640 1 --2- 172.21.15.123:0/2416537316 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f2fec14f250 0x7f2fec14f6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:17.969 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:17.967+0000 7f2ff1ca5640 1 --2- 172.21.15.123:0/2416537316 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2fec0ccc90 0x7f2fec1e7810 unknown :-1 s=CLOSED pgs=1240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:17.969 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:17.967+0000 7f2ff1ca5640 1 --2- 172.21.15.123:0/2416537316 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2fd4004010 0x7f2fec1e5d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:17.969 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:17.967+0000 7f2ff1ca5640 1 -- 172.21.15.123:0/2416537316 >> 172.21.15.123:0/2416537316 conn(0x7f2fec0b2c20 msgr2=0x7f2fec15e4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:49:17.969 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:17.967+0000 7f2ff1ca5640 1 -- 172.21.15.123:0/2416537316 shutdown_connections 2024-09-16T15:49:17.969 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:17.967+0000 7f2ff1ca5640 1 -- 172.21.15.123:0/2416537316 wait complete. 2024-09-16T15:49:17.993 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-09-16T15:49:18.131 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:18.131+0000 7f3f1dcee640 1 -- 172.21.15.123:0/4187292144 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3f00004810 msgr2=0x7f3f00004c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:18.132 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:18.131+0000 7f3f1dcee640 1 --2- 172.21.15.123:0/4187292144 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3f00004810 0x7f3f00004c90 secure :-1 s=READY pgs=1242 cs=0 l=1 rev1=1 crypto rx=0x7f3f0c00c700 tx=0x7f3f0c01c680 comp rx=0 tx=0).stop 2024-09-16T15:49:18.132 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:18.131+0000 7f3f1dcee640 1 -- 172.21.15.123:0/4187292144 shutdown_connections 2024-09-16T15:49:18.132 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:18.131+0000 7f3f1dcee640 1 --2- 172.21.15.123:0/4187292144 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3f00004810 0x7f3f00004c90 unknown :-1 s=CLOSED pgs=1242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:18.132 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:18.131+0000 7f3f1dcee640 1 -- 172.21.15.123:0/4187292144 >> 172.21.15.123:0/4187292144 conn(0x7f3f180b1710 msgr2=0x7f3f180b1b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:49:18.132 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:18.131+0000 7f3f1dcee640 1 -- 172.21.15.123:0/4187292144 shutdown_connections 2024-09-16T15:49:18.132 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:18.131+0000 7f3f1dcee640 1 -- 172.21.15.123:0/4187292144 wait complete. 2024-09-16T15:49:18.132 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:18.131+0000 7f3f1dcee640 1 Processor -- start 2024-09-16T15:49:18.133 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:18.131+0000 7f3f1dcee640 1 -- start start 2024-09-16T15:49:18.134 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:18.135+0000 7f3f1dcee640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f3f00004810 0x7f3f180c9a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:49:18.134 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:18.135+0000 7f3f1dcee640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3f180b4670 0x7f3f180b2ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:49:18.134 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:18.135+0000 7f3f1dcee640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3f181f57a0 0x7f3f180c5de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:49:18.134 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:18.135+0000 7f3f1dcee640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f3f1815e960 con 0x7f3f181f57a0 2024-09-16T15:49:18.135 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:18.135+0000 7f3f1dcee640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f3f1815e7e0 con 0x7f3f180b4670 2024-09-16T15:49:18.135 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:18.135+0000 7f3f1dcee640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f3f1815ae90 con 0x7f3f00004810 2024-09-16T15:49:18.135 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:18.135+0000 7f3f17fff640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3f181f57a0 0x7f3f180c5de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:49:18.135 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:18.135+0000 7f3f17fff640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3f181f57a0 0x7f3f180c5de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:52322/0 (socket says 172.21.15.123:52322) 2024-09-16T15:49:18.135 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:18.135+0000 7f3f17fff640 1 -- 172.21.15.123:0/2797342734 learned_addr learned my addr 172.21.15.123:0/2797342734 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:49:18.135 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:18.135+0000 7f3f177fe640 1 --2- 172.21.15.123:0/2797342734 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f3f00004810 0x7f3f180c9a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:49:18.135 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:18.135+0000 7f3f16ffd640 1 --2- 172.21.15.123:0/2797342734 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3f180b4670 0x7f3f180b2ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:49:18.135 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:18.135+0000 7f3f17fff640 1 -- 172.21.15.123:0/2797342734 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f3f00004810 msgr2=0x7f3f180c9a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:18.135 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:18.135+0000 7f3f17fff640 1 --2- 172.21.15.123:0/2797342734 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f3f00004810 0x7f3f180c9a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:18.135 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:18.135+0000 7f3f17fff640 1 -- 172.21.15.123:0/2797342734 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3f180b4670 msgr2=0x7f3f180b2ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:18.135 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:18.135+0000 7f3f17fff640 1 --2- 172.21.15.123:0/2797342734 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3f180b4670 0x7f3f180b2ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:18.135 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:18.135+0000 7f3f17fff640 1 -- 172.21.15.123:0/2797342734 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3f181f5c50 con 0x7f3f181f57a0 2024-09-16T15:49:18.135 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:18.135+0000 7f3f16ffd640 1 --2- 172.21.15.123:0/2797342734 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3f180b4670 0x7f3f180b2ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T15:49:18.136 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:18.135+0000 7f3f17fff640 1 --2- 172.21.15.123:0/2797342734 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3f181f57a0 0x7f3f180c5de0 secure :-1 s=READY pgs=1243 cs=0 l=1 rev1=1 crypto rx=0x7f3f180b27a0 tx=0x7f3f1000dd90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:49:18.136 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:18.135+0000 7f3efffff640 1 -- 172.21.15.123:0/2797342734 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f10004b30 con 0x7f3f181f57a0 2024-09-16T15:49:18.136 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:18.135+0000 7f3efffff640 1 -- 172.21.15.123:0/2797342734 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f3f10004de0 con 0x7f3f181f57a0 2024-09-16T15:49:18.136 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:18.135+0000 7f3efffff640 1 -- 172.21.15.123:0/2797342734 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f1000bc20 con 0x7f3f181f57a0 2024-09-16T15:49:18.136 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:18.135+0000 7f3f1dcee640 1 -- 172.21.15.123:0/2797342734 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3f180b35d0 con 0x7f3f181f57a0 2024-09-16T15:49:18.136 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:18.135+0000 7f3f1dcee640 1 -- 172.21.15.123:0/2797342734 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3f180b3b90 con 0x7f3f181f57a0 2024-09-16T15:49:18.137 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:18.135+0000 7f3f1dcee640 1 -- 172.21.15.123:0/2797342734 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3f180cc7a0 con 0x7f3f181f57a0 2024-09-16T15:49:18.137 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:18.135+0000 7f3efffff640 1 -- 172.21.15.123:0/2797342734 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 19) ==== 97488+0+0 (secure 0 0 0) 0x7f3f1001c020 con 0x7f3f181f57a0 2024-09-16T15:49:18.137 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:18.135+0000 7f3efffff640 1 --2- 172.21.15.123:0/2797342734 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f3f04073370 0x7f3f04075830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:49:18.137 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:18.135+0000 7f3efffff640 1 -- 172.21.15.123:0/2797342734 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(116..116 src has 1..116) ==== 5255+0+0 (secure 0 0 0) 0x7f3f10094840 con 0x7f3f181f57a0 2024-09-16T15:49:18.137 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:18.135+0000 7f3f177fe640 1 --2- 172.21.15.123:0/2797342734 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f3f04073370 0x7f3f04075830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:49:18.137 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:18.139+0000 7f3f177fe640 1 --2- 172.21.15.123:0/2797342734 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f3f04073370 0x7f3f04075830 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f3f0800aef0 tx=0x7f3f0800c040 comp rx=0 tx=0).ready entity=mgr.4150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:49:18.140 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:18.139+0000 7f3efffff640 1 -- 172.21.15.123:0/2797342734 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f3f181f5c50 con 0x7f3f181f57a0 2024-09-16T15:49:18.300 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:18.299+0000 7f3f1dcee640 1 -- 172.21.15.123:0/2797342734 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) -- 0x7f3f180cd440 con 0x7f3f181f57a0 2024-09-16T15:49:18.970 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:18.971+0000 7f3efffff640 1 -- 172.21.15.123:0/2797342734 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v117) ==== 170+0+0 (secure 0 0 0) 0x7f3f1005f6b0 con 0x7f3f181f57a0 2024-09-16T15:49:18.970 INFO:teuthology.orchestra.run.smithi123.stderr:pool 'cephfs_data' created 2024-09-16T15:49:18.973 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:18.975+0000 7f3f1dcee640 1 -- 172.21.15.123:0/2797342734 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f3f04073370 msgr2=0x7f3f04075830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:18.973 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:18.975+0000 7f3f1dcee640 1 --2- 172.21.15.123:0/2797342734 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f3f04073370 0x7f3f04075830 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f3f0800aef0 tx=0x7f3f0800c040 comp rx=0 tx=0).stop 2024-09-16T15:49:18.973 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:18.975+0000 7f3f1dcee640 1 -- 172.21.15.123:0/2797342734 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3f181f57a0 msgr2=0x7f3f180c5de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:18.973 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:18.975+0000 7f3f1dcee640 1 --2- 172.21.15.123:0/2797342734 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3f181f57a0 0x7f3f180c5de0 secure :-1 s=READY pgs=1243 cs=0 l=1 rev1=1 crypto rx=0x7f3f180b27a0 tx=0x7f3f1000dd90 comp rx=0 tx=0).stop 2024-09-16T15:49:18.973 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:18.975+0000 7f3f1dcee640 1 -- 172.21.15.123:0/2797342734 shutdown_connections 2024-09-16T15:49:18.974 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:18.975+0000 7f3f1dcee640 1 --2- 172.21.15.123:0/2797342734 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f3f04073370 0x7f3f04075830 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:18.974 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:18.975+0000 7f3f1dcee640 1 --2- 172.21.15.123:0/2797342734 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3f181f57a0 0x7f3f180c5de0 unknown :-1 s=CLOSED pgs=1243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:18.974 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:18.975+0000 7f3f1dcee640 1 --2- 172.21.15.123:0/2797342734 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3f180b4670 0x7f3f180b2ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:18.974 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:18.975+0000 7f3f1dcee640 1 --2- 172.21.15.123:0/2797342734 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f3f00004810 0x7f3f180c9a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:18.974 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:18.975+0000 7f3f1dcee640 1 -- 172.21.15.123:0/2797342734 >> 172.21.15.123:0/2797342734 conn(0x7f3f180b1710 msgr2=0x7f3f180aedf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:49:18.974 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:18.975+0000 7f3f1dcee640 1 -- 172.21.15.123:0/2797342734 shutdown_connections 2024-09-16T15:49:18.974 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:18.975+0000 7f3f1dcee640 1 -- 172.21.15.123:0/2797342734 wait complete. 2024-09-16T15:49:19.007 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-09-16T15:49:19.135 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:19.135+0000 7fd806cb6640 1 -- 172.21.15.123:0/73829434 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd7e8004810 msgr2=0x7fd7e8004c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:19.135 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:19.135+0000 7fd806cb6640 1 --2- 172.21.15.123:0/73829434 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd7e8004810 0x7fd7e8004c90 secure :-1 s=READY pgs=1245 cs=0 l=1 rev1=1 crypto rx=0x7fd7f0009b10 tx=0x7fd7f001d230 comp rx=0 tx=0).stop 2024-09-16T15:49:19.135 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:19.135+0000 7fd806cb6640 1 -- 172.21.15.123:0/73829434 shutdown_connections 2024-09-16T15:49:19.135 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:19.135+0000 7fd806cb6640 1 --2- 172.21.15.123:0/73829434 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd7e8004810 0x7fd7e8004c90 unknown :-1 s=CLOSED pgs=1245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:19.136 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:19.135+0000 7fd806cb6640 1 -- 172.21.15.123:0/73829434 >> 172.21.15.123:0/73829434 conn(0x7fd8000b1710 msgr2=0x7fd8000b1b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:49:19.136 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:19.135+0000 7fd806cb6640 1 -- 172.21.15.123:0/73829434 shutdown_connections 2024-09-16T15:49:19.136 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:19.135+0000 7fd806cb6640 1 -- 172.21.15.123:0/73829434 wait complete. 2024-09-16T15:49:19.136 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:19.135+0000 7fd806cb6640 1 Processor -- start 2024-09-16T15:49:19.136 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:19.135+0000 7fd806cb6640 1 -- start start 2024-09-16T15:49:19.138 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:19.139+0000 7fd806cb6640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd7e8004810 0x7fd80014e040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:49:19.138 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:19.139+0000 7fd806cb6640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fd8001f2770 0x7fd800159630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:49:19.138 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:19.139+0000 7fd806cb6640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd800159b70 0x7fd800157ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:49:19.138 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:19.139+0000 7fd806cb6640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fd8000cef50 con 0x7fd7e8004810 2024-09-16T15:49:19.138 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:19.139+0000 7fd806cb6640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fd8000cedd0 con 0x7fd800159b70 2024-09-16T15:49:19.138 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:19.139+0000 7fd806cb6640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fd8000cf0d0 con 0x7fd8001f2770 2024-09-16T15:49:19.139 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:19.139+0000 7fd804a52640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd7e8004810 0x7fd80014e040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:49:19.139 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:19.139+0000 7fd804a52640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd7e8004810 0x7fd80014e040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:52338/0 (socket says 172.21.15.123:52338) 2024-09-16T15:49:19.139 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:19.139+0000 7fd804a52640 1 -- 172.21.15.123:0/997718858 learned_addr learned my addr 172.21.15.123:0/997718858 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:49:19.139 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:19.139+0000 7fd7fffff640 1 --2- 172.21.15.123:0/997718858 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fd8001f2770 0x7fd800159630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:49:19.139 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:19.139+0000 7fd805253640 1 --2- 172.21.15.123:0/997718858 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd800159b70 0x7fd800157ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:49:19.139 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:19.139+0000 7fd804a52640 1 -- 172.21.15.123:0/997718858 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fd8001f2770 msgr2=0x7fd800159630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:19.139 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:19.139+0000 7fd804a52640 1 --2- 172.21.15.123:0/997718858 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fd8001f2770 0x7fd800159630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:19.139 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:19.139+0000 7fd804a52640 1 -- 172.21.15.123:0/997718858 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd800159b70 msgr2=0x7fd800157ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:19.139 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:19.139+0000 7fd804a52640 1 --2- 172.21.15.123:0/997718858 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd800159b70 0x7fd800157ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:19.139 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:19.139+0000 7fd804a52640 1 -- 172.21.15.123:0/997718858 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd800158ed0 con 0x7fd7e8004810 2024-09-16T15:49:19.139 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:19.139+0000 7fd804a52640 1 --2- 172.21.15.123:0/997718858 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd7e8004810 0x7fd80014e040 secure :-1 s=READY pgs=1246 cs=0 l=1 rev1=1 crypto rx=0x7fd7f4004f10 tx=0x7fd7f4001640 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:49:19.139 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:19.139+0000 7fd7fd7fa640 1 -- 172.21.15.123:0/997718858 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7f4015020 con 0x7fd7e8004810 2024-09-16T15:49:19.139 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:19.139+0000 7fd7fd7fa640 1 -- 172.21.15.123:0/997718858 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fd7f4011960 con 0x7fd7e8004810 2024-09-16T15:49:19.140 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:19.139+0000 7fd806cb6640 1 -- 172.21.15.123:0/997718858 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd800158bf0 con 0x7fd7e8004810 2024-09-16T15:49:19.140 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:19.139+0000 7fd7fd7fa640 1 -- 172.21.15.123:0/997718858 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7f4011d90 con 0x7fd7e8004810 2024-09-16T15:49:19.140 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:19.139+0000 7fd806cb6640 1 -- 172.21.15.123:0/997718858 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd80015c470 con 0x7fd7e8004810 2024-09-16T15:49:19.140 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:19.139+0000 7fd806cb6640 1 -- 172.21.15.123:0/997718858 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd80006ffb0 con 0x7fd7e8004810 2024-09-16T15:49:19.140 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:19.139+0000 7fd7fd7fa640 1 -- 172.21.15.123:0/997718858 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 19) ==== 97488+0+0 (secure 0 0 0) 0x7fd7f4018020 con 0x7fd7e8004810 2024-09-16T15:49:19.141 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:19.139+0000 7fd7fd7fa640 1 --2- 172.21.15.123:0/997718858 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fd7ec073370 0x7fd7ec075830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:49:19.141 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:19.143+0000 7fd7fd7fa640 1 -- 172.21.15.123:0/997718858 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(117..117 src has 1..117) ==== 5637+0+0 (secure 0 0 0) 0x7fd7f4095650 con 0x7fd7e8004810 2024-09-16T15:49:19.141 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:19.143+0000 7fd7fffff640 1 --2- 172.21.15.123:0/997718858 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fd7ec073370 0x7fd7ec075830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:49:19.141 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:19.143+0000 7fd7fffff640 1 --2- 172.21.15.123:0/997718858 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fd7ec073370 0x7fd7ec075830 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fd7f00ad000 tx=0x7fd7f0019fd0 comp rx=0 tx=0).ready entity=mgr.4150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:49:19.144 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:19.143+0000 7fd7fd7fa640 1 -- 172.21.15.123:0/997718858 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7fd7f401f020 con 0x7fd7e8004810 2024-09-16T15:49:19.358 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:19.359+0000 7fd806cb6640 1 -- 172.21.15.123:0/997718858 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) -- 0x7fd800159f50 con 0x7fd7e8004810 2024-09-16T15:49:19.989 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:19.991+0000 7fd7fd7fa640 1 -- 172.21.15.123:0/997718858 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 18 and data pool 19 v73) ==== 174+0+0 (secure 0 0 0) 0x7fd7f40602a0 con 0x7fd7e8004810 2024-09-16T15:49:19.989 INFO:teuthology.orchestra.run.smithi123.stderr:new fs with metadata pool 18 and data pool 19 2024-09-16T15:49:19.995 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:19.995+0000 7fd806cb6640 1 -- 172.21.15.123:0/997718858 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fd7ec073370 msgr2=0x7fd7ec075830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:19.996 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:19.995+0000 7fd806cb6640 1 --2- 172.21.15.123:0/997718858 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fd7ec073370 0x7fd7ec075830 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fd7f00ad000 tx=0x7fd7f0019fd0 comp rx=0 tx=0).stop 2024-09-16T15:49:19.996 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:19.995+0000 7fd806cb6640 1 -- 172.21.15.123:0/997718858 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd7e8004810 msgr2=0x7fd80014e040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:19.996 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:19.995+0000 7fd806cb6640 1 --2- 172.21.15.123:0/997718858 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd7e8004810 0x7fd80014e040 secure :-1 s=READY pgs=1246 cs=0 l=1 rev1=1 crypto rx=0x7fd7f4004f10 tx=0x7fd7f4001640 comp rx=0 tx=0).stop 2024-09-16T15:49:19.996 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:19.995+0000 7fd806cb6640 1 -- 172.21.15.123:0/997718858 shutdown_connections 2024-09-16T15:49:19.996 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:19.995+0000 7fd806cb6640 1 --2- 172.21.15.123:0/997718858 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fd7ec073370 0x7fd7ec075830 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:19.996 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:19.995+0000 7fd806cb6640 1 --2- 172.21.15.123:0/997718858 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd800159b70 0x7fd800157ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:19.996 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:19.995+0000 7fd806cb6640 1 --2- 172.21.15.123:0/997718858 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fd8001f2770 0x7fd800159630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:19.996 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:19.995+0000 7fd806cb6640 1 --2- 172.21.15.123:0/997718858 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd7e8004810 0x7fd80014e040 unknown :-1 s=CLOSED pgs=1246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:19.996 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:19.995+0000 7fd806cb6640 1 -- 172.21.15.123:0/997718858 >> 172.21.15.123:0/997718858 conn(0x7fd8000b1710 msgr2=0x7fd8000b2a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:49:19.997 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:19.995+0000 7fd806cb6640 1 -- 172.21.15.123:0/997718858 shutdown_connections 2024-09-16T15:49:19.997 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:19.995+0000 7fd806cb6640 1 -- 172.21.15.123:0/997718858 wait complete. 2024-09-16T15:49:20.033 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-16T15:49:20.158 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:20.159+0000 7fb88e4f4640 1 -- 172.21.15.123:0/3848718923 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb8880cf5a0 msgr2=0x7fb88806eee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:20.158 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:20.159+0000 7fb88e4f4640 1 --2- 172.21.15.123:0/3848718923 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb8880cf5a0 0x7fb88806eee0 secure :-1 s=READY pgs=1248 cs=0 l=1 rev1=1 crypto rx=0x7fb87c00aa10 tx=0x7fb87c01db10 comp rx=0 tx=0).stop 2024-09-16T15:49:20.158 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:20.159+0000 7fb88e4f4640 1 -- 172.21.15.123:0/3848718923 shutdown_connections 2024-09-16T15:49:20.158 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:20.159+0000 7fb88e4f4640 1 --2- 172.21.15.123:0/3848718923 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb8880cf5a0 0x7fb88806eee0 unknown :-1 s=CLOSED pgs=1248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:20.159 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:20.159+0000 7fb88e4f4640 1 -- 172.21.15.123:0/3848718923 >> 172.21.15.123:0/3848718923 conn(0x7fb8880cfb10 msgr2=0x7fb8880cff20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:49:20.159 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:20.159+0000 7fb88e4f4640 1 -- 172.21.15.123:0/3848718923 shutdown_connections 2024-09-16T15:49:20.159 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:20.159+0000 7fb88e4f4640 1 -- 172.21.15.123:0/3848718923 wait complete. 2024-09-16T15:49:20.159 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:20.159+0000 7fb88e4f4640 1 Processor -- start 2024-09-16T15:49:20.159 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:20.159+0000 7fb88e4f4640 1 -- start start 2024-09-16T15:49:20.161 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:20.159+0000 7fb88e4f4640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb8880cf5a0 0x7fb8881506d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:49:20.161 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:20.159+0000 7fb88e4f4640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fb888158120 0x7fb888072730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:49:20.161 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:20.159+0000 7fb88e4f4640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb88815ec40 0x7fb888072c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:49:20.161 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:20.159+0000 7fb88e4f4640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fb88815b7b0 con 0x7fb8880cf5a0 2024-09-16T15:49:20.161 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:20.159+0000 7fb88e4f4640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fb88815b630 con 0x7fb88815ec40 2024-09-16T15:49:20.162 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:20.159+0000 7fb88e4f4640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fb8881683e0 con 0x7fb888158120 2024-09-16T15:49:20.162 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:20.159+0000 7fb887fff640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb8880cf5a0 0x7fb8881506d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:49:20.162 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:20.159+0000 7fb887fff640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb8880cf5a0 0x7fb8881506d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:52362/0 (socket says 172.21.15.123:52362) 2024-09-16T15:49:20.162 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:20.159+0000 7fb887fff640 1 -- 172.21.15.123:0/1159378619 learned_addr learned my addr 172.21.15.123:0/1159378619 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:49:20.162 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:20.159+0000 7fb887fff640 1 -- 172.21.15.123:0/1159378619 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fb888158120 msgr2=0x7fb888072730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:20.162 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:20.159+0000 7fb887fff640 1 --2- 172.21.15.123:0/1159378619 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fb888158120 0x7fb888072730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:20.162 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:20.159+0000 7fb887fff640 1 -- 172.21.15.123:0/1159378619 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb88815ec40 msgr2=0x7fb888072c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:20.162 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:20.159+0000 7fb887fff640 1 --2- 172.21.15.123:0/1159378619 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb88815ec40 0x7fb888072c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:20.162 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:20.159+0000 7fb887fff640 1 -- 172.21.15.123:0/1159378619 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb8881f8b40 con 0x7fb8880cf5a0 2024-09-16T15:49:20.162 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:20.163+0000 7fb887fff640 1 --2- 172.21.15.123:0/1159378619 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb8880cf5a0 0x7fb8881506d0 secure :-1 s=READY pgs=1249 cs=0 l=1 rev1=1 crypto rx=0x7fb87800c500 tx=0x7fb87800c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:49:20.162 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:20.163+0000 7fb884ff9640 1 -- 172.21.15.123:0/1159378619 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb878018070 con 0x7fb8880cf5a0 2024-09-16T15:49:20.162 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:20.163+0000 7fb884ff9640 1 -- 172.21.15.123:0/1159378619 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fb878003600 con 0x7fb8880cf5a0 2024-09-16T15:49:20.162 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:20.163+0000 7fb88e4f4640 1 -- 172.21.15.123:0/1159378619 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb888154fb0 con 0x7fb8880cf5a0 2024-09-16T15:49:20.163 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:20.163+0000 7fb884ff9640 1 -- 172.21.15.123:0/1159378619 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb878003ad0 con 0x7fb8880cf5a0 2024-09-16T15:49:20.163 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:20.163+0000 7fb88e4f4640 1 -- 172.21.15.123:0/1159378619 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb888155570 con 0x7fb8880cf5a0 2024-09-16T15:49:20.163 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:20.163+0000 7fb88e4f4640 1 -- 172.21.15.123:0/1159378619 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb88806eee0 con 0x7fb8880cf5a0 2024-09-16T15:49:20.163 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:20.163+0000 7fb884ff9640 1 -- 172.21.15.123:0/1159378619 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 19) ==== 97488+0+0 (secure 0 0 0) 0x7fb87801b070 con 0x7fb8880cf5a0 2024-09-16T15:49:20.163 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:20.163+0000 7fb884ff9640 1 --2- 172.21.15.123:0/1159378619 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fb874073340 0x7fb874075800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:49:20.164 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:20.163+0000 7fb8877fe640 1 --2- 172.21.15.123:0/1159378619 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fb874073340 0x7fb874075800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:49:20.164 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:20.163+0000 7fb884ff9640 1 -- 172.21.15.123:0/1159378619 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(118..118 src has 1..118) ==== 5737+0+0 (secure 0 0 0) 0x7fb878093a50 con 0x7fb8880cf5a0 2024-09-16T15:49:20.164 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:20.163+0000 7fb8877fe640 1 --2- 172.21.15.123:0/1159378619 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fb874073340 0x7fb874075800 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fb87c0b0000 tx=0x7fb87c01a8b0 comp rx=0 tx=0).ready entity=mgr.4150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:49:20.167 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:20.167+0000 7fb884ff9640 1 -- 172.21.15.123:0/1159378619 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7fb878003c70 con 0x7fb8880cf5a0 2024-09-16T15:49:20.343 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:20.343+0000 7fb88e4f4640 1 -- 172.21.15.123:0/1159378619 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fb8880d06c0 con 0x7fb8880cf5a0 2024-09-16T15:49:20.346 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:20.347+0000 7fb884ff9640 1 -- 172.21.15.123:0/1159378619 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v118) ==== 74+0+16856 (secure 0 0 0) 0x7fb87807de20 con 0x7fb8880cf5a0 2024-09-16T15:49:20.346 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-09-16T15:49:20.346 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":118,"fsid":"56077f89-18c8-486f-ae7e-eecc0bb7822c","created":"2024-09-16T15:42:26.364892+0000","modified":"2024-09-16T15:49:19.361871+0000","last_up_change":"2024-09-16T15:42:38.707234+0000","last_in_change":"2024-09-16T15:42:29.423662+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":19,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-16T15:42:39.287764+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":18,"pool_name":"cephfs_metadata","create_time":"2024-09-16T15:49:17.014388+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"118","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":19,"pool_name":"cephfs_data","create_time":"2024-09-16T15:49:18.311134+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"118","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9e7c1955-a607-4671-9be7-f2ef760ceeaa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":117,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6800","nonce":3882044969},{"type":"v1","addr":"172.21.15.123:6801","nonce":3882044969}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6802","nonce":3882044969},{"type":"v1","addr":"172.21.15.123:6803","nonce":3882044969}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6806","nonce":3882044969},{"type":"v1","addr":"172.21.15.123:6807","nonce":3882044969}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6804","nonce":3882044969},{"type":"v1","addr":"172.21.15.123:6805","nonce":3882044969}]},"public_addr":"172.21.15.123:6801/3882044969","cluster_addr":"172.21.15.123:6803/3882044969","heartbeat_back_addr":"172.21.15.123:6807/3882044969","heartbeat_front_addr":"172.21.15.123:6805/3882044969","state":["exists","up"]},{"osd":1,"uuid":"c540cfde-0977-4742-9b17-fd634668e010","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":117,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6816","nonce":4037001850},{"type":"v1","addr":"172.21.15.123:6817","nonce":4037001850}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6818","nonce":4037001850},{"type":"v1","addr":"172.21.15.123:6819","nonce":4037001850}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6822","nonce":4037001850},{"type":"v1","addr":"172.21.15.123:6823","nonce":4037001850}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6820","nonce":4037001850},{"type":"v1","addr":"172.21.15.123:6821","nonce":4037001850}]},"public_addr":"172.21.15.123:6817/4037001850","cluster_addr":"172.21.15.123:6819/4037001850","heartbeat_back_addr":"172.21.15.123:6823/4037001850","heartbeat_front_addr":"172.21.15.123:6821/4037001850","state":["exists","up"]},{"osd":2,"uuid":"9864a496-f169-486b-95c4-1d59162793a7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":117,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6824","nonce":1332143088},{"type":"v1","addr":"172.21.15.123:6825","nonce":1332143088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6826","nonce":1332143088},{"type":"v1","addr":"172.21.15.123:6827","nonce":1332143088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6830","nonce":1332143088},{"type":"v1","addr":"172.21.15.123:6831","nonce":1332143088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6828","nonce":1332143088},{"type":"v1","addr":"172.21.15.123:6829","nonce":1332143088}]},"public_addr":"172.21.15.123:6825/1332143088","cluster_addr":"172.21.15.123:6827/1332143088","heartbeat_back_addr":"172.21.15.123:6831/1332143088","heartbeat_front_addr":"172.21.15.123:6829/1332143088","state":["exists","up"]},{"osd":3,"uuid":"c6d1f817-f074-4325-95e1-578eba13edda","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":117,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6808","nonce":988586998},{"type":"v1","addr":"172.21.15.123:6809","nonce":988586998}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6810","nonce":988586998},{"type":"v1","addr":"172.21.15.123:6811","nonce":988586998}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6814","nonce":988586998},{"type":"v1","addr":"172.21.15.123:6815","nonce":988586998}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6812","nonce":988586998},{"type":"v1","addr":"172.21.15.123:6813","nonce":988586998}]},"public_addr":"172.21.15.123:6809/988586998","cluster_addr":"172.21.15.123:6811/988586998","heartbeat_back_addr":"172.21.15.123:6815/988586998","heartbeat_front_addr":"172.21.15.123:6813/988586998","state":["exists","up"]},{"osd":4,"uuid":"200a8df4-50ef-40a5-b48a-58981269c086","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":117,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6816","nonce":3387813890},{"type":"v1","addr":"172.21.15.184:6817","nonce":3387813890}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6818","nonce":3387813890},{"type":"v1","addr":"172.21.15.184:6819","nonce":3387813890}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6822","nonce":3387813890},{"type":"v1","addr":"172.21.15.184:6823","nonce":3387813890}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6820","nonce":3387813890},{"type":"v1","addr":"172.21.15.184:6821","nonce":3387813890}]},"public_addr":"172.21.15.184:6817/3387813890","cluster_addr":"172.21.15.184:6819/3387813890","heartbeat_back_addr":"172.21.15.184:6823/3387813890","heartbeat_front_addr":"172.21.15.184:6821/3387813890","state":["exists","up"]},{"osd":5,"uuid":"32d5a900-da3d-412c-bc19-13861a2368d8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":117,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6808","nonce":2680916017},{"type":"v1","addr":"172.21.15.184:6809","nonce":2680916017}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6810","nonce":2680916017},{"type":"v1","addr":"172.21.15.184:6811","nonce":2680916017}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6814","nonce":2680916017},{"type":"v1","addr":"172.21.15.184:6815","nonce":2680916017}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6812","nonce":2680916017},{"type":"v1","addr":"172.21.15.184:6813","nonce":2680916017}]},"public_addr":"172.21.15.184:6809/2680916017","cluster_addr":"172.21.15.184:6811/2680916017","heartbeat_back_addr":"172.21.15.184:6815/2680916017","heartbeat_front_addr":"172.21.15.184:6813/2680916017","state":["exists","up"]},{"osd":6,"uuid":"cde6f39e-aa1b-4260-ac19-774ee6aaca72","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":117,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6824","nonce":2183120457},{"type":"v1","addr":"172.21.15.184:6825","nonce":2183120457}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6826","nonce":2183120457},{"type":"v1","addr":"172.21.15.184:6827","nonce":2183120457}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6830","nonce":2183120457},{"type":"v1","addr":"172.21.15.184:6831","nonce":2183120457}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6828","nonce":2183120457},{"type":"v1","addr":"172.21.15.184:6829","nonce":2183120457}]},"public_addr":"172.21.15.184:6825/2183120457","cluster_addr":"172.21.15.184:6827/2183120457","heartbeat_back_addr":"172.21.15.184:6831/2183120457","heartbeat_front_addr":"172.21.15.184:6829/2183120457","state":["exists","up"]},{"osd":7,"uuid":"4f381778-eee1-47c6-804f-df0e6b33e313","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":117,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6800","nonce":4102957285},{"type":"v1","addr":"172.21.15.184:6801","nonce":4102957285}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6802","nonce":4102957285},{"type":"v1","addr":"172.21.15.184:6803","nonce":4102957285}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6806","nonce":4102957285},{"type":"v1","addr":"172.21.15.184:6807","nonce":4102957285}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6804","nonce":4102957285},{"type":"v1","addr":"172.21.15.184:6805","nonce":4102957285}]},"public_addr":"172.21.15.184:6801/4102957285","cluster_addr":"172.21.15.184:6803/4102957285","heartbeat_back_addr":"172.21.15.184:6807/4102957285","heartbeat_front_addr":"172.21.15.184:6805/4102957285","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:35.972210+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.222007+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.198617+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.385663+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.358935+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.156066+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.752973+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:35.682846+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":17,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T15:49:20.349 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:20.347+0000 7fb88e4f4640 1 -- 172.21.15.123:0/1159378619 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fb874073340 msgr2=0x7fb874075800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:20.349 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:20.347+0000 7fb88e4f4640 1 --2- 172.21.15.123:0/1159378619 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fb874073340 0x7fb874075800 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fb87c0b0000 tx=0x7fb87c01a8b0 comp rx=0 tx=0).stop 2024-09-16T15:49:20.349 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:20.347+0000 7fb88e4f4640 1 -- 172.21.15.123:0/1159378619 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb8880cf5a0 msgr2=0x7fb8881506d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:20.349 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:20.347+0000 7fb88e4f4640 1 --2- 172.21.15.123:0/1159378619 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb8880cf5a0 0x7fb8881506d0 secure :-1 s=READY pgs=1249 cs=0 l=1 rev1=1 crypto rx=0x7fb87800c500 tx=0x7fb87800c9c0 comp rx=0 tx=0).stop 2024-09-16T15:49:20.349 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:20.347+0000 7fb88e4f4640 1 -- 172.21.15.123:0/1159378619 shutdown_connections 2024-09-16T15:49:20.349 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:20.347+0000 7fb88e4f4640 1 --2- 172.21.15.123:0/1159378619 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fb874073340 0x7fb874075800 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:20.349 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:20.347+0000 7fb88e4f4640 1 --2- 172.21.15.123:0/1159378619 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb88815ec40 0x7fb888072c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:20.349 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:20.347+0000 7fb88e4f4640 1 --2- 172.21.15.123:0/1159378619 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fb888158120 0x7fb888072730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:20.349 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:20.347+0000 7fb88e4f4640 1 --2- 172.21.15.123:0/1159378619 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb8880cf5a0 0x7fb8881506d0 unknown :-1 s=CLOSED pgs=1249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:20.349 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:20.347+0000 7fb88e4f4640 1 -- 172.21.15.123:0/1159378619 >> 172.21.15.123:0/1159378619 conn(0x7fb8880cfb10 msgr2=0x7fb88815d350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:49:20.350 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:20.347+0000 7fb88e4f4640 1 -- 172.21.15.123:0/1159378619 shutdown_connections 2024-09-16T15:49:20.350 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:20.347+0000 7fb88e4f4640 1 -- 172.21.15.123:0/1159378619 wait complete. 2024-09-16T15:49:20.366 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-16T15:49:20.482 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:20.483+0000 7f4cd9931640 1 -- 172.21.15.123:0/619354505 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4cd415bca0 msgr2=0x7f4cd40d0630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:20.482 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:20.483+0000 7f4cd9931640 1 --2- 172.21.15.123:0/619354505 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4cd415bca0 0x7f4cd40d0630 secure :-1 s=READY pgs=1251 cs=0 l=1 rev1=1 crypto rx=0x7f4cd406c970 tx=0x7f4cc401c740 comp rx=0 tx=0).stop 2024-09-16T15:49:20.482 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:20.483+0000 7f4cd9931640 1 -- 172.21.15.123:0/619354505 shutdown_connections 2024-09-16T15:49:20.482 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:20.483+0000 7f4cd9931640 1 --2- 172.21.15.123:0/619354505 >> v2:172.21.15.184:3300/0 conn(0x7f4cd4074600 0x7f4cd406e750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:20.482 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:20.483+0000 7f4cd9931640 1 --2- 172.21.15.123:0/619354505 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4cd415bca0 0x7f4cd40d0630 unknown :-1 s=CLOSED pgs=1251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:20.482 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:20.483+0000 7f4cd9931640 1 -- 172.21.15.123:0/619354505 >> 172.21.15.123:0/619354505 conn(0x7f4cd40cfb10 msgr2=0x7f4cd40cff20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:49:20.482 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:20.483+0000 7f4cd9931640 1 -- 172.21.15.123:0/619354505 shutdown_connections 2024-09-16T15:49:20.482 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:20.483+0000 7f4cd9931640 1 -- 172.21.15.123:0/619354505 wait complete. 2024-09-16T15:49:20.483 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:20.483+0000 7f4cd9931640 1 Processor -- start 2024-09-16T15:49:20.483 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:20.483+0000 7f4cd9931640 1 -- start start 2024-09-16T15:49:20.485 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:20.483+0000 7f4cd9931640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4cd40726c0 0x7f4cd41564a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:49:20.485 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:20.483+0000 7f4cd9931640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4cd4074600 0x7f4cd41582a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:49:20.485 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:20.483+0000 7f4cd9931640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f4cd415bca0 0x7f4cd40c9200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:49:20.485 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:20.483+0000 7f4cd9931640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f4cd4168bd0 con 0x7f4cd40726c0 2024-09-16T15:49:20.485 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:20.483+0000 7f4cd9931640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f4cd4168a50 con 0x7f4cd4074600 2024-09-16T15:49:20.485 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:20.483+0000 7f4cd9931640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f4cd4168d50 con 0x7f4cd415bca0 2024-09-16T15:49:20.485 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:20.483+0000 7f4cd2ffd640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4cd40726c0 0x7f4cd41564a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:49:20.485 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:20.483+0000 7f4cd2ffd640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4cd40726c0 0x7f4cd41564a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:52380/0 (socket says 172.21.15.123:52380) 2024-09-16T15:49:20.485 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:20.483+0000 7f4cd2ffd640 1 -- 172.21.15.123:0/4174268755 learned_addr learned my addr 172.21.15.123:0/4174268755 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:49:20.485 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:20.483+0000 7f4cd2ffd640 1 -- 172.21.15.123:0/4174268755 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f4cd415bca0 msgr2=0x7f4cd40c9200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:20.485 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:20.483+0000 7f4cd2ffd640 1 --2- 172.21.15.123:0/4174268755 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f4cd415bca0 0x7f4cd40c9200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:20.485 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:20.483+0000 7f4cd2ffd640 1 -- 172.21.15.123:0/4174268755 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4cd4074600 msgr2=0x7f4cd41582a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:20.485 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:20.483+0000 7f4cd2ffd640 1 --2- 172.21.15.123:0/4174268755 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4cd4074600 0x7f4cd41582a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:20.486 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:20.483+0000 7f4cd2ffd640 1 -- 172.21.15.123:0/4174268755 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4cd41f89c0 con 0x7f4cd40726c0 2024-09-16T15:49:20.486 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:20.483+0000 7f4cd2ffd640 1 --2- 172.21.15.123:0/4174268755 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4cd40726c0 0x7f4cd41564a0 secure :-1 s=READY pgs=1252 cs=0 l=1 rev1=1 crypto rx=0x7f4cc4009790 tx=0x7f4cc4004040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:49:20.486 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:20.483+0000 7f4cbbfff640 1 -- 172.21.15.123:0/4174268755 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4cc40bb070 con 0x7f4cd40726c0 2024-09-16T15:49:20.486 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:20.483+0000 7f4cbbfff640 1 -- 172.21.15.123:0/4174268755 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f4cc4005030 con 0x7f4cd40726c0 2024-09-16T15:49:20.486 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:20.483+0000 7f4cbbfff640 1 -- 172.21.15.123:0/4174268755 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4cc40b2040 con 0x7f4cd40726c0 2024-09-16T15:49:20.486 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:20.483+0000 7f4cd9931640 1 -- 172.21.15.123:0/4174268755 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4cd41f7db0 con 0x7f4cd40726c0 2024-09-16T15:49:20.486 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:20.483+0000 7f4cd9931640 1 -- 172.21.15.123:0/4174268755 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4cd41f8370 con 0x7f4cd40726c0 2024-09-16T15:49:20.487 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:20.487+0000 7f4cd9931640 1 -- 172.21.15.123:0/4174268755 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4cd40c53b0 con 0x7f4cd40726c0 2024-09-16T15:49:20.487 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:20.487+0000 7f4cbbfff640 1 -- 172.21.15.123:0/4174268755 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 19) ==== 97488+0+0 (secure 0 0 0) 0x7f4cc40be070 con 0x7f4cd40726c0 2024-09-16T15:49:20.487 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:20.487+0000 7f4cbbfff640 1 --2- 172.21.15.123:0/4174268755 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f4cc0073340 0x7f4cc0075800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:49:20.487 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:20.487+0000 7f4cd27fc640 1 --2- 172.21.15.123:0/4174268755 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f4cc0073340 0x7f4cc0075800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:49:20.487 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:20.487+0000 7f4cbbfff640 1 -- 172.21.15.123:0/4174268755 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(118..118 src has 1..118) ==== 5737+0+0 (secure 0 0 0) 0x7f4cc412e200 con 0x7f4cd40726c0 2024-09-16T15:49:20.487 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:20.487+0000 7f4cd27fc640 1 --2- 172.21.15.123:0/4174268755 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f4cc0073340 0x7f4cc0075800 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f4cd40cf870 tx=0x7f4cc8007a00 comp rx=0 tx=0).ready entity=mgr.4150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:49:20.490 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:20.491+0000 7f4cbbfff640 1 -- 172.21.15.123:0/4174268755 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f4cc40f30b0 con 0x7f4cd40726c0 2024-09-16T15:49:20.663 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:20.663+0000 7f4cd9931640 1 -- 172.21.15.123:0/4174268755 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f4cd415f630 con 0x7f4cd40726c0 2024-09-16T15:49:20.664 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:20.663+0000 7f4cbbfff640 1 -- 172.21.15.123:0/4174268755 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v118) ==== 74+0+16856 (secure 0 0 0) 0x7f4cc40163f0 con 0x7f4cd40726c0 2024-09-16T15:49:20.664 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-09-16T15:49:20.664 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":118,"fsid":"56077f89-18c8-486f-ae7e-eecc0bb7822c","created":"2024-09-16T15:42:26.364892+0000","modified":"2024-09-16T15:49:19.361871+0000","last_up_change":"2024-09-16T15:42:38.707234+0000","last_in_change":"2024-09-16T15:42:29.423662+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":19,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-16T15:42:39.287764+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":18,"pool_name":"cephfs_metadata","create_time":"2024-09-16T15:49:17.014388+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"118","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":19,"pool_name":"cephfs_data","create_time":"2024-09-16T15:49:18.311134+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"118","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9e7c1955-a607-4671-9be7-f2ef760ceeaa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":117,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6800","nonce":3882044969},{"type":"v1","addr":"172.21.15.123:6801","nonce":3882044969}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6802","nonce":3882044969},{"type":"v1","addr":"172.21.15.123:6803","nonce":3882044969}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6806","nonce":3882044969},{"type":"v1","addr":"172.21.15.123:6807","nonce":3882044969}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6804","nonce":3882044969},{"type":"v1","addr":"172.21.15.123:6805","nonce":3882044969}]},"public_addr":"172.21.15.123:6801/3882044969","cluster_addr":"172.21.15.123:6803/3882044969","heartbeat_back_addr":"172.21.15.123:6807/3882044969","heartbeat_front_addr":"172.21.15.123:6805/3882044969","state":["exists","up"]},{"osd":1,"uuid":"c540cfde-0977-4742-9b17-fd634668e010","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":117,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6816","nonce":4037001850},{"type":"v1","addr":"172.21.15.123:6817","nonce":4037001850}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6818","nonce":4037001850},{"type":"v1","addr":"172.21.15.123:6819","nonce":4037001850}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6822","nonce":4037001850},{"type":"v1","addr":"172.21.15.123:6823","nonce":4037001850}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6820","nonce":4037001850},{"type":"v1","addr":"172.21.15.123:6821","nonce":4037001850}]},"public_addr":"172.21.15.123:6817/4037001850","cluster_addr":"172.21.15.123:6819/4037001850","heartbeat_back_addr":"172.21.15.123:6823/4037001850","heartbeat_front_addr":"172.21.15.123:6821/4037001850","state":["exists","up"]},{"osd":2,"uuid":"9864a496-f169-486b-95c4-1d59162793a7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":117,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6824","nonce":1332143088},{"type":"v1","addr":"172.21.15.123:6825","nonce":1332143088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6826","nonce":1332143088},{"type":"v1","addr":"172.21.15.123:6827","nonce":1332143088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6830","nonce":1332143088},{"type":"v1","addr":"172.21.15.123:6831","nonce":1332143088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6828","nonce":1332143088},{"type":"v1","addr":"172.21.15.123:6829","nonce":1332143088}]},"public_addr":"172.21.15.123:6825/1332143088","cluster_addr":"172.21.15.123:6827/1332143088","heartbeat_back_addr":"172.21.15.123:6831/1332143088","heartbeat_front_addr":"172.21.15.123:6829/1332143088","state":["exists","up"]},{"osd":3,"uuid":"c6d1f817-f074-4325-95e1-578eba13edda","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":117,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6808","nonce":988586998},{"type":"v1","addr":"172.21.15.123:6809","nonce":988586998}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6810","nonce":988586998},{"type":"v1","addr":"172.21.15.123:6811","nonce":988586998}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6814","nonce":988586998},{"type":"v1","addr":"172.21.15.123:6815","nonce":988586998}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6812","nonce":988586998},{"type":"v1","addr":"172.21.15.123:6813","nonce":988586998}]},"public_addr":"172.21.15.123:6809/988586998","cluster_addr":"172.21.15.123:6811/988586998","heartbeat_back_addr":"172.21.15.123:6815/988586998","heartbeat_front_addr":"172.21.15.123:6813/988586998","state":["exists","up"]},{"osd":4,"uuid":"200a8df4-50ef-40a5-b48a-58981269c086","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":117,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6816","nonce":3387813890},{"type":"v1","addr":"172.21.15.184:6817","nonce":3387813890}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6818","nonce":3387813890},{"type":"v1","addr":"172.21.15.184:6819","nonce":3387813890}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6822","nonce":3387813890},{"type":"v1","addr":"172.21.15.184:6823","nonce":3387813890}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6820","nonce":3387813890},{"type":"v1","addr":"172.21.15.184:6821","nonce":3387813890}]},"public_addr":"172.21.15.184:6817/3387813890","cluster_addr":"172.21.15.184:6819/3387813890","heartbeat_back_addr":"172.21.15.184:6823/3387813890","heartbeat_front_addr":"172.21.15.184:6821/3387813890","state":["exists","up"]},{"osd":5,"uuid":"32d5a900-da3d-412c-bc19-13861a2368d8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":117,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6808","nonce":2680916017},{"type":"v1","addr":"172.21.15.184:6809","nonce":2680916017}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6810","nonce":2680916017},{"type":"v1","addr":"172.21.15.184:6811","nonce":2680916017}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6814","nonce":2680916017},{"type":"v1","addr":"172.21.15.184:6815","nonce":2680916017}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6812","nonce":2680916017},{"type":"v1","addr":"172.21.15.184:6813","nonce":2680916017}]},"public_addr":"172.21.15.184:6809/2680916017","cluster_addr":"172.21.15.184:6811/2680916017","heartbeat_back_addr":"172.21.15.184:6815/2680916017","heartbeat_front_addr":"172.21.15.184:6813/2680916017","state":["exists","up"]},{"osd":6,"uuid":"cde6f39e-aa1b-4260-ac19-774ee6aaca72","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":117,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6824","nonce":2183120457},{"type":"v1","addr":"172.21.15.184:6825","nonce":2183120457}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6826","nonce":2183120457},{"type":"v1","addr":"172.21.15.184:6827","nonce":2183120457}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6830","nonce":2183120457},{"type":"v1","addr":"172.21.15.184:6831","nonce":2183120457}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6828","nonce":2183120457},{"type":"v1","addr":"172.21.15.184:6829","nonce":2183120457}]},"public_addr":"172.21.15.184:6825/2183120457","cluster_addr":"172.21.15.184:6827/2183120457","heartbeat_back_addr":"172.21.15.184:6831/2183120457","heartbeat_front_addr":"172.21.15.184:6829/2183120457","state":["exists","up"]},{"osd":7,"uuid":"4f381778-eee1-47c6-804f-df0e6b33e313","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":117,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6800","nonce":4102957285},{"type":"v1","addr":"172.21.15.184:6801","nonce":4102957285}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6802","nonce":4102957285},{"type":"v1","addr":"172.21.15.184:6803","nonce":4102957285}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6806","nonce":4102957285},{"type":"v1","addr":"172.21.15.184:6807","nonce":4102957285}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6804","nonce":4102957285},{"type":"v1","addr":"172.21.15.184:6805","nonce":4102957285}]},"public_addr":"172.21.15.184:6801/4102957285","cluster_addr":"172.21.15.184:6803/4102957285","heartbeat_back_addr":"172.21.15.184:6807/4102957285","heartbeat_front_addr":"172.21.15.184:6805/4102957285","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:35.972210+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.222007+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.198617+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.385663+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.358935+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.156066+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.752973+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:35.682846+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":17,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T15:49:20.666 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:20.667+0000 7f4cd9931640 1 -- 172.21.15.123:0/4174268755 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f4cc0073340 msgr2=0x7f4cc0075800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:20.666 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:20.667+0000 7f4cd9931640 1 --2- 172.21.15.123:0/4174268755 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f4cc0073340 0x7f4cc0075800 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f4cd40cf870 tx=0x7f4cc8007a00 comp rx=0 tx=0).stop 2024-09-16T15:49:20.666 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:20.667+0000 7f4cd9931640 1 -- 172.21.15.123:0/4174268755 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4cd40726c0 msgr2=0x7f4cd41564a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:20.667 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:20.667+0000 7f4cd9931640 1 --2- 172.21.15.123:0/4174268755 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4cd40726c0 0x7f4cd41564a0 secure :-1 s=READY pgs=1252 cs=0 l=1 rev1=1 crypto rx=0x7f4cc4009790 tx=0x7f4cc4004040 comp rx=0 tx=0).stop 2024-09-16T15:49:20.667 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:20.667+0000 7f4cd9931640 1 -- 172.21.15.123:0/4174268755 shutdown_connections 2024-09-16T15:49:20.667 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:20.667+0000 7f4cd9931640 1 --2- 172.21.15.123:0/4174268755 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f4cc0073340 0x7f4cc0075800 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:20.667 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:20.667+0000 7f4cd9931640 1 --2- 172.21.15.123:0/4174268755 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f4cd415bca0 0x7f4cd40c9200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:20.667 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:20.667+0000 7f4cd9931640 1 --2- 172.21.15.123:0/4174268755 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4cd4074600 0x7f4cd41582a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:20.667 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:20.667+0000 7f4cd9931640 1 --2- 172.21.15.123:0/4174268755 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4cd40726c0 0x7f4cd41564a0 unknown :-1 s=CLOSED pgs=1252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:20.667 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:20.667+0000 7f4cd9931640 1 -- 172.21.15.123:0/4174268755 >> 172.21.15.123:0/4174268755 conn(0x7f4cd40cfb10 msgr2=0x7f4cd40ca140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:49:20.667 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:20.667+0000 7f4cd9931640 1 -- 172.21.15.123:0/4174268755 shutdown_connections 2024-09-16T15:49:20.667 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:20.667+0000 7f4cd9931640 1 -- 172.21.15.123:0/4174268755 wait complete. 2024-09-16T15:49:20.682 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-09-16T15:49:20.800 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:20.799+0000 7f3a468e9640 1 -- 172.21.15.123:0/1814706719 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3a2c003a60 msgr2=0x7f3a2c003e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:20.800 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:20.799+0000 7f3a468e9640 1 --2- 172.21.15.123:0/1814706719 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3a2c003a60 0x7f3a2c003e80 secure :-1 s=READY pgs=1254 cs=0 l=1 rev1=1 crypto rx=0x7f3a30009b30 tx=0x7f3a3001d250 comp rx=0 tx=0).stop 2024-09-16T15:49:20.800 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:20.799+0000 7f3a468e9640 1 -- 172.21.15.123:0/1814706719 shutdown_connections 2024-09-16T15:49:20.801 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:20.799+0000 7f3a468e9640 1 --2- 172.21.15.123:0/1814706719 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3a2c003a60 0x7f3a2c003e80 unknown :-1 s=CLOSED pgs=1254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:20.801 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:20.799+0000 7f3a468e9640 1 -- 172.21.15.123:0/1814706719 >> 172.21.15.123:0/1814706719 conn(0x7f3a400b1710 msgr2=0x7f3a400b1b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:49:20.801 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:20.799+0000 7f3a468e9640 1 -- 172.21.15.123:0/1814706719 shutdown_connections 2024-09-16T15:49:20.801 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:20.799+0000 7f3a468e9640 1 -- 172.21.15.123:0/1814706719 wait complete. 2024-09-16T15:49:20.801 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:20.803+0000 7f3a468e9640 1 Processor -- start 2024-09-16T15:49:20.801 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:20.803+0000 7f3a468e9640 1 -- start start 2024-09-16T15:49:20.804 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:20.803+0000 7f3a468e9640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3a2c003a60 0x7f3a401f4500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:49:20.804 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:20.803+0000 7f3a468e9640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3a40149650 0x7f3a401f73d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:49:20.804 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:20.803+0000 7f3a468e9640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f3a40148680 0x7f3a40148b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:49:20.804 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:20.803+0000 7f3a468e9640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f3a400732b0 con 0x7f3a2c003a60 2024-09-16T15:49:20.804 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:20.803+0000 7f3a468e9640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f3a40073130 con 0x7f3a40149650 2024-09-16T15:49:20.804 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:20.803+0000 7f3a468e9640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f3a40073430 con 0x7f3a40148680 2024-09-16T15:49:20.804 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:20.803+0000 7f3a3ffff640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3a2c003a60 0x7f3a401f4500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:49:20.804 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:20.803+0000 7f3a3ffff640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3a2c003a60 0x7f3a401f4500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:52404/0 (socket says 172.21.15.123:52404) 2024-09-16T15:49:20.804 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:20.803+0000 7f3a3ffff640 1 -- 172.21.15.123:0/1691847937 learned_addr learned my addr 172.21.15.123:0/1691847937 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:49:20.804 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:20.803+0000 7f3a37fff640 1 --2- 172.21.15.123:0/1691847937 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3a40149650 0x7f3a401f73d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:49:20.804 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:20.803+0000 7f3a44e86640 1 --2- 172.21.15.123:0/1691847937 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f3a40148680 0x7f3a40148b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:49:20.805 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:20.803+0000 7f3a3ffff640 1 -- 172.21.15.123:0/1691847937 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f3a40148680 msgr2=0x7f3a40148b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:20.805 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:20.803+0000 7f3a3ffff640 1 --2- 172.21.15.123:0/1691847937 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f3a40148680 0x7f3a40148b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:20.805 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:20.803+0000 7f3a3ffff640 1 -- 172.21.15.123:0/1691847937 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3a40149650 msgr2=0x7f3a401f73d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:20.805 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:20.803+0000 7f3a3ffff640 1 --2- 172.21.15.123:0/1691847937 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3a40149650 0x7f3a401f73d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:20.805 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:20.803+0000 7f3a3ffff640 1 -- 172.21.15.123:0/1691847937 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3a400c42e0 con 0x7f3a2c003a60 2024-09-16T15:49:20.805 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:20.803+0000 7f3a3ffff640 1 --2- 172.21.15.123:0/1691847937 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3a2c003a60 0x7f3a401f4500 secure :-1 s=READY pgs=1255 cs=0 l=1 rev1=1 crypto rx=0x7f3a30009a80 tx=0x7f3a300056c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:49:20.805 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:20.803+0000 7f3a3d7fa640 1 -- 172.21.15.123:0/1691847937 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a300ba020 con 0x7f3a2c003a60 2024-09-16T15:49:20.805 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:20.803+0000 7f3a3d7fa640 1 -- 172.21.15.123:0/1691847937 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f3a300acaa0 con 0x7f3a2c003a60 2024-09-16T15:49:20.805 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:20.803+0000 7f3a468e9640 1 -- 172.21.15.123:0/1691847937 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3a400a9850 con 0x7f3a2c003a60 2024-09-16T15:49:20.805 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:20.803+0000 7f3a3d7fa640 1 -- 172.21.15.123:0/1691847937 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a300b1040 con 0x7f3a2c003a60 2024-09-16T15:49:20.805 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:20.803+0000 7f3a468e9640 1 -- 172.21.15.123:0/1691847937 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3a400c4860 con 0x7f3a2c003a60 2024-09-16T15:49:20.805 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:20.803+0000 7f3a468e9640 1 -- 172.21.15.123:0/1691847937 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3a400ce090 con 0x7f3a2c003a60 2024-09-16T15:49:20.806 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:20.803+0000 7f3a3d7fa640 1 -- 172.21.15.123:0/1691847937 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 19) ==== 97488+0+0 (secure 0 0 0) 0x7f3a300bd020 con 0x7f3a2c003a60 2024-09-16T15:49:20.806 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:20.807+0000 7f3a3d7fa640 1 --2- 172.21.15.123:0/1691847937 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f3a20073370 0x7f3a20075830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:49:20.806 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:20.807+0000 7f3a3d7fa640 1 -- 172.21.15.123:0/1691847937 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(118..118 src has 1..118) ==== 5737+0+0 (secure 0 0 0) 0x7f3a3012f520 con 0x7f3a2c003a60 2024-09-16T15:49:20.806 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:20.807+0000 7f3a37fff640 1 --2- 172.21.15.123:0/1691847937 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f3a20073370 0x7f3a20075830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:49:20.806 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:20.807+0000 7f3a37fff640 1 --2- 172.21.15.123:0/1691847937 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f3a20073370 0x7f3a20075830 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f3a40075760 tx=0x7f3a28009040 comp rx=0 tx=0).ready entity=mgr.4150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:49:20.809 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:20.811+0000 7f3a3d7fa640 1 -- 172.21.15.123:0/1691847937 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f3a300c4030 con 0x7f3a2c003a60 2024-09-16T15:49:21.012 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:21.011+0000 7f3a468e9640 1 -- 172.21.15.123:0/1691847937 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) -- 0x7f3a400697b0 con 0x7f3a2c003a60 2024-09-16T15:49:22.028 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:22.027+0000 7f3a3d7fa640 1 -- 172.21.15.123:0/1691847937 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v75) ==== 118+0+0 (secure 0 0 0) 0x7f3a30016f60 con 0x7f3a2c003a60 2024-09-16T15:49:22.033 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:22.031+0000 7f3a468e9640 1 -- 172.21.15.123:0/1691847937 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f3a20073370 msgr2=0x7f3a20075830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:22.033 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:22.031+0000 7f3a468e9640 1 --2- 172.21.15.123:0/1691847937 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f3a20073370 0x7f3a20075830 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f3a40075760 tx=0x7f3a28009040 comp rx=0 tx=0).stop 2024-09-16T15:49:22.033 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:22.031+0000 7f3a468e9640 1 -- 172.21.15.123:0/1691847937 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3a2c003a60 msgr2=0x7f3a401f4500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:22.033 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:22.031+0000 7f3a468e9640 1 --2- 172.21.15.123:0/1691847937 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3a2c003a60 0x7f3a401f4500 secure :-1 s=READY pgs=1255 cs=0 l=1 rev1=1 crypto rx=0x7f3a30009a80 tx=0x7f3a300056c0 comp rx=0 tx=0).stop 2024-09-16T15:49:22.033 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:22.031+0000 7f3a468e9640 1 -- 172.21.15.123:0/1691847937 shutdown_connections 2024-09-16T15:49:22.033 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:22.031+0000 7f3a468e9640 1 --2- 172.21.15.123:0/1691847937 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f3a20073370 0x7f3a20075830 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:22.033 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:22.031+0000 7f3a468e9640 1 --2- 172.21.15.123:0/1691847937 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f3a40148680 0x7f3a40148b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:22.033 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:22.031+0000 7f3a468e9640 1 --2- 172.21.15.123:0/1691847937 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3a40149650 0x7f3a401f73d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:22.033 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:22.031+0000 7f3a468e9640 1 --2- 172.21.15.123:0/1691847937 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3a2c003a60 0x7f3a401f4500 unknown :-1 s=CLOSED pgs=1255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:22.033 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:22.031+0000 7f3a468e9640 1 -- 172.21.15.123:0/1691847937 >> 172.21.15.123:0/1691847937 conn(0x7f3a400b1710 msgr2=0x7f3a400c95e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:49:22.033 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:22.031+0000 7f3a468e9640 1 -- 172.21.15.123:0/1691847937 shutdown_connections 2024-09-16T15:49:22.033 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:22.031+0000 7f3a468e9640 1 -- 172.21.15.123:0/1691847937 wait complete. 2024-09-16T15:49:22.054 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-09-16T15:49:22.054 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-09-16T15:49:22.186 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:22.187+0000 7f09f6aa9640 1 -- 172.21.15.123:0/3286015822 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f09f015a9d0 msgr2=0x7f09f006ebe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:22.186 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:22.187+0000 7f09f6aa9640 1 --2- 172.21.15.123:0/3286015822 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f09f015a9d0 0x7f09f006ebe0 secure :-1 s=READY pgs=1257 cs=0 l=1 rev1=1 crypto rx=0x7f09f00af630 tx=0x7f09e001c740 comp rx=0 tx=0).stop 2024-09-16T15:49:22.186 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:22.187+0000 7f09f6aa9640 1 -- 172.21.15.123:0/3286015822 shutdown_connections 2024-09-16T15:49:22.186 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:22.187+0000 7f09f6aa9640 1 --2- 172.21.15.123:0/3286015822 >> v2:172.21.15.184:3300/0 conn(0x7f09f00ccad0 0x7f09f015e9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:22.186 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:22.187+0000 7f09f6aa9640 1 --2- 172.21.15.123:0/3286015822 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f09f015a9d0 0x7f09f006ebe0 unknown :-1 s=CLOSED pgs=1257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:22.186 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:22.187+0000 7f09f6aa9640 1 -- 172.21.15.123:0/3286015822 >> 172.21.15.123:0/3286015822 conn(0x7f09f00b1710 msgr2=0x7f09f00b1b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:49:22.187 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:22.187+0000 7f09f6aa9640 1 -- 172.21.15.123:0/3286015822 shutdown_connections 2024-09-16T15:49:22.187 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:22.187+0000 7f09f6aa9640 1 -- 172.21.15.123:0/3286015822 wait complete. 2024-09-16T15:49:22.187 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:22.187+0000 7f09f6aa9640 1 Processor -- start 2024-09-16T15:49:22.187 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:22.187+0000 7f09f6aa9640 1 -- start start 2024-09-16T15:49:22.189 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:22.187+0000 7f09f6aa9640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f09f0072c70 0x7f09f0156ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:49:22.189 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:22.187+0000 7f09f6aa9640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f09f00ccad0 0x7f09f00c9d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:49:22.189 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:22.187+0000 7f09f6aa9640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f09f015a9d0 0x7f09f01f57a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:49:22.189 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:22.187+0000 7f09f6aa9640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f09f0167e90 con 0x7f09f0072c70 2024-09-16T15:49:22.189 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:22.187+0000 7f09f6aa9640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f09f0167d10 con 0x7f09f00ccad0 2024-09-16T15:49:22.189 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:22.187+0000 7f09f6aa9640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f09f0168010 con 0x7f09f015a9d0 2024-09-16T15:49:22.190 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:22.187+0000 7f09f4845640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f09f0072c70 0x7f09f0156ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:49:22.190 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:22.187+0000 7f09f4845640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f09f0072c70 0x7f09f0156ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:52424/0 (socket says 172.21.15.123:52424) 2024-09-16T15:49:22.190 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:22.187+0000 7f09f4845640 1 -- 172.21.15.123:0/3406353869 learned_addr learned my addr 172.21.15.123:0/3406353869 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:49:22.190 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:22.187+0000 7f09effff640 1 --2- 172.21.15.123:0/3406353869 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f09f00ccad0 0x7f09f00c9d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:49:22.190 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:22.187+0000 7f09f4845640 1 -- 172.21.15.123:0/3406353869 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f09f015a9d0 msgr2=0x7f09f01f57a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:22.190 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:22.187+0000 7f09f4845640 1 --2- 172.21.15.123:0/3406353869 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f09f015a9d0 0x7f09f01f57a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:22.190 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:22.187+0000 7f09f4845640 1 -- 172.21.15.123:0/3406353869 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f09f00ccad0 msgr2=0x7f09f00c9d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:22.190 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:22.187+0000 7f09f4845640 1 --2- 172.21.15.123:0/3406353869 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f09f00ccad0 0x7f09f00c9d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:22.190 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:22.187+0000 7f09f4845640 1 -- 172.21.15.123:0/3406353869 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f09f01541c0 con 0x7f09f0072c70 2024-09-16T15:49:22.190 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:22.187+0000 7f09effff640 1 --2- 172.21.15.123:0/3406353869 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f09f00ccad0 0x7f09f00c9d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T15:49:22.190 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:22.187+0000 7f09f4845640 1 --2- 172.21.15.123:0/3406353869 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f09f0072c70 0x7f09f0156ea0 secure :-1 s=READY pgs=1258 cs=0 l=1 rev1=1 crypto rx=0x7f09e001cc20 tx=0x7f09e00a56a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:49:22.190 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:22.191+0000 7f09ed7fa640 1 -- 172.21.15.123:0/3406353869 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09e00b2040 con 0x7f09f0072c70 2024-09-16T15:49:22.190 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:22.191+0000 7f09ed7fa640 1 -- 172.21.15.123:0/3406353869 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f09e0004230 con 0x7f09f0072c70 2024-09-16T15:49:22.191 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:22.191+0000 7f09f6aa9640 1 -- 172.21.15.123:0/3406353869 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f09f00c6c10 con 0x7f09f0072c70 2024-09-16T15:49:22.191 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:22.191+0000 7f09ed7fa640 1 -- 172.21.15.123:0/3406353869 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09e0003600 con 0x7f09f0072c70 2024-09-16T15:49:22.191 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:22.191+0000 7f09f6aa9640 1 -- 172.21.15.123:0/3406353869 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f09f00c71d0 con 0x7f09f0072c70 2024-09-16T15:49:22.191 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:22.191+0000 7f09f6aa9640 1 -- 172.21.15.123:0/3406353869 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f09f015e9c0 con 0x7f09f0072c70 2024-09-16T15:49:22.191 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:22.191+0000 7f09ed7fa640 1 -- 172.21.15.123:0/3406353869 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 19) ==== 97488+0+0 (secure 0 0 0) 0x7f09e00bd030 con 0x7f09f0072c70 2024-09-16T15:49:22.192 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:22.191+0000 7f09ed7fa640 1 --2- 172.21.15.123:0/3406353869 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f09dc073340 0x7f09dc075800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:49:22.192 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:22.191+0000 7f09effff640 1 --2- 172.21.15.123:0/3406353869 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f09dc073340 0x7f09dc075800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:49:22.192 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:22.191+0000 7f09ed7fa640 1 -- 172.21.15.123:0/3406353869 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(119..119 src has 1..119) ==== 5737+0+0 (secure 0 0 0) 0x7f09e012e340 con 0x7f09f0072c70 2024-09-16T15:49:22.192 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:22.191+0000 7f09effff640 1 --2- 172.21.15.123:0/3406353869 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f09dc073340 0x7f09dc075800 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f09f015a9a0 tx=0x7f09e4009040 comp rx=0 tx=0).ready entity=mgr.4150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:49:22.195 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:22.195+0000 7f09ed7fa640 1 -- 172.21.15.123:0/3406353869 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f09e00c4060 con 0x7f09f0072c70 2024-09-16T15:49:22.406 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:22.407+0000 7f09f6aa9640 1 -- 172.21.15.123:0/3406353869 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) -- 0x7f09f00ca9a0 con 0x7f09f0072c70 2024-09-16T15:49:23.039 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:23.039+0000 7f09ed7fa640 1 -- 172.21.15.123:0/3406353869 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v76) ==== 122+0+0 (secure 0 0 0) 0x7f09e00163f0 con 0x7f09f0072c70 2024-09-16T15:49:23.044 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:23.043+0000 7f09f6aa9640 1 -- 172.21.15.123:0/3406353869 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f09dc073340 msgr2=0x7f09dc075800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:23.044 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:23.043+0000 7f09f6aa9640 1 --2- 172.21.15.123:0/3406353869 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f09dc073340 0x7f09dc075800 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f09f015a9a0 tx=0x7f09e4009040 comp rx=0 tx=0).stop 2024-09-16T15:49:23.045 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:23.043+0000 7f09f6aa9640 1 -- 172.21.15.123:0/3406353869 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f09f0072c70 msgr2=0x7f09f0156ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:23.045 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:23.043+0000 7f09f6aa9640 1 --2- 172.21.15.123:0/3406353869 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f09f0072c70 0x7f09f0156ea0 secure :-1 s=READY pgs=1258 cs=0 l=1 rev1=1 crypto rx=0x7f09e001cc20 tx=0x7f09e00a56a0 comp rx=0 tx=0).stop 2024-09-16T15:49:23.045 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:23.043+0000 7f09f6aa9640 1 -- 172.21.15.123:0/3406353869 shutdown_connections 2024-09-16T15:49:23.045 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:23.043+0000 7f09f6aa9640 1 --2- 172.21.15.123:0/3406353869 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f09dc073340 0x7f09dc075800 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:23.045 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:23.043+0000 7f09f6aa9640 1 --2- 172.21.15.123:0/3406353869 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f09f015a9d0 0x7f09f01f57a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:23.045 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:23.043+0000 7f09f6aa9640 1 --2- 172.21.15.123:0/3406353869 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f09f00ccad0 0x7f09f00c9d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:23.045 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:23.043+0000 7f09f6aa9640 1 --2- 172.21.15.123:0/3406353869 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f09f0072c70 0x7f09f0156ea0 unknown :-1 s=CLOSED pgs=1258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:23.045 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:23.043+0000 7f09f6aa9640 1 -- 172.21.15.123:0/3406353869 >> 172.21.15.123:0/3406353869 conn(0x7f09f00b1710 msgr2=0x7f09f00d04e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:49:23.045 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:23.043+0000 7f09f6aa9640 1 -- 172.21.15.123:0/3406353869 shutdown_connections 2024-09-16T15:49:23.045 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:23.043+0000 7f09f6aa9640 1 -- 172.21.15.123:0/3406353869 wait complete. 2024-09-16T15:49:23.068 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-09-16T15:49:23.069 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-16T15:49:23.198 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:23.199+0000 7f1287f4c640 1 -- 172.21.15.123:0/31779427 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f126c004810 msgr2=0x7f126c004c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:23.198 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:23.199+0000 7f1287f4c640 1 --2- 172.21.15.123:0/31779427 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f126c004810 0x7f126c004c90 secure :-1 s=READY pgs=1260 cs=0 l=1 rev1=1 crypto rx=0x7f1270006fa0 tx=0x7f127001cdd0 comp rx=0 tx=0).stop 2024-09-16T15:49:23.198 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:23.199+0000 7f1287f4c640 1 -- 172.21.15.123:0/31779427 shutdown_connections 2024-09-16T15:49:23.198 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:23.199+0000 7f1287f4c640 1 --2- 172.21.15.123:0/31779427 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f126c004810 0x7f126c004c90 unknown :-1 s=CLOSED pgs=1260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:23.198 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:23.199+0000 7f1287f4c640 1 -- 172.21.15.123:0/31779427 >> 172.21.15.123:0/31779427 conn(0x7f12800d3ea0 msgr2=0x7f12800d42b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:49:23.198 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:23.199+0000 7f1287f4c640 1 -- 172.21.15.123:0/31779427 shutdown_connections 2024-09-16T15:49:23.198 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:23.199+0000 7f1287f4c640 1 -- 172.21.15.123:0/31779427 wait complete. 2024-09-16T15:49:23.199 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:23.199+0000 7f1287f4c640 1 Processor -- start 2024-09-16T15:49:23.199 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:23.199+0000 7f1287f4c640 1 -- start start 2024-09-16T15:49:23.201 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:23.199+0000 7f1287f4c640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f126c004810 0x7f12801601c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:49:23.201 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:23.199+0000 7f1287f4c640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f12800b8f30 0x7f12800b9350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:49:23.201 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:23.199+0000 7f1287f4c640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f12800b9890 0x7f12801f7800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:49:23.201 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:23.199+0000 7f1287f4c640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f12800b1c20 con 0x7f12800b9890 2024-09-16T15:49:23.201 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:23.199+0000 7f1287f4c640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f12800b1aa0 con 0x7f126c004810 2024-09-16T15:49:23.201 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:23.199+0000 7f1287f4c640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f12800d1da0 con 0x7f12800b8f30 2024-09-16T15:49:23.201 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:23.199+0000 7f12864e9640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f12800b9890 0x7f12801f7800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:49:23.201 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:23.199+0000 7f12864e9640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f12800b9890 0x7f12801f7800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:35378/0 (socket says 172.21.15.123:35378) 2024-09-16T15:49:23.201 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:23.199+0000 7f12864e9640 1 -- 172.21.15.123:0/761582210 learned_addr learned my addr 172.21.15.123:0/761582210 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:49:23.202 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:23.199+0000 7f1285ce8640 1 --2- 172.21.15.123:0/761582210 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f126c004810 0x7f12801601c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:49:23.202 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:23.199+0000 7f12854e7640 1 --2- 172.21.15.123:0/761582210 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f12800b8f30 0x7f12800b9350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:49:23.202 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:23.199+0000 7f12864e9640 1 -- 172.21.15.123:0/761582210 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f12800b8f30 msgr2=0x7f12800b9350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:23.202 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:23.199+0000 7f12864e9640 1 --2- 172.21.15.123:0/761582210 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f12800b8f30 0x7f12800b9350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:23.202 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:23.199+0000 7f12864e9640 1 -- 172.21.15.123:0/761582210 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f126c004810 msgr2=0x7f12801601c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:23.202 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:23.199+0000 7f12864e9640 1 --2- 172.21.15.123:0/761582210 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f126c004810 0x7f12801601c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:23.202 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:23.199+0000 7f12864e9640 1 -- 172.21.15.123:0/761582210 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f12801fae30 con 0x7f12800b9890 2024-09-16T15:49:23.202 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:23.199+0000 7f12864e9640 1 --2- 172.21.15.123:0/761582210 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f12800b9890 0x7f12801f7800 secure :-1 s=READY pgs=1261 cs=0 l=1 rev1=1 crypto rx=0x7f128006f900 tx=0x7f127800db70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:49:23.202 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:23.199+0000 7f12767fc640 1 -- 172.21.15.123:0/761582210 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1278003b80 con 0x7f12800b9890 2024-09-16T15:49:23.202 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:23.199+0000 7f12767fc640 1 -- 172.21.15.123:0/761582210 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f1278003e10 con 0x7f12800b9890 2024-09-16T15:49:23.202 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:23.199+0000 7f1287f4c640 1 -- 172.21.15.123:0/761582210 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f12800b9d20 con 0x7f12800b9890 2024-09-16T15:49:23.202 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:23.199+0000 7f12767fc640 1 -- 172.21.15.123:0/761582210 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f127800ba60 con 0x7f12800b9890 2024-09-16T15:49:23.202 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:23.203+0000 7f1287f4c640 1 -- 172.21.15.123:0/761582210 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f12801fb3b0 con 0x7f12800b9890 2024-09-16T15:49:23.203 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:23.203+0000 7f1287f4c640 1 -- 172.21.15.123:0/761582210 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f12800d2300 con 0x7f12800b9890 2024-09-16T15:49:23.203 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:23.203+0000 7f12767fc640 1 -- 172.21.15.123:0/761582210 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 19) ==== 97488+0+0 (secure 0 0 0) 0x7f127800bc00 con 0x7f12800b9890 2024-09-16T15:49:23.203 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:23.203+0000 7f12767fc640 1 --2- 172.21.15.123:0/761582210 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f1264073340 0x7f1264075800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:49:23.204 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:23.203+0000 7f1285ce8640 1 --2- 172.21.15.123:0/761582210 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f1264073340 0x7f1264075800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:49:23.204 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:23.203+0000 7f12767fc640 1 -- 172.21.15.123:0/761582210 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(119..119 src has 1..119) ==== 5737+0+0 (secure 0 0 0) 0x7f1278094990 con 0x7f12800b9890 2024-09-16T15:49:23.204 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:23.203+0000 7f1285ce8640 1 --2- 172.21.15.123:0/761582210 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f1264073340 0x7f1264075800 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f127c005fd0 tx=0x7f127c005f40 comp rx=0 tx=0).ready entity=mgr.4150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:49:23.207 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:23.207+0000 7f12767fc640 1 -- 172.21.15.123:0/761582210 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f1278006070 con 0x7f12800b9890 2024-09-16T15:49:23.413 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:23.415+0000 7f1287f4c640 1 -- 172.21.15.123:0/761582210 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f12800ce2d0 con 0x7f12800b9890 2024-09-16T15:49:23.414 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:23.415+0000 7f12767fc640 1 -- 172.21.15.123:0/761582210 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 76 v76) ==== 94+0+5234 (secure 0 0 0) 0x7f127805f5e0 con 0x7f12800b9890 2024-09-16T15:49:23.414 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-09-16T15:49:23.414 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":76,"btime":"2024-09-16T15:49:23:031064+0000","default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6079,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6833/3932980815","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":3932980815},{"type":"v1","addr":"172.21.15.184:6833","nonce":3932980815}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":72},{"gid":6082,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6835/1809728014","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":1809728014},{"type":"v1","addr":"172.21.15.184:6835","nonce":1809728014}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":72},{"gid":7551,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6834/892095590","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6832","nonce":892095590},{"type":"v1","addr":"172.21.15.123:6834","nonce":892095590}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":72}],"filesystems":[{"mdsmap":{"epoch":76,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-16T15:49:19.361823+0000","modified":"2024-09-16T15:49:22.409036+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7554},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7554":{"gid":7554,"name":"a","rank":0,"incarnation":73,"state":"up:active","state_seq":3,"addr":"172.21.15.123:6835/2110793616","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6833","nonce":2110793616},{"type":"v1","addr":"172.21.15.123:6835","nonce":2110793616}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7554,"qdb_cluster":[7554]},"id":9}]} 2024-09-16T15:49:23.414 INFO:teuthology.orchestra.run.smithi123.stderr:dumped fsmap epoch 76 2024-09-16T15:49:23.416 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:23.415+0000 7f1287f4c640 1 -- 172.21.15.123:0/761582210 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f1264073340 msgr2=0x7f1264075800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:23.416 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:23.415+0000 7f1287f4c640 1 --2- 172.21.15.123:0/761582210 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f1264073340 0x7f1264075800 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f127c005fd0 tx=0x7f127c005f40 comp rx=0 tx=0).stop 2024-09-16T15:49:23.417 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:23.415+0000 7f1287f4c640 1 -- 172.21.15.123:0/761582210 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f12800b9890 msgr2=0x7f12801f7800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:23.417 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:23.415+0000 7f1287f4c640 1 --2- 172.21.15.123:0/761582210 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f12800b9890 0x7f12801f7800 secure :-1 s=READY pgs=1261 cs=0 l=1 rev1=1 crypto rx=0x7f128006f900 tx=0x7f127800db70 comp rx=0 tx=0).stop 2024-09-16T15:49:23.417 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:23.415+0000 7f1287f4c640 1 -- 172.21.15.123:0/761582210 shutdown_connections 2024-09-16T15:49:23.417 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:23.415+0000 7f1287f4c640 1 --2- 172.21.15.123:0/761582210 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f1264073340 0x7f1264075800 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:23.417 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:23.415+0000 7f1287f4c640 1 --2- 172.21.15.123:0/761582210 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f12800b9890 0x7f12801f7800 unknown :-1 s=CLOSED pgs=1261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:23.417 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:23.415+0000 7f1287f4c640 1 --2- 172.21.15.123:0/761582210 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f12800b8f30 0x7f12800b9350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:23.417 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:23.415+0000 7f1287f4c640 1 --2- 172.21.15.123:0/761582210 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f126c004810 0x7f12801601c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:23.417 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:23.415+0000 7f1287f4c640 1 -- 172.21.15.123:0/761582210 >> 172.21.15.123:0/761582210 conn(0x7f12800d3ea0 msgr2=0x7f128006dd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:49:23.417 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:23.415+0000 7f1287f4c640 1 -- 172.21.15.123:0/761582210 shutdown_connections 2024-09-16T15:49:23.417 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:23.415+0000 7f1287f4c640 1 -- 172.21.15.123:0/761582210 wait complete. 2024-09-16T15:49:23.432 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-16T15:49:23.558 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:23.559+0000 7f4648039640 1 -- 172.21.15.123:0/647978924 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f462c0052f0 msgr2=0x7f462c00b9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:23.558 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:23.559+0000 7f4648039640 1 --2- 172.21.15.123:0/647978924 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f462c0052f0 0x7f462c00b9d0 secure :-1 s=READY pgs=1263 cs=0 l=1 rev1=1 crypto rx=0x7f4638009e30 tx=0x7f463801d1b0 comp rx=0 tx=0).stop 2024-09-16T15:49:23.558 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:23.559+0000 7f4648039640 1 -- 172.21.15.123:0/647978924 shutdown_connections 2024-09-16T15:49:23.558 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:23.559+0000 7f4648039640 1 --2- 172.21.15.123:0/647978924 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f462c0052f0 0x7f462c00b9d0 unknown :-1 s=CLOSED pgs=1263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:23.558 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:23.559+0000 7f4648039640 1 -- 172.21.15.123:0/647978924 >> 172.21.15.123:0/647978924 conn(0x7f46400b2610 msgr2=0x7f46400b29f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:49:23.558 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:23.559+0000 7f4648039640 1 -- 172.21.15.123:0/647978924 shutdown_connections 2024-09-16T15:49:23.558 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:23.559+0000 7f4648039640 1 -- 172.21.15.123:0/647978924 wait complete. 2024-09-16T15:49:23.559 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:23.559+0000 7f4648039640 1 Processor -- start 2024-09-16T15:49:23.559 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:23.559+0000 7f4648039640 1 -- start start 2024-09-16T15:49:23.561 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:23.559+0000 7f4648039640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f462c0052f0 0x7f46401559b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:49:23.561 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:23.559+0000 7f4648039640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f46401f2b40 0x7f4640151e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:49:23.561 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:23.559+0000 7f4648039640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f464015e500 0x7f464015e980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:49:23.561 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:23.559+0000 7f4648039640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f464006dab0 con 0x7f46401f2b40 2024-09-16T15:49:23.561 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:23.559+0000 7f4648039640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f464006d930 con 0x7f462c0052f0 2024-09-16T15:49:23.561 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:23.559+0000 7f4648039640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f4640071d00 con 0x7f464015e500 2024-09-16T15:49:23.561 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:23.559+0000 7f46455d4640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f46401f2b40 0x7f4640151e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:49:23.561 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:23.559+0000 7f46455d4640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f46401f2b40 0x7f4640151e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:35392/0 (socket says 172.21.15.123:35392) 2024-09-16T15:49:23.562 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:23.559+0000 7f46455d4640 1 -- 172.21.15.123:0/3912160641 learned_addr learned my addr 172.21.15.123:0/3912160641 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:49:23.562 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:23.559+0000 7f46455d4640 1 -- 172.21.15.123:0/3912160641 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f464015e500 msgr2=0x7f464015e980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:23.562 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:23.559+0000 7f46455d4640 1 --2- 172.21.15.123:0/3912160641 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f464015e500 0x7f464015e980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:23.562 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:23.559+0000 7f46455d4640 1 -- 172.21.15.123:0/3912160641 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f462c0052f0 msgr2=0x7f46401559b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:23.562 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:23.559+0000 7f46455d4640 1 --2- 172.21.15.123:0/3912160641 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f462c0052f0 0x7f46401559b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:23.562 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:23.559+0000 7f46455d4640 1 -- 172.21.15.123:0/3912160641 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f46400c7d20 con 0x7f46401f2b40 2024-09-16T15:49:23.562 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:23.559+0000 7f46455d4640 1 --2- 172.21.15.123:0/3912160641 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f46401f2b40 0x7f4640151e20 secure :-1 s=READY pgs=1264 cs=0 l=1 rev1=1 crypto rx=0x7f463000d500 tx=0x7f463000d9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:49:23.562 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:23.559+0000 7f46367fc640 1 -- 172.21.15.123:0/3912160641 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4630005030 con 0x7f46401f2b40 2024-09-16T15:49:23.562 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:23.559+0000 7f46367fc640 1 -- 172.21.15.123:0/3912160641 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f46300051d0 con 0x7f46401f2b40 2024-09-16T15:49:23.562 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:23.559+0000 7f4648039640 1 -- 172.21.15.123:0/3912160641 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f46400c8000 con 0x7f46401f2b40 2024-09-16T15:49:23.562 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:23.559+0000 7f46367fc640 1 -- 172.21.15.123:0/3912160641 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4630012630 con 0x7f46401f2b40 2024-09-16T15:49:23.562 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:23.559+0000 7f4648039640 1 -- 172.21.15.123:0/3912160641 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f46400c85c0 con 0x7f46401f2b40 2024-09-16T15:49:23.563 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:23.563+0000 7f4648039640 1 -- 172.21.15.123:0/3912160641 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f464006cc40 con 0x7f46401f2b40 2024-09-16T15:49:23.563 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:23.563+0000 7f46367fc640 1 -- 172.21.15.123:0/3912160641 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 19) ==== 97488+0+0 (secure 0 0 0) 0x7f46300045f0 con 0x7f46401f2b40 2024-09-16T15:49:23.563 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:23.563+0000 7f46367fc640 1 --2- 172.21.15.123:0/3912160641 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f4624073340 0x7f4624075800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:49:23.563 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:23.563+0000 7f46367fc640 1 -- 172.21.15.123:0/3912160641 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(119..119 src has 1..119) ==== 5737+0+0 (secure 0 0 0) 0x7f46300947e0 con 0x7f46401f2b40 2024-09-16T15:49:23.564 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:23.563+0000 7f4645dd5640 1 --2- 172.21.15.123:0/3912160641 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f4624073340 0x7f4624075800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:49:23.564 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:23.563+0000 7f4645dd5640 1 --2- 172.21.15.123:0/3912160641 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f4624073340 0x7f4624075800 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f463c001250 tx=0x7f463c001140 comp rx=0 tx=0).ready entity=mgr.4150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:49:23.566 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:23.567+0000 7f46367fc640 1 -- 172.21.15.123:0/3912160641 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f463005f310 con 0x7f46401f2b40 2024-09-16T15:49:23.740 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:23.739+0000 7f4648039640 1 -- 172.21.15.123:0/3912160641 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f46400afb40 con 0x7f46401f2b40 2024-09-16T15:49:23.743 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:23.743+0000 7f46367fc640 1 -- 172.21.15.123:0/3912160641 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v119) ==== 74+0+16843 (secure 0 0 0) 0x7f463005ec10 con 0x7f46401f2b40 2024-09-16T15:49:23.743 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-09-16T15:49:23.743 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":119,"fsid":"56077f89-18c8-486f-ae7e-eecc0bb7822c","created":"2024-09-16T15:42:26.364892+0000","modified":"2024-09-16T15:49:20.991587+0000","last_up_change":"2024-09-16T15:42:38.707234+0000","last_in_change":"2024-09-16T15:42:29.423662+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":19,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-16T15:42:39.287764+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":18,"pool_name":"cephfs_metadata","create_time":"2024-09-16T15:49:17.014388+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"118","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":19,"pool_name":"cephfs_data","create_time":"2024-09-16T15:49:18.311134+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"119","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9e7c1955-a607-4671-9be7-f2ef760ceeaa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":117,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6800","nonce":3882044969},{"type":"v1","addr":"172.21.15.123:6801","nonce":3882044969}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6802","nonce":3882044969},{"type":"v1","addr":"172.21.15.123:6803","nonce":3882044969}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6806","nonce":3882044969},{"type":"v1","addr":"172.21.15.123:6807","nonce":3882044969}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6804","nonce":3882044969},{"type":"v1","addr":"172.21.15.123:6805","nonce":3882044969}]},"public_addr":"172.21.15.123:6801/3882044969","cluster_addr":"172.21.15.123:6803/3882044969","heartbeat_back_addr":"172.21.15.123:6807/3882044969","heartbeat_front_addr":"172.21.15.123:6805/3882044969","state":["exists","up"]},{"osd":1,"uuid":"c540cfde-0977-4742-9b17-fd634668e010","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":117,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6816","nonce":4037001850},{"type":"v1","addr":"172.21.15.123:6817","nonce":4037001850}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6818","nonce":4037001850},{"type":"v1","addr":"172.21.15.123:6819","nonce":4037001850}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6822","nonce":4037001850},{"type":"v1","addr":"172.21.15.123:6823","nonce":4037001850}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6820","nonce":4037001850},{"type":"v1","addr":"172.21.15.123:6821","nonce":4037001850}]},"public_addr":"172.21.15.123:6817/4037001850","cluster_addr":"172.21.15.123:6819/4037001850","heartbeat_back_addr":"172.21.15.123:6823/4037001850","heartbeat_front_addr":"172.21.15.123:6821/4037001850","state":["exists","up"]},{"osd":2,"uuid":"9864a496-f169-486b-95c4-1d59162793a7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":117,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6824","nonce":1332143088},{"type":"v1","addr":"172.21.15.123:6825","nonce":1332143088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6826","nonce":1332143088},{"type":"v1","addr":"172.21.15.123:6827","nonce":1332143088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6830","nonce":1332143088},{"type":"v1","addr":"172.21.15.123:6831","nonce":1332143088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6828","nonce":1332143088},{"type":"v1","addr":"172.21.15.123:6829","nonce":1332143088}]},"public_addr":"172.21.15.123:6825/1332143088","cluster_addr":"172.21.15.123:6827/1332143088","heartbeat_back_addr":"172.21.15.123:6831/1332143088","heartbeat_front_addr":"172.21.15.123:6829/1332143088","state":["exists","up"]},{"osd":3,"uuid":"c6d1f817-f074-4325-95e1-578eba13edda","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":117,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6808","nonce":988586998},{"type":"v1","addr":"172.21.15.123:6809","nonce":988586998}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6810","nonce":988586998},{"type":"v1","addr":"172.21.15.123:6811","nonce":988586998}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6814","nonce":988586998},{"type":"v1","addr":"172.21.15.123:6815","nonce":988586998}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6812","nonce":988586998},{"type":"v1","addr":"172.21.15.123:6813","nonce":988586998}]},"public_addr":"172.21.15.123:6809/988586998","cluster_addr":"172.21.15.123:6811/988586998","heartbeat_back_addr":"172.21.15.123:6815/988586998","heartbeat_front_addr":"172.21.15.123:6813/988586998","state":["exists","up"]},{"osd":4,"uuid":"200a8df4-50ef-40a5-b48a-58981269c086","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":117,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6816","nonce":3387813890},{"type":"v1","addr":"172.21.15.184:6817","nonce":3387813890}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6818","nonce":3387813890},{"type":"v1","addr":"172.21.15.184:6819","nonce":3387813890}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6822","nonce":3387813890},{"type":"v1","addr":"172.21.15.184:6823","nonce":3387813890}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6820","nonce":3387813890},{"type":"v1","addr":"172.21.15.184:6821","nonce":3387813890}]},"public_addr":"172.21.15.184:6817/3387813890","cluster_addr":"172.21.15.184:6819/3387813890","heartbeat_back_addr":"172.21.15.184:6823/3387813890","heartbeat_front_addr":"172.21.15.184:6821/3387813890","state":["exists","up"]},{"osd":5,"uuid":"32d5a900-da3d-412c-bc19-13861a2368d8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":117,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6808","nonce":2680916017},{"type":"v1","addr":"172.21.15.184:6809","nonce":2680916017}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6810","nonce":2680916017},{"type":"v1","addr":"172.21.15.184:6811","nonce":2680916017}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6814","nonce":2680916017},{"type":"v1","addr":"172.21.15.184:6815","nonce":2680916017}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6812","nonce":2680916017},{"type":"v1","addr":"172.21.15.184:6813","nonce":2680916017}]},"public_addr":"172.21.15.184:6809/2680916017","cluster_addr":"172.21.15.184:6811/2680916017","heartbeat_back_addr":"172.21.15.184:6815/2680916017","heartbeat_front_addr":"172.21.15.184:6813/2680916017","state":["exists","up"]},{"osd":6,"uuid":"cde6f39e-aa1b-4260-ac19-774ee6aaca72","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":117,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6824","nonce":2183120457},{"type":"v1","addr":"172.21.15.184:6825","nonce":2183120457}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6826","nonce":2183120457},{"type":"v1","addr":"172.21.15.184:6827","nonce":2183120457}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6830","nonce":2183120457},{"type":"v1","addr":"172.21.15.184:6831","nonce":2183120457}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6828","nonce":2183120457},{"type":"v1","addr":"172.21.15.184:6829","nonce":2183120457}]},"public_addr":"172.21.15.184:6825/2183120457","cluster_addr":"172.21.15.184:6827/2183120457","heartbeat_back_addr":"172.21.15.184:6831/2183120457","heartbeat_front_addr":"172.21.15.184:6829/2183120457","state":["exists","up"]},{"osd":7,"uuid":"4f381778-eee1-47c6-804f-df0e6b33e313","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":117,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6800","nonce":4102957285},{"type":"v1","addr":"172.21.15.184:6801","nonce":4102957285}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6802","nonce":4102957285},{"type":"v1","addr":"172.21.15.184:6803","nonce":4102957285}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6806","nonce":4102957285},{"type":"v1","addr":"172.21.15.184:6807","nonce":4102957285}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6804","nonce":4102957285},{"type":"v1","addr":"172.21.15.184:6805","nonce":4102957285}]},"public_addr":"172.21.15.184:6801/4102957285","cluster_addr":"172.21.15.184:6803/4102957285","heartbeat_back_addr":"172.21.15.184:6807/4102957285","heartbeat_front_addr":"172.21.15.184:6805/4102957285","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:35.972210+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.222007+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.198617+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.385663+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.358935+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.156066+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.752973+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:35.682846+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":17,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T15:49:23.745 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:23.747+0000 7f4648039640 1 -- 172.21.15.123:0/3912160641 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f4624073340 msgr2=0x7f4624075800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:23.745 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:23.747+0000 7f4648039640 1 --2- 172.21.15.123:0/3912160641 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f4624073340 0x7f4624075800 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f463c001250 tx=0x7f463c001140 comp rx=0 tx=0).stop 2024-09-16T15:49:23.745 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:23.747+0000 7f4648039640 1 -- 172.21.15.123:0/3912160641 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f46401f2b40 msgr2=0x7f4640151e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:23.745 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:23.747+0000 7f4648039640 1 --2- 172.21.15.123:0/3912160641 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f46401f2b40 0x7f4640151e20 secure :-1 s=READY pgs=1264 cs=0 l=1 rev1=1 crypto rx=0x7f463000d500 tx=0x7f463000d9c0 comp rx=0 tx=0).stop 2024-09-16T15:49:23.746 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:23.747+0000 7f4648039640 1 -- 172.21.15.123:0/3912160641 shutdown_connections 2024-09-16T15:49:23.746 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:23.747+0000 7f4648039640 1 --2- 172.21.15.123:0/3912160641 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f4624073340 0x7f4624075800 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:23.746 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:23.747+0000 7f4648039640 1 --2- 172.21.15.123:0/3912160641 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f464015e500 0x7f464015e980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:23.746 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:23.747+0000 7f4648039640 1 --2- 172.21.15.123:0/3912160641 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f46401f2b40 0x7f4640151e20 unknown :-1 s=CLOSED pgs=1264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:23.746 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:23.747+0000 7f4648039640 1 --2- 172.21.15.123:0/3912160641 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f462c0052f0 0x7f46401559b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:23.746 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:23.747+0000 7f4648039640 1 -- 172.21.15.123:0/3912160641 >> 172.21.15.123:0/3912160641 conn(0x7f46400b2610 msgr2=0x7f464015b0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:49:23.746 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:23.747+0000 7f4648039640 1 -- 172.21.15.123:0/3912160641 shutdown_connections 2024-09-16T15:49:23.746 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:23.747+0000 7f4648039640 1 -- 172.21.15.123:0/3912160641 wait complete. 2024-09-16T15:49:23.762 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-09-16T15:49:23.762 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T15:49:23.889 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:23.891+0000 7fd4d2e7f640 1 -- 172.21.15.123:0/2936003931 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd4cc15ae10 msgr2=0x7fd4cc0ca160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:23.890 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:23.891+0000 7fd4d2e7f640 1 --2- 172.21.15.123:0/2936003931 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd4cc15ae10 0x7fd4cc0ca160 secure :-1 s=READY pgs=1265 cs=0 l=1 rev1=1 crypto rx=0x7fd4cc0cad20 tx=0x7fd4b401d140 comp rx=0 tx=0).stop 2024-09-16T15:49:23.890 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:23.891+0000 7fd4d2e7f640 1 -- 172.21.15.123:0/2936003931 shutdown_connections 2024-09-16T15:49:23.890 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:23.891+0000 7fd4d2e7f640 1 --2- 172.21.15.123:0/2936003931 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd4cc15ae10 0x7fd4cc0ca160 unknown :-1 s=CLOSED pgs=1265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:23.890 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:23.891+0000 7fd4d2e7f640 1 --2- 172.21.15.123:0/2936003931 >> v2:172.21.15.184:3300/0 conn(0x7fd4cc06fc00 0x7fd4cc0cf3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:23.890 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:23.891+0000 7fd4d2e7f640 1 -- 172.21.15.123:0/2936003931 >> 172.21.15.123:0/2936003931 conn(0x7fd4cc0cfb10 msgr2=0x7fd4cc0cff20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:49:23.890 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:23.891+0000 7fd4d2e7f640 1 -- 172.21.15.123:0/2936003931 shutdown_connections 2024-09-16T15:49:23.890 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:23.891+0000 7fd4d2e7f640 1 -- 172.21.15.123:0/2936003931 wait complete. 2024-09-16T15:49:23.891 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:23.891+0000 7fd4d2e7f640 1 Processor -- start 2024-09-16T15:49:23.891 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:23.891+0000 7fd4d2e7f640 1 -- start start 2024-09-16T15:49:23.893 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:23.891+0000 7fd4d2e7f640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd4cc06fc00 0x7fd4cc14cda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:49:23.893 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:23.891+0000 7fd4d2e7f640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fd4cc0ca6a0 0x7fd4cc14c210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:49:23.893 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:23.891+0000 7fd4d2e7f640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd4cc15ae10 0x7fd4cc1f9270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:49:23.893 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:23.891+0000 7fd4d2e7f640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fd4cc06efe0 con 0x7fd4cc15ae10 2024-09-16T15:49:23.893 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:23.891+0000 7fd4d2e7f640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fd4cc06ee60 con 0x7fd4cc06fc00 2024-09-16T15:49:23.894 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:23.891+0000 7fd4d2e7f640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fd4cc06f160 con 0x7fd4cc0ca6a0 2024-09-16T15:49:23.894 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:23.891+0000 7fd4d141c640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd4cc15ae10 0x7fd4cc1f9270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:49:23.894 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:23.891+0000 7fd4d141c640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd4cc15ae10 0x7fd4cc1f9270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:35410/0 (socket says 172.21.15.123:35410) 2024-09-16T15:49:23.894 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:23.891+0000 7fd4d141c640 1 -- 172.21.15.123:0/3684023598 learned_addr learned my addr 172.21.15.123:0/3684023598 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:49:23.894 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:23.891+0000 7fd4cbfff640 1 --2- 172.21.15.123:0/3684023598 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fd4cc0ca6a0 0x7fd4cc14c210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:49:23.894 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:23.891+0000 7fd4d0c1b640 1 --2- 172.21.15.123:0/3684023598 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd4cc06fc00 0x7fd4cc14cda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:49:23.894 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:23.891+0000 7fd4d141c640 1 -- 172.21.15.123:0/3684023598 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fd4cc0ca6a0 msgr2=0x7fd4cc14c210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:23.894 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:23.891+0000 7fd4d141c640 1 --2- 172.21.15.123:0/3684023598 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fd4cc0ca6a0 0x7fd4cc14c210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:23.894 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:23.891+0000 7fd4d141c640 1 -- 172.21.15.123:0/3684023598 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd4cc06fc00 msgr2=0x7fd4cc14cda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:23.894 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:23.891+0000 7fd4d141c640 1 --2- 172.21.15.123:0/3684023598 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd4cc06fc00 0x7fd4cc14cda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:23.894 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:23.891+0000 7fd4d141c640 1 -- 172.21.15.123:0/3684023598 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd4cc156760 con 0x7fd4cc15ae10 2024-09-16T15:49:23.894 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:23.891+0000 7fd4d0c1b640 1 --2- 172.21.15.123:0/3684023598 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd4cc06fc00 0x7fd4cc14cda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T15:49:23.894 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:23.891+0000 7fd4d141c640 1 --2- 172.21.15.123:0/3684023598 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd4cc15ae10 0x7fd4cc1f9270 secure :-1 s=READY pgs=1266 cs=0 l=1 rev1=1 crypto rx=0x7fd4bc00c780 tx=0x7fd4bc00cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:49:23.895 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:23.895+0000 7fd4c97fa640 1 -- 172.21.15.123:0/3684023598 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd4bc004b70 con 0x7fd4cc15ae10 2024-09-16T15:49:23.895 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:23.895+0000 7fd4c97fa640 1 -- 172.21.15.123:0/3684023598 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fd4bc004e00 con 0x7fd4cc15ae10 2024-09-16T15:49:23.895 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:23.895+0000 7fd4d2e7f640 1 -- 172.21.15.123:0/3684023598 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd4cc14cac0 con 0x7fd4cc15ae10 2024-09-16T15:49:23.895 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:23.895+0000 7fd4c97fa640 1 -- 172.21.15.123:0/3684023598 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd4bc00aa90 con 0x7fd4cc15ae10 2024-09-16T15:49:23.895 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:23.895+0000 7fd4d2e7f640 1 -- 172.21.15.123:0/3684023598 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd4cc1fa080 con 0x7fd4cc15ae10 2024-09-16T15:49:23.896 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:23.895+0000 7fd4d2e7f640 1 -- 172.21.15.123:0/3684023598 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd4cc1569c0 con 0x7fd4cc15ae10 2024-09-16T15:49:23.896 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:23.895+0000 7fd4c97fa640 1 -- 172.21.15.123:0/3684023598 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 19) ==== 97488+0+0 (secure 0 0 0) 0x7fd4bc00ac30 con 0x7fd4cc15ae10 2024-09-16T15:49:23.896 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:23.895+0000 7fd4c97fa640 1 --2- 172.21.15.123:0/3684023598 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fd4ac073340 0x7fd4ac075800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:49:23.896 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:23.895+0000 7fd4c97fa640 1 -- 172.21.15.123:0/3684023598 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(119..119 src has 1..119) ==== 5737+0+0 (secure 0 0 0) 0x7fd4bc095110 con 0x7fd4cc15ae10 2024-09-16T15:49:23.896 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:23.895+0000 7fd4d0c1b640 1 --2- 172.21.15.123:0/3684023598 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fd4ac073340 0x7fd4ac075800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:49:23.896 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:23.895+0000 7fd4d0c1b640 1 --2- 172.21.15.123:0/3684023598 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fd4ac073340 0x7fd4ac075800 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fd4c4005fd0 tx=0x7fd4c4005d40 comp rx=0 tx=0).ready entity=mgr.4150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:49:23.899 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:23.899+0000 7fd4c97fa640 1 -- 172.21.15.123:0/3684023598 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7fd4bc020070 con 0x7fd4cc15ae10 2024-09-16T15:49:24.071 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:24.071+0000 7fd4d2e7f640 1 -- 172.21.15.123:0/3684023598 --> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fd4cc14c750 con 0x7fd4ac073340 2024-09-16T15:49:24.076 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:24.075+0000 7fd4c97fa640 1 -- 172.21.15.123:0/3684023598 <== mgr.4150 v2:172.21.15.123:6836/2464759676 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+252471 (secure 0 0 0) 0x7fd4cc14c750 con 0x7fd4ac073340 2024-09-16T15:49:24.079 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-09-16T15:49:24.080 INFO:teuthology.orchestra.run.smithi123.stderr:dumped all 2024-09-16T15:49:24.081 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:24.079+0000 7fd4d2e7f640 1 -- 172.21.15.123:0/3684023598 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fd4ac073340 msgr2=0x7fd4ac075800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:24.081 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:24.079+0000 7fd4d2e7f640 1 --2- 172.21.15.123:0/3684023598 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fd4ac073340 0x7fd4ac075800 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fd4c4005fd0 tx=0x7fd4c4005d40 comp rx=0 tx=0).stop 2024-09-16T15:49:24.081 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:24.079+0000 7fd4d2e7f640 1 -- 172.21.15.123:0/3684023598 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd4cc15ae10 msgr2=0x7fd4cc1f9270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:24.081 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:24.079+0000 7fd4d2e7f640 1 --2- 172.21.15.123:0/3684023598 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd4cc15ae10 0x7fd4cc1f9270 secure :-1 s=READY pgs=1266 cs=0 l=1 rev1=1 crypto rx=0x7fd4bc00c780 tx=0x7fd4bc00cc40 comp rx=0 tx=0).stop 2024-09-16T15:49:24.081 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:24.079+0000 7fd4d2e7f640 1 -- 172.21.15.123:0/3684023598 shutdown_connections 2024-09-16T15:49:24.082 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:24.079+0000 7fd4d2e7f640 1 --2- 172.21.15.123:0/3684023598 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fd4ac073340 0x7fd4ac075800 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:24.082 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:24.079+0000 7fd4d2e7f640 1 --2- 172.21.15.123:0/3684023598 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd4cc15ae10 0x7fd4cc1f9270 unknown :-1 s=CLOSED pgs=1266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:24.082 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:24.079+0000 7fd4d2e7f640 1 --2- 172.21.15.123:0/3684023598 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fd4cc0ca6a0 0x7fd4cc14c210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:24.082 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:24.079+0000 7fd4d2e7f640 1 --2- 172.21.15.123:0/3684023598 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd4cc06fc00 0x7fd4cc14cda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:24.082 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:24.079+0000 7fd4d2e7f640 1 -- 172.21.15.123:0/3684023598 >> 172.21.15.123:0/3684023598 conn(0x7fd4cc0cfb10 msgr2=0x7fd4cc153c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:49:24.082 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:24.079+0000 7fd4d2e7f640 1 -- 172.21.15.123:0/3684023598 shutdown_connections 2024-09-16T15:49:24.082 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:24.079+0000 7fd4d2e7f640 1 -- 172.21.15.123:0/3684023598 wait complete. 2024-09-16T15:49:24.097 INFO:teuthology.orchestra.run.smithi123.stdout:{"pg_ready":true,"pg_map":{"version":104,"stamp":"2024-09-16T15:49:22.099031+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":157,"num_write_kb":1642,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":74,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":262096,"kb_used_data":12568,"kb_used_omap":12,"kb_used_meta":249331,"kb_avail":754712624,"statfs":{"total":773094113280,"available":772825726976,"internally_reserved":0,"allocated":12869632,"data_stored":6311261,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":255315505},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":39,"apply_latency_ms":39,"commit_latency_ns":39000000,"apply_latency_ns":39000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.129691"},"pg_stats":[{"pgid":"19.3f","version":"0'0","reported_seq":10,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-09-16T15:49:20.026438+0000","last_change":"2024-09-16T15:49:20.025405+0000","last_active":"2024-09-16T15:49:20.026438+0000","last_peered":"2024-09-16T15:49:20.026438+0000","last_clean":"2024-09-16T15:49:20.026438+0000","last_became_active":"2024-09-16T15:49:20.025131+0000","last_became_peered":"2024-09-16T15:49:20.025131+0000","last_unstale":"2024-09-16T15:49:20.026438+0000","last_undegraded":"2024-09-16T15:49:20.026438+0000","last_fullsized":"2024-09-16T15:49:20.026438+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:49:18.967562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:49:18.967562+0000","last_clean_scrub_stamp":"2024-09-16T15:49:18.967562+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T01:55:12.894755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.3e","version":"0'0","reported_seq":10,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-09-16T15:49:20.022660+0000","last_change":"2024-09-16T15:49:20.018810+0000","last_active":"2024-09-16T15:49:20.022660+0000","last_peered":"2024-09-16T15:49:20.022660+0000","last_clean":"2024-09-16T15:49:20.022660+0000","last_became_active":"2024-09-16T15:49:20.018022+0000","last_became_peered":"2024-09-16T15:49:20.018022+0000","last_unstale":"2024-09-16T15:49:20.022660+0000","last_undegraded":"2024-09-16T15:49:20.022660+0000","last_fullsized":"2024-09-16T15:49:20.022660+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:49:18.967562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:49:18.967562+0000","last_clean_scrub_stamp":"2024-09-16T15:49:18.967562+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:59:36.978001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.3d","version":"0'0","reported_seq":10,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-09-16T15:49:20.043291+0000","last_change":"2024-09-16T15:49:20.034522+0000","last_active":"2024-09-16T15:49:20.043291+0000","last_peered":"2024-09-16T15:49:20.043291+0000","last_clean":"2024-09-16T15:49:20.043291+0000","last_became_active":"2024-09-16T15:49:20.033532+0000","last_became_peered":"2024-09-16T15:49:20.033532+0000","last_unstale":"2024-09-16T15:49:20.043291+0000","last_undegraded":"2024-09-16T15:49:20.043291+0000","last_fullsized":"2024-09-16T15:49:20.043291+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:49:18.967562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:49:18.967562+0000","last_clean_scrub_stamp":"2024-09-16T15:49:18.967562+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:41:41.203722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.3c","version":"0'0","reported_seq":10,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-09-16T15:49:20.051337+0000","last_change":"2024-09-16T15:49:20.050802+0000","last_active":"2024-09-16T15:49:20.051337+0000","last_peered":"2024-09-16T15:49:20.051337+0000","last_clean":"2024-09-16T15:49:20.051337+0000","last_became_active":"2024-09-16T15:49:20.050496+0000","last_became_peered":"2024-09-16T15:49:20.050496+0000","last_unstale":"2024-09-16T15:49:20.051337+0000","last_undegraded":"2024-09-16T15:49:20.051337+0000","last_fullsized":"2024-09-16T15:49:20.051337+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:49:18.967562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:49:18.967562+0000","last_clean_scrub_stamp":"2024-09-16T15:49:18.967562+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T02:00:42.471624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.3b","version":"0'0","reported_seq":10,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-09-16T15:49:20.058908+0000","last_change":"2024-09-16T15:49:20.055162+0000","last_active":"2024-09-16T15:49:20.058908+0000","last_peered":"2024-09-16T15:49:20.058908+0000","last_clean":"2024-09-16T15:49:20.058908+0000","last_became_active":"2024-09-16T15:49:20.050164+0000","last_became_peered":"2024-09-16T15:49:20.050164+0000","last_unstale":"2024-09-16T15:49:20.058908+0000","last_undegraded":"2024-09-16T15:49:20.058908+0000","last_fullsized":"2024-09-16T15:49:20.058908+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:49:18.967562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:49:18.967562+0000","last_clean_scrub_stamp":"2024-09-16T15:49:18.967562+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T22:54:20.092019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.3a","version":"0'0","reported_seq":10,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-09-16T15:49:20.046644+0000","last_change":"2024-09-16T15:49:20.034628+0000","last_active":"2024-09-16T15:49:20.046644+0000","last_peered":"2024-09-16T15:49:20.046644+0000","last_clean":"2024-09-16T15:49:20.046644+0000","last_became_active":"2024-09-16T15:49:20.030146+0000","last_became_peered":"2024-09-16T15:49:20.030146+0000","last_unstale":"2024-09-16T15:49:20.046644+0000","last_undegraded":"2024-09-16T15:49:20.046644+0000","last_fullsized":"2024-09-16T15:49:20.046644+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:49:18.967562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:49:18.967562+0000","last_clean_scrub_stamp":"2024-09-16T15:49:18.967562+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T01:16:16.527543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.39","version":"0'0","reported_seq":10,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-09-16T15:49:20.038358+0000","last_change":"2024-09-16T15:49:20.029027+0000","last_active":"2024-09-16T15:49:20.038358+0000","last_peered":"2024-09-16T15:49:20.038358+0000","last_clean":"2024-09-16T15:49:20.038358+0000","last_became_active":"2024-09-16T15:49:20.028199+0000","last_became_peered":"2024-09-16T15:49:20.028199+0000","last_unstale":"2024-09-16T15:49:20.038358+0000","last_undegraded":"2024-09-16T15:49:20.038358+0000","last_fullsized":"2024-09-16T15:49:20.038358+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:49:18.967562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:49:18.967562+0000","last_clean_scrub_stamp":"2024-09-16T15:49:18.967562+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T03:27:25.702064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.38","version":"0'0","reported_seq":10,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-09-16T15:49:20.052794+0000","last_change":"2024-09-16T15:49:20.033056+0000","last_active":"2024-09-16T15:49:20.052794+0000","last_peered":"2024-09-16T15:49:20.052794+0000","last_clean":"2024-09-16T15:49:20.052794+0000","last_became_active":"2024-09-16T15:49:20.027933+0000","last_became_peered":"2024-09-16T15:49:20.027933+0000","last_unstale":"2024-09-16T15:49:20.052794+0000","last_undegraded":"2024-09-16T15:49:20.052794+0000","last_fullsized":"2024-09-16T15:49:20.052794+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:49:18.967562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:49:18.967562+0000","last_clean_scrub_stamp":"2024-09-16T15:49:18.967562+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:17:29.529756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.37","version":"0'0","reported_seq":10,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-09-16T15:49:20.058001+0000","last_change":"2024-09-16T15:49:20.052915+0000","last_active":"2024-09-16T15:49:20.058001+0000","last_peered":"2024-09-16T15:49:20.058001+0000","last_clean":"2024-09-16T15:49:20.058001+0000","last_became_active":"2024-09-16T15:49:20.052189+0000","last_became_peered":"2024-09-16T15:49:20.052189+0000","last_unstale":"2024-09-16T15:49:20.058001+0000","last_undegraded":"2024-09-16T15:49:20.058001+0000","last_fullsized":"2024-09-16T15:49:20.058001+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:49:18.967562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:49:18.967562+0000","last_clean_scrub_stamp":"2024-09-16T15:49:18.967562+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:23:51.294104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.36","version":"0'0","reported_seq":10,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-09-16T15:49:20.022635+0000","last_change":"2024-09-16T15:49:20.018047+0000","last_active":"2024-09-16T15:49:20.022635+0000","last_peered":"2024-09-16T15:49:20.022635+0000","last_clean":"2024-09-16T15:49:20.022635+0000","last_became_active":"2024-09-16T15:49:20.017769+0000","last_became_peered":"2024-09-16T15:49:20.017769+0000","last_unstale":"2024-09-16T15:49:20.022635+0000","last_undegraded":"2024-09-16T15:49:20.022635+0000","last_fullsized":"2024-09-16T15:49:20.022635+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:49:18.967562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:49:18.967562+0000","last_clean_scrub_stamp":"2024-09-16T15:49:18.967562+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T00:46:04.825017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.35","version":"0'0","reported_seq":10,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-09-16T15:49:20.037376+0000","last_change":"2024-09-16T15:49:20.030040+0000","last_active":"2024-09-16T15:49:20.037376+0000","last_peered":"2024-09-16T15:49:20.037376+0000","last_clean":"2024-09-16T15:49:20.037376+0000","last_became_active":"2024-09-16T15:49:20.027913+0000","last_became_peered":"2024-09-16T15:49:20.027913+0000","last_unstale":"2024-09-16T15:49:20.037376+0000","last_undegraded":"2024-09-16T15:49:20.037376+0000","last_fullsized":"2024-09-16T15:49:20.037376+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:49:18.967562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:49:18.967562+0000","last_clean_scrub_stamp":"2024-09-16T15:49:18.967562+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:25:29.851210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.34","version":"0'0","reported_seq":10,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-09-16T15:49:20.042753+0000","last_change":"2024-09-16T15:49:20.031136+0000","last_active":"2024-09-16T15:49:20.042753+0000","last_peered":"2024-09-16T15:49:20.042753+0000","last_clean":"2024-09-16T15:49:20.042753+0000","last_became_active":"2024-09-16T15:49:20.026976+0000","last_became_peered":"2024-09-16T15:49:20.026976+0000","last_unstale":"2024-09-16T15:49:20.042753+0000","last_undegraded":"2024-09-16T15:49:20.042753+0000","last_fullsized":"2024-09-16T15:49:20.042753+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:49:18.967562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:49:18.967562+0000","last_clean_scrub_stamp":"2024-09-16T15:49:18.967562+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T18:28:59.419042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.33","version":"0'0","reported_seq":10,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-09-16T15:49:20.027590+0000","last_change":"2024-09-16T15:49:20.020663+0000","last_active":"2024-09-16T15:49:20.027590+0000","last_peered":"2024-09-16T15:49:20.027590+0000","last_clean":"2024-09-16T15:49:20.027590+0000","last_became_active":"2024-09-16T15:49:20.019315+0000","last_became_peered":"2024-09-16T15:49:20.019315+0000","last_unstale":"2024-09-16T15:49:20.027590+0000","last_undegraded":"2024-09-16T15:49:20.027590+0000","last_fullsized":"2024-09-16T15:49:20.027590+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:49:18.967562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:49:18.967562+0000","last_clean_scrub_stamp":"2024-09-16T15:49:18.967562+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T00:45:43.811211+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.32","version":"0'0","reported_seq":10,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-09-16T15:49:20.022362+0000","last_change":"2024-09-16T15:49:20.016990+0000","last_active":"2024-09-16T15:49:20.022362+0000","last_peered":"2024-09-16T15:49:20.022362+0000","last_clean":"2024-09-16T15:49:20.022362+0000","last_became_active":"2024-09-16T15:49:20.016648+0000","last_became_peered":"2024-09-16T15:49:20.016648+0000","last_unstale":"2024-09-16T15:49:20.022362+0000","last_undegraded":"2024-09-16T15:49:20.022362+0000","last_fullsized":"2024-09-16T15:49:20.022362+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:49:18.967562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:49:18.967562+0000","last_clean_scrub_stamp":"2024-09-16T15:49:18.967562+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T03:17:28.387500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.31","version":"0'0","reported_seq":10,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-09-16T15:49:20.026104+0000","last_change":"2024-09-16T15:49:20.022374+0000","last_active":"2024-09-16T15:49:20.026104+0000","last_peered":"2024-09-16T15:49:20.026104+0000","last_clean":"2024-09-16T15:49:20.026104+0000","last_became_active":"2024-09-16T15:49:20.021899+0000","last_became_peered":"2024-09-16T15:49:20.021899+0000","last_unstale":"2024-09-16T15:49:20.026104+0000","last_undegraded":"2024-09-16T15:49:20.026104+0000","last_fullsized":"2024-09-16T15:49:20.026104+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:49:18.967562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:49:18.967562+0000","last_clean_scrub_stamp":"2024-09-16T15:49:18.967562+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:20:43.185830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.30","version":"0'0","reported_seq":10,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-09-16T15:49:20.023423+0000","last_change":"2024-09-16T15:49:20.020306+0000","last_active":"2024-09-16T15:49:20.023423+0000","last_peered":"2024-09-16T15:49:20.023423+0000","last_clean":"2024-09-16T15:49:20.023423+0000","last_became_active":"2024-09-16T15:49:20.020046+0000","last_became_peered":"2024-09-16T15:49:20.020046+0000","last_unstale":"2024-09-16T15:49:20.023423+0000","last_undegraded":"2024-09-16T15:49:20.023423+0000","last_fullsized":"2024-09-16T15:49:20.023423+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:49:18.967562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:49:18.967562+0000","last_clean_scrub_stamp":"2024-09-16T15:49:18.967562+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T22:30:46.816501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.2f","version":"0'0","reported_seq":10,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-09-16T15:49:20.046739+0000","last_change":"2024-09-16T15:49:20.035937+0000","last_active":"2024-09-16T15:49:20.046739+0000","last_peered":"2024-09-16T15:49:20.046739+0000","last_clean":"2024-09-16T15:49:20.046739+0000","last_became_active":"2024-09-16T15:49:20.035175+0000","last_became_peered":"2024-09-16T15:49:20.035175+0000","last_unstale":"2024-09-16T15:49:20.046739+0000","last_undegraded":"2024-09-16T15:49:20.046739+0000","last_fullsized":"2024-09-16T15:49:20.046739+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:49:18.967562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:49:18.967562+0000","last_clean_scrub_stamp":"2024-09-16T15:49:18.967562+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:30:33.365956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.2e","version":"0'0","reported_seq":10,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-09-16T15:49:20.023098+0000","last_change":"2024-09-16T15:49:20.020864+0000","last_active":"2024-09-16T15:49:20.023098+0000","last_peered":"2024-09-16T15:49:20.023098+0000","last_clean":"2024-09-16T15:49:20.023098+0000","last_became_active":"2024-09-16T15:49:20.020624+0000","last_became_peered":"2024-09-16T15:49:20.020624+0000","last_unstale":"2024-09-16T15:49:20.023098+0000","last_undegraded":"2024-09-16T15:49:20.023098+0000","last_fullsized":"2024-09-16T15:49:20.023098+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:49:18.967562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:49:18.967562+0000","last_clean_scrub_stamp":"2024-09-16T15:49:18.967562+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:47:36.317556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.2d","version":"0'0","reported_seq":10,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-09-16T15:49:20.037465+0000","last_change":"2024-09-16T15:49:20.030653+0000","last_active":"2024-09-16T15:49:20.037465+0000","last_peered":"2024-09-16T15:49:20.037465+0000","last_clean":"2024-09-16T15:49:20.037465+0000","last_became_active":"2024-09-16T15:49:20.030433+0000","last_became_peered":"2024-09-16T15:49:20.030433+0000","last_unstale":"2024-09-16T15:49:20.037465+0000","last_undegraded":"2024-09-16T15:49:20.037465+0000","last_fullsized":"2024-09-16T15:49:20.037465+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:49:18.967562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:49:18.967562+0000","last_clean_scrub_stamp":"2024-09-16T15:49:18.967562+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:06:43.611308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.2c","version":"0'0","reported_seq":10,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-09-16T15:49:20.015023+0000","last_change":"2024-09-16T15:49:20.011995+0000","last_active":"2024-09-16T15:49:20.015023+0000","last_peered":"2024-09-16T15:49:20.015023+0000","last_clean":"2024-09-16T15:49:20.015023+0000","last_became_active":"2024-09-16T15:49:20.011483+0000","last_became_peered":"2024-09-16T15:49:20.011483+0000","last_unstale":"2024-09-16T15:49:20.015023+0000","last_undegraded":"2024-09-16T15:49:20.015023+0000","last_fullsized":"2024-09-16T15:49:20.015023+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:49:18.967562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:49:18.967562+0000","last_clean_scrub_stamp":"2024-09-16T15:49:18.967562+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T03:06:32.057244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.2b","version":"0'0","reported_seq":10,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-09-16T15:49:20.059412+0000","last_change":"2024-09-16T15:49:20.058046+0000","last_active":"2024-09-16T15:49:20.059412+0000","last_peered":"2024-09-16T15:49:20.059412+0000","last_clean":"2024-09-16T15:49:20.059412+0000","last_became_active":"2024-09-16T15:49:20.049107+0000","last_became_peered":"2024-09-16T15:49:20.049107+0000","last_unstale":"2024-09-16T15:49:20.059412+0000","last_undegraded":"2024-09-16T15:49:20.059412+0000","last_fullsized":"2024-09-16T15:49:20.059412+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:49:18.967562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:49:18.967562+0000","last_clean_scrub_stamp":"2024-09-16T15:49:18.967562+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:14:53.412945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.2a","version":"0'0","reported_seq":10,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-09-16T15:49:20.037304+0000","last_change":"2024-09-16T15:49:20.028064+0000","last_active":"2024-09-16T15:49:20.037304+0000","last_peered":"2024-09-16T15:49:20.037304+0000","last_clean":"2024-09-16T15:49:20.037304+0000","last_became_active":"2024-09-16T15:49:20.027664+0000","last_became_peered":"2024-09-16T15:49:20.027664+0000","last_unstale":"2024-09-16T15:49:20.037304+0000","last_undegraded":"2024-09-16T15:49:20.037304+0000","last_fullsized":"2024-09-16T15:49:20.037304+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:49:18.967562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:49:18.967562+0000","last_clean_scrub_stamp":"2024-09-16T15:49:18.967562+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T01:21:22.074609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.29","version":"0'0","reported_seq":10,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-09-16T15:49:20.043208+0000","last_change":"2024-09-16T15:49:20.033729+0000","last_active":"2024-09-16T15:49:20.043208+0000","last_peered":"2024-09-16T15:49:20.043208+0000","last_clean":"2024-09-16T15:49:20.043208+0000","last_became_active":"2024-09-16T15:49:20.026602+0000","last_became_peered":"2024-09-16T15:49:20.026602+0000","last_unstale":"2024-09-16T15:49:20.043208+0000","last_undegraded":"2024-09-16T15:49:20.043208+0000","last_fullsized":"2024-09-16T15:49:20.043208+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:49:18.967562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:49:18.967562+0000","last_clean_scrub_stamp":"2024-09-16T15:49:18.967562+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:59:22.673559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.28","version":"0'0","reported_seq":10,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-09-16T15:49:20.037398+0000","last_change":"2024-09-16T15:49:20.029115+0000","last_active":"2024-09-16T15:49:20.037398+0000","last_peered":"2024-09-16T15:49:20.037398+0000","last_clean":"2024-09-16T15:49:20.037398+0000","last_became_active":"2024-09-16T15:49:20.028609+0000","last_became_peered":"2024-09-16T15:49:20.028609+0000","last_unstale":"2024-09-16T15:49:20.037398+0000","last_undegraded":"2024-09-16T15:49:20.037398+0000","last_fullsized":"2024-09-16T15:49:20.037398+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:49:18.967562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:49:18.967562+0000","last_clean_scrub_stamp":"2024-09-16T15:49:18.967562+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T23:34:04.482788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.27","version":"0'0","reported_seq":10,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-09-16T15:49:20.045378+0000","last_change":"2024-09-16T15:49:20.041979+0000","last_active":"2024-09-16T15:49:20.045378+0000","last_peered":"2024-09-16T15:49:20.045378+0000","last_clean":"2024-09-16T15:49:20.045378+0000","last_became_active":"2024-09-16T15:49:20.041570+0000","last_became_peered":"2024-09-16T15:49:20.041570+0000","last_unstale":"2024-09-16T15:49:20.045378+0000","last_undegraded":"2024-09-16T15:49:20.045378+0000","last_fullsized":"2024-09-16T15:49:20.045378+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:49:18.967562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:49:18.967562+0000","last_clean_scrub_stamp":"2024-09-16T15:49:18.967562+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T22:59:42.998898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.26","version":"0'0","reported_seq":10,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-09-16T15:49:20.023069+0000","last_change":"2024-09-16T15:49:20.019624+0000","last_active":"2024-09-16T15:49:20.023069+0000","last_peered":"2024-09-16T15:49:20.023069+0000","last_clean":"2024-09-16T15:49:20.023069+0000","last_became_active":"2024-09-16T15:49:20.019378+0000","last_became_peered":"2024-09-16T15:49:20.019378+0000","last_unstale":"2024-09-16T15:49:20.023069+0000","last_undegraded":"2024-09-16T15:49:20.023069+0000","last_fullsized":"2024-09-16T15:49:20.023069+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:49:18.967562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:49:18.967562+0000","last_clean_scrub_stamp":"2024-09-16T15:49:18.967562+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:23:40.870521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.25","version":"0'0","reported_seq":10,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-09-16T15:49:20.021365+0000","last_change":"2024-09-16T15:49:20.019444+0000","last_active":"2024-09-16T15:49:20.021365+0000","last_peered":"2024-09-16T15:49:20.021365+0000","last_clean":"2024-09-16T15:49:20.021365+0000","last_became_active":"2024-09-16T15:49:20.019146+0000","last_became_peered":"2024-09-16T15:49:20.019146+0000","last_unstale":"2024-09-16T15:49:20.021365+0000","last_undegraded":"2024-09-16T15:49:20.021365+0000","last_fullsized":"2024-09-16T15:49:20.021365+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:49:18.967562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:49:18.967562+0000","last_clean_scrub_stamp":"2024-09-16T15:49:18.967562+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T02:29:37.284424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.24","version":"0'0","reported_seq":10,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-09-16T15:49:20.032552+0000","last_change":"2024-09-16T15:49:20.029167+0000","last_active":"2024-09-16T15:49:20.032552+0000","last_peered":"2024-09-16T15:49:20.032552+0000","last_clean":"2024-09-16T15:49:20.032552+0000","last_became_active":"2024-09-16T15:49:20.028409+0000","last_became_peered":"2024-09-16T15:49:20.028409+0000","last_unstale":"2024-09-16T15:49:20.032552+0000","last_undegraded":"2024-09-16T15:49:20.032552+0000","last_fullsized":"2024-09-16T15:49:20.032552+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:49:18.967562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:49:18.967562+0000","last_clean_scrub_stamp":"2024-09-16T15:49:18.967562+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T01:35:30.971956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.23","version":"0'0","reported_seq":10,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-09-16T15:49:20.027637+0000","last_change":"2024-09-16T15:49:20.023500+0000","last_active":"2024-09-16T15:49:20.027637+0000","last_peered":"2024-09-16T15:49:20.027637+0000","last_clean":"2024-09-16T15:49:20.027637+0000","last_became_active":"2024-09-16T15:49:20.023225+0000","last_became_peered":"2024-09-16T15:49:20.023225+0000","last_unstale":"2024-09-16T15:49:20.027637+0000","last_undegraded":"2024-09-16T15:49:20.027637+0000","last_fullsized":"2024-09-16T15:49:20.027637+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:49:18.967562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:49:18.967562+0000","last_clean_scrub_stamp":"2024-09-16T15:49:18.967562+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:15:15.039502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.22","version":"0'0","reported_seq":10,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-09-16T15:49:20.026433+0000","last_change":"2024-09-16T15:49:20.024936+0000","last_active":"2024-09-16T15:49:20.026433+0000","last_peered":"2024-09-16T15:49:20.026433+0000","last_clean":"2024-09-16T15:49:20.026433+0000","last_became_active":"2024-09-16T15:49:20.024607+0000","last_became_peered":"2024-09-16T15:49:20.024607+0000","last_unstale":"2024-09-16T15:49:20.026433+0000","last_undegraded":"2024-09-16T15:49:20.026433+0000","last_fullsized":"2024-09-16T15:49:20.026433+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:49:18.967562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:49:18.967562+0000","last_clean_scrub_stamp":"2024-09-16T15:49:18.967562+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:34:05.136599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.21","version":"0'0","reported_seq":10,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-09-16T15:49:20.059115+0000","last_change":"2024-09-16T15:49:20.055866+0000","last_active":"2024-09-16T15:49:20.059115+0000","last_peered":"2024-09-16T15:49:20.059115+0000","last_clean":"2024-09-16T15:49:20.059115+0000","last_became_active":"2024-09-16T15:49:20.043962+0000","last_became_peered":"2024-09-16T15:49:20.043962+0000","last_unstale":"2024-09-16T15:49:20.059115+0000","last_undegraded":"2024-09-16T15:49:20.059115+0000","last_fullsized":"2024-09-16T15:49:20.059115+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:49:18.967562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:49:18.967562+0000","last_clean_scrub_stamp":"2024-09-16T15:49:18.967562+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T22:09:02.693580+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.20","version":"0'0","reported_seq":10,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-09-16T15:49:20.020021+0000","last_change":"2024-09-16T15:49:20.017088+0000","last_active":"2024-09-16T15:49:20.020021+0000","last_peered":"2024-09-16T15:49:20.020021+0000","last_clean":"2024-09-16T15:49:20.020021+0000","last_became_active":"2024-09-16T15:49:20.012864+0000","last_became_peered":"2024-09-16T15:49:20.012864+0000","last_unstale":"2024-09-16T15:49:20.020021+0000","last_undegraded":"2024-09-16T15:49:20.020021+0000","last_fullsized":"2024-09-16T15:49:20.020021+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:49:18.967562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:49:18.967562+0000","last_clean_scrub_stamp":"2024-09-16T15:49:18.967562+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:54:14.214563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.1e","version":"0'0","reported_seq":10,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-09-16T15:49:20.037227+0000","last_change":"2024-09-16T15:49:20.032634+0000","last_active":"2024-09-16T15:49:20.037227+0000","last_peered":"2024-09-16T15:49:20.037227+0000","last_clean":"2024-09-16T15:49:20.037227+0000","last_became_active":"2024-09-16T15:49:20.032152+0000","last_became_peered":"2024-09-16T15:49:20.032152+0000","last_unstale":"2024-09-16T15:49:20.037227+0000","last_undegraded":"2024-09-16T15:49:20.037227+0000","last_fullsized":"2024-09-16T15:49:20.037227+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:49:18.967562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:49:18.967562+0000","last_clean_scrub_stamp":"2024-09-16T15:49:18.967562+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T03:01:45.514782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.1f","version":"118'4","reported_seq":17,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-09-16T15:49:20.049550+0000","last_change":"2024-09-16T15:49:19.016800+0000","last_active":"2024-09-16T15:49:20.049550+0000","last_peered":"2024-09-16T15:49:20.049550+0000","last_clean":"2024-09-16T15:49:20.049550+0000","last_became_active":"2024-09-16T15:49:19.016007+0000","last_became_peered":"2024-09-16T15:49:19.016007+0000","last_unstale":"2024-09-16T15:49:20.049550+0000","last_undegraded":"2024-09-16T15:49:20.049550+0000","last_fullsized":"2024-09-16T15:49:20.049550+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:49:17.964400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:49:17.964400+0000","last_clean_scrub_stamp":"2024-09-16T15:49:17.964400+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:21:30.845086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.1f","version":"0'0","reported_seq":10,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-09-16T15:49:20.026446+0000","last_change":"2024-09-16T15:49:20.021973+0000","last_active":"2024-09-16T15:49:20.026446+0000","last_peered":"2024-09-16T15:49:20.026446+0000","last_clean":"2024-09-16T15:49:20.026446+0000","last_became_active":"2024-09-16T15:49:20.021179+0000","last_became_peered":"2024-09-16T15:49:20.021179+0000","last_unstale":"2024-09-16T15:49:20.026446+0000","last_undegraded":"2024-09-16T15:49:20.026446+0000","last_fullsized":"2024-09-16T15:49:20.026446+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:49:18.967562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:49:18.967562+0000","last_clean_scrub_stamp":"2024-09-16T15:49:18.967562+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T22:19:23.440465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.1e","version":"118'1","reported_seq":14,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-09-16T15:49:20.038176+0000","last_change":"2024-09-16T15:49:18.991536+0000","last_active":"2024-09-16T15:49:20.038176+0000","last_peered":"2024-09-16T15:49:20.038176+0000","last_clean":"2024-09-16T15:49:20.038176+0000","last_became_active":"2024-09-16T15:49:18.990793+0000","last_became_peered":"2024-09-16T15:49:18.990793+0000","last_unstale":"2024-09-16T15:49:20.038176+0000","last_undegraded":"2024-09-16T15:49:20.038176+0000","last_fullsized":"2024-09-16T15:49:20.038176+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:49:17.964400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:49:17.964400+0000","last_clean_scrub_stamp":"2024-09-16T15:49:17.964400+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T01:36:48.358813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.1c","version":"0'0","reported_seq":10,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-09-16T15:49:20.058623+0000","last_change":"2024-09-16T15:49:20.050146+0000","last_active":"2024-09-16T15:49:20.058623+0000","last_peered":"2024-09-16T15:49:20.058623+0000","last_clean":"2024-09-16T15:49:20.058623+0000","last_became_active":"2024-09-16T15:49:20.048951+0000","last_became_peered":"2024-09-16T15:49:20.048951+0000","last_unstale":"2024-09-16T15:49:20.058623+0000","last_undegraded":"2024-09-16T15:49:20.058623+0000","last_fullsized":"2024-09-16T15:49:20.058623+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:49:18.967562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:49:18.967562+0000","last_clean_scrub_stamp":"2024-09-16T15:49:18.967562+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T01:27:31.814144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.1d","version":"118'2","reported_seq":15,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-09-16T15:49:20.034973+0000","last_change":"2024-09-16T15:49:18.994755+0000","last_active":"2024-09-16T15:49:20.034973+0000","last_peered":"2024-09-16T15:49:20.034973+0000","last_clean":"2024-09-16T15:49:20.034973+0000","last_became_active":"2024-09-16T15:49:18.994324+0000","last_became_peered":"2024-09-16T15:49:18.994324+0000","last_unstale":"2024-09-16T15:49:20.034973+0000","last_undegraded":"2024-09-16T15:49:20.034973+0000","last_fullsized":"2024-09-16T15:49:20.034973+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:49:17.964400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:49:17.964400+0000","last_clean_scrub_stamp":"2024-09-16T15:49:17.964400+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T22:48:23.896738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.1d","version":"0'0","reported_seq":10,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-09-16T15:49:20.057485+0000","last_change":"2024-09-16T15:49:20.048299+0000","last_active":"2024-09-16T15:49:20.057485+0000","last_peered":"2024-09-16T15:49:20.057485+0000","last_clean":"2024-09-16T15:49:20.057485+0000","last_became_active":"2024-09-16T15:49:20.043603+0000","last_became_peered":"2024-09-16T15:49:20.043603+0000","last_unstale":"2024-09-16T15:49:20.057485+0000","last_undegraded":"2024-09-16T15:49:20.057485+0000","last_fullsized":"2024-09-16T15:49:20.057485+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:49:18.967562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:49:18.967562+0000","last_clean_scrub_stamp":"2024-09-16T15:49:18.967562+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:39:06.552608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.1c","version":"118'2","reported_seq":15,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-09-16T15:49:20.028954+0000","last_change":"2024-09-16T15:49:19.002172+0000","last_active":"2024-09-16T15:49:20.028954+0000","last_peered":"2024-09-16T15:49:20.028954+0000","last_clean":"2024-09-16T15:49:20.028954+0000","last_became_active":"2024-09-16T15:49:19.001764+0000","last_became_peered":"2024-09-16T15:49:19.001764+0000","last_unstale":"2024-09-16T15:49:20.028954+0000","last_undegraded":"2024-09-16T15:49:20.028954+0000","last_fullsized":"2024-09-16T15:49:20.028954+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:49:17.964400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:49:17.964400+0000","last_clean_scrub_stamp":"2024-09-16T15:49:17.964400+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T01:29:07.726449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.1a","version":"0'0","reported_seq":10,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-09-16T15:49:20.037632+0000","last_change":"2024-09-16T15:49:20.036305+0000","last_active":"2024-09-16T15:49:20.037632+0000","last_peered":"2024-09-16T15:49:20.037632+0000","last_clean":"2024-09-16T15:49:20.037632+0000","last_became_active":"2024-09-16T15:49:20.036098+0000","last_became_peered":"2024-09-16T15:49:20.036098+0000","last_unstale":"2024-09-16T15:49:20.037632+0000","last_undegraded":"2024-09-16T15:49:20.037632+0000","last_fullsized":"2024-09-16T15:49:20.037632+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:49:18.967562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:49:18.967562+0000","last_clean_scrub_stamp":"2024-09-16T15:49:18.967562+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:30:37.221003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.1b","version":"118'1","reported_seq":14,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-09-16T15:49:20.015235+0000","last_change":"2024-09-16T15:49:18.997418+0000","last_active":"2024-09-16T15:49:20.015235+0000","last_peered":"2024-09-16T15:49:20.015235+0000","last_clean":"2024-09-16T15:49:20.015235+0000","last_became_active":"2024-09-16T15:49:18.997039+0000","last_became_peered":"2024-09-16T15:49:18.997039+0000","last_unstale":"2024-09-16T15:49:20.015235+0000","last_undegraded":"2024-09-16T15:49:20.015235+0000","last_fullsized":"2024-09-16T15:49:20.015235+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:49:17.964400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:49:17.964400+0000","last_clean_scrub_stamp":"2024-09-16T15:49:17.964400+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T22:21:18.524483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.1b","version":"0'0","reported_seq":10,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-09-16T15:49:20.038254+0000","last_change":"2024-09-16T15:49:20.029380+0000","last_active":"2024-09-16T15:49:20.038254+0000","last_peered":"2024-09-16T15:49:20.038254+0000","last_clean":"2024-09-16T15:49:20.038254+0000","last_became_active":"2024-09-16T15:49:20.028892+0000","last_became_peered":"2024-09-16T15:49:20.028892+0000","last_unstale":"2024-09-16T15:49:20.038254+0000","last_undegraded":"2024-09-16T15:49:20.038254+0000","last_fullsized":"2024-09-16T15:49:20.038254+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:49:18.967562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:49:18.967562+0000","last_clean_scrub_stamp":"2024-09-16T15:49:18.967562+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:47:36.317556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.1a","version":"118'1","reported_seq":14,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-09-16T15:49:20.022245+0000","last_change":"2024-09-16T15:49:18.998790+0000","last_active":"2024-09-16T15:49:20.022245+0000","last_peered":"2024-09-16T15:49:20.022245+0000","last_clean":"2024-09-16T15:49:20.022245+0000","last_became_active":"2024-09-16T15:49:18.998134+0000","last_became_peered":"2024-09-16T15:49:18.998134+0000","last_unstale":"2024-09-16T15:49:20.022245+0000","last_undegraded":"2024-09-16T15:49:20.022245+0000","last_fullsized":"2024-09-16T15:49:20.022245+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:49:17.964400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:49:17.964400+0000","last_clean_scrub_stamp":"2024-09-16T15:49:17.964400+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T23:43:15.143261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.18","version":"0'0","reported_seq":10,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-09-16T15:49:20.052821+0000","last_change":"2024-09-16T15:49:20.034428+0000","last_active":"2024-09-16T15:49:20.052821+0000","last_peered":"2024-09-16T15:49:20.052821+0000","last_clean":"2024-09-16T15:49:20.052821+0000","last_became_active":"2024-09-16T15:49:20.027556+0000","last_became_peered":"2024-09-16T15:49:20.027556+0000","last_unstale":"2024-09-16T15:49:20.052821+0000","last_undegraded":"2024-09-16T15:49:20.052821+0000","last_fullsized":"2024-09-16T15:49:20.052821+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:49:18.967562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:49:18.967562+0000","last_clean_scrub_stamp":"2024-09-16T15:49:18.967562+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:58:12.260025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.19","version":"0'0","reported_seq":13,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-09-16T15:49:20.014091+0000","last_change":"2024-09-16T15:49:19.018690+0000","last_active":"2024-09-16T15:49:20.014091+0000","last_peered":"2024-09-16T15:49:20.014091+0000","last_clean":"2024-09-16T15:49:20.014091+0000","last_became_active":"2024-09-16T15:49:19.017949+0000","last_became_peered":"2024-09-16T15:49:19.017949+0000","last_unstale":"2024-09-16T15:49:20.014091+0000","last_undegraded":"2024-09-16T15:49:20.014091+0000","last_fullsized":"2024-09-16T15:49:20.014091+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:49:17.964400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:49:17.964400+0000","last_clean_scrub_stamp":"2024-09-16T15:49:17.964400+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T02:22:55.594740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.19","version":"0'0","reported_seq":10,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-09-16T15:49:20.022308+0000","last_change":"2024-09-16T15:49:20.014659+0000","last_active":"2024-09-16T15:49:20.022308+0000","last_peered":"2024-09-16T15:49:20.022308+0000","last_clean":"2024-09-16T15:49:20.022308+0000","last_became_active":"2024-09-16T15:49:20.014342+0000","last_became_peered":"2024-09-16T15:49:20.014342+0000","last_unstale":"2024-09-16T15:49:20.022308+0000","last_undegraded":"2024-09-16T15:49:20.022308+0000","last_fullsized":"2024-09-16T15:49:20.022308+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:49:18.967562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:49:18.967562+0000","last_clean_scrub_stamp":"2024-09-16T15:49:18.967562+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T22:45:04.658721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.18","version":"0'0","reported_seq":13,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-09-16T15:49:19.997156+0000","last_change":"2024-09-16T15:49:19.003568+0000","last_active":"2024-09-16T15:49:19.997156+0000","last_peered":"2024-09-16T15:49:19.997156+0000","last_clean":"2024-09-16T15:49:19.997156+0000","last_became_active":"2024-09-16T15:49:19.003280+0000","last_became_peered":"2024-09-16T15:49:19.003280+0000","last_unstale":"2024-09-16T15:49:19.997156+0000","last_undegraded":"2024-09-16T15:49:19.997156+0000","last_fullsized":"2024-09-16T15:49:19.997156+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:49:17.964400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:49:17.964400+0000","last_clean_scrub_stamp":"2024-09-16T15:49:17.964400+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T18:20:00.931187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.16","version":"0'0","reported_seq":10,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-09-16T15:49:20.035435+0000","last_change":"2024-09-16T15:49:20.031021+0000","last_active":"2024-09-16T15:49:20.035435+0000","last_peered":"2024-09-16T15:49:20.035435+0000","last_clean":"2024-09-16T15:49:20.035435+0000","last_became_active":"2024-09-16T15:49:20.030747+0000","last_became_peered":"2024-09-16T15:49:20.030747+0000","last_unstale":"2024-09-16T15:49:20.035435+0000","last_undegraded":"2024-09-16T15:49:20.035435+0000","last_fullsized":"2024-09-16T15:49:20.035435+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:49:18.967562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:49:18.967562+0000","last_clean_scrub_stamp":"2024-09-16T15:49:18.967562+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:59:13.653253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.17","version":"0'0","reported_seq":13,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-09-16T15:49:20.043994+0000","last_change":"2024-09-16T15:49:19.006709+0000","last_active":"2024-09-16T15:49:20.043994+0000","last_peered":"2024-09-16T15:49:20.043994+0000","last_clean":"2024-09-16T15:49:20.043994+0000","last_became_active":"2024-09-16T15:49:19.005415+0000","last_became_peered":"2024-09-16T15:49:19.005415+0000","last_unstale":"2024-09-16T15:49:20.043994+0000","last_undegraded":"2024-09-16T15:49:20.043994+0000","last_fullsized":"2024-09-16T15:49:20.043994+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:49:17.964400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:49:17.964400+0000","last_clean_scrub_stamp":"2024-09-16T15:49:17.964400+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:31:42.555798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.17","version":"0'0","reported_seq":10,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-09-16T15:49:20.052583+0000","last_change":"2024-09-16T15:49:20.046340+0000","last_active":"2024-09-16T15:49:20.052583+0000","last_peered":"2024-09-16T15:49:20.052583+0000","last_clean":"2024-09-16T15:49:20.052583+0000","last_became_active":"2024-09-16T15:49:20.045340+0000","last_became_peered":"2024-09-16T15:49:20.045340+0000","last_unstale":"2024-09-16T15:49:20.052583+0000","last_undegraded":"2024-09-16T15:49:20.052583+0000","last_fullsized":"2024-09-16T15:49:20.052583+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:49:18.967562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:49:18.967562+0000","last_clean_scrub_stamp":"2024-09-16T15:49:18.967562+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:41:17.054562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.16","version":"0'0","reported_seq":13,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-09-16T15:49:20.009193+0000","last_change":"2024-09-16T15:49:18.999247+0000","last_active":"2024-09-16T15:49:20.009193+0000","last_peered":"2024-09-16T15:49:20.009193+0000","last_clean":"2024-09-16T15:49:20.009193+0000","last_became_active":"2024-09-16T15:49:18.998621+0000","last_became_peered":"2024-09-16T15:49:18.998621+0000","last_unstale":"2024-09-16T15:49:20.009193+0000","last_undegraded":"2024-09-16T15:49:20.009193+0000","last_fullsized":"2024-09-16T15:49:20.009193+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:49:17.964400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:49:17.964400+0000","last_clean_scrub_stamp":"2024-09-16T15:49:17.964400+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T02:26:02.649228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.14","version":"0'0","reported_seq":10,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-09-16T15:49:20.058517+0000","last_change":"2024-09-16T15:49:20.049655+0000","last_active":"2024-09-16T15:49:20.058517+0000","last_peered":"2024-09-16T15:49:20.058517+0000","last_clean":"2024-09-16T15:49:20.058517+0000","last_became_active":"2024-09-16T15:49:20.048533+0000","last_became_peered":"2024-09-16T15:49:20.048533+0000","last_unstale":"2024-09-16T15:49:20.058517+0000","last_undegraded":"2024-09-16T15:49:20.058517+0000","last_fullsized":"2024-09-16T15:49:20.058517+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:49:18.967562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:49:18.967562+0000","last_clean_scrub_stamp":"2024-09-16T15:49:18.967562+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:20:35.281539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.15","version":"118'3","reported_seq":16,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-09-16T15:49:20.031442+0000","last_change":"2024-09-16T15:49:18.994884+0000","last_active":"2024-09-16T15:49:20.031442+0000","last_peered":"2024-09-16T15:49:20.031442+0000","last_clean":"2024-09-16T15:49:20.031442+0000","last_became_active":"2024-09-16T15:49:18.994167+0000","last_became_peered":"2024-09-16T15:49:18.994167+0000","last_unstale":"2024-09-16T15:49:20.031442+0000","last_undegraded":"2024-09-16T15:49:20.031442+0000","last_fullsized":"2024-09-16T15:49:20.031442+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:49:17.964400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:49:17.964400+0000","last_clean_scrub_stamp":"2024-09-16T15:49:17.964400+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:07:21.707520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.15","version":"0'0","reported_seq":10,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-09-16T15:49:20.058744+0000","last_change":"2024-09-16T15:49:20.055174+0000","last_active":"2024-09-16T15:49:20.058744+0000","last_peered":"2024-09-16T15:49:20.058744+0000","last_clean":"2024-09-16T15:49:20.058744+0000","last_became_active":"2024-09-16T15:49:20.054624+0000","last_became_peered":"2024-09-16T15:49:20.054624+0000","last_unstale":"2024-09-16T15:49:20.058744+0000","last_undegraded":"2024-09-16T15:49:20.058744+0000","last_fullsized":"2024-09-16T15:49:20.058744+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:49:18.967562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:49:18.967562+0000","last_clean_scrub_stamp":"2024-09-16T15:49:18.967562+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:35:24.858013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.14","version":"118'6","reported_seq":19,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-09-16T15:49:20.062782+0000","last_change":"2024-09-16T15:49:19.010997+0000","last_active":"2024-09-16T15:49:20.062782+0000","last_peered":"2024-09-16T15:49:20.062782+0000","last_clean":"2024-09-16T15:49:20.062782+0000","last_became_active":"2024-09-16T15:49:19.010201+0000","last_became_peered":"2024-09-16T15:49:19.010201+0000","last_unstale":"2024-09-16T15:49:20.062782+0000","last_undegraded":"2024-09-16T15:49:20.062782+0000","last_fullsized":"2024-09-16T15:49:20.062782+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:49:17.964400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:49:17.964400+0000","last_clean_scrub_stamp":"2024-09-16T15:49:17.964400+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T18:54:40.001665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.13","version":"0'0","reported_seq":10,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-09-16T15:49:20.037239+0000","last_change":"2024-09-16T15:49:20.031568+0000","last_active":"2024-09-16T15:49:20.037239+0000","last_peered":"2024-09-16T15:49:20.037239+0000","last_clean":"2024-09-16T15:49:20.037239+0000","last_became_active":"2024-09-16T15:49:20.031308+0000","last_became_peered":"2024-09-16T15:49:20.031308+0000","last_unstale":"2024-09-16T15:49:20.037239+0000","last_undegraded":"2024-09-16T15:49:20.037239+0000","last_fullsized":"2024-09-16T15:49:20.037239+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:49:18.967562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:49:18.967562+0000","last_clean_scrub_stamp":"2024-09-16T15:49:18.967562+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:58:42.430549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.12","version":"0'0","reported_seq":13,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-09-16T15:49:20.024378+0000","last_change":"2024-09-16T15:49:19.000238+0000","last_active":"2024-09-16T15:49:20.024378+0000","last_peered":"2024-09-16T15:49:20.024378+0000","last_clean":"2024-09-16T15:49:20.024378+0000","last_became_active":"2024-09-16T15:49:18.999837+0000","last_became_peered":"2024-09-16T15:49:18.999837+0000","last_unstale":"2024-09-16T15:49:20.024378+0000","last_undegraded":"2024-09-16T15:49:20.024378+0000","last_fullsized":"2024-09-16T15:49:20.024378+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:49:17.964400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:49:17.964400+0000","last_clean_scrub_stamp":"2024-09-16T15:49:17.964400+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T18:57:09.054608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.10","version":"0'0","reported_seq":10,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-09-16T15:49:20.013249+0000","last_change":"2024-09-16T15:49:20.006545+0000","last_active":"2024-09-16T15:49:20.013249+0000","last_peered":"2024-09-16T15:49:20.013249+0000","last_clean":"2024-09-16T15:49:20.013249+0000","last_became_active":"2024-09-16T15:49:20.005832+0000","last_became_peered":"2024-09-16T15:49:20.005832+0000","last_unstale":"2024-09-16T15:49:20.013249+0000","last_undegraded":"2024-09-16T15:49:20.013249+0000","last_fullsized":"2024-09-16T15:49:20.013249+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:49:18.967562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:49:18.967562+0000","last_clean_scrub_stamp":"2024-09-16T15:49:18.967562+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T03:29:35.271830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.11","version":"0'0","reported_seq":13,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-09-16T15:49:19.997284+0000","last_change":"2024-09-16T15:49:19.010274+0000","last_active":"2024-09-16T15:49:19.997284+0000","last_peered":"2024-09-16T15:49:19.997284+0000","last_clean":"2024-09-16T15:49:19.997284+0000","last_became_active":"2024-09-16T15:49:19.009964+0000","last_became_peered":"2024-09-16T15:49:19.009964+0000","last_unstale":"2024-09-16T15:49:19.997284+0000","last_undegraded":"2024-09-16T15:49:19.997284+0000","last_fullsized":"2024-09-16T15:49:19.997284+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:49:17.964400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:49:17.964400+0000","last_clean_scrub_stamp":"2024-09-16T15:49:17.964400+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:57:29.053710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.11","version":"0'0","reported_seq":10,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-09-16T15:49:20.044676+0000","last_change":"2024-09-16T15:49:20.042183+0000","last_active":"2024-09-16T15:49:20.044676+0000","last_peered":"2024-09-16T15:49:20.044676+0000","last_clean":"2024-09-16T15:49:20.044676+0000","last_became_active":"2024-09-16T15:49:20.032992+0000","last_became_peered":"2024-09-16T15:49:20.032992+0000","last_unstale":"2024-09-16T15:49:20.044676+0000","last_undegraded":"2024-09-16T15:49:20.044676+0000","last_fullsized":"2024-09-16T15:49:20.044676+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:49:18.967562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:49:18.967562+0000","last_clean_scrub_stamp":"2024-09-16T15:49:18.967562+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:13:14.414577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.10","version":"118'8","reported_seq":21,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-09-16T15:49:20.062255+0000","last_change":"2024-09-16T15:49:19.000646+0000","last_active":"2024-09-16T15:49:20.062255+0000","last_peered":"2024-09-16T15:49:20.062255+0000","last_clean":"2024-09-16T15:49:20.062255+0000","last_became_active":"2024-09-16T15:49:18.999926+0000","last_became_peered":"2024-09-16T15:49:18.999926+0000","last_unstale":"2024-09-16T15:49:20.062255+0000","last_undegraded":"2024-09-16T15:49:20.062255+0000","last_fullsized":"2024-09-16T15:49:20.062255+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:49:17.964400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:49:17.964400+0000","last_clean_scrub_stamp":"2024-09-16T15:49:17.964400+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:43:38.769049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.e","version":"0'0","reported_seq":10,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-09-16T15:49:20.026542+0000","last_change":"2024-09-16T15:49:20.021838+0000","last_active":"2024-09-16T15:49:20.026542+0000","last_peered":"2024-09-16T15:49:20.026542+0000","last_clean":"2024-09-16T15:49:20.026542+0000","last_became_active":"2024-09-16T15:49:20.021142+0000","last_became_peered":"2024-09-16T15:49:20.021142+0000","last_unstale":"2024-09-16T15:49:20.026542+0000","last_undegraded":"2024-09-16T15:49:20.026542+0000","last_fullsized":"2024-09-16T15:49:20.026542+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:49:18.967562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:49:18.967562+0000","last_clean_scrub_stamp":"2024-09-16T15:49:18.967562+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:45:45.358751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.f","version":"118'1","reported_seq":14,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-09-16T15:49:20.056676+0000","last_change":"2024-09-16T15:49:19.011934+0000","last_active":"2024-09-16T15:49:20.056676+0000","last_peered":"2024-09-16T15:49:20.056676+0000","last_clean":"2024-09-16T15:49:20.056676+0000","last_became_active":"2024-09-16T15:49:19.011176+0000","last_became_peered":"2024-09-16T15:49:19.011176+0000","last_unstale":"2024-09-16T15:49:20.056676+0000","last_undegraded":"2024-09-16T15:49:20.056676+0000","last_fullsized":"2024-09-16T15:49:20.056676+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:49:17.964400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:49:17.964400+0000","last_clean_scrub_stamp":"2024-09-16T15:49:17.964400+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T22:51:13.621888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.f","version":"0'0","reported_seq":10,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-09-16T15:49:20.053874+0000","last_change":"2024-09-16T15:49:20.047290+0000","last_active":"2024-09-16T15:49:20.053874+0000","last_peered":"2024-09-16T15:49:20.053874+0000","last_clean":"2024-09-16T15:49:20.053874+0000","last_became_active":"2024-09-16T15:49:20.045875+0000","last_became_peered":"2024-09-16T15:49:20.045875+0000","last_unstale":"2024-09-16T15:49:20.053874+0000","last_undegraded":"2024-09-16T15:49:20.053874+0000","last_fullsized":"2024-09-16T15:49:20.053874+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:49:18.967562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:49:18.967562+0000","last_clean_scrub_stamp":"2024-09-16T15:49:18.967562+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:55:07.911006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.e","version":"118'2","reported_seq":15,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-09-16T15:49:20.036056+0000","last_change":"2024-09-16T15:49:19.006540+0000","last_active":"2024-09-16T15:49:20.036056+0000","last_peered":"2024-09-16T15:49:20.036056+0000","last_clean":"2024-09-16T15:49:20.036056+0000","last_became_active":"2024-09-16T15:49:19.006157+0000","last_became_peered":"2024-09-16T15:49:19.006157+0000","last_unstale":"2024-09-16T15:49:20.036056+0000","last_undegraded":"2024-09-16T15:49:20.036056+0000","last_fullsized":"2024-09-16T15:49:20.036056+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:49:17.964400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:49:17.964400+0000","last_clean_scrub_stamp":"2024-09-16T15:49:17.964400+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T23:32:33.274644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.c","version":"0'0","reported_seq":10,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-09-16T15:49:20.023091+0000","last_change":"2024-09-16T15:49:20.019570+0000","last_active":"2024-09-16T15:49:20.023091+0000","last_peered":"2024-09-16T15:49:20.023091+0000","last_clean":"2024-09-16T15:49:20.023091+0000","last_became_active":"2024-09-16T15:49:20.019303+0000","last_became_peered":"2024-09-16T15:49:20.019303+0000","last_unstale":"2024-09-16T15:49:20.023091+0000","last_undegraded":"2024-09-16T15:49:20.023091+0000","last_fullsized":"2024-09-16T15:49:20.023091+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:49:18.967562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:49:18.967562+0000","last_clean_scrub_stamp":"2024-09-16T15:49:18.967562+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T22:35:03.465884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.d","version":"118'1","reported_seq":14,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-09-16T15:49:20.028696+0000","last_change":"2024-09-16T15:49:19.005048+0000","last_active":"2024-09-16T15:49:20.028696+0000","last_peered":"2024-09-16T15:49:20.028696+0000","last_clean":"2024-09-16T15:49:20.028696+0000","last_became_active":"2024-09-16T15:49:19.003761+0000","last_became_peered":"2024-09-16T15:49:19.003761+0000","last_unstale":"2024-09-16T15:49:20.028696+0000","last_undegraded":"2024-09-16T15:49:20.028696+0000","last_fullsized":"2024-09-16T15:49:20.028696+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:49:17.964400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:49:17.964400+0000","last_clean_scrub_stamp":"2024-09-16T15:49:17.964400+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T00:06:54.427637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.d","version":"0'0","reported_seq":10,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-09-16T15:49:20.022226+0000","last_change":"2024-09-16T15:49:20.014483+0000","last_active":"2024-09-16T15:49:20.022226+0000","last_peered":"2024-09-16T15:49:20.022226+0000","last_clean":"2024-09-16T15:49:20.022226+0000","last_became_active":"2024-09-16T15:49:20.014250+0000","last_became_peered":"2024-09-16T15:49:20.014250+0000","last_unstale":"2024-09-16T15:49:20.022226+0000","last_undegraded":"2024-09-16T15:49:20.022226+0000","last_fullsized":"2024-09-16T15:49:20.022226+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:49:18.967562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:49:18.967562+0000","last_clean_scrub_stamp":"2024-09-16T15:49:18.967562+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T02:54:45.265261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.c","version":"0'0","reported_seq":13,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-09-16T15:49:20.019003+0000","last_change":"2024-09-16T15:49:18.993103+0000","last_active":"2024-09-16T15:49:20.019003+0000","last_peered":"2024-09-16T15:49:20.019003+0000","last_clean":"2024-09-16T15:49:20.019003+0000","last_became_active":"2024-09-16T15:49:18.992354+0000","last_became_peered":"2024-09-16T15:49:18.992354+0000","last_unstale":"2024-09-16T15:49:20.019003+0000","last_undegraded":"2024-09-16T15:49:20.019003+0000","last_fullsized":"2024-09-16T15:49:20.019003+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:49:17.964400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:49:17.964400+0000","last_clean_scrub_stamp":"2024-09-16T15:49:17.964400+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:36.121363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.a","version":"0'0","reported_seq":10,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-09-16T15:49:20.026407+0000","last_change":"2024-09-16T15:49:20.024410+0000","last_active":"2024-09-16T15:49:20.026407+0000","last_peered":"2024-09-16T15:49:20.026407+0000","last_clean":"2024-09-16T15:49:20.026407+0000","last_became_active":"2024-09-16T15:49:20.018974+0000","last_became_peered":"2024-09-16T15:49:20.018974+0000","last_unstale":"2024-09-16T15:49:20.026407+0000","last_undegraded":"2024-09-16T15:49:20.026407+0000","last_fullsized":"2024-09-16T15:49:20.026407+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:49:18.967562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:49:18.967562+0000","last_clean_scrub_stamp":"2024-09-16T15:49:18.967562+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:35:04.846769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.b","version":"118'2","reported_seq":15,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-09-16T15:49:20.046228+0000","last_change":"2024-09-16T15:49:18.991144+0000","last_active":"2024-09-16T15:49:20.046228+0000","last_peered":"2024-09-16T15:49:20.046228+0000","last_clean":"2024-09-16T15:49:20.046228+0000","last_became_active":"2024-09-16T15:49:18.990393+0000","last_became_peered":"2024-09-16T15:49:18.990393+0000","last_unstale":"2024-09-16T15:49:20.046228+0000","last_undegraded":"2024-09-16T15:49:20.046228+0000","last_fullsized":"2024-09-16T15:49:20.046228+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:49:17.964400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:49:17.964400+0000","last_clean_scrub_stamp":"2024-09-16T15:49:17.964400+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:32:40.744418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.b","version":"0'0","reported_seq":10,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-09-16T15:49:20.038875+0000","last_change":"2024-09-16T15:49:20.033198+0000","last_active":"2024-09-16T15:49:20.038875+0000","last_peered":"2024-09-16T15:49:20.038875+0000","last_clean":"2024-09-16T15:49:20.038875+0000","last_became_active":"2024-09-16T15:49:20.029975+0000","last_became_peered":"2024-09-16T15:49:20.029975+0000","last_unstale":"2024-09-16T15:49:20.038875+0000","last_undegraded":"2024-09-16T15:49:20.038875+0000","last_fullsized":"2024-09-16T15:49:20.038875+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:49:18.967562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:49:18.967562+0000","last_clean_scrub_stamp":"2024-09-16T15:49:18.967562+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T18:06:59.994313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.a","version":"0'0","reported_seq":13,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-09-16T15:49:20.008844+0000","last_change":"2024-09-16T15:49:18.981991+0000","last_active":"2024-09-16T15:49:20.008844+0000","last_peered":"2024-09-16T15:49:20.008844+0000","last_clean":"2024-09-16T15:49:20.008844+0000","last_became_active":"2024-09-16T15:49:18.981604+0000","last_became_peered":"2024-09-16T15:49:18.981604+0000","last_unstale":"2024-09-16T15:49:20.008844+0000","last_undegraded":"2024-09-16T15:49:20.008844+0000","last_fullsized":"2024-09-16T15:49:20.008844+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:49:17.964400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:49:17.964400+0000","last_clean_scrub_stamp":"2024-09-16T15:49:17.964400+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:59:17.318616+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.8","version":"0'0","reported_seq":10,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-09-16T15:49:20.034545+0000","last_change":"2024-09-16T15:49:20.030377+0000","last_active":"2024-09-16T15:49:20.034545+0000","last_peered":"2024-09-16T15:49:20.034545+0000","last_clean":"2024-09-16T15:49:20.034545+0000","last_became_active":"2024-09-16T15:49:20.029971+0000","last_became_peered":"2024-09-16T15:49:20.029971+0000","last_unstale":"2024-09-16T15:49:20.034545+0000","last_undegraded":"2024-09-16T15:49:20.034545+0000","last_fullsized":"2024-09-16T15:49:20.034545+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:49:18.967562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:49:18.967562+0000","last_clean_scrub_stamp":"2024-09-16T15:49:18.967562+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:11:43.275240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.9","version":"0'0","reported_seq":13,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-09-16T15:49:20.009045+0000","last_change":"2024-09-16T15:49:18.987931+0000","last_active":"2024-09-16T15:49:20.009045+0000","last_peered":"2024-09-16T15:49:20.009045+0000","last_clean":"2024-09-16T15:49:20.009045+0000","last_became_active":"2024-09-16T15:49:18.987524+0000","last_became_peered":"2024-09-16T15:49:18.987524+0000","last_unstale":"2024-09-16T15:49:20.009045+0000","last_undegraded":"2024-09-16T15:49:20.009045+0000","last_fullsized":"2024-09-16T15:49:20.009045+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:49:17.964400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:49:17.964400+0000","last_clean_scrub_stamp":"2024-09-16T15:49:17.964400+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:28:01.153450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.9","version":"0'0","reported_seq":10,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-09-16T15:49:20.059149+0000","last_change":"2024-09-16T15:49:20.056767+0000","last_active":"2024-09-16T15:49:20.059149+0000","last_peered":"2024-09-16T15:49:20.059149+0000","last_clean":"2024-09-16T15:49:20.059149+0000","last_became_active":"2024-09-16T15:49:20.048591+0000","last_became_peered":"2024-09-16T15:49:20.048591+0000","last_unstale":"2024-09-16T15:49:20.059149+0000","last_undegraded":"2024-09-16T15:49:20.059149+0000","last_fullsized":"2024-09-16T15:49:20.059149+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:49:18.967562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:49:18.967562+0000","last_clean_scrub_stamp":"2024-09-16T15:49:18.967562+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T02:01:48.218260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.8","version":"0'0","reported_seq":13,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-09-16T15:49:20.008200+0000","last_change":"2024-09-16T15:49:19.006665+0000","last_active":"2024-09-16T15:49:20.008200+0000","last_peered":"2024-09-16T15:49:20.008200+0000","last_clean":"2024-09-16T15:49:20.008200+0000","last_became_active":"2024-09-16T15:49:19.006118+0000","last_became_peered":"2024-09-16T15:49:19.006118+0000","last_unstale":"2024-09-16T15:49:20.008200+0000","last_undegraded":"2024-09-16T15:49:20.008200+0000","last_fullsized":"2024-09-16T15:49:20.008200+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:49:17.964400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:49:17.964400+0000","last_clean_scrub_stamp":"2024-09-16T15:49:17.964400+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T00:00:43.698889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.6","version":"0'0","reported_seq":10,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-09-16T15:49:20.038442+0000","last_change":"2024-09-16T15:49:20.029394+0000","last_active":"2024-09-16T15:49:20.038442+0000","last_peered":"2024-09-16T15:49:20.038442+0000","last_clean":"2024-09-16T15:49:20.038442+0000","last_became_active":"2024-09-16T15:49:20.023195+0000","last_became_peered":"2024-09-16T15:49:20.023195+0000","last_unstale":"2024-09-16T15:49:20.038442+0000","last_undegraded":"2024-09-16T15:49:20.038442+0000","last_fullsized":"2024-09-16T15:49:20.038442+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:49:18.967562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:49:18.967562+0000","last_clean_scrub_stamp":"2024-09-16T15:49:18.967562+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:19:27.788452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.7","version":"118'2","reported_seq":15,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-09-16T15:49:20.047609+0000","last_change":"2024-09-16T15:49:19.000107+0000","last_active":"2024-09-16T15:49:20.047609+0000","last_peered":"2024-09-16T15:49:20.047609+0000","last_clean":"2024-09-16T15:49:20.047609+0000","last_became_active":"2024-09-16T15:49:18.999087+0000","last_became_peered":"2024-09-16T15:49:18.999087+0000","last_unstale":"2024-09-16T15:49:20.047609+0000","last_undegraded":"2024-09-16T15:49:20.047609+0000","last_fullsized":"2024-09-16T15:49:20.047609+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:49:17.964400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:49:17.964400+0000","last_clean_scrub_stamp":"2024-09-16T15:49:17.964400+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:00:34.617481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.7","version":"0'0","reported_seq":10,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-09-16T15:49:20.026264+0000","last_change":"2024-09-16T15:49:20.023965+0000","last_active":"2024-09-16T15:49:20.026264+0000","last_peered":"2024-09-16T15:49:20.026264+0000","last_clean":"2024-09-16T15:49:20.026264+0000","last_became_active":"2024-09-16T15:49:20.022021+0000","last_became_peered":"2024-09-16T15:49:20.022021+0000","last_unstale":"2024-09-16T15:49:20.026264+0000","last_undegraded":"2024-09-16T15:49:20.026264+0000","last_fullsized":"2024-09-16T15:49:20.026264+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:49:18.967562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:49:18.967562+0000","last_clean_scrub_stamp":"2024-09-16T15:49:18.967562+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:13:14.966056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.6","version":"0'0","reported_seq":13,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-09-16T15:49:20.011948+0000","last_change":"2024-09-16T15:49:19.018906+0000","last_active":"2024-09-16T15:49:20.011948+0000","last_peered":"2024-09-16T15:49:20.011948+0000","last_clean":"2024-09-16T15:49:20.011948+0000","last_became_active":"2024-09-16T15:49:19.018066+0000","last_became_peered":"2024-09-16T15:49:19.018066+0000","last_unstale":"2024-09-16T15:49:20.011948+0000","last_undegraded":"2024-09-16T15:49:20.011948+0000","last_fullsized":"2024-09-16T15:49:20.011948+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:49:17.964400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:49:17.964400+0000","last_clean_scrub_stamp":"2024-09-16T15:49:17.964400+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:32.785433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.4","version":"0'0","reported_seq":10,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-09-16T15:49:20.045390+0000","last_change":"2024-09-16T15:49:20.041975+0000","last_active":"2024-09-16T15:49:20.045390+0000","last_peered":"2024-09-16T15:49:20.045390+0000","last_clean":"2024-09-16T15:49:20.045390+0000","last_became_active":"2024-09-16T15:49:20.041602+0000","last_became_peered":"2024-09-16T15:49:20.041602+0000","last_unstale":"2024-09-16T15:49:20.045390+0000","last_undegraded":"2024-09-16T15:49:20.045390+0000","last_fullsized":"2024-09-16T15:49:20.045390+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:49:18.967562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:49:18.967562+0000","last_clean_scrub_stamp":"2024-09-16T15:49:18.967562+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T01:11:05.029917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.5","version":"0'0","reported_seq":13,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-09-16T15:49:19.998114+0000","last_change":"2024-09-16T15:49:19.002146+0000","last_active":"2024-09-16T15:49:19.998114+0000","last_peered":"2024-09-16T15:49:19.998114+0000","last_clean":"2024-09-16T15:49:19.998114+0000","last_became_active":"2024-09-16T15:49:18.997889+0000","last_became_peered":"2024-09-16T15:49:18.997889+0000","last_unstale":"2024-09-16T15:49:19.998114+0000","last_undegraded":"2024-09-16T15:49:19.998114+0000","last_fullsized":"2024-09-16T15:49:19.998114+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:49:17.964400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:49:17.964400+0000","last_clean_scrub_stamp":"2024-09-16T15:49:17.964400+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T22:05:45.261671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.5","version":"0'0","reported_seq":10,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-09-16T15:49:20.036883+0000","last_change":"2024-09-16T15:49:20.021089+0000","last_active":"2024-09-16T15:49:20.036883+0000","last_peered":"2024-09-16T15:49:20.036883+0000","last_clean":"2024-09-16T15:49:20.036883+0000","last_became_active":"2024-09-16T15:49:20.020382+0000","last_became_peered":"2024-09-16T15:49:20.020382+0000","last_unstale":"2024-09-16T15:49:20.036883+0000","last_undegraded":"2024-09-16T15:49:20.036883+0000","last_fullsized":"2024-09-16T15:49:20.036883+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:49:18.967562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:49:18.967562+0000","last_clean_scrub_stamp":"2024-09-16T15:49:18.967562+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:15:25.349686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.4","version":"0'0","reported_seq":13,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-09-16T15:49:20.004842+0000","last_change":"2024-09-16T15:49:18.998714+0000","last_active":"2024-09-16T15:49:20.004842+0000","last_peered":"2024-09-16T15:49:20.004842+0000","last_clean":"2024-09-16T15:49:20.004842+0000","last_became_active":"2024-09-16T15:49:18.998264+0000","last_became_peered":"2024-09-16T15:49:18.998264+0000","last_unstale":"2024-09-16T15:49:20.004842+0000","last_undegraded":"2024-09-16T15:49:20.004842+0000","last_fullsized":"2024-09-16T15:49:20.004842+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:49:17.964400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:49:17.964400+0000","last_clean_scrub_stamp":"2024-09-16T15:49:17.964400+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:45:09.866648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.2","version":"0'0","reported_seq":10,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-09-16T15:49:20.058717+0000","last_change":"2024-09-16T15:49:20.055626+0000","last_active":"2024-09-16T15:49:20.058717+0000","last_peered":"2024-09-16T15:49:20.058717+0000","last_clean":"2024-09-16T15:49:20.058717+0000","last_became_active":"2024-09-16T15:49:20.055267+0000","last_became_peered":"2024-09-16T15:49:20.055267+0000","last_unstale":"2024-09-16T15:49:20.058717+0000","last_undegraded":"2024-09-16T15:49:20.058717+0000","last_fullsized":"2024-09-16T15:49:20.058717+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:49:18.967562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:49:18.967562+0000","last_clean_scrub_stamp":"2024-09-16T15:49:18.967562+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:43:02.183966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.3","version":"0'0","reported_seq":13,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-09-16T15:49:19.997007+0000","last_change":"2024-09-16T15:49:19.010214+0000","last_active":"2024-09-16T15:49:19.997007+0000","last_peered":"2024-09-16T15:49:19.997007+0000","last_clean":"2024-09-16T15:49:19.997007+0000","last_became_active":"2024-09-16T15:49:19.009877+0000","last_became_peered":"2024-09-16T15:49:19.009877+0000","last_unstale":"2024-09-16T15:49:19.997007+0000","last_undegraded":"2024-09-16T15:49:19.997007+0000","last_fullsized":"2024-09-16T15:49:19.997007+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:49:17.964400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:49:17.964400+0000","last_clean_scrub_stamp":"2024-09-16T15:49:17.964400+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:34:09.087537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.3","version":"0'0","reported_seq":10,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-09-16T15:49:20.047499+0000","last_change":"2024-09-16T15:49:20.044650+0000","last_active":"2024-09-16T15:49:20.047499+0000","last_peered":"2024-09-16T15:49:20.047499+0000","last_clean":"2024-09-16T15:49:20.047499+0000","last_became_active":"2024-09-16T15:49:20.043082+0000","last_became_peered":"2024-09-16T15:49:20.043082+0000","last_unstale":"2024-09-16T15:49:20.047499+0000","last_undegraded":"2024-09-16T15:49:20.047499+0000","last_fullsized":"2024-09-16T15:49:20.047499+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:49:18.967562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:49:18.967562+0000","last_clean_scrub_stamp":"2024-09-16T15:49:18.967562+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T22:55:49.755222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.2","version":"0'0","reported_seq":13,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-09-16T15:49:20.036237+0000","last_change":"2024-09-16T15:49:19.016650+0000","last_active":"2024-09-16T15:49:20.036237+0000","last_peered":"2024-09-16T15:49:20.036237+0000","last_clean":"2024-09-16T15:49:20.036237+0000","last_became_active":"2024-09-16T15:49:19.015887+0000","last_became_peered":"2024-09-16T15:49:19.015887+0000","last_unstale":"2024-09-16T15:49:20.036237+0000","last_undegraded":"2024-09-16T15:49:20.036237+0000","last_fullsized":"2024-09-16T15:49:20.036237+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:49:17.964400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:49:17.964400+0000","last_clean_scrub_stamp":"2024-09-16T15:49:17.964400+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:42:48.571248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.0","version":"0'0","reported_seq":10,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-09-16T15:49:20.045877+0000","last_change":"2024-09-16T15:49:20.043111+0000","last_active":"2024-09-16T15:49:20.045877+0000","last_peered":"2024-09-16T15:49:20.045877+0000","last_clean":"2024-09-16T15:49:20.045877+0000","last_became_active":"2024-09-16T15:49:20.042852+0000","last_became_peered":"2024-09-16T15:49:20.042852+0000","last_unstale":"2024-09-16T15:49:20.045877+0000","last_undegraded":"2024-09-16T15:49:20.045877+0000","last_fullsized":"2024-09-16T15:49:20.045877+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:49:18.967562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:49:18.967562+0000","last_clean_scrub_stamp":"2024-09-16T15:49:18.967562+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:02:03.068228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.1","version":"0'0","reported_seq":13,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-09-16T15:49:20.009202+0000","last_change":"2024-09-16T15:49:18.999499+0000","last_active":"2024-09-16T15:49:20.009202+0000","last_peered":"2024-09-16T15:49:20.009202+0000","last_clean":"2024-09-16T15:49:20.009202+0000","last_became_active":"2024-09-16T15:49:18.999058+0000","last_became_peered":"2024-09-16T15:49:18.999058+0000","last_unstale":"2024-09-16T15:49:20.009202+0000","last_undegraded":"2024-09-16T15:49:20.009202+0000","last_fullsized":"2024-09-16T15:49:20.009202+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:49:17.964400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:49:17.964400+0000","last_clean_scrub_stamp":"2024-09-16T15:49:17.964400+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:56:50.721641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.1","version":"0'0","reported_seq":10,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-09-16T15:49:20.046724+0000","last_change":"2024-09-16T15:49:20.045786+0000","last_active":"2024-09-16T15:49:20.046724+0000","last_peered":"2024-09-16T15:49:20.046724+0000","last_clean":"2024-09-16T15:49:20.046724+0000","last_became_active":"2024-09-16T15:49:20.045399+0000","last_became_peered":"2024-09-16T15:49:20.045399+0000","last_unstale":"2024-09-16T15:49:20.046724+0000","last_undegraded":"2024-09-16T15:49:20.046724+0000","last_fullsized":"2024-09-16T15:49:20.046724+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:49:18.967562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:49:18.967562+0000","last_clean_scrub_stamp":"2024-09-16T15:49:18.967562+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T18:10:58.632085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.0","version":"0'0","reported_seq":13,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-09-16T15:49:20.008451+0000","last_change":"2024-09-16T15:49:19.004933+0000","last_active":"2024-09-16T15:49:20.008451+0000","last_peered":"2024-09-16T15:49:20.008451+0000","last_clean":"2024-09-16T15:49:20.008451+0000","last_became_active":"2024-09-16T15:49:19.004577+0000","last_became_peered":"2024-09-16T15:49:19.004577+0000","last_unstale":"2024-09-16T15:49:20.008451+0000","last_undegraded":"2024-09-16T15:49:20.008451+0000","last_fullsized":"2024-09-16T15:49:20.008451+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:49:17.964400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:49:17.964400+0000","last_clean_scrub_stamp":"2024-09-16T15:49:17.964400+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:59:29.457235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.12","version":"0'0","reported_seq":10,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-09-16T15:49:20.058332+0000","last_change":"2024-09-16T15:49:20.049202+0000","last_active":"2024-09-16T15:49:20.058332+0000","last_peered":"2024-09-16T15:49:20.058332+0000","last_clean":"2024-09-16T15:49:20.058332+0000","last_became_active":"2024-09-16T15:49:20.048362+0000","last_became_peered":"2024-09-16T15:49:20.048362+0000","last_unstale":"2024-09-16T15:49:20.058332+0000","last_undegraded":"2024-09-16T15:49:20.058332+0000","last_fullsized":"2024-09-16T15:49:20.058332+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:49:18.967562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:49:18.967562+0000","last_clean_scrub_stamp":"2024-09-16T15:49:18.967562+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:11:48.910216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.13","version":"118'2","reported_seq":15,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-09-16T15:49:20.049481+0000","last_change":"2024-09-16T15:49:19.014597+0000","last_active":"2024-09-16T15:49:20.049481+0000","last_peered":"2024-09-16T15:49:20.049481+0000","last_clean":"2024-09-16T15:49:20.049481+0000","last_became_active":"2024-09-16T15:49:19.010375+0000","last_became_peered":"2024-09-16T15:49:19.010375+0000","last_unstale":"2024-09-16T15:49:20.049481+0000","last_undegraded":"2024-09-16T15:49:20.049481+0000","last_fullsized":"2024-09-16T15:49:20.049481+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:49:17.964400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:49:17.964400+0000","last_clean_scrub_stamp":"2024-09-16T15:49:17.964400+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T00:21:03.027521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"73'79","reported_seq":319,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-09-16T15:49:20.006250+0000","last_change":"2024-09-16T15:42:40.881820+0000","last_active":"2024-09-16T15:49:20.006250+0000","last_peered":"2024-09-16T15:49:20.006250+0000","last_clean":"2024-09-16T15:49:20.006250+0000","last_became_active":"2024-09-16T15:42:40.879965+0000","last_became_peered":"2024-09-16T15:42:40.879965+0000","last_unstale":"2024-09-16T15:49:20.006250+0000","last_undegraded":"2024-09-16T15:49:20.006250+0000","last_fullsized":"2024-09-16T15:49:20.006250+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:42:39.728762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:42:39.728762+0000","last_clean_scrub_stamp":"2024-09-16T15:42:39.728762+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:22:18.768932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":19,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":18,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79,"ondisk_log_size":79,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542228,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33192,"kb_used_data":2004,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338648,"statfs":{"total":96636764160,"available":96602775552,"internally_reserved":0,"allocated":2052096,"data_stored":1230822,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542227,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32636,"kb_used_data":1440,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339204,"statfs":{"total":96636764160,"available":96603344896,"internally_reserved":0,"allocated":1474560,"data_stored":647347,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":5,"up_from":14,"seq":60129542228,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32616,"kb_used_data":1424,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339224,"statfs":{"total":96636764160,"available":96603365376,"internally_reserved":0,"allocated":1458176,"data_stored":640454,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542226,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32612,"kb_used_data":1424,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339228,"statfs":{"total":96636764160,"available":96603369472,"internally_reserved":0,"allocated":1458176,"data_stored":640454,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":3,"up_from":14,"seq":60129542227,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32616,"kb_used_data":1424,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339224,"statfs":{"total":96636764160,"available":96603365376,"internally_reserved":0,"allocated":1458176,"data_stored":640454,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":14,"seq":60129542227,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32616,"kb_used_data":1424,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339224,"statfs":{"total":96636764160,"available":96603365376,"internally_reserved":0,"allocated":1458176,"data_stored":640454,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":14,"seq":60129542229,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32616,"kb_used_data":1424,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339224,"statfs":{"total":96636764160,"available":96603365376,"internally_reserved":0,"allocated":1458176,"data_stored":640454,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":14,"seq":60129542229,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33192,"kb_used_data":2004,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338648,"statfs":{"total":96636764160,"available":96602775552,"internally_reserved":0,"allocated":2052096,"data_stored":1230822,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":2250,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T15:49:24.100 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T15:49:24.270 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:24.267+0000 7f38a3b4d640 1 -- 172.21.15.123:0/3638303528 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f389c15bca0 msgr2=0x7f389c0d0630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:24.270 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:24.267+0000 7f38a3b4d640 1 --2- 172.21.15.123:0/3638303528 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f389c15bca0 0x7f389c0d0630 secure :-1 s=READY pgs=1267 cs=0 l=1 rev1=1 crypto rx=0x7f389c06df40 tx=0x7f389801c540 comp rx=0 tx=0).stop 2024-09-16T15:49:24.270 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:24.267+0000 7f38a3b4d640 1 -- 172.21.15.123:0/3638303528 shutdown_connections 2024-09-16T15:49:24.270 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:24.267+0000 7f38a3b4d640 1 --2- 172.21.15.123:0/3638303528 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f389c074430 0x7f389c0ca160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:24.270 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:24.267+0000 7f38a3b4d640 1 --2- 172.21.15.123:0/3638303528 >> v2:172.21.15.184:3300/0 conn(0x7f389c0726c0 0x7f389c073ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:24.270 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:24.267+0000 7f38a3b4d640 1 --2- 172.21.15.123:0/3638303528 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f389c15bca0 0x7f389c0d0630 unknown :-1 s=CLOSED pgs=1267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:24.270 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:24.267+0000 7f38a3b4d640 1 -- 172.21.15.123:0/3638303528 >> 172.21.15.123:0/3638303528 conn(0x7f389c0cfb10 msgr2=0x7f389c0cff20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:49:24.271 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:24.267+0000 7f38a3b4d640 1 -- 172.21.15.123:0/3638303528 shutdown_connections 2024-09-16T15:49:24.271 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:24.267+0000 7f38a3b4d640 1 -- 172.21.15.123:0/3638303528 wait complete. 2024-09-16T15:49:24.271 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:24.267+0000 7f38a3b4d640 1 Processor -- start 2024-09-16T15:49:24.271 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:24.267+0000 7f38a3b4d640 1 -- start start 2024-09-16T15:49:24.271 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:24.267+0000 7f38a3b4d640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f389c0726c0 0x7f389c0c7a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:49:24.271 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:24.267+0000 7f38a3b4d640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f389c074430 0x7f389c1553f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:49:24.271 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:24.267+0000 7f38a3b4d640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f389c15bca0 0x7f389c156550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:49:24.271 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:24.267+0000 7f38a3b4d640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f389c06eca0 con 0x7f389c074430 2024-09-16T15:49:24.271 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:24.267+0000 7f38a3b4d640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f389c06eb20 con 0x7f389c15bca0 2024-09-16T15:49:24.271 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:24.267+0000 7f38a3b4d640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f389c06ee20 con 0x7f389c0726c0 2024-09-16T15:49:24.271 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:24.267+0000 7f38a10e8640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f389c074430 0x7f389c1553f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:49:24.271 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:24.267+0000 7f38a10e8640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f389c074430 0x7f389c1553f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:35440/0 (socket says 172.21.15.123:35440) 2024-09-16T15:49:24.271 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:24.267+0000 7f38a10e8640 1 -- 172.21.15.123:0/4198831049 learned_addr learned my addr 172.21.15.123:0/4198831049 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:49:24.271 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:24.271+0000 7f38a18e9640 1 --2- 172.21.15.123:0/4198831049 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f389c0726c0 0x7f389c0c7a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:49:24.272 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:24.271+0000 7f38a20ea640 1 --2- 172.21.15.123:0/4198831049 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f389c15bca0 0x7f389c156550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:49:24.272 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:24.271+0000 7f38a10e8640 1 -- 172.21.15.123:0/4198831049 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f389c0726c0 msgr2=0x7f389c0c7a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:24.272 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:24.271+0000 7f38a10e8640 1 --2- 172.21.15.123:0/4198831049 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f389c0726c0 0x7f389c0c7a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:24.272 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:24.271+0000 7f38a10e8640 1 -- 172.21.15.123:0/4198831049 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f389c15bca0 msgr2=0x7f389c156550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:24.272 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:24.271+0000 7f38a10e8640 1 --2- 172.21.15.123:0/4198831049 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f389c15bca0 0x7f389c156550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:24.272 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:24.271+0000 7f38a10e8640 1 -- 172.21.15.123:0/4198831049 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f389c0c7f70 con 0x7f389c074430 2024-09-16T15:49:24.272 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:24.271+0000 7f38a18e9640 1 --2- 172.21.15.123:0/4198831049 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f389c0726c0 0x7f389c0c7a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T15:49:24.272 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:24.271+0000 7f38a10e8640 1 --2- 172.21.15.123:0/4198831049 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f389c074430 0x7f389c1553f0 secure :-1 s=READY pgs=1268 cs=0 l=1 rev1=1 crypto rx=0x7f389000c940 tx=0x7f389000ce00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:49:24.272 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:24.271+0000 7f388e7fc640 1 -- 172.21.15.123:0/4198831049 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3890007bb0 con 0x7f389c074430 2024-09-16T15:49:24.272 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:24.271+0000 7f388e7fc640 1 -- 172.21.15.123:0/4198831049 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f389000f450 con 0x7f389c074430 2024-09-16T15:49:24.272 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:24.271+0000 7f38a3b4d640 1 -- 172.21.15.123:0/4198831049 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f389c156f70 con 0x7f389c074430 2024-09-16T15:49:24.272 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:24.271+0000 7f388e7fc640 1 -- 172.21.15.123:0/4198831049 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3890007e60 con 0x7f389c074430 2024-09-16T15:49:24.272 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:24.271+0000 7f38a3b4d640 1 -- 172.21.15.123:0/4198831049 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f389c15d9b0 con 0x7f389c074430 2024-09-16T15:49:24.272 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:24.271+0000 7f38a3b4d640 1 -- 172.21.15.123:0/4198831049 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f389c0ca160 con 0x7f389c074430 2024-09-16T15:49:24.273 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:24.271+0000 7f388e7fc640 1 -- 172.21.15.123:0/4198831049 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 19) ==== 97488+0+0 (secure 0 0 0) 0x7f389000f8a0 con 0x7f389c074430 2024-09-16T15:49:24.273 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:24.271+0000 7f388e7fc640 1 --2- 172.21.15.123:0/4198831049 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f3880073340 0x7f3880075800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:49:24.273 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:24.271+0000 7f388e7fc640 1 -- 172.21.15.123:0/4198831049 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(119..119 src has 1..119) ==== 5737+0+0 (secure 0 0 0) 0x7f3890095060 con 0x7f389c074430 2024-09-16T15:49:24.273 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:24.271+0000 7f38a18e9640 1 --2- 172.21.15.123:0/4198831049 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f3880073340 0x7f3880075800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:49:24.274 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:24.271+0000 7f38a18e9640 1 --2- 172.21.15.123:0/4198831049 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f3880073340 0x7f3880075800 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f38980af000 tx=0x7f389801ad80 comp rx=0 tx=0).ready entity=mgr.4150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:49:24.276 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:24.275+0000 7f388e7fc640 1 -- 172.21.15.123:0/4198831049 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f3890014030 con 0x7f389c074430 2024-09-16T15:49:24.433 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:24.431+0000 7f38a3b4d640 1 -- 172.21.15.123:0/4198831049 --> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f389c0700e0 con 0x7f3880073340 2024-09-16T15:49:24.441 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:24.443+0000 7f388e7fc640 1 -- 172.21.15.123:0/4198831049 <== mgr.4150 v2:172.21.15.123:6836/2464759676 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+252471 (secure 0 0 0) 0x7f389c0700e0 con 0x7f3880073340 2024-09-16T15:49:24.444 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-09-16T15:49:24.445 INFO:teuthology.orchestra.run.smithi123.stderr:dumped all 2024-09-16T15:49:24.446 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:24.447+0000 7f38a3b4d640 1 -- 172.21.15.123:0/4198831049 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f3880073340 msgr2=0x7f3880075800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:24.446 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:24.447+0000 7f38a3b4d640 1 --2- 172.21.15.123:0/4198831049 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f3880073340 0x7f3880075800 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f38980af000 tx=0x7f389801ad80 comp rx=0 tx=0).stop 2024-09-16T15:49:24.446 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:24.447+0000 7f38a3b4d640 1 -- 172.21.15.123:0/4198831049 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f389c074430 msgr2=0x7f389c1553f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:24.446 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:24.447+0000 7f38a3b4d640 1 --2- 172.21.15.123:0/4198831049 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f389c074430 0x7f389c1553f0 secure :-1 s=READY pgs=1268 cs=0 l=1 rev1=1 crypto rx=0x7f389000c940 tx=0x7f389000ce00 comp rx=0 tx=0).stop 2024-09-16T15:49:24.447 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:24.447+0000 7f38a3b4d640 1 -- 172.21.15.123:0/4198831049 shutdown_connections 2024-09-16T15:49:24.447 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:24.447+0000 7f38a3b4d640 1 --2- 172.21.15.123:0/4198831049 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f3880073340 0x7f3880075800 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:24.447 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:24.447+0000 7f38a3b4d640 1 --2- 172.21.15.123:0/4198831049 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f389c15bca0 0x7f389c156550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:24.447 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:24.447+0000 7f38a3b4d640 1 --2- 172.21.15.123:0/4198831049 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f389c074430 0x7f389c1553f0 unknown :-1 s=CLOSED pgs=1268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:24.447 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:24.447+0000 7f38a3b4d640 1 --2- 172.21.15.123:0/4198831049 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f389c0726c0 0x7f389c0c7a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:24.447 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:24.447+0000 7f38a3b4d640 1 -- 172.21.15.123:0/4198831049 >> 172.21.15.123:0/4198831049 conn(0x7f389c0cfb10 msgr2=0x7f389c0b49b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:49:24.447 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:24.447+0000 7f38a3b4d640 1 -- 172.21.15.123:0/4198831049 shutdown_connections 2024-09-16T15:49:24.447 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:24.447+0000 7f38a3b4d640 1 -- 172.21.15.123:0/4198831049 wait complete. 2024-09-16T15:49:24.460 INFO:teuthology.orchestra.run.smithi123.stdout:{"pg_ready":true,"pg_map":{"version":105,"stamp":"2024-09-16T15:49:24.099466+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":157,"num_write_kb":1642,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":74,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":262096,"kb_used_data":12568,"kb_used_omap":12,"kb_used_meta":249331,"kb_avail":754712624,"statfs":{"total":773094113280,"available":772825726976,"internally_reserved":0,"allocated":12869632,"data_stored":6311261,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":255315505},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":39,"apply_latency_ms":39,"commit_latency_ns":39000000,"apply_latency_ns":39000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001434"},"pg_stats":[{"pgid":"19.3f","version":"0'0","reported_seq":10,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-09-16T15:49:20.026438+0000","last_change":"2024-09-16T15:49:20.025405+0000","last_active":"2024-09-16T15:49:20.026438+0000","last_peered":"2024-09-16T15:49:20.026438+0000","last_clean":"2024-09-16T15:49:20.026438+0000","last_became_active":"2024-09-16T15:49:20.025131+0000","last_became_peered":"2024-09-16T15:49:20.025131+0000","last_unstale":"2024-09-16T15:49:20.026438+0000","last_undegraded":"2024-09-16T15:49:20.026438+0000","last_fullsized":"2024-09-16T15:49:20.026438+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:49:18.967562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:49:18.967562+0000","last_clean_scrub_stamp":"2024-09-16T15:49:18.967562+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T01:55:12.894755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.3e","version":"0'0","reported_seq":10,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-09-16T15:49:20.022660+0000","last_change":"2024-09-16T15:49:20.018810+0000","last_active":"2024-09-16T15:49:20.022660+0000","last_peered":"2024-09-16T15:49:20.022660+0000","last_clean":"2024-09-16T15:49:20.022660+0000","last_became_active":"2024-09-16T15:49:20.018022+0000","last_became_peered":"2024-09-16T15:49:20.018022+0000","last_unstale":"2024-09-16T15:49:20.022660+0000","last_undegraded":"2024-09-16T15:49:20.022660+0000","last_fullsized":"2024-09-16T15:49:20.022660+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:49:18.967562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:49:18.967562+0000","last_clean_scrub_stamp":"2024-09-16T15:49:18.967562+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:59:36.978001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.3d","version":"0'0","reported_seq":10,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-09-16T15:49:20.043291+0000","last_change":"2024-09-16T15:49:20.034522+0000","last_active":"2024-09-16T15:49:20.043291+0000","last_peered":"2024-09-16T15:49:20.043291+0000","last_clean":"2024-09-16T15:49:20.043291+0000","last_became_active":"2024-09-16T15:49:20.033532+0000","last_became_peered":"2024-09-16T15:49:20.033532+0000","last_unstale":"2024-09-16T15:49:20.043291+0000","last_undegraded":"2024-09-16T15:49:20.043291+0000","last_fullsized":"2024-09-16T15:49:20.043291+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:49:18.967562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:49:18.967562+0000","last_clean_scrub_stamp":"2024-09-16T15:49:18.967562+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:41:41.203722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.3c","version":"0'0","reported_seq":10,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-09-16T15:49:20.051337+0000","last_change":"2024-09-16T15:49:20.050802+0000","last_active":"2024-09-16T15:49:20.051337+0000","last_peered":"2024-09-16T15:49:20.051337+0000","last_clean":"2024-09-16T15:49:20.051337+0000","last_became_active":"2024-09-16T15:49:20.050496+0000","last_became_peered":"2024-09-16T15:49:20.050496+0000","last_unstale":"2024-09-16T15:49:20.051337+0000","last_undegraded":"2024-09-16T15:49:20.051337+0000","last_fullsized":"2024-09-16T15:49:20.051337+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:49:18.967562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:49:18.967562+0000","last_clean_scrub_stamp":"2024-09-16T15:49:18.967562+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T02:00:42.471624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.3b","version":"0'0","reported_seq":10,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-09-16T15:49:20.058908+0000","last_change":"2024-09-16T15:49:20.055162+0000","last_active":"2024-09-16T15:49:20.058908+0000","last_peered":"2024-09-16T15:49:20.058908+0000","last_clean":"2024-09-16T15:49:20.058908+0000","last_became_active":"2024-09-16T15:49:20.050164+0000","last_became_peered":"2024-09-16T15:49:20.050164+0000","last_unstale":"2024-09-16T15:49:20.058908+0000","last_undegraded":"2024-09-16T15:49:20.058908+0000","last_fullsized":"2024-09-16T15:49:20.058908+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:49:18.967562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:49:18.967562+0000","last_clean_scrub_stamp":"2024-09-16T15:49:18.967562+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T22:54:20.092019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.3a","version":"0'0","reported_seq":10,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-09-16T15:49:20.046644+0000","last_change":"2024-09-16T15:49:20.034628+0000","last_active":"2024-09-16T15:49:20.046644+0000","last_peered":"2024-09-16T15:49:20.046644+0000","last_clean":"2024-09-16T15:49:20.046644+0000","last_became_active":"2024-09-16T15:49:20.030146+0000","last_became_peered":"2024-09-16T15:49:20.030146+0000","last_unstale":"2024-09-16T15:49:20.046644+0000","last_undegraded":"2024-09-16T15:49:20.046644+0000","last_fullsized":"2024-09-16T15:49:20.046644+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:49:18.967562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:49:18.967562+0000","last_clean_scrub_stamp":"2024-09-16T15:49:18.967562+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T01:16:16.527543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.39","version":"0'0","reported_seq":10,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-09-16T15:49:20.038358+0000","last_change":"2024-09-16T15:49:20.029027+0000","last_active":"2024-09-16T15:49:20.038358+0000","last_peered":"2024-09-16T15:49:20.038358+0000","last_clean":"2024-09-16T15:49:20.038358+0000","last_became_active":"2024-09-16T15:49:20.028199+0000","last_became_peered":"2024-09-16T15:49:20.028199+0000","last_unstale":"2024-09-16T15:49:20.038358+0000","last_undegraded":"2024-09-16T15:49:20.038358+0000","last_fullsized":"2024-09-16T15:49:20.038358+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:49:18.967562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:49:18.967562+0000","last_clean_scrub_stamp":"2024-09-16T15:49:18.967562+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T03:27:25.702064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.38","version":"0'0","reported_seq":10,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-09-16T15:49:20.052794+0000","last_change":"2024-09-16T15:49:20.033056+0000","last_active":"2024-09-16T15:49:20.052794+0000","last_peered":"2024-09-16T15:49:20.052794+0000","last_clean":"2024-09-16T15:49:20.052794+0000","last_became_active":"2024-09-16T15:49:20.027933+0000","last_became_peered":"2024-09-16T15:49:20.027933+0000","last_unstale":"2024-09-16T15:49:20.052794+0000","last_undegraded":"2024-09-16T15:49:20.052794+0000","last_fullsized":"2024-09-16T15:49:20.052794+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:49:18.967562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:49:18.967562+0000","last_clean_scrub_stamp":"2024-09-16T15:49:18.967562+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:17:29.529756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.37","version":"0'0","reported_seq":10,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-09-16T15:49:20.058001+0000","last_change":"2024-09-16T15:49:20.052915+0000","last_active":"2024-09-16T15:49:20.058001+0000","last_peered":"2024-09-16T15:49:20.058001+0000","last_clean":"2024-09-16T15:49:20.058001+0000","last_became_active":"2024-09-16T15:49:20.052189+0000","last_became_peered":"2024-09-16T15:49:20.052189+0000","last_unstale":"2024-09-16T15:49:20.058001+0000","last_undegraded":"2024-09-16T15:49:20.058001+0000","last_fullsized":"2024-09-16T15:49:20.058001+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:49:18.967562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:49:18.967562+0000","last_clean_scrub_stamp":"2024-09-16T15:49:18.967562+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:23:51.294104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.36","version":"0'0","reported_seq":10,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-09-16T15:49:20.022635+0000","last_change":"2024-09-16T15:49:20.018047+0000","last_active":"2024-09-16T15:49:20.022635+0000","last_peered":"2024-09-16T15:49:20.022635+0000","last_clean":"2024-09-16T15:49:20.022635+0000","last_became_active":"2024-09-16T15:49:20.017769+0000","last_became_peered":"2024-09-16T15:49:20.017769+0000","last_unstale":"2024-09-16T15:49:20.022635+0000","last_undegraded":"2024-09-16T15:49:20.022635+0000","last_fullsized":"2024-09-16T15:49:20.022635+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:49:18.967562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:49:18.967562+0000","last_clean_scrub_stamp":"2024-09-16T15:49:18.967562+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T00:46:04.825017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.35","version":"0'0","reported_seq":10,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-09-16T15:49:20.037376+0000","last_change":"2024-09-16T15:49:20.030040+0000","last_active":"2024-09-16T15:49:20.037376+0000","last_peered":"2024-09-16T15:49:20.037376+0000","last_clean":"2024-09-16T15:49:20.037376+0000","last_became_active":"2024-09-16T15:49:20.027913+0000","last_became_peered":"2024-09-16T15:49:20.027913+0000","last_unstale":"2024-09-16T15:49:20.037376+0000","last_undegraded":"2024-09-16T15:49:20.037376+0000","last_fullsized":"2024-09-16T15:49:20.037376+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:49:18.967562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:49:18.967562+0000","last_clean_scrub_stamp":"2024-09-16T15:49:18.967562+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:25:29.851210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.34","version":"0'0","reported_seq":10,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-09-16T15:49:20.042753+0000","last_change":"2024-09-16T15:49:20.031136+0000","last_active":"2024-09-16T15:49:20.042753+0000","last_peered":"2024-09-16T15:49:20.042753+0000","last_clean":"2024-09-16T15:49:20.042753+0000","last_became_active":"2024-09-16T15:49:20.026976+0000","last_became_peered":"2024-09-16T15:49:20.026976+0000","last_unstale":"2024-09-16T15:49:20.042753+0000","last_undegraded":"2024-09-16T15:49:20.042753+0000","last_fullsized":"2024-09-16T15:49:20.042753+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:49:18.967562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:49:18.967562+0000","last_clean_scrub_stamp":"2024-09-16T15:49:18.967562+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T18:28:59.419042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.33","version":"0'0","reported_seq":10,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-09-16T15:49:20.027590+0000","last_change":"2024-09-16T15:49:20.020663+0000","last_active":"2024-09-16T15:49:20.027590+0000","last_peered":"2024-09-16T15:49:20.027590+0000","last_clean":"2024-09-16T15:49:20.027590+0000","last_became_active":"2024-09-16T15:49:20.019315+0000","last_became_peered":"2024-09-16T15:49:20.019315+0000","last_unstale":"2024-09-16T15:49:20.027590+0000","last_undegraded":"2024-09-16T15:49:20.027590+0000","last_fullsized":"2024-09-16T15:49:20.027590+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:49:18.967562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:49:18.967562+0000","last_clean_scrub_stamp":"2024-09-16T15:49:18.967562+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T00:45:43.811211+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.32","version":"0'0","reported_seq":10,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-09-16T15:49:20.022362+0000","last_change":"2024-09-16T15:49:20.016990+0000","last_active":"2024-09-16T15:49:20.022362+0000","last_peered":"2024-09-16T15:49:20.022362+0000","last_clean":"2024-09-16T15:49:20.022362+0000","last_became_active":"2024-09-16T15:49:20.016648+0000","last_became_peered":"2024-09-16T15:49:20.016648+0000","last_unstale":"2024-09-16T15:49:20.022362+0000","last_undegraded":"2024-09-16T15:49:20.022362+0000","last_fullsized":"2024-09-16T15:49:20.022362+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:49:18.967562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:49:18.967562+0000","last_clean_scrub_stamp":"2024-09-16T15:49:18.967562+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T03:17:28.387500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.31","version":"0'0","reported_seq":10,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-09-16T15:49:20.026104+0000","last_change":"2024-09-16T15:49:20.022374+0000","last_active":"2024-09-16T15:49:20.026104+0000","last_peered":"2024-09-16T15:49:20.026104+0000","last_clean":"2024-09-16T15:49:20.026104+0000","last_became_active":"2024-09-16T15:49:20.021899+0000","last_became_peered":"2024-09-16T15:49:20.021899+0000","last_unstale":"2024-09-16T15:49:20.026104+0000","last_undegraded":"2024-09-16T15:49:20.026104+0000","last_fullsized":"2024-09-16T15:49:20.026104+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:49:18.967562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:49:18.967562+0000","last_clean_scrub_stamp":"2024-09-16T15:49:18.967562+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:20:43.185830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.30","version":"0'0","reported_seq":10,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-09-16T15:49:20.023423+0000","last_change":"2024-09-16T15:49:20.020306+0000","last_active":"2024-09-16T15:49:20.023423+0000","last_peered":"2024-09-16T15:49:20.023423+0000","last_clean":"2024-09-16T15:49:20.023423+0000","last_became_active":"2024-09-16T15:49:20.020046+0000","last_became_peered":"2024-09-16T15:49:20.020046+0000","last_unstale":"2024-09-16T15:49:20.023423+0000","last_undegraded":"2024-09-16T15:49:20.023423+0000","last_fullsized":"2024-09-16T15:49:20.023423+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:49:18.967562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:49:18.967562+0000","last_clean_scrub_stamp":"2024-09-16T15:49:18.967562+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T22:30:46.816501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.2f","version":"0'0","reported_seq":10,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-09-16T15:49:20.046739+0000","last_change":"2024-09-16T15:49:20.035937+0000","last_active":"2024-09-16T15:49:20.046739+0000","last_peered":"2024-09-16T15:49:20.046739+0000","last_clean":"2024-09-16T15:49:20.046739+0000","last_became_active":"2024-09-16T15:49:20.035175+0000","last_became_peered":"2024-09-16T15:49:20.035175+0000","last_unstale":"2024-09-16T15:49:20.046739+0000","last_undegraded":"2024-09-16T15:49:20.046739+0000","last_fullsized":"2024-09-16T15:49:20.046739+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:49:18.967562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:49:18.967562+0000","last_clean_scrub_stamp":"2024-09-16T15:49:18.967562+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:30:33.365956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.2e","version":"0'0","reported_seq":10,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-09-16T15:49:20.023098+0000","last_change":"2024-09-16T15:49:20.020864+0000","last_active":"2024-09-16T15:49:20.023098+0000","last_peered":"2024-09-16T15:49:20.023098+0000","last_clean":"2024-09-16T15:49:20.023098+0000","last_became_active":"2024-09-16T15:49:20.020624+0000","last_became_peered":"2024-09-16T15:49:20.020624+0000","last_unstale":"2024-09-16T15:49:20.023098+0000","last_undegraded":"2024-09-16T15:49:20.023098+0000","last_fullsized":"2024-09-16T15:49:20.023098+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:49:18.967562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:49:18.967562+0000","last_clean_scrub_stamp":"2024-09-16T15:49:18.967562+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:47:36.317556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.2d","version":"0'0","reported_seq":10,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-09-16T15:49:20.037465+0000","last_change":"2024-09-16T15:49:20.030653+0000","last_active":"2024-09-16T15:49:20.037465+0000","last_peered":"2024-09-16T15:49:20.037465+0000","last_clean":"2024-09-16T15:49:20.037465+0000","last_became_active":"2024-09-16T15:49:20.030433+0000","last_became_peered":"2024-09-16T15:49:20.030433+0000","last_unstale":"2024-09-16T15:49:20.037465+0000","last_undegraded":"2024-09-16T15:49:20.037465+0000","last_fullsized":"2024-09-16T15:49:20.037465+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:49:18.967562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:49:18.967562+0000","last_clean_scrub_stamp":"2024-09-16T15:49:18.967562+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:06:43.611308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.2c","version":"0'0","reported_seq":10,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-09-16T15:49:20.015023+0000","last_change":"2024-09-16T15:49:20.011995+0000","last_active":"2024-09-16T15:49:20.015023+0000","last_peered":"2024-09-16T15:49:20.015023+0000","last_clean":"2024-09-16T15:49:20.015023+0000","last_became_active":"2024-09-16T15:49:20.011483+0000","last_became_peered":"2024-09-16T15:49:20.011483+0000","last_unstale":"2024-09-16T15:49:20.015023+0000","last_undegraded":"2024-09-16T15:49:20.015023+0000","last_fullsized":"2024-09-16T15:49:20.015023+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:49:18.967562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:49:18.967562+0000","last_clean_scrub_stamp":"2024-09-16T15:49:18.967562+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T03:06:32.057244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.2b","version":"0'0","reported_seq":10,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-09-16T15:49:20.059412+0000","last_change":"2024-09-16T15:49:20.058046+0000","last_active":"2024-09-16T15:49:20.059412+0000","last_peered":"2024-09-16T15:49:20.059412+0000","last_clean":"2024-09-16T15:49:20.059412+0000","last_became_active":"2024-09-16T15:49:20.049107+0000","last_became_peered":"2024-09-16T15:49:20.049107+0000","last_unstale":"2024-09-16T15:49:20.059412+0000","last_undegraded":"2024-09-16T15:49:20.059412+0000","last_fullsized":"2024-09-16T15:49:20.059412+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:49:18.967562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:49:18.967562+0000","last_clean_scrub_stamp":"2024-09-16T15:49:18.967562+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:14:53.412945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.2a","version":"0'0","reported_seq":10,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-09-16T15:49:20.037304+0000","last_change":"2024-09-16T15:49:20.028064+0000","last_active":"2024-09-16T15:49:20.037304+0000","last_peered":"2024-09-16T15:49:20.037304+0000","last_clean":"2024-09-16T15:49:20.037304+0000","last_became_active":"2024-09-16T15:49:20.027664+0000","last_became_peered":"2024-09-16T15:49:20.027664+0000","last_unstale":"2024-09-16T15:49:20.037304+0000","last_undegraded":"2024-09-16T15:49:20.037304+0000","last_fullsized":"2024-09-16T15:49:20.037304+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:49:18.967562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:49:18.967562+0000","last_clean_scrub_stamp":"2024-09-16T15:49:18.967562+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T01:21:22.074609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.29","version":"0'0","reported_seq":10,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-09-16T15:49:20.043208+0000","last_change":"2024-09-16T15:49:20.033729+0000","last_active":"2024-09-16T15:49:20.043208+0000","last_peered":"2024-09-16T15:49:20.043208+0000","last_clean":"2024-09-16T15:49:20.043208+0000","last_became_active":"2024-09-16T15:49:20.026602+0000","last_became_peered":"2024-09-16T15:49:20.026602+0000","last_unstale":"2024-09-16T15:49:20.043208+0000","last_undegraded":"2024-09-16T15:49:20.043208+0000","last_fullsized":"2024-09-16T15:49:20.043208+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:49:18.967562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:49:18.967562+0000","last_clean_scrub_stamp":"2024-09-16T15:49:18.967562+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:59:22.673559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.28","version":"0'0","reported_seq":10,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-09-16T15:49:20.037398+0000","last_change":"2024-09-16T15:49:20.029115+0000","last_active":"2024-09-16T15:49:20.037398+0000","last_peered":"2024-09-16T15:49:20.037398+0000","last_clean":"2024-09-16T15:49:20.037398+0000","last_became_active":"2024-09-16T15:49:20.028609+0000","last_became_peered":"2024-09-16T15:49:20.028609+0000","last_unstale":"2024-09-16T15:49:20.037398+0000","last_undegraded":"2024-09-16T15:49:20.037398+0000","last_fullsized":"2024-09-16T15:49:20.037398+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:49:18.967562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:49:18.967562+0000","last_clean_scrub_stamp":"2024-09-16T15:49:18.967562+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T23:34:04.482788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.27","version":"0'0","reported_seq":10,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-09-16T15:49:20.045378+0000","last_change":"2024-09-16T15:49:20.041979+0000","last_active":"2024-09-16T15:49:20.045378+0000","last_peered":"2024-09-16T15:49:20.045378+0000","last_clean":"2024-09-16T15:49:20.045378+0000","last_became_active":"2024-09-16T15:49:20.041570+0000","last_became_peered":"2024-09-16T15:49:20.041570+0000","last_unstale":"2024-09-16T15:49:20.045378+0000","last_undegraded":"2024-09-16T15:49:20.045378+0000","last_fullsized":"2024-09-16T15:49:20.045378+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:49:18.967562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:49:18.967562+0000","last_clean_scrub_stamp":"2024-09-16T15:49:18.967562+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T22:59:42.998898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.26","version":"0'0","reported_seq":10,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-09-16T15:49:20.023069+0000","last_change":"2024-09-16T15:49:20.019624+0000","last_active":"2024-09-16T15:49:20.023069+0000","last_peered":"2024-09-16T15:49:20.023069+0000","last_clean":"2024-09-16T15:49:20.023069+0000","last_became_active":"2024-09-16T15:49:20.019378+0000","last_became_peered":"2024-09-16T15:49:20.019378+0000","last_unstale":"2024-09-16T15:49:20.023069+0000","last_undegraded":"2024-09-16T15:49:20.023069+0000","last_fullsized":"2024-09-16T15:49:20.023069+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:49:18.967562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:49:18.967562+0000","last_clean_scrub_stamp":"2024-09-16T15:49:18.967562+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:23:40.870521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.25","version":"0'0","reported_seq":10,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-09-16T15:49:20.021365+0000","last_change":"2024-09-16T15:49:20.019444+0000","last_active":"2024-09-16T15:49:20.021365+0000","last_peered":"2024-09-16T15:49:20.021365+0000","last_clean":"2024-09-16T15:49:20.021365+0000","last_became_active":"2024-09-16T15:49:20.019146+0000","last_became_peered":"2024-09-16T15:49:20.019146+0000","last_unstale":"2024-09-16T15:49:20.021365+0000","last_undegraded":"2024-09-16T15:49:20.021365+0000","last_fullsized":"2024-09-16T15:49:20.021365+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:49:18.967562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:49:18.967562+0000","last_clean_scrub_stamp":"2024-09-16T15:49:18.967562+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T02:29:37.284424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.24","version":"0'0","reported_seq":10,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-09-16T15:49:20.032552+0000","last_change":"2024-09-16T15:49:20.029167+0000","last_active":"2024-09-16T15:49:20.032552+0000","last_peered":"2024-09-16T15:49:20.032552+0000","last_clean":"2024-09-16T15:49:20.032552+0000","last_became_active":"2024-09-16T15:49:20.028409+0000","last_became_peered":"2024-09-16T15:49:20.028409+0000","last_unstale":"2024-09-16T15:49:20.032552+0000","last_undegraded":"2024-09-16T15:49:20.032552+0000","last_fullsized":"2024-09-16T15:49:20.032552+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:49:18.967562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:49:18.967562+0000","last_clean_scrub_stamp":"2024-09-16T15:49:18.967562+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T01:35:30.971956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.23","version":"0'0","reported_seq":10,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-09-16T15:49:20.027637+0000","last_change":"2024-09-16T15:49:20.023500+0000","last_active":"2024-09-16T15:49:20.027637+0000","last_peered":"2024-09-16T15:49:20.027637+0000","last_clean":"2024-09-16T15:49:20.027637+0000","last_became_active":"2024-09-16T15:49:20.023225+0000","last_became_peered":"2024-09-16T15:49:20.023225+0000","last_unstale":"2024-09-16T15:49:20.027637+0000","last_undegraded":"2024-09-16T15:49:20.027637+0000","last_fullsized":"2024-09-16T15:49:20.027637+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:49:18.967562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:49:18.967562+0000","last_clean_scrub_stamp":"2024-09-16T15:49:18.967562+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:15:15.039502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.22","version":"0'0","reported_seq":10,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-09-16T15:49:20.026433+0000","last_change":"2024-09-16T15:49:20.024936+0000","last_active":"2024-09-16T15:49:20.026433+0000","last_peered":"2024-09-16T15:49:20.026433+0000","last_clean":"2024-09-16T15:49:20.026433+0000","last_became_active":"2024-09-16T15:49:20.024607+0000","last_became_peered":"2024-09-16T15:49:20.024607+0000","last_unstale":"2024-09-16T15:49:20.026433+0000","last_undegraded":"2024-09-16T15:49:20.026433+0000","last_fullsized":"2024-09-16T15:49:20.026433+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:49:18.967562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:49:18.967562+0000","last_clean_scrub_stamp":"2024-09-16T15:49:18.967562+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:34:05.136599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.21","version":"0'0","reported_seq":10,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-09-16T15:49:20.059115+0000","last_change":"2024-09-16T15:49:20.055866+0000","last_active":"2024-09-16T15:49:20.059115+0000","last_peered":"2024-09-16T15:49:20.059115+0000","last_clean":"2024-09-16T15:49:20.059115+0000","last_became_active":"2024-09-16T15:49:20.043962+0000","last_became_peered":"2024-09-16T15:49:20.043962+0000","last_unstale":"2024-09-16T15:49:20.059115+0000","last_undegraded":"2024-09-16T15:49:20.059115+0000","last_fullsized":"2024-09-16T15:49:20.059115+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:49:18.967562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:49:18.967562+0000","last_clean_scrub_stamp":"2024-09-16T15:49:18.967562+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T22:09:02.693580+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.20","version":"0'0","reported_seq":10,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-09-16T15:49:20.020021+0000","last_change":"2024-09-16T15:49:20.017088+0000","last_active":"2024-09-16T15:49:20.020021+0000","last_peered":"2024-09-16T15:49:20.020021+0000","last_clean":"2024-09-16T15:49:20.020021+0000","last_became_active":"2024-09-16T15:49:20.012864+0000","last_became_peered":"2024-09-16T15:49:20.012864+0000","last_unstale":"2024-09-16T15:49:20.020021+0000","last_undegraded":"2024-09-16T15:49:20.020021+0000","last_fullsized":"2024-09-16T15:49:20.020021+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:49:18.967562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:49:18.967562+0000","last_clean_scrub_stamp":"2024-09-16T15:49:18.967562+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:54:14.214563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.1e","version":"0'0","reported_seq":10,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-09-16T15:49:20.037227+0000","last_change":"2024-09-16T15:49:20.032634+0000","last_active":"2024-09-16T15:49:20.037227+0000","last_peered":"2024-09-16T15:49:20.037227+0000","last_clean":"2024-09-16T15:49:20.037227+0000","last_became_active":"2024-09-16T15:49:20.032152+0000","last_became_peered":"2024-09-16T15:49:20.032152+0000","last_unstale":"2024-09-16T15:49:20.037227+0000","last_undegraded":"2024-09-16T15:49:20.037227+0000","last_fullsized":"2024-09-16T15:49:20.037227+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:49:18.967562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:49:18.967562+0000","last_clean_scrub_stamp":"2024-09-16T15:49:18.967562+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T03:01:45.514782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.1f","version":"118'4","reported_seq":17,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-09-16T15:49:20.049550+0000","last_change":"2024-09-16T15:49:19.016800+0000","last_active":"2024-09-16T15:49:20.049550+0000","last_peered":"2024-09-16T15:49:20.049550+0000","last_clean":"2024-09-16T15:49:20.049550+0000","last_became_active":"2024-09-16T15:49:19.016007+0000","last_became_peered":"2024-09-16T15:49:19.016007+0000","last_unstale":"2024-09-16T15:49:20.049550+0000","last_undegraded":"2024-09-16T15:49:20.049550+0000","last_fullsized":"2024-09-16T15:49:20.049550+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:49:17.964400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:49:17.964400+0000","last_clean_scrub_stamp":"2024-09-16T15:49:17.964400+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:21:30.845086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.1f","version":"0'0","reported_seq":10,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-09-16T15:49:20.026446+0000","last_change":"2024-09-16T15:49:20.021973+0000","last_active":"2024-09-16T15:49:20.026446+0000","last_peered":"2024-09-16T15:49:20.026446+0000","last_clean":"2024-09-16T15:49:20.026446+0000","last_became_active":"2024-09-16T15:49:20.021179+0000","last_became_peered":"2024-09-16T15:49:20.021179+0000","last_unstale":"2024-09-16T15:49:20.026446+0000","last_undegraded":"2024-09-16T15:49:20.026446+0000","last_fullsized":"2024-09-16T15:49:20.026446+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:49:18.967562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:49:18.967562+0000","last_clean_scrub_stamp":"2024-09-16T15:49:18.967562+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T22:19:23.440465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.1e","version":"118'1","reported_seq":14,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-09-16T15:49:20.038176+0000","last_change":"2024-09-16T15:49:18.991536+0000","last_active":"2024-09-16T15:49:20.038176+0000","last_peered":"2024-09-16T15:49:20.038176+0000","last_clean":"2024-09-16T15:49:20.038176+0000","last_became_active":"2024-09-16T15:49:18.990793+0000","last_became_peered":"2024-09-16T15:49:18.990793+0000","last_unstale":"2024-09-16T15:49:20.038176+0000","last_undegraded":"2024-09-16T15:49:20.038176+0000","last_fullsized":"2024-09-16T15:49:20.038176+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:49:17.964400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:49:17.964400+0000","last_clean_scrub_stamp":"2024-09-16T15:49:17.964400+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T01:36:48.358813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.1c","version":"0'0","reported_seq":10,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-09-16T15:49:20.058623+0000","last_change":"2024-09-16T15:49:20.050146+0000","last_active":"2024-09-16T15:49:20.058623+0000","last_peered":"2024-09-16T15:49:20.058623+0000","last_clean":"2024-09-16T15:49:20.058623+0000","last_became_active":"2024-09-16T15:49:20.048951+0000","last_became_peered":"2024-09-16T15:49:20.048951+0000","last_unstale":"2024-09-16T15:49:20.058623+0000","last_undegraded":"2024-09-16T15:49:20.058623+0000","last_fullsized":"2024-09-16T15:49:20.058623+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:49:18.967562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:49:18.967562+0000","last_clean_scrub_stamp":"2024-09-16T15:49:18.967562+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T01:27:31.814144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.1d","version":"118'2","reported_seq":15,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-09-16T15:49:20.034973+0000","last_change":"2024-09-16T15:49:18.994755+0000","last_active":"2024-09-16T15:49:20.034973+0000","last_peered":"2024-09-16T15:49:20.034973+0000","last_clean":"2024-09-16T15:49:20.034973+0000","last_became_active":"2024-09-16T15:49:18.994324+0000","last_became_peered":"2024-09-16T15:49:18.994324+0000","last_unstale":"2024-09-16T15:49:20.034973+0000","last_undegraded":"2024-09-16T15:49:20.034973+0000","last_fullsized":"2024-09-16T15:49:20.034973+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:49:17.964400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:49:17.964400+0000","last_clean_scrub_stamp":"2024-09-16T15:49:17.964400+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T22:48:23.896738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.1d","version":"0'0","reported_seq":10,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-09-16T15:49:20.057485+0000","last_change":"2024-09-16T15:49:20.048299+0000","last_active":"2024-09-16T15:49:20.057485+0000","last_peered":"2024-09-16T15:49:20.057485+0000","last_clean":"2024-09-16T15:49:20.057485+0000","last_became_active":"2024-09-16T15:49:20.043603+0000","last_became_peered":"2024-09-16T15:49:20.043603+0000","last_unstale":"2024-09-16T15:49:20.057485+0000","last_undegraded":"2024-09-16T15:49:20.057485+0000","last_fullsized":"2024-09-16T15:49:20.057485+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:49:18.967562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:49:18.967562+0000","last_clean_scrub_stamp":"2024-09-16T15:49:18.967562+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:39:06.552608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.1c","version":"118'2","reported_seq":15,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-09-16T15:49:20.028954+0000","last_change":"2024-09-16T15:49:19.002172+0000","last_active":"2024-09-16T15:49:20.028954+0000","last_peered":"2024-09-16T15:49:20.028954+0000","last_clean":"2024-09-16T15:49:20.028954+0000","last_became_active":"2024-09-16T15:49:19.001764+0000","last_became_peered":"2024-09-16T15:49:19.001764+0000","last_unstale":"2024-09-16T15:49:20.028954+0000","last_undegraded":"2024-09-16T15:49:20.028954+0000","last_fullsized":"2024-09-16T15:49:20.028954+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:49:17.964400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:49:17.964400+0000","last_clean_scrub_stamp":"2024-09-16T15:49:17.964400+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T01:29:07.726449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.1a","version":"0'0","reported_seq":10,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-09-16T15:49:20.037632+0000","last_change":"2024-09-16T15:49:20.036305+0000","last_active":"2024-09-16T15:49:20.037632+0000","last_peered":"2024-09-16T15:49:20.037632+0000","last_clean":"2024-09-16T15:49:20.037632+0000","last_became_active":"2024-09-16T15:49:20.036098+0000","last_became_peered":"2024-09-16T15:49:20.036098+0000","last_unstale":"2024-09-16T15:49:20.037632+0000","last_undegraded":"2024-09-16T15:49:20.037632+0000","last_fullsized":"2024-09-16T15:49:20.037632+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:49:18.967562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:49:18.967562+0000","last_clean_scrub_stamp":"2024-09-16T15:49:18.967562+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:30:37.221003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.1b","version":"118'1","reported_seq":14,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-09-16T15:49:20.015235+0000","last_change":"2024-09-16T15:49:18.997418+0000","last_active":"2024-09-16T15:49:20.015235+0000","last_peered":"2024-09-16T15:49:20.015235+0000","last_clean":"2024-09-16T15:49:20.015235+0000","last_became_active":"2024-09-16T15:49:18.997039+0000","last_became_peered":"2024-09-16T15:49:18.997039+0000","last_unstale":"2024-09-16T15:49:20.015235+0000","last_undegraded":"2024-09-16T15:49:20.015235+0000","last_fullsized":"2024-09-16T15:49:20.015235+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:49:17.964400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:49:17.964400+0000","last_clean_scrub_stamp":"2024-09-16T15:49:17.964400+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T22:21:18.524483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.1b","version":"0'0","reported_seq":10,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-09-16T15:49:20.038254+0000","last_change":"2024-09-16T15:49:20.029380+0000","last_active":"2024-09-16T15:49:20.038254+0000","last_peered":"2024-09-16T15:49:20.038254+0000","last_clean":"2024-09-16T15:49:20.038254+0000","last_became_active":"2024-09-16T15:49:20.028892+0000","last_became_peered":"2024-09-16T15:49:20.028892+0000","last_unstale":"2024-09-16T15:49:20.038254+0000","last_undegraded":"2024-09-16T15:49:20.038254+0000","last_fullsized":"2024-09-16T15:49:20.038254+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:49:18.967562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:49:18.967562+0000","last_clean_scrub_stamp":"2024-09-16T15:49:18.967562+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:47:36.317556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.1a","version":"118'1","reported_seq":14,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-09-16T15:49:20.022245+0000","last_change":"2024-09-16T15:49:18.998790+0000","last_active":"2024-09-16T15:49:20.022245+0000","last_peered":"2024-09-16T15:49:20.022245+0000","last_clean":"2024-09-16T15:49:20.022245+0000","last_became_active":"2024-09-16T15:49:18.998134+0000","last_became_peered":"2024-09-16T15:49:18.998134+0000","last_unstale":"2024-09-16T15:49:20.022245+0000","last_undegraded":"2024-09-16T15:49:20.022245+0000","last_fullsized":"2024-09-16T15:49:20.022245+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:49:17.964400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:49:17.964400+0000","last_clean_scrub_stamp":"2024-09-16T15:49:17.964400+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T23:43:15.143261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.18","version":"0'0","reported_seq":10,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-09-16T15:49:20.052821+0000","last_change":"2024-09-16T15:49:20.034428+0000","last_active":"2024-09-16T15:49:20.052821+0000","last_peered":"2024-09-16T15:49:20.052821+0000","last_clean":"2024-09-16T15:49:20.052821+0000","last_became_active":"2024-09-16T15:49:20.027556+0000","last_became_peered":"2024-09-16T15:49:20.027556+0000","last_unstale":"2024-09-16T15:49:20.052821+0000","last_undegraded":"2024-09-16T15:49:20.052821+0000","last_fullsized":"2024-09-16T15:49:20.052821+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:49:18.967562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:49:18.967562+0000","last_clean_scrub_stamp":"2024-09-16T15:49:18.967562+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:58:12.260025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.19","version":"0'0","reported_seq":13,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-09-16T15:49:20.014091+0000","last_change":"2024-09-16T15:49:19.018690+0000","last_active":"2024-09-16T15:49:20.014091+0000","last_peered":"2024-09-16T15:49:20.014091+0000","last_clean":"2024-09-16T15:49:20.014091+0000","last_became_active":"2024-09-16T15:49:19.017949+0000","last_became_peered":"2024-09-16T15:49:19.017949+0000","last_unstale":"2024-09-16T15:49:20.014091+0000","last_undegraded":"2024-09-16T15:49:20.014091+0000","last_fullsized":"2024-09-16T15:49:20.014091+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:49:17.964400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:49:17.964400+0000","last_clean_scrub_stamp":"2024-09-16T15:49:17.964400+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T02:22:55.594740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.19","version":"0'0","reported_seq":10,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-09-16T15:49:20.022308+0000","last_change":"2024-09-16T15:49:20.014659+0000","last_active":"2024-09-16T15:49:20.022308+0000","last_peered":"2024-09-16T15:49:20.022308+0000","last_clean":"2024-09-16T15:49:20.022308+0000","last_became_active":"2024-09-16T15:49:20.014342+0000","last_became_peered":"2024-09-16T15:49:20.014342+0000","last_unstale":"2024-09-16T15:49:20.022308+0000","last_undegraded":"2024-09-16T15:49:20.022308+0000","last_fullsized":"2024-09-16T15:49:20.022308+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:49:18.967562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:49:18.967562+0000","last_clean_scrub_stamp":"2024-09-16T15:49:18.967562+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T22:45:04.658721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.18","version":"0'0","reported_seq":13,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-09-16T15:49:19.997156+0000","last_change":"2024-09-16T15:49:19.003568+0000","last_active":"2024-09-16T15:49:19.997156+0000","last_peered":"2024-09-16T15:49:19.997156+0000","last_clean":"2024-09-16T15:49:19.997156+0000","last_became_active":"2024-09-16T15:49:19.003280+0000","last_became_peered":"2024-09-16T15:49:19.003280+0000","last_unstale":"2024-09-16T15:49:19.997156+0000","last_undegraded":"2024-09-16T15:49:19.997156+0000","last_fullsized":"2024-09-16T15:49:19.997156+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:49:17.964400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:49:17.964400+0000","last_clean_scrub_stamp":"2024-09-16T15:49:17.964400+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T18:20:00.931187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.16","version":"0'0","reported_seq":10,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-09-16T15:49:20.035435+0000","last_change":"2024-09-16T15:49:20.031021+0000","last_active":"2024-09-16T15:49:20.035435+0000","last_peered":"2024-09-16T15:49:20.035435+0000","last_clean":"2024-09-16T15:49:20.035435+0000","last_became_active":"2024-09-16T15:49:20.030747+0000","last_became_peered":"2024-09-16T15:49:20.030747+0000","last_unstale":"2024-09-16T15:49:20.035435+0000","last_undegraded":"2024-09-16T15:49:20.035435+0000","last_fullsized":"2024-09-16T15:49:20.035435+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:49:18.967562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:49:18.967562+0000","last_clean_scrub_stamp":"2024-09-16T15:49:18.967562+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:59:13.653253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.17","version":"0'0","reported_seq":13,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-09-16T15:49:20.043994+0000","last_change":"2024-09-16T15:49:19.006709+0000","last_active":"2024-09-16T15:49:20.043994+0000","last_peered":"2024-09-16T15:49:20.043994+0000","last_clean":"2024-09-16T15:49:20.043994+0000","last_became_active":"2024-09-16T15:49:19.005415+0000","last_became_peered":"2024-09-16T15:49:19.005415+0000","last_unstale":"2024-09-16T15:49:20.043994+0000","last_undegraded":"2024-09-16T15:49:20.043994+0000","last_fullsized":"2024-09-16T15:49:20.043994+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:49:17.964400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:49:17.964400+0000","last_clean_scrub_stamp":"2024-09-16T15:49:17.964400+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:31:42.555798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.17","version":"0'0","reported_seq":10,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-09-16T15:49:20.052583+0000","last_change":"2024-09-16T15:49:20.046340+0000","last_active":"2024-09-16T15:49:20.052583+0000","last_peered":"2024-09-16T15:49:20.052583+0000","last_clean":"2024-09-16T15:49:20.052583+0000","last_became_active":"2024-09-16T15:49:20.045340+0000","last_became_peered":"2024-09-16T15:49:20.045340+0000","last_unstale":"2024-09-16T15:49:20.052583+0000","last_undegraded":"2024-09-16T15:49:20.052583+0000","last_fullsized":"2024-09-16T15:49:20.052583+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:49:18.967562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:49:18.967562+0000","last_clean_scrub_stamp":"2024-09-16T15:49:18.967562+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:41:17.054562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.16","version":"0'0","reported_seq":13,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-09-16T15:49:20.009193+0000","last_change":"2024-09-16T15:49:18.999247+0000","last_active":"2024-09-16T15:49:20.009193+0000","last_peered":"2024-09-16T15:49:20.009193+0000","last_clean":"2024-09-16T15:49:20.009193+0000","last_became_active":"2024-09-16T15:49:18.998621+0000","last_became_peered":"2024-09-16T15:49:18.998621+0000","last_unstale":"2024-09-16T15:49:20.009193+0000","last_undegraded":"2024-09-16T15:49:20.009193+0000","last_fullsized":"2024-09-16T15:49:20.009193+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:49:17.964400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:49:17.964400+0000","last_clean_scrub_stamp":"2024-09-16T15:49:17.964400+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T02:26:02.649228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.14","version":"0'0","reported_seq":10,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-09-16T15:49:20.058517+0000","last_change":"2024-09-16T15:49:20.049655+0000","last_active":"2024-09-16T15:49:20.058517+0000","last_peered":"2024-09-16T15:49:20.058517+0000","last_clean":"2024-09-16T15:49:20.058517+0000","last_became_active":"2024-09-16T15:49:20.048533+0000","last_became_peered":"2024-09-16T15:49:20.048533+0000","last_unstale":"2024-09-16T15:49:20.058517+0000","last_undegraded":"2024-09-16T15:49:20.058517+0000","last_fullsized":"2024-09-16T15:49:20.058517+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:49:18.967562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:49:18.967562+0000","last_clean_scrub_stamp":"2024-09-16T15:49:18.967562+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:20:35.281539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.15","version":"118'3","reported_seq":16,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-09-16T15:49:20.031442+0000","last_change":"2024-09-16T15:49:18.994884+0000","last_active":"2024-09-16T15:49:20.031442+0000","last_peered":"2024-09-16T15:49:20.031442+0000","last_clean":"2024-09-16T15:49:20.031442+0000","last_became_active":"2024-09-16T15:49:18.994167+0000","last_became_peered":"2024-09-16T15:49:18.994167+0000","last_unstale":"2024-09-16T15:49:20.031442+0000","last_undegraded":"2024-09-16T15:49:20.031442+0000","last_fullsized":"2024-09-16T15:49:20.031442+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:49:17.964400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:49:17.964400+0000","last_clean_scrub_stamp":"2024-09-16T15:49:17.964400+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:07:21.707520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.15","version":"0'0","reported_seq":10,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-09-16T15:49:20.058744+0000","last_change":"2024-09-16T15:49:20.055174+0000","last_active":"2024-09-16T15:49:20.058744+0000","last_peered":"2024-09-16T15:49:20.058744+0000","last_clean":"2024-09-16T15:49:20.058744+0000","last_became_active":"2024-09-16T15:49:20.054624+0000","last_became_peered":"2024-09-16T15:49:20.054624+0000","last_unstale":"2024-09-16T15:49:20.058744+0000","last_undegraded":"2024-09-16T15:49:20.058744+0000","last_fullsized":"2024-09-16T15:49:20.058744+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:49:18.967562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:49:18.967562+0000","last_clean_scrub_stamp":"2024-09-16T15:49:18.967562+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:35:24.858013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.14","version":"118'6","reported_seq":19,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-09-16T15:49:20.062782+0000","last_change":"2024-09-16T15:49:19.010997+0000","last_active":"2024-09-16T15:49:20.062782+0000","last_peered":"2024-09-16T15:49:20.062782+0000","last_clean":"2024-09-16T15:49:20.062782+0000","last_became_active":"2024-09-16T15:49:19.010201+0000","last_became_peered":"2024-09-16T15:49:19.010201+0000","last_unstale":"2024-09-16T15:49:20.062782+0000","last_undegraded":"2024-09-16T15:49:20.062782+0000","last_fullsized":"2024-09-16T15:49:20.062782+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:49:17.964400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:49:17.964400+0000","last_clean_scrub_stamp":"2024-09-16T15:49:17.964400+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T18:54:40.001665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.13","version":"0'0","reported_seq":10,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-09-16T15:49:20.037239+0000","last_change":"2024-09-16T15:49:20.031568+0000","last_active":"2024-09-16T15:49:20.037239+0000","last_peered":"2024-09-16T15:49:20.037239+0000","last_clean":"2024-09-16T15:49:20.037239+0000","last_became_active":"2024-09-16T15:49:20.031308+0000","last_became_peered":"2024-09-16T15:49:20.031308+0000","last_unstale":"2024-09-16T15:49:20.037239+0000","last_undegraded":"2024-09-16T15:49:20.037239+0000","last_fullsized":"2024-09-16T15:49:20.037239+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:49:18.967562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:49:18.967562+0000","last_clean_scrub_stamp":"2024-09-16T15:49:18.967562+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:58:42.430549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.12","version":"0'0","reported_seq":13,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-09-16T15:49:20.024378+0000","last_change":"2024-09-16T15:49:19.000238+0000","last_active":"2024-09-16T15:49:20.024378+0000","last_peered":"2024-09-16T15:49:20.024378+0000","last_clean":"2024-09-16T15:49:20.024378+0000","last_became_active":"2024-09-16T15:49:18.999837+0000","last_became_peered":"2024-09-16T15:49:18.999837+0000","last_unstale":"2024-09-16T15:49:20.024378+0000","last_undegraded":"2024-09-16T15:49:20.024378+0000","last_fullsized":"2024-09-16T15:49:20.024378+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:49:17.964400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:49:17.964400+0000","last_clean_scrub_stamp":"2024-09-16T15:49:17.964400+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T18:57:09.054608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.10","version":"0'0","reported_seq":10,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-09-16T15:49:20.013249+0000","last_change":"2024-09-16T15:49:20.006545+0000","last_active":"2024-09-16T15:49:20.013249+0000","last_peered":"2024-09-16T15:49:20.013249+0000","last_clean":"2024-09-16T15:49:20.013249+0000","last_became_active":"2024-09-16T15:49:20.005832+0000","last_became_peered":"2024-09-16T15:49:20.005832+0000","last_unstale":"2024-09-16T15:49:20.013249+0000","last_undegraded":"2024-09-16T15:49:20.013249+0000","last_fullsized":"2024-09-16T15:49:20.013249+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:49:18.967562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:49:18.967562+0000","last_clean_scrub_stamp":"2024-09-16T15:49:18.967562+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T03:29:35.271830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.11","version":"0'0","reported_seq":13,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-09-16T15:49:19.997284+0000","last_change":"2024-09-16T15:49:19.010274+0000","last_active":"2024-09-16T15:49:19.997284+0000","last_peered":"2024-09-16T15:49:19.997284+0000","last_clean":"2024-09-16T15:49:19.997284+0000","last_became_active":"2024-09-16T15:49:19.009964+0000","last_became_peered":"2024-09-16T15:49:19.009964+0000","last_unstale":"2024-09-16T15:49:19.997284+0000","last_undegraded":"2024-09-16T15:49:19.997284+0000","last_fullsized":"2024-09-16T15:49:19.997284+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:49:17.964400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:49:17.964400+0000","last_clean_scrub_stamp":"2024-09-16T15:49:17.964400+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:57:29.053710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.11","version":"0'0","reported_seq":10,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-09-16T15:49:20.044676+0000","last_change":"2024-09-16T15:49:20.042183+0000","last_active":"2024-09-16T15:49:20.044676+0000","last_peered":"2024-09-16T15:49:20.044676+0000","last_clean":"2024-09-16T15:49:20.044676+0000","last_became_active":"2024-09-16T15:49:20.032992+0000","last_became_peered":"2024-09-16T15:49:20.032992+0000","last_unstale":"2024-09-16T15:49:20.044676+0000","last_undegraded":"2024-09-16T15:49:20.044676+0000","last_fullsized":"2024-09-16T15:49:20.044676+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:49:18.967562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:49:18.967562+0000","last_clean_scrub_stamp":"2024-09-16T15:49:18.967562+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:13:14.414577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.10","version":"118'8","reported_seq":21,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-09-16T15:49:20.062255+0000","last_change":"2024-09-16T15:49:19.000646+0000","last_active":"2024-09-16T15:49:20.062255+0000","last_peered":"2024-09-16T15:49:20.062255+0000","last_clean":"2024-09-16T15:49:20.062255+0000","last_became_active":"2024-09-16T15:49:18.999926+0000","last_became_peered":"2024-09-16T15:49:18.999926+0000","last_unstale":"2024-09-16T15:49:20.062255+0000","last_undegraded":"2024-09-16T15:49:20.062255+0000","last_fullsized":"2024-09-16T15:49:20.062255+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:49:17.964400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:49:17.964400+0000","last_clean_scrub_stamp":"2024-09-16T15:49:17.964400+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:43:38.769049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.e","version":"0'0","reported_seq":10,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-09-16T15:49:20.026542+0000","last_change":"2024-09-16T15:49:20.021838+0000","last_active":"2024-09-16T15:49:20.026542+0000","last_peered":"2024-09-16T15:49:20.026542+0000","last_clean":"2024-09-16T15:49:20.026542+0000","last_became_active":"2024-09-16T15:49:20.021142+0000","last_became_peered":"2024-09-16T15:49:20.021142+0000","last_unstale":"2024-09-16T15:49:20.026542+0000","last_undegraded":"2024-09-16T15:49:20.026542+0000","last_fullsized":"2024-09-16T15:49:20.026542+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:49:18.967562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:49:18.967562+0000","last_clean_scrub_stamp":"2024-09-16T15:49:18.967562+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:45:45.358751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.f","version":"118'1","reported_seq":14,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-09-16T15:49:20.056676+0000","last_change":"2024-09-16T15:49:19.011934+0000","last_active":"2024-09-16T15:49:20.056676+0000","last_peered":"2024-09-16T15:49:20.056676+0000","last_clean":"2024-09-16T15:49:20.056676+0000","last_became_active":"2024-09-16T15:49:19.011176+0000","last_became_peered":"2024-09-16T15:49:19.011176+0000","last_unstale":"2024-09-16T15:49:20.056676+0000","last_undegraded":"2024-09-16T15:49:20.056676+0000","last_fullsized":"2024-09-16T15:49:20.056676+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:49:17.964400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:49:17.964400+0000","last_clean_scrub_stamp":"2024-09-16T15:49:17.964400+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T22:51:13.621888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.f","version":"0'0","reported_seq":10,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-09-16T15:49:20.053874+0000","last_change":"2024-09-16T15:49:20.047290+0000","last_active":"2024-09-16T15:49:20.053874+0000","last_peered":"2024-09-16T15:49:20.053874+0000","last_clean":"2024-09-16T15:49:20.053874+0000","last_became_active":"2024-09-16T15:49:20.045875+0000","last_became_peered":"2024-09-16T15:49:20.045875+0000","last_unstale":"2024-09-16T15:49:20.053874+0000","last_undegraded":"2024-09-16T15:49:20.053874+0000","last_fullsized":"2024-09-16T15:49:20.053874+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:49:18.967562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:49:18.967562+0000","last_clean_scrub_stamp":"2024-09-16T15:49:18.967562+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:55:07.911006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.e","version":"118'2","reported_seq":15,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-09-16T15:49:20.036056+0000","last_change":"2024-09-16T15:49:19.006540+0000","last_active":"2024-09-16T15:49:20.036056+0000","last_peered":"2024-09-16T15:49:20.036056+0000","last_clean":"2024-09-16T15:49:20.036056+0000","last_became_active":"2024-09-16T15:49:19.006157+0000","last_became_peered":"2024-09-16T15:49:19.006157+0000","last_unstale":"2024-09-16T15:49:20.036056+0000","last_undegraded":"2024-09-16T15:49:20.036056+0000","last_fullsized":"2024-09-16T15:49:20.036056+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:49:17.964400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:49:17.964400+0000","last_clean_scrub_stamp":"2024-09-16T15:49:17.964400+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T23:32:33.274644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.c","version":"0'0","reported_seq":10,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-09-16T15:49:20.023091+0000","last_change":"2024-09-16T15:49:20.019570+0000","last_active":"2024-09-16T15:49:20.023091+0000","last_peered":"2024-09-16T15:49:20.023091+0000","last_clean":"2024-09-16T15:49:20.023091+0000","last_became_active":"2024-09-16T15:49:20.019303+0000","last_became_peered":"2024-09-16T15:49:20.019303+0000","last_unstale":"2024-09-16T15:49:20.023091+0000","last_undegraded":"2024-09-16T15:49:20.023091+0000","last_fullsized":"2024-09-16T15:49:20.023091+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:49:18.967562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:49:18.967562+0000","last_clean_scrub_stamp":"2024-09-16T15:49:18.967562+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T22:35:03.465884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.d","version":"118'1","reported_seq":14,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-09-16T15:49:20.028696+0000","last_change":"2024-09-16T15:49:19.005048+0000","last_active":"2024-09-16T15:49:20.028696+0000","last_peered":"2024-09-16T15:49:20.028696+0000","last_clean":"2024-09-16T15:49:20.028696+0000","last_became_active":"2024-09-16T15:49:19.003761+0000","last_became_peered":"2024-09-16T15:49:19.003761+0000","last_unstale":"2024-09-16T15:49:20.028696+0000","last_undegraded":"2024-09-16T15:49:20.028696+0000","last_fullsized":"2024-09-16T15:49:20.028696+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:49:17.964400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:49:17.964400+0000","last_clean_scrub_stamp":"2024-09-16T15:49:17.964400+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T00:06:54.427637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.d","version":"0'0","reported_seq":10,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-09-16T15:49:20.022226+0000","last_change":"2024-09-16T15:49:20.014483+0000","last_active":"2024-09-16T15:49:20.022226+0000","last_peered":"2024-09-16T15:49:20.022226+0000","last_clean":"2024-09-16T15:49:20.022226+0000","last_became_active":"2024-09-16T15:49:20.014250+0000","last_became_peered":"2024-09-16T15:49:20.014250+0000","last_unstale":"2024-09-16T15:49:20.022226+0000","last_undegraded":"2024-09-16T15:49:20.022226+0000","last_fullsized":"2024-09-16T15:49:20.022226+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:49:18.967562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:49:18.967562+0000","last_clean_scrub_stamp":"2024-09-16T15:49:18.967562+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T02:54:45.265261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.c","version":"0'0","reported_seq":13,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-09-16T15:49:20.019003+0000","last_change":"2024-09-16T15:49:18.993103+0000","last_active":"2024-09-16T15:49:20.019003+0000","last_peered":"2024-09-16T15:49:20.019003+0000","last_clean":"2024-09-16T15:49:20.019003+0000","last_became_active":"2024-09-16T15:49:18.992354+0000","last_became_peered":"2024-09-16T15:49:18.992354+0000","last_unstale":"2024-09-16T15:49:20.019003+0000","last_undegraded":"2024-09-16T15:49:20.019003+0000","last_fullsized":"2024-09-16T15:49:20.019003+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:49:17.964400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:49:17.964400+0000","last_clean_scrub_stamp":"2024-09-16T15:49:17.964400+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:36.121363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.a","version":"0'0","reported_seq":10,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-09-16T15:49:20.026407+0000","last_change":"2024-09-16T15:49:20.024410+0000","last_active":"2024-09-16T15:49:20.026407+0000","last_peered":"2024-09-16T15:49:20.026407+0000","last_clean":"2024-09-16T15:49:20.026407+0000","last_became_active":"2024-09-16T15:49:20.018974+0000","last_became_peered":"2024-09-16T15:49:20.018974+0000","last_unstale":"2024-09-16T15:49:20.026407+0000","last_undegraded":"2024-09-16T15:49:20.026407+0000","last_fullsized":"2024-09-16T15:49:20.026407+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:49:18.967562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:49:18.967562+0000","last_clean_scrub_stamp":"2024-09-16T15:49:18.967562+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:35:04.846769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.b","version":"118'2","reported_seq":15,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-09-16T15:49:20.046228+0000","last_change":"2024-09-16T15:49:18.991144+0000","last_active":"2024-09-16T15:49:20.046228+0000","last_peered":"2024-09-16T15:49:20.046228+0000","last_clean":"2024-09-16T15:49:20.046228+0000","last_became_active":"2024-09-16T15:49:18.990393+0000","last_became_peered":"2024-09-16T15:49:18.990393+0000","last_unstale":"2024-09-16T15:49:20.046228+0000","last_undegraded":"2024-09-16T15:49:20.046228+0000","last_fullsized":"2024-09-16T15:49:20.046228+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:49:17.964400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:49:17.964400+0000","last_clean_scrub_stamp":"2024-09-16T15:49:17.964400+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:32:40.744418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.b","version":"0'0","reported_seq":10,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-09-16T15:49:20.038875+0000","last_change":"2024-09-16T15:49:20.033198+0000","last_active":"2024-09-16T15:49:20.038875+0000","last_peered":"2024-09-16T15:49:20.038875+0000","last_clean":"2024-09-16T15:49:20.038875+0000","last_became_active":"2024-09-16T15:49:20.029975+0000","last_became_peered":"2024-09-16T15:49:20.029975+0000","last_unstale":"2024-09-16T15:49:20.038875+0000","last_undegraded":"2024-09-16T15:49:20.038875+0000","last_fullsized":"2024-09-16T15:49:20.038875+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:49:18.967562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:49:18.967562+0000","last_clean_scrub_stamp":"2024-09-16T15:49:18.967562+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T18:06:59.994313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.a","version":"0'0","reported_seq":13,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-09-16T15:49:20.008844+0000","last_change":"2024-09-16T15:49:18.981991+0000","last_active":"2024-09-16T15:49:20.008844+0000","last_peered":"2024-09-16T15:49:20.008844+0000","last_clean":"2024-09-16T15:49:20.008844+0000","last_became_active":"2024-09-16T15:49:18.981604+0000","last_became_peered":"2024-09-16T15:49:18.981604+0000","last_unstale":"2024-09-16T15:49:20.008844+0000","last_undegraded":"2024-09-16T15:49:20.008844+0000","last_fullsized":"2024-09-16T15:49:20.008844+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:49:17.964400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:49:17.964400+0000","last_clean_scrub_stamp":"2024-09-16T15:49:17.964400+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:59:17.318616+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.8","version":"0'0","reported_seq":10,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-09-16T15:49:20.034545+0000","last_change":"2024-09-16T15:49:20.030377+0000","last_active":"2024-09-16T15:49:20.034545+0000","last_peered":"2024-09-16T15:49:20.034545+0000","last_clean":"2024-09-16T15:49:20.034545+0000","last_became_active":"2024-09-16T15:49:20.029971+0000","last_became_peered":"2024-09-16T15:49:20.029971+0000","last_unstale":"2024-09-16T15:49:20.034545+0000","last_undegraded":"2024-09-16T15:49:20.034545+0000","last_fullsized":"2024-09-16T15:49:20.034545+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:49:18.967562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:49:18.967562+0000","last_clean_scrub_stamp":"2024-09-16T15:49:18.967562+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:11:43.275240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.9","version":"0'0","reported_seq":13,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-09-16T15:49:20.009045+0000","last_change":"2024-09-16T15:49:18.987931+0000","last_active":"2024-09-16T15:49:20.009045+0000","last_peered":"2024-09-16T15:49:20.009045+0000","last_clean":"2024-09-16T15:49:20.009045+0000","last_became_active":"2024-09-16T15:49:18.987524+0000","last_became_peered":"2024-09-16T15:49:18.987524+0000","last_unstale":"2024-09-16T15:49:20.009045+0000","last_undegraded":"2024-09-16T15:49:20.009045+0000","last_fullsized":"2024-09-16T15:49:20.009045+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:49:17.964400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:49:17.964400+0000","last_clean_scrub_stamp":"2024-09-16T15:49:17.964400+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:28:01.153450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.9","version":"0'0","reported_seq":10,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-09-16T15:49:20.059149+0000","last_change":"2024-09-16T15:49:20.056767+0000","last_active":"2024-09-16T15:49:20.059149+0000","last_peered":"2024-09-16T15:49:20.059149+0000","last_clean":"2024-09-16T15:49:20.059149+0000","last_became_active":"2024-09-16T15:49:20.048591+0000","last_became_peered":"2024-09-16T15:49:20.048591+0000","last_unstale":"2024-09-16T15:49:20.059149+0000","last_undegraded":"2024-09-16T15:49:20.059149+0000","last_fullsized":"2024-09-16T15:49:20.059149+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:49:18.967562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:49:18.967562+0000","last_clean_scrub_stamp":"2024-09-16T15:49:18.967562+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T02:01:48.218260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.8","version":"0'0","reported_seq":13,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-09-16T15:49:20.008200+0000","last_change":"2024-09-16T15:49:19.006665+0000","last_active":"2024-09-16T15:49:20.008200+0000","last_peered":"2024-09-16T15:49:20.008200+0000","last_clean":"2024-09-16T15:49:20.008200+0000","last_became_active":"2024-09-16T15:49:19.006118+0000","last_became_peered":"2024-09-16T15:49:19.006118+0000","last_unstale":"2024-09-16T15:49:20.008200+0000","last_undegraded":"2024-09-16T15:49:20.008200+0000","last_fullsized":"2024-09-16T15:49:20.008200+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:49:17.964400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:49:17.964400+0000","last_clean_scrub_stamp":"2024-09-16T15:49:17.964400+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T00:00:43.698889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.6","version":"0'0","reported_seq":10,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-09-16T15:49:20.038442+0000","last_change":"2024-09-16T15:49:20.029394+0000","last_active":"2024-09-16T15:49:20.038442+0000","last_peered":"2024-09-16T15:49:20.038442+0000","last_clean":"2024-09-16T15:49:20.038442+0000","last_became_active":"2024-09-16T15:49:20.023195+0000","last_became_peered":"2024-09-16T15:49:20.023195+0000","last_unstale":"2024-09-16T15:49:20.038442+0000","last_undegraded":"2024-09-16T15:49:20.038442+0000","last_fullsized":"2024-09-16T15:49:20.038442+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:49:18.967562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:49:18.967562+0000","last_clean_scrub_stamp":"2024-09-16T15:49:18.967562+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:19:27.788452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.7","version":"118'2","reported_seq":15,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-09-16T15:49:20.047609+0000","last_change":"2024-09-16T15:49:19.000107+0000","last_active":"2024-09-16T15:49:20.047609+0000","last_peered":"2024-09-16T15:49:20.047609+0000","last_clean":"2024-09-16T15:49:20.047609+0000","last_became_active":"2024-09-16T15:49:18.999087+0000","last_became_peered":"2024-09-16T15:49:18.999087+0000","last_unstale":"2024-09-16T15:49:20.047609+0000","last_undegraded":"2024-09-16T15:49:20.047609+0000","last_fullsized":"2024-09-16T15:49:20.047609+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:49:17.964400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:49:17.964400+0000","last_clean_scrub_stamp":"2024-09-16T15:49:17.964400+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:00:34.617481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.7","version":"0'0","reported_seq":10,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-09-16T15:49:20.026264+0000","last_change":"2024-09-16T15:49:20.023965+0000","last_active":"2024-09-16T15:49:20.026264+0000","last_peered":"2024-09-16T15:49:20.026264+0000","last_clean":"2024-09-16T15:49:20.026264+0000","last_became_active":"2024-09-16T15:49:20.022021+0000","last_became_peered":"2024-09-16T15:49:20.022021+0000","last_unstale":"2024-09-16T15:49:20.026264+0000","last_undegraded":"2024-09-16T15:49:20.026264+0000","last_fullsized":"2024-09-16T15:49:20.026264+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:49:18.967562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:49:18.967562+0000","last_clean_scrub_stamp":"2024-09-16T15:49:18.967562+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:13:14.966056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.6","version":"0'0","reported_seq":13,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-09-16T15:49:20.011948+0000","last_change":"2024-09-16T15:49:19.018906+0000","last_active":"2024-09-16T15:49:20.011948+0000","last_peered":"2024-09-16T15:49:20.011948+0000","last_clean":"2024-09-16T15:49:20.011948+0000","last_became_active":"2024-09-16T15:49:19.018066+0000","last_became_peered":"2024-09-16T15:49:19.018066+0000","last_unstale":"2024-09-16T15:49:20.011948+0000","last_undegraded":"2024-09-16T15:49:20.011948+0000","last_fullsized":"2024-09-16T15:49:20.011948+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:49:17.964400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:49:17.964400+0000","last_clean_scrub_stamp":"2024-09-16T15:49:17.964400+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:32.785433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.4","version":"0'0","reported_seq":10,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-09-16T15:49:20.045390+0000","last_change":"2024-09-16T15:49:20.041975+0000","last_active":"2024-09-16T15:49:20.045390+0000","last_peered":"2024-09-16T15:49:20.045390+0000","last_clean":"2024-09-16T15:49:20.045390+0000","last_became_active":"2024-09-16T15:49:20.041602+0000","last_became_peered":"2024-09-16T15:49:20.041602+0000","last_unstale":"2024-09-16T15:49:20.045390+0000","last_undegraded":"2024-09-16T15:49:20.045390+0000","last_fullsized":"2024-09-16T15:49:20.045390+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:49:18.967562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:49:18.967562+0000","last_clean_scrub_stamp":"2024-09-16T15:49:18.967562+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T01:11:05.029917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.5","version":"0'0","reported_seq":13,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-09-16T15:49:19.998114+0000","last_change":"2024-09-16T15:49:19.002146+0000","last_active":"2024-09-16T15:49:19.998114+0000","last_peered":"2024-09-16T15:49:19.998114+0000","last_clean":"2024-09-16T15:49:19.998114+0000","last_became_active":"2024-09-16T15:49:18.997889+0000","last_became_peered":"2024-09-16T15:49:18.997889+0000","last_unstale":"2024-09-16T15:49:19.998114+0000","last_undegraded":"2024-09-16T15:49:19.998114+0000","last_fullsized":"2024-09-16T15:49:19.998114+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:49:17.964400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:49:17.964400+0000","last_clean_scrub_stamp":"2024-09-16T15:49:17.964400+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T22:05:45.261671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.5","version":"0'0","reported_seq":10,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-09-16T15:49:20.036883+0000","last_change":"2024-09-16T15:49:20.021089+0000","last_active":"2024-09-16T15:49:20.036883+0000","last_peered":"2024-09-16T15:49:20.036883+0000","last_clean":"2024-09-16T15:49:20.036883+0000","last_became_active":"2024-09-16T15:49:20.020382+0000","last_became_peered":"2024-09-16T15:49:20.020382+0000","last_unstale":"2024-09-16T15:49:20.036883+0000","last_undegraded":"2024-09-16T15:49:20.036883+0000","last_fullsized":"2024-09-16T15:49:20.036883+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:49:18.967562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:49:18.967562+0000","last_clean_scrub_stamp":"2024-09-16T15:49:18.967562+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:15:25.349686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.4","version":"0'0","reported_seq":13,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-09-16T15:49:20.004842+0000","last_change":"2024-09-16T15:49:18.998714+0000","last_active":"2024-09-16T15:49:20.004842+0000","last_peered":"2024-09-16T15:49:20.004842+0000","last_clean":"2024-09-16T15:49:20.004842+0000","last_became_active":"2024-09-16T15:49:18.998264+0000","last_became_peered":"2024-09-16T15:49:18.998264+0000","last_unstale":"2024-09-16T15:49:20.004842+0000","last_undegraded":"2024-09-16T15:49:20.004842+0000","last_fullsized":"2024-09-16T15:49:20.004842+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:49:17.964400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:49:17.964400+0000","last_clean_scrub_stamp":"2024-09-16T15:49:17.964400+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:45:09.866648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.2","version":"0'0","reported_seq":10,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-09-16T15:49:20.058717+0000","last_change":"2024-09-16T15:49:20.055626+0000","last_active":"2024-09-16T15:49:20.058717+0000","last_peered":"2024-09-16T15:49:20.058717+0000","last_clean":"2024-09-16T15:49:20.058717+0000","last_became_active":"2024-09-16T15:49:20.055267+0000","last_became_peered":"2024-09-16T15:49:20.055267+0000","last_unstale":"2024-09-16T15:49:20.058717+0000","last_undegraded":"2024-09-16T15:49:20.058717+0000","last_fullsized":"2024-09-16T15:49:20.058717+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:49:18.967562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:49:18.967562+0000","last_clean_scrub_stamp":"2024-09-16T15:49:18.967562+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:43:02.183966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.3","version":"0'0","reported_seq":13,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-09-16T15:49:19.997007+0000","last_change":"2024-09-16T15:49:19.010214+0000","last_active":"2024-09-16T15:49:19.997007+0000","last_peered":"2024-09-16T15:49:19.997007+0000","last_clean":"2024-09-16T15:49:19.997007+0000","last_became_active":"2024-09-16T15:49:19.009877+0000","last_became_peered":"2024-09-16T15:49:19.009877+0000","last_unstale":"2024-09-16T15:49:19.997007+0000","last_undegraded":"2024-09-16T15:49:19.997007+0000","last_fullsized":"2024-09-16T15:49:19.997007+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:49:17.964400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:49:17.964400+0000","last_clean_scrub_stamp":"2024-09-16T15:49:17.964400+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:34:09.087537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.3","version":"0'0","reported_seq":10,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-09-16T15:49:20.047499+0000","last_change":"2024-09-16T15:49:20.044650+0000","last_active":"2024-09-16T15:49:20.047499+0000","last_peered":"2024-09-16T15:49:20.047499+0000","last_clean":"2024-09-16T15:49:20.047499+0000","last_became_active":"2024-09-16T15:49:20.043082+0000","last_became_peered":"2024-09-16T15:49:20.043082+0000","last_unstale":"2024-09-16T15:49:20.047499+0000","last_undegraded":"2024-09-16T15:49:20.047499+0000","last_fullsized":"2024-09-16T15:49:20.047499+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:49:18.967562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:49:18.967562+0000","last_clean_scrub_stamp":"2024-09-16T15:49:18.967562+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T22:55:49.755222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.2","version":"0'0","reported_seq":13,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-09-16T15:49:20.036237+0000","last_change":"2024-09-16T15:49:19.016650+0000","last_active":"2024-09-16T15:49:20.036237+0000","last_peered":"2024-09-16T15:49:20.036237+0000","last_clean":"2024-09-16T15:49:20.036237+0000","last_became_active":"2024-09-16T15:49:19.015887+0000","last_became_peered":"2024-09-16T15:49:19.015887+0000","last_unstale":"2024-09-16T15:49:20.036237+0000","last_undegraded":"2024-09-16T15:49:20.036237+0000","last_fullsized":"2024-09-16T15:49:20.036237+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:49:17.964400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:49:17.964400+0000","last_clean_scrub_stamp":"2024-09-16T15:49:17.964400+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:42:48.571248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.0","version":"0'0","reported_seq":10,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-09-16T15:49:20.045877+0000","last_change":"2024-09-16T15:49:20.043111+0000","last_active":"2024-09-16T15:49:20.045877+0000","last_peered":"2024-09-16T15:49:20.045877+0000","last_clean":"2024-09-16T15:49:20.045877+0000","last_became_active":"2024-09-16T15:49:20.042852+0000","last_became_peered":"2024-09-16T15:49:20.042852+0000","last_unstale":"2024-09-16T15:49:20.045877+0000","last_undegraded":"2024-09-16T15:49:20.045877+0000","last_fullsized":"2024-09-16T15:49:20.045877+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:49:18.967562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:49:18.967562+0000","last_clean_scrub_stamp":"2024-09-16T15:49:18.967562+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:02:03.068228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.1","version":"0'0","reported_seq":13,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-09-16T15:49:20.009202+0000","last_change":"2024-09-16T15:49:18.999499+0000","last_active":"2024-09-16T15:49:20.009202+0000","last_peered":"2024-09-16T15:49:20.009202+0000","last_clean":"2024-09-16T15:49:20.009202+0000","last_became_active":"2024-09-16T15:49:18.999058+0000","last_became_peered":"2024-09-16T15:49:18.999058+0000","last_unstale":"2024-09-16T15:49:20.009202+0000","last_undegraded":"2024-09-16T15:49:20.009202+0000","last_fullsized":"2024-09-16T15:49:20.009202+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:49:17.964400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:49:17.964400+0000","last_clean_scrub_stamp":"2024-09-16T15:49:17.964400+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:56:50.721641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.1","version":"0'0","reported_seq":10,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-09-16T15:49:20.046724+0000","last_change":"2024-09-16T15:49:20.045786+0000","last_active":"2024-09-16T15:49:20.046724+0000","last_peered":"2024-09-16T15:49:20.046724+0000","last_clean":"2024-09-16T15:49:20.046724+0000","last_became_active":"2024-09-16T15:49:20.045399+0000","last_became_peered":"2024-09-16T15:49:20.045399+0000","last_unstale":"2024-09-16T15:49:20.046724+0000","last_undegraded":"2024-09-16T15:49:20.046724+0000","last_fullsized":"2024-09-16T15:49:20.046724+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:49:18.967562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:49:18.967562+0000","last_clean_scrub_stamp":"2024-09-16T15:49:18.967562+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T18:10:58.632085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.0","version":"0'0","reported_seq":13,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-09-16T15:49:20.008451+0000","last_change":"2024-09-16T15:49:19.004933+0000","last_active":"2024-09-16T15:49:20.008451+0000","last_peered":"2024-09-16T15:49:20.008451+0000","last_clean":"2024-09-16T15:49:20.008451+0000","last_became_active":"2024-09-16T15:49:19.004577+0000","last_became_peered":"2024-09-16T15:49:19.004577+0000","last_unstale":"2024-09-16T15:49:20.008451+0000","last_undegraded":"2024-09-16T15:49:20.008451+0000","last_fullsized":"2024-09-16T15:49:20.008451+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:49:17.964400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:49:17.964400+0000","last_clean_scrub_stamp":"2024-09-16T15:49:17.964400+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:59:29.457235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.12","version":"0'0","reported_seq":10,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-09-16T15:49:20.058332+0000","last_change":"2024-09-16T15:49:20.049202+0000","last_active":"2024-09-16T15:49:20.058332+0000","last_peered":"2024-09-16T15:49:20.058332+0000","last_clean":"2024-09-16T15:49:20.058332+0000","last_became_active":"2024-09-16T15:49:20.048362+0000","last_became_peered":"2024-09-16T15:49:20.048362+0000","last_unstale":"2024-09-16T15:49:20.058332+0000","last_undegraded":"2024-09-16T15:49:20.058332+0000","last_fullsized":"2024-09-16T15:49:20.058332+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:49:18.967562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:49:18.967562+0000","last_clean_scrub_stamp":"2024-09-16T15:49:18.967562+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:11:48.910216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.13","version":"118'2","reported_seq":15,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-09-16T15:49:20.049481+0000","last_change":"2024-09-16T15:49:19.014597+0000","last_active":"2024-09-16T15:49:20.049481+0000","last_peered":"2024-09-16T15:49:20.049481+0000","last_clean":"2024-09-16T15:49:20.049481+0000","last_became_active":"2024-09-16T15:49:19.010375+0000","last_became_peered":"2024-09-16T15:49:19.010375+0000","last_unstale":"2024-09-16T15:49:20.049481+0000","last_undegraded":"2024-09-16T15:49:20.049481+0000","last_fullsized":"2024-09-16T15:49:20.049481+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:49:17.964400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:49:17.964400+0000","last_clean_scrub_stamp":"2024-09-16T15:49:17.964400+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T00:21:03.027521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"73'79","reported_seq":319,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-09-16T15:49:20.006250+0000","last_change":"2024-09-16T15:42:40.881820+0000","last_active":"2024-09-16T15:49:20.006250+0000","last_peered":"2024-09-16T15:49:20.006250+0000","last_clean":"2024-09-16T15:49:20.006250+0000","last_became_active":"2024-09-16T15:42:40.879965+0000","last_became_peered":"2024-09-16T15:42:40.879965+0000","last_unstale":"2024-09-16T15:49:20.006250+0000","last_undegraded":"2024-09-16T15:49:20.006250+0000","last_fullsized":"2024-09-16T15:49:20.006250+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:42:39.728762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:42:39.728762+0000","last_clean_scrub_stamp":"2024-09-16T15:42:39.728762+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:22:18.768932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":19,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":18,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79,"ondisk_log_size":79,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542228,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33192,"kb_used_data":2004,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338648,"statfs":{"total":96636764160,"available":96602775552,"internally_reserved":0,"allocated":2052096,"data_stored":1230822,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542227,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32636,"kb_used_data":1440,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339204,"statfs":{"total":96636764160,"available":96603344896,"internally_reserved":0,"allocated":1474560,"data_stored":647347,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":5,"up_from":14,"seq":60129542228,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32616,"kb_used_data":1424,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339224,"statfs":{"total":96636764160,"available":96603365376,"internally_reserved":0,"allocated":1458176,"data_stored":640454,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542226,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32612,"kb_used_data":1424,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339228,"statfs":{"total":96636764160,"available":96603369472,"internally_reserved":0,"allocated":1458176,"data_stored":640454,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":3,"up_from":14,"seq":60129542227,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32616,"kb_used_data":1424,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339224,"statfs":{"total":96636764160,"available":96603365376,"internally_reserved":0,"allocated":1458176,"data_stored":640454,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":14,"seq":60129542227,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32616,"kb_used_data":1424,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339224,"statfs":{"total":96636764160,"available":96603365376,"internally_reserved":0,"allocated":1458176,"data_stored":640454,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":14,"seq":60129542229,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32616,"kb_used_data":1424,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339224,"statfs":{"total":96636764160,"available":96603365376,"internally_reserved":0,"allocated":1458176,"data_stored":640454,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":14,"seq":60129542229,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33192,"kb_used_data":2004,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338648,"statfs":{"total":96636764160,"available":96602775552,"internally_reserved":0,"allocated":2052096,"data_stored":1230822,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":2250,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T15:49:24.464 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-09-16T15:49:24.464 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-09-16T15:49:24.643 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:24.643+0000 7fc950112640 1 -- 172.21.15.123:0/2182322991 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc948072c70 msgr2=0x7fc9480cc3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:24.643 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:24.643+0000 7fc950112640 1 --2- 172.21.15.123:0/2182322991 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc948072c70 0x7fc9480cc3c0 secure :-1 s=READY pgs=1270 cs=0 l=1 rev1=1 crypto rx=0x7fc93c009840 tx=0x7fc93c01c760 comp rx=0 tx=0).stop 2024-09-16T15:49:24.643 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:24.643+0000 7fc950112640 1 -- 172.21.15.123:0/2182322991 shutdown_connections 2024-09-16T15:49:24.643 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:24.643+0000 7fc950112640 1 --2- 172.21.15.123:0/2182322991 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc948072c70 0x7fc9480cc3c0 unknown :-1 s=CLOSED pgs=1270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:24.643 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:24.643+0000 7fc950112640 1 -- 172.21.15.123:0/2182322991 >> 172.21.15.123:0/2182322991 conn(0x7fc9480b1710 msgr2=0x7fc9480b1b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:49:24.643 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:24.643+0000 7fc950112640 1 -- 172.21.15.123:0/2182322991 shutdown_connections 2024-09-16T15:49:24.644 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:24.643+0000 7fc950112640 1 -- 172.21.15.123:0/2182322991 wait complete. 2024-09-16T15:49:24.644 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:24.643+0000 7fc950112640 1 Processor -- start 2024-09-16T15:49:24.644 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:24.643+0000 7fc950112640 1 -- start start 2024-09-16T15:49:24.647 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:24.647+0000 7fc950112640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc948072720 0x7fc948144140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:49:24.647 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:24.647+0000 7fc950112640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fc948072c70 0x7fc9480c59e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:49:24.647 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:24.647+0000 7fc950112640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc94815a9d0 0x7fc9481f9990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:49:24.647 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:24.647+0000 7fc950112640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fc9481695a0 con 0x7fc948072720 2024-09-16T15:49:24.647 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:24.647+0000 7fc950112640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fc948169420 con 0x7fc94815a9d0 2024-09-16T15:49:24.647 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:24.647+0000 7fc950112640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fc948169720 con 0x7fc948072c70 2024-09-16T15:49:24.647 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:24.647+0000 7fc94deae640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc948072720 0x7fc948144140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:49:24.647 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:24.647+0000 7fc94deae640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc948072720 0x7fc948144140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:35456/0 (socket says 172.21.15.123:35456) 2024-09-16T15:49:24.647 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:24.647+0000 7fc94deae640 1 -- 172.21.15.123:0/2608229841 learned_addr learned my addr 172.21.15.123:0/2608229841 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:49:24.647 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:24.647+0000 7fc94d6ad640 1 --2- 172.21.15.123:0/2608229841 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fc948072c70 0x7fc9480c59e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:49:24.647 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:24.647+0000 7fc94deae640 1 -- 172.21.15.123:0/2608229841 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fc948072c70 msgr2=0x7fc9480c59e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:24.647 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:24.647+0000 7fc94deae640 1 --2- 172.21.15.123:0/2608229841 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fc948072c70 0x7fc9480c59e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:24.648 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:24.647+0000 7fc94deae640 1 -- 172.21.15.123:0/2608229841 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc94815a9d0 msgr2=0x7fc9481f9990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:24.648 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:24.647+0000 7fc94e6af640 1 --2- 172.21.15.123:0/2608229841 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc94815a9d0 0x7fc9481f9990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:49:24.648 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:24.647+0000 7fc94deae640 1 --2- 172.21.15.123:0/2608229841 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc94815a9d0 0x7fc9481f9990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:24.648 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:24.647+0000 7fc94deae640 1 -- 172.21.15.123:0/2608229841 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc948144680 con 0x7fc948072720 2024-09-16T15:49:24.648 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:24.647+0000 7fc94e6af640 1 --2- 172.21.15.123:0/2608229841 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc94815a9d0 0x7fc9481f9990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T15:49:24.648 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:24.647+0000 7fc94deae640 1 --2- 172.21.15.123:0/2608229841 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc948072720 0x7fc948144140 secure :-1 s=READY pgs=1271 cs=0 l=1 rev1=1 crypto rx=0x7fc94000c450 tx=0x7fc94000c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:49:24.648 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:24.647+0000 7fc9467fc640 1 -- 172.21.15.123:0/2608229841 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc940011790 con 0x7fc948072720 2024-09-16T15:49:24.648 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:24.647+0000 7fc9467fc640 1 -- 172.21.15.123:0/2608229841 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fc940017070 con 0x7fc948072720 2024-09-16T15:49:24.648 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:24.647+0000 7fc9467fc640 1 -- 172.21.15.123:0/2608229841 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc940011ce0 con 0x7fc948072720 2024-09-16T15:49:24.648 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:24.647+0000 7fc950112640 1 -- 172.21.15.123:0/2608229841 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc9481fa300 con 0x7fc948072720 2024-09-16T15:49:24.648 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:24.647+0000 7fc950112640 1 -- 172.21.15.123:0/2608229841 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc948143940 con 0x7fc948072720 2024-09-16T15:49:24.648 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:24.647+0000 7fc950112640 1 -- 172.21.15.123:0/2608229841 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc948143750 con 0x7fc948072720 2024-09-16T15:49:24.649 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:24.647+0000 7fc9467fc640 1 -- 172.21.15.123:0/2608229841 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 19) ==== 97488+0+0 (secure 0 0 0) 0x7fc940007020 con 0x7fc948072720 2024-09-16T15:49:24.649 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:24.647+0000 7fc9467fc640 1 --2- 172.21.15.123:0/2608229841 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fc934073370 0x7fc934075830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:49:24.649 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:24.651+0000 7fc9467fc640 1 -- 172.21.15.123:0/2608229841 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(119..119 src has 1..119) ==== 5737+0+0 (secure 0 0 0) 0x7fc940094b30 con 0x7fc948072720 2024-09-16T15:49:24.649 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:24.651+0000 7fc94d6ad640 1 --2- 172.21.15.123:0/2608229841 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fc934073370 0x7fc934075830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:49:24.649 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:24.651+0000 7fc94d6ad640 1 --2- 172.21.15.123:0/2608229841 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fc934073370 0x7fc934075830 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fc93c0af000 tx=0x7fc93c01afa0 comp rx=0 tx=0).ready entity=mgr.4150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:49:24.652 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:24.651+0000 7fc9467fc640 1 -- 172.21.15.123:0/2608229841 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7fc94005f750 con 0x7fc948072720 2024-09-16T15:49:24.870 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:24.871+0000 7fc950112640 1 -- 172.21.15.123:0/2608229841 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) -- 0x7fc9480cc3c0 con 0x7fc948072720 2024-09-16T15:49:24.873 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:24.875+0000 7fc9467fc640 1 -- 172.21.15.123:0/2608229841 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v25) ==== 190+0+147 (secure 0 0 0) 0x7fc940066e20 con 0x7fc948072720 2024-09-16T15:49:24.874 INFO:teuthology.orchestra.run.smithi123.stdout:[client.0] 2024-09-16T15:49:24.874 INFO:teuthology.orchestra.run.smithi123.stdout: key = AQC1Uehm7GDTAhAAHWWVIyjVVnTeeozBR8DfFA== 2024-09-16T15:49:24.874 INFO:teuthology.orchestra.run.smithi123.stdout: caps mds = "allow" 2024-09-16T15:49:24.874 INFO:teuthology.orchestra.run.smithi123.stdout: caps mon = "allow r" 2024-09-16T15:49:24.874 INFO:teuthology.orchestra.run.smithi123.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-09-16T15:49:24.874 INFO:teuthology.orchestra.run.smithi123.stderr:updated caps for client.0 2024-09-16T15:49:24.876 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:24.875+0000 7fc950112640 1 -- 172.21.15.123:0/2608229841 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fc934073370 msgr2=0x7fc934075830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:24.876 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:24.875+0000 7fc950112640 1 --2- 172.21.15.123:0/2608229841 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fc934073370 0x7fc934075830 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fc93c0af000 tx=0x7fc93c01afa0 comp rx=0 tx=0).stop 2024-09-16T15:49:24.876 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:24.875+0000 7fc950112640 1 -- 172.21.15.123:0/2608229841 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc948072720 msgr2=0x7fc948144140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:24.876 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:24.875+0000 7fc950112640 1 --2- 172.21.15.123:0/2608229841 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc948072720 0x7fc948144140 secure :-1 s=READY pgs=1271 cs=0 l=1 rev1=1 crypto rx=0x7fc94000c450 tx=0x7fc94000c910 comp rx=0 tx=0).stop 2024-09-16T15:49:24.876 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:24.875+0000 7fc950112640 1 -- 172.21.15.123:0/2608229841 shutdown_connections 2024-09-16T15:49:24.876 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:24.875+0000 7fc950112640 1 --2- 172.21.15.123:0/2608229841 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fc934073370 0x7fc934075830 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:24.876 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:24.875+0000 7fc950112640 1 --2- 172.21.15.123:0/2608229841 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc94815a9d0 0x7fc9481f9990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:24.877 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:24.875+0000 7fc950112640 1 --2- 172.21.15.123:0/2608229841 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fc948072c70 0x7fc9480c59e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:24.877 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:24.875+0000 7fc950112640 1 --2- 172.21.15.123:0/2608229841 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc948072720 0x7fc948144140 unknown :-1 s=CLOSED pgs=1271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:24.877 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:24.875+0000 7fc950112640 1 -- 172.21.15.123:0/2608229841 >> 172.21.15.123:0/2608229841 conn(0x7fc9480b1710 msgr2=0x7fc9480d0340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:49:24.877 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:24.875+0000 7fc950112640 1 -- 172.21.15.123:0/2608229841 shutdown_connections 2024-09-16T15:49:24.877 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:24.875+0000 7fc950112640 1 -- 172.21.15.123:0/2608229841 wait complete. 2024-09-16T15:49:24.891 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-16T15:49:25.001 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:25.003+0000 7f1dd038a640 1 -- 172.21.15.123:0/2613950751 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1dc815e160 msgr2=0x7f1dc8073f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:25.001 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:25.003+0000 7f1dd038a640 1 --2- 172.21.15.123:0/2613950751 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1dc815e160 0x7f1dc8073f30 secure :-1 s=READY pgs=1273 cs=0 l=1 rev1=1 crypto rx=0x7f1dc0009840 tx=0x7f1dc001c700 comp rx=0 tx=0).stop 2024-09-16T15:49:25.001 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:25.003+0000 7f1dd038a640 1 -- 172.21.15.123:0/2613950751 shutdown_connections 2024-09-16T15:49:25.001 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:25.003+0000 7f1dd038a640 1 --2- 172.21.15.123:0/2613950751 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f1dc815e820 0x7f1dc815ecb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:25.001 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:25.003+0000 7f1dd038a640 1 --2- 172.21.15.123:0/2613950751 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1dc815e160 0x7f1dc8073f30 unknown :-1 s=CLOSED pgs=1273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:25.001 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:25.003+0000 7f1dd038a640 1 -- 172.21.15.123:0/2613950751 >> 172.21.15.123:0/2613950751 conn(0x7f1dc80cfb10 msgr2=0x7f1dc80cff20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:49:25.002 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:25.003+0000 7f1dd038a640 1 -- 172.21.15.123:0/2613950751 shutdown_connections 2024-09-16T15:49:25.002 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:25.003+0000 7f1dd038a640 1 -- 172.21.15.123:0/2613950751 wait complete. 2024-09-16T15:49:25.002 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:25.003+0000 7f1dd038a640 1 Processor -- start 2024-09-16T15:49:25.002 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:25.003+0000 7f1dd038a640 1 -- start start 2024-09-16T15:49:25.004 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:25.003+0000 7f1dd038a640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1dc8074470 0x7f1dc815d4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:49:25.004 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:25.003+0000 7f1dd038a640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f1dc815e160 0x7f1dc81567e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:49:25.004 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:25.003+0000 7f1dd038a640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1dc815e820 0x7f1dc81f7940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:49:25.004 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:25.003+0000 7f1dd038a640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f1dc80cf5b0 con 0x7f1dc815e820 2024-09-16T15:49:25.004 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:25.003+0000 7f1dd038a640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f1dc80cf430 con 0x7f1dc8074470 2024-09-16T15:49:25.004 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:25.003+0000 7f1dd038a640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f1dc81683d0 con 0x7f1dc815e160 2024-09-16T15:49:25.004 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:25.003+0000 7f1dce927640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1dc815e820 0x7f1dc81f7940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:49:25.004 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:25.003+0000 7f1dce927640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1dc815e820 0x7f1dc81f7940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:35472/0 (socket says 172.21.15.123:35472) 2024-09-16T15:49:25.005 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:25.003+0000 7f1dce927640 1 -- 172.21.15.123:0/1788487879 learned_addr learned my addr 172.21.15.123:0/1788487879 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:49:25.005 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:25.003+0000 7f1dce126640 1 --2- 172.21.15.123:0/1788487879 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1dc8074470 0x7f1dc815d4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:49:25.005 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:25.003+0000 7f1dcd925640 1 --2- 172.21.15.123:0/1788487879 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f1dc815e160 0x7f1dc81567e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:49:25.005 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:25.003+0000 7f1dce927640 1 -- 172.21.15.123:0/1788487879 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f1dc815e160 msgr2=0x7f1dc81567e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:25.005 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:25.003+0000 7f1dce927640 1 --2- 172.21.15.123:0/1788487879 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f1dc815e160 0x7f1dc81567e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:25.005 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:25.003+0000 7f1dce927640 1 -- 172.21.15.123:0/1788487879 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1dc8074470 msgr2=0x7f1dc815d4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:25.005 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:25.003+0000 7f1dce927640 1 --2- 172.21.15.123:0/1788487879 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1dc8074470 0x7f1dc815d4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:25.005 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:25.003+0000 7f1dce927640 1 -- 172.21.15.123:0/1788487879 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1dc815c120 con 0x7f1dc815e820 2024-09-16T15:49:25.005 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:25.003+0000 7f1dce927640 1 --2- 172.21.15.123:0/1788487879 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1dc815e820 0x7f1dc81f7940 secure :-1 s=READY pgs=1274 cs=0 l=1 rev1=1 crypto rx=0x7f1db000ca60 tx=0x7f1db000cf20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:49:25.005 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:25.003+0000 7f1dc6ffd640 1 -- 172.21.15.123:0/1788487879 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1db0007d90 con 0x7f1dc815e820 2024-09-16T15:49:25.005 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:25.003+0000 7f1dc6ffd640 1 -- 172.21.15.123:0/1788487879 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f1db0004c20 con 0x7f1dc815e820 2024-09-16T15:49:25.005 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:25.003+0000 7f1dd038a640 1 -- 172.21.15.123:0/1788487879 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1dc815c400 con 0x7f1dc815e820 2024-09-16T15:49:25.005 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:25.003+0000 7f1dc6ffd640 1 -- 172.21.15.123:0/1788487879 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1db0005280 con 0x7f1dc815e820 2024-09-16T15:49:25.005 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:25.003+0000 7f1dd038a640 1 -- 172.21.15.123:0/1788487879 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1dc815c9c0 con 0x7f1dc815e820 2024-09-16T15:49:25.006 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:25.007+0000 7f1dd038a640 1 -- 172.21.15.123:0/1788487879 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1dc80d06c0 con 0x7f1dc815e820 2024-09-16T15:49:25.006 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:25.007+0000 7f1dc6ffd640 1 -- 172.21.15.123:0/1788487879 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 19) ==== 97488+0+0 (secure 0 0 0) 0x7f1db0005420 con 0x7f1dc815e820 2024-09-16T15:49:25.006 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:25.007+0000 7f1dc6ffd640 1 --2- 172.21.15.123:0/1788487879 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f1da8073340 0x7f1da8075800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:49:25.007 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:25.007+0000 7f1dce126640 1 --2- 172.21.15.123:0/1788487879 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f1da8073340 0x7f1da8075800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:49:25.007 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:25.007+0000 7f1dc6ffd640 1 -- 172.21.15.123:0/1788487879 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(119..119 src has 1..119) ==== 5737+0+0 (secure 0 0 0) 0x7f1db0094e90 con 0x7f1dc815e820 2024-09-16T15:49:25.007 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:25.007+0000 7f1dce126640 1 --2- 172.21.15.123:0/1788487879 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f1da8073340 0x7f1da8075800 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f1dc00af000 tx=0x7f1dc001af40 comp rx=0 tx=0).ready entity=mgr.4150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:49:25.010 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:25.011+0000 7f1dc6ffd640 1 -- 172.21.15.123:0/1788487879 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f1db0014030 con 0x7f1dc815e820 2024-09-16T15:49:25.213 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:25.215+0000 7f1dd038a640 1 -- 172.21.15.123:0/1788487879 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f1dc8073f10 con 0x7f1dc815e820 2024-09-16T15:49:25.214 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:25.215+0000 7f1dc6ffd640 1 -- 172.21.15.123:0/1788487879 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 76 v76) ==== 94+0+5234 (secure 0 0 0) 0x7f1db005f980 con 0x7f1dc815e820 2024-09-16T15:49:25.214 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-09-16T15:49:25.215 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":76,"btime":"2024-09-16T15:49:23:031064+0000","default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6079,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6833/3932980815","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":3932980815},{"type":"v1","addr":"172.21.15.184:6833","nonce":3932980815}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":72},{"gid":6082,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6835/1809728014","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":1809728014},{"type":"v1","addr":"172.21.15.184:6835","nonce":1809728014}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":72},{"gid":7551,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6834/892095590","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6832","nonce":892095590},{"type":"v1","addr":"172.21.15.123:6834","nonce":892095590}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":72}],"filesystems":[{"mdsmap":{"epoch":76,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-16T15:49:19.361823+0000","modified":"2024-09-16T15:49:22.409036+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7554},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7554":{"gid":7554,"name":"a","rank":0,"incarnation":73,"state":"up:active","state_seq":3,"addr":"172.21.15.123:6835/2110793616","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6833","nonce":2110793616},{"type":"v1","addr":"172.21.15.123:6835","nonce":2110793616}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7554,"qdb_cluster":[7554]},"id":9}]} 2024-09-16T15:49:25.215 INFO:teuthology.orchestra.run.smithi123.stderr:dumped fsmap epoch 76 2024-09-16T15:49:25.217 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:25.215+0000 7f1dd038a640 1 -- 172.21.15.123:0/1788487879 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f1da8073340 msgr2=0x7f1da8075800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:25.217 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:25.215+0000 7f1dd038a640 1 --2- 172.21.15.123:0/1788487879 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f1da8073340 0x7f1da8075800 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f1dc00af000 tx=0x7f1dc001af40 comp rx=0 tx=0).stop 2024-09-16T15:49:25.217 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:25.215+0000 7f1dd038a640 1 -- 172.21.15.123:0/1788487879 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1dc815e820 msgr2=0x7f1dc81f7940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:25.217 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:25.215+0000 7f1dd038a640 1 --2- 172.21.15.123:0/1788487879 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1dc815e820 0x7f1dc81f7940 secure :-1 s=READY pgs=1274 cs=0 l=1 rev1=1 crypto rx=0x7f1db000ca60 tx=0x7f1db000cf20 comp rx=0 tx=0).stop 2024-09-16T15:49:25.217 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:25.215+0000 7f1dd038a640 1 -- 172.21.15.123:0/1788487879 shutdown_connections 2024-09-16T15:49:25.217 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:25.215+0000 7f1dd038a640 1 --2- 172.21.15.123:0/1788487879 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f1da8073340 0x7f1da8075800 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:25.217 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:25.215+0000 7f1dd038a640 1 --2- 172.21.15.123:0/1788487879 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1dc815e820 0x7f1dc81f7940 unknown :-1 s=CLOSED pgs=1274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:25.217 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:25.215+0000 7f1dd038a640 1 --2- 172.21.15.123:0/1788487879 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f1dc815e160 0x7f1dc81567e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:25.217 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:25.215+0000 7f1dd038a640 1 --2- 172.21.15.123:0/1788487879 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1dc8074470 0x7f1dc815d4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:25.217 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:25.219+0000 7f1dd038a640 1 -- 172.21.15.123:0/1788487879 >> 172.21.15.123:0/1788487879 conn(0x7f1dc80cfb10 msgr2=0x7f1dc80c5ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:49:25.218 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:25.219+0000 7f1dd038a640 1 -- 172.21.15.123:0/1788487879 shutdown_connections 2024-09-16T15:49:25.218 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:25.219+0000 7f1dd038a640 1 -- 172.21.15.123:0/1788487879 wait complete. 2024-09-16T15:49:25.292 INFO:tasks.ceph.mds.a.smithi123.stderr:2024-09-16T15:49:25.291+0000 7fe839ff0640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-09-16T15:49:25.636 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 76, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-09-16T15:49:19.361823+0000', 'modified': '2024-09-16T15:49:22.409036+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 7554}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7554': {'gid': 7554, 'name': 'a', 'rank': 0, 'incarnation': 73, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.123:6835/2110793616', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.123:6833', 'nonce': 2110793616}, {'type': 'v1', 'addr': '172.21.15.123:6835', 'nonce': 2110793616}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540719139924082687, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 7554, 'qdb_cluster': [7554]} 2024-09-16T15:49:25.636 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-09-16T15:49:25.636 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2024-09-16T15:49:25.771 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:25.771+0000 7f7b3a2f9640 1 -- 172.21.15.123:0/2527866603 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7b200051d0 msgr2=0x7f7b2000b9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:25.772 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:25.771+0000 7f7b3a2f9640 1 --2- 172.21.15.123:0/2527866603 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7b200051d0 0x7f7b2000b9d0 secure :-1 s=READY pgs=1276 cs=0 l=1 rev1=1 crypto rx=0x7f7b240016f0 tx=0x7f7b2401c7b0 comp rx=0 tx=0).stop 2024-09-16T15:49:25.772 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:25.771+0000 7f7b3a2f9640 1 -- 172.21.15.123:0/2527866603 shutdown_connections 2024-09-16T15:49:25.772 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:25.771+0000 7f7b3a2f9640 1 --2- 172.21.15.123:0/2527866603 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7b200051d0 0x7f7b2000b9d0 unknown :-1 s=CLOSED pgs=1276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:25.772 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:25.771+0000 7f7b3a2f9640 1 -- 172.21.15.123:0/2527866603 >> 172.21.15.123:0/2527866603 conn(0x7f7b3409aa40 msgr2=0x7f7b3409ae50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:49:25.772 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:25.771+0000 7f7b3a2f9640 1 -- 172.21.15.123:0/2527866603 shutdown_connections 2024-09-16T15:49:25.772 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:25.771+0000 7f7b3a2f9640 1 -- 172.21.15.123:0/2527866603 wait complete. 2024-09-16T15:49:25.773 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:25.771+0000 7f7b3a2f9640 1 Processor -- start 2024-09-16T15:49:25.773 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:25.771+0000 7f7b3a2f9640 1 -- start start 2024-09-16T15:49:25.775 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:25.775+0000 7f7b3a2f9640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7b200051d0 0x7f7b34159600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:49:25.775 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:25.775+0000 7f7b3a2f9640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7b34157680 0x7f7b34155500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:49:25.775 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:25.775+0000 7f7b3a2f9640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f7b341f6030 0x7f7b340b6cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:49:25.775 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:25.775+0000 7f7b3a2f9640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f7b340d1e80 con 0x7f7b200051d0 2024-09-16T15:49:25.775 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:25.775+0000 7f7b3a2f9640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f7b340d1d00 con 0x7f7b34157680 2024-09-16T15:49:25.775 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:25.775+0000 7f7b3a2f9640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f7b340b2350 con 0x7f7b341f6030 2024-09-16T15:49:25.775 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:25.775+0000 7f7b33fff640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7b200051d0 0x7f7b34159600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:49:25.775 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:25.775+0000 7f7b33fff640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7b200051d0 0x7f7b34159600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:35484/0 (socket says 172.21.15.123:35484) 2024-09-16T15:49:25.775 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:25.775+0000 7f7b33fff640 1 -- 172.21.15.123:0/1645735062 learned_addr learned my addr 172.21.15.123:0/1645735062 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:49:25.775 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:25.775+0000 7f7b38896640 1 --2- 172.21.15.123:0/1645735062 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f7b341f6030 0x7f7b340b6cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:49:25.775 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:25.775+0000 7f7b337fe640 1 --2- 172.21.15.123:0/1645735062 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7b34157680 0x7f7b34155500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:49:25.776 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:25.775+0000 7f7b33fff640 1 -- 172.21.15.123:0/1645735062 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f7b341f6030 msgr2=0x7f7b340b6cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:25.776 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:25.775+0000 7f7b33fff640 1 --2- 172.21.15.123:0/1645735062 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f7b341f6030 0x7f7b340b6cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:25.776 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:25.775+0000 7f7b33fff640 1 -- 172.21.15.123:0/1645735062 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7b34157680 msgr2=0x7f7b34155500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:25.776 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:25.775+0000 7f7b33fff640 1 --2- 172.21.15.123:0/1645735062 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7b34157680 0x7f7b34155500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:25.776 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:25.775+0000 7f7b33fff640 1 -- 172.21.15.123:0/1645735062 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7b3415c310 con 0x7f7b200051d0 2024-09-16T15:49:25.776 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:25.775+0000 7f7b38896640 1 --2- 172.21.15.123:0/1645735062 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f7b341f6030 0x7f7b340b6cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T15:49:25.776 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:25.775+0000 7f7b337fe640 1 --2- 172.21.15.123:0/1645735062 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7b34157680 0x7f7b34155500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T15:49:25.776 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:25.775+0000 7f7b33fff640 1 --2- 172.21.15.123:0/1645735062 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7b200051d0 0x7f7b34159600 secure :-1 s=READY pgs=1277 cs=0 l=1 rev1=1 crypto rx=0x7f7b2c00fc00 tx=0x7f7b2c00d6b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:49:25.776 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:25.775+0000 7f7b30ff9640 1 -- 172.21.15.123:0/1645735062 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b2c011640 con 0x7f7b200051d0 2024-09-16T15:49:25.776 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:25.775+0000 7f7b30ff9640 1 -- 172.21.15.123:0/1645735062 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f7b2c0117e0 con 0x7f7b200051d0 2024-09-16T15:49:25.776 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:25.775+0000 7f7b3a2f9640 1 -- 172.21.15.123:0/1645735062 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7b3415c5b0 con 0x7f7b200051d0 2024-09-16T15:49:25.776 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:25.775+0000 7f7b30ff9640 1 -- 172.21.15.123:0/1645735062 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b2c011c30 con 0x7f7b200051d0 2024-09-16T15:49:25.776 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:25.775+0000 7f7b3a2f9640 1 -- 172.21.15.123:0/1645735062 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7b34159b40 con 0x7f7b200051d0 2024-09-16T15:49:25.779 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:25.775+0000 7f7b30ff9640 1 -- 172.21.15.123:0/1645735062 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 19) ==== 97488+0+0 (secure 0 0 0) 0x7f7b2c024440 con 0x7f7b200051d0 2024-09-16T15:49:25.779 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:25.779+0000 7f7b30ff9640 1 --2- 172.21.15.123:0/1645735062 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f7b14073370 0x7f7b14075830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:49:25.779 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:25.779+0000 7f7b337fe640 1 --2- 172.21.15.123:0/1645735062 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f7b14073370 0x7f7b14075830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:49:25.779 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:25.779+0000 7f7b30ff9640 1 -- 172.21.15.123:0/1645735062 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(119..119 src has 1..119) ==== 5737+0+0 (secure 0 0 0) 0x7f7b2c095440 con 0x7f7b200051d0 2024-09-16T15:49:25.779 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:25.779+0000 7f7b337fe640 1 --2- 172.21.15.123:0/1645735062 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f7b14073370 0x7f7b14075830 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f7b1c000fc0 tx=0x7f7b1c001120 comp rx=0 tx=0).ready entity=mgr.4150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:49:25.780 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:25.779+0000 7f7b3d3e3000 1 Processor -- start 2024-09-16T15:49:25.780 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:25.779+0000 7f7b3d3e3000 1 -- start start 2024-09-16T15:49:25.780 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:25.779+0000 7f7b3d3e3000 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ee9e8af9d0 0x55ee9e8afdb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:49:25.780 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:25.779+0000 7f7b3d3e3000 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55ee9e8b02f0 0x55ee9e8638d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:49:25.780 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:25.779+0000 7f7b3d3e3000 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x55ee9e863e10 0x55ee9e8641f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:49:25.780 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:25.779+0000 7f7b3d3e3000 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x55ee9e7843f0 con 0x55ee9e8b02f0 2024-09-16T15:49:25.780 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:25.779+0000 7f7b3d3e3000 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x55ee9e6c1b20 con 0x55ee9e8af9d0 2024-09-16T15:49:25.780 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:25.779+0000 7f7b3d3e3000 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x55ee9e6b0f50 con 0x55ee9e863e10 2024-09-16T15:49:25.780 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:25.779+0000 7f7b337fe640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55ee9e8b02f0 0x55ee9e8638d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:49:25.780 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:25.779+0000 7f7b337fe640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55ee9e8b02f0 0x55ee9e8638d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:35492/0 (socket says 172.21.15.123:35492) 2024-09-16T15:49:25.780 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:25.779+0000 7f7b337fe640 1 -- 172.21.15.123:0/1597718549 learned_addr learned my addr 172.21.15.123:0/1597718549 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:49:25.780 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:25.779+0000 7f7b38896640 1 --2- 172.21.15.123:0/1597718549 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x55ee9e863e10 0x55ee9e8641f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:49:25.781 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:25.779+0000 7f7b337fe640 1 -- 172.21.15.123:0/1597718549 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x55ee9e863e10 msgr2=0x55ee9e8641f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T15:49:25.781 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:25.779+0000 7f7b337fe640 1 --2- 172.21.15.123:0/1597718549 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x55ee9e863e10 0x55ee9e8641f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:25.781 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:25.779+0000 7f7b337fe640 1 -- 172.21.15.123:0/1597718549 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ee9e8af9d0 msgr2=0x55ee9e8afdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T15:49:25.781 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:25.779+0000 7f7b337fe640 1 --2- 172.21.15.123:0/1597718549 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ee9e8af9d0 0x55ee9e8afdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:25.781 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:25.779+0000 7f7b337fe640 1 -- 172.21.15.123:0/1597718549 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ee9e678590 con 0x55ee9e8b02f0 2024-09-16T15:49:25.781 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:25.779+0000 7f7b337fe640 1 --2- 172.21.15.123:0/1597718549 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55ee9e8b02f0 0x55ee9e8638d0 secure :-1 s=READY pgs=1278 cs=0 l=1 rev1=1 crypto rx=0x7f7b1c001060 tx=0x7f7b1c009000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2eac79f2c0430ac5 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:49:25.781 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:25.779+0000 7f7b29ffb640 1 -- 172.21.15.123:0/1597718549 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b1c025070 con 0x55ee9e8b02f0 2024-09-16T15:49:25.781 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:25.779+0000 7f7b29ffb640 1 -- 172.21.15.123:0/1597718549 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f7b1c012ae0 con 0x55ee9e8b02f0 2024-09-16T15:49:25.781 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:25.779+0000 7f7b29ffb640 1 -- 172.21.15.123:0/1597718549 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b1c0113d0 con 0x55ee9e8b02f0 2024-09-16T15:49:25.786 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:25.779+0000 7f7b3d3e3000 1 -- 172.21.15.123:0/1597718549 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55ee9e8b02f0 msgr2=0x55ee9e8638d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:25.786 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:25.779+0000 7f7b3d3e3000 1 --2- 172.21.15.123:0/1597718549 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55ee9e8b02f0 0x55ee9e8638d0 secure :-1 s=READY pgs=1278 cs=0 l=1 rev1=1 crypto rx=0x7f7b1c001060 tx=0x7f7b1c009000 comp rx=0 tx=0).stop 2024-09-16T15:49:25.786 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:25.779+0000 7f7b3d3e3000 1 -- 172.21.15.123:0/1597718549 shutdown_connections 2024-09-16T15:49:25.786 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:25.779+0000 7f7b3d3e3000 1 --2- 172.21.15.123:0/1597718549 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x55ee9e863e10 0x55ee9e8641f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:25.786 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:25.779+0000 7f7b3d3e3000 1 --2- 172.21.15.123:0/1597718549 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55ee9e8b02f0 0x55ee9e8638d0 unknown :-1 s=CLOSED pgs=1278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:25.786 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:25.779+0000 7f7b3d3e3000 1 --2- 172.21.15.123:0/1597718549 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ee9e8af9d0 0x55ee9e8afdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:25.786 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:25.779+0000 7f7b3d3e3000 1 -- 172.21.15.123:0/1597718549 >> 172.21.15.123:0/1597718549 conn(0x55ee9e74d130 msgr2=0x55ee9e892e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:49:25.786 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:25.779+0000 7f7b3d3e3000 1 -- 172.21.15.123:0/1597718549 shutdown_connections 2024-09-16T15:49:25.786 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:25.779+0000 7f7b3d3e3000 1 -- 172.21.15.123:0/1597718549 wait complete. 2024-09-16T15:49:25.786 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:25.779+0000 7f7b3d3e3000 1 -- start start 2024-09-16T15:49:25.786 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:25.779+0000 7f7b3d3e3000 1 Processor -- start 2024-09-16T15:49:25.786 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:25.783+0000 7f7b3d3e3000 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x55ee9e863e10 0x55ee9e865240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:49:25.787 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:25.783+0000 7f7b3d3e3000 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ee9e8af9d0 0x55ee9e865780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:49:25.787 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:25.783+0000 7f7b3d3e3000 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55ee9e8b02f0 0x55ee9e865cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:49:25.787 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:25.783+0000 7f7b3d3e3000 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x55ee9e64b5d0 con 0x55ee9e8b02f0 2024-09-16T15:49:25.787 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:25.783+0000 7f7b3d3e3000 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x55ee9e6b4d60 con 0x55ee9e8af9d0 2024-09-16T15:49:25.787 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:25.783+0000 7f7b3d3e3000 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x55ee9e657500 con 0x55ee9e863e10 2024-09-16T15:49:25.787 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:25.783+0000 7f7b3d3e3000 10 client.0 _ll_register_callbacks cb 0x55ee9e65b810 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T15:49:25.787 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:25.783+0000 7f7b38896640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55ee9e8b02f0 0x55ee9e865cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:49:25.787 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:25.783+0000 7f7b38896640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55ee9e8b02f0 0x55ee9e865cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:35508/0 (socket says 172.21.15.123:35508) 2024-09-16T15:49:25.787 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:25.783+0000 7f7b38896640 1 -- 172.21.15.123:0/3040088393 learned_addr learned my addr 172.21.15.123:0/3040088393 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:49:25.787 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:25.783+0000 7f7b33fff640 1 --2- 172.21.15.123:0/3040088393 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x55ee9e863e10 0x55ee9e865240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:49:25.787 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:25.783+0000 7f7b38896640 1 -- 172.21.15.123:0/3040088393 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x55ee9e863e10 msgr2=0x55ee9e865240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T15:49:25.787 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:25.783+0000 7f7b38896640 1 --2- 172.21.15.123:0/3040088393 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x55ee9e863e10 0x55ee9e865240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:25.787 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:25.783+0000 7f7b38896640 1 -- 172.21.15.123:0/3040088393 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ee9e8af9d0 msgr2=0x55ee9e865780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T15:49:25.788 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:25.783+0000 7f7b38896640 1 --2- 172.21.15.123:0/3040088393 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ee9e8af9d0 0x55ee9e865780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:25.788 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:25.783+0000 7f7b38896640 1 -- 172.21.15.123:0/3040088393 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ee9e64f170 con 0x55ee9e8b02f0 2024-09-16T15:49:25.788 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:25.783+0000 7f7b38896640 1 --2- 172.21.15.123:0/3040088393 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55ee9e8b02f0 0x55ee9e865cc0 secure :-1 s=READY pgs=1279 cs=0 l=1 rev1=1 crypto rx=0x7f7b2401b080 tx=0x7f7b24006620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5635662ebebaf59d server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:49:25.788 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:25.783+0000 7f7b12ffd640 10 client.0 ms_handle_connect on v2:172.21.15.123:3300/0 2024-09-16T15:49:25.788 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:25.783+0000 7f7b12ffd640 1 -- 172.21.15.123:0/3040088393 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b24006930 con 0x55ee9e8b02f0 2024-09-16T15:49:25.788 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:25.783+0000 7f7b12ffd640 1 -- 172.21.15.123:0/3040088393 <== mon.0 v2:172.21.15.123:3300/0 2 ==== osd_map(119..119 src has 1..119) ==== 5737+0+0 (secure 0 0 0) 0x7f7b24001310 con 0x55ee9e8b02f0 2024-09-16T15:49:25.788 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:25.783+0000 7f7b3d3e3000 1 -- 172.21.15.123:0/3040088393 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55ee9e6490f0 con 0x55ee9e8b02f0 2024-09-16T15:49:25.788 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:25.783+0000 7f7b12ffd640 1 -- 172.21.15.123:0/3040088393 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_get_version_reply(handle=1 version=76) ==== 24+0+0 (secure 0 0 0) 0x7f7b240c0050 con 0x55ee9e8b02f0 2024-09-16T15:49:25.788 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:25.783+0000 7f7b3d3e3000 10 client.7689 fetch_fsmap learned FSMap version 76 2024-09-16T15:49:25.788 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:25.783+0000 7f7b3d3e3000 1 -- 172.21.15.123:0/3040088393 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,fsmap=76,monmap=2+}) -- 0x55ee9e735ef0 con 0x55ee9e8b02f0 2024-09-16T15:49:25.788 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:25.783+0000 7f7b12ffd640 1 -- 172.21.15.123:0/3040088393 <== mon.0 v2:172.21.15.123:3300/0 4 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x55ee9e6490f0 con 0x55ee9e8b02f0 2024-09-16T15:49:25.788 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:25.783+0000 7f7b12ffd640 1 -- 172.21.15.123:0/3040088393 <== mon.0 v2:172.21.15.123:3300/0 5 ==== fsmap(e 76) ==== 3502+0+0 (secure 0 0 0) 0x7f7b240a5610 con 0x55ee9e8b02f0 2024-09-16T15:49:25.789 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:25.783+0000 7f7b3d3e3000 10 client.7689 fetch_fsmap finished waiting for FSMap version 76 2024-09-16T15:49:25.789 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:25.783+0000 7f7b3d3e3000 10 client.7689 resolve_mds: resolved name 'a' to daemon mds.a 2024-09-16T15:49:25.789 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:25.783+0000 7f7b3d3e3000 20 client.7689 populate_metadata read hostname 'smithi123' 2024-09-16T15:49:25.789 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:25.783+0000 7f7b3d3e3000 1 --2- 172.21.15.123:0/3040088393 >> [v2:172.21.15.123:6833/2110793616,v1:172.21.15.123:6835/2110793616] conn(0x55ee9e8bec70 0x55ee9e8c1060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:49:25.789 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:25.783+0000 7f7b3d3e3000 4 client.7689 mds_command: new command op to 7554 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-16T15:49:25.789 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:25.783+0000 7f7b3d3e3000 1 -- 172.21.15.123:0/3040088393 --> [v2:172.21.15.123:6833/2110793616,v1:172.21.15.123:6835/2110793616] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ee9e6c3a70 con 0x55ee9e8bec70 2024-09-16T15:49:25.789 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:25.783+0000 7f7b33fff640 1 --2- 172.21.15.123:0/3040088393 >> [v2:172.21.15.123:6833/2110793616,v1:172.21.15.123:6835/2110793616] conn(0x55ee9e8bec70 0x55ee9e8c1060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:49:25.789 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:25.783+0000 7f7b12ffd640 10 client.7689 ms_handle_connect on v2:172.21.15.123:6833/2110793616 2024-09-16T15:49:25.789 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:25.783+0000 7f7b33fff640 1 --2- 172.21.15.123:0/3040088393 >> [v2:172.21.15.123:6833/2110793616,v1:172.21.15.123:6835/2110793616] conn(0x55ee9e8bec70 0x55ee9e8c1060 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f62043f8281f9def server_cookie=cba660a90192d3e in_seq=0 out_seq=0 2024-09-16T15:49:25.789 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:25.783+0000 7f7b12ffd640 1 -- 172.21.15.123:0/3040088393 <== mds.0 v2:172.21.15.123:6833/2110793616 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x55ee9e6c3a70 con 0x55ee9e8bec70 2024-09-16T15:49:25.789 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:25.783+0000 7f7b12ffd640 10 client.7689 handle_command_reply: tid=0 2024-09-16T15:49:25.789 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:25.783+0000 7f7b3d3e3000 1 client.7689 shutdown 2024-09-16T15:49:25.789 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:25.783+0000 7f7b3d3e3000 1 -- 172.21.15.123:0/3040088393 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55ee9e8b02f0 msgr2=0x55ee9e865cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:25.789 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:25.783+0000 7f7b3d3e3000 1 --2- 172.21.15.123:0/3040088393 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55ee9e8b02f0 0x55ee9e865cc0 secure :-1 s=READY pgs=1279 cs=0 l=1 rev1=1 crypto rx=0x7f7b2401b080 tx=0x7f7b24006620 comp rx=0 tx=0).stop 2024-09-16T15:49:25.790 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:25.783+0000 7f7b3d3e3000 1 -- 172.21.15.123:0/3040088393 shutdown_connections 2024-09-16T15:49:25.790 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:25.783+0000 7f7b3d3e3000 1 --2- 172.21.15.123:0/3040088393 >> [v2:172.21.15.123:6833/2110793616,v1:172.21.15.123:6835/2110793616] conn(0x55ee9e8bec70 0x55ee9e8c1060 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:25.790 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:25.783+0000 7f7b3d3e3000 1 --2- 172.21.15.123:0/3040088393 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55ee9e8b02f0 0x55ee9e865cc0 unknown :-1 s=CLOSED pgs=1279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:25.790 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:25.783+0000 7f7b3d3e3000 1 --2- 172.21.15.123:0/3040088393 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ee9e8af9d0 0x55ee9e865780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:25.790 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:25.783+0000 7f7b3d3e3000 1 --2- 172.21.15.123:0/3040088393 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x55ee9e863e10 0x55ee9e865240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:25.790 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:25.783+0000 7f7b12ffd640 0 client.7689 ms_handle_reset on v2:172.21.15.123:6833/2110793616 2024-09-16T15:49:25.790 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:25.783+0000 7f7b3d3e3000 1 -- 172.21.15.123:0/3040088393 >> 172.21.15.123:0/3040088393 conn(0x55ee9e74d130 msgr2=0x55ee9e862fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:49:25.790 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:25.783+0000 7f7b3d3e3000 1 -- 172.21.15.123:0/3040088393 shutdown_connections 2024-09-16T15:49:25.790 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:25.783+0000 7f7b3d3e3000 1 -- 172.21.15.123:0/3040088393 wait complete. 2024-09-16T15:49:25.790 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:25.783+0000 7f7b3d3e3000 20 client.7689 trim_cache size 0 max 16384 2024-09-16T15:49:25.803 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:25.803+0000 7f7b3d3e3000 1 Processor -- start 2024-09-16T15:49:25.803 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:25.803+0000 7f7b3d3e3000 1 -- start start 2024-09-16T15:49:25.805 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:25.803+0000 7f7b3d3e3000 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ee9e863e10 0x55ee9e925b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:49:25.806 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:25.803+0000 7f7b3d3e3000 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x55ee9e8af9d0 0x55ee9e926050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:49:25.806 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:25.803+0000 7f7b3d3e3000 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55ee9e8b02f0 0x55ee9e926590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:49:25.806 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:25.803+0000 7f7b3d3e3000 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f7b34156c70 con 0x55ee9e8b02f0 2024-09-16T15:49:25.806 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:25.803+0000 7f7b3d3e3000 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f7b341fdd60 con 0x55ee9e863e10 2024-09-16T15:49:25.806 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:25.803+0000 7f7b3d3e3000 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x55ee9e6b9390 con 0x55ee9e8af9d0 2024-09-16T15:49:25.806 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:25.803+0000 7f7b38896640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55ee9e8b02f0 0x55ee9e926590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:49:25.806 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:25.803+0000 7f7b38896640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55ee9e8b02f0 0x55ee9e926590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:35512/0 (socket says 172.21.15.123:35512) 2024-09-16T15:49:25.806 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:25.803+0000 7f7b38896640 1 -- 172.21.15.123:0/3017018132 learned_addr learned my addr 172.21.15.123:0/3017018132 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:49:25.806 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:25.803+0000 7f7b38896640 1 -- 172.21.15.123:0/3017018132 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x55ee9e8af9d0 msgr2=0x55ee9e926050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T15:49:25.806 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:25.803+0000 7f7b38896640 1 --2- 172.21.15.123:0/3017018132 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x55ee9e8af9d0 0x55ee9e926050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:25.807 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:25.803+0000 7f7b38896640 1 -- 172.21.15.123:0/3017018132 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ee9e863e10 msgr2=0x55ee9e925b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T15:49:25.807 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:25.803+0000 7f7b38896640 1 --2- 172.21.15.123:0/3017018132 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ee9e863e10 0x55ee9e925b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:25.807 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:25.803+0000 7f7b38896640 1 -- 172.21.15.123:0/3017018132 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ee9e70cd90 con 0x55ee9e8b02f0 2024-09-16T15:49:25.807 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:25.803+0000 7f7b38896640 1 --2- 172.21.15.123:0/3017018132 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55ee9e8b02f0 0x55ee9e926590 secure :-1 s=READY pgs=1280 cs=0 l=1 rev1=1 crypto rx=0x7f7b2401d050 tx=0x7f7b24006f50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3ee408a8de5cd38 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:49:25.807 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:25.803+0000 7f7b297fa640 1 -- 172.21.15.123:0/3017018132 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b240a5c60 con 0x55ee9e8b02f0 2024-09-16T15:49:25.807 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:25.803+0000 7f7b297fa640 1 -- 172.21.15.123:0/3017018132 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f7b240a5610 con 0x55ee9e8b02f0 2024-09-16T15:49:25.807 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:25.803+0000 7f7b297fa640 1 -- 172.21.15.123:0/3017018132 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b24003860 con 0x55ee9e8b02f0 2024-09-16T15:49:25.807 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:25.807+0000 7f7b3d3e3000 1 -- 172.21.15.123:0/3017018132 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55ee9e8b02f0 msgr2=0x55ee9e926590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:25.807 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:25.807+0000 7f7b3d3e3000 1 --2- 172.21.15.123:0/3017018132 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55ee9e8b02f0 0x55ee9e926590 secure :-1 s=READY pgs=1280 cs=0 l=1 rev1=1 crypto rx=0x7f7b2401d050 tx=0x7f7b24006f50 comp rx=0 tx=0).stop 2024-09-16T15:49:25.807 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:25.807+0000 7f7b3d3e3000 1 -- 172.21.15.123:0/3017018132 shutdown_connections 2024-09-16T15:49:25.807 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:25.807+0000 7f7b3d3e3000 1 --2- 172.21.15.123:0/3017018132 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55ee9e8b02f0 0x55ee9e926590 unknown :-1 s=CLOSED pgs=1280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:25.807 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:25.807+0000 7f7b3d3e3000 1 --2- 172.21.15.123:0/3017018132 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x55ee9e8af9d0 0x55ee9e926050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:25.807 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:25.807+0000 7f7b3d3e3000 1 --2- 172.21.15.123:0/3017018132 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ee9e863e10 0x55ee9e925b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:25.808 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:25.807+0000 7f7b3d3e3000 1 -- 172.21.15.123:0/3017018132 >> 172.21.15.123:0/3017018132 conn(0x55ee9e74d130 msgr2=0x55ee9e863470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:49:25.808 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:25.807+0000 7f7b3d3e3000 1 -- 172.21.15.123:0/3017018132 shutdown_connections 2024-09-16T15:49:25.808 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:25.807+0000 7f7b3d3e3000 1 -- 172.21.15.123:0/3017018132 wait complete. 2024-09-16T15:49:25.808 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:25.807+0000 7f7b3d3e3000 1 -- start start 2024-09-16T15:49:25.808 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:25.807+0000 7f7b3d3e3000 1 Processor -- start 2024-09-16T15:49:25.811 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:25.807+0000 7f7b3d3e3000 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x55ee9e863e10 0x55ee9e929a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:49:25.811 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:25.807+0000 7f7b3d3e3000 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ee9e8af9d0 0x55ee9e929f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:49:25.812 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:25.807+0000 7f7b3d3e3000 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55ee9e8b02f0 0x55ee9e8e4870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:49:25.812 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:25.807+0000 7f7b3d3e3000 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x55ee9e6c08f0 con 0x55ee9e8b02f0 2024-09-16T15:49:25.812 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:25.807+0000 7f7b3d3e3000 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x55ee9e770730 con 0x55ee9e8af9d0 2024-09-16T15:49:25.812 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:25.807+0000 7f7b3d3e3000 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x55ee9e6b4070 con 0x55ee9e863e10 2024-09-16T15:49:25.812 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:25.807+0000 7f7b3d3e3000 10 client.0 _ll_register_callbacks cb 0x7f7b20003700 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T15:49:25.812 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:25.807+0000 7f7b38896640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55ee9e8b02f0 0x55ee9e8e4870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:49:25.812 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:25.807+0000 7f7b38896640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55ee9e8b02f0 0x55ee9e8e4870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:35516/0 (socket says 172.21.15.123:35516) 2024-09-16T15:49:25.812 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:25.807+0000 7f7b38896640 1 -- 172.21.15.123:0/2929248140 learned_addr learned my addr 172.21.15.123:0/2929248140 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:49:25.812 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:25.807+0000 7f7b33fff640 1 --2- 172.21.15.123:0/2929248140 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x55ee9e863e10 0x55ee9e929a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:49:25.812 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:25.807+0000 7f7b337fe640 1 --2- 172.21.15.123:0/2929248140 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ee9e8af9d0 0x55ee9e929f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:49:25.812 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:25.807+0000 7f7b38896640 1 -- 172.21.15.123:0/2929248140 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x55ee9e863e10 msgr2=0x55ee9e929a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T15:49:25.812 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:25.807+0000 7f7b38896640 1 --2- 172.21.15.123:0/2929248140 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x55ee9e863e10 0x55ee9e929a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:25.812 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:25.807+0000 7f7b38896640 1 -- 172.21.15.123:0/2929248140 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ee9e8af9d0 msgr2=0x55ee9e929f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T15:49:25.813 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:25.807+0000 7f7b38896640 1 --2- 172.21.15.123:0/2929248140 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ee9e8af9d0 0x55ee9e929f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:25.813 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:25.807+0000 7f7b38896640 1 -- 172.21.15.123:0/2929248140 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ee9e677a90 con 0x55ee9e8b02f0 2024-09-16T15:49:25.813 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:25.807+0000 7f7b38896640 1 --2- 172.21.15.123:0/2929248140 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55ee9e8b02f0 0x55ee9e8e4870 secure :-1 s=READY pgs=1281 cs=0 l=1 rev1=1 crypto rx=0x7f7b240065f0 tx=0x7f7b240a5dc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6d371e96469751a1 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:49:25.813 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:25.807+0000 7f7b12ffd640 10 client.0 ms_handle_connect on v2:172.21.15.123:3300/0 2024-09-16T15:49:25.813 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:25.807+0000 7f7b12ffd640 1 -- 172.21.15.123:0/2929248140 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b240b4070 con 0x55ee9e8b02f0 2024-09-16T15:49:25.813 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:25.807+0000 7f7b12ffd640 1 -- 172.21.15.123:0/2929248140 <== mon.0 v2:172.21.15.123:3300/0 2 ==== osd_map(119..119 src has 1..119) ==== 5737+0+0 (secure 0 0 0) 0x7f7b240c6b50 con 0x55ee9e8b02f0 2024-09-16T15:49:25.813 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:25.807+0000 7f7b3d3e3000 1 -- 172.21.15.123:0/2929248140 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7b240a5610 con 0x55ee9e8b02f0 2024-09-16T15:49:25.813 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:25.807+0000 7f7b12ffd640 1 -- 172.21.15.123:0/2929248140 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_get_version_reply(handle=1 version=76) ==== 24+0+0 (secure 0 0 0) 0x7f7b24001130 con 0x55ee9e8b02f0 2024-09-16T15:49:25.813 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:25.807+0000 7f7b3d3e3000 10 client.7695 fetch_fsmap learned FSMap version 76 2024-09-16T15:49:25.813 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:25.807+0000 7f7b3d3e3000 1 -- 172.21.15.123:0/2929248140 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,fsmap=76,monmap=2+}) -- 0x55ee9e656050 con 0x55ee9e8b02f0 2024-09-16T15:49:25.813 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:25.807+0000 7f7b12ffd640 1 -- 172.21.15.123:0/2929248140 <== mon.0 v2:172.21.15.123:3300/0 4 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f7b240a5610 con 0x55ee9e8b02f0 2024-09-16T15:49:25.813 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:25.807+0000 7f7b12ffd640 1 -- 172.21.15.123:0/2929248140 <== mon.0 v2:172.21.15.123:3300/0 5 ==== fsmap(e 76) ==== 3502+0+0 (secure 0 0 0) 0x7f7b24003810 con 0x55ee9e8b02f0 2024-09-16T15:49:25.813 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:25.807+0000 7f7b3d3e3000 10 client.7695 fetch_fsmap finished waiting for FSMap version 76 2024-09-16T15:49:25.813 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:25.807+0000 7f7b3d3e3000 10 client.7695 resolve_mds: resolved name 'a' to daemon mds.a 2024-09-16T15:49:25.814 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:25.807+0000 7f7b3d3e3000 20 client.7695 populate_metadata read hostname 'smithi123' 2024-09-16T15:49:25.814 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:25.807+0000 7f7b3d3e3000 1 --2- 172.21.15.123:0/2929248140 >> [v2:172.21.15.123:6833/2110793616,v1:172.21.15.123:6835/2110793616] conn(0x55ee9e8e4db0 0x55ee9e8e5190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:49:25.814 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:25.807+0000 7f7b3d3e3000 4 client.7695 mds_command: new command op to 7554 tid=0 multi_id=0 [{"prefix": "status"}] 2024-09-16T15:49:25.814 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:25.807+0000 7f7b3d3e3000 1 -- 172.21.15.123:0/2929248140 --> [v2:172.21.15.123:6833/2110793616,v1:172.21.15.123:6835/2110793616] -- command(tid 0: {"prefix": "status"}) -- 0x55ee9e6bf520 con 0x55ee9e8e4db0 2024-09-16T15:49:25.814 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:25.807+0000 7f7b33fff640 1 --2- 172.21.15.123:0/2929248140 >> [v2:172.21.15.123:6833/2110793616,v1:172.21.15.123:6835/2110793616] conn(0x55ee9e8e4db0 0x55ee9e8e5190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:49:25.814 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:25.807+0000 7f7b12ffd640 10 client.7695 ms_handle_connect on v2:172.21.15.123:6833/2110793616 2024-09-16T15:49:25.814 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:25.807+0000 7f7b33fff640 1 --2- 172.21.15.123:0/2929248140 >> [v2:172.21.15.123:6833/2110793616,v1:172.21.15.123:6835/2110793616] conn(0x55ee9e8e4db0 0x55ee9e8e5190 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a393abc6d998efdb server_cookie=22b07d8bd74353b4 in_seq=0 out_seq=0 2024-09-16T15:49:25.814 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:25.807+0000 7f7b12ffd640 1 -- 172.21.15.123:0/2929248140 <== mds.0 v2:172.21.15.123:6833/2110793616 1 ==== command_reply(tid 0: 0 ) ==== 8+0+333 (crc 0 0 0) 0x55ee9e6bf520 con 0x55ee9e8e4db0 2024-09-16T15:49:25.814 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:25.807+0000 7f7b12ffd640 10 client.7695 handle_command_reply: tid=0 2024-09-16T15:49:25.814 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:25.807+0000 7f7b3d3e3000 1 client.7695 shutdown 2024-09-16T15:49:25.814 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:25.807+0000 7f7b3d3e3000 1 -- 172.21.15.123:0/2929248140 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55ee9e8b02f0 msgr2=0x55ee9e8e4870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:25.814 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:25.807+0000 7f7b3d3e3000 1 --2- 172.21.15.123:0/2929248140 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55ee9e8b02f0 0x55ee9e8e4870 secure :-1 s=READY pgs=1281 cs=0 l=1 rev1=1 crypto rx=0x7f7b240065f0 tx=0x7f7b240a5dc0 comp rx=0 tx=0).stop 2024-09-16T15:49:25.814 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:25.811+0000 7f7b3d3e3000 1 -- 172.21.15.123:0/2929248140 shutdown_connections 2024-09-16T15:49:25.814 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:25.811+0000 7f7b3d3e3000 1 --2- 172.21.15.123:0/2929248140 >> [v2:172.21.15.123:6833/2110793616,v1:172.21.15.123:6835/2110793616] conn(0x55ee9e8e4db0 0x55ee9e8e5190 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:25.815 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:25.811+0000 7f7b3d3e3000 1 --2- 172.21.15.123:0/2929248140 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55ee9e8b02f0 0x55ee9e8e4870 unknown :-1 s=CLOSED pgs=1281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:25.815 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:25.811+0000 7f7b3d3e3000 1 --2- 172.21.15.123:0/2929248140 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55ee9e8af9d0 0x55ee9e929f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:25.815 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:25.811+0000 7f7b12ffd640 0 client.7695 ms_handle_reset on v2:172.21.15.123:6833/2110793616 2024-09-16T15:49:25.815 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:25.811+0000 7f7b3d3e3000 1 --2- 172.21.15.123:0/2929248140 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x55ee9e863e10 0x55ee9e929a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:25.815 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:25.811+0000 7f7b3d3e3000 1 -- 172.21.15.123:0/2929248140 >> 172.21.15.123:0/2929248140 conn(0x55ee9e74d130 msgr2=0x55ee9e863470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:49:25.815 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:25.811+0000 7f7b3d3e3000 1 -- 172.21.15.123:0/2929248140 shutdown_connections 2024-09-16T15:49:25.815 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:25.811+0000 7f7b3d3e3000 1 -- 172.21.15.123:0/2929248140 wait complete. 2024-09-16T15:49:25.815 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:25.811+0000 7f7b3d3e3000 20 client.7695 trim_cache size 0 max 16384 2024-09-16T15:49:25.815 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2024-09-16T15:49:25.815 INFO:teuthology.orchestra.run.smithi123.stdout: "cluster_fsid": "56077f89-18c8-486f-ae7e-eecc0bb7822c", 2024-09-16T15:49:25.815 INFO:teuthology.orchestra.run.smithi123.stdout: "whoami": 0, 2024-09-16T15:49:25.816 INFO:teuthology.orchestra.run.smithi123.stdout: "id": 7554, 2024-09-16T15:49:25.816 INFO:teuthology.orchestra.run.smithi123.stdout: "want_state": "up:active", 2024-09-16T15:49:25.816 INFO:teuthology.orchestra.run.smithi123.stdout: "state": "up:active", 2024-09-16T15:49:25.816 INFO:teuthology.orchestra.run.smithi123.stdout: "fs_name": "cephfs", 2024-09-16T15:49:25.816 INFO:teuthology.orchestra.run.smithi123.stdout: "rank_uptime": 5.8183703749999998, 2024-09-16T15:49:25.816 INFO:teuthology.orchestra.run.smithi123.stdout: "mdsmap_epoch": 76, 2024-09-16T15:49:25.816 INFO:teuthology.orchestra.run.smithi123.stdout: "osdmap_epoch": 119, 2024-09-16T15:49:25.816 INFO:teuthology.orchestra.run.smithi123.stdout: "osdmap_epoch_barrier": 119, 2024-09-16T15:49:25.816 INFO:teuthology.orchestra.run.smithi123.stdout: "uptime": 11.154727512999999 2024-09-16T15:49:25.816 INFO:teuthology.orchestra.run.smithi123.stdout:} 2024-09-16T15:49:25.817 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:25.811+0000 7f7b3a2f9640 1 -- 172.21.15.123:0/1645735062 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f7b14073370 msgr2=0x7f7b14075830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:25.817 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:25.811+0000 7f7b3a2f9640 1 --2- 172.21.15.123:0/1645735062 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f7b14073370 0x7f7b14075830 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f7b1c000fc0 tx=0x7f7b1c001120 comp rx=0 tx=0).stop 2024-09-16T15:49:25.817 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:25.811+0000 7f7b3a2f9640 1 -- 172.21.15.123:0/1645735062 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7b200051d0 msgr2=0x7f7b34159600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:25.817 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:25.811+0000 7f7b3a2f9640 1 --2- 172.21.15.123:0/1645735062 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7b200051d0 0x7f7b34159600 secure :-1 s=READY pgs=1277 cs=0 l=1 rev1=1 crypto rx=0x7f7b2c00fc00 tx=0x7f7b2c00d6b0 comp rx=0 tx=0).stop 2024-09-16T15:49:25.817 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:25.811+0000 7f7b3a2f9640 1 -- 172.21.15.123:0/1645735062 shutdown_connections 2024-09-16T15:49:25.817 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:25.811+0000 7f7b3a2f9640 1 --2- 172.21.15.123:0/1645735062 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f7b14073370 0x7f7b14075830 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:25.818 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:25.811+0000 7f7b3a2f9640 1 --2- 172.21.15.123:0/1645735062 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f7b341f6030 0x7f7b340b6cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:25.818 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:25.811+0000 7f7b3a2f9640 1 --2- 172.21.15.123:0/1645735062 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7b34157680 0x7f7b34155500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:25.818 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:25.811+0000 7f7b3a2f9640 1 --2- 172.21.15.123:0/1645735062 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7b200051d0 0x7f7b34159600 unknown :-1 s=CLOSED pgs=1277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:25.818 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:25.811+0000 7f7b3a2f9640 1 -- 172.21.15.123:0/1645735062 >> 172.21.15.123:0/1645735062 conn(0x7f7b3409aa40 msgr2=0x7f7b34071a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:49:25.818 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:25.811+0000 7f7b3a2f9640 1 -- 172.21.15.123:0/1645735062 shutdown_connections 2024-09-16T15:49:25.818 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:25.811+0000 7f7b3a2f9640 1 -- 172.21.15.123:0/1645735062 wait complete. 2024-09-16T15:49:25.825 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-09-16T15:49:25.938 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:25.939+0000 7f9a0a847640 1 -- 172.21.15.123:0/3272523174 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f99ec004b70 msgr2=0x7f99ec004f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:25.938 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:25.939+0000 7f9a0a847640 1 --2- 172.21.15.123:0/3272523174 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f99ec004b70 0x7f99ec004f90 secure :-1 s=READY pgs=1283 cs=0 l=1 rev1=1 crypto rx=0x7f99f4009e30 tx=0x7f99f401d170 comp rx=0 tx=0).stop 2024-09-16T15:49:25.939 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:25.939+0000 7f9a0a847640 1 -- 172.21.15.123:0/3272523174 shutdown_connections 2024-09-16T15:49:25.939 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:25.939+0000 7f9a0a847640 1 --2- 172.21.15.123:0/3272523174 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f99ec004b70 0x7f99ec004f90 unknown :-1 s=CLOSED pgs=1283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:25.939 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:25.939+0000 7f9a0a847640 1 -- 172.21.15.123:0/3272523174 >> 172.21.15.123:0/3272523174 conn(0x7f9a040cfb10 msgr2=0x7f9a040cff20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:49:25.939 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:25.939+0000 7f9a0a847640 1 -- 172.21.15.123:0/3272523174 shutdown_connections 2024-09-16T15:49:25.939 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:25.939+0000 7f9a0a847640 1 -- 172.21.15.123:0/3272523174 wait complete. 2024-09-16T15:49:25.939 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:25.939+0000 7f9a0a847640 1 Processor -- start 2024-09-16T15:49:25.939 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:25.939+0000 7f9a0a847640 1 -- start start 2024-09-16T15:49:25.941 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:25.939+0000 7f9a0a847640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f99ec004b70 0x7f9a041f9620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:49:25.942 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:25.939+0000 7f9a0a847640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9a0414d850 0x7f9a0415ac70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:49:25.942 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:25.939+0000 7f9a0a847640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f9a04159310 0x7f9a04159790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:49:25.942 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:25.939+0000 7f9a0a847640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f9a0406ef00 con 0x7f9a0414d850 2024-09-16T15:49:25.942 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:25.939+0000 7f9a0a847640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f9a0406ed80 con 0x7f99ec004b70 2024-09-16T15:49:25.942 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:25.939+0000 7f9a0a847640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f9a0406f080 con 0x7f9a04159310 2024-09-16T15:49:25.942 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:25.939+0000 7f9a037fe640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9a0414d850 0x7f9a0415ac70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:49:25.942 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:25.939+0000 7f9a037fe640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9a0414d850 0x7f9a0415ac70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:35538/0 (socket says 172.21.15.123:35538) 2024-09-16T15:49:25.942 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:25.939+0000 7f9a037fe640 1 -- 172.21.15.123:0/1508625030 learned_addr learned my addr 172.21.15.123:0/1508625030 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:49:25.942 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:25.939+0000 7f9a03fff640 1 --2- 172.21.15.123:0/1508625030 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f99ec004b70 0x7f9a041f9620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:49:25.942 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:25.939+0000 7f9a08de4640 1 --2- 172.21.15.123:0/1508625030 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f9a04159310 0x7f9a04159790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:49:25.942 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:25.939+0000 7f9a037fe640 1 -- 172.21.15.123:0/1508625030 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f9a04159310 msgr2=0x7f9a04159790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:25.942 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:25.939+0000 7f9a037fe640 1 --2- 172.21.15.123:0/1508625030 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f9a04159310 0x7f9a04159790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:25.943 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:25.939+0000 7f9a037fe640 1 -- 172.21.15.123:0/1508625030 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f99ec004b70 msgr2=0x7f9a041f9620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:25.943 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:25.939+0000 7f9a037fe640 1 --2- 172.21.15.123:0/1508625030 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f99ec004b70 0x7f9a041f9620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:25.943 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:25.939+0000 7f9a037fe640 1 -- 172.21.15.123:0/1508625030 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9a041f9b60 con 0x7f9a0414d850 2024-09-16T15:49:25.943 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:25.943+0000 7f9a037fe640 1 --2- 172.21.15.123:0/1508625030 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9a0414d850 0x7f9a0415ac70 secure :-1 s=READY pgs=1284 cs=0 l=1 rev1=1 crypto rx=0x7f99f800d7b0 tx=0x7f99f800dc70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:49:25.943 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:25.943+0000 7f9a00ff9640 1 -- 172.21.15.123:0/1508625030 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99f8018070 con 0x7f9a0414d850 2024-09-16T15:49:25.943 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:25.943+0000 7f9a00ff9640 1 -- 172.21.15.123:0/1508625030 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f99f80052e0 con 0x7f9a0414d850 2024-09-16T15:49:25.943 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:25.943+0000 7f9a0a847640 1 -- 172.21.15.123:0/1508625030 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9a040c9620 con 0x7f9a0414d850 2024-09-16T15:49:25.943 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:25.943+0000 7f9a00ff9640 1 -- 172.21.15.123:0/1508625030 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99f800de60 con 0x7f9a0414d850 2024-09-16T15:49:25.943 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:25.943+0000 7f9a0a847640 1 -- 172.21.15.123:0/1508625030 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9a040c4880 con 0x7f9a0414d850 2024-09-16T15:49:25.943 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:25.943+0000 7f9a0a847640 1 -- 172.21.15.123:0/1508625030 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9a04159cd0 con 0x7f9a0414d850 2024-09-16T15:49:25.944 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:25.943+0000 7f9a00ff9640 1 -- 172.21.15.123:0/1508625030 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 19) ==== 97488+0+0 (secure 0 0 0) 0x7f99f801b070 con 0x7f9a0414d850 2024-09-16T15:49:25.944 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:25.943+0000 7f9a00ff9640 1 --2- 172.21.15.123:0/1508625030 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f99f0073370 0x7f99f0075830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:49:25.944 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:25.943+0000 7f9a03fff640 1 --2- 172.21.15.123:0/1508625030 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f99f0073370 0x7f99f0075830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:49:25.944 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:25.943+0000 7f9a00ff9640 1 -- 172.21.15.123:0/1508625030 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(119..119 src has 1..119) ==== 5737+0+0 (secure 0 0 0) 0x7f99f8093d20 con 0x7f9a0414d850 2024-09-16T15:49:25.944 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:25.943+0000 7f9a03fff640 1 --2- 172.21.15.123:0/1508625030 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f99f0073370 0x7f99f0075830 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f99f40ae000 tx=0x7f99f4019f10 comp rx=0 tx=0).ready entity=mgr.4150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:49:25.947 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:25.947+0000 7f9a00ff9640 1 -- 172.21.15.123:0/1508625030 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f99f8099020 con 0x7f9a0414d850 2024-09-16T15:49:26.152 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:26.151+0000 7f9a0a847640 1 -- 172.21.15.123:0/1508625030 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f9a040d06c0 con 0x7f9a0414d850 2024-09-16T15:49:26.152 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:26.151+0000 7f9a00ff9640 1 -- 172.21.15.123:0/1508625030 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v76) ==== 53+0+73 (secure 0 0 0) 0x7f99f805e1c0 con 0x7f9a0414d850 2024-09-16T15:49:26.152 INFO:teuthology.orchestra.run.smithi123.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-09-16T15:49:26.155 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:26.155+0000 7f9a0a847640 1 -- 172.21.15.123:0/1508625030 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f99f0073370 msgr2=0x7f99f0075830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:26.155 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:26.155+0000 7f9a0a847640 1 --2- 172.21.15.123:0/1508625030 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f99f0073370 0x7f99f0075830 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f99f40ae000 tx=0x7f99f4019f10 comp rx=0 tx=0).stop 2024-09-16T15:49:26.155 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:26.155+0000 7f9a0a847640 1 -- 172.21.15.123:0/1508625030 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9a0414d850 msgr2=0x7f9a0415ac70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:26.155 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:26.155+0000 7f9a0a847640 1 --2- 172.21.15.123:0/1508625030 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9a0414d850 0x7f9a0415ac70 secure :-1 s=READY pgs=1284 cs=0 l=1 rev1=1 crypto rx=0x7f99f800d7b0 tx=0x7f99f800dc70 comp rx=0 tx=0).stop 2024-09-16T15:49:26.155 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:26.155+0000 7f9a0a847640 1 -- 172.21.15.123:0/1508625030 shutdown_connections 2024-09-16T15:49:26.155 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:26.155+0000 7f9a0a847640 1 --2- 172.21.15.123:0/1508625030 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f99f0073370 0x7f99f0075830 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:26.155 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:26.155+0000 7f9a0a847640 1 --2- 172.21.15.123:0/1508625030 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f9a04159310 0x7f9a04159790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:26.155 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:26.155+0000 7f9a0a847640 1 --2- 172.21.15.123:0/1508625030 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9a0414d850 0x7f9a0415ac70 unknown :-1 s=CLOSED pgs=1284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:26.156 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:26.155+0000 7f9a0a847640 1 --2- 172.21.15.123:0/1508625030 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f99ec004b70 0x7f9a041f9620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:26.156 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:26.155+0000 7f9a0a847640 1 -- 172.21.15.123:0/1508625030 >> 172.21.15.123:0/1508625030 conn(0x7f9a040cfb10 msgr2=0x7f9a040b4d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:49:26.156 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:26.155+0000 7f9a0a847640 1 -- 172.21.15.123:0/1508625030 shutdown_connections 2024-09-16T15:49:26.156 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:26.155+0000 7f9a0a847640 1 -- 172.21.15.123:0/1508625030 wait complete. 2024-09-16T15:49:26.170 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-09-16T15:49:26.170 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-09-16T15:49:26.170 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi123.front.sepia.ceph.com 2024-09-16T15:49:26.170 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-09-16T15:49:26.170 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-09-16T15:49:26.171 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-09-16T15:49:26.171 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-09-16T15:49:26.171 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-09-16T15:49:26.171 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-09-16T15:49:26.171 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:49:26.171 DEBUG:teuthology.orchestra.run.smithi123:> ip addr 2024-09-16T15:49:26.177 INFO:teuthology.orchestra.run.smithi123.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-09-16T15:49:26.177 INFO:teuthology.orchestra.run.smithi123.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-09-16T15:49:26.177 INFO:teuthology.orchestra.run.smithi123.stdout: inet 127.0.0.1/8 scope host lo 2024-09-16T15:49:26.178 INFO:teuthology.orchestra.run.smithi123.stdout: valid_lft forever preferred_lft forever 2024-09-16T15:49:26.178 INFO:teuthology.orchestra.run.smithi123.stdout: inet6 ::1/128 scope host 2024-09-16T15:49:26.178 INFO:teuthology.orchestra.run.smithi123.stdout: valid_lft forever preferred_lft forever 2024-09-16T15:49:26.178 INFO:teuthology.orchestra.run.smithi123.stdout:2: eno1: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-09-16T15:49:26.178 INFO:teuthology.orchestra.run.smithi123.stdout: link/ether 0c:c4:7a:d8:da:6c brd ff:ff:ff:ff:ff:ff 2024-09-16T15:49:26.178 INFO:teuthology.orchestra.run.smithi123.stdout: altname enp7s0f0 2024-09-16T15:49:26.178 INFO:teuthology.orchestra.run.smithi123.stdout:3: eno2: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-09-16T15:49:26.178 INFO:teuthology.orchestra.run.smithi123.stdout: link/ether 0c:c4:7a:d8:da:6d brd ff:ff:ff:ff:ff:ff 2024-09-16T15:49:26.178 INFO:teuthology.orchestra.run.smithi123.stdout: altname enp7s0f1 2024-09-16T15:49:26.178 INFO:teuthology.orchestra.run.smithi123.stdout:4: enp3s0f0: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-09-16T15:49:26.178 INFO:teuthology.orchestra.run.smithi123.stdout: link/ether 0c:c4:7a:8f:d3:f0 brd ff:ff:ff:ff:ff:ff 2024-09-16T15:49:26.178 INFO:teuthology.orchestra.run.smithi123.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-09-16T15:49:26.178 INFO:teuthology.orchestra.run.smithi123.stdout: link/ether 0c:c4:7a:8f:d3:f1 brd ff:ff:ff:ff:ff:ff 2024-09-16T15:49:26.178 INFO:teuthology.orchestra.run.smithi123.stdout: inet 172.21.15.123/20 metric 100 brd 172.21.15.255 scope global dynamic enp3s0f1 2024-09-16T15:49:26.179 INFO:teuthology.orchestra.run.smithi123.stdout: valid_lft 42102sec preferred_lft 42102sec 2024-09-16T15:49:26.179 INFO:teuthology.orchestra.run.smithi123.stdout: inet6 fe80::ec4:7aff:fe8f:d3f1/64 scope link 2024-09-16T15:49:26.179 INFO:teuthology.orchestra.run.smithi123.stdout: valid_lft forever preferred_lft forever 2024-09-16T15:49:26.179 INFO:teuthology.orchestra.run.smithi123.stdout:6: docker0: mtu 1500 qdisc noqueue state DOWN group default 2024-09-16T15:49:26.179 INFO:teuthology.orchestra.run.smithi123.stdout: link/ether 02:42:dd:82:a2:6e brd ff:ff:ff:ff:ff:ff 2024-09-16T15:49:26.179 INFO:teuthology.orchestra.run.smithi123.stdout: inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0 2024-09-16T15:49:26.179 INFO:teuthology.orchestra.run.smithi123.stdout: valid_lft forever preferred_lft forever 2024-09-16T15:49:26.179 INFO:teuthology.orchestra.run.smithi123.stdout:7: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-09-16T15:49:26.179 INFO:teuthology.orchestra.run.smithi123.stdout: link/ether 5e:1f:84:6d:79:ae brd ff:ff:ff:ff:ff:ff 2024-09-16T15:49:26.179 INFO:teuthology.orchestra.run.smithi123.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-09-16T15:49:26.179 INFO:teuthology.orchestra.run.smithi123.stdout: valid_lft forever preferred_lft forever 2024-09-16T15:49:26.180 INFO:teuthology.orchestra.run.smithi123.stdout: inet6 fe80::5c1f:84ff:fe6d:79ae/64 scope link 2024-09-16T15:49:26.180 INFO:teuthology.orchestra.run.smithi123.stdout: valid_lft forever preferred_lft forever 2024-09-16T15:49:26.180 INFO:teuthology.orchestra.run.smithi123.stdout:8: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-09-16T15:49:26.180 INFO:teuthology.orchestra.run.smithi123.stdout: link/ether 0e:5c:f6:1f:82:2f brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-09-16T15:49:26.180 INFO:teuthology.orchestra.run.smithi123.stdout: inet6 fe80::c5c:f6ff:fe1f:822f/64 scope link 2024-09-16T15:49:26.180 INFO:teuthology.orchestra.run.smithi123.stdout: valid_lft forever preferred_lft forever 2024-09-16T15:49:26.180 INFO:teuthology.orchestra.run.smithi123.stdout:9: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-09-16T15:49:26.180 INFO:teuthology.orchestra.run.smithi123.stdout: link/ether c6:a6:3d:69:d0:db brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-09-16T15:49:26.180 INFO:teuthology.orchestra.run.smithi123.stdout: inet6 fe80::c4a6:3dff:fe69:d0db/64 scope link 2024-09-16T15:49:26.180 INFO:teuthology.orchestra.run.smithi123.stdout: valid_lft forever preferred_lft forever 2024-09-16T15:49:26.180 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:49:26.181 DEBUG:teuthology.orchestra.run.smithi123:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-09-16T15:49:26.228 INFO:teuthology.orchestra.run.smithi123.stdout:1 2024-09-16T15:49:26.230 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:49:26.230 DEBUG:teuthology.orchestra.run.smithi123:> route 2024-09-16T15:49:26.281 INFO:teuthology.orchestra.run.smithi123.stdout:Kernel IP routing table 2024-09-16T15:49:26.281 INFO:teuthology.orchestra.run.smithi123.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-09-16T15:49:26.281 INFO:teuthology.orchestra.run.smithi123.stdout:default _gateway 0.0.0.0 UG 0 0 0 enp3s0f1 2024-09-16T15:49:26.281 INFO:teuthology.orchestra.run.smithi123.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-09-16T15:49:26.281 INFO:teuthology.orchestra.run.smithi123.stdout:158.69.67.47 _gateway 255.255.255.255 UGH 100 0 0 enp3s0f1 2024-09-16T15:49:26.281 INFO:teuthology.orchestra.run.smithi123.stdout:172.17.0.0 0.0.0.0 255.255.0.0 U 0 0 0 docker0 2024-09-16T15:49:26.281 INFO:teuthology.orchestra.run.smithi123.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-09-16T15:49:26.281 INFO:teuthology.orchestra.run.smithi123.stdout:ns1.front.sepia 0.0.0.0 255.255.255.255 UH 100 0 0 enp3s0f1 2024-09-16T15:49:26.281 INFO:teuthology.orchestra.run.smithi123.stdout:172.21.0.2 0.0.0.0 255.255.255.255 UH 100 0 0 enp3s0f1 2024-09-16T15:49:26.281 INFO:teuthology.orchestra.run.smithi123.stdout:_gateway 0.0.0.0 255.255.255.255 UH 100 0 0 enp3s0f1 2024-09-16T15:49:26.281 INFO:teuthology.orchestra.run.smithi123.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-09-16T15:49:26.282 DEBUG:teuthology.orchestra.run.smithi123:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-09-16T15:49:26.282 DEBUG:teuthology.orchestra.run.smithi123:> set -e 2024-09-16T15:49:26.282 DEBUG:teuthology.orchestra.run.smithi123:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-09-16T15:49:26.282 DEBUG:teuthology.orchestra.run.smithi123:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-09-16T15:49:26.282 DEBUG:teuthology.orchestra.run.smithi123:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-09-16T15:49:26.282 DEBUG:teuthology.orchestra.run.smithi123:> ') 2024-09-16T15:49:26.373 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:26 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-09-16T15:49:26.399 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:26 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-16T15:49:26.405 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:49:26.405 DEBUG:teuthology.orchestra.run.smithi123:> ip netns list 2024-09-16T15:49:26.410 INFO:teuthology.orchestra.run.smithi123.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-09-16T15:49:26.410 INFO:teuthology.orchestra.run.smithi123.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-09-16T15:49:26.410 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-09-16T15:49:26.410 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:49:26.411 DEBUG:teuthology.orchestra.run.smithi123:> sudo ip link set brx.0 up 2024-09-16T15:49:26.464 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2024-09-16T15:49:26.464 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-09-16T15:49:26.464 DEBUG:teuthology.orchestra.run.smithi123:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-09-16T15:49:26.509 INFO:teuthology.orchestra.run.smithi123.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-09-16T15:49:26.509 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-09-16T15:49:26.509 DEBUG:teuthology.orchestra.run.smithi123:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-09-16T15:49:26.556 DEBUG:teuthology.orchestra.run.smithi123:> sudo modprobe fuse 2024-09-16T15:49:26.615 DEBUG:teuthology.orchestra.run.smithi123:> cat /proc/self/mounts | awk '{print $2}' 2024-09-16T15:49:26.662 INFO:teuthology.orchestra.run.smithi123.stdout:/sys 2024-09-16T15:49:26.663 INFO:teuthology.orchestra.run.smithi123.stdout:/proc 2024-09-16T15:49:26.663 INFO:teuthology.orchestra.run.smithi123.stdout:/dev 2024-09-16T15:49:26.663 INFO:teuthology.orchestra.run.smithi123.stdout:/dev/pts 2024-09-16T15:49:26.663 INFO:teuthology.orchestra.run.smithi123.stdout:/run 2024-09-16T15:49:26.663 INFO:teuthology.orchestra.run.smithi123.stdout:/ 2024-09-16T15:49:26.663 INFO:teuthology.orchestra.run.smithi123.stdout:/sys/kernel/security 2024-09-16T15:49:26.663 INFO:teuthology.orchestra.run.smithi123.stdout:/dev/shm 2024-09-16T15:49:26.663 INFO:teuthology.orchestra.run.smithi123.stdout:/run/lock 2024-09-16T15:49:26.663 INFO:teuthology.orchestra.run.smithi123.stdout:/sys/fs/cgroup 2024-09-16T15:49:26.663 INFO:teuthology.orchestra.run.smithi123.stdout:/sys/fs/pstore 2024-09-16T15:49:26.663 INFO:teuthology.orchestra.run.smithi123.stdout:/sys/fs/bpf 2024-09-16T15:49:26.664 INFO:teuthology.orchestra.run.smithi123.stdout:/proc/sys/fs/binfmt_misc 2024-09-16T15:49:26.664 INFO:teuthology.orchestra.run.smithi123.stdout:/dev/hugepages 2024-09-16T15:49:26.664 INFO:teuthology.orchestra.run.smithi123.stdout:/dev/mqueue 2024-09-16T15:49:26.664 INFO:teuthology.orchestra.run.smithi123.stdout:/sys/kernel/debug 2024-09-16T15:49:26.664 INFO:teuthology.orchestra.run.smithi123.stdout:/sys/kernel/tracing 2024-09-16T15:49:26.664 INFO:teuthology.orchestra.run.smithi123.stdout:/proc/fs/nfsd 2024-09-16T15:49:26.664 INFO:teuthology.orchestra.run.smithi123.stdout:/sys/fs/fuse/connections 2024-09-16T15:49:26.664 INFO:teuthology.orchestra.run.smithi123.stdout:/sys/kernel/config 2024-09-16T15:49:26.664 INFO:teuthology.orchestra.run.smithi123.stdout:/run/credentials/systemd-sysusers.service 2024-09-16T15:49:26.664 INFO:teuthology.orchestra.run.smithi123.stdout:/run/rpc_pipefs 2024-09-16T15:49:26.664 INFO:teuthology.orchestra.run.smithi123.stdout:/run/user/1000 2024-09-16T15:49:26.664 INFO:teuthology.orchestra.run.smithi123.stdout:/var/lib/ceph 2024-09-16T15:49:26.664 INFO:teuthology.orchestra.run.smithi123.stdout:/var/lib/ceph/osd/ceph-0 2024-09-16T15:49:26.665 INFO:teuthology.orchestra.run.smithi123.stdout:/var/lib/ceph/osd/ceph-1 2024-09-16T15:49:26.665 INFO:teuthology.orchestra.run.smithi123.stdout:/var/lib/ceph/osd/ceph-2 2024-09-16T15:49:26.665 INFO:teuthology.orchestra.run.smithi123.stdout:/var/lib/ceph/osd/ceph-3 2024-09-16T15:49:26.665 INFO:teuthology.orchestra.run.smithi123.stdout:/run/netns 2024-09-16T15:49:26.665 INFO:teuthology.orchestra.run.smithi123.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-09-16T15:49:26.665 INFO:teuthology.orchestra.run.smithi123.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-09-16T15:49:26.665 INFO:teuthology.orchestra.run.smithi123.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-09-16T15:49:26.665 INFO:teuthology.orchestra.run.smithi123.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-09-16T15:49:26.666 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:49:26.666 DEBUG:teuthology.orchestra.run.smithi123:> ls /sys/fs/fuse/connections 2024-09-16T15:49:26.713 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-09-16T15:49:26.713 DEBUG:teuthology.orchestra.run.smithi123:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2024-09-16T15:49:26.758 DEBUG:teuthology.orchestra.run.smithi123:> sudo modprobe fuse 2024-09-16T15:49:26.772 DEBUG:teuthology.orchestra.run.smithi123:> cat /proc/self/mounts | awk '{print $2}' 2024-09-16T15:49:26.822 INFO:teuthology.orchestra.run.smithi123.stdout:/sys 2024-09-16T15:49:26.822 INFO:teuthology.orchestra.run.smithi123.stdout:/proc 2024-09-16T15:49:26.822 INFO:teuthology.orchestra.run.smithi123.stdout:/dev 2024-09-16T15:49:26.822 INFO:teuthology.orchestra.run.smithi123.stdout:/dev/pts 2024-09-16T15:49:26.822 INFO:teuthology.orchestra.run.smithi123.stdout:/run 2024-09-16T15:49:26.822 INFO:teuthology.orchestra.run.smithi123.stdout:/ 2024-09-16T15:49:26.822 INFO:teuthology.orchestra.run.smithi123.stdout:/sys/kernel/security 2024-09-16T15:49:26.822 INFO:teuthology.orchestra.run.smithi123.stdout:/dev/shm 2024-09-16T15:49:26.822 INFO:teuthology.orchestra.run.smithi123.stdout:/run/lock 2024-09-16T15:49:26.822 INFO:teuthology.orchestra.run.smithi123.stdout:/sys/fs/cgroup 2024-09-16T15:49:26.822 INFO:teuthology.orchestra.run.smithi123.stdout:/sys/fs/pstore 2024-09-16T15:49:26.823 INFO:teuthology.orchestra.run.smithi123.stdout:/sys/fs/bpf 2024-09-16T15:49:26.823 INFO:teuthology.orchestra.run.smithi123.stdout:/proc/sys/fs/binfmt_misc 2024-09-16T15:49:26.823 INFO:teuthology.orchestra.run.smithi123.stdout:/dev/hugepages 2024-09-16T15:49:26.823 INFO:teuthology.orchestra.run.smithi123.stdout:/dev/mqueue 2024-09-16T15:49:26.823 INFO:teuthology.orchestra.run.smithi123.stdout:/sys/kernel/debug 2024-09-16T15:49:26.823 INFO:teuthology.orchestra.run.smithi123.stdout:/sys/kernel/tracing 2024-09-16T15:49:26.823 INFO:teuthology.orchestra.run.smithi123.stdout:/proc/fs/nfsd 2024-09-16T15:49:26.823 INFO:teuthology.orchestra.run.smithi123.stdout:/sys/fs/fuse/connections 2024-09-16T15:49:26.823 INFO:teuthology.orchestra.run.smithi123.stdout:/sys/kernel/config 2024-09-16T15:49:26.823 INFO:teuthology.orchestra.run.smithi123.stdout:/run/credentials/systemd-sysusers.service 2024-09-16T15:49:26.823 INFO:teuthology.orchestra.run.smithi123.stdout:/run/rpc_pipefs 2024-09-16T15:49:26.823 INFO:teuthology.orchestra.run.smithi123.stdout:/run/user/1000 2024-09-16T15:49:26.823 INFO:teuthology.orchestra.run.smithi123.stdout:/var/lib/ceph 2024-09-16T15:49:26.823 INFO:teuthology.orchestra.run.smithi123.stdout:/var/lib/ceph/osd/ceph-0 2024-09-16T15:49:26.824 INFO:teuthology.orchestra.run.smithi123.stdout:/var/lib/ceph/osd/ceph-1 2024-09-16T15:49:26.824 INFO:teuthology.orchestra.run.smithi123.stdout:/var/lib/ceph/osd/ceph-2 2024-09-16T15:49:26.824 INFO:teuthology.orchestra.run.smithi123.stdout:/var/lib/ceph/osd/ceph-3 2024-09-16T15:49:26.824 INFO:teuthology.orchestra.run.smithi123.stdout:/run/netns 2024-09-16T15:49:26.824 INFO:teuthology.orchestra.run.smithi123.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-09-16T15:49:26.824 INFO:teuthology.orchestra.run.smithi123.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-09-16T15:49:26.824 INFO:teuthology.orchestra.run.smithi123.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-09-16T15:49:26.824 INFO:teuthology.orchestra.run.smithi123.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-09-16T15:49:26.825 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:49:26.825 DEBUG:teuthology.orchestra.run.smithi123:> ls /sys/fs/fuse/connections 2024-09-16T15:49:26.836 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi123.stderr:2024-09-16T15:49:26.835+0000 7f311ef3f580 -1 init, newargv = 0x5597d2600f10 newargc=15 2024-09-16T15:49:26.836 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi123.stderr:ceph-fuse[48125]: starting ceph client 2024-09-16T15:49:26.843 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi123.stderr:ceph-fuse[48125]: starting fuse 2024-09-16T15:49:27.941 DEBUG:teuthology.orchestra.run.smithi123:> sudo modprobe fuse 2024-09-16T15:49:27.958 DEBUG:teuthology.orchestra.run.smithi123:> cat /proc/self/mounts | awk '{print $2}' 2024-09-16T15:49:28.008 INFO:teuthology.orchestra.run.smithi123.stdout:/sys 2024-09-16T15:49:28.008 INFO:teuthology.orchestra.run.smithi123.stdout:/proc 2024-09-16T15:49:28.008 INFO:teuthology.orchestra.run.smithi123.stdout:/dev 2024-09-16T15:49:28.008 INFO:teuthology.orchestra.run.smithi123.stdout:/dev/pts 2024-09-16T15:49:28.008 INFO:teuthology.orchestra.run.smithi123.stdout:/run 2024-09-16T15:49:28.008 INFO:teuthology.orchestra.run.smithi123.stdout:/ 2024-09-16T15:49:28.008 INFO:teuthology.orchestra.run.smithi123.stdout:/sys/kernel/security 2024-09-16T15:49:28.008 INFO:teuthology.orchestra.run.smithi123.stdout:/dev/shm 2024-09-16T15:49:28.009 INFO:teuthology.orchestra.run.smithi123.stdout:/run/lock 2024-09-16T15:49:28.009 INFO:teuthology.orchestra.run.smithi123.stdout:/sys/fs/cgroup 2024-09-16T15:49:28.009 INFO:teuthology.orchestra.run.smithi123.stdout:/sys/fs/pstore 2024-09-16T15:49:28.009 INFO:teuthology.orchestra.run.smithi123.stdout:/sys/fs/bpf 2024-09-16T15:49:28.009 INFO:teuthology.orchestra.run.smithi123.stdout:/proc/sys/fs/binfmt_misc 2024-09-16T15:49:28.009 INFO:teuthology.orchestra.run.smithi123.stdout:/dev/hugepages 2024-09-16T15:49:28.009 INFO:teuthology.orchestra.run.smithi123.stdout:/dev/mqueue 2024-09-16T15:49:28.009 INFO:teuthology.orchestra.run.smithi123.stdout:/sys/kernel/debug 2024-09-16T15:49:28.009 INFO:teuthology.orchestra.run.smithi123.stdout:/sys/kernel/tracing 2024-09-16T15:49:28.009 INFO:teuthology.orchestra.run.smithi123.stdout:/proc/fs/nfsd 2024-09-16T15:49:28.009 INFO:teuthology.orchestra.run.smithi123.stdout:/sys/fs/fuse/connections 2024-09-16T15:49:28.009 INFO:teuthology.orchestra.run.smithi123.stdout:/sys/kernel/config 2024-09-16T15:49:28.009 INFO:teuthology.orchestra.run.smithi123.stdout:/run/credentials/systemd-sysusers.service 2024-09-16T15:49:28.010 INFO:teuthology.orchestra.run.smithi123.stdout:/run/rpc_pipefs 2024-09-16T15:49:28.010 INFO:teuthology.orchestra.run.smithi123.stdout:/run/user/1000 2024-09-16T15:49:28.010 INFO:teuthology.orchestra.run.smithi123.stdout:/var/lib/ceph 2024-09-16T15:49:28.010 INFO:teuthology.orchestra.run.smithi123.stdout:/var/lib/ceph/osd/ceph-0 2024-09-16T15:49:28.010 INFO:teuthology.orchestra.run.smithi123.stdout:/var/lib/ceph/osd/ceph-1 2024-09-16T15:49:28.010 INFO:teuthology.orchestra.run.smithi123.stdout:/var/lib/ceph/osd/ceph-2 2024-09-16T15:49:28.010 INFO:teuthology.orchestra.run.smithi123.stdout:/var/lib/ceph/osd/ceph-3 2024-09-16T15:49:28.010 INFO:teuthology.orchestra.run.smithi123.stdout:/run/netns 2024-09-16T15:49:28.010 INFO:teuthology.orchestra.run.smithi123.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-09-16T15:49:28.010 INFO:teuthology.orchestra.run.smithi123.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-09-16T15:49:28.010 INFO:teuthology.orchestra.run.smithi123.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-09-16T15:49:28.010 INFO:teuthology.orchestra.run.smithi123.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-09-16T15:49:28.010 INFO:teuthology.orchestra.run.smithi123.stdout:/home/ubuntu/cephtest/mnt.0 2024-09-16T15:49:28.011 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:49:28.011 DEBUG:teuthology.orchestra.run.smithi123:> ls /sys/fs/fuse/connections 2024-09-16T15:49:28.058 INFO:teuthology.orchestra.run.smithi123.stdout:45 2024-09-16T15:49:28.058 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [45] 2024-09-16T15:49:28.058 DEBUG:teuthology.orchestra.run.smithi123:> sudo stdin-killer -- python3 -c ' 2024-09-16T15:49:28.058 DEBUG:teuthology.orchestra.run.smithi123:> import glob 2024-09-16T15:49:28.058 DEBUG:teuthology.orchestra.run.smithi123:> import re 2024-09-16T15:49:28.058 DEBUG:teuthology.orchestra.run.smithi123:> import os 2024-09-16T15:49:28.058 DEBUG:teuthology.orchestra.run.smithi123:> import subprocess 2024-09-16T15:49:28.058 DEBUG:teuthology.orchestra.run.smithi123:> 2024-09-16T15:49:28.059 DEBUG:teuthology.orchestra.run.smithi123:> def _find_admin_socket(client_name): 2024-09-16T15:49:28.059 DEBUG:teuthology.orchestra.run.smithi123:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2024-09-16T15:49:28.059 DEBUG:teuthology.orchestra.run.smithi123:> files = glob.glob(asok_path) 2024-09-16T15:49:28.059 DEBUG:teuthology.orchestra.run.smithi123:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-09-16T15:49:28.059 DEBUG:teuthology.orchestra.run.smithi123:> 2024-09-16T15:49:28.059 DEBUG:teuthology.orchestra.run.smithi123:> # Given a non-glob path, it better be there 2024-09-16T15:49:28.059 DEBUG:teuthology.orchestra.run.smithi123:> if "*" not in asok_path: 2024-09-16T15:49:28.059 DEBUG:teuthology.orchestra.run.smithi123:> assert(len(files) == 1) 2024-09-16T15:49:28.059 DEBUG:teuthology.orchestra.run.smithi123:> return files[0] 2024-09-16T15:49:28.059 DEBUG:teuthology.orchestra.run.smithi123:> 2024-09-16T15:49:28.059 DEBUG:teuthology.orchestra.run.smithi123:> for f in files: 2024-09-16T15:49:28.059 DEBUG:teuthology.orchestra.run.smithi123:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-09-16T15:49:28.059 DEBUG:teuthology.orchestra.run.smithi123:> if os.path.exists("/proc/{0}".format(pid)): 2024-09-16T15:49:28.059 DEBUG:teuthology.orchestra.run.smithi123:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-09-16T15:49:28.059 DEBUG:teuthology.orchestra.run.smithi123:> contents = proc_f.read() 2024-09-16T15:49:28.060 DEBUG:teuthology.orchestra.run.smithi123:> if mountpoint in contents: 2024-09-16T15:49:28.060 DEBUG:teuthology.orchestra.run.smithi123:> return f 2024-09-16T15:49:28.060 DEBUG:teuthology.orchestra.run.smithi123:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-09-16T15:49:28.060 DEBUG:teuthology.orchestra.run.smithi123:> 2024-09-16T15:49:28.060 DEBUG:teuthology.orchestra.run.smithi123:> print(_find_admin_socket("client.0")) 2024-09-16T15:49:28.060 DEBUG:teuthology.orchestra.run.smithi123:> ' 2024-09-16T15:49:28.182 INFO:teuthology.orchestra.run.smithi123.stdout:/var/run/ceph/ceph-client.0.48125.asok 2024-09-16T15:49:28.186 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:28 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-16T15:49:28.194 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.48125.asok 2024-09-16T15:49:28.194 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:49:28.194 DEBUG:teuthology.orchestra.run.smithi123:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.48125.asok status 2024-09-16T15:49:28.356 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2024-09-16T15:49:28.356 INFO:teuthology.orchestra.run.smithi123.stdout: "metadata": { 2024-09-16T15:49:28.356 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph_sha1": "33f95d2ae1bcbe7eb04404303f179eeeaa6982df", 2024-09-16T15:49:28.356 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph_version": "ceph version 19.3.0-4938-g33f95d2a (33f95d2ae1bcbe7eb04404303f179eeeaa6982df) squid (dev)", 2024-09-16T15:49:28.357 INFO:teuthology.orchestra.run.smithi123.stdout: "entity_id": "0", 2024-09-16T15:49:28.357 INFO:teuthology.orchestra.run.smithi123.stdout: "hostname": "smithi123", 2024-09-16T15:49:28.357 INFO:teuthology.orchestra.run.smithi123.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2024-09-16T15:49:28.357 INFO:teuthology.orchestra.run.smithi123.stdout: "pid": "48125", 2024-09-16T15:49:28.357 INFO:teuthology.orchestra.run.smithi123.stdout: "root": "/" 2024-09-16T15:49:28.357 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:49:28.357 INFO:teuthology.orchestra.run.smithi123.stdout: "dentry_count": 0, 2024-09-16T15:49:28.357 INFO:teuthology.orchestra.run.smithi123.stdout: "dentry_pinned_count": 0, 2024-09-16T15:49:28.357 INFO:teuthology.orchestra.run.smithi123.stdout: "id": 7704, 2024-09-16T15:49:28.357 INFO:teuthology.orchestra.run.smithi123.stdout: "inst": { 2024-09-16T15:49:28.357 INFO:teuthology.orchestra.run.smithi123.stdout: "name": { 2024-09-16T15:49:28.357 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "client", 2024-09-16T15:49:28.357 INFO:teuthology.orchestra.run.smithi123.stdout: "num": 7704 2024-09-16T15:49:28.358 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:49:28.358 INFO:teuthology.orchestra.run.smithi123.stdout: "addr": { 2024-09-16T15:49:28.358 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "v1", 2024-09-16T15:49:28.358 INFO:teuthology.orchestra.run.smithi123.stdout: "addr": "192.168.0.1:0", 2024-09-16T15:49:28.358 INFO:teuthology.orchestra.run.smithi123.stdout: "nonce": 2866416703 2024-09-16T15:49:28.358 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-16T15:49:28.358 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:49:28.358 INFO:teuthology.orchestra.run.smithi123.stdout: "addr": { 2024-09-16T15:49:28.358 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "v1", 2024-09-16T15:49:28.358 INFO:teuthology.orchestra.run.smithi123.stdout: "addr": "192.168.0.1:0", 2024-09-16T15:49:28.358 INFO:teuthology.orchestra.run.smithi123.stdout: "nonce": 2866416703 2024-09-16T15:49:28.358 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:49:28.358 INFO:teuthology.orchestra.run.smithi123.stdout: "inst_str": "client.7704 192.168.0.1:0/2866416703", 2024-09-16T15:49:28.359 INFO:teuthology.orchestra.run.smithi123.stdout: "addr_str": "192.168.0.1:0/2866416703", 2024-09-16T15:49:28.359 INFO:teuthology.orchestra.run.smithi123.stdout: "inode_count": 1, 2024-09-16T15:49:28.359 INFO:teuthology.orchestra.run.smithi123.stdout: "mds_epoch": 76, 2024-09-16T15:49:28.359 INFO:teuthology.orchestra.run.smithi123.stdout: "osd_epoch": 119, 2024-09-16T15:49:28.359 INFO:teuthology.orchestra.run.smithi123.stdout: "osd_epoch_barrier": 0, 2024-09-16T15:49:28.359 INFO:teuthology.orchestra.run.smithi123.stdout: "blocklisted": false, 2024-09-16T15:49:28.359 INFO:teuthology.orchestra.run.smithi123.stdout: "fs_name": "cephfs" 2024-09-16T15:49:28.359 INFO:teuthology.orchestra.run.smithi123.stdout:} 2024-09-16T15:49:28.370 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:49:28.370 DEBUG:teuthology.orchestra.run.smithi123:> stat --file-system '--printf=%T 2024-09-16T15:49:28.370 DEBUG:teuthology.orchestra.run.smithi123:> ' -- /home/ubuntu/cephtest/mnt.0 2024-09-16T15:49:28.376 INFO:teuthology.orchestra.run.smithi123.stdout:fuseblk 2024-09-16T15:49:28.376 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2024-09-16T15:49:28.377 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:49:28.377 DEBUG:teuthology.orchestra.run.smithi123:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-09-16T15:49:28.434 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-09-16T15:49:28.606 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:28.607+0000 7fde223a3640 1 -- 172.21.15.123:0/2245564515 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fde1c0b2b00 msgr2=0x7fde1c0d22e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:28.606 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:28.607+0000 7fde223a3640 1 --2- 172.21.15.123:0/2245564515 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fde1c0b2b00 0x7fde1c0d22e0 secure :-1 s=READY pgs=1286 cs=0 l=1 rev1=1 crypto rx=0x7fde1c08e2b0 tx=0x7fde1001c770 comp rx=0 tx=0).stop 2024-09-16T15:49:28.606 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:28.607+0000 7fde223a3640 1 -- 172.21.15.123:0/2245564515 shutdown_connections 2024-09-16T15:49:28.606 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:28.607+0000 7fde223a3640 1 --2- 172.21.15.123:0/2245564515 >> v2:172.21.15.184:3300/0 conn(0x7fde1c0d2820 0x7fde1c0cea00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:28.606 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:28.607+0000 7fde223a3640 1 --2- 172.21.15.123:0/2245564515 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fde1c0b2b00 0x7fde1c0d22e0 unknown :-1 s=CLOSED pgs=1286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:28.606 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:28.607+0000 7fde223a3640 1 --2- 172.21.15.123:0/2245564515 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fde1c161620 0x7fde1c0d1da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:28.606 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:28.607+0000 7fde223a3640 1 -- 172.21.15.123:0/2245564515 >> 172.21.15.123:0/2245564515 conn(0x7fde1c0d3ea0 msgr2=0x7fde1c0d42b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:49:28.606 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:28.607+0000 7fde223a3640 1 -- 172.21.15.123:0/2245564515 shutdown_connections 2024-09-16T15:49:28.606 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:28.607+0000 7fde223a3640 1 -- 172.21.15.123:0/2245564515 wait complete. 2024-09-16T15:49:28.607 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:28.607+0000 7fde223a3640 1 Processor -- start 2024-09-16T15:49:28.607 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:28.607+0000 7fde223a3640 1 -- start start 2024-09-16T15:49:28.609 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:28.607+0000 7fde223a3640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fde1c0b2b00 0x7fde1c0cc8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:49:28.609 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:28.607+0000 7fde223a3640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fde1c0d2820 0x7fde1c150060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:49:28.609 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:28.607+0000 7fde223a3640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fde1c161620 0x7fde1c1fb450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:49:28.609 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:28.607+0000 7fde223a3640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fde1c0b1140 con 0x7fde1c161620 2024-09-16T15:49:28.609 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:28.607+0000 7fde223a3640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fde1c0b0fc0 con 0x7fde1c0b2b00 2024-09-16T15:49:28.609 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:28.607+0000 7fde223a3640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fde1c0b12c0 con 0x7fde1c0d2820 2024-09-16T15:49:28.609 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:28.607+0000 7fde20940640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fde1c161620 0x7fde1c1fb450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:49:28.610 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:28.607+0000 7fde20940640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fde1c161620 0x7fde1c1fb450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:35548/0 (socket says 172.21.15.123:35548) 2024-09-16T15:49:28.610 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:28.607+0000 7fde20940640 1 -- 172.21.15.123:0/3290436074 learned_addr learned my addr 172.21.15.123:0/3290436074 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:49:28.610 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:28.607+0000 7fde1b7fe640 1 --2- 172.21.15.123:0/3290436074 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fde1c0d2820 0x7fde1c150060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:49:28.610 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:28.607+0000 7fde1bfff640 1 --2- 172.21.15.123:0/3290436074 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fde1c0b2b00 0x7fde1c0cc8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:49:28.610 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:28.607+0000 7fde20940640 1 -- 172.21.15.123:0/3290436074 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fde1c0d2820 msgr2=0x7fde1c150060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:28.610 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:28.607+0000 7fde20940640 1 --2- 172.21.15.123:0/3290436074 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fde1c0d2820 0x7fde1c150060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:28.610 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:28.607+0000 7fde20940640 1 -- 172.21.15.123:0/3290436074 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fde1c0b2b00 msgr2=0x7fde1c0cc8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:28.610 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:28.607+0000 7fde20940640 1 --2- 172.21.15.123:0/3290436074 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fde1c0b2b00 0x7fde1c0cc8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:28.610 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:28.607+0000 7fde20940640 1 -- 172.21.15.123:0/3290436074 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fde1c0cbef0 con 0x7fde1c161620 2024-09-16T15:49:28.610 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:28.607+0000 7fde20940640 1 --2- 172.21.15.123:0/3290436074 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fde1c161620 0x7fde1c1fb450 secure :-1 s=READY pgs=1287 cs=0 l=1 rev1=1 crypto rx=0x7fde1400e990 tx=0x7fde1400ee50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:49:28.610 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:28.607+0000 7fde18ff9640 1 -- 172.21.15.123:0/3290436074 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde14004d30 con 0x7fde1c161620 2024-09-16T15:49:28.610 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:28.611+0000 7fde18ff9640 1 -- 172.21.15.123:0/3290436074 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fde1400caa0 con 0x7fde1c161620 2024-09-16T15:49:28.610 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:28.611+0000 7fde223a3640 1 -- 172.21.15.123:0/3290436074 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fde1c06e0b0 con 0x7fde1c161620 2024-09-16T15:49:28.611 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:28.611+0000 7fde18ff9640 1 -- 172.21.15.123:0/3290436074 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde14005020 con 0x7fde1c161620 2024-09-16T15:49:28.611 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:28.611+0000 7fde223a3640 1 -- 172.21.15.123:0/3290436074 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fde1c06e670 con 0x7fde1c161620 2024-09-16T15:49:28.612 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:28.611+0000 7fde223a3640 1 -- 172.21.15.123:0/3290436074 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fde1c0cea00 con 0x7fde1c161620 2024-09-16T15:49:28.612 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:28.611+0000 7fde18ff9640 1 -- 172.21.15.123:0/3290436074 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 19) ==== 97488+0+0 (secure 0 0 0) 0x7fde140051c0 con 0x7fde1c161620 2024-09-16T15:49:28.612 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:28.611+0000 7fde18ff9640 1 --2- 172.21.15.123:0/3290436074 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fde08073340 0x7fde08075800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:49:28.612 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:28.611+0000 7fde18ff9640 1 -- 172.21.15.123:0/3290436074 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(119..119 src has 1..119) ==== 5737+0+0 (secure 0 0 0) 0x7fde14095310 con 0x7fde1c161620 2024-09-16T15:49:28.612 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:28.611+0000 7fde1bfff640 1 --2- 172.21.15.123:0/3290436074 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fde08073340 0x7fde08075800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:49:28.612 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:28.611+0000 7fde1bfff640 1 --2- 172.21.15.123:0/3290436074 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fde08073340 0x7fde08075800 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fde1c1fb3b0 tx=0x7fde0c00a250 comp rx=0 tx=0).ready entity=mgr.4150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:49:28.615 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:28.615+0000 7fde18ff9640 1 -- 172.21.15.123:0/3290436074 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7fde14016030 con 0x7fde1c161620 2024-09-16T15:49:28.827 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:28.827+0000 7fde223a3640 1 -- 172.21.15.123:0/3290436074 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7fde1c0b6780 con 0x7fde1c161620 2024-09-16T15:49:28.828 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:28.827+0000 7fde18ff9640 1 -- 172.21.15.123:0/3290436074 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v76) ==== 53+0+73 (secure 0 0 0) 0x7fde1405f820 con 0x7fde1c161620 2024-09-16T15:49:28.828 INFO:teuthology.orchestra.run.smithi123.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-09-16T15:49:28.830 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:28.831+0000 7fde223a3640 1 -- 172.21.15.123:0/3290436074 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fde08073340 msgr2=0x7fde08075800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:28.831 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:28.831+0000 7fde223a3640 1 --2- 172.21.15.123:0/3290436074 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fde08073340 0x7fde08075800 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fde1c1fb3b0 tx=0x7fde0c00a250 comp rx=0 tx=0).stop 2024-09-16T15:49:28.831 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:28.831+0000 7fde223a3640 1 -- 172.21.15.123:0/3290436074 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fde1c161620 msgr2=0x7fde1c1fb450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:28.831 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:28.831+0000 7fde223a3640 1 --2- 172.21.15.123:0/3290436074 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fde1c161620 0x7fde1c1fb450 secure :-1 s=READY pgs=1287 cs=0 l=1 rev1=1 crypto rx=0x7fde1400e990 tx=0x7fde1400ee50 comp rx=0 tx=0).stop 2024-09-16T15:49:28.831 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:28.831+0000 7fde223a3640 1 -- 172.21.15.123:0/3290436074 shutdown_connections 2024-09-16T15:49:28.831 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:28.831+0000 7fde223a3640 1 --2- 172.21.15.123:0/3290436074 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fde08073340 0x7fde08075800 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:28.831 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:28.831+0000 7fde223a3640 1 --2- 172.21.15.123:0/3290436074 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fde1c161620 0x7fde1c1fb450 unknown :-1 s=CLOSED pgs=1287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:28.831 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:28.831+0000 7fde223a3640 1 --2- 172.21.15.123:0/3290436074 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fde1c0d2820 0x7fde1c150060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:28.831 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:28.831+0000 7fde223a3640 1 --2- 172.21.15.123:0/3290436074 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fde1c0b2b00 0x7fde1c0cc8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:28.831 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:28.831+0000 7fde223a3640 1 -- 172.21.15.123:0/3290436074 >> 172.21.15.123:0/3290436074 conn(0x7fde1c0d3ea0 msgr2=0x7fde1c0d43f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:49:28.831 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:28.831+0000 7fde223a3640 1 -- 172.21.15.123:0/3290436074 shutdown_connections 2024-09-16T15:49:28.831 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:28.831+0000 7fde223a3640 1 -- 172.21.15.123:0/3290436074 wait complete. 2024-09-16T15:49:28.847 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-09-16T15:49:28.847 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-09-16T15:49:28.847 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi184.front.sepia.ceph.com 2024-09-16T15:49:28.847 INFO:tasks.cephfs.mount:self.client.name = client.1 2024-09-16T15:49:28.847 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2024-09-16T15:49:28.847 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-09-16T15:49:28.847 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-09-16T15:49:28.848 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-09-16T15:49:28.848 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.1' netns for '/home/ubuntu/cephtest/mnt.1' 2024-09-16T15:49:28.848 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:49:28.848 DEBUG:teuthology.orchestra.run.smithi184:> ip addr 2024-09-16T15:49:28.855 INFO:teuthology.orchestra.run.smithi184.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-09-16T15:49:28.855 INFO:teuthology.orchestra.run.smithi184.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-09-16T15:49:28.855 INFO:teuthology.orchestra.run.smithi184.stdout: inet 127.0.0.1/8 scope host lo 2024-09-16T15:49:28.855 INFO:teuthology.orchestra.run.smithi184.stdout: valid_lft forever preferred_lft forever 2024-09-16T15:49:28.855 INFO:teuthology.orchestra.run.smithi184.stdout: inet6 ::1/128 scope host 2024-09-16T15:49:28.855 INFO:teuthology.orchestra.run.smithi184.stdout: valid_lft forever preferred_lft forever 2024-09-16T15:49:28.855 INFO:teuthology.orchestra.run.smithi184.stdout:2: eno1: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-09-16T15:49:28.856 INFO:teuthology.orchestra.run.smithi184.stdout: link/ether 0c:c4:7a:d9:37:16 brd ff:ff:ff:ff:ff:ff 2024-09-16T15:49:28.856 INFO:teuthology.orchestra.run.smithi184.stdout: altname enp7s0f0 2024-09-16T15:49:28.856 INFO:teuthology.orchestra.run.smithi184.stdout:3: eno2: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-09-16T15:49:28.856 INFO:teuthology.orchestra.run.smithi184.stdout: link/ether 0c:c4:7a:d9:37:17 brd ff:ff:ff:ff:ff:ff 2024-09-16T15:49:28.856 INFO:teuthology.orchestra.run.smithi184.stdout: altname enp7s0f1 2024-09-16T15:49:28.856 INFO:teuthology.orchestra.run.smithi184.stdout:4: enp3s0f0: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-09-16T15:49:28.856 INFO:teuthology.orchestra.run.smithi184.stdout: link/ether 0c:c4:7a:88:7b:8e brd ff:ff:ff:ff:ff:ff 2024-09-16T15:49:28.856 INFO:teuthology.orchestra.run.smithi184.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-09-16T15:49:28.856 INFO:teuthology.orchestra.run.smithi184.stdout: link/ether 0c:c4:7a:88:7b:8f brd ff:ff:ff:ff:ff:ff 2024-09-16T15:49:28.856 INFO:teuthology.orchestra.run.smithi184.stdout: inet 172.21.15.184/20 metric 100 brd 172.21.15.255 scope global dynamic enp3s0f1 2024-09-16T15:49:28.856 INFO:teuthology.orchestra.run.smithi184.stdout: valid_lft 42088sec preferred_lft 42088sec 2024-09-16T15:49:28.856 INFO:teuthology.orchestra.run.smithi184.stdout: inet6 fe80::ec4:7aff:fe88:7b8f/64 scope link 2024-09-16T15:49:28.856 INFO:teuthology.orchestra.run.smithi184.stdout: valid_lft forever preferred_lft forever 2024-09-16T15:49:28.857 INFO:teuthology.orchestra.run.smithi184.stdout:6: docker0: mtu 1500 qdisc noqueue state DOWN group default 2024-09-16T15:49:28.857 INFO:teuthology.orchestra.run.smithi184.stdout: link/ether 02:42:5d:0a:44:b9 brd ff:ff:ff:ff:ff:ff 2024-09-16T15:49:28.857 INFO:teuthology.orchestra.run.smithi184.stdout: inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0 2024-09-16T15:49:28.857 INFO:teuthology.orchestra.run.smithi184.stdout: valid_lft forever preferred_lft forever 2024-09-16T15:49:28.857 INFO:teuthology.orchestra.run.smithi184.stdout:7: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-09-16T15:49:28.857 INFO:teuthology.orchestra.run.smithi184.stdout: link/ether 5e:1f:84:6d:79:ae brd ff:ff:ff:ff:ff:ff 2024-09-16T15:49:28.857 INFO:teuthology.orchestra.run.smithi184.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-09-16T15:49:28.857 INFO:teuthology.orchestra.run.smithi184.stdout: valid_lft forever preferred_lft forever 2024-09-16T15:49:28.857 INFO:teuthology.orchestra.run.smithi184.stdout: inet6 fe80::5c1f:84ff:fe6d:79ae/64 scope link 2024-09-16T15:49:28.857 INFO:teuthology.orchestra.run.smithi184.stdout: valid_lft forever preferred_lft forever 2024-09-16T15:49:28.857 INFO:teuthology.orchestra.run.smithi184.stdout:8: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-09-16T15:49:28.857 INFO:teuthology.orchestra.run.smithi184.stdout: link/ether 0e:5c:f6:1f:82:2f brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.1 2024-09-16T15:49:28.857 INFO:teuthology.orchestra.run.smithi184.stdout: inet6 fe80::c5c:f6ff:fe1f:822f/64 scope link 2024-09-16T15:49:28.857 INFO:teuthology.orchestra.run.smithi184.stdout: valid_lft forever preferred_lft forever 2024-09-16T15:49:28.858 INFO:teuthology.orchestra.run.smithi184.stdout:9: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-09-16T15:49:28.858 INFO:teuthology.orchestra.run.smithi184.stdout: link/ether c6:a6:3d:69:d0:db brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.3 2024-09-16T15:49:28.858 INFO:teuthology.orchestra.run.smithi184.stdout: inet6 fe80::c4a6:3dff:fe69:d0db/64 scope link 2024-09-16T15:49:28.858 INFO:teuthology.orchestra.run.smithi184.stdout: valid_lft forever preferred_lft forever 2024-09-16T15:49:28.858 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:49:28.858 DEBUG:teuthology.orchestra.run.smithi184:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-09-16T15:49:28.909 INFO:teuthology.orchestra.run.smithi184.stdout:1 2024-09-16T15:49:28.911 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:49:28.911 DEBUG:teuthology.orchestra.run.smithi184:> route 2024-09-16T15:49:28.963 INFO:teuthology.orchestra.run.smithi184.stdout:Kernel IP routing table 2024-09-16T15:49:28.963 INFO:teuthology.orchestra.run.smithi184.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-09-16T15:49:28.963 INFO:teuthology.orchestra.run.smithi184.stdout:default _gateway 0.0.0.0 UG 0 0 0 enp3s0f1 2024-09-16T15:49:28.963 INFO:teuthology.orchestra.run.smithi184.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-09-16T15:49:28.963 INFO:teuthology.orchestra.run.smithi184.stdout:158.69.67.47 _gateway 255.255.255.255 UGH 100 0 0 enp3s0f1 2024-09-16T15:49:28.964 INFO:teuthology.orchestra.run.smithi184.stdout:172.17.0.0 0.0.0.0 255.255.0.0 U 0 0 0 docker0 2024-09-16T15:49:28.964 INFO:teuthology.orchestra.run.smithi184.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-09-16T15:49:28.964 INFO:teuthology.orchestra.run.smithi184.stdout:ns1.front.sepia 0.0.0.0 255.255.255.255 UH 100 0 0 enp3s0f1 2024-09-16T15:49:28.964 INFO:teuthology.orchestra.run.smithi184.stdout:172.21.0.2 0.0.0.0 255.255.255.255 UH 100 0 0 enp3s0f1 2024-09-16T15:49:28.964 INFO:teuthology.orchestra.run.smithi184.stdout:_gateway 0.0.0.0 255.255.255.255 UH 100 0 0 enp3s0f1 2024-09-16T15:49:28.964 INFO:teuthology.orchestra.run.smithi184.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-09-16T15:49:28.964 DEBUG:teuthology.orchestra.run.smithi184:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-09-16T15:49:28.964 DEBUG:teuthology.orchestra.run.smithi184:> set -e 2024-09-16T15:49:28.964 DEBUG:teuthology.orchestra.run.smithi184:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-09-16T15:49:28.964 DEBUG:teuthology.orchestra.run.smithi184:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-09-16T15:49:28.964 DEBUG:teuthology.orchestra.run.smithi184:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-09-16T15:49:28.964 DEBUG:teuthology.orchestra.run.smithi184:> ') 2024-09-16T15:49:29.051 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:49:29 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-09-16T15:49:29.080 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:49:29 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-16T15:49:29.087 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:49:29.087 DEBUG:teuthology.orchestra.run.smithi184:> ip netns list 2024-09-16T15:49:29.093 INFO:teuthology.orchestra.run.smithi184.stdout:ceph-ns--home-ubuntu-cephtest-mnt.3 (id: 1) 2024-09-16T15:49:29.093 INFO:teuthology.orchestra.run.smithi184.stdout:ceph-ns--home-ubuntu-cephtest-mnt.1 (id: 0) 2024-09-16T15:49:29.093 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.1' netns for '/home/ubuntu/cephtest/mnt.1' 2024-09-16T15:49:29.093 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:49:29.094 DEBUG:teuthology.orchestra.run.smithi184:> sudo ip link set brx.0 up 2024-09-16T15:49:29.149 INFO:tasks.cephfs.fuse_mount:Client client.1 config is {} 2024-09-16T15:49:29.149 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-09-16T15:49:29.149 DEBUG:teuthology.orchestra.run.smithi184:> mkdir -p -v /home/ubuntu/cephtest/mnt.1 2024-09-16T15:49:29.202 INFO:teuthology.orchestra.run.smithi184.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.1' 2024-09-16T15:49:29.202 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-09-16T15:49:29.202 DEBUG:teuthology.orchestra.run.smithi184:> chmod 0000 /home/ubuntu/cephtest/mnt.1 2024-09-16T15:49:29.249 DEBUG:teuthology.orchestra.run.smithi184:> sudo modprobe fuse 2024-09-16T15:49:29.307 DEBUG:teuthology.orchestra.run.smithi184:> cat /proc/self/mounts | awk '{print $2}' 2024-09-16T15:49:29.355 INFO:teuthology.orchestra.run.smithi184.stdout:/sys 2024-09-16T15:49:29.355 INFO:teuthology.orchestra.run.smithi184.stdout:/proc 2024-09-16T15:49:29.355 INFO:teuthology.orchestra.run.smithi184.stdout:/dev 2024-09-16T15:49:29.355 INFO:teuthology.orchestra.run.smithi184.stdout:/dev/pts 2024-09-16T15:49:29.355 INFO:teuthology.orchestra.run.smithi184.stdout:/run 2024-09-16T15:49:29.355 INFO:teuthology.orchestra.run.smithi184.stdout:/ 2024-09-16T15:49:29.355 INFO:teuthology.orchestra.run.smithi184.stdout:/sys/kernel/security 2024-09-16T15:49:29.355 INFO:teuthology.orchestra.run.smithi184.stdout:/dev/shm 2024-09-16T15:49:29.355 INFO:teuthology.orchestra.run.smithi184.stdout:/run/lock 2024-09-16T15:49:29.355 INFO:teuthology.orchestra.run.smithi184.stdout:/sys/fs/cgroup 2024-09-16T15:49:29.355 INFO:teuthology.orchestra.run.smithi184.stdout:/sys/fs/pstore 2024-09-16T15:49:29.356 INFO:teuthology.orchestra.run.smithi184.stdout:/sys/fs/bpf 2024-09-16T15:49:29.356 INFO:teuthology.orchestra.run.smithi184.stdout:/proc/sys/fs/binfmt_misc 2024-09-16T15:49:29.356 INFO:teuthology.orchestra.run.smithi184.stdout:/dev/hugepages 2024-09-16T15:49:29.356 INFO:teuthology.orchestra.run.smithi184.stdout:/dev/mqueue 2024-09-16T15:49:29.356 INFO:teuthology.orchestra.run.smithi184.stdout:/sys/kernel/debug 2024-09-16T15:49:29.356 INFO:teuthology.orchestra.run.smithi184.stdout:/sys/kernel/tracing 2024-09-16T15:49:29.356 INFO:teuthology.orchestra.run.smithi184.stdout:/proc/fs/nfsd 2024-09-16T15:49:29.356 INFO:teuthology.orchestra.run.smithi184.stdout:/sys/fs/fuse/connections 2024-09-16T15:49:29.356 INFO:teuthology.orchestra.run.smithi184.stdout:/sys/kernel/config 2024-09-16T15:49:29.356 INFO:teuthology.orchestra.run.smithi184.stdout:/run/credentials/systemd-sysusers.service 2024-09-16T15:49:29.356 INFO:teuthology.orchestra.run.smithi184.stdout:/run/rpc_pipefs 2024-09-16T15:49:29.356 INFO:teuthology.orchestra.run.smithi184.stdout:/run/user/1000 2024-09-16T15:49:29.357 INFO:teuthology.orchestra.run.smithi184.stdout:/var/lib/ceph 2024-09-16T15:49:29.357 INFO:teuthology.orchestra.run.smithi184.stdout:/var/lib/ceph/osd/ceph-4 2024-09-16T15:49:29.357 INFO:teuthology.orchestra.run.smithi184.stdout:/var/lib/ceph/osd/ceph-5 2024-09-16T15:49:29.357 INFO:teuthology.orchestra.run.smithi184.stdout:/var/lib/ceph/osd/ceph-6 2024-09-16T15:49:29.357 INFO:teuthology.orchestra.run.smithi184.stdout:/var/lib/ceph/osd/ceph-7 2024-09-16T15:49:29.357 INFO:teuthology.orchestra.run.smithi184.stdout:/run/netns 2024-09-16T15:49:29.357 INFO:teuthology.orchestra.run.smithi184.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-09-16T15:49:29.357 INFO:teuthology.orchestra.run.smithi184.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-09-16T15:49:29.357 INFO:teuthology.orchestra.run.smithi184.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-09-16T15:49:29.357 INFO:teuthology.orchestra.run.smithi184.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-09-16T15:49:29.358 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:49:29.358 DEBUG:teuthology.orchestra.run.smithi184:> ls /sys/fs/fuse/connections 2024-09-16T15:49:29.405 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-09-16T15:49:29.406 DEBUG:teuthology.orchestra.run.smithi184:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.1 --id 1) 2024-09-16T15:49:29.451 DEBUG:teuthology.orchestra.run.smithi184:> sudo modprobe fuse 2024-09-16T15:49:29.465 DEBUG:teuthology.orchestra.run.smithi184:> cat /proc/self/mounts | awk '{print $2}' 2024-09-16T15:49:29.514 INFO:teuthology.orchestra.run.smithi184.stdout:/sys 2024-09-16T15:49:29.514 INFO:teuthology.orchestra.run.smithi184.stdout:/proc 2024-09-16T15:49:29.514 INFO:teuthology.orchestra.run.smithi184.stdout:/dev 2024-09-16T15:49:29.514 INFO:teuthology.orchestra.run.smithi184.stdout:/dev/pts 2024-09-16T15:49:29.514 INFO:teuthology.orchestra.run.smithi184.stdout:/run 2024-09-16T15:49:29.514 INFO:teuthology.orchestra.run.smithi184.stdout:/ 2024-09-16T15:49:29.514 INFO:teuthology.orchestra.run.smithi184.stdout:/sys/kernel/security 2024-09-16T15:49:29.514 INFO:teuthology.orchestra.run.smithi184.stdout:/dev/shm 2024-09-16T15:49:29.515 INFO:teuthology.orchestra.run.smithi184.stdout:/run/lock 2024-09-16T15:49:29.515 INFO:teuthology.orchestra.run.smithi184.stdout:/sys/fs/cgroup 2024-09-16T15:49:29.515 INFO:teuthology.orchestra.run.smithi184.stdout:/sys/fs/pstore 2024-09-16T15:49:29.515 INFO:teuthology.orchestra.run.smithi184.stdout:/sys/fs/bpf 2024-09-16T15:49:29.515 INFO:teuthology.orchestra.run.smithi184.stdout:/proc/sys/fs/binfmt_misc 2024-09-16T15:49:29.515 INFO:teuthology.orchestra.run.smithi184.stdout:/dev/hugepages 2024-09-16T15:49:29.515 INFO:teuthology.orchestra.run.smithi184.stdout:/dev/mqueue 2024-09-16T15:49:29.515 INFO:teuthology.orchestra.run.smithi184.stdout:/sys/kernel/debug 2024-09-16T15:49:29.515 INFO:teuthology.orchestra.run.smithi184.stdout:/sys/kernel/tracing 2024-09-16T15:49:29.515 INFO:teuthology.orchestra.run.smithi184.stdout:/proc/fs/nfsd 2024-09-16T15:49:29.515 INFO:teuthology.orchestra.run.smithi184.stdout:/sys/fs/fuse/connections 2024-09-16T15:49:29.515 INFO:teuthology.orchestra.run.smithi184.stdout:/sys/kernel/config 2024-09-16T15:49:29.515 INFO:teuthology.orchestra.run.smithi184.stdout:/run/credentials/systemd-sysusers.service 2024-09-16T15:49:29.515 INFO:teuthology.orchestra.run.smithi184.stdout:/run/rpc_pipefs 2024-09-16T15:49:29.515 INFO:teuthology.orchestra.run.smithi184.stdout:/run/user/1000 2024-09-16T15:49:29.516 INFO:teuthology.orchestra.run.smithi184.stdout:/var/lib/ceph 2024-09-16T15:49:29.516 INFO:teuthology.orchestra.run.smithi184.stdout:/var/lib/ceph/osd/ceph-4 2024-09-16T15:49:29.516 INFO:teuthology.orchestra.run.smithi184.stdout:/var/lib/ceph/osd/ceph-5 2024-09-16T15:49:29.516 INFO:teuthology.orchestra.run.smithi184.stdout:/var/lib/ceph/osd/ceph-6 2024-09-16T15:49:29.516 INFO:teuthology.orchestra.run.smithi184.stdout:/var/lib/ceph/osd/ceph-7 2024-09-16T15:49:29.516 INFO:teuthology.orchestra.run.smithi184.stdout:/run/netns 2024-09-16T15:49:29.516 INFO:teuthology.orchestra.run.smithi184.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-09-16T15:49:29.516 INFO:teuthology.orchestra.run.smithi184.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-09-16T15:49:29.516 INFO:teuthology.orchestra.run.smithi184.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-09-16T15:49:29.516 INFO:teuthology.orchestra.run.smithi184.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-09-16T15:49:29.517 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:49:29.517 DEBUG:teuthology.orchestra.run.smithi184:> ls /sys/fs/fuse/connections 2024-09-16T15:49:29.531 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi184.stderr:2024-09-16T15:49:29.528+0000 7f5991925580 -1 init, newargv = 0x55fa2daebb30 newargc=15 2024-09-16T15:49:29.531 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi184.stderr:ceph-fuse[35889]: starting ceph client 2024-09-16T15:49:29.538 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi184.stderr:ceph-fuse[35889]: starting fuse 2024-09-16T15:49:30.637 DEBUG:teuthology.orchestra.run.smithi184:> sudo modprobe fuse 2024-09-16T15:49:30.657 DEBUG:teuthology.orchestra.run.smithi184:> cat /proc/self/mounts | awk '{print $2}' 2024-09-16T15:49:30.707 INFO:teuthology.orchestra.run.smithi184.stdout:/sys 2024-09-16T15:49:30.707 INFO:teuthology.orchestra.run.smithi184.stdout:/proc 2024-09-16T15:49:30.707 INFO:teuthology.orchestra.run.smithi184.stdout:/dev 2024-09-16T15:49:30.707 INFO:teuthology.orchestra.run.smithi184.stdout:/dev/pts 2024-09-16T15:49:30.707 INFO:teuthology.orchestra.run.smithi184.stdout:/run 2024-09-16T15:49:30.707 INFO:teuthology.orchestra.run.smithi184.stdout:/ 2024-09-16T15:49:30.707 INFO:teuthology.orchestra.run.smithi184.stdout:/sys/kernel/security 2024-09-16T15:49:30.707 INFO:teuthology.orchestra.run.smithi184.stdout:/dev/shm 2024-09-16T15:49:30.707 INFO:teuthology.orchestra.run.smithi184.stdout:/run/lock 2024-09-16T15:49:30.708 INFO:teuthology.orchestra.run.smithi184.stdout:/sys/fs/cgroup 2024-09-16T15:49:30.708 INFO:teuthology.orchestra.run.smithi184.stdout:/sys/fs/pstore 2024-09-16T15:49:30.708 INFO:teuthology.orchestra.run.smithi184.stdout:/sys/fs/bpf 2024-09-16T15:49:30.708 INFO:teuthology.orchestra.run.smithi184.stdout:/proc/sys/fs/binfmt_misc 2024-09-16T15:49:30.708 INFO:teuthology.orchestra.run.smithi184.stdout:/dev/hugepages 2024-09-16T15:49:30.708 INFO:teuthology.orchestra.run.smithi184.stdout:/dev/mqueue 2024-09-16T15:49:30.708 INFO:teuthology.orchestra.run.smithi184.stdout:/sys/kernel/debug 2024-09-16T15:49:30.708 INFO:teuthology.orchestra.run.smithi184.stdout:/sys/kernel/tracing 2024-09-16T15:49:30.708 INFO:teuthology.orchestra.run.smithi184.stdout:/proc/fs/nfsd 2024-09-16T15:49:30.708 INFO:teuthology.orchestra.run.smithi184.stdout:/sys/fs/fuse/connections 2024-09-16T15:49:30.708 INFO:teuthology.orchestra.run.smithi184.stdout:/sys/kernel/config 2024-09-16T15:49:30.708 INFO:teuthology.orchestra.run.smithi184.stdout:/run/credentials/systemd-sysusers.service 2024-09-16T15:49:30.708 INFO:teuthology.orchestra.run.smithi184.stdout:/run/rpc_pipefs 2024-09-16T15:49:30.708 INFO:teuthology.orchestra.run.smithi184.stdout:/run/user/1000 2024-09-16T15:49:30.709 INFO:teuthology.orchestra.run.smithi184.stdout:/var/lib/ceph 2024-09-16T15:49:30.709 INFO:teuthology.orchestra.run.smithi184.stdout:/var/lib/ceph/osd/ceph-4 2024-09-16T15:49:30.709 INFO:teuthology.orchestra.run.smithi184.stdout:/var/lib/ceph/osd/ceph-5 2024-09-16T15:49:30.709 INFO:teuthology.orchestra.run.smithi184.stdout:/var/lib/ceph/osd/ceph-6 2024-09-16T15:49:30.709 INFO:teuthology.orchestra.run.smithi184.stdout:/var/lib/ceph/osd/ceph-7 2024-09-16T15:49:30.709 INFO:teuthology.orchestra.run.smithi184.stdout:/run/netns 2024-09-16T15:49:30.709 INFO:teuthology.orchestra.run.smithi184.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-09-16T15:49:30.709 INFO:teuthology.orchestra.run.smithi184.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-09-16T15:49:30.709 INFO:teuthology.orchestra.run.smithi184.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-09-16T15:49:30.709 INFO:teuthology.orchestra.run.smithi184.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-09-16T15:49:30.709 INFO:teuthology.orchestra.run.smithi184.stdout:/home/ubuntu/cephtest/mnt.1 2024-09-16T15:49:30.710 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:49:30.710 DEBUG:teuthology.orchestra.run.smithi184:> ls /sys/fs/fuse/connections 2024-09-16T15:49:30.757 INFO:teuthology.orchestra.run.smithi184.stdout:45 2024-09-16T15:49:30.758 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [45] 2024-09-16T15:49:30.758 DEBUG:teuthology.orchestra.run.smithi184:> sudo stdin-killer -- python3 -c ' 2024-09-16T15:49:30.758 DEBUG:teuthology.orchestra.run.smithi184:> import glob 2024-09-16T15:49:30.758 DEBUG:teuthology.orchestra.run.smithi184:> import re 2024-09-16T15:49:30.758 DEBUG:teuthology.orchestra.run.smithi184:> import os 2024-09-16T15:49:30.758 DEBUG:teuthology.orchestra.run.smithi184:> import subprocess 2024-09-16T15:49:30.758 DEBUG:teuthology.orchestra.run.smithi184:> 2024-09-16T15:49:30.758 DEBUG:teuthology.orchestra.run.smithi184:> def _find_admin_socket(client_name): 2024-09-16T15:49:30.758 DEBUG:teuthology.orchestra.run.smithi184:> asok_path = "/var/run/ceph/ceph-client.1.*.asok" 2024-09-16T15:49:30.758 DEBUG:teuthology.orchestra.run.smithi184:> files = glob.glob(asok_path) 2024-09-16T15:49:30.759 DEBUG:teuthology.orchestra.run.smithi184:> mountpoint = "/home/ubuntu/cephtest/mnt.1" 2024-09-16T15:49:30.759 DEBUG:teuthology.orchestra.run.smithi184:> 2024-09-16T15:49:30.759 DEBUG:teuthology.orchestra.run.smithi184:> # Given a non-glob path, it better be there 2024-09-16T15:49:30.759 DEBUG:teuthology.orchestra.run.smithi184:> if "*" not in asok_path: 2024-09-16T15:49:30.759 DEBUG:teuthology.orchestra.run.smithi184:> assert(len(files) == 1) 2024-09-16T15:49:30.759 DEBUG:teuthology.orchestra.run.smithi184:> return files[0] 2024-09-16T15:49:30.759 DEBUG:teuthology.orchestra.run.smithi184:> 2024-09-16T15:49:30.759 DEBUG:teuthology.orchestra.run.smithi184:> for f in files: 2024-09-16T15:49:30.759 DEBUG:teuthology.orchestra.run.smithi184:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-09-16T15:49:30.759 DEBUG:teuthology.orchestra.run.smithi184:> if os.path.exists("/proc/{0}".format(pid)): 2024-09-16T15:49:30.759 DEBUG:teuthology.orchestra.run.smithi184:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-09-16T15:49:30.759 DEBUG:teuthology.orchestra.run.smithi184:> contents = proc_f.read() 2024-09-16T15:49:30.759 DEBUG:teuthology.orchestra.run.smithi184:> if mountpoint in contents: 2024-09-16T15:49:30.759 DEBUG:teuthology.orchestra.run.smithi184:> return f 2024-09-16T15:49:30.759 DEBUG:teuthology.orchestra.run.smithi184:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-09-16T15:49:30.759 DEBUG:teuthology.orchestra.run.smithi184:> 2024-09-16T15:49:30.760 DEBUG:teuthology.orchestra.run.smithi184:> print(_find_admin_socket("client.1")) 2024-09-16T15:49:30.760 DEBUG:teuthology.orchestra.run.smithi184:> ' 2024-09-16T15:49:30.884 INFO:teuthology.orchestra.run.smithi184.stdout:/var/run/ceph/ceph-client.1.35889.asok 2024-09-16T15:49:30.887 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:49:30 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-16T15:49:30.896 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.1.35889.asok 2024-09-16T15:49:30.896 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:49:30.896 DEBUG:teuthology.orchestra.run.smithi184:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.1.35889.asok status 2024-09-16T15:49:31.048 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-16T15:49:31.048 INFO:teuthology.orchestra.run.smithi184.stdout: "metadata": { 2024-09-16T15:49:31.048 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph_sha1": "33f95d2ae1bcbe7eb04404303f179eeeaa6982df", 2024-09-16T15:49:31.048 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph_version": "ceph version 19.3.0-4938-g33f95d2a (33f95d2ae1bcbe7eb04404303f179eeeaa6982df) squid (dev)", 2024-09-16T15:49:31.048 INFO:teuthology.orchestra.run.smithi184.stdout: "entity_id": "1", 2024-09-16T15:49:31.049 INFO:teuthology.orchestra.run.smithi184.stdout: "hostname": "smithi184", 2024-09-16T15:49:31.049 INFO:teuthology.orchestra.run.smithi184.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.1", 2024-09-16T15:49:31.049 INFO:teuthology.orchestra.run.smithi184.stdout: "pid": "35889", 2024-09-16T15:49:31.049 INFO:teuthology.orchestra.run.smithi184.stdout: "root": "/" 2024-09-16T15:49:31.049 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-16T15:49:31.049 INFO:teuthology.orchestra.run.smithi184.stdout: "dentry_count": 0, 2024-09-16T15:49:31.049 INFO:teuthology.orchestra.run.smithi184.stdout: "dentry_pinned_count": 0, 2024-09-16T15:49:31.049 INFO:teuthology.orchestra.run.smithi184.stdout: "id": 5669, 2024-09-16T15:49:31.049 INFO:teuthology.orchestra.run.smithi184.stdout: "inst": { 2024-09-16T15:49:31.049 INFO:teuthology.orchestra.run.smithi184.stdout: "name": { 2024-09-16T15:49:31.049 INFO:teuthology.orchestra.run.smithi184.stdout: "type": "client", 2024-09-16T15:49:31.049 INFO:teuthology.orchestra.run.smithi184.stdout: "num": 5669 2024-09-16T15:49:31.050 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-16T15:49:31.050 INFO:teuthology.orchestra.run.smithi184.stdout: "addr": { 2024-09-16T15:49:31.050 INFO:teuthology.orchestra.run.smithi184.stdout: "type": "v1", 2024-09-16T15:49:31.050 INFO:teuthology.orchestra.run.smithi184.stdout: "addr": "192.168.0.1:0", 2024-09-16T15:49:31.050 INFO:teuthology.orchestra.run.smithi184.stdout: "nonce": 833378318 2024-09-16T15:49:31.050 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-16T15:49:31.051 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-16T15:49:31.051 INFO:teuthology.orchestra.run.smithi184.stdout: "addr": { 2024-09-16T15:49:31.051 INFO:teuthology.orchestra.run.smithi184.stdout: "type": "v1", 2024-09-16T15:49:31.051 INFO:teuthology.orchestra.run.smithi184.stdout: "addr": "192.168.0.1:0", 2024-09-16T15:49:31.051 INFO:teuthology.orchestra.run.smithi184.stdout: "nonce": 833378318 2024-09-16T15:49:31.051 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-16T15:49:31.051 INFO:teuthology.orchestra.run.smithi184.stdout: "inst_str": "client.5669 192.168.0.1:0/833378318", 2024-09-16T15:49:31.051 INFO:teuthology.orchestra.run.smithi184.stdout: "addr_str": "192.168.0.1:0/833378318", 2024-09-16T15:49:31.051 INFO:teuthology.orchestra.run.smithi184.stdout: "inode_count": 1, 2024-09-16T15:49:31.051 INFO:teuthology.orchestra.run.smithi184.stdout: "mds_epoch": 76, 2024-09-16T15:49:31.051 INFO:teuthology.orchestra.run.smithi184.stdout: "osd_epoch": 119, 2024-09-16T15:49:31.051 INFO:teuthology.orchestra.run.smithi184.stdout: "osd_epoch_barrier": 0, 2024-09-16T15:49:31.051 INFO:teuthology.orchestra.run.smithi184.stdout: "blocklisted": false, 2024-09-16T15:49:31.052 INFO:teuthology.orchestra.run.smithi184.stdout: "fs_name": "cephfs" 2024-09-16T15:49:31.052 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-16T15:49:31.061 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:49:31.062 DEBUG:teuthology.orchestra.run.smithi184:> stat --file-system '--printf=%T 2024-09-16T15:49:31.062 DEBUG:teuthology.orchestra.run.smithi184:> ' -- /home/ubuntu/cephtest/mnt.1 2024-09-16T15:49:31.067 INFO:teuthology.orchestra.run.smithi184.stdout:fuseblk 2024-09-16T15:49:31.068 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2024-09-16T15:49:31.068 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:49:31.068 DEBUG:teuthology.orchestra.run.smithi184:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.1 2024-09-16T15:49:31.124 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs flag set enable_multiple true --yes-i-really-mean-it 2024-09-16T15:49:31.265 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:31.263+0000 7ff04e9ad640 1 -- 172.21.15.123:0/4103723569 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff0480b9ad0 msgr2=0x7ff0480d0840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:31.265 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:31.263+0000 7ff04e9ad640 1 --2- 172.21.15.123:0/4103723569 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff0480b9ad0 0x7ff0480d0840 secure :-1 s=READY pgs=1288 cs=0 l=1 rev1=1 crypto rx=0x7ff0480d3830 tx=0x7ff03001c890 comp rx=0 tx=0).stop 2024-09-16T15:49:31.265 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:31.263+0000 7ff04e9ad640 1 -- 172.21.15.123:0/4103723569 shutdown_connections 2024-09-16T15:49:31.265 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:31.263+0000 7ff04e9ad640 1 --2- 172.21.15.123:0/4103723569 >> v2:172.21.15.184:3300/0 conn(0x7ff0480d4950 0x7ff0480d0d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:31.265 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:31.263+0000 7ff04e9ad640 1 --2- 172.21.15.123:0/4103723569 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff0480b9ad0 0x7ff0480d0840 unknown :-1 s=CLOSED pgs=1288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:31.265 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:31.263+0000 7ff04e9ad640 1 -- 172.21.15.123:0/4103723569 >> 172.21.15.123:0/4103723569 conn(0x7ff0480d3e30 msgr2=0x7ff0480d4240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:49:31.265 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:31.263+0000 7ff04e9ad640 1 -- 172.21.15.123:0/4103723569 shutdown_connections 2024-09-16T15:49:31.265 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:31.263+0000 7ff04e9ad640 1 -- 172.21.15.123:0/4103723569 wait complete. 2024-09-16T15:49:31.266 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:31.267+0000 7ff04e9ad640 1 Processor -- start 2024-09-16T15:49:31.266 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:31.267+0000 7ff04e9ad640 1 -- start start 2024-09-16T15:49:31.268 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:31.267+0000 7ff04e9ad640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7ff0480b9ad0 0x7ff04815ed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:49:31.268 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:31.267+0000 7ff04e9ad640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff0480d4950 0x7ff04815f2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:49:31.268 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:31.267+0000 7ff04e9ad640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff04815df80 0x7ff0480b3ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:49:31.268 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:31.267+0000 7ff04e9ad640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7ff04816ae70 con 0x7ff04815df80 2024-09-16T15:49:31.269 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:31.267+0000 7ff04e9ad640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7ff04816acf0 con 0x7ff0480d4950 2024-09-16T15:49:31.269 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:31.267+0000 7ff04e9ad640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7ff04816aff0 con 0x7ff0480b9ad0 2024-09-16T15:49:31.269 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:31.267+0000 7ff04cf4a640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff04815df80 0x7ff0480b3ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:49:31.269 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:31.267+0000 7ff04cf4a640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff04815df80 0x7ff0480b3ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:35552/0 (socket says 172.21.15.123:35552) 2024-09-16T15:49:31.269 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:31.267+0000 7ff04cf4a640 1 -- 172.21.15.123:0/2715191176 learned_addr learned my addr 172.21.15.123:0/2715191176 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:49:31.269 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:31.267+0000 7ff047fff640 1 --2- 172.21.15.123:0/2715191176 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7ff0480b9ad0 0x7ff04815ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:49:31.269 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:31.267+0000 7ff03f7fe640 1 --2- 172.21.15.123:0/2715191176 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff0480d4950 0x7ff04815f2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:49:31.269 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:31.267+0000 7ff04cf4a640 1 -- 172.21.15.123:0/2715191176 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7ff0480b9ad0 msgr2=0x7ff04815ed70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:31.269 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:31.267+0000 7ff04cf4a640 1 --2- 172.21.15.123:0/2715191176 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7ff0480b9ad0 0x7ff04815ed70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:31.269 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:31.267+0000 7ff04cf4a640 1 -- 172.21.15.123:0/2715191176 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff0480d4950 msgr2=0x7ff04815f2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:31.269 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:31.267+0000 7ff04cf4a640 1 --2- 172.21.15.123:0/2715191176 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff0480d4950 0x7ff04815f2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:31.269 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:31.267+0000 7ff04cf4a640 1 -- 172.21.15.123:0/2715191176 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff0480a1d60 con 0x7ff04815df80 2024-09-16T15:49:31.269 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:31.267+0000 7ff04cf4a640 1 --2- 172.21.15.123:0/2715191176 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff04815df80 0x7ff0480b3ca0 secure :-1 s=READY pgs=1289 cs=0 l=1 rev1=1 crypto rx=0x7ff04000ccb0 tx=0x7ff04000a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:49:31.269 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:31.267+0000 7ff0457fa640 1 -- 172.21.15.123:0/2715191176 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff040004b30 con 0x7ff04815df80 2024-09-16T15:49:31.270 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:31.267+0000 7ff0457fa640 1 -- 172.21.15.123:0/2715191176 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7ff040004de0 con 0x7ff04815df80 2024-09-16T15:49:31.270 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:31.267+0000 7ff04e9ad640 1 -- 172.21.15.123:0/2715191176 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff048159b80 con 0x7ff04815df80 2024-09-16T15:49:31.270 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:31.267+0000 7ff0457fa640 1 -- 172.21.15.123:0/2715191176 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff040005280 con 0x7ff04815df80 2024-09-16T15:49:31.270 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:31.267+0000 7ff04e9ad640 1 -- 172.21.15.123:0/2715191176 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff04815a140 con 0x7ff04815df80 2024-09-16T15:49:31.271 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:31.271+0000 7ff04e9ad640 1 -- 172.21.15.123:0/2715191176 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff0480d0620 con 0x7ff04815df80 2024-09-16T15:49:31.271 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:31.271+0000 7ff0457fa640 1 -- 172.21.15.123:0/2715191176 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 19) ==== 97488+0+0 (secure 0 0 0) 0x7ff0400054e0 con 0x7ff04815df80 2024-09-16T15:49:31.271 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:31.271+0000 7ff0457fa640 1 --2- 172.21.15.123:0/2715191176 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7ff028073340 0x7ff028075800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:49:31.271 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:31.271+0000 7ff0457fa640 1 -- 172.21.15.123:0/2715191176 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(119..119 src has 1..119) ==== 5737+0+0 (secure 0 0 0) 0x7ff0400954c0 con 0x7ff04815df80 2024-09-16T15:49:31.271 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:31.271+0000 7ff047fff640 1 --2- 172.21.15.123:0/2715191176 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7ff028073340 0x7ff028075800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:49:31.271 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:31.271+0000 7ff047fff640 1 --2- 172.21.15.123:0/2715191176 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7ff028073340 0x7ff028075800 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7ff03800b810 tx=0x7ff038008040 comp rx=0 tx=0).ready entity=mgr.4150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:49:31.275 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:31.275+0000 7ff0457fa640 1 -- 172.21.15.123:0/2715191176 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7ff040014030 con 0x7ff04815df80 2024-09-16T15:49:31.486 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:31.487+0000 7ff04e9ad640 1 -- 172.21.15.123:0/2715191176 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "fs flag set", "flag_name": "enable_multiple", "val": "true", "yes_i_really_mean_it": true} v 0) -- 0x7ff0481611d0 con 0x7ff04815df80 2024-09-16T15:49:32.168 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:32.167+0000 7ff0457fa640 1 -- 172.21.15.123:0/2715191176 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "fs flag set", "flag_name": "enable_multiple", "val": "true", "yes_i_really_mean_it": true}]=0 v77) ==== 136+0+0 (secure 0 0 0) 0x7ff040060020 con 0x7ff04815df80 2024-09-16T15:49:32.171 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:32.171+0000 7ff04e9ad640 1 -- 172.21.15.123:0/2715191176 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7ff028073340 msgr2=0x7ff028075800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:32.172 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:32.171+0000 7ff04e9ad640 1 --2- 172.21.15.123:0/2715191176 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7ff028073340 0x7ff028075800 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7ff03800b810 tx=0x7ff038008040 comp rx=0 tx=0).stop 2024-09-16T15:49:32.172 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:32.171+0000 7ff04e9ad640 1 -- 172.21.15.123:0/2715191176 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff04815df80 msgr2=0x7ff0480b3ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:32.172 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:32.171+0000 7ff04e9ad640 1 --2- 172.21.15.123:0/2715191176 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff04815df80 0x7ff0480b3ca0 secure :-1 s=READY pgs=1289 cs=0 l=1 rev1=1 crypto rx=0x7ff04000ccb0 tx=0x7ff04000a620 comp rx=0 tx=0).stop 2024-09-16T15:49:32.172 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:32.171+0000 7ff04e9ad640 1 -- 172.21.15.123:0/2715191176 shutdown_connections 2024-09-16T15:49:32.172 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:32.171+0000 7ff04e9ad640 1 --2- 172.21.15.123:0/2715191176 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7ff028073340 0x7ff028075800 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:32.172 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:32.171+0000 7ff04e9ad640 1 --2- 172.21.15.123:0/2715191176 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff04815df80 0x7ff0480b3ca0 unknown :-1 s=CLOSED pgs=1289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:32.172 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:32.171+0000 7ff04e9ad640 1 --2- 172.21.15.123:0/2715191176 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff0480d4950 0x7ff04815f2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:32.172 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:32.171+0000 7ff04e9ad640 1 --2- 172.21.15.123:0/2715191176 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7ff0480b9ad0 0x7ff04815ed70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:32.172 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:32.171+0000 7ff04e9ad640 1 -- 172.21.15.123:0/2715191176 >> 172.21.15.123:0/2715191176 conn(0x7ff0480d3e30 msgr2=0x7ff048160ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:49:32.173 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:32.171+0000 7ff04e9ad640 1 -- 172.21.15.123:0/2715191176 shutdown_connections 2024-09-16T15:49:32.173 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:32.171+0000 7ff04e9ad640 1 -- 172.21.15.123:0/2715191176 wait complete. 2024-09-16T15:49:32.193 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2024-09-16T15:49:32.324 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:32.323+0000 7fa983843640 1 -- 172.21.15.123:0/136167901 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fa97c15bca0 msgr2=0x7fa97c0d0630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:32.324 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:32.323+0000 7fa983843640 1 --2- 172.21.15.123:0/136167901 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fa97c15bca0 0x7fa97c0d0630 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fa974009900 tx=0x7fa97401c820 comp rx=0 tx=0).stop 2024-09-16T15:49:32.325 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:32.323+0000 7fa983843640 1 -- 172.21.15.123:0/136167901 shutdown_connections 2024-09-16T15:49:32.325 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:32.323+0000 7fa983843640 1 --2- 172.21.15.123:0/136167901 >> v2:172.21.15.184:3300/0 conn(0x7fa97c074600 0x7fa97c06e750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:32.325 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:32.323+0000 7fa983843640 1 --2- 172.21.15.123:0/136167901 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fa97c15bca0 0x7fa97c0d0630 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:32.325 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:32.323+0000 7fa983843640 1 -- 172.21.15.123:0/136167901 >> 172.21.15.123:0/136167901 conn(0x7fa97c0cfb10 msgr2=0x7fa97c0cff20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:49:32.325 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:32.323+0000 7fa983843640 1 -- 172.21.15.123:0/136167901 shutdown_connections 2024-09-16T15:49:32.325 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:32.323+0000 7fa983843640 1 -- 172.21.15.123:0/136167901 wait complete. 2024-09-16T15:49:32.327 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:32.327+0000 7fa983843640 1 Processor -- start 2024-09-16T15:49:32.327 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:32.327+0000 7fa983843640 1 -- start start 2024-09-16T15:49:32.327 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:32.327+0000 7fa983843640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa97c0726c0 0x7fa97c0c6be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:49:32.328 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:32.327+0000 7fa983843640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa97c074600 0x7fa97c1f9390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:49:32.328 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:32.327+0000 7fa983843640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fa97c15bca0 0x7fa97c1f98d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:49:32.328 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:32.327+0000 7fa983843640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fa97c168a50 con 0x7fa97c074600 2024-09-16T15:49:32.328 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:32.327+0000 7fa983843640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fa97c06fa60 con 0x7fa97c0726c0 2024-09-16T15:49:32.328 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:32.327+0000 7fa983843640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fa97c168bd0 con 0x7fa97c15bca0 2024-09-16T15:49:32.328 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:32.327+0000 7fa980dde640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa97c074600 0x7fa97c1f9390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:49:32.328 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:32.327+0000 7fa980dde640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa97c074600 0x7fa97c1f9390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:35556/0 (socket says 172.21.15.123:35556) 2024-09-16T15:49:32.329 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:32.327+0000 7fa980dde640 1 -- 172.21.15.123:0/4095849378 learned_addr learned my addr 172.21.15.123:0/4095849378 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:49:32.329 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:32.327+0000 7fa9815df640 1 --2- 172.21.15.123:0/4095849378 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa97c0726c0 0x7fa97c0c6be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:49:32.329 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:32.327+0000 7fa981de0640 1 --2- 172.21.15.123:0/4095849378 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fa97c15bca0 0x7fa97c1f98d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:49:32.329 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:32.327+0000 7fa980dde640 1 -- 172.21.15.123:0/4095849378 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fa97c15bca0 msgr2=0x7fa97c1f98d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:32.329 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:32.327+0000 7fa980dde640 1 --2- 172.21.15.123:0/4095849378 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fa97c15bca0 0x7fa97c1f98d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:32.329 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:32.327+0000 7fa980dde640 1 -- 172.21.15.123:0/4095849378 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa97c0726c0 msgr2=0x7fa97c0c6be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:32.329 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:32.327+0000 7fa980dde640 1 --2- 172.21.15.123:0/4095849378 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa97c0726c0 0x7fa97c0c6be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:32.329 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:32.327+0000 7fa980dde640 1 -- 172.21.15.123:0/4095849378 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa97c0c88c0 con 0x7fa97c074600 2024-09-16T15:49:32.329 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:32.327+0000 7fa980dde640 1 --2- 172.21.15.123:0/4095849378 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa97c074600 0x7fa97c1f9390 secure :-1 s=READY pgs=1290 cs=0 l=1 rev1=1 crypto rx=0x7fa97000c500 tx=0x7fa97000c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:49:32.329 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:32.327+0000 7fa979ffb640 1 -- 172.21.15.123:0/4095849378 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa970017070 con 0x7fa97c074600 2024-09-16T15:49:32.330 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:32.327+0000 7fa979ffb640 1 -- 172.21.15.123:0/4095849378 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fa9700053c0 con 0x7fa97c074600 2024-09-16T15:49:32.330 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:32.327+0000 7fa983843640 1 -- 172.21.15.123:0/4095849378 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa97c0c8b80 con 0x7fa97c074600 2024-09-16T15:49:32.330 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:32.327+0000 7fa979ffb640 1 -- 172.21.15.123:0/4095849378 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa970012890 con 0x7fa97c074600 2024-09-16T15:49:32.330 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:32.327+0000 7fa983843640 1 -- 172.21.15.123:0/4095849378 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa97c0c9420 con 0x7fa97c074600 2024-09-16T15:49:32.331 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:32.327+0000 7fa983843640 1 -- 172.21.15.123:0/4095849378 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa97c06e750 con 0x7fa97c074600 2024-09-16T15:49:32.331 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:32.331+0000 7fa979ffb640 1 -- 172.21.15.123:0/4095849378 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 19) ==== 97488+0+0 (secure 0 0 0) 0x7fa97001a070 con 0x7fa97c074600 2024-09-16T15:49:32.331 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:32.331+0000 7fa979ffb640 1 --2- 172.21.15.123:0/4095849378 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fa968073370 0x7fa968075830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:49:32.331 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:32.331+0000 7fa9815df640 1 --2- 172.21.15.123:0/4095849378 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fa968073370 0x7fa968075830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:49:32.331 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:32.331+0000 7fa979ffb640 1 -- 172.21.15.123:0/4095849378 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(119..119 src has 1..119) ==== 5737+0+0 (secure 0 0 0) 0x7fa970093a30 con 0x7fa97c074600 2024-09-16T15:49:32.331 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:32.331+0000 7fa9815df640 1 --2- 172.21.15.123:0/4095849378 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fa968073370 0x7fa968075830 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fa9740af000 tx=0x7fa97401b060 comp rx=0 tx=0).ready entity=mgr.4150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:49:32.333 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:32.335+0000 7fa979ffb640 1 -- 172.21.15.123:0/4095849378 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7fa97005e4a0 con 0x7fa97c074600 2024-09-16T15:49:32.524 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:32.523+0000 7fa983843640 1 -- 172.21.15.123:0/4095849378 --> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) -- 0x7fa97c15f630 con 0x7fa968073370 2024-09-16T15:49:32.526 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:32.527+0000 7fa979ffb640 1 -- 172.21.15.123:0/4095849378 <== mgr.4150 v2:172.21.15.123:6836/2464759676 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+41 (secure 0 0 0) 0x7fa97c15f630 con 0x7fa968073370 2024-09-16T15:49:32.526 INFO:teuthology.orchestra.run.smithi123.stdout:[ 2024-09-16T15:49:32.526 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-09-16T15:49:32.526 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "cephfs" 2024-09-16T15:49:32.527 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-16T15:49:32.527 INFO:teuthology.orchestra.run.smithi123.stdout:] 2024-09-16T15:49:32.529 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:32.531+0000 7fa983843640 1 -- 172.21.15.123:0/4095849378 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fa968073370 msgr2=0x7fa968075830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:32.529 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:32.531+0000 7fa983843640 1 --2- 172.21.15.123:0/4095849378 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fa968073370 0x7fa968075830 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fa9740af000 tx=0x7fa97401b060 comp rx=0 tx=0).stop 2024-09-16T15:49:32.529 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:32.531+0000 7fa983843640 1 -- 172.21.15.123:0/4095849378 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa97c074600 msgr2=0x7fa97c1f9390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:32.530 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:32.531+0000 7fa983843640 1 --2- 172.21.15.123:0/4095849378 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa97c074600 0x7fa97c1f9390 secure :-1 s=READY pgs=1290 cs=0 l=1 rev1=1 crypto rx=0x7fa97000c500 tx=0x7fa97000c9c0 comp rx=0 tx=0).stop 2024-09-16T15:49:32.530 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:32.531+0000 7fa983843640 1 -- 172.21.15.123:0/4095849378 shutdown_connections 2024-09-16T15:49:32.530 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:32.531+0000 7fa983843640 1 --2- 172.21.15.123:0/4095849378 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fa968073370 0x7fa968075830 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:32.530 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:32.531+0000 7fa983843640 1 --2- 172.21.15.123:0/4095849378 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fa97c15bca0 0x7fa97c1f98d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:32.530 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:32.531+0000 7fa983843640 1 --2- 172.21.15.123:0/4095849378 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa97c074600 0x7fa97c1f9390 unknown :-1 s=CLOSED pgs=1290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:32.530 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:32.531+0000 7fa983843640 1 --2- 172.21.15.123:0/4095849378 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa97c0726c0 0x7fa97c0c6be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:32.530 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:32.531+0000 7fa983843640 1 -- 172.21.15.123:0/4095849378 >> 172.21.15.123:0/4095849378 conn(0x7fa97c0cfb10 msgr2=0x7fa97c0c3640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:49:32.530 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:32.531+0000 7fa983843640 1 -- 172.21.15.123:0/4095849378 shutdown_connections 2024-09-16T15:49:32.530 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:32.531+0000 7fa983843640 1 -- 172.21.15.123:0/4095849378 wait complete. 2024-09-16T15:49:32.545 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-09-16T15:49:32.671 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:32.671+0000 7f97e3d95640 1 -- 172.21.15.123:0/1632230867 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f97dc0aff90 msgr2=0x7f97dc0b0440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:32.671 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:32.671+0000 7f97e3d95640 1 --2- 172.21.15.123:0/1632230867 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f97dc0aff90 0x7f97dc0b0440 secure :-1 s=READY pgs=1291 cs=0 l=1 rev1=1 crypto rx=0x7f97dc074070 tx=0x7f97d801cd50 comp rx=0 tx=0).stop 2024-09-16T15:49:32.671 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:32.671+0000 7f97e3d95640 1 -- 172.21.15.123:0/1632230867 shutdown_connections 2024-09-16T15:49:32.671 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:32.671+0000 7f97e3d95640 1 --2- 172.21.15.123:0/1632230867 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f97dc0aff90 0x7f97dc0b0440 unknown :-1 s=CLOSED pgs=1291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:32.671 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:32.671+0000 7f97e3d95640 1 --2- 172.21.15.123:0/1632230867 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f97dc0cc340 0x7f97dc0ce7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:32.671 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:32.671+0000 7f97e3d95640 1 --2- 172.21.15.123:0/1632230867 >> v2:172.21.15.184:3300/0 conn(0x7f97dc15adc0 0x7f97dc0ce2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:32.671 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:32.671+0000 7f97e3d95640 1 -- 172.21.15.123:0/1632230867 >> 172.21.15.123:0/1632230867 conn(0x7f97dc0741a0 msgr2=0x7f97dc0745b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:49:32.671 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:32.671+0000 7f97e3d95640 1 -- 172.21.15.123:0/1632230867 shutdown_connections 2024-09-16T15:49:32.672 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:32.671+0000 7f97e3d95640 1 -- 172.21.15.123:0/1632230867 wait complete. 2024-09-16T15:49:32.673 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:32.671+0000 7f97e3d95640 1 Processor -- start 2024-09-16T15:49:32.674 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:32.671+0000 7f97e3d95640 1 -- start start 2024-09-16T15:49:32.674 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:32.671+0000 7f97e3d95640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f97dc0aff90 0x7f97dc163ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:49:32.674 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:32.671+0000 7f97e3d95640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f97dc0cc340 0x7f97dc162d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:49:32.674 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:32.671+0000 7f97e3d95640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f97dc15adc0 0x7f97dc154240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:49:32.674 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:32.675+0000 7f97e3d95640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f97dc0cd0d0 con 0x7f97dc0aff90 2024-09-16T15:49:32.674 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:32.675+0000 7f97e3d95640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f97dc0ccf50 con 0x7f97dc0cc340 2024-09-16T15:49:32.674 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:32.675+0000 7f97e3d95640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f97dc0cd250 con 0x7f97dc15adc0 2024-09-16T15:49:32.674 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:32.675+0000 7f97e1b31640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f97dc0aff90 0x7f97dc163ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:49:32.674 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:32.675+0000 7f97e1b31640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f97dc0aff90 0x7f97dc163ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:56506/0 (socket says 172.21.15.123:56506) 2024-09-16T15:49:32.674 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:32.675+0000 7f97e1b31640 1 -- 172.21.15.123:0/2911484307 learned_addr learned my addr 172.21.15.123:0/2911484307 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:49:32.674 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:32.675+0000 7f97e2332640 1 --2- 172.21.15.123:0/2911484307 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f97dc15adc0 0x7f97dc154240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:49:32.675 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:32.675+0000 7f97e1330640 1 --2- 172.21.15.123:0/2911484307 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f97dc0cc340 0x7f97dc162d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:49:32.675 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:32.675+0000 7f97e1b31640 1 -- 172.21.15.123:0/2911484307 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f97dc15adc0 msgr2=0x7f97dc154240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:32.675 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:32.675+0000 7f97e1b31640 1 --2- 172.21.15.123:0/2911484307 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f97dc15adc0 0x7f97dc154240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:32.675 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:32.675+0000 7f97e1b31640 1 -- 172.21.15.123:0/2911484307 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f97dc0cc340 msgr2=0x7f97dc162d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:32.675 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:32.675+0000 7f97e1b31640 1 --2- 172.21.15.123:0/2911484307 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f97dc0cc340 0x7f97dc162d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:32.675 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:32.675+0000 7f97e1b31640 1 -- 172.21.15.123:0/2911484307 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f97dc156c70 con 0x7f97dc0aff90 2024-09-16T15:49:32.675 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:32.675+0000 7f97e2332640 1 --2- 172.21.15.123:0/2911484307 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f97dc15adc0 0x7f97dc154240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T15:49:32.675 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:32.675+0000 7f97e1330640 1 --2- 172.21.15.123:0/2911484307 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f97dc0cc340 0x7f97dc162d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T15:49:32.675 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:32.675+0000 7f97e1b31640 1 --2- 172.21.15.123:0/2911484307 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f97dc0aff90 0x7f97dc163ea0 secure :-1 s=READY pgs=1292 cs=0 l=1 rev1=1 crypto rx=0x7f97d400d950 tx=0x7f97d400de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:49:32.675 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:32.675+0000 7f97d27fc640 1 -- 172.21.15.123:0/2911484307 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97d4017070 con 0x7f97dc0aff90 2024-09-16T15:49:32.675 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:32.675+0000 7f97d27fc640 1 -- 172.21.15.123:0/2911484307 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f97d4009d20 con 0x7f97dc0aff90 2024-09-16T15:49:32.675 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:32.675+0000 7f97e3d95640 1 -- 172.21.15.123:0/2911484307 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f97dc156f50 con 0x7f97dc0aff90 2024-09-16T15:49:32.675 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:32.675+0000 7f97d27fc640 1 -- 172.21.15.123:0/2911484307 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97d40106c0 con 0x7f97dc0aff90 2024-09-16T15:49:32.676 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:32.675+0000 7f97e3d95640 1 -- 172.21.15.123:0/2911484307 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f97dc0b5e90 con 0x7f97dc0aff90 2024-09-16T15:49:32.676 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:32.675+0000 7f97e3d95640 1 -- 172.21.15.123:0/2911484307 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f97dc163770 con 0x7f97dc0aff90 2024-09-16T15:49:32.677 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:32.675+0000 7f97d27fc640 1 -- 172.21.15.123:0/2911484307 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 19) ==== 97488+0+0 (secure 0 0 0) 0x7f97d401a070 con 0x7f97dc0aff90 2024-09-16T15:49:32.677 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:32.675+0000 7f97d27fc640 1 --2- 172.21.15.123:0/2911484307 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f97c0073340 0x7f97c0075800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:49:32.677 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:32.675+0000 7f97d27fc640 1 -- 172.21.15.123:0/2911484307 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(119..119 src has 1..119) ==== 5737+0+0 (secure 0 0 0) 0x7f97d4094ce0 con 0x7f97dc0aff90 2024-09-16T15:49:32.677 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:32.675+0000 7f97e1330640 1 --2- 172.21.15.123:0/2911484307 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f97c0073340 0x7f97c0075800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:49:32.677 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:32.675+0000 7f97e1330640 1 --2- 172.21.15.123:0/2911484307 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f97c0073340 0x7f97c0075800 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f97cc005fd0 tx=0x7f97cc005e60 comp rx=0 tx=0).ready entity=mgr.4150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:49:32.679 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:32.679+0000 7f97d27fc640 1 -- 172.21.15.123:0/2911484307 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f97d405f7d0 con 0x7f97dc0aff90 2024-09-16T15:49:32.838 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:32.839+0000 7f97e3d95640 1 -- 172.21.15.123:0/2911484307 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) -- 0x7f97dc0b1c40 con 0x7f97dc0aff90 2024-09-16T15:49:32.839 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:32.839+0000 7f97d27fc640 1 -- 172.21.15.123:0/2911484307 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v40)=0 v40) ==== 122+0+0 (secure 0 0 0) 0x7f97d405f0d0 con 0x7f97dc0aff90 2024-09-16T15:49:32.842 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:32.843+0000 7f97e3d95640 1 -- 172.21.15.123:0/2911484307 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f97c0073340 msgr2=0x7f97c0075800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:32.842 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:32.843+0000 7f97e3d95640 1 --2- 172.21.15.123:0/2911484307 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f97c0073340 0x7f97c0075800 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f97cc005fd0 tx=0x7f97cc005e60 comp rx=0 tx=0).stop 2024-09-16T15:49:32.842 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:32.843+0000 7f97e3d95640 1 -- 172.21.15.123:0/2911484307 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f97dc0aff90 msgr2=0x7f97dc163ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:32.842 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:32.843+0000 7f97e3d95640 1 --2- 172.21.15.123:0/2911484307 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f97dc0aff90 0x7f97dc163ea0 secure :-1 s=READY pgs=1292 cs=0 l=1 rev1=1 crypto rx=0x7f97d400d950 tx=0x7f97d400de10 comp rx=0 tx=0).stop 2024-09-16T15:49:32.842 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:32.843+0000 7f97e3d95640 1 -- 172.21.15.123:0/2911484307 shutdown_connections 2024-09-16T15:49:32.842 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:32.843+0000 7f97e3d95640 1 --2- 172.21.15.123:0/2911484307 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f97c0073340 0x7f97c0075800 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:32.842 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:32.843+0000 7f97e3d95640 1 --2- 172.21.15.123:0/2911484307 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f97dc15adc0 0x7f97dc154240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:32.842 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:32.843+0000 7f97e3d95640 1 --2- 172.21.15.123:0/2911484307 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f97dc0cc340 0x7f97dc162d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:32.842 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:32.843+0000 7f97e3d95640 1 --2- 172.21.15.123:0/2911484307 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f97dc0aff90 0x7f97dc163ea0 unknown :-1 s=CLOSED pgs=1292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:32.842 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:32.843+0000 7f97e3d95640 1 -- 172.21.15.123:0/2911484307 >> 172.21.15.123:0/2911484307 conn(0x7f97dc0741a0 msgr2=0x7f97dc0c93d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:49:32.842 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:32.843+0000 7f97e3d95640 1 -- 172.21.15.123:0/2911484307 shutdown_connections 2024-09-16T15:49:32.843 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:32.843+0000 7f97e3d95640 1 -- 172.21.15.123:0/2911484307 wait complete. 2024-09-16T15:49:32.858 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:49:32.858 DEBUG:teuthology.orchestra.run.smithi123:> (cd /home/ubuntu/cephtest/mnt.0 && exec sudo mkdir -p volumes/_nogroup/subvol95170/a90a3f46-4eb8-4b4f-b475-d93b6556fce7) 2024-09-16T15:49:32.881 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:49:32.881 DEBUG:teuthology.orchestra.run.smithi123:> (cd /home/ubuntu/cephtest/mnt.0 && exec sudo mkdir -p volumes/_nogroup/subvol95170/a90a3f46-4eb8-4b4f-b475-d93b6556fce7/.snap/fake) 2024-09-16T15:49:32.946 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:49:32.946 DEBUG:teuthology.orchestra.run.smithi123:> (cd /home/ubuntu/cephtest/mnt.0 && exec getfattr --only-values -n ceph.dir.layout.pool .) 2024-09-16T15:49:32.995 INFO:teuthology.orchestra.run.smithi123.stdout:cephfs_data 2024-09-16T15:49:32.995 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:49:32.995 DEBUG:teuthology.orchestra.run.smithi123:> (cd /home/ubuntu/cephtest/mnt.0 && exec sudo setfattr -n ceph.dir.layout.pool -v cephfs_data volumes/_nogroup/subvol95170/a90a3f46-4eb8-4b4f-b475-d93b6556fce7) 2024-09-16T15:49:33.054 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2024-09-16T15:49:33.054 DEBUG:teuthology.orchestra.run.smithi123:> sudo dd of=/home/ubuntu/cephtest/mnt.0/volumes/_nogroup/subvol95170/.meta 2024-09-16T15:49:33.119 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:49:33.119 DEBUG:teuthology.orchestra.run.smithi123:> (cd /home/ubuntu/cephtest/mnt.0 && exec sudo mkdir -p volumes/subvol_grp86620/subvol97901/70aef543-253e-4adb-84ad-93162932d177) 2024-09-16T15:49:33.177 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:49:33.177 DEBUG:teuthology.orchestra.run.smithi123:> (cd /home/ubuntu/cephtest/mnt.0 && exec sudo mkdir -p volumes/subvol_grp86620/subvol97901/70aef543-253e-4adb-84ad-93162932d177/.snap/fake) 2024-09-16T15:49:33.245 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:49:33.245 DEBUG:teuthology.orchestra.run.smithi123:> (cd /home/ubuntu/cephtest/mnt.0 && exec getfattr --only-values -n ceph.dir.layout.pool .) 2024-09-16T15:49:33.295 INFO:teuthology.orchestra.run.smithi123.stdout:cephfs_data 2024-09-16T15:49:33.295 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:49:33.295 DEBUG:teuthology.orchestra.run.smithi123:> (cd /home/ubuntu/cephtest/mnt.0 && exec sudo setfattr -n ceph.dir.layout.pool -v cephfs_data volumes/subvol_grp86620/subvol97901/70aef543-253e-4adb-84ad-93162932d177) 2024-09-16T15:49:33.354 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2024-09-16T15:49:33.354 DEBUG:teuthology.orchestra.run.smithi123:> sudo dd of=/home/ubuntu/cephtest/mnt.0/volumes/subvol_grp86620/subvol97901/.meta 2024-09-16T15:49:33.410 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:49:33.410 DEBUG:teuthology.orchestra.run.smithi123:> (cd /home/ubuntu/cephtest/mnt.0 && exec sudo mkdir -p volumes/_nogroup/subvol4312/5bbaeaac-ff1e-47e6-b3e5-d523916453aa) 2024-09-16T15:49:33.468 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:49:33.468 DEBUG:teuthology.orchestra.run.smithi123:> (cd /home/ubuntu/cephtest/mnt.0 && exec getfattr --only-values -n ceph.dir.layout.pool .) 2024-09-16T15:49:33.519 INFO:teuthology.orchestra.run.smithi123.stdout:cephfs_data 2024-09-16T15:49:33.519 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:49:33.520 DEBUG:teuthology.orchestra.run.smithi123:> (cd /home/ubuntu/cephtest/mnt.0 && exec sudo setfattr -n ceph.dir.layout.pool -v cephfs_data volumes/_nogroup/subvol4312/5bbaeaac-ff1e-47e6-b3e5-d523916453aa) 2024-09-16T15:49:33.580 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2024-09-16T15:49:33.580 DEBUG:teuthology.orchestra.run.smithi123:> sudo dd of=/home/ubuntu/cephtest/mnt.0/volumes/_nogroup/subvol4312/.meta 2024-09-16T15:49:33.640 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs subvol95170 2024-09-16T15:49:33.823 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:33.823+0000 7f4a0ad98640 1 -- 172.21.15.123:0/1875149454 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4a041615c0 msgr2=0x7f4a040d0850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:33.823 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:33.823+0000 7f4a0ad98640 1 --2- 172.21.15.123:0/1875149454 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4a041615c0 0x7f4a040d0850 secure :-1 s=READY pgs=1293 cs=0 l=1 rev1=1 crypto rx=0x7f4a040ce420 tx=0x7f49f401c830 comp rx=0 tx=0).stop 2024-09-16T15:49:33.823 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:33.823+0000 7f4a0ad98640 1 -- 172.21.15.123:0/1875149454 shutdown_connections 2024-09-16T15:49:33.823 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:33.823+0000 7f4a0ad98640 1 --2- 172.21.15.123:0/1875149454 >> v2:172.21.15.184:3300/0 conn(0x7f4a040d0d90 0x7f4a040b3eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:33.823 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:33.823+0000 7f4a0ad98640 1 --2- 172.21.15.123:0/1875149454 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4a041615c0 0x7f4a040d0850 unknown :-1 s=CLOSED pgs=1293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:33.824 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:33.823+0000 7f4a0ad98640 1 -- 172.21.15.123:0/1875149454 >> 172.21.15.123:0/1875149454 conn(0x7f4a040d3e40 msgr2=0x7f4a040d4220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:49:33.824 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:33.823+0000 7f4a0ad98640 1 -- 172.21.15.123:0/1875149454 shutdown_connections 2024-09-16T15:49:33.824 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:33.823+0000 7f4a0ad98640 1 -- 172.21.15.123:0/1875149454 wait complete. 2024-09-16T15:49:33.824 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:33.823+0000 7f4a0ad98640 1 Processor -- start 2024-09-16T15:49:33.824 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:33.823+0000 7f4a0ad98640 1 -- start start 2024-09-16T15:49:33.827 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:33.827+0000 7f4a0ad98640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4a040d0d90 0x7f4a041fb880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:49:33.827 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:33.827+0000 7f4a0ad98640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f4a040d2120 0x7f4a04154e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:49:33.827 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:33.827+0000 7f4a0ad98640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4a041615c0 0x7f4a041f83c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:49:33.827 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:33.827+0000 7f4a0ad98640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f4a040b43f0 con 0x7f4a041615c0 2024-09-16T15:49:33.827 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:33.827+0000 7f4a0ad98640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f4a040b3120 con 0x7f4a040d0d90 2024-09-16T15:49:33.827 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:33.827+0000 7f4a0ad98640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f4a040b4570 con 0x7f4a040d2120 2024-09-16T15:49:33.827 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:33.827+0000 7f4a09335640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4a041615c0 0x7f4a041f83c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:49:33.827 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:33.827+0000 7f4a09335640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4a041615c0 0x7f4a041f83c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:56516/0 (socket says 172.21.15.123:56516) 2024-09-16T15:49:33.827 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:33.827+0000 7f4a09335640 1 -- 172.21.15.123:0/1664377414 learned_addr learned my addr 172.21.15.123:0/1664377414 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:49:33.827 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:33.827+0000 7f4a03fff640 1 --2- 172.21.15.123:0/1664377414 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f4a040d2120 0x7f4a04154e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:49:33.827 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:33.827+0000 7f4a08b34640 1 --2- 172.21.15.123:0/1664377414 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4a040d0d90 0x7f4a041fb880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:49:33.828 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:33.827+0000 7f4a09335640 1 -- 172.21.15.123:0/1664377414 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f4a040d2120 msgr2=0x7f4a04154e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:33.828 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:33.827+0000 7f4a09335640 1 --2- 172.21.15.123:0/1664377414 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f4a040d2120 0x7f4a04154e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:33.828 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:33.827+0000 7f4a09335640 1 -- 172.21.15.123:0/1664377414 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4a040d0d90 msgr2=0x7f4a041fb880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:33.828 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:33.827+0000 7f4a09335640 1 --2- 172.21.15.123:0/1664377414 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4a040d0d90 0x7f4a041fb880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:33.828 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:33.827+0000 7f4a09335640 1 -- 172.21.15.123:0/1664377414 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4a040ccc20 con 0x7f4a041615c0 2024-09-16T15:49:33.828 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:33.827+0000 7f4a08b34640 1 --2- 172.21.15.123:0/1664377414 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4a040d0d90 0x7f4a041fb880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T15:49:33.828 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:33.827+0000 7f4a09335640 1 --2- 172.21.15.123:0/1664377414 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4a041615c0 0x7f4a041f83c0 secure :-1 s=READY pgs=1294 cs=0 l=1 rev1=1 crypto rx=0x7f49fc00ea20 tx=0x7f49fc00eee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:49:33.828 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:33.827+0000 7f4a017fa640 1 -- 172.21.15.123:0/1664377414 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49fc00cd80 con 0x7f4a041615c0 2024-09-16T15:49:33.828 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:33.827+0000 7f4a017fa640 1 -- 172.21.15.123:0/1664377414 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f49fc004c40 con 0x7f4a041615c0 2024-09-16T15:49:33.828 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:33.827+0000 7f4a0ad98640 1 -- 172.21.15.123:0/1664377414 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4a040ccf00 con 0x7f4a041615c0 2024-09-16T15:49:33.828 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:33.827+0000 7f4a017fa640 1 -- 172.21.15.123:0/1664377414 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49fc005280 con 0x7f4a041615c0 2024-09-16T15:49:33.829 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:33.827+0000 7f4a0ad98640 1 -- 172.21.15.123:0/1664377414 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4a041f78d0 con 0x7f4a041615c0 2024-09-16T15:49:33.829 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:33.827+0000 7f4a0ad98640 1 -- 172.21.15.123:0/1664377414 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4a040d0850 con 0x7f4a041615c0 2024-09-16T15:49:33.830 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:33.827+0000 7f4a017fa640 1 -- 172.21.15.123:0/1664377414 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 19) ==== 97488+0+0 (secure 0 0 0) 0x7f49fc005420 con 0x7f4a041615c0 2024-09-16T15:49:33.830 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:33.827+0000 7f4a017fa640 1 --2- 172.21.15.123:0/1664377414 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f49f0073370 0x7f49f0075830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:49:33.830 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:33.831+0000 7f4a08b34640 1 --2- 172.21.15.123:0/1664377414 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f49f0073370 0x7f49f0075830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:49:33.830 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:33.831+0000 7f4a017fa640 1 -- 172.21.15.123:0/1664377414 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(119..119 src has 1..119) ==== 5737+0+0 (secure 0 0 0) 0x7f49fc0950a0 con 0x7f4a041615c0 2024-09-16T15:49:33.830 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:33.831+0000 7f4a08b34640 1 --2- 172.21.15.123:0/1664377414 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f49f0073370 0x7f49f0075830 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f49f40af000 tx=0x7f49f401b070 comp rx=0 tx=0).ready entity=mgr.4150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:49:33.832 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:33.835+0000 7f4a017fa640 1 -- 172.21.15.123:0/1664377414 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f49fc016030 con 0x7f4a041615c0 2024-09-16T15:49:34.031 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:34.031+0000 7f4a0ad98640 1 -- 172.21.15.123:0/1664377414 --> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "subvol95170", "target": ["mon-mgr", ""]}) -- 0x7f4a041fbdc0 con 0x7f49f0073370 2024-09-16T15:49:34.040 INFO:tasks.ceph.mgr.y.smithi123.stderr:2024-09-16T15:49:34.039+0000 7fa9b36f4640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-16T15:49:34.040 INFO:tasks.ceph.mgr.y.smithi123.stderr:2024-09-16T15:49:34.039+0000 7fa9b36f4640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-16T15:49:34.041 INFO:tasks.ceph.mgr.y.smithi123.stderr:2024-09-16T15:49:34.039+0000 7fa9b36f4640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-16T15:49:34.041 INFO:tasks.ceph.mgr.y.smithi123.stderr:2024-09-16T15:49:34.039+0000 7fa9b36f4640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-16T15:49:34.041 INFO:tasks.ceph.mgr.y.smithi123.stderr:2024-09-16T15:49:34.039+0000 7fa9b36f4640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-16T15:49:34.076 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:34.075+0000 7f4a017fa640 1 -- 172.21.15.123:0/1664377414 <== mgr.4150 v2:172.21.15.123:6836/2464759676 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+67 (secure 0 0 0) 0x7f49f4004220 con 0x7f49f0073370 2024-09-16T15:49:34.076 INFO:teuthology.orchestra.run.smithi123.stdout:/volumes/_nogroup/subvol95170/a90a3f46-4eb8-4b4f-b475-d93b6556fce7 2024-09-16T15:49:34.078 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:34.079+0000 7f4a0ad98640 1 -- 172.21.15.123:0/1664377414 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f49f0073370 msgr2=0x7f49f0075830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:34.079 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:34.079+0000 7f4a0ad98640 1 --2- 172.21.15.123:0/1664377414 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f49f0073370 0x7f49f0075830 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f49f40af000 tx=0x7f49f401b070 comp rx=0 tx=0).stop 2024-09-16T15:49:34.079 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:34.079+0000 7f4a0ad98640 1 -- 172.21.15.123:0/1664377414 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4a041615c0 msgr2=0x7f4a041f83c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:34.079 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:34.079+0000 7f4a0ad98640 1 --2- 172.21.15.123:0/1664377414 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4a041615c0 0x7f4a041f83c0 secure :-1 s=READY pgs=1294 cs=0 l=1 rev1=1 crypto rx=0x7f49fc00ea20 tx=0x7f49fc00eee0 comp rx=0 tx=0).stop 2024-09-16T15:49:34.079 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:34.079+0000 7f4a0ad98640 1 -- 172.21.15.123:0/1664377414 shutdown_connections 2024-09-16T15:49:34.079 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:34.079+0000 7f4a0ad98640 1 --2- 172.21.15.123:0/1664377414 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f49f0073370 0x7f49f0075830 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:34.079 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:34.079+0000 7f4a0ad98640 1 --2- 172.21.15.123:0/1664377414 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4a041615c0 0x7f4a041f83c0 unknown :-1 s=CLOSED pgs=1294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:34.079 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:34.079+0000 7f4a0ad98640 1 --2- 172.21.15.123:0/1664377414 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f4a040d2120 0x7f4a04154e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:34.079 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:34.079+0000 7f4a0ad98640 1 --2- 172.21.15.123:0/1664377414 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4a040d0d90 0x7f4a041fb880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:34.079 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:34.079+0000 7f4a0ad98640 1 -- 172.21.15.123:0/1664377414 >> 172.21.15.123:0/1664377414 conn(0x7f4a040d3e40 msgr2=0x7f4a040b1d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:49:34.079 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:34.079+0000 7f4a0ad98640 1 -- 172.21.15.123:0/1664377414 shutdown_connections 2024-09-16T15:49:34.079 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:34.079+0000 7f4a0ad98640 1 -- 172.21.15.123:0/1664377414 wait complete. 2024-09-16T15:49:34.095 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs subvol97901 subvol_grp86620 2024-09-16T15:49:34.215 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:34.215+0000 7fb4d02b5640 1 -- 172.21.15.123:0/2269670375 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb4c815fd80 msgr2=0x7fb4c80cebd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:34.215 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:34.215+0000 7fb4d02b5640 1 --2- 172.21.15.123:0/2269670375 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb4c815fd80 0x7fb4c80cebd0 secure :-1 s=READY pgs=1298 cs=0 l=1 rev1=1 crypto rx=0x7fb4c0009bc0 tx=0x7fb4c001cba0 comp rx=0 tx=0).stop 2024-09-16T15:49:34.216 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:34.215+0000 7fb4d02b5640 1 -- 172.21.15.123:0/2269670375 shutdown_connections 2024-09-16T15:49:34.216 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:34.215+0000 7fb4d02b5640 1 --2- 172.21.15.123:0/2269670375 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb4c815fd80 0x7fb4c80cebd0 unknown :-1 s=CLOSED pgs=1298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:34.216 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:34.215+0000 7fb4d02b5640 1 -- 172.21.15.123:0/2269670375 >> 172.21.15.123:0/2269670375 conn(0x7fb4c80b1630 msgr2=0x7fb4c80b1a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:49:34.216 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:34.215+0000 7fb4d02b5640 1 -- 172.21.15.123:0/2269670375 shutdown_connections 2024-09-16T15:49:34.216 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:34.215+0000 7fb4d02b5640 1 -- 172.21.15.123:0/2269670375 wait complete. 2024-09-16T15:49:34.219 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:34.215+0000 7fb4d02b5640 1 Processor -- start 2024-09-16T15:49:34.219 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:34.215+0000 7fb4d02b5640 1 -- start start 2024-09-16T15:49:34.219 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:34.219+0000 7fb4d02b5640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb4c80725e0 0x7fb4c815aac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:49:34.219 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:34.219+0000 7fb4d02b5640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fb4c8152cf0 0x7fb4c81fb390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:49:34.219 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:34.219+0000 7fb4d02b5640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb4c815fd80 0x7fb4c81fb8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:49:34.219 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:34.219+0000 7fb4d02b5640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fb4c816a030 con 0x7fb4c815fd80 2024-09-16T15:49:34.219 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:34.219+0000 7fb4d02b5640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fb4c8169eb0 con 0x7fb4c80725e0 2024-09-16T15:49:34.219 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:34.219+0000 7fb4d02b5640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fb4c816a1b0 con 0x7fb4c8152cf0 2024-09-16T15:49:34.219 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:34.219+0000 7fb4ce852640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb4c815fd80 0x7fb4c81fb8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:49:34.219 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:34.219+0000 7fb4ce852640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb4c815fd80 0x7fb4c81fb8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:56562/0 (socket says 172.21.15.123:56562) 2024-09-16T15:49:34.220 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:34.219+0000 7fb4ce852640 1 -- 172.21.15.123:0/1990052552 learned_addr learned my addr 172.21.15.123:0/1990052552 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:49:34.220 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:34.219+0000 7fb4ce051640 1 --2- 172.21.15.123:0/1990052552 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb4c80725e0 0x7fb4c815aac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:49:34.220 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:34.219+0000 7fb4cd850640 1 --2- 172.21.15.123:0/1990052552 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fb4c8152cf0 0x7fb4c81fb390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:49:34.220 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:34.219+0000 7fb4ce852640 1 -- 172.21.15.123:0/1990052552 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fb4c8152cf0 msgr2=0x7fb4c81fb390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:34.220 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:34.219+0000 7fb4ce852640 1 --2- 172.21.15.123:0/1990052552 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fb4c8152cf0 0x7fb4c81fb390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:34.220 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:34.219+0000 7fb4ce852640 1 -- 172.21.15.123:0/1990052552 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb4c80725e0 msgr2=0x7fb4c815aac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:34.220 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:34.219+0000 7fb4ce852640 1 --2- 172.21.15.123:0/1990052552 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb4c80725e0 0x7fb4c815aac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:34.220 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:34.219+0000 7fb4ce852640 1 -- 172.21.15.123:0/1990052552 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb4c8148c20 con 0x7fb4c815fd80 2024-09-16T15:49:34.220 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:34.219+0000 7fb4ce852640 1 --2- 172.21.15.123:0/1990052552 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb4c815fd80 0x7fb4c81fb8d0 secure :-1 s=READY pgs=1299 cs=0 l=1 rev1=1 crypto rx=0x7fb4c00b0040 tx=0x7fb4c0005a60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:49:34.220 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:34.219+0000 7fb4beffd640 1 -- 172.21.15.123:0/1990052552 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4c00b2040 con 0x7fb4c815fd80 2024-09-16T15:49:34.220 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:34.219+0000 7fb4beffd640 1 -- 172.21.15.123:0/1990052552 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fb4c00bb070 con 0x7fb4c815fd80 2024-09-16T15:49:34.220 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:34.219+0000 7fb4beffd640 1 -- 172.21.15.123:0/1990052552 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4c00045b0 con 0x7fb4c815fd80 2024-09-16T15:49:34.221 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:34.219+0000 7fb4d02b5640 1 -- 172.21.15.123:0/1990052552 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb4c8148f00 con 0x7fb4c815fd80 2024-09-16T15:49:34.221 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:34.219+0000 7fb4d02b5640 1 -- 172.21.15.123:0/1990052552 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb4c815b000 con 0x7fb4c815fd80 2024-09-16T15:49:34.221 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:34.219+0000 7fb4d02b5640 1 -- 172.21.15.123:0/1990052552 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb4c80cebd0 con 0x7fb4c815fd80 2024-09-16T15:49:34.221 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:34.219+0000 7fb4beffd640 1 -- 172.21.15.123:0/1990052552 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 19) ==== 97488+0+0 (secure 0 0 0) 0x7fb4c00a5a40 con 0x7fb4c815fd80 2024-09-16T15:49:34.222 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:34.219+0000 7fb4beffd640 1 --2- 172.21.15.123:0/1990052552 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fb4ac073340 0x7fb4ac075800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:49:34.222 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:34.219+0000 7fb4ce051640 1 --2- 172.21.15.123:0/1990052552 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fb4ac073340 0x7fb4ac075800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:49:34.222 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:34.219+0000 7fb4beffd640 1 -- 172.21.15.123:0/1990052552 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(119..119 src has 1..119) ==== 5737+0+0 (secure 0 0 0) 0x7fb4c012ec50 con 0x7fb4c815fd80 2024-09-16T15:49:34.222 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:34.219+0000 7fb4ce051640 1 --2- 172.21.15.123:0/1990052552 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fb4ac073340 0x7fb4ac075800 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fb4c8145080 tx=0x7fb4b8009040 comp rx=0 tx=0).ready entity=mgr.4150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:49:34.224 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:34.223+0000 7fb4beffd640 1 -- 172.21.15.123:0/1990052552 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7fb4c00a5e20 con 0x7fb4c815fd80 2024-09-16T15:49:34.423 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:34.423+0000 7fb4d02b5640 1 -- 172.21.15.123:0/1990052552 --> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "subvol97901", "group_name": "subvol_grp86620", "target": ["mon-mgr", ""]}) -- 0x7fb4c80741f0 con 0x7fb4ac073340 2024-09-16T15:49:34.469 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:34.467+0000 7fb4beffd640 1 -- 172.21.15.123:0/1990052552 <== mgr.4150 v2:172.21.15.123:6836/2464759676 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+74 (secure 0 0 0) 0x7fb4c80741f0 con 0x7fb4ac073340 2024-09-16T15:49:34.469 INFO:teuthology.orchestra.run.smithi123.stdout:/volumes/subvol_grp86620/subvol97901/70aef543-253e-4adb-84ad-93162932d177 2024-09-16T15:49:34.472 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:34.471+0000 7fb4d02b5640 1 -- 172.21.15.123:0/1990052552 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fb4ac073340 msgr2=0x7fb4ac075800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:34.472 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:34.471+0000 7fb4d02b5640 1 --2- 172.21.15.123:0/1990052552 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fb4ac073340 0x7fb4ac075800 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fb4c8145080 tx=0x7fb4b8009040 comp rx=0 tx=0).stop 2024-09-16T15:49:34.472 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:34.471+0000 7fb4d02b5640 1 -- 172.21.15.123:0/1990052552 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb4c815fd80 msgr2=0x7fb4c81fb8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:34.472 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:34.471+0000 7fb4d02b5640 1 --2- 172.21.15.123:0/1990052552 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb4c815fd80 0x7fb4c81fb8d0 secure :-1 s=READY pgs=1299 cs=0 l=1 rev1=1 crypto rx=0x7fb4c00b0040 tx=0x7fb4c0005a60 comp rx=0 tx=0).stop 2024-09-16T15:49:34.472 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:34.471+0000 7fb4d02b5640 1 -- 172.21.15.123:0/1990052552 shutdown_connections 2024-09-16T15:49:34.473 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:34.471+0000 7fb4d02b5640 1 --2- 172.21.15.123:0/1990052552 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fb4ac073340 0x7fb4ac075800 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:34.473 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:34.471+0000 7fb4d02b5640 1 --2- 172.21.15.123:0/1990052552 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb4c815fd80 0x7fb4c81fb8d0 unknown :-1 s=CLOSED pgs=1299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:34.473 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:34.471+0000 7fb4d02b5640 1 --2- 172.21.15.123:0/1990052552 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fb4c8152cf0 0x7fb4c81fb390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:34.473 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:34.471+0000 7fb4d02b5640 1 --2- 172.21.15.123:0/1990052552 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb4c80725e0 0x7fb4c815aac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:34.473 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:34.471+0000 7fb4d02b5640 1 -- 172.21.15.123:0/1990052552 >> 172.21.15.123:0/1990052552 conn(0x7fb4c80b1630 msgr2=0x7fb4c814bdb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:49:34.473 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:34.471+0000 7fb4d02b5640 1 -- 172.21.15.123:0/1990052552 shutdown_connections 2024-09-16T15:49:34.473 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:34.471+0000 7fb4d02b5640 1 -- 172.21.15.123:0/1990052552 wait complete. 2024-09-16T15:49:34.491 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol4312 2024-09-16T15:49:34.625 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:34.627+0000 7f70b9315640 1 -- 172.21.15.123:0/2048901199 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f70b415a9d0 msgr2=0x7f70b406ebe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:34.626 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:34.627+0000 7f70b9315640 1 --2- 172.21.15.123:0/2048901199 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f70b415a9d0 0x7f70b406ebe0 secure :-1 s=READY pgs=1301 cs=0 l=1 rev1=1 crypto rx=0x7f70b40af630 tx=0x7f70a401c740 comp rx=0 tx=0).stop 2024-09-16T15:49:34.626 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:34.627+0000 7f70b9315640 1 -- 172.21.15.123:0/2048901199 shutdown_connections 2024-09-16T15:49:34.626 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:34.627+0000 7f70b9315640 1 --2- 172.21.15.123:0/2048901199 >> v2:172.21.15.184:3300/0 conn(0x7f70b40ccad0 0x7f70b415e9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:34.626 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:34.627+0000 7f70b9315640 1 --2- 172.21.15.123:0/2048901199 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f70b415a9d0 0x7f70b406ebe0 unknown :-1 s=CLOSED pgs=1301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:34.626 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:34.627+0000 7f70b9315640 1 -- 172.21.15.123:0/2048901199 >> 172.21.15.123:0/2048901199 conn(0x7f70b40b1710 msgr2=0x7f70b40b1b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:49:34.626 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:34.627+0000 7f70b9315640 1 -- 172.21.15.123:0/2048901199 shutdown_connections 2024-09-16T15:49:34.626 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:34.627+0000 7f70b9315640 1 -- 172.21.15.123:0/2048901199 wait complete. 2024-09-16T15:49:34.627 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:34.627+0000 7f70b9315640 1 Processor -- start 2024-09-16T15:49:34.627 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:34.627+0000 7f70b9315640 1 -- start start 2024-09-16T15:49:34.629 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:34.627+0000 7f70b9315640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f70b4072c70 0x7f70b4156f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:49:34.629 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:34.627+0000 7f70b9315640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f70b40ccad0 0x7f70b40c77b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:49:34.629 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:34.627+0000 7f70b9315640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f70b415a9d0 0x7f70b40b0070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:49:34.629 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:34.627+0000 7f70b9315640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f70b4167ea0 con 0x7f70b4072c70 2024-09-16T15:49:34.629 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:34.627+0000 7f70b9315640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f70b4167d20 con 0x7f70b40ccad0 2024-09-16T15:49:34.629 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:34.627+0000 7f70b9315640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f70b4168020 con 0x7f70b415a9d0 2024-09-16T15:49:34.629 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:34.627+0000 7f70b2ffd640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f70b4072c70 0x7f70b4156f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:49:34.629 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:34.627+0000 7f70b2ffd640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f70b4072c70 0x7f70b4156f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:56586/0 (socket says 172.21.15.123:56586) 2024-09-16T15:49:34.630 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:34.627+0000 7f70b2ffd640 1 -- 172.21.15.123:0/2801449012 learned_addr learned my addr 172.21.15.123:0/2801449012 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:49:34.630 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:34.627+0000 7f70b27fc640 1 --2- 172.21.15.123:0/2801449012 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f70b40ccad0 0x7f70b40c77b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:49:34.630 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:34.627+0000 7f70b37fe640 1 --2- 172.21.15.123:0/2801449012 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f70b415a9d0 0x7f70b40b0070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:49:34.630 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:34.627+0000 7f70b2ffd640 1 -- 172.21.15.123:0/2801449012 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f70b415a9d0 msgr2=0x7f70b40b0070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:34.630 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:34.627+0000 7f70b2ffd640 1 --2- 172.21.15.123:0/2801449012 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f70b415a9d0 0x7f70b40b0070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:34.630 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:34.627+0000 7f70b2ffd640 1 -- 172.21.15.123:0/2801449012 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f70b40ccad0 msgr2=0x7f70b40c77b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:34.630 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:34.627+0000 7f70b2ffd640 1 --2- 172.21.15.123:0/2801449012 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f70b40ccad0 0x7f70b40c77b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:34.630 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:34.627+0000 7f70b2ffd640 1 -- 172.21.15.123:0/2801449012 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f70b40c8090 con 0x7f70b4072c70 2024-09-16T15:49:34.630 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:34.627+0000 7f70b2ffd640 1 --2- 172.21.15.123:0/2801449012 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f70b4072c70 0x7f70b4156f10 secure :-1 s=READY pgs=1302 cs=0 l=1 rev1=1 crypto rx=0x7f70a4000c10 tx=0x7f70a40a56c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:49:34.630 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:34.627+0000 7f709b7fe640 1 -- 172.21.15.123:0/2801449012 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70a40bc070 con 0x7f70b4072c70 2024-09-16T15:49:34.630 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:34.627+0000 7f709b7fe640 1 -- 172.21.15.123:0/2801449012 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f70a40044b0 con 0x7f70b4072c70 2024-09-16T15:49:34.631 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:34.627+0000 7f709b7fe640 1 -- 172.21.15.123:0/2801449012 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70a40b2040 con 0x7f70b4072c70 2024-09-16T15:49:34.631 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:34.627+0000 7f70b9315640 1 -- 172.21.15.123:0/2801449012 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f70b41f7dc0 con 0x7f70b4072c70 2024-09-16T15:49:34.631 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:34.627+0000 7f70b9315640 1 -- 172.21.15.123:0/2801449012 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f70b4157450 con 0x7f70b4072c70 2024-09-16T15:49:34.631 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:34.631+0000 7f70b9315640 1 -- 172.21.15.123:0/2801449012 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f70b415e9c0 con 0x7f70b4072c70 2024-09-16T15:49:34.631 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:34.631+0000 7f709b7fe640 1 -- 172.21.15.123:0/2801449012 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 19) ==== 97488+0+0 (secure 0 0 0) 0x7f70a4003840 con 0x7f70b4072c70 2024-09-16T15:49:34.633 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:34.631+0000 7f709b7fe640 1 --2- 172.21.15.123:0/2801449012 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f70a0073340 0x7f70a0075800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:49:34.633 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:34.631+0000 7f709b7fe640 1 -- 172.21.15.123:0/2801449012 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(119..119 src has 1..119) ==== 5737+0+0 (secure 0 0 0) 0x7f70a412e0e0 con 0x7f70b4072c70 2024-09-16T15:49:34.633 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:34.631+0000 7f70b27fc640 1 --2- 172.21.15.123:0/2801449012 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f70a0073340 0x7f70a0075800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:49:34.633 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:34.631+0000 7f70b27fc640 1 --2- 172.21.15.123:0/2801449012 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f70a0073340 0x7f70a0075800 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f70b41659b0 tx=0x7f70a8009040 comp rx=0 tx=0).ready entity=mgr.4150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:49:34.635 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:34.635+0000 7f709b7fe640 1 -- 172.21.15.123:0/2801449012 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f70a40163f0 con 0x7f70b4072c70 2024-09-16T15:49:34.844 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:34.843+0000 7f70b9315640 1 -- 172.21.15.123:0/2801449012 --> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvol4312", "target": ["mon-mgr", ""]}) -- 0x7f70b41f7ba0 con 0x7f70a0073340 2024-09-16T15:49:34.873 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:34.875+0000 7f709b7fe640 1 -- 172.21.15.123:0/2801449012 <== mgr.4150 v2:172.21.15.123:6836/2464759676 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+144 (secure 0 0 0) 0x7f70b41f7ba0 con 0x7f70a0073340 2024-09-16T15:49:34.874 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2024-09-16T15:49:34.874 INFO:teuthology.orchestra.run.smithi123.stdout: "status": { 2024-09-16T15:49:34.874 INFO:teuthology.orchestra.run.smithi123.stdout: "state": "pending", 2024-09-16T15:49:34.874 INFO:teuthology.orchestra.run.smithi123.stdout: "source": { 2024-09-16T15:49:34.874 INFO:teuthology.orchestra.run.smithi123.stdout: "volume": "fake", 2024-09-16T15:49:34.874 INFO:teuthology.orchestra.run.smithi123.stdout: "subvolume": "fake", 2024-09-16T15:49:34.874 INFO:teuthology.orchestra.run.smithi123.stdout: "snapshot": "fake" 2024-09-16T15:49:34.874 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-16T15:49:34.874 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-16T15:49:34.874 INFO:teuthology.orchestra.run.smithi123.stdout:} 2024-09-16T15:49:34.876 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:34.875+0000 7f70b9315640 1 -- 172.21.15.123:0/2801449012 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f70a0073340 msgr2=0x7f70a0075800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:34.876 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:34.875+0000 7f70b9315640 1 --2- 172.21.15.123:0/2801449012 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f70a0073340 0x7f70a0075800 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f70b41659b0 tx=0x7f70a8009040 comp rx=0 tx=0).stop 2024-09-16T15:49:34.876 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:34.875+0000 7f70b9315640 1 -- 172.21.15.123:0/2801449012 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f70b4072c70 msgr2=0x7f70b4156f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:34.876 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:34.875+0000 7f70b9315640 1 --2- 172.21.15.123:0/2801449012 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f70b4072c70 0x7f70b4156f10 secure :-1 s=READY pgs=1302 cs=0 l=1 rev1=1 crypto rx=0x7f70a4000c10 tx=0x7f70a40a56c0 comp rx=0 tx=0).stop 2024-09-16T15:49:34.876 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:34.875+0000 7f70b9315640 1 -- 172.21.15.123:0/2801449012 shutdown_connections 2024-09-16T15:49:34.876 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:34.875+0000 7f70b9315640 1 --2- 172.21.15.123:0/2801449012 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f70a0073340 0x7f70a0075800 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:34.877 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:34.875+0000 7f70b9315640 1 --2- 172.21.15.123:0/2801449012 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f70b415a9d0 0x7f70b40b0070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:34.877 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:34.875+0000 7f70b9315640 1 --2- 172.21.15.123:0/2801449012 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f70b40ccad0 0x7f70b40c77b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:34.877 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:34.875+0000 7f70b9315640 1 --2- 172.21.15.123:0/2801449012 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f70b4072c70 0x7f70b4156f10 unknown :-1 s=CLOSED pgs=1302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:34.877 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:34.875+0000 7f70b9315640 1 -- 172.21.15.123:0/2801449012 >> 172.21.15.123:0/2801449012 conn(0x7f70b40b1710 msgr2=0x7f70b4159220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:49:34.877 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:34.875+0000 7f70b9315640 1 -- 172.21.15.123:0/2801449012 shutdown_connections 2024-09-16T15:49:34.877 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:34.875+0000 7f70b9315640 1 -- 172.21.15.123:0/2801449012 wait complete. 2024-09-16T15:49:34.892 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot rm cephfs subvol95170 fake 2024-09-16T15:49:35.025 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:35.027+0000 7f0d893ed640 1 -- 172.21.15.123:0/3110732249 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f0d8406e590 msgr2=0x7f0d8406e970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:35.025 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:35.027+0000 7f0d893ed640 1 --2- 172.21.15.123:0/3110732249 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f0d8406e590 0x7f0d8406e970 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f0d74009840 tx=0x7f0d7401c7f0 comp rx=0 tx=0).stop 2024-09-16T15:49:35.025 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:35.027+0000 7f0d893ed640 1 -- 172.21.15.123:0/3110732249 shutdown_connections 2024-09-16T15:49:35.025 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:35.027+0000 7f0d893ed640 1 --2- 172.21.15.123:0/3110732249 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f0d840ce9e0 0x7f0d8406eeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:35.025 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:35.027+0000 7f0d893ed640 1 --2- 172.21.15.123:0/3110732249 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f0d8406e590 0x7f0d8406e970 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:35.026 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:35.027+0000 7f0d893ed640 1 -- 172.21.15.123:0/3110732249 >> 172.21.15.123:0/3110732249 conn(0x7f0d840b1630 msgr2=0x7f0d840b1a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:49:35.026 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:35.027+0000 7f0d893ed640 1 -- 172.21.15.123:0/3110732249 shutdown_connections 2024-09-16T15:49:35.026 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:35.027+0000 7f0d893ed640 1 -- 172.21.15.123:0/3110732249 wait complete. 2024-09-16T15:49:35.026 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:35.027+0000 7f0d893ed640 1 Processor -- start 2024-09-16T15:49:35.026 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:35.027+0000 7f0d893ed640 1 -- start start 2024-09-16T15:49:35.028 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:35.027+0000 7f0d893ed640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f0d8406e590 0x7f0d8414e010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:49:35.028 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:35.027+0000 7f0d893ed640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f0d84070400 0x7f0d8415ec20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:49:35.029 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:35.027+0000 7f0d893ed640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0d840ce9e0 0x7f0d841f8bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:49:35.029 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:35.027+0000 7f0d893ed640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f0d84168880 con 0x7f0d840ce9e0 2024-09-16T15:49:35.029 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:35.027+0000 7f0d893ed640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f0d84168700 con 0x7f0d8406e590 2024-09-16T15:49:35.029 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:35.027+0000 7f0d893ed640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f0d84168a00 con 0x7f0d84070400 2024-09-16T15:49:35.029 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:35.027+0000 7f0d837fe640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0d840ce9e0 0x7f0d841f8bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:49:35.029 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:35.027+0000 7f0d837fe640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0d840ce9e0 0x7f0d841f8bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:56602/0 (socket says 172.21.15.123:56602) 2024-09-16T15:49:35.029 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:35.027+0000 7f0d837fe640 1 -- 172.21.15.123:0/4249629785 learned_addr learned my addr 172.21.15.123:0/4249629785 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:49:35.029 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:35.027+0000 7f0d82ffd640 1 --2- 172.21.15.123:0/4249629785 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f0d8406e590 0x7f0d8414e010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:49:35.029 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:35.027+0000 7f0d827fc640 1 --2- 172.21.15.123:0/4249629785 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f0d84070400 0x7f0d8415ec20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:49:35.029 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:35.027+0000 7f0d837fe640 1 -- 172.21.15.123:0/4249629785 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f0d84070400 msgr2=0x7f0d8415ec20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:35.029 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:35.027+0000 7f0d837fe640 1 --2- 172.21.15.123:0/4249629785 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f0d84070400 0x7f0d8415ec20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:35.029 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:35.027+0000 7f0d837fe640 1 -- 172.21.15.123:0/4249629785 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f0d8406e590 msgr2=0x7f0d8414e010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:35.029 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:35.027+0000 7f0d837fe640 1 --2- 172.21.15.123:0/4249629785 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f0d8406e590 0x7f0d8414e010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:35.030 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:35.027+0000 7f0d837fe640 1 -- 172.21.15.123:0/4249629785 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0d8414f160 con 0x7f0d840ce9e0 2024-09-16T15:49:35.030 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:35.027+0000 7f0d837fe640 1 --2- 172.21.15.123:0/4249629785 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0d840ce9e0 0x7f0d841f8bf0 secure :-1 s=READY pgs=1303 cs=0 l=1 rev1=1 crypto rx=0x7f0d7c00cbc0 tx=0x7f0d7c00ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:49:35.030 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:35.027+0000 7f0d7b7fe640 1 -- 172.21.15.123:0/4249629785 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d7c016020 con 0x7f0d840ce9e0 2024-09-16T15:49:35.030 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:35.027+0000 7f0d7b7fe640 1 -- 172.21.15.123:0/4249629785 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f0d7c00ab40 con 0x7f0d840ce9e0 2024-09-16T15:49:35.030 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:35.027+0000 7f0d893ed640 1 -- 172.21.15.123:0/4249629785 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0d8414f3d0 con 0x7f0d840ce9e0 2024-09-16T15:49:35.030 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:35.027+0000 7f0d7b7fe640 1 -- 172.21.15.123:0/4249629785 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d7c012420 con 0x7f0d840ce9e0 2024-09-16T15:49:35.030 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:35.027+0000 7f0d893ed640 1 -- 172.21.15.123:0/4249629785 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0d84140540 con 0x7f0d840ce9e0 2024-09-16T15:49:35.031 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:35.031+0000 7f0d893ed640 1 -- 172.21.15.123:0/4249629785 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0d84072e40 con 0x7f0d840ce9e0 2024-09-16T15:49:35.031 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:35.031+0000 7f0d7b7fe640 1 -- 172.21.15.123:0/4249629785 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 19) ==== 97488+0+0 (secure 0 0 0) 0x7f0d7c019020 con 0x7f0d840ce9e0 2024-09-16T15:49:35.031 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:35.031+0000 7f0d7b7fe640 1 --2- 172.21.15.123:0/4249629785 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f0d64073340 0x7f0d64075800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:49:35.031 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:35.031+0000 7f0d82ffd640 1 --2- 172.21.15.123:0/4249629785 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f0d64073340 0x7f0d64075800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:49:35.031 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:35.031+0000 7f0d7b7fe640 1 -- 172.21.15.123:0/4249629785 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(119..119 src has 1..119) ==== 5737+0+0 (secure 0 0 0) 0x7f0d7c0945f0 con 0x7f0d840ce9e0 2024-09-16T15:49:35.031 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:35.031+0000 7f0d82ffd640 1 --2- 172.21.15.123:0/4249629785 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f0d64073340 0x7f0d64075800 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f0d740af000 tx=0x7f0d7401b030 comp rx=0 tx=0).ready entity=mgr.4150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:49:35.034 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:35.035+0000 7f0d7b7fe640 1 -- 172.21.15.123:0/4249629785 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f0d7c020030 con 0x7f0d840ce9e0 2024-09-16T15:49:35.246 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:35.247+0000 7f0d893ed640 1 -- 172.21.15.123:0/4249629785 --> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot rm", "vol_name": "cephfs", "sub_name": "subvol95170", "snap_name": "fake", "target": ["mon-mgr", ""]}) -- 0x7f0d8414f8f0 con 0x7f0d64073340 2024-09-16T15:49:35.299 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:35.299+0000 7f0d7b7fe640 1 -- 172.21.15.123:0/4249629785 <== mgr.4150 v2:172.21.15.123:6836/2464759676 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f0d8414f8f0 con 0x7f0d64073340 2024-09-16T15:49:35.302 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:35.303+0000 7f0d893ed640 1 -- 172.21.15.123:0/4249629785 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f0d64073340 msgr2=0x7f0d64075800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:35.303 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:35.303+0000 7f0d893ed640 1 --2- 172.21.15.123:0/4249629785 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f0d64073340 0x7f0d64075800 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f0d740af000 tx=0x7f0d7401b030 comp rx=0 tx=0).stop 2024-09-16T15:49:35.303 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:35.303+0000 7f0d893ed640 1 -- 172.21.15.123:0/4249629785 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0d840ce9e0 msgr2=0x7f0d841f8bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:35.303 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:35.303+0000 7f0d893ed640 1 --2- 172.21.15.123:0/4249629785 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0d840ce9e0 0x7f0d841f8bf0 secure :-1 s=READY pgs=1303 cs=0 l=1 rev1=1 crypto rx=0x7f0d7c00cbc0 tx=0x7f0d7c00ccb0 comp rx=0 tx=0).stop 2024-09-16T15:49:35.303 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:35.303+0000 7f0d893ed640 1 -- 172.21.15.123:0/4249629785 shutdown_connections 2024-09-16T15:49:35.303 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:35.303+0000 7f0d893ed640 1 --2- 172.21.15.123:0/4249629785 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f0d64073340 0x7f0d64075800 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:35.303 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:35.303+0000 7f0d893ed640 1 --2- 172.21.15.123:0/4249629785 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0d840ce9e0 0x7f0d841f8bf0 unknown :-1 s=CLOSED pgs=1303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:35.303 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:35.303+0000 7f0d893ed640 1 --2- 172.21.15.123:0/4249629785 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f0d84070400 0x7f0d8415ec20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:35.303 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:35.303+0000 7f0d893ed640 1 --2- 172.21.15.123:0/4249629785 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f0d8406e590 0x7f0d8414e010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:35.303 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:35.303+0000 7f0d893ed640 1 -- 172.21.15.123:0/4249629785 >> 172.21.15.123:0/4249629785 conn(0x7f0d840b1630 msgr2=0x7f0d8406ec00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:49:35.303 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:35.303+0000 7f0d893ed640 1 -- 172.21.15.123:0/4249629785 shutdown_connections 2024-09-16T15:49:35.303 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:35.303+0000 7f0d893ed640 1 -- 172.21.15.123:0/4249629785 wait complete. 2024-09-16T15:49:35.320 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot rm cephfs subvol97901 fake subvol_grp86620 2024-09-16T15:49:35.450 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:35.451+0000 7fce7d948640 1 -- 172.21.15.123:0/1142524613 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fce60004d50 msgr2=0x7fce600051d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:35.450 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:35.451+0000 7fce7d948640 1 --2- 172.21.15.123:0/1142524613 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fce60004d50 0x7fce600051d0 secure :-1 s=READY pgs=1305 cs=0 l=1 rev1=1 crypto rx=0x7fce780724c0 tx=0x7fce7001d1d0 comp rx=0 tx=0).stop 2024-09-16T15:49:35.450 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:35.451+0000 7fce777fe640 1 -- 172.21.15.123:0/1142524613 reap_dead start 2024-09-16T15:49:35.451 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:35.451+0000 7fce7d948640 1 -- 172.21.15.123:0/1142524613 shutdown_connections 2024-09-16T15:49:35.451 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:35.451+0000 7fce7d948640 1 -- 172.21.15.123:0/1142524613 >> 172.21.15.123:0/1142524613 conn(0x7fce780b1710 msgr2=0x7fce780b1b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:49:35.451 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:35.451+0000 7fce7d948640 1 -- 172.21.15.123:0/1142524613 shutdown_connections 2024-09-16T15:49:35.451 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:35.451+0000 7fce7d948640 1 -- 172.21.15.123:0/1142524613 wait complete. 2024-09-16T15:49:35.451 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:35.451+0000 7fce7d948640 1 Processor -- start 2024-09-16T15:49:35.451 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:35.451+0000 7fce7d948640 1 -- start start 2024-09-16T15:49:35.453 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:35.451+0000 7fce7d948640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fce60004d50 0x7fce78150e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:49:35.453 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:35.451+0000 7fce7d948640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fce780c3550 0x7fce780c39b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:49:35.453 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:35.451+0000 7fce7d948640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fce780c57e0 0x7fce780c5c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:49:35.453 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:35.451+0000 7fce7d948640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fce7806ebe0 con 0x7fce780c57e0 2024-09-16T15:49:35.454 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:35.451+0000 7fce7d948640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fce780729b0 con 0x7fce60004d50 2024-09-16T15:49:35.454 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:35.451+0000 7fce7d948640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fce7806ed60 con 0x7fce780c3550 2024-09-16T15:49:35.454 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:35.451+0000 7fce777fe640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fce780c57e0 0x7fce780c5c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:49:35.454 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:35.451+0000 7fce777fe640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fce780c57e0 0x7fce780c5c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:56620/0 (socket says 172.21.15.123:56620) 2024-09-16T15:49:35.454 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:35.451+0000 7fce777fe640 1 -- 172.21.15.123:0/3839676531 learned_addr learned my addr 172.21.15.123:0/3839676531 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:49:35.454 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:35.455+0000 7fce76ffd640 1 --2- 172.21.15.123:0/3839676531 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fce60004d50 0x7fce78150e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:49:35.454 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:35.455+0000 7fce767fc640 1 --2- 172.21.15.123:0/3839676531 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fce780c3550 0x7fce780c39b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:49:35.454 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:35.455+0000 7fce777fe640 1 -- 172.21.15.123:0/3839676531 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fce780c3550 msgr2=0x7fce780c39b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:35.454 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:35.455+0000 7fce777fe640 1 --2- 172.21.15.123:0/3839676531 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fce780c3550 0x7fce780c39b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:35.454 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:35.455+0000 7fce777fe640 1 -- 172.21.15.123:0/3839676531 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fce60004d50 msgr2=0x7fce78150e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:35.454 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:35.455+0000 7fce777fe640 1 --2- 172.21.15.123:0/3839676531 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fce60004d50 0x7fce78150e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:35.454 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:35.455+0000 7fce777fe640 1 -- 172.21.15.123:0/3839676531 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fce780c61a0 con 0x7fce780c57e0 2024-09-16T15:49:35.454 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:35.455+0000 7fce767fc640 1 --2- 172.21.15.123:0/3839676531 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fce780c3550 0x7fce780c39b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T15:49:35.455 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:35.455+0000 7fce777fe640 1 --2- 172.21.15.123:0/3839676531 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fce780c57e0 0x7fce780c5c60 secure :-1 s=READY pgs=1306 cs=0 l=1 rev1=1 crypto rx=0x7fce70002240 tx=0x7fce700094d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:49:35.455 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:35.455+0000 7fce5ffff640 1 -- 172.21.15.123:0/3839676531 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce70004a50 con 0x7fce780c57e0 2024-09-16T15:49:35.455 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:35.455+0000 7fce5ffff640 1 -- 172.21.15.123:0/3839676531 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fce700ac900 con 0x7fce780c57e0 2024-09-16T15:49:35.455 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:35.455+0000 7fce7d948640 1 -- 172.21.15.123:0/3839676531 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fce781f4ac0 con 0x7fce780c57e0 2024-09-16T15:49:35.455 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:35.455+0000 7fce5ffff640 1 -- 172.21.15.123:0/3839676531 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce700acaf0 con 0x7fce780c57e0 2024-09-16T15:49:35.455 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:35.455+0000 7fce7d948640 1 -- 172.21.15.123:0/3839676531 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fce780c6410 con 0x7fce780c57e0 2024-09-16T15:49:35.456 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:35.455+0000 7fce7d948640 1 -- 172.21.15.123:0/3839676531 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fce780cb360 con 0x7fce780c57e0 2024-09-16T15:49:35.456 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:35.455+0000 7fce5ffff640 1 -- 172.21.15.123:0/3839676531 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 19) ==== 97488+0+0 (secure 0 0 0) 0x7fce700c0030 con 0x7fce780c57e0 2024-09-16T15:49:35.456 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:35.455+0000 7fce5ffff640 1 --2- 172.21.15.123:0/3839676531 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fce64073340 0x7fce64075800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:49:35.456 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:35.455+0000 7fce5ffff640 1 -- 172.21.15.123:0/3839676531 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(120..120 src has 1..120) ==== 5795+0+0 (secure 0 0 0) 0x7fce7012f4d0 con 0x7fce780c57e0 2024-09-16T15:49:35.456 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:35.455+0000 7fce76ffd640 1 --2- 172.21.15.123:0/3839676531 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fce64073340 0x7fce64075800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:49:35.456 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:35.455+0000 7fce76ffd640 1 --2- 172.21.15.123:0/3839676531 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fce64073340 0x7fce64075800 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fce6c0057d0 tx=0x7fce6c009640 comp rx=0 tx=0).ready entity=mgr.4150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:49:35.459 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:35.459+0000 7fce5ffff640 1 -- 172.21.15.123:0/3839676531 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7fce700c7060 con 0x7fce780c57e0 2024-09-16T15:49:35.651 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:35.651+0000 7fce7d948640 1 -- 172.21.15.123:0/3839676531 --> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot rm", "vol_name": "cephfs", "sub_name": "subvol97901", "snap_name": "fake", "group_name": "subvol_grp86620", "target": ["mon-mgr", ""]}) -- 0x7fce780af7b0 con 0x7fce64073340 2024-09-16T15:49:35.698 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:35.699+0000 7fce5ffff640 1 -- 172.21.15.123:0/3839676531 <== mgr.4150 v2:172.21.15.123:6836/2464759676 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7fce780af7b0 con 0x7fce64073340 2024-09-16T15:49:35.701 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:35.699+0000 7fce7d948640 1 -- 172.21.15.123:0/3839676531 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fce64073340 msgr2=0x7fce64075800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:35.701 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:35.699+0000 7fce7d948640 1 --2- 172.21.15.123:0/3839676531 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fce64073340 0x7fce64075800 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fce6c0057d0 tx=0x7fce6c009640 comp rx=0 tx=0).stop 2024-09-16T15:49:35.701 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:35.703+0000 7fce7d948640 1 -- 172.21.15.123:0/3839676531 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fce780c57e0 msgr2=0x7fce780c5c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:35.701 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:35.703+0000 7fce7d948640 1 --2- 172.21.15.123:0/3839676531 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fce780c57e0 0x7fce780c5c60 secure :-1 s=READY pgs=1306 cs=0 l=1 rev1=1 crypto rx=0x7fce70002240 tx=0x7fce700094d0 comp rx=0 tx=0).stop 2024-09-16T15:49:35.701 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:35.703+0000 7fce7d948640 1 -- 172.21.15.123:0/3839676531 shutdown_connections 2024-09-16T15:49:35.701 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:35.703+0000 7fce7d948640 1 --2- 172.21.15.123:0/3839676531 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fce64073340 0x7fce64075800 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:35.702 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:35.703+0000 7fce7d948640 1 --2- 172.21.15.123:0/3839676531 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fce780c57e0 0x7fce780c5c60 unknown :-1 s=CLOSED pgs=1306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:35.702 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:35.703+0000 7fce7d948640 1 --2- 172.21.15.123:0/3839676531 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fce780c3550 0x7fce780c39b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:35.702 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:35.703+0000 7fce7d948640 1 --2- 172.21.15.123:0/3839676531 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fce60004d50 0x7fce78150e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:35.702 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:35.703+0000 7fce7d948640 1 -- 172.21.15.123:0/3839676531 >> 172.21.15.123:0/3839676531 conn(0x7fce780b1710 msgr2=0x7fce7815d0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:49:35.702 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:35.703+0000 7fce7d948640 1 -- 172.21.15.123:0/3839676531 shutdown_connections 2024-09-16T15:49:35.702 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:35.703+0000 7fce7d948640 1 -- 172.21.15.123:0/3839676531 wait complete. 2024-09-16T15:49:35.720 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:49:35.720 DEBUG:teuthology.orchestra.run.smithi123:> (cd /home/ubuntu/cephtest/mnt.0 && exec sudo cat ./volumes/_nogroup/subvol95170/.meta) 2024-09-16T15:49:35.737 INFO:teuthology.orchestra.run.smithi123.stdout:[GLOBAL] 2024-09-16T15:49:35.737 INFO:teuthology.orchestra.run.smithi123.stdout:version = 1 2024-09-16T15:49:35.737 INFO:teuthology.orchestra.run.smithi123.stdout:type = subvolume 2024-09-16T15:49:35.738 INFO:teuthology.orchestra.run.smithi123.stdout:path = /volumes/_nogroup/subvol95170/a90a3f46-4eb8-4b4f-b475-d93b6556fce7 2024-09-16T15:49:35.738 INFO:teuthology.orchestra.run.smithi123.stdout:state = complete 2024-09-16T15:49:35.738 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-09-16T15:49:35.739 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:49:35.739 DEBUG:teuthology.orchestra.run.smithi123:> (cd /home/ubuntu/cephtest/mnt.0 && exec sudo cat ./volumes/subvol_grp86620/subvol97901/.meta) 2024-09-16T15:49:35.797 INFO:teuthology.orchestra.run.smithi123.stdout:[GLOBAL] 2024-09-16T15:49:35.798 INFO:teuthology.orchestra.run.smithi123.stdout:version = 1 2024-09-16T15:49:35.798 INFO:teuthology.orchestra.run.smithi123.stdout:type = subvolume 2024-09-16T15:49:35.798 INFO:teuthology.orchestra.run.smithi123.stdout:path = /volumes/subvol_grp86620/subvol97901/70aef543-253e-4adb-84ad-93162932d177 2024-09-16T15:49:35.798 INFO:teuthology.orchestra.run.smithi123.stdout:state = complete 2024-09-16T15:49:35.798 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-09-16T15:49:35.799 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume rm cephfs subvol95170 2024-09-16T15:49:35.971 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:35.970+0000 7efc71955640 1 -- 172.21.15.123:0/2961346050 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7efc6c0c3230 msgr2=0x7efc6c0cedd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:35.971 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:35.970+0000 7efc71955640 1 --2- 172.21.15.123:0/2961346050 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7efc6c0c3230 0x7efc6c0cedd0 secure :-1 s=READY pgs=1308 cs=0 l=1 rev1=1 crypto rx=0x7efc60009900 tx=0x7efc6001d340 comp rx=0 tx=0).stop 2024-09-16T15:49:35.971 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:35.970+0000 7efc71955640 1 -- 172.21.15.123:0/2961346050 shutdown_connections 2024-09-16T15:49:35.971 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:35.970+0000 7efc71955640 1 --2- 172.21.15.123:0/2961346050 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7efc6c0c3230 0x7efc6c0cedd0 unknown :-1 s=CLOSED pgs=1308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:35.971 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:35.970+0000 7efc71955640 1 -- 172.21.15.123:0/2961346050 >> 172.21.15.123:0/2961346050 conn(0x7efc6c0b1710 msgr2=0x7efc6c0b1b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:49:35.972 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:35.970+0000 7efc71955640 1 -- 172.21.15.123:0/2961346050 shutdown_connections 2024-09-16T15:49:35.972 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:35.970+0000 7efc71955640 1 -- 172.21.15.123:0/2961346050 wait complete. 2024-09-16T15:49:35.972 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:35.970+0000 7efc71955640 1 Processor -- start 2024-09-16T15:49:35.972 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:35.970+0000 7efc71955640 1 -- start start 2024-09-16T15:49:35.974 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:35.974+0000 7efc71955640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7efc6c0c3230 0x7efc6c14d1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:49:35.974 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:35.974+0000 7efc71955640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7efc6c0cf310 0x7efc6c14f6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:49:35.974 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:35.974+0000 7efc71955640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7efc6c15b9f0 0x7efc6c156ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:49:35.974 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:35.974+0000 7efc71955640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7efc6c169cf0 con 0x7efc6c15b9f0 2024-09-16T15:49:35.974 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:35.974+0000 7efc71955640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7efc6c169b70 con 0x7efc6c0cf310 2024-09-16T15:49:35.974 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:35.974+0000 7efc71955640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7efc6c169e70 con 0x7efc6c0c3230 2024-09-16T15:49:35.975 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:35.974+0000 7efc6b7fe640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7efc6c15b9f0 0x7efc6c156ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:49:35.975 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:35.974+0000 7efc6b7fe640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7efc6c15b9f0 0x7efc6c156ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:56624/0 (socket says 172.21.15.123:56624) 2024-09-16T15:49:35.975 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:35.974+0000 7efc6b7fe640 1 -- 172.21.15.123:0/189904098 learned_addr learned my addr 172.21.15.123:0/189904098 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:49:35.975 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:35.974+0000 7efc6affd640 1 --2- 172.21.15.123:0/189904098 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7efc6c0c3230 0x7efc6c14d1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:49:35.975 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:35.974+0000 7efc6a7fc640 1 --2- 172.21.15.123:0/189904098 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7efc6c0cf310 0x7efc6c14f6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:49:35.975 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:35.974+0000 7efc6b7fe640 1 -- 172.21.15.123:0/189904098 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7efc6c0c3230 msgr2=0x7efc6c14d1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:35.975 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:35.974+0000 7efc6b7fe640 1 --2- 172.21.15.123:0/189904098 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7efc6c0c3230 0x7efc6c14d1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:35.975 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:35.974+0000 7efc6b7fe640 1 -- 172.21.15.123:0/189904098 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7efc6c0cf310 msgr2=0x7efc6c14f6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:35.975 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:35.974+0000 7efc6b7fe640 1 --2- 172.21.15.123:0/189904098 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7efc6c0cf310 0x7efc6c14f6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:35.975 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:35.974+0000 7efc6b7fe640 1 -- 172.21.15.123:0/189904098 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efc6c0b3bc0 con 0x7efc6c15b9f0 2024-09-16T15:49:35.975 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:35.974+0000 7efc6a7fc640 1 --2- 172.21.15.123:0/189904098 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7efc6c0cf310 0x7efc6c14f6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T15:49:35.976 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:35.974+0000 7efc6b7fe640 1 --2- 172.21.15.123:0/189904098 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7efc6c15b9f0 0x7efc6c156ac0 secure :-1 s=READY pgs=1309 cs=0 l=1 rev1=1 crypto rx=0x7efc5c00c460 tx=0x7efc5c00c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:49:35.976 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:35.974+0000 7efc53fff640 1 -- 172.21.15.123:0/189904098 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc5c017070 con 0x7efc6c15b9f0 2024-09-16T15:49:35.976 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:35.974+0000 7efc53fff640 1 -- 172.21.15.123:0/189904098 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7efc5c003600 con 0x7efc6c15b9f0 2024-09-16T15:49:35.976 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:35.974+0000 7efc71955640 1 -- 172.21.15.123:0/189904098 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efc6c0b3ea0 con 0x7efc6c15b9f0 2024-09-16T15:49:35.976 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:35.974+0000 7efc53fff640 1 -- 172.21.15.123:0/189904098 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc5c003a50 con 0x7efc6c15b9f0 2024-09-16T15:49:35.976 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:35.974+0000 7efc71955640 1 -- 172.21.15.123:0/189904098 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efc6c14d6e0 con 0x7efc6c15b9f0 2024-09-16T15:49:35.977 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:35.974+0000 7efc71955640 1 -- 172.21.15.123:0/189904098 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efc6c14ed60 con 0x7efc6c15b9f0 2024-09-16T15:49:35.977 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:35.974+0000 7efc53fff640 1 -- 172.21.15.123:0/189904098 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 19) ==== 97488+0+0 (secure 0 0 0) 0x7efc5c01a070 con 0x7efc6c15b9f0 2024-09-16T15:49:35.977 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:35.974+0000 7efc53fff640 1 --2- 172.21.15.123:0/189904098 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7efc58073340 0x7efc58075800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:49:35.977 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:35.974+0000 7efc53fff640 1 -- 172.21.15.123:0/189904098 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(120..120 src has 1..120) ==== 5795+0+0 (secure 0 0 0) 0x7efc5c093950 con 0x7efc6c15b9f0 2024-09-16T15:49:35.977 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:35.974+0000 7efc6affd640 1 --2- 172.21.15.123:0/189904098 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7efc58073340 0x7efc58075800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:49:35.977 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:35.978+0000 7efc6affd640 1 --2- 172.21.15.123:0/189904098 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7efc58073340 0x7efc58075800 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7efc64009f20 tx=0x7efc640094d0 comp rx=0 tx=0).ready entity=mgr.4150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:49:35.980 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:35.978+0000 7efc53fff640 1 -- 172.21.15.123:0/189904098 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7efc5c05ddb0 con 0x7efc6c15b9f0 2024-09-16T15:49:36.183 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:36.182+0000 7efc71955640 1 -- 172.21.15.123:0/189904098 --> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] -- mgr_command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "subvol95170", "target": ["mon-mgr", ""]}) -- 0x7efc6c1605d0 con 0x7efc58073340 2024-09-16T15:49:36.220 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:36.218+0000 7efc53fff640 1 -- 172.21.15.123:0/189904098 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mgrmap(e 20) ==== 97539+0+0 (secure 0 0 0) 0x7efc5c05a020 con 0x7efc6c15b9f0 2024-09-16T15:49:36.260 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:36.258+0000 7efc53fff640 1 -- 172.21.15.123:0/189904098 <== mgr.4150 v2:172.21.15.123:6836/2464759676 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7efc6c1605d0 con 0x7efc58073340 2024-09-16T15:49:36.263 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:36.262+0000 7efc71955640 1 -- 172.21.15.123:0/189904098 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7efc58073340 msgr2=0x7efc58075800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:36.263 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:36.262+0000 7efc71955640 1 --2- 172.21.15.123:0/189904098 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7efc58073340 0x7efc58075800 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7efc64009f20 tx=0x7efc640094d0 comp rx=0 tx=0).stop 2024-09-16T15:49:36.263 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:36.262+0000 7efc71955640 1 -- 172.21.15.123:0/189904098 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7efc6c15b9f0 msgr2=0x7efc6c156ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:36.263 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:36.262+0000 7efc71955640 1 --2- 172.21.15.123:0/189904098 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7efc6c15b9f0 0x7efc6c156ac0 secure :-1 s=READY pgs=1309 cs=0 l=1 rev1=1 crypto rx=0x7efc5c00c460 tx=0x7efc5c00c920 comp rx=0 tx=0).stop 2024-09-16T15:49:36.263 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:36.262+0000 7efc71955640 1 -- 172.21.15.123:0/189904098 shutdown_connections 2024-09-16T15:49:36.264 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:36.262+0000 7efc71955640 1 --2- 172.21.15.123:0/189904098 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7efc58073340 0x7efc58075800 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:36.264 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:36.262+0000 7efc71955640 1 --2- 172.21.15.123:0/189904098 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7efc6c15b9f0 0x7efc6c156ac0 unknown :-1 s=CLOSED pgs=1309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:36.264 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:36.262+0000 7efc71955640 1 --2- 172.21.15.123:0/189904098 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7efc6c0cf310 0x7efc6c14f6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:36.264 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:36.262+0000 7efc71955640 1 --2- 172.21.15.123:0/189904098 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7efc6c0c3230 0x7efc6c14d1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:36.264 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:36.262+0000 7efc71955640 1 -- 172.21.15.123:0/189904098 >> 172.21.15.123:0/189904098 conn(0x7efc6c0b1710 msgr2=0x7efc6c072410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:49:36.264 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:36.262+0000 7efc71955640 1 -- 172.21.15.123:0/189904098 shutdown_connections 2024-09-16T15:49:36.264 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:36.262+0000 7efc71955640 1 -- 172.21.15.123:0/189904098 wait complete. 2024-09-16T15:49:36.267 INFO:tasks.ceph.mgr.y.smithi123.stderr:2024-09-16T15:49:36.266+0000 7fa9b66fa640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-16T15:49:36.267 INFO:tasks.ceph.mgr.y.smithi123.stderr:2024-09-16T15:49:36.266+0000 7fa9b66fa640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-16T15:49:36.267 INFO:tasks.ceph.mgr.y.smithi123.stderr:2024-09-16T15:49:36.266+0000 7fa9b66fa640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-16T15:49:36.267 INFO:tasks.ceph.mgr.y.smithi123.stderr:2024-09-16T15:49:36.266+0000 7fa9b66fa640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-16T15:49:36.267 INFO:tasks.ceph.mgr.y.smithi123.stderr:2024-09-16T15:49:36.266+0000 7fa9b66fa640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-16T15:49:36.284 INFO:tasks.ceph.mgr.y.smithi123.stderr:2024-09-16T15:49:36.282+0000 7fa9b5ef9640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-16T15:49:36.284 INFO:tasks.ceph.mgr.y.smithi123.stderr:2024-09-16T15:49:36.282+0000 7fa9b5ef9640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-16T15:49:36.284 INFO:tasks.ceph.mgr.y.smithi123.stderr:2024-09-16T15:49:36.282+0000 7fa9b5ef9640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-16T15:49:36.284 INFO:tasks.ceph.mgr.y.smithi123.stderr:2024-09-16T15:49:36.282+0000 7fa9b5ef9640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-16T15:49:36.284 INFO:tasks.ceph.mgr.y.smithi123.stderr:2024-09-16T15:49:36.282+0000 7fa9b5ef9640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-16T15:49:36.286 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume rm cephfs subvol97901 subvol_grp86620 2024-09-16T15:49:36.414 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:36.414+0000 7f5cb9f47640 1 -- 172.21.15.123:0/1345231148 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5cb40ccd40 msgr2=0x7f5cb40cd1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:36.414 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:36.414+0000 7f5cb9f47640 1 --2- 172.21.15.123:0/1345231148 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5cb40ccd40 0x7f5cb40cd1d0 secure :-1 s=READY pgs=1314 cs=0 l=1 rev1=1 crypto rx=0x7f5cac009c80 tx=0x7f5cac01cda0 comp rx=0 tx=0).stop 2024-09-16T15:49:36.414 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:36.414+0000 7f5cb9f47640 1 -- 172.21.15.123:0/1345231148 shutdown_connections 2024-09-16T15:49:36.414 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:36.414+0000 7f5cb9f47640 1 --2- 172.21.15.123:0/1345231148 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5cb40ccd40 0x7f5cb40cd1d0 unknown :-1 s=CLOSED pgs=1314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:36.414 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:36.414+0000 7f5cb9f47640 1 --2- 172.21.15.123:0/1345231148 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f5cb40af480 0x7f5cb415f330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:36.414 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:36.414+0000 7f5cb9f47640 1 -- 172.21.15.123:0/1345231148 >> 172.21.15.123:0/1345231148 conn(0x7f5cb4074110 msgr2=0x7f5cb40744f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:49:36.414 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:36.414+0000 7f5cb9f47640 1 -- 172.21.15.123:0/1345231148 shutdown_connections 2024-09-16T15:49:36.414 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:36.414+0000 7f5cb9f47640 1 -- 172.21.15.123:0/1345231148 wait complete. 2024-09-16T15:49:36.415 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:36.414+0000 7f5cb9f47640 1 Processor -- start 2024-09-16T15:49:36.415 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:36.414+0000 7f5cb9f47640 1 -- start start 2024-09-16T15:49:36.417 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:36.414+0000 7f5cb9f47640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5cb40af480 0x7f5cb40c40c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:49:36.417 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:36.414+0000 7f5cb9f47640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f5cb40ccd40 0x7f5cb415ca10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:49:36.417 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:36.414+0000 7f5cb9f47640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5cb415ada0 0x7f5cb415cf50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:49:36.417 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:36.414+0000 7f5cb9f47640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f5cb4167eb0 con 0x7f5cb40af480 2024-09-16T15:49:36.417 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:36.414+0000 7f5cb9f47640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f5cb40cc980 con 0x7f5cb415ada0 2024-09-16T15:49:36.417 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:36.414+0000 7f5cb9f47640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f5cb4168030 con 0x7f5cb40ccd40 2024-09-16T15:49:36.417 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:36.414+0000 7f5cb37fe640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5cb40af480 0x7f5cb40c40c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:49:36.417 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:36.414+0000 7f5cb37fe640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5cb40af480 0x7f5cb40c40c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:56676/0 (socket says 172.21.15.123:56676) 2024-09-16T15:49:36.417 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:36.414+0000 7f5cb37fe640 1 -- 172.21.15.123:0/1502542526 learned_addr learned my addr 172.21.15.123:0/1502542526 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:49:36.417 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:36.414+0000 7f5cb2ffd640 1 --2- 172.21.15.123:0/1502542526 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f5cb40ccd40 0x7f5cb415ca10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:49:36.418 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:36.414+0000 7f5cb3fff640 1 --2- 172.21.15.123:0/1502542526 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5cb415ada0 0x7f5cb415cf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:49:36.418 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:36.414+0000 7f5cb37fe640 1 -- 172.21.15.123:0/1502542526 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f5cb40ccd40 msgr2=0x7f5cb415ca10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:36.418 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:36.414+0000 7f5cb37fe640 1 --2- 172.21.15.123:0/1502542526 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f5cb40ccd40 0x7f5cb415ca10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:36.418 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:36.414+0000 7f5cb37fe640 1 -- 172.21.15.123:0/1502542526 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5cb415ada0 msgr2=0x7f5cb415cf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:36.418 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:36.414+0000 7f5cb37fe640 1 --2- 172.21.15.123:0/1502542526 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5cb415ada0 0x7f5cb415cf50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:36.418 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:36.414+0000 7f5cb37fe640 1 -- 172.21.15.123:0/1502542526 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5cb40c4a30 con 0x7f5cb40af480 2024-09-16T15:49:36.418 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:36.414+0000 7f5cb37fe640 1 --2- 172.21.15.123:0/1502542526 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5cb40af480 0x7f5cb40c40c0 secure :-1 s=READY pgs=1315 cs=0 l=1 rev1=1 crypto rx=0x7f5ca400c3c0 tx=0x7f5ca400c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:49:36.418 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:36.414+0000 7f5c9bfff640 1 -- 172.21.15.123:0/1502542526 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ca4017070 con 0x7f5cb40af480 2024-09-16T15:49:36.418 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:36.414+0000 7f5c9bfff640 1 -- 172.21.15.123:0/1502542526 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f5ca40053c0 con 0x7f5cb40af480 2024-09-16T15:49:36.418 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:36.414+0000 7f5cb9f47640 1 -- 172.21.15.123:0/1502542526 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5cb40c6b00 con 0x7f5cb40af480 2024-09-16T15:49:36.418 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:36.414+0000 7f5c9bfff640 1 -- 172.21.15.123:0/1502542526 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ca4003860 con 0x7f5cb40af480 2024-09-16T15:49:36.418 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:36.414+0000 7f5cb9f47640 1 -- 172.21.15.123:0/1502542526 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5cb40c4fb0 con 0x7f5cb40af480 2024-09-16T15:49:36.419 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:36.418+0000 7f5cb9f47640 1 -- 172.21.15.123:0/1502542526 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5cb415f330 con 0x7f5cb40af480 2024-09-16T15:49:36.419 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:36.418+0000 7f5c9bfff640 1 -- 172.21.15.123:0/1502542526 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 20) ==== 97539+0+0 (secure 0 0 0) 0x7f5ca401a070 con 0x7f5cb40af480 2024-09-16T15:49:36.419 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:36.418+0000 7f5c9bfff640 1 --2- 172.21.15.123:0/1502542526 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f5ca00733d0 0x7f5ca0075890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:49:36.420 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:36.418+0000 7f5c9bfff640 1 -- 172.21.15.123:0/1502542526 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(121..121 src has 1..121) ==== 5795+0+0 (secure 0 0 0) 0x7f5ca4093be0 con 0x7f5cb40af480 2024-09-16T15:49:36.420 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:36.418+0000 7f5cb2ffd640 1 --2- 172.21.15.123:0/1502542526 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f5ca00733d0 0x7f5ca0075890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:49:36.420 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:36.418+0000 7f5cb2ffd640 1 --2- 172.21.15.123:0/1502542526 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f5ca00733d0 0x7f5ca0075890 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f5ca8005d90 tx=0x7f5ca8005c60 comp rx=0 tx=0).ready entity=mgr.4150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:49:36.422 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:36.422+0000 7f5c9bfff640 1 -- 172.21.15.123:0/1502542526 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f5ca405e530 con 0x7f5cb40af480 2024-09-16T15:49:36.610 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:36.610+0000 7f5cb9f47640 1 -- 172.21.15.123:0/1502542526 --> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] -- mgr_command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "subvol97901", "group_name": "subvol_grp86620", "target": ["mon-mgr", ""]}) -- 0x7f5cb40b62b0 con 0x7f5ca00733d0 2024-09-16T15:49:36.695 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:36.694+0000 7f5c9bfff640 1 -- 172.21.15.123:0/1502542526 <== mgr.4150 v2:172.21.15.123:6836/2464759676 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f5cb40b62b0 con 0x7f5ca00733d0 2024-09-16T15:49:36.698 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:36.698+0000 7f5cb9f47640 1 -- 172.21.15.123:0/1502542526 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f5ca00733d0 msgr2=0x7f5ca0075890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:36.698 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:36.698+0000 7f5cb9f47640 1 --2- 172.21.15.123:0/1502542526 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f5ca00733d0 0x7f5ca0075890 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f5ca8005d90 tx=0x7f5ca8005c60 comp rx=0 tx=0).stop 2024-09-16T15:49:36.698 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:36.698+0000 7f5cb9f47640 1 -- 172.21.15.123:0/1502542526 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5cb40af480 msgr2=0x7f5cb40c40c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:36.698 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:36.698+0000 7f5cb9f47640 1 --2- 172.21.15.123:0/1502542526 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5cb40af480 0x7f5cb40c40c0 secure :-1 s=READY pgs=1315 cs=0 l=1 rev1=1 crypto rx=0x7f5ca400c3c0 tx=0x7f5ca400c880 comp rx=0 tx=0).stop 2024-09-16T15:49:36.698 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:36.698+0000 7f5cb9f47640 1 -- 172.21.15.123:0/1502542526 shutdown_connections 2024-09-16T15:49:36.699 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:36.698+0000 7f5cb9f47640 1 --2- 172.21.15.123:0/1502542526 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f5ca00733d0 0x7f5ca0075890 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:36.699 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:36.698+0000 7f5cb9f47640 1 --2- 172.21.15.123:0/1502542526 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5cb415ada0 0x7f5cb415cf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:36.699 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:36.698+0000 7f5cb9f47640 1 --2- 172.21.15.123:0/1502542526 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f5cb40ccd40 0x7f5cb415ca10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:36.699 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:36.698+0000 7f5cb9f47640 1 --2- 172.21.15.123:0/1502542526 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5cb40af480 0x7f5cb40c40c0 unknown :-1 s=CLOSED pgs=1315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:36.699 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:36.698+0000 7f5cb9f47640 1 -- 172.21.15.123:0/1502542526 >> 172.21.15.123:0/1502542526 conn(0x7f5cb4074110 msgr2=0x7f5cb406ff90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:49:36.699 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:36.698+0000 7f5cb9f47640 1 -- 172.21.15.123:0/1502542526 shutdown_connections 2024-09-16T15:49:36.699 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:36.698+0000 7f5cb9f47640 1 -- 172.21.15.123:0/1502542526 wait complete. 2024-09-16T15:49:36.716 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume rm cephfs subvol4312 2024-09-16T15:49:36.842 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:36.842+0000 7f67abd11640 1 -- 172.21.15.123:0/3941646972 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f67a4151bf0 msgr2=0x7f67a406feb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:36.842 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:36.842+0000 7f67abd11640 1 --2- 172.21.15.123:0/3941646972 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f67a4151bf0 0x7f67a406feb0 secure :-1 s=READY pgs=1316 cs=0 l=1 rev1=1 crypto rx=0x7f67a40b6d80 tx=0x7f679c01cd80 comp rx=0 tx=0).stop 2024-09-16T15:49:36.843 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:36.842+0000 7f67abd11640 1 -- 172.21.15.123:0/3941646972 shutdown_connections 2024-09-16T15:49:36.843 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:36.842+0000 7f67abd11640 1 --2- 172.21.15.123:0/3941646972 >> v2:172.21.15.184:3300/0 conn(0x7f67a406e480 0x7f67a40c61a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:36.843 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:36.842+0000 7f67abd11640 1 --2- 172.21.15.123:0/3941646972 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f67a4151bf0 0x7f67a406feb0 unknown :-1 s=CLOSED pgs=1316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:36.843 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:36.842+0000 7f67abd11640 1 -- 172.21.15.123:0/3941646972 >> 172.21.15.123:0/3941646972 conn(0x7f67a40d0bf0 msgr2=0x7f67a40d1000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:49:36.843 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:36.842+0000 7f67abd11640 1 -- 172.21.15.123:0/3941646972 shutdown_connections 2024-09-16T15:49:36.843 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:36.842+0000 7f67abd11640 1 -- 172.21.15.123:0/3941646972 wait complete. 2024-09-16T15:49:36.843 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:36.842+0000 7f67abd11640 1 Processor -- start 2024-09-16T15:49:36.844 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:36.842+0000 7f67abd11640 1 -- start start 2024-09-16T15:49:36.845 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:36.842+0000 7f67abd11640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f67a406e480 0x7f67a4153d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:49:36.845 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:36.842+0000 7f67abd11640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f67a40703f0 0x7f67a4148c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:49:36.846 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:36.842+0000 7f67abd11640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f67a4151bf0 0x7f67a415e570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:49:36.846 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:36.842+0000 7f67abd11640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f67a4167980 con 0x7f67a40703f0 2024-09-16T15:49:36.846 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:36.842+0000 7f67abd11640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f67a40c74b0 con 0x7f67a4151bf0 2024-09-16T15:49:36.846 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:36.842+0000 7f67abd11640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f67a4167b00 con 0x7f67a406e480 2024-09-16T15:49:36.846 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:36.842+0000 7f67a92ac640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f67a40703f0 0x7f67a4148c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:49:36.846 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:36.842+0000 7f67a92ac640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f67a40703f0 0x7f67a4148c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:56686/0 (socket says 172.21.15.123:56686) 2024-09-16T15:49:36.846 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:36.842+0000 7f67a92ac640 1 -- 172.21.15.123:0/2030399769 learned_addr learned my addr 172.21.15.123:0/2030399769 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:49:36.846 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:36.846+0000 7f67aa2ae640 1 --2- 172.21.15.123:0/2030399769 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f67a4151bf0 0x7f67a415e570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:49:36.846 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:36.846+0000 7f67a9aad640 1 --2- 172.21.15.123:0/2030399769 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f67a406e480 0x7f67a4153d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:49:36.846 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:36.846+0000 7f67a92ac640 1 -- 172.21.15.123:0/2030399769 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f67a406e480 msgr2=0x7f67a4153d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:36.846 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:36.846+0000 7f67a92ac640 1 --2- 172.21.15.123:0/2030399769 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f67a406e480 0x7f67a4153d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:36.847 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:36.846+0000 7f67a92ac640 1 -- 172.21.15.123:0/2030399769 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f67a4151bf0 msgr2=0x7f67a415e570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:36.847 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:36.846+0000 7f67a92ac640 1 --2- 172.21.15.123:0/2030399769 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f67a4151bf0 0x7f67a415e570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:36.847 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:36.846+0000 7f67a92ac640 1 -- 172.21.15.123:0/2030399769 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f67a414c3d0 con 0x7f67a40703f0 2024-09-16T15:49:36.847 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:36.846+0000 7f67a92ac640 1 --2- 172.21.15.123:0/2030399769 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f67a40703f0 0x7f67a4148c40 secure :-1 s=READY pgs=1317 cs=0 l=1 rev1=1 crypto rx=0x7f679800c450 tx=0x7f679800c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:49:36.847 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:36.846+0000 7f67a27fc640 1 -- 172.21.15.123:0/2030399769 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6798012830 con 0x7f67a40703f0 2024-09-16T15:49:36.847 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:36.846+0000 7f67a27fc640 1 -- 172.21.15.123:0/2030399769 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f6798017070 con 0x7f67a40703f0 2024-09-16T15:49:36.847 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:36.846+0000 7f67a27fc640 1 -- 172.21.15.123:0/2030399769 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6798012d80 con 0x7f67a40703f0 2024-09-16T15:49:36.847 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:36.846+0000 7f67abd11640 1 -- 172.21.15.123:0/2030399769 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f67a4152fa0 con 0x7f67a40703f0 2024-09-16T15:49:36.847 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:36.846+0000 7f67abd11640 1 -- 172.21.15.123:0/2030399769 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f67a4153560 con 0x7f67a40703f0 2024-09-16T15:49:36.848 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:36.846+0000 7f67abd11640 1 -- 172.21.15.123:0/2030399769 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f67a406ea70 con 0x7f67a40703f0 2024-09-16T15:49:36.848 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:36.846+0000 7f67a27fc640 1 -- 172.21.15.123:0/2030399769 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 20) ==== 97539+0+0 (secure 0 0 0) 0x7f679801c460 con 0x7f67a40703f0 2024-09-16T15:49:36.848 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:36.846+0000 7f67a27fc640 1 --2- 172.21.15.123:0/2030399769 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f67900733d0 0x7f6790075890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:49:36.848 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:36.846+0000 7f67a9aad640 1 --2- 172.21.15.123:0/2030399769 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f67900733d0 0x7f6790075890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:49:36.848 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:36.846+0000 7f67a27fc640 1 -- 172.21.15.123:0/2030399769 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(121..121 src has 1..121) ==== 5795+0+0 (secure 0 0 0) 0x7f67980945f0 con 0x7f67a40703f0 2024-09-16T15:49:36.848 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:36.846+0000 7f67a9aad640 1 --2- 172.21.15.123:0/2030399769 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f67900733d0 0x7f6790075890 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f679c0af000 tx=0x7f679c01b5c0 comp rx=0 tx=0).ready entity=mgr.4150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:49:36.851 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:36.850+0000 7f67a27fc640 1 -- 172.21.15.123:0/2030399769 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f679805efd0 con 0x7f67a40703f0 2024-09-16T15:49:37.046 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:37.046+0000 7f67abd11640 1 -- 172.21.15.123:0/2030399769 --> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] -- mgr_command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "subvol4312", "target": ["mon-mgr", ""]}) -- 0x7f67a4149180 con 0x7f67900733d0 2024-09-16T15:49:37.056 INFO:tasks.ceph.mgr.y.smithi123.stderr:2024-09-16T15:49:37.054+0000 7fa9b36f4640 -1 mgr.server reply reply (11) Resource temporarily unavailable subvolume 'subvol4312' is not ready for operation rm (use --force to override) 2024-09-16T15:49:37.056 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:37.054+0000 7f67a27fc640 1 -- 172.21.15.123:0/2030399769 <== mgr.4150 v2:172.21.15.123:6836/2464759676 1 ==== mgr_command_reply(tid 0: -11 subvolume 'subvol4312' is not ready for operation rm (use --force to override)) ==== 86+0+0 (secure 0 0 0) 0x7f67a4149180 con 0x7f67900733d0 2024-09-16T15:49:37.056 INFO:teuthology.orchestra.run.smithi123.stderr:Error EAGAIN: subvolume 'subvol4312' is not ready for operation rm (use --force to override) 2024-09-16T15:49:37.062 DEBUG:teuthology.orchestra.run:got remote process result: 11 2024-09-16T15:49:37.063 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:49:37.063 DEBUG:teuthology.orchestra.run.smithi123:> (cd /home/ubuntu/cephtest/mnt.0 && exec sudo cat ./volumes/_nogroup/subvol4312/.meta) 2024-09-16T15:49:37.076 INFO:teuthology.orchestra.run.smithi123.stdout:[GLOBAL] 2024-09-16T15:49:37.076 INFO:teuthology.orchestra.run.smithi123.stdout:version = 1 2024-09-16T15:49:37.076 INFO:teuthology.orchestra.run.smithi123.stdout:type = clone 2024-09-16T15:49:37.076 INFO:teuthology.orchestra.run.smithi123.stdout:path = /volumes/_nogroup/subvol4312/5bbaeaac-ff1e-47e6-b3e5-d523916453aa 2024-09-16T15:49:37.076 INFO:teuthology.orchestra.run.smithi123.stdout:state = pending 2024-09-16T15:49:37.076 INFO:teuthology.orchestra.run.smithi123.stdout:[source] 2024-09-16T15:49:37.076 INFO:teuthology.orchestra.run.smithi123.stdout:volume = fake 2024-09-16T15:49:37.077 INFO:teuthology.orchestra.run.smithi123.stdout:subvolume = fake 2024-09-16T15:49:37.077 INFO:teuthology.orchestra.run.smithi123.stdout:snapshot = fake 2024-09-16T15:49:37.078 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume rm cephfs subvol4312 --force 2024-09-16T15:49:37.245 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:37.246+0000 7ff00e6b7640 1 -- 172.21.15.123:0/4099750063 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff008161390 msgr2=0x7ff0080d0630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:37.245 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:37.246+0000 7ff00e6b7640 1 --2- 172.21.15.123:0/4099750063 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff008161390 0x7ff0080d0630 secure :-1 s=READY pgs=1319 cs=0 l=1 rev1=1 crypto rx=0x7feff8009870 tx=0x7feff801c7f0 comp rx=0 tx=0).stop 2024-09-16T15:49:37.245 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:37.246+0000 7ff00e6b7640 1 -- 172.21.15.123:0/4099750063 shutdown_connections 2024-09-16T15:49:37.245 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:37.246+0000 7ff00e6b7640 1 --2- 172.21.15.123:0/4099750063 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff008161390 0x7ff0080d0630 unknown :-1 s=CLOSED pgs=1319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:37.245 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:37.246+0000 7ff00e6b7640 1 -- 172.21.15.123:0/4099750063 >> 172.21.15.123:0/4099750063 conn(0x7ff0080d3e30 msgr2=0x7ff0080d4240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:49:37.246 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:37.246+0000 7ff00e6b7640 1 -- 172.21.15.123:0/4099750063 shutdown_connections 2024-09-16T15:49:37.246 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:37.246+0000 7ff00e6b7640 1 -- 172.21.15.123:0/4099750063 wait complete. 2024-09-16T15:49:37.248 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:37.246+0000 7ff00e6b7640 1 Processor -- start 2024-09-16T15:49:37.248 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:37.246+0000 7ff00e6b7640 1 -- start start 2024-09-16T15:49:37.248 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:37.246+0000 7ff00e6b7640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7ff0080d0b70 0x7ff0081606a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:49:37.248 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:37.246+0000 7ff00e6b7640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff0080d2a10 0x7ff0080b90d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:49:37.248 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:37.246+0000 7ff00e6b7640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff008161390 0x7ff0080cce50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:49:37.249 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:37.246+0000 7ff00e6b7640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7ff00816c700 con 0x7ff0080d2a10 2024-09-16T15:49:37.249 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:37.246+0000 7ff00e6b7640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7ff00816c580 con 0x7ff008161390 2024-09-16T15:49:37.249 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:37.246+0000 7ff00e6b7640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7ff00816c880 con 0x7ff0080d0b70 2024-09-16T15:49:37.249 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:37.246+0000 7ff0077fe640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff0080d2a10 0x7ff0080b90d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:49:37.249 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:37.246+0000 7ff0077fe640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff0080d2a10 0x7ff0080b90d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:56702/0 (socket says 172.21.15.123:56702) 2024-09-16T15:49:37.249 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:37.246+0000 7ff0077fe640 1 -- 172.21.15.123:0/1895667183 learned_addr learned my addr 172.21.15.123:0/1895667183 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:49:37.249 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:37.246+0000 7ff007fff640 1 --2- 172.21.15.123:0/1895667183 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7ff0080d0b70 0x7ff0081606a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:49:37.249 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:37.246+0000 7ff0077fe640 1 -- 172.21.15.123:0/1895667183 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7ff0080d0b70 msgr2=0x7ff0081606a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:37.249 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:37.246+0000 7ff0077fe640 1 --2- 172.21.15.123:0/1895667183 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7ff0080d0b70 0x7ff0081606a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:37.249 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:37.246+0000 7ff00cc54640 1 --2- 172.21.15.123:0/1895667183 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff008161390 0x7ff0080cce50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:49:37.249 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:37.246+0000 7ff0077fe640 1 -- 172.21.15.123:0/1895667183 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff008161390 msgr2=0x7ff0080cce50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:37.250 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:37.246+0000 7ff0077fe640 1 --2- 172.21.15.123:0/1895667183 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff008161390 0x7ff0080cce50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:37.250 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:37.246+0000 7ff0077fe640 1 -- 172.21.15.123:0/1895667183 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff0080b9610 con 0x7ff0080d2a10 2024-09-16T15:49:37.250 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:37.246+0000 7ff0077fe640 1 --2- 172.21.15.123:0/1895667183 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff0080d2a10 0x7ff0080b90d0 secure :-1 s=READY pgs=1320 cs=0 l=1 rev1=1 crypto rx=0x7feffc00c460 tx=0x7feffc00c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:49:37.250 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:37.246+0000 7ff004ff9640 1 -- 172.21.15.123:0/1895667183 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feffc016020 con 0x7ff0080d2a10 2024-09-16T15:49:37.250 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:37.246+0000 7ff004ff9640 1 -- 172.21.15.123:0/1895667183 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7feffc012940 con 0x7ff0080d2a10 2024-09-16T15:49:37.250 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:37.246+0000 7ff004ff9640 1 -- 172.21.15.123:0/1895667183 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feffc012d90 con 0x7ff0080d2a10 2024-09-16T15:49:37.250 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:37.246+0000 7ff00e6b7640 1 -- 172.21.15.123:0/1895667183 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff0080b8030 con 0x7ff0080d2a10 2024-09-16T15:49:37.250 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:37.246+0000 7ff00e6b7640 1 -- 172.21.15.123:0/1895667183 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff0080b85f0 con 0x7ff0080d2a10 2024-09-16T15:49:37.251 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:37.250+0000 7ff00e6b7640 1 -- 172.21.15.123:0/1895667183 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff0080d0630 con 0x7ff0080d2a10 2024-09-16T15:49:37.251 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:37.250+0000 7ff004ff9640 1 -- 172.21.15.123:0/1895667183 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 20) ==== 97539+0+0 (secure 0 0 0) 0x7feffc019020 con 0x7ff0080d2a10 2024-09-16T15:49:37.251 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:37.250+0000 7ff004ff9640 1 --2- 172.21.15.123:0/1895667183 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7feff40733d0 0x7feff4075890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:49:37.251 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:37.250+0000 7ff007fff640 1 --2- 172.21.15.123:0/1895667183 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7feff40733d0 0x7feff4075890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:49:37.251 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:37.250+0000 7ff004ff9640 1 -- 172.21.15.123:0/1895667183 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(122..122 src has 1..122) ==== 5883+0+0 (secure 0 0 0) 0x7feffc0945e0 con 0x7ff0080d2a10 2024-09-16T15:49:37.252 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:37.250+0000 7ff007fff640 1 --2- 172.21.15.123:0/1895667183 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7feff40733d0 0x7feff4075890 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7feff80af000 tx=0x7feff801b030 comp rx=0 tx=0).ready entity=mgr.4150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:49:37.254 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:37.254+0000 7ff004ff9640 1 -- 172.21.15.123:0/1895667183 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7feffc020030 con 0x7ff0080d2a10 2024-09-16T15:49:37.443 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:37.442+0000 7ff00e6b7640 1 -- 172.21.15.123:0/1895667183 --> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] -- mgr_command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "subvol4312", "force": true, "target": ["mon-mgr", ""]}) -- 0x7ff0080b60d0 con 0x7feff40733d0 2024-09-16T15:49:37.466 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:37.466+0000 7ff004ff9640 1 -- 172.21.15.123:0/1895667183 <== mgr.4150 v2:172.21.15.123:6836/2464759676 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7ff0080b60d0 con 0x7feff40733d0 2024-09-16T15:49:37.469 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:37.470+0000 7ff00e6b7640 1 -- 172.21.15.123:0/1895667183 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7feff40733d0 msgr2=0x7feff4075890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:37.469 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:37.470+0000 7ff00e6b7640 1 --2- 172.21.15.123:0/1895667183 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7feff40733d0 0x7feff4075890 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7feff80af000 tx=0x7feff801b030 comp rx=0 tx=0).stop 2024-09-16T15:49:37.469 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:37.470+0000 7ff00e6b7640 1 -- 172.21.15.123:0/1895667183 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff0080d2a10 msgr2=0x7ff0080b90d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:37.469 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:37.470+0000 7ff00e6b7640 1 --2- 172.21.15.123:0/1895667183 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff0080d2a10 0x7ff0080b90d0 secure :-1 s=READY pgs=1320 cs=0 l=1 rev1=1 crypto rx=0x7feffc00c460 tx=0x7feffc00c920 comp rx=0 tx=0).stop 2024-09-16T15:49:37.469 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:37.470+0000 7ff00e6b7640 1 -- 172.21.15.123:0/1895667183 shutdown_connections 2024-09-16T15:49:37.469 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:37.470+0000 7ff00e6b7640 1 --2- 172.21.15.123:0/1895667183 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7feff40733d0 0x7feff4075890 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:37.469 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:37.470+0000 7ff00e6b7640 1 --2- 172.21.15.123:0/1895667183 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff008161390 0x7ff0080cce50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:37.469 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:37.470+0000 7ff00e6b7640 1 --2- 172.21.15.123:0/1895667183 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff0080d2a10 0x7ff0080b90d0 unknown :-1 s=CLOSED pgs=1320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:37.470 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:37.470+0000 7ff00e6b7640 1 --2- 172.21.15.123:0/1895667183 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7ff0080d0b70 0x7ff0081606a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:37.470 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:37.470+0000 7ff00e6b7640 1 -- 172.21.15.123:0/1895667183 >> 172.21.15.123:0/1895667183 conn(0x7ff0080d3e30 msgr2=0x7ff00806e810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:49:37.470 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:37.470+0000 7ff00e6b7640 1 -- 172.21.15.123:0/1895667183 shutdown_connections 2024-09-16T15:49:37.470 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:37.470+0000 7ff00e6b7640 1 -- 172.21.15.123:0/1895667183 wait complete. 2024-09-16T15:49:37.485 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume ls cephfs 2024-09-16T15:49:37.604 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:37.602+0000 7f81e27b0640 1 -- 172.21.15.123:0/4259844569 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f81bc004d50 msgr2=0x7f81bc0051d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:37.605 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:37.602+0000 7f81e27b0640 1 --2- 172.21.15.123:0/4259844569 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f81bc004d50 0x7f81bc0051d0 secure :-1 s=READY pgs=1322 cs=0 l=1 rev1=1 crypto rx=0x7f81cc005fd0 tx=0x7f81cc01cd80 comp rx=0 tx=0).stop 2024-09-16T15:49:37.605 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:37.602+0000 7f81e0d4d640 1 -- 172.21.15.123:0/4259844569 reap_dead start 2024-09-16T15:49:37.605 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:37.602+0000 7f81e27b0640 1 -- 172.21.15.123:0/4259844569 shutdown_connections 2024-09-16T15:49:37.605 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:37.602+0000 7f81e27b0640 1 -- 172.21.15.123:0/4259844569 >> 172.21.15.123:0/4259844569 conn(0x7f81dc0d3e40 msgr2=0x7f81dc0d4220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:49:37.605 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:37.602+0000 7f81e27b0640 1 -- 172.21.15.123:0/4259844569 shutdown_connections 2024-09-16T15:49:37.605 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:37.602+0000 7f81e27b0640 1 -- 172.21.15.123:0/4259844569 wait complete. 2024-09-16T15:49:37.605 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:37.606+0000 7f81e27b0640 1 Processor -- start 2024-09-16T15:49:37.606 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:37.606+0000 7f81e27b0640 1 -- start start 2024-09-16T15:49:37.608 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:37.606+0000 7f81e27b0640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f81bc004d50 0x7f81dc155250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:49:37.608 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:37.606+0000 7f81e27b0640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f81dc1f85f0 0x7f81dc0cd9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:49:37.608 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:37.606+0000 7f81e27b0640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f81dc0cdf00 0x7f81dc0b1760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:49:37.608 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:37.606+0000 7f81e27b0640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f81dc0d1ec0 con 0x7f81dc0cdf00 2024-09-16T15:49:37.608 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:37.606+0000 7f81e27b0640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f81dc0d1d40 con 0x7f81bc004d50 2024-09-16T15:49:37.608 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:37.606+0000 7f81e27b0640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f81dc0b2f10 con 0x7f81dc1f85f0 2024-09-16T15:49:37.608 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:37.606+0000 7f81e0d4d640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f81dc0cdf00 0x7f81dc0b1760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:49:37.608 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:37.606+0000 7f81e0d4d640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f81dc0cdf00 0x7f81dc0b1760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:56726/0 (socket says 172.21.15.123:56726) 2024-09-16T15:49:37.608 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:37.606+0000 7f81e0d4d640 1 -- 172.21.15.123:0/3706401344 learned_addr learned my addr 172.21.15.123:0/3706401344 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:49:37.608 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:37.606+0000 7f81db7fe640 1 --2- 172.21.15.123:0/3706401344 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f81dc1f85f0 0x7f81dc0cd9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:49:37.608 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:37.606+0000 7f81dbfff640 1 --2- 172.21.15.123:0/3706401344 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f81bc004d50 0x7f81dc155250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:49:37.608 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:37.606+0000 7f81e0d4d640 1 -- 172.21.15.123:0/3706401344 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f81dc1f85f0 msgr2=0x7f81dc0cd9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:37.609 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:37.606+0000 7f81e0d4d640 1 --2- 172.21.15.123:0/3706401344 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f81dc1f85f0 0x7f81dc0cd9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:37.609 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:37.606+0000 7f81e0d4d640 1 -- 172.21.15.123:0/3706401344 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f81bc004d50 msgr2=0x7f81dc155250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:37.609 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:37.606+0000 7f81e0d4d640 1 --2- 172.21.15.123:0/3706401344 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f81bc004d50 0x7f81dc155250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:37.609 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:37.606+0000 7f81e0d4d640 1 -- 172.21.15.123:0/3706401344 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f81dc155790 con 0x7f81dc0cdf00 2024-09-16T15:49:37.609 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:37.606+0000 7f81dbfff640 1 --2- 172.21.15.123:0/3706401344 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f81bc004d50 0x7f81dc155250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T15:49:37.609 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:37.606+0000 7f81e0d4d640 1 --2- 172.21.15.123:0/3706401344 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f81dc0cdf00 0x7f81dc0b1760 secure :-1 s=READY pgs=1323 cs=0 l=1 rev1=1 crypto rx=0x7f81cc0b1040 tx=0x7f81cc0055e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:49:37.609 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:37.606+0000 7f81d8ff9640 1 -- 172.21.15.123:0/3706401344 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81cc0be070 con 0x7f81dc0cdf00 2024-09-16T15:49:37.609 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:37.606+0000 7f81d8ff9640 1 -- 172.21.15.123:0/3706401344 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f81cc000e50 con 0x7f81dc0cdf00 2024-09-16T15:49:37.609 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:37.606+0000 7f81e27b0640 1 -- 172.21.15.123:0/3706401344 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f81dc0b8340 con 0x7f81dc0cdf00 2024-09-16T15:49:37.609 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:37.606+0000 7f81d8ff9640 1 -- 172.21.15.123:0/3706401344 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81cc0ab050 con 0x7f81dc0cdf00 2024-09-16T15:49:37.609 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:37.606+0000 7f81e27b0640 1 -- 172.21.15.123:0/3706401344 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f81dc0b8900 con 0x7f81dc0cdf00 2024-09-16T15:49:37.610 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:37.610+0000 7f81e27b0640 1 -- 172.21.15.123:0/3706401344 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f81dc0d0850 con 0x7f81dc0cdf00 2024-09-16T15:49:37.610 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:37.610+0000 7f81d8ff9640 1 -- 172.21.15.123:0/3706401344 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 20) ==== 97539+0+0 (secure 0 0 0) 0x7f81cc0c1070 con 0x7f81dc0cdf00 2024-09-16T15:49:37.610 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:37.610+0000 7f81d8ff9640 1 --2- 172.21.15.123:0/3706401344 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f81c00733d0 0x7f81c0075890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:49:37.611 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:37.610+0000 7f81d8ff9640 1 -- 172.21.15.123:0/3706401344 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(122..122 src has 1..122) ==== 5883+0+0 (secure 0 0 0) 0x7f81cc12e450 con 0x7f81dc0cdf00 2024-09-16T15:49:37.611 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:37.610+0000 7f81dbfff640 1 --2- 172.21.15.123:0/3706401344 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f81c00733d0 0x7f81c0075890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:49:37.611 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:37.610+0000 7f81dbfff640 1 --2- 172.21.15.123:0/3706401344 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f81c00733d0 0x7f81c0075890 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f81c4005fd0 tx=0x7f81c400b550 comp rx=0 tx=0).ready entity=mgr.4150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:49:37.613 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:37.614+0000 7f81d8ff9640 1 -- 172.21.15.123:0/3706401344 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f81cc016a90 con 0x7f81dc0cdf00 2024-09-16T15:49:37.819 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:37.818+0000 7f81e27b0640 1 -- 172.21.15.123:0/3706401344 --> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] -- mgr_command(tid 0: {"prefix": "fs subvolume ls", "vol_name": "cephfs", "target": ["mon-mgr", ""]}) -- 0x7f81dc0b3bb0 con 0x7f81c00733d0 2024-09-16T15:49:37.826 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:37.826+0000 7f81d8ff9640 1 -- 172.21.15.123:0/3706401344 <== mgr.4150 v2:172.21.15.123:6836/2464759676 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+3 (secure 0 0 0) 0x7f81dc0b3bb0 con 0x7f81c00733d0 2024-09-16T15:49:37.826 INFO:teuthology.orchestra.run.smithi123.stdout:[] 2024-09-16T15:49:37.829 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:37.826+0000 7f81e27b0640 1 -- 172.21.15.123:0/3706401344 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f81c00733d0 msgr2=0x7f81c0075890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:37.829 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:37.826+0000 7f81e27b0640 1 --2- 172.21.15.123:0/3706401344 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f81c00733d0 0x7f81c0075890 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f81c4005fd0 tx=0x7f81c400b550 comp rx=0 tx=0).stop 2024-09-16T15:49:37.829 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:37.826+0000 7f81e27b0640 1 -- 172.21.15.123:0/3706401344 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f81dc0cdf00 msgr2=0x7f81dc0b1760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:37.829 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:37.826+0000 7f81e27b0640 1 --2- 172.21.15.123:0/3706401344 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f81dc0cdf00 0x7f81dc0b1760 secure :-1 s=READY pgs=1323 cs=0 l=1 rev1=1 crypto rx=0x7f81cc0b1040 tx=0x7f81cc0055e0 comp rx=0 tx=0).stop 2024-09-16T15:49:37.829 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:37.826+0000 7f81e27b0640 1 -- 172.21.15.123:0/3706401344 shutdown_connections 2024-09-16T15:49:37.829 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:37.826+0000 7f81e27b0640 1 --2- 172.21.15.123:0/3706401344 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f81c00733d0 0x7f81c0075890 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:37.829 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:37.826+0000 7f81e27b0640 1 --2- 172.21.15.123:0/3706401344 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f81dc0cdf00 0x7f81dc0b1760 unknown :-1 s=CLOSED pgs=1323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:37.830 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:37.826+0000 7f81e27b0640 1 --2- 172.21.15.123:0/3706401344 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f81dc1f85f0 0x7f81dc0cd9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:37.830 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:37.826+0000 7f81e27b0640 1 --2- 172.21.15.123:0/3706401344 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f81bc004d50 0x7f81dc155250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:37.830 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:37.826+0000 7f81e27b0640 1 -- 172.21.15.123:0/3706401344 >> 172.21.15.123:0/3706401344 conn(0x7f81dc0d3e40 msgr2=0x7f81dc0ce950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:49:37.830 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:37.830+0000 7f81e27b0640 1 -- 172.21.15.123:0/3706401344 shutdown_connections 2024-09-16T15:49:37.830 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:37.830+0000 7f81e27b0640 1 -- 172.21.15.123:0/3706401344 wait complete. 2024-09-16T15:49:37.845 DEBUG:teuthology.orchestra.run.smithi123:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'stat -c %h /home/ubuntu/cephtest/mnt.0/./volumes/_deleting') 2024-09-16T15:49:37.885 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:37 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-09-16T15:49:37.889 INFO:teuthology.orchestra.run.smithi123.stdout:2 2024-09-16T15:49:37.889 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:37 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-16T15:49:37.895 INFO:teuthology.nuke:Clearing teuthology firewall rules... 2024-09-16T15:49:37.895 DEBUG:teuthology.orchestra.run.smithi123:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-09-16T15:49:37.994 DEBUG:teuthology.orchestra.run.smithi184:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-09-16T15:49:38.062 INFO:teuthology.nuke:Cleared teuthology firewall rules. 2024-09-16T15:49:38.062 INFO:tasks.cephfs.mount:Terminating background process 2024-09-16T15:49:38.062 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2024-09-16T15:49:38.063 DEBUG:teuthology.orchestra.run.smithi123:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-16T15:49:38.069 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2024-09-16T15:49:38.069 DEBUG:teuthology.orchestra.run.smithi123:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-16T15:49:38.118 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-09-16T15:49:38.303 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:38.302+0000 7fef1cadc640 1 -- 172.21.15.123:0/2671891602 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fef1815bca0 msgr2=0x7fef180d0630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:38.303 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:38.302+0000 7fef1cadc640 1 --2- 172.21.15.123:0/2671891602 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fef1815bca0 0x7fef180d0630 secure :-1 s=READY pgs=1324 cs=0 l=1 rev1=1 crypto rx=0x7fef1806df40 tx=0x7fef0801c800 comp rx=0 tx=0).stop 2024-09-16T15:49:38.303 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:38.302+0000 7fef1cadc640 1 -- 172.21.15.123:0/2671891602 shutdown_connections 2024-09-16T15:49:38.303 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:38.302+0000 7fef1cadc640 1 --2- 172.21.15.123:0/2671891602 >> v2:172.21.15.184:3300/0 conn(0x7fef180726c0 0x7fef18073ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:38.303 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:38.302+0000 7fef1cadc640 1 --2- 172.21.15.123:0/2671891602 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fef1815bca0 0x7fef180d0630 unknown :-1 s=CLOSED pgs=1324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:38.303 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:38.302+0000 7fef1cadc640 1 -- 172.21.15.123:0/2671891602 >> 172.21.15.123:0/2671891602 conn(0x7fef180cfb10 msgr2=0x7fef180cff20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:49:38.303 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:38.302+0000 7fef1cadc640 1 -- 172.21.15.123:0/2671891602 shutdown_connections 2024-09-16T15:49:38.303 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:38.302+0000 7fef1cadc640 1 -- 172.21.15.123:0/2671891602 wait complete. 2024-09-16T15:49:38.304 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:38.302+0000 7fef1cadc640 1 Processor -- start 2024-09-16T15:49:38.304 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:38.302+0000 7fef1cadc640 1 -- start start 2024-09-16T15:49:38.306 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:38.306+0000 7fef1cadc640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fef180726c0 0x7fef180c3230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:49:38.306 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:38.306+0000 7fef1cadc640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fef18074430 0x7fef180c8f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:49:38.306 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:38.306+0000 7fef1cadc640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fef1815bca0 0x7fef181f9820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:49:38.307 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:38.306+0000 7fef1cadc640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fef18168e90 con 0x7fef1815bca0 2024-09-16T15:49:38.307 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:38.306+0000 7fef1cadc640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fef18168d10 con 0x7fef18074430 2024-09-16T15:49:38.307 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:38.306+0000 7fef1cadc640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fef18169010 con 0x7fef180726c0 2024-09-16T15:49:38.307 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:38.306+0000 7fef16d9d640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fef1815bca0 0x7fef181f9820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:49:38.307 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:38.306+0000 7fef16d9d640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fef1815bca0 0x7fef181f9820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:56740/0 (socket says 172.21.15.123:56740) 2024-09-16T15:49:38.307 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:38.306+0000 7fef16d9d640 1 -- 172.21.15.123:0/1095595125 learned_addr learned my addr 172.21.15.123:0/1095595125 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:49:38.307 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:38.306+0000 7fef15d9b640 1 --2- 172.21.15.123:0/1095595125 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fef18074430 0x7fef180c8f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:49:38.307 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:38.306+0000 7fef1659c640 1 --2- 172.21.15.123:0/1095595125 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fef180726c0 0x7fef180c3230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:49:38.307 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:38.306+0000 7fef16d9d640 1 -- 172.21.15.123:0/1095595125 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fef180726c0 msgr2=0x7fef180c3230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:38.307 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:38.306+0000 7fef16d9d640 1 --2- 172.21.15.123:0/1095595125 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fef180726c0 0x7fef180c3230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:38.307 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:38.306+0000 7fef16d9d640 1 -- 172.21.15.123:0/1095595125 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fef18074430 msgr2=0x7fef180c8f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:38.307 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:38.306+0000 7fef16d9d640 1 --2- 172.21.15.123:0/1095595125 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fef18074430 0x7fef180c8f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:38.308 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:38.306+0000 7fef16d9d640 1 -- 172.21.15.123:0/1095595125 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fef1815f3b0 con 0x7fef1815bca0 2024-09-16T15:49:38.308 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:38.306+0000 7fef16d9d640 1 --2- 172.21.15.123:0/1095595125 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fef1815bca0 0x7fef181f9820 secure :-1 s=READY pgs=1325 cs=0 l=1 rev1=1 crypto rx=0x7fef1000c780 tx=0x7fef1000cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:49:38.308 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:38.306+0000 7fef06ffd640 1 -- 172.21.15.123:0/1095595125 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef10017070 con 0x7fef1815bca0 2024-09-16T15:49:38.308 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:38.306+0000 7fef06ffd640 1 -- 172.21.15.123:0/1095595125 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fef10005070 con 0x7fef1815bca0 2024-09-16T15:49:38.308 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:38.306+0000 7fef1cadc640 1 -- 172.21.15.123:0/1095595125 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fef1815ee30 con 0x7fef1815bca0 2024-09-16T15:49:38.308 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:38.306+0000 7fef06ffd640 1 -- 172.21.15.123:0/1095595125 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef10005500 con 0x7fef1815bca0 2024-09-16T15:49:38.308 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:38.306+0000 7fef1cadc640 1 -- 172.21.15.123:0/1095595125 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fef180c9440 con 0x7fef1815bca0 2024-09-16T15:49:38.309 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:38.306+0000 7fef1cadc640 1 -- 172.21.15.123:0/1095595125 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fef18073ef0 con 0x7fef1815bca0 2024-09-16T15:49:38.309 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:38.306+0000 7fef06ffd640 1 -- 172.21.15.123:0/1095595125 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 21) ==== 97641+0+0 (secure 0 0 0) 0x7fef1001a070 con 0x7fef1815bca0 2024-09-16T15:49:38.309 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:38.306+0000 7fef06ffd640 1 --2- 172.21.15.123:0/1095595125 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7feef80734f0 0x7feef80759b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:49:38.309 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:38.310+0000 7fef1659c640 1 --2- 172.21.15.123:0/1095595125 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7feef80734f0 0x7feef80759b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:49:38.309 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:38.310+0000 7fef06ffd640 1 -- 172.21.15.123:0/1095595125 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(123..123 src has 1..123) ==== 5867+0+0 (secure 0 0 0) 0x7fef10094150 con 0x7fef1815bca0 2024-09-16T15:49:38.309 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:38.310+0000 7fef1659c640 1 --2- 172.21.15.123:0/1095595125 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7feef80734f0 0x7feef80759b0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fef080af000 tx=0x7fef0801b040 comp rx=0 tx=0).ready entity=mgr.4150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:49:38.313 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:38.314+0000 7fef06ffd640 1 -- 172.21.15.123:0/1095595125 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7fef1005e940 con 0x7fef1815bca0 2024-09-16T15:49:38.503 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:38.502+0000 7fef1cadc640 1 -- 172.21.15.123:0/1095595125 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7fef180700e0 con 0x7fef1815bca0 2024-09-16T15:49:38.504 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:38.502+0000 7fef06ffd640 1 -- 172.21.15.123:0/1095595125 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v123) ==== 80+0+0 (secure 0 0 0) 0x7fef1005e240 con 0x7fef1815bca0 2024-09-16T15:49:38.504 INFO:teuthology.orchestra.run.smithi123.stderr:listed 0 entries 2024-09-16T15:49:38.506 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:38.506+0000 7fef1cadc640 1 -- 172.21.15.123:0/1095595125 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7feef80734f0 msgr2=0x7feef80759b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:38.506 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:38.506+0000 7fef1cadc640 1 --2- 172.21.15.123:0/1095595125 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7feef80734f0 0x7feef80759b0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fef080af000 tx=0x7fef0801b040 comp rx=0 tx=0).stop 2024-09-16T15:49:38.507 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:38.506+0000 7fef1cadc640 1 -- 172.21.15.123:0/1095595125 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fef1815bca0 msgr2=0x7fef181f9820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:38.507 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:38.506+0000 7fef1cadc640 1 --2- 172.21.15.123:0/1095595125 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fef1815bca0 0x7fef181f9820 secure :-1 s=READY pgs=1325 cs=0 l=1 rev1=1 crypto rx=0x7fef1000c780 tx=0x7fef1000cc40 comp rx=0 tx=0).stop 2024-09-16T15:49:38.507 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:38.506+0000 7fef1cadc640 1 -- 172.21.15.123:0/1095595125 shutdown_connections 2024-09-16T15:49:38.507 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:38.506+0000 7fef1cadc640 1 --2- 172.21.15.123:0/1095595125 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7feef80734f0 0x7feef80759b0 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:38.507 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:38.506+0000 7fef1cadc640 1 --2- 172.21.15.123:0/1095595125 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fef1815bca0 0x7fef181f9820 unknown :-1 s=CLOSED pgs=1325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:38.507 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:38.506+0000 7fef1cadc640 1 --2- 172.21.15.123:0/1095595125 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fef18074430 0x7fef180c8f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:38.507 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:38.506+0000 7fef1cadc640 1 --2- 172.21.15.123:0/1095595125 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fef180726c0 0x7fef180c3230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:38.507 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:38.506+0000 7fef1cadc640 1 -- 172.21.15.123:0/1095595125 >> 172.21.15.123:0/1095595125 conn(0x7fef180cfb10 msgr2=0x7fef1806e6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:49:38.507 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:38.506+0000 7fef1cadc640 1 -- 172.21.15.123:0/1095595125 shutdown_connections 2024-09-16T15:49:38.507 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:38.506+0000 7fef1cadc640 1 -- 172.21.15.123:0/1095595125 wait complete. 2024-09-16T15:49:38.522 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi123.front.sepia.ceph.com... 2024-09-16T15:49:38.522 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:49:38.523 DEBUG:teuthology.orchestra.run.smithi123:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2024-09-16T15:49:38.532 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi123.stderr:ceph-fuse[48125]: fuse finished with error 0 and tester_r 0 2024-09-16T15:49:38.533 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi123.front.sepia.ceph.com 2024-09-16T15:49:38.534 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:49:38.534 DEBUG:teuthology.orchestra.run.smithi123:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-09-16T15:49:38.637 INFO:tasks.cephfs.mount:Terminating background process 2024-09-16T15:49:38.637 DEBUG:teuthology.orchestra.run.smithi184:> set -ex 2024-09-16T15:49:38.637 DEBUG:teuthology.orchestra.run.smithi184:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-16T15:49:38.643 DEBUG:teuthology.orchestra.run.smithi184:> set -ex 2024-09-16T15:49:38.644 DEBUG:teuthology.orchestra.run.smithi184:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-16T15:49:38.690 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-09-16T15:49:38.816 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:38.814+0000 7f4972f4f640 1 -- 172.21.15.123:0/848130052 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f496c074470 msgr2=0x7f496c0cd6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:38.816 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:38.814+0000 7f4972f4f640 1 --2- 172.21.15.123:0/848130052 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f496c074470 0x7f496c0cd6e0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f4964009bc0 tx=0x7f496401cb10 comp rx=0 tx=0).stop 2024-09-16T15:49:38.816 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:38.814+0000 7f4972f4f640 1 -- 172.21.15.123:0/848130052 shutdown_connections 2024-09-16T15:49:38.816 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:38.814+0000 7f4972f4f640 1 --2- 172.21.15.123:0/848130052 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f496c074470 0x7f496c0cd6e0 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:38.816 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:38.814+0000 7f4972f4f640 1 --2- 172.21.15.123:0/848130052 >> v2:172.21.15.184:3300/0 conn(0x7f496c15e160 0x7f496c073f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:38.817 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:38.814+0000 7f4972f4f640 1 -- 172.21.15.123:0/848130052 >> 172.21.15.123:0/848130052 conn(0x7f496c0cfb10 msgr2=0x7f496c0cff20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:49:38.817 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:38.814+0000 7f4972f4f640 1 -- 172.21.15.123:0/848130052 shutdown_connections 2024-09-16T15:49:38.817 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:38.814+0000 7f4972f4f640 1 -- 172.21.15.123:0/848130052 wait complete. 2024-09-16T15:49:38.817 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:38.818+0000 7f4972f4f640 1 Processor -- start 2024-09-16T15:49:38.817 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:38.818+0000 7f4972f4f640 1 -- start start 2024-09-16T15:49:38.819 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:38.818+0000 7f4972f4f640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f496c074470 0x7f496c0ae820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:49:38.819 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:38.818+0000 7f4972f4f640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f496c0cdc20 0x7f496c152580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:49:38.819 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:38.818+0000 7f4972f4f640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f496c15e160 0x7f496c155460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:49:38.819 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:38.818+0000 7f4972f4f640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f496c167f50 con 0x7f496c15e160 2024-09-16T15:49:38.819 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:38.818+0000 7f4972f4f640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f496c167dd0 con 0x7f496c074470 2024-09-16T15:49:38.819 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:38.818+0000 7f4972f4f640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f496c1680d0 con 0x7f496c0cdc20 2024-09-16T15:49:38.819 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:38.818+0000 7f49714ec640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f496c15e160 0x7f496c155460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:49:38.820 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:38.818+0000 7f49714ec640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f496c15e160 0x7f496c155460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:56752/0 (socket says 172.21.15.123:56752) 2024-09-16T15:49:38.820 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:38.818+0000 7f49714ec640 1 -- 172.21.15.123:0/3636848688 learned_addr learned my addr 172.21.15.123:0/3636848688 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:49:38.820 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:38.818+0000 7f496bfff640 1 --2- 172.21.15.123:0/3636848688 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f496c0cdc20 0x7f496c152580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:49:38.820 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:38.818+0000 7f4970ceb640 1 --2- 172.21.15.123:0/3636848688 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f496c074470 0x7f496c0ae820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:49:38.820 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:38.818+0000 7f49714ec640 1 -- 172.21.15.123:0/3636848688 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f496c0cdc20 msgr2=0x7f496c152580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:38.820 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:38.818+0000 7f49714ec640 1 --2- 172.21.15.123:0/3636848688 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f496c0cdc20 0x7f496c152580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:38.820 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:38.818+0000 7f49714ec640 1 -- 172.21.15.123:0/3636848688 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f496c074470 msgr2=0x7f496c0ae820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:38.820 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:38.818+0000 7f49714ec640 1 --2- 172.21.15.123:0/3636848688 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f496c074470 0x7f496c0ae820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:38.820 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:38.818+0000 7f49714ec640 1 -- 172.21.15.123:0/3636848688 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f496c152c40 con 0x7f496c15e160 2024-09-16T15:49:38.820 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:38.818+0000 7f49714ec640 1 --2- 172.21.15.123:0/3636848688 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f496c15e160 0x7f496c155460 secure :-1 s=READY pgs=1326 cs=0 l=1 rev1=1 crypto rx=0x7f495400c470 tx=0x7f495400c930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:49:38.820 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:38.818+0000 7f49697fa640 1 -- 172.21.15.123:0/3636848688 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4954016020 con 0x7f496c15e160 2024-09-16T15:49:38.820 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:38.818+0000 7f4972f4f640 1 -- 172.21.15.123:0/3636848688 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f496c152f20 con 0x7f496c15e160 2024-09-16T15:49:38.820 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:38.818+0000 7f49697fa640 1 -- 172.21.15.123:0/3636848688 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f4954012920 con 0x7f496c15e160 2024-09-16T15:49:38.821 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:38.818+0000 7f49697fa640 1 -- 172.21.15.123:0/3636848688 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4954012d30 con 0x7f496c15e160 2024-09-16T15:49:38.821 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:38.818+0000 7f4972f4f640 1 -- 172.21.15.123:0/3636848688 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f496c1534e0 con 0x7f496c15e160 2024-09-16T15:49:38.822 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:38.818+0000 7f4972f4f640 1 -- 172.21.15.123:0/3636848688 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f496c0cd6e0 con 0x7f496c15e160 2024-09-16T15:49:38.822 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:38.822+0000 7f49697fa640 1 -- 172.21.15.123:0/3636848688 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 21) ==== 97641+0+0 (secure 0 0 0) 0x7f4954019020 con 0x7f496c15e160 2024-09-16T15:49:38.822 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:38.822+0000 7f49697fa640 1 --2- 172.21.15.123:0/3636848688 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f4950073520 0x7f49500759e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:49:38.822 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:38.822+0000 7f4970ceb640 1 --2- 172.21.15.123:0/3636848688 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f4950073520 0x7f49500759e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:49:38.822 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:38.822+0000 7f49697fa640 1 -- 172.21.15.123:0/3636848688 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(123..123 src has 1..123) ==== 5867+0+0 (secure 0 0 0) 0x7f4954094710 con 0x7f496c15e160 2024-09-16T15:49:38.822 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:38.822+0000 7f4970ceb640 1 --2- 172.21.15.123:0/3636848688 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f4950073520 0x7f49500759e0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f495c009ad0 tx=0x7f495c0091e0 comp rx=0 tx=0).ready entity=mgr.4150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:49:38.825 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:38.826+0000 7f49697fa640 1 -- 172.21.15.123:0/3636848688 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f4954020030 con 0x7f496c15e160 2024-09-16T15:49:39.003 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:39.002+0000 7f4972f4f640 1 -- 172.21.15.123:0/3636848688 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f496c0765d0 con 0x7f496c15e160 2024-09-16T15:49:39.004 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:39.002+0000 7f49697fa640 1 -- 172.21.15.123:0/3636848688 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v123) ==== 80+0+0 (secure 0 0 0) 0x7f495405eef0 con 0x7f496c15e160 2024-09-16T15:49:39.004 INFO:teuthology.orchestra.run.smithi123.stderr:listed 0 entries 2024-09-16T15:49:39.006 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:39.006+0000 7f4972f4f640 1 -- 172.21.15.123:0/3636848688 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f4950073520 msgr2=0x7f49500759e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:39.006 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:39.006+0000 7f4972f4f640 1 --2- 172.21.15.123:0/3636848688 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f4950073520 0x7f49500759e0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f495c009ad0 tx=0x7f495c0091e0 comp rx=0 tx=0).stop 2024-09-16T15:49:39.006 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:39.006+0000 7f4972f4f640 1 -- 172.21.15.123:0/3636848688 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f496c15e160 msgr2=0x7f496c155460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:39.006 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:39.006+0000 7f4972f4f640 1 --2- 172.21.15.123:0/3636848688 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f496c15e160 0x7f496c155460 secure :-1 s=READY pgs=1326 cs=0 l=1 rev1=1 crypto rx=0x7f495400c470 tx=0x7f495400c930 comp rx=0 tx=0).stop 2024-09-16T15:49:39.006 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:39.006+0000 7f4972f4f640 1 -- 172.21.15.123:0/3636848688 shutdown_connections 2024-09-16T15:49:39.006 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:39.006+0000 7f4972f4f640 1 --2- 172.21.15.123:0/3636848688 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f4950073520 0x7f49500759e0 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:39.006 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:39.006+0000 7f4972f4f640 1 --2- 172.21.15.123:0/3636848688 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f496c15e160 0x7f496c155460 unknown :-1 s=CLOSED pgs=1326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:39.006 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:39.006+0000 7f4972f4f640 1 --2- 172.21.15.123:0/3636848688 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f496c0cdc20 0x7f496c152580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:39.007 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:39.006+0000 7f4972f4f640 1 --2- 172.21.15.123:0/3636848688 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f496c074470 0x7f496c0ae820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:39.007 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:39.006+0000 7f4972f4f640 1 -- 172.21.15.123:0/3636848688 >> 172.21.15.123:0/3636848688 conn(0x7f496c0cfb10 msgr2=0x7f496c0758d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:49:39.007 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:39.006+0000 7f4972f4f640 1 -- 172.21.15.123:0/3636848688 shutdown_connections 2024-09-16T15:49:39.007 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:39.006+0000 7f4972f4f640 1 -- 172.21.15.123:0/3636848688 wait complete. 2024-09-16T15:49:39.023 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi184.front.sepia.ceph.com... 2024-09-16T15:49:39.023 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:49:39.024 DEBUG:teuthology.orchestra.run.smithi184:> sudo fusermount -u /home/ubuntu/cephtest/mnt.1 2024-09-16T15:49:39.036 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi184.stderr:ceph-fuse[35889]: fuse finished with error 0 and tester_r 0 2024-09-16T15:49:39.038 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi184.front.sepia.ceph.com 2024-09-16T15:49:39.038 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:49:39.039 DEBUG:teuthology.orchestra.run.smithi184:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2024-09-16T15:49:39.136 INFO:tasks.cephfs.mount:Terminating background process 2024-09-16T15:49:39.136 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2024-09-16T15:49:39.137 DEBUG:teuthology.orchestra.run.smithi123:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-16T15:49:39.143 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16336468k,nr_inodes=4084117,mode=755,inode64 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3274684k,mode=755,inode64 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev,inode64 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=17082 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3274680k,nr_inodes=818670,mode=700,uid=1000,gid=1271,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3274684k,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-09-16T15:49:39.143 INFO:tasks.cephfs.mount:Terminating background process 2024-09-16T15:49:39.143 DEBUG:teuthology.orchestra.run.smithi184:> set -ex 2024-09-16T15:49:39.144 DEBUG:teuthology.orchestra.run.smithi184:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-16T15:49:39.149 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16336488k,nr_inodes=4084122,mode=755,inode64 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3274688k,mode=755,inode64 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev,inode64 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=17834 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3274684k,nr_inodes=818671,mode=700,uid=1000,gid=1271,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3274688k,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-09-16T15:49:39.149 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-16T15:49:39.316 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:39.314+0000 7f77805f1640 1 -- 172.21.15.123:0/3603311164 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f777815f840 msgr2=0x7f7778073940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:39.316 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:39.314+0000 7f77805f1640 1 --2- 172.21.15.123:0/3603311164 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f777815f840 0x7f7778073940 secure :-1 s=READY pgs=1327 cs=0 l=1 rev1=1 crypto rx=0x7f7770009c80 tx=0x7f777001d7d0 comp rx=0 tx=0).stop 2024-09-16T15:49:39.317 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:39.314+0000 7f77805f1640 1 -- 172.21.15.123:0/3603311164 shutdown_connections 2024-09-16T15:49:39.317 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:39.314+0000 7f77805f1640 1 --2- 172.21.15.123:0/3603311164 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f777815f840 0x7f7778073940 unknown :-1 s=CLOSED pgs=1327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:39.317 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:39.314+0000 7f77805f1640 1 --2- 172.21.15.123:0/3603311164 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f777815bce0 0x7f777806cbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:39.317 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:39.314+0000 7f77805f1640 1 -- 172.21.15.123:0/3603311164 >> 172.21.15.123:0/3603311164 conn(0x7f77780d0b70 msgr2=0x7f77780d0f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:49:39.317 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:39.314+0000 7f77805f1640 1 -- 172.21.15.123:0/3603311164 shutdown_connections 2024-09-16T15:49:39.317 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:39.314+0000 7f77805f1640 1 -- 172.21.15.123:0/3603311164 wait complete. 2024-09-16T15:49:39.317 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:39.318+0000 7f77805f1640 1 Processor -- start 2024-09-16T15:49:39.318 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:39.318+0000 7f77805f1640 1 -- start start 2024-09-16T15:49:39.319 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:39.318+0000 7f77805f1640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7778070910 0x7f777815efe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:49:39.319 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:39.318+0000 7f77805f1640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f777815bce0 0x7f777815d220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:49:39.319 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:39.318+0000 7f77805f1640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f777815f840 0x7f77781fb230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:49:39.319 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:39.318+0000 7f77805f1640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f7778168d00 con 0x7f777815bce0 2024-09-16T15:49:39.319 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:39.318+0000 7f77805f1640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f7778168b80 con 0x7f7778070910 2024-09-16T15:49:39.319 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:39.318+0000 7f77805f1640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f7778168e80 con 0x7f777815f840 2024-09-16T15:49:39.320 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:39.318+0000 7f777db8c640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f777815bce0 0x7f777815d220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:49:39.320 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:39.318+0000 7f777db8c640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f777815bce0 0x7f777815d220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:56764/0 (socket says 172.21.15.123:56764) 2024-09-16T15:49:39.320 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:39.318+0000 7f777db8c640 1 -- 172.21.15.123:0/2095764402 learned_addr learned my addr 172.21.15.123:0/2095764402 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:49:39.320 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:39.318+0000 7f777e38d640 1 --2- 172.21.15.123:0/2095764402 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7778070910 0x7f777815efe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:49:39.320 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:39.318+0000 7f777eb8e640 1 --2- 172.21.15.123:0/2095764402 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f777815f840 0x7f77781fb230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:49:39.320 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:39.318+0000 7f777db8c640 1 -- 172.21.15.123:0/2095764402 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f777815f840 msgr2=0x7f77781fb230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:39.320 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:39.318+0000 7f777db8c640 1 --2- 172.21.15.123:0/2095764402 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f777815f840 0x7f77781fb230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:39.320 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:39.318+0000 7f777db8c640 1 -- 172.21.15.123:0/2095764402 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7778070910 msgr2=0x7f777815efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:39.320 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:39.318+0000 7f777db8c640 1 --2- 172.21.15.123:0/2095764402 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7778070910 0x7f777815efe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:39.320 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:39.318+0000 7f777db8c640 1 -- 172.21.15.123:0/2095764402 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f77780b1020 con 0x7f777815bce0 2024-09-16T15:49:39.320 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:39.318+0000 7f777db8c640 1 --2- 172.21.15.123:0/2095764402 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f777815bce0 0x7f777815d220 secure :-1 s=READY pgs=1328 cs=0 l=1 rev1=1 crypto rx=0x7f776800c430 tx=0x7f776800c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:49:39.320 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:39.318+0000 7f776effd640 1 -- 172.21.15.123:0/2095764402 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7768017070 con 0x7f777815bce0 2024-09-16T15:49:39.320 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:39.318+0000 7f776effd640 1 -- 172.21.15.123:0/2095764402 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f7768003600 con 0x7f777815bce0 2024-09-16T15:49:39.320 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:39.318+0000 7f77805f1640 1 -- 172.21.15.123:0/2095764402 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f77780b1300 con 0x7f777815bce0 2024-09-16T15:49:39.321 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:39.318+0000 7f776effd640 1 -- 172.21.15.123:0/2095764402 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7768003a50 con 0x7f777815bce0 2024-09-16T15:49:39.321 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:39.318+0000 7f77805f1640 1 -- 172.21.15.123:0/2095764402 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f777815c3b0 con 0x7f777815bce0 2024-09-16T15:49:39.321 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:39.318+0000 7f77805f1640 1 -- 172.21.15.123:0/2095764402 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7778073940 con 0x7f777815bce0 2024-09-16T15:49:39.322 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:39.322+0000 7f776effd640 1 -- 172.21.15.123:0/2095764402 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 21) ==== 97641+0+0 (secure 0 0 0) 0x7f776801a070 con 0x7f777815bce0 2024-09-16T15:49:39.322 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:39.322+0000 7f776effd640 1 --2- 172.21.15.123:0/2095764402 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f775c0734f0 0x7f775c0759b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:49:39.322 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:39.322+0000 7f776effd640 1 -- 172.21.15.123:0/2095764402 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(123..123 src has 1..123) ==== 5867+0+0 (secure 0 0 0) 0x7f7768094480 con 0x7f777815bce0 2024-09-16T15:49:39.322 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:39.322+0000 7f777e38d640 1 --2- 172.21.15.123:0/2095764402 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f775c0734f0 0x7f775c0759b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:49:39.322 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:39.322+0000 7f777e38d640 1 --2- 172.21.15.123:0/2095764402 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f775c0734f0 0x7f775c0759b0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f77740058d0 tx=0x7f7774005c70 comp rx=0 tx=0).ready entity=mgr.4150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:49:39.325 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:39.326+0000 7f776effd640 1 -- 172.21.15.123:0/2095764402 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f776805e720 con 0x7f777815bce0 2024-09-16T15:49:39.532 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:39.534+0000 7f77805f1640 1 -- 172.21.15.123:0/2095764402 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f77780a1f00 con 0x7f777815bce0 2024-09-16T15:49:39.534 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:39.534+0000 7f776effd640 1 -- 172.21.15.123:0/2095764402 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 77 v77) ==== 94+0+5234 (secure 0 0 0) 0x7f7768063150 con 0x7f777815bce0 2024-09-16T15:49:39.534 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-09-16T15:49:39.534 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":77,"btime":"2024-09-16T15:49:32:161575+0000","default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6079,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6833/3932980815","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":3932980815},{"type":"v1","addr":"172.21.15.184:6833","nonce":3932980815}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":72},{"gid":6082,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6835/1809728014","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":1809728014},{"type":"v1","addr":"172.21.15.184:6835","nonce":1809728014}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":72},{"gid":7551,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6834/892095590","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6832","nonce":892095590},{"type":"v1","addr":"172.21.15.123:6834","nonce":892095590}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":72}],"filesystems":[{"mdsmap":{"epoch":76,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-16T15:49:19.361823+0000","modified":"2024-09-16T15:49:22.409036+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7554},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7554":{"gid":7554,"name":"a","rank":0,"incarnation":73,"state":"up:active","state_seq":3,"addr":"172.21.15.123:6835/2110793616","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6833","nonce":2110793616},{"type":"v1","addr":"172.21.15.123:6835","nonce":2110793616}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7554,"qdb_cluster":[7554]},"id":9}]} 2024-09-16T15:49:39.534 INFO:teuthology.orchestra.run.smithi123.stderr:dumped fsmap epoch 77 2024-09-16T15:49:39.537 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:39.538+0000 7f77805f1640 1 -- 172.21.15.123:0/2095764402 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f775c0734f0 msgr2=0x7f775c0759b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:39.538 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:39.538+0000 7f77805f1640 1 --2- 172.21.15.123:0/2095764402 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f775c0734f0 0x7f775c0759b0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f77740058d0 tx=0x7f7774005c70 comp rx=0 tx=0).stop 2024-09-16T15:49:39.538 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:39.538+0000 7f77805f1640 1 -- 172.21.15.123:0/2095764402 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f777815bce0 msgr2=0x7f777815d220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:39.538 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:39.538+0000 7f77805f1640 1 --2- 172.21.15.123:0/2095764402 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f777815bce0 0x7f777815d220 secure :-1 s=READY pgs=1328 cs=0 l=1 rev1=1 crypto rx=0x7f776800c430 tx=0x7f776800c8f0 comp rx=0 tx=0).stop 2024-09-16T15:49:39.538 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:39.538+0000 7f77805f1640 1 -- 172.21.15.123:0/2095764402 shutdown_connections 2024-09-16T15:49:39.538 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:39.538+0000 7f77805f1640 1 --2- 172.21.15.123:0/2095764402 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f775c0734f0 0x7f775c0759b0 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:39.538 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:39.538+0000 7f77805f1640 1 --2- 172.21.15.123:0/2095764402 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f777815f840 0x7f77781fb230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:39.538 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:39.538+0000 7f77805f1640 1 --2- 172.21.15.123:0/2095764402 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f777815bce0 0x7f777815d220 unknown :-1 s=CLOSED pgs=1328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:39.538 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:39.538+0000 7f77805f1640 1 --2- 172.21.15.123:0/2095764402 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7778070910 0x7f777815efe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:39.538 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:39.538+0000 7f77805f1640 1 -- 172.21.15.123:0/2095764402 >> 172.21.15.123:0/2095764402 conn(0x7f77780d0b70 msgr2=0x7f77780d0f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:49:39.538 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:39.538+0000 7f77805f1640 1 -- 172.21.15.123:0/2095764402 shutdown_connections 2024-09-16T15:49:39.538 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:39.538+0000 7f77805f1640 1 -- 172.21.15.123:0/2095764402 wait complete. 2024-09-16T15:49:39.554 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-16T15:49:39.669 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:39.670+0000 7f14ca68c640 1 -- 172.21.15.123:0/1357571141 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f14c4074470 msgr2=0x7f14c40cd6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:39.669 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:39.670+0000 7f14ca68c640 1 --2- 172.21.15.123:0/1357571141 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f14c4074470 0x7f14c40cd6e0 secure :-1 s=READY pgs=1330 cs=0 l=1 rev1=1 crypto rx=0x7f14ac009900 tx=0x7f14ac01c850 comp rx=0 tx=0).stop 2024-09-16T15:49:39.669 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:39.670+0000 7f14ca68c640 1 -- 172.21.15.123:0/1357571141 shutdown_connections 2024-09-16T15:49:39.669 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:39.670+0000 7f14ca68c640 1 --2- 172.21.15.123:0/1357571141 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f14c4074470 0x7f14c40cd6e0 unknown :-1 s=CLOSED pgs=1330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:39.670 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:39.670+0000 7f14ca68c640 1 --2- 172.21.15.123:0/1357571141 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f14c415e160 0x7f14c4073f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:39.670 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:39.670+0000 7f14ca68c640 1 -- 172.21.15.123:0/1357571141 >> 172.21.15.123:0/1357571141 conn(0x7f14c40cfb10 msgr2=0x7f14c40cff20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:49:39.670 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:39.670+0000 7f14ca68c640 1 -- 172.21.15.123:0/1357571141 shutdown_connections 2024-09-16T15:49:39.670 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:39.670+0000 7f14ca68c640 1 -- 172.21.15.123:0/1357571141 wait complete. 2024-09-16T15:49:39.672 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:39.670+0000 7f14ca68c640 1 Processor -- start 2024-09-16T15:49:39.672 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:39.670+0000 7f14ca68c640 1 -- start start 2024-09-16T15:49:39.672 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:39.670+0000 7f14ca68c640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f14c4074470 0x7f14c415d340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:49:39.672 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:39.670+0000 7f14ca68c640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f14c40cdc20 0x7f14c40c4000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:49:39.672 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:39.670+0000 7f14ca68c640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f14c415e160 0x7f14c406eee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:49:39.672 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:39.670+0000 7f14ca68c640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f14c4167f50 con 0x7f14c40cdc20 2024-09-16T15:49:39.672 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:39.670+0000 7f14ca68c640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f14c4167dd0 con 0x7f14c415e160 2024-09-16T15:49:39.672 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:39.670+0000 7f14ca68c640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f14c41680d0 con 0x7f14c4074470 2024-09-16T15:49:39.673 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:39.670+0000 7f14bbfff640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f14c40cdc20 0x7f14c40c4000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:49:39.673 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:39.670+0000 7f14bbfff640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f14c40cdc20 0x7f14c40c4000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:56784/0 (socket says 172.21.15.123:56784) 2024-09-16T15:49:39.673 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:39.670+0000 7f14bbfff640 1 -- 172.21.15.123:0/3955515980 learned_addr learned my addr 172.21.15.123:0/3955515980 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:49:39.673 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:39.670+0000 7f14c3fff640 1 --2- 172.21.15.123:0/3955515980 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f14c4074470 0x7f14c415d340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:49:39.673 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:39.670+0000 7f14c8c29640 1 --2- 172.21.15.123:0/3955515980 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f14c415e160 0x7f14c406eee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:49:39.673 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:39.670+0000 7f14bbfff640 1 -- 172.21.15.123:0/3955515980 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f14c4074470 msgr2=0x7f14c415d340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:39.673 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:39.670+0000 7f14bbfff640 1 --2- 172.21.15.123:0/3955515980 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f14c4074470 0x7f14c415d340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:39.673 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:39.670+0000 7f14bbfff640 1 -- 172.21.15.123:0/3955515980 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f14c415e160 msgr2=0x7f14c406eee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:39.673 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:39.670+0000 7f14bbfff640 1 --2- 172.21.15.123:0/3955515980 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f14c415e160 0x7f14c406eee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:39.673 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:39.670+0000 7f14bbfff640 1 -- 172.21.15.123:0/3955515980 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f14c40c4c00 con 0x7f14c40cdc20 2024-09-16T15:49:39.673 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:39.670+0000 7f14bbfff640 1 --2- 172.21.15.123:0/3955515980 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f14c40cdc20 0x7f14c40c4000 secure :-1 s=READY pgs=1331 cs=0 l=1 rev1=1 crypto rx=0x7f14ac009850 tx=0x7f14ac0a57b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:49:39.673 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:39.670+0000 7f14c17fa640 1 -- 172.21.15.123:0/3955515980 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14ac0b2040 con 0x7f14c40cdc20 2024-09-16T15:49:39.674 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:39.670+0000 7f14c17fa640 1 -- 172.21.15.123:0/3955515980 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f14ac004480 con 0x7f14c40cdc20 2024-09-16T15:49:39.674 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:39.670+0000 7f14ca68c640 1 -- 172.21.15.123:0/3955515980 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f14c41ff050 con 0x7f14c40cdc20 2024-09-16T15:49:39.674 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:39.670+0000 7f14c17fa640 1 -- 172.21.15.123:0/3955515980 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14ac005290 con 0x7f14c40cdc20 2024-09-16T15:49:39.674 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:39.670+0000 7f14ca68c640 1 -- 172.21.15.123:0/3955515980 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f14c41ff610 con 0x7f14c40cdc20 2024-09-16T15:49:39.675 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:39.674+0000 7f14ca68c640 1 -- 172.21.15.123:0/3955515980 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f14c40cd6e0 con 0x7f14c40cdc20 2024-09-16T15:49:39.675 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:39.674+0000 7f14c17fa640 1 -- 172.21.15.123:0/3955515980 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 21) ==== 97641+0+0 (secure 0 0 0) 0x7f14ac005430 con 0x7f14c40cdc20 2024-09-16T15:49:39.675 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:39.674+0000 7f14c17fa640 1 --2- 172.21.15.123:0/3955515980 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f14a40734f0 0x7f14a40759b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:49:39.675 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:39.674+0000 7f14c17fa640 1 -- 172.21.15.123:0/3955515980 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(123..123 src has 1..123) ==== 5867+0+0 (secure 0 0 0) 0x7f14ac12e370 con 0x7f14c40cdc20 2024-09-16T15:49:39.675 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:39.674+0000 7f14c3fff640 1 --2- 172.21.15.123:0/3955515980 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f14a40734f0 0x7f14a40759b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:49:39.675 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:39.674+0000 7f14c3fff640 1 --2- 172.21.15.123:0/3955515980 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f14a40734f0 0x7f14a40759b0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f14b40058d0 tx=0x7f14b4005860 comp rx=0 tx=0).ready entity=mgr.4150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:49:39.678 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:39.678+0000 7f14c17fa640 1 -- 172.21.15.123:0/3955515980 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f14ac0164d0 con 0x7f14c40cdc20 2024-09-16T15:49:39.870 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:39.870+0000 7f14ca68c640 1 -- 172.21.15.123:0/3955515980 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f14c40d06c0 con 0x7f14c40cdc20 2024-09-16T15:49:39.871 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:39.870+0000 7f14c17fa640 1 -- 172.21.15.123:0/3955515980 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 77 v77) ==== 94+0+5234 (secure 0 0 0) 0x7f14ac0f8c80 con 0x7f14c40cdc20 2024-09-16T15:49:39.871 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-09-16T15:49:39.871 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":77,"btime":"2024-09-16T15:49:32:161575+0000","default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6079,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6833/3932980815","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":3932980815},{"type":"v1","addr":"172.21.15.184:6833","nonce":3932980815}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":72},{"gid":6082,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6835/1809728014","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":1809728014},{"type":"v1","addr":"172.21.15.184:6835","nonce":1809728014}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":72},{"gid":7551,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6834/892095590","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6832","nonce":892095590},{"type":"v1","addr":"172.21.15.123:6834","nonce":892095590}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":72}],"filesystems":[{"mdsmap":{"epoch":76,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-16T15:49:19.361823+0000","modified":"2024-09-16T15:49:22.409036+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7554},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7554":{"gid":7554,"name":"a","rank":0,"incarnation":73,"state":"up:active","state_seq":3,"addr":"172.21.15.123:6835/2110793616","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6833","nonce":2110793616},{"type":"v1","addr":"172.21.15.123:6835","nonce":2110793616}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7554,"qdb_cluster":[7554]},"id":9}]} 2024-09-16T15:49:39.872 INFO:teuthology.orchestra.run.smithi123.stderr:dumped fsmap epoch 77 2024-09-16T15:49:39.873 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:39.874+0000 7f14ca68c640 1 -- 172.21.15.123:0/3955515980 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f14a40734f0 msgr2=0x7f14a40759b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:39.874 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:39.874+0000 7f14ca68c640 1 --2- 172.21.15.123:0/3955515980 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f14a40734f0 0x7f14a40759b0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f14b40058d0 tx=0x7f14b4005860 comp rx=0 tx=0).stop 2024-09-16T15:49:39.874 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:39.874+0000 7f14ca68c640 1 -- 172.21.15.123:0/3955515980 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f14c40cdc20 msgr2=0x7f14c40c4000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:39.874 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:39.874+0000 7f14ca68c640 1 --2- 172.21.15.123:0/3955515980 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f14c40cdc20 0x7f14c40c4000 secure :-1 s=READY pgs=1331 cs=0 l=1 rev1=1 crypto rx=0x7f14ac009850 tx=0x7f14ac0a57b0 comp rx=0 tx=0).stop 2024-09-16T15:49:39.874 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:39.874+0000 7f14ca68c640 1 -- 172.21.15.123:0/3955515980 shutdown_connections 2024-09-16T15:49:39.874 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:39.874+0000 7f14ca68c640 1 --2- 172.21.15.123:0/3955515980 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f14a40734f0 0x7f14a40759b0 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:39.874 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:39.874+0000 7f14ca68c640 1 --2- 172.21.15.123:0/3955515980 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f14c415e160 0x7f14c406eee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:39.874 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:39.874+0000 7f14ca68c640 1 --2- 172.21.15.123:0/3955515980 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f14c40cdc20 0x7f14c40c4000 unknown :-1 s=CLOSED pgs=1331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:39.874 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:39.874+0000 7f14ca68c640 1 --2- 172.21.15.123:0/3955515980 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f14c4074470 0x7f14c415d340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:39.874 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:39.874+0000 7f14ca68c640 1 -- 172.21.15.123:0/3955515980 >> 172.21.15.123:0/3955515980 conn(0x7f14c40cfb10 msgr2=0x7f14c4151f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:49:39.874 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:39.874+0000 7f14ca68c640 1 -- 172.21.15.123:0/3955515980 shutdown_connections 2024-09-16T15:49:39.874 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:39.874+0000 7f14ca68c640 1 -- 172.21.15.123:0/3955515980 wait complete. 2024-09-16T15:49:39.889 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-16T15:49:39.889 DEBUG:teuthology.orchestra.run.smithi123:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-16T15:49:39.892 DEBUG:teuthology.orchestra.run.smithi184:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-16T15:49:40.010 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:40.006+0000 7f261e969640 1 -- 172.21.15.123:0/3612931367 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2604004d50 msgr2=0x7f26040051d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:40.010 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:40.006+0000 7f261e969640 1 --2- 172.21.15.123:0/3612931367 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2604004d50 0x7f26040051d0 secure :-1 s=READY pgs=1333 cs=0 l=1 rev1=1 crypto rx=0x7f2610009ec0 tx=0x7f261001d230 comp rx=0 tx=0).stop 2024-09-16T15:49:40.011 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:40.006+0000 7f261cf06640 1 -- 172.21.15.123:0/3612931367 reap_dead start 2024-09-16T15:49:40.011 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:40.006+0000 7f261e969640 1 -- 172.21.15.123:0/3612931367 shutdown_connections 2024-09-16T15:49:40.011 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:40.006+0000 7f261e969640 1 -- 172.21.15.123:0/3612931367 >> 172.21.15.123:0/3612931367 conn(0x7f26180d3e30 msgr2=0x7f26180d4240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:49:40.011 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:40.006+0000 7f261e969640 1 -- 172.21.15.123:0/3612931367 shutdown_connections 2024-09-16T15:49:40.011 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:40.006+0000 7f261e969640 1 -- 172.21.15.123:0/3612931367 wait complete. 2024-09-16T15:49:40.011 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:40.006+0000 7f261e969640 1 Processor -- start 2024-09-16T15:49:40.011 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:40.006+0000 7f261e969640 1 -- start start 2024-09-16T15:49:40.011 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:40.006+0000 7f261e969640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f2604004d50 0x7f261809edd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:49:40.011 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:40.006+0000 7f261e969640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f26181f86e0 0x7f26180c99c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:49:40.011 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:40.006+0000 7f261e969640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f261815d3c0 0x7f261806e020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:49:40.011 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:40.006+0000 7f261e969640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f26180b7620 con 0x7f26181f86e0 2024-09-16T15:49:40.011 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:40.006+0000 7f261e969640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f26180d4b60 con 0x7f261815d3c0 2024-09-16T15:49:40.012 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:40.006+0000 7f261e969640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f26180b77a0 con 0x7f2604004d50 2024-09-16T15:49:40.012 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:40.006+0000 7f260f7fe640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f26181f86e0 0x7f26180c99c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:49:40.012 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:40.006+0000 7f260f7fe640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f26181f86e0 0x7f26180c99c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:56812/0 (socket says 172.21.15.123:56812) 2024-09-16T15:49:40.012 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:40.006+0000 7f260f7fe640 1 -- 172.21.15.123:0/3453487760 learned_addr learned my addr 172.21.15.123:0/3453487760 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:49:40.012 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:40.006+0000 7f261cf06640 1 --2- 172.21.15.123:0/3453487760 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f261815d3c0 0x7f261806e020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:49:40.012 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:40.006+0000 7f2617fff640 1 --2- 172.21.15.123:0/3453487760 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f2604004d50 0x7f261809edd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:49:40.012 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:40.006+0000 7f260f7fe640 1 -- 172.21.15.123:0/3453487760 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f2604004d50 msgr2=0x7f261809edd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:40.012 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:40.010+0000 7f260f7fe640 1 --2- 172.21.15.123:0/3453487760 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f2604004d50 0x7f261809edd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:40.012 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:40.010+0000 7f260f7fe640 1 -- 172.21.15.123:0/3453487760 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f261815d3c0 msgr2=0x7f261806e020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:40.012 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:40.010+0000 7f260f7fe640 1 --2- 172.21.15.123:0/3453487760 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f261815d3c0 0x7f261806e020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:40.012 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:40.010+0000 7f260f7fe640 1 -- 172.21.15.123:0/3453487760 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f26180c9f00 con 0x7f26181f86e0 2024-09-16T15:49:40.012 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:40.010+0000 7f261cf06640 1 --2- 172.21.15.123:0/3453487760 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f261815d3c0 0x7f261806e020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T15:49:40.012 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:40.010+0000 7f260f7fe640 1 --2- 172.21.15.123:0/3453487760 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f26181f86e0 0x7f26180c99c0 secure :-1 s=READY pgs=1334 cs=0 l=1 rev1=1 crypto rx=0x7f2600011780 tx=0x7f2600011c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:49:40.013 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:40.010+0000 7f26157fa640 1 -- 172.21.15.123:0/3453487760 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2600004460 con 0x7f26181f86e0 2024-09-16T15:49:40.013 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:40.010+0000 7f26157fa640 1 -- 172.21.15.123:0/3453487760 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f2600016070 con 0x7f26181f86e0 2024-09-16T15:49:40.013 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:40.010+0000 7f261e969640 1 -- 172.21.15.123:0/3453487760 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f261809f460 con 0x7f26181f86e0 2024-09-16T15:49:40.013 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:40.010+0000 7f26157fa640 1 -- 172.21.15.123:0/3453487760 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f260000fb80 con 0x7f26181f86e0 2024-09-16T15:49:40.013 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:40.010+0000 7f261e969640 1 -- 172.21.15.123:0/3453487760 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f26181f7ee0 con 0x7f26181f86e0 2024-09-16T15:49:40.013 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:40.010+0000 7f261e969640 1 -- 172.21.15.123:0/3453487760 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f26180b7b20 con 0x7f26181f86e0 2024-09-16T15:49:40.014 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:40.010+0000 7f26157fa640 1 -- 172.21.15.123:0/3453487760 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 21) ==== 97641+0+0 (secure 0 0 0) 0x7f2600018030 con 0x7f26181f86e0 2024-09-16T15:49:40.014 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:40.010+0000 7f26157fa640 1 --2- 172.21.15.123:0/3453487760 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f25f80734f0 0x7f25f80759b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:49:40.014 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:40.010+0000 7f26157fa640 1 -- 172.21.15.123:0/3453487760 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(123..123 src has 1..123) ==== 5867+0+0 (secure 0 0 0) 0x7f26000950f0 con 0x7f26181f86e0 2024-09-16T15:49:40.014 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:40.010+0000 7f2617fff640 1 --2- 172.21.15.123:0/3453487760 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f25f80734f0 0x7f25f80759b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:49:40.014 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:40.014+0000 7f2617fff640 1 --2- 172.21.15.123:0/3453487760 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f25f80734f0 0x7f25f80759b0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f2608001ec0 tx=0x7f2608006040 comp rx=0 tx=0).ready entity=mgr.4150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:49:40.017 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:40.018+0000 7f26157fa640 1 -- 172.21.15.123:0/3453487760 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f260005f8e0 con 0x7f26181f86e0 2024-09-16T15:49:40.182 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:40.182+0000 7f261e969640 1 -- 172.21.15.123:0/3453487760 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f26180cf780 con 0x7f26181f86e0 2024-09-16T15:49:40.184 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:40.182+0000 7f26157fa640 1 -- 172.21.15.123:0/3453487760 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v123) ==== 74+0+17075 (secure 0 0 0) 0x7f260005f1e0 con 0x7f26181f86e0 2024-09-16T15:49:40.185 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-09-16T15:49:40.185 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":123,"fsid":"56077f89-18c8-486f-ae7e-eecc0bb7822c","created":"2024-09-16T15:42:26.364892+0000","modified":"2024-09-16T15:49:38.214268+0000","last_up_change":"2024-09-16T15:42:38.707234+0000","last_in_change":"2024-09-16T15:42:29.423662+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":19,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-16T15:42:39.287764+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":18,"pool_name":"cephfs_metadata","create_time":"2024-09-16T15:49:17.014388+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"118","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":19,"pool_name":"cephfs_data","create_time":"2024-09-16T15:49:18.311134+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"123","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":5,"snap_epoch":123,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9e7c1955-a607-4671-9be7-f2ef760ceeaa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":117,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6800","nonce":3882044969},{"type":"v1","addr":"172.21.15.123:6801","nonce":3882044969}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6802","nonce":3882044969},{"type":"v1","addr":"172.21.15.123:6803","nonce":3882044969}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6806","nonce":3882044969},{"type":"v1","addr":"172.21.15.123:6807","nonce":3882044969}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6804","nonce":3882044969},{"type":"v1","addr":"172.21.15.123:6805","nonce":3882044969}]},"public_addr":"172.21.15.123:6801/3882044969","cluster_addr":"172.21.15.123:6803/3882044969","heartbeat_back_addr":"172.21.15.123:6807/3882044969","heartbeat_front_addr":"172.21.15.123:6805/3882044969","state":["exists","up"]},{"osd":1,"uuid":"c540cfde-0977-4742-9b17-fd634668e010","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":117,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6816","nonce":4037001850},{"type":"v1","addr":"172.21.15.123:6817","nonce":4037001850}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6818","nonce":4037001850},{"type":"v1","addr":"172.21.15.123:6819","nonce":4037001850}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6822","nonce":4037001850},{"type":"v1","addr":"172.21.15.123:6823","nonce":4037001850}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6820","nonce":4037001850},{"type":"v1","addr":"172.21.15.123:6821","nonce":4037001850}]},"public_addr":"172.21.15.123:6817/4037001850","cluster_addr":"172.21.15.123:6819/4037001850","heartbeat_back_addr":"172.21.15.123:6823/4037001850","heartbeat_front_addr":"172.21.15.123:6821/4037001850","state":["exists","up"]},{"osd":2,"uuid":"9864a496-f169-486b-95c4-1d59162793a7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":117,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6824","nonce":1332143088},{"type":"v1","addr":"172.21.15.123:6825","nonce":1332143088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6826","nonce":1332143088},{"type":"v1","addr":"172.21.15.123:6827","nonce":1332143088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6830","nonce":1332143088},{"type":"v1","addr":"172.21.15.123:6831","nonce":1332143088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6828","nonce":1332143088},{"type":"v1","addr":"172.21.15.123:6829","nonce":1332143088}]},"public_addr":"172.21.15.123:6825/1332143088","cluster_addr":"172.21.15.123:6827/1332143088","heartbeat_back_addr":"172.21.15.123:6831/1332143088","heartbeat_front_addr":"172.21.15.123:6829/1332143088","state":["exists","up"]},{"osd":3,"uuid":"c6d1f817-f074-4325-95e1-578eba13edda","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":117,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6808","nonce":988586998},{"type":"v1","addr":"172.21.15.123:6809","nonce":988586998}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6810","nonce":988586998},{"type":"v1","addr":"172.21.15.123:6811","nonce":988586998}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6814","nonce":988586998},{"type":"v1","addr":"172.21.15.123:6815","nonce":988586998}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6812","nonce":988586998},{"type":"v1","addr":"172.21.15.123:6813","nonce":988586998}]},"public_addr":"172.21.15.123:6809/988586998","cluster_addr":"172.21.15.123:6811/988586998","heartbeat_back_addr":"172.21.15.123:6815/988586998","heartbeat_front_addr":"172.21.15.123:6813/988586998","state":["exists","up"]},{"osd":4,"uuid":"200a8df4-50ef-40a5-b48a-58981269c086","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":117,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6816","nonce":3387813890},{"type":"v1","addr":"172.21.15.184:6817","nonce":3387813890}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6818","nonce":3387813890},{"type":"v1","addr":"172.21.15.184:6819","nonce":3387813890}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6822","nonce":3387813890},{"type":"v1","addr":"172.21.15.184:6823","nonce":3387813890}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6820","nonce":3387813890},{"type":"v1","addr":"172.21.15.184:6821","nonce":3387813890}]},"public_addr":"172.21.15.184:6817/3387813890","cluster_addr":"172.21.15.184:6819/3387813890","heartbeat_back_addr":"172.21.15.184:6823/3387813890","heartbeat_front_addr":"172.21.15.184:6821/3387813890","state":["exists","up"]},{"osd":5,"uuid":"32d5a900-da3d-412c-bc19-13861a2368d8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":120,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6808","nonce":2680916017},{"type":"v1","addr":"172.21.15.184:6809","nonce":2680916017}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6810","nonce":2680916017},{"type":"v1","addr":"172.21.15.184:6811","nonce":2680916017}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6814","nonce":2680916017},{"type":"v1","addr":"172.21.15.184:6815","nonce":2680916017}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6812","nonce":2680916017},{"type":"v1","addr":"172.21.15.184:6813","nonce":2680916017}]},"public_addr":"172.21.15.184:6809/2680916017","cluster_addr":"172.21.15.184:6811/2680916017","heartbeat_back_addr":"172.21.15.184:6815/2680916017","heartbeat_front_addr":"172.21.15.184:6813/2680916017","state":["exists","up"]},{"osd":6,"uuid":"cde6f39e-aa1b-4260-ac19-774ee6aaca72","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":117,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6824","nonce":2183120457},{"type":"v1","addr":"172.21.15.184:6825","nonce":2183120457}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6826","nonce":2183120457},{"type":"v1","addr":"172.21.15.184:6827","nonce":2183120457}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6830","nonce":2183120457},{"type":"v1","addr":"172.21.15.184:6831","nonce":2183120457}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6828","nonce":2183120457},{"type":"v1","addr":"172.21.15.184:6829","nonce":2183120457}]},"public_addr":"172.21.15.184:6825/2183120457","cluster_addr":"172.21.15.184:6827/2183120457","heartbeat_back_addr":"172.21.15.184:6831/2183120457","heartbeat_front_addr":"172.21.15.184:6829/2183120457","state":["exists","up"]},{"osd":7,"uuid":"4f381778-eee1-47c6-804f-df0e6b33e313","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":120,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6800","nonce":4102957285},{"type":"v1","addr":"172.21.15.184:6801","nonce":4102957285}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6802","nonce":4102957285},{"type":"v1","addr":"172.21.15.184:6803","nonce":4102957285}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6806","nonce":4102957285},{"type":"v1","addr":"172.21.15.184:6807","nonce":4102957285}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6804","nonce":4102957285},{"type":"v1","addr":"172.21.15.184:6805","nonce":4102957285}]},"public_addr":"172.21.15.184:6801/4102957285","cluster_addr":"172.21.15.184:6803/4102957285","heartbeat_back_addr":"172.21.15.184:6807/4102957285","heartbeat_front_addr":"172.21.15.184:6805/4102957285","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:35.972210+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.222007+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.198617+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.385663+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.358935+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.156066+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.752973+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:35.682846+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"19.3","mappings":[{"from":2,"to":0}]},{"pgid":"19.12","mappings":[{"from":2,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":17,"snaps":[{"begin":2,"length":4}]},{"pool":19,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[{"pool":19,"snaps":[{"begin":3,"length":1},{"begin":5,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T15:49:40.187 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:40.186+0000 7f261e969640 1 -- 172.21.15.123:0/3453487760 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f25f80734f0 msgr2=0x7f25f80759b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:40.187 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:40.186+0000 7f261e969640 1 --2- 172.21.15.123:0/3453487760 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f25f80734f0 0x7f25f80759b0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f2608001ec0 tx=0x7f2608006040 comp rx=0 tx=0).stop 2024-09-16T15:49:40.187 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:40.186+0000 7f261e969640 1 -- 172.21.15.123:0/3453487760 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f26181f86e0 msgr2=0x7f26180c99c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:40.187 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:40.186+0000 7f261e969640 1 --2- 172.21.15.123:0/3453487760 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f26181f86e0 0x7f26180c99c0 secure :-1 s=READY pgs=1334 cs=0 l=1 rev1=1 crypto rx=0x7f2600011780 tx=0x7f2600011c40 comp rx=0 tx=0).stop 2024-09-16T15:49:40.187 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:40.186+0000 7f261e969640 1 -- 172.21.15.123:0/3453487760 shutdown_connections 2024-09-16T15:49:40.187 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:40.186+0000 7f261e969640 1 --2- 172.21.15.123:0/3453487760 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f25f80734f0 0x7f25f80759b0 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:40.187 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:40.186+0000 7f261e969640 1 --2- 172.21.15.123:0/3453487760 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f261815d3c0 0x7f261806e020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:40.187 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:40.186+0000 7f261e969640 1 --2- 172.21.15.123:0/3453487760 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f26181f86e0 0x7f26180c99c0 unknown :-1 s=CLOSED pgs=1334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:40.187 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:40.186+0000 7f261e969640 1 --2- 172.21.15.123:0/3453487760 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f2604004d50 0x7f261809edd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:40.187 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:40.186+0000 7f261e969640 1 -- 172.21.15.123:0/3453487760 >> 172.21.15.123:0/3453487760 conn(0x7f26180d3e30 msgr2=0x7f26180d0560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:49:40.187 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:40.186+0000 7f261e969640 1 -- 172.21.15.123:0/3453487760 shutdown_connections 2024-09-16T15:49:40.188 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:40.186+0000 7f261e969640 1 -- 172.21.15.123:0/3453487760 wait complete. 2024-09-16T15:49:40.203 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-09-16T15:49:40.203 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-16T15:49:40.373 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:40.374+0000 7f036d94d640 1 -- 172.21.15.123:0/3702354728 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0350004810 msgr2=0x7f0350004c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:40.373 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:40.374+0000 7f036d94d640 1 --2- 172.21.15.123:0/3702354728 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0350004810 0x7f0350004c90 secure :-1 s=READY pgs=1336 cs=0 l=1 rev1=1 crypto rx=0x7f035c00c2e0 tx=0x7f035c01cf60 comp rx=0 tx=0).stop 2024-09-16T15:49:40.374 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:40.374+0000 7f036d94d640 1 -- 172.21.15.123:0/3702354728 shutdown_connections 2024-09-16T15:49:40.374 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:40.374+0000 7f036d94d640 1 --2- 172.21.15.123:0/3702354728 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0350004810 0x7f0350004c90 unknown :-1 s=CLOSED pgs=1336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:40.374 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:40.374+0000 7f036d94d640 1 -- 172.21.15.123:0/3702354728 >> 172.21.15.123:0/3702354728 conn(0x7f03680cfb10 msgr2=0x7f03680cff20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:49:40.374 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:40.374+0000 7f036d94d640 1 -- 172.21.15.123:0/3702354728 shutdown_connections 2024-09-16T15:49:40.374 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:40.374+0000 7f036d94d640 1 -- 172.21.15.123:0/3702354728 wait complete. 2024-09-16T15:49:40.374 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:40.374+0000 7f036d94d640 1 Processor -- start 2024-09-16T15:49:40.374 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:40.374+0000 7f036d94d640 1 -- start start 2024-09-16T15:49:40.376 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:40.374+0000 7f036d94d640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f0350004810 0x7f0368151d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:49:40.376 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:40.374+0000 7f036d94d640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f03680afc60 0x7f03680b4b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:49:40.376 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:40.374+0000 7f036d94d640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0368152540 0x7f03680b50b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:49:40.377 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:40.374+0000 7f036d94d640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f036806eee0 con 0x7f0368152540 2024-09-16T15:49:40.377 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:40.374+0000 7f036d94d640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f036806ed60 con 0x7f03680afc60 2024-09-16T15:49:40.377 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:40.374+0000 7f036d94d640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f036806f060 con 0x7f0350004810 2024-09-16T15:49:40.377 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:40.374+0000 7f03677fe640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0368152540 0x7f03680b50b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:49:40.377 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:40.374+0000 7f03677fe640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0368152540 0x7f03680b50b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:56824/0 (socket says 172.21.15.123:56824) 2024-09-16T15:49:40.377 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:40.374+0000 7f03677fe640 1 -- 172.21.15.123:0/2331956861 learned_addr learned my addr 172.21.15.123:0/2331956861 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:49:40.377 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:40.374+0000 7f0366ffd640 1 --2- 172.21.15.123:0/2331956861 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f0350004810 0x7f0368151d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:49:40.377 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:40.374+0000 7f03667fc640 1 --2- 172.21.15.123:0/2331956861 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f03680afc60 0x7f03680b4b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:49:40.377 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:40.374+0000 7f03677fe640 1 -- 172.21.15.123:0/2331956861 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f0350004810 msgr2=0x7f0368151d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:40.377 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:40.374+0000 7f03677fe640 1 --2- 172.21.15.123:0/2331956861 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f0350004810 0x7f0368151d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:40.377 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:40.374+0000 7f03677fe640 1 -- 172.21.15.123:0/2331956861 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f03680afc60 msgr2=0x7f03680b4b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:40.377 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:40.374+0000 7f03677fe640 1 --2- 172.21.15.123:0/2331956861 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f03680afc60 0x7f03680b4b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:40.378 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:40.374+0000 7f03677fe640 1 -- 172.21.15.123:0/2331956861 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f03680b0120 con 0x7f0368152540 2024-09-16T15:49:40.378 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:40.374+0000 7f03677fe640 1 --2- 172.21.15.123:0/2331956861 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0368152540 0x7f03680b50b0 secure :-1 s=READY pgs=1337 cs=0 l=1 rev1=1 crypto rx=0x7f0368076250 tx=0x7f036000dd90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:49:40.378 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:40.374+0000 7f034ffff640 1 -- 172.21.15.123:0/2331956861 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0360004b30 con 0x7f0368152540 2024-09-16T15:49:40.378 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:40.374+0000 7f034ffff640 1 -- 172.21.15.123:0/2331956861 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f0360004de0 con 0x7f0368152540 2024-09-16T15:49:40.378 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:40.374+0000 7f036d94d640 1 -- 172.21.15.123:0/2331956861 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f03680c9620 con 0x7f0368152540 2024-09-16T15:49:40.378 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:40.374+0000 7f034ffff640 1 -- 172.21.15.123:0/2331956861 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f036000bc80 con 0x7f0368152540 2024-09-16T15:49:40.378 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:40.374+0000 7f036d94d640 1 -- 172.21.15.123:0/2331956861 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f03680c9be0 con 0x7f0368152540 2024-09-16T15:49:40.379 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:40.378+0000 7f036d94d640 1 -- 172.21.15.123:0/2331956861 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f03680c9410 con 0x7f0368152540 2024-09-16T15:49:40.379 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:40.378+0000 7f034ffff640 1 -- 172.21.15.123:0/2331956861 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 21) ==== 97641+0+0 (secure 0 0 0) 0x7f036001b020 con 0x7f0368152540 2024-09-16T15:49:40.379 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:40.378+0000 7f034ffff640 1 --2- 172.21.15.123:0/2331956861 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f03540734f0 0x7f03540759b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:49:40.379 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:40.378+0000 7f034ffff640 1 -- 172.21.15.123:0/2331956861 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(123..123 src has 1..123) ==== 5867+0+0 (secure 0 0 0) 0x7f0360094660 con 0x7f0368152540 2024-09-16T15:49:40.379 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:40.378+0000 7f0366ffd640 1 --2- 172.21.15.123:0/2331956861 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f03540734f0 0x7f03540759b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:49:40.379 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:40.378+0000 7f0366ffd640 1 --2- 172.21.15.123:0/2331956861 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f03540734f0 0x7f03540759b0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f036806b850 tx=0x7f0358006b70 comp rx=0 tx=0).ready entity=mgr.4150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:49:40.382 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:40.382+0000 7f034ffff640 1 -- 172.21.15.123:0/2331956861 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f036005eec0 con 0x7f0368152540 2024-09-16T15:49:40.592 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:40.590+0000 7f036d94d640 1 -- 172.21.15.123:0/2331956861 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f03680d06c0 con 0x7f0368152540 2024-09-16T15:49:40.592 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:40.590+0000 7f034ffff640 1 -- 172.21.15.123:0/2331956861 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 77 v77) ==== 94+0+5234 (secure 0 0 0) 0x7f036005e7c0 con 0x7f0368152540 2024-09-16T15:49:40.593 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-09-16T15:49:40.593 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":77,"btime":"2024-09-16T15:49:32:161575+0000","default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6079,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6833/3932980815","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":3932980815},{"type":"v1","addr":"172.21.15.184:6833","nonce":3932980815}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":72},{"gid":6082,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6835/1809728014","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":1809728014},{"type":"v1","addr":"172.21.15.184:6835","nonce":1809728014}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":72},{"gid":7551,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6834/892095590","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6832","nonce":892095590},{"type":"v1","addr":"172.21.15.123:6834","nonce":892095590}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":72}],"filesystems":[{"mdsmap":{"epoch":76,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-16T15:49:19.361823+0000","modified":"2024-09-16T15:49:22.409036+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7554},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7554":{"gid":7554,"name":"a","rank":0,"incarnation":73,"state":"up:active","state_seq":3,"addr":"172.21.15.123:6835/2110793616","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6833","nonce":2110793616},{"type":"v1","addr":"172.21.15.123:6835","nonce":2110793616}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7554,"qdb_cluster":[7554]},"id":9}]} 2024-09-16T15:49:40.593 INFO:teuthology.orchestra.run.smithi123.stderr:dumped fsmap epoch 77 2024-09-16T15:49:40.595 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:40.594+0000 7f036d94d640 1 -- 172.21.15.123:0/2331956861 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f03540734f0 msgr2=0x7f03540759b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:40.595 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:40.594+0000 7f036d94d640 1 --2- 172.21.15.123:0/2331956861 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f03540734f0 0x7f03540759b0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f036806b850 tx=0x7f0358006b70 comp rx=0 tx=0).stop 2024-09-16T15:49:40.595 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:40.594+0000 7f036d94d640 1 -- 172.21.15.123:0/2331956861 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0368152540 msgr2=0x7f03680b50b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:40.595 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:40.594+0000 7f036d94d640 1 --2- 172.21.15.123:0/2331956861 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0368152540 0x7f03680b50b0 secure :-1 s=READY pgs=1337 cs=0 l=1 rev1=1 crypto rx=0x7f0368076250 tx=0x7f036000dd90 comp rx=0 tx=0).stop 2024-09-16T15:49:40.595 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:40.594+0000 7f036d94d640 1 -- 172.21.15.123:0/2331956861 shutdown_connections 2024-09-16T15:49:40.595 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:40.594+0000 7f036d94d640 1 --2- 172.21.15.123:0/2331956861 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f03540734f0 0x7f03540759b0 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:40.595 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:40.594+0000 7f036d94d640 1 --2- 172.21.15.123:0/2331956861 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0368152540 0x7f03680b50b0 unknown :-1 s=CLOSED pgs=1337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:40.595 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:40.594+0000 7f036d94d640 1 --2- 172.21.15.123:0/2331956861 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f03680afc60 0x7f03680b4b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:40.595 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:40.594+0000 7f036d94d640 1 --2- 172.21.15.123:0/2331956861 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f0350004810 0x7f0368151d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:40.595 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:40.594+0000 7f036d94d640 1 -- 172.21.15.123:0/2331956861 >> 172.21.15.123:0/2331956861 conn(0x7f03680cfb10 msgr2=0x7f036815cf60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:49:40.595 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:40.594+0000 7f036d94d640 1 -- 172.21.15.123:0/2331956861 shutdown_connections 2024-09-16T15:49:40.595 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:40.594+0000 7f036d94d640 1 -- 172.21.15.123:0/2331956861 wait complete. 2024-09-16T15:49:40.611 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-16T15:49:40.739 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:40.738+0000 7fb3631d7640 1 -- 172.21.15.123:0/99255393 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb35c0cd070 msgr2=0x7fb35c0c74d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:40.739 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:40.738+0000 7fb3631d7640 1 --2- 172.21.15.123:0/99255393 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb35c0cd070 0x7fb35c0c74d0 secure :-1 s=READY pgs=1338 cs=0 l=1 rev1=1 crypto rx=0x7fb35c073310 tx=0x7fb35801c880 comp rx=0 tx=0).stop 2024-09-16T15:49:40.739 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:40.738+0000 7fb3631d7640 1 -- 172.21.15.123:0/99255393 shutdown_connections 2024-09-16T15:49:40.739 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:40.738+0000 7fb3631d7640 1 --2- 172.21.15.123:0/99255393 >> v2:172.21.15.184:3300/0 conn(0x7fb35c0ca590 0x7fb35c0cb980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:40.739 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:40.738+0000 7fb3631d7640 1 --2- 172.21.15.123:0/99255393 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb35c0cd070 0x7fb35c0c74d0 unknown :-1 s=CLOSED pgs=1338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:40.739 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:40.738+0000 7fb3631d7640 1 -- 172.21.15.123:0/99255393 >> 172.21.15.123:0/99255393 conn(0x7fb35c074230 msgr2=0x7fb35c074640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:49:40.740 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:40.738+0000 7fb3631d7640 1 -- 172.21.15.123:0/99255393 shutdown_connections 2024-09-16T15:49:40.740 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:40.738+0000 7fb3631d7640 1 -- 172.21.15.123:0/99255393 wait complete. 2024-09-16T15:49:40.742 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:40.738+0000 7fb3631d7640 1 Processor -- start 2024-09-16T15:49:40.742 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:40.738+0000 7fb3631d7640 1 -- start start 2024-09-16T15:49:40.742 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:40.742+0000 7fb3631d7640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb35c0ca590 0x7fb35c1fc2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:49:40.742 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:40.742+0000 7fb3631d7640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fb35c0cc8b0 0x7fb35c0b3230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:49:40.742 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:40.742+0000 7fb3631d7640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb35c0cd070 0x7fb35c0c67d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:49:40.742 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:40.742+0000 7fb3631d7640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fb35c1681c0 con 0x7fb35c0ca590 2024-09-16T15:49:40.742 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:40.742+0000 7fb3631d7640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fb35c15fc70 con 0x7fb35c0cd070 2024-09-16T15:49:40.742 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:40.742+0000 7fb3631d7640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fb35c168340 con 0x7fb35c0cc8b0 2024-09-16T15:49:40.742 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:40.742+0000 7fb360f73640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb35c0ca590 0x7fb35c1fc2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:49:40.742 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:40.742+0000 7fb360f73640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb35c0ca590 0x7fb35c1fc2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:56852/0 (socket says 172.21.15.123:56852) 2024-09-16T15:49:40.742 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:40.742+0000 7fb360f73640 1 -- 172.21.15.123:0/3069469326 learned_addr learned my addr 172.21.15.123:0/3069469326 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:49:40.742 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:40.742+0000 7fb353fff640 1 --2- 172.21.15.123:0/3069469326 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fb35c0cc8b0 0x7fb35c0b3230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:49:40.742 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:40.742+0000 7fb361774640 1 --2- 172.21.15.123:0/3069469326 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb35c0cd070 0x7fb35c0c67d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:49:40.743 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:40.742+0000 7fb360f73640 1 -- 172.21.15.123:0/3069469326 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fb35c0cc8b0 msgr2=0x7fb35c0b3230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:40.743 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:40.742+0000 7fb360f73640 1 --2- 172.21.15.123:0/3069469326 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fb35c0cc8b0 0x7fb35c0b3230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:40.743 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:40.742+0000 7fb360f73640 1 -- 172.21.15.123:0/3069469326 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb35c0cd070 msgr2=0x7fb35c0c67d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:40.743 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:40.742+0000 7fb360f73640 1 --2- 172.21.15.123:0/3069469326 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb35c0cd070 0x7fb35c0c67d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:40.743 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:40.742+0000 7fb360f73640 1 -- 172.21.15.123:0/3069469326 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb35c1f81f0 con 0x7fb35c0ca590 2024-09-16T15:49:40.743 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:40.742+0000 7fb360f73640 1 --2- 172.21.15.123:0/3069469326 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb35c0ca590 0x7fb35c1fc2a0 secure :-1 s=READY pgs=1339 cs=0 l=1 rev1=1 crypto rx=0x7fb358000c10 tx=0x7fb3580a5780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:49:40.743 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:40.742+0000 7fb3517fa640 1 -- 172.21.15.123:0/3069469326 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3580b2040 con 0x7fb35c0ca590 2024-09-16T15:49:40.743 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:40.742+0000 7fb3517fa640 1 -- 172.21.15.123:0/3069469326 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fb3580044b0 con 0x7fb35c0ca590 2024-09-16T15:49:40.743 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:40.742+0000 7fb3517fa640 1 -- 172.21.15.123:0/3069469326 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb358003860 con 0x7fb35c0ca590 2024-09-16T15:49:40.743 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:40.742+0000 7fb3631d7640 1 -- 172.21.15.123:0/3069469326 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb35c1f8470 con 0x7fb35c0ca590 2024-09-16T15:49:40.743 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:40.742+0000 7fb3631d7640 1 -- 172.21.15.123:0/3069469326 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb35c1f8a30 con 0x7fb35c0ca590 2024-09-16T15:49:40.744 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:40.742+0000 7fb3631d7640 1 -- 172.21.15.123:0/3069469326 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb35c0b15d0 con 0x7fb35c0ca590 2024-09-16T15:49:40.744 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:40.742+0000 7fb3517fa640 1 -- 172.21.15.123:0/3069469326 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 21) ==== 97641+0+0 (secure 0 0 0) 0x7fb3580bf060 con 0x7fb35c0ca590 2024-09-16T15:49:40.744 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:40.742+0000 7fb3517fa640 1 --2- 172.21.15.123:0/3069469326 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fb3480734f0 0x7fb3480759b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:49:40.744 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:40.742+0000 7fb353fff640 1 --2- 172.21.15.123:0/3069469326 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fb3480734f0 0x7fb3480759b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:49:40.744 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:40.742+0000 7fb3517fa640 1 -- 172.21.15.123:0/3069469326 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(123..123 src has 1..123) ==== 5867+0+0 (secure 0 0 0) 0x7fb35812e5f0 con 0x7fb35c0ca590 2024-09-16T15:49:40.744 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:40.742+0000 7fb353fff640 1 --2- 172.21.15.123:0/3069469326 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fb3480734f0 0x7fb3480759b0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fb35c06e7e0 tx=0x7fb34c005c70 comp rx=0 tx=0).ready entity=mgr.4150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:49:40.748 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:40.746+0000 7fb3517fa640 1 -- 172.21.15.123:0/3069469326 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7fb35c1f81f0 con 0x7fb35c0ca590 2024-09-16T15:49:40.943 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:40.942+0000 7fb3631d7640 1 -- 172.21.15.123:0/3069469326 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb35c0c74b0 con 0x7fb35c0ca590 2024-09-16T15:49:40.944 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:40.942+0000 7fb3517fa640 1 -- 172.21.15.123:0/3069469326 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 77 v77) ==== 94+0+5234 (secure 0 0 0) 0x7fb358016470 con 0x7fb35c0ca590 2024-09-16T15:49:40.944 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-09-16T15:49:40.944 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":77,"btime":"2024-09-16T15:49:32:161575+0000","default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6079,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6833/3932980815","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":3932980815},{"type":"v1","addr":"172.21.15.184:6833","nonce":3932980815}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":72},{"gid":6082,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6835/1809728014","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":1809728014},{"type":"v1","addr":"172.21.15.184:6835","nonce":1809728014}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":72},{"gid":7551,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6834/892095590","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6832","nonce":892095590},{"type":"v1","addr":"172.21.15.123:6834","nonce":892095590}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":72}],"filesystems":[{"mdsmap":{"epoch":76,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-16T15:49:19.361823+0000","modified":"2024-09-16T15:49:22.409036+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7554},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7554":{"gid":7554,"name":"a","rank":0,"incarnation":73,"state":"up:active","state_seq":3,"addr":"172.21.15.123:6835/2110793616","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6833","nonce":2110793616},{"type":"v1","addr":"172.21.15.123:6835","nonce":2110793616}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7554,"qdb_cluster":[7554]},"id":9}]} 2024-09-16T15:49:40.944 INFO:teuthology.orchestra.run.smithi123.stderr:dumped fsmap epoch 77 2024-09-16T15:49:40.947 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:40.946+0000 7fb3631d7640 1 -- 172.21.15.123:0/3069469326 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fb3480734f0 msgr2=0x7fb3480759b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:40.947 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:40.946+0000 7fb3631d7640 1 --2- 172.21.15.123:0/3069469326 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fb3480734f0 0x7fb3480759b0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fb35c06e7e0 tx=0x7fb34c005c70 comp rx=0 tx=0).stop 2024-09-16T15:49:40.947 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:40.946+0000 7fb3631d7640 1 -- 172.21.15.123:0/3069469326 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb35c0ca590 msgr2=0x7fb35c1fc2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:40.947 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:40.946+0000 7fb3631d7640 1 --2- 172.21.15.123:0/3069469326 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb35c0ca590 0x7fb35c1fc2a0 secure :-1 s=READY pgs=1339 cs=0 l=1 rev1=1 crypto rx=0x7fb358000c10 tx=0x7fb3580a5780 comp rx=0 tx=0).stop 2024-09-16T15:49:40.947 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:40.946+0000 7fb3631d7640 1 -- 172.21.15.123:0/3069469326 shutdown_connections 2024-09-16T15:49:40.947 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:40.946+0000 7fb3631d7640 1 --2- 172.21.15.123:0/3069469326 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fb3480734f0 0x7fb3480759b0 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:40.947 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:40.946+0000 7fb3631d7640 1 --2- 172.21.15.123:0/3069469326 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb35c0cd070 0x7fb35c0c67d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:40.947 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:40.946+0000 7fb3631d7640 1 --2- 172.21.15.123:0/3069469326 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fb35c0cc8b0 0x7fb35c0b3230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:40.947 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:40.946+0000 7fb3631d7640 1 --2- 172.21.15.123:0/3069469326 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb35c0ca590 0x7fb35c1fc2a0 unknown :-1 s=CLOSED pgs=1339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:40.947 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:40.946+0000 7fb3631d7640 1 -- 172.21.15.123:0/3069469326 >> 172.21.15.123:0/3069469326 conn(0x7fb35c074230 msgr2=0x7fb35c15b830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:49:40.948 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:40.946+0000 7fb3631d7640 1 -- 172.21.15.123:0/3069469326 shutdown_connections 2024-09-16T15:49:40.948 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:40.946+0000 7fb3631d7640 1 -- 172.21.15.123:0/3069469326 wait complete. 2024-09-16T15:49:40.962 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-16T15:49:41.082 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:41.082+0000 7f7d8e3fe640 1 -- 172.21.15.123:0/1931693407 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7d8815bca0 msgr2=0x7f7d880d0630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:41.083 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:41.082+0000 7f7d8e3fe640 1 --2- 172.21.15.123:0/1931693407 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7d8815bca0 0x7f7d880d0630 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f7d78009bc0 tx=0x7f7d7801cae0 comp rx=0 tx=0).stop 2024-09-16T15:49:41.083 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:41.082+0000 7f7d8e3fe640 1 -- 172.21.15.123:0/1931693407 shutdown_connections 2024-09-16T15:49:41.083 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:41.082+0000 7f7d8e3fe640 1 --2- 172.21.15.123:0/1931693407 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f7d88074600 0x7f7d8806e750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:41.083 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:41.082+0000 7f7d8e3fe640 1 --2- 172.21.15.123:0/1931693407 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7d8815bca0 0x7f7d880d0630 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:41.083 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:41.082+0000 7f7d8e3fe640 1 -- 172.21.15.123:0/1931693407 >> 172.21.15.123:0/1931693407 conn(0x7f7d880cfb10 msgr2=0x7f7d880cff20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:49:41.083 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:41.082+0000 7f7d8e3fe640 1 -- 172.21.15.123:0/1931693407 shutdown_connections 2024-09-16T15:49:41.083 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:41.082+0000 7f7d8e3fe640 1 -- 172.21.15.123:0/1931693407 wait complete. 2024-09-16T15:49:41.084 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:41.082+0000 7f7d8e3fe640 1 Processor -- start 2024-09-16T15:49:41.084 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:41.082+0000 7f7d8e3fe640 1 -- start start 2024-09-16T15:49:41.086 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:41.082+0000 7f7d8e3fe640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f7d880726c0 0x7f7d880c74c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:49:41.086 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:41.082+0000 7f7d8e3fe640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7d88074600 0x7f7d88164e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:49:41.086 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:41.082+0000 7f7d8e3fe640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7d88165340 0x7f7d881fad30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:49:41.086 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:41.082+0000 7f7d8e3fe640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f7d88168bd0 con 0x7f7d88165340 2024-09-16T15:49:41.086 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:41.082+0000 7f7d8e3fe640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f7d88168a50 con 0x7f7d88074600 2024-09-16T15:49:41.086 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:41.082+0000 7f7d8e3fe640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f7d88168d50 con 0x7f7d880726c0 2024-09-16T15:49:41.086 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:41.082+0000 7f7d8c99b640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7d88165340 0x7f7d881fad30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:49:41.086 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:41.086+0000 7f7d8c99b640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7d88165340 0x7f7d881fad30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:56868/0 (socket says 172.21.15.123:56868) 2024-09-16T15:49:41.086 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:41.086+0000 7f7d8c99b640 1 -- 172.21.15.123:0/3483654926 learned_addr learned my addr 172.21.15.123:0/3483654926 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:49:41.086 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:41.086+0000 7f7d87fff640 1 --2- 172.21.15.123:0/3483654926 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f7d880726c0 0x7f7d880c74c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:49:41.086 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:41.086+0000 7f7d8c99b640 1 -- 172.21.15.123:0/3483654926 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f7d880726c0 msgr2=0x7f7d880c74c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:41.087 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:41.086+0000 7f7d8c99b640 1 --2- 172.21.15.123:0/3483654926 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f7d880726c0 0x7f7d880c74c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:41.087 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:41.086+0000 7f7d877fe640 1 --2- 172.21.15.123:0/3483654926 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7d88074600 0x7f7d88164e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:49:41.087 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:41.086+0000 7f7d8c99b640 1 -- 172.21.15.123:0/3483654926 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7d88074600 msgr2=0x7f7d88164e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:41.087 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:41.086+0000 7f7d8c99b640 1 --2- 172.21.15.123:0/3483654926 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7d88074600 0x7f7d88164e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:41.087 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:41.086+0000 7f7d8c99b640 1 -- 172.21.15.123:0/3483654926 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7d881fba60 con 0x7f7d88165340 2024-09-16T15:49:41.087 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:41.086+0000 7f7d87fff640 1 --2- 172.21.15.123:0/3483654926 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f7d880726c0 0x7f7d880c74c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T15:49:41.087 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:41.086+0000 7f7d8c99b640 1 --2- 172.21.15.123:0/3483654926 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7d88165340 0x7f7d881fad30 secure :-1 s=READY pgs=1340 cs=0 l=1 rev1=1 crypto rx=0x7f7d8000e740 tx=0x7f7d8000ec00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:49:41.087 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:41.086+0000 7f7d84ff9640 1 -- 172.21.15.123:0/3483654926 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d8000cc60 con 0x7f7d88165340 2024-09-16T15:49:41.087 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:41.086+0000 7f7d84ff9640 1 -- 172.21.15.123:0/3483654926 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f7d8000ce20 con 0x7f7d88165340 2024-09-16T15:49:41.087 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:41.086+0000 7f7d8e3fe640 1 -- 172.21.15.123:0/3483654926 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7d881f8450 con 0x7f7d88165340 2024-09-16T15:49:41.087 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:41.086+0000 7f7d84ff9640 1 -- 172.21.15.123:0/3483654926 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d80005280 con 0x7f7d88165340 2024-09-16T15:49:41.087 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:41.086+0000 7f7d8e3fe640 1 -- 172.21.15.123:0/3483654926 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7d881f7970 con 0x7f7d88165340 2024-09-16T15:49:41.087 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:41.086+0000 7f7d8e3fe640 1 -- 172.21.15.123:0/3483654926 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7d880d0630 con 0x7f7d88165340 2024-09-16T15:49:41.088 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:41.086+0000 7f7d84ff9640 1 -- 172.21.15.123:0/3483654926 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 21) ==== 97641+0+0 (secure 0 0 0) 0x7f7d80005420 con 0x7f7d88165340 2024-09-16T15:49:41.088 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:41.086+0000 7f7d84ff9640 1 --2- 172.21.15.123:0/3483654926 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f7d740734f0 0x7f7d740759b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:49:41.088 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:41.086+0000 7f7d84ff9640 1 -- 172.21.15.123:0/3483654926 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(123..123 src has 1..123) ==== 5867+0+0 (secure 0 0 0) 0x7f7d800953f0 con 0x7f7d88165340 2024-09-16T15:49:41.088 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:41.086+0000 7f7d87fff640 1 --2- 172.21.15.123:0/3483654926 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f7d740734f0 0x7f7d740759b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:49:41.088 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:41.086+0000 7f7d87fff640 1 --2- 172.21.15.123:0/3483654926 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f7d740734f0 0x7f7d740759b0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f7d780af000 tx=0x7f7d7801b320 comp rx=0 tx=0).ready entity=mgr.4150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:49:41.091 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:41.090+0000 7f7d84ff9640 1 -- 172.21.15.123:0/3483654926 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f7d8005fba0 con 0x7f7d88165340 2024-09-16T15:49:41.262 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:41.262+0000 7f7d8e3fe640 1 -- 172.21.15.123:0/3483654926 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f7d8815f630 con 0x7f7d88165340 2024-09-16T15:49:41.264 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:41.262+0000 7f7d84ff9640 1 -- 172.21.15.123:0/3483654926 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v124) ==== 74+0+17007 (secure 0 0 0) 0x7f7d8005f4a0 con 0x7f7d88165340 2024-09-16T15:49:41.264 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-09-16T15:49:41.264 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":124,"fsid":"56077f89-18c8-486f-ae7e-eecc0bb7822c","created":"2024-09-16T15:42:26.364892+0000","modified":"2024-09-16T15:49:41.094532+0000","last_up_change":"2024-09-16T15:42:38.707234+0000","last_in_change":"2024-09-16T15:42:29.423662+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":19,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-16T15:42:39.287764+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":18,"pool_name":"cephfs_metadata","create_time":"2024-09-16T15:49:17.014388+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"118","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":19,"pool_name":"cephfs_data","create_time":"2024-09-16T15:49:18.311134+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"123","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":5,"snap_epoch":123,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9e7c1955-a607-4671-9be7-f2ef760ceeaa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":117,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6800","nonce":3882044969},{"type":"v1","addr":"172.21.15.123:6801","nonce":3882044969}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6802","nonce":3882044969},{"type":"v1","addr":"172.21.15.123:6803","nonce":3882044969}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6806","nonce":3882044969},{"type":"v1","addr":"172.21.15.123:6807","nonce":3882044969}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6804","nonce":3882044969},{"type":"v1","addr":"172.21.15.123:6805","nonce":3882044969}]},"public_addr":"172.21.15.123:6801/3882044969","cluster_addr":"172.21.15.123:6803/3882044969","heartbeat_back_addr":"172.21.15.123:6807/3882044969","heartbeat_front_addr":"172.21.15.123:6805/3882044969","state":["exists","up"]},{"osd":1,"uuid":"c540cfde-0977-4742-9b17-fd634668e010","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":117,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6816","nonce":4037001850},{"type":"v1","addr":"172.21.15.123:6817","nonce":4037001850}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6818","nonce":4037001850},{"type":"v1","addr":"172.21.15.123:6819","nonce":4037001850}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6822","nonce":4037001850},{"type":"v1","addr":"172.21.15.123:6823","nonce":4037001850}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6820","nonce":4037001850},{"type":"v1","addr":"172.21.15.123:6821","nonce":4037001850}]},"public_addr":"172.21.15.123:6817/4037001850","cluster_addr":"172.21.15.123:6819/4037001850","heartbeat_back_addr":"172.21.15.123:6823/4037001850","heartbeat_front_addr":"172.21.15.123:6821/4037001850","state":["exists","up"]},{"osd":2,"uuid":"9864a496-f169-486b-95c4-1d59162793a7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":117,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6824","nonce":1332143088},{"type":"v1","addr":"172.21.15.123:6825","nonce":1332143088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6826","nonce":1332143088},{"type":"v1","addr":"172.21.15.123:6827","nonce":1332143088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6830","nonce":1332143088},{"type":"v1","addr":"172.21.15.123:6831","nonce":1332143088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6828","nonce":1332143088},{"type":"v1","addr":"172.21.15.123:6829","nonce":1332143088}]},"public_addr":"172.21.15.123:6825/1332143088","cluster_addr":"172.21.15.123:6827/1332143088","heartbeat_back_addr":"172.21.15.123:6831/1332143088","heartbeat_front_addr":"172.21.15.123:6829/1332143088","state":["exists","up"]},{"osd":3,"uuid":"c6d1f817-f074-4325-95e1-578eba13edda","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":117,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6808","nonce":988586998},{"type":"v1","addr":"172.21.15.123:6809","nonce":988586998}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6810","nonce":988586998},{"type":"v1","addr":"172.21.15.123:6811","nonce":988586998}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6814","nonce":988586998},{"type":"v1","addr":"172.21.15.123:6815","nonce":988586998}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6812","nonce":988586998},{"type":"v1","addr":"172.21.15.123:6813","nonce":988586998}]},"public_addr":"172.21.15.123:6809/988586998","cluster_addr":"172.21.15.123:6811/988586998","heartbeat_back_addr":"172.21.15.123:6815/988586998","heartbeat_front_addr":"172.21.15.123:6813/988586998","state":["exists","up"]},{"osd":4,"uuid":"200a8df4-50ef-40a5-b48a-58981269c086","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":117,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6816","nonce":3387813890},{"type":"v1","addr":"172.21.15.184:6817","nonce":3387813890}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6818","nonce":3387813890},{"type":"v1","addr":"172.21.15.184:6819","nonce":3387813890}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6822","nonce":3387813890},{"type":"v1","addr":"172.21.15.184:6823","nonce":3387813890}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6820","nonce":3387813890},{"type":"v1","addr":"172.21.15.184:6821","nonce":3387813890}]},"public_addr":"172.21.15.184:6817/3387813890","cluster_addr":"172.21.15.184:6819/3387813890","heartbeat_back_addr":"172.21.15.184:6823/3387813890","heartbeat_front_addr":"172.21.15.184:6821/3387813890","state":["exists","up"]},{"osd":5,"uuid":"32d5a900-da3d-412c-bc19-13861a2368d8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":120,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6808","nonce":2680916017},{"type":"v1","addr":"172.21.15.184:6809","nonce":2680916017}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6810","nonce":2680916017},{"type":"v1","addr":"172.21.15.184:6811","nonce":2680916017}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6814","nonce":2680916017},{"type":"v1","addr":"172.21.15.184:6815","nonce":2680916017}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6812","nonce":2680916017},{"type":"v1","addr":"172.21.15.184:6813","nonce":2680916017}]},"public_addr":"172.21.15.184:6809/2680916017","cluster_addr":"172.21.15.184:6811/2680916017","heartbeat_back_addr":"172.21.15.184:6815/2680916017","heartbeat_front_addr":"172.21.15.184:6813/2680916017","state":["exists","up"]},{"osd":6,"uuid":"cde6f39e-aa1b-4260-ac19-774ee6aaca72","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":117,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6824","nonce":2183120457},{"type":"v1","addr":"172.21.15.184:6825","nonce":2183120457}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6826","nonce":2183120457},{"type":"v1","addr":"172.21.15.184:6827","nonce":2183120457}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6830","nonce":2183120457},{"type":"v1","addr":"172.21.15.184:6831","nonce":2183120457}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6828","nonce":2183120457},{"type":"v1","addr":"172.21.15.184:6829","nonce":2183120457}]},"public_addr":"172.21.15.184:6825/2183120457","cluster_addr":"172.21.15.184:6827/2183120457","heartbeat_back_addr":"172.21.15.184:6831/2183120457","heartbeat_front_addr":"172.21.15.184:6829/2183120457","state":["exists","up"]},{"osd":7,"uuid":"4f381778-eee1-47c6-804f-df0e6b33e313","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":120,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6800","nonce":4102957285},{"type":"v1","addr":"172.21.15.184:6801","nonce":4102957285}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6802","nonce":4102957285},{"type":"v1","addr":"172.21.15.184:6803","nonce":4102957285}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6806","nonce":4102957285},{"type":"v1","addr":"172.21.15.184:6807","nonce":4102957285}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6804","nonce":4102957285},{"type":"v1","addr":"172.21.15.184:6805","nonce":4102957285}]},"public_addr":"172.21.15.184:6801/4102957285","cluster_addr":"172.21.15.184:6803/4102957285","heartbeat_back_addr":"172.21.15.184:6807/4102957285","heartbeat_front_addr":"172.21.15.184:6805/4102957285","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:35.972210+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.222007+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.198617+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.385663+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.358935+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.156066+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.752973+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:35.682846+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"19.3","mappings":[{"from":2,"to":0}]},{"pgid":"19.12","mappings":[{"from":2,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":17,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[{"pool":19,"snaps":[{"begin":2,"length":4}]}],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T15:49:41.266 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:41.266+0000 7f7d8e3fe640 1 -- 172.21.15.123:0/3483654926 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f7d740734f0 msgr2=0x7f7d740759b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:41.266 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:41.266+0000 7f7d8e3fe640 1 --2- 172.21.15.123:0/3483654926 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f7d740734f0 0x7f7d740759b0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f7d780af000 tx=0x7f7d7801b320 comp rx=0 tx=0).stop 2024-09-16T15:49:41.266 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:41.266+0000 7f7d8e3fe640 1 -- 172.21.15.123:0/3483654926 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7d88165340 msgr2=0x7f7d881fad30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:41.266 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:41.266+0000 7f7d8e3fe640 1 --2- 172.21.15.123:0/3483654926 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7d88165340 0x7f7d881fad30 secure :-1 s=READY pgs=1340 cs=0 l=1 rev1=1 crypto rx=0x7f7d8000e740 tx=0x7f7d8000ec00 comp rx=0 tx=0).stop 2024-09-16T15:49:41.266 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:41.266+0000 7f7d8e3fe640 1 -- 172.21.15.123:0/3483654926 shutdown_connections 2024-09-16T15:49:41.266 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:41.266+0000 7f7d8e3fe640 1 --2- 172.21.15.123:0/3483654926 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f7d740734f0 0x7f7d740759b0 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:41.266 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:41.266+0000 7f7d8e3fe640 1 --2- 172.21.15.123:0/3483654926 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7d88165340 0x7f7d881fad30 unknown :-1 s=CLOSED pgs=1340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:41.267 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:41.266+0000 7f7d8e3fe640 1 --2- 172.21.15.123:0/3483654926 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7d88074600 0x7f7d88164e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:41.267 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:41.266+0000 7f7d8e3fe640 1 --2- 172.21.15.123:0/3483654926 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f7d880726c0 0x7f7d880c74c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:41.267 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:41.266+0000 7f7d8e3fe640 1 -- 172.21.15.123:0/3483654926 >> 172.21.15.123:0/3483654926 conn(0x7f7d880cfb10 msgr2=0x7f7d880ca140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:49:41.267 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:41.266+0000 7f7d8e3fe640 1 -- 172.21.15.123:0/3483654926 shutdown_connections 2024-09-16T15:49:41.267 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:41.266+0000 7f7d8e3fe640 1 -- 172.21.15.123:0/3483654926 wait complete. 2024-09-16T15:49:41.281 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-09-16T15:49:41.403 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:41.402+0000 7f780ad03640 1 -- 172.21.15.123:0/437260984 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f77f0003a60 msgr2=0x7f77f0003e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:41.404 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:41.402+0000 7f780ad03640 1 --2- 172.21.15.123:0/437260984 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f77f0003a60 0x7f77f0003e80 secure :-1 s=READY pgs=1342 cs=0 l=1 rev1=1 crypto rx=0x7f77f4001060 tx=0x7f77f401cf00 comp rx=0 tx=0).stop 2024-09-16T15:49:41.404 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:41.402+0000 7f780ad03640 1 -- 172.21.15.123:0/437260984 shutdown_connections 2024-09-16T15:49:41.404 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:41.402+0000 7f780ad03640 1 --2- 172.21.15.123:0/437260984 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f77f0003a60 0x7f77f0003e80 unknown :-1 s=CLOSED pgs=1342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:41.404 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:41.402+0000 7f780ad03640 1 -- 172.21.15.123:0/437260984 >> 172.21.15.123:0/437260984 conn(0x7f7804069650 msgr2=0x7f780409d170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:49:41.404 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:41.402+0000 7f780ad03640 1 -- 172.21.15.123:0/437260984 shutdown_connections 2024-09-16T15:49:41.404 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:41.402+0000 7f780ad03640 1 -- 172.21.15.123:0/437260984 wait complete. 2024-09-16T15:49:41.406 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:41.402+0000 7f780ad03640 1 Processor -- start 2024-09-16T15:49:41.406 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:41.406+0000 7f780ad03640 1 -- start start 2024-09-16T15:49:41.406 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:41.406+0000 7f780ad03640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f77f0003a60 0x7f78040ca770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:49:41.406 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:41.406+0000 7f780ad03640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f78040caf80 0x7f78040b9190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:49:41.407 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:41.406+0000 7f780ad03640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f780414a270 0x7f780414a6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:49:41.407 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:41.406+0000 7f780ad03640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f78040d2200 con 0x7f780414a270 2024-09-16T15:49:41.407 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:41.406+0000 7f780ad03640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f78040d2080 con 0x7f78040caf80 2024-09-16T15:49:41.407 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:41.406+0000 7f780ad03640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f78040d2380 con 0x7f77f0003a60 2024-09-16T15:49:41.407 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:41.406+0000 7f78092a0640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f780414a270 0x7f780414a6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:49:41.407 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:41.406+0000 7f78092a0640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f780414a270 0x7f780414a6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:56896/0 (socket says 172.21.15.123:56896) 2024-09-16T15:49:41.407 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:41.406+0000 7f78092a0640 1 -- 172.21.15.123:0/306196919 learned_addr learned my addr 172.21.15.123:0/306196919 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:49:41.407 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:41.406+0000 7f7803fff640 1 --2- 172.21.15.123:0/306196919 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f78040caf80 0x7f78040b9190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:49:41.407 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:41.406+0000 7f7808a9f640 1 --2- 172.21.15.123:0/306196919 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f77f0003a60 0x7f78040ca770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:49:41.407 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:41.406+0000 7f78092a0640 1 -- 172.21.15.123:0/306196919 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f77f0003a60 msgr2=0x7f78040ca770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:41.407 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:41.406+0000 7f78092a0640 1 --2- 172.21.15.123:0/306196919 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f77f0003a60 0x7f78040ca770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:41.408 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:41.406+0000 7f78092a0640 1 -- 172.21.15.123:0/306196919 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f78040caf80 msgr2=0x7f78040b9190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:41.408 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:41.406+0000 7f78092a0640 1 --2- 172.21.15.123:0/306196919 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f78040caf80 0x7f78040b9190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:41.408 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:41.406+0000 7f78092a0640 1 -- 172.21.15.123:0/306196919 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f780414ac30 con 0x7f780414a270 2024-09-16T15:49:41.408 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:41.406+0000 7f78092a0640 1 --2- 172.21.15.123:0/306196919 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f780414a270 0x7f780414a6f0 secure :-1 s=READY pgs=1343 cs=0 l=1 rev1=1 crypto rx=0x7f78040d4a80 tx=0x7f77fc010f40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:49:41.408 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:41.406+0000 7f78017fa640 1 -- 172.21.15.123:0/306196919 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77fc004b30 con 0x7f780414a270 2024-09-16T15:49:41.408 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:41.406+0000 7f78017fa640 1 -- 172.21.15.123:0/306196919 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f77fc015070 con 0x7f780414a270 2024-09-16T15:49:41.408 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:41.406+0000 7f780ad03640 1 -- 172.21.15.123:0/306196919 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f78040cdeb0 con 0x7f780414a270 2024-09-16T15:49:41.408 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:41.406+0000 7f78017fa640 1 -- 172.21.15.123:0/306196919 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77fc003980 con 0x7f780414a270 2024-09-16T15:49:41.408 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:41.406+0000 7f780ad03640 1 -- 172.21.15.123:0/306196919 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f78040b8990 con 0x7f780414a270 2024-09-16T15:49:41.409 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:41.406+0000 7f780ad03640 1 -- 172.21.15.123:0/306196919 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f78040d28e0 con 0x7f780414a270 2024-09-16T15:49:41.409 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:41.406+0000 7f78017fa640 1 -- 172.21.15.123:0/306196919 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 21) ==== 97641+0+0 (secure 0 0 0) 0x7f77fc003b20 con 0x7f780414a270 2024-09-16T15:49:41.409 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:41.406+0000 7f78017fa640 1 --2- 172.21.15.123:0/306196919 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f77e4073520 0x7f77e40759e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:49:41.409 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:41.406+0000 7f7808a9f640 1 --2- 172.21.15.123:0/306196919 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f77e4073520 0x7f77e40759e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:49:41.409 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:41.406+0000 7f78017fa640 1 -- 172.21.15.123:0/306196919 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(124..124 src has 1..124) ==== 5823+0+0 (secure 0 0 0) 0x7f77fc095140 con 0x7f780414a270 2024-09-16T15:49:41.410 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:41.410+0000 7f7808a9f640 1 --2- 172.21.15.123:0/306196919 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f77e4073520 0x7f77e40759e0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f77f40aa000 tx=0x7f77f4019d40 comp rx=0 tx=0).ready entity=mgr.4150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:49:41.412 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:41.410+0000 7f78017fa640 1 -- 172.21.15.123:0/306196919 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f77fc05f960 con 0x7f780414a270 2024-09-16T15:49:41.624 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:41.622+0000 7f780ad03640 1 -- 172.21.15.123:0/306196919 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7f780406dfd0 con 0x7f780414a270 2024-09-16T15:49:41.625 INFO:tasks.ceph.mon.a.smithi123.stderr:2024-09-16T15:49:41.626+0000 7f268d076640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-09-16T15:49:41.627 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:41.626+0000 7f78017fa640 1 -- 172.21.15.123:0/306196919 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v78) ==== 161+0+0 (secure 0 0 0) 0x7f77fc05f260 con 0x7f780414a270 2024-09-16T15:49:41.627 INFO:teuthology.orchestra.run.smithi123.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-09-16T15:49:41.630 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:41.630+0000 7f780ad03640 1 -- 172.21.15.123:0/306196919 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f77e4073520 msgr2=0x7f77e40759e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:41.631 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:41.630+0000 7f780ad03640 1 --2- 172.21.15.123:0/306196919 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f77e4073520 0x7f77e40759e0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f77f40aa000 tx=0x7f77f4019d40 comp rx=0 tx=0).stop 2024-09-16T15:49:41.631 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:41.630+0000 7f780ad03640 1 -- 172.21.15.123:0/306196919 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f780414a270 msgr2=0x7f780414a6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:41.631 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:41.630+0000 7f780ad03640 1 --2- 172.21.15.123:0/306196919 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f780414a270 0x7f780414a6f0 secure :-1 s=READY pgs=1343 cs=0 l=1 rev1=1 crypto rx=0x7f78040d4a80 tx=0x7f77fc010f40 comp rx=0 tx=0).stop 2024-09-16T15:49:41.631 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:41.630+0000 7f780ad03640 1 -- 172.21.15.123:0/306196919 shutdown_connections 2024-09-16T15:49:41.631 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:41.630+0000 7f780ad03640 1 --2- 172.21.15.123:0/306196919 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f77e4073520 0x7f77e40759e0 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:41.631 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:41.630+0000 7f780ad03640 1 --2- 172.21.15.123:0/306196919 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f780414a270 0x7f780414a6f0 unknown :-1 s=CLOSED pgs=1343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:41.631 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:41.630+0000 7f780ad03640 1 --2- 172.21.15.123:0/306196919 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f78040caf80 0x7f78040b9190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:41.631 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:41.630+0000 7f780ad03640 1 --2- 172.21.15.123:0/306196919 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f77f0003a60 0x7f78040ca770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:41.631 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:41.630+0000 7f780ad03640 1 -- 172.21.15.123:0/306196919 >> 172.21.15.123:0/306196919 conn(0x7f7804069650 msgr2=0x7f780406d3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:49:41.631 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:41.630+0000 7f780ad03640 1 -- 172.21.15.123:0/306196919 shutdown_connections 2024-09-16T15:49:41.631 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:41.630+0000 7f780ad03640 1 -- 172.21.15.123:0/306196919 wait complete. 2024-09-16T15:49:41.648 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-09-16T15:49:41.680 INFO:tasks.ceph.mds.a.smithi123.stdout:starting mds.a at 2024-09-16T15:49:41.767 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:41.766+0000 7f6e721d6640 1 -- 172.21.15.123:0/2756156354 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6e6c1615c0 msgr2=0x7f6e6c0d0850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:41.767 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:41.766+0000 7f6e721d6640 1 --2- 172.21.15.123:0/2756156354 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6e6c1615c0 0x7f6e6c0d0850 secure :-1 s=READY pgs=1346 cs=0 l=1 rev1=1 crypto rx=0x7f6e5c009840 tx=0x7f6e5c01c790 comp rx=0 tx=0).stop 2024-09-16T15:49:41.767 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:41.766+0000 7f6e721d6640 1 -- 172.21.15.123:0/2756156354 shutdown_connections 2024-09-16T15:49:41.767 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:41.766+0000 7f6e721d6640 1 --2- 172.21.15.123:0/2756156354 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f6e6c0d2120 0x7f6e6c0d0d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:41.767 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:41.766+0000 7f6e721d6640 1 --2- 172.21.15.123:0/2756156354 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6e6c1615c0 0x7f6e6c0d0850 unknown :-1 s=CLOSED pgs=1346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:41.767 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:41.766+0000 7f6e721d6640 1 -- 172.21.15.123:0/2756156354 >> 172.21.15.123:0/2756156354 conn(0x7f6e6c0d3e40 msgr2=0x7f6e6c0d4220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:49:41.768 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:41.766+0000 7f6e721d6640 1 -- 172.21.15.123:0/2756156354 shutdown_connections 2024-09-16T15:49:41.768 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:41.766+0000 7f6e721d6640 1 -- 172.21.15.123:0/2756156354 wait complete. 2024-09-16T15:49:41.768 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:41.766+0000 7f6e721d6640 1 Processor -- start 2024-09-16T15:49:41.768 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:41.766+0000 7f6e721d6640 1 -- start start 2024-09-16T15:49:41.770 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:41.770+0000 7f6e721d6640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6e6c0b37a0 0x7f6e6c150e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:49:41.770 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:41.770+0000 7f6e721d6640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f6e6c0d2120 0x7f6e6c1547a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:49:41.770 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:41.770+0000 7f6e721d6640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6e6c1615c0 0x7f6e6c153d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:49:41.770 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:41.770+0000 7f6e721d6640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f6e6c16b620 con 0x7f6e6c0b37a0 2024-09-16T15:49:41.770 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:41.770+0000 7f6e721d6640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f6e6c0b30d0 con 0x7f6e6c1615c0 2024-09-16T15:49:41.771 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:41.770+0000 7f6e721d6640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f6e6c16b7a0 con 0x7f6e6c0d2120 2024-09-16T15:49:41.771 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:41.770+0000 7f6e6b7fe640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6e6c0b37a0 0x7f6e6c150e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:49:41.771 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:41.770+0000 7f6e6b7fe640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6e6c0b37a0 0x7f6e6c150e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:56918/0 (socket says 172.21.15.123:56918) 2024-09-16T15:49:41.771 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:41.770+0000 7f6e6b7fe640 1 -- 172.21.15.123:0/3376702247 learned_addr learned my addr 172.21.15.123:0/3376702247 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:49:41.771 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:41.770+0000 7f6e6affd640 1 --2- 172.21.15.123:0/3376702247 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f6e6c0d2120 0x7f6e6c1547a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:49:41.771 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:41.770+0000 7f6e6b7fe640 1 -- 172.21.15.123:0/3376702247 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f6e6c0d2120 msgr2=0x7f6e6c1547a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:41.771 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:41.770+0000 7f6e6bfff640 1 --2- 172.21.15.123:0/3376702247 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6e6c1615c0 0x7f6e6c153d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:49:41.771 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:41.770+0000 7f6e6b7fe640 1 --2- 172.21.15.123:0/3376702247 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f6e6c0d2120 0x7f6e6c1547a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:41.771 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:41.770+0000 7f6e6b7fe640 1 -- 172.21.15.123:0/3376702247 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6e6c1615c0 msgr2=0x7f6e6c153d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:41.771 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:41.770+0000 7f6e6b7fe640 1 --2- 172.21.15.123:0/3376702247 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6e6c1615c0 0x7f6e6c153d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:41.771 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:41.770+0000 7f6e6b7fe640 1 -- 172.21.15.123:0/3376702247 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6e6c152ca0 con 0x7f6e6c0b37a0 2024-09-16T15:49:41.771 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:41.770+0000 7f6e6affd640 1 --2- 172.21.15.123:0/3376702247 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f6e6c0d2120 0x7f6e6c1547a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T15:49:41.772 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:41.770+0000 7f6e6b7fe640 1 --2- 172.21.15.123:0/3376702247 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6e6c0b37a0 0x7f6e6c150e50 secure :-1 s=READY pgs=1347 cs=0 l=1 rev1=1 crypto rx=0x7f6e5c000c10 tx=0x7f6e5c005040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:49:41.772 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:41.770+0000 7f6e53fff640 1 -- 172.21.15.123:0/3376702247 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e5c0ac030 con 0x7f6e6c0b37a0 2024-09-16T15:49:41.772 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:41.770+0000 7f6e53fff640 1 -- 172.21.15.123:0/3376702247 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f6e5c005520 con 0x7f6e6c0b37a0 2024-09-16T15:49:41.772 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:41.770+0000 7f6e721d6640 1 -- 172.21.15.123:0/3376702247 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6e6c152f80 con 0x7f6e6c0b37a0 2024-09-16T15:49:41.772 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:41.770+0000 7f6e53fff640 1 -- 172.21.15.123:0/3376702247 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e5c004930 con 0x7f6e6c0b37a0 2024-09-16T15:49:41.772 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:41.770+0000 7f6e721d6640 1 -- 172.21.15.123:0/3376702247 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6e6c153540 con 0x7f6e6c0b37a0 2024-09-16T15:49:41.773 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:41.770+0000 7f6e721d6640 1 -- 172.21.15.123:0/3376702247 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6e6c0b3c10 con 0x7f6e6c0b37a0 2024-09-16T15:49:41.773 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:41.770+0000 7f6e53fff640 1 -- 172.21.15.123:0/3376702247 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 21) ==== 97641+0+0 (secure 0 0 0) 0x7f6e5c0af070 con 0x7f6e6c0b37a0 2024-09-16T15:49:41.773 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:41.770+0000 7f6e53fff640 1 --2- 172.21.15.123:0/3376702247 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f6e58073520 0x7f6e580759e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:49:41.773 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:41.770+0000 7f6e53fff640 1 -- 172.21.15.123:0/3376702247 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(125..125 src has 1..125) ==== 5881+0+0 (secure 0 0 0) 0x7f6e5c12f3a0 con 0x7f6e6c0b37a0 2024-09-16T15:49:41.773 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:41.770+0000 7f6e6affd640 1 --2- 172.21.15.123:0/3376702247 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f6e58073520 0x7f6e580759e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:49:41.773 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:41.774+0000 7f6e6affd640 1 --2- 172.21.15.123:0/3376702247 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f6e58073520 0x7f6e580759e0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f6e6c0cc000 tx=0x7f6e60008040 comp rx=0 tx=0).ready entity=mgr.4150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:49:41.775 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:41.774+0000 7f6e53fff640 1 -- 172.21.15.123:0/3376702247 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f6e5c016410 con 0x7f6e6c0b37a0 2024-09-16T15:49:41.967 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:41.966+0000 7f6e721d6640 1 -- 172.21.15.123:0/3376702247 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) -- 0x7f6e6c0b28a0 con 0x7f6e6c0b37a0 2024-09-16T15:49:42.638 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:42.638+0000 7f6e53fff640 1 -- 172.21.15.123:0/3376702247 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v79) ==== 104+0+0 (secure 0 0 0) 0x7f6e5c0f9c80 con 0x7f6e6c0b37a0 2024-09-16T15:49:42.642 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:42.642+0000 7f6e721d6640 1 -- 172.21.15.123:0/3376702247 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f6e58073520 msgr2=0x7f6e580759e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:42.643 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:42.642+0000 7f6e721d6640 1 --2- 172.21.15.123:0/3376702247 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f6e58073520 0x7f6e580759e0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f6e6c0cc000 tx=0x7f6e60008040 comp rx=0 tx=0).stop 2024-09-16T15:49:42.643 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:42.642+0000 7f6e721d6640 1 -- 172.21.15.123:0/3376702247 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6e6c0b37a0 msgr2=0x7f6e6c150e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:42.643 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:42.642+0000 7f6e721d6640 1 --2- 172.21.15.123:0/3376702247 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6e6c0b37a0 0x7f6e6c150e50 secure :-1 s=READY pgs=1347 cs=0 l=1 rev1=1 crypto rx=0x7f6e5c000c10 tx=0x7f6e5c005040 comp rx=0 tx=0).stop 2024-09-16T15:49:42.643 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:42.642+0000 7f6e721d6640 1 -- 172.21.15.123:0/3376702247 shutdown_connections 2024-09-16T15:49:42.643 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:42.642+0000 7f6e721d6640 1 --2- 172.21.15.123:0/3376702247 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f6e58073520 0x7f6e580759e0 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:42.643 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:42.642+0000 7f6e721d6640 1 --2- 172.21.15.123:0/3376702247 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6e6c1615c0 0x7f6e6c153d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:42.643 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:42.642+0000 7f6e721d6640 1 --2- 172.21.15.123:0/3376702247 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f6e6c0d2120 0x7f6e6c1547a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:42.643 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:42.642+0000 7f6e721d6640 1 --2- 172.21.15.123:0/3376702247 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6e6c0b37a0 0x7f6e6c150e50 unknown :-1 s=CLOSED pgs=1347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:42.643 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:42.642+0000 7f6e721d6640 1 -- 172.21.15.123:0/3376702247 >> 172.21.15.123:0/3376702247 conn(0x7f6e6c0d3e40 msgr2=0x7f6e6c0d4220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:49:42.643 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:42.642+0000 7f6e721d6640 1 -- 172.21.15.123:0/3376702247 shutdown_connections 2024-09-16T15:49:42.643 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:42.642+0000 7f6e721d6640 1 -- 172.21.15.123:0/3376702247 wait complete. 2024-09-16T15:49:42.665 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-09-16T15:49:42.843 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:42.842+0000 7f1bcaef1640 1 -- 172.21.15.123:0/3407199570 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1bc40ce9e0 msgr2=0x7f1bc41636d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:42.844 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:42.842+0000 7f1bcaef1640 1 --2- 172.21.15.123:0/3407199570 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1bc40ce9e0 0x7f1bc41636d0 secure :-1 s=READY pgs=1348 cs=0 l=1 rev1=1 crypto rx=0x7f1bc4072570 tx=0x7f1bb401c8c0 comp rx=0 tx=0).stop 2024-09-16T15:49:42.844 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:42.842+0000 7f1bcaef1640 1 -- 172.21.15.123:0/3407199570 shutdown_connections 2024-09-16T15:49:42.844 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:42.842+0000 7f1bcaef1640 1 --2- 172.21.15.123:0/3407199570 >> v2:172.21.15.184:3300/0 conn(0x7f1bc4070400 0x7f1bc4165c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:42.844 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:42.842+0000 7f1bcaef1640 1 --2- 172.21.15.123:0/3407199570 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1bc40ce9e0 0x7f1bc41636d0 unknown :-1 s=CLOSED pgs=1348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:42.844 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:42.842+0000 7f1bcaef1640 1 -- 172.21.15.123:0/3407199570 >> 172.21.15.123:0/3407199570 conn(0x7f1bc40b1630 msgr2=0x7f1bc40b1a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:49:42.844 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:42.842+0000 7f1bcaef1640 1 -- 172.21.15.123:0/3407199570 shutdown_connections 2024-09-16T15:49:42.844 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:42.842+0000 7f1bcaef1640 1 -- 172.21.15.123:0/3407199570 wait complete. 2024-09-16T15:49:42.846 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:42.846+0000 7f1bcaef1640 1 Processor -- start 2024-09-16T15:49:42.846 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:42.846+0000 7f1bcaef1640 1 -- start start 2024-09-16T15:49:42.847 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:42.846+0000 7f1bcaef1640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f1bc406e590 0x7f1bc41448c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:49:42.847 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:42.846+0000 7f1bcaef1640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1bc4070400 0x7f1bc4148060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:49:42.847 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:42.846+0000 7f1bcaef1640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1bc40ce9e0 0x7f1bc41485a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:49:42.847 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:42.846+0000 7f1bcaef1640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f1bc4168d10 con 0x7f1bc40ce9e0 2024-09-16T15:49:42.847 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:42.846+0000 7f1bcaef1640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f1bc4168b90 con 0x7f1bc4070400 2024-09-16T15:49:42.847 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:42.846+0000 7f1bcaef1640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f1bc4168e90 con 0x7f1bc406e590 2024-09-16T15:49:42.847 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:42.846+0000 7f1bc948e640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1bc40ce9e0 0x7f1bc41485a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:49:42.847 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:42.846+0000 7f1bc948e640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1bc40ce9e0 0x7f1bc41485a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:57396/0 (socket says 172.21.15.123:57396) 2024-09-16T15:49:42.847 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:42.846+0000 7f1bc948e640 1 -- 172.21.15.123:0/1252823528 learned_addr learned my addr 172.21.15.123:0/1252823528 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:49:42.847 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:42.846+0000 7f1bbbfff640 1 --2- 172.21.15.123:0/1252823528 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1bc4070400 0x7f1bc4148060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:49:42.847 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:42.846+0000 7f1bc8c8d640 1 --2- 172.21.15.123:0/1252823528 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f1bc406e590 0x7f1bc41448c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:49:42.847 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:42.846+0000 7f1bc948e640 1 -- 172.21.15.123:0/1252823528 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f1bc406e590 msgr2=0x7f1bc41448c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:42.847 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:42.846+0000 7f1bc948e640 1 --2- 172.21.15.123:0/1252823528 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f1bc406e590 0x7f1bc41448c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:42.848 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:42.846+0000 7f1bc948e640 1 -- 172.21.15.123:0/1252823528 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1bc4070400 msgr2=0x7f1bc4148060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:42.848 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:42.846+0000 7f1bc948e640 1 --2- 172.21.15.123:0/1252823528 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1bc4070400 0x7f1bc4148060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:42.848 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:42.846+0000 7f1bc948e640 1 -- 172.21.15.123:0/1252823528 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1bc4145c00 con 0x7f1bc40ce9e0 2024-09-16T15:49:42.848 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:42.846+0000 7f1bc948e640 1 --2- 172.21.15.123:0/1252823528 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1bc40ce9e0 0x7f1bc41485a0 secure :-1 s=READY pgs=1349 cs=0 l=1 rev1=1 crypto rx=0x7f1bbc00ea50 tx=0x7f1bbc00ef10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:49:42.848 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:42.846+0000 7f1bb97fa640 1 -- 172.21.15.123:0/1252823528 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1bbc00cdc0 con 0x7f1bc40ce9e0 2024-09-16T15:49:42.848 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:42.846+0000 7f1bb97fa640 1 -- 172.21.15.123:0/1252823528 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f1bbc004c40 con 0x7f1bc40ce9e0 2024-09-16T15:49:42.848 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:42.846+0000 7f1bcaef1640 1 -- 172.21.15.123:0/1252823528 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1bc415a9a0 con 0x7f1bc40ce9e0 2024-09-16T15:49:42.848 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:42.846+0000 7f1bb97fa640 1 -- 172.21.15.123:0/1252823528 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1bbc005280 con 0x7f1bc40ce9e0 2024-09-16T15:49:42.848 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:42.846+0000 7f1bcaef1640 1 -- 172.21.15.123:0/1252823528 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1bc4145340 con 0x7f1bc40ce9e0 2024-09-16T15:49:42.849 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:42.846+0000 7f1bcaef1640 1 -- 172.21.15.123:0/1252823528 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1bc406e970 con 0x7f1bc40ce9e0 2024-09-16T15:49:42.849 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:42.846+0000 7f1bb97fa640 1 -- 172.21.15.123:0/1252823528 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 21) ==== 97641+0+0 (secure 0 0 0) 0x7f1bbc005420 con 0x7f1bc40ce9e0 2024-09-16T15:49:42.849 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:42.850+0000 7f1bb97fa640 1 --2- 172.21.15.123:0/1252823528 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f1bb00734f0 0x7f1bb00759b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:49:42.849 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:42.850+0000 7f1bb97fa640 1 -- 172.21.15.123:0/1252823528 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(125..125 src has 1..125) ==== 5881+0+0 (secure 0 0 0) 0x7f1bbc0953c0 con 0x7f1bc40ce9e0 2024-09-16T15:49:42.849 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:42.850+0000 7f1bc8c8d640 1 --2- 172.21.15.123:0/1252823528 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f1bb00734f0 0x7f1bb00759b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:49:42.849 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:42.850+0000 7f1bc8c8d640 1 --2- 172.21.15.123:0/1252823528 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f1bb00734f0 0x7f1bb00759b0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f1bc415a8d0 tx=0x7f1bc0009040 comp rx=0 tx=0).ready entity=mgr.4150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:49:42.852 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:42.854+0000 7f1bb97fa640 1 -- 172.21.15.123:0/1252823528 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f1bbc016030 con 0x7f1bc40ce9e0 2024-09-16T15:49:43.029 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:43.030+0000 7f1bcaef1640 1 -- 172.21.15.123:0/1252823528 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) -- 0x7f1bc415cd70 con 0x7f1bc40ce9e0 2024-09-16T15:49:43.641 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:43.638+0000 7f1bb97fa640 1 -- 172.21.15.123:0/1252823528 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v126) ==== 181+0+0 (secure 0 0 0) 0x7f1bbc05fba0 con 0x7f1bc40ce9e0 2024-09-16T15:49:43.641 INFO:teuthology.orchestra.run.smithi123.stderr:pool 'cephfs_metadata' removed 2024-09-16T15:49:43.647 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:43.646+0000 7f1bcaef1640 1 -- 172.21.15.123:0/1252823528 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f1bb00734f0 msgr2=0x7f1bb00759b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:43.647 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:43.646+0000 7f1bcaef1640 1 --2- 172.21.15.123:0/1252823528 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f1bb00734f0 0x7f1bb00759b0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f1bc415a8d0 tx=0x7f1bc0009040 comp rx=0 tx=0).stop 2024-09-16T15:49:43.648 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:43.646+0000 7f1bcaef1640 1 -- 172.21.15.123:0/1252823528 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1bc40ce9e0 msgr2=0x7f1bc41485a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:43.648 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:43.646+0000 7f1bcaef1640 1 --2- 172.21.15.123:0/1252823528 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1bc40ce9e0 0x7f1bc41485a0 secure :-1 s=READY pgs=1349 cs=0 l=1 rev1=1 crypto rx=0x7f1bbc00ea50 tx=0x7f1bbc00ef10 comp rx=0 tx=0).stop 2024-09-16T15:49:43.648 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:43.646+0000 7f1bcaef1640 1 -- 172.21.15.123:0/1252823528 shutdown_connections 2024-09-16T15:49:43.648 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:43.646+0000 7f1bcaef1640 1 --2- 172.21.15.123:0/1252823528 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f1bb00734f0 0x7f1bb00759b0 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:43.648 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:43.646+0000 7f1bcaef1640 1 --2- 172.21.15.123:0/1252823528 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1bc40ce9e0 0x7f1bc41485a0 unknown :-1 s=CLOSED pgs=1349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:43.648 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:43.646+0000 7f1bcaef1640 1 --2- 172.21.15.123:0/1252823528 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1bc4070400 0x7f1bc4148060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:43.648 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:43.646+0000 7f1bcaef1640 1 --2- 172.21.15.123:0/1252823528 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f1bc406e590 0x7f1bc41448c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:43.648 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:43.646+0000 7f1bcaef1640 1 -- 172.21.15.123:0/1252823528 >> 172.21.15.123:0/1252823528 conn(0x7f1bc40b1630 msgr2=0x7f1bc415c430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:49:43.648 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:43.646+0000 7f1bcaef1640 1 -- 172.21.15.123:0/1252823528 shutdown_connections 2024-09-16T15:49:43.648 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:43.646+0000 7f1bcaef1640 1 -- 172.21.15.123:0/1252823528 wait complete. 2024-09-16T15:49:43.672 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-09-16T15:49:43.810 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:43.810+0000 7fdee0e8f640 1 -- 172.21.15.123:0/1506566336 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdec4003a40 msgr2=0x7fdec4003e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:43.810 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:43.810+0000 7fdee0e8f640 1 --2- 172.21.15.123:0/1506566336 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdec4003a40 0x7fdec4003e60 secure :-1 s=READY pgs=1351 cs=0 l=1 rev1=1 crypto rx=0x7fdecc009ef0 tx=0x7fdecc01d1d0 comp rx=0 tx=0).stop 2024-09-16T15:49:43.810 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:43.810+0000 7fdedad9d640 1 -- 172.21.15.123:0/1506566336 reap_dead start 2024-09-16T15:49:43.810 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:43.810+0000 7fdee0e8f640 1 -- 172.21.15.123:0/1506566336 shutdown_connections 2024-09-16T15:49:43.810 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:43.810+0000 7fdee0e8f640 1 -- 172.21.15.123:0/1506566336 >> 172.21.15.123:0/1506566336 conn(0x7fdedc0b1630 msgr2=0x7fdedc0b1a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:49:43.810 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:43.810+0000 7fdee0e8f640 1 -- 172.21.15.123:0/1506566336 shutdown_connections 2024-09-16T15:49:43.810 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:43.810+0000 7fdee0e8f640 1 -- 172.21.15.123:0/1506566336 wait complete. 2024-09-16T15:49:43.811 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:43.810+0000 7fdee0e8f640 1 Processor -- start 2024-09-16T15:49:43.811 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:43.810+0000 7fdee0e8f640 1 -- start start 2024-09-16T15:49:43.813 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:43.810+0000 7fdee0e8f640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdec4003a40 0x7fdedc147660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:49:43.813 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:43.810+0000 7fdee0e8f640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fdedc1f7f20 0x7fdedc1f83a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:49:43.813 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:43.810+0000 7fdee0e8f640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fdedc15ddc0 0x7fdedc15e270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:49:43.813 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:43.810+0000 7fdee0e8f640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fdedc06f2f0 con 0x7fdec4003a40 2024-09-16T15:49:43.813 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:43.810+0000 7fdee0e8f640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fdedc06f170 con 0x7fdedc15ddc0 2024-09-16T15:49:43.813 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:43.810+0000 7fdee0e8f640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fdedc06f470 con 0x7fdedc1f7f20 2024-09-16T15:49:43.813 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:43.810+0000 7fdeda59c640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdec4003a40 0x7fdedc147660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:49:43.814 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:43.810+0000 7fdeda59c640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdec4003a40 0x7fdedc147660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:57412/0 (socket says 172.21.15.123:57412) 2024-09-16T15:49:43.814 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:43.810+0000 7fdeda59c640 1 -- 172.21.15.123:0/2454565598 learned_addr learned my addr 172.21.15.123:0/2454565598 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:49:43.814 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:43.810+0000 7fded9d9b640 1 --2- 172.21.15.123:0/2454565598 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fdedc1f7f20 0x7fdedc1f83a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:49:43.814 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:43.810+0000 7fdeda59c640 1 -- 172.21.15.123:0/2454565598 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fdedc1f7f20 msgr2=0x7fdedc1f83a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:43.814 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:43.810+0000 7fdedad9d640 1 --2- 172.21.15.123:0/2454565598 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fdedc15ddc0 0x7fdedc15e270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:49:43.814 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:43.810+0000 7fdeda59c640 1 --2- 172.21.15.123:0/2454565598 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fdedc1f7f20 0x7fdedc1f83a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:43.814 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:43.810+0000 7fdeda59c640 1 -- 172.21.15.123:0/2454565598 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fdedc15ddc0 msgr2=0x7fdedc15e270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:43.814 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:43.810+0000 7fdeda59c640 1 --2- 172.21.15.123:0/2454565598 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fdedc15ddc0 0x7fdedc15e270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:43.814 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:43.810+0000 7fdeda59c640 1 -- 172.21.15.123:0/2454565598 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdedc15c370 con 0x7fdec4003a40 2024-09-16T15:49:43.814 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:43.810+0000 7fded9d9b640 1 --2- 172.21.15.123:0/2454565598 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fdedc1f7f20 0x7fdedc1f83a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T15:49:43.814 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:43.810+0000 7fdedad9d640 1 --2- 172.21.15.123:0/2454565598 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fdedc15ddc0 0x7fdedc15e270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T15:49:43.814 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:43.810+0000 7fdeda59c640 1 --2- 172.21.15.123:0/2454565598 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdec4003a40 0x7fdedc147660 secure :-1 s=READY pgs=1352 cs=0 l=1 rev1=1 crypto rx=0x7fdecc0af040 tx=0x7fdecc0ada40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:49:43.815 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:43.814+0000 7fdecaffd640 1 -- 172.21.15.123:0/2454565598 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdecc0bb070 con 0x7fdec4003a40 2024-09-16T15:49:43.815 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:43.814+0000 7fdecaffd640 1 -- 172.21.15.123:0/2454565598 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fdecc004b30 con 0x7fdec4003a40 2024-09-16T15:49:43.815 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:43.814+0000 7fdecaffd640 1 -- 172.21.15.123:0/2454565598 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdecc009ac0 con 0x7fdec4003a40 2024-09-16T15:49:43.815 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:43.814+0000 7fdee0e8f640 1 -- 172.21.15.123:0/2454565598 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdedc1f8a10 con 0x7fdec4003a40 2024-09-16T15:49:43.815 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:43.814+0000 7fdee0e8f640 1 -- 172.21.15.123:0/2454565598 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdedc14ad70 con 0x7fdec4003a40 2024-09-16T15:49:43.815 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:43.814+0000 7fdee0e8f640 1 -- 172.21.15.123:0/2454565598 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdedc0b3260 con 0x7fdec4003a40 2024-09-16T15:49:43.816 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:43.814+0000 7fdecaffd640 1 -- 172.21.15.123:0/2454565598 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 21) ==== 97641+0+0 (secure 0 0 0) 0x7fdecc0b7460 con 0x7fdec4003a40 2024-09-16T15:49:43.816 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:43.814+0000 7fdecaffd640 1 --2- 172.21.15.123:0/2454565598 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fdebc073520 0x7fdebc0759e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:49:43.816 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:43.814+0000 7fdecaffd640 1 -- 172.21.15.123:0/2454565598 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(126..126 src has 1..126) ==== 5443+0+0 (secure 0 0 0) 0x7fdecc12fe00 con 0x7fdec4003a40 2024-09-16T15:49:43.816 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:43.814+0000 7fded9d9b640 1 --2- 172.21.15.123:0/2454565598 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fdebc073520 0x7fdebc0759e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:49:43.816 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:43.814+0000 7fded9d9b640 1 --2- 172.21.15.123:0/2454565598 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fdebc073520 0x7fdebc0759e0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fded0001870 tx=0x7fded0001c40 comp rx=0 tx=0).ready entity=mgr.4150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:49:43.818 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:43.818+0000 7fdecaffd640 1 -- 172.21.15.123:0/2454565598 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7fdecc016ee0 con 0x7fdec4003a40 2024-09-16T15:49:44.002 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:44.002+0000 7fdee0e8f640 1 -- 172.21.15.123:0/2454565598 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) -- 0x7fdedc1f8ec0 con 0x7fdec4003a40 2024-09-16T15:49:44.651 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:44.650+0000 7fdecaffd640 1 -- 172.21.15.123:0/2454565598 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v127) ==== 169+0+0 (secure 0 0 0) 0x7fdecc0fa890 con 0x7fdec4003a40 2024-09-16T15:49:44.651 INFO:teuthology.orchestra.run.smithi123.stderr:pool 'cephfs_data' removed 2024-09-16T15:49:44.654 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:44.654+0000 7fdee0e8f640 1 -- 172.21.15.123:0/2454565598 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fdebc073520 msgr2=0x7fdebc0759e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:44.654 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:44.654+0000 7fdee0e8f640 1 --2- 172.21.15.123:0/2454565598 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fdebc073520 0x7fdebc0759e0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fded0001870 tx=0x7fded0001c40 comp rx=0 tx=0).stop 2024-09-16T15:49:44.655 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:44.654+0000 7fdee0e8f640 1 -- 172.21.15.123:0/2454565598 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdec4003a40 msgr2=0x7fdedc147660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:44.655 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:44.654+0000 7fdee0e8f640 1 --2- 172.21.15.123:0/2454565598 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdec4003a40 0x7fdedc147660 secure :-1 s=READY pgs=1352 cs=0 l=1 rev1=1 crypto rx=0x7fdecc0af040 tx=0x7fdecc0ada40 comp rx=0 tx=0).stop 2024-09-16T15:49:44.655 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:44.654+0000 7fdee0e8f640 1 -- 172.21.15.123:0/2454565598 shutdown_connections 2024-09-16T15:49:44.655 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:44.654+0000 7fdee0e8f640 1 --2- 172.21.15.123:0/2454565598 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fdebc073520 0x7fdebc0759e0 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:44.655 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:44.654+0000 7fdee0e8f640 1 --2- 172.21.15.123:0/2454565598 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fdedc15ddc0 0x7fdedc15e270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:44.655 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:44.654+0000 7fdee0e8f640 1 --2- 172.21.15.123:0/2454565598 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fdedc1f7f20 0x7fdedc1f83a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:44.655 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:44.654+0000 7fdee0e8f640 1 --2- 172.21.15.123:0/2454565598 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdec4003a40 0x7fdedc147660 unknown :-1 s=CLOSED pgs=1352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:44.655 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:44.654+0000 7fdee0e8f640 1 -- 172.21.15.123:0/2454565598 >> 172.21.15.123:0/2454565598 conn(0x7fdedc0b1630 msgr2=0x7fdedc0cea50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:49:44.655 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:44.654+0000 7fdee0e8f640 1 -- 172.21.15.123:0/2454565598 shutdown_connections 2024-09-16T15:49:44.655 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:44.654+0000 7fdee0e8f640 1 -- 172.21.15.123:0/2454565598 wait complete. 2024-09-16T15:49:44.681 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config reset 27 2024-09-16T15:49:44.807 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:44.806+0000 7f9f01990640 1 -- 172.21.15.123:0/2693089616 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9efc15ec40 msgr2=0x7f9efc06f420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:44.807 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:44.806+0000 7f9f01990640 1 --2- 172.21.15.123:0/2693089616 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9efc15ec40 0x7f9efc06f420 secure :-1 s=READY pgs=1353 cs=0 l=1 rev1=1 crypto rx=0x7f9ef400ad90 tx=0x7f9ef401db90 comp rx=0 tx=0).stop 2024-09-16T15:49:44.807 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:44.806+0000 7f9f01990640 1 -- 172.21.15.123:0/2693089616 shutdown_connections 2024-09-16T15:49:44.807 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:44.806+0000 7f9f01990640 1 --2- 172.21.15.123:0/2693089616 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9efc15ec40 0x7f9efc06f420 unknown :-1 s=CLOSED pgs=1353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:44.807 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:44.806+0000 7f9f01990640 1 --2- 172.21.15.123:0/2693089616 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f9efc0cf5a0 0x7f9efc06eee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:44.807 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:44.806+0000 7f9f01990640 1 -- 172.21.15.123:0/2693089616 >> 172.21.15.123:0/2693089616 conn(0x7f9efc0cfb10 msgr2=0x7f9efc0cff20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:49:44.807 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:44.806+0000 7f9f01990640 1 -- 172.21.15.123:0/2693089616 shutdown_connections 2024-09-16T15:49:44.807 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:44.806+0000 7f9f01990640 1 -- 172.21.15.123:0/2693089616 wait complete. 2024-09-16T15:49:44.809 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:44.806+0000 7f9f01990640 1 Processor -- start 2024-09-16T15:49:44.809 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:44.806+0000 7f9f01990640 1 -- start start 2024-09-16T15:49:44.809 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:44.806+0000 7f9f01990640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9efc0cf5a0 0x7f9efc14b8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:49:44.809 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:44.810+0000 7f9f01990640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f9efc158120 0x7f9efc0b29a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:49:44.809 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:44.810+0000 7f9f01990640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f9efc15ec40 0x7f9efc153c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:49:44.810 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:44.810+0000 7f9f01990640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f9efc15b300 con 0x7f9efc0cf5a0 2024-09-16T15:49:44.810 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:44.810+0000 7f9f01990640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f9efc15b180 con 0x7f9efc158120 2024-09-16T15:49:44.810 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:44.810+0000 7f9f01990640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f9efc15b480 con 0x7f9efc15ec40 2024-09-16T15:49:44.810 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:44.810+0000 7f9efaffd640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9efc0cf5a0 0x7f9efc14b8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:49:44.810 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:44.810+0000 7f9efaffd640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9efc0cf5a0 0x7f9efc14b8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:57418/0 (socket says 172.21.15.123:57418) 2024-09-16T15:49:44.810 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:44.810+0000 7f9efaffd640 1 -- 172.21.15.123:0/3572765286 learned_addr learned my addr 172.21.15.123:0/3572765286 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:49:44.810 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:44.810+0000 7f9efb7fe640 1 --2- 172.21.15.123:0/3572765286 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f9efc15ec40 0x7f9efc153c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:49:44.810 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:44.810+0000 7f9efaffd640 1 -- 172.21.15.123:0/3572765286 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f9efc15ec40 msgr2=0x7f9efc153c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:44.810 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:44.810+0000 7f9efaffd640 1 --2- 172.21.15.123:0/3572765286 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f9efc15ec40 0x7f9efc153c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:44.810 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:44.810+0000 7f9efaffd640 1 -- 172.21.15.123:0/3572765286 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f9efc158120 msgr2=0x7f9efc0b29a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:44.810 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:44.810+0000 7f9efaffd640 1 --2- 172.21.15.123:0/3572765286 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f9efc158120 0x7f9efc0b29a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:44.810 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:44.810+0000 7f9efaffd640 1 -- 172.21.15.123:0/3572765286 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9efc15a100 con 0x7f9efc0cf5a0 2024-09-16T15:49:44.811 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:44.810+0000 7f9efaffd640 1 --2- 172.21.15.123:0/3572765286 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9efc0cf5a0 0x7f9efc14b8e0 secure :-1 s=READY pgs=1354 cs=0 l=1 rev1=1 crypto rx=0x7f9eec00c480 tx=0x7f9eec00c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:49:44.811 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:44.810+0000 7f9ee3fff640 1 -- 172.21.15.123:0/3572765286 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9eec012830 con 0x7f9efc0cf5a0 2024-09-16T15:49:44.811 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:44.810+0000 7f9ee3fff640 1 -- 172.21.15.123:0/3572765286 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f9eec017070 con 0x7f9efc0cf5a0 2024-09-16T15:49:44.811 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:44.810+0000 7f9f01990640 1 -- 172.21.15.123:0/3572765286 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9efc15a3e0 con 0x7f9efc0cf5a0 2024-09-16T15:49:44.811 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:44.810+0000 7f9ee3fff640 1 -- 172.21.15.123:0/3572765286 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9eec012d80 con 0x7f9efc0cf5a0 2024-09-16T15:49:44.811 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:44.810+0000 7f9f01990640 1 -- 172.21.15.123:0/3572765286 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9efc14c2b0 con 0x7f9efc0cf5a0 2024-09-16T15:49:44.812 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:44.810+0000 7f9f01990640 1 -- 172.21.15.123:0/3572765286 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9efc0ca320 con 0x7f9efc0cf5a0 2024-09-16T15:49:44.812 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:44.810+0000 7f9ee3fff640 1 -- 172.21.15.123:0/3572765286 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 21) ==== 97641+0+0 (secure 0 0 0) 0x7f9eec01c460 con 0x7f9efc0cf5a0 2024-09-16T15:49:44.812 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:44.810+0000 7f9ee3fff640 1 --2- 172.21.15.123:0/3572765286 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f9ee80734f0 0x7f9ee80759b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:49:44.812 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:44.810+0000 7f9ee3fff640 1 -- 172.21.15.123:0/3572765286 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(127..127 src has 1..127) ==== 4971+0+0 (secure 0 0 0) 0x7f9eec094570 con 0x7f9efc0cf5a0 2024-09-16T15:49:44.812 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:44.810+0000 7f9efa7fc640 1 --2- 172.21.15.123:0/3572765286 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f9ee80734f0 0x7f9ee80759b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:49:44.812 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:44.810+0000 7f9efa7fc640 1 --2- 172.21.15.123:0/3572765286 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f9ee80734f0 0x7f9ee80759b0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f9efc0c6850 tx=0x7f9ef0005c00 comp rx=0 tx=0).ready entity=mgr.4150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:49:44.815 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:44.814+0000 7f9ee3fff640 1 -- 172.21.15.123:0/3572765286 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f9eec05f0e0 con 0x7f9efc0cf5a0 2024-09-16T15:49:44.990 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:44.990+0000 7f9f01990640 1 -- 172.21.15.123:0/3572765286 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "config reset", "num": 27} v 0) -- 0x7f9efc0d06c0 con 0x7f9efc0cf5a0 2024-09-16T15:49:44.995 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:44.994+0000 7f9ee3fff640 1 -- 172.21.15.123:0/3572765286 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "config reset", "num": 27}]=0 v41) ==== 71+0+0 (secure 0 0 0) 0x7f9eec05e9e0 con 0x7f9efc0cf5a0 2024-09-16T15:49:44.998 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:44.998+0000 7f9f01990640 1 -- 172.21.15.123:0/3572765286 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f9ee80734f0 msgr2=0x7f9ee80759b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:44.998 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:44.998+0000 7f9f01990640 1 --2- 172.21.15.123:0/3572765286 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f9ee80734f0 0x7f9ee80759b0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f9efc0c6850 tx=0x7f9ef0005c00 comp rx=0 tx=0).stop 2024-09-16T15:49:44.998 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:44.998+0000 7f9f01990640 1 -- 172.21.15.123:0/3572765286 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9efc0cf5a0 msgr2=0x7f9efc14b8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:44.998 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:44.998+0000 7f9f01990640 1 --2- 172.21.15.123:0/3572765286 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9efc0cf5a0 0x7f9efc14b8e0 secure :-1 s=READY pgs=1354 cs=0 l=1 rev1=1 crypto rx=0x7f9eec00c480 tx=0x7f9eec00c940 comp rx=0 tx=0).stop 2024-09-16T15:49:44.998 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:44.998+0000 7f9f01990640 1 -- 172.21.15.123:0/3572765286 shutdown_connections 2024-09-16T15:49:44.998 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:44.998+0000 7f9f01990640 1 --2- 172.21.15.123:0/3572765286 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f9ee80734f0 0x7f9ee80759b0 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:44.998 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:44.998+0000 7f9f01990640 1 --2- 172.21.15.123:0/3572765286 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f9efc15ec40 0x7f9efc153c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:44.998 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:44.998+0000 7f9f01990640 1 --2- 172.21.15.123:0/3572765286 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f9efc158120 0x7f9efc0b29a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:44.998 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:44.998+0000 7f9f01990640 1 --2- 172.21.15.123:0/3572765286 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9efc0cf5a0 0x7f9efc14b8e0 unknown :-1 s=CLOSED pgs=1354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:44.998 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:44.998+0000 7f9f01990640 1 -- 172.21.15.123:0/3572765286 >> 172.21.15.123:0/3572765286 conn(0x7f9efc0cfb10 msgr2=0x7f9efc155f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:49:44.999 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:44.998+0000 7f9f01990640 1 -- 172.21.15.123:0/3572765286 shutdown_connections 2024-09-16T15:49:44.999 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:44.998+0000 7f9f01990640 1 -- 172.21.15.123:0/3572765286 wait complete. 2024-09-16T15:49:45.014 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_volumes.TestMisc.test_subvolume_no_upgrade_v1_to_v2' 2024-09-16T15:49:45.138 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:45.138+0000 7ff611991640 1 -- 172.21.15.123:0/494968652 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7ff60c0cf5a0 msgr2=0x7ff60c158120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:45.138 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:45.138+0000 7ff611991640 1 --2- 172.21.15.123:0/494968652 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7ff60c0cf5a0 0x7ff60c158120 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7ff5fc009840 tx=0x7ff5fc01c760 comp rx=0 tx=0).stop 2024-09-16T15:49:45.138 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:45.138+0000 7ff611991640 1 -- 172.21.15.123:0/494968652 shutdown_connections 2024-09-16T15:49:45.138 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:45.138+0000 7ff611991640 1 --2- 172.21.15.123:0/494968652 >> v2:172.21.15.184:3300/0 conn(0x7ff60c15aa60 0x7ff60c15af10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:45.138 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:45.138+0000 7ff611991640 1 --2- 172.21.15.123:0/494968652 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7ff60c0cf5a0 0x7ff60c158120 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:45.138 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:45.138+0000 7ff611991640 1 -- 172.21.15.123:0/494968652 >> 172.21.15.123:0/494968652 conn(0x7ff60c0cfb10 msgr2=0x7ff60c0cff20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:49:45.138 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:45.138+0000 7ff611991640 1 -- 172.21.15.123:0/494968652 shutdown_connections 2024-09-16T15:49:45.138 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:45.138+0000 7ff611991640 1 -- 172.21.15.123:0/494968652 wait complete. 2024-09-16T15:49:45.139 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:45.138+0000 7ff611991640 1 Processor -- start 2024-09-16T15:49:45.139 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:45.138+0000 7ff611991640 1 -- start start 2024-09-16T15:49:45.141 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:45.138+0000 7ff611991640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7ff60c0cf5a0 0x7ff60c0b51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:49:45.141 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:45.138+0000 7ff611991640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff60c15aa60 0x7ff60c14fc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:49:45.141 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:45.138+0000 7ff611991640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff60c15ec40 0x7ff60c15d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:49:45.142 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:45.138+0000 7ff611991640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7ff60c06e650 con 0x7ff60c15ec40 2024-09-16T15:49:45.142 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:45.138+0000 7ff611991640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7ff60c160160 con 0x7ff60c15aa60 2024-09-16T15:49:45.142 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:45.138+0000 7ff611991640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7ff60c06e7d0 con 0x7ff60c0cf5a0 2024-09-16T15:49:45.142 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:45.138+0000 7ff60b7fe640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff60c15ec40 0x7ff60c15d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:49:45.142 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:45.138+0000 7ff60b7fe640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff60c15ec40 0x7ff60c15d270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:57420/0 (socket says 172.21.15.123:57420) 2024-09-16T15:49:45.142 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:45.138+0000 7ff60b7fe640 1 -- 172.21.15.123:0/2035883415 learned_addr learned my addr 172.21.15.123:0/2035883415 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:49:45.142 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:45.138+0000 7ff60affd640 1 --2- 172.21.15.123:0/2035883415 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7ff60c0cf5a0 0x7ff60c0b51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:49:45.142 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:45.138+0000 7ff60a7fc640 1 --2- 172.21.15.123:0/2035883415 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff60c15aa60 0x7ff60c14fc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:49:45.142 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:45.138+0000 7ff60b7fe640 1 -- 172.21.15.123:0/2035883415 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7ff60c0cf5a0 msgr2=0x7ff60c0b51c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:45.142 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:45.138+0000 7ff60b7fe640 1 --2- 172.21.15.123:0/2035883415 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7ff60c0cf5a0 0x7ff60c0b51c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:45.142 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:45.138+0000 7ff60b7fe640 1 -- 172.21.15.123:0/2035883415 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff60c15aa60 msgr2=0x7ff60c14fc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:45.143 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:45.138+0000 7ff60b7fe640 1 --2- 172.21.15.123:0/2035883415 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff60c15aa60 0x7ff60c14fc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:45.143 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:45.138+0000 7ff60b7fe640 1 -- 172.21.15.123:0/2035883415 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff60c0b39e0 con 0x7ff60c15ec40 2024-09-16T15:49:45.143 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:45.142+0000 7ff60b7fe640 1 --2- 172.21.15.123:0/2035883415 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff60c15ec40 0x7ff60c15d270 secure :-1 s=READY pgs=1355 cs=0 l=1 rev1=1 crypto rx=0x7ff60400e970 tx=0x7ff60400ee30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:49:45.143 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:45.142+0000 7ff603fff640 1 -- 172.21.15.123:0/2035883415 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff60400cd80 con 0x7ff60c15ec40 2024-09-16T15:49:45.143 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:45.142+0000 7ff603fff640 1 -- 172.21.15.123:0/2035883415 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7ff604004030 con 0x7ff60c15ec40 2024-09-16T15:49:45.143 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:45.142+0000 7ff611991640 1 -- 172.21.15.123:0/2035883415 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff60c0b57a0 con 0x7ff60c15ec40 2024-09-16T15:49:45.143 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:45.142+0000 7ff603fff640 1 -- 172.21.15.123:0/2035883415 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff604004480 con 0x7ff60c15ec40 2024-09-16T15:49:45.143 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:45.142+0000 7ff611991640 1 -- 172.21.15.123:0/2035883415 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff60c1f9520 con 0x7ff60c15ec40 2024-09-16T15:49:45.143 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:45.142+0000 7ff611991640 1 -- 172.21.15.123:0/2035883415 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff60c158120 con 0x7ff60c15ec40 2024-09-16T15:49:45.144 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:45.142+0000 7ff603fff640 1 -- 172.21.15.123:0/2035883415 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 21) ==== 97641+0+0 (secure 0 0 0) 0x7ff604003600 con 0x7ff60c15ec40 2024-09-16T15:49:45.144 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:45.142+0000 7ff603fff640 1 --2- 172.21.15.123:0/2035883415 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7ff5ec0734f0 0x7ff5ec0759b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:49:45.144 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:45.142+0000 7ff603fff640 1 -- 172.21.15.123:0/2035883415 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(127..127 src has 1..127) ==== 4971+0+0 (secure 0 0 0) 0x7ff604095250 con 0x7ff60c15ec40 2024-09-16T15:49:45.144 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:45.142+0000 7ff60affd640 1 --2- 172.21.15.123:0/2035883415 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7ff5ec0734f0 0x7ff5ec0759b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:49:45.144 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:45.142+0000 7ff60affd640 1 --2- 172.21.15.123:0/2035883415 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7ff5ec0734f0 0x7ff5ec0759b0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7ff5fc0ae000 tx=0x7ff5fc01afa0 comp rx=0 tx=0).ready entity=mgr.4150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:49:45.147 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:45.146+0000 7ff603fff640 1 -- 172.21.15.123:0/2035883415 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7ff60c0b39e0 con 0x7ff60c15ec40 2024-09-16T15:49:45.382 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:45.382+0000 7ff611991640 1 -- 172.21.15.123:0/2035883415 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_volumes.TestMisc.test_subvolume_no_upgrade_v1_to_v2"]} v 0) -- 0x7ff60c0d06c0 con 0x7ff60c15ec40 2024-09-16T15:49:45.653 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:45.654+0000 7ff603fff640 1 -- 172.21.15.123:0/2035883415 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_volumes.TestMisc.test_subvolume_no_upgrade_v1_to_v2"]}]=0 v360) ==== 148+0+0 (secure 0 0 0) 0x7ff60405fd40 con 0x7ff60c15ec40 2024-09-16T15:49:45.658 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:45.658+0000 7ff611991640 1 -- 172.21.15.123:0/2035883415 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7ff5ec0734f0 msgr2=0x7ff5ec0759b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:45.658 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:45.658+0000 7ff611991640 1 --2- 172.21.15.123:0/2035883415 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7ff5ec0734f0 0x7ff5ec0759b0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7ff5fc0ae000 tx=0x7ff5fc01afa0 comp rx=0 tx=0).stop 2024-09-16T15:49:45.658 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:45.658+0000 7ff611991640 1 -- 172.21.15.123:0/2035883415 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff60c15ec40 msgr2=0x7ff60c15d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:45.658 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:45.658+0000 7ff611991640 1 --2- 172.21.15.123:0/2035883415 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff60c15ec40 0x7ff60c15d270 secure :-1 s=READY pgs=1355 cs=0 l=1 rev1=1 crypto rx=0x7ff60400e970 tx=0x7ff60400ee30 comp rx=0 tx=0).stop 2024-09-16T15:49:45.658 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:45.658+0000 7ff611991640 1 -- 172.21.15.123:0/2035883415 shutdown_connections 2024-09-16T15:49:45.658 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:45.658+0000 7ff611991640 1 --2- 172.21.15.123:0/2035883415 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7ff5ec0734f0 0x7ff5ec0759b0 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:45.658 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:45.658+0000 7ff611991640 1 --2- 172.21.15.123:0/2035883415 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff60c15ec40 0x7ff60c15d270 unknown :-1 s=CLOSED pgs=1355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:45.658 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:45.658+0000 7ff611991640 1 --2- 172.21.15.123:0/2035883415 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff60c15aa60 0x7ff60c14fc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:45.658 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:45.658+0000 7ff611991640 1 --2- 172.21.15.123:0/2035883415 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7ff60c0cf5a0 0x7ff60c0b51c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:45.658 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:45.658+0000 7ff611991640 1 -- 172.21.15.123:0/2035883415 >> 172.21.15.123:0/2035883415 conn(0x7ff60c0cfb10 msgr2=0x7ff60c15c270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:49:45.658 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:45.658+0000 7ff611991640 1 -- 172.21.15.123:0/2035883415 shutdown_connections 2024-09-16T15:49:45.658 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:45.658+0000 7ff611991640 1 -- 172.21.15.123:0/2035883415 wait complete. 2024-09-16T15:49:45.680 INFO:tasks.cephfs_test_runner:poor man's upgrade test -- theme continues... ... ok 2024-09-16T15:49:45.681 INFO:tasks.cephfs_test_runner:Starting test: test_subvolume_ops_on_nonexistent_vol (tasks.cephfs.test_volumes.TestMisc) 2024-09-16T15:49:45.681 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-16T15:49:45.810 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:45.810+0000 7fcc5392d640 1 -- 172.21.15.123:0/2785125468 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcc4c06f5f0 msgr2=0x7fcc4c15aa60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:45.810 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:45.810+0000 7fcc5392d640 1 --2- 172.21.15.123:0/2785125468 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcc4c06f5f0 0x7fcc4c15aa60 secure :-1 s=READY pgs=1357 cs=0 l=1 rev1=1 crypto rx=0x7fcc44009c80 tx=0x7fcc4401cd10 comp rx=0 tx=0).stop 2024-09-16T15:49:45.810 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:45.810+0000 7fcc5392d640 1 -- 172.21.15.123:0/2785125468 shutdown_connections 2024-09-16T15:49:45.810 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:45.810+0000 7fcc5392d640 1 --2- 172.21.15.123:0/2785125468 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcc4c06f5f0 0x7fcc4c15aa60 unknown :-1 s=CLOSED pgs=1357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:45.810 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:45.810+0000 7fcc5392d640 1 --2- 172.21.15.123:0/2785125468 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fcc4c158120 0x7fcc4c0ca320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:45.810 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:45.810+0000 7fcc5392d640 1 -- 172.21.15.123:0/2785125468 >> 172.21.15.123:0/2785125468 conn(0x7fcc4c0cfb10 msgr2=0x7fcc4c0cff20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:49:45.810 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:45.810+0000 7fcc5392d640 1 -- 172.21.15.123:0/2785125468 shutdown_connections 2024-09-16T15:49:45.811 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:45.810+0000 7fcc5392d640 1 -- 172.21.15.123:0/2785125468 wait complete. 2024-09-16T15:49:45.813 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:45.810+0000 7fcc5392d640 1 Processor -- start 2024-09-16T15:49:45.813 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:45.810+0000 7fcc5392d640 1 -- start start 2024-09-16T15:49:45.813 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:45.810+0000 7fcc5392d640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fcc4c06f5f0 0x7fcc4c0b41f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:49:45.813 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:45.810+0000 7fcc5392d640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcc4c0cf5a0 0x7fcc4c156c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:49:45.813 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:45.810+0000 7fcc5392d640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fcc4c158120 0x7fcc4c072710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:49:45.813 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:45.810+0000 7fcc5392d640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fcc4c167c10 con 0x7fcc4c0cf5a0 2024-09-16T15:49:45.813 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:45.810+0000 7fcc5392d640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fcc4c167a90 con 0x7fcc4c158120 2024-09-16T15:49:45.813 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:45.810+0000 7fcc5392d640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fcc4c167d90 con 0x7fcc4c06f5f0 2024-09-16T15:49:45.813 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:45.810+0000 7fcc50ec8640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcc4c0cf5a0 0x7fcc4c156c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:49:45.814 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:45.810+0000 7fcc50ec8640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcc4c0cf5a0 0x7fcc4c156c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:57434/0 (socket says 172.21.15.123:57434) 2024-09-16T15:49:45.814 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:45.810+0000 7fcc50ec8640 1 -- 172.21.15.123:0/2818562515 learned_addr learned my addr 172.21.15.123:0/2818562515 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:49:45.814 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:45.810+0000 7fcc516c9640 1 --2- 172.21.15.123:0/2818562515 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fcc4c06f5f0 0x7fcc4c0b41f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:49:45.814 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:45.810+0000 7fcc50ec8640 1 -- 172.21.15.123:0/2818562515 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fcc4c06f5f0 msgr2=0x7fcc4c0b41f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:45.814 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:45.810+0000 7fcc50ec8640 1 --2- 172.21.15.123:0/2818562515 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fcc4c06f5f0 0x7fcc4c0b41f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:45.814 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:45.810+0000 7fcc50ec8640 1 -- 172.21.15.123:0/2818562515 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fcc4c158120 msgr2=0x7fcc4c072710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:45.814 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:45.810+0000 7fcc50ec8640 1 --2- 172.21.15.123:0/2818562515 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fcc4c158120 0x7fcc4c072710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:45.814 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:45.810+0000 7fcc50ec8640 1 -- 172.21.15.123:0/2818562515 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcc4c072ff0 con 0x7fcc4c0cf5a0 2024-09-16T15:49:45.814 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:45.810+0000 7fcc50ec8640 1 --2- 172.21.15.123:0/2818562515 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcc4c0cf5a0 0x7fcc4c156c50 secure :-1 s=READY pgs=1358 cs=0 l=1 rev1=1 crypto rx=0x7fcc3c00c480 tx=0x7fcc3c00c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:49:45.814 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:45.810+0000 7fcc41ffb640 1 -- 172.21.15.123:0/2818562515 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc3c0052d0 con 0x7fcc4c0cf5a0 2024-09-16T15:49:45.814 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:45.810+0000 7fcc41ffb640 1 -- 172.21.15.123:0/2818562515 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fcc3c005470 con 0x7fcc4c0cf5a0 2024-09-16T15:49:45.814 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:45.814+0000 7fcc5392d640 1 -- 172.21.15.123:0/2818562515 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcc4c0732d0 con 0x7fcc4c0cf5a0 2024-09-16T15:49:45.814 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:45.814+0000 7fcc41ffb640 1 -- 172.21.15.123:0/2818562515 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc3c012870 con 0x7fcc4c0cf5a0 2024-09-16T15:49:45.815 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:45.814+0000 7fcc5392d640 1 -- 172.21.15.123:0/2818562515 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcc4c0b29a0 con 0x7fcc4c0cf5a0 2024-09-16T15:49:45.815 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:45.814+0000 7fcc5392d640 1 -- 172.21.15.123:0/2818562515 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcc4c0d06c0 con 0x7fcc4c0cf5a0 2024-09-16T15:49:45.815 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:45.814+0000 7fcc41ffb640 1 -- 172.21.15.123:0/2818562515 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 21) ==== 97641+0+0 (secure 0 0 0) 0x7fcc3c003600 con 0x7fcc4c0cf5a0 2024-09-16T15:49:45.816 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:45.814+0000 7fcc41ffb640 1 --2- 172.21.15.123:0/2818562515 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fcc300734f0 0x7fcc300759b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:49:45.816 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:45.814+0000 7fcc41ffb640 1 -- 172.21.15.123:0/2818562515 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(127..127 src has 1..127) ==== 4971+0+0 (secure 0 0 0) 0x7fcc3c094320 con 0x7fcc4c0cf5a0 2024-09-16T15:49:45.816 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:45.814+0000 7fcc516c9640 1 --2- 172.21.15.123:0/2818562515 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fcc300734f0 0x7fcc300759b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:49:45.816 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:45.814+0000 7fcc516c9640 1 --2- 172.21.15.123:0/2818562515 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fcc300734f0 0x7fcc300759b0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fcc480058d0 tx=0x7fcc48005bf0 comp rx=0 tx=0).ready entity=mgr.4150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:49:45.819 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:45.818+0000 7fcc41ffb640 1 -- 172.21.15.123:0/2818562515 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7fcc3c05ee90 con 0x7fcc4c0cf5a0 2024-09-16T15:49:45.995 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:45.994+0000 7fcc5392d640 1 -- 172.21.15.123:0/2818562515 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fcc4c0adba0 con 0x7fcc4c0cf5a0 2024-09-16T15:49:45.996 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:45.994+0000 7fcc41ffb640 1 -- 172.21.15.123:0/2818562515 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v127) ==== 74+0+13148 (secure 0 0 0) 0x7fcc3c05e790 con 0x7fcc4c0cf5a0 2024-09-16T15:49:45.997 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-09-16T15:49:45.997 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":127,"fsid":"56077f89-18c8-486f-ae7e-eecc0bb7822c","created":"2024-09-16T15:42:26.364892+0000","modified":"2024-09-16T15:49:44.643785+0000","last_up_change":"2024-09-16T15:42:38.707234+0000","last_in_change":"2024-09-16T15:42:29.423662+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":19,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-16T15:42:39.287764+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9e7c1955-a607-4671-9be7-f2ef760ceeaa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":117,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6800","nonce":3882044969},{"type":"v1","addr":"172.21.15.123:6801","nonce":3882044969}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6802","nonce":3882044969},{"type":"v1","addr":"172.21.15.123:6803","nonce":3882044969}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6806","nonce":3882044969},{"type":"v1","addr":"172.21.15.123:6807","nonce":3882044969}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6804","nonce":3882044969},{"type":"v1","addr":"172.21.15.123:6805","nonce":3882044969}]},"public_addr":"172.21.15.123:6801/3882044969","cluster_addr":"172.21.15.123:6803/3882044969","heartbeat_back_addr":"172.21.15.123:6807/3882044969","heartbeat_front_addr":"172.21.15.123:6805/3882044969","state":["exists","up"]},{"osd":1,"uuid":"c540cfde-0977-4742-9b17-fd634668e010","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":117,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6816","nonce":4037001850},{"type":"v1","addr":"172.21.15.123:6817","nonce":4037001850}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6818","nonce":4037001850},{"type":"v1","addr":"172.21.15.123:6819","nonce":4037001850}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6822","nonce":4037001850},{"type":"v1","addr":"172.21.15.123:6823","nonce":4037001850}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6820","nonce":4037001850},{"type":"v1","addr":"172.21.15.123:6821","nonce":4037001850}]},"public_addr":"172.21.15.123:6817/4037001850","cluster_addr":"172.21.15.123:6819/4037001850","heartbeat_back_addr":"172.21.15.123:6823/4037001850","heartbeat_front_addr":"172.21.15.123:6821/4037001850","state":["exists","up"]},{"osd":2,"uuid":"9864a496-f169-486b-95c4-1d59162793a7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":117,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6824","nonce":1332143088},{"type":"v1","addr":"172.21.15.123:6825","nonce":1332143088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6826","nonce":1332143088},{"type":"v1","addr":"172.21.15.123:6827","nonce":1332143088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6830","nonce":1332143088},{"type":"v1","addr":"172.21.15.123:6831","nonce":1332143088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6828","nonce":1332143088},{"type":"v1","addr":"172.21.15.123:6829","nonce":1332143088}]},"public_addr":"172.21.15.123:6825/1332143088","cluster_addr":"172.21.15.123:6827/1332143088","heartbeat_back_addr":"172.21.15.123:6831/1332143088","heartbeat_front_addr":"172.21.15.123:6829/1332143088","state":["exists","up"]},{"osd":3,"uuid":"c6d1f817-f074-4325-95e1-578eba13edda","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":117,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6808","nonce":988586998},{"type":"v1","addr":"172.21.15.123:6809","nonce":988586998}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6810","nonce":988586998},{"type":"v1","addr":"172.21.15.123:6811","nonce":988586998}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6814","nonce":988586998},{"type":"v1","addr":"172.21.15.123:6815","nonce":988586998}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6812","nonce":988586998},{"type":"v1","addr":"172.21.15.123:6813","nonce":988586998}]},"public_addr":"172.21.15.123:6809/988586998","cluster_addr":"172.21.15.123:6811/988586998","heartbeat_back_addr":"172.21.15.123:6815/988586998","heartbeat_front_addr":"172.21.15.123:6813/988586998","state":["exists","up"]},{"osd":4,"uuid":"200a8df4-50ef-40a5-b48a-58981269c086","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":117,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6816","nonce":3387813890},{"type":"v1","addr":"172.21.15.184:6817","nonce":3387813890}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6818","nonce":3387813890},{"type":"v1","addr":"172.21.15.184:6819","nonce":3387813890}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6822","nonce":3387813890},{"type":"v1","addr":"172.21.15.184:6823","nonce":3387813890}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6820","nonce":3387813890},{"type":"v1","addr":"172.21.15.184:6821","nonce":3387813890}]},"public_addr":"172.21.15.184:6817/3387813890","cluster_addr":"172.21.15.184:6819/3387813890","heartbeat_back_addr":"172.21.15.184:6823/3387813890","heartbeat_front_addr":"172.21.15.184:6821/3387813890","state":["exists","up"]},{"osd":5,"uuid":"32d5a900-da3d-412c-bc19-13861a2368d8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":120,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6808","nonce":2680916017},{"type":"v1","addr":"172.21.15.184:6809","nonce":2680916017}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6810","nonce":2680916017},{"type":"v1","addr":"172.21.15.184:6811","nonce":2680916017}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6814","nonce":2680916017},{"type":"v1","addr":"172.21.15.184:6815","nonce":2680916017}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6812","nonce":2680916017},{"type":"v1","addr":"172.21.15.184:6813","nonce":2680916017}]},"public_addr":"172.21.15.184:6809/2680916017","cluster_addr":"172.21.15.184:6811/2680916017","heartbeat_back_addr":"172.21.15.184:6815/2680916017","heartbeat_front_addr":"172.21.15.184:6813/2680916017","state":["exists","up"]},{"osd":6,"uuid":"cde6f39e-aa1b-4260-ac19-774ee6aaca72","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":117,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6824","nonce":2183120457},{"type":"v1","addr":"172.21.15.184:6825","nonce":2183120457}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6826","nonce":2183120457},{"type":"v1","addr":"172.21.15.184:6827","nonce":2183120457}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6830","nonce":2183120457},{"type":"v1","addr":"172.21.15.184:6831","nonce":2183120457}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6828","nonce":2183120457},{"type":"v1","addr":"172.21.15.184:6829","nonce":2183120457}]},"public_addr":"172.21.15.184:6825/2183120457","cluster_addr":"172.21.15.184:6827/2183120457","heartbeat_back_addr":"172.21.15.184:6831/2183120457","heartbeat_front_addr":"172.21.15.184:6829/2183120457","state":["exists","up"]},{"osd":7,"uuid":"4f381778-eee1-47c6-804f-df0e6b33e313","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":120,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6800","nonce":4102957285},{"type":"v1","addr":"172.21.15.184:6801","nonce":4102957285}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6802","nonce":4102957285},{"type":"v1","addr":"172.21.15.184:6803","nonce":4102957285}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6806","nonce":4102957285},{"type":"v1","addr":"172.21.15.184:6807","nonce":4102957285}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6804","nonce":4102957285},{"type":"v1","addr":"172.21.15.184:6805","nonce":4102957285}]},"public_addr":"172.21.15.184:6801/4102957285","cluster_addr":"172.21.15.184:6803/4102957285","heartbeat_back_addr":"172.21.15.184:6807/4102957285","heartbeat_front_addr":"172.21.15.184:6805/4102957285","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:35.972210+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.222007+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.198617+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.385663+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.358935+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.156066+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.752973+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:35.682846+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.123:6835/2110793616":"2024-09-17T15:49:41.627484+0000","172.21.15.123:6833/2110793616":"2024-09-17T15:49:41.627484+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":17,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T15:49:45.999 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:45.998+0000 7fcc5392d640 1 -- 172.21.15.123:0/2818562515 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fcc300734f0 msgr2=0x7fcc300759b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:45.999 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:45.998+0000 7fcc5392d640 1 --2- 172.21.15.123:0/2818562515 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fcc300734f0 0x7fcc300759b0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fcc480058d0 tx=0x7fcc48005bf0 comp rx=0 tx=0).stop 2024-09-16T15:49:45.999 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:45.998+0000 7fcc5392d640 1 -- 172.21.15.123:0/2818562515 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcc4c0cf5a0 msgr2=0x7fcc4c156c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:45.999 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:45.998+0000 7fcc5392d640 1 --2- 172.21.15.123:0/2818562515 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcc4c0cf5a0 0x7fcc4c156c50 secure :-1 s=READY pgs=1358 cs=0 l=1 rev1=1 crypto rx=0x7fcc3c00c480 tx=0x7fcc3c00c940 comp rx=0 tx=0).stop 2024-09-16T15:49:45.999 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:45.998+0000 7fcc5392d640 1 -- 172.21.15.123:0/2818562515 shutdown_connections 2024-09-16T15:49:45.999 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:45.998+0000 7fcc5392d640 1 --2- 172.21.15.123:0/2818562515 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fcc300734f0 0x7fcc300759b0 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:45.999 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:45.998+0000 7fcc5392d640 1 --2- 172.21.15.123:0/2818562515 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fcc4c158120 0x7fcc4c072710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:46.000 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:45.998+0000 7fcc5392d640 1 --2- 172.21.15.123:0/2818562515 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcc4c0cf5a0 0x7fcc4c156c50 unknown :-1 s=CLOSED pgs=1358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:46.000 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:45.998+0000 7fcc5392d640 1 --2- 172.21.15.123:0/2818562515 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fcc4c06f5f0 0x7fcc4c0b41f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:46.000 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:45.998+0000 7fcc5392d640 1 -- 172.21.15.123:0/2818562515 >> 172.21.15.123:0/2818562515 conn(0x7fcc4c0cfb10 msgr2=0x7fcc4c0b0c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:49:46.000 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:45.998+0000 7fcc5392d640 1 -- 172.21.15.123:0/2818562515 shutdown_connections 2024-09-16T15:49:46.000 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:45.998+0000 7fcc5392d640 1 -- 172.21.15.123:0/2818562515 wait complete. 2024-09-16T15:49:46.016 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-16T15:42:39.287764+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-09-16T15:49:46.017 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-09-16T15:49:46.149 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:46.150+0000 7f5d8ecbf640 1 -- 172.21.15.123:0/3916851502 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5d74003a10 msgr2=0x7f5d74003e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:46.149 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:46.150+0000 7f5d8ecbf640 1 --2- 172.21.15.123:0/3916851502 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5d74003a10 0x7f5d74003e30 secure :-1 s=READY pgs=1360 cs=0 l=1 rev1=1 crypto rx=0x7f5d70009b60 tx=0x7f5d7001d270 comp rx=0 tx=0).stop 2024-09-16T15:49:46.149 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:46.150+0000 7f5d8d25c640 1 -- 172.21.15.123:0/3916851502 reap_dead start 2024-09-16T15:49:46.149 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:46.150+0000 7f5d8ecbf640 1 -- 172.21.15.123:0/3916851502 shutdown_connections 2024-09-16T15:49:46.150 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:46.150+0000 7f5d8ecbf640 1 -- 172.21.15.123:0/3916851502 >> 172.21.15.123:0/3916851502 conn(0x7f5d880b1710 msgr2=0x7f5d880b1b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:49:46.150 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:46.150+0000 7f5d8ecbf640 1 -- 172.21.15.123:0/3916851502 shutdown_connections 2024-09-16T15:49:46.150 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:46.150+0000 7f5d8ecbf640 1 -- 172.21.15.123:0/3916851502 wait complete. 2024-09-16T15:49:46.150 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:46.150+0000 7f5d8ecbf640 1 Processor -- start 2024-09-16T15:49:46.150 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:46.150+0000 7f5d8ecbf640 1 -- start start 2024-09-16T15:49:46.152 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:46.150+0000 7f5d8ecbf640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5d74003a10 0x7f5d88158d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:49:46.152 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:46.150+0000 7f5d8ecbf640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f5d881f8cb0 0x7f5d880b4be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:49:46.153 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:46.150+0000 7f5d8ecbf640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5d880b5120 0x7f5d8815b6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:49:46.153 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:46.150+0000 7f5d8ecbf640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f5d8806fdb0 con 0x7f5d880b5120 2024-09-16T15:49:46.153 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:46.150+0000 7f5d8ecbf640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f5d8806fc30 con 0x7f5d74003a10 2024-09-16T15:49:46.153 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:46.150+0000 7f5d8ecbf640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f5d8806ff30 con 0x7f5d881f8cb0 2024-09-16T15:49:46.153 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:46.150+0000 7f5d8d25c640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5d880b5120 0x7f5d8815b6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:49:46.153 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:46.150+0000 7f5d8d25c640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5d880b5120 0x7f5d8815b6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:57444/0 (socket says 172.21.15.123:57444) 2024-09-16T15:49:46.153 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:46.150+0000 7f5d8d25c640 1 -- 172.21.15.123:0/244742830 learned_addr learned my addr 172.21.15.123:0/244742830 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:49:46.153 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:46.150+0000 7f5d87fff640 1 --2- 172.21.15.123:0/244742830 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f5d881f8cb0 0x7f5d880b4be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:49:46.153 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:46.150+0000 7f5d8ca5b640 1 --2- 172.21.15.123:0/244742830 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5d74003a10 0x7f5d88158d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:49:46.153 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:46.150+0000 7f5d8d25c640 1 -- 172.21.15.123:0/244742830 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f5d881f8cb0 msgr2=0x7f5d880b4be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:46.153 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:46.150+0000 7f5d8d25c640 1 --2- 172.21.15.123:0/244742830 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f5d881f8cb0 0x7f5d880b4be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:46.153 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:46.150+0000 7f5d8d25c640 1 -- 172.21.15.123:0/244742830 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5d74003a10 msgr2=0x7f5d88158d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:46.153 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:46.150+0000 7f5d8d25c640 1 --2- 172.21.15.123:0/244742830 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5d74003a10 0x7f5d88158d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:46.154 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:46.150+0000 7f5d8d25c640 1 -- 172.21.15.123:0/244742830 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5d8815cc60 con 0x7f5d880b5120 2024-09-16T15:49:46.154 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:46.150+0000 7f5d87fff640 1 --2- 172.21.15.123:0/244742830 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f5d881f8cb0 0x7f5d880b4be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T15:49:46.154 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:46.150+0000 7f5d8d25c640 1 --2- 172.21.15.123:0/244742830 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5d880b5120 0x7f5d8815b6f0 secure :-1 s=READY pgs=1361 cs=0 l=1 rev1=1 crypto rx=0x7f5d80004a60 tx=0x7f5d8000cd00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:49:46.154 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:46.150+0000 7f5d857fa640 1 -- 172.21.15.123:0/244742830 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d80017070 con 0x7f5d880b5120 2024-09-16T15:49:46.154 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:46.150+0000 7f5d857fa640 1 -- 172.21.15.123:0/244742830 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f5d8000aad0 con 0x7f5d880b5120 2024-09-16T15:49:46.154 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:46.150+0000 7f5d857fa640 1 -- 172.21.15.123:0/244742830 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d800070d0 con 0x7f5d880b5120 2024-09-16T15:49:46.154 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:46.150+0000 7f5d8ecbf640 1 -- 172.21.15.123:0/244742830 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5d88159c80 con 0x7f5d880b5120 2024-09-16T15:49:46.154 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:46.150+0000 7f5d8ecbf640 1 -- 172.21.15.123:0/244742830 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5d88158240 con 0x7f5d880b5120 2024-09-16T15:49:46.154 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:46.154+0000 7f5d8ecbf640 1 -- 172.21.15.123:0/244742830 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5d880d0810 con 0x7f5d880b5120 2024-09-16T15:49:46.155 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:46.154+0000 7f5d857fa640 1 -- 172.21.15.123:0/244742830 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 21) ==== 97641+0+0 (secure 0 0 0) 0x7f5d80007270 con 0x7f5d880b5120 2024-09-16T15:49:46.155 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:46.154+0000 7f5d857fa640 1 --2- 172.21.15.123:0/244742830 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f5d680734f0 0x7f5d680759b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:49:46.155 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:46.154+0000 7f5d857fa640 1 -- 172.21.15.123:0/244742830 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(127..127 src has 1..127) ==== 4971+0+0 (secure 0 0 0) 0x7f5d800943e0 con 0x7f5d880b5120 2024-09-16T15:49:46.155 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:46.154+0000 7f5d8ca5b640 1 --2- 172.21.15.123:0/244742830 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f5d680734f0 0x7f5d680759b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:49:46.155 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:46.154+0000 7f5d8ca5b640 1 --2- 172.21.15.123:0/244742830 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f5d680734f0 0x7f5d680759b0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f5d700ae000 tx=0x7f5d7001a010 comp rx=0 tx=0).ready entity=mgr.4150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:49:46.158 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:46.158+0000 7f5d857fa640 1 -- 172.21.15.123:0/244742830 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f5d80020020 con 0x7f5d880b5120 2024-09-16T15:49:46.332 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:46.330+0000 7f5d8ecbf640 1 -- 172.21.15.123:0/244742830 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f5d88157ff0 con 0x7f5d880b5120 2024-09-16T15:49:46.333 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:46.334+0000 7f5d857fa640 1 -- 172.21.15.123:0/244742830 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v127) ==== 93+0+10 (secure 0 0 0) 0x7f5d8005f050 con 0x7f5d880b5120 2024-09-16T15:49:46.333 INFO:teuthology.orchestra.run.smithi123.stdout:pg_num: 1 2024-09-16T15:49:46.336 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:46.334+0000 7f5d8ecbf640 1 -- 172.21.15.123:0/244742830 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f5d680734f0 msgr2=0x7f5d680759b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:46.336 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:46.334+0000 7f5d8ecbf640 1 --2- 172.21.15.123:0/244742830 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f5d680734f0 0x7f5d680759b0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f5d700ae000 tx=0x7f5d7001a010 comp rx=0 tx=0).stop 2024-09-16T15:49:46.336 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:46.334+0000 7f5d8ecbf640 1 -- 172.21.15.123:0/244742830 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5d880b5120 msgr2=0x7f5d8815b6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:46.336 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:46.334+0000 7f5d8ecbf640 1 --2- 172.21.15.123:0/244742830 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5d880b5120 0x7f5d8815b6f0 secure :-1 s=READY pgs=1361 cs=0 l=1 rev1=1 crypto rx=0x7f5d80004a60 tx=0x7f5d8000cd00 comp rx=0 tx=0).stop 2024-09-16T15:49:46.336 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:46.334+0000 7f5d8ecbf640 1 -- 172.21.15.123:0/244742830 shutdown_connections 2024-09-16T15:49:46.336 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:46.334+0000 7f5d8ecbf640 1 --2- 172.21.15.123:0/244742830 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f5d680734f0 0x7f5d680759b0 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:46.336 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:46.334+0000 7f5d8ecbf640 1 --2- 172.21.15.123:0/244742830 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5d880b5120 0x7f5d8815b6f0 unknown :-1 s=CLOSED pgs=1361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:46.336 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:46.334+0000 7f5d8ecbf640 1 --2- 172.21.15.123:0/244742830 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f5d881f8cb0 0x7f5d880b4be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:46.336 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:46.334+0000 7f5d8ecbf640 1 --2- 172.21.15.123:0/244742830 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5d74003a10 0x7f5d88158d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:46.336 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:46.334+0000 7f5d8ecbf640 1 -- 172.21.15.123:0/244742830 >> 172.21.15.123:0/244742830 conn(0x7f5d880b1710 msgr2=0x7f5d8815c550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:49:46.336 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:46.334+0000 7f5d8ecbf640 1 -- 172.21.15.123:0/244742830 shutdown_connections 2024-09-16T15:49:46.336 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:46.334+0000 7f5d8ecbf640 1 -- 172.21.15.123:0/244742830 wait complete. 2024-09-16T15:49:46.352 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_volumes.TestMisc.test_subvolume_ops_on_nonexistent_vol' 2024-09-16T15:49:46.468 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:46.466+0000 7f41cd4ac640 1 -- 172.21.15.123:0/721507354 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f41c8074600 msgr2=0x7f41c806e750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:46.469 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:46.466+0000 7f41cd4ac640 1 --2- 172.21.15.123:0/721507354 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f41c8074600 0x7f41c806e750 secure :-1 s=READY pgs=1363 cs=0 l=1 rev1=1 crypto rx=0x7f41c806df40 tx=0x7f41c001cac0 comp rx=0 tx=0).stop 2024-09-16T15:49:46.469 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:46.470+0000 7f41cd4ac640 1 -- 172.21.15.123:0/721507354 shutdown_connections 2024-09-16T15:49:46.469 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:46.470+0000 7f41cd4ac640 1 --2- 172.21.15.123:0/721507354 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f41c8074600 0x7f41c806e750 unknown :-1 s=CLOSED pgs=1363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:46.469 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:46.470+0000 7f41cd4ac640 1 --2- 172.21.15.123:0/721507354 >> v2:172.21.15.184:3300/0 conn(0x7f41c80726c0 0x7f41c80d0630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:46.469 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:46.470+0000 7f41cd4ac640 1 -- 172.21.15.123:0/721507354 >> 172.21.15.123:0/721507354 conn(0x7f41c80cfb10 msgr2=0x7f41c80cff20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:49:46.469 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:46.470+0000 7f41cd4ac640 1 -- 172.21.15.123:0/721507354 shutdown_connections 2024-09-16T15:49:46.469 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:46.470+0000 7f41cd4ac640 1 -- 172.21.15.123:0/721507354 wait complete. 2024-09-16T15:49:46.470 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:46.470+0000 7f41cd4ac640 1 Processor -- start 2024-09-16T15:49:46.470 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:46.470+0000 7f41cd4ac640 1 -- start start 2024-09-16T15:49:46.471 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:46.470+0000 7f41cd4ac640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f41c80726c0 0x7f41c815e390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:49:46.471 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:46.470+0000 7f41cd4ac640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f41c8074600 0x7f41c8166170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:49:46.472 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:46.470+0000 7f41cd4ac640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f41c815bca0 0x7f41c81fb1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:49:46.472 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:46.470+0000 7f41cd4ac640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f41c8168a50 con 0x7f41c80726c0 2024-09-16T15:49:46.472 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:46.470+0000 7f41cd4ac640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f41c806f980 con 0x7f41c8074600 2024-09-16T15:49:46.472 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:46.470+0000 7f41cd4ac640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f41c8168bd0 con 0x7f41c815bca0 2024-09-16T15:49:46.472 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:46.470+0000 7f41c6ffd640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f41c80726c0 0x7f41c815e390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:49:46.472 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:46.470+0000 7f41c6ffd640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f41c80726c0 0x7f41c815e390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:57448/0 (socket says 172.21.15.123:57448) 2024-09-16T15:49:46.472 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:46.470+0000 7f41c6ffd640 1 -- 172.21.15.123:0/3999445001 learned_addr learned my addr 172.21.15.123:0/3999445001 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:49:46.472 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:46.470+0000 7f41c6ffd640 1 -- 172.21.15.123:0/3999445001 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f41c815bca0 msgr2=0x7f41c81fb1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:46.472 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:46.470+0000 7f41c6ffd640 1 --2- 172.21.15.123:0/3999445001 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f41c815bca0 0x7f41c81fb1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:46.472 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:46.470+0000 7f41c67fc640 1 --2- 172.21.15.123:0/3999445001 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f41c8074600 0x7f41c8166170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:49:46.472 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:46.470+0000 7f41c6ffd640 1 -- 172.21.15.123:0/3999445001 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f41c8074600 msgr2=0x7f41c8166170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:46.472 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:46.470+0000 7f41c6ffd640 1 --2- 172.21.15.123:0/3999445001 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f41c8074600 0x7f41c8166170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:46.473 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:46.470+0000 7f41c6ffd640 1 -- 172.21.15.123:0/3999445001 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f41c81fa9f0 con 0x7f41c80726c0 2024-09-16T15:49:46.473 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:46.470+0000 7f41c67fc640 1 --2- 172.21.15.123:0/3999445001 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f41c8074600 0x7f41c8166170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T15:49:46.473 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:46.470+0000 7f41c6ffd640 1 --2- 172.21.15.123:0/3999445001 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f41c80726c0 0x7f41c815e390 secure :-1 s=READY pgs=1364 cs=0 l=1 rev1=1 crypto rx=0x7f41b800c400 tx=0x7f41b800c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:49:46.473 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:46.470+0000 7f41af7fe640 1 -- 172.21.15.123:0/3999445001 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41b8011790 con 0x7f41c80726c0 2024-09-16T15:49:46.473 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:46.470+0000 7f41af7fe640 1 -- 172.21.15.123:0/3999445001 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f41b8017070 con 0x7f41c80726c0 2024-09-16T15:49:46.473 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:46.470+0000 7f41cd4ac640 1 -- 172.21.15.123:0/3999445001 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f41c815ea70 con 0x7f41c80726c0 2024-09-16T15:49:46.473 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:46.470+0000 7f41af7fe640 1 -- 172.21.15.123:0/3999445001 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41b8011cc0 con 0x7f41c80726c0 2024-09-16T15:49:46.473 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:46.470+0000 7f41cd4ac640 1 -- 172.21.15.123:0/3999445001 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f41c815f030 con 0x7f41c80726c0 2024-09-16T15:49:46.474 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:46.470+0000 7f41cd4ac640 1 -- 172.21.15.123:0/3999445001 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f41c80d0630 con 0x7f41c80726c0 2024-09-16T15:49:46.474 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:46.474+0000 7f41af7fe640 1 -- 172.21.15.123:0/3999445001 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 21) ==== 97641+0+0 (secure 0 0 0) 0x7f41b801c460 con 0x7f41c80726c0 2024-09-16T15:49:46.474 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:46.474+0000 7f41af7fe640 1 --2- 172.21.15.123:0/3999445001 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f41b40734f0 0x7f41b40759b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:49:46.474 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:46.474+0000 7f41c67fc640 1 --2- 172.21.15.123:0/3999445001 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f41b40734f0 0x7f41b40759b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:49:46.474 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:46.474+0000 7f41af7fe640 1 -- 172.21.15.123:0/3999445001 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(127..127 src has 1..127) ==== 4971+0+0 (secure 0 0 0) 0x7f41b80944f0 con 0x7f41c80726c0 2024-09-16T15:49:46.474 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:46.474+0000 7f41c67fc640 1 --2- 172.21.15.123:0/3999445001 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f41b40734f0 0x7f41b40759b0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f41bc005fd0 tx=0x7f41bc005da0 comp rx=0 tx=0).ready entity=mgr.4150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:49:46.477 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:46.478+0000 7f41af7fe640 1 -- 172.21.15.123:0/3999445001 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f41b805efd0 con 0x7f41c80726c0 2024-09-16T15:49:46.708 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:46.706+0000 7f41cd4ac640 1 -- 172.21.15.123:0/3999445001 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_volumes.TestMisc.test_subvolume_ops_on_nonexistent_vol"]} v 0) -- 0x7f41c80700e0 con 0x7f41c80726c0 2024-09-16T15:49:47.663 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:47.662+0000 7f41af7fe640 1 -- 172.21.15.123:0/3999445001 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_volumes.TestMisc.test_subvolume_ops_on_nonexistent_vol"]}]=0 v362) ==== 154+0+0 (secure 0 0 0) 0x7f41b805e8d0 con 0x7f41c80726c0 2024-09-16T15:49:47.668 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:47.666+0000 7f41cd4ac640 1 -- 172.21.15.123:0/3999445001 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f41b40734f0 msgr2=0x7f41b40759b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:47.668 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:47.666+0000 7f41cd4ac640 1 --2- 172.21.15.123:0/3999445001 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f41b40734f0 0x7f41b40759b0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f41bc005fd0 tx=0x7f41bc005da0 comp rx=0 tx=0).stop 2024-09-16T15:49:47.668 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:47.666+0000 7f41cd4ac640 1 -- 172.21.15.123:0/3999445001 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f41c80726c0 msgr2=0x7f41c815e390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:47.668 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:47.666+0000 7f41cd4ac640 1 --2- 172.21.15.123:0/3999445001 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f41c80726c0 0x7f41c815e390 secure :-1 s=READY pgs=1364 cs=0 l=1 rev1=1 crypto rx=0x7f41b800c400 tx=0x7f41b800c8c0 comp rx=0 tx=0).stop 2024-09-16T15:49:47.668 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:47.666+0000 7f41cd4ac640 1 -- 172.21.15.123:0/3999445001 shutdown_connections 2024-09-16T15:49:47.669 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:47.666+0000 7f41cd4ac640 1 --2- 172.21.15.123:0/3999445001 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f41b40734f0 0x7f41b40759b0 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:47.669 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:47.666+0000 7f41cd4ac640 1 --2- 172.21.15.123:0/3999445001 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f41c815bca0 0x7f41c81fb1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:47.669 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:47.666+0000 7f41cd4ac640 1 --2- 172.21.15.123:0/3999445001 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f41c8074600 0x7f41c8166170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:47.669 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:47.666+0000 7f41cd4ac640 1 --2- 172.21.15.123:0/3999445001 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f41c80726c0 0x7f41c815e390 unknown :-1 s=CLOSED pgs=1364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:47.669 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:47.666+0000 7f41cd4ac640 1 -- 172.21.15.123:0/3999445001 >> 172.21.15.123:0/3999445001 conn(0x7f41c80cfb10 msgr2=0x7f41c80cda80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:49:47.669 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:47.670+0000 7f41cd4ac640 1 -- 172.21.15.123:0/3999445001 shutdown_connections 2024-09-16T15:49:47.669 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:47.670+0000 7f41cd4ac640 1 -- 172.21.15.123:0/3999445001 wait complete. 2024-09-16T15:49:47.689 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config dump 2024-09-16T15:49:47.810 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:47.810+0000 7f38c06c0640 1 -- 172.21.15.123:0/3918680589 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f38a4003a10 msgr2=0x7f38a4003e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:47.810 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:47.810+0000 7f38c06c0640 1 --2- 172.21.15.123:0/3918680589 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f38a4003a10 0x7f38a4003e30 secure :-1 s=READY pgs=1366 cs=0 l=1 rev1=1 crypto rx=0x7f38a800bb20 tx=0x7f38a801cd90 comp rx=0 tx=0).stop 2024-09-16T15:49:47.810 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:47.810+0000 7f38bec5d640 1 -- 172.21.15.123:0/3918680589 reap_dead start 2024-09-16T15:49:47.810 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:47.810+0000 7f38c06c0640 1 -- 172.21.15.123:0/3918680589 shutdown_connections 2024-09-16T15:49:47.810 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:47.810+0000 7f38c06c0640 1 -- 172.21.15.123:0/3918680589 >> 172.21.15.123:0/3918680589 conn(0x7f38b80d3ea0 msgr2=0x7f38b80d42b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:49:47.810 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:47.810+0000 7f38c06c0640 1 -- 172.21.15.123:0/3918680589 shutdown_connections 2024-09-16T15:49:47.811 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:47.810+0000 7f38c06c0640 1 -- 172.21.15.123:0/3918680589 wait complete. 2024-09-16T15:49:47.811 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:47.810+0000 7f38c06c0640 1 Processor -- start 2024-09-16T15:49:47.811 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:47.810+0000 7f38c06c0640 1 -- start start 2024-09-16T15:49:47.814 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:47.810+0000 7f38c06c0640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f38a4003a10 0x7f38b80c8b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:49:47.814 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:47.810+0000 7f38c06c0640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f38b80b36e0 0x7f38b8166870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:49:47.814 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:47.810+0000 7f38c06c0640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f38b80b3b60 0x7f38b8151cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:49:47.814 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:47.810+0000 7f38c06c0640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f38b80d1f20 con 0x7f38b80b3b60 2024-09-16T15:49:47.814 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:47.810+0000 7f38c06c0640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f38b80d1da0 con 0x7f38a4003a10 2024-09-16T15:49:47.814 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:47.810+0000 7f38c06c0640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f38b80d20a0 con 0x7f38b80b36e0 2024-09-16T15:49:47.814 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:47.814+0000 7f38bec5d640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f38b80b3b60 0x7f38b8151cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:49:47.814 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:47.814+0000 7f38bec5d640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f38b80b3b60 0x7f38b8151cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:57468/0 (socket says 172.21.15.123:57468) 2024-09-16T15:49:47.814 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:47.814+0000 7f38bec5d640 1 -- 172.21.15.123:0/1544274897 learned_addr learned my addr 172.21.15.123:0/1544274897 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:49:47.814 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:47.814+0000 7f38be45c640 1 --2- 172.21.15.123:0/1544274897 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f38a4003a10 0x7f38b80c8b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:49:47.815 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:47.814+0000 7f38bdc5b640 1 --2- 172.21.15.123:0/1544274897 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f38b80b36e0 0x7f38b8166870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:49:47.815 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:47.814+0000 7f38bec5d640 1 -- 172.21.15.123:0/1544274897 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f38b80b36e0 msgr2=0x7f38b8166870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:47.815 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:47.814+0000 7f38bec5d640 1 --2- 172.21.15.123:0/1544274897 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f38b80b36e0 0x7f38b8166870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:47.815 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:47.814+0000 7f38bec5d640 1 -- 172.21.15.123:0/1544274897 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f38a4003a10 msgr2=0x7f38b80c8b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:47.815 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:47.814+0000 7f38bec5d640 1 --2- 172.21.15.123:0/1544274897 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f38a4003a10 0x7f38b80c8b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:47.815 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:47.814+0000 7f38bec5d640 1 -- 172.21.15.123:0/1544274897 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f38b80c7a70 con 0x7f38b80b3b60 2024-09-16T15:49:47.815 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:47.814+0000 7f38bdc5b640 1 --2- 172.21.15.123:0/1544274897 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f38b80b36e0 0x7f38b8166870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T15:49:47.815 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:47.814+0000 7f38be45c640 1 --2- 172.21.15.123:0/1544274897 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f38a4003a10 0x7f38b80c8b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T15:49:47.815 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:47.814+0000 7f38bec5d640 1 --2- 172.21.15.123:0/1544274897 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f38b80b3b60 0x7f38b8151cf0 secure :-1 s=READY pgs=1367 cs=0 l=1 rev1=1 crypto rx=0x7f38b00014f0 tx=0x7f38b000cbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:49:47.815 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:47.814+0000 7f38aeffd640 1 -- 172.21.15.123:0/1544274897 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38b0017070 con 0x7f38b80b3b60 2024-09-16T15:49:47.815 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:47.814+0000 7f38aeffd640 1 -- 172.21.15.123:0/1544274897 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f38b000a8c0 con 0x7f38b80b3b60 2024-09-16T15:49:47.815 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:47.814+0000 7f38c06c0640 1 -- 172.21.15.123:0/1544274897 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f38b80c7d50 con 0x7f38b80b3b60 2024-09-16T15:49:47.815 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:47.814+0000 7f38aeffd640 1 -- 172.21.15.123:0/1544274897 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38b000ad10 con 0x7f38b80b3b60 2024-09-16T15:49:47.816 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:47.814+0000 7f38c06c0640 1 -- 172.21.15.123:0/1544274897 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f38b80c8310 con 0x7f38b80b3b60 2024-09-16T15:49:47.816 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:47.814+0000 7f38c06c0640 1 -- 172.21.15.123:0/1544274897 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f38b80d22b0 con 0x7f38b80b3b60 2024-09-16T15:49:47.816 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:47.814+0000 7f38aeffd640 1 -- 172.21.15.123:0/1544274897 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 21) ==== 97641+0+0 (secure 0 0 0) 0x7f38b001a070 con 0x7f38b80b3b60 2024-09-16T15:49:47.817 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:47.814+0000 7f38aeffd640 1 --2- 172.21.15.123:0/1544274897 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f389c073520 0x7f389c0759e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:49:47.817 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:47.814+0000 7f38aeffd640 1 -- 172.21.15.123:0/1544274897 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(127..127 src has 1..127) ==== 4971+0+0 (secure 0 0 0) 0x7f38b0093fb0 con 0x7f38b80b3b60 2024-09-16T15:49:47.817 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:47.814+0000 7f38be45c640 1 --2- 172.21.15.123:0/1544274897 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f389c073520 0x7f389c0759e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:49:47.817 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:47.814+0000 7f38be45c640 1 --2- 172.21.15.123:0/1544274897 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f389c073520 0x7f389c0759e0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f38a80b0000 tx=0x7f38a80a7040 comp rx=0 tx=0).ready entity=mgr.4150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:49:47.819 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:47.818+0000 7f38aeffd640 1 -- 172.21.15.123:0/1544274897 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f38b00c30b0 con 0x7f38b80b3b60 2024-09-16T15:49:47.983 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:47.982+0000 7f38c06c0640 1 -- 172.21.15.123:0/1544274897 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "config dump"} v 0) -- 0x7f38b80cece0 con 0x7f38b80b3b60 2024-09-16T15:49:47.984 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:47.982+0000 7f38aeffd640 1 -- 172.21.15.123:0/1544274897 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "config dump"}]=0 v41) ==== 59+0+1998 (secure 0 0 0) 0x7f38b005ec20 con 0x7f38b80b3b60 2024-09-16T15:49:47.984 INFO:teuthology.orchestra.run.smithi123.stdout:WHO MASK LEVEL OPTION VALUE RO 2024-09-16T15:49:47.984 INFO:teuthology.orchestra.run.smithi123.stdout:mon advanced mon_op_complaint_time 120 2024-09-16T15:49:47.985 INFO:teuthology.orchestra.run.smithi123.stdout:mon dev mon_warn_on_pool_no_app false 2024-09-16T15:49:47.985 INFO:teuthology.orchestra.run.smithi123.stdout:osd advanced osd_op_complaint_time 180.000000 2024-09-16T15:49:47.985 INFO:teuthology.orchestra.run.smithi123.stdout:osd.0 basic osd_mclock_max_capacity_iops_ssd 4805.300677 2024-09-16T15:49:47.985 INFO:teuthology.orchestra.run.smithi123.stdout:osd.1 basic osd_mclock_max_capacity_iops_ssd 4806.240607 2024-09-16T15:49:47.985 INFO:teuthology.orchestra.run.smithi123.stdout:osd.2 basic osd_mclock_max_capacity_iops_ssd 4046.441686 2024-09-16T15:49:47.985 INFO:teuthology.orchestra.run.smithi123.stdout:osd.3 basic osd_mclock_max_capacity_iops_ssd 6882.576123 2024-09-16T15:49:47.985 INFO:teuthology.orchestra.run.smithi123.stdout:osd.4 basic osd_mclock_max_capacity_iops_ssd 6108.444971 2024-09-16T15:49:47.985 INFO:teuthology.orchestra.run.smithi123.stdout:osd.5 basic osd_mclock_max_capacity_iops_ssd 4995.096306 2024-09-16T15:49:47.985 INFO:teuthology.orchestra.run.smithi123.stdout:osd.6 basic osd_mclock_max_capacity_iops_ssd 6237.822095 2024-09-16T15:49:47.985 INFO:teuthology.orchestra.run.smithi123.stdout:osd.7 basic osd_mclock_max_capacity_iops_ssd 5202.385138 2024-09-16T15:49:47.985 INFO:teuthology.orchestra.run.smithi123.stdout:mds advanced debug_mds 20/20 2024-09-16T15:49:47.985 INFO:teuthology.orchestra.run.smithi123.stdout:mds advanced debug_mds_balancer 20/20 2024-09-16T15:49:47.986 INFO:teuthology.orchestra.run.smithi123.stdout:mds advanced debug_ms 2/2 2024-09-16T15:49:47.986 INFO:teuthology.orchestra.run.smithi123.stdout:mds dev mds_debug_frag true 2024-09-16T15:49:47.986 INFO:teuthology.orchestra.run.smithi123.stdout:mds dev mds_debug_scatterstat true 2024-09-16T15:49:47.986 INFO:teuthology.orchestra.run.smithi123.stdout:mds advanced mds_op_complaint_time 180.000000 2024-09-16T15:49:47.986 INFO:teuthology.orchestra.run.smithi123.stdout:mds dev mds_verify_scatter true 2024-09-16T15:49:47.986 INFO:teuthology.orchestra.run.smithi123.stdout:mds advanced osd_op_complaint_time 180.000000 2024-09-16T15:49:47.986 INFO:teuthology.orchestra.run.smithi123.stdout:mds advanced rados_mon_op_timeout 900 2024-09-16T15:49:47.986 INFO:teuthology.orchestra.run.smithi123.stdout:mds advanced rados_osd_op_timeout 900 2024-09-16T15:49:47.986 INFO:teuthology.orchestra.run.smithi123.stdout:client advanced client_mount_timeout 600 2024-09-16T15:49:47.986 INFO:teuthology.orchestra.run.smithi123.stdout:client advanced debug_client 20/20 2024-09-16T15:49:47.986 INFO:teuthology.orchestra.run.smithi123.stdout:client advanced debug_ms 1/1 2024-09-16T15:49:47.986 INFO:teuthology.orchestra.run.smithi123.stdout:client advanced rados_mon_op_timeout 900 2024-09-16T15:49:47.986 INFO:teuthology.orchestra.run.smithi123.stdout:client advanced rados_osd_op_timeout 900 2024-09-16T15:49:47.987 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:47.986+0000 7f38c06c0640 1 -- 172.21.15.123:0/1544274897 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f389c073520 msgr2=0x7f389c0759e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:47.987 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:47.986+0000 7f38c06c0640 1 --2- 172.21.15.123:0/1544274897 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f389c073520 0x7f389c0759e0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f38a80b0000 tx=0x7f38a80a7040 comp rx=0 tx=0).stop 2024-09-16T15:49:47.987 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:47.986+0000 7f38c06c0640 1 -- 172.21.15.123:0/1544274897 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f38b80b3b60 msgr2=0x7f38b8151cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:47.987 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:47.986+0000 7f38c06c0640 1 --2- 172.21.15.123:0/1544274897 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f38b80b3b60 0x7f38b8151cf0 secure :-1 s=READY pgs=1367 cs=0 l=1 rev1=1 crypto rx=0x7f38b00014f0 tx=0x7f38b000cbe0 comp rx=0 tx=0).stop 2024-09-16T15:49:47.987 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:47.986+0000 7f38c06c0640 1 -- 172.21.15.123:0/1544274897 shutdown_connections 2024-09-16T15:49:47.987 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:47.986+0000 7f38c06c0640 1 --2- 172.21.15.123:0/1544274897 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f389c073520 0x7f389c0759e0 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:47.987 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:47.986+0000 7f38c06c0640 1 --2- 172.21.15.123:0/1544274897 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f38b80b3b60 0x7f38b8151cf0 unknown :-1 s=CLOSED pgs=1367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:47.988 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:47.986+0000 7f38c06c0640 1 --2- 172.21.15.123:0/1544274897 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f38b80b36e0 0x7f38b8166870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:47.988 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:47.986+0000 7f38c06c0640 1 --2- 172.21.15.123:0/1544274897 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f38a4003a10 0x7f38b80c8b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:47.988 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:47.986+0000 7f38c06c0640 1 -- 172.21.15.123:0/1544274897 >> 172.21.15.123:0/1544274897 conn(0x7f38b80d3ea0 msgr2=0x7f38b815ff60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:49:47.988 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:47.986+0000 7f38c06c0640 1 -- 172.21.15.123:0/1544274897 shutdown_connections 2024-09-16T15:49:47.988 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:47.986+0000 7f38c06c0640 1 -- 172.21.15.123:0/1544274897 wait complete. 2024-09-16T15:49:48.002 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-09-16T15:49:48.130 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:48.130+0000 7f4753575640 1 -- 172.21.15.123:0/3731224089 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f474c0c3230 msgr2=0x7f474c0cedd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:48.130 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:48.130+0000 7f4753575640 1 --2- 172.21.15.123:0/3731224089 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f474c0c3230 0x7f474c0cedd0 secure :-1 s=READY pgs=1368 cs=0 l=1 rev1=1 crypto rx=0x7f474c0cdb70 tx=0x7f473c01c870 comp rx=0 tx=0).stop 2024-09-16T15:49:48.130 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:48.130+0000 7f4753575640 1 -- 172.21.15.123:0/3731224089 shutdown_connections 2024-09-16T15:49:48.130 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:48.130+0000 7f4753575640 1 --2- 172.21.15.123:0/3731224089 >> v2:172.21.15.184:3300/0 conn(0x7f474c0701e0 0x7f474c0cf310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:48.130 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:48.130+0000 7f4753575640 1 --2- 172.21.15.123:0/3731224089 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f474c0c3230 0x7f474c0cedd0 unknown :-1 s=CLOSED pgs=1368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:48.130 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:48.130+0000 7f4753575640 1 --2- 172.21.15.123:0/3731224089 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f474c15b9f0 0x7f474c06fca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:48.130 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:48.130+0000 7f4753575640 1 -- 172.21.15.123:0/3731224089 >> 172.21.15.123:0/3731224089 conn(0x7f474c0b1710 msgr2=0x7f474c0b1b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:49:48.131 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:48.130+0000 7f4753575640 1 -- 172.21.15.123:0/3731224089 shutdown_connections 2024-09-16T15:49:48.131 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:48.130+0000 7f4753575640 1 -- 172.21.15.123:0/3731224089 wait complete. 2024-09-16T15:49:48.133 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:48.130+0000 7f4753575640 1 Processor -- start 2024-09-16T15:49:48.133 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:48.130+0000 7f4753575640 1 -- start start 2024-09-16T15:49:48.133 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:48.130+0000 7f4753575640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f474c0701e0 0x7f474c14d720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:49:48.133 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:48.130+0000 7f4753575640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f474c0c3230 0x7f474c14d0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:49:48.133 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:48.130+0000 7f4753575640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f474c15b9f0 0x7f474c15de50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:49:48.133 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:48.130+0000 7f4753575640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f474c073350 con 0x7f474c0c3230 2024-09-16T15:49:48.133 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:48.130+0000 7f4753575640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f474c0731d0 con 0x7f474c0701e0 2024-09-16T15:49:48.133 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:48.130+0000 7f4753575640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f474c15f6b0 con 0x7f474c15b9f0 2024-09-16T15:49:48.133 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:48.130+0000 7f4750b10640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f474c0c3230 0x7f474c14d0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:49:48.134 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:48.130+0000 7f4750b10640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f474c0c3230 0x7f474c14d0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:57488/0 (socket says 172.21.15.123:57488) 2024-09-16T15:49:48.134 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:48.130+0000 7f4750b10640 1 -- 172.21.15.123:0/1945344785 learned_addr learned my addr 172.21.15.123:0/1945344785 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:49:48.134 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:48.130+0000 7f4751b12640 1 --2- 172.21.15.123:0/1945344785 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f474c15b9f0 0x7f474c15de50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:49:48.134 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:48.130+0000 7f4750b10640 1 -- 172.21.15.123:0/1945344785 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f474c15b9f0 msgr2=0x7f474c15de50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:48.134 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:48.130+0000 7f4751311640 1 --2- 172.21.15.123:0/1945344785 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f474c0701e0 0x7f474c14d720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:49:48.134 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:48.130+0000 7f4750b10640 1 --2- 172.21.15.123:0/1945344785 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f474c15b9f0 0x7f474c15de50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:48.134 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:48.130+0000 7f4750b10640 1 -- 172.21.15.123:0/1945344785 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f474c0701e0 msgr2=0x7f474c14d720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:48.134 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:48.130+0000 7f4750b10640 1 --2- 172.21.15.123:0/1945344785 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f474c0701e0 0x7f474c14d720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:48.134 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:48.130+0000 7f4750b10640 1 -- 172.21.15.123:0/1945344785 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f474c15ec20 con 0x7f474c0c3230 2024-09-16T15:49:48.134 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:48.130+0000 7f4750b10640 1 --2- 172.21.15.123:0/1945344785 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f474c0c3230 0x7f474c14d0a0 secure :-1 s=READY pgs=1369 cs=0 l=1 rev1=1 crypto rx=0x7f473c01af20 tx=0x7f473c004040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:49:48.134 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:48.130+0000 7f4741ffb640 1 -- 172.21.15.123:0/1945344785 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f473c0ba020 con 0x7f474c0c3230 2024-09-16T15:49:48.134 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:48.130+0000 7f4741ffb640 1 -- 172.21.15.123:0/1945344785 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f473c005030 con 0x7f474c0c3230 2024-09-16T15:49:48.134 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:48.130+0000 7f4753575640 1 -- 172.21.15.123:0/1945344785 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f474c15ef00 con 0x7f474c0c3230 2024-09-16T15:49:48.135 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:48.130+0000 7f4741ffb640 1 -- 172.21.15.123:0/1945344785 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f473c0b2040 con 0x7f474c0c3230 2024-09-16T15:49:48.135 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:48.130+0000 7f4753575640 1 -- 172.21.15.123:0/1945344785 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f474c15c930 con 0x7f474c0c3230 2024-09-16T15:49:48.135 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:48.134+0000 7f4753575640 1 -- 172.21.15.123:0/1945344785 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f474c06fca0 con 0x7f474c0c3230 2024-09-16T15:49:48.135 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:48.134+0000 7f4741ffb640 1 -- 172.21.15.123:0/1945344785 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 21) ==== 97641+0+0 (secure 0 0 0) 0x7f473c0046d0 con 0x7f474c0c3230 2024-09-16T15:49:48.136 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:48.134+0000 7f4741ffb640 1 --2- 172.21.15.123:0/1945344785 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f4730073520 0x7f47300759e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:49:48.136 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:48.134+0000 7f4751311640 1 --2- 172.21.15.123:0/1945344785 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f4730073520 0x7f47300759e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:49:48.136 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:48.134+0000 7f4741ffb640 1 -- 172.21.15.123:0/1945344785 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(127..127 src has 1..127) ==== 4971+0+0 (secure 0 0 0) 0x7f473c12e580 con 0x7f474c0c3230 2024-09-16T15:49:48.136 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:48.134+0000 7f4751311640 1 --2- 172.21.15.123:0/1945344785 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f4730073520 0x7f47300759e0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f474c0b4790 tx=0x7f4748008040 comp rx=0 tx=0).ready entity=mgr.4150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:49:48.138 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:48.138+0000 7f4741ffb640 1 -- 172.21.15.123:0/1945344785 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f473c016490 con 0x7f474c0c3230 2024-09-16T15:49:48.304 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:48.302+0000 7f4753575640 1 -- 172.21.15.123:0/1945344785 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) -- 0x7f474c073a90 con 0x7f474c0c3230 2024-09-16T15:49:48.311 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:48.310+0000 7f4741ffb640 1 -- 172.21.15.123:0/1945344785 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v42)=0 v42) ==== 122+0+0 (secure 0 0 0) 0x7f473c12e0c0 con 0x7f474c0c3230 2024-09-16T15:49:48.313 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:48.314+0000 7f4753575640 1 -- 172.21.15.123:0/1945344785 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f4730073520 msgr2=0x7f47300759e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:48.313 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:48.314+0000 7f4753575640 1 --2- 172.21.15.123:0/1945344785 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f4730073520 0x7f47300759e0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f474c0b4790 tx=0x7f4748008040 comp rx=0 tx=0).stop 2024-09-16T15:49:48.313 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:48.314+0000 7f4753575640 1 -- 172.21.15.123:0/1945344785 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f474c0c3230 msgr2=0x7f474c14d0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:48.313 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:48.314+0000 7f4753575640 1 --2- 172.21.15.123:0/1945344785 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f474c0c3230 0x7f474c14d0a0 secure :-1 s=READY pgs=1369 cs=0 l=1 rev1=1 crypto rx=0x7f473c01af20 tx=0x7f473c004040 comp rx=0 tx=0).stop 2024-09-16T15:49:48.313 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:48.314+0000 7f4753575640 1 -- 172.21.15.123:0/1945344785 shutdown_connections 2024-09-16T15:49:48.314 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:48.314+0000 7f4753575640 1 --2- 172.21.15.123:0/1945344785 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f4730073520 0x7f47300759e0 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:48.314 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:48.314+0000 7f4753575640 1 --2- 172.21.15.123:0/1945344785 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f474c15b9f0 0x7f474c15de50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:48.314 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:48.314+0000 7f4753575640 1 --2- 172.21.15.123:0/1945344785 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f474c0c3230 0x7f474c14d0a0 unknown :-1 s=CLOSED pgs=1369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:48.314 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:48.314+0000 7f4753575640 1 --2- 172.21.15.123:0/1945344785 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f474c0701e0 0x7f474c14d720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:48.314 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:48.314+0000 7f4753575640 1 -- 172.21.15.123:0/1945344785 >> 172.21.15.123:0/1945344785 conn(0x7f474c0b1710 msgr2=0x7f474c15a7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:49:48.314 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:48.314+0000 7f4753575640 1 -- 172.21.15.123:0/1945344785 shutdown_connections 2024-09-16T15:49:48.314 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:48.314+0000 7f4753575640 1 -- 172.21.15.123:0/1945344785 wait complete. 2024-09-16T15:49:48.329 INFO:teuthology.nuke:Clearing teuthology firewall rules... 2024-09-16T15:49:48.329 DEBUG:teuthology.orchestra.run.smithi123:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-09-16T15:49:48.406 DEBUG:teuthology.orchestra.run.smithi184:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-09-16T15:49:48.483 INFO:teuthology.nuke:Cleared teuthology firewall rules. 2024-09-16T15:49:48.483 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2024-09-16T15:49:48.483 DEBUG:teuthology.orchestra.run.smithi123:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-16T15:49:48.489 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16336468k,nr_inodes=4084117,mode=755,inode64 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3274684k,mode=755,inode64 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev,inode64 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=17082 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3274680k,nr_inodes=818670,mode=700,uid=1000,gid=1271,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3274684k,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-09-16T15:49:48.489 DEBUG:teuthology.orchestra.run.smithi184:> set -ex 2024-09-16T15:49:48.489 DEBUG:teuthology.orchestra.run.smithi184:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-16T15:49:48.495 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16336488k,nr_inodes=4084122,mode=755,inode64 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3274688k,mode=755,inode64 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev,inode64 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=17834 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3274684k,nr_inodes=818671,mode=700,uid=1000,gid=1271,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3274688k,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-09-16T15:49:48.495 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2024-09-16T15:49:48.495 DEBUG:teuthology.orchestra.run.smithi123:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-16T15:49:48.537 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16336468k,nr_inodes=4084117,mode=755,inode64 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3274684k,mode=755,inode64 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev,inode64 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=17082 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3274680k,nr_inodes=818670,mode=700,uid=1000,gid=1271,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3274684k,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-09-16T15:49:48.537 DEBUG:teuthology.orchestra.run.smithi184:> set -ex 2024-09-16T15:49:48.538 DEBUG:teuthology.orchestra.run.smithi184:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-16T15:49:48.543 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16336488k,nr_inodes=4084122,mode=755,inode64 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3274688k,mode=755,inode64 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev,inode64 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=17834 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3274684k,nr_inodes=818671,mode=700,uid=1000,gid=1271,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3274688k,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-09-16T15:49:48.543 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-09-16T15:49:48.544 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-16T15:49:48.713 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:48.714+0000 7f9b76f5f640 1 -- 172.21.15.123:0/1249299355 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9b58004030 msgr2=0x7f9b58004810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:48.713 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:48.714+0000 7f9b76f5f640 1 --2- 172.21.15.123:0/1249299355 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9b58004030 0x7f9b58004810 secure :-1 s=READY pgs=1371 cs=0 l=1 rev1=1 crypto rx=0x7f9b640016f0 tx=0x7f9b6401c780 comp rx=0 tx=0).stop 2024-09-16T15:49:48.713 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:48.714+0000 7f9b754fc640 1 -- 172.21.15.123:0/1249299355 reap_dead start 2024-09-16T15:49:48.714 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:48.714+0000 7f9b76f5f640 1 -- 172.21.15.123:0/1249299355 shutdown_connections 2024-09-16T15:49:48.714 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:48.714+0000 7f9b76f5f640 1 -- 172.21.15.123:0/1249299355 >> 172.21.15.123:0/1249299355 conn(0x7f9b700cfb10 msgr2=0x7f9b700cff20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:49:48.714 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:48.714+0000 7f9b76f5f640 1 -- 172.21.15.123:0/1249299355 shutdown_connections 2024-09-16T15:49:48.714 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:48.714+0000 7f9b76f5f640 1 -- 172.21.15.123:0/1249299355 wait complete. 2024-09-16T15:49:48.716 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:48.714+0000 7f9b76f5f640 1 Processor -- start 2024-09-16T15:49:48.716 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:48.714+0000 7f9b76f5f640 1 -- start start 2024-09-16T15:49:48.716 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:48.714+0000 7f9b76f5f640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9b58004030 0x7f9b70148ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:49:48.716 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:48.714+0000 7f9b76f5f640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f9b70162a20 0x7f9b70165d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:49:48.716 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:48.714+0000 7f9b76f5f640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f9b70162ea0 0x7f9b70166270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:49:48.716 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:48.714+0000 7f9b76f5f640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f9b70167bd0 con 0x7f9b58004030 2024-09-16T15:49:48.717 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:48.714+0000 7f9b76f5f640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f9b70167a50 con 0x7f9b70162ea0 2024-09-16T15:49:48.717 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:48.714+0000 7f9b76f5f640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f9b70167d50 con 0x7f9b70162a20 2024-09-16T15:49:48.717 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:48.714+0000 7f9b74cfb640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9b58004030 0x7f9b70148ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:49:48.717 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:48.714+0000 7f9b74cfb640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9b58004030 0x7f9b70148ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:57506/0 (socket says 172.21.15.123:57506) 2024-09-16T15:49:48.717 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:48.714+0000 7f9b74cfb640 1 -- 172.21.15.123:0/780792723 learned_addr learned my addr 172.21.15.123:0/780792723 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:49:48.717 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:48.714+0000 7f9b754fc640 1 --2- 172.21.15.123:0/780792723 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f9b70162ea0 0x7f9b70166270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:49:48.717 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:48.714+0000 7f9b6ffff640 1 --2- 172.21.15.123:0/780792723 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f9b70162a20 0x7f9b70165d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:49:48.717 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:48.714+0000 7f9b74cfb640 1 -- 172.21.15.123:0/780792723 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f9b70162a20 msgr2=0x7f9b70165d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:48.717 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:48.714+0000 7f9b74cfb640 1 --2- 172.21.15.123:0/780792723 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f9b70162a20 0x7f9b70165d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:48.717 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:48.714+0000 7f9b74cfb640 1 -- 172.21.15.123:0/780792723 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f9b70162ea0 msgr2=0x7f9b70166270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:48.717 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:48.714+0000 7f9b74cfb640 1 --2- 172.21.15.123:0/780792723 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f9b70162ea0 0x7f9b70166270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:48.717 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:48.714+0000 7f9b74cfb640 1 -- 172.21.15.123:0/780792723 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9b700c6c80 con 0x7f9b58004030 2024-09-16T15:49:48.718 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:48.714+0000 7f9b74cfb640 1 --2- 172.21.15.123:0/780792723 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9b58004030 0x7f9b70148ed0 secure :-1 s=READY pgs=1372 cs=0 l=1 rev1=1 crypto rx=0x7f9b60010a10 tx=0x7f9b60010ed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:49:48.718 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:48.714+0000 7f9b6d7fa640 1 -- 172.21.15.123:0/780792723 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b6000ec80 con 0x7f9b58004030 2024-09-16T15:49:48.718 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:48.714+0000 7f9b6d7fa640 1 -- 172.21.15.123:0/780792723 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f9b6000ee20 con 0x7f9b58004030 2024-09-16T15:49:48.718 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:48.714+0000 7f9b76f5f640 1 -- 172.21.15.123:0/780792723 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9b700c6f60 con 0x7f9b58004030 2024-09-16T15:49:48.718 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:48.714+0000 7f9b6d7fa640 1 -- 172.21.15.123:0/780792723 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b60005280 con 0x7f9b58004030 2024-09-16T15:49:48.718 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:48.714+0000 7f9b76f5f640 1 -- 172.21.15.123:0/780792723 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9b700c7520 con 0x7f9b58004030 2024-09-16T15:49:48.719 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:48.718+0000 7f9b76f5f640 1 -- 172.21.15.123:0/780792723 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9b70073ef0 con 0x7f9b58004030 2024-09-16T15:49:48.719 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:48.718+0000 7f9b6d7fa640 1 -- 172.21.15.123:0/780792723 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 21) ==== 97641+0+0 (secure 0 0 0) 0x7f9b60005420 con 0x7f9b58004030 2024-09-16T15:49:48.719 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:48.718+0000 7f9b6d7fa640 1 --2- 172.21.15.123:0/780792723 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f9b5c073520 0x7f9b5c0759e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:49:48.719 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:48.718+0000 7f9b6d7fa640 1 -- 172.21.15.123:0/780792723 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(127..127 src has 1..127) ==== 4971+0+0 (secure 0 0 0) 0x7f9b600955e0 con 0x7f9b58004030 2024-09-16T15:49:48.719 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:48.718+0000 7f9b6ffff640 1 --2- 172.21.15.123:0/780792723 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f9b5c073520 0x7f9b5c0759e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:49:48.719 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:48.718+0000 7f9b6ffff640 1 --2- 172.21.15.123:0/780792723 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f9b5c073520 0x7f9b5c0759e0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f9b640016c0 tx=0x7f9b640a8040 comp rx=0 tx=0).ready entity=mgr.4150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:49:48.722 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:48.722+0000 7f9b6d7fa640 1 -- 172.21.15.123:0/780792723 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f9b600600c0 con 0x7f9b58004030 2024-09-16T15:49:48.932 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:48.930+0000 7f9b76f5f640 1 -- 172.21.15.123:0/780792723 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f9b7015f660 con 0x7f9b58004030 2024-09-16T15:49:48.932 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:48.934+0000 7f9b6d7fa640 1 -- 172.21.15.123:0/780792723 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 79 v79) ==== 94+0+3920 (secure 0 0 0) 0x7f9b6005f9c0 con 0x7f9b58004030 2024-09-16T15:49:48.933 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-09-16T15:49:48.933 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":79,"btime":"2024-09-16T15:49:42:630370+0000","default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6079,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6833/3932980815","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":3932980815},{"type":"v1","addr":"172.21.15.184:6833","nonce":3932980815}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":72},{"gid":6082,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6835/1809728014","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":1809728014},{"type":"v1","addr":"172.21.15.184:6835","nonce":1809728014}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":72},{"gid":7551,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6834/892095590","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6832","nonce":892095590},{"type":"v1","addr":"172.21.15.123:6834","nonce":892095590}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":72},{"gid":7863,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6835/2341780134","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6833","nonce":2341780134},{"type":"v1","addr":"172.21.15.123:6835","nonce":2341780134}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":79}],"filesystems":[]} 2024-09-16T15:49:48.933 INFO:teuthology.orchestra.run.smithi123.stderr:dumped fsmap epoch 79 2024-09-16T15:49:48.935 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:48.934+0000 7f9b76f5f640 1 -- 172.21.15.123:0/780792723 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f9b5c073520 msgr2=0x7f9b5c0759e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:48.935 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:48.934+0000 7f9b76f5f640 1 --2- 172.21.15.123:0/780792723 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f9b5c073520 0x7f9b5c0759e0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f9b640016c0 tx=0x7f9b640a8040 comp rx=0 tx=0).stop 2024-09-16T15:49:48.935 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:48.934+0000 7f9b76f5f640 1 -- 172.21.15.123:0/780792723 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9b58004030 msgr2=0x7f9b70148ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:48.935 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:48.934+0000 7f9b76f5f640 1 --2- 172.21.15.123:0/780792723 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9b58004030 0x7f9b70148ed0 secure :-1 s=READY pgs=1372 cs=0 l=1 rev1=1 crypto rx=0x7f9b60010a10 tx=0x7f9b60010ed0 comp rx=0 tx=0).stop 2024-09-16T15:49:48.935 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:48.934+0000 7f9b76f5f640 1 -- 172.21.15.123:0/780792723 shutdown_connections 2024-09-16T15:49:48.935 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:48.934+0000 7f9b76f5f640 1 --2- 172.21.15.123:0/780792723 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f9b5c073520 0x7f9b5c0759e0 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:48.936 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:48.934+0000 7f9b76f5f640 1 --2- 172.21.15.123:0/780792723 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f9b70162ea0 0x7f9b70166270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:48.936 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:48.934+0000 7f9b76f5f640 1 --2- 172.21.15.123:0/780792723 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f9b70162a20 0x7f9b70165d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:48.936 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:48.934+0000 7f9b76f5f640 1 --2- 172.21.15.123:0/780792723 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9b58004030 0x7f9b70148ed0 unknown :-1 s=CLOSED pgs=1372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:48.936 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:48.934+0000 7f9b76f5f640 1 -- 172.21.15.123:0/780792723 >> 172.21.15.123:0/780792723 conn(0x7f9b700cfb10 msgr2=0x7f9b700ca140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:49:48.936 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:48.934+0000 7f9b76f5f640 1 -- 172.21.15.123:0/780792723 shutdown_connections 2024-09-16T15:49:48.936 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:48.934+0000 7f9b76f5f640 1 -- 172.21.15.123:0/780792723 wait complete. 2024-09-16T15:49:48.952 INFO:tasks.ceph.mds.a:Restarting daemon 2024-09-16T15:49:48.952 INFO:tasks.ceph.mds.a:Stopping old one... 2024-09-16T15:49:48.952 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-09-16T15:49:48.953 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T15:49:48.953 INFO:tasks.ceph.mds.c:Restarting daemon 2024-09-16T15:49:48.953 INFO:tasks.ceph.mds.c:Stopping old one... 2024-09-16T15:49:48.953 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-09-16T15:49:48.953 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T15:49:48.953 INFO:tasks.ceph.mds.b:Restarting daemon 2024-09-16T15:49:48.953 INFO:tasks.ceph.mds.b:Stopping old one... 2024-09-16T15:49:48.954 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-09-16T15:49:48.954 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T15:49:48.954 INFO:tasks.ceph.mds.d:Restarting daemon 2024-09-16T15:49:48.954 INFO:tasks.ceph.mds.d:Stopping old one... 2024-09-16T15:49:48.954 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-09-16T15:49:48.954 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T15:49:48.964 INFO:tasks.ceph.mds.a:Stopped 2024-09-16T15:49:48.964 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-09-16T15:49:48.965 INFO:tasks.ceph.mds.c:Stopped 2024-09-16T15:49:48.965 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-09-16T15:49:48.969 INFO:tasks.ceph.mds.b:Stopped 2024-09-16T15:49:48.969 DEBUG:teuthology.orchestra.run.smithi184:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-09-16T15:49:48.975 INFO:tasks.ceph.mds.d:Stopped 2024-09-16T15:49:48.975 DEBUG:teuthology.orchestra.run.smithi184:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-09-16T15:49:49.005 INFO:tasks.ceph.mds.a:Started 2024-09-16T15:49:49.006 DEBUG:teuthology.parallel:result is None 2024-09-16T15:49:49.006 INFO:tasks.ceph.mds.c:Started 2024-09-16T15:49:49.006 DEBUG:teuthology.parallel:result is None 2024-09-16T15:49:49.017 INFO:tasks.ceph.mds.b:Started 2024-09-16T15:49:49.018 DEBUG:teuthology.parallel:result is None 2024-09-16T15:49:49.019 INFO:tasks.ceph.mds.d:Started 2024-09-16T15:49:49.019 DEBUG:teuthology.parallel:result is None 2024-09-16T15:49:49.019 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-09-16T15:49:49.063 INFO:tasks.ceph.mds.c.smithi123.stdout:starting mds.c at 2024-09-16T15:49:49.073 INFO:tasks.ceph.mds.a.smithi123.stdout:starting mds.a at 2024-09-16T15:49:49.092 INFO:tasks.ceph.mds.d.smithi184.stdout:starting mds.d at 2024-09-16T15:49:49.092 INFO:tasks.ceph.mds.b.smithi184.stdout:starting mds.b at 2024-09-16T15:49:49.131 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:49.130+0000 7f224226a640 1 -- 172.21.15.123:0/1836725197 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f223c15f720 msgr2=0x7f223c0cd7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:49.131 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:49.130+0000 7f224226a640 1 --2- 172.21.15.123:0/1836725197 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f223c15f720 0x7f223c0cd7e0 secure :-1 s=READY pgs=1379 cs=0 l=1 rev1=1 crypto rx=0x7f2234009c80 tx=0x7f223401cda0 comp rx=0 tx=0).stop 2024-09-16T15:49:49.132 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:49.130+0000 7f224226a640 1 -- 172.21.15.123:0/1836725197 shutdown_connections 2024-09-16T15:49:49.132 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:49.130+0000 7f224226a640 1 --2- 172.21.15.123:0/1836725197 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f223c15f720 0x7f223c0cd7e0 unknown :-1 s=CLOSED pgs=1379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:49.132 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:49.130+0000 7f224226a640 1 -- 172.21.15.123:0/1836725197 >> 172.21.15.123:0/1836725197 conn(0x7f223c0d1c70 msgr2=0x7f223c0d2050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:49:49.132 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:49.130+0000 7f224226a640 1 -- 172.21.15.123:0/1836725197 shutdown_connections 2024-09-16T15:49:49.132 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:49.130+0000 7f224226a640 1 -- 172.21.15.123:0/1836725197 wait complete. 2024-09-16T15:49:49.132 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:49.130+0000 7f224226a640 1 Processor -- start 2024-09-16T15:49:49.133 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:49.134+0000 7f224226a640 1 -- start start 2024-09-16T15:49:49.134 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:49.134+0000 7f224226a640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f223c0b0f40 0x7f223c14cd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:49:49.134 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:49.134+0000 7f224226a640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f223c15f340 0x7f223c1f9730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:49:49.135 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:49.134+0000 7f224226a640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f223c15f720 0x7f223c0afb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:49:49.135 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:49.134+0000 7f224226a640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f223c16a870 con 0x7f223c15f340 2024-09-16T15:49:49.135 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:49.134+0000 7f224226a640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f223c16a6f0 con 0x7f223c15f720 2024-09-16T15:49:49.135 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:49.134+0000 7f224226a640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f223c16a9f0 con 0x7f223c0b0f40 2024-09-16T15:49:49.135 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:49.134+0000 7f223b7fe640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f223c15f340 0x7f223c1f9730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:49:49.135 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:49.134+0000 7f223b7fe640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f223c15f340 0x7f223c1f9730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:57548/0 (socket says 172.21.15.123:57548) 2024-09-16T15:49:49.135 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:49.134+0000 7f223b7fe640 1 -- 172.21.15.123:0/1614285433 learned_addr learned my addr 172.21.15.123:0/1614285433 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:49:49.135 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:49.134+0000 7f223bfff640 1 --2- 172.21.15.123:0/1614285433 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f223c0b0f40 0x7f223c14cd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:49:49.135 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:49.134+0000 7f2240807640 1 --2- 172.21.15.123:0/1614285433 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f223c15f720 0x7f223c0afb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:49:49.135 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:49.134+0000 7f223b7fe640 1 -- 172.21.15.123:0/1614285433 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f223c0b0f40 msgr2=0x7f223c14cd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:49.135 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:49.134+0000 7f223b7fe640 1 --2- 172.21.15.123:0/1614285433 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f223c0b0f40 0x7f223c14cd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:49.135 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:49.134+0000 7f223b7fe640 1 -- 172.21.15.123:0/1614285433 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f223c15f720 msgr2=0x7f223c0afb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:49.135 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:49.134+0000 7f223b7fe640 1 --2- 172.21.15.123:0/1614285433 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f223c15f720 0x7f223c0afb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:49.135 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:49.134+0000 7f223b7fe640 1 -- 172.21.15.123:0/1614285433 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f223c1fa0a0 con 0x7f223c15f340 2024-09-16T15:49:49.136 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:49.134+0000 7f223b7fe640 1 --2- 172.21.15.123:0/1614285433 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f223c15f340 0x7f223c1f9730 secure :-1 s=READY pgs=1380 cs=0 l=1 rev1=1 crypto rx=0x7f222c00c430 tx=0x7f222c00c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:49:49.136 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:49.134+0000 7f2238ff9640 1 -- 172.21.15.123:0/1614285433 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f222c017070 con 0x7f223c15f340 2024-09-16T15:49:49.136 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:49.134+0000 7f2238ff9640 1 -- 172.21.15.123:0/1614285433 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f222c003600 con 0x7f223c15f340 2024-09-16T15:49:49.136 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:49.134+0000 7f224226a640 1 -- 172.21.15.123:0/1614285433 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f223c1fa380 con 0x7f223c15f340 2024-09-16T15:49:49.136 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:49.134+0000 7f2238ff9640 1 -- 172.21.15.123:0/1614285433 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f222c003a50 con 0x7f223c15f340 2024-09-16T15:49:49.136 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:49.134+0000 7f224226a640 1 -- 172.21.15.123:0/1614285433 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f223c1f9c70 con 0x7f223c15f340 2024-09-16T15:49:49.136 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:49.134+0000 7f224226a640 1 -- 172.21.15.123:0/1614285433 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f223c0ce7b0 con 0x7f223c15f340 2024-09-16T15:49:49.137 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:49.134+0000 7f2238ff9640 1 -- 172.21.15.123:0/1614285433 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 21) ==== 97641+0+0 (secure 0 0 0) 0x7f222c01a070 con 0x7f223c15f340 2024-09-16T15:49:49.137 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:49.134+0000 7f2238ff9640 1 --2- 172.21.15.123:0/1614285433 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f221c0734f0 0x7f221c0759b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:49:49.137 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:49.134+0000 7f223bfff640 1 --2- 172.21.15.123:0/1614285433 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f221c0734f0 0x7f221c0759b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:49:49.137 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:49.134+0000 7f2238ff9640 1 -- 172.21.15.123:0/1614285433 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(127..127 src has 1..127) ==== 4971+0+0 (secure 0 0 0) 0x7f222c093980 con 0x7f223c15f340 2024-09-16T15:49:49.137 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:49.138+0000 7f223bfff640 1 --2- 172.21.15.123:0/1614285433 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f221c0734f0 0x7f221c0759b0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f2224007fd0 tx=0x7f2224009040 comp rx=0 tx=0).ready entity=mgr.4150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:49:49.140 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:49.142+0000 7f2238ff9640 1 -- 172.21.15.123:0/1614285433 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f222c05dea0 con 0x7f223c15f340 2024-09-16T15:49:49.323 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:49.322+0000 7f224226a640 1 -- 172.21.15.123:0/1614285433 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) -- 0x7f223c155850 con 0x7f223c15f340 2024-09-16T15:49:49.697 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:49.698+0000 7f2238ff9640 1 -- 172.21.15.123:0/1614285433 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v129) ==== 97+0+0 (secure 0 0 0) 0x7f222c0628d0 con 0x7f223c15f340 2024-09-16T15:49:49.697 INFO:teuthology.orchestra.run.smithi123.stderr: removed all blocklist entries 2024-09-16T15:49:49.701 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:49.702+0000 7f224226a640 1 -- 172.21.15.123:0/1614285433 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f221c0734f0 msgr2=0x7f221c0759b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:49.701 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:49.702+0000 7f224226a640 1 --2- 172.21.15.123:0/1614285433 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f221c0734f0 0x7f221c0759b0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f2224007fd0 tx=0x7f2224009040 comp rx=0 tx=0).stop 2024-09-16T15:49:49.701 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:49.702+0000 7f224226a640 1 -- 172.21.15.123:0/1614285433 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f223c15f340 msgr2=0x7f223c1f9730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:49.701 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:49.702+0000 7f224226a640 1 --2- 172.21.15.123:0/1614285433 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f223c15f340 0x7f223c1f9730 secure :-1 s=READY pgs=1380 cs=0 l=1 rev1=1 crypto rx=0x7f222c00c430 tx=0x7f222c00c8f0 comp rx=0 tx=0).stop 2024-09-16T15:49:49.702 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:49.702+0000 7f224226a640 1 -- 172.21.15.123:0/1614285433 shutdown_connections 2024-09-16T15:49:49.702 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:49.702+0000 7f224226a640 1 --2- 172.21.15.123:0/1614285433 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f221c0734f0 0x7f221c0759b0 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:49.702 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:49.702+0000 7f224226a640 1 --2- 172.21.15.123:0/1614285433 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f223c15f720 0x7f223c0afb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:49.702 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:49.702+0000 7f224226a640 1 --2- 172.21.15.123:0/1614285433 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f223c15f340 0x7f223c1f9730 unknown :-1 s=CLOSED pgs=1380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:49.702 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:49.702+0000 7f224226a640 1 --2- 172.21.15.123:0/1614285433 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f223c0b0f40 0x7f223c14cd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:49.702 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:49.702+0000 7f224226a640 1 -- 172.21.15.123:0/1614285433 >> 172.21.15.123:0/1614285433 conn(0x7f223c0d1c70 msgr2=0x7f223c0cfb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:49:49.702 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:49.702+0000 7f224226a640 1 -- 172.21.15.123:0/1614285433 shutdown_connections 2024-09-16T15:49:49.702 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:49.702+0000 7f224226a640 1 -- 172.21.15.123:0/1614285433 wait complete. 2024-09-16T15:49:49.727 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-09-16T15:49:49.902 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:49.902+0000 7f85aa877640 1 -- 172.21.15.123:0/3047800784 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f85a40b25a0 msgr2=0x7f85a40d1c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:49.902 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:49.902+0000 7f85aa877640 1 --2- 172.21.15.123:0/3047800784 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f85a40b25a0 0x7f85a40d1c10 secure :-1 s=READY pgs=1381 cs=0 l=1 rev1=1 crypto rx=0x7f858c009c80 tx=0x7f858c01ccb0 comp rx=0 tx=0).stop 2024-09-16T15:49:49.902 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:49.902+0000 7f85aa877640 1 -- 172.21.15.123:0/3047800784 shutdown_connections 2024-09-16T15:49:49.902 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:49.902+0000 7f85aa877640 1 --2- 172.21.15.123:0/3047800784 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f85a40b25a0 0x7f85a40d1c10 unknown :-1 s=CLOSED pgs=1381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:49.902 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:49.902+0000 7f85aa877640 1 --2- 172.21.15.123:0/3047800784 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f85a415bda0 0x7f85a40d0310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:49.902 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:49.902+0000 7f85aa877640 1 -- 172.21.15.123:0/3047800784 >> 172.21.15.123:0/3047800784 conn(0x7f85a4074bd0 msgr2=0x7f85a409aa40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:49:49.903 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:49.902+0000 7f85aa877640 1 -- 172.21.15.123:0/3047800784 shutdown_connections 2024-09-16T15:49:49.903 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:49.902+0000 7f85aa877640 1 -- 172.21.15.123:0/3047800784 wait complete. 2024-09-16T15:49:49.903 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:49.902+0000 7f85aa877640 1 Processor -- start 2024-09-16T15:49:49.903 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:49.902+0000 7f85aa877640 1 -- start start 2024-09-16T15:49:49.905 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:49.902+0000 7f85aa877640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f85a40b25a0 0x7f85a4154e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:49:49.905 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:49.902+0000 7f85aa877640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f85a40d0850 0x7f85a4158660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:49:49.905 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:49.902+0000 7f85aa877640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f85a415bda0 0x7f85a414e3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:49:49.906 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:49.902+0000 7f85aa877640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f85a4169250 con 0x7f85a40d0850 2024-09-16T15:49:49.906 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:49.902+0000 7f85aa877640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f85a41690d0 con 0x7f85a415bda0 2024-09-16T15:49:49.906 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:49.902+0000 7f85aa877640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f85a41693d0 con 0x7f85a40b25a0 2024-09-16T15:49:49.906 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:49.902+0000 7f85a37fe640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f85a40d0850 0x7f85a4158660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:49:49.906 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:49.902+0000 7f85a37fe640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f85a40d0850 0x7f85a4158660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:57572/0 (socket says 172.21.15.123:57572) 2024-09-16T15:49:49.906 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:49.902+0000 7f85a37fe640 1 -- 172.21.15.123:0/1866988412 learned_addr learned my addr 172.21.15.123:0/1866988412 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:49:49.906 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:49.906+0000 7f85a3fff640 1 --2- 172.21.15.123:0/1866988412 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f85a40b25a0 0x7f85a4154e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:49:49.906 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:49.906+0000 7f85a8e14640 1 --2- 172.21.15.123:0/1866988412 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f85a415bda0 0x7f85a414e3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:49:49.906 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:49.906+0000 7f85a37fe640 1 -- 172.21.15.123:0/1866988412 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f85a40b25a0 msgr2=0x7f85a4154e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:49.906 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:49.906+0000 7f85a37fe640 1 --2- 172.21.15.123:0/1866988412 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f85a40b25a0 0x7f85a4154e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:49.906 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:49.906+0000 7f85a37fe640 1 -- 172.21.15.123:0/1866988412 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f85a415bda0 msgr2=0x7f85a414e3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:49.906 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:49.906+0000 7f85a37fe640 1 --2- 172.21.15.123:0/1866988412 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f85a415bda0 0x7f85a414e3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:49.906 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:49.906+0000 7f85a37fe640 1 -- 172.21.15.123:0/1866988412 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f85a4158ba0 con 0x7f85a40d0850 2024-09-16T15:49:49.906 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:49.906+0000 7f85a37fe640 1 --2- 172.21.15.123:0/1866988412 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f85a40d0850 0x7f85a4158660 secure :-1 s=READY pgs=1382 cs=0 l=1 rev1=1 crypto rx=0x7f858c009bd0 tx=0x7f858c003d70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:49:49.907 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:49.906+0000 7f85a0ff9640 1 -- 172.21.15.123:0/1866988412 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f858c0bb070 con 0x7f85a40d0850 2024-09-16T15:49:49.907 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:49.906+0000 7f85a0ff9640 1 -- 172.21.15.123:0/1866988412 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f858c004e00 con 0x7f85a40d0850 2024-09-16T15:49:49.907 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:49.906+0000 7f85aa877640 1 -- 172.21.15.123:0/1866988412 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f85a414a110 con 0x7f85a40d0850 2024-09-16T15:49:49.907 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:49.906+0000 7f85a0ff9640 1 -- 172.21.15.123:0/1866988412 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f858c0b2040 con 0x7f85a40d0850 2024-09-16T15:49:49.907 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:49.906+0000 7f85aa877640 1 -- 172.21.15.123:0/1866988412 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f85a414a6d0 con 0x7f85a40d0850 2024-09-16T15:49:49.908 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:49.906+0000 7f85aa877640 1 -- 172.21.15.123:0/1866988412 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f85a40d1c10 con 0x7f85a40d0850 2024-09-16T15:49:49.908 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:49.906+0000 7f85a0ff9640 1 -- 172.21.15.123:0/1866988412 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 21) ==== 97641+0+0 (secure 0 0 0) 0x7f858c0be070 con 0x7f85a40d0850 2024-09-16T15:49:49.908 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:49.906+0000 7f85a0ff9640 1 --2- 172.21.15.123:0/1866988412 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f85840734f0 0x7f85840759b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:49:49.908 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:49.906+0000 7f85a0ff9640 1 -- 172.21.15.123:0/1866988412 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(131..131 src has 1..131) ==== 4885+0+0 (secure 0 0 0) 0x7f858c12e650 con 0x7f85a40d0850 2024-09-16T15:49:49.908 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:49.906+0000 7f85a3fff640 1 --2- 172.21.15.123:0/1866988412 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f85840734f0 0x7f85840759b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:49:49.908 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:49.906+0000 7f85a3fff640 1 --2- 172.21.15.123:0/1866988412 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f85840734f0 0x7f85840759b0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f85940058d0 tx=0x7f8594005860 comp rx=0 tx=0).ready entity=mgr.4150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:49:49.911 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:49.910+0000 7f85a0ff9640 1 -- 172.21.15.123:0/1866988412 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f858c016990 con 0x7f85a40d0850 2024-09-16T15:49:50.149 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:50.150+0000 7f85aa877640 1 -- 172.21.15.123:0/1866988412 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) -- 0x7f85a4159550 con 0x7f85a40d0850 2024-09-16T15:49:50.150 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:50.150+0000 7f85a0ff9640 1 -- 172.21.15.123:0/1866988412 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v25) ==== 80+0+5304 (secure 0 0 0) 0x7f858c0f9210 con 0x7f85a40d0850 2024-09-16T15:49:50.150 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-09-16T15:49:50.150 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2024-09-16T15:49:50.150 INFO:teuthology.orchestra.run.smithi123.stdout: "auth_dump": [ 2024-09-16T15:49:50.151 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-09-16T15:49:50.151 INFO:teuthology.orchestra.run.smithi123.stdout: "entity": "mds.a", 2024-09-16T15:49:50.151 INFO:teuthology.orchestra.run.smithi123.stdout: "key": "AQC0UehmbZpOJBAAaYGyDcWrt4k0FS5P/bb+4A==", 2024-09-16T15:49:50.151 INFO:teuthology.orchestra.run.smithi123.stdout: "caps": { 2024-09-16T15:49:50.151 INFO:teuthology.orchestra.run.smithi123.stdout: "mds": "allow", 2024-09-16T15:49:50.151 INFO:teuthology.orchestra.run.smithi123.stdout: "mgr": "allow *", 2024-09-16T15:49:50.151 INFO:teuthology.orchestra.run.smithi123.stdout: "mon": "allow *", 2024-09-16T15:49:50.151 INFO:teuthology.orchestra.run.smithi123.stdout: "osd": "allow *" 2024-09-16T15:49:50.151 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-16T15:49:50.151 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:49:50.151 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-09-16T15:49:50.151 INFO:teuthology.orchestra.run.smithi123.stdout: "entity": "mds.b", 2024-09-16T15:49:50.151 INFO:teuthology.orchestra.run.smithi123.stdout: "key": "AQC0UehmF1HFMhAADcvqXNKdHZMPAbLc+se4Aw==", 2024-09-16T15:49:50.152 INFO:teuthology.orchestra.run.smithi123.stdout: "caps": { 2024-09-16T15:49:50.152 INFO:teuthology.orchestra.run.smithi123.stdout: "mds": "allow", 2024-09-16T15:49:50.152 INFO:teuthology.orchestra.run.smithi123.stdout: "mgr": "allow *", 2024-09-16T15:49:50.152 INFO:teuthology.orchestra.run.smithi123.stdout: "mon": "allow *", 2024-09-16T15:49:50.152 INFO:teuthology.orchestra.run.smithi123.stdout: "osd": "allow *" 2024-09-16T15:49:50.152 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-16T15:49:50.152 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:49:50.152 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-09-16T15:49:50.152 INFO:teuthology.orchestra.run.smithi123.stdout: "entity": "mds.c", 2024-09-16T15:49:50.152 INFO:teuthology.orchestra.run.smithi123.stdout: "key": "AQC0UehmCBjCLBAAXd85J25r7cgrXeDCG7pahg==", 2024-09-16T15:49:50.152 INFO:teuthology.orchestra.run.smithi123.stdout: "caps": { 2024-09-16T15:49:50.152 INFO:teuthology.orchestra.run.smithi123.stdout: "mds": "allow", 2024-09-16T15:49:50.152 INFO:teuthology.orchestra.run.smithi123.stdout: "mgr": "allow *", 2024-09-16T15:49:50.152 INFO:teuthology.orchestra.run.smithi123.stdout: "mon": "allow *", 2024-09-16T15:49:50.153 INFO:teuthology.orchestra.run.smithi123.stdout: "osd": "allow *" 2024-09-16T15:49:50.153 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-16T15:49:50.153 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:49:50.153 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-09-16T15:49:50.153 INFO:teuthology.orchestra.run.smithi123.stdout: "entity": "mds.d", 2024-09-16T15:49:50.153 INFO:teuthology.orchestra.run.smithi123.stdout: "key": "AQC0UehmMh3ZOBAAx0RCa3R9RMT93/icZhPGyg==", 2024-09-16T15:49:50.153 INFO:teuthology.orchestra.run.smithi123.stdout: "caps": { 2024-09-16T15:49:50.153 INFO:teuthology.orchestra.run.smithi123.stdout: "mds": "allow", 2024-09-16T15:49:50.153 INFO:teuthology.orchestra.run.smithi123.stdout: "mgr": "allow *", 2024-09-16T15:49:50.153 INFO:teuthology.orchestra.run.smithi123.stdout: "mon": "allow *", 2024-09-16T15:49:50.153 INFO:teuthology.orchestra.run.smithi123.stdout: "osd": "allow *" 2024-09-16T15:49:50.153 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-16T15:49:50.153 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:49:50.153 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-09-16T15:49:50.154 INFO:teuthology.orchestra.run.smithi123.stdout: "entity": "osd.0", 2024-09-16T15:49:50.154 INFO:teuthology.orchestra.run.smithi123.stdout: "key": "AQC4UehmT633CRAAIhpkeIBOzwJO0kiXLu0x9Q==", 2024-09-16T15:49:50.154 INFO:teuthology.orchestra.run.smithi123.stdout: "caps": { 2024-09-16T15:49:50.154 INFO:teuthology.orchestra.run.smithi123.stdout: "mgr": "allow profile osd", 2024-09-16T15:49:50.154 INFO:teuthology.orchestra.run.smithi123.stdout: "mon": "allow profile osd", 2024-09-16T15:49:50.154 INFO:teuthology.orchestra.run.smithi123.stdout: "osd": "allow *" 2024-09-16T15:49:50.154 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-16T15:49:50.154 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:49:50.154 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-09-16T15:49:50.154 INFO:teuthology.orchestra.run.smithi123.stdout: "entity": "osd.1", 2024-09-16T15:49:50.154 INFO:teuthology.orchestra.run.smithi123.stdout: "key": "AQC7UehmFwF7GBAAvOUG+rWjANKT8wm/ai4xGg==", 2024-09-16T15:49:50.154 INFO:teuthology.orchestra.run.smithi123.stdout: "caps": { 2024-09-16T15:49:50.154 INFO:teuthology.orchestra.run.smithi123.stdout: "mgr": "allow profile osd", 2024-09-16T15:49:50.154 INFO:teuthology.orchestra.run.smithi123.stdout: "mon": "allow profile osd", 2024-09-16T15:49:50.155 INFO:teuthology.orchestra.run.smithi123.stdout: "osd": "allow *" 2024-09-16T15:49:50.155 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-16T15:49:50.155 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:49:50.155 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-09-16T15:49:50.155 INFO:teuthology.orchestra.run.smithi123.stdout: "entity": "osd.2", 2024-09-16T15:49:50.155 INFO:teuthology.orchestra.run.smithi123.stdout: "key": "AQC+UehmlCwWKxAAl3iO7ZALWyaorQH2urHKCw==", 2024-09-16T15:49:50.155 INFO:teuthology.orchestra.run.smithi123.stdout: "caps": { 2024-09-16T15:49:50.155 INFO:teuthology.orchestra.run.smithi123.stdout: "mgr": "allow profile osd", 2024-09-16T15:49:50.155 INFO:teuthology.orchestra.run.smithi123.stdout: "mon": "allow profile osd", 2024-09-16T15:49:50.155 INFO:teuthology.orchestra.run.smithi123.stdout: "osd": "allow *" 2024-09-16T15:49:50.155 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-16T15:49:50.155 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:49:50.155 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-09-16T15:49:50.156 INFO:teuthology.orchestra.run.smithi123.stdout: "entity": "osd.3", 2024-09-16T15:49:50.156 INFO:teuthology.orchestra.run.smithi123.stdout: "key": "AQDCUehmD8VgAhAAneoIeGr3MflSmi+nyavj2A==", 2024-09-16T15:49:50.156 INFO:teuthology.orchestra.run.smithi123.stdout: "caps": { 2024-09-16T15:49:50.156 INFO:teuthology.orchestra.run.smithi123.stdout: "mgr": "allow profile osd", 2024-09-16T15:49:50.156 INFO:teuthology.orchestra.run.smithi123.stdout: "mon": "allow profile osd", 2024-09-16T15:49:50.156 INFO:teuthology.orchestra.run.smithi123.stdout: "osd": "allow *" 2024-09-16T15:49:50.156 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-16T15:49:50.156 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:49:50.156 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-09-16T15:49:50.156 INFO:teuthology.orchestra.run.smithi123.stdout: "entity": "osd.4", 2024-09-16T15:49:50.156 INFO:teuthology.orchestra.run.smithi123.stdout: "key": "AQDIUehmA9B9EBAAsu0X5LPx1TmatZeHzjnVig==", 2024-09-16T15:49:50.156 INFO:teuthology.orchestra.run.smithi123.stdout: "caps": { 2024-09-16T15:49:50.156 INFO:teuthology.orchestra.run.smithi123.stdout: "mgr": "allow profile osd", 2024-09-16T15:49:50.157 INFO:teuthology.orchestra.run.smithi123.stdout: "mon": "allow profile osd", 2024-09-16T15:49:50.157 INFO:teuthology.orchestra.run.smithi123.stdout: "osd": "allow *" 2024-09-16T15:49:50.157 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-16T15:49:50.157 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:49:50.157 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-09-16T15:49:50.157 INFO:teuthology.orchestra.run.smithi123.stdout: "entity": "osd.5", 2024-09-16T15:49:50.157 INFO:teuthology.orchestra.run.smithi123.stdout: "key": "AQDLUehmcugdHxAAQMSS1ZH71FUzCx5A+yNnZg==", 2024-09-16T15:49:50.157 INFO:teuthology.orchestra.run.smithi123.stdout: "caps": { 2024-09-16T15:49:50.157 INFO:teuthology.orchestra.run.smithi123.stdout: "mgr": "allow profile osd", 2024-09-16T15:49:50.157 INFO:teuthology.orchestra.run.smithi123.stdout: "mon": "allow profile osd", 2024-09-16T15:49:50.157 INFO:teuthology.orchestra.run.smithi123.stdout: "osd": "allow *" 2024-09-16T15:49:50.157 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-16T15:49:50.157 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:49:50.157 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-09-16T15:49:50.158 INFO:teuthology.orchestra.run.smithi123.stdout: "entity": "osd.6", 2024-09-16T15:49:50.158 INFO:teuthology.orchestra.run.smithi123.stdout: "key": "AQDOUehmWLwYLhAAhOlQLepcWElsVPMnHlZm4w==", 2024-09-16T15:49:50.158 INFO:teuthology.orchestra.run.smithi123.stdout: "caps": { 2024-09-16T15:49:50.158 INFO:teuthology.orchestra.run.smithi123.stdout: "mgr": "allow profile osd", 2024-09-16T15:49:50.158 INFO:teuthology.orchestra.run.smithi123.stdout: "mon": "allow profile osd", 2024-09-16T15:49:50.158 INFO:teuthology.orchestra.run.smithi123.stdout: "osd": "allow *" 2024-09-16T15:49:50.158 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-16T15:49:50.158 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:49:50.158 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-09-16T15:49:50.158 INFO:teuthology.orchestra.run.smithi123.stdout: "entity": "osd.7", 2024-09-16T15:49:50.158 INFO:teuthology.orchestra.run.smithi123.stdout: "key": "AQDSUehmipkbABAAQuL/2O7w0tsM1OKeOil97w==", 2024-09-16T15:49:50.158 INFO:teuthology.orchestra.run.smithi123.stdout: "caps": { 2024-09-16T15:49:50.158 INFO:teuthology.orchestra.run.smithi123.stdout: "mgr": "allow profile osd", 2024-09-16T15:49:50.158 INFO:teuthology.orchestra.run.smithi123.stdout: "mon": "allow profile osd", 2024-09-16T15:49:50.159 INFO:teuthology.orchestra.run.smithi123.stdout: "osd": "allow *" 2024-09-16T15:49:50.159 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-16T15:49:50.159 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:49:50.159 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-09-16T15:49:50.159 INFO:teuthology.orchestra.run.smithi123.stdout: "entity": "client.0", 2024-09-16T15:49:50.159 INFO:teuthology.orchestra.run.smithi123.stdout: "key": "AQC1Uehm7GDTAhAAHWWVIyjVVnTeeozBR8DfFA==", 2024-09-16T15:49:50.159 INFO:teuthology.orchestra.run.smithi123.stdout: "caps": { 2024-09-16T15:49:50.159 INFO:teuthology.orchestra.run.smithi123.stdout: "mds": "allow", 2024-09-16T15:49:50.159 INFO:teuthology.orchestra.run.smithi123.stdout: "mon": "allow r", 2024-09-16T15:49:50.159 INFO:teuthology.orchestra.run.smithi123.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-09-16T15:49:50.159 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-16T15:49:50.159 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:49:50.159 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-09-16T15:49:50.159 INFO:teuthology.orchestra.run.smithi123.stdout: "entity": "client.1", 2024-09-16T15:49:50.160 INFO:teuthology.orchestra.run.smithi123.stdout: "key": "AQC1UehmFbaWChAAFtRmKucm+pbKtCNvRCTNqA==", 2024-09-16T15:49:50.160 INFO:teuthology.orchestra.run.smithi123.stdout: "caps": { 2024-09-16T15:49:50.160 INFO:teuthology.orchestra.run.smithi123.stdout: "mds": "allow", 2024-09-16T15:49:50.160 INFO:teuthology.orchestra.run.smithi123.stdout: "mgr": "allow r", 2024-09-16T15:49:50.160 INFO:teuthology.orchestra.run.smithi123.stdout: "mon": "allow rw", 2024-09-16T15:49:50.160 INFO:teuthology.orchestra.run.smithi123.stdout: "osd": "allow rwx" 2024-09-16T15:49:50.160 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-16T15:49:50.160 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:49:50.160 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-09-16T15:49:50.160 INFO:teuthology.orchestra.run.smithi123.stdout: "entity": "client.2", 2024-09-16T15:49:50.160 INFO:teuthology.orchestra.run.smithi123.stdout: "key": "AQC1Uehm1ADrBxAAW/8ayNHI6Wq/YlI4ivFZiw==", 2024-09-16T15:49:50.160 INFO:teuthology.orchestra.run.smithi123.stdout: "caps": { 2024-09-16T15:49:50.160 INFO:teuthology.orchestra.run.smithi123.stdout: "mds": "allow", 2024-09-16T15:49:50.161 INFO:teuthology.orchestra.run.smithi123.stdout: "mgr": "allow r", 2024-09-16T15:49:50.161 INFO:teuthology.orchestra.run.smithi123.stdout: "mon": "allow rw", 2024-09-16T15:49:50.161 INFO:teuthology.orchestra.run.smithi123.stdout: "osd": "allow rwx" 2024-09-16T15:49:50.161 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-16T15:49:50.161 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:49:50.161 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-09-16T15:49:50.161 INFO:teuthology.orchestra.run.smithi123.stdout: "entity": "client.3", 2024-09-16T15:49:50.161 INFO:teuthology.orchestra.run.smithi123.stdout: "key": "AQC1UehmLv/3DxAAuGloBxSuza95uajpfiZ9MQ==", 2024-09-16T15:49:50.161 INFO:teuthology.orchestra.run.smithi123.stdout: "caps": { 2024-09-16T15:49:50.161 INFO:teuthology.orchestra.run.smithi123.stdout: "mds": "allow", 2024-09-16T15:49:50.161 INFO:teuthology.orchestra.run.smithi123.stdout: "mgr": "allow r", 2024-09-16T15:49:50.161 INFO:teuthology.orchestra.run.smithi123.stdout: "mon": "allow rw", 2024-09-16T15:49:50.161 INFO:teuthology.orchestra.run.smithi123.stdout: "osd": "allow rwx" 2024-09-16T15:49:50.161 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-16T15:49:50.162 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:49:50.162 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-09-16T15:49:50.162 INFO:teuthology.orchestra.run.smithi123.stdout: "entity": "client.admin", 2024-09-16T15:49:50.162 INFO:teuthology.orchestra.run.smithi123.stdout: "key": "AQC0Uehm1ZDRDBAAhyQ1BmJg1Mw6wQ04I2CbFA==", 2024-09-16T15:49:50.162 INFO:teuthology.orchestra.run.smithi123.stdout: "caps": { 2024-09-16T15:49:50.162 INFO:teuthology.orchestra.run.smithi123.stdout: "mds": "allow *", 2024-09-16T15:49:50.162 INFO:teuthology.orchestra.run.smithi123.stdout: "mgr": "allow *", 2024-09-16T15:49:50.162 INFO:teuthology.orchestra.run.smithi123.stdout: "mon": "allow *", 2024-09-16T15:49:50.162 INFO:teuthology.orchestra.run.smithi123.stdout: "osd": "allow *" 2024-09-16T15:49:50.162 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-16T15:49:50.162 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:49:50.162 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-09-16T15:49:50.162 INFO:teuthology.orchestra.run.smithi123.stdout: "entity": "mgr.x", 2024-09-16T15:49:50.162 INFO:teuthology.orchestra.run.smithi123.stdout: "key": "AQC0UehmfPilIBAAERlBbduOgAU9ba0n/1tl3g==", 2024-09-16T15:49:50.163 INFO:teuthology.orchestra.run.smithi123.stdout: "caps": { 2024-09-16T15:49:50.163 INFO:teuthology.orchestra.run.smithi123.stdout: "mds": "allow *", 2024-09-16T15:49:50.163 INFO:teuthology.orchestra.run.smithi123.stdout: "mon": "allow profile mgr", 2024-09-16T15:49:50.163 INFO:teuthology.orchestra.run.smithi123.stdout: "osd": "allow *" 2024-09-16T15:49:50.163 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-16T15:49:50.163 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:49:50.163 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-09-16T15:49:50.163 INFO:teuthology.orchestra.run.smithi123.stdout: "entity": "mgr.y", 2024-09-16T15:49:50.163 INFO:teuthology.orchestra.run.smithi123.stdout: "key": "AQC0UehmkjrlHRAA+3xbQC+SPWGWW1gND7xcAg==", 2024-09-16T15:49:50.163 INFO:teuthology.orchestra.run.smithi123.stdout: "caps": { 2024-09-16T15:49:50.163 INFO:teuthology.orchestra.run.smithi123.stdout: "mds": "allow *", 2024-09-16T15:49:50.163 INFO:teuthology.orchestra.run.smithi123.stdout: "mon": "allow profile mgr", 2024-09-16T15:49:50.163 INFO:teuthology.orchestra.run.smithi123.stdout: "osd": "allow *" 2024-09-16T15:49:50.163 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-16T15:49:50.164 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-16T15:49:50.164 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-09-16T15:49:50.164 INFO:teuthology.orchestra.run.smithi123.stdout:} 2024-09-16T15:49:50.164 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:50.150+0000 7f85aa877640 1 -- 172.21.15.123:0/1866988412 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f85840734f0 msgr2=0x7f85840759b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:50.165 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:50.150+0000 7f85aa877640 1 --2- 172.21.15.123:0/1866988412 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f85840734f0 0x7f85840759b0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f85940058d0 tx=0x7f8594005860 comp rx=0 tx=0).stop 2024-09-16T15:49:50.165 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:50.150+0000 7f85aa877640 1 -- 172.21.15.123:0/1866988412 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f85a40d0850 msgr2=0x7f85a4158660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:50.165 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:50.150+0000 7f85aa877640 1 --2- 172.21.15.123:0/1866988412 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f85a40d0850 0x7f85a4158660 secure :-1 s=READY pgs=1382 cs=0 l=1 rev1=1 crypto rx=0x7f858c009bd0 tx=0x7f858c003d70 comp rx=0 tx=0).stop 2024-09-16T15:49:50.165 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:50.150+0000 7f85aa877640 1 -- 172.21.15.123:0/1866988412 shutdown_connections 2024-09-16T15:49:50.165 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:50.150+0000 7f85aa877640 1 --2- 172.21.15.123:0/1866988412 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f85840734f0 0x7f85840759b0 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:50.165 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:50.150+0000 7f85aa877640 1 --2- 172.21.15.123:0/1866988412 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f85a415bda0 0x7f85a414e3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:50.165 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:50.150+0000 7f85aa877640 1 --2- 172.21.15.123:0/1866988412 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f85a40d0850 0x7f85a4158660 unknown :-1 s=CLOSED pgs=1382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:50.165 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:50.150+0000 7f85aa877640 1 --2- 172.21.15.123:0/1866988412 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f85a40b25a0 0x7f85a4154e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:50.165 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:50.150+0000 7f85aa877640 1 -- 172.21.15.123:0/1866988412 >> 172.21.15.123:0/1866988412 conn(0x7f85a4074bd0 msgr2=0x7f85a40aee40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:49:50.165 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:50.150+0000 7f85aa877640 1 -- 172.21.15.123:0/1866988412 shutdown_connections 2024-09-16T15:49:50.165 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:50.150+0000 7f85aa877640 1 -- 172.21.15.123:0/1866988412 wait complete. 2024-09-16T15:49:50.169 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-09-16T15:49:50.292 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:50.290+0000 7fb918137640 1 -- 172.21.15.123:0/2508185083 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb8fc003a60 msgr2=0x7fb8fc003e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:50.292 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:50.290+0000 7fb918137640 1 --2- 172.21.15.123:0/2508185083 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb8fc003a60 0x7fb8fc003e80 secure :-1 s=READY pgs=1384 cs=0 l=1 rev1=1 crypto rx=0x7fb90c00c470 tx=0x7fb90c01ced0 comp rx=0 tx=0).stop 2024-09-16T15:49:50.292 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:50.290+0000 7fb918137640 1 -- 172.21.15.123:0/2508185083 shutdown_connections 2024-09-16T15:49:50.292 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:50.290+0000 7fb918137640 1 --2- 172.21.15.123:0/2508185083 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb8fc003a60 0x7fb8fc003e80 unknown :-1 s=CLOSED pgs=1384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:50.292 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:50.290+0000 7fb918137640 1 -- 172.21.15.123:0/2508185083 >> 172.21.15.123:0/2508185083 conn(0x7fb9100d0b70 msgr2=0x7fb9100d0f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:49:50.292 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:50.290+0000 7fb918137640 1 -- 172.21.15.123:0/2508185083 shutdown_connections 2024-09-16T15:49:50.292 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:50.290+0000 7fb918137640 1 -- 172.21.15.123:0/2508185083 wait complete. 2024-09-16T15:49:50.293 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:50.294+0000 7fb918137640 1 Processor -- start 2024-09-16T15:49:50.293 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:50.294+0000 7fb918137640 1 -- start start 2024-09-16T15:49:50.295 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:50.294+0000 7fb918137640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fb8fc003a60 0x7fb9100d0410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:49:50.295 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:50.294+0000 7fb918137640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb91009ee50 0x7fb91009f2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:49:50.295 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:50.294+0000 7fb918137640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb9100a0f60 0x7fb9100a1410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:49:50.295 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:50.294+0000 7fb918137640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fb91006d840 con 0x7fb9100a0f60 2024-09-16T15:49:50.295 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:50.294+0000 7fb918137640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fb91006d6c0 con 0x7fb91009ee50 2024-09-16T15:49:50.295 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:50.294+0000 7fb918137640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fb91006d9c0 con 0x7fb8fc003a60 2024-09-16T15:49:50.295 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:50.294+0000 7fb9166d4640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb9100a0f60 0x7fb9100a1410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:49:50.295 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:50.294+0000 7fb9166d4640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb9100a0f60 0x7fb9100a1410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:57590/0 (socket says 172.21.15.123:57590) 2024-09-16T15:49:50.296 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:50.294+0000 7fb9166d4640 1 -- 172.21.15.123:0/1450208597 learned_addr learned my addr 172.21.15.123:0/1450208597 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:49:50.296 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:50.294+0000 7fb915ed3640 1 --2- 172.21.15.123:0/1450208597 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fb8fc003a60 0x7fb9100d0410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:49:50.296 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:50.294+0000 7fb9156d2640 1 --2- 172.21.15.123:0/1450208597 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb91009ee50 0x7fb91009f2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:49:50.296 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:50.294+0000 7fb9166d4640 1 -- 172.21.15.123:0/1450208597 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fb8fc003a60 msgr2=0x7fb9100d0410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:50.296 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:50.294+0000 7fb9166d4640 1 --2- 172.21.15.123:0/1450208597 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fb8fc003a60 0x7fb9100d0410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:50.296 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:50.294+0000 7fb9166d4640 1 -- 172.21.15.123:0/1450208597 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb91009ee50 msgr2=0x7fb91009f2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:50.296 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:50.294+0000 7fb9166d4640 1 --2- 172.21.15.123:0/1450208597 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb91009ee50 0x7fb91009f2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:50.296 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:50.294+0000 7fb9166d4640 1 -- 172.21.15.123:0/1450208597 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb9101f88a0 con 0x7fb9100a0f60 2024-09-16T15:49:50.296 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:50.294+0000 7fb9166d4640 1 --2- 172.21.15.123:0/1450208597 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb9100a0f60 0x7fb9100a1410 secure :-1 s=READY pgs=1385 cs=0 l=1 rev1=1 crypto rx=0x7fb910069320 tx=0x7fb90800fed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:49:50.296 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:50.294+0000 7fb9067fc640 1 -- 172.21.15.123:0/1450208597 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb908003d90 con 0x7fb9100a0f60 2024-09-16T15:49:50.296 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:50.294+0000 7fb9067fc640 1 -- 172.21.15.123:0/1450208597 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fb90800d9b0 con 0x7fb9100a0f60 2024-09-16T15:49:50.296 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:50.294+0000 7fb9067fc640 1 -- 172.21.15.123:0/1450208597 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb90800de00 con 0x7fb9100a0f60 2024-09-16T15:49:50.296 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:50.294+0000 7fb918137640 1 -- 172.21.15.123:0/1450208597 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb9100a1950 con 0x7fb9100a0f60 2024-09-16T15:49:50.297 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:50.294+0000 7fb918137640 1 -- 172.21.15.123:0/1450208597 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb9100b18f0 con 0x7fb9100a0f60 2024-09-16T15:49:50.297 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:50.294+0000 7fb918137640 1 -- 172.21.15.123:0/1450208597 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb910073d20 con 0x7fb9100a0f60 2024-09-16T15:49:50.297 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:50.298+0000 7fb9067fc640 1 -- 172.21.15.123:0/1450208597 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 21) ==== 97641+0+0 (secure 0 0 0) 0x7fb908019020 con 0x7fb9100a0f60 2024-09-16T15:49:50.297 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:50.298+0000 7fb9067fc640 1 --2- 172.21.15.123:0/1450208597 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fb8f4073520 0x7fb8f40759e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:49:50.298 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:50.298+0000 7fb9067fc640 1 -- 172.21.15.123:0/1450208597 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(131..131 src has 1..131) ==== 4885+0+0 (secure 0 0 0) 0x7fb908094d90 con 0x7fb9100a0f60 2024-09-16T15:49:50.298 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:50.298+0000 7fb915ed3640 1 --2- 172.21.15.123:0/1450208597 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fb8f4073520 0x7fb8f40759e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:49:50.298 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:50.298+0000 7fb915ed3640 1 --2- 172.21.15.123:0/1450208597 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fb8f4073520 0x7fb8f40759e0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fb90c0ac000 tx=0x7fb90c019d10 comp rx=0 tx=0).ready entity=mgr.4150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:49:50.300 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:50.302+0000 7fb9067fc640 1 -- 172.21.15.123:0/1450208597 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7fb908022030 con 0x7fb9100a0f60 2024-09-16T15:49:50.463 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:50.462+0000 7fb918137640 1 -- 172.21.15.123:0/1450208597 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) -- 0x7fb9100a1f00 con 0x7fb9100a0f60 2024-09-16T15:49:50.464 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:50.462+0000 7fb9067fc640 1 -- 172.21.15.123:0/1450208597 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v131) ==== 84+0+65 (secure 0 0 0) 0x7fb90805fa50 con 0x7fb9100a0f60 2024-09-16T15:49:50.464 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-09-16T15:49:50.464 INFO:teuthology.orchestra.run.smithi123.stdout:[ 2024-09-16T15:49:50.464 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-09-16T15:49:50.464 INFO:teuthology.orchestra.run.smithi123.stdout: "poolnum": 1, 2024-09-16T15:49:50.465 INFO:teuthology.orchestra.run.smithi123.stdout: "poolname": ".mgr" 2024-09-16T15:49:50.465 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-16T15:49:50.465 INFO:teuthology.orchestra.run.smithi123.stdout:] 2024-09-16T15:49:50.467 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:50.466+0000 7fb918137640 1 -- 172.21.15.123:0/1450208597 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fb8f4073520 msgr2=0x7fb8f40759e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:50.467 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:50.466+0000 7fb918137640 1 --2- 172.21.15.123:0/1450208597 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fb8f4073520 0x7fb8f40759e0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fb90c0ac000 tx=0x7fb90c019d10 comp rx=0 tx=0).stop 2024-09-16T15:49:50.467 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:50.466+0000 7fb918137640 1 -- 172.21.15.123:0/1450208597 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb9100a0f60 msgr2=0x7fb9100a1410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:50.467 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:50.466+0000 7fb918137640 1 --2- 172.21.15.123:0/1450208597 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb9100a0f60 0x7fb9100a1410 secure :-1 s=READY pgs=1385 cs=0 l=1 rev1=1 crypto rx=0x7fb910069320 tx=0x7fb90800fed0 comp rx=0 tx=0).stop 2024-09-16T15:49:50.467 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:50.466+0000 7fb918137640 1 -- 172.21.15.123:0/1450208597 shutdown_connections 2024-09-16T15:49:50.467 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:50.466+0000 7fb918137640 1 --2- 172.21.15.123:0/1450208597 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fb8f4073520 0x7fb8f40759e0 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:50.467 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:50.466+0000 7fb918137640 1 --2- 172.21.15.123:0/1450208597 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb9100a0f60 0x7fb9100a1410 unknown :-1 s=CLOSED pgs=1385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:50.467 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:50.466+0000 7fb918137640 1 --2- 172.21.15.123:0/1450208597 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb91009ee50 0x7fb91009f2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:50.467 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:50.466+0000 7fb918137640 1 --2- 172.21.15.123:0/1450208597 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fb8fc003a60 0x7fb9100d0410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:50.467 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:50.466+0000 7fb918137640 1 -- 172.21.15.123:0/1450208597 >> 172.21.15.123:0/1450208597 conn(0x7fb9100d0b70 msgr2=0x7fb910070d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:49:50.467 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:50.466+0000 7fb918137640 1 -- 172.21.15.123:0/1450208597 shutdown_connections 2024-09-16T15:49:50.468 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:50.466+0000 7fb918137640 1 -- 172.21.15.123:0/1450208597 wait complete. 2024-09-16T15:49:50.482 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-09-16T15:49:50.483 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-09-16T15:49:50.611 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:50.610+0000 7f27c9969640 1 -- 172.21.15.123:0/1665411024 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f27ac003a10 msgr2=0x7f27ac003e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:50.611 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:50.610+0000 7f27c9969640 1 --2- 172.21.15.123:0/1665411024 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f27ac003a10 0x7f27ac003e30 secure :-1 s=READY pgs=1387 cs=0 l=1 rev1=1 crypto rx=0x7f27b4009b60 tx=0x7f27b401d2b0 comp rx=0 tx=0).stop 2024-09-16T15:49:50.611 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:50.610+0000 7f27c37fe640 1 -- 172.21.15.123:0/1665411024 reap_dead start 2024-09-16T15:49:50.611 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:50.610+0000 7f27c9969640 1 -- 172.21.15.123:0/1665411024 shutdown_connections 2024-09-16T15:49:50.611 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:50.610+0000 7f27c9969640 1 -- 172.21.15.123:0/1665411024 >> 172.21.15.123:0/1665411024 conn(0x7f27c40b1630 msgr2=0x7f27c40b1a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:49:50.612 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:50.610+0000 7f27c9969640 1 -- 172.21.15.123:0/1665411024 shutdown_connections 2024-09-16T15:49:50.612 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:50.610+0000 7f27c9969640 1 -- 172.21.15.123:0/1665411024 wait complete. 2024-09-16T15:49:50.612 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:50.610+0000 7f27c9969640 1 Processor -- start 2024-09-16T15:49:50.612 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:50.610+0000 7f27c9969640 1 -- start start 2024-09-16T15:49:50.614 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:50.614+0000 7f27c9969640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f27ac003a10 0x7f27c4156fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:49:50.614 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:50.614+0000 7f27c9969640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f27c4157ca0 0x7f27c414c420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:49:50.614 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:50.614+0000 7f27c9969640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f27c415c1f0 0x7f27c4159cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:49:50.615 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:50.614+0000 7f27c9969640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f27c406fb10 con 0x7f27ac003a10 2024-09-16T15:49:50.615 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:50.614+0000 7f27c9969640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f27c406f990 con 0x7f27c4157ca0 2024-09-16T15:49:50.615 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:50.614+0000 7f27c9969640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f27c406fc90 con 0x7f27c415c1f0 2024-09-16T15:49:50.615 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:50.614+0000 7f27c2ffd640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f27ac003a10 0x7f27c4156fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:49:50.615 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:50.614+0000 7f27c2ffd640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f27ac003a10 0x7f27c4156fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:57606/0 (socket says 172.21.15.123:57606) 2024-09-16T15:49:50.615 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:50.614+0000 7f27c2ffd640 1 -- 172.21.15.123:0/1742704552 learned_addr learned my addr 172.21.15.123:0/1742704552 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:49:50.615 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:50.614+0000 7f27c27fc640 1 --2- 172.21.15.123:0/1742704552 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f27c4157ca0 0x7f27c414c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:49:50.615 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:50.614+0000 7f27c37fe640 1 --2- 172.21.15.123:0/1742704552 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f27c415c1f0 0x7f27c4159cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:49:50.615 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:50.614+0000 7f27c2ffd640 1 -- 172.21.15.123:0/1742704552 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f27c415c1f0 msgr2=0x7f27c4159cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:50.615 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:50.614+0000 7f27c2ffd640 1 --2- 172.21.15.123:0/1742704552 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f27c415c1f0 0x7f27c4159cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:50.615 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:50.614+0000 7f27c2ffd640 1 -- 172.21.15.123:0/1742704552 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f27c4157ca0 msgr2=0x7f27c414c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:50.615 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:50.614+0000 7f27c2ffd640 1 --2- 172.21.15.123:0/1742704552 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f27c4157ca0 0x7f27c414c420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:50.615 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:50.614+0000 7f27c2ffd640 1 -- 172.21.15.123:0/1742704552 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f27c41581f0 con 0x7f27ac003a10 2024-09-16T15:49:50.616 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:50.614+0000 7f27c2ffd640 1 --2- 172.21.15.123:0/1742704552 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f27ac003a10 0x7f27c4156fd0 secure :-1 s=READY pgs=1388 cs=0 l=1 rev1=1 crypto rx=0x7f27b401d0e0 tx=0x7f27b40055d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:49:50.616 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:50.614+0000 7f27abfff640 1 -- 172.21.15.123:0/1742704552 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27b40b1040 con 0x7f27ac003a10 2024-09-16T15:49:50.616 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:50.614+0000 7f27abfff640 1 -- 172.21.15.123:0/1742704552 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f27b40bb070 con 0x7f27ac003a10 2024-09-16T15:49:50.616 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:50.614+0000 7f27c9969640 1 -- 172.21.15.123:0/1742704552 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f27c41584d0 con 0x7f27ac003a10 2024-09-16T15:49:50.616 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:50.614+0000 7f27abfff640 1 -- 172.21.15.123:0/1742704552 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27b4004290 con 0x7f27ac003a10 2024-09-16T15:49:50.616 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:50.614+0000 7f27c9969640 1 -- 172.21.15.123:0/1742704552 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f27c414c960 con 0x7f27ac003a10 2024-09-16T15:49:50.616 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:50.614+0000 7f27c9969640 1 -- 172.21.15.123:0/1742704552 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f27c415b050 con 0x7f27ac003a10 2024-09-16T15:49:50.617 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:50.614+0000 7f27abfff640 1 -- 172.21.15.123:0/1742704552 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 21) ==== 97641+0+0 (secure 0 0 0) 0x7f27b40046d0 con 0x7f27ac003a10 2024-09-16T15:49:50.617 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:50.614+0000 7f27abfff640 1 --2- 172.21.15.123:0/1742704552 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f27b00734f0 0x7f27b00759b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:49:50.617 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:50.614+0000 7f27abfff640 1 -- 172.21.15.123:0/1742704552 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(131..131 src has 1..131) ==== 4885+0+0 (secure 0 0 0) 0x7f27b412f2f0 con 0x7f27ac003a10 2024-09-16T15:49:50.617 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:50.614+0000 7f27c27fc640 1 --2- 172.21.15.123:0/1742704552 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f27b00734f0 0x7f27b00759b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:49:50.617 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:50.618+0000 7f27c27fc640 1 --2- 172.21.15.123:0/1742704552 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f27b00734f0 0x7f27b00759b0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f27c41513a0 tx=0x7f27b800f040 comp rx=0 tx=0).ready entity=mgr.4150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:49:50.620 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:50.618+0000 7f27abfff640 1 -- 172.21.15.123:0/1742704552 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f27b4016fc0 con 0x7f27ac003a10 2024-09-16T15:49:50.800 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:50.798+0000 7f27c9969640 1 -- 172.21.15.123:0/1742704552 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) -- 0x7f27c406e7b0 con 0x7f27ac003a10 2024-09-16T15:49:51.722 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:51.722+0000 7f27abfff640 1 -- 172.21.15.123:0/1742704552 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v132) ==== 138+0+0 (secure 0 0 0) 0x7f27b40fa090 con 0x7f27ac003a10 2024-09-16T15:49:51.722 INFO:teuthology.orchestra.run.smithi123.stderr:pool 'cephfs_metadata' created 2024-09-16T15:49:51.726 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:51.726+0000 7f27c9969640 1 -- 172.21.15.123:0/1742704552 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f27b00734f0 msgr2=0x7f27b00759b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:51.726 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:51.726+0000 7f27c9969640 1 --2- 172.21.15.123:0/1742704552 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f27b00734f0 0x7f27b00759b0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f27c41513a0 tx=0x7f27b800f040 comp rx=0 tx=0).stop 2024-09-16T15:49:51.726 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:51.726+0000 7f27c9969640 1 -- 172.21.15.123:0/1742704552 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f27ac003a10 msgr2=0x7f27c4156fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:51.726 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:51.726+0000 7f27c9969640 1 --2- 172.21.15.123:0/1742704552 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f27ac003a10 0x7f27c4156fd0 secure :-1 s=READY pgs=1388 cs=0 l=1 rev1=1 crypto rx=0x7f27b401d0e0 tx=0x7f27b40055d0 comp rx=0 tx=0).stop 2024-09-16T15:49:51.727 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:51.726+0000 7f27c9969640 1 -- 172.21.15.123:0/1742704552 shutdown_connections 2024-09-16T15:49:51.727 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:51.726+0000 7f27c9969640 1 --2- 172.21.15.123:0/1742704552 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f27b00734f0 0x7f27b00759b0 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:51.727 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:51.726+0000 7f27c9969640 1 --2- 172.21.15.123:0/1742704552 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f27c415c1f0 0x7f27c4159cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:51.727 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:51.726+0000 7f27c9969640 1 --2- 172.21.15.123:0/1742704552 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f27c4157ca0 0x7f27c414c420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:51.727 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:51.726+0000 7f27c9969640 1 --2- 172.21.15.123:0/1742704552 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f27ac003a10 0x7f27c4156fd0 unknown :-1 s=CLOSED pgs=1388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:51.727 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:51.726+0000 7f27c9969640 1 -- 172.21.15.123:0/1742704552 >> 172.21.15.123:0/1742704552 conn(0x7f27c40b1630 msgr2=0x7f27c40cd930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:49:51.728 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:51.726+0000 7f27c9969640 1 -- 172.21.15.123:0/1742704552 shutdown_connections 2024-09-16T15:49:51.728 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:51.726+0000 7f27c9969640 1 -- 172.21.15.123:0/1742704552 wait complete. 2024-09-16T15:49:51.750 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-09-16T15:49:51.884 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:51.886+0000 7fb7afe58640 1 -- 172.21.15.123:0/3808076351 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb7a806dd00 msgr2=0x7fb7a806e190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:51.885 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:51.886+0000 7fb7afe58640 1 --2- 172.21.15.123:0/3808076351 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb7a806dd00 0x7fb7a806e190 secure :-1 s=READY pgs=1390 cs=0 l=1 rev1=1 crypto rx=0x7fb798009840 tx=0x7fb79801c790 comp rx=0 tx=0).stop 2024-09-16T15:49:51.885 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:51.886+0000 7fb7afe58640 1 -- 172.21.15.123:0/3808076351 shutdown_connections 2024-09-16T15:49:51.885 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:51.886+0000 7fb7afe58640 1 --2- 172.21.15.123:0/3808076351 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fb7a815b050 0x7fb7a815b4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:51.885 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:51.886+0000 7fb7afe58640 1 --2- 172.21.15.123:0/3808076351 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb7a806dd00 0x7fb7a806e190 unknown :-1 s=CLOSED pgs=1390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:51.885 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:51.886+0000 7fb7afe58640 1 -- 172.21.15.123:0/3808076351 >> 172.21.15.123:0/3808076351 conn(0x7fb7a80d0bc0 msgr2=0x7fb7a80d0fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:49:51.885 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:51.886+0000 7fb7afe58640 1 -- 172.21.15.123:0/3808076351 shutdown_connections 2024-09-16T15:49:51.885 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:51.886+0000 7fb7afe58640 1 -- 172.21.15.123:0/3808076351 wait complete. 2024-09-16T15:49:51.886 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:51.886+0000 7fb7afe58640 1 Processor -- start 2024-09-16T15:49:51.886 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:51.886+0000 7fb7afe58640 1 -- start start 2024-09-16T15:49:51.887 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:51.886+0000 7fb7afe58640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb7a806dd00 0x7fb7a814ebb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:49:51.888 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:51.886+0000 7fb7afe58640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fb7a815b050 0x7fb7a80b5a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:49:51.888 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:51.886+0000 7fb7afe58640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb7a815ef00 0x7fb7a80b5fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:49:51.888 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:51.886+0000 7fb7afe58640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fb7a8154b40 con 0x7fb7a806dd00 2024-09-16T15:49:51.888 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:51.886+0000 7fb7afe58640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fb7a81549c0 con 0x7fb7a815ef00 2024-09-16T15:49:51.888 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:51.886+0000 7fb7afe58640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fb7a8154cc0 con 0x7fb7a815b050 2024-09-16T15:49:51.888 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:51.886+0000 7fb7adbf4640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb7a806dd00 0x7fb7a814ebb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:49:51.888 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:51.886+0000 7fb7adbf4640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb7a806dd00 0x7fb7a814ebb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:57624/0 (socket says 172.21.15.123:57624) 2024-09-16T15:49:51.888 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:51.886+0000 7fb7adbf4640 1 -- 172.21.15.123:0/157577154 learned_addr learned my addr 172.21.15.123:0/157577154 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:49:51.888 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:51.886+0000 7fb7ad3f3640 1 --2- 172.21.15.123:0/157577154 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fb7a815b050 0x7fb7a80b5a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:49:51.888 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:51.886+0000 7fb7ae3f5640 1 --2- 172.21.15.123:0/157577154 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb7a815ef00 0x7fb7a80b5fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:49:51.888 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:51.886+0000 7fb7adbf4640 1 -- 172.21.15.123:0/157577154 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fb7a815b050 msgr2=0x7fb7a80b5a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:51.888 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:51.886+0000 7fb7adbf4640 1 --2- 172.21.15.123:0/157577154 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fb7a815b050 0x7fb7a80b5a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:51.888 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:51.886+0000 7fb7adbf4640 1 -- 172.21.15.123:0/157577154 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb7a815ef00 msgr2=0x7fb7a80b5fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:51.889 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:51.886+0000 7fb7adbf4640 1 --2- 172.21.15.123:0/157577154 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb7a815ef00 0x7fb7a80b5fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:51.889 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:51.886+0000 7fb7adbf4640 1 -- 172.21.15.123:0/157577154 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb7a80b6650 con 0x7fb7a806dd00 2024-09-16T15:49:51.889 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:51.886+0000 7fb7ad3f3640 1 --2- 172.21.15.123:0/157577154 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fb7a815b050 0x7fb7a80b5a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T15:49:51.889 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:51.886+0000 7fb7adbf4640 1 --2- 172.21.15.123:0/157577154 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb7a806dd00 0x7fb7a814ebb0 secure :-1 s=READY pgs=1391 cs=0 l=1 rev1=1 crypto rx=0x7fb7a400c400 tx=0x7fb7a400c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:49:51.889 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:51.886+0000 7fb79e7fc640 1 -- 172.21.15.123:0/157577154 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7a4017070 con 0x7fb7a806dd00 2024-09-16T15:49:51.889 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:51.886+0000 7fb79e7fc640 1 -- 172.21.15.123:0/157577154 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fb7a4003600 con 0x7fb7a806dd00 2024-09-16T15:49:51.889 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:51.886+0000 7fb79e7fc640 1 -- 172.21.15.123:0/157577154 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7a4003a50 con 0x7fb7a806dd00 2024-09-16T15:49:51.889 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:51.886+0000 7fb7afe58640 1 -- 172.21.15.123:0/157577154 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb7a81f50e0 con 0x7fb7a806dd00 2024-09-16T15:49:51.889 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:51.886+0000 7fb7afe58640 1 -- 172.21.15.123:0/157577154 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb7a81f56a0 con 0x7fb7a806dd00 2024-09-16T15:49:51.890 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:51.890+0000 7fb7afe58640 1 -- 172.21.15.123:0/157577154 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb7a81551b0 con 0x7fb7a806dd00 2024-09-16T15:49:51.890 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:51.890+0000 7fb79e7fc640 1 -- 172.21.15.123:0/157577154 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 21) ==== 97641+0+0 (secure 0 0 0) 0x7fb7a401a070 con 0x7fb7a806dd00 2024-09-16T15:49:51.890 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:51.890+0000 7fb79e7fc640 1 --2- 172.21.15.123:0/157577154 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fb78c0734f0 0x7fb78c0759b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:49:51.890 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:51.890+0000 7fb7ad3f3640 1 --2- 172.21.15.123:0/157577154 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fb78c0734f0 0x7fb78c0759b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:49:51.890 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:51.890+0000 7fb79e7fc640 1 -- 172.21.15.123:0/157577154 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(132..132 src has 1..132) ==== 5255+0+0 (secure 0 0 0) 0x7fb7a4093f20 con 0x7fb7a806dd00 2024-09-16T15:49:51.890 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:51.890+0000 7fb7ad3f3640 1 --2- 172.21.15.123:0/157577154 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fb78c0734f0 0x7fb78c0759b0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fb7980ae000 tx=0x7fb79801afd0 comp rx=0 tx=0).ready entity=mgr.4150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:49:51.893 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:51.894+0000 7fb79e7fc640 1 -- 172.21.15.123:0/157577154 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7fb7a4021020 con 0x7fb7a806dd00 2024-09-16T15:49:52.069 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:52.070+0000 7fb7afe58640 1 -- 172.21.15.123:0/157577154 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) -- 0x7fb7a8152a60 con 0x7fb7a806dd00 2024-09-16T15:49:52.736 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:52.734+0000 7fb79e7fc640 1 -- 172.21.15.123:0/157577154 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v133) ==== 170+0+0 (secure 0 0 0) 0x7fb7a405e9e0 con 0x7fb7a806dd00 2024-09-16T15:49:52.736 INFO:teuthology.orchestra.run.smithi123.stderr:pool 'cephfs_data' created 2024-09-16T15:49:52.739 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:52.738+0000 7fb7afe58640 1 -- 172.21.15.123:0/157577154 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fb78c0734f0 msgr2=0x7fb78c0759b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:52.740 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:52.738+0000 7fb7afe58640 1 --2- 172.21.15.123:0/157577154 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fb78c0734f0 0x7fb78c0759b0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fb7980ae000 tx=0x7fb79801afd0 comp rx=0 tx=0).stop 2024-09-16T15:49:52.740 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:52.738+0000 7fb7afe58640 1 -- 172.21.15.123:0/157577154 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb7a806dd00 msgr2=0x7fb7a814ebb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:52.740 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:52.738+0000 7fb7afe58640 1 --2- 172.21.15.123:0/157577154 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb7a806dd00 0x7fb7a814ebb0 secure :-1 s=READY pgs=1391 cs=0 l=1 rev1=1 crypto rx=0x7fb7a400c400 tx=0x7fb7a400c8c0 comp rx=0 tx=0).stop 2024-09-16T15:49:52.740 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:52.742+0000 7fb7afe58640 1 -- 172.21.15.123:0/157577154 shutdown_connections 2024-09-16T15:49:52.740 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:52.742+0000 7fb7afe58640 1 --2- 172.21.15.123:0/157577154 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fb78c0734f0 0x7fb78c0759b0 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:52.740 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:52.742+0000 7fb7afe58640 1 --2- 172.21.15.123:0/157577154 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb7a815ef00 0x7fb7a80b5fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:52.740 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:52.742+0000 7fb7afe58640 1 --2- 172.21.15.123:0/157577154 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fb7a815b050 0x7fb7a80b5a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:52.740 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:52.742+0000 7fb7afe58640 1 --2- 172.21.15.123:0/157577154 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb7a806dd00 0x7fb7a814ebb0 unknown :-1 s=CLOSED pgs=1391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:52.741 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:52.742+0000 7fb7afe58640 1 -- 172.21.15.123:0/157577154 >> 172.21.15.123:0/157577154 conn(0x7fb7a80d0bc0 msgr2=0x7fb7a80d0fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:49:52.741 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:52.742+0000 7fb7afe58640 1 -- 172.21.15.123:0/157577154 shutdown_connections 2024-09-16T15:49:52.741 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:52.742+0000 7fb7afe58640 1 -- 172.21.15.123:0/157577154 wait complete. 2024-09-16T15:49:52.775 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-09-16T15:49:52.905 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:52.906+0000 7f72d0dde640 1 -- 172.21.15.123:0/3909009255 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f72cc0c3230 msgr2=0x7f72cc0cedd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:52.905 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:52.906+0000 7f72d0dde640 1 --2- 172.21.15.123:0/3909009255 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f72cc0c3230 0x7f72cc0cedd0 secure :-1 s=READY pgs=1393 cs=0 l=1 rev1=1 crypto rx=0x7f72c0009900 tx=0x7f72c001c8e0 comp rx=0 tx=0).stop 2024-09-16T15:49:52.905 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:52.906+0000 7f72d0dde640 1 -- 172.21.15.123:0/3909009255 shutdown_connections 2024-09-16T15:49:52.905 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:52.906+0000 7f72d0dde640 1 --2- 172.21.15.123:0/3909009255 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f72cc0c3230 0x7f72cc0cedd0 unknown :-1 s=CLOSED pgs=1393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:52.905 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:52.906+0000 7f72d0dde640 1 --2- 172.21.15.123:0/3909009255 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f72cc15b860 0x7f72cc06fca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:52.905 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:52.906+0000 7f72d0dde640 1 -- 172.21.15.123:0/3909009255 >> 172.21.15.123:0/3909009255 conn(0x7f72cc0b1710 msgr2=0x7f72cc0b1b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:49:52.905 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:52.906+0000 7f72d0dde640 1 -- 172.21.15.123:0/3909009255 shutdown_connections 2024-09-16T15:49:52.906 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:52.906+0000 7f72d0dde640 1 -- 172.21.15.123:0/3909009255 wait complete. 2024-09-16T15:49:52.906 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:52.906+0000 7f72d0dde640 1 Processor -- start 2024-09-16T15:49:52.906 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:52.906+0000 7f72d0dde640 1 -- start start 2024-09-16T15:49:52.908 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:52.906+0000 7f72d0dde640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f72cc0701e0 0x7f72cc1fad10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:49:52.908 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:52.906+0000 7f72d0dde640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f72cc0c3230 0x7f72cc164b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:49:52.908 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:52.906+0000 7f72d0dde640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f72cc15b860 0x7f72cc151be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:49:52.908 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:52.906+0000 7f72d0dde640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f72cc169140 con 0x7f72cc0c3230 2024-09-16T15:49:52.908 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:52.906+0000 7f72d0dde640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f72cc168fc0 con 0x7f72cc0701e0 2024-09-16T15:49:52.908 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:52.906+0000 7f72d0dde640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f72cc1692c0 con 0x7f72cc15b860 2024-09-16T15:49:52.908 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:52.906+0000 7f72c9d9b640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f72cc0c3230 0x7f72cc164b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:49:52.908 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:52.906+0000 7f72c9d9b640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f72cc0c3230 0x7f72cc164b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:58668/0 (socket says 172.21.15.123:58668) 2024-09-16T15:49:52.908 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:52.906+0000 7f72c9d9b640 1 -- 172.21.15.123:0/1213801811 learned_addr learned my addr 172.21.15.123:0/1213801811 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:49:52.908 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:52.906+0000 7f72c9d9b640 1 -- 172.21.15.123:0/1213801811 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f72cc15b860 msgr2=0x7f72cc151be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:52.908 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:52.906+0000 7f72c9d9b640 1 --2- 172.21.15.123:0/1213801811 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f72cc15b860 0x7f72cc151be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:52.909 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:52.906+0000 7f72c9d9b640 1 -- 172.21.15.123:0/1213801811 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f72cc0701e0 msgr2=0x7f72cc1fad10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:52.909 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:52.906+0000 7f72c9d9b640 1 --2- 172.21.15.123:0/1213801811 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f72cc0701e0 0x7f72cc1fad10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:52.909 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:52.906+0000 7f72c9d9b640 1 -- 172.21.15.123:0/1213801811 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f72cc1650d0 con 0x7f72cc0c3230 2024-09-16T15:49:52.909 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:52.906+0000 7f72c9d9b640 1 --2- 172.21.15.123:0/1213801811 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f72cc0c3230 0x7f72cc164b90 secure :-1 s=READY pgs=1394 cs=0 l=1 rev1=1 crypto rx=0x7f72c0009850 tx=0x7f72c00a5730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:49:52.909 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:52.906+0000 7f72baffd640 1 -- 172.21.15.123:0/1213801811 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72c00b2040 con 0x7f72cc0c3230 2024-09-16T15:49:52.909 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:52.906+0000 7f72baffd640 1 -- 172.21.15.123:0/1213801811 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f72c00044b0 con 0x7f72cc0c3230 2024-09-16T15:49:52.909 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:52.906+0000 7f72d0dde640 1 -- 172.21.15.123:0/1213801811 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f72cc145e30 con 0x7f72cc0c3230 2024-09-16T15:49:52.909 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:52.906+0000 7f72baffd640 1 -- 172.21.15.123:0/1213801811 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72c0003860 con 0x7f72cc0c3230 2024-09-16T15:49:52.909 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:52.906+0000 7f72d0dde640 1 -- 172.21.15.123:0/1213801811 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f72cc165650 con 0x7f72cc0c3230 2024-09-16T15:49:52.910 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:52.910+0000 7f72d0dde640 1 -- 172.21.15.123:0/1213801811 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f72cc0cedd0 con 0x7f72cc0c3230 2024-09-16T15:49:52.910 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:52.910+0000 7f72baffd640 1 -- 172.21.15.123:0/1213801811 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 21) ==== 97641+0+0 (secure 0 0 0) 0x7f72c00bd060 con 0x7f72cc0c3230 2024-09-16T15:49:52.910 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:52.910+0000 7f72baffd640 1 --2- 172.21.15.123:0/1213801811 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f72ac0734f0 0x7f72ac0759b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:49:52.910 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:52.910+0000 7f72baffd640 1 -- 172.21.15.123:0/1213801811 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(133..133 src has 1..133) ==== 5637+0+0 (secure 0 0 0) 0x7f72c012e4c0 con 0x7f72cc0c3230 2024-09-16T15:49:52.910 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:52.910+0000 7f72ca59c640 1 --2- 172.21.15.123:0/1213801811 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f72ac0734f0 0x7f72ac0759b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:49:52.911 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:52.910+0000 7f72ca59c640 1 --2- 172.21.15.123:0/1213801811 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f72ac0734f0 0x7f72ac0759b0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f72c4005fd0 tx=0x7f72c4005f60 comp rx=0 tx=0).ready entity=mgr.4150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:49:52.914 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:52.914+0000 7f72baffd640 1 -- 172.21.15.123:0/1213801811 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f72c00164d0 con 0x7f72cc0c3230 2024-09-16T15:49:53.123 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:53.122+0000 7f72d0dde640 1 -- 172.21.15.123:0/1213801811 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) -- 0x7f72cc155300 con 0x7f72cc0c3230 2024-09-16T15:49:53.738 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:53.738+0000 7f72baffd640 1 -- 172.21.15.123:0/1213801811 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 20 and data pool 21 v82) ==== 174+0+0 (secure 0 0 0) 0x7f72c00f8e90 con 0x7f72cc0c3230 2024-09-16T15:49:53.739 INFO:teuthology.orchestra.run.smithi123.stderr:new fs with metadata pool 20 and data pool 21 2024-09-16T15:49:53.743 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:53.742+0000 7f72d0dde640 1 -- 172.21.15.123:0/1213801811 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f72ac0734f0 msgr2=0x7f72ac0759b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:53.743 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:53.742+0000 7f72d0dde640 1 --2- 172.21.15.123:0/1213801811 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f72ac0734f0 0x7f72ac0759b0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f72c4005fd0 tx=0x7f72c4005f60 comp rx=0 tx=0).stop 2024-09-16T15:49:53.743 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:53.742+0000 7f72d0dde640 1 -- 172.21.15.123:0/1213801811 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f72cc0c3230 msgr2=0x7f72cc164b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:53.743 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:53.742+0000 7f72d0dde640 1 --2- 172.21.15.123:0/1213801811 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f72cc0c3230 0x7f72cc164b90 secure :-1 s=READY pgs=1394 cs=0 l=1 rev1=1 crypto rx=0x7f72c0009850 tx=0x7f72c00a5730 comp rx=0 tx=0).stop 2024-09-16T15:49:53.743 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:53.742+0000 7f72d0dde640 1 -- 172.21.15.123:0/1213801811 shutdown_connections 2024-09-16T15:49:53.743 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:53.742+0000 7f72d0dde640 1 --2- 172.21.15.123:0/1213801811 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f72ac0734f0 0x7f72ac0759b0 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:53.743 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:53.742+0000 7f72d0dde640 1 --2- 172.21.15.123:0/1213801811 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f72cc15b860 0x7f72cc151be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:53.743 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:53.742+0000 7f72d0dde640 1 --2- 172.21.15.123:0/1213801811 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f72cc0c3230 0x7f72cc164b90 unknown :-1 s=CLOSED pgs=1394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:53.744 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:53.742+0000 7f72d0dde640 1 --2- 172.21.15.123:0/1213801811 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f72cc0701e0 0x7f72cc1fad10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:53.744 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:53.742+0000 7f72d0dde640 1 -- 172.21.15.123:0/1213801811 >> 172.21.15.123:0/1213801811 conn(0x7f72cc0b1710 msgr2=0x7f72cc0cc550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:49:53.744 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:53.742+0000 7f72d0dde640 1 -- 172.21.15.123:0/1213801811 shutdown_connections 2024-09-16T15:49:53.744 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:53.742+0000 7f72d0dde640 1 -- 172.21.15.123:0/1213801811 wait complete. 2024-09-16T15:49:53.781 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-16T15:49:53.906 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:53.906+0000 7fdf39db7640 1 -- 172.21.15.123:0/4215422598 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdf340b25a0 msgr2=0x7fdf340d1c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:53.906 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:53.906+0000 7fdf39db7640 1 --2- 172.21.15.123:0/4215422598 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdf340b25a0 0x7fdf340d1c10 secure :-1 s=READY pgs=1395 cs=0 l=1 rev1=1 crypto rx=0x7fdf24009900 tx=0x7fdf2401c7f0 comp rx=0 tx=0).stop 2024-09-16T15:49:53.906 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:53.906+0000 7fdf39db7640 1 -- 172.21.15.123:0/4215422598 shutdown_connections 2024-09-16T15:49:53.906 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:53.906+0000 7fdf39db7640 1 --2- 172.21.15.123:0/4215422598 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdf340b25a0 0x7fdf340d1c10 unknown :-1 s=CLOSED pgs=1395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:53.906 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:53.906+0000 7fdf39db7640 1 --2- 172.21.15.123:0/4215422598 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fdf3415bda0 0x7fdf340d0310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:53.906 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:53.906+0000 7fdf39db7640 1 -- 172.21.15.123:0/4215422598 >> 172.21.15.123:0/4215422598 conn(0x7fdf34074bd0 msgr2=0x7fdf3409aa40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:49:53.906 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:53.906+0000 7fdf39db7640 1 -- 172.21.15.123:0/4215422598 shutdown_connections 2024-09-16T15:49:53.906 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:53.906+0000 7fdf39db7640 1 -- 172.21.15.123:0/4215422598 wait complete. 2024-09-16T15:49:53.907 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:53.906+0000 7fdf39db7640 1 Processor -- start 2024-09-16T15:49:53.907 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:53.906+0000 7fdf39db7640 1 -- start start 2024-09-16T15:49:53.908 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:53.906+0000 7fdf39db7640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdf340b25a0 0x7fdf340b6590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:49:53.909 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:53.906+0000 7fdf39db7640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fdf340d0850 0x7fdf340b55d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:49:53.909 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:53.906+0000 7fdf39db7640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fdf3415bda0 0x7fdf340b5b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:49:53.909 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:53.906+0000 7fdf39db7640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fdf341692a0 con 0x7fdf340b25a0 2024-09-16T15:49:53.909 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:53.906+0000 7fdf39db7640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fdf34169120 con 0x7fdf3415bda0 2024-09-16T15:49:53.909 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:53.906+0000 7fdf39db7640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fdf34169420 con 0x7fdf340d0850 2024-09-16T15:49:53.909 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:53.906+0000 7fdf337fe640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdf340b25a0 0x7fdf340b6590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:49:53.909 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:53.906+0000 7fdf337fe640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdf340b25a0 0x7fdf340b6590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:58684/0 (socket says 172.21.15.123:58684) 2024-09-16T15:49:53.909 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:53.906+0000 7fdf337fe640 1 -- 172.21.15.123:0/655350712 learned_addr learned my addr 172.21.15.123:0/655350712 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:49:53.909 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:53.906+0000 7fdf33fff640 1 --2- 172.21.15.123:0/655350712 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fdf3415bda0 0x7fdf340b5b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:49:53.909 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:53.906+0000 7fdf337fe640 1 -- 172.21.15.123:0/655350712 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fdf340d0850 msgr2=0x7fdf340b55d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:53.909 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:53.906+0000 7fdf32ffd640 1 --2- 172.21.15.123:0/655350712 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fdf340d0850 0x7fdf340b55d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:49:53.909 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:53.906+0000 7fdf337fe640 1 --2- 172.21.15.123:0/655350712 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fdf340d0850 0x7fdf340b55d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:53.909 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:53.906+0000 7fdf337fe640 1 -- 172.21.15.123:0/655350712 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fdf3415bda0 msgr2=0x7fdf340b5b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:53.910 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:53.906+0000 7fdf337fe640 1 --2- 172.21.15.123:0/655350712 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fdf3415bda0 0x7fdf340b5b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:53.910 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:53.906+0000 7fdf337fe640 1 -- 172.21.15.123:0/655350712 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdf34153f10 con 0x7fdf340b25a0 2024-09-16T15:49:53.910 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:53.906+0000 7fdf33fff640 1 --2- 172.21.15.123:0/655350712 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fdf3415bda0 0x7fdf340b5b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T15:49:53.910 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:53.906+0000 7fdf337fe640 1 --2- 172.21.15.123:0/655350712 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdf340b25a0 0x7fdf340b6590 secure :-1 s=READY pgs=1396 cs=0 l=1 rev1=1 crypto rx=0x7fdf2800c940 tx=0x7fdf2800ce00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:49:53.910 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:53.910+0000 7fdf1bfff640 1 -- 172.21.15.123:0/655350712 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf28004220 con 0x7fdf340b25a0 2024-09-16T15:49:53.910 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:53.910+0000 7fdf1bfff640 1 -- 172.21.15.123:0/655350712 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fdf280079b0 con 0x7fdf340b25a0 2024-09-16T15:49:53.910 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:53.910+0000 7fdf39db7640 1 -- 172.21.15.123:0/655350712 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdf341541f0 con 0x7fdf340b25a0 2024-09-16T15:49:53.910 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:53.910+0000 7fdf1bfff640 1 -- 172.21.15.123:0/655350712 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf280044b0 con 0x7fdf340b25a0 2024-09-16T15:49:53.910 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:53.910+0000 7fdf39db7640 1 -- 172.21.15.123:0/655350712 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdf341547b0 con 0x7fdf340b25a0 2024-09-16T15:49:53.911 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:53.910+0000 7fdf39db7640 1 -- 172.21.15.123:0/655350712 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdf340d1c10 con 0x7fdf340b25a0 2024-09-16T15:49:53.911 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:53.910+0000 7fdf1bfff640 1 -- 172.21.15.123:0/655350712 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 21) ==== 97641+0+0 (secure 0 0 0) 0x7fdf28003600 con 0x7fdf340b25a0 2024-09-16T15:49:53.911 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:53.910+0000 7fdf1bfff640 1 --2- 172.21.15.123:0/655350712 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fdf200734f0 0x7fdf200759b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:49:53.911 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:53.910+0000 7fdf1bfff640 1 -- 172.21.15.123:0/655350712 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(134..134 src has 1..134) ==== 5737+0+0 (secure 0 0 0) 0x7fdf28094ba0 con 0x7fdf340b25a0 2024-09-16T15:49:53.911 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:53.910+0000 7fdf32ffd640 1 --2- 172.21.15.123:0/655350712 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fdf200734f0 0x7fdf200759b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:49:53.911 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:53.910+0000 7fdf32ffd640 1 --2- 172.21.15.123:0/655350712 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fdf200734f0 0x7fdf200759b0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fdf240ae000 tx=0x7fdf2401b090 comp rx=0 tx=0).ready entity=mgr.4150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:49:53.915 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:53.914+0000 7fdf1bfff640 1 -- 172.21.15.123:0/655350712 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7fdf28014030 con 0x7fdf340b25a0 2024-09-16T15:49:54.092 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:54.094+0000 7fdf39db7640 1 -- 172.21.15.123:0/655350712 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fdf34155c50 con 0x7fdf340b25a0 2024-09-16T15:49:54.094 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:54.094+0000 7fdf1bfff640 1 -- 172.21.15.123:0/655350712 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v134) ==== 74+0+16856 (secure 0 0 0) 0x7fdf2805f380 con 0x7fdf340b25a0 2024-09-16T15:49:54.094 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-09-16T15:49:54.094 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":134,"fsid":"56077f89-18c8-486f-ae7e-eecc0bb7822c","created":"2024-09-16T15:42:26.364892+0000","modified":"2024-09-16T15:49:53.127543+0000","last_up_change":"2024-09-16T15:42:38.707234+0000","last_in_change":"2024-09-16T15:42:29.423662+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":21,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-16T15:42:39.287764+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":20,"pool_name":"cephfs_metadata","create_time":"2024-09-16T15:49:50.809741+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"134","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":21,"pool_name":"cephfs_data","create_time":"2024-09-16T15:49:52.080476+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"134","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9e7c1955-a607-4671-9be7-f2ef760ceeaa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":133,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6800","nonce":3882044969},{"type":"v1","addr":"172.21.15.123:6801","nonce":3882044969}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6802","nonce":3882044969},{"type":"v1","addr":"172.21.15.123:6803","nonce":3882044969}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6806","nonce":3882044969},{"type":"v1","addr":"172.21.15.123:6807","nonce":3882044969}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6804","nonce":3882044969},{"type":"v1","addr":"172.21.15.123:6805","nonce":3882044969}]},"public_addr":"172.21.15.123:6801/3882044969","cluster_addr":"172.21.15.123:6803/3882044969","heartbeat_back_addr":"172.21.15.123:6807/3882044969","heartbeat_front_addr":"172.21.15.123:6805/3882044969","state":["exists","up"]},{"osd":1,"uuid":"c540cfde-0977-4742-9b17-fd634668e010","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":133,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6816","nonce":4037001850},{"type":"v1","addr":"172.21.15.123:6817","nonce":4037001850}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6818","nonce":4037001850},{"type":"v1","addr":"172.21.15.123:6819","nonce":4037001850}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6822","nonce":4037001850},{"type":"v1","addr":"172.21.15.123:6823","nonce":4037001850}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6820","nonce":4037001850},{"type":"v1","addr":"172.21.15.123:6821","nonce":4037001850}]},"public_addr":"172.21.15.123:6817/4037001850","cluster_addr":"172.21.15.123:6819/4037001850","heartbeat_back_addr":"172.21.15.123:6823/4037001850","heartbeat_front_addr":"172.21.15.123:6821/4037001850","state":["exists","up"]},{"osd":2,"uuid":"9864a496-f169-486b-95c4-1d59162793a7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":133,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6824","nonce":1332143088},{"type":"v1","addr":"172.21.15.123:6825","nonce":1332143088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6826","nonce":1332143088},{"type":"v1","addr":"172.21.15.123:6827","nonce":1332143088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6830","nonce":1332143088},{"type":"v1","addr":"172.21.15.123:6831","nonce":1332143088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6828","nonce":1332143088},{"type":"v1","addr":"172.21.15.123:6829","nonce":1332143088}]},"public_addr":"172.21.15.123:6825/1332143088","cluster_addr":"172.21.15.123:6827/1332143088","heartbeat_back_addr":"172.21.15.123:6831/1332143088","heartbeat_front_addr":"172.21.15.123:6829/1332143088","state":["exists","up"]},{"osd":3,"uuid":"c6d1f817-f074-4325-95e1-578eba13edda","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":133,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6808","nonce":988586998},{"type":"v1","addr":"172.21.15.123:6809","nonce":988586998}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6810","nonce":988586998},{"type":"v1","addr":"172.21.15.123:6811","nonce":988586998}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6814","nonce":988586998},{"type":"v1","addr":"172.21.15.123:6815","nonce":988586998}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6812","nonce":988586998},{"type":"v1","addr":"172.21.15.123:6813","nonce":988586998}]},"public_addr":"172.21.15.123:6809/988586998","cluster_addr":"172.21.15.123:6811/988586998","heartbeat_back_addr":"172.21.15.123:6815/988586998","heartbeat_front_addr":"172.21.15.123:6813/988586998","state":["exists","up"]},{"osd":4,"uuid":"200a8df4-50ef-40a5-b48a-58981269c086","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":133,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6816","nonce":3387813890},{"type":"v1","addr":"172.21.15.184:6817","nonce":3387813890}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6818","nonce":3387813890},{"type":"v1","addr":"172.21.15.184:6819","nonce":3387813890}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6822","nonce":3387813890},{"type":"v1","addr":"172.21.15.184:6823","nonce":3387813890}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6820","nonce":3387813890},{"type":"v1","addr":"172.21.15.184:6821","nonce":3387813890}]},"public_addr":"172.21.15.184:6817/3387813890","cluster_addr":"172.21.15.184:6819/3387813890","heartbeat_back_addr":"172.21.15.184:6823/3387813890","heartbeat_front_addr":"172.21.15.184:6821/3387813890","state":["exists","up"]},{"osd":5,"uuid":"32d5a900-da3d-412c-bc19-13861a2368d8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":133,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6808","nonce":2680916017},{"type":"v1","addr":"172.21.15.184:6809","nonce":2680916017}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6810","nonce":2680916017},{"type":"v1","addr":"172.21.15.184:6811","nonce":2680916017}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6814","nonce":2680916017},{"type":"v1","addr":"172.21.15.184:6815","nonce":2680916017}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6812","nonce":2680916017},{"type":"v1","addr":"172.21.15.184:6813","nonce":2680916017}]},"public_addr":"172.21.15.184:6809/2680916017","cluster_addr":"172.21.15.184:6811/2680916017","heartbeat_back_addr":"172.21.15.184:6815/2680916017","heartbeat_front_addr":"172.21.15.184:6813/2680916017","state":["exists","up"]},{"osd":6,"uuid":"cde6f39e-aa1b-4260-ac19-774ee6aaca72","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":133,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6824","nonce":2183120457},{"type":"v1","addr":"172.21.15.184:6825","nonce":2183120457}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6826","nonce":2183120457},{"type":"v1","addr":"172.21.15.184:6827","nonce":2183120457}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6830","nonce":2183120457},{"type":"v1","addr":"172.21.15.184:6831","nonce":2183120457}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6828","nonce":2183120457},{"type":"v1","addr":"172.21.15.184:6829","nonce":2183120457}]},"public_addr":"172.21.15.184:6825/2183120457","cluster_addr":"172.21.15.184:6827/2183120457","heartbeat_back_addr":"172.21.15.184:6831/2183120457","heartbeat_front_addr":"172.21.15.184:6829/2183120457","state":["exists","up"]},{"osd":7,"uuid":"4f381778-eee1-47c6-804f-df0e6b33e313","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":133,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6800","nonce":4102957285},{"type":"v1","addr":"172.21.15.184:6801","nonce":4102957285}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6802","nonce":4102957285},{"type":"v1","addr":"172.21.15.184:6803","nonce":4102957285}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6806","nonce":4102957285},{"type":"v1","addr":"172.21.15.184:6807","nonce":4102957285}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6804","nonce":4102957285},{"type":"v1","addr":"172.21.15.184:6805","nonce":4102957285}]},"public_addr":"172.21.15.184:6801/4102957285","cluster_addr":"172.21.15.184:6803/4102957285","heartbeat_back_addr":"172.21.15.184:6807/4102957285","heartbeat_front_addr":"172.21.15.184:6805/4102957285","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:35.972210+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.222007+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.198617+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.385663+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.358935+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.156066+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.752973+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:35.682846+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":17,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T15:49:54.096 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:54.098+0000 7fdf39db7640 1 -- 172.21.15.123:0/655350712 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fdf200734f0 msgr2=0x7fdf200759b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:54.097 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:54.098+0000 7fdf39db7640 1 --2- 172.21.15.123:0/655350712 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fdf200734f0 0x7fdf200759b0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fdf240ae000 tx=0x7fdf2401b090 comp rx=0 tx=0).stop 2024-09-16T15:49:54.097 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:54.098+0000 7fdf39db7640 1 -- 172.21.15.123:0/655350712 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdf340b25a0 msgr2=0x7fdf340b6590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:54.097 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:54.098+0000 7fdf39db7640 1 --2- 172.21.15.123:0/655350712 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdf340b25a0 0x7fdf340b6590 secure :-1 s=READY pgs=1396 cs=0 l=1 rev1=1 crypto rx=0x7fdf2800c940 tx=0x7fdf2800ce00 comp rx=0 tx=0).stop 2024-09-16T15:49:54.097 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:54.098+0000 7fdf39db7640 1 -- 172.21.15.123:0/655350712 shutdown_connections 2024-09-16T15:49:54.097 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:54.098+0000 7fdf39db7640 1 --2- 172.21.15.123:0/655350712 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fdf200734f0 0x7fdf200759b0 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:54.097 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:54.098+0000 7fdf39db7640 1 --2- 172.21.15.123:0/655350712 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fdf3415bda0 0x7fdf340b5b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:54.097 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:54.098+0000 7fdf39db7640 1 --2- 172.21.15.123:0/655350712 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fdf340d0850 0x7fdf340b55d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:54.097 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:54.098+0000 7fdf39db7640 1 --2- 172.21.15.123:0/655350712 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdf340b25a0 0x7fdf340b6590 unknown :-1 s=CLOSED pgs=1396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:54.097 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:54.098+0000 7fdf39db7640 1 -- 172.21.15.123:0/655350712 >> 172.21.15.123:0/655350712 conn(0x7fdf34074bd0 msgr2=0x7fdf340b71f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:49:54.097 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:54.098+0000 7fdf39db7640 1 -- 172.21.15.123:0/655350712 shutdown_connections 2024-09-16T15:49:54.097 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:54.098+0000 7fdf39db7640 1 -- 172.21.15.123:0/655350712 wait complete. 2024-09-16T15:49:54.114 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-16T15:49:54.242 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:54.242+0000 7f2ae5cf0640 1 -- 172.21.15.123:0/117602154 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f2ae00ce560 msgr2=0x7f2ae00d1ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:54.242 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:54.242+0000 7f2ae5cf0640 1 --2- 172.21.15.123:0/117602154 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f2ae00ce560 0x7f2ae00d1ec0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f2ad8009840 tx=0x7f2ad801c7f0 comp rx=0 tx=0).stop 2024-09-16T15:49:54.242 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:54.242+0000 7f2ae5cf0640 1 -- 172.21.15.123:0/117602154 shutdown_connections 2024-09-16T15:49:54.242 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:54.242+0000 7f2ae5cf0640 1 --2- 172.21.15.123:0/117602154 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f2ae00ce560 0x7f2ae00d1ec0 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:54.242 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:54.242+0000 7f2ae5cf0640 1 --2- 172.21.15.123:0/117602154 >> v2:172.21.15.184:3300/0 conn(0x7f2ae015ddc0 0x7f2ae00d1980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:54.242 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:54.242+0000 7f2ae5cf0640 1 -- 172.21.15.123:0/117602154 >> 172.21.15.123:0/117602154 conn(0x7f2ae0074bd0 msgr2=0x7f2ae009aa40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:49:54.243 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:54.242+0000 7f2ae5cf0640 1 -- 172.21.15.123:0/117602154 shutdown_connections 2024-09-16T15:49:54.243 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:54.242+0000 7f2ae5cf0640 1 -- 172.21.15.123:0/117602154 wait complete. 2024-09-16T15:49:54.243 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:54.242+0000 7f2ae5cf0640 1 Processor -- start 2024-09-16T15:49:54.243 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:54.242+0000 7f2ae5cf0640 1 -- start start 2024-09-16T15:49:54.245 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:54.242+0000 7f2ae5cf0640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2ae00b32f0 0x7f2ae00c7720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:49:54.245 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:54.242+0000 7f2ae5cf0640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f2ae00ce560 0x7f2ae014f270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:49:54.245 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:54.242+0000 7f2ae5cf0640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2ae015ddc0 0x7f2ae014eaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:49:54.245 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:54.242+0000 7f2ae5cf0640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f2ae016ad70 con 0x7f2ae00b32f0 2024-09-16T15:49:54.245 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:54.242+0000 7f2ae5cf0640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f2ae006efa0 con 0x7f2ae015ddc0 2024-09-16T15:49:54.246 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:54.242+0000 7f2ae5cf0640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f2ae016aef0 con 0x7f2ae00ce560 2024-09-16T15:49:54.246 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:54.246+0000 7f2adf7fe640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2ae00b32f0 0x7f2ae00c7720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:49:54.246 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:54.246+0000 7f2adf7fe640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2ae00b32f0 0x7f2ae00c7720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:58688/0 (socket says 172.21.15.123:58688) 2024-09-16T15:49:54.246 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:54.246+0000 7f2adf7fe640 1 -- 172.21.15.123:0/4097284332 learned_addr learned my addr 172.21.15.123:0/4097284332 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:49:54.246 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:54.246+0000 7f2adeffd640 1 --2- 172.21.15.123:0/4097284332 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f2ae00ce560 0x7f2ae014f270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:49:54.246 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:54.246+0000 7f2adffff640 1 --2- 172.21.15.123:0/4097284332 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2ae015ddc0 0x7f2ae014eaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:49:54.246 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:54.246+0000 7f2adf7fe640 1 -- 172.21.15.123:0/4097284332 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f2ae00ce560 msgr2=0x7f2ae014f270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:54.246 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:54.246+0000 7f2adf7fe640 1 --2- 172.21.15.123:0/4097284332 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f2ae00ce560 0x7f2ae014f270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:54.246 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:54.246+0000 7f2adf7fe640 1 -- 172.21.15.123:0/4097284332 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2ae015ddc0 msgr2=0x7f2ae014eaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:54.246 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:54.246+0000 7f2adf7fe640 1 --2- 172.21.15.123:0/4097284332 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2ae015ddc0 0x7f2ae014eaa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:54.246 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:54.246+0000 7f2adf7fe640 1 -- 172.21.15.123:0/4097284332 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2ae014f7b0 con 0x7f2ae00b32f0 2024-09-16T15:49:54.246 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:54.246+0000 7f2adeffd640 1 --2- 172.21.15.123:0/4097284332 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f2ae00ce560 0x7f2ae014f270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T15:49:54.246 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:54.246+0000 7f2adf7fe640 1 --2- 172.21.15.123:0/4097284332 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2ae00b32f0 0x7f2ae00c7720 secure :-1 s=READY pgs=1397 cs=0 l=1 rev1=1 crypto rx=0x7f2ad000e9f0 tx=0x7f2ad000edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:49:54.247 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:54.246+0000 7f2ad7fff640 1 -- 172.21.15.123:0/4097284332 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ad000cd80 con 0x7f2ae00b32f0 2024-09-16T15:49:54.247 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:54.246+0000 7f2ad7fff640 1 -- 172.21.15.123:0/4097284332 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f2ad0004020 con 0x7f2ae00b32f0 2024-09-16T15:49:54.247 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:54.246+0000 7f2ae5cf0640 1 -- 172.21.15.123:0/4097284332 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2ae0154000 con 0x7f2ae00b32f0 2024-09-16T15:49:54.247 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:54.246+0000 7f2ad7fff640 1 -- 172.21.15.123:0/4097284332 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ad0004470 con 0x7f2ae00b32f0 2024-09-16T15:49:54.247 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:54.246+0000 7f2ae5cf0640 1 -- 172.21.15.123:0/4097284332 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2ae014fd10 con 0x7f2ae00b32f0 2024-09-16T15:49:54.248 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:54.246+0000 7f2ae5cf0640 1 -- 172.21.15.123:0/4097284332 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2ae01619d0 con 0x7f2ae00b32f0 2024-09-16T15:49:54.248 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:54.246+0000 7f2ad7fff640 1 -- 172.21.15.123:0/4097284332 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 21) ==== 97641+0+0 (secure 0 0 0) 0x7f2ad0003600 con 0x7f2ae00b32f0 2024-09-16T15:49:54.248 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:54.246+0000 7f2ad7fff640 1 --2- 172.21.15.123:0/4097284332 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f2ac00734f0 0x7f2ac00759b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:49:54.248 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:54.246+0000 7f2ad7fff640 1 -- 172.21.15.123:0/4097284332 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(134..134 src has 1..134) ==== 5737+0+0 (secure 0 0 0) 0x7f2ad0095470 con 0x7f2ae00b32f0 2024-09-16T15:49:54.248 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:54.246+0000 7f2adeffd640 1 --2- 172.21.15.123:0/4097284332 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f2ac00734f0 0x7f2ac00759b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:49:54.248 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:54.246+0000 7f2adeffd640 1 --2- 172.21.15.123:0/4097284332 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f2ac00734f0 0x7f2ac00759b0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f2ae00c8020 tx=0x7f2ac8009040 comp rx=0 tx=0).ready entity=mgr.4150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:49:54.251 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:54.250+0000 7f2ad7fff640 1 -- 172.21.15.123:0/4097284332 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f2ae014f7b0 con 0x7f2ae00b32f0 2024-09-16T15:49:54.426 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:54.426+0000 7f2ae5cf0640 1 -- 172.21.15.123:0/4097284332 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f2ae00d0330 con 0x7f2ae00b32f0 2024-09-16T15:49:54.429 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:54.430+0000 7f2ad7fff640 1 -- 172.21.15.123:0/4097284332 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v134) ==== 74+0+16856 (secure 0 0 0) 0x7f2ad005fc60 con 0x7f2ae00b32f0 2024-09-16T15:49:54.429 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-09-16T15:49:54.429 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":134,"fsid":"56077f89-18c8-486f-ae7e-eecc0bb7822c","created":"2024-09-16T15:42:26.364892+0000","modified":"2024-09-16T15:49:53.127543+0000","last_up_change":"2024-09-16T15:42:38.707234+0000","last_in_change":"2024-09-16T15:42:29.423662+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":21,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-16T15:42:39.287764+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":20,"pool_name":"cephfs_metadata","create_time":"2024-09-16T15:49:50.809741+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"134","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":21,"pool_name":"cephfs_data","create_time":"2024-09-16T15:49:52.080476+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"134","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9e7c1955-a607-4671-9be7-f2ef760ceeaa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":133,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6800","nonce":3882044969},{"type":"v1","addr":"172.21.15.123:6801","nonce":3882044969}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6802","nonce":3882044969},{"type":"v1","addr":"172.21.15.123:6803","nonce":3882044969}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6806","nonce":3882044969},{"type":"v1","addr":"172.21.15.123:6807","nonce":3882044969}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6804","nonce":3882044969},{"type":"v1","addr":"172.21.15.123:6805","nonce":3882044969}]},"public_addr":"172.21.15.123:6801/3882044969","cluster_addr":"172.21.15.123:6803/3882044969","heartbeat_back_addr":"172.21.15.123:6807/3882044969","heartbeat_front_addr":"172.21.15.123:6805/3882044969","state":["exists","up"]},{"osd":1,"uuid":"c540cfde-0977-4742-9b17-fd634668e010","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":133,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6816","nonce":4037001850},{"type":"v1","addr":"172.21.15.123:6817","nonce":4037001850}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6818","nonce":4037001850},{"type":"v1","addr":"172.21.15.123:6819","nonce":4037001850}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6822","nonce":4037001850},{"type":"v1","addr":"172.21.15.123:6823","nonce":4037001850}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6820","nonce":4037001850},{"type":"v1","addr":"172.21.15.123:6821","nonce":4037001850}]},"public_addr":"172.21.15.123:6817/4037001850","cluster_addr":"172.21.15.123:6819/4037001850","heartbeat_back_addr":"172.21.15.123:6823/4037001850","heartbeat_front_addr":"172.21.15.123:6821/4037001850","state":["exists","up"]},{"osd":2,"uuid":"9864a496-f169-486b-95c4-1d59162793a7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":133,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6824","nonce":1332143088},{"type":"v1","addr":"172.21.15.123:6825","nonce":1332143088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6826","nonce":1332143088},{"type":"v1","addr":"172.21.15.123:6827","nonce":1332143088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6830","nonce":1332143088},{"type":"v1","addr":"172.21.15.123:6831","nonce":1332143088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6828","nonce":1332143088},{"type":"v1","addr":"172.21.15.123:6829","nonce":1332143088}]},"public_addr":"172.21.15.123:6825/1332143088","cluster_addr":"172.21.15.123:6827/1332143088","heartbeat_back_addr":"172.21.15.123:6831/1332143088","heartbeat_front_addr":"172.21.15.123:6829/1332143088","state":["exists","up"]},{"osd":3,"uuid":"c6d1f817-f074-4325-95e1-578eba13edda","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":133,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6808","nonce":988586998},{"type":"v1","addr":"172.21.15.123:6809","nonce":988586998}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6810","nonce":988586998},{"type":"v1","addr":"172.21.15.123:6811","nonce":988586998}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6814","nonce":988586998},{"type":"v1","addr":"172.21.15.123:6815","nonce":988586998}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6812","nonce":988586998},{"type":"v1","addr":"172.21.15.123:6813","nonce":988586998}]},"public_addr":"172.21.15.123:6809/988586998","cluster_addr":"172.21.15.123:6811/988586998","heartbeat_back_addr":"172.21.15.123:6815/988586998","heartbeat_front_addr":"172.21.15.123:6813/988586998","state":["exists","up"]},{"osd":4,"uuid":"200a8df4-50ef-40a5-b48a-58981269c086","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":133,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6816","nonce":3387813890},{"type":"v1","addr":"172.21.15.184:6817","nonce":3387813890}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6818","nonce":3387813890},{"type":"v1","addr":"172.21.15.184:6819","nonce":3387813890}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6822","nonce":3387813890},{"type":"v1","addr":"172.21.15.184:6823","nonce":3387813890}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6820","nonce":3387813890},{"type":"v1","addr":"172.21.15.184:6821","nonce":3387813890}]},"public_addr":"172.21.15.184:6817/3387813890","cluster_addr":"172.21.15.184:6819/3387813890","heartbeat_back_addr":"172.21.15.184:6823/3387813890","heartbeat_front_addr":"172.21.15.184:6821/3387813890","state":["exists","up"]},{"osd":5,"uuid":"32d5a900-da3d-412c-bc19-13861a2368d8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":133,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6808","nonce":2680916017},{"type":"v1","addr":"172.21.15.184:6809","nonce":2680916017}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6810","nonce":2680916017},{"type":"v1","addr":"172.21.15.184:6811","nonce":2680916017}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6814","nonce":2680916017},{"type":"v1","addr":"172.21.15.184:6815","nonce":2680916017}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6812","nonce":2680916017},{"type":"v1","addr":"172.21.15.184:6813","nonce":2680916017}]},"public_addr":"172.21.15.184:6809/2680916017","cluster_addr":"172.21.15.184:6811/2680916017","heartbeat_back_addr":"172.21.15.184:6815/2680916017","heartbeat_front_addr":"172.21.15.184:6813/2680916017","state":["exists","up"]},{"osd":6,"uuid":"cde6f39e-aa1b-4260-ac19-774ee6aaca72","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":133,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6824","nonce":2183120457},{"type":"v1","addr":"172.21.15.184:6825","nonce":2183120457}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6826","nonce":2183120457},{"type":"v1","addr":"172.21.15.184:6827","nonce":2183120457}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6830","nonce":2183120457},{"type":"v1","addr":"172.21.15.184:6831","nonce":2183120457}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6828","nonce":2183120457},{"type":"v1","addr":"172.21.15.184:6829","nonce":2183120457}]},"public_addr":"172.21.15.184:6825/2183120457","cluster_addr":"172.21.15.184:6827/2183120457","heartbeat_back_addr":"172.21.15.184:6831/2183120457","heartbeat_front_addr":"172.21.15.184:6829/2183120457","state":["exists","up"]},{"osd":7,"uuid":"4f381778-eee1-47c6-804f-df0e6b33e313","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":133,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6800","nonce":4102957285},{"type":"v1","addr":"172.21.15.184:6801","nonce":4102957285}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6802","nonce":4102957285},{"type":"v1","addr":"172.21.15.184:6803","nonce":4102957285}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6806","nonce":4102957285},{"type":"v1","addr":"172.21.15.184:6807","nonce":4102957285}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6804","nonce":4102957285},{"type":"v1","addr":"172.21.15.184:6805","nonce":4102957285}]},"public_addr":"172.21.15.184:6801/4102957285","cluster_addr":"172.21.15.184:6803/4102957285","heartbeat_back_addr":"172.21.15.184:6807/4102957285","heartbeat_front_addr":"172.21.15.184:6805/4102957285","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:35.972210+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.222007+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.198617+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.385663+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.358935+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.156066+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.752973+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:35.682846+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":17,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T15:49:54.431 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:54.430+0000 7f2ae5cf0640 1 -- 172.21.15.123:0/4097284332 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f2ac00734f0 msgr2=0x7f2ac00759b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:54.431 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:54.430+0000 7f2ae5cf0640 1 --2- 172.21.15.123:0/4097284332 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f2ac00734f0 0x7f2ac00759b0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f2ae00c8020 tx=0x7f2ac8009040 comp rx=0 tx=0).stop 2024-09-16T15:49:54.431 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:54.430+0000 7f2ae5cf0640 1 -- 172.21.15.123:0/4097284332 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2ae00b32f0 msgr2=0x7f2ae00c7720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:54.431 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:54.430+0000 7f2ae5cf0640 1 --2- 172.21.15.123:0/4097284332 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2ae00b32f0 0x7f2ae00c7720 secure :-1 s=READY pgs=1397 cs=0 l=1 rev1=1 crypto rx=0x7f2ad000e9f0 tx=0x7f2ad000edf0 comp rx=0 tx=0).stop 2024-09-16T15:49:54.432 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:54.430+0000 7f2ae5cf0640 1 -- 172.21.15.123:0/4097284332 shutdown_connections 2024-09-16T15:49:54.432 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:54.430+0000 7f2ae5cf0640 1 --2- 172.21.15.123:0/4097284332 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f2ac00734f0 0x7f2ac00759b0 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:54.432 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:54.430+0000 7f2ae5cf0640 1 --2- 172.21.15.123:0/4097284332 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2ae015ddc0 0x7f2ae014eaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:54.432 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:54.430+0000 7f2ae5cf0640 1 --2- 172.21.15.123:0/4097284332 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f2ae00ce560 0x7f2ae014f270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:54.432 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:54.430+0000 7f2ae5cf0640 1 --2- 172.21.15.123:0/4097284332 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2ae00b32f0 0x7f2ae00c7720 unknown :-1 s=CLOSED pgs=1397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:54.432 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:54.430+0000 7f2ae5cf0640 1 -- 172.21.15.123:0/4097284332 >> 172.21.15.123:0/4097284332 conn(0x7f2ae0074bd0 msgr2=0x7f2ae0071b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:49:54.432 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:54.430+0000 7f2ae5cf0640 1 -- 172.21.15.123:0/4097284332 shutdown_connections 2024-09-16T15:49:54.432 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:54.430+0000 7f2ae5cf0640 1 -- 172.21.15.123:0/4097284332 wait complete. 2024-09-16T15:49:54.449 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-09-16T15:49:54.581 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:54.582+0000 7f819b198640 1 -- 172.21.15.123:0/2380415150 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f81940cf4e0 msgr2=0x7f819415f6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:54.582 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:54.582+0000 7f819b198640 1 --2- 172.21.15.123:0/2380415150 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f81940cf4e0 0x7f819415f6b0 secure :-1 s=READY pgs=1399 cs=0 l=1 rev1=1 crypto rx=0x7f81940b27d0 tx=0x7f818c01c800 comp rx=0 tx=0).stop 2024-09-16T15:49:54.582 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:54.582+0000 7f819b198640 1 -- 172.21.15.123:0/2380415150 shutdown_connections 2024-09-16T15:49:54.582 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:54.582+0000 7f819b198640 1 --2- 172.21.15.123:0/2380415150 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f81940cf4e0 0x7f819415f6b0 unknown :-1 s=CLOSED pgs=1399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:54.582 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:54.582+0000 7f819b198640 1 --2- 172.21.15.123:0/2380415150 >> v2:172.21.15.184:3300/0 conn(0x7f819415b860 0x7f819406fca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:54.582 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:54.582+0000 7f819b198640 1 -- 172.21.15.123:0/2380415150 >> 172.21.15.123:0/2380415150 conn(0x7f81940b1710 msgr2=0x7f81940b1b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:49:54.582 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:54.582+0000 7f819b198640 1 -- 172.21.15.123:0/2380415150 shutdown_connections 2024-09-16T15:49:54.582 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:54.582+0000 7f819b198640 1 -- 172.21.15.123:0/2380415150 wait complete. 2024-09-16T15:49:54.582 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:54.582+0000 7f819b198640 1 Processor -- start 2024-09-16T15:49:54.583 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:54.582+0000 7f819b198640 1 -- start start 2024-09-16T15:49:54.584 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:54.582+0000 7f819b198640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f81940c3230 0x7f81941550b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:49:54.584 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:54.582+0000 7f819b198640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f81940cf4e0 0x7f819415cd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:49:54.584 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:54.582+0000 7f819b198640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f819415b860 0x7f819415d280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:49:54.584 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:54.582+0000 7f819b198640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f81941690a0 con 0x7f819415b860 2024-09-16T15:49:54.584 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:54.582+0000 7f819b198640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f8194168f20 con 0x7f81940c3230 2024-09-16T15:49:54.584 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:54.582+0000 7f819b198640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f8194169220 con 0x7f81940cf4e0 2024-09-16T15:49:54.585 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:54.582+0000 7f8199735640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f819415b860 0x7f819415d280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:49:54.585 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:54.582+0000 7f8199735640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f819415b860 0x7f819415d280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:58696/0 (socket says 172.21.15.123:58696) 2024-09-16T15:49:54.585 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:54.582+0000 7f8199735640 1 -- 172.21.15.123:0/1634820677 learned_addr learned my addr 172.21.15.123:0/1634820677 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:49:54.585 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:54.582+0000 7f8198f34640 1 --2- 172.21.15.123:0/1634820677 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f81940c3230 0x7f81941550b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:49:54.585 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:54.582+0000 7f818bfff640 1 --2- 172.21.15.123:0/1634820677 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f81940cf4e0 0x7f819415cd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:49:54.585 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:54.582+0000 7f8199735640 1 -- 172.21.15.123:0/1634820677 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f81940cf4e0 msgr2=0x7f819415cd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:54.585 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:54.582+0000 7f8199735640 1 --2- 172.21.15.123:0/1634820677 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f81940cf4e0 0x7f819415cd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:54.585 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:54.582+0000 7f8199735640 1 -- 172.21.15.123:0/1634820677 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f81940c3230 msgr2=0x7f81941550b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:54.585 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:54.582+0000 7f8199735640 1 --2- 172.21.15.123:0/1634820677 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f81940c3230 0x7f81941550b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:54.585 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:54.582+0000 7f8199735640 1 -- 172.21.15.123:0/1634820677 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8194150160 con 0x7f819415b860 2024-09-16T15:49:54.585 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:54.582+0000 7f8199735640 1 --2- 172.21.15.123:0/1634820677 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f819415b860 0x7f819415d280 secure :-1 s=READY pgs=1400 cs=0 l=1 rev1=1 crypto rx=0x7f818c01a430 tx=0x7f818c0a67b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:49:54.585 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:54.582+0000 7f81897fa640 1 -- 172.21.15.123:0/1634820677 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f818c004340 con 0x7f819415b860 2024-09-16T15:49:54.585 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:54.582+0000 7f81897fa640 1 -- 172.21.15.123:0/1634820677 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f818c0be070 con 0x7f819415b860 2024-09-16T15:49:54.585 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:54.582+0000 7f81897fa640 1 -- 172.21.15.123:0/1634820677 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f818c005290 con 0x7f819415b860 2024-09-16T15:49:54.586 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:54.582+0000 7f819b198640 1 -- 172.21.15.123:0/1634820677 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8194150440 con 0x7f819415b860 2024-09-16T15:49:54.586 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:54.586+0000 7f819b198640 1 -- 172.21.15.123:0/1634820677 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8194150a00 con 0x7f819415b860 2024-09-16T15:49:54.586 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:54.586+0000 7f819b198640 1 -- 172.21.15.123:0/1634820677 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8194150ee0 con 0x7f819415b860 2024-09-16T15:49:54.586 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:54.586+0000 7f81897fa640 1 -- 172.21.15.123:0/1634820677 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 21) ==== 97641+0+0 (secure 0 0 0) 0x7f818c005430 con 0x7f819415b860 2024-09-16T15:49:54.586 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:54.586+0000 7f81897fa640 1 --2- 172.21.15.123:0/1634820677 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f81800734f0 0x7f81800759b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:49:54.587 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:54.586+0000 7f81897fa640 1 -- 172.21.15.123:0/1634820677 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(134..134 src has 1..134) ==== 5737+0+0 (secure 0 0 0) 0x7f818c12f960 con 0x7f819415b860 2024-09-16T15:49:54.587 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:54.586+0000 7f8198f34640 1 --2- 172.21.15.123:0/1634820677 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f81800734f0 0x7f81800759b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:49:54.587 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:54.586+0000 7f8198f34640 1 --2- 172.21.15.123:0/1634820677 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f81800734f0 0x7f81800759b0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f8190005c50 tx=0x7f8190005be0 comp rx=0 tx=0).ready entity=mgr.4150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:49:54.590 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:54.590+0000 7f81897fa640 1 -- 172.21.15.123:0/1634820677 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f8194150ee0 con 0x7f819415b860 2024-09-16T15:49:54.802 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:54.802+0000 7f819b198640 1 -- 172.21.15.123:0/1634820677 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) -- 0x7f819415f6b0 con 0x7f819415b860 2024-09-16T15:49:55.709 INFO:tasks.ceph.mds.a.smithi123.stderr:2024-09-16T15:49:55.710+0000 7f145ca6e640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-09-16T15:49:55.784 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:55.782+0000 7f81897fa640 1 -- 172.21.15.123:0/1634820677 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v85) ==== 118+0+0 (secure 0 0 0) 0x7f818c0163f0 con 0x7f819415b860 2024-09-16T15:49:55.787 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:55.786+0000 7f819b198640 1 -- 172.21.15.123:0/1634820677 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f81800734f0 msgr2=0x7f81800759b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:55.787 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:55.786+0000 7f819b198640 1 --2- 172.21.15.123:0/1634820677 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f81800734f0 0x7f81800759b0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f8190005c50 tx=0x7f8190005be0 comp rx=0 tx=0).stop 2024-09-16T15:49:55.787 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:55.786+0000 7f819b198640 1 -- 172.21.15.123:0/1634820677 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f819415b860 msgr2=0x7f819415d280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:55.787 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:55.786+0000 7f819b198640 1 --2- 172.21.15.123:0/1634820677 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f819415b860 0x7f819415d280 secure :-1 s=READY pgs=1400 cs=0 l=1 rev1=1 crypto rx=0x7f818c01a430 tx=0x7f818c0a67b0 comp rx=0 tx=0).stop 2024-09-16T15:49:55.788 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:55.786+0000 7f819b198640 1 -- 172.21.15.123:0/1634820677 shutdown_connections 2024-09-16T15:49:55.788 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:55.786+0000 7f819b198640 1 --2- 172.21.15.123:0/1634820677 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f81800734f0 0x7f81800759b0 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:55.788 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:55.786+0000 7f819b198640 1 --2- 172.21.15.123:0/1634820677 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f819415b860 0x7f819415d280 unknown :-1 s=CLOSED pgs=1400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:55.788 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:55.786+0000 7f819b198640 1 --2- 172.21.15.123:0/1634820677 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f81940cf4e0 0x7f819415cd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:55.788 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:55.786+0000 7f819b198640 1 --2- 172.21.15.123:0/1634820677 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f81940c3230 0x7f81941550b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:55.788 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:55.786+0000 7f819b198640 1 -- 172.21.15.123:0/1634820677 >> 172.21.15.123:0/1634820677 conn(0x7f81940b1710 msgr2=0x7f81940b38a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:49:55.788 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:55.786+0000 7f819b198640 1 -- 172.21.15.123:0/1634820677 shutdown_connections 2024-09-16T15:49:55.788 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:55.786+0000 7f819b198640 1 -- 172.21.15.123:0/1634820677 wait complete. 2024-09-16T15:49:55.808 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-09-16T15:49:55.809 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-09-16T15:49:55.941 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:55.942+0000 7fcdeadc6640 1 -- 172.21.15.123:0/262543009 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcde415aa00 msgr2=0x7fcde415e5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:55.941 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:55.942+0000 7fcdeadc6640 1 --2- 172.21.15.123:0/262543009 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcde415aa00 0x7fcde415e5d0 secure :-1 s=READY pgs=1401 cs=0 l=1 rev1=1 crypto rx=0x7fcdcc009900 tx=0x7fcdcc01c8e0 comp rx=0 tx=0).stop 2024-09-16T15:49:55.941 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:55.942+0000 7fcdeadc6640 1 -- 172.21.15.123:0/262543009 shutdown_connections 2024-09-16T15:49:55.941 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:55.942+0000 7fcdeadc6640 1 --2- 172.21.15.123:0/262543009 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fcde4072720 0x7fcde40cabc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:55.941 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:55.942+0000 7fcdeadc6640 1 --2- 172.21.15.123:0/262543009 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcde415aa00 0x7fcde415e5d0 unknown :-1 s=CLOSED pgs=1401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:55.941 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:55.942+0000 7fcdeadc6640 1 -- 172.21.15.123:0/262543009 >> 172.21.15.123:0/262543009 conn(0x7fcde40b1710 msgr2=0x7fcde40b1b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:49:55.941 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:55.942+0000 7fcdeadc6640 1 -- 172.21.15.123:0/262543009 shutdown_connections 2024-09-16T15:49:55.942 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:55.942+0000 7fcdeadc6640 1 -- 172.21.15.123:0/262543009 wait complete. 2024-09-16T15:49:55.942 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:55.942+0000 7fcdeadc6640 1 Processor -- start 2024-09-16T15:49:55.942 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:55.942+0000 7fcdeadc6640 1 -- start start 2024-09-16T15:49:55.944 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:55.942+0000 7fcdeadc6640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fcde4072720 0x7fcde41f6c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:49:55.944 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:55.942+0000 7fcdeadc6640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcde40cc3c0 0x7fcde414bb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:49:55.944 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:55.942+0000 7fcdeadc6640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fcde415aa00 0x7fcde40b0010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:49:55.944 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:55.942+0000 7fcdeadc6640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fcde41662f0 con 0x7fcde40cc3c0 2024-09-16T15:49:55.944 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:55.942+0000 7fcdeadc6640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fcde4166170 con 0x7fcde4072720 2024-09-16T15:49:55.944 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:55.942+0000 7fcdeadc6640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fcde4166470 con 0x7fcde415aa00 2024-09-16T15:49:55.944 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:55.942+0000 7fcde3fff640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcde40cc3c0 0x7fcde414bb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:49:55.945 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:55.942+0000 7fcde3fff640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcde40cc3c0 0x7fcde414bb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:58700/0 (socket says 172.21.15.123:58700) 2024-09-16T15:49:55.945 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:55.942+0000 7fcde3fff640 1 -- 172.21.15.123:0/273743824 learned_addr learned my addr 172.21.15.123:0/273743824 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:49:55.945 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:55.942+0000 7fcde8b62640 1 --2- 172.21.15.123:0/273743824 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fcde4072720 0x7fcde41f6c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:49:55.945 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:55.942+0000 7fcde3fff640 1 -- 172.21.15.123:0/273743824 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fcde415aa00 msgr2=0x7fcde40b0010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:55.945 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:55.942+0000 7fcde9363640 1 --2- 172.21.15.123:0/273743824 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fcde415aa00 0x7fcde40b0010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:49:55.945 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:55.942+0000 7fcde3fff640 1 --2- 172.21.15.123:0/273743824 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fcde415aa00 0x7fcde40b0010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:55.945 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:55.942+0000 7fcde3fff640 1 -- 172.21.15.123:0/273743824 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fcde4072720 msgr2=0x7fcde41f6c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:55.945 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:55.942+0000 7fcde3fff640 1 --2- 172.21.15.123:0/273743824 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fcde4072720 0x7fcde41f6c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:55.945 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:55.942+0000 7fcde3fff640 1 -- 172.21.15.123:0/273743824 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcde415b1f0 con 0x7fcde40cc3c0 2024-09-16T15:49:55.945 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:55.942+0000 7fcde3fff640 1 --2- 172.21.15.123:0/273743824 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcde40cc3c0 0x7fcde414bb40 secure :-1 s=READY pgs=1402 cs=0 l=1 rev1=1 crypto rx=0x7fcdd400d980 tx=0x7fcdd400de40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:49:55.945 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:55.942+0000 7fcde17fa640 1 -- 172.21.15.123:0/273743824 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcdd4017070 con 0x7fcde40cc3c0 2024-09-16T15:49:55.945 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:55.942+0000 7fcde17fa640 1 -- 172.21.15.123:0/273743824 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fcdd4009d60 con 0x7fcde40cc3c0 2024-09-16T15:49:55.946 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:55.942+0000 7fcdeadc6640 1 -- 172.21.15.123:0/273743824 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcde415b4d0 con 0x7fcde40cc3c0 2024-09-16T15:49:55.946 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:55.942+0000 7fcde17fa640 1 -- 172.21.15.123:0/273743824 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcdd4005280 con 0x7fcde40cc3c0 2024-09-16T15:49:55.946 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:55.942+0000 7fcdeadc6640 1 -- 172.21.15.123:0/273743824 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcde415ba90 con 0x7fcde40cc3c0 2024-09-16T15:49:55.947 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:55.946+0000 7fcdeadc6640 1 -- 172.21.15.123:0/273743824 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcde415bed0 con 0x7fcde40cc3c0 2024-09-16T15:49:55.947 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:55.946+0000 7fcde17fa640 1 -- 172.21.15.123:0/273743824 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 21) ==== 97641+0+0 (secure 0 0 0) 0x7fcdd4005420 con 0x7fcde40cc3c0 2024-09-16T15:49:55.947 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:55.946+0000 7fcde17fa640 1 --2- 172.21.15.123:0/273743824 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fcdc40734f0 0x7fcdc40759b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:49:55.947 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:55.946+0000 7fcde17fa640 1 -- 172.21.15.123:0/273743824 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(135..135 src has 1..135) ==== 5737+0+0 (secure 0 0 0) 0x7fcdd4094c90 con 0x7fcde40cc3c0 2024-09-16T15:49:55.947 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:55.946+0000 7fcde8b62640 1 --2- 172.21.15.123:0/273743824 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fcdc40734f0 0x7fcdc40759b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:49:55.947 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:55.946+0000 7fcde8b62640 1 --2- 172.21.15.123:0/273743824 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fcdc40734f0 0x7fcdc40759b0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fcdcc0af000 tx=0x7fcdcc01b120 comp rx=0 tx=0).ready entity=mgr.4150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:49:55.950 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:55.950+0000 7fcde17fa640 1 -- 172.21.15.123:0/273743824 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7fcdd405f470 con 0x7fcde40cc3c0 2024-09-16T15:49:56.157 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:56.158+0000 7fcdeadc6640 1 -- 172.21.15.123:0/273743824 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) -- 0x7fcde40731a0 con 0x7fcde40cc3c0 2024-09-16T15:49:56.796 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:56.794+0000 7fcde17fa640 1 -- 172.21.15.123:0/273743824 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v86) ==== 122+0+0 (secure 0 0 0) 0x7fcdd405ed70 con 0x7fcde40cc3c0 2024-09-16T15:49:56.799 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:56.798+0000 7fcdeadc6640 1 -- 172.21.15.123:0/273743824 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fcdc40734f0 msgr2=0x7fcdc40759b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:56.799 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:56.798+0000 7fcdeadc6640 1 --2- 172.21.15.123:0/273743824 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fcdc40734f0 0x7fcdc40759b0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fcdcc0af000 tx=0x7fcdcc01b120 comp rx=0 tx=0).stop 2024-09-16T15:49:56.799 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:56.798+0000 7fcdeadc6640 1 -- 172.21.15.123:0/273743824 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcde40cc3c0 msgr2=0x7fcde414bb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:56.799 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:56.798+0000 7fcdeadc6640 1 --2- 172.21.15.123:0/273743824 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcde40cc3c0 0x7fcde414bb40 secure :-1 s=READY pgs=1402 cs=0 l=1 rev1=1 crypto rx=0x7fcdd400d980 tx=0x7fcdd400de40 comp rx=0 tx=0).stop 2024-09-16T15:49:56.799 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:56.798+0000 7fcdeadc6640 1 -- 172.21.15.123:0/273743824 shutdown_connections 2024-09-16T15:49:56.799 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:56.798+0000 7fcdeadc6640 1 --2- 172.21.15.123:0/273743824 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fcdc40734f0 0x7fcdc40759b0 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:56.799 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:56.798+0000 7fcdeadc6640 1 --2- 172.21.15.123:0/273743824 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fcde415aa00 0x7fcde40b0010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:56.799 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:56.798+0000 7fcdeadc6640 1 --2- 172.21.15.123:0/273743824 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcde40cc3c0 0x7fcde414bb40 unknown :-1 s=CLOSED pgs=1402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:56.799 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:56.798+0000 7fcdeadc6640 1 --2- 172.21.15.123:0/273743824 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fcde4072720 0x7fcde41f6c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:56.799 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:56.798+0000 7fcdeadc6640 1 -- 172.21.15.123:0/273743824 >> 172.21.15.123:0/273743824 conn(0x7fcde40b1710 msgr2=0x7fcde40cd6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:49:56.799 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:56.798+0000 7fcdeadc6640 1 -- 172.21.15.123:0/273743824 shutdown_connections 2024-09-16T15:49:56.799 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:56.798+0000 7fcdeadc6640 1 -- 172.21.15.123:0/273743824 wait complete. 2024-09-16T15:49:56.818 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-09-16T15:49:56.818 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-16T15:49:56.945 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:56.946+0000 7f00a94dd640 1 -- 172.21.15.123:0/3878048933 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f00a40b0a50 msgr2=0x7f00a40b0ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:56.945 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:56.946+0000 7f00a94dd640 1 --2- 172.21.15.123:0/3878048933 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f00a40b0a50 0x7f00a40b0ee0 secure :-1 s=READY pgs=1403 cs=0 l=1 rev1=1 crypto rx=0x7f009c009c80 tx=0x7f009c01cc50 comp rx=0 tx=0).stop 2024-09-16T15:49:56.946 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:56.946+0000 7f00a94dd640 1 -- 172.21.15.123:0/3878048933 shutdown_connections 2024-09-16T15:49:56.946 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:56.946+0000 7f00a94dd640 1 --2- 172.21.15.123:0/3878048933 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f00a40b0a50 0x7f00a40b0ee0 unknown :-1 s=CLOSED pgs=1403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:56.946 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:56.946+0000 7f00a94dd640 1 --2- 172.21.15.123:0/3878048933 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f00a40b2b00 0x7f00a40d24b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:56.946 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:56.946+0000 7f00a94dd640 1 -- 172.21.15.123:0/3878048933 >> 172.21.15.123:0/3878048933 conn(0x7f00a40d3ea0 msgr2=0x7f00a40d42b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:49:56.946 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:56.946+0000 7f00a94dd640 1 -- 172.21.15.123:0/3878048933 shutdown_connections 2024-09-16T15:49:56.946 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:56.946+0000 7f00a94dd640 1 -- 172.21.15.123:0/3878048933 wait complete. 2024-09-16T15:49:56.946 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:56.946+0000 7f00a94dd640 1 Processor -- start 2024-09-16T15:49:56.947 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:56.946+0000 7f00a94dd640 1 -- start start 2024-09-16T15:49:56.949 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:56.946+0000 7f00a94dd640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f00a40b0a50 0x7f00a4157060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:49:56.949 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:56.946+0000 7f00a94dd640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f00a40b2b00 0x7f00a414fea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:49:56.949 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:56.946+0000 7f00a94dd640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f00a4161620 0x7f00a41f7aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:49:56.949 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:56.946+0000 7f00a94dd640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f00a416b0f0 con 0x7f00a4161620 2024-09-16T15:49:56.949 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:56.946+0000 7f00a94dd640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f00a416af70 con 0x7f00a40b0a50 2024-09-16T15:49:56.949 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:56.946+0000 7f00a94dd640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f00a416b270 con 0x7f00a40b2b00 2024-09-16T15:49:56.949 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:56.946+0000 7f00a37fe640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f00a4161620 0x7f00a41f7aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:49:56.949 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:56.946+0000 7f00a37fe640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f00a4161620 0x7f00a41f7aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:58712/0 (socket says 172.21.15.123:58712) 2024-09-16T15:49:56.949 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:56.946+0000 7f00a37fe640 1 -- 172.21.15.123:0/2616349271 learned_addr learned my addr 172.21.15.123:0/2616349271 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:49:56.949 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:56.946+0000 7f00a2ffd640 1 --2- 172.21.15.123:0/2616349271 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f00a40b0a50 0x7f00a4157060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:49:56.949 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:56.946+0000 7f00a37fe640 1 -- 172.21.15.123:0/2616349271 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f00a40b2b00 msgr2=0x7f00a414fea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:56.949 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:56.946+0000 7f00a37fe640 1 --2- 172.21.15.123:0/2616349271 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f00a40b2b00 0x7f00a414fea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:56.950 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:56.946+0000 7f00a37fe640 1 -- 172.21.15.123:0/2616349271 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f00a40b0a50 msgr2=0x7f00a4157060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:56.950 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:56.946+0000 7f00a37fe640 1 --2- 172.21.15.123:0/2616349271 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f00a40b0a50 0x7f00a4157060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:56.950 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:56.946+0000 7f00a37fe640 1 -- 172.21.15.123:0/2616349271 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f00a4157640 con 0x7f00a4161620 2024-09-16T15:49:56.950 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:56.946+0000 7f00a37fe640 1 --2- 172.21.15.123:0/2616349271 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f00a4161620 0x7f00a41f7aa0 secure :-1 s=READY pgs=1404 cs=0 l=1 rev1=1 crypto rx=0x7f009c009bd0 tx=0x7f009c009390 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:49:56.950 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:56.950+0000 7f008b7fe640 1 -- 172.21.15.123:0/2616349271 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f009c0bb070 con 0x7f00a4161620 2024-09-16T15:49:56.950 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:56.950+0000 7f008b7fe640 1 -- 172.21.15.123:0/2616349271 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f009c004e00 con 0x7f00a4161620 2024-09-16T15:49:56.950 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:56.950+0000 7f00a94dd640 1 -- 172.21.15.123:0/2616349271 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f00a4157920 con 0x7f00a4161620 2024-09-16T15:49:56.950 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:56.950+0000 7f008b7fe640 1 -- 172.21.15.123:0/2616349271 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f009c0b2040 con 0x7f00a4161620 2024-09-16T15:49:56.950 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:56.950+0000 7f00a94dd640 1 -- 172.21.15.123:0/2616349271 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f00a4157ee0 con 0x7f00a4161620 2024-09-16T15:49:56.951 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:56.950+0000 7f00a94dd640 1 -- 172.21.15.123:0/2616349271 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f00a40b0ee0 con 0x7f00a4161620 2024-09-16T15:49:56.951 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:56.950+0000 7f008b7fe640 1 -- 172.21.15.123:0/2616349271 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 21) ==== 97641+0+0 (secure 0 0 0) 0x7f009c0be070 con 0x7f00a4161620 2024-09-16T15:49:56.951 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:56.950+0000 7f008b7fe640 1 --2- 172.21.15.123:0/2616349271 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f00900734f0 0x7f00900759b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:49:56.951 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:56.950+0000 7f00a2ffd640 1 --2- 172.21.15.123:0/2616349271 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f00900734f0 0x7f00900759b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:49:56.951 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:56.950+0000 7f008b7fe640 1 -- 172.21.15.123:0/2616349271 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(135..135 src has 1..135) ==== 5737+0+0 (secure 0 0 0) 0x7f009c12eef0 con 0x7f00a4161620 2024-09-16T15:49:56.951 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:56.950+0000 7f00a2ffd640 1 --2- 172.21.15.123:0/2616349271 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f00900734f0 0x7f00900759b0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f00a41fc110 tx=0x7f0094008040 comp rx=0 tx=0).ready entity=mgr.4150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:49:56.954 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:56.954+0000 7f008b7fe640 1 -- 172.21.15.123:0/2616349271 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f009c016990 con 0x7f00a4161620 2024-09-16T15:49:57.147 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:57.146+0000 7f00a94dd640 1 -- 172.21.15.123:0/2616349271 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f00a4159a70 con 0x7f00a4161620 2024-09-16T15:49:57.148 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:57.146+0000 7f008b7fe640 1 -- 172.21.15.123:0/2616349271 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 86 v86) ==== 94+0+5230 (secure 0 0 0) 0x7f009c0f97d0 con 0x7f00a4161620 2024-09-16T15:49:57.148 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-09-16T15:49:57.148 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":86,"btime":"2024-09-16T15:49:56:787440+0000","default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5801,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6834/730567479","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":730567479},{"type":"v1","addr":"172.21.15.184:6834","nonce":730567479}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":81},{"gid":6346,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6835/584518652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6833","nonce":584518652},{"type":"v1","addr":"172.21.15.184:6835","nonce":584518652}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":81},{"gid":7938,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6833/3935767668","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6832","nonce":3935767668},{"type":"v1","addr":"172.21.15.123:6833","nonce":3935767668}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":80}],"filesystems":[{"mdsmap":{"epoch":86,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-16T15:49:53.127484+0000","modified":"2024-09-16T15:49:56.160938+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7944},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7944":{"gid":7944,"name":"a","rank":0,"incarnation":82,"state":"up:active","state_seq":3,"addr":"172.21.15.123:6835/333722639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6834","nonce":333722639},{"type":"v1","addr":"172.21.15.123:6835","nonce":333722639}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7944,"qdb_cluster":[7944]},"id":10}]} 2024-09-16T15:49:57.149 INFO:teuthology.orchestra.run.smithi123.stderr:dumped fsmap epoch 86 2024-09-16T15:49:57.150 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:57.150+0000 7f00a94dd640 1 -- 172.21.15.123:0/2616349271 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f00900734f0 msgr2=0x7f00900759b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:57.150 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:57.150+0000 7f00a94dd640 1 --2- 172.21.15.123:0/2616349271 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f00900734f0 0x7f00900759b0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f00a41fc110 tx=0x7f0094008040 comp rx=0 tx=0).stop 2024-09-16T15:49:57.151 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:57.150+0000 7f00a94dd640 1 -- 172.21.15.123:0/2616349271 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f00a4161620 msgr2=0x7f00a41f7aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:57.151 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:57.150+0000 7f00a94dd640 1 --2- 172.21.15.123:0/2616349271 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f00a4161620 0x7f00a41f7aa0 secure :-1 s=READY pgs=1404 cs=0 l=1 rev1=1 crypto rx=0x7f009c009bd0 tx=0x7f009c009390 comp rx=0 tx=0).stop 2024-09-16T15:49:57.151 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:57.150+0000 7f00a94dd640 1 -- 172.21.15.123:0/2616349271 shutdown_connections 2024-09-16T15:49:57.151 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:57.150+0000 7f00a94dd640 1 --2- 172.21.15.123:0/2616349271 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f00900734f0 0x7f00900759b0 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:57.151 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:57.150+0000 7f00a94dd640 1 --2- 172.21.15.123:0/2616349271 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f00a4161620 0x7f00a41f7aa0 unknown :-1 s=CLOSED pgs=1404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:57.151 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:57.150+0000 7f00a94dd640 1 --2- 172.21.15.123:0/2616349271 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f00a40b2b00 0x7f00a414fea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:57.151 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:57.150+0000 7f00a94dd640 1 --2- 172.21.15.123:0/2616349271 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f00a40b0a50 0x7f00a4157060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:57.151 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:57.150+0000 7f00a94dd640 1 -- 172.21.15.123:0/2616349271 >> 172.21.15.123:0/2616349271 conn(0x7f00a40d3ea0 msgr2=0x7f00a415c820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:49:57.151 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:57.150+0000 7f00a94dd640 1 -- 172.21.15.123:0/2616349271 shutdown_connections 2024-09-16T15:49:57.151 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:57.150+0000 7f00a94dd640 1 -- 172.21.15.123:0/2616349271 wait complete. 2024-09-16T15:49:57.166 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-16T15:49:57.292 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:57.290+0000 7fc87b43a640 1 -- 172.21.15.123:0/365800619 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc874074600 msgr2=0x7fc87406e750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:57.292 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:57.290+0000 7fc87b43a640 1 --2- 172.21.15.123:0/365800619 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc874074600 0x7fc87406e750 secure :-1 s=READY pgs=1405 cs=0 l=1 rev1=1 crypto rx=0x7fc8740cd650 tx=0x7fc86c01ccc0 comp rx=0 tx=0).stop 2024-09-16T15:49:57.292 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:57.290+0000 7fc87b43a640 1 -- 172.21.15.123:0/365800619 shutdown_connections 2024-09-16T15:49:57.292 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:57.290+0000 7fc87b43a640 1 --2- 172.21.15.123:0/365800619 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc874074600 0x7fc87406e750 unknown :-1 s=CLOSED pgs=1405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:57.292 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:57.290+0000 7fc87b43a640 1 --2- 172.21.15.123:0/365800619 >> v2:172.21.15.184:3300/0 conn(0x7fc87415bca0 0x7fc8740d0630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:57.292 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:57.290+0000 7fc87b43a640 1 -- 172.21.15.123:0/365800619 >> 172.21.15.123:0/365800619 conn(0x7fc8740cfb10 msgr2=0x7fc8740cff20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:49:57.292 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:57.290+0000 7fc87b43a640 1 -- 172.21.15.123:0/365800619 shutdown_connections 2024-09-16T15:49:57.292 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:57.290+0000 7fc87b43a640 1 -- 172.21.15.123:0/365800619 wait complete. 2024-09-16T15:49:57.293 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:57.294+0000 7fc87b43a640 1 Processor -- start 2024-09-16T15:49:57.293 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:57.294+0000 7fc87b43a640 1 -- start start 2024-09-16T15:49:57.295 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:57.294+0000 7fc87b43a640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc8740726c0 0x7fc8740c96b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:49:57.295 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:57.294+0000 7fc87b43a640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc874074600 0x7fc8741f7f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:49:57.295 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:57.294+0000 7fc87b43a640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fc87415bca0 0x7fc8741f88d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:49:57.295 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:57.294+0000 7fc87b43a640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fc874168a50 con 0x7fc874074600 2024-09-16T15:49:57.296 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:57.294+0000 7fc87b43a640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fc87406f9e0 con 0x7fc8740726c0 2024-09-16T15:49:57.296 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:57.294+0000 7fc87b43a640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fc874168bd0 con 0x7fc87415bca0 2024-09-16T15:49:57.296 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:57.294+0000 7fc8789d5640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc874074600 0x7fc8741f7f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:49:57.296 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:57.294+0000 7fc8789d5640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc874074600 0x7fc8741f7f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:58732/0 (socket says 172.21.15.123:58732) 2024-09-16T15:49:57.296 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:57.294+0000 7fc8789d5640 1 -- 172.21.15.123:0/144798284 learned_addr learned my addr 172.21.15.123:0/144798284 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:49:57.296 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:57.294+0000 7fc8799d7640 1 --2- 172.21.15.123:0/144798284 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fc87415bca0 0x7fc8741f88d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:49:57.296 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:57.294+0000 7fc8791d6640 1 --2- 172.21.15.123:0/144798284 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc8740726c0 0x7fc8740c96b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:49:57.296 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:57.294+0000 7fc8789d5640 1 -- 172.21.15.123:0/144798284 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fc87415bca0 msgr2=0x7fc8741f88d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:57.296 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:57.294+0000 7fc8789d5640 1 --2- 172.21.15.123:0/144798284 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fc87415bca0 0x7fc8741f88d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:57.296 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:57.294+0000 7fc8789d5640 1 -- 172.21.15.123:0/144798284 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc8740726c0 msgr2=0x7fc8740c96b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:57.296 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:57.294+0000 7fc8789d5640 1 --2- 172.21.15.123:0/144798284 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc8740726c0 0x7fc8740c96b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:57.296 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:57.294+0000 7fc8789d5640 1 -- 172.21.15.123:0/144798284 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc8740c9bf0 con 0x7fc874074600 2024-09-16T15:49:57.297 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:57.294+0000 7fc8789d5640 1 --2- 172.21.15.123:0/144798284 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc874074600 0x7fc8741f7f60 secure :-1 s=READY pgs=1406 cs=0 l=1 rev1=1 crypto rx=0x7fc86400c490 tx=0x7fc864001e80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:49:57.297 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:57.294+0000 7fc869ffb640 1 -- 172.21.15.123:0/144798284 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc864017070 con 0x7fc874074600 2024-09-16T15:49:57.297 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:57.294+0000 7fc869ffb640 1 -- 172.21.15.123:0/144798284 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fc864003600 con 0x7fc874074600 2024-09-16T15:49:57.297 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:57.294+0000 7fc87b43a640 1 -- 172.21.15.123:0/144798284 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc8740c35b0 con 0x7fc874074600 2024-09-16T15:49:57.297 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:57.294+0000 7fc869ffb640 1 -- 172.21.15.123:0/144798284 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc864003a90 con 0x7fc874074600 2024-09-16T15:49:57.297 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:57.294+0000 7fc87b43a640 1 -- 172.21.15.123:0/144798284 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc874151440 con 0x7fc874074600 2024-09-16T15:49:57.298 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:57.294+0000 7fc87b43a640 1 -- 172.21.15.123:0/144798284 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc87406e750 con 0x7fc874074600 2024-09-16T15:49:57.298 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:57.298+0000 7fc869ffb640 1 -- 172.21.15.123:0/144798284 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 21) ==== 97641+0+0 (secure 0 0 0) 0x7fc86401a070 con 0x7fc874074600 2024-09-16T15:49:57.298 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:57.298+0000 7fc869ffb640 1 --2- 172.21.15.123:0/144798284 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fc8580734f0 0x7fc8580759b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:49:57.298 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:57.298+0000 7fc869ffb640 1 -- 172.21.15.123:0/144798284 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(135..135 src has 1..135) ==== 5737+0+0 (secure 0 0 0) 0x7fc864094190 con 0x7fc874074600 2024-09-16T15:49:57.298 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:57.298+0000 7fc8791d6640 1 --2- 172.21.15.123:0/144798284 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fc8580734f0 0x7fc8580759b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:49:57.298 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:57.298+0000 7fc8791d6640 1 --2- 172.21.15.123:0/144798284 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fc8580734f0 0x7fc8580759b0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fc8700058d0 tx=0x7fc870005860 comp rx=0 tx=0).ready entity=mgr.4150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:49:57.301 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:57.302+0000 7fc869ffb640 1 -- 172.21.15.123:0/144798284 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7fc864021020 con 0x7fc874074600 2024-09-16T15:49:57.462 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:57.462+0000 7fc87b43a640 1 -- 172.21.15.123:0/144798284 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fc87415f630 con 0x7fc874074600 2024-09-16T15:49:57.464 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:57.462+0000 7fc869ffb640 1 -- 172.21.15.123:0/144798284 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v135) ==== 74+0+16843 (secure 0 0 0) 0x7fc86405eb00 con 0x7fc874074600 2024-09-16T15:49:57.464 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-09-16T15:49:57.465 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":135,"fsid":"56077f89-18c8-486f-ae7e-eecc0bb7822c","created":"2024-09-16T15:42:26.364892+0000","modified":"2024-09-16T15:49:54.741464+0000","last_up_change":"2024-09-16T15:42:38.707234+0000","last_in_change":"2024-09-16T15:42:29.423662+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":21,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-16T15:42:39.287764+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":20,"pool_name":"cephfs_metadata","create_time":"2024-09-16T15:49:50.809741+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"134","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":21,"pool_name":"cephfs_data","create_time":"2024-09-16T15:49:52.080476+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"135","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9e7c1955-a607-4671-9be7-f2ef760ceeaa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":133,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6800","nonce":3882044969},{"type":"v1","addr":"172.21.15.123:6801","nonce":3882044969}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6802","nonce":3882044969},{"type":"v1","addr":"172.21.15.123:6803","nonce":3882044969}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6806","nonce":3882044969},{"type":"v1","addr":"172.21.15.123:6807","nonce":3882044969}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6804","nonce":3882044969},{"type":"v1","addr":"172.21.15.123:6805","nonce":3882044969}]},"public_addr":"172.21.15.123:6801/3882044969","cluster_addr":"172.21.15.123:6803/3882044969","heartbeat_back_addr":"172.21.15.123:6807/3882044969","heartbeat_front_addr":"172.21.15.123:6805/3882044969","state":["exists","up"]},{"osd":1,"uuid":"c540cfde-0977-4742-9b17-fd634668e010","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":133,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6816","nonce":4037001850},{"type":"v1","addr":"172.21.15.123:6817","nonce":4037001850}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6818","nonce":4037001850},{"type":"v1","addr":"172.21.15.123:6819","nonce":4037001850}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6822","nonce":4037001850},{"type":"v1","addr":"172.21.15.123:6823","nonce":4037001850}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6820","nonce":4037001850},{"type":"v1","addr":"172.21.15.123:6821","nonce":4037001850}]},"public_addr":"172.21.15.123:6817/4037001850","cluster_addr":"172.21.15.123:6819/4037001850","heartbeat_back_addr":"172.21.15.123:6823/4037001850","heartbeat_front_addr":"172.21.15.123:6821/4037001850","state":["exists","up"]},{"osd":2,"uuid":"9864a496-f169-486b-95c4-1d59162793a7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":133,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6824","nonce":1332143088},{"type":"v1","addr":"172.21.15.123:6825","nonce":1332143088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6826","nonce":1332143088},{"type":"v1","addr":"172.21.15.123:6827","nonce":1332143088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6830","nonce":1332143088},{"type":"v1","addr":"172.21.15.123:6831","nonce":1332143088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6828","nonce":1332143088},{"type":"v1","addr":"172.21.15.123:6829","nonce":1332143088}]},"public_addr":"172.21.15.123:6825/1332143088","cluster_addr":"172.21.15.123:6827/1332143088","heartbeat_back_addr":"172.21.15.123:6831/1332143088","heartbeat_front_addr":"172.21.15.123:6829/1332143088","state":["exists","up"]},{"osd":3,"uuid":"c6d1f817-f074-4325-95e1-578eba13edda","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":133,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6808","nonce":988586998},{"type":"v1","addr":"172.21.15.123:6809","nonce":988586998}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6810","nonce":988586998},{"type":"v1","addr":"172.21.15.123:6811","nonce":988586998}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6814","nonce":988586998},{"type":"v1","addr":"172.21.15.123:6815","nonce":988586998}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6812","nonce":988586998},{"type":"v1","addr":"172.21.15.123:6813","nonce":988586998}]},"public_addr":"172.21.15.123:6809/988586998","cluster_addr":"172.21.15.123:6811/988586998","heartbeat_back_addr":"172.21.15.123:6815/988586998","heartbeat_front_addr":"172.21.15.123:6813/988586998","state":["exists","up"]},{"osd":4,"uuid":"200a8df4-50ef-40a5-b48a-58981269c086","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":133,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6816","nonce":3387813890},{"type":"v1","addr":"172.21.15.184:6817","nonce":3387813890}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6818","nonce":3387813890},{"type":"v1","addr":"172.21.15.184:6819","nonce":3387813890}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6822","nonce":3387813890},{"type":"v1","addr":"172.21.15.184:6823","nonce":3387813890}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6820","nonce":3387813890},{"type":"v1","addr":"172.21.15.184:6821","nonce":3387813890}]},"public_addr":"172.21.15.184:6817/3387813890","cluster_addr":"172.21.15.184:6819/3387813890","heartbeat_back_addr":"172.21.15.184:6823/3387813890","heartbeat_front_addr":"172.21.15.184:6821/3387813890","state":["exists","up"]},{"osd":5,"uuid":"32d5a900-da3d-412c-bc19-13861a2368d8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":133,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6808","nonce":2680916017},{"type":"v1","addr":"172.21.15.184:6809","nonce":2680916017}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6810","nonce":2680916017},{"type":"v1","addr":"172.21.15.184:6811","nonce":2680916017}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6814","nonce":2680916017},{"type":"v1","addr":"172.21.15.184:6815","nonce":2680916017}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6812","nonce":2680916017},{"type":"v1","addr":"172.21.15.184:6813","nonce":2680916017}]},"public_addr":"172.21.15.184:6809/2680916017","cluster_addr":"172.21.15.184:6811/2680916017","heartbeat_back_addr":"172.21.15.184:6815/2680916017","heartbeat_front_addr":"172.21.15.184:6813/2680916017","state":["exists","up"]},{"osd":6,"uuid":"cde6f39e-aa1b-4260-ac19-774ee6aaca72","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":133,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6824","nonce":2183120457},{"type":"v1","addr":"172.21.15.184:6825","nonce":2183120457}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6826","nonce":2183120457},{"type":"v1","addr":"172.21.15.184:6827","nonce":2183120457}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6830","nonce":2183120457},{"type":"v1","addr":"172.21.15.184:6831","nonce":2183120457}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6828","nonce":2183120457},{"type":"v1","addr":"172.21.15.184:6829","nonce":2183120457}]},"public_addr":"172.21.15.184:6825/2183120457","cluster_addr":"172.21.15.184:6827/2183120457","heartbeat_back_addr":"172.21.15.184:6831/2183120457","heartbeat_front_addr":"172.21.15.184:6829/2183120457","state":["exists","up"]},{"osd":7,"uuid":"4f381778-eee1-47c6-804f-df0e6b33e313","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":133,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6800","nonce":4102957285},{"type":"v1","addr":"172.21.15.184:6801","nonce":4102957285}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6802","nonce":4102957285},{"type":"v1","addr":"172.21.15.184:6803","nonce":4102957285}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6806","nonce":4102957285},{"type":"v1","addr":"172.21.15.184:6807","nonce":4102957285}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6804","nonce":4102957285},{"type":"v1","addr":"172.21.15.184:6805","nonce":4102957285}]},"public_addr":"172.21.15.184:6801/4102957285","cluster_addr":"172.21.15.184:6803/4102957285","heartbeat_back_addr":"172.21.15.184:6807/4102957285","heartbeat_front_addr":"172.21.15.184:6805/4102957285","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:35.972210+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.222007+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.198617+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.385663+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.358935+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.156066+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.752973+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:35.682846+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":17,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T15:49:57.467 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:57.466+0000 7fc87b43a640 1 -- 172.21.15.123:0/144798284 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fc8580734f0 msgr2=0x7fc8580759b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:57.467 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:57.466+0000 7fc87b43a640 1 --2- 172.21.15.123:0/144798284 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fc8580734f0 0x7fc8580759b0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fc8700058d0 tx=0x7fc870005860 comp rx=0 tx=0).stop 2024-09-16T15:49:57.467 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:57.466+0000 7fc87b43a640 1 -- 172.21.15.123:0/144798284 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc874074600 msgr2=0x7fc8741f7f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:57.467 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:57.466+0000 7fc87b43a640 1 --2- 172.21.15.123:0/144798284 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc874074600 0x7fc8741f7f60 secure :-1 s=READY pgs=1406 cs=0 l=1 rev1=1 crypto rx=0x7fc86400c490 tx=0x7fc864001e80 comp rx=0 tx=0).stop 2024-09-16T15:49:57.467 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:57.466+0000 7fc87b43a640 1 -- 172.21.15.123:0/144798284 shutdown_connections 2024-09-16T15:49:57.467 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:57.466+0000 7fc87b43a640 1 --2- 172.21.15.123:0/144798284 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fc8580734f0 0x7fc8580759b0 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:57.468 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:57.466+0000 7fc87b43a640 1 --2- 172.21.15.123:0/144798284 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fc87415bca0 0x7fc8741f88d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:57.468 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:57.466+0000 7fc87b43a640 1 --2- 172.21.15.123:0/144798284 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc874074600 0x7fc8741f7f60 unknown :-1 s=CLOSED pgs=1406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:57.468 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:57.466+0000 7fc87b43a640 1 --2- 172.21.15.123:0/144798284 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc8740726c0 0x7fc8740c96b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:57.468 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:57.466+0000 7fc87b43a640 1 -- 172.21.15.123:0/144798284 >> 172.21.15.123:0/144798284 conn(0x7fc8740cfb10 msgr2=0x7fc8740b29a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:49:57.468 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:57.466+0000 7fc87b43a640 1 -- 172.21.15.123:0/144798284 shutdown_connections 2024-09-16T15:49:57.468 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:57.466+0000 7fc87b43a640 1 -- 172.21.15.123:0/144798284 wait complete. 2024-09-16T15:49:57.483 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-09-16T15:49:57.483 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T15:49:57.610 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:57.610+0000 7fbfc3586640 1 -- 172.21.15.123:0/3474988791 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbfbc0b2b00 msgr2=0x7fbfbc0d22e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:57.610 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:57.610+0000 7fbfc3586640 1 --2- 172.21.15.123:0/3474988791 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbfbc0b2b00 0x7fbfbc0d22e0 secure :-1 s=READY pgs=1407 cs=0 l=1 rev1=1 crypto rx=0x7fbfbc06f930 tx=0x7fbfac01c7a0 comp rx=0 tx=0).stop 2024-09-16T15:49:57.610 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:57.610+0000 7fbfc3586640 1 -- 172.21.15.123:0/3474988791 shutdown_connections 2024-09-16T15:49:57.610 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:57.610+0000 7fbfc3586640 1 --2- 172.21.15.123:0/3474988791 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbfbc0b2b00 0x7fbfbc0d22e0 unknown :-1 s=CLOSED pgs=1407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:57.610 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:57.610+0000 7fbfc3586640 1 --2- 172.21.15.123:0/3474988791 >> v2:172.21.15.184:3300/0 conn(0x7fbfbc161620 0x7fbfbc0d1da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:57.610 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:57.610+0000 7fbfc3586640 1 -- 172.21.15.123:0/3474988791 >> 172.21.15.123:0/3474988791 conn(0x7fbfbc0d3ea0 msgr2=0x7fbfbc0d42b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:49:57.610 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:57.610+0000 7fbfc3586640 1 -- 172.21.15.123:0/3474988791 shutdown_connections 2024-09-16T15:49:57.610 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:57.610+0000 7fbfc3586640 1 -- 172.21.15.123:0/3474988791 wait complete. 2024-09-16T15:49:57.612 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:57.610+0000 7fbfc3586640 1 Processor -- start 2024-09-16T15:49:57.613 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:57.610+0000 7fbfc3586640 1 -- start start 2024-09-16T15:49:57.613 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:57.610+0000 7fbfc3586640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbfbc0b2b00 0x7fbfbc149380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:49:57.613 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:57.610+0000 7fbfc3586640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fbfbc0d2820 0x7fbfbc15d1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:49:57.613 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:57.610+0000 7fbfc3586640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fbfbc161620 0x7fbfbc15d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:49:57.613 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:57.610+0000 7fbfc3586640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fbfbc16aee0 con 0x7fbfbc0b2b00 2024-09-16T15:49:57.613 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:57.610+0000 7fbfc3586640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fbfbc16ad60 con 0x7fbfbc0d2820 2024-09-16T15:49:57.613 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:57.610+0000 7fbfc3586640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fbfbc16b060 con 0x7fbfbc161620 2024-09-16T15:49:57.613 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:57.610+0000 7fbfc1322640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbfbc0b2b00 0x7fbfbc149380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:49:57.613 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:57.610+0000 7fbfc1322640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbfbc0b2b00 0x7fbfbc149380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:58752/0 (socket says 172.21.15.123:58752) 2024-09-16T15:49:57.613 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:57.610+0000 7fbfc1322640 1 -- 172.21.15.123:0/1532453240 learned_addr learned my addr 172.21.15.123:0/1532453240 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:49:57.613 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:57.610+0000 7fbfc0b21640 1 --2- 172.21.15.123:0/1532453240 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fbfbc0d2820 0x7fbfbc15d1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:49:57.614 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:57.610+0000 7fbfc1b23640 1 --2- 172.21.15.123:0/1532453240 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fbfbc161620 0x7fbfbc15d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:49:57.614 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:57.610+0000 7fbfc1322640 1 -- 172.21.15.123:0/1532453240 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fbfbc161620 msgr2=0x7fbfbc15d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:57.614 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:57.610+0000 7fbfc1322640 1 --2- 172.21.15.123:0/1532453240 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fbfbc161620 0x7fbfbc15d6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:57.614 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:57.610+0000 7fbfc1322640 1 -- 172.21.15.123:0/1532453240 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fbfbc0d2820 msgr2=0x7fbfbc15d1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:57.614 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:57.610+0000 7fbfc1322640 1 --2- 172.21.15.123:0/1532453240 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fbfbc0d2820 0x7fbfbc15d1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:57.614 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:57.610+0000 7fbfc1322640 1 -- 172.21.15.123:0/1532453240 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbfbc1572a0 con 0x7fbfbc0b2b00 2024-09-16T15:49:57.614 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:57.610+0000 7fbfc1322640 1 --2- 172.21.15.123:0/1532453240 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbfbc0b2b00 0x7fbfbc149380 secure :-1 s=READY pgs=1408 cs=0 l=1 rev1=1 crypto rx=0x7fbfb800c9a0 tx=0x7fbfb800ce60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:49:57.614 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:57.610+0000 7fbfb1ffb640 1 -- 172.21.15.123:0/1532453240 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbfb8007bb0 con 0x7fbfbc0b2b00 2024-09-16T15:49:57.614 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:57.610+0000 7fbfb1ffb640 1 -- 172.21.15.123:0/1532453240 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fbfb8007e40 con 0x7fbfbc0b2b00 2024-09-16T15:49:57.614 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:57.610+0000 7fbfc3586640 1 -- 172.21.15.123:0/1532453240 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbfbc1fad40 con 0x7fbfbc0b2b00 2024-09-16T15:49:57.614 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:57.610+0000 7fbfb1ffb640 1 -- 172.21.15.123:0/1532453240 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbfb8005270 con 0x7fbfbc0b2b00 2024-09-16T15:49:57.614 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:57.610+0000 7fbfc3586640 1 -- 172.21.15.123:0/1532453240 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbfbc1fb300 con 0x7fbfbc0b2b00 2024-09-16T15:49:57.615 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:57.614+0000 7fbfc3586640 1 -- 172.21.15.123:0/1532453240 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbfbc0cea00 con 0x7fbfbc0b2b00 2024-09-16T15:49:57.615 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:57.614+0000 7fbfb1ffb640 1 -- 172.21.15.123:0/1532453240 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 21) ==== 97641+0+0 (secure 0 0 0) 0x7fbfb8005410 con 0x7fbfbc0b2b00 2024-09-16T15:49:57.615 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:57.614+0000 7fbfb1ffb640 1 --2- 172.21.15.123:0/1532453240 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fbfa00734f0 0x7fbfa00759b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:49:57.615 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:57.614+0000 7fbfc0b21640 1 --2- 172.21.15.123:0/1532453240 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fbfa00734f0 0x7fbfa00759b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:49:57.615 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:57.614+0000 7fbfb1ffb640 1 -- 172.21.15.123:0/1532453240 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(135..135 src has 1..135) ==== 5737+0+0 (secure 0 0 0) 0x7fbfb8094f00 con 0x7fbfbc0b2b00 2024-09-16T15:49:57.616 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:57.614+0000 7fbfc0b21640 1 --2- 172.21.15.123:0/1532453240 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fbfa00734f0 0x7fbfa00759b0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fbfac0af000 tx=0x7fbfac01afe0 comp rx=0 tx=0).ready entity=mgr.4150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:49:57.618 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:57.618+0000 7fbfb1ffb640 1 -- 172.21.15.123:0/1532453240 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7fbfbc1572a0 con 0x7fbfbc0b2b00 2024-09-16T15:49:57.788 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:57.786+0000 7fbfc3586640 1 -- 172.21.15.123:0/1532453240 --> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fbfbc0a1b30 con 0x7fbfa00734f0 2024-09-16T15:49:57.796 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:57.794+0000 7fbfb1ffb640 1 -- 172.21.15.123:0/1532453240 <== mgr.4150 v2:172.21.15.123:6836/2464759676 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+252480 (secure 0 0 0) 0x7fbfbc0a1b30 con 0x7fbfa00734f0 2024-09-16T15:49:57.799 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-09-16T15:49:57.801 INFO:teuthology.orchestra.run.smithi123.stderr:dumped all 2024-09-16T15:49:57.802 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:57.798+0000 7fbfc3586640 1 -- 172.21.15.123:0/1532453240 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fbfa00734f0 msgr2=0x7fbfa00759b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:57.802 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:57.798+0000 7fbfc3586640 1 --2- 172.21.15.123:0/1532453240 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fbfa00734f0 0x7fbfa00759b0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fbfac0af000 tx=0x7fbfac01afe0 comp rx=0 tx=0).stop 2024-09-16T15:49:57.802 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:57.798+0000 7fbfc3586640 1 -- 172.21.15.123:0/1532453240 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbfbc0b2b00 msgr2=0x7fbfbc149380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:57.802 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:57.798+0000 7fbfc3586640 1 --2- 172.21.15.123:0/1532453240 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbfbc0b2b00 0x7fbfbc149380 secure :-1 s=READY pgs=1408 cs=0 l=1 rev1=1 crypto rx=0x7fbfb800c9a0 tx=0x7fbfb800ce60 comp rx=0 tx=0).stop 2024-09-16T15:49:57.802 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:57.798+0000 7fbfc3586640 1 -- 172.21.15.123:0/1532453240 shutdown_connections 2024-09-16T15:49:57.802 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:57.798+0000 7fbfc3586640 1 --2- 172.21.15.123:0/1532453240 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fbfa00734f0 0x7fbfa00759b0 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:57.802 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:57.798+0000 7fbfc3586640 1 --2- 172.21.15.123:0/1532453240 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fbfbc161620 0x7fbfbc15d6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:57.802 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:57.798+0000 7fbfc3586640 1 --2- 172.21.15.123:0/1532453240 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fbfbc0d2820 0x7fbfbc15d1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:57.802 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:57.798+0000 7fbfc3586640 1 --2- 172.21.15.123:0/1532453240 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbfbc0b2b00 0x7fbfbc149380 unknown :-1 s=CLOSED pgs=1408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:57.802 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:57.798+0000 7fbfc3586640 1 -- 172.21.15.123:0/1532453240 >> 172.21.15.123:0/1532453240 conn(0x7fbfbc0d3ea0 msgr2=0x7fbfbc15c350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:49:57.803 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:57.798+0000 7fbfc3586640 1 -- 172.21.15.123:0/1532453240 shutdown_connections 2024-09-16T15:49:57.803 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:57.798+0000 7fbfc3586640 1 -- 172.21.15.123:0/1532453240 wait complete. 2024-09-16T15:49:57.817 INFO:teuthology.orchestra.run.smithi123.stdout:{"pg_ready":true,"pg_map":{"version":137,"stamp":"2024-09-16T15:49:56.108698+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":157,"num_write_kb":1642,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":80,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":263860,"kb_used_data":14336,"kb_used_omap":12,"kb_used_meta":249331,"kb_avail":754710860,"statfs":{"total":773094113280,"available":772823920640,"internally_reserved":0,"allocated":14680064,"data_stored":7134612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":255315505},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":46,"apply_latency_ms":46,"commit_latency_ns":46000000,"apply_latency_ns":46000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.380126"},"pg_stats":[{"pgid":"21.3f","version":"0'0","reported_seq":13,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-09-16T15:49:55.017428+0000","last_change":"2024-09-16T15:49:53.776624+0000","last_active":"2024-09-16T15:49:55.017428+0000","last_peered":"2024-09-16T15:49:55.017428+0000","last_clean":"2024-09-16T15:49:55.017428+0000","last_became_active":"2024-09-16T15:49:53.776370+0000","last_became_peered":"2024-09-16T15:49:53.776370+0000","last_unstale":"2024-09-16T15:49:55.017428+0000","last_undegraded":"2024-09-16T15:49:55.017428+0000","last_fullsized":"2024-09-16T15:49:55.017428+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:49:52.725252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:49:52.725252+0000","last_clean_scrub_stamp":"2024-09-16T15:49:52.725252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:26:03.608900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.3e","version":"0'0","reported_seq":13,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-09-16T15:49:54.749305+0000","last_change":"2024-09-16T15:49:53.772932+0000","last_active":"2024-09-16T15:49:54.749305+0000","last_peered":"2024-09-16T15:49:54.749305+0000","last_clean":"2024-09-16T15:49:54.749305+0000","last_became_active":"2024-09-16T15:49:53.772471+0000","last_became_peered":"2024-09-16T15:49:53.772471+0000","last_unstale":"2024-09-16T15:49:54.749305+0000","last_undegraded":"2024-09-16T15:49:54.749305+0000","last_fullsized":"2024-09-16T15:49:54.749305+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:49:52.725252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:49:52.725252+0000","last_clean_scrub_stamp":"2024-09-16T15:49:52.725252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T01:36:37.006276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.3d","version":"0'0","reported_seq":13,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-09-16T15:49:54.752437+0000","last_change":"2024-09-16T15:49:53.808252+0000","last_active":"2024-09-16T15:49:54.752437+0000","last_peered":"2024-09-16T15:49:54.752437+0000","last_clean":"2024-09-16T15:49:54.752437+0000","last_became_active":"2024-09-16T15:49:53.807204+0000","last_became_peered":"2024-09-16T15:49:53.807204+0000","last_unstale":"2024-09-16T15:49:54.752437+0000","last_undegraded":"2024-09-16T15:49:54.752437+0000","last_fullsized":"2024-09-16T15:49:54.752437+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:49:52.725252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:49:52.725252+0000","last_clean_scrub_stamp":"2024-09-16T15:49:52.725252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T23:32:34.768446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"21.3c","version":"0'0","reported_seq":13,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-09-16T15:49:54.752083+0000","last_change":"2024-09-16T15:49:53.809842+0000","last_active":"2024-09-16T15:49:54.752083+0000","last_peered":"2024-09-16T15:49:54.752083+0000","last_clean":"2024-09-16T15:49:54.752083+0000","last_became_active":"2024-09-16T15:49:53.809353+0000","last_became_peered":"2024-09-16T15:49:53.809353+0000","last_unstale":"2024-09-16T15:49:54.752083+0000","last_undegraded":"2024-09-16T15:49:54.752083+0000","last_fullsized":"2024-09-16T15:49:54.752083+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:49:52.725252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:49:52.725252+0000","last_clean_scrub_stamp":"2024-09-16T15:49:52.725252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T22:01:27.459937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.3b","version":"0'0","reported_seq":13,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-09-16T15:49:55.014985+0000","last_change":"2024-09-16T15:49:53.794026+0000","last_active":"2024-09-16T15:49:55.014985+0000","last_peered":"2024-09-16T15:49:55.014985+0000","last_clean":"2024-09-16T15:49:55.014985+0000","last_became_active":"2024-09-16T15:49:53.790246+0000","last_became_peered":"2024-09-16T15:49:53.790246+0000","last_unstale":"2024-09-16T15:49:55.014985+0000","last_undegraded":"2024-09-16T15:49:55.014985+0000","last_fullsized":"2024-09-16T15:49:55.014985+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:49:52.725252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:49:52.725252+0000","last_clean_scrub_stamp":"2024-09-16T15:49:52.725252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T22:23:38.505887+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"21.3a","version":"0'0","reported_seq":13,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-09-16T15:49:55.015658+0000","last_change":"2024-09-16T15:49:53.808447+0000","last_active":"2024-09-16T15:49:55.015658+0000","last_peered":"2024-09-16T15:49:55.015658+0000","last_clean":"2024-09-16T15:49:55.015658+0000","last_became_active":"2024-09-16T15:49:53.808152+0000","last_became_peered":"2024-09-16T15:49:53.808152+0000","last_unstale":"2024-09-16T15:49:55.015658+0000","last_undegraded":"2024-09-16T15:49:55.015658+0000","last_fullsized":"2024-09-16T15:49:55.015658+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:49:52.725252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:49:52.725252+0000","last_clean_scrub_stamp":"2024-09-16T15:49:52.725252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T22:48:58.657590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.39","version":"0'0","reported_seq":13,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-09-16T15:49:55.016077+0000","last_change":"2024-09-16T15:49:53.774744+0000","last_active":"2024-09-16T15:49:55.016077+0000","last_peered":"2024-09-16T15:49:55.016077+0000","last_clean":"2024-09-16T15:49:55.016077+0000","last_became_active":"2024-09-16T15:49:53.774466+0000","last_became_peered":"2024-09-16T15:49:53.774466+0000","last_unstale":"2024-09-16T15:49:55.016077+0000","last_undegraded":"2024-09-16T15:49:55.016077+0000","last_fullsized":"2024-09-16T15:49:55.016077+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:49:52.725252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:49:52.725252+0000","last_clean_scrub_stamp":"2024-09-16T15:49:52.725252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:44:50.941421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"21.38","version":"0'0","reported_seq":13,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-09-16T15:49:55.014928+0000","last_change":"2024-09-16T15:49:53.785007+0000","last_active":"2024-09-16T15:49:55.014928+0000","last_peered":"2024-09-16T15:49:55.014928+0000","last_clean":"2024-09-16T15:49:55.014928+0000","last_became_active":"2024-09-16T15:49:53.784695+0000","last_became_peered":"2024-09-16T15:49:53.784695+0000","last_unstale":"2024-09-16T15:49:55.014928+0000","last_undegraded":"2024-09-16T15:49:55.014928+0000","last_fullsized":"2024-09-16T15:49:55.014928+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:49:52.725252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:49:52.725252+0000","last_clean_scrub_stamp":"2024-09-16T15:49:52.725252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:45:44.627500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.37","version":"0'0","reported_seq":13,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-09-16T15:49:54.749146+0000","last_change":"2024-09-16T15:49:53.773613+0000","last_active":"2024-09-16T15:49:54.749146+0000","last_peered":"2024-09-16T15:49:54.749146+0000","last_clean":"2024-09-16T15:49:54.749146+0000","last_became_active":"2024-09-16T15:49:53.773284+0000","last_became_peered":"2024-09-16T15:49:53.773284+0000","last_unstale":"2024-09-16T15:49:54.749146+0000","last_undegraded":"2024-09-16T15:49:54.749146+0000","last_fullsized":"2024-09-16T15:49:54.749146+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:49:52.725252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:49:52.725252+0000","last_clean_scrub_stamp":"2024-09-16T15:49:52.725252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:43:49.339223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.36","version":"0'0","reported_seq":13,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-09-16T15:49:55.015877+0000","last_change":"2024-09-16T15:49:53.799514+0000","last_active":"2024-09-16T15:49:55.015877+0000","last_peered":"2024-09-16T15:49:55.015877+0000","last_clean":"2024-09-16T15:49:55.015877+0000","last_became_active":"2024-09-16T15:49:53.798835+0000","last_became_peered":"2024-09-16T15:49:53.798835+0000","last_unstale":"2024-09-16T15:49:55.015877+0000","last_undegraded":"2024-09-16T15:49:55.015877+0000","last_fullsized":"2024-09-16T15:49:55.015877+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:49:52.725252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:49:52.725252+0000","last_clean_scrub_stamp":"2024-09-16T15:49:52.725252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T23:31:40.351018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.35","version":"0'0","reported_seq":13,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-09-16T15:49:55.015824+0000","last_change":"2024-09-16T15:49:53.783766+0000","last_active":"2024-09-16T15:49:55.015824+0000","last_peered":"2024-09-16T15:49:55.015824+0000","last_clean":"2024-09-16T15:49:55.015824+0000","last_became_active":"2024-09-16T15:49:53.783442+0000","last_became_peered":"2024-09-16T15:49:53.783442+0000","last_unstale":"2024-09-16T15:49:55.015824+0000","last_undegraded":"2024-09-16T15:49:55.015824+0000","last_fullsized":"2024-09-16T15:49:55.015824+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:49:52.725252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:49:52.725252+0000","last_clean_scrub_stamp":"2024-09-16T15:49:52.725252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:00:04.218087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.34","version":"0'0","reported_seq":13,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-09-16T15:49:55.016097+0000","last_change":"2024-09-16T15:49:53.801949+0000","last_active":"2024-09-16T15:49:55.016097+0000","last_peered":"2024-09-16T15:49:55.016097+0000","last_clean":"2024-09-16T15:49:55.016097+0000","last_became_active":"2024-09-16T15:49:53.800152+0000","last_became_peered":"2024-09-16T15:49:53.800152+0000","last_unstale":"2024-09-16T15:49:55.016097+0000","last_undegraded":"2024-09-16T15:49:55.016097+0000","last_fullsized":"2024-09-16T15:49:55.016097+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:49:52.725252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:49:52.725252+0000","last_clean_scrub_stamp":"2024-09-16T15:49:52.725252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:23:25.956451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.33","version":"0'0","reported_seq":13,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-09-16T15:49:55.013657+0000","last_change":"2024-09-16T15:49:53.793781+0000","last_active":"2024-09-16T15:49:55.013657+0000","last_peered":"2024-09-16T15:49:55.013657+0000","last_clean":"2024-09-16T15:49:55.013657+0000","last_became_active":"2024-09-16T15:49:53.784790+0000","last_became_peered":"2024-09-16T15:49:53.784790+0000","last_unstale":"2024-09-16T15:49:55.013657+0000","last_undegraded":"2024-09-16T15:49:55.013657+0000","last_fullsized":"2024-09-16T15:49:55.013657+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:49:52.725252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:49:52.725252+0000","last_clean_scrub_stamp":"2024-09-16T15:49:52.725252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T02:02:21.975949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"21.32","version":"0'0","reported_seq":13,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-09-16T15:49:54.750051+0000","last_change":"2024-09-16T15:49:53.778571+0000","last_active":"2024-09-16T15:49:54.750051+0000","last_peered":"2024-09-16T15:49:54.750051+0000","last_clean":"2024-09-16T15:49:54.750051+0000","last_became_active":"2024-09-16T15:49:53.777156+0000","last_became_peered":"2024-09-16T15:49:53.777156+0000","last_unstale":"2024-09-16T15:49:54.750051+0000","last_undegraded":"2024-09-16T15:49:54.750051+0000","last_fullsized":"2024-09-16T15:49:54.750051+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:49:52.725252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:49:52.725252+0000","last_clean_scrub_stamp":"2024-09-16T15:49:52.725252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T22:22:36.895961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.31","version":"0'0","reported_seq":13,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-09-16T15:49:55.014470+0000","last_change":"2024-09-16T15:49:53.782437+0000","last_active":"2024-09-16T15:49:55.014470+0000","last_peered":"2024-09-16T15:49:55.014470+0000","last_clean":"2024-09-16T15:49:55.014470+0000","last_became_active":"2024-09-16T15:49:53.781726+0000","last_became_peered":"2024-09-16T15:49:53.781726+0000","last_unstale":"2024-09-16T15:49:55.014470+0000","last_undegraded":"2024-09-16T15:49:55.014470+0000","last_fullsized":"2024-09-16T15:49:55.014470+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:49:52.725252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:49:52.725252+0000","last_clean_scrub_stamp":"2024-09-16T15:49:52.725252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T01:18:42.700556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.30","version":"0'0","reported_seq":13,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-09-16T15:49:55.015366+0000","last_change":"2024-09-16T15:49:53.762868+0000","last_active":"2024-09-16T15:49:55.015366+0000","last_peered":"2024-09-16T15:49:55.015366+0000","last_clean":"2024-09-16T15:49:55.015366+0000","last_became_active":"2024-09-16T15:49:53.762584+0000","last_became_peered":"2024-09-16T15:49:53.762584+0000","last_unstale":"2024-09-16T15:49:55.015366+0000","last_undegraded":"2024-09-16T15:49:55.015366+0000","last_fullsized":"2024-09-16T15:49:55.015366+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:49:52.725252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:49:52.725252+0000","last_clean_scrub_stamp":"2024-09-16T15:49:52.725252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T01:51:32.315200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.2f","version":"0'0","reported_seq":13,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-09-16T15:49:55.015773+0000","last_change":"2024-09-16T15:49:53.798961+0000","last_active":"2024-09-16T15:49:55.015773+0000","last_peered":"2024-09-16T15:49:55.015773+0000","last_clean":"2024-09-16T15:49:55.015773+0000","last_became_active":"2024-09-16T15:49:53.797676+0000","last_became_peered":"2024-09-16T15:49:53.797676+0000","last_unstale":"2024-09-16T15:49:55.015773+0000","last_undegraded":"2024-09-16T15:49:55.015773+0000","last_fullsized":"2024-09-16T15:49:55.015773+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:49:52.725252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:49:52.725252+0000","last_clean_scrub_stamp":"2024-09-16T15:49:52.725252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T03:05:01.704120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.2e","version":"0'0","reported_seq":13,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-09-16T15:49:54.753203+0000","last_change":"2024-09-16T15:49:53.806273+0000","last_active":"2024-09-16T15:49:54.753203+0000","last_peered":"2024-09-16T15:49:54.753203+0000","last_clean":"2024-09-16T15:49:54.753203+0000","last_became_active":"2024-09-16T15:49:53.805609+0000","last_became_peered":"2024-09-16T15:49:53.805609+0000","last_unstale":"2024-09-16T15:49:54.753203+0000","last_undegraded":"2024-09-16T15:49:54.753203+0000","last_fullsized":"2024-09-16T15:49:54.753203+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:49:52.725252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:49:52.725252+0000","last_clean_scrub_stamp":"2024-09-16T15:49:52.725252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T02:15:44.638806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"21.2d","version":"0'0","reported_seq":13,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-09-16T15:49:55.017511+0000","last_change":"2024-09-16T15:49:53.761753+0000","last_active":"2024-09-16T15:49:55.017511+0000","last_peered":"2024-09-16T15:49:55.017511+0000","last_clean":"2024-09-16T15:49:55.017511+0000","last_became_active":"2024-09-16T15:49:53.761452+0000","last_became_peered":"2024-09-16T15:49:53.761452+0000","last_unstale":"2024-09-16T15:49:55.017511+0000","last_undegraded":"2024-09-16T15:49:55.017511+0000","last_fullsized":"2024-09-16T15:49:55.017511+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:49:52.725252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:49:52.725252+0000","last_clean_scrub_stamp":"2024-09-16T15:49:52.725252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:07:17.368998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.2c","version":"0'0","reported_seq":13,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-09-16T15:49:55.017110+0000","last_change":"2024-09-16T15:49:53.772294+0000","last_active":"2024-09-16T15:49:55.017110+0000","last_peered":"2024-09-16T15:49:55.017110+0000","last_clean":"2024-09-16T15:49:55.017110+0000","last_became_active":"2024-09-16T15:49:53.771986+0000","last_became_peered":"2024-09-16T15:49:53.771986+0000","last_unstale":"2024-09-16T15:49:55.017110+0000","last_undegraded":"2024-09-16T15:49:55.017110+0000","last_fullsized":"2024-09-16T15:49:55.017110+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:49:52.725252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:49:52.725252+0000","last_clean_scrub_stamp":"2024-09-16T15:49:52.725252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:55:21.714737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.2b","version":"0'0","reported_seq":13,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-09-16T15:49:55.015237+0000","last_change":"2024-09-16T15:49:53.790512+0000","last_active":"2024-09-16T15:49:55.015237+0000","last_peered":"2024-09-16T15:49:55.015237+0000","last_clean":"2024-09-16T15:49:55.015237+0000","last_became_active":"2024-09-16T15:49:53.789779+0000","last_became_peered":"2024-09-16T15:49:53.789779+0000","last_unstale":"2024-09-16T15:49:55.015237+0000","last_undegraded":"2024-09-16T15:49:55.015237+0000","last_fullsized":"2024-09-16T15:49:55.015237+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:49:52.725252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:49:52.725252+0000","last_clean_scrub_stamp":"2024-09-16T15:49:52.725252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T18:23:11.685238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.2a","version":"0'0","reported_seq":13,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-09-16T15:49:54.750453+0000","last_change":"2024-09-16T15:49:53.776142+0000","last_active":"2024-09-16T15:49:54.750453+0000","last_peered":"2024-09-16T15:49:54.750453+0000","last_clean":"2024-09-16T15:49:54.750453+0000","last_became_active":"2024-09-16T15:49:53.772977+0000","last_became_peered":"2024-09-16T15:49:53.772977+0000","last_unstale":"2024-09-16T15:49:54.750453+0000","last_undegraded":"2024-09-16T15:49:54.750453+0000","last_fullsized":"2024-09-16T15:49:54.750453+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:49:52.725252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:49:52.725252+0000","last_clean_scrub_stamp":"2024-09-16T15:49:52.725252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T23:39:37.396671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.29","version":"0'0","reported_seq":13,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-09-16T15:49:54.750006+0000","last_change":"2024-09-16T15:49:53.767624+0000","last_active":"2024-09-16T15:49:54.750006+0000","last_peered":"2024-09-16T15:49:54.750006+0000","last_clean":"2024-09-16T15:49:54.750006+0000","last_became_active":"2024-09-16T15:49:53.766908+0000","last_became_peered":"2024-09-16T15:49:53.766908+0000","last_unstale":"2024-09-16T15:49:54.750006+0000","last_undegraded":"2024-09-16T15:49:54.750006+0000","last_fullsized":"2024-09-16T15:49:54.750006+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:49:52.725252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:49:52.725252+0000","last_clean_scrub_stamp":"2024-09-16T15:49:52.725252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T01:08:26.895306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.28","version":"0'0","reported_seq":13,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-09-16T15:49:55.014366+0000","last_change":"2024-09-16T15:49:53.789465+0000","last_active":"2024-09-16T15:49:55.014366+0000","last_peered":"2024-09-16T15:49:55.014366+0000","last_clean":"2024-09-16T15:49:55.014366+0000","last_became_active":"2024-09-16T15:49:53.788737+0000","last_became_peered":"2024-09-16T15:49:53.788737+0000","last_unstale":"2024-09-16T15:49:55.014366+0000","last_undegraded":"2024-09-16T15:49:55.014366+0000","last_fullsized":"2024-09-16T15:49:55.014366+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:49:52.725252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:49:52.725252+0000","last_clean_scrub_stamp":"2024-09-16T15:49:52.725252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:10:45.743329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.27","version":"0'0","reported_seq":13,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-09-16T15:49:55.016881+0000","last_change":"2024-09-16T15:49:53.770290+0000","last_active":"2024-09-16T15:49:55.016881+0000","last_peered":"2024-09-16T15:49:55.016881+0000","last_clean":"2024-09-16T15:49:55.016881+0000","last_became_active":"2024-09-16T15:49:53.770020+0000","last_became_peered":"2024-09-16T15:49:53.770020+0000","last_unstale":"2024-09-16T15:49:55.016881+0000","last_undegraded":"2024-09-16T15:49:55.016881+0000","last_fullsized":"2024-09-16T15:49:55.016881+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:49:52.725252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:49:52.725252+0000","last_clean_scrub_stamp":"2024-09-16T15:49:52.725252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:21:26.353413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.26","version":"0'0","reported_seq":13,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-09-16T15:49:55.015999+0000","last_change":"2024-09-16T15:49:53.761671+0000","last_active":"2024-09-16T15:49:55.015999+0000","last_peered":"2024-09-16T15:49:55.015999+0000","last_clean":"2024-09-16T15:49:55.015999+0000","last_became_active":"2024-09-16T15:49:53.761225+0000","last_became_peered":"2024-09-16T15:49:53.761225+0000","last_unstale":"2024-09-16T15:49:55.015999+0000","last_undegraded":"2024-09-16T15:49:55.015999+0000","last_fullsized":"2024-09-16T15:49:55.015999+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:49:52.725252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:49:52.725252+0000","last_clean_scrub_stamp":"2024-09-16T15:49:52.725252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T23:33:08.035496+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.25","version":"0'0","reported_seq":13,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-09-16T15:49:55.016927+0000","last_change":"2024-09-16T15:49:53.809122+0000","last_active":"2024-09-16T15:49:55.016927+0000","last_peered":"2024-09-16T15:49:55.016927+0000","last_clean":"2024-09-16T15:49:55.016927+0000","last_became_active":"2024-09-16T15:49:53.808805+0000","last_became_peered":"2024-09-16T15:49:53.808805+0000","last_unstale":"2024-09-16T15:49:55.016927+0000","last_undegraded":"2024-09-16T15:49:55.016927+0000","last_fullsized":"2024-09-16T15:49:55.016927+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:49:52.725252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:49:52.725252+0000","last_clean_scrub_stamp":"2024-09-16T15:49:52.725252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:53:15.380959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.24","version":"0'0","reported_seq":13,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-09-16T15:49:55.017233+0000","last_change":"2024-09-16T15:49:53.814338+0000","last_active":"2024-09-16T15:49:55.017233+0000","last_peered":"2024-09-16T15:49:55.017233+0000","last_clean":"2024-09-16T15:49:55.017233+0000","last_became_active":"2024-09-16T15:49:53.813945+0000","last_became_peered":"2024-09-16T15:49:53.813945+0000","last_unstale":"2024-09-16T15:49:55.017233+0000","last_undegraded":"2024-09-16T15:49:55.017233+0000","last_fullsized":"2024-09-16T15:49:55.017233+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:49:52.725252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:49:52.725252+0000","last_clean_scrub_stamp":"2024-09-16T15:49:52.725252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T02:59:39.191888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"21.23","version":"0'0","reported_seq":13,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-09-16T15:49:55.014885+0000","last_change":"2024-09-16T15:49:53.806453+0000","last_active":"2024-09-16T15:49:55.014885+0000","last_peered":"2024-09-16T15:49:55.014885+0000","last_clean":"2024-09-16T15:49:55.014885+0000","last_became_active":"2024-09-16T15:49:53.806200+0000","last_became_peered":"2024-09-16T15:49:53.806200+0000","last_unstale":"2024-09-16T15:49:55.014885+0000","last_undegraded":"2024-09-16T15:49:55.014885+0000","last_fullsized":"2024-09-16T15:49:55.014885+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:49:52.725252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:49:52.725252+0000","last_clean_scrub_stamp":"2024-09-16T15:49:52.725252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:11:19.892285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.22","version":"0'0","reported_seq":13,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-09-16T15:49:54.750774+0000","last_change":"2024-09-16T15:49:53.758217+0000","last_active":"2024-09-16T15:49:54.750774+0000","last_peered":"2024-09-16T15:49:54.750774+0000","last_clean":"2024-09-16T15:49:54.750774+0000","last_became_active":"2024-09-16T15:49:53.757252+0000","last_became_peered":"2024-09-16T15:49:53.757252+0000","last_unstale":"2024-09-16T15:49:54.750774+0000","last_undegraded":"2024-09-16T15:49:54.750774+0000","last_fullsized":"2024-09-16T15:49:54.750774+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:49:52.725252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:49:52.725252+0000","last_clean_scrub_stamp":"2024-09-16T15:49:52.725252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T00:21:32.615337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.21","version":"0'0","reported_seq":13,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-09-16T15:49:54.748650+0000","last_change":"2024-09-16T15:49:53.810297+0000","last_active":"2024-09-16T15:49:54.748650+0000","last_peered":"2024-09-16T15:49:54.748650+0000","last_clean":"2024-09-16T15:49:54.748650+0000","last_became_active":"2024-09-16T15:49:53.810018+0000","last_became_peered":"2024-09-16T15:49:53.810018+0000","last_unstale":"2024-09-16T15:49:54.748650+0000","last_undegraded":"2024-09-16T15:49:54.748650+0000","last_fullsized":"2024-09-16T15:49:54.748650+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:49:52.725252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:49:52.725252+0000","last_clean_scrub_stamp":"2024-09-16T15:49:52.725252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T23:50:35.196885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.20","version":"0'0","reported_seq":13,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-09-16T15:49:55.015940+0000","last_change":"2024-09-16T15:49:53.785301+0000","last_active":"2024-09-16T15:49:55.015940+0000","last_peered":"2024-09-16T15:49:55.015940+0000","last_clean":"2024-09-16T15:49:55.015940+0000","last_became_active":"2024-09-16T15:49:53.784991+0000","last_became_peered":"2024-09-16T15:49:53.784991+0000","last_unstale":"2024-09-16T15:49:55.015940+0000","last_undegraded":"2024-09-16T15:49:55.015940+0000","last_fullsized":"2024-09-16T15:49:55.015940+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:49:52.725252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:49:52.725252+0000","last_clean_scrub_stamp":"2024-09-16T15:49:52.725252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:28:00.897124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.1e","version":"0'0","reported_seq":13,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-09-16T15:49:55.015777+0000","last_change":"2024-09-16T15:49:53.811225+0000","last_active":"2024-09-16T15:49:55.015777+0000","last_peered":"2024-09-16T15:49:55.015777+0000","last_clean":"2024-09-16T15:49:55.015777+0000","last_became_active":"2024-09-16T15:49:53.810759+0000","last_became_peered":"2024-09-16T15:49:53.810759+0000","last_unstale":"2024-09-16T15:49:55.015777+0000","last_undegraded":"2024-09-16T15:49:55.015777+0000","last_fullsized":"2024-09-16T15:49:55.015777+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:49:52.725252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:49:52.725252+0000","last_clean_scrub_stamp":"2024-09-16T15:49:52.725252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T22:09:52.767577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.1f","version":"134'4","reported_seq":19,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-09-16T15:49:55.015563+0000","last_change":"2024-09-16T15:49:52.775873+0000","last_active":"2024-09-16T15:49:55.015563+0000","last_peered":"2024-09-16T15:49:55.015563+0000","last_clean":"2024-09-16T15:49:55.015563+0000","last_became_active":"2024-09-16T15:49:52.773732+0000","last_became_peered":"2024-09-16T15:49:52.773732+0000","last_unstale":"2024-09-16T15:49:55.015563+0000","last_undegraded":"2024-09-16T15:49:55.015563+0000","last_fullsized":"2024-09-16T15:49:55.015563+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:49:51.721471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:49:51.721471+0000","last_clean_scrub_stamp":"2024-09-16T15:49:51.721471+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:23:53.699929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.1f","version":"0'0","reported_seq":13,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-09-16T15:49:54.749395+0000","last_change":"2024-09-16T15:49:53.772719+0000","last_active":"2024-09-16T15:49:54.749395+0000","last_peered":"2024-09-16T15:49:54.749395+0000","last_clean":"2024-09-16T15:49:54.749395+0000","last_became_active":"2024-09-16T15:49:53.766969+0000","last_became_peered":"2024-09-16T15:49:53.766969+0000","last_unstale":"2024-09-16T15:49:54.749395+0000","last_undegraded":"2024-09-16T15:49:54.749395+0000","last_fullsized":"2024-09-16T15:49:54.749395+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:49:52.725252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:49:52.725252+0000","last_clean_scrub_stamp":"2024-09-16T15:49:52.725252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T03:06:22.126597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.1e","version":"134'1","reported_seq":16,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-09-16T15:49:54.749159+0000","last_change":"2024-09-16T15:49:52.777280+0000","last_active":"2024-09-16T15:49:54.749159+0000","last_peered":"2024-09-16T15:49:54.749159+0000","last_clean":"2024-09-16T15:49:54.749159+0000","last_became_active":"2024-09-16T15:49:52.776932+0000","last_became_peered":"2024-09-16T15:49:52.776932+0000","last_unstale":"2024-09-16T15:49:54.749159+0000","last_undegraded":"2024-09-16T15:49:54.749159+0000","last_fullsized":"2024-09-16T15:49:54.749159+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:49:51.721471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:49:51.721471+0000","last_clean_scrub_stamp":"2024-09-16T15:49:51.721471+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T03:18:54.303040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.1c","version":"0'0","reported_seq":13,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-09-16T15:49:55.015542+0000","last_change":"2024-09-16T15:49:53.760772+0000","last_active":"2024-09-16T15:49:55.015542+0000","last_peered":"2024-09-16T15:49:55.015542+0000","last_clean":"2024-09-16T15:49:55.015542+0000","last_became_active":"2024-09-16T15:49:53.760233+0000","last_became_peered":"2024-09-16T15:49:53.760233+0000","last_unstale":"2024-09-16T15:49:55.015542+0000","last_undegraded":"2024-09-16T15:49:55.015542+0000","last_fullsized":"2024-09-16T15:49:55.015542+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:49:52.725252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:49:52.725252+0000","last_clean_scrub_stamp":"2024-09-16T15:49:52.725252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T00:01:18.459741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.1d","version":"134'2","reported_seq":17,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-09-16T15:49:54.750973+0000","last_change":"2024-09-16T15:49:52.759269+0000","last_active":"2024-09-16T15:49:54.750973+0000","last_peered":"2024-09-16T15:49:54.750973+0000","last_clean":"2024-09-16T15:49:54.750973+0000","last_became_active":"2024-09-16T15:49:52.758529+0000","last_became_peered":"2024-09-16T15:49:52.758529+0000","last_unstale":"2024-09-16T15:49:54.750973+0000","last_undegraded":"2024-09-16T15:49:54.750973+0000","last_fullsized":"2024-09-16T15:49:54.750973+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:49:51.721471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:49:51.721471+0000","last_clean_scrub_stamp":"2024-09-16T15:49:51.721471+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:34:39.636337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.1d","version":"0'0","reported_seq":13,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-09-16T15:49:55.015207+0000","last_change":"2024-09-16T15:49:53.801699+0000","last_active":"2024-09-16T15:49:55.015207+0000","last_peered":"2024-09-16T15:49:55.015207+0000","last_clean":"2024-09-16T15:49:55.015207+0000","last_became_active":"2024-09-16T15:49:53.801058+0000","last_became_peered":"2024-09-16T15:49:53.801058+0000","last_unstale":"2024-09-16T15:49:55.015207+0000","last_undegraded":"2024-09-16T15:49:55.015207+0000","last_fullsized":"2024-09-16T15:49:55.015207+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:49:52.725252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:49:52.725252+0000","last_clean_scrub_stamp":"2024-09-16T15:49:52.725252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T22:37:54.641296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.1c","version":"134'2","reported_seq":17,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-09-16T15:49:54.752366+0000","last_change":"2024-09-16T15:49:52.763863+0000","last_active":"2024-09-16T15:49:54.752366+0000","last_peered":"2024-09-16T15:49:54.752366+0000","last_clean":"2024-09-16T15:49:54.752366+0000","last_became_active":"2024-09-16T15:49:52.762989+0000","last_became_peered":"2024-09-16T15:49:52.762989+0000","last_unstale":"2024-09-16T15:49:54.752366+0000","last_undegraded":"2024-09-16T15:49:54.752366+0000","last_fullsized":"2024-09-16T15:49:54.752366+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:49:51.721471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:49:51.721471+0000","last_clean_scrub_stamp":"2024-09-16T15:49:51.721471+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:39:55.480968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.1a","version":"0'0","reported_seq":13,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-09-16T15:49:54.748602+0000","last_change":"2024-09-16T15:49:53.815933+0000","last_active":"2024-09-16T15:49:54.748602+0000","last_peered":"2024-09-16T15:49:54.748602+0000","last_clean":"2024-09-16T15:49:54.748602+0000","last_became_active":"2024-09-16T15:49:53.815649+0000","last_became_peered":"2024-09-16T15:49:53.815649+0000","last_unstale":"2024-09-16T15:49:54.748602+0000","last_undegraded":"2024-09-16T15:49:54.748602+0000","last_fullsized":"2024-09-16T15:49:54.748602+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:49:52.725252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:49:52.725252+0000","last_clean_scrub_stamp":"2024-09-16T15:49:52.725252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T00:02:37.101218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.1b","version":"134'1","reported_seq":16,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-09-16T15:49:55.016063+0000","last_change":"2024-09-16T15:49:52.774467+0000","last_active":"2024-09-16T15:49:55.016063+0000","last_peered":"2024-09-16T15:49:55.016063+0000","last_clean":"2024-09-16T15:49:55.016063+0000","last_became_active":"2024-09-16T15:49:52.774038+0000","last_became_peered":"2024-09-16T15:49:52.774038+0000","last_unstale":"2024-09-16T15:49:55.016063+0000","last_undegraded":"2024-09-16T15:49:55.016063+0000","last_fullsized":"2024-09-16T15:49:55.016063+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:49:51.721471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:49:51.721471+0000","last_clean_scrub_stamp":"2024-09-16T15:49:51.721471+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T18:49:34.407287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.1b","version":"0'0","reported_seq":13,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-09-16T15:49:54.750414+0000","last_change":"2024-09-16T15:49:53.806188+0000","last_active":"2024-09-16T15:49:54.750414+0000","last_peered":"2024-09-16T15:49:54.750414+0000","last_clean":"2024-09-16T15:49:54.750414+0000","last_became_active":"2024-09-16T15:49:53.805684+0000","last_became_peered":"2024-09-16T15:49:53.805684+0000","last_unstale":"2024-09-16T15:49:54.750414+0000","last_undegraded":"2024-09-16T15:49:54.750414+0000","last_fullsized":"2024-09-16T15:49:54.750414+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:49:52.725252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:49:52.725252+0000","last_clean_scrub_stamp":"2024-09-16T15:49:52.725252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:17:49.186332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"20.1a","version":"134'1","reported_seq":16,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-09-16T15:49:55.013465+0000","last_change":"2024-09-16T15:49:52.781580+0000","last_active":"2024-09-16T15:49:55.013465+0000","last_peered":"2024-09-16T15:49:55.013465+0000","last_clean":"2024-09-16T15:49:55.013465+0000","last_became_active":"2024-09-16T15:49:52.780852+0000","last_became_peered":"2024-09-16T15:49:52.780852+0000","last_unstale":"2024-09-16T15:49:55.013465+0000","last_undegraded":"2024-09-16T15:49:55.013465+0000","last_fullsized":"2024-09-16T15:49:55.013465+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:49:51.721471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:49:51.721471+0000","last_clean_scrub_stamp":"2024-09-16T15:49:51.721471+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:28:10.260758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"21.18","version":"0'0","reported_seq":13,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-09-16T15:49:54.748388+0000","last_change":"2024-09-16T15:49:53.794339+0000","last_active":"2024-09-16T15:49:54.748388+0000","last_peered":"2024-09-16T15:49:54.748388+0000","last_clean":"2024-09-16T15:49:54.748388+0000","last_became_active":"2024-09-16T15:49:53.794054+0000","last_became_peered":"2024-09-16T15:49:53.794054+0000","last_unstale":"2024-09-16T15:49:54.748388+0000","last_undegraded":"2024-09-16T15:49:54.748388+0000","last_fullsized":"2024-09-16T15:49:54.748388+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:49:52.725252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:49:52.725252+0000","last_clean_scrub_stamp":"2024-09-16T15:49:52.725252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:38:50.572143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.19","version":"0'0","reported_seq":15,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-09-16T15:49:55.015756+0000","last_change":"2024-09-16T15:49:52.781835+0000","last_active":"2024-09-16T15:49:55.015756+0000","last_peered":"2024-09-16T15:49:55.015756+0000","last_clean":"2024-09-16T15:49:55.015756+0000","last_became_active":"2024-09-16T15:49:52.781449+0000","last_became_peered":"2024-09-16T15:49:52.781449+0000","last_unstale":"2024-09-16T15:49:55.015756+0000","last_undegraded":"2024-09-16T15:49:55.015756+0000","last_fullsized":"2024-09-16T15:49:55.015756+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:49:51.721471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:49:51.721471+0000","last_clean_scrub_stamp":"2024-09-16T15:49:51.721471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T23:24:41.912346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.19","version":"0'0","reported_seq":13,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-09-16T15:49:55.013260+0000","last_change":"2024-09-16T15:49:53.784500+0000","last_active":"2024-09-16T15:49:55.013260+0000","last_peered":"2024-09-16T15:49:55.013260+0000","last_clean":"2024-09-16T15:49:55.013260+0000","last_became_active":"2024-09-16T15:49:53.782672+0000","last_became_peered":"2024-09-16T15:49:53.782672+0000","last_unstale":"2024-09-16T15:49:55.013260+0000","last_undegraded":"2024-09-16T15:49:55.013260+0000","last_fullsized":"2024-09-16T15:49:55.013260+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:49:52.725252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:49:52.725252+0000","last_clean_scrub_stamp":"2024-09-16T15:49:52.725252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T01:11:40.211819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.18","version":"0'0","reported_seq":15,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-09-16T15:49:54.750020+0000","last_change":"2024-09-16T15:49:52.778794+0000","last_active":"2024-09-16T15:49:54.750020+0000","last_peered":"2024-09-16T15:49:54.750020+0000","last_clean":"2024-09-16T15:49:54.750020+0000","last_became_active":"2024-09-16T15:49:52.778051+0000","last_became_peered":"2024-09-16T15:49:52.778051+0000","last_unstale":"2024-09-16T15:49:54.750020+0000","last_undegraded":"2024-09-16T15:49:54.750020+0000","last_fullsized":"2024-09-16T15:49:54.750020+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:49:51.721471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:49:51.721471+0000","last_clean_scrub_stamp":"2024-09-16T15:49:51.721471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T02:15:43.635025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.16","version":"0'0","reported_seq":13,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-09-16T15:49:55.014022+0000","last_change":"2024-09-16T15:49:53.785294+0000","last_active":"2024-09-16T15:49:55.014022+0000","last_peered":"2024-09-16T15:49:55.014022+0000","last_clean":"2024-09-16T15:49:55.014022+0000","last_became_active":"2024-09-16T15:49:53.784515+0000","last_became_peered":"2024-09-16T15:49:53.784515+0000","last_unstale":"2024-09-16T15:49:55.014022+0000","last_undegraded":"2024-09-16T15:49:55.014022+0000","last_fullsized":"2024-09-16T15:49:55.014022+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:49:52.725252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:49:52.725252+0000","last_clean_scrub_stamp":"2024-09-16T15:49:52.725252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:15:27.170635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.17","version":"0'0","reported_seq":15,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-09-16T15:49:54.749258+0000","last_change":"2024-09-16T15:49:52.769159+0000","last_active":"2024-09-16T15:49:54.749258+0000","last_peered":"2024-09-16T15:49:54.749258+0000","last_clean":"2024-09-16T15:49:54.749258+0000","last_became_active":"2024-09-16T15:49:52.768366+0000","last_became_peered":"2024-09-16T15:49:52.768366+0000","last_unstale":"2024-09-16T15:49:54.749258+0000","last_undegraded":"2024-09-16T15:49:54.749258+0000","last_fullsized":"2024-09-16T15:49:54.749258+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:49:51.721471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:49:51.721471+0000","last_clean_scrub_stamp":"2024-09-16T15:49:51.721471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:36:59.572984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.17","version":"0'0","reported_seq":13,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-09-16T15:49:55.015378+0000","last_change":"2024-09-16T15:49:53.798297+0000","last_active":"2024-09-16T15:49:55.015378+0000","last_peered":"2024-09-16T15:49:55.015378+0000","last_clean":"2024-09-16T15:49:55.015378+0000","last_became_active":"2024-09-16T15:49:53.796876+0000","last_became_peered":"2024-09-16T15:49:53.796876+0000","last_unstale":"2024-09-16T15:49:55.015378+0000","last_undegraded":"2024-09-16T15:49:55.015378+0000","last_fullsized":"2024-09-16T15:49:55.015378+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:49:52.725252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:49:52.725252+0000","last_clean_scrub_stamp":"2024-09-16T15:49:52.725252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:59:47.880031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.16","version":"0'0","reported_seq":15,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-09-16T15:49:55.016139+0000","last_change":"2024-09-16T15:49:52.782354+0000","last_active":"2024-09-16T15:49:55.016139+0000","last_peered":"2024-09-16T15:49:55.016139+0000","last_clean":"2024-09-16T15:49:55.016139+0000","last_became_active":"2024-09-16T15:49:52.781604+0000","last_became_peered":"2024-09-16T15:49:52.781604+0000","last_unstale":"2024-09-16T15:49:55.016139+0000","last_undegraded":"2024-09-16T15:49:55.016139+0000","last_fullsized":"2024-09-16T15:49:55.016139+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:49:51.721471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:49:51.721471+0000","last_clean_scrub_stamp":"2024-09-16T15:49:51.721471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T23:50:37.482485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.15","version":"0'0","reported_seq":13,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-09-16T15:49:55.017525+0000","last_change":"2024-09-16T15:49:53.761206+0000","last_active":"2024-09-16T15:49:55.017525+0000","last_peered":"2024-09-16T15:49:55.017525+0000","last_clean":"2024-09-16T15:49:55.017525+0000","last_became_active":"2024-09-16T15:49:53.760908+0000","last_became_peered":"2024-09-16T15:49:53.760908+0000","last_unstale":"2024-09-16T15:49:55.017525+0000","last_undegraded":"2024-09-16T15:49:55.017525+0000","last_fullsized":"2024-09-16T15:49:55.017525+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:49:52.725252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:49:52.725252+0000","last_clean_scrub_stamp":"2024-09-16T15:49:52.725252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T22:54:53.849709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.14","version":"134'6","reported_seq":21,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-09-16T15:49:55.018014+0000","last_change":"2024-09-16T15:49:52.762448+0000","last_active":"2024-09-16T15:49:55.018014+0000","last_peered":"2024-09-16T15:49:55.018014+0000","last_clean":"2024-09-16T15:49:55.018014+0000","last_became_active":"2024-09-16T15:49:52.761695+0000","last_became_peered":"2024-09-16T15:49:52.761695+0000","last_unstale":"2024-09-16T15:49:55.018014+0000","last_undegraded":"2024-09-16T15:49:55.018014+0000","last_fullsized":"2024-09-16T15:49:55.018014+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:49:51.721471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:49:51.721471+0000","last_clean_scrub_stamp":"2024-09-16T15:49:51.721471+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T00:28:15.684923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"21.12","version":"0'0","reported_seq":13,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-09-16T15:49:54.750130+0000","last_change":"2024-09-16T15:49:53.806120+0000","last_active":"2024-09-16T15:49:54.750130+0000","last_peered":"2024-09-16T15:49:54.750130+0000","last_clean":"2024-09-16T15:49:54.750130+0000","last_became_active":"2024-09-16T15:49:53.805381+0000","last_became_peered":"2024-09-16T15:49:53.805381+0000","last_unstale":"2024-09-16T15:49:54.750130+0000","last_undegraded":"2024-09-16T15:49:54.750130+0000","last_fullsized":"2024-09-16T15:49:54.750130+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:49:52.725252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:49:52.725252+0000","last_clean_scrub_stamp":"2024-09-16T15:49:52.725252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:11:15.664173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.13","version":"134'2","reported_seq":17,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-09-16T15:49:54.752944+0000","last_change":"2024-09-16T15:49:52.774793+0000","last_active":"2024-09-16T15:49:54.752944+0000","last_peered":"2024-09-16T15:49:54.752944+0000","last_clean":"2024-09-16T15:49:54.752944+0000","last_became_active":"2024-09-16T15:49:52.774116+0000","last_became_peered":"2024-09-16T15:49:52.774116+0000","last_unstale":"2024-09-16T15:49:54.752944+0000","last_undegraded":"2024-09-16T15:49:54.752944+0000","last_fullsized":"2024-09-16T15:49:54.752944+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:49:51.721471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:49:51.721471+0000","last_clean_scrub_stamp":"2024-09-16T15:49:51.721471+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T23:49:56.154646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"21.13","version":"0'0","reported_seq":13,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-09-16T15:49:54.748633+0000","last_change":"2024-09-16T15:49:53.772319+0000","last_active":"2024-09-16T15:49:54.748633+0000","last_peered":"2024-09-16T15:49:54.748633+0000","last_clean":"2024-09-16T15:49:54.748633+0000","last_became_active":"2024-09-16T15:49:53.767119+0000","last_became_peered":"2024-09-16T15:49:53.767119+0000","last_unstale":"2024-09-16T15:49:54.748633+0000","last_undegraded":"2024-09-16T15:49:54.748633+0000","last_fullsized":"2024-09-16T15:49:54.748633+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:49:52.725252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:49:52.725252+0000","last_clean_scrub_stamp":"2024-09-16T15:49:52.725252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T22:28:29.014751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.12","version":"0'0","reported_seq":15,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-09-16T15:49:54.750536+0000","last_change":"2024-09-16T15:49:52.785819+0000","last_active":"2024-09-16T15:49:54.750536+0000","last_peered":"2024-09-16T15:49:54.750536+0000","last_clean":"2024-09-16T15:49:54.750536+0000","last_became_active":"2024-09-16T15:49:52.785063+0000","last_became_peered":"2024-09-16T15:49:52.785063+0000","last_unstale":"2024-09-16T15:49:54.750536+0000","last_undegraded":"2024-09-16T15:49:54.750536+0000","last_fullsized":"2024-09-16T15:49:54.750536+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:49:51.721471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:49:51.721471+0000","last_clean_scrub_stamp":"2024-09-16T15:49:51.721471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T02:37:28.590611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"21.10","version":"0'0","reported_seq":13,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-09-16T15:49:55.013330+0000","last_change":"2024-09-16T15:49:53.795683+0000","last_active":"2024-09-16T15:49:55.013330+0000","last_peered":"2024-09-16T15:49:55.013330+0000","last_clean":"2024-09-16T15:49:55.013330+0000","last_became_active":"2024-09-16T15:49:53.791581+0000","last_became_peered":"2024-09-16T15:49:53.791581+0000","last_unstale":"2024-09-16T15:49:55.013330+0000","last_undegraded":"2024-09-16T15:49:55.013330+0000","last_fullsized":"2024-09-16T15:49:55.013330+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:49:52.725252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:49:52.725252+0000","last_clean_scrub_stamp":"2024-09-16T15:49:52.725252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:43:35.941656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.11","version":"0'0","reported_seq":15,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-09-16T15:49:55.015152+0000","last_change":"2024-09-16T15:49:52.757579+0000","last_active":"2024-09-16T15:49:55.015152+0000","last_peered":"2024-09-16T15:49:55.015152+0000","last_clean":"2024-09-16T15:49:55.015152+0000","last_became_active":"2024-09-16T15:49:52.757085+0000","last_became_peered":"2024-09-16T15:49:52.757085+0000","last_unstale":"2024-09-16T15:49:55.015152+0000","last_undegraded":"2024-09-16T15:49:55.015152+0000","last_fullsized":"2024-09-16T15:49:55.015152+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:49:51.721471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:49:51.721471+0000","last_clean_scrub_stamp":"2024-09-16T15:49:51.721471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T00:53:44.706141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.11","version":"0'0","reported_seq":13,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-09-16T15:49:54.751549+0000","last_change":"2024-09-16T15:49:53.807125+0000","last_active":"2024-09-16T15:49:54.751549+0000","last_peered":"2024-09-16T15:49:54.751549+0000","last_clean":"2024-09-16T15:49:54.751549+0000","last_became_active":"2024-09-16T15:49:53.805178+0000","last_became_peered":"2024-09-16T15:49:53.805178+0000","last_unstale":"2024-09-16T15:49:54.751549+0000","last_undegraded":"2024-09-16T15:49:54.751549+0000","last_fullsized":"2024-09-16T15:49:54.751549+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:49:52.725252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:49:52.725252+0000","last_clean_scrub_stamp":"2024-09-16T15:49:52.725252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:53:44.753622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.10","version":"134'8","reported_seq":23,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-09-16T15:49:55.016256+0000","last_change":"2024-09-16T15:49:52.766519+0000","last_active":"2024-09-16T15:49:55.016256+0000","last_peered":"2024-09-16T15:49:55.016256+0000","last_clean":"2024-09-16T15:49:55.016256+0000","last_became_active":"2024-09-16T15:49:52.766156+0000","last_became_peered":"2024-09-16T15:49:52.766156+0000","last_unstale":"2024-09-16T15:49:55.016256+0000","last_undegraded":"2024-09-16T15:49:55.016256+0000","last_fullsized":"2024-09-16T15:49:55.016256+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:49:51.721471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:49:51.721471+0000","last_clean_scrub_stamp":"2024-09-16T15:49:51.721471+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T22:50:26.176905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.e","version":"0'0","reported_seq":13,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-09-16T15:49:55.016284+0000","last_change":"2024-09-16T15:49:53.795034+0000","last_active":"2024-09-16T15:49:55.016284+0000","last_peered":"2024-09-16T15:49:55.016284+0000","last_clean":"2024-09-16T15:49:55.016284+0000","last_became_active":"2024-09-16T15:49:53.794234+0000","last_became_peered":"2024-09-16T15:49:53.794234+0000","last_unstale":"2024-09-16T15:49:55.016284+0000","last_undegraded":"2024-09-16T15:49:55.016284+0000","last_fullsized":"2024-09-16T15:49:55.016284+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:49:52.725252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:49:52.725252+0000","last_clean_scrub_stamp":"2024-09-16T15:49:52.725252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:52:42.730798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.f","version":"134'1","reported_seq":16,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-09-16T15:49:55.013653+0000","last_change":"2024-09-16T15:49:52.784454+0000","last_active":"2024-09-16T15:49:55.013653+0000","last_peered":"2024-09-16T15:49:55.013653+0000","last_clean":"2024-09-16T15:49:55.013653+0000","last_became_active":"2024-09-16T15:49:52.783843+0000","last_became_peered":"2024-09-16T15:49:52.783843+0000","last_unstale":"2024-09-16T15:49:55.013653+0000","last_undegraded":"2024-09-16T15:49:55.013653+0000","last_fullsized":"2024-09-16T15:49:55.013653+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:49:51.721471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:49:51.721471+0000","last_clean_scrub_stamp":"2024-09-16T15:49:51.721471+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:22:04.602157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"21.f","version":"0'0","reported_seq":13,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-09-16T15:49:55.017447+0000","last_change":"2024-09-16T15:49:53.782494+0000","last_active":"2024-09-16T15:49:55.017447+0000","last_peered":"2024-09-16T15:49:55.017447+0000","last_clean":"2024-09-16T15:49:55.017447+0000","last_became_active":"2024-09-16T15:49:53.782211+0000","last_became_peered":"2024-09-16T15:49:53.782211+0000","last_unstale":"2024-09-16T15:49:55.017447+0000","last_undegraded":"2024-09-16T15:49:55.017447+0000","last_fullsized":"2024-09-16T15:49:55.017447+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:49:52.725252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:49:52.725252+0000","last_clean_scrub_stamp":"2024-09-16T15:49:52.725252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T18:27:22.512705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.e","version":"134'2","reported_seq":17,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-09-16T15:49:54.749430+0000","last_change":"2024-09-16T15:49:52.774535+0000","last_active":"2024-09-16T15:49:54.749430+0000","last_peered":"2024-09-16T15:49:54.749430+0000","last_clean":"2024-09-16T15:49:54.749430+0000","last_became_active":"2024-09-16T15:49:52.774230+0000","last_became_peered":"2024-09-16T15:49:52.774230+0000","last_unstale":"2024-09-16T15:49:54.749430+0000","last_undegraded":"2024-09-16T15:49:54.749430+0000","last_fullsized":"2024-09-16T15:49:54.749430+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:49:51.721471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:49:51.721471+0000","last_clean_scrub_stamp":"2024-09-16T15:49:51.721471+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:06:41.060737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.c","version":"0'0","reported_seq":13,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-09-16T15:49:54.752223+0000","last_change":"2024-09-16T15:49:53.805839+0000","last_active":"2024-09-16T15:49:54.752223+0000","last_peered":"2024-09-16T15:49:54.752223+0000","last_clean":"2024-09-16T15:49:54.752223+0000","last_became_active":"2024-09-16T15:49:53.805178+0000","last_became_peered":"2024-09-16T15:49:53.805178+0000","last_unstale":"2024-09-16T15:49:54.752223+0000","last_undegraded":"2024-09-16T15:49:54.752223+0000","last_fullsized":"2024-09-16T15:49:54.752223+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:49:52.725252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:49:52.725252+0000","last_clean_scrub_stamp":"2024-09-16T15:49:52.725252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:51:57.391541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.d","version":"134'1","reported_seq":16,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-09-16T15:49:55.017991+0000","last_change":"2024-09-16T15:49:52.773668+0000","last_active":"2024-09-16T15:49:55.017991+0000","last_peered":"2024-09-16T15:49:55.017991+0000","last_clean":"2024-09-16T15:49:55.017991+0000","last_became_active":"2024-09-16T15:49:52.773184+0000","last_became_peered":"2024-09-16T15:49:52.773184+0000","last_unstale":"2024-09-16T15:49:55.017991+0000","last_undegraded":"2024-09-16T15:49:55.017991+0000","last_fullsized":"2024-09-16T15:49:55.017991+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:49:51.721471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:49:51.721471+0000","last_clean_scrub_stamp":"2024-09-16T15:49:51.721471+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:30:28.622437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.d","version":"0'0","reported_seq":13,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-09-16T15:49:55.016400+0000","last_change":"2024-09-16T15:49:53.809332+0000","last_active":"2024-09-16T15:49:55.016400+0000","last_peered":"2024-09-16T15:49:55.016400+0000","last_clean":"2024-09-16T15:49:55.016400+0000","last_became_active":"2024-09-16T15:49:53.809054+0000","last_became_peered":"2024-09-16T15:49:53.809054+0000","last_unstale":"2024-09-16T15:49:55.016400+0000","last_undegraded":"2024-09-16T15:49:55.016400+0000","last_fullsized":"2024-09-16T15:49:55.016400+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:49:52.725252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:49:52.725252+0000","last_clean_scrub_stamp":"2024-09-16T15:49:52.725252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T01:21:55.832298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.c","version":"0'0","reported_seq":15,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-09-16T15:49:54.748349+0000","last_change":"2024-09-16T15:49:52.768132+0000","last_active":"2024-09-16T15:49:54.748349+0000","last_peered":"2024-09-16T15:49:54.748349+0000","last_clean":"2024-09-16T15:49:54.748349+0000","last_became_active":"2024-09-16T15:49:52.767738+0000","last_became_peered":"2024-09-16T15:49:52.767738+0000","last_unstale":"2024-09-16T15:49:54.748349+0000","last_undegraded":"2024-09-16T15:49:54.748349+0000","last_fullsized":"2024-09-16T15:49:54.748349+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:49:51.721471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:49:51.721471+0000","last_clean_scrub_stamp":"2024-09-16T15:49:51.721471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T03:32:13.600777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.a","version":"0'0","reported_seq":13,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-09-16T15:49:54.751189+0000","last_change":"2024-09-16T15:49:53.757825+0000","last_active":"2024-09-16T15:49:54.751189+0000","last_peered":"2024-09-16T15:49:54.751189+0000","last_clean":"2024-09-16T15:49:54.751189+0000","last_became_active":"2024-09-16T15:49:53.756978+0000","last_became_peered":"2024-09-16T15:49:53.756978+0000","last_unstale":"2024-09-16T15:49:54.751189+0000","last_undegraded":"2024-09-16T15:49:54.751189+0000","last_fullsized":"2024-09-16T15:49:54.751189+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:49:52.725252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:49:52.725252+0000","last_clean_scrub_stamp":"2024-09-16T15:49:52.725252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T22:28:29.014751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"20.b","version":"134'2","reported_seq":17,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-09-16T15:49:55.017698+0000","last_change":"2024-09-16T15:49:52.770565+0000","last_active":"2024-09-16T15:49:55.017698+0000","last_peered":"2024-09-16T15:49:55.017698+0000","last_clean":"2024-09-16T15:49:55.017698+0000","last_became_active":"2024-09-16T15:49:52.770244+0000","last_became_peered":"2024-09-16T15:49:52.770244+0000","last_unstale":"2024-09-16T15:49:55.017698+0000","last_undegraded":"2024-09-16T15:49:55.017698+0000","last_fullsized":"2024-09-16T15:49:55.017698+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:49:51.721471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:49:51.721471+0000","last_clean_scrub_stamp":"2024-09-16T15:49:51.721471+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T22:38:55.209721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"21.b","version":"0'0","reported_seq":13,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-09-16T15:49:55.017231+0000","last_change":"2024-09-16T15:49:53.777929+0000","last_active":"2024-09-16T15:49:55.017231+0000","last_peered":"2024-09-16T15:49:55.017231+0000","last_clean":"2024-09-16T15:49:55.017231+0000","last_became_active":"2024-09-16T15:49:53.777689+0000","last_became_peered":"2024-09-16T15:49:53.777689+0000","last_unstale":"2024-09-16T15:49:55.017231+0000","last_undegraded":"2024-09-16T15:49:55.017231+0000","last_fullsized":"2024-09-16T15:49:55.017231+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:49:52.725252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:49:52.725252+0000","last_clean_scrub_stamp":"2024-09-16T15:49:52.725252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:40:39.342110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"20.a","version":"0'0","reported_seq":15,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-09-16T15:49:55.015205+0000","last_change":"2024-09-16T15:49:52.771440+0000","last_active":"2024-09-16T15:49:55.015205+0000","last_peered":"2024-09-16T15:49:55.015205+0000","last_clean":"2024-09-16T15:49:55.015205+0000","last_became_active":"2024-09-16T15:49:52.770609+0000","last_became_peered":"2024-09-16T15:49:52.770609+0000","last_unstale":"2024-09-16T15:49:55.015205+0000","last_undegraded":"2024-09-16T15:49:55.015205+0000","last_fullsized":"2024-09-16T15:49:55.015205+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:49:51.721471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:49:51.721471+0000","last_clean_scrub_stamp":"2024-09-16T15:49:51.721471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T22:28:38.317070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.8","version":"0'0","reported_seq":13,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-09-16T15:49:54.749697+0000","last_change":"2024-09-16T15:49:53.763650+0000","last_active":"2024-09-16T15:49:54.749697+0000","last_peered":"2024-09-16T15:49:54.749697+0000","last_clean":"2024-09-16T15:49:54.749697+0000","last_became_active":"2024-09-16T15:49:53.762879+0000","last_became_peered":"2024-09-16T15:49:53.762879+0000","last_unstale":"2024-09-16T15:49:54.749697+0000","last_undegraded":"2024-09-16T15:49:54.749697+0000","last_fullsized":"2024-09-16T15:49:54.749697+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:49:52.725252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:49:52.725252+0000","last_clean_scrub_stamp":"2024-09-16T15:49:52.725252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:00:52.258152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"20.9","version":"0'0","reported_seq":15,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-09-16T15:49:55.014517+0000","last_change":"2024-09-16T15:49:52.771244+0000","last_active":"2024-09-16T15:49:55.014517+0000","last_peered":"2024-09-16T15:49:55.014517+0000","last_clean":"2024-09-16T15:49:55.014517+0000","last_became_active":"2024-09-16T15:49:52.770510+0000","last_became_peered":"2024-09-16T15:49:52.770510+0000","last_unstale":"2024-09-16T15:49:55.014517+0000","last_undegraded":"2024-09-16T15:49:55.014517+0000","last_fullsized":"2024-09-16T15:49:55.014517+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:49:51.721471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:49:51.721471+0000","last_clean_scrub_stamp":"2024-09-16T15:49:51.721471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:59:48.118024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.9","version":"0'0","reported_seq":13,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-09-16T15:49:55.015523+0000","last_change":"2024-09-16T15:49:53.774586+0000","last_active":"2024-09-16T15:49:55.015523+0000","last_peered":"2024-09-16T15:49:55.015523+0000","last_clean":"2024-09-16T15:49:55.015523+0000","last_became_active":"2024-09-16T15:49:53.774211+0000","last_became_peered":"2024-09-16T15:49:53.774211+0000","last_unstale":"2024-09-16T15:49:55.015523+0000","last_undegraded":"2024-09-16T15:49:55.015523+0000","last_fullsized":"2024-09-16T15:49:55.015523+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:49:52.725252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:49:52.725252+0000","last_clean_scrub_stamp":"2024-09-16T15:49:52.725252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T03:33:15.612475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"20.8","version":"0'0","reported_seq":15,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-09-16T15:49:54.749002+0000","last_change":"2024-09-16T15:49:52.777568+0000","last_active":"2024-09-16T15:49:54.749002+0000","last_peered":"2024-09-16T15:49:54.749002+0000","last_clean":"2024-09-16T15:49:54.749002+0000","last_became_active":"2024-09-16T15:49:52.776903+0000","last_became_peered":"2024-09-16T15:49:52.776903+0000","last_unstale":"2024-09-16T15:49:54.749002+0000","last_undegraded":"2024-09-16T15:49:54.749002+0000","last_fullsized":"2024-09-16T15:49:54.749002+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:49:51.721471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:49:51.721471+0000","last_clean_scrub_stamp":"2024-09-16T15:49:51.721471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T23:11:07.659450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"21.6","version":"0'0","reported_seq":13,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-09-16T15:49:55.014578+0000","last_change":"2024-09-16T15:49:53.804004+0000","last_active":"2024-09-16T15:49:55.014578+0000","last_peered":"2024-09-16T15:49:55.014578+0000","last_clean":"2024-09-16T15:49:55.014578+0000","last_became_active":"2024-09-16T15:49:53.803397+0000","last_became_peered":"2024-09-16T15:49:53.803397+0000","last_unstale":"2024-09-16T15:49:55.014578+0000","last_undegraded":"2024-09-16T15:49:55.014578+0000","last_fullsized":"2024-09-16T15:49:55.014578+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:49:52.725252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:49:52.725252+0000","last_clean_scrub_stamp":"2024-09-16T15:49:52.725252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T18:28:28.606171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.7","version":"134'2","reported_seq":17,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-09-16T15:49:55.015849+0000","last_change":"2024-09-16T15:49:52.775976+0000","last_active":"2024-09-16T15:49:55.015849+0000","last_peered":"2024-09-16T15:49:55.015849+0000","last_clean":"2024-09-16T15:49:55.015849+0000","last_became_active":"2024-09-16T15:49:52.775249+0000","last_became_peered":"2024-09-16T15:49:52.775249+0000","last_unstale":"2024-09-16T15:49:55.015849+0000","last_undegraded":"2024-09-16T15:49:55.015849+0000","last_fullsized":"2024-09-16T15:49:55.015849+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:49:51.721471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:49:51.721471+0000","last_clean_scrub_stamp":"2024-09-16T15:49:51.721471+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:57:08.808070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.7","version":"0'0","reported_seq":13,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-09-16T15:49:54.753391+0000","last_change":"2024-09-16T15:49:53.806912+0000","last_active":"2024-09-16T15:49:54.753391+0000","last_peered":"2024-09-16T15:49:54.753391+0000","last_clean":"2024-09-16T15:49:54.753391+0000","last_became_active":"2024-09-16T15:49:53.805374+0000","last_became_peered":"2024-09-16T15:49:53.805374+0000","last_unstale":"2024-09-16T15:49:54.753391+0000","last_undegraded":"2024-09-16T15:49:54.753391+0000","last_fullsized":"2024-09-16T15:49:54.753391+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:49:52.725252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:49:52.725252+0000","last_clean_scrub_stamp":"2024-09-16T15:49:52.725252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T00:21:39.029022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.6","version":"0'0","reported_seq":15,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-09-16T15:49:55.016280+0000","last_change":"2024-09-16T15:49:52.774453+0000","last_active":"2024-09-16T15:49:55.016280+0000","last_peered":"2024-09-16T15:49:55.016280+0000","last_clean":"2024-09-16T15:49:55.016280+0000","last_became_active":"2024-09-16T15:49:52.773886+0000","last_became_peered":"2024-09-16T15:49:52.773886+0000","last_unstale":"2024-09-16T15:49:55.016280+0000","last_undegraded":"2024-09-16T15:49:55.016280+0000","last_fullsized":"2024-09-16T15:49:55.016280+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:49:51.721471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:49:51.721471+0000","last_clean_scrub_stamp":"2024-09-16T15:49:51.721471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:16:42.423423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.4","version":"0'0","reported_seq":13,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-09-16T15:49:54.748622+0000","last_change":"2024-09-16T15:49:53.783444+0000","last_active":"2024-09-16T15:49:54.748622+0000","last_peered":"2024-09-16T15:49:54.748622+0000","last_clean":"2024-09-16T15:49:54.748622+0000","last_became_active":"2024-09-16T15:49:53.782851+0000","last_became_peered":"2024-09-16T15:49:53.782851+0000","last_unstale":"2024-09-16T15:49:54.748622+0000","last_undegraded":"2024-09-16T15:49:54.748622+0000","last_fullsized":"2024-09-16T15:49:54.748622+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:49:52.725252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:49:52.725252+0000","last_clean_scrub_stamp":"2024-09-16T15:49:52.725252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T00:21:32.615337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.5","version":"0'0","reported_seq":15,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-09-16T15:49:55.017773+0000","last_change":"2024-09-16T15:49:52.757797+0000","last_active":"2024-09-16T15:49:55.017773+0000","last_peered":"2024-09-16T15:49:55.017773+0000","last_clean":"2024-09-16T15:49:55.017773+0000","last_became_active":"2024-09-16T15:49:52.757406+0000","last_became_peered":"2024-09-16T15:49:52.757406+0000","last_unstale":"2024-09-16T15:49:55.017773+0000","last_undegraded":"2024-09-16T15:49:55.017773+0000","last_fullsized":"2024-09-16T15:49:55.017773+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:49:51.721471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:49:51.721471+0000","last_clean_scrub_stamp":"2024-09-16T15:49:51.721471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T18:55:22.565226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.5","version":"0'0","reported_seq":13,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-09-16T15:49:55.016967+0000","last_change":"2024-09-16T15:49:53.760718+0000","last_active":"2024-09-16T15:49:55.016967+0000","last_peered":"2024-09-16T15:49:55.016967+0000","last_clean":"2024-09-16T15:49:55.016967+0000","last_became_active":"2024-09-16T15:49:53.760349+0000","last_became_peered":"2024-09-16T15:49:53.760349+0000","last_unstale":"2024-09-16T15:49:55.016967+0000","last_undegraded":"2024-09-16T15:49:55.016967+0000","last_fullsized":"2024-09-16T15:49:55.016967+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:49:52.725252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:49:52.725252+0000","last_clean_scrub_stamp":"2024-09-16T15:49:52.725252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T23:34:38.240478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.4","version":"0'0","reported_seq":15,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-09-16T15:49:55.016179+0000","last_change":"2024-09-16T15:49:52.768224+0000","last_active":"2024-09-16T15:49:55.016179+0000","last_peered":"2024-09-16T15:49:55.016179+0000","last_clean":"2024-09-16T15:49:55.016179+0000","last_became_active":"2024-09-16T15:49:52.767916+0000","last_became_peered":"2024-09-16T15:49:52.767916+0000","last_unstale":"2024-09-16T15:49:55.016179+0000","last_undegraded":"2024-09-16T15:49:55.016179+0000","last_fullsized":"2024-09-16T15:49:55.016179+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:49:51.721471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:49:51.721471+0000","last_clean_scrub_stamp":"2024-09-16T15:49:51.721471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T23:56:11.721811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.2","version":"0'0","reported_seq":13,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-09-16T15:49:55.016164+0000","last_change":"2024-09-16T15:49:53.775623+0000","last_active":"2024-09-16T15:49:55.016164+0000","last_peered":"2024-09-16T15:49:55.016164+0000","last_clean":"2024-09-16T15:49:55.016164+0000","last_became_active":"2024-09-16T15:49:53.775132+0000","last_became_peered":"2024-09-16T15:49:53.775132+0000","last_unstale":"2024-09-16T15:49:55.016164+0000","last_undegraded":"2024-09-16T15:49:55.016164+0000","last_fullsized":"2024-09-16T15:49:55.016164+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:49:52.725252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:49:52.725252+0000","last_clean_scrub_stamp":"2024-09-16T15:49:52.725252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T00:11:48.212471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.3","version":"0'0","reported_seq":15,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-09-16T15:49:55.017859+0000","last_change":"2024-09-16T15:49:52.757448+0000","last_active":"2024-09-16T15:49:55.017859+0000","last_peered":"2024-09-16T15:49:55.017859+0000","last_clean":"2024-09-16T15:49:55.017859+0000","last_became_active":"2024-09-16T15:49:52.755623+0000","last_became_peered":"2024-09-16T15:49:52.755623+0000","last_unstale":"2024-09-16T15:49:55.017859+0000","last_undegraded":"2024-09-16T15:49:55.017859+0000","last_fullsized":"2024-09-16T15:49:55.017859+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:49:51.721471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:49:51.721471+0000","last_clean_scrub_stamp":"2024-09-16T15:49:51.721471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T03:42:03.536282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"21.3","version":"0'0","reported_seq":13,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-09-16T15:49:55.014186+0000","last_change":"2024-09-16T15:49:53.814161+0000","last_active":"2024-09-16T15:49:55.014186+0000","last_peered":"2024-09-16T15:49:55.014186+0000","last_clean":"2024-09-16T15:49:55.014186+0000","last_became_active":"2024-09-16T15:49:53.813475+0000","last_became_peered":"2024-09-16T15:49:53.813475+0000","last_unstale":"2024-09-16T15:49:55.014186+0000","last_undegraded":"2024-09-16T15:49:55.014186+0000","last_fullsized":"2024-09-16T15:49:55.014186+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:49:52.725252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:49:52.725252+0000","last_clean_scrub_stamp":"2024-09-16T15:49:52.725252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T23:23:59.844906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.2","version":"0'0","reported_seq":15,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-09-16T15:49:55.015118+0000","last_change":"2024-09-16T15:49:52.767818+0000","last_active":"2024-09-16T15:49:55.015118+0000","last_peered":"2024-09-16T15:49:55.015118+0000","last_clean":"2024-09-16T15:49:55.015118+0000","last_became_active":"2024-09-16T15:49:52.765724+0000","last_became_peered":"2024-09-16T15:49:52.765724+0000","last_unstale":"2024-09-16T15:49:55.015118+0000","last_undegraded":"2024-09-16T15:49:55.015118+0000","last_fullsized":"2024-09-16T15:49:55.015118+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:49:51.721471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:49:51.721471+0000","last_clean_scrub_stamp":"2024-09-16T15:49:51.721471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:52:33.088661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.0","version":"0'0","reported_seq":13,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-09-16T15:49:55.015432+0000","last_change":"2024-09-16T15:49:53.786223+0000","last_active":"2024-09-16T15:49:55.015432+0000","last_peered":"2024-09-16T15:49:55.015432+0000","last_clean":"2024-09-16T15:49:55.015432+0000","last_became_active":"2024-09-16T15:49:53.785450+0000","last_became_peered":"2024-09-16T15:49:53.785450+0000","last_unstale":"2024-09-16T15:49:55.015432+0000","last_undegraded":"2024-09-16T15:49:55.015432+0000","last_fullsized":"2024-09-16T15:49:55.015432+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:49:52.725252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:49:52.725252+0000","last_clean_scrub_stamp":"2024-09-16T15:49:52.725252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T22:50:14.368364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.1","version":"0'0","reported_seq":15,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-09-16T15:49:54.751466+0000","last_change":"2024-09-16T15:49:52.774681+0000","last_active":"2024-09-16T15:49:54.751466+0000","last_peered":"2024-09-16T15:49:54.751466+0000","last_clean":"2024-09-16T15:49:54.751466+0000","last_became_active":"2024-09-16T15:49:52.773868+0000","last_became_peered":"2024-09-16T15:49:52.773868+0000","last_unstale":"2024-09-16T15:49:54.751466+0000","last_undegraded":"2024-09-16T15:49:54.751466+0000","last_fullsized":"2024-09-16T15:49:54.751466+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:49:51.721471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:49:51.721471+0000","last_clean_scrub_stamp":"2024-09-16T15:49:51.721471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:36:12.259846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"21.1","version":"0'0","reported_seq":13,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-09-16T15:49:55.013352+0000","last_change":"2024-09-16T15:49:53.791806+0000","last_active":"2024-09-16T15:49:55.013352+0000","last_peered":"2024-09-16T15:49:55.013352+0000","last_clean":"2024-09-16T15:49:55.013352+0000","last_became_active":"2024-09-16T15:49:53.783641+0000","last_became_peered":"2024-09-16T15:49:53.783641+0000","last_unstale":"2024-09-16T15:49:55.013352+0000","last_undegraded":"2024-09-16T15:49:55.013352+0000","last_fullsized":"2024-09-16T15:49:55.013352+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:49:52.725252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:49:52.725252+0000","last_clean_scrub_stamp":"2024-09-16T15:49:52.725252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T02:13:51.898581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.0","version":"0'0","reported_seq":15,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-09-16T15:49:54.747977+0000","last_change":"2024-09-16T15:49:52.779185+0000","last_active":"2024-09-16T15:49:54.747977+0000","last_peered":"2024-09-16T15:49:54.747977+0000","last_clean":"2024-09-16T15:49:54.747977+0000","last_became_active":"2024-09-16T15:49:52.778952+0000","last_became_peered":"2024-09-16T15:49:52.778952+0000","last_unstale":"2024-09-16T15:49:54.747977+0000","last_undegraded":"2024-09-16T15:49:54.747977+0000","last_fullsized":"2024-09-16T15:49:54.747977+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:49:51.721471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:49:51.721471+0000","last_clean_scrub_stamp":"2024-09-16T15:49:51.721471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:39:55.480968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.14","version":"0'0","reported_seq":13,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-09-16T15:49:55.017797+0000","last_change":"2024-09-16T15:49:53.774718+0000","last_active":"2024-09-16T15:49:55.017797+0000","last_peered":"2024-09-16T15:49:55.017797+0000","last_clean":"2024-09-16T15:49:55.017797+0000","last_became_active":"2024-09-16T15:49:53.774232+0000","last_became_peered":"2024-09-16T15:49:53.774232+0000","last_unstale":"2024-09-16T15:49:55.017797+0000","last_undegraded":"2024-09-16T15:49:55.017797+0000","last_fullsized":"2024-09-16T15:49:55.017797+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:49:52.725252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:49:52.725252+0000","last_clean_scrub_stamp":"2024-09-16T15:49:52.725252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T01:08:23.372694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"20.15","version":"134'3","reported_seq":18,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-09-16T15:49:54.752729+0000","last_change":"2024-09-16T15:49:52.775063+0000","last_active":"2024-09-16T15:49:54.752729+0000","last_peered":"2024-09-16T15:49:54.752729+0000","last_clean":"2024-09-16T15:49:54.752729+0000","last_became_active":"2024-09-16T15:49:52.774426+0000","last_became_peered":"2024-09-16T15:49:52.774426+0000","last_unstale":"2024-09-16T15:49:54.752729+0000","last_undegraded":"2024-09-16T15:49:54.752729+0000","last_fullsized":"2024-09-16T15:49:54.752729+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:49:51.721471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:49:51.721471+0000","last_clean_scrub_stamp":"2024-09-16T15:49:51.721471+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:00:27.297007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"73'79","reported_seq":349,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-09-16T15:49:55.014545+0000","last_change":"2024-09-16T15:42:40.881820+0000","last_active":"2024-09-16T15:49:55.014545+0000","last_peered":"2024-09-16T15:49:55.014545+0000","last_clean":"2024-09-16T15:49:55.014545+0000","last_became_active":"2024-09-16T15:42:40.879965+0000","last_became_peered":"2024-09-16T15:42:40.879965+0000","last_unstale":"2024-09-16T15:49:55.014545+0000","last_undegraded":"2024-09-16T15:49:55.014545+0000","last_fullsized":"2024-09-16T15:49:55.014545+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:42:39.728762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:42:39.728762+0000","last_clean_scrub_stamp":"2024-09-16T15:42:39.728762+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:22:18.768932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":21,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":20,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79,"ondisk_log_size":79,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542235,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33416,"kb_used_data":2224,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338424,"statfs":{"total":96636764160,"available":96602546176,"internally_reserved":0,"allocated":2277376,"data_stored":1333070,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542234,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32840,"kb_used_data":1648,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339000,"statfs":{"total":96636764160,"available":96603136000,"internally_reserved":0,"allocated":1687552,"data_stored":742196,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":15,"apply_latency_ms":15,"commit_latency_ns":15000000,"apply_latency_ns":15000000},"alerts":[]},{"osd":5,"up_from":14,"seq":60129542235,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32828,"kb_used_data":1640,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339012,"statfs":{"total":96636764160,"available":96603148288,"internally_reserved":0,"allocated":1679360,"data_stored":742072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542233,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32836,"kb_used_data":1648,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339004,"statfs":{"total":96636764160,"available":96603140096,"internally_reserved":0,"allocated":1687552,"data_stored":748378,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":3,"up_from":14,"seq":60129542234,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32828,"kb_used_data":1640,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339012,"statfs":{"total":96636764160,"available":96603148288,"internally_reserved":0,"allocated":1679360,"data_stored":742140,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":2,"up_from":14,"seq":60129542234,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32856,"kb_used_data":1664,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338984,"statfs":{"total":96636764160,"available":96603119616,"internally_reserved":0,"allocated":1703936,"data_stored":750060,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":14,"seq":60129542236,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32828,"kb_used_data":1640,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339012,"statfs":{"total":96636764160,"available":96603148288,"internally_reserved":0,"allocated":1679360,"data_stored":742632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":14,"seq":60129542236,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33428,"kb_used_data":2232,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338412,"statfs":{"total":96636764160,"available":96602533888,"internally_reserved":0,"allocated":2285568,"data_stored":1334064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1646,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T15:49:57.821 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T15:49:57.990 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:57.990+0000 7fdf86123640 1 -- 172.21.15.123:0/1514161379 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdf6c004e70 msgr2=0x7fdf6c00b9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:57.990 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:57.990+0000 7fdf86123640 1 --2- 172.21.15.123:0/1514161379 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdf6c004e70 0x7fdf6c00b9d0 secure :-1 s=READY pgs=1410 cs=0 l=1 rev1=1 crypto rx=0x7fdf78009e00 tx=0x7fdf7801d0c0 comp rx=0 tx=0).stop 2024-09-16T15:49:57.990 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:57.990+0000 7fdf7ffff640 1 -- 172.21.15.123:0/1514161379 reap_dead start 2024-09-16T15:49:57.990 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:57.990+0000 7fdf86123640 1 -- 172.21.15.123:0/1514161379 shutdown_connections 2024-09-16T15:49:57.990 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:57.990+0000 7fdf86123640 1 -- 172.21.15.123:0/1514161379 >> 172.21.15.123:0/1514161379 conn(0x7fdf800d3ea0 msgr2=0x7fdf800d42b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:49:57.990 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:57.990+0000 7fdf86123640 1 -- 172.21.15.123:0/1514161379 shutdown_connections 2024-09-16T15:49:57.990 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:57.990+0000 7fdf86123640 1 -- 172.21.15.123:0/1514161379 wait complete. 2024-09-16T15:49:57.991 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:57.990+0000 7fdf86123640 1 Processor -- start 2024-09-16T15:49:57.991 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:57.990+0000 7fdf86123640 1 -- start start 2024-09-16T15:49:57.993 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:57.990+0000 7fdf86123640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fdf6c004e70 0x7fdf800b98b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:49:57.993 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:57.990+0000 7fdf86123640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdf8014f2d0 0x7fdf8014f750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:49:57.993 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:57.990+0000 7fdf86123640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fdf80150e70 0x7fdf801512f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:49:57.993 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:57.990+0000 7fdf86123640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fdf801698f0 con 0x7fdf8014f2d0 2024-09-16T15:49:57.993 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:57.990+0000 7fdf86123640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fdf80169770 con 0x7fdf6c004e70 2024-09-16T15:49:57.993 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:57.990+0000 7fdf86123640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fdf80169a70 con 0x7fdf80150e70 2024-09-16T15:49:57.993 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:57.990+0000 7fdf77fff640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdf8014f2d0 0x7fdf8014f750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:49:57.993 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:57.990+0000 7fdf77fff640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdf8014f2d0 0x7fdf8014f750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:58766/0 (socket says 172.21.15.123:58766) 2024-09-16T15:49:57.993 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:57.990+0000 7fdf77fff640 1 -- 172.21.15.123:0/120465913 learned_addr learned my addr 172.21.15.123:0/120465913 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:49:57.994 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:57.990+0000 7fdf7ffff640 1 --2- 172.21.15.123:0/120465913 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fdf80150e70 0x7fdf801512f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:49:57.994 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:57.990+0000 7fdf7f7fe640 1 --2- 172.21.15.123:0/120465913 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fdf6c004e70 0x7fdf800b98b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:49:57.994 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:57.990+0000 7fdf77fff640 1 -- 172.21.15.123:0/120465913 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fdf80150e70 msgr2=0x7fdf801512f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:57.994 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:57.990+0000 7fdf77fff640 1 --2- 172.21.15.123:0/120465913 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fdf80150e70 0x7fdf801512f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:57.994 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:57.990+0000 7fdf77fff640 1 -- 172.21.15.123:0/120465913 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fdf6c004e70 msgr2=0x7fdf800b98b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:57.994 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:57.990+0000 7fdf77fff640 1 --2- 172.21.15.123:0/120465913 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fdf6c004e70 0x7fdf800b98b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:57.994 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:57.990+0000 7fdf77fff640 1 -- 172.21.15.123:0/120465913 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdf801f7e10 con 0x7fdf8014f2d0 2024-09-16T15:49:57.994 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:57.990+0000 7fdf7f7fe640 1 --2- 172.21.15.123:0/120465913 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fdf6c004e70 0x7fdf800b98b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T15:49:57.994 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:57.994+0000 7fdf77fff640 1 --2- 172.21.15.123:0/120465913 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdf8014f2d0 0x7fdf8014f750 secure :-1 s=READY pgs=1411 cs=0 l=1 rev1=1 crypto rx=0x7fdf6800d810 tx=0x7fdf6800dcd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:49:57.994 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:57.994+0000 7fdf7cff9640 1 -- 172.21.15.123:0/120465913 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf68019070 con 0x7fdf8014f2d0 2024-09-16T15:49:57.994 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:57.994+0000 7fdf7cff9640 1 -- 172.21.15.123:0/120465913 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fdf680050d0 con 0x7fdf8014f2d0 2024-09-16T15:49:57.994 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:57.994+0000 7fdf86123640 1 -- 172.21.15.123:0/120465913 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdf801f80f0 con 0x7fdf8014f2d0 2024-09-16T15:49:57.994 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:57.994+0000 7fdf7cff9640 1 -- 172.21.15.123:0/120465913 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf6800b660 con 0x7fdf8014f2d0 2024-09-16T15:49:57.995 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:57.994+0000 7fdf86123640 1 -- 172.21.15.123:0/120465913 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdf801f86b0 con 0x7fdf8014f2d0 2024-09-16T15:49:57.995 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:57.994+0000 7fdf86123640 1 -- 172.21.15.123:0/120465913 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdf800d1da0 con 0x7fdf8014f2d0 2024-09-16T15:49:57.995 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:57.994+0000 7fdf7cff9640 1 -- 172.21.15.123:0/120465913 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 21) ==== 97641+0+0 (secure 0 0 0) 0x7fdf6801c070 con 0x7fdf8014f2d0 2024-09-16T15:49:57.995 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:57.994+0000 7fdf7cff9640 1 --2- 172.21.15.123:0/120465913 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fdf60073520 0x7fdf600759e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:49:57.996 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:57.994+0000 7fdf7f7fe640 1 --2- 172.21.15.123:0/120465913 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fdf60073520 0x7fdf600759e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:49:57.996 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:57.994+0000 7fdf7cff9640 1 -- 172.21.15.123:0/120465913 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(135..135 src has 1..135) ==== 5737+0+0 (secure 0 0 0) 0x7fdf68094090 con 0x7fdf8014f2d0 2024-09-16T15:49:57.996 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:57.994+0000 7fdf7f7fe640 1 --2- 172.21.15.123:0/120465913 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fdf60073520 0x7fdf600759e0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7fdf70006fd0 tx=0x7fdf70006b90 comp rx=0 tx=0).ready entity=mgr.4150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:49:57.998 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:57.998+0000 7fdf7cff9640 1 -- 172.21.15.123:0/120465913 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7fdf6805e900 con 0x7fdf8014f2d0 2024-09-16T15:49:58.167 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:58.166+0000 7fdf86123640 1 -- 172.21.15.123:0/120465913 --> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fdf800747b0 con 0x7fdf60073520 2024-09-16T15:49:58.171 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:58.170+0000 7fdf7cff9640 1 -- 172.21.15.123:0/120465913 <== mgr.4150 v2:172.21.15.123:6836/2464759676 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+252480 (secure 0 0 0) 0x7fdf800747b0 con 0x7fdf60073520 2024-09-16T15:49:58.174 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-09-16T15:49:58.175 INFO:teuthology.orchestra.run.smithi123.stderr:dumped all 2024-09-16T15:49:58.176 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:58.174+0000 7fdf86123640 1 -- 172.21.15.123:0/120465913 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fdf60073520 msgr2=0x7fdf600759e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:58.176 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:58.174+0000 7fdf86123640 1 --2- 172.21.15.123:0/120465913 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fdf60073520 0x7fdf600759e0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7fdf70006fd0 tx=0x7fdf70006b90 comp rx=0 tx=0).stop 2024-09-16T15:49:58.176 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:58.174+0000 7fdf86123640 1 -- 172.21.15.123:0/120465913 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdf8014f2d0 msgr2=0x7fdf8014f750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:58.176 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:58.174+0000 7fdf86123640 1 --2- 172.21.15.123:0/120465913 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdf8014f2d0 0x7fdf8014f750 secure :-1 s=READY pgs=1411 cs=0 l=1 rev1=1 crypto rx=0x7fdf6800d810 tx=0x7fdf6800dcd0 comp rx=0 tx=0).stop 2024-09-16T15:49:58.176 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:58.174+0000 7fdf86123640 1 -- 172.21.15.123:0/120465913 shutdown_connections 2024-09-16T15:49:58.176 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:58.174+0000 7fdf86123640 1 --2- 172.21.15.123:0/120465913 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fdf60073520 0x7fdf600759e0 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:58.176 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:58.174+0000 7fdf86123640 1 --2- 172.21.15.123:0/120465913 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fdf80150e70 0x7fdf801512f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:58.176 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:58.174+0000 7fdf86123640 1 --2- 172.21.15.123:0/120465913 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdf8014f2d0 0x7fdf8014f750 unknown :-1 s=CLOSED pgs=1411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:58.177 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:58.174+0000 7fdf86123640 1 --2- 172.21.15.123:0/120465913 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fdf6c004e70 0x7fdf800b98b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:58.177 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:58.174+0000 7fdf86123640 1 -- 172.21.15.123:0/120465913 >> 172.21.15.123:0/120465913 conn(0x7fdf800d3ea0 msgr2=0x7fdf80154580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:49:58.177 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:58.174+0000 7fdf86123640 1 -- 172.21.15.123:0/120465913 shutdown_connections 2024-09-16T15:49:58.177 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:58.174+0000 7fdf86123640 1 -- 172.21.15.123:0/120465913 wait complete. 2024-09-16T15:49:58.191 INFO:teuthology.orchestra.run.smithi123.stdout:{"pg_ready":true,"pg_map":{"version":138,"stamp":"2024-09-16T15:49:58.109587+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":157,"num_write_kb":1642,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":80,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":263860,"kb_used_data":14336,"kb_used_omap":12,"kb_used_meta":249331,"kb_avail":754710860,"statfs":{"total":773094113280,"available":772823920640,"internally_reserved":0,"allocated":14680064,"data_stored":7134612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":255315505},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":46,"apply_latency_ms":46,"commit_latency_ns":46000000,"apply_latency_ns":46000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001831"},"pg_stats":[{"pgid":"21.3f","version":"0'0","reported_seq":13,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-09-16T15:49:55.017428+0000","last_change":"2024-09-16T15:49:53.776624+0000","last_active":"2024-09-16T15:49:55.017428+0000","last_peered":"2024-09-16T15:49:55.017428+0000","last_clean":"2024-09-16T15:49:55.017428+0000","last_became_active":"2024-09-16T15:49:53.776370+0000","last_became_peered":"2024-09-16T15:49:53.776370+0000","last_unstale":"2024-09-16T15:49:55.017428+0000","last_undegraded":"2024-09-16T15:49:55.017428+0000","last_fullsized":"2024-09-16T15:49:55.017428+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:49:52.725252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:49:52.725252+0000","last_clean_scrub_stamp":"2024-09-16T15:49:52.725252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:26:03.608900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.3e","version":"0'0","reported_seq":13,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-09-16T15:49:54.749305+0000","last_change":"2024-09-16T15:49:53.772932+0000","last_active":"2024-09-16T15:49:54.749305+0000","last_peered":"2024-09-16T15:49:54.749305+0000","last_clean":"2024-09-16T15:49:54.749305+0000","last_became_active":"2024-09-16T15:49:53.772471+0000","last_became_peered":"2024-09-16T15:49:53.772471+0000","last_unstale":"2024-09-16T15:49:54.749305+0000","last_undegraded":"2024-09-16T15:49:54.749305+0000","last_fullsized":"2024-09-16T15:49:54.749305+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:49:52.725252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:49:52.725252+0000","last_clean_scrub_stamp":"2024-09-16T15:49:52.725252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T01:36:37.006276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.3d","version":"0'0","reported_seq":13,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-09-16T15:49:54.752437+0000","last_change":"2024-09-16T15:49:53.808252+0000","last_active":"2024-09-16T15:49:54.752437+0000","last_peered":"2024-09-16T15:49:54.752437+0000","last_clean":"2024-09-16T15:49:54.752437+0000","last_became_active":"2024-09-16T15:49:53.807204+0000","last_became_peered":"2024-09-16T15:49:53.807204+0000","last_unstale":"2024-09-16T15:49:54.752437+0000","last_undegraded":"2024-09-16T15:49:54.752437+0000","last_fullsized":"2024-09-16T15:49:54.752437+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:49:52.725252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:49:52.725252+0000","last_clean_scrub_stamp":"2024-09-16T15:49:52.725252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T23:32:34.768446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"21.3c","version":"0'0","reported_seq":13,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-09-16T15:49:54.752083+0000","last_change":"2024-09-16T15:49:53.809842+0000","last_active":"2024-09-16T15:49:54.752083+0000","last_peered":"2024-09-16T15:49:54.752083+0000","last_clean":"2024-09-16T15:49:54.752083+0000","last_became_active":"2024-09-16T15:49:53.809353+0000","last_became_peered":"2024-09-16T15:49:53.809353+0000","last_unstale":"2024-09-16T15:49:54.752083+0000","last_undegraded":"2024-09-16T15:49:54.752083+0000","last_fullsized":"2024-09-16T15:49:54.752083+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:49:52.725252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:49:52.725252+0000","last_clean_scrub_stamp":"2024-09-16T15:49:52.725252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T22:01:27.459937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.3b","version":"0'0","reported_seq":13,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-09-16T15:49:55.014985+0000","last_change":"2024-09-16T15:49:53.794026+0000","last_active":"2024-09-16T15:49:55.014985+0000","last_peered":"2024-09-16T15:49:55.014985+0000","last_clean":"2024-09-16T15:49:55.014985+0000","last_became_active":"2024-09-16T15:49:53.790246+0000","last_became_peered":"2024-09-16T15:49:53.790246+0000","last_unstale":"2024-09-16T15:49:55.014985+0000","last_undegraded":"2024-09-16T15:49:55.014985+0000","last_fullsized":"2024-09-16T15:49:55.014985+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:49:52.725252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:49:52.725252+0000","last_clean_scrub_stamp":"2024-09-16T15:49:52.725252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T22:23:38.505887+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"21.3a","version":"0'0","reported_seq":13,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-09-16T15:49:55.015658+0000","last_change":"2024-09-16T15:49:53.808447+0000","last_active":"2024-09-16T15:49:55.015658+0000","last_peered":"2024-09-16T15:49:55.015658+0000","last_clean":"2024-09-16T15:49:55.015658+0000","last_became_active":"2024-09-16T15:49:53.808152+0000","last_became_peered":"2024-09-16T15:49:53.808152+0000","last_unstale":"2024-09-16T15:49:55.015658+0000","last_undegraded":"2024-09-16T15:49:55.015658+0000","last_fullsized":"2024-09-16T15:49:55.015658+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:49:52.725252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:49:52.725252+0000","last_clean_scrub_stamp":"2024-09-16T15:49:52.725252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T22:48:58.657590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.39","version":"0'0","reported_seq":13,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-09-16T15:49:55.016077+0000","last_change":"2024-09-16T15:49:53.774744+0000","last_active":"2024-09-16T15:49:55.016077+0000","last_peered":"2024-09-16T15:49:55.016077+0000","last_clean":"2024-09-16T15:49:55.016077+0000","last_became_active":"2024-09-16T15:49:53.774466+0000","last_became_peered":"2024-09-16T15:49:53.774466+0000","last_unstale":"2024-09-16T15:49:55.016077+0000","last_undegraded":"2024-09-16T15:49:55.016077+0000","last_fullsized":"2024-09-16T15:49:55.016077+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:49:52.725252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:49:52.725252+0000","last_clean_scrub_stamp":"2024-09-16T15:49:52.725252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:44:50.941421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"21.38","version":"0'0","reported_seq":13,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-09-16T15:49:55.014928+0000","last_change":"2024-09-16T15:49:53.785007+0000","last_active":"2024-09-16T15:49:55.014928+0000","last_peered":"2024-09-16T15:49:55.014928+0000","last_clean":"2024-09-16T15:49:55.014928+0000","last_became_active":"2024-09-16T15:49:53.784695+0000","last_became_peered":"2024-09-16T15:49:53.784695+0000","last_unstale":"2024-09-16T15:49:55.014928+0000","last_undegraded":"2024-09-16T15:49:55.014928+0000","last_fullsized":"2024-09-16T15:49:55.014928+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:49:52.725252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:49:52.725252+0000","last_clean_scrub_stamp":"2024-09-16T15:49:52.725252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:45:44.627500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.37","version":"0'0","reported_seq":13,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-09-16T15:49:54.749146+0000","last_change":"2024-09-16T15:49:53.773613+0000","last_active":"2024-09-16T15:49:54.749146+0000","last_peered":"2024-09-16T15:49:54.749146+0000","last_clean":"2024-09-16T15:49:54.749146+0000","last_became_active":"2024-09-16T15:49:53.773284+0000","last_became_peered":"2024-09-16T15:49:53.773284+0000","last_unstale":"2024-09-16T15:49:54.749146+0000","last_undegraded":"2024-09-16T15:49:54.749146+0000","last_fullsized":"2024-09-16T15:49:54.749146+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:49:52.725252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:49:52.725252+0000","last_clean_scrub_stamp":"2024-09-16T15:49:52.725252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:43:49.339223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.36","version":"0'0","reported_seq":13,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-09-16T15:49:55.015877+0000","last_change":"2024-09-16T15:49:53.799514+0000","last_active":"2024-09-16T15:49:55.015877+0000","last_peered":"2024-09-16T15:49:55.015877+0000","last_clean":"2024-09-16T15:49:55.015877+0000","last_became_active":"2024-09-16T15:49:53.798835+0000","last_became_peered":"2024-09-16T15:49:53.798835+0000","last_unstale":"2024-09-16T15:49:55.015877+0000","last_undegraded":"2024-09-16T15:49:55.015877+0000","last_fullsized":"2024-09-16T15:49:55.015877+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:49:52.725252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:49:52.725252+0000","last_clean_scrub_stamp":"2024-09-16T15:49:52.725252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T23:31:40.351018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.35","version":"0'0","reported_seq":13,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-09-16T15:49:55.015824+0000","last_change":"2024-09-16T15:49:53.783766+0000","last_active":"2024-09-16T15:49:55.015824+0000","last_peered":"2024-09-16T15:49:55.015824+0000","last_clean":"2024-09-16T15:49:55.015824+0000","last_became_active":"2024-09-16T15:49:53.783442+0000","last_became_peered":"2024-09-16T15:49:53.783442+0000","last_unstale":"2024-09-16T15:49:55.015824+0000","last_undegraded":"2024-09-16T15:49:55.015824+0000","last_fullsized":"2024-09-16T15:49:55.015824+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:49:52.725252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:49:52.725252+0000","last_clean_scrub_stamp":"2024-09-16T15:49:52.725252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:00:04.218087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.34","version":"0'0","reported_seq":13,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-09-16T15:49:55.016097+0000","last_change":"2024-09-16T15:49:53.801949+0000","last_active":"2024-09-16T15:49:55.016097+0000","last_peered":"2024-09-16T15:49:55.016097+0000","last_clean":"2024-09-16T15:49:55.016097+0000","last_became_active":"2024-09-16T15:49:53.800152+0000","last_became_peered":"2024-09-16T15:49:53.800152+0000","last_unstale":"2024-09-16T15:49:55.016097+0000","last_undegraded":"2024-09-16T15:49:55.016097+0000","last_fullsized":"2024-09-16T15:49:55.016097+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:49:52.725252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:49:52.725252+0000","last_clean_scrub_stamp":"2024-09-16T15:49:52.725252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:23:25.956451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.33","version":"0'0","reported_seq":13,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-09-16T15:49:55.013657+0000","last_change":"2024-09-16T15:49:53.793781+0000","last_active":"2024-09-16T15:49:55.013657+0000","last_peered":"2024-09-16T15:49:55.013657+0000","last_clean":"2024-09-16T15:49:55.013657+0000","last_became_active":"2024-09-16T15:49:53.784790+0000","last_became_peered":"2024-09-16T15:49:53.784790+0000","last_unstale":"2024-09-16T15:49:55.013657+0000","last_undegraded":"2024-09-16T15:49:55.013657+0000","last_fullsized":"2024-09-16T15:49:55.013657+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:49:52.725252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:49:52.725252+0000","last_clean_scrub_stamp":"2024-09-16T15:49:52.725252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T02:02:21.975949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"21.32","version":"0'0","reported_seq":13,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-09-16T15:49:54.750051+0000","last_change":"2024-09-16T15:49:53.778571+0000","last_active":"2024-09-16T15:49:54.750051+0000","last_peered":"2024-09-16T15:49:54.750051+0000","last_clean":"2024-09-16T15:49:54.750051+0000","last_became_active":"2024-09-16T15:49:53.777156+0000","last_became_peered":"2024-09-16T15:49:53.777156+0000","last_unstale":"2024-09-16T15:49:54.750051+0000","last_undegraded":"2024-09-16T15:49:54.750051+0000","last_fullsized":"2024-09-16T15:49:54.750051+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:49:52.725252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:49:52.725252+0000","last_clean_scrub_stamp":"2024-09-16T15:49:52.725252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T22:22:36.895961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.31","version":"0'0","reported_seq":13,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-09-16T15:49:55.014470+0000","last_change":"2024-09-16T15:49:53.782437+0000","last_active":"2024-09-16T15:49:55.014470+0000","last_peered":"2024-09-16T15:49:55.014470+0000","last_clean":"2024-09-16T15:49:55.014470+0000","last_became_active":"2024-09-16T15:49:53.781726+0000","last_became_peered":"2024-09-16T15:49:53.781726+0000","last_unstale":"2024-09-16T15:49:55.014470+0000","last_undegraded":"2024-09-16T15:49:55.014470+0000","last_fullsized":"2024-09-16T15:49:55.014470+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:49:52.725252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:49:52.725252+0000","last_clean_scrub_stamp":"2024-09-16T15:49:52.725252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T01:18:42.700556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.30","version":"0'0","reported_seq":13,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-09-16T15:49:55.015366+0000","last_change":"2024-09-16T15:49:53.762868+0000","last_active":"2024-09-16T15:49:55.015366+0000","last_peered":"2024-09-16T15:49:55.015366+0000","last_clean":"2024-09-16T15:49:55.015366+0000","last_became_active":"2024-09-16T15:49:53.762584+0000","last_became_peered":"2024-09-16T15:49:53.762584+0000","last_unstale":"2024-09-16T15:49:55.015366+0000","last_undegraded":"2024-09-16T15:49:55.015366+0000","last_fullsized":"2024-09-16T15:49:55.015366+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:49:52.725252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:49:52.725252+0000","last_clean_scrub_stamp":"2024-09-16T15:49:52.725252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T01:51:32.315200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.2f","version":"0'0","reported_seq":13,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-09-16T15:49:55.015773+0000","last_change":"2024-09-16T15:49:53.798961+0000","last_active":"2024-09-16T15:49:55.015773+0000","last_peered":"2024-09-16T15:49:55.015773+0000","last_clean":"2024-09-16T15:49:55.015773+0000","last_became_active":"2024-09-16T15:49:53.797676+0000","last_became_peered":"2024-09-16T15:49:53.797676+0000","last_unstale":"2024-09-16T15:49:55.015773+0000","last_undegraded":"2024-09-16T15:49:55.015773+0000","last_fullsized":"2024-09-16T15:49:55.015773+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:49:52.725252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:49:52.725252+0000","last_clean_scrub_stamp":"2024-09-16T15:49:52.725252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T03:05:01.704120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.2e","version":"0'0","reported_seq":13,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-09-16T15:49:54.753203+0000","last_change":"2024-09-16T15:49:53.806273+0000","last_active":"2024-09-16T15:49:54.753203+0000","last_peered":"2024-09-16T15:49:54.753203+0000","last_clean":"2024-09-16T15:49:54.753203+0000","last_became_active":"2024-09-16T15:49:53.805609+0000","last_became_peered":"2024-09-16T15:49:53.805609+0000","last_unstale":"2024-09-16T15:49:54.753203+0000","last_undegraded":"2024-09-16T15:49:54.753203+0000","last_fullsized":"2024-09-16T15:49:54.753203+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:49:52.725252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:49:52.725252+0000","last_clean_scrub_stamp":"2024-09-16T15:49:52.725252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T02:15:44.638806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"21.2d","version":"0'0","reported_seq":13,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-09-16T15:49:55.017511+0000","last_change":"2024-09-16T15:49:53.761753+0000","last_active":"2024-09-16T15:49:55.017511+0000","last_peered":"2024-09-16T15:49:55.017511+0000","last_clean":"2024-09-16T15:49:55.017511+0000","last_became_active":"2024-09-16T15:49:53.761452+0000","last_became_peered":"2024-09-16T15:49:53.761452+0000","last_unstale":"2024-09-16T15:49:55.017511+0000","last_undegraded":"2024-09-16T15:49:55.017511+0000","last_fullsized":"2024-09-16T15:49:55.017511+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:49:52.725252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:49:52.725252+0000","last_clean_scrub_stamp":"2024-09-16T15:49:52.725252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:07:17.368998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.2c","version":"0'0","reported_seq":13,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-09-16T15:49:55.017110+0000","last_change":"2024-09-16T15:49:53.772294+0000","last_active":"2024-09-16T15:49:55.017110+0000","last_peered":"2024-09-16T15:49:55.017110+0000","last_clean":"2024-09-16T15:49:55.017110+0000","last_became_active":"2024-09-16T15:49:53.771986+0000","last_became_peered":"2024-09-16T15:49:53.771986+0000","last_unstale":"2024-09-16T15:49:55.017110+0000","last_undegraded":"2024-09-16T15:49:55.017110+0000","last_fullsized":"2024-09-16T15:49:55.017110+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:49:52.725252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:49:52.725252+0000","last_clean_scrub_stamp":"2024-09-16T15:49:52.725252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:55:21.714737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.2b","version":"0'0","reported_seq":13,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-09-16T15:49:55.015237+0000","last_change":"2024-09-16T15:49:53.790512+0000","last_active":"2024-09-16T15:49:55.015237+0000","last_peered":"2024-09-16T15:49:55.015237+0000","last_clean":"2024-09-16T15:49:55.015237+0000","last_became_active":"2024-09-16T15:49:53.789779+0000","last_became_peered":"2024-09-16T15:49:53.789779+0000","last_unstale":"2024-09-16T15:49:55.015237+0000","last_undegraded":"2024-09-16T15:49:55.015237+0000","last_fullsized":"2024-09-16T15:49:55.015237+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:49:52.725252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:49:52.725252+0000","last_clean_scrub_stamp":"2024-09-16T15:49:52.725252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T18:23:11.685238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.2a","version":"0'0","reported_seq":13,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-09-16T15:49:54.750453+0000","last_change":"2024-09-16T15:49:53.776142+0000","last_active":"2024-09-16T15:49:54.750453+0000","last_peered":"2024-09-16T15:49:54.750453+0000","last_clean":"2024-09-16T15:49:54.750453+0000","last_became_active":"2024-09-16T15:49:53.772977+0000","last_became_peered":"2024-09-16T15:49:53.772977+0000","last_unstale":"2024-09-16T15:49:54.750453+0000","last_undegraded":"2024-09-16T15:49:54.750453+0000","last_fullsized":"2024-09-16T15:49:54.750453+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:49:52.725252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:49:52.725252+0000","last_clean_scrub_stamp":"2024-09-16T15:49:52.725252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T23:39:37.396671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.29","version":"0'0","reported_seq":13,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-09-16T15:49:54.750006+0000","last_change":"2024-09-16T15:49:53.767624+0000","last_active":"2024-09-16T15:49:54.750006+0000","last_peered":"2024-09-16T15:49:54.750006+0000","last_clean":"2024-09-16T15:49:54.750006+0000","last_became_active":"2024-09-16T15:49:53.766908+0000","last_became_peered":"2024-09-16T15:49:53.766908+0000","last_unstale":"2024-09-16T15:49:54.750006+0000","last_undegraded":"2024-09-16T15:49:54.750006+0000","last_fullsized":"2024-09-16T15:49:54.750006+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:49:52.725252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:49:52.725252+0000","last_clean_scrub_stamp":"2024-09-16T15:49:52.725252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T01:08:26.895306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.28","version":"0'0","reported_seq":13,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-09-16T15:49:55.014366+0000","last_change":"2024-09-16T15:49:53.789465+0000","last_active":"2024-09-16T15:49:55.014366+0000","last_peered":"2024-09-16T15:49:55.014366+0000","last_clean":"2024-09-16T15:49:55.014366+0000","last_became_active":"2024-09-16T15:49:53.788737+0000","last_became_peered":"2024-09-16T15:49:53.788737+0000","last_unstale":"2024-09-16T15:49:55.014366+0000","last_undegraded":"2024-09-16T15:49:55.014366+0000","last_fullsized":"2024-09-16T15:49:55.014366+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:49:52.725252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:49:52.725252+0000","last_clean_scrub_stamp":"2024-09-16T15:49:52.725252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:10:45.743329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.27","version":"0'0","reported_seq":13,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-09-16T15:49:55.016881+0000","last_change":"2024-09-16T15:49:53.770290+0000","last_active":"2024-09-16T15:49:55.016881+0000","last_peered":"2024-09-16T15:49:55.016881+0000","last_clean":"2024-09-16T15:49:55.016881+0000","last_became_active":"2024-09-16T15:49:53.770020+0000","last_became_peered":"2024-09-16T15:49:53.770020+0000","last_unstale":"2024-09-16T15:49:55.016881+0000","last_undegraded":"2024-09-16T15:49:55.016881+0000","last_fullsized":"2024-09-16T15:49:55.016881+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:49:52.725252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:49:52.725252+0000","last_clean_scrub_stamp":"2024-09-16T15:49:52.725252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:21:26.353413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.26","version":"0'0","reported_seq":13,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-09-16T15:49:55.015999+0000","last_change":"2024-09-16T15:49:53.761671+0000","last_active":"2024-09-16T15:49:55.015999+0000","last_peered":"2024-09-16T15:49:55.015999+0000","last_clean":"2024-09-16T15:49:55.015999+0000","last_became_active":"2024-09-16T15:49:53.761225+0000","last_became_peered":"2024-09-16T15:49:53.761225+0000","last_unstale":"2024-09-16T15:49:55.015999+0000","last_undegraded":"2024-09-16T15:49:55.015999+0000","last_fullsized":"2024-09-16T15:49:55.015999+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:49:52.725252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:49:52.725252+0000","last_clean_scrub_stamp":"2024-09-16T15:49:52.725252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T23:33:08.035496+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.25","version":"0'0","reported_seq":13,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-09-16T15:49:55.016927+0000","last_change":"2024-09-16T15:49:53.809122+0000","last_active":"2024-09-16T15:49:55.016927+0000","last_peered":"2024-09-16T15:49:55.016927+0000","last_clean":"2024-09-16T15:49:55.016927+0000","last_became_active":"2024-09-16T15:49:53.808805+0000","last_became_peered":"2024-09-16T15:49:53.808805+0000","last_unstale":"2024-09-16T15:49:55.016927+0000","last_undegraded":"2024-09-16T15:49:55.016927+0000","last_fullsized":"2024-09-16T15:49:55.016927+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:49:52.725252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:49:52.725252+0000","last_clean_scrub_stamp":"2024-09-16T15:49:52.725252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:53:15.380959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.24","version":"0'0","reported_seq":13,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-09-16T15:49:55.017233+0000","last_change":"2024-09-16T15:49:53.814338+0000","last_active":"2024-09-16T15:49:55.017233+0000","last_peered":"2024-09-16T15:49:55.017233+0000","last_clean":"2024-09-16T15:49:55.017233+0000","last_became_active":"2024-09-16T15:49:53.813945+0000","last_became_peered":"2024-09-16T15:49:53.813945+0000","last_unstale":"2024-09-16T15:49:55.017233+0000","last_undegraded":"2024-09-16T15:49:55.017233+0000","last_fullsized":"2024-09-16T15:49:55.017233+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:49:52.725252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:49:52.725252+0000","last_clean_scrub_stamp":"2024-09-16T15:49:52.725252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T02:59:39.191888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"21.23","version":"0'0","reported_seq":13,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-09-16T15:49:55.014885+0000","last_change":"2024-09-16T15:49:53.806453+0000","last_active":"2024-09-16T15:49:55.014885+0000","last_peered":"2024-09-16T15:49:55.014885+0000","last_clean":"2024-09-16T15:49:55.014885+0000","last_became_active":"2024-09-16T15:49:53.806200+0000","last_became_peered":"2024-09-16T15:49:53.806200+0000","last_unstale":"2024-09-16T15:49:55.014885+0000","last_undegraded":"2024-09-16T15:49:55.014885+0000","last_fullsized":"2024-09-16T15:49:55.014885+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:49:52.725252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:49:52.725252+0000","last_clean_scrub_stamp":"2024-09-16T15:49:52.725252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:11:19.892285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.22","version":"0'0","reported_seq":13,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-09-16T15:49:54.750774+0000","last_change":"2024-09-16T15:49:53.758217+0000","last_active":"2024-09-16T15:49:54.750774+0000","last_peered":"2024-09-16T15:49:54.750774+0000","last_clean":"2024-09-16T15:49:54.750774+0000","last_became_active":"2024-09-16T15:49:53.757252+0000","last_became_peered":"2024-09-16T15:49:53.757252+0000","last_unstale":"2024-09-16T15:49:54.750774+0000","last_undegraded":"2024-09-16T15:49:54.750774+0000","last_fullsized":"2024-09-16T15:49:54.750774+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:49:52.725252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:49:52.725252+0000","last_clean_scrub_stamp":"2024-09-16T15:49:52.725252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T00:21:32.615337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.21","version":"0'0","reported_seq":13,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-09-16T15:49:54.748650+0000","last_change":"2024-09-16T15:49:53.810297+0000","last_active":"2024-09-16T15:49:54.748650+0000","last_peered":"2024-09-16T15:49:54.748650+0000","last_clean":"2024-09-16T15:49:54.748650+0000","last_became_active":"2024-09-16T15:49:53.810018+0000","last_became_peered":"2024-09-16T15:49:53.810018+0000","last_unstale":"2024-09-16T15:49:54.748650+0000","last_undegraded":"2024-09-16T15:49:54.748650+0000","last_fullsized":"2024-09-16T15:49:54.748650+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:49:52.725252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:49:52.725252+0000","last_clean_scrub_stamp":"2024-09-16T15:49:52.725252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T23:50:35.196885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.20","version":"0'0","reported_seq":13,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-09-16T15:49:55.015940+0000","last_change":"2024-09-16T15:49:53.785301+0000","last_active":"2024-09-16T15:49:55.015940+0000","last_peered":"2024-09-16T15:49:55.015940+0000","last_clean":"2024-09-16T15:49:55.015940+0000","last_became_active":"2024-09-16T15:49:53.784991+0000","last_became_peered":"2024-09-16T15:49:53.784991+0000","last_unstale":"2024-09-16T15:49:55.015940+0000","last_undegraded":"2024-09-16T15:49:55.015940+0000","last_fullsized":"2024-09-16T15:49:55.015940+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:49:52.725252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:49:52.725252+0000","last_clean_scrub_stamp":"2024-09-16T15:49:52.725252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:28:00.897124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.1e","version":"0'0","reported_seq":13,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-09-16T15:49:55.015777+0000","last_change":"2024-09-16T15:49:53.811225+0000","last_active":"2024-09-16T15:49:55.015777+0000","last_peered":"2024-09-16T15:49:55.015777+0000","last_clean":"2024-09-16T15:49:55.015777+0000","last_became_active":"2024-09-16T15:49:53.810759+0000","last_became_peered":"2024-09-16T15:49:53.810759+0000","last_unstale":"2024-09-16T15:49:55.015777+0000","last_undegraded":"2024-09-16T15:49:55.015777+0000","last_fullsized":"2024-09-16T15:49:55.015777+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:49:52.725252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:49:52.725252+0000","last_clean_scrub_stamp":"2024-09-16T15:49:52.725252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T22:09:52.767577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.1f","version":"134'4","reported_seq":19,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-09-16T15:49:55.015563+0000","last_change":"2024-09-16T15:49:52.775873+0000","last_active":"2024-09-16T15:49:55.015563+0000","last_peered":"2024-09-16T15:49:55.015563+0000","last_clean":"2024-09-16T15:49:55.015563+0000","last_became_active":"2024-09-16T15:49:52.773732+0000","last_became_peered":"2024-09-16T15:49:52.773732+0000","last_unstale":"2024-09-16T15:49:55.015563+0000","last_undegraded":"2024-09-16T15:49:55.015563+0000","last_fullsized":"2024-09-16T15:49:55.015563+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:49:51.721471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:49:51.721471+0000","last_clean_scrub_stamp":"2024-09-16T15:49:51.721471+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:23:53.699929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.1f","version":"0'0","reported_seq":13,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-09-16T15:49:54.749395+0000","last_change":"2024-09-16T15:49:53.772719+0000","last_active":"2024-09-16T15:49:54.749395+0000","last_peered":"2024-09-16T15:49:54.749395+0000","last_clean":"2024-09-16T15:49:54.749395+0000","last_became_active":"2024-09-16T15:49:53.766969+0000","last_became_peered":"2024-09-16T15:49:53.766969+0000","last_unstale":"2024-09-16T15:49:54.749395+0000","last_undegraded":"2024-09-16T15:49:54.749395+0000","last_fullsized":"2024-09-16T15:49:54.749395+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:49:52.725252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:49:52.725252+0000","last_clean_scrub_stamp":"2024-09-16T15:49:52.725252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T03:06:22.126597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.1e","version":"134'1","reported_seq":16,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-09-16T15:49:54.749159+0000","last_change":"2024-09-16T15:49:52.777280+0000","last_active":"2024-09-16T15:49:54.749159+0000","last_peered":"2024-09-16T15:49:54.749159+0000","last_clean":"2024-09-16T15:49:54.749159+0000","last_became_active":"2024-09-16T15:49:52.776932+0000","last_became_peered":"2024-09-16T15:49:52.776932+0000","last_unstale":"2024-09-16T15:49:54.749159+0000","last_undegraded":"2024-09-16T15:49:54.749159+0000","last_fullsized":"2024-09-16T15:49:54.749159+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:49:51.721471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:49:51.721471+0000","last_clean_scrub_stamp":"2024-09-16T15:49:51.721471+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T03:18:54.303040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.1c","version":"0'0","reported_seq":13,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-09-16T15:49:55.015542+0000","last_change":"2024-09-16T15:49:53.760772+0000","last_active":"2024-09-16T15:49:55.015542+0000","last_peered":"2024-09-16T15:49:55.015542+0000","last_clean":"2024-09-16T15:49:55.015542+0000","last_became_active":"2024-09-16T15:49:53.760233+0000","last_became_peered":"2024-09-16T15:49:53.760233+0000","last_unstale":"2024-09-16T15:49:55.015542+0000","last_undegraded":"2024-09-16T15:49:55.015542+0000","last_fullsized":"2024-09-16T15:49:55.015542+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:49:52.725252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:49:52.725252+0000","last_clean_scrub_stamp":"2024-09-16T15:49:52.725252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T00:01:18.459741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.1d","version":"134'2","reported_seq":17,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-09-16T15:49:54.750973+0000","last_change":"2024-09-16T15:49:52.759269+0000","last_active":"2024-09-16T15:49:54.750973+0000","last_peered":"2024-09-16T15:49:54.750973+0000","last_clean":"2024-09-16T15:49:54.750973+0000","last_became_active":"2024-09-16T15:49:52.758529+0000","last_became_peered":"2024-09-16T15:49:52.758529+0000","last_unstale":"2024-09-16T15:49:54.750973+0000","last_undegraded":"2024-09-16T15:49:54.750973+0000","last_fullsized":"2024-09-16T15:49:54.750973+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:49:51.721471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:49:51.721471+0000","last_clean_scrub_stamp":"2024-09-16T15:49:51.721471+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:34:39.636337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.1d","version":"0'0","reported_seq":13,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-09-16T15:49:55.015207+0000","last_change":"2024-09-16T15:49:53.801699+0000","last_active":"2024-09-16T15:49:55.015207+0000","last_peered":"2024-09-16T15:49:55.015207+0000","last_clean":"2024-09-16T15:49:55.015207+0000","last_became_active":"2024-09-16T15:49:53.801058+0000","last_became_peered":"2024-09-16T15:49:53.801058+0000","last_unstale":"2024-09-16T15:49:55.015207+0000","last_undegraded":"2024-09-16T15:49:55.015207+0000","last_fullsized":"2024-09-16T15:49:55.015207+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:49:52.725252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:49:52.725252+0000","last_clean_scrub_stamp":"2024-09-16T15:49:52.725252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T22:37:54.641296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.1c","version":"134'2","reported_seq":17,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-09-16T15:49:54.752366+0000","last_change":"2024-09-16T15:49:52.763863+0000","last_active":"2024-09-16T15:49:54.752366+0000","last_peered":"2024-09-16T15:49:54.752366+0000","last_clean":"2024-09-16T15:49:54.752366+0000","last_became_active":"2024-09-16T15:49:52.762989+0000","last_became_peered":"2024-09-16T15:49:52.762989+0000","last_unstale":"2024-09-16T15:49:54.752366+0000","last_undegraded":"2024-09-16T15:49:54.752366+0000","last_fullsized":"2024-09-16T15:49:54.752366+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:49:51.721471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:49:51.721471+0000","last_clean_scrub_stamp":"2024-09-16T15:49:51.721471+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:39:55.480968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.1a","version":"0'0","reported_seq":13,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-09-16T15:49:54.748602+0000","last_change":"2024-09-16T15:49:53.815933+0000","last_active":"2024-09-16T15:49:54.748602+0000","last_peered":"2024-09-16T15:49:54.748602+0000","last_clean":"2024-09-16T15:49:54.748602+0000","last_became_active":"2024-09-16T15:49:53.815649+0000","last_became_peered":"2024-09-16T15:49:53.815649+0000","last_unstale":"2024-09-16T15:49:54.748602+0000","last_undegraded":"2024-09-16T15:49:54.748602+0000","last_fullsized":"2024-09-16T15:49:54.748602+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:49:52.725252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:49:52.725252+0000","last_clean_scrub_stamp":"2024-09-16T15:49:52.725252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T00:02:37.101218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.1b","version":"134'1","reported_seq":16,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-09-16T15:49:55.016063+0000","last_change":"2024-09-16T15:49:52.774467+0000","last_active":"2024-09-16T15:49:55.016063+0000","last_peered":"2024-09-16T15:49:55.016063+0000","last_clean":"2024-09-16T15:49:55.016063+0000","last_became_active":"2024-09-16T15:49:52.774038+0000","last_became_peered":"2024-09-16T15:49:52.774038+0000","last_unstale":"2024-09-16T15:49:55.016063+0000","last_undegraded":"2024-09-16T15:49:55.016063+0000","last_fullsized":"2024-09-16T15:49:55.016063+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:49:51.721471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:49:51.721471+0000","last_clean_scrub_stamp":"2024-09-16T15:49:51.721471+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T18:49:34.407287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.1b","version":"0'0","reported_seq":13,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-09-16T15:49:54.750414+0000","last_change":"2024-09-16T15:49:53.806188+0000","last_active":"2024-09-16T15:49:54.750414+0000","last_peered":"2024-09-16T15:49:54.750414+0000","last_clean":"2024-09-16T15:49:54.750414+0000","last_became_active":"2024-09-16T15:49:53.805684+0000","last_became_peered":"2024-09-16T15:49:53.805684+0000","last_unstale":"2024-09-16T15:49:54.750414+0000","last_undegraded":"2024-09-16T15:49:54.750414+0000","last_fullsized":"2024-09-16T15:49:54.750414+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:49:52.725252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:49:52.725252+0000","last_clean_scrub_stamp":"2024-09-16T15:49:52.725252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:17:49.186332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"20.1a","version":"134'1","reported_seq":16,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-09-16T15:49:55.013465+0000","last_change":"2024-09-16T15:49:52.781580+0000","last_active":"2024-09-16T15:49:55.013465+0000","last_peered":"2024-09-16T15:49:55.013465+0000","last_clean":"2024-09-16T15:49:55.013465+0000","last_became_active":"2024-09-16T15:49:52.780852+0000","last_became_peered":"2024-09-16T15:49:52.780852+0000","last_unstale":"2024-09-16T15:49:55.013465+0000","last_undegraded":"2024-09-16T15:49:55.013465+0000","last_fullsized":"2024-09-16T15:49:55.013465+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:49:51.721471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:49:51.721471+0000","last_clean_scrub_stamp":"2024-09-16T15:49:51.721471+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:28:10.260758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"21.18","version":"0'0","reported_seq":13,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-09-16T15:49:54.748388+0000","last_change":"2024-09-16T15:49:53.794339+0000","last_active":"2024-09-16T15:49:54.748388+0000","last_peered":"2024-09-16T15:49:54.748388+0000","last_clean":"2024-09-16T15:49:54.748388+0000","last_became_active":"2024-09-16T15:49:53.794054+0000","last_became_peered":"2024-09-16T15:49:53.794054+0000","last_unstale":"2024-09-16T15:49:54.748388+0000","last_undegraded":"2024-09-16T15:49:54.748388+0000","last_fullsized":"2024-09-16T15:49:54.748388+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:49:52.725252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:49:52.725252+0000","last_clean_scrub_stamp":"2024-09-16T15:49:52.725252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:38:50.572143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.19","version":"0'0","reported_seq":15,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-09-16T15:49:55.015756+0000","last_change":"2024-09-16T15:49:52.781835+0000","last_active":"2024-09-16T15:49:55.015756+0000","last_peered":"2024-09-16T15:49:55.015756+0000","last_clean":"2024-09-16T15:49:55.015756+0000","last_became_active":"2024-09-16T15:49:52.781449+0000","last_became_peered":"2024-09-16T15:49:52.781449+0000","last_unstale":"2024-09-16T15:49:55.015756+0000","last_undegraded":"2024-09-16T15:49:55.015756+0000","last_fullsized":"2024-09-16T15:49:55.015756+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:49:51.721471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:49:51.721471+0000","last_clean_scrub_stamp":"2024-09-16T15:49:51.721471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T23:24:41.912346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.19","version":"0'0","reported_seq":13,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-09-16T15:49:55.013260+0000","last_change":"2024-09-16T15:49:53.784500+0000","last_active":"2024-09-16T15:49:55.013260+0000","last_peered":"2024-09-16T15:49:55.013260+0000","last_clean":"2024-09-16T15:49:55.013260+0000","last_became_active":"2024-09-16T15:49:53.782672+0000","last_became_peered":"2024-09-16T15:49:53.782672+0000","last_unstale":"2024-09-16T15:49:55.013260+0000","last_undegraded":"2024-09-16T15:49:55.013260+0000","last_fullsized":"2024-09-16T15:49:55.013260+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:49:52.725252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:49:52.725252+0000","last_clean_scrub_stamp":"2024-09-16T15:49:52.725252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T01:11:40.211819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.18","version":"0'0","reported_seq":15,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-09-16T15:49:54.750020+0000","last_change":"2024-09-16T15:49:52.778794+0000","last_active":"2024-09-16T15:49:54.750020+0000","last_peered":"2024-09-16T15:49:54.750020+0000","last_clean":"2024-09-16T15:49:54.750020+0000","last_became_active":"2024-09-16T15:49:52.778051+0000","last_became_peered":"2024-09-16T15:49:52.778051+0000","last_unstale":"2024-09-16T15:49:54.750020+0000","last_undegraded":"2024-09-16T15:49:54.750020+0000","last_fullsized":"2024-09-16T15:49:54.750020+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:49:51.721471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:49:51.721471+0000","last_clean_scrub_stamp":"2024-09-16T15:49:51.721471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T02:15:43.635025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.16","version":"0'0","reported_seq":13,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-09-16T15:49:55.014022+0000","last_change":"2024-09-16T15:49:53.785294+0000","last_active":"2024-09-16T15:49:55.014022+0000","last_peered":"2024-09-16T15:49:55.014022+0000","last_clean":"2024-09-16T15:49:55.014022+0000","last_became_active":"2024-09-16T15:49:53.784515+0000","last_became_peered":"2024-09-16T15:49:53.784515+0000","last_unstale":"2024-09-16T15:49:55.014022+0000","last_undegraded":"2024-09-16T15:49:55.014022+0000","last_fullsized":"2024-09-16T15:49:55.014022+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:49:52.725252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:49:52.725252+0000","last_clean_scrub_stamp":"2024-09-16T15:49:52.725252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:15:27.170635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.17","version":"0'0","reported_seq":15,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-09-16T15:49:54.749258+0000","last_change":"2024-09-16T15:49:52.769159+0000","last_active":"2024-09-16T15:49:54.749258+0000","last_peered":"2024-09-16T15:49:54.749258+0000","last_clean":"2024-09-16T15:49:54.749258+0000","last_became_active":"2024-09-16T15:49:52.768366+0000","last_became_peered":"2024-09-16T15:49:52.768366+0000","last_unstale":"2024-09-16T15:49:54.749258+0000","last_undegraded":"2024-09-16T15:49:54.749258+0000","last_fullsized":"2024-09-16T15:49:54.749258+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:49:51.721471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:49:51.721471+0000","last_clean_scrub_stamp":"2024-09-16T15:49:51.721471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:36:59.572984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.17","version":"0'0","reported_seq":13,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-09-16T15:49:55.015378+0000","last_change":"2024-09-16T15:49:53.798297+0000","last_active":"2024-09-16T15:49:55.015378+0000","last_peered":"2024-09-16T15:49:55.015378+0000","last_clean":"2024-09-16T15:49:55.015378+0000","last_became_active":"2024-09-16T15:49:53.796876+0000","last_became_peered":"2024-09-16T15:49:53.796876+0000","last_unstale":"2024-09-16T15:49:55.015378+0000","last_undegraded":"2024-09-16T15:49:55.015378+0000","last_fullsized":"2024-09-16T15:49:55.015378+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:49:52.725252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:49:52.725252+0000","last_clean_scrub_stamp":"2024-09-16T15:49:52.725252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:59:47.880031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.16","version":"0'0","reported_seq":15,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-09-16T15:49:55.016139+0000","last_change":"2024-09-16T15:49:52.782354+0000","last_active":"2024-09-16T15:49:55.016139+0000","last_peered":"2024-09-16T15:49:55.016139+0000","last_clean":"2024-09-16T15:49:55.016139+0000","last_became_active":"2024-09-16T15:49:52.781604+0000","last_became_peered":"2024-09-16T15:49:52.781604+0000","last_unstale":"2024-09-16T15:49:55.016139+0000","last_undegraded":"2024-09-16T15:49:55.016139+0000","last_fullsized":"2024-09-16T15:49:55.016139+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:49:51.721471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:49:51.721471+0000","last_clean_scrub_stamp":"2024-09-16T15:49:51.721471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T23:50:37.482485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.15","version":"0'0","reported_seq":13,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-09-16T15:49:55.017525+0000","last_change":"2024-09-16T15:49:53.761206+0000","last_active":"2024-09-16T15:49:55.017525+0000","last_peered":"2024-09-16T15:49:55.017525+0000","last_clean":"2024-09-16T15:49:55.017525+0000","last_became_active":"2024-09-16T15:49:53.760908+0000","last_became_peered":"2024-09-16T15:49:53.760908+0000","last_unstale":"2024-09-16T15:49:55.017525+0000","last_undegraded":"2024-09-16T15:49:55.017525+0000","last_fullsized":"2024-09-16T15:49:55.017525+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:49:52.725252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:49:52.725252+0000","last_clean_scrub_stamp":"2024-09-16T15:49:52.725252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T22:54:53.849709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.14","version":"134'6","reported_seq":21,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-09-16T15:49:55.018014+0000","last_change":"2024-09-16T15:49:52.762448+0000","last_active":"2024-09-16T15:49:55.018014+0000","last_peered":"2024-09-16T15:49:55.018014+0000","last_clean":"2024-09-16T15:49:55.018014+0000","last_became_active":"2024-09-16T15:49:52.761695+0000","last_became_peered":"2024-09-16T15:49:52.761695+0000","last_unstale":"2024-09-16T15:49:55.018014+0000","last_undegraded":"2024-09-16T15:49:55.018014+0000","last_fullsized":"2024-09-16T15:49:55.018014+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:49:51.721471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:49:51.721471+0000","last_clean_scrub_stamp":"2024-09-16T15:49:51.721471+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T00:28:15.684923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"21.12","version":"0'0","reported_seq":13,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-09-16T15:49:54.750130+0000","last_change":"2024-09-16T15:49:53.806120+0000","last_active":"2024-09-16T15:49:54.750130+0000","last_peered":"2024-09-16T15:49:54.750130+0000","last_clean":"2024-09-16T15:49:54.750130+0000","last_became_active":"2024-09-16T15:49:53.805381+0000","last_became_peered":"2024-09-16T15:49:53.805381+0000","last_unstale":"2024-09-16T15:49:54.750130+0000","last_undegraded":"2024-09-16T15:49:54.750130+0000","last_fullsized":"2024-09-16T15:49:54.750130+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:49:52.725252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:49:52.725252+0000","last_clean_scrub_stamp":"2024-09-16T15:49:52.725252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:11:15.664173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.13","version":"134'2","reported_seq":17,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-09-16T15:49:54.752944+0000","last_change":"2024-09-16T15:49:52.774793+0000","last_active":"2024-09-16T15:49:54.752944+0000","last_peered":"2024-09-16T15:49:54.752944+0000","last_clean":"2024-09-16T15:49:54.752944+0000","last_became_active":"2024-09-16T15:49:52.774116+0000","last_became_peered":"2024-09-16T15:49:52.774116+0000","last_unstale":"2024-09-16T15:49:54.752944+0000","last_undegraded":"2024-09-16T15:49:54.752944+0000","last_fullsized":"2024-09-16T15:49:54.752944+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:49:51.721471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:49:51.721471+0000","last_clean_scrub_stamp":"2024-09-16T15:49:51.721471+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T23:49:56.154646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"21.13","version":"0'0","reported_seq":13,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-09-16T15:49:54.748633+0000","last_change":"2024-09-16T15:49:53.772319+0000","last_active":"2024-09-16T15:49:54.748633+0000","last_peered":"2024-09-16T15:49:54.748633+0000","last_clean":"2024-09-16T15:49:54.748633+0000","last_became_active":"2024-09-16T15:49:53.767119+0000","last_became_peered":"2024-09-16T15:49:53.767119+0000","last_unstale":"2024-09-16T15:49:54.748633+0000","last_undegraded":"2024-09-16T15:49:54.748633+0000","last_fullsized":"2024-09-16T15:49:54.748633+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:49:52.725252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:49:52.725252+0000","last_clean_scrub_stamp":"2024-09-16T15:49:52.725252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T22:28:29.014751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.12","version":"0'0","reported_seq":15,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-09-16T15:49:54.750536+0000","last_change":"2024-09-16T15:49:52.785819+0000","last_active":"2024-09-16T15:49:54.750536+0000","last_peered":"2024-09-16T15:49:54.750536+0000","last_clean":"2024-09-16T15:49:54.750536+0000","last_became_active":"2024-09-16T15:49:52.785063+0000","last_became_peered":"2024-09-16T15:49:52.785063+0000","last_unstale":"2024-09-16T15:49:54.750536+0000","last_undegraded":"2024-09-16T15:49:54.750536+0000","last_fullsized":"2024-09-16T15:49:54.750536+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:49:51.721471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:49:51.721471+0000","last_clean_scrub_stamp":"2024-09-16T15:49:51.721471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T02:37:28.590611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"21.10","version":"0'0","reported_seq":13,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-09-16T15:49:55.013330+0000","last_change":"2024-09-16T15:49:53.795683+0000","last_active":"2024-09-16T15:49:55.013330+0000","last_peered":"2024-09-16T15:49:55.013330+0000","last_clean":"2024-09-16T15:49:55.013330+0000","last_became_active":"2024-09-16T15:49:53.791581+0000","last_became_peered":"2024-09-16T15:49:53.791581+0000","last_unstale":"2024-09-16T15:49:55.013330+0000","last_undegraded":"2024-09-16T15:49:55.013330+0000","last_fullsized":"2024-09-16T15:49:55.013330+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:49:52.725252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:49:52.725252+0000","last_clean_scrub_stamp":"2024-09-16T15:49:52.725252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:43:35.941656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.11","version":"0'0","reported_seq":15,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-09-16T15:49:55.015152+0000","last_change":"2024-09-16T15:49:52.757579+0000","last_active":"2024-09-16T15:49:55.015152+0000","last_peered":"2024-09-16T15:49:55.015152+0000","last_clean":"2024-09-16T15:49:55.015152+0000","last_became_active":"2024-09-16T15:49:52.757085+0000","last_became_peered":"2024-09-16T15:49:52.757085+0000","last_unstale":"2024-09-16T15:49:55.015152+0000","last_undegraded":"2024-09-16T15:49:55.015152+0000","last_fullsized":"2024-09-16T15:49:55.015152+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:49:51.721471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:49:51.721471+0000","last_clean_scrub_stamp":"2024-09-16T15:49:51.721471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T00:53:44.706141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.11","version":"0'0","reported_seq":13,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-09-16T15:49:54.751549+0000","last_change":"2024-09-16T15:49:53.807125+0000","last_active":"2024-09-16T15:49:54.751549+0000","last_peered":"2024-09-16T15:49:54.751549+0000","last_clean":"2024-09-16T15:49:54.751549+0000","last_became_active":"2024-09-16T15:49:53.805178+0000","last_became_peered":"2024-09-16T15:49:53.805178+0000","last_unstale":"2024-09-16T15:49:54.751549+0000","last_undegraded":"2024-09-16T15:49:54.751549+0000","last_fullsized":"2024-09-16T15:49:54.751549+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:49:52.725252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:49:52.725252+0000","last_clean_scrub_stamp":"2024-09-16T15:49:52.725252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:53:44.753622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.10","version":"134'8","reported_seq":23,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-09-16T15:49:55.016256+0000","last_change":"2024-09-16T15:49:52.766519+0000","last_active":"2024-09-16T15:49:55.016256+0000","last_peered":"2024-09-16T15:49:55.016256+0000","last_clean":"2024-09-16T15:49:55.016256+0000","last_became_active":"2024-09-16T15:49:52.766156+0000","last_became_peered":"2024-09-16T15:49:52.766156+0000","last_unstale":"2024-09-16T15:49:55.016256+0000","last_undegraded":"2024-09-16T15:49:55.016256+0000","last_fullsized":"2024-09-16T15:49:55.016256+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:49:51.721471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:49:51.721471+0000","last_clean_scrub_stamp":"2024-09-16T15:49:51.721471+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T22:50:26.176905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.e","version":"0'0","reported_seq":13,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-09-16T15:49:55.016284+0000","last_change":"2024-09-16T15:49:53.795034+0000","last_active":"2024-09-16T15:49:55.016284+0000","last_peered":"2024-09-16T15:49:55.016284+0000","last_clean":"2024-09-16T15:49:55.016284+0000","last_became_active":"2024-09-16T15:49:53.794234+0000","last_became_peered":"2024-09-16T15:49:53.794234+0000","last_unstale":"2024-09-16T15:49:55.016284+0000","last_undegraded":"2024-09-16T15:49:55.016284+0000","last_fullsized":"2024-09-16T15:49:55.016284+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:49:52.725252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:49:52.725252+0000","last_clean_scrub_stamp":"2024-09-16T15:49:52.725252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:52:42.730798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.f","version":"134'1","reported_seq":16,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-09-16T15:49:55.013653+0000","last_change":"2024-09-16T15:49:52.784454+0000","last_active":"2024-09-16T15:49:55.013653+0000","last_peered":"2024-09-16T15:49:55.013653+0000","last_clean":"2024-09-16T15:49:55.013653+0000","last_became_active":"2024-09-16T15:49:52.783843+0000","last_became_peered":"2024-09-16T15:49:52.783843+0000","last_unstale":"2024-09-16T15:49:55.013653+0000","last_undegraded":"2024-09-16T15:49:55.013653+0000","last_fullsized":"2024-09-16T15:49:55.013653+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:49:51.721471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:49:51.721471+0000","last_clean_scrub_stamp":"2024-09-16T15:49:51.721471+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:22:04.602157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"21.f","version":"0'0","reported_seq":13,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-09-16T15:49:55.017447+0000","last_change":"2024-09-16T15:49:53.782494+0000","last_active":"2024-09-16T15:49:55.017447+0000","last_peered":"2024-09-16T15:49:55.017447+0000","last_clean":"2024-09-16T15:49:55.017447+0000","last_became_active":"2024-09-16T15:49:53.782211+0000","last_became_peered":"2024-09-16T15:49:53.782211+0000","last_unstale":"2024-09-16T15:49:55.017447+0000","last_undegraded":"2024-09-16T15:49:55.017447+0000","last_fullsized":"2024-09-16T15:49:55.017447+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:49:52.725252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:49:52.725252+0000","last_clean_scrub_stamp":"2024-09-16T15:49:52.725252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T18:27:22.512705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.e","version":"134'2","reported_seq":17,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-09-16T15:49:54.749430+0000","last_change":"2024-09-16T15:49:52.774535+0000","last_active":"2024-09-16T15:49:54.749430+0000","last_peered":"2024-09-16T15:49:54.749430+0000","last_clean":"2024-09-16T15:49:54.749430+0000","last_became_active":"2024-09-16T15:49:52.774230+0000","last_became_peered":"2024-09-16T15:49:52.774230+0000","last_unstale":"2024-09-16T15:49:54.749430+0000","last_undegraded":"2024-09-16T15:49:54.749430+0000","last_fullsized":"2024-09-16T15:49:54.749430+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:49:51.721471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:49:51.721471+0000","last_clean_scrub_stamp":"2024-09-16T15:49:51.721471+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:06:41.060737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.c","version":"0'0","reported_seq":13,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-09-16T15:49:54.752223+0000","last_change":"2024-09-16T15:49:53.805839+0000","last_active":"2024-09-16T15:49:54.752223+0000","last_peered":"2024-09-16T15:49:54.752223+0000","last_clean":"2024-09-16T15:49:54.752223+0000","last_became_active":"2024-09-16T15:49:53.805178+0000","last_became_peered":"2024-09-16T15:49:53.805178+0000","last_unstale":"2024-09-16T15:49:54.752223+0000","last_undegraded":"2024-09-16T15:49:54.752223+0000","last_fullsized":"2024-09-16T15:49:54.752223+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:49:52.725252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:49:52.725252+0000","last_clean_scrub_stamp":"2024-09-16T15:49:52.725252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:51:57.391541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.d","version":"134'1","reported_seq":16,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-09-16T15:49:55.017991+0000","last_change":"2024-09-16T15:49:52.773668+0000","last_active":"2024-09-16T15:49:55.017991+0000","last_peered":"2024-09-16T15:49:55.017991+0000","last_clean":"2024-09-16T15:49:55.017991+0000","last_became_active":"2024-09-16T15:49:52.773184+0000","last_became_peered":"2024-09-16T15:49:52.773184+0000","last_unstale":"2024-09-16T15:49:55.017991+0000","last_undegraded":"2024-09-16T15:49:55.017991+0000","last_fullsized":"2024-09-16T15:49:55.017991+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:49:51.721471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:49:51.721471+0000","last_clean_scrub_stamp":"2024-09-16T15:49:51.721471+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:30:28.622437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.d","version":"0'0","reported_seq":13,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-09-16T15:49:55.016400+0000","last_change":"2024-09-16T15:49:53.809332+0000","last_active":"2024-09-16T15:49:55.016400+0000","last_peered":"2024-09-16T15:49:55.016400+0000","last_clean":"2024-09-16T15:49:55.016400+0000","last_became_active":"2024-09-16T15:49:53.809054+0000","last_became_peered":"2024-09-16T15:49:53.809054+0000","last_unstale":"2024-09-16T15:49:55.016400+0000","last_undegraded":"2024-09-16T15:49:55.016400+0000","last_fullsized":"2024-09-16T15:49:55.016400+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:49:52.725252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:49:52.725252+0000","last_clean_scrub_stamp":"2024-09-16T15:49:52.725252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T01:21:55.832298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.c","version":"0'0","reported_seq":15,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-09-16T15:49:54.748349+0000","last_change":"2024-09-16T15:49:52.768132+0000","last_active":"2024-09-16T15:49:54.748349+0000","last_peered":"2024-09-16T15:49:54.748349+0000","last_clean":"2024-09-16T15:49:54.748349+0000","last_became_active":"2024-09-16T15:49:52.767738+0000","last_became_peered":"2024-09-16T15:49:52.767738+0000","last_unstale":"2024-09-16T15:49:54.748349+0000","last_undegraded":"2024-09-16T15:49:54.748349+0000","last_fullsized":"2024-09-16T15:49:54.748349+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:49:51.721471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:49:51.721471+0000","last_clean_scrub_stamp":"2024-09-16T15:49:51.721471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T03:32:13.600777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.a","version":"0'0","reported_seq":13,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-09-16T15:49:54.751189+0000","last_change":"2024-09-16T15:49:53.757825+0000","last_active":"2024-09-16T15:49:54.751189+0000","last_peered":"2024-09-16T15:49:54.751189+0000","last_clean":"2024-09-16T15:49:54.751189+0000","last_became_active":"2024-09-16T15:49:53.756978+0000","last_became_peered":"2024-09-16T15:49:53.756978+0000","last_unstale":"2024-09-16T15:49:54.751189+0000","last_undegraded":"2024-09-16T15:49:54.751189+0000","last_fullsized":"2024-09-16T15:49:54.751189+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:49:52.725252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:49:52.725252+0000","last_clean_scrub_stamp":"2024-09-16T15:49:52.725252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T22:28:29.014751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"20.b","version":"134'2","reported_seq":17,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-09-16T15:49:55.017698+0000","last_change":"2024-09-16T15:49:52.770565+0000","last_active":"2024-09-16T15:49:55.017698+0000","last_peered":"2024-09-16T15:49:55.017698+0000","last_clean":"2024-09-16T15:49:55.017698+0000","last_became_active":"2024-09-16T15:49:52.770244+0000","last_became_peered":"2024-09-16T15:49:52.770244+0000","last_unstale":"2024-09-16T15:49:55.017698+0000","last_undegraded":"2024-09-16T15:49:55.017698+0000","last_fullsized":"2024-09-16T15:49:55.017698+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:49:51.721471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:49:51.721471+0000","last_clean_scrub_stamp":"2024-09-16T15:49:51.721471+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T22:38:55.209721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"21.b","version":"0'0","reported_seq":13,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-09-16T15:49:55.017231+0000","last_change":"2024-09-16T15:49:53.777929+0000","last_active":"2024-09-16T15:49:55.017231+0000","last_peered":"2024-09-16T15:49:55.017231+0000","last_clean":"2024-09-16T15:49:55.017231+0000","last_became_active":"2024-09-16T15:49:53.777689+0000","last_became_peered":"2024-09-16T15:49:53.777689+0000","last_unstale":"2024-09-16T15:49:55.017231+0000","last_undegraded":"2024-09-16T15:49:55.017231+0000","last_fullsized":"2024-09-16T15:49:55.017231+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:49:52.725252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:49:52.725252+0000","last_clean_scrub_stamp":"2024-09-16T15:49:52.725252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:40:39.342110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"20.a","version":"0'0","reported_seq":15,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-09-16T15:49:55.015205+0000","last_change":"2024-09-16T15:49:52.771440+0000","last_active":"2024-09-16T15:49:55.015205+0000","last_peered":"2024-09-16T15:49:55.015205+0000","last_clean":"2024-09-16T15:49:55.015205+0000","last_became_active":"2024-09-16T15:49:52.770609+0000","last_became_peered":"2024-09-16T15:49:52.770609+0000","last_unstale":"2024-09-16T15:49:55.015205+0000","last_undegraded":"2024-09-16T15:49:55.015205+0000","last_fullsized":"2024-09-16T15:49:55.015205+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:49:51.721471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:49:51.721471+0000","last_clean_scrub_stamp":"2024-09-16T15:49:51.721471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T22:28:38.317070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.8","version":"0'0","reported_seq":13,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-09-16T15:49:54.749697+0000","last_change":"2024-09-16T15:49:53.763650+0000","last_active":"2024-09-16T15:49:54.749697+0000","last_peered":"2024-09-16T15:49:54.749697+0000","last_clean":"2024-09-16T15:49:54.749697+0000","last_became_active":"2024-09-16T15:49:53.762879+0000","last_became_peered":"2024-09-16T15:49:53.762879+0000","last_unstale":"2024-09-16T15:49:54.749697+0000","last_undegraded":"2024-09-16T15:49:54.749697+0000","last_fullsized":"2024-09-16T15:49:54.749697+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:49:52.725252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:49:52.725252+0000","last_clean_scrub_stamp":"2024-09-16T15:49:52.725252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:00:52.258152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"20.9","version":"0'0","reported_seq":15,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-09-16T15:49:55.014517+0000","last_change":"2024-09-16T15:49:52.771244+0000","last_active":"2024-09-16T15:49:55.014517+0000","last_peered":"2024-09-16T15:49:55.014517+0000","last_clean":"2024-09-16T15:49:55.014517+0000","last_became_active":"2024-09-16T15:49:52.770510+0000","last_became_peered":"2024-09-16T15:49:52.770510+0000","last_unstale":"2024-09-16T15:49:55.014517+0000","last_undegraded":"2024-09-16T15:49:55.014517+0000","last_fullsized":"2024-09-16T15:49:55.014517+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:49:51.721471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:49:51.721471+0000","last_clean_scrub_stamp":"2024-09-16T15:49:51.721471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:59:48.118024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.9","version":"0'0","reported_seq":13,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-09-16T15:49:55.015523+0000","last_change":"2024-09-16T15:49:53.774586+0000","last_active":"2024-09-16T15:49:55.015523+0000","last_peered":"2024-09-16T15:49:55.015523+0000","last_clean":"2024-09-16T15:49:55.015523+0000","last_became_active":"2024-09-16T15:49:53.774211+0000","last_became_peered":"2024-09-16T15:49:53.774211+0000","last_unstale":"2024-09-16T15:49:55.015523+0000","last_undegraded":"2024-09-16T15:49:55.015523+0000","last_fullsized":"2024-09-16T15:49:55.015523+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:49:52.725252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:49:52.725252+0000","last_clean_scrub_stamp":"2024-09-16T15:49:52.725252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T03:33:15.612475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"20.8","version":"0'0","reported_seq":15,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-09-16T15:49:54.749002+0000","last_change":"2024-09-16T15:49:52.777568+0000","last_active":"2024-09-16T15:49:54.749002+0000","last_peered":"2024-09-16T15:49:54.749002+0000","last_clean":"2024-09-16T15:49:54.749002+0000","last_became_active":"2024-09-16T15:49:52.776903+0000","last_became_peered":"2024-09-16T15:49:52.776903+0000","last_unstale":"2024-09-16T15:49:54.749002+0000","last_undegraded":"2024-09-16T15:49:54.749002+0000","last_fullsized":"2024-09-16T15:49:54.749002+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:49:51.721471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:49:51.721471+0000","last_clean_scrub_stamp":"2024-09-16T15:49:51.721471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T23:11:07.659450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"21.6","version":"0'0","reported_seq":13,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-09-16T15:49:55.014578+0000","last_change":"2024-09-16T15:49:53.804004+0000","last_active":"2024-09-16T15:49:55.014578+0000","last_peered":"2024-09-16T15:49:55.014578+0000","last_clean":"2024-09-16T15:49:55.014578+0000","last_became_active":"2024-09-16T15:49:53.803397+0000","last_became_peered":"2024-09-16T15:49:53.803397+0000","last_unstale":"2024-09-16T15:49:55.014578+0000","last_undegraded":"2024-09-16T15:49:55.014578+0000","last_fullsized":"2024-09-16T15:49:55.014578+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:49:52.725252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:49:52.725252+0000","last_clean_scrub_stamp":"2024-09-16T15:49:52.725252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T18:28:28.606171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.7","version":"134'2","reported_seq":17,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-09-16T15:49:55.015849+0000","last_change":"2024-09-16T15:49:52.775976+0000","last_active":"2024-09-16T15:49:55.015849+0000","last_peered":"2024-09-16T15:49:55.015849+0000","last_clean":"2024-09-16T15:49:55.015849+0000","last_became_active":"2024-09-16T15:49:52.775249+0000","last_became_peered":"2024-09-16T15:49:52.775249+0000","last_unstale":"2024-09-16T15:49:55.015849+0000","last_undegraded":"2024-09-16T15:49:55.015849+0000","last_fullsized":"2024-09-16T15:49:55.015849+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:49:51.721471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:49:51.721471+0000","last_clean_scrub_stamp":"2024-09-16T15:49:51.721471+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:57:08.808070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.7","version":"0'0","reported_seq":13,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-09-16T15:49:54.753391+0000","last_change":"2024-09-16T15:49:53.806912+0000","last_active":"2024-09-16T15:49:54.753391+0000","last_peered":"2024-09-16T15:49:54.753391+0000","last_clean":"2024-09-16T15:49:54.753391+0000","last_became_active":"2024-09-16T15:49:53.805374+0000","last_became_peered":"2024-09-16T15:49:53.805374+0000","last_unstale":"2024-09-16T15:49:54.753391+0000","last_undegraded":"2024-09-16T15:49:54.753391+0000","last_fullsized":"2024-09-16T15:49:54.753391+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:49:52.725252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:49:52.725252+0000","last_clean_scrub_stamp":"2024-09-16T15:49:52.725252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T00:21:39.029022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.6","version":"0'0","reported_seq":15,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-09-16T15:49:55.016280+0000","last_change":"2024-09-16T15:49:52.774453+0000","last_active":"2024-09-16T15:49:55.016280+0000","last_peered":"2024-09-16T15:49:55.016280+0000","last_clean":"2024-09-16T15:49:55.016280+0000","last_became_active":"2024-09-16T15:49:52.773886+0000","last_became_peered":"2024-09-16T15:49:52.773886+0000","last_unstale":"2024-09-16T15:49:55.016280+0000","last_undegraded":"2024-09-16T15:49:55.016280+0000","last_fullsized":"2024-09-16T15:49:55.016280+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:49:51.721471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:49:51.721471+0000","last_clean_scrub_stamp":"2024-09-16T15:49:51.721471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:16:42.423423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.4","version":"0'0","reported_seq":13,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-09-16T15:49:54.748622+0000","last_change":"2024-09-16T15:49:53.783444+0000","last_active":"2024-09-16T15:49:54.748622+0000","last_peered":"2024-09-16T15:49:54.748622+0000","last_clean":"2024-09-16T15:49:54.748622+0000","last_became_active":"2024-09-16T15:49:53.782851+0000","last_became_peered":"2024-09-16T15:49:53.782851+0000","last_unstale":"2024-09-16T15:49:54.748622+0000","last_undegraded":"2024-09-16T15:49:54.748622+0000","last_fullsized":"2024-09-16T15:49:54.748622+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:49:52.725252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:49:52.725252+0000","last_clean_scrub_stamp":"2024-09-16T15:49:52.725252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T00:21:32.615337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.5","version":"0'0","reported_seq":15,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-09-16T15:49:55.017773+0000","last_change":"2024-09-16T15:49:52.757797+0000","last_active":"2024-09-16T15:49:55.017773+0000","last_peered":"2024-09-16T15:49:55.017773+0000","last_clean":"2024-09-16T15:49:55.017773+0000","last_became_active":"2024-09-16T15:49:52.757406+0000","last_became_peered":"2024-09-16T15:49:52.757406+0000","last_unstale":"2024-09-16T15:49:55.017773+0000","last_undegraded":"2024-09-16T15:49:55.017773+0000","last_fullsized":"2024-09-16T15:49:55.017773+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:49:51.721471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:49:51.721471+0000","last_clean_scrub_stamp":"2024-09-16T15:49:51.721471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T18:55:22.565226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.5","version":"0'0","reported_seq":13,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-09-16T15:49:55.016967+0000","last_change":"2024-09-16T15:49:53.760718+0000","last_active":"2024-09-16T15:49:55.016967+0000","last_peered":"2024-09-16T15:49:55.016967+0000","last_clean":"2024-09-16T15:49:55.016967+0000","last_became_active":"2024-09-16T15:49:53.760349+0000","last_became_peered":"2024-09-16T15:49:53.760349+0000","last_unstale":"2024-09-16T15:49:55.016967+0000","last_undegraded":"2024-09-16T15:49:55.016967+0000","last_fullsized":"2024-09-16T15:49:55.016967+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:49:52.725252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:49:52.725252+0000","last_clean_scrub_stamp":"2024-09-16T15:49:52.725252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T23:34:38.240478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.4","version":"0'0","reported_seq":15,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-09-16T15:49:55.016179+0000","last_change":"2024-09-16T15:49:52.768224+0000","last_active":"2024-09-16T15:49:55.016179+0000","last_peered":"2024-09-16T15:49:55.016179+0000","last_clean":"2024-09-16T15:49:55.016179+0000","last_became_active":"2024-09-16T15:49:52.767916+0000","last_became_peered":"2024-09-16T15:49:52.767916+0000","last_unstale":"2024-09-16T15:49:55.016179+0000","last_undegraded":"2024-09-16T15:49:55.016179+0000","last_fullsized":"2024-09-16T15:49:55.016179+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:49:51.721471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:49:51.721471+0000","last_clean_scrub_stamp":"2024-09-16T15:49:51.721471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T23:56:11.721811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.2","version":"0'0","reported_seq":13,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-09-16T15:49:55.016164+0000","last_change":"2024-09-16T15:49:53.775623+0000","last_active":"2024-09-16T15:49:55.016164+0000","last_peered":"2024-09-16T15:49:55.016164+0000","last_clean":"2024-09-16T15:49:55.016164+0000","last_became_active":"2024-09-16T15:49:53.775132+0000","last_became_peered":"2024-09-16T15:49:53.775132+0000","last_unstale":"2024-09-16T15:49:55.016164+0000","last_undegraded":"2024-09-16T15:49:55.016164+0000","last_fullsized":"2024-09-16T15:49:55.016164+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:49:52.725252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:49:52.725252+0000","last_clean_scrub_stamp":"2024-09-16T15:49:52.725252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T00:11:48.212471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.3","version":"0'0","reported_seq":15,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-09-16T15:49:55.017859+0000","last_change":"2024-09-16T15:49:52.757448+0000","last_active":"2024-09-16T15:49:55.017859+0000","last_peered":"2024-09-16T15:49:55.017859+0000","last_clean":"2024-09-16T15:49:55.017859+0000","last_became_active":"2024-09-16T15:49:52.755623+0000","last_became_peered":"2024-09-16T15:49:52.755623+0000","last_unstale":"2024-09-16T15:49:55.017859+0000","last_undegraded":"2024-09-16T15:49:55.017859+0000","last_fullsized":"2024-09-16T15:49:55.017859+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:49:51.721471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:49:51.721471+0000","last_clean_scrub_stamp":"2024-09-16T15:49:51.721471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T03:42:03.536282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"21.3","version":"0'0","reported_seq":13,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-09-16T15:49:55.014186+0000","last_change":"2024-09-16T15:49:53.814161+0000","last_active":"2024-09-16T15:49:55.014186+0000","last_peered":"2024-09-16T15:49:55.014186+0000","last_clean":"2024-09-16T15:49:55.014186+0000","last_became_active":"2024-09-16T15:49:53.813475+0000","last_became_peered":"2024-09-16T15:49:53.813475+0000","last_unstale":"2024-09-16T15:49:55.014186+0000","last_undegraded":"2024-09-16T15:49:55.014186+0000","last_fullsized":"2024-09-16T15:49:55.014186+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:49:52.725252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:49:52.725252+0000","last_clean_scrub_stamp":"2024-09-16T15:49:52.725252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T23:23:59.844906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.2","version":"0'0","reported_seq":15,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-09-16T15:49:55.015118+0000","last_change":"2024-09-16T15:49:52.767818+0000","last_active":"2024-09-16T15:49:55.015118+0000","last_peered":"2024-09-16T15:49:55.015118+0000","last_clean":"2024-09-16T15:49:55.015118+0000","last_became_active":"2024-09-16T15:49:52.765724+0000","last_became_peered":"2024-09-16T15:49:52.765724+0000","last_unstale":"2024-09-16T15:49:55.015118+0000","last_undegraded":"2024-09-16T15:49:55.015118+0000","last_fullsized":"2024-09-16T15:49:55.015118+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:49:51.721471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:49:51.721471+0000","last_clean_scrub_stamp":"2024-09-16T15:49:51.721471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:52:33.088661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.0","version":"0'0","reported_seq":13,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-09-16T15:49:55.015432+0000","last_change":"2024-09-16T15:49:53.786223+0000","last_active":"2024-09-16T15:49:55.015432+0000","last_peered":"2024-09-16T15:49:55.015432+0000","last_clean":"2024-09-16T15:49:55.015432+0000","last_became_active":"2024-09-16T15:49:53.785450+0000","last_became_peered":"2024-09-16T15:49:53.785450+0000","last_unstale":"2024-09-16T15:49:55.015432+0000","last_undegraded":"2024-09-16T15:49:55.015432+0000","last_fullsized":"2024-09-16T15:49:55.015432+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:49:52.725252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:49:52.725252+0000","last_clean_scrub_stamp":"2024-09-16T15:49:52.725252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T22:50:14.368364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.1","version":"0'0","reported_seq":15,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-09-16T15:49:54.751466+0000","last_change":"2024-09-16T15:49:52.774681+0000","last_active":"2024-09-16T15:49:54.751466+0000","last_peered":"2024-09-16T15:49:54.751466+0000","last_clean":"2024-09-16T15:49:54.751466+0000","last_became_active":"2024-09-16T15:49:52.773868+0000","last_became_peered":"2024-09-16T15:49:52.773868+0000","last_unstale":"2024-09-16T15:49:54.751466+0000","last_undegraded":"2024-09-16T15:49:54.751466+0000","last_fullsized":"2024-09-16T15:49:54.751466+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:49:51.721471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:49:51.721471+0000","last_clean_scrub_stamp":"2024-09-16T15:49:51.721471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:36:12.259846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"21.1","version":"0'0","reported_seq":13,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-09-16T15:49:55.013352+0000","last_change":"2024-09-16T15:49:53.791806+0000","last_active":"2024-09-16T15:49:55.013352+0000","last_peered":"2024-09-16T15:49:55.013352+0000","last_clean":"2024-09-16T15:49:55.013352+0000","last_became_active":"2024-09-16T15:49:53.783641+0000","last_became_peered":"2024-09-16T15:49:53.783641+0000","last_unstale":"2024-09-16T15:49:55.013352+0000","last_undegraded":"2024-09-16T15:49:55.013352+0000","last_fullsized":"2024-09-16T15:49:55.013352+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:49:52.725252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:49:52.725252+0000","last_clean_scrub_stamp":"2024-09-16T15:49:52.725252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T02:13:51.898581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.0","version":"0'0","reported_seq":15,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-09-16T15:49:54.747977+0000","last_change":"2024-09-16T15:49:52.779185+0000","last_active":"2024-09-16T15:49:54.747977+0000","last_peered":"2024-09-16T15:49:54.747977+0000","last_clean":"2024-09-16T15:49:54.747977+0000","last_became_active":"2024-09-16T15:49:52.778952+0000","last_became_peered":"2024-09-16T15:49:52.778952+0000","last_unstale":"2024-09-16T15:49:54.747977+0000","last_undegraded":"2024-09-16T15:49:54.747977+0000","last_fullsized":"2024-09-16T15:49:54.747977+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:49:51.721471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:49:51.721471+0000","last_clean_scrub_stamp":"2024-09-16T15:49:51.721471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:39:55.480968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.14","version":"0'0","reported_seq":13,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-09-16T15:49:55.017797+0000","last_change":"2024-09-16T15:49:53.774718+0000","last_active":"2024-09-16T15:49:55.017797+0000","last_peered":"2024-09-16T15:49:55.017797+0000","last_clean":"2024-09-16T15:49:55.017797+0000","last_became_active":"2024-09-16T15:49:53.774232+0000","last_became_peered":"2024-09-16T15:49:53.774232+0000","last_unstale":"2024-09-16T15:49:55.017797+0000","last_undegraded":"2024-09-16T15:49:55.017797+0000","last_fullsized":"2024-09-16T15:49:55.017797+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:49:52.725252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:49:52.725252+0000","last_clean_scrub_stamp":"2024-09-16T15:49:52.725252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T01:08:23.372694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"20.15","version":"134'3","reported_seq":18,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-09-16T15:49:54.752729+0000","last_change":"2024-09-16T15:49:52.775063+0000","last_active":"2024-09-16T15:49:54.752729+0000","last_peered":"2024-09-16T15:49:54.752729+0000","last_clean":"2024-09-16T15:49:54.752729+0000","last_became_active":"2024-09-16T15:49:52.774426+0000","last_became_peered":"2024-09-16T15:49:52.774426+0000","last_unstale":"2024-09-16T15:49:54.752729+0000","last_undegraded":"2024-09-16T15:49:54.752729+0000","last_fullsized":"2024-09-16T15:49:54.752729+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:49:51.721471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:49:51.721471+0000","last_clean_scrub_stamp":"2024-09-16T15:49:51.721471+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:00:27.297007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"73'79","reported_seq":349,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-09-16T15:49:55.014545+0000","last_change":"2024-09-16T15:42:40.881820+0000","last_active":"2024-09-16T15:49:55.014545+0000","last_peered":"2024-09-16T15:49:55.014545+0000","last_clean":"2024-09-16T15:49:55.014545+0000","last_became_active":"2024-09-16T15:42:40.879965+0000","last_became_peered":"2024-09-16T15:42:40.879965+0000","last_unstale":"2024-09-16T15:49:55.014545+0000","last_undegraded":"2024-09-16T15:49:55.014545+0000","last_fullsized":"2024-09-16T15:49:55.014545+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:42:39.728762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:42:39.728762+0000","last_clean_scrub_stamp":"2024-09-16T15:42:39.728762+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:22:18.768932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":21,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":20,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79,"ondisk_log_size":79,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542235,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33416,"kb_used_data":2224,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338424,"statfs":{"total":96636764160,"available":96602546176,"internally_reserved":0,"allocated":2277376,"data_stored":1333070,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542234,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32840,"kb_used_data":1648,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339000,"statfs":{"total":96636764160,"available":96603136000,"internally_reserved":0,"allocated":1687552,"data_stored":742196,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":15,"apply_latency_ms":15,"commit_latency_ns":15000000,"apply_latency_ns":15000000},"alerts":[]},{"osd":5,"up_from":14,"seq":60129542235,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32828,"kb_used_data":1640,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339012,"statfs":{"total":96636764160,"available":96603148288,"internally_reserved":0,"allocated":1679360,"data_stored":742072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542233,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32836,"kb_used_data":1648,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339004,"statfs":{"total":96636764160,"available":96603140096,"internally_reserved":0,"allocated":1687552,"data_stored":748378,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":3,"up_from":14,"seq":60129542234,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32828,"kb_used_data":1640,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339012,"statfs":{"total":96636764160,"available":96603148288,"internally_reserved":0,"allocated":1679360,"data_stored":742140,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":2,"up_from":14,"seq":60129542234,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32856,"kb_used_data":1664,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338984,"statfs":{"total":96636764160,"available":96603119616,"internally_reserved":0,"allocated":1703936,"data_stored":750060,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":14,"seq":60129542236,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32828,"kb_used_data":1640,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339012,"statfs":{"total":96636764160,"available":96603148288,"internally_reserved":0,"allocated":1679360,"data_stored":742632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":14,"seq":60129542236,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33428,"kb_used_data":2232,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338412,"statfs":{"total":96636764160,"available":96602533888,"internally_reserved":0,"allocated":2285568,"data_stored":1334064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1646,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T15:49:58.194 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-09-16T15:49:58.195 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-09-16T15:49:58.370 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:58.370+0000 7f427d846640 1 -- 172.21.15.123:0/1802963381 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f427815ddc0 msgr2=0x7f427815e270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:58.370 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:58.370+0000 7f427d846640 1 --2- 172.21.15.123:0/1802963381 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f427815ddc0 0x7f427815e270 secure :-1 s=READY pgs=1412 cs=0 l=1 rev1=1 crypto rx=0x7f4270009c80 tx=0x7f427001cd10 comp rx=0 tx=0).stop 2024-09-16T15:49:58.370 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:58.370+0000 7f427d846640 1 -- 172.21.15.123:0/1802963381 shutdown_connections 2024-09-16T15:49:58.371 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:58.370+0000 7f427d846640 1 --2- 172.21.15.123:0/1802963381 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f427815ddc0 0x7f427815e270 unknown :-1 s=CLOSED pgs=1412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:58.371 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:58.370+0000 7f427d846640 1 --2- 172.21.15.123:0/1802963381 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f427815a170 0x7f4278072f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:58.371 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:58.370+0000 7f427d846640 1 -- 172.21.15.123:0/1802963381 >> 172.21.15.123:0/1802963381 conn(0x7f42780b1710 msgr2=0x7f42780b1b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:49:58.371 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:58.370+0000 7f427d846640 1 -- 172.21.15.123:0/1802963381 shutdown_connections 2024-09-16T15:49:58.371 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:58.370+0000 7f427d846640 1 -- 172.21.15.123:0/1802963381 wait complete. 2024-09-16T15:49:58.371 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:58.370+0000 7f427d846640 1 Processor -- start 2024-09-16T15:49:58.371 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:58.370+0000 7f427d846640 1 -- start start 2024-09-16T15:49:58.373 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:58.370+0000 7f427d846640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f42780ca650 0x7f42780c6600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:49:58.373 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:58.374+0000 7f427d846640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f427815a170 0x7f42780c51f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:49:58.374 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:58.374+0000 7f427d846640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f427815ddc0 0x7f42781f8950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:49:58.374 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:58.374+0000 7f427d846640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f4278167160 con 0x7f427815a170 2024-09-16T15:49:58.374 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:58.374+0000 7f427d846640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f427806ee60 con 0x7f427815ddc0 2024-09-16T15:49:58.374 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:58.374+0000 7f427d846640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f42781672e0 con 0x7f42780ca650 2024-09-16T15:49:58.374 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:58.374+0000 7f426ffff640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f427815a170 0x7f42780c51f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:49:58.374 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:58.374+0000 7f426ffff640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f427815a170 0x7f42780c51f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:58796/0 (socket says 172.21.15.123:58796) 2024-09-16T15:49:58.374 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:58.374+0000 7f426ffff640 1 -- 172.21.15.123:0/251500821 learned_addr learned my addr 172.21.15.123:0/251500821 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:49:58.374 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:58.374+0000 7f42777fe640 1 --2- 172.21.15.123:0/251500821 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f427815ddc0 0x7f42781f8950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:49:58.374 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:58.374+0000 7f4276ffd640 1 --2- 172.21.15.123:0/251500821 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f42780ca650 0x7f42780c6600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:49:58.374 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:58.374+0000 7f426ffff640 1 -- 172.21.15.123:0/251500821 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f42780ca650 msgr2=0x7f42780c6600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:58.374 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:58.374+0000 7f426ffff640 1 --2- 172.21.15.123:0/251500821 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f42780ca650 0x7f42780c6600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:58.374 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:58.374+0000 7f426ffff640 1 -- 172.21.15.123:0/251500821 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f427815ddc0 msgr2=0x7f42781f8950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:58.374 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:58.374+0000 7f426ffff640 1 --2- 172.21.15.123:0/251500821 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f427815ddc0 0x7f42781f8950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:58.375 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:58.374+0000 7f426ffff640 1 -- 172.21.15.123:0/251500821 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f42780c5990 con 0x7f427815a170 2024-09-16T15:49:58.375 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:58.374+0000 7f426ffff640 1 --2- 172.21.15.123:0/251500821 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f427815a170 0x7f42780c51f0 secure :-1 s=READY pgs=1413 cs=0 l=1 rev1=1 crypto rx=0x7f426000c440 tx=0x7f426000c900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:49:58.375 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:58.374+0000 7f4274ff9640 1 -- 172.21.15.123:0/251500821 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4260011790 con 0x7f427815a170 2024-09-16T15:49:58.375 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:58.374+0000 7f4274ff9640 1 -- 172.21.15.123:0/251500821 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f4260017070 con 0x7f427815a170 2024-09-16T15:49:58.375 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:58.374+0000 7f427d846640 1 -- 172.21.15.123:0/251500821 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f42780c3de0 con 0x7f427815a170 2024-09-16T15:49:58.375 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:58.374+0000 7f4274ff9640 1 -- 172.21.15.123:0/251500821 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4260011cc0 con 0x7f427815a170 2024-09-16T15:49:58.375 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:58.374+0000 7f427d846640 1 -- 172.21.15.123:0/251500821 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f42780c43a0 con 0x7f427815a170 2024-09-16T15:49:58.376 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:58.374+0000 7f427d846640 1 -- 172.21.15.123:0/251500821 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4278072f30 con 0x7f427815a170 2024-09-16T15:49:58.376 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:58.374+0000 7f4274ff9640 1 -- 172.21.15.123:0/251500821 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 21) ==== 97641+0+0 (secure 0 0 0) 0x7f426001c460 con 0x7f427815a170 2024-09-16T15:49:58.376 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:58.374+0000 7f4274ff9640 1 --2- 172.21.15.123:0/251500821 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f42580734f0 0x7f42580759b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:49:58.376 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:58.374+0000 7f4276ffd640 1 --2- 172.21.15.123:0/251500821 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f42580734f0 0x7f42580759b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:49:58.376 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:58.374+0000 7f4274ff9640 1 -- 172.21.15.123:0/251500821 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(135..135 src has 1..135) ==== 5737+0+0 (secure 0 0 0) 0x7f4260094830 con 0x7f427815a170 2024-09-16T15:49:58.376 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:58.374+0000 7f4276ffd640 1 --2- 172.21.15.123:0/251500821 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f42580734f0 0x7f42580759b0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f426800abe0 tx=0x7f426800a2d0 comp rx=0 tx=0).ready entity=mgr.4150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:49:58.379 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:58.378+0000 7f4274ff9640 1 -- 172.21.15.123:0/251500821 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f426005e940 con 0x7f427815a170 2024-09-16T15:49:58.617 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:58.618+0000 7f427d846640 1 -- 172.21.15.123:0/251500821 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) -- 0x7f42780697b0 con 0x7f427815a170 2024-09-16T15:49:58.620 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:58.618+0000 7f4274ff9640 1 -- 172.21.15.123:0/251500821 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v26) ==== 190+0+147 (secure 0 0 0) 0x7f4260063370 con 0x7f427815a170 2024-09-16T15:49:58.620 INFO:teuthology.orchestra.run.smithi123.stdout:[client.0] 2024-09-16T15:49:58.621 INFO:teuthology.orchestra.run.smithi123.stdout: key = AQC1Uehm7GDTAhAAHWWVIyjVVnTeeozBR8DfFA== 2024-09-16T15:49:58.621 INFO:teuthology.orchestra.run.smithi123.stdout: caps mds = "allow" 2024-09-16T15:49:58.621 INFO:teuthology.orchestra.run.smithi123.stdout: caps mon = "allow r" 2024-09-16T15:49:58.621 INFO:teuthology.orchestra.run.smithi123.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-09-16T15:49:58.621 INFO:teuthology.orchestra.run.smithi123.stderr:updated caps for client.0 2024-09-16T15:49:58.623 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:58.622+0000 7f427d846640 1 -- 172.21.15.123:0/251500821 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f42580734f0 msgr2=0x7f42580759b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:58.623 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:58.622+0000 7f427d846640 1 --2- 172.21.15.123:0/251500821 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f42580734f0 0x7f42580759b0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f426800abe0 tx=0x7f426800a2d0 comp rx=0 tx=0).stop 2024-09-16T15:49:58.623 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:58.622+0000 7f427d846640 1 -- 172.21.15.123:0/251500821 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f427815a170 msgr2=0x7f42780c51f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:58.623 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:58.622+0000 7f427d846640 1 --2- 172.21.15.123:0/251500821 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f427815a170 0x7f42780c51f0 secure :-1 s=READY pgs=1413 cs=0 l=1 rev1=1 crypto rx=0x7f426000c440 tx=0x7f426000c900 comp rx=0 tx=0).stop 2024-09-16T15:49:58.623 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:58.622+0000 7f427d846640 1 -- 172.21.15.123:0/251500821 shutdown_connections 2024-09-16T15:49:58.623 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:58.622+0000 7f427d846640 1 --2- 172.21.15.123:0/251500821 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f42580734f0 0x7f42580759b0 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:58.623 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:58.622+0000 7f427d846640 1 --2- 172.21.15.123:0/251500821 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f427815ddc0 0x7f42781f8950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:58.623 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:58.622+0000 7f427d846640 1 --2- 172.21.15.123:0/251500821 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f427815a170 0x7f42780c51f0 unknown :-1 s=CLOSED pgs=1413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:58.623 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:58.622+0000 7f427d846640 1 --2- 172.21.15.123:0/251500821 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f42780ca650 0x7f42780c6600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:58.624 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:58.622+0000 7f427d846640 1 -- 172.21.15.123:0/251500821 >> 172.21.15.123:0/251500821 conn(0x7f42780b1710 msgr2=0x7f42780b4c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:49:58.624 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:58.622+0000 7f427d846640 1 -- 172.21.15.123:0/251500821 shutdown_connections 2024-09-16T15:49:58.624 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:58.622+0000 7f427d846640 1 -- 172.21.15.123:0/251500821 wait complete. 2024-09-16T15:49:58.639 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-16T15:49:58.753 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:58.754+0000 7f6c2d3ea640 1 -- 172.21.15.123:0/1137160664 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6c10004810 msgr2=0x7f6c10004c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:58.753 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:58.754+0000 7f6c2d3ea640 1 --2- 172.21.15.123:0/1137160664 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6c10004810 0x7f6c10004c90 secure :-1 s=READY pgs=1415 cs=0 l=1 rev1=1 crypto rx=0x7f6c1c00bb10 tx=0x7f6c1c01cd70 comp rx=0 tx=0).stop 2024-09-16T15:49:58.753 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:58.754+0000 7f6c277fe640 1 -- 172.21.15.123:0/1137160664 reap_dead start 2024-09-16T15:49:58.753 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:58.754+0000 7f6c2d3ea640 1 -- 172.21.15.123:0/1137160664 shutdown_connections 2024-09-16T15:49:58.753 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:58.754+0000 7f6c2d3ea640 1 -- 172.21.15.123:0/1137160664 >> 172.21.15.123:0/1137160664 conn(0x7f6c280d3ea0 msgr2=0x7f6c280d42b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:49:58.754 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:58.754+0000 7f6c2d3ea640 1 -- 172.21.15.123:0/1137160664 shutdown_connections 2024-09-16T15:49:58.754 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:58.754+0000 7f6c2d3ea640 1 -- 172.21.15.123:0/1137160664 wait complete. 2024-09-16T15:49:58.754 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:58.754+0000 7f6c2d3ea640 1 Processor -- start 2024-09-16T15:49:58.754 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:58.754+0000 7f6c2d3ea640 1 -- start start 2024-09-16T15:49:58.756 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:58.754+0000 7f6c2d3ea640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6c10004810 0x7f6c280b9860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:49:58.756 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:58.754+0000 7f6c2d3ea640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f6c28164500 0x7f6c2814a0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:49:58.757 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:58.754+0000 7f6c2d3ea640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6c2814a630 0x7f6c2806d2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:49:58.757 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:58.754+0000 7f6c2d3ea640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f6c280d2b10 con 0x7f6c10004810 2024-09-16T15:49:58.757 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:58.754+0000 7f6c2d3ea640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f6c280d2990 con 0x7f6c2814a630 2024-09-16T15:49:58.757 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:58.754+0000 7f6c2d3ea640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f6c28169d60 con 0x7f6c28164500 2024-09-16T15:49:58.757 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:58.754+0000 7f6c26ffd640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6c10004810 0x7f6c280b9860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:49:58.757 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:58.754+0000 7f6c26ffd640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6c10004810 0x7f6c280b9860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:58806/0 (socket says 172.21.15.123:58806) 2024-09-16T15:49:58.757 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:58.754+0000 7f6c26ffd640 1 -- 172.21.15.123:0/832221898 learned_addr learned my addr 172.21.15.123:0/832221898 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:49:58.757 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:58.754+0000 7f6c26ffd640 1 -- 172.21.15.123:0/832221898 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f6c28164500 msgr2=0x7f6c2814a0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:58.757 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:58.754+0000 7f6c26ffd640 1 --2- 172.21.15.123:0/832221898 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f6c28164500 0x7f6c2814a0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:58.757 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:58.754+0000 7f6c26ffd640 1 -- 172.21.15.123:0/832221898 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6c2814a630 msgr2=0x7f6c2806d2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:58.757 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:58.754+0000 7f6c277fe640 1 --2- 172.21.15.123:0/832221898 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6c2814a630 0x7f6c2806d2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:49:58.757 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:58.754+0000 7f6c26ffd640 1 --2- 172.21.15.123:0/832221898 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6c2814a630 0x7f6c2806d2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:58.758 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:58.754+0000 7f6c26ffd640 1 -- 172.21.15.123:0/832221898 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6c280b8520 con 0x7f6c10004810 2024-09-16T15:49:58.758 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:58.754+0000 7f6c26ffd640 1 --2- 172.21.15.123:0/832221898 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6c10004810 0x7f6c280b9860 secure :-1 s=READY pgs=1416 cs=0 l=1 rev1=1 crypto rx=0x7f6c1800c760 tx=0x7f6c1800cc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:49:58.758 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:58.754+0000 7f6c0f7fe640 1 -- 172.21.15.123:0/832221898 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c18017070 con 0x7f6c10004810 2024-09-16T15:49:58.758 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:58.754+0000 7f6c0f7fe640 1 -- 172.21.15.123:0/832221898 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f6c18005360 con 0x7f6c10004810 2024-09-16T15:49:58.758 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:58.754+0000 7f6c2d3ea640 1 -- 172.21.15.123:0/832221898 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6c280b87c0 con 0x7f6c10004810 2024-09-16T15:49:58.758 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:58.754+0000 7f6c0f7fe640 1 -- 172.21.15.123:0/832221898 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c18003860 con 0x7f6c10004810 2024-09-16T15:49:58.758 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:58.754+0000 7f6c2d3ea640 1 -- 172.21.15.123:0/832221898 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6c280b8d80 con 0x7f6c10004810 2024-09-16T15:49:58.758 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:58.758+0000 7f6c2d3ea640 1 -- 172.21.15.123:0/832221898 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6c280b0ee0 con 0x7f6c10004810 2024-09-16T15:49:58.759 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:58.758+0000 7f6c0f7fe640 1 -- 172.21.15.123:0/832221898 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 21) ==== 97641+0+0 (secure 0 0 0) 0x7f6c1801a070 con 0x7f6c10004810 2024-09-16T15:49:58.759 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:58.758+0000 7f6c0f7fe640 1 --2- 172.21.15.123:0/832221898 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f6c14073520 0x7f6c140759e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:49:58.759 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:58.758+0000 7f6c267fc640 1 --2- 172.21.15.123:0/832221898 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f6c14073520 0x7f6c140759e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:49:58.759 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:58.758+0000 7f6c0f7fe640 1 -- 172.21.15.123:0/832221898 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(135..135 src has 1..135) ==== 5737+0+0 (secure 0 0 0) 0x7f6c18093e10 con 0x7f6c10004810 2024-09-16T15:49:58.759 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:58.758+0000 7f6c267fc640 1 --2- 172.21.15.123:0/832221898 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f6c14073520 0x7f6c140759e0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f6c1c0b0000 tx=0x7f6c10005840 comp rx=0 tx=0).ready entity=mgr.4150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:49:58.762 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:58.762+0000 7f6c0f7fe640 1 -- 172.21.15.123:0/832221898 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f6c1805df60 con 0x7f6c10004810 2024-09-16T15:49:58.969 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:58.970+0000 7f6c2d3ea640 1 -- 172.21.15.123:0/832221898 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f6c28169680 con 0x7f6c10004810 2024-09-16T15:49:58.969 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:58.970+0000 7f6c0f7fe640 1 -- 172.21.15.123:0/832221898 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 86 v86) ==== 94+0+5230 (secure 0 0 0) 0x7f6c18062990 con 0x7f6c10004810 2024-09-16T15:49:58.970 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-09-16T15:49:58.970 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":86,"btime":"2024-09-16T15:49:56:787440+0000","default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5801,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6834/730567479","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":730567479},{"type":"v1","addr":"172.21.15.184:6834","nonce":730567479}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":81},{"gid":6346,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6835/584518652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6833","nonce":584518652},{"type":"v1","addr":"172.21.15.184:6835","nonce":584518652}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":81},{"gid":7938,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6833/3935767668","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6832","nonce":3935767668},{"type":"v1","addr":"172.21.15.123:6833","nonce":3935767668}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":80}],"filesystems":[{"mdsmap":{"epoch":86,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-16T15:49:53.127484+0000","modified":"2024-09-16T15:49:56.160938+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7944},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7944":{"gid":7944,"name":"a","rank":0,"incarnation":82,"state":"up:active","state_seq":3,"addr":"172.21.15.123:6835/333722639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6834","nonce":333722639},{"type":"v1","addr":"172.21.15.123:6835","nonce":333722639}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7944,"qdb_cluster":[7944]},"id":10}]} 2024-09-16T15:49:58.970 INFO:teuthology.orchestra.run.smithi123.stderr:dumped fsmap epoch 86 2024-09-16T15:49:58.972 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:58.970+0000 7f6c2d3ea640 1 -- 172.21.15.123:0/832221898 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f6c14073520 msgr2=0x7f6c140759e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:58.972 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:58.970+0000 7f6c2d3ea640 1 --2- 172.21.15.123:0/832221898 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f6c14073520 0x7f6c140759e0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f6c1c0b0000 tx=0x7f6c10005840 comp rx=0 tx=0).stop 2024-09-16T15:49:58.972 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:58.970+0000 7f6c2d3ea640 1 -- 172.21.15.123:0/832221898 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6c10004810 msgr2=0x7f6c280b9860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:58.972 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:58.970+0000 7f6c2d3ea640 1 --2- 172.21.15.123:0/832221898 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6c10004810 0x7f6c280b9860 secure :-1 s=READY pgs=1416 cs=0 l=1 rev1=1 crypto rx=0x7f6c1800c760 tx=0x7f6c1800cc20 comp rx=0 tx=0).stop 2024-09-16T15:49:58.972 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:58.970+0000 7f6c2d3ea640 1 -- 172.21.15.123:0/832221898 shutdown_connections 2024-09-16T15:49:58.972 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:58.970+0000 7f6c2d3ea640 1 --2- 172.21.15.123:0/832221898 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f6c14073520 0x7f6c140759e0 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:58.972 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:58.970+0000 7f6c2d3ea640 1 --2- 172.21.15.123:0/832221898 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6c2814a630 0x7f6c2806d2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:58.972 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:58.970+0000 7f6c2d3ea640 1 --2- 172.21.15.123:0/832221898 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f6c28164500 0x7f6c2814a0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:58.972 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:58.970+0000 7f6c2d3ea640 1 --2- 172.21.15.123:0/832221898 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6c10004810 0x7f6c280b9860 unknown :-1 s=CLOSED pgs=1416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:58.973 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:58.970+0000 7f6c2d3ea640 1 -- 172.21.15.123:0/832221898 >> 172.21.15.123:0/832221898 conn(0x7f6c280d3ea0 msgr2=0x7f6c280d4670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:49:58.973 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:58.970+0000 7f6c2d3ea640 1 -- 172.21.15.123:0/832221898 shutdown_connections 2024-09-16T15:49:58.973 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:58.970+0000 7f6c2d3ea640 1 -- 172.21.15.123:0/832221898 wait complete. 2024-09-16T15:49:59.390 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 86, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-09-16T15:49:53.127484+0000', 'modified': '2024-09-16T15:49:56.160938+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 7944}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7944': {'gid': 7944, 'name': 'a', 'rank': 0, 'incarnation': 82, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.123:6835/333722639', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.123:6834', 'nonce': 333722639}, {'type': 'v1', 'addr': '172.21.15.123:6835', 'nonce': 333722639}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540719139924082687, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 7944, 'qdb_cluster': [7944]} 2024-09-16T15:49:59.390 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-09-16T15:49:59.390 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2024-09-16T15:49:59.527 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:59.526+0000 7f031d8e9640 1 -- 172.21.15.123:0/2823298435 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0300004d50 msgr2=0x7f03000051d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:59.527 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:59.526+0000 7f031d8e9640 1 --2- 172.21.15.123:0/2823298435 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0300004d50 0x7f03000051d0 secure :-1 s=READY pgs=1418 cs=0 l=1 rev1=1 crypto rx=0x7f0310009e30 tx=0x7f031001d1b0 comp rx=0 tx=0).stop 2024-09-16T15:49:59.527 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:59.526+0000 7f03177fe640 1 -- 172.21.15.123:0/2823298435 reap_dead start 2024-09-16T15:49:59.527 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:59.526+0000 7f031d8e9640 1 -- 172.21.15.123:0/2823298435 shutdown_connections 2024-09-16T15:49:59.527 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:59.526+0000 7f031d8e9640 1 -- 172.21.15.123:0/2823298435 >> 172.21.15.123:0/2823298435 conn(0x7f03180cfb10 msgr2=0x7f03180cff20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:49:59.527 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:59.526+0000 7f031d8e9640 1 -- 172.21.15.123:0/2823298435 shutdown_connections 2024-09-16T15:49:59.527 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:59.526+0000 7f031d8e9640 1 -- 172.21.15.123:0/2823298435 wait complete. 2024-09-16T15:49:59.528 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:59.526+0000 7f031d8e9640 1 Processor -- start 2024-09-16T15:49:59.528 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:59.526+0000 7f031d8e9640 1 -- start start 2024-09-16T15:49:59.529 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:59.530+0000 7f031d8e9640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f03181f1d30 0x7f0318158b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:49:59.529 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:59.530+0000 7f031d8e9640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f0318159060 0x7f03180c7c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:49:59.529 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:59.530+0000 7f031d8e9640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f03180c8170 0x7f0318155f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:49:59.530 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:59.530+0000 7f031d8e9640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f03180d09c0 con 0x7f03180c8170 2024-09-16T15:49:59.530 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:59.530+0000 7f031d8e9640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f03180d0840 con 0x7f03181f1d30 2024-09-16T15:49:59.530 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:59.530+0000 7f031d8e9640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f031806ec90 con 0x7f0318159060 2024-09-16T15:49:59.530 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:59.530+0000 7f03177fe640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f03180c8170 0x7f0318155f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:49:59.530 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:59.530+0000 7f03177fe640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f03180c8170 0x7f0318155f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:58832/0 (socket says 172.21.15.123:58832) 2024-09-16T15:49:59.530 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:59.530+0000 7f03177fe640 1 -- 172.21.15.123:0/1599040388 learned_addr learned my addr 172.21.15.123:0/1599040388 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:49:59.530 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:59.530+0000 7f0316ffd640 1 --2- 172.21.15.123:0/1599040388 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f03181f1d30 0x7f0318158b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:49:59.530 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:59.530+0000 7f03167fc640 1 --2- 172.21.15.123:0/1599040388 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f0318159060 0x7f03180c7c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:49:59.530 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:59.530+0000 7f03177fe640 1 -- 172.21.15.123:0/1599040388 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f0318159060 msgr2=0x7f03180c7c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:59.530 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:59.530+0000 7f03177fe640 1 --2- 172.21.15.123:0/1599040388 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f0318159060 0x7f03180c7c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:59.530 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:59.530+0000 7f03177fe640 1 -- 172.21.15.123:0/1599040388 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f03181f1d30 msgr2=0x7f0318158b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:59.530 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:59.530+0000 7f03177fe640 1 --2- 172.21.15.123:0/1599040388 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f03181f1d30 0x7f0318158b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:59.530 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:59.530+0000 7f03177fe640 1 -- 172.21.15.123:0/1599040388 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f03181f3130 con 0x7f03180c8170 2024-09-16T15:49:59.530 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:59.530+0000 7f03177fe640 1 --2- 172.21.15.123:0/1599040388 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f03180c8170 0x7f0318155f90 secure :-1 s=READY pgs=1419 cs=0 l=1 rev1=1 crypto rx=0x7f0310002240 tx=0x7f03100add50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:49:59.531 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:59.530+0000 7f02fffff640 1 -- 172.21.15.123:0/1599040388 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03100b1040 con 0x7f03180c8170 2024-09-16T15:49:59.531 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:59.530+0000 7f02fffff640 1 -- 172.21.15.123:0/1599040388 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f03100bc070 con 0x7f03180c8170 2024-09-16T15:49:59.531 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:59.530+0000 7f02fffff640 1 -- 172.21.15.123:0/1599040388 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0310009a00 con 0x7f03180c8170 2024-09-16T15:49:59.531 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:59.530+0000 7f031d8e9640 1 -- 172.21.15.123:0/1599040388 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f03181f3410 con 0x7f03180c8170 2024-09-16T15:49:59.531 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:59.530+0000 7f031d8e9640 1 -- 172.21.15.123:0/1599040388 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0318158320 con 0x7f03180c8170 2024-09-16T15:49:59.531 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:59.530+0000 7f02fffff640 1 -- 172.21.15.123:0/1599040388 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 21) ==== 97641+0+0 (secure 0 0 0) 0x7f0310004a50 con 0x7f03180c8170 2024-09-16T15:49:59.532 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:59.530+0000 7f02fffff640 1 --2- 172.21.15.123:0/1599040388 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f0304073520 0x7f03040759e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:49:59.532 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:59.530+0000 7f02fffff640 1 -- 172.21.15.123:0/1599040388 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(135..135 src has 1..135) ==== 5737+0+0 (secure 0 0 0) 0x7f031012f390 con 0x7f03180c8170 2024-09-16T15:49:59.532 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:59.530+0000 7f0316ffd640 1 --2- 172.21.15.123:0/1599040388 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f0304073520 0x7f03040759e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:49:59.532 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:59.534+0000 7f0316ffd640 1 --2- 172.21.15.123:0/1599040388 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f0304073520 0x7f03040759e0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f0308000f70 tx=0x7f0308001670 comp rx=0 tx=0).ready entity=mgr.4150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:49:59.532 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:59.534+0000 7f03209d3000 1 Processor -- start 2024-09-16T15:49:59.532 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:59.534+0000 7f03209d3000 1 -- start start 2024-09-16T15:49:59.534 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:59.534+0000 7f03209d3000 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x557b7a0d99d0 0x557b7a0d9db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:49:59.534 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:59.534+0000 7f03209d3000 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x557b7a0da2f0 0x557b7a08d8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:49:59.534 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:59.534+0000 7f03209d3000 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x557b7a08de10 0x557b7a08e1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:49:59.534 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:59.534+0000 7f03209d3000 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x557b79fae3f0 con 0x557b7a0da2f0 2024-09-16T15:49:59.534 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:59.534+0000 7f03209d3000 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x557b79eebb20 con 0x557b7a0d99d0 2024-09-16T15:49:59.534 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:59.534+0000 7f03209d3000 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x557b79edaf50 con 0x557b7a08de10 2024-09-16T15:49:59.534 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:59.534+0000 7f03167fc640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x557b7a0da2f0 0x557b7a08d8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:49:59.534 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:59.534+0000 7f03167fc640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x557b7a0da2f0 0x557b7a08d8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:58842/0 (socket says 172.21.15.123:58842) 2024-09-16T15:49:59.535 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:59.534+0000 7f03167fc640 1 -- 172.21.15.123:0/2141405920 learned_addr learned my addr 172.21.15.123:0/2141405920 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:49:59.535 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:59.534+0000 7f03177fe640 1 --2- 172.21.15.123:0/2141405920 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x557b7a08de10 0x557b7a08e1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:49:59.535 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:59.534+0000 7f0316ffd640 1 --2- 172.21.15.123:0/2141405920 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x557b7a0d99d0 0x557b7a0d9db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:49:59.535 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:59.534+0000 7f03167fc640 1 -- 172.21.15.123:0/2141405920 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x557b7a08de10 msgr2=0x557b7a08e1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T15:49:59.535 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:59.534+0000 7f03167fc640 1 --2- 172.21.15.123:0/2141405920 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x557b7a08de10 0x557b7a08e1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:59.535 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:59.534+0000 7f03167fc640 1 -- 172.21.15.123:0/2141405920 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x557b7a0d99d0 msgr2=0x557b7a0d9db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T15:49:59.535 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:59.534+0000 7f03167fc640 1 --2- 172.21.15.123:0/2141405920 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x557b7a0d99d0 0x557b7a0d9db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:59.535 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:59.534+0000 7f03167fc640 1 -- 172.21.15.123:0/2141405920 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557b79ea2590 con 0x557b7a0da2f0 2024-09-16T15:49:59.535 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:59.534+0000 7f03167fc640 1 --2- 172.21.15.123:0/2141405920 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x557b7a0da2f0 0x557b7a08d8d0 secure :-1 s=READY pgs=1420 cs=0 l=1 rev1=1 crypto rx=0x7f030c009b40 tx=0x7f030c0113e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1bb0fab39174bdb server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:49:59.535 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:59.534+0000 7f02fd7fa640 1 -- 172.21.15.123:0/2141405920 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f030c011e10 con 0x557b7a0da2f0 2024-09-16T15:49:59.535 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:59.534+0000 7f02fd7fa640 1 -- 172.21.15.123:0/2141405920 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f030c004030 con 0x557b7a0da2f0 2024-09-16T15:49:59.535 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:59.534+0000 7f02fd7fa640 1 -- 172.21.15.123:0/2141405920 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f030c004480 con 0x557b7a0da2f0 2024-09-16T15:49:59.539 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:59.534+0000 7f03209d3000 1 -- 172.21.15.123:0/2141405920 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x557b7a0da2f0 msgr2=0x557b7a08d8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:59.539 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:59.534+0000 7f03209d3000 1 --2- 172.21.15.123:0/2141405920 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x557b7a0da2f0 0x557b7a08d8d0 secure :-1 s=READY pgs=1420 cs=0 l=1 rev1=1 crypto rx=0x7f030c009b40 tx=0x7f030c0113e0 comp rx=0 tx=0).stop 2024-09-16T15:49:59.540 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:59.534+0000 7f03209d3000 1 -- 172.21.15.123:0/2141405920 shutdown_connections 2024-09-16T15:49:59.540 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:59.534+0000 7f03209d3000 1 --2- 172.21.15.123:0/2141405920 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x557b7a08de10 0x557b7a08e1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:59.540 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:59.534+0000 7f03209d3000 1 --2- 172.21.15.123:0/2141405920 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x557b7a0da2f0 0x557b7a08d8d0 unknown :-1 s=CLOSED pgs=1420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:59.540 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:59.534+0000 7f03209d3000 1 --2- 172.21.15.123:0/2141405920 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x557b7a0d99d0 0x557b7a0d9db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:59.540 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:59.534+0000 7f03209d3000 1 -- 172.21.15.123:0/2141405920 >> 172.21.15.123:0/2141405920 conn(0x557b79f77130 msgr2=0x557b7a0bce80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:49:59.540 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:59.534+0000 7f03209d3000 1 -- 172.21.15.123:0/2141405920 shutdown_connections 2024-09-16T15:49:59.540 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:59.534+0000 7f03209d3000 1 -- 172.21.15.123:0/2141405920 wait complete. 2024-09-16T15:49:59.540 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:59.534+0000 7f03209d3000 1 -- start start 2024-09-16T15:49:59.540 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:59.534+0000 7f03209d3000 1 Processor -- start 2024-09-16T15:49:59.540 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:59.538+0000 7f03209d3000 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x557b7a08de10 0x557b7a0ba6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:49:59.540 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:59.538+0000 7f03209d3000 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x557b7a0d99d0 0x557b7a0bae70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:49:59.540 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:59.538+0000 7f03209d3000 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x557b7a0da2f0 0x557b7a0bb3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:49:59.540 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:59.538+0000 7f03209d3000 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x557b79e755d0 con 0x557b7a0da2f0 2024-09-16T15:49:59.540 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:59.538+0000 7f03209d3000 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x557b79eded60 con 0x557b7a0d99d0 2024-09-16T15:49:59.541 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:59.538+0000 7f03209d3000 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x557b79e81500 con 0x557b7a08de10 2024-09-16T15:49:59.541 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:59.538+0000 7f03209d3000 10 client.0 _ll_register_callbacks cb 0x557b79e85810 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T15:49:59.541 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:59.538+0000 7f03177fe640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x557b7a0da2f0 0x557b7a0bb3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:49:59.541 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:59.538+0000 7f03177fe640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x557b7a0da2f0 0x557b7a0bb3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:58844/0 (socket says 172.21.15.123:58844) 2024-09-16T15:49:59.541 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:59.538+0000 7f03177fe640 1 -- 172.21.15.123:0/1822767544 learned_addr learned my addr 172.21.15.123:0/1822767544 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:49:59.541 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:59.538+0000 7f0316ffd640 1 --2- 172.21.15.123:0/1822767544 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x557b7a08de10 0x557b7a0ba6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:49:59.541 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:59.538+0000 7f03167fc640 1 --2- 172.21.15.123:0/1822767544 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x557b7a0d99d0 0x557b7a0bae70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:49:59.541 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:59.538+0000 7f03177fe640 1 -- 172.21.15.123:0/1822767544 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x557b7a08de10 msgr2=0x557b7a0ba6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T15:49:59.541 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:59.538+0000 7f03177fe640 1 --2- 172.21.15.123:0/1822767544 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x557b7a08de10 0x557b7a0ba6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:59.541 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:59.538+0000 7f03177fe640 1 -- 172.21.15.123:0/1822767544 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x557b7a0d99d0 msgr2=0x557b7a0bae70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T15:49:59.541 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:59.538+0000 7f03177fe640 1 --2- 172.21.15.123:0/1822767544 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x557b7a0d99d0 0x557b7a0bae70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:59.541 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:59.538+0000 7f03177fe640 1 -- 172.21.15.123:0/1822767544 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557b79e79170 con 0x557b7a0da2f0 2024-09-16T15:49:59.541 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:59.538+0000 7f03167fc640 1 --2- 172.21.15.123:0/1822767544 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x557b7a0d99d0 0x557b7a0bae70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T15:49:59.541 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:59.538+0000 7f03177fe640 1 --2- 172.21.15.123:0/1822767544 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x557b7a0da2f0 0x557b7a0bb3b0 secure :-1 s=READY pgs=1421 cs=0 l=1 rev1=1 crypto rx=0x7f03100f5690 tx=0x7f03100f8ff0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3747d817be597af1 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:49:59.541 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:59.538+0000 7f02f67fc640 10 client.0 ms_handle_connect on v2:172.21.15.123:3300/0 2024-09-16T15:49:59.542 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:59.538+0000 7f02f67fc640 1 -- 172.21.15.123:0/1822767544 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03100f95e0 con 0x557b7a0da2f0 2024-09-16T15:49:59.542 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:59.538+0000 7f02f67fc640 1 -- 172.21.15.123:0/1822767544 <== mon.0 v2:172.21.15.123:3300/0 2 ==== osd_map(135..135 src has 1..135) ==== 5737+0+0 (secure 0 0 0) 0x7f0310102ae0 con 0x557b7a0da2f0 2024-09-16T15:49:59.542 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:59.538+0000 7f03209d3000 1 -- 172.21.15.123:0/1822767544 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x557b79e730f0 con 0x557b7a0da2f0 2024-09-16T15:49:59.542 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:59.538+0000 7f02f67fc640 1 -- 172.21.15.123:0/1822767544 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_get_version_reply(handle=1 version=86) ==== 24+0+0 (secure 0 0 0) 0x7f03100fe0c0 con 0x557b7a0da2f0 2024-09-16T15:49:59.542 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:59.538+0000 7f03209d3000 10 client.8073 fetch_fsmap learned FSMap version 86 2024-09-16T15:49:59.542 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:59.538+0000 7f03209d3000 1 -- 172.21.15.123:0/1822767544 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,fsmap=86,monmap=2+}) -- 0x557b79f5fef0 con 0x557b7a0da2f0 2024-09-16T15:49:59.542 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:59.538+0000 7f02f67fc640 1 -- 172.21.15.123:0/1822767544 <== mon.0 v2:172.21.15.123:3300/0 4 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x557b79e730f0 con 0x557b7a0da2f0 2024-09-16T15:49:59.542 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:59.538+0000 7f02f67fc640 1 -- 172.21.15.123:0/1822767544 <== mon.0 v2:172.21.15.123:3300/0 5 ==== fsmap(e 86) ==== 3502+0+0 (secure 0 0 0) 0x7f031000f040 con 0x557b7a0da2f0 2024-09-16T15:49:59.542 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:59.538+0000 7f03209d3000 10 client.8073 fetch_fsmap finished waiting for FSMap version 86 2024-09-16T15:49:59.542 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:59.538+0000 7f03209d3000 10 client.8073 resolve_mds: resolved name 'a' to daemon mds.a 2024-09-16T15:49:59.542 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:59.538+0000 7f03209d3000 20 client.8073 populate_metadata read hostname 'smithi123' 2024-09-16T15:49:59.542 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:59.538+0000 7f03209d3000 1 --2- 172.21.15.123:0/1822767544 >> [v2:172.21.15.123:6834/333722639,v1:172.21.15.123:6835/333722639] conn(0x557b7a0bc0f0 0x557b7a0ebc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:49:59.543 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:59.538+0000 7f03209d3000 4 client.8073 mds_command: new command op to 7944 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-16T15:49:59.543 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:59.538+0000 7f03209d3000 1 -- 172.21.15.123:0/1822767544 --> [v2:172.21.15.123:6834/333722639,v1:172.21.15.123:6835/333722639] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x557b79eeda70 con 0x557b7a0bc0f0 2024-09-16T15:49:59.543 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:59.538+0000 7f0316ffd640 1 --2- 172.21.15.123:0/1822767544 >> [v2:172.21.15.123:6834/333722639,v1:172.21.15.123:6835/333722639] conn(0x557b7a0bc0f0 0x557b7a0ebc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:49:59.543 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:59.538+0000 7f02f67fc640 10 client.8073 ms_handle_connect on v2:172.21.15.123:6834/333722639 2024-09-16T15:49:59.543 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:59.538+0000 7f0316ffd640 1 --2- 172.21.15.123:0/1822767544 >> [v2:172.21.15.123:6834/333722639,v1:172.21.15.123:6835/333722639] conn(0x557b7a0bc0f0 0x557b7a0ebc90 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=88280565d656d70b server_cookie=1b7d7f94aa907dec in_seq=0 out_seq=0 2024-09-16T15:49:59.543 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:59.538+0000 7f02f67fc640 1 -- 172.21.15.123:0/1822767544 <== mds.0 v2:172.21.15.123:6834/333722639 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x557b79eeda70 con 0x557b7a0bc0f0 2024-09-16T15:49:59.543 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:59.538+0000 7f02f67fc640 10 client.8073 handle_command_reply: tid=0 2024-09-16T15:49:59.543 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:59.538+0000 7f03209d3000 1 client.8073 shutdown 2024-09-16T15:49:59.543 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:59.538+0000 7f03209d3000 1 -- 172.21.15.123:0/1822767544 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x557b7a0da2f0 msgr2=0x557b7a0bb3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:59.543 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:59.538+0000 7f03209d3000 1 --2- 172.21.15.123:0/1822767544 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x557b7a0da2f0 0x557b7a0bb3b0 secure :-1 s=READY pgs=1421 cs=0 l=1 rev1=1 crypto rx=0x7f03100f5690 tx=0x7f03100f8ff0 comp rx=0 tx=0).stop 2024-09-16T15:49:59.543 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:59.538+0000 7f03209d3000 1 -- 172.21.15.123:0/1822767544 shutdown_connections 2024-09-16T15:49:59.543 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:59.538+0000 7f03209d3000 1 --2- 172.21.15.123:0/1822767544 >> [v2:172.21.15.123:6834/333722639,v1:172.21.15.123:6835/333722639] conn(0x557b7a0bc0f0 0x557b7a0ebc90 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:59.543 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:59.538+0000 7f03209d3000 1 --2- 172.21.15.123:0/1822767544 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x557b7a0da2f0 0x557b7a0bb3b0 unknown :-1 s=CLOSED pgs=1421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:59.543 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:59.538+0000 7f03209d3000 1 --2- 172.21.15.123:0/1822767544 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x557b7a0d99d0 0x557b7a0bae70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:59.544 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:59.538+0000 7f02f67fc640 0 client.8073 ms_handle_reset on v2:172.21.15.123:6834/333722639 2024-09-16T15:49:59.544 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:59.538+0000 7f03209d3000 1 --2- 172.21.15.123:0/1822767544 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x557b7a08de10 0x557b7a0ba6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:59.544 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:59.538+0000 7f03209d3000 1 -- 172.21.15.123:0/1822767544 >> 172.21.15.123:0/1822767544 conn(0x557b79f77130 msgr2=0x557b7a08cfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:49:59.544 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:59.538+0000 7f03209d3000 1 -- 172.21.15.123:0/1822767544 shutdown_connections 2024-09-16T15:49:59.544 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:59.538+0000 7f03209d3000 1 -- 172.21.15.123:0/1822767544 wait complete. 2024-09-16T15:49:59.544 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:59.538+0000 7f03209d3000 20 client.8073 trim_cache size 0 max 16384 2024-09-16T15:49:59.558 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:59.558+0000 7f03209d3000 1 Processor -- start 2024-09-16T15:49:59.558 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:59.558+0000 7f03209d3000 1 -- start start 2024-09-16T15:49:59.561 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:59.558+0000 7f03209d3000 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x557b7a08de10 0x557b7a14bbc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:49:59.561 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:59.558+0000 7f03209d3000 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x557b7a0d99d0 0x557b7a14c100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:49:59.561 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:59.558+0000 7f03209d3000 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x557b7a0da2f0 0x557b7a14c640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:49:59.561 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:59.558+0000 7f03209d3000 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f03181ff780 con 0x557b7a0da2f0 2024-09-16T15:49:59.561 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:59.558+0000 7f03209d3000 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f03181fe7e0 con 0x557b7a0d99d0 2024-09-16T15:49:59.561 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:59.558+0000 7f03209d3000 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f0318145760 con 0x557b7a08de10 2024-09-16T15:49:59.561 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:59.558+0000 7f03177fe640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x557b7a0da2f0 0x557b7a14c640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:49:59.561 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:59.558+0000 7f03177fe640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x557b7a0da2f0 0x557b7a14c640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:58858/0 (socket says 172.21.15.123:58858) 2024-09-16T15:49:59.562 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:59.558+0000 7f03177fe640 1 -- 172.21.15.123:0/1701440338 learned_addr learned my addr 172.21.15.123:0/1701440338 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:49:59.562 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:59.558+0000 7f0316ffd640 1 --2- 172.21.15.123:0/1701440338 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x557b7a08de10 0x557b7a14bbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:49:59.562 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:59.558+0000 7f03167fc640 1 --2- 172.21.15.123:0/1701440338 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x557b7a0d99d0 0x557b7a14c100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:49:59.562 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:59.558+0000 7f03177fe640 1 -- 172.21.15.123:0/1701440338 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x557b7a08de10 msgr2=0x557b7a14bbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T15:49:59.562 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:59.558+0000 7f03177fe640 1 --2- 172.21.15.123:0/1701440338 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x557b7a08de10 0x557b7a14bbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:59.562 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:59.558+0000 7f03177fe640 1 -- 172.21.15.123:0/1701440338 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x557b7a0d99d0 msgr2=0x557b7a14c100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T15:49:59.562 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:59.558+0000 7f03177fe640 1 --2- 172.21.15.123:0/1701440338 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x557b7a0d99d0 0x557b7a14c100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:59.562 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:59.558+0000 7f03177fe640 1 -- 172.21.15.123:0/1701440338 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557b79f36d90 con 0x557b7a0da2f0 2024-09-16T15:49:59.562 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:59.558+0000 7f03167fc640 1 --2- 172.21.15.123:0/1701440338 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x557b7a0d99d0 0x557b7a14c100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T15:49:59.562 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:59.558+0000 7f03177fe640 1 --2- 172.21.15.123:0/1701440338 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x557b7a0da2f0 0x557b7a14c640 secure :-1 s=READY pgs=1422 cs=0 l=1 rev1=1 crypto rx=0x7f03100a6070 tx=0x7f03100f9060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=36d6caa6c3cac4d3 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:49:59.562 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:59.558+0000 7f02fcff9640 1 -- 172.21.15.123:0/1701440338 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0310102d30 con 0x557b7a0da2f0 2024-09-16T15:49:59.562 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:59.558+0000 7f02fcff9640 1 -- 172.21.15.123:0/1701440338 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f03100f95e0 con 0x557b7a0da2f0 2024-09-16T15:49:59.562 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:59.558+0000 7f02fcff9640 1 -- 172.21.15.123:0/1701440338 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f031000f040 con 0x557b7a0da2f0 2024-09-16T15:49:59.563 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:59.562+0000 7f03209d3000 1 -- 172.21.15.123:0/1701440338 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x557b7a0da2f0 msgr2=0x557b7a14c640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:59.563 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:59.562+0000 7f03209d3000 1 --2- 172.21.15.123:0/1701440338 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x557b7a0da2f0 0x557b7a14c640 secure :-1 s=READY pgs=1422 cs=0 l=1 rev1=1 crypto rx=0x7f03100a6070 tx=0x7f03100f9060 comp rx=0 tx=0).stop 2024-09-16T15:49:59.563 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:59.562+0000 7f03209d3000 1 -- 172.21.15.123:0/1701440338 shutdown_connections 2024-09-16T15:49:59.563 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:59.562+0000 7f03209d3000 1 --2- 172.21.15.123:0/1701440338 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x557b7a0da2f0 0x557b7a14c640 unknown :-1 s=CLOSED pgs=1422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:59.563 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:59.562+0000 7f03209d3000 1 --2- 172.21.15.123:0/1701440338 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x557b7a0d99d0 0x557b7a14c100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:59.563 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:59.562+0000 7f03209d3000 1 --2- 172.21.15.123:0/1701440338 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x557b7a08de10 0x557b7a14bbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:59.563 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:59.562+0000 7f03209d3000 1 -- 172.21.15.123:0/1701440338 >> 172.21.15.123:0/1701440338 conn(0x557b79f77130 msgr2=0x557b7a095aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:49:59.563 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:59.562+0000 7f03209d3000 1 -- 172.21.15.123:0/1701440338 shutdown_connections 2024-09-16T15:49:59.563 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:59.562+0000 7f03209d3000 1 -- 172.21.15.123:0/1701440338 wait complete. 2024-09-16T15:49:59.563 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:59.562+0000 7f03209d3000 1 -- start start 2024-09-16T15:49:59.563 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:59.562+0000 7f03209d3000 1 Processor -- start 2024-09-16T15:49:59.567 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:59.562+0000 7f03209d3000 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x557b7a08de10 0x557b7a14f9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:49:59.567 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:59.562+0000 7f03209d3000 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x557b7a0d99d0 0x557b7a14ff30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:49:59.567 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:59.562+0000 7f03209d3000 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x557b7a0da2f0 0x557b7a150470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:49:59.567 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:59.562+0000 7f03209d3000 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x557b79eded60 con 0x557b7a08de10 2024-09-16T15:49:59.568 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:59.562+0000 7f03209d3000 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f03181fe190 con 0x557b7a0d99d0 2024-09-16T15:49:59.568 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:59.562+0000 7f03209d3000 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x557b79e755d0 con 0x557b7a0da2f0 2024-09-16T15:49:59.568 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:59.562+0000 7f03209d3000 10 client.0 _ll_register_callbacks cb 0x7f03100f93b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T15:49:59.568 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:59.562+0000 7f0316ffd640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x557b7a08de10 0x557b7a14f9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:49:59.568 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:59.562+0000 7f0316ffd640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x557b7a08de10 0x557b7a14f9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:58874/0 (socket says 172.21.15.123:58874) 2024-09-16T15:49:59.568 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:59.562+0000 7f0316ffd640 1 -- 172.21.15.123:0/2650130149 learned_addr learned my addr 172.21.15.123:0/2650130149 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:49:59.568 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:59.562+0000 7f0316ffd640 1 -- 172.21.15.123:0/2650130149 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x557b7a0da2f0 msgr2=0x557b7a150470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T15:49:59.568 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:59.562+0000 7f0316ffd640 1 --2- 172.21.15.123:0/2650130149 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x557b7a0da2f0 0x557b7a150470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:59.568 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:59.562+0000 7f0316ffd640 1 -- 172.21.15.123:0/2650130149 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x557b7a0d99d0 msgr2=0x557b7a14ff30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T15:49:59.568 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:59.562+0000 7f03167fc640 1 --2- 172.21.15.123:0/2650130149 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x557b7a0d99d0 0x557b7a14ff30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:49:59.568 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:59.562+0000 7f0316ffd640 1 --2- 172.21.15.123:0/2650130149 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x557b7a0d99d0 0x557b7a14ff30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:59.568 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:59.562+0000 7f0316ffd640 1 -- 172.21.15.123:0/2650130149 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557b79ea1a90 con 0x557b7a08de10 2024-09-16T15:49:59.569 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:59.562+0000 7f0316ffd640 1 --2- 172.21.15.123:0/2650130149 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x557b7a08de10 0x557b7a14f9f0 secure :-1 s=READY pgs=1423 cs=0 l=1 rev1=1 crypto rx=0x7f030800f3c0 tx=0x7f0308015000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f94195041c3a329d server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:49:59.569 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:59.562+0000 7f02f67fc640 10 client.0 ms_handle_connect on v2:172.21.15.123:3300/0 2024-09-16T15:49:59.569 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:59.562+0000 7f02f67fc640 1 -- 172.21.15.123:0/2650130149 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0308021650 con 0x557b7a08de10 2024-09-16T15:49:59.569 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:59.562+0000 7f02f67fc640 1 -- 172.21.15.123:0/2650130149 <== mon.0 v2:172.21.15.123:3300/0 2 ==== osd_map(135..135 src has 1..135) ==== 5737+0+0 (secure 0 0 0) 0x7f030802e930 con 0x557b7a08de10 2024-09-16T15:49:59.569 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:59.562+0000 7f03209d3000 1 -- 172.21.15.123:0/2650130149 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f03100f95e0 con 0x557b7a08de10 2024-09-16T15:49:59.569 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:59.562+0000 7f02f67fc640 1 -- 172.21.15.123:0/2650130149 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_get_version_reply(handle=1 version=86) ==== 24+0+0 (secure 0 0 0) 0x7f030802eb30 con 0x557b7a08de10 2024-09-16T15:49:59.569 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:59.562+0000 7f03209d3000 10 client.8079 fetch_fsmap learned FSMap version 86 2024-09-16T15:49:59.569 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:59.562+0000 7f03209d3000 1 -- 172.21.15.123:0/2650130149 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,fsmap=86,monmap=2+}) -- 0x557b79e80050 con 0x557b7a08de10 2024-09-16T15:49:59.569 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:59.562+0000 7f02f67fc640 1 -- 172.21.15.123:0/2650130149 <== mon.0 v2:172.21.15.123:3300/0 4 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f03100f95e0 con 0x557b7a08de10 2024-09-16T15:49:59.569 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:59.562+0000 7f02f67fc640 1 -- 172.21.15.123:0/2650130149 <== mon.0 v2:172.21.15.123:3300/0 5 ==== fsmap(e 86) ==== 3502+0+0 (secure 0 0 0) 0x7f030803b070 con 0x557b7a08de10 2024-09-16T15:49:59.569 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:59.562+0000 7f03209d3000 10 client.8079 fetch_fsmap finished waiting for FSMap version 86 2024-09-16T15:49:59.569 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:59.562+0000 7f03209d3000 10 client.8079 resolve_mds: resolved name 'a' to daemon mds.a 2024-09-16T15:49:59.569 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:59.562+0000 7f03209d3000 20 client.8079 populate_metadata read hostname 'smithi123' 2024-09-16T15:49:59.569 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:59.562+0000 7f03209d3000 1 --2- 172.21.15.123:0/2650130149 >> [v2:172.21.15.123:6834/333722639,v1:172.21.15.123:6835/333722639] conn(0x557b7a0ef340 0x557b7a158c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:49:59.570 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:59.562+0000 7f03209d3000 4 client.8079 mds_command: new command op to 7944 tid=0 multi_id=0 [{"prefix": "status"}] 2024-09-16T15:49:59.570 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:59.562+0000 7f03209d3000 1 -- 172.21.15.123:0/2650130149 --> [v2:172.21.15.123:6834/333722639,v1:172.21.15.123:6835/333722639] -- command(tid 0: {"prefix": "status"}) -- 0x557b79ee9520 con 0x557b7a0ef340 2024-09-16T15:49:59.570 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:59.562+0000 7f03167fc640 1 --2- 172.21.15.123:0/2650130149 >> [v2:172.21.15.123:6834/333722639,v1:172.21.15.123:6835/333722639] conn(0x557b7a0ef340 0x557b7a158c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:49:59.570 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:59.562+0000 7f02f67fc640 10 client.8079 ms_handle_connect on v2:172.21.15.123:6834/333722639 2024-09-16T15:49:59.570 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:59.562+0000 7f03167fc640 1 --2- 172.21.15.123:0/2650130149 >> [v2:172.21.15.123:6834/333722639,v1:172.21.15.123:6835/333722639] conn(0x557b7a0ef340 0x557b7a158c50 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6afec9e3cef77c41 server_cookie=77bda9f1aacbb906 in_seq=0 out_seq=0 2024-09-16T15:49:59.570 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:59.562+0000 7f02f67fc640 1 -- 172.21.15.123:0/2650130149 <== mds.0 v2:172.21.15.123:6834/333722639 1 ==== command_reply(tid 0: 0 ) ==== 8+0+333 (crc 0 0 0) 0x557b79ee9520 con 0x557b7a0ef340 2024-09-16T15:49:59.570 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:59.562+0000 7f02f67fc640 10 client.8079 handle_command_reply: tid=0 2024-09-16T15:49:59.570 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:59.562+0000 7f03209d3000 1 client.8079 shutdown 2024-09-16T15:49:59.570 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:59.562+0000 7f03209d3000 1 -- 172.21.15.123:0/2650130149 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x557b7a08de10 msgr2=0x557b7a14f9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:59.570 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:59.562+0000 7f03209d3000 1 --2- 172.21.15.123:0/2650130149 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x557b7a08de10 0x557b7a14f9f0 secure :-1 s=READY pgs=1423 cs=0 l=1 rev1=1 crypto rx=0x7f030800f3c0 tx=0x7f0308015000 comp rx=0 tx=0).stop 2024-09-16T15:49:59.570 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:59.562+0000 7f03209d3000 1 -- 172.21.15.123:0/2650130149 shutdown_connections 2024-09-16T15:49:59.570 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:59.562+0000 7f03209d3000 1 --2- 172.21.15.123:0/2650130149 >> [v2:172.21.15.123:6834/333722639,v1:172.21.15.123:6835/333722639] conn(0x557b7a0ef340 0x557b7a158c50 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:59.570 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:59.562+0000 7f03209d3000 1 --2- 172.21.15.123:0/2650130149 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x557b7a0da2f0 0x557b7a150470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:59.571 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:59.562+0000 7f02f67fc640 0 client.8079 ms_handle_reset on v2:172.21.15.123:6834/333722639 2024-09-16T15:49:59.571 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:59.562+0000 7f03209d3000 1 --2- 172.21.15.123:0/2650130149 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x557b7a0d99d0 0x557b7a14ff30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:59.571 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:59.562+0000 7f03209d3000 1 --2- 172.21.15.123:0/2650130149 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x557b7a08de10 0x557b7a14f9f0 unknown :-1 s=CLOSED pgs=1423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:59.571 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:59.562+0000 7f03209d3000 1 -- 172.21.15.123:0/2650130149 >> 172.21.15.123:0/2650130149 conn(0x557b79f77130 msgr2=0x557b7a0d85b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:49:59.571 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:59.562+0000 7f03209d3000 1 -- 172.21.15.123:0/2650130149 shutdown_connections 2024-09-16T15:49:59.571 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:59.562+0000 7f03209d3000 1 -- 172.21.15.123:0/2650130149 wait complete. 2024-09-16T15:49:59.571 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:59.562+0000 7f03209d3000 20 client.8079 trim_cache size 0 max 16384 2024-09-16T15:49:59.571 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2024-09-16T15:49:59.571 INFO:teuthology.orchestra.run.smithi123.stdout: "cluster_fsid": "56077f89-18c8-486f-ae7e-eecc0bb7822c", 2024-09-16T15:49:59.571 INFO:teuthology.orchestra.run.smithi123.stdout: "whoami": 0, 2024-09-16T15:49:59.571 INFO:teuthology.orchestra.run.smithi123.stdout: "id": 7944, 2024-09-16T15:49:59.572 INFO:teuthology.orchestra.run.smithi123.stdout: "want_state": "up:active", 2024-09-16T15:49:59.572 INFO:teuthology.orchestra.run.smithi123.stdout: "state": "up:active", 2024-09-16T15:49:59.572 INFO:teuthology.orchestra.run.smithi123.stdout: "fs_name": "cephfs", 2024-09-16T15:49:59.572 INFO:teuthology.orchestra.run.smithi123.stdout: "rank_uptime": 5.8238958140000001, 2024-09-16T15:49:59.572 INFO:teuthology.orchestra.run.smithi123.stdout: "mdsmap_epoch": 86, 2024-09-16T15:49:59.572 INFO:teuthology.orchestra.run.smithi123.stdout: "osdmap_epoch": 135, 2024-09-16T15:49:59.572 INFO:teuthology.orchestra.run.smithi123.stdout: "osdmap_epoch_barrier": 135, 2024-09-16T15:49:59.572 INFO:teuthology.orchestra.run.smithi123.stdout: "uptime": 10.489170854999999 2024-09-16T15:49:59.572 INFO:teuthology.orchestra.run.smithi123.stdout:} 2024-09-16T15:49:59.573 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:59.566+0000 7f031d8e9640 1 -- 172.21.15.123:0/1599040388 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f0304073520 msgr2=0x7f03040759e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:59.573 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:59.566+0000 7f031d8e9640 1 --2- 172.21.15.123:0/1599040388 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f0304073520 0x7f03040759e0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f0308000f70 tx=0x7f0308001670 comp rx=0 tx=0).stop 2024-09-16T15:49:59.573 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:59.566+0000 7f031d8e9640 1 -- 172.21.15.123:0/1599040388 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f03180c8170 msgr2=0x7f0318155f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:59.573 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:59.566+0000 7f031d8e9640 1 --2- 172.21.15.123:0/1599040388 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f03180c8170 0x7f0318155f90 secure :-1 s=READY pgs=1419 cs=0 l=1 rev1=1 crypto rx=0x7f0310002240 tx=0x7f03100add50 comp rx=0 tx=0).stop 2024-09-16T15:49:59.573 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:59.566+0000 7f031d8e9640 1 -- 172.21.15.123:0/1599040388 shutdown_connections 2024-09-16T15:49:59.573 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:59.566+0000 7f031d8e9640 1 --2- 172.21.15.123:0/1599040388 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f0304073520 0x7f03040759e0 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:59.573 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:59.566+0000 7f031d8e9640 1 --2- 172.21.15.123:0/1599040388 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f03180c8170 0x7f0318155f90 unknown :-1 s=CLOSED pgs=1419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:59.573 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:59.566+0000 7f031d8e9640 1 --2- 172.21.15.123:0/1599040388 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f0318159060 0x7f03180c7c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:59.574 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:59.566+0000 7f031d8e9640 1 --2- 172.21.15.123:0/1599040388 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f03181f1d30 0x7f0318158b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:59.574 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:59.566+0000 7f031d8e9640 1 -- 172.21.15.123:0/1599040388 >> 172.21.15.123:0/1599040388 conn(0x7f03180cfb10 msgr2=0x7f0318073ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:49:59.574 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:59.566+0000 7f031d8e9640 1 -- 172.21.15.123:0/1599040388 shutdown_connections 2024-09-16T15:49:59.574 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:59.566+0000 7f031d8e9640 1 -- 172.21.15.123:0/1599040388 wait complete. 2024-09-16T15:49:59.580 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-09-16T15:49:59.696 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:59.694+0000 7fd411c2d640 1 -- 172.21.15.123:0/3462107925 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd40c074470 msgr2=0x7fd40c0cd6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:59.696 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:59.694+0000 7fd411c2d640 1 --2- 172.21.15.123:0/3462107925 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd40c074470 0x7fd40c0cd6e0 secure :-1 s=READY pgs=1424 cs=0 l=1 rev1=1 crypto rx=0x7fd400009900 tx=0x7fd40001c850 comp rx=0 tx=0).stop 2024-09-16T15:49:59.696 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:59.694+0000 7fd411c2d640 1 -- 172.21.15.123:0/3462107925 shutdown_connections 2024-09-16T15:49:59.697 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:59.694+0000 7fd411c2d640 1 --2- 172.21.15.123:0/3462107925 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd40c074470 0x7fd40c0cd6e0 unknown :-1 s=CLOSED pgs=1424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:59.697 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:59.694+0000 7fd411c2d640 1 --2- 172.21.15.123:0/3462107925 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fd40c15e160 0x7fd40c073f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:59.697 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:59.694+0000 7fd411c2d640 1 -- 172.21.15.123:0/3462107925 >> 172.21.15.123:0/3462107925 conn(0x7fd40c0cfb10 msgr2=0x7fd40c0cff20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:49:59.697 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:59.694+0000 7fd411c2d640 1 -- 172.21.15.123:0/3462107925 shutdown_connections 2024-09-16T15:49:59.697 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:59.698+0000 7fd411c2d640 1 -- 172.21.15.123:0/3462107925 wait complete. 2024-09-16T15:49:59.697 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:59.698+0000 7fd411c2d640 1 Processor -- start 2024-09-16T15:49:59.697 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:59.698+0000 7fd411c2d640 1 -- start start 2024-09-16T15:49:59.699 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:59.698+0000 7fd411c2d640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fd40c074470 0x7fd40c151940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:49:59.700 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:59.698+0000 7fd411c2d640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd40c0cdc20 0x7fd40c0b4e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:49:59.700 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:59.698+0000 7fd411c2d640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd40c15e160 0x7fd40c0b5380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:49:59.700 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:59.698+0000 7fd411c2d640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fd40c167f50 con 0x7fd40c15e160 2024-09-16T15:49:59.700 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:59.698+0000 7fd411c2d640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fd40c167dd0 con 0x7fd40c0cdc20 2024-09-16T15:49:59.700 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:59.698+0000 7fd411c2d640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fd40c1680d0 con 0x7fd40c074470 2024-09-16T15:49:59.700 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:59.698+0000 7fd40bfff640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd40c15e160 0x7fd40c0b5380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:49:59.700 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:59.698+0000 7fd40bfff640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd40c15e160 0x7fd40c0b5380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:58890/0 (socket says 172.21.15.123:58890) 2024-09-16T15:49:59.700 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:59.698+0000 7fd40bfff640 1 -- 172.21.15.123:0/1789968404 learned_addr learned my addr 172.21.15.123:0/1789968404 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:49:59.700 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:59.698+0000 7fd40b7fe640 1 --2- 172.21.15.123:0/1789968404 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fd40c074470 0x7fd40c151940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:49:59.700 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:59.698+0000 7fd40affd640 1 --2- 172.21.15.123:0/1789968404 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd40c0cdc20 0x7fd40c0b4e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:49:59.700 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:59.698+0000 7fd40bfff640 1 -- 172.21.15.123:0/1789968404 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fd40c074470 msgr2=0x7fd40c151940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:59.701 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:59.698+0000 7fd40bfff640 1 --2- 172.21.15.123:0/1789968404 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fd40c074470 0x7fd40c151940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:59.701 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:59.698+0000 7fd40bfff640 1 -- 172.21.15.123:0/1789968404 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd40c0cdc20 msgr2=0x7fd40c0b4e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:59.701 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:59.698+0000 7fd40bfff640 1 --2- 172.21.15.123:0/1789968404 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd40c0cdc20 0x7fd40c0b4e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:59.701 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:59.698+0000 7fd40bfff640 1 -- 172.21.15.123:0/1789968404 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd40c14e500 con 0x7fd40c15e160 2024-09-16T15:49:59.701 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:59.698+0000 7fd40b7fe640 1 --2- 172.21.15.123:0/1789968404 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fd40c074470 0x7fd40c151940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T15:49:59.701 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:59.698+0000 7fd40bfff640 1 --2- 172.21.15.123:0/1789968404 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd40c15e160 0x7fd40c0b5380 secure :-1 s=READY pgs=1425 cs=0 l=1 rev1=1 crypto rx=0x7fd40400c410 tx=0x7fd40400c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:49:59.701 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:59.698+0000 7fd3f3fff640 1 -- 172.21.15.123:0/1789968404 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd404017070 con 0x7fd40c15e160 2024-09-16T15:49:59.701 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:59.698+0000 7fd3f3fff640 1 -- 172.21.15.123:0/1789968404 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fd404003600 con 0x7fd40c15e160 2024-09-16T15:49:59.701 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:59.698+0000 7fd411c2d640 1 -- 172.21.15.123:0/1789968404 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd40c14e230 con 0x7fd40c15e160 2024-09-16T15:49:59.701 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:59.698+0000 7fd3f3fff640 1 -- 172.21.15.123:0/1789968404 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd404003a50 con 0x7fd40c15e160 2024-09-16T15:49:59.701 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:59.698+0000 7fd411c2d640 1 -- 172.21.15.123:0/1789968404 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd40c14ea80 con 0x7fd40c15e160 2024-09-16T15:49:59.701 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:59.698+0000 7fd411c2d640 1 -- 172.21.15.123:0/1789968404 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd40c0cd6e0 con 0x7fd40c15e160 2024-09-16T15:49:59.702 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:59.702+0000 7fd3f3fff640 1 -- 172.21.15.123:0/1789968404 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 21) ==== 97641+0+0 (secure 0 0 0) 0x7fd40401a070 con 0x7fd40c15e160 2024-09-16T15:49:59.702 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:59.702+0000 7fd3f3fff640 1 --2- 172.21.15.123:0/1789968404 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fd3f8073520 0x7fd3f80759e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:49:59.702 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:59.702+0000 7fd3f3fff640 1 -- 172.21.15.123:0/1789968404 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(135..135 src has 1..135) ==== 5737+0+0 (secure 0 0 0) 0x7fd404093d30 con 0x7fd40c15e160 2024-09-16T15:49:59.702 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:59.702+0000 7fd40b7fe640 1 --2- 172.21.15.123:0/1789968404 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fd3f8073520 0x7fd3f80759e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:49:59.703 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:59.702+0000 7fd40b7fe640 1 --2- 172.21.15.123:0/1789968404 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fd3f8073520 0x7fd3f80759e0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7fd3fc008d60 tx=0x7fd3fc0082b0 comp rx=0 tx=0).ready entity=mgr.4150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:49:59.705 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:59.706+0000 7fd3f3fff640 1 -- 172.21.15.123:0/1789968404 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7fd40405e5a0 con 0x7fd40c15e160 2024-09-16T15:49:59.903 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:59.902+0000 7fd411c2d640 1 -- 172.21.15.123:0/1789968404 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7fd40c0cef90 con 0x7fd40c15e160 2024-09-16T15:49:59.903 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:59.902+0000 7fd3f3fff640 1 -- 172.21.15.123:0/1789968404 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v86) ==== 53+0+73 (secure 0 0 0) 0x7fd40405dea0 con 0x7fd40c15e160 2024-09-16T15:49:59.904 INFO:teuthology.orchestra.run.smithi123.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-09-16T15:49:59.906 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:59.906+0000 7fd411c2d640 1 -- 172.21.15.123:0/1789968404 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fd3f8073520 msgr2=0x7fd3f80759e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:59.906 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:59.906+0000 7fd411c2d640 1 --2- 172.21.15.123:0/1789968404 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fd3f8073520 0x7fd3f80759e0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7fd3fc008d60 tx=0x7fd3fc0082b0 comp rx=0 tx=0).stop 2024-09-16T15:49:59.906 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:59.906+0000 7fd411c2d640 1 -- 172.21.15.123:0/1789968404 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd40c15e160 msgr2=0x7fd40c0b5380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:49:59.906 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:59.906+0000 7fd411c2d640 1 --2- 172.21.15.123:0/1789968404 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd40c15e160 0x7fd40c0b5380 secure :-1 s=READY pgs=1425 cs=0 l=1 rev1=1 crypto rx=0x7fd40400c410 tx=0x7fd40400c8d0 comp rx=0 tx=0).stop 2024-09-16T15:49:59.907 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:59.906+0000 7fd411c2d640 1 -- 172.21.15.123:0/1789968404 shutdown_connections 2024-09-16T15:49:59.907 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:59.906+0000 7fd411c2d640 1 --2- 172.21.15.123:0/1789968404 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fd3f8073520 0x7fd3f80759e0 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:59.907 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:59.906+0000 7fd411c2d640 1 --2- 172.21.15.123:0/1789968404 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd40c15e160 0x7fd40c0b5380 unknown :-1 s=CLOSED pgs=1425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:59.907 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:59.906+0000 7fd411c2d640 1 --2- 172.21.15.123:0/1789968404 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd40c0cdc20 0x7fd40c0b4e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:59.907 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:59.906+0000 7fd411c2d640 1 --2- 172.21.15.123:0/1789968404 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fd40c074470 0x7fd40c151940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:49:59.907 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:59.906+0000 7fd411c2d640 1 -- 172.21.15.123:0/1789968404 >> 172.21.15.123:0/1789968404 conn(0x7fd40c0cfb10 msgr2=0x7fd40c0af5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:49:59.907 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:59.906+0000 7fd411c2d640 1 -- 172.21.15.123:0/1789968404 shutdown_connections 2024-09-16T15:49:59.907 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:49:59.906+0000 7fd411c2d640 1 -- 172.21.15.123:0/1789968404 wait complete. 2024-09-16T15:49:59.921 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-09-16T15:49:59.921 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-09-16T15:49:59.921 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi123.front.sepia.ceph.com 2024-09-16T15:49:59.922 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-09-16T15:49:59.922 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-09-16T15:49:59.922 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-09-16T15:49:59.922 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-09-16T15:49:59.922 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-09-16T15:49:59.922 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-09-16T15:49:59.922 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:49:59.922 DEBUG:teuthology.orchestra.run.smithi123:> ip addr 2024-09-16T15:49:59.927 INFO:teuthology.orchestra.run.smithi123.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-09-16T15:49:59.927 INFO:teuthology.orchestra.run.smithi123.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-09-16T15:49:59.927 INFO:teuthology.orchestra.run.smithi123.stdout: inet 127.0.0.1/8 scope host lo 2024-09-16T15:49:59.927 INFO:teuthology.orchestra.run.smithi123.stdout: valid_lft forever preferred_lft forever 2024-09-16T15:49:59.927 INFO:teuthology.orchestra.run.smithi123.stdout: inet6 ::1/128 scope host 2024-09-16T15:49:59.928 INFO:teuthology.orchestra.run.smithi123.stdout: valid_lft forever preferred_lft forever 2024-09-16T15:49:59.928 INFO:teuthology.orchestra.run.smithi123.stdout:2: eno1: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-09-16T15:49:59.928 INFO:teuthology.orchestra.run.smithi123.stdout: link/ether 0c:c4:7a:d8:da:6c brd ff:ff:ff:ff:ff:ff 2024-09-16T15:49:59.928 INFO:teuthology.orchestra.run.smithi123.stdout: altname enp7s0f0 2024-09-16T15:49:59.928 INFO:teuthology.orchestra.run.smithi123.stdout:3: eno2: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-09-16T15:49:59.928 INFO:teuthology.orchestra.run.smithi123.stdout: link/ether 0c:c4:7a:d8:da:6d brd ff:ff:ff:ff:ff:ff 2024-09-16T15:49:59.928 INFO:teuthology.orchestra.run.smithi123.stdout: altname enp7s0f1 2024-09-16T15:49:59.928 INFO:teuthology.orchestra.run.smithi123.stdout:4: enp3s0f0: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-09-16T15:49:59.928 INFO:teuthology.orchestra.run.smithi123.stdout: link/ether 0c:c4:7a:8f:d3:f0 brd ff:ff:ff:ff:ff:ff 2024-09-16T15:49:59.928 INFO:teuthology.orchestra.run.smithi123.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-09-16T15:49:59.928 INFO:teuthology.orchestra.run.smithi123.stdout: link/ether 0c:c4:7a:8f:d3:f1 brd ff:ff:ff:ff:ff:ff 2024-09-16T15:49:59.928 INFO:teuthology.orchestra.run.smithi123.stdout: inet 172.21.15.123/20 metric 100 brd 172.21.15.255 scope global dynamic enp3s0f1 2024-09-16T15:49:59.928 INFO:teuthology.orchestra.run.smithi123.stdout: valid_lft 42068sec preferred_lft 42068sec 2024-09-16T15:49:59.929 INFO:teuthology.orchestra.run.smithi123.stdout: inet6 fe80::ec4:7aff:fe8f:d3f1/64 scope link 2024-09-16T15:49:59.929 INFO:teuthology.orchestra.run.smithi123.stdout: valid_lft forever preferred_lft forever 2024-09-16T15:49:59.929 INFO:teuthology.orchestra.run.smithi123.stdout:6: docker0: mtu 1500 qdisc noqueue state DOWN group default 2024-09-16T15:49:59.929 INFO:teuthology.orchestra.run.smithi123.stdout: link/ether 02:42:dd:82:a2:6e brd ff:ff:ff:ff:ff:ff 2024-09-16T15:49:59.929 INFO:teuthology.orchestra.run.smithi123.stdout: inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0 2024-09-16T15:49:59.929 INFO:teuthology.orchestra.run.smithi123.stdout: valid_lft forever preferred_lft forever 2024-09-16T15:49:59.929 INFO:teuthology.orchestra.run.smithi123.stdout:7: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-09-16T15:49:59.929 INFO:teuthology.orchestra.run.smithi123.stdout: link/ether 5e:1f:84:6d:79:ae brd ff:ff:ff:ff:ff:ff 2024-09-16T15:49:59.929 INFO:teuthology.orchestra.run.smithi123.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-09-16T15:49:59.929 INFO:teuthology.orchestra.run.smithi123.stdout: valid_lft forever preferred_lft forever 2024-09-16T15:49:59.929 INFO:teuthology.orchestra.run.smithi123.stdout: inet6 fe80::5c1f:84ff:fe6d:79ae/64 scope link 2024-09-16T15:49:59.929 INFO:teuthology.orchestra.run.smithi123.stdout: valid_lft forever preferred_lft forever 2024-09-16T15:49:59.929 INFO:teuthology.orchestra.run.smithi123.stdout:8: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-09-16T15:49:59.930 INFO:teuthology.orchestra.run.smithi123.stdout: link/ether 0e:5c:f6:1f:82:2f brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-09-16T15:49:59.930 INFO:teuthology.orchestra.run.smithi123.stdout: inet6 fe80::c5c:f6ff:fe1f:822f/64 scope link 2024-09-16T15:49:59.930 INFO:teuthology.orchestra.run.smithi123.stdout: valid_lft forever preferred_lft forever 2024-09-16T15:49:59.930 INFO:teuthology.orchestra.run.smithi123.stdout:9: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-09-16T15:49:59.930 INFO:teuthology.orchestra.run.smithi123.stdout: link/ether c6:a6:3d:69:d0:db brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-09-16T15:49:59.930 INFO:teuthology.orchestra.run.smithi123.stdout: inet6 fe80::c4a6:3dff:fe69:d0db/64 scope link 2024-09-16T15:49:59.930 INFO:teuthology.orchestra.run.smithi123.stdout: valid_lft forever preferred_lft forever 2024-09-16T15:49:59.930 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:49:59.930 DEBUG:teuthology.orchestra.run.smithi123:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-09-16T15:49:59.978 INFO:teuthology.orchestra.run.smithi123.stdout:1 2024-09-16T15:49:59.979 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:49:59.979 DEBUG:teuthology.orchestra.run.smithi123:> route 2024-09-16T15:50:00.027 INFO:teuthology.orchestra.run.smithi123.stdout:Kernel IP routing table 2024-09-16T15:50:00.027 INFO:teuthology.orchestra.run.smithi123.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-09-16T15:50:00.027 INFO:teuthology.orchestra.run.smithi123.stdout:default _gateway 0.0.0.0 UG 0 0 0 enp3s0f1 2024-09-16T15:50:00.027 INFO:teuthology.orchestra.run.smithi123.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-09-16T15:50:00.027 INFO:teuthology.orchestra.run.smithi123.stdout:158.69.67.47 _gateway 255.255.255.255 UGH 100 0 0 enp3s0f1 2024-09-16T15:50:00.027 INFO:teuthology.orchestra.run.smithi123.stdout:172.17.0.0 0.0.0.0 255.255.0.0 U 0 0 0 docker0 2024-09-16T15:50:00.027 INFO:teuthology.orchestra.run.smithi123.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-09-16T15:50:00.027 INFO:teuthology.orchestra.run.smithi123.stdout:ns1.front.sepia 0.0.0.0 255.255.255.255 UH 100 0 0 enp3s0f1 2024-09-16T15:50:00.027 INFO:teuthology.orchestra.run.smithi123.stdout:172.21.0.2 0.0.0.0 255.255.255.255 UH 100 0 0 enp3s0f1 2024-09-16T15:50:00.027 INFO:teuthology.orchestra.run.smithi123.stdout:_gateway 0.0.0.0 255.255.255.255 UH 100 0 0 enp3s0f1 2024-09-16T15:50:00.027 INFO:teuthology.orchestra.run.smithi123.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-09-16T15:50:00.028 DEBUG:teuthology.orchestra.run.smithi123:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-09-16T15:50:00.028 DEBUG:teuthology.orchestra.run.smithi123:> set -e 2024-09-16T15:50:00.028 DEBUG:teuthology.orchestra.run.smithi123:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-09-16T15:50:00.028 DEBUG:teuthology.orchestra.run.smithi123:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-09-16T15:50:00.028 DEBUG:teuthology.orchestra.run.smithi123:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-09-16T15:50:00.028 DEBUG:teuthology.orchestra.run.smithi123:> ') 2024-09-16T15:50:00.110 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:00 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-09-16T15:50:00.138 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:00 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-16T15:50:00.144 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:50:00.145 DEBUG:teuthology.orchestra.run.smithi123:> ip netns list 2024-09-16T15:50:00.192 INFO:teuthology.orchestra.run.smithi123.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-09-16T15:50:00.192 INFO:teuthology.orchestra.run.smithi123.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-09-16T15:50:00.193 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-09-16T15:50:00.193 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:50:00.193 DEBUG:teuthology.orchestra.run.smithi123:> sudo ip link set brx.0 up 2024-09-16T15:50:00.248 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2024-09-16T15:50:00.249 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-09-16T15:50:00.249 DEBUG:teuthology.orchestra.run.smithi123:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-09-16T15:50:00.296 INFO:teuthology.orchestra.run.smithi123.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-09-16T15:50:00.297 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-09-16T15:50:00.297 DEBUG:teuthology.orchestra.run.smithi123:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-09-16T15:50:00.345 DEBUG:teuthology.orchestra.run.smithi123:> sudo modprobe fuse 2024-09-16T15:50:00.402 DEBUG:teuthology.orchestra.run.smithi123:> cat /proc/self/mounts | awk '{print $2}' 2024-09-16T15:50:00.450 INFO:teuthology.orchestra.run.smithi123.stdout:/sys 2024-09-16T15:50:00.450 INFO:teuthology.orchestra.run.smithi123.stdout:/proc 2024-09-16T15:50:00.450 INFO:teuthology.orchestra.run.smithi123.stdout:/dev 2024-09-16T15:50:00.450 INFO:teuthology.orchestra.run.smithi123.stdout:/dev/pts 2024-09-16T15:50:00.450 INFO:teuthology.orchestra.run.smithi123.stdout:/run 2024-09-16T15:50:00.450 INFO:teuthology.orchestra.run.smithi123.stdout:/ 2024-09-16T15:50:00.450 INFO:teuthology.orchestra.run.smithi123.stdout:/sys/kernel/security 2024-09-16T15:50:00.450 INFO:teuthology.orchestra.run.smithi123.stdout:/dev/shm 2024-09-16T15:50:00.450 INFO:teuthology.orchestra.run.smithi123.stdout:/run/lock 2024-09-16T15:50:00.450 INFO:teuthology.orchestra.run.smithi123.stdout:/sys/fs/cgroup 2024-09-16T15:50:00.450 INFO:teuthology.orchestra.run.smithi123.stdout:/sys/fs/pstore 2024-09-16T15:50:00.450 INFO:teuthology.orchestra.run.smithi123.stdout:/sys/fs/bpf 2024-09-16T15:50:00.451 INFO:teuthology.orchestra.run.smithi123.stdout:/proc/sys/fs/binfmt_misc 2024-09-16T15:50:00.451 INFO:teuthology.orchestra.run.smithi123.stdout:/dev/hugepages 2024-09-16T15:50:00.451 INFO:teuthology.orchestra.run.smithi123.stdout:/dev/mqueue 2024-09-16T15:50:00.451 INFO:teuthology.orchestra.run.smithi123.stdout:/sys/kernel/debug 2024-09-16T15:50:00.451 INFO:teuthology.orchestra.run.smithi123.stdout:/sys/kernel/tracing 2024-09-16T15:50:00.451 INFO:teuthology.orchestra.run.smithi123.stdout:/proc/fs/nfsd 2024-09-16T15:50:00.451 INFO:teuthology.orchestra.run.smithi123.stdout:/sys/fs/fuse/connections 2024-09-16T15:50:00.451 INFO:teuthology.orchestra.run.smithi123.stdout:/sys/kernel/config 2024-09-16T15:50:00.451 INFO:teuthology.orchestra.run.smithi123.stdout:/run/credentials/systemd-sysusers.service 2024-09-16T15:50:00.451 INFO:teuthology.orchestra.run.smithi123.stdout:/run/rpc_pipefs 2024-09-16T15:50:00.451 INFO:teuthology.orchestra.run.smithi123.stdout:/run/user/1000 2024-09-16T15:50:00.451 INFO:teuthology.orchestra.run.smithi123.stdout:/var/lib/ceph 2024-09-16T15:50:00.451 INFO:teuthology.orchestra.run.smithi123.stdout:/var/lib/ceph/osd/ceph-0 2024-09-16T15:50:00.452 INFO:teuthology.orchestra.run.smithi123.stdout:/var/lib/ceph/osd/ceph-1 2024-09-16T15:50:00.452 INFO:teuthology.orchestra.run.smithi123.stdout:/var/lib/ceph/osd/ceph-2 2024-09-16T15:50:00.452 INFO:teuthology.orchestra.run.smithi123.stdout:/var/lib/ceph/osd/ceph-3 2024-09-16T15:50:00.452 INFO:teuthology.orchestra.run.smithi123.stdout:/run/netns 2024-09-16T15:50:00.452 INFO:teuthology.orchestra.run.smithi123.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-09-16T15:50:00.452 INFO:teuthology.orchestra.run.smithi123.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-09-16T15:50:00.452 INFO:teuthology.orchestra.run.smithi123.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-09-16T15:50:00.452 INFO:teuthology.orchestra.run.smithi123.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-09-16T15:50:00.453 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:50:00.453 DEBUG:teuthology.orchestra.run.smithi123:> ls /sys/fs/fuse/connections 2024-09-16T15:50:00.501 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-09-16T15:50:00.501 DEBUG:teuthology.orchestra.run.smithi123:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2024-09-16T15:50:00.546 DEBUG:teuthology.orchestra.run.smithi123:> sudo modprobe fuse 2024-09-16T15:50:00.558 DEBUG:teuthology.orchestra.run.smithi123:> cat /proc/self/mounts | awk '{print $2}' 2024-09-16T15:50:00.605 INFO:teuthology.orchestra.run.smithi123.stdout:/sys 2024-09-16T15:50:00.605 INFO:teuthology.orchestra.run.smithi123.stdout:/proc 2024-09-16T15:50:00.606 INFO:teuthology.orchestra.run.smithi123.stdout:/dev 2024-09-16T15:50:00.606 INFO:teuthology.orchestra.run.smithi123.stdout:/dev/pts 2024-09-16T15:50:00.606 INFO:teuthology.orchestra.run.smithi123.stdout:/run 2024-09-16T15:50:00.606 INFO:teuthology.orchestra.run.smithi123.stdout:/ 2024-09-16T15:50:00.606 INFO:teuthology.orchestra.run.smithi123.stdout:/sys/kernel/security 2024-09-16T15:50:00.606 INFO:teuthology.orchestra.run.smithi123.stdout:/dev/shm 2024-09-16T15:50:00.606 INFO:teuthology.orchestra.run.smithi123.stdout:/run/lock 2024-09-16T15:50:00.606 INFO:teuthology.orchestra.run.smithi123.stdout:/sys/fs/cgroup 2024-09-16T15:50:00.606 INFO:teuthology.orchestra.run.smithi123.stdout:/sys/fs/pstore 2024-09-16T15:50:00.606 INFO:teuthology.orchestra.run.smithi123.stdout:/sys/fs/bpf 2024-09-16T15:50:00.606 INFO:teuthology.orchestra.run.smithi123.stdout:/proc/sys/fs/binfmt_misc 2024-09-16T15:50:00.606 INFO:teuthology.orchestra.run.smithi123.stdout:/dev/hugepages 2024-09-16T15:50:00.606 INFO:teuthology.orchestra.run.smithi123.stdout:/dev/mqueue 2024-09-16T15:50:00.606 INFO:teuthology.orchestra.run.smithi123.stdout:/sys/kernel/debug 2024-09-16T15:50:00.607 INFO:teuthology.orchestra.run.smithi123.stdout:/sys/kernel/tracing 2024-09-16T15:50:00.607 INFO:teuthology.orchestra.run.smithi123.stdout:/proc/fs/nfsd 2024-09-16T15:50:00.607 INFO:teuthology.orchestra.run.smithi123.stdout:/sys/fs/fuse/connections 2024-09-16T15:50:00.607 INFO:teuthology.orchestra.run.smithi123.stdout:/sys/kernel/config 2024-09-16T15:50:00.607 INFO:teuthology.orchestra.run.smithi123.stdout:/run/credentials/systemd-sysusers.service 2024-09-16T15:50:00.607 INFO:teuthology.orchestra.run.smithi123.stdout:/run/rpc_pipefs 2024-09-16T15:50:00.607 INFO:teuthology.orchestra.run.smithi123.stdout:/run/user/1000 2024-09-16T15:50:00.607 INFO:teuthology.orchestra.run.smithi123.stdout:/var/lib/ceph 2024-09-16T15:50:00.607 INFO:teuthology.orchestra.run.smithi123.stdout:/var/lib/ceph/osd/ceph-0 2024-09-16T15:50:00.607 INFO:teuthology.orchestra.run.smithi123.stdout:/var/lib/ceph/osd/ceph-1 2024-09-16T15:50:00.607 INFO:teuthology.orchestra.run.smithi123.stdout:/var/lib/ceph/osd/ceph-2 2024-09-16T15:50:00.607 INFO:teuthology.orchestra.run.smithi123.stdout:/var/lib/ceph/osd/ceph-3 2024-09-16T15:50:00.607 INFO:teuthology.orchestra.run.smithi123.stdout:/run/netns 2024-09-16T15:50:00.608 INFO:teuthology.orchestra.run.smithi123.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-09-16T15:50:00.608 INFO:teuthology.orchestra.run.smithi123.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-09-16T15:50:00.608 INFO:teuthology.orchestra.run.smithi123.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-09-16T15:50:00.608 INFO:teuthology.orchestra.run.smithi123.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-09-16T15:50:00.608 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:50:00.608 DEBUG:teuthology.orchestra.run.smithi123:> ls /sys/fs/fuse/connections 2024-09-16T15:50:00.626 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi123.stderr:2024-09-16T15:50:00.626+0000 7ff8566b4580 -1 init, newargv = 0x55b3e4fd9ca0 newargc=15 2024-09-16T15:50:00.626 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi123.stderr:ceph-fuse[49774]: starting ceph client 2024-09-16T15:50:00.631 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi123.stderr:ceph-fuse[49774]: starting fuse 2024-09-16T15:50:01.733 DEBUG:teuthology.orchestra.run.smithi123:> sudo modprobe fuse 2024-09-16T15:50:01.749 DEBUG:teuthology.orchestra.run.smithi123:> cat /proc/self/mounts | awk '{print $2}' 2024-09-16T15:50:01.799 INFO:teuthology.orchestra.run.smithi123.stdout:/sys 2024-09-16T15:50:01.800 INFO:teuthology.orchestra.run.smithi123.stdout:/proc 2024-09-16T15:50:01.800 INFO:teuthology.orchestra.run.smithi123.stdout:/dev 2024-09-16T15:50:01.800 INFO:teuthology.orchestra.run.smithi123.stdout:/dev/pts 2024-09-16T15:50:01.800 INFO:teuthology.orchestra.run.smithi123.stdout:/run 2024-09-16T15:50:01.800 INFO:teuthology.orchestra.run.smithi123.stdout:/ 2024-09-16T15:50:01.800 INFO:teuthology.orchestra.run.smithi123.stdout:/sys/kernel/security 2024-09-16T15:50:01.800 INFO:teuthology.orchestra.run.smithi123.stdout:/dev/shm 2024-09-16T15:50:01.800 INFO:teuthology.orchestra.run.smithi123.stdout:/run/lock 2024-09-16T15:50:01.800 INFO:teuthology.orchestra.run.smithi123.stdout:/sys/fs/cgroup 2024-09-16T15:50:01.800 INFO:teuthology.orchestra.run.smithi123.stdout:/sys/fs/pstore 2024-09-16T15:50:01.800 INFO:teuthology.orchestra.run.smithi123.stdout:/sys/fs/bpf 2024-09-16T15:50:01.800 INFO:teuthology.orchestra.run.smithi123.stdout:/proc/sys/fs/binfmt_misc 2024-09-16T15:50:01.801 INFO:teuthology.orchestra.run.smithi123.stdout:/dev/hugepages 2024-09-16T15:50:01.801 INFO:teuthology.orchestra.run.smithi123.stdout:/dev/mqueue 2024-09-16T15:50:01.801 INFO:teuthology.orchestra.run.smithi123.stdout:/sys/kernel/debug 2024-09-16T15:50:01.801 INFO:teuthology.orchestra.run.smithi123.stdout:/sys/kernel/tracing 2024-09-16T15:50:01.801 INFO:teuthology.orchestra.run.smithi123.stdout:/proc/fs/nfsd 2024-09-16T15:50:01.801 INFO:teuthology.orchestra.run.smithi123.stdout:/sys/fs/fuse/connections 2024-09-16T15:50:01.801 INFO:teuthology.orchestra.run.smithi123.stdout:/sys/kernel/config 2024-09-16T15:50:01.801 INFO:teuthology.orchestra.run.smithi123.stdout:/run/credentials/systemd-sysusers.service 2024-09-16T15:50:01.801 INFO:teuthology.orchestra.run.smithi123.stdout:/run/rpc_pipefs 2024-09-16T15:50:01.801 INFO:teuthology.orchestra.run.smithi123.stdout:/run/user/1000 2024-09-16T15:50:01.801 INFO:teuthology.orchestra.run.smithi123.stdout:/var/lib/ceph 2024-09-16T15:50:01.801 INFO:teuthology.orchestra.run.smithi123.stdout:/var/lib/ceph/osd/ceph-0 2024-09-16T15:50:01.801 INFO:teuthology.orchestra.run.smithi123.stdout:/var/lib/ceph/osd/ceph-1 2024-09-16T15:50:01.802 INFO:teuthology.orchestra.run.smithi123.stdout:/var/lib/ceph/osd/ceph-2 2024-09-16T15:50:01.802 INFO:teuthology.orchestra.run.smithi123.stdout:/var/lib/ceph/osd/ceph-3 2024-09-16T15:50:01.802 INFO:teuthology.orchestra.run.smithi123.stdout:/run/netns 2024-09-16T15:50:01.802 INFO:teuthology.orchestra.run.smithi123.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-09-16T15:50:01.802 INFO:teuthology.orchestra.run.smithi123.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-09-16T15:50:01.802 INFO:teuthology.orchestra.run.smithi123.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-09-16T15:50:01.802 INFO:teuthology.orchestra.run.smithi123.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-09-16T15:50:01.802 INFO:teuthology.orchestra.run.smithi123.stdout:/home/ubuntu/cephtest/mnt.0 2024-09-16T15:50:01.803 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:50:01.803 DEBUG:teuthology.orchestra.run.smithi123:> ls /sys/fs/fuse/connections 2024-09-16T15:50:01.849 INFO:teuthology.orchestra.run.smithi123.stdout:45 2024-09-16T15:50:01.850 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [45] 2024-09-16T15:50:01.850 DEBUG:teuthology.orchestra.run.smithi123:> sudo stdin-killer -- python3 -c ' 2024-09-16T15:50:01.850 DEBUG:teuthology.orchestra.run.smithi123:> import glob 2024-09-16T15:50:01.850 DEBUG:teuthology.orchestra.run.smithi123:> import re 2024-09-16T15:50:01.850 DEBUG:teuthology.orchestra.run.smithi123:> import os 2024-09-16T15:50:01.850 DEBUG:teuthology.orchestra.run.smithi123:> import subprocess 2024-09-16T15:50:01.850 DEBUG:teuthology.orchestra.run.smithi123:> 2024-09-16T15:50:01.850 DEBUG:teuthology.orchestra.run.smithi123:> def _find_admin_socket(client_name): 2024-09-16T15:50:01.850 DEBUG:teuthology.orchestra.run.smithi123:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2024-09-16T15:50:01.850 DEBUG:teuthology.orchestra.run.smithi123:> files = glob.glob(asok_path) 2024-09-16T15:50:01.850 DEBUG:teuthology.orchestra.run.smithi123:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-09-16T15:50:01.851 DEBUG:teuthology.orchestra.run.smithi123:> 2024-09-16T15:50:01.851 DEBUG:teuthology.orchestra.run.smithi123:> # Given a non-glob path, it better be there 2024-09-16T15:50:01.851 DEBUG:teuthology.orchestra.run.smithi123:> if "*" not in asok_path: 2024-09-16T15:50:01.851 DEBUG:teuthology.orchestra.run.smithi123:> assert(len(files) == 1) 2024-09-16T15:50:01.851 DEBUG:teuthology.orchestra.run.smithi123:> return files[0] 2024-09-16T15:50:01.851 DEBUG:teuthology.orchestra.run.smithi123:> 2024-09-16T15:50:01.851 DEBUG:teuthology.orchestra.run.smithi123:> for f in files: 2024-09-16T15:50:01.851 DEBUG:teuthology.orchestra.run.smithi123:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-09-16T15:50:01.851 DEBUG:teuthology.orchestra.run.smithi123:> if os.path.exists("/proc/{0}".format(pid)): 2024-09-16T15:50:01.851 DEBUG:teuthology.orchestra.run.smithi123:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-09-16T15:50:01.851 DEBUG:teuthology.orchestra.run.smithi123:> contents = proc_f.read() 2024-09-16T15:50:01.851 DEBUG:teuthology.orchestra.run.smithi123:> if mountpoint in contents: 2024-09-16T15:50:01.851 DEBUG:teuthology.orchestra.run.smithi123:> return f 2024-09-16T15:50:01.851 DEBUG:teuthology.orchestra.run.smithi123:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-09-16T15:50:01.851 DEBUG:teuthology.orchestra.run.smithi123:> 2024-09-16T15:50:01.851 DEBUG:teuthology.orchestra.run.smithi123:> print(_find_admin_socket("client.0")) 2024-09-16T15:50:01.851 DEBUG:teuthology.orchestra.run.smithi123:> ' 2024-09-16T15:50:01.974 INFO:teuthology.orchestra.run.smithi123.stdout:/var/run/ceph/ceph-client.0.49774.asok 2024-09-16T15:50:01.978 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:01 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-16T15:50:01.986 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.49774.asok 2024-09-16T15:50:01.987 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:50:01.987 DEBUG:teuthology.orchestra.run.smithi123:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.49774.asok status 2024-09-16T15:50:02.137 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2024-09-16T15:50:02.138 INFO:teuthology.orchestra.run.smithi123.stdout: "metadata": { 2024-09-16T15:50:02.138 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph_sha1": "33f95d2ae1bcbe7eb04404303f179eeeaa6982df", 2024-09-16T15:50:02.138 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph_version": "ceph version 19.3.0-4938-g33f95d2a (33f95d2ae1bcbe7eb04404303f179eeeaa6982df) squid (dev)", 2024-09-16T15:50:02.138 INFO:teuthology.orchestra.run.smithi123.stdout: "entity_id": "0", 2024-09-16T15:50:02.138 INFO:teuthology.orchestra.run.smithi123.stdout: "hostname": "smithi123", 2024-09-16T15:50:02.138 INFO:teuthology.orchestra.run.smithi123.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2024-09-16T15:50:02.138 INFO:teuthology.orchestra.run.smithi123.stdout: "pid": "49774", 2024-09-16T15:50:02.138 INFO:teuthology.orchestra.run.smithi123.stdout: "root": "/" 2024-09-16T15:50:02.138 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:50:02.138 INFO:teuthology.orchestra.run.smithi123.stdout: "dentry_count": 0, 2024-09-16T15:50:02.138 INFO:teuthology.orchestra.run.smithi123.stdout: "dentry_pinned_count": 0, 2024-09-16T15:50:02.138 INFO:teuthology.orchestra.run.smithi123.stdout: "id": 8091, 2024-09-16T15:50:02.138 INFO:teuthology.orchestra.run.smithi123.stdout: "inst": { 2024-09-16T15:50:02.138 INFO:teuthology.orchestra.run.smithi123.stdout: "name": { 2024-09-16T15:50:02.139 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "client", 2024-09-16T15:50:02.139 INFO:teuthology.orchestra.run.smithi123.stdout: "num": 8091 2024-09-16T15:50:02.139 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:50:02.139 INFO:teuthology.orchestra.run.smithi123.stdout: "addr": { 2024-09-16T15:50:02.139 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "v1", 2024-09-16T15:50:02.139 INFO:teuthology.orchestra.run.smithi123.stdout: "addr": "192.168.0.1:0", 2024-09-16T15:50:02.139 INFO:teuthology.orchestra.run.smithi123.stdout: "nonce": 754969012 2024-09-16T15:50:02.139 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-16T15:50:02.139 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:50:02.139 INFO:teuthology.orchestra.run.smithi123.stdout: "addr": { 2024-09-16T15:50:02.139 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "v1", 2024-09-16T15:50:02.139 INFO:teuthology.orchestra.run.smithi123.stdout: "addr": "192.168.0.1:0", 2024-09-16T15:50:02.139 INFO:teuthology.orchestra.run.smithi123.stdout: "nonce": 754969012 2024-09-16T15:50:02.139 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:50:02.139 INFO:teuthology.orchestra.run.smithi123.stdout: "inst_str": "client.8091 192.168.0.1:0/754969012", 2024-09-16T15:50:02.140 INFO:teuthology.orchestra.run.smithi123.stdout: "addr_str": "192.168.0.1:0/754969012", 2024-09-16T15:50:02.140 INFO:teuthology.orchestra.run.smithi123.stdout: "inode_count": 1, 2024-09-16T15:50:02.140 INFO:teuthology.orchestra.run.smithi123.stdout: "mds_epoch": 86, 2024-09-16T15:50:02.140 INFO:teuthology.orchestra.run.smithi123.stdout: "osd_epoch": 135, 2024-09-16T15:50:02.140 INFO:teuthology.orchestra.run.smithi123.stdout: "osd_epoch_barrier": 0, 2024-09-16T15:50:02.140 INFO:teuthology.orchestra.run.smithi123.stdout: "blocklisted": false, 2024-09-16T15:50:02.140 INFO:teuthology.orchestra.run.smithi123.stdout: "fs_name": "cephfs" 2024-09-16T15:50:02.140 INFO:teuthology.orchestra.run.smithi123.stdout:} 2024-09-16T15:50:02.152 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:50:02.152 DEBUG:teuthology.orchestra.run.smithi123:> stat --file-system '--printf=%T 2024-09-16T15:50:02.153 DEBUG:teuthology.orchestra.run.smithi123:> ' -- /home/ubuntu/cephtest/mnt.0 2024-09-16T15:50:02.159 INFO:teuthology.orchestra.run.smithi123.stdout:fuseblk 2024-09-16T15:50:02.159 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2024-09-16T15:50:02.159 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:50:02.159 DEBUG:teuthology.orchestra.run.smithi123:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-09-16T15:50:02.216 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-09-16T15:50:02.389 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:02.390+0000 7fbb88e96640 1 -- 172.21.15.123:0/1982833323 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbb70004810 msgr2=0x7fbb70004c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:50:02.390 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:02.390+0000 7fbb88e96640 1 --2- 172.21.15.123:0/1982833323 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbb70004810 0x7fbb70004c90 secure :-1 s=READY pgs=1429 cs=0 l=1 rev1=1 crypto rx=0x7fbb6c009b30 tx=0x7fbb6c01d270 comp rx=0 tx=0).stop 2024-09-16T15:50:02.390 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:02.390+0000 7fbb82d9d640 1 -- 172.21.15.123:0/1982833323 reap_dead start 2024-09-16T15:50:02.390 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:02.390+0000 7fbb88e96640 1 -- 172.21.15.123:0/1982833323 shutdown_connections 2024-09-16T15:50:02.390 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:02.390+0000 7fbb88e96640 1 -- 172.21.15.123:0/1982833323 >> 172.21.15.123:0/1982833323 conn(0x7fbb840cfb10 msgr2=0x7fbb840cff20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:50:02.390 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:02.390+0000 7fbb88e96640 1 -- 172.21.15.123:0/1982833323 shutdown_connections 2024-09-16T15:50:02.390 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:02.390+0000 7fbb88e96640 1 -- 172.21.15.123:0/1982833323 wait complete. 2024-09-16T15:50:02.391 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:02.390+0000 7fbb88e96640 1 Processor -- start 2024-09-16T15:50:02.391 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:02.390+0000 7fbb88e96640 1 -- start start 2024-09-16T15:50:02.392 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:02.390+0000 7fbb88e96640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fbb70004810 0x7fbb840ae2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:50:02.392 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:02.390+0000 7fbb88e96640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbb8415d010 0x7fbb841f82e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:50:02.393 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:02.390+0000 7fbb88e96640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fbb8415d490 0x7fbb841f8820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:50:02.393 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:02.390+0000 7fbb88e96640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fbb84165d30 con 0x7fbb8415d010 2024-09-16T15:50:02.393 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:02.390+0000 7fbb88e96640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fbb84165bb0 con 0x7fbb8415d490 2024-09-16T15:50:02.393 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:02.390+0000 7fbb88e96640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fbb84165eb0 con 0x7fbb70004810 2024-09-16T15:50:02.393 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:02.390+0000 7fbb79d9b640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbb8415d010 0x7fbb841f82e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:50:02.393 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:02.390+0000 7fbb79d9b640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbb8415d010 0x7fbb841f82e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:58902/0 (socket says 172.21.15.123:58902) 2024-09-16T15:50:02.393 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:02.390+0000 7fbb79d9b640 1 -- 172.21.15.123:0/4043844443 learned_addr learned my addr 172.21.15.123:0/4043844443 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:50:02.393 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:02.390+0000 7fbb8259c640 1 --2- 172.21.15.123:0/4043844443 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fbb70004810 0x7fbb840ae2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:50:02.393 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:02.390+0000 7fbb82d9d640 1 --2- 172.21.15.123:0/4043844443 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fbb8415d490 0x7fbb841f8820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:50:02.393 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:02.390+0000 7fbb79d9b640 1 -- 172.21.15.123:0/4043844443 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fbb70004810 msgr2=0x7fbb840ae2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:50:02.393 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:02.390+0000 7fbb79d9b640 1 --2- 172.21.15.123:0/4043844443 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fbb70004810 0x7fbb840ae2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:02.393 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:02.390+0000 7fbb79d9b640 1 -- 172.21.15.123:0/4043844443 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fbb8415d490 msgr2=0x7fbb841f8820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:50:02.394 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:02.394+0000 7fbb79d9b640 1 --2- 172.21.15.123:0/4043844443 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fbb8415d490 0x7fbb841f8820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:02.394 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:02.394+0000 7fbb79d9b640 1 -- 172.21.15.123:0/4043844443 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbb840ae810 con 0x7fbb8415d010 2024-09-16T15:50:02.394 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:02.394+0000 7fbb8259c640 1 --2- 172.21.15.123:0/4043844443 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fbb70004810 0x7fbb840ae2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T15:50:02.394 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:02.394+0000 7fbb79d9b640 1 --2- 172.21.15.123:0/4043844443 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbb8415d010 0x7fbb841f82e0 secure :-1 s=READY pgs=1430 cs=0 l=1 rev1=1 crypto rx=0x7fbb6c0af040 tx=0x7fbb6c0acdb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:50:02.394 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:02.394+0000 7fbb7b7fe640 1 -- 172.21.15.123:0/4043844443 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb6c0b1040 con 0x7fbb8415d010 2024-09-16T15:50:02.394 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:02.394+0000 7fbb7b7fe640 1 -- 172.21.15.123:0/4043844443 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fbb6c005070 con 0x7fbb8415d010 2024-09-16T15:50:02.394 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:02.394+0000 7fbb7b7fe640 1 -- 172.21.15.123:0/4043844443 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb6c009730 con 0x7fbb8415d010 2024-09-16T15:50:02.394 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:02.394+0000 7fbb88e96640 1 -- 172.21.15.123:0/4043844443 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbb841f8e10 con 0x7fbb8415d010 2024-09-16T15:50:02.394 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:02.394+0000 7fbb88e96640 1 -- 172.21.15.123:0/4043844443 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbb840adad0 con 0x7fbb8415d010 2024-09-16T15:50:02.395 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:02.394+0000 7fbb88e96640 1 -- 172.21.15.123:0/4043844443 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbb840ad8c0 con 0x7fbb8415d010 2024-09-16T15:50:02.395 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:02.394+0000 7fbb7b7fe640 1 -- 172.21.15.123:0/4043844443 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 21) ==== 97641+0+0 (secure 0 0 0) 0x7fbb6c0c0060 con 0x7fbb8415d010 2024-09-16T15:50:02.395 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:02.394+0000 7fbb7b7fe640 1 --2- 172.21.15.123:0/4043844443 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fbb640734f0 0x7fbb640759b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:50:02.395 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:02.394+0000 7fbb7b7fe640 1 -- 172.21.15.123:0/4043844443 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(135..135 src has 1..135) ==== 5737+0+0 (secure 0 0 0) 0x7fbb6c12e840 con 0x7fbb8415d010 2024-09-16T15:50:02.395 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:02.394+0000 7fbb8259c640 1 --2- 172.21.15.123:0/4043844443 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fbb640734f0 0x7fbb640759b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:50:02.395 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:02.394+0000 7fbb8259c640 1 --2- 172.21.15.123:0/4043844443 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fbb640734f0 0x7fbb640759b0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7fbb74005d10 tx=0x7fbb74005f40 comp rx=0 tx=0).ready entity=mgr.4150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:50:02.398 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:02.398+0000 7fbb7b7fe640 1 -- 172.21.15.123:0/4043844443 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7fbb6c016f80 con 0x7fbb8415d010 2024-09-16T15:50:02.608 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:02.610+0000 7fbb88e96640 1 -- 172.21.15.123:0/4043844443 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7fbb84073f30 con 0x7fbb8415d010 2024-09-16T15:50:02.609 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:02.610+0000 7fbb7b7fe640 1 -- 172.21.15.123:0/4043844443 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v86) ==== 53+0+73 (secure 0 0 0) 0x7fbb6c0f93b0 con 0x7fbb8415d010 2024-09-16T15:50:02.609 INFO:teuthology.orchestra.run.smithi123.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-09-16T15:50:02.611 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:02.610+0000 7fbb88e96640 1 -- 172.21.15.123:0/4043844443 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fbb640734f0 msgr2=0x7fbb640759b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:50:02.612 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:02.610+0000 7fbb88e96640 1 --2- 172.21.15.123:0/4043844443 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fbb640734f0 0x7fbb640759b0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7fbb74005d10 tx=0x7fbb74005f40 comp rx=0 tx=0).stop 2024-09-16T15:50:02.612 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:02.610+0000 7fbb88e96640 1 -- 172.21.15.123:0/4043844443 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbb8415d010 msgr2=0x7fbb841f82e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:50:02.612 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:02.610+0000 7fbb88e96640 1 --2- 172.21.15.123:0/4043844443 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbb8415d010 0x7fbb841f82e0 secure :-1 s=READY pgs=1430 cs=0 l=1 rev1=1 crypto rx=0x7fbb6c0af040 tx=0x7fbb6c0acdb0 comp rx=0 tx=0).stop 2024-09-16T15:50:02.612 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:02.610+0000 7fbb88e96640 1 -- 172.21.15.123:0/4043844443 shutdown_connections 2024-09-16T15:50:02.612 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:02.610+0000 7fbb88e96640 1 --2- 172.21.15.123:0/4043844443 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fbb640734f0 0x7fbb640759b0 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:02.612 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:02.610+0000 7fbb88e96640 1 --2- 172.21.15.123:0/4043844443 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fbb8415d490 0x7fbb841f8820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:02.612 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:02.610+0000 7fbb88e96640 1 --2- 172.21.15.123:0/4043844443 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbb8415d010 0x7fbb841f82e0 unknown :-1 s=CLOSED pgs=1430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:02.612 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:02.610+0000 7fbb88e96640 1 --2- 172.21.15.123:0/4043844443 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fbb70004810 0x7fbb840ae2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:02.612 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:02.610+0000 7fbb88e96640 1 -- 172.21.15.123:0/4043844443 >> 172.21.15.123:0/4043844443 conn(0x7fbb840cfb10 msgr2=0x7fbb84072f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:50:02.612 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:02.610+0000 7fbb88e96640 1 -- 172.21.15.123:0/4043844443 shutdown_connections 2024-09-16T15:50:02.612 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:02.610+0000 7fbb88e96640 1 -- 172.21.15.123:0/4043844443 wait complete. 2024-09-16T15:50:02.628 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-09-16T15:50:02.628 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-09-16T15:50:02.628 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi184.front.sepia.ceph.com 2024-09-16T15:50:02.628 INFO:tasks.cephfs.mount:self.client.name = client.1 2024-09-16T15:50:02.628 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2024-09-16T15:50:02.628 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-09-16T15:50:02.628 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-09-16T15:50:02.628 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-09-16T15:50:02.628 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.1' netns for '/home/ubuntu/cephtest/mnt.1' 2024-09-16T15:50:02.629 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:50:02.629 DEBUG:teuthology.orchestra.run.smithi184:> ip addr 2024-09-16T15:50:02.636 INFO:teuthology.orchestra.run.smithi184.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-09-16T15:50:02.636 INFO:teuthology.orchestra.run.smithi184.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-09-16T15:50:02.636 INFO:teuthology.orchestra.run.smithi184.stdout: inet 127.0.0.1/8 scope host lo 2024-09-16T15:50:02.636 INFO:teuthology.orchestra.run.smithi184.stdout: valid_lft forever preferred_lft forever 2024-09-16T15:50:02.636 INFO:teuthology.orchestra.run.smithi184.stdout: inet6 ::1/128 scope host 2024-09-16T15:50:02.636 INFO:teuthology.orchestra.run.smithi184.stdout: valid_lft forever preferred_lft forever 2024-09-16T15:50:02.636 INFO:teuthology.orchestra.run.smithi184.stdout:2: eno1: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-09-16T15:50:02.636 INFO:teuthology.orchestra.run.smithi184.stdout: link/ether 0c:c4:7a:d9:37:16 brd ff:ff:ff:ff:ff:ff 2024-09-16T15:50:02.636 INFO:teuthology.orchestra.run.smithi184.stdout: altname enp7s0f0 2024-09-16T15:50:02.636 INFO:teuthology.orchestra.run.smithi184.stdout:3: eno2: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-09-16T15:50:02.637 INFO:teuthology.orchestra.run.smithi184.stdout: link/ether 0c:c4:7a:d9:37:17 brd ff:ff:ff:ff:ff:ff 2024-09-16T15:50:02.637 INFO:teuthology.orchestra.run.smithi184.stdout: altname enp7s0f1 2024-09-16T15:50:02.637 INFO:teuthology.orchestra.run.smithi184.stdout:4: enp3s0f0: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-09-16T15:50:02.637 INFO:teuthology.orchestra.run.smithi184.stdout: link/ether 0c:c4:7a:88:7b:8e brd ff:ff:ff:ff:ff:ff 2024-09-16T15:50:02.637 INFO:teuthology.orchestra.run.smithi184.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-09-16T15:50:02.637 INFO:teuthology.orchestra.run.smithi184.stdout: link/ether 0c:c4:7a:88:7b:8f brd ff:ff:ff:ff:ff:ff 2024-09-16T15:50:02.637 INFO:teuthology.orchestra.run.smithi184.stdout: inet 172.21.15.184/20 metric 100 brd 172.21.15.255 scope global dynamic enp3s0f1 2024-09-16T15:50:02.637 INFO:teuthology.orchestra.run.smithi184.stdout: valid_lft 42054sec preferred_lft 42054sec 2024-09-16T15:50:02.637 INFO:teuthology.orchestra.run.smithi184.stdout: inet6 fe80::ec4:7aff:fe88:7b8f/64 scope link 2024-09-16T15:50:02.637 INFO:teuthology.orchestra.run.smithi184.stdout: valid_lft forever preferred_lft forever 2024-09-16T15:50:02.637 INFO:teuthology.orchestra.run.smithi184.stdout:6: docker0: mtu 1500 qdisc noqueue state DOWN group default 2024-09-16T15:50:02.637 INFO:teuthology.orchestra.run.smithi184.stdout: link/ether 02:42:5d:0a:44:b9 brd ff:ff:ff:ff:ff:ff 2024-09-16T15:50:02.637 INFO:teuthology.orchestra.run.smithi184.stdout: inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0 2024-09-16T15:50:02.637 INFO:teuthology.orchestra.run.smithi184.stdout: valid_lft forever preferred_lft forever 2024-09-16T15:50:02.638 INFO:teuthology.orchestra.run.smithi184.stdout:7: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-09-16T15:50:02.638 INFO:teuthology.orchestra.run.smithi184.stdout: link/ether 5e:1f:84:6d:79:ae brd ff:ff:ff:ff:ff:ff 2024-09-16T15:50:02.638 INFO:teuthology.orchestra.run.smithi184.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-09-16T15:50:02.638 INFO:teuthology.orchestra.run.smithi184.stdout: valid_lft forever preferred_lft forever 2024-09-16T15:50:02.638 INFO:teuthology.orchestra.run.smithi184.stdout: inet6 fe80::5c1f:84ff:fe6d:79ae/64 scope link 2024-09-16T15:50:02.638 INFO:teuthology.orchestra.run.smithi184.stdout: valid_lft forever preferred_lft forever 2024-09-16T15:50:02.638 INFO:teuthology.orchestra.run.smithi184.stdout:8: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-09-16T15:50:02.638 INFO:teuthology.orchestra.run.smithi184.stdout: link/ether 0e:5c:f6:1f:82:2f brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.1 2024-09-16T15:50:02.638 INFO:teuthology.orchestra.run.smithi184.stdout: inet6 fe80::c5c:f6ff:fe1f:822f/64 scope link 2024-09-16T15:50:02.638 INFO:teuthology.orchestra.run.smithi184.stdout: valid_lft forever preferred_lft forever 2024-09-16T15:50:02.638 INFO:teuthology.orchestra.run.smithi184.stdout:9: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-09-16T15:50:02.638 INFO:teuthology.orchestra.run.smithi184.stdout: link/ether c6:a6:3d:69:d0:db brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.3 2024-09-16T15:50:02.638 INFO:teuthology.orchestra.run.smithi184.stdout: inet6 fe80::c4a6:3dff:fe69:d0db/64 scope link 2024-09-16T15:50:02.639 INFO:teuthology.orchestra.run.smithi184.stdout: valid_lft forever preferred_lft forever 2024-09-16T15:50:02.639 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:50:02.639 DEBUG:teuthology.orchestra.run.smithi184:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-09-16T15:50:02.695 INFO:teuthology.orchestra.run.smithi184.stdout:1 2024-09-16T15:50:02.696 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:50:02.696 DEBUG:teuthology.orchestra.run.smithi184:> route 2024-09-16T15:50:02.751 INFO:teuthology.orchestra.run.smithi184.stdout:Kernel IP routing table 2024-09-16T15:50:02.751 INFO:teuthology.orchestra.run.smithi184.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-09-16T15:50:02.751 INFO:teuthology.orchestra.run.smithi184.stdout:default _gateway 0.0.0.0 UG 0 0 0 enp3s0f1 2024-09-16T15:50:02.751 INFO:teuthology.orchestra.run.smithi184.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-09-16T15:50:02.751 INFO:teuthology.orchestra.run.smithi184.stdout:158.69.67.47 _gateway 255.255.255.255 UGH 100 0 0 enp3s0f1 2024-09-16T15:50:02.751 INFO:teuthology.orchestra.run.smithi184.stdout:172.17.0.0 0.0.0.0 255.255.0.0 U 0 0 0 docker0 2024-09-16T15:50:02.751 INFO:teuthology.orchestra.run.smithi184.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-09-16T15:50:02.751 INFO:teuthology.orchestra.run.smithi184.stdout:ns1.front.sepia 0.0.0.0 255.255.255.255 UH 100 0 0 enp3s0f1 2024-09-16T15:50:02.751 INFO:teuthology.orchestra.run.smithi184.stdout:172.21.0.2 0.0.0.0 255.255.255.255 UH 100 0 0 enp3s0f1 2024-09-16T15:50:02.751 INFO:teuthology.orchestra.run.smithi184.stdout:_gateway 0.0.0.0 255.255.255.255 UH 100 0 0 enp3s0f1 2024-09-16T15:50:02.751 INFO:teuthology.orchestra.run.smithi184.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-09-16T15:50:02.752 DEBUG:teuthology.orchestra.run.smithi184:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-09-16T15:50:02.752 DEBUG:teuthology.orchestra.run.smithi184:> set -e 2024-09-16T15:50:02.752 DEBUG:teuthology.orchestra.run.smithi184:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-09-16T15:50:02.752 DEBUG:teuthology.orchestra.run.smithi184:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-09-16T15:50:02.752 DEBUG:teuthology.orchestra.run.smithi184:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-09-16T15:50:02.752 DEBUG:teuthology.orchestra.run.smithi184:> ') 2024-09-16T15:50:02.840 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:50:02 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-09-16T15:50:02.868 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:50:02 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-16T15:50:02.876 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:50:02.876 DEBUG:teuthology.orchestra.run.smithi184:> ip netns list 2024-09-16T15:50:02.881 INFO:teuthology.orchestra.run.smithi184.stdout:ceph-ns--home-ubuntu-cephtest-mnt.3 (id: 1) 2024-09-16T15:50:02.881 INFO:teuthology.orchestra.run.smithi184.stdout:ceph-ns--home-ubuntu-cephtest-mnt.1 (id: 0) 2024-09-16T15:50:02.881 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.1' netns for '/home/ubuntu/cephtest/mnt.1' 2024-09-16T15:50:02.881 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:50:02.881 DEBUG:teuthology.orchestra.run.smithi184:> sudo ip link set brx.0 up 2024-09-16T15:50:02.935 INFO:tasks.cephfs.fuse_mount:Client client.1 config is {} 2024-09-16T15:50:02.936 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-09-16T15:50:02.936 DEBUG:teuthology.orchestra.run.smithi184:> mkdir -p -v /home/ubuntu/cephtest/mnt.1 2024-09-16T15:50:02.981 INFO:teuthology.orchestra.run.smithi184.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.1' 2024-09-16T15:50:02.981 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-09-16T15:50:02.981 DEBUG:teuthology.orchestra.run.smithi184:> chmod 0000 /home/ubuntu/cephtest/mnt.1 2024-09-16T15:50:03.029 DEBUG:teuthology.orchestra.run.smithi184:> sudo modprobe fuse 2024-09-16T15:50:03.087 DEBUG:teuthology.orchestra.run.smithi184:> cat /proc/self/mounts | awk '{print $2}' 2024-09-16T15:50:03.134 INFO:teuthology.orchestra.run.smithi184.stdout:/sys 2024-09-16T15:50:03.134 INFO:teuthology.orchestra.run.smithi184.stdout:/proc 2024-09-16T15:50:03.134 INFO:teuthology.orchestra.run.smithi184.stdout:/dev 2024-09-16T15:50:03.134 INFO:teuthology.orchestra.run.smithi184.stdout:/dev/pts 2024-09-16T15:50:03.135 INFO:teuthology.orchestra.run.smithi184.stdout:/run 2024-09-16T15:50:03.135 INFO:teuthology.orchestra.run.smithi184.stdout:/ 2024-09-16T15:50:03.135 INFO:teuthology.orchestra.run.smithi184.stdout:/sys/kernel/security 2024-09-16T15:50:03.135 INFO:teuthology.orchestra.run.smithi184.stdout:/dev/shm 2024-09-16T15:50:03.135 INFO:teuthology.orchestra.run.smithi184.stdout:/run/lock 2024-09-16T15:50:03.135 INFO:teuthology.orchestra.run.smithi184.stdout:/sys/fs/cgroup 2024-09-16T15:50:03.135 INFO:teuthology.orchestra.run.smithi184.stdout:/sys/fs/pstore 2024-09-16T15:50:03.135 INFO:teuthology.orchestra.run.smithi184.stdout:/sys/fs/bpf 2024-09-16T15:50:03.135 INFO:teuthology.orchestra.run.smithi184.stdout:/proc/sys/fs/binfmt_misc 2024-09-16T15:50:03.135 INFO:teuthology.orchestra.run.smithi184.stdout:/dev/hugepages 2024-09-16T15:50:03.135 INFO:teuthology.orchestra.run.smithi184.stdout:/dev/mqueue 2024-09-16T15:50:03.135 INFO:teuthology.orchestra.run.smithi184.stdout:/sys/kernel/debug 2024-09-16T15:50:03.135 INFO:teuthology.orchestra.run.smithi184.stdout:/sys/kernel/tracing 2024-09-16T15:50:03.135 INFO:teuthology.orchestra.run.smithi184.stdout:/proc/fs/nfsd 2024-09-16T15:50:03.136 INFO:teuthology.orchestra.run.smithi184.stdout:/sys/fs/fuse/connections 2024-09-16T15:50:03.136 INFO:teuthology.orchestra.run.smithi184.stdout:/sys/kernel/config 2024-09-16T15:50:03.136 INFO:teuthology.orchestra.run.smithi184.stdout:/run/credentials/systemd-sysusers.service 2024-09-16T15:50:03.136 INFO:teuthology.orchestra.run.smithi184.stdout:/run/rpc_pipefs 2024-09-16T15:50:03.136 INFO:teuthology.orchestra.run.smithi184.stdout:/run/user/1000 2024-09-16T15:50:03.136 INFO:teuthology.orchestra.run.smithi184.stdout:/var/lib/ceph 2024-09-16T15:50:03.136 INFO:teuthology.orchestra.run.smithi184.stdout:/var/lib/ceph/osd/ceph-4 2024-09-16T15:50:03.136 INFO:teuthology.orchestra.run.smithi184.stdout:/var/lib/ceph/osd/ceph-5 2024-09-16T15:50:03.136 INFO:teuthology.orchestra.run.smithi184.stdout:/var/lib/ceph/osd/ceph-6 2024-09-16T15:50:03.136 INFO:teuthology.orchestra.run.smithi184.stdout:/var/lib/ceph/osd/ceph-7 2024-09-16T15:50:03.136 INFO:teuthology.orchestra.run.smithi184.stdout:/run/netns 2024-09-16T15:50:03.136 INFO:teuthology.orchestra.run.smithi184.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-09-16T15:50:03.136 INFO:teuthology.orchestra.run.smithi184.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-09-16T15:50:03.136 INFO:teuthology.orchestra.run.smithi184.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-09-16T15:50:03.137 INFO:teuthology.orchestra.run.smithi184.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-09-16T15:50:03.137 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:50:03.137 DEBUG:teuthology.orchestra.run.smithi184:> ls /sys/fs/fuse/connections 2024-09-16T15:50:03.190 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-09-16T15:50:03.190 DEBUG:teuthology.orchestra.run.smithi184:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.1 --id 1) 2024-09-16T15:50:03.234 DEBUG:teuthology.orchestra.run.smithi184:> sudo modprobe fuse 2024-09-16T15:50:03.249 DEBUG:teuthology.orchestra.run.smithi184:> cat /proc/self/mounts | awk '{print $2}' 2024-09-16T15:50:03.300 INFO:teuthology.orchestra.run.smithi184.stdout:/sys 2024-09-16T15:50:03.300 INFO:teuthology.orchestra.run.smithi184.stdout:/proc 2024-09-16T15:50:03.301 INFO:teuthology.orchestra.run.smithi184.stdout:/dev 2024-09-16T15:50:03.301 INFO:teuthology.orchestra.run.smithi184.stdout:/dev/pts 2024-09-16T15:50:03.301 INFO:teuthology.orchestra.run.smithi184.stdout:/run 2024-09-16T15:50:03.301 INFO:teuthology.orchestra.run.smithi184.stdout:/ 2024-09-16T15:50:03.301 INFO:teuthology.orchestra.run.smithi184.stdout:/sys/kernel/security 2024-09-16T15:50:03.301 INFO:teuthology.orchestra.run.smithi184.stdout:/dev/shm 2024-09-16T15:50:03.301 INFO:teuthology.orchestra.run.smithi184.stdout:/run/lock 2024-09-16T15:50:03.301 INFO:teuthology.orchestra.run.smithi184.stdout:/sys/fs/cgroup 2024-09-16T15:50:03.301 INFO:teuthology.orchestra.run.smithi184.stdout:/sys/fs/pstore 2024-09-16T15:50:03.301 INFO:teuthology.orchestra.run.smithi184.stdout:/sys/fs/bpf 2024-09-16T15:50:03.301 INFO:teuthology.orchestra.run.smithi184.stdout:/proc/sys/fs/binfmt_misc 2024-09-16T15:50:03.301 INFO:teuthology.orchestra.run.smithi184.stdout:/dev/hugepages 2024-09-16T15:50:03.301 INFO:teuthology.orchestra.run.smithi184.stdout:/dev/mqueue 2024-09-16T15:50:03.302 INFO:teuthology.orchestra.run.smithi184.stdout:/sys/kernel/debug 2024-09-16T15:50:03.302 INFO:teuthology.orchestra.run.smithi184.stdout:/sys/kernel/tracing 2024-09-16T15:50:03.302 INFO:teuthology.orchestra.run.smithi184.stdout:/proc/fs/nfsd 2024-09-16T15:50:03.302 INFO:teuthology.orchestra.run.smithi184.stdout:/sys/fs/fuse/connections 2024-09-16T15:50:03.302 INFO:teuthology.orchestra.run.smithi184.stdout:/sys/kernel/config 2024-09-16T15:50:03.302 INFO:teuthology.orchestra.run.smithi184.stdout:/run/credentials/systemd-sysusers.service 2024-09-16T15:50:03.302 INFO:teuthology.orchestra.run.smithi184.stdout:/run/rpc_pipefs 2024-09-16T15:50:03.302 INFO:teuthology.orchestra.run.smithi184.stdout:/run/user/1000 2024-09-16T15:50:03.302 INFO:teuthology.orchestra.run.smithi184.stdout:/var/lib/ceph 2024-09-16T15:50:03.302 INFO:teuthology.orchestra.run.smithi184.stdout:/var/lib/ceph/osd/ceph-4 2024-09-16T15:50:03.302 INFO:teuthology.orchestra.run.smithi184.stdout:/var/lib/ceph/osd/ceph-5 2024-09-16T15:50:03.302 INFO:teuthology.orchestra.run.smithi184.stdout:/var/lib/ceph/osd/ceph-6 2024-09-16T15:50:03.302 INFO:teuthology.orchestra.run.smithi184.stdout:/var/lib/ceph/osd/ceph-7 2024-09-16T15:50:03.302 INFO:teuthology.orchestra.run.smithi184.stdout:/run/netns 2024-09-16T15:50:03.303 INFO:teuthology.orchestra.run.smithi184.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-09-16T15:50:03.303 INFO:teuthology.orchestra.run.smithi184.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-09-16T15:50:03.303 INFO:teuthology.orchestra.run.smithi184.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-09-16T15:50:03.303 INFO:teuthology.orchestra.run.smithi184.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-09-16T15:50:03.303 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:50:03.303 DEBUG:teuthology.orchestra.run.smithi184:> ls /sys/fs/fuse/connections 2024-09-16T15:50:03.311 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi184.stderr:2024-09-16T15:50:03.308+0000 7f939f7c3580 -1 init, newargv = 0x562698225be0 newargc=15 2024-09-16T15:50:03.311 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi184.stderr:ceph-fuse[36044]: starting ceph client 2024-09-16T15:50:03.319 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi184.stderr:ceph-fuse[36044]: starting fuse 2024-09-16T15:50:04.417 DEBUG:teuthology.orchestra.run.smithi184:> sudo modprobe fuse 2024-09-16T15:50:04.433 DEBUG:teuthology.orchestra.run.smithi184:> cat /proc/self/mounts | awk '{print $2}' 2024-09-16T15:50:04.482 INFO:teuthology.orchestra.run.smithi184.stdout:/sys 2024-09-16T15:50:04.482 INFO:teuthology.orchestra.run.smithi184.stdout:/proc 2024-09-16T15:50:04.482 INFO:teuthology.orchestra.run.smithi184.stdout:/dev 2024-09-16T15:50:04.483 INFO:teuthology.orchestra.run.smithi184.stdout:/dev/pts 2024-09-16T15:50:04.483 INFO:teuthology.orchestra.run.smithi184.stdout:/run 2024-09-16T15:50:04.483 INFO:teuthology.orchestra.run.smithi184.stdout:/ 2024-09-16T15:50:04.483 INFO:teuthology.orchestra.run.smithi184.stdout:/sys/kernel/security 2024-09-16T15:50:04.483 INFO:teuthology.orchestra.run.smithi184.stdout:/dev/shm 2024-09-16T15:50:04.483 INFO:teuthology.orchestra.run.smithi184.stdout:/run/lock 2024-09-16T15:50:04.483 INFO:teuthology.orchestra.run.smithi184.stdout:/sys/fs/cgroup 2024-09-16T15:50:04.483 INFO:teuthology.orchestra.run.smithi184.stdout:/sys/fs/pstore 2024-09-16T15:50:04.483 INFO:teuthology.orchestra.run.smithi184.stdout:/sys/fs/bpf 2024-09-16T15:50:04.483 INFO:teuthology.orchestra.run.smithi184.stdout:/proc/sys/fs/binfmt_misc 2024-09-16T15:50:04.483 INFO:teuthology.orchestra.run.smithi184.stdout:/dev/hugepages 2024-09-16T15:50:04.483 INFO:teuthology.orchestra.run.smithi184.stdout:/dev/mqueue 2024-09-16T15:50:04.484 INFO:teuthology.orchestra.run.smithi184.stdout:/sys/kernel/debug 2024-09-16T15:50:04.484 INFO:teuthology.orchestra.run.smithi184.stdout:/sys/kernel/tracing 2024-09-16T15:50:04.484 INFO:teuthology.orchestra.run.smithi184.stdout:/proc/fs/nfsd 2024-09-16T15:50:04.484 INFO:teuthology.orchestra.run.smithi184.stdout:/sys/fs/fuse/connections 2024-09-16T15:50:04.484 INFO:teuthology.orchestra.run.smithi184.stdout:/sys/kernel/config 2024-09-16T15:50:04.484 INFO:teuthology.orchestra.run.smithi184.stdout:/run/credentials/systemd-sysusers.service 2024-09-16T15:50:04.484 INFO:teuthology.orchestra.run.smithi184.stdout:/run/rpc_pipefs 2024-09-16T15:50:04.484 INFO:teuthology.orchestra.run.smithi184.stdout:/run/user/1000 2024-09-16T15:50:04.484 INFO:teuthology.orchestra.run.smithi184.stdout:/var/lib/ceph 2024-09-16T15:50:04.484 INFO:teuthology.orchestra.run.smithi184.stdout:/var/lib/ceph/osd/ceph-4 2024-09-16T15:50:04.484 INFO:teuthology.orchestra.run.smithi184.stdout:/var/lib/ceph/osd/ceph-5 2024-09-16T15:50:04.484 INFO:teuthology.orchestra.run.smithi184.stdout:/var/lib/ceph/osd/ceph-6 2024-09-16T15:50:04.484 INFO:teuthology.orchestra.run.smithi184.stdout:/var/lib/ceph/osd/ceph-7 2024-09-16T15:50:04.484 INFO:teuthology.orchestra.run.smithi184.stdout:/run/netns 2024-09-16T15:50:04.485 INFO:teuthology.orchestra.run.smithi184.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-09-16T15:50:04.485 INFO:teuthology.orchestra.run.smithi184.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-09-16T15:50:04.485 INFO:teuthology.orchestra.run.smithi184.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-09-16T15:50:04.485 INFO:teuthology.orchestra.run.smithi184.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-09-16T15:50:04.485 INFO:teuthology.orchestra.run.smithi184.stdout:/home/ubuntu/cephtest/mnt.1 2024-09-16T15:50:04.486 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:50:04.486 DEBUG:teuthology.orchestra.run.smithi184:> ls /sys/fs/fuse/connections 2024-09-16T15:50:04.537 INFO:teuthology.orchestra.run.smithi184.stdout:45 2024-09-16T15:50:04.537 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [45] 2024-09-16T15:50:04.537 DEBUG:teuthology.orchestra.run.smithi184:> sudo stdin-killer -- python3 -c ' 2024-09-16T15:50:04.537 DEBUG:teuthology.orchestra.run.smithi184:> import glob 2024-09-16T15:50:04.538 DEBUG:teuthology.orchestra.run.smithi184:> import re 2024-09-16T15:50:04.538 DEBUG:teuthology.orchestra.run.smithi184:> import os 2024-09-16T15:50:04.538 DEBUG:teuthology.orchestra.run.smithi184:> import subprocess 2024-09-16T15:50:04.538 DEBUG:teuthology.orchestra.run.smithi184:> 2024-09-16T15:50:04.538 DEBUG:teuthology.orchestra.run.smithi184:> def _find_admin_socket(client_name): 2024-09-16T15:50:04.538 DEBUG:teuthology.orchestra.run.smithi184:> asok_path = "/var/run/ceph/ceph-client.1.*.asok" 2024-09-16T15:50:04.538 DEBUG:teuthology.orchestra.run.smithi184:> files = glob.glob(asok_path) 2024-09-16T15:50:04.538 DEBUG:teuthology.orchestra.run.smithi184:> mountpoint = "/home/ubuntu/cephtest/mnt.1" 2024-09-16T15:50:04.538 DEBUG:teuthology.orchestra.run.smithi184:> 2024-09-16T15:50:04.538 DEBUG:teuthology.orchestra.run.smithi184:> # Given a non-glob path, it better be there 2024-09-16T15:50:04.538 DEBUG:teuthology.orchestra.run.smithi184:> if "*" not in asok_path: 2024-09-16T15:50:04.538 DEBUG:teuthology.orchestra.run.smithi184:> assert(len(files) == 1) 2024-09-16T15:50:04.538 DEBUG:teuthology.orchestra.run.smithi184:> return files[0] 2024-09-16T15:50:04.538 DEBUG:teuthology.orchestra.run.smithi184:> 2024-09-16T15:50:04.538 DEBUG:teuthology.orchestra.run.smithi184:> for f in files: 2024-09-16T15:50:04.538 DEBUG:teuthology.orchestra.run.smithi184:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-09-16T15:50:04.539 DEBUG:teuthology.orchestra.run.smithi184:> if os.path.exists("/proc/{0}".format(pid)): 2024-09-16T15:50:04.539 DEBUG:teuthology.orchestra.run.smithi184:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-09-16T15:50:04.539 DEBUG:teuthology.orchestra.run.smithi184:> contents = proc_f.read() 2024-09-16T15:50:04.539 DEBUG:teuthology.orchestra.run.smithi184:> if mountpoint in contents: 2024-09-16T15:50:04.539 DEBUG:teuthology.orchestra.run.smithi184:> return f 2024-09-16T15:50:04.539 DEBUG:teuthology.orchestra.run.smithi184:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-09-16T15:50:04.539 DEBUG:teuthology.orchestra.run.smithi184:> 2024-09-16T15:50:04.539 DEBUG:teuthology.orchestra.run.smithi184:> print(_find_admin_socket("client.1")) 2024-09-16T15:50:04.539 DEBUG:teuthology.orchestra.run.smithi184:> ' 2024-09-16T15:50:04.663 INFO:teuthology.orchestra.run.smithi184.stdout:/var/run/ceph/ceph-client.1.36044.asok 2024-09-16T15:50:04.666 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:50:04 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-16T15:50:04.675 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.1.36044.asok 2024-09-16T15:50:04.675 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:50:04.675 DEBUG:teuthology.orchestra.run.smithi184:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.1.36044.asok status 2024-09-16T15:50:04.827 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-16T15:50:04.827 INFO:teuthology.orchestra.run.smithi184.stdout: "metadata": { 2024-09-16T15:50:04.827 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph_sha1": "33f95d2ae1bcbe7eb04404303f179eeeaa6982df", 2024-09-16T15:50:04.828 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph_version": "ceph version 19.3.0-4938-g33f95d2a (33f95d2ae1bcbe7eb04404303f179eeeaa6982df) squid (dev)", 2024-09-16T15:50:04.828 INFO:teuthology.orchestra.run.smithi184.stdout: "entity_id": "1", 2024-09-16T15:50:04.828 INFO:teuthology.orchestra.run.smithi184.stdout: "hostname": "smithi184", 2024-09-16T15:50:04.828 INFO:teuthology.orchestra.run.smithi184.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.1", 2024-09-16T15:50:04.828 INFO:teuthology.orchestra.run.smithi184.stdout: "pid": "36044", 2024-09-16T15:50:04.828 INFO:teuthology.orchestra.run.smithi184.stdout: "root": "/" 2024-09-16T15:50:04.828 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-16T15:50:04.828 INFO:teuthology.orchestra.run.smithi184.stdout: "dentry_count": 0, 2024-09-16T15:50:04.828 INFO:teuthology.orchestra.run.smithi184.stdout: "dentry_pinned_count": 0, 2024-09-16T15:50:04.828 INFO:teuthology.orchestra.run.smithi184.stdout: "id": 6412, 2024-09-16T15:50:04.828 INFO:teuthology.orchestra.run.smithi184.stdout: "inst": { 2024-09-16T15:50:04.828 INFO:teuthology.orchestra.run.smithi184.stdout: "name": { 2024-09-16T15:50:04.828 INFO:teuthology.orchestra.run.smithi184.stdout: "type": "client", 2024-09-16T15:50:04.829 INFO:teuthology.orchestra.run.smithi184.stdout: "num": 6412 2024-09-16T15:50:04.829 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-16T15:50:04.829 INFO:teuthology.orchestra.run.smithi184.stdout: "addr": { 2024-09-16T15:50:04.829 INFO:teuthology.orchestra.run.smithi184.stdout: "type": "v1", 2024-09-16T15:50:04.829 INFO:teuthology.orchestra.run.smithi184.stdout: "addr": "192.168.0.1:0", 2024-09-16T15:50:04.829 INFO:teuthology.orchestra.run.smithi184.stdout: "nonce": 3006908787 2024-09-16T15:50:04.829 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-16T15:50:04.829 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-16T15:50:04.829 INFO:teuthology.orchestra.run.smithi184.stdout: "addr": { 2024-09-16T15:50:04.829 INFO:teuthology.orchestra.run.smithi184.stdout: "type": "v1", 2024-09-16T15:50:04.829 INFO:teuthology.orchestra.run.smithi184.stdout: "addr": "192.168.0.1:0", 2024-09-16T15:50:04.829 INFO:teuthology.orchestra.run.smithi184.stdout: "nonce": 3006908787 2024-09-16T15:50:04.829 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-16T15:50:04.830 INFO:teuthology.orchestra.run.smithi184.stdout: "inst_str": "client.6412 192.168.0.1:0/3006908787", 2024-09-16T15:50:04.830 INFO:teuthology.orchestra.run.smithi184.stdout: "addr_str": "192.168.0.1:0/3006908787", 2024-09-16T15:50:04.830 INFO:teuthology.orchestra.run.smithi184.stdout: "inode_count": 1, 2024-09-16T15:50:04.830 INFO:teuthology.orchestra.run.smithi184.stdout: "mds_epoch": 86, 2024-09-16T15:50:04.830 INFO:teuthology.orchestra.run.smithi184.stdout: "osd_epoch": 135, 2024-09-16T15:50:04.830 INFO:teuthology.orchestra.run.smithi184.stdout: "osd_epoch_barrier": 0, 2024-09-16T15:50:04.830 INFO:teuthology.orchestra.run.smithi184.stdout: "blocklisted": false, 2024-09-16T15:50:04.830 INFO:teuthology.orchestra.run.smithi184.stdout: "fs_name": "cephfs" 2024-09-16T15:50:04.830 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-16T15:50:04.841 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:50:04.841 DEBUG:teuthology.orchestra.run.smithi184:> stat --file-system '--printf=%T 2024-09-16T15:50:04.841 DEBUG:teuthology.orchestra.run.smithi184:> ' -- /home/ubuntu/cephtest/mnt.1 2024-09-16T15:50:04.847 INFO:teuthology.orchestra.run.smithi184.stdout:fuseblk 2024-09-16T15:50:04.847 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2024-09-16T15:50:04.847 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:50:04.847 DEBUG:teuthology.orchestra.run.smithi184:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.1 2024-09-16T15:50:04.898 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs flag set enable_multiple true --yes-i-really-mean-it 2024-09-16T15:50:05.040 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:05.038+0000 7f7006d6c640 1 -- 172.21.15.123:0/2300265120 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f700015d7c0 msgr2=0x7f70000b31e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:50:05.040 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:05.038+0000 7f7006d6c640 1 --2- 172.21.15.123:0/2300265120 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f700015d7c0 0x7f70000b31e0 secure :-1 s=READY pgs=1433 cs=0 l=1 rev1=1 crypto rx=0x7f6ff4009840 tx=0x7f6ff401f010 comp rx=0 tx=0).stop 2024-09-16T15:50:05.040 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:05.038+0000 7f7006d6c640 1 -- 172.21.15.123:0/2300265120 shutdown_connections 2024-09-16T15:50:05.040 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:05.038+0000 7f7006d6c640 1 --2- 172.21.15.123:0/2300265120 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f700015d7c0 0x7f70000b31e0 unknown :-1 s=CLOSED pgs=1433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:05.040 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:05.038+0000 7f7006d6c640 1 -- 172.21.15.123:0/2300265120 >> 172.21.15.123:0/2300265120 conn(0x7f70000b1630 msgr2=0x7f70000b1a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:50:05.040 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:05.038+0000 7f7006d6c640 1 -- 172.21.15.123:0/2300265120 shutdown_connections 2024-09-16T15:50:05.040 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:05.038+0000 7f7006d6c640 1 -- 172.21.15.123:0/2300265120 wait complete. 2024-09-16T15:50:05.041 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:05.042+0000 7f7006d6c640 1 Processor -- start 2024-09-16T15:50:05.041 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:05.042+0000 7f7006d6c640 1 -- start start 2024-09-16T15:50:05.042 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:05.042+0000 7f7006d6c640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f70000b3720 0x7f7000148ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:50:05.043 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:05.042+0000 7f7006d6c640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f700015d7c0 0x7f70000adb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:50:05.043 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:05.042+0000 7f7006d6c640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f700015e4b0 0x7f70000cd910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:50:05.043 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:05.042+0000 7f7006d6c640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f700006eae0 con 0x7f70000b3720 2024-09-16T15:50:05.043 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:05.042+0000 7f7006d6c640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f700006e960 con 0x7f700015e4b0 2024-09-16T15:50:05.043 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:05.042+0000 7f7006d6c640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f700006ec60 con 0x7f700015d7c0 2024-09-16T15:50:05.043 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:05.042+0000 7f7004b08640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f70000b3720 0x7f7000148ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:50:05.043 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:05.042+0000 7f7004b08640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f70000b3720 0x7f7000148ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:53196/0 (socket says 172.21.15.123:53196) 2024-09-16T15:50:05.043 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:05.042+0000 7f7004b08640 1 -- 172.21.15.123:0/3335304668 learned_addr learned my addr 172.21.15.123:0/3335304668 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:50:05.043 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:05.042+0000 7f7004b08640 1 -- 172.21.15.123:0/3335304668 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f700015d7c0 msgr2=0x7f70000adb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:50:05.043 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:05.042+0000 7f7004b08640 1 --2- 172.21.15.123:0/3335304668 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f700015d7c0 0x7f70000adb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:05.043 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:05.042+0000 7f7004b08640 1 -- 172.21.15.123:0/3335304668 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f700015e4b0 msgr2=0x7f70000cd910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:50:05.043 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:05.042+0000 7f7004b08640 1 --2- 172.21.15.123:0/3335304668 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f700015e4b0 0x7f70000cd910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:05.043 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:05.042+0000 7f7004b08640 1 -- 172.21.15.123:0/3335304668 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f70000cd130 con 0x7f70000b3720 2024-09-16T15:50:05.044 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:05.042+0000 7f7004b08640 1 --2- 172.21.15.123:0/3335304668 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f70000b3720 0x7f7000148ff0 secure :-1 s=READY pgs=1434 cs=0 l=1 rev1=1 crypto rx=0x7f6ff4009790 tx=0x7f6ff40a8d80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:50:05.044 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:05.042+0000 7f6ffd7fa640 1 -- 172.21.15.123:0/3335304668 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ff40bf070 con 0x7f70000b3720 2024-09-16T15:50:05.044 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:05.042+0000 7f6ffd7fa640 1 -- 172.21.15.123:0/3335304668 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f6ff40052f0 con 0x7f70000b3720 2024-09-16T15:50:05.044 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:05.042+0000 7f7006d6c640 1 -- 172.21.15.123:0/3335304668 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f700016a350 con 0x7f70000b3720 2024-09-16T15:50:05.044 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:05.042+0000 7f6ffd7fa640 1 -- 172.21.15.123:0/3335304668 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ff40b5040 con 0x7f70000b3720 2024-09-16T15:50:05.044 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:05.042+0000 7f7006d6c640 1 -- 172.21.15.123:0/3335304668 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7000149530 con 0x7f70000b3720 2024-09-16T15:50:05.044 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:05.042+0000 7f7006d6c640 1 -- 172.21.15.123:0/3335304668 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f70000b31e0 con 0x7f70000b3720 2024-09-16T15:50:05.044 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:05.046+0000 7f6ffd7fa640 1 -- 172.21.15.123:0/3335304668 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 21) ==== 97641+0+0 (secure 0 0 0) 0x7f6ff40c2070 con 0x7f70000b3720 2024-09-16T15:50:05.045 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:05.046+0000 7f6ffd7fa640 1 --2- 172.21.15.123:0/3335304668 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f6fe40734f0 0x7f6fe40759b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:50:05.045 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:05.046+0000 7f6ffd7fa640 1 -- 172.21.15.123:0/3335304668 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(135..135 src has 1..135) ==== 5737+0+0 (secure 0 0 0) 0x7f6ff4131930 con 0x7f70000b3720 2024-09-16T15:50:05.045 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:05.046+0000 7f6ffffff640 1 --2- 172.21.15.123:0/3335304668 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f6fe40734f0 0x7f6fe40759b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:50:05.045 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:05.046+0000 7f6ffffff640 1 --2- 172.21.15.123:0/3335304668 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f6fe40734f0 0x7f6fe40759b0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f700016abe0 tx=0x7f6fec0079c0 comp rx=0 tx=0).ready entity=mgr.4150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:50:05.049 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:05.050+0000 7f6ffd7fa640 1 -- 172.21.15.123:0/3335304668 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f6ff4018b70 con 0x7f70000b3720 2024-09-16T15:50:05.255 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:05.254+0000 7f7006d6c640 1 -- 172.21.15.123:0/3335304668 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "fs flag set", "flag_name": "enable_multiple", "val": "true", "yes_i_really_mean_it": true} v 0) -- 0x7f700016a800 con 0x7f70000b3720 2024-09-16T15:50:06.177 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:06.178+0000 7f6ffd7fa640 1 -- 172.21.15.123:0/3335304668 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "fs flag set", "flag_name": "enable_multiple", "val": "true", "yes_i_really_mean_it": true}]=0 v87) ==== 136+0+0 (secure 0 0 0) 0x7f6ff40fc2a0 con 0x7f70000b3720 2024-09-16T15:50:06.181 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:06.182+0000 7f7006d6c640 1 -- 172.21.15.123:0/3335304668 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f6fe40734f0 msgr2=0x7f6fe40759b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:50:06.181 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:06.182+0000 7f7006d6c640 1 --2- 172.21.15.123:0/3335304668 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f6fe40734f0 0x7f6fe40759b0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f700016abe0 tx=0x7f6fec0079c0 comp rx=0 tx=0).stop 2024-09-16T15:50:06.181 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:06.182+0000 7f7006d6c640 1 -- 172.21.15.123:0/3335304668 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f70000b3720 msgr2=0x7f7000148ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:50:06.181 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:06.182+0000 7f7006d6c640 1 --2- 172.21.15.123:0/3335304668 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f70000b3720 0x7f7000148ff0 secure :-1 s=READY pgs=1434 cs=0 l=1 rev1=1 crypto rx=0x7f6ff4009790 tx=0x7f6ff40a8d80 comp rx=0 tx=0).stop 2024-09-16T15:50:06.181 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:06.182+0000 7f7006d6c640 1 -- 172.21.15.123:0/3335304668 shutdown_connections 2024-09-16T15:50:06.181 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:06.182+0000 7f7006d6c640 1 --2- 172.21.15.123:0/3335304668 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f6fe40734f0 0x7f6fe40759b0 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:06.181 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:06.182+0000 7f7006d6c640 1 --2- 172.21.15.123:0/3335304668 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f700015e4b0 0x7f70000cd910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:06.181 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:06.182+0000 7f7006d6c640 1 --2- 172.21.15.123:0/3335304668 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f700015d7c0 0x7f70000adb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:06.181 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:06.182+0000 7f7006d6c640 1 --2- 172.21.15.123:0/3335304668 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f70000b3720 0x7f7000148ff0 unknown :-1 s=CLOSED pgs=1434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:06.182 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:06.182+0000 7f7006d6c640 1 -- 172.21.15.123:0/3335304668 >> 172.21.15.123:0/3335304668 conn(0x7f70000b1630 msgr2=0x7f70000ae8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:50:06.182 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:06.182+0000 7f7006d6c640 1 -- 172.21.15.123:0/3335304668 shutdown_connections 2024-09-16T15:50:06.182 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:06.182+0000 7f7006d6c640 1 -- 172.21.15.123:0/3335304668 wait complete. 2024-09-16T15:50:06.202 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2024-09-16T15:50:06.331 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:06.330+0000 7fde652bb640 1 -- 172.21.15.123:0/3386318725 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fde4c004d50 msgr2=0x7fde4c0051d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:50:06.331 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:06.330+0000 7fde652bb640 1 --2- 172.21.15.123:0/3386318725 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fde4c004d50 0x7fde4c0051d0 secure :-1 s=READY pgs=1436 cs=0 l=1 rev1=1 crypto rx=0x7fde5800c640 tx=0x7fde5801fa80 comp rx=0 tx=0).stop 2024-09-16T15:50:06.331 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:06.330+0000 7fde5f7fe640 1 -- 172.21.15.123:0/3386318725 reap_dead start 2024-09-16T15:50:06.331 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:06.330+0000 7fde652bb640 1 -- 172.21.15.123:0/3386318725 shutdown_connections 2024-09-16T15:50:06.331 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:06.330+0000 7fde652bb640 1 -- 172.21.15.123:0/3386318725 >> 172.21.15.123:0/3386318725 conn(0x7fde600cfb10 msgr2=0x7fde600cff20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:50:06.331 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:06.330+0000 7fde652bb640 1 -- 172.21.15.123:0/3386318725 shutdown_connections 2024-09-16T15:50:06.331 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:06.330+0000 7fde652bb640 1 -- 172.21.15.123:0/3386318725 wait complete. 2024-09-16T15:50:06.332 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:06.334+0000 7fde652bb640 1 Processor -- start 2024-09-16T15:50:06.332 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:06.334+0000 7fde652bb640 1 -- start start 2024-09-16T15:50:06.334 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:06.334+0000 7fde652bb640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fde4c004d50 0x7fde6014b010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:50:06.335 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:06.334+0000 7fde652bb640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fde6014dce0 0x7fde601f6090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:50:06.335 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:06.334+0000 7fde652bb640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fde6015e880 0x7fde6014a330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:50:06.335 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:06.334+0000 7fde652bb640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fde6006f8e0 con 0x7fde6014dce0 2024-09-16T15:50:06.335 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:06.334+0000 7fde652bb640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fde6006f760 con 0x7fde6015e880 2024-09-16T15:50:06.335 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:06.334+0000 7fde652bb640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fde6006fa60 con 0x7fde4c004d50 2024-09-16T15:50:06.335 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:06.334+0000 7fde57fff640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fde6014dce0 0x7fde601f6090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:50:06.335 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:06.334+0000 7fde57fff640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fde6014dce0 0x7fde601f6090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:53202/0 (socket says 172.21.15.123:53202) 2024-09-16T15:50:06.335 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:06.334+0000 7fde57fff640 1 -- 172.21.15.123:0/4019724072 learned_addr learned my addr 172.21.15.123:0/4019724072 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:50:06.335 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:06.334+0000 7fde5f7fe640 1 --2- 172.21.15.123:0/4019724072 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fde6015e880 0x7fde6014a330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:50:06.335 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:06.334+0000 7fde5effd640 1 --2- 172.21.15.123:0/4019724072 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fde4c004d50 0x7fde6014b010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:50:06.335 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:06.334+0000 7fde57fff640 1 -- 172.21.15.123:0/4019724072 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fde4c004d50 msgr2=0x7fde6014b010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:50:06.335 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:06.334+0000 7fde57fff640 1 --2- 172.21.15.123:0/4019724072 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fde4c004d50 0x7fde6014b010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:06.336 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:06.334+0000 7fde57fff640 1 -- 172.21.15.123:0/4019724072 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fde6015e880 msgr2=0x7fde6014a330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:50:06.336 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:06.334+0000 7fde57fff640 1 --2- 172.21.15.123:0/4019724072 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fde6015e880 0x7fde6014a330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:06.336 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:06.334+0000 7fde57fff640 1 -- 172.21.15.123:0/4019724072 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fde601f68c0 con 0x7fde6014dce0 2024-09-16T15:50:06.336 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:06.334+0000 7fde5f7fe640 1 --2- 172.21.15.123:0/4019724072 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fde6015e880 0x7fde6014a330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T15:50:06.336 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:06.334+0000 7fde57fff640 1 --2- 172.21.15.123:0/4019724072 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fde6014dce0 0x7fde601f6090 secure :-1 s=READY pgs=1437 cs=0 l=1 rev1=1 crypto rx=0x7fde50010a10 tx=0x7fde50010ed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:50:06.336 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:06.334+0000 7fde577fe640 1 -- 172.21.15.123:0/4019724072 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde5000ece0 con 0x7fde6014dce0 2024-09-16T15:50:06.336 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:06.334+0000 7fde577fe640 1 -- 172.21.15.123:0/4019724072 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fde5000ee80 con 0x7fde6014dce0 2024-09-16T15:50:06.336 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:06.334+0000 7fde652bb640 1 -- 172.21.15.123:0/4019724072 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fde601f6ba0 con 0x7fde6014dce0 2024-09-16T15:50:06.336 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:06.334+0000 7fde577fe640 1 -- 172.21.15.123:0/4019724072 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde50005260 con 0x7fde6014dce0 2024-09-16T15:50:06.336 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:06.334+0000 7fde652bb640 1 -- 172.21.15.123:0/4019724072 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fde600c8b20 con 0x7fde6014dce0 2024-09-16T15:50:06.336 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:06.334+0000 7fde652bb640 1 -- 172.21.15.123:0/4019724072 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fde6006a110 con 0x7fde6014dce0 2024-09-16T15:50:06.337 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:06.334+0000 7fde577fe640 1 -- 172.21.15.123:0/4019724072 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 21) ==== 97641+0+0 (secure 0 0 0) 0x7fde50005400 con 0x7fde6014dce0 2024-09-16T15:50:06.337 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:06.334+0000 7fde577fe640 1 --2- 172.21.15.123:0/4019724072 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fde40073520 0x7fde400759e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:50:06.337 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:06.334+0000 7fde577fe640 1 -- 172.21.15.123:0/4019724072 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(135..135 src has 1..135) ==== 5737+0+0 (secure 0 0 0) 0x7fde50095970 con 0x7fde6014dce0 2024-09-16T15:50:06.337 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:06.334+0000 7fde5effd640 1 --2- 172.21.15.123:0/4019724072 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fde40073520 0x7fde400759e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:50:06.337 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:06.338+0000 7fde5effd640 1 --2- 172.21.15.123:0/4019724072 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fde40073520 0x7fde400759e0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7fde48009a30 tx=0x7fde480091e0 comp rx=0 tx=0).ready entity=mgr.4150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:50:06.340 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:06.342+0000 7fde577fe640 1 -- 172.21.15.123:0/4019724072 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7fde50060150 con 0x7fde6014dce0 2024-09-16T15:50:06.537 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:06.538+0000 7fde652bb640 1 -- 172.21.15.123:0/4019724072 --> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) -- 0x7fde60074210 con 0x7fde40073520 2024-09-16T15:50:06.539 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:06.538+0000 7fde577fe640 1 -- 172.21.15.123:0/4019724072 <== mgr.4150 v2:172.21.15.123:6836/2464759676 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+41 (secure 0 0 0) 0x7fde60074210 con 0x7fde40073520 2024-09-16T15:50:06.539 INFO:teuthology.orchestra.run.smithi123.stdout:[ 2024-09-16T15:50:06.539 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-09-16T15:50:06.539 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "cephfs" 2024-09-16T15:50:06.539 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-16T15:50:06.540 INFO:teuthology.orchestra.run.smithi123.stdout:] 2024-09-16T15:50:06.542 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:06.542+0000 7fde652bb640 1 -- 172.21.15.123:0/4019724072 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fde40073520 msgr2=0x7fde400759e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:50:06.542 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:06.542+0000 7fde652bb640 1 --2- 172.21.15.123:0/4019724072 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fde40073520 0x7fde400759e0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7fde48009a30 tx=0x7fde480091e0 comp rx=0 tx=0).stop 2024-09-16T15:50:06.542 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:06.542+0000 7fde652bb640 1 -- 172.21.15.123:0/4019724072 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fde6014dce0 msgr2=0x7fde601f6090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:50:06.542 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:06.542+0000 7fde652bb640 1 --2- 172.21.15.123:0/4019724072 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fde6014dce0 0x7fde601f6090 secure :-1 s=READY pgs=1437 cs=0 l=1 rev1=1 crypto rx=0x7fde50010a10 tx=0x7fde50010ed0 comp rx=0 tx=0).stop 2024-09-16T15:50:06.542 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:06.542+0000 7fde652bb640 1 -- 172.21.15.123:0/4019724072 shutdown_connections 2024-09-16T15:50:06.542 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:06.542+0000 7fde652bb640 1 --2- 172.21.15.123:0/4019724072 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fde40073520 0x7fde400759e0 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:06.542 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:06.542+0000 7fde652bb640 1 --2- 172.21.15.123:0/4019724072 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fde6015e880 0x7fde6014a330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:06.542 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:06.542+0000 7fde652bb640 1 --2- 172.21.15.123:0/4019724072 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fde6014dce0 0x7fde601f6090 unknown :-1 s=CLOSED pgs=1437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:06.542 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:06.542+0000 7fde652bb640 1 --2- 172.21.15.123:0/4019724072 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fde4c004d50 0x7fde6014b010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:06.542 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:06.542+0000 7fde652bb640 1 -- 172.21.15.123:0/4019724072 >> 172.21.15.123:0/4019724072 conn(0x7fde600cfb10 msgr2=0x7fde600b2db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:50:06.543 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:06.542+0000 7fde652bb640 1 -- 172.21.15.123:0/4019724072 shutdown_connections 2024-09-16T15:50:06.543 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:06.542+0000 7fde652bb640 1 -- 172.21.15.123:0/4019724072 wait complete. 2024-09-16T15:50:06.557 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-09-16T15:50:06.686 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:06.686+0000 7f5d04f5b640 1 -- 172.21.15.123:0/2508301567 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5d000cf410 msgr2=0x7f5d0015bab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:50:06.686 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:06.686+0000 7f5d04f5b640 1 --2- 172.21.15.123:0/2508301567 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5d000cf410 0x7f5d0015bab0 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f5cf8009bc0 tx=0x7f5cf801cb10 comp rx=0 tx=0).stop 2024-09-16T15:50:06.686 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:06.686+0000 7f5d04f5b640 1 -- 172.21.15.123:0/2508301567 shutdown_connections 2024-09-16T15:50:06.687 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:06.686+0000 7f5d04f5b640 1 --2- 172.21.15.123:0/2508301567 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5d000cf410 0x7f5d0015bab0 unknown :-1 s=CLOSED pgs=461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:06.687 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:06.686+0000 7f5d04f5b640 1 --2- 172.21.15.123:0/2508301567 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f5d000c31c0 0x7f5d0015b570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:06.687 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:06.686+0000 7f5d04f5b640 1 -- 172.21.15.123:0/2508301567 >> 172.21.15.123:0/2508301567 conn(0x7f5d000b1710 msgr2=0x7f5d000b1b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:50:06.687 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:06.686+0000 7f5d04f5b640 1 -- 172.21.15.123:0/2508301567 shutdown_connections 2024-09-16T15:50:06.687 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:06.686+0000 7f5d04f5b640 1 -- 172.21.15.123:0/2508301567 wait complete. 2024-09-16T15:50:06.687 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:06.686+0000 7f5d04f5b640 1 Processor -- start 2024-09-16T15:50:06.687 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:06.686+0000 7f5d04f5b640 1 -- start start 2024-09-16T15:50:06.689 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:06.686+0000 7f5d04f5b640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5d000c31c0 0x7f5d0015a7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:50:06.689 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:06.690+0000 7f5d04f5b640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5d000cf030 0x7f5d00147720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:50:06.689 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:06.690+0000 7f5d04f5b640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f5d000cf410 0x7f5d000c8de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:50:06.689 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:06.690+0000 7f5d04f5b640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f5d00168bd0 con 0x7f5d000c31c0 2024-09-16T15:50:06.690 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:06.690+0000 7f5d04f5b640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f5d00168a50 con 0x7f5d000cf030 2024-09-16T15:50:06.690 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:06.690+0000 7f5d04f5b640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f5d00168d50 con 0x7f5d000cf410 2024-09-16T15:50:06.690 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:06.690+0000 7f5cfe59c640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5d000c31c0 0x7f5d0015a7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:50:06.690 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:06.690+0000 7f5cfe59c640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5d000c31c0 0x7f5d0015a7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:53210/0 (socket says 172.21.15.123:53210) 2024-09-16T15:50:06.690 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:06.690+0000 7f5cfe59c640 1 -- 172.21.15.123:0/920252065 learned_addr learned my addr 172.21.15.123:0/920252065 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:50:06.690 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:06.690+0000 7f5cfdd9b640 1 --2- 172.21.15.123:0/920252065 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5d000cf030 0x7f5d00147720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:50:06.690 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:06.690+0000 7f5cfed9d640 1 --2- 172.21.15.123:0/920252065 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f5d000cf410 0x7f5d000c8de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:50:06.690 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:06.690+0000 7f5cfe59c640 1 -- 172.21.15.123:0/920252065 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f5d000cf410 msgr2=0x7f5d000c8de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:50:06.690 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:06.690+0000 7f5cfe59c640 1 --2- 172.21.15.123:0/920252065 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f5d000cf410 0x7f5d000c8de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:06.690 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:06.690+0000 7f5cfe59c640 1 -- 172.21.15.123:0/920252065 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5d000cf030 msgr2=0x7f5d00147720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:50:06.690 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:06.690+0000 7f5cfe59c640 1 --2- 172.21.15.123:0/920252065 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5d000cf030 0x7f5d00147720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:06.690 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:06.690+0000 7f5cfe59c640 1 -- 172.21.15.123:0/920252065 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5d00073140 con 0x7f5d000c31c0 2024-09-16T15:50:06.690 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:06.690+0000 7f5cfe59c640 1 --2- 172.21.15.123:0/920252065 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5d000c31c0 0x7f5d0015a7e0 secure :-1 s=READY pgs=1438 cs=0 l=1 rev1=1 crypto rx=0x7f5cf000c500 tx=0x7f5cf000c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:50:06.691 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:06.690+0000 7f5ceeffd640 1 -- 172.21.15.123:0/920252065 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5cf0017070 con 0x7f5d000c31c0 2024-09-16T15:50:06.691 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:06.690+0000 7f5ceeffd640 1 -- 172.21.15.123:0/920252065 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f5cf00053c0 con 0x7f5d000c31c0 2024-09-16T15:50:06.691 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:06.690+0000 7f5d04f5b640 1 -- 172.21.15.123:0/920252065 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5d001661f0 con 0x7f5d000c31c0 2024-09-16T15:50:06.691 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:06.690+0000 7f5ceeffd640 1 -- 172.21.15.123:0/920252065 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5cf0012890 con 0x7f5d000c31c0 2024-09-16T15:50:06.691 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:06.690+0000 7f5d04f5b640 1 -- 172.21.15.123:0/920252065 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5d00072940 con 0x7f5d000c31c0 2024-09-16T15:50:06.692 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:06.690+0000 7f5d04f5b640 1 -- 172.21.15.123:0/920252065 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5d0006fca0 con 0x7f5d000c31c0 2024-09-16T15:50:06.692 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:06.690+0000 7f5ceeffd640 1 -- 172.21.15.123:0/920252065 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 21) ==== 97641+0+0 (secure 0 0 0) 0x7f5cf001a070 con 0x7f5d000c31c0 2024-09-16T15:50:06.692 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:06.690+0000 7f5ceeffd640 1 --2- 172.21.15.123:0/920252065 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f5ce0073520 0x7f5ce00759e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:50:06.692 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:06.690+0000 7f5cfdd9b640 1 --2- 172.21.15.123:0/920252065 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f5ce0073520 0x7f5ce00759e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:50:06.692 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:06.690+0000 7f5ceeffd640 1 -- 172.21.15.123:0/920252065 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(135..135 src has 1..135) ==== 5737+0+0 (secure 0 0 0) 0x7f5cf0093d50 con 0x7f5d000c31c0 2024-09-16T15:50:06.692 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:06.690+0000 7f5cfdd9b640 1 --2- 172.21.15.123:0/920252065 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f5ce0073520 0x7f5ce00759e0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f5d001589a0 tx=0x7f5cf4009260 comp rx=0 tx=0).ready entity=mgr.4150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:50:06.695 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:06.694+0000 7f5ceeffd640 1 -- 172.21.15.123:0/920252065 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f5cf009a080 con 0x7f5d000c31c0 2024-09-16T15:50:06.869 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:06.870+0000 7f5d04f5b640 1 -- 172.21.15.123:0/920252065 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) -- 0x7f5d00073a60 con 0x7f5d000c31c0 2024-09-16T15:50:06.869 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:06.870+0000 7f5ceeffd640 1 -- 172.21.15.123:0/920252065 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v42)=0 v42) ==== 122+0+0 (secure 0 0 0) 0x7f5cf005e540 con 0x7f5d000c31c0 2024-09-16T15:50:06.872 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:06.870+0000 7f5d04f5b640 1 -- 172.21.15.123:0/920252065 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f5ce0073520 msgr2=0x7f5ce00759e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:50:06.872 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:06.870+0000 7f5d04f5b640 1 --2- 172.21.15.123:0/920252065 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f5ce0073520 0x7f5ce00759e0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f5d001589a0 tx=0x7f5cf4009260 comp rx=0 tx=0).stop 2024-09-16T15:50:06.872 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:06.870+0000 7f5d04f5b640 1 -- 172.21.15.123:0/920252065 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5d000c31c0 msgr2=0x7f5d0015a7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:50:06.872 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:06.870+0000 7f5d04f5b640 1 --2- 172.21.15.123:0/920252065 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5d000c31c0 0x7f5d0015a7e0 secure :-1 s=READY pgs=1438 cs=0 l=1 rev1=1 crypto rx=0x7f5cf000c500 tx=0x7f5cf000c9c0 comp rx=0 tx=0).stop 2024-09-16T15:50:06.872 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:06.870+0000 7f5d04f5b640 1 -- 172.21.15.123:0/920252065 shutdown_connections 2024-09-16T15:50:06.872 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:06.870+0000 7f5d04f5b640 1 --2- 172.21.15.123:0/920252065 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f5ce0073520 0x7f5ce00759e0 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:06.872 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:06.870+0000 7f5d04f5b640 1 --2- 172.21.15.123:0/920252065 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f5d000cf410 0x7f5d000c8de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:06.872 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:06.870+0000 7f5d04f5b640 1 --2- 172.21.15.123:0/920252065 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5d000cf030 0x7f5d00147720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:06.872 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:06.870+0000 7f5d04f5b640 1 --2- 172.21.15.123:0/920252065 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5d000c31c0 0x7f5d0015a7e0 unknown :-1 s=CLOSED pgs=1438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:06.872 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:06.870+0000 7f5d04f5b640 1 -- 172.21.15.123:0/920252065 >> 172.21.15.123:0/920252065 conn(0x7f5d000b1710 msgr2=0x7f5d0015da40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:50:06.873 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:06.870+0000 7f5d04f5b640 1 -- 172.21.15.123:0/920252065 shutdown_connections 2024-09-16T15:50:06.873 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:06.870+0000 7f5d04f5b640 1 -- 172.21.15.123:0/920252065 wait complete. 2024-09-16T15:50:06.889 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume create non_existent_subvolume subvolume_1 2024-09-16T15:50:07.017 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:07.018+0000 7f67f87d5640 1 -- 172.21.15.123:0/966997007 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f67dc004d50 msgr2=0x7f67dc0051d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:50:07.018 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:07.018+0000 7f67f87d5640 1 --2- 172.21.15.123:0/966997007 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f67dc004d50 0x7f67dc0051d0 secure :-1 s=READY pgs=1440 cs=0 l=1 rev1=1 crypto rx=0x7f67e800c6a0 tx=0x7f67e801d2e0 comp rx=0 tx=0).stop 2024-09-16T15:50:07.018 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:07.018+0000 7f67f6d72640 1 -- 172.21.15.123:0/966997007 reap_dead start 2024-09-16T15:50:07.018 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:07.018+0000 7f67f87d5640 1 -- 172.21.15.123:0/966997007 shutdown_connections 2024-09-16T15:50:07.018 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:07.018+0000 7f67f87d5640 1 -- 172.21.15.123:0/966997007 >> 172.21.15.123:0/966997007 conn(0x7f67f00b1710 msgr2=0x7f67f00b1b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:50:07.018 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:07.018+0000 7f67f87d5640 1 -- 172.21.15.123:0/966997007 shutdown_connections 2024-09-16T15:50:07.018 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:07.018+0000 7f67f87d5640 1 -- 172.21.15.123:0/966997007 wait complete. 2024-09-16T15:50:07.018 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:07.018+0000 7f67f87d5640 1 Processor -- start 2024-09-16T15:50:07.019 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:07.018+0000 7f67f87d5640 1 -- start start 2024-09-16T15:50:07.020 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:07.018+0000 7f67f87d5640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f67dc004d50 0x7f67f01f2580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:50:07.020 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:07.018+0000 7f67f87d5640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f67f00c99f0 0x7f67f00c8840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:50:07.021 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:07.018+0000 7f67f87d5640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f67f0154480 0x7f67f0154910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:50:07.021 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:07.018+0000 7f67f87d5640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f67f00cf530 con 0x7f67dc004d50 2024-09-16T15:50:07.021 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:07.018+0000 7f67f87d5640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f67f00cf3b0 con 0x7f67f00c99f0 2024-09-16T15:50:07.021 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:07.018+0000 7f67f87d5640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f67f00cf6b0 con 0x7f67f0154480 2024-09-16T15:50:07.021 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:07.018+0000 7f67f6571640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f67dc004d50 0x7f67f01f2580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:50:07.021 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:07.018+0000 7f67f6571640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f67dc004d50 0x7f67f01f2580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:53240/0 (socket says 172.21.15.123:53240) 2024-09-16T15:50:07.021 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:07.018+0000 7f67f6571640 1 -- 172.21.15.123:0/1078543316 learned_addr learned my addr 172.21.15.123:0/1078543316 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:50:07.021 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:07.018+0000 7f67f6d72640 1 --2- 172.21.15.123:0/1078543316 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f67f0154480 0x7f67f0154910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:50:07.021 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:07.018+0000 7f67f5d70640 1 --2- 172.21.15.123:0/1078543316 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f67f00c99f0 0x7f67f00c8840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:50:07.021 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:07.018+0000 7f67f6571640 1 -- 172.21.15.123:0/1078543316 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f67f0154480 msgr2=0x7f67f0154910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:50:07.021 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:07.018+0000 7f67f6571640 1 --2- 172.21.15.123:0/1078543316 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f67f0154480 0x7f67f0154910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:07.021 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:07.018+0000 7f67f6571640 1 -- 172.21.15.123:0/1078543316 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f67f00c99f0 msgr2=0x7f67f00c8840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:50:07.022 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:07.018+0000 7f67f6571640 1 --2- 172.21.15.123:0/1078543316 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f67f00c99f0 0x7f67f00c8840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:07.022 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:07.018+0000 7f67f6571640 1 -- 172.21.15.123:0/1078543316 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f67f0153520 con 0x7f67dc004d50 2024-09-16T15:50:07.022 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:07.022+0000 7f67f6571640 1 --2- 172.21.15.123:0/1078543316 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f67dc004d50 0x7f67f01f2580 secure :-1 s=READY pgs=1441 cs=0 l=1 rev1=1 crypto rx=0x7f67ec004e30 tx=0x7f67ec0015e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:50:07.022 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:07.022+0000 7f67e6ffd640 1 -- 172.21.15.123:0/1078543316 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67ec015020 con 0x7f67dc004d50 2024-09-16T15:50:07.022 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:07.022+0000 7f67e6ffd640 1 -- 172.21.15.123:0/1078543316 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f67ec011960 con 0x7f67dc004d50 2024-09-16T15:50:07.022 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:07.022+0000 7f67f87d5640 1 -- 172.21.15.123:0/1078543316 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f67f0153800 con 0x7f67dc004d50 2024-09-16T15:50:07.022 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:07.022+0000 7f67e6ffd640 1 -- 172.21.15.123:0/1078543316 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67ec001d50 con 0x7f67dc004d50 2024-09-16T15:50:07.022 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:07.022+0000 7f67f87d5640 1 -- 172.21.15.123:0/1078543316 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f67f0153dc0 con 0x7f67dc004d50 2024-09-16T15:50:07.023 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:07.022+0000 7f67f87d5640 1 -- 172.21.15.123:0/1078543316 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f67f00cc550 con 0x7f67dc004d50 2024-09-16T15:50:07.023 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:07.022+0000 7f67e6ffd640 1 -- 172.21.15.123:0/1078543316 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 21) ==== 97641+0+0 (secure 0 0 0) 0x7f67ec018020 con 0x7f67dc004d50 2024-09-16T15:50:07.023 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:07.022+0000 7f67e6ffd640 1 --2- 172.21.15.123:0/1078543316 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f67d40734f0 0x7f67d40759b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:50:07.023 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:07.022+0000 7f67e6ffd640 1 -- 172.21.15.123:0/1078543316 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(135..135 src has 1..135) ==== 5737+0+0 (secure 0 0 0) 0x7f67ec095a40 con 0x7f67dc004d50 2024-09-16T15:50:07.023 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:07.022+0000 7f67f5d70640 1 --2- 172.21.15.123:0/1078543316 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f67d40734f0 0x7f67d40759b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:50:07.023 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:07.022+0000 7f67f5d70640 1 --2- 172.21.15.123:0/1078543316 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f67d40734f0 0x7f67d40759b0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f67f00c9e10 tx=0x7f67e00091e0 comp rx=0 tx=0).ready entity=mgr.4150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:50:07.026 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:07.026+0000 7f67e6ffd640 1 -- 172.21.15.123:0/1078543316 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f67ec01f020 con 0x7f67dc004d50 2024-09-16T15:50:07.225 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:07.226+0000 7f67f87d5640 1 -- 172.21.15.123:0/1078543316 --> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "non_existent_subvolume", "sub_name": "subvolume_1", "target": ["mon-mgr", ""]}) -- 0x7f67f0073fe0 con 0x7f67d40734f0 2024-09-16T15:50:07.227 INFO:tasks.ceph.mgr.y.smithi123.stderr:2024-09-16T15:50:07.226+0000 7fa9b36f4640 -1 mgr.server reply reply (2) No such file or directory FS 'non_existent_subvolume' not found 2024-09-16T15:50:07.227 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:07.226+0000 7f67e6ffd640 1 -- 172.21.15.123:0/1078543316 <== mgr.4150 v2:172.21.15.123:6836/2464759676 1 ==== mgr_command_reply(tid 0: -2 FS 'non_existent_subvolume' not found) ==== 45+0+0 (secure 0 0 0) 0x7f67f0073fe0 con 0x7f67d40734f0 2024-09-16T15:50:07.227 INFO:teuthology.orchestra.run.smithi123.stderr:Error ENOENT: FS 'non_existent_subvolume' not found 2024-09-16T15:50:07.233 DEBUG:teuthology.orchestra.run:got remote process result: 2 2024-09-16T15:50:07.234 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume rm non_existent_subvolume subvolume_1 2024-09-16T15:50:07.358 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:07.358+0000 7f1f80ede640 1 -- 172.21.15.123:0/2155617399 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1f7c15bae0 msgr2=0x7f1f7c0d1bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:50:07.358 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:07.358+0000 7f1f80ede640 1 --2- 172.21.15.123:0/2155617399 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1f7c15bae0 0x7f1f7c0d1bc0 secure :-1 s=READY pgs=1443 cs=0 l=1 rev1=1 crypto rx=0x7f1f74009840 tx=0x7f1f7401d280 comp rx=0 tx=0).stop 2024-09-16T15:50:07.358 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:07.358+0000 7f1f80ede640 1 -- 172.21.15.123:0/2155617399 shutdown_connections 2024-09-16T15:50:07.358 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:07.358+0000 7f1f80ede640 1 --2- 172.21.15.123:0/2155617399 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f1f7c0b2d00 0x7f1f7c0aeae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:07.358 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:07.358+0000 7f1f80ede640 1 --2- 172.21.15.123:0/2155617399 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1f7c15bae0 0x7f1f7c0d1bc0 unknown :-1 s=CLOSED pgs=1443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:07.358 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:07.358+0000 7f1f80ede640 1 -- 172.21.15.123:0/2155617399 >> 172.21.15.123:0/2155617399 conn(0x7f1f7c09aa40 msgr2=0x7f1f7c09ae50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:50:07.358 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:07.358+0000 7f1f80ede640 1 -- 172.21.15.123:0/2155617399 shutdown_connections 2024-09-16T15:50:07.359 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:07.358+0000 7f1f80ede640 1 -- 172.21.15.123:0/2155617399 wait complete. 2024-09-16T15:50:07.359 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:07.358+0000 7f1f80ede640 1 Processor -- start 2024-09-16T15:50:07.359 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:07.358+0000 7f1f80ede640 1 -- start start 2024-09-16T15:50:07.361 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:07.358+0000 7f1f80ede640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1f7c0b2d00 0x7f1f7c0cb9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:50:07.361 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:07.358+0000 7f1f80ede640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f1f7c0d2100 0x7f1f7c0b5930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:50:07.361 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:07.358+0000 7f1f80ede640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1f7c15bae0 0x7f1f7c06f1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:50:07.361 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:07.358+0000 7f1f80ede640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f1f7c168910 con 0x7f1f7c15bae0 2024-09-16T15:50:07.361 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:07.358+0000 7f1f80ede640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f1f7c168790 con 0x7f1f7c0b2d00 2024-09-16T15:50:07.361 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:07.358+0000 7f1f80ede640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f1f7c168a90 con 0x7f1f7c0d2100 2024-09-16T15:50:07.362 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:07.362+0000 7f1f7ad9d640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1f7c15bae0 0x7f1f7c06f1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:50:07.362 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:07.362+0000 7f1f7ad9d640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1f7c15bae0 0x7f1f7c06f1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:53252/0 (socket says 172.21.15.123:53252) 2024-09-16T15:50:07.362 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:07.362+0000 7f1f7ad9d640 1 -- 172.21.15.123:0/3746416818 learned_addr learned my addr 172.21.15.123:0/3746416818 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:50:07.362 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:07.362+0000 7f1f79d9b640 1 --2- 172.21.15.123:0/3746416818 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f1f7c0d2100 0x7f1f7c0b5930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:50:07.362 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:07.362+0000 7f1f7a59c640 1 --2- 172.21.15.123:0/3746416818 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1f7c0b2d00 0x7f1f7c0cb9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:50:07.362 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:07.362+0000 7f1f7ad9d640 1 -- 172.21.15.123:0/3746416818 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f1f7c0d2100 msgr2=0x7f1f7c0b5930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:50:07.362 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:07.362+0000 7f1f7ad9d640 1 --2- 172.21.15.123:0/3746416818 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f1f7c0d2100 0x7f1f7c0b5930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:07.362 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:07.362+0000 7f1f7ad9d640 1 -- 172.21.15.123:0/3746416818 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1f7c0b2d00 msgr2=0x7f1f7c0cb9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:50:07.362 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:07.362+0000 7f1f7ad9d640 1 --2- 172.21.15.123:0/3746416818 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1f7c0b2d00 0x7f1f7c0cb9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:07.362 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:07.362+0000 7f1f7ad9d640 1 -- 172.21.15.123:0/3746416818 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1f7c06fd30 con 0x7f1f7c15bae0 2024-09-16T15:50:07.362 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:07.362+0000 7f1f79d9b640 1 --2- 172.21.15.123:0/3746416818 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f1f7c0d2100 0x7f1f7c0b5930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T15:50:07.362 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:07.362+0000 7f1f7ad9d640 1 --2- 172.21.15.123:0/3746416818 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1f7c15bae0 0x7f1f7c06f1e0 secure :-1 s=READY pgs=1444 cs=0 l=1 rev1=1 crypto rx=0x7f1f6c00ea80 tx=0x7f1f6c00ef40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:50:07.362 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:07.362+0000 7f1f72ffd640 1 -- 172.21.15.123:0/3746416818 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f6c00ce40 con 0x7f1f7c15bae0 2024-09-16T15:50:07.363 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:07.362+0000 7f1f72ffd640 1 -- 172.21.15.123:0/3746416818 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f1f6c004c40 con 0x7f1f7c15bae0 2024-09-16T15:50:07.363 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:07.362+0000 7f1f80ede640 1 -- 172.21.15.123:0/3746416818 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1f7c0cc020 con 0x7f1f7c15bae0 2024-09-16T15:50:07.363 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:07.362+0000 7f1f72ffd640 1 -- 172.21.15.123:0/3746416818 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f6c005280 con 0x7f1f7c15bae0 2024-09-16T15:50:07.363 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:07.362+0000 7f1f80ede640 1 -- 172.21.15.123:0/3746416818 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1f7c0b4e50 con 0x7f1f7c15bae0 2024-09-16T15:50:07.364 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:07.362+0000 7f1f80ede640 1 -- 172.21.15.123:0/3746416818 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1f7c0aeae0 con 0x7f1f7c15bae0 2024-09-16T15:50:07.364 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:07.362+0000 7f1f72ffd640 1 -- 172.21.15.123:0/3746416818 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 21) ==== 97641+0+0 (secure 0 0 0) 0x7f1f6c005420 con 0x7f1f7c15bae0 2024-09-16T15:50:07.364 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:07.362+0000 7f1f72ffd640 1 --2- 172.21.15.123:0/3746416818 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f1f5c073520 0x7f1f5c0759e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:50:07.364 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:07.362+0000 7f1f7a59c640 1 --2- 172.21.15.123:0/3746416818 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f1f5c073520 0x7f1f5c0759e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:50:07.364 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:07.362+0000 7f1f72ffd640 1 -- 172.21.15.123:0/3746416818 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(135..135 src has 1..135) ==== 5737+0+0 (secure 0 0 0) 0x7f1f6c0953c0 con 0x7f1f7c15bae0 2024-09-16T15:50:07.364 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:07.362+0000 7f1f7a59c640 1 --2- 172.21.15.123:0/3746416818 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f1f5c073520 0x7f1f5c0759e0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f1f740af000 tx=0x7f1f7401b060 comp rx=0 tx=0).ready entity=mgr.4150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:50:07.367 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:07.366+0000 7f1f72ffd640 1 -- 172.21.15.123:0/3746416818 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f1f7c06fd30 con 0x7f1f7c15bae0 2024-09-16T15:50:07.553 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:07.554+0000 7f1f80ede640 1 -- 172.21.15.123:0/3746416818 --> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] -- mgr_command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "non_existent_subvolume", "sub_name": "subvolume_1", "target": ["mon-mgr", ""]}) -- 0x7f1f7c1590f0 con 0x7f1f5c073520 2024-09-16T15:50:07.556 INFO:tasks.ceph.mgr.y.smithi123.stderr:2024-09-16T15:50:07.554+0000 7fa9b36f4640 -1 mgr.server reply reply (2) No such file or directory FS 'non_existent_subvolume' not found 2024-09-16T15:50:07.556 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:07.554+0000 7f1f72ffd640 1 -- 172.21.15.123:0/3746416818 <== mgr.4150 v2:172.21.15.123:6836/2464759676 1 ==== mgr_command_reply(tid 0: -2 FS 'non_existent_subvolume' not found) ==== 45+0+0 (secure 0 0 0) 0x7f1f7c1590f0 con 0x7f1f5c073520 2024-09-16T15:50:07.556 INFO:teuthology.orchestra.run.smithi123.stderr:Error ENOENT: FS 'non_existent_subvolume' not found 2024-09-16T15:50:07.562 DEBUG:teuthology.orchestra.run:got remote process result: 2 2024-09-16T15:50:07.562 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath non_existent_subvolume subvolume_1 2024-09-16T15:50:07.696 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:07.694+0000 7f39d285a640 1 -- 172.21.15.123:0/2571504541 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f39b4003fe0 msgr2=0x7f39b4004810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:50:07.696 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:07.694+0000 7f39d285a640 1 --2- 172.21.15.123:0/2571504541 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f39b4003fe0 0x7f39b4004810 secure :-1 s=READY pgs=1446 cs=0 l=1 rev1=1 crypto rx=0x7f39c000c290 tx=0x7f39c001cfa0 comp rx=0 tx=0).stop 2024-09-16T15:50:07.696 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:07.694+0000 7f39d0df7640 1 -- 172.21.15.123:0/2571504541 reap_dead start 2024-09-16T15:50:07.696 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:07.694+0000 7f39d285a640 1 -- 172.21.15.123:0/2571504541 shutdown_connections 2024-09-16T15:50:07.696 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:07.694+0000 7f39d285a640 1 -- 172.21.15.123:0/2571504541 >> 172.21.15.123:0/2571504541 conn(0x7f39cc0b1710 msgr2=0x7f39cc0b1b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:50:07.696 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:07.694+0000 7f39d285a640 1 -- 172.21.15.123:0/2571504541 shutdown_connections 2024-09-16T15:50:07.697 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:07.694+0000 7f39d285a640 1 -- 172.21.15.123:0/2571504541 wait complete. 2024-09-16T15:50:07.697 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:07.698+0000 7f39d285a640 1 Processor -- start 2024-09-16T15:50:07.697 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:07.698+0000 7f39d285a640 1 -- start start 2024-09-16T15:50:07.699 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:07.698+0000 7f39d285a640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f39b4003fe0 0x7f39cc072da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:50:07.700 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:07.698+0000 7f39d285a640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f39cc1f7d80 0x7f39cc0c5130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:50:07.700 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:07.698+0000 7f39d285a640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f39cc0b55a0 0x7f39cc0c5670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:50:07.700 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:07.698+0000 7f39d285a640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f39cc166d80 con 0x7f39b4003fe0 2024-09-16T15:50:07.700 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:07.698+0000 7f39d285a640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f39cc166c00 con 0x7f39cc1f7d80 2024-09-16T15:50:07.700 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:07.698+0000 7f39d285a640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f39cc166f00 con 0x7f39cc0b55a0 2024-09-16T15:50:07.700 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:07.698+0000 7f39cbfff640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f39b4003fe0 0x7f39cc072da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:50:07.700 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:07.698+0000 7f39cbfff640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f39b4003fe0 0x7f39cc072da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:53266/0 (socket says 172.21.15.123:53266) 2024-09-16T15:50:07.700 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:07.698+0000 7f39cbfff640 1 -- 172.21.15.123:0/1920314669 learned_addr learned my addr 172.21.15.123:0/1920314669 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:50:07.700 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:07.698+0000 7f39cb7fe640 1 --2- 172.21.15.123:0/1920314669 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f39cc1f7d80 0x7f39cc0c5130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:50:07.700 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:07.698+0000 7f39d0df7640 1 --2- 172.21.15.123:0/1920314669 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f39cc0b55a0 0x7f39cc0c5670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:50:07.700 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:07.698+0000 7f39cbfff640 1 -- 172.21.15.123:0/1920314669 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f39cc0b55a0 msgr2=0x7f39cc0c5670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:50:07.700 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:07.698+0000 7f39cbfff640 1 --2- 172.21.15.123:0/1920314669 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f39cc0b55a0 0x7f39cc0c5670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:07.701 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:07.698+0000 7f39cbfff640 1 -- 172.21.15.123:0/1920314669 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f39cc1f7d80 msgr2=0x7f39cc0c5130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:50:07.701 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:07.698+0000 7f39cbfff640 1 --2- 172.21.15.123:0/1920314669 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f39cc1f7d80 0x7f39cc0c5130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:07.701 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:07.698+0000 7f39cbfff640 1 -- 172.21.15.123:0/1920314669 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f39cc073380 con 0x7f39b4003fe0 2024-09-16T15:50:07.701 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:07.698+0000 7f39cbfff640 1 --2- 172.21.15.123:0/1920314669 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f39b4003fe0 0x7f39cc072da0 secure :-1 s=READY pgs=1447 cs=0 l=1 rev1=1 crypto rx=0x7f39bc010a10 tx=0x7f39bc010ed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:50:07.701 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:07.698+0000 7f39c8ff9640 1 -- 172.21.15.123:0/1920314669 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39bc00ec80 con 0x7f39b4003fe0 2024-09-16T15:50:07.701 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:07.698+0000 7f39c8ff9640 1 -- 172.21.15.123:0/1920314669 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f39bc00ee20 con 0x7f39b4003fe0 2024-09-16T15:50:07.701 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:07.698+0000 7f39d285a640 1 -- 172.21.15.123:0/1920314669 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f39cc0c8710 con 0x7f39b4003fe0 2024-09-16T15:50:07.701 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:07.698+0000 7f39c8ff9640 1 -- 172.21.15.123:0/1920314669 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39bc005280 con 0x7f39b4003fe0 2024-09-16T15:50:07.701 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:07.698+0000 7f39d285a640 1 -- 172.21.15.123:0/1920314669 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f39cc0c8cd0 con 0x7f39b4003fe0 2024-09-16T15:50:07.702 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:07.702+0000 7f39d285a640 1 -- 172.21.15.123:0/1920314669 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f39cc15f6b0 con 0x7f39b4003fe0 2024-09-16T15:50:07.702 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:07.702+0000 7f39c8ff9640 1 -- 172.21.15.123:0/1920314669 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 21) ==== 97641+0+0 (secure 0 0 0) 0x7f39bc005420 con 0x7f39b4003fe0 2024-09-16T15:50:07.702 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:07.702+0000 7f39c8ff9640 1 --2- 172.21.15.123:0/1920314669 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f39b80734f0 0x7f39b80759b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:50:07.702 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:07.702+0000 7f39cb7fe640 1 --2- 172.21.15.123:0/1920314669 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f39b80734f0 0x7f39b80759b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:50:07.702 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:07.702+0000 7f39c8ff9640 1 -- 172.21.15.123:0/1920314669 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(135..135 src has 1..135) ==== 5737+0+0 (secure 0 0 0) 0x7f39bc095660 con 0x7f39b4003fe0 2024-09-16T15:50:07.703 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:07.702+0000 7f39cb7fe640 1 --2- 172.21.15.123:0/1920314669 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f39b80734f0 0x7f39b80759b0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f39c00ac000 tx=0x7f39c0019de0 comp rx=0 tx=0).ready entity=mgr.4150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:50:07.706 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:07.706+0000 7f39c8ff9640 1 -- 172.21.15.123:0/1920314669 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f39bc05ff40 con 0x7f39b4003fe0 2024-09-16T15:50:07.890 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:07.890+0000 7f39d285a640 1 -- 172.21.15.123:0/1920314669 --> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "non_existent_subvolume", "sub_name": "subvolume_1", "target": ["mon-mgr", ""]}) -- 0x7f39cc0b0c20 con 0x7f39b80734f0 2024-09-16T15:50:07.893 INFO:tasks.ceph.mgr.y.smithi123.stderr:2024-09-16T15:50:07.894+0000 7fa9b36f4640 -1 mgr.server reply reply (2) No such file or directory FS 'non_existent_subvolume' not found 2024-09-16T15:50:07.893 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:07.894+0000 7f39c8ff9640 1 -- 172.21.15.123:0/1920314669 <== mgr.4150 v2:172.21.15.123:6836/2464759676 1 ==== mgr_command_reply(tid 0: -2 FS 'non_existent_subvolume' not found) ==== 45+0+0 (secure 0 0 0) 0x7f39cc0b0c20 con 0x7f39b80734f0 2024-09-16T15:50:07.893 INFO:teuthology.orchestra.run.smithi123.stderr:Error ENOENT: FS 'non_existent_subvolume' not found 2024-09-16T15:50:07.899 DEBUG:teuthology.orchestra.run:got remote process result: 2 2024-09-16T15:50:07.899 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume info non_existent_subvolume subvolume_1 2024-09-16T15:50:08.018 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:08.018+0000 7f93b39b7640 1 -- 172.21.15.123:0/1840276873 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f93ac072720 msgr2=0x7f93ac0cabc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:50:08.018 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:08.018+0000 7f93b39b7640 1 --2- 172.21.15.123:0/1840276873 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f93ac072720 0x7f93ac0cabc0 secure :-1 s=READY pgs=1448 cs=0 l=1 rev1=1 crypto rx=0x7f93ac0b26a0 tx=0x7f939c01c8c0 comp rx=0 tx=0).stop 2024-09-16T15:50:08.018 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:08.018+0000 7f93b39b7640 1 -- 172.21.15.123:0/1840276873 shutdown_connections 2024-09-16T15:50:08.018 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:08.018+0000 7f93b39b7640 1 --2- 172.21.15.123:0/1840276873 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f93ac0cc3c0 0x7f93ac0cc850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:08.018 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:08.018+0000 7f93b39b7640 1 --2- 172.21.15.123:0/1840276873 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f93ac072720 0x7f93ac0cabc0 unknown :-1 s=CLOSED pgs=1448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:08.018 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:08.018+0000 7f93b39b7640 1 --2- 172.21.15.123:0/1840276873 >> v2:172.21.15.184:3300/0 conn(0x7f93ac15aa00 0x7f93ac15e5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:08.018 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:08.018+0000 7f93b39b7640 1 -- 172.21.15.123:0/1840276873 >> 172.21.15.123:0/1840276873 conn(0x7f93ac0b1710 msgr2=0x7f93ac0b1b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:50:08.018 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:08.018+0000 7f93b39b7640 1 -- 172.21.15.123:0/1840276873 shutdown_connections 2024-09-16T15:50:08.018 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:08.018+0000 7f93b39b7640 1 -- 172.21.15.123:0/1840276873 wait complete. 2024-09-16T15:50:08.021 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:08.018+0000 7f93b39b7640 1 Processor -- start 2024-09-16T15:50:08.021 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:08.018+0000 7f93b39b7640 1 -- start start 2024-09-16T15:50:08.021 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:08.018+0000 7f93b39b7640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f93ac072720 0x7f93ac153ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:50:08.021 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:08.018+0000 7f93b39b7640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f93ac0cc3c0 0x7f93ac157510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:50:08.022 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:08.018+0000 7f93b39b7640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f93ac15aa00 0x7f93ac0c83e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:50:08.022 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:08.018+0000 7f93b39b7640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f93ac167b70 con 0x7f93ac0cc3c0 2024-09-16T15:50:08.022 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:08.018+0000 7f93b39b7640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f93ac06ff00 con 0x7f93ac15aa00 2024-09-16T15:50:08.022 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:08.018+0000 7f93b39b7640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f93ac167cf0 con 0x7f93ac072720 2024-09-16T15:50:08.022 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:08.018+0000 7f93b0f52640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f93ac0cc3c0 0x7f93ac157510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:50:08.022 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:08.018+0000 7f93b0f52640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f93ac0cc3c0 0x7f93ac157510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:53284/0 (socket says 172.21.15.123:53284) 2024-09-16T15:50:08.022 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:08.018+0000 7f93b0f52640 1 -- 172.21.15.123:0/2185282179 learned_addr learned my addr 172.21.15.123:0/2185282179 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:50:08.022 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:08.018+0000 7f93b1753640 1 --2- 172.21.15.123:0/2185282179 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f93ac072720 0x7f93ac153ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:50:08.022 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:08.018+0000 7f93b1f54640 1 --2- 172.21.15.123:0/2185282179 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f93ac15aa00 0x7f93ac0c83e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:50:08.022 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:08.018+0000 7f93b0f52640 1 -- 172.21.15.123:0/2185282179 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f93ac072720 msgr2=0x7f93ac153ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:50:08.022 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:08.018+0000 7f93b0f52640 1 --2- 172.21.15.123:0/2185282179 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f93ac072720 0x7f93ac153ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:08.023 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:08.018+0000 7f93b0f52640 1 -- 172.21.15.123:0/2185282179 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f93ac15aa00 msgr2=0x7f93ac0c83e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:50:08.023 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:08.018+0000 7f93b0f52640 1 --2- 172.21.15.123:0/2185282179 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f93ac15aa00 0x7f93ac0c83e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:08.023 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:08.018+0000 7f93b0f52640 1 -- 172.21.15.123:0/2185282179 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f93ac0c6e20 con 0x7f93ac0cc3c0 2024-09-16T15:50:08.023 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:08.018+0000 7f93b1753640 1 --2- 172.21.15.123:0/2185282179 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f93ac072720 0x7f93ac153ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T15:50:08.023 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:08.022+0000 7f93b0f52640 1 --2- 172.21.15.123:0/2185282179 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f93ac0cc3c0 0x7f93ac157510 secure :-1 s=READY pgs=1449 cs=0 l=1 rev1=1 crypto rx=0x7f939c01af70 tx=0x7f939c0a5690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:50:08.023 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:08.022+0000 7f93a1ffb640 1 -- 172.21.15.123:0/2185282179 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f939c0b2040 con 0x7f93ac0cc3c0 2024-09-16T15:50:08.023 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:08.022+0000 7f93a1ffb640 1 -- 172.21.15.123:0/2185282179 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f939c004430 con 0x7f93ac0cc3c0 2024-09-16T15:50:08.023 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:08.022+0000 7f93b39b7640 1 -- 172.21.15.123:0/2185282179 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f93ac0c7100 con 0x7f93ac0cc3c0 2024-09-16T15:50:08.023 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:08.022+0000 7f93a1ffb640 1 -- 172.21.15.123:0/2185282179 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f939c005290 con 0x7f93ac0cc3c0 2024-09-16T15:50:08.023 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:08.022+0000 7f93b39b7640 1 -- 172.21.15.123:0/2185282179 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f93ac0c76c0 con 0x7f93ac0cc3c0 2024-09-16T15:50:08.023 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:08.022+0000 7f93b39b7640 1 -- 172.21.15.123:0/2185282179 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f93ac0cc850 con 0x7f93ac0cc3c0 2024-09-16T15:50:08.024 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:08.022+0000 7f93a1ffb640 1 -- 172.21.15.123:0/2185282179 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 21) ==== 97641+0+0 (secure 0 0 0) 0x7f939c0be060 con 0x7f93ac0cc3c0 2024-09-16T15:50:08.024 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:08.022+0000 7f93a1ffb640 1 --2- 172.21.15.123:0/2185282179 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f93900734f0 0x7f93900759b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:50:08.024 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:08.022+0000 7f93a1ffb640 1 -- 172.21.15.123:0/2185282179 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(135..135 src has 1..135) ==== 5737+0+0 (secure 0 0 0) 0x7f939c12e3a0 con 0x7f93ac0cc3c0 2024-09-16T15:50:08.024 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:08.022+0000 7f93b1753640 1 --2- 172.21.15.123:0/2185282179 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f93900734f0 0x7f93900759b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:50:08.024 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:08.022+0000 7f93b1753640 1 --2- 172.21.15.123:0/2185282179 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f93900734f0 0x7f93900759b0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f93a80058d0 tx=0x7f93a8005ca0 comp rx=0 tx=0).ready entity=mgr.4150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:50:08.026 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:08.026+0000 7f93a1ffb640 1 -- 172.21.15.123:0/2185282179 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f939c0164b0 con 0x7f93ac0cc3c0 2024-09-16T15:50:08.212 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:08.214+0000 7f93b39b7640 1 -- 172.21.15.123:0/2185282179 --> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] -- mgr_command(tid 0: {"prefix": "fs subvolume info", "vol_name": "non_existent_subvolume", "sub_name": "subvolume_1", "target": ["mon-mgr", ""]}) -- 0x7f93ac0ca9a0 con 0x7f93900734f0 2024-09-16T15:50:08.214 INFO:tasks.ceph.mgr.y.smithi123.stderr:2024-09-16T15:50:08.214+0000 7fa9b36f4640 -1 mgr.server reply reply (2) No such file or directory FS 'non_existent_subvolume' not found 2024-09-16T15:50:08.214 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:08.214+0000 7f93a1ffb640 1 -- 172.21.15.123:0/2185282179 <== mgr.4150 v2:172.21.15.123:6836/2464759676 1 ==== mgr_command_reply(tid 0: -2 FS 'non_existent_subvolume' not found) ==== 45+0+0 (secure 0 0 0) 0x7f93ac0ca9a0 con 0x7f93900734f0 2024-09-16T15:50:08.214 INFO:teuthology.orchestra.run.smithi123.stderr:Error ENOENT: FS 'non_existent_subvolume' not found 2024-09-16T15:50:08.220 DEBUG:teuthology.orchestra.run:got remote process result: 2 2024-09-16T15:50:08.220 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume resize non_existent_subvolume subvolname_1 inf 2024-09-16T15:50:08.349 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:08.350+0000 7f5d85e63640 1 -- 172.21.15.123:0/4046494482 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5d800cfad0 msgr2=0x7f5d800cff00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:50:08.349 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:08.350+0000 7f5d85e63640 1 --2- 172.21.15.123:0/4046494482 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5d800cfad0 0x7f5d800cff00 secure :-1 s=READY pgs=1451 cs=0 l=1 rev1=1 crypto rx=0x7f5d78009bc0 tx=0x7f5d7801cb10 comp rx=0 tx=0).stop 2024-09-16T15:50:08.350 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:08.350+0000 7f5d85e63640 1 -- 172.21.15.123:0/4046494482 shutdown_connections 2024-09-16T15:50:08.350 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:08.350+0000 7f5d85e63640 1 --2- 172.21.15.123:0/4046494482 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5d800cfad0 0x7f5d800cff00 unknown :-1 s=CLOSED pgs=1451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:08.350 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:08.350+0000 7f5d85e63640 1 -- 172.21.15.123:0/4046494482 >> 172.21.15.123:0/4046494482 conn(0x7f5d80074110 msgr2=0x7f5d800744f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:50:08.350 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:08.350+0000 7f5d85e63640 1 -- 172.21.15.123:0/4046494482 shutdown_connections 2024-09-16T15:50:08.350 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:08.350+0000 7f5d85e63640 1 -- 172.21.15.123:0/4046494482 wait complete. 2024-09-16T15:50:08.350 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:08.350+0000 7f5d85e63640 1 Processor -- start 2024-09-16T15:50:08.350 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:08.350+0000 7f5d85e63640 1 -- start start 2024-09-16T15:50:08.352 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:08.350+0000 7f5d85e63640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5d800b14e0 0x7f5d80099680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:50:08.352 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:08.350+0000 7f5d85e63640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5d800cfad0 0x7f5d8015b810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:50:08.353 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:08.350+0000 7f5d85e63640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f5d8015f450 0x7f5d800ae710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:50:08.353 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:08.350+0000 7f5d85e63640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f5d8006e8d0 con 0x7f5d800cfad0 2024-09-16T15:50:08.353 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:08.350+0000 7f5d85e63640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f5d8006e750 con 0x7f5d800b14e0 2024-09-16T15:50:08.353 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:08.350+0000 7f5d85e63640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f5d8016a4f0 con 0x7f5d8015f450 2024-09-16T15:50:08.353 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:08.350+0000 7f5d76ffd640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5d800cfad0 0x7f5d8015b810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:50:08.353 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:08.350+0000 7f5d76ffd640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5d800cfad0 0x7f5d8015b810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:53304/0 (socket says 172.21.15.123:53304) 2024-09-16T15:50:08.353 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:08.350+0000 7f5d76ffd640 1 -- 172.21.15.123:0/1594182186 learned_addr learned my addr 172.21.15.123:0/1594182186 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:50:08.353 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:08.350+0000 7f5d7f7fe640 1 --2- 172.21.15.123:0/1594182186 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5d800b14e0 0x7f5d80099680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:50:08.353 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:08.350+0000 7f5d7ffff640 1 --2- 172.21.15.123:0/1594182186 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f5d8015f450 0x7f5d800ae710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:50:08.353 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:08.350+0000 7f5d76ffd640 1 -- 172.21.15.123:0/1594182186 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f5d8015f450 msgr2=0x7f5d800ae710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:50:08.353 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:08.350+0000 7f5d76ffd640 1 --2- 172.21.15.123:0/1594182186 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f5d8015f450 0x7f5d800ae710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:08.353 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:08.350+0000 7f5d76ffd640 1 -- 172.21.15.123:0/1594182186 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5d800b14e0 msgr2=0x7f5d80099680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:50:08.354 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:08.350+0000 7f5d76ffd640 1 --2- 172.21.15.123:0/1594182186 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5d800b14e0 0x7f5d80099680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:08.354 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:08.350+0000 7f5d76ffd640 1 -- 172.21.15.123:0/1594182186 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5d8015c730 con 0x7f5d800cfad0 2024-09-16T15:50:08.354 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:08.350+0000 7f5d76ffd640 1 --2- 172.21.15.123:0/1594182186 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5d800cfad0 0x7f5d8015b810 secure :-1 s=READY pgs=1452 cs=0 l=1 rev1=1 crypto rx=0x7f5d6800c450 tx=0x7f5d6800c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:50:08.354 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:08.354+0000 7f5d7cff9640 1 -- 172.21.15.123:0/1594182186 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d68012830 con 0x7f5d800cfad0 2024-09-16T15:50:08.354 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:08.354+0000 7f5d7cff9640 1 -- 172.21.15.123:0/1594182186 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f5d68017070 con 0x7f5d800cfad0 2024-09-16T15:50:08.354 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:08.354+0000 7f5d85e63640 1 -- 172.21.15.123:0/1594182186 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5d8015c450 con 0x7f5d800cfad0 2024-09-16T15:50:08.354 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:08.354+0000 7f5d7cff9640 1 -- 172.21.15.123:0/1594182186 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d68012d80 con 0x7f5d800cfad0 2024-09-16T15:50:08.354 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:08.354+0000 7f5d85e63640 1 -- 172.21.15.123:0/1594182186 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5d8015b2f0 con 0x7f5d800cfad0 2024-09-16T15:50:08.355 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:08.354+0000 7f5d85e63640 1 -- 172.21.15.123:0/1594182186 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5d800cff00 con 0x7f5d800cfad0 2024-09-16T15:50:08.355 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:08.354+0000 7f5d7cff9640 1 -- 172.21.15.123:0/1594182186 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 21) ==== 97641+0+0 (secure 0 0 0) 0x7f5d6801c460 con 0x7f5d800cfad0 2024-09-16T15:50:08.355 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:08.354+0000 7f5d7cff9640 1 --2- 172.21.15.123:0/1594182186 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f5d60073520 0x7f5d600759e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:50:08.355 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:08.354+0000 7f5d7cff9640 1 -- 172.21.15.123:0/1594182186 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(135..135 src has 1..135) ==== 5737+0+0 (secure 0 0 0) 0x7f5d68094850 con 0x7f5d800cfad0 2024-09-16T15:50:08.355 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:08.354+0000 7f5d7f7fe640 1 --2- 172.21.15.123:0/1594182186 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f5d60073520 0x7f5d600759e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:50:08.355 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:08.354+0000 7f5d7f7fe640 1 --2- 172.21.15.123:0/1594182186 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f5d60073520 0x7f5d600759e0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f5d70001640 tx=0x7f5d70009040 comp rx=0 tx=0).ready entity=mgr.4150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:50:08.358 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:08.358+0000 7f5d7cff9640 1 -- 172.21.15.123:0/1594182186 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f5d6805f030 con 0x7f5d800cfad0 2024-09-16T15:50:08.565 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:08.566+0000 7f5d85e63640 1 -- 172.21.15.123:0/1594182186 --> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] -- mgr_command(tid 0: {"prefix": "fs subvolume resize", "vol_name": "non_existent_subvolume", "sub_name": "subvolname_1", "new_size": "inf", "target": ["mon-mgr", ""]}) -- 0x7f5d800b62b0 con 0x7f5d60073520 2024-09-16T15:50:08.568 INFO:tasks.ceph.mgr.y.smithi123.stderr:2024-09-16T15:50:08.566+0000 7fa9b36f4640 -1 mgr.server reply reply (2) No such file or directory FS 'non_existent_subvolume' not found 2024-09-16T15:50:08.568 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:08.566+0000 7f5d7cff9640 1 -- 172.21.15.123:0/1594182186 <== mgr.4150 v2:172.21.15.123:6836/2464759676 1 ==== mgr_command_reply(tid 0: -2 FS 'non_existent_subvolume' not found) ==== 45+0+0 (secure 0 0 0) 0x7f5d800b62b0 con 0x7f5d60073520 2024-09-16T15:50:08.568 INFO:teuthology.orchestra.run.smithi123.stderr:Error ENOENT: FS 'non_existent_subvolume' not found 2024-09-16T15:50:08.573 DEBUG:teuthology.orchestra.run:got remote process result: 2 2024-09-16T15:50:08.573 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume pin non_existent_subvolume subvolname_1 export 1 2024-09-16T15:50:08.704 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:08.706+0000 7fe3a27b4640 1 -- 172.21.15.123:0/3262774854 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe39c152cf0 msgr2=0x7fe39c0725e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:50:08.705 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:08.706+0000 7fe3a27b4640 1 --2- 172.21.15.123:0/3262774854 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe39c152cf0 0x7fe39c0725e0 secure :-1 s=READY pgs=1453 cs=0 l=1 rev1=1 crypto rx=0x7fe39c0768d0 tx=0x7fe38401c6c0 comp rx=0 tx=0).stop 2024-09-16T15:50:08.705 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:08.706+0000 7fe3a27b4640 1 -- 172.21.15.123:0/3262774854 shutdown_connections 2024-09-16T15:50:08.705 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:08.706+0000 7fe3a27b4640 1 --2- 172.21.15.123:0/3262774854 >> v2:172.21.15.184:3300/0 conn(0x7fe39c070270 0x7fe39c15f670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:08.705 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:08.706+0000 7fe3a27b4640 1 --2- 172.21.15.123:0/3262774854 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe39c152cf0 0x7fe39c0725e0 unknown :-1 s=CLOSED pgs=1453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:08.705 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:08.706+0000 7fe3a27b4640 1 -- 172.21.15.123:0/3262774854 >> 172.21.15.123:0/3262774854 conn(0x7fe39c0b1630 msgr2=0x7fe39c0b1a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:50:08.705 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:08.706+0000 7fe3a27b4640 1 -- 172.21.15.123:0/3262774854 shutdown_connections 2024-09-16T15:50:08.705 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:08.706+0000 7fe3a27b4640 1 -- 172.21.15.123:0/3262774854 wait complete. 2024-09-16T15:50:08.706 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:08.706+0000 7fe3a27b4640 1 Processor -- start 2024-09-16T15:50:08.706 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:08.706+0000 7fe3a27b4640 1 -- start start 2024-09-16T15:50:08.708 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:08.706+0000 7fe3a27b4640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe39c070270 0x7fe39c13c160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:50:08.708 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:08.706+0000 7fe3a27b4640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fe39c0706c0 0x7fe39c15d1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:50:08.708 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:08.706+0000 7fe3a27b4640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe39c152cf0 0x7fe39c15d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:50:08.708 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:08.706+0000 7fe3a27b4640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fe39c1695b0 con 0x7fe39c152cf0 2024-09-16T15:50:08.708 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:08.706+0000 7fe3a27b4640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fe39c169430 con 0x7fe39c070270 2024-09-16T15:50:08.708 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:08.706+0000 7fe3a27b4640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fe39c169730 con 0x7fe39c0706c0 2024-09-16T15:50:08.708 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:08.706+0000 7fe3a0d51640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe39c152cf0 0x7fe39c15d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:50:08.708 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:08.706+0000 7fe3a0d51640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe39c152cf0 0x7fe39c15d6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:53312/0 (socket says 172.21.15.123:53312) 2024-09-16T15:50:08.708 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:08.706+0000 7fe3a0d51640 1 -- 172.21.15.123:0/1159010955 learned_addr learned my addr 172.21.15.123:0/1159010955 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:50:08.708 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:08.706+0000 7fe39bfff640 1 --2- 172.21.15.123:0/1159010955 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe39c070270 0x7fe39c13c160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:50:08.708 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:08.706+0000 7fe3937fe640 1 --2- 172.21.15.123:0/1159010955 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fe39c0706c0 0x7fe39c15d1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:50:08.708 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:08.706+0000 7fe3a0d51640 1 -- 172.21.15.123:0/1159010955 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fe39c0706c0 msgr2=0x7fe39c15d1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:50:08.709 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:08.706+0000 7fe3a0d51640 1 --2- 172.21.15.123:0/1159010955 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fe39c0706c0 0x7fe39c15d1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:08.709 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:08.706+0000 7fe3a0d51640 1 -- 172.21.15.123:0/1159010955 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe39c070270 msgr2=0x7fe39c13c160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:50:08.709 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:08.706+0000 7fe3a0d51640 1 --2- 172.21.15.123:0/1159010955 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe39c070270 0x7fe39c13c160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:08.709 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:08.706+0000 7fe3a0d51640 1 -- 172.21.15.123:0/1159010955 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe39c14a710 con 0x7fe39c152cf0 2024-09-16T15:50:08.709 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:08.706+0000 7fe39bfff640 1 --2- 172.21.15.123:0/1159010955 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe39c070270 0x7fe39c13c160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T15:50:08.709 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:08.706+0000 7fe3a0d51640 1 --2- 172.21.15.123:0/1159010955 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe39c152cf0 0x7fe39c15d6f0 secure :-1 s=READY pgs=1454 cs=0 l=1 rev1=1 crypto rx=0x7fe39400c780 tx=0x7fe39400cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:50:08.709 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:08.706+0000 7fe3997fa640 1 -- 172.21.15.123:0/1159010955 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe394016020 con 0x7fe39c152cf0 2024-09-16T15:50:08.709 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:08.706+0000 7fe3997fa640 1 -- 172.21.15.123:0/1159010955 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fe39400aae0 con 0x7fe39c152cf0 2024-09-16T15:50:08.709 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:08.706+0000 7fe3a27b4640 1 -- 172.21.15.123:0/1159010955 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe39c14a9f0 con 0x7fe39c152cf0 2024-09-16T15:50:08.709 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:08.706+0000 7fe3997fa640 1 -- 172.21.15.123:0/1159010955 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe394012420 con 0x7fe39c152cf0 2024-09-16T15:50:08.709 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:08.706+0000 7fe3a27b4640 1 -- 172.21.15.123:0/1159010955 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe39c14afb0 con 0x7fe39c152cf0 2024-09-16T15:50:08.710 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:08.710+0000 7fe3a27b4640 1 -- 172.21.15.123:0/1159010955 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe39c15f670 con 0x7fe39c152cf0 2024-09-16T15:50:08.710 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:08.710+0000 7fe3997fa640 1 -- 172.21.15.123:0/1159010955 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 21) ==== 97641+0+0 (secure 0 0 0) 0x7fe394019020 con 0x7fe39c152cf0 2024-09-16T15:50:08.710 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:08.710+0000 7fe3997fa640 1 --2- 172.21.15.123:0/1159010955 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fe380073520 0x7fe3800759e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:50:08.710 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:08.710+0000 7fe3997fa640 1 -- 172.21.15.123:0/1159010955 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(135..135 src has 1..135) ==== 5737+0+0 (secure 0 0 0) 0x7fe394094a30 con 0x7fe39c152cf0 2024-09-16T15:50:08.710 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:08.710+0000 7fe39bfff640 1 --2- 172.21.15.123:0/1159010955 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fe380073520 0x7fe3800759e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:50:08.710 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:08.710+0000 7fe39bfff640 1 --2- 172.21.15.123:0/1159010955 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fe380073520 0x7fe3800759e0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7fe3840ae000 tx=0x7fe38401af00 comp rx=0 tx=0).ready entity=mgr.4150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:50:08.713 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:08.714+0000 7fe3997fa640 1 -- 172.21.15.123:0/1159010955 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7fe394020030 con 0x7fe39c152cf0 2024-09-16T15:50:08.908 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:08.910+0000 7fe3a27b4640 1 -- 172.21.15.123:0/1159010955 --> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] -- mgr_command(tid 0: {"prefix": "fs subvolume pin", "vol_name": "non_existent_subvolume", "sub_name": "subvolname_1", "pin_type": "export", "pin_setting": "1", "target": ["mon-mgr", ""]}) -- 0x7fe39c0741f0 con 0x7fe380073520 2024-09-16T15:50:08.911 INFO:tasks.ceph.mgr.y.smithi123.stderr:2024-09-16T15:50:08.910+0000 7fa9b36f4640 -1 mgr.server reply reply (2) No such file or directory FS 'non_existent_subvolume' not found 2024-09-16T15:50:08.912 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:08.910+0000 7fe3997fa640 1 -- 172.21.15.123:0/1159010955 <== mgr.4150 v2:172.21.15.123:6836/2464759676 1 ==== mgr_command_reply(tid 0: -2 FS 'non_existent_subvolume' not found) ==== 45+0+0 (secure 0 0 0) 0x7fe39c0741f0 con 0x7fe380073520 2024-09-16T15:50:08.912 INFO:teuthology.orchestra.run.smithi123.stderr:Error ENOENT: FS 'non_existent_subvolume' not found 2024-09-16T15:50:08.918 DEBUG:teuthology.orchestra.run:got remote process result: 2 2024-09-16T15:50:08.919 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume ls non_existent_subvolume 2024-09-16T15:50:09.045 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:09.046+0000 7f00d8fd1640 1 -- 172.21.15.123:0/1367164178 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f00d40d0510 msgr2=0x7f00d40d0980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:50:09.045 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:09.046+0000 7f00d8fd1640 1 --2- 172.21.15.123:0/1367164178 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f00d40d0510 0x7f00d40d0980 secure :-1 s=READY pgs=1456 cs=0 l=1 rev1=1 crypto rx=0x7f00cc009bc0 tx=0x7f00cc01cba0 comp rx=0 tx=0).stop 2024-09-16T15:50:09.045 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:09.046+0000 7f00d8fd1640 1 -- 172.21.15.123:0/1367164178 shutdown_connections 2024-09-16T15:50:09.045 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:09.046+0000 7f00d8fd1640 1 --2- 172.21.15.123:0/1367164178 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f00d40d0510 0x7f00d40d0980 unknown :-1 s=CLOSED pgs=1456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:09.045 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:09.046+0000 7f00d8fd1640 1 --2- 172.21.15.123:0/1367164178 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f00d40cc070 0x7f00d415b410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:09.046 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:09.046+0000 7f00d8fd1640 1 -- 172.21.15.123:0/1367164178 >> 172.21.15.123:0/1367164178 conn(0x7f00d406d670 msgr2=0x7f00d40743a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:50:09.046 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:09.046+0000 7f00d8fd1640 1 -- 172.21.15.123:0/1367164178 shutdown_connections 2024-09-16T15:50:09.046 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:09.046+0000 7f00d8fd1640 1 -- 172.21.15.123:0/1367164178 wait complete. 2024-09-16T15:50:09.046 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:09.046+0000 7f00d8fd1640 1 Processor -- start 2024-09-16T15:50:09.046 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:09.046+0000 7f00d8fd1640 1 -- start start 2024-09-16T15:50:09.047 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:09.046+0000 7f00d8fd1640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f00d40cc070 0x7f00d40d0920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:50:09.047 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:09.046+0000 7f00d8fd1640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f00d40cc450 0x7f00d4154d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:50:09.048 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:09.046+0000 7f00d8fd1640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f00d40d0510 0x7f00d41f9610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:50:09.048 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:09.046+0000 7f00d8fd1640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f00d415a450 con 0x7f00d40cc070 2024-09-16T15:50:09.048 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:09.046+0000 7f00d8fd1640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f00d415a2d0 con 0x7f00d40cc450 2024-09-16T15:50:09.048 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:09.046+0000 7f00d8fd1640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f00d415a5d0 con 0x7f00d40d0510 2024-09-16T15:50:09.048 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:09.046+0000 7f00d259c640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f00d40cc070 0x7f00d40d0920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:50:09.048 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:09.046+0000 7f00d259c640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f00d40cc070 0x7f00d40d0920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:53344/0 (socket says 172.21.15.123:53344) 2024-09-16T15:50:09.048 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:09.046+0000 7f00d259c640 1 -- 172.21.15.123:0/2599379965 learned_addr learned my addr 172.21.15.123:0/2599379965 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:50:09.048 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:09.046+0000 7f00d2d9d640 1 --2- 172.21.15.123:0/2599379965 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f00d40d0510 0x7f00d41f9610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:50:09.049 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:09.046+0000 7f00d259c640 1 -- 172.21.15.123:0/2599379965 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f00d40d0510 msgr2=0x7f00d41f9610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:50:09.049 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:09.046+0000 7f00d259c640 1 --2- 172.21.15.123:0/2599379965 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f00d40d0510 0x7f00d41f9610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:09.049 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:09.046+0000 7f00d259c640 1 -- 172.21.15.123:0/2599379965 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f00d40cc450 msgr2=0x7f00d4154d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:50:09.049 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:09.046+0000 7f00d259c640 1 --2- 172.21.15.123:0/2599379965 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f00d40cc450 0x7f00d4154d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:09.049 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:09.046+0000 7f00d259c640 1 -- 172.21.15.123:0/2599379965 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f00d40cddb0 con 0x7f00d40cc070 2024-09-16T15:50:09.049 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:09.046+0000 7f00d259c640 1 --2- 172.21.15.123:0/2599379965 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f00d40cc070 0x7f00d40d0920 secure :-1 s=READY pgs=1457 cs=0 l=1 rev1=1 crypto rx=0x7f00c400d950 tx=0x7f00c400de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:50:09.050 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:09.050+0000 7f00c2ffd640 1 -- 172.21.15.123:0/2599379965 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00c4017070 con 0x7f00d40cc070 2024-09-16T15:50:09.050 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:09.050+0000 7f00c2ffd640 1 -- 172.21.15.123:0/2599379965 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f00c4009d20 con 0x7f00d40cc070 2024-09-16T15:50:09.050 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:09.050+0000 7f00d8fd1640 1 -- 172.21.15.123:0/2599379965 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f00d40b3570 con 0x7f00d40cc070 2024-09-16T15:50:09.050 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:09.050+0000 7f00c2ffd640 1 -- 172.21.15.123:0/2599379965 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00c40106c0 con 0x7f00d40cc070 2024-09-16T15:50:09.050 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:09.050+0000 7f00d8fd1640 1 -- 172.21.15.123:0/2599379965 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f00d40b3b30 con 0x7f00d40cc070 2024-09-16T15:50:09.050 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:09.050+0000 7f00d8fd1640 1 -- 172.21.15.123:0/2599379965 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f00d40d11d0 con 0x7f00d40cc070 2024-09-16T15:50:09.050 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:09.050+0000 7f00c2ffd640 1 -- 172.21.15.123:0/2599379965 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 21) ==== 97641+0+0 (secure 0 0 0) 0x7f00c4010860 con 0x7f00d40cc070 2024-09-16T15:50:09.050 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:09.050+0000 7f00c2ffd640 1 --2- 172.21.15.123:0/2599379965 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f00b4073520 0x7f00b40759e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:50:09.051 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:09.050+0000 7f00c2ffd640 1 -- 172.21.15.123:0/2599379965 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(135..135 src has 1..135) ==== 5737+0+0 (secure 0 0 0) 0x7f00c4094e90 con 0x7f00d40cc070 2024-09-16T15:50:09.051 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:09.050+0000 7f00d1d9b640 1 --2- 172.21.15.123:0/2599379965 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f00b4073520 0x7f00b40759e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:50:09.051 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:09.050+0000 7f00d1d9b640 1 --2- 172.21.15.123:0/2599379965 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f00b4073520 0x7f00b40759e0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f00d40ca630 tx=0x7f00c8007a00 comp rx=0 tx=0).ready entity=mgr.4150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:50:09.054 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:09.054+0000 7f00c2ffd640 1 -- 172.21.15.123:0/2599379965 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f00c405f670 con 0x7f00d40cc070 2024-09-16T15:50:09.238 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:09.238+0000 7f00d8fd1640 1 -- 172.21.15.123:0/2599379965 --> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] -- mgr_command(tid 0: {"prefix": "fs subvolume ls", "vol_name": "non_existent_subvolume", "target": ["mon-mgr", ""]}) -- 0x7f00d415b3f0 con 0x7f00b4073520 2024-09-16T15:50:09.240 INFO:tasks.ceph.mgr.y.smithi123.stderr:2024-09-16T15:50:09.238+0000 7fa9b36f4640 -1 mgr.server reply reply (2) No such file or directory FS 'non_existent_subvolume' not found 2024-09-16T15:50:09.240 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:09.238+0000 7f00c2ffd640 1 -- 172.21.15.123:0/2599379965 <== mgr.4150 v2:172.21.15.123:6836/2464759676 1 ==== mgr_command_reply(tid 0: -2 FS 'non_existent_subvolume' not found) ==== 45+0+0 (secure 0 0 0) 0x7f00d415b3f0 con 0x7f00b4073520 2024-09-16T15:50:09.240 INFO:teuthology.orchestra.run.smithi123.stderr:Error ENOENT: FS 'non_existent_subvolume' not found 2024-09-16T15:50:09.245 DEBUG:teuthology.orchestra.run:got remote process result: 2 2024-09-16T15:50:09.245 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot create non_existent_subvolume subvolume_1 snapshot_1 2024-09-16T15:50:09.372 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:09.370+0000 7ff355f18640 1 -- 172.21.15.123:0/3179091467 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff35015bc40 msgr2=0x7ff350070470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:50:09.372 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:09.370+0000 7ff355f18640 1 --2- 172.21.15.123:0/3179091467 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff35015bc40 0x7ff350070470 secure :-1 s=READY pgs=1459 cs=0 l=1 rev1=1 crypto rx=0x7ff340009840 tx=0x7ff34001c760 comp rx=0 tx=0).stop 2024-09-16T15:50:09.372 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:09.370+0000 7ff355f18640 1 -- 172.21.15.123:0/3179091467 shutdown_connections 2024-09-16T15:50:09.372 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:09.370+0000 7ff355f18640 1 --2- 172.21.15.123:0/3179091467 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff35015bc40 0x7ff350070470 unknown :-1 s=CLOSED pgs=1459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:09.373 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:09.370+0000 7ff355f18640 1 -- 172.21.15.123:0/3179091467 >> 172.21.15.123:0/3179091467 conn(0x7ff3500b1630 msgr2=0x7ff3500b1a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:50:09.373 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:09.370+0000 7ff355f18640 1 -- 172.21.15.123:0/3179091467 shutdown_connections 2024-09-16T15:50:09.373 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:09.370+0000 7ff355f18640 1 -- 172.21.15.123:0/3179091467 wait complete. 2024-09-16T15:50:09.373 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:09.374+0000 7ff355f18640 1 Processor -- start 2024-09-16T15:50:09.373 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:09.374+0000 7ff355f18640 1 -- start start 2024-09-16T15:50:09.375 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:09.374+0000 7ff355f18640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7ff35015bc40 0x7ff350151e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:50:09.375 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:09.374+0000 7ff355f18640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff35015f690 0x7ff35014ce50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:50:09.375 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:09.374+0000 7ff355f18640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff350160310 0x7ff350148aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:50:09.375 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:09.374+0000 7ff355f18640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7ff35016a5e0 con 0x7ff350160310 2024-09-16T15:50:09.376 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:09.374+0000 7ff355f18640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7ff35016a460 con 0x7ff35015f690 2024-09-16T15:50:09.376 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:09.374+0000 7ff355f18640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7ff35016a760 con 0x7ff35015bc40 2024-09-16T15:50:09.376 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:09.374+0000 7ff34ffff640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff350160310 0x7ff350148aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:50:09.376 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:09.374+0000 7ff34ffff640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff350160310 0x7ff350148aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:53366/0 (socket says 172.21.15.123:53366) 2024-09-16T15:50:09.376 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:09.374+0000 7ff34ffff640 1 -- 172.21.15.123:0/2405891841 learned_addr learned my addr 172.21.15.123:0/2405891841 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:50:09.376 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:09.374+0000 7ff34effd640 1 --2- 172.21.15.123:0/2405891841 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff35015f690 0x7ff35014ce50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:50:09.376 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:09.374+0000 7ff34ffff640 1 -- 172.21.15.123:0/2405891841 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7ff35015bc40 msgr2=0x7ff350151e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:50:09.376 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:09.374+0000 7ff34ffff640 1 --2- 172.21.15.123:0/2405891841 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7ff35015bc40 0x7ff350151e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:09.376 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:09.374+0000 7ff34ffff640 1 -- 172.21.15.123:0/2405891841 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff35015f690 msgr2=0x7ff35014ce50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:50:09.376 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:09.374+0000 7ff34ffff640 1 --2- 172.21.15.123:0/2405891841 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff35015f690 0x7ff35014ce50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:09.376 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:09.374+0000 7ff34ffff640 1 -- 172.21.15.123:0/2405891841 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff350148fe0 con 0x7ff350160310 2024-09-16T15:50:09.376 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:09.374+0000 7ff34ffff640 1 --2- 172.21.15.123:0/2405891841 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff350160310 0x7ff350148aa0 secure :-1 s=READY pgs=1460 cs=0 l=1 rev1=1 crypto rx=0x7ff34800c770 tx=0x7ff34800cc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:50:09.377 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:09.374+0000 7ff337fff640 1 -- 172.21.15.123:0/2405891841 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff348017070 con 0x7ff350160310 2024-09-16T15:50:09.377 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:09.374+0000 7ff337fff640 1 -- 172.21.15.123:0/2405891841 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7ff348005070 con 0x7ff350160310 2024-09-16T15:50:09.377 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:09.374+0000 7ff355f18640 1 -- 172.21.15.123:0/2405891841 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff3501fb9b0 con 0x7ff350160310 2024-09-16T15:50:09.377 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:09.374+0000 7ff337fff640 1 -- 172.21.15.123:0/2405891841 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3480054c0 con 0x7ff350160310 2024-09-16T15:50:09.377 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:09.374+0000 7ff355f18640 1 -- 172.21.15.123:0/2405891841 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff3501fbf70 con 0x7ff350160310 2024-09-16T15:50:09.377 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:09.374+0000 7ff355f18640 1 -- 172.21.15.123:0/2405891841 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff3500766e0 con 0x7ff350160310 2024-09-16T15:50:09.378 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:09.378+0000 7ff337fff640 1 -- 172.21.15.123:0/2405891841 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 21) ==== 97641+0+0 (secure 0 0 0) 0x7ff34801a070 con 0x7ff350160310 2024-09-16T15:50:09.378 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:09.378+0000 7ff337fff640 1 --2- 172.21.15.123:0/2405891841 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7ff33c073520 0x7ff33c0759e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:50:09.378 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:09.378+0000 7ff34f7fe640 1 --2- 172.21.15.123:0/2405891841 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7ff33c073520 0x7ff33c0759e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:50:09.378 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:09.378+0000 7ff337fff640 1 -- 172.21.15.123:0/2405891841 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(135..135 src has 1..135) ==== 5737+0+0 (secure 0 0 0) 0x7ff348094030 con 0x7ff350160310 2024-09-16T15:50:09.378 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:09.378+0000 7ff34f7fe640 1 --2- 172.21.15.123:0/2405891841 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7ff33c073520 0x7ff33c0759e0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7ff3400af000 tx=0x7ff340019500 comp rx=0 tx=0).ready entity=mgr.4150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:50:09.380 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:09.382+0000 7ff337fff640 1 -- 172.21.15.123:0/2405891841 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7ff34805e8a0 con 0x7ff350160310 2024-09-16T15:50:09.567 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:09.566+0000 7ff355f18640 1 -- 172.21.15.123:0/2405891841 --> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot create", "vol_name": "non_existent_subvolume", "sub_name": "subvolume_1", "snap_name": "snapshot_1", "target": ["mon-mgr", ""]}) -- 0x7ff35014d390 con 0x7ff33c073520 2024-09-16T15:50:09.569 INFO:tasks.ceph.mgr.y.smithi123.stderr:2024-09-16T15:50:09.570+0000 7fa9b36f4640 -1 mgr.server reply reply (2) No such file or directory FS 'non_existent_subvolume' not found 2024-09-16T15:50:09.569 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:09.570+0000 7ff337fff640 1 -- 172.21.15.123:0/2405891841 <== mgr.4150 v2:172.21.15.123:6836/2464759676 1 ==== mgr_command_reply(tid 0: -2 FS 'non_existent_subvolume' not found) ==== 45+0+0 (secure 0 0 0) 0x7ff35014d390 con 0x7ff33c073520 2024-09-16T15:50:09.569 INFO:teuthology.orchestra.run.smithi123.stderr:Error ENOENT: FS 'non_existent_subvolume' not found 2024-09-16T15:50:09.576 DEBUG:teuthology.orchestra.run:got remote process result: 2 2024-09-16T15:50:09.576 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot rm non_existent_subvolume subvolume_1 snapshot_1 2024-09-16T15:50:09.709 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:09.710+0000 7fb838f4a640 1 -- 172.21.15.123:0/3878511462 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb83415c940 msgr2=0x7fb8340d0590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:50:09.709 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:09.710+0000 7fb838f4a640 1 --2- 172.21.15.123:0/3878511462 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb83415c940 0x7fb8340d0590 secure :-1 s=READY pgs=1462 cs=0 l=1 rev1=1 crypto rx=0x7fb83415ced0 tx=0x7fb82c01cac0 comp rx=0 tx=0).stop 2024-09-16T15:50:09.709 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:09.710+0000 7fb838f4a640 1 -- 172.21.15.123:0/3878511462 shutdown_connections 2024-09-16T15:50:09.709 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:09.710+0000 7fb838f4a640 1 --2- 172.21.15.123:0/3878511462 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb83415c940 0x7fb8340d0590 unknown :-1 s=CLOSED pgs=1462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:09.709 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:09.710+0000 7fb838f4a640 1 -- 172.21.15.123:0/3878511462 >> 172.21.15.123:0/3878511462 conn(0x7fb83409aa40 msgr2=0x7fb83409ae50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:50:09.709 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:09.710+0000 7fb838f4a640 1 -- 172.21.15.123:0/3878511462 shutdown_connections 2024-09-16T15:50:09.709 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:09.710+0000 7fb838f4a640 1 -- 172.21.15.123:0/3878511462 wait complete. 2024-09-16T15:50:09.710 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:09.710+0000 7fb838f4a640 1 Processor -- start 2024-09-16T15:50:09.710 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:09.710+0000 7fb838f4a640 1 -- start start 2024-09-16T15:50:09.712 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:09.710+0000 7fb838f4a640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fb8340b2840 0x7fb834158fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:50:09.712 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:09.710+0000 7fb838f4a640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb8340d21a0 0x7fb8340b7150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:50:09.712 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:09.710+0000 7fb838f4a640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb83415c940 0x7fb834071860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:50:09.712 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:09.710+0000 7fb838f4a640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fb834169f50 con 0x7fb8340d21a0 2024-09-16T15:50:09.712 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:09.710+0000 7fb838f4a640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fb834169dd0 con 0x7fb83415c940 2024-09-16T15:50:09.712 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:09.710+0000 7fb838f4a640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fb83416a0d0 con 0x7fb8340b2840 2024-09-16T15:50:09.712 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:09.710+0000 7fb82bfff640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb8340d21a0 0x7fb8340b7150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:50:09.712 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:09.710+0000 7fb82bfff640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb8340d21a0 0x7fb8340b7150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:53382/0 (socket says 172.21.15.123:53382) 2024-09-16T15:50:09.712 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:09.710+0000 7fb82bfff640 1 -- 172.21.15.123:0/1320323210 learned_addr learned my addr 172.21.15.123:0/1320323210 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:50:09.713 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:09.710+0000 7fb83259c640 1 --2- 172.21.15.123:0/1320323210 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fb8340b2840 0x7fb834158fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:50:09.713 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:09.710+0000 7fb832d9d640 1 --2- 172.21.15.123:0/1320323210 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb83415c940 0x7fb834071860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:50:09.713 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:09.710+0000 7fb82bfff640 1 -- 172.21.15.123:0/1320323210 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fb8340b2840 msgr2=0x7fb834158fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:50:09.713 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:09.710+0000 7fb82bfff640 1 --2- 172.21.15.123:0/1320323210 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fb8340b2840 0x7fb834158fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:09.713 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:09.710+0000 7fb82bfff640 1 -- 172.21.15.123:0/1320323210 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb83415c940 msgr2=0x7fb834071860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:50:09.713 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:09.710+0000 7fb82bfff640 1 --2- 172.21.15.123:0/1320323210 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb83415c940 0x7fb834071860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:09.713 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:09.710+0000 7fb82bfff640 1 -- 172.21.15.123:0/1320323210 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb8340c8740 con 0x7fb8340d21a0 2024-09-16T15:50:09.713 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:09.710+0000 7fb82bfff640 1 --2- 172.21.15.123:0/1320323210 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb8340d21a0 0x7fb8340b7150 secure :-1 s=READY pgs=1463 cs=0 l=1 rev1=1 crypto rx=0x7fb81c00c700 tx=0x7fb81c00cbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:50:09.713 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:09.710+0000 7fb82affd640 1 -- 172.21.15.123:0/1320323210 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb81c016020 con 0x7fb8340d21a0 2024-09-16T15:50:09.713 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:09.710+0000 7fb82affd640 1 -- 172.21.15.123:0/1320323210 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fb81c00a930 con 0x7fb8340d21a0 2024-09-16T15:50:09.713 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:09.710+0000 7fb838f4a640 1 -- 172.21.15.123:0/1320323210 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb8340c7bc0 con 0x7fb8340d21a0 2024-09-16T15:50:09.713 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:09.710+0000 7fb82affd640 1 -- 172.21.15.123:0/1320323210 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb81c00ad60 con 0x7fb8340d21a0 2024-09-16T15:50:09.713 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:09.710+0000 7fb838f4a640 1 -- 172.21.15.123:0/1320323210 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb8340c8460 con 0x7fb8340d21a0 2024-09-16T15:50:09.714 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:09.714+0000 7fb838f4a640 1 -- 172.21.15.123:0/1320323210 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb8340d0590 con 0x7fb8340d21a0 2024-09-16T15:50:09.714 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:09.714+0000 7fb82affd640 1 -- 172.21.15.123:0/1320323210 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 21) ==== 97641+0+0 (secure 0 0 0) 0x7fb81c019020 con 0x7fb8340d21a0 2024-09-16T15:50:09.714 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:09.714+0000 7fb82affd640 1 --2- 172.21.15.123:0/1320323210 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fb8140734f0 0x7fb8140759b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:50:09.714 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:09.714+0000 7fb82affd640 1 -- 172.21.15.123:0/1320323210 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(135..135 src has 1..135) ==== 5737+0+0 (secure 0 0 0) 0x7fb81c094bd0 con 0x7fb8340d21a0 2024-09-16T15:50:09.714 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:09.714+0000 7fb83259c640 1 --2- 172.21.15.123:0/1320323210 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fb8140734f0 0x7fb8140759b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:50:09.714 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:09.714+0000 7fb83259c640 1 --2- 172.21.15.123:0/1320323210 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fb8140734f0 0x7fb8140759b0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7fb8240016f0 tx=0x7fb824009040 comp rx=0 tx=0).ready entity=mgr.4150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:50:09.718 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:09.718+0000 7fb82affd640 1 -- 172.21.15.123:0/1320323210 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7fb81c020020 con 0x7fb8340d21a0 2024-09-16T15:50:09.927 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:09.926+0000 7fb838f4a640 1 -- 172.21.15.123:0/1320323210 --> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot rm", "vol_name": "non_existent_subvolume", "sub_name": "subvolume_1", "snap_name": "snapshot_1", "target": ["mon-mgr", ""]}) -- 0x7fb834074a00 con 0x7fb8140734f0 2024-09-16T15:50:09.930 INFO:tasks.ceph.mgr.y.smithi123.stderr:2024-09-16T15:50:09.930+0000 7fa9b36f4640 -1 mgr.server reply reply (2) No such file or directory FS 'non_existent_subvolume' not found 2024-09-16T15:50:09.930 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:09.930+0000 7fb82affd640 1 -- 172.21.15.123:0/1320323210 <== mgr.4150 v2:172.21.15.123:6836/2464759676 1 ==== mgr_command_reply(tid 0: -2 FS 'non_existent_subvolume' not found) ==== 45+0+0 (secure 0 0 0) 0x7fb834074a00 con 0x7fb8140734f0 2024-09-16T15:50:09.930 INFO:teuthology.orchestra.run.smithi123.stderr:Error ENOENT: FS 'non_existent_subvolume' not found 2024-09-16T15:50:09.936 DEBUG:teuthology.orchestra.run:got remote process result: 2 2024-09-16T15:50:09.936 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot info non_existent_subvolume subvolume_1 snapshot_1 2024-09-16T15:50:10.005 DEBUG:teuthology.orchestra.run.smithi123:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-16T15:50:10.007 DEBUG:teuthology.orchestra.run.smithi184:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-16T15:50:10.066 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:10.066+0000 7f16cc916640 1 -- 172.21.15.123:0/2078459566 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f16b00052f0 msgr2=0x7f16b000b9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:50:10.067 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:10.066+0000 7f16cc916640 1 --2- 172.21.15.123:0/2078459566 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f16b00052f0 0x7f16b000b9d0 secure :-1 s=READY pgs=1465 cs=0 l=1 rev1=1 crypto rx=0x7f16c0006fd0 tx=0x7f16c001caf0 comp rx=0 tx=0).stop 2024-09-16T15:50:10.067 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:10.066+0000 7f16cc916640 1 -- 172.21.15.123:0/2078459566 shutdown_connections 2024-09-16T15:50:10.067 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:10.066+0000 7f16cc916640 1 --2- 172.21.15.123:0/2078459566 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f16b00052f0 0x7f16b000b9d0 unknown :-1 s=CLOSED pgs=1465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:10.067 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:10.066+0000 7f16cc916640 1 -- 172.21.15.123:0/2078459566 >> 172.21.15.123:0/2078459566 conn(0x7f16c80d0b40 msgr2=0x7f16c80d0f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:50:10.067 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:10.066+0000 7f16cc916640 1 -- 172.21.15.123:0/2078459566 shutdown_connections 2024-09-16T15:50:10.067 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:10.066+0000 7f16cc916640 1 -- 172.21.15.123:0/2078459566 wait complete. 2024-09-16T15:50:10.067 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:10.066+0000 7f16cc916640 1 Processor -- start 2024-09-16T15:50:10.068 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:10.066+0000 7f16cc916640 1 -- start start 2024-09-16T15:50:10.069 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:10.070+0000 7f16cc916640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f16b00052f0 0x7f16c814e7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:50:10.069 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:10.070+0000 7f16cc916640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f16c81473b0 0x7f16c813e390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:50:10.069 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:10.070+0000 7f16cc916640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f16c80b0620 0x7f16c80b0ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:50:10.070 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:10.070+0000 7f16cc916640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f16c815b3d0 con 0x7f16c81473b0 2024-09-16T15:50:10.070 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:10.070+0000 7f16cc916640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f16c815b250 con 0x7f16b00052f0 2024-09-16T15:50:10.070 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:10.070+0000 7f16cc916640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f16c815b550 con 0x7f16c80b0620 2024-09-16T15:50:10.070 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:10.070+0000 7f16c5d9b640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f16c81473b0 0x7f16c813e390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:50:10.070 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:10.070+0000 7f16c5d9b640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f16c81473b0 0x7f16c813e390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:53396/0 (socket says 172.21.15.123:53396) 2024-09-16T15:50:10.070 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:10.070+0000 7f16c5d9b640 1 -- 172.21.15.123:0/1605333521 learned_addr learned my addr 172.21.15.123:0/1605333521 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:50:10.070 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:10.070+0000 7f16c659c640 1 --2- 172.21.15.123:0/1605333521 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f16b00052f0 0x7f16c814e7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:50:10.070 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:10.070+0000 7f16c6d9d640 1 --2- 172.21.15.123:0/1605333521 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f16c80b0620 0x7f16c80b0ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:50:10.070 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:10.070+0000 7f16c5d9b640 1 -- 172.21.15.123:0/1605333521 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f16c80b0620 msgr2=0x7f16c80b0ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:50:10.070 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:10.070+0000 7f16c5d9b640 1 --2- 172.21.15.123:0/1605333521 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f16c80b0620 0x7f16c80b0ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:10.070 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:10.070+0000 7f16c5d9b640 1 -- 172.21.15.123:0/1605333521 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f16b00052f0 msgr2=0x7f16c814e7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:50:10.070 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:10.070+0000 7f16c5d9b640 1 --2- 172.21.15.123:0/1605333521 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f16b00052f0 0x7f16c814e7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:10.071 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:10.070+0000 7f16c5d9b640 1 -- 172.21.15.123:0/1605333521 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f16c8153060 con 0x7f16c81473b0 2024-09-16T15:50:10.071 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:10.070+0000 7f16c5d9b640 1 --2- 172.21.15.123:0/1605333521 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f16c81473b0 0x7f16c813e390 secure :-1 s=READY pgs=1466 cs=0 l=1 rev1=1 crypto rx=0x7f16bc000fc0 tx=0x7f16bc00ea20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:50:10.071 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:10.070+0000 7f16b6ffd640 1 -- 172.21.15.123:0/1605333521 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16bc015020 con 0x7f16c81473b0 2024-09-16T15:50:10.071 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:10.070+0000 7f16b6ffd640 1 -- 172.21.15.123:0/1605333521 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f16bc010980 con 0x7f16c81473b0 2024-09-16T15:50:10.071 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:10.070+0000 7f16b6ffd640 1 -- 172.21.15.123:0/1605333521 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16bc010dd0 con 0x7f16c81473b0 2024-09-16T15:50:10.071 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:10.070+0000 7f16cc916640 1 -- 172.21.15.123:0/1605333521 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f16c8151fc0 con 0x7f16c81473b0 2024-09-16T15:50:10.071 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:10.070+0000 7f16cc916640 1 -- 172.21.15.123:0/1605333521 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f16c8152580 con 0x7f16c81473b0 2024-09-16T15:50:10.072 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:10.070+0000 7f16cc916640 1 -- 172.21.15.123:0/1605333521 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f16c80b4630 con 0x7f16c81473b0 2024-09-16T15:50:10.072 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:10.070+0000 7f16b6ffd640 1 -- 172.21.15.123:0/1605333521 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 21) ==== 97641+0+0 (secure 0 0 0) 0x7f16bc018020 con 0x7f16c81473b0 2024-09-16T15:50:10.072 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:10.070+0000 7f16b6ffd640 1 --2- 172.21.15.123:0/1605333521 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f16a8073520 0x7f16a80759e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:50:10.072 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:10.070+0000 7f16c659c640 1 --2- 172.21.15.123:0/1605333521 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f16a8073520 0x7f16a80759e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:50:10.072 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:10.070+0000 7f16b6ffd640 1 -- 172.21.15.123:0/1605333521 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(135..135 src has 1..135) ==== 5737+0+0 (secure 0 0 0) 0x7f16bc094fd0 con 0x7f16c81473b0 2024-09-16T15:50:10.072 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:10.070+0000 7f16c659c640 1 --2- 172.21.15.123:0/1605333521 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f16a8073520 0x7f16a80759e0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f16b8005fd0 tx=0x7f16b800b260 comp rx=0 tx=0).ready entity=mgr.4150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:50:10.075 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:10.074+0000 7f16b6ffd640 1 -- 172.21.15.123:0/1605333521 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f16bc01f020 con 0x7f16c81473b0 2024-09-16T15:50:10.280 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:10.278+0000 7f16cc916640 1 -- 172.21.15.123:0/1605333521 --> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot info", "vol_name": "non_existent_subvolume", "sub_name": "subvolume_1", "snap_name": "snapshot_1", "target": ["mon-mgr", ""]}) -- 0x7f16c815d730 con 0x7f16a8073520 2024-09-16T15:50:10.281 INFO:tasks.ceph.mgr.y.smithi123.stderr:2024-09-16T15:50:10.282+0000 7fa9b36f4640 -1 mgr.server reply reply (2) No such file or directory FS 'non_existent_subvolume' not found 2024-09-16T15:50:10.282 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:10.282+0000 7f16b6ffd640 1 -- 172.21.15.123:0/1605333521 <== mgr.4150 v2:172.21.15.123:6836/2464759676 1 ==== mgr_command_reply(tid 0: -2 FS 'non_existent_subvolume' not found) ==== 45+0+0 (secure 0 0 0) 0x7f16c815d730 con 0x7f16a8073520 2024-09-16T15:50:10.282 INFO:teuthology.orchestra.run.smithi123.stderr:Error ENOENT: FS 'non_existent_subvolume' not found 2024-09-16T15:50:10.289 DEBUG:teuthology.orchestra.run:got remote process result: 2 2024-09-16T15:50:10.290 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot protect non_existent_subvolume subvolume_1 snapshot_1 2024-09-16T15:50:10.461 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:10.462+0000 7fc0cd4c1640 1 -- 172.21.15.123:0/279525050 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc0c80d1a90 msgr2=0x7fc0c80d1f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:50:10.461 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:10.462+0000 7fc0cd4c1640 1 --2- 172.21.15.123:0/279525050 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc0c80d1a90 0x7fc0c80d1f00 secure :-1 s=READY pgs=1468 cs=0 l=1 rev1=1 crypto rx=0x7fc0c80cf690 tx=0x7fc0bc01c800 comp rx=0 tx=0).stop 2024-09-16T15:50:10.461 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:10.462+0000 7fc0cd4c1640 1 -- 172.21.15.123:0/279525050 shutdown_connections 2024-09-16T15:50:10.461 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:10.462+0000 7fc0cd4c1640 1 --2- 172.21.15.123:0/279525050 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc0c80d1a90 0x7fc0c80d1f00 unknown :-1 s=CLOSED pgs=1468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:10.461 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:10.462+0000 7fc0cd4c1640 1 --2- 172.21.15.123:0/279525050 >> v2:172.21.15.184:3300/0 conn(0x7fc0c80b2840 0x7fc0c80d0590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:10.461 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:10.462+0000 7fc0cd4c1640 1 -- 172.21.15.123:0/279525050 >> 172.21.15.123:0/279525050 conn(0x7fc0c809aa40 msgr2=0x7fc0c809ae50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:50:10.461 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:10.462+0000 7fc0cd4c1640 1 -- 172.21.15.123:0/279525050 shutdown_connections 2024-09-16T15:50:10.461 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:10.462+0000 7fc0cd4c1640 1 -- 172.21.15.123:0/279525050 wait complete. 2024-09-16T15:50:10.464 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:10.462+0000 7fc0cd4c1640 1 Processor -- start 2024-09-16T15:50:10.464 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:10.462+0000 7fc0cd4c1640 1 -- start start 2024-09-16T15:50:10.464 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:10.462+0000 7fc0cd4c1640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fc0c80b2840 0x7fc0c81558a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:50:10.464 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:10.462+0000 7fc0cd4c1640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc0c80d1a90 0x7fc0c8156970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:50:10.464 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:10.462+0000 7fc0cd4c1640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc0c815c230 0x7fc0c80b5e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:50:10.464 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:10.462+0000 7fc0cd4c1640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fc0c8168e00 con 0x7fc0c815c230 2024-09-16T15:50:10.464 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:10.462+0000 7fc0cd4c1640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fc0c8168c80 con 0x7fc0c80d1a90 2024-09-16T15:50:10.464 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:10.462+0000 7fc0cd4c1640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fc0c8168f80 con 0x7fc0c80b2840 2024-09-16T15:50:10.464 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:10.462+0000 7fc0c77fe640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc0c815c230 0x7fc0c80b5e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:50:10.464 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:10.462+0000 7fc0c77fe640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc0c815c230 0x7fc0c80b5e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:53418/0 (socket says 172.21.15.123:53418) 2024-09-16T15:50:10.465 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:10.462+0000 7fc0c77fe640 1 -- 172.21.15.123:0/997209726 learned_addr learned my addr 172.21.15.123:0/997209726 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:50:10.465 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:10.462+0000 7fc0c67fc640 1 --2- 172.21.15.123:0/997209726 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc0c80d1a90 0x7fc0c8156970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:50:10.465 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:10.462+0000 7fc0c6ffd640 1 --2- 172.21.15.123:0/997209726 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fc0c80b2840 0x7fc0c81558a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:50:10.465 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:10.462+0000 7fc0c77fe640 1 -- 172.21.15.123:0/997209726 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fc0c80b2840 msgr2=0x7fc0c81558a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:50:10.465 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:10.462+0000 7fc0c77fe640 1 --2- 172.21.15.123:0/997209726 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fc0c80b2840 0x7fc0c81558a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:10.465 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:10.462+0000 7fc0c77fe640 1 -- 172.21.15.123:0/997209726 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc0c80d1a90 msgr2=0x7fc0c8156970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:50:10.465 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:10.462+0000 7fc0c77fe640 1 --2- 172.21.15.123:0/997209726 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc0c80d1a90 0x7fc0c8156970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:10.465 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:10.462+0000 7fc0c77fe640 1 -- 172.21.15.123:0/997209726 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc0c81571c0 con 0x7fc0c815c230 2024-09-16T15:50:10.465 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:10.462+0000 7fc0c67fc640 1 --2- 172.21.15.123:0/997209726 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc0c80d1a90 0x7fc0c8156970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T15:50:10.465 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:10.462+0000 7fc0c77fe640 1 --2- 172.21.15.123:0/997209726 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc0c815c230 0x7fc0c80b5e30 secure :-1 s=READY pgs=1469 cs=0 l=1 rev1=1 crypto rx=0x7fc0c000cb10 tx=0x7fc0c000cc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:50:10.465 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:10.462+0000 7fc0af7fe640 1 -- 172.21.15.123:0/997209726 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0c0016020 con 0x7fc0c815c230 2024-09-16T15:50:10.465 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:10.462+0000 7fc0af7fe640 1 -- 172.21.15.123:0/997209726 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fc0c000ce50 con 0x7fc0c815c230 2024-09-16T15:50:10.466 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:10.462+0000 7fc0cd4c1640 1 -- 172.21.15.123:0/997209726 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc0c80c9390 con 0x7fc0c815c230 2024-09-16T15:50:10.466 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:10.462+0000 7fc0af7fe640 1 -- 172.21.15.123:0/997209726 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0c000ae00 con 0x7fc0c815c230 2024-09-16T15:50:10.466 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:10.462+0000 7fc0cd4c1640 1 -- 172.21.15.123:0/997209726 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc0c8156eb0 con 0x7fc0c815c230 2024-09-16T15:50:10.466 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:10.466+0000 7fc0cd4c1640 1 -- 172.21.15.123:0/997209726 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc0c80d0590 con 0x7fc0c815c230 2024-09-16T15:50:10.467 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:10.466+0000 7fc0af7fe640 1 -- 172.21.15.123:0/997209726 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 21) ==== 97641+0+0 (secure 0 0 0) 0x7fc0c0019020 con 0x7fc0c815c230 2024-09-16T15:50:10.467 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:10.466+0000 7fc0af7fe640 1 --2- 172.21.15.123:0/997209726 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fc0b40734f0 0x7fc0b40759b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:50:10.467 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:10.466+0000 7fc0af7fe640 1 -- 172.21.15.123:0/997209726 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(135..135 src has 1..135) ==== 5737+0+0 (secure 0 0 0) 0x7fc0c0094d90 con 0x7fc0c815c230 2024-09-16T15:50:10.467 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:10.466+0000 7fc0c6ffd640 1 --2- 172.21.15.123:0/997209726 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fc0b40734f0 0x7fc0b40759b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:50:10.467 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:10.466+0000 7fc0c6ffd640 1 --2- 172.21.15.123:0/997209726 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fc0b40734f0 0x7fc0b40759b0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7fc0b80058d0 tx=0x7fc0b8005860 comp rx=0 tx=0).ready entity=mgr.4150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:50:10.470 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:10.470+0000 7fc0af7fe640 1 -- 172.21.15.123:0/997209726 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7fc0c0020020 con 0x7fc0c815c230 2024-09-16T15:50:10.661 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:10.662+0000 7fc0cd4c1640 1 -- 172.21.15.123:0/997209726 --> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot protect", "vol_name": "non_existent_subvolume", "sub_name": "subvolume_1", "snap_name": "snapshot_1", "target": ["mon-mgr", ""]}) -- 0x7fc0c8074a00 con 0x7fc0b40734f0 2024-09-16T15:50:10.663 INFO:tasks.ceph.mgr.y.smithi123.stderr:2024-09-16T15:50:10.662+0000 7fa9b36f4640 -1 mgr.server reply reply (2) No such file or directory FS 'non_existent_subvolume' not found 2024-09-16T15:50:10.664 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:10.662+0000 7fc0af7fe640 1 -- 172.21.15.123:0/997209726 <== mgr.4150 v2:172.21.15.123:6836/2464759676 1 ==== mgr_command_reply(tid 0: -2 FS 'non_existent_subvolume' not found) ==== 45+0+0 (secure 0 0 0) 0x7fc0c8074a00 con 0x7fc0b40734f0 2024-09-16T15:50:10.664 INFO:teuthology.orchestra.run.smithi123.stderr:Error ENOENT: FS 'non_existent_subvolume' not found 2024-09-16T15:50:10.669 DEBUG:teuthology.orchestra.run:got remote process result: 2 2024-09-16T15:50:10.670 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot unprotect non_existent_subvolume subvolume_1 snapshot_1 2024-09-16T15:50:10.803 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:10.802+0000 7fd0660d6640 1 -- 172.21.15.123:0/3181079839 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd06015bc40 msgr2=0x7fd060070470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:50:10.803 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:10.802+0000 7fd0660d6640 1 --2- 172.21.15.123:0/3181079839 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd06015bc40 0x7fd060070470 secure :-1 s=READY pgs=1470 cs=0 l=1 rev1=1 crypto rx=0x7fd058009c80 tx=0x7fd05801cce0 comp rx=0 tx=0).stop 2024-09-16T15:50:10.803 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:10.802+0000 7fd0660d6640 1 -- 172.21.15.123:0/3181079839 shutdown_connections 2024-09-16T15:50:10.804 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:10.802+0000 7fd0660d6640 1 --2- 172.21.15.123:0/3181079839 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fd060160310 0x7fd0600725e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:10.804 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:10.802+0000 7fd0660d6640 1 --2- 172.21.15.123:0/3181079839 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd06015bc40 0x7fd060070470 unknown :-1 s=CLOSED pgs=1470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:10.804 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:10.802+0000 7fd0660d6640 1 -- 172.21.15.123:0/3181079839 >> 172.21.15.123:0/3181079839 conn(0x7fd0600b1630 msgr2=0x7fd0600b1a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:50:10.804 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:10.802+0000 7fd0660d6640 1 -- 172.21.15.123:0/3181079839 shutdown_connections 2024-09-16T15:50:10.804 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:10.802+0000 7fd0660d6640 1 -- 172.21.15.123:0/3181079839 wait complete. 2024-09-16T15:50:10.804 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:10.806+0000 7fd0660d6640 1 Processor -- start 2024-09-16T15:50:10.804 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:10.806+0000 7fd0660d6640 1 -- start start 2024-09-16T15:50:10.806 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:10.806+0000 7fd0660d6640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd06015bc40 0x7fd060149ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:50:10.807 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:10.806+0000 7fd0660d6640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fd06015f690 0x7fd06014ece0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:50:10.807 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:10.806+0000 7fd0660d6640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd060160310 0x7fd06014cbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:50:10.807 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:10.806+0000 7fd0660d6640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fd0601693c0 con 0x7fd060160310 2024-09-16T15:50:10.807 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:10.806+0000 7fd0660d6640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fd060169240 con 0x7fd06015bc40 2024-09-16T15:50:10.807 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:10.806+0000 7fd0660d6640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fd060169540 con 0x7fd06015f690 2024-09-16T15:50:10.807 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:10.806+0000 7fd05ffff640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd060160310 0x7fd06014cbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:50:10.807 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:10.806+0000 7fd05ffff640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd060160310 0x7fd06014cbd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:53428/0 (socket says 172.21.15.123:53428) 2024-09-16T15:50:10.807 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:10.806+0000 7fd05ffff640 1 -- 172.21.15.123:0/878113504 learned_addr learned my addr 172.21.15.123:0/878113504 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:50:10.807 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:10.806+0000 7fd05effd640 1 --2- 172.21.15.123:0/878113504 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fd06015f690 0x7fd06014ece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:50:10.807 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:10.806+0000 7fd05f7fe640 1 --2- 172.21.15.123:0/878113504 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd06015bc40 0x7fd060149ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:50:10.807 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:10.806+0000 7fd05ffff640 1 -- 172.21.15.123:0/878113504 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fd06015f690 msgr2=0x7fd06014ece0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:50:10.807 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:10.806+0000 7fd05ffff640 1 --2- 172.21.15.123:0/878113504 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fd06015f690 0x7fd06014ece0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:10.807 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:10.806+0000 7fd05ffff640 1 -- 172.21.15.123:0/878113504 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd06015bc40 msgr2=0x7fd060149ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:50:10.808 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:10.806+0000 7fd05ffff640 1 --2- 172.21.15.123:0/878113504 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd06015bc40 0x7fd060149ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:10.808 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:10.806+0000 7fd05ffff640 1 -- 172.21.15.123:0/878113504 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd0600b08b0 con 0x7fd060160310 2024-09-16T15:50:10.808 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:10.806+0000 7fd05effd640 1 --2- 172.21.15.123:0/878113504 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fd06015f690 0x7fd06014ece0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T15:50:10.808 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:10.806+0000 7fd05f7fe640 1 --2- 172.21.15.123:0/878113504 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd06015bc40 0x7fd060149ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T15:50:10.808 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:10.806+0000 7fd05ffff640 1 --2- 172.21.15.123:0/878113504 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd060160310 0x7fd06014cbd0 secure :-1 s=READY pgs=1471 cs=0 l=1 rev1=1 crypto rx=0x7fd05000c3c0 tx=0x7fd05000c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:50:10.808 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:10.806+0000 7fd057fff640 1 -- 172.21.15.123:0/878113504 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd050011790 con 0x7fd060160310 2024-09-16T15:50:10.808 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:10.806+0000 7fd057fff640 1 -- 172.21.15.123:0/878113504 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fd050017070 con 0x7fd060160310 2024-09-16T15:50:10.808 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:10.806+0000 7fd0660d6640 1 -- 172.21.15.123:0/878113504 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd0600b0b90 con 0x7fd060160310 2024-09-16T15:50:10.808 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:10.806+0000 7fd0660d6640 1 -- 172.21.15.123:0/878113504 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd06014b050 con 0x7fd060160310 2024-09-16T15:50:10.808 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:10.806+0000 7fd057fff640 1 -- 172.21.15.123:0/878113504 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd050011cc0 con 0x7fd060160310 2024-09-16T15:50:10.809 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:10.806+0000 7fd0660d6640 1 -- 172.21.15.123:0/878113504 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd0440052a0 con 0x7fd060160310 2024-09-16T15:50:10.809 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:10.806+0000 7fd057fff640 1 -- 172.21.15.123:0/878113504 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 21) ==== 97641+0+0 (secure 0 0 0) 0x7fd05001c460 con 0x7fd060160310 2024-09-16T15:50:10.809 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:10.810+0000 7fd057fff640 1 --2- 172.21.15.123:0/878113504 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fd0400734f0 0x7fd0400759b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:50:10.809 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:10.810+0000 7fd05f7fe640 1 --2- 172.21.15.123:0/878113504 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fd0400734f0 0x7fd0400759b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:50:10.809 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:10.810+0000 7fd057fff640 1 -- 172.21.15.123:0/878113504 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(135..135 src has 1..135) ==== 5737+0+0 (secure 0 0 0) 0x7fd050094620 con 0x7fd060160310 2024-09-16T15:50:10.810 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:10.810+0000 7fd05f7fe640 1 --2- 172.21.15.123:0/878113504 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fd0400734f0 0x7fd0400759b0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7fd0580af000 tx=0x7fd05801b520 comp rx=0 tx=0).ready entity=mgr.4150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:50:10.812 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:10.814+0000 7fd057fff640 1 -- 172.21.15.123:0/878113504 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7fd05005ef90 con 0x7fd060160310 2024-09-16T15:50:11.007 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:11.006+0000 7fd0660d6640 1 -- 172.21.15.123:0/878113504 --> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot unprotect", "vol_name": "non_existent_subvolume", "sub_name": "subvolume_1", "snap_name": "snapshot_1", "target": ["mon-mgr", ""]}) -- 0x7fd044000c80 con 0x7fd0400734f0 2024-09-16T15:50:11.009 INFO:tasks.ceph.mgr.y.smithi123.stderr:2024-09-16T15:50:11.010+0000 7fa9b36f4640 -1 mgr.server reply reply (2) No such file or directory FS 'non_existent_subvolume' not found 2024-09-16T15:50:11.010 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:11.010+0000 7fd057fff640 1 -- 172.21.15.123:0/878113504 <== mgr.4150 v2:172.21.15.123:6836/2464759676 1 ==== mgr_command_reply(tid 0: -2 FS 'non_existent_subvolume' not found) ==== 45+0+0 (secure 0 0 0) 0x7fd044000c80 con 0x7fd0400734f0 2024-09-16T15:50:11.010 INFO:teuthology.orchestra.run.smithi123.stderr:Error ENOENT: FS 'non_existent_subvolume' not found 2024-09-16T15:50:11.016 DEBUG:teuthology.orchestra.run:got remote process result: 2 2024-09-16T15:50:11.016 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot ls non_existent_subvolume subvolume_1 2024-09-16T15:50:11.147 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:11.146+0000 7fd74f60c640 1 -- 172.21.15.123:0/375758043 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd748160310 msgr2=0x7fd7480725e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:50:11.147 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:11.146+0000 7fd74f60c640 1 --2- 172.21.15.123:0/375758043 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd748160310 0x7fd7480725e0 secure :-1 s=READY pgs=1472 cs=0 l=1 rev1=1 crypto rx=0x7fd748051540 tx=0x7fd73801c800 comp rx=0 tx=0).stop 2024-09-16T15:50:11.147 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:11.146+0000 7fd74f60c640 1 -- 172.21.15.123:0/375758043 shutdown_connections 2024-09-16T15:50:11.147 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:11.146+0000 7fd74f60c640 1 --2- 172.21.15.123:0/375758043 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fd74815f690 0x7fd74815faf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:11.147 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:11.146+0000 7fd74f60c640 1 --2- 172.21.15.123:0/375758043 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd748160310 0x7fd7480725e0 unknown :-1 s=CLOSED pgs=1472 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:11.147 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:11.146+0000 7fd74f60c640 1 --2- 172.21.15.123:0/375758043 >> v2:172.21.15.184:3300/0 conn(0x7fd74815bc40 0x7fd748070470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:11.147 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:11.146+0000 7fd74f60c640 1 -- 172.21.15.123:0/375758043 >> 172.21.15.123:0/375758043 conn(0x7fd7480b1630 msgr2=0x7fd7480b1a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:50:11.147 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:11.146+0000 7fd74f60c640 1 -- 172.21.15.123:0/375758043 shutdown_connections 2024-09-16T15:50:11.147 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:11.146+0000 7fd74f60c640 1 -- 172.21.15.123:0/375758043 wait complete. 2024-09-16T15:50:11.148 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:11.146+0000 7fd74f60c640 1 Processor -- start 2024-09-16T15:50:11.148 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:11.146+0000 7fd74f60c640 1 -- start start 2024-09-16T15:50:11.150 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:11.150+0000 7fd74f60c640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fd74815bc40 0x7fd7481340a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:50:11.150 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:11.150+0000 7fd74f60c640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd74815f690 0x7fd74815ab00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:50:11.150 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:11.150+0000 7fd74f60c640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd748160310 0x7fd7481508d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:50:11.150 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:11.150+0000 7fd74f60c640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fd7480cf460 con 0x7fd74815f690 2024-09-16T15:50:11.150 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:11.150+0000 7fd74f60c640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fd7480cf2e0 con 0x7fd748160310 2024-09-16T15:50:11.150 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:11.150+0000 7fd74f60c640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fd7480cb9a0 con 0x7fd74815bc40 2024-09-16T15:50:11.150 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:11.150+0000 7fd74cba7640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd74815f690 0x7fd74815ab00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:50:11.150 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:11.150+0000 7fd74cba7640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd74815f690 0x7fd74815ab00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:53444/0 (socket says 172.21.15.123:53444) 2024-09-16T15:50:11.150 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:11.150+0000 7fd74cba7640 1 -- 172.21.15.123:0/3371368169 learned_addr learned my addr 172.21.15.123:0/3371368169 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:50:11.150 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:11.150+0000 7fd74dba9640 1 --2- 172.21.15.123:0/3371368169 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd748160310 0x7fd7481508d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:50:11.150 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:11.150+0000 7fd74cba7640 1 -- 172.21.15.123:0/3371368169 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fd74815bc40 msgr2=0x7fd7481340a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:50:11.150 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:11.150+0000 7fd74cba7640 1 --2- 172.21.15.123:0/3371368169 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fd74815bc40 0x7fd7481340a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:11.151 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:11.150+0000 7fd74cba7640 1 -- 172.21.15.123:0/3371368169 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd748160310 msgr2=0x7fd7481508d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:50:11.151 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:11.150+0000 7fd74cba7640 1 --2- 172.21.15.123:0/3371368169 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd748160310 0x7fd7481508d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:11.151 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:11.150+0000 7fd74cba7640 1 -- 172.21.15.123:0/3371368169 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd74814c1f0 con 0x7fd74815f690 2024-09-16T15:50:11.151 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:11.150+0000 7fd74dba9640 1 --2- 172.21.15.123:0/3371368169 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd748160310 0x7fd7481508d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T15:50:11.151 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:11.150+0000 7fd74cba7640 1 --2- 172.21.15.123:0/3371368169 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd74815f690 0x7fd74815ab00 secure :-1 s=READY pgs=1473 cs=0 l=1 rev1=1 crypto rx=0x7fd7380b0040 tx=0x7fd7380a5790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:50:11.151 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:11.150+0000 7fd73dffb640 1 -- 172.21.15.123:0/3371368169 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7380b2040 con 0x7fd74815f690 2024-09-16T15:50:11.151 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:11.150+0000 7fd73dffb640 1 -- 172.21.15.123:0/3371368169 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fd738004430 con 0x7fd74815f690 2024-09-16T15:50:11.151 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:11.150+0000 7fd74f60c640 1 -- 172.21.15.123:0/3371368169 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd748149250 con 0x7fd74815f690 2024-09-16T15:50:11.151 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:11.150+0000 7fd73dffb640 1 -- 172.21.15.123:0/3371368169 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd738005280 con 0x7fd74815f690 2024-09-16T15:50:11.151 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:11.150+0000 7fd74f60c640 1 -- 172.21.15.123:0/3371368169 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd748149810 con 0x7fd74815f690 2024-09-16T15:50:11.152 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:11.150+0000 7fd74f60c640 1 -- 172.21.15.123:0/3371368169 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd74815faf0 con 0x7fd74815f690 2024-09-16T15:50:11.152 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:11.150+0000 7fd73dffb640 1 -- 172.21.15.123:0/3371368169 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 21) ==== 97641+0+0 (secure 0 0 0) 0x7fd738005420 con 0x7fd74815f690 2024-09-16T15:50:11.152 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:11.150+0000 7fd73dffb640 1 --2- 172.21.15.123:0/3371368169 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fd72c0734f0 0x7fd72c0759b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:50:11.152 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:11.150+0000 7fd74d3a8640 1 --2- 172.21.15.123:0/3371368169 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fd72c0734f0 0x7fd72c0759b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:50:11.152 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:11.150+0000 7fd73dffb640 1 -- 172.21.15.123:0/3371368169 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(135..135 src has 1..135) ==== 5737+0+0 (secure 0 0 0) 0x7fd73812ea60 con 0x7fd74815f690 2024-09-16T15:50:11.152 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:11.154+0000 7fd74d3a8640 1 --2- 172.21.15.123:0/3371368169 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fd72c0734f0 0x7fd72c0759b0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7fd744005d90 tx=0x7fd744005ca0 comp rx=0 tx=0).ready entity=mgr.4150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:50:11.156 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:11.158+0000 7fd73dffb640 1 -- 172.21.15.123:0/3371368169 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7fd7380164b0 con 0x7fd74815f690 2024-09-16T15:50:11.345 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:11.346+0000 7fd74f60c640 1 -- 172.21.15.123:0/3371368169 --> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot ls", "vol_name": "non_existent_subvolume", "sub_name": "subvolume_1", "target": ["mon-mgr", ""]}) -- 0x7fd748074220 con 0x7fd72c0734f0 2024-09-16T15:50:11.347 INFO:tasks.ceph.mgr.y.smithi123.stderr:2024-09-16T15:50:11.346+0000 7fa9b36f4640 -1 mgr.server reply reply (2) No such file or directory FS 'non_existent_subvolume' not found 2024-09-16T15:50:11.348 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:11.346+0000 7fd73dffb640 1 -- 172.21.15.123:0/3371368169 <== mgr.4150 v2:172.21.15.123:6836/2464759676 1 ==== mgr_command_reply(tid 0: -2 FS 'non_existent_subvolume' not found) ==== 45+0+0 (secure 0 0 0) 0x7fd748074220 con 0x7fd72c0734f0 2024-09-16T15:50:11.348 INFO:teuthology.orchestra.run.smithi123.stderr:Error ENOENT: FS 'non_existent_subvolume' not found 2024-09-16T15:50:11.354 DEBUG:teuthology.orchestra.run:got remote process result: 2 2024-09-16T15:50:11.354 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot clone non_existent_subvolume subvolume_1 snapshot_1 clone_1 2024-09-16T15:50:11.492 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:11.494+0000 7fb9ba953640 1 -- 172.21.15.123:0/1458635783 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb9a00052f0 msgr2=0x7fb9a000b9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:50:11.493 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:11.494+0000 7fb9ba953640 1 --2- 172.21.15.123:0/1458635783 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb9a00052f0 0x7fb9a000b9d0 secure :-1 s=READY pgs=1475 cs=0 l=1 rev1=1 crypto rx=0x7fb9ac006fd0 tx=0x7fb9ac01ca90 comp rx=0 tx=0).stop 2024-09-16T15:50:11.493 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:11.494+0000 7fb9ba953640 1 -- 172.21.15.123:0/1458635783 shutdown_connections 2024-09-16T15:50:11.493 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:11.494+0000 7fb9ba953640 1 --2- 172.21.15.123:0/1458635783 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb9a00052f0 0x7fb9a000b9d0 unknown :-1 s=CLOSED pgs=1475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:11.493 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:11.494+0000 7fb9ba953640 1 -- 172.21.15.123:0/1458635783 >> 172.21.15.123:0/1458635783 conn(0x7fb9b40b1710 msgr2=0x7fb9b40b1b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:50:11.493 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:11.494+0000 7fb9ba953640 1 -- 172.21.15.123:0/1458635783 shutdown_connections 2024-09-16T15:50:11.493 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:11.494+0000 7fb9ba953640 1 -- 172.21.15.123:0/1458635783 wait complete. 2024-09-16T15:50:11.493 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:11.494+0000 7fb9ba953640 1 Processor -- start 2024-09-16T15:50:11.494 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:11.494+0000 7fb9ba953640 1 -- start start 2024-09-16T15:50:11.495 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:11.494+0000 7fb9ba953640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fb9a00052f0 0x7fb9b41f6500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:50:11.496 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:11.494+0000 7fb9ba953640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb9b40c8030 0x7fb9b40c9410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:50:11.496 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:11.494+0000 7fb9ba953640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb9b41f1920 0x7fb9b41f1da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:50:11.496 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:11.494+0000 7fb9ba953640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fb9b40730b0 con 0x7fb9b41f1920 2024-09-16T15:50:11.496 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:11.494+0000 7fb9ba953640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fb9b4072f30 con 0x7fb9b40c8030 2024-09-16T15:50:11.496 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:11.494+0000 7fb9ba953640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fb9b4073230 con 0x7fb9a00052f0 2024-09-16T15:50:11.496 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:11.494+0000 7fb9b8ef0640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb9b41f1920 0x7fb9b41f1da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:50:11.496 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:11.494+0000 7fb9b8ef0640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb9b41f1920 0x7fb9b41f1da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:53466/0 (socket says 172.21.15.123:53466) 2024-09-16T15:50:11.496 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:11.494+0000 7fb9b8ef0640 1 -- 172.21.15.123:0/3247217763 learned_addr learned my addr 172.21.15.123:0/3247217763 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:50:11.496 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:11.494+0000 7fb9b3fff640 1 --2- 172.21.15.123:0/3247217763 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fb9a00052f0 0x7fb9b41f6500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:50:11.496 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:11.494+0000 7fb9abfff640 1 --2- 172.21.15.123:0/3247217763 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb9b40c8030 0x7fb9b40c9410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:50:11.496 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:11.494+0000 7fb9b8ef0640 1 -- 172.21.15.123:0/3247217763 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fb9a00052f0 msgr2=0x7fb9b41f6500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:50:11.496 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:11.494+0000 7fb9b8ef0640 1 --2- 172.21.15.123:0/3247217763 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fb9a00052f0 0x7fb9b41f6500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:11.497 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:11.494+0000 7fb9b8ef0640 1 -- 172.21.15.123:0/3247217763 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb9b40c8030 msgr2=0x7fb9b40c9410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:50:11.497 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:11.494+0000 7fb9b8ef0640 1 --2- 172.21.15.123:0/3247217763 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb9b40c8030 0x7fb9b40c9410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:11.497 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:11.494+0000 7fb9b8ef0640 1 -- 172.21.15.123:0/3247217763 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb9b41f6a40 con 0x7fb9b41f1920 2024-09-16T15:50:11.497 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:11.494+0000 7fb9abfff640 1 --2- 172.21.15.123:0/3247217763 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb9b40c8030 0x7fb9b40c9410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T15:50:11.497 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:11.494+0000 7fb9b8ef0640 1 --2- 172.21.15.123:0/3247217763 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb9b41f1920 0x7fb9b41f1da0 secure :-1 s=READY pgs=1476 cs=0 l=1 rev1=1 crypto rx=0x7fb9ac007010 tx=0x7fb9ac0019f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:50:11.497 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:11.494+0000 7fb9b17fa640 1 -- 172.21.15.123:0/3247217763 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9ac003d90 con 0x7fb9b41f1920 2024-09-16T15:50:11.497 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:11.494+0000 7fb9b17fa640 1 -- 172.21.15.123:0/3247217763 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fb9ac0060d0 con 0x7fb9b41f1920 2024-09-16T15:50:11.497 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:11.494+0000 7fb9ba953640 1 -- 172.21.15.123:0/3247217763 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb9b40c6f90 con 0x7fb9b41f1920 2024-09-16T15:50:11.497 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:11.494+0000 7fb9b17fa640 1 -- 172.21.15.123:0/3247217763 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9ac0064c0 con 0x7fb9b41f1920 2024-09-16T15:50:11.497 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:11.494+0000 7fb9ba953640 1 -- 172.21.15.123:0/3247217763 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb9b40c7550 con 0x7fb9b41f1920 2024-09-16T15:50:11.498 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:11.498+0000 7fb9ba953640 1 -- 172.21.15.123:0/3247217763 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb9b40cc5c0 con 0x7fb9b41f1920 2024-09-16T15:50:11.498 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:11.498+0000 7fb9b17fa640 1 -- 172.21.15.123:0/3247217763 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 21) ==== 97641+0+0 (secure 0 0 0) 0x7fb9ac006660 con 0x7fb9b41f1920 2024-09-16T15:50:11.498 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:11.498+0000 7fb9b17fa640 1 --2- 172.21.15.123:0/3247217763 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fb9940734f0 0x7fb9940759b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:50:11.498 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:11.498+0000 7fb9b3fff640 1 --2- 172.21.15.123:0/3247217763 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fb9940734f0 0x7fb9940759b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:50:11.498 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:11.498+0000 7fb9b17fa640 1 -- 172.21.15.123:0/3247217763 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(135..135 src has 1..135) ==== 5737+0+0 (secure 0 0 0) 0x7fb9ac12f010 con 0x7fb9b41f1920 2024-09-16T15:50:11.499 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:11.498+0000 7fb9b3fff640 1 --2- 172.21.15.123:0/3247217763 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fb9940734f0 0x7fb9940759b0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7fb9a4012640 tx=0x7fb9a4000ee0 comp rx=0 tx=0).ready entity=mgr.4150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:50:11.501 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:11.502+0000 7fb9b17fa640 1 -- 172.21.15.123:0/3247217763 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7fb9ac0bf030 con 0x7fb9b41f1920 2024-09-16T15:50:11.692 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:11.690+0000 7fb9ba953640 1 -- 172.21.15.123:0/3247217763 --> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot clone", "vol_name": "non_existent_subvolume", "sub_name": "subvolume_1", "snap_name": "snapshot_1", "target_sub_name": "clone_1", "target": ["mon-mgr", ""]}) -- 0x7fb9b40b1af0 con 0x7fb9940734f0 2024-09-16T15:50:11.701 INFO:tasks.ceph.mgr.y.smithi123.stderr:2024-09-16T15:50:11.702+0000 7fa9b36f4640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-16T15:50:11.701 INFO:tasks.ceph.mgr.y.smithi123.stderr:2024-09-16T15:50:11.702+0000 7fa9b36f4640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-16T15:50:11.701 INFO:tasks.ceph.mgr.y.smithi123.stderr:2024-09-16T15:50:11.702+0000 7fa9b36f4640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-16T15:50:11.701 INFO:tasks.ceph.mgr.y.smithi123.stderr:2024-09-16T15:50:11.702+0000 7fa9b36f4640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-16T15:50:11.701 INFO:tasks.ceph.mgr.y.smithi123.stderr:2024-09-16T15:50:11.702+0000 7fa9b36f4640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-16T15:50:11.706 INFO:tasks.ceph.mgr.y.smithi123.stderr:2024-09-16T15:50:11.706+0000 7fa9b36f4640 -1 mgr.server reply reply (2) No such file or directory FS 'non_existent_subvolume' not found 2024-09-16T15:50:11.706 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:11.706+0000 7fb9b17fa640 1 -- 172.21.15.123:0/3247217763 <== mgr.4150 v2:172.21.15.123:6836/2464759676 1 ==== mgr_command_reply(tid 0: -2 FS 'non_existent_subvolume' not found) ==== 45+0+0 (secure 0 0 0) 0x7fb9b40b1af0 con 0x7fb9940734f0 2024-09-16T15:50:11.706 INFO:teuthology.orchestra.run.smithi123.stderr:Error ENOENT: FS 'non_existent_subvolume' not found 2024-09-16T15:50:11.710 DEBUG:teuthology.orchestra.run:got remote process result: 2 2024-09-16T15:50:11.711 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status non_existent_subvolume clone_1 2024-09-16T15:50:11.832 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:11.830+0000 7f4109b75640 1 -- 172.21.15.123:0/2067502888 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f41040cf4e0 msgr2=0x7f410415f6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:50:11.832 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:11.830+0000 7f4109b75640 1 --2- 172.21.15.123:0/2067502888 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f41040cf4e0 0x7f410415f6b0 secure :-1 s=READY pgs=1479 cs=0 l=1 rev1=1 crypto rx=0x7f40fc009c80 tx=0x7f40fc01cda0 comp rx=0 tx=0).stop 2024-09-16T15:50:11.832 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:11.830+0000 7f4109b75640 1 -- 172.21.15.123:0/2067502888 shutdown_connections 2024-09-16T15:50:11.832 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:11.830+0000 7f4109b75640 1 --2- 172.21.15.123:0/2067502888 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f41040cf4e0 0x7f410415f6b0 unknown :-1 s=CLOSED pgs=1479 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:11.833 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:11.830+0000 7f4109b75640 1 --2- 172.21.15.123:0/2067502888 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f410415b860 0x7f410406fca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:11.833 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:11.830+0000 7f4109b75640 1 -- 172.21.15.123:0/2067502888 >> 172.21.15.123:0/2067502888 conn(0x7f41040b1710 msgr2=0x7f41040b1b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:50:11.833 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:11.830+0000 7f4109b75640 1 -- 172.21.15.123:0/2067502888 shutdown_connections 2024-09-16T15:50:11.833 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:11.830+0000 7f4109b75640 1 -- 172.21.15.123:0/2067502888 wait complete. 2024-09-16T15:50:11.833 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:11.834+0000 7f4109b75640 1 Processor -- start 2024-09-16T15:50:11.833 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:11.834+0000 7f4109b75640 1 -- start start 2024-09-16T15:50:11.835 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:11.834+0000 7f4109b75640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f41040c3230 0x7f410415a4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:50:11.835 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:11.834+0000 7f4109b75640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f41040cf4e0 0x7f41041f9540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:50:11.835 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:11.834+0000 7f4109b75640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f410415b860 0x7f41041f9a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:50:11.835 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:11.834+0000 7f4109b75640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f41041691e0 con 0x7f41040c3230 2024-09-16T15:50:11.835 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:11.834+0000 7f4109b75640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f4104169060 con 0x7f410415b860 2024-09-16T15:50:11.835 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:11.834+0000 7f4109b75640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f4104169360 con 0x7f41040cf4e0 2024-09-16T15:50:11.836 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:11.834+0000 7f41037fe640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f41040c3230 0x7f410415a4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:50:11.836 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:11.834+0000 7f41037fe640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f41040c3230 0x7f410415a4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:53504/0 (socket says 172.21.15.123:53504) 2024-09-16T15:50:11.836 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:11.834+0000 7f41037fe640 1 -- 172.21.15.123:0/1685426957 learned_addr learned my addr 172.21.15.123:0/1685426957 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:50:11.836 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:11.834+0000 7f4103fff640 1 --2- 172.21.15.123:0/1685426957 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f410415b860 0x7f41041f9a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:50:11.836 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:11.834+0000 7f41037fe640 1 -- 172.21.15.123:0/1685426957 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f41040cf4e0 msgr2=0x7f41041f9540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:50:11.836 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:11.834+0000 7f41037fe640 1 --2- 172.21.15.123:0/1685426957 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f41040cf4e0 0x7f41041f9540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:11.836 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:11.834+0000 7f41037fe640 1 -- 172.21.15.123:0/1685426957 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f410415b860 msgr2=0x7f41041f9a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:50:11.836 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:11.834+0000 7f41037fe640 1 --2- 172.21.15.123:0/1685426957 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f410415b860 0x7f41041f9a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:11.836 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:11.834+0000 7f41037fe640 1 -- 172.21.15.123:0/1685426957 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f41041fa3c0 con 0x7f41040c3230 2024-09-16T15:50:11.836 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:11.834+0000 7f41037fe640 1 --2- 172.21.15.123:0/1685426957 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f41040c3230 0x7f410415a4c0 secure :-1 s=READY pgs=1480 cs=0 l=1 rev1=1 crypto rx=0x7f40f400d720 tx=0x7f40f400dbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:50:11.836 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:11.834+0000 7f40ebfff640 1 -- 172.21.15.123:0/1685426957 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40f4017070 con 0x7f41040c3230 2024-09-16T15:50:11.836 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:11.834+0000 7f40ebfff640 1 -- 172.21.15.123:0/1685426957 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f40f4009cc0 con 0x7f41040c3230 2024-09-16T15:50:11.836 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:11.834+0000 7f4109b75640 1 -- 172.21.15.123:0/1685426957 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4104147550 con 0x7f41040c3230 2024-09-16T15:50:11.837 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:11.834+0000 7f40ebfff640 1 -- 172.21.15.123:0/1685426957 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40f4005280 con 0x7f41040c3230 2024-09-16T15:50:11.837 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:11.834+0000 7f4109b75640 1 -- 172.21.15.123:0/1685426957 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4104147b10 con 0x7f41040c3230 2024-09-16T15:50:11.838 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:11.834+0000 7f4109b75640 1 -- 172.21.15.123:0/1685426957 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f410415f6b0 con 0x7f41040c3230 2024-09-16T15:50:11.838 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:11.838+0000 7f40ebfff640 1 -- 172.21.15.123:0/1685426957 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 21) ==== 97641+0+0 (secure 0 0 0) 0x7f40f4005420 con 0x7f41040c3230 2024-09-16T15:50:11.838 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:11.838+0000 7f40ebfff640 1 --2- 172.21.15.123:0/1685426957 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f40f00734f0 0x7f40f00759b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:50:11.838 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:11.838+0000 7f40ebfff640 1 -- 172.21.15.123:0/1685426957 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(135..135 src has 1..135) ==== 5737+0+0 (secure 0 0 0) 0x7f40f4094ab0 con 0x7f41040c3230 2024-09-16T15:50:11.838 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:11.838+0000 7f4102ffd640 1 --2- 172.21.15.123:0/1685426957 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f40f00734f0 0x7f40f00759b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:50:11.838 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:11.838+0000 7f4102ffd640 1 --2- 172.21.15.123:0/1685426957 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f40f00734f0 0x7f40f00759b0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f40f8000f70 tx=0x7f40f80079c0 comp rx=0 tx=0).ready entity=mgr.4150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:50:11.841 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:11.842+0000 7f40ebfff640 1 -- 172.21.15.123:0/1685426957 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f40f405f320 con 0x7f41040c3230 2024-09-16T15:50:12.045 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:12.046+0000 7f4109b75640 1 -- 172.21.15.123:0/1685426957 --> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "non_existent_subvolume", "clone_name": "clone_1", "target": ["mon-mgr", ""]}) -- 0x7f41040b52d0 con 0x7f40f00734f0 2024-09-16T15:50:12.047 INFO:tasks.ceph.mgr.y.smithi123.stderr:2024-09-16T15:50:12.046+0000 7fa9b36f4640 -1 mgr.server reply reply (2) No such file or directory FS 'non_existent_subvolume' not found 2024-09-16T15:50:12.047 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:12.046+0000 7f40ebfff640 1 -- 172.21.15.123:0/1685426957 <== mgr.4150 v2:172.21.15.123:6836/2464759676 1 ==== mgr_command_reply(tid 0: -2 FS 'non_existent_subvolume' not found) ==== 45+0+0 (secure 0 0 0) 0x7f41040b52d0 con 0x7f40f00734f0 2024-09-16T15:50:12.047 INFO:teuthology.orchestra.run.smithi123.stderr:Error ENOENT: FS 'non_existent_subvolume' not found 2024-09-16T15:50:12.055 DEBUG:teuthology.orchestra.run:got remote process result: 2 2024-09-16T15:50:12.055 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolumegroup create non_existent_subvolume group_1 2024-09-16T15:50:12.174 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:12.174+0000 7f833a165640 1 -- 172.21.15.123:0/3716165091 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f833415a9d0 msgr2=0x7f833406ebe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:50:12.174 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:12.174+0000 7f833a165640 1 --2- 172.21.15.123:0/3716165091 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f833415a9d0 0x7f833406ebe0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f8324009900 tx=0x7f832401c820 comp rx=0 tx=0).stop 2024-09-16T15:50:12.174 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:12.174+0000 7f833a165640 1 -- 172.21.15.123:0/3716165091 shutdown_connections 2024-09-16T15:50:12.174 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:12.174+0000 7f833a165640 1 --2- 172.21.15.123:0/3716165091 >> v2:172.21.15.184:3300/0 conn(0x7f8334072c70 0x7f83340cc3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:12.174 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:12.174+0000 7f833a165640 1 --2- 172.21.15.123:0/3716165091 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f833415a9d0 0x7f833406ebe0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:12.174 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:12.174+0000 7f833a165640 1 -- 172.21.15.123:0/3716165091 >> 172.21.15.123:0/3716165091 conn(0x7f83340b1710 msgr2=0x7f83340b1b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:50:12.174 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:12.174+0000 7f833a165640 1 -- 172.21.15.123:0/3716165091 shutdown_connections 2024-09-16T15:50:12.174 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:12.174+0000 7f833a165640 1 -- 172.21.15.123:0/3716165091 wait complete. 2024-09-16T15:50:12.175 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:12.174+0000 7f833a165640 1 Processor -- start 2024-09-16T15:50:12.175 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:12.174+0000 7f833a165640 1 -- start start 2024-09-16T15:50:12.177 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:12.174+0000 7f833a165640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f8334072720 0x7f8334156bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:50:12.177 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:12.174+0000 7f833a165640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f8334072c70 0x7f8334165010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:50:12.177 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:12.174+0000 7f833a165640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f833415a9d0 0x7f83340afd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:50:12.177 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:12.174+0000 7f833a165640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f8334168380 con 0x7f833415a9d0 2024-09-16T15:50:12.177 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:12.174+0000 7f833a165640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f8334168200 con 0x7f8334072720 2024-09-16T15:50:12.178 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:12.174+0000 7f833a165640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f8334168500 con 0x7f8334072c70 2024-09-16T15:50:12.178 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:12.174+0000 7f8333fff640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f833415a9d0 0x7f83340afd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:50:12.178 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:12.174+0000 7f8333fff640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f833415a9d0 0x7f83340afd40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:53518/0 (socket says 172.21.15.123:53518) 2024-09-16T15:50:12.178 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:12.174+0000 7f8333fff640 1 -- 172.21.15.123:0/3443128033 learned_addr learned my addr 172.21.15.123:0/3443128033 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:50:12.178 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:12.174+0000 7f832b7fe640 1 --2- 172.21.15.123:0/3443128033 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f8334072720 0x7f8334156bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:50:12.178 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:12.174+0000 7f83337fe640 1 --2- 172.21.15.123:0/3443128033 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f8334072c70 0x7f8334165010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:50:12.178 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:12.174+0000 7f8333fff640 1 -- 172.21.15.123:0/3443128033 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f8334072c70 msgr2=0x7f8334165010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:50:12.178 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:12.174+0000 7f8333fff640 1 --2- 172.21.15.123:0/3443128033 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f8334072c70 0x7f8334165010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:12.178 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:12.174+0000 7f8333fff640 1 -- 172.21.15.123:0/3443128033 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f8334072720 msgr2=0x7f8334156bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:50:12.178 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:12.174+0000 7f8333fff640 1 --2- 172.21.15.123:0/3443128033 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f8334072720 0x7f8334156bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:12.178 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:12.174+0000 7f8333fff640 1 -- 172.21.15.123:0/3443128033 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f83340c9420 con 0x7f833415a9d0 2024-09-16T15:50:12.179 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:12.174+0000 7f832b7fe640 1 --2- 172.21.15.123:0/3443128033 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f8334072720 0x7f8334156bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T15:50:12.179 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:12.178+0000 7f8333fff640 1 --2- 172.21.15.123:0/3443128033 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f833415a9d0 0x7f83340afd40 secure :-1 s=READY pgs=1481 cs=0 l=1 rev1=1 crypto rx=0x7f832c00c730 tx=0x7f832c00cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:50:12.179 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:12.178+0000 7f8330ff9640 1 -- 172.21.15.123:0/3443128033 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f832c003b80 con 0x7f833415a9d0 2024-09-16T15:50:12.179 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:12.178+0000 7f8330ff9640 1 -- 172.21.15.123:0/3443128033 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f832c003d20 con 0x7f833415a9d0 2024-09-16T15:50:12.179 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:12.178+0000 7f833a165640 1 -- 172.21.15.123:0/3443128033 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8334157700 con 0x7f833415a9d0 2024-09-16T15:50:12.179 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:12.178+0000 7f8330ff9640 1 -- 172.21.15.123:0/3443128033 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f832c00aab0 con 0x7f833415a9d0 2024-09-16T15:50:12.179 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:12.178+0000 7f833a165640 1 -- 172.21.15.123:0/3443128033 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f83340c9c60 con 0x7f833415a9d0 2024-09-16T15:50:12.179 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:12.178+0000 7f833a165640 1 -- 172.21.15.123:0/3443128033 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f83340cc3c0 con 0x7f833415a9d0 2024-09-16T15:50:12.180 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:12.178+0000 7f8330ff9640 1 -- 172.21.15.123:0/3443128033 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 21) ==== 97641+0+0 (secure 0 0 0) 0x7f832c00ac50 con 0x7f833415a9d0 2024-09-16T15:50:12.180 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:12.178+0000 7f8330ff9640 1 --2- 172.21.15.123:0/3443128033 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f8314073520 0x7f83140759e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:50:12.180 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:12.178+0000 7f8330ff9640 1 -- 172.21.15.123:0/3443128033 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(135..135 src has 1..135) ==== 5737+0+0 (secure 0 0 0) 0x7f832c094880 con 0x7f833415a9d0 2024-09-16T15:50:12.180 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:12.178+0000 7f832b7fe640 1 --2- 172.21.15.123:0/3443128033 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f8314073520 0x7f83140759e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:50:12.180 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:12.178+0000 7f832b7fe640 1 --2- 172.21.15.123:0/3443128033 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f8314073520 0x7f83140759e0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f83240af000 tx=0x7f832401b060 comp rx=0 tx=0).ready entity=mgr.4150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:50:12.182 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:12.182+0000 7f8330ff9640 1 -- 172.21.15.123:0/3443128033 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f83340c9420 con 0x7f833415a9d0 2024-09-16T15:50:12.219 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:12.218+0000 7f8330ff9640 1 -- 172.21.15.123:0/3443128033 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mgrmap(e 22) ==== 97692+0+0 (secure 0 0 0) 0x7f832c05abd0 con 0x7f833415a9d0 2024-09-16T15:50:12.368 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:12.370+0000 7f833a165640 1 -- 172.21.15.123:0/3443128033 --> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] -- mgr_command(tid 0: {"prefix": "fs subvolumegroup create", "vol_name": "non_existent_subvolume", "group_name": "group_1", "target": ["mon-mgr", ""]}) -- 0x7f83340ca9a0 con 0x7f8314073520 2024-09-16T15:50:12.371 INFO:tasks.ceph.mgr.y.smithi123.stderr:2024-09-16T15:50:12.370+0000 7fa9b36f4640 -1 mgr.server reply reply (2) No such file or directory FS 'non_existent_subvolume' not found 2024-09-16T15:50:12.371 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:12.370+0000 7f8330ff9640 1 -- 172.21.15.123:0/3443128033 <== mgr.4150 v2:172.21.15.123:6836/2464759676 1 ==== mgr_command_reply(tid 0: -2 FS 'non_existent_subvolume' not found) ==== 45+0+0 (secure 0 0 0) 0x7f83340ca9a0 con 0x7f8314073520 2024-09-16T15:50:12.371 INFO:teuthology.orchestra.run.smithi123.stderr:Error ENOENT: FS 'non_existent_subvolume' not found 2024-09-16T15:50:12.379 DEBUG:teuthology.orchestra.run:got remote process result: 2 2024-09-16T15:50:12.379 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolumegroup rm non_existent_subvolume group_1 2024-09-16T15:50:12.510 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:12.510+0000 7f6571ed3640 1 -- 172.21.15.123:0/2808444879 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6554004d50 msgr2=0x7f65540051b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:50:12.510 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:12.510+0000 7f6571ed3640 1 --2- 172.21.15.123:0/2808444879 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6554004d50 0x7f65540051b0 secure :-1 s=READY pgs=1483 cs=0 l=1 rev1=1 crypto rx=0x7f656c074a00 tx=0x7f656401d5d0 comp rx=0 tx=0).stop 2024-09-16T15:50:12.510 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:12.510+0000 7f656bfff640 1 -- 172.21.15.123:0/2808444879 reap_dead start 2024-09-16T15:50:12.510 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:12.510+0000 7f6571ed3640 1 -- 172.21.15.123:0/2808444879 shutdown_connections 2024-09-16T15:50:12.510 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:12.510+0000 7f6571ed3640 1 -- 172.21.15.123:0/2808444879 >> 172.21.15.123:0/2808444879 conn(0x7f656c0b1710 msgr2=0x7f656c0b1b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:50:12.510 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:12.510+0000 7f6571ed3640 1 -- 172.21.15.123:0/2808444879 shutdown_connections 2024-09-16T15:50:12.510 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:12.510+0000 7f6571ed3640 1 -- 172.21.15.123:0/2808444879 wait complete. 2024-09-16T15:50:12.511 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:12.510+0000 7f6571ed3640 1 Processor -- start 2024-09-16T15:50:12.511 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:12.510+0000 7f6571ed3640 1 -- start start 2024-09-16T15:50:12.513 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:12.510+0000 7f6571ed3640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6554004d50 0x7f656c1f64c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:50:12.513 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:12.510+0000 7f6571ed3640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f656c146d00 0x7f656c0b5000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:50:12.513 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:12.510+0000 7f6571ed3640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f656c0cd7a0 0x7f656c0cdc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:50:12.513 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:12.510+0000 7f6571ed3640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f656c15e960 con 0x7f656c0cd7a0 2024-09-16T15:50:12.514 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:12.510+0000 7f6571ed3640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f656c15e7e0 con 0x7f6554004d50 2024-09-16T15:50:12.514 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:12.510+0000 7f6571ed3640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f656c147180 con 0x7f656c146d00 2024-09-16T15:50:12.514 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:12.514+0000 7f656bfff640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f656c0cd7a0 0x7f656c0cdc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:50:12.514 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:12.514+0000 7f656bfff640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f656c0cd7a0 0x7f656c0cdc20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:37012/0 (socket says 172.21.15.123:37012) 2024-09-16T15:50:12.514 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:12.514+0000 7f656bfff640 1 -- 172.21.15.123:0/3473931253 learned_addr learned my addr 172.21.15.123:0/3473931253 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:50:12.514 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:12.514+0000 7f656b7fe640 1 --2- 172.21.15.123:0/3473931253 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6554004d50 0x7f656c1f64c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:50:12.514 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:12.514+0000 7f656affd640 1 --2- 172.21.15.123:0/3473931253 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f656c146d00 0x7f656c0b5000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:50:12.514 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:12.514+0000 7f656bfff640 1 -- 172.21.15.123:0/3473931253 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f656c146d00 msgr2=0x7f656c0b5000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:50:12.514 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:12.514+0000 7f656bfff640 1 --2- 172.21.15.123:0/3473931253 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f656c146d00 0x7f656c0b5000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:12.514 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:12.514+0000 7f656bfff640 1 -- 172.21.15.123:0/3473931253 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6554004d50 msgr2=0x7f656c1f64c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:50:12.514 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:12.514+0000 7f656bfff640 1 --2- 172.21.15.123:0/3473931253 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6554004d50 0x7f656c1f64c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:12.514 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:12.514+0000 7f656bfff640 1 -- 172.21.15.123:0/3473931253 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f656c1f6ae0 con 0x7f656c0cd7a0 2024-09-16T15:50:12.515 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:12.514+0000 7f656bfff640 1 --2- 172.21.15.123:0/3473931253 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f656c0cd7a0 0x7f656c0cdc20 secure :-1 s=READY pgs=1484 cs=0 l=1 rev1=1 crypto rx=0x7f6564099100 tx=0x7f6564004e80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:50:12.515 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:12.514+0000 7f6553fff640 1 -- 172.21.15.123:0/3473931253 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6564009ab0 con 0x7f656c0cd7a0 2024-09-16T15:50:12.515 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:12.514+0000 7f6553fff640 1 -- 172.21.15.123:0/3473931253 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f65640a9510 con 0x7f656c0cd7a0 2024-09-16T15:50:12.515 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:12.514+0000 7f6571ed3640 1 -- 172.21.15.123:0/3473931253 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f656c1f6dc0 con 0x7f656c0cd7a0 2024-09-16T15:50:12.515 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:12.514+0000 7f6553fff640 1 -- 172.21.15.123:0/3473931253 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6564009200 con 0x7f656c0cd7a0 2024-09-16T15:50:12.515 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:12.514+0000 7f6571ed3640 1 -- 172.21.15.123:0/3473931253 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f656c14f820 con 0x7f656c0cd7a0 2024-09-16T15:50:12.515 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:12.514+0000 7f6571ed3640 1 -- 172.21.15.123:0/3473931253 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f656c0cc3c0 con 0x7f656c0cd7a0 2024-09-16T15:50:12.516 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:12.514+0000 7f6553fff640 1 -- 172.21.15.123:0/3473931253 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 22) ==== 97692+0+0 (secure 0 0 0) 0x7f65640b9030 con 0x7f656c0cd7a0 2024-09-16T15:50:12.516 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:12.514+0000 7f6553fff640 1 --2- 172.21.15.123:0/3473931253 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f6558073580 0x7f6558075a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:50:12.516 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:12.514+0000 7f6553fff640 1 -- 172.21.15.123:0/3473931253 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(135..135 src has 1..135) ==== 5737+0+0 (secure 0 0 0) 0x7f656412f9f0 con 0x7f656c0cd7a0 2024-09-16T15:50:12.516 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:12.514+0000 7f656b7fe640 1 --2- 172.21.15.123:0/3473931253 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f6558073580 0x7f6558075a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:50:12.516 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:12.514+0000 7f656b7fe640 1 --2- 172.21.15.123:0/3473931253 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f6558073580 0x7f6558075a40 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f655c006610 tx=0x7f655c006ba0 comp rx=0 tx=0).ready entity=mgr.4150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:50:12.518 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:12.518+0000 7f6553fff640 1 -- 172.21.15.123:0/3473931253 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f65640ad030 con 0x7f656c0cd7a0 2024-09-16T15:50:12.716 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:12.714+0000 7f6571ed3640 1 -- 172.21.15.123:0/3473931253 --> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] -- mgr_command(tid 0: {"prefix": "fs subvolumegroup rm", "vol_name": "non_existent_subvolume", "group_name": "group_1", "target": ["mon-mgr", ""]}) -- 0x7f656c0cd440 con 0x7f6558073580 2024-09-16T15:50:12.718 INFO:tasks.ceph.mgr.y.smithi123.stderr:2024-09-16T15:50:12.718+0000 7fa9b36f4640 -1 mgr.server reply reply (2) No such file or directory FS 'non_existent_subvolume' not found 2024-09-16T15:50:12.718 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:12.718+0000 7f6553fff640 1 -- 172.21.15.123:0/3473931253 <== mgr.4150 v2:172.21.15.123:6836/2464759676 1 ==== mgr_command_reply(tid 0: -2 FS 'non_existent_subvolume' not found) ==== 45+0+0 (secure 0 0 0) 0x7f656c0cd440 con 0x7f6558073580 2024-09-16T15:50:12.718 INFO:teuthology.orchestra.run.smithi123.stderr:Error ENOENT: FS 'non_existent_subvolume' not found 2024-09-16T15:50:12.724 DEBUG:teuthology.orchestra.run:got remote process result: 2 2024-09-16T15:50:12.724 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolumegroup getpath non_existent_subvolume group_1 2024-09-16T15:50:12.849 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:12.850+0000 7fd8dfe9a640 1 -- 172.21.15.123:0/3946303767 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd8d815bc40 msgr2=0x7fd8d80cf310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:50:12.849 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:12.850+0000 7fd8dfe9a640 1 --2- 172.21.15.123:0/3946303767 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd8d815bc40 0x7fd8d80cf310 secure :-1 s=READY pgs=1486 cs=0 l=1 rev1=1 crypto rx=0x7fd8d0009bc0 tx=0x7fd8d001d540 comp rx=0 tx=0).stop 2024-09-16T15:50:12.849 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:12.850+0000 7fd8dfe9a640 1 -- 172.21.15.123:0/3946303767 shutdown_connections 2024-09-16T15:50:12.849 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:12.850+0000 7fd8dfe9a640 1 --2- 172.21.15.123:0/3946303767 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd8d815bc40 0x7fd8d80cf310 unknown :-1 s=CLOSED pgs=1486 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:12.849 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:12.850+0000 7fd8dfe9a640 1 --2- 172.21.15.123:0/3946303767 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fd8d80c3230 0x7fd8d80cedd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:12.849 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:12.850+0000 7fd8dfe9a640 1 -- 172.21.15.123:0/3946303767 >> 172.21.15.123:0/3946303767 conn(0x7fd8d80b1710 msgr2=0x7fd8d80b1b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:50:12.850 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:12.850+0000 7fd8dfe9a640 1 -- 172.21.15.123:0/3946303767 shutdown_connections 2024-09-16T15:50:12.850 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:12.850+0000 7fd8dfe9a640 1 -- 172.21.15.123:0/3946303767 wait complete. 2024-09-16T15:50:12.850 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:12.850+0000 7fd8dfe9a640 1 Processor -- start 2024-09-16T15:50:12.850 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:12.850+0000 7fd8dfe9a640 1 -- start start 2024-09-16T15:50:12.853 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:12.850+0000 7fd8dfe9a640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd8d80c3230 0x7fd8d814fed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:50:12.853 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:12.850+0000 7fd8dfe9a640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd8d815b860 0x7fd8d814af80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:50:12.853 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:12.850+0000 7fd8dfe9a640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fd8d815bc40 0x7fd8d81504d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:50:12.853 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:12.850+0000 7fd8dfe9a640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fd8d81689c0 con 0x7fd8d815b860 2024-09-16T15:50:12.853 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:12.850+0000 7fd8dfe9a640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fd8d8168840 con 0x7fd8d80c3230 2024-09-16T15:50:12.853 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:12.850+0000 7fd8dfe9a640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fd8d8168b40 con 0x7fd8d815bc40 2024-09-16T15:50:12.853 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:12.850+0000 7fd8dd435640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd8d815b860 0x7fd8d814af80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:50:12.853 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:12.850+0000 7fd8dd435640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd8d815b860 0x7fd8d814af80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:37034/0 (socket says 172.21.15.123:37034) 2024-09-16T15:50:12.853 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:12.850+0000 7fd8dd435640 1 -- 172.21.15.123:0/4171096506 learned_addr learned my addr 172.21.15.123:0/4171096506 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:50:12.853 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:12.850+0000 7fd8ddc36640 1 --2- 172.21.15.123:0/4171096506 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd8d80c3230 0x7fd8d814fed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:50:12.853 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:12.850+0000 7fd8de437640 1 --2- 172.21.15.123:0/4171096506 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fd8d815bc40 0x7fd8d81504d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:50:12.853 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:12.850+0000 7fd8dd435640 1 -- 172.21.15.123:0/4171096506 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fd8d815bc40 msgr2=0x7fd8d81504d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:50:12.854 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:12.850+0000 7fd8dd435640 1 --2- 172.21.15.123:0/4171096506 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fd8d815bc40 0x7fd8d81504d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:12.854 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:12.850+0000 7fd8dd435640 1 -- 172.21.15.123:0/4171096506 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd8d80c3230 msgr2=0x7fd8d814fed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:50:12.854 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:12.850+0000 7fd8dd435640 1 --2- 172.21.15.123:0/4171096506 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd8d80c3230 0x7fd8d814fed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:12.854 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:12.850+0000 7fd8dd435640 1 -- 172.21.15.123:0/4171096506 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd8d8151000 con 0x7fd8d815b860 2024-09-16T15:50:12.854 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:12.850+0000 7fd8dd435640 1 --2- 172.21.15.123:0/4171096506 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd8d815b860 0x7fd8d814af80 secure :-1 s=READY pgs=1487 cs=0 l=1 rev1=1 crypto rx=0x7fd8c800d980 tx=0x7fd8c800de40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:50:12.854 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:12.850+0000 7fd8ce7fc640 1 -- 172.21.15.123:0/4171096506 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8c8017070 con 0x7fd8d815b860 2024-09-16T15:50:12.854 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:12.850+0000 7fd8ce7fc640 1 -- 172.21.15.123:0/4171096506 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fd8c8009d20 con 0x7fd8d815b860 2024-09-16T15:50:12.854 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:12.850+0000 7fd8dfe9a640 1 -- 172.21.15.123:0/4171096506 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd8d8150d20 con 0x7fd8d815b860 2024-09-16T15:50:12.854 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:12.850+0000 7fd8ce7fc640 1 -- 172.21.15.123:0/4171096506 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8c80106c0 con 0x7fd8d815b860 2024-09-16T15:50:12.854 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:12.850+0000 7fd8dfe9a640 1 -- 172.21.15.123:0/4171096506 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd8d80c8560 con 0x7fd8d815b860 2024-09-16T15:50:12.854 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:12.854+0000 7fd8dfe9a640 1 -- 172.21.15.123:0/4171096506 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd8d806fc80 con 0x7fd8d815b860 2024-09-16T15:50:12.855 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:12.854+0000 7fd8ce7fc640 1 -- 172.21.15.123:0/4171096506 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 22) ==== 97692+0+0 (secure 0 0 0) 0x7fd8c801a070 con 0x7fd8d815b860 2024-09-16T15:50:12.855 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:12.854+0000 7fd8ce7fc640 1 --2- 172.21.15.123:0/4171096506 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fd8bc0735b0 0x7fd8bc075a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:50:12.855 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:12.854+0000 7fd8ce7fc640 1 -- 172.21.15.123:0/4171096506 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(135..135 src has 1..135) ==== 5737+0+0 (secure 0 0 0) 0x7fd8c8095050 con 0x7fd8d815b860 2024-09-16T15:50:12.855 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:12.854+0000 7fd8ddc36640 1 --2- 172.21.15.123:0/4171096506 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fd8bc0735b0 0x7fd8bc075a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:50:12.855 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:12.854+0000 7fd8ddc36640 1 --2- 172.21.15.123:0/4171096506 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fd8bc0735b0 0x7fd8bc075a70 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7fd8d4000f70 tx=0x7fd8d4009040 comp rx=0 tx=0).ready entity=mgr.4150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:50:12.857 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:12.858+0000 7fd8ce7fc640 1 -- 172.21.15.123:0/4171096506 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7fd8c805f8e0 con 0x7fd8d815b860 2024-09-16T15:50:13.051 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:13.050+0000 7fd8dfe9a640 1 -- 172.21.15.123:0/4171096506 --> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] -- mgr_command(tid 0: {"prefix": "fs subvolumegroup getpath", "vol_name": "non_existent_subvolume", "group_name": "group_1", "target": ["mon-mgr", ""]}) -- 0x7fd8d80b3220 con 0x7fd8bc0735b0 2024-09-16T15:50:13.054 INFO:tasks.ceph.mgr.y.smithi123.stderr:2024-09-16T15:50:13.054+0000 7fa9b36f4640 -1 mgr.server reply reply (2) No such file or directory FS 'non_existent_subvolume' not found 2024-09-16T15:50:13.054 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:13.054+0000 7fd8ce7fc640 1 -- 172.21.15.123:0/4171096506 <== mgr.4150 v2:172.21.15.123:6836/2464759676 1 ==== mgr_command_reply(tid 0: -2 FS 'non_existent_subvolume' not found) ==== 45+0+0 (secure 0 0 0) 0x7fd8d80b3220 con 0x7fd8bc0735b0 2024-09-16T15:50:13.054 INFO:teuthology.orchestra.run.smithi123.stderr:Error ENOENT: FS 'non_existent_subvolume' not found 2024-09-16T15:50:13.060 DEBUG:teuthology.orchestra.run:got remote process result: 2 2024-09-16T15:50:13.060 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolumegroup pin non_existent_subvolume group_1 export 0 2024-09-16T15:50:13.184 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:13.182+0000 7f7c23dab640 1 -- 172.21.15.123:0/2973615973 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7c1c070470 msgr2=0x7f7c1c157d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:50:13.184 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:13.182+0000 7f7c23dab640 1 --2- 172.21.15.123:0/2973615973 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7c1c070470 0x7f7c1c157d00 secure :-1 s=READY pgs=1488 cs=0 l=1 rev1=1 crypto rx=0x7f7c1c0ccbf0 tx=0x7f7c1401cd80 comp rx=0 tx=0).stop 2024-09-16T15:50:13.184 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:13.182+0000 7f7c23dab640 1 -- 172.21.15.123:0/2973615973 shutdown_connections 2024-09-16T15:50:13.184 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:13.182+0000 7f7c23dab640 1 --2- 172.21.15.123:0/2973615973 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7c1c070470 0x7f7c1c157d00 unknown :-1 s=CLOSED pgs=1488 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:13.184 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:13.182+0000 7f7c23dab640 1 --2- 172.21.15.123:0/2973615973 >> v2:172.21.15.184:3300/0 conn(0x7f7c1c073390 0x7f7c1c06ff30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:13.184 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:13.182+0000 7f7c23dab640 1 --2- 172.21.15.123:0/2973615973 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f7c1c0aee40 0x7f7c1c15b980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:13.184 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:13.182+0000 7f7c23dab640 1 -- 172.21.15.123:0/2973615973 >> 172.21.15.123:0/2973615973 conn(0x7f7c1c0b1630 msgr2=0x7f7c1c0b1a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:50:13.184 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:13.182+0000 7f7c23dab640 1 -- 172.21.15.123:0/2973615973 shutdown_connections 2024-09-16T15:50:13.184 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:13.182+0000 7f7c23dab640 1 -- 172.21.15.123:0/2973615973 wait complete. 2024-09-16T15:50:13.185 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:13.186+0000 7f7c23dab640 1 Processor -- start 2024-09-16T15:50:13.185 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:13.186+0000 7f7c23dab640 1 -- start start 2024-09-16T15:50:13.187 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:13.186+0000 7f7c23dab640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7c1c070470 0x7f7c1c147860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:50:13.187 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:13.186+0000 7f7c23dab640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7c1c073390 0x7f7c1c14f470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:50:13.187 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:13.186+0000 7f7c23dab640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f7c1c0aee40 0x7f7c1c0b3e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:50:13.187 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:13.186+0000 7f7c23dab640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f7c1c15ebf0 con 0x7f7c1c070470 2024-09-16T15:50:13.187 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:13.186+0000 7f7c23dab640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f7c1c15ea70 con 0x7f7c1c073390 2024-09-16T15:50:13.187 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:13.186+0000 7f7c23dab640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f7c1c15ed70 con 0x7f7c1c0aee40 2024-09-16T15:50:13.187 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:13.186+0000 7f7c21b47640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7c1c070470 0x7f7c1c147860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:50:13.187 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:13.186+0000 7f7c21b47640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7c1c070470 0x7f7c1c147860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:37048/0 (socket says 172.21.15.123:37048) 2024-09-16T15:50:13.187 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:13.186+0000 7f7c21b47640 1 -- 172.21.15.123:0/3976606250 learned_addr learned my addr 172.21.15.123:0/3976606250 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:50:13.188 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:13.186+0000 7f7c22348640 1 --2- 172.21.15.123:0/3976606250 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f7c1c0aee40 0x7f7c1c0b3e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:50:13.188 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:13.186+0000 7f7c21b47640 1 -- 172.21.15.123:0/3976606250 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f7c1c0aee40 msgr2=0x7f7c1c0b3e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:50:13.188 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:13.186+0000 7f7c21b47640 1 --2- 172.21.15.123:0/3976606250 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f7c1c0aee40 0x7f7c1c0b3e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:13.188 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:13.186+0000 7f7c21b47640 1 -- 172.21.15.123:0/3976606250 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7c1c073390 msgr2=0x7f7c1c14f470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:50:13.188 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:13.186+0000 7f7c21346640 1 --2- 172.21.15.123:0/3976606250 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7c1c073390 0x7f7c1c14f470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:50:13.188 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:13.186+0000 7f7c21b47640 1 --2- 172.21.15.123:0/3976606250 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7c1c073390 0x7f7c1c14f470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:13.188 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:13.186+0000 7f7c21b47640 1 -- 172.21.15.123:0/3976606250 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7c1c157f60 con 0x7f7c1c070470 2024-09-16T15:50:13.188 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:13.186+0000 7f7c21b47640 1 --2- 172.21.15.123:0/3976606250 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7c1c070470 0x7f7c1c147860 secure :-1 s=READY pgs=1489 cs=0 l=1 rev1=1 crypto rx=0x7f7c180058d0 tx=0x7f7c1800dde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:50:13.188 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:13.186+0000 7f7c127fc640 1 -- 172.21.15.123:0/3976606250 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c18017070 con 0x7f7c1c070470 2024-09-16T15:50:13.188 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:13.186+0000 7f7c127fc640 1 -- 172.21.15.123:0/3976606250 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f7c18009cc0 con 0x7f7c1c070470 2024-09-16T15:50:13.188 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:13.186+0000 7f7c23dab640 1 -- 172.21.15.123:0/3976606250 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7c1c15a640 con 0x7f7c1c070470 2024-09-16T15:50:13.188 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:13.186+0000 7f7c127fc640 1 -- 172.21.15.123:0/3976606250 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c180106c0 con 0x7f7c1c070470 2024-09-16T15:50:13.188 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:13.186+0000 7f7c23dab640 1 -- 172.21.15.123:0/3976606250 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7c1c147da0 con 0x7f7c1c070470 2024-09-16T15:50:13.189 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:13.186+0000 7f7c23dab640 1 -- 172.21.15.123:0/3976606250 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7c1c15f230 con 0x7f7c1c070470 2024-09-16T15:50:13.189 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:13.190+0000 7f7c127fc640 1 -- 172.21.15.123:0/3976606250 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 22) ==== 97692+0+0 (secure 0 0 0) 0x7f7c1801a070 con 0x7f7c1c070470 2024-09-16T15:50:13.189 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:13.190+0000 7f7c127fc640 1 --2- 172.21.15.123:0/3976606250 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f7c00073580 0x7f7c00075a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:50:13.189 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:13.190+0000 7f7c21346640 1 --2- 172.21.15.123:0/3976606250 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f7c00073580 0x7f7c00075a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:50:13.190 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:13.190+0000 7f7c127fc640 1 -- 172.21.15.123:0/3976606250 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(135..135 src has 1..135) ==== 5737+0+0 (secure 0 0 0) 0x7f7c18094c50 con 0x7f7c1c070470 2024-09-16T15:50:13.190 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:13.190+0000 7f7c21346640 1 --2- 172.21.15.123:0/3976606250 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f7c00073580 0x7f7c00075a40 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f7c0c005950 tx=0x7f7c0c005bd0 comp rx=0 tx=0).ready entity=mgr.4150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:50:13.193 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:13.194+0000 7f7c127fc640 1 -- 172.21.15.123:0/3976606250 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f7c1805f3e0 con 0x7f7c1c070470 2024-09-16T15:50:13.381 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:13.382+0000 7f7c23dab640 1 -- 172.21.15.123:0/3976606250 --> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] -- mgr_command(tid 0: {"prefix": "fs subvolumegroup pin", "vol_name": "non_existent_subvolume", "group_name": "group_1", "pin_type": "export", "pin_setting": "0", "target": ["mon-mgr", ""]}) -- 0x7f7c1c1666d0 con 0x7f7c00073580 2024-09-16T15:50:13.384 INFO:tasks.ceph.mgr.y.smithi123.stderr:2024-09-16T15:50:13.382+0000 7fa9b36f4640 -1 mgr.server reply reply (2) No such file or directory FS 'non_existent_subvolume' not found 2024-09-16T15:50:13.384 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:13.386+0000 7f7c127fc640 1 -- 172.21.15.123:0/3976606250 <== mgr.4150 v2:172.21.15.123:6836/2464759676 1 ==== mgr_command_reply(tid 0: -2 FS 'non_existent_subvolume' not found) ==== 45+0+0 (secure 0 0 0) 0x7f7c1c1666d0 con 0x7f7c00073580 2024-09-16T15:50:13.384 INFO:teuthology.orchestra.run.smithi123.stderr:Error ENOENT: FS 'non_existent_subvolume' not found 2024-09-16T15:50:13.389 DEBUG:teuthology.orchestra.run:got remote process result: 2 2024-09-16T15:50:13.389 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolumegroup ls non_existent_subvolume 2024-09-16T15:50:13.516 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:13.518+0000 7f6b6be21640 1 -- 172.21.15.123:0/600986037 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6b640cd710 msgr2=0x7f6b640cc550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:50:13.517 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:13.518+0000 7f6b6be21640 1 --2- 172.21.15.123:0/600986037 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6b640cd710 0x7f6b640cc550 secure :-1 s=READY pgs=1491 cs=0 l=1 rev1=1 crypto rx=0x7f6b640ced60 tx=0x7f6b5c01cb50 comp rx=0 tx=0).stop 2024-09-16T15:50:13.517 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:13.518+0000 7f6b6be21640 1 -- 172.21.15.123:0/600986037 shutdown_connections 2024-09-16T15:50:13.517 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:13.518+0000 7f6b6be21640 1 --2- 172.21.15.123:0/600986037 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6b640cd710 0x7f6b640cc550 unknown :-1 s=CLOSED pgs=1491 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:13.517 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:13.518+0000 7f6b6be21640 1 --2- 172.21.15.123:0/600986037 >> v2:172.21.15.184:3300/0 conn(0x7f6b64152d60 0x7f6b640cf3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:13.517 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:13.518+0000 7f6b6be21640 1 -- 172.21.15.123:0/600986037 >> 172.21.15.123:0/600986037 conn(0x7f6b640b1710 msgr2=0x7f6b640b1b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:50:13.517 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:13.518+0000 7f6b6be21640 1 -- 172.21.15.123:0/600986037 shutdown_connections 2024-09-16T15:50:13.517 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:13.518+0000 7f6b6be21640 1 -- 172.21.15.123:0/600986037 wait complete. 2024-09-16T15:50:13.518 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:13.518+0000 7f6b6be21640 1 Processor -- start 2024-09-16T15:50:13.518 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:13.518+0000 7f6b6be21640 1 -- start start 2024-09-16T15:50:13.520 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:13.518+0000 7f6b6be21640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f6b640cd710 0x7f6b640739b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:50:13.520 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:13.518+0000 7f6b6be21640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6b64152980 0x7f6b6414ab30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:50:13.520 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:13.518+0000 7f6b6be21640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6b64152d60 0x7f6b64072d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:50:13.520 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:13.518+0000 7f6b6be21640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f6b6406fe20 con 0x7f6b64152980 2024-09-16T15:50:13.520 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:13.518+0000 7f6b6be21640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f6b6406fca0 con 0x7f6b64152d60 2024-09-16T15:50:13.520 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:13.518+0000 7f6b6be21640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f6b6406ffa0 con 0x7f6b640cd710 2024-09-16T15:50:13.520 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:13.518+0000 7f6b693bc640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6b64152980 0x7f6b6414ab30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:50:13.520 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:13.518+0000 7f6b693bc640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6b64152980 0x7f6b6414ab30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:37066/0 (socket says 172.21.15.123:37066) 2024-09-16T15:50:13.520 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:13.518+0000 7f6b693bc640 1 -- 172.21.15.123:0/423132434 learned_addr learned my addr 172.21.15.123:0/423132434 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:50:13.520 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:13.518+0000 7f6b69bbd640 1 --2- 172.21.15.123:0/423132434 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f6b640cd710 0x7f6b640739b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:50:13.520 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:13.518+0000 7f6b6a3be640 1 --2- 172.21.15.123:0/423132434 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6b64152d60 0x7f6b64072d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:50:13.521 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:13.518+0000 7f6b693bc640 1 -- 172.21.15.123:0/423132434 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f6b640cd710 msgr2=0x7f6b640739b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:50:13.521 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:13.518+0000 7f6b693bc640 1 --2- 172.21.15.123:0/423132434 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f6b640cd710 0x7f6b640739b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:13.521 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:13.518+0000 7f6b693bc640 1 -- 172.21.15.123:0/423132434 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6b64152d60 msgr2=0x7f6b64072d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:50:13.521 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:13.518+0000 7f6b693bc640 1 --2- 172.21.15.123:0/423132434 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6b64152d60 0x7f6b64072d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:13.521 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:13.518+0000 7f6b693bc640 1 -- 172.21.15.123:0/423132434 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6b6414b070 con 0x7f6b64152980 2024-09-16T15:50:13.521 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:13.518+0000 7f6b693bc640 1 --2- 172.21.15.123:0/423132434 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6b64152980 0x7f6b6414ab30 secure :-1 s=READY pgs=1492 cs=0 l=1 rev1=1 crypto rx=0x7f6b5400c9f0 tx=0x7f6b5400ceb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:50:13.521 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:13.518+0000 7f6b5a7fc640 1 -- 172.21.15.123:0/423132434 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b54007bb0 con 0x7f6b64152980 2024-09-16T15:50:13.521 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:13.518+0000 7f6b5a7fc640 1 -- 172.21.15.123:0/423132434 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f6b54007e40 con 0x7f6b64152980 2024-09-16T15:50:13.521 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:13.518+0000 7f6b6be21640 1 -- 172.21.15.123:0/423132434 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6b640742d0 con 0x7f6b64152980 2024-09-16T15:50:13.521 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:13.518+0000 7f6b5a7fc640 1 -- 172.21.15.123:0/423132434 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b5400f690 con 0x7f6b64152980 2024-09-16T15:50:13.521 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:13.518+0000 7f6b6be21640 1 -- 172.21.15.123:0/423132434 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6b64072590 con 0x7f6b64152980 2024-09-16T15:50:13.522 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:13.522+0000 7f6b6be21640 1 -- 172.21.15.123:0/423132434 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6b640cc550 con 0x7f6b64152980 2024-09-16T15:50:13.522 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:13.522+0000 7f6b5a7fc640 1 -- 172.21.15.123:0/423132434 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 22) ==== 97692+0+0 (secure 0 0 0) 0x7f6b5400f830 con 0x7f6b64152980 2024-09-16T15:50:13.522 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:13.522+0000 7f6b5a7fc640 1 --2- 172.21.15.123:0/423132434 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f6b48073580 0x7f6b48075a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:50:13.522 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:13.522+0000 7f6b69bbd640 1 --2- 172.21.15.123:0/423132434 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f6b48073580 0x7f6b48075a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:50:13.523 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:13.522+0000 7f6b5a7fc640 1 -- 172.21.15.123:0/423132434 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(135..135 src has 1..135) ==== 5737+0+0 (secure 0 0 0) 0x7f6b54095390 con 0x7f6b64152980 2024-09-16T15:50:13.523 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:13.522+0000 7f6b69bbd640 1 --2- 172.21.15.123:0/423132434 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f6b48073580 0x7f6b48075a40 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f6b600016e0 tx=0x7f6b60009040 comp rx=0 tx=0).ready entity=mgr.4150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:50:13.526 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:13.526+0000 7f6b5a7fc640 1 -- 172.21.15.123:0/423132434 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f6b54014030 con 0x7f6b64152980 2024-09-16T15:50:13.723 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:13.722+0000 7f6b6be21640 1 -- 172.21.15.123:0/423132434 --> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] -- mgr_command(tid 0: {"prefix": "fs subvolumegroup ls", "vol_name": "non_existent_subvolume", "target": ["mon-mgr", ""]}) -- 0x7f6b640b4590 con 0x7f6b48073580 2024-09-16T15:50:13.725 INFO:tasks.ceph.mgr.y.smithi123.stderr:2024-09-16T15:50:13.726+0000 7fa9b36f4640 -1 mgr.server reply reply (2) No such file or directory FS 'non_existent_subvolume' not found 2024-09-16T15:50:13.726 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:13.726+0000 7f6b5a7fc640 1 -- 172.21.15.123:0/423132434 <== mgr.4150 v2:172.21.15.123:6836/2464759676 1 ==== mgr_command_reply(tid 0: -2 FS 'non_existent_subvolume' not found) ==== 45+0+0 (secure 0 0 0) 0x7f6b640b4590 con 0x7f6b48073580 2024-09-16T15:50:13.726 INFO:teuthology.orchestra.run.smithi123.stderr:Error ENOENT: FS 'non_existent_subvolume' not found 2024-09-16T15:50:13.734 DEBUG:teuthology.orchestra.run:got remote process result: 2 2024-09-16T15:50:13.734 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolumegroup snapshot create non_existent_subvolume group_1 snapshot_1 2024-09-16T15:50:13.865 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:13.866+0000 7fe15a144640 1 -- 172.21.15.123:0/3633533696 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe15415dc40 msgr2=0x7fe15415e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:50:13.865 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:13.866+0000 7fe15a144640 1 --2- 172.21.15.123:0/3633533696 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe15415dc40 0x7fe15415e090 secure :-1 s=READY pgs=1494 cs=0 l=1 rev1=1 crypto rx=0x7fe148009900 tx=0x7fe14801f0d0 comp rx=0 tx=0).stop 2024-09-16T15:50:13.865 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:13.866+0000 7fe15a144640 1 -- 172.21.15.123:0/3633533696 shutdown_connections 2024-09-16T15:50:13.865 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:13.866+0000 7fe15a144640 1 --2- 172.21.15.123:0/3633533696 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fe15406f6c0 0x7fe15406fb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:13.865 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:13.866+0000 7fe15a144640 1 --2- 172.21.15.123:0/3633533696 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe15415dc40 0x7fe15415e090 unknown :-1 s=CLOSED pgs=1494 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:13.865 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:13.866+0000 7fe15a144640 1 -- 172.21.15.123:0/3633533696 >> 172.21.15.123:0/3633533696 conn(0x7fe1540b1630 msgr2=0x7fe1540b1a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:50:13.865 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:13.866+0000 7fe15a144640 1 -- 172.21.15.123:0/3633533696 shutdown_connections 2024-09-16T15:50:13.866 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:13.866+0000 7fe15a144640 1 -- 172.21.15.123:0/3633533696 wait complete. 2024-09-16T15:50:13.866 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:13.866+0000 7fe15a144640 1 Processor -- start 2024-09-16T15:50:13.866 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:13.866+0000 7fe15a144640 1 -- start start 2024-09-16T15:50:13.868 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:13.866+0000 7fe15a144640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe15406f6c0 0x7fe1541f4900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:50:13.868 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:13.866+0000 7fe15a144640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe15415a1b0 0x7fe154155f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:50:13.868 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:13.866+0000 7fe15a144640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fe15415dc40 0x7fe1540cd340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:50:13.868 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:13.866+0000 7fe15a144640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fe15406e710 con 0x7fe15406f6c0 2024-09-16T15:50:13.869 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:13.866+0000 7fe15a144640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fe15406e590 con 0x7fe15415a1b0 2024-09-16T15:50:13.869 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:13.866+0000 7fe15a144640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fe15406e890 con 0x7fe15415dc40 2024-09-16T15:50:13.869 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:13.866+0000 7fe1537fe640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe15406f6c0 0x7fe1541f4900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:50:13.869 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:13.866+0000 7fe1537fe640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe15406f6c0 0x7fe1541f4900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:37082/0 (socket says 172.21.15.123:37082) 2024-09-16T15:50:13.869 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:13.866+0000 7fe1537fe640 1 -- 172.21.15.123:0/2110771908 learned_addr learned my addr 172.21.15.123:0/2110771908 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:50:13.869 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:13.866+0000 7fe1537fe640 1 -- 172.21.15.123:0/2110771908 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fe15415dc40 msgr2=0x7fe1540cd340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:50:13.869 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:13.866+0000 7fe153fff640 1 --2- 172.21.15.123:0/2110771908 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fe15415dc40 0x7fe1540cd340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:50:13.869 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:13.866+0000 7fe1537fe640 1 --2- 172.21.15.123:0/2110771908 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fe15415dc40 0x7fe1540cd340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:13.869 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:13.866+0000 7fe1537fe640 1 -- 172.21.15.123:0/2110771908 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe15415a1b0 msgr2=0x7fe154155f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:50:13.869 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:13.866+0000 7fe1537fe640 1 --2- 172.21.15.123:0/2110771908 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe15415a1b0 0x7fe154155f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:13.869 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:13.866+0000 7fe1537fe640 1 -- 172.21.15.123:0/2110771908 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe1541f5150 con 0x7fe15406f6c0 2024-09-16T15:50:13.869 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:13.866+0000 7fe153fff640 1 --2- 172.21.15.123:0/2110771908 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fe15415dc40 0x7fe1540cd340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T15:50:13.869 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:13.866+0000 7fe1537fe640 1 --2- 172.21.15.123:0/2110771908 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe15406f6c0 0x7fe1541f4900 secure :-1 s=READY pgs=1495 cs=0 l=1 rev1=1 crypto rx=0x7fe14400c500 tx=0x7fe14400c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:50:13.869 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:13.866+0000 7fe13bfff640 1 -- 172.21.15.123:0/2110771908 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe144017070 con 0x7fe15406f6c0 2024-09-16T15:50:13.870 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:13.866+0000 7fe13bfff640 1 -- 172.21.15.123:0/2110771908 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fe1440053c0 con 0x7fe15406f6c0 2024-09-16T15:50:13.870 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:13.866+0000 7fe15a144640 1 -- 172.21.15.123:0/2110771908 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe1541f5430 con 0x7fe15406f6c0 2024-09-16T15:50:13.870 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:13.866+0000 7fe13bfff640 1 -- 172.21.15.123:0/2110771908 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe144012890 con 0x7fe15406f6c0 2024-09-16T15:50:13.870 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:13.866+0000 7fe15a144640 1 -- 172.21.15.123:0/2110771908 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe1540ccb40 con 0x7fe15406f6c0 2024-09-16T15:50:13.870 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:13.870+0000 7fe15a144640 1 -- 172.21.15.123:0/2110771908 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe15415e090 con 0x7fe15406f6c0 2024-09-16T15:50:13.871 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:13.870+0000 7fe13bfff640 1 -- 172.21.15.123:0/2110771908 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 22) ==== 97692+0+0 (secure 0 0 0) 0x7fe14401a070 con 0x7fe15406f6c0 2024-09-16T15:50:13.871 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:13.870+0000 7fe13bfff640 1 --2- 172.21.15.123:0/2110771908 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fe140073580 0x7fe140075a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:50:13.871 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:13.870+0000 7fe152ffd640 1 --2- 172.21.15.123:0/2110771908 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fe140073580 0x7fe140075a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:50:13.871 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:13.870+0000 7fe13bfff640 1 -- 172.21.15.123:0/2110771908 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(135..135 src has 1..135) ==== 5737+0+0 (secure 0 0 0) 0x7fe144093e60 con 0x7fe15406f6c0 2024-09-16T15:50:13.871 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:13.870+0000 7fe152ffd640 1 --2- 172.21.15.123:0/2110771908 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fe140073580 0x7fe140075a40 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7fe1480b2000 tx=0x7fe14801be70 comp rx=0 tx=0).ready entity=mgr.4150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:50:13.874 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:13.874+0000 7fe13bfff640 1 -- 172.21.15.123:0/2110771908 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7fe14405e570 con 0x7fe15406f6c0 2024-09-16T15:50:14.059 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:14.058+0000 7fe15a144640 1 -- 172.21.15.123:0/2110771908 --> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] -- mgr_command(tid 0: {"prefix": "fs subvolumegroup snapshot create", "vol_name": "non_existent_subvolume", "group_name": "group_1", "snap_name": "snapshot_1", "target": ["mon-mgr", ""]}) -- 0x7fe1540ccf60 con 0x7fe140073580 2024-09-16T15:50:14.061 INFO:tasks.ceph.mgr.y.smithi123.stderr:2024-09-16T15:50:14.062+0000 7fa9b36f4640 -1 mgr.server reply reply (2) No such file or directory FS 'non_existent_subvolume' not found 2024-09-16T15:50:14.061 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:14.062+0000 7fe13bfff640 1 -- 172.21.15.123:0/2110771908 <== mgr.4150 v2:172.21.15.123:6836/2464759676 1 ==== mgr_command_reply(tid 0: -2 FS 'non_existent_subvolume' not found) ==== 45+0+0 (secure 0 0 0) 0x7fe1540ccf60 con 0x7fe140073580 2024-09-16T15:50:14.061 INFO:teuthology.orchestra.run.smithi123.stderr:Error ENOENT: FS 'non_existent_subvolume' not found 2024-09-16T15:50:14.066 DEBUG:teuthology.orchestra.run:got remote process result: 2 2024-09-16T15:50:14.067 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolumegroup snapshot rm non_existent_subvolume group_1 snapshot_1 2024-09-16T15:50:14.198 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:14.198+0000 7f9ff60cd640 1 -- 172.21.15.123:0/1565384077 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9ff0070470 msgr2=0x7f9ff0157d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:50:14.198 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:14.198+0000 7f9ff60cd640 1 --2- 172.21.15.123:0/1565384077 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9ff0070470 0x7f9ff0157d00 secure :-1 s=READY pgs=1496 cs=0 l=1 rev1=1 crypto rx=0x7f9ff0074d80 tx=0x7f9fe801c8c0 comp rx=0 tx=0).stop 2024-09-16T15:50:14.198 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:14.198+0000 7f9ff60cd640 1 -- 172.21.15.123:0/1565384077 shutdown_connections 2024-09-16T15:50:14.198 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:14.198+0000 7f9ff60cd640 1 --2- 172.21.15.123:0/1565384077 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9ff0070470 0x7f9ff0157d00 unknown :-1 s=CLOSED pgs=1496 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:14.198 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:14.198+0000 7f9ff60cd640 1 --2- 172.21.15.123:0/1565384077 >> v2:172.21.15.184:3300/0 conn(0x7f9ff0073390 0x7f9ff006ff30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:14.199 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:14.198+0000 7f9ff60cd640 1 -- 172.21.15.123:0/1565384077 >> 172.21.15.123:0/1565384077 conn(0x7f9ff00b1630 msgr2=0x7f9ff00b1a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:50:14.199 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:14.198+0000 7f9ff60cd640 1 -- 172.21.15.123:0/1565384077 shutdown_connections 2024-09-16T15:50:14.199 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:14.198+0000 7f9ff60cd640 1 -- 172.21.15.123:0/1565384077 wait complete. 2024-09-16T15:50:14.199 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:14.198+0000 7f9ff60cd640 1 Processor -- start 2024-09-16T15:50:14.199 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:14.198+0000 7f9ff60cd640 1 -- start start 2024-09-16T15:50:14.201 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:14.202+0000 7f9ff60cd640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9ff0070470 0x7f9ff01661a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:50:14.201 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:14.202+0000 7f9ff60cd640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f9ff0073390 0x7f9ff014f9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:50:14.201 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:14.202+0000 7f9ff60cd640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f9ff00aee40 0x7f9ff0154930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:50:14.201 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:14.202+0000 7f9ff60cd640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f9ff00b3a10 con 0x7f9ff0070470 2024-09-16T15:50:14.201 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:14.202+0000 7f9ff60cd640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f9ff00b3890 con 0x7f9ff0073390 2024-09-16T15:50:14.201 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:14.202+0000 7f9ff60cd640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f9ff00b3b90 con 0x7f9ff00aee40 2024-09-16T15:50:14.201 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:14.202+0000 7f9fef7fe640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9ff0070470 0x7f9ff01661a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:50:14.202 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:14.202+0000 7f9fef7fe640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9ff0070470 0x7f9ff01661a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:37090/0 (socket says 172.21.15.123:37090) 2024-09-16T15:50:14.202 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:14.202+0000 7f9fef7fe640 1 -- 172.21.15.123:0/750221395 learned_addr learned my addr 172.21.15.123:0/750221395 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:50:14.202 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:14.202+0000 7f9fef7fe640 1 -- 172.21.15.123:0/750221395 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f9ff00aee40 msgr2=0x7f9ff0154930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:50:14.202 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:14.202+0000 7f9fef7fe640 1 --2- 172.21.15.123:0/750221395 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f9ff00aee40 0x7f9ff0154930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:14.202 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:14.202+0000 7f9fef7fe640 1 -- 172.21.15.123:0/750221395 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f9ff0073390 msgr2=0x7f9ff014f9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:50:14.202 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:14.202+0000 7f9fef7fe640 1 --2- 172.21.15.123:0/750221395 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f9ff0073390 0x7f9ff014f9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:14.202 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:14.202+0000 7f9fef7fe640 1 -- 172.21.15.123:0/750221395 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9ff01666e0 con 0x7f9ff0070470 2024-09-16T15:50:14.202 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:14.202+0000 7f9fef7fe640 1 --2- 172.21.15.123:0/750221395 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9ff0070470 0x7f9ff01661a0 secure :-1 s=READY pgs=1497 cs=0 l=1 rev1=1 crypto rx=0x7f9fe000ba50 tx=0x7f9fe000bf10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:50:14.202 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:14.202+0000 7f9fe7fff640 1 -- 172.21.15.123:0/750221395 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9fe0019070 con 0x7f9ff0070470 2024-09-16T15:50:14.202 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:14.202+0000 7f9fe7fff640 1 -- 172.21.15.123:0/750221395 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f9fe000cd10 con 0x7f9ff0070470 2024-09-16T15:50:14.202 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:14.202+0000 7f9ff60cd640 1 -- 172.21.15.123:0/750221395 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9ff0149900 con 0x7f9ff0070470 2024-09-16T15:50:14.202 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:14.202+0000 7f9fe7fff640 1 -- 172.21.15.123:0/750221395 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9fe00146a0 con 0x7f9ff0070470 2024-09-16T15:50:14.202 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:14.202+0000 7f9ff60cd640 1 -- 172.21.15.123:0/750221395 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9ff0149ec0 con 0x7f9ff0070470 2024-09-16T15:50:14.203 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:14.202+0000 7f9ff60cd640 1 -- 172.21.15.123:0/750221395 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9ff00b4100 con 0x7f9ff0070470 2024-09-16T15:50:14.203 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:14.202+0000 7f9fe7fff640 1 -- 172.21.15.123:0/750221395 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 22) ==== 97692+0+0 (secure 0 0 0) 0x7f9fe001c070 con 0x7f9ff0070470 2024-09-16T15:50:14.203 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:14.202+0000 7f9fe7fff640 1 --2- 172.21.15.123:0/750221395 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f9fd0073580 0x7f9fd0075a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:50:14.203 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:14.202+0000 7f9fe7fff640 1 -- 172.21.15.123:0/750221395 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(135..135 src has 1..135) ==== 5737+0+0 (secure 0 0 0) 0x7f9fe00945d0 con 0x7f9ff0070470 2024-09-16T15:50:14.204 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:14.202+0000 7f9feeffd640 1 --2- 172.21.15.123:0/750221395 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f9fd0073580 0x7f9fd0075a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:50:14.204 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:14.202+0000 7f9feeffd640 1 --2- 172.21.15.123:0/750221395 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f9fd0073580 0x7f9fd0075a40 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f9ff01f91f0 tx=0x7f9fd8005c00 comp rx=0 tx=0).ready entity=mgr.4150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:50:14.207 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:14.206+0000 7f9fe7fff640 1 -- 172.21.15.123:0/750221395 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f9fe005e910 con 0x7f9ff0070470 2024-09-16T15:50:14.392 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:14.394+0000 7f9ff60cd640 1 -- 172.21.15.123:0/750221395 --> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] -- mgr_command(tid 0: {"prefix": "fs subvolumegroup snapshot rm", "vol_name": "non_existent_subvolume", "group_name": "group_1", "snap_name": "snapshot_1", "target": ["mon-mgr", ""]}) -- 0x7f9ff00cf7b0 con 0x7f9fd0073580 2024-09-16T15:50:14.395 INFO:tasks.ceph.mgr.y.smithi123.stderr:2024-09-16T15:50:14.394+0000 7fa9b36f4640 -1 mgr.server reply reply (2) No such file or directory FS 'non_existent_subvolume' not found 2024-09-16T15:50:14.396 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:14.394+0000 7f9fe7fff640 1 -- 172.21.15.123:0/750221395 <== mgr.4150 v2:172.21.15.123:6836/2464759676 1 ==== mgr_command_reply(tid 0: -2 FS 'non_existent_subvolume' not found) ==== 45+0+0 (secure 0 0 0) 0x7f9ff00cf7b0 con 0x7f9fd0073580 2024-09-16T15:50:14.396 INFO:teuthology.orchestra.run.smithi123.stderr:Error ENOENT: FS 'non_existent_subvolume' not found 2024-09-16T15:50:14.401 DEBUG:teuthology.orchestra.run:got remote process result: 2 2024-09-16T15:50:14.401 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolumegroup snapshot ls non_existent_subvolume group_1 2024-09-16T15:50:14.534 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:14.534+0000 7f89c8d2b640 1 -- 172.21.15.123:0/4050962883 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f89c406fc20 msgr2=0x7f89c406cbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:50:14.534 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:14.534+0000 7f89c8d2b640 1 --2- 172.21.15.123:0/4050962883 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f89c406fc20 0x7f89c406cbf0 secure :-1 s=READY pgs=1498 cs=0 l=1 rev1=1 crypto rx=0x7f89c4073630 tx=0x7f89bc01c8c0 comp rx=0 tx=0).stop 2024-09-16T15:50:14.534 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:14.534+0000 7f89c8d2b640 1 -- 172.21.15.123:0/4050962883 shutdown_connections 2024-09-16T15:50:14.534 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:14.534+0000 7f89c8d2b640 1 --2- 172.21.15.123:0/4050962883 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f89c406fc20 0x7f89c406cbf0 unknown :-1 s=CLOSED pgs=1498 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:14.534 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:14.534+0000 7f89c8d2b640 1 --2- 172.21.15.123:0/4050962883 >> v2:172.21.15.184:3300/0 conn(0x7f89c4071cf0 0x7f89c406f6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:14.535 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:14.534+0000 7f89c8d2b640 1 -- 172.21.15.123:0/4050962883 >> 172.21.15.123:0/4050962883 conn(0x7f89c40d0ba0 msgr2=0x7f89c40d0fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:50:14.535 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:14.534+0000 7f89c8d2b640 1 -- 172.21.15.123:0/4050962883 shutdown_connections 2024-09-16T15:50:14.535 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:14.534+0000 7f89c8d2b640 1 -- 172.21.15.123:0/4050962883 wait complete. 2024-09-16T15:50:14.535 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:14.534+0000 7f89c8d2b640 1 Processor -- start 2024-09-16T15:50:14.535 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:14.534+0000 7f89c8d2b640 1 -- start start 2024-09-16T15:50:14.537 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:14.538+0000 7f89c8d2b640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f89c406fc20 0x7f89c414fa20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:50:14.537 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:14.538+0000 7f89c8d2b640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f89c4071cf0 0x7f89c415aa60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:50:14.537 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:14.538+0000 7f89c8d2b640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f89c415bc20 0x7f89c40b0700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:50:14.537 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:14.538+0000 7f89c8d2b640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f89c4169030 con 0x7f89c415bc20 2024-09-16T15:50:14.538 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:14.538+0000 7f89c8d2b640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f89c4168eb0 con 0x7f89c406fc20 2024-09-16T15:50:14.538 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:14.538+0000 7f89c8d2b640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f89c41691b0 con 0x7f89c4071cf0 2024-09-16T15:50:14.538 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:14.538+0000 7f89c2d9d640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f89c415bc20 0x7f89c40b0700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:50:14.538 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:14.538+0000 7f89c2d9d640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f89c415bc20 0x7f89c40b0700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:37104/0 (socket says 172.21.15.123:37104) 2024-09-16T15:50:14.538 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:14.538+0000 7f89c2d9d640 1 -- 172.21.15.123:0/1784260801 learned_addr learned my addr 172.21.15.123:0/1784260801 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:50:14.538 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:14.538+0000 7f89c1d9b640 1 --2- 172.21.15.123:0/1784260801 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f89c4071cf0 0x7f89c415aa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:50:14.538 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:14.538+0000 7f89c259c640 1 --2- 172.21.15.123:0/1784260801 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f89c406fc20 0x7f89c414fa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:50:14.538 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:14.538+0000 7f89c2d9d640 1 -- 172.21.15.123:0/1784260801 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f89c4071cf0 msgr2=0x7f89c415aa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:50:14.538 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:14.538+0000 7f89c2d9d640 1 --2- 172.21.15.123:0/1784260801 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f89c4071cf0 0x7f89c415aa60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:14.538 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:14.538+0000 7f89c2d9d640 1 -- 172.21.15.123:0/1784260801 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f89c406fc20 msgr2=0x7f89c414fa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:50:14.538 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:14.538+0000 7f89c2d9d640 1 --2- 172.21.15.123:0/1784260801 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f89c406fc20 0x7f89c414fa20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:14.538 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:14.538+0000 7f89c2d9d640 1 -- 172.21.15.123:0/1784260801 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f89c40b1410 con 0x7f89c415bc20 2024-09-16T15:50:14.539 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:14.538+0000 7f89c259c640 1 --2- 172.21.15.123:0/1784260801 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f89c406fc20 0x7f89c414fa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T15:50:14.539 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:14.538+0000 7f89c1d9b640 1 --2- 172.21.15.123:0/1784260801 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f89c4071cf0 0x7f89c415aa60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T15:50:14.539 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:14.538+0000 7f89c2d9d640 1 --2- 172.21.15.123:0/1784260801 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f89c415bc20 0x7f89c40b0700 secure :-1 s=READY pgs=1499 cs=0 l=1 rev1=1 crypto rx=0x7f89bc0b0040 tx=0x7f89bc0a56a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:50:14.539 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:14.538+0000 7f89baffd640 1 -- 172.21.15.123:0/1784260801 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89bc0b2040 con 0x7f89c415bc20 2024-09-16T15:50:14.539 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:14.538+0000 7f89baffd640 1 -- 172.21.15.123:0/1784260801 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f89bc004430 con 0x7f89c415bc20 2024-09-16T15:50:14.539 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:14.538+0000 7f89c8d2b640 1 -- 172.21.15.123:0/1784260801 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f89c40b3fd0 con 0x7f89c415bc20 2024-09-16T15:50:14.539 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:14.538+0000 7f89baffd640 1 -- 172.21.15.123:0/1784260801 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89bc003860 con 0x7f89c415bc20 2024-09-16T15:50:14.539 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:14.538+0000 7f89c8d2b640 1 -- 172.21.15.123:0/1784260801 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f89c40b4590 con 0x7f89c415bc20 2024-09-16T15:50:14.540 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:14.538+0000 7f89c8d2b640 1 -- 172.21.15.123:0/1784260801 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f89c406cbf0 con 0x7f89c415bc20 2024-09-16T15:50:14.540 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:14.538+0000 7f89baffd640 1 -- 172.21.15.123:0/1784260801 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 22) ==== 97692+0+0 (secure 0 0 0) 0x7f89bc0bf060 con 0x7f89c415bc20 2024-09-16T15:50:14.540 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:14.538+0000 7f89baffd640 1 --2- 172.21.15.123:0/1784260801 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f89a4073580 0x7f89a4075a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:50:14.540 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:14.538+0000 7f89baffd640 1 -- 172.21.15.123:0/1784260801 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(135..135 src has 1..135) ==== 5737+0+0 (secure 0 0 0) 0x7f89bc12ec00 con 0x7f89c415bc20 2024-09-16T15:50:14.540 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:14.538+0000 7f89c259c640 1 --2- 172.21.15.123:0/1784260801 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f89a4073580 0x7f89a4075a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:50:14.540 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:14.538+0000 7f89c259c640 1 --2- 172.21.15.123:0/1784260801 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f89a4073580 0x7f89a4075a40 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f89c40b1a30 tx=0x7f89b40073e0 comp rx=0 tx=0).ready entity=mgr.4150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:50:14.542 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:14.542+0000 7f89baffd640 1 -- 172.21.15.123:0/1784260801 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f89bc0164b0 con 0x7f89c415bc20 2024-09-16T15:50:14.725 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:14.726+0000 7f89c8d2b640 1 -- 172.21.15.123:0/1784260801 --> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] -- mgr_command(tid 0: {"prefix": "fs subvolumegroup snapshot ls", "vol_name": "non_existent_subvolume", "group_name": "group_1", "target": ["mon-mgr", ""]}) -- 0x7f89c40cd030 con 0x7f89a4073580 2024-09-16T15:50:14.728 INFO:tasks.ceph.mgr.y.smithi123.stderr:2024-09-16T15:50:14.730+0000 7fa9b36f4640 -1 mgr.server reply reply (2) No such file or directory FS 'non_existent_subvolume' not found 2024-09-16T15:50:14.728 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:14.730+0000 7f89baffd640 1 -- 172.21.15.123:0/1784260801 <== mgr.4150 v2:172.21.15.123:6836/2464759676 1 ==== mgr_command_reply(tid 0: -2 FS 'non_existent_subvolume' not found) ==== 45+0+0 (secure 0 0 0) 0x7f89c40cd030 con 0x7f89a4073580 2024-09-16T15:50:14.728 INFO:teuthology.orchestra.run.smithi123.stderr:Error ENOENT: FS 'non_existent_subvolume' not found 2024-09-16T15:50:14.734 DEBUG:teuthology.orchestra.run:got remote process result: 2 2024-09-16T15:50:14.735 INFO:teuthology.nuke:Clearing teuthology firewall rules... 2024-09-16T15:50:14.735 DEBUG:teuthology.orchestra.run.smithi123:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-09-16T15:50:14.813 DEBUG:teuthology.orchestra.run.smithi184:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-09-16T15:50:14.902 INFO:teuthology.nuke:Cleared teuthology firewall rules. 2024-09-16T15:50:14.902 INFO:tasks.cephfs.mount:Terminating background process 2024-09-16T15:50:14.902 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2024-09-16T15:50:14.902 DEBUG:teuthology.orchestra.run.smithi123:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-16T15:50:14.908 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2024-09-16T15:50:14.908 DEBUG:teuthology.orchestra.run.smithi123:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-16T15:50:14.957 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-09-16T15:50:15.131 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:15.130+0000 7f578f5f4640 1 -- 172.21.15.123:0/3128827976 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5788158120 msgr2=0x7f57880ca320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:50:15.131 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:15.130+0000 7f578f5f4640 1 --2- 172.21.15.123:0/3128827976 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5788158120 0x7f57880ca320 secure :-1 s=READY pgs=1501 cs=0 l=1 rev1=1 crypto rx=0x7f5788072250 tx=0x7f578401c6e0 comp rx=0 tx=0).stop 2024-09-16T15:50:15.131 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:15.130+0000 7f578f5f4640 1 -- 172.21.15.123:0/3128827976 shutdown_connections 2024-09-16T15:50:15.131 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:15.130+0000 7f578f5f4640 1 --2- 172.21.15.123:0/3128827976 >> v2:172.21.15.184:3300/0 conn(0x7f57880cf5a0 0x7f578806eee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:15.131 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:15.130+0000 7f578f5f4640 1 --2- 172.21.15.123:0/3128827976 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5788158120 0x7f57880ca320 unknown :-1 s=CLOSED pgs=1501 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:15.132 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:15.130+0000 7f578f5f4640 1 -- 172.21.15.123:0/3128827976 >> 172.21.15.123:0/3128827976 conn(0x7f57880cfb10 msgr2=0x7f57880cff20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:50:15.132 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:15.130+0000 7f578f5f4640 1 -- 172.21.15.123:0/3128827976 shutdown_connections 2024-09-16T15:50:15.132 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:15.130+0000 7f578f5f4640 1 -- 172.21.15.123:0/3128827976 wait complete. 2024-09-16T15:50:15.132 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:15.134+0000 7f578f5f4640 1 Processor -- start 2024-09-16T15:50:15.132 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:15.134+0000 7f578f5f4640 1 -- start start 2024-09-16T15:50:15.134 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:15.134+0000 7f578f5f4640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f57880cf5a0 0x7f57881f8b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:50:15.134 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:15.134+0000 7f578f5f4640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f5788158120 0x7f57881565b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:50:15.134 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:15.134+0000 7f578f5f4640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f578815ec40 0x7f57881523e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:50:15.134 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:15.134+0000 7f578f5f4640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f578806ebd0 con 0x7f57880cf5a0 2024-09-16T15:50:15.134 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:15.134+0000 7f578f5f4640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f578806ea50 con 0x7f578815ec40 2024-09-16T15:50:15.135 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:15.134+0000 7f578f5f4640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f57880b2e30 con 0x7f5788158120 2024-09-16T15:50:15.135 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:15.134+0000 7f578d390640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f57880cf5a0 0x7f57881f8b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:50:15.135 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:15.134+0000 7f578d390640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f57880cf5a0 0x7f57881f8b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:37126/0 (socket says 172.21.15.123:37126) 2024-09-16T15:50:15.135 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:15.134+0000 7f578d390640 1 -- 172.21.15.123:0/1356131042 learned_addr learned my addr 172.21.15.123:0/1356131042 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:50:15.135 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:15.134+0000 7f578db91640 1 --2- 172.21.15.123:0/1356131042 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f578815ec40 0x7f57881523e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:50:15.135 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:15.134+0000 7f578cb8f640 1 --2- 172.21.15.123:0/1356131042 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f5788158120 0x7f57881565b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:50:15.135 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:15.134+0000 7f578d390640 1 -- 172.21.15.123:0/1356131042 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f5788158120 msgr2=0x7f57881565b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:50:15.135 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:15.134+0000 7f578d390640 1 --2- 172.21.15.123:0/1356131042 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f5788158120 0x7f57881565b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:15.135 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:15.134+0000 7f578d390640 1 -- 172.21.15.123:0/1356131042 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f578815ec40 msgr2=0x7f57881523e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:50:15.135 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:15.134+0000 7f578d390640 1 --2- 172.21.15.123:0/1356131042 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f578815ec40 0x7f57881523e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:15.135 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:15.134+0000 7f578d390640 1 -- 172.21.15.123:0/1356131042 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f57880b3110 con 0x7f57880cf5a0 2024-09-16T15:50:15.135 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:15.134+0000 7f578db91640 1 --2- 172.21.15.123:0/1356131042 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f578815ec40 0x7f57881523e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T15:50:15.135 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:15.134+0000 7f578cb8f640 1 --2- 172.21.15.123:0/1356131042 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f5788158120 0x7f57881565b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T15:50:15.135 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:15.134+0000 7f578d390640 1 --2- 172.21.15.123:0/1356131042 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f57880cf5a0 0x7f57881f8b80 secure :-1 s=READY pgs=1502 cs=0 l=1 rev1=1 crypto rx=0x7f578401a310 tx=0x7f57840a8600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:50:15.135 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:15.134+0000 7f577dffb640 1 -- 172.21.15.123:0/1356131042 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57840bf070 con 0x7f57880cf5a0 2024-09-16T15:50:15.136 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:15.134+0000 7f577dffb640 1 -- 172.21.15.123:0/1356131042 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f5784005100 con 0x7f57880cf5a0 2024-09-16T15:50:15.136 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:15.134+0000 7f578f5f4640 1 -- 172.21.15.123:0/1356131042 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f57880b33f0 con 0x7f57880cf5a0 2024-09-16T15:50:15.136 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:15.134+0000 7f577dffb640 1 -- 172.21.15.123:0/1356131042 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5784005530 con 0x7f57880cf5a0 2024-09-16T15:50:15.136 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:15.134+0000 7f578f5f4640 1 -- 172.21.15.123:0/1356131042 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f57880b39b0 con 0x7f57880cf5a0 2024-09-16T15:50:15.137 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:15.134+0000 7f578f5f4640 1 -- 172.21.15.123:0/1356131042 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f57880b3bb0 con 0x7f57880cf5a0 2024-09-16T15:50:15.137 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:15.134+0000 7f577dffb640 1 -- 172.21.15.123:0/1356131042 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 22) ==== 97692+0+0 (secure 0 0 0) 0x7f57840c2070 con 0x7f57880cf5a0 2024-09-16T15:50:15.137 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:15.138+0000 7f577dffb640 1 --2- 172.21.15.123:0/1356131042 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f576c073580 0x7f576c075a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:50:15.137 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:15.138+0000 7f578cb8f640 1 --2- 172.21.15.123:0/1356131042 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f576c073580 0x7f576c075a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:50:15.137 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:15.138+0000 7f577dffb640 1 -- 172.21.15.123:0/1356131042 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(135..135 src has 1..135) ==== 5737+0+0 (secure 0 0 0) 0x7f578412ec40 con 0x7f57880cf5a0 2024-09-16T15:50:15.137 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:15.138+0000 7f578cb8f640 1 --2- 172.21.15.123:0/1356131042 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f576c073580 0x7f576c075a40 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f5778005fd0 tx=0x7f5778005d40 comp rx=0 tx=0).ready entity=mgr.4150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:50:15.141 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:15.142+0000 7f577dffb640 1 -- 172.21.15.123:0/1356131042 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f57840ac030 con 0x7f57880cf5a0 2024-09-16T15:50:15.323 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:15.322+0000 7f578f5f4640 1 -- 172.21.15.123:0/1356131042 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f57880d06c0 con 0x7f57880cf5a0 2024-09-16T15:50:15.323 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:15.322+0000 7f577dffb640 1 -- 172.21.15.123:0/1356131042 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v135) ==== 80+0+0 (secure 0 0 0) 0x7f57840163f0 con 0x7f57880cf5a0 2024-09-16T15:50:15.324 INFO:teuthology.orchestra.run.smithi123.stderr:listed 0 entries 2024-09-16T15:50:15.326 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:15.326+0000 7f578f5f4640 1 -- 172.21.15.123:0/1356131042 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f576c073580 msgr2=0x7f576c075a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:50:15.326 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:15.326+0000 7f578f5f4640 1 --2- 172.21.15.123:0/1356131042 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f576c073580 0x7f576c075a40 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f5778005fd0 tx=0x7f5778005d40 comp rx=0 tx=0).stop 2024-09-16T15:50:15.326 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:15.326+0000 7f578f5f4640 1 -- 172.21.15.123:0/1356131042 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f57880cf5a0 msgr2=0x7f57881f8b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:50:15.326 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:15.326+0000 7f578f5f4640 1 --2- 172.21.15.123:0/1356131042 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f57880cf5a0 0x7f57881f8b80 secure :-1 s=READY pgs=1502 cs=0 l=1 rev1=1 crypto rx=0x7f578401a310 tx=0x7f57840a8600 comp rx=0 tx=0).stop 2024-09-16T15:50:15.326 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:15.326+0000 7f578f5f4640 1 -- 172.21.15.123:0/1356131042 shutdown_connections 2024-09-16T15:50:15.326 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:15.326+0000 7f578f5f4640 1 --2- 172.21.15.123:0/1356131042 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f576c073580 0x7f576c075a40 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:15.326 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:15.326+0000 7f578f5f4640 1 --2- 172.21.15.123:0/1356131042 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f578815ec40 0x7f57881523e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:15.326 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:15.326+0000 7f578f5f4640 1 --2- 172.21.15.123:0/1356131042 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f5788158120 0x7f57881565b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:15.327 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:15.326+0000 7f578f5f4640 1 --2- 172.21.15.123:0/1356131042 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f57880cf5a0 0x7f57881f8b80 unknown :-1 s=CLOSED pgs=1502 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:15.327 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:15.326+0000 7f578f5f4640 1 -- 172.21.15.123:0/1356131042 >> 172.21.15.123:0/1356131042 conn(0x7f57880cfb10 msgr2=0x7f57880cd650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:50:15.327 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:15.326+0000 7f578f5f4640 1 -- 172.21.15.123:0/1356131042 shutdown_connections 2024-09-16T15:50:15.327 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:15.326+0000 7f578f5f4640 1 -- 172.21.15.123:0/1356131042 wait complete. 2024-09-16T15:50:15.342 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi123.front.sepia.ceph.com... 2024-09-16T15:50:15.342 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:50:15.342 DEBUG:teuthology.orchestra.run.smithi123:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2024-09-16T15:50:15.352 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi123.stderr:ceph-fuse[49774]: fuse finished with error 0 and tester_r 0 2024-09-16T15:50:15.353 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi123.front.sepia.ceph.com 2024-09-16T15:50:15.353 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:50:15.353 DEBUG:teuthology.orchestra.run.smithi123:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-09-16T15:50:15.431 INFO:tasks.cephfs.mount:Terminating background process 2024-09-16T15:50:15.431 DEBUG:teuthology.orchestra.run.smithi184:> set -ex 2024-09-16T15:50:15.431 DEBUG:teuthology.orchestra.run.smithi184:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-16T15:50:15.437 DEBUG:teuthology.orchestra.run.smithi184:> set -ex 2024-09-16T15:50:15.438 DEBUG:teuthology.orchestra.run.smithi184:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-16T15:50:15.485 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-09-16T15:50:15.614 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:15.614+0000 7fdaf0fe6640 1 -- 172.21.15.123:0/3598200598 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdaec15e820 msgr2=0x7fdaec15ec90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:50:15.614 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:15.614+0000 7fdaf0fe6640 1 --2- 172.21.15.123:0/3598200598 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdaec15e820 0x7fdaec15ec90 secure :-1 s=READY pgs=1503 cs=0 l=1 rev1=1 crypto rx=0x7fdaec070730 tx=0x7fdae401cc60 comp rx=0 tx=0).stop 2024-09-16T15:50:15.614 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:15.614+0000 7fdaf0fe6640 1 -- 172.21.15.123:0/3598200598 shutdown_connections 2024-09-16T15:50:15.615 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:15.614+0000 7fdaf0fe6640 1 --2- 172.21.15.123:0/3598200598 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdaec15e820 0x7fdaec15ec90 unknown :-1 s=CLOSED pgs=1503 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:15.615 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:15.614+0000 7fdaf0fe6640 1 --2- 172.21.15.123:0/3598200598 >> v2:172.21.15.184:3300/0 conn(0x7fdaec15e160 0x7fdaec073f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:15.615 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:15.614+0000 7fdaf0fe6640 1 -- 172.21.15.123:0/3598200598 >> 172.21.15.123:0/3598200598 conn(0x7fdaec0cfb10 msgr2=0x7fdaec0cff20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:50:15.615 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:15.614+0000 7fdaf0fe6640 1 -- 172.21.15.123:0/3598200598 shutdown_connections 2024-09-16T15:50:15.615 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:15.614+0000 7fdaf0fe6640 1 -- 172.21.15.123:0/3598200598 wait complete. 2024-09-16T15:50:15.615 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:15.614+0000 7fdaf0fe6640 1 Processor -- start 2024-09-16T15:50:15.615 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:15.614+0000 7fdaf0fe6640 1 -- start start 2024-09-16T15:50:15.617 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:15.618+0000 7fdaf0fe6640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdaec074470 0x7fdaec154c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:50:15.617 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:15.618+0000 7fdaf0fe6640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fdaec15e160 0x7fdaec154630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:50:15.617 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:15.618+0000 7fdaf0fe6640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fdaec15e820 0x7fdaec15b2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:50:15.618 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:15.618+0000 7fdaf0fe6640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fdaec0cf5b0 con 0x7fdaec074470 2024-09-16T15:50:15.618 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:15.618+0000 7fdaf0fe6640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fdaec0cf430 con 0x7fdaec15e160 2024-09-16T15:50:15.618 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:15.618+0000 7fdaf0fe6640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fdaec1683d0 con 0x7fdaec15e820 2024-09-16T15:50:15.618 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:15.618+0000 7fdaea59c640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdaec074470 0x7fdaec154c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:50:15.618 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:15.618+0000 7fdaea59c640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdaec074470 0x7fdaec154c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:37144/0 (socket says 172.21.15.123:37144) 2024-09-16T15:50:15.618 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:15.618+0000 7fdaea59c640 1 -- 172.21.15.123:0/1804039600 learned_addr learned my addr 172.21.15.123:0/1804039600 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:50:15.618 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:15.618+0000 7fdae9d9b640 1 --2- 172.21.15.123:0/1804039600 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fdaec15e160 0x7fdaec154630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:50:15.618 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:15.618+0000 7fdaead9d640 1 --2- 172.21.15.123:0/1804039600 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fdaec15e820 0x7fdaec15b2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:50:15.618 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:15.618+0000 7fdaea59c640 1 -- 172.21.15.123:0/1804039600 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fdaec15e820 msgr2=0x7fdaec15b2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:50:15.618 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:15.618+0000 7fdaea59c640 1 --2- 172.21.15.123:0/1804039600 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fdaec15e820 0x7fdaec15b2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:15.618 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:15.618+0000 7fdaea59c640 1 -- 172.21.15.123:0/1804039600 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fdaec15e160 msgr2=0x7fdaec154630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:50:15.618 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:15.618+0000 7fdaea59c640 1 --2- 172.21.15.123:0/1804039600 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fdaec15e160 0x7fdaec154630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:15.618 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:15.618+0000 7fdaea59c640 1 -- 172.21.15.123:0/1804039600 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdaec0b4aa0 con 0x7fdaec074470 2024-09-16T15:50:15.619 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:15.618+0000 7fdaead9d640 1 --2- 172.21.15.123:0/1804039600 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fdaec15e820 0x7fdaec15b2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T15:50:15.619 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:15.618+0000 7fdaea59c640 1 --2- 172.21.15.123:0/1804039600 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdaec074470 0x7fdaec154c20 secure :-1 s=READY pgs=1504 cs=0 l=1 rev1=1 crypto rx=0x7fdadc00c970 tx=0x7fdadc00ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:50:15.619 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:15.618+0000 7fdadaffd640 1 -- 172.21.15.123:0/1804039600 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdadc007c70 con 0x7fdaec074470 2024-09-16T15:50:15.619 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:15.618+0000 7fdadaffd640 1 -- 172.21.15.123:0/1804039600 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fdadc00f450 con 0x7fdaec074470 2024-09-16T15:50:15.619 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:15.618+0000 7fdaf0fe6640 1 -- 172.21.15.123:0/1804039600 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdaec15a7d0 con 0x7fdaec074470 2024-09-16T15:50:15.619 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:15.618+0000 7fdadaffd640 1 -- 172.21.15.123:0/1804039600 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdadc00f8a0 con 0x7fdaec074470 2024-09-16T15:50:15.619 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:15.618+0000 7fdaf0fe6640 1 -- 172.21.15.123:0/1804039600 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdaec15ad90 con 0x7fdaec074470 2024-09-16T15:50:15.620 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:15.618+0000 7fdaf0fe6640 1 -- 172.21.15.123:0/1804039600 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdaec073f30 con 0x7fdaec074470 2024-09-16T15:50:15.620 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:15.618+0000 7fdadaffd640 1 -- 172.21.15.123:0/1804039600 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 22) ==== 97692+0+0 (secure 0 0 0) 0x7fdadc00fa40 con 0x7fdaec074470 2024-09-16T15:50:15.620 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:15.618+0000 7fdadaffd640 1 --2- 172.21.15.123:0/1804039600 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fdacc073580 0x7fdacc075a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:50:15.620 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:15.618+0000 7fdadaffd640 1 -- 172.21.15.123:0/1804039600 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(135..135 src has 1..135) ==== 5737+0+0 (secure 0 0 0) 0x7fdadc0953d0 con 0x7fdaec074470 2024-09-16T15:50:15.620 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:15.618+0000 7fdae9d9b640 1 --2- 172.21.15.123:0/1804039600 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fdacc073580 0x7fdacc075a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:50:15.620 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:15.622+0000 7fdae9d9b640 1 --2- 172.21.15.123:0/1804039600 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fdacc073580 0x7fdacc075a40 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7fdaec1f8ac0 tx=0x7fdae0009040 comp rx=0 tx=0).ready entity=mgr.4150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:50:15.624 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:15.622+0000 7fdadaffd640 1 -- 172.21.15.123:0/1804039600 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7fdadc014030 con 0x7fdaec074470 2024-09-16T15:50:15.800 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:15.802+0000 7fdaf0fe6640 1 -- 172.21.15.123:0/1804039600 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7fdaec0d06c0 con 0x7fdaec074470 2024-09-16T15:50:15.801 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:15.802+0000 7fdadaffd640 1 -- 172.21.15.123:0/1804039600 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v135) ==== 80+0+0 (secure 0 0 0) 0x7fdadc05fb60 con 0x7fdaec074470 2024-09-16T15:50:15.801 INFO:teuthology.orchestra.run.smithi123.stderr:listed 0 entries 2024-09-16T15:50:15.803 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:15.802+0000 7fdaf0fe6640 1 -- 172.21.15.123:0/1804039600 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fdacc073580 msgr2=0x7fdacc075a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:50:15.804 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:15.802+0000 7fdaf0fe6640 1 --2- 172.21.15.123:0/1804039600 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fdacc073580 0x7fdacc075a40 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7fdaec1f8ac0 tx=0x7fdae0009040 comp rx=0 tx=0).stop 2024-09-16T15:50:15.804 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:15.802+0000 7fdaf0fe6640 1 -- 172.21.15.123:0/1804039600 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdaec074470 msgr2=0x7fdaec154c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:50:15.804 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:15.802+0000 7fdaf0fe6640 1 --2- 172.21.15.123:0/1804039600 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdaec074470 0x7fdaec154c20 secure :-1 s=READY pgs=1504 cs=0 l=1 rev1=1 crypto rx=0x7fdadc00c970 tx=0x7fdadc00ce30 comp rx=0 tx=0).stop 2024-09-16T15:50:15.804 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:15.802+0000 7fdaf0fe6640 1 -- 172.21.15.123:0/1804039600 shutdown_connections 2024-09-16T15:50:15.804 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:15.802+0000 7fdaf0fe6640 1 --2- 172.21.15.123:0/1804039600 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fdacc073580 0x7fdacc075a40 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:15.804 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:15.802+0000 7fdaf0fe6640 1 --2- 172.21.15.123:0/1804039600 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fdaec15e820 0x7fdaec15b2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:15.804 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:15.802+0000 7fdaf0fe6640 1 --2- 172.21.15.123:0/1804039600 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fdaec15e160 0x7fdaec154630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:15.804 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:15.802+0000 7fdaf0fe6640 1 --2- 172.21.15.123:0/1804039600 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdaec074470 0x7fdaec154c20 unknown :-1 s=CLOSED pgs=1504 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:15.804 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:15.802+0000 7fdaf0fe6640 1 -- 172.21.15.123:0/1804039600 >> 172.21.15.123:0/1804039600 conn(0x7fdaec0cfb10 msgr2=0x7fdaec158f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:50:15.804 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:15.802+0000 7fdaf0fe6640 1 -- 172.21.15.123:0/1804039600 shutdown_connections 2024-09-16T15:50:15.804 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:15.802+0000 7fdaf0fe6640 1 -- 172.21.15.123:0/1804039600 wait complete. 2024-09-16T15:50:15.818 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi184.front.sepia.ceph.com... 2024-09-16T15:50:15.819 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:50:15.819 DEBUG:teuthology.orchestra.run.smithi184:> sudo fusermount -u /home/ubuntu/cephtest/mnt.1 2024-09-16T15:50:15.832 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi184.stderr:ceph-fuse[36044]: fuse finished with error 0 and tester_r 0 2024-09-16T15:50:15.834 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi184.front.sepia.ceph.com 2024-09-16T15:50:15.834 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:50:15.834 DEBUG:teuthology.orchestra.run.smithi184:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2024-09-16T15:50:15.919 INFO:tasks.cephfs.mount:Terminating background process 2024-09-16T15:50:15.919 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2024-09-16T15:50:15.919 DEBUG:teuthology.orchestra.run.smithi123:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-16T15:50:15.925 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16336468k,nr_inodes=4084117,mode=755,inode64 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3274684k,mode=755,inode64 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev,inode64 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=17082 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3274680k,nr_inodes=818670,mode=700,uid=1000,gid=1271,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3274684k,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-09-16T15:50:15.925 INFO:tasks.cephfs.mount:Terminating background process 2024-09-16T15:50:15.926 DEBUG:teuthology.orchestra.run.smithi184:> set -ex 2024-09-16T15:50:15.926 DEBUG:teuthology.orchestra.run.smithi184:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-16T15:50:15.965 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16336488k,nr_inodes=4084122,mode=755,inode64 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3274688k,mode=755,inode64 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev,inode64 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=17834 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3274684k,nr_inodes=818671,mode=700,uid=1000,gid=1271,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3274688k,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-09-16T15:50:15.966 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-16T15:50:16.093 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:16.094+0000 7fc397bc7640 1 -- 172.21.15.123:0/2096217708 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc39015e820 msgr2=0x7fc39015ecd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:50:16.093 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:16.094+0000 7fc397bc7640 1 --2- 172.21.15.123:0/2096217708 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc39015e820 0x7fc39015ecd0 secure :-1 s=READY pgs=1505 cs=0 l=1 rev1=1 crypto rx=0x7fc388009c80 tx=0x7fc38801cd10 comp rx=0 tx=0).stop 2024-09-16T15:50:16.093 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:16.094+0000 7fc397bc7640 1 -- 172.21.15.123:0/2096217708 shutdown_connections 2024-09-16T15:50:16.093 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:16.094+0000 7fc397bc7640 1 --2- 172.21.15.123:0/2096217708 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc39015e820 0x7fc39015ecd0 unknown :-1 s=CLOSED pgs=1505 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:16.093 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:16.094+0000 7fc397bc7640 1 --2- 172.21.15.123:0/2096217708 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fc3900cd6e0 0x7fc3900cdb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:16.093 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:16.094+0000 7fc397bc7640 1 -- 172.21.15.123:0/2096217708 >> 172.21.15.123:0/2096217708 conn(0x7fc3900cfb10 msgr2=0x7fc3900cff20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:50:16.094 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:16.094+0000 7fc397bc7640 1 -- 172.21.15.123:0/2096217708 shutdown_connections 2024-09-16T15:50:16.094 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:16.094+0000 7fc397bc7640 1 -- 172.21.15.123:0/2096217708 wait complete. 2024-09-16T15:50:16.094 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:16.094+0000 7fc397bc7640 1 Processor -- start 2024-09-16T15:50:16.094 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:16.094+0000 7fc397bc7640 1 -- start start 2024-09-16T15:50:16.096 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:16.094+0000 7fc397bc7640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc3900cd6e0 0x7fc390159200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:50:16.096 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:16.094+0000 7fc397bc7640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc39015e160 0x7fc390155060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:50:16.096 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:16.094+0000 7fc397bc7640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fc39015e820 0x7fc3900c5a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:50:16.096 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:16.094+0000 7fc397bc7640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fc3900cf5b0 con 0x7fc3900cd6e0 2024-09-16T15:50:16.096 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:16.094+0000 7fc397bc7640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fc3900cf430 con 0x7fc39015e160 2024-09-16T15:50:16.096 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:16.094+0000 7fc397bc7640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fc3901683d0 con 0x7fc39015e820 2024-09-16T15:50:16.096 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:16.094+0000 7fc395963640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc3900cd6e0 0x7fc390159200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:50:16.096 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:16.094+0000 7fc395963640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc3900cd6e0 0x7fc390159200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:37162/0 (socket says 172.21.15.123:37162) 2024-09-16T15:50:16.096 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:16.094+0000 7fc395963640 1 -- 172.21.15.123:0/237602215 learned_addr learned my addr 172.21.15.123:0/237602215 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:50:16.097 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:16.094+0000 7fc395162640 1 --2- 172.21.15.123:0/237602215 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc39015e160 0x7fc390155060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:50:16.097 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:16.094+0000 7fc396164640 1 --2- 172.21.15.123:0/237602215 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fc39015e820 0x7fc3900c5a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:50:16.097 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:16.094+0000 7fc395963640 1 -- 172.21.15.123:0/237602215 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fc39015e820 msgr2=0x7fc3900c5a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:50:16.097 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:16.094+0000 7fc395963640 1 --2- 172.21.15.123:0/237602215 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fc39015e820 0x7fc3900c5a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:16.097 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:16.094+0000 7fc395963640 1 -- 172.21.15.123:0/237602215 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc39015e160 msgr2=0x7fc390155060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:50:16.097 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:16.094+0000 7fc395963640 1 --2- 172.21.15.123:0/237602215 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc39015e160 0x7fc390155060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:16.097 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:16.094+0000 7fc395963640 1 -- 172.21.15.123:0/237602215 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc3901555a0 con 0x7fc3900cd6e0 2024-09-16T15:50:16.097 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:16.094+0000 7fc395162640 1 --2- 172.21.15.123:0/237602215 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc39015e160 0x7fc390155060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T15:50:16.097 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:16.094+0000 7fc395963640 1 --2- 172.21.15.123:0/237602215 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc3900cd6e0 0x7fc390159200 secure :-1 s=READY pgs=1506 cs=0 l=1 rev1=1 crypto rx=0x7fc38c00c500 tx=0x7fc38c00c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:50:16.097 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:16.094+0000 7fc3867fc640 1 -- 172.21.15.123:0/237602215 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc38c017070 con 0x7fc3900cd6e0 2024-09-16T15:50:16.097 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:16.094+0000 7fc3867fc640 1 -- 172.21.15.123:0/237602215 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fc38c0053c0 con 0x7fc3900cd6e0 2024-09-16T15:50:16.097 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:16.094+0000 7fc397bc7640 1 -- 172.21.15.123:0/237602215 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc3901582e0 con 0x7fc3900cd6e0 2024-09-16T15:50:16.097 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:16.094+0000 7fc3867fc640 1 -- 172.21.15.123:0/237602215 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc38c012890 con 0x7fc3900cd6e0 2024-09-16T15:50:16.098 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:16.094+0000 7fc397bc7640 1 -- 172.21.15.123:0/237602215 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc3901588a0 con 0x7fc3900cd6e0 2024-09-16T15:50:16.098 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:16.098+0000 7fc397bc7640 1 -- 172.21.15.123:0/237602215 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc3900cdb20 con 0x7fc3900cd6e0 2024-09-16T15:50:16.098 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:16.098+0000 7fc3867fc640 1 -- 172.21.15.123:0/237602215 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 22) ==== 97692+0+0 (secure 0 0 0) 0x7fc38c01a070 con 0x7fc3900cd6e0 2024-09-16T15:50:16.099 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:16.098+0000 7fc3867fc640 1 --2- 172.21.15.123:0/237602215 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fc374073580 0x7fc374075a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:50:16.099 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:16.098+0000 7fc3867fc640 1 -- 172.21.15.123:0/237602215 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(135..135 src has 1..135) ==== 5737+0+0 (secure 0 0 0) 0x7fc38c094070 con 0x7fc3900cd6e0 2024-09-16T15:50:16.099 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:16.098+0000 7fc395162640 1 --2- 172.21.15.123:0/237602215 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fc374073580 0x7fc374075a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:50:16.099 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:16.098+0000 7fc395162640 1 --2- 172.21.15.123:0/237602215 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fc374073580 0x7fc374075a40 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7fc380005fd0 tx=0x7fc380005e80 comp rx=0 tx=0).ready entity=mgr.4150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:50:16.102 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:16.102+0000 7fc3867fc640 1 -- 172.21.15.123:0/237602215 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7fc38c021080 con 0x7fc3900cd6e0 2024-09-16T15:50:16.312 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:16.314+0000 7fc397bc7640 1 -- 172.21.15.123:0/237602215 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc3900d06c0 con 0x7fc3900cd6e0 2024-09-16T15:50:16.313 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:16.314+0000 7fc3867fc640 1 -- 172.21.15.123:0/237602215 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 87 v87) ==== 94+0+5230 (secure 0 0 0) 0x7fc38c05e7c0 con 0x7fc3900cd6e0 2024-09-16T15:50:16.313 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-09-16T15:50:16.313 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":87,"btime":"2024-09-16T15:50:06:171067+0000","default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5801,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6834/730567479","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":730567479},{"type":"v1","addr":"172.21.15.184:6834","nonce":730567479}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":81},{"gid":6346,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6835/584518652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6833","nonce":584518652},{"type":"v1","addr":"172.21.15.184:6835","nonce":584518652}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":81},{"gid":7938,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6833/3935767668","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6832","nonce":3935767668},{"type":"v1","addr":"172.21.15.123:6833","nonce":3935767668}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":80}],"filesystems":[{"mdsmap":{"epoch":86,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-16T15:49:53.127484+0000","modified":"2024-09-16T15:49:56.160938+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7944},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7944":{"gid":7944,"name":"a","rank":0,"incarnation":82,"state":"up:active","state_seq":3,"addr":"172.21.15.123:6835/333722639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6834","nonce":333722639},{"type":"v1","addr":"172.21.15.123:6835","nonce":333722639}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7944,"qdb_cluster":[7944]},"id":10}]} 2024-09-16T15:50:16.314 INFO:teuthology.orchestra.run.smithi123.stderr:dumped fsmap epoch 87 2024-09-16T15:50:16.316 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:16.314+0000 7fc397bc7640 1 -- 172.21.15.123:0/237602215 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fc374073580 msgr2=0x7fc374075a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:50:16.316 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:16.314+0000 7fc397bc7640 1 --2- 172.21.15.123:0/237602215 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fc374073580 0x7fc374075a40 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7fc380005fd0 tx=0x7fc380005e80 comp rx=0 tx=0).stop 2024-09-16T15:50:16.316 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:16.314+0000 7fc397bc7640 1 -- 172.21.15.123:0/237602215 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc3900cd6e0 msgr2=0x7fc390159200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:50:16.316 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:16.314+0000 7fc397bc7640 1 --2- 172.21.15.123:0/237602215 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc3900cd6e0 0x7fc390159200 secure :-1 s=READY pgs=1506 cs=0 l=1 rev1=1 crypto rx=0x7fc38c00c500 tx=0x7fc38c00c9c0 comp rx=0 tx=0).stop 2024-09-16T15:50:16.316 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:16.314+0000 7fc397bc7640 1 -- 172.21.15.123:0/237602215 shutdown_connections 2024-09-16T15:50:16.316 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:16.314+0000 7fc397bc7640 1 --2- 172.21.15.123:0/237602215 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fc374073580 0x7fc374075a40 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:16.316 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:16.314+0000 7fc397bc7640 1 --2- 172.21.15.123:0/237602215 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fc39015e820 0x7fc3900c5a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:16.316 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:16.314+0000 7fc397bc7640 1 --2- 172.21.15.123:0/237602215 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc39015e160 0x7fc390155060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:16.316 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:16.314+0000 7fc397bc7640 1 --2- 172.21.15.123:0/237602215 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc3900cd6e0 0x7fc390159200 unknown :-1 s=CLOSED pgs=1506 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:16.316 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:16.314+0000 7fc397bc7640 1 -- 172.21.15.123:0/237602215 >> 172.21.15.123:0/237602215 conn(0x7fc3900cfb10 msgr2=0x7fc3900758d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:50:16.316 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:16.314+0000 7fc397bc7640 1 -- 172.21.15.123:0/237602215 shutdown_connections 2024-09-16T15:50:16.317 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:16.314+0000 7fc397bc7640 1 -- 172.21.15.123:0/237602215 wait complete. 2024-09-16T15:50:16.332 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-16T15:50:16.441 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:16.442+0000 7fbce4c17640 1 -- 172.21.15.123:0/2486771130 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbce0070770 msgr2=0x7fbce00d1760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:50:16.441 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:16.442+0000 7fbce4c17640 1 --2- 172.21.15.123:0/2486771130 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbce0070770 0x7fbce00d1760 secure :-1 s=READY pgs=1508 cs=0 l=1 rev1=1 crypto rx=0x7fbcd8009bc0 tx=0x7fbcd801cb70 comp rx=0 tx=0).stop 2024-09-16T15:50:16.441 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:16.442+0000 7fbce4c17640 1 -- 172.21.15.123:0/2486771130 shutdown_connections 2024-09-16T15:50:16.441 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:16.442+0000 7fbce4c17640 1 --2- 172.21.15.123:0/2486771130 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbce0070770 0x7fbce00d1760 unknown :-1 s=CLOSED pgs=1508 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:16.441 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:16.442+0000 7fbce4c17640 1 --2- 172.21.15.123:0/2486771130 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fbce00cfef0 0x7fbce00cdb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:16.441 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:16.442+0000 7fbce4c17640 1 -- 172.21.15.123:0/2486771130 >> 172.21.15.123:0/2486771130 conn(0x7fbce0072150 msgr2=0x7fbce0072560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:50:16.442 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:16.442+0000 7fbce4c17640 1 -- 172.21.15.123:0/2486771130 shutdown_connections 2024-09-16T15:50:16.442 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:16.442+0000 7fbce4c17640 1 -- 172.21.15.123:0/2486771130 wait complete. 2024-09-16T15:50:16.444 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:16.442+0000 7fbce4c17640 1 Processor -- start 2024-09-16T15:50:16.444 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:16.442+0000 7fbce4c17640 1 -- start start 2024-09-16T15:50:16.444 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:16.442+0000 7fbce4c17640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbce0070770 0x7fbce00cb9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:50:16.444 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:16.442+0000 7fbce4c17640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fbce00b6f10 0x7fbce015b740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:50:16.444 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:16.442+0000 7fbce4c17640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fbce00cfef0 0x7fbce0164ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:50:16.444 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:16.442+0000 7fbce4c17640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fbce0168f40 con 0x7fbce0070770 2024-09-16T15:50:16.444 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:16.442+0000 7fbce4c17640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fbce0168dc0 con 0x7fbce00cfef0 2024-09-16T15:50:16.444 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:16.442+0000 7fbce4c17640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fbce01690c0 con 0x7fbce00b6f10 2024-09-16T15:50:16.444 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:16.442+0000 7fbcde59c640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbce0070770 0x7fbce00cb9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:50:16.445 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:16.442+0000 7fbcde59c640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbce0070770 0x7fbce00cb9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:37188/0 (socket says 172.21.15.123:37188) 2024-09-16T15:50:16.445 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:16.442+0000 7fbcde59c640 1 -- 172.21.15.123:0/876466716 learned_addr learned my addr 172.21.15.123:0/876466716 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:50:16.445 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:16.442+0000 7fbcde59c640 1 -- 172.21.15.123:0/876466716 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fbce00b6f10 msgr2=0x7fbce015b740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:50:16.445 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:16.442+0000 7fbcd7fff640 1 --2- 172.21.15.123:0/876466716 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fbce00b6f10 0x7fbce015b740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:50:16.445 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:16.442+0000 7fbcde59c640 1 --2- 172.21.15.123:0/876466716 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fbce00b6f10 0x7fbce015b740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:16.445 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:16.442+0000 7fbcde59c640 1 -- 172.21.15.123:0/876466716 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fbce00cfef0 msgr2=0x7fbce0164ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:50:16.445 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:16.442+0000 7fbcde59c640 1 --2- 172.21.15.123:0/876466716 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fbce00cfef0 0x7fbce0164ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:16.445 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:16.442+0000 7fbcde59c640 1 -- 172.21.15.123:0/876466716 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbce015bc80 con 0x7fbce0070770 2024-09-16T15:50:16.445 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:16.442+0000 7fbcde59c640 1 --2- 172.21.15.123:0/876466716 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbce0070770 0x7fbce00cb9d0 secure :-1 s=READY pgs=1509 cs=0 l=1 rev1=1 crypto rx=0x7fbcd000c530 tx=0x7fbcd000c9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:50:16.445 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:16.442+0000 7fbcd6ffd640 1 -- 172.21.15.123:0/876466716 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbcd0017070 con 0x7fbce0070770 2024-09-16T15:50:16.445 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:16.442+0000 7fbcd6ffd640 1 -- 172.21.15.123:0/876466716 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fbcd0003600 con 0x7fbce0070770 2024-09-16T15:50:16.445 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:16.442+0000 7fbcd6ffd640 1 -- 172.21.15.123:0/876466716 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbcd0003a50 con 0x7fbce0070770 2024-09-16T15:50:16.445 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:16.442+0000 7fbce4c17640 1 -- 172.21.15.123:0/876466716 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbce006dea0 con 0x7fbce0070770 2024-09-16T15:50:16.446 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:16.446+0000 7fbce4c17640 1 -- 172.21.15.123:0/876466716 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbce006e460 con 0x7fbce0070770 2024-09-16T15:50:16.446 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:16.446+0000 7fbce4c17640 1 -- 172.21.15.123:0/876466716 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbce00d1760 con 0x7fbce0070770 2024-09-16T15:50:16.446 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:16.446+0000 7fbcd6ffd640 1 -- 172.21.15.123:0/876466716 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 22) ==== 97692+0+0 (secure 0 0 0) 0x7fbcd001a070 con 0x7fbce0070770 2024-09-16T15:50:16.447 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:16.446+0000 7fbcd6ffd640 1 --2- 172.21.15.123:0/876466716 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fbcc0073580 0x7fbcc0075a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:50:16.447 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:16.446+0000 7fbcd7fff640 1 --2- 172.21.15.123:0/876466716 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fbcc0073580 0x7fbcc0075a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:50:16.447 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:16.446+0000 7fbcd6ffd640 1 -- 172.21.15.123:0/876466716 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(135..135 src has 1..135) ==== 5737+0+0 (secure 0 0 0) 0x7fbcd0093ee0 con 0x7fbce0070770 2024-09-16T15:50:16.447 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:16.446+0000 7fbcd7fff640 1 --2- 172.21.15.123:0/876466716 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fbcc0073580 0x7fbcc0075a40 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7fbce0165a80 tx=0x7fbcc8008040 comp rx=0 tx=0).ready entity=mgr.4150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:50:16.450 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:16.450+0000 7fbcd6ffd640 1 -- 172.21.15.123:0/876466716 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7fbcd0097030 con 0x7fbce0070770 2024-09-16T15:50:16.641 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:16.642+0000 7fbce4c17640 1 -- 172.21.15.123:0/876466716 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fbce00b50c0 con 0x7fbce0070770 2024-09-16T15:50:16.642 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:16.642+0000 7fbcd6ffd640 1 -- 172.21.15.123:0/876466716 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 87 v87) ==== 94+0+5230 (secure 0 0 0) 0x7fbcd005e730 con 0x7fbce0070770 2024-09-16T15:50:16.643 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-09-16T15:50:16.643 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":87,"btime":"2024-09-16T15:50:06:171067+0000","default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5801,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6834/730567479","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":730567479},{"type":"v1","addr":"172.21.15.184:6834","nonce":730567479}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":81},{"gid":6346,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6835/584518652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6833","nonce":584518652},{"type":"v1","addr":"172.21.15.184:6835","nonce":584518652}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":81},{"gid":7938,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6833/3935767668","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6832","nonce":3935767668},{"type":"v1","addr":"172.21.15.123:6833","nonce":3935767668}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":80}],"filesystems":[{"mdsmap":{"epoch":86,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-16T15:49:53.127484+0000","modified":"2024-09-16T15:49:56.160938+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7944},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7944":{"gid":7944,"name":"a","rank":0,"incarnation":82,"state":"up:active","state_seq":3,"addr":"172.21.15.123:6835/333722639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6834","nonce":333722639},{"type":"v1","addr":"172.21.15.123:6835","nonce":333722639}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7944,"qdb_cluster":[7944]},"id":10}]} 2024-09-16T15:50:16.643 INFO:teuthology.orchestra.run.smithi123.stderr:dumped fsmap epoch 87 2024-09-16T15:50:16.645 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:16.646+0000 7fbce4c17640 1 -- 172.21.15.123:0/876466716 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fbcc0073580 msgr2=0x7fbcc0075a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:50:16.645 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:16.646+0000 7fbce4c17640 1 --2- 172.21.15.123:0/876466716 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fbcc0073580 0x7fbcc0075a40 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7fbce0165a80 tx=0x7fbcc8008040 comp rx=0 tx=0).stop 2024-09-16T15:50:16.645 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:16.646+0000 7fbce4c17640 1 -- 172.21.15.123:0/876466716 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbce0070770 msgr2=0x7fbce00cb9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:50:16.645 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:16.646+0000 7fbce4c17640 1 --2- 172.21.15.123:0/876466716 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbce0070770 0x7fbce00cb9d0 secure :-1 s=READY pgs=1509 cs=0 l=1 rev1=1 crypto rx=0x7fbcd000c530 tx=0x7fbcd000c9f0 comp rx=0 tx=0).stop 2024-09-16T15:50:16.646 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:16.646+0000 7fbce4c17640 1 -- 172.21.15.123:0/876466716 shutdown_connections 2024-09-16T15:50:16.646 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:16.646+0000 7fbce4c17640 1 --2- 172.21.15.123:0/876466716 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fbcc0073580 0x7fbcc0075a40 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:16.646 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:16.646+0000 7fbce4c17640 1 --2- 172.21.15.123:0/876466716 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fbce00cfef0 0x7fbce0164ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:16.646 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:16.646+0000 7fbce4c17640 1 --2- 172.21.15.123:0/876466716 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fbce00b6f10 0x7fbce015b740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:16.646 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:16.646+0000 7fbce4c17640 1 --2- 172.21.15.123:0/876466716 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbce0070770 0x7fbce00cb9d0 unknown :-1 s=CLOSED pgs=1509 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:16.646 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:16.646+0000 7fbce4c17640 1 -- 172.21.15.123:0/876466716 >> 172.21.15.123:0/876466716 conn(0x7fbce0072150 msgr2=0x7fbce00ce130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:50:16.646 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:16.646+0000 7fbce4c17640 1 -- 172.21.15.123:0/876466716 shutdown_connections 2024-09-16T15:50:16.646 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:16.646+0000 7fbce4c17640 1 -- 172.21.15.123:0/876466716 wait complete. 2024-09-16T15:50:16.660 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-16T15:50:16.780 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:16.778+0000 7f763c4d8640 1 -- 172.21.15.123:0/479498764 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f76340cf530 msgr2=0x7f76340d2270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:50:16.780 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:16.778+0000 7f763c4d8640 1 --2- 172.21.15.123:0/479498764 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f76340cf530 0x7f76340d2270 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f7624009900 tx=0x7f762401f0d0 comp rx=0 tx=0).stop 2024-09-16T15:50:16.780 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:16.778+0000 7f763c4d8640 1 -- 172.21.15.123:0/479498764 shutdown_connections 2024-09-16T15:50:16.780 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:16.778+0000 7f763c4d8640 1 --2- 172.21.15.123:0/479498764 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f76340cf530 0x7f76340d2270 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:16.780 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:16.778+0000 7f763c4d8640 1 --2- 172.21.15.123:0/479498764 >> v2:172.21.15.184:3300/0 conn(0x7f76340d0aa0 0x7f76340d1d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:16.780 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:16.778+0000 7f763c4d8640 1 -- 172.21.15.123:0/479498764 >> 172.21.15.123:0/479498764 conn(0x7f76340d3e30 msgr2=0x7f76340d4240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:50:16.780 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:16.778+0000 7f763c4d8640 1 -- 172.21.15.123:0/479498764 shutdown_connections 2024-09-16T15:50:16.780 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:16.778+0000 7f763c4d8640 1 -- 172.21.15.123:0/479498764 wait complete. 2024-09-16T15:50:16.781 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:16.782+0000 7f763c4d8640 1 Processor -- start 2024-09-16T15:50:16.781 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:16.782+0000 7f763c4d8640 1 -- start start 2024-09-16T15:50:16.782 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:16.782+0000 7f763c4d8640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f76340cf530 0x7f763415f090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:50:16.783 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:16.782+0000 7f763c4d8640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f76340d0200 0x7f76340caf00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:50:16.783 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:16.782+0000 7f763c4d8640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f76340d0aa0 0x7f76341f46d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:50:16.783 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:16.782+0000 7f763c4d8640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f76340b52a0 con 0x7f76340cf530 2024-09-16T15:50:16.783 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:16.782+0000 7f763c4d8640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f76340d1090 con 0x7f76340d0aa0 2024-09-16T15:50:16.783 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:16.782+0000 7f763c4d8640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f76340d27b0 con 0x7f76340d0200 2024-09-16T15:50:16.783 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:16.782+0000 7f763a274640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f76340cf530 0x7f763415f090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:50:16.783 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:16.782+0000 7f763a274640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f76340cf530 0x7f763415f090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:37196/0 (socket says 172.21.15.123:37196) 2024-09-16T15:50:16.783 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:16.782+0000 7f763a274640 1 -- 172.21.15.123:0/3239021622 learned_addr learned my addr 172.21.15.123:0/3239021622 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:50:16.783 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:16.782+0000 7f763aa75640 1 --2- 172.21.15.123:0/3239021622 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f76340d0aa0 0x7f76341f46d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:50:16.783 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:16.782+0000 7f7639a73640 1 --2- 172.21.15.123:0/3239021622 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f76340d0200 0x7f76340caf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:50:16.783 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:16.782+0000 7f763a274640 1 -- 172.21.15.123:0/3239021622 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f76340d0200 msgr2=0x7f76340caf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:50:16.784 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:16.782+0000 7f763a274640 1 --2- 172.21.15.123:0/3239021622 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f76340d0200 0x7f76340caf00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:16.784 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:16.782+0000 7f763a274640 1 -- 172.21.15.123:0/3239021622 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f76340d0aa0 msgr2=0x7f76341f46d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:50:16.784 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:16.782+0000 7f763a274640 1 --2- 172.21.15.123:0/3239021622 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f76340d0aa0 0x7f76341f46d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:16.784 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:16.782+0000 7f763a274640 1 -- 172.21.15.123:0/3239021622 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f76341f51e0 con 0x7f76340cf530 2024-09-16T15:50:16.784 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:16.782+0000 7f763a274640 1 --2- 172.21.15.123:0/3239021622 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f76340cf530 0x7f763415f090 secure :-1 s=READY pgs=1510 cs=0 l=1 rev1=1 crypto rx=0x7f763000e7a0 tx=0x7f763000ec60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:50:16.784 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:16.782+0000 7f762affd640 1 -- 172.21.15.123:0/3239021622 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f763000ccc0 con 0x7f76340cf530 2024-09-16T15:50:16.784 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:16.782+0000 7f762affd640 1 -- 172.21.15.123:0/3239021622 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f7630010430 con 0x7f76340cf530 2024-09-16T15:50:16.784 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:16.782+0000 7f763c4d8640 1 -- 172.21.15.123:0/3239021622 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f76341f54c0 con 0x7f76340cf530 2024-09-16T15:50:16.784 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:16.782+0000 7f762affd640 1 -- 172.21.15.123:0/3239021622 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7630010860 con 0x7f76340cf530 2024-09-16T15:50:16.785 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:16.782+0000 7f763c4d8640 1 -- 172.21.15.123:0/3239021622 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f763415f5d0 con 0x7f76340cf530 2024-09-16T15:50:16.785 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:16.782+0000 7f763c4d8640 1 -- 172.21.15.123:0/3239021622 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f76340c9000 con 0x7f76340cf530 2024-09-16T15:50:16.785 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:16.786+0000 7f762affd640 1 -- 172.21.15.123:0/3239021622 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 22) ==== 97692+0+0 (secure 0 0 0) 0x7f7630010a00 con 0x7f76340cf530 2024-09-16T15:50:16.785 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:16.786+0000 7f762affd640 1 --2- 172.21.15.123:0/3239021622 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f76180735b0 0x7f7618075a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:50:16.785 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:16.786+0000 7f762affd640 1 -- 172.21.15.123:0/3239021622 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(135..135 src has 1..135) ==== 5737+0+0 (secure 0 0 0) 0x7f7630095620 con 0x7f76340cf530 2024-09-16T15:50:16.785 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:16.786+0000 7f7639a73640 1 --2- 172.21.15.123:0/3239021622 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f76180735b0 0x7f7618075a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:50:16.786 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:16.786+0000 7f7639a73640 1 --2- 172.21.15.123:0/3239021622 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f76180735b0 0x7f7618075a70 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f76240b2000 tx=0x7f762401d910 comp rx=0 tx=0).ready entity=mgr.4150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:50:16.789 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:16.790+0000 7f762affd640 1 -- 172.21.15.123:0/3239021622 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f763005fca0 con 0x7f76340cf530 2024-09-16T15:50:16.949 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:16.950+0000 7f763c4d8640 1 -- 172.21.15.123:0/3239021622 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f76340d1d30 con 0x7f76340cf530 2024-09-16T15:50:16.951 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:16.950+0000 7f762affd640 1 -- 172.21.15.123:0/3239021622 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v135) ==== 74+0+16843 (secure 0 0 0) 0x7f763005f5a0 con 0x7f76340cf530 2024-09-16T15:50:16.951 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-09-16T15:50:16.951 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":135,"fsid":"56077f89-18c8-486f-ae7e-eecc0bb7822c","created":"2024-09-16T15:42:26.364892+0000","modified":"2024-09-16T15:49:54.741464+0000","last_up_change":"2024-09-16T15:42:38.707234+0000","last_in_change":"2024-09-16T15:42:29.423662+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":21,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-16T15:42:39.287764+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":20,"pool_name":"cephfs_metadata","create_time":"2024-09-16T15:49:50.809741+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"134","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":21,"pool_name":"cephfs_data","create_time":"2024-09-16T15:49:52.080476+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"135","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9e7c1955-a607-4671-9be7-f2ef760ceeaa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":133,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6800","nonce":3882044969},{"type":"v1","addr":"172.21.15.123:6801","nonce":3882044969}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6802","nonce":3882044969},{"type":"v1","addr":"172.21.15.123:6803","nonce":3882044969}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6806","nonce":3882044969},{"type":"v1","addr":"172.21.15.123:6807","nonce":3882044969}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6804","nonce":3882044969},{"type":"v1","addr":"172.21.15.123:6805","nonce":3882044969}]},"public_addr":"172.21.15.123:6801/3882044969","cluster_addr":"172.21.15.123:6803/3882044969","heartbeat_back_addr":"172.21.15.123:6807/3882044969","heartbeat_front_addr":"172.21.15.123:6805/3882044969","state":["exists","up"]},{"osd":1,"uuid":"c540cfde-0977-4742-9b17-fd634668e010","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":133,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6816","nonce":4037001850},{"type":"v1","addr":"172.21.15.123:6817","nonce":4037001850}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6818","nonce":4037001850},{"type":"v1","addr":"172.21.15.123:6819","nonce":4037001850}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6822","nonce":4037001850},{"type":"v1","addr":"172.21.15.123:6823","nonce":4037001850}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6820","nonce":4037001850},{"type":"v1","addr":"172.21.15.123:6821","nonce":4037001850}]},"public_addr":"172.21.15.123:6817/4037001850","cluster_addr":"172.21.15.123:6819/4037001850","heartbeat_back_addr":"172.21.15.123:6823/4037001850","heartbeat_front_addr":"172.21.15.123:6821/4037001850","state":["exists","up"]},{"osd":2,"uuid":"9864a496-f169-486b-95c4-1d59162793a7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":133,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6824","nonce":1332143088},{"type":"v1","addr":"172.21.15.123:6825","nonce":1332143088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6826","nonce":1332143088},{"type":"v1","addr":"172.21.15.123:6827","nonce":1332143088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6830","nonce":1332143088},{"type":"v1","addr":"172.21.15.123:6831","nonce":1332143088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6828","nonce":1332143088},{"type":"v1","addr":"172.21.15.123:6829","nonce":1332143088}]},"public_addr":"172.21.15.123:6825/1332143088","cluster_addr":"172.21.15.123:6827/1332143088","heartbeat_back_addr":"172.21.15.123:6831/1332143088","heartbeat_front_addr":"172.21.15.123:6829/1332143088","state":["exists","up"]},{"osd":3,"uuid":"c6d1f817-f074-4325-95e1-578eba13edda","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":133,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6808","nonce":988586998},{"type":"v1","addr":"172.21.15.123:6809","nonce":988586998}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6810","nonce":988586998},{"type":"v1","addr":"172.21.15.123:6811","nonce":988586998}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6814","nonce":988586998},{"type":"v1","addr":"172.21.15.123:6815","nonce":988586998}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6812","nonce":988586998},{"type":"v1","addr":"172.21.15.123:6813","nonce":988586998}]},"public_addr":"172.21.15.123:6809/988586998","cluster_addr":"172.21.15.123:6811/988586998","heartbeat_back_addr":"172.21.15.123:6815/988586998","heartbeat_front_addr":"172.21.15.123:6813/988586998","state":["exists","up"]},{"osd":4,"uuid":"200a8df4-50ef-40a5-b48a-58981269c086","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":133,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6816","nonce":3387813890},{"type":"v1","addr":"172.21.15.184:6817","nonce":3387813890}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6818","nonce":3387813890},{"type":"v1","addr":"172.21.15.184:6819","nonce":3387813890}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6822","nonce":3387813890},{"type":"v1","addr":"172.21.15.184:6823","nonce":3387813890}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6820","nonce":3387813890},{"type":"v1","addr":"172.21.15.184:6821","nonce":3387813890}]},"public_addr":"172.21.15.184:6817/3387813890","cluster_addr":"172.21.15.184:6819/3387813890","heartbeat_back_addr":"172.21.15.184:6823/3387813890","heartbeat_front_addr":"172.21.15.184:6821/3387813890","state":["exists","up"]},{"osd":5,"uuid":"32d5a900-da3d-412c-bc19-13861a2368d8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":133,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6808","nonce":2680916017},{"type":"v1","addr":"172.21.15.184:6809","nonce":2680916017}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6810","nonce":2680916017},{"type":"v1","addr":"172.21.15.184:6811","nonce":2680916017}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6814","nonce":2680916017},{"type":"v1","addr":"172.21.15.184:6815","nonce":2680916017}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6812","nonce":2680916017},{"type":"v1","addr":"172.21.15.184:6813","nonce":2680916017}]},"public_addr":"172.21.15.184:6809/2680916017","cluster_addr":"172.21.15.184:6811/2680916017","heartbeat_back_addr":"172.21.15.184:6815/2680916017","heartbeat_front_addr":"172.21.15.184:6813/2680916017","state":["exists","up"]},{"osd":6,"uuid":"cde6f39e-aa1b-4260-ac19-774ee6aaca72","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":133,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6824","nonce":2183120457},{"type":"v1","addr":"172.21.15.184:6825","nonce":2183120457}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6826","nonce":2183120457},{"type":"v1","addr":"172.21.15.184:6827","nonce":2183120457}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6830","nonce":2183120457},{"type":"v1","addr":"172.21.15.184:6831","nonce":2183120457}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6828","nonce":2183120457},{"type":"v1","addr":"172.21.15.184:6829","nonce":2183120457}]},"public_addr":"172.21.15.184:6825/2183120457","cluster_addr":"172.21.15.184:6827/2183120457","heartbeat_back_addr":"172.21.15.184:6831/2183120457","heartbeat_front_addr":"172.21.15.184:6829/2183120457","state":["exists","up"]},{"osd":7,"uuid":"4f381778-eee1-47c6-804f-df0e6b33e313","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":133,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6800","nonce":4102957285},{"type":"v1","addr":"172.21.15.184:6801","nonce":4102957285}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6802","nonce":4102957285},{"type":"v1","addr":"172.21.15.184:6803","nonce":4102957285}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6806","nonce":4102957285},{"type":"v1","addr":"172.21.15.184:6807","nonce":4102957285}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6804","nonce":4102957285},{"type":"v1","addr":"172.21.15.184:6805","nonce":4102957285}]},"public_addr":"172.21.15.184:6801/4102957285","cluster_addr":"172.21.15.184:6803/4102957285","heartbeat_back_addr":"172.21.15.184:6807/4102957285","heartbeat_front_addr":"172.21.15.184:6805/4102957285","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:35.972210+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.222007+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.198617+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.385663+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.358935+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.156066+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.752973+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:35.682846+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":17,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T15:50:16.953 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:16.954+0000 7f763c4d8640 1 -- 172.21.15.123:0/3239021622 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f76180735b0 msgr2=0x7f7618075a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:50:16.953 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:16.954+0000 7f763c4d8640 1 --2- 172.21.15.123:0/3239021622 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f76180735b0 0x7f7618075a70 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f76240b2000 tx=0x7f762401d910 comp rx=0 tx=0).stop 2024-09-16T15:50:16.954 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:16.954+0000 7f763c4d8640 1 -- 172.21.15.123:0/3239021622 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f76340cf530 msgr2=0x7f763415f090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:50:16.954 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:16.954+0000 7f763c4d8640 1 --2- 172.21.15.123:0/3239021622 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f76340cf530 0x7f763415f090 secure :-1 s=READY pgs=1510 cs=0 l=1 rev1=1 crypto rx=0x7f763000e7a0 tx=0x7f763000ec60 comp rx=0 tx=0).stop 2024-09-16T15:50:16.954 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:16.954+0000 7f763c4d8640 1 -- 172.21.15.123:0/3239021622 shutdown_connections 2024-09-16T15:50:16.954 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:16.954+0000 7f763c4d8640 1 --2- 172.21.15.123:0/3239021622 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f76180735b0 0x7f7618075a70 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:16.954 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:16.954+0000 7f763c4d8640 1 --2- 172.21.15.123:0/3239021622 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f76340d0aa0 0x7f76341f46d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:16.954 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:16.954+0000 7f763c4d8640 1 --2- 172.21.15.123:0/3239021622 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f76340d0200 0x7f76340caf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:16.954 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:16.954+0000 7f763c4d8640 1 --2- 172.21.15.123:0/3239021622 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f76340cf530 0x7f763415f090 unknown :-1 s=CLOSED pgs=1510 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:16.954 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:16.954+0000 7f763c4d8640 1 -- 172.21.15.123:0/3239021622 >> 172.21.15.123:0/3239021622 conn(0x7f76340d3e30 msgr2=0x7f76341613c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:50:16.954 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:16.954+0000 7f763c4d8640 1 -- 172.21.15.123:0/3239021622 shutdown_connections 2024-09-16T15:50:16.954 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:16.954+0000 7f763c4d8640 1 -- 172.21.15.123:0/3239021622 wait complete. 2024-09-16T15:50:16.969 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-09-16T15:50:16.969 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-16T15:50:17.085 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:17.086+0000 7f814b349640 1 -- 172.21.15.123:0/2977151449 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f81300051d0 msgr2=0x7f813000b9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:50:17.085 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:17.086+0000 7f814b349640 1 --2- 172.21.15.123:0/2977151449 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f81300051d0 0x7f813000b9d0 secure :-1 s=READY pgs=1512 cs=0 l=1 rev1=1 crypto rx=0x7f81440d2aa0 tx=0x7f813c01d200 comp rx=0 tx=0).stop 2024-09-16T15:50:17.085 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:17.086+0000 7f81498e6640 1 -- 172.21.15.123:0/2977151449 reap_dead start 2024-09-16T15:50:17.086 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:17.086+0000 7f814b349640 1 -- 172.21.15.123:0/2977151449 shutdown_connections 2024-09-16T15:50:17.086 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:17.086+0000 7f814b349640 1 -- 172.21.15.123:0/2977151449 >> 172.21.15.123:0/2977151449 conn(0x7f814408bfd0 msgr2=0x7f814409b150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:50:17.086 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:17.086+0000 7f814b349640 1 -- 172.21.15.123:0/2977151449 shutdown_connections 2024-09-16T15:50:17.086 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:17.086+0000 7f814b349640 1 -- 172.21.15.123:0/2977151449 wait complete. 2024-09-16T15:50:17.086 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:17.086+0000 7f814b349640 1 Processor -- start 2024-09-16T15:50:17.086 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:17.086+0000 7f814b349640 1 -- start start 2024-09-16T15:50:17.088 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:17.086+0000 7f814b349640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f81441f45d0 0x7f81441f49b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:50:17.089 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:17.086+0000 7f814b349640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f814406f5e0 0x7f81441f4ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:50:17.089 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:17.086+0000 7f814b349640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f814406ea30 0x7f81440d0cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:50:17.089 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:17.086+0000 7f814b349640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f81440b00e0 con 0x7f814406f5e0 2024-09-16T15:50:17.089 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:17.086+0000 7f814b349640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f81440aff60 con 0x7f814406ea30 2024-09-16T15:50:17.089 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:17.086+0000 7f814b349640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f81440b0260 con 0x7f81441f45d0 2024-09-16T15:50:17.089 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:17.086+0000 7f81488e4640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f814406f5e0 0x7f81441f4ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:50:17.089 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:17.086+0000 7f81488e4640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f814406f5e0 0x7f81441f4ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:37204/0 (socket says 172.21.15.123:37204) 2024-09-16T15:50:17.089 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:17.086+0000 7f81488e4640 1 -- 172.21.15.123:0/4254008337 learned_addr learned my addr 172.21.15.123:0/4254008337 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:50:17.089 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:17.086+0000 7f81490e5640 1 --2- 172.21.15.123:0/4254008337 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f81441f45d0 0x7f81441f49b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:50:17.089 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:17.086+0000 7f81498e6640 1 --2- 172.21.15.123:0/4254008337 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f814406ea30 0x7f81440d0cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:50:17.089 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:17.086+0000 7f81488e4640 1 -- 172.21.15.123:0/4254008337 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f81441f45d0 msgr2=0x7f81441f49b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:50:17.089 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:17.086+0000 7f81488e4640 1 --2- 172.21.15.123:0/4254008337 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f81441f45d0 0x7f81441f49b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:17.089 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:17.086+0000 7f81488e4640 1 -- 172.21.15.123:0/4254008337 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f814406ea30 msgr2=0x7f81440d0cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:50:17.090 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:17.086+0000 7f81488e4640 1 --2- 172.21.15.123:0/4254008337 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f814406ea30 0x7f81440d0cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:17.090 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:17.086+0000 7f81488e4640 1 -- 172.21.15.123:0/4254008337 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f814406fb10 con 0x7f814406f5e0 2024-09-16T15:50:17.090 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:17.086+0000 7f81498e6640 1 --2- 172.21.15.123:0/4254008337 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f814406ea30 0x7f81440d0cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T15:50:17.090 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:17.090+0000 7f81488e4640 1 --2- 172.21.15.123:0/4254008337 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f814406f5e0 0x7f81441f4ef0 secure :-1 s=READY pgs=1513 cs=0 l=1 rev1=1 crypto rx=0x7f813400e9c0 tx=0x7f813400ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:50:17.090 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:17.090+0000 7f8139ffb640 1 -- 172.21.15.123:0/4254008337 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f813400cc60 con 0x7f814406f5e0 2024-09-16T15:50:17.090 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:17.090+0000 7f8139ffb640 1 -- 172.21.15.123:0/4254008337 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f8134011470 con 0x7f814406f5e0 2024-09-16T15:50:17.090 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:17.090+0000 7f814b349640 1 -- 172.21.15.123:0/4254008337 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f814406df50 con 0x7f814406f5e0 2024-09-16T15:50:17.090 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:17.090+0000 7f8139ffb640 1 -- 172.21.15.123:0/4254008337 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81340118a0 con 0x7f814406f5e0 2024-09-16T15:50:17.090 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:17.090+0000 7f814b349640 1 -- 172.21.15.123:0/4254008337 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f814406e510 con 0x7f814406f5e0 2024-09-16T15:50:17.091 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:17.090+0000 7f814b349640 1 -- 172.21.15.123:0/4254008337 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f81440712b0 con 0x7f814406f5e0 2024-09-16T15:50:17.091 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:17.090+0000 7f8139ffb640 1 -- 172.21.15.123:0/4254008337 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 22) ==== 97692+0+0 (secure 0 0 0) 0x7f8134011a40 con 0x7f814406f5e0 2024-09-16T15:50:17.091 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:17.090+0000 7f8139ffb640 1 --2- 172.21.15.123:0/4254008337 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f81280735b0 0x7f8128075a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:50:17.091 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:17.090+0000 7f8139ffb640 1 -- 172.21.15.123:0/4254008337 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(135..135 src has 1..135) ==== 5737+0+0 (secure 0 0 0) 0x7f8134095670 con 0x7f814406f5e0 2024-09-16T15:50:17.091 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:17.090+0000 7f81490e5640 1 --2- 172.21.15.123:0/4254008337 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f81280735b0 0x7f8128075a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:50:17.091 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:17.090+0000 7f81490e5640 1 --2- 172.21.15.123:0/4254008337 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f81280735b0 0x7f8128075a70 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f81440cbcf0 tx=0x7f8140009040 comp rx=0 tx=0).ready entity=mgr.4150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:50:17.094 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:17.094+0000 7f8139ffb640 1 -- 172.21.15.123:0/4254008337 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f8134016030 con 0x7f814406f5e0 2024-09-16T15:50:17.297 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:17.298+0000 7f814b349640 1 -- 172.21.15.123:0/4254008337 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f81440cc230 con 0x7f814406f5e0 2024-09-16T15:50:17.298 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:17.298+0000 7f8139ffb640 1 -- 172.21.15.123:0/4254008337 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 87 v87) ==== 94+0+5230 (secure 0 0 0) 0x7f813405fe00 con 0x7f814406f5e0 2024-09-16T15:50:17.298 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-09-16T15:50:17.298 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":87,"btime":"2024-09-16T15:50:06:171067+0000","default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5801,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6834/730567479","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":730567479},{"type":"v1","addr":"172.21.15.184:6834","nonce":730567479}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":81},{"gid":6346,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6835/584518652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6833","nonce":584518652},{"type":"v1","addr":"172.21.15.184:6835","nonce":584518652}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":81},{"gid":7938,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6833/3935767668","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6832","nonce":3935767668},{"type":"v1","addr":"172.21.15.123:6833","nonce":3935767668}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":80}],"filesystems":[{"mdsmap":{"epoch":86,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-16T15:49:53.127484+0000","modified":"2024-09-16T15:49:56.160938+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7944},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7944":{"gid":7944,"name":"a","rank":0,"incarnation":82,"state":"up:active","state_seq":3,"addr":"172.21.15.123:6835/333722639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6834","nonce":333722639},{"type":"v1","addr":"172.21.15.123:6835","nonce":333722639}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7944,"qdb_cluster":[7944]},"id":10}]} 2024-09-16T15:50:17.298 INFO:teuthology.orchestra.run.smithi123.stderr:dumped fsmap epoch 87 2024-09-16T15:50:17.300 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:17.298+0000 7f814b349640 1 -- 172.21.15.123:0/4254008337 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f81280735b0 msgr2=0x7f8128075a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:50:17.300 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:17.298+0000 7f814b349640 1 --2- 172.21.15.123:0/4254008337 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f81280735b0 0x7f8128075a70 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f81440cbcf0 tx=0x7f8140009040 comp rx=0 tx=0).stop 2024-09-16T15:50:17.300 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:17.298+0000 7f814b349640 1 -- 172.21.15.123:0/4254008337 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f814406f5e0 msgr2=0x7f81441f4ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:50:17.301 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:17.298+0000 7f814b349640 1 --2- 172.21.15.123:0/4254008337 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f814406f5e0 0x7f81441f4ef0 secure :-1 s=READY pgs=1513 cs=0 l=1 rev1=1 crypto rx=0x7f813400e9c0 tx=0x7f813400ee80 comp rx=0 tx=0).stop 2024-09-16T15:50:17.301 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:17.302+0000 7f814b349640 1 -- 172.21.15.123:0/4254008337 shutdown_connections 2024-09-16T15:50:17.301 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:17.302+0000 7f814b349640 1 --2- 172.21.15.123:0/4254008337 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f81280735b0 0x7f8128075a70 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:17.301 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:17.302+0000 7f814b349640 1 --2- 172.21.15.123:0/4254008337 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f814406ea30 0x7f81440d0cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:17.301 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:17.302+0000 7f814b349640 1 --2- 172.21.15.123:0/4254008337 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f814406f5e0 0x7f81441f4ef0 unknown :-1 s=CLOSED pgs=1513 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:17.301 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:17.302+0000 7f814b349640 1 --2- 172.21.15.123:0/4254008337 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f81441f45d0 0x7f81441f49b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:17.301 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:17.302+0000 7f814b349640 1 -- 172.21.15.123:0/4254008337 >> 172.21.15.123:0/4254008337 conn(0x7f814408bfd0 msgr2=0x7f81440b0770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:50:17.301 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:17.302+0000 7f814b349640 1 -- 172.21.15.123:0/4254008337 shutdown_connections 2024-09-16T15:50:17.301 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:17.302+0000 7f814b349640 1 -- 172.21.15.123:0/4254008337 wait complete. 2024-09-16T15:50:17.316 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-16T15:50:17.429 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:17.430+0000 7f00175f6640 1 -- 172.21.15.123:0/353145530 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7efffc003a10 msgr2=0x7efffc003e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:50:17.429 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:17.430+0000 7f00175f6640 1 --2- 172.21.15.123:0/353145530 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7efffc003a10 0x7efffc003e30 secure :-1 s=READY pgs=1515 cs=0 l=1 rev1=1 crypto rx=0x7f000c00bb10 tx=0x7f000c01cd70 comp rx=0 tx=0).stop 2024-09-16T15:50:17.429 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:17.430+0000 7f00175f6640 1 -- 172.21.15.123:0/353145530 shutdown_connections 2024-09-16T15:50:17.430 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:17.430+0000 7f00175f6640 1 --2- 172.21.15.123:0/353145530 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7efffc003a10 0x7efffc003e30 unknown :-1 s=CLOSED pgs=1515 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:17.430 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:17.430+0000 7f00175f6640 1 -- 172.21.15.123:0/353145530 >> 172.21.15.123:0/353145530 conn(0x7f00100cfb10 msgr2=0x7f00100cff20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:50:17.430 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:17.430+0000 7f00175f6640 1 -- 172.21.15.123:0/353145530 shutdown_connections 2024-09-16T15:50:17.430 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:17.430+0000 7f00175f6640 1 -- 172.21.15.123:0/353145530 wait complete. 2024-09-16T15:50:17.430 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:17.430+0000 7f00175f6640 1 Processor -- start 2024-09-16T15:50:17.430 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:17.430+0000 7f00175f6640 1 -- start start 2024-09-16T15:50:17.432 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:17.430+0000 7f00175f6640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7efffc003a10 0x7f0010147110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:50:17.432 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:17.430+0000 7f00175f6640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f00101450d0 0x7f00100cf220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:50:17.432 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:17.430+0000 7f00175f6640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f0010144840 0x7f00100cdd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:50:17.432 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:17.430+0000 7f00175f6640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f00100cad00 con 0x7f00101450d0 2024-09-16T15:50:17.432 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:17.430+0000 7f00175f6640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f00100cab80 con 0x7f0010144840 2024-09-16T15:50:17.432 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:17.430+0000 7f00175f6640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f00100cae80 con 0x7efffc003a10 2024-09-16T15:50:17.433 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:17.430+0000 7f0014b91640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f00101450d0 0x7f00100cf220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:50:17.433 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:17.430+0000 7f0014b91640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f00101450d0 0x7f00100cf220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:37210/0 (socket says 172.21.15.123:37210) 2024-09-16T15:50:17.433 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:17.430+0000 7f0014b91640 1 -- 172.21.15.123:0/3004139070 learned_addr learned my addr 172.21.15.123:0/3004139070 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:50:17.433 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:17.430+0000 7f0015392640 1 --2- 172.21.15.123:0/3004139070 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7efffc003a10 0x7f0010147110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:50:17.433 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:17.430+0000 7f0015b93640 1 --2- 172.21.15.123:0/3004139070 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f0010144840 0x7f00100cdd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:50:17.433 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:17.430+0000 7f0014b91640 1 -- 172.21.15.123:0/3004139070 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7efffc003a10 msgr2=0x7f0010147110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:50:17.433 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:17.430+0000 7f0014b91640 1 --2- 172.21.15.123:0/3004139070 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7efffc003a10 0x7f0010147110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:17.433 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:17.430+0000 7f0014b91640 1 -- 172.21.15.123:0/3004139070 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f0010144840 msgr2=0x7f00100cdd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:50:17.433 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:17.430+0000 7f0014b91640 1 --2- 172.21.15.123:0/3004139070 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f0010144840 0x7f00100cdd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:17.433 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:17.430+0000 7f0014b91640 1 -- 172.21.15.123:0/3004139070 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f00101480c0 con 0x7f00101450d0 2024-09-16T15:50:17.433 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:17.430+0000 7f0014b91640 1 --2- 172.21.15.123:0/3004139070 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f00101450d0 0x7f00100cf220 secure :-1 s=READY pgs=1516 cs=0 l=1 rev1=1 crypto rx=0x7f000400c7c0 tx=0x7f000400cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:50:17.433 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:17.430+0000 7f0001ffb640 1 -- 172.21.15.123:0/3004139070 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0004005480 con 0x7f00101450d0 2024-09-16T15:50:17.433 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:17.430+0000 7f0001ffb640 1 -- 172.21.15.123:0/3004139070 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f0004003600 con 0x7f00101450d0 2024-09-16T15:50:17.434 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:17.434+0000 7f00175f6640 1 -- 172.21.15.123:0/3004139070 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0010145800 con 0x7f00101450d0 2024-09-16T15:50:17.434 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:17.434+0000 7f0001ffb640 1 -- 172.21.15.123:0/3004139070 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0004003a30 con 0x7f00101450d0 2024-09-16T15:50:17.434 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:17.434+0000 7f00175f6640 1 -- 172.21.15.123:0/3004139070 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0010147820 con 0x7f00101450d0 2024-09-16T15:50:17.434 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:17.434+0000 7f00175f6640 1 -- 172.21.15.123:0/3004139070 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f001006e980 con 0x7f00101450d0 2024-09-16T15:50:17.435 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:17.434+0000 7f0001ffb640 1 -- 172.21.15.123:0/3004139070 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 22) ==== 97692+0+0 (secure 0 0 0) 0x7f000400a5e0 con 0x7f00101450d0 2024-09-16T15:50:17.435 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:17.434+0000 7f0001ffb640 1 --2- 172.21.15.123:0/3004139070 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7efff4073580 0x7efff4075a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:50:17.435 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:17.434+0000 7f0001ffb640 1 -- 172.21.15.123:0/3004139070 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(135..135 src has 1..135) ==== 5737+0+0 (secure 0 0 0) 0x7f0004095240 con 0x7f00101450d0 2024-09-16T15:50:17.435 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:17.434+0000 7f0015392640 1 --2- 172.21.15.123:0/3004139070 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7efff4073580 0x7efff4075a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:50:17.435 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:17.434+0000 7f0015392640 1 --2- 172.21.15.123:0/3004139070 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7efff4073580 0x7efff4075a40 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f000c0b0000 tx=0x7f000c019b10 comp rx=0 tx=0).ready entity=mgr.4150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:50:17.437 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:17.438+0000 7f0001ffb640 1 -- 172.21.15.123:0/3004139070 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f000405fbd0 con 0x7f00101450d0 2024-09-16T15:50:17.629 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:17.630+0000 7f00175f6640 1 -- 172.21.15.123:0/3004139070 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f001015f630 con 0x7f00101450d0 2024-09-16T15:50:17.630 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:17.630+0000 7f0001ffb640 1 -- 172.21.15.123:0/3004139070 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 87 v87) ==== 94+0+5230 (secure 0 0 0) 0x7f000405f4d0 con 0x7f00101450d0 2024-09-16T15:50:17.631 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-09-16T15:50:17.631 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":87,"btime":"2024-09-16T15:50:06:171067+0000","default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5801,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6834/730567479","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":730567479},{"type":"v1","addr":"172.21.15.184:6834","nonce":730567479}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":81},{"gid":6346,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6835/584518652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6833","nonce":584518652},{"type":"v1","addr":"172.21.15.184:6835","nonce":584518652}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":81},{"gid":7938,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6833/3935767668","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6832","nonce":3935767668},{"type":"v1","addr":"172.21.15.123:6833","nonce":3935767668}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":80}],"filesystems":[{"mdsmap":{"epoch":86,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-16T15:49:53.127484+0000","modified":"2024-09-16T15:49:56.160938+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7944},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7944":{"gid":7944,"name":"a","rank":0,"incarnation":82,"state":"up:active","state_seq":3,"addr":"172.21.15.123:6835/333722639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6834","nonce":333722639},{"type":"v1","addr":"172.21.15.123:6835","nonce":333722639}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7944,"qdb_cluster":[7944]},"id":10}]} 2024-09-16T15:50:17.631 INFO:teuthology.orchestra.run.smithi123.stderr:dumped fsmap epoch 87 2024-09-16T15:50:17.633 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:17.634+0000 7f00175f6640 1 -- 172.21.15.123:0/3004139070 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7efff4073580 msgr2=0x7efff4075a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:50:17.633 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:17.634+0000 7f00175f6640 1 --2- 172.21.15.123:0/3004139070 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7efff4073580 0x7efff4075a40 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f000c0b0000 tx=0x7f000c019b10 comp rx=0 tx=0).stop 2024-09-16T15:50:17.633 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:17.634+0000 7f00175f6640 1 -- 172.21.15.123:0/3004139070 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f00101450d0 msgr2=0x7f00100cf220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:50:17.633 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:17.634+0000 7f00175f6640 1 --2- 172.21.15.123:0/3004139070 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f00101450d0 0x7f00100cf220 secure :-1 s=READY pgs=1516 cs=0 l=1 rev1=1 crypto rx=0x7f000400c7c0 tx=0x7f000400cc80 comp rx=0 tx=0).stop 2024-09-16T15:50:17.633 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:17.634+0000 7f00175f6640 1 -- 172.21.15.123:0/3004139070 shutdown_connections 2024-09-16T15:50:17.633 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:17.634+0000 7f00175f6640 1 --2- 172.21.15.123:0/3004139070 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7efff4073580 0x7efff4075a40 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:17.633 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:17.634+0000 7f00175f6640 1 --2- 172.21.15.123:0/3004139070 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f0010144840 0x7f00100cdd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:17.633 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:17.634+0000 7f00175f6640 1 --2- 172.21.15.123:0/3004139070 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f00101450d0 0x7f00100cf220 unknown :-1 s=CLOSED pgs=1516 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:17.634 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:17.634+0000 7f00175f6640 1 --2- 172.21.15.123:0/3004139070 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7efffc003a10 0x7f0010147110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:17.634 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:17.634+0000 7f00175f6640 1 -- 172.21.15.123:0/3004139070 >> 172.21.15.123:0/3004139070 conn(0x7f00100cfb10 msgr2=0x7f00100c3cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:50:17.634 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:17.634+0000 7f00175f6640 1 -- 172.21.15.123:0/3004139070 shutdown_connections 2024-09-16T15:50:17.634 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:17.634+0000 7f00175f6640 1 -- 172.21.15.123:0/3004139070 wait complete. 2024-09-16T15:50:17.648 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-16T15:50:17.762 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:17.762+0000 7f9ba87ce640 1 -- 172.21.15.123:0/2077174686 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9b88004d50 msgr2=0x7f9b880051d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:50:17.762 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:17.762+0000 7f9ba87ce640 1 --2- 172.21.15.123:0/2077174686 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9b88004d50 0x7f9b880051d0 secure :-1 s=READY pgs=1518 cs=0 l=1 rev1=1 crypto rx=0x7f9b9800c290 tx=0x7f9b9801cf40 comp rx=0 tx=0).stop 2024-09-16T15:50:17.762 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:17.762+0000 7f9ba6d6b640 1 -- 172.21.15.123:0/2077174686 reap_dead start 2024-09-16T15:50:17.762 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:17.762+0000 7f9ba87ce640 1 -- 172.21.15.123:0/2077174686 shutdown_connections 2024-09-16T15:50:17.762 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:17.762+0000 7f9ba87ce640 1 -- 172.21.15.123:0/2077174686 >> 172.21.15.123:0/2077174686 conn(0x7f9ba00cfb10 msgr2=0x7f9ba00cff20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:50:17.762 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:17.762+0000 7f9ba87ce640 1 -- 172.21.15.123:0/2077174686 shutdown_connections 2024-09-16T15:50:17.762 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:17.762+0000 7f9ba87ce640 1 -- 172.21.15.123:0/2077174686 wait complete. 2024-09-16T15:50:17.763 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:17.762+0000 7f9ba87ce640 1 Processor -- start 2024-09-16T15:50:17.763 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:17.762+0000 7f9ba87ce640 1 -- start start 2024-09-16T15:50:17.764 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:17.762+0000 7f9ba87ce640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f9b88004d50 0x7f9ba0147820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:50:17.765 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:17.762+0000 7f9ba87ce640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9ba014ae50 0x7f9ba01444a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:50:17.765 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:17.762+0000 7f9ba87ce640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f9ba0144db0 0x7f9ba0157060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:50:17.765 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:17.762+0000 7f9ba87ce640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f9ba00cf5b0 con 0x7f9ba014ae50 2024-09-16T15:50:17.765 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:17.762+0000 7f9ba87ce640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f9ba00cf430 con 0x7f9b88004d50 2024-09-16T15:50:17.765 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:17.762+0000 7f9ba87ce640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f9ba006f6f0 con 0x7f9ba0144db0 2024-09-16T15:50:17.765 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:17.762+0000 7f9ba5d69640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9ba014ae50 0x7f9ba01444a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:50:17.765 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:17.762+0000 7f9ba5d69640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9ba014ae50 0x7f9ba01444a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:37236/0 (socket says 172.21.15.123:37236) 2024-09-16T15:50:17.765 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:17.762+0000 7f9ba5d69640 1 -- 172.21.15.123:0/3299215945 learned_addr learned my addr 172.21.15.123:0/3299215945 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:50:17.765 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:17.766+0000 7f9ba656a640 1 --2- 172.21.15.123:0/3299215945 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f9b88004d50 0x7f9ba0147820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:50:17.765 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:17.766+0000 7f9ba6d6b640 1 --2- 172.21.15.123:0/3299215945 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f9ba0144db0 0x7f9ba0157060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:50:17.765 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:17.766+0000 7f9ba5d69640 1 -- 172.21.15.123:0/3299215945 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f9ba0144db0 msgr2=0x7f9ba0157060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:50:17.765 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:17.766+0000 7f9ba5d69640 1 --2- 172.21.15.123:0/3299215945 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f9ba0144db0 0x7f9ba0157060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:17.765 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:17.766+0000 7f9ba5d69640 1 -- 172.21.15.123:0/3299215945 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f9b88004d50 msgr2=0x7f9ba0147820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:50:17.765 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:17.766+0000 7f9ba5d69640 1 --2- 172.21.15.123:0/3299215945 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f9b88004d50 0x7f9ba0147820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:17.766 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:17.766+0000 7f9ba5d69640 1 -- 172.21.15.123:0/3299215945 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9ba00c6d80 con 0x7f9ba014ae50 2024-09-16T15:50:17.766 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:17.766+0000 7f9ba5d69640 1 --2- 172.21.15.123:0/3299215945 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9ba014ae50 0x7f9ba01444a0 secure :-1 s=READY pgs=1519 cs=0 l=1 rev1=1 crypto rx=0x7f9b9400c890 tx=0x7f9b9400cd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:50:17.766 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:17.766+0000 7f9b9effd640 1 -- 172.21.15.123:0/3299215945 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b94017070 con 0x7f9ba014ae50 2024-09-16T15:50:17.766 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:17.766+0000 7f9b9effd640 1 -- 172.21.15.123:0/3299215945 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f9b940050f0 con 0x7f9ba014ae50 2024-09-16T15:50:17.766 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:17.766+0000 7f9ba87ce640 1 -- 172.21.15.123:0/3299215945 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9ba00c7060 con 0x7f9ba014ae50 2024-09-16T15:50:17.766 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:17.766+0000 7f9b9effd640 1 -- 172.21.15.123:0/3299215945 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b94005540 con 0x7f9ba014ae50 2024-09-16T15:50:17.766 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:17.766+0000 7f9ba87ce640 1 -- 172.21.15.123:0/3299215945 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9ba014ab90 con 0x7f9ba014ae50 2024-09-16T15:50:17.767 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:17.766+0000 7f9ba87ce640 1 -- 172.21.15.123:0/3299215945 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9ba006fc50 con 0x7f9ba014ae50 2024-09-16T15:50:17.767 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:17.766+0000 7f9b9effd640 1 -- 172.21.15.123:0/3299215945 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 22) ==== 97692+0+0 (secure 0 0 0) 0x7f9b9401a070 con 0x7f9ba014ae50 2024-09-16T15:50:17.767 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:17.766+0000 7f9b9effd640 1 --2- 172.21.15.123:0/3299215945 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f9b8c073580 0x7f9b8c075a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:50:17.767 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:17.766+0000 7f9b9effd640 1 -- 172.21.15.123:0/3299215945 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(135..135 src has 1..135) ==== 5737+0+0 (secure 0 0 0) 0x7f9b94094670 con 0x7f9ba014ae50 2024-09-16T15:50:17.767 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:17.766+0000 7f9ba656a640 1 --2- 172.21.15.123:0/3299215945 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f9b8c073580 0x7f9b8c075a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:50:17.767 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:17.766+0000 7f9ba656a640 1 --2- 172.21.15.123:0/3299215945 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f9b8c073580 0x7f9b8c075a40 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f9b900058d0 tx=0x7f9b90005860 comp rx=0 tx=0).ready entity=mgr.4150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:50:17.770 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:17.770+0000 7f9b9effd640 1 -- 172.21.15.123:0/3299215945 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f9b94021020 con 0x7f9ba014ae50 2024-09-16T15:50:17.932 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:17.930+0000 7f9ba87ce640 1 -- 172.21.15.123:0/3299215945 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f9ba0050f80 con 0x7f9ba014ae50 2024-09-16T15:50:17.934 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:17.934+0000 7f9b9effd640 1 -- 172.21.15.123:0/3299215945 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v135) ==== 74+0+16843 (secure 0 0 0) 0x7f9b9405ef00 con 0x7f9ba014ae50 2024-09-16T15:50:17.934 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-09-16T15:50:17.934 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":135,"fsid":"56077f89-18c8-486f-ae7e-eecc0bb7822c","created":"2024-09-16T15:42:26.364892+0000","modified":"2024-09-16T15:49:54.741464+0000","last_up_change":"2024-09-16T15:42:38.707234+0000","last_in_change":"2024-09-16T15:42:29.423662+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":21,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-16T15:42:39.287764+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":20,"pool_name":"cephfs_metadata","create_time":"2024-09-16T15:49:50.809741+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"134","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":21,"pool_name":"cephfs_data","create_time":"2024-09-16T15:49:52.080476+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"135","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9e7c1955-a607-4671-9be7-f2ef760ceeaa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":133,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6800","nonce":3882044969},{"type":"v1","addr":"172.21.15.123:6801","nonce":3882044969}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6802","nonce":3882044969},{"type":"v1","addr":"172.21.15.123:6803","nonce":3882044969}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6806","nonce":3882044969},{"type":"v1","addr":"172.21.15.123:6807","nonce":3882044969}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6804","nonce":3882044969},{"type":"v1","addr":"172.21.15.123:6805","nonce":3882044969}]},"public_addr":"172.21.15.123:6801/3882044969","cluster_addr":"172.21.15.123:6803/3882044969","heartbeat_back_addr":"172.21.15.123:6807/3882044969","heartbeat_front_addr":"172.21.15.123:6805/3882044969","state":["exists","up"]},{"osd":1,"uuid":"c540cfde-0977-4742-9b17-fd634668e010","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":133,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6816","nonce":4037001850},{"type":"v1","addr":"172.21.15.123:6817","nonce":4037001850}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6818","nonce":4037001850},{"type":"v1","addr":"172.21.15.123:6819","nonce":4037001850}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6822","nonce":4037001850},{"type":"v1","addr":"172.21.15.123:6823","nonce":4037001850}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6820","nonce":4037001850},{"type":"v1","addr":"172.21.15.123:6821","nonce":4037001850}]},"public_addr":"172.21.15.123:6817/4037001850","cluster_addr":"172.21.15.123:6819/4037001850","heartbeat_back_addr":"172.21.15.123:6823/4037001850","heartbeat_front_addr":"172.21.15.123:6821/4037001850","state":["exists","up"]},{"osd":2,"uuid":"9864a496-f169-486b-95c4-1d59162793a7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":133,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6824","nonce":1332143088},{"type":"v1","addr":"172.21.15.123:6825","nonce":1332143088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6826","nonce":1332143088},{"type":"v1","addr":"172.21.15.123:6827","nonce":1332143088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6830","nonce":1332143088},{"type":"v1","addr":"172.21.15.123:6831","nonce":1332143088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6828","nonce":1332143088},{"type":"v1","addr":"172.21.15.123:6829","nonce":1332143088}]},"public_addr":"172.21.15.123:6825/1332143088","cluster_addr":"172.21.15.123:6827/1332143088","heartbeat_back_addr":"172.21.15.123:6831/1332143088","heartbeat_front_addr":"172.21.15.123:6829/1332143088","state":["exists","up"]},{"osd":3,"uuid":"c6d1f817-f074-4325-95e1-578eba13edda","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":133,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6808","nonce":988586998},{"type":"v1","addr":"172.21.15.123:6809","nonce":988586998}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6810","nonce":988586998},{"type":"v1","addr":"172.21.15.123:6811","nonce":988586998}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6814","nonce":988586998},{"type":"v1","addr":"172.21.15.123:6815","nonce":988586998}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6812","nonce":988586998},{"type":"v1","addr":"172.21.15.123:6813","nonce":988586998}]},"public_addr":"172.21.15.123:6809/988586998","cluster_addr":"172.21.15.123:6811/988586998","heartbeat_back_addr":"172.21.15.123:6815/988586998","heartbeat_front_addr":"172.21.15.123:6813/988586998","state":["exists","up"]},{"osd":4,"uuid":"200a8df4-50ef-40a5-b48a-58981269c086","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":133,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6816","nonce":3387813890},{"type":"v1","addr":"172.21.15.184:6817","nonce":3387813890}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6818","nonce":3387813890},{"type":"v1","addr":"172.21.15.184:6819","nonce":3387813890}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6822","nonce":3387813890},{"type":"v1","addr":"172.21.15.184:6823","nonce":3387813890}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6820","nonce":3387813890},{"type":"v1","addr":"172.21.15.184:6821","nonce":3387813890}]},"public_addr":"172.21.15.184:6817/3387813890","cluster_addr":"172.21.15.184:6819/3387813890","heartbeat_back_addr":"172.21.15.184:6823/3387813890","heartbeat_front_addr":"172.21.15.184:6821/3387813890","state":["exists","up"]},{"osd":5,"uuid":"32d5a900-da3d-412c-bc19-13861a2368d8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":133,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6808","nonce":2680916017},{"type":"v1","addr":"172.21.15.184:6809","nonce":2680916017}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6810","nonce":2680916017},{"type":"v1","addr":"172.21.15.184:6811","nonce":2680916017}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6814","nonce":2680916017},{"type":"v1","addr":"172.21.15.184:6815","nonce":2680916017}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6812","nonce":2680916017},{"type":"v1","addr":"172.21.15.184:6813","nonce":2680916017}]},"public_addr":"172.21.15.184:6809/2680916017","cluster_addr":"172.21.15.184:6811/2680916017","heartbeat_back_addr":"172.21.15.184:6815/2680916017","heartbeat_front_addr":"172.21.15.184:6813/2680916017","state":["exists","up"]},{"osd":6,"uuid":"cde6f39e-aa1b-4260-ac19-774ee6aaca72","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":133,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6824","nonce":2183120457},{"type":"v1","addr":"172.21.15.184:6825","nonce":2183120457}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6826","nonce":2183120457},{"type":"v1","addr":"172.21.15.184:6827","nonce":2183120457}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6830","nonce":2183120457},{"type":"v1","addr":"172.21.15.184:6831","nonce":2183120457}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6828","nonce":2183120457},{"type":"v1","addr":"172.21.15.184:6829","nonce":2183120457}]},"public_addr":"172.21.15.184:6825/2183120457","cluster_addr":"172.21.15.184:6827/2183120457","heartbeat_back_addr":"172.21.15.184:6831/2183120457","heartbeat_front_addr":"172.21.15.184:6829/2183120457","state":["exists","up"]},{"osd":7,"uuid":"4f381778-eee1-47c6-804f-df0e6b33e313","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":133,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6800","nonce":4102957285},{"type":"v1","addr":"172.21.15.184:6801","nonce":4102957285}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6802","nonce":4102957285},{"type":"v1","addr":"172.21.15.184:6803","nonce":4102957285}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6806","nonce":4102957285},{"type":"v1","addr":"172.21.15.184:6807","nonce":4102957285}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6804","nonce":4102957285},{"type":"v1","addr":"172.21.15.184:6805","nonce":4102957285}]},"public_addr":"172.21.15.184:6801/4102957285","cluster_addr":"172.21.15.184:6803/4102957285","heartbeat_back_addr":"172.21.15.184:6807/4102957285","heartbeat_front_addr":"172.21.15.184:6805/4102957285","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:35.972210+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.222007+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.198617+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.385663+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.358935+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.156066+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.752973+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:35.682846+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":17,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T15:50:17.936 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:17.938+0000 7f9ba87ce640 1 -- 172.21.15.123:0/3299215945 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f9b8c073580 msgr2=0x7f9b8c075a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:50:17.937 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:17.938+0000 7f9ba87ce640 1 --2- 172.21.15.123:0/3299215945 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f9b8c073580 0x7f9b8c075a40 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f9b900058d0 tx=0x7f9b90005860 comp rx=0 tx=0).stop 2024-09-16T15:50:17.937 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:17.938+0000 7f9ba87ce640 1 -- 172.21.15.123:0/3299215945 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9ba014ae50 msgr2=0x7f9ba01444a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:50:17.937 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:17.938+0000 7f9ba87ce640 1 --2- 172.21.15.123:0/3299215945 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9ba014ae50 0x7f9ba01444a0 secure :-1 s=READY pgs=1519 cs=0 l=1 rev1=1 crypto rx=0x7f9b9400c890 tx=0x7f9b9400cd50 comp rx=0 tx=0).stop 2024-09-16T15:50:17.937 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:17.938+0000 7f9ba87ce640 1 -- 172.21.15.123:0/3299215945 shutdown_connections 2024-09-16T15:50:17.937 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:17.938+0000 7f9ba87ce640 1 --2- 172.21.15.123:0/3299215945 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f9b8c073580 0x7f9b8c075a40 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:17.937 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:17.938+0000 7f9ba87ce640 1 --2- 172.21.15.123:0/3299215945 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f9ba0144db0 0x7f9ba0157060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:17.937 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:17.938+0000 7f9ba87ce640 1 --2- 172.21.15.123:0/3299215945 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9ba014ae50 0x7f9ba01444a0 unknown :-1 s=CLOSED pgs=1519 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:17.937 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:17.938+0000 7f9ba87ce640 1 --2- 172.21.15.123:0/3299215945 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f9b88004d50 0x7f9ba0147820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:17.937 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:17.938+0000 7f9ba87ce640 1 -- 172.21.15.123:0/3299215945 >> 172.21.15.123:0/3299215945 conn(0x7f9ba00cfb10 msgr2=0x7f9ba00c3640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:50:17.937 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:17.938+0000 7f9ba87ce640 1 -- 172.21.15.123:0/3299215945 shutdown_connections 2024-09-16T15:50:17.937 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:17.938+0000 7f9ba87ce640 1 -- 172.21.15.123:0/3299215945 wait complete. 2024-09-16T15:50:17.952 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-09-16T15:50:18.069 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:18.070+0000 7f7d5ba2b640 1 -- 172.21.15.123:0/890753081 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7d40004da0 msgr2=0x7f7d40005220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:50:18.069 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:18.070+0000 7f7d5ba2b640 1 --2- 172.21.15.123:0/890753081 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7d40004da0 0x7f7d40005220 secure :-1 s=READY pgs=1521 cs=0 l=1 rev1=1 crypto rx=0x7f7d4c0013e0 tx=0x7f7d4c01f9a0 comp rx=0 tx=0).stop 2024-09-16T15:50:18.069 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:18.070+0000 7f7d59fc8640 1 -- 172.21.15.123:0/890753081 reap_dead start 2024-09-16T15:50:18.069 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:18.070+0000 7f7d5ba2b640 1 -- 172.21.15.123:0/890753081 shutdown_connections 2024-09-16T15:50:18.069 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:18.070+0000 7f7d5ba2b640 1 -- 172.21.15.123:0/890753081 >> 172.21.15.123:0/890753081 conn(0x7f7d5409aa40 msgr2=0x7f7d5409ae50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:50:18.069 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:18.070+0000 7f7d5ba2b640 1 -- 172.21.15.123:0/890753081 shutdown_connections 2024-09-16T15:50:18.070 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:18.070+0000 7f7d5ba2b640 1 -- 172.21.15.123:0/890753081 wait complete. 2024-09-16T15:50:18.070 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:18.070+0000 7f7d5ba2b640 1 Processor -- start 2024-09-16T15:50:18.070 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:18.070+0000 7f7d5ba2b640 1 -- start start 2024-09-16T15:50:18.072 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:18.070+0000 7f7d5ba2b640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f7d40004da0 0x7f7d541f5dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:50:18.072 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:18.070+0000 7f7d5ba2b640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7d54147d40 0x7f7d541481c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:50:18.072 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:18.070+0000 7f7d5ba2b640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7d540c78d0 0x7f7d540c7d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:50:18.072 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:18.070+0000 7f7d5ba2b640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f7d540af2f0 con 0x7f7d540c78d0 2024-09-16T15:50:18.073 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:18.070+0000 7f7d5ba2b640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f7d540b2700 con 0x7f7d54147d40 2024-09-16T15:50:18.073 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:18.070+0000 7f7d5ba2b640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f7d540af470 con 0x7f7d40004da0 2024-09-16T15:50:18.073 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:18.070+0000 7f7d59fc8640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7d540c78d0 0x7f7d540c7d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:50:18.073 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:18.070+0000 7f7d59fc8640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7d540c78d0 0x7f7d540c7d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:37252/0 (socket says 172.21.15.123:37252) 2024-09-16T15:50:18.073 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:18.070+0000 7f7d59fc8640 1 -- 172.21.15.123:0/1550497963 learned_addr learned my addr 172.21.15.123:0/1550497963 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:50:18.073 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:18.070+0000 7f7d597c7640 1 --2- 172.21.15.123:0/1550497963 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f7d40004da0 0x7f7d541f5dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:50:18.073 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:18.070+0000 7f7d58fc6640 1 --2- 172.21.15.123:0/1550497963 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7d54147d40 0x7f7d541481c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:50:18.073 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:18.070+0000 7f7d59fc8640 1 -- 172.21.15.123:0/1550497963 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f7d40004da0 msgr2=0x7f7d541f5dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:50:18.073 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:18.070+0000 7f7d59fc8640 1 --2- 172.21.15.123:0/1550497963 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f7d40004da0 0x7f7d541f5dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:18.073 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:18.070+0000 7f7d59fc8640 1 -- 172.21.15.123:0/1550497963 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7d54147d40 msgr2=0x7f7d541481c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:50:18.073 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:18.070+0000 7f7d59fc8640 1 --2- 172.21.15.123:0/1550497963 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7d54147d40 0x7f7d541481c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:18.073 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:18.070+0000 7f7d59fc8640 1 -- 172.21.15.123:0/1550497963 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7d540c8630 con 0x7f7d540c78d0 2024-09-16T15:50:18.074 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:18.070+0000 7f7d58fc6640 1 --2- 172.21.15.123:0/1550497963 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7d54147d40 0x7f7d541481c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T15:50:18.074 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:18.070+0000 7f7d597c7640 1 --2- 172.21.15.123:0/1550497963 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f7d40004da0 0x7f7d541f5dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T15:50:18.074 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:18.070+0000 7f7d59fc8640 1 --2- 172.21.15.123:0/1550497963 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7d540c78d0 0x7f7d540c7d60 secure :-1 s=READY pgs=1522 cs=0 l=1 rev1=1 crypto rx=0x7f7d4c020240 tx=0x7f7d4c01fbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:50:18.074 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:18.074+0000 7f7d49ffb640 1 -- 172.21.15.123:0/1550497963 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d4c004e40 con 0x7f7d540c78d0 2024-09-16T15:50:18.074 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:18.074+0000 7f7d49ffb640 1 -- 172.21.15.123:0/1550497963 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f7d4c001210 con 0x7f7d540c78d0 2024-09-16T15:50:18.074 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:18.074+0000 7f7d5ba2b640 1 -- 172.21.15.123:0/1550497963 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7d540b5b60 con 0x7f7d540c78d0 2024-09-16T15:50:18.074 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:18.074+0000 7f7d49ffb640 1 -- 172.21.15.123:0/1550497963 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d4c001a80 con 0x7f7d540c78d0 2024-09-16T15:50:18.074 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:18.074+0000 7f7d5ba2b640 1 -- 172.21.15.123:0/1550497963 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7d540b6120 con 0x7f7d540c78d0 2024-09-16T15:50:18.075 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:18.074+0000 7f7d5ba2b640 1 -- 172.21.15.123:0/1550497963 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7d540d1b50 con 0x7f7d540c78d0 2024-09-16T15:50:18.075 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:18.074+0000 7f7d49ffb640 1 -- 172.21.15.123:0/1550497963 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 22) ==== 97692+0+0 (secure 0 0 0) 0x7f7d4c0bb030 con 0x7f7d540c78d0 2024-09-16T15:50:18.075 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:18.074+0000 7f7d49ffb640 1 --2- 172.21.15.123:0/1550497963 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f7d380735b0 0x7f7d38075a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:50:18.075 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:18.074+0000 7f7d49ffb640 1 -- 172.21.15.123:0/1550497963 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(135..135 src has 1..135) ==== 5737+0+0 (secure 0 0 0) 0x7f7d4c1314b0 con 0x7f7d540c78d0 2024-09-16T15:50:18.075 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:18.074+0000 7f7d597c7640 1 --2- 172.21.15.123:0/1550497963 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f7d380735b0 0x7f7d38075a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:50:18.075 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:18.074+0000 7f7d597c7640 1 --2- 172.21.15.123:0/1550497963 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f7d380735b0 0x7f7d38075a70 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f7d50005d90 tx=0x7f7d50005d20 comp rx=0 tx=0).ready entity=mgr.4150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:50:18.078 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:18.078+0000 7f7d49ffb640 1 -- 172.21.15.123:0/1550497963 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f7d4c019750 con 0x7f7d540c78d0 2024-09-16T15:50:18.286 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:18.286+0000 7f7d5ba2b640 1 -- 172.21.15.123:0/1550497963 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7f7d540708d0 con 0x7f7d540c78d0 2024-09-16T15:50:18.287 INFO:tasks.ceph.mon.a.smithi123.stderr:2024-09-16T15:50:18.286+0000 7f268d076640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-09-16T15:50:18.289 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:18.290+0000 7f7d49ffb640 1 -- 172.21.15.123:0/1550497963 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v88) ==== 161+0+0 (secure 0 0 0) 0x7f7d4c0fbb20 con 0x7f7d540c78d0 2024-09-16T15:50:18.289 INFO:teuthology.orchestra.run.smithi123.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-09-16T15:50:18.294 INFO:tasks.ceph.mds.a.smithi123.stderr: -1762> 2024-09-16T15:49:55.710+0000 7f145ca6e640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-09-16T15:50:18.294 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:18.294+0000 7f7d5ba2b640 1 -- 172.21.15.123:0/1550497963 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f7d380735b0 msgr2=0x7f7d38075a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:50:18.294 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:18.294+0000 7f7d5ba2b640 1 --2- 172.21.15.123:0/1550497963 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f7d380735b0 0x7f7d38075a70 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f7d50005d90 tx=0x7f7d50005d20 comp rx=0 tx=0).stop 2024-09-16T15:50:18.294 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:18.294+0000 7f7d5ba2b640 1 -- 172.21.15.123:0/1550497963 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7d540c78d0 msgr2=0x7f7d540c7d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:50:18.295 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:18.294+0000 7f7d5ba2b640 1 --2- 172.21.15.123:0/1550497963 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7d540c78d0 0x7f7d540c7d60 secure :-1 s=READY pgs=1522 cs=0 l=1 rev1=1 crypto rx=0x7f7d4c020240 tx=0x7f7d4c01fbf0 comp rx=0 tx=0).stop 2024-09-16T15:50:18.295 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:18.294+0000 7f7d5ba2b640 1 -- 172.21.15.123:0/1550497963 shutdown_connections 2024-09-16T15:50:18.295 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:18.294+0000 7f7d5ba2b640 1 --2- 172.21.15.123:0/1550497963 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f7d380735b0 0x7f7d38075a70 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:18.295 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:18.294+0000 7f7d5ba2b640 1 --2- 172.21.15.123:0/1550497963 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7d540c78d0 0x7f7d540c7d60 unknown :-1 s=CLOSED pgs=1522 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:18.295 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:18.294+0000 7f7d5ba2b640 1 --2- 172.21.15.123:0/1550497963 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7d54147d40 0x7f7d541481c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:18.295 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:18.294+0000 7f7d5ba2b640 1 --2- 172.21.15.123:0/1550497963 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f7d40004da0 0x7f7d541f5dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:18.295 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:18.294+0000 7f7d5ba2b640 1 -- 172.21.15.123:0/1550497963 >> 172.21.15.123:0/1550497963 conn(0x7f7d5409aa40 msgr2=0x7f7d5409ae50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:50:18.295 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:18.294+0000 7f7d5ba2b640 1 -- 172.21.15.123:0/1550497963 shutdown_connections 2024-09-16T15:50:18.295 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:18.294+0000 7f7d5ba2b640 1 -- 172.21.15.123:0/1550497963 wait complete. 2024-09-16T15:50:18.311 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-09-16T15:50:18.322 INFO:tasks.ceph.mds.a.smithi123.stdout:starting mds.a at 2024-09-16T15:50:18.434 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:18.434+0000 7f15a7eb0640 1 -- 172.21.15.123:0/1309884830 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f15a0150700 msgr2=0x7f15a0150b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:50:18.434 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:18.434+0000 7f15a7eb0640 1 --2- 172.21.15.123:0/1309884830 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f15a0150700 0x7f15a0150b50 secure :-1 s=READY pgs=1524 cs=0 l=1 rev1=1 crypto rx=0x7f15a00cd230 tx=0x7f159801e310 comp rx=0 tx=0).stop 2024-09-16T15:50:18.434 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:18.434+0000 7f15a7eb0640 1 -- 172.21.15.123:0/1309884830 shutdown_connections 2024-09-16T15:50:18.434 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:18.434+0000 7f15a7eb0640 1 --2- 172.21.15.123:0/1309884830 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f15a0150700 0x7f15a0150b50 unknown :-1 s=CLOSED pgs=1524 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:18.434 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:18.434+0000 7f15a7eb0640 1 --2- 172.21.15.123:0/1309884830 >> v2:172.21.15.184:3300/0 conn(0x7f15a00cf140 0x7f15a00cf580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:18.434 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:18.434+0000 7f15a7eb0640 1 --2- 172.21.15.123:0/1309884830 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f15a00cc550 0x7f15a00cc930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:18.434 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:18.434+0000 7f15a7eb0640 1 -- 172.21.15.123:0/1309884830 >> 172.21.15.123:0/1309884830 conn(0x7f15a00b1710 msgr2=0x7f15a00b1b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:50:18.434 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:18.434+0000 7f15a7eb0640 1 -- 172.21.15.123:0/1309884830 shutdown_connections 2024-09-16T15:50:18.435 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:18.434+0000 7f15a7eb0640 1 -- 172.21.15.123:0/1309884830 wait complete. 2024-09-16T15:50:18.435 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:18.434+0000 7f15a7eb0640 1 Processor -- start 2024-09-16T15:50:18.435 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:18.434+0000 7f15a7eb0640 1 -- start start 2024-09-16T15:50:18.437 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:18.434+0000 7f15a7eb0640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f15a00cc550 0x7f15a014dd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:50:18.437 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:18.434+0000 7f15a7eb0640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f15a00cf140 0x7f15a014ad90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:50:18.437 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:18.434+0000 7f15a7eb0640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f15a0150700 0x7f15a014b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:50:18.437 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:18.434+0000 7f15a7eb0640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f15a00cdaa0 con 0x7f15a0150700 2024-09-16T15:50:18.437 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:18.434+0000 7f15a7eb0640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f15a00cd920 con 0x7f15a00cf140 2024-09-16T15:50:18.437 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:18.434+0000 7f15a7eb0640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f15a00cdc20 con 0x7f15a00cc550 2024-09-16T15:50:18.437 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:18.434+0000 7f15a644d640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f15a0150700 0x7f15a014b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:50:18.437 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:18.434+0000 7f15a644d640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f15a0150700 0x7f15a014b2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:37282/0 (socket says 172.21.15.123:37282) 2024-09-16T15:50:18.437 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:18.438+0000 7f15a644d640 1 -- 172.21.15.123:0/3075329335 learned_addr learned my addr 172.21.15.123:0/3075329335 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:50:18.438 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:18.438+0000 7f15a5c4c640 1 --2- 172.21.15.123:0/3075329335 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f15a00cc550 0x7f15a014dd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:50:18.438 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:18.438+0000 7f15a544b640 1 --2- 172.21.15.123:0/3075329335 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f15a00cf140 0x7f15a014ad90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:50:18.438 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:18.438+0000 7f15a644d640 1 -- 172.21.15.123:0/3075329335 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f15a00cc550 msgr2=0x7f15a014dd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:50:18.438 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:18.438+0000 7f15a644d640 1 --2- 172.21.15.123:0/3075329335 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f15a00cc550 0x7f15a014dd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:18.438 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:18.438+0000 7f15a644d640 1 -- 172.21.15.123:0/3075329335 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f15a00cf140 msgr2=0x7f15a014ad90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:50:18.438 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:18.438+0000 7f15a644d640 1 --2- 172.21.15.123:0/3075329335 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f15a00cf140 0x7f15a014ad90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:18.438 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:18.438+0000 7f15a644d640 1 -- 172.21.15.123:0/3075329335 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f15a014bc20 con 0x7f15a0150700 2024-09-16T15:50:18.438 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:18.438+0000 7f15a544b640 1 --2- 172.21.15.123:0/3075329335 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f15a00cf140 0x7f15a014ad90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T15:50:18.438 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:18.438+0000 7f15a5c4c640 1 --2- 172.21.15.123:0/3075329335 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f15a00cc550 0x7f15a014dd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T15:50:18.438 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:18.438+0000 7f15a644d640 1 --2- 172.21.15.123:0/3075329335 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f15a0150700 0x7f15a014b2d0 secure :-1 s=READY pgs=1525 cs=0 l=1 rev1=1 crypto rx=0x7f159801e150 tx=0x7f15980039a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:50:18.438 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:18.438+0000 7f15967fc640 1 -- 172.21.15.123:0/3075329335 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15980ba020 con 0x7f15a0150700 2024-09-16T15:50:18.438 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:18.438+0000 7f15967fc640 1 -- 172.21.15.123:0/3075329335 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f15980a6650 con 0x7f15a0150700 2024-09-16T15:50:18.439 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:18.438+0000 7f15967fc640 1 -- 172.21.15.123:0/3075329335 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15980a6a80 con 0x7f15a0150700 2024-09-16T15:50:18.439 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:18.438+0000 7f15a7eb0640 1 -- 172.21.15.123:0/3075329335 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f15a01f8e20 con 0x7f15a0150700 2024-09-16T15:50:18.439 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:18.438+0000 7f15a7eb0640 1 -- 172.21.15.123:0/3075329335 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f15a01f93e0 con 0x7f15a0150700 2024-09-16T15:50:18.439 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:18.438+0000 7f15a7eb0640 1 -- 172.21.15.123:0/3075329335 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f15a00cf580 con 0x7f15a0150700 2024-09-16T15:50:18.440 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:18.438+0000 7f15967fc640 1 -- 172.21.15.123:0/3075329335 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 22) ==== 97692+0+0 (secure 0 0 0) 0x7f15980bd070 con 0x7f15a0150700 2024-09-16T15:50:18.440 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:18.438+0000 7f15967fc640 1 --2- 172.21.15.123:0/3075329335 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f1584073580 0x7f1584075a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:50:18.440 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:18.438+0000 7f15967fc640 1 -- 172.21.15.123:0/3075329335 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(136..136 src has 1..136) ==== 5823+0+0 (secure 0 0 0) 0x7f159812db10 con 0x7f15a0150700 2024-09-16T15:50:18.440 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:18.438+0000 7f15a5c4c640 1 --2- 172.21.15.123:0/3075329335 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f1584073580 0x7f1584075a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:50:18.440 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:18.438+0000 7f15a5c4c640 1 --2- 172.21.15.123:0/3075329335 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f1584073580 0x7f1584075a40 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f15900058d0 tx=0x7f1590005ca0 comp rx=0 tx=0).ready entity=mgr.4150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:50:18.443 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:18.442+0000 7f15967fc640 1 -- 172.21.15.123:0/3075329335 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f15980174a0 con 0x7f15a0150700 2024-09-16T15:50:18.646 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:18.646+0000 7f15a7eb0640 1 -- 172.21.15.123:0/3075329335 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) -- 0x7f15a00ccb30 con 0x7f15a0150700 2024-09-16T15:50:19.292 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:19.290+0000 7f15967fc640 1 -- 172.21.15.123:0/3075329335 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v89) ==== 104+0+0 (secure 0 0 0) 0x7f15980f8350 con 0x7f15a0150700 2024-09-16T15:50:19.296 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:19.294+0000 7f15a7eb0640 1 -- 172.21.15.123:0/3075329335 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f1584073580 msgr2=0x7f1584075a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:50:19.296 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:19.294+0000 7f15a7eb0640 1 --2- 172.21.15.123:0/3075329335 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f1584073580 0x7f1584075a40 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f15900058d0 tx=0x7f1590005ca0 comp rx=0 tx=0).stop 2024-09-16T15:50:19.296 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:19.294+0000 7f15a7eb0640 1 -- 172.21.15.123:0/3075329335 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f15a0150700 msgr2=0x7f15a014b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:50:19.296 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:19.294+0000 7f15a7eb0640 1 --2- 172.21.15.123:0/3075329335 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f15a0150700 0x7f15a014b2d0 secure :-1 s=READY pgs=1525 cs=0 l=1 rev1=1 crypto rx=0x7f159801e150 tx=0x7f15980039a0 comp rx=0 tx=0).stop 2024-09-16T15:50:19.296 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:19.294+0000 7f15a7eb0640 1 -- 172.21.15.123:0/3075329335 shutdown_connections 2024-09-16T15:50:19.296 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:19.294+0000 7f15a7eb0640 1 --2- 172.21.15.123:0/3075329335 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f1584073580 0x7f1584075a40 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:19.296 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:19.294+0000 7f15a7eb0640 1 --2- 172.21.15.123:0/3075329335 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f15a0150700 0x7f15a014b2d0 unknown :-1 s=CLOSED pgs=1525 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:19.296 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:19.294+0000 7f15a7eb0640 1 --2- 172.21.15.123:0/3075329335 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f15a00cf140 0x7f15a014ad90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:19.296 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:19.294+0000 7f15a7eb0640 1 --2- 172.21.15.123:0/3075329335 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f15a00cc550 0x7f15a014dd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:19.296 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:19.294+0000 7f15a7eb0640 1 -- 172.21.15.123:0/3075329335 >> 172.21.15.123:0/3075329335 conn(0x7f15a00b1710 msgr2=0x7f15a01592c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:50:19.296 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:19.294+0000 7f15a7eb0640 1 -- 172.21.15.123:0/3075329335 shutdown_connections 2024-09-16T15:50:19.296 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:19.294+0000 7f15a7eb0640 1 -- 172.21.15.123:0/3075329335 wait complete. 2024-09-16T15:50:19.318 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-09-16T15:50:19.491 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:19.490+0000 7f994e0bc640 1 -- 172.21.15.123:0/4069845462 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f994815e1d0 msgr2=0x7f994806f5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:50:19.491 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:19.490+0000 7f994e0bc640 1 --2- 172.21.15.123:0/4069845462 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f994815e1d0 0x7f994806f5a0 secure :-1 s=READY pgs=1527 cs=0 l=1 rev1=1 crypto rx=0x7f994000acd0 tx=0x7f99400206d0 comp rx=0 tx=0).stop 2024-09-16T15:50:19.491 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:19.490+0000 7f994e0bc640 1 -- 172.21.15.123:0/4069845462 shutdown_connections 2024-09-16T15:50:19.491 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:19.490+0000 7f994e0bc640 1 --2- 172.21.15.123:0/4069845462 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f994815e1d0 0x7f994806f5a0 unknown :-1 s=CLOSED pgs=1527 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:19.491 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:19.490+0000 7f994e0bc640 1 -- 172.21.15.123:0/4069845462 >> 172.21.15.123:0/4069845462 conn(0x7f99480b1630 msgr2=0x7f99480b1a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:50:19.491 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:19.490+0000 7f994e0bc640 1 -- 172.21.15.123:0/4069845462 shutdown_connections 2024-09-16T15:50:19.491 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:19.490+0000 7f994e0bc640 1 -- 172.21.15.123:0/4069845462 wait complete. 2024-09-16T15:50:19.492 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:19.494+0000 7f994e0bc640 1 Processor -- start 2024-09-16T15:50:19.492 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:19.494+0000 7f994e0bc640 1 -- start start 2024-09-16T15:50:19.494 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:19.494+0000 7f994e0bc640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f994815d7c0 0x7f99480c52a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:50:19.494 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:19.494+0000 7f994e0bc640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f994815dda0 0x7f99480cc5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:50:19.494 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:19.494+0000 7f994e0bc640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f994815e1d0 0x7f99480ccae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:50:19.494 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:19.494+0000 7f994e0bc640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f994806fae0 con 0x7f994815dda0 2024-09-16T15:50:19.494 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:19.494+0000 7f994e0bc640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f99480b3b00 con 0x7f994815d7c0 2024-09-16T15:50:19.494 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:19.494+0000 7f994e0bc640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f994806fc60 con 0x7f994815e1d0 2024-09-16T15:50:19.494 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:19.494+0000 7f993ffff640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f994815dda0 0x7f99480cc5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:50:19.494 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:19.494+0000 7f993ffff640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f994815dda0 0x7f99480cc5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:37314/0 (socket says 172.21.15.123:37314) 2024-09-16T15:50:19.494 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:19.494+0000 7f993ffff640 1 -- 172.21.15.123:0/2859777985 learned_addr learned my addr 172.21.15.123:0/2859777985 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:50:19.494 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:19.494+0000 7f993ffff640 1 -- 172.21.15.123:0/2859777985 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f994815e1d0 msgr2=0x7f99480ccae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:50:19.494 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:19.494+0000 7f993ffff640 1 --2- 172.21.15.123:0/2859777985 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f994815e1d0 0x7f99480ccae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:19.494 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:19.494+0000 7f993ffff640 1 -- 172.21.15.123:0/2859777985 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f994815d7c0 msgr2=0x7f99480c52a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:50:19.495 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:19.494+0000 7f99477fe640 1 --2- 172.21.15.123:0/2859777985 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f994815d7c0 0x7f99480c52a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:50:19.495 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:19.494+0000 7f993ffff640 1 --2- 172.21.15.123:0/2859777985 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f994815d7c0 0x7f99480c52a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:19.495 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:19.494+0000 7f993ffff640 1 -- 172.21.15.123:0/2859777985 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f99481f2cf0 con 0x7f994815dda0 2024-09-16T15:50:19.495 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:19.494+0000 7f993ffff640 1 --2- 172.21.15.123:0/2859777985 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f994815dda0 0x7f99480cc5a0 secure :-1 s=READY pgs=1528 cs=0 l=1 rev1=1 crypto rx=0x7f993000c480 tx=0x7f993000c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:50:19.495 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:19.494+0000 7f9944ff9640 1 -- 172.21.15.123:0/2859777985 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9930012830 con 0x7f994815dda0 2024-09-16T15:50:19.495 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:19.494+0000 7f9944ff9640 1 -- 172.21.15.123:0/2859777985 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f9930017070 con 0x7f994815dda0 2024-09-16T15:50:19.495 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:19.494+0000 7f994e0bc640 1 -- 172.21.15.123:0/2859777985 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f99481f2fd0 con 0x7f994815dda0 2024-09-16T15:50:19.495 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:19.494+0000 7f9944ff9640 1 -- 172.21.15.123:0/2859777985 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9930012d80 con 0x7f994815dda0 2024-09-16T15:50:19.495 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:19.494+0000 7f994e0bc640 1 -- 172.21.15.123:0/2859777985 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f99481f3590 con 0x7f994815dda0 2024-09-16T15:50:19.496 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:19.494+0000 7f994e0bc640 1 -- 172.21.15.123:0/2859777985 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9948070100 con 0x7f994815dda0 2024-09-16T15:50:19.496 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:19.494+0000 7f9944ff9640 1 -- 172.21.15.123:0/2859777985 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 22) ==== 97692+0+0 (secure 0 0 0) 0x7f9930007020 con 0x7f994815dda0 2024-09-16T15:50:19.496 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:19.498+0000 7f9944ff9640 1 --2- 172.21.15.123:0/2859777985 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f9928073580 0x7f9928075a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:50:19.496 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:19.498+0000 7f9944ff9640 1 -- 172.21.15.123:0/2859777985 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(136..136 src has 1..136) ==== 5823+0+0 (secure 0 0 0) 0x7f9930094f00 con 0x7f994815dda0 2024-09-16T15:50:19.496 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:19.498+0000 7f99477fe640 1 --2- 172.21.15.123:0/2859777985 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f9928073580 0x7f9928075a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:50:19.497 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:19.498+0000 7f99477fe640 1 --2- 172.21.15.123:0/2859777985 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f9928073580 0x7f9928075a40 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f9938007fd0 tx=0x7f9938009040 comp rx=0 tx=0).ready entity=mgr.4150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:50:19.499 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:19.498+0000 7f9944ff9640 1 -- 172.21.15.123:0/2859777985 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f993005f8d0 con 0x7f994815dda0 2024-09-16T15:50:19.669 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:19.670+0000 7f994e0bc640 1 -- 172.21.15.123:0/2859777985 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) -- 0x7f99481f3760 con 0x7f994815dda0 2024-09-16T15:50:20.300 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:20.298+0000 7f9944ff9640 1 -- 172.21.15.123:0/2859777985 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v137) ==== 181+0+0 (secure 0 0 0) 0x7f993005f1d0 con 0x7f994815dda0 2024-09-16T15:50:20.300 INFO:teuthology.orchestra.run.smithi123.stderr:pool 'cephfs_metadata' removed 2024-09-16T15:50:20.306 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:20.306+0000 7f994e0bc640 1 -- 172.21.15.123:0/2859777985 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f9928073580 msgr2=0x7f9928075a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:50:20.306 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:20.306+0000 7f994e0bc640 1 --2- 172.21.15.123:0/2859777985 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f9928073580 0x7f9928075a40 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f9938007fd0 tx=0x7f9938009040 comp rx=0 tx=0).stop 2024-09-16T15:50:20.306 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:20.306+0000 7f994e0bc640 1 -- 172.21.15.123:0/2859777985 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f994815dda0 msgr2=0x7f99480cc5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:50:20.307 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:20.306+0000 7f994e0bc640 1 --2- 172.21.15.123:0/2859777985 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f994815dda0 0x7f99480cc5a0 secure :-1 s=READY pgs=1528 cs=0 l=1 rev1=1 crypto rx=0x7f993000c480 tx=0x7f993000c940 comp rx=0 tx=0).stop 2024-09-16T15:50:20.307 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:20.306+0000 7f994e0bc640 1 -- 172.21.15.123:0/2859777985 shutdown_connections 2024-09-16T15:50:20.307 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:20.306+0000 7f994e0bc640 1 --2- 172.21.15.123:0/2859777985 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f9928073580 0x7f9928075a40 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:20.307 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:20.306+0000 7f994e0bc640 1 --2- 172.21.15.123:0/2859777985 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f994815e1d0 0x7f99480ccae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:20.307 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:20.306+0000 7f994e0bc640 1 --2- 172.21.15.123:0/2859777985 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f994815dda0 0x7f99480cc5a0 secure :-1 s=CLOSED pgs=1528 cs=0 l=1 rev1=1 crypto rx=0x7f993000c480 tx=0x7f993000c940 comp rx=0 tx=0).stop 2024-09-16T15:50:20.307 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:20.306+0000 7f994e0bc640 1 --2- 172.21.15.123:0/2859777985 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f994815d7c0 0x7f99480c52a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:20.308 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:20.306+0000 7f994e0bc640 1 -- 172.21.15.123:0/2859777985 >> 172.21.15.123:0/2859777985 conn(0x7f99480b1630 msgr2=0x7f99480b31e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:50:20.308 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:20.310+0000 7f994e0bc640 1 -- 172.21.15.123:0/2859777985 shutdown_connections 2024-09-16T15:50:20.308 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:20.310+0000 7f994e0bc640 1 -- 172.21.15.123:0/2859777985 wait complete. 2024-09-16T15:50:20.330 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-09-16T15:50:20.461 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:20.462+0000 7fb0a3143640 1 -- 172.21.15.123:0/2603365676 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb09c0b1660 msgr2=0x7fb09c0afbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:50:20.461 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:20.462+0000 7fb0a3143640 1 --2- 172.21.15.123:0/2603365676 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb09c0b1660 0x7fb09c0afbf0 secure :-1 s=READY pgs=1530 cs=0 l=1 rev1=1 crypto rx=0x7fb09c0d1350 tx=0x7fb09401dc70 comp rx=0 tx=0).stop 2024-09-16T15:50:20.462 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:20.462+0000 7fb0a3143640 1 -- 172.21.15.123:0/2603365676 shutdown_connections 2024-09-16T15:50:20.462 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:20.462+0000 7fb0a3143640 1 --2- 172.21.15.123:0/2603365676 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb09c0b1660 0x7fb09c0afbf0 unknown :-1 s=CLOSED pgs=1530 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:20.462 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:20.462+0000 7fb0a3143640 1 --2- 172.21.15.123:0/2603365676 >> v2:172.21.15.184:3300/0 conn(0x7fb09c15aed0 0x7fb09c15b2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:20.462 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:20.462+0000 7fb0a3143640 1 -- 172.21.15.123:0/2603365676 >> 172.21.15.123:0/2603365676 conn(0x7fb09c074150 msgr2=0x7fb09c074530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:50:20.462 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:20.462+0000 7fb0a3143640 1 -- 172.21.15.123:0/2603365676 shutdown_connections 2024-09-16T15:50:20.462 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:20.462+0000 7fb0a3143640 1 -- 172.21.15.123:0/2603365676 wait complete. 2024-09-16T15:50:20.462 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:20.462+0000 7fb0a3143640 1 Processor -- start 2024-09-16T15:50:20.463 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:20.462+0000 7fb0a3143640 1 -- start start 2024-09-16T15:50:20.464 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:20.462+0000 7fb0a3143640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb09c0b1660 0x7fb09c099980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:50:20.464 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:20.462+0000 7fb0a3143640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb09c0ce1d0 0x7fb09c0d0a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:50:20.464 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:20.462+0000 7fb0a3143640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fb09c15aed0 0x7fb09c1f92b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:50:20.464 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:20.462+0000 7fb0a3143640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fb09c0b0660 con 0x7fb09c0ce1d0 2024-09-16T15:50:20.464 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:20.462+0000 7fb0a3143640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fb09c0b04e0 con 0x7fb09c0b1660 2024-09-16T15:50:20.465 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:20.462+0000 7fb0a3143640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fb09c0b07e0 con 0x7fb09c15aed0 2024-09-16T15:50:20.465 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:20.462+0000 7fb09bfff640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb09c0ce1d0 0x7fb09c0d0a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:50:20.465 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:20.462+0000 7fb09bfff640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb09c0ce1d0 0x7fb09c0d0a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:37332/0 (socket says 172.21.15.123:37332) 2024-09-16T15:50:20.465 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:20.462+0000 7fb09bfff640 1 -- 172.21.15.123:0/153310260 learned_addr learned my addr 172.21.15.123:0/153310260 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:50:20.465 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:20.462+0000 7fb09bfff640 1 -- 172.21.15.123:0/153310260 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fb09c15aed0 msgr2=0x7fb09c1f92b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:50:20.465 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:20.462+0000 7fb09bfff640 1 --2- 172.21.15.123:0/153310260 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fb09c15aed0 0x7fb09c1f92b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:20.465 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:20.462+0000 7fb09bfff640 1 -- 172.21.15.123:0/153310260 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb09c0b1660 msgr2=0x7fb09c099980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:50:20.465 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:20.462+0000 7fb0a0edf640 1 --2- 172.21.15.123:0/153310260 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb09c0b1660 0x7fb09c099980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:50:20.465 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:20.462+0000 7fb09bfff640 1 --2- 172.21.15.123:0/153310260 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb09c0b1660 0x7fb09c099980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:20.465 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:20.462+0000 7fb09bfff640 1 -- 172.21.15.123:0/153310260 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb09c15a750 con 0x7fb09c0ce1d0 2024-09-16T15:50:20.465 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:20.462+0000 7fb09bfff640 1 --2- 172.21.15.123:0/153310260 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb09c0ce1d0 0x7fb09c0d0a20 secure :-1 s=READY pgs=1531 cs=0 l=1 rev1=1 crypto rx=0x7fb08800c430 tx=0x7fb08800c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:50:20.465 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:20.466+0000 7fb0997fa640 1 -- 172.21.15.123:0/153310260 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb088017070 con 0x7fb09c0ce1d0 2024-09-16T15:50:20.465 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:20.466+0000 7fb0997fa640 1 -- 172.21.15.123:0/153310260 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fb0880053c0 con 0x7fb09c0ce1d0 2024-09-16T15:50:20.466 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:20.466+0000 7fb0a3143640 1 -- 172.21.15.123:0/153310260 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb09c0c90d0 con 0x7fb09c0ce1d0 2024-09-16T15:50:20.466 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:20.466+0000 7fb0997fa640 1 -- 172.21.15.123:0/153310260 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb088003880 con 0x7fb09c0ce1d0 2024-09-16T15:50:20.466 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:20.466+0000 7fb0a3143640 1 -- 172.21.15.123:0/153310260 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb09c0c9340 con 0x7fb09c0ce1d0 2024-09-16T15:50:20.466 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:20.466+0000 7fb0a3143640 1 -- 172.21.15.123:0/153310260 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb09c0696c0 con 0x7fb09c0ce1d0 2024-09-16T15:50:20.467 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:20.466+0000 7fb0997fa640 1 -- 172.21.15.123:0/153310260 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 22) ==== 97692+0+0 (secure 0 0 0) 0x7fb08801a070 con 0x7fb09c0ce1d0 2024-09-16T15:50:20.467 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:20.466+0000 7fb0997fa640 1 --2- 172.21.15.123:0/153310260 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fb080073580 0x7fb080075a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:50:20.467 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:20.466+0000 7fb0997fa640 1 -- 172.21.15.123:0/153310260 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(137..137 src has 1..137) ==== 5385+0+0 (secure 0 0 0) 0x7fb088093e40 con 0x7fb09c0ce1d0 2024-09-16T15:50:20.467 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:20.466+0000 7fb0a0edf640 1 --2- 172.21.15.123:0/153310260 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fb080073580 0x7fb080075a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:50:20.467 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:20.466+0000 7fb0a0edf640 1 --2- 172.21.15.123:0/153310260 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fb080073580 0x7fb080075a40 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7fb0900058d0 tx=0x7fb090005860 comp rx=0 tx=0).ready entity=mgr.4150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:50:20.470 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:20.470+0000 7fb0997fa640 1 -- 172.21.15.123:0/153310260 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7fb088001130 con 0x7fb09c0ce1d0 2024-09-16T15:50:20.650 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:20.650+0000 7fb0a3143640 1 -- 172.21.15.123:0/153310260 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) -- 0x7fb09c0cbaf0 con 0x7fb09c0ce1d0 2024-09-16T15:50:21.302 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:21.302+0000 7fb0997fa640 1 -- 172.21.15.123:0/153310260 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v138) ==== 169+0+0 (secure 0 0 0) 0x7fb08805e6a0 con 0x7fb09c0ce1d0 2024-09-16T15:50:21.302 INFO:teuthology.orchestra.run.smithi123.stderr:pool 'cephfs_data' removed 2024-09-16T15:50:21.307 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:21.306+0000 7fb0a3143640 1 -- 172.21.15.123:0/153310260 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fb080073580 msgr2=0x7fb080075a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:50:21.307 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:21.306+0000 7fb0a3143640 1 --2- 172.21.15.123:0/153310260 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fb080073580 0x7fb080075a40 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7fb0900058d0 tx=0x7fb090005860 comp rx=0 tx=0).stop 2024-09-16T15:50:21.307 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:21.306+0000 7fb0a3143640 1 -- 172.21.15.123:0/153310260 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb09c0ce1d0 msgr2=0x7fb09c0d0a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:50:21.307 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:21.306+0000 7fb0a3143640 1 --2- 172.21.15.123:0/153310260 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb09c0ce1d0 0x7fb09c0d0a20 secure :-1 s=READY pgs=1531 cs=0 l=1 rev1=1 crypto rx=0x7fb08800c430 tx=0x7fb08800c8f0 comp rx=0 tx=0).stop 2024-09-16T15:50:21.307 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:21.306+0000 7fb0a3143640 1 -- 172.21.15.123:0/153310260 shutdown_connections 2024-09-16T15:50:21.307 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:21.306+0000 7fb0a3143640 1 --2- 172.21.15.123:0/153310260 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fb080073580 0x7fb080075a40 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:21.307 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:21.306+0000 7fb0a3143640 1 --2- 172.21.15.123:0/153310260 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fb09c15aed0 0x7fb09c1f92b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:21.308 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:21.306+0000 7fb0a3143640 1 --2- 172.21.15.123:0/153310260 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb09c0ce1d0 0x7fb09c0d0a20 unknown :-1 s=CLOSED pgs=1531 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:21.308 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:21.306+0000 7fb0a3143640 1 --2- 172.21.15.123:0/153310260 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb09c0b1660 0x7fb09c099980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:21.308 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:21.306+0000 7fb0a3143640 1 -- 172.21.15.123:0/153310260 >> 172.21.15.123:0/153310260 conn(0x7fb09c074150 msgr2=0x7fb09c15a040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:50:21.308 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:21.306+0000 7fb0a3143640 1 -- 172.21.15.123:0/153310260 shutdown_connections 2024-09-16T15:50:21.308 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:21.306+0000 7fb0a3143640 1 -- 172.21.15.123:0/153310260 wait complete. 2024-09-16T15:50:21.333 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config reset 27 2024-09-16T15:50:21.463 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:21.462+0000 7fb1d26df640 1 -- 172.21.15.123:0/1635550109 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb1cc0cf5a0 msgr2=0x7fb1cc06eee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:50:21.463 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:21.462+0000 7fb1d26df640 1 --2- 172.21.15.123:0/1635550109 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb1cc0cf5a0 0x7fb1cc06eee0 secure :-1 s=READY pgs=1532 cs=0 l=1 rev1=1 crypto rx=0x7fb1cc076280 tx=0x7fb1c001c7a0 comp rx=0 tx=0).stop 2024-09-16T15:50:21.463 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:21.462+0000 7fb1d26df640 1 -- 172.21.15.123:0/1635550109 shutdown_connections 2024-09-16T15:50:21.463 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:21.462+0000 7fb1d26df640 1 --2- 172.21.15.123:0/1635550109 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb1cc0cf5a0 0x7fb1cc06eee0 unknown :-1 s=CLOSED pgs=1532 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:21.463 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:21.462+0000 7fb1d26df640 1 --2- 172.21.15.123:0/1635550109 >> v2:172.21.15.184:3300/0 conn(0x7fb1cc158120 0x7fb1cc0ca320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:21.463 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:21.462+0000 7fb1d26df640 1 -- 172.21.15.123:0/1635550109 >> 172.21.15.123:0/1635550109 conn(0x7fb1cc0cfb10 msgr2=0x7fb1cc0cff20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:50:21.464 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:21.462+0000 7fb1d26df640 1 -- 172.21.15.123:0/1635550109 shutdown_connections 2024-09-16T15:50:21.464 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:21.462+0000 7fb1d26df640 1 -- 172.21.15.123:0/1635550109 wait complete. 2024-09-16T15:50:21.464 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:21.466+0000 7fb1d26df640 1 Processor -- start 2024-09-16T15:50:21.464 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:21.466+0000 7fb1d26df640 1 -- start start 2024-09-16T15:50:21.466 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:21.466+0000 7fb1d26df640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fb1cc0cf5a0 0x7fb1cc1f9a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:50:21.466 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:21.466+0000 7fb1d26df640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb1cc158120 0x7fb1cc150fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:50:21.466 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:21.466+0000 7fb1d26df640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb1cc15ec40 0x7fb1cc0b0660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:50:21.467 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:21.466+0000 7fb1d26df640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fb1cc15b6f0 con 0x7fb1cc158120 2024-09-16T15:50:21.467 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:21.466+0000 7fb1d26df640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fb1cc15b570 con 0x7fb1cc15ec40 2024-09-16T15:50:21.467 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:21.466+0000 7fb1d26df640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fb1cc15b870 con 0x7fb1cc0cf5a0 2024-09-16T15:50:21.467 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:21.466+0000 7fb1cb7fe640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb1cc158120 0x7fb1cc150fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:50:21.467 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:21.466+0000 7fb1cb7fe640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb1cc158120 0x7fb1cc150fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:37350/0 (socket says 172.21.15.123:37350) 2024-09-16T15:50:21.467 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:21.466+0000 7fb1cb7fe640 1 -- 172.21.15.123:0/3484937625 learned_addr learned my addr 172.21.15.123:0/3484937625 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:50:21.467 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:21.466+0000 7fb1d0c7c640 1 --2- 172.21.15.123:0/3484937625 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb1cc15ec40 0x7fb1cc0b0660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:50:21.467 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:21.466+0000 7fb1cbfff640 1 --2- 172.21.15.123:0/3484937625 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fb1cc0cf5a0 0x7fb1cc1f9a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:50:21.467 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:21.466+0000 7fb1cb7fe640 1 -- 172.21.15.123:0/3484937625 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fb1cc0cf5a0 msgr2=0x7fb1cc1f9a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:50:21.467 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:21.466+0000 7fb1cb7fe640 1 --2- 172.21.15.123:0/3484937625 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fb1cc0cf5a0 0x7fb1cc1f9a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:21.467 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:21.466+0000 7fb1cb7fe640 1 -- 172.21.15.123:0/3484937625 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb1cc15ec40 msgr2=0x7fb1cc0b0660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:50:21.467 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:21.466+0000 7fb1cb7fe640 1 --2- 172.21.15.123:0/3484937625 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb1cc15ec40 0x7fb1cc0b0660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:21.467 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:21.466+0000 7fb1cb7fe640 1 -- 172.21.15.123:0/3484937625 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb1cc1f9fa0 con 0x7fb1cc158120 2024-09-16T15:50:21.468 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:21.466+0000 7fb1cbfff640 1 --2- 172.21.15.123:0/3484937625 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fb1cc0cf5a0 0x7fb1cc1f9a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T15:50:21.468 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:21.466+0000 7fb1cb7fe640 1 --2- 172.21.15.123:0/3484937625 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb1cc158120 0x7fb1cc150fb0 secure :-1 s=READY pgs=1533 cs=0 l=1 rev1=1 crypto rx=0x7fb1c001ae50 tx=0x7fb1c00a84a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:50:21.468 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:21.466+0000 7fb1c8ff9640 1 -- 172.21.15.123:0/3484937625 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1c00be070 con 0x7fb1cc158120 2024-09-16T15:50:21.468 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:21.466+0000 7fb1c8ff9640 1 -- 172.21.15.123:0/3484937625 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fb1c0004040 con 0x7fb1cc158120 2024-09-16T15:50:21.468 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:21.466+0000 7fb1c8ff9640 1 -- 172.21.15.123:0/3484937625 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1c0004490 con 0x7fb1cc158120 2024-09-16T15:50:21.468 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:21.466+0000 7fb1d26df640 1 -- 172.21.15.123:0/3484937625 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb1cc151f60 con 0x7fb1cc158120 2024-09-16T15:50:21.468 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:21.466+0000 7fb1d26df640 1 -- 172.21.15.123:0/3484937625 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb1cc1524e0 con 0x7fb1cc158120 2024-09-16T15:50:21.469 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:21.466+0000 7fb1d26df640 1 -- 172.21.15.123:0/3484937625 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb1cc06eee0 con 0x7fb1cc158120 2024-09-16T15:50:21.469 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:21.470+0000 7fb1c8ff9640 1 -- 172.21.15.123:0/3484937625 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 22) ==== 97692+0+0 (secure 0 0 0) 0x7fb1c0005030 con 0x7fb1cc158120 2024-09-16T15:50:21.469 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:21.470+0000 7fb1c8ff9640 1 --2- 172.21.15.123:0/3484937625 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fb1b8073580 0x7fb1b8075a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:50:21.469 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:21.470+0000 7fb1cbfff640 1 --2- 172.21.15.123:0/3484937625 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fb1b8073580 0x7fb1b8075a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:50:21.469 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:21.470+0000 7fb1c8ff9640 1 -- 172.21.15.123:0/3484937625 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(138..138 src has 1..138) ==== 4971+0+0 (secure 0 0 0) 0x7fb1c012e9c0 con 0x7fb1cc158120 2024-09-16T15:50:21.469 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:21.470+0000 7fb1cbfff640 1 --2- 172.21.15.123:0/3484937625 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fb1b8073580 0x7fb1b8075a40 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7fb1bc005fd0 tx=0x7fb1bc005e60 comp rx=0 tx=0).ready entity=mgr.4150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:50:21.473 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:21.474+0000 7fb1c8ff9640 1 -- 172.21.15.123:0/3484937625 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7fb1cc1f9fa0 con 0x7fb1cc158120 2024-09-16T15:50:21.647 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:21.646+0000 7fb1d26df640 1 -- 172.21.15.123:0/3484937625 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "config reset", "num": 27} v 0) -- 0x7fb1cc0d06c0 con 0x7fb1cc158120 2024-09-16T15:50:21.653 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:21.654+0000 7fb1c8ff9640 1 -- 172.21.15.123:0/3484937625 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "config reset", "num": 27}]=0 v43) ==== 71+0+0 (secure 0 0 0) 0x7fb1c00ac030 con 0x7fb1cc158120 2024-09-16T15:50:21.656 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:21.654+0000 7fb1d26df640 1 -- 172.21.15.123:0/3484937625 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fb1b8073580 msgr2=0x7fb1b8075a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:50:21.656 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:21.654+0000 7fb1d26df640 1 --2- 172.21.15.123:0/3484937625 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fb1b8073580 0x7fb1b8075a40 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7fb1bc005fd0 tx=0x7fb1bc005e60 comp rx=0 tx=0).stop 2024-09-16T15:50:21.656 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:21.658+0000 7fb1d26df640 1 -- 172.21.15.123:0/3484937625 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb1cc158120 msgr2=0x7fb1cc150fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:50:21.656 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:21.658+0000 7fb1d26df640 1 --2- 172.21.15.123:0/3484937625 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb1cc158120 0x7fb1cc150fb0 secure :-1 s=READY pgs=1533 cs=0 l=1 rev1=1 crypto rx=0x7fb1c001ae50 tx=0x7fb1c00a84a0 comp rx=0 tx=0).stop 2024-09-16T15:50:21.657 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:21.658+0000 7fb1d26df640 1 -- 172.21.15.123:0/3484937625 shutdown_connections 2024-09-16T15:50:21.657 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:21.658+0000 7fb1d26df640 1 --2- 172.21.15.123:0/3484937625 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fb1b8073580 0x7fb1b8075a40 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:21.657 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:21.658+0000 7fb1d26df640 1 --2- 172.21.15.123:0/3484937625 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb1cc15ec40 0x7fb1cc0b0660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:21.657 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:21.658+0000 7fb1d26df640 1 --2- 172.21.15.123:0/3484937625 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb1cc158120 0x7fb1cc150fb0 unknown :-1 s=CLOSED pgs=1533 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:21.657 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:21.658+0000 7fb1d26df640 1 --2- 172.21.15.123:0/3484937625 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fb1cc0cf5a0 0x7fb1cc1f9a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:21.657 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:21.658+0000 7fb1d26df640 1 -- 172.21.15.123:0/3484937625 >> 172.21.15.123:0/3484937625 conn(0x7fb1cc0cfb10 msgr2=0x7fb1cc15cda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:50:21.657 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:21.658+0000 7fb1d26df640 1 -- 172.21.15.123:0/3484937625 shutdown_connections 2024-09-16T15:50:21.657 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:21.658+0000 7fb1d26df640 1 -- 172.21.15.123:0/3484937625 wait complete. 2024-09-16T15:50:21.674 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_volumes.TestMisc.test_subvolume_ops_on_nonexistent_vol' 2024-09-16T15:50:21.801 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:21.802+0000 7f59e2596640 1 -- 172.21.15.123:0/3722212331 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f59dc074470 msgr2=0x7f59dc0cd6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:50:21.801 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:21.802+0000 7f59e2596640 1 --2- 172.21.15.123:0/3722212331 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f59dc074470 0x7f59dc0cd6e0 secure :-1 s=READY pgs=1534 cs=0 l=1 rev1=1 crypto rx=0x7f59c4009c80 tx=0x7f59c401cd10 comp rx=0 tx=0).stop 2024-09-16T15:50:21.801 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:21.802+0000 7f59e2596640 1 -- 172.21.15.123:0/3722212331 shutdown_connections 2024-09-16T15:50:21.801 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:21.802+0000 7f59e2596640 1 --2- 172.21.15.123:0/3722212331 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f59dc074470 0x7f59dc0cd6e0 unknown :-1 s=CLOSED pgs=1534 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:21.801 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:21.802+0000 7f59e2596640 1 --2- 172.21.15.123:0/3722212331 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f59dc15e160 0x7f59dc073f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:21.801 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:21.802+0000 7f59e2596640 1 -- 172.21.15.123:0/3722212331 >> 172.21.15.123:0/3722212331 conn(0x7f59dc0cfb10 msgr2=0x7f59dc0cff20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:50:21.801 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:21.802+0000 7f59e2596640 1 -- 172.21.15.123:0/3722212331 shutdown_connections 2024-09-16T15:50:21.802 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:21.802+0000 7f59e2596640 1 -- 172.21.15.123:0/3722212331 wait complete. 2024-09-16T15:50:21.802 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:21.802+0000 7f59e2596640 1 Processor -- start 2024-09-16T15:50:21.802 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:21.802+0000 7f59e2596640 1 -- start start 2024-09-16T15:50:21.805 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:21.802+0000 7f59e2596640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f59dc074470 0x7f59dc14ebd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:50:21.805 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:21.802+0000 7f59e2596640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f59dc0cdc20 0x7f59dc1f6260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:50:21.805 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:21.802+0000 7f59e2596640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f59dc15e160 0x7f59dc0aff20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:50:21.805 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:21.802+0000 7f59e2596640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f59dc167f50 con 0x7f59dc074470 2024-09-16T15:50:21.805 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:21.802+0000 7f59e2596640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f59dc167dd0 con 0x7f59dc0cdc20 2024-09-16T15:50:21.805 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:21.802+0000 7f59e2596640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f59dc1680d0 con 0x7f59dc15e160 2024-09-16T15:50:21.805 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:21.802+0000 7f59dbfff640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f59dc074470 0x7f59dc14ebd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:50:21.805 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:21.806+0000 7f59dbfff640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f59dc074470 0x7f59dc14ebd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:37358/0 (socket says 172.21.15.123:37358) 2024-09-16T15:50:21.805 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:21.806+0000 7f59dbfff640 1 -- 172.21.15.123:0/3385114559 learned_addr learned my addr 172.21.15.123:0/3385114559 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:50:21.805 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:21.806+0000 7f59e0b33640 1 --2- 172.21.15.123:0/3385114559 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f59dc15e160 0x7f59dc0aff20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:50:21.806 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:21.806+0000 7f59db7fe640 1 --2- 172.21.15.123:0/3385114559 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f59dc0cdc20 0x7f59dc1f6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:50:21.806 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:21.806+0000 7f59dbfff640 1 -- 172.21.15.123:0/3385114559 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f59dc15e160 msgr2=0x7f59dc0aff20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:50:21.806 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:21.806+0000 7f59dbfff640 1 --2- 172.21.15.123:0/3385114559 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f59dc15e160 0x7f59dc0aff20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:21.806 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:21.806+0000 7f59dbfff640 1 -- 172.21.15.123:0/3385114559 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f59dc0cdc20 msgr2=0x7f59dc1f6260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:50:21.806 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:21.806+0000 7f59dbfff640 1 --2- 172.21.15.123:0/3385114559 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f59dc0cdc20 0x7f59dc1f6260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:21.806 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:21.806+0000 7f59dbfff640 1 -- 172.21.15.123:0/3385114559 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f59dc1f9000 con 0x7f59dc074470 2024-09-16T15:50:21.806 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:21.806+0000 7f59e0b33640 1 --2- 172.21.15.123:0/3385114559 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f59dc15e160 0x7f59dc0aff20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T15:50:21.806 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:21.806+0000 7f59db7fe640 1 --2- 172.21.15.123:0/3385114559 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f59dc0cdc20 0x7f59dc1f6260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T15:50:21.806 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:21.806+0000 7f59dbfff640 1 --2- 172.21.15.123:0/3385114559 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f59dc074470 0x7f59dc14ebd0 secure :-1 s=READY pgs=1535 cs=0 l=1 rev1=1 crypto rx=0x7f59cc00ca20 tx=0x7f59cc00cee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:50:21.806 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:21.806+0000 7f59d8ff9640 1 -- 172.21.15.123:0/3385114559 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59cc007c70 con 0x7f59dc074470 2024-09-16T15:50:21.806 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:21.806+0000 7f59d8ff9640 1 -- 172.21.15.123:0/3385114559 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f59cc00f450 con 0x7f59dc074470 2024-09-16T15:50:21.806 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:21.806+0000 7f59d8ff9640 1 -- 172.21.15.123:0/3385114559 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59cc00f8a0 con 0x7f59dc074470 2024-09-16T15:50:21.806 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:21.806+0000 7f59e2596640 1 -- 172.21.15.123:0/3385114559 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f59dc1f92e0 con 0x7f59dc074470 2024-09-16T15:50:21.807 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:21.806+0000 7f59e2596640 1 -- 172.21.15.123:0/3385114559 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f59dc1f98a0 con 0x7f59dc074470 2024-09-16T15:50:21.807 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:21.806+0000 7f59e2596640 1 -- 172.21.15.123:0/3385114559 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f59dc0cd6e0 con 0x7f59dc074470 2024-09-16T15:50:21.807 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:21.806+0000 7f59d8ff9640 1 -- 172.21.15.123:0/3385114559 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 22) ==== 97692+0+0 (secure 0 0 0) 0x7f59cc00fa40 con 0x7f59dc074470 2024-09-16T15:50:21.807 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:21.806+0000 7f59d8ff9640 1 --2- 172.21.15.123:0/3385114559 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f59bc073580 0x7f59bc075a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:50:21.808 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:21.806+0000 7f59db7fe640 1 --2- 172.21.15.123:0/3385114559 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f59bc073580 0x7f59bc075a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:50:21.808 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:21.806+0000 7f59d8ff9640 1 -- 172.21.15.123:0/3385114559 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(138..138 src has 1..138) ==== 4971+0+0 (secure 0 0 0) 0x7f59cc095040 con 0x7f59dc074470 2024-09-16T15:50:21.808 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:21.806+0000 7f59db7fe640 1 --2- 172.21.15.123:0/3385114559 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f59bc073580 0x7f59bc075a40 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f59c40af000 tx=0x7f59c401b550 comp rx=0 tx=0).ready entity=mgr.4150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:50:21.811 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:21.810+0000 7f59d8ff9640 1 -- 172.21.15.123:0/3385114559 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f59cc014030 con 0x7f59dc074470 2024-09-16T15:50:22.045 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:22.046+0000 7f59e2596640 1 -- 172.21.15.123:0/3385114559 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_volumes.TestMisc.test_subvolume_ops_on_nonexistent_vol"]} v 0) -- 0x7f59dc076060 con 0x7f59dc074470 2024-09-16T15:50:22.320 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:22.322+0000 7f59d8ff9640 1 -- 172.21.15.123:0/3385114559 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_volumes.TestMisc.test_subvolume_ops_on_nonexistent_vol"]}]=0 v394) ==== 151+0+0 (secure 0 0 0) 0x7f59cc05fa40 con 0x7f59dc074470 2024-09-16T15:50:22.323 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:22.322+0000 7f59e2596640 1 -- 172.21.15.123:0/3385114559 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f59bc073580 msgr2=0x7f59bc075a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:50:22.324 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:22.322+0000 7f59e2596640 1 --2- 172.21.15.123:0/3385114559 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f59bc073580 0x7f59bc075a40 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f59c40af000 tx=0x7f59c401b550 comp rx=0 tx=0).stop 2024-09-16T15:50:22.324 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:22.322+0000 7f59e2596640 1 -- 172.21.15.123:0/3385114559 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f59dc074470 msgr2=0x7f59dc14ebd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:50:22.324 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:22.322+0000 7f59e2596640 1 --2- 172.21.15.123:0/3385114559 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f59dc074470 0x7f59dc14ebd0 secure :-1 s=READY pgs=1535 cs=0 l=1 rev1=1 crypto rx=0x7f59cc00ca20 tx=0x7f59cc00cee0 comp rx=0 tx=0).stop 2024-09-16T15:50:22.324 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:22.322+0000 7f59e2596640 1 -- 172.21.15.123:0/3385114559 shutdown_connections 2024-09-16T15:50:22.324 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:22.322+0000 7f59e2596640 1 --2- 172.21.15.123:0/3385114559 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f59bc073580 0x7f59bc075a40 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:22.324 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:22.322+0000 7f59e2596640 1 --2- 172.21.15.123:0/3385114559 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f59dc15e160 0x7f59dc0aff20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:22.324 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:22.322+0000 7f59e2596640 1 --2- 172.21.15.123:0/3385114559 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f59dc0cdc20 0x7f59dc1f6260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:22.324 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:22.322+0000 7f59e2596640 1 --2- 172.21.15.123:0/3385114559 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f59dc074470 0x7f59dc14ebd0 unknown :-1 s=CLOSED pgs=1535 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:22.324 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:22.322+0000 7f59e2596640 1 -- 172.21.15.123:0/3385114559 >> 172.21.15.123:0/3385114559 conn(0x7f59dc0cfb10 msgr2=0x7f59dc0b0dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:50:22.325 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:22.322+0000 7f59e2596640 1 -- 172.21.15.123:0/3385114559 shutdown_connections 2024-09-16T15:50:22.325 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:22.326+0000 7f59e2596640 1 -- 172.21.15.123:0/3385114559 wait complete. 2024-09-16T15:50:22.343 INFO:tasks.cephfs_test_runner:test_subvolume_ops_on_nonexistent_vol (tasks.cephfs.test_volumes.TestMisc) ... ok 2024-09-16T15:50:22.344 INFO:tasks.cephfs_test_runner:Starting test: test_subvolume_upgrade_legacy_to_v1 (tasks.cephfs.test_volumes.TestMisc) poor man's upgrade test -- rather than going through a full upgrade cycle, 2024-09-16T15:50:22.344 INFO:tasks.cephfs_test_runner:test_subvolume_upgrade_legacy_to_v1 (tasks.cephfs.test_volumes.TestMisc) 2024-09-16T15:50:22.344 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-16T15:50:22.473 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:22.474+0000 7f9c3744c640 1 -- 172.21.15.123:0/782250104 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9c3015e160 msgr2=0x7f9c30073f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:50:22.473 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:22.474+0000 7f9c3744c640 1 --2- 172.21.15.123:0/782250104 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9c3015e160 0x7f9c30073f30 secure :-1 s=READY pgs=1537 cs=0 l=1 rev1=1 crypto rx=0x7f9c2c009840 tx=0x7f9c2c01c790 comp rx=0 tx=0).stop 2024-09-16T15:50:22.473 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:22.474+0000 7f9c3744c640 1 -- 172.21.15.123:0/782250104 shutdown_connections 2024-09-16T15:50:22.473 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:22.474+0000 7f9c3744c640 1 --2- 172.21.15.123:0/782250104 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f9c30074470 0x7f9c300cd6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:22.474 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:22.474+0000 7f9c3744c640 1 --2- 172.21.15.123:0/782250104 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9c3015e160 0x7f9c30073f30 unknown :-1 s=CLOSED pgs=1537 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:22.474 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:22.474+0000 7f9c3744c640 1 -- 172.21.15.123:0/782250104 >> 172.21.15.123:0/782250104 conn(0x7f9c300cfb10 msgr2=0x7f9c300cff20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:50:22.474 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:22.474+0000 7f9c3744c640 1 -- 172.21.15.123:0/782250104 shutdown_connections 2024-09-16T15:50:22.474 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:22.474+0000 7f9c3744c640 1 -- 172.21.15.123:0/782250104 wait complete. 2024-09-16T15:50:22.474 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:22.474+0000 7f9c3744c640 1 Processor -- start 2024-09-16T15:50:22.474 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:22.474+0000 7f9c3744c640 1 -- start start 2024-09-16T15:50:22.476 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:22.474+0000 7f9c3744c640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9c30074470 0x7f9c3014ab00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:50:22.476 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:22.474+0000 7f9c3744c640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f9c300cdc20 0x7f9c3015a630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:50:22.476 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:22.474+0000 7f9c3744c640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f9c3015e160 0x7f9c300c91f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:50:22.476 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:22.474+0000 7f9c3744c640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f9c30167f50 con 0x7f9c30074470 2024-09-16T15:50:22.477 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:22.474+0000 7f9c3744c640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f9c30167dd0 con 0x7f9c3015e160 2024-09-16T15:50:22.477 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:22.474+0000 7f9c3744c640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f9c301680d0 con 0x7f9c300cdc20 2024-09-16T15:50:22.477 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:22.474+0000 7f9c351e8640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9c30074470 0x7f9c3014ab00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:50:22.477 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:22.474+0000 7f9c351e8640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9c30074470 0x7f9c3014ab00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:55810/0 (socket says 172.21.15.123:55810) 2024-09-16T15:50:22.477 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:22.474+0000 7f9c351e8640 1 -- 172.21.15.123:0/2773569064 learned_addr learned my addr 172.21.15.123:0/2773569064 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:50:22.477 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:22.474+0000 7f9c351e8640 1 -- 172.21.15.123:0/2773569064 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f9c300cdc20 msgr2=0x7f9c3015a630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:50:22.477 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:22.474+0000 7f9c349e7640 1 --2- 172.21.15.123:0/2773569064 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f9c300cdc20 0x7f9c3015a630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:50:22.477 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:22.474+0000 7f9c351e8640 1 --2- 172.21.15.123:0/2773569064 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f9c300cdc20 0x7f9c3015a630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:22.477 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:22.474+0000 7f9c351e8640 1 -- 172.21.15.123:0/2773569064 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f9c3015e160 msgr2=0x7f9c300c91f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:50:22.477 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:22.474+0000 7f9c351e8640 1 --2- 172.21.15.123:0/2773569064 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f9c3015e160 0x7f9c300c91f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:22.477 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:22.474+0000 7f9c351e8640 1 -- 172.21.15.123:0/2773569064 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9c3015ab70 con 0x7f9c30074470 2024-09-16T15:50:22.477 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:22.474+0000 7f9c351e8640 1 --2- 172.21.15.123:0/2773569064 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9c30074470 0x7f9c3014ab00 secure :-1 s=READY pgs=1538 cs=0 l=1 rev1=1 crypto rx=0x7f9c2c009790 tx=0x7f9c2c0a56f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:50:22.477 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:22.474+0000 7f9c25ffb640 1 -- 172.21.15.123:0/2773569064 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c2c0bb070 con 0x7f9c30074470 2024-09-16T15:50:22.478 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:22.474+0000 7f9c25ffb640 1 -- 172.21.15.123:0/2773569064 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f9c2c0044b0 con 0x7f9c30074470 2024-09-16T15:50:22.478 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:22.474+0000 7f9c3744c640 1 -- 172.21.15.123:0/2773569064 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9c3014b3b0 con 0x7f9c30074470 2024-09-16T15:50:22.478 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:22.474+0000 7f9c25ffb640 1 -- 172.21.15.123:0/2773569064 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c2c0b2040 con 0x7f9c30074470 2024-09-16T15:50:22.478 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:22.474+0000 7f9c3744c640 1 -- 172.21.15.123:0/2773569064 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9c300b3b70 con 0x7f9c30074470 2024-09-16T15:50:22.478 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:22.478+0000 7f9c3744c640 1 -- 172.21.15.123:0/2773569064 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9c300cd6e0 con 0x7f9c30074470 2024-09-16T15:50:22.479 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:22.478+0000 7f9c25ffb640 1 -- 172.21.15.123:0/2773569064 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 22) ==== 97692+0+0 (secure 0 0 0) 0x7f9c2c0be070 con 0x7f9c30074470 2024-09-16T15:50:22.479 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:22.478+0000 7f9c25ffb640 1 --2- 172.21.15.123:0/2773569064 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f9c14073580 0x7f9c14075a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:50:22.479 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:22.478+0000 7f9c25ffb640 1 -- 172.21.15.123:0/2773569064 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(138..138 src has 1..138) ==== 4971+0+0 (secure 0 0 0) 0x7f9c2c12dfd0 con 0x7f9c30074470 2024-09-16T15:50:22.479 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:22.478+0000 7f9c349e7640 1 --2- 172.21.15.123:0/2773569064 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f9c14073580 0x7f9c14075a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:50:22.479 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:22.478+0000 7f9c349e7640 1 --2- 172.21.15.123:0/2773569064 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f9c14073580 0x7f9c14075a40 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f9c3014b610 tx=0x7f9c20005c70 comp rx=0 tx=0).ready entity=mgr.4150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:50:22.482 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:22.482+0000 7f9c25ffb640 1 -- 172.21.15.123:0/2773569064 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f9c2c0f30b0 con 0x7f9c30074470 2024-09-16T15:50:22.655 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:22.654+0000 7f9c3744c640 1 -- 172.21.15.123:0/2773569064 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f9c300d06c0 con 0x7f9c30074470 2024-09-16T15:50:22.658 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:22.658+0000 7f9c25ffb640 1 -- 172.21.15.123:0/2773569064 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v138) ==== 74+0+13146 (secure 0 0 0) 0x7f9c2c016410 con 0x7f9c30074470 2024-09-16T15:50:22.659 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-09-16T15:50:22.659 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":138,"fsid":"56077f89-18c8-486f-ae7e-eecc0bb7822c","created":"2024-09-16T15:42:26.364892+0000","modified":"2024-09-16T15:50:21.303144+0000","last_up_change":"2024-09-16T15:42:38.707234+0000","last_in_change":"2024-09-16T15:42:29.423662+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":21,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-16T15:42:39.287764+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9e7c1955-a607-4671-9be7-f2ef760ceeaa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":133,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6800","nonce":3882044969},{"type":"v1","addr":"172.21.15.123:6801","nonce":3882044969}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6802","nonce":3882044969},{"type":"v1","addr":"172.21.15.123:6803","nonce":3882044969}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6806","nonce":3882044969},{"type":"v1","addr":"172.21.15.123:6807","nonce":3882044969}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6804","nonce":3882044969},{"type":"v1","addr":"172.21.15.123:6805","nonce":3882044969}]},"public_addr":"172.21.15.123:6801/3882044969","cluster_addr":"172.21.15.123:6803/3882044969","heartbeat_back_addr":"172.21.15.123:6807/3882044969","heartbeat_front_addr":"172.21.15.123:6805/3882044969","state":["exists","up"]},{"osd":1,"uuid":"c540cfde-0977-4742-9b17-fd634668e010","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":133,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6816","nonce":4037001850},{"type":"v1","addr":"172.21.15.123:6817","nonce":4037001850}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6818","nonce":4037001850},{"type":"v1","addr":"172.21.15.123:6819","nonce":4037001850}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6822","nonce":4037001850},{"type":"v1","addr":"172.21.15.123:6823","nonce":4037001850}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6820","nonce":4037001850},{"type":"v1","addr":"172.21.15.123:6821","nonce":4037001850}]},"public_addr":"172.21.15.123:6817/4037001850","cluster_addr":"172.21.15.123:6819/4037001850","heartbeat_back_addr":"172.21.15.123:6823/4037001850","heartbeat_front_addr":"172.21.15.123:6821/4037001850","state":["exists","up"]},{"osd":2,"uuid":"9864a496-f169-486b-95c4-1d59162793a7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":133,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6824","nonce":1332143088},{"type":"v1","addr":"172.21.15.123:6825","nonce":1332143088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6826","nonce":1332143088},{"type":"v1","addr":"172.21.15.123:6827","nonce":1332143088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6830","nonce":1332143088},{"type":"v1","addr":"172.21.15.123:6831","nonce":1332143088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6828","nonce":1332143088},{"type":"v1","addr":"172.21.15.123:6829","nonce":1332143088}]},"public_addr":"172.21.15.123:6825/1332143088","cluster_addr":"172.21.15.123:6827/1332143088","heartbeat_back_addr":"172.21.15.123:6831/1332143088","heartbeat_front_addr":"172.21.15.123:6829/1332143088","state":["exists","up"]},{"osd":3,"uuid":"c6d1f817-f074-4325-95e1-578eba13edda","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":133,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6808","nonce":988586998},{"type":"v1","addr":"172.21.15.123:6809","nonce":988586998}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6810","nonce":988586998},{"type":"v1","addr":"172.21.15.123:6811","nonce":988586998}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6814","nonce":988586998},{"type":"v1","addr":"172.21.15.123:6815","nonce":988586998}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6812","nonce":988586998},{"type":"v1","addr":"172.21.15.123:6813","nonce":988586998}]},"public_addr":"172.21.15.123:6809/988586998","cluster_addr":"172.21.15.123:6811/988586998","heartbeat_back_addr":"172.21.15.123:6815/988586998","heartbeat_front_addr":"172.21.15.123:6813/988586998","state":["exists","up"]},{"osd":4,"uuid":"200a8df4-50ef-40a5-b48a-58981269c086","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":133,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6816","nonce":3387813890},{"type":"v1","addr":"172.21.15.184:6817","nonce":3387813890}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6818","nonce":3387813890},{"type":"v1","addr":"172.21.15.184:6819","nonce":3387813890}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6822","nonce":3387813890},{"type":"v1","addr":"172.21.15.184:6823","nonce":3387813890}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6820","nonce":3387813890},{"type":"v1","addr":"172.21.15.184:6821","nonce":3387813890}]},"public_addr":"172.21.15.184:6817/3387813890","cluster_addr":"172.21.15.184:6819/3387813890","heartbeat_back_addr":"172.21.15.184:6823/3387813890","heartbeat_front_addr":"172.21.15.184:6821/3387813890","state":["exists","up"]},{"osd":5,"uuid":"32d5a900-da3d-412c-bc19-13861a2368d8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":133,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6808","nonce":2680916017},{"type":"v1","addr":"172.21.15.184:6809","nonce":2680916017}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6810","nonce":2680916017},{"type":"v1","addr":"172.21.15.184:6811","nonce":2680916017}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6814","nonce":2680916017},{"type":"v1","addr":"172.21.15.184:6815","nonce":2680916017}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6812","nonce":2680916017},{"type":"v1","addr":"172.21.15.184:6813","nonce":2680916017}]},"public_addr":"172.21.15.184:6809/2680916017","cluster_addr":"172.21.15.184:6811/2680916017","heartbeat_back_addr":"172.21.15.184:6815/2680916017","heartbeat_front_addr":"172.21.15.184:6813/2680916017","state":["exists","up"]},{"osd":6,"uuid":"cde6f39e-aa1b-4260-ac19-774ee6aaca72","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":133,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6824","nonce":2183120457},{"type":"v1","addr":"172.21.15.184:6825","nonce":2183120457}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6826","nonce":2183120457},{"type":"v1","addr":"172.21.15.184:6827","nonce":2183120457}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6830","nonce":2183120457},{"type":"v1","addr":"172.21.15.184:6831","nonce":2183120457}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6828","nonce":2183120457},{"type":"v1","addr":"172.21.15.184:6829","nonce":2183120457}]},"public_addr":"172.21.15.184:6825/2183120457","cluster_addr":"172.21.15.184:6827/2183120457","heartbeat_back_addr":"172.21.15.184:6831/2183120457","heartbeat_front_addr":"172.21.15.184:6829/2183120457","state":["exists","up"]},{"osd":7,"uuid":"4f381778-eee1-47c6-804f-df0e6b33e313","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":133,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6800","nonce":4102957285},{"type":"v1","addr":"172.21.15.184:6801","nonce":4102957285}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6802","nonce":4102957285},{"type":"v1","addr":"172.21.15.184:6803","nonce":4102957285}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6806","nonce":4102957285},{"type":"v1","addr":"172.21.15.184:6807","nonce":4102957285}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6804","nonce":4102957285},{"type":"v1","addr":"172.21.15.184:6805","nonce":4102957285}]},"public_addr":"172.21.15.184:6801/4102957285","cluster_addr":"172.21.15.184:6803/4102957285","heartbeat_back_addr":"172.21.15.184:6807/4102957285","heartbeat_front_addr":"172.21.15.184:6805/4102957285","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:35.972210+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.222007+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.198617+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.385663+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.358935+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.156066+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.752973+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:35.682846+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.123:6835/333722639":"2024-09-17T15:50:18.290323+0000","172.21.15.123:6834/333722639":"2024-09-17T15:50:18.290323+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":17,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T15:50:22.662 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:22.662+0000 7f9c3744c640 1 -- 172.21.15.123:0/2773569064 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f9c14073580 msgr2=0x7f9c14075a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:50:22.662 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:22.662+0000 7f9c3744c640 1 --2- 172.21.15.123:0/2773569064 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f9c14073580 0x7f9c14075a40 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f9c3014b610 tx=0x7f9c20005c70 comp rx=0 tx=0).stop 2024-09-16T15:50:22.662 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:22.662+0000 7f9c3744c640 1 -- 172.21.15.123:0/2773569064 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9c30074470 msgr2=0x7f9c3014ab00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:50:22.662 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:22.662+0000 7f9c3744c640 1 --2- 172.21.15.123:0/2773569064 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9c30074470 0x7f9c3014ab00 secure :-1 s=READY pgs=1538 cs=0 l=1 rev1=1 crypto rx=0x7f9c2c009790 tx=0x7f9c2c0a56f0 comp rx=0 tx=0).stop 2024-09-16T15:50:22.662 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:22.662+0000 7f9c3744c640 1 -- 172.21.15.123:0/2773569064 shutdown_connections 2024-09-16T15:50:22.662 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:22.662+0000 7f9c3744c640 1 --2- 172.21.15.123:0/2773569064 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f9c14073580 0x7f9c14075a40 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:22.662 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:22.662+0000 7f9c3744c640 1 --2- 172.21.15.123:0/2773569064 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f9c3015e160 0x7f9c300c91f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:22.662 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:22.662+0000 7f9c3744c640 1 --2- 172.21.15.123:0/2773569064 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f9c300cdc20 0x7f9c3015a630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:22.662 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:22.662+0000 7f9c3744c640 1 --2- 172.21.15.123:0/2773569064 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9c30074470 0x7f9c3014ab00 unknown :-1 s=CLOSED pgs=1538 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:22.662 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:22.662+0000 7f9c3744c640 1 -- 172.21.15.123:0/2773569064 >> 172.21.15.123:0/2773569064 conn(0x7f9c300cfb10 msgr2=0x7f9c30075a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:50:22.662 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:22.662+0000 7f9c3744c640 1 -- 172.21.15.123:0/2773569064 shutdown_connections 2024-09-16T15:50:22.662 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:22.662+0000 7f9c3744c640 1 -- 172.21.15.123:0/2773569064 wait complete. 2024-09-16T15:50:22.679 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-16T15:42:39.287764+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-09-16T15:50:22.679 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-09-16T15:50:22.810 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:22.810+0000 7f46f6845640 1 -- 172.21.15.123:0/1447031587 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f46d8003fe0 msgr2=0x7f46d8004810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:50:22.810 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:22.810+0000 7f46f6845640 1 --2- 172.21.15.123:0/1447031587 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f46d8003fe0 0x7f46d8004810 secure :-1 s=READY pgs=1540 cs=0 l=1 rev1=1 crypto rx=0x7f46e4009b40 tx=0x7f46e401d220 comp rx=0 tx=0).stop 2024-09-16T15:50:22.810 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:22.810+0000 7f46f4de2640 1 -- 172.21.15.123:0/1447031587 reap_dead start 2024-09-16T15:50:22.811 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:22.810+0000 7f46f6845640 1 -- 172.21.15.123:0/1447031587 shutdown_connections 2024-09-16T15:50:22.811 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:22.810+0000 7f46f6845640 1 -- 172.21.15.123:0/1447031587 >> 172.21.15.123:0/1447031587 conn(0x7f46f00b1710 msgr2=0x7f46f00b1b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:50:22.811 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:22.810+0000 7f46f6845640 1 -- 172.21.15.123:0/1447031587 shutdown_connections 2024-09-16T15:50:22.811 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:22.810+0000 7f46f6845640 1 -- 172.21.15.123:0/1447031587 wait complete. 2024-09-16T15:50:22.811 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:22.810+0000 7f46f6845640 1 Processor -- start 2024-09-16T15:50:22.811 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:22.810+0000 7f46f6845640 1 -- start start 2024-09-16T15:50:22.813 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:22.814+0000 7f46f6845640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f46d8003fe0 0x7f46f00c7bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:50:22.813 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:22.814+0000 7f46f6845640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f46f00b42b0 0x7f46f01fb540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:50:22.814 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:22.814+0000 7f46f6845640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f46f00b4730 0x7f46f01fba80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:50:22.814 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:22.814+0000 7f46f6845640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f46f006e9e0 con 0x7f46d8003fe0 2024-09-16T15:50:22.814 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:22.814+0000 7f46f6845640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f46f006e860 con 0x7f46f00b42b0 2024-09-16T15:50:22.814 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:22.814+0000 7f46f6845640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f46f006eb60 con 0x7f46f00b4730 2024-09-16T15:50:22.814 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:22.814+0000 7f46effff640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f46d8003fe0 0x7f46f00c7bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:50:22.814 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:22.814+0000 7f46effff640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f46d8003fe0 0x7f46f00c7bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:55816/0 (socket says 172.21.15.123:55816) 2024-09-16T15:50:22.814 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:22.814+0000 7f46effff640 1 -- 172.21.15.123:0/546984040 learned_addr learned my addr 172.21.15.123:0/546984040 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:50:22.814 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:22.814+0000 7f46ef7fe640 1 --2- 172.21.15.123:0/546984040 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f46f00b42b0 0x7f46f01fb540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:50:22.814 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:22.814+0000 7f46f4de2640 1 --2- 172.21.15.123:0/546984040 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f46f00b4730 0x7f46f01fba80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:50:22.814 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:22.814+0000 7f46effff640 1 -- 172.21.15.123:0/546984040 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f46f00b4730 msgr2=0x7f46f01fba80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:50:22.814 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:22.814+0000 7f46effff640 1 --2- 172.21.15.123:0/546984040 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f46f00b4730 0x7f46f01fba80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:22.814 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:22.814+0000 7f46effff640 1 -- 172.21.15.123:0/546984040 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f46f00b42b0 msgr2=0x7f46f01fb540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:50:22.815 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:22.814+0000 7f46effff640 1 --2- 172.21.15.123:0/546984040 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f46f00b42b0 0x7f46f01fb540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:22.815 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:22.814+0000 7f46effff640 1 -- 172.21.15.123:0/546984040 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f46f00b2a10 con 0x7f46d8003fe0 2024-09-16T15:50:22.815 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:22.814+0000 7f46effff640 1 --2- 172.21.15.123:0/546984040 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f46d8003fe0 0x7f46f00c7bf0 secure :-1 s=READY pgs=1541 cs=0 l=1 rev1=1 crypto rx=0x7f46e000e620 tx=0x7f46e000eae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:50:22.815 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:22.814+0000 7f46ecff9640 1 -- 172.21.15.123:0/546984040 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46e0015070 con 0x7f46d8003fe0 2024-09-16T15:50:22.815 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:22.814+0000 7f46ecff9640 1 -- 172.21.15.123:0/546984040 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f46e0004310 con 0x7f46d8003fe0 2024-09-16T15:50:22.815 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:22.814+0000 7f46f6845640 1 -- 172.21.15.123:0/546984040 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f46f01574e0 con 0x7f46d8003fe0 2024-09-16T15:50:22.815 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:22.814+0000 7f46ecff9640 1 -- 172.21.15.123:0/546984040 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46e000ca00 con 0x7f46d8003fe0 2024-09-16T15:50:22.815 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:22.814+0000 7f46f6845640 1 -- 172.21.15.123:0/546984040 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f46f0157780 con 0x7f46d8003fe0 2024-09-16T15:50:22.816 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:22.814+0000 7f46f6845640 1 -- 172.21.15.123:0/546984040 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f46f015ddc0 con 0x7f46d8003fe0 2024-09-16T15:50:22.816 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:22.814+0000 7f46ecff9640 1 -- 172.21.15.123:0/546984040 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 22) ==== 97692+0+0 (secure 0 0 0) 0x7f46e000cba0 con 0x7f46d8003fe0 2024-09-16T15:50:22.816 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:22.814+0000 7f46ecff9640 1 --2- 172.21.15.123:0/546984040 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f46dc073580 0x7f46dc075a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:50:22.816 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:22.818+0000 7f46ecff9640 1 -- 172.21.15.123:0/546984040 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(138..138 src has 1..138) ==== 4971+0+0 (secure 0 0 0) 0x7f46e00950e0 con 0x7f46d8003fe0 2024-09-16T15:50:22.816 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:22.818+0000 7f46ef7fe640 1 --2- 172.21.15.123:0/546984040 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f46dc073580 0x7f46dc075a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:50:22.816 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:22.818+0000 7f46ef7fe640 1 --2- 172.21.15.123:0/546984040 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f46dc073580 0x7f46dc075a40 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f46e40ad000 tx=0x7f46e4019fc0 comp rx=0 tx=0).ready entity=mgr.4150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:50:22.818 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:22.818+0000 7f46ecff9640 1 -- 172.21.15.123:0/546984040 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f46f00b2a10 con 0x7f46d8003fe0 2024-09-16T15:50:22.991 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:22.990+0000 7f46f6845640 1 -- 172.21.15.123:0/546984040 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f46f00b5650 con 0x7f46d8003fe0 2024-09-16T15:50:22.992 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:22.994+0000 7f46ecff9640 1 -- 172.21.15.123:0/546984040 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v138) ==== 93+0+10 (secure 0 0 0) 0x7f46e005fbe0 con 0x7f46d8003fe0 2024-09-16T15:50:22.992 INFO:teuthology.orchestra.run.smithi123.stdout:pg_num: 1 2024-09-16T15:50:22.995 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:22.994+0000 7f46f6845640 1 -- 172.21.15.123:0/546984040 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f46dc073580 msgr2=0x7f46dc075a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:50:22.995 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:22.994+0000 7f46f6845640 1 --2- 172.21.15.123:0/546984040 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f46dc073580 0x7f46dc075a40 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f46e40ad000 tx=0x7f46e4019fc0 comp rx=0 tx=0).stop 2024-09-16T15:50:22.995 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:22.994+0000 7f46f6845640 1 -- 172.21.15.123:0/546984040 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f46d8003fe0 msgr2=0x7f46f00c7bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:50:22.995 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:22.994+0000 7f46f6845640 1 --2- 172.21.15.123:0/546984040 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f46d8003fe0 0x7f46f00c7bf0 secure :-1 s=READY pgs=1541 cs=0 l=1 rev1=1 crypto rx=0x7f46e000e620 tx=0x7f46e000eae0 comp rx=0 tx=0).stop 2024-09-16T15:50:22.995 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:22.994+0000 7f46f6845640 1 -- 172.21.15.123:0/546984040 shutdown_connections 2024-09-16T15:50:22.995 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:22.994+0000 7f46f6845640 1 --2- 172.21.15.123:0/546984040 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f46dc073580 0x7f46dc075a40 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:22.995 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:22.994+0000 7f46f6845640 1 --2- 172.21.15.123:0/546984040 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f46f00b4730 0x7f46f01fba80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:22.995 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:22.994+0000 7f46f6845640 1 --2- 172.21.15.123:0/546984040 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f46f00b42b0 0x7f46f01fb540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:22.995 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:22.994+0000 7f46f6845640 1 --2- 172.21.15.123:0/546984040 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f46d8003fe0 0x7f46f00c7bf0 unknown :-1 s=CLOSED pgs=1541 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:22.995 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:22.994+0000 7f46f6845640 1 -- 172.21.15.123:0/546984040 >> 172.21.15.123:0/546984040 conn(0x7f46f00b1710 msgr2=0x7f46f01560d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:50:22.996 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:22.994+0000 7f46f6845640 1 -- 172.21.15.123:0/546984040 shutdown_connections 2024-09-16T15:50:22.996 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:22.994+0000 7f46f6845640 1 -- 172.21.15.123:0/546984040 wait complete. 2024-09-16T15:50:23.015 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_volumes.TestMisc.test_subvolume_upgrade_legacy_to_v1' 2024-09-16T15:50:23.138 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:23.138+0000 7f786cbdf640 1 -- 172.21.15.123:0/1159318622 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7850004d50 msgr2=0x7f78500051d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:50:23.138 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:23.138+0000 7f786cbdf640 1 --2- 172.21.15.123:0/1159318622 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7850004d50 0x7f78500051d0 secure :-1 s=READY pgs=1543 cs=0 l=1 rev1=1 crypto rx=0x7f786000c660 tx=0x7f786001d2e0 comp rx=0 tx=0).stop 2024-09-16T15:50:23.138 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:23.138+0000 7f7866d9d640 1 -- 172.21.15.123:0/1159318622 reap_dead start 2024-09-16T15:50:23.138 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:23.138+0000 7f786cbdf640 1 -- 172.21.15.123:0/1159318622 shutdown_connections 2024-09-16T15:50:23.138 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:23.138+0000 7f786cbdf640 1 -- 172.21.15.123:0/1159318622 >> 172.21.15.123:0/1159318622 conn(0x7f7868074bd0 msgr2=0x7f786809aa40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:50:23.138 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:23.138+0000 7f786cbdf640 1 -- 172.21.15.123:0/1159318622 shutdown_connections 2024-09-16T15:50:23.138 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:23.138+0000 7f786cbdf640 1 -- 172.21.15.123:0/1159318622 wait complete. 2024-09-16T15:50:23.139 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:23.138+0000 7f786cbdf640 1 Processor -- start 2024-09-16T15:50:23.139 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:23.138+0000 7f786cbdf640 1 -- start start 2024-09-16T15:50:23.141 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:23.142+0000 7f786cbdf640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7850004d50 0x7f7868154200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:50:23.141 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:23.142+0000 7f786cbdf640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7868153d20 0x7f78680b6f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:50:23.141 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:23.142+0000 7f786cbdf640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f78680b7440 0x7f78680b5b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:50:23.141 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:23.142+0000 7f786cbdf640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f786806ee10 con 0x7f7868153d20 2024-09-16T15:50:23.141 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:23.142+0000 7f786cbdf640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f786815e280 con 0x7f7850004d50 2024-09-16T15:50:23.141 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:23.142+0000 7f786cbdf640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f786806ef90 con 0x7f78680b7440 2024-09-16T15:50:23.141 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:23.142+0000 7f7865d9b640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7868153d20 0x7f78680b6f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:50:23.141 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:23.142+0000 7f7865d9b640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7868153d20 0x7f78680b6f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:55840/0 (socket says 172.21.15.123:55840) 2024-09-16T15:50:23.142 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:23.142+0000 7f7865d9b640 1 -- 172.21.15.123:0/2844422467 learned_addr learned my addr 172.21.15.123:0/2844422467 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:50:23.142 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:23.142+0000 7f7866d9d640 1 --2- 172.21.15.123:0/2844422467 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f78680b7440 0x7f78680b5b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:50:23.142 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:23.142+0000 7f786659c640 1 --2- 172.21.15.123:0/2844422467 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7850004d50 0x7f7868154200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:50:23.142 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:23.142+0000 7f7865d9b640 1 -- 172.21.15.123:0/2844422467 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f78680b7440 msgr2=0x7f78680b5b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:50:23.142 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:23.142+0000 7f7865d9b640 1 --2- 172.21.15.123:0/2844422467 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f78680b7440 0x7f78680b5b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:23.142 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:23.142+0000 7f7865d9b640 1 -- 172.21.15.123:0/2844422467 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7850004d50 msgr2=0x7f7868154200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:50:23.142 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:23.142+0000 7f7865d9b640 1 --2- 172.21.15.123:0/2844422467 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7850004d50 0x7f7868154200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:23.142 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:23.142+0000 7f7865d9b640 1 -- 172.21.15.123:0/2844422467 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f78680b4d40 con 0x7f7868153d20 2024-09-16T15:50:23.142 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:23.142+0000 7f7865d9b640 1 --2- 172.21.15.123:0/2844422467 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7868153d20 0x7f78680b6f00 secure :-1 s=READY pgs=1544 cs=0 l=1 rev1=1 crypto rx=0x7f785c00d5e0 tx=0x7f785c00daa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:50:23.142 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:23.142+0000 7f7856ffd640 1 -- 172.21.15.123:0/2844422467 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f785c016020 con 0x7f7868153d20 2024-09-16T15:50:23.142 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:23.142+0000 7f7856ffd640 1 -- 172.21.15.123:0/2844422467 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f785c012630 con 0x7f7868153d20 2024-09-16T15:50:23.142 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:23.142+0000 7f7856ffd640 1 -- 172.21.15.123:0/2844422467 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f785c012a60 con 0x7f7868153d20 2024-09-16T15:50:23.142 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:23.142+0000 7f786cbdf640 1 -- 172.21.15.123:0/2844422467 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f78680b5020 con 0x7f7868153d20 2024-09-16T15:50:23.143 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:23.142+0000 7f786cbdf640 1 -- 172.21.15.123:0/2844422467 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7868154740 con 0x7f7868153d20 2024-09-16T15:50:23.143 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:23.142+0000 7f786cbdf640 1 -- 172.21.15.123:0/2844422467 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f78680d1c20 con 0x7f7868153d20 2024-09-16T15:50:23.143 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:23.142+0000 7f7856ffd640 1 -- 172.21.15.123:0/2844422467 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 22) ==== 97692+0+0 (secure 0 0 0) 0x7f785c019020 con 0x7f7868153d20 2024-09-16T15:50:23.144 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:23.142+0000 7f7856ffd640 1 --2- 172.21.15.123:0/2844422467 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f7848073580 0x7f7848075a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:50:23.144 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:23.142+0000 7f7856ffd640 1 -- 172.21.15.123:0/2844422467 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(138..138 src has 1..138) ==== 4971+0+0 (secure 0 0 0) 0x7f785c094350 con 0x7f7868153d20 2024-09-16T15:50:23.144 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:23.142+0000 7f786659c640 1 --2- 172.21.15.123:0/2844422467 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f7848073580 0x7f7848075a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:50:23.144 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:23.142+0000 7f786659c640 1 --2- 172.21.15.123:0/2844422467 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f7848073580 0x7f7848075a40 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f7858009ab0 tx=0x7f78580091e0 comp rx=0 tx=0).ready entity=mgr.4150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:50:23.147 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:23.146+0000 7f7856ffd640 1 -- 172.21.15.123:0/2844422467 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f785c020020 con 0x7f7868153d20 2024-09-16T15:50:23.385 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:23.386+0000 7f786cbdf640 1 -- 172.21.15.123:0/2844422467 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_volumes.TestMisc.test_subvolume_upgrade_legacy_to_v1"]} v 0) -- 0x7f78680b7820 con 0x7f7868153d20 2024-09-16T15:50:24.333 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:24.334+0000 7f7856ffd640 1 -- 172.21.15.123:0/2844422467 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_volumes.TestMisc.test_subvolume_upgrade_legacy_to_v1"]}]=0 v396) ==== 152+0+0 (secure 0 0 0) 0x7f785c05f070 con 0x7f7868153d20 2024-09-16T15:50:24.337 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:24.338+0000 7f786cbdf640 1 -- 172.21.15.123:0/2844422467 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f7848073580 msgr2=0x7f7848075a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:50:24.337 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:24.338+0000 7f786cbdf640 1 --2- 172.21.15.123:0/2844422467 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f7848073580 0x7f7848075a40 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f7858009ab0 tx=0x7f78580091e0 comp rx=0 tx=0).stop 2024-09-16T15:50:24.337 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:24.338+0000 7f786cbdf640 1 -- 172.21.15.123:0/2844422467 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7868153d20 msgr2=0x7f78680b6f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:50:24.337 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:24.338+0000 7f786cbdf640 1 --2- 172.21.15.123:0/2844422467 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7868153d20 0x7f78680b6f00 secure :-1 s=READY pgs=1544 cs=0 l=1 rev1=1 crypto rx=0x7f785c00d5e0 tx=0x7f785c00daa0 comp rx=0 tx=0).stop 2024-09-16T15:50:24.337 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:24.338+0000 7f786cbdf640 1 -- 172.21.15.123:0/2844422467 shutdown_connections 2024-09-16T15:50:24.337 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:24.338+0000 7f786cbdf640 1 --2- 172.21.15.123:0/2844422467 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f7848073580 0x7f7848075a40 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:24.337 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:24.338+0000 7f786cbdf640 1 --2- 172.21.15.123:0/2844422467 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f78680b7440 0x7f78680b5b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:24.337 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:24.338+0000 7f786cbdf640 1 --2- 172.21.15.123:0/2844422467 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7868153d20 0x7f78680b6f00 unknown :-1 s=CLOSED pgs=1544 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:24.337 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:24.338+0000 7f786cbdf640 1 --2- 172.21.15.123:0/2844422467 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7850004d50 0x7f7868154200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:24.337 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:24.338+0000 7f786cbdf640 1 -- 172.21.15.123:0/2844422467 >> 172.21.15.123:0/2844422467 conn(0x7f7868074bd0 msgr2=0x7f786815d360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:50:24.338 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:24.338+0000 7f786cbdf640 1 -- 172.21.15.123:0/2844422467 shutdown_connections 2024-09-16T15:50:24.338 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:24.338+0000 7f786cbdf640 1 -- 172.21.15.123:0/2844422467 wait complete. 2024-09-16T15:50:24.356 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config dump 2024-09-16T15:50:24.486 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:24.486+0000 7fc4a7d84640 1 -- 172.21.15.123:0/198370781 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc48c004d50 msgr2=0x7fc48c0051d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:50:24.486 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:24.486+0000 7fc4a7d84640 1 --2- 172.21.15.123:0/198370781 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc48c004d50 0x7fc48c0051d0 secure :-1 s=READY pgs=1546 cs=0 l=1 rev1=1 crypto rx=0x7fc49800bb10 tx=0x7fc49801cde0 comp rx=0 tx=0).stop 2024-09-16T15:50:24.486 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:24.486+0000 7fc4a6321640 1 -- 172.21.15.123:0/198370781 reap_dead start 2024-09-16T15:50:24.486 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:24.486+0000 7fc4a7d84640 1 -- 172.21.15.123:0/198370781 shutdown_connections 2024-09-16T15:50:24.486 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:24.486+0000 7fc4a7d84640 1 -- 172.21.15.123:0/198370781 >> 172.21.15.123:0/198370781 conn(0x7fc4a00cfb10 msgr2=0x7fc4a00cff20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:50:24.486 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:24.486+0000 7fc4a7d84640 1 -- 172.21.15.123:0/198370781 shutdown_connections 2024-09-16T15:50:24.487 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:24.486+0000 7fc4a7d84640 1 -- 172.21.15.123:0/198370781 wait complete. 2024-09-16T15:50:24.487 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:24.486+0000 7fc4a7d84640 1 Processor -- start 2024-09-16T15:50:24.487 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:24.486+0000 7fc4a7d84640 1 -- start start 2024-09-16T15:50:24.490 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:24.490+0000 7fc4a7d84640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc48c004d50 0x7fc4a00b38c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:50:24.490 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:24.490+0000 7fc4a7d84640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fc4a0148a10 0x7fc4a0148e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:50:24.490 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:24.490+0000 7fc4a7d84640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc4a01f5b40 0x7fc4a01f5fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:50:24.490 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:24.490+0000 7fc4a7d84640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fc4a00d09c0 con 0x7fc4a01f5b40 2024-09-16T15:50:24.490 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:24.490+0000 7fc4a7d84640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fc4a00d0840 con 0x7fc48c004d50 2024-09-16T15:50:24.490 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:24.490+0000 7fc4a7d84640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fc4a006ec90 con 0x7fc4a0148a10 2024-09-16T15:50:24.490 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:24.490+0000 7fc4a6321640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc4a01f5b40 0x7fc4a01f5fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:50:24.490 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:24.490+0000 7fc4a6321640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc4a01f5b40 0x7fc4a01f5fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:55862/0 (socket says 172.21.15.123:55862) 2024-09-16T15:50:24.490 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:24.490+0000 7fc4a6321640 1 -- 172.21.15.123:0/720549823 learned_addr learned my addr 172.21.15.123:0/720549823 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:50:24.490 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:24.490+0000 7fc4a5b20640 1 --2- 172.21.15.123:0/720549823 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc48c004d50 0x7fc4a00b38c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:50:24.491 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:24.490+0000 7fc4a531f640 1 --2- 172.21.15.123:0/720549823 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fc4a0148a10 0x7fc4a0148e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:50:24.491 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:24.490+0000 7fc4a6321640 1 -- 172.21.15.123:0/720549823 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fc4a0148a10 msgr2=0x7fc4a0148e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:50:24.491 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:24.490+0000 7fc4a6321640 1 --2- 172.21.15.123:0/720549823 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fc4a0148a10 0x7fc4a0148e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:24.491 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:24.490+0000 7fc4a6321640 1 -- 172.21.15.123:0/720549823 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc48c004d50 msgr2=0x7fc4a00b38c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:50:24.491 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:24.490+0000 7fc4a6321640 1 --2- 172.21.15.123:0/720549823 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc48c004d50 0x7fc4a00b38c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:24.491 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:24.490+0000 7fc4a6321640 1 -- 172.21.15.123:0/720549823 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc4a01f6600 con 0x7fc4a01f5b40 2024-09-16T15:50:24.491 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:24.490+0000 7fc4a6321640 1 --2- 172.21.15.123:0/720549823 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc4a01f5b40 0x7fc4a01f5fd0 secure :-1 s=READY pgs=1547 cs=0 l=1 rev1=1 crypto rx=0x7fc4980b1040 tx=0x7fc498005570 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:50:24.491 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:24.490+0000 7fc4927fc640 1 -- 172.21.15.123:0/720549823 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4980ab050 con 0x7fc4a01f5b40 2024-09-16T15:50:24.491 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:24.490+0000 7fc4927fc640 1 -- 172.21.15.123:0/720549823 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fc498000e50 con 0x7fc4a01f5b40 2024-09-16T15:50:24.491 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:24.490+0000 7fc4a7d84640 1 -- 172.21.15.123:0/720549823 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc4a01f68e0 con 0x7fc4a01f5b40 2024-09-16T15:50:24.491 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:24.490+0000 7fc4927fc640 1 -- 172.21.15.123:0/720549823 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc498004280 con 0x7fc4a01f5b40 2024-09-16T15:50:24.491 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:24.490+0000 7fc4a7d84640 1 -- 172.21.15.123:0/720549823 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc4a00c8120 con 0x7fc4a01f5b40 2024-09-16T15:50:24.492 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:24.490+0000 7fc4a7d84640 1 -- 172.21.15.123:0/720549823 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc4a00c7c30 con 0x7fc4a01f5b40 2024-09-16T15:50:24.492 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:24.490+0000 7fc4927fc640 1 -- 172.21.15.123:0/720549823 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 22) ==== 97692+0+0 (secure 0 0 0) 0x7fc4980c0060 con 0x7fc4a01f5b40 2024-09-16T15:50:24.492 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:24.490+0000 7fc4927fc640 1 --2- 172.21.15.123:0/720549823 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fc4840735b0 0x7fc484075a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:50:24.492 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:24.490+0000 7fc4927fc640 1 -- 172.21.15.123:0/720549823 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(138..138 src has 1..138) ==== 4971+0+0 (secure 0 0 0) 0x7fc49812e240 con 0x7fc4a01f5b40 2024-09-16T15:50:24.492 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:24.490+0000 7fc4a5b20640 1 --2- 172.21.15.123:0/720549823 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fc4840735b0 0x7fc484075a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:50:24.493 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:24.494+0000 7fc4a5b20640 1 --2- 172.21.15.123:0/720549823 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fc4840735b0 0x7fc484075a70 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7fc49400cc30 tx=0x7fc494005f30 comp rx=0 tx=0).ready entity=mgr.4150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:50:24.496 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:24.494+0000 7fc4927fc640 1 -- 172.21.15.123:0/720549823 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7fc498016af0 con 0x7fc4a01f5b40 2024-09-16T15:50:24.668 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:24.670+0000 7fc4a7d84640 1 -- 172.21.15.123:0/720549823 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "config dump"} v 0) -- 0x7fc4a006e750 con 0x7fc4a01f5b40 2024-09-16T15:50:24.669 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:24.670+0000 7fc4927fc640 1 -- 172.21.15.123:0/720549823 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "config dump"}]=0 v43) ==== 59+0+1998 (secure 0 0 0) 0x7fc4980f8c40 con 0x7fc4a01f5b40 2024-09-16T15:50:24.669 INFO:teuthology.orchestra.run.smithi123.stdout:WHO MASK LEVEL OPTION VALUE RO 2024-09-16T15:50:24.669 INFO:teuthology.orchestra.run.smithi123.stdout:mon advanced mon_op_complaint_time 120 2024-09-16T15:50:24.669 INFO:teuthology.orchestra.run.smithi123.stdout:mon dev mon_warn_on_pool_no_app false 2024-09-16T15:50:24.669 INFO:teuthology.orchestra.run.smithi123.stdout:osd advanced osd_op_complaint_time 180.000000 2024-09-16T15:50:24.670 INFO:teuthology.orchestra.run.smithi123.stdout:osd.0 basic osd_mclock_max_capacity_iops_ssd 4805.300677 2024-09-16T15:50:24.670 INFO:teuthology.orchestra.run.smithi123.stdout:osd.1 basic osd_mclock_max_capacity_iops_ssd 4806.240607 2024-09-16T15:50:24.670 INFO:teuthology.orchestra.run.smithi123.stdout:osd.2 basic osd_mclock_max_capacity_iops_ssd 4046.441686 2024-09-16T15:50:24.670 INFO:teuthology.orchestra.run.smithi123.stdout:osd.3 basic osd_mclock_max_capacity_iops_ssd 6882.576123 2024-09-16T15:50:24.670 INFO:teuthology.orchestra.run.smithi123.stdout:osd.4 basic osd_mclock_max_capacity_iops_ssd 6108.444971 2024-09-16T15:50:24.670 INFO:teuthology.orchestra.run.smithi123.stdout:osd.5 basic osd_mclock_max_capacity_iops_ssd 4995.096306 2024-09-16T15:50:24.670 INFO:teuthology.orchestra.run.smithi123.stdout:osd.6 basic osd_mclock_max_capacity_iops_ssd 6237.822095 2024-09-16T15:50:24.670 INFO:teuthology.orchestra.run.smithi123.stdout:osd.7 basic osd_mclock_max_capacity_iops_ssd 5202.385138 2024-09-16T15:50:24.670 INFO:teuthology.orchestra.run.smithi123.stdout:mds advanced debug_mds 20/20 2024-09-16T15:50:24.670 INFO:teuthology.orchestra.run.smithi123.stdout:mds advanced debug_mds_balancer 20/20 2024-09-16T15:50:24.670 INFO:teuthology.orchestra.run.smithi123.stdout:mds advanced debug_ms 2/2 2024-09-16T15:50:24.670 INFO:teuthology.orchestra.run.smithi123.stdout:mds dev mds_debug_frag true 2024-09-16T15:50:24.670 INFO:teuthology.orchestra.run.smithi123.stdout:mds dev mds_debug_scatterstat true 2024-09-16T15:50:24.671 INFO:teuthology.orchestra.run.smithi123.stdout:mds advanced mds_op_complaint_time 180.000000 2024-09-16T15:50:24.671 INFO:teuthology.orchestra.run.smithi123.stdout:mds dev mds_verify_scatter true 2024-09-16T15:50:24.671 INFO:teuthology.orchestra.run.smithi123.stdout:mds advanced osd_op_complaint_time 180.000000 2024-09-16T15:50:24.671 INFO:teuthology.orchestra.run.smithi123.stdout:mds advanced rados_mon_op_timeout 900 2024-09-16T15:50:24.671 INFO:teuthology.orchestra.run.smithi123.stdout:mds advanced rados_osd_op_timeout 900 2024-09-16T15:50:24.671 INFO:teuthology.orchestra.run.smithi123.stdout:client advanced client_mount_timeout 600 2024-09-16T15:50:24.671 INFO:teuthology.orchestra.run.smithi123.stdout:client advanced debug_client 20/20 2024-09-16T15:50:24.671 INFO:teuthology.orchestra.run.smithi123.stdout:client advanced debug_ms 1/1 2024-09-16T15:50:24.671 INFO:teuthology.orchestra.run.smithi123.stdout:client advanced rados_mon_op_timeout 900 2024-09-16T15:50:24.671 INFO:teuthology.orchestra.run.smithi123.stdout:client advanced rados_osd_op_timeout 900 2024-09-16T15:50:24.672 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:24.670+0000 7fc4a7d84640 1 -- 172.21.15.123:0/720549823 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fc4840735b0 msgr2=0x7fc484075a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:50:24.672 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:24.670+0000 7fc4a7d84640 1 --2- 172.21.15.123:0/720549823 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fc4840735b0 0x7fc484075a70 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7fc49400cc30 tx=0x7fc494005f30 comp rx=0 tx=0).stop 2024-09-16T15:50:24.672 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:24.670+0000 7fc4a7d84640 1 -- 172.21.15.123:0/720549823 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc4a01f5b40 msgr2=0x7fc4a01f5fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:50:24.672 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:24.670+0000 7fc4a7d84640 1 --2- 172.21.15.123:0/720549823 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc4a01f5b40 0x7fc4a01f5fd0 secure :-1 s=READY pgs=1547 cs=0 l=1 rev1=1 crypto rx=0x7fc4980b1040 tx=0x7fc498005570 comp rx=0 tx=0).stop 2024-09-16T15:50:24.672 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:24.670+0000 7fc4a7d84640 1 -- 172.21.15.123:0/720549823 shutdown_connections 2024-09-16T15:50:24.672 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:24.670+0000 7fc4a7d84640 1 --2- 172.21.15.123:0/720549823 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fc4840735b0 0x7fc484075a70 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:24.673 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:24.670+0000 7fc4a7d84640 1 --2- 172.21.15.123:0/720549823 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc4a01f5b40 0x7fc4a01f5fd0 unknown :-1 s=CLOSED pgs=1547 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:24.673 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:24.670+0000 7fc4a7d84640 1 --2- 172.21.15.123:0/720549823 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fc4a0148a10 0x7fc4a0148e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:24.673 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:24.670+0000 7fc4a7d84640 1 --2- 172.21.15.123:0/720549823 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc48c004d50 0x7fc4a00b38c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:24.673 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:24.670+0000 7fc4a7d84640 1 -- 172.21.15.123:0/720549823 >> 172.21.15.123:0/720549823 conn(0x7fc4a00cfb10 msgr2=0x7fc4a00b4810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:50:24.673 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:24.674+0000 7fc4a7d84640 1 -- 172.21.15.123:0/720549823 shutdown_connections 2024-09-16T15:50:24.673 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:24.674+0000 7fc4a7d84640 1 -- 172.21.15.123:0/720549823 wait complete. 2024-09-16T15:50:24.690 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-09-16T15:50:24.825 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:24.826+0000 7ff2c543d640 1 -- 172.21.15.123:0/470414355 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff2c00d2120 msgr2=0x7ff2c00cef00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:50:24.825 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:24.826+0000 7ff2c543d640 1 --2- 172.21.15.123:0/470414355 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff2c00d2120 0x7ff2c00cef00 secure :-1 s=READY pgs=1548 cs=0 l=1 rev1=1 crypto rx=0x7ff2b8009c80 tx=0x7ff2b801cc80 comp rx=0 tx=0).stop 2024-09-16T15:50:24.825 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:24.826+0000 7ff2c543d640 1 -- 172.21.15.123:0/470414355 shutdown_connections 2024-09-16T15:50:24.825 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:24.826+0000 7ff2c543d640 1 --2- 172.21.15.123:0/470414355 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff2c00d2120 0x7ff2c00cef00 unknown :-1 s=CLOSED pgs=1548 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:24.826 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:24.826+0000 7ff2c543d640 1 --2- 172.21.15.123:0/470414355 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7ff2c00d0f60 0x7ff2c00ce9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:24.826 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:24.826+0000 7ff2c543d640 1 -- 172.21.15.123:0/470414355 >> 172.21.15.123:0/470414355 conn(0x7ff2c00d3e40 msgr2=0x7ff2c00d4220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:50:24.826 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:24.826+0000 7ff2c543d640 1 -- 172.21.15.123:0/470414355 shutdown_connections 2024-09-16T15:50:24.826 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:24.826+0000 7ff2c543d640 1 -- 172.21.15.123:0/470414355 wait complete. 2024-09-16T15:50:24.828 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:24.826+0000 7ff2c543d640 1 Processor -- start 2024-09-16T15:50:24.828 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:24.826+0000 7ff2c543d640 1 -- start start 2024-09-16T15:50:24.828 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:24.826+0000 7ff2c543d640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff2c00d0f60 0x7ff2c0150450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:50:24.828 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:24.826+0000 7ff2c543d640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff2c00d2120 0x7ff2c0150e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:50:24.828 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:24.826+0000 7ff2c543d640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7ff2c01615c0 0x7ff2c00b95a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:50:24.828 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:24.826+0000 7ff2c543d640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7ff2c016b590 con 0x7ff2c00d2120 2024-09-16T15:50:24.828 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:24.826+0000 7ff2c543d640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7ff2c016b410 con 0x7ff2c00d0f60 2024-09-16T15:50:24.828 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:24.826+0000 7ff2c543d640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7ff2c016b710 con 0x7ff2c01615c0 2024-09-16T15:50:24.828 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:24.826+0000 7ff2be7fc640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff2c00d2120 0x7ff2c0150e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:50:24.829 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:24.826+0000 7ff2be7fc640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff2c00d2120 0x7ff2c0150e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:55872/0 (socket says 172.21.15.123:55872) 2024-09-16T15:50:24.829 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:24.826+0000 7ff2be7fc640 1 -- 172.21.15.123:0/1640104338 learned_addr learned my addr 172.21.15.123:0/1640104338 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:50:24.829 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:24.826+0000 7ff2beffd640 1 --2- 172.21.15.123:0/1640104338 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff2c00d0f60 0x7ff2c0150450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:50:24.829 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:24.826+0000 7ff2bf7fe640 1 --2- 172.21.15.123:0/1640104338 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7ff2c01615c0 0x7ff2c00b95a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:50:24.829 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:24.826+0000 7ff2be7fc640 1 -- 172.21.15.123:0/1640104338 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7ff2c01615c0 msgr2=0x7ff2c00b95a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:50:24.829 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:24.826+0000 7ff2be7fc640 1 --2- 172.21.15.123:0/1640104338 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7ff2c01615c0 0x7ff2c00b95a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:24.829 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:24.826+0000 7ff2be7fc640 1 -- 172.21.15.123:0/1640104338 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff2c00d0f60 msgr2=0x7ff2c0150450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:50:24.829 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:24.826+0000 7ff2be7fc640 1 --2- 172.21.15.123:0/1640104338 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff2c00d0f60 0x7ff2c0150450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:24.829 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:24.826+0000 7ff2be7fc640 1 -- 172.21.15.123:0/1640104338 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff2c0152c10 con 0x7ff2c00d2120 2024-09-16T15:50:24.829 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:24.826+0000 7ff2be7fc640 1 --2- 172.21.15.123:0/1640104338 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff2c00d2120 0x7ff2c0150e60 secure :-1 s=READY pgs=1549 cs=0 l=1 rev1=1 crypto rx=0x7ff2b400d770 tx=0x7ff2b400dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:50:24.829 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:24.826+0000 7ff2a77fe640 1 -- 172.21.15.123:0/1640104338 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff2b4017020 con 0x7ff2c00d2120 2024-09-16T15:50:24.829 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:24.830+0000 7ff2a77fe640 1 -- 172.21.15.123:0/1640104338 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7ff2b4009820 con 0x7ff2c00d2120 2024-09-16T15:50:24.830 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:24.830+0000 7ff2c543d640 1 -- 172.21.15.123:0/1640104338 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff2c0152ef0 con 0x7ff2c00d2120 2024-09-16T15:50:24.830 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:24.830+0000 7ff2a77fe640 1 -- 172.21.15.123:0/1640104338 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff2b4009c50 con 0x7ff2c00d2120 2024-09-16T15:50:24.830 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:24.830+0000 7ff2c543d640 1 -- 172.21.15.123:0/1640104338 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff2c01528e0 con 0x7ff2c00d2120 2024-09-16T15:50:24.831 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:24.830+0000 7ff2c543d640 1 -- 172.21.15.123:0/1640104338 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff2c00d0850 con 0x7ff2c00d2120 2024-09-16T15:50:24.831 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:24.830+0000 7ff2a77fe640 1 -- 172.21.15.123:0/1640104338 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 22) ==== 97692+0+0 (secure 0 0 0) 0x7ff2b401a020 con 0x7ff2c00d2120 2024-09-16T15:50:24.831 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:24.830+0000 7ff2a77fe640 1 --2- 172.21.15.123:0/1640104338 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7ff2ac073580 0x7ff2ac075a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:50:24.831 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:24.830+0000 7ff2a77fe640 1 -- 172.21.15.123:0/1640104338 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(138..138 src has 1..138) ==== 4971+0+0 (secure 0 0 0) 0x7ff2b4095380 con 0x7ff2c00d2120 2024-09-16T15:50:24.831 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:24.830+0000 7ff2beffd640 1 --2- 172.21.15.123:0/1640104338 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7ff2ac073580 0x7ff2ac075a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:50:24.831 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:24.830+0000 7ff2beffd640 1 --2- 172.21.15.123:0/1640104338 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7ff2ac073580 0x7ff2ac075a40 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7ff2b0001570 tx=0x7ff2b0009040 comp rx=0 tx=0).ready entity=mgr.4150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:50:24.834 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:24.834+0000 7ff2a77fe640 1 -- 172.21.15.123:0/1640104338 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7ff2b4021020 con 0x7ff2c00d2120 2024-09-16T15:50:25.002 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:25.002+0000 7ff2c543d640 1 -- 172.21.15.123:0/1640104338 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) -- 0x7ff2c006ed10 con 0x7ff2c00d2120 2024-09-16T15:50:25.007 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:25.006+0000 7ff2a77fe640 1 -- 172.21.15.123:0/1640104338 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v44)=0 v44) ==== 122+0+0 (secure 0 0 0) 0x7ff2b405fe10 con 0x7ff2c00d2120 2024-09-16T15:50:25.010 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:25.010+0000 7ff2c543d640 1 -- 172.21.15.123:0/1640104338 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7ff2ac073580 msgr2=0x7ff2ac075a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:50:25.010 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:25.010+0000 7ff2c543d640 1 --2- 172.21.15.123:0/1640104338 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7ff2ac073580 0x7ff2ac075a40 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7ff2b0001570 tx=0x7ff2b0009040 comp rx=0 tx=0).stop 2024-09-16T15:50:25.010 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:25.010+0000 7ff2c543d640 1 -- 172.21.15.123:0/1640104338 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff2c00d2120 msgr2=0x7ff2c0150e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:50:25.010 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:25.010+0000 7ff2c543d640 1 --2- 172.21.15.123:0/1640104338 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff2c00d2120 0x7ff2c0150e60 secure :-1 s=READY pgs=1549 cs=0 l=1 rev1=1 crypto rx=0x7ff2b400d770 tx=0x7ff2b400dc30 comp rx=0 tx=0).stop 2024-09-16T15:50:25.010 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:25.010+0000 7ff2c543d640 1 -- 172.21.15.123:0/1640104338 shutdown_connections 2024-09-16T15:50:25.010 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:25.010+0000 7ff2c543d640 1 --2- 172.21.15.123:0/1640104338 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7ff2ac073580 0x7ff2ac075a40 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:25.011 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:25.010+0000 7ff2c543d640 1 --2- 172.21.15.123:0/1640104338 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7ff2c01615c0 0x7ff2c00b95a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:25.011 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:25.010+0000 7ff2c543d640 1 --2- 172.21.15.123:0/1640104338 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff2c00d2120 0x7ff2c0150e60 unknown :-1 s=CLOSED pgs=1549 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:25.011 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:25.010+0000 7ff2c543d640 1 --2- 172.21.15.123:0/1640104338 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff2c00d0f60 0x7ff2c0150450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:25.011 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:25.010+0000 7ff2c543d640 1 -- 172.21.15.123:0/1640104338 >> 172.21.15.123:0/1640104338 conn(0x7ff2c00d3e40 msgr2=0x7ff2c006e470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:50:25.011 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:25.010+0000 7ff2c543d640 1 -- 172.21.15.123:0/1640104338 shutdown_connections 2024-09-16T15:50:25.011 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:25.010+0000 7ff2c543d640 1 -- 172.21.15.123:0/1640104338 wait complete. 2024-09-16T15:50:25.026 INFO:teuthology.nuke:Clearing teuthology firewall rules... 2024-09-16T15:50:25.026 DEBUG:teuthology.orchestra.run.smithi123:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-09-16T15:50:25.093 DEBUG:teuthology.orchestra.run.smithi184:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-09-16T15:50:25.170 INFO:teuthology.nuke:Cleared teuthology firewall rules. 2024-09-16T15:50:25.170 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2024-09-16T15:50:25.170 DEBUG:teuthology.orchestra.run.smithi123:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-16T15:50:25.176 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16336468k,nr_inodes=4084117,mode=755,inode64 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3274684k,mode=755,inode64 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev,inode64 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=17082 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3274680k,nr_inodes=818670,mode=700,uid=1000,gid=1271,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3274684k,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-09-16T15:50:25.176 DEBUG:teuthology.orchestra.run.smithi184:> set -ex 2024-09-16T15:50:25.176 DEBUG:teuthology.orchestra.run.smithi184:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-16T15:50:25.181 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16336488k,nr_inodes=4084122,mode=755,inode64 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3274688k,mode=755,inode64 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev,inode64 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=17834 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3274684k,nr_inodes=818671,mode=700,uid=1000,gid=1271,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3274688k,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-09-16T15:50:25.182 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2024-09-16T15:50:25.182 DEBUG:teuthology.orchestra.run.smithi123:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-16T15:50:25.225 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16336468k,nr_inodes=4084117,mode=755,inode64 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3274684k,mode=755,inode64 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev,inode64 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=17082 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3274680k,nr_inodes=818670,mode=700,uid=1000,gid=1271,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3274684k,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-09-16T15:50:25.225 DEBUG:teuthology.orchestra.run.smithi184:> set -ex 2024-09-16T15:50:25.225 DEBUG:teuthology.orchestra.run.smithi184:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-16T15:50:25.233 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16336488k,nr_inodes=4084122,mode=755,inode64 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3274688k,mode=755,inode64 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev,inode64 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=17834 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3274684k,nr_inodes=818671,mode=700,uid=1000,gid=1271,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3274688k,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-09-16T15:50:25.233 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-09-16T15:50:25.234 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-16T15:50:25.402 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:25.402+0000 7fd1d73ad640 1 -- 172.21.15.123:0/406056546 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fd1d015e820 msgr2=0x7fd1d015ecb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:50:25.402 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:25.402+0000 7fd1d73ad640 1 --2- 172.21.15.123:0/406056546 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fd1d015e820 0x7fd1d015ecb0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fd1c8009c80 tx=0x7fd1c801cd10 comp rx=0 tx=0).stop 2024-09-16T15:50:25.402 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:25.402+0000 7fd1d73ad640 1 -- 172.21.15.123:0/406056546 shutdown_connections 2024-09-16T15:50:25.402 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:25.402+0000 7fd1d73ad640 1 --2- 172.21.15.123:0/406056546 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fd1d015e820 0x7fd1d015ecb0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:25.402 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:25.402+0000 7fd1d73ad640 1 --2- 172.21.15.123:0/406056546 >> v2:172.21.15.184:3300/0 conn(0x7fd1d00cd6e0 0x7fd1d00cdb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:25.403 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:25.402+0000 7fd1d73ad640 1 -- 172.21.15.123:0/406056546 >> 172.21.15.123:0/406056546 conn(0x7fd1d00cfb10 msgr2=0x7fd1d00cff20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:50:25.403 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:25.402+0000 7fd1d73ad640 1 -- 172.21.15.123:0/406056546 shutdown_connections 2024-09-16T15:50:25.403 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:25.402+0000 7fd1d73ad640 1 -- 172.21.15.123:0/406056546 wait complete. 2024-09-16T15:50:25.403 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:25.402+0000 7fd1d73ad640 1 Processor -- start 2024-09-16T15:50:25.403 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:25.402+0000 7fd1d73ad640 1 -- start start 2024-09-16T15:50:25.405 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:25.406+0000 7fd1d73ad640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd1d00cd6e0 0x7fd1d01f7c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:50:25.406 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:25.406+0000 7fd1d73ad640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fd1d015e160 0x7fd1d01577b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:50:25.406 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:25.406+0000 7fd1d73ad640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd1d015e820 0x7fd1d00caa60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:50:25.406 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:25.406+0000 7fd1d73ad640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fd1d00cf5b0 con 0x7fd1d015e820 2024-09-16T15:50:25.406 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:25.406+0000 7fd1d73ad640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fd1d00cf430 con 0x7fd1d00cd6e0 2024-09-16T15:50:25.406 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:25.406+0000 7fd1d73ad640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fd1d006eee0 con 0x7fd1d015e160 2024-09-16T15:50:25.406 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:25.406+0000 7fd1d594a640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd1d015e820 0x7fd1d00caa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:50:25.406 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:25.406+0000 7fd1d594a640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd1d015e820 0x7fd1d00caa60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:55878/0 (socket says 172.21.15.123:55878) 2024-09-16T15:50:25.406 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:25.406+0000 7fd1d594a640 1 -- 172.21.15.123:0/3728247216 learned_addr learned my addr 172.21.15.123:0/3728247216 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:50:25.406 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:25.406+0000 7fd1d4948640 1 --2- 172.21.15.123:0/3728247216 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fd1d015e160 0x7fd1d01577b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:50:25.406 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:25.406+0000 7fd1d5149640 1 --2- 172.21.15.123:0/3728247216 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd1d00cd6e0 0x7fd1d01f7c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:50:25.407 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:25.406+0000 7fd1d594a640 1 -- 172.21.15.123:0/3728247216 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fd1d015e160 msgr2=0x7fd1d01577b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:50:25.407 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:25.406+0000 7fd1d594a640 1 --2- 172.21.15.123:0/3728247216 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fd1d015e160 0x7fd1d01577b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:25.407 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:25.406+0000 7fd1d594a640 1 -- 172.21.15.123:0/3728247216 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd1d00cd6e0 msgr2=0x7fd1d01f7c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:50:25.407 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:25.406+0000 7fd1d594a640 1 --2- 172.21.15.123:0/3728247216 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd1d00cd6e0 0x7fd1d01f7c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:25.407 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:25.406+0000 7fd1d594a640 1 -- 172.21.15.123:0/3728247216 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd1d0157cf0 con 0x7fd1d015e820 2024-09-16T15:50:25.407 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:25.406+0000 7fd1d4948640 1 --2- 172.21.15.123:0/3728247216 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fd1d015e160 0x7fd1d01577b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T15:50:25.407 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:25.406+0000 7fd1d594a640 1 --2- 172.21.15.123:0/3728247216 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd1d015e820 0x7fd1d00caa60 secure :-1 s=READY pgs=1550 cs=0 l=1 rev1=1 crypto rx=0x7fd1c801d5b0 tx=0x7fd1c8005ae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:50:25.407 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:25.406+0000 7fd1c5ffb640 1 -- 172.21.15.123:0/3728247216 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1c80bc070 con 0x7fd1d015e820 2024-09-16T15:50:25.407 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:25.406+0000 7fd1c5ffb640 1 -- 172.21.15.123:0/3728247216 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fd1c80040c0 con 0x7fd1d015e820 2024-09-16T15:50:25.407 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:25.406+0000 7fd1c5ffb640 1 -- 172.21.15.123:0/3728247216 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1c80b2040 con 0x7fd1d015e820 2024-09-16T15:50:25.407 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:25.406+0000 7fd1d73ad640 1 -- 172.21.15.123:0/3728247216 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd1d0157fc0 con 0x7fd1d015e820 2024-09-16T15:50:25.407 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:25.406+0000 7fd1d73ad640 1 -- 172.21.15.123:0/3728247216 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd1d0158580 con 0x7fd1d015e820 2024-09-16T15:50:25.408 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:25.406+0000 7fd1d73ad640 1 -- 172.21.15.123:0/3728247216 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd1d00cdb50 con 0x7fd1d015e820 2024-09-16T15:50:25.408 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:25.406+0000 7fd1c5ffb640 1 -- 172.21.15.123:0/3728247216 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 22) ==== 97692+0+0 (secure 0 0 0) 0x7fd1c80bf070 con 0x7fd1d015e820 2024-09-16T15:50:25.408 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:25.406+0000 7fd1c5ffb640 1 --2- 172.21.15.123:0/3728247216 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fd1b4073580 0x7fd1b4075a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:50:25.408 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:25.406+0000 7fd1c5ffb640 1 -- 172.21.15.123:0/3728247216 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(138..138 src has 1..138) ==== 4971+0+0 (secure 0 0 0) 0x7fd1c812e300 con 0x7fd1d015e820 2024-09-16T15:50:25.409 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:25.406+0000 7fd1d5149640 1 --2- 172.21.15.123:0/3728247216 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fd1b4073580 0x7fd1b4075a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:50:25.409 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:25.410+0000 7fd1d5149640 1 --2- 172.21.15.123:0/3728247216 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fd1b4073580 0x7fd1b4075a40 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7fd1cc0016f0 tx=0x7fd1cc009040 comp rx=0 tx=0).ready entity=mgr.4150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:50:25.411 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:25.410+0000 7fd1c5ffb640 1 -- 172.21.15.123:0/3728247216 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7fd1c8016990 con 0x7fd1d015e820 2024-09-16T15:50:25.620 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:25.618+0000 7fd1d73ad640 1 -- 172.21.15.123:0/3728247216 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd1d00d06c0 con 0x7fd1d015e820 2024-09-16T15:50:25.621 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:25.622+0000 7fd1c5ffb640 1 -- 172.21.15.123:0/3728247216 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 89 v89) ==== 94+0+3914 (secure 0 0 0) 0x7fd1c80f8f90 con 0x7fd1d015e820 2024-09-16T15:50:25.621 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-09-16T15:50:25.621 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":89,"btime":"2024-09-16T15:50:19:293046+0000","default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5801,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6834/730567479","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":730567479},{"type":"v1","addr":"172.21.15.184:6834","nonce":730567479}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":81},{"gid":6346,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6835/584518652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6833","nonce":584518652},{"type":"v1","addr":"172.21.15.184:6835","nonce":584518652}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":81},{"gid":7938,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6833/3935767668","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6832","nonce":3935767668},{"type":"v1","addr":"172.21.15.123:6833","nonce":3935767668}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":80},{"gid":8343,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6835/903430319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6834","nonce":903430319},{"type":"v1","addr":"172.21.15.123:6835","nonce":903430319}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":89}],"filesystems":[]} 2024-09-16T15:50:25.621 INFO:teuthology.orchestra.run.smithi123.stderr:dumped fsmap epoch 89 2024-09-16T15:50:25.623 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:25.622+0000 7fd1d73ad640 1 -- 172.21.15.123:0/3728247216 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fd1b4073580 msgr2=0x7fd1b4075a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:50:25.623 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:25.622+0000 7fd1d73ad640 1 --2- 172.21.15.123:0/3728247216 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fd1b4073580 0x7fd1b4075a40 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7fd1cc0016f0 tx=0x7fd1cc009040 comp rx=0 tx=0).stop 2024-09-16T15:50:25.624 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:25.622+0000 7fd1d73ad640 1 -- 172.21.15.123:0/3728247216 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd1d015e820 msgr2=0x7fd1d00caa60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:50:25.624 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:25.622+0000 7fd1d73ad640 1 --2- 172.21.15.123:0/3728247216 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd1d015e820 0x7fd1d00caa60 secure :-1 s=READY pgs=1550 cs=0 l=1 rev1=1 crypto rx=0x7fd1c801d5b0 tx=0x7fd1c8005ae0 comp rx=0 tx=0).stop 2024-09-16T15:50:25.624 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:25.622+0000 7fd1d73ad640 1 -- 172.21.15.123:0/3728247216 shutdown_connections 2024-09-16T15:50:25.624 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:25.622+0000 7fd1d73ad640 1 --2- 172.21.15.123:0/3728247216 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fd1b4073580 0x7fd1b4075a40 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:25.624 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:25.622+0000 7fd1d73ad640 1 --2- 172.21.15.123:0/3728247216 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd1d015e820 0x7fd1d00caa60 unknown :-1 s=CLOSED pgs=1550 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:25.624 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:25.622+0000 7fd1d73ad640 1 --2- 172.21.15.123:0/3728247216 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fd1d015e160 0x7fd1d01577b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:25.624 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:25.622+0000 7fd1d73ad640 1 --2- 172.21.15.123:0/3728247216 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd1d00cd6e0 0x7fd1d01f7c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:25.624 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:25.622+0000 7fd1d73ad640 1 -- 172.21.15.123:0/3728247216 >> 172.21.15.123:0/3728247216 conn(0x7fd1d00cfb10 msgr2=0x7fd1d0153100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:50:25.624 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:25.622+0000 7fd1d73ad640 1 -- 172.21.15.123:0/3728247216 shutdown_connections 2024-09-16T15:50:25.624 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:25.622+0000 7fd1d73ad640 1 -- 172.21.15.123:0/3728247216 wait complete. 2024-09-16T15:50:25.640 INFO:tasks.ceph.mds.a:Restarting daemon 2024-09-16T15:50:25.640 INFO:tasks.ceph.mds.a:Stopping old one... 2024-09-16T15:50:25.641 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-09-16T15:50:25.641 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T15:50:25.641 INFO:tasks.ceph.mds.c:Restarting daemon 2024-09-16T15:50:25.641 INFO:tasks.ceph.mds.c:Stopping old one... 2024-09-16T15:50:25.641 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-09-16T15:50:25.641 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T15:50:25.642 INFO:tasks.ceph.mds.b:Restarting daemon 2024-09-16T15:50:25.642 INFO:tasks.ceph.mds.b:Stopping old one... 2024-09-16T15:50:25.642 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-09-16T15:50:25.642 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T15:50:25.642 INFO:tasks.ceph.mds.d:Restarting daemon 2024-09-16T15:50:25.642 INFO:tasks.ceph.mds.d:Stopping old one... 2024-09-16T15:50:25.642 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-09-16T15:50:25.642 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T15:50:25.652 INFO:tasks.ceph.mds.a:Stopped 2024-09-16T15:50:25.652 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-09-16T15:50:25.653 INFO:tasks.ceph.mds.c:Stopped 2024-09-16T15:50:25.653 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-09-16T15:50:25.657 INFO:tasks.ceph.mds.d:Stopped 2024-09-16T15:50:25.657 DEBUG:teuthology.orchestra.run.smithi184:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-09-16T15:50:25.660 INFO:tasks.ceph.mds.b:Stopped 2024-09-16T15:50:25.660 DEBUG:teuthology.orchestra.run.smithi184:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-09-16T15:50:25.697 INFO:tasks.ceph.mds.a:Started 2024-09-16T15:50:25.697 DEBUG:teuthology.parallel:result is None 2024-09-16T15:50:25.698 INFO:tasks.ceph.mds.c:Started 2024-09-16T15:50:25.698 DEBUG:teuthology.parallel:result is None 2024-09-16T15:50:25.709 INFO:tasks.ceph.mds.d:Started 2024-09-16T15:50:25.709 DEBUG:teuthology.parallel:result is None 2024-09-16T15:50:25.710 INFO:tasks.ceph.mds.b:Started 2024-09-16T15:50:25.711 DEBUG:teuthology.parallel:result is None 2024-09-16T15:50:25.711 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-09-16T15:50:25.756 INFO:tasks.ceph.mds.c.smithi123.stdout:starting mds.c at 2024-09-16T15:50:25.758 INFO:tasks.ceph.mds.a.smithi123.stdout:starting mds.a at 2024-09-16T15:50:25.781 INFO:tasks.ceph.mds.b.smithi184.stdout:starting mds.b at 2024-09-16T15:50:25.785 INFO:tasks.ceph.mds.d.smithi184.stdout:starting mds.d at 2024-09-16T15:50:25.824 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:25.826+0000 7fd010294640 1 -- 172.21.15.123:0/437877737 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd008161620 msgr2=0x7fd0080d1da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:50:25.824 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:25.826+0000 7fd010294640 1 --2- 172.21.15.123:0/437877737 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd008161620 0x7fd0080d1da0 secure :-1 s=READY pgs=1561 cs=0 l=1 rev1=1 crypto rx=0x7fd004005fd0 tx=0x7fd00401c610 comp rx=0 tx=0).stop 2024-09-16T15:50:25.824 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:25.826+0000 7fd010294640 1 -- 172.21.15.123:0/437877737 shutdown_connections 2024-09-16T15:50:25.824 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:25.826+0000 7fd010294640 1 --2- 172.21.15.123:0/437877737 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd008161620 0x7fd0080d1da0 unknown :-1 s=CLOSED pgs=1561 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:25.825 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:25.826+0000 7fd010294640 1 -- 172.21.15.123:0/437877737 >> 172.21.15.123:0/437877737 conn(0x7fd0080d3ea0 msgr2=0x7fd0080d42b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:50:25.825 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:25.826+0000 7fd010294640 1 -- 172.21.15.123:0/437877737 shutdown_connections 2024-09-16T15:50:25.825 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:25.826+0000 7fd010294640 1 -- 172.21.15.123:0/437877737 wait complete. 2024-09-16T15:50:25.825 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:25.826+0000 7fd010294640 1 Processor -- start 2024-09-16T15:50:25.825 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:25.826+0000 7fd010294640 1 -- start start 2024-09-16T15:50:25.827 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:25.826+0000 7fd010294640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd0080b0a50 0x7fd00814f7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:50:25.827 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:25.826+0000 7fd010294640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd0080b2b00 0x7fd008158790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:50:25.827 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:25.826+0000 7fd010294640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fd008161620 0x7fd0080b47c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:50:25.827 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:25.826+0000 7fd010294640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fd00816c100 con 0x7fd0080b0a50 2024-09-16T15:50:25.827 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:25.826+0000 7fd010294640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fd00816bf80 con 0x7fd0080b2b00 2024-09-16T15:50:25.828 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:25.826+0000 7fd010294640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fd00816c280 con 0x7fd008161620 2024-09-16T15:50:25.828 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:25.826+0000 7fd00e030640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd0080b0a50 0x7fd00814f7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:50:25.828 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:25.826+0000 7fd00e030640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd0080b0a50 0x7fd00814f7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:55934/0 (socket says 172.21.15.123:55934) 2024-09-16T15:50:25.828 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:25.826+0000 7fd00e030640 1 -- 172.21.15.123:0/1557028376 learned_addr learned my addr 172.21.15.123:0/1557028376 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:50:25.828 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:25.826+0000 7fd00e030640 1 -- 172.21.15.123:0/1557028376 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fd008161620 msgr2=0x7fd0080b47c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:50:25.828 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:25.826+0000 7fd00e030640 1 --2- 172.21.15.123:0/1557028376 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fd008161620 0x7fd0080b47c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:25.828 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:25.826+0000 7fd00e030640 1 -- 172.21.15.123:0/1557028376 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd0080b2b00 msgr2=0x7fd008158790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:50:25.828 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:25.826+0000 7fd00e030640 1 --2- 172.21.15.123:0/1557028376 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd0080b2b00 0x7fd008158790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:25.828 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:25.826+0000 7fd00e030640 1 -- 172.21.15.123:0/1557028376 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd008159270 con 0x7fd0080b0a50 2024-09-16T15:50:25.828 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:25.826+0000 7fd00e030640 1 --2- 172.21.15.123:0/1557028376 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd0080b0a50 0x7fd00814f7a0 secure :-1 s=READY pgs=1562 cs=0 l=1 rev1=1 crypto rx=0x7fd004009790 tx=0x7fd004003e20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:50:25.829 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:25.826+0000 7fcffeffd640 1 -- 172.21.15.123:0/1557028376 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0040be070 con 0x7fd0080b0a50 2024-09-16T15:50:25.829 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:25.826+0000 7fcffeffd640 1 -- 172.21.15.123:0/1557028376 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fd0040b9480 con 0x7fd0080b0a50 2024-09-16T15:50:25.829 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:25.826+0000 7fd010294640 1 -- 172.21.15.123:0/1557028376 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd00815a8e0 con 0x7fd0080b0a50 2024-09-16T15:50:25.829 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:25.826+0000 7fcffeffd640 1 -- 172.21.15.123:0/1557028376 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0040b98b0 con 0x7fd0080b0a50 2024-09-16T15:50:25.829 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:25.826+0000 7fd010294640 1 -- 172.21.15.123:0/1557028376 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd00814fce0 con 0x7fd0080b0a50 2024-09-16T15:50:25.830 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:25.826+0000 7fd010294640 1 -- 172.21.15.123:0/1557028376 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd0080b0ea0 con 0x7fd0080b0a50 2024-09-16T15:50:25.830 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:25.830+0000 7fcffeffd640 1 -- 172.21.15.123:0/1557028376 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 22) ==== 97692+0+0 (secure 0 0 0) 0x7fd0040c1070 con 0x7fd0080b0a50 2024-09-16T15:50:25.830 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:25.830+0000 7fcffeffd640 1 --2- 172.21.15.123:0/1557028376 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fcfec073580 0x7fcfec075a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:50:25.830 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:25.830+0000 7fcffeffd640 1 -- 172.21.15.123:0/1557028376 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(138..138 src has 1..138) ==== 4971+0+0 (secure 0 0 0) 0x7fd00412ef30 con 0x7fd0080b0a50 2024-09-16T15:50:25.830 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:25.830+0000 7fd00d82f640 1 --2- 172.21.15.123:0/1557028376 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fcfec073580 0x7fcfec075a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:50:25.830 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:25.830+0000 7fd00d82f640 1 --2- 172.21.15.123:0/1557028376 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fcfec073580 0x7fcfec075a40 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7fcff8007fd0 tx=0x7fcff80078c0 comp rx=0 tx=0).ready entity=mgr.4150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:50:25.833 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:25.834+0000 7fcffeffd640 1 -- 172.21.15.123:0/1557028376 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7fd0040ac020 con 0x7fd0080b0a50 2024-09-16T15:50:26.002 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:26.002+0000 7fd010294640 1 -- 172.21.15.123:0/1557028376 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) -- 0x7fd0080d26c0 con 0x7fd0080b0a50 2024-09-16T15:50:26.353 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:26.354+0000 7fcffeffd640 1 -- 172.21.15.123:0/1557028376 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v140) ==== 97+0+0 (secure 0 0 0) 0x7fd004016410 con 0x7fd0080b0a50 2024-09-16T15:50:26.353 INFO:teuthology.orchestra.run.smithi123.stderr: removed all blocklist entries 2024-09-16T15:50:26.357 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:26.358+0000 7fd010294640 1 -- 172.21.15.123:0/1557028376 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fcfec073580 msgr2=0x7fcfec075a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:50:26.357 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:26.358+0000 7fd010294640 1 --2- 172.21.15.123:0/1557028376 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fcfec073580 0x7fcfec075a40 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7fcff8007fd0 tx=0x7fcff80078c0 comp rx=0 tx=0).stop 2024-09-16T15:50:26.357 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:26.358+0000 7fd010294640 1 -- 172.21.15.123:0/1557028376 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd0080b0a50 msgr2=0x7fd00814f7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:50:26.357 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:26.358+0000 7fd010294640 1 --2- 172.21.15.123:0/1557028376 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd0080b0a50 0x7fd00814f7a0 secure :-1 s=READY pgs=1562 cs=0 l=1 rev1=1 crypto rx=0x7fd004009790 tx=0x7fd004003e20 comp rx=0 tx=0).stop 2024-09-16T15:50:26.357 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:26.358+0000 7fd010294640 1 -- 172.21.15.123:0/1557028376 shutdown_connections 2024-09-16T15:50:26.357 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:26.358+0000 7fd010294640 1 --2- 172.21.15.123:0/1557028376 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fcfec073580 0x7fcfec075a40 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:26.357 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:26.358+0000 7fd010294640 1 --2- 172.21.15.123:0/1557028376 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fd008161620 0x7fd0080b47c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:26.357 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:26.358+0000 7fd010294640 1 --2- 172.21.15.123:0/1557028376 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd0080b2b00 0x7fd008158790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:26.357 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:26.358+0000 7fd010294640 1 --2- 172.21.15.123:0/1557028376 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd0080b0a50 0x7fd00814f7a0 unknown :-1 s=CLOSED pgs=1562 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:26.358 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:26.358+0000 7fd010294640 1 -- 172.21.15.123:0/1557028376 >> 172.21.15.123:0/1557028376 conn(0x7fd0080d3ea0 msgr2=0x7fd0080b5920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:50:26.358 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:26.358+0000 7fd010294640 1 -- 172.21.15.123:0/1557028376 shutdown_connections 2024-09-16T15:50:26.358 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:26.358+0000 7fd010294640 1 -- 172.21.15.123:0/1557028376 wait complete. 2024-09-16T15:50:26.379 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-09-16T15:50:26.545 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:26.546+0000 7f333d5ee640 1 -- 172.21.15.123:0/3132229429 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f333815ec40 msgr2=0x7f333806f420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:50:26.545 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:26.546+0000 7f333d5ee640 1 --2- 172.21.15.123:0/3132229429 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f333815ec40 0x7f333806f420 secure :-1 s=READY pgs=1563 cs=0 l=1 rev1=1 crypto rx=0x7f3330009c80 tx=0x7f333001cc80 comp rx=0 tx=0).stop 2024-09-16T15:50:26.545 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:26.546+0000 7f333d5ee640 1 -- 172.21.15.123:0/3132229429 shutdown_connections 2024-09-16T15:50:26.545 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:26.546+0000 7f333d5ee640 1 --2- 172.21.15.123:0/3132229429 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f333815ec40 0x7f333806f420 unknown :-1 s=CLOSED pgs=1563 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:26.546 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:26.546+0000 7f333d5ee640 1 --2- 172.21.15.123:0/3132229429 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f33380cf5a0 0x7f333806eee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:26.546 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:26.546+0000 7f333d5ee640 1 -- 172.21.15.123:0/3132229429 >> 172.21.15.123:0/3132229429 conn(0x7f33380cfb10 msgr2=0x7f33380cff20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:50:26.546 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:26.546+0000 7f333d5ee640 1 -- 172.21.15.123:0/3132229429 shutdown_connections 2024-09-16T15:50:26.546 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:26.546+0000 7f333d5ee640 1 -- 172.21.15.123:0/3132229429 wait complete. 2024-09-16T15:50:26.546 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:26.546+0000 7f333d5ee640 1 Processor -- start 2024-09-16T15:50:26.546 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:26.546+0000 7f333d5ee640 1 -- start start 2024-09-16T15:50:26.548 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:26.546+0000 7f333d5ee640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f33380cf5a0 0x7f33381597f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:50:26.549 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:26.546+0000 7f333d5ee640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3338158120 0x7f333814e660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:50:26.549 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:26.546+0000 7f333d5ee640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f333815ec40 0x7f3338158660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:50:26.549 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:26.546+0000 7f333d5ee640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f333815b300 con 0x7f333815ec40 2024-09-16T15:50:26.549 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:26.546+0000 7f333d5ee640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f333815b180 con 0x7f3338158120 2024-09-16T15:50:26.549 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:26.546+0000 7f333d5ee640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f333815b480 con 0x7f33380cf5a0 2024-09-16T15:50:26.549 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:26.546+0000 7f33377fe640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f333815ec40 0x7f3338158660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:50:26.549 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:26.546+0000 7f33377fe640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f333815ec40 0x7f3338158660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:55960/0 (socket says 172.21.15.123:55960) 2024-09-16T15:50:26.549 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:26.546+0000 7f33377fe640 1 -- 172.21.15.123:0/214867718 learned_addr learned my addr 172.21.15.123:0/214867718 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:50:26.549 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:26.546+0000 7f3336ffd640 1 --2- 172.21.15.123:0/214867718 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f33380cf5a0 0x7f33381597f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:50:26.549 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:26.546+0000 7f33367fc640 1 --2- 172.21.15.123:0/214867718 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3338158120 0x7f333814e660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:50:26.549 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:26.546+0000 7f33377fe640 1 -- 172.21.15.123:0/214867718 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f33380cf5a0 msgr2=0x7f33381597f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:50:26.550 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:26.546+0000 7f33377fe640 1 --2- 172.21.15.123:0/214867718 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f33380cf5a0 0x7f33381597f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:26.550 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:26.546+0000 7f33377fe640 1 -- 172.21.15.123:0/214867718 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3338158120 msgr2=0x7f333814e660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:50:26.550 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:26.546+0000 7f33377fe640 1 --2- 172.21.15.123:0/214867718 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3338158120 0x7f333814e660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:26.550 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:26.546+0000 7f33377fe640 1 -- 172.21.15.123:0/214867718 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f33380c5840 con 0x7f333815ec40 2024-09-16T15:50:26.550 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:26.546+0000 7f33367fc640 1 --2- 172.21.15.123:0/214867718 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3338158120 0x7f333814e660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T15:50:26.550 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:26.546+0000 7f33377fe640 1 --2- 172.21.15.123:0/214867718 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f333815ec40 0x7f3338158660 secure :-1 s=READY pgs=1564 cs=0 l=1 rev1=1 crypto rx=0x7f333001d520 tx=0x7f33300093d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:50:26.550 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:26.546+0000 7f331f7fe640 1 -- 172.21.15.123:0/214867718 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33300b2040 con 0x7f333815ec40 2024-09-16T15:50:26.550 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:26.546+0000 7f331f7fe640 1 -- 172.21.15.123:0/214867718 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f3330004200 con 0x7f333815ec40 2024-09-16T15:50:26.550 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:26.546+0000 7f333d5ee640 1 -- 172.21.15.123:0/214867718 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f33380c5ac0 con 0x7f333815ec40 2024-09-16T15:50:26.550 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:26.550+0000 7f331f7fe640 1 -- 172.21.15.123:0/214867718 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3330004630 con 0x7f333815ec40 2024-09-16T15:50:26.550 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:26.550+0000 7f333d5ee640 1 -- 172.21.15.123:0/214867718 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f33380c6080 con 0x7f333815ec40 2024-09-16T15:50:26.550 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:26.550+0000 7f333d5ee640 1 -- 172.21.15.123:0/214867718 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f33380ca320 con 0x7f333815ec40 2024-09-16T15:50:26.551 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:26.550+0000 7f331f7fe640 1 -- 172.21.15.123:0/214867718 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 22) ==== 97692+0+0 (secure 0 0 0) 0x7f33300a5ac0 con 0x7f333815ec40 2024-09-16T15:50:26.551 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:26.550+0000 7f331f7fe640 1 --2- 172.21.15.123:0/214867718 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f3324073580 0x7f3324075a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:50:26.551 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:26.550+0000 7f3336ffd640 1 --2- 172.21.15.123:0/214867718 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f3324073580 0x7f3324075a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:50:26.551 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:26.550+0000 7f331f7fe640 1 -- 172.21.15.123:0/214867718 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(142..142 src has 1..142) ==== 4885+0+0 (secure 0 0 0) 0x7f333012ef10 con 0x7f333815ec40 2024-09-16T15:50:26.552 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:26.550+0000 7f3336ffd640 1 --2- 172.21.15.123:0/214867718 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f3324073580 0x7f3324075a40 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f3328004e30 tx=0x7f3328009040 comp rx=0 tx=0).ready entity=mgr.4150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:50:26.554 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:26.554+0000 7f331f7fe640 1 -- 172.21.15.123:0/214867718 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f33380c5840 con 0x7f333815ec40 2024-09-16T15:50:26.783 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:26.782+0000 7f333d5ee640 1 -- 172.21.15.123:0/214867718 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) -- 0x7f3338157750 con 0x7f333815ec40 2024-09-16T15:50:26.784 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:26.782+0000 7f331f7fe640 1 -- 172.21.15.123:0/214867718 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v26) ==== 80+0+5304 (secure 0 0 0) 0x7f3330016990 con 0x7f333815ec40 2024-09-16T15:50:26.784 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-09-16T15:50:26.784 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2024-09-16T15:50:26.784 INFO:teuthology.orchestra.run.smithi123.stdout: "auth_dump": [ 2024-09-16T15:50:26.784 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-09-16T15:50:26.784 INFO:teuthology.orchestra.run.smithi123.stdout: "entity": "mds.a", 2024-09-16T15:50:26.784 INFO:teuthology.orchestra.run.smithi123.stdout: "key": "AQC0UehmbZpOJBAAaYGyDcWrt4k0FS5P/bb+4A==", 2024-09-16T15:50:26.785 INFO:teuthology.orchestra.run.smithi123.stdout: "caps": { 2024-09-16T15:50:26.785 INFO:teuthology.orchestra.run.smithi123.stdout: "mds": "allow", 2024-09-16T15:50:26.785 INFO:teuthology.orchestra.run.smithi123.stdout: "mgr": "allow *", 2024-09-16T15:50:26.785 INFO:teuthology.orchestra.run.smithi123.stdout: "mon": "allow *", 2024-09-16T15:50:26.785 INFO:teuthology.orchestra.run.smithi123.stdout: "osd": "allow *" 2024-09-16T15:50:26.785 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-16T15:50:26.785 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:50:26.785 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-09-16T15:50:26.785 INFO:teuthology.orchestra.run.smithi123.stdout: "entity": "mds.b", 2024-09-16T15:50:26.785 INFO:teuthology.orchestra.run.smithi123.stdout: "key": "AQC0UehmF1HFMhAADcvqXNKdHZMPAbLc+se4Aw==", 2024-09-16T15:50:26.785 INFO:teuthology.orchestra.run.smithi123.stdout: "caps": { 2024-09-16T15:50:26.786 INFO:teuthology.orchestra.run.smithi123.stdout: "mds": "allow", 2024-09-16T15:50:26.786 INFO:teuthology.orchestra.run.smithi123.stdout: "mgr": "allow *", 2024-09-16T15:50:26.786 INFO:teuthology.orchestra.run.smithi123.stdout: "mon": "allow *", 2024-09-16T15:50:26.786 INFO:teuthology.orchestra.run.smithi123.stdout: "osd": "allow *" 2024-09-16T15:50:26.786 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-16T15:50:26.786 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:50:26.786 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-09-16T15:50:26.786 INFO:teuthology.orchestra.run.smithi123.stdout: "entity": "mds.c", 2024-09-16T15:50:26.786 INFO:teuthology.orchestra.run.smithi123.stdout: "key": "AQC0UehmCBjCLBAAXd85J25r7cgrXeDCG7pahg==", 2024-09-16T15:50:26.786 INFO:teuthology.orchestra.run.smithi123.stdout: "caps": { 2024-09-16T15:50:26.786 INFO:teuthology.orchestra.run.smithi123.stdout: "mds": "allow", 2024-09-16T15:50:26.786 INFO:teuthology.orchestra.run.smithi123.stdout: "mgr": "allow *", 2024-09-16T15:50:26.786 INFO:teuthology.orchestra.run.smithi123.stdout: "mon": "allow *", 2024-09-16T15:50:26.787 INFO:teuthology.orchestra.run.smithi123.stdout: "osd": "allow *" 2024-09-16T15:50:26.787 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-16T15:50:26.787 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:50:26.787 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-09-16T15:50:26.787 INFO:teuthology.orchestra.run.smithi123.stdout: "entity": "mds.d", 2024-09-16T15:50:26.787 INFO:teuthology.orchestra.run.smithi123.stdout: "key": "AQC0UehmMh3ZOBAAx0RCa3R9RMT93/icZhPGyg==", 2024-09-16T15:50:26.787 INFO:teuthology.orchestra.run.smithi123.stdout: "caps": { 2024-09-16T15:50:26.787 INFO:teuthology.orchestra.run.smithi123.stdout: "mds": "allow", 2024-09-16T15:50:26.787 INFO:teuthology.orchestra.run.smithi123.stdout: "mgr": "allow *", 2024-09-16T15:50:26.787 INFO:teuthology.orchestra.run.smithi123.stdout: "mon": "allow *", 2024-09-16T15:50:26.787 INFO:teuthology.orchestra.run.smithi123.stdout: "osd": "allow *" 2024-09-16T15:50:26.787 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-16T15:50:26.788 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:50:26.788 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-09-16T15:50:26.788 INFO:teuthology.orchestra.run.smithi123.stdout: "entity": "osd.0", 2024-09-16T15:50:26.788 INFO:teuthology.orchestra.run.smithi123.stdout: "key": "AQC4UehmT633CRAAIhpkeIBOzwJO0kiXLu0x9Q==", 2024-09-16T15:50:26.788 INFO:teuthology.orchestra.run.smithi123.stdout: "caps": { 2024-09-16T15:50:26.788 INFO:teuthology.orchestra.run.smithi123.stdout: "mgr": "allow profile osd", 2024-09-16T15:50:26.788 INFO:teuthology.orchestra.run.smithi123.stdout: "mon": "allow profile osd", 2024-09-16T15:50:26.788 INFO:teuthology.orchestra.run.smithi123.stdout: "osd": "allow *" 2024-09-16T15:50:26.788 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-16T15:50:26.788 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:50:26.788 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-09-16T15:50:26.788 INFO:teuthology.orchestra.run.smithi123.stdout: "entity": "osd.1", 2024-09-16T15:50:26.789 INFO:teuthology.orchestra.run.smithi123.stdout: "key": "AQC7UehmFwF7GBAAvOUG+rWjANKT8wm/ai4xGg==", 2024-09-16T15:50:26.789 INFO:teuthology.orchestra.run.smithi123.stdout: "caps": { 2024-09-16T15:50:26.789 INFO:teuthology.orchestra.run.smithi123.stdout: "mgr": "allow profile osd", 2024-09-16T15:50:26.789 INFO:teuthology.orchestra.run.smithi123.stdout: "mon": "allow profile osd", 2024-09-16T15:50:26.789 INFO:teuthology.orchestra.run.smithi123.stdout: "osd": "allow *" 2024-09-16T15:50:26.789 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-16T15:50:26.789 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:50:26.789 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-09-16T15:50:26.789 INFO:teuthology.orchestra.run.smithi123.stdout: "entity": "osd.2", 2024-09-16T15:50:26.789 INFO:teuthology.orchestra.run.smithi123.stdout: "key": "AQC+UehmlCwWKxAAl3iO7ZALWyaorQH2urHKCw==", 2024-09-16T15:50:26.789 INFO:teuthology.orchestra.run.smithi123.stdout: "caps": { 2024-09-16T15:50:26.789 INFO:teuthology.orchestra.run.smithi123.stdout: "mgr": "allow profile osd", 2024-09-16T15:50:26.789 INFO:teuthology.orchestra.run.smithi123.stdout: "mon": "allow profile osd", 2024-09-16T15:50:26.790 INFO:teuthology.orchestra.run.smithi123.stdout: "osd": "allow *" 2024-09-16T15:50:26.790 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-16T15:50:26.790 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:50:26.790 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-09-16T15:50:26.790 INFO:teuthology.orchestra.run.smithi123.stdout: "entity": "osd.3", 2024-09-16T15:50:26.790 INFO:teuthology.orchestra.run.smithi123.stdout: "key": "AQDCUehmD8VgAhAAneoIeGr3MflSmi+nyavj2A==", 2024-09-16T15:50:26.790 INFO:teuthology.orchestra.run.smithi123.stdout: "caps": { 2024-09-16T15:50:26.790 INFO:teuthology.orchestra.run.smithi123.stdout: "mgr": "allow profile osd", 2024-09-16T15:50:26.790 INFO:teuthology.orchestra.run.smithi123.stdout: "mon": "allow profile osd", 2024-09-16T15:50:26.790 INFO:teuthology.orchestra.run.smithi123.stdout: "osd": "allow *" 2024-09-16T15:50:26.790 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-16T15:50:26.791 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:50:26.791 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-09-16T15:50:26.791 INFO:teuthology.orchestra.run.smithi123.stdout: "entity": "osd.4", 2024-09-16T15:50:26.791 INFO:teuthology.orchestra.run.smithi123.stdout: "key": "AQDIUehmA9B9EBAAsu0X5LPx1TmatZeHzjnVig==", 2024-09-16T15:50:26.791 INFO:teuthology.orchestra.run.smithi123.stdout: "caps": { 2024-09-16T15:50:26.791 INFO:teuthology.orchestra.run.smithi123.stdout: "mgr": "allow profile osd", 2024-09-16T15:50:26.791 INFO:teuthology.orchestra.run.smithi123.stdout: "mon": "allow profile osd", 2024-09-16T15:50:26.791 INFO:teuthology.orchestra.run.smithi123.stdout: "osd": "allow *" 2024-09-16T15:50:26.791 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-16T15:50:26.791 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:50:26.791 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-09-16T15:50:26.792 INFO:teuthology.orchestra.run.smithi123.stdout: "entity": "osd.5", 2024-09-16T15:50:26.792 INFO:teuthology.orchestra.run.smithi123.stdout: "key": "AQDLUehmcugdHxAAQMSS1ZH71FUzCx5A+yNnZg==", 2024-09-16T15:50:26.792 INFO:teuthology.orchestra.run.smithi123.stdout: "caps": { 2024-09-16T15:50:26.792 INFO:teuthology.orchestra.run.smithi123.stdout: "mgr": "allow profile osd", 2024-09-16T15:50:26.792 INFO:teuthology.orchestra.run.smithi123.stdout: "mon": "allow profile osd", 2024-09-16T15:50:26.792 INFO:teuthology.orchestra.run.smithi123.stdout: "osd": "allow *" 2024-09-16T15:50:26.792 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-16T15:50:26.792 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:50:26.792 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-09-16T15:50:26.792 INFO:teuthology.orchestra.run.smithi123.stdout: "entity": "osd.6", 2024-09-16T15:50:26.793 INFO:teuthology.orchestra.run.smithi123.stdout: "key": "AQDOUehmWLwYLhAAhOlQLepcWElsVPMnHlZm4w==", 2024-09-16T15:50:26.793 INFO:teuthology.orchestra.run.smithi123.stdout: "caps": { 2024-09-16T15:50:26.793 INFO:teuthology.orchestra.run.smithi123.stdout: "mgr": "allow profile osd", 2024-09-16T15:50:26.793 INFO:teuthology.orchestra.run.smithi123.stdout: "mon": "allow profile osd", 2024-09-16T15:50:26.793 INFO:teuthology.orchestra.run.smithi123.stdout: "osd": "allow *" 2024-09-16T15:50:26.793 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-16T15:50:26.793 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:50:26.793 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-09-16T15:50:26.793 INFO:teuthology.orchestra.run.smithi123.stdout: "entity": "osd.7", 2024-09-16T15:50:26.793 INFO:teuthology.orchestra.run.smithi123.stdout: "key": "AQDSUehmipkbABAAQuL/2O7w0tsM1OKeOil97w==", 2024-09-16T15:50:26.793 INFO:teuthology.orchestra.run.smithi123.stdout: "caps": { 2024-09-16T15:50:26.793 INFO:teuthology.orchestra.run.smithi123.stdout: "mgr": "allow profile osd", 2024-09-16T15:50:26.793 INFO:teuthology.orchestra.run.smithi123.stdout: "mon": "allow profile osd", 2024-09-16T15:50:26.794 INFO:teuthology.orchestra.run.smithi123.stdout: "osd": "allow *" 2024-09-16T15:50:26.794 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-16T15:50:26.794 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:50:26.794 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-09-16T15:50:26.794 INFO:teuthology.orchestra.run.smithi123.stdout: "entity": "client.0", 2024-09-16T15:50:26.794 INFO:teuthology.orchestra.run.smithi123.stdout: "key": "AQC1Uehm7GDTAhAAHWWVIyjVVnTeeozBR8DfFA==", 2024-09-16T15:50:26.794 INFO:teuthology.orchestra.run.smithi123.stdout: "caps": { 2024-09-16T15:50:26.794 INFO:teuthology.orchestra.run.smithi123.stdout: "mds": "allow", 2024-09-16T15:50:26.794 INFO:teuthology.orchestra.run.smithi123.stdout: "mon": "allow r", 2024-09-16T15:50:26.794 INFO:teuthology.orchestra.run.smithi123.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-09-16T15:50:26.794 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-16T15:50:26.794 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:50:26.794 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-09-16T15:50:26.795 INFO:teuthology.orchestra.run.smithi123.stdout: "entity": "client.1", 2024-09-16T15:50:26.795 INFO:teuthology.orchestra.run.smithi123.stdout: "key": "AQC1UehmFbaWChAAFtRmKucm+pbKtCNvRCTNqA==", 2024-09-16T15:50:26.795 INFO:teuthology.orchestra.run.smithi123.stdout: "caps": { 2024-09-16T15:50:26.795 INFO:teuthology.orchestra.run.smithi123.stdout: "mds": "allow", 2024-09-16T15:50:26.795 INFO:teuthology.orchestra.run.smithi123.stdout: "mgr": "allow r", 2024-09-16T15:50:26.795 INFO:teuthology.orchestra.run.smithi123.stdout: "mon": "allow rw", 2024-09-16T15:50:26.795 INFO:teuthology.orchestra.run.smithi123.stdout: "osd": "allow rwx" 2024-09-16T15:50:26.795 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-16T15:50:26.795 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:50:26.795 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-09-16T15:50:26.795 INFO:teuthology.orchestra.run.smithi123.stdout: "entity": "client.2", 2024-09-16T15:50:26.795 INFO:teuthology.orchestra.run.smithi123.stdout: "key": "AQC1Uehm1ADrBxAAW/8ayNHI6Wq/YlI4ivFZiw==", 2024-09-16T15:50:26.796 INFO:teuthology.orchestra.run.smithi123.stdout: "caps": { 2024-09-16T15:50:26.796 INFO:teuthology.orchestra.run.smithi123.stdout: "mds": "allow", 2024-09-16T15:50:26.796 INFO:teuthology.orchestra.run.smithi123.stdout: "mgr": "allow r", 2024-09-16T15:50:26.796 INFO:teuthology.orchestra.run.smithi123.stdout: "mon": "allow rw", 2024-09-16T15:50:26.796 INFO:teuthology.orchestra.run.smithi123.stdout: "osd": "allow rwx" 2024-09-16T15:50:26.796 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-16T15:50:26.796 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:50:26.796 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-09-16T15:50:26.796 INFO:teuthology.orchestra.run.smithi123.stdout: "entity": "client.3", 2024-09-16T15:50:26.796 INFO:teuthology.orchestra.run.smithi123.stdout: "key": "AQC1UehmLv/3DxAAuGloBxSuza95uajpfiZ9MQ==", 2024-09-16T15:50:26.796 INFO:teuthology.orchestra.run.smithi123.stdout: "caps": { 2024-09-16T15:50:26.796 INFO:teuthology.orchestra.run.smithi123.stdout: "mds": "allow", 2024-09-16T15:50:26.796 INFO:teuthology.orchestra.run.smithi123.stdout: "mgr": "allow r", 2024-09-16T15:50:26.797 INFO:teuthology.orchestra.run.smithi123.stdout: "mon": "allow rw", 2024-09-16T15:50:26.797 INFO:teuthology.orchestra.run.smithi123.stdout: "osd": "allow rwx" 2024-09-16T15:50:26.797 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-16T15:50:26.797 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:50:26.797 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-09-16T15:50:26.797 INFO:teuthology.orchestra.run.smithi123.stdout: "entity": "client.admin", 2024-09-16T15:50:26.797 INFO:teuthology.orchestra.run.smithi123.stdout: "key": "AQC0Uehm1ZDRDBAAhyQ1BmJg1Mw6wQ04I2CbFA==", 2024-09-16T15:50:26.797 INFO:teuthology.orchestra.run.smithi123.stdout: "caps": { 2024-09-16T15:50:26.797 INFO:teuthology.orchestra.run.smithi123.stdout: "mds": "allow *", 2024-09-16T15:50:26.797 INFO:teuthology.orchestra.run.smithi123.stdout: "mgr": "allow *", 2024-09-16T15:50:26.797 INFO:teuthology.orchestra.run.smithi123.stdout: "mon": "allow *", 2024-09-16T15:50:26.797 INFO:teuthology.orchestra.run.smithi123.stdout: "osd": "allow *" 2024-09-16T15:50:26.798 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-16T15:50:26.798 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:50:26.798 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-09-16T15:50:26.798 INFO:teuthology.orchestra.run.smithi123.stdout: "entity": "mgr.x", 2024-09-16T15:50:26.798 INFO:teuthology.orchestra.run.smithi123.stdout: "key": "AQC0UehmfPilIBAAERlBbduOgAU9ba0n/1tl3g==", 2024-09-16T15:50:26.798 INFO:teuthology.orchestra.run.smithi123.stdout: "caps": { 2024-09-16T15:50:26.798 INFO:teuthology.orchestra.run.smithi123.stdout: "mds": "allow *", 2024-09-16T15:50:26.798 INFO:teuthology.orchestra.run.smithi123.stdout: "mon": "allow profile mgr", 2024-09-16T15:50:26.798 INFO:teuthology.orchestra.run.smithi123.stdout: "osd": "allow *" 2024-09-16T15:50:26.798 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-16T15:50:26.798 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:50:26.798 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-09-16T15:50:26.799 INFO:teuthology.orchestra.run.smithi123.stdout: "entity": "mgr.y", 2024-09-16T15:50:26.799 INFO:teuthology.orchestra.run.smithi123.stdout: "key": "AQC0UehmkjrlHRAA+3xbQC+SPWGWW1gND7xcAg==", 2024-09-16T15:50:26.799 INFO:teuthology.orchestra.run.smithi123.stdout: "caps": { 2024-09-16T15:50:26.799 INFO:teuthology.orchestra.run.smithi123.stdout: "mds": "allow *", 2024-09-16T15:50:26.799 INFO:teuthology.orchestra.run.smithi123.stdout: "mon": "allow profile mgr", 2024-09-16T15:50:26.799 INFO:teuthology.orchestra.run.smithi123.stdout: "osd": "allow *" 2024-09-16T15:50:26.799 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-16T15:50:26.799 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-16T15:50:26.799 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-09-16T15:50:26.799 INFO:teuthology.orchestra.run.smithi123.stdout:} 2024-09-16T15:50:26.800 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:26.786+0000 7f333d5ee640 1 -- 172.21.15.123:0/214867718 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f3324073580 msgr2=0x7f3324075a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:50:26.800 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:26.786+0000 7f333d5ee640 1 --2- 172.21.15.123:0/214867718 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f3324073580 0x7f3324075a40 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f3328004e30 tx=0x7f3328009040 comp rx=0 tx=0).stop 2024-09-16T15:50:26.800 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:26.786+0000 7f333d5ee640 1 -- 172.21.15.123:0/214867718 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f333815ec40 msgr2=0x7f3338158660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:50:26.800 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:26.786+0000 7f333d5ee640 1 --2- 172.21.15.123:0/214867718 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f333815ec40 0x7f3338158660 secure :-1 s=READY pgs=1564 cs=0 l=1 rev1=1 crypto rx=0x7f333001d520 tx=0x7f33300093d0 comp rx=0 tx=0).stop 2024-09-16T15:50:26.800 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:26.786+0000 7f333d5ee640 1 -- 172.21.15.123:0/214867718 shutdown_connections 2024-09-16T15:50:26.801 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:26.786+0000 7f333d5ee640 1 --2- 172.21.15.123:0/214867718 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f3324073580 0x7f3324075a40 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:26.801 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:26.786+0000 7f333d5ee640 1 --2- 172.21.15.123:0/214867718 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f333815ec40 0x7f3338158660 unknown :-1 s=CLOSED pgs=1564 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:26.801 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:26.786+0000 7f333d5ee640 1 --2- 172.21.15.123:0/214867718 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3338158120 0x7f333814e660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:26.801 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:26.786+0000 7f333d5ee640 1 --2- 172.21.15.123:0/214867718 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f33380cf5a0 0x7f33381597f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:26.801 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:26.786+0000 7f333d5ee640 1 -- 172.21.15.123:0/214867718 >> 172.21.15.123:0/214867718 conn(0x7f33380cfb10 msgr2=0x7f3338152dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:50:26.801 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:26.786+0000 7f333d5ee640 1 -- 172.21.15.123:0/214867718 shutdown_connections 2024-09-16T15:50:26.801 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:26.786+0000 7f333d5ee640 1 -- 172.21.15.123:0/214867718 wait complete. 2024-09-16T15:50:26.802 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-09-16T15:50:26.913 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:26.914+0000 7f0929c5e640 1 -- 172.21.15.123:0/3065136510 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0910004810 msgr2=0x7f0910004c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:50:26.914 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:26.914+0000 7f0929c5e640 1 --2- 172.21.15.123:0/3065136510 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0910004810 0x7f0910004c90 secure :-1 s=READY pgs=1566 cs=0 l=1 rev1=1 crypto rx=0x7f0914001690 tx=0x7f091401c710 comp rx=0 tx=0).stop 2024-09-16T15:50:26.914 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:26.914+0000 7f0923fff640 1 -- 172.21.15.123:0/3065136510 reap_dead start 2024-09-16T15:50:26.914 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:26.914+0000 7f0929c5e640 1 -- 172.21.15.123:0/3065136510 shutdown_connections 2024-09-16T15:50:26.914 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:26.914+0000 7f0929c5e640 1 -- 172.21.15.123:0/3065136510 >> 172.21.15.123:0/3065136510 conn(0x7f09240cfb10 msgr2=0x7f09240cff20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:50:26.914 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:26.914+0000 7f0929c5e640 1 -- 172.21.15.123:0/3065136510 shutdown_connections 2024-09-16T15:50:26.914 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:26.914+0000 7f0929c5e640 1 -- 172.21.15.123:0/3065136510 wait complete. 2024-09-16T15:50:26.914 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:26.914+0000 7f0929c5e640 1 Processor -- start 2024-09-16T15:50:26.915 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:26.914+0000 7f0929c5e640 1 -- start start 2024-09-16T15:50:26.917 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:26.914+0000 7f0929c5e640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f0910004810 0x7f0924147eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:50:26.917 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:26.914+0000 7f0929c5e640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f092415a630 0x7f09240c49d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:50:26.917 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:26.914+0000 7f0929c5e640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0924165fa0 0x7f0924166450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:50:26.917 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:26.914+0000 7f0929c5e640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f09240cf5b0 con 0x7f0924165fa0 2024-09-16T15:50:26.917 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:26.914+0000 7f0929c5e640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f09240cf430 con 0x7f092415a630 2024-09-16T15:50:26.917 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:26.914+0000 7f0929c5e640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f09241673d0 con 0x7f0910004810 2024-09-16T15:50:26.917 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:26.914+0000 7f0923fff640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0924165fa0 0x7f0924166450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:50:26.917 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:26.914+0000 7f0923fff640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0924165fa0 0x7f0924166450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:55978/0 (socket says 172.21.15.123:55978) 2024-09-16T15:50:26.918 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:26.914+0000 7f0923fff640 1 -- 172.21.15.123:0/4125471635 learned_addr learned my addr 172.21.15.123:0/4125471635 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:50:26.918 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:26.914+0000 7f0922ffd640 1 --2- 172.21.15.123:0/4125471635 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f092415a630 0x7f09240c49d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:50:26.918 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:26.914+0000 7f09237fe640 1 --2- 172.21.15.123:0/4125471635 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f0910004810 0x7f0924147eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:50:26.918 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:26.918+0000 7f0923fff640 1 -- 172.21.15.123:0/4125471635 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f0910004810 msgr2=0x7f0924147eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:50:26.918 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:26.918+0000 7f0923fff640 1 --2- 172.21.15.123:0/4125471635 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f0910004810 0x7f0924147eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:26.918 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:26.918+0000 7f0923fff640 1 -- 172.21.15.123:0/4125471635 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f092415a630 msgr2=0x7f09240c49d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:50:26.918 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:26.918+0000 7f0923fff640 1 --2- 172.21.15.123:0/4125471635 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f092415a630 0x7f09240c49d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:26.918 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:26.918+0000 7f0923fff640 1 -- 172.21.15.123:0/4125471635 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0924073720 con 0x7f0924165fa0 2024-09-16T15:50:26.918 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:26.918+0000 7f0923fff640 1 --2- 172.21.15.123:0/4125471635 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0924165fa0 0x7f0924166450 secure :-1 s=READY pgs=1567 cs=0 l=1 rev1=1 crypto rx=0x7f092415ec00 tx=0x7f091c00e7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:50:26.918 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:26.918+0000 7f091bfff640 1 -- 172.21.15.123:0/4125471635 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f091c010890 con 0x7f0924165fa0 2024-09-16T15:50:26.918 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:26.918+0000 7f091bfff640 1 -- 172.21.15.123:0/4125471635 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f091c016070 con 0x7f0924165fa0 2024-09-16T15:50:26.918 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:26.918+0000 7f0929c5e640 1 -- 172.21.15.123:0/4125471635 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f09241f8070 con 0x7f0924165fa0 2024-09-16T15:50:26.919 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:26.918+0000 7f091bfff640 1 -- 172.21.15.123:0/4125471635 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f091c010dc0 con 0x7f0924165fa0 2024-09-16T15:50:26.919 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:26.918+0000 7f0929c5e640 1 -- 172.21.15.123:0/4125471635 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f09241f85f0 con 0x7f0924165fa0 2024-09-16T15:50:26.919 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:26.918+0000 7f0929c5e640 1 -- 172.21.15.123:0/4125471635 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f09240cd6e0 con 0x7f0924165fa0 2024-09-16T15:50:26.919 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:26.918+0000 7f091bfff640 1 -- 172.21.15.123:0/4125471635 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 22) ==== 97692+0+0 (secure 0 0 0) 0x7f091c01b460 con 0x7f0924165fa0 2024-09-16T15:50:26.920 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:26.918+0000 7f091bfff640 1 --2- 172.21.15.123:0/4125471635 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f09040735b0 0x7f0904075a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:50:26.920 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:26.918+0000 7f091bfff640 1 -- 172.21.15.123:0/4125471635 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(142..142 src has 1..142) ==== 4885+0+0 (secure 0 0 0) 0x7f091c094ac0 con 0x7f0924165fa0 2024-09-16T15:50:26.920 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:26.918+0000 7f09237fe640 1 --2- 172.21.15.123:0/4125471635 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f09040735b0 0x7f0904075a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:50:26.920 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:26.918+0000 7f09237fe640 1 --2- 172.21.15.123:0/4125471635 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f09040735b0 0x7f0904075a70 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f090c009cb0 tx=0x7f090c009690 comp rx=0 tx=0).ready entity=mgr.4150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:50:26.922 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:26.922+0000 7f091bfff640 1 -- 172.21.15.123:0/4125471635 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f091c05f6a0 con 0x7f0924165fa0 2024-09-16T15:50:27.082 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:27.082+0000 7f0929c5e640 1 -- 172.21.15.123:0/4125471635 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) -- 0x7f092415bfe0 con 0x7f0924165fa0 2024-09-16T15:50:27.082 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:27.082+0000 7f091bfff640 1 -- 172.21.15.123:0/4125471635 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v142) ==== 84+0+65 (secure 0 0 0) 0x7f091c05efa0 con 0x7f0924165fa0 2024-09-16T15:50:27.083 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-09-16T15:50:27.083 INFO:teuthology.orchestra.run.smithi123.stdout:[ 2024-09-16T15:50:27.083 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-09-16T15:50:27.083 INFO:teuthology.orchestra.run.smithi123.stdout: "poolnum": 1, 2024-09-16T15:50:27.083 INFO:teuthology.orchestra.run.smithi123.stdout: "poolname": ".mgr" 2024-09-16T15:50:27.083 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-16T15:50:27.083 INFO:teuthology.orchestra.run.smithi123.stdout:] 2024-09-16T15:50:27.085 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:27.086+0000 7f0929c5e640 1 -- 172.21.15.123:0/4125471635 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f09040735b0 msgr2=0x7f0904075a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:50:27.085 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:27.086+0000 7f0929c5e640 1 --2- 172.21.15.123:0/4125471635 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f09040735b0 0x7f0904075a70 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f090c009cb0 tx=0x7f090c009690 comp rx=0 tx=0).stop 2024-09-16T15:50:27.085 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:27.086+0000 7f0929c5e640 1 -- 172.21.15.123:0/4125471635 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0924165fa0 msgr2=0x7f0924166450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:50:27.086 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:27.086+0000 7f0929c5e640 1 --2- 172.21.15.123:0/4125471635 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0924165fa0 0x7f0924166450 secure :-1 s=READY pgs=1567 cs=0 l=1 rev1=1 crypto rx=0x7f092415ec00 tx=0x7f091c00e7f0 comp rx=0 tx=0).stop 2024-09-16T15:50:27.086 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:27.086+0000 7f0929c5e640 1 -- 172.21.15.123:0/4125471635 shutdown_connections 2024-09-16T15:50:27.086 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:27.086+0000 7f0929c5e640 1 --2- 172.21.15.123:0/4125471635 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f09040735b0 0x7f0904075a70 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:27.086 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:27.086+0000 7f0929c5e640 1 --2- 172.21.15.123:0/4125471635 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0924165fa0 0x7f0924166450 unknown :-1 s=CLOSED pgs=1567 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:27.086 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:27.086+0000 7f0929c5e640 1 --2- 172.21.15.123:0/4125471635 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f092415a630 0x7f09240c49d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:27.086 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:27.086+0000 7f0929c5e640 1 --2- 172.21.15.123:0/4125471635 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f0910004810 0x7f0924147eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:27.086 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:27.086+0000 7f0929c5e640 1 -- 172.21.15.123:0/4125471635 >> 172.21.15.123:0/4125471635 conn(0x7f09240cfb10 msgr2=0x7f0924158fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:50:27.086 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:27.086+0000 7f0929c5e640 1 -- 172.21.15.123:0/4125471635 shutdown_connections 2024-09-16T15:50:27.086 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:27.086+0000 7f0929c5e640 1 -- 172.21.15.123:0/4125471635 wait complete. 2024-09-16T15:50:27.100 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-09-16T15:50:27.101 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-09-16T15:50:27.226 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:27.226+0000 7fc429efd640 1 -- 172.21.15.123:0/50566740 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc40c004090 msgr2=0x7fc40c004810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:50:27.226 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:27.226+0000 7fc429efd640 1 --2- 172.21.15.123:0/50566740 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc40c004090 0x7fc40c004810 secure :-1 s=READY pgs=1569 cs=0 l=1 rev1=1 crypto rx=0x7fc418001ec0 tx=0x7fc41801f630 comp rx=0 tx=0).stop 2024-09-16T15:50:27.226 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:27.226+0000 7fc423fff640 1 -- 172.21.15.123:0/50566740 reap_dead start 2024-09-16T15:50:27.226 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:27.226+0000 7fc429efd640 1 -- 172.21.15.123:0/50566740 shutdown_connections 2024-09-16T15:50:27.226 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:27.226+0000 7fc429efd640 1 -- 172.21.15.123:0/50566740 >> 172.21.15.123:0/50566740 conn(0x7fc4240b1630 msgr2=0x7fc4240b1a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:50:27.227 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:27.226+0000 7fc429efd640 1 -- 172.21.15.123:0/50566740 shutdown_connections 2024-09-16T15:50:27.227 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:27.226+0000 7fc429efd640 1 -- 172.21.15.123:0/50566740 wait complete. 2024-09-16T15:50:27.227 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:27.226+0000 7fc429efd640 1 Processor -- start 2024-09-16T15:50:27.227 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:27.226+0000 7fc429efd640 1 -- start start 2024-09-16T15:50:27.229 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:27.230+0000 7fc429efd640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fc40c004090 0x7fc42414b250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:50:27.229 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:27.230+0000 7fc429efd640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc424143df0 0x7fc424151280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:50:27.229 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:27.230+0000 7fc429efd640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc4240ad8b0 0x7fc4240add30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:50:27.230 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:27.230+0000 7fc429efd640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fc4240ccfa0 con 0x7fc424143df0 2024-09-16T15:50:27.230 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:27.230+0000 7fc429efd640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fc4240cce20 con 0x7fc4240ad8b0 2024-09-16T15:50:27.230 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:27.230+0000 7fc429efd640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fc42415ec30 con 0x7fc40c004090 2024-09-16T15:50:27.230 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:27.230+0000 7fc422ffd640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc424143df0 0x7fc424151280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:50:27.230 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:27.230+0000 7fc422ffd640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc424143df0 0x7fc424151280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:56010/0 (socket says 172.21.15.123:56010) 2024-09-16T15:50:27.230 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:27.230+0000 7fc422ffd640 1 -- 172.21.15.123:0/2773642559 learned_addr learned my addr 172.21.15.123:0/2773642559 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:50:27.230 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:27.230+0000 7fc423fff640 1 --2- 172.21.15.123:0/2773642559 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc4240ad8b0 0x7fc4240add30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:50:27.230 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:27.230+0000 7fc4237fe640 1 --2- 172.21.15.123:0/2773642559 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fc40c004090 0x7fc42414b250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:50:27.230 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:27.230+0000 7fc422ffd640 1 -- 172.21.15.123:0/2773642559 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fc40c004090 msgr2=0x7fc42414b250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:50:27.230 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:27.230+0000 7fc422ffd640 1 --2- 172.21.15.123:0/2773642559 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fc40c004090 0x7fc42414b250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:27.231 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:27.230+0000 7fc422ffd640 1 -- 172.21.15.123:0/2773642559 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc4240ad8b0 msgr2=0x7fc4240add30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:50:27.231 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:27.230+0000 7fc422ffd640 1 --2- 172.21.15.123:0/2773642559 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc4240ad8b0 0x7fc4240add30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:27.231 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:27.230+0000 7fc422ffd640 1 -- 172.21.15.123:0/2773642559 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc424151af0 con 0x7fc424143df0 2024-09-16T15:50:27.231 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:27.230+0000 7fc4237fe640 1 --2- 172.21.15.123:0/2773642559 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fc40c004090 0x7fc42414b250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T15:50:27.231 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:27.230+0000 7fc422ffd640 1 --2- 172.21.15.123:0/2773642559 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc424143df0 0x7fc424151280 secure :-1 s=READY pgs=1570 cs=0 l=1 rev1=1 crypto rx=0x7fc41801dd30 tx=0x7fc418006d70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:50:27.231 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:27.230+0000 7fc40bfff640 1 -- 172.21.15.123:0/2773642559 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4180bf020 con 0x7fc424143df0 2024-09-16T15:50:27.231 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:27.230+0000 7fc40bfff640 1 -- 172.21.15.123:0/2773642559 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fc418003710 con 0x7fc424143df0 2024-09-16T15:50:27.231 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:27.230+0000 7fc429efd640 1 -- 172.21.15.123:0/2773642559 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc424151dd0 con 0x7fc424143df0 2024-09-16T15:50:27.231 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:27.230+0000 7fc40bfff640 1 -- 172.21.15.123:0/2773642559 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc418003b60 con 0x7fc424143df0 2024-09-16T15:50:27.231 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:27.230+0000 7fc429efd640 1 -- 172.21.15.123:0/2773642559 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc424152390 con 0x7fc424143df0 2024-09-16T15:50:27.231 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:27.230+0000 7fc429efd640 1 -- 172.21.15.123:0/2773642559 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc42415f190 con 0x7fc424143df0 2024-09-16T15:50:27.232 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:27.230+0000 7fc40bfff640 1 -- 172.21.15.123:0/2773642559 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 22) ==== 97692+0+0 (secure 0 0 0) 0x7fc4180c2020 con 0x7fc424143df0 2024-09-16T15:50:27.232 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:27.230+0000 7fc40bfff640 1 --2- 172.21.15.123:0/2773642559 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fc410073580 0x7fc410075a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:50:27.232 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:27.230+0000 7fc40bfff640 1 -- 172.21.15.123:0/2773642559 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(142..142 src has 1..142) ==== 4885+0+0 (secure 0 0 0) 0x7fc418132cf0 con 0x7fc424143df0 2024-09-16T15:50:27.232 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:27.230+0000 7fc4237fe640 1 --2- 172.21.15.123:0/2773642559 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fc410073580 0x7fc410075a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:50:27.232 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:27.230+0000 7fc4237fe640 1 --2- 172.21.15.123:0/2773642559 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fc410073580 0x7fc410075a40 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7fc414005fd0 tx=0x7fc414005e60 comp rx=0 tx=0).ready entity=mgr.4150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:50:27.234 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:27.234+0000 7fc40bfff640 1 -- 172.21.15.123:0/2773642559 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7fc4180c9030 con 0x7fc424143df0 2024-09-16T15:50:27.397 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:27.398+0000 7fc429efd640 1 -- 172.21.15.123:0/2773642559 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) -- 0x7fc4240b3cf0 con 0x7fc424143df0 2024-09-16T15:50:28.370 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:28.370+0000 7fc40bfff640 1 -- 172.21.15.123:0/2773642559 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v143) ==== 138+0+0 (secure 0 0 0) 0x7fc4180193e0 con 0x7fc424143df0 2024-09-16T15:50:28.370 INFO:teuthology.orchestra.run.smithi123.stderr:pool 'cephfs_metadata' created 2024-09-16T15:50:28.374 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:28.374+0000 7fc429efd640 1 -- 172.21.15.123:0/2773642559 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fc410073580 msgr2=0x7fc410075a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:50:28.374 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:28.374+0000 7fc429efd640 1 --2- 172.21.15.123:0/2773642559 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fc410073580 0x7fc410075a40 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7fc414005fd0 tx=0x7fc414005e60 comp rx=0 tx=0).stop 2024-09-16T15:50:28.374 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:28.374+0000 7fc429efd640 1 -- 172.21.15.123:0/2773642559 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc424143df0 msgr2=0x7fc424151280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:50:28.374 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:28.374+0000 7fc429efd640 1 --2- 172.21.15.123:0/2773642559 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc424143df0 0x7fc424151280 secure :-1 s=READY pgs=1570 cs=0 l=1 rev1=1 crypto rx=0x7fc41801dd30 tx=0x7fc418006d70 comp rx=0 tx=0).stop 2024-09-16T15:50:28.374 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:28.374+0000 7fc429efd640 1 -- 172.21.15.123:0/2773642559 shutdown_connections 2024-09-16T15:50:28.374 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:28.374+0000 7fc429efd640 1 --2- 172.21.15.123:0/2773642559 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fc410073580 0x7fc410075a40 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:28.374 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:28.374+0000 7fc429efd640 1 --2- 172.21.15.123:0/2773642559 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc4240ad8b0 0x7fc4240add30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:28.374 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:28.374+0000 7fc429efd640 1 --2- 172.21.15.123:0/2773642559 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc424143df0 0x7fc424151280 unknown :-1 s=CLOSED pgs=1570 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:28.374 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:28.374+0000 7fc429efd640 1 --2- 172.21.15.123:0/2773642559 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fc40c004090 0x7fc42414b250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:28.374 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:28.374+0000 7fc429efd640 1 -- 172.21.15.123:0/2773642559 >> 172.21.15.123:0/2773642559 conn(0x7fc4240b1630 msgr2=0x7fc4240b0090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:50:28.374 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:28.374+0000 7fc429efd640 1 -- 172.21.15.123:0/2773642559 shutdown_connections 2024-09-16T15:50:28.374 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:28.374+0000 7fc429efd640 1 -- 172.21.15.123:0/2773642559 wait complete. 2024-09-16T15:50:28.399 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-09-16T15:50:28.538 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:28.538+0000 7fc41bfc8640 1 -- 172.21.15.123:0/1227826080 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc4140cc1a0 msgr2=0x7fc41415ddc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:50:28.538 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:28.538+0000 7fc41bfc8640 1 --2- 172.21.15.123:0/1227826080 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc4140cc1a0 0x7fc41415ddc0 secure :-1 s=READY pgs=1571 cs=0 l=1 rev1=1 crypto rx=0x7fc40c00ad90 tx=0x7fc40c01dbf0 comp rx=0 tx=0).stop 2024-09-16T15:50:28.538 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:28.538+0000 7fc41bfc8640 1 -- 172.21.15.123:0/1227826080 shutdown_connections 2024-09-16T15:50:28.538 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:28.538+0000 7fc41bfc8640 1 --2- 172.21.15.123:0/1227826080 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc4140cc1a0 0x7fc41415ddc0 unknown :-1 s=CLOSED pgs=1571 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:28.538 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:28.538+0000 7fc41bfc8640 1 --2- 172.21.15.123:0/1227826080 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fc4140ca650 0x7fc4140cbc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:28.538 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:28.538+0000 7fc41bfc8640 1 -- 172.21.15.123:0/1227826080 >> 172.21.15.123:0/1227826080 conn(0x7fc4140b1710 msgr2=0x7fc4140b1b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:50:28.538 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:28.538+0000 7fc41bfc8640 1 -- 172.21.15.123:0/1227826080 shutdown_connections 2024-09-16T15:50:28.538 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:28.538+0000 7fc41bfc8640 1 -- 172.21.15.123:0/1227826080 wait complete. 2024-09-16T15:50:28.539 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:28.538+0000 7fc41bfc8640 1 Processor -- start 2024-09-16T15:50:28.539 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:28.538+0000 7fc41bfc8640 1 -- start start 2024-09-16T15:50:28.541 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:28.538+0000 7fc41bfc8640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc4140ca650 0x7fc414149f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:50:28.541 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:28.538+0000 7fc41bfc8640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fc41415a170 0x7fc41415ca10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:50:28.541 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:28.538+0000 7fc41bfc8640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc41415cf50 0x7fc4140aa5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:50:28.541 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:28.538+0000 7fc41bfc8640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fc41406ed40 con 0x7fc4140ca650 2024-09-16T15:50:28.541 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:28.538+0000 7fc41bfc8640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fc41406ebc0 con 0x7fc41415cf50 2024-09-16T15:50:28.541 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:28.538+0000 7fc41bfc8640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fc41415d460 con 0x7fc41415a170 2024-09-16T15:50:28.541 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:28.538+0000 7fc419d64640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc4140ca650 0x7fc414149f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:50:28.542 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:28.538+0000 7fc419d64640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc4140ca650 0x7fc414149f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:56014/0 (socket says 172.21.15.123:56014) 2024-09-16T15:50:28.542 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:28.538+0000 7fc419d64640 1 -- 172.21.15.123:0/3196802981 learned_addr learned my addr 172.21.15.123:0/3196802981 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:50:28.542 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:28.538+0000 7fc41a565640 1 --2- 172.21.15.123:0/3196802981 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc41415cf50 0x7fc4140aa5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:50:28.542 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:28.542+0000 7fc419563640 1 --2- 172.21.15.123:0/3196802981 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fc41415a170 0x7fc41415ca10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:50:28.542 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:28.542+0000 7fc419d64640 1 -- 172.21.15.123:0/3196802981 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fc41415a170 msgr2=0x7fc41415ca10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:50:28.542 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:28.542+0000 7fc419d64640 1 --2- 172.21.15.123:0/3196802981 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fc41415a170 0x7fc41415ca10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:28.542 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:28.542+0000 7fc419d64640 1 -- 172.21.15.123:0/3196802981 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc41415cf50 msgr2=0x7fc4140aa5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:50:28.542 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:28.542+0000 7fc419d64640 1 --2- 172.21.15.123:0/3196802981 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc41415cf50 0x7fc4140aa5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:28.542 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:28.542+0000 7fc419d64640 1 -- 172.21.15.123:0/3196802981 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc41415d740 con 0x7fc4140ca650 2024-09-16T15:50:28.542 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:28.542+0000 7fc41a565640 1 --2- 172.21.15.123:0/3196802981 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc41415cf50 0x7fc4140aa5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T15:50:28.542 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:28.542+0000 7fc419d64640 1 --2- 172.21.15.123:0/3196802981 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc4140ca650 0x7fc414149f00 secure :-1 s=READY pgs=1572 cs=0 l=1 rev1=1 crypto rx=0x7fc41000b410 tx=0x7fc41000b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:50:28.542 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:28.542+0000 7fc40a7fc640 1 -- 172.21.15.123:0/3196802981 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc410015030 con 0x7fc4140ca650 2024-09-16T15:50:28.542 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:28.542+0000 7fc40a7fc640 1 -- 172.21.15.123:0/3196802981 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fc410004020 con 0x7fc4140ca650 2024-09-16T15:50:28.543 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:28.542+0000 7fc41bfc8640 1 -- 172.21.15.123:0/3196802981 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc41415da20 con 0x7fc4140ca650 2024-09-16T15:50:28.543 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:28.542+0000 7fc40a7fc640 1 -- 172.21.15.123:0/3196802981 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc410004470 con 0x7fc4140ca650 2024-09-16T15:50:28.543 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:28.542+0000 7fc41bfc8640 1 -- 172.21.15.123:0/3196802981 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc41415dfe0 con 0x7fc4140ca650 2024-09-16T15:50:28.543 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:28.542+0000 7fc41bfc8640 1 -- 172.21.15.123:0/3196802981 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc4140cbc60 con 0x7fc4140ca650 2024-09-16T15:50:28.543 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:28.542+0000 7fc40a7fc640 1 -- 172.21.15.123:0/3196802981 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 22) ==== 97692+0+0 (secure 0 0 0) 0x7fc410018070 con 0x7fc4140ca650 2024-09-16T15:50:28.544 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:28.542+0000 7fc40a7fc640 1 --2- 172.21.15.123:0/3196802981 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fc3f80735b0 0x7fc3f8075a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:50:28.544 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:28.542+0000 7fc419563640 1 --2- 172.21.15.123:0/3196802981 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fc3f80735b0 0x7fc3f8075a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:50:28.544 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:28.542+0000 7fc40a7fc640 1 -- 172.21.15.123:0/3196802981 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(143..143 src has 1..143) ==== 5255+0+0 (secure 0 0 0) 0x7fc4100947b0 con 0x7fc4140ca650 2024-09-16T15:50:28.544 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:28.542+0000 7fc419563640 1 --2- 172.21.15.123:0/3196802981 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fc3f80735b0 0x7fc3f8075a70 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7fc414159980 tx=0x7fc404005e30 comp rx=0 tx=0).ready entity=mgr.4150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:50:28.546 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:28.546+0000 7fc40a7fc640 1 -- 172.21.15.123:0/3196802981 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7fc410018e40 con 0x7fc4140ca650 2024-09-16T15:50:28.718 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:28.718+0000 7fc41bfc8640 1 -- 172.21.15.123:0/3196802981 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) -- 0x7fc41414a440 con 0x7fc4140ca650 2024-09-16T15:50:29.382 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:29.382+0000 7fc40a7fc640 1 -- 172.21.15.123:0/3196802981 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v144) ==== 170+0+0 (secure 0 0 0) 0x7fc41005f220 con 0x7fc4140ca650 2024-09-16T15:50:29.382 INFO:teuthology.orchestra.run.smithi123.stderr:pool 'cephfs_data' created 2024-09-16T15:50:29.386 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:29.386+0000 7fc41bfc8640 1 -- 172.21.15.123:0/3196802981 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fc3f80735b0 msgr2=0x7fc3f8075a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:50:29.386 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:29.386+0000 7fc41bfc8640 1 --2- 172.21.15.123:0/3196802981 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fc3f80735b0 0x7fc3f8075a70 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7fc414159980 tx=0x7fc404005e30 comp rx=0 tx=0).stop 2024-09-16T15:50:29.386 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:29.386+0000 7fc41bfc8640 1 -- 172.21.15.123:0/3196802981 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc4140ca650 msgr2=0x7fc414149f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:50:29.386 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:29.386+0000 7fc41bfc8640 1 --2- 172.21.15.123:0/3196802981 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc4140ca650 0x7fc414149f00 secure :-1 s=READY pgs=1572 cs=0 l=1 rev1=1 crypto rx=0x7fc41000b410 tx=0x7fc41000b8d0 comp rx=0 tx=0).stop 2024-09-16T15:50:29.386 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:29.386+0000 7fc41bfc8640 1 -- 172.21.15.123:0/3196802981 shutdown_connections 2024-09-16T15:50:29.386 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:29.386+0000 7fc41bfc8640 1 --2- 172.21.15.123:0/3196802981 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fc3f80735b0 0x7fc3f8075a70 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:29.387 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:29.386+0000 7fc41bfc8640 1 --2- 172.21.15.123:0/3196802981 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc41415cf50 0x7fc4140aa5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:29.387 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:29.386+0000 7fc41bfc8640 1 --2- 172.21.15.123:0/3196802981 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fc41415a170 0x7fc41415ca10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:29.387 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:29.386+0000 7fc41bfc8640 1 --2- 172.21.15.123:0/3196802981 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc4140ca650 0x7fc414149f00 unknown :-1 s=CLOSED pgs=1572 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:29.387 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:29.386+0000 7fc41bfc8640 1 -- 172.21.15.123:0/3196802981 >> 172.21.15.123:0/3196802981 conn(0x7fc4140b1710 msgr2=0x7fc4140b2b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:50:29.387 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:29.386+0000 7fc41bfc8640 1 -- 172.21.15.123:0/3196802981 shutdown_connections 2024-09-16T15:50:29.387 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:29.386+0000 7fc41bfc8640 1 -- 172.21.15.123:0/3196802981 wait complete. 2024-09-16T15:50:29.421 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-09-16T15:50:29.546 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:29.546+0000 7fa18961a640 1 -- 172.21.15.123:0/3074110868 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fa18415a350 msgr2=0x7fa18406e3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:50:29.546 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:29.546+0000 7fa18961a640 1 --2- 172.21.15.123:0/3074110868 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fa18415a350 0x7fa18406e3b0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fa16c00a950 tx=0x7fa16c01db70 comp rx=0 tx=0).stop 2024-09-16T15:50:29.546 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:29.546+0000 7fa18961a640 1 -- 172.21.15.123:0/3074110868 shutdown_connections 2024-09-16T15:50:29.546 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:29.546+0000 7fa18961a640 1 --2- 172.21.15.123:0/3074110868 >> v2:172.21.15.184:3300/0 conn(0x7fa18406e8f0 0x7fa18415ddb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:29.546 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:29.546+0000 7fa18961a640 1 --2- 172.21.15.123:0/3074110868 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fa18415a350 0x7fa18406e3b0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:29.546 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:29.546+0000 7fa18961a640 1 -- 172.21.15.123:0/3074110868 >> 172.21.15.123:0/3074110868 conn(0x7fa1840b1710 msgr2=0x7fa1840b1b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:50:29.546 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:29.546+0000 7fa18961a640 1 -- 172.21.15.123:0/3074110868 shutdown_connections 2024-09-16T15:50:29.547 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:29.546+0000 7fa18961a640 1 -- 172.21.15.123:0/3074110868 wait complete. 2024-09-16T15:50:29.547 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:29.546+0000 7fa18961a640 1 Processor -- start 2024-09-16T15:50:29.547 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:29.546+0000 7fa18961a640 1 -- start start 2024-09-16T15:50:29.549 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:29.546+0000 7fa18961a640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa18406e8f0 0x7fa1840c97f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:50:29.549 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:29.546+0000 7fa18961a640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fa184159f70 0x7fa184155a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:50:29.549 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:29.546+0000 7fa18961a640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa18415a350 0x7fa1840b4ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:50:29.549 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:29.546+0000 7fa18961a640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fa1840ccbe0 con 0x7fa18406e8f0 2024-09-16T15:50:29.549 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:29.546+0000 7fa18961a640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fa1840cca60 con 0x7fa18415a350 2024-09-16T15:50:29.549 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:29.546+0000 7fa18961a640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fa184166f40 con 0x7fa184159f70 2024-09-16T15:50:29.549 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:29.550+0000 7fa182ffd640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa18406e8f0 0x7fa1840c97f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:50:29.550 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:29.550+0000 7fa182ffd640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa18406e8f0 0x7fa1840c97f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:56024/0 (socket says 172.21.15.123:56024) 2024-09-16T15:50:29.550 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:29.550+0000 7fa182ffd640 1 -- 172.21.15.123:0/788518674 learned_addr learned my addr 172.21.15.123:0/788518674 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:50:29.550 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:29.550+0000 7fa1837fe640 1 --2- 172.21.15.123:0/788518674 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa18415a350 0x7fa1840b4ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:50:29.550 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:29.550+0000 7fa1827fc640 1 --2- 172.21.15.123:0/788518674 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fa184159f70 0x7fa184155a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:50:29.550 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:29.550+0000 7fa182ffd640 1 -- 172.21.15.123:0/788518674 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fa184159f70 msgr2=0x7fa184155a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:50:29.550 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:29.550+0000 7fa182ffd640 1 --2- 172.21.15.123:0/788518674 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fa184159f70 0x7fa184155a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:29.550 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:29.550+0000 7fa182ffd640 1 -- 172.21.15.123:0/788518674 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa18415a350 msgr2=0x7fa1840b4ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:50:29.550 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:29.550+0000 7fa182ffd640 1 --2- 172.21.15.123:0/788518674 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa18415a350 0x7fa1840b4ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:29.550 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:29.550+0000 7fa182ffd640 1 -- 172.21.15.123:0/788518674 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa1841563e0 con 0x7fa18406e8f0 2024-09-16T15:50:29.550 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:29.550+0000 7fa182ffd640 1 --2- 172.21.15.123:0/788518674 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa18406e8f0 0x7fa1840c97f0 secure :-1 s=READY pgs=1573 cs=0 l=1 rev1=1 crypto rx=0x7fa17c00cba0 tx=0x7fa17c00cc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:50:29.550 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:29.550+0000 7fa17b7fe640 1 -- 172.21.15.123:0/788518674 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa17c016020 con 0x7fa18406e8f0 2024-09-16T15:50:29.551 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:29.550+0000 7fa17b7fe640 1 -- 172.21.15.123:0/788518674 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fa17c00ab40 con 0x7fa18406e8f0 2024-09-16T15:50:29.551 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:29.550+0000 7fa18961a640 1 -- 172.21.15.123:0/788518674 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa1841566c0 con 0x7fa18406e8f0 2024-09-16T15:50:29.551 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:29.550+0000 7fa18961a640 1 -- 172.21.15.123:0/788518674 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa184156c80 con 0x7fa18406e8f0 2024-09-16T15:50:29.551 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:29.550+0000 7fa17b7fe640 1 -- 172.21.15.123:0/788518674 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa17c012420 con 0x7fa18406e8f0 2024-09-16T15:50:29.552 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:29.550+0000 7fa18961a640 1 -- 172.21.15.123:0/788518674 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa18415ddb0 con 0x7fa18406e8f0 2024-09-16T15:50:29.552 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:29.550+0000 7fa17b7fe640 1 -- 172.21.15.123:0/788518674 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 22) ==== 97692+0+0 (secure 0 0 0) 0x7fa17c019020 con 0x7fa18406e8f0 2024-09-16T15:50:29.552 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:29.550+0000 7fa17b7fe640 1 --2- 172.21.15.123:0/788518674 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fa164073580 0x7fa164075a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:50:29.552 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:29.550+0000 7fa17b7fe640 1 -- 172.21.15.123:0/788518674 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(144..144 src has 1..144) ==== 5637+0+0 (secure 0 0 0) 0x7fa17c095230 con 0x7fa18406e8f0 2024-09-16T15:50:29.552 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:29.550+0000 7fa1827fc640 1 --2- 172.21.15.123:0/788518674 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fa164073580 0x7fa164075a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:50:29.552 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:29.554+0000 7fa1827fc640 1 --2- 172.21.15.123:0/788518674 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fa164073580 0x7fa164075a40 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7fa16c0b0000 tx=0x7fa16c01c3b0 comp rx=0 tx=0).ready entity=mgr.4150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:50:29.555 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:29.554+0000 7fa17b7fe640 1 -- 172.21.15.123:0/788518674 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7fa17c020020 con 0x7fa18406e8f0 2024-09-16T15:50:29.767 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:29.766+0000 7fa18961a640 1 -- 172.21.15.123:0/788518674 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) -- 0x7fa1841f5b40 con 0x7fa18406e8f0 2024-09-16T15:50:30.414 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:30.414+0000 7fa17b7fe640 1 -- 172.21.15.123:0/788518674 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 22 and data pool 23 v92) ==== 174+0+0 (secure 0 0 0) 0x7fa17c05fa90 con 0x7fa18406e8f0 2024-09-16T15:50:30.416 INFO:teuthology.orchestra.run.smithi123.stderr:new fs with metadata pool 22 and data pool 23 2024-09-16T15:50:30.422 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:30.422+0000 7fa18961a640 1 -- 172.21.15.123:0/788518674 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fa164073580 msgr2=0x7fa164075a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:50:30.423 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:30.422+0000 7fa18961a640 1 --2- 172.21.15.123:0/788518674 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fa164073580 0x7fa164075a40 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7fa16c0b0000 tx=0x7fa16c01c3b0 comp rx=0 tx=0).stop 2024-09-16T15:50:30.423 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:30.422+0000 7fa18961a640 1 -- 172.21.15.123:0/788518674 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa18406e8f0 msgr2=0x7fa1840c97f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:50:30.423 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:30.422+0000 7fa18961a640 1 --2- 172.21.15.123:0/788518674 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa18406e8f0 0x7fa1840c97f0 secure :-1 s=READY pgs=1573 cs=0 l=1 rev1=1 crypto rx=0x7fa17c00cba0 tx=0x7fa17c00cc90 comp rx=0 tx=0).stop 2024-09-16T15:50:30.423 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:30.422+0000 7fa18961a640 1 -- 172.21.15.123:0/788518674 shutdown_connections 2024-09-16T15:50:30.423 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:30.422+0000 7fa18961a640 1 --2- 172.21.15.123:0/788518674 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fa164073580 0x7fa164075a40 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:30.423 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:30.422+0000 7fa18961a640 1 --2- 172.21.15.123:0/788518674 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa18415a350 0x7fa1840b4ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:30.423 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:30.422+0000 7fa18961a640 1 --2- 172.21.15.123:0/788518674 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fa184159f70 0x7fa184155a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:30.423 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:30.422+0000 7fa18961a640 1 --2- 172.21.15.123:0/788518674 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa18406e8f0 0x7fa1840c97f0 unknown :-1 s=CLOSED pgs=1573 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:30.423 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:30.422+0000 7fa18961a640 1 -- 172.21.15.123:0/788518674 >> 172.21.15.123:0/788518674 conn(0x7fa1840b1710 msgr2=0x7fa184164710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:50:30.423 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:30.422+0000 7fa18961a640 1 -- 172.21.15.123:0/788518674 shutdown_connections 2024-09-16T15:50:30.424 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:30.422+0000 7fa18961a640 1 -- 172.21.15.123:0/788518674 wait complete. 2024-09-16T15:50:30.455 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-16T15:50:30.577 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:30.578+0000 7f47a39b8640 1 -- 172.21.15.123:0/1806811581 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f479c074470 msgr2=0x7f479c0cd6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:50:30.577 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:30.578+0000 7f47a39b8640 1 --2- 172.21.15.123:0/1806811581 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f479c074470 0x7f479c0cd6e0 secure :-1 s=READY pgs=1574 cs=0 l=1 rev1=1 crypto rx=0x7f4790009900 tx=0x7f479001c850 comp rx=0 tx=0).stop 2024-09-16T15:50:30.577 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:30.578+0000 7f47a39b8640 1 -- 172.21.15.123:0/1806811581 shutdown_connections 2024-09-16T15:50:30.578 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:30.578+0000 7f47a39b8640 1 --2- 172.21.15.123:0/1806811581 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f479c074470 0x7f479c0cd6e0 unknown :-1 s=CLOSED pgs=1574 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:30.578 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:30.578+0000 7f47a39b8640 1 --2- 172.21.15.123:0/1806811581 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f479c15e160 0x7f479c073f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:30.578 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:30.578+0000 7f47a39b8640 1 -- 172.21.15.123:0/1806811581 >> 172.21.15.123:0/1806811581 conn(0x7f479c0cfb10 msgr2=0x7f479c0cff20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:50:30.578 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:30.578+0000 7f47a39b8640 1 -- 172.21.15.123:0/1806811581 shutdown_connections 2024-09-16T15:50:30.578 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:30.578+0000 7f47a39b8640 1 -- 172.21.15.123:0/1806811581 wait complete. 2024-09-16T15:50:30.578 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:30.578+0000 7f47a39b8640 1 Processor -- start 2024-09-16T15:50:30.578 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:30.578+0000 7f47a39b8640 1 -- start start 2024-09-16T15:50:30.580 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:30.578+0000 7f47a39b8640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f479c074470 0x7f479c1faf10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:50:30.580 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:30.578+0000 7f47a39b8640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f479c0cdc20 0x7f479c15d000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:50:30.580 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:30.578+0000 7f47a39b8640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f479c15e160 0x7f479c14dc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:50:30.580 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:30.578+0000 7f47a39b8640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f479c167f50 con 0x7f479c15e160 2024-09-16T15:50:30.581 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:30.578+0000 7f47a39b8640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f479c167dd0 con 0x7f479c074470 2024-09-16T15:50:30.581 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:30.578+0000 7f47a39b8640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f479c1680d0 con 0x7f479c0cdc20 2024-09-16T15:50:30.581 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:30.578+0000 7f47a1f55640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f479c15e160 0x7f479c14dc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:50:30.581 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:30.578+0000 7f47a1f55640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f479c15e160 0x7f479c14dc50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:56032/0 (socket says 172.21.15.123:56032) 2024-09-16T15:50:30.581 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:30.578+0000 7f47a1f55640 1 -- 172.21.15.123:0/2258330436 learned_addr learned my addr 172.21.15.123:0/2258330436 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:50:30.581 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:30.578+0000 7f47a1754640 1 --2- 172.21.15.123:0/2258330436 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f479c074470 0x7f479c1faf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:50:30.581 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:30.578+0000 7f47a0f53640 1 --2- 172.21.15.123:0/2258330436 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f479c0cdc20 0x7f479c15d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:50:30.581 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:30.578+0000 7f47a1f55640 1 -- 172.21.15.123:0/2258330436 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f479c0cdc20 msgr2=0x7f479c15d000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:50:30.581 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:30.578+0000 7f47a1f55640 1 --2- 172.21.15.123:0/2258330436 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f479c0cdc20 0x7f479c15d000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:30.581 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:30.578+0000 7f47a1f55640 1 -- 172.21.15.123:0/2258330436 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f479c074470 msgr2=0x7f479c1faf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:50:30.581 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:30.578+0000 7f47a1f55640 1 --2- 172.21.15.123:0/2258330436 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f479c074470 0x7f479c1faf10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:30.581 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:30.578+0000 7f47a1f55640 1 -- 172.21.15.123:0/2258330436 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f479c0d08e0 con 0x7f479c15e160 2024-09-16T15:50:30.581 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:30.582+0000 7f47a1f55640 1 --2- 172.21.15.123:0/2258330436 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f479c15e160 0x7f479c14dc50 secure :-1 s=READY pgs=1575 cs=0 l=1 rev1=1 crypto rx=0x7f479400c3c0 tx=0x7f479400c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:50:30.581 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:30.582+0000 7f4799ffb640 1 -- 172.21.15.123:0/2258330436 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4794011790 con 0x7f479c15e160 2024-09-16T15:50:30.582 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:30.582+0000 7f4799ffb640 1 -- 172.21.15.123:0/2258330436 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f4794017070 con 0x7f479c15e160 2024-09-16T15:50:30.582 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:30.582+0000 7f47a39b8640 1 -- 172.21.15.123:0/2258330436 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f479c075ca0 con 0x7f479c15e160 2024-09-16T15:50:30.582 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:30.582+0000 7f4799ffb640 1 -- 172.21.15.123:0/2258330436 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4794011cc0 con 0x7f479c15e160 2024-09-16T15:50:30.582 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:30.582+0000 7f47a39b8640 1 -- 172.21.15.123:0/2258330436 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f479c1fb4f0 con 0x7f479c15e160 2024-09-16T15:50:30.583 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:30.582+0000 7f47a39b8640 1 -- 172.21.15.123:0/2258330436 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f479c0cd6e0 con 0x7f479c15e160 2024-09-16T15:50:30.583 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:30.582+0000 7f4799ffb640 1 -- 172.21.15.123:0/2258330436 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 22) ==== 97692+0+0 (secure 0 0 0) 0x7f479401c460 con 0x7f479c15e160 2024-09-16T15:50:30.583 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:30.582+0000 7f4799ffb640 1 --2- 172.21.15.123:0/2258330436 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f4788073580 0x7f4788075a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:50:30.583 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:30.582+0000 7f47a1754640 1 --2- 172.21.15.123:0/2258330436 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f4788073580 0x7f4788075a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:50:30.583 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:30.582+0000 7f4799ffb640 1 -- 172.21.15.123:0/2258330436 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(145..145 src has 1..145) ==== 5737+0+0 (secure 0 0 0) 0x7f4794095270 con 0x7f479c15e160 2024-09-16T15:50:30.583 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:30.582+0000 7f47a1754640 1 --2- 172.21.15.123:0/2258330436 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f4788073580 0x7f4788075a40 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f478c005c50 tx=0x7f478c00a550 comp rx=0 tx=0).ready entity=mgr.4150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:50:30.586 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:30.586+0000 7f4799ffb640 1 -- 172.21.15.123:0/2258330436 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f479405fc00 con 0x7f479c15e160 2024-09-16T15:50:30.762 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:30.762+0000 7f47a39b8640 1 -- 172.21.15.123:0/2258330436 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f479c0d06c0 con 0x7f479c15e160 2024-09-16T15:50:30.765 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:30.766+0000 7f4799ffb640 1 -- 172.21.15.123:0/2258330436 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v145) ==== 74+0+16848 (secure 0 0 0) 0x7f479405f500 con 0x7f479c15e160 2024-09-16T15:50:30.765 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-09-16T15:50:30.765 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":145,"fsid":"56077f89-18c8-486f-ae7e-eecc0bb7822c","created":"2024-09-16T15:42:26.364892+0000","modified":"2024-09-16T15:50:29.771715+0000","last_up_change":"2024-09-16T15:42:38.707234+0000","last_in_change":"2024-09-16T15:42:29.423662+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":23,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-16T15:42:39.287764+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":22,"pool_name":"cephfs_metadata","create_time":"2024-09-16T15:50:27.407822+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"145","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":23,"pool_name":"cephfs_data","create_time":"2024-09-16T15:50:28.729831+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"145","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9e7c1955-a607-4671-9be7-f2ef760ceeaa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":144,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6800","nonce":3882044969},{"type":"v1","addr":"172.21.15.123:6801","nonce":3882044969}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6802","nonce":3882044969},{"type":"v1","addr":"172.21.15.123:6803","nonce":3882044969}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6806","nonce":3882044969},{"type":"v1","addr":"172.21.15.123:6807","nonce":3882044969}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6804","nonce":3882044969},{"type":"v1","addr":"172.21.15.123:6805","nonce":3882044969}]},"public_addr":"172.21.15.123:6801/3882044969","cluster_addr":"172.21.15.123:6803/3882044969","heartbeat_back_addr":"172.21.15.123:6807/3882044969","heartbeat_front_addr":"172.21.15.123:6805/3882044969","state":["exists","up"]},{"osd":1,"uuid":"c540cfde-0977-4742-9b17-fd634668e010","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":144,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6816","nonce":4037001850},{"type":"v1","addr":"172.21.15.123:6817","nonce":4037001850}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6818","nonce":4037001850},{"type":"v1","addr":"172.21.15.123:6819","nonce":4037001850}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6822","nonce":4037001850},{"type":"v1","addr":"172.21.15.123:6823","nonce":4037001850}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6820","nonce":4037001850},{"type":"v1","addr":"172.21.15.123:6821","nonce":4037001850}]},"public_addr":"172.21.15.123:6817/4037001850","cluster_addr":"172.21.15.123:6819/4037001850","heartbeat_back_addr":"172.21.15.123:6823/4037001850","heartbeat_front_addr":"172.21.15.123:6821/4037001850","state":["exists","up"]},{"osd":2,"uuid":"9864a496-f169-486b-95c4-1d59162793a7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":144,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6824","nonce":1332143088},{"type":"v1","addr":"172.21.15.123:6825","nonce":1332143088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6826","nonce":1332143088},{"type":"v1","addr":"172.21.15.123:6827","nonce":1332143088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6830","nonce":1332143088},{"type":"v1","addr":"172.21.15.123:6831","nonce":1332143088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6828","nonce":1332143088},{"type":"v1","addr":"172.21.15.123:6829","nonce":1332143088}]},"public_addr":"172.21.15.123:6825/1332143088","cluster_addr":"172.21.15.123:6827/1332143088","heartbeat_back_addr":"172.21.15.123:6831/1332143088","heartbeat_front_addr":"172.21.15.123:6829/1332143088","state":["exists","up"]},{"osd":3,"uuid":"c6d1f817-f074-4325-95e1-578eba13edda","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":144,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6808","nonce":988586998},{"type":"v1","addr":"172.21.15.123:6809","nonce":988586998}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6810","nonce":988586998},{"type":"v1","addr":"172.21.15.123:6811","nonce":988586998}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6814","nonce":988586998},{"type":"v1","addr":"172.21.15.123:6815","nonce":988586998}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6812","nonce":988586998},{"type":"v1","addr":"172.21.15.123:6813","nonce":988586998}]},"public_addr":"172.21.15.123:6809/988586998","cluster_addr":"172.21.15.123:6811/988586998","heartbeat_back_addr":"172.21.15.123:6815/988586998","heartbeat_front_addr":"172.21.15.123:6813/988586998","state":["exists","up"]},{"osd":4,"uuid":"200a8df4-50ef-40a5-b48a-58981269c086","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":144,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6816","nonce":3387813890},{"type":"v1","addr":"172.21.15.184:6817","nonce":3387813890}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6818","nonce":3387813890},{"type":"v1","addr":"172.21.15.184:6819","nonce":3387813890}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6822","nonce":3387813890},{"type":"v1","addr":"172.21.15.184:6823","nonce":3387813890}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6820","nonce":3387813890},{"type":"v1","addr":"172.21.15.184:6821","nonce":3387813890}]},"public_addr":"172.21.15.184:6817/3387813890","cluster_addr":"172.21.15.184:6819/3387813890","heartbeat_back_addr":"172.21.15.184:6823/3387813890","heartbeat_front_addr":"172.21.15.184:6821/3387813890","state":["exists","up"]},{"osd":5,"uuid":"32d5a900-da3d-412c-bc19-13861a2368d8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":144,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6808","nonce":2680916017},{"type":"v1","addr":"172.21.15.184:6809","nonce":2680916017}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6810","nonce":2680916017},{"type":"v1","addr":"172.21.15.184:6811","nonce":2680916017}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6814","nonce":2680916017},{"type":"v1","addr":"172.21.15.184:6815","nonce":2680916017}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6812","nonce":2680916017},{"type":"v1","addr":"172.21.15.184:6813","nonce":2680916017}]},"public_addr":"172.21.15.184:6809/2680916017","cluster_addr":"172.21.15.184:6811/2680916017","heartbeat_back_addr":"172.21.15.184:6815/2680916017","heartbeat_front_addr":"172.21.15.184:6813/2680916017","state":["exists","up"]},{"osd":6,"uuid":"cde6f39e-aa1b-4260-ac19-774ee6aaca72","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":144,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6824","nonce":2183120457},{"type":"v1","addr":"172.21.15.184:6825","nonce":2183120457}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6826","nonce":2183120457},{"type":"v1","addr":"172.21.15.184:6827","nonce":2183120457}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6830","nonce":2183120457},{"type":"v1","addr":"172.21.15.184:6831","nonce":2183120457}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6828","nonce":2183120457},{"type":"v1","addr":"172.21.15.184:6829","nonce":2183120457}]},"public_addr":"172.21.15.184:6825/2183120457","cluster_addr":"172.21.15.184:6827/2183120457","heartbeat_back_addr":"172.21.15.184:6831/2183120457","heartbeat_front_addr":"172.21.15.184:6829/2183120457","state":["exists","up"]},{"osd":7,"uuid":"4f381778-eee1-47c6-804f-df0e6b33e313","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":144,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6800","nonce":4102957285},{"type":"v1","addr":"172.21.15.184:6801","nonce":4102957285}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6802","nonce":4102957285},{"type":"v1","addr":"172.21.15.184:6803","nonce":4102957285}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6806","nonce":4102957285},{"type":"v1","addr":"172.21.15.184:6807","nonce":4102957285}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6804","nonce":4102957285},{"type":"v1","addr":"172.21.15.184:6805","nonce":4102957285}]},"public_addr":"172.21.15.184:6801/4102957285","cluster_addr":"172.21.15.184:6803/4102957285","heartbeat_back_addr":"172.21.15.184:6807/4102957285","heartbeat_front_addr":"172.21.15.184:6805/4102957285","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:35.972210+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.222007+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.198617+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.385663+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.358935+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.156066+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.752973+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:35.682846+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":17,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T15:50:30.767 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:30.766+0000 7f47a39b8640 1 -- 172.21.15.123:0/2258330436 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f4788073580 msgr2=0x7f4788075a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:50:30.767 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:30.766+0000 7f47a39b8640 1 --2- 172.21.15.123:0/2258330436 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f4788073580 0x7f4788075a40 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f478c005c50 tx=0x7f478c00a550 comp rx=0 tx=0).stop 2024-09-16T15:50:30.768 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:30.766+0000 7f47a39b8640 1 -- 172.21.15.123:0/2258330436 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f479c15e160 msgr2=0x7f479c14dc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:50:30.768 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:30.766+0000 7f47a39b8640 1 --2- 172.21.15.123:0/2258330436 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f479c15e160 0x7f479c14dc50 secure :-1 s=READY pgs=1575 cs=0 l=1 rev1=1 crypto rx=0x7f479400c3c0 tx=0x7f479400c880 comp rx=0 tx=0).stop 2024-09-16T15:50:30.768 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:30.766+0000 7f47a39b8640 1 -- 172.21.15.123:0/2258330436 shutdown_connections 2024-09-16T15:50:30.768 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:30.766+0000 7f47a39b8640 1 --2- 172.21.15.123:0/2258330436 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f4788073580 0x7f4788075a40 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:30.768 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:30.766+0000 7f47a39b8640 1 --2- 172.21.15.123:0/2258330436 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f479c15e160 0x7f479c14dc50 unknown :-1 s=CLOSED pgs=1575 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:30.768 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:30.766+0000 7f47a39b8640 1 --2- 172.21.15.123:0/2258330436 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f479c0cdc20 0x7f479c15d000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:30.768 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:30.766+0000 7f47a39b8640 1 --2- 172.21.15.123:0/2258330436 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f479c074470 0x7f479c1faf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:30.768 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:30.766+0000 7f47a39b8640 1 -- 172.21.15.123:0/2258330436 >> 172.21.15.123:0/2258330436 conn(0x7f479c0cfb10 msgr2=0x7f479c0af810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:50:30.768 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:30.766+0000 7f47a39b8640 1 -- 172.21.15.123:0/2258330436 shutdown_connections 2024-09-16T15:50:30.768 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:30.766+0000 7f47a39b8640 1 -- 172.21.15.123:0/2258330436 wait complete. 2024-09-16T15:50:30.785 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-16T15:50:30.906 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:30.906+0000 7f0bc3d19640 1 -- 172.21.15.123:0/910301241 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0bbc15bca0 msgr2=0x7f0bbc0d0630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:50:30.906 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:30.906+0000 7f0bc3d19640 1 --2- 172.21.15.123:0/910301241 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0bbc15bca0 0x7f0bbc0d0630 secure :-1 s=READY pgs=1577 cs=0 l=1 rev1=1 crypto rx=0x7f0bbc06df40 tx=0x7f0bb401c800 comp rx=0 tx=0).stop 2024-09-16T15:50:30.906 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:30.906+0000 7f0bc3d19640 1 -- 172.21.15.123:0/910301241 shutdown_connections 2024-09-16T15:50:30.906 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:30.906+0000 7f0bc3d19640 1 --2- 172.21.15.123:0/910301241 >> v2:172.21.15.184:3300/0 conn(0x7f0bbc0726c0 0x7f0bbc073ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:30.906 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:30.906+0000 7f0bc3d19640 1 --2- 172.21.15.123:0/910301241 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0bbc15bca0 0x7f0bbc0d0630 unknown :-1 s=CLOSED pgs=1577 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:30.906 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:30.906+0000 7f0bc3d19640 1 -- 172.21.15.123:0/910301241 >> 172.21.15.123:0/910301241 conn(0x7f0bbc0cfb10 msgr2=0x7f0bbc0cff20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:50:30.906 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:30.906+0000 7f0bc3d19640 1 -- 172.21.15.123:0/910301241 shutdown_connections 2024-09-16T15:50:30.906 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:30.906+0000 7f0bc3d19640 1 -- 172.21.15.123:0/910301241 wait complete. 2024-09-16T15:50:30.907 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:30.906+0000 7f0bc3d19640 1 Processor -- start 2024-09-16T15:50:30.907 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:30.906+0000 7f0bc3d19640 1 -- start start 2024-09-16T15:50:30.908 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:30.906+0000 7f0bc3d19640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0bbc0726c0 0x7f0bbc0c81b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:50:30.909 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:30.906+0000 7f0bc3d19640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f0bbc074430 0x7f0bbc1fbb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:50:30.909 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:30.910+0000 7f0bc3d19640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f0bbc15bca0 0x7f0bbc165c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:50:30.909 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:30.910+0000 7f0bc3d19640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f0bbc168e90 con 0x7f0bbc0726c0 2024-09-16T15:50:30.909 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:30.910+0000 7f0bc3d19640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f0bbc168d10 con 0x7f0bbc15bca0 2024-09-16T15:50:30.909 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:30.910+0000 7f0bc3d19640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f0bbc169010 con 0x7f0bbc074430 2024-09-16T15:50:30.909 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:30.910+0000 7f0bc1ab5640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0bbc0726c0 0x7f0bbc0c81b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:50:30.909 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:30.910+0000 7f0bc1ab5640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0bbc0726c0 0x7f0bbc0c81b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:56048/0 (socket says 172.21.15.123:56048) 2024-09-16T15:50:30.909 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:30.910+0000 7f0bc1ab5640 1 -- 172.21.15.123:0/2260799699 learned_addr learned my addr 172.21.15.123:0/2260799699 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:50:30.909 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:30.910+0000 7f0bc12b4640 1 --2- 172.21.15.123:0/2260799699 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f0bbc074430 0x7f0bbc1fbb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:50:30.909 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:30.910+0000 7f0bc22b6640 1 --2- 172.21.15.123:0/2260799699 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f0bbc15bca0 0x7f0bbc165c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:50:30.909 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:30.910+0000 7f0bc1ab5640 1 -- 172.21.15.123:0/2260799699 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f0bbc074430 msgr2=0x7f0bbc1fbb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:50:30.909 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:30.910+0000 7f0bc1ab5640 1 --2- 172.21.15.123:0/2260799699 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f0bbc074430 0x7f0bbc1fbb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:30.909 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:30.910+0000 7f0bc1ab5640 1 -- 172.21.15.123:0/2260799699 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f0bbc15bca0 msgr2=0x7f0bbc165c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:50:30.910 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:30.910+0000 7f0bc1ab5640 1 --2- 172.21.15.123:0/2260799699 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f0bbc15bca0 0x7f0bbc165c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:30.910 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:30.910+0000 7f0bc1ab5640 1 -- 172.21.15.123:0/2260799699 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0bbc0c86f0 con 0x7f0bbc0726c0 2024-09-16T15:50:30.910 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:30.910+0000 7f0bc1ab5640 1 --2- 172.21.15.123:0/2260799699 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0bbc0726c0 0x7f0bbc0c81b0 secure :-1 s=READY pgs=1578 cs=0 l=1 rev1=1 crypto rx=0x7f0bb40b0040 tx=0x7f0bb40a5740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:50:30.910 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:30.910+0000 7f0bba7fc640 1 -- 172.21.15.123:0/2260799699 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0bb40b2040 con 0x7f0bbc0726c0 2024-09-16T15:50:30.910 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:30.910+0000 7f0bba7fc640 1 -- 172.21.15.123:0/2260799699 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f0bb40043b0 con 0x7f0bbc0726c0 2024-09-16T15:50:30.910 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:30.910+0000 7f0bc3d19640 1 -- 172.21.15.123:0/2260799699 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0bbc0c8990 con 0x7f0bbc0726c0 2024-09-16T15:50:30.910 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:30.910+0000 7f0bba7fc640 1 -- 172.21.15.123:0/2260799699 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0bb4005280 con 0x7f0bbc0726c0 2024-09-16T15:50:30.910 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:30.910+0000 7f0bc3d19640 1 -- 172.21.15.123:0/2260799699 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0bbc0c8f50 con 0x7f0bbc0726c0 2024-09-16T15:50:30.911 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:30.910+0000 7f0bc3d19640 1 -- 172.21.15.123:0/2260799699 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0bbc073ef0 con 0x7f0bbc0726c0 2024-09-16T15:50:30.911 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:30.910+0000 7f0bba7fc640 1 -- 172.21.15.123:0/2260799699 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 22) ==== 97692+0+0 (secure 0 0 0) 0x7f0bb4005420 con 0x7f0bbc0726c0 2024-09-16T15:50:30.911 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:30.910+0000 7f0bba7fc640 1 --2- 172.21.15.123:0/2260799699 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f0ba8073580 0x7f0ba8075a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:50:30.911 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:30.910+0000 7f0bc12b4640 1 --2- 172.21.15.123:0/2260799699 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f0ba8073580 0x7f0ba8075a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:50:30.911 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:30.910+0000 7f0bba7fc640 1 -- 172.21.15.123:0/2260799699 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(145..145 src has 1..145) ==== 5737+0+0 (secure 0 0 0) 0x7f0bb412e630 con 0x7f0bbc0726c0 2024-09-16T15:50:30.911 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:30.910+0000 7f0bc12b4640 1 --2- 172.21.15.123:0/2260799699 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f0ba8073580 0x7f0ba8075a40 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f0bb0005c50 tx=0x7f0bb0005be0 comp rx=0 tx=0).ready entity=mgr.4150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:50:30.914 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:30.914+0000 7f0bba7fc640 1 -- 172.21.15.123:0/2260799699 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f0bb40164b0 con 0x7f0bbc0726c0 2024-09-16T15:50:31.089 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:31.090+0000 7f0bc3d19640 1 -- 172.21.15.123:0/2260799699 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f0bbc165740 con 0x7f0bbc0726c0 2024-09-16T15:50:31.091 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:31.090+0000 7f0bba7fc640 1 -- 172.21.15.123:0/2260799699 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v145) ==== 74+0+16848 (secure 0 0 0) 0x7f0bb40f8f50 con 0x7f0bbc0726c0 2024-09-16T15:50:31.091 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-09-16T15:50:31.091 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":145,"fsid":"56077f89-18c8-486f-ae7e-eecc0bb7822c","created":"2024-09-16T15:42:26.364892+0000","modified":"2024-09-16T15:50:29.771715+0000","last_up_change":"2024-09-16T15:42:38.707234+0000","last_in_change":"2024-09-16T15:42:29.423662+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":23,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-16T15:42:39.287764+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":22,"pool_name":"cephfs_metadata","create_time":"2024-09-16T15:50:27.407822+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"145","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":23,"pool_name":"cephfs_data","create_time":"2024-09-16T15:50:28.729831+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"145","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9e7c1955-a607-4671-9be7-f2ef760ceeaa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":144,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6800","nonce":3882044969},{"type":"v1","addr":"172.21.15.123:6801","nonce":3882044969}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6802","nonce":3882044969},{"type":"v1","addr":"172.21.15.123:6803","nonce":3882044969}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6806","nonce":3882044969},{"type":"v1","addr":"172.21.15.123:6807","nonce":3882044969}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6804","nonce":3882044969},{"type":"v1","addr":"172.21.15.123:6805","nonce":3882044969}]},"public_addr":"172.21.15.123:6801/3882044969","cluster_addr":"172.21.15.123:6803/3882044969","heartbeat_back_addr":"172.21.15.123:6807/3882044969","heartbeat_front_addr":"172.21.15.123:6805/3882044969","state":["exists","up"]},{"osd":1,"uuid":"c540cfde-0977-4742-9b17-fd634668e010","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":144,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6816","nonce":4037001850},{"type":"v1","addr":"172.21.15.123:6817","nonce":4037001850}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6818","nonce":4037001850},{"type":"v1","addr":"172.21.15.123:6819","nonce":4037001850}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6822","nonce":4037001850},{"type":"v1","addr":"172.21.15.123:6823","nonce":4037001850}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6820","nonce":4037001850},{"type":"v1","addr":"172.21.15.123:6821","nonce":4037001850}]},"public_addr":"172.21.15.123:6817/4037001850","cluster_addr":"172.21.15.123:6819/4037001850","heartbeat_back_addr":"172.21.15.123:6823/4037001850","heartbeat_front_addr":"172.21.15.123:6821/4037001850","state":["exists","up"]},{"osd":2,"uuid":"9864a496-f169-486b-95c4-1d59162793a7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":144,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6824","nonce":1332143088},{"type":"v1","addr":"172.21.15.123:6825","nonce":1332143088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6826","nonce":1332143088},{"type":"v1","addr":"172.21.15.123:6827","nonce":1332143088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6830","nonce":1332143088},{"type":"v1","addr":"172.21.15.123:6831","nonce":1332143088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6828","nonce":1332143088},{"type":"v1","addr":"172.21.15.123:6829","nonce":1332143088}]},"public_addr":"172.21.15.123:6825/1332143088","cluster_addr":"172.21.15.123:6827/1332143088","heartbeat_back_addr":"172.21.15.123:6831/1332143088","heartbeat_front_addr":"172.21.15.123:6829/1332143088","state":["exists","up"]},{"osd":3,"uuid":"c6d1f817-f074-4325-95e1-578eba13edda","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":144,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6808","nonce":988586998},{"type":"v1","addr":"172.21.15.123:6809","nonce":988586998}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6810","nonce":988586998},{"type":"v1","addr":"172.21.15.123:6811","nonce":988586998}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6814","nonce":988586998},{"type":"v1","addr":"172.21.15.123:6815","nonce":988586998}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6812","nonce":988586998},{"type":"v1","addr":"172.21.15.123:6813","nonce":988586998}]},"public_addr":"172.21.15.123:6809/988586998","cluster_addr":"172.21.15.123:6811/988586998","heartbeat_back_addr":"172.21.15.123:6815/988586998","heartbeat_front_addr":"172.21.15.123:6813/988586998","state":["exists","up"]},{"osd":4,"uuid":"200a8df4-50ef-40a5-b48a-58981269c086","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":144,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6816","nonce":3387813890},{"type":"v1","addr":"172.21.15.184:6817","nonce":3387813890}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6818","nonce":3387813890},{"type":"v1","addr":"172.21.15.184:6819","nonce":3387813890}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6822","nonce":3387813890},{"type":"v1","addr":"172.21.15.184:6823","nonce":3387813890}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6820","nonce":3387813890},{"type":"v1","addr":"172.21.15.184:6821","nonce":3387813890}]},"public_addr":"172.21.15.184:6817/3387813890","cluster_addr":"172.21.15.184:6819/3387813890","heartbeat_back_addr":"172.21.15.184:6823/3387813890","heartbeat_front_addr":"172.21.15.184:6821/3387813890","state":["exists","up"]},{"osd":5,"uuid":"32d5a900-da3d-412c-bc19-13861a2368d8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":144,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6808","nonce":2680916017},{"type":"v1","addr":"172.21.15.184:6809","nonce":2680916017}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6810","nonce":2680916017},{"type":"v1","addr":"172.21.15.184:6811","nonce":2680916017}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6814","nonce":2680916017},{"type":"v1","addr":"172.21.15.184:6815","nonce":2680916017}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6812","nonce":2680916017},{"type":"v1","addr":"172.21.15.184:6813","nonce":2680916017}]},"public_addr":"172.21.15.184:6809/2680916017","cluster_addr":"172.21.15.184:6811/2680916017","heartbeat_back_addr":"172.21.15.184:6815/2680916017","heartbeat_front_addr":"172.21.15.184:6813/2680916017","state":["exists","up"]},{"osd":6,"uuid":"cde6f39e-aa1b-4260-ac19-774ee6aaca72","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":144,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6824","nonce":2183120457},{"type":"v1","addr":"172.21.15.184:6825","nonce":2183120457}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6826","nonce":2183120457},{"type":"v1","addr":"172.21.15.184:6827","nonce":2183120457}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6830","nonce":2183120457},{"type":"v1","addr":"172.21.15.184:6831","nonce":2183120457}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6828","nonce":2183120457},{"type":"v1","addr":"172.21.15.184:6829","nonce":2183120457}]},"public_addr":"172.21.15.184:6825/2183120457","cluster_addr":"172.21.15.184:6827/2183120457","heartbeat_back_addr":"172.21.15.184:6831/2183120457","heartbeat_front_addr":"172.21.15.184:6829/2183120457","state":["exists","up"]},{"osd":7,"uuid":"4f381778-eee1-47c6-804f-df0e6b33e313","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":144,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6800","nonce":4102957285},{"type":"v1","addr":"172.21.15.184:6801","nonce":4102957285}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6802","nonce":4102957285},{"type":"v1","addr":"172.21.15.184:6803","nonce":4102957285}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6806","nonce":4102957285},{"type":"v1","addr":"172.21.15.184:6807","nonce":4102957285}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6804","nonce":4102957285},{"type":"v1","addr":"172.21.15.184:6805","nonce":4102957285}]},"public_addr":"172.21.15.184:6801/4102957285","cluster_addr":"172.21.15.184:6803/4102957285","heartbeat_back_addr":"172.21.15.184:6807/4102957285","heartbeat_front_addr":"172.21.15.184:6805/4102957285","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:35.972210+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.222007+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.198617+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.385663+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.358935+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.156066+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.752973+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:35.682846+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":17,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T15:50:31.093 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:31.094+0000 7f0bc3d19640 1 -- 172.21.15.123:0/2260799699 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f0ba8073580 msgr2=0x7f0ba8075a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:50:31.094 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:31.094+0000 7f0bc3d19640 1 --2- 172.21.15.123:0/2260799699 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f0ba8073580 0x7f0ba8075a40 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f0bb0005c50 tx=0x7f0bb0005be0 comp rx=0 tx=0).stop 2024-09-16T15:50:31.094 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:31.094+0000 7f0bc3d19640 1 -- 172.21.15.123:0/2260799699 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0bbc0726c0 msgr2=0x7f0bbc0c81b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:50:31.094 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:31.094+0000 7f0bc3d19640 1 --2- 172.21.15.123:0/2260799699 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0bbc0726c0 0x7f0bbc0c81b0 secure :-1 s=READY pgs=1578 cs=0 l=1 rev1=1 crypto rx=0x7f0bb40b0040 tx=0x7f0bb40a5740 comp rx=0 tx=0).stop 2024-09-16T15:50:31.094 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:31.094+0000 7f0bc3d19640 1 -- 172.21.15.123:0/2260799699 shutdown_connections 2024-09-16T15:50:31.094 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:31.094+0000 7f0bc3d19640 1 --2- 172.21.15.123:0/2260799699 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f0ba8073580 0x7f0ba8075a40 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:31.094 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:31.094+0000 7f0bc3d19640 1 --2- 172.21.15.123:0/2260799699 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f0bbc15bca0 0x7f0bbc165c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:31.094 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:31.094+0000 7f0bc3d19640 1 --2- 172.21.15.123:0/2260799699 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f0bbc074430 0x7f0bbc1fbb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:31.094 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:31.094+0000 7f0bc3d19640 1 --2- 172.21.15.123:0/2260799699 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0bbc0726c0 0x7f0bbc0c81b0 unknown :-1 s=CLOSED pgs=1578 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:31.094 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:31.094+0000 7f0bc3d19640 1 -- 172.21.15.123:0/2260799699 >> 172.21.15.123:0/2260799699 conn(0x7f0bbc0cfb10 msgr2=0x7f0bbc0b49b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:50:31.094 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:31.094+0000 7f0bc3d19640 1 -- 172.21.15.123:0/2260799699 shutdown_connections 2024-09-16T15:50:31.094 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:31.094+0000 7f0bc3d19640 1 -- 172.21.15.123:0/2260799699 wait complete. 2024-09-16T15:50:31.111 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-09-16T15:50:31.238 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:31.238+0000 7fe7e7a0d640 1 -- 172.21.15.123:0/2918165122 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe7e0150700 msgr2=0x7fe7e0150b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:50:31.238 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:31.238+0000 7fe7e7a0d640 1 --2- 172.21.15.123:0/2918165122 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe7e0150700 0x7fe7e0150b50 secure :-1 s=READY pgs=1579 cs=0 l=1 rev1=1 crypto rx=0x7fe7e006db30 tx=0x7fe7d801e6d0 comp rx=0 tx=0).stop 2024-09-16T15:50:31.238 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:31.238+0000 7fe7e7a0d640 1 -- 172.21.15.123:0/2918165122 shutdown_connections 2024-09-16T15:50:31.238 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:31.238+0000 7fe7e7a0d640 1 --2- 172.21.15.123:0/2918165122 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe7e0150700 0x7fe7e0150b50 unknown :-1 s=CLOSED pgs=1579 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:31.239 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:31.238+0000 7fe7e7a0d640 1 --2- 172.21.15.123:0/2918165122 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fe7e00cf140 0x7fe7e00cf580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:31.239 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:31.238+0000 7fe7e7a0d640 1 --2- 172.21.15.123:0/2918165122 >> v2:172.21.15.184:3300/0 conn(0x7fe7e00cc550 0x7fe7e00cc930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:31.239 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:31.238+0000 7fe7e7a0d640 1 -- 172.21.15.123:0/2918165122 >> 172.21.15.123:0/2918165122 conn(0x7fe7e00b1710 msgr2=0x7fe7e00b1b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:50:31.239 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:31.238+0000 7fe7e7a0d640 1 -- 172.21.15.123:0/2918165122 shutdown_connections 2024-09-16T15:50:31.239 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:31.238+0000 7fe7e7a0d640 1 -- 172.21.15.123:0/2918165122 wait complete. 2024-09-16T15:50:31.239 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:31.238+0000 7fe7e7a0d640 1 Processor -- start 2024-09-16T15:50:31.239 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:31.238+0000 7fe7e7a0d640 1 -- start start 2024-09-16T15:50:31.241 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:31.242+0000 7fe7e7a0d640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe7e00cc550 0x7fe7e00aede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:50:31.241 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:31.242+0000 7fe7e7a0d640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fe7e00cf140 0x7fe7e014f6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:50:31.242 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:31.242+0000 7fe7e7a0d640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe7e0150700 0x7fe7e0073460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:50:31.242 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:31.242+0000 7fe7e7a0d640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fe7e00cdaa0 con 0x7fe7e0150700 2024-09-16T15:50:31.242 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:31.242+0000 7fe7e7a0d640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fe7e00cd920 con 0x7fe7e00cc550 2024-09-16T15:50:31.242 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:31.242+0000 7fe7e7a0d640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fe7e00cdc20 con 0x7fe7e00cf140 2024-09-16T15:50:31.242 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:31.242+0000 7fe7e5faa640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe7e0150700 0x7fe7e0073460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:50:31.242 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:31.242+0000 7fe7e5faa640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe7e0150700 0x7fe7e0073460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:56074/0 (socket says 172.21.15.123:56074) 2024-09-16T15:50:31.242 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:31.242+0000 7fe7e5faa640 1 -- 172.21.15.123:0/1917785830 learned_addr learned my addr 172.21.15.123:0/1917785830 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:50:31.242 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:31.242+0000 7fe7e57a9640 1 --2- 172.21.15.123:0/1917785830 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe7e00cc550 0x7fe7e00aede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:50:31.242 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:31.242+0000 7fe7e4fa8640 1 --2- 172.21.15.123:0/1917785830 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fe7e00cf140 0x7fe7e014f6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:50:31.242 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:31.242+0000 7fe7e5faa640 1 -- 172.21.15.123:0/1917785830 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fe7e00cf140 msgr2=0x7fe7e014f6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:50:31.242 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:31.242+0000 7fe7e5faa640 1 --2- 172.21.15.123:0/1917785830 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fe7e00cf140 0x7fe7e014f6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:31.242 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:31.242+0000 7fe7e5faa640 1 -- 172.21.15.123:0/1917785830 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe7e00cc550 msgr2=0x7fe7e00aede0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:50:31.242 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:31.242+0000 7fe7e5faa640 1 --2- 172.21.15.123:0/1917785830 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe7e00cc550 0x7fe7e00aede0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:31.242 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:31.242+0000 7fe7e5faa640 1 -- 172.21.15.123:0/1917785830 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe7e00ceee0 con 0x7fe7e0150700 2024-09-16T15:50:31.243 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:31.242+0000 7fe7e57a9640 1 --2- 172.21.15.123:0/1917785830 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe7e00cc550 0x7fe7e00aede0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T15:50:31.243 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:31.242+0000 7fe7e5faa640 1 --2- 172.21.15.123:0/1917785830 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe7e0150700 0x7fe7e0073460 secure :-1 s=READY pgs=1580 cs=0 l=1 rev1=1 crypto rx=0x7fe7d801dcb0 tx=0x7fe7d80a6a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:50:31.243 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:31.242+0000 7fe7d5ffb640 1 -- 172.21.15.123:0/1917785830 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7d80b3040 con 0x7fe7e0150700 2024-09-16T15:50:31.243 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:31.242+0000 7fe7d5ffb640 1 -- 172.21.15.123:0/1917785830 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fe7d80043b0 con 0x7fe7e0150700 2024-09-16T15:50:31.243 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:31.242+0000 7fe7e7a0d640 1 -- 172.21.15.123:0/1917785830 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe7e00c8e20 con 0x7fe7e0150700 2024-09-16T15:50:31.243 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:31.242+0000 7fe7d5ffb640 1 -- 172.21.15.123:0/1917785830 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7d80047c0 con 0x7fe7e0150700 2024-09-16T15:50:31.243 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:31.242+0000 7fe7e7a0d640 1 -- 172.21.15.123:0/1917785830 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe7e01513b0 con 0x7fe7e0150700 2024-09-16T15:50:31.243 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:31.242+0000 7fe7e7a0d640 1 -- 172.21.15.123:0/1917785830 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe7e00d0710 con 0x7fe7e0150700 2024-09-16T15:50:31.244 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:31.242+0000 7fe7d5ffb640 1 -- 172.21.15.123:0/1917785830 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 22) ==== 97692+0+0 (secure 0 0 0) 0x7fe7d80c0060 con 0x7fe7e0150700 2024-09-16T15:50:31.244 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:31.242+0000 7fe7d5ffb640 1 --2- 172.21.15.123:0/1917785830 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fe7c40735b0 0x7fe7c4075a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:50:31.244 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:31.242+0000 7fe7d5ffb640 1 -- 172.21.15.123:0/1917785830 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(145..145 src has 1..145) ==== 5737+0+0 (secure 0 0 0) 0x7fe7d812f0c0 con 0x7fe7e0150700 2024-09-16T15:50:31.244 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:31.242+0000 7fe7e57a9640 1 --2- 172.21.15.123:0/1917785830 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fe7c40735b0 0x7fe7c4075a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:50:31.244 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:31.242+0000 7fe7e57a9640 1 --2- 172.21.15.123:0/1917785830 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fe7c40735b0 0x7fe7c4075a70 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7fe7d0005fd0 tx=0x7fe7d0005d00 comp rx=0 tx=0).ready entity=mgr.4150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:50:31.247 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:31.246+0000 7fe7d5ffb640 1 -- 172.21.15.123:0/1917785830 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7fe7d8017860 con 0x7fe7e0150700 2024-09-16T15:50:31.446 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:31.446+0000 7fe7e7a0d640 1 -- 172.21.15.123:0/1917785830 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) -- 0x7fe7e0150b10 con 0x7fe7e0150700 2024-09-16T15:50:32.368 INFO:tasks.ceph.mds.b.smithi184.stderr:2024-09-16T15:50:32.363+0000 7f8f4afa3640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-09-16T15:50:32.422 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:32.422+0000 7fe7d5ffb640 1 -- 172.21.15.123:0/1917785830 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v94) ==== 118+0+0 (secure 0 0 0) 0x7fe7d80f9390 con 0x7fe7e0150700 2024-09-16T15:50:32.426 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:32.426+0000 7fe7e7a0d640 1 -- 172.21.15.123:0/1917785830 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fe7c40735b0 msgr2=0x7fe7c4075a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:50:32.426 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:32.426+0000 7fe7e7a0d640 1 --2- 172.21.15.123:0/1917785830 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fe7c40735b0 0x7fe7c4075a70 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7fe7d0005fd0 tx=0x7fe7d0005d00 comp rx=0 tx=0).stop 2024-09-16T15:50:32.426 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:32.426+0000 7fe7e7a0d640 1 -- 172.21.15.123:0/1917785830 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe7e0150700 msgr2=0x7fe7e0073460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:50:32.426 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:32.426+0000 7fe7e7a0d640 1 --2- 172.21.15.123:0/1917785830 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe7e0150700 0x7fe7e0073460 secure :-1 s=READY pgs=1580 cs=0 l=1 rev1=1 crypto rx=0x7fe7d801dcb0 tx=0x7fe7d80a6a00 comp rx=0 tx=0).stop 2024-09-16T15:50:32.426 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:32.426+0000 7fe7e7a0d640 1 -- 172.21.15.123:0/1917785830 shutdown_connections 2024-09-16T15:50:32.426 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:32.426+0000 7fe7e7a0d640 1 --2- 172.21.15.123:0/1917785830 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fe7c40735b0 0x7fe7c4075a70 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:32.426 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:32.426+0000 7fe7e7a0d640 1 --2- 172.21.15.123:0/1917785830 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe7e0150700 0x7fe7e0073460 unknown :-1 s=CLOSED pgs=1580 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:32.426 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:32.426+0000 7fe7e7a0d640 1 --2- 172.21.15.123:0/1917785830 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fe7e00cf140 0x7fe7e014f6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:32.426 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:32.426+0000 7fe7e7a0d640 1 --2- 172.21.15.123:0/1917785830 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe7e00cc550 0x7fe7e00aede0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:32.426 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:32.426+0000 7fe7e7a0d640 1 -- 172.21.15.123:0/1917785830 >> 172.21.15.123:0/1917785830 conn(0x7fe7e00b1710 msgr2=0x7fe7e015ced0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:50:32.426 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:32.426+0000 7fe7e7a0d640 1 -- 172.21.15.123:0/1917785830 shutdown_connections 2024-09-16T15:50:32.427 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:32.426+0000 7fe7e7a0d640 1 -- 172.21.15.123:0/1917785830 wait complete. 2024-09-16T15:50:32.445 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-09-16T15:50:32.445 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-09-16T15:50:32.578 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:32.578+0000 7f5ee80de640 1 -- 172.21.15.123:0/1362639715 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5ee015bb70 msgr2=0x7f5ee00b1c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:50:32.578 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:32.578+0000 7f5ee80de640 1 --2- 172.21.15.123:0/1362639715 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5ee015bb70 0x7f5ee00b1c10 secure :-1 s=READY pgs=1581 cs=0 l=1 rev1=1 crypto rx=0x7f5edc009900 tx=0x7f5edc01c8e0 comp rx=0 tx=0).stop 2024-09-16T15:50:32.578 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:32.578+0000 7f5ee80de640 1 -- 172.21.15.123:0/1362639715 shutdown_connections 2024-09-16T15:50:32.578 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:32.578+0000 7f5ee80de640 1 --2- 172.21.15.123:0/1362639715 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f5ee00b2150 0x7f5ee00cfdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:32.578 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:32.578+0000 7f5ee80de640 1 --2- 172.21.15.123:0/1362639715 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5ee015bb70 0x7f5ee00b1c10 unknown :-1 s=CLOSED pgs=1581 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:32.578 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:32.578+0000 7f5ee80de640 1 -- 172.21.15.123:0/1362639715 >> 172.21.15.123:0/1362639715 conn(0x7f5ee009aa40 msgr2=0x7f5ee009ae50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:50:32.579 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:32.578+0000 7f5ee80de640 1 -- 172.21.15.123:0/1362639715 shutdown_connections 2024-09-16T15:50:32.579 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:32.578+0000 7f5ee80de640 1 -- 172.21.15.123:0/1362639715 wait complete. 2024-09-16T15:50:32.579 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:32.578+0000 7f5ee80de640 1 Processor -- start 2024-09-16T15:50:32.579 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:32.578+0000 7f5ee80de640 1 -- start start 2024-09-16T15:50:32.581 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:32.582+0000 7f5ee80de640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5ee00b2150 0x7f5ee01f8760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:50:32.581 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:32.582+0000 7f5ee80de640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f5ee00d0300 0x7f5ee0154100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:50:32.581 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:32.582+0000 7f5ee80de640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5ee015bb70 0x7f5ee0154640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:50:32.581 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:32.582+0000 7f5ee80de640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f5ee0169060 con 0x7f5ee00b2150 2024-09-16T15:50:32.582 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:32.582+0000 7f5ee80de640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f5ee0168ee0 con 0x7f5ee015bb70 2024-09-16T15:50:32.582 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:32.582+0000 7f5ee80de640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f5ee01691e0 con 0x7f5ee00d0300 2024-09-16T15:50:32.582 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:32.582+0000 7f5ee5e7a640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5ee00b2150 0x7f5ee01f8760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:50:32.582 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:32.582+0000 7f5ee5e7a640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5ee00b2150 0x7f5ee01f8760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:49612/0 (socket says 172.21.15.123:49612) 2024-09-16T15:50:32.582 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:32.582+0000 7f5ee5e7a640 1 -- 172.21.15.123:0/809843157 learned_addr learned my addr 172.21.15.123:0/809843157 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:50:32.582 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:32.582+0000 7f5ee5679640 1 --2- 172.21.15.123:0/809843157 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f5ee00d0300 0x7f5ee0154100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:50:32.582 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:32.582+0000 7f5ee667b640 1 --2- 172.21.15.123:0/809843157 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5ee015bb70 0x7f5ee0154640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:50:32.582 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:32.582+0000 7f5ee5e7a640 1 -- 172.21.15.123:0/809843157 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f5ee00d0300 msgr2=0x7f5ee0154100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:50:32.582 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:32.582+0000 7f5ee5e7a640 1 --2- 172.21.15.123:0/809843157 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f5ee00d0300 0x7f5ee0154100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:32.582 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:32.582+0000 7f5ee5e7a640 1 -- 172.21.15.123:0/809843157 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5ee015bb70 msgr2=0x7f5ee0154640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:50:32.582 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:32.582+0000 7f5ee5e7a640 1 --2- 172.21.15.123:0/809843157 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5ee015bb70 0x7f5ee0154640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:32.582 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:32.582+0000 7f5ee5e7a640 1 -- 172.21.15.123:0/809843157 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5ee0152500 con 0x7f5ee00b2150 2024-09-16T15:50:32.582 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:32.582+0000 7f5ee667b640 1 --2- 172.21.15.123:0/809843157 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5ee015bb70 0x7f5ee0154640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T15:50:32.583 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:32.582+0000 7f5ee5e7a640 1 --2- 172.21.15.123:0/809843157 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5ee00b2150 0x7f5ee01f8760 secure :-1 s=READY pgs=1582 cs=0 l=1 rev1=1 crypto rx=0x7f5edc009850 tx=0x7f5edc005760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:50:32.583 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:32.582+0000 7f5ed67fc640 1 -- 172.21.15.123:0/809843157 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5edc0b2040 con 0x7f5ee00b2150 2024-09-16T15:50:32.583 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:32.582+0000 7f5ed67fc640 1 -- 172.21.15.123:0/809843157 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f5edc0a5c40 con 0x7f5ee00b2150 2024-09-16T15:50:32.583 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:32.582+0000 7f5ee80de640 1 -- 172.21.15.123:0/809843157 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5ee01fbd40 con 0x7f5ee00b2150 2024-09-16T15:50:32.583 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:32.582+0000 7f5ed67fc640 1 -- 172.21.15.123:0/809843157 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5edc005030 con 0x7f5ee00b2150 2024-09-16T15:50:32.583 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:32.582+0000 7f5ee80de640 1 -- 172.21.15.123:0/809843157 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5ee01f7c80 con 0x7f5ee00b2150 2024-09-16T15:50:32.584 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:32.582+0000 7f5ee80de640 1 -- 172.21.15.123:0/809843157 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5ee00cfdc0 con 0x7f5ee00b2150 2024-09-16T15:50:32.584 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:32.582+0000 7f5ed67fc640 1 -- 172.21.15.123:0/809843157 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 22) ==== 97692+0+0 (secure 0 0 0) 0x7f5edc0051d0 con 0x7f5ee00b2150 2024-09-16T15:50:32.584 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:32.582+0000 7f5ed67fc640 1 --2- 172.21.15.123:0/809843157 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f5ec4073570 0x7f5ec4075a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:50:32.584 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:32.582+0000 7f5ed67fc640 1 -- 172.21.15.123:0/809843157 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(146..146 src has 1..146) ==== 5737+0+0 (secure 0 0 0) 0x7f5edc12e5f0 con 0x7f5ee00b2150 2024-09-16T15:50:32.584 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:32.582+0000 7f5ee5679640 1 --2- 172.21.15.123:0/809843157 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f5ec4073570 0x7f5ec4075a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:50:32.584 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:32.582+0000 7f5ee5679640 1 --2- 172.21.15.123:0/809843157 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f5ec4073570 0x7f5ec4075a30 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f5ed00058d0 tx=0x7f5ed0005860 comp rx=0 tx=0).ready entity=mgr.4150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:50:32.587 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:32.586+0000 7f5ed67fc640 1 -- 172.21.15.123:0/809843157 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f5ee0152500 con 0x7f5ee00b2150 2024-09-16T15:50:32.795 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:32.794+0000 7f5ee80de640 1 -- 172.21.15.123:0/809843157 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) -- 0x7f5ee00cc210 con 0x7f5ee00b2150 2024-09-16T15:50:33.424 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:33.426+0000 7f5ed67fc640 1 -- 172.21.15.123:0/809843157 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v95) ==== 122+0+0 (secure 0 0 0) 0x7f5edc0164d0 con 0x7f5ee00b2150 2024-09-16T15:50:33.428 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:33.426+0000 7f5ee80de640 1 -- 172.21.15.123:0/809843157 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f5ec4073570 msgr2=0x7f5ec4075a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:50:33.428 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:33.426+0000 7f5ee80de640 1 --2- 172.21.15.123:0/809843157 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f5ec4073570 0x7f5ec4075a30 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f5ed00058d0 tx=0x7f5ed0005860 comp rx=0 tx=0).stop 2024-09-16T15:50:33.428 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:33.426+0000 7f5ee80de640 1 -- 172.21.15.123:0/809843157 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5ee00b2150 msgr2=0x7f5ee01f8760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:50:33.428 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:33.426+0000 7f5ee80de640 1 --2- 172.21.15.123:0/809843157 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5ee00b2150 0x7f5ee01f8760 secure :-1 s=READY pgs=1582 cs=0 l=1 rev1=1 crypto rx=0x7f5edc009850 tx=0x7f5edc005760 comp rx=0 tx=0).stop 2024-09-16T15:50:33.428 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:33.426+0000 7f5ee80de640 1 -- 172.21.15.123:0/809843157 shutdown_connections 2024-09-16T15:50:33.428 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:33.426+0000 7f5ee80de640 1 --2- 172.21.15.123:0/809843157 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f5ec4073570 0x7f5ec4075a30 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:33.428 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:33.426+0000 7f5ee80de640 1 --2- 172.21.15.123:0/809843157 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5ee015bb70 0x7f5ee0154640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:33.428 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:33.426+0000 7f5ee80de640 1 --2- 172.21.15.123:0/809843157 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f5ee00d0300 0x7f5ee0154100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:33.428 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:33.426+0000 7f5ee80de640 1 --2- 172.21.15.123:0/809843157 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5ee00b2150 0x7f5ee01f8760 unknown :-1 s=CLOSED pgs=1582 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:33.428 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:33.426+0000 7f5ee80de640 1 -- 172.21.15.123:0/809843157 >> 172.21.15.123:0/809843157 conn(0x7f5ee009aa40 msgr2=0x7f5ee00d2130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:50:33.428 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:33.430+0000 7f5ee80de640 1 -- 172.21.15.123:0/809843157 shutdown_connections 2024-09-16T15:50:33.428 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:33.430+0000 7f5ee80de640 1 -- 172.21.15.123:0/809843157 wait complete. 2024-09-16T15:50:33.450 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-09-16T15:50:33.450 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-16T15:50:33.578 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:33.578+0000 7f5eb29f3640 1 -- 172.21.15.123:0/3151155498 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5e94003a10 msgr2=0x7f5e94003e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:50:33.578 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:33.578+0000 7f5eb29f3640 1 --2- 172.21.15.123:0/3151155498 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5e94003a10 0x7f5e94003e30 secure :-1 s=READY pgs=1584 cs=0 l=1 rev1=1 crypto rx=0x7f5e9c009b60 tx=0x7f5e9c01d280 comp rx=0 tx=0).stop 2024-09-16T15:50:33.578 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:33.578+0000 7f5eb0f90640 1 -- 172.21.15.123:0/3151155498 reap_dead start 2024-09-16T15:50:33.578 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:33.578+0000 7f5eb29f3640 1 -- 172.21.15.123:0/3151155498 shutdown_connections 2024-09-16T15:50:33.578 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:33.578+0000 7f5eb29f3640 1 -- 172.21.15.123:0/3151155498 >> 172.21.15.123:0/3151155498 conn(0x7f5eac074bd0 msgr2=0x7f5eac09aa40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:50:33.578 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:33.578+0000 7f5eb29f3640 1 -- 172.21.15.123:0/3151155498 shutdown_connections 2024-09-16T15:50:33.579 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:33.578+0000 7f5eb29f3640 1 -- 172.21.15.123:0/3151155498 wait complete. 2024-09-16T15:50:33.579 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:33.578+0000 7f5eb29f3640 1 Processor -- start 2024-09-16T15:50:33.579 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:33.578+0000 7f5eb29f3640 1 -- start start 2024-09-16T15:50:33.581 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:33.582+0000 7f5eb29f3640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5e94003a10 0x7f5eac153620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:50:33.581 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:33.582+0000 7f5eb29f3640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5eac158460 0x7f5eac15b770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:50:33.581 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:33.582+0000 7f5eb29f3640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f5eac1588e0 0x7f5eac15bcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:50:33.581 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:33.582+0000 7f5eb29f3640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f5eac168cd0 con 0x7f5e94003a10 2024-09-16T15:50:33.582 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:33.582+0000 7f5eb29f3640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f5eac168b50 con 0x7f5eac158460 2024-09-16T15:50:33.582 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:33.582+0000 7f5eb29f3640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f5eac168e50 con 0x7f5eac1588e0 2024-09-16T15:50:33.582 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:33.582+0000 7f5eabfff640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5e94003a10 0x7f5eac153620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:50:33.582 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:33.582+0000 7f5eabfff640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5e94003a10 0x7f5eac153620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:49626/0 (socket says 172.21.15.123:49626) 2024-09-16T15:50:33.582 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:33.582+0000 7f5eabfff640 1 -- 172.21.15.123:0/2332434432 learned_addr learned my addr 172.21.15.123:0/2332434432 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:50:33.582 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:33.582+0000 7f5eab7fe640 1 --2- 172.21.15.123:0/2332434432 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5eac158460 0x7f5eac15b770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:50:33.582 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:33.582+0000 7f5eb0f90640 1 --2- 172.21.15.123:0/2332434432 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f5eac1588e0 0x7f5eac15bcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:50:33.582 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:33.582+0000 7f5eabfff640 1 -- 172.21.15.123:0/2332434432 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f5eac1588e0 msgr2=0x7f5eac15bcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:50:33.582 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:33.582+0000 7f5eabfff640 1 --2- 172.21.15.123:0/2332434432 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f5eac1588e0 0x7f5eac15bcb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:33.582 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:33.582+0000 7f5eabfff640 1 -- 172.21.15.123:0/2332434432 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5eac158460 msgr2=0x7f5eac15b770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:50:33.582 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:33.582+0000 7f5eabfff640 1 --2- 172.21.15.123:0/2332434432 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5eac158460 0x7f5eac15b770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:33.582 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:33.582+0000 7f5eabfff640 1 -- 172.21.15.123:0/2332434432 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5eac153b60 con 0x7f5e94003a10 2024-09-16T15:50:33.583 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:33.582+0000 7f5eb0f90640 1 --2- 172.21.15.123:0/2332434432 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f5eac1588e0 0x7f5eac15bcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T15:50:33.583 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:33.582+0000 7f5eabfff640 1 --2- 172.21.15.123:0/2332434432 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5e94003a10 0x7f5eac153620 secure :-1 s=READY pgs=1585 cs=0 l=1 rev1=1 crypto rx=0x7f5e9c01d0b0 tx=0x7f5e9c0055d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:50:33.583 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:33.582+0000 7f5ea8ff9640 1 -- 172.21.15.123:0/2332434432 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e9c0b1040 con 0x7f5e94003a10 2024-09-16T15:50:33.583 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:33.582+0000 7f5ea8ff9640 1 -- 172.21.15.123:0/2332434432 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f5e9c0bb070 con 0x7f5e94003a10 2024-09-16T15:50:33.583 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:33.582+0000 7f5eb29f3640 1 -- 172.21.15.123:0/2332434432 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5eac158e80 con 0x7f5e94003a10 2024-09-16T15:50:33.583 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:33.582+0000 7f5ea8ff9640 1 -- 172.21.15.123:0/2332434432 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e9c0a5e60 con 0x7f5e94003a10 2024-09-16T15:50:33.583 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:33.582+0000 7f5eb29f3640 1 -- 172.21.15.123:0/2332434432 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5eac0b5a60 con 0x7f5e94003a10 2024-09-16T15:50:33.584 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:33.582+0000 7f5eb29f3640 1 -- 172.21.15.123:0/2332434432 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5eac0d1980 con 0x7f5e94003a10 2024-09-16T15:50:33.584 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:33.582+0000 7f5ea8ff9640 1 -- 172.21.15.123:0/2332434432 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 22) ==== 97692+0+0 (secure 0 0 0) 0x7f5e9c004290 con 0x7f5e94003a10 2024-09-16T15:50:33.584 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:33.582+0000 7f5ea8ff9640 1 --2- 172.21.15.123:0/2332434432 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f5e98073580 0x7f5e98075a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:50:33.584 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:33.582+0000 7f5eab7fe640 1 --2- 172.21.15.123:0/2332434432 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f5e98073580 0x7f5e98075a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:50:33.584 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:33.582+0000 7f5ea8ff9640 1 -- 172.21.15.123:0/2332434432 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(146..146 src has 1..146) ==== 5737+0+0 (secure 0 0 0) 0x7f5e9c12f610 con 0x7f5e94003a10 2024-09-16T15:50:33.584 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:33.582+0000 7f5eab7fe640 1 --2- 172.21.15.123:0/2332434432 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f5e98073580 0x7f5e98075a40 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f5eac15b6a0 tx=0x7f5ea00091e0 comp rx=0 tx=0).ready entity=mgr.4150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:50:33.587 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:33.586+0000 7f5ea8ff9640 1 -- 172.21.15.123:0/2332434432 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f5e9c016fc0 con 0x7f5e94003a10 2024-09-16T15:50:33.796 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:33.798+0000 7f5eb29f3640 1 -- 172.21.15.123:0/2332434432 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f5eac159bc0 con 0x7f5e94003a10 2024-09-16T15:50:33.796 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:33.798+0000 7f5ea8ff9640 1 -- 172.21.15.123:0/2332434432 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 95 v95) ==== 94+0+5236 (secure 0 0 0) 0x7f5e9c0f9ea0 con 0x7f5e94003a10 2024-09-16T15:50:33.797 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-09-16T15:50:33.797 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":95,"btime":"2024-09-16T15:50:33:425542+0000","default_fscid":11,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6580,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6835/430907849","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":430907849},{"type":"v1","addr":"172.21.15.184:6835","nonce":430907849}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":91},{"gid":8424,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6833/1494334556","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6832","nonce":1494334556},{"type":"v1","addr":"172.21.15.123:6833","nonce":1494334556}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":90},{"gid":8427,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6835/2168742484","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6834","nonce":2168742484},{"type":"v1","addr":"172.21.15.123:6835","nonce":2168742484}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":91}],"filesystems":[{"mdsmap":{"epoch":95,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-16T15:50:29.771624+0000","modified":"2024-09-16T15:50:32.799612+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8436},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8436":{"gid":8436,"name":"b","rank":0,"incarnation":92,"state":"up:active","state_seq":3,"addr":"172.21.15.184:6833/1547569255","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":1547569255},{"type":"v1","addr":"172.21.15.184:6833","nonce":1547569255}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[23],"metadata_pool":22,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8436,"qdb_cluster":[8436]},"id":11}]} 2024-09-16T15:50:33.797 INFO:teuthology.orchestra.run.smithi123.stderr:dumped fsmap epoch 95 2024-09-16T15:50:33.799 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:33.798+0000 7f5eb29f3640 1 -- 172.21.15.123:0/2332434432 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f5e98073580 msgr2=0x7f5e98075a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:50:33.799 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:33.798+0000 7f5eb29f3640 1 --2- 172.21.15.123:0/2332434432 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f5e98073580 0x7f5e98075a40 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f5eac15b6a0 tx=0x7f5ea00091e0 comp rx=0 tx=0).stop 2024-09-16T15:50:33.799 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:33.798+0000 7f5eb29f3640 1 -- 172.21.15.123:0/2332434432 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5e94003a10 msgr2=0x7f5eac153620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:50:33.799 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:33.798+0000 7f5eb29f3640 1 --2- 172.21.15.123:0/2332434432 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5e94003a10 0x7f5eac153620 secure :-1 s=READY pgs=1585 cs=0 l=1 rev1=1 crypto rx=0x7f5e9c01d0b0 tx=0x7f5e9c0055d0 comp rx=0 tx=0).stop 2024-09-16T15:50:33.799 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:33.798+0000 7f5eb29f3640 1 -- 172.21.15.123:0/2332434432 shutdown_connections 2024-09-16T15:50:33.799 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:33.798+0000 7f5eb29f3640 1 --2- 172.21.15.123:0/2332434432 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f5e98073580 0x7f5e98075a40 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:33.799 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:33.798+0000 7f5eb29f3640 1 --2- 172.21.15.123:0/2332434432 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f5eac1588e0 0x7f5eac15bcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:33.799 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:33.798+0000 7f5eb29f3640 1 --2- 172.21.15.123:0/2332434432 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5eac158460 0x7f5eac15b770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:33.800 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:33.798+0000 7f5eb29f3640 1 --2- 172.21.15.123:0/2332434432 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5e94003a10 0x7f5eac153620 unknown :-1 s=CLOSED pgs=1585 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:33.800 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:33.798+0000 7f5eb29f3640 1 -- 172.21.15.123:0/2332434432 >> 172.21.15.123:0/2332434432 conn(0x7f5eac074bd0 msgr2=0x7f5eac1674e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:50:33.800 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:33.798+0000 7f5eb29f3640 1 -- 172.21.15.123:0/2332434432 shutdown_connections 2024-09-16T15:50:33.800 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:33.798+0000 7f5eb29f3640 1 -- 172.21.15.123:0/2332434432 wait complete. 2024-09-16T15:50:33.816 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-16T15:50:33.942 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:33.942+0000 7f14721ec640 1 -- 172.21.15.123:0/4056232986 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f14540053d0 msgr2=0x7f145400ba30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:50:33.942 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:33.942+0000 7f14721ec640 1 --2- 172.21.15.123:0/4056232986 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f14540053d0 0x7f145400ba30 secure :-1 s=READY pgs=1587 cs=0 l=1 rev1=1 crypto rx=0x7f1464001dc0 tx=0x7f146401cae0 comp rx=0 tx=0).stop 2024-09-16T15:50:33.942 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:33.942+0000 7f14721ec640 1 -- 172.21.15.123:0/4056232986 shutdown_connections 2024-09-16T15:50:33.942 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:33.942+0000 7f14721ec640 1 --2- 172.21.15.123:0/4056232986 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f14540053d0 0x7f145400ba30 unknown :-1 s=CLOSED pgs=1587 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:33.942 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:33.942+0000 7f14721ec640 1 -- 172.21.15.123:0/4056232986 >> 172.21.15.123:0/4056232986 conn(0x7f146c0cfb10 msgr2=0x7f146c0cff20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:50:33.942 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:33.942+0000 7f14721ec640 1 -- 172.21.15.123:0/4056232986 shutdown_connections 2024-09-16T15:50:33.942 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:33.942+0000 7f14721ec640 1 -- 172.21.15.123:0/4056232986 wait complete. 2024-09-16T15:50:33.943 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:33.942+0000 7f14721ec640 1 Processor -- start 2024-09-16T15:50:33.943 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:33.942+0000 7f14721ec640 1 -- start start 2024-09-16T15:50:33.945 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:33.942+0000 7f14721ec640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f14540053d0 0x7f146c156b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:50:33.945 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:33.942+0000 7f14721ec640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f146c145450 0x7f146c1fb340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:50:33.945 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:33.942+0000 7f14721ec640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f146c0b39f0 0x7f146c0b3e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:50:33.945 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:33.946+0000 7f14721ec640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f146c074090 con 0x7f14540053d0 2024-09-16T15:50:33.945 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:33.946+0000 7f14721ec640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f146c073f10 con 0x7f146c145450 2024-09-16T15:50:33.945 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:33.946+0000 7f14721ec640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f146c074210 con 0x7f146c0b39f0 2024-09-16T15:50:33.945 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:33.946+0000 7f146b7fe640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f14540053d0 0x7f146c156b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:50:33.945 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:33.946+0000 7f146b7fe640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f14540053d0 0x7f146c156b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:49642/0 (socket says 172.21.15.123:49642) 2024-09-16T15:50:33.945 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:33.946+0000 7f146b7fe640 1 -- 172.21.15.123:0/4188821201 learned_addr learned my addr 172.21.15.123:0/4188821201 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:50:33.945 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:33.946+0000 7f146affd640 1 --2- 172.21.15.123:0/4188821201 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f146c145450 0x7f146c1fb340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:50:33.946 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:33.946+0000 7f146bfff640 1 --2- 172.21.15.123:0/4188821201 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f146c0b39f0 0x7f146c0b3e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:50:33.946 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:33.946+0000 7f146b7fe640 1 -- 172.21.15.123:0/4188821201 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f146c0b39f0 msgr2=0x7f146c0b3e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:50:33.946 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:33.946+0000 7f146b7fe640 1 --2- 172.21.15.123:0/4188821201 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f146c0b39f0 0x7f146c0b3e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:33.946 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:33.946+0000 7f146b7fe640 1 -- 172.21.15.123:0/4188821201 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f146c145450 msgr2=0x7f146c1fb340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:50:33.946 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:33.946+0000 7f146b7fe640 1 --2- 172.21.15.123:0/4188821201 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f146c145450 0x7f146c1fb340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:33.946 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:33.946+0000 7f146b7fe640 1 -- 172.21.15.123:0/4188821201 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f146c158890 con 0x7f14540053d0 2024-09-16T15:50:33.946 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:33.946+0000 7f146b7fe640 1 --2- 172.21.15.123:0/4188821201 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f14540053d0 0x7f146c156b80 secure :-1 s=READY pgs=1588 cs=0 l=1 rev1=1 crypto rx=0x7f145c00e990 tx=0x7f145c00ee50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:50:33.946 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:33.946+0000 7f1453fff640 1 -- 172.21.15.123:0/4188821201 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f145c00cbe0 con 0x7f14540053d0 2024-09-16T15:50:33.946 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:33.946+0000 7f1453fff640 1 -- 172.21.15.123:0/4188821201 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f145c005020 con 0x7f14540053d0 2024-09-16T15:50:33.946 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:33.946+0000 7f14721ec640 1 -- 172.21.15.123:0/4188821201 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f146c158b70 con 0x7f14540053d0 2024-09-16T15:50:33.946 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:33.946+0000 7f1453fff640 1 -- 172.21.15.123:0/4188821201 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f145c005450 con 0x7f14540053d0 2024-09-16T15:50:33.946 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:33.946+0000 7f14721ec640 1 -- 172.21.15.123:0/4188821201 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f146c159130 con 0x7f14540053d0 2024-09-16T15:50:33.947 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:33.946+0000 7f14721ec640 1 -- 172.21.15.123:0/4188821201 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f146c15eeb0 con 0x7f14540053d0 2024-09-16T15:50:33.947 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:33.946+0000 7f1453fff640 1 -- 172.21.15.123:0/4188821201 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 22) ==== 97692+0+0 (secure 0 0 0) 0x7f145c0047b0 con 0x7f14540053d0 2024-09-16T15:50:33.947 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:33.946+0000 7f1453fff640 1 --2- 172.21.15.123:0/4188821201 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f1458073580 0x7f1458075a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:50:33.948 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:33.946+0000 7f146affd640 1 --2- 172.21.15.123:0/4188821201 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f1458073580 0x7f1458075a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:50:33.948 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:33.946+0000 7f1453fff640 1 -- 172.21.15.123:0/4188821201 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(146..146 src has 1..146) ==== 5737+0+0 (secure 0 0 0) 0x7f145c0958e0 con 0x7f14540053d0 2024-09-16T15:50:33.948 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:33.946+0000 7f146affd640 1 --2- 172.21.15.123:0/4188821201 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f1458073580 0x7f1458075a40 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f1460000c10 tx=0x7f1460001150 comp rx=0 tx=0).ready entity=mgr.4150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:50:33.950 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:33.950+0000 7f1453fff640 1 -- 172.21.15.123:0/4188821201 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f145c016030 con 0x7f14540053d0 2024-09-16T15:50:34.111 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:34.110+0000 7f14721ec640 1 -- 172.21.15.123:0/4188821201 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f146c0d06c0 con 0x7f14540053d0 2024-09-16T15:50:34.113 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:34.114+0000 7f1453fff640 1 -- 172.21.15.123:0/4188821201 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v146) ==== 74+0+16835 (secure 0 0 0) 0x7f145c060050 con 0x7f14540053d0 2024-09-16T15:50:34.114 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-09-16T15:50:34.114 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":146,"fsid":"56077f89-18c8-486f-ae7e-eecc0bb7822c","created":"2024-09-16T15:42:26.364892+0000","modified":"2024-09-16T15:50:31.413800+0000","last_up_change":"2024-09-16T15:42:38.707234+0000","last_in_change":"2024-09-16T15:42:29.423662+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":23,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-16T15:42:39.287764+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":22,"pool_name":"cephfs_metadata","create_time":"2024-09-16T15:50:27.407822+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"145","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":23,"pool_name":"cephfs_data","create_time":"2024-09-16T15:50:28.729831+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"146","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9e7c1955-a607-4671-9be7-f2ef760ceeaa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":144,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6800","nonce":3882044969},{"type":"v1","addr":"172.21.15.123:6801","nonce":3882044969}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6802","nonce":3882044969},{"type":"v1","addr":"172.21.15.123:6803","nonce":3882044969}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6806","nonce":3882044969},{"type":"v1","addr":"172.21.15.123:6807","nonce":3882044969}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6804","nonce":3882044969},{"type":"v1","addr":"172.21.15.123:6805","nonce":3882044969}]},"public_addr":"172.21.15.123:6801/3882044969","cluster_addr":"172.21.15.123:6803/3882044969","heartbeat_back_addr":"172.21.15.123:6807/3882044969","heartbeat_front_addr":"172.21.15.123:6805/3882044969","state":["exists","up"]},{"osd":1,"uuid":"c540cfde-0977-4742-9b17-fd634668e010","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":144,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6816","nonce":4037001850},{"type":"v1","addr":"172.21.15.123:6817","nonce":4037001850}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6818","nonce":4037001850},{"type":"v1","addr":"172.21.15.123:6819","nonce":4037001850}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6822","nonce":4037001850},{"type":"v1","addr":"172.21.15.123:6823","nonce":4037001850}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6820","nonce":4037001850},{"type":"v1","addr":"172.21.15.123:6821","nonce":4037001850}]},"public_addr":"172.21.15.123:6817/4037001850","cluster_addr":"172.21.15.123:6819/4037001850","heartbeat_back_addr":"172.21.15.123:6823/4037001850","heartbeat_front_addr":"172.21.15.123:6821/4037001850","state":["exists","up"]},{"osd":2,"uuid":"9864a496-f169-486b-95c4-1d59162793a7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":144,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6824","nonce":1332143088},{"type":"v1","addr":"172.21.15.123:6825","nonce":1332143088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6826","nonce":1332143088},{"type":"v1","addr":"172.21.15.123:6827","nonce":1332143088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6830","nonce":1332143088},{"type":"v1","addr":"172.21.15.123:6831","nonce":1332143088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6828","nonce":1332143088},{"type":"v1","addr":"172.21.15.123:6829","nonce":1332143088}]},"public_addr":"172.21.15.123:6825/1332143088","cluster_addr":"172.21.15.123:6827/1332143088","heartbeat_back_addr":"172.21.15.123:6831/1332143088","heartbeat_front_addr":"172.21.15.123:6829/1332143088","state":["exists","up"]},{"osd":3,"uuid":"c6d1f817-f074-4325-95e1-578eba13edda","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":144,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6808","nonce":988586998},{"type":"v1","addr":"172.21.15.123:6809","nonce":988586998}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6810","nonce":988586998},{"type":"v1","addr":"172.21.15.123:6811","nonce":988586998}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6814","nonce":988586998},{"type":"v1","addr":"172.21.15.123:6815","nonce":988586998}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6812","nonce":988586998},{"type":"v1","addr":"172.21.15.123:6813","nonce":988586998}]},"public_addr":"172.21.15.123:6809/988586998","cluster_addr":"172.21.15.123:6811/988586998","heartbeat_back_addr":"172.21.15.123:6815/988586998","heartbeat_front_addr":"172.21.15.123:6813/988586998","state":["exists","up"]},{"osd":4,"uuid":"200a8df4-50ef-40a5-b48a-58981269c086","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":144,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6816","nonce":3387813890},{"type":"v1","addr":"172.21.15.184:6817","nonce":3387813890}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6818","nonce":3387813890},{"type":"v1","addr":"172.21.15.184:6819","nonce":3387813890}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6822","nonce":3387813890},{"type":"v1","addr":"172.21.15.184:6823","nonce":3387813890}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6820","nonce":3387813890},{"type":"v1","addr":"172.21.15.184:6821","nonce":3387813890}]},"public_addr":"172.21.15.184:6817/3387813890","cluster_addr":"172.21.15.184:6819/3387813890","heartbeat_back_addr":"172.21.15.184:6823/3387813890","heartbeat_front_addr":"172.21.15.184:6821/3387813890","state":["exists","up"]},{"osd":5,"uuid":"32d5a900-da3d-412c-bc19-13861a2368d8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":144,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6808","nonce":2680916017},{"type":"v1","addr":"172.21.15.184:6809","nonce":2680916017}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6810","nonce":2680916017},{"type":"v1","addr":"172.21.15.184:6811","nonce":2680916017}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6814","nonce":2680916017},{"type":"v1","addr":"172.21.15.184:6815","nonce":2680916017}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6812","nonce":2680916017},{"type":"v1","addr":"172.21.15.184:6813","nonce":2680916017}]},"public_addr":"172.21.15.184:6809/2680916017","cluster_addr":"172.21.15.184:6811/2680916017","heartbeat_back_addr":"172.21.15.184:6815/2680916017","heartbeat_front_addr":"172.21.15.184:6813/2680916017","state":["exists","up"]},{"osd":6,"uuid":"cde6f39e-aa1b-4260-ac19-774ee6aaca72","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":144,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6824","nonce":2183120457},{"type":"v1","addr":"172.21.15.184:6825","nonce":2183120457}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6826","nonce":2183120457},{"type":"v1","addr":"172.21.15.184:6827","nonce":2183120457}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6830","nonce":2183120457},{"type":"v1","addr":"172.21.15.184:6831","nonce":2183120457}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6828","nonce":2183120457},{"type":"v1","addr":"172.21.15.184:6829","nonce":2183120457}]},"public_addr":"172.21.15.184:6825/2183120457","cluster_addr":"172.21.15.184:6827/2183120457","heartbeat_back_addr":"172.21.15.184:6831/2183120457","heartbeat_front_addr":"172.21.15.184:6829/2183120457","state":["exists","up"]},{"osd":7,"uuid":"4f381778-eee1-47c6-804f-df0e6b33e313","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":144,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6800","nonce":4102957285},{"type":"v1","addr":"172.21.15.184:6801","nonce":4102957285}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6802","nonce":4102957285},{"type":"v1","addr":"172.21.15.184:6803","nonce":4102957285}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6806","nonce":4102957285},{"type":"v1","addr":"172.21.15.184:6807","nonce":4102957285}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6804","nonce":4102957285},{"type":"v1","addr":"172.21.15.184:6805","nonce":4102957285}]},"public_addr":"172.21.15.184:6801/4102957285","cluster_addr":"172.21.15.184:6803/4102957285","heartbeat_back_addr":"172.21.15.184:6807/4102957285","heartbeat_front_addr":"172.21.15.184:6805/4102957285","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:35.972210+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.222007+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.198617+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.385663+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.358935+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.156066+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.752973+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:35.682846+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":17,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T15:50:34.116 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:34.114+0000 7f14721ec640 1 -- 172.21.15.123:0/4188821201 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f1458073580 msgr2=0x7f1458075a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:50:34.116 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:34.114+0000 7f14721ec640 1 --2- 172.21.15.123:0/4188821201 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f1458073580 0x7f1458075a40 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f1460000c10 tx=0x7f1460001150 comp rx=0 tx=0).stop 2024-09-16T15:50:34.116 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:34.114+0000 7f14721ec640 1 -- 172.21.15.123:0/4188821201 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f14540053d0 msgr2=0x7f146c156b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:50:34.116 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:34.114+0000 7f14721ec640 1 --2- 172.21.15.123:0/4188821201 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f14540053d0 0x7f146c156b80 secure :-1 s=READY pgs=1588 cs=0 l=1 rev1=1 crypto rx=0x7f145c00e990 tx=0x7f145c00ee50 comp rx=0 tx=0).stop 2024-09-16T15:50:34.116 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:34.118+0000 7f14721ec640 1 -- 172.21.15.123:0/4188821201 shutdown_connections 2024-09-16T15:50:34.116 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:34.118+0000 7f14721ec640 1 --2- 172.21.15.123:0/4188821201 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f1458073580 0x7f1458075a40 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:34.116 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:34.118+0000 7f14721ec640 1 --2- 172.21.15.123:0/4188821201 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f146c0b39f0 0x7f146c0b3e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:34.117 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:34.118+0000 7f14721ec640 1 --2- 172.21.15.123:0/4188821201 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f146c145450 0x7f146c1fb340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:34.117 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:34.118+0000 7f14721ec640 1 --2- 172.21.15.123:0/4188821201 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f14540053d0 0x7f146c156b80 unknown :-1 s=CLOSED pgs=1588 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:34.117 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:34.118+0000 7f14721ec640 1 -- 172.21.15.123:0/4188821201 >> 172.21.15.123:0/4188821201 conn(0x7f146c0cfb10 msgr2=0x7f146c0cd6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:50:34.117 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:34.118+0000 7f14721ec640 1 -- 172.21.15.123:0/4188821201 shutdown_connections 2024-09-16T15:50:34.117 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:34.118+0000 7f14721ec640 1 -- 172.21.15.123:0/4188821201 wait complete. 2024-09-16T15:50:34.132 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-09-16T15:50:34.132 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T15:50:34.251 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:34.250+0000 7fd2605bb640 1 -- 172.21.15.123:0/2878180346 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd244004810 msgr2=0x7fd244004c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:50:34.252 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:34.250+0000 7fd2605bb640 1 --2- 172.21.15.123:0/2878180346 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd244004810 0x7fd244004c90 secure :-1 s=READY pgs=1590 cs=0 l=1 rev1=1 crypto rx=0x7fd24800c290 tx=0x7fd24801cf70 comp rx=0 tx=0).stop 2024-09-16T15:50:34.252 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:34.250+0000 7fd2605bb640 1 -- 172.21.15.123:0/2878180346 shutdown_connections 2024-09-16T15:50:34.252 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:34.250+0000 7fd2605bb640 1 --2- 172.21.15.123:0/2878180346 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd244004810 0x7fd244004c90 unknown :-1 s=CLOSED pgs=1590 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:34.252 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:34.250+0000 7fd2605bb640 1 -- 172.21.15.123:0/2878180346 >> 172.21.15.123:0/2878180346 conn(0x7fd2580d3ea0 msgr2=0x7fd2580d42b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:50:34.252 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:34.250+0000 7fd2605bb640 1 -- 172.21.15.123:0/2878180346 shutdown_connections 2024-09-16T15:50:34.252 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:34.250+0000 7fd2605bb640 1 -- 172.21.15.123:0/2878180346 wait complete. 2024-09-16T15:50:34.253 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:34.254+0000 7fd2605bb640 1 Processor -- start 2024-09-16T15:50:34.253 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:34.254+0000 7fd2605bb640 1 -- start start 2024-09-16T15:50:34.254 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:34.254+0000 7fd2605bb640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fd244004810 0x7fd2581555d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:50:34.254 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:34.254+0000 7fd2605bb640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd2580c9cc0 0x7fd2580c8830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:50:34.255 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:34.254+0000 7fd2605bb640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd2581e9a40 0x7fd2581e9ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:50:34.255 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:34.254+0000 7fd2605bb640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fd2580cf4d0 con 0x7fd2581e9a40 2024-09-16T15:50:34.255 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:34.254+0000 7fd2605bb640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fd2580cf350 con 0x7fd2580c9cc0 2024-09-16T15:50:34.255 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:34.254+0000 7fd2605bb640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fd2580cf650 con 0x7fd244004810 2024-09-16T15:50:34.255 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:34.254+0000 7fd25eb58640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd2581e9a40 0x7fd2581e9ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:50:34.255 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:34.254+0000 7fd25eb58640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd2581e9a40 0x7fd2581e9ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:49646/0 (socket says 172.21.15.123:49646) 2024-09-16T15:50:34.255 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:34.254+0000 7fd25eb58640 1 -- 172.21.15.123:0/731566901 learned_addr learned my addr 172.21.15.123:0/731566901 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:50:34.255 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:34.254+0000 7fd25e357640 1 --2- 172.21.15.123:0/731566901 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fd244004810 0x7fd2581555d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:50:34.255 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:34.254+0000 7fd25db56640 1 --2- 172.21.15.123:0/731566901 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd2580c9cc0 0x7fd2580c8830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:50:34.255 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:34.254+0000 7fd25eb58640 1 -- 172.21.15.123:0/731566901 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fd244004810 msgr2=0x7fd2581555d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:50:34.255 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:34.254+0000 7fd25eb58640 1 --2- 172.21.15.123:0/731566901 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fd244004810 0x7fd2581555d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:34.255 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:34.254+0000 7fd25eb58640 1 -- 172.21.15.123:0/731566901 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd2580c9cc0 msgr2=0x7fd2580c8830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:50:34.255 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:34.254+0000 7fd25eb58640 1 --2- 172.21.15.123:0/731566901 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd2580c9cc0 0x7fd2580c8830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:34.256 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:34.254+0000 7fd25eb58640 1 -- 172.21.15.123:0/731566901 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd2580ca4a0 con 0x7fd2581e9a40 2024-09-16T15:50:34.256 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:34.254+0000 7fd25eb58640 1 --2- 172.21.15.123:0/731566901 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd2581e9a40 0x7fd2581e9ec0 secure :-1 s=READY pgs=1591 cs=0 l=1 rev1=1 crypto rx=0x7fd2580caf00 tx=0x7fd25000fe10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:50:34.256 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:34.254+0000 7fd24effd640 1 -- 172.21.15.123:0/731566901 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd250003b90 con 0x7fd2581e9a40 2024-09-16T15:50:34.256 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:34.254+0000 7fd24effd640 1 -- 172.21.15.123:0/731566901 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fd250003e20 con 0x7fd2581e9a40 2024-09-16T15:50:34.256 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:34.254+0000 7fd2605bb640 1 -- 172.21.15.123:0/731566901 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd2580c7a70 con 0x7fd2581e9a40 2024-09-16T15:50:34.256 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:34.254+0000 7fd24effd640 1 -- 172.21.15.123:0/731566901 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd250004d70 con 0x7fd2581e9a40 2024-09-16T15:50:34.256 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:34.254+0000 7fd2605bb640 1 -- 172.21.15.123:0/731566901 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd2580c8030 con 0x7fd2581e9a40 2024-09-16T15:50:34.257 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:34.254+0000 7fd2605bb640 1 -- 172.21.15.123:0/731566901 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd2580d1da0 con 0x7fd2581e9a40 2024-09-16T15:50:34.257 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:34.258+0000 7fd24effd640 1 -- 172.21.15.123:0/731566901 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 22) ==== 97692+0+0 (secure 0 0 0) 0x7fd25001b020 con 0x7fd2581e9a40 2024-09-16T15:50:34.257 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:34.258+0000 7fd24effd640 1 --2- 172.21.15.123:0/731566901 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fd23c0735b0 0x7fd23c075a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:50:34.257 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:34.258+0000 7fd25e357640 1 --2- 172.21.15.123:0/731566901 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fd23c0735b0 0x7fd23c075a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:50:34.257 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:34.258+0000 7fd24effd640 1 -- 172.21.15.123:0/731566901 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(146..146 src has 1..146) ==== 5737+0+0 (secure 0 0 0) 0x7fd2500947e0 con 0x7fd2581e9a40 2024-09-16T15:50:34.257 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:34.258+0000 7fd25e357640 1 --2- 172.21.15.123:0/731566901 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fd23c0735b0 0x7fd23c075a70 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7fd25806b850 tx=0x7fd254009040 comp rx=0 tx=0).ready entity=mgr.4150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:50:34.261 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:34.262+0000 7fd24effd640 1 -- 172.21.15.123:0/731566901 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7fd25005f070 con 0x7fd2581e9a40 2024-09-16T15:50:34.431 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:34.430+0000 7fd2605bb640 1 -- 172.21.15.123:0/731566901 --> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fd2580a1b30 con 0x7fd23c0735b0 2024-09-16T15:50:34.440 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:34.442+0000 7fd24effd640 1 -- 172.21.15.123:0/731566901 <== mgr.4150 v2:172.21.15.123:6836/2464759676 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+247115 (secure 0 0 0) 0x7fd2580a1b30 con 0x7fd23c0735b0 2024-09-16T15:50:34.443 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-09-16T15:50:34.444 INFO:teuthology.orchestra.run.smithi123.stderr:dumped all 2024-09-16T15:50:34.445 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:34.446+0000 7fd2605bb640 1 -- 172.21.15.123:0/731566901 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fd23c0735b0 msgr2=0x7fd23c075a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:50:34.445 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:34.446+0000 7fd2605bb640 1 --2- 172.21.15.123:0/731566901 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fd23c0735b0 0x7fd23c075a70 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7fd25806b850 tx=0x7fd254009040 comp rx=0 tx=0).stop 2024-09-16T15:50:34.445 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:34.446+0000 7fd2605bb640 1 -- 172.21.15.123:0/731566901 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd2581e9a40 msgr2=0x7fd2581e9ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:50:34.445 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:34.446+0000 7fd2605bb640 1 --2- 172.21.15.123:0/731566901 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd2581e9a40 0x7fd2581e9ec0 secure :-1 s=READY pgs=1591 cs=0 l=1 rev1=1 crypto rx=0x7fd2580caf00 tx=0x7fd25000fe10 comp rx=0 tx=0).stop 2024-09-16T15:50:34.445 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:34.446+0000 7fd2605bb640 1 -- 172.21.15.123:0/731566901 shutdown_connections 2024-09-16T15:50:34.446 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:34.446+0000 7fd2605bb640 1 --2- 172.21.15.123:0/731566901 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fd23c0735b0 0x7fd23c075a70 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:34.446 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:34.446+0000 7fd2605bb640 1 --2- 172.21.15.123:0/731566901 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd2581e9a40 0x7fd2581e9ec0 unknown :-1 s=CLOSED pgs=1591 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:34.446 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:34.446+0000 7fd2605bb640 1 --2- 172.21.15.123:0/731566901 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd2580c9cc0 0x7fd2580c8830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:34.446 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:34.446+0000 7fd2605bb640 1 --2- 172.21.15.123:0/731566901 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fd244004810 0x7fd2581555d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:34.446 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:34.446+0000 7fd2605bb640 1 -- 172.21.15.123:0/731566901 >> 172.21.15.123:0/731566901 conn(0x7fd2580d3ea0 msgr2=0x7fd2581608e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:50:34.446 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:34.446+0000 7fd2605bb640 1 -- 172.21.15.123:0/731566901 shutdown_connections 2024-09-16T15:50:34.446 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:34.446+0000 7fd2605bb640 1 -- 172.21.15.123:0/731566901 wait complete. 2024-09-16T15:50:34.461 INFO:teuthology.orchestra.run.smithi123.stdout:{"pg_ready":true,"pg_map":{"version":167,"stamp":"2024-09-16T15:50:34.119725+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79,"ondisk_log_size":79,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":39,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":268904,"kb_used_data":15224,"kb_used_omap":12,"kb_used_meta":253491,"kb_avail":754705816,"statfs":{"total":773094113280,"available":772818755584,"internally_reserved":0,"allocated":15589376,"data_stored":7545564,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":259575345},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":29,"apply_latency_ms":29,"commit_latency_ns":29000000,"apply_latency_ns":29000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.743992"},"pg_stats":[{"pgid":"23.3f","version":"0'0","reported_seq":1,"reported_epoch":144,"state":"creating+peering","last_fresh":"2024-09-16T15:50:29.396826+0000","last_change":"2024-09-16T15:50:29.396826+0000","last_active":"2024-09-16T15:50:29.373642+0000","last_peered":"2024-09-16T15:50:29.373642+0000","last_clean":"2024-09-16T15:50:29.373642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:50:29.396826+0000","last_undegraded":"2024-09-16T15:50:29.396826+0000","last_fullsized":"2024-09-16T15:50:29.396826+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_clean_scrub_stamp":"2024-09-16T15:50:29.373642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.3e","version":"0'0","reported_seq":1,"reported_epoch":144,"state":"creating+peering","last_fresh":"2024-09-16T15:50:29.405611+0000","last_change":"2024-09-16T15:50:29.405611+0000","last_active":"2024-09-16T15:50:29.373642+0000","last_peered":"2024-09-16T15:50:29.373642+0000","last_clean":"2024-09-16T15:50:29.373642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:50:29.405611+0000","last_undegraded":"2024-09-16T15:50:29.405611+0000","last_fullsized":"2024-09-16T15:50:29.405611+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_clean_scrub_stamp":"2024-09-16T15:50:29.373642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.3d","version":"0'0","reported_seq":1,"reported_epoch":144,"state":"creating+peering","last_fresh":"2024-09-16T15:50:29.414816+0000","last_change":"2024-09-16T15:50:29.414816+0000","last_active":"2024-09-16T15:50:29.373642+0000","last_peered":"2024-09-16T15:50:29.373642+0000","last_clean":"2024-09-16T15:50:29.373642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:50:29.414816+0000","last_undegraded":"2024-09-16T15:50:29.414816+0000","last_fullsized":"2024-09-16T15:50:29.414816+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_clean_scrub_stamp":"2024-09-16T15:50:29.373642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.3c","version":"0'0","reported_seq":1,"reported_epoch":144,"state":"creating+peering","last_fresh":"2024-09-16T15:50:29.400948+0000","last_change":"2024-09-16T15:50:29.400948+0000","last_active":"2024-09-16T15:50:29.373642+0000","last_peered":"2024-09-16T15:50:29.373642+0000","last_clean":"2024-09-16T15:50:29.373642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:50:29.400948+0000","last_undegraded":"2024-09-16T15:50:29.400948+0000","last_fullsized":"2024-09-16T15:50:29.400948+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_clean_scrub_stamp":"2024-09-16T15:50:29.373642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.3b","version":"0'0","reported_seq":1,"reported_epoch":144,"state":"creating+peering","last_fresh":"2024-09-16T15:50:29.408612+0000","last_change":"2024-09-16T15:50:29.408612+0000","last_active":"2024-09-16T15:50:29.373642+0000","last_peered":"2024-09-16T15:50:29.373642+0000","last_clean":"2024-09-16T15:50:29.373642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:50:29.408612+0000","last_undegraded":"2024-09-16T15:50:29.408612+0000","last_fullsized":"2024-09-16T15:50:29.408612+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_clean_scrub_stamp":"2024-09-16T15:50:29.373642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.3a","version":"0'0","reported_seq":1,"reported_epoch":144,"state":"creating+peering","last_fresh":"2024-09-16T15:50:29.399825+0000","last_change":"2024-09-16T15:50:29.399825+0000","last_active":"2024-09-16T15:50:29.373642+0000","last_peered":"2024-09-16T15:50:29.373642+0000","last_clean":"2024-09-16T15:50:29.373642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:50:29.399825+0000","last_undegraded":"2024-09-16T15:50:29.399825+0000","last_fullsized":"2024-09-16T15:50:29.399825+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_clean_scrub_stamp":"2024-09-16T15:50:29.373642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.39","version":"0'0","reported_seq":1,"reported_epoch":144,"state":"creating+peering","last_fresh":"2024-09-16T15:50:29.397580+0000","last_change":"2024-09-16T15:50:29.397580+0000","last_active":"2024-09-16T15:50:29.373642+0000","last_peered":"2024-09-16T15:50:29.373642+0000","last_clean":"2024-09-16T15:50:29.373642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:50:29.397580+0000","last_undegraded":"2024-09-16T15:50:29.397580+0000","last_fullsized":"2024-09-16T15:50:29.397580+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_clean_scrub_stamp":"2024-09-16T15:50:29.373642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"23.38","version":"0'0","reported_seq":1,"reported_epoch":144,"state":"creating+peering","last_fresh":"2024-09-16T15:50:29.409708+0000","last_change":"2024-09-16T15:50:29.409708+0000","last_active":"2024-09-16T15:50:29.373642+0000","last_peered":"2024-09-16T15:50:29.373642+0000","last_clean":"2024-09-16T15:50:29.373642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:50:29.409708+0000","last_undegraded":"2024-09-16T15:50:29.409708+0000","last_fullsized":"2024-09-16T15:50:29.409708+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_clean_scrub_stamp":"2024-09-16T15:50:29.373642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.37","version":"0'0","reported_seq":1,"reported_epoch":144,"state":"creating+peering","last_fresh":"2024-09-16T15:50:29.406549+0000","last_change":"2024-09-16T15:50:29.406549+0000","last_active":"2024-09-16T15:50:29.373642+0000","last_peered":"2024-09-16T15:50:29.373642+0000","last_clean":"2024-09-16T15:50:29.373642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:50:29.406549+0000","last_undegraded":"2024-09-16T15:50:29.406549+0000","last_fullsized":"2024-09-16T15:50:29.406549+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_clean_scrub_stamp":"2024-09-16T15:50:29.373642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.36","version":"0'0","reported_seq":1,"reported_epoch":144,"state":"creating+peering","last_fresh":"2024-09-16T15:50:29.404538+0000","last_change":"2024-09-16T15:50:29.404538+0000","last_active":"2024-09-16T15:50:29.373642+0000","last_peered":"2024-09-16T15:50:29.373642+0000","last_clean":"2024-09-16T15:50:29.373642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:50:29.404538+0000","last_undegraded":"2024-09-16T15:50:29.404538+0000","last_fullsized":"2024-09-16T15:50:29.404538+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_clean_scrub_stamp":"2024-09-16T15:50:29.373642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.35","version":"0'0","reported_seq":1,"reported_epoch":144,"state":"creating+peering","last_fresh":"2024-09-16T15:50:29.418503+0000","last_change":"2024-09-16T15:50:29.418503+0000","last_active":"2024-09-16T15:50:29.373642+0000","last_peered":"2024-09-16T15:50:29.373642+0000","last_clean":"2024-09-16T15:50:29.373642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:50:29.418503+0000","last_undegraded":"2024-09-16T15:50:29.418503+0000","last_fullsized":"2024-09-16T15:50:29.418503+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_clean_scrub_stamp":"2024-09-16T15:50:29.373642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.34","version":"0'0","reported_seq":1,"reported_epoch":144,"state":"creating+peering","last_fresh":"2024-09-16T15:50:29.398426+0000","last_change":"2024-09-16T15:50:29.398426+0000","last_active":"2024-09-16T15:50:29.373642+0000","last_peered":"2024-09-16T15:50:29.373642+0000","last_clean":"2024-09-16T15:50:29.373642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:50:29.398426+0000","last_undegraded":"2024-09-16T15:50:29.398426+0000","last_fullsized":"2024-09-16T15:50:29.398426+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_clean_scrub_stamp":"2024-09-16T15:50:29.373642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.33","version":"0'0","reported_seq":1,"reported_epoch":144,"state":"creating+peering","last_fresh":"2024-09-16T15:50:29.415630+0000","last_change":"2024-09-16T15:50:29.415630+0000","last_active":"2024-09-16T15:50:29.373642+0000","last_peered":"2024-09-16T15:50:29.373642+0000","last_clean":"2024-09-16T15:50:29.373642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:50:29.415630+0000","last_undegraded":"2024-09-16T15:50:29.415630+0000","last_fullsized":"2024-09-16T15:50:29.415630+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_clean_scrub_stamp":"2024-09-16T15:50:29.373642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.32","version":"0'0","reported_seq":1,"reported_epoch":144,"state":"creating+peering","last_fresh":"2024-09-16T15:50:29.411622+0000","last_change":"2024-09-16T15:50:29.411622+0000","last_active":"2024-09-16T15:50:29.373642+0000","last_peered":"2024-09-16T15:50:29.373642+0000","last_clean":"2024-09-16T15:50:29.373642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:50:29.411622+0000","last_undegraded":"2024-09-16T15:50:29.411622+0000","last_fullsized":"2024-09-16T15:50:29.411622+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_clean_scrub_stamp":"2024-09-16T15:50:29.373642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.31","version":"0'0","reported_seq":1,"reported_epoch":144,"state":"creating+peering","last_fresh":"2024-09-16T15:50:29.396532+0000","last_change":"2024-09-16T15:50:29.396532+0000","last_active":"2024-09-16T15:50:29.373642+0000","last_peered":"2024-09-16T15:50:29.373642+0000","last_clean":"2024-09-16T15:50:29.373642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:50:29.396532+0000","last_undegraded":"2024-09-16T15:50:29.396532+0000","last_fullsized":"2024-09-16T15:50:29.396532+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_clean_scrub_stamp":"2024-09-16T15:50:29.373642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.30","version":"0'0","reported_seq":1,"reported_epoch":144,"state":"creating+peering","last_fresh":"2024-09-16T15:50:29.398051+0000","last_change":"2024-09-16T15:50:29.398051+0000","last_active":"2024-09-16T15:50:29.373642+0000","last_peered":"2024-09-16T15:50:29.373642+0000","last_clean":"2024-09-16T15:50:29.373642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:50:29.398051+0000","last_undegraded":"2024-09-16T15:50:29.398051+0000","last_fullsized":"2024-09-16T15:50:29.398051+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_clean_scrub_stamp":"2024-09-16T15:50:29.373642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"23.2f","version":"0'0","reported_seq":1,"reported_epoch":144,"state":"creating+peering","last_fresh":"2024-09-16T15:50:29.399338+0000","last_change":"2024-09-16T15:50:29.399338+0000","last_active":"2024-09-16T15:50:29.373642+0000","last_peered":"2024-09-16T15:50:29.373642+0000","last_clean":"2024-09-16T15:50:29.373642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:50:29.399338+0000","last_undegraded":"2024-09-16T15:50:29.399338+0000","last_fullsized":"2024-09-16T15:50:29.399338+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_clean_scrub_stamp":"2024-09-16T15:50:29.373642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.2e","version":"0'0","reported_seq":1,"reported_epoch":144,"state":"creating+peering","last_fresh":"2024-09-16T15:50:29.403155+0000","last_change":"2024-09-16T15:50:29.403155+0000","last_active":"2024-09-16T15:50:29.373642+0000","last_peered":"2024-09-16T15:50:29.373642+0000","last_clean":"2024-09-16T15:50:29.373642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:50:29.403155+0000","last_undegraded":"2024-09-16T15:50:29.403155+0000","last_fullsized":"2024-09-16T15:50:29.403155+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_clean_scrub_stamp":"2024-09-16T15:50:29.373642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"23.2d","version":"0'0","reported_seq":1,"reported_epoch":144,"state":"creating+peering","last_fresh":"2024-09-16T15:50:29.399089+0000","last_change":"2024-09-16T15:50:29.399089+0000","last_active":"2024-09-16T15:50:29.373642+0000","last_peered":"2024-09-16T15:50:29.373642+0000","last_clean":"2024-09-16T15:50:29.373642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:50:29.399089+0000","last_undegraded":"2024-09-16T15:50:29.399089+0000","last_fullsized":"2024-09-16T15:50:29.399089+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_clean_scrub_stamp":"2024-09-16T15:50:29.373642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.2c","version":"0'0","reported_seq":1,"reported_epoch":144,"state":"creating+peering","last_fresh":"2024-09-16T15:50:29.400996+0000","last_change":"2024-09-16T15:50:29.400996+0000","last_active":"2024-09-16T15:50:29.373642+0000","last_peered":"2024-09-16T15:50:29.373642+0000","last_clean":"2024-09-16T15:50:29.373642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:50:29.400996+0000","last_undegraded":"2024-09-16T15:50:29.400996+0000","last_fullsized":"2024-09-16T15:50:29.400996+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_clean_scrub_stamp":"2024-09-16T15:50:29.373642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.2b","version":"0'0","reported_seq":1,"reported_epoch":144,"state":"creating+peering","last_fresh":"2024-09-16T15:50:29.395481+0000","last_change":"2024-09-16T15:50:29.395481+0000","last_active":"2024-09-16T15:50:29.373642+0000","last_peered":"2024-09-16T15:50:29.373642+0000","last_clean":"2024-09-16T15:50:29.373642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:50:29.395481+0000","last_undegraded":"2024-09-16T15:50:29.395481+0000","last_fullsized":"2024-09-16T15:50:29.395481+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_clean_scrub_stamp":"2024-09-16T15:50:29.373642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.2a","version":"0'0","reported_seq":1,"reported_epoch":144,"state":"creating+peering","last_fresh":"2024-09-16T15:50:29.396706+0000","last_change":"2024-09-16T15:50:29.396706+0000","last_active":"2024-09-16T15:50:29.373642+0000","last_peered":"2024-09-16T15:50:29.373642+0000","last_clean":"2024-09-16T15:50:29.373642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:50:29.396706+0000","last_undegraded":"2024-09-16T15:50:29.396706+0000","last_fullsized":"2024-09-16T15:50:29.396706+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_clean_scrub_stamp":"2024-09-16T15:50:29.373642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.29","version":"0'0","reported_seq":1,"reported_epoch":144,"state":"creating+peering","last_fresh":"2024-09-16T15:50:29.397404+0000","last_change":"2024-09-16T15:50:29.397404+0000","last_active":"2024-09-16T15:50:29.373642+0000","last_peered":"2024-09-16T15:50:29.373642+0000","last_clean":"2024-09-16T15:50:29.373642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:50:29.397404+0000","last_undegraded":"2024-09-16T15:50:29.397404+0000","last_fullsized":"2024-09-16T15:50:29.397404+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_clean_scrub_stamp":"2024-09-16T15:50:29.373642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.28","version":"0'0","reported_seq":1,"reported_epoch":144,"state":"creating+peering","last_fresh":"2024-09-16T15:50:29.396706+0000","last_change":"2024-09-16T15:50:29.396706+0000","last_active":"2024-09-16T15:50:29.373642+0000","last_peered":"2024-09-16T15:50:29.373642+0000","last_clean":"2024-09-16T15:50:29.373642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:50:29.396706+0000","last_undegraded":"2024-09-16T15:50:29.396706+0000","last_fullsized":"2024-09-16T15:50:29.396706+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_clean_scrub_stamp":"2024-09-16T15:50:29.373642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.27","version":"0'0","reported_seq":1,"reported_epoch":144,"state":"creating+peering","last_fresh":"2024-09-16T15:50:29.401802+0000","last_change":"2024-09-16T15:50:29.401802+0000","last_active":"2024-09-16T15:50:29.373642+0000","last_peered":"2024-09-16T15:50:29.373642+0000","last_clean":"2024-09-16T15:50:29.373642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:50:29.401802+0000","last_undegraded":"2024-09-16T15:50:29.401802+0000","last_fullsized":"2024-09-16T15:50:29.401802+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_clean_scrub_stamp":"2024-09-16T15:50:29.373642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.26","version":"0'0","reported_seq":1,"reported_epoch":144,"state":"creating+peering","last_fresh":"2024-09-16T15:50:29.397726+0000","last_change":"2024-09-16T15:50:29.397726+0000","last_active":"2024-09-16T15:50:29.373642+0000","last_peered":"2024-09-16T15:50:29.373642+0000","last_clean":"2024-09-16T15:50:29.373642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:50:29.397726+0000","last_undegraded":"2024-09-16T15:50:29.397726+0000","last_fullsized":"2024-09-16T15:50:29.397726+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_clean_scrub_stamp":"2024-09-16T15:50:29.373642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"23.25","version":"0'0","reported_seq":1,"reported_epoch":144,"state":"creating+peering","last_fresh":"2024-09-16T15:50:29.413671+0000","last_change":"2024-09-16T15:50:29.413671+0000","last_active":"2024-09-16T15:50:29.373642+0000","last_peered":"2024-09-16T15:50:29.373642+0000","last_clean":"2024-09-16T15:50:29.373642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:50:29.413671+0000","last_undegraded":"2024-09-16T15:50:29.413671+0000","last_fullsized":"2024-09-16T15:50:29.413671+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_clean_scrub_stamp":"2024-09-16T15:50:29.373642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.24","version":"0'0","reported_seq":1,"reported_epoch":144,"state":"creating+peering","last_fresh":"2024-09-16T15:50:29.400095+0000","last_change":"2024-09-16T15:50:29.400095+0000","last_active":"2024-09-16T15:50:29.373642+0000","last_peered":"2024-09-16T15:50:29.373642+0000","last_clean":"2024-09-16T15:50:29.373642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:50:29.400095+0000","last_undegraded":"2024-09-16T15:50:29.400095+0000","last_fullsized":"2024-09-16T15:50:29.400095+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_clean_scrub_stamp":"2024-09-16T15:50:29.373642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.23","version":"0'0","reported_seq":1,"reported_epoch":144,"state":"creating+peering","last_fresh":"2024-09-16T15:50:29.403015+0000","last_change":"2024-09-16T15:50:29.403015+0000","last_active":"2024-09-16T15:50:29.373642+0000","last_peered":"2024-09-16T15:50:29.373642+0000","last_clean":"2024-09-16T15:50:29.373642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:50:29.403015+0000","last_undegraded":"2024-09-16T15:50:29.403015+0000","last_fullsized":"2024-09-16T15:50:29.403015+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_clean_scrub_stamp":"2024-09-16T15:50:29.373642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.22","version":"0'0","reported_seq":1,"reported_epoch":144,"state":"creating+peering","last_fresh":"2024-09-16T15:50:29.390228+0000","last_change":"2024-09-16T15:50:29.390228+0000","last_active":"2024-09-16T15:50:29.373642+0000","last_peered":"2024-09-16T15:50:29.373642+0000","last_clean":"2024-09-16T15:50:29.373642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:50:29.390228+0000","last_undegraded":"2024-09-16T15:50:29.390228+0000","last_fullsized":"2024-09-16T15:50:29.390228+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_clean_scrub_stamp":"2024-09-16T15:50:29.373642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"23.21","version":"0'0","reported_seq":1,"reported_epoch":144,"state":"creating+peering","last_fresh":"2024-09-16T15:50:29.392793+0000","last_change":"2024-09-16T15:50:29.392793+0000","last_active":"2024-09-16T15:50:29.373642+0000","last_peered":"2024-09-16T15:50:29.373642+0000","last_clean":"2024-09-16T15:50:29.373642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:50:29.392793+0000","last_undegraded":"2024-09-16T15:50:29.392793+0000","last_fullsized":"2024-09-16T15:50:29.392793+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_clean_scrub_stamp":"2024-09-16T15:50:29.373642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"23.20","version":"0'0","reported_seq":1,"reported_epoch":144,"state":"creating+peering","last_fresh":"2024-09-16T15:50:29.400381+0000","last_change":"2024-09-16T15:50:29.400381+0000","last_active":"2024-09-16T15:50:29.373642+0000","last_peered":"2024-09-16T15:50:29.373642+0000","last_clean":"2024-09-16T15:50:29.373642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:50:29.400381+0000","last_undegraded":"2024-09-16T15:50:29.400381+0000","last_fullsized":"2024-09-16T15:50:29.400381+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_clean_scrub_stamp":"2024-09-16T15:50:29.373642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"23.1e","version":"0'0","reported_seq":1,"reported_epoch":144,"state":"creating+peering","last_fresh":"2024-09-16T15:50:29.396097+0000","last_change":"2024-09-16T15:50:29.396097+0000","last_active":"2024-09-16T15:50:29.373642+0000","last_peered":"2024-09-16T15:50:29.373642+0000","last_clean":"2024-09-16T15:50:29.373642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:50:29.396097+0000","last_undegraded":"2024-09-16T15:50:29.396097+0000","last_fullsized":"2024-09-16T15:50:29.396097+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_clean_scrub_stamp":"2024-09-16T15:50:29.373642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"22.1f","version":"0'0","reported_seq":10,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-16T15:50:29.430449+0000","last_change":"2024-09-16T15:50:29.428405+0000","last_active":"2024-09-16T15:50:29.430449+0000","last_peered":"2024-09-16T15:50:29.430449+0000","last_clean":"2024-09-16T15:50:29.430449+0000","last_became_active":"2024-09-16T15:50:29.423819+0000","last_became_peered":"2024-09-16T15:50:29.423819+0000","last_unstale":"2024-09-16T15:50:29.430449+0000","last_undegraded":"2024-09-16T15:50:29.430449+0000","last_fullsized":"2024-09-16T15:50:29.430449+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:28.359944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:28.359944+0000","last_clean_scrub_stamp":"2024-09-16T15:50:28.359944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T22:06:16.541534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.1f","version":"0'0","reported_seq":1,"reported_epoch":144,"state":"creating+peering","last_fresh":"2024-09-16T15:50:29.395019+0000","last_change":"2024-09-16T15:50:29.395019+0000","last_active":"2024-09-16T15:50:29.373642+0000","last_peered":"2024-09-16T15:50:29.373642+0000","last_clean":"2024-09-16T15:50:29.373642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:50:29.395019+0000","last_undegraded":"2024-09-16T15:50:29.395019+0000","last_fullsized":"2024-09-16T15:50:29.395019+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_clean_scrub_stamp":"2024-09-16T15:50:29.373642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.1e","version":"0'0","reported_seq":10,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-16T15:50:29.433442+0000","last_change":"2024-09-16T15:50:29.431326+0000","last_active":"2024-09-16T15:50:29.433442+0000","last_peered":"2024-09-16T15:50:29.433442+0000","last_clean":"2024-09-16T15:50:29.433442+0000","last_became_active":"2024-09-16T15:50:29.430610+0000","last_became_peered":"2024-09-16T15:50:29.430610+0000","last_unstale":"2024-09-16T15:50:29.433442+0000","last_undegraded":"2024-09-16T15:50:29.433442+0000","last_fullsized":"2024-09-16T15:50:29.433442+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:28.359944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:28.359944+0000","last_clean_scrub_stamp":"2024-09-16T15:50:28.359944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T00:28:56.379780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"23.1c","version":"0'0","reported_seq":1,"reported_epoch":144,"state":"creating+peering","last_fresh":"2024-09-16T15:50:29.395809+0000","last_change":"2024-09-16T15:50:29.395809+0000","last_active":"2024-09-16T15:50:29.373642+0000","last_peered":"2024-09-16T15:50:29.373642+0000","last_clean":"2024-09-16T15:50:29.373642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:50:29.395809+0000","last_undegraded":"2024-09-16T15:50:29.395809+0000","last_fullsized":"2024-09-16T15:50:29.395809+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_clean_scrub_stamp":"2024-09-16T15:50:29.373642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.1d","version":"0'0","reported_seq":10,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-16T15:50:29.431196+0000","last_change":"2024-09-16T15:50:29.425771+0000","last_active":"2024-09-16T15:50:29.431196+0000","last_peered":"2024-09-16T15:50:29.431196+0000","last_clean":"2024-09-16T15:50:29.431196+0000","last_became_active":"2024-09-16T15:50:29.425507+0000","last_became_peered":"2024-09-16T15:50:29.425507+0000","last_unstale":"2024-09-16T15:50:29.431196+0000","last_undegraded":"2024-09-16T15:50:29.431196+0000","last_fullsized":"2024-09-16T15:50:29.431196+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:28.359944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:28.359944+0000","last_clean_scrub_stamp":"2024-09-16T15:50:28.359944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:16:32.881957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.1d","version":"0'0","reported_seq":1,"reported_epoch":144,"state":"creating+peering","last_fresh":"2024-09-16T15:50:29.412090+0000","last_change":"2024-09-16T15:50:29.412090+0000","last_active":"2024-09-16T15:50:29.373642+0000","last_peered":"2024-09-16T15:50:29.373642+0000","last_clean":"2024-09-16T15:50:29.373642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:50:29.412090+0000","last_undegraded":"2024-09-16T15:50:29.412090+0000","last_fullsized":"2024-09-16T15:50:29.412090+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_clean_scrub_stamp":"2024-09-16T15:50:29.373642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.1c","version":"0'0","reported_seq":10,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-16T15:50:29.411810+0000","last_change":"2024-09-16T15:50:29.408868+0000","last_active":"2024-09-16T15:50:29.411810+0000","last_peered":"2024-09-16T15:50:29.411810+0000","last_clean":"2024-09-16T15:50:29.411810+0000","last_became_active":"2024-09-16T15:50:29.408550+0000","last_became_peered":"2024-09-16T15:50:29.408550+0000","last_unstale":"2024-09-16T15:50:29.411810+0000","last_undegraded":"2024-09-16T15:50:29.411810+0000","last_fullsized":"2024-09-16T15:50:29.411810+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:28.359944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:28.359944+0000","last_clean_scrub_stamp":"2024-09-16T15:50:28.359944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:57:02.247484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.1a","version":"0'0","reported_seq":1,"reported_epoch":144,"state":"creating+peering","last_fresh":"2024-09-16T15:50:29.396181+0000","last_change":"2024-09-16T15:50:29.396181+0000","last_active":"2024-09-16T15:50:29.373642+0000","last_peered":"2024-09-16T15:50:29.373642+0000","last_clean":"2024-09-16T15:50:29.373642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:50:29.396181+0000","last_undegraded":"2024-09-16T15:50:29.396181+0000","last_fullsized":"2024-09-16T15:50:29.396181+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_clean_scrub_stamp":"2024-09-16T15:50:29.373642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"22.1b","version":"0'0","reported_seq":10,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-16T15:50:29.433993+0000","last_change":"2024-09-16T15:50:29.429045+0000","last_active":"2024-09-16T15:50:29.433993+0000","last_peered":"2024-09-16T15:50:29.433993+0000","last_clean":"2024-09-16T15:50:29.433993+0000","last_became_active":"2024-09-16T15:50:29.428317+0000","last_became_peered":"2024-09-16T15:50:29.428317+0000","last_unstale":"2024-09-16T15:50:29.433993+0000","last_undegraded":"2024-09-16T15:50:29.433993+0000","last_fullsized":"2024-09-16T15:50:29.433993+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:28.359944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:28.359944+0000","last_clean_scrub_stamp":"2024-09-16T15:50:28.359944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:54:05.773197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.1b","version":"0'0","reported_seq":1,"reported_epoch":144,"state":"creating+peering","last_fresh":"2024-09-16T15:50:29.396991+0000","last_change":"2024-09-16T15:50:29.396991+0000","last_active":"2024-09-16T15:50:29.373642+0000","last_peered":"2024-09-16T15:50:29.373642+0000","last_clean":"2024-09-16T15:50:29.373642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:50:29.396991+0000","last_undegraded":"2024-09-16T15:50:29.396991+0000","last_fullsized":"2024-09-16T15:50:29.396991+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_clean_scrub_stamp":"2024-09-16T15:50:29.373642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.1a","version":"0'0","reported_seq":10,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-16T15:50:29.411180+0000","last_change":"2024-09-16T15:50:29.408149+0000","last_active":"2024-09-16T15:50:29.411180+0000","last_peered":"2024-09-16T15:50:29.411180+0000","last_clean":"2024-09-16T15:50:29.411180+0000","last_became_active":"2024-09-16T15:50:29.407778+0000","last_became_peered":"2024-09-16T15:50:29.407778+0000","last_unstale":"2024-09-16T15:50:29.411180+0000","last_undegraded":"2024-09-16T15:50:29.411180+0000","last_fullsized":"2024-09-16T15:50:29.411180+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:28.359944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:28.359944+0000","last_clean_scrub_stamp":"2024-09-16T15:50:28.359944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:02:10.632687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.18","version":"0'0","reported_seq":1,"reported_epoch":144,"state":"creating+peering","last_fresh":"2024-09-16T15:50:29.397299+0000","last_change":"2024-09-16T15:50:29.397299+0000","last_active":"2024-09-16T15:50:29.373642+0000","last_peered":"2024-09-16T15:50:29.373642+0000","last_clean":"2024-09-16T15:50:29.373642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:50:29.397299+0000","last_undegraded":"2024-09-16T15:50:29.397299+0000","last_fullsized":"2024-09-16T15:50:29.397299+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_clean_scrub_stamp":"2024-09-16T15:50:29.373642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.19","version":"0'0","reported_seq":10,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-16T15:50:29.433714+0000","last_change":"2024-09-16T15:50:29.427582+0000","last_active":"2024-09-16T15:50:29.433714+0000","last_peered":"2024-09-16T15:50:29.433714+0000","last_clean":"2024-09-16T15:50:29.433714+0000","last_became_active":"2024-09-16T15:50:29.426124+0000","last_became_peered":"2024-09-16T15:50:29.426124+0000","last_unstale":"2024-09-16T15:50:29.433714+0000","last_undegraded":"2024-09-16T15:50:29.433714+0000","last_fullsized":"2024-09-16T15:50:29.433714+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:28.359944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:28.359944+0000","last_clean_scrub_stamp":"2024-09-16T15:50:28.359944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:01:27.892843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.19","version":"0'0","reported_seq":1,"reported_epoch":144,"state":"creating+peering","last_fresh":"2024-09-16T15:50:29.396885+0000","last_change":"2024-09-16T15:50:29.396885+0000","last_active":"2024-09-16T15:50:29.373642+0000","last_peered":"2024-09-16T15:50:29.373642+0000","last_clean":"2024-09-16T15:50:29.373642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:50:29.396885+0000","last_undegraded":"2024-09-16T15:50:29.396885+0000","last_fullsized":"2024-09-16T15:50:29.396885+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_clean_scrub_stamp":"2024-09-16T15:50:29.373642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.18","version":"0'0","reported_seq":10,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-16T15:50:29.409060+0000","last_change":"2024-09-16T15:50:29.406218+0000","last_active":"2024-09-16T15:50:29.409060+0000","last_peered":"2024-09-16T15:50:29.409060+0000","last_clean":"2024-09-16T15:50:29.409060+0000","last_became_active":"2024-09-16T15:50:29.405872+0000","last_became_peered":"2024-09-16T15:50:29.405872+0000","last_unstale":"2024-09-16T15:50:29.409060+0000","last_undegraded":"2024-09-16T15:50:29.409060+0000","last_fullsized":"2024-09-16T15:50:29.409060+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:28.359944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:28.359944+0000","last_clean_scrub_stamp":"2024-09-16T15:50:28.359944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T22:00:23.514722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"23.17","version":"0'0","reported_seq":1,"reported_epoch":144,"state":"creating+peering","last_fresh":"2024-09-16T15:50:29.403320+0000","last_change":"2024-09-16T15:50:29.403320+0000","last_active":"2024-09-16T15:50:29.373642+0000","last_peered":"2024-09-16T15:50:29.373642+0000","last_clean":"2024-09-16T15:50:29.373642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:50:29.403320+0000","last_undegraded":"2024-09-16T15:50:29.403320+0000","last_fullsized":"2024-09-16T15:50:29.403320+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_clean_scrub_stamp":"2024-09-16T15:50:29.373642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.16","version":"0'0","reported_seq":10,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-16T15:50:29.419433+0000","last_change":"2024-09-16T15:50:29.418819+0000","last_active":"2024-09-16T15:50:29.419433+0000","last_peered":"2024-09-16T15:50:29.419433+0000","last_clean":"2024-09-16T15:50:29.419433+0000","last_became_active":"2024-09-16T15:50:29.418365+0000","last_became_peered":"2024-09-16T15:50:29.418365+0000","last_unstale":"2024-09-16T15:50:29.419433+0000","last_undegraded":"2024-09-16T15:50:29.419433+0000","last_fullsized":"2024-09-16T15:50:29.419433+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:28.359944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:28.359944+0000","last_clean_scrub_stamp":"2024-09-16T15:50:28.359944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T23:56:48.360283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.14","version":"0'0","reported_seq":1,"reported_epoch":144,"state":"creating+peering","last_fresh":"2024-09-16T15:50:29.401245+0000","last_change":"2024-09-16T15:50:29.401245+0000","last_active":"2024-09-16T15:50:29.373642+0000","last_peered":"2024-09-16T15:50:29.373642+0000","last_clean":"2024-09-16T15:50:29.373642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:50:29.401245+0000","last_undegraded":"2024-09-16T15:50:29.401245+0000","last_fullsized":"2024-09-16T15:50:29.401245+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_clean_scrub_stamp":"2024-09-16T15:50:29.373642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.15","version":"0'0","reported_seq":10,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-16T15:50:29.417183+0000","last_change":"2024-09-16T15:50:29.414494+0000","last_active":"2024-09-16T15:50:29.417183+0000","last_peered":"2024-09-16T15:50:29.417183+0000","last_clean":"2024-09-16T15:50:29.417183+0000","last_became_active":"2024-09-16T15:50:29.413903+0000","last_became_peered":"2024-09-16T15:50:29.413903+0000","last_unstale":"2024-09-16T15:50:29.417183+0000","last_undegraded":"2024-09-16T15:50:29.417183+0000","last_fullsized":"2024-09-16T15:50:29.417183+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:28.359944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:28.359944+0000","last_clean_scrub_stamp":"2024-09-16T15:50:28.359944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T00:54:21.344614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.15","version":"0'0","reported_seq":1,"reported_epoch":144,"state":"creating+peering","last_fresh":"2024-09-16T15:50:29.390531+0000","last_change":"2024-09-16T15:50:29.390531+0000","last_active":"2024-09-16T15:50:29.373642+0000","last_peered":"2024-09-16T15:50:29.373642+0000","last_clean":"2024-09-16T15:50:29.373642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:50:29.390531+0000","last_undegraded":"2024-09-16T15:50:29.390531+0000","last_fullsized":"2024-09-16T15:50:29.390531+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_clean_scrub_stamp":"2024-09-16T15:50:29.373642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.14","version":"0'0","reported_seq":10,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-16T15:50:29.420922+0000","last_change":"2024-09-16T15:50:29.406502+0000","last_active":"2024-09-16T15:50:29.420922+0000","last_peered":"2024-09-16T15:50:29.420922+0000","last_clean":"2024-09-16T15:50:29.420922+0000","last_became_active":"2024-09-16T15:50:29.404490+0000","last_became_peered":"2024-09-16T15:50:29.404490+0000","last_unstale":"2024-09-16T15:50:29.420922+0000","last_undegraded":"2024-09-16T15:50:29.420922+0000","last_fullsized":"2024-09-16T15:50:29.420922+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:28.359944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:28.359944+0000","last_clean_scrub_stamp":"2024-09-16T15:50:28.359944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T03:40:13.610653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"23.12","version":"0'0","reported_seq":1,"reported_epoch":144,"state":"creating+peering","last_fresh":"2024-09-16T15:50:29.401262+0000","last_change":"2024-09-16T15:50:29.401262+0000","last_active":"2024-09-16T15:50:29.373642+0000","last_peered":"2024-09-16T15:50:29.373642+0000","last_clean":"2024-09-16T15:50:29.373642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:50:29.401262+0000","last_undegraded":"2024-09-16T15:50:29.401262+0000","last_fullsized":"2024-09-16T15:50:29.401262+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_clean_scrub_stamp":"2024-09-16T15:50:29.373642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.13","version":"0'0","reported_seq":10,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-16T15:50:29.411283+0000","last_change":"2024-09-16T15:50:29.408013+0000","last_active":"2024-09-16T15:50:29.411283+0000","last_peered":"2024-09-16T15:50:29.411283+0000","last_clean":"2024-09-16T15:50:29.411283+0000","last_became_active":"2024-09-16T15:50:29.407703+0000","last_became_peered":"2024-09-16T15:50:29.407703+0000","last_unstale":"2024-09-16T15:50:29.411283+0000","last_undegraded":"2024-09-16T15:50:29.411283+0000","last_fullsized":"2024-09-16T15:50:29.411283+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:28.359944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:28.359944+0000","last_clean_scrub_stamp":"2024-09-16T15:50:28.359944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:05:33.631752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.13","version":"0'0","reported_seq":1,"reported_epoch":144,"state":"creating+peering","last_fresh":"2024-09-16T15:50:29.396084+0000","last_change":"2024-09-16T15:50:29.396084+0000","last_active":"2024-09-16T15:50:29.373642+0000","last_peered":"2024-09-16T15:50:29.373642+0000","last_clean":"2024-09-16T15:50:29.373642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:50:29.396084+0000","last_undegraded":"2024-09-16T15:50:29.396084+0000","last_fullsized":"2024-09-16T15:50:29.396084+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_clean_scrub_stamp":"2024-09-16T15:50:29.373642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"22.12","version":"0'0","reported_seq":10,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-16T15:50:29.411692+0000","last_change":"2024-09-16T15:50:29.410142+0000","last_active":"2024-09-16T15:50:29.411692+0000","last_peered":"2024-09-16T15:50:29.411692+0000","last_clean":"2024-09-16T15:50:29.411692+0000","last_became_active":"2024-09-16T15:50:29.409779+0000","last_became_peered":"2024-09-16T15:50:29.409779+0000","last_unstale":"2024-09-16T15:50:29.411692+0000","last_undegraded":"2024-09-16T15:50:29.411692+0000","last_fullsized":"2024-09-16T15:50:29.411692+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:28.359944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:28.359944+0000","last_clean_scrub_stamp":"2024-09-16T15:50:28.359944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:51:34.842420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"23.10","version":"0'0","reported_seq":1,"reported_epoch":144,"state":"creating+peering","last_fresh":"2024-09-16T15:50:29.396732+0000","last_change":"2024-09-16T15:50:29.396732+0000","last_active":"2024-09-16T15:50:29.373642+0000","last_peered":"2024-09-16T15:50:29.373642+0000","last_clean":"2024-09-16T15:50:29.373642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:50:29.396732+0000","last_undegraded":"2024-09-16T15:50:29.396732+0000","last_fullsized":"2024-09-16T15:50:29.396732+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_clean_scrub_stamp":"2024-09-16T15:50:29.373642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.11","version":"0'0","reported_seq":10,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-16T15:50:29.431189+0000","last_change":"2024-09-16T15:50:29.425647+0000","last_active":"2024-09-16T15:50:29.431189+0000","last_peered":"2024-09-16T15:50:29.431189+0000","last_clean":"2024-09-16T15:50:29.431189+0000","last_became_active":"2024-09-16T15:50:29.425266+0000","last_became_peered":"2024-09-16T15:50:29.425266+0000","last_unstale":"2024-09-16T15:50:29.431189+0000","last_undegraded":"2024-09-16T15:50:29.431189+0000","last_fullsized":"2024-09-16T15:50:29.431189+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:28.359944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:28.359944+0000","last_clean_scrub_stamp":"2024-09-16T15:50:28.359944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:18:55.430269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.11","version":"0'0","reported_seq":1,"reported_epoch":144,"state":"creating+peering","last_fresh":"2024-09-16T15:50:29.418052+0000","last_change":"2024-09-16T15:50:29.418052+0000","last_active":"2024-09-16T15:50:29.373642+0000","last_peered":"2024-09-16T15:50:29.373642+0000","last_clean":"2024-09-16T15:50:29.373642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:50:29.418052+0000","last_undegraded":"2024-09-16T15:50:29.418052+0000","last_fullsized":"2024-09-16T15:50:29.418052+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_clean_scrub_stamp":"2024-09-16T15:50:29.373642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.10","version":"0'0","reported_seq":10,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-16T15:50:29.421423+0000","last_change":"2024-09-16T15:50:29.408571+0000","last_active":"2024-09-16T15:50:29.421423+0000","last_peered":"2024-09-16T15:50:29.421423+0000","last_clean":"2024-09-16T15:50:29.421423+0000","last_became_active":"2024-09-16T15:50:29.407840+0000","last_became_peered":"2024-09-16T15:50:29.407840+0000","last_unstale":"2024-09-16T15:50:29.421423+0000","last_undegraded":"2024-09-16T15:50:29.421423+0000","last_fullsized":"2024-09-16T15:50:29.421423+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:28.359944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:28.359944+0000","last_clean_scrub_stamp":"2024-09-16T15:50:28.359944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:00:52.523701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"23.e","version":"0'0","reported_seq":1,"reported_epoch":144,"state":"creating+peering","last_fresh":"2024-09-16T15:50:29.396821+0000","last_change":"2024-09-16T15:50:29.396821+0000","last_active":"2024-09-16T15:50:29.373642+0000","last_peered":"2024-09-16T15:50:29.373642+0000","last_clean":"2024-09-16T15:50:29.373642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:50:29.396821+0000","last_undegraded":"2024-09-16T15:50:29.396821+0000","last_fullsized":"2024-09-16T15:50:29.396821+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_clean_scrub_stamp":"2024-09-16T15:50:29.373642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.f","version":"0'0","reported_seq":10,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-16T15:50:29.435965+0000","last_change":"2024-09-16T15:50:29.434785+0000","last_active":"2024-09-16T15:50:29.435965+0000","last_peered":"2024-09-16T15:50:29.435965+0000","last_clean":"2024-09-16T15:50:29.435965+0000","last_became_active":"2024-09-16T15:50:29.434102+0000","last_became_peered":"2024-09-16T15:50:29.434102+0000","last_unstale":"2024-09-16T15:50:29.435965+0000","last_undegraded":"2024-09-16T15:50:29.435965+0000","last_fullsized":"2024-09-16T15:50:29.435965+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:28.359944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:28.359944+0000","last_clean_scrub_stamp":"2024-09-16T15:50:28.359944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T02:25:34.661941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.f","version":"0'0","reported_seq":1,"reported_epoch":144,"state":"creating+peering","last_fresh":"2024-09-16T15:50:29.402843+0000","last_change":"2024-09-16T15:50:29.402843+0000","last_active":"2024-09-16T15:50:29.373642+0000","last_peered":"2024-09-16T15:50:29.373642+0000","last_clean":"2024-09-16T15:50:29.373642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:50:29.402843+0000","last_undegraded":"2024-09-16T15:50:29.402843+0000","last_fullsized":"2024-09-16T15:50:29.402843+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_clean_scrub_stamp":"2024-09-16T15:50:29.373642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.e","version":"0'0","reported_seq":10,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-16T15:50:29.429265+0000","last_change":"2024-09-16T15:50:29.426483+0000","last_active":"2024-09-16T15:50:29.429265+0000","last_peered":"2024-09-16T15:50:29.429265+0000","last_clean":"2024-09-16T15:50:29.429265+0000","last_became_active":"2024-09-16T15:50:29.425712+0000","last_became_peered":"2024-09-16T15:50:29.425712+0000","last_unstale":"2024-09-16T15:50:29.429265+0000","last_undegraded":"2024-09-16T15:50:29.429265+0000","last_fullsized":"2024-09-16T15:50:29.429265+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:28.359944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:28.359944+0000","last_clean_scrub_stamp":"2024-09-16T15:50:28.359944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:31:59.513487+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.c","version":"0'0","reported_seq":1,"reported_epoch":144,"state":"creating+peering","last_fresh":"2024-09-16T15:50:29.399219+0000","last_change":"2024-09-16T15:50:29.399219+0000","last_active":"2024-09-16T15:50:29.373642+0000","last_peered":"2024-09-16T15:50:29.373642+0000","last_clean":"2024-09-16T15:50:29.373642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:50:29.399219+0000","last_undegraded":"2024-09-16T15:50:29.399219+0000","last_fullsized":"2024-09-16T15:50:29.399219+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_clean_scrub_stamp":"2024-09-16T15:50:29.373642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.d","version":"0'0","reported_seq":10,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-16T15:50:29.430249+0000","last_change":"2024-09-16T15:50:29.427670+0000","last_active":"2024-09-16T15:50:29.430249+0000","last_peered":"2024-09-16T15:50:29.430249+0000","last_clean":"2024-09-16T15:50:29.430249+0000","last_became_active":"2024-09-16T15:50:29.426920+0000","last_became_peered":"2024-09-16T15:50:29.426920+0000","last_unstale":"2024-09-16T15:50:29.430249+0000","last_undegraded":"2024-09-16T15:50:29.430249+0000","last_fullsized":"2024-09-16T15:50:29.430249+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:28.359944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:28.359944+0000","last_clean_scrub_stamp":"2024-09-16T15:50:28.359944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:53:40.560984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.d","version":"0'0","reported_seq":1,"reported_epoch":144,"state":"creating+peering","last_fresh":"2024-09-16T15:50:29.395185+0000","last_change":"2024-09-16T15:50:29.395185+0000","last_active":"2024-09-16T15:50:29.373642+0000","last_peered":"2024-09-16T15:50:29.373642+0000","last_clean":"2024-09-16T15:50:29.373642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:50:29.395185+0000","last_undegraded":"2024-09-16T15:50:29.395185+0000","last_fullsized":"2024-09-16T15:50:29.395185+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_clean_scrub_stamp":"2024-09-16T15:50:29.373642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"22.c","version":"0'0","reported_seq":10,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-16T15:50:29.430128+0000","last_change":"2024-09-16T15:50:29.427240+0000","last_active":"2024-09-16T15:50:29.430128+0000","last_peered":"2024-09-16T15:50:29.430128+0000","last_clean":"2024-09-16T15:50:29.430128+0000","last_became_active":"2024-09-16T15:50:29.423280+0000","last_became_peered":"2024-09-16T15:50:29.423280+0000","last_unstale":"2024-09-16T15:50:29.430128+0000","last_undegraded":"2024-09-16T15:50:29.430128+0000","last_fullsized":"2024-09-16T15:50:29.430128+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:28.359944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:28.359944+0000","last_clean_scrub_stamp":"2024-09-16T15:50:28.359944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:36:54.628281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.a","version":"0'0","reported_seq":1,"reported_epoch":144,"state":"creating+peering","last_fresh":"2024-09-16T15:50:29.400460+0000","last_change":"2024-09-16T15:50:29.400460+0000","last_active":"2024-09-16T15:50:29.373642+0000","last_peered":"2024-09-16T15:50:29.373642+0000","last_clean":"2024-09-16T15:50:29.373642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:50:29.400460+0000","last_undegraded":"2024-09-16T15:50:29.400460+0000","last_fullsized":"2024-09-16T15:50:29.400460+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_clean_scrub_stamp":"2024-09-16T15:50:29.373642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.b","version":"0'0","reported_seq":10,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-16T15:50:29.431651+0000","last_change":"2024-09-16T15:50:29.425798+0000","last_active":"2024-09-16T15:50:29.431651+0000","last_peered":"2024-09-16T15:50:29.431651+0000","last_clean":"2024-09-16T15:50:29.431651+0000","last_became_active":"2024-09-16T15:50:29.425459+0000","last_became_peered":"2024-09-16T15:50:29.425459+0000","last_unstale":"2024-09-16T15:50:29.431651+0000","last_undegraded":"2024-09-16T15:50:29.431651+0000","last_fullsized":"2024-09-16T15:50:29.431651+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:28.359944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:28.359944+0000","last_clean_scrub_stamp":"2024-09-16T15:50:28.359944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:59:55.828797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.b","version":"0'0","reported_seq":1,"reported_epoch":144,"state":"creating+peering","last_fresh":"2024-09-16T15:50:29.394330+0000","last_change":"2024-09-16T15:50:29.394330+0000","last_active":"2024-09-16T15:50:29.373642+0000","last_peered":"2024-09-16T15:50:29.373642+0000","last_clean":"2024-09-16T15:50:29.373642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:50:29.394330+0000","last_undegraded":"2024-09-16T15:50:29.394330+0000","last_fullsized":"2024-09-16T15:50:29.394330+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_clean_scrub_stamp":"2024-09-16T15:50:29.373642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"22.a","version":"0'0","reported_seq":10,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-16T15:50:29.429851+0000","last_change":"2024-09-16T15:50:29.426752+0000","last_active":"2024-09-16T15:50:29.429851+0000","last_peered":"2024-09-16T15:50:29.429851+0000","last_clean":"2024-09-16T15:50:29.429851+0000","last_became_active":"2024-09-16T15:50:29.422755+0000","last_became_peered":"2024-09-16T15:50:29.422755+0000","last_unstale":"2024-09-16T15:50:29.429851+0000","last_undegraded":"2024-09-16T15:50:29.429851+0000","last_fullsized":"2024-09-16T15:50:29.429851+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:28.359944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:28.359944+0000","last_clean_scrub_stamp":"2024-09-16T15:50:28.359944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T18:14:40.770591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.8","version":"0'0","reported_seq":1,"reported_epoch":144,"state":"creating+peering","last_fresh":"2024-09-16T15:50:29.397861+0000","last_change":"2024-09-16T15:50:29.397861+0000","last_active":"2024-09-16T15:50:29.373642+0000","last_peered":"2024-09-16T15:50:29.373642+0000","last_clean":"2024-09-16T15:50:29.373642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:50:29.397861+0000","last_undegraded":"2024-09-16T15:50:29.397861+0000","last_fullsized":"2024-09-16T15:50:29.397861+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_clean_scrub_stamp":"2024-09-16T15:50:29.373642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.9","version":"0'0","reported_seq":10,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-16T15:50:29.424612+0000","last_change":"2024-09-16T15:50:29.423325+0000","last_active":"2024-09-16T15:50:29.424612+0000","last_peered":"2024-09-16T15:50:29.424612+0000","last_clean":"2024-09-16T15:50:29.424612+0000","last_became_active":"2024-09-16T15:50:29.423053+0000","last_became_peered":"2024-09-16T15:50:29.423053+0000","last_unstale":"2024-09-16T15:50:29.424612+0000","last_undegraded":"2024-09-16T15:50:29.424612+0000","last_fullsized":"2024-09-16T15:50:29.424612+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:28.359944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:28.359944+0000","last_clean_scrub_stamp":"2024-09-16T15:50:28.359944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T22:05:53.341097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.9","version":"0'0","reported_seq":1,"reported_epoch":144,"state":"creating+peering","last_fresh":"2024-09-16T15:50:29.396384+0000","last_change":"2024-09-16T15:50:29.396384+0000","last_active":"2024-09-16T15:50:29.373642+0000","last_peered":"2024-09-16T15:50:29.373642+0000","last_clean":"2024-09-16T15:50:29.373642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:50:29.396384+0000","last_undegraded":"2024-09-16T15:50:29.396384+0000","last_fullsized":"2024-09-16T15:50:29.396384+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_clean_scrub_stamp":"2024-09-16T15:50:29.373642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.8","version":"0'0","reported_seq":10,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-16T15:50:29.421906+0000","last_change":"2024-09-16T15:50:29.418765+0000","last_active":"2024-09-16T15:50:29.421906+0000","last_peered":"2024-09-16T15:50:29.421906+0000","last_clean":"2024-09-16T15:50:29.421906+0000","last_became_active":"2024-09-16T15:50:29.417737+0000","last_became_peered":"2024-09-16T15:50:29.417737+0000","last_unstale":"2024-09-16T15:50:29.421906+0000","last_undegraded":"2024-09-16T15:50:29.421906+0000","last_fullsized":"2024-09-16T15:50:29.421906+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:28.359944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:28.359944+0000","last_clean_scrub_stamp":"2024-09-16T15:50:28.359944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T00:16:50.967923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.6","version":"0'0","reported_seq":1,"reported_epoch":144,"state":"creating+peering","last_fresh":"2024-09-16T15:50:29.400497+0000","last_change":"2024-09-16T15:50:29.400497+0000","last_active":"2024-09-16T15:50:29.373642+0000","last_peered":"2024-09-16T15:50:29.373642+0000","last_clean":"2024-09-16T15:50:29.373642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:50:29.400497+0000","last_undegraded":"2024-09-16T15:50:29.400497+0000","last_fullsized":"2024-09-16T15:50:29.400497+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_clean_scrub_stamp":"2024-09-16T15:50:29.373642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.7","version":"0'0","reported_seq":10,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-16T15:50:29.423515+0000","last_change":"2024-09-16T15:50:29.422588+0000","last_active":"2024-09-16T15:50:29.423515+0000","last_peered":"2024-09-16T15:50:29.423515+0000","last_clean":"2024-09-16T15:50:29.423515+0000","last_became_active":"2024-09-16T15:50:29.422240+0000","last_became_peered":"2024-09-16T15:50:29.422240+0000","last_unstale":"2024-09-16T15:50:29.423515+0000","last_undegraded":"2024-09-16T15:50:29.423515+0000","last_fullsized":"2024-09-16T15:50:29.423515+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:28.359944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:28.359944+0000","last_clean_scrub_stamp":"2024-09-16T15:50:28.359944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T01:48:32.528831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"23.7","version":"0'0","reported_seq":1,"reported_epoch":144,"state":"creating+peering","last_fresh":"2024-09-16T15:50:29.394083+0000","last_change":"2024-09-16T15:50:29.394083+0000","last_active":"2024-09-16T15:50:29.373642+0000","last_peered":"2024-09-16T15:50:29.373642+0000","last_clean":"2024-09-16T15:50:29.373642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:50:29.394083+0000","last_undegraded":"2024-09-16T15:50:29.394083+0000","last_fullsized":"2024-09-16T15:50:29.394083+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_clean_scrub_stamp":"2024-09-16T15:50:29.373642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"22.6","version":"0'0","reported_seq":10,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-16T15:50:29.428409+0000","last_change":"2024-09-16T15:50:29.424687+0000","last_active":"2024-09-16T15:50:29.428409+0000","last_peered":"2024-09-16T15:50:29.428409+0000","last_clean":"2024-09-16T15:50:29.428409+0000","last_became_active":"2024-09-16T15:50:29.423323+0000","last_became_peered":"2024-09-16T15:50:29.423323+0000","last_unstale":"2024-09-16T15:50:29.428409+0000","last_undegraded":"2024-09-16T15:50:29.428409+0000","last_fullsized":"2024-09-16T15:50:29.428409+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:28.359944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:28.359944+0000","last_clean_scrub_stamp":"2024-09-16T15:50:28.359944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:26:50.321081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.4","version":"0'0","reported_seq":1,"reported_epoch":144,"state":"creating+peering","last_fresh":"2024-09-16T15:50:29.395179+0000","last_change":"2024-09-16T15:50:29.395179+0000","last_active":"2024-09-16T15:50:29.373642+0000","last_peered":"2024-09-16T15:50:29.373642+0000","last_clean":"2024-09-16T15:50:29.373642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:50:29.395179+0000","last_undegraded":"2024-09-16T15:50:29.395179+0000","last_fullsized":"2024-09-16T15:50:29.395179+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_clean_scrub_stamp":"2024-09-16T15:50:29.373642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.5","version":"0'0","reported_seq":10,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-16T15:50:29.429727+0000","last_change":"2024-09-16T15:50:29.421716+0000","last_active":"2024-09-16T15:50:29.429727+0000","last_peered":"2024-09-16T15:50:29.429727+0000","last_clean":"2024-09-16T15:50:29.429727+0000","last_became_active":"2024-09-16T15:50:29.420529+0000","last_became_peered":"2024-09-16T15:50:29.420529+0000","last_unstale":"2024-09-16T15:50:29.429727+0000","last_undegraded":"2024-09-16T15:50:29.429727+0000","last_fullsized":"2024-09-16T15:50:29.429727+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:28.359944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:28.359944+0000","last_clean_scrub_stamp":"2024-09-16T15:50:28.359944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T03:46:32.264684+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.5","version":"0'0","reported_seq":1,"reported_epoch":144,"state":"creating+peering","last_fresh":"2024-09-16T15:50:29.400308+0000","last_change":"2024-09-16T15:50:29.400308+0000","last_active":"2024-09-16T15:50:29.373642+0000","last_peered":"2024-09-16T15:50:29.373642+0000","last_clean":"2024-09-16T15:50:29.373642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:50:29.400308+0000","last_undegraded":"2024-09-16T15:50:29.400308+0000","last_fullsized":"2024-09-16T15:50:29.400308+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_clean_scrub_stamp":"2024-09-16T15:50:29.373642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.4","version":"0'0","reported_seq":10,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-16T15:50:29.417851+0000","last_change":"2024-09-16T15:50:29.417210+0000","last_active":"2024-09-16T15:50:29.417851+0000","last_peered":"2024-09-16T15:50:29.417851+0000","last_clean":"2024-09-16T15:50:29.417851+0000","last_became_active":"2024-09-16T15:50:29.416831+0000","last_became_peered":"2024-09-16T15:50:29.416831+0000","last_unstale":"2024-09-16T15:50:29.417851+0000","last_undegraded":"2024-09-16T15:50:29.417851+0000","last_fullsized":"2024-09-16T15:50:29.417851+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:28.359944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:28.359944+0000","last_clean_scrub_stamp":"2024-09-16T15:50:28.359944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:00:59.918701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"23.2","version":"0'0","reported_seq":1,"reported_epoch":144,"state":"creating+peering","last_fresh":"2024-09-16T15:50:29.400856+0000","last_change":"2024-09-16T15:50:29.400856+0000","last_active":"2024-09-16T15:50:29.373642+0000","last_peered":"2024-09-16T15:50:29.373642+0000","last_clean":"2024-09-16T15:50:29.373642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:50:29.400856+0000","last_undegraded":"2024-09-16T15:50:29.400856+0000","last_fullsized":"2024-09-16T15:50:29.400856+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_clean_scrub_stamp":"2024-09-16T15:50:29.373642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.3","version":"0'0","reported_seq":10,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-16T15:50:29.421008+0000","last_change":"2024-09-16T15:50:29.407104+0000","last_active":"2024-09-16T15:50:29.421008+0000","last_peered":"2024-09-16T15:50:29.421008+0000","last_clean":"2024-09-16T15:50:29.421008+0000","last_became_active":"2024-09-16T15:50:29.406182+0000","last_became_peered":"2024-09-16T15:50:29.406182+0000","last_unstale":"2024-09-16T15:50:29.421008+0000","last_undegraded":"2024-09-16T15:50:29.421008+0000","last_fullsized":"2024-09-16T15:50:29.421008+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:28.359944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:28.359944+0000","last_clean_scrub_stamp":"2024-09-16T15:50:28.359944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:36:55.485971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"23.3","version":"0'0","reported_seq":1,"reported_epoch":144,"state":"creating+peering","last_fresh":"2024-09-16T15:50:29.391321+0000","last_change":"2024-09-16T15:50:29.391321+0000","last_active":"2024-09-16T15:50:29.373642+0000","last_peered":"2024-09-16T15:50:29.373642+0000","last_clean":"2024-09-16T15:50:29.373642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:50:29.391321+0000","last_undegraded":"2024-09-16T15:50:29.391321+0000","last_fullsized":"2024-09-16T15:50:29.391321+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_clean_scrub_stamp":"2024-09-16T15:50:29.373642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.2","version":"0'0","reported_seq":10,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-16T15:50:29.411842+0000","last_change":"2024-09-16T15:50:29.410821+0000","last_active":"2024-09-16T15:50:29.411842+0000","last_peered":"2024-09-16T15:50:29.411842+0000","last_clean":"2024-09-16T15:50:29.411842+0000","last_became_active":"2024-09-16T15:50:29.410554+0000","last_became_peered":"2024-09-16T15:50:29.410554+0000","last_unstale":"2024-09-16T15:50:29.411842+0000","last_undegraded":"2024-09-16T15:50:29.411842+0000","last_fullsized":"2024-09-16T15:50:29.411842+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:28.359944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:28.359944+0000","last_clean_scrub_stamp":"2024-09-16T15:50:28.359944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:55:15.802642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.0","version":"0'0","reported_seq":1,"reported_epoch":144,"state":"creating+peering","last_fresh":"2024-09-16T15:50:29.394199+0000","last_change":"2024-09-16T15:50:29.394199+0000","last_active":"2024-09-16T15:50:29.373642+0000","last_peered":"2024-09-16T15:50:29.373642+0000","last_clean":"2024-09-16T15:50:29.373642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:50:29.394199+0000","last_undegraded":"2024-09-16T15:50:29.394199+0000","last_fullsized":"2024-09-16T15:50:29.394199+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_clean_scrub_stamp":"2024-09-16T15:50:29.373642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"22.1","version":"0'0","reported_seq":10,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-16T15:50:29.421042+0000","last_change":"2024-09-16T15:50:29.416751+0000","last_active":"2024-09-16T15:50:29.421042+0000","last_peered":"2024-09-16T15:50:29.421042+0000","last_clean":"2024-09-16T15:50:29.421042+0000","last_became_active":"2024-09-16T15:50:29.415995+0000","last_became_peered":"2024-09-16T15:50:29.415995+0000","last_unstale":"2024-09-16T15:50:29.421042+0000","last_undegraded":"2024-09-16T15:50:29.421042+0000","last_fullsized":"2024-09-16T15:50:29.421042+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:28.359944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:28.359944+0000","last_clean_scrub_stamp":"2024-09-16T15:50:28.359944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T02:16:57.915820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.1","version":"0'0","reported_seq":1,"reported_epoch":144,"state":"creating+peering","last_fresh":"2024-09-16T15:50:29.409753+0000","last_change":"2024-09-16T15:50:29.409753+0000","last_active":"2024-09-16T15:50:29.373642+0000","last_peered":"2024-09-16T15:50:29.373642+0000","last_clean":"2024-09-16T15:50:29.373642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:50:29.409753+0000","last_undegraded":"2024-09-16T15:50:29.409753+0000","last_fullsized":"2024-09-16T15:50:29.409753+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_clean_scrub_stamp":"2024-09-16T15:50:29.373642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.0","version":"0'0","reported_seq":10,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-16T15:50:29.406562+0000","last_change":"2024-09-16T15:50:29.395361+0000","last_active":"2024-09-16T15:50:29.406562+0000","last_peered":"2024-09-16T15:50:29.406562+0000","last_clean":"2024-09-16T15:50:29.406562+0000","last_became_active":"2024-09-16T15:50:29.394578+0000","last_became_peered":"2024-09-16T15:50:29.394578+0000","last_unstale":"2024-09-16T15:50:29.406562+0000","last_undegraded":"2024-09-16T15:50:29.406562+0000","last_fullsized":"2024-09-16T15:50:29.406562+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:28.359944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:28.359944+0000","last_clean_scrub_stamp":"2024-09-16T15:50:28.359944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T02:06:32.966504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"23.16","version":"0'0","reported_seq":1,"reported_epoch":144,"state":"creating+peering","last_fresh":"2024-09-16T15:50:29.401057+0000","last_change":"2024-09-16T15:50:29.401057+0000","last_active":"2024-09-16T15:50:29.373642+0000","last_peered":"2024-09-16T15:50:29.373642+0000","last_clean":"2024-09-16T15:50:29.373642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:50:29.401057+0000","last_undegraded":"2024-09-16T15:50:29.401057+0000","last_fullsized":"2024-09-16T15:50:29.401057+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_clean_scrub_stamp":"2024-09-16T15:50:29.373642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.17","version":"0'0","reported_seq":10,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-16T15:50:29.421257+0000","last_change":"2024-09-16T15:50:29.410992+0000","last_active":"2024-09-16T15:50:29.421257+0000","last_peered":"2024-09-16T15:50:29.421257+0000","last_clean":"2024-09-16T15:50:29.421257+0000","last_became_active":"2024-09-16T15:50:29.410215+0000","last_became_peered":"2024-09-16T15:50:29.410215+0000","last_unstale":"2024-09-16T15:50:29.421257+0000","last_undegraded":"2024-09-16T15:50:29.421257+0000","last_fullsized":"2024-09-16T15:50:29.421257+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:28.359944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:28.359944+0000","last_clean_scrub_stamp":"2024-09-16T15:50:28.359944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T00:24:19.013129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"73'79","reported_seq":366,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-16T15:50:29.402844+0000","last_change":"2024-09-16T15:42:40.881820+0000","last_active":"2024-09-16T15:50:29.402844+0000","last_peered":"2024-09-16T15:50:29.402844+0000","last_clean":"2024-09-16T15:50:29.402844+0000","last_became_active":"2024-09-16T15:42:40.879965+0000","last_became_peered":"2024-09-16T15:42:40.879965+0000","last_unstale":"2024-09-16T15:50:29.402844+0000","last_undegraded":"2024-09-16T15:50:29.402844+0000","last_fullsized":"2024-09-16T15:50:29.402844+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:42:39.728762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:42:39.728762+0000","last_clean_scrub_stamp":"2024-09-16T15:42:39.728762+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:22:18.768932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":23,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":22,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79,"ondisk_log_size":79,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542242,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33536,"kb_used_data":2344,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338304,"statfs":{"total":96636764160,"available":96602423296,"internally_reserved":0,"allocated":2400256,"data_stored":1389123,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542241,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32956,"kb_used_data":1764,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338884,"statfs":{"total":96636764160,"available":96603017216,"internally_reserved":0,"allocated":1806336,"data_stored":798755,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":5,"up_from":14,"seq":60129542242,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32940,"kb_used_data":1752,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338900,"statfs":{"total":96636764160,"available":96603033600,"internally_reserved":0,"allocated":1794048,"data_stored":792452,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542240,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32940,"kb_used_data":1752,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338900,"statfs":{"total":96636764160,"available":96603033600,"internally_reserved":0,"allocated":1794048,"data_stored":792452,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":3,"up_from":14,"seq":60129542241,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37100,"kb_used_data":1752,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334740,"statfs":{"total":96636764160,"available":96598773760,"internally_reserved":0,"allocated":1794048,"data_stored":792452,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":14,"seq":60129542241,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32956,"kb_used_data":1764,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338884,"statfs":{"total":96636764160,"available":96603017216,"internally_reserved":0,"allocated":1806336,"data_stored":798755,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":14,"seq":60129542243,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32956,"kb_used_data":1764,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338884,"statfs":{"total":96636764160,"available":96603017216,"internally_reserved":0,"allocated":1806336,"data_stored":798755,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":14,"seq":60129542243,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33520,"kb_used_data":2332,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338320,"statfs":{"total":96636764160,"available":96602439680,"internally_reserved":0,"allocated":2387968,"data_stored":1382820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T15:50:34.464 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T15:50:34.622 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:34.622+0000 7fc1fb9d3640 1 -- 172.21.15.123:0/426327280 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc1f415e820 msgr2=0x7fc1f415ecd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:50:34.623 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:34.622+0000 7fc1fb9d3640 1 --2- 172.21.15.123:0/426327280 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc1f415e820 0x7fc1f415ecd0 secure :-1 s=READY pgs=1593 cs=0 l=1 rev1=1 crypto rx=0x7fc1ec009c80 tx=0x7fc1ec01cd10 comp rx=0 tx=0).stop 2024-09-16T15:50:34.623 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:34.622+0000 7fc1fb9d3640 1 -- 172.21.15.123:0/426327280 shutdown_connections 2024-09-16T15:50:34.623 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:34.622+0000 7fc1fb9d3640 1 --2- 172.21.15.123:0/426327280 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc1f415e820 0x7fc1f415ecd0 unknown :-1 s=CLOSED pgs=1593 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:34.623 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:34.622+0000 7fc1fb9d3640 1 --2- 172.21.15.123:0/426327280 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fc1f415e160 0x7fc1f4073f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:34.623 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:34.622+0000 7fc1fb9d3640 1 -- 172.21.15.123:0/426327280 >> 172.21.15.123:0/426327280 conn(0x7fc1f40cfb10 msgr2=0x7fc1f40cff20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:50:34.623 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:34.622+0000 7fc1fb9d3640 1 -- 172.21.15.123:0/426327280 shutdown_connections 2024-09-16T15:50:34.623 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:34.622+0000 7fc1fb9d3640 1 -- 172.21.15.123:0/426327280 wait complete. 2024-09-16T15:50:34.624 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:34.622+0000 7fc1fb9d3640 1 Processor -- start 2024-09-16T15:50:34.624 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:34.622+0000 7fc1fb9d3640 1 -- start start 2024-09-16T15:50:34.626 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:34.626+0000 7fc1fb9d3640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc1f4074470 0x7fc1f40c88d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:50:34.626 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:34.626+0000 7fc1fb9d3640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc1f415e160 0x7fc1f4153a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:50:34.626 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:34.626+0000 7fc1fb9d3640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fc1f415e820 0x7fc1f4159460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:50:34.626 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:34.626+0000 7fc1fb9d3640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fc1f40cf5b0 con 0x7fc1f4074470 2024-09-16T15:50:34.626 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:34.626+0000 7fc1fb9d3640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fc1f40cf430 con 0x7fc1f415e160 2024-09-16T15:50:34.626 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:34.626+0000 7fc1fb9d3640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fc1f41683d0 con 0x7fc1f415e820 2024-09-16T15:50:34.626 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:34.626+0000 7fc1f976f640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc1f4074470 0x7fc1f40c88d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:50:34.627 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:34.626+0000 7fc1f976f640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc1f4074470 0x7fc1f40c88d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:49660/0 (socket says 172.21.15.123:49660) 2024-09-16T15:50:34.627 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:34.626+0000 7fc1f976f640 1 -- 172.21.15.123:0/544862788 learned_addr learned my addr 172.21.15.123:0/544862788 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:50:34.627 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:34.626+0000 7fc1f976f640 1 -- 172.21.15.123:0/544862788 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fc1f415e820 msgr2=0x7fc1f4159460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:50:34.627 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:34.626+0000 7fc1f976f640 1 --2- 172.21.15.123:0/544862788 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fc1f415e820 0x7fc1f4159460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:34.627 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:34.626+0000 7fc1f976f640 1 -- 172.21.15.123:0/544862788 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc1f415e160 msgr2=0x7fc1f4153a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:50:34.627 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:34.626+0000 7fc1f976f640 1 --2- 172.21.15.123:0/544862788 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc1f415e160 0x7fc1f4153a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:34.627 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:34.626+0000 7fc1f976f640 1 -- 172.21.15.123:0/544862788 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc1f4152df0 con 0x7fc1f4074470 2024-09-16T15:50:34.627 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:34.626+0000 7fc1f976f640 1 --2- 172.21.15.123:0/544862788 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc1f4074470 0x7fc1f40c88d0 secure :-1 s=READY pgs=1594 cs=0 l=1 rev1=1 crypto rx=0x7fc1e800d950 tx=0x7fc1e800de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:50:34.627 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:34.626+0000 7fc1e5ffb640 1 -- 172.21.15.123:0/544862788 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1e8017070 con 0x7fc1f4074470 2024-09-16T15:50:34.627 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:34.626+0000 7fc1e5ffb640 1 -- 172.21.15.123:0/544862788 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fc1e8009d20 con 0x7fc1f4074470 2024-09-16T15:50:34.628 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:34.626+0000 7fc1fb9d3640 1 -- 172.21.15.123:0/544862788 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc1f41530d0 con 0x7fc1f4074470 2024-09-16T15:50:34.628 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:34.626+0000 7fc1e5ffb640 1 -- 172.21.15.123:0/544862788 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1e80106c0 con 0x7fc1f4074470 2024-09-16T15:50:34.628 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:34.626+0000 7fc1fb9d3640 1 -- 172.21.15.123:0/544862788 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc1f4153690 con 0x7fc1f4074470 2024-09-16T15:50:34.628 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:34.626+0000 7fc1fb9d3640 1 -- 172.21.15.123:0/544862788 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc1f4073f30 con 0x7fc1f4074470 2024-09-16T15:50:34.629 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:34.626+0000 7fc1e5ffb640 1 -- 172.21.15.123:0/544862788 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 22) ==== 97692+0+0 (secure 0 0 0) 0x7fc1e8010860 con 0x7fc1f4074470 2024-09-16T15:50:34.629 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:34.626+0000 7fc1e5ffb640 1 --2- 172.21.15.123:0/544862788 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fc1d8073580 0x7fc1d8075a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:50:34.629 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:34.626+0000 7fc1e5ffb640 1 -- 172.21.15.123:0/544862788 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(146..146 src has 1..146) ==== 5737+0+0 (secure 0 0 0) 0x7fc1e8095000 con 0x7fc1f4074470 2024-09-16T15:50:34.629 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:34.626+0000 7fc1f8f6e640 1 --2- 172.21.15.123:0/544862788 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fc1d8073580 0x7fc1d8075a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:50:34.629 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:34.626+0000 7fc1f8f6e640 1 --2- 172.21.15.123:0/544862788 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fc1d8073580 0x7fc1d8075a40 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7fc1f0007fd0 tx=0x7fc1f00078e0 comp rx=0 tx=0).ready entity=mgr.4150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:50:34.631 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:34.630+0000 7fc1e5ffb640 1 -- 172.21.15.123:0/544862788 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7fc1e805f700 con 0x7fc1f4074470 2024-09-16T15:50:34.787 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:34.786+0000 7fc1fb9d3640 1 -- 172.21.15.123:0/544862788 --> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fc1f40c8e10 con 0x7fc1d8073580 2024-09-16T15:50:34.798 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:34.798+0000 7fc1e5ffb640 1 -- 172.21.15.123:0/544862788 <== mgr.4150 v2:172.21.15.123:6836/2464759676 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+247115 (secure 0 0 0) 0x7fc1f40c8e10 con 0x7fc1d8073580 2024-09-16T15:50:34.800 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-09-16T15:50:34.801 INFO:teuthology.orchestra.run.smithi123.stderr:dumped all 2024-09-16T15:50:34.802 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:34.802+0000 7fc1fb9d3640 1 -- 172.21.15.123:0/544862788 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fc1d8073580 msgr2=0x7fc1d8075a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:50:34.802 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:34.802+0000 7fc1fb9d3640 1 --2- 172.21.15.123:0/544862788 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fc1d8073580 0x7fc1d8075a40 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7fc1f0007fd0 tx=0x7fc1f00078e0 comp rx=0 tx=0).stop 2024-09-16T15:50:34.802 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:34.802+0000 7fc1fb9d3640 1 -- 172.21.15.123:0/544862788 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc1f4074470 msgr2=0x7fc1f40c88d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:50:34.802 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:34.802+0000 7fc1fb9d3640 1 --2- 172.21.15.123:0/544862788 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc1f4074470 0x7fc1f40c88d0 secure :-1 s=READY pgs=1594 cs=0 l=1 rev1=1 crypto rx=0x7fc1e800d950 tx=0x7fc1e800de10 comp rx=0 tx=0).stop 2024-09-16T15:50:34.802 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:34.802+0000 7fc1fb9d3640 1 -- 172.21.15.123:0/544862788 shutdown_connections 2024-09-16T15:50:34.803 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:34.802+0000 7fc1fb9d3640 1 --2- 172.21.15.123:0/544862788 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fc1d8073580 0x7fc1d8075a40 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:34.803 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:34.802+0000 7fc1fb9d3640 1 --2- 172.21.15.123:0/544862788 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fc1f415e820 0x7fc1f4159460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:34.803 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:34.802+0000 7fc1fb9d3640 1 --2- 172.21.15.123:0/544862788 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc1f415e160 0x7fc1f4153a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:34.803 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:34.802+0000 7fc1fb9d3640 1 --2- 172.21.15.123:0/544862788 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc1f4074470 0x7fc1f40c88d0 unknown :-1 s=CLOSED pgs=1594 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:34.803 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:34.802+0000 7fc1fb9d3640 1 -- 172.21.15.123:0/544862788 >> 172.21.15.123:0/544862788 conn(0x7fc1f40cfb10 msgr2=0x7fc1f40ccc30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:50:34.803 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:34.802+0000 7fc1fb9d3640 1 -- 172.21.15.123:0/544862788 shutdown_connections 2024-09-16T15:50:34.803 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:34.802+0000 7fc1fb9d3640 1 -- 172.21.15.123:0/544862788 wait complete. 2024-09-16T15:50:34.816 INFO:teuthology.orchestra.run.smithi123.stdout:{"pg_ready":true,"pg_map":{"version":167,"stamp":"2024-09-16T15:50:34.119725+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79,"ondisk_log_size":79,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":39,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":268904,"kb_used_data":15224,"kb_used_omap":12,"kb_used_meta":253491,"kb_avail":754705816,"statfs":{"total":773094113280,"available":772818755584,"internally_reserved":0,"allocated":15589376,"data_stored":7545564,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":259575345},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":29,"apply_latency_ms":29,"commit_latency_ns":29000000,"apply_latency_ns":29000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.743992"},"pg_stats":[{"pgid":"23.3f","version":"0'0","reported_seq":1,"reported_epoch":144,"state":"creating+peering","last_fresh":"2024-09-16T15:50:29.396826+0000","last_change":"2024-09-16T15:50:29.396826+0000","last_active":"2024-09-16T15:50:29.373642+0000","last_peered":"2024-09-16T15:50:29.373642+0000","last_clean":"2024-09-16T15:50:29.373642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:50:29.396826+0000","last_undegraded":"2024-09-16T15:50:29.396826+0000","last_fullsized":"2024-09-16T15:50:29.396826+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_clean_scrub_stamp":"2024-09-16T15:50:29.373642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.3e","version":"0'0","reported_seq":1,"reported_epoch":144,"state":"creating+peering","last_fresh":"2024-09-16T15:50:29.405611+0000","last_change":"2024-09-16T15:50:29.405611+0000","last_active":"2024-09-16T15:50:29.373642+0000","last_peered":"2024-09-16T15:50:29.373642+0000","last_clean":"2024-09-16T15:50:29.373642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:50:29.405611+0000","last_undegraded":"2024-09-16T15:50:29.405611+0000","last_fullsized":"2024-09-16T15:50:29.405611+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_clean_scrub_stamp":"2024-09-16T15:50:29.373642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.3d","version":"0'0","reported_seq":1,"reported_epoch":144,"state":"creating+peering","last_fresh":"2024-09-16T15:50:29.414816+0000","last_change":"2024-09-16T15:50:29.414816+0000","last_active":"2024-09-16T15:50:29.373642+0000","last_peered":"2024-09-16T15:50:29.373642+0000","last_clean":"2024-09-16T15:50:29.373642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:50:29.414816+0000","last_undegraded":"2024-09-16T15:50:29.414816+0000","last_fullsized":"2024-09-16T15:50:29.414816+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_clean_scrub_stamp":"2024-09-16T15:50:29.373642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.3c","version":"0'0","reported_seq":1,"reported_epoch":144,"state":"creating+peering","last_fresh":"2024-09-16T15:50:29.400948+0000","last_change":"2024-09-16T15:50:29.400948+0000","last_active":"2024-09-16T15:50:29.373642+0000","last_peered":"2024-09-16T15:50:29.373642+0000","last_clean":"2024-09-16T15:50:29.373642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:50:29.400948+0000","last_undegraded":"2024-09-16T15:50:29.400948+0000","last_fullsized":"2024-09-16T15:50:29.400948+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_clean_scrub_stamp":"2024-09-16T15:50:29.373642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.3b","version":"0'0","reported_seq":1,"reported_epoch":144,"state":"creating+peering","last_fresh":"2024-09-16T15:50:29.408612+0000","last_change":"2024-09-16T15:50:29.408612+0000","last_active":"2024-09-16T15:50:29.373642+0000","last_peered":"2024-09-16T15:50:29.373642+0000","last_clean":"2024-09-16T15:50:29.373642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:50:29.408612+0000","last_undegraded":"2024-09-16T15:50:29.408612+0000","last_fullsized":"2024-09-16T15:50:29.408612+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_clean_scrub_stamp":"2024-09-16T15:50:29.373642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.3a","version":"0'0","reported_seq":1,"reported_epoch":144,"state":"creating+peering","last_fresh":"2024-09-16T15:50:29.399825+0000","last_change":"2024-09-16T15:50:29.399825+0000","last_active":"2024-09-16T15:50:29.373642+0000","last_peered":"2024-09-16T15:50:29.373642+0000","last_clean":"2024-09-16T15:50:29.373642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:50:29.399825+0000","last_undegraded":"2024-09-16T15:50:29.399825+0000","last_fullsized":"2024-09-16T15:50:29.399825+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_clean_scrub_stamp":"2024-09-16T15:50:29.373642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.39","version":"0'0","reported_seq":1,"reported_epoch":144,"state":"creating+peering","last_fresh":"2024-09-16T15:50:29.397580+0000","last_change":"2024-09-16T15:50:29.397580+0000","last_active":"2024-09-16T15:50:29.373642+0000","last_peered":"2024-09-16T15:50:29.373642+0000","last_clean":"2024-09-16T15:50:29.373642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:50:29.397580+0000","last_undegraded":"2024-09-16T15:50:29.397580+0000","last_fullsized":"2024-09-16T15:50:29.397580+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_clean_scrub_stamp":"2024-09-16T15:50:29.373642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"23.38","version":"0'0","reported_seq":1,"reported_epoch":144,"state":"creating+peering","last_fresh":"2024-09-16T15:50:29.409708+0000","last_change":"2024-09-16T15:50:29.409708+0000","last_active":"2024-09-16T15:50:29.373642+0000","last_peered":"2024-09-16T15:50:29.373642+0000","last_clean":"2024-09-16T15:50:29.373642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:50:29.409708+0000","last_undegraded":"2024-09-16T15:50:29.409708+0000","last_fullsized":"2024-09-16T15:50:29.409708+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_clean_scrub_stamp":"2024-09-16T15:50:29.373642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.37","version":"0'0","reported_seq":1,"reported_epoch":144,"state":"creating+peering","last_fresh":"2024-09-16T15:50:29.406549+0000","last_change":"2024-09-16T15:50:29.406549+0000","last_active":"2024-09-16T15:50:29.373642+0000","last_peered":"2024-09-16T15:50:29.373642+0000","last_clean":"2024-09-16T15:50:29.373642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:50:29.406549+0000","last_undegraded":"2024-09-16T15:50:29.406549+0000","last_fullsized":"2024-09-16T15:50:29.406549+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_clean_scrub_stamp":"2024-09-16T15:50:29.373642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.36","version":"0'0","reported_seq":1,"reported_epoch":144,"state":"creating+peering","last_fresh":"2024-09-16T15:50:29.404538+0000","last_change":"2024-09-16T15:50:29.404538+0000","last_active":"2024-09-16T15:50:29.373642+0000","last_peered":"2024-09-16T15:50:29.373642+0000","last_clean":"2024-09-16T15:50:29.373642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:50:29.404538+0000","last_undegraded":"2024-09-16T15:50:29.404538+0000","last_fullsized":"2024-09-16T15:50:29.404538+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_clean_scrub_stamp":"2024-09-16T15:50:29.373642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.35","version":"0'0","reported_seq":1,"reported_epoch":144,"state":"creating+peering","last_fresh":"2024-09-16T15:50:29.418503+0000","last_change":"2024-09-16T15:50:29.418503+0000","last_active":"2024-09-16T15:50:29.373642+0000","last_peered":"2024-09-16T15:50:29.373642+0000","last_clean":"2024-09-16T15:50:29.373642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:50:29.418503+0000","last_undegraded":"2024-09-16T15:50:29.418503+0000","last_fullsized":"2024-09-16T15:50:29.418503+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_clean_scrub_stamp":"2024-09-16T15:50:29.373642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.34","version":"0'0","reported_seq":1,"reported_epoch":144,"state":"creating+peering","last_fresh":"2024-09-16T15:50:29.398426+0000","last_change":"2024-09-16T15:50:29.398426+0000","last_active":"2024-09-16T15:50:29.373642+0000","last_peered":"2024-09-16T15:50:29.373642+0000","last_clean":"2024-09-16T15:50:29.373642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:50:29.398426+0000","last_undegraded":"2024-09-16T15:50:29.398426+0000","last_fullsized":"2024-09-16T15:50:29.398426+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_clean_scrub_stamp":"2024-09-16T15:50:29.373642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.33","version":"0'0","reported_seq":1,"reported_epoch":144,"state":"creating+peering","last_fresh":"2024-09-16T15:50:29.415630+0000","last_change":"2024-09-16T15:50:29.415630+0000","last_active":"2024-09-16T15:50:29.373642+0000","last_peered":"2024-09-16T15:50:29.373642+0000","last_clean":"2024-09-16T15:50:29.373642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:50:29.415630+0000","last_undegraded":"2024-09-16T15:50:29.415630+0000","last_fullsized":"2024-09-16T15:50:29.415630+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_clean_scrub_stamp":"2024-09-16T15:50:29.373642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.32","version":"0'0","reported_seq":1,"reported_epoch":144,"state":"creating+peering","last_fresh":"2024-09-16T15:50:29.411622+0000","last_change":"2024-09-16T15:50:29.411622+0000","last_active":"2024-09-16T15:50:29.373642+0000","last_peered":"2024-09-16T15:50:29.373642+0000","last_clean":"2024-09-16T15:50:29.373642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:50:29.411622+0000","last_undegraded":"2024-09-16T15:50:29.411622+0000","last_fullsized":"2024-09-16T15:50:29.411622+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_clean_scrub_stamp":"2024-09-16T15:50:29.373642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.31","version":"0'0","reported_seq":1,"reported_epoch":144,"state":"creating+peering","last_fresh":"2024-09-16T15:50:29.396532+0000","last_change":"2024-09-16T15:50:29.396532+0000","last_active":"2024-09-16T15:50:29.373642+0000","last_peered":"2024-09-16T15:50:29.373642+0000","last_clean":"2024-09-16T15:50:29.373642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:50:29.396532+0000","last_undegraded":"2024-09-16T15:50:29.396532+0000","last_fullsized":"2024-09-16T15:50:29.396532+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_clean_scrub_stamp":"2024-09-16T15:50:29.373642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.30","version":"0'0","reported_seq":1,"reported_epoch":144,"state":"creating+peering","last_fresh":"2024-09-16T15:50:29.398051+0000","last_change":"2024-09-16T15:50:29.398051+0000","last_active":"2024-09-16T15:50:29.373642+0000","last_peered":"2024-09-16T15:50:29.373642+0000","last_clean":"2024-09-16T15:50:29.373642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:50:29.398051+0000","last_undegraded":"2024-09-16T15:50:29.398051+0000","last_fullsized":"2024-09-16T15:50:29.398051+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_clean_scrub_stamp":"2024-09-16T15:50:29.373642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"23.2f","version":"0'0","reported_seq":1,"reported_epoch":144,"state":"creating+peering","last_fresh":"2024-09-16T15:50:29.399338+0000","last_change":"2024-09-16T15:50:29.399338+0000","last_active":"2024-09-16T15:50:29.373642+0000","last_peered":"2024-09-16T15:50:29.373642+0000","last_clean":"2024-09-16T15:50:29.373642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:50:29.399338+0000","last_undegraded":"2024-09-16T15:50:29.399338+0000","last_fullsized":"2024-09-16T15:50:29.399338+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_clean_scrub_stamp":"2024-09-16T15:50:29.373642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.2e","version":"0'0","reported_seq":1,"reported_epoch":144,"state":"creating+peering","last_fresh":"2024-09-16T15:50:29.403155+0000","last_change":"2024-09-16T15:50:29.403155+0000","last_active":"2024-09-16T15:50:29.373642+0000","last_peered":"2024-09-16T15:50:29.373642+0000","last_clean":"2024-09-16T15:50:29.373642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:50:29.403155+0000","last_undegraded":"2024-09-16T15:50:29.403155+0000","last_fullsized":"2024-09-16T15:50:29.403155+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_clean_scrub_stamp":"2024-09-16T15:50:29.373642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"23.2d","version":"0'0","reported_seq":1,"reported_epoch":144,"state":"creating+peering","last_fresh":"2024-09-16T15:50:29.399089+0000","last_change":"2024-09-16T15:50:29.399089+0000","last_active":"2024-09-16T15:50:29.373642+0000","last_peered":"2024-09-16T15:50:29.373642+0000","last_clean":"2024-09-16T15:50:29.373642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:50:29.399089+0000","last_undegraded":"2024-09-16T15:50:29.399089+0000","last_fullsized":"2024-09-16T15:50:29.399089+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_clean_scrub_stamp":"2024-09-16T15:50:29.373642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.2c","version":"0'0","reported_seq":1,"reported_epoch":144,"state":"creating+peering","last_fresh":"2024-09-16T15:50:29.400996+0000","last_change":"2024-09-16T15:50:29.400996+0000","last_active":"2024-09-16T15:50:29.373642+0000","last_peered":"2024-09-16T15:50:29.373642+0000","last_clean":"2024-09-16T15:50:29.373642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:50:29.400996+0000","last_undegraded":"2024-09-16T15:50:29.400996+0000","last_fullsized":"2024-09-16T15:50:29.400996+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_clean_scrub_stamp":"2024-09-16T15:50:29.373642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.2b","version":"0'0","reported_seq":1,"reported_epoch":144,"state":"creating+peering","last_fresh":"2024-09-16T15:50:29.395481+0000","last_change":"2024-09-16T15:50:29.395481+0000","last_active":"2024-09-16T15:50:29.373642+0000","last_peered":"2024-09-16T15:50:29.373642+0000","last_clean":"2024-09-16T15:50:29.373642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:50:29.395481+0000","last_undegraded":"2024-09-16T15:50:29.395481+0000","last_fullsized":"2024-09-16T15:50:29.395481+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_clean_scrub_stamp":"2024-09-16T15:50:29.373642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.2a","version":"0'0","reported_seq":1,"reported_epoch":144,"state":"creating+peering","last_fresh":"2024-09-16T15:50:29.396706+0000","last_change":"2024-09-16T15:50:29.396706+0000","last_active":"2024-09-16T15:50:29.373642+0000","last_peered":"2024-09-16T15:50:29.373642+0000","last_clean":"2024-09-16T15:50:29.373642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:50:29.396706+0000","last_undegraded":"2024-09-16T15:50:29.396706+0000","last_fullsized":"2024-09-16T15:50:29.396706+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_clean_scrub_stamp":"2024-09-16T15:50:29.373642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.29","version":"0'0","reported_seq":1,"reported_epoch":144,"state":"creating+peering","last_fresh":"2024-09-16T15:50:29.397404+0000","last_change":"2024-09-16T15:50:29.397404+0000","last_active":"2024-09-16T15:50:29.373642+0000","last_peered":"2024-09-16T15:50:29.373642+0000","last_clean":"2024-09-16T15:50:29.373642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:50:29.397404+0000","last_undegraded":"2024-09-16T15:50:29.397404+0000","last_fullsized":"2024-09-16T15:50:29.397404+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_clean_scrub_stamp":"2024-09-16T15:50:29.373642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.28","version":"0'0","reported_seq":1,"reported_epoch":144,"state":"creating+peering","last_fresh":"2024-09-16T15:50:29.396706+0000","last_change":"2024-09-16T15:50:29.396706+0000","last_active":"2024-09-16T15:50:29.373642+0000","last_peered":"2024-09-16T15:50:29.373642+0000","last_clean":"2024-09-16T15:50:29.373642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:50:29.396706+0000","last_undegraded":"2024-09-16T15:50:29.396706+0000","last_fullsized":"2024-09-16T15:50:29.396706+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_clean_scrub_stamp":"2024-09-16T15:50:29.373642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.27","version":"0'0","reported_seq":1,"reported_epoch":144,"state":"creating+peering","last_fresh":"2024-09-16T15:50:29.401802+0000","last_change":"2024-09-16T15:50:29.401802+0000","last_active":"2024-09-16T15:50:29.373642+0000","last_peered":"2024-09-16T15:50:29.373642+0000","last_clean":"2024-09-16T15:50:29.373642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:50:29.401802+0000","last_undegraded":"2024-09-16T15:50:29.401802+0000","last_fullsized":"2024-09-16T15:50:29.401802+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_clean_scrub_stamp":"2024-09-16T15:50:29.373642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.26","version":"0'0","reported_seq":1,"reported_epoch":144,"state":"creating+peering","last_fresh":"2024-09-16T15:50:29.397726+0000","last_change":"2024-09-16T15:50:29.397726+0000","last_active":"2024-09-16T15:50:29.373642+0000","last_peered":"2024-09-16T15:50:29.373642+0000","last_clean":"2024-09-16T15:50:29.373642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:50:29.397726+0000","last_undegraded":"2024-09-16T15:50:29.397726+0000","last_fullsized":"2024-09-16T15:50:29.397726+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_clean_scrub_stamp":"2024-09-16T15:50:29.373642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"23.25","version":"0'0","reported_seq":1,"reported_epoch":144,"state":"creating+peering","last_fresh":"2024-09-16T15:50:29.413671+0000","last_change":"2024-09-16T15:50:29.413671+0000","last_active":"2024-09-16T15:50:29.373642+0000","last_peered":"2024-09-16T15:50:29.373642+0000","last_clean":"2024-09-16T15:50:29.373642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:50:29.413671+0000","last_undegraded":"2024-09-16T15:50:29.413671+0000","last_fullsized":"2024-09-16T15:50:29.413671+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_clean_scrub_stamp":"2024-09-16T15:50:29.373642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.24","version":"0'0","reported_seq":1,"reported_epoch":144,"state":"creating+peering","last_fresh":"2024-09-16T15:50:29.400095+0000","last_change":"2024-09-16T15:50:29.400095+0000","last_active":"2024-09-16T15:50:29.373642+0000","last_peered":"2024-09-16T15:50:29.373642+0000","last_clean":"2024-09-16T15:50:29.373642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:50:29.400095+0000","last_undegraded":"2024-09-16T15:50:29.400095+0000","last_fullsized":"2024-09-16T15:50:29.400095+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_clean_scrub_stamp":"2024-09-16T15:50:29.373642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.23","version":"0'0","reported_seq":1,"reported_epoch":144,"state":"creating+peering","last_fresh":"2024-09-16T15:50:29.403015+0000","last_change":"2024-09-16T15:50:29.403015+0000","last_active":"2024-09-16T15:50:29.373642+0000","last_peered":"2024-09-16T15:50:29.373642+0000","last_clean":"2024-09-16T15:50:29.373642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:50:29.403015+0000","last_undegraded":"2024-09-16T15:50:29.403015+0000","last_fullsized":"2024-09-16T15:50:29.403015+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_clean_scrub_stamp":"2024-09-16T15:50:29.373642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.22","version":"0'0","reported_seq":1,"reported_epoch":144,"state":"creating+peering","last_fresh":"2024-09-16T15:50:29.390228+0000","last_change":"2024-09-16T15:50:29.390228+0000","last_active":"2024-09-16T15:50:29.373642+0000","last_peered":"2024-09-16T15:50:29.373642+0000","last_clean":"2024-09-16T15:50:29.373642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:50:29.390228+0000","last_undegraded":"2024-09-16T15:50:29.390228+0000","last_fullsized":"2024-09-16T15:50:29.390228+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_clean_scrub_stamp":"2024-09-16T15:50:29.373642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"23.21","version":"0'0","reported_seq":1,"reported_epoch":144,"state":"creating+peering","last_fresh":"2024-09-16T15:50:29.392793+0000","last_change":"2024-09-16T15:50:29.392793+0000","last_active":"2024-09-16T15:50:29.373642+0000","last_peered":"2024-09-16T15:50:29.373642+0000","last_clean":"2024-09-16T15:50:29.373642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:50:29.392793+0000","last_undegraded":"2024-09-16T15:50:29.392793+0000","last_fullsized":"2024-09-16T15:50:29.392793+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_clean_scrub_stamp":"2024-09-16T15:50:29.373642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"23.20","version":"0'0","reported_seq":1,"reported_epoch":144,"state":"creating+peering","last_fresh":"2024-09-16T15:50:29.400381+0000","last_change":"2024-09-16T15:50:29.400381+0000","last_active":"2024-09-16T15:50:29.373642+0000","last_peered":"2024-09-16T15:50:29.373642+0000","last_clean":"2024-09-16T15:50:29.373642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:50:29.400381+0000","last_undegraded":"2024-09-16T15:50:29.400381+0000","last_fullsized":"2024-09-16T15:50:29.400381+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_clean_scrub_stamp":"2024-09-16T15:50:29.373642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"23.1e","version":"0'0","reported_seq":1,"reported_epoch":144,"state":"creating+peering","last_fresh":"2024-09-16T15:50:29.396097+0000","last_change":"2024-09-16T15:50:29.396097+0000","last_active":"2024-09-16T15:50:29.373642+0000","last_peered":"2024-09-16T15:50:29.373642+0000","last_clean":"2024-09-16T15:50:29.373642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:50:29.396097+0000","last_undegraded":"2024-09-16T15:50:29.396097+0000","last_fullsized":"2024-09-16T15:50:29.396097+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_clean_scrub_stamp":"2024-09-16T15:50:29.373642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"22.1f","version":"0'0","reported_seq":10,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-16T15:50:29.430449+0000","last_change":"2024-09-16T15:50:29.428405+0000","last_active":"2024-09-16T15:50:29.430449+0000","last_peered":"2024-09-16T15:50:29.430449+0000","last_clean":"2024-09-16T15:50:29.430449+0000","last_became_active":"2024-09-16T15:50:29.423819+0000","last_became_peered":"2024-09-16T15:50:29.423819+0000","last_unstale":"2024-09-16T15:50:29.430449+0000","last_undegraded":"2024-09-16T15:50:29.430449+0000","last_fullsized":"2024-09-16T15:50:29.430449+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:28.359944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:28.359944+0000","last_clean_scrub_stamp":"2024-09-16T15:50:28.359944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T22:06:16.541534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.1f","version":"0'0","reported_seq":1,"reported_epoch":144,"state":"creating+peering","last_fresh":"2024-09-16T15:50:29.395019+0000","last_change":"2024-09-16T15:50:29.395019+0000","last_active":"2024-09-16T15:50:29.373642+0000","last_peered":"2024-09-16T15:50:29.373642+0000","last_clean":"2024-09-16T15:50:29.373642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:50:29.395019+0000","last_undegraded":"2024-09-16T15:50:29.395019+0000","last_fullsized":"2024-09-16T15:50:29.395019+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_clean_scrub_stamp":"2024-09-16T15:50:29.373642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.1e","version":"0'0","reported_seq":10,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-16T15:50:29.433442+0000","last_change":"2024-09-16T15:50:29.431326+0000","last_active":"2024-09-16T15:50:29.433442+0000","last_peered":"2024-09-16T15:50:29.433442+0000","last_clean":"2024-09-16T15:50:29.433442+0000","last_became_active":"2024-09-16T15:50:29.430610+0000","last_became_peered":"2024-09-16T15:50:29.430610+0000","last_unstale":"2024-09-16T15:50:29.433442+0000","last_undegraded":"2024-09-16T15:50:29.433442+0000","last_fullsized":"2024-09-16T15:50:29.433442+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:28.359944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:28.359944+0000","last_clean_scrub_stamp":"2024-09-16T15:50:28.359944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T00:28:56.379780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"23.1c","version":"0'0","reported_seq":1,"reported_epoch":144,"state":"creating+peering","last_fresh":"2024-09-16T15:50:29.395809+0000","last_change":"2024-09-16T15:50:29.395809+0000","last_active":"2024-09-16T15:50:29.373642+0000","last_peered":"2024-09-16T15:50:29.373642+0000","last_clean":"2024-09-16T15:50:29.373642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:50:29.395809+0000","last_undegraded":"2024-09-16T15:50:29.395809+0000","last_fullsized":"2024-09-16T15:50:29.395809+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_clean_scrub_stamp":"2024-09-16T15:50:29.373642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.1d","version":"0'0","reported_seq":10,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-16T15:50:29.431196+0000","last_change":"2024-09-16T15:50:29.425771+0000","last_active":"2024-09-16T15:50:29.431196+0000","last_peered":"2024-09-16T15:50:29.431196+0000","last_clean":"2024-09-16T15:50:29.431196+0000","last_became_active":"2024-09-16T15:50:29.425507+0000","last_became_peered":"2024-09-16T15:50:29.425507+0000","last_unstale":"2024-09-16T15:50:29.431196+0000","last_undegraded":"2024-09-16T15:50:29.431196+0000","last_fullsized":"2024-09-16T15:50:29.431196+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:28.359944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:28.359944+0000","last_clean_scrub_stamp":"2024-09-16T15:50:28.359944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:16:32.881957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.1d","version":"0'0","reported_seq":1,"reported_epoch":144,"state":"creating+peering","last_fresh":"2024-09-16T15:50:29.412090+0000","last_change":"2024-09-16T15:50:29.412090+0000","last_active":"2024-09-16T15:50:29.373642+0000","last_peered":"2024-09-16T15:50:29.373642+0000","last_clean":"2024-09-16T15:50:29.373642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:50:29.412090+0000","last_undegraded":"2024-09-16T15:50:29.412090+0000","last_fullsized":"2024-09-16T15:50:29.412090+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_clean_scrub_stamp":"2024-09-16T15:50:29.373642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.1c","version":"0'0","reported_seq":10,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-16T15:50:29.411810+0000","last_change":"2024-09-16T15:50:29.408868+0000","last_active":"2024-09-16T15:50:29.411810+0000","last_peered":"2024-09-16T15:50:29.411810+0000","last_clean":"2024-09-16T15:50:29.411810+0000","last_became_active":"2024-09-16T15:50:29.408550+0000","last_became_peered":"2024-09-16T15:50:29.408550+0000","last_unstale":"2024-09-16T15:50:29.411810+0000","last_undegraded":"2024-09-16T15:50:29.411810+0000","last_fullsized":"2024-09-16T15:50:29.411810+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:28.359944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:28.359944+0000","last_clean_scrub_stamp":"2024-09-16T15:50:28.359944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:57:02.247484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.1a","version":"0'0","reported_seq":1,"reported_epoch":144,"state":"creating+peering","last_fresh":"2024-09-16T15:50:29.396181+0000","last_change":"2024-09-16T15:50:29.396181+0000","last_active":"2024-09-16T15:50:29.373642+0000","last_peered":"2024-09-16T15:50:29.373642+0000","last_clean":"2024-09-16T15:50:29.373642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:50:29.396181+0000","last_undegraded":"2024-09-16T15:50:29.396181+0000","last_fullsized":"2024-09-16T15:50:29.396181+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_clean_scrub_stamp":"2024-09-16T15:50:29.373642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"22.1b","version":"0'0","reported_seq":10,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-16T15:50:29.433993+0000","last_change":"2024-09-16T15:50:29.429045+0000","last_active":"2024-09-16T15:50:29.433993+0000","last_peered":"2024-09-16T15:50:29.433993+0000","last_clean":"2024-09-16T15:50:29.433993+0000","last_became_active":"2024-09-16T15:50:29.428317+0000","last_became_peered":"2024-09-16T15:50:29.428317+0000","last_unstale":"2024-09-16T15:50:29.433993+0000","last_undegraded":"2024-09-16T15:50:29.433993+0000","last_fullsized":"2024-09-16T15:50:29.433993+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:28.359944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:28.359944+0000","last_clean_scrub_stamp":"2024-09-16T15:50:28.359944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:54:05.773197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.1b","version":"0'0","reported_seq":1,"reported_epoch":144,"state":"creating+peering","last_fresh":"2024-09-16T15:50:29.396991+0000","last_change":"2024-09-16T15:50:29.396991+0000","last_active":"2024-09-16T15:50:29.373642+0000","last_peered":"2024-09-16T15:50:29.373642+0000","last_clean":"2024-09-16T15:50:29.373642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:50:29.396991+0000","last_undegraded":"2024-09-16T15:50:29.396991+0000","last_fullsized":"2024-09-16T15:50:29.396991+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_clean_scrub_stamp":"2024-09-16T15:50:29.373642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.1a","version":"0'0","reported_seq":10,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-16T15:50:29.411180+0000","last_change":"2024-09-16T15:50:29.408149+0000","last_active":"2024-09-16T15:50:29.411180+0000","last_peered":"2024-09-16T15:50:29.411180+0000","last_clean":"2024-09-16T15:50:29.411180+0000","last_became_active":"2024-09-16T15:50:29.407778+0000","last_became_peered":"2024-09-16T15:50:29.407778+0000","last_unstale":"2024-09-16T15:50:29.411180+0000","last_undegraded":"2024-09-16T15:50:29.411180+0000","last_fullsized":"2024-09-16T15:50:29.411180+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:28.359944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:28.359944+0000","last_clean_scrub_stamp":"2024-09-16T15:50:28.359944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:02:10.632687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.18","version":"0'0","reported_seq":1,"reported_epoch":144,"state":"creating+peering","last_fresh":"2024-09-16T15:50:29.397299+0000","last_change":"2024-09-16T15:50:29.397299+0000","last_active":"2024-09-16T15:50:29.373642+0000","last_peered":"2024-09-16T15:50:29.373642+0000","last_clean":"2024-09-16T15:50:29.373642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:50:29.397299+0000","last_undegraded":"2024-09-16T15:50:29.397299+0000","last_fullsized":"2024-09-16T15:50:29.397299+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_clean_scrub_stamp":"2024-09-16T15:50:29.373642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.19","version":"0'0","reported_seq":10,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-16T15:50:29.433714+0000","last_change":"2024-09-16T15:50:29.427582+0000","last_active":"2024-09-16T15:50:29.433714+0000","last_peered":"2024-09-16T15:50:29.433714+0000","last_clean":"2024-09-16T15:50:29.433714+0000","last_became_active":"2024-09-16T15:50:29.426124+0000","last_became_peered":"2024-09-16T15:50:29.426124+0000","last_unstale":"2024-09-16T15:50:29.433714+0000","last_undegraded":"2024-09-16T15:50:29.433714+0000","last_fullsized":"2024-09-16T15:50:29.433714+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:28.359944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:28.359944+0000","last_clean_scrub_stamp":"2024-09-16T15:50:28.359944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:01:27.892843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.19","version":"0'0","reported_seq":1,"reported_epoch":144,"state":"creating+peering","last_fresh":"2024-09-16T15:50:29.396885+0000","last_change":"2024-09-16T15:50:29.396885+0000","last_active":"2024-09-16T15:50:29.373642+0000","last_peered":"2024-09-16T15:50:29.373642+0000","last_clean":"2024-09-16T15:50:29.373642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:50:29.396885+0000","last_undegraded":"2024-09-16T15:50:29.396885+0000","last_fullsized":"2024-09-16T15:50:29.396885+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_clean_scrub_stamp":"2024-09-16T15:50:29.373642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.18","version":"0'0","reported_seq":10,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-16T15:50:29.409060+0000","last_change":"2024-09-16T15:50:29.406218+0000","last_active":"2024-09-16T15:50:29.409060+0000","last_peered":"2024-09-16T15:50:29.409060+0000","last_clean":"2024-09-16T15:50:29.409060+0000","last_became_active":"2024-09-16T15:50:29.405872+0000","last_became_peered":"2024-09-16T15:50:29.405872+0000","last_unstale":"2024-09-16T15:50:29.409060+0000","last_undegraded":"2024-09-16T15:50:29.409060+0000","last_fullsized":"2024-09-16T15:50:29.409060+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:28.359944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:28.359944+0000","last_clean_scrub_stamp":"2024-09-16T15:50:28.359944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T22:00:23.514722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"23.17","version":"0'0","reported_seq":1,"reported_epoch":144,"state":"creating+peering","last_fresh":"2024-09-16T15:50:29.403320+0000","last_change":"2024-09-16T15:50:29.403320+0000","last_active":"2024-09-16T15:50:29.373642+0000","last_peered":"2024-09-16T15:50:29.373642+0000","last_clean":"2024-09-16T15:50:29.373642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:50:29.403320+0000","last_undegraded":"2024-09-16T15:50:29.403320+0000","last_fullsized":"2024-09-16T15:50:29.403320+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_clean_scrub_stamp":"2024-09-16T15:50:29.373642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.16","version":"0'0","reported_seq":10,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-16T15:50:29.419433+0000","last_change":"2024-09-16T15:50:29.418819+0000","last_active":"2024-09-16T15:50:29.419433+0000","last_peered":"2024-09-16T15:50:29.419433+0000","last_clean":"2024-09-16T15:50:29.419433+0000","last_became_active":"2024-09-16T15:50:29.418365+0000","last_became_peered":"2024-09-16T15:50:29.418365+0000","last_unstale":"2024-09-16T15:50:29.419433+0000","last_undegraded":"2024-09-16T15:50:29.419433+0000","last_fullsized":"2024-09-16T15:50:29.419433+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:28.359944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:28.359944+0000","last_clean_scrub_stamp":"2024-09-16T15:50:28.359944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T23:56:48.360283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.14","version":"0'0","reported_seq":1,"reported_epoch":144,"state":"creating+peering","last_fresh":"2024-09-16T15:50:29.401245+0000","last_change":"2024-09-16T15:50:29.401245+0000","last_active":"2024-09-16T15:50:29.373642+0000","last_peered":"2024-09-16T15:50:29.373642+0000","last_clean":"2024-09-16T15:50:29.373642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:50:29.401245+0000","last_undegraded":"2024-09-16T15:50:29.401245+0000","last_fullsized":"2024-09-16T15:50:29.401245+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_clean_scrub_stamp":"2024-09-16T15:50:29.373642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.15","version":"0'0","reported_seq":10,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-16T15:50:29.417183+0000","last_change":"2024-09-16T15:50:29.414494+0000","last_active":"2024-09-16T15:50:29.417183+0000","last_peered":"2024-09-16T15:50:29.417183+0000","last_clean":"2024-09-16T15:50:29.417183+0000","last_became_active":"2024-09-16T15:50:29.413903+0000","last_became_peered":"2024-09-16T15:50:29.413903+0000","last_unstale":"2024-09-16T15:50:29.417183+0000","last_undegraded":"2024-09-16T15:50:29.417183+0000","last_fullsized":"2024-09-16T15:50:29.417183+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:28.359944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:28.359944+0000","last_clean_scrub_stamp":"2024-09-16T15:50:28.359944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T00:54:21.344614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.15","version":"0'0","reported_seq":1,"reported_epoch":144,"state":"creating+peering","last_fresh":"2024-09-16T15:50:29.390531+0000","last_change":"2024-09-16T15:50:29.390531+0000","last_active":"2024-09-16T15:50:29.373642+0000","last_peered":"2024-09-16T15:50:29.373642+0000","last_clean":"2024-09-16T15:50:29.373642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:50:29.390531+0000","last_undegraded":"2024-09-16T15:50:29.390531+0000","last_fullsized":"2024-09-16T15:50:29.390531+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_clean_scrub_stamp":"2024-09-16T15:50:29.373642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.14","version":"0'0","reported_seq":10,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-16T15:50:29.420922+0000","last_change":"2024-09-16T15:50:29.406502+0000","last_active":"2024-09-16T15:50:29.420922+0000","last_peered":"2024-09-16T15:50:29.420922+0000","last_clean":"2024-09-16T15:50:29.420922+0000","last_became_active":"2024-09-16T15:50:29.404490+0000","last_became_peered":"2024-09-16T15:50:29.404490+0000","last_unstale":"2024-09-16T15:50:29.420922+0000","last_undegraded":"2024-09-16T15:50:29.420922+0000","last_fullsized":"2024-09-16T15:50:29.420922+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:28.359944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:28.359944+0000","last_clean_scrub_stamp":"2024-09-16T15:50:28.359944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T03:40:13.610653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"23.12","version":"0'0","reported_seq":1,"reported_epoch":144,"state":"creating+peering","last_fresh":"2024-09-16T15:50:29.401262+0000","last_change":"2024-09-16T15:50:29.401262+0000","last_active":"2024-09-16T15:50:29.373642+0000","last_peered":"2024-09-16T15:50:29.373642+0000","last_clean":"2024-09-16T15:50:29.373642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:50:29.401262+0000","last_undegraded":"2024-09-16T15:50:29.401262+0000","last_fullsized":"2024-09-16T15:50:29.401262+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_clean_scrub_stamp":"2024-09-16T15:50:29.373642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.13","version":"0'0","reported_seq":10,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-16T15:50:29.411283+0000","last_change":"2024-09-16T15:50:29.408013+0000","last_active":"2024-09-16T15:50:29.411283+0000","last_peered":"2024-09-16T15:50:29.411283+0000","last_clean":"2024-09-16T15:50:29.411283+0000","last_became_active":"2024-09-16T15:50:29.407703+0000","last_became_peered":"2024-09-16T15:50:29.407703+0000","last_unstale":"2024-09-16T15:50:29.411283+0000","last_undegraded":"2024-09-16T15:50:29.411283+0000","last_fullsized":"2024-09-16T15:50:29.411283+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:28.359944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:28.359944+0000","last_clean_scrub_stamp":"2024-09-16T15:50:28.359944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:05:33.631752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.13","version":"0'0","reported_seq":1,"reported_epoch":144,"state":"creating+peering","last_fresh":"2024-09-16T15:50:29.396084+0000","last_change":"2024-09-16T15:50:29.396084+0000","last_active":"2024-09-16T15:50:29.373642+0000","last_peered":"2024-09-16T15:50:29.373642+0000","last_clean":"2024-09-16T15:50:29.373642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:50:29.396084+0000","last_undegraded":"2024-09-16T15:50:29.396084+0000","last_fullsized":"2024-09-16T15:50:29.396084+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_clean_scrub_stamp":"2024-09-16T15:50:29.373642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"22.12","version":"0'0","reported_seq":10,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-16T15:50:29.411692+0000","last_change":"2024-09-16T15:50:29.410142+0000","last_active":"2024-09-16T15:50:29.411692+0000","last_peered":"2024-09-16T15:50:29.411692+0000","last_clean":"2024-09-16T15:50:29.411692+0000","last_became_active":"2024-09-16T15:50:29.409779+0000","last_became_peered":"2024-09-16T15:50:29.409779+0000","last_unstale":"2024-09-16T15:50:29.411692+0000","last_undegraded":"2024-09-16T15:50:29.411692+0000","last_fullsized":"2024-09-16T15:50:29.411692+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:28.359944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:28.359944+0000","last_clean_scrub_stamp":"2024-09-16T15:50:28.359944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:51:34.842420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"23.10","version":"0'0","reported_seq":1,"reported_epoch":144,"state":"creating+peering","last_fresh":"2024-09-16T15:50:29.396732+0000","last_change":"2024-09-16T15:50:29.396732+0000","last_active":"2024-09-16T15:50:29.373642+0000","last_peered":"2024-09-16T15:50:29.373642+0000","last_clean":"2024-09-16T15:50:29.373642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:50:29.396732+0000","last_undegraded":"2024-09-16T15:50:29.396732+0000","last_fullsized":"2024-09-16T15:50:29.396732+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_clean_scrub_stamp":"2024-09-16T15:50:29.373642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.11","version":"0'0","reported_seq":10,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-16T15:50:29.431189+0000","last_change":"2024-09-16T15:50:29.425647+0000","last_active":"2024-09-16T15:50:29.431189+0000","last_peered":"2024-09-16T15:50:29.431189+0000","last_clean":"2024-09-16T15:50:29.431189+0000","last_became_active":"2024-09-16T15:50:29.425266+0000","last_became_peered":"2024-09-16T15:50:29.425266+0000","last_unstale":"2024-09-16T15:50:29.431189+0000","last_undegraded":"2024-09-16T15:50:29.431189+0000","last_fullsized":"2024-09-16T15:50:29.431189+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:28.359944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:28.359944+0000","last_clean_scrub_stamp":"2024-09-16T15:50:28.359944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:18:55.430269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.11","version":"0'0","reported_seq":1,"reported_epoch":144,"state":"creating+peering","last_fresh":"2024-09-16T15:50:29.418052+0000","last_change":"2024-09-16T15:50:29.418052+0000","last_active":"2024-09-16T15:50:29.373642+0000","last_peered":"2024-09-16T15:50:29.373642+0000","last_clean":"2024-09-16T15:50:29.373642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:50:29.418052+0000","last_undegraded":"2024-09-16T15:50:29.418052+0000","last_fullsized":"2024-09-16T15:50:29.418052+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_clean_scrub_stamp":"2024-09-16T15:50:29.373642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.10","version":"0'0","reported_seq":10,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-16T15:50:29.421423+0000","last_change":"2024-09-16T15:50:29.408571+0000","last_active":"2024-09-16T15:50:29.421423+0000","last_peered":"2024-09-16T15:50:29.421423+0000","last_clean":"2024-09-16T15:50:29.421423+0000","last_became_active":"2024-09-16T15:50:29.407840+0000","last_became_peered":"2024-09-16T15:50:29.407840+0000","last_unstale":"2024-09-16T15:50:29.421423+0000","last_undegraded":"2024-09-16T15:50:29.421423+0000","last_fullsized":"2024-09-16T15:50:29.421423+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:28.359944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:28.359944+0000","last_clean_scrub_stamp":"2024-09-16T15:50:28.359944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:00:52.523701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"23.e","version":"0'0","reported_seq":1,"reported_epoch":144,"state":"creating+peering","last_fresh":"2024-09-16T15:50:29.396821+0000","last_change":"2024-09-16T15:50:29.396821+0000","last_active":"2024-09-16T15:50:29.373642+0000","last_peered":"2024-09-16T15:50:29.373642+0000","last_clean":"2024-09-16T15:50:29.373642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:50:29.396821+0000","last_undegraded":"2024-09-16T15:50:29.396821+0000","last_fullsized":"2024-09-16T15:50:29.396821+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_clean_scrub_stamp":"2024-09-16T15:50:29.373642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.f","version":"0'0","reported_seq":10,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-16T15:50:29.435965+0000","last_change":"2024-09-16T15:50:29.434785+0000","last_active":"2024-09-16T15:50:29.435965+0000","last_peered":"2024-09-16T15:50:29.435965+0000","last_clean":"2024-09-16T15:50:29.435965+0000","last_became_active":"2024-09-16T15:50:29.434102+0000","last_became_peered":"2024-09-16T15:50:29.434102+0000","last_unstale":"2024-09-16T15:50:29.435965+0000","last_undegraded":"2024-09-16T15:50:29.435965+0000","last_fullsized":"2024-09-16T15:50:29.435965+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:28.359944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:28.359944+0000","last_clean_scrub_stamp":"2024-09-16T15:50:28.359944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T02:25:34.661941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.f","version":"0'0","reported_seq":1,"reported_epoch":144,"state":"creating+peering","last_fresh":"2024-09-16T15:50:29.402843+0000","last_change":"2024-09-16T15:50:29.402843+0000","last_active":"2024-09-16T15:50:29.373642+0000","last_peered":"2024-09-16T15:50:29.373642+0000","last_clean":"2024-09-16T15:50:29.373642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:50:29.402843+0000","last_undegraded":"2024-09-16T15:50:29.402843+0000","last_fullsized":"2024-09-16T15:50:29.402843+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_clean_scrub_stamp":"2024-09-16T15:50:29.373642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.e","version":"0'0","reported_seq":10,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-16T15:50:29.429265+0000","last_change":"2024-09-16T15:50:29.426483+0000","last_active":"2024-09-16T15:50:29.429265+0000","last_peered":"2024-09-16T15:50:29.429265+0000","last_clean":"2024-09-16T15:50:29.429265+0000","last_became_active":"2024-09-16T15:50:29.425712+0000","last_became_peered":"2024-09-16T15:50:29.425712+0000","last_unstale":"2024-09-16T15:50:29.429265+0000","last_undegraded":"2024-09-16T15:50:29.429265+0000","last_fullsized":"2024-09-16T15:50:29.429265+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:28.359944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:28.359944+0000","last_clean_scrub_stamp":"2024-09-16T15:50:28.359944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:31:59.513487+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.c","version":"0'0","reported_seq":1,"reported_epoch":144,"state":"creating+peering","last_fresh":"2024-09-16T15:50:29.399219+0000","last_change":"2024-09-16T15:50:29.399219+0000","last_active":"2024-09-16T15:50:29.373642+0000","last_peered":"2024-09-16T15:50:29.373642+0000","last_clean":"2024-09-16T15:50:29.373642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:50:29.399219+0000","last_undegraded":"2024-09-16T15:50:29.399219+0000","last_fullsized":"2024-09-16T15:50:29.399219+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_clean_scrub_stamp":"2024-09-16T15:50:29.373642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.d","version":"0'0","reported_seq":10,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-16T15:50:29.430249+0000","last_change":"2024-09-16T15:50:29.427670+0000","last_active":"2024-09-16T15:50:29.430249+0000","last_peered":"2024-09-16T15:50:29.430249+0000","last_clean":"2024-09-16T15:50:29.430249+0000","last_became_active":"2024-09-16T15:50:29.426920+0000","last_became_peered":"2024-09-16T15:50:29.426920+0000","last_unstale":"2024-09-16T15:50:29.430249+0000","last_undegraded":"2024-09-16T15:50:29.430249+0000","last_fullsized":"2024-09-16T15:50:29.430249+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:28.359944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:28.359944+0000","last_clean_scrub_stamp":"2024-09-16T15:50:28.359944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:53:40.560984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.d","version":"0'0","reported_seq":1,"reported_epoch":144,"state":"creating+peering","last_fresh":"2024-09-16T15:50:29.395185+0000","last_change":"2024-09-16T15:50:29.395185+0000","last_active":"2024-09-16T15:50:29.373642+0000","last_peered":"2024-09-16T15:50:29.373642+0000","last_clean":"2024-09-16T15:50:29.373642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:50:29.395185+0000","last_undegraded":"2024-09-16T15:50:29.395185+0000","last_fullsized":"2024-09-16T15:50:29.395185+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_clean_scrub_stamp":"2024-09-16T15:50:29.373642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"22.c","version":"0'0","reported_seq":10,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-16T15:50:29.430128+0000","last_change":"2024-09-16T15:50:29.427240+0000","last_active":"2024-09-16T15:50:29.430128+0000","last_peered":"2024-09-16T15:50:29.430128+0000","last_clean":"2024-09-16T15:50:29.430128+0000","last_became_active":"2024-09-16T15:50:29.423280+0000","last_became_peered":"2024-09-16T15:50:29.423280+0000","last_unstale":"2024-09-16T15:50:29.430128+0000","last_undegraded":"2024-09-16T15:50:29.430128+0000","last_fullsized":"2024-09-16T15:50:29.430128+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:28.359944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:28.359944+0000","last_clean_scrub_stamp":"2024-09-16T15:50:28.359944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:36:54.628281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.a","version":"0'0","reported_seq":1,"reported_epoch":144,"state":"creating+peering","last_fresh":"2024-09-16T15:50:29.400460+0000","last_change":"2024-09-16T15:50:29.400460+0000","last_active":"2024-09-16T15:50:29.373642+0000","last_peered":"2024-09-16T15:50:29.373642+0000","last_clean":"2024-09-16T15:50:29.373642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:50:29.400460+0000","last_undegraded":"2024-09-16T15:50:29.400460+0000","last_fullsized":"2024-09-16T15:50:29.400460+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_clean_scrub_stamp":"2024-09-16T15:50:29.373642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.b","version":"0'0","reported_seq":10,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-16T15:50:29.431651+0000","last_change":"2024-09-16T15:50:29.425798+0000","last_active":"2024-09-16T15:50:29.431651+0000","last_peered":"2024-09-16T15:50:29.431651+0000","last_clean":"2024-09-16T15:50:29.431651+0000","last_became_active":"2024-09-16T15:50:29.425459+0000","last_became_peered":"2024-09-16T15:50:29.425459+0000","last_unstale":"2024-09-16T15:50:29.431651+0000","last_undegraded":"2024-09-16T15:50:29.431651+0000","last_fullsized":"2024-09-16T15:50:29.431651+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:28.359944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:28.359944+0000","last_clean_scrub_stamp":"2024-09-16T15:50:28.359944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:59:55.828797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.b","version":"0'0","reported_seq":1,"reported_epoch":144,"state":"creating+peering","last_fresh":"2024-09-16T15:50:29.394330+0000","last_change":"2024-09-16T15:50:29.394330+0000","last_active":"2024-09-16T15:50:29.373642+0000","last_peered":"2024-09-16T15:50:29.373642+0000","last_clean":"2024-09-16T15:50:29.373642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:50:29.394330+0000","last_undegraded":"2024-09-16T15:50:29.394330+0000","last_fullsized":"2024-09-16T15:50:29.394330+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_clean_scrub_stamp":"2024-09-16T15:50:29.373642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"22.a","version":"0'0","reported_seq":10,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-16T15:50:29.429851+0000","last_change":"2024-09-16T15:50:29.426752+0000","last_active":"2024-09-16T15:50:29.429851+0000","last_peered":"2024-09-16T15:50:29.429851+0000","last_clean":"2024-09-16T15:50:29.429851+0000","last_became_active":"2024-09-16T15:50:29.422755+0000","last_became_peered":"2024-09-16T15:50:29.422755+0000","last_unstale":"2024-09-16T15:50:29.429851+0000","last_undegraded":"2024-09-16T15:50:29.429851+0000","last_fullsized":"2024-09-16T15:50:29.429851+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:28.359944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:28.359944+0000","last_clean_scrub_stamp":"2024-09-16T15:50:28.359944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T18:14:40.770591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.8","version":"0'0","reported_seq":1,"reported_epoch":144,"state":"creating+peering","last_fresh":"2024-09-16T15:50:29.397861+0000","last_change":"2024-09-16T15:50:29.397861+0000","last_active":"2024-09-16T15:50:29.373642+0000","last_peered":"2024-09-16T15:50:29.373642+0000","last_clean":"2024-09-16T15:50:29.373642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:50:29.397861+0000","last_undegraded":"2024-09-16T15:50:29.397861+0000","last_fullsized":"2024-09-16T15:50:29.397861+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_clean_scrub_stamp":"2024-09-16T15:50:29.373642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.9","version":"0'0","reported_seq":10,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-16T15:50:29.424612+0000","last_change":"2024-09-16T15:50:29.423325+0000","last_active":"2024-09-16T15:50:29.424612+0000","last_peered":"2024-09-16T15:50:29.424612+0000","last_clean":"2024-09-16T15:50:29.424612+0000","last_became_active":"2024-09-16T15:50:29.423053+0000","last_became_peered":"2024-09-16T15:50:29.423053+0000","last_unstale":"2024-09-16T15:50:29.424612+0000","last_undegraded":"2024-09-16T15:50:29.424612+0000","last_fullsized":"2024-09-16T15:50:29.424612+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:28.359944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:28.359944+0000","last_clean_scrub_stamp":"2024-09-16T15:50:28.359944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T22:05:53.341097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.9","version":"0'0","reported_seq":1,"reported_epoch":144,"state":"creating+peering","last_fresh":"2024-09-16T15:50:29.396384+0000","last_change":"2024-09-16T15:50:29.396384+0000","last_active":"2024-09-16T15:50:29.373642+0000","last_peered":"2024-09-16T15:50:29.373642+0000","last_clean":"2024-09-16T15:50:29.373642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:50:29.396384+0000","last_undegraded":"2024-09-16T15:50:29.396384+0000","last_fullsized":"2024-09-16T15:50:29.396384+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_clean_scrub_stamp":"2024-09-16T15:50:29.373642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.8","version":"0'0","reported_seq":10,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-16T15:50:29.421906+0000","last_change":"2024-09-16T15:50:29.418765+0000","last_active":"2024-09-16T15:50:29.421906+0000","last_peered":"2024-09-16T15:50:29.421906+0000","last_clean":"2024-09-16T15:50:29.421906+0000","last_became_active":"2024-09-16T15:50:29.417737+0000","last_became_peered":"2024-09-16T15:50:29.417737+0000","last_unstale":"2024-09-16T15:50:29.421906+0000","last_undegraded":"2024-09-16T15:50:29.421906+0000","last_fullsized":"2024-09-16T15:50:29.421906+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:28.359944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:28.359944+0000","last_clean_scrub_stamp":"2024-09-16T15:50:28.359944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T00:16:50.967923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.6","version":"0'0","reported_seq":1,"reported_epoch":144,"state":"creating+peering","last_fresh":"2024-09-16T15:50:29.400497+0000","last_change":"2024-09-16T15:50:29.400497+0000","last_active":"2024-09-16T15:50:29.373642+0000","last_peered":"2024-09-16T15:50:29.373642+0000","last_clean":"2024-09-16T15:50:29.373642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:50:29.400497+0000","last_undegraded":"2024-09-16T15:50:29.400497+0000","last_fullsized":"2024-09-16T15:50:29.400497+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_clean_scrub_stamp":"2024-09-16T15:50:29.373642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.7","version":"0'0","reported_seq":10,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-16T15:50:29.423515+0000","last_change":"2024-09-16T15:50:29.422588+0000","last_active":"2024-09-16T15:50:29.423515+0000","last_peered":"2024-09-16T15:50:29.423515+0000","last_clean":"2024-09-16T15:50:29.423515+0000","last_became_active":"2024-09-16T15:50:29.422240+0000","last_became_peered":"2024-09-16T15:50:29.422240+0000","last_unstale":"2024-09-16T15:50:29.423515+0000","last_undegraded":"2024-09-16T15:50:29.423515+0000","last_fullsized":"2024-09-16T15:50:29.423515+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:28.359944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:28.359944+0000","last_clean_scrub_stamp":"2024-09-16T15:50:28.359944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T01:48:32.528831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"23.7","version":"0'0","reported_seq":1,"reported_epoch":144,"state":"creating+peering","last_fresh":"2024-09-16T15:50:29.394083+0000","last_change":"2024-09-16T15:50:29.394083+0000","last_active":"2024-09-16T15:50:29.373642+0000","last_peered":"2024-09-16T15:50:29.373642+0000","last_clean":"2024-09-16T15:50:29.373642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:50:29.394083+0000","last_undegraded":"2024-09-16T15:50:29.394083+0000","last_fullsized":"2024-09-16T15:50:29.394083+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_clean_scrub_stamp":"2024-09-16T15:50:29.373642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"22.6","version":"0'0","reported_seq":10,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-16T15:50:29.428409+0000","last_change":"2024-09-16T15:50:29.424687+0000","last_active":"2024-09-16T15:50:29.428409+0000","last_peered":"2024-09-16T15:50:29.428409+0000","last_clean":"2024-09-16T15:50:29.428409+0000","last_became_active":"2024-09-16T15:50:29.423323+0000","last_became_peered":"2024-09-16T15:50:29.423323+0000","last_unstale":"2024-09-16T15:50:29.428409+0000","last_undegraded":"2024-09-16T15:50:29.428409+0000","last_fullsized":"2024-09-16T15:50:29.428409+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:28.359944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:28.359944+0000","last_clean_scrub_stamp":"2024-09-16T15:50:28.359944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:26:50.321081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.4","version":"0'0","reported_seq":1,"reported_epoch":144,"state":"creating+peering","last_fresh":"2024-09-16T15:50:29.395179+0000","last_change":"2024-09-16T15:50:29.395179+0000","last_active":"2024-09-16T15:50:29.373642+0000","last_peered":"2024-09-16T15:50:29.373642+0000","last_clean":"2024-09-16T15:50:29.373642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:50:29.395179+0000","last_undegraded":"2024-09-16T15:50:29.395179+0000","last_fullsized":"2024-09-16T15:50:29.395179+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_clean_scrub_stamp":"2024-09-16T15:50:29.373642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.5","version":"0'0","reported_seq":10,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-16T15:50:29.429727+0000","last_change":"2024-09-16T15:50:29.421716+0000","last_active":"2024-09-16T15:50:29.429727+0000","last_peered":"2024-09-16T15:50:29.429727+0000","last_clean":"2024-09-16T15:50:29.429727+0000","last_became_active":"2024-09-16T15:50:29.420529+0000","last_became_peered":"2024-09-16T15:50:29.420529+0000","last_unstale":"2024-09-16T15:50:29.429727+0000","last_undegraded":"2024-09-16T15:50:29.429727+0000","last_fullsized":"2024-09-16T15:50:29.429727+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:28.359944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:28.359944+0000","last_clean_scrub_stamp":"2024-09-16T15:50:28.359944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T03:46:32.264684+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.5","version":"0'0","reported_seq":1,"reported_epoch":144,"state":"creating+peering","last_fresh":"2024-09-16T15:50:29.400308+0000","last_change":"2024-09-16T15:50:29.400308+0000","last_active":"2024-09-16T15:50:29.373642+0000","last_peered":"2024-09-16T15:50:29.373642+0000","last_clean":"2024-09-16T15:50:29.373642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:50:29.400308+0000","last_undegraded":"2024-09-16T15:50:29.400308+0000","last_fullsized":"2024-09-16T15:50:29.400308+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_clean_scrub_stamp":"2024-09-16T15:50:29.373642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.4","version":"0'0","reported_seq":10,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-16T15:50:29.417851+0000","last_change":"2024-09-16T15:50:29.417210+0000","last_active":"2024-09-16T15:50:29.417851+0000","last_peered":"2024-09-16T15:50:29.417851+0000","last_clean":"2024-09-16T15:50:29.417851+0000","last_became_active":"2024-09-16T15:50:29.416831+0000","last_became_peered":"2024-09-16T15:50:29.416831+0000","last_unstale":"2024-09-16T15:50:29.417851+0000","last_undegraded":"2024-09-16T15:50:29.417851+0000","last_fullsized":"2024-09-16T15:50:29.417851+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:28.359944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:28.359944+0000","last_clean_scrub_stamp":"2024-09-16T15:50:28.359944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:00:59.918701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"23.2","version":"0'0","reported_seq":1,"reported_epoch":144,"state":"creating+peering","last_fresh":"2024-09-16T15:50:29.400856+0000","last_change":"2024-09-16T15:50:29.400856+0000","last_active":"2024-09-16T15:50:29.373642+0000","last_peered":"2024-09-16T15:50:29.373642+0000","last_clean":"2024-09-16T15:50:29.373642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:50:29.400856+0000","last_undegraded":"2024-09-16T15:50:29.400856+0000","last_fullsized":"2024-09-16T15:50:29.400856+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_clean_scrub_stamp":"2024-09-16T15:50:29.373642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.3","version":"0'0","reported_seq":10,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-16T15:50:29.421008+0000","last_change":"2024-09-16T15:50:29.407104+0000","last_active":"2024-09-16T15:50:29.421008+0000","last_peered":"2024-09-16T15:50:29.421008+0000","last_clean":"2024-09-16T15:50:29.421008+0000","last_became_active":"2024-09-16T15:50:29.406182+0000","last_became_peered":"2024-09-16T15:50:29.406182+0000","last_unstale":"2024-09-16T15:50:29.421008+0000","last_undegraded":"2024-09-16T15:50:29.421008+0000","last_fullsized":"2024-09-16T15:50:29.421008+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:28.359944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:28.359944+0000","last_clean_scrub_stamp":"2024-09-16T15:50:28.359944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:36:55.485971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"23.3","version":"0'0","reported_seq":1,"reported_epoch":144,"state":"creating+peering","last_fresh":"2024-09-16T15:50:29.391321+0000","last_change":"2024-09-16T15:50:29.391321+0000","last_active":"2024-09-16T15:50:29.373642+0000","last_peered":"2024-09-16T15:50:29.373642+0000","last_clean":"2024-09-16T15:50:29.373642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:50:29.391321+0000","last_undegraded":"2024-09-16T15:50:29.391321+0000","last_fullsized":"2024-09-16T15:50:29.391321+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_clean_scrub_stamp":"2024-09-16T15:50:29.373642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.2","version":"0'0","reported_seq":10,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-16T15:50:29.411842+0000","last_change":"2024-09-16T15:50:29.410821+0000","last_active":"2024-09-16T15:50:29.411842+0000","last_peered":"2024-09-16T15:50:29.411842+0000","last_clean":"2024-09-16T15:50:29.411842+0000","last_became_active":"2024-09-16T15:50:29.410554+0000","last_became_peered":"2024-09-16T15:50:29.410554+0000","last_unstale":"2024-09-16T15:50:29.411842+0000","last_undegraded":"2024-09-16T15:50:29.411842+0000","last_fullsized":"2024-09-16T15:50:29.411842+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:28.359944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:28.359944+0000","last_clean_scrub_stamp":"2024-09-16T15:50:28.359944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:55:15.802642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.0","version":"0'0","reported_seq":1,"reported_epoch":144,"state":"creating+peering","last_fresh":"2024-09-16T15:50:29.394199+0000","last_change":"2024-09-16T15:50:29.394199+0000","last_active":"2024-09-16T15:50:29.373642+0000","last_peered":"2024-09-16T15:50:29.373642+0000","last_clean":"2024-09-16T15:50:29.373642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:50:29.394199+0000","last_undegraded":"2024-09-16T15:50:29.394199+0000","last_fullsized":"2024-09-16T15:50:29.394199+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_clean_scrub_stamp":"2024-09-16T15:50:29.373642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"22.1","version":"0'0","reported_seq":10,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-16T15:50:29.421042+0000","last_change":"2024-09-16T15:50:29.416751+0000","last_active":"2024-09-16T15:50:29.421042+0000","last_peered":"2024-09-16T15:50:29.421042+0000","last_clean":"2024-09-16T15:50:29.421042+0000","last_became_active":"2024-09-16T15:50:29.415995+0000","last_became_peered":"2024-09-16T15:50:29.415995+0000","last_unstale":"2024-09-16T15:50:29.421042+0000","last_undegraded":"2024-09-16T15:50:29.421042+0000","last_fullsized":"2024-09-16T15:50:29.421042+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:28.359944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:28.359944+0000","last_clean_scrub_stamp":"2024-09-16T15:50:28.359944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T02:16:57.915820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.1","version":"0'0","reported_seq":1,"reported_epoch":144,"state":"creating+peering","last_fresh":"2024-09-16T15:50:29.409753+0000","last_change":"2024-09-16T15:50:29.409753+0000","last_active":"2024-09-16T15:50:29.373642+0000","last_peered":"2024-09-16T15:50:29.373642+0000","last_clean":"2024-09-16T15:50:29.373642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:50:29.409753+0000","last_undegraded":"2024-09-16T15:50:29.409753+0000","last_fullsized":"2024-09-16T15:50:29.409753+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_clean_scrub_stamp":"2024-09-16T15:50:29.373642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.0","version":"0'0","reported_seq":10,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-16T15:50:29.406562+0000","last_change":"2024-09-16T15:50:29.395361+0000","last_active":"2024-09-16T15:50:29.406562+0000","last_peered":"2024-09-16T15:50:29.406562+0000","last_clean":"2024-09-16T15:50:29.406562+0000","last_became_active":"2024-09-16T15:50:29.394578+0000","last_became_peered":"2024-09-16T15:50:29.394578+0000","last_unstale":"2024-09-16T15:50:29.406562+0000","last_undegraded":"2024-09-16T15:50:29.406562+0000","last_fullsized":"2024-09-16T15:50:29.406562+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:28.359944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:28.359944+0000","last_clean_scrub_stamp":"2024-09-16T15:50:28.359944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T02:06:32.966504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"23.16","version":"0'0","reported_seq":1,"reported_epoch":144,"state":"creating+peering","last_fresh":"2024-09-16T15:50:29.401057+0000","last_change":"2024-09-16T15:50:29.401057+0000","last_active":"2024-09-16T15:50:29.373642+0000","last_peered":"2024-09-16T15:50:29.373642+0000","last_clean":"2024-09-16T15:50:29.373642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:50:29.401057+0000","last_undegraded":"2024-09-16T15:50:29.401057+0000","last_fullsized":"2024-09-16T15:50:29.401057+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_clean_scrub_stamp":"2024-09-16T15:50:29.373642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.17","version":"0'0","reported_seq":10,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-16T15:50:29.421257+0000","last_change":"2024-09-16T15:50:29.410992+0000","last_active":"2024-09-16T15:50:29.421257+0000","last_peered":"2024-09-16T15:50:29.421257+0000","last_clean":"2024-09-16T15:50:29.421257+0000","last_became_active":"2024-09-16T15:50:29.410215+0000","last_became_peered":"2024-09-16T15:50:29.410215+0000","last_unstale":"2024-09-16T15:50:29.421257+0000","last_undegraded":"2024-09-16T15:50:29.421257+0000","last_fullsized":"2024-09-16T15:50:29.421257+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:28.359944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:28.359944+0000","last_clean_scrub_stamp":"2024-09-16T15:50:28.359944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T00:24:19.013129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"73'79","reported_seq":366,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-16T15:50:29.402844+0000","last_change":"2024-09-16T15:42:40.881820+0000","last_active":"2024-09-16T15:50:29.402844+0000","last_peered":"2024-09-16T15:50:29.402844+0000","last_clean":"2024-09-16T15:50:29.402844+0000","last_became_active":"2024-09-16T15:42:40.879965+0000","last_became_peered":"2024-09-16T15:42:40.879965+0000","last_unstale":"2024-09-16T15:50:29.402844+0000","last_undegraded":"2024-09-16T15:50:29.402844+0000","last_fullsized":"2024-09-16T15:50:29.402844+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:42:39.728762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:42:39.728762+0000","last_clean_scrub_stamp":"2024-09-16T15:42:39.728762+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:22:18.768932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":23,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":22,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79,"ondisk_log_size":79,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542242,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33536,"kb_used_data":2344,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338304,"statfs":{"total":96636764160,"available":96602423296,"internally_reserved":0,"allocated":2400256,"data_stored":1389123,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542241,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32956,"kb_used_data":1764,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338884,"statfs":{"total":96636764160,"available":96603017216,"internally_reserved":0,"allocated":1806336,"data_stored":798755,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":5,"up_from":14,"seq":60129542242,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32940,"kb_used_data":1752,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338900,"statfs":{"total":96636764160,"available":96603033600,"internally_reserved":0,"allocated":1794048,"data_stored":792452,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542240,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32940,"kb_used_data":1752,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338900,"statfs":{"total":96636764160,"available":96603033600,"internally_reserved":0,"allocated":1794048,"data_stored":792452,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":3,"up_from":14,"seq":60129542241,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37100,"kb_used_data":1752,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334740,"statfs":{"total":96636764160,"available":96598773760,"internally_reserved":0,"allocated":1794048,"data_stored":792452,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":14,"seq":60129542241,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32956,"kb_used_data":1764,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338884,"statfs":{"total":96636764160,"available":96603017216,"internally_reserved":0,"allocated":1806336,"data_stored":798755,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":14,"seq":60129542243,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32956,"kb_used_data":1764,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338884,"statfs":{"total":96636764160,"available":96603017216,"internally_reserved":0,"allocated":1806336,"data_stored":798755,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":14,"seq":60129542243,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33520,"kb_used_data":2332,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338320,"statfs":{"total":96636764160,"available":96602439680,"internally_reserved":0,"allocated":2387968,"data_stored":1382820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T15:50:34.820 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T15:50:34.972 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:34.970+0000 7fab8a03e640 1 -- 172.21.15.123:0/1706393126 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fab8415e230 msgr2=0x7fab840cd6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:50:34.972 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:34.970+0000 7fab8a03e640 1 --2- 172.21.15.123:0/1706393126 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fab8415e230 0x7fab840cd6e0 secure :-1 s=READY pgs=1596 cs=0 l=1 rev1=1 crypto rx=0x7fab6c009840 tx=0x7fab6c01c820 comp rx=0 tx=0).stop 2024-09-16T15:50:34.972 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:34.970+0000 7fab8a03e640 1 -- 172.21.15.123:0/1706393126 shutdown_connections 2024-09-16T15:50:34.972 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:34.970+0000 7fab8a03e640 1 --2- 172.21.15.123:0/1706393126 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fab8415e230 0x7fab840cd6e0 unknown :-1 s=CLOSED pgs=1596 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:34.972 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:34.970+0000 7fab8a03e640 1 --2- 172.21.15.123:0/1706393126 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fab8415de50 0x7fab84162470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:34.972 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:34.970+0000 7fab8a03e640 1 -- 172.21.15.123:0/1706393126 >> 172.21.15.123:0/1706393126 conn(0x7fab840cfb10 msgr2=0x7fab840cff20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:50:34.972 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:34.970+0000 7fab8a03e640 1 -- 172.21.15.123:0/1706393126 shutdown_connections 2024-09-16T15:50:34.972 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:34.970+0000 7fab8a03e640 1 -- 172.21.15.123:0/1706393126 wait complete. 2024-09-16T15:50:34.973 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:34.974+0000 7fab8a03e640 1 Processor -- start 2024-09-16T15:50:34.973 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:34.974+0000 7fab8a03e640 1 -- start start 2024-09-16T15:50:34.975 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:34.974+0000 7fab8a03e640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fab840701d0 0x7fab8415a210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:50:34.975 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:34.974+0000 7fab8a03e640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fab8415de50 0x7fab84152680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:50:34.975 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:34.974+0000 7fab8a03e640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fab8415e230 0x7fab841f7fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:50:34.975 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:34.974+0000 7fab8a03e640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fab8416b3a0 con 0x7fab840701d0 2024-09-16T15:50:34.975 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:34.974+0000 7fab8a03e640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fab8416b220 con 0x7fab8415de50 2024-09-16T15:50:34.976 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:34.974+0000 7fab8a03e640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fab8416b520 con 0x7fab8415e230 2024-09-16T15:50:34.976 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:34.974+0000 7fab837fe640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fab840701d0 0x7fab8415a210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:50:34.976 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:34.974+0000 7fab837fe640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fab840701d0 0x7fab8415a210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:49690/0 (socket says 172.21.15.123:49690) 2024-09-16T15:50:34.976 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:34.974+0000 7fab837fe640 1 -- 172.21.15.123:0/1315550884 learned_addr learned my addr 172.21.15.123:0/1315550884 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:50:34.976 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:34.974+0000 7fab82ffd640 1 --2- 172.21.15.123:0/1315550884 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fab8415de50 0x7fab84152680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:50:34.976 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:34.974+0000 7fab83fff640 1 --2- 172.21.15.123:0/1315550884 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fab8415e230 0x7fab841f7fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:50:34.976 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:34.974+0000 7fab837fe640 1 -- 172.21.15.123:0/1315550884 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fab8415e230 msgr2=0x7fab841f7fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:50:34.976 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:34.974+0000 7fab837fe640 1 --2- 172.21.15.123:0/1315550884 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fab8415e230 0x7fab841f7fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:34.976 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:34.974+0000 7fab837fe640 1 -- 172.21.15.123:0/1315550884 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fab8415de50 msgr2=0x7fab84152680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:50:34.976 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:34.974+0000 7fab837fe640 1 --2- 172.21.15.123:0/1315550884 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fab8415de50 0x7fab84152680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:34.976 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:34.974+0000 7fab837fe640 1 -- 172.21.15.123:0/1315550884 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fab841f7810 con 0x7fab840701d0 2024-09-16T15:50:34.976 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:34.974+0000 7fab82ffd640 1 --2- 172.21.15.123:0/1315550884 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fab8415de50 0x7fab84152680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T15:50:34.976 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:34.974+0000 7fab83fff640 1 --2- 172.21.15.123:0/1315550884 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fab8415e230 0x7fab841f7fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T15:50:34.977 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:34.974+0000 7fab837fe640 1 --2- 172.21.15.123:0/1315550884 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fab840701d0 0x7fab8415a210 secure :-1 s=READY pgs=1597 cs=0 l=1 rev1=1 crypto rx=0x7fab7c009fd0 tx=0x7fab7c00c270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:50:34.977 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:34.974+0000 7fab7bfff640 1 -- 172.21.15.123:0/1315550884 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab7c011470 con 0x7fab840701d0 2024-09-16T15:50:34.977 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:34.974+0000 7fab7bfff640 1 -- 172.21.15.123:0/1315550884 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fab7c011700 con 0x7fab840701d0 2024-09-16T15:50:34.977 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:34.974+0000 7fab7bfff640 1 -- 172.21.15.123:0/1315550884 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab7c011b30 con 0x7fab840701d0 2024-09-16T15:50:34.977 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:34.974+0000 7fab8a03e640 1 -- 172.21.15.123:0/1315550884 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fab8415a890 con 0x7fab840701d0 2024-09-16T15:50:34.977 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:34.974+0000 7fab8a03e640 1 -- 172.21.15.123:0/1315550884 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fab8415ae50 con 0x7fab840701d0 2024-09-16T15:50:34.977 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:34.974+0000 7fab8a03e640 1 -- 172.21.15.123:0/1315550884 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fab84162470 con 0x7fab840701d0 2024-09-16T15:50:34.978 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:34.978+0000 7fab7bfff640 1 -- 172.21.15.123:0/1315550884 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 22) ==== 97692+0+0 (secure 0 0 0) 0x7fab7c011cd0 con 0x7fab840701d0 2024-09-16T15:50:34.978 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:34.978+0000 7fab7bfff640 1 --2- 172.21.15.123:0/1315550884 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fab680735b0 0x7fab68075a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:50:34.978 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:34.978+0000 7fab82ffd640 1 --2- 172.21.15.123:0/1315550884 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fab680735b0 0x7fab68075a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:50:34.978 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:34.978+0000 7fab7bfff640 1 -- 172.21.15.123:0/1315550884 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(146..146 src has 1..146) ==== 5737+0+0 (secure 0 0 0) 0x7fab7c095830 con 0x7fab840701d0 2024-09-16T15:50:34.978 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:34.978+0000 7fab82ffd640 1 --2- 172.21.15.123:0/1315550884 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fab680735b0 0x7fab68075a70 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7fab6c0af000 tx=0x7fab6c01b060 comp rx=0 tx=0).ready entity=mgr.4150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:50:34.980 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:34.982+0000 7fab7bfff640 1 -- 172.21.15.123:0/1315550884 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7fab7c016030 con 0x7fab840701d0 2024-09-16T15:50:35.147 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:35.146+0000 7fab8a03e640 1 -- 172.21.15.123:0/1315550884 --> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fab8415b170 con 0x7fab680735b0 2024-09-16T15:50:35.157 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:35.158+0000 7fab7bfff640 1 -- 172.21.15.123:0/1315550884 <== mgr.4150 v2:172.21.15.123:6836/2464759676 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+247115 (secure 0 0 0) 0x7fab8415b170 con 0x7fab680735b0 2024-09-16T15:50:35.160 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-09-16T15:50:35.161 INFO:teuthology.orchestra.run.smithi123.stderr:dumped all 2024-09-16T15:50:35.162 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:35.162+0000 7fab8a03e640 1 -- 172.21.15.123:0/1315550884 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fab680735b0 msgr2=0x7fab68075a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:50:35.162 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:35.162+0000 7fab8a03e640 1 --2- 172.21.15.123:0/1315550884 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fab680735b0 0x7fab68075a70 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7fab6c0af000 tx=0x7fab6c01b060 comp rx=0 tx=0).stop 2024-09-16T15:50:35.162 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:35.162+0000 7fab8a03e640 1 -- 172.21.15.123:0/1315550884 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fab840701d0 msgr2=0x7fab8415a210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:50:35.163 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:35.162+0000 7fab8a03e640 1 --2- 172.21.15.123:0/1315550884 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fab840701d0 0x7fab8415a210 secure :-1 s=READY pgs=1597 cs=0 l=1 rev1=1 crypto rx=0x7fab7c009fd0 tx=0x7fab7c00c270 comp rx=0 tx=0).stop 2024-09-16T15:50:35.163 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:35.162+0000 7fab8a03e640 1 -- 172.21.15.123:0/1315550884 shutdown_connections 2024-09-16T15:50:35.163 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:35.162+0000 7fab8a03e640 1 --2- 172.21.15.123:0/1315550884 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fab680735b0 0x7fab68075a70 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:35.163 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:35.162+0000 7fab8a03e640 1 --2- 172.21.15.123:0/1315550884 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fab8415e230 0x7fab841f7fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:35.163 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:35.162+0000 7fab8a03e640 1 --2- 172.21.15.123:0/1315550884 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fab8415de50 0x7fab84152680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:35.163 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:35.162+0000 7fab8a03e640 1 --2- 172.21.15.123:0/1315550884 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fab840701d0 0x7fab8415a210 unknown :-1 s=CLOSED pgs=1597 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:35.163 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:35.162+0000 7fab8a03e640 1 -- 172.21.15.123:0/1315550884 >> 172.21.15.123:0/1315550884 conn(0x7fab840cfb10 msgr2=0x7fab84153f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:50:35.163 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:35.162+0000 7fab8a03e640 1 -- 172.21.15.123:0/1315550884 shutdown_connections 2024-09-16T15:50:35.163 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:35.162+0000 7fab8a03e640 1 -- 172.21.15.123:0/1315550884 wait complete. 2024-09-16T15:50:35.178 INFO:teuthology.orchestra.run.smithi123.stdout:{"pg_ready":true,"pg_map":{"version":167,"stamp":"2024-09-16T15:50:34.119725+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79,"ondisk_log_size":79,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":39,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":268904,"kb_used_data":15224,"kb_used_omap":12,"kb_used_meta":253491,"kb_avail":754705816,"statfs":{"total":773094113280,"available":772818755584,"internally_reserved":0,"allocated":15589376,"data_stored":7545564,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":259575345},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":29,"apply_latency_ms":29,"commit_latency_ns":29000000,"apply_latency_ns":29000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.743992"},"pg_stats":[{"pgid":"23.3f","version":"0'0","reported_seq":1,"reported_epoch":144,"state":"creating+peering","last_fresh":"2024-09-16T15:50:29.396826+0000","last_change":"2024-09-16T15:50:29.396826+0000","last_active":"2024-09-16T15:50:29.373642+0000","last_peered":"2024-09-16T15:50:29.373642+0000","last_clean":"2024-09-16T15:50:29.373642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:50:29.396826+0000","last_undegraded":"2024-09-16T15:50:29.396826+0000","last_fullsized":"2024-09-16T15:50:29.396826+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_clean_scrub_stamp":"2024-09-16T15:50:29.373642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.3e","version":"0'0","reported_seq":1,"reported_epoch":144,"state":"creating+peering","last_fresh":"2024-09-16T15:50:29.405611+0000","last_change":"2024-09-16T15:50:29.405611+0000","last_active":"2024-09-16T15:50:29.373642+0000","last_peered":"2024-09-16T15:50:29.373642+0000","last_clean":"2024-09-16T15:50:29.373642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:50:29.405611+0000","last_undegraded":"2024-09-16T15:50:29.405611+0000","last_fullsized":"2024-09-16T15:50:29.405611+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_clean_scrub_stamp":"2024-09-16T15:50:29.373642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.3d","version":"0'0","reported_seq":1,"reported_epoch":144,"state":"creating+peering","last_fresh":"2024-09-16T15:50:29.414816+0000","last_change":"2024-09-16T15:50:29.414816+0000","last_active":"2024-09-16T15:50:29.373642+0000","last_peered":"2024-09-16T15:50:29.373642+0000","last_clean":"2024-09-16T15:50:29.373642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:50:29.414816+0000","last_undegraded":"2024-09-16T15:50:29.414816+0000","last_fullsized":"2024-09-16T15:50:29.414816+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_clean_scrub_stamp":"2024-09-16T15:50:29.373642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.3c","version":"0'0","reported_seq":1,"reported_epoch":144,"state":"creating+peering","last_fresh":"2024-09-16T15:50:29.400948+0000","last_change":"2024-09-16T15:50:29.400948+0000","last_active":"2024-09-16T15:50:29.373642+0000","last_peered":"2024-09-16T15:50:29.373642+0000","last_clean":"2024-09-16T15:50:29.373642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:50:29.400948+0000","last_undegraded":"2024-09-16T15:50:29.400948+0000","last_fullsized":"2024-09-16T15:50:29.400948+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_clean_scrub_stamp":"2024-09-16T15:50:29.373642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.3b","version":"0'0","reported_seq":1,"reported_epoch":144,"state":"creating+peering","last_fresh":"2024-09-16T15:50:29.408612+0000","last_change":"2024-09-16T15:50:29.408612+0000","last_active":"2024-09-16T15:50:29.373642+0000","last_peered":"2024-09-16T15:50:29.373642+0000","last_clean":"2024-09-16T15:50:29.373642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:50:29.408612+0000","last_undegraded":"2024-09-16T15:50:29.408612+0000","last_fullsized":"2024-09-16T15:50:29.408612+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_clean_scrub_stamp":"2024-09-16T15:50:29.373642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.3a","version":"0'0","reported_seq":1,"reported_epoch":144,"state":"creating+peering","last_fresh":"2024-09-16T15:50:29.399825+0000","last_change":"2024-09-16T15:50:29.399825+0000","last_active":"2024-09-16T15:50:29.373642+0000","last_peered":"2024-09-16T15:50:29.373642+0000","last_clean":"2024-09-16T15:50:29.373642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:50:29.399825+0000","last_undegraded":"2024-09-16T15:50:29.399825+0000","last_fullsized":"2024-09-16T15:50:29.399825+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_clean_scrub_stamp":"2024-09-16T15:50:29.373642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.39","version":"0'0","reported_seq":1,"reported_epoch":144,"state":"creating+peering","last_fresh":"2024-09-16T15:50:29.397580+0000","last_change":"2024-09-16T15:50:29.397580+0000","last_active":"2024-09-16T15:50:29.373642+0000","last_peered":"2024-09-16T15:50:29.373642+0000","last_clean":"2024-09-16T15:50:29.373642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:50:29.397580+0000","last_undegraded":"2024-09-16T15:50:29.397580+0000","last_fullsized":"2024-09-16T15:50:29.397580+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_clean_scrub_stamp":"2024-09-16T15:50:29.373642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"23.38","version":"0'0","reported_seq":1,"reported_epoch":144,"state":"creating+peering","last_fresh":"2024-09-16T15:50:29.409708+0000","last_change":"2024-09-16T15:50:29.409708+0000","last_active":"2024-09-16T15:50:29.373642+0000","last_peered":"2024-09-16T15:50:29.373642+0000","last_clean":"2024-09-16T15:50:29.373642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:50:29.409708+0000","last_undegraded":"2024-09-16T15:50:29.409708+0000","last_fullsized":"2024-09-16T15:50:29.409708+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_clean_scrub_stamp":"2024-09-16T15:50:29.373642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.37","version":"0'0","reported_seq":1,"reported_epoch":144,"state":"creating+peering","last_fresh":"2024-09-16T15:50:29.406549+0000","last_change":"2024-09-16T15:50:29.406549+0000","last_active":"2024-09-16T15:50:29.373642+0000","last_peered":"2024-09-16T15:50:29.373642+0000","last_clean":"2024-09-16T15:50:29.373642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:50:29.406549+0000","last_undegraded":"2024-09-16T15:50:29.406549+0000","last_fullsized":"2024-09-16T15:50:29.406549+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_clean_scrub_stamp":"2024-09-16T15:50:29.373642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.36","version":"0'0","reported_seq":1,"reported_epoch":144,"state":"creating+peering","last_fresh":"2024-09-16T15:50:29.404538+0000","last_change":"2024-09-16T15:50:29.404538+0000","last_active":"2024-09-16T15:50:29.373642+0000","last_peered":"2024-09-16T15:50:29.373642+0000","last_clean":"2024-09-16T15:50:29.373642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:50:29.404538+0000","last_undegraded":"2024-09-16T15:50:29.404538+0000","last_fullsized":"2024-09-16T15:50:29.404538+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_clean_scrub_stamp":"2024-09-16T15:50:29.373642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.35","version":"0'0","reported_seq":1,"reported_epoch":144,"state":"creating+peering","last_fresh":"2024-09-16T15:50:29.418503+0000","last_change":"2024-09-16T15:50:29.418503+0000","last_active":"2024-09-16T15:50:29.373642+0000","last_peered":"2024-09-16T15:50:29.373642+0000","last_clean":"2024-09-16T15:50:29.373642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:50:29.418503+0000","last_undegraded":"2024-09-16T15:50:29.418503+0000","last_fullsized":"2024-09-16T15:50:29.418503+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_clean_scrub_stamp":"2024-09-16T15:50:29.373642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.34","version":"0'0","reported_seq":1,"reported_epoch":144,"state":"creating+peering","last_fresh":"2024-09-16T15:50:29.398426+0000","last_change":"2024-09-16T15:50:29.398426+0000","last_active":"2024-09-16T15:50:29.373642+0000","last_peered":"2024-09-16T15:50:29.373642+0000","last_clean":"2024-09-16T15:50:29.373642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:50:29.398426+0000","last_undegraded":"2024-09-16T15:50:29.398426+0000","last_fullsized":"2024-09-16T15:50:29.398426+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_clean_scrub_stamp":"2024-09-16T15:50:29.373642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.33","version":"0'0","reported_seq":1,"reported_epoch":144,"state":"creating+peering","last_fresh":"2024-09-16T15:50:29.415630+0000","last_change":"2024-09-16T15:50:29.415630+0000","last_active":"2024-09-16T15:50:29.373642+0000","last_peered":"2024-09-16T15:50:29.373642+0000","last_clean":"2024-09-16T15:50:29.373642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:50:29.415630+0000","last_undegraded":"2024-09-16T15:50:29.415630+0000","last_fullsized":"2024-09-16T15:50:29.415630+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_clean_scrub_stamp":"2024-09-16T15:50:29.373642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.32","version":"0'0","reported_seq":1,"reported_epoch":144,"state":"creating+peering","last_fresh":"2024-09-16T15:50:29.411622+0000","last_change":"2024-09-16T15:50:29.411622+0000","last_active":"2024-09-16T15:50:29.373642+0000","last_peered":"2024-09-16T15:50:29.373642+0000","last_clean":"2024-09-16T15:50:29.373642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:50:29.411622+0000","last_undegraded":"2024-09-16T15:50:29.411622+0000","last_fullsized":"2024-09-16T15:50:29.411622+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_clean_scrub_stamp":"2024-09-16T15:50:29.373642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.31","version":"0'0","reported_seq":1,"reported_epoch":144,"state":"creating+peering","last_fresh":"2024-09-16T15:50:29.396532+0000","last_change":"2024-09-16T15:50:29.396532+0000","last_active":"2024-09-16T15:50:29.373642+0000","last_peered":"2024-09-16T15:50:29.373642+0000","last_clean":"2024-09-16T15:50:29.373642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:50:29.396532+0000","last_undegraded":"2024-09-16T15:50:29.396532+0000","last_fullsized":"2024-09-16T15:50:29.396532+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_clean_scrub_stamp":"2024-09-16T15:50:29.373642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.30","version":"0'0","reported_seq":1,"reported_epoch":144,"state":"creating+peering","last_fresh":"2024-09-16T15:50:29.398051+0000","last_change":"2024-09-16T15:50:29.398051+0000","last_active":"2024-09-16T15:50:29.373642+0000","last_peered":"2024-09-16T15:50:29.373642+0000","last_clean":"2024-09-16T15:50:29.373642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:50:29.398051+0000","last_undegraded":"2024-09-16T15:50:29.398051+0000","last_fullsized":"2024-09-16T15:50:29.398051+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_clean_scrub_stamp":"2024-09-16T15:50:29.373642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"23.2f","version":"0'0","reported_seq":1,"reported_epoch":144,"state":"creating+peering","last_fresh":"2024-09-16T15:50:29.399338+0000","last_change":"2024-09-16T15:50:29.399338+0000","last_active":"2024-09-16T15:50:29.373642+0000","last_peered":"2024-09-16T15:50:29.373642+0000","last_clean":"2024-09-16T15:50:29.373642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:50:29.399338+0000","last_undegraded":"2024-09-16T15:50:29.399338+0000","last_fullsized":"2024-09-16T15:50:29.399338+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_clean_scrub_stamp":"2024-09-16T15:50:29.373642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.2e","version":"0'0","reported_seq":1,"reported_epoch":144,"state":"creating+peering","last_fresh":"2024-09-16T15:50:29.403155+0000","last_change":"2024-09-16T15:50:29.403155+0000","last_active":"2024-09-16T15:50:29.373642+0000","last_peered":"2024-09-16T15:50:29.373642+0000","last_clean":"2024-09-16T15:50:29.373642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:50:29.403155+0000","last_undegraded":"2024-09-16T15:50:29.403155+0000","last_fullsized":"2024-09-16T15:50:29.403155+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_clean_scrub_stamp":"2024-09-16T15:50:29.373642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"23.2d","version":"0'0","reported_seq":1,"reported_epoch":144,"state":"creating+peering","last_fresh":"2024-09-16T15:50:29.399089+0000","last_change":"2024-09-16T15:50:29.399089+0000","last_active":"2024-09-16T15:50:29.373642+0000","last_peered":"2024-09-16T15:50:29.373642+0000","last_clean":"2024-09-16T15:50:29.373642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:50:29.399089+0000","last_undegraded":"2024-09-16T15:50:29.399089+0000","last_fullsized":"2024-09-16T15:50:29.399089+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_clean_scrub_stamp":"2024-09-16T15:50:29.373642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.2c","version":"0'0","reported_seq":1,"reported_epoch":144,"state":"creating+peering","last_fresh":"2024-09-16T15:50:29.400996+0000","last_change":"2024-09-16T15:50:29.400996+0000","last_active":"2024-09-16T15:50:29.373642+0000","last_peered":"2024-09-16T15:50:29.373642+0000","last_clean":"2024-09-16T15:50:29.373642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:50:29.400996+0000","last_undegraded":"2024-09-16T15:50:29.400996+0000","last_fullsized":"2024-09-16T15:50:29.400996+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_clean_scrub_stamp":"2024-09-16T15:50:29.373642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.2b","version":"0'0","reported_seq":1,"reported_epoch":144,"state":"creating+peering","last_fresh":"2024-09-16T15:50:29.395481+0000","last_change":"2024-09-16T15:50:29.395481+0000","last_active":"2024-09-16T15:50:29.373642+0000","last_peered":"2024-09-16T15:50:29.373642+0000","last_clean":"2024-09-16T15:50:29.373642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:50:29.395481+0000","last_undegraded":"2024-09-16T15:50:29.395481+0000","last_fullsized":"2024-09-16T15:50:29.395481+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_clean_scrub_stamp":"2024-09-16T15:50:29.373642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.2a","version":"0'0","reported_seq":1,"reported_epoch":144,"state":"creating+peering","last_fresh":"2024-09-16T15:50:29.396706+0000","last_change":"2024-09-16T15:50:29.396706+0000","last_active":"2024-09-16T15:50:29.373642+0000","last_peered":"2024-09-16T15:50:29.373642+0000","last_clean":"2024-09-16T15:50:29.373642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:50:29.396706+0000","last_undegraded":"2024-09-16T15:50:29.396706+0000","last_fullsized":"2024-09-16T15:50:29.396706+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_clean_scrub_stamp":"2024-09-16T15:50:29.373642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.29","version":"0'0","reported_seq":1,"reported_epoch":144,"state":"creating+peering","last_fresh":"2024-09-16T15:50:29.397404+0000","last_change":"2024-09-16T15:50:29.397404+0000","last_active":"2024-09-16T15:50:29.373642+0000","last_peered":"2024-09-16T15:50:29.373642+0000","last_clean":"2024-09-16T15:50:29.373642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:50:29.397404+0000","last_undegraded":"2024-09-16T15:50:29.397404+0000","last_fullsized":"2024-09-16T15:50:29.397404+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_clean_scrub_stamp":"2024-09-16T15:50:29.373642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.28","version":"0'0","reported_seq":1,"reported_epoch":144,"state":"creating+peering","last_fresh":"2024-09-16T15:50:29.396706+0000","last_change":"2024-09-16T15:50:29.396706+0000","last_active":"2024-09-16T15:50:29.373642+0000","last_peered":"2024-09-16T15:50:29.373642+0000","last_clean":"2024-09-16T15:50:29.373642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:50:29.396706+0000","last_undegraded":"2024-09-16T15:50:29.396706+0000","last_fullsized":"2024-09-16T15:50:29.396706+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_clean_scrub_stamp":"2024-09-16T15:50:29.373642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.27","version":"0'0","reported_seq":1,"reported_epoch":144,"state":"creating+peering","last_fresh":"2024-09-16T15:50:29.401802+0000","last_change":"2024-09-16T15:50:29.401802+0000","last_active":"2024-09-16T15:50:29.373642+0000","last_peered":"2024-09-16T15:50:29.373642+0000","last_clean":"2024-09-16T15:50:29.373642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:50:29.401802+0000","last_undegraded":"2024-09-16T15:50:29.401802+0000","last_fullsized":"2024-09-16T15:50:29.401802+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_clean_scrub_stamp":"2024-09-16T15:50:29.373642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.26","version":"0'0","reported_seq":1,"reported_epoch":144,"state":"creating+peering","last_fresh":"2024-09-16T15:50:29.397726+0000","last_change":"2024-09-16T15:50:29.397726+0000","last_active":"2024-09-16T15:50:29.373642+0000","last_peered":"2024-09-16T15:50:29.373642+0000","last_clean":"2024-09-16T15:50:29.373642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:50:29.397726+0000","last_undegraded":"2024-09-16T15:50:29.397726+0000","last_fullsized":"2024-09-16T15:50:29.397726+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_clean_scrub_stamp":"2024-09-16T15:50:29.373642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"23.25","version":"0'0","reported_seq":1,"reported_epoch":144,"state":"creating+peering","last_fresh":"2024-09-16T15:50:29.413671+0000","last_change":"2024-09-16T15:50:29.413671+0000","last_active":"2024-09-16T15:50:29.373642+0000","last_peered":"2024-09-16T15:50:29.373642+0000","last_clean":"2024-09-16T15:50:29.373642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:50:29.413671+0000","last_undegraded":"2024-09-16T15:50:29.413671+0000","last_fullsized":"2024-09-16T15:50:29.413671+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_clean_scrub_stamp":"2024-09-16T15:50:29.373642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.24","version":"0'0","reported_seq":1,"reported_epoch":144,"state":"creating+peering","last_fresh":"2024-09-16T15:50:29.400095+0000","last_change":"2024-09-16T15:50:29.400095+0000","last_active":"2024-09-16T15:50:29.373642+0000","last_peered":"2024-09-16T15:50:29.373642+0000","last_clean":"2024-09-16T15:50:29.373642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:50:29.400095+0000","last_undegraded":"2024-09-16T15:50:29.400095+0000","last_fullsized":"2024-09-16T15:50:29.400095+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_clean_scrub_stamp":"2024-09-16T15:50:29.373642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.23","version":"0'0","reported_seq":1,"reported_epoch":144,"state":"creating+peering","last_fresh":"2024-09-16T15:50:29.403015+0000","last_change":"2024-09-16T15:50:29.403015+0000","last_active":"2024-09-16T15:50:29.373642+0000","last_peered":"2024-09-16T15:50:29.373642+0000","last_clean":"2024-09-16T15:50:29.373642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:50:29.403015+0000","last_undegraded":"2024-09-16T15:50:29.403015+0000","last_fullsized":"2024-09-16T15:50:29.403015+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_clean_scrub_stamp":"2024-09-16T15:50:29.373642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.22","version":"0'0","reported_seq":1,"reported_epoch":144,"state":"creating+peering","last_fresh":"2024-09-16T15:50:29.390228+0000","last_change":"2024-09-16T15:50:29.390228+0000","last_active":"2024-09-16T15:50:29.373642+0000","last_peered":"2024-09-16T15:50:29.373642+0000","last_clean":"2024-09-16T15:50:29.373642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:50:29.390228+0000","last_undegraded":"2024-09-16T15:50:29.390228+0000","last_fullsized":"2024-09-16T15:50:29.390228+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_clean_scrub_stamp":"2024-09-16T15:50:29.373642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"23.21","version":"0'0","reported_seq":1,"reported_epoch":144,"state":"creating+peering","last_fresh":"2024-09-16T15:50:29.392793+0000","last_change":"2024-09-16T15:50:29.392793+0000","last_active":"2024-09-16T15:50:29.373642+0000","last_peered":"2024-09-16T15:50:29.373642+0000","last_clean":"2024-09-16T15:50:29.373642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:50:29.392793+0000","last_undegraded":"2024-09-16T15:50:29.392793+0000","last_fullsized":"2024-09-16T15:50:29.392793+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_clean_scrub_stamp":"2024-09-16T15:50:29.373642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"23.20","version":"0'0","reported_seq":1,"reported_epoch":144,"state":"creating+peering","last_fresh":"2024-09-16T15:50:29.400381+0000","last_change":"2024-09-16T15:50:29.400381+0000","last_active":"2024-09-16T15:50:29.373642+0000","last_peered":"2024-09-16T15:50:29.373642+0000","last_clean":"2024-09-16T15:50:29.373642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:50:29.400381+0000","last_undegraded":"2024-09-16T15:50:29.400381+0000","last_fullsized":"2024-09-16T15:50:29.400381+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_clean_scrub_stamp":"2024-09-16T15:50:29.373642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"23.1e","version":"0'0","reported_seq":1,"reported_epoch":144,"state":"creating+peering","last_fresh":"2024-09-16T15:50:29.396097+0000","last_change":"2024-09-16T15:50:29.396097+0000","last_active":"2024-09-16T15:50:29.373642+0000","last_peered":"2024-09-16T15:50:29.373642+0000","last_clean":"2024-09-16T15:50:29.373642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:50:29.396097+0000","last_undegraded":"2024-09-16T15:50:29.396097+0000","last_fullsized":"2024-09-16T15:50:29.396097+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_clean_scrub_stamp":"2024-09-16T15:50:29.373642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"22.1f","version":"0'0","reported_seq":10,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-16T15:50:29.430449+0000","last_change":"2024-09-16T15:50:29.428405+0000","last_active":"2024-09-16T15:50:29.430449+0000","last_peered":"2024-09-16T15:50:29.430449+0000","last_clean":"2024-09-16T15:50:29.430449+0000","last_became_active":"2024-09-16T15:50:29.423819+0000","last_became_peered":"2024-09-16T15:50:29.423819+0000","last_unstale":"2024-09-16T15:50:29.430449+0000","last_undegraded":"2024-09-16T15:50:29.430449+0000","last_fullsized":"2024-09-16T15:50:29.430449+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:28.359944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:28.359944+0000","last_clean_scrub_stamp":"2024-09-16T15:50:28.359944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T22:06:16.541534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.1f","version":"0'0","reported_seq":1,"reported_epoch":144,"state":"creating+peering","last_fresh":"2024-09-16T15:50:29.395019+0000","last_change":"2024-09-16T15:50:29.395019+0000","last_active":"2024-09-16T15:50:29.373642+0000","last_peered":"2024-09-16T15:50:29.373642+0000","last_clean":"2024-09-16T15:50:29.373642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:50:29.395019+0000","last_undegraded":"2024-09-16T15:50:29.395019+0000","last_fullsized":"2024-09-16T15:50:29.395019+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_clean_scrub_stamp":"2024-09-16T15:50:29.373642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.1e","version":"0'0","reported_seq":10,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-16T15:50:29.433442+0000","last_change":"2024-09-16T15:50:29.431326+0000","last_active":"2024-09-16T15:50:29.433442+0000","last_peered":"2024-09-16T15:50:29.433442+0000","last_clean":"2024-09-16T15:50:29.433442+0000","last_became_active":"2024-09-16T15:50:29.430610+0000","last_became_peered":"2024-09-16T15:50:29.430610+0000","last_unstale":"2024-09-16T15:50:29.433442+0000","last_undegraded":"2024-09-16T15:50:29.433442+0000","last_fullsized":"2024-09-16T15:50:29.433442+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:28.359944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:28.359944+0000","last_clean_scrub_stamp":"2024-09-16T15:50:28.359944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T00:28:56.379780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"23.1c","version":"0'0","reported_seq":1,"reported_epoch":144,"state":"creating+peering","last_fresh":"2024-09-16T15:50:29.395809+0000","last_change":"2024-09-16T15:50:29.395809+0000","last_active":"2024-09-16T15:50:29.373642+0000","last_peered":"2024-09-16T15:50:29.373642+0000","last_clean":"2024-09-16T15:50:29.373642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:50:29.395809+0000","last_undegraded":"2024-09-16T15:50:29.395809+0000","last_fullsized":"2024-09-16T15:50:29.395809+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_clean_scrub_stamp":"2024-09-16T15:50:29.373642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.1d","version":"0'0","reported_seq":10,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-16T15:50:29.431196+0000","last_change":"2024-09-16T15:50:29.425771+0000","last_active":"2024-09-16T15:50:29.431196+0000","last_peered":"2024-09-16T15:50:29.431196+0000","last_clean":"2024-09-16T15:50:29.431196+0000","last_became_active":"2024-09-16T15:50:29.425507+0000","last_became_peered":"2024-09-16T15:50:29.425507+0000","last_unstale":"2024-09-16T15:50:29.431196+0000","last_undegraded":"2024-09-16T15:50:29.431196+0000","last_fullsized":"2024-09-16T15:50:29.431196+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:28.359944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:28.359944+0000","last_clean_scrub_stamp":"2024-09-16T15:50:28.359944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:16:32.881957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.1d","version":"0'0","reported_seq":1,"reported_epoch":144,"state":"creating+peering","last_fresh":"2024-09-16T15:50:29.412090+0000","last_change":"2024-09-16T15:50:29.412090+0000","last_active":"2024-09-16T15:50:29.373642+0000","last_peered":"2024-09-16T15:50:29.373642+0000","last_clean":"2024-09-16T15:50:29.373642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:50:29.412090+0000","last_undegraded":"2024-09-16T15:50:29.412090+0000","last_fullsized":"2024-09-16T15:50:29.412090+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_clean_scrub_stamp":"2024-09-16T15:50:29.373642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.1c","version":"0'0","reported_seq":10,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-16T15:50:29.411810+0000","last_change":"2024-09-16T15:50:29.408868+0000","last_active":"2024-09-16T15:50:29.411810+0000","last_peered":"2024-09-16T15:50:29.411810+0000","last_clean":"2024-09-16T15:50:29.411810+0000","last_became_active":"2024-09-16T15:50:29.408550+0000","last_became_peered":"2024-09-16T15:50:29.408550+0000","last_unstale":"2024-09-16T15:50:29.411810+0000","last_undegraded":"2024-09-16T15:50:29.411810+0000","last_fullsized":"2024-09-16T15:50:29.411810+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:28.359944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:28.359944+0000","last_clean_scrub_stamp":"2024-09-16T15:50:28.359944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:57:02.247484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.1a","version":"0'0","reported_seq":1,"reported_epoch":144,"state":"creating+peering","last_fresh":"2024-09-16T15:50:29.396181+0000","last_change":"2024-09-16T15:50:29.396181+0000","last_active":"2024-09-16T15:50:29.373642+0000","last_peered":"2024-09-16T15:50:29.373642+0000","last_clean":"2024-09-16T15:50:29.373642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:50:29.396181+0000","last_undegraded":"2024-09-16T15:50:29.396181+0000","last_fullsized":"2024-09-16T15:50:29.396181+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_clean_scrub_stamp":"2024-09-16T15:50:29.373642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"22.1b","version":"0'0","reported_seq":10,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-16T15:50:29.433993+0000","last_change":"2024-09-16T15:50:29.429045+0000","last_active":"2024-09-16T15:50:29.433993+0000","last_peered":"2024-09-16T15:50:29.433993+0000","last_clean":"2024-09-16T15:50:29.433993+0000","last_became_active":"2024-09-16T15:50:29.428317+0000","last_became_peered":"2024-09-16T15:50:29.428317+0000","last_unstale":"2024-09-16T15:50:29.433993+0000","last_undegraded":"2024-09-16T15:50:29.433993+0000","last_fullsized":"2024-09-16T15:50:29.433993+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:28.359944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:28.359944+0000","last_clean_scrub_stamp":"2024-09-16T15:50:28.359944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:54:05.773197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.1b","version":"0'0","reported_seq":1,"reported_epoch":144,"state":"creating+peering","last_fresh":"2024-09-16T15:50:29.396991+0000","last_change":"2024-09-16T15:50:29.396991+0000","last_active":"2024-09-16T15:50:29.373642+0000","last_peered":"2024-09-16T15:50:29.373642+0000","last_clean":"2024-09-16T15:50:29.373642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:50:29.396991+0000","last_undegraded":"2024-09-16T15:50:29.396991+0000","last_fullsized":"2024-09-16T15:50:29.396991+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_clean_scrub_stamp":"2024-09-16T15:50:29.373642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.1a","version":"0'0","reported_seq":10,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-16T15:50:29.411180+0000","last_change":"2024-09-16T15:50:29.408149+0000","last_active":"2024-09-16T15:50:29.411180+0000","last_peered":"2024-09-16T15:50:29.411180+0000","last_clean":"2024-09-16T15:50:29.411180+0000","last_became_active":"2024-09-16T15:50:29.407778+0000","last_became_peered":"2024-09-16T15:50:29.407778+0000","last_unstale":"2024-09-16T15:50:29.411180+0000","last_undegraded":"2024-09-16T15:50:29.411180+0000","last_fullsized":"2024-09-16T15:50:29.411180+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:28.359944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:28.359944+0000","last_clean_scrub_stamp":"2024-09-16T15:50:28.359944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:02:10.632687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.18","version":"0'0","reported_seq":1,"reported_epoch":144,"state":"creating+peering","last_fresh":"2024-09-16T15:50:29.397299+0000","last_change":"2024-09-16T15:50:29.397299+0000","last_active":"2024-09-16T15:50:29.373642+0000","last_peered":"2024-09-16T15:50:29.373642+0000","last_clean":"2024-09-16T15:50:29.373642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:50:29.397299+0000","last_undegraded":"2024-09-16T15:50:29.397299+0000","last_fullsized":"2024-09-16T15:50:29.397299+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_clean_scrub_stamp":"2024-09-16T15:50:29.373642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.19","version":"0'0","reported_seq":10,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-16T15:50:29.433714+0000","last_change":"2024-09-16T15:50:29.427582+0000","last_active":"2024-09-16T15:50:29.433714+0000","last_peered":"2024-09-16T15:50:29.433714+0000","last_clean":"2024-09-16T15:50:29.433714+0000","last_became_active":"2024-09-16T15:50:29.426124+0000","last_became_peered":"2024-09-16T15:50:29.426124+0000","last_unstale":"2024-09-16T15:50:29.433714+0000","last_undegraded":"2024-09-16T15:50:29.433714+0000","last_fullsized":"2024-09-16T15:50:29.433714+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:28.359944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:28.359944+0000","last_clean_scrub_stamp":"2024-09-16T15:50:28.359944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:01:27.892843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.19","version":"0'0","reported_seq":1,"reported_epoch":144,"state":"creating+peering","last_fresh":"2024-09-16T15:50:29.396885+0000","last_change":"2024-09-16T15:50:29.396885+0000","last_active":"2024-09-16T15:50:29.373642+0000","last_peered":"2024-09-16T15:50:29.373642+0000","last_clean":"2024-09-16T15:50:29.373642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:50:29.396885+0000","last_undegraded":"2024-09-16T15:50:29.396885+0000","last_fullsized":"2024-09-16T15:50:29.396885+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_clean_scrub_stamp":"2024-09-16T15:50:29.373642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.18","version":"0'0","reported_seq":10,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-16T15:50:29.409060+0000","last_change":"2024-09-16T15:50:29.406218+0000","last_active":"2024-09-16T15:50:29.409060+0000","last_peered":"2024-09-16T15:50:29.409060+0000","last_clean":"2024-09-16T15:50:29.409060+0000","last_became_active":"2024-09-16T15:50:29.405872+0000","last_became_peered":"2024-09-16T15:50:29.405872+0000","last_unstale":"2024-09-16T15:50:29.409060+0000","last_undegraded":"2024-09-16T15:50:29.409060+0000","last_fullsized":"2024-09-16T15:50:29.409060+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:28.359944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:28.359944+0000","last_clean_scrub_stamp":"2024-09-16T15:50:28.359944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T22:00:23.514722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"23.17","version":"0'0","reported_seq":1,"reported_epoch":144,"state":"creating+peering","last_fresh":"2024-09-16T15:50:29.403320+0000","last_change":"2024-09-16T15:50:29.403320+0000","last_active":"2024-09-16T15:50:29.373642+0000","last_peered":"2024-09-16T15:50:29.373642+0000","last_clean":"2024-09-16T15:50:29.373642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:50:29.403320+0000","last_undegraded":"2024-09-16T15:50:29.403320+0000","last_fullsized":"2024-09-16T15:50:29.403320+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_clean_scrub_stamp":"2024-09-16T15:50:29.373642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.16","version":"0'0","reported_seq":10,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-16T15:50:29.419433+0000","last_change":"2024-09-16T15:50:29.418819+0000","last_active":"2024-09-16T15:50:29.419433+0000","last_peered":"2024-09-16T15:50:29.419433+0000","last_clean":"2024-09-16T15:50:29.419433+0000","last_became_active":"2024-09-16T15:50:29.418365+0000","last_became_peered":"2024-09-16T15:50:29.418365+0000","last_unstale":"2024-09-16T15:50:29.419433+0000","last_undegraded":"2024-09-16T15:50:29.419433+0000","last_fullsized":"2024-09-16T15:50:29.419433+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:28.359944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:28.359944+0000","last_clean_scrub_stamp":"2024-09-16T15:50:28.359944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T23:56:48.360283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.14","version":"0'0","reported_seq":1,"reported_epoch":144,"state":"creating+peering","last_fresh":"2024-09-16T15:50:29.401245+0000","last_change":"2024-09-16T15:50:29.401245+0000","last_active":"2024-09-16T15:50:29.373642+0000","last_peered":"2024-09-16T15:50:29.373642+0000","last_clean":"2024-09-16T15:50:29.373642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:50:29.401245+0000","last_undegraded":"2024-09-16T15:50:29.401245+0000","last_fullsized":"2024-09-16T15:50:29.401245+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_clean_scrub_stamp":"2024-09-16T15:50:29.373642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.15","version":"0'0","reported_seq":10,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-16T15:50:29.417183+0000","last_change":"2024-09-16T15:50:29.414494+0000","last_active":"2024-09-16T15:50:29.417183+0000","last_peered":"2024-09-16T15:50:29.417183+0000","last_clean":"2024-09-16T15:50:29.417183+0000","last_became_active":"2024-09-16T15:50:29.413903+0000","last_became_peered":"2024-09-16T15:50:29.413903+0000","last_unstale":"2024-09-16T15:50:29.417183+0000","last_undegraded":"2024-09-16T15:50:29.417183+0000","last_fullsized":"2024-09-16T15:50:29.417183+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:28.359944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:28.359944+0000","last_clean_scrub_stamp":"2024-09-16T15:50:28.359944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T00:54:21.344614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.15","version":"0'0","reported_seq":1,"reported_epoch":144,"state":"creating+peering","last_fresh":"2024-09-16T15:50:29.390531+0000","last_change":"2024-09-16T15:50:29.390531+0000","last_active":"2024-09-16T15:50:29.373642+0000","last_peered":"2024-09-16T15:50:29.373642+0000","last_clean":"2024-09-16T15:50:29.373642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:50:29.390531+0000","last_undegraded":"2024-09-16T15:50:29.390531+0000","last_fullsized":"2024-09-16T15:50:29.390531+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_clean_scrub_stamp":"2024-09-16T15:50:29.373642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.14","version":"0'0","reported_seq":10,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-16T15:50:29.420922+0000","last_change":"2024-09-16T15:50:29.406502+0000","last_active":"2024-09-16T15:50:29.420922+0000","last_peered":"2024-09-16T15:50:29.420922+0000","last_clean":"2024-09-16T15:50:29.420922+0000","last_became_active":"2024-09-16T15:50:29.404490+0000","last_became_peered":"2024-09-16T15:50:29.404490+0000","last_unstale":"2024-09-16T15:50:29.420922+0000","last_undegraded":"2024-09-16T15:50:29.420922+0000","last_fullsized":"2024-09-16T15:50:29.420922+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:28.359944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:28.359944+0000","last_clean_scrub_stamp":"2024-09-16T15:50:28.359944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T03:40:13.610653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"23.12","version":"0'0","reported_seq":1,"reported_epoch":144,"state":"creating+peering","last_fresh":"2024-09-16T15:50:29.401262+0000","last_change":"2024-09-16T15:50:29.401262+0000","last_active":"2024-09-16T15:50:29.373642+0000","last_peered":"2024-09-16T15:50:29.373642+0000","last_clean":"2024-09-16T15:50:29.373642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:50:29.401262+0000","last_undegraded":"2024-09-16T15:50:29.401262+0000","last_fullsized":"2024-09-16T15:50:29.401262+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_clean_scrub_stamp":"2024-09-16T15:50:29.373642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.13","version":"0'0","reported_seq":10,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-16T15:50:29.411283+0000","last_change":"2024-09-16T15:50:29.408013+0000","last_active":"2024-09-16T15:50:29.411283+0000","last_peered":"2024-09-16T15:50:29.411283+0000","last_clean":"2024-09-16T15:50:29.411283+0000","last_became_active":"2024-09-16T15:50:29.407703+0000","last_became_peered":"2024-09-16T15:50:29.407703+0000","last_unstale":"2024-09-16T15:50:29.411283+0000","last_undegraded":"2024-09-16T15:50:29.411283+0000","last_fullsized":"2024-09-16T15:50:29.411283+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:28.359944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:28.359944+0000","last_clean_scrub_stamp":"2024-09-16T15:50:28.359944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:05:33.631752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.13","version":"0'0","reported_seq":1,"reported_epoch":144,"state":"creating+peering","last_fresh":"2024-09-16T15:50:29.396084+0000","last_change":"2024-09-16T15:50:29.396084+0000","last_active":"2024-09-16T15:50:29.373642+0000","last_peered":"2024-09-16T15:50:29.373642+0000","last_clean":"2024-09-16T15:50:29.373642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:50:29.396084+0000","last_undegraded":"2024-09-16T15:50:29.396084+0000","last_fullsized":"2024-09-16T15:50:29.396084+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_clean_scrub_stamp":"2024-09-16T15:50:29.373642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"22.12","version":"0'0","reported_seq":10,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-16T15:50:29.411692+0000","last_change":"2024-09-16T15:50:29.410142+0000","last_active":"2024-09-16T15:50:29.411692+0000","last_peered":"2024-09-16T15:50:29.411692+0000","last_clean":"2024-09-16T15:50:29.411692+0000","last_became_active":"2024-09-16T15:50:29.409779+0000","last_became_peered":"2024-09-16T15:50:29.409779+0000","last_unstale":"2024-09-16T15:50:29.411692+0000","last_undegraded":"2024-09-16T15:50:29.411692+0000","last_fullsized":"2024-09-16T15:50:29.411692+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:28.359944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:28.359944+0000","last_clean_scrub_stamp":"2024-09-16T15:50:28.359944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:51:34.842420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"23.10","version":"0'0","reported_seq":1,"reported_epoch":144,"state":"creating+peering","last_fresh":"2024-09-16T15:50:29.396732+0000","last_change":"2024-09-16T15:50:29.396732+0000","last_active":"2024-09-16T15:50:29.373642+0000","last_peered":"2024-09-16T15:50:29.373642+0000","last_clean":"2024-09-16T15:50:29.373642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:50:29.396732+0000","last_undegraded":"2024-09-16T15:50:29.396732+0000","last_fullsized":"2024-09-16T15:50:29.396732+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_clean_scrub_stamp":"2024-09-16T15:50:29.373642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.11","version":"0'0","reported_seq":10,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-16T15:50:29.431189+0000","last_change":"2024-09-16T15:50:29.425647+0000","last_active":"2024-09-16T15:50:29.431189+0000","last_peered":"2024-09-16T15:50:29.431189+0000","last_clean":"2024-09-16T15:50:29.431189+0000","last_became_active":"2024-09-16T15:50:29.425266+0000","last_became_peered":"2024-09-16T15:50:29.425266+0000","last_unstale":"2024-09-16T15:50:29.431189+0000","last_undegraded":"2024-09-16T15:50:29.431189+0000","last_fullsized":"2024-09-16T15:50:29.431189+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:28.359944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:28.359944+0000","last_clean_scrub_stamp":"2024-09-16T15:50:28.359944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:18:55.430269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.11","version":"0'0","reported_seq":1,"reported_epoch":144,"state":"creating+peering","last_fresh":"2024-09-16T15:50:29.418052+0000","last_change":"2024-09-16T15:50:29.418052+0000","last_active":"2024-09-16T15:50:29.373642+0000","last_peered":"2024-09-16T15:50:29.373642+0000","last_clean":"2024-09-16T15:50:29.373642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:50:29.418052+0000","last_undegraded":"2024-09-16T15:50:29.418052+0000","last_fullsized":"2024-09-16T15:50:29.418052+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_clean_scrub_stamp":"2024-09-16T15:50:29.373642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.10","version":"0'0","reported_seq":10,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-16T15:50:29.421423+0000","last_change":"2024-09-16T15:50:29.408571+0000","last_active":"2024-09-16T15:50:29.421423+0000","last_peered":"2024-09-16T15:50:29.421423+0000","last_clean":"2024-09-16T15:50:29.421423+0000","last_became_active":"2024-09-16T15:50:29.407840+0000","last_became_peered":"2024-09-16T15:50:29.407840+0000","last_unstale":"2024-09-16T15:50:29.421423+0000","last_undegraded":"2024-09-16T15:50:29.421423+0000","last_fullsized":"2024-09-16T15:50:29.421423+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:28.359944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:28.359944+0000","last_clean_scrub_stamp":"2024-09-16T15:50:28.359944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:00:52.523701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"23.e","version":"0'0","reported_seq":1,"reported_epoch":144,"state":"creating+peering","last_fresh":"2024-09-16T15:50:29.396821+0000","last_change":"2024-09-16T15:50:29.396821+0000","last_active":"2024-09-16T15:50:29.373642+0000","last_peered":"2024-09-16T15:50:29.373642+0000","last_clean":"2024-09-16T15:50:29.373642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:50:29.396821+0000","last_undegraded":"2024-09-16T15:50:29.396821+0000","last_fullsized":"2024-09-16T15:50:29.396821+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_clean_scrub_stamp":"2024-09-16T15:50:29.373642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.f","version":"0'0","reported_seq":10,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-16T15:50:29.435965+0000","last_change":"2024-09-16T15:50:29.434785+0000","last_active":"2024-09-16T15:50:29.435965+0000","last_peered":"2024-09-16T15:50:29.435965+0000","last_clean":"2024-09-16T15:50:29.435965+0000","last_became_active":"2024-09-16T15:50:29.434102+0000","last_became_peered":"2024-09-16T15:50:29.434102+0000","last_unstale":"2024-09-16T15:50:29.435965+0000","last_undegraded":"2024-09-16T15:50:29.435965+0000","last_fullsized":"2024-09-16T15:50:29.435965+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:28.359944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:28.359944+0000","last_clean_scrub_stamp":"2024-09-16T15:50:28.359944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T02:25:34.661941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.f","version":"0'0","reported_seq":1,"reported_epoch":144,"state":"creating+peering","last_fresh":"2024-09-16T15:50:29.402843+0000","last_change":"2024-09-16T15:50:29.402843+0000","last_active":"2024-09-16T15:50:29.373642+0000","last_peered":"2024-09-16T15:50:29.373642+0000","last_clean":"2024-09-16T15:50:29.373642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:50:29.402843+0000","last_undegraded":"2024-09-16T15:50:29.402843+0000","last_fullsized":"2024-09-16T15:50:29.402843+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_clean_scrub_stamp":"2024-09-16T15:50:29.373642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.e","version":"0'0","reported_seq":10,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-16T15:50:29.429265+0000","last_change":"2024-09-16T15:50:29.426483+0000","last_active":"2024-09-16T15:50:29.429265+0000","last_peered":"2024-09-16T15:50:29.429265+0000","last_clean":"2024-09-16T15:50:29.429265+0000","last_became_active":"2024-09-16T15:50:29.425712+0000","last_became_peered":"2024-09-16T15:50:29.425712+0000","last_unstale":"2024-09-16T15:50:29.429265+0000","last_undegraded":"2024-09-16T15:50:29.429265+0000","last_fullsized":"2024-09-16T15:50:29.429265+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:28.359944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:28.359944+0000","last_clean_scrub_stamp":"2024-09-16T15:50:28.359944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:31:59.513487+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.c","version":"0'0","reported_seq":1,"reported_epoch":144,"state":"creating+peering","last_fresh":"2024-09-16T15:50:29.399219+0000","last_change":"2024-09-16T15:50:29.399219+0000","last_active":"2024-09-16T15:50:29.373642+0000","last_peered":"2024-09-16T15:50:29.373642+0000","last_clean":"2024-09-16T15:50:29.373642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:50:29.399219+0000","last_undegraded":"2024-09-16T15:50:29.399219+0000","last_fullsized":"2024-09-16T15:50:29.399219+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_clean_scrub_stamp":"2024-09-16T15:50:29.373642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.d","version":"0'0","reported_seq":10,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-16T15:50:29.430249+0000","last_change":"2024-09-16T15:50:29.427670+0000","last_active":"2024-09-16T15:50:29.430249+0000","last_peered":"2024-09-16T15:50:29.430249+0000","last_clean":"2024-09-16T15:50:29.430249+0000","last_became_active":"2024-09-16T15:50:29.426920+0000","last_became_peered":"2024-09-16T15:50:29.426920+0000","last_unstale":"2024-09-16T15:50:29.430249+0000","last_undegraded":"2024-09-16T15:50:29.430249+0000","last_fullsized":"2024-09-16T15:50:29.430249+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:28.359944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:28.359944+0000","last_clean_scrub_stamp":"2024-09-16T15:50:28.359944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:53:40.560984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.d","version":"0'0","reported_seq":1,"reported_epoch":144,"state":"creating+peering","last_fresh":"2024-09-16T15:50:29.395185+0000","last_change":"2024-09-16T15:50:29.395185+0000","last_active":"2024-09-16T15:50:29.373642+0000","last_peered":"2024-09-16T15:50:29.373642+0000","last_clean":"2024-09-16T15:50:29.373642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:50:29.395185+0000","last_undegraded":"2024-09-16T15:50:29.395185+0000","last_fullsized":"2024-09-16T15:50:29.395185+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_clean_scrub_stamp":"2024-09-16T15:50:29.373642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"22.c","version":"0'0","reported_seq":10,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-16T15:50:29.430128+0000","last_change":"2024-09-16T15:50:29.427240+0000","last_active":"2024-09-16T15:50:29.430128+0000","last_peered":"2024-09-16T15:50:29.430128+0000","last_clean":"2024-09-16T15:50:29.430128+0000","last_became_active":"2024-09-16T15:50:29.423280+0000","last_became_peered":"2024-09-16T15:50:29.423280+0000","last_unstale":"2024-09-16T15:50:29.430128+0000","last_undegraded":"2024-09-16T15:50:29.430128+0000","last_fullsized":"2024-09-16T15:50:29.430128+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:28.359944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:28.359944+0000","last_clean_scrub_stamp":"2024-09-16T15:50:28.359944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:36:54.628281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.a","version":"0'0","reported_seq":1,"reported_epoch":144,"state":"creating+peering","last_fresh":"2024-09-16T15:50:29.400460+0000","last_change":"2024-09-16T15:50:29.400460+0000","last_active":"2024-09-16T15:50:29.373642+0000","last_peered":"2024-09-16T15:50:29.373642+0000","last_clean":"2024-09-16T15:50:29.373642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:50:29.400460+0000","last_undegraded":"2024-09-16T15:50:29.400460+0000","last_fullsized":"2024-09-16T15:50:29.400460+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_clean_scrub_stamp":"2024-09-16T15:50:29.373642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.b","version":"0'0","reported_seq":10,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-16T15:50:29.431651+0000","last_change":"2024-09-16T15:50:29.425798+0000","last_active":"2024-09-16T15:50:29.431651+0000","last_peered":"2024-09-16T15:50:29.431651+0000","last_clean":"2024-09-16T15:50:29.431651+0000","last_became_active":"2024-09-16T15:50:29.425459+0000","last_became_peered":"2024-09-16T15:50:29.425459+0000","last_unstale":"2024-09-16T15:50:29.431651+0000","last_undegraded":"2024-09-16T15:50:29.431651+0000","last_fullsized":"2024-09-16T15:50:29.431651+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:28.359944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:28.359944+0000","last_clean_scrub_stamp":"2024-09-16T15:50:28.359944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:59:55.828797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.b","version":"0'0","reported_seq":1,"reported_epoch":144,"state":"creating+peering","last_fresh":"2024-09-16T15:50:29.394330+0000","last_change":"2024-09-16T15:50:29.394330+0000","last_active":"2024-09-16T15:50:29.373642+0000","last_peered":"2024-09-16T15:50:29.373642+0000","last_clean":"2024-09-16T15:50:29.373642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:50:29.394330+0000","last_undegraded":"2024-09-16T15:50:29.394330+0000","last_fullsized":"2024-09-16T15:50:29.394330+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_clean_scrub_stamp":"2024-09-16T15:50:29.373642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"22.a","version":"0'0","reported_seq":10,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-16T15:50:29.429851+0000","last_change":"2024-09-16T15:50:29.426752+0000","last_active":"2024-09-16T15:50:29.429851+0000","last_peered":"2024-09-16T15:50:29.429851+0000","last_clean":"2024-09-16T15:50:29.429851+0000","last_became_active":"2024-09-16T15:50:29.422755+0000","last_became_peered":"2024-09-16T15:50:29.422755+0000","last_unstale":"2024-09-16T15:50:29.429851+0000","last_undegraded":"2024-09-16T15:50:29.429851+0000","last_fullsized":"2024-09-16T15:50:29.429851+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:28.359944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:28.359944+0000","last_clean_scrub_stamp":"2024-09-16T15:50:28.359944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T18:14:40.770591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.8","version":"0'0","reported_seq":1,"reported_epoch":144,"state":"creating+peering","last_fresh":"2024-09-16T15:50:29.397861+0000","last_change":"2024-09-16T15:50:29.397861+0000","last_active":"2024-09-16T15:50:29.373642+0000","last_peered":"2024-09-16T15:50:29.373642+0000","last_clean":"2024-09-16T15:50:29.373642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:50:29.397861+0000","last_undegraded":"2024-09-16T15:50:29.397861+0000","last_fullsized":"2024-09-16T15:50:29.397861+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_clean_scrub_stamp":"2024-09-16T15:50:29.373642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.9","version":"0'0","reported_seq":10,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-16T15:50:29.424612+0000","last_change":"2024-09-16T15:50:29.423325+0000","last_active":"2024-09-16T15:50:29.424612+0000","last_peered":"2024-09-16T15:50:29.424612+0000","last_clean":"2024-09-16T15:50:29.424612+0000","last_became_active":"2024-09-16T15:50:29.423053+0000","last_became_peered":"2024-09-16T15:50:29.423053+0000","last_unstale":"2024-09-16T15:50:29.424612+0000","last_undegraded":"2024-09-16T15:50:29.424612+0000","last_fullsized":"2024-09-16T15:50:29.424612+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:28.359944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:28.359944+0000","last_clean_scrub_stamp":"2024-09-16T15:50:28.359944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T22:05:53.341097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.9","version":"0'0","reported_seq":1,"reported_epoch":144,"state":"creating+peering","last_fresh":"2024-09-16T15:50:29.396384+0000","last_change":"2024-09-16T15:50:29.396384+0000","last_active":"2024-09-16T15:50:29.373642+0000","last_peered":"2024-09-16T15:50:29.373642+0000","last_clean":"2024-09-16T15:50:29.373642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:50:29.396384+0000","last_undegraded":"2024-09-16T15:50:29.396384+0000","last_fullsized":"2024-09-16T15:50:29.396384+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_clean_scrub_stamp":"2024-09-16T15:50:29.373642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.8","version":"0'0","reported_seq":10,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-16T15:50:29.421906+0000","last_change":"2024-09-16T15:50:29.418765+0000","last_active":"2024-09-16T15:50:29.421906+0000","last_peered":"2024-09-16T15:50:29.421906+0000","last_clean":"2024-09-16T15:50:29.421906+0000","last_became_active":"2024-09-16T15:50:29.417737+0000","last_became_peered":"2024-09-16T15:50:29.417737+0000","last_unstale":"2024-09-16T15:50:29.421906+0000","last_undegraded":"2024-09-16T15:50:29.421906+0000","last_fullsized":"2024-09-16T15:50:29.421906+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:28.359944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:28.359944+0000","last_clean_scrub_stamp":"2024-09-16T15:50:28.359944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T00:16:50.967923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.6","version":"0'0","reported_seq":1,"reported_epoch":144,"state":"creating+peering","last_fresh":"2024-09-16T15:50:29.400497+0000","last_change":"2024-09-16T15:50:29.400497+0000","last_active":"2024-09-16T15:50:29.373642+0000","last_peered":"2024-09-16T15:50:29.373642+0000","last_clean":"2024-09-16T15:50:29.373642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:50:29.400497+0000","last_undegraded":"2024-09-16T15:50:29.400497+0000","last_fullsized":"2024-09-16T15:50:29.400497+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_clean_scrub_stamp":"2024-09-16T15:50:29.373642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.7","version":"0'0","reported_seq":10,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-16T15:50:29.423515+0000","last_change":"2024-09-16T15:50:29.422588+0000","last_active":"2024-09-16T15:50:29.423515+0000","last_peered":"2024-09-16T15:50:29.423515+0000","last_clean":"2024-09-16T15:50:29.423515+0000","last_became_active":"2024-09-16T15:50:29.422240+0000","last_became_peered":"2024-09-16T15:50:29.422240+0000","last_unstale":"2024-09-16T15:50:29.423515+0000","last_undegraded":"2024-09-16T15:50:29.423515+0000","last_fullsized":"2024-09-16T15:50:29.423515+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:28.359944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:28.359944+0000","last_clean_scrub_stamp":"2024-09-16T15:50:28.359944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T01:48:32.528831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"23.7","version":"0'0","reported_seq":1,"reported_epoch":144,"state":"creating+peering","last_fresh":"2024-09-16T15:50:29.394083+0000","last_change":"2024-09-16T15:50:29.394083+0000","last_active":"2024-09-16T15:50:29.373642+0000","last_peered":"2024-09-16T15:50:29.373642+0000","last_clean":"2024-09-16T15:50:29.373642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:50:29.394083+0000","last_undegraded":"2024-09-16T15:50:29.394083+0000","last_fullsized":"2024-09-16T15:50:29.394083+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_clean_scrub_stamp":"2024-09-16T15:50:29.373642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"22.6","version":"0'0","reported_seq":10,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-16T15:50:29.428409+0000","last_change":"2024-09-16T15:50:29.424687+0000","last_active":"2024-09-16T15:50:29.428409+0000","last_peered":"2024-09-16T15:50:29.428409+0000","last_clean":"2024-09-16T15:50:29.428409+0000","last_became_active":"2024-09-16T15:50:29.423323+0000","last_became_peered":"2024-09-16T15:50:29.423323+0000","last_unstale":"2024-09-16T15:50:29.428409+0000","last_undegraded":"2024-09-16T15:50:29.428409+0000","last_fullsized":"2024-09-16T15:50:29.428409+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:28.359944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:28.359944+0000","last_clean_scrub_stamp":"2024-09-16T15:50:28.359944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:26:50.321081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.4","version":"0'0","reported_seq":1,"reported_epoch":144,"state":"creating+peering","last_fresh":"2024-09-16T15:50:29.395179+0000","last_change":"2024-09-16T15:50:29.395179+0000","last_active":"2024-09-16T15:50:29.373642+0000","last_peered":"2024-09-16T15:50:29.373642+0000","last_clean":"2024-09-16T15:50:29.373642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:50:29.395179+0000","last_undegraded":"2024-09-16T15:50:29.395179+0000","last_fullsized":"2024-09-16T15:50:29.395179+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_clean_scrub_stamp":"2024-09-16T15:50:29.373642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.5","version":"0'0","reported_seq":10,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-16T15:50:29.429727+0000","last_change":"2024-09-16T15:50:29.421716+0000","last_active":"2024-09-16T15:50:29.429727+0000","last_peered":"2024-09-16T15:50:29.429727+0000","last_clean":"2024-09-16T15:50:29.429727+0000","last_became_active":"2024-09-16T15:50:29.420529+0000","last_became_peered":"2024-09-16T15:50:29.420529+0000","last_unstale":"2024-09-16T15:50:29.429727+0000","last_undegraded":"2024-09-16T15:50:29.429727+0000","last_fullsized":"2024-09-16T15:50:29.429727+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:28.359944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:28.359944+0000","last_clean_scrub_stamp":"2024-09-16T15:50:28.359944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T03:46:32.264684+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.5","version":"0'0","reported_seq":1,"reported_epoch":144,"state":"creating+peering","last_fresh":"2024-09-16T15:50:29.400308+0000","last_change":"2024-09-16T15:50:29.400308+0000","last_active":"2024-09-16T15:50:29.373642+0000","last_peered":"2024-09-16T15:50:29.373642+0000","last_clean":"2024-09-16T15:50:29.373642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:50:29.400308+0000","last_undegraded":"2024-09-16T15:50:29.400308+0000","last_fullsized":"2024-09-16T15:50:29.400308+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_clean_scrub_stamp":"2024-09-16T15:50:29.373642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.4","version":"0'0","reported_seq":10,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-16T15:50:29.417851+0000","last_change":"2024-09-16T15:50:29.417210+0000","last_active":"2024-09-16T15:50:29.417851+0000","last_peered":"2024-09-16T15:50:29.417851+0000","last_clean":"2024-09-16T15:50:29.417851+0000","last_became_active":"2024-09-16T15:50:29.416831+0000","last_became_peered":"2024-09-16T15:50:29.416831+0000","last_unstale":"2024-09-16T15:50:29.417851+0000","last_undegraded":"2024-09-16T15:50:29.417851+0000","last_fullsized":"2024-09-16T15:50:29.417851+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:28.359944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:28.359944+0000","last_clean_scrub_stamp":"2024-09-16T15:50:28.359944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:00:59.918701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"23.2","version":"0'0","reported_seq":1,"reported_epoch":144,"state":"creating+peering","last_fresh":"2024-09-16T15:50:29.400856+0000","last_change":"2024-09-16T15:50:29.400856+0000","last_active":"2024-09-16T15:50:29.373642+0000","last_peered":"2024-09-16T15:50:29.373642+0000","last_clean":"2024-09-16T15:50:29.373642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:50:29.400856+0000","last_undegraded":"2024-09-16T15:50:29.400856+0000","last_fullsized":"2024-09-16T15:50:29.400856+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_clean_scrub_stamp":"2024-09-16T15:50:29.373642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.3","version":"0'0","reported_seq":10,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-16T15:50:29.421008+0000","last_change":"2024-09-16T15:50:29.407104+0000","last_active":"2024-09-16T15:50:29.421008+0000","last_peered":"2024-09-16T15:50:29.421008+0000","last_clean":"2024-09-16T15:50:29.421008+0000","last_became_active":"2024-09-16T15:50:29.406182+0000","last_became_peered":"2024-09-16T15:50:29.406182+0000","last_unstale":"2024-09-16T15:50:29.421008+0000","last_undegraded":"2024-09-16T15:50:29.421008+0000","last_fullsized":"2024-09-16T15:50:29.421008+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:28.359944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:28.359944+0000","last_clean_scrub_stamp":"2024-09-16T15:50:28.359944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:36:55.485971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"23.3","version":"0'0","reported_seq":1,"reported_epoch":144,"state":"creating+peering","last_fresh":"2024-09-16T15:50:29.391321+0000","last_change":"2024-09-16T15:50:29.391321+0000","last_active":"2024-09-16T15:50:29.373642+0000","last_peered":"2024-09-16T15:50:29.373642+0000","last_clean":"2024-09-16T15:50:29.373642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:50:29.391321+0000","last_undegraded":"2024-09-16T15:50:29.391321+0000","last_fullsized":"2024-09-16T15:50:29.391321+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_clean_scrub_stamp":"2024-09-16T15:50:29.373642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.2","version":"0'0","reported_seq":10,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-16T15:50:29.411842+0000","last_change":"2024-09-16T15:50:29.410821+0000","last_active":"2024-09-16T15:50:29.411842+0000","last_peered":"2024-09-16T15:50:29.411842+0000","last_clean":"2024-09-16T15:50:29.411842+0000","last_became_active":"2024-09-16T15:50:29.410554+0000","last_became_peered":"2024-09-16T15:50:29.410554+0000","last_unstale":"2024-09-16T15:50:29.411842+0000","last_undegraded":"2024-09-16T15:50:29.411842+0000","last_fullsized":"2024-09-16T15:50:29.411842+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:28.359944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:28.359944+0000","last_clean_scrub_stamp":"2024-09-16T15:50:28.359944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:55:15.802642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.0","version":"0'0","reported_seq":1,"reported_epoch":144,"state":"creating+peering","last_fresh":"2024-09-16T15:50:29.394199+0000","last_change":"2024-09-16T15:50:29.394199+0000","last_active":"2024-09-16T15:50:29.373642+0000","last_peered":"2024-09-16T15:50:29.373642+0000","last_clean":"2024-09-16T15:50:29.373642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:50:29.394199+0000","last_undegraded":"2024-09-16T15:50:29.394199+0000","last_fullsized":"2024-09-16T15:50:29.394199+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_clean_scrub_stamp":"2024-09-16T15:50:29.373642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"22.1","version":"0'0","reported_seq":10,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-16T15:50:29.421042+0000","last_change":"2024-09-16T15:50:29.416751+0000","last_active":"2024-09-16T15:50:29.421042+0000","last_peered":"2024-09-16T15:50:29.421042+0000","last_clean":"2024-09-16T15:50:29.421042+0000","last_became_active":"2024-09-16T15:50:29.415995+0000","last_became_peered":"2024-09-16T15:50:29.415995+0000","last_unstale":"2024-09-16T15:50:29.421042+0000","last_undegraded":"2024-09-16T15:50:29.421042+0000","last_fullsized":"2024-09-16T15:50:29.421042+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:28.359944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:28.359944+0000","last_clean_scrub_stamp":"2024-09-16T15:50:28.359944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T02:16:57.915820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.1","version":"0'0","reported_seq":1,"reported_epoch":144,"state":"creating+peering","last_fresh":"2024-09-16T15:50:29.409753+0000","last_change":"2024-09-16T15:50:29.409753+0000","last_active":"2024-09-16T15:50:29.373642+0000","last_peered":"2024-09-16T15:50:29.373642+0000","last_clean":"2024-09-16T15:50:29.373642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:50:29.409753+0000","last_undegraded":"2024-09-16T15:50:29.409753+0000","last_fullsized":"2024-09-16T15:50:29.409753+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_clean_scrub_stamp":"2024-09-16T15:50:29.373642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.0","version":"0'0","reported_seq":10,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-16T15:50:29.406562+0000","last_change":"2024-09-16T15:50:29.395361+0000","last_active":"2024-09-16T15:50:29.406562+0000","last_peered":"2024-09-16T15:50:29.406562+0000","last_clean":"2024-09-16T15:50:29.406562+0000","last_became_active":"2024-09-16T15:50:29.394578+0000","last_became_peered":"2024-09-16T15:50:29.394578+0000","last_unstale":"2024-09-16T15:50:29.406562+0000","last_undegraded":"2024-09-16T15:50:29.406562+0000","last_fullsized":"2024-09-16T15:50:29.406562+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:28.359944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:28.359944+0000","last_clean_scrub_stamp":"2024-09-16T15:50:28.359944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T02:06:32.966504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"23.16","version":"0'0","reported_seq":1,"reported_epoch":144,"state":"creating+peering","last_fresh":"2024-09-16T15:50:29.401057+0000","last_change":"2024-09-16T15:50:29.401057+0000","last_active":"2024-09-16T15:50:29.373642+0000","last_peered":"2024-09-16T15:50:29.373642+0000","last_clean":"2024-09-16T15:50:29.373642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:50:29.401057+0000","last_undegraded":"2024-09-16T15:50:29.401057+0000","last_fullsized":"2024-09-16T15:50:29.401057+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_clean_scrub_stamp":"2024-09-16T15:50:29.373642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.17","version":"0'0","reported_seq":10,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-16T15:50:29.421257+0000","last_change":"2024-09-16T15:50:29.410992+0000","last_active":"2024-09-16T15:50:29.421257+0000","last_peered":"2024-09-16T15:50:29.421257+0000","last_clean":"2024-09-16T15:50:29.421257+0000","last_became_active":"2024-09-16T15:50:29.410215+0000","last_became_peered":"2024-09-16T15:50:29.410215+0000","last_unstale":"2024-09-16T15:50:29.421257+0000","last_undegraded":"2024-09-16T15:50:29.421257+0000","last_fullsized":"2024-09-16T15:50:29.421257+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:28.359944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:28.359944+0000","last_clean_scrub_stamp":"2024-09-16T15:50:28.359944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T00:24:19.013129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"73'79","reported_seq":366,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-16T15:50:29.402844+0000","last_change":"2024-09-16T15:42:40.881820+0000","last_active":"2024-09-16T15:50:29.402844+0000","last_peered":"2024-09-16T15:50:29.402844+0000","last_clean":"2024-09-16T15:50:29.402844+0000","last_became_active":"2024-09-16T15:42:40.879965+0000","last_became_peered":"2024-09-16T15:42:40.879965+0000","last_unstale":"2024-09-16T15:50:29.402844+0000","last_undegraded":"2024-09-16T15:50:29.402844+0000","last_fullsized":"2024-09-16T15:50:29.402844+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:42:39.728762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:42:39.728762+0000","last_clean_scrub_stamp":"2024-09-16T15:42:39.728762+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:22:18.768932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":23,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":22,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79,"ondisk_log_size":79,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542242,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33536,"kb_used_data":2344,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338304,"statfs":{"total":96636764160,"available":96602423296,"internally_reserved":0,"allocated":2400256,"data_stored":1389123,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542241,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32956,"kb_used_data":1764,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338884,"statfs":{"total":96636764160,"available":96603017216,"internally_reserved":0,"allocated":1806336,"data_stored":798755,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":5,"up_from":14,"seq":60129542242,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32940,"kb_used_data":1752,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338900,"statfs":{"total":96636764160,"available":96603033600,"internally_reserved":0,"allocated":1794048,"data_stored":792452,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542240,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32940,"kb_used_data":1752,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338900,"statfs":{"total":96636764160,"available":96603033600,"internally_reserved":0,"allocated":1794048,"data_stored":792452,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":3,"up_from":14,"seq":60129542241,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37100,"kb_used_data":1752,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334740,"statfs":{"total":96636764160,"available":96598773760,"internally_reserved":0,"allocated":1794048,"data_stored":792452,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":14,"seq":60129542241,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32956,"kb_used_data":1764,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338884,"statfs":{"total":96636764160,"available":96603017216,"internally_reserved":0,"allocated":1806336,"data_stored":798755,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":14,"seq":60129542243,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32956,"kb_used_data":1764,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338884,"statfs":{"total":96636764160,"available":96603017216,"internally_reserved":0,"allocated":1806336,"data_stored":798755,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":14,"seq":60129542243,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33520,"kb_used_data":2332,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338320,"statfs":{"total":96636764160,"available":96602439680,"internally_reserved":0,"allocated":2387968,"data_stored":1382820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T15:50:35.181 INFO:tasks.ceph.ceph_manager.ceph:PG 23.3f is not active+clean 2024-09-16T15:50:35.182 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '23.3f', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 144, 'state': 'creating+peering', 'last_fresh': '2024-09-16T15:50:29.396826+0000', 'last_change': '2024-09-16T15:50:29.396826+0000', 'last_active': '2024-09-16T15:50:29.373642+0000', 'last_peered': '2024-09-16T15:50:29.373642+0000', 'last_clean': '2024-09-16T15:50:29.373642+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-16T15:50:29.396826+0000', 'last_undegraded': '2024-09-16T15:50:29.396826+0000', 'last_fullsized': '2024-09-16T15:50:29.396826+0000', 'mapping_epoch': 144, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 144, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T15:50:29.373642+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T15:50:29.373642+0000', 'last_clean_scrub_stamp': '2024-09-16T15:50:29.373642+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 5], 'acting': [7, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-09-16T15:50:35.182 INFO:tasks.ceph.ceph_manager.ceph:PG 23.3e is not active+clean 2024-09-16T15:50:35.182 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '23.3e', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 144, 'state': 'creating+peering', 'last_fresh': '2024-09-16T15:50:29.405611+0000', 'last_change': '2024-09-16T15:50:29.405611+0000', 'last_active': '2024-09-16T15:50:29.373642+0000', 'last_peered': '2024-09-16T15:50:29.373642+0000', 'last_clean': '2024-09-16T15:50:29.373642+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-16T15:50:29.405611+0000', 'last_undegraded': '2024-09-16T15:50:29.405611+0000', 'last_fullsized': '2024-09-16T15:50:29.405611+0000', 'mapping_epoch': 144, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 144, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T15:50:29.373642+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T15:50:29.373642+0000', 'last_clean_scrub_stamp': '2024-09-16T15:50:29.373642+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 7], 'acting': [6, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-09-16T15:50:35.182 INFO:tasks.ceph.ceph_manager.ceph:PG 23.3d is not active+clean 2024-09-16T15:50:35.182 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '23.3d', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 144, 'state': 'creating+peering', 'last_fresh': '2024-09-16T15:50:29.414816+0000', 'last_change': '2024-09-16T15:50:29.414816+0000', 'last_active': '2024-09-16T15:50:29.373642+0000', 'last_peered': '2024-09-16T15:50:29.373642+0000', 'last_clean': '2024-09-16T15:50:29.373642+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-16T15:50:29.414816+0000', 'last_undegraded': '2024-09-16T15:50:29.414816+0000', 'last_fullsized': '2024-09-16T15:50:29.414816+0000', 'mapping_epoch': 144, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 144, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T15:50:29.373642+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T15:50:29.373642+0000', 'last_clean_scrub_stamp': '2024-09-16T15:50:29.373642+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 4], 'acting': [5, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-09-16T15:50:35.182 INFO:tasks.ceph.ceph_manager.ceph:PG 23.3c is not active+clean 2024-09-16T15:50:35.182 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '23.3c', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 144, 'state': 'creating+peering', 'last_fresh': '2024-09-16T15:50:29.400948+0000', 'last_change': '2024-09-16T15:50:29.400948+0000', 'last_active': '2024-09-16T15:50:29.373642+0000', 'last_peered': '2024-09-16T15:50:29.373642+0000', 'last_clean': '2024-09-16T15:50:29.373642+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-16T15:50:29.400948+0000', 'last_undegraded': '2024-09-16T15:50:29.400948+0000', 'last_fullsized': '2024-09-16T15:50:29.400948+0000', 'mapping_epoch': 144, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 144, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T15:50:29.373642+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T15:50:29.373642+0000', 'last_clean_scrub_stamp': '2024-09-16T15:50:29.373642+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3], 'acting': [1, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-09-16T15:50:35.182 INFO:tasks.ceph.ceph_manager.ceph:PG 23.3b is not active+clean 2024-09-16T15:50:35.182 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '23.3b', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 144, 'state': 'creating+peering', 'last_fresh': '2024-09-16T15:50:29.408612+0000', 'last_change': '2024-09-16T15:50:29.408612+0000', 'last_active': '2024-09-16T15:50:29.373642+0000', 'last_peered': '2024-09-16T15:50:29.373642+0000', 'last_clean': '2024-09-16T15:50:29.373642+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-16T15:50:29.408612+0000', 'last_undegraded': '2024-09-16T15:50:29.408612+0000', 'last_fullsized': '2024-09-16T15:50:29.408612+0000', 'mapping_epoch': 144, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 144, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T15:50:29.373642+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T15:50:29.373642+0000', 'last_clean_scrub_stamp': '2024-09-16T15:50:29.373642+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 4], 'acting': [6, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-09-16T15:50:35.182 INFO:tasks.ceph.ceph_manager.ceph:PG 23.3a is not active+clean 2024-09-16T15:50:35.182 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '23.3a', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 144, 'state': 'creating+peering', 'last_fresh': '2024-09-16T15:50:29.399825+0000', 'last_change': '2024-09-16T15:50:29.399825+0000', 'last_active': '2024-09-16T15:50:29.373642+0000', 'last_peered': '2024-09-16T15:50:29.373642+0000', 'last_clean': '2024-09-16T15:50:29.373642+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-16T15:50:29.399825+0000', 'last_undegraded': '2024-09-16T15:50:29.399825+0000', 'last_fullsized': '2024-09-16T15:50:29.399825+0000', 'mapping_epoch': 144, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 144, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T15:50:29.373642+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T15:50:29.373642+0000', 'last_clean_scrub_stamp': '2024-09-16T15:50:29.373642+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0], 'acting': [2, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T15:50:35.182 INFO:tasks.ceph.ceph_manager.ceph:PG 23.39 is not active+clean 2024-09-16T15:50:35.183 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '23.39', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 144, 'state': 'creating+peering', 'last_fresh': '2024-09-16T15:50:29.397580+0000', 'last_change': '2024-09-16T15:50:29.397580+0000', 'last_active': '2024-09-16T15:50:29.373642+0000', 'last_peered': '2024-09-16T15:50:29.373642+0000', 'last_clean': '2024-09-16T15:50:29.373642+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-16T15:50:29.397580+0000', 'last_undegraded': '2024-09-16T15:50:29.397580+0000', 'last_fullsized': '2024-09-16T15:50:29.397580+0000', 'mapping_epoch': 144, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 144, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T15:50:29.373642+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T15:50:29.373642+0000', 'last_clean_scrub_stamp': '2024-09-16T15:50:29.373642+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 4], 'acting': [0, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T15:50:35.183 INFO:tasks.ceph.ceph_manager.ceph:PG 23.38 is not active+clean 2024-09-16T15:50:35.183 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '23.38', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 144, 'state': 'creating+peering', 'last_fresh': '2024-09-16T15:50:29.409708+0000', 'last_change': '2024-09-16T15:50:29.409708+0000', 'last_active': '2024-09-16T15:50:29.373642+0000', 'last_peered': '2024-09-16T15:50:29.373642+0000', 'last_clean': '2024-09-16T15:50:29.373642+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-16T15:50:29.409708+0000', 'last_undegraded': '2024-09-16T15:50:29.409708+0000', 'last_fullsized': '2024-09-16T15:50:29.409708+0000', 'mapping_epoch': 144, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 144, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T15:50:29.373642+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T15:50:29.373642+0000', 'last_clean_scrub_stamp': '2024-09-16T15:50:29.373642+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 4], 'acting': [6, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-09-16T15:50:35.183 INFO:tasks.ceph.ceph_manager.ceph:PG 23.37 is not active+clean 2024-09-16T15:50:35.183 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '23.37', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 144, 'state': 'creating+peering', 'last_fresh': '2024-09-16T15:50:29.406549+0000', 'last_change': '2024-09-16T15:50:29.406549+0000', 'last_active': '2024-09-16T15:50:29.373642+0000', 'last_peered': '2024-09-16T15:50:29.373642+0000', 'last_clean': '2024-09-16T15:50:29.373642+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-16T15:50:29.406549+0000', 'last_undegraded': '2024-09-16T15:50:29.406549+0000', 'last_fullsized': '2024-09-16T15:50:29.406549+0000', 'mapping_epoch': 144, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 144, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T15:50:29.373642+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T15:50:29.373642+0000', 'last_clean_scrub_stamp': '2024-09-16T15:50:29.373642+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 7], 'acting': [6, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-09-16T15:50:35.183 INFO:tasks.ceph.ceph_manager.ceph:PG 23.36 is not active+clean 2024-09-16T15:50:35.183 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '23.36', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 144, 'state': 'creating+peering', 'last_fresh': '2024-09-16T15:50:29.404538+0000', 'last_change': '2024-09-16T15:50:29.404538+0000', 'last_active': '2024-09-16T15:50:29.373642+0000', 'last_peered': '2024-09-16T15:50:29.373642+0000', 'last_clean': '2024-09-16T15:50:29.373642+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-16T15:50:29.404538+0000', 'last_undegraded': '2024-09-16T15:50:29.404538+0000', 'last_fullsized': '2024-09-16T15:50:29.404538+0000', 'mapping_epoch': 144, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 144, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T15:50:29.373642+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T15:50:29.373642+0000', 'last_clean_scrub_stamp': '2024-09-16T15:50:29.373642+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 1], 'acting': [6, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-09-16T15:50:35.183 INFO:tasks.ceph.ceph_manager.ceph:PG 23.35 is not active+clean 2024-09-16T15:50:35.183 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '23.35', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 144, 'state': 'creating+peering', 'last_fresh': '2024-09-16T15:50:29.418503+0000', 'last_change': '2024-09-16T15:50:29.418503+0000', 'last_active': '2024-09-16T15:50:29.373642+0000', 'last_peered': '2024-09-16T15:50:29.373642+0000', 'last_clean': '2024-09-16T15:50:29.373642+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-16T15:50:29.418503+0000', 'last_undegraded': '2024-09-16T15:50:29.418503+0000', 'last_fullsized': '2024-09-16T15:50:29.418503+0000', 'mapping_epoch': 144, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 144, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T15:50:29.373642+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T15:50:29.373642+0000', 'last_clean_scrub_stamp': '2024-09-16T15:50:29.373642+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 4], 'acting': [6, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-09-16T15:50:35.183 INFO:tasks.ceph.ceph_manager.ceph:PG 23.34 is not active+clean 2024-09-16T15:50:35.183 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '23.34', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 144, 'state': 'creating+peering', 'last_fresh': '2024-09-16T15:50:29.398426+0000', 'last_change': '2024-09-16T15:50:29.398426+0000', 'last_active': '2024-09-16T15:50:29.373642+0000', 'last_peered': '2024-09-16T15:50:29.373642+0000', 'last_clean': '2024-09-16T15:50:29.373642+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-16T15:50:29.398426+0000', 'last_undegraded': '2024-09-16T15:50:29.398426+0000', 'last_fullsized': '2024-09-16T15:50:29.398426+0000', 'mapping_epoch': 144, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 144, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T15:50:29.373642+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T15:50:29.373642+0000', 'last_clean_scrub_stamp': '2024-09-16T15:50:29.373642+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 7], 'acting': [6, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-09-16T15:50:35.183 INFO:tasks.ceph.ceph_manager.ceph:PG 23.33 is not active+clean 2024-09-16T15:50:35.184 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '23.33', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 144, 'state': 'creating+peering', 'last_fresh': '2024-09-16T15:50:29.415630+0000', 'last_change': '2024-09-16T15:50:29.415630+0000', 'last_active': '2024-09-16T15:50:29.373642+0000', 'last_peered': '2024-09-16T15:50:29.373642+0000', 'last_clean': '2024-09-16T15:50:29.373642+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-16T15:50:29.415630+0000', 'last_undegraded': '2024-09-16T15:50:29.415630+0000', 'last_fullsized': '2024-09-16T15:50:29.415630+0000', 'mapping_epoch': 144, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 144, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T15:50:29.373642+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T15:50:29.373642+0000', 'last_clean_scrub_stamp': '2024-09-16T15:50:29.373642+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 3], 'acting': [5, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-09-16T15:50:35.184 INFO:tasks.ceph.ceph_manager.ceph:PG 23.32 is not active+clean 2024-09-16T15:50:35.184 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '23.32', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 144, 'state': 'creating+peering', 'last_fresh': '2024-09-16T15:50:29.411622+0000', 'last_change': '2024-09-16T15:50:29.411622+0000', 'last_active': '2024-09-16T15:50:29.373642+0000', 'last_peered': '2024-09-16T15:50:29.373642+0000', 'last_clean': '2024-09-16T15:50:29.373642+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-16T15:50:29.411622+0000', 'last_undegraded': '2024-09-16T15:50:29.411622+0000', 'last_fullsized': '2024-09-16T15:50:29.411622+0000', 'mapping_epoch': 144, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 144, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T15:50:29.373642+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T15:50:29.373642+0000', 'last_clean_scrub_stamp': '2024-09-16T15:50:29.373642+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 7], 'acting': [5, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-09-16T15:50:35.184 INFO:tasks.ceph.ceph_manager.ceph:PG 23.31 is not active+clean 2024-09-16T15:50:35.184 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '23.31', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 144, 'state': 'creating+peering', 'last_fresh': '2024-09-16T15:50:29.396532+0000', 'last_change': '2024-09-16T15:50:29.396532+0000', 'last_active': '2024-09-16T15:50:29.373642+0000', 'last_peered': '2024-09-16T15:50:29.373642+0000', 'last_clean': '2024-09-16T15:50:29.373642+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-16T15:50:29.396532+0000', 'last_undegraded': '2024-09-16T15:50:29.396532+0000', 'last_fullsized': '2024-09-16T15:50:29.396532+0000', 'mapping_epoch': 144, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 144, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T15:50:29.373642+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T15:50:29.373642+0000', 'last_clean_scrub_stamp': '2024-09-16T15:50:29.373642+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 1], 'acting': [7, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-09-16T15:50:35.184 INFO:tasks.ceph.ceph_manager.ceph:PG 23.30 is not active+clean 2024-09-16T15:50:35.184 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '23.30', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 144, 'state': 'creating+peering', 'last_fresh': '2024-09-16T15:50:29.398051+0000', 'last_change': '2024-09-16T15:50:29.398051+0000', 'last_active': '2024-09-16T15:50:29.373642+0000', 'last_peered': '2024-09-16T15:50:29.373642+0000', 'last_clean': '2024-09-16T15:50:29.373642+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-16T15:50:29.398051+0000', 'last_undegraded': '2024-09-16T15:50:29.398051+0000', 'last_fullsized': '2024-09-16T15:50:29.398051+0000', 'mapping_epoch': 144, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 144, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T15:50:29.373642+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T15:50:29.373642+0000', 'last_clean_scrub_stamp': '2024-09-16T15:50:29.373642+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 7], 'acting': [0, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T15:50:35.184 INFO:tasks.ceph.ceph_manager.ceph:PG 23.2f is not active+clean 2024-09-16T15:50:35.184 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '23.2f', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 144, 'state': 'creating+peering', 'last_fresh': '2024-09-16T15:50:29.399338+0000', 'last_change': '2024-09-16T15:50:29.399338+0000', 'last_active': '2024-09-16T15:50:29.373642+0000', 'last_peered': '2024-09-16T15:50:29.373642+0000', 'last_clean': '2024-09-16T15:50:29.373642+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-16T15:50:29.399338+0000', 'last_undegraded': '2024-09-16T15:50:29.399338+0000', 'last_fullsized': '2024-09-16T15:50:29.399338+0000', 'mapping_epoch': 144, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 144, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T15:50:29.373642+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T15:50:29.373642+0000', 'last_clean_scrub_stamp': '2024-09-16T15:50:29.373642+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 5], 'acting': [2, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T15:50:35.184 INFO:tasks.ceph.ceph_manager.ceph:PG 23.2e is not active+clean 2024-09-16T15:50:35.184 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '23.2e', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 144, 'state': 'creating+peering', 'last_fresh': '2024-09-16T15:50:29.403155+0000', 'last_change': '2024-09-16T15:50:29.403155+0000', 'last_active': '2024-09-16T15:50:29.373642+0000', 'last_peered': '2024-09-16T15:50:29.373642+0000', 'last_clean': '2024-09-16T15:50:29.373642+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-16T15:50:29.403155+0000', 'last_undegraded': '2024-09-16T15:50:29.403155+0000', 'last_fullsized': '2024-09-16T15:50:29.403155+0000', 'mapping_epoch': 144, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 144, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T15:50:29.373642+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T15:50:29.373642+0000', 'last_clean_scrub_stamp': '2024-09-16T15:50:29.373642+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T15:50:35.185 INFO:tasks.ceph.ceph_manager.ceph:PG 23.2d is not active+clean 2024-09-16T15:50:35.185 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '23.2d', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 144, 'state': 'creating+peering', 'last_fresh': '2024-09-16T15:50:29.399089+0000', 'last_change': '2024-09-16T15:50:29.399089+0000', 'last_active': '2024-09-16T15:50:29.373642+0000', 'last_peered': '2024-09-16T15:50:29.373642+0000', 'last_clean': '2024-09-16T15:50:29.373642+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-16T15:50:29.399089+0000', 'last_undegraded': '2024-09-16T15:50:29.399089+0000', 'last_fullsized': '2024-09-16T15:50:29.399089+0000', 'mapping_epoch': 144, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 144, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T15:50:29.373642+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T15:50:29.373642+0000', 'last_clean_scrub_stamp': '2024-09-16T15:50:29.373642+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 4], 'acting': [2, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T15:50:35.185 INFO:tasks.ceph.ceph_manager.ceph:PG 23.2c is not active+clean 2024-09-16T15:50:35.185 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '23.2c', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 144, 'state': 'creating+peering', 'last_fresh': '2024-09-16T15:50:29.400996+0000', 'last_change': '2024-09-16T15:50:29.400996+0000', 'last_active': '2024-09-16T15:50:29.373642+0000', 'last_peered': '2024-09-16T15:50:29.373642+0000', 'last_clean': '2024-09-16T15:50:29.373642+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-16T15:50:29.400996+0000', 'last_undegraded': '2024-09-16T15:50:29.400996+0000', 'last_fullsized': '2024-09-16T15:50:29.400996+0000', 'mapping_epoch': 144, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 144, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T15:50:29.373642+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T15:50:29.373642+0000', 'last_clean_scrub_stamp': '2024-09-16T15:50:29.373642+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 7], 'acting': [5, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-09-16T15:50:35.185 INFO:tasks.ceph.ceph_manager.ceph:PG 23.2b is not active+clean 2024-09-16T15:50:35.185 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '23.2b', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 144, 'state': 'creating+peering', 'last_fresh': '2024-09-16T15:50:29.395481+0000', 'last_change': '2024-09-16T15:50:29.395481+0000', 'last_active': '2024-09-16T15:50:29.373642+0000', 'last_peered': '2024-09-16T15:50:29.373642+0000', 'last_clean': '2024-09-16T15:50:29.373642+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-16T15:50:29.395481+0000', 'last_undegraded': '2024-09-16T15:50:29.395481+0000', 'last_fullsized': '2024-09-16T15:50:29.395481+0000', 'mapping_epoch': 144, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 144, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T15:50:29.373642+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T15:50:29.373642+0000', 'last_clean_scrub_stamp': '2024-09-16T15:50:29.373642+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 5], 'acting': [2, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T15:50:35.185 INFO:tasks.ceph.ceph_manager.ceph:PG 23.2a is not active+clean 2024-09-16T15:50:35.185 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '23.2a', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 144, 'state': 'creating+peering', 'last_fresh': '2024-09-16T15:50:29.396706+0000', 'last_change': '2024-09-16T15:50:29.396706+0000', 'last_active': '2024-09-16T15:50:29.373642+0000', 'last_peered': '2024-09-16T15:50:29.373642+0000', 'last_clean': '2024-09-16T15:50:29.373642+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-16T15:50:29.396706+0000', 'last_undegraded': '2024-09-16T15:50:29.396706+0000', 'last_fullsized': '2024-09-16T15:50:29.396706+0000', 'mapping_epoch': 144, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 144, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T15:50:29.373642+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T15:50:29.373642+0000', 'last_clean_scrub_stamp': '2024-09-16T15:50:29.373642+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 0], 'acting': [6, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-09-16T15:50:35.185 INFO:tasks.ceph.ceph_manager.ceph:PG 23.29 is not active+clean 2024-09-16T15:50:35.185 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '23.29', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 144, 'state': 'creating+peering', 'last_fresh': '2024-09-16T15:50:29.397404+0000', 'last_change': '2024-09-16T15:50:29.397404+0000', 'last_active': '2024-09-16T15:50:29.373642+0000', 'last_peered': '2024-09-16T15:50:29.373642+0000', 'last_clean': '2024-09-16T15:50:29.373642+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-16T15:50:29.397404+0000', 'last_undegraded': '2024-09-16T15:50:29.397404+0000', 'last_fullsized': '2024-09-16T15:50:29.397404+0000', 'mapping_epoch': 144, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 144, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T15:50:29.373642+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T15:50:29.373642+0000', 'last_clean_scrub_stamp': '2024-09-16T15:50:29.373642+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 6], 'acting': [2, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T15:50:35.185 INFO:tasks.ceph.ceph_manager.ceph:PG 23.28 is not active+clean 2024-09-16T15:50:35.185 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '23.28', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 144, 'state': 'creating+peering', 'last_fresh': '2024-09-16T15:50:29.396706+0000', 'last_change': '2024-09-16T15:50:29.396706+0000', 'last_active': '2024-09-16T15:50:29.373642+0000', 'last_peered': '2024-09-16T15:50:29.373642+0000', 'last_clean': '2024-09-16T15:50:29.373642+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-16T15:50:29.396706+0000', 'last_undegraded': '2024-09-16T15:50:29.396706+0000', 'last_fullsized': '2024-09-16T15:50:29.396706+0000', 'mapping_epoch': 144, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 144, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T15:50:29.373642+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T15:50:29.373642+0000', 'last_clean_scrub_stamp': '2024-09-16T15:50:29.373642+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 5], 'acting': [6, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-09-16T15:50:35.186 INFO:tasks.ceph.ceph_manager.ceph:PG 23.27 is not active+clean 2024-09-16T15:50:35.186 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '23.27', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 144, 'state': 'creating+peering', 'last_fresh': '2024-09-16T15:50:29.401802+0000', 'last_change': '2024-09-16T15:50:29.401802+0000', 'last_active': '2024-09-16T15:50:29.373642+0000', 'last_peered': '2024-09-16T15:50:29.373642+0000', 'last_clean': '2024-09-16T15:50:29.373642+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-16T15:50:29.401802+0000', 'last_undegraded': '2024-09-16T15:50:29.401802+0000', 'last_fullsized': '2024-09-16T15:50:29.401802+0000', 'mapping_epoch': 144, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 144, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T15:50:29.373642+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T15:50:29.373642+0000', 'last_clean_scrub_stamp': '2024-09-16T15:50:29.373642+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 4], 'acting': [5, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-09-16T15:50:35.186 INFO:tasks.ceph.ceph_manager.ceph:PG 23.26 is not active+clean 2024-09-16T15:50:35.186 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '23.26', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 144, 'state': 'creating+peering', 'last_fresh': '2024-09-16T15:50:29.397726+0000', 'last_change': '2024-09-16T15:50:29.397726+0000', 'last_active': '2024-09-16T15:50:29.373642+0000', 'last_peered': '2024-09-16T15:50:29.373642+0000', 'last_clean': '2024-09-16T15:50:29.373642+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-16T15:50:29.397726+0000', 'last_undegraded': '2024-09-16T15:50:29.397726+0000', 'last_fullsized': '2024-09-16T15:50:29.397726+0000', 'mapping_epoch': 144, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 144, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T15:50:29.373642+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T15:50:29.373642+0000', 'last_clean_scrub_stamp': '2024-09-16T15:50:29.373642+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T15:50:35.186 INFO:tasks.ceph.ceph_manager.ceph:PG 23.25 is not active+clean 2024-09-16T15:50:35.186 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '23.25', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 144, 'state': 'creating+peering', 'last_fresh': '2024-09-16T15:50:29.413671+0000', 'last_change': '2024-09-16T15:50:29.413671+0000', 'last_active': '2024-09-16T15:50:29.373642+0000', 'last_peered': '2024-09-16T15:50:29.373642+0000', 'last_clean': '2024-09-16T15:50:29.373642+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-16T15:50:29.413671+0000', 'last_undegraded': '2024-09-16T15:50:29.413671+0000', 'last_fullsized': '2024-09-16T15:50:29.413671+0000', 'mapping_epoch': 144, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 144, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T15:50:29.373642+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T15:50:29.373642+0000', 'last_clean_scrub_stamp': '2024-09-16T15:50:29.373642+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 1], 'acting': [5, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-09-16T15:50:35.186 INFO:tasks.ceph.ceph_manager.ceph:PG 23.24 is not active+clean 2024-09-16T15:50:35.186 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '23.24', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 144, 'state': 'creating+peering', 'last_fresh': '2024-09-16T15:50:29.400095+0000', 'last_change': '2024-09-16T15:50:29.400095+0000', 'last_active': '2024-09-16T15:50:29.373642+0000', 'last_peered': '2024-09-16T15:50:29.373642+0000', 'last_clean': '2024-09-16T15:50:29.373642+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-16T15:50:29.400095+0000', 'last_undegraded': '2024-09-16T15:50:29.400095+0000', 'last_fullsized': '2024-09-16T15:50:29.400095+0000', 'mapping_epoch': 144, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 144, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T15:50:29.373642+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T15:50:29.373642+0000', 'last_clean_scrub_stamp': '2024-09-16T15:50:29.373642+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 4], 'acting': [1, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-09-16T15:50:35.186 INFO:tasks.ceph.ceph_manager.ceph:PG 23.23 is not active+clean 2024-09-16T15:50:35.186 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '23.23', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 144, 'state': 'creating+peering', 'last_fresh': '2024-09-16T15:50:29.403015+0000', 'last_change': '2024-09-16T15:50:29.403015+0000', 'last_active': '2024-09-16T15:50:29.373642+0000', 'last_peered': '2024-09-16T15:50:29.373642+0000', 'last_clean': '2024-09-16T15:50:29.373642+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-16T15:50:29.403015+0000', 'last_undegraded': '2024-09-16T15:50:29.403015+0000', 'last_fullsized': '2024-09-16T15:50:29.403015+0000', 'mapping_epoch': 144, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 144, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T15:50:29.373642+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T15:50:29.373642+0000', 'last_clean_scrub_stamp': '2024-09-16T15:50:29.373642+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 1], 'acting': [5, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-09-16T15:50:35.186 INFO:tasks.ceph.ceph_manager.ceph:PG 23.22 is not active+clean 2024-09-16T15:50:35.186 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '23.22', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 144, 'state': 'creating+peering', 'last_fresh': '2024-09-16T15:50:29.390228+0000', 'last_change': '2024-09-16T15:50:29.390228+0000', 'last_active': '2024-09-16T15:50:29.373642+0000', 'last_peered': '2024-09-16T15:50:29.373642+0000', 'last_clean': '2024-09-16T15:50:29.373642+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-16T15:50:29.390228+0000', 'last_undegraded': '2024-09-16T15:50:29.390228+0000', 'last_fullsized': '2024-09-16T15:50:29.390228+0000', 'mapping_epoch': 144, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 144, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T15:50:29.373642+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T15:50:29.373642+0000', 'last_clean_scrub_stamp': '2024-09-16T15:50:29.373642+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 7], 'acting': [4, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-09-16T15:50:35.187 INFO:tasks.ceph.ceph_manager.ceph:PG 23.21 is not active+clean 2024-09-16T15:50:35.187 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '23.21', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 144, 'state': 'creating+peering', 'last_fresh': '2024-09-16T15:50:29.392793+0000', 'last_change': '2024-09-16T15:50:29.392793+0000', 'last_active': '2024-09-16T15:50:29.373642+0000', 'last_peered': '2024-09-16T15:50:29.373642+0000', 'last_clean': '2024-09-16T15:50:29.373642+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-16T15:50:29.392793+0000', 'last_undegraded': '2024-09-16T15:50:29.392793+0000', 'last_fullsized': '2024-09-16T15:50:29.392793+0000', 'mapping_epoch': 144, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 144, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T15:50:29.373642+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T15:50:29.373642+0000', 'last_clean_scrub_stamp': '2024-09-16T15:50:29.373642+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 0], 'acting': [4, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-09-16T15:50:35.187 INFO:tasks.ceph.ceph_manager.ceph:PG 23.20 is not active+clean 2024-09-16T15:50:35.187 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '23.20', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 144, 'state': 'creating+peering', 'last_fresh': '2024-09-16T15:50:29.400381+0000', 'last_change': '2024-09-16T15:50:29.400381+0000', 'last_active': '2024-09-16T15:50:29.373642+0000', 'last_peered': '2024-09-16T15:50:29.373642+0000', 'last_clean': '2024-09-16T15:50:29.373642+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-16T15:50:29.400381+0000', 'last_undegraded': '2024-09-16T15:50:29.400381+0000', 'last_fullsized': '2024-09-16T15:50:29.400381+0000', 'mapping_epoch': 144, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 144, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T15:50:29.373642+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T15:50:29.373642+0000', 'last_clean_scrub_stamp': '2024-09-16T15:50:29.373642+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T15:50:35.187 INFO:tasks.ceph.ceph_manager.ceph:PG 23.1e is not active+clean 2024-09-16T15:50:35.187 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '23.1e', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 144, 'state': 'creating+peering', 'last_fresh': '2024-09-16T15:50:29.396097+0000', 'last_change': '2024-09-16T15:50:29.396097+0000', 'last_active': '2024-09-16T15:50:29.373642+0000', 'last_peered': '2024-09-16T15:50:29.373642+0000', 'last_clean': '2024-09-16T15:50:29.373642+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-16T15:50:29.396097+0000', 'last_undegraded': '2024-09-16T15:50:29.396097+0000', 'last_fullsized': '2024-09-16T15:50:29.396097+0000', 'mapping_epoch': 144, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 144, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T15:50:29.373642+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T15:50:29.373642+0000', 'last_clean_scrub_stamp': '2024-09-16T15:50:29.373642+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T15:50:35.187 INFO:tasks.ceph.ceph_manager.ceph:PG 23.1f is not active+clean 2024-09-16T15:50:35.187 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '23.1f', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 144, 'state': 'creating+peering', 'last_fresh': '2024-09-16T15:50:29.395019+0000', 'last_change': '2024-09-16T15:50:29.395019+0000', 'last_active': '2024-09-16T15:50:29.373642+0000', 'last_peered': '2024-09-16T15:50:29.373642+0000', 'last_clean': '2024-09-16T15:50:29.373642+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-16T15:50:29.395019+0000', 'last_undegraded': '2024-09-16T15:50:29.395019+0000', 'last_fullsized': '2024-09-16T15:50:29.395019+0000', 'mapping_epoch': 144, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 144, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T15:50:29.373642+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T15:50:29.373642+0000', 'last_clean_scrub_stamp': '2024-09-16T15:50:29.373642+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 3], 'acting': [4, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-09-16T15:50:35.187 INFO:tasks.ceph.ceph_manager.ceph:PG 23.1c is not active+clean 2024-09-16T15:50:35.187 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '23.1c', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 144, 'state': 'creating+peering', 'last_fresh': '2024-09-16T15:50:29.395809+0000', 'last_change': '2024-09-16T15:50:29.395809+0000', 'last_active': '2024-09-16T15:50:29.373642+0000', 'last_peered': '2024-09-16T15:50:29.373642+0000', 'last_clean': '2024-09-16T15:50:29.373642+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-16T15:50:29.395809+0000', 'last_undegraded': '2024-09-16T15:50:29.395809+0000', 'last_fullsized': '2024-09-16T15:50:29.395809+0000', 'mapping_epoch': 144, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 144, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T15:50:29.373642+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T15:50:29.373642+0000', 'last_clean_scrub_stamp': '2024-09-16T15:50:29.373642+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 7], 'acting': [2, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T15:50:35.187 INFO:tasks.ceph.ceph_manager.ceph:PG 23.1d is not active+clean 2024-09-16T15:50:35.188 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '23.1d', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 144, 'state': 'creating+peering', 'last_fresh': '2024-09-16T15:50:29.412090+0000', 'last_change': '2024-09-16T15:50:29.412090+0000', 'last_active': '2024-09-16T15:50:29.373642+0000', 'last_peered': '2024-09-16T15:50:29.373642+0000', 'last_clean': '2024-09-16T15:50:29.373642+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-16T15:50:29.412090+0000', 'last_undegraded': '2024-09-16T15:50:29.412090+0000', 'last_fullsized': '2024-09-16T15:50:29.412090+0000', 'mapping_epoch': 144, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 144, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T15:50:29.373642+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T15:50:29.373642+0000', 'last_clean_scrub_stamp': '2024-09-16T15:50:29.373642+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 2], 'acting': [5, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-09-16T15:50:35.188 INFO:tasks.ceph.ceph_manager.ceph:PG 23.1a is not active+clean 2024-09-16T15:50:35.188 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '23.1a', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 144, 'state': 'creating+peering', 'last_fresh': '2024-09-16T15:50:29.396181+0000', 'last_change': '2024-09-16T15:50:29.396181+0000', 'last_active': '2024-09-16T15:50:29.373642+0000', 'last_peered': '2024-09-16T15:50:29.373642+0000', 'last_clean': '2024-09-16T15:50:29.373642+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-16T15:50:29.396181+0000', 'last_undegraded': '2024-09-16T15:50:29.396181+0000', 'last_fullsized': '2024-09-16T15:50:29.396181+0000', 'mapping_epoch': 144, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 144, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T15:50:29.373642+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T15:50:29.373642+0000', 'last_clean_scrub_stamp': '2024-09-16T15:50:29.373642+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3], 'acting': [0, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T15:50:35.188 INFO:tasks.ceph.ceph_manager.ceph:PG 23.1b is not active+clean 2024-09-16T15:50:35.188 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '23.1b', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 144, 'state': 'creating+peering', 'last_fresh': '2024-09-16T15:50:29.396991+0000', 'last_change': '2024-09-16T15:50:29.396991+0000', 'last_active': '2024-09-16T15:50:29.373642+0000', 'last_peered': '2024-09-16T15:50:29.373642+0000', 'last_clean': '2024-09-16T15:50:29.373642+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-16T15:50:29.396991+0000', 'last_undegraded': '2024-09-16T15:50:29.396991+0000', 'last_fullsized': '2024-09-16T15:50:29.396991+0000', 'mapping_epoch': 144, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 144, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T15:50:29.373642+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T15:50:29.373642+0000', 'last_clean_scrub_stamp': '2024-09-16T15:50:29.373642+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [1, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-09-16T15:50:35.188 INFO:tasks.ceph.ceph_manager.ceph:PG 23.18 is not active+clean 2024-09-16T15:50:35.188 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '23.18', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 144, 'state': 'creating+peering', 'last_fresh': '2024-09-16T15:50:29.397299+0000', 'last_change': '2024-09-16T15:50:29.397299+0000', 'last_active': '2024-09-16T15:50:29.373642+0000', 'last_peered': '2024-09-16T15:50:29.373642+0000', 'last_clean': '2024-09-16T15:50:29.373642+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-16T15:50:29.397299+0000', 'last_undegraded': '2024-09-16T15:50:29.397299+0000', 'last_fullsized': '2024-09-16T15:50:29.397299+0000', 'mapping_epoch': 144, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 144, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T15:50:29.373642+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T15:50:29.373642+0000', 'last_clean_scrub_stamp': '2024-09-16T15:50:29.373642+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 7], 'acting': [1, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-09-16T15:50:35.188 INFO:tasks.ceph.ceph_manager.ceph:PG 23.19 is not active+clean 2024-09-16T15:50:35.188 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '23.19', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 144, 'state': 'creating+peering', 'last_fresh': '2024-09-16T15:50:29.396885+0000', 'last_change': '2024-09-16T15:50:29.396885+0000', 'last_active': '2024-09-16T15:50:29.373642+0000', 'last_peered': '2024-09-16T15:50:29.373642+0000', 'last_clean': '2024-09-16T15:50:29.373642+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-16T15:50:29.396885+0000', 'last_undegraded': '2024-09-16T15:50:29.396885+0000', 'last_fullsized': '2024-09-16T15:50:29.396885+0000', 'mapping_epoch': 144, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 144, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T15:50:29.373642+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T15:50:29.373642+0000', 'last_clean_scrub_stamp': '2024-09-16T15:50:29.373642+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0], 'acting': [2, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T15:50:35.188 INFO:tasks.ceph.ceph_manager.ceph:PG 23.17 is not active+clean 2024-09-16T15:50:35.188 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '23.17', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 144, 'state': 'creating+peering', 'last_fresh': '2024-09-16T15:50:29.403320+0000', 'last_change': '2024-09-16T15:50:29.403320+0000', 'last_active': '2024-09-16T15:50:29.373642+0000', 'last_peered': '2024-09-16T15:50:29.373642+0000', 'last_clean': '2024-09-16T15:50:29.373642+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-16T15:50:29.403320+0000', 'last_undegraded': '2024-09-16T15:50:29.403320+0000', 'last_fullsized': '2024-09-16T15:50:29.403320+0000', 'mapping_epoch': 144, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 144, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T15:50:29.373642+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T15:50:29.373642+0000', 'last_clean_scrub_stamp': '2024-09-16T15:50:29.373642+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1], 'acting': [3, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T15:50:35.189 INFO:tasks.ceph.ceph_manager.ceph:PG 23.14 is not active+clean 2024-09-16T15:50:35.189 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '23.14', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 144, 'state': 'creating+peering', 'last_fresh': '2024-09-16T15:50:29.401245+0000', 'last_change': '2024-09-16T15:50:29.401245+0000', 'last_active': '2024-09-16T15:50:29.373642+0000', 'last_peered': '2024-09-16T15:50:29.373642+0000', 'last_clean': '2024-09-16T15:50:29.373642+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-16T15:50:29.401245+0000', 'last_undegraded': '2024-09-16T15:50:29.401245+0000', 'last_fullsized': '2024-09-16T15:50:29.401245+0000', 'mapping_epoch': 144, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 144, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T15:50:29.373642+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T15:50:29.373642+0000', 'last_clean_scrub_stamp': '2024-09-16T15:50:29.373642+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1], 'acting': [3, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T15:50:35.189 INFO:tasks.ceph.ceph_manager.ceph:PG 23.15 is not active+clean 2024-09-16T15:50:35.189 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '23.15', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 144, 'state': 'creating+peering', 'last_fresh': '2024-09-16T15:50:29.390531+0000', 'last_change': '2024-09-16T15:50:29.390531+0000', 'last_active': '2024-09-16T15:50:29.373642+0000', 'last_peered': '2024-09-16T15:50:29.373642+0000', 'last_clean': '2024-09-16T15:50:29.373642+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-16T15:50:29.390531+0000', 'last_undegraded': '2024-09-16T15:50:29.390531+0000', 'last_fullsized': '2024-09-16T15:50:29.390531+0000', 'mapping_epoch': 144, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 144, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T15:50:29.373642+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T15:50:29.373642+0000', 'last_clean_scrub_stamp': '2024-09-16T15:50:29.373642+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 5], 'acting': [4, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-09-16T15:50:35.189 INFO:tasks.ceph.ceph_manager.ceph:PG 23.12 is not active+clean 2024-09-16T15:50:35.189 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '23.12', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 144, 'state': 'creating+peering', 'last_fresh': '2024-09-16T15:50:29.401262+0000', 'last_change': '2024-09-16T15:50:29.401262+0000', 'last_active': '2024-09-16T15:50:29.373642+0000', 'last_peered': '2024-09-16T15:50:29.373642+0000', 'last_clean': '2024-09-16T15:50:29.373642+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-16T15:50:29.401262+0000', 'last_undegraded': '2024-09-16T15:50:29.401262+0000', 'last_fullsized': '2024-09-16T15:50:29.401262+0000', 'mapping_epoch': 144, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 144, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T15:50:29.373642+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T15:50:29.373642+0000', 'last_clean_scrub_stamp': '2024-09-16T15:50:29.373642+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 6], 'acting': [3, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T15:50:35.189 INFO:tasks.ceph.ceph_manager.ceph:PG 23.13 is not active+clean 2024-09-16T15:50:35.189 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '23.13', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 144, 'state': 'creating+peering', 'last_fresh': '2024-09-16T15:50:29.396084+0000', 'last_change': '2024-09-16T15:50:29.396084+0000', 'last_active': '2024-09-16T15:50:29.373642+0000', 'last_peered': '2024-09-16T15:50:29.373642+0000', 'last_clean': '2024-09-16T15:50:29.373642+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-16T15:50:29.396084+0000', 'last_undegraded': '2024-09-16T15:50:29.396084+0000', 'last_fullsized': '2024-09-16T15:50:29.396084+0000', 'mapping_epoch': 144, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 144, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T15:50:29.373642+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T15:50:29.373642+0000', 'last_clean_scrub_stamp': '2024-09-16T15:50:29.373642+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 5], 'acting': [0, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T15:50:35.189 INFO:tasks.ceph.ceph_manager.ceph:PG 23.10 is not active+clean 2024-09-16T15:50:35.189 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '23.10', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 144, 'state': 'creating+peering', 'last_fresh': '2024-09-16T15:50:29.396732+0000', 'last_change': '2024-09-16T15:50:29.396732+0000', 'last_active': '2024-09-16T15:50:29.373642+0000', 'last_peered': '2024-09-16T15:50:29.373642+0000', 'last_clean': '2024-09-16T15:50:29.373642+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-16T15:50:29.396732+0000', 'last_undegraded': '2024-09-16T15:50:29.396732+0000', 'last_fullsized': '2024-09-16T15:50:29.396732+0000', 'mapping_epoch': 144, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 144, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T15:50:29.373642+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T15:50:29.373642+0000', 'last_clean_scrub_stamp': '2024-09-16T15:50:29.373642+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 5], 'acting': [1, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-09-16T15:50:35.189 INFO:tasks.ceph.ceph_manager.ceph:PG 23.11 is not active+clean 2024-09-16T15:50:35.190 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '23.11', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 144, 'state': 'creating+peering', 'last_fresh': '2024-09-16T15:50:29.418052+0000', 'last_change': '2024-09-16T15:50:29.418052+0000', 'last_active': '2024-09-16T15:50:29.373642+0000', 'last_peered': '2024-09-16T15:50:29.373642+0000', 'last_clean': '2024-09-16T15:50:29.373642+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-16T15:50:29.418052+0000', 'last_undegraded': '2024-09-16T15:50:29.418052+0000', 'last_fullsized': '2024-09-16T15:50:29.418052+0000', 'mapping_epoch': 144, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 144, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T15:50:29.373642+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T15:50:29.373642+0000', 'last_clean_scrub_stamp': '2024-09-16T15:50:29.373642+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 0], 'acting': [5, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-09-16T15:50:35.190 INFO:tasks.ceph.ceph_manager.ceph:PG 23.e is not active+clean 2024-09-16T15:50:35.190 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '23.e', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 144, 'state': 'creating+peering', 'last_fresh': '2024-09-16T15:50:29.396821+0000', 'last_change': '2024-09-16T15:50:29.396821+0000', 'last_active': '2024-09-16T15:50:29.373642+0000', 'last_peered': '2024-09-16T15:50:29.373642+0000', 'last_clean': '2024-09-16T15:50:29.373642+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-16T15:50:29.396821+0000', 'last_undegraded': '2024-09-16T15:50:29.396821+0000', 'last_fullsized': '2024-09-16T15:50:29.396821+0000', 'mapping_epoch': 144, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 144, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T15:50:29.373642+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T15:50:29.373642+0000', 'last_clean_scrub_stamp': '2024-09-16T15:50:29.373642+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [1, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-09-16T15:50:35.190 INFO:tasks.ceph.ceph_manager.ceph:PG 23.f is not active+clean 2024-09-16T15:50:35.190 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '23.f', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 144, 'state': 'creating+peering', 'last_fresh': '2024-09-16T15:50:29.402843+0000', 'last_change': '2024-09-16T15:50:29.402843+0000', 'last_active': '2024-09-16T15:50:29.373642+0000', 'last_peered': '2024-09-16T15:50:29.373642+0000', 'last_clean': '2024-09-16T15:50:29.373642+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-16T15:50:29.402843+0000', 'last_undegraded': '2024-09-16T15:50:29.402843+0000', 'last_fullsized': '2024-09-16T15:50:29.402843+0000', 'mapping_epoch': 144, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 144, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T15:50:29.373642+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T15:50:29.373642+0000', 'last_clean_scrub_stamp': '2024-09-16T15:50:29.373642+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 4], 'acting': [3, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T15:50:35.190 INFO:tasks.ceph.ceph_manager.ceph:PG 23.c is not active+clean 2024-09-16T15:50:35.190 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '23.c', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 144, 'state': 'creating+peering', 'last_fresh': '2024-09-16T15:50:29.399219+0000', 'last_change': '2024-09-16T15:50:29.399219+0000', 'last_active': '2024-09-16T15:50:29.373642+0000', 'last_peered': '2024-09-16T15:50:29.373642+0000', 'last_clean': '2024-09-16T15:50:29.373642+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-16T15:50:29.399219+0000', 'last_undegraded': '2024-09-16T15:50:29.399219+0000', 'last_fullsized': '2024-09-16T15:50:29.399219+0000', 'mapping_epoch': 144, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 144, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T15:50:29.373642+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T15:50:29.373642+0000', 'last_clean_scrub_stamp': '2024-09-16T15:50:29.373642+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [1, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-09-16T15:50:35.190 INFO:tasks.ceph.ceph_manager.ceph:PG 23.d is not active+clean 2024-09-16T15:50:35.190 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '23.d', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 144, 'state': 'creating+peering', 'last_fresh': '2024-09-16T15:50:29.395185+0000', 'last_change': '2024-09-16T15:50:29.395185+0000', 'last_active': '2024-09-16T15:50:29.373642+0000', 'last_peered': '2024-09-16T15:50:29.373642+0000', 'last_clean': '2024-09-16T15:50:29.373642+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-16T15:50:29.395185+0000', 'last_undegraded': '2024-09-16T15:50:29.395185+0000', 'last_fullsized': '2024-09-16T15:50:29.395185+0000', 'mapping_epoch': 144, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 144, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T15:50:29.373642+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T15:50:29.373642+0000', 'last_clean_scrub_stamp': '2024-09-16T15:50:29.373642+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 4], 'acting': [0, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T15:50:35.190 INFO:tasks.ceph.ceph_manager.ceph:PG 23.a is not active+clean 2024-09-16T15:50:35.190 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '23.a', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 144, 'state': 'creating+peering', 'last_fresh': '2024-09-16T15:50:29.400460+0000', 'last_change': '2024-09-16T15:50:29.400460+0000', 'last_active': '2024-09-16T15:50:29.373642+0000', 'last_peered': '2024-09-16T15:50:29.373642+0000', 'last_clean': '2024-09-16T15:50:29.373642+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-16T15:50:29.400460+0000', 'last_undegraded': '2024-09-16T15:50:29.400460+0000', 'last_fullsized': '2024-09-16T15:50:29.400460+0000', 'mapping_epoch': 144, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 144, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T15:50:29.373642+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T15:50:29.373642+0000', 'last_clean_scrub_stamp': '2024-09-16T15:50:29.373642+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 3], 'acting': [5, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-09-16T15:50:35.191 INFO:tasks.ceph.ceph_manager.ceph:PG 23.b is not active+clean 2024-09-16T15:50:35.191 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '23.b', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 144, 'state': 'creating+peering', 'last_fresh': '2024-09-16T15:50:29.394330+0000', 'last_change': '2024-09-16T15:50:29.394330+0000', 'last_active': '2024-09-16T15:50:29.373642+0000', 'last_peered': '2024-09-16T15:50:29.373642+0000', 'last_clean': '2024-09-16T15:50:29.373642+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-16T15:50:29.394330+0000', 'last_undegraded': '2024-09-16T15:50:29.394330+0000', 'last_fullsized': '2024-09-16T15:50:29.394330+0000', 'mapping_epoch': 144, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 144, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T15:50:29.373642+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T15:50:29.373642+0000', 'last_clean_scrub_stamp': '2024-09-16T15:50:29.373642+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 7], 'acting': [0, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T15:50:35.191 INFO:tasks.ceph.ceph_manager.ceph:PG 23.8 is not active+clean 2024-09-16T15:50:35.191 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '23.8', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 144, 'state': 'creating+peering', 'last_fresh': '2024-09-16T15:50:29.397861+0000', 'last_change': '2024-09-16T15:50:29.397861+0000', 'last_active': '2024-09-16T15:50:29.373642+0000', 'last_peered': '2024-09-16T15:50:29.373642+0000', 'last_clean': '2024-09-16T15:50:29.373642+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-16T15:50:29.397861+0000', 'last_undegraded': '2024-09-16T15:50:29.397861+0000', 'last_fullsized': '2024-09-16T15:50:29.397861+0000', 'mapping_epoch': 144, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 144, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T15:50:29.373642+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T15:50:29.373642+0000', 'last_clean_scrub_stamp': '2024-09-16T15:50:29.373642+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 6], 'acting': [7, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-09-16T15:50:35.191 INFO:tasks.ceph.ceph_manager.ceph:PG 23.9 is not active+clean 2024-09-16T15:50:35.191 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '23.9', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 144, 'state': 'creating+peering', 'last_fresh': '2024-09-16T15:50:29.396384+0000', 'last_change': '2024-09-16T15:50:29.396384+0000', 'last_active': '2024-09-16T15:50:29.373642+0000', 'last_peered': '2024-09-16T15:50:29.373642+0000', 'last_clean': '2024-09-16T15:50:29.373642+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-16T15:50:29.396384+0000', 'last_undegraded': '2024-09-16T15:50:29.396384+0000', 'last_fullsized': '2024-09-16T15:50:29.396384+0000', 'mapping_epoch': 144, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 144, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T15:50:29.373642+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T15:50:29.373642+0000', 'last_clean_scrub_stamp': '2024-09-16T15:50:29.373642+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 6], 'acting': [2, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T15:50:35.191 INFO:tasks.ceph.ceph_manager.ceph:PG 23.6 is not active+clean 2024-09-16T15:50:35.191 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '23.6', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 144, 'state': 'creating+peering', 'last_fresh': '2024-09-16T15:50:29.400497+0000', 'last_change': '2024-09-16T15:50:29.400497+0000', 'last_active': '2024-09-16T15:50:29.373642+0000', 'last_peered': '2024-09-16T15:50:29.373642+0000', 'last_clean': '2024-09-16T15:50:29.373642+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-16T15:50:29.400497+0000', 'last_undegraded': '2024-09-16T15:50:29.400497+0000', 'last_fullsized': '2024-09-16T15:50:29.400497+0000', 'mapping_epoch': 144, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 144, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T15:50:29.373642+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T15:50:29.373642+0000', 'last_clean_scrub_stamp': '2024-09-16T15:50:29.373642+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1], 'acting': [3, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T15:50:35.191 INFO:tasks.ceph.ceph_manager.ceph:PG 23.7 is not active+clean 2024-09-16T15:50:35.191 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '23.7', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 144, 'state': 'creating+peering', 'last_fresh': '2024-09-16T15:50:29.394083+0000', 'last_change': '2024-09-16T15:50:29.394083+0000', 'last_active': '2024-09-16T15:50:29.373642+0000', 'last_peered': '2024-09-16T15:50:29.373642+0000', 'last_clean': '2024-09-16T15:50:29.373642+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-16T15:50:29.394083+0000', 'last_undegraded': '2024-09-16T15:50:29.394083+0000', 'last_fullsized': '2024-09-16T15:50:29.394083+0000', 'mapping_epoch': 144, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 144, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T15:50:29.373642+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T15:50:29.373642+0000', 'last_clean_scrub_stamp': '2024-09-16T15:50:29.373642+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 6], 'acting': [0, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T15:50:35.191 INFO:tasks.ceph.ceph_manager.ceph:PG 23.4 is not active+clean 2024-09-16T15:50:35.192 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '23.4', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 144, 'state': 'creating+peering', 'last_fresh': '2024-09-16T15:50:29.395179+0000', 'last_change': '2024-09-16T15:50:29.395179+0000', 'last_active': '2024-09-16T15:50:29.373642+0000', 'last_peered': '2024-09-16T15:50:29.373642+0000', 'last_clean': '2024-09-16T15:50:29.373642+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-16T15:50:29.395179+0000', 'last_undegraded': '2024-09-16T15:50:29.395179+0000', 'last_fullsized': '2024-09-16T15:50:29.395179+0000', 'mapping_epoch': 144, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 144, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T15:50:29.373642+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T15:50:29.373642+0000', 'last_clean_scrub_stamp': '2024-09-16T15:50:29.373642+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T15:50:35.192 INFO:tasks.ceph.ceph_manager.ceph:PG 23.5 is not active+clean 2024-09-16T15:50:35.192 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '23.5', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 144, 'state': 'creating+peering', 'last_fresh': '2024-09-16T15:50:29.400308+0000', 'last_change': '2024-09-16T15:50:29.400308+0000', 'last_active': '2024-09-16T15:50:29.373642+0000', 'last_peered': '2024-09-16T15:50:29.373642+0000', 'last_clean': '2024-09-16T15:50:29.373642+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-16T15:50:29.400308+0000', 'last_undegraded': '2024-09-16T15:50:29.400308+0000', 'last_fullsized': '2024-09-16T15:50:29.400308+0000', 'mapping_epoch': 144, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 144, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T15:50:29.373642+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T15:50:29.373642+0000', 'last_clean_scrub_stamp': '2024-09-16T15:50:29.373642+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 7], 'acting': [3, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T15:50:35.192 INFO:tasks.ceph.ceph_manager.ceph:PG 23.2 is not active+clean 2024-09-16T15:50:35.192 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '23.2', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 144, 'state': 'creating+peering', 'last_fresh': '2024-09-16T15:50:29.400856+0000', 'last_change': '2024-09-16T15:50:29.400856+0000', 'last_active': '2024-09-16T15:50:29.373642+0000', 'last_peered': '2024-09-16T15:50:29.373642+0000', 'last_clean': '2024-09-16T15:50:29.373642+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-16T15:50:29.400856+0000', 'last_undegraded': '2024-09-16T15:50:29.400856+0000', 'last_fullsized': '2024-09-16T15:50:29.400856+0000', 'mapping_epoch': 144, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 144, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T15:50:29.373642+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T15:50:29.373642+0000', 'last_clean_scrub_stamp': '2024-09-16T15:50:29.373642+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T15:50:35.192 INFO:tasks.ceph.ceph_manager.ceph:PG 23.3 is not active+clean 2024-09-16T15:50:35.192 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '23.3', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 144, 'state': 'creating+peering', 'last_fresh': '2024-09-16T15:50:29.391321+0000', 'last_change': '2024-09-16T15:50:29.391321+0000', 'last_active': '2024-09-16T15:50:29.373642+0000', 'last_peered': '2024-09-16T15:50:29.373642+0000', 'last_clean': '2024-09-16T15:50:29.373642+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-16T15:50:29.391321+0000', 'last_undegraded': '2024-09-16T15:50:29.391321+0000', 'last_fullsized': '2024-09-16T15:50:29.391321+0000', 'mapping_epoch': 144, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 144, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T15:50:29.373642+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T15:50:29.373642+0000', 'last_clean_scrub_stamp': '2024-09-16T15:50:29.373642+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 0], 'acting': [4, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-09-16T15:50:35.192 INFO:tasks.ceph.ceph_manager.ceph:PG 23.0 is not active+clean 2024-09-16T15:50:35.192 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '23.0', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 144, 'state': 'creating+peering', 'last_fresh': '2024-09-16T15:50:29.394199+0000', 'last_change': '2024-09-16T15:50:29.394199+0000', 'last_active': '2024-09-16T15:50:29.373642+0000', 'last_peered': '2024-09-16T15:50:29.373642+0000', 'last_clean': '2024-09-16T15:50:29.373642+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-16T15:50:29.394199+0000', 'last_undegraded': '2024-09-16T15:50:29.394199+0000', 'last_fullsized': '2024-09-16T15:50:29.394199+0000', 'mapping_epoch': 144, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 144, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T15:50:29.373642+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T15:50:29.373642+0000', 'last_clean_scrub_stamp': '2024-09-16T15:50:29.373642+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 7], 'acting': [0, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T15:50:35.192 INFO:tasks.ceph.ceph_manager.ceph:PG 23.1 is not active+clean 2024-09-16T15:50:35.193 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '23.1', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 144, 'state': 'creating+peering', 'last_fresh': '2024-09-16T15:50:29.409753+0000', 'last_change': '2024-09-16T15:50:29.409753+0000', 'last_active': '2024-09-16T15:50:29.373642+0000', 'last_peered': '2024-09-16T15:50:29.373642+0000', 'last_clean': '2024-09-16T15:50:29.373642+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-16T15:50:29.409753+0000', 'last_undegraded': '2024-09-16T15:50:29.409753+0000', 'last_fullsized': '2024-09-16T15:50:29.409753+0000', 'mapping_epoch': 144, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 144, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T15:50:29.373642+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T15:50:29.373642+0000', 'last_clean_scrub_stamp': '2024-09-16T15:50:29.373642+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 2], 'acting': [5, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-09-16T15:50:35.193 INFO:tasks.ceph.ceph_manager.ceph:PG 23.16 is not active+clean 2024-09-16T15:50:35.193 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '23.16', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 144, 'state': 'creating+peering', 'last_fresh': '2024-09-16T15:50:29.401057+0000', 'last_change': '2024-09-16T15:50:29.401057+0000', 'last_active': '2024-09-16T15:50:29.373642+0000', 'last_peered': '2024-09-16T15:50:29.373642+0000', 'last_clean': '2024-09-16T15:50:29.373642+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-16T15:50:29.401057+0000', 'last_undegraded': '2024-09-16T15:50:29.401057+0000', 'last_fullsized': '2024-09-16T15:50:29.401057+0000', 'mapping_epoch': 144, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 144, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T15:50:29.373642+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T15:50:29.373642+0000', 'last_clean_scrub_stamp': '2024-09-16T15:50:29.373642+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1], 'acting': [3, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T15:50:35.193 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T15:50:35.332 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:35.334+0000 7f3f10368640 1 -- 172.21.15.123:0/1295375182 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f3f080c5f20 msgr2=0x7f3f080cd900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:50:35.332 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:35.334+0000 7f3f10368640 1 --2- 172.21.15.123:0/1295375182 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f3f080c5f20 0x7f3f080cd900 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f3efc00aa10 tx=0x7f3efc020420 comp rx=0 tx=0).stop 2024-09-16T15:50:35.332 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:35.334+0000 7f3f10368640 1 -- 172.21.15.123:0/1295375182 shutdown_connections 2024-09-16T15:50:35.333 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:35.334+0000 7f3f10368640 1 --2- 172.21.15.123:0/1295375182 >> v2:172.21.15.184:3300/0 conn(0x7f3f080cde40 0x7f3f080ca200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:35.333 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:35.334+0000 7f3f10368640 1 --2- 172.21.15.123:0/1295375182 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f3f080c5f20 0x7f3f080cd900 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:35.333 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:35.334+0000 7f3f10368640 1 -- 172.21.15.123:0/1295375182 >> 172.21.15.123:0/1295375182 conn(0x7f3f080cfa90 msgr2=0x7f3f080cfea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:50:35.333 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:35.334+0000 7f3f10368640 1 -- 172.21.15.123:0/1295375182 shutdown_connections 2024-09-16T15:50:35.333 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:35.334+0000 7f3f10368640 1 -- 172.21.15.123:0/1295375182 wait complete. 2024-09-16T15:50:35.333 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:35.334+0000 7f3f10368640 1 Processor -- start 2024-09-16T15:50:35.333 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:35.334+0000 7f3f10368640 1 -- start start 2024-09-16T15:50:35.335 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:35.334+0000 7f3f10368640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3f080c5f20 0x7f3f081594c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:50:35.336 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:35.334+0000 7f3f10368640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f3f080cde40 0x7f3f0814df10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:50:35.336 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:35.334+0000 7f3f10368640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3f08151400 0x7f3f08152b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:50:35.336 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:35.334+0000 7f3f10368640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f3f0806e860 con 0x7f3f08151400 2024-09-16T15:50:35.336 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:35.334+0000 7f3f10368640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f3f0806e6e0 con 0x7f3f080c5f20 2024-09-16T15:50:35.336 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:35.334+0000 7f3f10368640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f3f0806e9e0 con 0x7f3f080cde40 2024-09-16T15:50:35.336 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:35.334+0000 7f3f0e905640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3f08151400 0x7f3f08152b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:50:35.336 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:35.334+0000 7f3f0e905640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3f08151400 0x7f3f08152b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:49704/0 (socket says 172.21.15.123:49704) 2024-09-16T15:50:35.336 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:35.334+0000 7f3f0e905640 1 -- 172.21.15.123:0/2718684887 learned_addr learned my addr 172.21.15.123:0/2718684887 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:50:35.336 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:35.334+0000 7f3f0d903640 1 --2- 172.21.15.123:0/2718684887 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f3f080cde40 0x7f3f0814df10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:50:35.336 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:35.334+0000 7f3f0e104640 1 --2- 172.21.15.123:0/2718684887 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3f080c5f20 0x7f3f081594c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:50:35.336 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:35.334+0000 7f3f0e905640 1 -- 172.21.15.123:0/2718684887 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f3f080cde40 msgr2=0x7f3f0814df10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:50:35.336 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:35.334+0000 7f3f0e905640 1 --2- 172.21.15.123:0/2718684887 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f3f080cde40 0x7f3f0814df10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:35.337 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:35.334+0000 7f3f0e905640 1 -- 172.21.15.123:0/2718684887 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3f080c5f20 msgr2=0x7f3f081594c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:50:35.337 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:35.334+0000 7f3f0e905640 1 --2- 172.21.15.123:0/2718684887 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3f080c5f20 0x7f3f081594c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:35.337 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:35.334+0000 7f3f0e905640 1 -- 172.21.15.123:0/2718684887 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3f081f52d0 con 0x7f3f08151400 2024-09-16T15:50:35.337 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:35.334+0000 7f3f0e905640 1 --2- 172.21.15.123:0/2718684887 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3f08151400 0x7f3f08152b80 secure :-1 s=READY pgs=1598 cs=0 l=1 rev1=1 crypto rx=0x7f3f0000e8c0 tx=0x7f3f0000ed80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:50:35.337 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:35.334+0000 7f3f06ffd640 1 -- 172.21.15.123:0/2718684887 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f0000ccc0 con 0x7f3f08151400 2024-09-16T15:50:35.337 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:35.334+0000 7f3f06ffd640 1 -- 172.21.15.123:0/2718684887 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f3f0000ce60 con 0x7f3f08151400 2024-09-16T15:50:35.337 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:35.334+0000 7f3f10368640 1 -- 172.21.15.123:0/2718684887 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3f081f55b0 con 0x7f3f08151400 2024-09-16T15:50:35.337 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:35.334+0000 7f3f06ffd640 1 -- 172.21.15.123:0/2718684887 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f00005280 con 0x7f3f08151400 2024-09-16T15:50:35.337 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:35.334+0000 7f3f10368640 1 -- 172.21.15.123:0/2718684887 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3f081f5b70 con 0x7f3f08151400 2024-09-16T15:50:35.338 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:35.338+0000 7f3f10368640 1 -- 172.21.15.123:0/2718684887 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3f080cd900 con 0x7f3f08151400 2024-09-16T15:50:35.338 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:35.338+0000 7f3f06ffd640 1 -- 172.21.15.123:0/2718684887 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 22) ==== 97692+0+0 (secure 0 0 0) 0x7f3f00005420 con 0x7f3f08151400 2024-09-16T15:50:35.338 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:35.338+0000 7f3f06ffd640 1 --2- 172.21.15.123:0/2718684887 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f3ef4073580 0x7f3ef4075a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:50:35.338 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:35.338+0000 7f3f06ffd640 1 -- 172.21.15.123:0/2718684887 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(146..146 src has 1..146) ==== 5737+0+0 (secure 0 0 0) 0x7f3f00095480 con 0x7f3f08151400 2024-09-16T15:50:35.338 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:35.338+0000 7f3f0e104640 1 --2- 172.21.15.123:0/2718684887 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f3ef4073580 0x7f3ef4075a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:50:35.338 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:35.338+0000 7f3f0e104640 1 --2- 172.21.15.123:0/2718684887 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f3ef4073580 0x7f3ef4075a40 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f3ef8001cf0 tx=0x7f3ef8009040 comp rx=0 tx=0).ready entity=mgr.4150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:50:35.341 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:35.342+0000 7f3f06ffd640 1 -- 172.21.15.123:0/2718684887 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f3f0005fda0 con 0x7f3f08151400 2024-09-16T15:50:35.561 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:35.562+0000 7f3f10368640 1 -- 172.21.15.123:0/2718684887 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "status", "format": "json"} v 0) -- 0x7f3f0815be80 con 0x7f3f08151400 2024-09-16T15:50:35.562 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:35.562+0000 7f3f06ffd640 1 -- 172.21.15.123:0/2718684887 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json"}]=0 v0) ==== 72+0+1499 (secure 0 0 0) 0x7f3f0005f6a0 con 0x7f3f08151400 2024-09-16T15:50:35.562 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-09-16T15:50:35.562 INFO:teuthology.orchestra.run.smithi123.stdout:{"fsid":"56077f89-18c8-486f-ae7e-eecc0bb7822c","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":489,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":146,"num_osds":8,"num_up_osds":8,"osd_up_since":1726501358,"num_in_osds":8,"osd_in_since":1726501349,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"creating+peering","count":64},{"state_name":"active+clean","count":33}],"num_pgs":97,"num_pools":3,"num_objects":2,"data_bytes":590368,"bytes_used":275357696,"bytes_avail":772818755584,"bytes_total":773094113280,"inactive_pgs_ratio":0.65979379415512085},"fsmap":{"epoch":95,"btime":"2024-09-16T15:50:33:425542+0000","id":11,"up":1,"in":1,"max":1,"by_rank":[{"filesystem_id":11,"rank":0,"name":"b","status":"up:active","gid":8436}],"up:standby":3},"mgrmap":{"available":true,"num_standbys":1,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":13,"modified":"2024-09-16T15:49:42.105365+0000","services":{"mds":{"daemons":{"summary":"","b":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"c":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"d":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-09-16T15:50:35.564 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:35.566+0000 7f3f10368640 1 -- 172.21.15.123:0/2718684887 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f3ef4073580 msgr2=0x7f3ef4075a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:50:35.564 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:35.566+0000 7f3f10368640 1 --2- 172.21.15.123:0/2718684887 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f3ef4073580 0x7f3ef4075a40 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f3ef8001cf0 tx=0x7f3ef8009040 comp rx=0 tx=0).stop 2024-09-16T15:50:35.564 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:35.566+0000 7f3f10368640 1 -- 172.21.15.123:0/2718684887 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3f08151400 msgr2=0x7f3f08152b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:50:35.564 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:35.566+0000 7f3f10368640 1 --2- 172.21.15.123:0/2718684887 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3f08151400 0x7f3f08152b80 secure :-1 s=READY pgs=1598 cs=0 l=1 rev1=1 crypto rx=0x7f3f0000e8c0 tx=0x7f3f0000ed80 comp rx=0 tx=0).stop 2024-09-16T15:50:35.564 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:35.566+0000 7f3f10368640 1 -- 172.21.15.123:0/2718684887 shutdown_connections 2024-09-16T15:50:35.565 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:35.566+0000 7f3f10368640 1 --2- 172.21.15.123:0/2718684887 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f3ef4073580 0x7f3ef4075a40 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:35.565 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:35.566+0000 7f3f10368640 1 --2- 172.21.15.123:0/2718684887 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3f08151400 0x7f3f08152b80 unknown :-1 s=CLOSED pgs=1598 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:35.565 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:35.566+0000 7f3f10368640 1 --2- 172.21.15.123:0/2718684887 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f3f080cde40 0x7f3f0814df10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:35.565 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:35.566+0000 7f3f10368640 1 --2- 172.21.15.123:0/2718684887 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3f080c5f20 0x7f3f081594c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:35.565 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:35.566+0000 7f3f10368640 1 -- 172.21.15.123:0/2718684887 >> 172.21.15.123:0/2718684887 conn(0x7f3f080cfa90 msgr2=0x7f3f081503c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:50:35.565 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:35.566+0000 7f3f10368640 1 -- 172.21.15.123:0/2718684887 shutdown_connections 2024-09-16T15:50:35.565 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:35.566+0000 7f3f10368640 1 -- 172.21.15.123:0/2718684887 wait complete. 2024-09-16T15:50:35.579 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2024-09-16T15:50:35.580 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T15:50:35.692 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:35.694+0000 7f78dd01a640 1 -- 172.21.15.123:0/2242280729 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f78d80b2b00 msgr2=0x7f78d80d22e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:50:35.692 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:35.694+0000 7f78dd01a640 1 --2- 172.21.15.123:0/2242280729 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f78d80b2b00 0x7f78d80d22e0 secure :-1 s=READY pgs=1599 cs=0 l=1 rev1=1 crypto rx=0x7f78cc009900 tx=0x7f78cc01c790 comp rx=0 tx=0).stop 2024-09-16T15:50:35.693 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:35.694+0000 7f78dd01a640 1 -- 172.21.15.123:0/2242280729 shutdown_connections 2024-09-16T15:50:35.693 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:35.694+0000 7f78dd01a640 1 --2- 172.21.15.123:0/2242280729 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f78d80b2b00 0x7f78d80d22e0 unknown :-1 s=CLOSED pgs=1599 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:35.693 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:35.694+0000 7f78dd01a640 1 --2- 172.21.15.123:0/2242280729 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f78d8161620 0x7f78d80d1da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:35.693 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:35.694+0000 7f78dd01a640 1 -- 172.21.15.123:0/2242280729 >> 172.21.15.123:0/2242280729 conn(0x7f78d80d3ea0 msgr2=0x7f78d80d42b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:50:35.693 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:35.694+0000 7f78dd01a640 1 -- 172.21.15.123:0/2242280729 shutdown_connections 2024-09-16T15:50:35.693 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:35.694+0000 7f78dd01a640 1 -- 172.21.15.123:0/2242280729 wait complete. 2024-09-16T15:50:35.693 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:35.694+0000 7f78dd01a640 1 Processor -- start 2024-09-16T15:50:35.693 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:35.694+0000 7f78dd01a640 1 -- start start 2024-09-16T15:50:35.695 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:35.694+0000 7f78dd01a640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f78d80b2b00 0x7f78d8152ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:50:35.695 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:35.694+0000 7f78dd01a640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f78d80d2820 0x7f78d8158ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:50:35.695 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:35.694+0000 7f78dd01a640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f78d8161620 0x7f78d815a640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:50:35.695 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:35.694+0000 7f78dd01a640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f78d816af70 con 0x7f78d8161620 2024-09-16T15:50:35.696 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:35.694+0000 7f78dd01a640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f78d80b1cb0 con 0x7f78d80b2b00 2024-09-16T15:50:35.696 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:35.694+0000 7f78dd01a640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f78d816b0f0 con 0x7f78d80d2820 2024-09-16T15:50:35.696 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:35.694+0000 7f78d759e640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f78d8161620 0x7f78d815a640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:50:35.696 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:35.694+0000 7f78d759e640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f78d8161620 0x7f78d815a640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:49726/0 (socket says 172.21.15.123:49726) 2024-09-16T15:50:35.696 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:35.694+0000 7f78d759e640 1 -- 172.21.15.123:0/234686626 learned_addr learned my addr 172.21.15.123:0/234686626 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:50:35.696 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:35.694+0000 7f78d659c640 1 --2- 172.21.15.123:0/234686626 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f78d80d2820 0x7f78d8158ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:50:35.696 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:35.694+0000 7f78d6d9d640 1 --2- 172.21.15.123:0/234686626 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f78d80b2b00 0x7f78d8152ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:50:35.696 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:35.694+0000 7f78d759e640 1 -- 172.21.15.123:0/234686626 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f78d80d2820 msgr2=0x7f78d8158ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:50:35.696 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:35.694+0000 7f78d759e640 1 --2- 172.21.15.123:0/234686626 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f78d80d2820 0x7f78d8158ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:35.696 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:35.694+0000 7f78d759e640 1 -- 172.21.15.123:0/234686626 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f78d80b2b00 msgr2=0x7f78d8152ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:50:35.696 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:35.694+0000 7f78d759e640 1 --2- 172.21.15.123:0/234686626 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f78d80b2b00 0x7f78d8152ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:35.696 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:35.694+0000 7f78d759e640 1 -- 172.21.15.123:0/234686626 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f78d8153950 con 0x7f78d8161620 2024-09-16T15:50:35.696 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:35.694+0000 7f78d759e640 1 --2- 172.21.15.123:0/234686626 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f78d8161620 0x7f78d815a640 secure :-1 s=READY pgs=1600 cs=0 l=1 rev1=1 crypto rx=0x7f78d000c780 tx=0x7f78d000cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:50:35.696 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:35.694+0000 7f78bf7fe640 1 -- 172.21.15.123:0/234686626 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78d0017070 con 0x7f78d8161620 2024-09-16T15:50:35.697 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:35.694+0000 7f78bf7fe640 1 -- 172.21.15.123:0/234686626 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f78d0005070 con 0x7f78d8161620 2024-09-16T15:50:35.697 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:35.694+0000 7f78dd01a640 1 -- 172.21.15.123:0/234686626 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f78d8153c30 con 0x7f78d8161620 2024-09-16T15:50:35.697 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:35.694+0000 7f78bf7fe640 1 -- 172.21.15.123:0/234686626 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78d0005500 con 0x7f78d8161620 2024-09-16T15:50:35.697 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:35.694+0000 7f78dd01a640 1 -- 172.21.15.123:0/234686626 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f78d81541f0 con 0x7f78d8161620 2024-09-16T15:50:35.698 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:35.698+0000 7f78dd01a640 1 -- 172.21.15.123:0/234686626 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f78d80cea00 con 0x7f78d8161620 2024-09-16T15:50:35.698 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:35.698+0000 7f78bf7fe640 1 -- 172.21.15.123:0/234686626 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 22) ==== 97692+0+0 (secure 0 0 0) 0x7f78d001a070 con 0x7f78d8161620 2024-09-16T15:50:35.698 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:35.698+0000 7f78bf7fe640 1 --2- 172.21.15.123:0/234686626 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f78c40735b0 0x7f78c4075a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:50:35.698 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:35.698+0000 7f78bf7fe640 1 -- 172.21.15.123:0/234686626 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(146..146 src has 1..146) ==== 5737+0+0 (secure 0 0 0) 0x7f78d00941c0 con 0x7f78d8161620 2024-09-16T15:50:35.698 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:35.698+0000 7f78d6d9d640 1 --2- 172.21.15.123:0/234686626 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f78c40735b0 0x7f78c4075a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:50:35.698 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:35.698+0000 7f78d6d9d640 1 --2- 172.21.15.123:0/234686626 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f78c40735b0 0x7f78c4075a70 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f78c8005870 tx=0x7f78c800c440 comp rx=0 tx=0).ready entity=mgr.4150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:50:35.701 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:35.702+0000 7f78bf7fe640 1 -- 172.21.15.123:0/234686626 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f78d005e8c0 con 0x7f78d8161620 2024-09-16T15:50:35.857 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:35.858+0000 7f78dd01a640 1 -- 172.21.15.123:0/234686626 --> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f78d806da30 con 0x7f78c40735b0 2024-09-16T15:50:35.864 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:35.866+0000 7f78bf7fe640 1 -- 172.21.15.123:0/234686626 <== mgr.4150 v2:172.21.15.123:6836/2464759676 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+247115 (secure 0 0 0) 0x7f78d806da30 con 0x7f78c40735b0 2024-09-16T15:50:35.866 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-09-16T15:50:35.868 INFO:teuthology.orchestra.run.smithi123.stderr:dumped all 2024-09-16T15:50:35.869 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:35.870+0000 7f78dd01a640 1 -- 172.21.15.123:0/234686626 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f78c40735b0 msgr2=0x7f78c4075a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:50:35.869 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:35.870+0000 7f78dd01a640 1 --2- 172.21.15.123:0/234686626 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f78c40735b0 0x7f78c4075a70 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f78c8005870 tx=0x7f78c800c440 comp rx=0 tx=0).stop 2024-09-16T15:50:35.869 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:35.870+0000 7f78dd01a640 1 -- 172.21.15.123:0/234686626 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f78d8161620 msgr2=0x7f78d815a640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:50:35.869 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:35.870+0000 7f78dd01a640 1 --2- 172.21.15.123:0/234686626 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f78d8161620 0x7f78d815a640 secure :-1 s=READY pgs=1600 cs=0 l=1 rev1=1 crypto rx=0x7f78d000c780 tx=0x7f78d000cc40 comp rx=0 tx=0).stop 2024-09-16T15:50:35.869 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:35.870+0000 7f78dd01a640 1 -- 172.21.15.123:0/234686626 shutdown_connections 2024-09-16T15:50:35.869 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:35.870+0000 7f78dd01a640 1 --2- 172.21.15.123:0/234686626 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f78c40735b0 0x7f78c4075a70 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:35.869 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:35.870+0000 7f78dd01a640 1 --2- 172.21.15.123:0/234686626 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f78d8161620 0x7f78d815a640 unknown :-1 s=CLOSED pgs=1600 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:35.869 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:35.870+0000 7f78dd01a640 1 --2- 172.21.15.123:0/234686626 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f78d80d2820 0x7f78d8158ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:35.869 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:35.870+0000 7f78dd01a640 1 --2- 172.21.15.123:0/234686626 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f78d80b2b00 0x7f78d8152ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:35.869 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:35.870+0000 7f78dd01a640 1 -- 172.21.15.123:0/234686626 >> 172.21.15.123:0/234686626 conn(0x7f78d80d3ea0 msgr2=0x7f78d80b5920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:50:35.870 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:35.870+0000 7f78dd01a640 1 -- 172.21.15.123:0/234686626 shutdown_connections 2024-09-16T15:50:35.870 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:35.870+0000 7f78dd01a640 1 -- 172.21.15.123:0/234686626 wait complete. 2024-09-16T15:50:35.884 INFO:teuthology.orchestra.run.smithi123.stdout:{"pg_ready":true,"pg_map":{"version":167,"stamp":"2024-09-16T15:50:34.119725+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79,"ondisk_log_size":79,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":39,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":268904,"kb_used_data":15224,"kb_used_omap":12,"kb_used_meta":253491,"kb_avail":754705816,"statfs":{"total":773094113280,"available":772818755584,"internally_reserved":0,"allocated":15589376,"data_stored":7545564,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":259575345},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":29,"apply_latency_ms":29,"commit_latency_ns":29000000,"apply_latency_ns":29000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.743992"},"pg_stats":[{"pgid":"23.3f","version":"0'0","reported_seq":1,"reported_epoch":144,"state":"creating+peering","last_fresh":"2024-09-16T15:50:29.396826+0000","last_change":"2024-09-16T15:50:29.396826+0000","last_active":"2024-09-16T15:50:29.373642+0000","last_peered":"2024-09-16T15:50:29.373642+0000","last_clean":"2024-09-16T15:50:29.373642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:50:29.396826+0000","last_undegraded":"2024-09-16T15:50:29.396826+0000","last_fullsized":"2024-09-16T15:50:29.396826+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_clean_scrub_stamp":"2024-09-16T15:50:29.373642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.3e","version":"0'0","reported_seq":1,"reported_epoch":144,"state":"creating+peering","last_fresh":"2024-09-16T15:50:29.405611+0000","last_change":"2024-09-16T15:50:29.405611+0000","last_active":"2024-09-16T15:50:29.373642+0000","last_peered":"2024-09-16T15:50:29.373642+0000","last_clean":"2024-09-16T15:50:29.373642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:50:29.405611+0000","last_undegraded":"2024-09-16T15:50:29.405611+0000","last_fullsized":"2024-09-16T15:50:29.405611+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_clean_scrub_stamp":"2024-09-16T15:50:29.373642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.3d","version":"0'0","reported_seq":1,"reported_epoch":144,"state":"creating+peering","last_fresh":"2024-09-16T15:50:29.414816+0000","last_change":"2024-09-16T15:50:29.414816+0000","last_active":"2024-09-16T15:50:29.373642+0000","last_peered":"2024-09-16T15:50:29.373642+0000","last_clean":"2024-09-16T15:50:29.373642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:50:29.414816+0000","last_undegraded":"2024-09-16T15:50:29.414816+0000","last_fullsized":"2024-09-16T15:50:29.414816+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_clean_scrub_stamp":"2024-09-16T15:50:29.373642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.3c","version":"0'0","reported_seq":1,"reported_epoch":144,"state":"creating+peering","last_fresh":"2024-09-16T15:50:29.400948+0000","last_change":"2024-09-16T15:50:29.400948+0000","last_active":"2024-09-16T15:50:29.373642+0000","last_peered":"2024-09-16T15:50:29.373642+0000","last_clean":"2024-09-16T15:50:29.373642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:50:29.400948+0000","last_undegraded":"2024-09-16T15:50:29.400948+0000","last_fullsized":"2024-09-16T15:50:29.400948+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_clean_scrub_stamp":"2024-09-16T15:50:29.373642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.3b","version":"0'0","reported_seq":1,"reported_epoch":144,"state":"creating+peering","last_fresh":"2024-09-16T15:50:29.408612+0000","last_change":"2024-09-16T15:50:29.408612+0000","last_active":"2024-09-16T15:50:29.373642+0000","last_peered":"2024-09-16T15:50:29.373642+0000","last_clean":"2024-09-16T15:50:29.373642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:50:29.408612+0000","last_undegraded":"2024-09-16T15:50:29.408612+0000","last_fullsized":"2024-09-16T15:50:29.408612+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_clean_scrub_stamp":"2024-09-16T15:50:29.373642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.3a","version":"0'0","reported_seq":1,"reported_epoch":144,"state":"creating+peering","last_fresh":"2024-09-16T15:50:29.399825+0000","last_change":"2024-09-16T15:50:29.399825+0000","last_active":"2024-09-16T15:50:29.373642+0000","last_peered":"2024-09-16T15:50:29.373642+0000","last_clean":"2024-09-16T15:50:29.373642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:50:29.399825+0000","last_undegraded":"2024-09-16T15:50:29.399825+0000","last_fullsized":"2024-09-16T15:50:29.399825+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_clean_scrub_stamp":"2024-09-16T15:50:29.373642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.39","version":"0'0","reported_seq":1,"reported_epoch":144,"state":"creating+peering","last_fresh":"2024-09-16T15:50:29.397580+0000","last_change":"2024-09-16T15:50:29.397580+0000","last_active":"2024-09-16T15:50:29.373642+0000","last_peered":"2024-09-16T15:50:29.373642+0000","last_clean":"2024-09-16T15:50:29.373642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:50:29.397580+0000","last_undegraded":"2024-09-16T15:50:29.397580+0000","last_fullsized":"2024-09-16T15:50:29.397580+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_clean_scrub_stamp":"2024-09-16T15:50:29.373642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"23.38","version":"0'0","reported_seq":1,"reported_epoch":144,"state":"creating+peering","last_fresh":"2024-09-16T15:50:29.409708+0000","last_change":"2024-09-16T15:50:29.409708+0000","last_active":"2024-09-16T15:50:29.373642+0000","last_peered":"2024-09-16T15:50:29.373642+0000","last_clean":"2024-09-16T15:50:29.373642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:50:29.409708+0000","last_undegraded":"2024-09-16T15:50:29.409708+0000","last_fullsized":"2024-09-16T15:50:29.409708+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_clean_scrub_stamp":"2024-09-16T15:50:29.373642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.37","version":"0'0","reported_seq":1,"reported_epoch":144,"state":"creating+peering","last_fresh":"2024-09-16T15:50:29.406549+0000","last_change":"2024-09-16T15:50:29.406549+0000","last_active":"2024-09-16T15:50:29.373642+0000","last_peered":"2024-09-16T15:50:29.373642+0000","last_clean":"2024-09-16T15:50:29.373642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:50:29.406549+0000","last_undegraded":"2024-09-16T15:50:29.406549+0000","last_fullsized":"2024-09-16T15:50:29.406549+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_clean_scrub_stamp":"2024-09-16T15:50:29.373642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.36","version":"0'0","reported_seq":1,"reported_epoch":144,"state":"creating+peering","last_fresh":"2024-09-16T15:50:29.404538+0000","last_change":"2024-09-16T15:50:29.404538+0000","last_active":"2024-09-16T15:50:29.373642+0000","last_peered":"2024-09-16T15:50:29.373642+0000","last_clean":"2024-09-16T15:50:29.373642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:50:29.404538+0000","last_undegraded":"2024-09-16T15:50:29.404538+0000","last_fullsized":"2024-09-16T15:50:29.404538+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_clean_scrub_stamp":"2024-09-16T15:50:29.373642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.35","version":"0'0","reported_seq":1,"reported_epoch":144,"state":"creating+peering","last_fresh":"2024-09-16T15:50:29.418503+0000","last_change":"2024-09-16T15:50:29.418503+0000","last_active":"2024-09-16T15:50:29.373642+0000","last_peered":"2024-09-16T15:50:29.373642+0000","last_clean":"2024-09-16T15:50:29.373642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:50:29.418503+0000","last_undegraded":"2024-09-16T15:50:29.418503+0000","last_fullsized":"2024-09-16T15:50:29.418503+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_clean_scrub_stamp":"2024-09-16T15:50:29.373642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.34","version":"0'0","reported_seq":1,"reported_epoch":144,"state":"creating+peering","last_fresh":"2024-09-16T15:50:29.398426+0000","last_change":"2024-09-16T15:50:29.398426+0000","last_active":"2024-09-16T15:50:29.373642+0000","last_peered":"2024-09-16T15:50:29.373642+0000","last_clean":"2024-09-16T15:50:29.373642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:50:29.398426+0000","last_undegraded":"2024-09-16T15:50:29.398426+0000","last_fullsized":"2024-09-16T15:50:29.398426+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_clean_scrub_stamp":"2024-09-16T15:50:29.373642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.33","version":"0'0","reported_seq":1,"reported_epoch":144,"state":"creating+peering","last_fresh":"2024-09-16T15:50:29.415630+0000","last_change":"2024-09-16T15:50:29.415630+0000","last_active":"2024-09-16T15:50:29.373642+0000","last_peered":"2024-09-16T15:50:29.373642+0000","last_clean":"2024-09-16T15:50:29.373642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:50:29.415630+0000","last_undegraded":"2024-09-16T15:50:29.415630+0000","last_fullsized":"2024-09-16T15:50:29.415630+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_clean_scrub_stamp":"2024-09-16T15:50:29.373642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.32","version":"0'0","reported_seq":1,"reported_epoch":144,"state":"creating+peering","last_fresh":"2024-09-16T15:50:29.411622+0000","last_change":"2024-09-16T15:50:29.411622+0000","last_active":"2024-09-16T15:50:29.373642+0000","last_peered":"2024-09-16T15:50:29.373642+0000","last_clean":"2024-09-16T15:50:29.373642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:50:29.411622+0000","last_undegraded":"2024-09-16T15:50:29.411622+0000","last_fullsized":"2024-09-16T15:50:29.411622+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_clean_scrub_stamp":"2024-09-16T15:50:29.373642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.31","version":"0'0","reported_seq":1,"reported_epoch":144,"state":"creating+peering","last_fresh":"2024-09-16T15:50:29.396532+0000","last_change":"2024-09-16T15:50:29.396532+0000","last_active":"2024-09-16T15:50:29.373642+0000","last_peered":"2024-09-16T15:50:29.373642+0000","last_clean":"2024-09-16T15:50:29.373642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:50:29.396532+0000","last_undegraded":"2024-09-16T15:50:29.396532+0000","last_fullsized":"2024-09-16T15:50:29.396532+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_clean_scrub_stamp":"2024-09-16T15:50:29.373642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.30","version":"0'0","reported_seq":1,"reported_epoch":144,"state":"creating+peering","last_fresh":"2024-09-16T15:50:29.398051+0000","last_change":"2024-09-16T15:50:29.398051+0000","last_active":"2024-09-16T15:50:29.373642+0000","last_peered":"2024-09-16T15:50:29.373642+0000","last_clean":"2024-09-16T15:50:29.373642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:50:29.398051+0000","last_undegraded":"2024-09-16T15:50:29.398051+0000","last_fullsized":"2024-09-16T15:50:29.398051+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_clean_scrub_stamp":"2024-09-16T15:50:29.373642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"23.2f","version":"0'0","reported_seq":1,"reported_epoch":144,"state":"creating+peering","last_fresh":"2024-09-16T15:50:29.399338+0000","last_change":"2024-09-16T15:50:29.399338+0000","last_active":"2024-09-16T15:50:29.373642+0000","last_peered":"2024-09-16T15:50:29.373642+0000","last_clean":"2024-09-16T15:50:29.373642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:50:29.399338+0000","last_undegraded":"2024-09-16T15:50:29.399338+0000","last_fullsized":"2024-09-16T15:50:29.399338+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_clean_scrub_stamp":"2024-09-16T15:50:29.373642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.2e","version":"0'0","reported_seq":1,"reported_epoch":144,"state":"creating+peering","last_fresh":"2024-09-16T15:50:29.403155+0000","last_change":"2024-09-16T15:50:29.403155+0000","last_active":"2024-09-16T15:50:29.373642+0000","last_peered":"2024-09-16T15:50:29.373642+0000","last_clean":"2024-09-16T15:50:29.373642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:50:29.403155+0000","last_undegraded":"2024-09-16T15:50:29.403155+0000","last_fullsized":"2024-09-16T15:50:29.403155+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_clean_scrub_stamp":"2024-09-16T15:50:29.373642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"23.2d","version":"0'0","reported_seq":1,"reported_epoch":144,"state":"creating+peering","last_fresh":"2024-09-16T15:50:29.399089+0000","last_change":"2024-09-16T15:50:29.399089+0000","last_active":"2024-09-16T15:50:29.373642+0000","last_peered":"2024-09-16T15:50:29.373642+0000","last_clean":"2024-09-16T15:50:29.373642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:50:29.399089+0000","last_undegraded":"2024-09-16T15:50:29.399089+0000","last_fullsized":"2024-09-16T15:50:29.399089+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_clean_scrub_stamp":"2024-09-16T15:50:29.373642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.2c","version":"0'0","reported_seq":1,"reported_epoch":144,"state":"creating+peering","last_fresh":"2024-09-16T15:50:29.400996+0000","last_change":"2024-09-16T15:50:29.400996+0000","last_active":"2024-09-16T15:50:29.373642+0000","last_peered":"2024-09-16T15:50:29.373642+0000","last_clean":"2024-09-16T15:50:29.373642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:50:29.400996+0000","last_undegraded":"2024-09-16T15:50:29.400996+0000","last_fullsized":"2024-09-16T15:50:29.400996+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_clean_scrub_stamp":"2024-09-16T15:50:29.373642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.2b","version":"0'0","reported_seq":1,"reported_epoch":144,"state":"creating+peering","last_fresh":"2024-09-16T15:50:29.395481+0000","last_change":"2024-09-16T15:50:29.395481+0000","last_active":"2024-09-16T15:50:29.373642+0000","last_peered":"2024-09-16T15:50:29.373642+0000","last_clean":"2024-09-16T15:50:29.373642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:50:29.395481+0000","last_undegraded":"2024-09-16T15:50:29.395481+0000","last_fullsized":"2024-09-16T15:50:29.395481+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_clean_scrub_stamp":"2024-09-16T15:50:29.373642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.2a","version":"0'0","reported_seq":1,"reported_epoch":144,"state":"creating+peering","last_fresh":"2024-09-16T15:50:29.396706+0000","last_change":"2024-09-16T15:50:29.396706+0000","last_active":"2024-09-16T15:50:29.373642+0000","last_peered":"2024-09-16T15:50:29.373642+0000","last_clean":"2024-09-16T15:50:29.373642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:50:29.396706+0000","last_undegraded":"2024-09-16T15:50:29.396706+0000","last_fullsized":"2024-09-16T15:50:29.396706+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_clean_scrub_stamp":"2024-09-16T15:50:29.373642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.29","version":"0'0","reported_seq":1,"reported_epoch":144,"state":"creating+peering","last_fresh":"2024-09-16T15:50:29.397404+0000","last_change":"2024-09-16T15:50:29.397404+0000","last_active":"2024-09-16T15:50:29.373642+0000","last_peered":"2024-09-16T15:50:29.373642+0000","last_clean":"2024-09-16T15:50:29.373642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:50:29.397404+0000","last_undegraded":"2024-09-16T15:50:29.397404+0000","last_fullsized":"2024-09-16T15:50:29.397404+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_clean_scrub_stamp":"2024-09-16T15:50:29.373642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.28","version":"0'0","reported_seq":1,"reported_epoch":144,"state":"creating+peering","last_fresh":"2024-09-16T15:50:29.396706+0000","last_change":"2024-09-16T15:50:29.396706+0000","last_active":"2024-09-16T15:50:29.373642+0000","last_peered":"2024-09-16T15:50:29.373642+0000","last_clean":"2024-09-16T15:50:29.373642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:50:29.396706+0000","last_undegraded":"2024-09-16T15:50:29.396706+0000","last_fullsized":"2024-09-16T15:50:29.396706+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_clean_scrub_stamp":"2024-09-16T15:50:29.373642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.27","version":"0'0","reported_seq":1,"reported_epoch":144,"state":"creating+peering","last_fresh":"2024-09-16T15:50:29.401802+0000","last_change":"2024-09-16T15:50:29.401802+0000","last_active":"2024-09-16T15:50:29.373642+0000","last_peered":"2024-09-16T15:50:29.373642+0000","last_clean":"2024-09-16T15:50:29.373642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:50:29.401802+0000","last_undegraded":"2024-09-16T15:50:29.401802+0000","last_fullsized":"2024-09-16T15:50:29.401802+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_clean_scrub_stamp":"2024-09-16T15:50:29.373642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.26","version":"0'0","reported_seq":1,"reported_epoch":144,"state":"creating+peering","last_fresh":"2024-09-16T15:50:29.397726+0000","last_change":"2024-09-16T15:50:29.397726+0000","last_active":"2024-09-16T15:50:29.373642+0000","last_peered":"2024-09-16T15:50:29.373642+0000","last_clean":"2024-09-16T15:50:29.373642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:50:29.397726+0000","last_undegraded":"2024-09-16T15:50:29.397726+0000","last_fullsized":"2024-09-16T15:50:29.397726+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_clean_scrub_stamp":"2024-09-16T15:50:29.373642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"23.25","version":"0'0","reported_seq":1,"reported_epoch":144,"state":"creating+peering","last_fresh":"2024-09-16T15:50:29.413671+0000","last_change":"2024-09-16T15:50:29.413671+0000","last_active":"2024-09-16T15:50:29.373642+0000","last_peered":"2024-09-16T15:50:29.373642+0000","last_clean":"2024-09-16T15:50:29.373642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:50:29.413671+0000","last_undegraded":"2024-09-16T15:50:29.413671+0000","last_fullsized":"2024-09-16T15:50:29.413671+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_clean_scrub_stamp":"2024-09-16T15:50:29.373642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.24","version":"0'0","reported_seq":1,"reported_epoch":144,"state":"creating+peering","last_fresh":"2024-09-16T15:50:29.400095+0000","last_change":"2024-09-16T15:50:29.400095+0000","last_active":"2024-09-16T15:50:29.373642+0000","last_peered":"2024-09-16T15:50:29.373642+0000","last_clean":"2024-09-16T15:50:29.373642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:50:29.400095+0000","last_undegraded":"2024-09-16T15:50:29.400095+0000","last_fullsized":"2024-09-16T15:50:29.400095+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_clean_scrub_stamp":"2024-09-16T15:50:29.373642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.23","version":"0'0","reported_seq":1,"reported_epoch":144,"state":"creating+peering","last_fresh":"2024-09-16T15:50:29.403015+0000","last_change":"2024-09-16T15:50:29.403015+0000","last_active":"2024-09-16T15:50:29.373642+0000","last_peered":"2024-09-16T15:50:29.373642+0000","last_clean":"2024-09-16T15:50:29.373642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:50:29.403015+0000","last_undegraded":"2024-09-16T15:50:29.403015+0000","last_fullsized":"2024-09-16T15:50:29.403015+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_clean_scrub_stamp":"2024-09-16T15:50:29.373642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.22","version":"0'0","reported_seq":1,"reported_epoch":144,"state":"creating+peering","last_fresh":"2024-09-16T15:50:29.390228+0000","last_change":"2024-09-16T15:50:29.390228+0000","last_active":"2024-09-16T15:50:29.373642+0000","last_peered":"2024-09-16T15:50:29.373642+0000","last_clean":"2024-09-16T15:50:29.373642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:50:29.390228+0000","last_undegraded":"2024-09-16T15:50:29.390228+0000","last_fullsized":"2024-09-16T15:50:29.390228+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_clean_scrub_stamp":"2024-09-16T15:50:29.373642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"23.21","version":"0'0","reported_seq":1,"reported_epoch":144,"state":"creating+peering","last_fresh":"2024-09-16T15:50:29.392793+0000","last_change":"2024-09-16T15:50:29.392793+0000","last_active":"2024-09-16T15:50:29.373642+0000","last_peered":"2024-09-16T15:50:29.373642+0000","last_clean":"2024-09-16T15:50:29.373642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:50:29.392793+0000","last_undegraded":"2024-09-16T15:50:29.392793+0000","last_fullsized":"2024-09-16T15:50:29.392793+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_clean_scrub_stamp":"2024-09-16T15:50:29.373642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"23.20","version":"0'0","reported_seq":1,"reported_epoch":144,"state":"creating+peering","last_fresh":"2024-09-16T15:50:29.400381+0000","last_change":"2024-09-16T15:50:29.400381+0000","last_active":"2024-09-16T15:50:29.373642+0000","last_peered":"2024-09-16T15:50:29.373642+0000","last_clean":"2024-09-16T15:50:29.373642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:50:29.400381+0000","last_undegraded":"2024-09-16T15:50:29.400381+0000","last_fullsized":"2024-09-16T15:50:29.400381+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_clean_scrub_stamp":"2024-09-16T15:50:29.373642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"23.1e","version":"0'0","reported_seq":1,"reported_epoch":144,"state":"creating+peering","last_fresh":"2024-09-16T15:50:29.396097+0000","last_change":"2024-09-16T15:50:29.396097+0000","last_active":"2024-09-16T15:50:29.373642+0000","last_peered":"2024-09-16T15:50:29.373642+0000","last_clean":"2024-09-16T15:50:29.373642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:50:29.396097+0000","last_undegraded":"2024-09-16T15:50:29.396097+0000","last_fullsized":"2024-09-16T15:50:29.396097+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_clean_scrub_stamp":"2024-09-16T15:50:29.373642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"22.1f","version":"0'0","reported_seq":10,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-16T15:50:29.430449+0000","last_change":"2024-09-16T15:50:29.428405+0000","last_active":"2024-09-16T15:50:29.430449+0000","last_peered":"2024-09-16T15:50:29.430449+0000","last_clean":"2024-09-16T15:50:29.430449+0000","last_became_active":"2024-09-16T15:50:29.423819+0000","last_became_peered":"2024-09-16T15:50:29.423819+0000","last_unstale":"2024-09-16T15:50:29.430449+0000","last_undegraded":"2024-09-16T15:50:29.430449+0000","last_fullsized":"2024-09-16T15:50:29.430449+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:28.359944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:28.359944+0000","last_clean_scrub_stamp":"2024-09-16T15:50:28.359944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T22:06:16.541534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.1f","version":"0'0","reported_seq":1,"reported_epoch":144,"state":"creating+peering","last_fresh":"2024-09-16T15:50:29.395019+0000","last_change":"2024-09-16T15:50:29.395019+0000","last_active":"2024-09-16T15:50:29.373642+0000","last_peered":"2024-09-16T15:50:29.373642+0000","last_clean":"2024-09-16T15:50:29.373642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:50:29.395019+0000","last_undegraded":"2024-09-16T15:50:29.395019+0000","last_fullsized":"2024-09-16T15:50:29.395019+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_clean_scrub_stamp":"2024-09-16T15:50:29.373642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.1e","version":"0'0","reported_seq":10,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-16T15:50:29.433442+0000","last_change":"2024-09-16T15:50:29.431326+0000","last_active":"2024-09-16T15:50:29.433442+0000","last_peered":"2024-09-16T15:50:29.433442+0000","last_clean":"2024-09-16T15:50:29.433442+0000","last_became_active":"2024-09-16T15:50:29.430610+0000","last_became_peered":"2024-09-16T15:50:29.430610+0000","last_unstale":"2024-09-16T15:50:29.433442+0000","last_undegraded":"2024-09-16T15:50:29.433442+0000","last_fullsized":"2024-09-16T15:50:29.433442+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:28.359944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:28.359944+0000","last_clean_scrub_stamp":"2024-09-16T15:50:28.359944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T00:28:56.379780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"23.1c","version":"0'0","reported_seq":1,"reported_epoch":144,"state":"creating+peering","last_fresh":"2024-09-16T15:50:29.395809+0000","last_change":"2024-09-16T15:50:29.395809+0000","last_active":"2024-09-16T15:50:29.373642+0000","last_peered":"2024-09-16T15:50:29.373642+0000","last_clean":"2024-09-16T15:50:29.373642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:50:29.395809+0000","last_undegraded":"2024-09-16T15:50:29.395809+0000","last_fullsized":"2024-09-16T15:50:29.395809+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_clean_scrub_stamp":"2024-09-16T15:50:29.373642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.1d","version":"0'0","reported_seq":10,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-16T15:50:29.431196+0000","last_change":"2024-09-16T15:50:29.425771+0000","last_active":"2024-09-16T15:50:29.431196+0000","last_peered":"2024-09-16T15:50:29.431196+0000","last_clean":"2024-09-16T15:50:29.431196+0000","last_became_active":"2024-09-16T15:50:29.425507+0000","last_became_peered":"2024-09-16T15:50:29.425507+0000","last_unstale":"2024-09-16T15:50:29.431196+0000","last_undegraded":"2024-09-16T15:50:29.431196+0000","last_fullsized":"2024-09-16T15:50:29.431196+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:28.359944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:28.359944+0000","last_clean_scrub_stamp":"2024-09-16T15:50:28.359944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:16:32.881957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.1d","version":"0'0","reported_seq":1,"reported_epoch":144,"state":"creating+peering","last_fresh":"2024-09-16T15:50:29.412090+0000","last_change":"2024-09-16T15:50:29.412090+0000","last_active":"2024-09-16T15:50:29.373642+0000","last_peered":"2024-09-16T15:50:29.373642+0000","last_clean":"2024-09-16T15:50:29.373642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:50:29.412090+0000","last_undegraded":"2024-09-16T15:50:29.412090+0000","last_fullsized":"2024-09-16T15:50:29.412090+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_clean_scrub_stamp":"2024-09-16T15:50:29.373642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.1c","version":"0'0","reported_seq":10,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-16T15:50:29.411810+0000","last_change":"2024-09-16T15:50:29.408868+0000","last_active":"2024-09-16T15:50:29.411810+0000","last_peered":"2024-09-16T15:50:29.411810+0000","last_clean":"2024-09-16T15:50:29.411810+0000","last_became_active":"2024-09-16T15:50:29.408550+0000","last_became_peered":"2024-09-16T15:50:29.408550+0000","last_unstale":"2024-09-16T15:50:29.411810+0000","last_undegraded":"2024-09-16T15:50:29.411810+0000","last_fullsized":"2024-09-16T15:50:29.411810+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:28.359944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:28.359944+0000","last_clean_scrub_stamp":"2024-09-16T15:50:28.359944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:57:02.247484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.1a","version":"0'0","reported_seq":1,"reported_epoch":144,"state":"creating+peering","last_fresh":"2024-09-16T15:50:29.396181+0000","last_change":"2024-09-16T15:50:29.396181+0000","last_active":"2024-09-16T15:50:29.373642+0000","last_peered":"2024-09-16T15:50:29.373642+0000","last_clean":"2024-09-16T15:50:29.373642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:50:29.396181+0000","last_undegraded":"2024-09-16T15:50:29.396181+0000","last_fullsized":"2024-09-16T15:50:29.396181+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_clean_scrub_stamp":"2024-09-16T15:50:29.373642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"22.1b","version":"0'0","reported_seq":10,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-16T15:50:29.433993+0000","last_change":"2024-09-16T15:50:29.429045+0000","last_active":"2024-09-16T15:50:29.433993+0000","last_peered":"2024-09-16T15:50:29.433993+0000","last_clean":"2024-09-16T15:50:29.433993+0000","last_became_active":"2024-09-16T15:50:29.428317+0000","last_became_peered":"2024-09-16T15:50:29.428317+0000","last_unstale":"2024-09-16T15:50:29.433993+0000","last_undegraded":"2024-09-16T15:50:29.433993+0000","last_fullsized":"2024-09-16T15:50:29.433993+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:28.359944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:28.359944+0000","last_clean_scrub_stamp":"2024-09-16T15:50:28.359944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:54:05.773197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.1b","version":"0'0","reported_seq":1,"reported_epoch":144,"state":"creating+peering","last_fresh":"2024-09-16T15:50:29.396991+0000","last_change":"2024-09-16T15:50:29.396991+0000","last_active":"2024-09-16T15:50:29.373642+0000","last_peered":"2024-09-16T15:50:29.373642+0000","last_clean":"2024-09-16T15:50:29.373642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:50:29.396991+0000","last_undegraded":"2024-09-16T15:50:29.396991+0000","last_fullsized":"2024-09-16T15:50:29.396991+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_clean_scrub_stamp":"2024-09-16T15:50:29.373642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.1a","version":"0'0","reported_seq":10,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-16T15:50:29.411180+0000","last_change":"2024-09-16T15:50:29.408149+0000","last_active":"2024-09-16T15:50:29.411180+0000","last_peered":"2024-09-16T15:50:29.411180+0000","last_clean":"2024-09-16T15:50:29.411180+0000","last_became_active":"2024-09-16T15:50:29.407778+0000","last_became_peered":"2024-09-16T15:50:29.407778+0000","last_unstale":"2024-09-16T15:50:29.411180+0000","last_undegraded":"2024-09-16T15:50:29.411180+0000","last_fullsized":"2024-09-16T15:50:29.411180+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:28.359944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:28.359944+0000","last_clean_scrub_stamp":"2024-09-16T15:50:28.359944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:02:10.632687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.18","version":"0'0","reported_seq":1,"reported_epoch":144,"state":"creating+peering","last_fresh":"2024-09-16T15:50:29.397299+0000","last_change":"2024-09-16T15:50:29.397299+0000","last_active":"2024-09-16T15:50:29.373642+0000","last_peered":"2024-09-16T15:50:29.373642+0000","last_clean":"2024-09-16T15:50:29.373642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:50:29.397299+0000","last_undegraded":"2024-09-16T15:50:29.397299+0000","last_fullsized":"2024-09-16T15:50:29.397299+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_clean_scrub_stamp":"2024-09-16T15:50:29.373642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.19","version":"0'0","reported_seq":10,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-16T15:50:29.433714+0000","last_change":"2024-09-16T15:50:29.427582+0000","last_active":"2024-09-16T15:50:29.433714+0000","last_peered":"2024-09-16T15:50:29.433714+0000","last_clean":"2024-09-16T15:50:29.433714+0000","last_became_active":"2024-09-16T15:50:29.426124+0000","last_became_peered":"2024-09-16T15:50:29.426124+0000","last_unstale":"2024-09-16T15:50:29.433714+0000","last_undegraded":"2024-09-16T15:50:29.433714+0000","last_fullsized":"2024-09-16T15:50:29.433714+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:28.359944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:28.359944+0000","last_clean_scrub_stamp":"2024-09-16T15:50:28.359944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:01:27.892843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.19","version":"0'0","reported_seq":1,"reported_epoch":144,"state":"creating+peering","last_fresh":"2024-09-16T15:50:29.396885+0000","last_change":"2024-09-16T15:50:29.396885+0000","last_active":"2024-09-16T15:50:29.373642+0000","last_peered":"2024-09-16T15:50:29.373642+0000","last_clean":"2024-09-16T15:50:29.373642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:50:29.396885+0000","last_undegraded":"2024-09-16T15:50:29.396885+0000","last_fullsized":"2024-09-16T15:50:29.396885+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_clean_scrub_stamp":"2024-09-16T15:50:29.373642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.18","version":"0'0","reported_seq":10,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-16T15:50:29.409060+0000","last_change":"2024-09-16T15:50:29.406218+0000","last_active":"2024-09-16T15:50:29.409060+0000","last_peered":"2024-09-16T15:50:29.409060+0000","last_clean":"2024-09-16T15:50:29.409060+0000","last_became_active":"2024-09-16T15:50:29.405872+0000","last_became_peered":"2024-09-16T15:50:29.405872+0000","last_unstale":"2024-09-16T15:50:29.409060+0000","last_undegraded":"2024-09-16T15:50:29.409060+0000","last_fullsized":"2024-09-16T15:50:29.409060+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:28.359944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:28.359944+0000","last_clean_scrub_stamp":"2024-09-16T15:50:28.359944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T22:00:23.514722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"23.17","version":"0'0","reported_seq":1,"reported_epoch":144,"state":"creating+peering","last_fresh":"2024-09-16T15:50:29.403320+0000","last_change":"2024-09-16T15:50:29.403320+0000","last_active":"2024-09-16T15:50:29.373642+0000","last_peered":"2024-09-16T15:50:29.373642+0000","last_clean":"2024-09-16T15:50:29.373642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:50:29.403320+0000","last_undegraded":"2024-09-16T15:50:29.403320+0000","last_fullsized":"2024-09-16T15:50:29.403320+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_clean_scrub_stamp":"2024-09-16T15:50:29.373642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.16","version":"0'0","reported_seq":10,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-16T15:50:29.419433+0000","last_change":"2024-09-16T15:50:29.418819+0000","last_active":"2024-09-16T15:50:29.419433+0000","last_peered":"2024-09-16T15:50:29.419433+0000","last_clean":"2024-09-16T15:50:29.419433+0000","last_became_active":"2024-09-16T15:50:29.418365+0000","last_became_peered":"2024-09-16T15:50:29.418365+0000","last_unstale":"2024-09-16T15:50:29.419433+0000","last_undegraded":"2024-09-16T15:50:29.419433+0000","last_fullsized":"2024-09-16T15:50:29.419433+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:28.359944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:28.359944+0000","last_clean_scrub_stamp":"2024-09-16T15:50:28.359944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T23:56:48.360283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.14","version":"0'0","reported_seq":1,"reported_epoch":144,"state":"creating+peering","last_fresh":"2024-09-16T15:50:29.401245+0000","last_change":"2024-09-16T15:50:29.401245+0000","last_active":"2024-09-16T15:50:29.373642+0000","last_peered":"2024-09-16T15:50:29.373642+0000","last_clean":"2024-09-16T15:50:29.373642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:50:29.401245+0000","last_undegraded":"2024-09-16T15:50:29.401245+0000","last_fullsized":"2024-09-16T15:50:29.401245+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_clean_scrub_stamp":"2024-09-16T15:50:29.373642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.15","version":"0'0","reported_seq":10,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-16T15:50:29.417183+0000","last_change":"2024-09-16T15:50:29.414494+0000","last_active":"2024-09-16T15:50:29.417183+0000","last_peered":"2024-09-16T15:50:29.417183+0000","last_clean":"2024-09-16T15:50:29.417183+0000","last_became_active":"2024-09-16T15:50:29.413903+0000","last_became_peered":"2024-09-16T15:50:29.413903+0000","last_unstale":"2024-09-16T15:50:29.417183+0000","last_undegraded":"2024-09-16T15:50:29.417183+0000","last_fullsized":"2024-09-16T15:50:29.417183+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:28.359944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:28.359944+0000","last_clean_scrub_stamp":"2024-09-16T15:50:28.359944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T00:54:21.344614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.15","version":"0'0","reported_seq":1,"reported_epoch":144,"state":"creating+peering","last_fresh":"2024-09-16T15:50:29.390531+0000","last_change":"2024-09-16T15:50:29.390531+0000","last_active":"2024-09-16T15:50:29.373642+0000","last_peered":"2024-09-16T15:50:29.373642+0000","last_clean":"2024-09-16T15:50:29.373642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:50:29.390531+0000","last_undegraded":"2024-09-16T15:50:29.390531+0000","last_fullsized":"2024-09-16T15:50:29.390531+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_clean_scrub_stamp":"2024-09-16T15:50:29.373642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.14","version":"0'0","reported_seq":10,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-16T15:50:29.420922+0000","last_change":"2024-09-16T15:50:29.406502+0000","last_active":"2024-09-16T15:50:29.420922+0000","last_peered":"2024-09-16T15:50:29.420922+0000","last_clean":"2024-09-16T15:50:29.420922+0000","last_became_active":"2024-09-16T15:50:29.404490+0000","last_became_peered":"2024-09-16T15:50:29.404490+0000","last_unstale":"2024-09-16T15:50:29.420922+0000","last_undegraded":"2024-09-16T15:50:29.420922+0000","last_fullsized":"2024-09-16T15:50:29.420922+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:28.359944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:28.359944+0000","last_clean_scrub_stamp":"2024-09-16T15:50:28.359944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T03:40:13.610653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"23.12","version":"0'0","reported_seq":1,"reported_epoch":144,"state":"creating+peering","last_fresh":"2024-09-16T15:50:29.401262+0000","last_change":"2024-09-16T15:50:29.401262+0000","last_active":"2024-09-16T15:50:29.373642+0000","last_peered":"2024-09-16T15:50:29.373642+0000","last_clean":"2024-09-16T15:50:29.373642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:50:29.401262+0000","last_undegraded":"2024-09-16T15:50:29.401262+0000","last_fullsized":"2024-09-16T15:50:29.401262+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_clean_scrub_stamp":"2024-09-16T15:50:29.373642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.13","version":"0'0","reported_seq":10,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-16T15:50:29.411283+0000","last_change":"2024-09-16T15:50:29.408013+0000","last_active":"2024-09-16T15:50:29.411283+0000","last_peered":"2024-09-16T15:50:29.411283+0000","last_clean":"2024-09-16T15:50:29.411283+0000","last_became_active":"2024-09-16T15:50:29.407703+0000","last_became_peered":"2024-09-16T15:50:29.407703+0000","last_unstale":"2024-09-16T15:50:29.411283+0000","last_undegraded":"2024-09-16T15:50:29.411283+0000","last_fullsized":"2024-09-16T15:50:29.411283+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:28.359944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:28.359944+0000","last_clean_scrub_stamp":"2024-09-16T15:50:28.359944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:05:33.631752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.13","version":"0'0","reported_seq":1,"reported_epoch":144,"state":"creating+peering","last_fresh":"2024-09-16T15:50:29.396084+0000","last_change":"2024-09-16T15:50:29.396084+0000","last_active":"2024-09-16T15:50:29.373642+0000","last_peered":"2024-09-16T15:50:29.373642+0000","last_clean":"2024-09-16T15:50:29.373642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:50:29.396084+0000","last_undegraded":"2024-09-16T15:50:29.396084+0000","last_fullsized":"2024-09-16T15:50:29.396084+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_clean_scrub_stamp":"2024-09-16T15:50:29.373642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"22.12","version":"0'0","reported_seq":10,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-16T15:50:29.411692+0000","last_change":"2024-09-16T15:50:29.410142+0000","last_active":"2024-09-16T15:50:29.411692+0000","last_peered":"2024-09-16T15:50:29.411692+0000","last_clean":"2024-09-16T15:50:29.411692+0000","last_became_active":"2024-09-16T15:50:29.409779+0000","last_became_peered":"2024-09-16T15:50:29.409779+0000","last_unstale":"2024-09-16T15:50:29.411692+0000","last_undegraded":"2024-09-16T15:50:29.411692+0000","last_fullsized":"2024-09-16T15:50:29.411692+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:28.359944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:28.359944+0000","last_clean_scrub_stamp":"2024-09-16T15:50:28.359944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:51:34.842420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"23.10","version":"0'0","reported_seq":1,"reported_epoch":144,"state":"creating+peering","last_fresh":"2024-09-16T15:50:29.396732+0000","last_change":"2024-09-16T15:50:29.396732+0000","last_active":"2024-09-16T15:50:29.373642+0000","last_peered":"2024-09-16T15:50:29.373642+0000","last_clean":"2024-09-16T15:50:29.373642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:50:29.396732+0000","last_undegraded":"2024-09-16T15:50:29.396732+0000","last_fullsized":"2024-09-16T15:50:29.396732+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_clean_scrub_stamp":"2024-09-16T15:50:29.373642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.11","version":"0'0","reported_seq":10,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-16T15:50:29.431189+0000","last_change":"2024-09-16T15:50:29.425647+0000","last_active":"2024-09-16T15:50:29.431189+0000","last_peered":"2024-09-16T15:50:29.431189+0000","last_clean":"2024-09-16T15:50:29.431189+0000","last_became_active":"2024-09-16T15:50:29.425266+0000","last_became_peered":"2024-09-16T15:50:29.425266+0000","last_unstale":"2024-09-16T15:50:29.431189+0000","last_undegraded":"2024-09-16T15:50:29.431189+0000","last_fullsized":"2024-09-16T15:50:29.431189+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:28.359944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:28.359944+0000","last_clean_scrub_stamp":"2024-09-16T15:50:28.359944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:18:55.430269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.11","version":"0'0","reported_seq":1,"reported_epoch":144,"state":"creating+peering","last_fresh":"2024-09-16T15:50:29.418052+0000","last_change":"2024-09-16T15:50:29.418052+0000","last_active":"2024-09-16T15:50:29.373642+0000","last_peered":"2024-09-16T15:50:29.373642+0000","last_clean":"2024-09-16T15:50:29.373642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:50:29.418052+0000","last_undegraded":"2024-09-16T15:50:29.418052+0000","last_fullsized":"2024-09-16T15:50:29.418052+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_clean_scrub_stamp":"2024-09-16T15:50:29.373642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.10","version":"0'0","reported_seq":10,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-16T15:50:29.421423+0000","last_change":"2024-09-16T15:50:29.408571+0000","last_active":"2024-09-16T15:50:29.421423+0000","last_peered":"2024-09-16T15:50:29.421423+0000","last_clean":"2024-09-16T15:50:29.421423+0000","last_became_active":"2024-09-16T15:50:29.407840+0000","last_became_peered":"2024-09-16T15:50:29.407840+0000","last_unstale":"2024-09-16T15:50:29.421423+0000","last_undegraded":"2024-09-16T15:50:29.421423+0000","last_fullsized":"2024-09-16T15:50:29.421423+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:28.359944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:28.359944+0000","last_clean_scrub_stamp":"2024-09-16T15:50:28.359944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:00:52.523701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"23.e","version":"0'0","reported_seq":1,"reported_epoch":144,"state":"creating+peering","last_fresh":"2024-09-16T15:50:29.396821+0000","last_change":"2024-09-16T15:50:29.396821+0000","last_active":"2024-09-16T15:50:29.373642+0000","last_peered":"2024-09-16T15:50:29.373642+0000","last_clean":"2024-09-16T15:50:29.373642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:50:29.396821+0000","last_undegraded":"2024-09-16T15:50:29.396821+0000","last_fullsized":"2024-09-16T15:50:29.396821+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_clean_scrub_stamp":"2024-09-16T15:50:29.373642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.f","version":"0'0","reported_seq":10,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-16T15:50:29.435965+0000","last_change":"2024-09-16T15:50:29.434785+0000","last_active":"2024-09-16T15:50:29.435965+0000","last_peered":"2024-09-16T15:50:29.435965+0000","last_clean":"2024-09-16T15:50:29.435965+0000","last_became_active":"2024-09-16T15:50:29.434102+0000","last_became_peered":"2024-09-16T15:50:29.434102+0000","last_unstale":"2024-09-16T15:50:29.435965+0000","last_undegraded":"2024-09-16T15:50:29.435965+0000","last_fullsized":"2024-09-16T15:50:29.435965+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:28.359944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:28.359944+0000","last_clean_scrub_stamp":"2024-09-16T15:50:28.359944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T02:25:34.661941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.f","version":"0'0","reported_seq":1,"reported_epoch":144,"state":"creating+peering","last_fresh":"2024-09-16T15:50:29.402843+0000","last_change":"2024-09-16T15:50:29.402843+0000","last_active":"2024-09-16T15:50:29.373642+0000","last_peered":"2024-09-16T15:50:29.373642+0000","last_clean":"2024-09-16T15:50:29.373642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:50:29.402843+0000","last_undegraded":"2024-09-16T15:50:29.402843+0000","last_fullsized":"2024-09-16T15:50:29.402843+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_clean_scrub_stamp":"2024-09-16T15:50:29.373642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.e","version":"0'0","reported_seq":10,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-16T15:50:29.429265+0000","last_change":"2024-09-16T15:50:29.426483+0000","last_active":"2024-09-16T15:50:29.429265+0000","last_peered":"2024-09-16T15:50:29.429265+0000","last_clean":"2024-09-16T15:50:29.429265+0000","last_became_active":"2024-09-16T15:50:29.425712+0000","last_became_peered":"2024-09-16T15:50:29.425712+0000","last_unstale":"2024-09-16T15:50:29.429265+0000","last_undegraded":"2024-09-16T15:50:29.429265+0000","last_fullsized":"2024-09-16T15:50:29.429265+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:28.359944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:28.359944+0000","last_clean_scrub_stamp":"2024-09-16T15:50:28.359944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:31:59.513487+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.c","version":"0'0","reported_seq":1,"reported_epoch":144,"state":"creating+peering","last_fresh":"2024-09-16T15:50:29.399219+0000","last_change":"2024-09-16T15:50:29.399219+0000","last_active":"2024-09-16T15:50:29.373642+0000","last_peered":"2024-09-16T15:50:29.373642+0000","last_clean":"2024-09-16T15:50:29.373642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:50:29.399219+0000","last_undegraded":"2024-09-16T15:50:29.399219+0000","last_fullsized":"2024-09-16T15:50:29.399219+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_clean_scrub_stamp":"2024-09-16T15:50:29.373642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.d","version":"0'0","reported_seq":10,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-16T15:50:29.430249+0000","last_change":"2024-09-16T15:50:29.427670+0000","last_active":"2024-09-16T15:50:29.430249+0000","last_peered":"2024-09-16T15:50:29.430249+0000","last_clean":"2024-09-16T15:50:29.430249+0000","last_became_active":"2024-09-16T15:50:29.426920+0000","last_became_peered":"2024-09-16T15:50:29.426920+0000","last_unstale":"2024-09-16T15:50:29.430249+0000","last_undegraded":"2024-09-16T15:50:29.430249+0000","last_fullsized":"2024-09-16T15:50:29.430249+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:28.359944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:28.359944+0000","last_clean_scrub_stamp":"2024-09-16T15:50:28.359944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:53:40.560984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.d","version":"0'0","reported_seq":1,"reported_epoch":144,"state":"creating+peering","last_fresh":"2024-09-16T15:50:29.395185+0000","last_change":"2024-09-16T15:50:29.395185+0000","last_active":"2024-09-16T15:50:29.373642+0000","last_peered":"2024-09-16T15:50:29.373642+0000","last_clean":"2024-09-16T15:50:29.373642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:50:29.395185+0000","last_undegraded":"2024-09-16T15:50:29.395185+0000","last_fullsized":"2024-09-16T15:50:29.395185+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_clean_scrub_stamp":"2024-09-16T15:50:29.373642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"22.c","version":"0'0","reported_seq":10,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-16T15:50:29.430128+0000","last_change":"2024-09-16T15:50:29.427240+0000","last_active":"2024-09-16T15:50:29.430128+0000","last_peered":"2024-09-16T15:50:29.430128+0000","last_clean":"2024-09-16T15:50:29.430128+0000","last_became_active":"2024-09-16T15:50:29.423280+0000","last_became_peered":"2024-09-16T15:50:29.423280+0000","last_unstale":"2024-09-16T15:50:29.430128+0000","last_undegraded":"2024-09-16T15:50:29.430128+0000","last_fullsized":"2024-09-16T15:50:29.430128+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:28.359944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:28.359944+0000","last_clean_scrub_stamp":"2024-09-16T15:50:28.359944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:36:54.628281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.a","version":"0'0","reported_seq":1,"reported_epoch":144,"state":"creating+peering","last_fresh":"2024-09-16T15:50:29.400460+0000","last_change":"2024-09-16T15:50:29.400460+0000","last_active":"2024-09-16T15:50:29.373642+0000","last_peered":"2024-09-16T15:50:29.373642+0000","last_clean":"2024-09-16T15:50:29.373642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:50:29.400460+0000","last_undegraded":"2024-09-16T15:50:29.400460+0000","last_fullsized":"2024-09-16T15:50:29.400460+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_clean_scrub_stamp":"2024-09-16T15:50:29.373642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.b","version":"0'0","reported_seq":10,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-16T15:50:29.431651+0000","last_change":"2024-09-16T15:50:29.425798+0000","last_active":"2024-09-16T15:50:29.431651+0000","last_peered":"2024-09-16T15:50:29.431651+0000","last_clean":"2024-09-16T15:50:29.431651+0000","last_became_active":"2024-09-16T15:50:29.425459+0000","last_became_peered":"2024-09-16T15:50:29.425459+0000","last_unstale":"2024-09-16T15:50:29.431651+0000","last_undegraded":"2024-09-16T15:50:29.431651+0000","last_fullsized":"2024-09-16T15:50:29.431651+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:28.359944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:28.359944+0000","last_clean_scrub_stamp":"2024-09-16T15:50:28.359944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:59:55.828797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.b","version":"0'0","reported_seq":1,"reported_epoch":144,"state":"creating+peering","last_fresh":"2024-09-16T15:50:29.394330+0000","last_change":"2024-09-16T15:50:29.394330+0000","last_active":"2024-09-16T15:50:29.373642+0000","last_peered":"2024-09-16T15:50:29.373642+0000","last_clean":"2024-09-16T15:50:29.373642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:50:29.394330+0000","last_undegraded":"2024-09-16T15:50:29.394330+0000","last_fullsized":"2024-09-16T15:50:29.394330+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_clean_scrub_stamp":"2024-09-16T15:50:29.373642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"22.a","version":"0'0","reported_seq":10,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-16T15:50:29.429851+0000","last_change":"2024-09-16T15:50:29.426752+0000","last_active":"2024-09-16T15:50:29.429851+0000","last_peered":"2024-09-16T15:50:29.429851+0000","last_clean":"2024-09-16T15:50:29.429851+0000","last_became_active":"2024-09-16T15:50:29.422755+0000","last_became_peered":"2024-09-16T15:50:29.422755+0000","last_unstale":"2024-09-16T15:50:29.429851+0000","last_undegraded":"2024-09-16T15:50:29.429851+0000","last_fullsized":"2024-09-16T15:50:29.429851+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:28.359944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:28.359944+0000","last_clean_scrub_stamp":"2024-09-16T15:50:28.359944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T18:14:40.770591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.8","version":"0'0","reported_seq":1,"reported_epoch":144,"state":"creating+peering","last_fresh":"2024-09-16T15:50:29.397861+0000","last_change":"2024-09-16T15:50:29.397861+0000","last_active":"2024-09-16T15:50:29.373642+0000","last_peered":"2024-09-16T15:50:29.373642+0000","last_clean":"2024-09-16T15:50:29.373642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:50:29.397861+0000","last_undegraded":"2024-09-16T15:50:29.397861+0000","last_fullsized":"2024-09-16T15:50:29.397861+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_clean_scrub_stamp":"2024-09-16T15:50:29.373642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.9","version":"0'0","reported_seq":10,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-16T15:50:29.424612+0000","last_change":"2024-09-16T15:50:29.423325+0000","last_active":"2024-09-16T15:50:29.424612+0000","last_peered":"2024-09-16T15:50:29.424612+0000","last_clean":"2024-09-16T15:50:29.424612+0000","last_became_active":"2024-09-16T15:50:29.423053+0000","last_became_peered":"2024-09-16T15:50:29.423053+0000","last_unstale":"2024-09-16T15:50:29.424612+0000","last_undegraded":"2024-09-16T15:50:29.424612+0000","last_fullsized":"2024-09-16T15:50:29.424612+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:28.359944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:28.359944+0000","last_clean_scrub_stamp":"2024-09-16T15:50:28.359944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T22:05:53.341097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.9","version":"0'0","reported_seq":1,"reported_epoch":144,"state":"creating+peering","last_fresh":"2024-09-16T15:50:29.396384+0000","last_change":"2024-09-16T15:50:29.396384+0000","last_active":"2024-09-16T15:50:29.373642+0000","last_peered":"2024-09-16T15:50:29.373642+0000","last_clean":"2024-09-16T15:50:29.373642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:50:29.396384+0000","last_undegraded":"2024-09-16T15:50:29.396384+0000","last_fullsized":"2024-09-16T15:50:29.396384+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_clean_scrub_stamp":"2024-09-16T15:50:29.373642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.8","version":"0'0","reported_seq":10,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-16T15:50:29.421906+0000","last_change":"2024-09-16T15:50:29.418765+0000","last_active":"2024-09-16T15:50:29.421906+0000","last_peered":"2024-09-16T15:50:29.421906+0000","last_clean":"2024-09-16T15:50:29.421906+0000","last_became_active":"2024-09-16T15:50:29.417737+0000","last_became_peered":"2024-09-16T15:50:29.417737+0000","last_unstale":"2024-09-16T15:50:29.421906+0000","last_undegraded":"2024-09-16T15:50:29.421906+0000","last_fullsized":"2024-09-16T15:50:29.421906+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:28.359944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:28.359944+0000","last_clean_scrub_stamp":"2024-09-16T15:50:28.359944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T00:16:50.967923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.6","version":"0'0","reported_seq":1,"reported_epoch":144,"state":"creating+peering","last_fresh":"2024-09-16T15:50:29.400497+0000","last_change":"2024-09-16T15:50:29.400497+0000","last_active":"2024-09-16T15:50:29.373642+0000","last_peered":"2024-09-16T15:50:29.373642+0000","last_clean":"2024-09-16T15:50:29.373642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:50:29.400497+0000","last_undegraded":"2024-09-16T15:50:29.400497+0000","last_fullsized":"2024-09-16T15:50:29.400497+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_clean_scrub_stamp":"2024-09-16T15:50:29.373642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.7","version":"0'0","reported_seq":10,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-16T15:50:29.423515+0000","last_change":"2024-09-16T15:50:29.422588+0000","last_active":"2024-09-16T15:50:29.423515+0000","last_peered":"2024-09-16T15:50:29.423515+0000","last_clean":"2024-09-16T15:50:29.423515+0000","last_became_active":"2024-09-16T15:50:29.422240+0000","last_became_peered":"2024-09-16T15:50:29.422240+0000","last_unstale":"2024-09-16T15:50:29.423515+0000","last_undegraded":"2024-09-16T15:50:29.423515+0000","last_fullsized":"2024-09-16T15:50:29.423515+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:28.359944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:28.359944+0000","last_clean_scrub_stamp":"2024-09-16T15:50:28.359944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T01:48:32.528831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"23.7","version":"0'0","reported_seq":1,"reported_epoch":144,"state":"creating+peering","last_fresh":"2024-09-16T15:50:29.394083+0000","last_change":"2024-09-16T15:50:29.394083+0000","last_active":"2024-09-16T15:50:29.373642+0000","last_peered":"2024-09-16T15:50:29.373642+0000","last_clean":"2024-09-16T15:50:29.373642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:50:29.394083+0000","last_undegraded":"2024-09-16T15:50:29.394083+0000","last_fullsized":"2024-09-16T15:50:29.394083+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_clean_scrub_stamp":"2024-09-16T15:50:29.373642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"22.6","version":"0'0","reported_seq":10,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-16T15:50:29.428409+0000","last_change":"2024-09-16T15:50:29.424687+0000","last_active":"2024-09-16T15:50:29.428409+0000","last_peered":"2024-09-16T15:50:29.428409+0000","last_clean":"2024-09-16T15:50:29.428409+0000","last_became_active":"2024-09-16T15:50:29.423323+0000","last_became_peered":"2024-09-16T15:50:29.423323+0000","last_unstale":"2024-09-16T15:50:29.428409+0000","last_undegraded":"2024-09-16T15:50:29.428409+0000","last_fullsized":"2024-09-16T15:50:29.428409+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:28.359944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:28.359944+0000","last_clean_scrub_stamp":"2024-09-16T15:50:28.359944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:26:50.321081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.4","version":"0'0","reported_seq":1,"reported_epoch":144,"state":"creating+peering","last_fresh":"2024-09-16T15:50:29.395179+0000","last_change":"2024-09-16T15:50:29.395179+0000","last_active":"2024-09-16T15:50:29.373642+0000","last_peered":"2024-09-16T15:50:29.373642+0000","last_clean":"2024-09-16T15:50:29.373642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:50:29.395179+0000","last_undegraded":"2024-09-16T15:50:29.395179+0000","last_fullsized":"2024-09-16T15:50:29.395179+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_clean_scrub_stamp":"2024-09-16T15:50:29.373642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.5","version":"0'0","reported_seq":10,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-16T15:50:29.429727+0000","last_change":"2024-09-16T15:50:29.421716+0000","last_active":"2024-09-16T15:50:29.429727+0000","last_peered":"2024-09-16T15:50:29.429727+0000","last_clean":"2024-09-16T15:50:29.429727+0000","last_became_active":"2024-09-16T15:50:29.420529+0000","last_became_peered":"2024-09-16T15:50:29.420529+0000","last_unstale":"2024-09-16T15:50:29.429727+0000","last_undegraded":"2024-09-16T15:50:29.429727+0000","last_fullsized":"2024-09-16T15:50:29.429727+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:28.359944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:28.359944+0000","last_clean_scrub_stamp":"2024-09-16T15:50:28.359944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T03:46:32.264684+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.5","version":"0'0","reported_seq":1,"reported_epoch":144,"state":"creating+peering","last_fresh":"2024-09-16T15:50:29.400308+0000","last_change":"2024-09-16T15:50:29.400308+0000","last_active":"2024-09-16T15:50:29.373642+0000","last_peered":"2024-09-16T15:50:29.373642+0000","last_clean":"2024-09-16T15:50:29.373642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:50:29.400308+0000","last_undegraded":"2024-09-16T15:50:29.400308+0000","last_fullsized":"2024-09-16T15:50:29.400308+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_clean_scrub_stamp":"2024-09-16T15:50:29.373642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.4","version":"0'0","reported_seq":10,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-16T15:50:29.417851+0000","last_change":"2024-09-16T15:50:29.417210+0000","last_active":"2024-09-16T15:50:29.417851+0000","last_peered":"2024-09-16T15:50:29.417851+0000","last_clean":"2024-09-16T15:50:29.417851+0000","last_became_active":"2024-09-16T15:50:29.416831+0000","last_became_peered":"2024-09-16T15:50:29.416831+0000","last_unstale":"2024-09-16T15:50:29.417851+0000","last_undegraded":"2024-09-16T15:50:29.417851+0000","last_fullsized":"2024-09-16T15:50:29.417851+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:28.359944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:28.359944+0000","last_clean_scrub_stamp":"2024-09-16T15:50:28.359944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:00:59.918701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"23.2","version":"0'0","reported_seq":1,"reported_epoch":144,"state":"creating+peering","last_fresh":"2024-09-16T15:50:29.400856+0000","last_change":"2024-09-16T15:50:29.400856+0000","last_active":"2024-09-16T15:50:29.373642+0000","last_peered":"2024-09-16T15:50:29.373642+0000","last_clean":"2024-09-16T15:50:29.373642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:50:29.400856+0000","last_undegraded":"2024-09-16T15:50:29.400856+0000","last_fullsized":"2024-09-16T15:50:29.400856+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_clean_scrub_stamp":"2024-09-16T15:50:29.373642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.3","version":"0'0","reported_seq":10,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-16T15:50:29.421008+0000","last_change":"2024-09-16T15:50:29.407104+0000","last_active":"2024-09-16T15:50:29.421008+0000","last_peered":"2024-09-16T15:50:29.421008+0000","last_clean":"2024-09-16T15:50:29.421008+0000","last_became_active":"2024-09-16T15:50:29.406182+0000","last_became_peered":"2024-09-16T15:50:29.406182+0000","last_unstale":"2024-09-16T15:50:29.421008+0000","last_undegraded":"2024-09-16T15:50:29.421008+0000","last_fullsized":"2024-09-16T15:50:29.421008+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:28.359944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:28.359944+0000","last_clean_scrub_stamp":"2024-09-16T15:50:28.359944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:36:55.485971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"23.3","version":"0'0","reported_seq":1,"reported_epoch":144,"state":"creating+peering","last_fresh":"2024-09-16T15:50:29.391321+0000","last_change":"2024-09-16T15:50:29.391321+0000","last_active":"2024-09-16T15:50:29.373642+0000","last_peered":"2024-09-16T15:50:29.373642+0000","last_clean":"2024-09-16T15:50:29.373642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:50:29.391321+0000","last_undegraded":"2024-09-16T15:50:29.391321+0000","last_fullsized":"2024-09-16T15:50:29.391321+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_clean_scrub_stamp":"2024-09-16T15:50:29.373642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.2","version":"0'0","reported_seq":10,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-16T15:50:29.411842+0000","last_change":"2024-09-16T15:50:29.410821+0000","last_active":"2024-09-16T15:50:29.411842+0000","last_peered":"2024-09-16T15:50:29.411842+0000","last_clean":"2024-09-16T15:50:29.411842+0000","last_became_active":"2024-09-16T15:50:29.410554+0000","last_became_peered":"2024-09-16T15:50:29.410554+0000","last_unstale":"2024-09-16T15:50:29.411842+0000","last_undegraded":"2024-09-16T15:50:29.411842+0000","last_fullsized":"2024-09-16T15:50:29.411842+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:28.359944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:28.359944+0000","last_clean_scrub_stamp":"2024-09-16T15:50:28.359944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:55:15.802642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.0","version":"0'0","reported_seq":1,"reported_epoch":144,"state":"creating+peering","last_fresh":"2024-09-16T15:50:29.394199+0000","last_change":"2024-09-16T15:50:29.394199+0000","last_active":"2024-09-16T15:50:29.373642+0000","last_peered":"2024-09-16T15:50:29.373642+0000","last_clean":"2024-09-16T15:50:29.373642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:50:29.394199+0000","last_undegraded":"2024-09-16T15:50:29.394199+0000","last_fullsized":"2024-09-16T15:50:29.394199+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_clean_scrub_stamp":"2024-09-16T15:50:29.373642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"22.1","version":"0'0","reported_seq":10,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-16T15:50:29.421042+0000","last_change":"2024-09-16T15:50:29.416751+0000","last_active":"2024-09-16T15:50:29.421042+0000","last_peered":"2024-09-16T15:50:29.421042+0000","last_clean":"2024-09-16T15:50:29.421042+0000","last_became_active":"2024-09-16T15:50:29.415995+0000","last_became_peered":"2024-09-16T15:50:29.415995+0000","last_unstale":"2024-09-16T15:50:29.421042+0000","last_undegraded":"2024-09-16T15:50:29.421042+0000","last_fullsized":"2024-09-16T15:50:29.421042+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:28.359944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:28.359944+0000","last_clean_scrub_stamp":"2024-09-16T15:50:28.359944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T02:16:57.915820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.1","version":"0'0","reported_seq":1,"reported_epoch":144,"state":"creating+peering","last_fresh":"2024-09-16T15:50:29.409753+0000","last_change":"2024-09-16T15:50:29.409753+0000","last_active":"2024-09-16T15:50:29.373642+0000","last_peered":"2024-09-16T15:50:29.373642+0000","last_clean":"2024-09-16T15:50:29.373642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:50:29.409753+0000","last_undegraded":"2024-09-16T15:50:29.409753+0000","last_fullsized":"2024-09-16T15:50:29.409753+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_clean_scrub_stamp":"2024-09-16T15:50:29.373642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.0","version":"0'0","reported_seq":10,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-16T15:50:29.406562+0000","last_change":"2024-09-16T15:50:29.395361+0000","last_active":"2024-09-16T15:50:29.406562+0000","last_peered":"2024-09-16T15:50:29.406562+0000","last_clean":"2024-09-16T15:50:29.406562+0000","last_became_active":"2024-09-16T15:50:29.394578+0000","last_became_peered":"2024-09-16T15:50:29.394578+0000","last_unstale":"2024-09-16T15:50:29.406562+0000","last_undegraded":"2024-09-16T15:50:29.406562+0000","last_fullsized":"2024-09-16T15:50:29.406562+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:28.359944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:28.359944+0000","last_clean_scrub_stamp":"2024-09-16T15:50:28.359944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T02:06:32.966504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"23.16","version":"0'0","reported_seq":1,"reported_epoch":144,"state":"creating+peering","last_fresh":"2024-09-16T15:50:29.401057+0000","last_change":"2024-09-16T15:50:29.401057+0000","last_active":"2024-09-16T15:50:29.373642+0000","last_peered":"2024-09-16T15:50:29.373642+0000","last_clean":"2024-09-16T15:50:29.373642+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T15:50:29.401057+0000","last_undegraded":"2024-09-16T15:50:29.401057+0000","last_fullsized":"2024-09-16T15:50:29.401057+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_clean_scrub_stamp":"2024-09-16T15:50:29.373642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.17","version":"0'0","reported_seq":10,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-16T15:50:29.421257+0000","last_change":"2024-09-16T15:50:29.410992+0000","last_active":"2024-09-16T15:50:29.421257+0000","last_peered":"2024-09-16T15:50:29.421257+0000","last_clean":"2024-09-16T15:50:29.421257+0000","last_became_active":"2024-09-16T15:50:29.410215+0000","last_became_peered":"2024-09-16T15:50:29.410215+0000","last_unstale":"2024-09-16T15:50:29.421257+0000","last_undegraded":"2024-09-16T15:50:29.421257+0000","last_fullsized":"2024-09-16T15:50:29.421257+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:28.359944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:28.359944+0000","last_clean_scrub_stamp":"2024-09-16T15:50:28.359944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T00:24:19.013129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"73'79","reported_seq":366,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-16T15:50:29.402844+0000","last_change":"2024-09-16T15:42:40.881820+0000","last_active":"2024-09-16T15:50:29.402844+0000","last_peered":"2024-09-16T15:50:29.402844+0000","last_clean":"2024-09-16T15:50:29.402844+0000","last_became_active":"2024-09-16T15:42:40.879965+0000","last_became_peered":"2024-09-16T15:42:40.879965+0000","last_unstale":"2024-09-16T15:50:29.402844+0000","last_undegraded":"2024-09-16T15:50:29.402844+0000","last_fullsized":"2024-09-16T15:50:29.402844+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:42:39.728762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:42:39.728762+0000","last_clean_scrub_stamp":"2024-09-16T15:42:39.728762+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:22:18.768932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":23,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":22,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79,"ondisk_log_size":79,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542242,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33536,"kb_used_data":2344,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338304,"statfs":{"total":96636764160,"available":96602423296,"internally_reserved":0,"allocated":2400256,"data_stored":1389123,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542241,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32956,"kb_used_data":1764,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338884,"statfs":{"total":96636764160,"available":96603017216,"internally_reserved":0,"allocated":1806336,"data_stored":798755,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":5,"up_from":14,"seq":60129542242,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32940,"kb_used_data":1752,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338900,"statfs":{"total":96636764160,"available":96603033600,"internally_reserved":0,"allocated":1794048,"data_stored":792452,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542240,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32940,"kb_used_data":1752,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338900,"statfs":{"total":96636764160,"available":96603033600,"internally_reserved":0,"allocated":1794048,"data_stored":792452,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":3,"up_from":14,"seq":60129542241,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37100,"kb_used_data":1752,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334740,"statfs":{"total":96636764160,"available":96598773760,"internally_reserved":0,"allocated":1794048,"data_stored":792452,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":14,"seq":60129542241,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32956,"kb_used_data":1764,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338884,"statfs":{"total":96636764160,"available":96603017216,"internally_reserved":0,"allocated":1806336,"data_stored":798755,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":14,"seq":60129542243,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32956,"kb_used_data":1764,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338884,"statfs":{"total":96636764160,"available":96603017216,"internally_reserved":0,"allocated":1806336,"data_stored":798755,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":14,"seq":60129542243,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33520,"kb_used_data":2332,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338320,"statfs":{"total":96636764160,"available":96602439680,"internally_reserved":0,"allocated":2387968,"data_stored":1382820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T15:50:38.888 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T15:50:39.025 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:39.026+0000 7f9356426640 1 -- 172.21.15.123:0/1753627401 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f93500cd6e0 msgr2=0x7f93500cdb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:50:39.025 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:39.026+0000 7f9356426640 1 --2- 172.21.15.123:0/1753627401 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f93500cd6e0 0x7f93500cdb50 secure :-1 s=READY pgs=1602 cs=0 l=1 rev1=1 crypto rx=0x7f9344009840 tx=0x7f934401c790 comp rx=0 tx=0).stop 2024-09-16T15:50:39.025 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:39.026+0000 7f9356426640 1 -- 172.21.15.123:0/1753627401 shutdown_connections 2024-09-16T15:50:39.025 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:39.026+0000 7f9356426640 1 --2- 172.21.15.123:0/1753627401 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f935015e820 0x7f935015ecb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:39.025 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:39.026+0000 7f9356426640 1 --2- 172.21.15.123:0/1753627401 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f93500cd6e0 0x7f93500cdb50 unknown :-1 s=CLOSED pgs=1602 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:39.025 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:39.026+0000 7f9356426640 1 -- 172.21.15.123:0/1753627401 >> 172.21.15.123:0/1753627401 conn(0x7f93500cfb10 msgr2=0x7f93500cff20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:50:39.026 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:39.026+0000 7f9356426640 1 -- 172.21.15.123:0/1753627401 shutdown_connections 2024-09-16T15:50:39.026 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:39.026+0000 7f9356426640 1 -- 172.21.15.123:0/1753627401 wait complete. 2024-09-16T15:50:39.026 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:39.026+0000 7f9356426640 1 Processor -- start 2024-09-16T15:50:39.026 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:39.026+0000 7f9356426640 1 -- start start 2024-09-16T15:50:39.029 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:39.026+0000 7f9356426640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f93500cd6e0 0x7f9350155cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:50:39.029 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:39.026+0000 7f9356426640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f935015e160 0x7f9350150030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:50:39.029 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:39.026+0000 7f9356426640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f935015e820 0x7f93500b3cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:50:39.029 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:39.026+0000 7f9356426640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f93500cf5b0 con 0x7f935015e820 2024-09-16T15:50:39.029 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:39.026+0000 7f9356426640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f93500cf430 con 0x7f93500cd6e0 2024-09-16T15:50:39.029 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:39.026+0000 7f9356426640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f93501663c0 con 0x7f935015e160 2024-09-16T15:50:39.029 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:39.030+0000 7f93549c3640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f935015e820 0x7f93500b3cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:50:39.029 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:39.030+0000 7f93549c3640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f935015e820 0x7f93500b3cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:49754/0 (socket says 172.21.15.123:49754) 2024-09-16T15:50:39.029 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:39.030+0000 7f93549c3640 1 -- 172.21.15.123:0/3898794804 learned_addr learned my addr 172.21.15.123:0/3898794804 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:50:39.029 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:39.030+0000 7f934f7fe640 1 --2- 172.21.15.123:0/3898794804 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f935015e160 0x7f9350150030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:50:39.029 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:39.030+0000 7f934ffff640 1 --2- 172.21.15.123:0/3898794804 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f93500cd6e0 0x7f9350155cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:50:39.029 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:39.030+0000 7f93549c3640 1 -- 172.21.15.123:0/3898794804 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f935015e160 msgr2=0x7f9350150030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:50:39.030 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:39.030+0000 7f93549c3640 1 --2- 172.21.15.123:0/3898794804 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f935015e160 0x7f9350150030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:39.030 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:39.030+0000 7f93549c3640 1 -- 172.21.15.123:0/3898794804 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f93500cd6e0 msgr2=0x7f9350155cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:50:39.030 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:39.030+0000 7f93549c3640 1 --2- 172.21.15.123:0/3898794804 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f93500cd6e0 0x7f9350155cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:39.030 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:39.030+0000 7f93549c3640 1 -- 172.21.15.123:0/3898794804 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9350168c00 con 0x7f935015e820 2024-09-16T15:50:39.030 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:39.030+0000 7f934ffff640 1 --2- 172.21.15.123:0/3898794804 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f93500cd6e0 0x7f9350155cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T15:50:39.030 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:39.030+0000 7f93549c3640 1 --2- 172.21.15.123:0/3898794804 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f935015e820 0x7f93500b3cc0 secure :-1 s=READY pgs=1603 cs=0 l=1 rev1=1 crypto rx=0x7f934800b970 tx=0x7f934800be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:50:39.030 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:39.030+0000 7f934cff9640 1 -- 172.21.15.123:0/3898794804 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9348019070 con 0x7f935015e820 2024-09-16T15:50:39.030 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:39.030+0000 7f934cff9640 1 -- 172.21.15.123:0/3898794804 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f934800cc50 con 0x7f935015e820 2024-09-16T15:50:39.030 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:39.030+0000 7f934cff9640 1 -- 172.21.15.123:0/3898794804 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93480146a0 con 0x7f935015e820 2024-09-16T15:50:39.030 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:39.030+0000 7f9356426640 1 -- 172.21.15.123:0/3898794804 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9350150570 con 0x7f935015e820 2024-09-16T15:50:39.030 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:39.030+0000 7f9356426640 1 -- 172.21.15.123:0/3898794804 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9350168640 con 0x7f935015e820 2024-09-16T15:50:39.031 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:39.030+0000 7f9356426640 1 -- 172.21.15.123:0/3898794804 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f93501668c0 con 0x7f935015e820 2024-09-16T15:50:39.031 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:39.030+0000 7f934cff9640 1 -- 172.21.15.123:0/3898794804 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 22) ==== 97692+0+0 (secure 0 0 0) 0x7f934801c070 con 0x7f935015e820 2024-09-16T15:50:39.031 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:39.030+0000 7f934cff9640 1 --2- 172.21.15.123:0/3898794804 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f933c0735b0 0x7f933c075a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:50:39.031 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:39.030+0000 7f934cff9640 1 -- 172.21.15.123:0/3898794804 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(146..146 src has 1..146) ==== 5737+0+0 (secure 0 0 0) 0x7f9348094b40 con 0x7f935015e820 2024-09-16T15:50:39.031 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:39.030+0000 7f934ffff640 1 --2- 172.21.15.123:0/3898794804 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f933c0735b0 0x7f933c075a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:50:39.031 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:39.030+0000 7f934ffff640 1 --2- 172.21.15.123:0/3898794804 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f933c0735b0 0x7f933c075a70 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f93400016d0 tx=0x7f9340009040 comp rx=0 tx=0).ready entity=mgr.4150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:50:39.034 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:39.034+0000 7f934cff9640 1 -- 172.21.15.123:0/3898794804 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f9348023020 con 0x7f935015e820 2024-09-16T15:50:39.204 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:39.206+0000 7f9356426640 1 -- 172.21.15.123:0/3898794804 --> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f9350156200 con 0x7f933c0735b0 2024-09-16T15:50:39.209 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:39.210+0000 7f934cff9640 1 -- 172.21.15.123:0/3898794804 <== mgr.4150 v2:172.21.15.123:6836/2464759676 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+252484 (secure 0 0 0) 0x7f9340001cb0 con 0x7f933c0735b0 2024-09-16T15:50:39.212 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-09-16T15:50:39.214 INFO:teuthology.orchestra.run.smithi123.stderr:dumped all 2024-09-16T15:50:39.215 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:39.214+0000 7f9356426640 1 -- 172.21.15.123:0/3898794804 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f933c0735b0 msgr2=0x7f933c075a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:50:39.215 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:39.214+0000 7f9356426640 1 --2- 172.21.15.123:0/3898794804 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f933c0735b0 0x7f933c075a70 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f93400016d0 tx=0x7f9340009040 comp rx=0 tx=0).stop 2024-09-16T15:50:39.215 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:39.214+0000 7f9356426640 1 -- 172.21.15.123:0/3898794804 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f935015e820 msgr2=0x7f93500b3cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:50:39.215 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:39.214+0000 7f9356426640 1 --2- 172.21.15.123:0/3898794804 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f935015e820 0x7f93500b3cc0 secure :-1 s=READY pgs=1603 cs=0 l=1 rev1=1 crypto rx=0x7f934800b970 tx=0x7f934800be30 comp rx=0 tx=0).stop 2024-09-16T15:50:39.215 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:39.214+0000 7f9356426640 1 -- 172.21.15.123:0/3898794804 shutdown_connections 2024-09-16T15:50:39.215 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:39.214+0000 7f9356426640 1 --2- 172.21.15.123:0/3898794804 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f933c0735b0 0x7f933c075a70 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:39.215 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:39.214+0000 7f9356426640 1 --2- 172.21.15.123:0/3898794804 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f935015e820 0x7f93500b3cc0 unknown :-1 s=CLOSED pgs=1603 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:39.215 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:39.214+0000 7f9356426640 1 --2- 172.21.15.123:0/3898794804 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f935015e160 0x7f9350150030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:39.215 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:39.214+0000 7f9356426640 1 --2- 172.21.15.123:0/3898794804 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f93500cd6e0 0x7f9350155cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:39.215 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:39.214+0000 7f9356426640 1 -- 172.21.15.123:0/3898794804 >> 172.21.15.123:0/3898794804 conn(0x7f93500cfb10 msgr2=0x7f9350156430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:50:39.215 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:39.214+0000 7f9356426640 1 -- 172.21.15.123:0/3898794804 shutdown_connections 2024-09-16T15:50:39.215 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:39.214+0000 7f9356426640 1 -- 172.21.15.123:0/3898794804 wait complete. 2024-09-16T15:50:39.232 INFO:teuthology.orchestra.run.smithi123.stdout:{"pg_ready":true,"pg_map":{"version":169,"stamp":"2024-09-16T15:50:38.121219+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":158,"num_write_kb":1642,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":118,"ondisk_log_size":118,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":186,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":269212,"kb_used_data":15528,"kb_used_omap":12,"kb_used_meta":253491,"kb_avail":754705508,"statfs":{"total":773094113280,"available":772818440192,"internally_reserved":0,"allocated":15900672,"data_stored":7680644,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":259575345},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":42,"apply_latency_ms":42,"commit_latency_ns":42000000,"apply_latency_ns":42000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002829"},"pg_stats":[{"pgid":"23.3f","version":"0'0","reported_seq":13,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-09-16T15:50:32.108822+0000","last_change":"2024-09-16T15:50:30.443060+0000","last_active":"2024-09-16T15:50:32.108822+0000","last_peered":"2024-09-16T15:50:32.108822+0000","last_clean":"2024-09-16T15:50:32.108822+0000","last_became_active":"2024-09-16T15:50:30.442767+0000","last_became_peered":"2024-09-16T15:50:30.442767+0000","last_unstale":"2024-09-16T15:50:32.108822+0000","last_undegraded":"2024-09-16T15:50:32.108822+0000","last_fullsized":"2024-09-16T15:50:32.108822+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_clean_scrub_stamp":"2024-09-16T15:50:29.373642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T23:38:42.640768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.3e","version":"0'0","reported_seq":13,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-09-16T15:50:31.423681+0000","last_change":"2024-09-16T15:50:30.441809+0000","last_active":"2024-09-16T15:50:31.423681+0000","last_peered":"2024-09-16T15:50:31.423681+0000","last_clean":"2024-09-16T15:50:31.423681+0000","last_became_active":"2024-09-16T15:50:30.441493+0000","last_became_peered":"2024-09-16T15:50:30.441493+0000","last_unstale":"2024-09-16T15:50:31.423681+0000","last_undegraded":"2024-09-16T15:50:31.423681+0000","last_fullsized":"2024-09-16T15:50:31.423681+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_clean_scrub_stamp":"2024-09-16T15:50:29.373642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:41:41.769424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.3d","version":"0'0","reported_seq":13,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-09-16T15:50:31.427284+0000","last_change":"2024-09-16T15:50:30.425528+0000","last_active":"2024-09-16T15:50:31.427284+0000","last_peered":"2024-09-16T15:50:31.427284+0000","last_clean":"2024-09-16T15:50:31.427284+0000","last_became_active":"2024-09-16T15:50:30.425166+0000","last_became_peered":"2024-09-16T15:50:30.425166+0000","last_unstale":"2024-09-16T15:50:31.427284+0000","last_undegraded":"2024-09-16T15:50:31.427284+0000","last_fullsized":"2024-09-16T15:50:31.427284+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_clean_scrub_stamp":"2024-09-16T15:50:29.373642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T22:07:14.754017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.3c","version":"0'0","reported_seq":13,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-09-16T15:50:31.420855+0000","last_change":"2024-09-16T15:50:30.427136+0000","last_active":"2024-09-16T15:50:31.420855+0000","last_peered":"2024-09-16T15:50:31.420855+0000","last_clean":"2024-09-16T15:50:31.420855+0000","last_became_active":"2024-09-16T15:50:30.426565+0000","last_became_peered":"2024-09-16T15:50:30.426565+0000","last_unstale":"2024-09-16T15:50:31.420855+0000","last_undegraded":"2024-09-16T15:50:31.420855+0000","last_fullsized":"2024-09-16T15:50:31.420855+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_clean_scrub_stamp":"2024-09-16T15:50:29.373642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:07:03.900408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.3b","version":"0'0","reported_seq":13,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-09-16T15:50:31.422430+0000","last_change":"2024-09-16T15:50:30.429446+0000","last_active":"2024-09-16T15:50:31.422430+0000","last_peered":"2024-09-16T15:50:31.422430+0000","last_clean":"2024-09-16T15:50:31.422430+0000","last_became_active":"2024-09-16T15:50:30.428886+0000","last_became_peered":"2024-09-16T15:50:30.428886+0000","last_unstale":"2024-09-16T15:50:31.422430+0000","last_undegraded":"2024-09-16T15:50:31.422430+0000","last_fullsized":"2024-09-16T15:50:31.422430+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_clean_scrub_stamp":"2024-09-16T15:50:29.373642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:36:56.499669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.3a","version":"0'0","reported_seq":13,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-09-16T15:50:32.118415+0000","last_change":"2024-09-16T15:50:30.433158+0000","last_active":"2024-09-16T15:50:32.118415+0000","last_peered":"2024-09-16T15:50:32.118415+0000","last_clean":"2024-09-16T15:50:32.118415+0000","last_became_active":"2024-09-16T15:50:30.432824+0000","last_became_peered":"2024-09-16T15:50:30.432824+0000","last_unstale":"2024-09-16T15:50:32.118415+0000","last_undegraded":"2024-09-16T15:50:32.118415+0000","last_fullsized":"2024-09-16T15:50:32.118415+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_clean_scrub_stamp":"2024-09-16T15:50:29.373642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:15:49.525325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.39","version":"0'0","reported_seq":13,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-09-16T15:50:32.577780+0000","last_change":"2024-09-16T15:50:30.417657+0000","last_active":"2024-09-16T15:50:32.577780+0000","last_peered":"2024-09-16T15:50:32.577780+0000","last_clean":"2024-09-16T15:50:32.577780+0000","last_became_active":"2024-09-16T15:50:30.416794+0000","last_became_peered":"2024-09-16T15:50:30.416794+0000","last_unstale":"2024-09-16T15:50:32.577780+0000","last_undegraded":"2024-09-16T15:50:32.577780+0000","last_fullsized":"2024-09-16T15:50:32.577780+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_clean_scrub_stamp":"2024-09-16T15:50:29.373642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:04:53.897644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"23.38","version":"0'0","reported_seq":13,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-09-16T15:50:31.422727+0000","last_change":"2024-09-16T15:50:30.418525+0000","last_active":"2024-09-16T15:50:31.422727+0000","last_peered":"2024-09-16T15:50:31.422727+0000","last_clean":"2024-09-16T15:50:31.422727+0000","last_became_active":"2024-09-16T15:50:30.418231+0000","last_became_peered":"2024-09-16T15:50:30.418231+0000","last_unstale":"2024-09-16T15:50:31.422727+0000","last_undegraded":"2024-09-16T15:50:31.422727+0000","last_fullsized":"2024-09-16T15:50:31.422727+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_clean_scrub_stamp":"2024-09-16T15:50:29.373642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T00:24:20.026827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.37","version":"0'0","reported_seq":13,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-09-16T15:50:31.424409+0000","last_change":"2024-09-16T15:50:30.434165+0000","last_active":"2024-09-16T15:50:31.424409+0000","last_peered":"2024-09-16T15:50:31.424409+0000","last_clean":"2024-09-16T15:50:31.424409+0000","last_became_active":"2024-09-16T15:50:30.433852+0000","last_became_peered":"2024-09-16T15:50:30.433852+0000","last_unstale":"2024-09-16T15:50:31.424409+0000","last_undegraded":"2024-09-16T15:50:31.424409+0000","last_fullsized":"2024-09-16T15:50:31.424409+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_clean_scrub_stamp":"2024-09-16T15:50:29.373642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T03:20:18.058943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.36","version":"0'0","reported_seq":13,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-09-16T15:50:31.423770+0000","last_change":"2024-09-16T15:50:30.428599+0000","last_active":"2024-09-16T15:50:31.423770+0000","last_peered":"2024-09-16T15:50:31.423770+0000","last_clean":"2024-09-16T15:50:31.423770+0000","last_became_active":"2024-09-16T15:50:30.427981+0000","last_became_peered":"2024-09-16T15:50:30.427981+0000","last_unstale":"2024-09-16T15:50:31.423770+0000","last_undegraded":"2024-09-16T15:50:31.423770+0000","last_fullsized":"2024-09-16T15:50:31.423770+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_clean_scrub_stamp":"2024-09-16T15:50:29.373642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:38:18.213928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.35","version":"0'0","reported_seq":13,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-09-16T15:50:31.422312+0000","last_change":"2024-09-16T15:50:30.430444+0000","last_active":"2024-09-16T15:50:31.422312+0000","last_peered":"2024-09-16T15:50:31.422312+0000","last_clean":"2024-09-16T15:50:31.422312+0000","last_became_active":"2024-09-16T15:50:30.429558+0000","last_became_peered":"2024-09-16T15:50:30.429558+0000","last_unstale":"2024-09-16T15:50:31.422312+0000","last_undegraded":"2024-09-16T15:50:31.422312+0000","last_fullsized":"2024-09-16T15:50:31.422312+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_clean_scrub_stamp":"2024-09-16T15:50:29.373642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T03:40:14.624351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.34","version":"0'0","reported_seq":13,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-09-16T15:50:31.422136+0000","last_change":"2024-09-16T15:50:30.442082+0000","last_active":"2024-09-16T15:50:31.422136+0000","last_peered":"2024-09-16T15:50:31.422136+0000","last_clean":"2024-09-16T15:50:31.422136+0000","last_became_active":"2024-09-16T15:50:30.441713+0000","last_became_peered":"2024-09-16T15:50:30.441713+0000","last_unstale":"2024-09-16T15:50:31.422136+0000","last_undegraded":"2024-09-16T15:50:31.422136+0000","last_fullsized":"2024-09-16T15:50:31.422136+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_clean_scrub_stamp":"2024-09-16T15:50:29.373642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T02:06:33.980202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.33","version":"0'0","reported_seq":13,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-09-16T15:50:31.426772+0000","last_change":"2024-09-16T15:50:30.428594+0000","last_active":"2024-09-16T15:50:31.426772+0000","last_peered":"2024-09-16T15:50:31.426772+0000","last_clean":"2024-09-16T15:50:31.426772+0000","last_became_active":"2024-09-16T15:50:30.428244+0000","last_became_peered":"2024-09-16T15:50:30.428244+0000","last_unstale":"2024-09-16T15:50:31.426772+0000","last_undegraded":"2024-09-16T15:50:31.426772+0000","last_fullsized":"2024-09-16T15:50:31.426772+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_clean_scrub_stamp":"2024-09-16T15:50:29.373642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:58:59.456368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.32","version":"0'0","reported_seq":13,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-09-16T15:50:31.426574+0000","last_change":"2024-09-16T15:50:30.416017+0000","last_active":"2024-09-16T15:50:31.426574+0000","last_peered":"2024-09-16T15:50:31.426574+0000","last_clean":"2024-09-16T15:50:31.426574+0000","last_became_active":"2024-09-16T15:50:30.415247+0000","last_became_peered":"2024-09-16T15:50:30.415247+0000","last_unstale":"2024-09-16T15:50:31.426574+0000","last_undegraded":"2024-09-16T15:50:31.426574+0000","last_fullsized":"2024-09-16T15:50:31.426574+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_clean_scrub_stamp":"2024-09-16T15:50:29.373642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:02:11.646385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.31","version":"0'0","reported_seq":13,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-09-16T15:50:32.108173+0000","last_change":"2024-09-16T15:50:30.438270+0000","last_active":"2024-09-16T15:50:32.108173+0000","last_peered":"2024-09-16T15:50:32.108173+0000","last_clean":"2024-09-16T15:50:32.108173+0000","last_became_active":"2024-09-16T15:50:30.437937+0000","last_became_peered":"2024-09-16T15:50:30.437937+0000","last_unstale":"2024-09-16T15:50:32.108173+0000","last_undegraded":"2024-09-16T15:50:32.108173+0000","last_fullsized":"2024-09-16T15:50:32.108173+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_clean_scrub_stamp":"2024-09-16T15:50:29.373642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T03:26:49.980495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.30","version":"0'0","reported_seq":13,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-09-16T15:50:32.577636+0000","last_change":"2024-09-16T15:50:30.442346+0000","last_active":"2024-09-16T15:50:32.577636+0000","last_peered":"2024-09-16T15:50:32.577636+0000","last_clean":"2024-09-16T15:50:32.577636+0000","last_became_active":"2024-09-16T15:50:30.441749+0000","last_became_peered":"2024-09-16T15:50:30.441749+0000","last_unstale":"2024-09-16T15:50:32.577636+0000","last_undegraded":"2024-09-16T15:50:32.577636+0000","last_fullsized":"2024-09-16T15:50:32.577636+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_clean_scrub_stamp":"2024-09-16T15:50:29.373642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T18:00:48.580097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"23.2f","version":"0'0","reported_seq":13,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-09-16T15:50:32.118492+0000","last_change":"2024-09-16T15:50:30.424700+0000","last_active":"2024-09-16T15:50:32.118492+0000","last_peered":"2024-09-16T15:50:32.118492+0000","last_clean":"2024-09-16T15:50:32.118492+0000","last_became_active":"2024-09-16T15:50:30.424203+0000","last_became_peered":"2024-09-16T15:50:30.424203+0000","last_unstale":"2024-09-16T15:50:32.118492+0000","last_undegraded":"2024-09-16T15:50:32.118492+0000","last_fullsized":"2024-09-16T15:50:32.118492+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_clean_scrub_stamp":"2024-09-16T15:50:29.373642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T22:49:35.305979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.2e","version":"0'0","reported_seq":13,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-09-16T15:50:31.712053+0000","last_change":"2024-09-16T15:50:30.429486+0000","last_active":"2024-09-16T15:50:31.712053+0000","last_peered":"2024-09-16T15:50:31.712053+0000","last_clean":"2024-09-16T15:50:31.712053+0000","last_became_active":"2024-09-16T15:50:30.428398+0000","last_became_peered":"2024-09-16T15:50:30.428398+0000","last_unstale":"2024-09-16T15:50:31.712053+0000","last_undegraded":"2024-09-16T15:50:31.712053+0000","last_fullsized":"2024-09-16T15:50:31.712053+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_clean_scrub_stamp":"2024-09-16T15:50:29.373642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T00:28:16.941253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"23.2d","version":"0'0","reported_seq":13,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-09-16T15:50:32.118727+0000","last_change":"2024-09-16T15:50:30.425401+0000","last_active":"2024-09-16T15:50:32.118727+0000","last_peered":"2024-09-16T15:50:32.118727+0000","last_clean":"2024-09-16T15:50:32.118727+0000","last_became_active":"2024-09-16T15:50:30.425040+0000","last_became_peered":"2024-09-16T15:50:30.425040+0000","last_unstale":"2024-09-16T15:50:32.118727+0000","last_undegraded":"2024-09-16T15:50:32.118727+0000","last_fullsized":"2024-09-16T15:50:32.118727+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_clean_scrub_stamp":"2024-09-16T15:50:29.373642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T01:55:06.149734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.2c","version":"0'0","reported_seq":13,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-09-16T15:50:31.427752+0000","last_change":"2024-09-16T15:50:30.442057+0000","last_active":"2024-09-16T15:50:31.427752+0000","last_peered":"2024-09-16T15:50:31.427752+0000","last_clean":"2024-09-16T15:50:31.427752+0000","last_became_active":"2024-09-16T15:50:30.441521+0000","last_became_peered":"2024-09-16T15:50:30.441521+0000","last_unstale":"2024-09-16T15:50:31.427752+0000","last_undegraded":"2024-09-16T15:50:31.427752+0000","last_fullsized":"2024-09-16T15:50:31.427752+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_clean_scrub_stamp":"2024-09-16T15:50:29.373642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T22:52:44.779303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.2b","version":"0'0","reported_seq":13,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-09-16T15:50:32.118296+0000","last_change":"2024-09-16T15:50:30.422147+0000","last_active":"2024-09-16T15:50:32.118296+0000","last_peered":"2024-09-16T15:50:32.118296+0000","last_clean":"2024-09-16T15:50:32.118296+0000","last_became_active":"2024-09-16T15:50:30.421723+0000","last_became_peered":"2024-09-16T15:50:30.421723+0000","last_unstale":"2024-09-16T15:50:32.118296+0000","last_undegraded":"2024-09-16T15:50:32.118296+0000","last_fullsized":"2024-09-16T15:50:32.118296+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_clean_scrub_stamp":"2024-09-16T15:50:29.373642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T22:12:14.156255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.2a","version":"0'0","reported_seq":13,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-09-16T15:50:31.422499+0000","last_change":"2024-09-16T15:50:30.420826+0000","last_active":"2024-09-16T15:50:31.422499+0000","last_peered":"2024-09-16T15:50:31.422499+0000","last_clean":"2024-09-16T15:50:31.422499+0000","last_became_active":"2024-09-16T15:50:30.420072+0000","last_became_peered":"2024-09-16T15:50:30.420072+0000","last_unstale":"2024-09-16T15:50:31.422499+0000","last_undegraded":"2024-09-16T15:50:31.422499+0000","last_fullsized":"2024-09-16T15:50:31.422499+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_clean_scrub_stamp":"2024-09-16T15:50:29.373642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:00:53.537399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.29","version":"0'0","reported_seq":13,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-09-16T15:50:32.118150+0000","last_change":"2024-09-16T15:50:30.433635+0000","last_active":"2024-09-16T15:50:32.118150+0000","last_peered":"2024-09-16T15:50:32.118150+0000","last_clean":"2024-09-16T15:50:32.118150+0000","last_became_active":"2024-09-16T15:50:30.433251+0000","last_became_peered":"2024-09-16T15:50:30.433251+0000","last_unstale":"2024-09-16T15:50:32.118150+0000","last_undegraded":"2024-09-16T15:50:32.118150+0000","last_fullsized":"2024-09-16T15:50:32.118150+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_clean_scrub_stamp":"2024-09-16T15:50:29.373642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T01:28:31.142734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.28","version":"0'0","reported_seq":13,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-09-16T15:50:31.423167+0000","last_change":"2024-09-16T15:50:30.423158+0000","last_active":"2024-09-16T15:50:31.423167+0000","last_peered":"2024-09-16T15:50:31.423167+0000","last_clean":"2024-09-16T15:50:31.423167+0000","last_became_active":"2024-09-16T15:50:30.422877+0000","last_became_peered":"2024-09-16T15:50:30.422877+0000","last_unstale":"2024-09-16T15:50:31.423167+0000","last_undegraded":"2024-09-16T15:50:31.423167+0000","last_fullsized":"2024-09-16T15:50:31.423167+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_clean_scrub_stamp":"2024-09-16T15:50:29.373642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T00:28:57.393478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.27","version":"0'0","reported_seq":13,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-09-16T15:50:31.427512+0000","last_change":"2024-09-16T15:50:30.427329+0000","last_active":"2024-09-16T15:50:31.427512+0000","last_peered":"2024-09-16T15:50:31.427512+0000","last_clean":"2024-09-16T15:50:31.427512+0000","last_became_active":"2024-09-16T15:50:30.427083+0000","last_became_peered":"2024-09-16T15:50:30.427083+0000","last_unstale":"2024-09-16T15:50:31.427512+0000","last_undegraded":"2024-09-16T15:50:31.427512+0000","last_fullsized":"2024-09-16T15:50:31.427512+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_clean_scrub_stamp":"2024-09-16T15:50:29.373642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:37:57.461841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.26","version":"0'0","reported_seq":13,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-09-16T15:50:32.578154+0000","last_change":"2024-09-16T15:50:30.431967+0000","last_active":"2024-09-16T15:50:32.578154+0000","last_peered":"2024-09-16T15:50:32.578154+0000","last_clean":"2024-09-16T15:50:32.578154+0000","last_became_active":"2024-09-16T15:50:30.431577+0000","last_became_peered":"2024-09-16T15:50:30.431577+0000","last_unstale":"2024-09-16T15:50:32.578154+0000","last_undegraded":"2024-09-16T15:50:32.578154+0000","last_fullsized":"2024-09-16T15:50:32.578154+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_clean_scrub_stamp":"2024-09-16T15:50:29.373642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:26:17.597557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"23.25","version":"0'0","reported_seq":13,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-09-16T15:50:31.427242+0000","last_change":"2024-09-16T15:50:30.423530+0000","last_active":"2024-09-16T15:50:31.427242+0000","last_peered":"2024-09-16T15:50:31.427242+0000","last_clean":"2024-09-16T15:50:31.427242+0000","last_became_active":"2024-09-16T15:50:30.423183+0000","last_became_peered":"2024-09-16T15:50:30.423183+0000","last_unstale":"2024-09-16T15:50:31.427242+0000","last_undegraded":"2024-09-16T15:50:31.427242+0000","last_fullsized":"2024-09-16T15:50:31.427242+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_clean_scrub_stamp":"2024-09-16T15:50:29.373642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T00:01:17.578723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.24","version":"0'0","reported_seq":13,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-09-16T15:50:31.421054+0000","last_change":"2024-09-16T15:50:30.425134+0000","last_active":"2024-09-16T15:50:31.421054+0000","last_peered":"2024-09-16T15:50:31.421054+0000","last_clean":"2024-09-16T15:50:31.421054+0000","last_became_active":"2024-09-16T15:50:30.421610+0000","last_became_peered":"2024-09-16T15:50:30.421610+0000","last_unstale":"2024-09-16T15:50:31.421054+0000","last_undegraded":"2024-09-16T15:50:31.421054+0000","last_fullsized":"2024-09-16T15:50:31.421054+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_clean_scrub_stamp":"2024-09-16T15:50:29.373642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:46:46.434245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.23","version":"0'0","reported_seq":13,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-09-16T15:50:31.427118+0000","last_change":"2024-09-16T15:50:30.426780+0000","last_active":"2024-09-16T15:50:31.427118+0000","last_peered":"2024-09-16T15:50:31.427118+0000","last_clean":"2024-09-16T15:50:31.427118+0000","last_became_active":"2024-09-16T15:50:30.426398+0000","last_became_peered":"2024-09-16T15:50:30.426398+0000","last_unstale":"2024-09-16T15:50:31.427118+0000","last_undegraded":"2024-09-16T15:50:31.427118+0000","last_fullsized":"2024-09-16T15:50:31.427118+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_clean_scrub_stamp":"2024-09-16T15:50:29.373642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T23:38:32.126583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.22","version":"0'0","reported_seq":13,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-09-16T15:50:31.420668+0000","last_change":"2024-09-16T15:50:30.422377+0000","last_active":"2024-09-16T15:50:31.420668+0000","last_peered":"2024-09-16T15:50:31.420668+0000","last_clean":"2024-09-16T15:50:31.420668+0000","last_became_active":"2024-09-16T15:50:30.418522+0000","last_became_peered":"2024-09-16T15:50:30.418522+0000","last_unstale":"2024-09-16T15:50:31.420668+0000","last_undegraded":"2024-09-16T15:50:31.420668+0000","last_fullsized":"2024-09-16T15:50:31.420668+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_clean_scrub_stamp":"2024-09-16T15:50:29.373642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T02:01:29.970565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"23.21","version":"0'0","reported_seq":13,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-09-16T15:50:31.420614+0000","last_change":"2024-09-16T15:50:30.429499+0000","last_active":"2024-09-16T15:50:31.420614+0000","last_peered":"2024-09-16T15:50:31.420614+0000","last_clean":"2024-09-16T15:50:31.420614+0000","last_became_active":"2024-09-16T15:50:30.429250+0000","last_became_peered":"2024-09-16T15:50:30.429250+0000","last_unstale":"2024-09-16T15:50:31.420614+0000","last_undegraded":"2024-09-16T15:50:31.420614+0000","last_fullsized":"2024-09-16T15:50:31.420614+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_clean_scrub_stamp":"2024-09-16T15:50:29.373642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:48:04.966878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"23.20","version":"0'0","reported_seq":13,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-09-16T15:50:31.711379+0000","last_change":"2024-09-16T15:50:30.421531+0000","last_active":"2024-09-16T15:50:31.711379+0000","last_peered":"2024-09-16T15:50:31.711379+0000","last_clean":"2024-09-16T15:50:31.711379+0000","last_became_active":"2024-09-16T15:50:30.421060+0000","last_became_peered":"2024-09-16T15:50:30.421060+0000","last_unstale":"2024-09-16T15:50:31.711379+0000","last_undegraded":"2024-09-16T15:50:31.711379+0000","last_fullsized":"2024-09-16T15:50:31.711379+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_clean_scrub_stamp":"2024-09-16T15:50:29.373642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T22:58:36.525635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"23.1e","version":"0'0","reported_seq":13,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-09-16T15:50:32.578327+0000","last_change":"2024-09-16T15:50:30.429101+0000","last_active":"2024-09-16T15:50:32.578327+0000","last_peered":"2024-09-16T15:50:32.578327+0000","last_clean":"2024-09-16T15:50:32.578327+0000","last_became_active":"2024-09-16T15:50:30.428824+0000","last_became_peered":"2024-09-16T15:50:30.428824+0000","last_unstale":"2024-09-16T15:50:32.578327+0000","last_undegraded":"2024-09-16T15:50:32.578327+0000","last_fullsized":"2024-09-16T15:50:32.578327+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_clean_scrub_stamp":"2024-09-16T15:50:29.373642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T23:50:30.465958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"22.1f","version":"145'4","reported_seq":19,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-09-16T15:50:32.108812+0000","last_change":"2024-09-16T15:50:29.428405+0000","last_active":"2024-09-16T15:50:32.108812+0000","last_peered":"2024-09-16T15:50:32.108812+0000","last_clean":"2024-09-16T15:50:32.108812+0000","last_became_active":"2024-09-16T15:50:29.423819+0000","last_became_peered":"2024-09-16T15:50:29.423819+0000","last_unstale":"2024-09-16T15:50:32.108812+0000","last_undegraded":"2024-09-16T15:50:32.108812+0000","last_fullsized":"2024-09-16T15:50:32.108812+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:28.359944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:28.359944+0000","last_clean_scrub_stamp":"2024-09-16T15:50:28.359944+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:08:52.550227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.1f","version":"0'0","reported_seq":13,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-09-16T15:50:31.421893+0000","last_change":"2024-09-16T15:50:30.432456+0000","last_active":"2024-09-16T15:50:31.421893+0000","last_peered":"2024-09-16T15:50:31.421893+0000","last_clean":"2024-09-16T15:50:31.421893+0000","last_became_active":"2024-09-16T15:50:30.430570+0000","last_became_peered":"2024-09-16T15:50:30.430570+0000","last_unstale":"2024-09-16T15:50:31.421893+0000","last_undegraded":"2024-09-16T15:50:31.421893+0000","last_fullsized":"2024-09-16T15:50:31.421893+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_clean_scrub_stamp":"2024-09-16T15:50:29.373642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:47:22.979699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.1e","version":"145'1","reported_seq":16,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-09-16T15:50:31.421490+0000","last_change":"2024-09-16T15:50:29.431326+0000","last_active":"2024-09-16T15:50:31.421490+0000","last_peered":"2024-09-16T15:50:31.421490+0000","last_clean":"2024-09-16T15:50:31.421490+0000","last_became_active":"2024-09-16T15:50:29.430610+0000","last_became_peered":"2024-09-16T15:50:29.430610+0000","last_unstale":"2024-09-16T15:50:31.421490+0000","last_undegraded":"2024-09-16T15:50:31.421490+0000","last_fullsized":"2024-09-16T15:50:31.421490+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:28.359944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:28.359944+0000","last_clean_scrub_stamp":"2024-09-16T15:50:28.359944+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T03:15:34.163477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"23.1c","version":"0'0","reported_seq":13,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-09-16T15:50:32.118356+0000","last_change":"2024-09-16T15:50:30.441740+0000","last_active":"2024-09-16T15:50:32.118356+0000","last_peered":"2024-09-16T15:50:32.118356+0000","last_clean":"2024-09-16T15:50:32.118356+0000","last_became_active":"2024-09-16T15:50:30.441137+0000","last_became_peered":"2024-09-16T15:50:30.441137+0000","last_unstale":"2024-09-16T15:50:32.118356+0000","last_undegraded":"2024-09-16T15:50:32.118356+0000","last_fullsized":"2024-09-16T15:50:32.118356+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_clean_scrub_stamp":"2024-09-16T15:50:29.373642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:23:19.227212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.1d","version":"145'2","reported_seq":17,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-09-16T15:50:31.421196+0000","last_change":"2024-09-16T15:50:29.425771+0000","last_active":"2024-09-16T15:50:31.421196+0000","last_peered":"2024-09-16T15:50:31.421196+0000","last_clean":"2024-09-16T15:50:31.421196+0000","last_became_active":"2024-09-16T15:50:29.425507+0000","last_became_peered":"2024-09-16T15:50:29.425507+0000","last_unstale":"2024-09-16T15:50:31.421196+0000","last_undegraded":"2024-09-16T15:50:31.421196+0000","last_fullsized":"2024-09-16T15:50:31.421196+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:28.359944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:28.359944+0000","last_clean_scrub_stamp":"2024-09-16T15:50:28.359944+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:24:30.338402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.1d","version":"0'0","reported_seq":13,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-09-16T15:50:31.427632+0000","last_change":"2024-09-16T15:50:30.420346+0000","last_active":"2024-09-16T15:50:31.427632+0000","last_peered":"2024-09-16T15:50:31.427632+0000","last_clean":"2024-09-16T15:50:31.427632+0000","last_became_active":"2024-09-16T15:50:30.420100+0000","last_became_peered":"2024-09-16T15:50:30.420100+0000","last_unstale":"2024-09-16T15:50:31.427632+0000","last_undegraded":"2024-09-16T15:50:31.427632+0000","last_fullsized":"2024-09-16T15:50:31.427632+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_clean_scrub_stamp":"2024-09-16T15:50:29.373642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:16:33.895655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.1c","version":"145'2","reported_seq":17,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-09-16T15:50:31.421229+0000","last_change":"2024-09-16T15:50:29.408868+0000","last_active":"2024-09-16T15:50:31.421229+0000","last_peered":"2024-09-16T15:50:31.421229+0000","last_clean":"2024-09-16T15:50:31.421229+0000","last_became_active":"2024-09-16T15:50:29.408550+0000","last_became_peered":"2024-09-16T15:50:29.408550+0000","last_unstale":"2024-09-16T15:50:31.421229+0000","last_undegraded":"2024-09-16T15:50:31.421229+0000","last_fullsized":"2024-09-16T15:50:31.421229+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:28.359944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:28.359944+0000","last_clean_scrub_stamp":"2024-09-16T15:50:28.359944+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:57:45.446543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.1a","version":"0'0","reported_seq":13,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-09-16T15:50:32.577931+0000","last_change":"2024-09-16T15:50:30.431163+0000","last_active":"2024-09-16T15:50:32.577931+0000","last_peered":"2024-09-16T15:50:32.577931+0000","last_clean":"2024-09-16T15:50:32.577931+0000","last_became_active":"2024-09-16T15:50:30.430801+0000","last_became_peered":"2024-09-16T15:50:30.430801+0000","last_unstale":"2024-09-16T15:50:32.577931+0000","last_undegraded":"2024-09-16T15:50:32.577931+0000","last_fullsized":"2024-09-16T15:50:32.577931+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_clean_scrub_stamp":"2024-09-16T15:50:29.373642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:53:41.574681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"22.1b","version":"145'1","reported_seq":16,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-09-16T15:50:31.427009+0000","last_change":"2024-09-16T15:50:29.429045+0000","last_active":"2024-09-16T15:50:31.427009+0000","last_peered":"2024-09-16T15:50:31.427009+0000","last_clean":"2024-09-16T15:50:31.427009+0000","last_became_active":"2024-09-16T15:50:29.428317+0000","last_became_peered":"2024-09-16T15:50:29.428317+0000","last_unstale":"2024-09-16T15:50:31.427009+0000","last_undegraded":"2024-09-16T15:50:31.427009+0000","last_fullsized":"2024-09-16T15:50:31.427009+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:28.359944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:28.359944+0000","last_clean_scrub_stamp":"2024-09-16T15:50:28.359944+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:46:20.262191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.1b","version":"0'0","reported_seq":13,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-09-16T15:50:31.421441+0000","last_change":"2024-09-16T15:50:30.420634+0000","last_active":"2024-09-16T15:50:31.421441+0000","last_peered":"2024-09-16T15:50:31.421441+0000","last_clean":"2024-09-16T15:50:31.421441+0000","last_became_active":"2024-09-16T15:50:30.420354+0000","last_became_peered":"2024-09-16T15:50:30.420354+0000","last_unstale":"2024-09-16T15:50:31.421441+0000","last_undegraded":"2024-09-16T15:50:31.421441+0000","last_fullsized":"2024-09-16T15:50:31.421441+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_clean_scrub_stamp":"2024-09-16T15:50:29.373642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:11:22.391718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.1a","version":"145'1","reported_seq":16,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-09-16T15:50:31.420486+0000","last_change":"2024-09-16T15:50:29.408149+0000","last_active":"2024-09-16T15:50:31.420486+0000","last_peered":"2024-09-16T15:50:31.420486+0000","last_clean":"2024-09-16T15:50:31.420486+0000","last_became_active":"2024-09-16T15:50:29.407778+0000","last_became_peered":"2024-09-16T15:50:29.407778+0000","last_unstale":"2024-09-16T15:50:31.420486+0000","last_undegraded":"2024-09-16T15:50:31.420486+0000","last_fullsized":"2024-09-16T15:50:31.420486+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:28.359944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:28.359944+0000","last_clean_scrub_stamp":"2024-09-16T15:50:28.359944+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T22:29:14.955543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.18","version":"0'0","reported_seq":13,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-09-16T15:50:31.420313+0000","last_change":"2024-09-16T15:50:30.435105+0000","last_active":"2024-09-16T15:50:31.420313+0000","last_peered":"2024-09-16T15:50:31.420313+0000","last_clean":"2024-09-16T15:50:31.420313+0000","last_became_active":"2024-09-16T15:50:30.434866+0000","last_became_peered":"2024-09-16T15:50:30.434866+0000","last_unstale":"2024-09-16T15:50:31.420313+0000","last_undegraded":"2024-09-16T15:50:31.420313+0000","last_fullsized":"2024-09-16T15:50:31.420313+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_clean_scrub_stamp":"2024-09-16T15:50:29.373642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T01:36:10.500854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.19","version":"0'0","reported_seq":15,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-09-16T15:50:31.423782+0000","last_change":"2024-09-16T15:50:29.427582+0000","last_active":"2024-09-16T15:50:31.423782+0000","last_peered":"2024-09-16T15:50:31.423782+0000","last_clean":"2024-09-16T15:50:31.423782+0000","last_became_active":"2024-09-16T15:50:29.426124+0000","last_became_peered":"2024-09-16T15:50:29.426124+0000","last_unstale":"2024-09-16T15:50:31.423782+0000","last_undegraded":"2024-09-16T15:50:31.423782+0000","last_fullsized":"2024-09-16T15:50:31.423782+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:28.359944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:28.359944+0000","last_clean_scrub_stamp":"2024-09-16T15:50:28.359944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T22:23:12.530653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.19","version":"0'0","reported_seq":13,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-09-16T15:50:32.118350+0000","last_change":"2024-09-16T15:50:30.418942+0000","last_active":"2024-09-16T15:50:32.118350+0000","last_peered":"2024-09-16T15:50:32.118350+0000","last_clean":"2024-09-16T15:50:32.118350+0000","last_became_active":"2024-09-16T15:50:30.418636+0000","last_became_peered":"2024-09-16T15:50:30.418636+0000","last_unstale":"2024-09-16T15:50:32.118350+0000","last_undegraded":"2024-09-16T15:50:32.118350+0000","last_fullsized":"2024-09-16T15:50:32.118350+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_clean_scrub_stamp":"2024-09-16T15:50:29.373642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T01:52:41.276229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.18","version":"0'0","reported_seq":15,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-09-16T15:50:32.577894+0000","last_change":"2024-09-16T15:50:29.406218+0000","last_active":"2024-09-16T15:50:32.577894+0000","last_peered":"2024-09-16T15:50:32.577894+0000","last_clean":"2024-09-16T15:50:32.577894+0000","last_became_active":"2024-09-16T15:50:29.405872+0000","last_became_peered":"2024-09-16T15:50:29.405872+0000","last_unstale":"2024-09-16T15:50:32.577894+0000","last_undegraded":"2024-09-16T15:50:32.577894+0000","last_fullsized":"2024-09-16T15:50:32.577894+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:28.359944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:28.359944+0000","last_clean_scrub_stamp":"2024-09-16T15:50:28.359944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T02:38:59.050296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"23.17","version":"0'0","reported_seq":13,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-09-16T15:50:31.712155+0000","last_change":"2024-09-16T15:50:30.428692+0000","last_active":"2024-09-16T15:50:31.712155+0000","last_peered":"2024-09-16T15:50:31.712155+0000","last_clean":"2024-09-16T15:50:31.712155+0000","last_became_active":"2024-09-16T15:50:30.427697+0000","last_became_peered":"2024-09-16T15:50:30.427697+0000","last_unstale":"2024-09-16T15:50:31.712155+0000","last_undegraded":"2024-09-16T15:50:31.712155+0000","last_fullsized":"2024-09-16T15:50:31.712155+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_clean_scrub_stamp":"2024-09-16T15:50:29.373642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T23:58:05.626535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.16","version":"0'0","reported_seq":15,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-09-16T15:50:32.118548+0000","last_change":"2024-09-16T15:50:29.418819+0000","last_active":"2024-09-16T15:50:32.118548+0000","last_peered":"2024-09-16T15:50:32.118548+0000","last_clean":"2024-09-16T15:50:32.118548+0000","last_became_active":"2024-09-16T15:50:29.418365+0000","last_became_peered":"2024-09-16T15:50:29.418365+0000","last_unstale":"2024-09-16T15:50:32.118548+0000","last_undegraded":"2024-09-16T15:50:32.118548+0000","last_fullsized":"2024-09-16T15:50:32.118548+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:28.359944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:28.359944+0000","last_clean_scrub_stamp":"2024-09-16T15:50:28.359944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T18:55:59.203699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.14","version":"0'0","reported_seq":13,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-09-16T15:50:31.711963+0000","last_change":"2024-09-16T15:50:30.428073+0000","last_active":"2024-09-16T15:50:31.711963+0000","last_peered":"2024-09-16T15:50:31.711963+0000","last_clean":"2024-09-16T15:50:31.711963+0000","last_became_active":"2024-09-16T15:50:30.427379+0000","last_became_peered":"2024-09-16T15:50:30.427379+0000","last_unstale":"2024-09-16T15:50:31.711963+0000","last_undegraded":"2024-09-16T15:50:31.711963+0000","last_fullsized":"2024-09-16T15:50:31.711963+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_clean_scrub_stamp":"2024-09-16T15:50:29.373642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T00:01:56.799532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.15","version":"145'3","reported_seq":18,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-09-16T15:50:32.118845+0000","last_change":"2024-09-16T15:50:29.414494+0000","last_active":"2024-09-16T15:50:32.118845+0000","last_peered":"2024-09-16T15:50:32.118845+0000","last_clean":"2024-09-16T15:50:32.118845+0000","last_became_active":"2024-09-16T15:50:29.413903+0000","last_became_peered":"2024-09-16T15:50:29.413903+0000","last_unstale":"2024-09-16T15:50:32.118845+0000","last_undegraded":"2024-09-16T15:50:32.118845+0000","last_fullsized":"2024-09-16T15:50:32.118845+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:28.359944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:28.359944+0000","last_clean_scrub_stamp":"2024-09-16T15:50:28.359944+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:17:19.061895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.15","version":"0'0","reported_seq":13,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-09-16T15:50:31.421810+0000","last_change":"2024-09-16T15:50:30.422941+0000","last_active":"2024-09-16T15:50:31.421810+0000","last_peered":"2024-09-16T15:50:31.421810+0000","last_clean":"2024-09-16T15:50:31.421810+0000","last_became_active":"2024-09-16T15:50:30.422640+0000","last_became_peered":"2024-09-16T15:50:30.422640+0000","last_unstale":"2024-09-16T15:50:31.421810+0000","last_undegraded":"2024-09-16T15:50:31.421810+0000","last_fullsized":"2024-09-16T15:50:31.421810+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_clean_scrub_stamp":"2024-09-16T15:50:29.373642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:37:04.545451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.14","version":"145'6","reported_seq":21,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-09-16T15:50:31.421315+0000","last_change":"2024-09-16T15:50:29.406502+0000","last_active":"2024-09-16T15:50:31.421315+0000","last_peered":"2024-09-16T15:50:31.421315+0000","last_clean":"2024-09-16T15:50:31.421315+0000","last_became_active":"2024-09-16T15:50:29.404490+0000","last_became_peered":"2024-09-16T15:50:29.404490+0000","last_unstale":"2024-09-16T15:50:31.421315+0000","last_undegraded":"2024-09-16T15:50:31.421315+0000","last_fullsized":"2024-09-16T15:50:31.421315+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:28.359944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:28.359944+0000","last_clean_scrub_stamp":"2024-09-16T15:50:28.359944+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:44:56.750085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"23.12","version":"0'0","reported_seq":13,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-09-16T15:50:31.711870+0000","last_change":"2024-09-16T15:50:30.421787+0000","last_active":"2024-09-16T15:50:31.711870+0000","last_peered":"2024-09-16T15:50:31.711870+0000","last_clean":"2024-09-16T15:50:31.711870+0000","last_became_active":"2024-09-16T15:50:30.421302+0000","last_became_peered":"2024-09-16T15:50:30.421302+0000","last_unstale":"2024-09-16T15:50:31.711870+0000","last_undegraded":"2024-09-16T15:50:31.711870+0000","last_fullsized":"2024-09-16T15:50:31.711870+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_clean_scrub_stamp":"2024-09-16T15:50:29.373642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T23:59:34.139432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.13","version":"145'2","reported_seq":17,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-09-16T15:50:31.420710+0000","last_change":"2024-09-16T15:50:29.408013+0000","last_active":"2024-09-16T15:50:31.420710+0000","last_peered":"2024-09-16T15:50:31.420710+0000","last_clean":"2024-09-16T15:50:31.420710+0000","last_became_active":"2024-09-16T15:50:29.407703+0000","last_became_peered":"2024-09-16T15:50:29.407703+0000","last_unstale":"2024-09-16T15:50:31.420710+0000","last_undegraded":"2024-09-16T15:50:31.420710+0000","last_fullsized":"2024-09-16T15:50:31.420710+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:28.359944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:28.359944+0000","last_clean_scrub_stamp":"2024-09-16T15:50:28.359944+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T01:59:12.540770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.13","version":"0'0","reported_seq":13,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-09-16T15:50:32.578374+0000","last_change":"2024-09-16T15:50:30.430250+0000","last_active":"2024-09-16T15:50:32.578374+0000","last_peered":"2024-09-16T15:50:32.578374+0000","last_clean":"2024-09-16T15:50:32.578374+0000","last_became_active":"2024-09-16T15:50:30.429506+0000","last_became_peered":"2024-09-16T15:50:30.429506+0000","last_unstale":"2024-09-16T15:50:32.578374+0000","last_undegraded":"2024-09-16T15:50:32.578374+0000","last_fullsized":"2024-09-16T15:50:32.578374+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_clean_scrub_stamp":"2024-09-16T15:50:29.373642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T01:50:01.172772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"22.12","version":"0'0","reported_seq":15,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-09-16T15:50:31.712484+0000","last_change":"2024-09-16T15:50:29.410142+0000","last_active":"2024-09-16T15:50:31.712484+0000","last_peered":"2024-09-16T15:50:31.712484+0000","last_clean":"2024-09-16T15:50:31.712484+0000","last_became_active":"2024-09-16T15:50:29.409779+0000","last_became_peered":"2024-09-16T15:50:29.409779+0000","last_unstale":"2024-09-16T15:50:31.712484+0000","last_undegraded":"2024-09-16T15:50:31.712484+0000","last_fullsized":"2024-09-16T15:50:31.712484+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:28.359944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:28.359944+0000","last_clean_scrub_stamp":"2024-09-16T15:50:28.359944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:18:05.631146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"23.10","version":"0'0","reported_seq":13,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-09-16T15:50:31.420370+0000","last_change":"2024-09-16T15:50:30.410405+0000","last_active":"2024-09-16T15:50:31.420370+0000","last_peered":"2024-09-16T15:50:31.420370+0000","last_clean":"2024-09-16T15:50:31.420370+0000","last_became_active":"2024-09-16T15:50:30.409675+0000","last_became_peered":"2024-09-16T15:50:30.409675+0000","last_unstale":"2024-09-16T15:50:31.420370+0000","last_undegraded":"2024-09-16T15:50:31.420370+0000","last_fullsized":"2024-09-16T15:50:31.420370+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_clean_scrub_stamp":"2024-09-16T15:50:29.373642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T03:00:15.840278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.11","version":"0'0","reported_seq":15,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-09-16T15:50:31.420250+0000","last_change":"2024-09-16T15:50:29.425647+0000","last_active":"2024-09-16T15:50:31.420250+0000","last_peered":"2024-09-16T15:50:31.420250+0000","last_clean":"2024-09-16T15:50:31.420250+0000","last_became_active":"2024-09-16T15:50:29.425266+0000","last_became_peered":"2024-09-16T15:50:29.425266+0000","last_unstale":"2024-09-16T15:50:31.420250+0000","last_undegraded":"2024-09-16T15:50:31.420250+0000","last_fullsized":"2024-09-16T15:50:31.420250+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:28.359944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:28.359944+0000","last_clean_scrub_stamp":"2024-09-16T15:50:28.359944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T23:17:20.472543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.11","version":"0'0","reported_seq":13,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-09-16T15:50:31.426563+0000","last_change":"2024-09-16T15:50:30.416966+0000","last_active":"2024-09-16T15:50:31.426563+0000","last_peered":"2024-09-16T15:50:31.426563+0000","last_clean":"2024-09-16T15:50:31.426563+0000","last_became_active":"2024-09-16T15:50:30.416672+0000","last_became_peered":"2024-09-16T15:50:30.416672+0000","last_unstale":"2024-09-16T15:50:31.426563+0000","last_undegraded":"2024-09-16T15:50:31.426563+0000","last_fullsized":"2024-09-16T15:50:31.426563+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_clean_scrub_stamp":"2024-09-16T15:50:29.373642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T23:37:14.673774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.10","version":"145'8","reported_seq":23,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-09-16T15:50:31.420335+0000","last_change":"2024-09-16T15:50:29.408571+0000","last_active":"2024-09-16T15:50:31.420335+0000","last_peered":"2024-09-16T15:50:31.420335+0000","last_clean":"2024-09-16T15:50:31.420335+0000","last_became_active":"2024-09-16T15:50:29.407840+0000","last_became_peered":"2024-09-16T15:50:29.407840+0000","last_unstale":"2024-09-16T15:50:31.420335+0000","last_undegraded":"2024-09-16T15:50:31.420335+0000","last_fullsized":"2024-09-16T15:50:31.420335+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:28.359944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:28.359944+0000","last_clean_scrub_stamp":"2024-09-16T15:50:28.359944+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T03:20:17.045245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"23.e","version":"0'0","reported_seq":13,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-09-16T15:50:31.422543+0000","last_change":"2024-09-16T15:50:30.416130+0000","last_active":"2024-09-16T15:50:31.422543+0000","last_peered":"2024-09-16T15:50:31.422543+0000","last_clean":"2024-09-16T15:50:31.422543+0000","last_became_active":"2024-09-16T15:50:30.411687+0000","last_became_peered":"2024-09-16T15:50:30.411687+0000","last_unstale":"2024-09-16T15:50:31.422543+0000","last_undegraded":"2024-09-16T15:50:31.422543+0000","last_fullsized":"2024-09-16T15:50:31.422543+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_clean_scrub_stamp":"2024-09-16T15:50:29.373642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T01:19:19.348945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.f","version":"145'1","reported_seq":16,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-09-16T15:50:31.427351+0000","last_change":"2024-09-16T15:50:29.434785+0000","last_active":"2024-09-16T15:50:31.427351+0000","last_peered":"2024-09-16T15:50:31.427351+0000","last_clean":"2024-09-16T15:50:31.427351+0000","last_became_active":"2024-09-16T15:50:29.434102+0000","last_became_peered":"2024-09-16T15:50:29.434102+0000","last_unstale":"2024-09-16T15:50:31.427351+0000","last_undegraded":"2024-09-16T15:50:31.427351+0000","last_fullsized":"2024-09-16T15:50:31.427351+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:28.359944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:28.359944+0000","last_clean_scrub_stamp":"2024-09-16T15:50:28.359944+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:11:55.526977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.f","version":"0'0","reported_seq":13,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-09-16T15:50:31.712405+0000","last_change":"2024-09-16T15:50:30.430840+0000","last_active":"2024-09-16T15:50:31.712405+0000","last_peered":"2024-09-16T15:50:31.712405+0000","last_clean":"2024-09-16T15:50:31.712405+0000","last_became_active":"2024-09-16T15:50:30.430465+0000","last_became_peered":"2024-09-16T15:50:30.430465+0000","last_unstale":"2024-09-16T15:50:31.712405+0000","last_undegraded":"2024-09-16T15:50:31.712405+0000","last_fullsized":"2024-09-16T15:50:31.712405+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_clean_scrub_stamp":"2024-09-16T15:50:29.373642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T22:27:20.876644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.e","version":"145'2","reported_seq":17,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-09-16T15:50:31.427191+0000","last_change":"2024-09-16T15:50:29.426483+0000","last_active":"2024-09-16T15:50:31.427191+0000","last_peered":"2024-09-16T15:50:31.427191+0000","last_clean":"2024-09-16T15:50:31.427191+0000","last_became_active":"2024-09-16T15:50:29.425712+0000","last_became_peered":"2024-09-16T15:50:29.425712+0000","last_unstale":"2024-09-16T15:50:31.427191+0000","last_undegraded":"2024-09-16T15:50:31.427191+0000","last_fullsized":"2024-09-16T15:50:31.427191+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:28.359944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:28.359944+0000","last_clean_scrub_stamp":"2024-09-16T15:50:28.359944+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:24:14.974365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.c","version":"0'0","reported_seq":13,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-09-16T15:50:31.421066+0000","last_change":"2024-09-16T15:50:30.422935+0000","last_active":"2024-09-16T15:50:31.421066+0000","last_peered":"2024-09-16T15:50:31.421066+0000","last_clean":"2024-09-16T15:50:31.421066+0000","last_became_active":"2024-09-16T15:50:30.422403+0000","last_became_peered":"2024-09-16T15:50:30.422403+0000","last_unstale":"2024-09-16T15:50:31.421066+0000","last_undegraded":"2024-09-16T15:50:31.421066+0000","last_fullsized":"2024-09-16T15:50:31.421066+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_clean_scrub_stamp":"2024-09-16T15:50:29.373642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T00:17:11.137582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.d","version":"145'1","reported_seq":16,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-09-16T15:50:32.108834+0000","last_change":"2024-09-16T15:50:29.427670+0000","last_active":"2024-09-16T15:50:32.108834+0000","last_peered":"2024-09-16T15:50:32.108834+0000","last_clean":"2024-09-16T15:50:32.108834+0000","last_became_active":"2024-09-16T15:50:29.426920+0000","last_became_peered":"2024-09-16T15:50:29.426920+0000","last_unstale":"2024-09-16T15:50:32.108834+0000","last_undegraded":"2024-09-16T15:50:32.108834+0000","last_fullsized":"2024-09-16T15:50:32.108834+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:28.359944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:28.359944+0000","last_clean_scrub_stamp":"2024-09-16T15:50:28.359944+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:16:26.927855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.d","version":"0'0","reported_seq":13,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-09-16T15:50:32.578048+0000","last_change":"2024-09-16T15:50:30.420386+0000","last_active":"2024-09-16T15:50:32.578048+0000","last_peered":"2024-09-16T15:50:32.578048+0000","last_clean":"2024-09-16T15:50:32.578048+0000","last_became_active":"2024-09-16T15:50:30.419835+0000","last_became_peered":"2024-09-16T15:50:30.419835+0000","last_unstale":"2024-09-16T15:50:32.578048+0000","last_undegraded":"2024-09-16T15:50:32.578048+0000","last_fullsized":"2024-09-16T15:50:32.578048+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_clean_scrub_stamp":"2024-09-16T15:50:29.373642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:54:03.217773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"22.c","version":"0'0","reported_seq":15,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-09-16T15:50:32.109094+0000","last_change":"2024-09-16T15:50:29.427240+0000","last_active":"2024-09-16T15:50:32.109094+0000","last_peered":"2024-09-16T15:50:32.109094+0000","last_clean":"2024-09-16T15:50:32.109094+0000","last_became_active":"2024-09-16T15:50:29.423280+0000","last_became_peered":"2024-09-16T15:50:29.423280+0000","last_unstale":"2024-09-16T15:50:32.109094+0000","last_undegraded":"2024-09-16T15:50:32.109094+0000","last_fullsized":"2024-09-16T15:50:32.109094+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:28.359944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:28.359944+0000","last_clean_scrub_stamp":"2024-09-16T15:50:28.359944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:58:21.241616+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.a","version":"0'0","reported_seq":13,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-09-16T15:50:31.426833+0000","last_change":"2024-09-16T15:50:30.431357+0000","last_active":"2024-09-16T15:50:31.426833+0000","last_peered":"2024-09-16T15:50:31.426833+0000","last_clean":"2024-09-16T15:50:31.426833+0000","last_became_active":"2024-09-16T15:50:30.430742+0000","last_became_peered":"2024-09-16T15:50:30.430742+0000","last_unstale":"2024-09-16T15:50:31.426833+0000","last_undegraded":"2024-09-16T15:50:31.426833+0000","last_fullsized":"2024-09-16T15:50:31.426833+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_clean_scrub_stamp":"2024-09-16T15:50:29.373642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:51.358615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.b","version":"145'2","reported_seq":17,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-09-16T15:50:31.420978+0000","last_change":"2024-09-16T15:50:29.425798+0000","last_active":"2024-09-16T15:50:31.420978+0000","last_peered":"2024-09-16T15:50:31.420978+0000","last_clean":"2024-09-16T15:50:31.420978+0000","last_became_active":"2024-09-16T15:50:29.425459+0000","last_became_peered":"2024-09-16T15:50:29.425459+0000","last_unstale":"2024-09-16T15:50:31.420978+0000","last_undegraded":"2024-09-16T15:50:31.420978+0000","last_fullsized":"2024-09-16T15:50:31.420978+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:28.359944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:28.359944+0000","last_clean_scrub_stamp":"2024-09-16T15:50:28.359944+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:53:09.727133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.b","version":"0'0","reported_seq":13,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-09-16T15:50:32.578481+0000","last_change":"2024-09-16T15:50:30.439064+0000","last_active":"2024-09-16T15:50:32.578481+0000","last_peered":"2024-09-16T15:50:32.578481+0000","last_clean":"2024-09-16T15:50:32.578481+0000","last_became_active":"2024-09-16T15:50:30.438780+0000","last_became_peered":"2024-09-16T15:50:30.438780+0000","last_unstale":"2024-09-16T15:50:32.578481+0000","last_undegraded":"2024-09-16T15:50:32.578481+0000","last_fullsized":"2024-09-16T15:50:32.578481+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_clean_scrub_stamp":"2024-09-16T15:50:29.373642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T18:03:11.433879+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"22.a","version":"0'0","reported_seq":15,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-09-16T15:50:32.108248+0000","last_change":"2024-09-16T15:50:29.426752+0000","last_active":"2024-09-16T15:50:32.108248+0000","last_peered":"2024-09-16T15:50:32.108248+0000","last_clean":"2024-09-16T15:50:32.108248+0000","last_became_active":"2024-09-16T15:50:29.422755+0000","last_became_peered":"2024-09-16T15:50:29.422755+0000","last_unstale":"2024-09-16T15:50:32.108248+0000","last_undegraded":"2024-09-16T15:50:32.108248+0000","last_fullsized":"2024-09-16T15:50:32.108248+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:28.359944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:28.359944+0000","last_clean_scrub_stamp":"2024-09-16T15:50:28.359944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T22:21:22.794435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.8","version":"0'0","reported_seq":13,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-09-16T15:50:32.108290+0000","last_change":"2024-09-16T15:50:30.439554+0000","last_active":"2024-09-16T15:50:32.108290+0000","last_peered":"2024-09-16T15:50:32.108290+0000","last_clean":"2024-09-16T15:50:32.108290+0000","last_became_active":"2024-09-16T15:50:30.439106+0000","last_became_peered":"2024-09-16T15:50:30.439106+0000","last_unstale":"2024-09-16T15:50:32.108290+0000","last_undegraded":"2024-09-16T15:50:32.108290+0000","last_fullsized":"2024-09-16T15:50:32.108290+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_clean_scrub_stamp":"2024-09-16T15:50:29.373642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T02:33:45.139752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.9","version":"0'0","reported_seq":15,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-09-16T15:50:31.420352+0000","last_change":"2024-09-16T15:50:29.423325+0000","last_active":"2024-09-16T15:50:31.420352+0000","last_peered":"2024-09-16T15:50:31.420352+0000","last_clean":"2024-09-16T15:50:31.420352+0000","last_became_active":"2024-09-16T15:50:29.423053+0000","last_became_peered":"2024-09-16T15:50:29.423053+0000","last_unstale":"2024-09-16T15:50:31.420352+0000","last_undegraded":"2024-09-16T15:50:31.420352+0000","last_fullsized":"2024-09-16T15:50:31.420352+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:28.359944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:28.359944+0000","last_clean_scrub_stamp":"2024-09-16T15:50:28.359944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T02:23:42.181119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.9","version":"0'0","reported_seq":13,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-09-16T15:50:32.118940+0000","last_change":"2024-09-16T15:50:30.432926+0000","last_active":"2024-09-16T15:50:32.118940+0000","last_peered":"2024-09-16T15:50:32.118940+0000","last_clean":"2024-09-16T15:50:32.118940+0000","last_became_active":"2024-09-16T15:50:30.432592+0000","last_became_peered":"2024-09-16T15:50:30.432592+0000","last_unstale":"2024-09-16T15:50:32.118940+0000","last_undegraded":"2024-09-16T15:50:32.118940+0000","last_fullsized":"2024-09-16T15:50:32.118940+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_clean_scrub_stamp":"2024-09-16T15:50:29.373642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T03:02:11.063591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.8","version":"0'0","reported_seq":15,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-09-16T15:50:32.108589+0000","last_change":"2024-09-16T15:50:29.418765+0000","last_active":"2024-09-16T15:50:32.108589+0000","last_peered":"2024-09-16T15:50:32.108589+0000","last_clean":"2024-09-16T15:50:32.108589+0000","last_became_active":"2024-09-16T15:50:29.417737+0000","last_became_peered":"2024-09-16T15:50:29.417737+0000","last_unstale":"2024-09-16T15:50:32.108589+0000","last_undegraded":"2024-09-16T15:50:32.108589+0000","last_fullsized":"2024-09-16T15:50:32.108589+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:28.359944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:28.359944+0000","last_clean_scrub_stamp":"2024-09-16T15:50:28.359944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T00:35:20.473412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.6","version":"0'0","reported_seq":13,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-09-16T15:50:31.712748+0000","last_change":"2024-09-16T15:50:30.430405+0000","last_active":"2024-09-16T15:50:31.712748+0000","last_peered":"2024-09-16T15:50:31.712748+0000","last_clean":"2024-09-16T15:50:31.712748+0000","last_became_active":"2024-09-16T15:50:30.428381+0000","last_became_peered":"2024-09-16T15:50:30.428381+0000","last_unstale":"2024-09-16T15:50:31.712748+0000","last_undegraded":"2024-09-16T15:50:31.712748+0000","last_fullsized":"2024-09-16T15:50:31.712748+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_clean_scrub_stamp":"2024-09-16T15:50:29.373642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:28:31.939476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.7","version":"145'2","reported_seq":17,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-09-16T15:50:31.712563+0000","last_change":"2024-09-16T15:50:29.422588+0000","last_active":"2024-09-16T15:50:31.712563+0000","last_peered":"2024-09-16T15:50:31.712563+0000","last_clean":"2024-09-16T15:50:31.712563+0000","last_became_active":"2024-09-16T15:50:29.422240+0000","last_became_peered":"2024-09-16T15:50:29.422240+0000","last_unstale":"2024-09-16T15:50:31.712563+0000","last_undegraded":"2024-09-16T15:50:31.712563+0000","last_fullsized":"2024-09-16T15:50:31.712563+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:28.359944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:28.359944+0000","last_clean_scrub_stamp":"2024-09-16T15:50:28.359944+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T01:13:16.860020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"23.7","version":"0'0","reported_seq":13,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-09-16T15:50:32.578390+0000","last_change":"2024-09-16T15:50:30.434311+0000","last_active":"2024-09-16T15:50:32.578390+0000","last_peered":"2024-09-16T15:50:32.578390+0000","last_clean":"2024-09-16T15:50:32.578390+0000","last_became_active":"2024-09-16T15:50:30.433949+0000","last_became_peered":"2024-09-16T15:50:30.433949+0000","last_unstale":"2024-09-16T15:50:32.578390+0000","last_undegraded":"2024-09-16T15:50:32.578390+0000","last_fullsized":"2024-09-16T15:50:32.578390+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_clean_scrub_stamp":"2024-09-16T15:50:29.373642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:08:53.563925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"22.6","version":"146'1","reported_seq":16,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-09-16T15:50:31.427589+0000","last_change":"2024-09-16T15:50:29.424687+0000","last_active":"2024-09-16T15:50:31.427589+0000","last_peered":"2024-09-16T15:50:31.427589+0000","last_clean":"2024-09-16T15:50:31.427589+0000","last_became_active":"2024-09-16T15:50:29.423323+0000","last_became_peered":"2024-09-16T15:50:29.423323+0000","last_unstale":"2024-09-16T15:50:31.427589+0000","last_undegraded":"2024-09-16T15:50:31.427589+0000","last_fullsized":"2024-09-16T15:50:31.427589+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:28.359944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:28.359944+0000","last_clean_scrub_stamp":"2024-09-16T15:50:28.359944+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T01:30:34.197202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.4","version":"0'0","reported_seq":13,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-09-16T15:50:32.119019+0000","last_change":"2024-09-16T15:50:30.422831+0000","last_active":"2024-09-16T15:50:32.119019+0000","last_peered":"2024-09-16T15:50:32.119019+0000","last_clean":"2024-09-16T15:50:32.119019+0000","last_became_active":"2024-09-16T15:50:30.422260+0000","last_became_peered":"2024-09-16T15:50:30.422260+0000","last_unstale":"2024-09-16T15:50:32.119019+0000","last_undegraded":"2024-09-16T15:50:32.119019+0000","last_fullsized":"2024-09-16T15:50:32.119019+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_clean_scrub_stamp":"2024-09-16T15:50:29.373642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T23:35:14.888868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.5","version":"0'0","reported_seq":15,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-09-16T15:50:31.423502+0000","last_change":"2024-09-16T15:50:29.421716+0000","last_active":"2024-09-16T15:50:31.423502+0000","last_peered":"2024-09-16T15:50:31.423502+0000","last_clean":"2024-09-16T15:50:31.423502+0000","last_became_active":"2024-09-16T15:50:29.420529+0000","last_became_peered":"2024-09-16T15:50:29.420529+0000","last_unstale":"2024-09-16T15:50:31.423502+0000","last_undegraded":"2024-09-16T15:50:31.423502+0000","last_fullsized":"2024-09-16T15:50:31.423502+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:28.359944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:28.359944+0000","last_clean_scrub_stamp":"2024-09-16T15:50:28.359944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T01:09:02.529997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.5","version":"0'0","reported_seq":13,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-09-16T15:50:31.712147+0000","last_change":"2024-09-16T15:50:30.428112+0000","last_active":"2024-09-16T15:50:31.712147+0000","last_peered":"2024-09-16T15:50:31.712147+0000","last_clean":"2024-09-16T15:50:31.712147+0000","last_became_active":"2024-09-16T15:50:30.427549+0000","last_became_peered":"2024-09-16T15:50:30.427549+0000","last_unstale":"2024-09-16T15:50:31.712147+0000","last_undegraded":"2024-09-16T15:50:31.712147+0000","last_fullsized":"2024-09-16T15:50:31.712147+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_clean_scrub_stamp":"2024-09-16T15:50:29.373642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:15:00.250920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.4","version":"0'0","reported_seq":15,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-09-16T15:50:31.711865+0000","last_change":"2024-09-16T15:50:29.417210+0000","last_active":"2024-09-16T15:50:31.711865+0000","last_peered":"2024-09-16T15:50:31.711865+0000","last_clean":"2024-09-16T15:50:31.711865+0000","last_became_active":"2024-09-16T15:50:29.416831+0000","last_became_peered":"2024-09-16T15:50:29.416831+0000","last_unstale":"2024-09-16T15:50:31.711865+0000","last_undegraded":"2024-09-16T15:50:31.711865+0000","last_fullsized":"2024-09-16T15:50:31.711865+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:28.359944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:28.359944+0000","last_clean_scrub_stamp":"2024-09-16T15:50:28.359944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:41:40.755726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"23.2","version":"0'0","reported_seq":13,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-09-16T15:50:31.712128+0000","last_change":"2024-09-16T15:50:30.420699+0000","last_active":"2024-09-16T15:50:31.712128+0000","last_peered":"2024-09-16T15:50:31.712128+0000","last_clean":"2024-09-16T15:50:31.712128+0000","last_became_active":"2024-09-16T15:50:30.420194+0000","last_became_peered":"2024-09-16T15:50:30.420194+0000","last_unstale":"2024-09-16T15:50:31.712128+0000","last_undegraded":"2024-09-16T15:50:31.712128+0000","last_fullsized":"2024-09-16T15:50:31.712128+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_clean_scrub_stamp":"2024-09-16T15:50:29.373642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T18:54:12.256787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.3","version":"0'0","reported_seq":15,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-09-16T15:50:31.421342+0000","last_change":"2024-09-16T15:50:29.407104+0000","last_active":"2024-09-16T15:50:31.421342+0000","last_peered":"2024-09-16T15:50:31.421342+0000","last_clean":"2024-09-16T15:50:31.421342+0000","last_became_active":"2024-09-16T15:50:29.406182+0000","last_became_peered":"2024-09-16T15:50:29.406182+0000","last_unstale":"2024-09-16T15:50:31.421342+0000","last_undegraded":"2024-09-16T15:50:31.421342+0000","last_fullsized":"2024-09-16T15:50:31.421342+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:28.359944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:28.359944+0000","last_clean_scrub_stamp":"2024-09-16T15:50:28.359944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:38:17.200230+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"23.3","version":"0'0","reported_seq":13,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-09-16T15:50:31.420953+0000","last_change":"2024-09-16T15:50:30.429649+0000","last_active":"2024-09-16T15:50:31.420953+0000","last_peered":"2024-09-16T15:50:31.420953+0000","last_clean":"2024-09-16T15:50:31.420953+0000","last_became_active":"2024-09-16T15:50:30.429327+0000","last_became_peered":"2024-09-16T15:50:30.429327+0000","last_unstale":"2024-09-16T15:50:31.420953+0000","last_undegraded":"2024-09-16T15:50:31.420953+0000","last_fullsized":"2024-09-16T15:50:31.420953+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_clean_scrub_stamp":"2024-09-16T15:50:29.373642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T02:39:08.299476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.2","version":"0'0","reported_seq":15,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-09-16T15:50:31.420613+0000","last_change":"2024-09-16T15:50:29.410821+0000","last_active":"2024-09-16T15:50:31.420613+0000","last_peered":"2024-09-16T15:50:31.420613+0000","last_clean":"2024-09-16T15:50:31.420613+0000","last_became_active":"2024-09-16T15:50:29.410554+0000","last_became_peered":"2024-09-16T15:50:29.410554+0000","last_unstale":"2024-09-16T15:50:31.420613+0000","last_undegraded":"2024-09-16T15:50:31.420613+0000","last_fullsized":"2024-09-16T15:50:31.420613+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:28.359944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:28.359944+0000","last_clean_scrub_stamp":"2024-09-16T15:50:28.359944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:00:24.756496+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.0","version":"0'0","reported_seq":13,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-09-16T15:50:32.577768+0000","last_change":"2024-09-16T15:50:30.442321+0000","last_active":"2024-09-16T15:50:32.577768+0000","last_peered":"2024-09-16T15:50:32.577768+0000","last_clean":"2024-09-16T15:50:32.577768+0000","last_became_active":"2024-09-16T15:50:30.441963+0000","last_became_peered":"2024-09-16T15:50:30.441963+0000","last_unstale":"2024-09-16T15:50:32.577768+0000","last_undegraded":"2024-09-16T15:50:32.577768+0000","last_fullsized":"2024-09-16T15:50:32.577768+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_clean_scrub_stamp":"2024-09-16T15:50:29.373642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:48:00.744274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"22.1","version":"0'0","reported_seq":15,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-09-16T15:50:31.424120+0000","last_change":"2024-09-16T15:50:29.416751+0000","last_active":"2024-09-16T15:50:31.424120+0000","last_peered":"2024-09-16T15:50:31.424120+0000","last_clean":"2024-09-16T15:50:31.424120+0000","last_became_active":"2024-09-16T15:50:29.415995+0000","last_became_peered":"2024-09-16T15:50:29.415995+0000","last_unstale":"2024-09-16T15:50:31.424120+0000","last_undegraded":"2024-09-16T15:50:31.424120+0000","last_fullsized":"2024-09-16T15:50:31.424120+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:28.359944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:28.359944+0000","last_clean_scrub_stamp":"2024-09-16T15:50:28.359944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:18:24.821023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.1","version":"0'0","reported_seq":13,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-09-16T15:50:31.426640+0000","last_change":"2024-09-16T15:50:30.409553+0000","last_active":"2024-09-16T15:50:31.426640+0000","last_peered":"2024-09-16T15:50:31.426640+0000","last_clean":"2024-09-16T15:50:31.426640+0000","last_became_active":"2024-09-16T15:50:30.408870+0000","last_became_peered":"2024-09-16T15:50:30.408870+0000","last_unstale":"2024-09-16T15:50:31.426640+0000","last_undegraded":"2024-09-16T15:50:31.426640+0000","last_fullsized":"2024-09-16T15:50:31.426640+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_clean_scrub_stamp":"2024-09-16T15:50:29.373642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T02:32:44.791900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.0","version":"0'0","reported_seq":15,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-09-16T15:50:31.420501+0000","last_change":"2024-09-16T15:50:29.395361+0000","last_active":"2024-09-16T15:50:31.420501+0000","last_peered":"2024-09-16T15:50:31.420501+0000","last_clean":"2024-09-16T15:50:31.420501+0000","last_became_active":"2024-09-16T15:50:29.394578+0000","last_became_peered":"2024-09-16T15:50:29.394578+0000","last_unstale":"2024-09-16T15:50:31.420501+0000","last_undegraded":"2024-09-16T15:50:31.420501+0000","last_fullsized":"2024-09-16T15:50:31.420501+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:28.359944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:28.359944+0000","last_clean_scrub_stamp":"2024-09-16T15:50:28.359944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:58:09.004093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"23.16","version":"0'0","reported_seq":13,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-09-16T15:50:31.712679+0000","last_change":"2024-09-16T15:50:30.428674+0000","last_active":"2024-09-16T15:50:31.712679+0000","last_peered":"2024-09-16T15:50:31.712679+0000","last_clean":"2024-09-16T15:50:31.712679+0000","last_became_active":"2024-09-16T15:50:30.427544+0000","last_became_peered":"2024-09-16T15:50:30.427544+0000","last_unstale":"2024-09-16T15:50:31.712679+0000","last_undegraded":"2024-09-16T15:50:31.712679+0000","last_fullsized":"2024-09-16T15:50:31.712679+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:29.373642+0000","last_clean_scrub_stamp":"2024-09-16T15:50:29.373642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T02:43:22.615920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.17","version":"0'0","reported_seq":15,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-09-16T15:50:31.421873+0000","last_change":"2024-09-16T15:50:29.410992+0000","last_active":"2024-09-16T15:50:31.421873+0000","last_peered":"2024-09-16T15:50:31.421873+0000","last_clean":"2024-09-16T15:50:31.421873+0000","last_became_active":"2024-09-16T15:50:29.410215+0000","last_became_peered":"2024-09-16T15:50:29.410215+0000","last_unstale":"2024-09-16T15:50:31.421873+0000","last_undegraded":"2024-09-16T15:50:31.421873+0000","last_fullsized":"2024-09-16T15:50:31.421873+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:50:28.359944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:50:28.359944+0000","last_clean_scrub_stamp":"2024-09-16T15:50:28.359944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T18:46:56.837077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"73'79","reported_seq":370,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-09-16T15:50:32.108765+0000","last_change":"2024-09-16T15:42:40.881820+0000","last_active":"2024-09-16T15:50:32.108765+0000","last_peered":"2024-09-16T15:50:32.108765+0000","last_clean":"2024-09-16T15:50:32.108765+0000","last_became_active":"2024-09-16T15:42:40.879965+0000","last_became_peered":"2024-09-16T15:42:40.879965+0000","last_unstale":"2024-09-16T15:50:32.108765+0000","last_undegraded":"2024-09-16T15:50:32.108765+0000","last_fullsized":"2024-09-16T15:50:32.108765+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:42:39.728762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:42:39.728762+0000","last_clean_scrub_stamp":"2024-09-16T15:42:39.728762+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:22:18.768932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":23,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":22,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79,"ondisk_log_size":79,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542243,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33580,"kb_used_data":2392,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338260,"statfs":{"total":96636764160,"available":96602378240,"internally_reserved":0,"allocated":2449408,"data_stored":1404608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542242,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32980,"kb_used_data":1792,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338860,"statfs":{"total":96636764160,"available":96602992640,"internally_reserved":0,"allocated":1835008,"data_stored":811908,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":5,"up_from":14,"seq":60129542243,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32988,"kb_used_data":1796,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338852,"statfs":{"total":96636764160,"available":96602984448,"internally_reserved":0,"allocated":1839104,"data_stored":812490,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542241,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32992,"kb_used_data":1804,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338848,"statfs":{"total":96636764160,"available":96602980352,"internally_reserved":0,"allocated":1847296,"data_stored":813568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":3,"up_from":14,"seq":60129542242,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37144,"kb_used_data":1792,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334696,"statfs":{"total":96636764160,"available":96598728704,"internally_reserved":0,"allocated":1835008,"data_stored":811920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[]},{"osd":2,"up_from":14,"seq":60129542242,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32984,"kb_used_data":1792,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338856,"statfs":{"total":96636764160,"available":96602988544,"internally_reserved":0,"allocated":1835008,"data_stored":811976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":14,"seq":60129542244,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32984,"kb_used_data":1792,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338856,"statfs":{"total":96636764160,"available":96602988544,"internally_reserved":0,"allocated":1835008,"data_stored":811920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":0,"up_from":14,"seq":60129542244,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33560,"kb_used_data":2368,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338280,"statfs":{"total":96636764160,"available":96602398720,"internally_reserved":0,"allocated":2424832,"data_stored":1402254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":2354,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T15:50:39.235 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-09-16T15:50:39.235 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-09-16T15:50:39.415 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:39.414+0000 7fc476feb640 1 -- 172.21.15.123:0/1286967291 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc470152d60 msgr2=0x7fc4700cf3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:50:39.415 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:39.414+0000 7fc476feb640 1 --2- 172.21.15.123:0/1286967291 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc470152d60 0x7fc4700cf3b0 secure :-1 s=READY pgs=1605 cs=0 l=1 rev1=1 crypto rx=0x7fc458009900 tx=0x7fc45801c8b0 comp rx=0 tx=0).stop 2024-09-16T15:50:39.415 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:39.414+0000 7fc476feb640 1 -- 172.21.15.123:0/1286967291 shutdown_connections 2024-09-16T15:50:39.415 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:39.414+0000 7fc476feb640 1 --2- 172.21.15.123:0/1286967291 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc470152d60 0x7fc4700cf3b0 unknown :-1 s=CLOSED pgs=1605 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:39.415 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:39.414+0000 7fc476feb640 1 -- 172.21.15.123:0/1286967291 >> 172.21.15.123:0/1286967291 conn(0x7fc4700b1710 msgr2=0x7fc4700b1b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:50:39.415 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:39.414+0000 7fc476feb640 1 -- 172.21.15.123:0/1286967291 shutdown_connections 2024-09-16T15:50:39.416 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:39.414+0000 7fc476feb640 1 -- 172.21.15.123:0/1286967291 wait complete. 2024-09-16T15:50:39.416 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:39.418+0000 7fc476feb640 1 Processor -- start 2024-09-16T15:50:39.416 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:39.418+0000 7fc476feb640 1 -- start start 2024-09-16T15:50:39.418 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:39.418+0000 7fc476feb640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fc4700cd710 0x7fc4701fb7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:50:39.418 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:39.418+0000 7fc476feb640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc470152980 0x7fc470151370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:50:39.418 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:39.418+0000 7fc476feb640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc470152d60 0x7fc4701532a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:50:39.418 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:39.418+0000 7fc476feb640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fc47006fe20 con 0x7fc470152980 2024-09-16T15:50:39.419 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:39.418+0000 7fc476feb640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fc47006fca0 con 0x7fc470152d60 2024-09-16T15:50:39.419 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:39.418+0000 7fc476feb640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fc47006ffa0 con 0x7fc4700cd710 2024-09-16T15:50:39.419 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:39.418+0000 7fc46ffff640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc470152980 0x7fc470151370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:50:39.419 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:39.418+0000 7fc46ffff640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc470152980 0x7fc470151370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:49768/0 (socket says 172.21.15.123:49768) 2024-09-16T15:50:39.419 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:39.418+0000 7fc46ffff640 1 -- 172.21.15.123:0/2703393171 learned_addr learned my addr 172.21.15.123:0/2703393171 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:50:39.419 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:39.418+0000 7fc474d87640 1 --2- 172.21.15.123:0/2703393171 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fc4700cd710 0x7fc4701fb7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:50:39.419 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:39.418+0000 7fc475588640 1 --2- 172.21.15.123:0/2703393171 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc470152d60 0x7fc4701532a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:50:39.419 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:39.418+0000 7fc46ffff640 1 -- 172.21.15.123:0/2703393171 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fc4700cd710 msgr2=0x7fc4701fb7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:50:39.419 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:39.418+0000 7fc46ffff640 1 --2- 172.21.15.123:0/2703393171 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fc4700cd710 0x7fc4701fb7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:39.419 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:39.418+0000 7fc46ffff640 1 -- 172.21.15.123:0/2703393171 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc470152d60 msgr2=0x7fc4701532a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:50:39.419 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:39.418+0000 7fc46ffff640 1 --2- 172.21.15.123:0/2703393171 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc470152d60 0x7fc4701532a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:39.419 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:39.418+0000 7fc46ffff640 1 -- 172.21.15.123:0/2703393171 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc4700c9960 con 0x7fc470152980 2024-09-16T15:50:39.420 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:39.418+0000 7fc475588640 1 --2- 172.21.15.123:0/2703393171 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc470152d60 0x7fc4701532a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T15:50:39.420 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:39.418+0000 7fc46ffff640 1 --2- 172.21.15.123:0/2703393171 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc470152980 0x7fc470151370 secure :-1 s=READY pgs=1606 cs=0 l=1 rev1=1 crypto rx=0x7fc458009850 tx=0x7fc4580a58a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:50:39.420 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:39.418+0000 7fc46d7fa640 1 -- 172.21.15.123:0/2703393171 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4580b2040 con 0x7fc470152980 2024-09-16T15:50:39.420 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:39.418+0000 7fc46d7fa640 1 -- 172.21.15.123:0/2703393171 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fc458005320 con 0x7fc470152980 2024-09-16T15:50:39.420 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:39.418+0000 7fc476feb640 1 -- 172.21.15.123:0/2703393171 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc4700c9c40 con 0x7fc470152980 2024-09-16T15:50:39.420 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:39.418+0000 7fc46d7fa640 1 -- 172.21.15.123:0/2703393171 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4580046d0 con 0x7fc470152980 2024-09-16T15:50:39.420 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:39.418+0000 7fc476feb640 1 -- 172.21.15.123:0/2703393171 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc4700cbb10 con 0x7fc470152980 2024-09-16T15:50:39.421 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:39.418+0000 7fc476feb640 1 -- 172.21.15.123:0/2703393171 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc470070520 con 0x7fc470152980 2024-09-16T15:50:39.421 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:39.422+0000 7fc46d7fa640 1 -- 172.21.15.123:0/2703393171 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 22) ==== 97692+0+0 (secure 0 0 0) 0x7fc458004890 con 0x7fc470152980 2024-09-16T15:50:39.421 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:39.422+0000 7fc46d7fa640 1 --2- 172.21.15.123:0/2703393171 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fc450073580 0x7fc450075a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:50:39.421 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:39.422+0000 7fc474d87640 1 --2- 172.21.15.123:0/2703393171 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fc450073580 0x7fc450075a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:50:39.421 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:39.422+0000 7fc46d7fa640 1 -- 172.21.15.123:0/2703393171 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(146..146 src has 1..146) ==== 5737+0+0 (secure 0 0 0) 0x7fc45812ec80 con 0x7fc470152980 2024-09-16T15:50:39.421 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:39.422+0000 7fc474d87640 1 --2- 172.21.15.123:0/2703393171 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fc450073580 0x7fc450075a40 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7fc460004e30 tx=0x7fc460009040 comp rx=0 tx=0).ready entity=mgr.4150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:50:39.424 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:39.426+0000 7fc46d7fa640 1 -- 172.21.15.123:0/2703393171 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7fc4580164d0 con 0x7fc470152980 2024-09-16T15:50:39.655 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:39.654+0000 7fc476feb640 1 -- 172.21.15.123:0/2703393171 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) -- 0x7fc47015cda0 con 0x7fc470152980 2024-09-16T15:50:39.659 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:39.658+0000 7fc46d7fa640 1 -- 172.21.15.123:0/2703393171 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v28) ==== 190+0+147 (secure 0 0 0) 0x7fc4580f95a0 con 0x7fc470152980 2024-09-16T15:50:39.659 INFO:teuthology.orchestra.run.smithi123.stdout:[client.0] 2024-09-16T15:50:39.660 INFO:teuthology.orchestra.run.smithi123.stdout: key = AQC1Uehm7GDTAhAAHWWVIyjVVnTeeozBR8DfFA== 2024-09-16T15:50:39.660 INFO:teuthology.orchestra.run.smithi123.stdout: caps mds = "allow" 2024-09-16T15:50:39.660 INFO:teuthology.orchestra.run.smithi123.stdout: caps mon = "allow r" 2024-09-16T15:50:39.660 INFO:teuthology.orchestra.run.smithi123.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-09-16T15:50:39.660 INFO:teuthology.orchestra.run.smithi123.stderr:updated caps for client.0 2024-09-16T15:50:39.662 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:39.662+0000 7fc476feb640 1 -- 172.21.15.123:0/2703393171 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fc450073580 msgr2=0x7fc450075a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:50:39.662 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:39.662+0000 7fc476feb640 1 --2- 172.21.15.123:0/2703393171 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fc450073580 0x7fc450075a40 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7fc460004e30 tx=0x7fc460009040 comp rx=0 tx=0).stop 2024-09-16T15:50:39.662 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:39.662+0000 7fc476feb640 1 -- 172.21.15.123:0/2703393171 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc470152980 msgr2=0x7fc470151370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:50:39.662 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:39.662+0000 7fc476feb640 1 --2- 172.21.15.123:0/2703393171 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc470152980 0x7fc470151370 secure :-1 s=READY pgs=1606 cs=0 l=1 rev1=1 crypto rx=0x7fc458009850 tx=0x7fc4580a58a0 comp rx=0 tx=0).stop 2024-09-16T15:50:39.662 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:39.662+0000 7fc476feb640 1 -- 172.21.15.123:0/2703393171 shutdown_connections 2024-09-16T15:50:39.662 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:39.662+0000 7fc476feb640 1 --2- 172.21.15.123:0/2703393171 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fc450073580 0x7fc450075a40 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:39.662 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:39.662+0000 7fc476feb640 1 --2- 172.21.15.123:0/2703393171 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc470152d60 0x7fc4701532a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:39.663 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:39.662+0000 7fc476feb640 1 --2- 172.21.15.123:0/2703393171 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc470152980 0x7fc470151370 unknown :-1 s=CLOSED pgs=1606 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:39.663 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:39.662+0000 7fc476feb640 1 --2- 172.21.15.123:0/2703393171 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fc4700cd710 0x7fc4701fb7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:39.663 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:39.662+0000 7fc476feb640 1 -- 172.21.15.123:0/2703393171 >> 172.21.15.123:0/2703393171 conn(0x7fc4700b1710 msgr2=0x7fc4700b1af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:50:39.663 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:39.662+0000 7fc476feb640 1 -- 172.21.15.123:0/2703393171 shutdown_connections 2024-09-16T15:50:39.663 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:39.662+0000 7fc476feb640 1 -- 172.21.15.123:0/2703393171 wait complete. 2024-09-16T15:50:39.677 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-16T15:50:39.800 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:39.802+0000 7ff298abf640 1 -- 172.21.15.123:0/1564057016 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff29415ec40 msgr2=0x7ff29406f420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:50:39.800 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:39.802+0000 7ff298abf640 1 --2- 172.21.15.123:0/1564057016 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff29415ec40 0x7ff29406f420 secure :-1 s=READY pgs=1607 cs=0 l=1 rev1=1 crypto rx=0x7ff294076280 tx=0x7ff28c01dad0 comp rx=0 tx=0).stop 2024-09-16T15:50:39.800 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:39.802+0000 7ff298abf640 1 -- 172.21.15.123:0/1564057016 shutdown_connections 2024-09-16T15:50:39.800 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:39.802+0000 7ff298abf640 1 --2- 172.21.15.123:0/1564057016 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff29415ec40 0x7ff29406f420 unknown :-1 s=CLOSED pgs=1607 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:39.800 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:39.802+0000 7ff298abf640 1 --2- 172.21.15.123:0/1564057016 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7ff2940cf5a0 0x7ff29406eee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:39.801 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:39.802+0000 7ff298abf640 1 --2- 172.21.15.123:0/1564057016 >> v2:172.21.15.184:3300/0 conn(0x7ff294158120 0x7ff2940ca320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:39.801 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:39.802+0000 7ff298abf640 1 -- 172.21.15.123:0/1564057016 >> 172.21.15.123:0/1564057016 conn(0x7ff2940cfb10 msgr2=0x7ff2940cff20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:50:39.801 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:39.802+0000 7ff298abf640 1 -- 172.21.15.123:0/1564057016 shutdown_connections 2024-09-16T15:50:39.801 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:39.802+0000 7ff298abf640 1 -- 172.21.15.123:0/1564057016 wait complete. 2024-09-16T15:50:39.801 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:39.802+0000 7ff298abf640 1 Processor -- start 2024-09-16T15:50:39.801 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:39.802+0000 7ff298abf640 1 -- start start 2024-09-16T15:50:39.803 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:39.802+0000 7ff298abf640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff2940cf5a0 0x7ff2941fbab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:50:39.803 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:39.802+0000 7ff298abf640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7ff294158120 0x7ff2940b5130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:50:39.803 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:39.802+0000 7ff298abf640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff29415ec40 0x7ff294159670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:50:39.803 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:39.802+0000 7ff298abf640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7ff29415b210 con 0x7ff2940cf5a0 2024-09-16T15:50:39.804 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:39.802+0000 7ff298abf640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7ff29415b090 con 0x7ff29415ec40 2024-09-16T15:50:39.804 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:39.802+0000 7ff298abf640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7ff29415b390 con 0x7ff294158120 2024-09-16T15:50:39.804 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:39.802+0000 7ff29259c640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff2940cf5a0 0x7ff2941fbab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:50:39.804 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:39.802+0000 7ff29259c640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff2940cf5a0 0x7ff2941fbab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:49794/0 (socket says 172.21.15.123:49794) 2024-09-16T15:50:39.804 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:39.802+0000 7ff29259c640 1 -- 172.21.15.123:0/3670052449 learned_addr learned my addr 172.21.15.123:0/3670052449 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:50:39.804 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:39.802+0000 7ff291d9b640 1 --2- 172.21.15.123:0/3670052449 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7ff294158120 0x7ff2940b5130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:50:39.804 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:39.802+0000 7ff292d9d640 1 --2- 172.21.15.123:0/3670052449 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff29415ec40 0x7ff294159670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:50:39.804 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:39.802+0000 7ff29259c640 1 -- 172.21.15.123:0/3670052449 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7ff294158120 msgr2=0x7ff2940b5130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:50:39.804 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:39.802+0000 7ff29259c640 1 --2- 172.21.15.123:0/3670052449 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7ff294158120 0x7ff2940b5130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:39.804 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:39.802+0000 7ff29259c640 1 -- 172.21.15.123:0/3670052449 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff29415ec40 msgr2=0x7ff294159670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:50:39.804 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:39.802+0000 7ff29259c640 1 --2- 172.21.15.123:0/3670052449 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff29415ec40 0x7ff294159670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:39.804 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:39.802+0000 7ff29259c640 1 -- 172.21.15.123:0/3670052449 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff2940ad8c0 con 0x7ff2940cf5a0 2024-09-16T15:50:39.804 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:39.802+0000 7ff292d9d640 1 --2- 172.21.15.123:0/3670052449 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff29415ec40 0x7ff294159670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T15:50:39.805 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:39.802+0000 7ff29259c640 1 --2- 172.21.15.123:0/3670052449 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff2940cf5a0 0x7ff2941fbab0 secure :-1 s=READY pgs=1608 cs=0 l=1 rev1=1 crypto rx=0x7ff28800ccb0 tx=0x7ff28800a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:50:39.805 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:39.802+0000 7ff282ffd640 1 -- 172.21.15.123:0/3670052449 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff288004b30 con 0x7ff2940cf5a0 2024-09-16T15:50:39.805 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:39.802+0000 7ff282ffd640 1 -- 172.21.15.123:0/3670052449 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7ff288004de0 con 0x7ff2940cf5a0 2024-09-16T15:50:39.805 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:39.802+0000 7ff298abf640 1 -- 172.21.15.123:0/3670052449 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff2940adba0 con 0x7ff2940cf5a0 2024-09-16T15:50:39.805 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:39.802+0000 7ff282ffd640 1 -- 172.21.15.123:0/3670052449 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff288005280 con 0x7ff2940cf5a0 2024-09-16T15:50:39.805 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:39.802+0000 7ff298abf640 1 -- 172.21.15.123:0/3670052449 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff2940ae160 con 0x7ff2940cf5a0 2024-09-16T15:50:39.806 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:39.806+0000 7ff298abf640 1 -- 172.21.15.123:0/3670052449 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff2940ca320 con 0x7ff2940cf5a0 2024-09-16T15:50:39.806 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:39.806+0000 7ff282ffd640 1 -- 172.21.15.123:0/3670052449 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 22) ==== 97692+0+0 (secure 0 0 0) 0x7ff288005420 con 0x7ff2940cf5a0 2024-09-16T15:50:39.806 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:39.806+0000 7ff282ffd640 1 --2- 172.21.15.123:0/3670052449 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7ff274073580 0x7ff274075a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:50:39.806 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:39.806+0000 7ff291d9b640 1 --2- 172.21.15.123:0/3670052449 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7ff274073580 0x7ff274075a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:50:39.806 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:39.806+0000 7ff282ffd640 1 -- 172.21.15.123:0/3670052449 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(146..146 src has 1..146) ==== 5737+0+0 (secure 0 0 0) 0x7ff2880953b0 con 0x7ff2940cf5a0 2024-09-16T15:50:39.806 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:39.806+0000 7ff291d9b640 1 --2- 172.21.15.123:0/3670052449 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7ff274073580 0x7ff274075a40 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7ff284005fd0 tx=0x7ff28400c480 comp rx=0 tx=0).ready entity=mgr.4150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:50:39.810 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:39.810+0000 7ff282ffd640 1 -- 172.21.15.123:0/3670052449 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7ff288014030 con 0x7ff2940cf5a0 2024-09-16T15:50:40.019 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:40.018+0000 7ff298abf640 1 -- 172.21.15.123:0/3670052449 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff2940d06c0 con 0x7ff2940cf5a0 2024-09-16T15:50:40.020 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:40.022+0000 7ff282ffd640 1 -- 172.21.15.123:0/3670052449 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 95 v95) ==== 94+0+5236 (secure 0 0 0) 0x7ff28805fb40 con 0x7ff2940cf5a0 2024-09-16T15:50:40.020 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-09-16T15:50:40.020 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":95,"btime":"2024-09-16T15:50:33:425542+0000","default_fscid":11,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6580,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6835/430907849","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":430907849},{"type":"v1","addr":"172.21.15.184:6835","nonce":430907849}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":91},{"gid":8424,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6833/1494334556","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6832","nonce":1494334556},{"type":"v1","addr":"172.21.15.123:6833","nonce":1494334556}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":90},{"gid":8427,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6835/2168742484","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6834","nonce":2168742484},{"type":"v1","addr":"172.21.15.123:6835","nonce":2168742484}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":91}],"filesystems":[{"mdsmap":{"epoch":95,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-16T15:50:29.771624+0000","modified":"2024-09-16T15:50:32.799612+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8436},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8436":{"gid":8436,"name":"b","rank":0,"incarnation":92,"state":"up:active","state_seq":3,"addr":"172.21.15.184:6833/1547569255","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":1547569255},{"type":"v1","addr":"172.21.15.184:6833","nonce":1547569255}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[23],"metadata_pool":22,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8436,"qdb_cluster":[8436]},"id":11}]} 2024-09-16T15:50:40.020 INFO:teuthology.orchestra.run.smithi123.stderr:dumped fsmap epoch 95 2024-09-16T15:50:40.022 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:40.022+0000 7ff298abf640 1 -- 172.21.15.123:0/3670052449 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7ff274073580 msgr2=0x7ff274075a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:50:40.023 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:40.022+0000 7ff298abf640 1 --2- 172.21.15.123:0/3670052449 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7ff274073580 0x7ff274075a40 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7ff284005fd0 tx=0x7ff28400c480 comp rx=0 tx=0).stop 2024-09-16T15:50:40.023 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:40.022+0000 7ff298abf640 1 -- 172.21.15.123:0/3670052449 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff2940cf5a0 msgr2=0x7ff2941fbab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:50:40.023 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:40.022+0000 7ff298abf640 1 --2- 172.21.15.123:0/3670052449 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff2940cf5a0 0x7ff2941fbab0 secure :-1 s=READY pgs=1608 cs=0 l=1 rev1=1 crypto rx=0x7ff28800ccb0 tx=0x7ff28800a620 comp rx=0 tx=0).stop 2024-09-16T15:50:40.023 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:40.022+0000 7ff298abf640 1 -- 172.21.15.123:0/3670052449 shutdown_connections 2024-09-16T15:50:40.023 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:40.022+0000 7ff298abf640 1 --2- 172.21.15.123:0/3670052449 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7ff274073580 0x7ff274075a40 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:40.023 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:40.022+0000 7ff298abf640 1 --2- 172.21.15.123:0/3670052449 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff29415ec40 0x7ff294159670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:40.023 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:40.022+0000 7ff298abf640 1 --2- 172.21.15.123:0/3670052449 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7ff294158120 0x7ff2940b5130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:40.023 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:40.022+0000 7ff298abf640 1 --2- 172.21.15.123:0/3670052449 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff2940cf5a0 0x7ff2941fbab0 unknown :-1 s=CLOSED pgs=1608 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:40.023 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:40.022+0000 7ff298abf640 1 -- 172.21.15.123:0/3670052449 >> 172.21.15.123:0/3670052449 conn(0x7ff2940cfb10 msgr2=0x7ff29414d3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:50:40.023 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:40.022+0000 7ff298abf640 1 -- 172.21.15.123:0/3670052449 shutdown_connections 2024-09-16T15:50:40.023 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:40.022+0000 7ff298abf640 1 -- 172.21.15.123:0/3670052449 wait complete. 2024-09-16T15:50:40.048 DEBUG:teuthology.orchestra.run.smithi123:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-16T15:50:40.050 DEBUG:teuthology.orchestra.run.smithi184:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-16T15:50:40.441 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 95, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-09-16T15:50:29.771624+0000', 'modified': '2024-09-16T15:50:32.799612+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 8436}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8436': {'gid': 8436, 'name': 'b', 'rank': 0, 'incarnation': 92, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.184:6833/1547569255', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.184:6832', 'nonce': 1547569255}, {'type': 'v1', 'addr': '172.21.15.184:6833', 'nonce': 1547569255}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540719139924082687, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [23], 'metadata_pool': 22, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8436, 'qdb_cluster': [8436]} 2024-09-16T15:50:40.441 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-09-16T15:50:40.442 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-09-16T15:50:40.578 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:40.578+0000 7f5e1994c640 1 -- 172.21.15.123:0/3356680053 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5e1415f420 msgr2=0x7f5e14068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:50:40.578 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:40.578+0000 7f5e1994c640 1 --2- 172.21.15.123:0/3356680053 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5e1415f420 0x7f5e14068960 secure :-1 s=READY pgs=1609 cs=0 l=1 rev1=1 crypto rx=0x7f5e140740b0 tx=0x7f5e0c01ccf0 comp rx=0 tx=0).stop 2024-09-16T15:50:40.578 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:40.578+0000 7f5e1994c640 1 -- 172.21.15.123:0/3356680053 shutdown_connections 2024-09-16T15:50:40.578 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:40.578+0000 7f5e1994c640 1 --2- 172.21.15.123:0/3356680053 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f5e14074700 0x7f5e140cbb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:40.578 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:40.578+0000 7f5e1994c640 1 --2- 172.21.15.123:0/3356680053 >> v2:172.21.15.184:3300/0 conn(0x7f5e140b2910 0x7f5e140741c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:40.578 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:40.578+0000 7f5e1994c640 1 --2- 172.21.15.123:0/3356680053 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5e1415f420 0x7f5e14068960 unknown :-1 s=CLOSED pgs=1609 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:40.578 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:40.578+0000 7f5e1994c640 1 -- 172.21.15.123:0/3356680053 >> 172.21.15.123:0/3356680053 conn(0x7f5e140b1df0 msgr2=0x7f5e140b2200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:50:40.578 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:40.578+0000 7f5e1994c640 1 -- 172.21.15.123:0/3356680053 shutdown_connections 2024-09-16T15:50:40.579 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:40.578+0000 7f5e1994c640 1 -- 172.21.15.123:0/3356680053 wait complete. 2024-09-16T15:50:40.579 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:40.578+0000 7f5e1994c640 1 Processor -- start 2024-09-16T15:50:40.579 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:40.578+0000 7f5e1994c640 1 -- start start 2024-09-16T15:50:40.581 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:40.582+0000 7f5e1994c640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5e14074700 0x7f5e14157ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:50:40.581 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:40.582+0000 7f5e1994c640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5e140b2910 0x7f5e140c4990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:50:40.581 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:40.582+0000 7f5e1994c640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f5e1415f420 0x7f5e1415c9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:50:40.581 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:40.582+0000 7f5e1994c640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f5e140b5420 con 0x7f5e14074700 2024-09-16T15:50:40.581 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:40.582+0000 7f5e1994c640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f5e140b52a0 con 0x7f5e140b2910 2024-09-16T15:50:40.581 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:40.582+0000 7f5e1994c640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f5e140b55a0 con 0x7f5e1415f420 2024-09-16T15:50:40.581 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:40.582+0000 7f5e12ffd640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5e14074700 0x7f5e14157ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:50:40.582 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:40.582+0000 7f5e12ffd640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5e14074700 0x7f5e14157ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:49808/0 (socket says 172.21.15.123:49808) 2024-09-16T15:50:40.582 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:40.582+0000 7f5e12ffd640 1 -- 172.21.15.123:0/3620421730 learned_addr learned my addr 172.21.15.123:0/3620421730 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:50:40.582 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:40.582+0000 7f5e137fe640 1 --2- 172.21.15.123:0/3620421730 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f5e1415f420 0x7f5e1415c9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:50:40.582 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:40.582+0000 7f5e127fc640 1 --2- 172.21.15.123:0/3620421730 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5e140b2910 0x7f5e140c4990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:50:40.582 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:40.582+0000 7f5e12ffd640 1 -- 172.21.15.123:0/3620421730 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f5e1415f420 msgr2=0x7f5e1415c9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:50:40.582 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:40.582+0000 7f5e12ffd640 1 --2- 172.21.15.123:0/3620421730 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f5e1415f420 0x7f5e1415c9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:40.582 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:40.582+0000 7f5e12ffd640 1 -- 172.21.15.123:0/3620421730 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5e140b2910 msgr2=0x7f5e140c4990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:50:40.582 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:40.582+0000 7f5e12ffd640 1 --2- 172.21.15.123:0/3620421730 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5e140b2910 0x7f5e140c4990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:40.582 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:40.582+0000 7f5e12ffd640 1 -- 172.21.15.123:0/3620421730 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5e1409e3f0 con 0x7f5e14074700 2024-09-16T15:50:40.582 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:40.582+0000 7f5e12ffd640 1 --2- 172.21.15.123:0/3620421730 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5e14074700 0x7f5e14157ed0 secure :-1 s=READY pgs=1610 cs=0 l=1 rev1=1 crypto rx=0x7f5e0c000f90 tx=0x7f5e0c009360 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:50:40.582 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:40.582+0000 7f5dfbfff640 1 -- 172.21.15.123:0/3620421730 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e0c0bb070 con 0x7f5e14074700 2024-09-16T15:50:40.582 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:40.582+0000 7f5dfbfff640 1 -- 172.21.15.123:0/3620421730 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f5e0c004d40 con 0x7f5e14074700 2024-09-16T15:50:40.582 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:40.582+0000 7f5e1994c640 1 -- 172.21.15.123:0/3620421730 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5e14158410 con 0x7f5e14074700 2024-09-16T15:50:40.582 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:40.582+0000 7f5dfbfff640 1 -- 172.21.15.123:0/3620421730 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e0c0b2040 con 0x7f5e14074700 2024-09-16T15:50:40.583 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:40.582+0000 7f5e1994c640 1 -- 172.21.15.123:0/3620421730 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5e140cdf70 con 0x7f5e14074700 2024-09-16T15:50:40.583 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:40.582+0000 7f5dfbfff640 1 -- 172.21.15.123:0/3620421730 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 22) ==== 97692+0+0 (secure 0 0 0) 0x7f5e0c0be070 con 0x7f5e14074700 2024-09-16T15:50:40.583 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:40.582+0000 7f5dfbfff640 1 --2- 172.21.15.123:0/3620421730 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f5e00073580 0x7f5e00075a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:50:40.584 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:40.582+0000 7f5e127fc640 1 --2- 172.21.15.123:0/3620421730 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f5e00073580 0x7f5e00075a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:50:40.584 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:40.582+0000 7f5dfbfff640 1 -- 172.21.15.123:0/3620421730 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(146..146 src has 1..146) ==== 5737+0+0 (secure 0 0 0) 0x7f5e0c12e620 con 0x7f5e14074700 2024-09-16T15:50:40.584 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:40.582+0000 7f5e1ca36000 1 Processor -- start 2024-09-16T15:50:40.584 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:40.582+0000 7f5e127fc640 1 --2- 172.21.15.123:0/3620421730 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f5e00073580 0x7f5e00075a40 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f5e080058d0 tx=0x7f5e08005860 comp rx=0 tx=0).ready entity=mgr.4150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:50:40.584 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:40.582+0000 7f5e1ca36000 1 -- start start 2024-09-16T15:50:40.586 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:40.586+0000 7f5e1ca36000 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55f96594e9d0 0x55f96594edb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:50:40.587 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:40.586+0000 7f5e1ca36000 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55f96594f2f0 0x55f9659028d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:50:40.587 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:40.586+0000 7f5e1ca36000 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x55f965902e10 0x55f9659031f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:50:40.587 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:40.586+0000 7f5e1ca36000 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x55f9658233f0 con 0x55f96594f2f0 2024-09-16T15:50:40.587 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:40.586+0000 7f5e1ca36000 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x55f965760b20 con 0x55f96594e9d0 2024-09-16T15:50:40.587 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:40.586+0000 7f5e1ca36000 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x55f96574ff50 con 0x55f965902e10 2024-09-16T15:50:40.587 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:40.586+0000 7f5e127fc640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55f96594f2f0 0x55f9659028d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:50:40.587 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:40.586+0000 7f5e127fc640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55f96594f2f0 0x55f9659028d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:49820/0 (socket says 172.21.15.123:49820) 2024-09-16T15:50:40.587 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:40.586+0000 7f5e127fc640 1 -- 172.21.15.123:0/522174073 learned_addr learned my addr 172.21.15.123:0/522174073 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:50:40.587 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:40.586+0000 7f5e12ffd640 1 --2- 172.21.15.123:0/522174073 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55f96594e9d0 0x55f96594edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:50:40.587 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:40.586+0000 7f5e137fe640 1 --2- 172.21.15.123:0/522174073 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x55f965902e10 0x55f9659031f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:50:40.587 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:40.586+0000 7f5e127fc640 1 -- 172.21.15.123:0/522174073 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x55f965902e10 msgr2=0x55f9659031f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T15:50:40.587 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:40.586+0000 7f5e127fc640 1 --2- 172.21.15.123:0/522174073 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x55f965902e10 0x55f9659031f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:40.587 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:40.586+0000 7f5e127fc640 1 -- 172.21.15.123:0/522174073 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55f96594e9d0 msgr2=0x55f96594edb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T15:50:40.588 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:40.586+0000 7f5e127fc640 1 --2- 172.21.15.123:0/522174073 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55f96594e9d0 0x55f96594edb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:40.588 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:40.586+0000 7f5e127fc640 1 -- 172.21.15.123:0/522174073 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f965717590 con 0x55f96594f2f0 2024-09-16T15:50:40.588 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:40.586+0000 7f5e137fe640 1 --2- 172.21.15.123:0/522174073 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x55f965902e10 0x55f9659031f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T15:50:40.588 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:40.586+0000 7f5e12ffd640 1 --2- 172.21.15.123:0/522174073 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55f96594e9d0 0x55f96594edb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T15:50:40.588 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:40.586+0000 7f5e127fc640 1 --2- 172.21.15.123:0/522174073 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55f96594f2f0 0x55f9659028d0 secure :-1 s=READY pgs=1611 cs=0 l=1 rev1=1 crypto rx=0x7f5e0800fde0 tx=0x7f5e08012000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a0a898163691ea58 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:50:40.588 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:40.586+0000 7f5df97fa640 1 -- 172.21.15.123:0/522174073 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e08022070 con 0x55f96594f2f0 2024-09-16T15:50:40.588 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:40.586+0000 7f5df97fa640 1 -- 172.21.15.123:0/522174073 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f5e0801e450 con 0x55f96594f2f0 2024-09-16T15:50:40.588 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:40.586+0000 7f5df97fa640 1 -- 172.21.15.123:0/522174073 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e0801e880 con 0x55f96594f2f0 2024-09-16T15:50:40.588 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:40.586+0000 7f5e1ca36000 1 -- 172.21.15.123:0/522174073 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55f96594f2f0 msgr2=0x55f9659028d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:50:40.588 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:40.586+0000 7f5e1ca36000 1 --2- 172.21.15.123:0/522174073 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55f96594f2f0 0x55f9659028d0 secure :-1 s=READY pgs=1611 cs=0 l=1 rev1=1 crypto rx=0x7f5e0800fde0 tx=0x7f5e08012000 comp rx=0 tx=0).stop 2024-09-16T15:50:40.588 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:40.586+0000 7f5e1ca36000 1 -- 172.21.15.123:0/522174073 shutdown_connections 2024-09-16T15:50:40.588 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:40.586+0000 7f5e1ca36000 1 --2- 172.21.15.123:0/522174073 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x55f965902e10 0x55f9659031f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:40.588 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:40.586+0000 7f5e1ca36000 1 --2- 172.21.15.123:0/522174073 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55f96594f2f0 0x55f9659028d0 unknown :-1 s=CLOSED pgs=1611 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:40.588 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:40.586+0000 7f5e1ca36000 1 --2- 172.21.15.123:0/522174073 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55f96594e9d0 0x55f96594edb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:40.589 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:40.586+0000 7f5e1ca36000 1 -- 172.21.15.123:0/522174073 >> 172.21.15.123:0/522174073 conn(0x55f9657ec130 msgr2=0x55f965931e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:50:40.589 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:40.586+0000 7f5e1ca36000 1 -- 172.21.15.123:0/522174073 shutdown_connections 2024-09-16T15:50:40.589 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:40.586+0000 7f5e1ca36000 1 -- 172.21.15.123:0/522174073 wait complete. 2024-09-16T15:50:40.589 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:40.586+0000 7f5e1ca36000 1 -- start start 2024-09-16T15:50:40.589 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:40.586+0000 7f5e1ca36000 1 Processor -- start 2024-09-16T15:50:40.592 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:40.586+0000 7f5e1ca36000 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55f965902e10 0x55f96592f6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:50:40.592 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:40.586+0000 7f5e1ca36000 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x55f96594e9d0 0x55f96592fe70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:50:40.592 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:40.586+0000 7f5e1ca36000 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55f96594f2f0 0x55f9659303b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:50:40.592 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:40.586+0000 7f5e1ca36000 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x55f9656ea5d0 con 0x55f96594f2f0 2024-09-16T15:50:40.592 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:40.586+0000 7f5e1ca36000 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x55f965753d60 con 0x55f965902e10 2024-09-16T15:50:40.593 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:40.586+0000 7f5e1ca36000 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x55f9656f6500 con 0x55f96594e9d0 2024-09-16T15:50:40.593 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:40.586+0000 7f5e1ca36000 10 client.0 _ll_register_callbacks cb 0x55f9656fa810 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T15:50:40.593 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:40.586+0000 7f5e137fe640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55f96594f2f0 0x55f9659303b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:50:40.593 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:40.586+0000 7f5e137fe640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55f96594f2f0 0x55f9659303b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:49822/0 (socket says 172.21.15.123:49822) 2024-09-16T15:50:40.593 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:40.586+0000 7f5e137fe640 1 -- 172.21.15.123:0/295119805 learned_addr learned my addr 172.21.15.123:0/295119805 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:50:40.593 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:40.590+0000 7f5e127fc640 1 --2- 172.21.15.123:0/295119805 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x55f96594e9d0 0x55f96592fe70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:50:40.593 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:40.590+0000 7f5e12ffd640 1 --2- 172.21.15.123:0/295119805 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55f965902e10 0x55f96592f6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:50:40.593 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:40.590+0000 7f5e137fe640 1 -- 172.21.15.123:0/295119805 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x55f96594e9d0 msgr2=0x55f96592fe70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T15:50:40.593 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:40.590+0000 7f5e137fe640 1 --2- 172.21.15.123:0/295119805 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x55f96594e9d0 0x55f96592fe70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:40.593 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:40.590+0000 7f5e137fe640 1 -- 172.21.15.123:0/295119805 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55f965902e10 msgr2=0x55f96592f6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T15:50:40.593 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:40.590+0000 7f5e137fe640 1 --2- 172.21.15.123:0/295119805 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55f965902e10 0x55f96592f6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:40.593 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:40.590+0000 7f5e137fe640 1 -- 172.21.15.123:0/295119805 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f9656ee170 con 0x55f96594f2f0 2024-09-16T15:50:40.593 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:40.590+0000 7f5e137fe640 1 --2- 172.21.15.123:0/295119805 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55f96594f2f0 0x55f9659303b0 secure :-1 s=READY pgs=1612 cs=0 l=1 rev1=1 crypto rx=0x7f5e04010440 tx=0x7f5e04010900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a33ccd4acca32baa server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:50:40.593 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:40.590+0000 7f5df27fc640 10 client.0 ms_handle_connect on v2:172.21.15.123:3300/0 2024-09-16T15:50:40.594 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:40.590+0000 7f5df27fc640 1 -- 172.21.15.123:0/295119805 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e04012920 con 0x55f96594f2f0 2024-09-16T15:50:40.594 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:40.590+0000 7f5df27fc640 1 -- 172.21.15.123:0/295119805 <== mon.0 v2:172.21.15.123:3300/0 2 ==== osd_map(146..146 src has 1..146) ==== 5737+0+0 (secure 0 0 0) 0x7f5e04012ac0 con 0x55f96594f2f0 2024-09-16T15:50:40.594 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:40.590+0000 7f5e1ca36000 1 -- 172.21.15.123:0/295119805 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55f9656e80f0 con 0x55f96594f2f0 2024-09-16T15:50:40.594 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:40.590+0000 7f5df27fc640 1 -- 172.21.15.123:0/295119805 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_get_version_reply(handle=1 version=95) ==== 24+0+0 (secure 0 0 0) 0x7f5e04024050 con 0x55f96594f2f0 2024-09-16T15:50:40.594 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:40.590+0000 7f5e1ca36000 10 client.8583 fetch_fsmap learned FSMap version 95 2024-09-16T15:50:40.594 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:40.590+0000 7f5e1ca36000 1 -- 172.21.15.123:0/295119805 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,fsmap=95,monmap=2+}) -- 0x55f9657d4ef0 con 0x55f96594f2f0 2024-09-16T15:50:40.594 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:40.590+0000 7f5df27fc640 1 -- 172.21.15.123:0/295119805 <== mon.0 v2:172.21.15.123:3300/0 4 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x55f9656e80f0 con 0x55f96594f2f0 2024-09-16T15:50:40.594 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:40.590+0000 7f5df27fc640 1 -- 172.21.15.123:0/295119805 <== mon.0 v2:172.21.15.123:3300/0 5 ==== fsmap(e 95) ==== 3502+0+0 (secure 0 0 0) 0x7f5e04027020 con 0x55f96594f2f0 2024-09-16T15:50:40.594 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:40.590+0000 7f5e1ca36000 10 client.8583 fetch_fsmap finished waiting for FSMap version 95 2024-09-16T15:50:40.594 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:40.590+0000 7f5e1ca36000 10 client.8583 resolve_mds: resolved name 'b' to daemon mds.b 2024-09-16T15:50:40.594 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:40.590+0000 7f5e1ca36000 20 client.8583 populate_metadata read hostname 'smithi123' 2024-09-16T15:50:40.594 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:40.590+0000 7f5e1ca36000 1 --2- 172.21.15.123:0/295119805 >> [v2:172.21.15.184:6832/1547569255,v1:172.21.15.184:6833/1547569255] conn(0x55f9659310f0 0x55f965960c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:50:40.594 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:40.590+0000 7f5e1ca36000 4 client.8583 mds_command: new command op to 8436 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-16T15:50:40.594 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:40.590+0000 7f5e1ca36000 1 -- 172.21.15.123:0/295119805 --> [v2:172.21.15.184:6832/1547569255,v1:172.21.15.184:6833/1547569255] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f965762a70 con 0x55f9659310f0 2024-09-16T15:50:40.595 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:40.590+0000 7f5e12ffd640 1 --2- 172.21.15.123:0/295119805 >> [v2:172.21.15.184:6832/1547569255,v1:172.21.15.184:6833/1547569255] conn(0x55f9659310f0 0x55f965960c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:50:40.595 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:40.590+0000 7f5df27fc640 10 client.8583 ms_handle_connect on v2:172.21.15.184:6832/1547569255 2024-09-16T15:50:40.595 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:40.590+0000 7f5e12ffd640 1 --2- 172.21.15.123:0/295119805 >> [v2:172.21.15.184:6832/1547569255,v1:172.21.15.184:6833/1547569255] conn(0x55f9659310f0 0x55f965960c90 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=460718405f7710ed server_cookie=70fe8feaed3c9714 in_seq=0 out_seq=0 2024-09-16T15:50:40.595 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:40.590+0000 7f5df27fc640 1 -- 172.21.15.123:0/295119805 <== mds.0 v2:172.21.15.184:6832/1547569255 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x55f965762a70 con 0x55f9659310f0 2024-09-16T15:50:40.595 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:40.590+0000 7f5df27fc640 10 client.8583 handle_command_reply: tid=0 2024-09-16T15:50:40.595 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:40.590+0000 7f5e1ca36000 1 client.8583 shutdown 2024-09-16T15:50:40.595 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:40.594+0000 7f5e1ca36000 1 -- 172.21.15.123:0/295119805 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55f96594f2f0 msgr2=0x55f9659303b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:50:40.595 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:40.594+0000 7f5e1ca36000 1 --2- 172.21.15.123:0/295119805 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55f96594f2f0 0x55f9659303b0 secure :-1 s=READY pgs=1612 cs=0 l=1 rev1=1 crypto rx=0x7f5e04010440 tx=0x7f5e04010900 comp rx=0 tx=0).stop 2024-09-16T15:50:40.595 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:40.594+0000 7f5e1ca36000 1 -- 172.21.15.123:0/295119805 shutdown_connections 2024-09-16T15:50:40.595 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:40.594+0000 7f5e1ca36000 1 --2- 172.21.15.123:0/295119805 >> [v2:172.21.15.184:6832/1547569255,v1:172.21.15.184:6833/1547569255] conn(0x55f9659310f0 0x55f965960c90 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:40.595 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:40.594+0000 7f5e1ca36000 1 --2- 172.21.15.123:0/295119805 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55f96594f2f0 0x55f9659303b0 unknown :-1 s=CLOSED pgs=1612 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:40.595 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:40.594+0000 7f5e1ca36000 1 --2- 172.21.15.123:0/295119805 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x55f96594e9d0 0x55f96592fe70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:40.595 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:40.594+0000 7f5e1ca36000 1 --2- 172.21.15.123:0/295119805 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55f965902e10 0x55f96592f6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:40.596 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:40.594+0000 7f5df27fc640 0 client.8583 ms_handle_reset on v2:172.21.15.184:6832/1547569255 2024-09-16T15:50:40.596 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:40.594+0000 7f5e1ca36000 1 -- 172.21.15.123:0/295119805 >> 172.21.15.123:0/295119805 conn(0x55f9657ec130 msgr2=0x55f965901fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:50:40.596 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:40.594+0000 7f5e1ca36000 1 -- 172.21.15.123:0/295119805 shutdown_connections 2024-09-16T15:50:40.596 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:40.594+0000 7f5e1ca36000 1 -- 172.21.15.123:0/295119805 wait complete. 2024-09-16T15:50:40.596 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:40.594+0000 7f5e1ca36000 20 client.8583 trim_cache size 0 max 16384 2024-09-16T15:50:40.610 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:40.610+0000 7f5e1ca36000 1 Processor -- start 2024-09-16T15:50:40.610 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:40.610+0000 7f5e1ca36000 1 -- start start 2024-09-16T15:50:40.611 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:40.610+0000 7f5e1ca36000 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55f965902e10 0x55f9659c0fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:50:40.611 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:40.610+0000 7f5e1ca36000 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x55f96594e9d0 0x55f9659c1520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:50:40.611 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:40.610+0000 7f5e1ca36000 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55f96594f2f0 0x55f9659c1a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:50:40.611 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:40.610+0000 7f5e1ca36000 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f5e14204240 con 0x55f965902e10 2024-09-16T15:50:40.612 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:40.610+0000 7f5e1ca36000 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f5e141647d0 con 0x55f96594f2f0 2024-09-16T15:50:40.612 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:40.610+0000 7f5e1ca36000 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f5e141fc050 con 0x55f96594e9d0 2024-09-16T15:50:40.612 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:40.610+0000 7f5e12ffd640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55f965902e10 0x55f9659c0fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:50:40.612 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:40.610+0000 7f5e12ffd640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55f965902e10 0x55f9659c0fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:49838/0 (socket says 172.21.15.123:49838) 2024-09-16T15:50:40.612 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:40.610+0000 7f5e12ffd640 1 -- 172.21.15.123:0/1786598455 learned_addr learned my addr 172.21.15.123:0/1786598455 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:50:40.612 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:40.610+0000 7f5e12ffd640 1 -- 172.21.15.123:0/1786598455 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x55f96594e9d0 msgr2=0x55f9659c1520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T15:50:40.612 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:40.610+0000 7f5e12ffd640 1 --2- 172.21.15.123:0/1786598455 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x55f96594e9d0 0x55f9659c1520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:40.612 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:40.610+0000 7f5e12ffd640 1 -- 172.21.15.123:0/1786598455 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55f96594f2f0 msgr2=0x55f9659c1a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T15:50:40.612 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:40.610+0000 7f5e12ffd640 1 --2- 172.21.15.123:0/1786598455 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55f96594f2f0 0x55f9659c1a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:40.612 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:40.610+0000 7f5e12ffd640 1 -- 172.21.15.123:0/1786598455 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f9657abd90 con 0x55f965902e10 2024-09-16T15:50:40.612 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:40.610+0000 7f5e12ffd640 1 --2- 172.21.15.123:0/1786598455 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55f965902e10 0x55f9659c0fe0 secure :-1 s=READY pgs=1613 cs=0 l=1 rev1=1 crypto rx=0x7f5e0c01c760 tx=0x7f5e0c01ccf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=497875eee12cb60c server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:50:40.612 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:40.610+0000 7f5df8ff9640 1 -- 172.21.15.123:0/1786598455 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e0c0f8030 con 0x55f965902e10 2024-09-16T15:50:40.612 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:40.610+0000 7f5df8ff9640 1 -- 172.21.15.123:0/1786598455 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f5e0c0f8fc0 con 0x55f965902e10 2024-09-16T15:50:40.612 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:40.610+0000 7f5df8ff9640 1 -- 172.21.15.123:0/1786598455 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e0c0f88c0 con 0x55f965902e10 2024-09-16T15:50:40.615 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:40.614+0000 7f5e1ca36000 1 -- 172.21.15.123:0/1786598455 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55f965902e10 msgr2=0x55f9659c0fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:50:40.615 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:40.614+0000 7f5e1ca36000 1 --2- 172.21.15.123:0/1786598455 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55f965902e10 0x55f9659c0fe0 secure :-1 s=READY pgs=1613 cs=0 l=1 rev1=1 crypto rx=0x7f5e0c01c760 tx=0x7f5e0c01ccf0 comp rx=0 tx=0).stop 2024-09-16T15:50:40.615 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:40.614+0000 7f5e1ca36000 1 -- 172.21.15.123:0/1786598455 shutdown_connections 2024-09-16T15:50:40.615 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:40.614+0000 7f5e1ca36000 1 --2- 172.21.15.123:0/1786598455 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55f96594f2f0 0x55f9659c1a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:40.616 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:40.614+0000 7f5e1ca36000 1 --2- 172.21.15.123:0/1786598455 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x55f96594e9d0 0x55f9659c1520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:40.616 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:40.614+0000 7f5e1ca36000 1 --2- 172.21.15.123:0/1786598455 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55f965902e10 0x55f9659c0fe0 unknown :-1 s=CLOSED pgs=1613 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:40.616 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:40.614+0000 7f5e1ca36000 1 -- 172.21.15.123:0/1786598455 >> 172.21.15.123:0/1786598455 conn(0x55f9657ec130 msgr2=0x55f96590aa60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:50:40.616 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:40.614+0000 7f5e1ca36000 1 -- 172.21.15.123:0/1786598455 shutdown_connections 2024-09-16T15:50:40.616 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:40.614+0000 7f5e1ca36000 1 -- 172.21.15.123:0/1786598455 wait complete. 2024-09-16T15:50:40.616 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:40.614+0000 7f5e1ca36000 1 -- start start 2024-09-16T15:50:40.616 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:40.614+0000 7f5e1ca36000 1 Processor -- start 2024-09-16T15:50:40.616 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:40.614+0000 7f5e1ca36000 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x55f965902e10 0x55f9659631b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:50:40.616 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:40.614+0000 7f5e1ca36000 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55f96594e9d0 0x55f9659636f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:50:40.616 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:40.614+0000 7f5e1ca36000 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55f96594f2f0 0x55f965963c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:50:40.616 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:40.614+0000 7f5e1ca36000 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x55f96575f8f0 con 0x55f96594e9d0 2024-09-16T15:50:40.616 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:40.614+0000 7f5e1ca36000 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x55f965758390 con 0x55f96594f2f0 2024-09-16T15:50:40.616 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:40.614+0000 7f5e1ca36000 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x55f9657d4a60 con 0x55f965902e10 2024-09-16T15:50:40.617 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:40.614+0000 7f5e1ca36000 10 client.0 _ll_register_callbacks cb 0x7f5e0400b3a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T15:50:40.617 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:40.614+0000 7f5e127fc640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55f96594e9d0 0x55f9659636f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:50:40.617 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:40.614+0000 7f5e127fc640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55f96594e9d0 0x55f9659636f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:49844/0 (socket says 172.21.15.123:49844) 2024-09-16T15:50:40.617 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:40.614+0000 7f5e127fc640 1 -- 172.21.15.123:0/3233743309 learned_addr learned my addr 172.21.15.123:0/3233743309 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:50:40.617 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:40.614+0000 7f5e127fc640 1 -- 172.21.15.123:0/3233743309 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x55f965902e10 msgr2=0x55f9659631b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T15:50:40.617 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:40.614+0000 7f5e137fe640 1 --2- 172.21.15.123:0/3233743309 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55f96594f2f0 0x55f965963c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:50:40.617 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:40.614+0000 7f5e127fc640 1 --2- 172.21.15.123:0/3233743309 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x55f965902e10 0x55f9659631b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:40.617 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:40.614+0000 7f5e127fc640 1 -- 172.21.15.123:0/3233743309 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55f96594f2f0 msgr2=0x55f965963c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T15:50:40.617 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:40.614+0000 7f5e127fc640 1 --2- 172.21.15.123:0/3233743309 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55f96594f2f0 0x55f965963c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:40.617 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:40.614+0000 7f5e127fc640 1 -- 172.21.15.123:0/3233743309 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f965716a90 con 0x55f96594e9d0 2024-09-16T15:50:40.617 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:40.614+0000 7f5e127fc640 1 --2- 172.21.15.123:0/3233743309 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55f96594e9d0 0x55f9659636f0 secure :-1 s=READY pgs=1614 cs=0 l=1 rev1=1 crypto rx=0x55f965701130 tx=0x7f5e08020470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ccf294225fefdc30 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:50:40.617 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:40.614+0000 7f5df27fc640 10 client.0 ms_handle_connect on v2:172.21.15.123:3300/0 2024-09-16T15:50:40.617 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:40.614+0000 7f5df27fc640 1 -- 172.21.15.123:0/3233743309 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e08035060 con 0x55f96594e9d0 2024-09-16T15:50:40.617 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:40.614+0000 7f5df27fc640 1 -- 172.21.15.123:0/3233743309 <== mon.0 v2:172.21.15.123:3300/0 2 ==== osd_map(146..146 src has 1..146) ==== 5737+0+0 (secure 0 0 0) 0x7f5e08020860 con 0x55f96594e9d0 2024-09-16T15:50:40.617 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:40.614+0000 7f5e1ca36000 1 -- 172.21.15.123:0/3233743309 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5e0c0f8fc0 con 0x55f96594e9d0 2024-09-16T15:50:40.618 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:40.614+0000 7f5df27fc640 1 -- 172.21.15.123:0/3233743309 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_get_version_reply(handle=1 version=95) ==== 24+0+0 (secure 0 0 0) 0x7f5e0803b050 con 0x55f96594e9d0 2024-09-16T15:50:40.618 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:40.614+0000 7f5e1ca36000 10 client.8589 fetch_fsmap learned FSMap version 95 2024-09-16T15:50:40.618 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:40.614+0000 7f5e1ca36000 1 -- 172.21.15.123:0/3233743309 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,fsmap=95,monmap=2+}) -- 0x55f9656f5050 con 0x55f96594e9d0 2024-09-16T15:50:40.618 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:40.614+0000 7f5df27fc640 1 -- 172.21.15.123:0/3233743309 <== mon.0 v2:172.21.15.123:3300/0 4 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f5e0c0f8fc0 con 0x55f96594e9d0 2024-09-16T15:50:40.618 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:40.614+0000 7f5df27fc640 1 -- 172.21.15.123:0/3233743309 <== mon.0 v2:172.21.15.123:3300/0 5 ==== fsmap(e 95) ==== 3502+0+0 (secure 0 0 0) 0x7f5e0803e020 con 0x55f96594e9d0 2024-09-16T15:50:40.618 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:40.614+0000 7f5e1ca36000 10 client.8589 fetch_fsmap finished waiting for FSMap version 95 2024-09-16T15:50:40.618 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:40.614+0000 7f5e1ca36000 10 client.8589 resolve_mds: resolved name 'b' to daemon mds.b 2024-09-16T15:50:40.618 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:40.614+0000 7f5e1ca36000 20 client.8589 populate_metadata read hostname 'smithi123' 2024-09-16T15:50:40.618 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:40.614+0000 7f5e1ca36000 1 --2- 172.21.15.123:0/3233743309 >> [v2:172.21.15.184:6832/1547569255,v1:172.21.15.184:6833/1547569255] conn(0x55f9659cb050 0x55f9659cd440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:50:40.618 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:40.614+0000 7f5e1ca36000 4 client.8589 mds_command: new command op to 8436 tid=0 multi_id=0 [{"prefix": "status"}] 2024-09-16T15:50:40.618 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:40.614+0000 7f5e1ca36000 1 -- 172.21.15.123:0/3233743309 --> [v2:172.21.15.184:6832/1547569255,v1:172.21.15.184:6833/1547569255] -- command(tid 0: {"prefix": "status"}) -- 0x55f96575e520 con 0x55f9659cb050 2024-09-16T15:50:40.618 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:40.614+0000 7f5e12ffd640 1 --2- 172.21.15.123:0/3233743309 >> [v2:172.21.15.184:6832/1547569255,v1:172.21.15.184:6833/1547569255] conn(0x55f9659cb050 0x55f9659cd440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:50:40.618 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:40.618+0000 7f5df27fc640 10 client.8589 ms_handle_connect on v2:172.21.15.184:6832/1547569255 2024-09-16T15:50:40.618 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:40.618+0000 7f5e12ffd640 1 --2- 172.21.15.123:0/3233743309 >> [v2:172.21.15.184:6832/1547569255,v1:172.21.15.184:6833/1547569255] conn(0x55f9659cb050 0x55f9659cd440 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9d24ab2830319ff1 server_cookie=18a322cb19f64a9a in_seq=0 out_seq=0 2024-09-16T15:50:40.619 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:40.618+0000 7f5df27fc640 1 -- 172.21.15.123:0/3233743309 <== mds.0 v2:172.21.15.184:6832/1547569255 1 ==== command_reply(tid 0: 0 ) ==== 8+0+321 (crc 0 0 0) 0x55f96575e520 con 0x55f9659cb050 2024-09-16T15:50:40.620 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:40.618+0000 7f5df27fc640 10 client.8589 handle_command_reply: tid=0 2024-09-16T15:50:40.620 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:40.618+0000 7f5e1ca36000 1 client.8589 shutdown 2024-09-16T15:50:40.620 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:40.618+0000 7f5e1ca36000 1 -- 172.21.15.123:0/3233743309 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55f96594e9d0 msgr2=0x55f9659636f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:50:40.620 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:40.618+0000 7f5e1ca36000 1 --2- 172.21.15.123:0/3233743309 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55f96594e9d0 0x55f9659636f0 secure :-1 s=READY pgs=1614 cs=0 l=1 rev1=1 crypto rx=0x55f965701130 tx=0x7f5e08020470 comp rx=0 tx=0).stop 2024-09-16T15:50:40.620 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:40.618+0000 7f5e1ca36000 1 -- 172.21.15.123:0/3233743309 shutdown_connections 2024-09-16T15:50:40.620 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:40.618+0000 7f5e1ca36000 1 --2- 172.21.15.123:0/3233743309 >> [v2:172.21.15.184:6832/1547569255,v1:172.21.15.184:6833/1547569255] conn(0x55f9659cb050 0x55f9659cd440 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:40.620 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:40.618+0000 7f5e1ca36000 1 --2- 172.21.15.123:0/3233743309 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x55f96594f2f0 0x55f965963c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:40.620 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:40.618+0000 7f5df27fc640 0 client.8589 ms_handle_reset on v2:172.21.15.184:6832/1547569255 2024-09-16T15:50:40.620 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:40.618+0000 7f5e1ca36000 1 --2- 172.21.15.123:0/3233743309 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55f96594e9d0 0x55f9659636f0 unknown :-1 s=CLOSED pgs=1614 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:40.620 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:40.618+0000 7f5e1ca36000 1 --2- 172.21.15.123:0/3233743309 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x55f965902e10 0x55f9659631b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:40.620 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:40.618+0000 7f5e1ca36000 1 -- 172.21.15.123:0/3233743309 >> 172.21.15.123:0/3233743309 conn(0x55f9657ec130 msgr2=0x55f96594d5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:50:40.620 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:40.618+0000 7f5e1ca36000 1 -- 172.21.15.123:0/3233743309 shutdown_connections 2024-09-16T15:50:40.620 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:40.618+0000 7f5e1ca36000 1 -- 172.21.15.123:0/3233743309 wait complete. 2024-09-16T15:50:40.620 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:40.618+0000 7f5e1ca36000 20 client.8589 trim_cache size 0 max 16384 2024-09-16T15:50:40.621 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2024-09-16T15:50:40.621 INFO:teuthology.orchestra.run.smithi123.stdout: "cluster_fsid": "56077f89-18c8-486f-ae7e-eecc0bb7822c", 2024-09-16T15:50:40.621 INFO:teuthology.orchestra.run.smithi123.stdout: "whoami": 0, 2024-09-16T15:50:40.621 INFO:teuthology.orchestra.run.smithi123.stdout: "id": 8436, 2024-09-16T15:50:40.621 INFO:teuthology.orchestra.run.smithi123.stdout: "want_state": "up:active", 2024-09-16T15:50:40.621 INFO:teuthology.orchestra.run.smithi123.stdout: "state": "up:active", 2024-09-16T15:50:40.621 INFO:teuthology.orchestra.run.smithi123.stdout: "fs_name": "cephfs", 2024-09-16T15:50:40.621 INFO:teuthology.orchestra.run.smithi123.stdout: "rank_uptime": 10.203300851, 2024-09-16T15:50:40.621 INFO:teuthology.orchestra.run.smithi123.stdout: "mdsmap_epoch": 95, 2024-09-16T15:50:40.621 INFO:teuthology.orchestra.run.smithi123.stdout: "osdmap_epoch": 146, 2024-09-16T15:50:40.622 INFO:teuthology.orchestra.run.smithi123.stdout: "osdmap_epoch_barrier": 146, 2024-09-16T15:50:40.622 INFO:teuthology.orchestra.run.smithi123.stdout: "uptime": 14.833140831 2024-09-16T15:50:40.622 INFO:teuthology.orchestra.run.smithi123.stdout:} 2024-09-16T15:50:40.622 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:40.618+0000 7f5e1994c640 1 -- 172.21.15.123:0/3620421730 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f5e00073580 msgr2=0x7f5e00075a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:50:40.622 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:40.618+0000 7f5e1994c640 1 --2- 172.21.15.123:0/3620421730 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f5e00073580 0x7f5e00075a40 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f5e080058d0 tx=0x7f5e08005860 comp rx=0 tx=0).stop 2024-09-16T15:50:40.622 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:40.618+0000 7f5e1994c640 1 -- 172.21.15.123:0/3620421730 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5e14074700 msgr2=0x7f5e14157ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:50:40.623 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:40.618+0000 7f5e1994c640 1 --2- 172.21.15.123:0/3620421730 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5e14074700 0x7f5e14157ed0 secure :-1 s=READY pgs=1610 cs=0 l=1 rev1=1 crypto rx=0x7f5e0c000f90 tx=0x7f5e0c009360 comp rx=0 tx=0).stop 2024-09-16T15:50:40.623 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:40.618+0000 7f5e1994c640 1 -- 172.21.15.123:0/3620421730 shutdown_connections 2024-09-16T15:50:40.623 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:40.618+0000 7f5e1994c640 1 --2- 172.21.15.123:0/3620421730 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f5e00073580 0x7f5e00075a40 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:40.623 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:40.618+0000 7f5e1994c640 1 --2- 172.21.15.123:0/3620421730 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f5e1415f420 0x7f5e1415c9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:40.623 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:40.618+0000 7f5e1994c640 1 --2- 172.21.15.123:0/3620421730 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5e140b2910 0x7f5e140c4990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:40.623 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:40.618+0000 7f5e1994c640 1 --2- 172.21.15.123:0/3620421730 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5e14074700 0x7f5e14157ed0 unknown :-1 s=CLOSED pgs=1610 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:40.623 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:40.618+0000 7f5e1994c640 1 -- 172.21.15.123:0/3620421730 >> 172.21.15.123:0/3620421730 conn(0x7f5e140b1df0 msgr2=0x7f5e140c7de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:50:40.623 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:40.618+0000 7f5e1994c640 1 -- 172.21.15.123:0/3620421730 shutdown_connections 2024-09-16T15:50:40.623 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:40.618+0000 7f5e1994c640 1 -- 172.21.15.123:0/3620421730 wait complete. 2024-09-16T15:50:40.632 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-09-16T15:50:40.748 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:40.750+0000 7f6a5a023640 1 -- 172.21.15.123:0/2670716147 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6a540cd6e0 msgr2=0x7f6a540cdb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:50:40.748 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:40.750+0000 7f6a5a023640 1 --2- 172.21.15.123:0/2670716147 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6a540cd6e0 0x7f6a540cdb50 secure :-1 s=READY pgs=1615 cs=0 l=1 rev1=1 crypto rx=0x7f6a48009900 tx=0x7f6a4801c850 comp rx=0 tx=0).stop 2024-09-16T15:50:40.748 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:40.750+0000 7f6a5a023640 1 -- 172.21.15.123:0/2670716147 shutdown_connections 2024-09-16T15:50:40.748 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:40.750+0000 7f6a5a023640 1 --2- 172.21.15.123:0/2670716147 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f6a5415e820 0x7f6a5415ecb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:40.748 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:40.750+0000 7f6a5a023640 1 --2- 172.21.15.123:0/2670716147 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6a540cd6e0 0x7f6a540cdb50 unknown :-1 s=CLOSED pgs=1615 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:40.748 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:40.750+0000 7f6a5a023640 1 -- 172.21.15.123:0/2670716147 >> 172.21.15.123:0/2670716147 conn(0x7f6a540cfb10 msgr2=0x7f6a540cff20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:50:40.748 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:40.750+0000 7f6a5a023640 1 -- 172.21.15.123:0/2670716147 shutdown_connections 2024-09-16T15:50:40.748 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:40.750+0000 7f6a5a023640 1 -- 172.21.15.123:0/2670716147 wait complete. 2024-09-16T15:50:40.749 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:40.750+0000 7f6a5a023640 1 Processor -- start 2024-09-16T15:50:40.749 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:40.750+0000 7f6a5a023640 1 -- start start 2024-09-16T15:50:40.750 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:40.750+0000 7f6a5a023640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f6a540cd6e0 0x7f6a5414ec00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:50:40.751 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:40.750+0000 7f6a5a023640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6a5415e160 0x7f6a5414e540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:50:40.751 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:40.750+0000 7f6a5a023640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6a5415e820 0x7f6a5406eee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:50:40.751 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:40.750+0000 7f6a5a023640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f6a540cf5b0 con 0x7f6a5415e160 2024-09-16T15:50:40.751 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:40.750+0000 7f6a5a023640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f6a540cf430 con 0x7f6a5415e820 2024-09-16T15:50:40.751 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:40.750+0000 7f6a5a023640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f6a541683d0 con 0x7f6a540cd6e0 2024-09-16T15:50:40.751 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:40.750+0000 7f6a52ffd640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6a5415e160 0x7f6a5414e540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:50:40.751 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:40.750+0000 7f6a52ffd640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6a5415e160 0x7f6a5414e540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:49862/0 (socket says 172.21.15.123:49862) 2024-09-16T15:50:40.751 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:40.750+0000 7f6a52ffd640 1 -- 172.21.15.123:0/3908188682 learned_addr learned my addr 172.21.15.123:0/3908188682 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:50:40.751 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:40.750+0000 7f6a53fff640 1 --2- 172.21.15.123:0/3908188682 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6a5415e820 0x7f6a5406eee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:50:40.751 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:40.750+0000 7f6a537fe640 1 --2- 172.21.15.123:0/3908188682 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f6a540cd6e0 0x7f6a5414ec00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:50:40.751 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:40.750+0000 7f6a52ffd640 1 -- 172.21.15.123:0/3908188682 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f6a540cd6e0 msgr2=0x7f6a5414ec00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:50:40.751 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:40.750+0000 7f6a52ffd640 1 --2- 172.21.15.123:0/3908188682 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f6a540cd6e0 0x7f6a5414ec00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:40.751 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:40.750+0000 7f6a52ffd640 1 -- 172.21.15.123:0/3908188682 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6a5415e820 msgr2=0x7f6a5406eee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:50:40.751 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:40.750+0000 7f6a52ffd640 1 --2- 172.21.15.123:0/3908188682 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6a5415e820 0x7f6a5406eee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:40.752 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:40.750+0000 7f6a52ffd640 1 -- 172.21.15.123:0/3908188682 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6a540c5760 con 0x7f6a5415e160 2024-09-16T15:50:40.752 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:40.750+0000 7f6a52ffd640 1 --2- 172.21.15.123:0/3908188682 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6a5415e160 0x7f6a5414e540 secure :-1 s=READY pgs=1616 cs=0 l=1 rev1=1 crypto rx=0x7f6a48009850 tx=0x7f6a480a5700 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:50:40.752 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:40.750+0000 7f6a3bfff640 1 -- 172.21.15.123:0/3908188682 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a480b2040 con 0x7f6a5415e160 2024-09-16T15:50:40.752 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:40.750+0000 7f6a3bfff640 1 -- 172.21.15.123:0/3908188682 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f6a48004400 con 0x7f6a5415e160 2024-09-16T15:50:40.752 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:40.750+0000 7f6a5a023640 1 -- 172.21.15.123:0/3908188682 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6a540b4340 con 0x7f6a5415e160 2024-09-16T15:50:40.752 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:40.750+0000 7f6a3bfff640 1 -- 172.21.15.123:0/3908188682 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a48005290 con 0x7f6a5415e160 2024-09-16T15:50:40.752 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:40.750+0000 7f6a5a023640 1 -- 172.21.15.123:0/3908188682 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6a540b4900 con 0x7f6a5415e160 2024-09-16T15:50:40.753 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:40.750+0000 7f6a5a023640 1 -- 172.21.15.123:0/3908188682 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6a540cdb50 con 0x7f6a5415e160 2024-09-16T15:50:40.753 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:40.754+0000 7f6a3bfff640 1 -- 172.21.15.123:0/3908188682 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 22) ==== 97692+0+0 (secure 0 0 0) 0x7f6a48005430 con 0x7f6a5415e160 2024-09-16T15:50:40.753 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:40.754+0000 7f6a3bfff640 1 --2- 172.21.15.123:0/3908188682 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f6a40073580 0x7f6a40075a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:50:40.753 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:40.754+0000 7f6a3bfff640 1 -- 172.21.15.123:0/3908188682 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(146..146 src has 1..146) ==== 5737+0+0 (secure 0 0 0) 0x7f6a4812e4c0 con 0x7f6a5415e160 2024-09-16T15:50:40.753 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:40.754+0000 7f6a537fe640 1 --2- 172.21.15.123:0/3908188682 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f6a40073580 0x7f6a40075a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:50:40.753 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:40.754+0000 7f6a537fe640 1 --2- 172.21.15.123:0/3908188682 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f6a40073580 0x7f6a40075a40 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f6a541f8a70 tx=0x7f6a44007e40 comp rx=0 tx=0).ready entity=mgr.4150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:50:40.757 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:40.758+0000 7f6a3bfff640 1 -- 172.21.15.123:0/3908188682 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f6a480164d0 con 0x7f6a5415e160 2024-09-16T15:50:40.951 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:40.950+0000 7f6a5a023640 1 -- 172.21.15.123:0/3908188682 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f6a540d06c0 con 0x7f6a5415e160 2024-09-16T15:50:40.952 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:40.954+0000 7f6a3bfff640 1 -- 172.21.15.123:0/3908188682 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v95) ==== 53+0+73 (secure 0 0 0) 0x7f6a480f8790 con 0x7f6a5415e160 2024-09-16T15:50:40.952 INFO:teuthology.orchestra.run.smithi123.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-09-16T15:50:40.954 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:40.954+0000 7f6a5a023640 1 -- 172.21.15.123:0/3908188682 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f6a40073580 msgr2=0x7f6a40075a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:50:40.955 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:40.954+0000 7f6a5a023640 1 --2- 172.21.15.123:0/3908188682 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f6a40073580 0x7f6a40075a40 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f6a541f8a70 tx=0x7f6a44007e40 comp rx=0 tx=0).stop 2024-09-16T15:50:40.955 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:40.954+0000 7f6a5a023640 1 -- 172.21.15.123:0/3908188682 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6a5415e160 msgr2=0x7f6a5414e540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:50:40.955 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:40.954+0000 7f6a5a023640 1 --2- 172.21.15.123:0/3908188682 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6a5415e160 0x7f6a5414e540 secure :-1 s=READY pgs=1616 cs=0 l=1 rev1=1 crypto rx=0x7f6a48009850 tx=0x7f6a480a5700 comp rx=0 tx=0).stop 2024-09-16T15:50:40.955 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:40.954+0000 7f6a5a023640 1 -- 172.21.15.123:0/3908188682 shutdown_connections 2024-09-16T15:50:40.955 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:40.954+0000 7f6a5a023640 1 --2- 172.21.15.123:0/3908188682 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f6a40073580 0x7f6a40075a40 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:40.955 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:40.954+0000 7f6a5a023640 1 --2- 172.21.15.123:0/3908188682 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6a5415e820 0x7f6a5406eee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:40.956 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:40.954+0000 7f6a5a023640 1 --2- 172.21.15.123:0/3908188682 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6a5415e160 0x7f6a5414e540 unknown :-1 s=CLOSED pgs=1616 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:40.956 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:40.954+0000 7f6a5a023640 1 --2- 172.21.15.123:0/3908188682 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f6a540cd6e0 0x7f6a5414ec00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:40.956 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:40.954+0000 7f6a5a023640 1 -- 172.21.15.123:0/3908188682 >> 172.21.15.123:0/3908188682 conn(0x7f6a540cfb10 msgr2=0x7f6a540b0160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:50:40.956 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:40.954+0000 7f6a5a023640 1 -- 172.21.15.123:0/3908188682 shutdown_connections 2024-09-16T15:50:40.956 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:40.954+0000 7f6a5a023640 1 -- 172.21.15.123:0/3908188682 wait complete. 2024-09-16T15:50:40.969 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-09-16T15:50:40.969 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-09-16T15:50:40.969 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi123.front.sepia.ceph.com 2024-09-16T15:50:40.969 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-09-16T15:50:40.970 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-09-16T15:50:40.970 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-09-16T15:50:40.970 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-09-16T15:50:40.970 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-09-16T15:50:40.970 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-09-16T15:50:40.970 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:50:40.970 DEBUG:teuthology.orchestra.run.smithi123:> ip addr 2024-09-16T15:50:40.976 INFO:teuthology.orchestra.run.smithi123.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-09-16T15:50:40.976 INFO:teuthology.orchestra.run.smithi123.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-09-16T15:50:40.976 INFO:teuthology.orchestra.run.smithi123.stdout: inet 127.0.0.1/8 scope host lo 2024-09-16T15:50:40.976 INFO:teuthology.orchestra.run.smithi123.stdout: valid_lft forever preferred_lft forever 2024-09-16T15:50:40.976 INFO:teuthology.orchestra.run.smithi123.stdout: inet6 ::1/128 scope host 2024-09-16T15:50:40.976 INFO:teuthology.orchestra.run.smithi123.stdout: valid_lft forever preferred_lft forever 2024-09-16T15:50:40.977 INFO:teuthology.orchestra.run.smithi123.stdout:2: eno1: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-09-16T15:50:40.977 INFO:teuthology.orchestra.run.smithi123.stdout: link/ether 0c:c4:7a:d8:da:6c brd ff:ff:ff:ff:ff:ff 2024-09-16T15:50:40.977 INFO:teuthology.orchestra.run.smithi123.stdout: altname enp7s0f0 2024-09-16T15:50:40.977 INFO:teuthology.orchestra.run.smithi123.stdout:3: eno2: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-09-16T15:50:40.977 INFO:teuthology.orchestra.run.smithi123.stdout: link/ether 0c:c4:7a:d8:da:6d brd ff:ff:ff:ff:ff:ff 2024-09-16T15:50:40.977 INFO:teuthology.orchestra.run.smithi123.stdout: altname enp7s0f1 2024-09-16T15:50:40.977 INFO:teuthology.orchestra.run.smithi123.stdout:4: enp3s0f0: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-09-16T15:50:40.977 INFO:teuthology.orchestra.run.smithi123.stdout: link/ether 0c:c4:7a:8f:d3:f0 brd ff:ff:ff:ff:ff:ff 2024-09-16T15:50:40.977 INFO:teuthology.orchestra.run.smithi123.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-09-16T15:50:40.977 INFO:teuthology.orchestra.run.smithi123.stdout: link/ether 0c:c4:7a:8f:d3:f1 brd ff:ff:ff:ff:ff:ff 2024-09-16T15:50:40.977 INFO:teuthology.orchestra.run.smithi123.stdout: inet 172.21.15.123/20 metric 100 brd 172.21.15.255 scope global dynamic enp3s0f1 2024-09-16T15:50:40.977 INFO:teuthology.orchestra.run.smithi123.stdout: valid_lft 42027sec preferred_lft 42027sec 2024-09-16T15:50:40.978 INFO:teuthology.orchestra.run.smithi123.stdout: inet6 fe80::ec4:7aff:fe8f:d3f1/64 scope link 2024-09-16T15:50:40.978 INFO:teuthology.orchestra.run.smithi123.stdout: valid_lft forever preferred_lft forever 2024-09-16T15:50:40.978 INFO:teuthology.orchestra.run.smithi123.stdout:6: docker0: mtu 1500 qdisc noqueue state DOWN group default 2024-09-16T15:50:40.978 INFO:teuthology.orchestra.run.smithi123.stdout: link/ether 02:42:dd:82:a2:6e brd ff:ff:ff:ff:ff:ff 2024-09-16T15:50:40.978 INFO:teuthology.orchestra.run.smithi123.stdout: inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0 2024-09-16T15:50:40.978 INFO:teuthology.orchestra.run.smithi123.stdout: valid_lft forever preferred_lft forever 2024-09-16T15:50:40.978 INFO:teuthology.orchestra.run.smithi123.stdout:7: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-09-16T15:50:40.978 INFO:teuthology.orchestra.run.smithi123.stdout: link/ether 5e:1f:84:6d:79:ae brd ff:ff:ff:ff:ff:ff 2024-09-16T15:50:40.978 INFO:teuthology.orchestra.run.smithi123.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-09-16T15:50:40.978 INFO:teuthology.orchestra.run.smithi123.stdout: valid_lft forever preferred_lft forever 2024-09-16T15:50:40.978 INFO:teuthology.orchestra.run.smithi123.stdout: inet6 fe80::5c1f:84ff:fe6d:79ae/64 scope link 2024-09-16T15:50:40.978 INFO:teuthology.orchestra.run.smithi123.stdout: valid_lft forever preferred_lft forever 2024-09-16T15:50:40.978 INFO:teuthology.orchestra.run.smithi123.stdout:8: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-09-16T15:50:40.978 INFO:teuthology.orchestra.run.smithi123.stdout: link/ether 0e:5c:f6:1f:82:2f brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-09-16T15:50:40.979 INFO:teuthology.orchestra.run.smithi123.stdout: inet6 fe80::c5c:f6ff:fe1f:822f/64 scope link 2024-09-16T15:50:40.979 INFO:teuthology.orchestra.run.smithi123.stdout: valid_lft forever preferred_lft forever 2024-09-16T15:50:40.979 INFO:teuthology.orchestra.run.smithi123.stdout:9: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-09-16T15:50:40.979 INFO:teuthology.orchestra.run.smithi123.stdout: link/ether c6:a6:3d:69:d0:db brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-09-16T15:50:40.979 INFO:teuthology.orchestra.run.smithi123.stdout: inet6 fe80::c4a6:3dff:fe69:d0db/64 scope link 2024-09-16T15:50:40.979 INFO:teuthology.orchestra.run.smithi123.stdout: valid_lft forever preferred_lft forever 2024-09-16T15:50:40.979 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:50:40.979 DEBUG:teuthology.orchestra.run.smithi123:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-09-16T15:50:41.030 INFO:teuthology.orchestra.run.smithi123.stdout:1 2024-09-16T15:50:41.031 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:50:41.031 DEBUG:teuthology.orchestra.run.smithi123:> route 2024-09-16T15:50:41.084 INFO:teuthology.orchestra.run.smithi123.stdout:Kernel IP routing table 2024-09-16T15:50:41.084 INFO:teuthology.orchestra.run.smithi123.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-09-16T15:50:41.084 INFO:teuthology.orchestra.run.smithi123.stdout:default _gateway 0.0.0.0 UG 0 0 0 enp3s0f1 2024-09-16T15:50:41.084 INFO:teuthology.orchestra.run.smithi123.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-09-16T15:50:41.084 INFO:teuthology.orchestra.run.smithi123.stdout:158.69.67.47 _gateway 255.255.255.255 UGH 100 0 0 enp3s0f1 2024-09-16T15:50:41.084 INFO:teuthology.orchestra.run.smithi123.stdout:172.17.0.0 0.0.0.0 255.255.0.0 U 0 0 0 docker0 2024-09-16T15:50:41.084 INFO:teuthology.orchestra.run.smithi123.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-09-16T15:50:41.084 INFO:teuthology.orchestra.run.smithi123.stdout:ns1.front.sepia 0.0.0.0 255.255.255.255 UH 100 0 0 enp3s0f1 2024-09-16T15:50:41.084 INFO:teuthology.orchestra.run.smithi123.stdout:172.21.0.2 0.0.0.0 255.255.255.255 UH 100 0 0 enp3s0f1 2024-09-16T15:50:41.084 INFO:teuthology.orchestra.run.smithi123.stdout:_gateway 0.0.0.0 255.255.255.255 UH 100 0 0 enp3s0f1 2024-09-16T15:50:41.084 INFO:teuthology.orchestra.run.smithi123.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-09-16T15:50:41.085 DEBUG:teuthology.orchestra.run.smithi123:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-09-16T15:50:41.085 DEBUG:teuthology.orchestra.run.smithi123:> set -e 2024-09-16T15:50:41.085 DEBUG:teuthology.orchestra.run.smithi123:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-09-16T15:50:41.085 DEBUG:teuthology.orchestra.run.smithi123:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-09-16T15:50:41.085 DEBUG:teuthology.orchestra.run.smithi123:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-09-16T15:50:41.085 DEBUG:teuthology.orchestra.run.smithi123:> ') 2024-09-16T15:50:41.170 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:41 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-09-16T15:50:41.199 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:41 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-16T15:50:41.206 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:50:41.206 DEBUG:teuthology.orchestra.run.smithi123:> ip netns list 2024-09-16T15:50:41.252 INFO:teuthology.orchestra.run.smithi123.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-09-16T15:50:41.266 INFO:teuthology.orchestra.run.smithi123.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-09-16T15:50:41.266 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-09-16T15:50:41.267 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:50:41.267 DEBUG:teuthology.orchestra.run.smithi123:> sudo ip link set brx.0 up 2024-09-16T15:50:41.312 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2024-09-16T15:50:41.312 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-09-16T15:50:41.312 DEBUG:teuthology.orchestra.run.smithi123:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-09-16T15:50:41.360 INFO:teuthology.orchestra.run.smithi123.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-09-16T15:50:41.360 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-09-16T15:50:41.360 DEBUG:teuthology.orchestra.run.smithi123:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-09-16T15:50:41.412 DEBUG:teuthology.orchestra.run.smithi123:> sudo modprobe fuse 2024-09-16T15:50:41.470 DEBUG:teuthology.orchestra.run.smithi123:> cat /proc/self/mounts | awk '{print $2}' 2024-09-16T15:50:41.519 INFO:teuthology.orchestra.run.smithi123.stdout:/sys 2024-09-16T15:50:41.519 INFO:teuthology.orchestra.run.smithi123.stdout:/proc 2024-09-16T15:50:41.519 INFO:teuthology.orchestra.run.smithi123.stdout:/dev 2024-09-16T15:50:41.519 INFO:teuthology.orchestra.run.smithi123.stdout:/dev/pts 2024-09-16T15:50:41.519 INFO:teuthology.orchestra.run.smithi123.stdout:/run 2024-09-16T15:50:41.519 INFO:teuthology.orchestra.run.smithi123.stdout:/ 2024-09-16T15:50:41.519 INFO:teuthology.orchestra.run.smithi123.stdout:/sys/kernel/security 2024-09-16T15:50:41.519 INFO:teuthology.orchestra.run.smithi123.stdout:/dev/shm 2024-09-16T15:50:41.519 INFO:teuthology.orchestra.run.smithi123.stdout:/run/lock 2024-09-16T15:50:41.519 INFO:teuthology.orchestra.run.smithi123.stdout:/sys/fs/cgroup 2024-09-16T15:50:41.519 INFO:teuthology.orchestra.run.smithi123.stdout:/sys/fs/pstore 2024-09-16T15:50:41.519 INFO:teuthology.orchestra.run.smithi123.stdout:/sys/fs/bpf 2024-09-16T15:50:41.520 INFO:teuthology.orchestra.run.smithi123.stdout:/proc/sys/fs/binfmt_misc 2024-09-16T15:50:41.520 INFO:teuthology.orchestra.run.smithi123.stdout:/dev/hugepages 2024-09-16T15:50:41.520 INFO:teuthology.orchestra.run.smithi123.stdout:/dev/mqueue 2024-09-16T15:50:41.520 INFO:teuthology.orchestra.run.smithi123.stdout:/sys/kernel/debug 2024-09-16T15:50:41.520 INFO:teuthology.orchestra.run.smithi123.stdout:/sys/kernel/tracing 2024-09-16T15:50:41.520 INFO:teuthology.orchestra.run.smithi123.stdout:/proc/fs/nfsd 2024-09-16T15:50:41.520 INFO:teuthology.orchestra.run.smithi123.stdout:/sys/fs/fuse/connections 2024-09-16T15:50:41.520 INFO:teuthology.orchestra.run.smithi123.stdout:/sys/kernel/config 2024-09-16T15:50:41.520 INFO:teuthology.orchestra.run.smithi123.stdout:/run/credentials/systemd-sysusers.service 2024-09-16T15:50:41.520 INFO:teuthology.orchestra.run.smithi123.stdout:/run/rpc_pipefs 2024-09-16T15:50:41.520 INFO:teuthology.orchestra.run.smithi123.stdout:/run/user/1000 2024-09-16T15:50:41.520 INFO:teuthology.orchestra.run.smithi123.stdout:/var/lib/ceph 2024-09-16T15:50:41.520 INFO:teuthology.orchestra.run.smithi123.stdout:/var/lib/ceph/osd/ceph-0 2024-09-16T15:50:41.521 INFO:teuthology.orchestra.run.smithi123.stdout:/var/lib/ceph/osd/ceph-1 2024-09-16T15:50:41.521 INFO:teuthology.orchestra.run.smithi123.stdout:/var/lib/ceph/osd/ceph-2 2024-09-16T15:50:41.521 INFO:teuthology.orchestra.run.smithi123.stdout:/var/lib/ceph/osd/ceph-3 2024-09-16T15:50:41.521 INFO:teuthology.orchestra.run.smithi123.stdout:/run/netns 2024-09-16T15:50:41.521 INFO:teuthology.orchestra.run.smithi123.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-09-16T15:50:41.521 INFO:teuthology.orchestra.run.smithi123.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-09-16T15:50:41.521 INFO:teuthology.orchestra.run.smithi123.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-09-16T15:50:41.521 INFO:teuthology.orchestra.run.smithi123.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-09-16T15:50:41.522 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:50:41.522 DEBUG:teuthology.orchestra.run.smithi123:> ls /sys/fs/fuse/connections 2024-09-16T15:50:41.568 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-09-16T15:50:41.568 DEBUG:teuthology.orchestra.run.smithi123:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2024-09-16T15:50:41.613 DEBUG:teuthology.orchestra.run.smithi123:> sudo modprobe fuse 2024-09-16T15:50:41.629 DEBUG:teuthology.orchestra.run.smithi123:> cat /proc/self/mounts | awk '{print $2}' 2024-09-16T15:50:41.676 INFO:teuthology.orchestra.run.smithi123.stdout:/sys 2024-09-16T15:50:41.676 INFO:teuthology.orchestra.run.smithi123.stdout:/proc 2024-09-16T15:50:41.676 INFO:teuthology.orchestra.run.smithi123.stdout:/dev 2024-09-16T15:50:41.676 INFO:teuthology.orchestra.run.smithi123.stdout:/dev/pts 2024-09-16T15:50:41.676 INFO:teuthology.orchestra.run.smithi123.stdout:/run 2024-09-16T15:50:41.677 INFO:teuthology.orchestra.run.smithi123.stdout:/ 2024-09-16T15:50:41.677 INFO:teuthology.orchestra.run.smithi123.stdout:/sys/kernel/security 2024-09-16T15:50:41.677 INFO:teuthology.orchestra.run.smithi123.stdout:/dev/shm 2024-09-16T15:50:41.677 INFO:teuthology.orchestra.run.smithi123.stdout:/run/lock 2024-09-16T15:50:41.677 INFO:teuthology.orchestra.run.smithi123.stdout:/sys/fs/cgroup 2024-09-16T15:50:41.677 INFO:teuthology.orchestra.run.smithi123.stdout:/sys/fs/pstore 2024-09-16T15:50:41.677 INFO:teuthology.orchestra.run.smithi123.stdout:/sys/fs/bpf 2024-09-16T15:50:41.680 INFO:teuthology.orchestra.run.smithi123.stdout:/proc/sys/fs/binfmt_misc 2024-09-16T15:50:41.680 INFO:teuthology.orchestra.run.smithi123.stdout:/dev/hugepages 2024-09-16T15:50:41.680 INFO:teuthology.orchestra.run.smithi123.stdout:/dev/mqueue 2024-09-16T15:50:41.680 INFO:teuthology.orchestra.run.smithi123.stdout:/sys/kernel/debug 2024-09-16T15:50:41.680 INFO:teuthology.orchestra.run.smithi123.stdout:/sys/kernel/tracing 2024-09-16T15:50:41.680 INFO:teuthology.orchestra.run.smithi123.stdout:/proc/fs/nfsd 2024-09-16T15:50:41.680 INFO:teuthology.orchestra.run.smithi123.stdout:/sys/fs/fuse/connections 2024-09-16T15:50:41.680 INFO:teuthology.orchestra.run.smithi123.stdout:/sys/kernel/config 2024-09-16T15:50:41.680 INFO:teuthology.orchestra.run.smithi123.stdout:/run/credentials/systemd-sysusers.service 2024-09-16T15:50:41.680 INFO:teuthology.orchestra.run.smithi123.stdout:/run/rpc_pipefs 2024-09-16T15:50:41.681 INFO:teuthology.orchestra.run.smithi123.stdout:/run/user/1000 2024-09-16T15:50:41.681 INFO:teuthology.orchestra.run.smithi123.stdout:/var/lib/ceph 2024-09-16T15:50:41.681 INFO:teuthology.orchestra.run.smithi123.stdout:/var/lib/ceph/osd/ceph-0 2024-09-16T15:50:41.681 INFO:teuthology.orchestra.run.smithi123.stdout:/var/lib/ceph/osd/ceph-1 2024-09-16T15:50:41.681 INFO:teuthology.orchestra.run.smithi123.stdout:/var/lib/ceph/osd/ceph-2 2024-09-16T15:50:41.681 INFO:teuthology.orchestra.run.smithi123.stdout:/var/lib/ceph/osd/ceph-3 2024-09-16T15:50:41.681 INFO:teuthology.orchestra.run.smithi123.stdout:/run/netns 2024-09-16T15:50:41.681 INFO:teuthology.orchestra.run.smithi123.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-09-16T15:50:41.681 INFO:teuthology.orchestra.run.smithi123.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-09-16T15:50:41.681 INFO:teuthology.orchestra.run.smithi123.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-09-16T15:50:41.681 INFO:teuthology.orchestra.run.smithi123.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-09-16T15:50:41.682 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:50:41.682 DEBUG:teuthology.orchestra.run.smithi123:> ls /sys/fs/fuse/connections 2024-09-16T15:50:41.695 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi123.stderr:2024-09-16T15:50:41.694+0000 7f8f1d374580 -1 init, newargv = 0x558239eadd20 newargc=15 2024-09-16T15:50:41.695 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi123.stderr:ceph-fuse[51760]: starting ceph client 2024-09-16T15:50:41.703 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi123.stderr:ceph-fuse[51760]: starting fuse 2024-09-16T15:50:42.800 DEBUG:teuthology.orchestra.run.smithi123:> sudo modprobe fuse 2024-09-16T15:50:42.816 DEBUG:teuthology.orchestra.run.smithi123:> cat /proc/self/mounts | awk '{print $2}' 2024-09-16T15:50:42.865 INFO:teuthology.orchestra.run.smithi123.stdout:/sys 2024-09-16T15:50:42.865 INFO:teuthology.orchestra.run.smithi123.stdout:/proc 2024-09-16T15:50:42.865 INFO:teuthology.orchestra.run.smithi123.stdout:/dev 2024-09-16T15:50:42.865 INFO:teuthology.orchestra.run.smithi123.stdout:/dev/pts 2024-09-16T15:50:42.866 INFO:teuthology.orchestra.run.smithi123.stdout:/run 2024-09-16T15:50:42.866 INFO:teuthology.orchestra.run.smithi123.stdout:/ 2024-09-16T15:50:42.866 INFO:teuthology.orchestra.run.smithi123.stdout:/sys/kernel/security 2024-09-16T15:50:42.866 INFO:teuthology.orchestra.run.smithi123.stdout:/dev/shm 2024-09-16T15:50:42.866 INFO:teuthology.orchestra.run.smithi123.stdout:/run/lock 2024-09-16T15:50:42.866 INFO:teuthology.orchestra.run.smithi123.stdout:/sys/fs/cgroup 2024-09-16T15:50:42.866 INFO:teuthology.orchestra.run.smithi123.stdout:/sys/fs/pstore 2024-09-16T15:50:42.866 INFO:teuthology.orchestra.run.smithi123.stdout:/sys/fs/bpf 2024-09-16T15:50:42.866 INFO:teuthology.orchestra.run.smithi123.stdout:/proc/sys/fs/binfmt_misc 2024-09-16T15:50:42.866 INFO:teuthology.orchestra.run.smithi123.stdout:/dev/hugepages 2024-09-16T15:50:42.866 INFO:teuthology.orchestra.run.smithi123.stdout:/dev/mqueue 2024-09-16T15:50:42.866 INFO:teuthology.orchestra.run.smithi123.stdout:/sys/kernel/debug 2024-09-16T15:50:42.866 INFO:teuthology.orchestra.run.smithi123.stdout:/sys/kernel/tracing 2024-09-16T15:50:42.866 INFO:teuthology.orchestra.run.smithi123.stdout:/proc/fs/nfsd 2024-09-16T15:50:42.867 INFO:teuthology.orchestra.run.smithi123.stdout:/sys/fs/fuse/connections 2024-09-16T15:50:42.867 INFO:teuthology.orchestra.run.smithi123.stdout:/sys/kernel/config 2024-09-16T15:50:42.867 INFO:teuthology.orchestra.run.smithi123.stdout:/run/credentials/systemd-sysusers.service 2024-09-16T15:50:42.867 INFO:teuthology.orchestra.run.smithi123.stdout:/run/rpc_pipefs 2024-09-16T15:50:42.867 INFO:teuthology.orchestra.run.smithi123.stdout:/run/user/1000 2024-09-16T15:50:42.867 INFO:teuthology.orchestra.run.smithi123.stdout:/var/lib/ceph 2024-09-16T15:50:42.867 INFO:teuthology.orchestra.run.smithi123.stdout:/var/lib/ceph/osd/ceph-0 2024-09-16T15:50:42.867 INFO:teuthology.orchestra.run.smithi123.stdout:/var/lib/ceph/osd/ceph-1 2024-09-16T15:50:42.867 INFO:teuthology.orchestra.run.smithi123.stdout:/var/lib/ceph/osd/ceph-2 2024-09-16T15:50:42.867 INFO:teuthology.orchestra.run.smithi123.stdout:/var/lib/ceph/osd/ceph-3 2024-09-16T15:50:42.867 INFO:teuthology.orchestra.run.smithi123.stdout:/run/netns 2024-09-16T15:50:42.867 INFO:teuthology.orchestra.run.smithi123.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-09-16T15:50:42.868 INFO:teuthology.orchestra.run.smithi123.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-09-16T15:50:42.868 INFO:teuthology.orchestra.run.smithi123.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-09-16T15:50:42.868 INFO:teuthology.orchestra.run.smithi123.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-09-16T15:50:42.868 INFO:teuthology.orchestra.run.smithi123.stdout:/home/ubuntu/cephtest/mnt.0 2024-09-16T15:50:42.869 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:50:42.869 DEBUG:teuthology.orchestra.run.smithi123:> ls /sys/fs/fuse/connections 2024-09-16T15:50:42.920 INFO:teuthology.orchestra.run.smithi123.stdout:45 2024-09-16T15:50:42.920 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [45] 2024-09-16T15:50:42.920 DEBUG:teuthology.orchestra.run.smithi123:> sudo stdin-killer -- python3 -c ' 2024-09-16T15:50:42.920 DEBUG:teuthology.orchestra.run.smithi123:> import glob 2024-09-16T15:50:42.921 DEBUG:teuthology.orchestra.run.smithi123:> import re 2024-09-16T15:50:42.921 DEBUG:teuthology.orchestra.run.smithi123:> import os 2024-09-16T15:50:42.921 DEBUG:teuthology.orchestra.run.smithi123:> import subprocess 2024-09-16T15:50:42.921 DEBUG:teuthology.orchestra.run.smithi123:> 2024-09-16T15:50:42.921 DEBUG:teuthology.orchestra.run.smithi123:> def _find_admin_socket(client_name): 2024-09-16T15:50:42.921 DEBUG:teuthology.orchestra.run.smithi123:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2024-09-16T15:50:42.921 DEBUG:teuthology.orchestra.run.smithi123:> files = glob.glob(asok_path) 2024-09-16T15:50:42.921 DEBUG:teuthology.orchestra.run.smithi123:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-09-16T15:50:42.921 DEBUG:teuthology.orchestra.run.smithi123:> 2024-09-16T15:50:42.921 DEBUG:teuthology.orchestra.run.smithi123:> # Given a non-glob path, it better be there 2024-09-16T15:50:42.921 DEBUG:teuthology.orchestra.run.smithi123:> if "*" not in asok_path: 2024-09-16T15:50:42.921 DEBUG:teuthology.orchestra.run.smithi123:> assert(len(files) == 1) 2024-09-16T15:50:42.921 DEBUG:teuthology.orchestra.run.smithi123:> return files[0] 2024-09-16T15:50:42.921 DEBUG:teuthology.orchestra.run.smithi123:> 2024-09-16T15:50:42.921 DEBUG:teuthology.orchestra.run.smithi123:> for f in files: 2024-09-16T15:50:42.922 DEBUG:teuthology.orchestra.run.smithi123:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-09-16T15:50:42.922 DEBUG:teuthology.orchestra.run.smithi123:> if os.path.exists("/proc/{0}".format(pid)): 2024-09-16T15:50:42.922 DEBUG:teuthology.orchestra.run.smithi123:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-09-16T15:50:42.922 DEBUG:teuthology.orchestra.run.smithi123:> contents = proc_f.read() 2024-09-16T15:50:42.922 DEBUG:teuthology.orchestra.run.smithi123:> if mountpoint in contents: 2024-09-16T15:50:42.922 DEBUG:teuthology.orchestra.run.smithi123:> return f 2024-09-16T15:50:42.922 DEBUG:teuthology.orchestra.run.smithi123:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-09-16T15:50:42.922 DEBUG:teuthology.orchestra.run.smithi123:> 2024-09-16T15:50:42.922 DEBUG:teuthology.orchestra.run.smithi123:> print(_find_admin_socket("client.0")) 2024-09-16T15:50:42.922 DEBUG:teuthology.orchestra.run.smithi123:> ' 2024-09-16T15:50:43.044 INFO:teuthology.orchestra.run.smithi123.stdout:/var/run/ceph/ceph-client.0.51760.asok 2024-09-16T15:50:43.048 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:43 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-16T15:50:43.056 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.51760.asok 2024-09-16T15:50:43.056 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:50:43.056 DEBUG:teuthology.orchestra.run.smithi123:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.51760.asok status 2024-09-16T15:50:43.210 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2024-09-16T15:50:43.210 INFO:teuthology.orchestra.run.smithi123.stdout: "metadata": { 2024-09-16T15:50:43.210 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph_sha1": "33f95d2ae1bcbe7eb04404303f179eeeaa6982df", 2024-09-16T15:50:43.210 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph_version": "ceph version 19.3.0-4938-g33f95d2a (33f95d2ae1bcbe7eb04404303f179eeeaa6982df) squid (dev)", 2024-09-16T15:50:43.210 INFO:teuthology.orchestra.run.smithi123.stdout: "entity_id": "0", 2024-09-16T15:50:43.210 INFO:teuthology.orchestra.run.smithi123.stdout: "hostname": "smithi123", 2024-09-16T15:50:43.210 INFO:teuthology.orchestra.run.smithi123.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2024-09-16T15:50:43.210 INFO:teuthology.orchestra.run.smithi123.stdout: "pid": "51760", 2024-09-16T15:50:43.210 INFO:teuthology.orchestra.run.smithi123.stdout: "root": "/" 2024-09-16T15:50:43.211 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:50:43.211 INFO:teuthology.orchestra.run.smithi123.stdout: "dentry_count": 0, 2024-09-16T15:50:43.211 INFO:teuthology.orchestra.run.smithi123.stdout: "dentry_pinned_count": 0, 2024-09-16T15:50:43.211 INFO:teuthology.orchestra.run.smithi123.stdout: "id": 8598, 2024-09-16T15:50:43.211 INFO:teuthology.orchestra.run.smithi123.stdout: "inst": { 2024-09-16T15:50:43.211 INFO:teuthology.orchestra.run.smithi123.stdout: "name": { 2024-09-16T15:50:43.211 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "client", 2024-09-16T15:50:43.211 INFO:teuthology.orchestra.run.smithi123.stdout: "num": 8598 2024-09-16T15:50:43.211 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:50:43.211 INFO:teuthology.orchestra.run.smithi123.stdout: "addr": { 2024-09-16T15:50:43.211 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "v1", 2024-09-16T15:50:43.211 INFO:teuthology.orchestra.run.smithi123.stdout: "addr": "192.168.0.1:0", 2024-09-16T15:50:43.211 INFO:teuthology.orchestra.run.smithi123.stdout: "nonce": 4074325308 2024-09-16T15:50:43.211 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-16T15:50:43.212 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:50:43.212 INFO:teuthology.orchestra.run.smithi123.stdout: "addr": { 2024-09-16T15:50:43.212 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "v1", 2024-09-16T15:50:43.212 INFO:teuthology.orchestra.run.smithi123.stdout: "addr": "192.168.0.1:0", 2024-09-16T15:50:43.212 INFO:teuthology.orchestra.run.smithi123.stdout: "nonce": 4074325308 2024-09-16T15:50:43.212 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:50:43.212 INFO:teuthology.orchestra.run.smithi123.stdout: "inst_str": "client.8598 192.168.0.1:0/4074325308", 2024-09-16T15:50:43.212 INFO:teuthology.orchestra.run.smithi123.stdout: "addr_str": "192.168.0.1:0/4074325308", 2024-09-16T15:50:43.212 INFO:teuthology.orchestra.run.smithi123.stdout: "inode_count": 1, 2024-09-16T15:50:43.212 INFO:teuthology.orchestra.run.smithi123.stdout: "mds_epoch": 95, 2024-09-16T15:50:43.212 INFO:teuthology.orchestra.run.smithi123.stdout: "osd_epoch": 146, 2024-09-16T15:50:43.212 INFO:teuthology.orchestra.run.smithi123.stdout: "osd_epoch_barrier": 0, 2024-09-16T15:50:43.212 INFO:teuthology.orchestra.run.smithi123.stdout: "blocklisted": false, 2024-09-16T15:50:43.213 INFO:teuthology.orchestra.run.smithi123.stdout: "fs_name": "cephfs" 2024-09-16T15:50:43.213 INFO:teuthology.orchestra.run.smithi123.stdout:} 2024-09-16T15:50:43.223 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:50:43.223 DEBUG:teuthology.orchestra.run.smithi123:> stat --file-system '--printf=%T 2024-09-16T15:50:43.223 DEBUG:teuthology.orchestra.run.smithi123:> ' -- /home/ubuntu/cephtest/mnt.0 2024-09-16T15:50:43.229 INFO:teuthology.orchestra.run.smithi123.stdout:fuseblk 2024-09-16T15:50:43.229 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2024-09-16T15:50:43.229 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:50:43.230 DEBUG:teuthology.orchestra.run.smithi123:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-09-16T15:50:43.283 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-09-16T15:50:43.454 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:43.454+0000 7fb6112d7640 1 -- 172.21.15.123:0/2600886426 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb60c0cf5a0 msgr2=0x7fb60c06eee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:50:43.454 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:43.454+0000 7fb6112d7640 1 --2- 172.21.15.123:0/2600886426 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb60c0cf5a0 0x7fb60c06eee0 secure :-1 s=READY pgs=1618 cs=0 l=1 rev1=1 crypto rx=0x7fb60c076230 tx=0x7fb60001c830 comp rx=0 tx=0).stop 2024-09-16T15:50:43.454 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:43.454+0000 7fb6112d7640 1 -- 172.21.15.123:0/2600886426 shutdown_connections 2024-09-16T15:50:43.454 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:43.454+0000 7fb6112d7640 1 --2- 172.21.15.123:0/2600886426 >> v2:172.21.15.184:3300/0 conn(0x7fb60c15ec40 0x7fb60c06f420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:43.454 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:43.454+0000 7fb6112d7640 1 --2- 172.21.15.123:0/2600886426 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb60c0cf5a0 0x7fb60c06eee0 unknown :-1 s=CLOSED pgs=1618 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:43.454 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:43.454+0000 7fb6112d7640 1 --2- 172.21.15.123:0/2600886426 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fb60c158120 0x7fb60c0ca320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:43.454 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:43.454+0000 7fb6112d7640 1 -- 172.21.15.123:0/2600886426 >> 172.21.15.123:0/2600886426 conn(0x7fb60c0cfb10 msgr2=0x7fb60c0cff20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:50:43.454 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:43.454+0000 7fb6112d7640 1 -- 172.21.15.123:0/2600886426 shutdown_connections 2024-09-16T15:50:43.454 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:43.454+0000 7fb6112d7640 1 -- 172.21.15.123:0/2600886426 wait complete. 2024-09-16T15:50:43.455 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:43.454+0000 7fb6112d7640 1 Processor -- start 2024-09-16T15:50:43.455 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:43.454+0000 7fb6112d7640 1 -- start start 2024-09-16T15:50:43.456 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:43.458+0000 7fb6112d7640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb60c0cf5a0 0x7fb60c138dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:50:43.457 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:43.458+0000 7fb6112d7640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fb60c158120 0x7fb60c14c4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:50:43.457 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:43.458+0000 7fb6112d7640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb60c15ec40 0x7fb60c14b5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:50:43.457 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:43.458+0000 7fb6112d7640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fb60c15b120 con 0x7fb60c0cf5a0 2024-09-16T15:50:43.457 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:43.458+0000 7fb6112d7640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fb60c15afa0 con 0x7fb60c15ec40 2024-09-16T15:50:43.457 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:43.458+0000 7fb6112d7640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fb60c15b2a0 con 0x7fb60c158120 2024-09-16T15:50:43.457 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:43.458+0000 7fb60affd640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb60c0cf5a0 0x7fb60c138dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:50:43.457 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:43.458+0000 7fb60affd640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb60c0cf5a0 0x7fb60c138dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:40276/0 (socket says 172.21.15.123:40276) 2024-09-16T15:50:43.457 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:43.458+0000 7fb60affd640 1 -- 172.21.15.123:0/712844181 learned_addr learned my addr 172.21.15.123:0/712844181 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:50:43.457 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:43.458+0000 7fb60affd640 1 -- 172.21.15.123:0/712844181 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fb60c158120 msgr2=0x7fb60c14c4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:50:43.457 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:43.458+0000 7fb60affd640 1 --2- 172.21.15.123:0/712844181 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fb60c158120 0x7fb60c14c4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:43.457 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:43.458+0000 7fb60affd640 1 -- 172.21.15.123:0/712844181 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb60c15ec40 msgr2=0x7fb60c14b5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:50:43.457 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:43.458+0000 7fb60affd640 1 --2- 172.21.15.123:0/712844181 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb60c15ec40 0x7fb60c14b5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:43.458 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:43.458+0000 7fb60affd640 1 -- 172.21.15.123:0/712844181 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb60c0adb10 con 0x7fb60c0cf5a0 2024-09-16T15:50:43.458 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:43.458+0000 7fb60affd640 1 --2- 172.21.15.123:0/712844181 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb60c0cf5a0 0x7fb60c138dd0 secure :-1 s=READY pgs=1619 cs=0 l=1 rev1=1 crypto rx=0x7fb5fc00c940 tx=0x7fb5fc00ce00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:50:43.458 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:43.458+0000 7fb5f37fe640 1 -- 172.21.15.123:0/712844181 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5fc007bb0 con 0x7fb60c0cf5a0 2024-09-16T15:50:43.458 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:43.458+0000 7fb6112d7640 1 -- 172.21.15.123:0/712844181 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb60c0addf0 con 0x7fb60c0cf5a0 2024-09-16T15:50:43.458 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:43.458+0000 7fb6112d7640 1 -- 172.21.15.123:0/712844181 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb60c0ae3b0 con 0x7fb60c0cf5a0 2024-09-16T15:50:43.458 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:43.458+0000 7fb5f37fe640 1 -- 172.21.15.123:0/712844181 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fb5fc007e40 con 0x7fb60c0cf5a0 2024-09-16T15:50:43.458 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:43.458+0000 7fb5f37fe640 1 -- 172.21.15.123:0/712844181 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5fc005240 con 0x7fb60c0cf5a0 2024-09-16T15:50:43.458 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:43.458+0000 7fb6112d7640 1 -- 172.21.15.123:0/712844181 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb5ec0052a0 con 0x7fb60c0cf5a0 2024-09-16T15:50:43.458 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:43.458+0000 7fb5f37fe640 1 -- 172.21.15.123:0/712844181 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 22) ==== 97692+0+0 (secure 0 0 0) 0x7fb5fc0053e0 con 0x7fb60c0cf5a0 2024-09-16T15:50:43.459 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:43.458+0000 7fb5f37fe640 1 --2- 172.21.15.123:0/712844181 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fb5f8073580 0x7fb5f8075a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:50:43.459 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:43.458+0000 7fb5f37fe640 1 -- 172.21.15.123:0/712844181 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(146..146 src has 1..146) ==== 5737+0+0 (secure 0 0 0) 0x7fb5fc094fc0 con 0x7fb60c0cf5a0 2024-09-16T15:50:43.459 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:43.458+0000 7fb60a7fc640 1 --2- 172.21.15.123:0/712844181 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fb5f8073580 0x7fb5f8075a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:50:43.459 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:43.462+0000 7fb60a7fc640 1 --2- 172.21.15.123:0/712844181 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fb5f8073580 0x7fb5f8075a40 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7fb6000ae000 tx=0x7fb6000195d0 comp rx=0 tx=0).ready entity=mgr.4150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:50:43.463 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:43.466+0000 7fb5f37fe640 1 -- 172.21.15.123:0/712844181 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7fb5fc014030 con 0x7fb60c0cf5a0 2024-09-16T15:50:43.669 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:43.670+0000 7fb6112d7640 1 -- 172.21.15.123:0/712844181 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7fb5ec005860 con 0x7fb60c0cf5a0 2024-09-16T15:50:43.669 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:43.670+0000 7fb5f37fe640 1 -- 172.21.15.123:0/712844181 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v95) ==== 53+0+73 (secure 0 0 0) 0x7fb5fc05f6d0 con 0x7fb60c0cf5a0 2024-09-16T15:50:43.670 INFO:teuthology.orchestra.run.smithi123.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-09-16T15:50:43.672 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:43.674+0000 7fb6112d7640 1 -- 172.21.15.123:0/712844181 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fb5f8073580 msgr2=0x7fb5f8075a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:50:43.672 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:43.674+0000 7fb6112d7640 1 --2- 172.21.15.123:0/712844181 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fb5f8073580 0x7fb5f8075a40 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7fb6000ae000 tx=0x7fb6000195d0 comp rx=0 tx=0).stop 2024-09-16T15:50:43.672 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:43.674+0000 7fb6112d7640 1 -- 172.21.15.123:0/712844181 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb60c0cf5a0 msgr2=0x7fb60c138dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:50:43.672 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:43.674+0000 7fb6112d7640 1 --2- 172.21.15.123:0/712844181 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb60c0cf5a0 0x7fb60c138dd0 secure :-1 s=READY pgs=1619 cs=0 l=1 rev1=1 crypto rx=0x7fb5fc00c940 tx=0x7fb5fc00ce00 comp rx=0 tx=0).stop 2024-09-16T15:50:43.672 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:43.674+0000 7fb6112d7640 1 -- 172.21.15.123:0/712844181 shutdown_connections 2024-09-16T15:50:43.672 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:43.674+0000 7fb6112d7640 1 --2- 172.21.15.123:0/712844181 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fb5f8073580 0x7fb5f8075a40 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:43.672 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:43.674+0000 7fb6112d7640 1 --2- 172.21.15.123:0/712844181 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb60c15ec40 0x7fb60c14b5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:43.673 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:43.674+0000 7fb6112d7640 1 --2- 172.21.15.123:0/712844181 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fb60c158120 0x7fb60c14c4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:43.673 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:43.674+0000 7fb6112d7640 1 --2- 172.21.15.123:0/712844181 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb60c0cf5a0 0x7fb60c138dd0 unknown :-1 s=CLOSED pgs=1619 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:43.673 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:43.674+0000 7fb6112d7640 1 -- 172.21.15.123:0/712844181 >> 172.21.15.123:0/712844181 conn(0x7fb60c0cfb10 msgr2=0x7fb60c159f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:50:43.673 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:43.674+0000 7fb6112d7640 1 -- 172.21.15.123:0/712844181 shutdown_connections 2024-09-16T15:50:43.673 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:43.674+0000 7fb6112d7640 1 -- 172.21.15.123:0/712844181 wait complete. 2024-09-16T15:50:43.686 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-09-16T15:50:43.686 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-09-16T15:50:43.687 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi184.front.sepia.ceph.com 2024-09-16T15:50:43.687 INFO:tasks.cephfs.mount:self.client.name = client.1 2024-09-16T15:50:43.687 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2024-09-16T15:50:43.687 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-09-16T15:50:43.687 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-09-16T15:50:43.687 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-09-16T15:50:43.687 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.1' netns for '/home/ubuntu/cephtest/mnt.1' 2024-09-16T15:50:43.687 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:50:43.687 DEBUG:teuthology.orchestra.run.smithi184:> ip addr 2024-09-16T15:50:43.694 INFO:teuthology.orchestra.run.smithi184.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-09-16T15:50:43.694 INFO:teuthology.orchestra.run.smithi184.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-09-16T15:50:43.694 INFO:teuthology.orchestra.run.smithi184.stdout: inet 127.0.0.1/8 scope host lo 2024-09-16T15:50:43.694 INFO:teuthology.orchestra.run.smithi184.stdout: valid_lft forever preferred_lft forever 2024-09-16T15:50:43.694 INFO:teuthology.orchestra.run.smithi184.stdout: inet6 ::1/128 scope host 2024-09-16T15:50:43.694 INFO:teuthology.orchestra.run.smithi184.stdout: valid_lft forever preferred_lft forever 2024-09-16T15:50:43.694 INFO:teuthology.orchestra.run.smithi184.stdout:2: eno1: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-09-16T15:50:43.695 INFO:teuthology.orchestra.run.smithi184.stdout: link/ether 0c:c4:7a:d9:37:16 brd ff:ff:ff:ff:ff:ff 2024-09-16T15:50:43.695 INFO:teuthology.orchestra.run.smithi184.stdout: altname enp7s0f0 2024-09-16T15:50:43.695 INFO:teuthology.orchestra.run.smithi184.stdout:3: eno2: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-09-16T15:50:43.695 INFO:teuthology.orchestra.run.smithi184.stdout: link/ether 0c:c4:7a:d9:37:17 brd ff:ff:ff:ff:ff:ff 2024-09-16T15:50:43.695 INFO:teuthology.orchestra.run.smithi184.stdout: altname enp7s0f1 2024-09-16T15:50:43.695 INFO:teuthology.orchestra.run.smithi184.stdout:4: enp3s0f0: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-09-16T15:50:43.695 INFO:teuthology.orchestra.run.smithi184.stdout: link/ether 0c:c4:7a:88:7b:8e brd ff:ff:ff:ff:ff:ff 2024-09-16T15:50:43.695 INFO:teuthology.orchestra.run.smithi184.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-09-16T15:50:43.695 INFO:teuthology.orchestra.run.smithi184.stdout: link/ether 0c:c4:7a:88:7b:8f brd ff:ff:ff:ff:ff:ff 2024-09-16T15:50:43.695 INFO:teuthology.orchestra.run.smithi184.stdout: inet 172.21.15.184/20 metric 100 brd 172.21.15.255 scope global dynamic enp3s0f1 2024-09-16T15:50:43.695 INFO:teuthology.orchestra.run.smithi184.stdout: valid_lft 42013sec preferred_lft 42013sec 2024-09-16T15:50:43.695 INFO:teuthology.orchestra.run.smithi184.stdout: inet6 fe80::ec4:7aff:fe88:7b8f/64 scope link 2024-09-16T15:50:43.695 INFO:teuthology.orchestra.run.smithi184.stdout: valid_lft forever preferred_lft forever 2024-09-16T15:50:43.696 INFO:teuthology.orchestra.run.smithi184.stdout:6: docker0: mtu 1500 qdisc noqueue state DOWN group default 2024-09-16T15:50:43.696 INFO:teuthology.orchestra.run.smithi184.stdout: link/ether 02:42:5d:0a:44:b9 brd ff:ff:ff:ff:ff:ff 2024-09-16T15:50:43.696 INFO:teuthology.orchestra.run.smithi184.stdout: inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0 2024-09-16T15:50:43.696 INFO:teuthology.orchestra.run.smithi184.stdout: valid_lft forever preferred_lft forever 2024-09-16T15:50:43.696 INFO:teuthology.orchestra.run.smithi184.stdout:7: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-09-16T15:50:43.696 INFO:teuthology.orchestra.run.smithi184.stdout: link/ether 5e:1f:84:6d:79:ae brd ff:ff:ff:ff:ff:ff 2024-09-16T15:50:43.696 INFO:teuthology.orchestra.run.smithi184.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-09-16T15:50:43.696 INFO:teuthology.orchestra.run.smithi184.stdout: valid_lft forever preferred_lft forever 2024-09-16T15:50:43.696 INFO:teuthology.orchestra.run.smithi184.stdout: inet6 fe80::5c1f:84ff:fe6d:79ae/64 scope link 2024-09-16T15:50:43.696 INFO:teuthology.orchestra.run.smithi184.stdout: valid_lft forever preferred_lft forever 2024-09-16T15:50:43.696 INFO:teuthology.orchestra.run.smithi184.stdout:8: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-09-16T15:50:43.696 INFO:teuthology.orchestra.run.smithi184.stdout: link/ether 0e:5c:f6:1f:82:2f brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.1 2024-09-16T15:50:43.696 INFO:teuthology.orchestra.run.smithi184.stdout: inet6 fe80::c5c:f6ff:fe1f:822f/64 scope link 2024-09-16T15:50:43.696 INFO:teuthology.orchestra.run.smithi184.stdout: valid_lft forever preferred_lft forever 2024-09-16T15:50:43.697 INFO:teuthology.orchestra.run.smithi184.stdout:9: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-09-16T15:50:43.697 INFO:teuthology.orchestra.run.smithi184.stdout: link/ether c6:a6:3d:69:d0:db brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.3 2024-09-16T15:50:43.697 INFO:teuthology.orchestra.run.smithi184.stdout: inet6 fe80::c4a6:3dff:fe69:d0db/64 scope link 2024-09-16T15:50:43.697 INFO:teuthology.orchestra.run.smithi184.stdout: valid_lft forever preferred_lft forever 2024-09-16T15:50:43.697 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:50:43.697 DEBUG:teuthology.orchestra.run.smithi184:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-09-16T15:50:43.748 INFO:teuthology.orchestra.run.smithi184.stdout:1 2024-09-16T15:50:43.750 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:50:43.750 DEBUG:teuthology.orchestra.run.smithi184:> route 2024-09-16T15:50:43.801 INFO:teuthology.orchestra.run.smithi184.stdout:Kernel IP routing table 2024-09-16T15:50:43.801 INFO:teuthology.orchestra.run.smithi184.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-09-16T15:50:43.801 INFO:teuthology.orchestra.run.smithi184.stdout:default _gateway 0.0.0.0 UG 0 0 0 enp3s0f1 2024-09-16T15:50:43.801 INFO:teuthology.orchestra.run.smithi184.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-09-16T15:50:43.802 INFO:teuthology.orchestra.run.smithi184.stdout:158.69.67.47 _gateway 255.255.255.255 UGH 100 0 0 enp3s0f1 2024-09-16T15:50:43.802 INFO:teuthology.orchestra.run.smithi184.stdout:172.17.0.0 0.0.0.0 255.255.0.0 U 0 0 0 docker0 2024-09-16T15:50:43.802 INFO:teuthology.orchestra.run.smithi184.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-09-16T15:50:43.802 INFO:teuthology.orchestra.run.smithi184.stdout:ns1.front.sepia 0.0.0.0 255.255.255.255 UH 100 0 0 enp3s0f1 2024-09-16T15:50:43.802 INFO:teuthology.orchestra.run.smithi184.stdout:172.21.0.2 0.0.0.0 255.255.255.255 UH 100 0 0 enp3s0f1 2024-09-16T15:50:43.802 INFO:teuthology.orchestra.run.smithi184.stdout:_gateway 0.0.0.0 255.255.255.255 UH 100 0 0 enp3s0f1 2024-09-16T15:50:43.802 INFO:teuthology.orchestra.run.smithi184.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-09-16T15:50:43.802 DEBUG:teuthology.orchestra.run.smithi184:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-09-16T15:50:43.802 DEBUG:teuthology.orchestra.run.smithi184:> set -e 2024-09-16T15:50:43.802 DEBUG:teuthology.orchestra.run.smithi184:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-09-16T15:50:43.802 DEBUG:teuthology.orchestra.run.smithi184:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-09-16T15:50:43.803 DEBUG:teuthology.orchestra.run.smithi184:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-09-16T15:50:43.803 DEBUG:teuthology.orchestra.run.smithi184:> ') 2024-09-16T15:50:43.889 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:50:43 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-09-16T15:50:43.917 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:50:43 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-16T15:50:43.925 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:50:43.925 DEBUG:teuthology.orchestra.run.smithi184:> ip netns list 2024-09-16T15:50:43.930 INFO:teuthology.orchestra.run.smithi184.stdout:ceph-ns--home-ubuntu-cephtest-mnt.3 (id: 1) 2024-09-16T15:50:43.930 INFO:teuthology.orchestra.run.smithi184.stdout:ceph-ns--home-ubuntu-cephtest-mnt.1 (id: 0) 2024-09-16T15:50:43.930 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.1' netns for '/home/ubuntu/cephtest/mnt.1' 2024-09-16T15:50:43.930 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:50:43.930 DEBUG:teuthology.orchestra.run.smithi184:> sudo ip link set brx.0 up 2024-09-16T15:50:43.983 INFO:tasks.cephfs.fuse_mount:Client client.1 config is {} 2024-09-16T15:50:43.983 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-09-16T15:50:43.983 DEBUG:teuthology.orchestra.run.smithi184:> mkdir -p -v /home/ubuntu/cephtest/mnt.1 2024-09-16T15:50:44.028 INFO:teuthology.orchestra.run.smithi184.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.1' 2024-09-16T15:50:44.029 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-09-16T15:50:44.029 DEBUG:teuthology.orchestra.run.smithi184:> chmod 0000 /home/ubuntu/cephtest/mnt.1 2024-09-16T15:50:44.076 DEBUG:teuthology.orchestra.run.smithi184:> sudo modprobe fuse 2024-09-16T15:50:44.133 DEBUG:teuthology.orchestra.run.smithi184:> cat /proc/self/mounts | awk '{print $2}' 2024-09-16T15:50:44.182 INFO:teuthology.orchestra.run.smithi184.stdout:/sys 2024-09-16T15:50:44.182 INFO:teuthology.orchestra.run.smithi184.stdout:/proc 2024-09-16T15:50:44.182 INFO:teuthology.orchestra.run.smithi184.stdout:/dev 2024-09-16T15:50:44.182 INFO:teuthology.orchestra.run.smithi184.stdout:/dev/pts 2024-09-16T15:50:44.182 INFO:teuthology.orchestra.run.smithi184.stdout:/run 2024-09-16T15:50:44.182 INFO:teuthology.orchestra.run.smithi184.stdout:/ 2024-09-16T15:50:44.182 INFO:teuthology.orchestra.run.smithi184.stdout:/sys/kernel/security 2024-09-16T15:50:44.182 INFO:teuthology.orchestra.run.smithi184.stdout:/dev/shm 2024-09-16T15:50:44.182 INFO:teuthology.orchestra.run.smithi184.stdout:/run/lock 2024-09-16T15:50:44.183 INFO:teuthology.orchestra.run.smithi184.stdout:/sys/fs/cgroup 2024-09-16T15:50:44.183 INFO:teuthology.orchestra.run.smithi184.stdout:/sys/fs/pstore 2024-09-16T15:50:44.183 INFO:teuthology.orchestra.run.smithi184.stdout:/sys/fs/bpf 2024-09-16T15:50:44.183 INFO:teuthology.orchestra.run.smithi184.stdout:/proc/sys/fs/binfmt_misc 2024-09-16T15:50:44.183 INFO:teuthology.orchestra.run.smithi184.stdout:/dev/hugepages 2024-09-16T15:50:44.183 INFO:teuthology.orchestra.run.smithi184.stdout:/dev/mqueue 2024-09-16T15:50:44.183 INFO:teuthology.orchestra.run.smithi184.stdout:/sys/kernel/debug 2024-09-16T15:50:44.183 INFO:teuthology.orchestra.run.smithi184.stdout:/sys/kernel/tracing 2024-09-16T15:50:44.183 INFO:teuthology.orchestra.run.smithi184.stdout:/proc/fs/nfsd 2024-09-16T15:50:44.183 INFO:teuthology.orchestra.run.smithi184.stdout:/sys/fs/fuse/connections 2024-09-16T15:50:44.183 INFO:teuthology.orchestra.run.smithi184.stdout:/sys/kernel/config 2024-09-16T15:50:44.183 INFO:teuthology.orchestra.run.smithi184.stdout:/run/credentials/systemd-sysusers.service 2024-09-16T15:50:44.183 INFO:teuthology.orchestra.run.smithi184.stdout:/run/rpc_pipefs 2024-09-16T15:50:44.183 INFO:teuthology.orchestra.run.smithi184.stdout:/run/user/1000 2024-09-16T15:50:44.183 INFO:teuthology.orchestra.run.smithi184.stdout:/var/lib/ceph 2024-09-16T15:50:44.184 INFO:teuthology.orchestra.run.smithi184.stdout:/var/lib/ceph/osd/ceph-4 2024-09-16T15:50:44.184 INFO:teuthology.orchestra.run.smithi184.stdout:/var/lib/ceph/osd/ceph-5 2024-09-16T15:50:44.184 INFO:teuthology.orchestra.run.smithi184.stdout:/var/lib/ceph/osd/ceph-6 2024-09-16T15:50:44.184 INFO:teuthology.orchestra.run.smithi184.stdout:/var/lib/ceph/osd/ceph-7 2024-09-16T15:50:44.184 INFO:teuthology.orchestra.run.smithi184.stdout:/run/netns 2024-09-16T15:50:44.184 INFO:teuthology.orchestra.run.smithi184.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-09-16T15:50:44.184 INFO:teuthology.orchestra.run.smithi184.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-09-16T15:50:44.184 INFO:teuthology.orchestra.run.smithi184.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-09-16T15:50:44.184 INFO:teuthology.orchestra.run.smithi184.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-09-16T15:50:44.185 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:50:44.185 DEBUG:teuthology.orchestra.run.smithi184:> ls /sys/fs/fuse/connections 2024-09-16T15:50:44.233 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-09-16T15:50:44.233 DEBUG:teuthology.orchestra.run.smithi184:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.1 --id 1) 2024-09-16T15:50:44.278 DEBUG:teuthology.orchestra.run.smithi184:> sudo modprobe fuse 2024-09-16T15:50:44.291 DEBUG:teuthology.orchestra.run.smithi184:> cat /proc/self/mounts | awk '{print $2}' 2024-09-16T15:50:44.340 INFO:teuthology.orchestra.run.smithi184.stdout:/sys 2024-09-16T15:50:44.341 INFO:teuthology.orchestra.run.smithi184.stdout:/proc 2024-09-16T15:50:44.341 INFO:teuthology.orchestra.run.smithi184.stdout:/dev 2024-09-16T15:50:44.341 INFO:teuthology.orchestra.run.smithi184.stdout:/dev/pts 2024-09-16T15:50:44.341 INFO:teuthology.orchestra.run.smithi184.stdout:/run 2024-09-16T15:50:44.341 INFO:teuthology.orchestra.run.smithi184.stdout:/ 2024-09-16T15:50:44.341 INFO:teuthology.orchestra.run.smithi184.stdout:/sys/kernel/security 2024-09-16T15:50:44.341 INFO:teuthology.orchestra.run.smithi184.stdout:/dev/shm 2024-09-16T15:50:44.341 INFO:teuthology.orchestra.run.smithi184.stdout:/run/lock 2024-09-16T15:50:44.341 INFO:teuthology.orchestra.run.smithi184.stdout:/sys/fs/cgroup 2024-09-16T15:50:44.341 INFO:teuthology.orchestra.run.smithi184.stdout:/sys/fs/pstore 2024-09-16T15:50:44.341 INFO:teuthology.orchestra.run.smithi184.stdout:/sys/fs/bpf 2024-09-16T15:50:44.341 INFO:teuthology.orchestra.run.smithi184.stdout:/proc/sys/fs/binfmt_misc 2024-09-16T15:50:44.341 INFO:teuthology.orchestra.run.smithi184.stdout:/dev/hugepages 2024-09-16T15:50:44.342 INFO:teuthology.orchestra.run.smithi184.stdout:/dev/mqueue 2024-09-16T15:50:44.342 INFO:teuthology.orchestra.run.smithi184.stdout:/sys/kernel/debug 2024-09-16T15:50:44.342 INFO:teuthology.orchestra.run.smithi184.stdout:/sys/kernel/tracing 2024-09-16T15:50:44.342 INFO:teuthology.orchestra.run.smithi184.stdout:/proc/fs/nfsd 2024-09-16T15:50:44.342 INFO:teuthology.orchestra.run.smithi184.stdout:/sys/fs/fuse/connections 2024-09-16T15:50:44.342 INFO:teuthology.orchestra.run.smithi184.stdout:/sys/kernel/config 2024-09-16T15:50:44.342 INFO:teuthology.orchestra.run.smithi184.stdout:/run/credentials/systemd-sysusers.service 2024-09-16T15:50:44.342 INFO:teuthology.orchestra.run.smithi184.stdout:/run/rpc_pipefs 2024-09-16T15:50:44.342 INFO:teuthology.orchestra.run.smithi184.stdout:/run/user/1000 2024-09-16T15:50:44.342 INFO:teuthology.orchestra.run.smithi184.stdout:/var/lib/ceph 2024-09-16T15:50:44.342 INFO:teuthology.orchestra.run.smithi184.stdout:/var/lib/ceph/osd/ceph-4 2024-09-16T15:50:44.342 INFO:teuthology.orchestra.run.smithi184.stdout:/var/lib/ceph/osd/ceph-5 2024-09-16T15:50:44.342 INFO:teuthology.orchestra.run.smithi184.stdout:/var/lib/ceph/osd/ceph-6 2024-09-16T15:50:44.342 INFO:teuthology.orchestra.run.smithi184.stdout:/var/lib/ceph/osd/ceph-7 2024-09-16T15:50:44.343 INFO:teuthology.orchestra.run.smithi184.stdout:/run/netns 2024-09-16T15:50:44.343 INFO:teuthology.orchestra.run.smithi184.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-09-16T15:50:44.343 INFO:teuthology.orchestra.run.smithi184.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-09-16T15:50:44.343 INFO:teuthology.orchestra.run.smithi184.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-09-16T15:50:44.343 INFO:teuthology.orchestra.run.smithi184.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-09-16T15:50:44.344 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:50:44.344 DEBUG:teuthology.orchestra.run.smithi184:> ls /sys/fs/fuse/connections 2024-09-16T15:50:44.358 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi184.stderr:2024-09-16T15:50:44.355+0000 7f685915d580 -1 init, newargv = 0x563476626ec0 newargc=15 2024-09-16T15:50:44.358 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi184.stderr:ceph-fuse[36213]: starting ceph client 2024-09-16T15:50:44.363 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi184.stderr:ceph-fuse[36213]: starting fuse 2024-09-16T15:50:45.463 DEBUG:teuthology.orchestra.run.smithi184:> sudo modprobe fuse 2024-09-16T15:50:45.479 DEBUG:teuthology.orchestra.run.smithi184:> cat /proc/self/mounts | awk '{print $2}' 2024-09-16T15:50:45.531 INFO:teuthology.orchestra.run.smithi184.stdout:/sys 2024-09-16T15:50:45.531 INFO:teuthology.orchestra.run.smithi184.stdout:/proc 2024-09-16T15:50:45.531 INFO:teuthology.orchestra.run.smithi184.stdout:/dev 2024-09-16T15:50:45.531 INFO:teuthology.orchestra.run.smithi184.stdout:/dev/pts 2024-09-16T15:50:45.532 INFO:teuthology.orchestra.run.smithi184.stdout:/run 2024-09-16T15:50:45.532 INFO:teuthology.orchestra.run.smithi184.stdout:/ 2024-09-16T15:50:45.532 INFO:teuthology.orchestra.run.smithi184.stdout:/sys/kernel/security 2024-09-16T15:50:45.532 INFO:teuthology.orchestra.run.smithi184.stdout:/dev/shm 2024-09-16T15:50:45.532 INFO:teuthology.orchestra.run.smithi184.stdout:/run/lock 2024-09-16T15:50:45.532 INFO:teuthology.orchestra.run.smithi184.stdout:/sys/fs/cgroup 2024-09-16T15:50:45.532 INFO:teuthology.orchestra.run.smithi184.stdout:/sys/fs/pstore 2024-09-16T15:50:45.532 INFO:teuthology.orchestra.run.smithi184.stdout:/sys/fs/bpf 2024-09-16T15:50:45.532 INFO:teuthology.orchestra.run.smithi184.stdout:/proc/sys/fs/binfmt_misc 2024-09-16T15:50:45.532 INFO:teuthology.orchestra.run.smithi184.stdout:/dev/hugepages 2024-09-16T15:50:45.533 INFO:teuthology.orchestra.run.smithi184.stdout:/dev/mqueue 2024-09-16T15:50:45.533 INFO:teuthology.orchestra.run.smithi184.stdout:/sys/kernel/debug 2024-09-16T15:50:45.533 INFO:teuthology.orchestra.run.smithi184.stdout:/sys/kernel/tracing 2024-09-16T15:50:45.533 INFO:teuthology.orchestra.run.smithi184.stdout:/proc/fs/nfsd 2024-09-16T15:50:45.533 INFO:teuthology.orchestra.run.smithi184.stdout:/sys/fs/fuse/connections 2024-09-16T15:50:45.533 INFO:teuthology.orchestra.run.smithi184.stdout:/sys/kernel/config 2024-09-16T15:50:45.533 INFO:teuthology.orchestra.run.smithi184.stdout:/run/credentials/systemd-sysusers.service 2024-09-16T15:50:45.533 INFO:teuthology.orchestra.run.smithi184.stdout:/run/rpc_pipefs 2024-09-16T15:50:45.533 INFO:teuthology.orchestra.run.smithi184.stdout:/run/user/1000 2024-09-16T15:50:45.533 INFO:teuthology.orchestra.run.smithi184.stdout:/var/lib/ceph 2024-09-16T15:50:45.533 INFO:teuthology.orchestra.run.smithi184.stdout:/var/lib/ceph/osd/ceph-4 2024-09-16T15:50:45.533 INFO:teuthology.orchestra.run.smithi184.stdout:/var/lib/ceph/osd/ceph-5 2024-09-16T15:50:45.533 INFO:teuthology.orchestra.run.smithi184.stdout:/var/lib/ceph/osd/ceph-6 2024-09-16T15:50:45.534 INFO:teuthology.orchestra.run.smithi184.stdout:/var/lib/ceph/osd/ceph-7 2024-09-16T15:50:45.534 INFO:teuthology.orchestra.run.smithi184.stdout:/run/netns 2024-09-16T15:50:45.534 INFO:teuthology.orchestra.run.smithi184.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-09-16T15:50:45.534 INFO:teuthology.orchestra.run.smithi184.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-09-16T15:50:45.534 INFO:teuthology.orchestra.run.smithi184.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-09-16T15:50:45.534 INFO:teuthology.orchestra.run.smithi184.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-09-16T15:50:45.534 INFO:teuthology.orchestra.run.smithi184.stdout:/home/ubuntu/cephtest/mnt.1 2024-09-16T15:50:45.535 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:50:45.535 DEBUG:teuthology.orchestra.run.smithi184:> ls /sys/fs/fuse/connections 2024-09-16T15:50:45.581 INFO:teuthology.orchestra.run.smithi184.stdout:45 2024-09-16T15:50:45.581 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [45] 2024-09-16T15:50:45.581 DEBUG:teuthology.orchestra.run.smithi184:> sudo stdin-killer -- python3 -c ' 2024-09-16T15:50:45.581 DEBUG:teuthology.orchestra.run.smithi184:> import glob 2024-09-16T15:50:45.582 DEBUG:teuthology.orchestra.run.smithi184:> import re 2024-09-16T15:50:45.582 DEBUG:teuthology.orchestra.run.smithi184:> import os 2024-09-16T15:50:45.582 DEBUG:teuthology.orchestra.run.smithi184:> import subprocess 2024-09-16T15:50:45.582 DEBUG:teuthology.orchestra.run.smithi184:> 2024-09-16T15:50:45.582 DEBUG:teuthology.orchestra.run.smithi184:> def _find_admin_socket(client_name): 2024-09-16T15:50:45.582 DEBUG:teuthology.orchestra.run.smithi184:> asok_path = "/var/run/ceph/ceph-client.1.*.asok" 2024-09-16T15:50:45.582 DEBUG:teuthology.orchestra.run.smithi184:> files = glob.glob(asok_path) 2024-09-16T15:50:45.582 DEBUG:teuthology.orchestra.run.smithi184:> mountpoint = "/home/ubuntu/cephtest/mnt.1" 2024-09-16T15:50:45.582 DEBUG:teuthology.orchestra.run.smithi184:> 2024-09-16T15:50:45.582 DEBUG:teuthology.orchestra.run.smithi184:> # Given a non-glob path, it better be there 2024-09-16T15:50:45.582 DEBUG:teuthology.orchestra.run.smithi184:> if "*" not in asok_path: 2024-09-16T15:50:45.582 DEBUG:teuthology.orchestra.run.smithi184:> assert(len(files) == 1) 2024-09-16T15:50:45.582 DEBUG:teuthology.orchestra.run.smithi184:> return files[0] 2024-09-16T15:50:45.582 DEBUG:teuthology.orchestra.run.smithi184:> 2024-09-16T15:50:45.583 DEBUG:teuthology.orchestra.run.smithi184:> for f in files: 2024-09-16T15:50:45.583 DEBUG:teuthology.orchestra.run.smithi184:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-09-16T15:50:45.583 DEBUG:teuthology.orchestra.run.smithi184:> if os.path.exists("/proc/{0}".format(pid)): 2024-09-16T15:50:45.583 DEBUG:teuthology.orchestra.run.smithi184:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-09-16T15:50:45.583 DEBUG:teuthology.orchestra.run.smithi184:> contents = proc_f.read() 2024-09-16T15:50:45.583 DEBUG:teuthology.orchestra.run.smithi184:> if mountpoint in contents: 2024-09-16T15:50:45.583 DEBUG:teuthology.orchestra.run.smithi184:> return f 2024-09-16T15:50:45.583 DEBUG:teuthology.orchestra.run.smithi184:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-09-16T15:50:45.583 DEBUG:teuthology.orchestra.run.smithi184:> 2024-09-16T15:50:45.583 DEBUG:teuthology.orchestra.run.smithi184:> print(_find_admin_socket("client.1")) 2024-09-16T15:50:45.583 DEBUG:teuthology.orchestra.run.smithi184:> ' 2024-09-16T15:50:45.705 INFO:teuthology.orchestra.run.smithi184.stdout:/var/run/ceph/ceph-client.1.36213.asok 2024-09-16T15:50:45.709 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:50:45 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-16T15:50:45.718 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.1.36213.asok 2024-09-16T15:50:45.718 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:50:45.718 DEBUG:teuthology.orchestra.run.smithi184:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.1.36213.asok status 2024-09-16T15:50:45.869 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-16T15:50:45.869 INFO:teuthology.orchestra.run.smithi184.stdout: "metadata": { 2024-09-16T15:50:45.869 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph_sha1": "33f95d2ae1bcbe7eb04404303f179eeeaa6982df", 2024-09-16T15:50:45.869 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph_version": "ceph version 19.3.0-4938-g33f95d2a (33f95d2ae1bcbe7eb04404303f179eeeaa6982df) squid (dev)", 2024-09-16T15:50:45.869 INFO:teuthology.orchestra.run.smithi184.stdout: "entity_id": "1", 2024-09-16T15:50:45.869 INFO:teuthology.orchestra.run.smithi184.stdout: "hostname": "smithi184", 2024-09-16T15:50:45.869 INFO:teuthology.orchestra.run.smithi184.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.1", 2024-09-16T15:50:45.870 INFO:teuthology.orchestra.run.smithi184.stdout: "pid": "36213", 2024-09-16T15:50:45.870 INFO:teuthology.orchestra.run.smithi184.stdout: "root": "/" 2024-09-16T15:50:45.870 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-16T15:50:45.870 INFO:teuthology.orchestra.run.smithi184.stdout: "dentry_count": 0, 2024-09-16T15:50:45.870 INFO:teuthology.orchestra.run.smithi184.stdout: "dentry_pinned_count": 0, 2024-09-16T15:50:45.870 INFO:teuthology.orchestra.run.smithi184.stdout: "id": 6110, 2024-09-16T15:50:45.870 INFO:teuthology.orchestra.run.smithi184.stdout: "inst": { 2024-09-16T15:50:45.870 INFO:teuthology.orchestra.run.smithi184.stdout: "name": { 2024-09-16T15:50:45.870 INFO:teuthology.orchestra.run.smithi184.stdout: "type": "client", 2024-09-16T15:50:45.870 INFO:teuthology.orchestra.run.smithi184.stdout: "num": 6110 2024-09-16T15:50:45.870 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-16T15:50:45.870 INFO:teuthology.orchestra.run.smithi184.stdout: "addr": { 2024-09-16T15:50:45.870 INFO:teuthology.orchestra.run.smithi184.stdout: "type": "v1", 2024-09-16T15:50:45.871 INFO:teuthology.orchestra.run.smithi184.stdout: "addr": "192.168.0.1:0", 2024-09-16T15:50:45.871 INFO:teuthology.orchestra.run.smithi184.stdout: "nonce": 3680618210 2024-09-16T15:50:45.871 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-16T15:50:45.871 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-16T15:50:45.871 INFO:teuthology.orchestra.run.smithi184.stdout: "addr": { 2024-09-16T15:50:45.871 INFO:teuthology.orchestra.run.smithi184.stdout: "type": "v1", 2024-09-16T15:50:45.871 INFO:teuthology.orchestra.run.smithi184.stdout: "addr": "192.168.0.1:0", 2024-09-16T15:50:45.871 INFO:teuthology.orchestra.run.smithi184.stdout: "nonce": 3680618210 2024-09-16T15:50:45.871 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-16T15:50:45.871 INFO:teuthology.orchestra.run.smithi184.stdout: "inst_str": "client.6110 192.168.0.1:0/3680618210", 2024-09-16T15:50:45.871 INFO:teuthology.orchestra.run.smithi184.stdout: "addr_str": "192.168.0.1:0/3680618210", 2024-09-16T15:50:45.871 INFO:teuthology.orchestra.run.smithi184.stdout: "inode_count": 1, 2024-09-16T15:50:45.871 INFO:teuthology.orchestra.run.smithi184.stdout: "mds_epoch": 95, 2024-09-16T15:50:45.871 INFO:teuthology.orchestra.run.smithi184.stdout: "osd_epoch": 146, 2024-09-16T15:50:45.872 INFO:teuthology.orchestra.run.smithi184.stdout: "osd_epoch_barrier": 0, 2024-09-16T15:50:45.872 INFO:teuthology.orchestra.run.smithi184.stdout: "blocklisted": false, 2024-09-16T15:50:45.872 INFO:teuthology.orchestra.run.smithi184.stdout: "fs_name": "cephfs" 2024-09-16T15:50:45.872 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-16T15:50:45.882 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:50:45.883 DEBUG:teuthology.orchestra.run.smithi184:> stat --file-system '--printf=%T 2024-09-16T15:50:45.883 DEBUG:teuthology.orchestra.run.smithi184:> ' -- /home/ubuntu/cephtest/mnt.1 2024-09-16T15:50:45.889 INFO:teuthology.orchestra.run.smithi184.stdout:fuseblk 2024-09-16T15:50:45.889 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2024-09-16T15:50:45.889 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:50:45.890 DEBUG:teuthology.orchestra.run.smithi184:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.1 2024-09-16T15:50:45.944 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs flag set enable_multiple true --yes-i-really-mean-it 2024-09-16T15:50:46.086 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:46.086+0000 7fe473f0b640 1 -- 172.21.15.123:0/1375065661 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe46c0b0c30 msgr2=0x7fe46c0d1980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:50:46.086 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:46.086+0000 7fe473f0b640 1 --2- 172.21.15.123:0/1375065661 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe46c0b0c30 0x7fe46c0d1980 secure :-1 s=READY pgs=1622 cs=0 l=1 rev1=1 crypto rx=0x7fe46c0cf6f0 tx=0x7fe46801d1a0 comp rx=0 tx=0).stop 2024-09-16T15:50:46.086 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:46.086+0000 7fe473f0b640 1 -- 172.21.15.123:0/1375065661 shutdown_connections 2024-09-16T15:50:46.086 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:46.086+0000 7fe473f0b640 1 --2- 172.21.15.123:0/1375065661 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe46c0b0c30 0x7fe46c0d1980 unknown :-1 s=CLOSED pgs=1622 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:46.086 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:46.086+0000 7fe473f0b640 1 --2- 172.21.15.123:0/1375065661 >> v2:172.21.15.184:3300/0 conn(0x7fe46c15bcf0 0x7fe46c0b2e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:46.087 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:46.086+0000 7fe473f0b640 1 -- 172.21.15.123:0/1375065661 >> 172.21.15.123:0/1375065661 conn(0x7fe46c09aa40 msgr2=0x7fe46c09ae50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:50:46.087 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:46.086+0000 7fe473f0b640 1 -- 172.21.15.123:0/1375065661 shutdown_connections 2024-09-16T15:50:46.087 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:46.086+0000 7fe473f0b640 1 -- 172.21.15.123:0/1375065661 wait complete. 2024-09-16T15:50:46.087 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:46.086+0000 7fe473f0b640 1 Processor -- start 2024-09-16T15:50:46.087 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:46.086+0000 7fe473f0b640 1 -- start start 2024-09-16T15:50:46.089 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:46.090+0000 7fe473f0b640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe46c0b0c30 0x7fe46c15ad40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:50:46.089 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:46.090+0000 7fe473f0b640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe46c0d1ec0 0x7fe46c143910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:50:46.089 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:46.090+0000 7fe473f0b640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fe46c15bcf0 0x7fe46c0c7aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:50:46.089 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:46.090+0000 7fe473f0b640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fe46c168f20 con 0x7fe46c0d1ec0 2024-09-16T15:50:46.089 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:46.090+0000 7fe473f0b640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fe46c168da0 con 0x7fe46c0b0c30 2024-09-16T15:50:46.089 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:46.090+0000 7fe473f0b640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fe46c1690a0 con 0x7fe46c15bcf0 2024-09-16T15:50:46.089 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:46.090+0000 7fe4714a6640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe46c0d1ec0 0x7fe46c143910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:50:46.089 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:46.090+0000 7fe4714a6640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe46c0d1ec0 0x7fe46c143910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:40306/0 (socket says 172.21.15.123:40306) 2024-09-16T15:50:46.089 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:46.090+0000 7fe4714a6640 1 -- 172.21.15.123:0/589924446 learned_addr learned my addr 172.21.15.123:0/589924446 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:50:46.090 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:46.090+0000 7fe471ca7640 1 --2- 172.21.15.123:0/589924446 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe46c0b0c30 0x7fe46c15ad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:50:46.090 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:46.090+0000 7fe4714a6640 1 -- 172.21.15.123:0/589924446 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fe46c15bcf0 msgr2=0x7fe46c0c7aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:50:46.090 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:46.090+0000 7fe4714a6640 1 --2- 172.21.15.123:0/589924446 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fe46c15bcf0 0x7fe46c0c7aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:46.090 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:46.090+0000 7fe4714a6640 1 -- 172.21.15.123:0/589924446 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe46c0b0c30 msgr2=0x7fe46c15ad40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:50:46.090 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:46.090+0000 7fe4714a6640 1 --2- 172.21.15.123:0/589924446 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe46c0b0c30 0x7fe46c15ad40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:46.090 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:46.090+0000 7fe4714a6640 1 -- 172.21.15.123:0/589924446 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe46c1f5c50 con 0x7fe46c0d1ec0 2024-09-16T15:50:46.090 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:46.090+0000 7fe4714a6640 1 --2- 172.21.15.123:0/589924446 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe46c0d1ec0 0x7fe46c143910 secure :-1 s=READY pgs=1623 cs=0 l=1 rev1=1 crypto rx=0x7fe468000c10 tx=0x7fe4680a56a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:50:46.090 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:46.090+0000 7fe4627fc640 1 -- 172.21.15.123:0/589924446 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4680bb070 con 0x7fe46c0d1ec0 2024-09-16T15:50:46.090 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:46.090+0000 7fe4627fc640 1 -- 172.21.15.123:0/589924446 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fe468004470 con 0x7fe46c0d1ec0 2024-09-16T15:50:46.090 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:46.090+0000 7fe4627fc640 1 -- 172.21.15.123:0/589924446 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4680b2040 con 0x7fe46c0d1ec0 2024-09-16T15:50:46.090 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:46.090+0000 7fe473f0b640 1 -- 172.21.15.123:0/589924446 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe46c1f5ec0 con 0x7fe46c0d1ec0 2024-09-16T15:50:46.090 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:46.090+0000 7fe473f0b640 1 -- 172.21.15.123:0/589924446 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe46c1f6480 con 0x7fe46c0d1ec0 2024-09-16T15:50:46.091 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:46.090+0000 7fe473f0b640 1 -- 172.21.15.123:0/589924446 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe46c0d1980 con 0x7fe46c0d1ec0 2024-09-16T15:50:46.091 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:46.090+0000 7fe4627fc640 1 -- 172.21.15.123:0/589924446 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 22) ==== 97692+0+0 (secure 0 0 0) 0x7fe468005230 con 0x7fe46c0d1ec0 2024-09-16T15:50:46.091 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:46.090+0000 7fe4627fc640 1 --2- 172.21.15.123:0/589924446 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fe450073580 0x7fe450075a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:50:46.091 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:46.090+0000 7fe4627fc640 1 -- 172.21.15.123:0/589924446 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(146..146 src has 1..146) ==== 5737+0+0 (secure 0 0 0) 0x7fe46812e420 con 0x7fe46c0d1ec0 2024-09-16T15:50:46.092 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:46.090+0000 7fe471ca7640 1 --2- 172.21.15.123:0/589924446 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fe450073580 0x7fe450075a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:50:46.092 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:46.094+0000 7fe471ca7640 1 --2- 172.21.15.123:0/589924446 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fe450073580 0x7fe450075a40 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7fe45c0058d0 tx=0x7fe45c005860 comp rx=0 tx=0).ready entity=mgr.4150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:50:46.095 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:46.094+0000 7fe4627fc640 1 -- 172.21.15.123:0/589924446 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7fe4680163f0 con 0x7fe46c0d1ec0 2024-09-16T15:50:46.307 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:46.306+0000 7fe473f0b640 1 -- 172.21.15.123:0/589924446 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "fs flag set", "flag_name": "enable_multiple", "val": "true", "yes_i_really_mean_it": true} v 0) -- 0x7fe46c0d2ae0 con 0x7fe46c0d1ec0 2024-09-16T15:50:46.490 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:46.490+0000 7fe4627fc640 1 -- 172.21.15.123:0/589924446 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "fs flag set", "flag_name": "enable_multiple", "val": "true", "yes_i_really_mean_it": true}]=0 v96) ==== 136+0+0 (secure 0 0 0) 0x7fe4680f8cc0 con 0x7fe46c0d1ec0 2024-09-16T15:50:46.495 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:46.494+0000 7fe473f0b640 1 -- 172.21.15.123:0/589924446 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fe450073580 msgr2=0x7fe450075a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:50:46.495 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:46.494+0000 7fe473f0b640 1 --2- 172.21.15.123:0/589924446 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fe450073580 0x7fe450075a40 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7fe45c0058d0 tx=0x7fe45c005860 comp rx=0 tx=0).stop 2024-09-16T15:50:46.495 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:46.494+0000 7fe473f0b640 1 -- 172.21.15.123:0/589924446 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe46c0d1ec0 msgr2=0x7fe46c143910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:50:46.496 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:46.494+0000 7fe473f0b640 1 --2- 172.21.15.123:0/589924446 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe46c0d1ec0 0x7fe46c143910 secure :-1 s=READY pgs=1623 cs=0 l=1 rev1=1 crypto rx=0x7fe468000c10 tx=0x7fe4680a56a0 comp rx=0 tx=0).stop 2024-09-16T15:50:46.496 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:46.494+0000 7fe473f0b640 1 -- 172.21.15.123:0/589924446 shutdown_connections 2024-09-16T15:50:46.496 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:46.494+0000 7fe473f0b640 1 --2- 172.21.15.123:0/589924446 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fe450073580 0x7fe450075a40 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:46.496 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:46.494+0000 7fe473f0b640 1 --2- 172.21.15.123:0/589924446 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fe46c15bcf0 0x7fe46c0c7aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:46.496 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:46.494+0000 7fe473f0b640 1 --2- 172.21.15.123:0/589924446 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe46c0d1ec0 0x7fe46c143910 unknown :-1 s=CLOSED pgs=1623 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:46.496 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:46.494+0000 7fe473f0b640 1 --2- 172.21.15.123:0/589924446 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe46c0b0c30 0x7fe46c15ad40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:46.496 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:46.494+0000 7fe473f0b640 1 -- 172.21.15.123:0/589924446 >> 172.21.15.123:0/589924446 conn(0x7fe46c09aa40 msgr2=0x7fe46c0af2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:50:46.496 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:46.494+0000 7fe473f0b640 1 -- 172.21.15.123:0/589924446 shutdown_connections 2024-09-16T15:50:46.496 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:46.494+0000 7fe473f0b640 1 -- 172.21.15.123:0/589924446 wait complete. 2024-09-16T15:50:46.514 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2024-09-16T15:50:46.634 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:46.634+0000 7fbc3f33d640 1 -- 172.21.15.123:0/3474656875 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbc3815bca0 msgr2=0x7fbc380d0630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:50:46.634 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:46.634+0000 7fbc3f33d640 1 --2- 172.21.15.123:0/3474656875 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbc3815bca0 0x7fbc380d0630 secure :-1 s=READY pgs=1624 cs=0 l=1 rev1=1 crypto rx=0x7fbc3806df40 tx=0x7fbc3401c800 comp rx=0 tx=0).stop 2024-09-16T15:50:46.634 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:46.634+0000 7fbc3f33d640 1 -- 172.21.15.123:0/3474656875 shutdown_connections 2024-09-16T15:50:46.634 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:46.634+0000 7fbc3f33d640 1 --2- 172.21.15.123:0/3474656875 >> v2:172.21.15.184:3300/0 conn(0x7fbc380726c0 0x7fbc38073ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:46.634 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:46.634+0000 7fbc3f33d640 1 --2- 172.21.15.123:0/3474656875 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbc3815bca0 0x7fbc380d0630 unknown :-1 s=CLOSED pgs=1624 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:46.634 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:46.634+0000 7fbc3f33d640 1 -- 172.21.15.123:0/3474656875 >> 172.21.15.123:0/3474656875 conn(0x7fbc380cfb10 msgr2=0x7fbc380cff20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:50:46.634 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:46.634+0000 7fbc3f33d640 1 -- 172.21.15.123:0/3474656875 shutdown_connections 2024-09-16T15:50:46.634 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:46.634+0000 7fbc3f33d640 1 -- 172.21.15.123:0/3474656875 wait complete. 2024-09-16T15:50:46.635 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:46.634+0000 7fbc3f33d640 1 Processor -- start 2024-09-16T15:50:46.635 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:46.634+0000 7fbc3f33d640 1 -- start start 2024-09-16T15:50:46.637 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:46.638+0000 7fbc3f33d640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fbc380726c0 0x7fbc380ce300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:50:46.637 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:46.638+0000 7fbc3f33d640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbc38074430 0x7fbc3814bef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:50:46.638 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:46.638+0000 7fbc3f33d640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fbc3815bca0 0x7fbc380c5c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:50:46.638 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:46.638+0000 7fbc3f33d640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fbc38168f40 con 0x7fbc38074430 2024-09-16T15:50:46.638 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:46.638+0000 7fbc3f33d640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fbc38168dc0 con 0x7fbc380726c0 2024-09-16T15:50:46.638 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:46.638+0000 7fbc3f33d640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fbc381690c0 con 0x7fbc3815bca0 2024-09-16T15:50:46.638 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:46.638+0000 7fbc3c8d8640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbc38074430 0x7fbc3814bef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:50:46.638 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:46.638+0000 7fbc3c8d8640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbc38074430 0x7fbc3814bef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:40324/0 (socket says 172.21.15.123:40324) 2024-09-16T15:50:46.638 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:46.638+0000 7fbc3c8d8640 1 -- 172.21.15.123:0/1397258697 learned_addr learned my addr 172.21.15.123:0/1397258697 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:50:46.638 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:46.638+0000 7fbc3d0d9640 1 --2- 172.21.15.123:0/1397258697 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fbc380726c0 0x7fbc380ce300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:50:46.638 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:46.638+0000 7fbc3d8da640 1 --2- 172.21.15.123:0/1397258697 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fbc3815bca0 0x7fbc380c5c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:50:46.638 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:46.638+0000 7fbc3c8d8640 1 -- 172.21.15.123:0/1397258697 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fbc3815bca0 msgr2=0x7fbc380c5c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:50:46.639 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:46.638+0000 7fbc3c8d8640 1 --2- 172.21.15.123:0/1397258697 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fbc3815bca0 0x7fbc380c5c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:46.639 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:46.638+0000 7fbc3c8d8640 1 -- 172.21.15.123:0/1397258697 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fbc380726c0 msgr2=0x7fbc380ce300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:50:46.639 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:46.638+0000 7fbc3c8d8640 1 --2- 172.21.15.123:0/1397258697 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fbc380726c0 0x7fbc380ce300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:46.639 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:46.638+0000 7fbc3c8d8640 1 -- 172.21.15.123:0/1397258697 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbc3806ff10 con 0x7fbc38074430 2024-09-16T15:50:46.639 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:46.638+0000 7fbc3d0d9640 1 --2- 172.21.15.123:0/1397258697 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fbc380726c0 0x7fbc380ce300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T15:50:46.639 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:46.638+0000 7fbc3c8d8640 1 --2- 172.21.15.123:0/1397258697 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbc38074430 0x7fbc3814bef0 secure :-1 s=READY pgs=1625 cs=0 l=1 rev1=1 crypto rx=0x7fbc2800b550 tx=0x7fbc2800ba10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:50:46.639 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:46.638+0000 7fbc2dffb640 1 -- 172.21.15.123:0/1397258697 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc28014070 con 0x7fbc38074430 2024-09-16T15:50:46.639 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:46.638+0000 7fbc2dffb640 1 -- 172.21.15.123:0/1397258697 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fbc28010980 con 0x7fbc38074430 2024-09-16T15:50:46.639 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:46.638+0000 7fbc3f33d640 1 -- 172.21.15.123:0/1397258697 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbc380c5200 con 0x7fbc38074430 2024-09-16T15:50:46.639 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:46.638+0000 7fbc2dffb640 1 -- 172.21.15.123:0/1397258697 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc28010dd0 con 0x7fbc38074430 2024-09-16T15:50:46.639 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:46.638+0000 7fbc3f33d640 1 -- 172.21.15.123:0/1397258697 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbc380c5470 con 0x7fbc38074430 2024-09-16T15:50:46.640 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:46.638+0000 7fbc3f33d640 1 -- 172.21.15.123:0/1397258697 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbc38073ef0 con 0x7fbc38074430 2024-09-16T15:50:46.640 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:46.638+0000 7fbc2dffb640 1 -- 172.21.15.123:0/1397258697 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 22) ==== 97692+0+0 (secure 0 0 0) 0x7fbc28017070 con 0x7fbc38074430 2024-09-16T15:50:46.640 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:46.638+0000 7fbc2dffb640 1 --2- 172.21.15.123:0/1397258697 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fbc1c073580 0x7fbc1c075a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:50:46.640 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:46.638+0000 7fbc3d0d9640 1 --2- 172.21.15.123:0/1397258697 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fbc1c073580 0x7fbc1c075a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:50:46.641 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:46.638+0000 7fbc2dffb640 1 -- 172.21.15.123:0/1397258697 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(146..146 src has 1..146) ==== 5737+0+0 (secure 0 0 0) 0x7fbc28094a30 con 0x7fbc38074430 2024-09-16T15:50:46.641 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:46.638+0000 7fbc3d0d9640 1 --2- 172.21.15.123:0/1397258697 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fbc1c073580 0x7fbc1c075a40 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7fbc340af000 tx=0x7fbc3401b040 comp rx=0 tx=0).ready entity=mgr.4150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:50:46.642 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:46.642+0000 7fbc2dffb640 1 -- 172.21.15.123:0/1397258697 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7fbc2805f130 con 0x7fbc38074430 2024-09-16T15:50:46.825 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:46.826+0000 7fbc3f33d640 1 -- 172.21.15.123:0/1397258697 --> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) -- 0x7fbc3815f630 con 0x7fbc1c073580 2024-09-16T15:50:46.827 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:46.826+0000 7fbc2dffb640 1 -- 172.21.15.123:0/1397258697 <== mgr.4150 v2:172.21.15.123:6836/2464759676 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+41 (secure 0 0 0) 0x7fbc3815f630 con 0x7fbc1c073580 2024-09-16T15:50:46.827 INFO:teuthology.orchestra.run.smithi123.stdout:[ 2024-09-16T15:50:46.827 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-09-16T15:50:46.827 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "cephfs" 2024-09-16T15:50:46.828 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-16T15:50:46.828 INFO:teuthology.orchestra.run.smithi123.stdout:] 2024-09-16T15:50:46.829 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:46.830+0000 7fbc3f33d640 1 -- 172.21.15.123:0/1397258697 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fbc1c073580 msgr2=0x7fbc1c075a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:50:46.829 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:46.830+0000 7fbc3f33d640 1 --2- 172.21.15.123:0/1397258697 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fbc1c073580 0x7fbc1c075a40 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7fbc340af000 tx=0x7fbc3401b040 comp rx=0 tx=0).stop 2024-09-16T15:50:46.829 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:46.830+0000 7fbc3f33d640 1 -- 172.21.15.123:0/1397258697 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbc38074430 msgr2=0x7fbc3814bef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:50:46.830 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:46.830+0000 7fbc3f33d640 1 --2- 172.21.15.123:0/1397258697 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbc38074430 0x7fbc3814bef0 secure :-1 s=READY pgs=1625 cs=0 l=1 rev1=1 crypto rx=0x7fbc2800b550 tx=0x7fbc2800ba10 comp rx=0 tx=0).stop 2024-09-16T15:50:46.830 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:46.830+0000 7fbc3f33d640 1 -- 172.21.15.123:0/1397258697 shutdown_connections 2024-09-16T15:50:46.830 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:46.830+0000 7fbc3f33d640 1 --2- 172.21.15.123:0/1397258697 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fbc1c073580 0x7fbc1c075a40 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:46.830 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:46.830+0000 7fbc3f33d640 1 --2- 172.21.15.123:0/1397258697 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fbc3815bca0 0x7fbc380c5c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:46.830 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:46.830+0000 7fbc3f33d640 1 --2- 172.21.15.123:0/1397258697 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbc38074430 0x7fbc3814bef0 unknown :-1 s=CLOSED pgs=1625 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:46.830 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:46.830+0000 7fbc3f33d640 1 --2- 172.21.15.123:0/1397258697 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fbc380726c0 0x7fbc380ce300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:46.830 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:46.830+0000 7fbc3f33d640 1 -- 172.21.15.123:0/1397258697 >> 172.21.15.123:0/1397258697 conn(0x7fbc380cfb10 msgr2=0x7fbc380ca140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:50:46.830 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:46.830+0000 7fbc3f33d640 1 -- 172.21.15.123:0/1397258697 shutdown_connections 2024-09-16T15:50:46.830 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:46.830+0000 7fbc3f33d640 1 -- 172.21.15.123:0/1397258697 wait complete. 2024-09-16T15:50:46.845 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-09-16T15:50:46.961 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:46.962+0000 7f5dad684640 1 -- 172.21.15.123:0/1315503960 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5d90003a90 msgr2=0x7f5d90003eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:50:46.961 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:46.962+0000 7f5dad684640 1 --2- 172.21.15.123:0/1315503960 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5d90003a90 0x7f5d90003eb0 secure :-1 s=READY pgs=1627 cs=0 l=1 rev1=1 crypto rx=0x7f5da80ce090 tx=0x7f5d9c01cc60 comp rx=0 tx=0).stop 2024-09-16T15:50:46.962 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:46.962+0000 7f5da77fe640 1 -- 172.21.15.123:0/1315503960 reap_dead start 2024-09-16T15:50:46.962 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:46.962+0000 7f5dad684640 1 -- 172.21.15.123:0/1315503960 shutdown_connections 2024-09-16T15:50:46.962 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:46.962+0000 7f5dad684640 1 -- 172.21.15.123:0/1315503960 >> 172.21.15.123:0/1315503960 conn(0x7f5da806d670 msgr2=0x7f5da80743a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:50:46.962 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:46.962+0000 7f5dad684640 1 -- 172.21.15.123:0/1315503960 shutdown_connections 2024-09-16T15:50:46.962 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:46.962+0000 7f5dad684640 1 -- 172.21.15.123:0/1315503960 wait complete. 2024-09-16T15:50:46.962 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:46.962+0000 7f5dad684640 1 Processor -- start 2024-09-16T15:50:46.962 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:46.962+0000 7f5dad684640 1 -- start start 2024-09-16T15:50:46.965 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:46.962+0000 7f5dad684640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5d90003a90 0x7f5da81562c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:50:46.965 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:46.962+0000 7f5dad684640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f5da81599a0 0x7f5da8159e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:50:46.965 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:46.962+0000 7f5dad684640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5da80ae300 0x7f5da80ae740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:50:46.965 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:46.962+0000 7f5dad684640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f5da8167e80 con 0x7f5da80ae300 2024-09-16T15:50:46.965 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:46.962+0000 7f5dad684640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f5da815faa0 con 0x7f5d90003a90 2024-09-16T15:50:46.965 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:46.962+0000 7f5dad684640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f5da8168000 con 0x7f5da81599a0 2024-09-16T15:50:46.965 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:46.966+0000 7f5da77fe640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5da80ae300 0x7f5da80ae740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:50:46.966 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:46.966+0000 7f5da77fe640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5da80ae300 0x7f5da80ae740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:40340/0 (socket says 172.21.15.123:40340) 2024-09-16T15:50:46.966 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:46.966+0000 7f5da77fe640 1 -- 172.21.15.123:0/3793777882 learned_addr learned my addr 172.21.15.123:0/3793777882 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:50:46.966 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:46.966+0000 7f5da67fc640 1 --2- 172.21.15.123:0/3793777882 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f5da81599a0 0x7f5da8159e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:50:46.966 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:46.966+0000 7f5da6ffd640 1 --2- 172.21.15.123:0/3793777882 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5d90003a90 0x7f5da81562c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:50:46.966 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:46.966+0000 7f5da77fe640 1 -- 172.21.15.123:0/3793777882 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f5da81599a0 msgr2=0x7f5da8159e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:50:46.966 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:46.966+0000 7f5da77fe640 1 --2- 172.21.15.123:0/3793777882 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f5da81599a0 0x7f5da8159e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:46.966 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:46.966+0000 7f5da77fe640 1 -- 172.21.15.123:0/3793777882 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5d90003a90 msgr2=0x7f5da81562c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:50:46.966 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:46.966+0000 7f5da77fe640 1 --2- 172.21.15.123:0/3793777882 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5d90003a90 0x7f5da81562c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:46.966 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:46.966+0000 7f5da77fe640 1 -- 172.21.15.123:0/3793777882 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5da80af070 con 0x7f5da80ae300 2024-09-16T15:50:46.966 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:46.966+0000 7f5da77fe640 1 --2- 172.21.15.123:0/3793777882 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5da80ae300 0x7f5da80ae740 secure :-1 s=READY pgs=1628 cs=0 l=1 rev1=1 crypto rx=0x7f5da8075340 tx=0x7f5da000dd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:50:46.966 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:46.966+0000 7f5d8f7fe640 1 -- 172.21.15.123:0/3793777882 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5da0016020 con 0x7f5da80ae300 2024-09-16T15:50:46.967 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:46.966+0000 7f5d8f7fe640 1 -- 172.21.15.123:0/3793777882 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f5da000b8a0 con 0x7f5da80ae300 2024-09-16T15:50:46.967 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:46.966+0000 7f5dad684640 1 -- 172.21.15.123:0/3793777882 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5da80c4300 con 0x7f5da80ae300 2024-09-16T15:50:46.967 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:46.966+0000 7f5d8f7fe640 1 -- 172.21.15.123:0/3793777882 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5da000bcf0 con 0x7f5da80ae300 2024-09-16T15:50:46.967 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:46.966+0000 7f5dad684640 1 -- 172.21.15.123:0/3793777882 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5da80aec80 con 0x7f5da80ae300 2024-09-16T15:50:46.967 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:46.966+0000 7f5dad684640 1 -- 172.21.15.123:0/3793777882 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5da815a360 con 0x7f5da80ae300 2024-09-16T15:50:46.968 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:46.966+0000 7f5d8f7fe640 1 -- 172.21.15.123:0/3793777882 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 22) ==== 97692+0+0 (secure 0 0 0) 0x7f5da0019020 con 0x7f5da80ae300 2024-09-16T15:50:46.968 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:46.966+0000 7f5d8f7fe640 1 --2- 172.21.15.123:0/3793777882 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f5d94073580 0x7f5d94075a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:50:46.968 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:46.966+0000 7f5d8f7fe640 1 -- 172.21.15.123:0/3793777882 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(146..146 src has 1..146) ==== 5737+0+0 (secure 0 0 0) 0x7f5da0094690 con 0x7f5da80ae300 2024-09-16T15:50:46.968 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:46.966+0000 7f5da6ffd640 1 --2- 172.21.15.123:0/3793777882 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f5d94073580 0x7f5d94075a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:50:46.968 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:46.966+0000 7f5da6ffd640 1 --2- 172.21.15.123:0/3793777882 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f5d94073580 0x7f5d94075a40 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f5d9c0b0000 tx=0x7f5d9c0a7040 comp rx=0 tx=0).ready entity=mgr.4150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:50:46.970 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:46.970+0000 7f5d8f7fe640 1 -- 172.21.15.123:0/3793777882 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f5da0020030 con 0x7f5da80ae300 2024-09-16T15:50:47.129 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:47.130+0000 7f5dad684640 1 -- 172.21.15.123:0/3793777882 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) -- 0x7f5da80d10b0 con 0x7f5da80ae300 2024-09-16T15:50:47.130 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:47.130+0000 7f5d8f7fe640 1 -- 172.21.15.123:0/3793777882 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v44)=0 v44) ==== 122+0+0 (secure 0 0 0) 0x7f5da005efb0 con 0x7f5da80ae300 2024-09-16T15:50:47.132 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:47.134+0000 7f5dad684640 1 -- 172.21.15.123:0/3793777882 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f5d94073580 msgr2=0x7f5d94075a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:50:47.132 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:47.134+0000 7f5dad684640 1 --2- 172.21.15.123:0/3793777882 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f5d94073580 0x7f5d94075a40 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f5d9c0b0000 tx=0x7f5d9c0a7040 comp rx=0 tx=0).stop 2024-09-16T15:50:47.132 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:47.134+0000 7f5dad684640 1 -- 172.21.15.123:0/3793777882 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5da80ae300 msgr2=0x7f5da80ae740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:50:47.132 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:47.134+0000 7f5dad684640 1 --2- 172.21.15.123:0/3793777882 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5da80ae300 0x7f5da80ae740 secure :-1 s=READY pgs=1628 cs=0 l=1 rev1=1 crypto rx=0x7f5da8075340 tx=0x7f5da000dd50 comp rx=0 tx=0).stop 2024-09-16T15:50:47.132 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:47.134+0000 7f5dad684640 1 -- 172.21.15.123:0/3793777882 shutdown_connections 2024-09-16T15:50:47.132 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:47.134+0000 7f5dad684640 1 --2- 172.21.15.123:0/3793777882 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f5d94073580 0x7f5d94075a40 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:47.133 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:47.134+0000 7f5dad684640 1 --2- 172.21.15.123:0/3793777882 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5da80ae300 0x7f5da80ae740 unknown :-1 s=CLOSED pgs=1628 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:47.133 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:47.134+0000 7f5dad684640 1 --2- 172.21.15.123:0/3793777882 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f5da81599a0 0x7f5da8159e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:47.133 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:47.134+0000 7f5dad684640 1 --2- 172.21.15.123:0/3793777882 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5d90003a90 0x7f5da81562c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:47.133 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:47.134+0000 7f5dad684640 1 -- 172.21.15.123:0/3793777882 >> 172.21.15.123:0/3793777882 conn(0x7f5da806d670 msgr2=0x7f5da80b03e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:50:47.133 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:47.134+0000 7f5dad684640 1 -- 172.21.15.123:0/3793777882 shutdown_connections 2024-09-16T15:50:47.133 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:47.134+0000 7f5dad684640 1 -- 172.21.15.123:0/3793777882 wait complete. 2024-09-16T15:50:47.147 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:50:47.148 DEBUG:teuthology.orchestra.run.smithi123:> (cd /home/ubuntu/cephtest/mnt.0 && exec sudo mkdir -p ./volumes/_nogroup/subvol75470) 2024-09-16T15:50:47.176 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:50:47.176 DEBUG:teuthology.orchestra.run.smithi123:> (cd /home/ubuntu/cephtest/mnt.0 && exec sudo mkdir -p ./volumes/subvol_grp44440/subvol82901) 2024-09-16T15:50:47.239 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs subvol75470 2024-09-16T15:50:47.413 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:47.414+0000 7fd0fab45640 1 -- 172.21.15.123:0/1394545421 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd0f40b0b10 msgr2=0x7fd0f40b3d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:50:47.413 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:47.414+0000 7fd0fab45640 1 --2- 172.21.15.123:0/1394545421 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd0f40b0b10 0x7fd0f40b3d70 secure :-1 s=READY pgs=1629 cs=0 l=1 rev1=1 crypto rx=0x7fd0e8009900 tx=0x7fd0e801c820 comp rx=0 tx=0).stop 2024-09-16T15:50:47.413 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:47.414+0000 7fd0fab45640 1 -- 172.21.15.123:0/1394545421 shutdown_connections 2024-09-16T15:50:47.413 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:47.414+0000 7fd0fab45640 1 --2- 172.21.15.123:0/1394545421 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fd0f40d44e0 0x7fd0f40b42b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:47.414 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:47.414+0000 7fd0fab45640 1 --2- 172.21.15.123:0/1394545421 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd0f40b0b10 0x7fd0f40b3d70 unknown :-1 s=CLOSED pgs=1629 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:47.414 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:47.414+0000 7fd0fab45640 1 -- 172.21.15.123:0/1394545421 >> 172.21.15.123:0/1394545421 conn(0x7fd0f40696d0 msgr2=0x7fd0f409d1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:50:47.414 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:47.414+0000 7fd0fab45640 1 -- 172.21.15.123:0/1394545421 shutdown_connections 2024-09-16T15:50:47.414 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:47.414+0000 7fd0fab45640 1 -- 172.21.15.123:0/1394545421 wait complete. 2024-09-16T15:50:47.416 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:47.414+0000 7fd0fab45640 1 Processor -- start 2024-09-16T15:50:47.416 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:47.414+0000 7fd0fab45640 1 -- start start 2024-09-16T15:50:47.416 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:47.414+0000 7fd0fab45640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd0f40d44e0 0x7fd0f414fc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:50:47.416 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:47.414+0000 7fd0fab45640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fd0f415dcc0 0x7fd0f40caf10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:50:47.416 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:47.414+0000 7fd0fab45640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd0f40b8840 0x7fd0f4156580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:50:47.416 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:47.414+0000 7fd0fab45640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fd0f416b310 con 0x7fd0f40d44e0 2024-09-16T15:50:47.416 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:47.414+0000 7fd0fab45640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fd0f416b190 con 0x7fd0f40b8840 2024-09-16T15:50:47.416 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:47.414+0000 7fd0fab45640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fd0f416b490 con 0x7fd0f415dcc0 2024-09-16T15:50:47.416 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:47.414+0000 7fd0f88e1640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd0f40d44e0 0x7fd0f414fc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:50:47.416 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:47.414+0000 7fd0f88e1640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd0f40d44e0 0x7fd0f414fc10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:40350/0 (socket says 172.21.15.123:40350) 2024-09-16T15:50:47.416 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:47.414+0000 7fd0f88e1640 1 -- 172.21.15.123:0/1276163718 learned_addr learned my addr 172.21.15.123:0/1276163718 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:50:47.417 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:47.414+0000 7fd0f88e1640 1 -- 172.21.15.123:0/1276163718 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fd0f415dcc0 msgr2=0x7fd0f40caf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:50:47.417 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:47.414+0000 7fd0f88e1640 1 --2- 172.21.15.123:0/1276163718 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fd0f415dcc0 0x7fd0f40caf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:47.417 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:47.414+0000 7fd0f88e1640 1 -- 172.21.15.123:0/1276163718 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd0f40b8840 msgr2=0x7fd0f4156580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:50:47.417 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:47.414+0000 7fd0f88e1640 1 --2- 172.21.15.123:0/1276163718 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd0f40b8840 0x7fd0f4156580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:47.417 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:47.414+0000 7fd0f88e1640 1 -- 172.21.15.123:0/1276163718 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd0f41fb730 con 0x7fd0f40d44e0 2024-09-16T15:50:47.417 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:47.418+0000 7fd0f88e1640 1 --2- 172.21.15.123:0/1276163718 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd0f40d44e0 0x7fd0f414fc10 secure :-1 s=READY pgs=1630 cs=0 l=1 rev1=1 crypto rx=0x7fd0e400c490 tx=0x7fd0e4001e80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:50:47.417 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:47.418+0000 7fd0f17fa640 1 -- 172.21.15.123:0/1276163718 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0e4017070 con 0x7fd0f40d44e0 2024-09-16T15:50:47.417 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:47.418+0000 7fd0f17fa640 1 -- 172.21.15.123:0/1276163718 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fd0e4003600 con 0x7fd0f40d44e0 2024-09-16T15:50:47.417 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:47.418+0000 7fd0fab45640 1 -- 172.21.15.123:0/1276163718 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd0f41fba10 con 0x7fd0f40d44e0 2024-09-16T15:50:47.417 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:47.418+0000 7fd0f17fa640 1 -- 172.21.15.123:0/1276163718 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0e4003a90 con 0x7fd0f40d44e0 2024-09-16T15:50:47.417 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:47.418+0000 7fd0fab45640 1 -- 172.21.15.123:0/1276163718 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd0f41fbfd0 con 0x7fd0f40d44e0 2024-09-16T15:50:47.418 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:47.418+0000 7fd0fab45640 1 -- 172.21.15.123:0/1276163718 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd0f40d3dd0 con 0x7fd0f40d44e0 2024-09-16T15:50:47.418 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:47.418+0000 7fd0f17fa640 1 -- 172.21.15.123:0/1276163718 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 22) ==== 97692+0+0 (secure 0 0 0) 0x7fd0e401a070 con 0x7fd0f40d44e0 2024-09-16T15:50:47.418 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:47.418+0000 7fd0f17fa640 1 --2- 172.21.15.123:0/1276163718 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fd0e00735b0 0x7fd0e0075a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:50:47.418 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:47.418+0000 7fd0f3fff640 1 --2- 172.21.15.123:0/1276163718 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fd0e00735b0 0x7fd0e0075a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:50:47.418 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:47.418+0000 7fd0f17fa640 1 -- 172.21.15.123:0/1276163718 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(146..146 src has 1..146) ==== 5737+0+0 (secure 0 0 0) 0x7fd0e4093f00 con 0x7fd0f40d44e0 2024-09-16T15:50:47.419 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:47.418+0000 7fd0f3fff640 1 --2- 172.21.15.123:0/1276163718 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fd0e00735b0 0x7fd0e0075a70 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7fd0e80af000 tx=0x7fd0e80195c0 comp rx=0 tx=0).ready entity=mgr.4150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:50:47.421 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:47.422+0000 7fd0f17fa640 1 -- 172.21.15.123:0/1276163718 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7fd0e405e600 con 0x7fd0f40d44e0 2024-09-16T15:50:47.620 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:47.622+0000 7fd0fab45640 1 -- 172.21.15.123:0/1276163718 --> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "subvol75470", "target": ["mon-mgr", ""]}) -- 0x7fd0f4157e90 con 0x7fd0e00735b0 2024-09-16T15:50:47.627 INFO:tasks.ceph.mgr.y.smithi123.stderr:2024-09-16T15:50:47.626+0000 7fa9b36f4640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-16T15:50:47.627 INFO:tasks.ceph.mgr.y.smithi123.stderr:2024-09-16T15:50:47.626+0000 7fa9b36f4640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-16T15:50:47.627 INFO:tasks.ceph.mgr.y.smithi123.stderr:2024-09-16T15:50:47.626+0000 7fa9b36f4640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-16T15:50:47.627 INFO:tasks.ceph.mgr.y.smithi123.stderr:2024-09-16T15:50:47.626+0000 7fa9b36f4640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-16T15:50:47.627 INFO:tasks.ceph.mgr.y.smithi123.stderr:2024-09-16T15:50:47.626+0000 7fa9b36f4640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-16T15:50:47.683 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:47.682+0000 7fd0f17fa640 1 -- 172.21.15.123:0/1276163718 <== mgr.4150 v2:172.21.15.123:6836/2464759676 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+30 (secure 0 0 0) 0x7fd0f4157e90 con 0x7fd0e00735b0 2024-09-16T15:50:47.683 INFO:teuthology.orchestra.run.smithi123.stdout:/volumes/_nogroup/subvol75470 2024-09-16T15:50:47.685 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:47.686+0000 7fd0fab45640 1 -- 172.21.15.123:0/1276163718 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fd0e00735b0 msgr2=0x7fd0e0075a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:50:47.686 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:47.686+0000 7fd0fab45640 1 --2- 172.21.15.123:0/1276163718 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fd0e00735b0 0x7fd0e0075a70 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7fd0e80af000 tx=0x7fd0e80195c0 comp rx=0 tx=0).stop 2024-09-16T15:50:47.686 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:47.686+0000 7fd0fab45640 1 -- 172.21.15.123:0/1276163718 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd0f40d44e0 msgr2=0x7fd0f414fc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:50:47.686 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:47.686+0000 7fd0fab45640 1 --2- 172.21.15.123:0/1276163718 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd0f40d44e0 0x7fd0f414fc10 secure :-1 s=READY pgs=1630 cs=0 l=1 rev1=1 crypto rx=0x7fd0e400c490 tx=0x7fd0e4001e80 comp rx=0 tx=0).stop 2024-09-16T15:50:47.686 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:47.686+0000 7fd0fab45640 1 -- 172.21.15.123:0/1276163718 shutdown_connections 2024-09-16T15:50:47.686 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:47.686+0000 7fd0fab45640 1 --2- 172.21.15.123:0/1276163718 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fd0e00735b0 0x7fd0e0075a70 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:47.686 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:47.686+0000 7fd0fab45640 1 --2- 172.21.15.123:0/1276163718 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd0f40b8840 0x7fd0f4156580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:47.686 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:47.686+0000 7fd0fab45640 1 --2- 172.21.15.123:0/1276163718 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fd0f415dcc0 0x7fd0f40caf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:47.686 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:47.686+0000 7fd0fab45640 1 --2- 172.21.15.123:0/1276163718 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd0f40d44e0 0x7fd0f414fc10 unknown :-1 s=CLOSED pgs=1630 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:47.686 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:47.686+0000 7fd0fab45640 1 -- 172.21.15.123:0/1276163718 >> 172.21.15.123:0/1276163718 conn(0x7fd0f40696d0 msgr2=0x7fd0f40d0520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:50:47.686 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:47.686+0000 7fd0fab45640 1 -- 172.21.15.123:0/1276163718 shutdown_connections 2024-09-16T15:50:47.686 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:47.686+0000 7fd0fab45640 1 -- 172.21.15.123:0/1276163718 wait complete. 2024-09-16T15:50:47.702 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs subvol82901 subvol_grp44440 2024-09-16T15:50:47.822 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:47.822+0000 7fc186ad9640 1 -- 172.21.15.123:0/1849732040 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc168004dc0 msgr2=0x7fc168005240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:50:47.822 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:47.822+0000 7fc186ad9640 1 --2- 172.21.15.123:0/1849732040 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc168004dc0 0x7fc168005240 secure :-1 s=READY pgs=1634 cs=0 l=1 rev1=1 crypto rx=0x7fc17800c770 tx=0x7fc17801d300 comp rx=0 tx=0).stop 2024-09-16T15:50:47.822 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:47.822+0000 7fc185076640 1 -- 172.21.15.123:0/1849732040 reap_dead start 2024-09-16T15:50:47.822 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:47.822+0000 7fc186ad9640 1 -- 172.21.15.123:0/1849732040 shutdown_connections 2024-09-16T15:50:47.822 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:47.822+0000 7fc186ad9640 1 -- 172.21.15.123:0/1849732040 >> 172.21.15.123:0/1849732040 conn(0x7fc1800b1630 msgr2=0x7fc1800b1a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:50:47.822 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:47.822+0000 7fc186ad9640 1 -- 172.21.15.123:0/1849732040 shutdown_connections 2024-09-16T15:50:47.822 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:47.822+0000 7fc186ad9640 1 -- 172.21.15.123:0/1849732040 wait complete. 2024-09-16T15:50:47.824 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:47.822+0000 7fc186ad9640 1 Processor -- start 2024-09-16T15:50:47.824 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:47.822+0000 7fc186ad9640 1 -- start start 2024-09-16T15:50:47.824 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:47.822+0000 7fc186ad9640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc1800afd90 0x7fc1800b2a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:50:47.824 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:47.822+0000 7fc186ad9640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fc1800b0170 0x7fc1800c6280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:50:47.825 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:47.826+0000 7fc186ad9640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc1800c67c0 0x7fc1800b40b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:50:47.825 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:47.826+0000 7fc186ad9640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fc18015f670 con 0x7fc1800afd90 2024-09-16T15:50:47.825 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:47.826+0000 7fc186ad9640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fc18006f1b0 con 0x7fc1800c67c0 2024-09-16T15:50:47.825 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:47.826+0000 7fc186ad9640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fc18015f7f0 con 0x7fc1800b0170 2024-09-16T15:50:47.825 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:47.826+0000 7fc184875640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc1800afd90 0x7fc1800b2a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:50:47.825 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:47.826+0000 7fc184875640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc1800afd90 0x7fc1800b2a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:40406/0 (socket says 172.21.15.123:40406) 2024-09-16T15:50:47.825 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:47.826+0000 7fc184875640 1 -- 172.21.15.123:0/3226858047 learned_addr learned my addr 172.21.15.123:0/3226858047 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:50:47.825 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:47.826+0000 7fc184875640 1 -- 172.21.15.123:0/3226858047 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fc1800b0170 msgr2=0x7fc1800c6280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:50:47.825 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:47.826+0000 7fc184875640 1 --2- 172.21.15.123:0/3226858047 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fc1800b0170 0x7fc1800c6280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:47.825 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:47.826+0000 7fc185076640 1 --2- 172.21.15.123:0/3226858047 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc1800c67c0 0x7fc1800b40b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:50:47.825 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:47.826+0000 7fc184875640 1 -- 172.21.15.123:0/3226858047 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc1800c67c0 msgr2=0x7fc1800b40b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:50:47.826 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:47.826+0000 7fc184875640 1 --2- 172.21.15.123:0/3226858047 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc1800c67c0 0x7fc1800b40b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:47.826 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:47.826+0000 7fc184875640 1 -- 172.21.15.123:0/3226858047 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc1801f7e40 con 0x7fc1800afd90 2024-09-16T15:50:47.826 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:47.826+0000 7fc184875640 1 --2- 172.21.15.123:0/3226858047 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc1800afd90 0x7fc1800b2a30 secure :-1 s=READY pgs=1635 cs=0 l=1 rev1=1 crypto rx=0x7fc17000c430 tx=0x7fc17000c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:50:47.826 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:47.826+0000 7fc17d7fa640 1 -- 172.21.15.123:0/3226858047 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc170011790 con 0x7fc1800afd90 2024-09-16T15:50:47.826 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:47.826+0000 7fc17d7fa640 1 -- 172.21.15.123:0/3226858047 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fc170017070 con 0x7fc1800afd90 2024-09-16T15:50:47.826 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:47.826+0000 7fc186ad9640 1 -- 172.21.15.123:0/3226858047 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc1801f8120 con 0x7fc1800afd90 2024-09-16T15:50:47.826 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:47.826+0000 7fc17d7fa640 1 -- 172.21.15.123:0/3226858047 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc170011cc0 con 0x7fc1800afd90 2024-09-16T15:50:47.826 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:47.826+0000 7fc186ad9640 1 -- 172.21.15.123:0/3226858047 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc1801f86e0 con 0x7fc1800afd90 2024-09-16T15:50:47.827 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:47.826+0000 7fc186ad9640 1 -- 172.21.15.123:0/3226858047 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc1800c6ba0 con 0x7fc1800afd90 2024-09-16T15:50:47.827 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:47.826+0000 7fc17d7fa640 1 -- 172.21.15.123:0/3226858047 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 22) ==== 97692+0+0 (secure 0 0 0) 0x7fc17001c460 con 0x7fc1800afd90 2024-09-16T15:50:47.827 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:47.826+0000 7fc17d7fa640 1 --2- 172.21.15.123:0/3226858047 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fc16c0735b0 0x7fc16c075a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:50:47.827 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:47.826+0000 7fc17ffff640 1 --2- 172.21.15.123:0/3226858047 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fc16c0735b0 0x7fc16c075a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:50:47.827 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:47.826+0000 7fc17d7fa640 1 -- 172.21.15.123:0/3226858047 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(146..146 src has 1..146) ==== 5737+0+0 (secure 0 0 0) 0x7fc170094880 con 0x7fc1800afd90 2024-09-16T15:50:47.827 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:47.826+0000 7fc17ffff640 1 --2- 172.21.15.123:0/3226858047 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fc16c0735b0 0x7fc16c075a70 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7fc174005d90 tx=0x7fc174005ca0 comp rx=0 tx=0).ready entity=mgr.4150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:50:47.830 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:47.830+0000 7fc17d7fa640 1 -- 172.21.15.123:0/3226858047 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7fc17005f110 con 0x7fc1800afd90 2024-09-16T15:50:48.034 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:48.034+0000 7fc186ad9640 1 -- 172.21.15.123:0/3226858047 --> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "subvol82901", "group_name": "subvol_grp44440", "target": ["mon-mgr", ""]}) -- 0x7fc1801f4650 con 0x7fc16c0735b0 2024-09-16T15:50:48.073 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:48.074+0000 7fc17d7fa640 1 -- 172.21.15.123:0/3226858047 <== mgr.4150 v2:172.21.15.123:6836/2464759676 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7fc17400e940 con 0x7fc16c0735b0 2024-09-16T15:50:48.073 INFO:teuthology.orchestra.run.smithi123.stdout:/volumes/subvol_grp44440/subvol82901 2024-09-16T15:50:48.076 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:48.074+0000 7fc186ad9640 1 -- 172.21.15.123:0/3226858047 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fc16c0735b0 msgr2=0x7fc16c075a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:50:48.076 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:48.074+0000 7fc186ad9640 1 --2- 172.21.15.123:0/3226858047 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fc16c0735b0 0x7fc16c075a70 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7fc174005d90 tx=0x7fc174005ca0 comp rx=0 tx=0).stop 2024-09-16T15:50:48.076 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:48.074+0000 7fc186ad9640 1 -- 172.21.15.123:0/3226858047 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc1800afd90 msgr2=0x7fc1800b2a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:50:48.076 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:48.074+0000 7fc186ad9640 1 --2- 172.21.15.123:0/3226858047 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc1800afd90 0x7fc1800b2a30 secure :-1 s=READY pgs=1635 cs=0 l=1 rev1=1 crypto rx=0x7fc17000c430 tx=0x7fc17000c8f0 comp rx=0 tx=0).stop 2024-09-16T15:50:48.076 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:48.074+0000 7fc186ad9640 1 -- 172.21.15.123:0/3226858047 shutdown_connections 2024-09-16T15:50:48.076 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:48.074+0000 7fc186ad9640 1 --2- 172.21.15.123:0/3226858047 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fc16c0735b0 0x7fc16c075a70 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:48.076 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:48.074+0000 7fc186ad9640 1 --2- 172.21.15.123:0/3226858047 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc1800c67c0 0x7fc1800b40b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:48.076 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:48.074+0000 7fc186ad9640 1 --2- 172.21.15.123:0/3226858047 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fc1800b0170 0x7fc1800c6280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:48.076 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:48.074+0000 7fc186ad9640 1 --2- 172.21.15.123:0/3226858047 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc1800afd90 0x7fc1800b2a30 unknown :-1 s=CLOSED pgs=1635 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:48.076 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:48.074+0000 7fc186ad9640 1 -- 172.21.15.123:0/3226858047 >> 172.21.15.123:0/3226858047 conn(0x7fc1800b1630 msgr2=0x7fc180151f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:50:48.076 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:48.078+0000 7fc186ad9640 1 -- 172.21.15.123:0/3226858047 shutdown_connections 2024-09-16T15:50:48.077 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:48.078+0000 7fc186ad9640 1 -- 172.21.15.123:0/3226858047 wait complete. 2024-09-16T15:50:48.092 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs subvol75470 2024-09-16T15:50:48.208 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:48.210+0000 7f34e800f640 1 -- 172.21.15.123:0/3205072745 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f34e015c9b0 msgr2=0x7f34e0073500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:50:48.208 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:48.210+0000 7f34e800f640 1 --2- 172.21.15.123:0/3205072745 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f34e015c9b0 0x7f34e0073500 secure :-1 s=READY pgs=1636 cs=0 l=1 rev1=1 crypto rx=0x7f34e00741e0 tx=0x7f34dc01c800 comp rx=0 tx=0).stop 2024-09-16T15:50:48.208 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:48.210+0000 7f34e800f640 1 -- 172.21.15.123:0/3205072745 shutdown_connections 2024-09-16T15:50:48.209 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:48.210+0000 7f34e800f640 1 --2- 172.21.15.123:0/3205072745 >> v2:172.21.15.184:3300/0 conn(0x7f34e00cc7f0 0x7f34e00ccc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:48.209 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:48.210+0000 7f34e800f640 1 --2- 172.21.15.123:0/3205072745 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f34e015cd90 0x7f34e0073a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:48.209 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:48.210+0000 7f34e800f640 1 --2- 172.21.15.123:0/3205072745 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f34e015c9b0 0x7f34e0073500 unknown :-1 s=CLOSED pgs=1636 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:48.209 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:48.210+0000 7f34e800f640 1 -- 172.21.15.123:0/3205072745 >> 172.21.15.123:0/3205072745 conn(0x7f34e00d0bc0 msgr2=0x7f34e00d0fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:50:48.209 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:48.210+0000 7f34e800f640 1 -- 172.21.15.123:0/3205072745 shutdown_connections 2024-09-16T15:50:48.209 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:48.210+0000 7f34e800f640 1 -- 172.21.15.123:0/3205072745 wait complete. 2024-09-16T15:50:48.209 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:48.210+0000 7f34e800f640 1 Processor -- start 2024-09-16T15:50:48.210 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:48.210+0000 7f34e800f640 1 -- start start 2024-09-16T15:50:48.211 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:48.210+0000 7f34e800f640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f34e00cc7f0 0x7f34e014b1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:50:48.211 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:48.210+0000 7f34e800f640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f34e015c9b0 0x7f34e014dee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:50:48.211 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:48.210+0000 7f34e800f640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f34e015cd90 0x7f34e0154a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:50:48.211 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:48.210+0000 7f34e800f640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f34e0156500 con 0x7f34e015cd90 2024-09-16T15:50:48.212 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:48.210+0000 7f34e800f640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f34e0156380 con 0x7f34e00cc7f0 2024-09-16T15:50:48.212 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:48.210+0000 7f34e800f640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f34e00ce790 con 0x7f34e015c9b0 2024-09-16T15:50:48.212 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:48.210+0000 7f34e65ac640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f34e015cd90 0x7f34e0154a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:50:48.212 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:48.210+0000 7f34e65ac640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f34e015cd90 0x7f34e0154a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:40422/0 (socket says 172.21.15.123:40422) 2024-09-16T15:50:48.212 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:48.210+0000 7f34e65ac640 1 -- 172.21.15.123:0/1836114239 learned_addr learned my addr 172.21.15.123:0/1836114239 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:50:48.212 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:48.210+0000 7f34e55aa640 1 --2- 172.21.15.123:0/1836114239 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f34e015c9b0 0x7f34e014dee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:50:48.212 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:48.210+0000 7f34e65ac640 1 -- 172.21.15.123:0/1836114239 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f34e015c9b0 msgr2=0x7f34e014dee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:50:48.212 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:48.210+0000 7f34e65ac640 1 --2- 172.21.15.123:0/1836114239 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f34e015c9b0 0x7f34e014dee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:48.212 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:48.210+0000 7f34e65ac640 1 -- 172.21.15.123:0/1836114239 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f34e00cc7f0 msgr2=0x7f34e014b1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:50:48.212 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:48.210+0000 7f34e65ac640 1 --2- 172.21.15.123:0/1836114239 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f34e00cc7f0 0x7f34e014b1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:48.212 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:48.210+0000 7f34e65ac640 1 -- 172.21.15.123:0/1836114239 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f34e014b730 con 0x7f34e015cd90 2024-09-16T15:50:48.212 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:48.210+0000 7f34e65ac640 1 --2- 172.21.15.123:0/1836114239 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f34e015cd90 0x7f34e0154a70 secure :-1 s=READY pgs=1637 cs=0 l=1 rev1=1 crypto rx=0x7f34d800ccb0 tx=0x7f34d800a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:50:48.212 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:48.210+0000 7f34d67fc640 1 -- 172.21.15.123:0/1836114239 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34d8004b30 con 0x7f34e015cd90 2024-09-16T15:50:48.213 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:48.210+0000 7f34d67fc640 1 -- 172.21.15.123:0/1836114239 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f34d8004de0 con 0x7f34e015cd90 2024-09-16T15:50:48.213 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:48.210+0000 7f34d67fc640 1 -- 172.21.15.123:0/1836114239 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34d8005280 con 0x7f34e015cd90 2024-09-16T15:50:48.213 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:48.210+0000 7f34e800f640 1 -- 172.21.15.123:0/1836114239 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f34e00c8d20 con 0x7f34e015cd90 2024-09-16T15:50:48.213 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:48.210+0000 7f34e800f640 1 -- 172.21.15.123:0/1836114239 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f34e014ba50 con 0x7f34e015cd90 2024-09-16T15:50:48.214 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:48.214+0000 7f34e800f640 1 -- 172.21.15.123:0/1836114239 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f34e00ccc40 con 0x7f34e015cd90 2024-09-16T15:50:48.214 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:48.214+0000 7f34d67fc640 1 -- 172.21.15.123:0/1836114239 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 22) ==== 97692+0+0 (secure 0 0 0) 0x7f34d8005420 con 0x7f34e015cd90 2024-09-16T15:50:48.214 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:48.214+0000 7f34d67fc640 1 --2- 172.21.15.123:0/1836114239 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f34c40735b0 0x7f34c4075a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:50:48.214 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:48.214+0000 7f34d67fc640 1 -- 172.21.15.123:0/1836114239 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(146..146 src has 1..146) ==== 5737+0+0 (secure 0 0 0) 0x7f34d8095650 con 0x7f34e015cd90 2024-09-16T15:50:48.214 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:48.214+0000 7f34e5dab640 1 --2- 172.21.15.123:0/1836114239 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f34c40735b0 0x7f34c4075a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:50:48.214 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:48.214+0000 7f34e5dab640 1 --2- 172.21.15.123:0/1836114239 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f34c40735b0 0x7f34c4075a70 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f34dc0af000 tx=0x7f34dc0195a0 comp rx=0 tx=0).ready entity=mgr.4150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:50:48.217 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:48.218+0000 7f34d67fc640 1 -- 172.21.15.123:0/1836114239 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f34d8014030 con 0x7f34e015cd90 2024-09-16T15:50:48.404 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:48.406+0000 7f34e800f640 1 -- 172.21.15.123:0/1836114239 --> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "subvol75470", "target": ["mon-mgr", ""]}) -- 0x7f34e006d7c0 con 0x7f34c40735b0 2024-09-16T15:50:48.410 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:48.410+0000 7f34d67fc640 1 -- 172.21.15.123:0/1836114239 <== mgr.4150 v2:172.21.15.123:6836/2464759676 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+30 (secure 0 0 0) 0x7f34e006d7c0 con 0x7f34c40735b0 2024-09-16T15:50:48.410 INFO:teuthology.orchestra.run.smithi123.stdout:/volumes/_nogroup/subvol75470 2024-09-16T15:50:48.413 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:48.414+0000 7f34e800f640 1 -- 172.21.15.123:0/1836114239 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f34c40735b0 msgr2=0x7f34c4075a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:50:48.413 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:48.414+0000 7f34e800f640 1 --2- 172.21.15.123:0/1836114239 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f34c40735b0 0x7f34c4075a70 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f34dc0af000 tx=0x7f34dc0195a0 comp rx=0 tx=0).stop 2024-09-16T15:50:48.413 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:48.414+0000 7f34e800f640 1 -- 172.21.15.123:0/1836114239 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f34e015cd90 msgr2=0x7f34e0154a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:50:48.413 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:48.414+0000 7f34e800f640 1 --2- 172.21.15.123:0/1836114239 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f34e015cd90 0x7f34e0154a70 secure :-1 s=READY pgs=1637 cs=0 l=1 rev1=1 crypto rx=0x7f34d800ccb0 tx=0x7f34d800a620 comp rx=0 tx=0).stop 2024-09-16T15:50:48.413 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:48.414+0000 7f34e800f640 1 -- 172.21.15.123:0/1836114239 shutdown_connections 2024-09-16T15:50:48.413 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:48.414+0000 7f34e800f640 1 --2- 172.21.15.123:0/1836114239 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f34c40735b0 0x7f34c4075a70 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:48.413 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:48.414+0000 7f34e800f640 1 --2- 172.21.15.123:0/1836114239 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f34e015cd90 0x7f34e0154a70 unknown :-1 s=CLOSED pgs=1637 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:48.413 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:48.414+0000 7f34e800f640 1 --2- 172.21.15.123:0/1836114239 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f34e015c9b0 0x7f34e014dee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:48.413 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:48.414+0000 7f34e800f640 1 --2- 172.21.15.123:0/1836114239 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f34e00cc7f0 0x7f34e014b1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:48.413 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:48.414+0000 7f34e800f640 1 -- 172.21.15.123:0/1836114239 >> 172.21.15.123:0/1836114239 conn(0x7f34e00d0bc0 msgr2=0x7f34e006cbf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:50:48.413 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:48.414+0000 7f34e800f640 1 -- 172.21.15.123:0/1836114239 shutdown_connections 2024-09-16T15:50:48.413 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:48.414+0000 7f34e800f640 1 -- 172.21.15.123:0/1836114239 wait complete. 2024-09-16T15:50:48.427 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:50:48.428 DEBUG:teuthology.orchestra.run.smithi123:> (cd /home/ubuntu/cephtest/mnt.0 && exec sudo cat ./volumes/_legacy/96aabe54b09eae17916b4247524ad6d1.meta) 2024-09-16T15:50:50.798 INFO:teuthology.orchestra.run.smithi123.stdout:[GLOBAL] 2024-09-16T15:50:50.798 INFO:teuthology.orchestra.run.smithi123.stdout:version = 1 2024-09-16T15:50:50.798 INFO:teuthology.orchestra.run.smithi123.stdout:type = subvolume 2024-09-16T15:50:50.798 INFO:teuthology.orchestra.run.smithi123.stdout:path = /volumes/_nogroup/subvol75470 2024-09-16T15:50:50.798 INFO:teuthology.orchestra.run.smithi123.stdout:state = complete 2024-09-16T15:50:50.798 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-09-16T15:50:50.800 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs subvol82901 subvol_grp44440 2024-09-16T15:50:50.940 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:50.942+0000 7f9f7faee640 1 -- 172.21.15.123:0/3732200197 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9f780d1a90 msgr2=0x7f9f780d1f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:50:50.940 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:50.942+0000 7f9f7faee640 1 --2- 172.21.15.123:0/3732200197 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9f780d1a90 0x7f9f780d1f00 secure :-1 s=READY pgs=1638 cs=0 l=1 rev1=1 crypto rx=0x7f9f780cf690 tx=0x7f9f6801c8c0 comp rx=0 tx=0).stop 2024-09-16T15:50:50.940 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:50.942+0000 7f9f7faee640 1 -- 172.21.15.123:0/3732200197 shutdown_connections 2024-09-16T15:50:50.940 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:50.942+0000 7f9f7faee640 1 --2- 172.21.15.123:0/3732200197 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9f780d1a90 0x7f9f780d1f00 unknown :-1 s=CLOSED pgs=1638 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:50.940 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:50.942+0000 7f9f7faee640 1 --2- 172.21.15.123:0/3732200197 >> v2:172.21.15.184:3300/0 conn(0x7f9f780b2840 0x7f9f780d0590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:50.940 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:50.942+0000 7f9f7faee640 1 -- 172.21.15.123:0/3732200197 >> 172.21.15.123:0/3732200197 conn(0x7f9f7809aa40 msgr2=0x7f9f7809ae50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:50:50.940 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:50.942+0000 7f9f7faee640 1 -- 172.21.15.123:0/3732200197 shutdown_connections 2024-09-16T15:50:50.940 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:50.942+0000 7f9f7faee640 1 -- 172.21.15.123:0/3732200197 wait complete. 2024-09-16T15:50:50.941 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:50.942+0000 7f9f7faee640 1 Processor -- start 2024-09-16T15:50:50.941 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:50.942+0000 7f9f7faee640 1 -- start start 2024-09-16T15:50:50.943 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:50.942+0000 7f9f7faee640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f9f780b2840 0x7f9f781568c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:50:50.943 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:50.942+0000 7f9f7faee640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9f780d1a90 0x7f9f7806e710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:50:50.943 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:50.942+0000 7f9f7faee640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f9f7815c230 0x7f9f78154e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:50:50.943 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:50.942+0000 7f9f7faee640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f9f78168e00 con 0x7f9f780d1a90 2024-09-16T15:50:50.943 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:50.942+0000 7f9f7faee640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f9f78168c80 con 0x7f9f7815c230 2024-09-16T15:50:50.943 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:50.942+0000 7f9f7faee640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f9f78168f80 con 0x7f9f780b2840 2024-09-16T15:50:50.943 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:50.942+0000 7f9f7d089640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9f780d1a90 0x7f9f7806e710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:50:50.943 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:50.942+0000 7f9f7d089640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9f780d1a90 0x7f9f7806e710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:40440/0 (socket says 172.21.15.123:40440) 2024-09-16T15:50:50.943 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:50.942+0000 7f9f7d089640 1 -- 172.21.15.123:0/2327304313 learned_addr learned my addr 172.21.15.123:0/2327304313 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:50:50.943 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:50.942+0000 7f9f7d88a640 1 --2- 172.21.15.123:0/2327304313 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f9f780b2840 0x7f9f781568c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:50:50.944 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:50.942+0000 7f9f7e08b640 1 --2- 172.21.15.123:0/2327304313 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f9f7815c230 0x7f9f78154e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:50:50.944 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:50.942+0000 7f9f7d089640 1 -- 172.21.15.123:0/2327304313 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f9f780b2840 msgr2=0x7f9f781568c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:50:50.944 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:50.942+0000 7f9f7d089640 1 --2- 172.21.15.123:0/2327304313 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f9f780b2840 0x7f9f781568c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:50.944 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:50.942+0000 7f9f7d089640 1 -- 172.21.15.123:0/2327304313 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f9f7815c230 msgr2=0x7f9f78154e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:50:50.944 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:50.942+0000 7f9f7d089640 1 --2- 172.21.15.123:0/2327304313 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f9f7815c230 0x7f9f78154e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:50.944 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:50.942+0000 7f9f7d089640 1 -- 172.21.15.123:0/2327304313 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9f7806f650 con 0x7f9f780d1a90 2024-09-16T15:50:50.944 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:50.942+0000 7f9f7d089640 1 --2- 172.21.15.123:0/2327304313 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9f780d1a90 0x7f9f7806e710 secure :-1 s=READY pgs=1639 cs=0 l=1 rev1=1 crypto rx=0x7f9f680b0040 tx=0x7f9f680a5720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:50:50.944 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:50.942+0000 7f9f6e7fc640 1 -- 172.21.15.123:0/2327304313 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f680b2040 con 0x7f9f780d1a90 2024-09-16T15:50:50.944 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:50.942+0000 7f9f6e7fc640 1 -- 172.21.15.123:0/2327304313 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f9f68005030 con 0x7f9f780d1a90 2024-09-16T15:50:50.944 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:50.942+0000 7f9f7faee640 1 -- 172.21.15.123:0/2327304313 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9f7806f090 con 0x7f9f780d1a90 2024-09-16T15:50:50.944 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:50.942+0000 7f9f6e7fc640 1 -- 172.21.15.123:0/2327304313 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f680044f0 con 0x7f9f780d1a90 2024-09-16T15:50:50.944 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:50.942+0000 7f9f7faee640 1 -- 172.21.15.123:0/2327304313 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9f78155de0 con 0x7f9f780d1a90 2024-09-16T15:50:50.945 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:50.946+0000 7f9f7faee640 1 -- 172.21.15.123:0/2327304313 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9f780d0590 con 0x7f9f780d1a90 2024-09-16T15:50:50.945 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:50.946+0000 7f9f6e7fc640 1 -- 172.21.15.123:0/2327304313 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 23) ==== 97743+0+0 (secure 0 0 0) 0x7f9f68005460 con 0x7f9f780d1a90 2024-09-16T15:50:50.945 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:50.946+0000 7f9f6e7fc640 1 --2- 172.21.15.123:0/2327304313 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f9f5c073610 0x7f9f5c075ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:50:50.945 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:50.946+0000 7f9f6e7fc640 1 -- 172.21.15.123:0/2327304313 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(146..146 src has 1..146) ==== 5737+0+0 (secure 0 0 0) 0x7f9f6812ed30 con 0x7f9f780d1a90 2024-09-16T15:50:50.945 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:50.946+0000 7f9f7d88a640 1 --2- 172.21.15.123:0/2327304313 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f9f5c073610 0x7f9f5c075ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:50:50.946 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:50.946+0000 7f9f7d88a640 1 --2- 172.21.15.123:0/2327304313 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f9f5c073610 0x7f9f5c075ad0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f9f740058d0 tx=0x7f9f74005860 comp rx=0 tx=0).ready entity=mgr.4150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:50:50.949 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:50.950+0000 7f9f6e7fc640 1 -- 172.21.15.123:0/2327304313 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f9f680164b0 con 0x7f9f780d1a90 2024-09-16T15:50:51.154 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:51.154+0000 7f9f7faee640 1 -- 172.21.15.123:0/2327304313 --> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "subvol82901", "group_name": "subvol_grp44440", "target": ["mon-mgr", ""]}) -- 0x7f9f780b3400 con 0x7f9f5c073610 2024-09-16T15:50:51.160 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:51.162+0000 7f9f6e7fc640 1 -- 172.21.15.123:0/2327304313 <== mgr.4150 v2:172.21.15.123:6836/2464759676 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f9f780b3400 con 0x7f9f5c073610 2024-09-16T15:50:51.161 INFO:teuthology.orchestra.run.smithi123.stdout:/volumes/subvol_grp44440/subvol82901 2024-09-16T15:50:51.163 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:51.162+0000 7f9f7faee640 1 -- 172.21.15.123:0/2327304313 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f9f5c073610 msgr2=0x7f9f5c075ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:50:51.163 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:51.162+0000 7f9f7faee640 1 --2- 172.21.15.123:0/2327304313 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f9f5c073610 0x7f9f5c075ad0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f9f740058d0 tx=0x7f9f74005860 comp rx=0 tx=0).stop 2024-09-16T15:50:51.163 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:51.162+0000 7f9f7faee640 1 -- 172.21.15.123:0/2327304313 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9f780d1a90 msgr2=0x7f9f7806e710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:50:51.163 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:51.162+0000 7f9f7faee640 1 --2- 172.21.15.123:0/2327304313 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9f780d1a90 0x7f9f7806e710 secure :-1 s=READY pgs=1639 cs=0 l=1 rev1=1 crypto rx=0x7f9f680b0040 tx=0x7f9f680a5720 comp rx=0 tx=0).stop 2024-09-16T15:50:51.163 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:51.162+0000 7f9f7faee640 1 -- 172.21.15.123:0/2327304313 shutdown_connections 2024-09-16T15:50:51.163 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:51.162+0000 7f9f7faee640 1 --2- 172.21.15.123:0/2327304313 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f9f5c073610 0x7f9f5c075ad0 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:51.163 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:51.162+0000 7f9f7faee640 1 --2- 172.21.15.123:0/2327304313 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f9f7815c230 0x7f9f78154e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:51.164 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:51.162+0000 7f9f7faee640 1 --2- 172.21.15.123:0/2327304313 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9f780d1a90 0x7f9f7806e710 unknown :-1 s=CLOSED pgs=1639 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:51.164 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:51.162+0000 7f9f7faee640 1 --2- 172.21.15.123:0/2327304313 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f9f780b2840 0x7f9f781568c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:51.164 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:51.162+0000 7f9f7faee640 1 -- 172.21.15.123:0/2327304313 >> 172.21.15.123:0/2327304313 conn(0x7f9f7809aa40 msgr2=0x7f9f780ce090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:50:51.164 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:51.162+0000 7f9f7faee640 1 -- 172.21.15.123:0/2327304313 shutdown_connections 2024-09-16T15:50:51.164 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:51.162+0000 7f9f7faee640 1 -- 172.21.15.123:0/2327304313 wait complete. 2024-09-16T15:50:51.180 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:50:51.180 DEBUG:teuthology.orchestra.run.smithi123:> (cd /home/ubuntu/cephtest/mnt.0 && exec sudo cat ./volumes/_legacy/123fd69a5f770b439a41e3d4db3e29de.meta) 2024-09-16T15:50:55.795 INFO:teuthology.orchestra.run.smithi123.stdout:[GLOBAL] 2024-09-16T15:50:55.795 INFO:teuthology.orchestra.run.smithi123.stdout:version = 1 2024-09-16T15:50:55.795 INFO:teuthology.orchestra.run.smithi123.stdout:type = subvolume 2024-09-16T15:50:55.795 INFO:teuthology.orchestra.run.smithi123.stdout:path = /volumes/subvol_grp44440/subvol82901 2024-09-16T15:50:55.796 INFO:teuthology.orchestra.run.smithi123.stdout:state = complete 2024-09-16T15:50:55.796 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-09-16T15:50:55.798 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume rm cephfs subvol75470 2024-09-16T15:50:55.937 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:55.938+0000 7f901c84f640 1 -- 172.21.15.123:0/2556182067 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9000004810 msgr2=0x7f9000004c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:50:55.938 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:55.938+0000 7f901c84f640 1 --2- 172.21.15.123:0/2556182067 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9000004810 0x7f9000004c90 secure :-1 s=READY pgs=1641 cs=0 l=1 rev1=1 crypto rx=0x7f900c009b60 tx=0x7f900c01d270 comp rx=0 tx=0).stop 2024-09-16T15:50:55.938 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:55.938+0000 7f9016d9d640 1 -- 172.21.15.123:0/2556182067 reap_dead start 2024-09-16T15:50:55.938 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:55.938+0000 7f901c84f640 1 -- 172.21.15.123:0/2556182067 shutdown_connections 2024-09-16T15:50:55.938 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:55.938+0000 7f901c84f640 1 -- 172.21.15.123:0/2556182067 >> 172.21.15.123:0/2556182067 conn(0x7f901809aa40 msgr2=0x7f901809ae50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:50:55.938 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:55.938+0000 7f901c84f640 1 -- 172.21.15.123:0/2556182067 shutdown_connections 2024-09-16T15:50:55.938 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:55.938+0000 7f901c84f640 1 -- 172.21.15.123:0/2556182067 wait complete. 2024-09-16T15:50:55.938 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:55.938+0000 7f901c84f640 1 Processor -- start 2024-09-16T15:50:55.938 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:55.938+0000 7f901c84f640 1 -- start start 2024-09-16T15:50:55.941 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:55.942+0000 7f901c84f640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9000004810 0x7f90180b6510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:50:55.941 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:55.942+0000 7f901c84f640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f9018154170 0x7f901814bf10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:50:55.941 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:55.942+0000 7f901c84f640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f90180c56e0 0x7f90180c5b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:50:55.941 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:55.942+0000 7f901c84f640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f90180ccb60 con 0x7f9000004810 2024-09-16T15:50:55.941 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:55.942+0000 7f901c84f640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f90180cc9e0 con 0x7f90180c56e0 2024-09-16T15:50:55.941 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:55.942+0000 7f901c84f640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f90180ccce0 con 0x7f9018154170 2024-09-16T15:50:55.941 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:55.942+0000 7f901659c640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9000004810 0x7f90180b6510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:50:55.941 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:55.942+0000 7f901659c640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9000004810 0x7f90180b6510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:49470/0 (socket says 172.21.15.123:49470) 2024-09-16T15:50:55.941 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:55.942+0000 7f901659c640 1 -- 172.21.15.123:0/2175492139 learned_addr learned my addr 172.21.15.123:0/2175492139 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:50:55.941 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:55.942+0000 7f9015d9b640 1 --2- 172.21.15.123:0/2175492139 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f9018154170 0x7f901814bf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:50:55.941 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:55.942+0000 7f9016d9d640 1 --2- 172.21.15.123:0/2175492139 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f90180c56e0 0x7f90180c5b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:50:55.941 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:55.942+0000 7f901659c640 1 -- 172.21.15.123:0/2175492139 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f9018154170 msgr2=0x7f901814bf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:50:55.942 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:55.942+0000 7f901659c640 1 --2- 172.21.15.123:0/2175492139 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f9018154170 0x7f901814bf10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:55.942 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:55.942+0000 7f901659c640 1 -- 172.21.15.123:0/2175492139 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f90180c56e0 msgr2=0x7f90180c5b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:50:55.942 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:55.942+0000 7f901659c640 1 --2- 172.21.15.123:0/2175492139 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f90180c56e0 0x7f90180c5b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:55.942 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:55.942+0000 7f901659c640 1 -- 172.21.15.123:0/2175492139 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f90180b6b60 con 0x7f9000004810 2024-09-16T15:50:55.942 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:55.942+0000 7f9016d9d640 1 --2- 172.21.15.123:0/2175492139 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f90180c56e0 0x7f90180c5b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T15:50:55.942 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:55.942+0000 7f901659c640 1 --2- 172.21.15.123:0/2175492139 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9000004810 0x7f90180b6510 secure :-1 s=READY pgs=1642 cs=0 l=1 rev1=1 crypto rx=0x7f9008010c50 tx=0x7f900800e720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:50:55.942 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:55.942+0000 7f9006ffd640 1 -- 172.21.15.123:0/2175492139 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9008003b80 con 0x7f9000004810 2024-09-16T15:50:55.942 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:55.942+0000 7f9006ffd640 1 -- 172.21.15.123:0/2175492139 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f9008003d20 con 0x7f9000004810 2024-09-16T15:50:55.942 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:55.942+0000 7f901c84f640 1 -- 172.21.15.123:0/2175492139 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f90180b6e40 con 0x7f9000004810 2024-09-16T15:50:55.942 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:55.942+0000 7f9006ffd640 1 -- 172.21.15.123:0/2175492139 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90080042b0 con 0x7f9000004810 2024-09-16T15:50:55.942 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:55.942+0000 7f901c84f640 1 -- 172.21.15.123:0/2175492139 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f90180b7400 con 0x7f9000004810 2024-09-16T15:50:55.943 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:55.942+0000 7f901c84f640 1 -- 172.21.15.123:0/2175492139 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f90180c60a0 con 0x7f9000004810 2024-09-16T15:50:55.943 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:55.942+0000 7f9006ffd640 1 -- 172.21.15.123:0/2175492139 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 23) ==== 97743+0+0 (secure 0 0 0) 0x7f9008004450 con 0x7f9000004810 2024-09-16T15:50:55.943 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:55.942+0000 7f9006ffd640 1 --2- 172.21.15.123:0/2175492139 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f8ff8073610 0x7f8ff8075ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:50:55.943 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:55.942+0000 7f9006ffd640 1 -- 172.21.15.123:0/2175492139 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(146..146 src has 1..146) ==== 5737+0+0 (secure 0 0 0) 0x7f9008095c40 con 0x7f9000004810 2024-09-16T15:50:55.944 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:55.942+0000 7f9015d9b640 1 --2- 172.21.15.123:0/2175492139 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f8ff8073610 0x7f8ff8075ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:50:55.944 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:55.942+0000 7f9015d9b640 1 --2- 172.21.15.123:0/2175492139 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f8ff8073610 0x7f8ff8075ad0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f900c0ae000 tx=0x7f900c01a010 comp rx=0 tx=0).ready entity=mgr.4150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:50:55.947 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:55.946+0000 7f9006ffd640 1 -- 172.21.15.123:0/2175492139 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f90180c60a0 con 0x7f9000004810 2024-09-16T15:50:56.144 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:56.146+0000 7f901c84f640 1 -- 172.21.15.123:0/2175492139 --> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] -- mgr_command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "subvol75470", "target": ["mon-mgr", ""]}) -- 0x7f9018071450 con 0x7f8ff8073610 2024-09-16T15:50:56.174 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:56.174+0000 7f9006ffd640 1 -- 172.21.15.123:0/2175492139 <== mgr.4150 v2:172.21.15.123:6836/2464759676 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f9018071450 con 0x7f8ff8073610 2024-09-16T15:50:56.178 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:56.178+0000 7f901c84f640 1 -- 172.21.15.123:0/2175492139 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f8ff8073610 msgr2=0x7f8ff8075ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:50:56.178 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:56.178+0000 7f901c84f640 1 --2- 172.21.15.123:0/2175492139 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f8ff8073610 0x7f8ff8075ad0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f900c0ae000 tx=0x7f900c01a010 comp rx=0 tx=0).stop 2024-09-16T15:50:56.178 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:56.178+0000 7f901c84f640 1 -- 172.21.15.123:0/2175492139 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9000004810 msgr2=0x7f90180b6510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:50:56.178 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:56.178+0000 7f901c84f640 1 --2- 172.21.15.123:0/2175492139 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9000004810 0x7f90180b6510 secure :-1 s=READY pgs=1642 cs=0 l=1 rev1=1 crypto rx=0x7f9008010c50 tx=0x7f900800e720 comp rx=0 tx=0).stop 2024-09-16T15:50:56.178 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:56.178+0000 7f901c84f640 1 -- 172.21.15.123:0/2175492139 shutdown_connections 2024-09-16T15:50:56.178 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:56.178+0000 7f901c84f640 1 --2- 172.21.15.123:0/2175492139 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f8ff8073610 0x7f8ff8075ad0 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:56.178 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:56.178+0000 7f901c84f640 1 --2- 172.21.15.123:0/2175492139 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f90180c56e0 0x7f90180c5b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:56.178 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:56.178+0000 7f901c84f640 1 --2- 172.21.15.123:0/2175492139 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f9018154170 0x7f901814bf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:56.178 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:56.178+0000 7f901c84f640 1 --2- 172.21.15.123:0/2175492139 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9000004810 0x7f90180b6510 unknown :-1 s=CLOSED pgs=1642 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:56.178 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:56.178+0000 7f901c84f640 1 -- 172.21.15.123:0/2175492139 >> 172.21.15.123:0/2175492139 conn(0x7f901809aa40 msgr2=0x7f9018071be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:50:56.178 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:56.178+0000 7f901c84f640 1 -- 172.21.15.123:0/2175492139 shutdown_connections 2024-09-16T15:50:56.178 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:56.178+0000 7f901c84f640 1 -- 172.21.15.123:0/2175492139 wait complete. 2024-09-16T15:50:56.183 INFO:tasks.ceph.mgr.y.smithi123.stderr:2024-09-16T15:50:56.182+0000 7fa9b56f8640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-16T15:50:56.183 INFO:tasks.ceph.mgr.y.smithi123.stderr:2024-09-16T15:50:56.182+0000 7fa9b56f8640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-16T15:50:56.183 INFO:tasks.ceph.mgr.y.smithi123.stderr:2024-09-16T15:50:56.182+0000 7fa9b56f8640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-16T15:50:56.183 INFO:tasks.ceph.mgr.y.smithi123.stderr:2024-09-16T15:50:56.182+0000 7fa9b56f8640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-16T15:50:56.184 INFO:tasks.ceph.mgr.y.smithi123.stderr:2024-09-16T15:50:56.182+0000 7fa9b56f8640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-16T15:50:56.196 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume rm cephfs subvol82901 subvol_grp44440 2024-09-16T15:50:56.206 INFO:tasks.ceph.mgr.y.smithi123.stderr:2024-09-16T15:50:56.206+0000 7fa9b5ef9640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-16T15:50:56.206 INFO:tasks.ceph.mgr.y.smithi123.stderr:2024-09-16T15:50:56.206+0000 7fa9b5ef9640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-16T15:50:56.206 INFO:tasks.ceph.mgr.y.smithi123.stderr:2024-09-16T15:50:56.206+0000 7fa9b5ef9640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-16T15:50:56.206 INFO:tasks.ceph.mgr.y.smithi123.stderr:2024-09-16T15:50:56.206+0000 7fa9b5ef9640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-16T15:50:56.206 INFO:tasks.ceph.mgr.y.smithi123.stderr:2024-09-16T15:50:56.206+0000 7fa9b5ef9640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-16T15:50:56.307 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:56.306+0000 7f2d07659640 1 -- 172.21.15.123:0/1264462335 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2d00073590 msgr2=0x7f2d00074210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:50:56.307 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:56.306+0000 7f2d07659640 1 --2- 172.21.15.123:0/1264462335 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2d00073590 0x7f2d00074210 secure :-1 s=READY pgs=1648 cs=0 l=1 rev1=1 crypto rx=0x7f2cfc009900 tx=0x7f2cfc01f090 comp rx=0 tx=0).stop 2024-09-16T15:50:56.307 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:56.310+0000 7f2d07659640 1 -- 172.21.15.123:0/1264462335 shutdown_connections 2024-09-16T15:50:56.308 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:56.310+0000 7f2d07659640 1 --2- 172.21.15.123:0/1264462335 >> v2:172.21.15.184:3300/0 conn(0x7f2d000cc150 0x7f2d0015bb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:56.308 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:56.310+0000 7f2d07659640 1 --2- 172.21.15.123:0/1264462335 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2d00073590 0x7f2d00074210 unknown :-1 s=CLOSED pgs=1648 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:56.308 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:56.310+0000 7f2d07659640 1 -- 172.21.15.123:0/1264462335 >> 172.21.15.123:0/1264462335 conn(0x7f2d000d0b30 msgr2=0x7f2d000d0f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:50:56.308 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:56.310+0000 7f2d07659640 1 -- 172.21.15.123:0/1264462335 shutdown_connections 2024-09-16T15:50:56.308 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:56.310+0000 7f2d07659640 1 -- 172.21.15.123:0/1264462335 wait complete. 2024-09-16T15:50:56.308 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:56.310+0000 7f2d07659640 1 Processor -- start 2024-09-16T15:50:56.308 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:56.310+0000 7f2d07659640 1 -- start start 2024-09-16T15:50:56.310 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:56.310+0000 7f2d07659640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2d00073590 0x7f2d001f5730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:50:56.310 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:56.310+0000 7f2d07659640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2d00073970 0x7f2d00155690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:50:56.310 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:56.310+0000 7f2d07659640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f2d000cc150 0x7f2d00155bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:50:56.310 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:56.310+0000 7f2d07659640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f2d0015ef00 con 0x7f2d00073590 2024-09-16T15:50:56.311 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:56.310+0000 7f2d07659640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f2d0015ed80 con 0x7f2d00073970 2024-09-16T15:50:56.311 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:56.310+0000 7f2d07659640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f2d0015f080 con 0x7f2d000cc150 2024-09-16T15:50:56.311 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:56.310+0000 7f2d053f5640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2d00073590 0x7f2d001f5730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:50:56.311 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:56.310+0000 7f2d053f5640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2d00073590 0x7f2d001f5730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:49518/0 (socket says 172.21.15.123:49518) 2024-09-16T15:50:56.311 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:56.310+0000 7f2d053f5640 1 -- 172.21.15.123:0/4217993708 learned_addr learned my addr 172.21.15.123:0/4217993708 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:50:56.311 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:56.310+0000 7f2d05bf6640 1 --2- 172.21.15.123:0/4217993708 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f2d000cc150 0x7f2d00155bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:50:56.311 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:56.310+0000 7f2d04bf4640 1 --2- 172.21.15.123:0/4217993708 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2d00073970 0x7f2d00155690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:50:56.311 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:56.310+0000 7f2d053f5640 1 -- 172.21.15.123:0/4217993708 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f2d000cc150 msgr2=0x7f2d00155bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:50:56.311 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:56.310+0000 7f2d053f5640 1 --2- 172.21.15.123:0/4217993708 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f2d000cc150 0x7f2d00155bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:56.311 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:56.310+0000 7f2d053f5640 1 -- 172.21.15.123:0/4217993708 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2d00073970 msgr2=0x7f2d00155690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:50:56.311 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:56.310+0000 7f2d053f5640 1 --2- 172.21.15.123:0/4217993708 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2d00073970 0x7f2d00155690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:56.311 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:56.310+0000 7f2d053f5640 1 -- 172.21.15.123:0/4217993708 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2d001f37e0 con 0x7f2d00073590 2024-09-16T15:50:56.311 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:56.310+0000 7f2d053f5640 1 --2- 172.21.15.123:0/4217993708 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2d00073590 0x7f2d001f5730 secure :-1 s=READY pgs=1649 cs=0 l=1 rev1=1 crypto rx=0x7f2cfc009850 tx=0x7f2cfc0a8db0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:50:56.311 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:56.310+0000 7f2cf5ffb640 1 -- 172.21.15.123:0/4217993708 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2cfc0be070 con 0x7f2d00073590 2024-09-16T15:50:56.312 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:56.310+0000 7f2cf5ffb640 1 -- 172.21.15.123:0/4217993708 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f2cfc0039f0 con 0x7f2d00073590 2024-09-16T15:50:56.312 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:56.310+0000 7f2d07659640 1 -- 172.21.15.123:0/4217993708 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2d000b6270 con 0x7f2d00073590 2024-09-16T15:50:56.312 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:56.310+0000 7f2cf5ffb640 1 -- 172.21.15.123:0/4217993708 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2cfc0b5040 con 0x7f2d00073590 2024-09-16T15:50:56.312 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:56.310+0000 7f2d07659640 1 -- 172.21.15.123:0/4217993708 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2d001f3d10 con 0x7f2d00073590 2024-09-16T15:50:56.313 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:56.310+0000 7f2d07659640 1 -- 172.21.15.123:0/4217993708 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2d00074210 con 0x7f2d00073590 2024-09-16T15:50:56.313 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:56.314+0000 7f2cf5ffb640 1 -- 172.21.15.123:0/4217993708 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 23) ==== 97743+0+0 (secure 0 0 0) 0x7f2cfc0c1070 con 0x7f2d00073590 2024-09-16T15:50:56.313 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:56.314+0000 7f2cf5ffb640 1 --2- 172.21.15.123:0/4217993708 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f2ce4073610 0x7f2ce4075ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:50:56.313 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:56.314+0000 7f2cf5ffb640 1 -- 172.21.15.123:0/4217993708 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(146..146 src has 1..146) ==== 5737+0+0 (secure 0 0 0) 0x7f2cfc131b80 con 0x7f2d00073590 2024-09-16T15:50:56.313 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:56.314+0000 7f2d04bf4640 1 --2- 172.21.15.123:0/4217993708 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f2ce4073610 0x7f2ce4075ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:50:56.313 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:56.314+0000 7f2d04bf4640 1 --2- 172.21.15.123:0/4217993708 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f2ce4073610 0x7f2ce4075ad0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f2d0015ba70 tx=0x7f2cf0005c30 comp rx=0 tx=0).ready entity=mgr.4150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:50:56.316 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:56.318+0000 7f2cf5ffb640 1 -- 172.21.15.123:0/4217993708 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f2cfc018bf0 con 0x7f2d00073590 2024-09-16T15:50:56.499 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:56.498+0000 7f2d07659640 1 -- 172.21.15.123:0/4217993708 --> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] -- mgr_command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "subvol82901", "group_name": "subvol_grp44440", "target": ["mon-mgr", ""]}) -- 0x7f2d000d1850 con 0x7f2ce4073610 2024-09-16T15:50:56.534 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:56.534+0000 7f2cf5ffb640 1 -- 172.21.15.123:0/4217993708 <== mgr.4150 v2:172.21.15.123:6836/2464759676 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f2d000d1850 con 0x7f2ce4073610 2024-09-16T15:50:56.537 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:56.538+0000 7f2d07659640 1 -- 172.21.15.123:0/4217993708 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f2ce4073610 msgr2=0x7f2ce4075ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:50:56.537 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:56.538+0000 7f2d07659640 1 --2- 172.21.15.123:0/4217993708 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f2ce4073610 0x7f2ce4075ad0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f2d0015ba70 tx=0x7f2cf0005c30 comp rx=0 tx=0).stop 2024-09-16T15:50:56.537 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:56.538+0000 7f2d07659640 1 -- 172.21.15.123:0/4217993708 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2d00073590 msgr2=0x7f2d001f5730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:50:56.537 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:56.538+0000 7f2d07659640 1 --2- 172.21.15.123:0/4217993708 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2d00073590 0x7f2d001f5730 secure :-1 s=READY pgs=1649 cs=0 l=1 rev1=1 crypto rx=0x7f2cfc009850 tx=0x7f2cfc0a8db0 comp rx=0 tx=0).stop 2024-09-16T15:50:56.537 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:56.538+0000 7f2d07659640 1 -- 172.21.15.123:0/4217993708 shutdown_connections 2024-09-16T15:50:56.537 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:56.538+0000 7f2d07659640 1 --2- 172.21.15.123:0/4217993708 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f2ce4073610 0x7f2ce4075ad0 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:56.538 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:56.538+0000 7f2d07659640 1 --2- 172.21.15.123:0/4217993708 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f2d000cc150 0x7f2d00155bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:56.538 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:56.538+0000 7f2d07659640 1 --2- 172.21.15.123:0/4217993708 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2d00073970 0x7f2d00155690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:56.538 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:56.538+0000 7f2d07659640 1 --2- 172.21.15.123:0/4217993708 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2d00073590 0x7f2d001f5730 unknown :-1 s=CLOSED pgs=1649 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:50:56.538 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:56.538+0000 7f2d07659640 1 -- 172.21.15.123:0/4217993708 >> 172.21.15.123:0/4217993708 conn(0x7f2d000d0b30 msgr2=0x7f2d000b1670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:50:56.538 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:56.538+0000 7f2d07659640 1 -- 172.21.15.123:0/4217993708 shutdown_connections 2024-09-16T15:50:56.538 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:56.538+0000 7f2d07659640 1 -- 172.21.15.123:0/4217993708 wait complete. 2024-09-16T15:50:56.554 DEBUG:teuthology.orchestra.run.smithi123:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'stat -c %h /home/ubuntu/cephtest/mnt.0/./volumes/_deleting') 2024-09-16T15:50:56.634 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:50:56 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-09-16T15:51:00.793 INFO:teuthology.orchestra.run.smithi123.stdout:2 2024-09-16T15:51:00.793 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:00 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-16T15:51:00.803 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolumegroup rm cephfs subvol_grp44440 2024-09-16T15:51:00.981 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:00.982+0000 7f70d4ce7640 1 -- 172.21.15.123:0/813863123 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f70d015a170 msgr2=0x7f70d0072f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:51:00.981 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:00.982+0000 7f70d4ce7640 1 --2- 172.21.15.123:0/813863123 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f70d015a170 0x7f70d0072f30 secure :-1 s=READY pgs=1650 cs=0 l=1 rev1=1 crypto rx=0x7f70c0009900 tx=0x7f70c001c850 comp rx=0 tx=0).stop 2024-09-16T15:51:00.981 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:00.982+0000 7f70d4ce7640 1 -- 172.21.15.123:0/813863123 shutdown_connections 2024-09-16T15:51:00.981 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:00.982+0000 7f70d4ce7640 1 --2- 172.21.15.123:0/813863123 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f70d015ddc0 0x7f70d015e250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:00.981 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:00.982+0000 7f70d4ce7640 1 --2- 172.21.15.123:0/813863123 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f70d015a170 0x7f70d0072f30 unknown :-1 s=CLOSED pgs=1650 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:00.981 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:00.982+0000 7f70d4ce7640 1 -- 172.21.15.123:0/813863123 >> 172.21.15.123:0/813863123 conn(0x7f70d00b1710 msgr2=0x7f70d00b1b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:51:00.982 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:00.982+0000 7f70d4ce7640 1 -- 172.21.15.123:0/813863123 shutdown_connections 2024-09-16T15:51:00.982 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:00.982+0000 7f70d4ce7640 1 -- 172.21.15.123:0/813863123 wait complete. 2024-09-16T15:51:00.984 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:00.982+0000 7f70d4ce7640 1 Processor -- start 2024-09-16T15:51:00.984 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:00.982+0000 7f70d4ce7640 1 -- start start 2024-09-16T15:51:00.984 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:00.982+0000 7f70d4ce7640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f70d00ca650 0x7f70d01f5cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:51:00.984 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:00.982+0000 7f70d4ce7640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f70d015a170 0x7f70d01f9180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:51:00.985 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:00.982+0000 7f70d4ce7640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f70d015ddc0 0x7f70d0155980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:51:00.985 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:00.982+0000 7f70d4ce7640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f70d01670d0 con 0x7f70d015ddc0 2024-09-16T15:51:00.985 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:00.982+0000 7f70d4ce7640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f70d0166f50 con 0x7f70d015a170 2024-09-16T15:51:00.985 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:00.982+0000 7f70d4ce7640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f70d0167250 con 0x7f70d00ca650 2024-09-16T15:51:00.985 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:00.982+0000 7f70ced9d640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f70d015ddc0 0x7f70d0155980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:51:00.985 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:00.982+0000 7f70ced9d640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f70d015ddc0 0x7f70d0155980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:49544/0 (socket says 172.21.15.123:49544) 2024-09-16T15:51:00.985 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:00.982+0000 7f70ced9d640 1 -- 172.21.15.123:0/3744006810 learned_addr learned my addr 172.21.15.123:0/3744006810 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:51:00.985 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:00.982+0000 7f70ce59c640 1 --2- 172.21.15.123:0/3744006810 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f70d00ca650 0x7f70d01f5cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:51:00.985 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:00.982+0000 7f70cdd9b640 1 --2- 172.21.15.123:0/3744006810 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f70d015a170 0x7f70d01f9180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:51:00.985 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:00.982+0000 7f70ced9d640 1 -- 172.21.15.123:0/3744006810 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f70d00ca650 msgr2=0x7f70d01f5cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:51:00.985 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:00.982+0000 7f70ced9d640 1 --2- 172.21.15.123:0/3744006810 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f70d00ca650 0x7f70d01f5cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:00.986 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:00.982+0000 7f70ced9d640 1 -- 172.21.15.123:0/3744006810 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f70d015a170 msgr2=0x7f70d01f9180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:51:00.986 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:00.982+0000 7f70ced9d640 1 --2- 172.21.15.123:0/3744006810 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f70d015a170 0x7f70d01f9180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:00.986 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:00.982+0000 7f70ced9d640 1 -- 172.21.15.123:0/3744006810 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f70d00c8e80 con 0x7f70d015ddc0 2024-09-16T15:51:00.986 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:00.986+0000 7f70ced9d640 1 --2- 172.21.15.123:0/3744006810 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f70d015ddc0 0x7f70d0155980 secure :-1 s=READY pgs=1651 cs=0 l=1 rev1=1 crypto rx=0x7f70c800ca60 tx=0x7f70c800cf20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:51:00.986 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:00.986+0000 7f70c6ffd640 1 -- 172.21.15.123:0/3744006810 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70c8004d30 con 0x7f70d015ddc0 2024-09-16T15:51:00.986 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:00.986+0000 7f70c6ffd640 1 -- 172.21.15.123:0/3744006810 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f70c8007b30 con 0x7f70d015ddc0 2024-09-16T15:51:00.986 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:00.986+0000 7f70c6ffd640 1 -- 172.21.15.123:0/3744006810 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70c8005020 con 0x7f70d015ddc0 2024-09-16T15:51:00.986 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:00.986+0000 7f70d4ce7640 1 -- 172.21.15.123:0/3744006810 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f70d01f54c0 con 0x7f70d015ddc0 2024-09-16T15:51:00.986 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:00.986+0000 7f70d4ce7640 1 -- 172.21.15.123:0/3744006810 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f70d01f96c0 con 0x7f70d015ddc0 2024-09-16T15:51:00.987 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:00.986+0000 7f70d4ce7640 1 -- 172.21.15.123:0/3744006810 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f70d0072f30 con 0x7f70d015ddc0 2024-09-16T15:51:00.987 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:00.986+0000 7f70c6ffd640 1 -- 172.21.15.123:0/3744006810 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 24) ==== 97845+0+0 (secure 0 0 0) 0x7f70c80051c0 con 0x7f70d015ddc0 2024-09-16T15:51:00.987 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:00.986+0000 7f70c6ffd640 1 --2- 172.21.15.123:0/3744006810 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f70b0073730 0x7f70b0075bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:51:00.987 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:00.986+0000 7f70c6ffd640 1 -- 172.21.15.123:0/3744006810 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(146..146 src has 1..146) ==== 5737+0+0 (secure 0 0 0) 0x7f70c80955e0 con 0x7f70d015ddc0 2024-09-16T15:51:00.987 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:00.986+0000 7f70ce59c640 1 --2- 172.21.15.123:0/3744006810 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f70b0073730 0x7f70b0075bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:51:00.988 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:00.986+0000 7f70ce59c640 1 --2- 172.21.15.123:0/3744006810 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f70b0073730 0x7f70b0075bf0 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f70c00af000 tx=0x7f70c001b090 comp rx=0 tx=0).ready entity=mgr.4150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:51:00.990 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:00.990+0000 7f70c6ffd640 1 -- 172.21.15.123:0/3744006810 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f70c8014030 con 0x7f70d015ddc0 2024-09-16T15:51:01.188 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:01.190+0000 7f70d4ce7640 1 -- 172.21.15.123:0/3744006810 --> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] -- mgr_command(tid 0: {"prefix": "fs subvolumegroup rm", "vol_name": "cephfs", "group_name": "subvol_grp44440", "target": ["mon-mgr", ""]}) -- 0x7f70d00ced40 con 0x7f70b0073730 2024-09-16T15:51:01.198 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:01.198+0000 7f70c6ffd640 1 -- 172.21.15.123:0/3744006810 <== mgr.4150 v2:172.21.15.123:6836/2464759676 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f70d00ced40 con 0x7f70b0073730 2024-09-16T15:51:01.200 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:01.202+0000 7f70d4ce7640 1 -- 172.21.15.123:0/3744006810 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f70b0073730 msgr2=0x7f70b0075bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:51:01.200 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:01.202+0000 7f70d4ce7640 1 --2- 172.21.15.123:0/3744006810 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f70b0073730 0x7f70b0075bf0 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f70c00af000 tx=0x7f70c001b090 comp rx=0 tx=0).stop 2024-09-16T15:51:01.201 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:01.202+0000 7f70d4ce7640 1 -- 172.21.15.123:0/3744006810 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f70d015ddc0 msgr2=0x7f70d0155980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:51:01.201 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:01.202+0000 7f70d4ce7640 1 --2- 172.21.15.123:0/3744006810 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f70d015ddc0 0x7f70d0155980 secure :-1 s=READY pgs=1651 cs=0 l=1 rev1=1 crypto rx=0x7f70c800ca60 tx=0x7f70c800cf20 comp rx=0 tx=0).stop 2024-09-16T15:51:01.201 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:01.202+0000 7f70d4ce7640 1 -- 172.21.15.123:0/3744006810 shutdown_connections 2024-09-16T15:51:01.201 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:01.202+0000 7f70d4ce7640 1 --2- 172.21.15.123:0/3744006810 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f70b0073730 0x7f70b0075bf0 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:01.201 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:01.202+0000 7f70d4ce7640 1 --2- 172.21.15.123:0/3744006810 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f70d015ddc0 0x7f70d0155980 unknown :-1 s=CLOSED pgs=1651 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:01.201 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:01.202+0000 7f70d4ce7640 1 --2- 172.21.15.123:0/3744006810 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f70d015a170 0x7f70d01f9180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:01.201 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:01.202+0000 7f70d4ce7640 1 --2- 172.21.15.123:0/3744006810 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f70d00ca650 0x7f70d01f5cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:01.201 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:01.202+0000 7f70d4ce7640 1 -- 172.21.15.123:0/3744006810 >> 172.21.15.123:0/3744006810 conn(0x7f70d00b1710 msgr2=0x7f70d015a8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:51:01.201 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:01.202+0000 7f70d4ce7640 1 -- 172.21.15.123:0/3744006810 shutdown_connections 2024-09-16T15:51:01.201 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:01.202+0000 7f70d4ce7640 1 -- 172.21.15.123:0/3744006810 wait complete. 2024-09-16T15:51:01.217 INFO:teuthology.nuke:Clearing teuthology firewall rules... 2024-09-16T15:51:01.217 DEBUG:teuthology.orchestra.run.smithi123:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-09-16T15:51:01.309 DEBUG:teuthology.orchestra.run.smithi184:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-09-16T15:51:01.374 INFO:teuthology.nuke:Cleared teuthology firewall rules. 2024-09-16T15:51:01.374 INFO:tasks.cephfs.mount:Terminating background process 2024-09-16T15:51:01.374 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2024-09-16T15:51:01.374 DEBUG:teuthology.orchestra.run.smithi123:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-16T15:51:01.381 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2024-09-16T15:51:01.381 DEBUG:teuthology.orchestra.run.smithi123:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-16T15:51:01.433 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-09-16T15:51:01.611 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:01.610+0000 7fe334b54640 1 -- 172.21.15.123:0/2564255187 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe33015bca0 msgr2=0x7fe3300d0630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:51:01.611 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:01.610+0000 7fe334b54640 1 --2- 172.21.15.123:0/2564255187 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe33015bca0 0x7fe3300d0630 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7fe328009c80 tx=0x7fe32801cce0 comp rx=0 tx=0).stop 2024-09-16T15:51:01.611 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:01.610+0000 7fe334b54640 1 -- 172.21.15.123:0/2564255187 shutdown_connections 2024-09-16T15:51:01.611 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:01.610+0000 7fe334b54640 1 --2- 172.21.15.123:0/2564255187 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fe3300726c0 0x7fe330073ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:01.611 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:01.610+0000 7fe334b54640 1 --2- 172.21.15.123:0/2564255187 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe33015bca0 0x7fe3300d0630 unknown :-1 s=CLOSED pgs=511 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:01.611 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:01.610+0000 7fe334b54640 1 -- 172.21.15.123:0/2564255187 >> 172.21.15.123:0/2564255187 conn(0x7fe3300cfb10 msgr2=0x7fe3300cff20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:51:01.611 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:01.610+0000 7fe334b54640 1 -- 172.21.15.123:0/2564255187 shutdown_connections 2024-09-16T15:51:01.612 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:01.610+0000 7fe334b54640 1 -- 172.21.15.123:0/2564255187 wait complete. 2024-09-16T15:51:01.612 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:01.614+0000 7fe334b54640 1 Processor -- start 2024-09-16T15:51:01.612 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:01.614+0000 7fe334b54640 1 -- start start 2024-09-16T15:51:01.613 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:01.614+0000 7fe334b54640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe3300726c0 0x7fe3301f64d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:51:01.614 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:01.614+0000 7fe334b54640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe330074430 0x7fe3300c5340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:51:01.614 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:01.614+0000 7fe334b54640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fe33015bca0 0x7fe3300c8cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:51:01.614 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:01.614+0000 7fe334b54640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fe330168e60 con 0x7fe330074430 2024-09-16T15:51:01.614 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:01.614+0000 7fe334b54640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fe330168ce0 con 0x7fe3300726c0 2024-09-16T15:51:01.614 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:01.614+0000 7fe334b54640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fe330168fe0 con 0x7fe33015bca0 2024-09-16T15:51:01.614 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:01.614+0000 7fe32dd9b640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe330074430 0x7fe3300c5340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:51:01.614 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:01.614+0000 7fe32dd9b640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe330074430 0x7fe3300c5340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:49558/0 (socket says 172.21.15.123:49558) 2024-09-16T15:51:01.614 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:01.614+0000 7fe32dd9b640 1 -- 172.21.15.123:0/3165783724 learned_addr learned my addr 172.21.15.123:0/3165783724 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:51:01.614 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:01.614+0000 7fe32e59c640 1 --2- 172.21.15.123:0/3165783724 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe3300726c0 0x7fe3301f64d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:51:01.614 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:01.614+0000 7fe32ed9d640 1 --2- 172.21.15.123:0/3165783724 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fe33015bca0 0x7fe3300c8cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:51:01.614 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:01.614+0000 7fe32dd9b640 1 -- 172.21.15.123:0/3165783724 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fe33015bca0 msgr2=0x7fe3300c8cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:51:01.614 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:01.614+0000 7fe32dd9b640 1 --2- 172.21.15.123:0/3165783724 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fe33015bca0 0x7fe3300c8cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:01.615 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:01.614+0000 7fe32dd9b640 1 -- 172.21.15.123:0/3165783724 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe3300726c0 msgr2=0x7fe3301f64d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:51:01.615 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:01.614+0000 7fe32dd9b640 1 --2- 172.21.15.123:0/3165783724 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe3300726c0 0x7fe3301f64d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:01.615 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:01.614+0000 7fe32dd9b640 1 -- 172.21.15.123:0/3165783724 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe3301f9250 con 0x7fe330074430 2024-09-16T15:51:01.615 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:01.614+0000 7fe32dd9b640 1 --2- 172.21.15.123:0/3165783724 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe330074430 0x7fe3300c5340 secure :-1 s=READY pgs=1652 cs=0 l=1 rev1=1 crypto rx=0x7fe31800ea10 tx=0x7fe31800eed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:51:01.615 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:01.614+0000 7fe326ffd640 1 -- 172.21.15.123:0/3165783724 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe31800cde0 con 0x7fe330074430 2024-09-16T15:51:01.615 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:01.614+0000 7fe334b54640 1 -- 172.21.15.123:0/3165783724 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe3301f9530 con 0x7fe330074430 2024-09-16T15:51:01.615 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:01.614+0000 7fe326ffd640 1 -- 172.21.15.123:0/3165783724 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fe318004020 con 0x7fe330074430 2024-09-16T15:51:01.615 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:01.614+0000 7fe326ffd640 1 -- 172.21.15.123:0/3165783724 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe318004470 con 0x7fe330074430 2024-09-16T15:51:01.616 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:01.614+0000 7fe334b54640 1 -- 172.21.15.123:0/3165783724 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe3301f9af0 con 0x7fe330074430 2024-09-16T15:51:01.616 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:01.614+0000 7fe334b54640 1 -- 172.21.15.123:0/3165783724 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe3301f9ed0 con 0x7fe330074430 2024-09-16T15:51:01.616 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:01.618+0000 7fe326ffd640 1 -- 172.21.15.123:0/3165783724 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 24) ==== 97845+0+0 (secure 0 0 0) 0x7fe318003600 con 0x7fe330074430 2024-09-16T15:51:01.616 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:01.618+0000 7fe326ffd640 1 --2- 172.21.15.123:0/3165783724 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fe310073730 0x7fe310075bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:51:01.616 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:01.618+0000 7fe326ffd640 1 -- 172.21.15.123:0/3165783724 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(146..146 src has 1..146) ==== 5737+0+0 (secure 0 0 0) 0x7fe318093110 con 0x7fe330074430 2024-09-16T15:51:01.616 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:01.618+0000 7fe32e59c640 1 --2- 172.21.15.123:0/3165783724 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fe310073730 0x7fe310075bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:51:01.616 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:01.618+0000 7fe32e59c640 1 --2- 172.21.15.123:0/3165783724 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fe310073730 0x7fe310075bf0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7fe3280af000 tx=0x7fe32801b520 comp rx=0 tx=0).ready entity=mgr.4150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:51:01.620 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:01.622+0000 7fe326ffd640 1 -- 172.21.15.123:0/3165783724 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7fe3301f9ed0 con 0x7fe330074430 2024-09-16T15:51:01.805 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:01.806+0000 7fe334b54640 1 -- 172.21.15.123:0/3165783724 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7fe330073ef0 con 0x7fe330074430 2024-09-16T15:51:01.805 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:01.806+0000 7fe326ffd640 1 -- 172.21.15.123:0/3165783724 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v146) ==== 80+0+0 (secure 0 0 0) 0x7fe31805fd90 con 0x7fe330074430 2024-09-16T15:51:01.806 INFO:teuthology.orchestra.run.smithi123.stderr:listed 0 entries 2024-09-16T15:51:01.808 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:01.810+0000 7fe334b54640 1 -- 172.21.15.123:0/3165783724 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fe310073730 msgr2=0x7fe310075bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:51:01.808 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:01.810+0000 7fe334b54640 1 --2- 172.21.15.123:0/3165783724 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fe310073730 0x7fe310075bf0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7fe3280af000 tx=0x7fe32801b520 comp rx=0 tx=0).stop 2024-09-16T15:51:01.808 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:01.810+0000 7fe334b54640 1 -- 172.21.15.123:0/3165783724 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe330074430 msgr2=0x7fe3300c5340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:51:01.808 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:01.810+0000 7fe334b54640 1 --2- 172.21.15.123:0/3165783724 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe330074430 0x7fe3300c5340 secure :-1 s=READY pgs=1652 cs=0 l=1 rev1=1 crypto rx=0x7fe31800ea10 tx=0x7fe31800eed0 comp rx=0 tx=0).stop 2024-09-16T15:51:01.809 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:01.810+0000 7fe334b54640 1 -- 172.21.15.123:0/3165783724 shutdown_connections 2024-09-16T15:51:01.809 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:01.810+0000 7fe334b54640 1 --2- 172.21.15.123:0/3165783724 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fe310073730 0x7fe310075bf0 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:01.809 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:01.810+0000 7fe334b54640 1 --2- 172.21.15.123:0/3165783724 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fe33015bca0 0x7fe3300c8cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:01.809 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:01.810+0000 7fe334b54640 1 --2- 172.21.15.123:0/3165783724 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe330074430 0x7fe3300c5340 unknown :-1 s=CLOSED pgs=1652 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:01.809 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:01.810+0000 7fe334b54640 1 --2- 172.21.15.123:0/3165783724 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe3300726c0 0x7fe3301f64d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:01.809 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:01.810+0000 7fe334b54640 1 -- 172.21.15.123:0/3165783724 >> 172.21.15.123:0/3165783724 conn(0x7fe3300cfb10 msgr2=0x7fe3300ca140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:51:01.809 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:01.810+0000 7fe334b54640 1 -- 172.21.15.123:0/3165783724 shutdown_connections 2024-09-16T15:51:01.809 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:01.810+0000 7fe334b54640 1 -- 172.21.15.123:0/3165783724 wait complete. 2024-09-16T15:51:01.823 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi123.front.sepia.ceph.com... 2024-09-16T15:51:01.823 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:51:01.823 DEBUG:teuthology.orchestra.run.smithi123:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2024-09-16T15:51:01.834 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi123.stderr:ceph-fuse[51760]: fuse finished with error 0 and tester_r 0 2024-09-16T15:51:01.835 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi123.front.sepia.ceph.com 2024-09-16T15:51:01.835 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:51:01.835 DEBUG:teuthology.orchestra.run.smithi123:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-09-16T15:51:01.907 INFO:tasks.cephfs.mount:Terminating background process 2024-09-16T15:51:01.907 DEBUG:teuthology.orchestra.run.smithi184:> set -ex 2024-09-16T15:51:01.907 DEBUG:teuthology.orchestra.run.smithi184:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-16T15:51:01.913 DEBUG:teuthology.orchestra.run.smithi184:> set -ex 2024-09-16T15:51:01.913 DEBUG:teuthology.orchestra.run.smithi184:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-16T15:51:01.961 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-09-16T15:51:02.090 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:02.090+0000 7f1d6e2b9640 1 -- 172.21.15.123:0/584134972 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1d500053d0 msgr2=0x7f1d5000b9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:51:02.090 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:02.090+0000 7f1d6e2b9640 1 --2- 172.21.15.123:0/584134972 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1d500053d0 0x7f1d5000b9d0 secure :-1 s=READY pgs=1654 cs=0 l=1 rev1=1 crypto rx=0x7f1d60001dc0 tx=0x7f1d6001cb30 comp rx=0 tx=0).stop 2024-09-16T15:51:02.090 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:02.090+0000 7f1d6e2b9640 1 -- 172.21.15.123:0/584134972 shutdown_connections 2024-09-16T15:51:02.090 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:02.090+0000 7f1d6e2b9640 1 --2- 172.21.15.123:0/584134972 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1d500053d0 0x7f1d5000b9d0 unknown :-1 s=CLOSED pgs=1654 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:02.090 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:02.090+0000 7f1d6e2b9640 1 -- 172.21.15.123:0/584134972 >> 172.21.15.123:0/584134972 conn(0x7f1d680cfb10 msgr2=0x7f1d680cff20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:51:02.090 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:02.090+0000 7f1d6e2b9640 1 -- 172.21.15.123:0/584134972 shutdown_connections 2024-09-16T15:51:02.091 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:02.090+0000 7f1d6e2b9640 1 -- 172.21.15.123:0/584134972 wait complete. 2024-09-16T15:51:02.091 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:02.090+0000 7f1d6e2b9640 1 Processor -- start 2024-09-16T15:51:02.091 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:02.090+0000 7f1d6e2b9640 1 -- start start 2024-09-16T15:51:02.093 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:02.094+0000 7f1d6e2b9640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1d500053d0 0x7f1d6814ecf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:51:02.093 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:02.094+0000 7f1d6e2b9640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f1d681f66b0 0x7f1d680c4380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:51:02.093 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:02.094+0000 7f1d6e2b9640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1d680b4eb0 0x7f1d680c4b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:51:02.093 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:02.094+0000 7f1d6e2b9640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f1d680ca320 con 0x7f1d680b4eb0 2024-09-16T15:51:02.093 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:02.094+0000 7f1d6e2b9640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f1d6806f1a0 con 0x7f1d500053d0 2024-09-16T15:51:02.093 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:02.094+0000 7f1d6e2b9640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f1d680ca4a0 con 0x7f1d681f66b0 2024-09-16T15:51:02.093 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:02.094+0000 7f1d6c856640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1d680b4eb0 0x7f1d680c4b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:51:02.093 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:02.094+0000 7f1d6c856640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1d680b4eb0 0x7f1d680c4b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:49574/0 (socket says 172.21.15.123:49574) 2024-09-16T15:51:02.094 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:02.094+0000 7f1d6c856640 1 -- 172.21.15.123:0/455773499 learned_addr learned my addr 172.21.15.123:0/455773499 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:51:02.094 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:02.094+0000 7f1d677fe640 1 --2- 172.21.15.123:0/455773499 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f1d681f66b0 0x7f1d680c4380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:51:02.094 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:02.094+0000 7f1d67fff640 1 --2- 172.21.15.123:0/455773499 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1d500053d0 0x7f1d6814ecf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:51:02.094 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:02.094+0000 7f1d6c856640 1 -- 172.21.15.123:0/455773499 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f1d681f66b0 msgr2=0x7f1d680c4380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:51:02.094 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:02.094+0000 7f1d6c856640 1 --2- 172.21.15.123:0/455773499 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f1d681f66b0 0x7f1d680c4380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:02.094 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:02.094+0000 7f1d6c856640 1 -- 172.21.15.123:0/455773499 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1d500053d0 msgr2=0x7f1d6814ecf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:51:02.094 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:02.094+0000 7f1d6c856640 1 --2- 172.21.15.123:0/455773499 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1d500053d0 0x7f1d6814ecf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:02.094 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:02.094+0000 7f1d6c856640 1 -- 172.21.15.123:0/455773499 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1d680b56c0 con 0x7f1d680b4eb0 2024-09-16T15:51:02.094 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:02.094+0000 7f1d6c856640 1 --2- 172.21.15.123:0/455773499 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1d680b4eb0 0x7f1d680c4b50 secure :-1 s=READY pgs=1655 cs=0 l=1 rev1=1 crypto rx=0x7f1d6001d3d0 tx=0x7f1d60006d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:51:02.094 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:02.094+0000 7f1d64ff9640 1 -- 172.21.15.123:0/455773499 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d60003980 con 0x7f1d680b4eb0 2024-09-16T15:51:02.094 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:02.094+0000 7f1d64ff9640 1 -- 172.21.15.123:0/455773499 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f1d60003c10 con 0x7f1d680b4eb0 2024-09-16T15:51:02.094 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:02.094+0000 7f1d6e2b9640 1 -- 172.21.15.123:0/455773499 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1d680c35c0 con 0x7f1d680b4eb0 2024-09-16T15:51:02.094 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:02.094+0000 7f1d64ff9640 1 -- 172.21.15.123:0/455773499 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d600060d0 con 0x7f1d680b4eb0 2024-09-16T15:51:02.094 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:02.094+0000 7f1d6e2b9640 1 -- 172.21.15.123:0/455773499 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1d680c3b80 con 0x7f1d680b4eb0 2024-09-16T15:51:02.095 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:02.094+0000 7f1d6e2b9640 1 -- 172.21.15.123:0/455773499 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1d6815aa60 con 0x7f1d680b4eb0 2024-09-16T15:51:02.095 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:02.094+0000 7f1d64ff9640 1 -- 172.21.15.123:0/455773499 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 24) ==== 97845+0+0 (secure 0 0 0) 0x7f1d60006270 con 0x7f1d680b4eb0 2024-09-16T15:51:02.095 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:02.094+0000 7f1d64ff9640 1 --2- 172.21.15.123:0/455773499 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f1d54073760 0x7f1d54075c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:51:02.095 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:02.094+0000 7f1d67fff640 1 --2- 172.21.15.123:0/455773499 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f1d54073760 0x7f1d54075c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:51:02.095 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:02.094+0000 7f1d64ff9640 1 -- 172.21.15.123:0/455773499 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(146..146 src has 1..146) ==== 5737+0+0 (secure 0 0 0) 0x7f1d6012c9e0 con 0x7f1d680b4eb0 2024-09-16T15:51:02.096 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:02.098+0000 7f1d67fff640 1 --2- 172.21.15.123:0/455773499 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f1d54073760 0x7f1d54075c20 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f1d58001870 tx=0x7f1d58001d20 comp rx=0 tx=0).ready entity=mgr.4150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:51:02.099 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:02.098+0000 7f1d64ff9640 1 -- 172.21.15.123:0/455773499 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f1d600bf030 con 0x7f1d680b4eb0 2024-09-16T15:51:02.269 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:02.270+0000 7f1d6e2b9640 1 -- 172.21.15.123:0/455773499 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f1d680ca0d0 con 0x7f1d680b4eb0 2024-09-16T15:51:02.270 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:02.270+0000 7f1d64ff9640 1 -- 172.21.15.123:0/455773499 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v146) ==== 80+0+0 (secure 0 0 0) 0x7f1d60016890 con 0x7f1d680b4eb0 2024-09-16T15:51:02.270 INFO:teuthology.orchestra.run.smithi123.stderr:listed 0 entries 2024-09-16T15:51:02.272 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:02.274+0000 7f1d6e2b9640 1 -- 172.21.15.123:0/455773499 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f1d54073760 msgr2=0x7f1d54075c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:51:02.272 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:02.274+0000 7f1d6e2b9640 1 --2- 172.21.15.123:0/455773499 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f1d54073760 0x7f1d54075c20 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f1d58001870 tx=0x7f1d58001d20 comp rx=0 tx=0).stop 2024-09-16T15:51:02.272 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:02.274+0000 7f1d6e2b9640 1 -- 172.21.15.123:0/455773499 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1d680b4eb0 msgr2=0x7f1d680c4b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:51:02.273 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:02.274+0000 7f1d6e2b9640 1 --2- 172.21.15.123:0/455773499 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1d680b4eb0 0x7f1d680c4b50 secure :-1 s=READY pgs=1655 cs=0 l=1 rev1=1 crypto rx=0x7f1d6001d3d0 tx=0x7f1d60006d50 comp rx=0 tx=0).stop 2024-09-16T15:51:02.273 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:02.274+0000 7f1d6e2b9640 1 -- 172.21.15.123:0/455773499 shutdown_connections 2024-09-16T15:51:02.273 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:02.274+0000 7f1d6e2b9640 1 --2- 172.21.15.123:0/455773499 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f1d54073760 0x7f1d54075c20 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:02.273 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:02.274+0000 7f1d6e2b9640 1 --2- 172.21.15.123:0/455773499 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1d680b4eb0 0x7f1d680c4b50 unknown :-1 s=CLOSED pgs=1655 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:02.273 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:02.274+0000 7f1d6e2b9640 1 --2- 172.21.15.123:0/455773499 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f1d681f66b0 0x7f1d680c4380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:02.273 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:02.274+0000 7f1d6e2b9640 1 --2- 172.21.15.123:0/455773499 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1d500053d0 0x7f1d6814ecf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:02.273 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:02.274+0000 7f1d6e2b9640 1 -- 172.21.15.123:0/455773499 >> 172.21.15.123:0/455773499 conn(0x7f1d680cfb10 msgr2=0x7f1d680b0cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:51:02.273 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:02.274+0000 7f1d6e2b9640 1 -- 172.21.15.123:0/455773499 shutdown_connections 2024-09-16T15:51:02.273 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:02.274+0000 7f1d6e2b9640 1 -- 172.21.15.123:0/455773499 wait complete. 2024-09-16T15:51:02.288 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi184.front.sepia.ceph.com... 2024-09-16T15:51:02.288 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:51:02.288 DEBUG:teuthology.orchestra.run.smithi184:> sudo fusermount -u /home/ubuntu/cephtest/mnt.1 2024-09-16T15:51:02.302 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi184.stderr:ceph-fuse[36213]: fuse finished with error 0 and tester_r 0 2024-09-16T15:51:02.304 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi184.front.sepia.ceph.com 2024-09-16T15:51:02.304 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:51:02.304 DEBUG:teuthology.orchestra.run.smithi184:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2024-09-16T15:51:02.372 INFO:tasks.cephfs.mount:Terminating background process 2024-09-16T15:51:02.372 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2024-09-16T15:51:02.372 DEBUG:teuthology.orchestra.run.smithi123:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-16T15:51:02.378 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16336468k,nr_inodes=4084117,mode=755,inode64 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3274684k,mode=755,inode64 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev,inode64 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=17082 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3274680k,nr_inodes=818670,mode=700,uid=1000,gid=1271,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3274684k,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-09-16T15:51:02.378 INFO:tasks.cephfs.mount:Terminating background process 2024-09-16T15:51:02.378 DEBUG:teuthology.orchestra.run.smithi184:> set -ex 2024-09-16T15:51:02.378 DEBUG:teuthology.orchestra.run.smithi184:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-16T15:51:02.421 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16336488k,nr_inodes=4084122,mode=755,inode64 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3274688k,mode=755,inode64 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev,inode64 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=17834 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3274684k,nr_inodes=818671,mode=700,uid=1000,gid=1271,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3274688k,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-09-16T15:51:02.421 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-16T15:51:02.549 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:02.550+0000 7f40076ee640 1 -- 172.21.15.123:0/3460066963 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3fec0050f0 msgr2=0x7f3fec005920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:51:02.550 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:02.550+0000 7f40076ee640 1 --2- 172.21.15.123:0/3460066963 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3fec0050f0 0x7f3fec005920 secure :-1 s=READY pgs=1657 cs=0 l=1 rev1=1 crypto rx=0x7f3ff000c2e0 tx=0x7f3ff001cfc0 comp rx=0 tx=0).stop 2024-09-16T15:51:02.550 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:02.550+0000 7f4005c8b640 1 -- 172.21.15.123:0/3460066963 reap_dead start 2024-09-16T15:51:02.550 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:02.550+0000 7f40076ee640 1 -- 172.21.15.123:0/3460066963 shutdown_connections 2024-09-16T15:51:02.550 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:02.550+0000 7f40076ee640 1 -- 172.21.15.123:0/3460066963 >> 172.21.15.123:0/3460066963 conn(0x7f40000cfb10 msgr2=0x7f40000cff20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:51:02.550 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:02.550+0000 7f40076ee640 1 -- 172.21.15.123:0/3460066963 shutdown_connections 2024-09-16T15:51:02.550 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:02.550+0000 7f40076ee640 1 -- 172.21.15.123:0/3460066963 wait complete. 2024-09-16T15:51:02.551 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:02.550+0000 7f40076ee640 1 Processor -- start 2024-09-16T15:51:02.551 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:02.550+0000 7f40076ee640 1 -- start start 2024-09-16T15:51:02.552 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:02.550+0000 7f40076ee640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3fec0050f0 0x7f400015ca70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:51:02.552 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:02.550+0000 7f40076ee640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f40001f76c0 0x7f4000156d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:51:02.552 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:02.550+0000 7f40076ee640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f40001f7b40 0x7f4000157250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:51:02.553 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:02.550+0000 7f40076ee640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f400015b190 con 0x7f3fec0050f0 2024-09-16T15:51:02.553 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:02.554+0000 7f40076ee640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f400015b010 con 0x7f40001f7b40 2024-09-16T15:51:02.553 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:02.554+0000 7f40076ee640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f400015b310 con 0x7f40001f76c0 2024-09-16T15:51:02.553 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:02.554+0000 7f400548a640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3fec0050f0 0x7f400015ca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:51:02.553 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:02.554+0000 7f400548a640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3fec0050f0 0x7f400015ca70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:55386/0 (socket says 172.21.15.123:55386) 2024-09-16T15:51:02.553 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:02.554+0000 7f400548a640 1 -- 172.21.15.123:0/1799487992 learned_addr learned my addr 172.21.15.123:0/1799487992 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:51:02.553 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:02.554+0000 7f4004c89640 1 --2- 172.21.15.123:0/1799487992 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f40001f76c0 0x7f4000156d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:51:02.553 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:02.554+0000 7f4005c8b640 1 --2- 172.21.15.123:0/1799487992 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f40001f7b40 0x7f4000157250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:51:02.553 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:02.554+0000 7f400548a640 1 -- 172.21.15.123:0/1799487992 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f40001f76c0 msgr2=0x7f4000156d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:51:02.553 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:02.554+0000 7f400548a640 1 --2- 172.21.15.123:0/1799487992 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f40001f76c0 0x7f4000156d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:02.553 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:02.554+0000 7f400548a640 1 -- 172.21.15.123:0/1799487992 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f40001f7b40 msgr2=0x7f4000157250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:51:02.553 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:02.554+0000 7f400548a640 1 --2- 172.21.15.123:0/1799487992 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f40001f7b40 0x7f4000157250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:02.554 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:02.554+0000 7f400548a640 1 -- 172.21.15.123:0/1799487992 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f40000c6fc0 con 0x7f3fec0050f0 2024-09-16T15:51:02.554 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:02.554+0000 7f400548a640 1 --2- 172.21.15.123:0/1799487992 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3fec0050f0 0x7f400015ca70 secure :-1 s=READY pgs=1658 cs=0 l=1 rev1=1 crypto rx=0x7f3ffc010990 tx=0x7f3ffc010e50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:51:02.554 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:02.554+0000 7f3ff5ffb640 1 -- 172.21.15.123:0/1799487992 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ffc00ed80 con 0x7f3fec0050f0 2024-09-16T15:51:02.554 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:02.554+0000 7f3ff5ffb640 1 -- 172.21.15.123:0/1799487992 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f3ffc005020 con 0x7f3fec0050f0 2024-09-16T15:51:02.554 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:02.554+0000 7f40076ee640 1 -- 172.21.15.123:0/1799487992 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f40000c72a0 con 0x7f3fec0050f0 2024-09-16T15:51:02.554 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:02.554+0000 7f3ff5ffb640 1 -- 172.21.15.123:0/1799487992 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ffc005450 con 0x7f3fec0050f0 2024-09-16T15:51:02.554 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:02.554+0000 7f40076ee640 1 -- 172.21.15.123:0/1799487992 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f40000c7860 con 0x7f3fec0050f0 2024-09-16T15:51:02.555 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:02.554+0000 7f40076ee640 1 -- 172.21.15.123:0/1799487992 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f40000ca320 con 0x7f3fec0050f0 2024-09-16T15:51:02.555 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:02.554+0000 7f3ff5ffb640 1 -- 172.21.15.123:0/1799487992 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 24) ==== 97845+0+0 (secure 0 0 0) 0x7f3ffc0047b0 con 0x7f3fec0050f0 2024-09-16T15:51:02.555 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:02.554+0000 7f3ff5ffb640 1 --2- 172.21.15.123:0/1799487992 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f3fe4073760 0x7f3fe4075c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:51:02.555 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:02.554+0000 7f3ff5ffb640 1 -- 172.21.15.123:0/1799487992 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(146..146 src has 1..146) ==== 5737+0+0 (secure 0 0 0) 0x7f3ffc093450 con 0x7f3fec0050f0 2024-09-16T15:51:02.555 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:02.554+0000 7f4004c89640 1 --2- 172.21.15.123:0/1799487992 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f3fe4073760 0x7f3fe4075c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:51:02.555 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:02.554+0000 7f4004c89640 1 --2- 172.21.15.123:0/1799487992 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f3fe4073760 0x7f3fe4075c20 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f3ff00ac000 tx=0x7f3ff0019e00 comp rx=0 tx=0).ready entity=mgr.4150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:51:02.558 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:02.558+0000 7f3ff5ffb640 1 -- 172.21.15.123:0/1799487992 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f3ffc060150 con 0x7f3fec0050f0 2024-09-16T15:51:02.765 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:02.766+0000 7f40076ee640 1 -- 172.21.15.123:0/1799487992 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f40000d06c0 con 0x7f3fec0050f0 2024-09-16T15:51:02.766 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:02.766+0000 7f3ff5ffb640 1 -- 172.21.15.123:0/1799487992 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 96 v96) ==== 94+0+5236 (secure 0 0 0) 0x7f3ffc05fa50 con 0x7f3fec0050f0 2024-09-16T15:51:02.766 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-09-16T15:51:02.767 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":96,"btime":"2024-09-16T15:50:46:485057+0000","default_fscid":11,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6580,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6835/430907849","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":430907849},{"type":"v1","addr":"172.21.15.184:6835","nonce":430907849}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":91},{"gid":8424,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6833/1494334556","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6832","nonce":1494334556},{"type":"v1","addr":"172.21.15.123:6833","nonce":1494334556}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":90},{"gid":8427,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6835/2168742484","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6834","nonce":2168742484},{"type":"v1","addr":"172.21.15.123:6835","nonce":2168742484}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":91}],"filesystems":[{"mdsmap":{"epoch":95,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-16T15:50:29.771624+0000","modified":"2024-09-16T15:50:32.799612+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8436},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8436":{"gid":8436,"name":"b","rank":0,"incarnation":92,"state":"up:active","state_seq":3,"addr":"172.21.15.184:6833/1547569255","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":1547569255},{"type":"v1","addr":"172.21.15.184:6833","nonce":1547569255}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[23],"metadata_pool":22,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8436,"qdb_cluster":[8436]},"id":11}]} 2024-09-16T15:51:02.767 INFO:teuthology.orchestra.run.smithi123.stderr:dumped fsmap epoch 96 2024-09-16T15:51:02.769 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:02.770+0000 7f40076ee640 1 -- 172.21.15.123:0/1799487992 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f3fe4073760 msgr2=0x7f3fe4075c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:51:02.769 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:02.770+0000 7f40076ee640 1 --2- 172.21.15.123:0/1799487992 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f3fe4073760 0x7f3fe4075c20 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f3ff00ac000 tx=0x7f3ff0019e00 comp rx=0 tx=0).stop 2024-09-16T15:51:02.769 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:02.770+0000 7f40076ee640 1 -- 172.21.15.123:0/1799487992 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3fec0050f0 msgr2=0x7f400015ca70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:51:02.769 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:02.770+0000 7f40076ee640 1 --2- 172.21.15.123:0/1799487992 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3fec0050f0 0x7f400015ca70 secure :-1 s=READY pgs=1658 cs=0 l=1 rev1=1 crypto rx=0x7f3ffc010990 tx=0x7f3ffc010e50 comp rx=0 tx=0).stop 2024-09-16T15:51:02.769 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:02.770+0000 7f40076ee640 1 -- 172.21.15.123:0/1799487992 shutdown_connections 2024-09-16T15:51:02.769 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:02.770+0000 7f40076ee640 1 --2- 172.21.15.123:0/1799487992 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f3fe4073760 0x7f3fe4075c20 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:02.769 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:02.770+0000 7f40076ee640 1 --2- 172.21.15.123:0/1799487992 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f40001f7b40 0x7f4000157250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:02.769 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:02.770+0000 7f40076ee640 1 --2- 172.21.15.123:0/1799487992 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f40001f76c0 0x7f4000156d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:02.769 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:02.770+0000 7f40076ee640 1 --2- 172.21.15.123:0/1799487992 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3fec0050f0 0x7f400015ca70 unknown :-1 s=CLOSED pgs=1658 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:02.769 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:02.770+0000 7f40076ee640 1 -- 172.21.15.123:0/1799487992 >> 172.21.15.123:0/1799487992 conn(0x7f40000cfb10 msgr2=0x7f400015d690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:51:02.769 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:02.770+0000 7f40076ee640 1 -- 172.21.15.123:0/1799487992 shutdown_connections 2024-09-16T15:51:02.770 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:02.770+0000 7f40076ee640 1 -- 172.21.15.123:0/1799487992 wait complete. 2024-09-16T15:51:02.786 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-16T15:51:02.912 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:02.914+0000 7f244b124640 1 -- 172.21.15.123:0/3259838808 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2444074470 msgr2=0x7f24440cd6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:51:02.912 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:02.914+0000 7f244b124640 1 --2- 172.21.15.123:0/3259838808 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2444074470 0x7f24440cd6e0 secure :-1 s=READY pgs=1659 cs=0 l=1 rev1=1 crypto rx=0x7f24440c3570 tx=0x7f243001c570 comp rx=0 tx=0).stop 2024-09-16T15:51:02.912 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:02.914+0000 7f244b124640 1 -- 172.21.15.123:0/3259838808 shutdown_connections 2024-09-16T15:51:02.913 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:02.914+0000 7f244b124640 1 --2- 172.21.15.123:0/3259838808 >> v2:172.21.15.184:3300/0 conn(0x7f24440cdc20 0x7f24440ca860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:02.913 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:02.914+0000 7f244b124640 1 --2- 172.21.15.123:0/3259838808 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2444074470 0x7f24440cd6e0 unknown :-1 s=CLOSED pgs=1659 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:02.913 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:02.914+0000 7f244b124640 1 --2- 172.21.15.123:0/3259838808 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f244415e160 0x7f2444073f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:02.913 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:02.914+0000 7f244b124640 1 -- 172.21.15.123:0/3259838808 >> 172.21.15.123:0/3259838808 conn(0x7f24440cfb10 msgr2=0x7f24440cff20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:51:02.913 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:02.914+0000 7f244b124640 1 -- 172.21.15.123:0/3259838808 shutdown_connections 2024-09-16T15:51:02.913 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:02.914+0000 7f244b124640 1 -- 172.21.15.123:0/3259838808 wait complete. 2024-09-16T15:51:02.915 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:02.914+0000 7f244b124640 1 Processor -- start 2024-09-16T15:51:02.915 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:02.914+0000 7f244b124640 1 -- start start 2024-09-16T15:51:02.915 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:02.914+0000 7f244b124640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2444074470 0x7f2444167380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:51:02.915 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:02.914+0000 7f244b124640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f24440cdc20 0x7f24441f7640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:51:02.915 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:02.914+0000 7f244b124640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f244415e160 0x7f24441596a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:51:02.915 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:02.914+0000 7f244b124640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f244415ebb0 con 0x7f244415e160 2024-09-16T15:51:02.916 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:02.914+0000 7f244b124640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f244415ea30 con 0x7f2444074470 2024-09-16T15:51:02.916 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:02.914+0000 7f244b124640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f244415ed30 con 0x7f24440cdc20 2024-09-16T15:51:02.916 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:02.914+0000 7f24496c1640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f244415e160 0x7f24441596a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:51:02.916 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:02.914+0000 7f24496c1640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f244415e160 0x7f24441596a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:55402/0 (socket says 172.21.15.123:55402) 2024-09-16T15:51:02.916 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:02.914+0000 7f24496c1640 1 -- 172.21.15.123:0/1178618494 learned_addr learned my addr 172.21.15.123:0/1178618494 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:51:02.916 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:02.914+0000 7f2448ec0640 1 --2- 172.21.15.123:0/1178618494 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2444074470 0x7f2444167380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:51:02.916 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:02.914+0000 7f2443fff640 1 --2- 172.21.15.123:0/1178618494 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f24440cdc20 0x7f24441f7640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:51:02.916 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:02.914+0000 7f24496c1640 1 -- 172.21.15.123:0/1178618494 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f24440cdc20 msgr2=0x7f24441f7640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:51:02.916 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:02.914+0000 7f24496c1640 1 --2- 172.21.15.123:0/1178618494 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f24440cdc20 0x7f24441f7640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:02.916 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:02.914+0000 7f24496c1640 1 -- 172.21.15.123:0/1178618494 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2444074470 msgr2=0x7f2444167380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:51:02.916 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:02.914+0000 7f24496c1640 1 --2- 172.21.15.123:0/1178618494 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2444074470 0x7f2444167380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:02.917 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:02.914+0000 7f24496c1640 1 -- 172.21.15.123:0/1178618494 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f24441f8020 con 0x7f244415e160 2024-09-16T15:51:02.917 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:02.914+0000 7f24496c1640 1 --2- 172.21.15.123:0/1178618494 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f244415e160 0x7f24441596a0 secure :-1 s=READY pgs=1660 cs=0 l=1 rev1=1 crypto rx=0x7f243c00cce0 tx=0x7f243c00a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:51:02.917 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:02.914+0000 7f24417fa640 1 -- 172.21.15.123:0/1178618494 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f243c00f450 con 0x7f244415e160 2024-09-16T15:51:02.917 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:02.914+0000 7f24417fa640 1 -- 172.21.15.123:0/1178618494 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f243c00f6e0 con 0x7f244415e160 2024-09-16T15:51:02.917 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:02.914+0000 7f24417fa640 1 -- 172.21.15.123:0/1178618494 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f243c00fb30 con 0x7f244415e160 2024-09-16T15:51:02.917 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:02.918+0000 7f244b124640 1 -- 172.21.15.123:0/1178618494 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f24441f8300 con 0x7f244415e160 2024-09-16T15:51:02.917 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:02.918+0000 7f244b124640 1 -- 172.21.15.123:0/1178618494 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f24441f7b80 con 0x7f244415e160 2024-09-16T15:51:02.918 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:02.918+0000 7f244b124640 1 -- 172.21.15.123:0/1178618494 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f24440ca860 con 0x7f244415e160 2024-09-16T15:51:02.918 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:02.918+0000 7f24417fa640 1 -- 172.21.15.123:0/1178618494 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 24) ==== 97845+0+0 (secure 0 0 0) 0x7f243c00fcd0 con 0x7f244415e160 2024-09-16T15:51:02.918 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:02.918+0000 7f24417fa640 1 --2- 172.21.15.123:0/1178618494 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f242c073730 0x7f242c075bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:51:02.918 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:02.918+0000 7f2448ec0640 1 --2- 172.21.15.123:0/1178618494 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f242c073730 0x7f242c075bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:51:02.918 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:02.918+0000 7f24417fa640 1 -- 172.21.15.123:0/1178618494 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(146..146 src has 1..146) ==== 5737+0+0 (secure 0 0 0) 0x7f243c0932a0 con 0x7f244415e160 2024-09-16T15:51:02.918 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:02.918+0000 7f2448ec0640 1 --2- 172.21.15.123:0/1178618494 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f242c073730 0x7f242c075bf0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f2434005fd0 tx=0x7f243400c460 comp rx=0 tx=0).ready entity=mgr.4150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:51:02.922 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:02.922+0000 7f24417fa640 1 -- 172.21.15.123:0/1178618494 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f243c014030 con 0x7f244415e160 2024-09-16T15:51:03.131 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:03.130+0000 7f244b124640 1 -- 172.21.15.123:0/1178618494 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f24440d06c0 con 0x7f244415e160 2024-09-16T15:51:03.132 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:03.134+0000 7f24417fa640 1 -- 172.21.15.123:0/1178618494 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 96 v96) ==== 94+0+5236 (secure 0 0 0) 0x7f243c05f950 con 0x7f244415e160 2024-09-16T15:51:03.132 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-09-16T15:51:03.132 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":96,"btime":"2024-09-16T15:50:46:485057+0000","default_fscid":11,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6580,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6835/430907849","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":430907849},{"type":"v1","addr":"172.21.15.184:6835","nonce":430907849}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":91},{"gid":8424,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6833/1494334556","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6832","nonce":1494334556},{"type":"v1","addr":"172.21.15.123:6833","nonce":1494334556}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":90},{"gid":8427,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6835/2168742484","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6834","nonce":2168742484},{"type":"v1","addr":"172.21.15.123:6835","nonce":2168742484}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":91}],"filesystems":[{"mdsmap":{"epoch":95,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-16T15:50:29.771624+0000","modified":"2024-09-16T15:50:32.799612+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8436},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8436":{"gid":8436,"name":"b","rank":0,"incarnation":92,"state":"up:active","state_seq":3,"addr":"172.21.15.184:6833/1547569255","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":1547569255},{"type":"v1","addr":"172.21.15.184:6833","nonce":1547569255}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[23],"metadata_pool":22,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8436,"qdb_cluster":[8436]},"id":11}]} 2024-09-16T15:51:03.132 INFO:teuthology.orchestra.run.smithi123.stderr:dumped fsmap epoch 96 2024-09-16T15:51:03.135 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:03.134+0000 7f244b124640 1 -- 172.21.15.123:0/1178618494 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f242c073730 msgr2=0x7f242c075bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:51:03.135 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:03.134+0000 7f244b124640 1 --2- 172.21.15.123:0/1178618494 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f242c073730 0x7f242c075bf0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f2434005fd0 tx=0x7f243400c460 comp rx=0 tx=0).stop 2024-09-16T15:51:03.135 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:03.134+0000 7f244b124640 1 -- 172.21.15.123:0/1178618494 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f244415e160 msgr2=0x7f24441596a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:51:03.135 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:03.134+0000 7f244b124640 1 --2- 172.21.15.123:0/1178618494 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f244415e160 0x7f24441596a0 secure :-1 s=READY pgs=1660 cs=0 l=1 rev1=1 crypto rx=0x7f243c00cce0 tx=0x7f243c00a620 comp rx=0 tx=0).stop 2024-09-16T15:51:03.135 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:03.134+0000 7f244b124640 1 -- 172.21.15.123:0/1178618494 shutdown_connections 2024-09-16T15:51:03.135 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:03.134+0000 7f244b124640 1 --2- 172.21.15.123:0/1178618494 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f242c073730 0x7f242c075bf0 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:03.135 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:03.134+0000 7f244b124640 1 --2- 172.21.15.123:0/1178618494 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f244415e160 0x7f24441596a0 unknown :-1 s=CLOSED pgs=1660 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:03.135 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:03.134+0000 7f244b124640 1 --2- 172.21.15.123:0/1178618494 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f24440cdc20 0x7f24441f7640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:03.136 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:03.134+0000 7f244b124640 1 --2- 172.21.15.123:0/1178618494 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2444074470 0x7f2444167380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:03.136 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:03.134+0000 7f244b124640 1 -- 172.21.15.123:0/1178618494 >> 172.21.15.123:0/1178618494 conn(0x7f24440cfb10 msgr2=0x7f24440b0400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:51:03.136 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:03.134+0000 7f244b124640 1 -- 172.21.15.123:0/1178618494 shutdown_connections 2024-09-16T15:51:03.136 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:03.134+0000 7f244b124640 1 -- 172.21.15.123:0/1178618494 wait complete. 2024-09-16T15:51:03.152 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-16T15:51:03.281 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:03.282+0000 7f44e31c7640 1 -- 172.21.15.123:0/3970557383 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f44dc158120 msgr2=0x7f44dc0ca320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:51:03.281 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:03.282+0000 7f44e31c7640 1 --2- 172.21.15.123:0/3970557383 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f44dc158120 0x7f44dc0ca320 secure :-1 s=READY pgs=1662 cs=0 l=1 rev1=1 crypto rx=0x7f44dc072250 tx=0x7f44cc01da10 comp rx=0 tx=0).stop 2024-09-16T15:51:03.281 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:03.282+0000 7f44e31c7640 1 -- 172.21.15.123:0/3970557383 shutdown_connections 2024-09-16T15:51:03.281 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:03.282+0000 7f44e31c7640 1 --2- 172.21.15.123:0/3970557383 >> v2:172.21.15.184:3300/0 conn(0x7f44dc0cf5a0 0x7f44dc06eee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:03.281 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:03.282+0000 7f44e31c7640 1 --2- 172.21.15.123:0/3970557383 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f44dc158120 0x7f44dc0ca320 unknown :-1 s=CLOSED pgs=1662 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:03.281 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:03.282+0000 7f44e31c7640 1 -- 172.21.15.123:0/3970557383 >> 172.21.15.123:0/3970557383 conn(0x7f44dc0cfb10 msgr2=0x7f44dc0cff20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:51:03.282 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:03.282+0000 7f44e31c7640 1 -- 172.21.15.123:0/3970557383 shutdown_connections 2024-09-16T15:51:03.282 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:03.282+0000 7f44e31c7640 1 -- 172.21.15.123:0/3970557383 wait complete. 2024-09-16T15:51:03.283 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:03.282+0000 7f44e31c7640 1 Processor -- start 2024-09-16T15:51:03.284 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:03.282+0000 7f44e31c7640 1 -- start start 2024-09-16T15:51:03.284 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:03.282+0000 7f44e31c7640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f44dc0cf5a0 0x7f44dc152950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:51:03.284 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:03.282+0000 7f44e31c7640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f44dc158120 0x7f44dc1f8180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:51:03.284 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:03.282+0000 7f44e31c7640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f44dc15ec40 0x7f44dc0c5b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:51:03.284 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:03.282+0000 7f44e31c7640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f44dc15b740 con 0x7f44dc15ec40 2024-09-16T15:51:03.284 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:03.282+0000 7f44e31c7640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f44dc15b5c0 con 0x7f44dc158120 2024-09-16T15:51:03.284 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:03.282+0000 7f44e31c7640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f44dc1673d0 con 0x7f44dc0cf5a0 2024-09-16T15:51:03.284 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:03.282+0000 7f44e1764640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f44dc15ec40 0x7f44dc0c5b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:51:03.284 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:03.282+0000 7f44e1764640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f44dc15ec40 0x7f44dc0c5b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:55414/0 (socket says 172.21.15.123:55414) 2024-09-16T15:51:03.284 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:03.282+0000 7f44e1764640 1 -- 172.21.15.123:0/1402653980 learned_addr learned my addr 172.21.15.123:0/1402653980 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:51:03.284 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:03.282+0000 7f44e0f63640 1 --2- 172.21.15.123:0/1402653980 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f44dc0cf5a0 0x7f44dc152950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:51:03.284 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:03.282+0000 7f44dbfff640 1 --2- 172.21.15.123:0/1402653980 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f44dc158120 0x7f44dc1f8180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:51:03.285 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:03.282+0000 7f44e1764640 1 -- 172.21.15.123:0/1402653980 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f44dc0cf5a0 msgr2=0x7f44dc152950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:51:03.285 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:03.282+0000 7f44e1764640 1 --2- 172.21.15.123:0/1402653980 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f44dc0cf5a0 0x7f44dc152950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:03.285 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:03.282+0000 7f44e1764640 1 -- 172.21.15.123:0/1402653980 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f44dc158120 msgr2=0x7f44dc1f8180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:51:03.285 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:03.282+0000 7f44e1764640 1 --2- 172.21.15.123:0/1402653980 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f44dc158120 0x7f44dc1f8180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:03.285 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:03.282+0000 7f44e1764640 1 -- 172.21.15.123:0/1402653980 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f44dc0c3a00 con 0x7f44dc15ec40 2024-09-16T15:51:03.285 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:03.286+0000 7f44e1764640 1 --2- 172.21.15.123:0/1402653980 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f44dc15ec40 0x7f44dc0c5b30 secure :-1 s=READY pgs=1663 cs=0 l=1 rev1=1 crypto rx=0x7f44d400cab0 tx=0x7f44d400cba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:51:03.285 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:03.286+0000 7f44d97fa640 1 -- 172.21.15.123:0/1402653980 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44d4016020 con 0x7f44dc15ec40 2024-09-16T15:51:03.285 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:03.286+0000 7f44d97fa640 1 -- 172.21.15.123:0/1402653980 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f44d400a930 con 0x7f44dc15ec40 2024-09-16T15:51:03.285 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:03.286+0000 7f44d97fa640 1 -- 172.21.15.123:0/1402653980 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44d400ad80 con 0x7f44dc15ec40 2024-09-16T15:51:03.285 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:03.286+0000 7f44e31c7640 1 -- 172.21.15.123:0/1402653980 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f44dc15aa40 con 0x7f44dc15ec40 2024-09-16T15:51:03.285 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:03.286+0000 7f44e31c7640 1 -- 172.21.15.123:0/1402653980 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f44dc0c40b0 con 0x7f44dc15ec40 2024-09-16T15:51:03.286 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:03.286+0000 7f44e31c7640 1 -- 172.21.15.123:0/1402653980 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f44dc06eee0 con 0x7f44dc15ec40 2024-09-16T15:51:03.286 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:03.286+0000 7f44d97fa640 1 -- 172.21.15.123:0/1402653980 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 24) ==== 97845+0+0 (secure 0 0 0) 0x7f44d4019020 con 0x7f44dc15ec40 2024-09-16T15:51:03.287 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:03.286+0000 7f44d97fa640 1 --2- 172.21.15.123:0/1402653980 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f44c8073730 0x7f44c8075bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:51:03.287 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:03.286+0000 7f44d97fa640 1 -- 172.21.15.123:0/1402653980 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(146..146 src has 1..146) ==== 5737+0+0 (secure 0 0 0) 0x7f44d4092cf0 con 0x7f44dc15ec40 2024-09-16T15:51:03.287 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:03.286+0000 7f44e0f63640 1 --2- 172.21.15.123:0/1402653980 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f44c8073730 0x7f44c8075bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:51:03.287 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:03.286+0000 7f44e0f63640 1 --2- 172.21.15.123:0/1402653980 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f44c8073730 0x7f44c8075bf0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f44cc0b0000 tx=0x7f44cc01c250 comp rx=0 tx=0).ready entity=mgr.4150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:51:03.289 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:03.290+0000 7f44d97fa640 1 -- 172.21.15.123:0/1402653980 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f44d4020020 con 0x7f44dc15ec40 2024-09-16T15:51:03.463 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:03.466+0000 7f44e31c7640 1 -- 172.21.15.123:0/1402653980 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f44dc0c6800 con 0x7f44dc15ec40 2024-09-16T15:51:03.465 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:03.466+0000 7f44d97fa640 1 -- 172.21.15.123:0/1402653980 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v146) ==== 74+0+16835 (secure 0 0 0) 0x7f44d405f9d0 con 0x7f44dc15ec40 2024-09-16T15:51:03.465 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-09-16T15:51:03.465 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":146,"fsid":"56077f89-18c8-486f-ae7e-eecc0bb7822c","created":"2024-09-16T15:42:26.364892+0000","modified":"2024-09-16T15:50:31.413800+0000","last_up_change":"2024-09-16T15:42:38.707234+0000","last_in_change":"2024-09-16T15:42:29.423662+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":23,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-16T15:42:39.287764+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":22,"pool_name":"cephfs_metadata","create_time":"2024-09-16T15:50:27.407822+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"145","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":23,"pool_name":"cephfs_data","create_time":"2024-09-16T15:50:28.729831+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"146","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9e7c1955-a607-4671-9be7-f2ef760ceeaa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":144,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6800","nonce":3882044969},{"type":"v1","addr":"172.21.15.123:6801","nonce":3882044969}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6802","nonce":3882044969},{"type":"v1","addr":"172.21.15.123:6803","nonce":3882044969}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6806","nonce":3882044969},{"type":"v1","addr":"172.21.15.123:6807","nonce":3882044969}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6804","nonce":3882044969},{"type":"v1","addr":"172.21.15.123:6805","nonce":3882044969}]},"public_addr":"172.21.15.123:6801/3882044969","cluster_addr":"172.21.15.123:6803/3882044969","heartbeat_back_addr":"172.21.15.123:6807/3882044969","heartbeat_front_addr":"172.21.15.123:6805/3882044969","state":["exists","up"]},{"osd":1,"uuid":"c540cfde-0977-4742-9b17-fd634668e010","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":144,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6816","nonce":4037001850},{"type":"v1","addr":"172.21.15.123:6817","nonce":4037001850}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6818","nonce":4037001850},{"type":"v1","addr":"172.21.15.123:6819","nonce":4037001850}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6822","nonce":4037001850},{"type":"v1","addr":"172.21.15.123:6823","nonce":4037001850}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6820","nonce":4037001850},{"type":"v1","addr":"172.21.15.123:6821","nonce":4037001850}]},"public_addr":"172.21.15.123:6817/4037001850","cluster_addr":"172.21.15.123:6819/4037001850","heartbeat_back_addr":"172.21.15.123:6823/4037001850","heartbeat_front_addr":"172.21.15.123:6821/4037001850","state":["exists","up"]},{"osd":2,"uuid":"9864a496-f169-486b-95c4-1d59162793a7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":144,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6824","nonce":1332143088},{"type":"v1","addr":"172.21.15.123:6825","nonce":1332143088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6826","nonce":1332143088},{"type":"v1","addr":"172.21.15.123:6827","nonce":1332143088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6830","nonce":1332143088},{"type":"v1","addr":"172.21.15.123:6831","nonce":1332143088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6828","nonce":1332143088},{"type":"v1","addr":"172.21.15.123:6829","nonce":1332143088}]},"public_addr":"172.21.15.123:6825/1332143088","cluster_addr":"172.21.15.123:6827/1332143088","heartbeat_back_addr":"172.21.15.123:6831/1332143088","heartbeat_front_addr":"172.21.15.123:6829/1332143088","state":["exists","up"]},{"osd":3,"uuid":"c6d1f817-f074-4325-95e1-578eba13edda","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":144,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6808","nonce":988586998},{"type":"v1","addr":"172.21.15.123:6809","nonce":988586998}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6810","nonce":988586998},{"type":"v1","addr":"172.21.15.123:6811","nonce":988586998}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6814","nonce":988586998},{"type":"v1","addr":"172.21.15.123:6815","nonce":988586998}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6812","nonce":988586998},{"type":"v1","addr":"172.21.15.123:6813","nonce":988586998}]},"public_addr":"172.21.15.123:6809/988586998","cluster_addr":"172.21.15.123:6811/988586998","heartbeat_back_addr":"172.21.15.123:6815/988586998","heartbeat_front_addr":"172.21.15.123:6813/988586998","state":["exists","up"]},{"osd":4,"uuid":"200a8df4-50ef-40a5-b48a-58981269c086","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":144,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6816","nonce":3387813890},{"type":"v1","addr":"172.21.15.184:6817","nonce":3387813890}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6818","nonce":3387813890},{"type":"v1","addr":"172.21.15.184:6819","nonce":3387813890}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6822","nonce":3387813890},{"type":"v1","addr":"172.21.15.184:6823","nonce":3387813890}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6820","nonce":3387813890},{"type":"v1","addr":"172.21.15.184:6821","nonce":3387813890}]},"public_addr":"172.21.15.184:6817/3387813890","cluster_addr":"172.21.15.184:6819/3387813890","heartbeat_back_addr":"172.21.15.184:6823/3387813890","heartbeat_front_addr":"172.21.15.184:6821/3387813890","state":["exists","up"]},{"osd":5,"uuid":"32d5a900-da3d-412c-bc19-13861a2368d8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":144,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6808","nonce":2680916017},{"type":"v1","addr":"172.21.15.184:6809","nonce":2680916017}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6810","nonce":2680916017},{"type":"v1","addr":"172.21.15.184:6811","nonce":2680916017}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6814","nonce":2680916017},{"type":"v1","addr":"172.21.15.184:6815","nonce":2680916017}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6812","nonce":2680916017},{"type":"v1","addr":"172.21.15.184:6813","nonce":2680916017}]},"public_addr":"172.21.15.184:6809/2680916017","cluster_addr":"172.21.15.184:6811/2680916017","heartbeat_back_addr":"172.21.15.184:6815/2680916017","heartbeat_front_addr":"172.21.15.184:6813/2680916017","state":["exists","up"]},{"osd":6,"uuid":"cde6f39e-aa1b-4260-ac19-774ee6aaca72","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":144,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6824","nonce":2183120457},{"type":"v1","addr":"172.21.15.184:6825","nonce":2183120457}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6826","nonce":2183120457},{"type":"v1","addr":"172.21.15.184:6827","nonce":2183120457}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6830","nonce":2183120457},{"type":"v1","addr":"172.21.15.184:6831","nonce":2183120457}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6828","nonce":2183120457},{"type":"v1","addr":"172.21.15.184:6829","nonce":2183120457}]},"public_addr":"172.21.15.184:6825/2183120457","cluster_addr":"172.21.15.184:6827/2183120457","heartbeat_back_addr":"172.21.15.184:6831/2183120457","heartbeat_front_addr":"172.21.15.184:6829/2183120457","state":["exists","up"]},{"osd":7,"uuid":"4f381778-eee1-47c6-804f-df0e6b33e313","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":144,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6800","nonce":4102957285},{"type":"v1","addr":"172.21.15.184:6801","nonce":4102957285}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6802","nonce":4102957285},{"type":"v1","addr":"172.21.15.184:6803","nonce":4102957285}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6806","nonce":4102957285},{"type":"v1","addr":"172.21.15.184:6807","nonce":4102957285}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6804","nonce":4102957285},{"type":"v1","addr":"172.21.15.184:6805","nonce":4102957285}]},"public_addr":"172.21.15.184:6801/4102957285","cluster_addr":"172.21.15.184:6803/4102957285","heartbeat_back_addr":"172.21.15.184:6807/4102957285","heartbeat_front_addr":"172.21.15.184:6805/4102957285","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:35.972210+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.222007+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.198617+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.385663+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.358935+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.156066+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.752973+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:35.682846+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":17,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T15:51:03.468 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:03.466+0000 7f44e31c7640 1 -- 172.21.15.123:0/1402653980 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f44c8073730 msgr2=0x7f44c8075bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:51:03.468 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:03.466+0000 7f44e31c7640 1 --2- 172.21.15.123:0/1402653980 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f44c8073730 0x7f44c8075bf0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f44cc0b0000 tx=0x7f44cc01c250 comp rx=0 tx=0).stop 2024-09-16T15:51:03.468 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:03.466+0000 7f44e31c7640 1 -- 172.21.15.123:0/1402653980 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f44dc15ec40 msgr2=0x7f44dc0c5b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:51:03.468 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:03.466+0000 7f44e31c7640 1 --2- 172.21.15.123:0/1402653980 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f44dc15ec40 0x7f44dc0c5b30 secure :-1 s=READY pgs=1663 cs=0 l=1 rev1=1 crypto rx=0x7f44d400cab0 tx=0x7f44d400cba0 comp rx=0 tx=0).stop 2024-09-16T15:51:03.468 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:03.470+0000 7f44e31c7640 1 -- 172.21.15.123:0/1402653980 shutdown_connections 2024-09-16T15:51:03.468 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:03.470+0000 7f44e31c7640 1 --2- 172.21.15.123:0/1402653980 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f44c8073730 0x7f44c8075bf0 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:03.468 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:03.470+0000 7f44e31c7640 1 --2- 172.21.15.123:0/1402653980 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f44dc15ec40 0x7f44dc0c5b30 unknown :-1 s=CLOSED pgs=1663 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:03.468 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:03.470+0000 7f44e31c7640 1 --2- 172.21.15.123:0/1402653980 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f44dc158120 0x7f44dc1f8180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:03.468 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:03.470+0000 7f44e31c7640 1 --2- 172.21.15.123:0/1402653980 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f44dc0cf5a0 0x7f44dc152950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:03.468 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:03.470+0000 7f44e31c7640 1 -- 172.21.15.123:0/1402653980 >> 172.21.15.123:0/1402653980 conn(0x7f44dc0cfb10 msgr2=0x7f44dc0cc520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:51:03.468 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:03.470+0000 7f44e31c7640 1 -- 172.21.15.123:0/1402653980 shutdown_connections 2024-09-16T15:51:03.468 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:03.470+0000 7f44e31c7640 1 -- 172.21.15.123:0/1402653980 wait complete. 2024-09-16T15:51:03.485 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-09-16T15:51:03.485 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-16T15:51:03.614 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:03.614+0000 7f827a633640 1 -- 172.21.15.123:0/3332003747 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f82740d2410 msgr2=0x7f82740b4550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:51:03.614 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:03.614+0000 7f827a633640 1 --2- 172.21.15.123:0/3332003747 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f82740d2410 0x7f82740b4550 secure :-1 s=READY pgs=1665 cs=0 l=1 rev1=1 crypto rx=0x7f826c009bf0 tx=0x7f826c01cae0 comp rx=0 tx=0).stop 2024-09-16T15:51:03.614 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:03.614+0000 7f827a633640 1 -- 172.21.15.123:0/3332003747 shutdown_connections 2024-09-16T15:51:03.614 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:03.614+0000 7f827a633640 1 --2- 172.21.15.123:0/3332003747 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f82740d2410 0x7f82740b4550 unknown :-1 s=CLOSED pgs=1665 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:03.614 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:03.614+0000 7f827a633640 1 -- 172.21.15.123:0/3332003747 >> 172.21.15.123:0/3332003747 conn(0x7f82740d3e60 msgr2=0x7f82740d4240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:51:03.614 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:03.614+0000 7f827a633640 1 -- 172.21.15.123:0/3332003747 shutdown_connections 2024-09-16T15:51:03.614 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:03.614+0000 7f827a633640 1 -- 172.21.15.123:0/3332003747 wait complete. 2024-09-16T15:51:03.615 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:03.614+0000 7f827a633640 1 Processor -- start 2024-09-16T15:51:03.615 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:03.614+0000 7f827a633640 1 -- start start 2024-09-16T15:51:03.618 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:03.618+0000 7f827a633640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f82740ce7d0 0x7f82740c82a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:51:03.618 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:03.618+0000 7f827a633640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f82740d2030 0x7f827415d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:51:03.618 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:03.618+0000 7f827a633640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f82740d2410 0x7f8274150060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:51:03.618 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:03.618+0000 7f827a633640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f827416c290 con 0x7f82740ce7d0 2024-09-16T15:51:03.618 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:03.618+0000 7f827a633640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f827416c110 con 0x7f82740d2030 2024-09-16T15:51:03.618 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:03.618+0000 7f827a633640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f827416c410 con 0x7f82740d2410 2024-09-16T15:51:03.618 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:03.618+0000 7f8273fff640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f82740ce7d0 0x7f82740c82a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:51:03.619 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:03.618+0000 7f8273fff640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f82740ce7d0 0x7f82740c82a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:55436/0 (socket says 172.21.15.123:55436) 2024-09-16T15:51:03.619 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:03.618+0000 7f8273fff640 1 -- 172.21.15.123:0/2638223320 learned_addr learned my addr 172.21.15.123:0/2638223320 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:51:03.619 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:03.618+0000 7f826b7fe640 1 --2- 172.21.15.123:0/2638223320 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f82740d2030 0x7f827415d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:51:03.619 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:03.618+0000 7f8278bd0640 1 --2- 172.21.15.123:0/2638223320 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f82740d2410 0x7f8274150060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:51:03.619 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:03.618+0000 7f8273fff640 1 -- 172.21.15.123:0/2638223320 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f82740d2410 msgr2=0x7f8274150060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:51:03.619 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:03.618+0000 7f8273fff640 1 --2- 172.21.15.123:0/2638223320 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f82740d2410 0x7f8274150060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:03.619 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:03.618+0000 7f8273fff640 1 -- 172.21.15.123:0/2638223320 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f82740d2030 msgr2=0x7f827415d190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:51:03.619 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:03.618+0000 7f8273fff640 1 --2- 172.21.15.123:0/2638223320 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f82740d2030 0x7f827415d190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:03.619 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:03.618+0000 7f8273fff640 1 -- 172.21.15.123:0/2638223320 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f827415d6d0 con 0x7f82740ce7d0 2024-09-16T15:51:03.619 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:03.618+0000 7f826b7fe640 1 --2- 172.21.15.123:0/2638223320 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f82740d2030 0x7f827415d190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T15:51:03.619 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:03.618+0000 7f8273fff640 1 --2- 172.21.15.123:0/2638223320 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f82740ce7d0 0x7f82740c82a0 secure :-1 s=READY pgs=1666 cs=0 l=1 rev1=1 crypto rx=0x7f826400c4c0 tx=0x7f826400c980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:51:03.619 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:03.618+0000 7f82717fa640 1 -- 172.21.15.123:0/2638223320 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8264016020 con 0x7f82740ce7d0 2024-09-16T15:51:03.620 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:03.618+0000 7f82717fa640 1 -- 172.21.15.123:0/2638223320 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f8264012940 con 0x7f82740ce7d0 2024-09-16T15:51:03.620 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:03.618+0000 7f827a633640 1 -- 172.21.15.123:0/2638223320 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f827415cd40 con 0x7f82740ce7d0 2024-09-16T15:51:03.620 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:03.618+0000 7f82717fa640 1 -- 172.21.15.123:0/2638223320 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8264012d90 con 0x7f82740ce7d0 2024-09-16T15:51:03.620 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:03.618+0000 7f827a633640 1 -- 172.21.15.123:0/2638223320 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f82740b7bf0 con 0x7f82740ce7d0 2024-09-16T15:51:03.620 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:03.618+0000 7f827a633640 1 -- 172.21.15.123:0/2638223320 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f82740b8030 con 0x7f82740ce7d0 2024-09-16T15:51:03.621 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:03.618+0000 7f82717fa640 1 -- 172.21.15.123:0/2638223320 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 24) ==== 97845+0+0 (secure 0 0 0) 0x7f8264019020 con 0x7f82740ce7d0 2024-09-16T15:51:03.621 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:03.618+0000 7f82717fa640 1 --2- 172.21.15.123:0/2638223320 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f8254073730 0x7f8254075bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:51:03.621 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:03.618+0000 7f82717fa640 1 -- 172.21.15.123:0/2638223320 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(146..146 src has 1..146) ==== 5737+0+0 (secure 0 0 0) 0x7f8264092c10 con 0x7f82740ce7d0 2024-09-16T15:51:03.621 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:03.618+0000 7f826b7fe640 1 --2- 172.21.15.123:0/2638223320 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f8254073730 0x7f8254075bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:51:03.621 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:03.622+0000 7f826b7fe640 1 --2- 172.21.15.123:0/2638223320 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f8254073730 0x7f8254075bf0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f82740caba0 tx=0x7f825c009040 comp rx=0 tx=0).ready entity=mgr.4150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:51:03.622 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:03.622+0000 7f82717fa640 1 -- 172.21.15.123:0/2638223320 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f8264020020 con 0x7f82740ce7d0 2024-09-16T15:51:03.834 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:03.834+0000 7f827a633640 1 -- 172.21.15.123:0/2638223320 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f8274161d40 con 0x7f82740ce7d0 2024-09-16T15:51:03.835 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:03.838+0000 7f82717fa640 1 -- 172.21.15.123:0/2638223320 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 96 v96) ==== 94+0+5236 (secure 0 0 0) 0x7f826405f9b0 con 0x7f82740ce7d0 2024-09-16T15:51:03.836 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-09-16T15:51:03.836 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":96,"btime":"2024-09-16T15:50:46:485057+0000","default_fscid":11,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6580,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6835/430907849","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":430907849},{"type":"v1","addr":"172.21.15.184:6835","nonce":430907849}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":91},{"gid":8424,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6833/1494334556","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6832","nonce":1494334556},{"type":"v1","addr":"172.21.15.123:6833","nonce":1494334556}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":90},{"gid":8427,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6835/2168742484","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6834","nonce":2168742484},{"type":"v1","addr":"172.21.15.123:6835","nonce":2168742484}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":91}],"filesystems":[{"mdsmap":{"epoch":95,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-16T15:50:29.771624+0000","modified":"2024-09-16T15:50:32.799612+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8436},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8436":{"gid":8436,"name":"b","rank":0,"incarnation":92,"state":"up:active","state_seq":3,"addr":"172.21.15.184:6833/1547569255","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":1547569255},{"type":"v1","addr":"172.21.15.184:6833","nonce":1547569255}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[23],"metadata_pool":22,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8436,"qdb_cluster":[8436]},"id":11}]} 2024-09-16T15:51:03.836 INFO:teuthology.orchestra.run.smithi123.stderr:dumped fsmap epoch 96 2024-09-16T15:51:03.838 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:03.838+0000 7f827a633640 1 -- 172.21.15.123:0/2638223320 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f8254073730 msgr2=0x7f8254075bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:51:03.838 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:03.838+0000 7f827a633640 1 --2- 172.21.15.123:0/2638223320 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f8254073730 0x7f8254075bf0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f82740caba0 tx=0x7f825c009040 comp rx=0 tx=0).stop 2024-09-16T15:51:03.838 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:03.838+0000 7f827a633640 1 -- 172.21.15.123:0/2638223320 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f82740ce7d0 msgr2=0x7f82740c82a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:51:03.838 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:03.838+0000 7f827a633640 1 --2- 172.21.15.123:0/2638223320 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f82740ce7d0 0x7f82740c82a0 secure :-1 s=READY pgs=1666 cs=0 l=1 rev1=1 crypto rx=0x7f826400c4c0 tx=0x7f826400c980 comp rx=0 tx=0).stop 2024-09-16T15:51:03.838 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:03.838+0000 7f827a633640 1 -- 172.21.15.123:0/2638223320 shutdown_connections 2024-09-16T15:51:03.838 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:03.838+0000 7f827a633640 1 --2- 172.21.15.123:0/2638223320 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f8254073730 0x7f8254075bf0 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:03.838 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:03.838+0000 7f827a633640 1 --2- 172.21.15.123:0/2638223320 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f82740d2410 0x7f8274150060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:03.839 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:03.838+0000 7f827a633640 1 --2- 172.21.15.123:0/2638223320 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f82740d2030 0x7f827415d190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:03.839 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:03.838+0000 7f827a633640 1 --2- 172.21.15.123:0/2638223320 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f82740ce7d0 0x7f82740c82a0 unknown :-1 s=CLOSED pgs=1666 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:03.839 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:03.838+0000 7f827a633640 1 -- 172.21.15.123:0/2638223320 >> 172.21.15.123:0/2638223320 conn(0x7f82740d3e60 msgr2=0x7f82740b3500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:51:03.839 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:03.838+0000 7f827a633640 1 -- 172.21.15.123:0/2638223320 shutdown_connections 2024-09-16T15:51:03.839 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:03.838+0000 7f827a633640 1 -- 172.21.15.123:0/2638223320 wait complete. 2024-09-16T15:51:03.854 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-16T15:51:03.982 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:03.982+0000 7fdbda6e3640 1 -- 172.21.15.123:0/877505568 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdbd4074600 msgr2=0x7fdbd406e750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:51:03.982 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:03.982+0000 7fdbda6e3640 1 --2- 172.21.15.123:0/877505568 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdbd4074600 0x7fdbd406e750 secure :-1 s=READY pgs=1667 cs=0 l=1 rev1=1 crypto rx=0x7fdbcc009c80 tx=0x7fdbcc01cce0 comp rx=0 tx=0).stop 2024-09-16T15:51:03.983 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:03.982+0000 7fdbda6e3640 1 -- 172.21.15.123:0/877505568 shutdown_connections 2024-09-16T15:51:03.983 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:03.982+0000 7fdbda6e3640 1 --2- 172.21.15.123:0/877505568 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdbd4074600 0x7fdbd406e750 unknown :-1 s=CLOSED pgs=1667 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:03.983 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:03.982+0000 7fdbda6e3640 1 --2- 172.21.15.123:0/877505568 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fdbd415bca0 0x7fdbd40d0630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:03.983 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:03.982+0000 7fdbda6e3640 1 -- 172.21.15.123:0/877505568 >> 172.21.15.123:0/877505568 conn(0x7fdbd40cfb10 msgr2=0x7fdbd40cff20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:51:03.983 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:03.982+0000 7fdbda6e3640 1 -- 172.21.15.123:0/877505568 shutdown_connections 2024-09-16T15:51:03.983 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:03.982+0000 7fdbda6e3640 1 -- 172.21.15.123:0/877505568 wait complete. 2024-09-16T15:51:03.985 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:03.982+0000 7fdbda6e3640 1 Processor -- start 2024-09-16T15:51:03.985 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:03.986+0000 7fdbda6e3640 1 -- start start 2024-09-16T15:51:03.985 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:03.986+0000 7fdbda6e3640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fdbd40726c0 0x7fdbd4166390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:51:03.985 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:03.986+0000 7fdbda6e3640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdbd4074600 0x7fdbd4154ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:51:03.986 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:03.986+0000 7fdbda6e3640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fdbd415bca0 0x7fdbd41fb460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:51:03.986 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:03.986+0000 7fdbda6e3640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fdbd4168bd0 con 0x7fdbd4074600 2024-09-16T15:51:03.986 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:03.986+0000 7fdbda6e3640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fdbd4168a50 con 0x7fdbd415bca0 2024-09-16T15:51:03.986 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:03.986+0000 7fdbda6e3640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fdbd4168d50 con 0x7fdbd40726c0 2024-09-16T15:51:03.986 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:03.986+0000 7fdbd37fe640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdbd4074600 0x7fdbd4154ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:51:03.986 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:03.986+0000 7fdbd37fe640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdbd4074600 0x7fdbd4154ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:55452/0 (socket says 172.21.15.123:55452) 2024-09-16T15:51:03.986 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:03.986+0000 7fdbd37fe640 1 -- 172.21.15.123:0/1842758153 learned_addr learned my addr 172.21.15.123:0/1842758153 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:51:03.986 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:03.986+0000 7fdbd8c80640 1 --2- 172.21.15.123:0/1842758153 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fdbd415bca0 0x7fdbd41fb460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:51:03.986 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:03.986+0000 7fdbd3fff640 1 --2- 172.21.15.123:0/1842758153 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fdbd40726c0 0x7fdbd4166390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:51:03.986 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:03.986+0000 7fdbd37fe640 1 -- 172.21.15.123:0/1842758153 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fdbd40726c0 msgr2=0x7fdbd4166390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:51:03.986 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:03.986+0000 7fdbd37fe640 1 --2- 172.21.15.123:0/1842758153 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fdbd40726c0 0x7fdbd4166390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:03.987 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:03.986+0000 7fdbd37fe640 1 -- 172.21.15.123:0/1842758153 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fdbd415bca0 msgr2=0x7fdbd41fb460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:51:03.987 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:03.986+0000 7fdbd37fe640 1 --2- 172.21.15.123:0/1842758153 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fdbd415bca0 0x7fdbd41fb460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:03.987 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:03.986+0000 7fdbd37fe640 1 -- 172.21.15.123:0/1842758153 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdbd415c3a0 con 0x7fdbd4074600 2024-09-16T15:51:03.987 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:03.986+0000 7fdbd37fe640 1 --2- 172.21.15.123:0/1842758153 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdbd4074600 0x7fdbd4154ba0 secure :-1 s=READY pgs=1668 cs=0 l=1 rev1=1 crypto rx=0x7fdbc400c450 tx=0x7fdbc400c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:51:03.987 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:03.986+0000 7fdbd0ff9640 1 -- 172.21.15.123:0/1842758153 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdbc4012830 con 0x7fdbd4074600 2024-09-16T15:51:03.987 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:03.986+0000 7fdbd0ff9640 1 -- 172.21.15.123:0/1842758153 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fdbc4017070 con 0x7fdbd4074600 2024-09-16T15:51:03.987 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:03.986+0000 7fdbda6e3640 1 -- 172.21.15.123:0/1842758153 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdbd415c680 con 0x7fdbd4074600 2024-09-16T15:51:03.987 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:03.986+0000 7fdbd0ff9640 1 -- 172.21.15.123:0/1842758153 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdbc4012d80 con 0x7fdbd4074600 2024-09-16T15:51:03.987 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:03.986+0000 7fdbda6e3640 1 -- 172.21.15.123:0/1842758153 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdbd415cc40 con 0x7fdbd4074600 2024-09-16T15:51:03.988 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:03.986+0000 7fdbda6e3640 1 -- 172.21.15.123:0/1842758153 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdbd406e750 con 0x7fdbd4074600 2024-09-16T15:51:03.988 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:03.986+0000 7fdbd0ff9640 1 -- 172.21.15.123:0/1842758153 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 24) ==== 97845+0+0 (secure 0 0 0) 0x7fdbc4007020 con 0x7fdbd4074600 2024-09-16T15:51:03.988 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:03.990+0000 7fdbd0ff9640 1 --2- 172.21.15.123:0/1842758153 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fdbc0073760 0x7fdbc0075c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:51:03.988 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:03.990+0000 7fdbd0ff9640 1 -- 172.21.15.123:0/1842758153 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(146..146 src has 1..146) ==== 5737+0+0 (secure 0 0 0) 0x7fdbc4092f00 con 0x7fdbd4074600 2024-09-16T15:51:03.988 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:03.990+0000 7fdbd3fff640 1 --2- 172.21.15.123:0/1842758153 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fdbc0073760 0x7fdbc0075c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:51:03.989 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:03.990+0000 7fdbd3fff640 1 --2- 172.21.15.123:0/1842758153 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fdbc0073760 0x7fdbc0075c20 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7fdbc8005fd0 tx=0x7fdbc800c460 comp rx=0 tx=0).ready entity=mgr.4150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:51:03.991 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:03.990+0000 7fdbd0ff9640 1 -- 172.21.15.123:0/1842758153 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7fdbc405fc00 con 0x7fdbd4074600 2024-09-16T15:51:04.194 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:04.194+0000 7fdbda6e3640 1 -- 172.21.15.123:0/1842758153 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fdbd415f630 con 0x7fdbd4074600 2024-09-16T15:51:04.195 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:04.194+0000 7fdbd0ff9640 1 -- 172.21.15.123:0/1842758153 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 96 v96) ==== 94+0+5236 (secure 0 0 0) 0x7fdbc405f500 con 0x7fdbd4074600 2024-09-16T15:51:04.195 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-09-16T15:51:04.196 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":96,"btime":"2024-09-16T15:50:46:485057+0000","default_fscid":11,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6580,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6835/430907849","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":430907849},{"type":"v1","addr":"172.21.15.184:6835","nonce":430907849}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":91},{"gid":8424,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6833/1494334556","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6832","nonce":1494334556},{"type":"v1","addr":"172.21.15.123:6833","nonce":1494334556}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":90},{"gid":8427,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6835/2168742484","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6834","nonce":2168742484},{"type":"v1","addr":"172.21.15.123:6835","nonce":2168742484}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":91}],"filesystems":[{"mdsmap":{"epoch":95,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-16T15:50:29.771624+0000","modified":"2024-09-16T15:50:32.799612+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8436},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8436":{"gid":8436,"name":"b","rank":0,"incarnation":92,"state":"up:active","state_seq":3,"addr":"172.21.15.184:6833/1547569255","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":1547569255},{"type":"v1","addr":"172.21.15.184:6833","nonce":1547569255}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[23],"metadata_pool":22,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8436,"qdb_cluster":[8436]},"id":11}]} 2024-09-16T15:51:04.196 INFO:teuthology.orchestra.run.smithi123.stderr:dumped fsmap epoch 96 2024-09-16T15:51:04.197 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:04.198+0000 7fdbda6e3640 1 -- 172.21.15.123:0/1842758153 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fdbc0073760 msgr2=0x7fdbc0075c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:51:04.198 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:04.198+0000 7fdbda6e3640 1 --2- 172.21.15.123:0/1842758153 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fdbc0073760 0x7fdbc0075c20 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7fdbc8005fd0 tx=0x7fdbc800c460 comp rx=0 tx=0).stop 2024-09-16T15:51:04.198 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:04.198+0000 7fdbda6e3640 1 -- 172.21.15.123:0/1842758153 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdbd4074600 msgr2=0x7fdbd4154ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:51:04.198 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:04.198+0000 7fdbda6e3640 1 --2- 172.21.15.123:0/1842758153 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdbd4074600 0x7fdbd4154ba0 secure :-1 s=READY pgs=1668 cs=0 l=1 rev1=1 crypto rx=0x7fdbc400c450 tx=0x7fdbc400c910 comp rx=0 tx=0).stop 2024-09-16T15:51:04.198 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:04.198+0000 7fdbda6e3640 1 -- 172.21.15.123:0/1842758153 shutdown_connections 2024-09-16T15:51:04.198 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:04.198+0000 7fdbda6e3640 1 --2- 172.21.15.123:0/1842758153 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fdbc0073760 0x7fdbc0075c20 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:04.198 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:04.198+0000 7fdbda6e3640 1 --2- 172.21.15.123:0/1842758153 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fdbd415bca0 0x7fdbd41fb460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:04.198 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:04.198+0000 7fdbda6e3640 1 --2- 172.21.15.123:0/1842758153 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdbd4074600 0x7fdbd4154ba0 unknown :-1 s=CLOSED pgs=1668 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:04.198 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:04.198+0000 7fdbda6e3640 1 --2- 172.21.15.123:0/1842758153 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fdbd40726c0 0x7fdbd4166390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:04.198 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:04.198+0000 7fdbda6e3640 1 -- 172.21.15.123:0/1842758153 >> 172.21.15.123:0/1842758153 conn(0x7fdbd40cfb10 msgr2=0x7fdbd41580e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:51:04.198 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:04.198+0000 7fdbda6e3640 1 -- 172.21.15.123:0/1842758153 shutdown_connections 2024-09-16T15:51:04.198 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:04.198+0000 7fdbda6e3640 1 -- 172.21.15.123:0/1842758153 wait complete. 2024-09-16T15:51:04.213 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-16T15:51:04.328 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:04.330+0000 7f64759a0640 1 -- 172.21.15.123:0/2678673966 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6470074470 msgr2=0x7f64700cd6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:51:04.328 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:04.330+0000 7f64759a0640 1 --2- 172.21.15.123:0/2678673966 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6470074470 0x7f64700cd6e0 secure :-1 s=READY pgs=1670 cs=0 l=1 rev1=1 crypto rx=0x7f6470070730 tx=0x7f646001ca60 comp rx=0 tx=0).stop 2024-09-16T15:51:04.328 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:04.330+0000 7f64759a0640 1 -- 172.21.15.123:0/2678673966 shutdown_connections 2024-09-16T15:51:04.328 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:04.330+0000 7f64759a0640 1 --2- 172.21.15.123:0/2678673966 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6470074470 0x7f64700cd6e0 unknown :-1 s=CLOSED pgs=1670 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:04.328 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:04.330+0000 7f64759a0640 1 --2- 172.21.15.123:0/2678673966 >> v2:172.21.15.184:3300/0 conn(0x7f647015e160 0x7f6470073f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:04.328 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:04.330+0000 7f64759a0640 1 -- 172.21.15.123:0/2678673966 >> 172.21.15.123:0/2678673966 conn(0x7f64700cfb10 msgr2=0x7f64700cff20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:51:04.329 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:04.330+0000 7f64759a0640 1 -- 172.21.15.123:0/2678673966 shutdown_connections 2024-09-16T15:51:04.329 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:04.330+0000 7f64759a0640 1 -- 172.21.15.123:0/2678673966 wait complete. 2024-09-16T15:51:04.331 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:04.330+0000 7f64759a0640 1 Processor -- start 2024-09-16T15:51:04.331 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:04.330+0000 7f64759a0640 1 -- start start 2024-09-16T15:51:04.331 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:04.330+0000 7f64759a0640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6470074470 0x7f647014dbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:51:04.331 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:04.330+0000 7f64759a0640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f64700cdc20 0x7f6470154230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:51:04.332 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:04.330+0000 7f64759a0640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f647015e160 0x7f64700c6be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:51:04.332 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:04.330+0000 7f64759a0640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f6470167f50 con 0x7f6470074470 2024-09-16T15:51:04.332 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:04.330+0000 7f64759a0640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f6470167dd0 con 0x7f647015e160 2024-09-16T15:51:04.332 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:04.330+0000 7f64759a0640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f64701680d0 con 0x7f64700cdc20 2024-09-16T15:51:04.332 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:04.330+0000 7f646effd640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6470074470 0x7f647014dbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:51:04.332 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:04.330+0000 7f646effd640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6470074470 0x7f647014dbc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:55468/0 (socket says 172.21.15.123:55468) 2024-09-16T15:51:04.332 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:04.330+0000 7f646effd640 1 -- 172.21.15.123:0/193348494 learned_addr learned my addr 172.21.15.123:0/193348494 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:51:04.332 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:04.330+0000 7f646effd640 1 -- 172.21.15.123:0/193348494 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f64700cdc20 msgr2=0x7f6470154230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:51:04.332 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:04.330+0000 7f646f7fe640 1 --2- 172.21.15.123:0/193348494 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f647015e160 0x7f64700c6be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:51:04.332 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:04.330+0000 7f646effd640 1 --2- 172.21.15.123:0/193348494 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f64700cdc20 0x7f6470154230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:04.333 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:04.330+0000 7f646effd640 1 -- 172.21.15.123:0/193348494 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f647015e160 msgr2=0x7f64700c6be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:51:04.333 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:04.330+0000 7f646effd640 1 --2- 172.21.15.123:0/193348494 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f647015e160 0x7f64700c6be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:04.333 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:04.330+0000 7f646effd640 1 -- 172.21.15.123:0/193348494 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f647014e2d0 con 0x7f6470074470 2024-09-16T15:51:04.333 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:04.330+0000 7f646effd640 1 --2- 172.21.15.123:0/193348494 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6470074470 0x7f647014dbc0 secure :-1 s=READY pgs=1671 cs=0 l=1 rev1=1 crypto rx=0x7f645800b410 tx=0x7f645800b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:51:04.333 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:04.330+0000 7f6467fff640 1 -- 172.21.15.123:0/193348494 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6458015020 con 0x7f6470074470 2024-09-16T15:51:04.333 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:04.330+0000 7f6467fff640 1 -- 172.21.15.123:0/193348494 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f6458011920 con 0x7f6470074470 2024-09-16T15:51:04.333 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:04.330+0000 7f64759a0640 1 -- 172.21.15.123:0/193348494 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f64700c7660 con 0x7f6470074470 2024-09-16T15:51:04.333 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:04.330+0000 7f6467fff640 1 -- 172.21.15.123:0/193348494 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6458011d70 con 0x7f6470074470 2024-09-16T15:51:04.333 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:04.330+0000 7f64759a0640 1 -- 172.21.15.123:0/193348494 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f64700c7c20 con 0x7f6470074470 2024-09-16T15:51:04.333 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:04.330+0000 7f64759a0640 1 -- 172.21.15.123:0/193348494 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f64700cd6e0 con 0x7f6470074470 2024-09-16T15:51:04.334 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:04.334+0000 7f6467fff640 1 -- 172.21.15.123:0/193348494 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 24) ==== 97845+0+0 (secure 0 0 0) 0x7f6458018020 con 0x7f6470074470 2024-09-16T15:51:04.334 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:04.334+0000 7f6467fff640 1 --2- 172.21.15.123:0/193348494 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f6450073730 0x7f6450075bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:51:04.334 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:04.334+0000 7f646e7fc640 1 --2- 172.21.15.123:0/193348494 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f6450073730 0x7f6450075bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:51:04.334 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:04.334+0000 7f6467fff640 1 -- 172.21.15.123:0/193348494 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(146..146 src has 1..146) ==== 5737+0+0 (secure 0 0 0) 0x7f6458094d80 con 0x7f6470074470 2024-09-16T15:51:04.334 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:04.334+0000 7f646e7fc640 1 --2- 172.21.15.123:0/193348494 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f6450073730 0x7f6450075bf0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f64600af000 tx=0x7f6460019800 comp rx=0 tx=0).ready entity=mgr.4150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:51:04.336 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:04.338+0000 7f6467fff640 1 -- 172.21.15.123:0/193348494 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f645801f030 con 0x7f6470074470 2024-09-16T15:51:04.502 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:04.502+0000 7f64759a0640 1 -- 172.21.15.123:0/193348494 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f64700d06c0 con 0x7f6470074470 2024-09-16T15:51:04.504 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:04.506+0000 7f6467fff640 1 -- 172.21.15.123:0/193348494 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v146) ==== 74+0+16835 (secure 0 0 0) 0x7f645805ef90 con 0x7f6470074470 2024-09-16T15:51:04.505 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-09-16T15:51:04.505 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":146,"fsid":"56077f89-18c8-486f-ae7e-eecc0bb7822c","created":"2024-09-16T15:42:26.364892+0000","modified":"2024-09-16T15:50:31.413800+0000","last_up_change":"2024-09-16T15:42:38.707234+0000","last_in_change":"2024-09-16T15:42:29.423662+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":23,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-16T15:42:39.287764+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":22,"pool_name":"cephfs_metadata","create_time":"2024-09-16T15:50:27.407822+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"145","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":23,"pool_name":"cephfs_data","create_time":"2024-09-16T15:50:28.729831+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"146","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9e7c1955-a607-4671-9be7-f2ef760ceeaa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":144,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6800","nonce":3882044969},{"type":"v1","addr":"172.21.15.123:6801","nonce":3882044969}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6802","nonce":3882044969},{"type":"v1","addr":"172.21.15.123:6803","nonce":3882044969}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6806","nonce":3882044969},{"type":"v1","addr":"172.21.15.123:6807","nonce":3882044969}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6804","nonce":3882044969},{"type":"v1","addr":"172.21.15.123:6805","nonce":3882044969}]},"public_addr":"172.21.15.123:6801/3882044969","cluster_addr":"172.21.15.123:6803/3882044969","heartbeat_back_addr":"172.21.15.123:6807/3882044969","heartbeat_front_addr":"172.21.15.123:6805/3882044969","state":["exists","up"]},{"osd":1,"uuid":"c540cfde-0977-4742-9b17-fd634668e010","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":144,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6816","nonce":4037001850},{"type":"v1","addr":"172.21.15.123:6817","nonce":4037001850}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6818","nonce":4037001850},{"type":"v1","addr":"172.21.15.123:6819","nonce":4037001850}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6822","nonce":4037001850},{"type":"v1","addr":"172.21.15.123:6823","nonce":4037001850}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6820","nonce":4037001850},{"type":"v1","addr":"172.21.15.123:6821","nonce":4037001850}]},"public_addr":"172.21.15.123:6817/4037001850","cluster_addr":"172.21.15.123:6819/4037001850","heartbeat_back_addr":"172.21.15.123:6823/4037001850","heartbeat_front_addr":"172.21.15.123:6821/4037001850","state":["exists","up"]},{"osd":2,"uuid":"9864a496-f169-486b-95c4-1d59162793a7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":144,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6824","nonce":1332143088},{"type":"v1","addr":"172.21.15.123:6825","nonce":1332143088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6826","nonce":1332143088},{"type":"v1","addr":"172.21.15.123:6827","nonce":1332143088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6830","nonce":1332143088},{"type":"v1","addr":"172.21.15.123:6831","nonce":1332143088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6828","nonce":1332143088},{"type":"v1","addr":"172.21.15.123:6829","nonce":1332143088}]},"public_addr":"172.21.15.123:6825/1332143088","cluster_addr":"172.21.15.123:6827/1332143088","heartbeat_back_addr":"172.21.15.123:6831/1332143088","heartbeat_front_addr":"172.21.15.123:6829/1332143088","state":["exists","up"]},{"osd":3,"uuid":"c6d1f817-f074-4325-95e1-578eba13edda","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":144,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6808","nonce":988586998},{"type":"v1","addr":"172.21.15.123:6809","nonce":988586998}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6810","nonce":988586998},{"type":"v1","addr":"172.21.15.123:6811","nonce":988586998}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6814","nonce":988586998},{"type":"v1","addr":"172.21.15.123:6815","nonce":988586998}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6812","nonce":988586998},{"type":"v1","addr":"172.21.15.123:6813","nonce":988586998}]},"public_addr":"172.21.15.123:6809/988586998","cluster_addr":"172.21.15.123:6811/988586998","heartbeat_back_addr":"172.21.15.123:6815/988586998","heartbeat_front_addr":"172.21.15.123:6813/988586998","state":["exists","up"]},{"osd":4,"uuid":"200a8df4-50ef-40a5-b48a-58981269c086","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":144,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6816","nonce":3387813890},{"type":"v1","addr":"172.21.15.184:6817","nonce":3387813890}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6818","nonce":3387813890},{"type":"v1","addr":"172.21.15.184:6819","nonce":3387813890}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6822","nonce":3387813890},{"type":"v1","addr":"172.21.15.184:6823","nonce":3387813890}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6820","nonce":3387813890},{"type":"v1","addr":"172.21.15.184:6821","nonce":3387813890}]},"public_addr":"172.21.15.184:6817/3387813890","cluster_addr":"172.21.15.184:6819/3387813890","heartbeat_back_addr":"172.21.15.184:6823/3387813890","heartbeat_front_addr":"172.21.15.184:6821/3387813890","state":["exists","up"]},{"osd":5,"uuid":"32d5a900-da3d-412c-bc19-13861a2368d8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":144,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6808","nonce":2680916017},{"type":"v1","addr":"172.21.15.184:6809","nonce":2680916017}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6810","nonce":2680916017},{"type":"v1","addr":"172.21.15.184:6811","nonce":2680916017}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6814","nonce":2680916017},{"type":"v1","addr":"172.21.15.184:6815","nonce":2680916017}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6812","nonce":2680916017},{"type":"v1","addr":"172.21.15.184:6813","nonce":2680916017}]},"public_addr":"172.21.15.184:6809/2680916017","cluster_addr":"172.21.15.184:6811/2680916017","heartbeat_back_addr":"172.21.15.184:6815/2680916017","heartbeat_front_addr":"172.21.15.184:6813/2680916017","state":["exists","up"]},{"osd":6,"uuid":"cde6f39e-aa1b-4260-ac19-774ee6aaca72","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":144,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6824","nonce":2183120457},{"type":"v1","addr":"172.21.15.184:6825","nonce":2183120457}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6826","nonce":2183120457},{"type":"v1","addr":"172.21.15.184:6827","nonce":2183120457}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6830","nonce":2183120457},{"type":"v1","addr":"172.21.15.184:6831","nonce":2183120457}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6828","nonce":2183120457},{"type":"v1","addr":"172.21.15.184:6829","nonce":2183120457}]},"public_addr":"172.21.15.184:6825/2183120457","cluster_addr":"172.21.15.184:6827/2183120457","heartbeat_back_addr":"172.21.15.184:6831/2183120457","heartbeat_front_addr":"172.21.15.184:6829/2183120457","state":["exists","up"]},{"osd":7,"uuid":"4f381778-eee1-47c6-804f-df0e6b33e313","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":144,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6800","nonce":4102957285},{"type":"v1","addr":"172.21.15.184:6801","nonce":4102957285}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6802","nonce":4102957285},{"type":"v1","addr":"172.21.15.184:6803","nonce":4102957285}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6806","nonce":4102957285},{"type":"v1","addr":"172.21.15.184:6807","nonce":4102957285}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6804","nonce":4102957285},{"type":"v1","addr":"172.21.15.184:6805","nonce":4102957285}]},"public_addr":"172.21.15.184:6801/4102957285","cluster_addr":"172.21.15.184:6803/4102957285","heartbeat_back_addr":"172.21.15.184:6807/4102957285","heartbeat_front_addr":"172.21.15.184:6805/4102957285","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:35.972210+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.222007+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.198617+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.385663+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.358935+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.156066+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.752973+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:35.682846+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":17,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T15:51:04.507 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:04.506+0000 7f64759a0640 1 -- 172.21.15.123:0/193348494 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f6450073730 msgr2=0x7f6450075bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:51:04.507 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:04.506+0000 7f64759a0640 1 --2- 172.21.15.123:0/193348494 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f6450073730 0x7f6450075bf0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f64600af000 tx=0x7f6460019800 comp rx=0 tx=0).stop 2024-09-16T15:51:04.507 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:04.506+0000 7f64759a0640 1 -- 172.21.15.123:0/193348494 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6470074470 msgr2=0x7f647014dbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:51:04.507 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:04.506+0000 7f64759a0640 1 --2- 172.21.15.123:0/193348494 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6470074470 0x7f647014dbc0 secure :-1 s=READY pgs=1671 cs=0 l=1 rev1=1 crypto rx=0x7f645800b410 tx=0x7f645800b8d0 comp rx=0 tx=0).stop 2024-09-16T15:51:04.507 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:04.506+0000 7f64759a0640 1 -- 172.21.15.123:0/193348494 shutdown_connections 2024-09-16T15:51:04.507 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:04.506+0000 7f64759a0640 1 --2- 172.21.15.123:0/193348494 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f6450073730 0x7f6450075bf0 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:04.507 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:04.506+0000 7f64759a0640 1 --2- 172.21.15.123:0/193348494 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f647015e160 0x7f64700c6be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:04.508 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:04.506+0000 7f64759a0640 1 --2- 172.21.15.123:0/193348494 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f64700cdc20 0x7f6470154230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:04.508 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:04.506+0000 7f64759a0640 1 --2- 172.21.15.123:0/193348494 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6470074470 0x7f647014dbc0 unknown :-1 s=CLOSED pgs=1671 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:04.508 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:04.506+0000 7f64759a0640 1 -- 172.21.15.123:0/193348494 >> 172.21.15.123:0/193348494 conn(0x7f64700cfb10 msgr2=0x7f64700ce170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:51:04.508 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:04.506+0000 7f64759a0640 1 -- 172.21.15.123:0/193348494 shutdown_connections 2024-09-16T15:51:04.508 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:04.506+0000 7f64759a0640 1 -- 172.21.15.123:0/193348494 wait complete. 2024-09-16T15:51:04.523 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-09-16T15:51:04.644 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:04.642+0000 7f1c54afe640 1 -- 172.21.15.123:0/3518013587 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1c5015bda0 msgr2=0x7f1c500d0310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:51:04.644 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:04.642+0000 7f1c54afe640 1 --2- 172.21.15.123:0/3518013587 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1c5015bda0 0x7f1c500d0310 secure :-1 s=READY pgs=1673 cs=0 l=1 rev1=1 crypto rx=0x7f1c40009bc0 tx=0x7f1c4001cb10 comp rx=0 tx=0).stop 2024-09-16T15:51:04.644 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:04.646+0000 7f1c54afe640 1 -- 172.21.15.123:0/3518013587 shutdown_connections 2024-09-16T15:51:04.644 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:04.646+0000 7f1c54afe640 1 --2- 172.21.15.123:0/3518013587 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f1c500b25a0 0x7f1c500d1c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:04.644 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:04.646+0000 7f1c54afe640 1 --2- 172.21.15.123:0/3518013587 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1c5015bda0 0x7f1c500d0310 unknown :-1 s=CLOSED pgs=1673 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:04.644 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:04.646+0000 7f1c54afe640 1 -- 172.21.15.123:0/3518013587 >> 172.21.15.123:0/3518013587 conn(0x7f1c50074bd0 msgr2=0x7f1c5009aa40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:51:04.644 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:04.646+0000 7f1c54afe640 1 -- 172.21.15.123:0/3518013587 shutdown_connections 2024-09-16T15:51:04.644 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:04.646+0000 7f1c54afe640 1 -- 172.21.15.123:0/3518013587 wait complete. 2024-09-16T15:51:04.645 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:04.646+0000 7f1c54afe640 1 Processor -- start 2024-09-16T15:51:04.645 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:04.646+0000 7f1c54afe640 1 -- start start 2024-09-16T15:51:04.647 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:04.646+0000 7f1c54afe640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f1c500b25a0 0x7f1c5014b300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:51:04.647 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:04.646+0000 7f1c54afe640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1c500d0850 0x7f1c500b5950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:51:04.647 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:04.646+0000 7f1c54afe640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1c5015bda0 0x7f1c5014df60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:51:04.647 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:04.646+0000 7f1c54afe640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f1c501692f0 con 0x7f1c5015bda0 2024-09-16T15:51:04.647 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:04.646+0000 7f1c54afe640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f1c50169170 con 0x7f1c500d0850 2024-09-16T15:51:04.647 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:04.646+0000 7f1c54afe640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f1c50169470 con 0x7f1c500b25a0 2024-09-16T15:51:04.648 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:04.646+0000 7f1c4ed9d640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1c5015bda0 0x7f1c5014df60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:51:04.648 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:04.646+0000 7f1c4ed9d640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1c5015bda0 0x7f1c5014df60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:55488/0 (socket says 172.21.15.123:55488) 2024-09-16T15:51:04.648 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:04.646+0000 7f1c4ed9d640 1 -- 172.21.15.123:0/2868149830 learned_addr learned my addr 172.21.15.123:0/2868149830 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:51:04.648 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:04.646+0000 7f1c4ed9d640 1 -- 172.21.15.123:0/2868149830 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f1c500b25a0 msgr2=0x7f1c5014b300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:51:04.648 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:04.646+0000 7f1c4ed9d640 1 --2- 172.21.15.123:0/2868149830 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f1c500b25a0 0x7f1c5014b300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:04.648 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:04.646+0000 7f1c4ed9d640 1 -- 172.21.15.123:0/2868149830 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1c500d0850 msgr2=0x7f1c500b5950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:51:04.648 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:04.646+0000 7f1c4ed9d640 1 --2- 172.21.15.123:0/2868149830 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1c500d0850 0x7f1c500b5950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:04.648 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:04.646+0000 7f1c4ed9d640 1 -- 172.21.15.123:0/2868149830 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1c500b6350 con 0x7f1c5015bda0 2024-09-16T15:51:04.648 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:04.646+0000 7f1c4ed9d640 1 --2- 172.21.15.123:0/2868149830 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1c5015bda0 0x7f1c5014df60 secure :-1 s=READY pgs=1674 cs=0 l=1 rev1=1 crypto rx=0x7f1c4800c420 tx=0x7f1c4800c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:51:04.648 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:04.646+0000 7f1c3effd640 1 -- 172.21.15.123:0/2868149830 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c48017070 con 0x7f1c5015bda0 2024-09-16T15:51:04.648 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:04.646+0000 7f1c3effd640 1 -- 172.21.15.123:0/2868149830 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f1c48003600 con 0x7f1c5015bda0 2024-09-16T15:51:04.649 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:04.646+0000 7f1c54afe640 1 -- 172.21.15.123:0/2868149830 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1c500b6630 con 0x7f1c5015bda0 2024-09-16T15:51:04.649 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:04.646+0000 7f1c3effd640 1 -- 172.21.15.123:0/2868149830 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c48003a50 con 0x7f1c5015bda0 2024-09-16T15:51:04.649 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:04.646+0000 7f1c54afe640 1 -- 172.21.15.123:0/2868149830 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1c500b6bf0 con 0x7f1c5015bda0 2024-09-16T15:51:04.649 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:04.650+0000 7f1c54afe640 1 -- 172.21.15.123:0/2868149830 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1c50070860 con 0x7f1c5015bda0 2024-09-16T15:51:04.650 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:04.650+0000 7f1c3effd640 1 -- 172.21.15.123:0/2868149830 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 24) ==== 97845+0+0 (secure 0 0 0) 0x7f1c4801a070 con 0x7f1c5015bda0 2024-09-16T15:51:04.650 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:04.650+0000 7f1c3effd640 1 --2- 172.21.15.123:0/2868149830 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f1c30073730 0x7f1c30075bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:51:04.650 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:04.650+0000 7f1c4e59c640 1 --2- 172.21.15.123:0/2868149830 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f1c30073730 0x7f1c30075bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:51:04.650 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:04.650+0000 7f1c3effd640 1 -- 172.21.15.123:0/2868149830 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(146..146 src has 1..146) ==== 5737+0+0 (secure 0 0 0) 0x7f1c48091d80 con 0x7f1c5015bda0 2024-09-16T15:51:04.650 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:04.650+0000 7f1c4e59c640 1 --2- 172.21.15.123:0/2868149830 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f1c30073730 0x7f1c30075bf0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f1c400af000 tx=0x7f1c400198b0 comp rx=0 tx=0).ready entity=mgr.4150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:51:04.652 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:04.654+0000 7f1c3effd640 1 -- 172.21.15.123:0/2868149830 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f1c48021020 con 0x7f1c5015bda0 2024-09-16T15:51:04.843 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:04.846+0000 7f1c54afe640 1 -- 172.21.15.123:0/2868149830 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7f1c50069d70 con 0x7f1c5015bda0 2024-09-16T15:51:04.844 INFO:tasks.ceph.mon.a.smithi123.stderr:2024-09-16T15:51:04.846+0000 7f268d076640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-09-16T15:51:04.847 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:04.850+0000 7f1c3effd640 1 -- 172.21.15.123:0/2868149830 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v97) ==== 161+0+0 (secure 0 0 0) 0x7f1c4805ea60 con 0x7f1c5015bda0 2024-09-16T15:51:04.847 INFO:teuthology.orchestra.run.smithi123.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-09-16T15:51:04.850 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:04.850+0000 7f1c54afe640 1 -- 172.21.15.123:0/2868149830 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f1c30073730 msgr2=0x7f1c30075bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:51:04.850 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:04.850+0000 7f1c54afe640 1 --2- 172.21.15.123:0/2868149830 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f1c30073730 0x7f1c30075bf0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f1c400af000 tx=0x7f1c400198b0 comp rx=0 tx=0).stop 2024-09-16T15:51:04.850 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:04.850+0000 7f1c54afe640 1 -- 172.21.15.123:0/2868149830 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1c5015bda0 msgr2=0x7f1c5014df60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:51:04.850 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:04.850+0000 7f1c54afe640 1 --2- 172.21.15.123:0/2868149830 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1c5015bda0 0x7f1c5014df60 secure :-1 s=READY pgs=1674 cs=0 l=1 rev1=1 crypto rx=0x7f1c4800c420 tx=0x7f1c4800c8e0 comp rx=0 tx=0).stop 2024-09-16T15:51:04.851 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:04.850+0000 7f1c54afe640 1 -- 172.21.15.123:0/2868149830 shutdown_connections 2024-09-16T15:51:04.851 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:04.850+0000 7f1c54afe640 1 --2- 172.21.15.123:0/2868149830 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f1c30073730 0x7f1c30075bf0 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:04.851 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:04.850+0000 7f1c54afe640 1 --2- 172.21.15.123:0/2868149830 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1c5015bda0 0x7f1c5014df60 unknown :-1 s=CLOSED pgs=1674 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:04.851 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:04.850+0000 7f1c54afe640 1 --2- 172.21.15.123:0/2868149830 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1c500d0850 0x7f1c500b5950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:04.851 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:04.850+0000 7f1c54afe640 1 --2- 172.21.15.123:0/2868149830 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f1c500b25a0 0x7f1c5014b300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:04.852 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:04.850+0000 7f1c54afe640 1 -- 172.21.15.123:0/2868149830 >> 172.21.15.123:0/2868149830 conn(0x7f1c50074bd0 msgr2=0x7f1c500ae8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:51:04.852 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:04.850+0000 7f1c54afe640 1 -- 172.21.15.123:0/2868149830 shutdown_connections 2024-09-16T15:51:04.852 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:04.850+0000 7f1c54afe640 1 -- 172.21.15.123:0/2868149830 wait complete. 2024-09-16T15:51:04.866 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-09-16T15:51:04.941 INFO:tasks.ceph.mds.b.smithi184.stdout:starting mds.b at 2024-09-16T15:51:04.987 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:04.986+0000 7fcd5b1fd640 1 -- 172.21.15.123:0/52420748 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcd3c003fe0 msgr2=0x7fcd3c004810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:51:04.987 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:04.986+0000 7fcd5b1fd640 1 --2- 172.21.15.123:0/52420748 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcd3c003fe0 0x7fcd3c004810 secure :-1 s=READY pgs=1677 cs=0 l=1 rev1=1 crypto rx=0x7fcd48009b40 tx=0x7fcd4801d250 comp rx=0 tx=0).stop 2024-09-16T15:51:04.987 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:04.986+0000 7fcd5979a640 1 -- 172.21.15.123:0/52420748 reap_dead start 2024-09-16T15:51:04.987 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:04.986+0000 7fcd5b1fd640 1 -- 172.21.15.123:0/52420748 shutdown_connections 2024-09-16T15:51:04.987 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:04.986+0000 7fcd5b1fd640 1 -- 172.21.15.123:0/52420748 >> 172.21.15.123:0/52420748 conn(0x7fcd540d1bc0 msgr2=0x7fcd540d1fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:51:04.987 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:04.986+0000 7fcd5b1fd640 1 -- 172.21.15.123:0/52420748 shutdown_connections 2024-09-16T15:51:04.988 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:04.986+0000 7fcd5b1fd640 1 -- 172.21.15.123:0/52420748 wait complete. 2024-09-16T15:51:04.988 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:04.990+0000 7fcd5b1fd640 1 Processor -- start 2024-09-16T15:51:04.988 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:04.990+0000 7fcd5b1fd640 1 -- start start 2024-09-16T15:51:04.990 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:04.990+0000 7fcd5b1fd640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fcd3c003fe0 0x7fcd540c96a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:51:04.990 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:04.990+0000 7fcd5b1fd640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fcd5415a390 0x7fcd54159a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:51:04.990 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:04.990+0000 7fcd5b1fd640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcd540cae30 0x7fcd540cb290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:51:04.990 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:04.990+0000 7fcd5b1fd640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fcd54166ff0 con 0x7fcd540cae30 2024-09-16T15:51:04.990 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:04.990+0000 7fcd5b1fd640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fcd54166e70 con 0x7fcd3c003fe0 2024-09-16T15:51:04.990 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:04.990+0000 7fcd5b1fd640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fcd54167170 con 0x7fcd5415a390 2024-09-16T15:51:04.990 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:04.990+0000 7fcd5979a640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcd540cae30 0x7fcd540cb290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:51:04.990 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:04.990+0000 7fcd5979a640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcd540cae30 0x7fcd540cb290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:55494/0 (socket says 172.21.15.123:55494) 2024-09-16T15:51:04.990 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:04.990+0000 7fcd5979a640 1 -- 172.21.15.123:0/819902278 learned_addr learned my addr 172.21.15.123:0/819902278 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:51:04.991 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:04.990+0000 7fcd53fff640 1 --2- 172.21.15.123:0/819902278 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fcd5415a390 0x7fcd54159a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:51:04.991 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:04.990+0000 7fcd58f99640 1 --2- 172.21.15.123:0/819902278 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fcd3c003fe0 0x7fcd540c96a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:51:04.991 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:04.990+0000 7fcd5979a640 1 -- 172.21.15.123:0/819902278 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fcd5415a390 msgr2=0x7fcd54159a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:51:04.991 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:04.990+0000 7fcd5979a640 1 --2- 172.21.15.123:0/819902278 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fcd5415a390 0x7fcd54159a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:04.991 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:04.990+0000 7fcd5979a640 1 -- 172.21.15.123:0/819902278 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fcd3c003fe0 msgr2=0x7fcd540c96a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:51:04.991 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:04.990+0000 7fcd5979a640 1 --2- 172.21.15.123:0/819902278 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fcd3c003fe0 0x7fcd540c96a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:04.991 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:04.990+0000 7fcd5979a640 1 -- 172.21.15.123:0/819902278 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcd540c9f60 con 0x7fcd540cae30 2024-09-16T15:51:04.991 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:04.990+0000 7fcd5979a640 1 --2- 172.21.15.123:0/819902278 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcd540cae30 0x7fcd540cb290 secure :-1 s=READY pgs=1678 cs=0 l=1 rev1=1 crypto rx=0x7fcd540703f0 tx=0x7fcd4c00dd80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:51:04.991 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:04.990+0000 7fcd517fa640 1 -- 172.21.15.123:0/819902278 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd4c003d90 con 0x7fcd540cae30 2024-09-16T15:51:04.991 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:04.990+0000 7fcd517fa640 1 -- 172.21.15.123:0/819902278 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fcd4c00b910 con 0x7fcd540cae30 2024-09-16T15:51:04.991 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:04.990+0000 7fcd517fa640 1 -- 172.21.15.123:0/819902278 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd4c00bd60 con 0x7fcd540cae30 2024-09-16T15:51:04.991 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:04.990+0000 7fcd5b1fd640 1 -- 172.21.15.123:0/819902278 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcd540ca240 con 0x7fcd540cae30 2024-09-16T15:51:04.991 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:04.990+0000 7fcd5b1fd640 1 -- 172.21.15.123:0/819902278 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcd540b4b90 con 0x7fcd540cae30 2024-09-16T15:51:04.992 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:04.990+0000 7fcd5b1fd640 1 -- 172.21.15.123:0/819902278 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcd540ce470 con 0x7fcd540cae30 2024-09-16T15:51:04.992 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:04.994+0000 7fcd517fa640 1 -- 172.21.15.123:0/819902278 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 24) ==== 97845+0+0 (secure 0 0 0) 0x7fcd4c006110 con 0x7fcd540cae30 2024-09-16T15:51:04.992 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:04.994+0000 7fcd517fa640 1 --2- 172.21.15.123:0/819902278 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fcd40073730 0x7fcd40075bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:51:04.992 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:04.994+0000 7fcd58f99640 1 --2- 172.21.15.123:0/819902278 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fcd40073730 0x7fcd40075bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:51:04.993 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:04.994+0000 7fcd517fa640 1 -- 172.21.15.123:0/819902278 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(147..147 src has 1..147) ==== 5823+0+0 (secure 0 0 0) 0x7fcd4c092ab0 con 0x7fcd540cae30 2024-09-16T15:51:04.993 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:04.994+0000 7fcd58f99640 1 --2- 172.21.15.123:0/819902278 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fcd40073730 0x7fcd40075bf0 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7fcd44009be0 tx=0x7fcd440091e0 comp rx=0 tx=0).ready entity=mgr.4150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:51:04.996 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:04.998+0000 7fcd517fa640 1 -- 172.21.15.123:0/819902278 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7fcd4c05f160 con 0x7fcd540cae30 2024-09-16T15:51:05.188 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:05.190+0000 7fcd5b1fd640 1 -- 172.21.15.123:0/819902278 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) -- 0x7fcd540cf170 con 0x7fcd540cae30 2024-09-16T15:51:05.850 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:05.850+0000 7fcd517fa640 1 -- 172.21.15.123:0/819902278 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v98) ==== 104+0+0 (secure 0 0 0) 0x7fcd4c063b90 con 0x7fcd540cae30 2024-09-16T15:51:05.853 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:05.854+0000 7fcd5b1fd640 1 -- 172.21.15.123:0/819902278 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fcd40073730 msgr2=0x7fcd40075bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:51:05.853 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:05.854+0000 7fcd5b1fd640 1 --2- 172.21.15.123:0/819902278 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fcd40073730 0x7fcd40075bf0 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7fcd44009be0 tx=0x7fcd440091e0 comp rx=0 tx=0).stop 2024-09-16T15:51:05.853 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:05.854+0000 7fcd5b1fd640 1 -- 172.21.15.123:0/819902278 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcd540cae30 msgr2=0x7fcd540cb290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:51:05.853 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:05.854+0000 7fcd5b1fd640 1 --2- 172.21.15.123:0/819902278 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcd540cae30 0x7fcd540cb290 secure :-1 s=READY pgs=1678 cs=0 l=1 rev1=1 crypto rx=0x7fcd540703f0 tx=0x7fcd4c00dd80 comp rx=0 tx=0).stop 2024-09-16T15:51:05.853 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:05.854+0000 7fcd5b1fd640 1 -- 172.21.15.123:0/819902278 shutdown_connections 2024-09-16T15:51:05.853 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:05.854+0000 7fcd5b1fd640 1 --2- 172.21.15.123:0/819902278 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fcd40073730 0x7fcd40075bf0 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:05.854 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:05.854+0000 7fcd5b1fd640 1 --2- 172.21.15.123:0/819902278 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcd540cae30 0x7fcd540cb290 unknown :-1 s=CLOSED pgs=1678 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:05.854 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:05.854+0000 7fcd5b1fd640 1 --2- 172.21.15.123:0/819902278 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fcd5415a390 0x7fcd54159a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:05.854 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:05.854+0000 7fcd5b1fd640 1 --2- 172.21.15.123:0/819902278 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fcd3c003fe0 0x7fcd540c96a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:05.854 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:05.854+0000 7fcd5b1fd640 1 -- 172.21.15.123:0/819902278 >> 172.21.15.123:0/819902278 conn(0x7fcd540d1bc0 msgr2=0x7fcd54166420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:51:05.854 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:05.854+0000 7fcd5b1fd640 1 -- 172.21.15.123:0/819902278 shutdown_connections 2024-09-16T15:51:05.854 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:05.854+0000 7fcd5b1fd640 1 -- 172.21.15.123:0/819902278 wait complete. 2024-09-16T15:51:05.876 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-09-16T15:51:06.014 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:06.014+0000 7f4ad10f8640 1 -- 172.21.15.123:0/793207099 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4acc0d1b70 msgr2=0x7f4acc0d2020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:51:06.014 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:06.014+0000 7f4ad10f8640 1 --2- 172.21.15.123:0/793207099 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4acc0d1b70 0x7f4acc0d2020 secure :-1 s=READY pgs=1680 cs=0 l=1 rev1=1 crypto rx=0x7f4abc009840 tx=0x7f4abc01c820 comp rx=0 tx=0).stop 2024-09-16T15:51:06.015 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:06.014+0000 7f4ad10f8640 1 -- 172.21.15.123:0/793207099 shutdown_connections 2024-09-16T15:51:06.015 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:06.014+0000 7f4ad10f8640 1 --2- 172.21.15.123:0/793207099 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4acc0d1b70 0x7f4acc0d2020 unknown :-1 s=CLOSED pgs=1680 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:06.015 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:06.014+0000 7f4ad10f8640 1 --2- 172.21.15.123:0/793207099 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f4acc160310 0x7f4acc0d05b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:06.015 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:06.014+0000 7f4ad10f8640 1 -- 172.21.15.123:0/793207099 >> 172.21.15.123:0/793207099 conn(0x7f4acc09aa40 msgr2=0x7f4acc09ae50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:51:06.015 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:06.014+0000 7f4ad10f8640 1 -- 172.21.15.123:0/793207099 shutdown_connections 2024-09-16T15:51:06.015 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:06.014+0000 7f4ad10f8640 1 -- 172.21.15.123:0/793207099 wait complete. 2024-09-16T15:51:06.015 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:06.018+0000 7f4ad10f8640 1 Processor -- start 2024-09-16T15:51:06.016 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:06.018+0000 7f4ad10f8640 1 -- start start 2024-09-16T15:51:06.017 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:06.018+0000 7f4ad10f8640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f4acc0d1b70 0x7f4acc0c6be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:51:06.017 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:06.018+0000 7f4ad10f8640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4acc15bd80 0x7f4acc06f7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:51:06.017 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:06.018+0000 7f4ad10f8640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4acc160310 0x7f4acc158100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:51:06.018 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:06.018+0000 7f4ad10f8640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f4acc168d30 con 0x7f4acc160310 2024-09-16T15:51:06.018 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:06.018+0000 7f4ad10f8640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f4acc168bb0 con 0x7f4acc15bd80 2024-09-16T15:51:06.018 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:06.018+0000 7f4ad10f8640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f4acc168eb0 con 0x7f4acc0d1b70 2024-09-16T15:51:06.018 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:06.018+0000 7f4acb59e640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4acc160310 0x7f4acc158100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:51:06.018 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:06.018+0000 7f4acb59e640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4acc160310 0x7f4acc158100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:55520/0 (socket says 172.21.15.123:55520) 2024-09-16T15:51:06.018 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:06.018+0000 7f4acb59e640 1 -- 172.21.15.123:0/110014305 learned_addr learned my addr 172.21.15.123:0/110014305 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:51:06.018 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:06.018+0000 7f4acad9d640 1 --2- 172.21.15.123:0/110014305 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f4acc0d1b70 0x7f4acc0c6be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:51:06.018 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:06.018+0000 7f4acb59e640 1 -- 172.21.15.123:0/110014305 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f4acc0d1b70 msgr2=0x7f4acc0c6be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:51:06.018 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:06.018+0000 7f4acb59e640 1 --2- 172.21.15.123:0/110014305 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f4acc0d1b70 0x7f4acc0c6be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:06.018 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:06.018+0000 7f4acb59e640 1 -- 172.21.15.123:0/110014305 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4acc15bd80 msgr2=0x7f4acc06f7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:51:06.018 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:06.018+0000 7f4acb59e640 1 --2- 172.21.15.123:0/110014305 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4acc15bd80 0x7f4acc06f7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:06.018 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:06.018+0000 7f4acb59e640 1 -- 172.21.15.123:0/110014305 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4acc06fd10 con 0x7f4acc160310 2024-09-16T15:51:06.019 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:06.018+0000 7f4acad9d640 1 --2- 172.21.15.123:0/110014305 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f4acc0d1b70 0x7f4acc0c6be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T15:51:06.019 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:06.018+0000 7f4acb59e640 1 --2- 172.21.15.123:0/110014305 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4acc160310 0x7f4acc158100 secure :-1 s=READY pgs=1681 cs=0 l=1 rev1=1 crypto rx=0x7f4abc01a450 tx=0x7f4abc0a5790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:51:06.019 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:06.018+0000 7f4ac37fe640 1 -- 172.21.15.123:0/110014305 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4abc0b2040 con 0x7f4acc160310 2024-09-16T15:51:06.019 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:06.018+0000 7f4ac37fe640 1 -- 172.21.15.123:0/110014305 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f4abc004430 con 0x7f4acc160310 2024-09-16T15:51:06.019 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:06.018+0000 7f4ad10f8640 1 -- 172.21.15.123:0/110014305 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4acc164f40 con 0x7f4acc160310 2024-09-16T15:51:06.019 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:06.018+0000 7f4ac37fe640 1 -- 172.21.15.123:0/110014305 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4abc005280 con 0x7f4acc160310 2024-09-16T15:51:06.019 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:06.018+0000 7f4ad10f8640 1 -- 172.21.15.123:0/110014305 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4acc165500 con 0x7f4acc160310 2024-09-16T15:51:06.020 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:06.018+0000 7f4ad10f8640 1 -- 172.21.15.123:0/110014305 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4acc0d2020 con 0x7f4acc160310 2024-09-16T15:51:06.020 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:06.018+0000 7f4ac37fe640 1 -- 172.21.15.123:0/110014305 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 24) ==== 97845+0+0 (secure 0 0 0) 0x7f4abc0be060 con 0x7f4acc160310 2024-09-16T15:51:06.020 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:06.022+0000 7f4ac37fe640 1 --2- 172.21.15.123:0/110014305 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f4aac073730 0x7f4aac075bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:51:06.020 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:06.022+0000 7f4acad9d640 1 --2- 172.21.15.123:0/110014305 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f4aac073730 0x7f4aac075bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:51:06.020 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:06.022+0000 7f4ac37fe640 1 -- 172.21.15.123:0/110014305 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(147..147 src has 1..147) ==== 5823+0+0 (secure 0 0 0) 0x7f4abc12bf80 con 0x7f4acc160310 2024-09-16T15:51:06.020 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:06.022+0000 7f4acad9d640 1 --2- 172.21.15.123:0/110014305 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f4aac073730 0x7f4aac075bf0 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f4ac4001780 tx=0x7f4ac4009040 comp rx=0 tx=0).ready entity=mgr.4150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:51:06.023 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:06.026+0000 7f4ac37fe640 1 -- 172.21.15.123:0/110014305 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f4abc016410 con 0x7f4acc160310 2024-09-16T15:51:06.199 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:06.198+0000 7f4ad10f8640 1 -- 172.21.15.123:0/110014305 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) -- 0x7f4acc070830 con 0x7f4acc160310 2024-09-16T15:51:06.858 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:06.858+0000 7f4ac37fe640 1 -- 172.21.15.123:0/110014305 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v148) ==== 181+0+0 (secure 0 0 0) 0x7f4abc0f8c80 con 0x7f4acc160310 2024-09-16T15:51:06.858 INFO:teuthology.orchestra.run.smithi123.stderr:pool 'cephfs_metadata' removed 2024-09-16T15:51:06.865 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:06.866+0000 7f4ad10f8640 1 -- 172.21.15.123:0/110014305 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f4aac073730 msgr2=0x7f4aac075bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:51:06.865 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:06.866+0000 7f4ad10f8640 1 --2- 172.21.15.123:0/110014305 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f4aac073730 0x7f4aac075bf0 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f4ac4001780 tx=0x7f4ac4009040 comp rx=0 tx=0).stop 2024-09-16T15:51:06.865 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:06.866+0000 7f4ad10f8640 1 -- 172.21.15.123:0/110014305 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4acc160310 msgr2=0x7f4acc158100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:51:06.865 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:06.866+0000 7f4ad10f8640 1 --2- 172.21.15.123:0/110014305 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4acc160310 0x7f4acc158100 secure :-1 s=READY pgs=1681 cs=0 l=1 rev1=1 crypto rx=0x7f4abc01a450 tx=0x7f4abc0a5790 comp rx=0 tx=0).stop 2024-09-16T15:51:06.865 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:06.866+0000 7f4ad10f8640 1 -- 172.21.15.123:0/110014305 shutdown_connections 2024-09-16T15:51:06.865 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:06.866+0000 7f4ad10f8640 1 --2- 172.21.15.123:0/110014305 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f4aac073730 0x7f4aac075bf0 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:06.866 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:06.866+0000 7f4ad10f8640 1 --2- 172.21.15.123:0/110014305 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4acc160310 0x7f4acc158100 unknown :-1 s=CLOSED pgs=1681 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:06.866 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:06.866+0000 7f4ad10f8640 1 --2- 172.21.15.123:0/110014305 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4acc15bd80 0x7f4acc06f7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:06.866 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:06.866+0000 7f4ad10f8640 1 --2- 172.21.15.123:0/110014305 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f4acc0d1b70 0x7f4acc0c6be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:06.866 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:06.866+0000 7f4ad10f8640 1 -- 172.21.15.123:0/110014305 >> 172.21.15.123:0/110014305 conn(0x7f4acc09aa40 msgr2=0x7f4acc09ae20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:51:06.866 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:06.866+0000 7f4ad10f8640 1 -- 172.21.15.123:0/110014305 shutdown_connections 2024-09-16T15:51:06.866 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:06.866+0000 7f4ad10f8640 1 -- 172.21.15.123:0/110014305 wait complete. 2024-09-16T15:51:06.886 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-09-16T15:51:07.023 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:07.026+0000 7fdecb767640 1 -- 172.21.15.123:0/3893441626 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdec40cc7e0 msgr2=0x7fdec40d0800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:51:07.024 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:07.026+0000 7fdecb767640 1 --2- 172.21.15.123:0/3893441626 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdec40cc7e0 0x7fdec40d0800 secure :-1 s=READY pgs=1682 cs=0 l=1 rev1=1 crypto rx=0x7fdebc009c80 tx=0x7fdebc01cd30 comp rx=0 tx=0).stop 2024-09-16T15:51:07.024 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:07.026+0000 7fdecb767640 1 -- 172.21.15.123:0/3893441626 shutdown_connections 2024-09-16T15:51:07.024 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:07.026+0000 7fdecb767640 1 --2- 172.21.15.123:0/3893441626 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdec40cc7e0 0x7fdec40d0800 unknown :-1 s=CLOSED pgs=1682 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:07.024 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:07.026+0000 7fdecb767640 1 --2- 172.21.15.123:0/3893441626 >> v2:172.21.15.184:3300/0 conn(0x7fdec40cc390 0x7fdec40d02c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:07.024 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:07.026+0000 7fdecb767640 1 -- 172.21.15.123:0/3893441626 >> 172.21.15.123:0/3893441626 conn(0x7fdec4074110 msgr2=0x7fdec40744f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:51:07.024 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:07.026+0000 7fdecb767640 1 -- 172.21.15.123:0/3893441626 shutdown_connections 2024-09-16T15:51:07.024 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:07.026+0000 7fdecb767640 1 -- 172.21.15.123:0/3893441626 wait complete. 2024-09-16T15:51:07.024 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:07.026+0000 7fdecb767640 1 Processor -- start 2024-09-16T15:51:07.025 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:07.026+0000 7fdecb767640 1 -- start start 2024-09-16T15:51:07.026 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:07.026+0000 7fdecb767640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fdec40cc390 0x7fdec40982f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:51:07.026 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:07.026+0000 7fdecb767640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdec40cc7e0 0x7fdec40b40b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:51:07.026 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:07.026+0000 7fdecb767640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fdec415e0f0 0x7fdec40aeea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:51:07.027 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:07.026+0000 7fdecb767640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fdec416b310 con 0x7fdec40cc7e0 2024-09-16T15:51:07.027 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:07.026+0000 7fdecb767640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fdec416b190 con 0x7fdec40cc390 2024-09-16T15:51:07.027 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:07.026+0000 7fdecb767640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fdec416b490 con 0x7fdec415e0f0 2024-09-16T15:51:07.027 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:07.026+0000 7fdec8d02640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdec40cc7e0 0x7fdec40b40b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:51:07.027 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:07.026+0000 7fdec8d02640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdec40cc7e0 0x7fdec40b40b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:55544/0 (socket says 172.21.15.123:55544) 2024-09-16T15:51:07.027 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:07.026+0000 7fdec8d02640 1 -- 172.21.15.123:0/4186788910 learned_addr learned my addr 172.21.15.123:0/4186788910 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:51:07.027 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:07.026+0000 7fdec9d04640 1 --2- 172.21.15.123:0/4186788910 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fdec415e0f0 0x7fdec40aeea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:51:07.027 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:07.026+0000 7fdec9503640 1 --2- 172.21.15.123:0/4186788910 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fdec40cc390 0x7fdec40982f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:51:07.027 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:07.026+0000 7fdec8d02640 1 -- 172.21.15.123:0/4186788910 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fdec415e0f0 msgr2=0x7fdec40aeea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:51:07.027 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:07.026+0000 7fdec8d02640 1 --2- 172.21.15.123:0/4186788910 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fdec415e0f0 0x7fdec40aeea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:07.027 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:07.026+0000 7fdec8d02640 1 -- 172.21.15.123:0/4186788910 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fdec40cc390 msgr2=0x7fdec40982f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:51:07.027 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:07.026+0000 7fdec8d02640 1 --2- 172.21.15.123:0/4186788910 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fdec40cc390 0x7fdec40982f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:07.028 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:07.026+0000 7fdec8d02640 1 -- 172.21.15.123:0/4186788910 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdec40b5bf0 con 0x7fdec40cc7e0 2024-09-16T15:51:07.028 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:07.026+0000 7fdec9d04640 1 --2- 172.21.15.123:0/4186788910 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fdec415e0f0 0x7fdec40aeea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T15:51:07.028 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:07.026+0000 7fdec8d02640 1 --2- 172.21.15.123:0/4186788910 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdec40cc7e0 0x7fdec40b40b0 secure :-1 s=READY pgs=1683 cs=0 l=1 rev1=1 crypto rx=0x7fdeb400b550 tx=0x7fdeb400ba10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:51:07.028 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:07.026+0000 7fdeb9ffb640 1 -- 172.21.15.123:0/4186788910 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdeb4014070 con 0x7fdec40cc7e0 2024-09-16T15:51:07.028 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:07.026+0000 7fdeb9ffb640 1 -- 172.21.15.123:0/4186788910 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fdeb4010980 con 0x7fdec40cc7e0 2024-09-16T15:51:07.028 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:07.026+0000 7fdecb767640 1 -- 172.21.15.123:0/4186788910 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdec40b5e90 con 0x7fdec40cc7e0 2024-09-16T15:51:07.028 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:07.026+0000 7fdeb9ffb640 1 -- 172.21.15.123:0/4186788910 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdeb4010dd0 con 0x7fdec40cc7e0 2024-09-16T15:51:07.028 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:07.026+0000 7fdecb767640 1 -- 172.21.15.123:0/4186788910 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdec40afe10 con 0x7fdec40cc7e0 2024-09-16T15:51:07.029 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:07.030+0000 7fdecb767640 1 -- 172.21.15.123:0/4186788910 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdec40ce230 con 0x7fdec40cc7e0 2024-09-16T15:51:07.029 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:07.030+0000 7fdeb9ffb640 1 -- 172.21.15.123:0/4186788910 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 24) ==== 97845+0+0 (secure 0 0 0) 0x7fdeb4017070 con 0x7fdec40cc7e0 2024-09-16T15:51:07.029 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:07.030+0000 7fdeb9ffb640 1 --2- 172.21.15.123:0/4186788910 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fdea8073730 0x7fdea8075bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:51:07.029 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:07.030+0000 7fdec9503640 1 --2- 172.21.15.123:0/4186788910 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fdea8073730 0x7fdea8075bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:51:07.029 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:07.030+0000 7fdeb9ffb640 1 -- 172.21.15.123:0/4186788910 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(148..148 src has 1..148) ==== 5385+0+0 (secure 0 0 0) 0x7fdeb40924b0 con 0x7fdec40cc7e0 2024-09-16T15:51:07.029 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:07.030+0000 7fdec9503640 1 --2- 172.21.15.123:0/4186788910 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fdea8073730 0x7fdea8075bf0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7fdec000a790 tx=0x7fdec0008040 comp rx=0 tx=0).ready entity=mgr.4150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:51:07.032 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:07.034+0000 7fdeb9ffb640 1 -- 172.21.15.123:0/4186788910 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7fdeb405f1b0 con 0x7fdec40cc7e0 2024-09-16T15:51:07.210 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:07.210+0000 7fdecb767640 1 -- 172.21.15.123:0/4186788910 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) -- 0x7fdec415bf60 con 0x7fdec40cc7e0 2024-09-16T15:51:07.861 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:07.862+0000 7fdeb9ffb640 1 -- 172.21.15.123:0/4186788910 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v149) ==== 169+0+0 (secure 0 0 0) 0x7fdeb405eab0 con 0x7fdec40cc7e0 2024-09-16T15:51:07.861 INFO:teuthology.orchestra.run.smithi123.stderr:pool 'cephfs_data' removed 2024-09-16T15:51:07.865 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:07.866+0000 7fdecb767640 1 -- 172.21.15.123:0/4186788910 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fdea8073730 msgr2=0x7fdea8075bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:51:07.866 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:07.866+0000 7fdecb767640 1 --2- 172.21.15.123:0/4186788910 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fdea8073730 0x7fdea8075bf0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7fdec000a790 tx=0x7fdec0008040 comp rx=0 tx=0).stop 2024-09-16T15:51:07.866 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:07.866+0000 7fdecb767640 1 -- 172.21.15.123:0/4186788910 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdec40cc7e0 msgr2=0x7fdec40b40b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:51:07.866 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:07.866+0000 7fdecb767640 1 --2- 172.21.15.123:0/4186788910 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdec40cc7e0 0x7fdec40b40b0 secure :-1 s=READY pgs=1683 cs=0 l=1 rev1=1 crypto rx=0x7fdeb400b550 tx=0x7fdeb400ba10 comp rx=0 tx=0).stop 2024-09-16T15:51:07.868 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:07.870+0000 7fdecb767640 1 -- 172.21.15.123:0/4186788910 shutdown_connections 2024-09-16T15:51:07.868 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:07.870+0000 7fdecb767640 1 --2- 172.21.15.123:0/4186788910 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fdea8073730 0x7fdea8075bf0 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:07.868 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:07.870+0000 7fdecb767640 1 --2- 172.21.15.123:0/4186788910 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fdec415e0f0 0x7fdec40aeea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:07.868 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:07.870+0000 7fdecb767640 1 --2- 172.21.15.123:0/4186788910 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdec40cc7e0 0x7fdec40b40b0 unknown :-1 s=CLOSED pgs=1683 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:07.868 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:07.870+0000 7fdecb767640 1 --2- 172.21.15.123:0/4186788910 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fdec40cc390 0x7fdec40982f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:07.868 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:07.870+0000 7fdecb767640 1 -- 172.21.15.123:0/4186788910 >> 172.21.15.123:0/4186788910 conn(0x7fdec4074110 msgr2=0x7fdec4160e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:51:07.868 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:07.870+0000 7fdecb767640 1 -- 172.21.15.123:0/4186788910 shutdown_connections 2024-09-16T15:51:07.868 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:07.870+0000 7fdecb767640 1 -- 172.21.15.123:0/4186788910 wait complete. 2024-09-16T15:51:07.895 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config reset 27 2024-09-16T15:51:08.024 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:08.026+0000 7f90c17ea640 1 -- 172.21.15.123:0/3355585406 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f90bc0cc6f0 msgr2=0x7f90bc0cf650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:51:08.024 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:08.026+0000 7f90c17ea640 1 --2- 172.21.15.123:0/3355585406 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f90bc0cc6f0 0x7f90bc0cf650 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7f90b4009c80 tx=0x7f90b401cda0 comp rx=0 tx=0).stop 2024-09-16T15:51:08.024 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:08.026+0000 7f90c17ea640 1 -- 172.21.15.123:0/3355585406 shutdown_connections 2024-09-16T15:51:08.024 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:08.026+0000 7f90c17ea640 1 --2- 172.21.15.123:0/3355585406 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f90bc0cc6f0 0x7f90bc0cf650 unknown :-1 s=CLOSED pgs=518 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:08.024 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:08.026+0000 7f90c17ea640 1 --2- 172.21.15.123:0/3355585406 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f90bc0b3100 0x7f90bc0cc1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:08.024 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:08.026+0000 7f90c17ea640 1 -- 172.21.15.123:0/3355585406 >> 172.21.15.123:0/3355585406 conn(0x7f90bc0b2610 msgr2=0x7f90bc0b29f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:51:08.024 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:08.026+0000 7f90c17ea640 1 -- 172.21.15.123:0/3355585406 shutdown_connections 2024-09-16T15:51:08.024 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:08.026+0000 7f90c17ea640 1 -- 172.21.15.123:0/3355585406 wait complete. 2024-09-16T15:51:08.025 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:08.026+0000 7f90c17ea640 1 Processor -- start 2024-09-16T15:51:08.025 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:08.026+0000 7f90c17ea640 1 -- start start 2024-09-16T15:51:08.027 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:08.026+0000 7f90c17ea640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f90bc0b3100 0x7f90bc13fd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:51:08.027 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:08.026+0000 7f90c17ea640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f90bc0cc6f0 0x7f90bc153780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:51:08.027 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:08.026+0000 7f90c17ea640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f90bc15ac00 0x7f90bc159cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:51:08.027 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:08.026+0000 7f90c17ea640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f90bc1682d0 con 0x7f90bc0cc6f0 2024-09-16T15:51:08.027 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:08.026+0000 7f90c17ea640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f90bc070a00 con 0x7f90bc15ac00 2024-09-16T15:51:08.027 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:08.026+0000 7f90c17ea640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f90bc168450 con 0x7f90bc0b3100 2024-09-16T15:51:08.027 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:08.026+0000 7f90b27fc640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f90bc0cc6f0 0x7f90bc153780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:51:08.027 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:08.026+0000 7f90b27fc640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f90bc0cc6f0 0x7f90bc153780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:55550/0 (socket says 172.21.15.123:55550) 2024-09-16T15:51:08.027 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:08.026+0000 7f90b27fc640 1 -- 172.21.15.123:0/1163770440 learned_addr learned my addr 172.21.15.123:0/1163770440 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:51:08.027 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:08.026+0000 7f90baffd640 1 --2- 172.21.15.123:0/1163770440 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f90bc0b3100 0x7f90bc13fd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:51:08.027 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:08.026+0000 7f90bb7fe640 1 --2- 172.21.15.123:0/1163770440 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f90bc15ac00 0x7f90bc159cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:51:08.027 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:08.026+0000 7f90b27fc640 1 -- 172.21.15.123:0/1163770440 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f90bc0b3100 msgr2=0x7f90bc13fd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:51:08.027 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:08.026+0000 7f90b27fc640 1 --2- 172.21.15.123:0/1163770440 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f90bc0b3100 0x7f90bc13fd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:08.027 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:08.026+0000 7f90b27fc640 1 -- 172.21.15.123:0/1163770440 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f90bc15ac00 msgr2=0x7f90bc159cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:51:08.028 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:08.026+0000 7f90b27fc640 1 --2- 172.21.15.123:0/1163770440 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f90bc15ac00 0x7f90bc159cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:08.028 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:08.026+0000 7f90b27fc640 1 -- 172.21.15.123:0/1163770440 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f90bc1421d0 con 0x7f90bc0cc6f0 2024-09-16T15:51:08.028 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:08.026+0000 7f90b27fc640 1 --2- 172.21.15.123:0/1163770440 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f90bc0cc6f0 0x7f90bc153780 secure :-1 s=READY pgs=1684 cs=0 l=1 rev1=1 crypto rx=0x7f90a400e9f0 tx=0x7f90a400eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:51:08.028 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:08.026+0000 7f90b3fff640 1 -- 172.21.15.123:0/1163770440 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90a400cd80 con 0x7f90bc0cc6f0 2024-09-16T15:51:08.028 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:08.026+0000 7f90b3fff640 1 -- 172.21.15.123:0/1163770440 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f90a4010430 con 0x7f90bc0cc6f0 2024-09-16T15:51:08.028 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:08.026+0000 7f90b3fff640 1 -- 172.21.15.123:0/1163770440 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90a4010860 con 0x7f90bc0cc6f0 2024-09-16T15:51:08.028 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:08.026+0000 7f90c17ea640 1 -- 172.21.15.123:0/1163770440 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f90bc1424b0 con 0x7f90bc0cc6f0 2024-09-16T15:51:08.028 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:08.026+0000 7f90c17ea640 1 -- 172.21.15.123:0/1163770440 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f90bc142a70 con 0x7f90bc0cc6f0 2024-09-16T15:51:08.028 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:08.030+0000 7f90c17ea640 1 -- 172.21.15.123:0/1163770440 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f90bc0cc1b0 con 0x7f90bc0cc6f0 2024-09-16T15:51:08.029 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:08.030+0000 7f90b3fff640 1 -- 172.21.15.123:0/1163770440 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 24) ==== 97845+0+0 (secure 0 0 0) 0x7f90a4010a00 con 0x7f90bc0cc6f0 2024-09-16T15:51:08.029 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:08.030+0000 7f90b3fff640 1 --2- 172.21.15.123:0/1163770440 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f909c073760 0x7f909c075c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:51:08.029 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:08.030+0000 7f90b3fff640 1 -- 172.21.15.123:0/1163770440 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(149..149 src has 1..149) ==== 4971+0+0 (secure 0 0 0) 0x7f90a4093a30 con 0x7f90bc0cc6f0 2024-09-16T15:51:08.029 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:08.030+0000 7f90baffd640 1 --2- 172.21.15.123:0/1163770440 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f909c073760 0x7f909c075c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:51:08.030 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:08.030+0000 7f90baffd640 1 --2- 172.21.15.123:0/1163770440 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f909c073760 0x7f909c075c20 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f90ac001650 tx=0x7f90ac009040 comp rx=0 tx=0).ready entity=mgr.4150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:51:08.033 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:08.034+0000 7f90b3fff640 1 -- 172.21.15.123:0/1163770440 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f90a40607d0 con 0x7f90bc0cc6f0 2024-09-16T15:51:08.208 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:08.210+0000 7f90c17ea640 1 -- 172.21.15.123:0/1163770440 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "config reset", "num": 27} v 0) -- 0x7f90bc0b6aa0 con 0x7f90bc0cc6f0 2024-09-16T15:51:08.214 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:08.214+0000 7f90b3fff640 1 -- 172.21.15.123:0/1163770440 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "config reset", "num": 27}]=0 v45) ==== 71+0+0 (secure 0 0 0) 0x7f90a40600d0 con 0x7f90bc0cc6f0 2024-09-16T15:51:08.216 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:08.218+0000 7f90c17ea640 1 -- 172.21.15.123:0/1163770440 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f909c073760 msgr2=0x7f909c075c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:51:08.216 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:08.218+0000 7f90c17ea640 1 --2- 172.21.15.123:0/1163770440 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f909c073760 0x7f909c075c20 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f90ac001650 tx=0x7f90ac009040 comp rx=0 tx=0).stop 2024-09-16T15:51:08.216 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:08.218+0000 7f90c17ea640 1 -- 172.21.15.123:0/1163770440 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f90bc0cc6f0 msgr2=0x7f90bc153780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:51:08.216 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:08.218+0000 7f90c17ea640 1 --2- 172.21.15.123:0/1163770440 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f90bc0cc6f0 0x7f90bc153780 secure :-1 s=READY pgs=1684 cs=0 l=1 rev1=1 crypto rx=0x7f90a400e9f0 tx=0x7f90a400eeb0 comp rx=0 tx=0).stop 2024-09-16T15:51:08.217 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:08.218+0000 7f90c17ea640 1 -- 172.21.15.123:0/1163770440 shutdown_connections 2024-09-16T15:51:08.217 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:08.218+0000 7f90c17ea640 1 --2- 172.21.15.123:0/1163770440 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f909c073760 0x7f909c075c20 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:08.217 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:08.218+0000 7f90c17ea640 1 --2- 172.21.15.123:0/1163770440 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f90bc15ac00 0x7f90bc159cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:08.217 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:08.218+0000 7f90c17ea640 1 --2- 172.21.15.123:0/1163770440 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f90bc0cc6f0 0x7f90bc153780 unknown :-1 s=CLOSED pgs=1684 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:08.217 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:08.218+0000 7f90c17ea640 1 --2- 172.21.15.123:0/1163770440 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f90bc0b3100 0x7f90bc13fd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:08.217 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:08.218+0000 7f90c17ea640 1 -- 172.21.15.123:0/1163770440 >> 172.21.15.123:0/1163770440 conn(0x7f90bc0b2610 msgr2=0x7f90bc0b5b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:51:08.217 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:08.218+0000 7f90c17ea640 1 -- 172.21.15.123:0/1163770440 shutdown_connections 2024-09-16T15:51:08.217 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:08.218+0000 7f90c17ea640 1 -- 172.21.15.123:0/1163770440 wait complete. 2024-09-16T15:51:08.233 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_volumes.TestMisc.test_subvolume_upgrade_legacy_to_v1' 2024-09-16T15:51:08.361 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:08.362+0000 7f4e330c9640 1 -- 172.21.15.123:0/3218778602 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4e18003a10 msgr2=0x7f4e18003e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:51:08.361 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:08.362+0000 7f4e330c9640 1 --2- 172.21.15.123:0/3218778602 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4e18003a10 0x7f4e18003e10 secure :-1 s=READY pgs=1686 cs=0 l=1 rev1=1 crypto rx=0x7f4e1c009e30 tx=0x7f4e1c01d190 comp rx=0 tx=0).stop 2024-09-16T15:51:08.361 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:08.362+0000 7f4e330c9640 1 -- 172.21.15.123:0/3218778602 shutdown_connections 2024-09-16T15:51:08.362 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:08.362+0000 7f4e330c9640 1 --2- 172.21.15.123:0/3218778602 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4e18003a10 0x7f4e18003e10 unknown :-1 s=CLOSED pgs=1686 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:08.362 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:08.362+0000 7f4e330c9640 1 -- 172.21.15.123:0/3218778602 >> 172.21.15.123:0/3218778602 conn(0x7f4e2c074bd0 msgr2=0x7f4e2c09aa40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:51:08.362 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:08.362+0000 7f4e330c9640 1 -- 172.21.15.123:0/3218778602 shutdown_connections 2024-09-16T15:51:08.362 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:08.362+0000 7f4e330c9640 1 -- 172.21.15.123:0/3218778602 wait complete. 2024-09-16T15:51:08.362 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:08.362+0000 7f4e330c9640 1 Processor -- start 2024-09-16T15:51:08.363 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:08.362+0000 7f4e330c9640 1 -- start start 2024-09-16T15:51:08.364 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:08.366+0000 7f4e330c9640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4e18003a10 0x7f4e2c1526a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:51:08.365 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:08.366+0000 7f4e330c9640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4e2c0b5930 0x7f4e2c06f050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:51:08.365 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:08.366+0000 7f4e330c9640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f4e2c0b6060 0x7f4e2c0b64f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:51:08.365 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:08.366+0000 7f4e330c9640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f4e2c0cf050 con 0x7f4e18003a10 2024-09-16T15:51:08.365 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:08.366+0000 7f4e330c9640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f4e2c0ceed0 con 0x7f4e2c0b5930 2024-09-16T15:51:08.365 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:08.366+0000 7f4e330c9640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f4e2c0cf1d0 con 0x7f4e2c0b6060 2024-09-16T15:51:08.365 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:08.366+0000 7f4e30e65640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4e18003a10 0x7f4e2c1526a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:51:08.365 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:08.366+0000 7f4e30e65640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4e18003a10 0x7f4e2c1526a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:55568/0 (socket says 172.21.15.123:55568) 2024-09-16T15:51:08.365 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:08.366+0000 7f4e30e65640 1 -- 172.21.15.123:0/2849775395 learned_addr learned my addr 172.21.15.123:0/2849775395 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:51:08.365 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:08.366+0000 7f4e2bfff640 1 --2- 172.21.15.123:0/2849775395 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4e2c0b5930 0x7f4e2c06f050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:51:08.365 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:08.366+0000 7f4e31666640 1 --2- 172.21.15.123:0/2849775395 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f4e2c0b6060 0x7f4e2c0b64f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:51:08.365 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:08.366+0000 7f4e30e65640 1 -- 172.21.15.123:0/2849775395 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f4e2c0b6060 msgr2=0x7f4e2c0b64f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:51:08.365 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:08.366+0000 7f4e30e65640 1 --2- 172.21.15.123:0/2849775395 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f4e2c0b6060 0x7f4e2c0b64f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:08.365 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:08.366+0000 7f4e30e65640 1 -- 172.21.15.123:0/2849775395 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4e2c0b5930 msgr2=0x7f4e2c06f050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:51:08.365 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:08.366+0000 7f4e30e65640 1 --2- 172.21.15.123:0/2849775395 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4e2c0b5930 0x7f4e2c06f050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:08.366 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:08.366+0000 7f4e30e65640 1 -- 172.21.15.123:0/2849775395 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4e2c0c5d50 con 0x7f4e18003a10 2024-09-16T15:51:08.366 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:08.366+0000 7f4e30e65640 1 --2- 172.21.15.123:0/2849775395 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4e18003a10 0x7f4e2c1526a0 secure :-1 s=READY pgs=1687 cs=0 l=1 rev1=1 crypto rx=0x7f4e1c0af040 tx=0x7f4e1c005a80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:51:08.366 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:08.366+0000 7f4e297fa640 1 -- 172.21.15.123:0/2849775395 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e1c0ba020 con 0x7f4e18003a10 2024-09-16T15:51:08.366 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:08.366+0000 7f4e297fa640 1 -- 172.21.15.123:0/2849775395 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f4e1c0ac8a0 con 0x7f4e18003a10 2024-09-16T15:51:08.366 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:08.366+0000 7f4e330c9640 1 -- 172.21.15.123:0/2849775395 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4e2c152be0 con 0x7f4e18003a10 2024-09-16T15:51:08.366 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:08.366+0000 7f4e297fa640 1 -- 172.21.15.123:0/2849775395 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e1c0b1040 con 0x7f4e18003a10 2024-09-16T15:51:08.366 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:08.366+0000 7f4e330c9640 1 -- 172.21.15.123:0/2849775395 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4e2c06f860 con 0x7f4e18003a10 2024-09-16T15:51:08.367 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:08.366+0000 7f4e330c9640 1 -- 172.21.15.123:0/2849775395 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4e2c0cdcf0 con 0x7f4e18003a10 2024-09-16T15:51:08.367 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:08.366+0000 7f4e297fa640 1 -- 172.21.15.123:0/2849775395 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 24) ==== 97845+0+0 (secure 0 0 0) 0x7f4e1c0bd020 con 0x7f4e18003a10 2024-09-16T15:51:08.367 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:08.366+0000 7f4e297fa640 1 --2- 172.21.15.123:0/2849775395 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f4e0c073760 0x7f4e0c075c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:51:08.367 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:08.366+0000 7f4e297fa640 1 -- 172.21.15.123:0/2849775395 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(149..149 src has 1..149) ==== 4971+0+0 (secure 0 0 0) 0x7f4e1c12d960 con 0x7f4e18003a10 2024-09-16T15:51:08.367 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:08.366+0000 7f4e2bfff640 1 --2- 172.21.15.123:0/2849775395 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f4e0c073760 0x7f4e0c075c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:51:08.367 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:08.370+0000 7f4e2bfff640 1 --2- 172.21.15.123:0/2849775395 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f4e0c073760 0x7f4e0c075c20 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f4e2c02eb60 tx=0x7f4e14006b30 comp rx=0 tx=0).ready entity=mgr.4150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:51:08.370 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:08.370+0000 7f4e297fa640 1 -- 172.21.15.123:0/2849775395 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f4e1c0c4030 con 0x7f4e18003a10 2024-09-16T15:51:08.605 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:08.606+0000 7f4e330c9640 1 -- 172.21.15.123:0/2849775395 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_volumes.TestMisc.test_subvolume_upgrade_legacy_to_v1"]} v 0) -- 0x7f4e2c15b8f0 con 0x7f4e18003a10 2024-09-16T15:51:08.868 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:08.870+0000 7f4e297fa640 1 -- 172.21.15.123:0/2849775395 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_volumes.TestMisc.test_subvolume_upgrade_legacy_to_v1"]}]=0 v434) ==== 149+0+0 (secure 0 0 0) 0x7f4e2c0d0b20 con 0x7f4e18003a10 2024-09-16T15:51:08.873 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:08.874+0000 7f4e330c9640 1 -- 172.21.15.123:0/2849775395 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f4e0c073760 msgr2=0x7f4e0c075c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:51:08.873 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:08.874+0000 7f4e330c9640 1 --2- 172.21.15.123:0/2849775395 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f4e0c073760 0x7f4e0c075c20 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f4e2c02eb60 tx=0x7f4e14006b30 comp rx=0 tx=0).stop 2024-09-16T15:51:08.873 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:08.874+0000 7f4e330c9640 1 -- 172.21.15.123:0/2849775395 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4e18003a10 msgr2=0x7f4e2c1526a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:51:08.873 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:08.874+0000 7f4e330c9640 1 --2- 172.21.15.123:0/2849775395 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4e18003a10 0x7f4e2c1526a0 secure :-1 s=READY pgs=1687 cs=0 l=1 rev1=1 crypto rx=0x7f4e1c0af040 tx=0x7f4e1c005a80 comp rx=0 tx=0).stop 2024-09-16T15:51:08.873 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:08.874+0000 7f4e330c9640 1 -- 172.21.15.123:0/2849775395 shutdown_connections 2024-09-16T15:51:08.873 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:08.874+0000 7f4e330c9640 1 --2- 172.21.15.123:0/2849775395 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f4e0c073760 0x7f4e0c075c20 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:08.873 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:08.874+0000 7f4e330c9640 1 --2- 172.21.15.123:0/2849775395 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f4e2c0b6060 0x7f4e2c0b64f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:08.873 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:08.874+0000 7f4e330c9640 1 --2- 172.21.15.123:0/2849775395 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4e2c0b5930 0x7f4e2c06f050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:08.873 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:08.874+0000 7f4e330c9640 1 --2- 172.21.15.123:0/2849775395 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4e18003a10 0x7f4e2c1526a0 unknown :-1 s=CLOSED pgs=1687 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:08.873 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:08.874+0000 7f4e330c9640 1 -- 172.21.15.123:0/2849775395 >> 172.21.15.123:0/2849775395 conn(0x7f4e2c074bd0 msgr2=0x7f4e2c159db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:51:08.874 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:08.874+0000 7f4e330c9640 1 -- 172.21.15.123:0/2849775395 shutdown_connections 2024-09-16T15:51:08.874 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:08.874+0000 7f4e330c9640 1 -- 172.21.15.123:0/2849775395 wait complete. 2024-09-16T15:51:08.894 INFO:tasks.cephfs_test_runner:poor man's upgrade test -- rather than going through a full upgrade cycle, ... ok 2024-09-16T15:51:08.894 INFO:tasks.cephfs_test_runner:Starting test: test_subvolume_upgrade_v1_to_v2 (tasks.cephfs.test_volumes.TestMisc) poor man's upgrade test -- theme continues... 2024-09-16T15:51:08.894 INFO:tasks.cephfs_test_runner:test_subvolume_upgrade_v1_to_v2 (tasks.cephfs.test_volumes.TestMisc) 2024-09-16T15:51:08.895 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-16T15:51:09.025 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:09.026+0000 7fbf1c7a2640 1 -- 172.21.15.123:0/2367307093 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbefc005320 msgr2=0x7fbefc00b9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:51:09.025 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:09.026+0000 7fbf1c7a2640 1 --2- 172.21.15.123:0/2367307093 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbefc005320 0x7fbefc00b9d0 secure :-1 s=READY pgs=1689 cs=0 l=1 rev1=1 crypto rx=0x7fbf0400c610 tx=0x7fbf0401d2e0 comp rx=0 tx=0).stop 2024-09-16T15:51:09.025 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:09.026+0000 7fbf1c7a2640 1 -- 172.21.15.123:0/2367307093 shutdown_connections 2024-09-16T15:51:09.025 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:09.026+0000 7fbf1c7a2640 1 --2- 172.21.15.123:0/2367307093 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbefc005320 0x7fbefc00b9d0 unknown :-1 s=CLOSED pgs=1689 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:09.025 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:09.026+0000 7fbf1c7a2640 1 -- 172.21.15.123:0/2367307093 >> 172.21.15.123:0/2367307093 conn(0x7fbf140cfb10 msgr2=0x7fbf140cff20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:51:09.025 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:09.026+0000 7fbf1c7a2640 1 -- 172.21.15.123:0/2367307093 shutdown_connections 2024-09-16T15:51:09.025 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:09.026+0000 7fbf1c7a2640 1 -- 172.21.15.123:0/2367307093 wait complete. 2024-09-16T15:51:09.026 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:09.026+0000 7fbf1c7a2640 1 Processor -- start 2024-09-16T15:51:09.026 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:09.026+0000 7fbf1c7a2640 1 -- start start 2024-09-16T15:51:09.028 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:09.026+0000 7fbf1c7a2640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbefc005320 0x7fbf14159870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:51:09.028 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:09.026+0000 7fbf1c7a2640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fbf140cdde0 0x7fbf14151a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:51:09.028 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:09.026+0000 7fbf1c7a2640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fbf14153b20 0x7fbf14153fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:51:09.028 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:09.026+0000 7fbf1c7a2640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fbf14073f10 con 0x7fbefc005320 2024-09-16T15:51:09.028 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:09.026+0000 7fbf1c7a2640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fbf14074090 con 0x7fbf14153b20 2024-09-16T15:51:09.028 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:09.026+0000 7fbf1c7a2640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fbf14074210 con 0x7fbf140cdde0 2024-09-16T15:51:09.028 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:09.026+0000 7fbf1a53e640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbefc005320 0x7fbf14159870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:51:09.028 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:09.026+0000 7fbf1a53e640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbefc005320 0x7fbf14159870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:55592/0 (socket says 172.21.15.123:55592) 2024-09-16T15:51:09.028 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:09.026+0000 7fbf1a53e640 1 -- 172.21.15.123:0/2640635757 learned_addr learned my addr 172.21.15.123:0/2640635757 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:51:09.029 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:09.030+0000 7fbf1ad3f640 1 --2- 172.21.15.123:0/2640635757 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fbf14153b20 0x7fbf14153fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:51:09.029 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:09.030+0000 7fbf19d3d640 1 --2- 172.21.15.123:0/2640635757 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fbf140cdde0 0x7fbf14151a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:51:09.029 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:09.030+0000 7fbf1a53e640 1 -- 172.21.15.123:0/2640635757 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fbf140cdde0 msgr2=0x7fbf14151a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:51:09.029 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:09.030+0000 7fbf1a53e640 1 --2- 172.21.15.123:0/2640635757 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fbf140cdde0 0x7fbf14151a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:09.029 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:09.030+0000 7fbf1a53e640 1 -- 172.21.15.123:0/2640635757 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fbf14153b20 msgr2=0x7fbf14153fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:51:09.029 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:09.030+0000 7fbf1a53e640 1 --2- 172.21.15.123:0/2640635757 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fbf14153b20 0x7fbf14153fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:09.029 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:09.030+0000 7fbf1a53e640 1 -- 172.21.15.123:0/2640635757 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbf14151580 con 0x7fbefc005320 2024-09-16T15:51:09.029 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:09.030+0000 7fbf1a53e640 1 --2- 172.21.15.123:0/2640635757 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbefc005320 0x7fbf14159870 secure :-1 s=READY pgs=1690 cs=0 l=1 rev1=1 crypto rx=0x7fbf0c010760 tx=0x7fbf0c010c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:51:09.029 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:09.030+0000 7fbf12ffd640 1 -- 172.21.15.123:0/2640635757 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf0c00ece0 con 0x7fbefc005320 2024-09-16T15:51:09.029 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:09.030+0000 7fbf12ffd640 1 -- 172.21.15.123:0/2640635757 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fbf0c00ee80 con 0x7fbefc005320 2024-09-16T15:51:09.029 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:09.030+0000 7fbf12ffd640 1 -- 172.21.15.123:0/2640635757 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf0c005260 con 0x7fbefc005320 2024-09-16T15:51:09.029 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:09.030+0000 7fbf1c7a2640 1 -- 172.21.15.123:0/2640635757 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbf14152110 con 0x7fbefc005320 2024-09-16T15:51:09.029 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:09.030+0000 7fbf1c7a2640 1 -- 172.21.15.123:0/2640635757 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbf14158d90 con 0x7fbefc005320 2024-09-16T15:51:09.030 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:09.030+0000 7fbf1c7a2640 1 -- 172.21.15.123:0/2640635757 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbf1415efd0 con 0x7fbefc005320 2024-09-16T15:51:09.030 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:09.030+0000 7fbf12ffd640 1 -- 172.21.15.123:0/2640635757 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 24) ==== 97845+0+0 (secure 0 0 0) 0x7fbf0c005400 con 0x7fbefc005320 2024-09-16T15:51:09.030 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:09.030+0000 7fbf12ffd640 1 --2- 172.21.15.123:0/2640635757 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fbf00073760 0x7fbf00075c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:51:09.030 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:09.030+0000 7fbf19d3d640 1 --2- 172.21.15.123:0/2640635757 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fbf00073760 0x7fbf00075c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:51:09.030 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:09.030+0000 7fbf12ffd640 1 -- 172.21.15.123:0/2640635757 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(149..149 src has 1..149) ==== 4971+0+0 (secure 0 0 0) 0x7fbf0c093270 con 0x7fbefc005320 2024-09-16T15:51:09.031 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:09.030+0000 7fbf19d3d640 1 --2- 172.21.15.123:0/2640635757 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fbf00073760 0x7fbf00075c20 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7fbf14153510 tx=0x7fbf08006ad0 comp rx=0 tx=0).ready entity=mgr.4150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:51:09.034 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:09.034+0000 7fbf12ffd640 1 -- 172.21.15.123:0/2640635757 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7fbf0c05ff70 con 0x7fbefc005320 2024-09-16T15:51:09.194 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:09.194+0000 7fbf1c7a2640 1 -- 172.21.15.123:0/2640635757 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fbf140d06c0 con 0x7fbefc005320 2024-09-16T15:51:09.195 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:09.194+0000 7fbf12ffd640 1 -- 172.21.15.123:0/2640635757 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v149) ==== 74+0+13148 (secure 0 0 0) 0x7fbf0c05f870 con 0x7fbefc005320 2024-09-16T15:51:09.195 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-09-16T15:51:09.195 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":149,"fsid":"56077f89-18c8-486f-ae7e-eecc0bb7822c","created":"2024-09-16T15:42:26.364892+0000","modified":"2024-09-16T15:51:07.862206+0000","last_up_change":"2024-09-16T15:42:38.707234+0000","last_in_change":"2024-09-16T15:42:29.423662+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":23,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-16T15:42:39.287764+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9e7c1955-a607-4671-9be7-f2ef760ceeaa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":144,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6800","nonce":3882044969},{"type":"v1","addr":"172.21.15.123:6801","nonce":3882044969}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6802","nonce":3882044969},{"type":"v1","addr":"172.21.15.123:6803","nonce":3882044969}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6806","nonce":3882044969},{"type":"v1","addr":"172.21.15.123:6807","nonce":3882044969}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6804","nonce":3882044969},{"type":"v1","addr":"172.21.15.123:6805","nonce":3882044969}]},"public_addr":"172.21.15.123:6801/3882044969","cluster_addr":"172.21.15.123:6803/3882044969","heartbeat_back_addr":"172.21.15.123:6807/3882044969","heartbeat_front_addr":"172.21.15.123:6805/3882044969","state":["exists","up"]},{"osd":1,"uuid":"c540cfde-0977-4742-9b17-fd634668e010","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":144,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6816","nonce":4037001850},{"type":"v1","addr":"172.21.15.123:6817","nonce":4037001850}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6818","nonce":4037001850},{"type":"v1","addr":"172.21.15.123:6819","nonce":4037001850}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6822","nonce":4037001850},{"type":"v1","addr":"172.21.15.123:6823","nonce":4037001850}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6820","nonce":4037001850},{"type":"v1","addr":"172.21.15.123:6821","nonce":4037001850}]},"public_addr":"172.21.15.123:6817/4037001850","cluster_addr":"172.21.15.123:6819/4037001850","heartbeat_back_addr":"172.21.15.123:6823/4037001850","heartbeat_front_addr":"172.21.15.123:6821/4037001850","state":["exists","up"]},{"osd":2,"uuid":"9864a496-f169-486b-95c4-1d59162793a7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":144,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6824","nonce":1332143088},{"type":"v1","addr":"172.21.15.123:6825","nonce":1332143088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6826","nonce":1332143088},{"type":"v1","addr":"172.21.15.123:6827","nonce":1332143088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6830","nonce":1332143088},{"type":"v1","addr":"172.21.15.123:6831","nonce":1332143088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6828","nonce":1332143088},{"type":"v1","addr":"172.21.15.123:6829","nonce":1332143088}]},"public_addr":"172.21.15.123:6825/1332143088","cluster_addr":"172.21.15.123:6827/1332143088","heartbeat_back_addr":"172.21.15.123:6831/1332143088","heartbeat_front_addr":"172.21.15.123:6829/1332143088","state":["exists","up"]},{"osd":3,"uuid":"c6d1f817-f074-4325-95e1-578eba13edda","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":144,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6808","nonce":988586998},{"type":"v1","addr":"172.21.15.123:6809","nonce":988586998}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6810","nonce":988586998},{"type":"v1","addr":"172.21.15.123:6811","nonce":988586998}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6814","nonce":988586998},{"type":"v1","addr":"172.21.15.123:6815","nonce":988586998}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6812","nonce":988586998},{"type":"v1","addr":"172.21.15.123:6813","nonce":988586998}]},"public_addr":"172.21.15.123:6809/988586998","cluster_addr":"172.21.15.123:6811/988586998","heartbeat_back_addr":"172.21.15.123:6815/988586998","heartbeat_front_addr":"172.21.15.123:6813/988586998","state":["exists","up"]},{"osd":4,"uuid":"200a8df4-50ef-40a5-b48a-58981269c086","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":144,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6816","nonce":3387813890},{"type":"v1","addr":"172.21.15.184:6817","nonce":3387813890}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6818","nonce":3387813890},{"type":"v1","addr":"172.21.15.184:6819","nonce":3387813890}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6822","nonce":3387813890},{"type":"v1","addr":"172.21.15.184:6823","nonce":3387813890}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6820","nonce":3387813890},{"type":"v1","addr":"172.21.15.184:6821","nonce":3387813890}]},"public_addr":"172.21.15.184:6817/3387813890","cluster_addr":"172.21.15.184:6819/3387813890","heartbeat_back_addr":"172.21.15.184:6823/3387813890","heartbeat_front_addr":"172.21.15.184:6821/3387813890","state":["exists","up"]},{"osd":5,"uuid":"32d5a900-da3d-412c-bc19-13861a2368d8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":144,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6808","nonce":2680916017},{"type":"v1","addr":"172.21.15.184:6809","nonce":2680916017}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6810","nonce":2680916017},{"type":"v1","addr":"172.21.15.184:6811","nonce":2680916017}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6814","nonce":2680916017},{"type":"v1","addr":"172.21.15.184:6815","nonce":2680916017}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6812","nonce":2680916017},{"type":"v1","addr":"172.21.15.184:6813","nonce":2680916017}]},"public_addr":"172.21.15.184:6809/2680916017","cluster_addr":"172.21.15.184:6811/2680916017","heartbeat_back_addr":"172.21.15.184:6815/2680916017","heartbeat_front_addr":"172.21.15.184:6813/2680916017","state":["exists","up"]},{"osd":6,"uuid":"cde6f39e-aa1b-4260-ac19-774ee6aaca72","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":144,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6824","nonce":2183120457},{"type":"v1","addr":"172.21.15.184:6825","nonce":2183120457}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6826","nonce":2183120457},{"type":"v1","addr":"172.21.15.184:6827","nonce":2183120457}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6830","nonce":2183120457},{"type":"v1","addr":"172.21.15.184:6831","nonce":2183120457}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6828","nonce":2183120457},{"type":"v1","addr":"172.21.15.184:6829","nonce":2183120457}]},"public_addr":"172.21.15.184:6825/2183120457","cluster_addr":"172.21.15.184:6827/2183120457","heartbeat_back_addr":"172.21.15.184:6831/2183120457","heartbeat_front_addr":"172.21.15.184:6829/2183120457","state":["exists","up"]},{"osd":7,"uuid":"4f381778-eee1-47c6-804f-df0e6b33e313","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":144,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6800","nonce":4102957285},{"type":"v1","addr":"172.21.15.184:6801","nonce":4102957285}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6802","nonce":4102957285},{"type":"v1","addr":"172.21.15.184:6803","nonce":4102957285}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6806","nonce":4102957285},{"type":"v1","addr":"172.21.15.184:6807","nonce":4102957285}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6804","nonce":4102957285},{"type":"v1","addr":"172.21.15.184:6805","nonce":4102957285}]},"public_addr":"172.21.15.184:6801/4102957285","cluster_addr":"172.21.15.184:6803/4102957285","heartbeat_back_addr":"172.21.15.184:6807/4102957285","heartbeat_front_addr":"172.21.15.184:6805/4102957285","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:35.972210+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.222007+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.198617+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.385663+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.358935+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.156066+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.752973+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:35.682846+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.184:6833/1547569255":"2024-09-17T15:51:04.848452+0000","172.21.15.184:6832/1547569255":"2024-09-17T15:51:04.848452+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":17,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T15:51:09.197 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:09.198+0000 7fbf1c7a2640 1 -- 172.21.15.123:0/2640635757 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fbf00073760 msgr2=0x7fbf00075c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:51:09.197 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:09.198+0000 7fbf1c7a2640 1 --2- 172.21.15.123:0/2640635757 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fbf00073760 0x7fbf00075c20 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7fbf14153510 tx=0x7fbf08006ad0 comp rx=0 tx=0).stop 2024-09-16T15:51:09.197 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:09.198+0000 7fbf1c7a2640 1 -- 172.21.15.123:0/2640635757 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbefc005320 msgr2=0x7fbf14159870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:51:09.197 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:09.198+0000 7fbf1c7a2640 1 --2- 172.21.15.123:0/2640635757 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbefc005320 0x7fbf14159870 secure :-1 s=READY pgs=1690 cs=0 l=1 rev1=1 crypto rx=0x7fbf0c010760 tx=0x7fbf0c010c20 comp rx=0 tx=0).stop 2024-09-16T15:51:09.197 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:09.198+0000 7fbf1c7a2640 1 -- 172.21.15.123:0/2640635757 shutdown_connections 2024-09-16T15:51:09.197 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:09.198+0000 7fbf1c7a2640 1 --2- 172.21.15.123:0/2640635757 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fbf00073760 0x7fbf00075c20 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:09.197 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:09.198+0000 7fbf1c7a2640 1 --2- 172.21.15.123:0/2640635757 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fbf14153b20 0x7fbf14153fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:09.197 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:09.198+0000 7fbf1c7a2640 1 --2- 172.21.15.123:0/2640635757 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fbf140cdde0 0x7fbf14151a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:09.197 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:09.198+0000 7fbf1c7a2640 1 --2- 172.21.15.123:0/2640635757 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbefc005320 0x7fbf14159870 unknown :-1 s=CLOSED pgs=1690 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:09.198 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:09.198+0000 7fbf1c7a2640 1 -- 172.21.15.123:0/2640635757 >> 172.21.15.123:0/2640635757 conn(0x7fbf140cfb10 msgr2=0x7fbf140b2e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:51:09.198 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:09.198+0000 7fbf1c7a2640 1 -- 172.21.15.123:0/2640635757 shutdown_connections 2024-09-16T15:51:09.198 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:09.198+0000 7fbf1c7a2640 1 -- 172.21.15.123:0/2640635757 wait complete. 2024-09-16T15:51:09.212 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-16T15:42:39.287764+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-09-16T15:51:09.212 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-09-16T15:51:09.341 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:09.342+0000 7f3d16521640 1 -- 172.21.15.123:0/4218995391 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3cf8004810 msgr2=0x7f3cf8004c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:51:09.341 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:09.342+0000 7f3d16521640 1 --2- 172.21.15.123:0/4218995391 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3cf8004810 0x7f3cf8004c90 secure :-1 s=READY pgs=1692 cs=0 l=1 rev1=1 crypto rx=0x7f3d0400c710 tx=0x7f3d0401c690 comp rx=0 tx=0).stop 2024-09-16T15:51:09.341 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:09.342+0000 7f3d16521640 1 -- 172.21.15.123:0/4218995391 shutdown_connections 2024-09-16T15:51:09.341 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:09.342+0000 7f3d16521640 1 --2- 172.21.15.123:0/4218995391 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3cf8004810 0x7f3cf8004c90 unknown :-1 s=CLOSED pgs=1692 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:09.341 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:09.342+0000 7f3d16521640 1 -- 172.21.15.123:0/4218995391 >> 172.21.15.123:0/4218995391 conn(0x7f3d100b1710 msgr2=0x7f3d100b1b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:51:09.341 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:09.342+0000 7f3d16521640 1 -- 172.21.15.123:0/4218995391 shutdown_connections 2024-09-16T15:51:09.341 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:09.342+0000 7f3d16521640 1 -- 172.21.15.123:0/4218995391 wait complete. 2024-09-16T15:51:09.342 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:09.342+0000 7f3d16521640 1 Processor -- start 2024-09-16T15:51:09.342 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:09.342+0000 7f3d16521640 1 -- start start 2024-09-16T15:51:09.344 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:09.342+0000 7f3d16521640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3cf8004810 0x7f3d10158610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:51:09.344 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:09.342+0000 7f3d16521640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f3d101578e0 0x7f3d100b3d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:51:09.344 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:09.342+0000 7f3d16521640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3d10158dc0 0x7f3d10159270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:51:09.344 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:09.342+0000 7f3d16521640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f3d1015f560 con 0x7f3cf8004810 2024-09-16T15:51:09.344 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:09.342+0000 7f3d16521640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f3d1015f3e0 con 0x7f3d10158dc0 2024-09-16T15:51:09.344 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:09.342+0000 7f3d16521640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f3d1006ebe0 con 0x7f3d101578e0 2024-09-16T15:51:09.344 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:09.342+0000 7f3d0ffff640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3cf8004810 0x7f3d10158610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:51:09.344 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:09.342+0000 7f3d0ffff640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3cf8004810 0x7f3d10158610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:55616/0 (socket says 172.21.15.123:55616) 2024-09-16T15:51:09.344 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:09.342+0000 7f3d0ffff640 1 -- 172.21.15.123:0/1430718113 learned_addr learned my addr 172.21.15.123:0/1430718113 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:51:09.344 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:09.342+0000 7f3d14abe640 1 --2- 172.21.15.123:0/1430718113 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3d10158dc0 0x7f3d10159270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:51:09.344 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:09.342+0000 7f3d0ffff640 1 -- 172.21.15.123:0/1430718113 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f3d101578e0 msgr2=0x7f3d100b3d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:51:09.344 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:09.342+0000 7f3d0ffff640 1 --2- 172.21.15.123:0/1430718113 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f3d101578e0 0x7f3d100b3d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:09.345 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:09.342+0000 7f3d0ffff640 1 -- 172.21.15.123:0/1430718113 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3d10158dc0 msgr2=0x7f3d10159270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:51:09.345 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:09.342+0000 7f3d0ffff640 1 --2- 172.21.15.123:0/1430718113 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3d10158dc0 0x7f3d10159270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:09.345 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:09.342+0000 7f3d0ffff640 1 -- 172.21.15.123:0/1430718113 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3d100b4e80 con 0x7f3cf8004810 2024-09-16T15:51:09.345 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:09.342+0000 7f3d14abe640 1 --2- 172.21.15.123:0/1430718113 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3d10158dc0 0x7f3d10159270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T15:51:09.345 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:09.346+0000 7f3d0ffff640 1 --2- 172.21.15.123:0/1430718113 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3cf8004810 0x7f3d10158610 secure :-1 s=READY pgs=1693 cs=0 l=1 rev1=1 crypto rx=0x7f3d00001230 tx=0x7f3d0000cae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:51:09.345 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:09.346+0000 7f3d0cff9640 1 -- 172.21.15.123:0/1430718113 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d00017070 con 0x7f3cf8004810 2024-09-16T15:51:09.345 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:09.346+0000 7f3d0cff9640 1 -- 172.21.15.123:0/1430718113 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f3d00003600 con 0x7f3cf8004810 2024-09-16T15:51:09.345 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:09.346+0000 7f3d16521640 1 -- 172.21.15.123:0/1430718113 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3d100b45e0 con 0x7f3cf8004810 2024-09-16T15:51:09.345 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:09.346+0000 7f3d0cff9640 1 -- 172.21.15.123:0/1430718113 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d00003a30 con 0x7f3cf8004810 2024-09-16T15:51:09.345 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:09.346+0000 7f3d16521640 1 -- 172.21.15.123:0/1430718113 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3d100b4ba0 con 0x7f3cf8004810 2024-09-16T15:51:09.346 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:09.346+0000 7f3d16521640 1 -- 172.21.15.123:0/1430718113 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3d1006f140 con 0x7f3cf8004810 2024-09-16T15:51:09.346 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:09.346+0000 7f3d0cff9640 1 -- 172.21.15.123:0/1430718113 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 24) ==== 97845+0+0 (secure 0 0 0) 0x7f3d0001a070 con 0x7f3cf8004810 2024-09-16T15:51:09.346 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:09.346+0000 7f3d0cff9640 1 --2- 172.21.15.123:0/1430718113 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f3cfc073760 0x7f3cfc075c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:51:09.346 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:09.346+0000 7f3d0cff9640 1 -- 172.21.15.123:0/1430718113 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(149..149 src has 1..149) ==== 4971+0+0 (secure 0 0 0) 0x7f3d00092230 con 0x7f3cf8004810 2024-09-16T15:51:09.346 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:09.346+0000 7f3d0f7fe640 1 --2- 172.21.15.123:0/1430718113 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f3cfc073760 0x7f3cfc075c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:51:09.346 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:09.346+0000 7f3d0f7fe640 1 --2- 172.21.15.123:0/1430718113 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f3cfc073760 0x7f3cfc075c20 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f3d0400c6e0 tx=0x7f3cf8005840 comp rx=0 tx=0).ready entity=mgr.4150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:51:09.349 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:09.350+0000 7f3d0cff9640 1 -- 172.21.15.123:0/1430718113 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f3d0005e8c0 con 0x7f3cf8004810 2024-09-16T15:51:09.521 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:09.522+0000 7f3d16521640 1 -- 172.21.15.123:0/1430718113 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f3d10158ba0 con 0x7f3cf8004810 2024-09-16T15:51:09.522 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:09.522+0000 7f3d0cff9640 1 -- 172.21.15.123:0/1430718113 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v149) ==== 93+0+10 (secure 0 0 0) 0x7f3d000632f0 con 0x7f3cf8004810 2024-09-16T15:51:09.522 INFO:teuthology.orchestra.run.smithi123.stdout:pg_num: 1 2024-09-16T15:51:09.525 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:09.526+0000 7f3d16521640 1 -- 172.21.15.123:0/1430718113 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f3cfc073760 msgr2=0x7f3cfc075c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:51:09.525 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:09.526+0000 7f3d16521640 1 --2- 172.21.15.123:0/1430718113 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f3cfc073760 0x7f3cfc075c20 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f3d0400c6e0 tx=0x7f3cf8005840 comp rx=0 tx=0).stop 2024-09-16T15:51:09.525 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:09.526+0000 7f3d16521640 1 -- 172.21.15.123:0/1430718113 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3cf8004810 msgr2=0x7f3d10158610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:51:09.525 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:09.526+0000 7f3d16521640 1 --2- 172.21.15.123:0/1430718113 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3cf8004810 0x7f3d10158610 secure :-1 s=READY pgs=1693 cs=0 l=1 rev1=1 crypto rx=0x7f3d00001230 tx=0x7f3d0000cae0 comp rx=0 tx=0).stop 2024-09-16T15:51:09.525 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:09.526+0000 7f3d16521640 1 -- 172.21.15.123:0/1430718113 shutdown_connections 2024-09-16T15:51:09.525 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:09.526+0000 7f3d16521640 1 --2- 172.21.15.123:0/1430718113 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f3cfc073760 0x7f3cfc075c20 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:09.525 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:09.526+0000 7f3d16521640 1 --2- 172.21.15.123:0/1430718113 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3d10158dc0 0x7f3d10159270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:09.526 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:09.526+0000 7f3d16521640 1 --2- 172.21.15.123:0/1430718113 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f3d101578e0 0x7f3d100b3d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:09.526 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:09.526+0000 7f3d16521640 1 --2- 172.21.15.123:0/1430718113 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3cf8004810 0x7f3d10158610 unknown :-1 s=CLOSED pgs=1693 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:09.526 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:09.526+0000 7f3d16521640 1 -- 172.21.15.123:0/1430718113 >> 172.21.15.123:0/1430718113 conn(0x7f3d100b1710 msgr2=0x7f3d10153230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:51:09.526 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:09.526+0000 7f3d16521640 1 -- 172.21.15.123:0/1430718113 shutdown_connections 2024-09-16T15:51:09.526 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:09.526+0000 7f3d16521640 1 -- 172.21.15.123:0/1430718113 wait complete. 2024-09-16T15:51:09.542 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_volumes.TestMisc.test_subvolume_upgrade_v1_to_v2' 2024-09-16T15:51:09.665 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:09.666+0000 7f54069ea640 1 -- 172.21.15.123:0/1290736580 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f53ec004030 msgr2=0x7f53ec004810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:51:09.665 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:09.666+0000 7f54069ea640 1 --2- 172.21.15.123:0/1290736580 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f53ec004030 0x7f53ec004810 secure :-1 s=READY pgs=1695 cs=0 l=1 rev1=1 crypto rx=0x7f53e8006fa0 tx=0x7f53e801ce90 comp rx=0 tx=0).stop 2024-09-16T15:51:09.665 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:09.666+0000 7f5404f87640 1 -- 172.21.15.123:0/1290736580 reap_dead start 2024-09-16T15:51:09.665 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:09.666+0000 7f54069ea640 1 -- 172.21.15.123:0/1290736580 shutdown_connections 2024-09-16T15:51:09.665 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:09.666+0000 7f54069ea640 1 -- 172.21.15.123:0/1290736580 >> 172.21.15.123:0/1290736580 conn(0x7f54000d3ea0 msgr2=0x7f54000d42b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:51:09.665 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:09.666+0000 7f54069ea640 1 -- 172.21.15.123:0/1290736580 shutdown_connections 2024-09-16T15:51:09.665 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:09.666+0000 7f54069ea640 1 -- 172.21.15.123:0/1290736580 wait complete. 2024-09-16T15:51:09.666 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:09.666+0000 7f54069ea640 1 Processor -- start 2024-09-16T15:51:09.666 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:09.666+0000 7f54069ea640 1 -- start start 2024-09-16T15:51:09.668 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:09.670+0000 7f54069ea640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f53ec004030 0x7f54001f8c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:51:09.668 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:09.670+0000 7f54069ea640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f54000b8ea0 0x7f540015ea20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:51:09.668 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:09.670+0000 7f54069ea640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f54001655f0 0x7f54000c86f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:51:09.668 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:09.670+0000 7f54069ea640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f54001690e0 con 0x7f53ec004030 2024-09-16T15:51:09.668 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:09.670+0000 7f54069ea640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f5400168f60 con 0x7f54001655f0 2024-09-16T15:51:09.669 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:09.670+0000 7f54069ea640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f5400169260 con 0x7f54000b8ea0 2024-09-16T15:51:09.669 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:09.670+0000 7f53fffff640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f53ec004030 0x7f54001f8c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:51:09.669 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:09.670+0000 7f53fffff640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f53ec004030 0x7f54001f8c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:55626/0 (socket says 172.21.15.123:55626) 2024-09-16T15:51:09.669 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:09.670+0000 7f53fffff640 1 -- 172.21.15.123:0/1906744022 learned_addr learned my addr 172.21.15.123:0/1906744022 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:51:09.669 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:09.670+0000 7f53fffff640 1 -- 172.21.15.123:0/1906744022 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f54000b8ea0 msgr2=0x7f540015ea20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:51:09.669 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:09.670+0000 7f53fffff640 1 --2- 172.21.15.123:0/1906744022 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f54000b8ea0 0x7f540015ea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:09.669 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:09.670+0000 7f53fffff640 1 -- 172.21.15.123:0/1906744022 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f54001655f0 msgr2=0x7f54000c86f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:51:09.669 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:09.670+0000 7f53fffff640 1 --2- 172.21.15.123:0/1906744022 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f54001655f0 0x7f54000c86f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:09.669 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:09.670+0000 7f53fffff640 1 -- 172.21.15.123:0/1906744022 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f54001f9180 con 0x7f53ec004030 2024-09-16T15:51:09.669 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:09.670+0000 7f53fffff640 1 --2- 172.21.15.123:0/1906744022 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f53ec004030 0x7f54001f8c40 secure :-1 s=READY pgs=1696 cs=0 l=1 rev1=1 crypto rx=0x7f53f8011b40 tx=0x7f53f8009e70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:51:09.669 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:09.670+0000 7f53fcff9640 1 -- 172.21.15.123:0/1906744022 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53f801e070 con 0x7f53ec004030 2024-09-16T15:51:09.669 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:09.670+0000 7f53fcff9640 1 -- 172.21.15.123:0/1906744022 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f53f800fb90 con 0x7f53ec004030 2024-09-16T15:51:09.669 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:09.670+0000 7f54069ea640 1 -- 172.21.15.123:0/1906744022 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f54000b99e0 con 0x7f53ec004030 2024-09-16T15:51:09.670 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:09.670+0000 7f53fcff9640 1 -- 172.21.15.123:0/1906744022 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53f8019410 con 0x7f53ec004030 2024-09-16T15:51:09.670 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:09.670+0000 7f54069ea640 1 -- 172.21.15.123:0/1906744022 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f54000b9700 con 0x7f53ec004030 2024-09-16T15:51:09.670 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:09.670+0000 7f54069ea640 1 -- 172.21.15.123:0/1906744022 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f54000b0f00 con 0x7f53ec004030 2024-09-16T15:51:09.670 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:09.670+0000 7f53fcff9640 1 -- 172.21.15.123:0/1906744022 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 24) ==== 97845+0+0 (secure 0 0 0) 0x7f53f8021070 con 0x7f53ec004030 2024-09-16T15:51:09.671 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:09.670+0000 7f53fcff9640 1 --2- 172.21.15.123:0/1906744022 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f53e0073730 0x7f53e0075bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:51:09.671 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:09.670+0000 7f53ff7fe640 1 --2- 172.21.15.123:0/1906744022 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f53e0073730 0x7f53e0075bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:51:09.671 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:09.670+0000 7f53fcff9640 1 -- 172.21.15.123:0/1906744022 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(149..149 src has 1..149) ==== 4971+0+0 (secure 0 0 0) 0x7f53f8092d40 con 0x7f53ec004030 2024-09-16T15:51:09.671 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:09.670+0000 7f53ff7fe640 1 --2- 172.21.15.123:0/1906744022 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f53e0073730 0x7f53e0075bf0 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f53e80ab000 tx=0x7f53ec005840 comp rx=0 tx=0).ready entity=mgr.4150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:51:09.674 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:09.674+0000 7f53fcff9640 1 -- 172.21.15.123:0/1906744022 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f54001f9180 con 0x7f53ec004030 2024-09-16T15:51:09.905 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:09.906+0000 7f54069ea640 1 -- 172.21.15.123:0/1906744022 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_volumes.TestMisc.test_subvolume_upgrade_v1_to_v2"]} v 0) -- 0x7f540006d2d0 con 0x7f53ec004030 2024-09-16T15:51:10.195 DEBUG:teuthology.orchestra.run.smithi123:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-16T15:51:10.198 DEBUG:teuthology.orchestra.run.smithi184:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-16T15:51:10.879 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:10.878+0000 7f53fcff9640 1 -- 172.21.15.123:0/1906744022 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_volumes.TestMisc.test_subvolume_upgrade_v1_to_v2"]}]=0 v436) ==== 148+0+0 (secure 0 0 0) 0x7f53f805f3f0 con 0x7f53ec004030 2024-09-16T15:51:10.883 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:10.882+0000 7f54069ea640 1 -- 172.21.15.123:0/1906744022 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f53e0073730 msgr2=0x7f53e0075bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:51:10.883 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:10.882+0000 7f54069ea640 1 --2- 172.21.15.123:0/1906744022 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f53e0073730 0x7f53e0075bf0 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f53e80ab000 tx=0x7f53ec005840 comp rx=0 tx=0).stop 2024-09-16T15:51:10.883 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:10.882+0000 7f54069ea640 1 -- 172.21.15.123:0/1906744022 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f53ec004030 msgr2=0x7f54001f8c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:51:10.883 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:10.882+0000 7f54069ea640 1 --2- 172.21.15.123:0/1906744022 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f53ec004030 0x7f54001f8c40 secure :-1 s=READY pgs=1696 cs=0 l=1 rev1=1 crypto rx=0x7f53f8011b40 tx=0x7f53f8009e70 comp rx=0 tx=0).stop 2024-09-16T15:51:10.883 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:10.882+0000 7f54069ea640 1 -- 172.21.15.123:0/1906744022 shutdown_connections 2024-09-16T15:51:10.884 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:10.882+0000 7f54069ea640 1 --2- 172.21.15.123:0/1906744022 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f53e0073730 0x7f53e0075bf0 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:10.884 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:10.882+0000 7f54069ea640 1 --2- 172.21.15.123:0/1906744022 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f54001655f0 0x7f54000c86f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:10.884 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:10.882+0000 7f54069ea640 1 --2- 172.21.15.123:0/1906744022 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f54000b8ea0 0x7f540015ea20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:10.884 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:10.882+0000 7f54069ea640 1 --2- 172.21.15.123:0/1906744022 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f53ec004030 0x7f54001f8c40 unknown :-1 s=CLOSED pgs=1696 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:10.884 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:10.882+0000 7f54069ea640 1 -- 172.21.15.123:0/1906744022 >> 172.21.15.123:0/1906744022 conn(0x7f54000d3ea0 msgr2=0x7f540015f370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:51:10.884 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:10.886+0000 7f54069ea640 1 -- 172.21.15.123:0/1906744022 shutdown_connections 2024-09-16T15:51:10.884 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:10.886+0000 7f54069ea640 1 -- 172.21.15.123:0/1906744022 wait complete. 2024-09-16T15:51:10.904 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config dump 2024-09-16T15:51:11.033 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:11.034+0000 7fedfe601640 1 -- 172.21.15.123:0/252167947 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fedf8158120 msgr2=0x7fedf80ca320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:51:11.033 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:11.034+0000 7fedfe601640 1 --2- 172.21.15.123:0/252167947 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fedf8158120 0x7fedf80ca320 secure :-1 s=READY pgs=1698 cs=0 l=1 rev1=1 crypto rx=0x7fedf806df40 tx=0x7fede001c850 comp rx=0 tx=0).stop 2024-09-16T15:51:11.033 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:11.034+0000 7fedfe601640 1 -- 172.21.15.123:0/252167947 shutdown_connections 2024-09-16T15:51:11.033 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:11.034+0000 7fedfe601640 1 --2- 172.21.15.123:0/252167947 >> v2:172.21.15.184:3300/0 conn(0x7fedf806f5f0 0x7fedf815aa60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:11.033 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:11.034+0000 7fedfe601640 1 --2- 172.21.15.123:0/252167947 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fedf8158120 0x7fedf80ca320 unknown :-1 s=CLOSED pgs=1698 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:11.033 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:11.034+0000 7fedfe601640 1 -- 172.21.15.123:0/252167947 >> 172.21.15.123:0/252167947 conn(0x7fedf80cfb10 msgr2=0x7fedf80cff20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:51:11.033 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:11.034+0000 7fedfe601640 1 -- 172.21.15.123:0/252167947 shutdown_connections 2024-09-16T15:51:11.033 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:11.034+0000 7fedfe601640 1 -- 172.21.15.123:0/252167947 wait complete. 2024-09-16T15:51:11.034 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:11.034+0000 7fedfe601640 1 Processor -- start 2024-09-16T15:51:11.034 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:11.034+0000 7fedfe601640 1 -- start start 2024-09-16T15:51:11.036 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:11.034+0000 7fedfe601640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fedf806f5f0 0x7fedf8152580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:51:11.036 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:11.034+0000 7fedfe601640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fedf80cf5a0 0x7fedf8158900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:51:11.036 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:11.034+0000 7fedfe601640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fedf8158120 0x7fedf80b3100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:51:11.036 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:11.034+0000 7fedfe601640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fedf815b710 con 0x7fedf8158120 2024-09-16T15:51:11.036 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:11.034+0000 7fedfe601640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fedf815b590 con 0x7fedf80cf5a0 2024-09-16T15:51:11.036 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:11.034+0000 7fedfe601640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fedf815a550 con 0x7fedf806f5f0 2024-09-16T15:51:11.036 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:11.034+0000 7fedfcb9e640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fedf8158120 0x7fedf80b3100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:51:11.036 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:11.034+0000 7fedfcb9e640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fedf8158120 0x7fedf80b3100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:55656/0 (socket says 172.21.15.123:55656) 2024-09-16T15:51:11.036 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:11.034+0000 7fedfcb9e640 1 -- 172.21.15.123:0/905279971 learned_addr learned my addr 172.21.15.123:0/905279971 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:51:11.036 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:11.034+0000 7fedf7fff640 1 --2- 172.21.15.123:0/905279971 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fedf806f5f0 0x7fedf8152580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:51:11.037 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:11.038+0000 7fedf77fe640 1 --2- 172.21.15.123:0/905279971 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fedf80cf5a0 0x7fedf8158900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:51:11.037 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:11.038+0000 7fedfcb9e640 1 -- 172.21.15.123:0/905279971 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fedf806f5f0 msgr2=0x7fedf8152580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:51:11.037 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:11.038+0000 7fedfcb9e640 1 --2- 172.21.15.123:0/905279971 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fedf806f5f0 0x7fedf8152580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:11.037 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:11.038+0000 7fedfcb9e640 1 -- 172.21.15.123:0/905279971 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fedf80cf5a0 msgr2=0x7fedf8158900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:51:11.037 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:11.038+0000 7fedfcb9e640 1 --2- 172.21.15.123:0/905279971 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fedf80cf5a0 0x7fedf8158900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:11.037 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:11.038+0000 7fedfcb9e640 1 -- 172.21.15.123:0/905279971 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fedf8152ac0 con 0x7fedf8158120 2024-09-16T15:51:11.037 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:11.038+0000 7fedf77fe640 1 --2- 172.21.15.123:0/905279971 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fedf80cf5a0 0x7fedf8158900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T15:51:11.037 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:11.038+0000 7fedfcb9e640 1 --2- 172.21.15.123:0/905279971 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fedf8158120 0x7fedf80b3100 secure :-1 s=READY pgs=1699 cs=0 l=1 rev1=1 crypto rx=0x7fedf000b9e0 tx=0x7fedf000bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:51:11.037 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:11.038+0000 7fedf4ff9640 1 -- 172.21.15.123:0/905279971 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fedf0019070 con 0x7fedf8158120 2024-09-16T15:51:11.037 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:11.038+0000 7fedf4ff9640 1 -- 172.21.15.123:0/905279971 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fedf000cc30 con 0x7fedf8158120 2024-09-16T15:51:11.037 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:11.038+0000 7fedfe601640 1 -- 172.21.15.123:0/905279971 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fedf8159f90 con 0x7fedf8158120 2024-09-16T15:51:11.037 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:11.038+0000 7fedf4ff9640 1 -- 172.21.15.123:0/905279971 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fedf00146a0 con 0x7fedf8158120 2024-09-16T15:51:11.037 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:11.038+0000 7fedfe601640 1 -- 172.21.15.123:0/905279971 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fedf8155d30 con 0x7fedf8158120 2024-09-16T15:51:11.038 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:11.038+0000 7fedfe601640 1 -- 172.21.15.123:0/905279971 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fedf815aa60 con 0x7fedf8158120 2024-09-16T15:51:11.038 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:11.038+0000 7fedf4ff9640 1 -- 172.21.15.123:0/905279971 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 24) ==== 97845+0+0 (secure 0 0 0) 0x7fedf001c070 con 0x7fedf8158120 2024-09-16T15:51:11.038 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:11.038+0000 7fedf4ff9640 1 --2- 172.21.15.123:0/905279971 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fedd8073730 0x7fedd8075bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:51:11.038 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:11.038+0000 7fedf4ff9640 1 -- 172.21.15.123:0/905279971 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(149..149 src has 1..149) ==== 4971+0+0 (secure 0 0 0) 0x7fedf00922d0 con 0x7fedf8158120 2024-09-16T15:51:11.038 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:11.038+0000 7fedf7fff640 1 --2- 172.21.15.123:0/905279971 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fedd8073730 0x7fedd8075bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:51:11.039 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:11.038+0000 7fedf7fff640 1 --2- 172.21.15.123:0/905279971 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fedd8073730 0x7fedd8075bf0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7fede00af000 tx=0x7fede001b090 comp rx=0 tx=0).ready entity=mgr.4150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:51:11.042 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:11.042+0000 7fedf4ff9640 1 -- 172.21.15.123:0/905279971 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7fedf000ce20 con 0x7fedf8158120 2024-09-16T15:51:11.220 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:11.222+0000 7fedfe601640 1 -- 172.21.15.123:0/905279971 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "config dump"} v 0) -- 0x7fedf80d06c0 con 0x7fedf8158120 2024-09-16T15:51:11.220 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:11.222+0000 7fedf4ff9640 1 -- 172.21.15.123:0/905279971 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "config dump"}]=0 v45) ==== 59+0+1998 (secure 0 0 0) 0x7fedf005efd0 con 0x7fedf8158120 2024-09-16T15:51:11.221 INFO:teuthology.orchestra.run.smithi123.stdout:WHO MASK LEVEL OPTION VALUE RO 2024-09-16T15:51:11.221 INFO:teuthology.orchestra.run.smithi123.stdout:mon advanced mon_op_complaint_time 120 2024-09-16T15:51:11.221 INFO:teuthology.orchestra.run.smithi123.stdout:mon dev mon_warn_on_pool_no_app false 2024-09-16T15:51:11.221 INFO:teuthology.orchestra.run.smithi123.stdout:osd advanced osd_op_complaint_time 180.000000 2024-09-16T15:51:11.221 INFO:teuthology.orchestra.run.smithi123.stdout:osd.0 basic osd_mclock_max_capacity_iops_ssd 4805.300677 2024-09-16T15:51:11.221 INFO:teuthology.orchestra.run.smithi123.stdout:osd.1 basic osd_mclock_max_capacity_iops_ssd 4806.240607 2024-09-16T15:51:11.221 INFO:teuthology.orchestra.run.smithi123.stdout:osd.2 basic osd_mclock_max_capacity_iops_ssd 4046.441686 2024-09-16T15:51:11.221 INFO:teuthology.orchestra.run.smithi123.stdout:osd.3 basic osd_mclock_max_capacity_iops_ssd 6882.576123 2024-09-16T15:51:11.221 INFO:teuthology.orchestra.run.smithi123.stdout:osd.4 basic osd_mclock_max_capacity_iops_ssd 6108.444971 2024-09-16T15:51:11.221 INFO:teuthology.orchestra.run.smithi123.stdout:osd.5 basic osd_mclock_max_capacity_iops_ssd 4995.096306 2024-09-16T15:51:11.222 INFO:teuthology.orchestra.run.smithi123.stdout:osd.6 basic osd_mclock_max_capacity_iops_ssd 6237.822095 2024-09-16T15:51:11.222 INFO:teuthology.orchestra.run.smithi123.stdout:osd.7 basic osd_mclock_max_capacity_iops_ssd 5202.385138 2024-09-16T15:51:11.222 INFO:teuthology.orchestra.run.smithi123.stdout:mds advanced debug_mds 20/20 2024-09-16T15:51:11.222 INFO:teuthology.orchestra.run.smithi123.stdout:mds advanced debug_mds_balancer 20/20 2024-09-16T15:51:11.222 INFO:teuthology.orchestra.run.smithi123.stdout:mds advanced debug_ms 2/2 2024-09-16T15:51:11.222 INFO:teuthology.orchestra.run.smithi123.stdout:mds dev mds_debug_frag true 2024-09-16T15:51:11.222 INFO:teuthology.orchestra.run.smithi123.stdout:mds dev mds_debug_scatterstat true 2024-09-16T15:51:11.222 INFO:teuthology.orchestra.run.smithi123.stdout:mds advanced mds_op_complaint_time 180.000000 2024-09-16T15:51:11.222 INFO:teuthology.orchestra.run.smithi123.stdout:mds dev mds_verify_scatter true 2024-09-16T15:51:11.222 INFO:teuthology.orchestra.run.smithi123.stdout:mds advanced osd_op_complaint_time 180.000000 2024-09-16T15:51:11.222 INFO:teuthology.orchestra.run.smithi123.stdout:mds advanced rados_mon_op_timeout 900 2024-09-16T15:51:11.222 INFO:teuthology.orchestra.run.smithi123.stdout:mds advanced rados_osd_op_timeout 900 2024-09-16T15:51:11.222 INFO:teuthology.orchestra.run.smithi123.stdout:client advanced client_mount_timeout 600 2024-09-16T15:51:11.222 INFO:teuthology.orchestra.run.smithi123.stdout:client advanced debug_client 20/20 2024-09-16T15:51:11.223 INFO:teuthology.orchestra.run.smithi123.stdout:client advanced debug_ms 1/1 2024-09-16T15:51:11.223 INFO:teuthology.orchestra.run.smithi123.stdout:client advanced rados_mon_op_timeout 900 2024-09-16T15:51:11.223 INFO:teuthology.orchestra.run.smithi123.stdout:client advanced rados_osd_op_timeout 900 2024-09-16T15:51:11.223 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:11.222+0000 7fedfe601640 1 -- 172.21.15.123:0/905279971 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fedd8073730 msgr2=0x7fedd8075bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:51:11.224 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:11.222+0000 7fedfe601640 1 --2- 172.21.15.123:0/905279971 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fedd8073730 0x7fedd8075bf0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7fede00af000 tx=0x7fede001b090 comp rx=0 tx=0).stop 2024-09-16T15:51:11.224 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:11.222+0000 7fedfe601640 1 -- 172.21.15.123:0/905279971 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fedf8158120 msgr2=0x7fedf80b3100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:51:11.224 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:11.222+0000 7fedfe601640 1 --2- 172.21.15.123:0/905279971 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fedf8158120 0x7fedf80b3100 secure :-1 s=READY pgs=1699 cs=0 l=1 rev1=1 crypto rx=0x7fedf000b9e0 tx=0x7fedf000bea0 comp rx=0 tx=0).stop 2024-09-16T15:51:11.224 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:11.222+0000 7fedfe601640 1 -- 172.21.15.123:0/905279971 shutdown_connections 2024-09-16T15:51:11.224 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:11.222+0000 7fedfe601640 1 --2- 172.21.15.123:0/905279971 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fedd8073730 0x7fedd8075bf0 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:11.224 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:11.222+0000 7fedfe601640 1 --2- 172.21.15.123:0/905279971 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fedf8158120 0x7fedf80b3100 unknown :-1 s=CLOSED pgs=1699 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:11.224 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:11.222+0000 7fedfe601640 1 --2- 172.21.15.123:0/905279971 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fedf80cf5a0 0x7fedf8158900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:11.224 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:11.222+0000 7fedfe601640 1 --2- 172.21.15.123:0/905279971 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fedf806f5f0 0x7fedf8152580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:11.224 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:11.222+0000 7fedfe601640 1 -- 172.21.15.123:0/905279971 >> 172.21.15.123:0/905279971 conn(0x7fedf80cfb10 msgr2=0x7fedf80cc450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:51:11.224 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:11.222+0000 7fedfe601640 1 -- 172.21.15.123:0/905279971 shutdown_connections 2024-09-16T15:51:11.224 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:11.226+0000 7fedfe601640 1 -- 172.21.15.123:0/905279971 wait complete. 2024-09-16T15:51:11.240 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-09-16T15:51:11.369 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:11.370+0000 7f61d59f8640 1 -- 172.21.15.123:0/2291257419 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f61b8005190 msgr2=0x7f61b800b9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:51:11.369 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:11.370+0000 7f61d59f8640 1 --2- 172.21.15.123:0/2291257419 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f61b8005190 0x7f61b800b9d0 secure :-1 s=READY pgs=1701 cs=0 l=1 rev1=1 crypto rx=0x7f61c8006fd0 tx=0x7f61c801cae0 comp rx=0 tx=0).stop 2024-09-16T15:51:11.369 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:11.370+0000 7f61cf7fe640 1 -- 172.21.15.123:0/2291257419 reap_dead start 2024-09-16T15:51:11.369 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:11.370+0000 7f61d59f8640 1 -- 172.21.15.123:0/2291257419 shutdown_connections 2024-09-16T15:51:11.369 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:11.370+0000 7f61d59f8640 1 -- 172.21.15.123:0/2291257419 >> 172.21.15.123:0/2291257419 conn(0x7f61d00741a0 msgr2=0x7f61d00745b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:51:11.369 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:11.370+0000 7f61d59f8640 1 -- 172.21.15.123:0/2291257419 shutdown_connections 2024-09-16T15:51:11.370 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:11.370+0000 7f61d59f8640 1 -- 172.21.15.123:0/2291257419 wait complete. 2024-09-16T15:51:11.370 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:11.370+0000 7f61d59f8640 1 Processor -- start 2024-09-16T15:51:11.370 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:11.370+0000 7f61d59f8640 1 -- start start 2024-09-16T15:51:11.372 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:11.374+0000 7f61d59f8640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f61b8005190 0x7f61d00b58f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:51:11.372 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:11.374+0000 7f61d59f8640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f61d00c9ee0 0x7f61d01f31b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:51:11.372 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:11.374+0000 7f61d59f8640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f61d0159db0 0x7f61d009c930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:51:11.372 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:11.374+0000 7f61d59f8640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f61d00cbda0 con 0x7f61b8005190 2024-09-16T15:51:11.372 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:11.374+0000 7f61d59f8640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f61d00cbc20 con 0x7f61d0159db0 2024-09-16T15:51:11.372 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:11.374+0000 7f61d59f8640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f61d00cbf20 con 0x7f61d00c9ee0 2024-09-16T15:51:11.373 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:11.374+0000 7f61ceffd640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f61b8005190 0x7f61d00b58f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:51:11.373 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:11.374+0000 7f61ceffd640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f61b8005190 0x7f61d00b58f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:55674/0 (socket says 172.21.15.123:55674) 2024-09-16T15:51:11.373 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:11.374+0000 7f61ceffd640 1 -- 172.21.15.123:0/1582224588 learned_addr learned my addr 172.21.15.123:0/1582224588 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:51:11.373 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:11.374+0000 7f61ce7fc640 1 --2- 172.21.15.123:0/1582224588 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f61d00c9ee0 0x7f61d01f31b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:51:11.373 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:11.374+0000 7f61ceffd640 1 -- 172.21.15.123:0/1582224588 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f61d00c9ee0 msgr2=0x7f61d01f31b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:51:11.373 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:11.374+0000 7f61ceffd640 1 --2- 172.21.15.123:0/1582224588 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f61d00c9ee0 0x7f61d01f31b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:11.373 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:11.374+0000 7f61cf7fe640 1 --2- 172.21.15.123:0/1582224588 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f61d0159db0 0x7f61d009c930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:51:11.373 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:11.374+0000 7f61ceffd640 1 -- 172.21.15.123:0/1582224588 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f61d0159db0 msgr2=0x7f61d009c930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:51:11.373 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:11.374+0000 7f61ceffd640 1 --2- 172.21.15.123:0/1582224588 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f61d0159db0 0x7f61d009c930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:11.373 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:11.374+0000 7f61ceffd640 1 -- 172.21.15.123:0/1582224588 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f61d01f64f0 con 0x7f61b8005190 2024-09-16T15:51:11.373 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:11.374+0000 7f61ceffd640 1 --2- 172.21.15.123:0/1582224588 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f61b8005190 0x7f61d00b58f0 secure :-1 s=READY pgs=1702 cs=0 l=1 rev1=1 crypto rx=0x7f61c000d730 tx=0x7f61c000dbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:51:11.373 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:11.374+0000 7f61b7fff640 1 -- 172.21.15.123:0/1582224588 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61c0018070 con 0x7f61b8005190 2024-09-16T15:51:11.374 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:11.374+0000 7f61b7fff640 1 -- 172.21.15.123:0/1582224588 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f61c000de40 con 0x7f61b8005190 2024-09-16T15:51:11.374 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:11.374+0000 7f61d59f8640 1 -- 172.21.15.123:0/1582224588 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f61d009ddb0 con 0x7f61b8005190 2024-09-16T15:51:11.374 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:11.374+0000 7f61b7fff640 1 -- 172.21.15.123:0/1582224588 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61c000b6c0 con 0x7f61b8005190 2024-09-16T15:51:11.374 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:11.374+0000 7f61d59f8640 1 -- 172.21.15.123:0/1582224588 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f61d01f36f0 con 0x7f61b8005190 2024-09-16T15:51:11.374 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:11.374+0000 7f61d59f8640 1 -- 172.21.15.123:0/1582224588 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f61d009c560 con 0x7f61b8005190 2024-09-16T15:51:11.374 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:11.374+0000 7f61b7fff640 1 -- 172.21.15.123:0/1582224588 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 24) ==== 97845+0+0 (secure 0 0 0) 0x7f61c001b070 con 0x7f61b8005190 2024-09-16T15:51:11.375 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:11.374+0000 7f61b7fff640 1 --2- 172.21.15.123:0/1582224588 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f61bc073730 0x7f61bc075bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:51:11.375 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:11.374+0000 7f61b7fff640 1 -- 172.21.15.123:0/1582224588 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(149..149 src has 1..149) ==== 4971+0+0 (secure 0 0 0) 0x7f61c0091de0 con 0x7f61b8005190 2024-09-16T15:51:11.375 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:11.374+0000 7f61ce7fc640 1 --2- 172.21.15.123:0/1582224588 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f61bc073730 0x7f61bc075bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:51:11.375 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:11.374+0000 7f61ce7fc640 1 --2- 172.21.15.123:0/1582224588 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f61bc073730 0x7f61bc075bf0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f61c40058d0 tx=0x7f61c4005860 comp rx=0 tx=0).ready entity=mgr.4150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:51:11.378 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:11.378+0000 7f61b7fff640 1 -- 172.21.15.123:0/1582224588 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f61c005e380 con 0x7f61b8005190 2024-09-16T15:51:11.551 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:11.550+0000 7f61d59f8640 1 -- 172.21.15.123:0/1582224588 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) -- 0x7f61d015bdd0 con 0x7f61b8005190 2024-09-16T15:51:11.557 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:11.558+0000 7f61b7fff640 1 -- 172.21.15.123:0/1582224588 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v46)=0 v46) ==== 122+0+0 (secure 0 0 0) 0x7f61c0062db0 con 0x7f61b8005190 2024-09-16T15:51:11.560 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:11.562+0000 7f61d59f8640 1 -- 172.21.15.123:0/1582224588 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f61bc073730 msgr2=0x7f61bc075bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:51:11.560 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:11.562+0000 7f61d59f8640 1 --2- 172.21.15.123:0/1582224588 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f61bc073730 0x7f61bc075bf0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f61c40058d0 tx=0x7f61c4005860 comp rx=0 tx=0).stop 2024-09-16T15:51:11.560 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:11.562+0000 7f61d59f8640 1 -- 172.21.15.123:0/1582224588 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f61b8005190 msgr2=0x7f61d00b58f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:51:11.560 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:11.562+0000 7f61d59f8640 1 --2- 172.21.15.123:0/1582224588 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f61b8005190 0x7f61d00b58f0 secure :-1 s=READY pgs=1702 cs=0 l=1 rev1=1 crypto rx=0x7f61c000d730 tx=0x7f61c000dbf0 comp rx=0 tx=0).stop 2024-09-16T15:51:11.560 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:11.562+0000 7f61d59f8640 1 -- 172.21.15.123:0/1582224588 shutdown_connections 2024-09-16T15:51:11.560 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:11.562+0000 7f61d59f8640 1 --2- 172.21.15.123:0/1582224588 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f61bc073730 0x7f61bc075bf0 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:11.560 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:11.562+0000 7f61d59f8640 1 --2- 172.21.15.123:0/1582224588 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f61d0159db0 0x7f61d009c930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:11.560 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:11.562+0000 7f61d59f8640 1 --2- 172.21.15.123:0/1582224588 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f61d00c9ee0 0x7f61d01f31b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:11.560 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:11.562+0000 7f61d59f8640 1 --2- 172.21.15.123:0/1582224588 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f61b8005190 0x7f61d00b58f0 unknown :-1 s=CLOSED pgs=1702 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:11.561 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:11.562+0000 7f61d59f8640 1 -- 172.21.15.123:0/1582224588 >> 172.21.15.123:0/1582224588 conn(0x7f61d00741a0 msgr2=0x7f61d00ad960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:51:11.561 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:11.562+0000 7f61d59f8640 1 -- 172.21.15.123:0/1582224588 shutdown_connections 2024-09-16T15:51:11.561 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:11.562+0000 7f61d59f8640 1 -- 172.21.15.123:0/1582224588 wait complete. 2024-09-16T15:51:11.577 INFO:teuthology.nuke:Clearing teuthology firewall rules... 2024-09-16T15:51:11.577 DEBUG:teuthology.orchestra.run.smithi123:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-09-16T15:51:11.656 DEBUG:teuthology.orchestra.run.smithi184:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-09-16T15:51:11.737 INFO:teuthology.nuke:Cleared teuthology firewall rules. 2024-09-16T15:51:11.738 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2024-09-16T15:51:11.738 DEBUG:teuthology.orchestra.run.smithi123:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-16T15:51:11.744 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16336468k,nr_inodes=4084117,mode=755,inode64 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3274684k,mode=755,inode64 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev,inode64 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=17082 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3274680k,nr_inodes=818670,mode=700,uid=1000,gid=1271,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3274684k,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-09-16T15:51:11.744 DEBUG:teuthology.orchestra.run.smithi184:> set -ex 2024-09-16T15:51:11.744 DEBUG:teuthology.orchestra.run.smithi184:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-16T15:51:11.752 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16336488k,nr_inodes=4084122,mode=755,inode64 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3274688k,mode=755,inode64 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev,inode64 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=17834 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3274684k,nr_inodes=818671,mode=700,uid=1000,gid=1271,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3274688k,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-09-16T15:51:11.752 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2024-09-16T15:51:11.752 DEBUG:teuthology.orchestra.run.smithi123:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-16T15:51:11.792 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16336468k,nr_inodes=4084117,mode=755,inode64 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3274684k,mode=755,inode64 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev,inode64 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=17082 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3274680k,nr_inodes=818670,mode=700,uid=1000,gid=1271,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3274684k,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-09-16T15:51:11.792 DEBUG:teuthology.orchestra.run.smithi184:> set -ex 2024-09-16T15:51:11.792 DEBUG:teuthology.orchestra.run.smithi184:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-16T15:51:11.800 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16336488k,nr_inodes=4084122,mode=755,inode64 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3274688k,mode=755,inode64 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev,inode64 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=17834 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3274684k,nr_inodes=818671,mode=700,uid=1000,gid=1271,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3274688k,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-09-16T15:51:11.800 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-09-16T15:51:11.801 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-16T15:51:11.969 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:11.970+0000 7fba92549640 1 -- 172.21.15.123:0/826963836 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fba8c158120 msgr2=0x7fba8c0ca320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:51:12.001 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:11.970+0000 7fba92549640 1 --2- 172.21.15.123:0/826963836 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fba8c158120 0x7fba8c0ca320 secure :-1 s=READY pgs=1704 cs=0 l=1 rev1=1 crypto rx=0x7fba7c00a950 tx=0x7fba7c01dae0 comp rx=0 tx=0).stop 2024-09-16T15:51:12.001 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:11.970+0000 7fba92549640 1 -- 172.21.15.123:0/826963836 shutdown_connections 2024-09-16T15:51:12.001 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:11.970+0000 7fba92549640 1 --2- 172.21.15.123:0/826963836 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fba8c158120 0x7fba8c0ca320 unknown :-1 s=CLOSED pgs=1704 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:12.001 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:11.970+0000 7fba92549640 1 -- 172.21.15.123:0/826963836 >> 172.21.15.123:0/826963836 conn(0x7fba8c0cfb10 msgr2=0x7fba8c0cff20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:51:12.001 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:11.970+0000 7fba92549640 1 -- 172.21.15.123:0/826963836 shutdown_connections 2024-09-16T15:51:12.001 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:11.970+0000 7fba92549640 1 -- 172.21.15.123:0/826963836 wait complete. 2024-09-16T15:51:12.003 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:11.970+0000 7fba92549640 1 Processor -- start 2024-09-16T15:51:12.004 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:11.970+0000 7fba92549640 1 -- start start 2024-09-16T15:51:12.004 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:11.974+0000 7fba92549640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fba8c06f5f0 0x7fba8c153b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:51:12.004 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:11.974+0000 7fba92549640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fba8c0cf5a0 0x7fba8c15d010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:51:12.004 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:11.974+0000 7fba92549640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fba8c158120 0x7fba8c15d550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:51:12.004 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:11.974+0000 7fba92549640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fba8c1683e0 con 0x7fba8c06f5f0 2024-09-16T15:51:12.004 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:11.974+0000 7fba92549640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fba8c15b830 con 0x7fba8c0cf5a0 2024-09-16T15:51:12.004 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:11.974+0000 7fba92549640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fba8c168560 con 0x7fba8c158120 2024-09-16T15:51:12.004 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:11.974+0000 7fba8bfff640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fba8c06f5f0 0x7fba8c153b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:51:12.004 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:11.974+0000 7fba8bfff640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fba8c06f5f0 0x7fba8c153b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:55686/0 (socket says 172.21.15.123:55686) 2024-09-16T15:51:12.004 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:11.974+0000 7fba8bfff640 1 -- 172.21.15.123:0/2503850213 learned_addr learned my addr 172.21.15.123:0/2503850213 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:51:12.004 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:11.974+0000 7fba8bfff640 1 -- 172.21.15.123:0/2503850213 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fba8c158120 msgr2=0x7fba8c15d550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:51:12.004 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:11.974+0000 7fba90ae6640 1 --2- 172.21.15.123:0/2503850213 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fba8c158120 0x7fba8c15d550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:51:12.005 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:11.974+0000 7fba8b7fe640 1 --2- 172.21.15.123:0/2503850213 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fba8c0cf5a0 0x7fba8c15d010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:51:12.005 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:11.974+0000 7fba8bfff640 1 --2- 172.21.15.123:0/2503850213 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fba8c158120 0x7fba8c15d550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:12.005 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:11.974+0000 7fba8bfff640 1 -- 172.21.15.123:0/2503850213 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fba8c0cf5a0 msgr2=0x7fba8c15d010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:51:12.005 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:11.974+0000 7fba8bfff640 1 --2- 172.21.15.123:0/2503850213 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fba8c0cf5a0 0x7fba8c15d010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:12.005 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:11.974+0000 7fba8bfff640 1 -- 172.21.15.123:0/2503850213 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fba8c1540b0 con 0x7fba8c06f5f0 2024-09-16T15:51:12.005 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:11.974+0000 7fba8bfff640 1 --2- 172.21.15.123:0/2503850213 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fba8c06f5f0 0x7fba8c153b70 secure :-1 s=READY pgs=1705 cs=0 l=1 rev1=1 crypto rx=0x7fba7c01f1c0 tx=0x7fba7c005ca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:51:12.005 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:11.974+0000 7fba88ff9640 1 -- 172.21.15.123:0/2503850213 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba7c0be070 con 0x7fba8c06f5f0 2024-09-16T15:51:12.005 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:11.974+0000 7fba92549640 1 -- 172.21.15.123:0/2503850213 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fba8c165260 con 0x7fba8c06f5f0 2024-09-16T15:51:12.005 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:11.974+0000 7fba92549640 1 -- 172.21.15.123:0/2503850213 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fba8c165820 con 0x7fba8c06f5f0 2024-09-16T15:51:12.005 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:11.974+0000 7fba88ff9640 1 -- 172.21.15.123:0/2503850213 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fba7c0038f0 con 0x7fba8c06f5f0 2024-09-16T15:51:12.005 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:11.974+0000 7fba88ff9640 1 -- 172.21.15.123:0/2503850213 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba7c0b3040 con 0x7fba8c06f5f0 2024-09-16T15:51:12.005 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:11.974+0000 7fba92549640 1 -- 172.21.15.123:0/2503850213 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fba6c0052a0 con 0x7fba8c06f5f0 2024-09-16T15:51:12.005 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:11.974+0000 7fba88ff9640 1 -- 172.21.15.123:0/2503850213 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 24) ==== 97845+0+0 (secure 0 0 0) 0x7fba7c003ca0 con 0x7fba8c06f5f0 2024-09-16T15:51:12.005 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:11.974+0000 7fba88ff9640 1 --2- 172.21.15.123:0/2503850213 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fba78073750 0x7fba78075c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:51:12.006 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:11.974+0000 7fba8b7fe640 1 --2- 172.21.15.123:0/2503850213 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fba78073750 0x7fba78075c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:51:12.006 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:11.974+0000 7fba88ff9640 1 -- 172.21.15.123:0/2503850213 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(149..149 src has 1..149) ==== 4971+0+0 (secure 0 0 0) 0x7fba7c12cb50 con 0x7fba8c06f5f0 2024-09-16T15:51:12.006 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:11.974+0000 7fba8b7fe640 1 --2- 172.21.15.123:0/2503850213 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fba78073750 0x7fba78075c10 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7fba8c073aa0 tx=0x7fba80009040 comp rx=0 tx=0).ready entity=mgr.4150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:51:12.006 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:11.978+0000 7fba88ff9640 1 -- 172.21.15.123:0/2503850213 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7fba7c017760 con 0x7fba8c06f5f0 2024-09-16T15:51:12.180 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:12.182+0000 7fba92549640 1 -- 172.21.15.123:0/2503850213 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fba6c005860 con 0x7fba8c06f5f0 2024-09-16T15:51:12.180 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:12.182+0000 7fba88ff9640 1 -- 172.21.15.123:0/2503850213 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 98 v98) ==== 94+0+3920 (secure 0 0 0) 0x7fba7c0f97d0 con 0x7fba8c06f5f0 2024-09-16T15:51:12.181 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-09-16T15:51:12.181 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":98,"btime":"2024-09-16T15:51:05:851343+0000","default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6155,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6833/1845295626","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":1845295626},{"type":"v1","addr":"172.21.15.184:6833","nonce":1845295626}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":98},{"gid":6580,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6835/430907849","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":430907849},{"type":"v1","addr":"172.21.15.184:6835","nonce":430907849}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":91},{"gid":8424,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6833/1494334556","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6832","nonce":1494334556},{"type":"v1","addr":"172.21.15.123:6833","nonce":1494334556}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":90},{"gid":8427,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6835/2168742484","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6834","nonce":2168742484},{"type":"v1","addr":"172.21.15.123:6835","nonce":2168742484}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":91}],"filesystems":[]} 2024-09-16T15:51:12.181 INFO:teuthology.orchestra.run.smithi123.stderr:dumped fsmap epoch 98 2024-09-16T15:51:12.183 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:12.182+0000 7fba92549640 1 -- 172.21.15.123:0/2503850213 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fba78073750 msgr2=0x7fba78075c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:51:12.183 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:12.182+0000 7fba92549640 1 --2- 172.21.15.123:0/2503850213 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fba78073750 0x7fba78075c10 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7fba8c073aa0 tx=0x7fba80009040 comp rx=0 tx=0).stop 2024-09-16T15:51:12.183 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:12.182+0000 7fba92549640 1 -- 172.21.15.123:0/2503850213 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fba8c06f5f0 msgr2=0x7fba8c153b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:51:12.183 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:12.182+0000 7fba92549640 1 --2- 172.21.15.123:0/2503850213 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fba8c06f5f0 0x7fba8c153b70 secure :-1 s=READY pgs=1705 cs=0 l=1 rev1=1 crypto rx=0x7fba7c01f1c0 tx=0x7fba7c005ca0 comp rx=0 tx=0).stop 2024-09-16T15:51:12.183 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:12.182+0000 7fba92549640 1 -- 172.21.15.123:0/2503850213 shutdown_connections 2024-09-16T15:51:12.183 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:12.182+0000 7fba92549640 1 --2- 172.21.15.123:0/2503850213 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fba78073750 0x7fba78075c10 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:12.183 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:12.182+0000 7fba92549640 1 --2- 172.21.15.123:0/2503850213 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fba8c158120 0x7fba8c15d550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:12.184 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:12.182+0000 7fba92549640 1 --2- 172.21.15.123:0/2503850213 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fba8c0cf5a0 0x7fba8c15d010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:12.184 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:12.182+0000 7fba92549640 1 --2- 172.21.15.123:0/2503850213 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fba8c06f5f0 0x7fba8c153b70 unknown :-1 s=CLOSED pgs=1705 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:12.184 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:12.182+0000 7fba92549640 1 -- 172.21.15.123:0/2503850213 >> 172.21.15.123:0/2503850213 conn(0x7fba8c0cfb10 msgr2=0x7fba8c075aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:51:12.184 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:12.182+0000 7fba92549640 1 -- 172.21.15.123:0/2503850213 shutdown_connections 2024-09-16T15:51:12.184 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:12.182+0000 7fba92549640 1 -- 172.21.15.123:0/2503850213 wait complete. 2024-09-16T15:51:12.199 INFO:tasks.ceph.mds.a:Restarting daemon 2024-09-16T15:51:12.199 INFO:tasks.ceph.mds.a:Stopping old one... 2024-09-16T15:51:12.200 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-09-16T15:51:12.200 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T15:51:12.200 INFO:tasks.ceph.mds.c:Restarting daemon 2024-09-16T15:51:12.200 INFO:tasks.ceph.mds.c:Stopping old one... 2024-09-16T15:51:12.200 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-09-16T15:51:12.200 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T15:51:12.200 INFO:tasks.ceph.mds.b:Restarting daemon 2024-09-16T15:51:12.201 INFO:tasks.ceph.mds.b:Stopping old one... 2024-09-16T15:51:12.201 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-09-16T15:51:12.201 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T15:51:12.201 INFO:tasks.ceph.mds.d:Restarting daemon 2024-09-16T15:51:12.201 INFO:tasks.ceph.mds.d:Stopping old one... 2024-09-16T15:51:12.201 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-09-16T15:51:12.202 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T15:51:12.210 INFO:tasks.ceph.mds.a:Stopped 2024-09-16T15:51:12.210 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-09-16T15:51:12.211 INFO:tasks.ceph.mds.c:Stopped 2024-09-16T15:51:12.211 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-09-16T15:51:12.214 INFO:tasks.ceph.mds.d:Stopped 2024-09-16T15:51:12.214 DEBUG:teuthology.orchestra.run.smithi184:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-09-16T15:51:12.216 INFO:tasks.ceph.mds.b:Stopped 2024-09-16T15:51:12.217 DEBUG:teuthology.orchestra.run.smithi184:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-09-16T15:51:12.256 INFO:tasks.ceph.mds.a:Started 2024-09-16T15:51:12.256 DEBUG:teuthology.parallel:result is None 2024-09-16T15:51:12.258 INFO:tasks.ceph.mds.c:Started 2024-09-16T15:51:12.258 DEBUG:teuthology.parallel:result is None 2024-09-16T15:51:12.264 INFO:tasks.ceph.mds.d:Started 2024-09-16T15:51:12.265 DEBUG:teuthology.parallel:result is None 2024-09-16T15:51:12.266 INFO:tasks.ceph.mds.b:Started 2024-09-16T15:51:12.267 DEBUG:teuthology.parallel:result is None 2024-09-16T15:51:12.267 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-09-16T15:51:12.321 INFO:tasks.ceph.mds.c.smithi123.stdout:starting mds.c at 2024-09-16T15:51:12.321 INFO:tasks.ceph.mds.a.smithi123.stdout:starting mds.a at 2024-09-16T15:51:12.337 INFO:tasks.ceph.mds.b.smithi184.stdout:starting mds.b at 2024-09-16T15:51:12.337 INFO:tasks.ceph.mds.d.smithi184.stdout:starting mds.d at 2024-09-16T15:51:12.379 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:12.382+0000 7f634d686640 1 -- 172.21.15.123:0/507751860 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f63480b2b00 msgr2=0x7f63480d24b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:51:12.379 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:12.382+0000 7f634d686640 1 --2- 172.21.15.123:0/507751860 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f63480b2b00 0x7f63480d24b0 secure :-1 s=READY pgs=1716 cs=0 l=1 rev1=1 crypto rx=0x7f634808d220 tx=0x7f633c01c6b0 comp rx=0 tx=0).stop 2024-09-16T15:51:12.379 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:12.382+0000 7f634d686640 1 -- 172.21.15.123:0/507751860 shutdown_connections 2024-09-16T15:51:12.380 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:12.382+0000 7f634d686640 1 --2- 172.21.15.123:0/507751860 >> v2:172.21.15.184:3300/0 conn(0x7f63480b0a50 0x7f63480b0ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:12.380 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:12.382+0000 7f634d686640 1 --2- 172.21.15.123:0/507751860 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f63480b2b00 0x7f63480d24b0 unknown :-1 s=CLOSED pgs=1716 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:12.380 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:12.382+0000 7f634d686640 1 -- 172.21.15.123:0/507751860 >> 172.21.15.123:0/507751860 conn(0x7f63480d3ea0 msgr2=0x7f63480d42b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:51:12.380 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:12.382+0000 7f634d686640 1 -- 172.21.15.123:0/507751860 shutdown_connections 2024-09-16T15:51:12.380 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:12.382+0000 7f634d686640 1 -- 172.21.15.123:0/507751860 wait complete. 2024-09-16T15:51:12.380 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:12.382+0000 7f634d686640 1 Processor -- start 2024-09-16T15:51:12.380 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:12.382+0000 7f634d686640 1 -- start start 2024-09-16T15:51:12.382 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:12.382+0000 7f634d686640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f63480b0a50 0x7f63481515b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:51:12.382 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:12.382+0000 7f634d686640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f63480b2b00 0x7f6348152cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:51:12.382 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:12.382+0000 7f634d686640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6348161620 0x7f63480b5920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:51:12.382 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:12.382+0000 7f634d686640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f634816b0f0 con 0x7f63480b2b00 2024-09-16T15:51:12.382 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:12.382+0000 7f634d686640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f634816af70 con 0x7f6348161620 2024-09-16T15:51:12.382 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:12.382+0000 7f634d686640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f634816b270 con 0x7f63480b0a50 2024-09-16T15:51:12.383 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:12.382+0000 7f63467fc640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f63480b2b00 0x7f6348152cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:51:12.383 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:12.382+0000 7f63467fc640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f63480b2b00 0x7f6348152cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:55738/0 (socket says 172.21.15.123:55738) 2024-09-16T15:51:12.383 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:12.382+0000 7f63467fc640 1 -- 172.21.15.123:0/2179313832 learned_addr learned my addr 172.21.15.123:0/2179313832 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:51:12.383 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:12.382+0000 7f63477fe640 1 --2- 172.21.15.123:0/2179313832 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6348161620 0x7f63480b5920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:51:12.383 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:12.382+0000 7f6346ffd640 1 --2- 172.21.15.123:0/2179313832 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f63480b0a50 0x7f63481515b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:51:12.383 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:12.382+0000 7f63467fc640 1 -- 172.21.15.123:0/2179313832 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f63480b0a50 msgr2=0x7f63481515b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:51:12.383 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:12.382+0000 7f63467fc640 1 --2- 172.21.15.123:0/2179313832 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f63480b0a50 0x7f63481515b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:12.383 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:12.382+0000 7f63467fc640 1 -- 172.21.15.123:0/2179313832 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6348161620 msgr2=0x7f63480b5920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:51:12.383 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:12.382+0000 7f63467fc640 1 --2- 172.21.15.123:0/2179313832 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6348161620 0x7f63480b5920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:12.383 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:12.382+0000 7f63467fc640 1 -- 172.21.15.123:0/2179313832 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f63481537b0 con 0x7f63480b2b00 2024-09-16T15:51:12.383 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:12.382+0000 7f6346ffd640 1 --2- 172.21.15.123:0/2179313832 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f63480b0a50 0x7f63481515b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T15:51:12.383 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:12.382+0000 7f63467fc640 1 --2- 172.21.15.123:0/2179313832 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f63480b2b00 0x7f6348152cf0 secure :-1 s=READY pgs=1717 cs=0 l=1 rev1=1 crypto rx=0x7f633c01cb90 tx=0x7f633c0a6620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:51:12.383 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:12.382+0000 7f632f7fe640 1 -- 172.21.15.123:0/2179313832 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f633c0be020 con 0x7f63480b2b00 2024-09-16T15:51:12.384 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:12.382+0000 7f632f7fe640 1 -- 172.21.15.123:0/2179313832 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f633c004330 con 0x7f63480b2b00 2024-09-16T15:51:12.384 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:12.382+0000 7f634d686640 1 -- 172.21.15.123:0/2179313832 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6348153230 con 0x7f63480b2b00 2024-09-16T15:51:12.384 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:12.382+0000 7f632f7fe640 1 -- 172.21.15.123:0/2179313832 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f633c003840 con 0x7f63480b2b00 2024-09-16T15:51:12.384 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:12.382+0000 7f634d686640 1 -- 172.21.15.123:0/2179313832 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f63481f7c90 con 0x7f63480b2b00 2024-09-16T15:51:12.385 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:12.382+0000 7f634d686640 1 -- 172.21.15.123:0/2179313832 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f63480b0ea0 con 0x7f63480b2b00 2024-09-16T15:51:12.385 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:12.386+0000 7f632f7fe640 1 -- 172.21.15.123:0/2179313832 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 24) ==== 97845+0+0 (secure 0 0 0) 0x7f633c0c1020 con 0x7f63480b2b00 2024-09-16T15:51:12.385 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:12.386+0000 7f632f7fe640 1 --2- 172.21.15.123:0/2179313832 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f6334073730 0x7f6334075bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:51:12.385 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:12.386+0000 7f6346ffd640 1 --2- 172.21.15.123:0/2179313832 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f6334073730 0x7f6334075bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:51:12.385 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:12.386+0000 7f632f7fe640 1 -- 172.21.15.123:0/2179313832 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(149..149 src has 1..149) ==== 4971+0+0 (secure 0 0 0) 0x7f633c12d7f0 con 0x7f63480b2b00 2024-09-16T15:51:12.385 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:12.386+0000 7f6346ffd640 1 --2- 172.21.15.123:0/2179313832 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f6334073730 0x7f6334075bf0 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f6348156ef0 tx=0x7f6338009040 comp rx=0 tx=0).ready entity=mgr.4150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:51:12.388 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:12.390+0000 7f632f7fe640 1 -- 172.21.15.123:0/2179313832 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f633c0c8030 con 0x7f63480b2b00 2024-09-16T15:51:12.571 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:12.570+0000 7f634d686640 1 -- 172.21.15.123:0/2179313832 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) -- 0x7f63480d33d0 con 0x7f63480b2b00 2024-09-16T15:51:12.902 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:12.902+0000 7f632f7fe640 1 -- 172.21.15.123:0/2179313832 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v151) ==== 97+0+0 (secure 0 0 0) 0x7f633c0163f0 con 0x7f63480b2b00 2024-09-16T15:51:12.902 INFO:teuthology.orchestra.run.smithi123.stderr: removed all blocklist entries 2024-09-16T15:51:12.905 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:12.906+0000 7f634d686640 1 -- 172.21.15.123:0/2179313832 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f6334073730 msgr2=0x7f6334075bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:51:12.905 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:12.906+0000 7f634d686640 1 --2- 172.21.15.123:0/2179313832 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f6334073730 0x7f6334075bf0 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f6348156ef0 tx=0x7f6338009040 comp rx=0 tx=0).stop 2024-09-16T15:51:12.905 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:12.906+0000 7f634d686640 1 -- 172.21.15.123:0/2179313832 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f63480b2b00 msgr2=0x7f6348152cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:51:12.905 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:12.906+0000 7f634d686640 1 --2- 172.21.15.123:0/2179313832 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f63480b2b00 0x7f6348152cf0 secure :-1 s=READY pgs=1717 cs=0 l=1 rev1=1 crypto rx=0x7f633c01cb90 tx=0x7f633c0a6620 comp rx=0 tx=0).stop 2024-09-16T15:51:12.905 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:12.906+0000 7f634d686640 1 -- 172.21.15.123:0/2179313832 shutdown_connections 2024-09-16T15:51:12.905 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:12.906+0000 7f634d686640 1 --2- 172.21.15.123:0/2179313832 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f6334073730 0x7f6334075bf0 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:12.906 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:12.906+0000 7f634d686640 1 --2- 172.21.15.123:0/2179313832 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6348161620 0x7f63480b5920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:12.906 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:12.906+0000 7f634d686640 1 --2- 172.21.15.123:0/2179313832 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f63480b2b00 0x7f6348152cf0 unknown :-1 s=CLOSED pgs=1717 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:12.906 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:12.906+0000 7f634d686640 1 --2- 172.21.15.123:0/2179313832 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f63480b0a50 0x7f63481515b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:12.906 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:12.906+0000 7f634d686640 1 -- 172.21.15.123:0/2179313832 >> 172.21.15.123:0/2179313832 conn(0x7f63480d3ea0 msgr2=0x7f634815d110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:51:12.906 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:12.906+0000 7f634d686640 1 -- 172.21.15.123:0/2179313832 shutdown_connections 2024-09-16T15:51:12.906 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:12.906+0000 7f634d686640 1 -- 172.21.15.123:0/2179313832 wait complete. 2024-09-16T15:51:12.930 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-09-16T15:51:13.057 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:13.058+0000 7f2b14eef640 1 -- 172.21.15.123:0/3054086853 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2af8004810 msgr2=0x7f2af8004c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:51:13.057 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:13.058+0000 7f2b14eef640 1 --2- 172.21.15.123:0/3054086853 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2af8004810 0x7f2af8004c90 secure :-1 s=READY pgs=1719 cs=0 l=1 rev1=1 crypto rx=0x7f2b04009b60 tx=0x7f2b0401d240 comp rx=0 tx=0).stop 2024-09-16T15:51:13.057 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:13.058+0000 7f2b14eef640 1 -- 172.21.15.123:0/3054086853 shutdown_connections 2024-09-16T15:51:13.057 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:13.058+0000 7f2b14eef640 1 --2- 172.21.15.123:0/3054086853 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2af8004810 0x7f2af8004c90 unknown :-1 s=CLOSED pgs=1719 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:13.057 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:13.058+0000 7f2b14eef640 1 -- 172.21.15.123:0/3054086853 >> 172.21.15.123:0/3054086853 conn(0x7f2b100d3ea0 msgr2=0x7f2b100d42b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:51:13.057 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:13.058+0000 7f2b14eef640 1 -- 172.21.15.123:0/3054086853 shutdown_connections 2024-09-16T15:51:13.057 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:13.058+0000 7f2b14eef640 1 -- 172.21.15.123:0/3054086853 wait complete. 2024-09-16T15:51:13.058 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:13.058+0000 7f2b14eef640 1 Processor -- start 2024-09-16T15:51:13.058 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:13.058+0000 7f2b14eef640 1 -- start start 2024-09-16T15:51:13.060 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:13.058+0000 7f2b14eef640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2af8004810 0x7f2b100cc8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:51:13.060 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:13.058+0000 7f2b14eef640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f2b10156510 0x7f2b10155a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:51:13.060 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:13.058+0000 7f2b14eef640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2b10155fb0 0x7f2b101eca70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:51:13.060 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:13.058+0000 7f2b14eef640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f2b100d2be0 con 0x7f2b10155fb0 2024-09-16T15:51:13.060 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:13.058+0000 7f2b14eef640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f2b100d2a60 con 0x7f2af8004810 2024-09-16T15:51:13.060 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:13.058+0000 7f2b14eef640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f2b100b0e30 con 0x7f2b10156510 2024-09-16T15:51:13.060 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:13.058+0000 7f2b0ed9d640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2b10155fb0 0x7f2b101eca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:51:13.060 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:13.058+0000 7f2b0ed9d640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2b10155fb0 0x7f2b101eca70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:44948/0 (socket says 172.21.15.123:44948) 2024-09-16T15:51:13.061 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:13.058+0000 7f2b0ed9d640 1 -- 172.21.15.123:0/560539495 learned_addr learned my addr 172.21.15.123:0/560539495 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:51:13.061 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:13.062+0000 7f2b0e59c640 1 --2- 172.21.15.123:0/560539495 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2af8004810 0x7f2b100cc8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:51:13.061 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:13.062+0000 7f2b0dd9b640 1 --2- 172.21.15.123:0/560539495 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f2b10156510 0x7f2b10155a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:51:13.061 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:13.062+0000 7f2b0ed9d640 1 -- 172.21.15.123:0/560539495 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f2b10156510 msgr2=0x7f2b10155a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:51:13.061 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:13.062+0000 7f2b0ed9d640 1 --2- 172.21.15.123:0/560539495 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f2b10156510 0x7f2b10155a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:13.061 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:13.062+0000 7f2b0ed9d640 1 -- 172.21.15.123:0/560539495 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2af8004810 msgr2=0x7f2b100cc8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:51:13.061 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:13.062+0000 7f2b0ed9d640 1 --2- 172.21.15.123:0/560539495 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2af8004810 0x7f2b100cc8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:13.061 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:13.062+0000 7f2b0ed9d640 1 -- 172.21.15.123:0/560539495 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2b100cce20 con 0x7f2b10155fb0 2024-09-16T15:51:13.061 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:13.062+0000 7f2b0ed9d640 1 --2- 172.21.15.123:0/560539495 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2b10155fb0 0x7f2b101eca70 secure :-1 s=READY pgs=1720 cs=0 l=1 rev1=1 crypto rx=0x7f2b100b5870 tx=0x7f2b0800cb90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:51:13.061 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:13.062+0000 7f2afeffd640 1 -- 172.21.15.123:0/560539495 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b08003b80 con 0x7f2b10155fb0 2024-09-16T15:51:13.061 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:13.062+0000 7f2afeffd640 1 -- 172.21.15.123:0/560539495 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f2b08003d40 con 0x7f2b10155fb0 2024-09-16T15:51:13.061 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:13.062+0000 7f2b14eef640 1 -- 172.21.15.123:0/560539495 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2b100b9990 con 0x7f2b10155fb0 2024-09-16T15:51:13.061 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:13.062+0000 7f2afeffd640 1 -- 172.21.15.123:0/560539495 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b08005330 con 0x7f2b10155fb0 2024-09-16T15:51:13.061 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:13.062+0000 7f2b14eef640 1 -- 172.21.15.123:0/560539495 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2b100cbe00 con 0x7f2b10155fb0 2024-09-16T15:51:13.062 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:13.062+0000 7f2b14eef640 1 -- 172.21.15.123:0/560539495 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2b100b1340 con 0x7f2b10155fb0 2024-09-16T15:51:13.062 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:13.062+0000 7f2afeffd640 1 -- 172.21.15.123:0/560539495 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 24) ==== 97845+0+0 (secure 0 0 0) 0x7f2b080054d0 con 0x7f2b10155fb0 2024-09-16T15:51:13.062 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:13.062+0000 7f2afeffd640 1 --2- 172.21.15.123:0/560539495 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f2af0073730 0x7f2af0075bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:51:13.062 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:13.062+0000 7f2afeffd640 1 -- 172.21.15.123:0/560539495 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(153..153 src has 1..153) ==== 4885+0+0 (secure 0 0 0) 0x7f2b08092160 con 0x7f2b10155fb0 2024-09-16T15:51:13.063 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:13.062+0000 7f2b0e59c640 1 --2- 172.21.15.123:0/560539495 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f2af0073730 0x7f2af0075bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:51:13.063 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:13.062+0000 7f2b0e59c640 1 --2- 172.21.15.123:0/560539495 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f2af0073730 0x7f2af0075bf0 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f2b000010f0 tx=0x7f2b00000fe0 comp rx=0 tx=0).ready entity=mgr.4150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:51:13.066 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:13.066+0000 7f2afeffd640 1 -- 172.21.15.123:0/560539495 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f2b0805ee60 con 0x7f2b10155fb0 2024-09-16T15:51:13.307 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:13.306+0000 7f2b14eef640 1 -- 172.21.15.123:0/560539495 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) -- 0x7f2b100b6260 con 0x7f2b10155fb0 2024-09-16T15:51:13.308 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:13.310+0000 7f2afeffd640 1 -- 172.21.15.123:0/560539495 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v28) ==== 80+0+5304 (secure 0 0 0) 0x7f2b0805e760 con 0x7f2b10155fb0 2024-09-16T15:51:13.308 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-09-16T15:51:13.308 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2024-09-16T15:51:13.308 INFO:teuthology.orchestra.run.smithi123.stdout: "auth_dump": [ 2024-09-16T15:51:13.308 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-09-16T15:51:13.308 INFO:teuthology.orchestra.run.smithi123.stdout: "entity": "mds.a", 2024-09-16T15:51:13.308 INFO:teuthology.orchestra.run.smithi123.stdout: "key": "AQC0UehmbZpOJBAAaYGyDcWrt4k0FS5P/bb+4A==", 2024-09-16T15:51:13.308 INFO:teuthology.orchestra.run.smithi123.stdout: "caps": { 2024-09-16T15:51:13.308 INFO:teuthology.orchestra.run.smithi123.stdout: "mds": "allow", 2024-09-16T15:51:13.308 INFO:teuthology.orchestra.run.smithi123.stdout: "mgr": "allow *", 2024-09-16T15:51:13.309 INFO:teuthology.orchestra.run.smithi123.stdout: "mon": "allow *", 2024-09-16T15:51:13.309 INFO:teuthology.orchestra.run.smithi123.stdout: "osd": "allow *" 2024-09-16T15:51:13.309 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-16T15:51:13.309 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:51:13.309 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-09-16T15:51:13.309 INFO:teuthology.orchestra.run.smithi123.stdout: "entity": "mds.b", 2024-09-16T15:51:13.309 INFO:teuthology.orchestra.run.smithi123.stdout: "key": "AQC0UehmF1HFMhAADcvqXNKdHZMPAbLc+se4Aw==", 2024-09-16T15:51:13.309 INFO:teuthology.orchestra.run.smithi123.stdout: "caps": { 2024-09-16T15:51:13.309 INFO:teuthology.orchestra.run.smithi123.stdout: "mds": "allow", 2024-09-16T15:51:13.309 INFO:teuthology.orchestra.run.smithi123.stdout: "mgr": "allow *", 2024-09-16T15:51:13.309 INFO:teuthology.orchestra.run.smithi123.stdout: "mon": "allow *", 2024-09-16T15:51:13.309 INFO:teuthology.orchestra.run.smithi123.stdout: "osd": "allow *" 2024-09-16T15:51:13.309 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-16T15:51:13.309 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:51:13.310 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-09-16T15:51:13.310 INFO:teuthology.orchestra.run.smithi123.stdout: "entity": "mds.c", 2024-09-16T15:51:13.310 INFO:teuthology.orchestra.run.smithi123.stdout: "key": "AQC0UehmCBjCLBAAXd85J25r7cgrXeDCG7pahg==", 2024-09-16T15:51:13.310 INFO:teuthology.orchestra.run.smithi123.stdout: "caps": { 2024-09-16T15:51:13.310 INFO:teuthology.orchestra.run.smithi123.stdout: "mds": "allow", 2024-09-16T15:51:13.310 INFO:teuthology.orchestra.run.smithi123.stdout: "mgr": "allow *", 2024-09-16T15:51:13.310 INFO:teuthology.orchestra.run.smithi123.stdout: "mon": "allow *", 2024-09-16T15:51:13.310 INFO:teuthology.orchestra.run.smithi123.stdout: "osd": "allow *" 2024-09-16T15:51:13.310 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-16T15:51:13.310 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:51:13.310 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-09-16T15:51:13.310 INFO:teuthology.orchestra.run.smithi123.stdout: "entity": "mds.d", 2024-09-16T15:51:13.310 INFO:teuthology.orchestra.run.smithi123.stdout: "key": "AQC0UehmMh3ZOBAAx0RCa3R9RMT93/icZhPGyg==", 2024-09-16T15:51:13.310 INFO:teuthology.orchestra.run.smithi123.stdout: "caps": { 2024-09-16T15:51:13.310 INFO:teuthology.orchestra.run.smithi123.stdout: "mds": "allow", 2024-09-16T15:51:13.311 INFO:teuthology.orchestra.run.smithi123.stdout: "mgr": "allow *", 2024-09-16T15:51:13.311 INFO:teuthology.orchestra.run.smithi123.stdout: "mon": "allow *", 2024-09-16T15:51:13.311 INFO:teuthology.orchestra.run.smithi123.stdout: "osd": "allow *" 2024-09-16T15:51:13.311 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-16T15:51:13.311 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:51:13.311 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-09-16T15:51:13.311 INFO:teuthology.orchestra.run.smithi123.stdout: "entity": "osd.0", 2024-09-16T15:51:13.311 INFO:teuthology.orchestra.run.smithi123.stdout: "key": "AQC4UehmT633CRAAIhpkeIBOzwJO0kiXLu0x9Q==", 2024-09-16T15:51:13.311 INFO:teuthology.orchestra.run.smithi123.stdout: "caps": { 2024-09-16T15:51:13.311 INFO:teuthology.orchestra.run.smithi123.stdout: "mgr": "allow profile osd", 2024-09-16T15:51:13.311 INFO:teuthology.orchestra.run.smithi123.stdout: "mon": "allow profile osd", 2024-09-16T15:51:13.311 INFO:teuthology.orchestra.run.smithi123.stdout: "osd": "allow *" 2024-09-16T15:51:13.311 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-16T15:51:13.311 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:51:13.311 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-09-16T15:51:13.312 INFO:teuthology.orchestra.run.smithi123.stdout: "entity": "osd.1", 2024-09-16T15:51:13.312 INFO:teuthology.orchestra.run.smithi123.stdout: "key": "AQC7UehmFwF7GBAAvOUG+rWjANKT8wm/ai4xGg==", 2024-09-16T15:51:13.312 INFO:teuthology.orchestra.run.smithi123.stdout: "caps": { 2024-09-16T15:51:13.312 INFO:teuthology.orchestra.run.smithi123.stdout: "mgr": "allow profile osd", 2024-09-16T15:51:13.312 INFO:teuthology.orchestra.run.smithi123.stdout: "mon": "allow profile osd", 2024-09-16T15:51:13.312 INFO:teuthology.orchestra.run.smithi123.stdout: "osd": "allow *" 2024-09-16T15:51:13.312 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-16T15:51:13.312 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:51:13.312 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-09-16T15:51:13.312 INFO:teuthology.orchestra.run.smithi123.stdout: "entity": "osd.2", 2024-09-16T15:51:13.312 INFO:teuthology.orchestra.run.smithi123.stdout: "key": "AQC+UehmlCwWKxAAl3iO7ZALWyaorQH2urHKCw==", 2024-09-16T15:51:13.312 INFO:teuthology.orchestra.run.smithi123.stdout: "caps": { 2024-09-16T15:51:13.312 INFO:teuthology.orchestra.run.smithi123.stdout: "mgr": "allow profile osd", 2024-09-16T15:51:13.312 INFO:teuthology.orchestra.run.smithi123.stdout: "mon": "allow profile osd", 2024-09-16T15:51:13.312 INFO:teuthology.orchestra.run.smithi123.stdout: "osd": "allow *" 2024-09-16T15:51:13.313 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-16T15:51:13.313 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:51:13.313 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-09-16T15:51:13.313 INFO:teuthology.orchestra.run.smithi123.stdout: "entity": "osd.3", 2024-09-16T15:51:13.313 INFO:teuthology.orchestra.run.smithi123.stdout: "key": "AQDCUehmD8VgAhAAneoIeGr3MflSmi+nyavj2A==", 2024-09-16T15:51:13.313 INFO:teuthology.orchestra.run.smithi123.stdout: "caps": { 2024-09-16T15:51:13.313 INFO:teuthology.orchestra.run.smithi123.stdout: "mgr": "allow profile osd", 2024-09-16T15:51:13.313 INFO:teuthology.orchestra.run.smithi123.stdout: "mon": "allow profile osd", 2024-09-16T15:51:13.313 INFO:teuthology.orchestra.run.smithi123.stdout: "osd": "allow *" 2024-09-16T15:51:13.313 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-16T15:51:13.313 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:51:13.313 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-09-16T15:51:13.313 INFO:teuthology.orchestra.run.smithi123.stdout: "entity": "osd.4", 2024-09-16T15:51:13.313 INFO:teuthology.orchestra.run.smithi123.stdout: "key": "AQDIUehmA9B9EBAAsu0X5LPx1TmatZeHzjnVig==", 2024-09-16T15:51:13.314 INFO:teuthology.orchestra.run.smithi123.stdout: "caps": { 2024-09-16T15:51:13.314 INFO:teuthology.orchestra.run.smithi123.stdout: "mgr": "allow profile osd", 2024-09-16T15:51:13.314 INFO:teuthology.orchestra.run.smithi123.stdout: "mon": "allow profile osd", 2024-09-16T15:51:13.314 INFO:teuthology.orchestra.run.smithi123.stdout: "osd": "allow *" 2024-09-16T15:51:13.314 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-16T15:51:13.314 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:51:13.314 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-09-16T15:51:13.314 INFO:teuthology.orchestra.run.smithi123.stdout: "entity": "osd.5", 2024-09-16T15:51:13.314 INFO:teuthology.orchestra.run.smithi123.stdout: "key": "AQDLUehmcugdHxAAQMSS1ZH71FUzCx5A+yNnZg==", 2024-09-16T15:51:13.314 INFO:teuthology.orchestra.run.smithi123.stdout: "caps": { 2024-09-16T15:51:13.314 INFO:teuthology.orchestra.run.smithi123.stdout: "mgr": "allow profile osd", 2024-09-16T15:51:13.314 INFO:teuthology.orchestra.run.smithi123.stdout: "mon": "allow profile osd", 2024-09-16T15:51:13.314 INFO:teuthology.orchestra.run.smithi123.stdout: "osd": "allow *" 2024-09-16T15:51:13.314 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-16T15:51:13.314 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:51:13.315 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-09-16T15:51:13.315 INFO:teuthology.orchestra.run.smithi123.stdout: "entity": "osd.6", 2024-09-16T15:51:13.315 INFO:teuthology.orchestra.run.smithi123.stdout: "key": "AQDOUehmWLwYLhAAhOlQLepcWElsVPMnHlZm4w==", 2024-09-16T15:51:13.315 INFO:teuthology.orchestra.run.smithi123.stdout: "caps": { 2024-09-16T15:51:13.315 INFO:teuthology.orchestra.run.smithi123.stdout: "mgr": "allow profile osd", 2024-09-16T15:51:13.315 INFO:teuthology.orchestra.run.smithi123.stdout: "mon": "allow profile osd", 2024-09-16T15:51:13.315 INFO:teuthology.orchestra.run.smithi123.stdout: "osd": "allow *" 2024-09-16T15:51:13.315 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-16T15:51:13.315 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:51:13.315 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-09-16T15:51:13.315 INFO:teuthology.orchestra.run.smithi123.stdout: "entity": "osd.7", 2024-09-16T15:51:13.315 INFO:teuthology.orchestra.run.smithi123.stdout: "key": "AQDSUehmipkbABAAQuL/2O7w0tsM1OKeOil97w==", 2024-09-16T15:51:13.315 INFO:teuthology.orchestra.run.smithi123.stdout: "caps": { 2024-09-16T15:51:13.315 INFO:teuthology.orchestra.run.smithi123.stdout: "mgr": "allow profile osd", 2024-09-16T15:51:13.315 INFO:teuthology.orchestra.run.smithi123.stdout: "mon": "allow profile osd", 2024-09-16T15:51:13.316 INFO:teuthology.orchestra.run.smithi123.stdout: "osd": "allow *" 2024-09-16T15:51:13.316 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-16T15:51:13.316 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:51:13.316 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-09-16T15:51:13.316 INFO:teuthology.orchestra.run.smithi123.stdout: "entity": "client.0", 2024-09-16T15:51:13.316 INFO:teuthology.orchestra.run.smithi123.stdout: "key": "AQC1Uehm7GDTAhAAHWWVIyjVVnTeeozBR8DfFA==", 2024-09-16T15:51:13.316 INFO:teuthology.orchestra.run.smithi123.stdout: "caps": { 2024-09-16T15:51:13.316 INFO:teuthology.orchestra.run.smithi123.stdout: "mds": "allow", 2024-09-16T15:51:13.316 INFO:teuthology.orchestra.run.smithi123.stdout: "mon": "allow r", 2024-09-16T15:51:13.316 INFO:teuthology.orchestra.run.smithi123.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-09-16T15:51:13.316 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-16T15:51:13.316 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:51:13.316 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-09-16T15:51:13.316 INFO:teuthology.orchestra.run.smithi123.stdout: "entity": "client.1", 2024-09-16T15:51:13.316 INFO:teuthology.orchestra.run.smithi123.stdout: "key": "AQC1UehmFbaWChAAFtRmKucm+pbKtCNvRCTNqA==", 2024-09-16T15:51:13.317 INFO:teuthology.orchestra.run.smithi123.stdout: "caps": { 2024-09-16T15:51:13.317 INFO:teuthology.orchestra.run.smithi123.stdout: "mds": "allow", 2024-09-16T15:51:13.317 INFO:teuthology.orchestra.run.smithi123.stdout: "mgr": "allow r", 2024-09-16T15:51:13.317 INFO:teuthology.orchestra.run.smithi123.stdout: "mon": "allow rw", 2024-09-16T15:51:13.317 INFO:teuthology.orchestra.run.smithi123.stdout: "osd": "allow rwx" 2024-09-16T15:51:13.317 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-16T15:51:13.317 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:51:13.317 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-09-16T15:51:13.317 INFO:teuthology.orchestra.run.smithi123.stdout: "entity": "client.2", 2024-09-16T15:51:13.317 INFO:teuthology.orchestra.run.smithi123.stdout: "key": "AQC1Uehm1ADrBxAAW/8ayNHI6Wq/YlI4ivFZiw==", 2024-09-16T15:51:13.317 INFO:teuthology.orchestra.run.smithi123.stdout: "caps": { 2024-09-16T15:51:13.317 INFO:teuthology.orchestra.run.smithi123.stdout: "mds": "allow", 2024-09-16T15:51:13.317 INFO:teuthology.orchestra.run.smithi123.stdout: "mgr": "allow r", 2024-09-16T15:51:13.318 INFO:teuthology.orchestra.run.smithi123.stdout: "mon": "allow rw", 2024-09-16T15:51:13.318 INFO:teuthology.orchestra.run.smithi123.stdout: "osd": "allow rwx" 2024-09-16T15:51:13.318 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-16T15:51:13.318 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:51:13.318 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-09-16T15:51:13.318 INFO:teuthology.orchestra.run.smithi123.stdout: "entity": "client.3", 2024-09-16T15:51:13.318 INFO:teuthology.orchestra.run.smithi123.stdout: "key": "AQC1UehmLv/3DxAAuGloBxSuza95uajpfiZ9MQ==", 2024-09-16T15:51:13.318 INFO:teuthology.orchestra.run.smithi123.stdout: "caps": { 2024-09-16T15:51:13.318 INFO:teuthology.orchestra.run.smithi123.stdout: "mds": "allow", 2024-09-16T15:51:13.318 INFO:teuthology.orchestra.run.smithi123.stdout: "mgr": "allow r", 2024-09-16T15:51:13.318 INFO:teuthology.orchestra.run.smithi123.stdout: "mon": "allow rw", 2024-09-16T15:51:13.318 INFO:teuthology.orchestra.run.smithi123.stdout: "osd": "allow rwx" 2024-09-16T15:51:13.318 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-16T15:51:13.318 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:51:13.318 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-09-16T15:51:13.319 INFO:teuthology.orchestra.run.smithi123.stdout: "entity": "client.admin", 2024-09-16T15:51:13.319 INFO:teuthology.orchestra.run.smithi123.stdout: "key": "AQC0Uehm1ZDRDBAAhyQ1BmJg1Mw6wQ04I2CbFA==", 2024-09-16T15:51:13.319 INFO:teuthology.orchestra.run.smithi123.stdout: "caps": { 2024-09-16T15:51:13.319 INFO:teuthology.orchestra.run.smithi123.stdout: "mds": "allow *", 2024-09-16T15:51:13.319 INFO:teuthology.orchestra.run.smithi123.stdout: "mgr": "allow *", 2024-09-16T15:51:13.319 INFO:teuthology.orchestra.run.smithi123.stdout: "mon": "allow *", 2024-09-16T15:51:13.319 INFO:teuthology.orchestra.run.smithi123.stdout: "osd": "allow *" 2024-09-16T15:51:13.319 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-16T15:51:13.319 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:51:13.319 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-09-16T15:51:13.319 INFO:teuthology.orchestra.run.smithi123.stdout: "entity": "mgr.x", 2024-09-16T15:51:13.319 INFO:teuthology.orchestra.run.smithi123.stdout: "key": "AQC0UehmfPilIBAAERlBbduOgAU9ba0n/1tl3g==", 2024-09-16T15:51:13.319 INFO:teuthology.orchestra.run.smithi123.stdout: "caps": { 2024-09-16T15:51:13.319 INFO:teuthology.orchestra.run.smithi123.stdout: "mds": "allow *", 2024-09-16T15:51:13.319 INFO:teuthology.orchestra.run.smithi123.stdout: "mon": "allow profile mgr", 2024-09-16T15:51:13.319 INFO:teuthology.orchestra.run.smithi123.stdout: "osd": "allow *" 2024-09-16T15:51:13.320 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-16T15:51:13.320 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:51:13.320 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-09-16T15:51:13.320 INFO:teuthology.orchestra.run.smithi123.stdout: "entity": "mgr.y", 2024-09-16T15:51:13.320 INFO:teuthology.orchestra.run.smithi123.stdout: "key": "AQC0UehmkjrlHRAA+3xbQC+SPWGWW1gND7xcAg==", 2024-09-16T15:51:13.320 INFO:teuthology.orchestra.run.smithi123.stdout: "caps": { 2024-09-16T15:51:13.320 INFO:teuthology.orchestra.run.smithi123.stdout: "mds": "allow *", 2024-09-16T15:51:13.320 INFO:teuthology.orchestra.run.smithi123.stdout: "mon": "allow profile mgr", 2024-09-16T15:51:13.320 INFO:teuthology.orchestra.run.smithi123.stdout: "osd": "allow *" 2024-09-16T15:51:13.320 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-16T15:51:13.320 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-16T15:51:13.320 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-09-16T15:51:13.320 INFO:teuthology.orchestra.run.smithi123.stdout:} 2024-09-16T15:51:13.324 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:13.310+0000 7f2b14eef640 1 -- 172.21.15.123:0/560539495 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f2af0073730 msgr2=0x7f2af0075bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:51:13.324 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:13.310+0000 7f2b14eef640 1 --2- 172.21.15.123:0/560539495 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f2af0073730 0x7f2af0075bf0 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f2b000010f0 tx=0x7f2b00000fe0 comp rx=0 tx=0).stop 2024-09-16T15:51:13.324 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:13.310+0000 7f2b14eef640 1 -- 172.21.15.123:0/560539495 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2b10155fb0 msgr2=0x7f2b101eca70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:51:13.324 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:13.310+0000 7f2b14eef640 1 --2- 172.21.15.123:0/560539495 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2b10155fb0 0x7f2b101eca70 secure :-1 s=READY pgs=1720 cs=0 l=1 rev1=1 crypto rx=0x7f2b100b5870 tx=0x7f2b0800cb90 comp rx=0 tx=0).stop 2024-09-16T15:51:13.324 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:13.310+0000 7f2b14eef640 1 -- 172.21.15.123:0/560539495 shutdown_connections 2024-09-16T15:51:13.325 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:13.310+0000 7f2b14eef640 1 --2- 172.21.15.123:0/560539495 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f2af0073730 0x7f2af0075bf0 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:13.325 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:13.310+0000 7f2b14eef640 1 --2- 172.21.15.123:0/560539495 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2b10155fb0 0x7f2b101eca70 unknown :-1 s=CLOSED pgs=1720 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:13.325 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:13.310+0000 7f2b14eef640 1 --2- 172.21.15.123:0/560539495 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f2b10156510 0x7f2b10155a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:13.325 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:13.310+0000 7f2b14eef640 1 --2- 172.21.15.123:0/560539495 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2af8004810 0x7f2b100cc8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:13.325 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:13.310+0000 7f2b14eef640 1 -- 172.21.15.123:0/560539495 >> 172.21.15.123:0/560539495 conn(0x7f2b100d3ea0 msgr2=0x7f2b100b1550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:51:13.325 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:13.310+0000 7f2b14eef640 1 -- 172.21.15.123:0/560539495 shutdown_connections 2024-09-16T15:51:13.325 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:13.310+0000 7f2b14eef640 1 -- 172.21.15.123:0/560539495 wait complete. 2024-09-16T15:51:13.328 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-09-16T15:51:13.449 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:13.450+0000 7fe9b9fd6640 1 -- 172.21.15.123:0/3459885953 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe9b40ca870 msgr2=0x7fe9b41506d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:51:13.449 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:13.450+0000 7fe9b9fd6640 1 --2- 172.21.15.123:0/3459885953 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe9b40ca870 0x7fe9b41506d0 secure :-1 s=READY pgs=1722 cs=0 l=1 rev1=1 crypto rx=0x7fe9b40cad20 tx=0x7fe9ac01cb80 comp rx=0 tx=0).stop 2024-09-16T15:51:13.449 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:13.450+0000 7fe9b9fd6640 1 -- 172.21.15.123:0/3459885953 shutdown_connections 2024-09-16T15:51:13.449 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:13.450+0000 7fe9b9fd6640 1 --2- 172.21.15.123:0/3459885953 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe9b40ca870 0x7fe9b41506d0 unknown :-1 s=CLOSED pgs=1722 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:13.449 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:13.450+0000 7fe9b9fd6640 1 --2- 172.21.15.123:0/3459885953 >> v2:172.21.15.184:3300/0 conn(0x7fe9b406fc00 0x7fe9b40cf3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:13.449 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:13.450+0000 7fe9b9fd6640 1 -- 172.21.15.123:0/3459885953 >> 172.21.15.123:0/3459885953 conn(0x7fe9b40cfb10 msgr2=0x7fe9b40cff20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:51:13.449 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:13.450+0000 7fe9b9fd6640 1 -- 172.21.15.123:0/3459885953 shutdown_connections 2024-09-16T15:51:13.449 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:13.450+0000 7fe9b9fd6640 1 -- 172.21.15.123:0/3459885953 wait complete. 2024-09-16T15:51:13.450 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:13.450+0000 7fe9b9fd6640 1 Processor -- start 2024-09-16T15:51:13.450 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:13.450+0000 7fe9b9fd6640 1 -- start start 2024-09-16T15:51:13.452 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:13.454+0000 7fe9b9fd6640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fe9b406fc00 0x7fe9b4147ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:51:13.452 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:13.454+0000 7fe9b9fd6640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe9b40ca870 0x7fe9b414d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:51:13.452 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:13.454+0000 7fe9b9fd6640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe9b415ae10 0x7fe9b414d8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:51:13.452 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:13.454+0000 7fe9b9fd6640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fe9b406eb80 con 0x7fe9b415ae10 2024-09-16T15:51:13.452 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:13.454+0000 7fe9b9fd6640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fe9b406ea00 con 0x7fe9b40ca870 2024-09-16T15:51:13.452 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:13.454+0000 7fe9b9fd6640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fe9b406ed00 con 0x7fe9b406fc00 2024-09-16T15:51:13.452 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:13.454+0000 7fe9b3fff640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe9b415ae10 0x7fe9b414d8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:51:13.452 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:13.454+0000 7fe9b3fff640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe9b415ae10 0x7fe9b414d8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:44968/0 (socket says 172.21.15.123:44968) 2024-09-16T15:51:13.452 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:13.454+0000 7fe9b3fff640 1 -- 172.21.15.123:0/4215458964 learned_addr learned my addr 172.21.15.123:0/4215458964 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:51:13.452 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:13.454+0000 7fe9b3fff640 1 -- 172.21.15.123:0/4215458964 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fe9b406fc00 msgr2=0x7fe9b4147ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:51:13.452 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:13.454+0000 7fe9b3fff640 1 --2- 172.21.15.123:0/4215458964 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fe9b406fc00 0x7fe9b4147ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:13.453 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:13.454+0000 7fe9b3fff640 1 -- 172.21.15.123:0/4215458964 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe9b40ca870 msgr2=0x7fe9b414d390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:51:13.453 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:13.454+0000 7fe9b3fff640 1 --2- 172.21.15.123:0/4215458964 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe9b40ca870 0x7fe9b414d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:13.453 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:13.454+0000 7fe9b3fff640 1 -- 172.21.15.123:0/4215458964 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe9b40c6be0 con 0x7fe9b415ae10 2024-09-16T15:51:13.453 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:13.454+0000 7fe9b3fff640 1 --2- 172.21.15.123:0/4215458964 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe9b415ae10 0x7fe9b414d8d0 secure :-1 s=READY pgs=1723 cs=0 l=1 rev1=1 crypto rx=0x7fe9ac01a7b0 tx=0x7fe9ac003f60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:51:13.453 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:13.454+0000 7fe99bfff640 1 -- 172.21.15.123:0/4215458964 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9ac0bd070 con 0x7fe9b415ae10 2024-09-16T15:51:13.453 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:13.454+0000 7fe99bfff640 1 -- 172.21.15.123:0/4215458964 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fe9ac0a5b90 con 0x7fe9b415ae10 2024-09-16T15:51:13.453 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:13.454+0000 7fe99bfff640 1 -- 172.21.15.123:0/4215458964 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9ac004a50 con 0x7fe9b415ae10 2024-09-16T15:51:13.453 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:13.454+0000 7fe9b9fd6640 1 -- 172.21.15.123:0/4215458964 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe9b40c6e80 con 0x7fe9b415ae10 2024-09-16T15:51:13.454 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:13.454+0000 7fe9b9fd6640 1 -- 172.21.15.123:0/4215458964 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe9b40c7440 con 0x7fe9b415ae10 2024-09-16T15:51:13.454 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:13.454+0000 7fe9b9fd6640 1 -- 172.21.15.123:0/4215458964 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe9b40cf3f0 con 0x7fe9b415ae10 2024-09-16T15:51:13.454 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:13.454+0000 7fe99bfff640 1 -- 172.21.15.123:0/4215458964 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 24) ==== 97845+0+0 (secure 0 0 0) 0x7fe9ac0b5070 con 0x7fe9b415ae10 2024-09-16T15:51:13.454 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:13.454+0000 7fe99bfff640 1 --2- 172.21.15.123:0/4215458964 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fe9a0073730 0x7fe9a0075bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:51:13.455 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:13.454+0000 7fe9b37fe640 1 --2- 172.21.15.123:0/4215458964 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fe9a0073730 0x7fe9a0075bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:51:13.455 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:13.454+0000 7fe99bfff640 1 -- 172.21.15.123:0/4215458964 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(153..153 src has 1..153) ==== 4885+0+0 (secure 0 0 0) 0x7fe9ac12e8d0 con 0x7fe9b415ae10 2024-09-16T15:51:13.455 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:13.458+0000 7fe9b37fe640 1 --2- 172.21.15.123:0/4215458964 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fe9a0073730 0x7fe9a0075bf0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7fe9b40c86c0 tx=0x7fe9a4005f50 comp rx=0 tx=0).ready entity=mgr.4150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:51:13.458 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:13.458+0000 7fe99bfff640 1 -- 172.21.15.123:0/4215458964 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7fe9ac016770 con 0x7fe9b415ae10 2024-09-16T15:51:13.631 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:13.634+0000 7fe9b9fd6640 1 -- 172.21.15.123:0/4215458964 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) -- 0x7fe9b40cd6e0 con 0x7fe9b415ae10 2024-09-16T15:51:13.631 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:13.634+0000 7fe99bfff640 1 -- 172.21.15.123:0/4215458964 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v153) ==== 84+0+65 (secure 0 0 0) 0x7fe9ac0f9230 con 0x7fe9b415ae10 2024-09-16T15:51:13.632 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-09-16T15:51:13.632 INFO:teuthology.orchestra.run.smithi123.stdout:[ 2024-09-16T15:51:13.632 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-09-16T15:51:13.632 INFO:teuthology.orchestra.run.smithi123.stdout: "poolnum": 1, 2024-09-16T15:51:13.632 INFO:teuthology.orchestra.run.smithi123.stdout: "poolname": ".mgr" 2024-09-16T15:51:13.632 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-16T15:51:13.632 INFO:teuthology.orchestra.run.smithi123.stdout:] 2024-09-16T15:51:13.634 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:13.634+0000 7fe9b9fd6640 1 -- 172.21.15.123:0/4215458964 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fe9a0073730 msgr2=0x7fe9a0075bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:51:13.634 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:13.634+0000 7fe9b9fd6640 1 --2- 172.21.15.123:0/4215458964 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fe9a0073730 0x7fe9a0075bf0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7fe9b40c86c0 tx=0x7fe9a4005f50 comp rx=0 tx=0).stop 2024-09-16T15:51:13.634 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:13.634+0000 7fe9b9fd6640 1 -- 172.21.15.123:0/4215458964 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe9b415ae10 msgr2=0x7fe9b414d8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:51:13.634 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:13.634+0000 7fe9b9fd6640 1 --2- 172.21.15.123:0/4215458964 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe9b415ae10 0x7fe9b414d8d0 secure :-1 s=READY pgs=1723 cs=0 l=1 rev1=1 crypto rx=0x7fe9ac01a7b0 tx=0x7fe9ac003f60 comp rx=0 tx=0).stop 2024-09-16T15:51:13.634 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:13.634+0000 7fe9b9fd6640 1 -- 172.21.15.123:0/4215458964 shutdown_connections 2024-09-16T15:51:13.634 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:13.634+0000 7fe9b9fd6640 1 --2- 172.21.15.123:0/4215458964 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fe9a0073730 0x7fe9a0075bf0 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:13.634 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:13.634+0000 7fe9b9fd6640 1 --2- 172.21.15.123:0/4215458964 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe9b415ae10 0x7fe9b414d8d0 unknown :-1 s=CLOSED pgs=1723 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:13.635 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:13.634+0000 7fe9b9fd6640 1 --2- 172.21.15.123:0/4215458964 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe9b40ca870 0x7fe9b414d390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:13.635 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:13.634+0000 7fe9b9fd6640 1 --2- 172.21.15.123:0/4215458964 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fe9b406fc00 0x7fe9b4147ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:13.635 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:13.634+0000 7fe9b9fd6640 1 -- 172.21.15.123:0/4215458964 >> 172.21.15.123:0/4215458964 conn(0x7fe9b40cfb10 msgr2=0x7fe9b40b29a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:51:13.635 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:13.634+0000 7fe9b9fd6640 1 -- 172.21.15.123:0/4215458964 shutdown_connections 2024-09-16T15:51:13.635 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:13.634+0000 7fe9b9fd6640 1 -- 172.21.15.123:0/4215458964 wait complete. 2024-09-16T15:51:13.650 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-09-16T15:51:13.650 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-09-16T15:51:13.772 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:13.774+0000 7f49f2843640 1 -- 172.21.15.123:0/364549612 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f49d8005220 msgr2=0x7f49d800b9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:51:13.772 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:13.774+0000 7f49f2843640 1 --2- 172.21.15.123:0/364549612 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f49d8005220 0x7f49d800b9d0 secure :-1 s=READY pgs=1725 cs=0 l=1 rev1=1 crypto rx=0x7f49e40043e0 tx=0x7f49e401ca60 comp rx=0 tx=0).stop 2024-09-16T15:51:13.772 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:13.774+0000 7f49f2843640 1 -- 172.21.15.123:0/364549612 shutdown_connections 2024-09-16T15:51:13.772 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:13.774+0000 7f49f2843640 1 --2- 172.21.15.123:0/364549612 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f49d8005220 0x7f49d800b9d0 unknown :-1 s=CLOSED pgs=1725 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:13.772 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:13.774+0000 7f49f2843640 1 -- 172.21.15.123:0/364549612 >> 172.21.15.123:0/364549612 conn(0x7f49ec09aa40 msgr2=0x7f49ec09ae50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:51:13.772 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:13.774+0000 7f49f2843640 1 -- 172.21.15.123:0/364549612 shutdown_connections 2024-09-16T15:51:13.772 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:13.774+0000 7f49f2843640 1 -- 172.21.15.123:0/364549612 wait complete. 2024-09-16T15:51:13.773 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:13.774+0000 7f49f2843640 1 Processor -- start 2024-09-16T15:51:13.773 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:13.774+0000 7f49f2843640 1 -- start start 2024-09-16T15:51:13.775 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:13.774+0000 7f49f2843640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f49d8005220 0x7f49ec0b66f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:51:13.775 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:13.774+0000 7f49f2843640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f49ec1f3b90 0x7f49ec1557f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:51:13.775 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:13.774+0000 7f49f2843640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f49ec155d30 0x7f49ec157b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:51:13.775 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:13.774+0000 7f49f2843640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f49ec0d1c10 con 0x7f49ec155d30 2024-09-16T15:51:13.775 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:13.774+0000 7f49f2843640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f49ec0d1a90 con 0x7f49ec1f3b90 2024-09-16T15:51:13.775 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:13.774+0000 7f49f2843640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f49ec0d1d90 con 0x7f49d8005220 2024-09-16T15:51:13.775 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:13.774+0000 7f49f0de0640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f49ec155d30 0x7f49ec157b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:51:13.775 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:13.774+0000 7f49f0de0640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f49ec155d30 0x7f49ec157b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:44990/0 (socket says 172.21.15.123:44990) 2024-09-16T15:51:13.775 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:13.774+0000 7f49f0de0640 1 -- 172.21.15.123:0/3216291462 learned_addr learned my addr 172.21.15.123:0/3216291462 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:51:13.775 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:13.774+0000 7f49ebfff640 1 --2- 172.21.15.123:0/3216291462 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f49d8005220 0x7f49ec0b66f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:51:13.775 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:13.774+0000 7f49f0de0640 1 -- 172.21.15.123:0/3216291462 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f49d8005220 msgr2=0x7f49ec0b66f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:51:13.776 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:13.774+0000 7f49f0de0640 1 --2- 172.21.15.123:0/3216291462 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f49d8005220 0x7f49ec0b66f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:13.776 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:13.774+0000 7f49f0de0640 1 -- 172.21.15.123:0/3216291462 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f49ec1f3b90 msgr2=0x7f49ec1557f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:51:13.776 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:13.774+0000 7f49e37fe640 1 --2- 172.21.15.123:0/3216291462 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f49ec1f3b90 0x7f49ec1557f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:51:13.776 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:13.774+0000 7f49f0de0640 1 --2- 172.21.15.123:0/3216291462 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f49ec1f3b90 0x7f49ec1557f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:13.776 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:13.774+0000 7f49f0de0640 1 -- 172.21.15.123:0/3216291462 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f49ec0b4c80 con 0x7f49ec155d30 2024-09-16T15:51:13.776 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:13.774+0000 7f49f0de0640 1 --2- 172.21.15.123:0/3216291462 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f49ec155d30 0x7f49ec157b00 secure :-1 s=READY pgs=1726 cs=0 l=1 rev1=1 crypto rx=0x7f49e401d300 tx=0x7f49e4006cb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:51:13.776 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:13.774+0000 7f49e97fa640 1 -- 172.21.15.123:0/3216291462 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49e40a5a60 con 0x7f49ec155d30 2024-09-16T15:51:13.776 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:13.774+0000 7f49e97fa640 1 -- 172.21.15.123:0/3216291462 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f49e40b2070 con 0x7f49ec155d30 2024-09-16T15:51:13.776 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:13.774+0000 7f49f2843640 1 -- 172.21.15.123:0/3216291462 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f49ec0b4f60 con 0x7f49ec155d30 2024-09-16T15:51:13.776 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:13.774+0000 7f49e97fa640 1 -- 172.21.15.123:0/3216291462 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49e40060d0 con 0x7f49ec155d30 2024-09-16T15:51:13.776 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:13.774+0000 7f49f2843640 1 -- 172.21.15.123:0/3216291462 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f49ec0b5a40 con 0x7f49ec155d30 2024-09-16T15:51:13.777 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:13.778+0000 7f49f2843640 1 -- 172.21.15.123:0/3216291462 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f49ec15c690 con 0x7f49ec155d30 2024-09-16T15:51:13.777 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:13.778+0000 7f49e97fa640 1 -- 172.21.15.123:0/3216291462 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 24) ==== 97845+0+0 (secure 0 0 0) 0x7f49e40bd060 con 0x7f49ec155d30 2024-09-16T15:51:13.777 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:13.778+0000 7f49e97fa640 1 --2- 172.21.15.123:0/3216291462 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f49cc073760 0x7f49cc075c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:51:13.777 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:13.778+0000 7f49e97fa640 1 -- 172.21.15.123:0/3216291462 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(153..153 src has 1..153) ==== 4885+0+0 (secure 0 0 0) 0x7f49e412c330 con 0x7f49ec155d30 2024-09-16T15:51:13.778 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:13.778+0000 7f49ebfff640 1 --2- 172.21.15.123:0/3216291462 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f49cc073760 0x7f49cc075c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:51:13.778 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:13.778+0000 7f49ebfff640 1 --2- 172.21.15.123:0/3216291462 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f49cc073760 0x7f49cc075c20 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f49dc001250 tx=0x7f49dc0011c0 comp rx=0 tx=0).ready entity=mgr.4150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:51:13.780 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:13.782+0000 7f49e97fa640 1 -- 172.21.15.123:0/3216291462 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f49e4016770 con 0x7f49ec155d30 2024-09-16T15:51:13.943 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:13.946+0000 7f49f2843640 1 -- 172.21.15.123:0/3216291462 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) -- 0x7f49ec0b3400 con 0x7f49ec155d30 2024-09-16T15:51:14.913 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:14.914+0000 7f49e97fa640 1 -- 172.21.15.123:0/3216291462 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v154) ==== 138+0+0 (secure 0 0 0) 0x7f49e40f9120 con 0x7f49ec155d30 2024-09-16T15:51:14.913 INFO:teuthology.orchestra.run.smithi123.stderr:pool 'cephfs_metadata' created 2024-09-16T15:51:14.919 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:14.918+0000 7f49f2843640 1 -- 172.21.15.123:0/3216291462 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f49cc073760 msgr2=0x7f49cc075c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:51:14.919 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:14.918+0000 7f49f2843640 1 --2- 172.21.15.123:0/3216291462 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f49cc073760 0x7f49cc075c20 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f49dc001250 tx=0x7f49dc0011c0 comp rx=0 tx=0).stop 2024-09-16T15:51:14.919 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:14.918+0000 7f49f2843640 1 -- 172.21.15.123:0/3216291462 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f49ec155d30 msgr2=0x7f49ec157b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:51:14.919 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:14.918+0000 7f49f2843640 1 --2- 172.21.15.123:0/3216291462 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f49ec155d30 0x7f49ec157b00 secure :-1 s=READY pgs=1726 cs=0 l=1 rev1=1 crypto rx=0x7f49e401d300 tx=0x7f49e4006cb0 comp rx=0 tx=0).stop 2024-09-16T15:51:14.919 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:14.918+0000 7f49f2843640 1 -- 172.21.15.123:0/3216291462 shutdown_connections 2024-09-16T15:51:14.920 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:14.918+0000 7f49f2843640 1 --2- 172.21.15.123:0/3216291462 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f49cc073760 0x7f49cc075c20 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:14.920 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:14.918+0000 7f49f2843640 1 --2- 172.21.15.123:0/3216291462 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f49ec155d30 0x7f49ec157b00 unknown :-1 s=CLOSED pgs=1726 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:14.920 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:14.918+0000 7f49f2843640 1 --2- 172.21.15.123:0/3216291462 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f49ec1f3b90 0x7f49ec1557f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:14.920 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:14.918+0000 7f49f2843640 1 --2- 172.21.15.123:0/3216291462 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f49d8005220 0x7f49ec0b66f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:14.920 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:14.918+0000 7f49f2843640 1 -- 172.21.15.123:0/3216291462 >> 172.21.15.123:0/3216291462 conn(0x7f49ec09aa40 msgr2=0x7f49ec15b2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:51:14.920 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:14.918+0000 7f49f2843640 1 -- 172.21.15.123:0/3216291462 shutdown_connections 2024-09-16T15:51:14.920 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:14.922+0000 7f49f2843640 1 -- 172.21.15.123:0/3216291462 wait complete. 2024-09-16T15:51:14.947 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-09-16T15:51:15.087 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:15.090+0000 7fdc31110640 1 -- 172.21.15.123:0/2720548144 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdc18005220 msgr2=0x7fdc1800b9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:51:15.088 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:15.090+0000 7fdc31110640 1 --2- 172.21.15.123:0/2720548144 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdc18005220 0x7fdc1800b9d0 secure :-1 s=READY pgs=1728 cs=0 l=1 rev1=1 crypto rx=0x7fdc2400c610 tx=0x7fdc2401d2e0 comp rx=0 tx=0).stop 2024-09-16T15:51:15.088 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:15.090+0000 7fdc31110640 1 -- 172.21.15.123:0/2720548144 shutdown_connections 2024-09-16T15:51:15.088 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:15.090+0000 7fdc31110640 1 --2- 172.21.15.123:0/2720548144 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdc18005220 0x7fdc1800b9d0 unknown :-1 s=CLOSED pgs=1728 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:15.088 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:15.090+0000 7fdc31110640 1 -- 172.21.15.123:0/2720548144 >> 172.21.15.123:0/2720548144 conn(0x7fdc2c0b1710 msgr2=0x7fdc2c0b1b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:51:15.088 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:15.090+0000 7fdc31110640 1 -- 172.21.15.123:0/2720548144 shutdown_connections 2024-09-16T15:51:15.088 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:15.090+0000 7fdc31110640 1 -- 172.21.15.123:0/2720548144 wait complete. 2024-09-16T15:51:15.089 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:15.090+0000 7fdc31110640 1 Processor -- start 2024-09-16T15:51:15.089 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:15.090+0000 7fdc31110640 1 -- start start 2024-09-16T15:51:15.090 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:15.090+0000 7fdc31110640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fdc18005220 0x7fdc2c1f86c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:51:15.090 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:15.090+0000 7fdc31110640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fdc2c0c8ab0 0x7fdc2c15aa70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:51:15.091 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:15.090+0000 7fdc31110640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdc2c0afcd0 0x7fdc2c0b0150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:51:15.091 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:15.090+0000 7fdc31110640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fdc2c15e1f0 con 0x7fdc2c0afcd0 2024-09-16T15:51:15.091 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:15.090+0000 7fdc31110640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fdc2c0ca8d0 con 0x7fdc18005220 2024-09-16T15:51:15.091 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:15.090+0000 7fdc31110640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fdc2c15e370 con 0x7fdc2c0c8ab0 2024-09-16T15:51:15.091 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:15.090+0000 7fdc2b59e640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdc2c0afcd0 0x7fdc2c0b0150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:51:15.091 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:15.090+0000 7fdc2b59e640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdc2c0afcd0 0x7fdc2c0b0150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:45016/0 (socket says 172.21.15.123:45016) 2024-09-16T15:51:15.091 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:15.090+0000 7fdc2b59e640 1 -- 172.21.15.123:0/726206885 learned_addr learned my addr 172.21.15.123:0/726206885 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:51:15.091 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:15.090+0000 7fdc2ad9d640 1 --2- 172.21.15.123:0/726206885 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fdc18005220 0x7fdc2c1f86c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:51:15.091 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:15.090+0000 7fdc2259c640 1 --2- 172.21.15.123:0/726206885 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fdc2c0c8ab0 0x7fdc2c15aa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:51:15.091 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:15.090+0000 7fdc2b59e640 1 -- 172.21.15.123:0/726206885 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fdc2c0c8ab0 msgr2=0x7fdc2c15aa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:51:15.091 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:15.090+0000 7fdc2b59e640 1 --2- 172.21.15.123:0/726206885 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fdc2c0c8ab0 0x7fdc2c15aa70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:15.091 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:15.090+0000 7fdc2b59e640 1 -- 172.21.15.123:0/726206885 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fdc18005220 msgr2=0x7fdc2c1f86c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:51:15.092 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:15.090+0000 7fdc2b59e640 1 --2- 172.21.15.123:0/726206885 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fdc18005220 0x7fdc2c1f86c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:15.092 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:15.090+0000 7fdc2b59e640 1 -- 172.21.15.123:0/726206885 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdc2c1f8c00 con 0x7fdc2c0afcd0 2024-09-16T15:51:15.092 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:15.090+0000 7fdc2259c640 1 --2- 172.21.15.123:0/726206885 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fdc2c0c8ab0 0x7fdc2c15aa70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T15:51:15.092 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:15.090+0000 7fdc2ad9d640 1 --2- 172.21.15.123:0/726206885 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fdc18005220 0x7fdc2c1f86c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T15:51:15.092 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:15.090+0000 7fdc2b59e640 1 --2- 172.21.15.123:0/726206885 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdc2c0afcd0 0x7fdc2c0b0150 secure :-1 s=READY pgs=1729 cs=0 l=1 rev1=1 crypto rx=0x7fdc2401b9e0 tx=0x7fdc240020e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:51:15.092 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:15.090+0000 7fdc23fff640 1 -- 172.21.15.123:0/726206885 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc24003e50 con 0x7fdc2c0afcd0 2024-09-16T15:51:15.092 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:15.090+0000 7fdc31110640 1 -- 172.21.15.123:0/726206885 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdc2c14e600 con 0x7fdc2c0afcd0 2024-09-16T15:51:15.092 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:15.090+0000 7fdc23fff640 1 -- 172.21.15.123:0/726206885 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fdc24006160 con 0x7fdc2c0afcd0 2024-09-16T15:51:15.092 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:15.090+0000 7fdc23fff640 1 -- 172.21.15.123:0/726206885 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc24006570 con 0x7fdc2c0afcd0 2024-09-16T15:51:15.092 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:15.090+0000 7fdc31110640 1 -- 172.21.15.123:0/726206885 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdc2c0c7cf0 con 0x7fdc2c0afcd0 2024-09-16T15:51:15.093 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:15.094+0000 7fdc31110640 1 -- 172.21.15.123:0/726206885 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdc2c06e890 con 0x7fdc2c0afcd0 2024-09-16T15:51:15.093 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:15.094+0000 7fdc23fff640 1 -- 172.21.15.123:0/726206885 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 24) ==== 97845+0+0 (secure 0 0 0) 0x7fdc240ab030 con 0x7fdc2c0afcd0 2024-09-16T15:51:15.093 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:15.094+0000 7fdc23fff640 1 --2- 172.21.15.123:0/726206885 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fdc0c0738d0 0x7fdc0c075d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:51:15.093 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:15.094+0000 7fdc23fff640 1 -- 172.21.15.123:0/726206885 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(154..154 src has 1..154) ==== 5255+0+0 (secure 0 0 0) 0x7fdc2412e4a0 con 0x7fdc2c0afcd0 2024-09-16T15:51:15.094 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:15.094+0000 7fdc2ad9d640 1 --2- 172.21.15.123:0/726206885 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fdc0c0738d0 0x7fdc0c075d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:51:15.094 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:15.094+0000 7fdc2ad9d640 1 --2- 172.21.15.123:0/726206885 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fdc0c0738d0 0x7fdc0c075d90 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7fdc1c006e20 tx=0x7fdc1c006190 comp rx=0 tx=0).ready entity=mgr.4150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:51:15.097 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:15.098+0000 7fdc23fff640 1 -- 172.21.15.123:0/726206885 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7fdc2c1f8c00 con 0x7fdc2c0afcd0 2024-09-16T15:51:15.268 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:15.270+0000 7fdc31110640 1 -- 172.21.15.123:0/726206885 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) -- 0x7fdc2c0cc680 con 0x7fdc2c0afcd0 2024-09-16T15:51:15.917 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:15.918+0000 7fdc23fff640 1 -- 172.21.15.123:0/726206885 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v155) ==== 170+0+0 (secure 0 0 0) 0x7fdc24017090 con 0x7fdc2c0afcd0 2024-09-16T15:51:15.918 INFO:teuthology.orchestra.run.smithi123.stderr:pool 'cephfs_data' created 2024-09-16T15:51:15.921 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:15.922+0000 7fdc31110640 1 -- 172.21.15.123:0/726206885 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fdc0c0738d0 msgr2=0x7fdc0c075d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:51:15.921 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:15.922+0000 7fdc31110640 1 --2- 172.21.15.123:0/726206885 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fdc0c0738d0 0x7fdc0c075d90 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7fdc1c006e20 tx=0x7fdc1c006190 comp rx=0 tx=0).stop 2024-09-16T15:51:15.921 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:15.922+0000 7fdc31110640 1 -- 172.21.15.123:0/726206885 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdc2c0afcd0 msgr2=0x7fdc2c0b0150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:51:15.921 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:15.922+0000 7fdc31110640 1 --2- 172.21.15.123:0/726206885 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdc2c0afcd0 0x7fdc2c0b0150 secure :-1 s=READY pgs=1729 cs=0 l=1 rev1=1 crypto rx=0x7fdc2401b9e0 tx=0x7fdc240020e0 comp rx=0 tx=0).stop 2024-09-16T15:51:15.921 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:15.922+0000 7fdc31110640 1 -- 172.21.15.123:0/726206885 shutdown_connections 2024-09-16T15:51:15.922 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:15.922+0000 7fdc31110640 1 --2- 172.21.15.123:0/726206885 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fdc0c0738d0 0x7fdc0c075d90 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:15.922 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:15.922+0000 7fdc31110640 1 --2- 172.21.15.123:0/726206885 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdc2c0afcd0 0x7fdc2c0b0150 unknown :-1 s=CLOSED pgs=1729 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:15.922 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:15.922+0000 7fdc31110640 1 --2- 172.21.15.123:0/726206885 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fdc2c0c8ab0 0x7fdc2c15aa70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:15.922 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:15.922+0000 7fdc31110640 1 --2- 172.21.15.123:0/726206885 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fdc18005220 0x7fdc2c1f86c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:15.922 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:15.922+0000 7fdc31110640 1 -- 172.21.15.123:0/726206885 >> 172.21.15.123:0/726206885 conn(0x7fdc2c0b1710 msgr2=0x7fdc2c06eaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:51:15.922 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:15.922+0000 7fdc31110640 1 -- 172.21.15.123:0/726206885 shutdown_connections 2024-09-16T15:51:15.922 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:15.922+0000 7fdc31110640 1 -- 172.21.15.123:0/726206885 wait complete. 2024-09-16T15:51:15.953 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-09-16T15:51:16.077 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:16.078+0000 7f1fb6b32640 1 -- 172.21.15.123:0/4175312922 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f1fb00ccf00 msgr2=0x7f1fb0073a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:51:16.077 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:16.078+0000 7f1fb6b32640 1 --2- 172.21.15.123:0/4175312922 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f1fb00ccf00 0x7f1fb0073a40 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f1fa8009c80 tx=0x7f1fa801cce0 comp rx=0 tx=0).stop 2024-09-16T15:51:16.077 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:16.078+0000 7f1fb6b32640 1 -- 172.21.15.123:0/4175312922 shutdown_connections 2024-09-16T15:51:16.077 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:16.078+0000 7f1fb6b32640 1 --2- 172.21.15.123:0/4175312922 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f1fb00ccf00 0x7f1fb0073a40 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:16.078 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:16.078+0000 7f1fb6b32640 1 --2- 172.21.15.123:0/4175312922 >> v2:172.21.15.184:3300/0 conn(0x7f1fb015c9b0 0x7f1fb0073500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:16.078 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:16.078+0000 7f1fb6b32640 1 -- 172.21.15.123:0/4175312922 >> 172.21.15.123:0/4175312922 conn(0x7f1fb00d0bc0 msgr2=0x7f1fb00d0fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:51:16.078 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:16.078+0000 7f1fb6b32640 1 -- 172.21.15.123:0/4175312922 shutdown_connections 2024-09-16T15:51:16.078 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:16.078+0000 7f1fb6b32640 1 -- 172.21.15.123:0/4175312922 wait complete. 2024-09-16T15:51:16.078 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:16.078+0000 7f1fb6b32640 1 Processor -- start 2024-09-16T15:51:16.078 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:16.078+0000 7f1fb6b32640 1 -- start start 2024-09-16T15:51:16.080 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:16.082+0000 7f1fb6b32640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1fb00ccf00 0x7f1fb01f6930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:51:16.080 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:16.082+0000 7f1fb6b32640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1fb015c9b0 0x7f1fb01f8580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:51:16.080 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:16.082+0000 7f1fb6b32640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f1fb015cd90 0x7f1fb014ef50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:51:16.081 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:16.082+0000 7f1fb6b32640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f1fb006cd70 con 0x7f1fb015c9b0 2024-09-16T15:51:16.081 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:16.082+0000 7f1fb6b32640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f1fb006cbf0 con 0x7f1fb00ccf00 2024-09-16T15:51:16.081 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:16.082+0000 7f1fb6b32640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f1fb006cef0 con 0x7f1fb015cd90 2024-09-16T15:51:16.081 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:16.082+0000 7f1faffff640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1fb015c9b0 0x7f1fb01f8580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:51:16.081 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:16.082+0000 7f1faffff640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1fb015c9b0 0x7f1fb01f8580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:45020/0 (socket says 172.21.15.123:45020) 2024-09-16T15:51:16.081 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:16.082+0000 7f1faffff640 1 -- 172.21.15.123:0/914999140 learned_addr learned my addr 172.21.15.123:0/914999140 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:51:16.081 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:16.082+0000 7f1fb48ce640 1 --2- 172.21.15.123:0/914999140 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1fb00ccf00 0x7f1fb01f6930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:51:16.081 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:16.082+0000 7f1fb50cf640 1 --2- 172.21.15.123:0/914999140 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f1fb015cd90 0x7f1fb014ef50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:51:16.081 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:16.082+0000 7f1faffff640 1 -- 172.21.15.123:0/914999140 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f1fb015cd90 msgr2=0x7f1fb014ef50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:51:16.081 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:16.082+0000 7f1faffff640 1 --2- 172.21.15.123:0/914999140 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f1fb015cd90 0x7f1fb014ef50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:16.081 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:16.082+0000 7f1faffff640 1 -- 172.21.15.123:0/914999140 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1fb00ccf00 msgr2=0x7f1fb01f6930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:51:16.081 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:16.082+0000 7f1faffff640 1 --2- 172.21.15.123:0/914999140 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1fb00ccf00 0x7f1fb01f6930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:16.081 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:16.082+0000 7f1faffff640 1 -- 172.21.15.123:0/914999140 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1fb0148e70 con 0x7f1fb015c9b0 2024-09-16T15:51:16.082 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:16.082+0000 7f1fb50cf640 1 --2- 172.21.15.123:0/914999140 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f1fb015cd90 0x7f1fb014ef50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T15:51:16.082 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:16.082+0000 7f1faffff640 1 --2- 172.21.15.123:0/914999140 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1fb015c9b0 0x7f1fb01f8580 secure :-1 s=READY pgs=1730 cs=0 l=1 rev1=1 crypto rx=0x7f1fa400c3f0 tx=0x7f1fa400c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:51:16.082 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:16.082+0000 7f1fad7fa640 1 -- 172.21.15.123:0/914999140 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1fa40052d0 con 0x7f1fb015c9b0 2024-09-16T15:51:16.082 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:16.082+0000 7f1fb6b32640 1 -- 172.21.15.123:0/914999140 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1fb01f8ac0 con 0x7f1fb015c9b0 2024-09-16T15:51:16.082 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:16.082+0000 7f1fad7fa640 1 -- 172.21.15.123:0/914999140 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f1fa4007020 con 0x7f1fb015c9b0 2024-09-16T15:51:16.082 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:16.082+0000 7f1fad7fa640 1 -- 172.21.15.123:0/914999140 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1fa4007470 con 0x7f1fb015c9b0 2024-09-16T15:51:16.082 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:16.082+0000 7f1fb6b32640 1 -- 172.21.15.123:0/914999140 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1fb014fb30 con 0x7f1fb015c9b0 2024-09-16T15:51:16.083 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:16.082+0000 7f1fb6b32640 1 -- 172.21.15.123:0/914999140 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1fb00cc7f0 con 0x7f1fb015c9b0 2024-09-16T15:51:16.083 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:16.082+0000 7f1fad7fa640 1 -- 172.21.15.123:0/914999140 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 24) ==== 97845+0+0 (secure 0 0 0) 0x7f1fa40067e0 con 0x7f1fb015c9b0 2024-09-16T15:51:16.083 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:16.082+0000 7f1fad7fa640 1 --2- 172.21.15.123:0/914999140 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f1f9c073730 0x7f1f9c075bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:51:16.083 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:16.082+0000 7f1fad7fa640 1 -- 172.21.15.123:0/914999140 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(155..155 src has 1..155) ==== 5637+0+0 (secure 0 0 0) 0x7f1fa4091fa0 con 0x7f1fb015c9b0 2024-09-16T15:51:16.083 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:16.082+0000 7f1fb48ce640 1 --2- 172.21.15.123:0/914999140 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f1f9c073730 0x7f1f9c075bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:51:16.083 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:16.082+0000 7f1fb48ce640 1 --2- 172.21.15.123:0/914999140 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f1f9c073730 0x7f1f9c075bf0 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f1fa0009be0 tx=0x7f1fa00091e0 comp rx=0 tx=0).ready entity=mgr.4150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:51:16.086 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:16.086+0000 7f1fad7fa640 1 -- 172.21.15.123:0/914999140 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f1fa405e650 con 0x7f1fb015c9b0 2024-09-16T15:51:16.297 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:16.298+0000 7f1fb6b32640 1 -- 172.21.15.123:0/914999140 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) -- 0x7f1fb0149040 con 0x7f1fb015c9b0 2024-09-16T15:51:16.989 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:16.990+0000 7f1fad7fa640 1 -- 172.21.15.123:0/914999140 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 24 and data pool 25 v101) ==== 174+0+0 (secure 0 0 0) 0x7f1fa4063080 con 0x7f1fb015c9b0 2024-09-16T15:51:16.991 INFO:teuthology.orchestra.run.smithi123.stderr:new fs with metadata pool 24 and data pool 25 2024-09-16T15:51:16.998 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:16.998+0000 7f1fb6b32640 1 -- 172.21.15.123:0/914999140 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f1f9c073730 msgr2=0x7f1f9c075bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:51:16.998 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:16.998+0000 7f1fb6b32640 1 --2- 172.21.15.123:0/914999140 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f1f9c073730 0x7f1f9c075bf0 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f1fa0009be0 tx=0x7f1fa00091e0 comp rx=0 tx=0).stop 2024-09-16T15:51:16.998 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:16.998+0000 7f1fb6b32640 1 -- 172.21.15.123:0/914999140 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1fb015c9b0 msgr2=0x7f1fb01f8580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:51:16.998 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:16.998+0000 7f1fb6b32640 1 --2- 172.21.15.123:0/914999140 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1fb015c9b0 0x7f1fb01f8580 secure :-1 s=READY pgs=1730 cs=0 l=1 rev1=1 crypto rx=0x7f1fa400c3f0 tx=0x7f1fa400c8b0 comp rx=0 tx=0).stop 2024-09-16T15:51:17.002 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:17.002+0000 7f1fb6b32640 1 -- 172.21.15.123:0/914999140 shutdown_connections 2024-09-16T15:51:17.002 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:17.002+0000 7f1fb6b32640 1 --2- 172.21.15.123:0/914999140 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f1f9c073730 0x7f1f9c075bf0 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:17.002 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:17.002+0000 7f1fb6b32640 1 --2- 172.21.15.123:0/914999140 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f1fb015cd90 0x7f1fb014ef50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:17.002 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:17.002+0000 7f1fb6b32640 1 --2- 172.21.15.123:0/914999140 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1fb015c9b0 0x7f1fb01f8580 unknown :-1 s=CLOSED pgs=1730 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:17.003 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:17.002+0000 7f1fb6b32640 1 --2- 172.21.15.123:0/914999140 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1fb00ccf00 0x7f1fb01f6930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:17.003 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:17.002+0000 7f1fb6b32640 1 -- 172.21.15.123:0/914999140 >> 172.21.15.123:0/914999140 conn(0x7f1fb00d0bc0 msgr2=0x7f1fb0154b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:51:17.003 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:17.002+0000 7f1fb6b32640 1 -- 172.21.15.123:0/914999140 shutdown_connections 2024-09-16T15:51:17.003 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:17.002+0000 7f1fb6b32640 1 -- 172.21.15.123:0/914999140 wait complete. 2024-09-16T15:51:17.033 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-16T15:51:17.156 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:17.158+0000 7f64752f4640 1 -- 172.21.15.123:0/866415355 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f647015e160 msgr2=0x7f6470073f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:51:17.156 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:17.158+0000 7f64752f4640 1 --2- 172.21.15.123:0/866415355 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f647015e160 0x7f6470073f30 secure :-1 s=READY pgs=1732 cs=0 l=1 rev1=1 crypto rx=0x7f6470073c20 tx=0x7f646001c7c0 comp rx=0 tx=0).stop 2024-09-16T15:51:17.156 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:17.158+0000 7f64752f4640 1 -- 172.21.15.123:0/866415355 shutdown_connections 2024-09-16T15:51:17.156 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:17.158+0000 7f64752f4640 1 --2- 172.21.15.123:0/866415355 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f647015e160 0x7f6470073f30 unknown :-1 s=CLOSED pgs=1732 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:17.156 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:17.158+0000 7f64752f4640 1 -- 172.21.15.123:0/866415355 >> 172.21.15.123:0/866415355 conn(0x7f64700cfb10 msgr2=0x7f64700cff20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:51:17.156 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:17.158+0000 7f64752f4640 1 -- 172.21.15.123:0/866415355 shutdown_connections 2024-09-16T15:51:17.156 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:17.158+0000 7f64752f4640 1 -- 172.21.15.123:0/866415355 wait complete. 2024-09-16T15:51:17.157 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:17.158+0000 7f64752f4640 1 Processor -- start 2024-09-16T15:51:17.157 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:17.158+0000 7f64752f4640 1 -- start start 2024-09-16T15:51:17.158 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:17.158+0000 7f64752f4640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6470074470 0x7f647014aee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:51:17.158 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:17.158+0000 7f64752f4640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f647015e160 0x7f64701fbfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:51:17.158 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:17.158+0000 7f64752f4640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f647015e820 0x7f64701f8030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:51:17.159 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:17.158+0000 7f64752f4640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f64701697f0 con 0x7f647015e820 2024-09-16T15:51:17.159 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:17.158+0000 7f64752f4640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f64700cf640 con 0x7f6470074470 2024-09-16T15:51:17.159 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:17.158+0000 7f64752f4640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f6470169970 con 0x7f647015e160 2024-09-16T15:51:17.159 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:17.158+0000 7f646f7fe640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f647015e820 0x7f64701f8030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:51:17.159 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:17.158+0000 7f646f7fe640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f647015e820 0x7f64701f8030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:45038/0 (socket says 172.21.15.123:45038) 2024-09-16T15:51:17.159 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:17.158+0000 7f646f7fe640 1 -- 172.21.15.123:0/1436776498 learned_addr learned my addr 172.21.15.123:0/1436776498 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:51:17.159 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:17.158+0000 7f646e7fc640 1 --2- 172.21.15.123:0/1436776498 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f647015e160 0x7f64701fbfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:51:17.159 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:17.158+0000 7f646f7fe640 1 -- 172.21.15.123:0/1436776498 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f647015e160 msgr2=0x7f64701fbfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:51:17.159 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:17.158+0000 7f646effd640 1 --2- 172.21.15.123:0/1436776498 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6470074470 0x7f647014aee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:51:17.159 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:17.158+0000 7f646f7fe640 1 --2- 172.21.15.123:0/1436776498 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f647015e160 0x7f64701fbfe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:17.159 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:17.158+0000 7f646f7fe640 1 -- 172.21.15.123:0/1436776498 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6470074470 msgr2=0x7f647014aee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:51:17.159 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:17.158+0000 7f646f7fe640 1 --2- 172.21.15.123:0/1436776498 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6470074470 0x7f647014aee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:17.160 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:17.158+0000 7f646f7fe640 1 -- 172.21.15.123:0/1436776498 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f64701f8b50 con 0x7f647015e820 2024-09-16T15:51:17.160 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:17.158+0000 7f646f7fe640 1 --2- 172.21.15.123:0/1436776498 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f647015e820 0x7f64701f8030 secure :-1 s=READY pgs=1733 cs=0 l=1 rev1=1 crypto rx=0x7f646800b990 tx=0x7f646800be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:51:17.160 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:17.158+0000 7f64577fe640 1 -- 172.21.15.123:0/1436776498 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6468019070 con 0x7f647015e820 2024-09-16T15:51:17.160 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:17.158+0000 7f64577fe640 1 -- 172.21.15.123:0/1436776498 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f646800cc30 con 0x7f647015e820 2024-09-16T15:51:17.160 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:17.158+0000 7f64752f4640 1 -- 172.21.15.123:0/1436776498 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f64701f8e30 con 0x7f647015e820 2024-09-16T15:51:17.160 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:17.158+0000 7f64577fe640 1 -- 172.21.15.123:0/1436776498 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f646800ce60 con 0x7f647015e820 2024-09-16T15:51:17.161 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:17.158+0000 7f64752f4640 1 -- 172.21.15.123:0/1436776498 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f647014b420 con 0x7f647015e820 2024-09-16T15:51:17.161 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:17.162+0000 7f64752f4640 1 -- 172.21.15.123:0/1436776498 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f64700d06c0 con 0x7f647015e820 2024-09-16T15:51:17.161 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:17.162+0000 7f64577fe640 1 -- 172.21.15.123:0/1436776498 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 24) ==== 97845+0+0 (secure 0 0 0) 0x7f646801c070 con 0x7f647015e820 2024-09-16T15:51:17.161 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:17.162+0000 7f64577fe640 1 --2- 172.21.15.123:0/1436776498 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f645c073730 0x7f645c075bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:51:17.161 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:17.162+0000 7f646effd640 1 --2- 172.21.15.123:0/1436776498 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f645c073730 0x7f645c075bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:51:17.161 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:17.162+0000 7f64577fe640 1 -- 172.21.15.123:0/1436776498 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(156..156 src has 1..156) ==== 5737+0+0 (secure 0 0 0) 0x7f6468092400 con 0x7f647015e820 2024-09-16T15:51:17.161 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:17.162+0000 7f646effd640 1 --2- 172.21.15.123:0/1436776498 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f645c073730 0x7f645c075bf0 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f64600af000 tx=0x7f646001b000 comp rx=0 tx=0).ready entity=mgr.4150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:51:17.165 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:17.166+0000 7f64577fe640 1 -- 172.21.15.123:0/1436776498 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f646805f100 con 0x7f647015e820 2024-09-16T15:51:17.338 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:17.338+0000 7f64752f4640 1 -- 172.21.15.123:0/1436776498 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f6470073f10 con 0x7f647015e820 2024-09-16T15:51:17.341 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:17.342+0000 7f64577fe640 1 -- 172.21.15.123:0/1436776498 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v156) ==== 74+0+16856 (secure 0 0 0) 0x7f646805ea00 con 0x7f647015e820 2024-09-16T15:51:17.342 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-09-16T15:51:17.342 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":156,"fsid":"56077f89-18c8-486f-ae7e-eecc0bb7822c","created":"2024-09-16T15:42:26.364892+0000","modified":"2024-09-16T15:51:16.301997+0000","last_up_change":"2024-09-16T15:42:38.707234+0000","last_in_change":"2024-09-16T15:42:29.423662+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":25,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-16T15:42:39.287764+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":24,"pool_name":"cephfs_metadata","create_time":"2024-09-16T15:51:13.953990+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"156","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":25,"pool_name":"cephfs_data","create_time":"2024-09-16T15:51:15.281388+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"156","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9e7c1955-a607-4671-9be7-f2ef760ceeaa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":155,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6800","nonce":3882044969},{"type":"v1","addr":"172.21.15.123:6801","nonce":3882044969}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6802","nonce":3882044969},{"type":"v1","addr":"172.21.15.123:6803","nonce":3882044969}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6806","nonce":3882044969},{"type":"v1","addr":"172.21.15.123:6807","nonce":3882044969}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6804","nonce":3882044969},{"type":"v1","addr":"172.21.15.123:6805","nonce":3882044969}]},"public_addr":"172.21.15.123:6801/3882044969","cluster_addr":"172.21.15.123:6803/3882044969","heartbeat_back_addr":"172.21.15.123:6807/3882044969","heartbeat_front_addr":"172.21.15.123:6805/3882044969","state":["exists","up"]},{"osd":1,"uuid":"c540cfde-0977-4742-9b17-fd634668e010","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":155,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6816","nonce":4037001850},{"type":"v1","addr":"172.21.15.123:6817","nonce":4037001850}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6818","nonce":4037001850},{"type":"v1","addr":"172.21.15.123:6819","nonce":4037001850}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6822","nonce":4037001850},{"type":"v1","addr":"172.21.15.123:6823","nonce":4037001850}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6820","nonce":4037001850},{"type":"v1","addr":"172.21.15.123:6821","nonce":4037001850}]},"public_addr":"172.21.15.123:6817/4037001850","cluster_addr":"172.21.15.123:6819/4037001850","heartbeat_back_addr":"172.21.15.123:6823/4037001850","heartbeat_front_addr":"172.21.15.123:6821/4037001850","state":["exists","up"]},{"osd":2,"uuid":"9864a496-f169-486b-95c4-1d59162793a7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":155,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6824","nonce":1332143088},{"type":"v1","addr":"172.21.15.123:6825","nonce":1332143088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6826","nonce":1332143088},{"type":"v1","addr":"172.21.15.123:6827","nonce":1332143088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6830","nonce":1332143088},{"type":"v1","addr":"172.21.15.123:6831","nonce":1332143088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6828","nonce":1332143088},{"type":"v1","addr":"172.21.15.123:6829","nonce":1332143088}]},"public_addr":"172.21.15.123:6825/1332143088","cluster_addr":"172.21.15.123:6827/1332143088","heartbeat_back_addr":"172.21.15.123:6831/1332143088","heartbeat_front_addr":"172.21.15.123:6829/1332143088","state":["exists","up"]},{"osd":3,"uuid":"c6d1f817-f074-4325-95e1-578eba13edda","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":155,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6808","nonce":988586998},{"type":"v1","addr":"172.21.15.123:6809","nonce":988586998}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6810","nonce":988586998},{"type":"v1","addr":"172.21.15.123:6811","nonce":988586998}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6814","nonce":988586998},{"type":"v1","addr":"172.21.15.123:6815","nonce":988586998}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6812","nonce":988586998},{"type":"v1","addr":"172.21.15.123:6813","nonce":988586998}]},"public_addr":"172.21.15.123:6809/988586998","cluster_addr":"172.21.15.123:6811/988586998","heartbeat_back_addr":"172.21.15.123:6815/988586998","heartbeat_front_addr":"172.21.15.123:6813/988586998","state":["exists","up"]},{"osd":4,"uuid":"200a8df4-50ef-40a5-b48a-58981269c086","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":155,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6816","nonce":3387813890},{"type":"v1","addr":"172.21.15.184:6817","nonce":3387813890}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6818","nonce":3387813890},{"type":"v1","addr":"172.21.15.184:6819","nonce":3387813890}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6822","nonce":3387813890},{"type":"v1","addr":"172.21.15.184:6823","nonce":3387813890}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6820","nonce":3387813890},{"type":"v1","addr":"172.21.15.184:6821","nonce":3387813890}]},"public_addr":"172.21.15.184:6817/3387813890","cluster_addr":"172.21.15.184:6819/3387813890","heartbeat_back_addr":"172.21.15.184:6823/3387813890","heartbeat_front_addr":"172.21.15.184:6821/3387813890","state":["exists","up"]},{"osd":5,"uuid":"32d5a900-da3d-412c-bc19-13861a2368d8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":155,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6808","nonce":2680916017},{"type":"v1","addr":"172.21.15.184:6809","nonce":2680916017}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6810","nonce":2680916017},{"type":"v1","addr":"172.21.15.184:6811","nonce":2680916017}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6814","nonce":2680916017},{"type":"v1","addr":"172.21.15.184:6815","nonce":2680916017}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6812","nonce":2680916017},{"type":"v1","addr":"172.21.15.184:6813","nonce":2680916017}]},"public_addr":"172.21.15.184:6809/2680916017","cluster_addr":"172.21.15.184:6811/2680916017","heartbeat_back_addr":"172.21.15.184:6815/2680916017","heartbeat_front_addr":"172.21.15.184:6813/2680916017","state":["exists","up"]},{"osd":6,"uuid":"cde6f39e-aa1b-4260-ac19-774ee6aaca72","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":155,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6824","nonce":2183120457},{"type":"v1","addr":"172.21.15.184:6825","nonce":2183120457}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6826","nonce":2183120457},{"type":"v1","addr":"172.21.15.184:6827","nonce":2183120457}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6830","nonce":2183120457},{"type":"v1","addr":"172.21.15.184:6831","nonce":2183120457}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6828","nonce":2183120457},{"type":"v1","addr":"172.21.15.184:6829","nonce":2183120457}]},"public_addr":"172.21.15.184:6825/2183120457","cluster_addr":"172.21.15.184:6827/2183120457","heartbeat_back_addr":"172.21.15.184:6831/2183120457","heartbeat_front_addr":"172.21.15.184:6829/2183120457","state":["exists","up"]},{"osd":7,"uuid":"4f381778-eee1-47c6-804f-df0e6b33e313","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":155,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6800","nonce":4102957285},{"type":"v1","addr":"172.21.15.184:6801","nonce":4102957285}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6802","nonce":4102957285},{"type":"v1","addr":"172.21.15.184:6803","nonce":4102957285}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6806","nonce":4102957285},{"type":"v1","addr":"172.21.15.184:6807","nonce":4102957285}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6804","nonce":4102957285},{"type":"v1","addr":"172.21.15.184:6805","nonce":4102957285}]},"public_addr":"172.21.15.184:6801/4102957285","cluster_addr":"172.21.15.184:6803/4102957285","heartbeat_back_addr":"172.21.15.184:6807/4102957285","heartbeat_front_addr":"172.21.15.184:6805/4102957285","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:35.972210+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.222007+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.198617+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.385663+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.358935+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.156066+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.752973+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:35.682846+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":17,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T15:51:17.343 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:17.346+0000 7f64752f4640 1 -- 172.21.15.123:0/1436776498 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f645c073730 msgr2=0x7f645c075bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:51:17.344 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:17.346+0000 7f64752f4640 1 --2- 172.21.15.123:0/1436776498 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f645c073730 0x7f645c075bf0 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f64600af000 tx=0x7f646001b000 comp rx=0 tx=0).stop 2024-09-16T15:51:17.344 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:17.346+0000 7f64752f4640 1 -- 172.21.15.123:0/1436776498 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f647015e820 msgr2=0x7f64701f8030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:51:17.344 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:17.346+0000 7f64752f4640 1 --2- 172.21.15.123:0/1436776498 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f647015e820 0x7f64701f8030 secure :-1 s=READY pgs=1733 cs=0 l=1 rev1=1 crypto rx=0x7f646800b990 tx=0x7f646800be50 comp rx=0 tx=0).stop 2024-09-16T15:51:17.344 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:17.346+0000 7f64752f4640 1 -- 172.21.15.123:0/1436776498 shutdown_connections 2024-09-16T15:51:17.344 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:17.346+0000 7f64752f4640 1 --2- 172.21.15.123:0/1436776498 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f645c073730 0x7f645c075bf0 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:17.344 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:17.346+0000 7f64752f4640 1 --2- 172.21.15.123:0/1436776498 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f647015e820 0x7f64701f8030 unknown :-1 s=CLOSED pgs=1733 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:17.344 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:17.346+0000 7f64752f4640 1 --2- 172.21.15.123:0/1436776498 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f647015e160 0x7f64701fbfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:17.344 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:17.346+0000 7f64752f4640 1 --2- 172.21.15.123:0/1436776498 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6470074470 0x7f647014aee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:17.344 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:17.346+0000 7f64752f4640 1 -- 172.21.15.123:0/1436776498 >> 172.21.15.123:0/1436776498 conn(0x7f64700cfb10 msgr2=0x7f64700c54b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:51:17.344 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:17.346+0000 7f64752f4640 1 -- 172.21.15.123:0/1436776498 shutdown_connections 2024-09-16T15:51:17.344 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:17.346+0000 7f64752f4640 1 -- 172.21.15.123:0/1436776498 wait complete. 2024-09-16T15:51:17.362 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-16T15:51:17.483 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:17.486+0000 7f4f06b13640 1 -- 172.21.15.123:0/2291163064 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4f00161620 msgr2=0x7f4f000d1da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:51:17.483 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:17.486+0000 7f4f06b13640 1 --2- 172.21.15.123:0/2291163064 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4f00161620 0x7f4f000d1da0 secure :-1 s=READY pgs=1734 cs=0 l=1 rev1=1 crypto rx=0x7f4ef00098f0 tx=0x7f4ef001c790 comp rx=0 tx=0).stop 2024-09-16T15:51:17.484 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:17.486+0000 7f4f06b13640 1 -- 172.21.15.123:0/2291163064 shutdown_connections 2024-09-16T15:51:17.484 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:17.486+0000 7f4f06b13640 1 --2- 172.21.15.123:0/2291163064 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f4f000b0a50 0x7f4f000b0ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:17.484 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:17.486+0000 7f4f06b13640 1 --2- 172.21.15.123:0/2291163064 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4f00161620 0x7f4f000d1da0 unknown :-1 s=CLOSED pgs=1734 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:17.484 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:17.486+0000 7f4f06b13640 1 -- 172.21.15.123:0/2291163064 >> 172.21.15.123:0/2291163064 conn(0x7f4f000d3ea0 msgr2=0x7f4f000d42b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:51:17.484 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:17.486+0000 7f4f06b13640 1 -- 172.21.15.123:0/2291163064 shutdown_connections 2024-09-16T15:51:17.484 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:17.486+0000 7f4f06b13640 1 -- 172.21.15.123:0/2291163064 wait complete. 2024-09-16T15:51:17.484 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:17.486+0000 7f4f06b13640 1 Processor -- start 2024-09-16T15:51:17.485 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:17.486+0000 7f4f06b13640 1 -- start start 2024-09-16T15:51:17.487 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:17.486+0000 7f4f06b13640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4f000b0a50 0x7f4f0014e5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:51:17.487 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:17.486+0000 7f4f06b13640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4f000b2b00 0x7f4f000caf70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:51:17.488 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:17.486+0000 7f4f06b13640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f4f00161620 0x7f4f0015b720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:51:17.488 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:17.486+0000 7f4f06b13640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f4f0016af70 con 0x7f4f000b2b00 2024-09-16T15:51:17.488 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:17.486+0000 7f4f06b13640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f4f000d2c10 con 0x7f4f000b0a50 2024-09-16T15:51:17.488 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:17.486+0000 7f4f06b13640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f4f0016b0f0 con 0x7f4f00161620 2024-09-16T15:51:17.488 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:17.486+0000 7f4efffff640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4f000b2b00 0x7f4f000caf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:51:17.489 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:17.486+0000 7f4efffff640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4f000b2b00 0x7f4f000caf70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:45064/0 (socket says 172.21.15.123:45064) 2024-09-16T15:51:17.489 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:17.486+0000 7f4efffff640 1 -- 172.21.15.123:0/3428683191 learned_addr learned my addr 172.21.15.123:0/3428683191 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:51:17.489 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:17.486+0000 7f4f048af640 1 --2- 172.21.15.123:0/3428683191 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4f000b0a50 0x7f4f0014e5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:51:17.489 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:17.486+0000 7f4efffff640 1 -- 172.21.15.123:0/3428683191 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f4f00161620 msgr2=0x7f4f0015b720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:51:17.490 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:17.486+0000 7f4f050b0640 1 --2- 172.21.15.123:0/3428683191 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f4f00161620 0x7f4f0015b720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:51:17.490 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:17.486+0000 7f4efffff640 1 --2- 172.21.15.123:0/3428683191 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f4f00161620 0x7f4f0015b720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:17.490 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:17.486+0000 7f4efffff640 1 -- 172.21.15.123:0/3428683191 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4f000b0a50 msgr2=0x7f4f0014e5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:51:17.490 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:17.486+0000 7f4efffff640 1 --2- 172.21.15.123:0/3428683191 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4f000b0a50 0x7f4f0014e5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:17.490 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:17.486+0000 7f4efffff640 1 -- 172.21.15.123:0/3428683191 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4f001fb450 con 0x7f4f000b2b00 2024-09-16T15:51:17.490 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:17.486+0000 7f4f050b0640 1 --2- 172.21.15.123:0/3428683191 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f4f00161620 0x7f4f0015b720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T15:51:17.491 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:17.486+0000 7f4efffff640 1 --2- 172.21.15.123:0/3428683191 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4f000b2b00 0x7f4f000caf70 secure :-1 s=READY pgs=1735 cs=0 l=1 rev1=1 crypto rx=0x7f4ef400c450 tx=0x7f4ef400c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:51:17.491 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:17.486+0000 7f4efd7fa640 1 -- 172.21.15.123:0/3428683191 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ef4012830 con 0x7f4f000b2b00 2024-09-16T15:51:17.491 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:17.486+0000 7f4efd7fa640 1 -- 172.21.15.123:0/3428683191 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f4ef4017070 con 0x7f4f000b2b00 2024-09-16T15:51:17.491 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:17.486+0000 7f4f06b13640 1 -- 172.21.15.123:0/3428683191 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4f001f8780 con 0x7f4f000b2b00 2024-09-16T15:51:17.491 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:17.486+0000 7f4efd7fa640 1 -- 172.21.15.123:0/3428683191 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ef4012d80 con 0x7f4f000b2b00 2024-09-16T15:51:17.492 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:17.486+0000 7f4f06b13640 1 -- 172.21.15.123:0/3428683191 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4f0015af20 con 0x7f4f000b2b00 2024-09-16T15:51:17.492 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:17.490+0000 7f4f06b13640 1 -- 172.21.15.123:0/3428683191 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4f000b0ee0 con 0x7f4f000b2b00 2024-09-16T15:51:17.493 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:17.490+0000 7f4efd7fa640 1 -- 172.21.15.123:0/3428683191 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 24) ==== 97845+0+0 (secure 0 0 0) 0x7f4ef4007020 con 0x7f4f000b2b00 2024-09-16T15:51:17.494 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:17.490+0000 7f4efd7fa640 1 --2- 172.21.15.123:0/3428683191 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f4eec073760 0x7f4eec075c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:51:17.494 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:17.490+0000 7f4efd7fa640 1 -- 172.21.15.123:0/3428683191 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(156..156 src has 1..156) ==== 5737+0+0 (secure 0 0 0) 0x7f4ef4092bd0 con 0x7f4f000b2b00 2024-09-16T15:51:17.494 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:17.490+0000 7f4f048af640 1 --2- 172.21.15.123:0/3428683191 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f4eec073760 0x7f4eec075c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:51:17.494 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:17.490+0000 7f4f048af640 1 --2- 172.21.15.123:0/3428683191 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f4eec073760 0x7f4eec075c20 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f4ef00af000 tx=0x7f4ef001afd0 comp rx=0 tx=0).ready entity=mgr.4150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:51:17.494 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:17.494+0000 7f4efd7fa640 1 -- 172.21.15.123:0/3428683191 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f4ef405f8d0 con 0x7f4f000b2b00 2024-09-16T15:51:17.665 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:17.666+0000 7f4f06b13640 1 -- 172.21.15.123:0/3428683191 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f4f000b3bd0 con 0x7f4f000b2b00 2024-09-16T15:51:17.666 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:17.666+0000 7f4efd7fa640 1 -- 172.21.15.123:0/3428683191 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v156) ==== 74+0+16856 (secure 0 0 0) 0x7f4ef405f1d0 con 0x7f4f000b2b00 2024-09-16T15:51:17.667 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-09-16T15:51:17.667 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":156,"fsid":"56077f89-18c8-486f-ae7e-eecc0bb7822c","created":"2024-09-16T15:42:26.364892+0000","modified":"2024-09-16T15:51:16.301997+0000","last_up_change":"2024-09-16T15:42:38.707234+0000","last_in_change":"2024-09-16T15:42:29.423662+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":25,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-16T15:42:39.287764+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":24,"pool_name":"cephfs_metadata","create_time":"2024-09-16T15:51:13.953990+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"156","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":25,"pool_name":"cephfs_data","create_time":"2024-09-16T15:51:15.281388+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"156","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9e7c1955-a607-4671-9be7-f2ef760ceeaa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":155,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6800","nonce":3882044969},{"type":"v1","addr":"172.21.15.123:6801","nonce":3882044969}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6802","nonce":3882044969},{"type":"v1","addr":"172.21.15.123:6803","nonce":3882044969}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6806","nonce":3882044969},{"type":"v1","addr":"172.21.15.123:6807","nonce":3882044969}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6804","nonce":3882044969},{"type":"v1","addr":"172.21.15.123:6805","nonce":3882044969}]},"public_addr":"172.21.15.123:6801/3882044969","cluster_addr":"172.21.15.123:6803/3882044969","heartbeat_back_addr":"172.21.15.123:6807/3882044969","heartbeat_front_addr":"172.21.15.123:6805/3882044969","state":["exists","up"]},{"osd":1,"uuid":"c540cfde-0977-4742-9b17-fd634668e010","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":155,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6816","nonce":4037001850},{"type":"v1","addr":"172.21.15.123:6817","nonce":4037001850}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6818","nonce":4037001850},{"type":"v1","addr":"172.21.15.123:6819","nonce":4037001850}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6822","nonce":4037001850},{"type":"v1","addr":"172.21.15.123:6823","nonce":4037001850}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6820","nonce":4037001850},{"type":"v1","addr":"172.21.15.123:6821","nonce":4037001850}]},"public_addr":"172.21.15.123:6817/4037001850","cluster_addr":"172.21.15.123:6819/4037001850","heartbeat_back_addr":"172.21.15.123:6823/4037001850","heartbeat_front_addr":"172.21.15.123:6821/4037001850","state":["exists","up"]},{"osd":2,"uuid":"9864a496-f169-486b-95c4-1d59162793a7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":155,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6824","nonce":1332143088},{"type":"v1","addr":"172.21.15.123:6825","nonce":1332143088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6826","nonce":1332143088},{"type":"v1","addr":"172.21.15.123:6827","nonce":1332143088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6830","nonce":1332143088},{"type":"v1","addr":"172.21.15.123:6831","nonce":1332143088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6828","nonce":1332143088},{"type":"v1","addr":"172.21.15.123:6829","nonce":1332143088}]},"public_addr":"172.21.15.123:6825/1332143088","cluster_addr":"172.21.15.123:6827/1332143088","heartbeat_back_addr":"172.21.15.123:6831/1332143088","heartbeat_front_addr":"172.21.15.123:6829/1332143088","state":["exists","up"]},{"osd":3,"uuid":"c6d1f817-f074-4325-95e1-578eba13edda","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":155,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6808","nonce":988586998},{"type":"v1","addr":"172.21.15.123:6809","nonce":988586998}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6810","nonce":988586998},{"type":"v1","addr":"172.21.15.123:6811","nonce":988586998}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6814","nonce":988586998},{"type":"v1","addr":"172.21.15.123:6815","nonce":988586998}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6812","nonce":988586998},{"type":"v1","addr":"172.21.15.123:6813","nonce":988586998}]},"public_addr":"172.21.15.123:6809/988586998","cluster_addr":"172.21.15.123:6811/988586998","heartbeat_back_addr":"172.21.15.123:6815/988586998","heartbeat_front_addr":"172.21.15.123:6813/988586998","state":["exists","up"]},{"osd":4,"uuid":"200a8df4-50ef-40a5-b48a-58981269c086","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":155,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6816","nonce":3387813890},{"type":"v1","addr":"172.21.15.184:6817","nonce":3387813890}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6818","nonce":3387813890},{"type":"v1","addr":"172.21.15.184:6819","nonce":3387813890}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6822","nonce":3387813890},{"type":"v1","addr":"172.21.15.184:6823","nonce":3387813890}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6820","nonce":3387813890},{"type":"v1","addr":"172.21.15.184:6821","nonce":3387813890}]},"public_addr":"172.21.15.184:6817/3387813890","cluster_addr":"172.21.15.184:6819/3387813890","heartbeat_back_addr":"172.21.15.184:6823/3387813890","heartbeat_front_addr":"172.21.15.184:6821/3387813890","state":["exists","up"]},{"osd":5,"uuid":"32d5a900-da3d-412c-bc19-13861a2368d8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":155,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6808","nonce":2680916017},{"type":"v1","addr":"172.21.15.184:6809","nonce":2680916017}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6810","nonce":2680916017},{"type":"v1","addr":"172.21.15.184:6811","nonce":2680916017}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6814","nonce":2680916017},{"type":"v1","addr":"172.21.15.184:6815","nonce":2680916017}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6812","nonce":2680916017},{"type":"v1","addr":"172.21.15.184:6813","nonce":2680916017}]},"public_addr":"172.21.15.184:6809/2680916017","cluster_addr":"172.21.15.184:6811/2680916017","heartbeat_back_addr":"172.21.15.184:6815/2680916017","heartbeat_front_addr":"172.21.15.184:6813/2680916017","state":["exists","up"]},{"osd":6,"uuid":"cde6f39e-aa1b-4260-ac19-774ee6aaca72","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":155,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6824","nonce":2183120457},{"type":"v1","addr":"172.21.15.184:6825","nonce":2183120457}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6826","nonce":2183120457},{"type":"v1","addr":"172.21.15.184:6827","nonce":2183120457}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6830","nonce":2183120457},{"type":"v1","addr":"172.21.15.184:6831","nonce":2183120457}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6828","nonce":2183120457},{"type":"v1","addr":"172.21.15.184:6829","nonce":2183120457}]},"public_addr":"172.21.15.184:6825/2183120457","cluster_addr":"172.21.15.184:6827/2183120457","heartbeat_back_addr":"172.21.15.184:6831/2183120457","heartbeat_front_addr":"172.21.15.184:6829/2183120457","state":["exists","up"]},{"osd":7,"uuid":"4f381778-eee1-47c6-804f-df0e6b33e313","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":155,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6800","nonce":4102957285},{"type":"v1","addr":"172.21.15.184:6801","nonce":4102957285}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6802","nonce":4102957285},{"type":"v1","addr":"172.21.15.184:6803","nonce":4102957285}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6806","nonce":4102957285},{"type":"v1","addr":"172.21.15.184:6807","nonce":4102957285}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6804","nonce":4102957285},{"type":"v1","addr":"172.21.15.184:6805","nonce":4102957285}]},"public_addr":"172.21.15.184:6801/4102957285","cluster_addr":"172.21.15.184:6803/4102957285","heartbeat_back_addr":"172.21.15.184:6807/4102957285","heartbeat_front_addr":"172.21.15.184:6805/4102957285","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:35.972210+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.222007+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.198617+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.385663+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.358935+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.156066+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.752973+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:35.682846+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":17,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T15:51:17.669 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:17.670+0000 7f4f06b13640 1 -- 172.21.15.123:0/3428683191 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f4eec073760 msgr2=0x7f4eec075c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:51:17.669 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:17.670+0000 7f4f06b13640 1 --2- 172.21.15.123:0/3428683191 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f4eec073760 0x7f4eec075c20 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f4ef00af000 tx=0x7f4ef001afd0 comp rx=0 tx=0).stop 2024-09-16T15:51:17.669 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:17.670+0000 7f4f06b13640 1 -- 172.21.15.123:0/3428683191 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4f000b2b00 msgr2=0x7f4f000caf70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:51:17.669 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:17.670+0000 7f4f06b13640 1 --2- 172.21.15.123:0/3428683191 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4f000b2b00 0x7f4f000caf70 secure :-1 s=READY pgs=1735 cs=0 l=1 rev1=1 crypto rx=0x7f4ef400c450 tx=0x7f4ef400c910 comp rx=0 tx=0).stop 2024-09-16T15:51:17.670 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:17.670+0000 7f4f06b13640 1 -- 172.21.15.123:0/3428683191 shutdown_connections 2024-09-16T15:51:17.670 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:17.670+0000 7f4f06b13640 1 --2- 172.21.15.123:0/3428683191 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f4eec073760 0x7f4eec075c20 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:17.670 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:17.670+0000 7f4f06b13640 1 --2- 172.21.15.123:0/3428683191 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f4f00161620 0x7f4f0015b720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:17.670 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:17.670+0000 7f4f06b13640 1 --2- 172.21.15.123:0/3428683191 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4f000b2b00 0x7f4f000caf70 unknown :-1 s=CLOSED pgs=1735 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:17.670 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:17.670+0000 7f4f06b13640 1 --2- 172.21.15.123:0/3428683191 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4f000b0a50 0x7f4f0014e5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:17.670 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:17.670+0000 7f4f06b13640 1 -- 172.21.15.123:0/3428683191 >> 172.21.15.123:0/3428683191 conn(0x7f4f000d3ea0 msgr2=0x7f4f00158ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:51:17.670 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:17.670+0000 7f4f06b13640 1 -- 172.21.15.123:0/3428683191 shutdown_connections 2024-09-16T15:51:17.670 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:17.670+0000 7f4f06b13640 1 -- 172.21.15.123:0/3428683191 wait complete. 2024-09-16T15:51:17.686 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-09-16T15:51:17.812 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:17.814+0000 7fdfb6fc4640 1 -- 172.21.15.123:0/1556804721 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdfb015aa00 msgr2=0x7fdfb015e5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:51:17.813 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:17.814+0000 7fdfb6fc4640 1 --2- 172.21.15.123:0/1556804721 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdfb015aa00 0x7fdfb015e5d0 secure :-1 s=READY pgs=1736 cs=0 l=1 rev1=1 crypto rx=0x7fdfb00ca660 tx=0x7fdfa401c8c0 comp rx=0 tx=0).stop 2024-09-16T15:51:17.813 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:17.814+0000 7fdfb6fc4640 1 -- 172.21.15.123:0/1556804721 shutdown_connections 2024-09-16T15:51:17.813 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:17.814+0000 7fdfb6fc4640 1 --2- 172.21.15.123:0/1556804721 >> v2:172.21.15.184:3300/0 conn(0x7fdfb00cc3c0 0x7fdfb00cc810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:17.813 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:17.814+0000 7fdfb6fc4640 1 --2- 172.21.15.123:0/1556804721 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdfb015aa00 0x7fdfb015e5d0 unknown :-1 s=CLOSED pgs=1736 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:17.813 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:17.814+0000 7fdfb6fc4640 1 -- 172.21.15.123:0/1556804721 >> 172.21.15.123:0/1556804721 conn(0x7fdfb00b1710 msgr2=0x7fdfb00b1b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:51:17.813 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:17.814+0000 7fdfb6fc4640 1 -- 172.21.15.123:0/1556804721 shutdown_connections 2024-09-16T15:51:17.813 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:17.814+0000 7fdfb6fc4640 1 -- 172.21.15.123:0/1556804721 wait complete. 2024-09-16T15:51:17.814 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:17.814+0000 7fdfb6fc4640 1 Processor -- start 2024-09-16T15:51:17.814 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:17.814+0000 7fdfb6fc4640 1 -- start start 2024-09-16T15:51:17.815 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:17.814+0000 7fdfb6fc4640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fdfb0072720 0x7fdfb00b2a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:51:17.815 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:17.814+0000 7fdfb6fc4640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdfb00cc3c0 0x7fdfb01530f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:51:17.815 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:17.814+0000 7fdfb6fc4640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fdfb015aa00 0x7fdfb0152080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:51:17.815 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:17.814+0000 7fdfb6fc4640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fdfb0167af0 con 0x7fdfb00cc3c0 2024-09-16T15:51:17.816 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:17.814+0000 7fdfb6fc4640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fdfb0167970 con 0x7fdfb015aa00 2024-09-16T15:51:17.816 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:17.814+0000 7fdfb6fc4640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fdfb0167c70 con 0x7fdfb0072720 2024-09-16T15:51:17.816 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:17.814+0000 7fdfaffff640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdfb00cc3c0 0x7fdfb01530f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:51:17.816 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:17.814+0000 7fdfaffff640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdfb00cc3c0 0x7fdfb01530f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:45096/0 (socket says 172.21.15.123:45096) 2024-09-16T15:51:17.816 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:17.814+0000 7fdfaffff640 1 -- 172.21.15.123:0/3305273361 learned_addr learned my addr 172.21.15.123:0/3305273361 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:51:17.816 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:17.814+0000 7fdfb5561640 1 --2- 172.21.15.123:0/3305273361 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fdfb015aa00 0x7fdfb0152080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:51:17.816 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:17.814+0000 7fdfb4d60640 1 --2- 172.21.15.123:0/3305273361 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fdfb0072720 0x7fdfb00b2a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:51:17.816 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:17.814+0000 7fdfaffff640 1 -- 172.21.15.123:0/3305273361 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fdfb0072720 msgr2=0x7fdfb00b2a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:51:17.816 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:17.814+0000 7fdfaffff640 1 --2- 172.21.15.123:0/3305273361 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fdfb0072720 0x7fdfb00b2a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:17.816 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:17.814+0000 7fdfaffff640 1 -- 172.21.15.123:0/3305273361 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fdfb015aa00 msgr2=0x7fdfb0152080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:51:17.816 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:17.814+0000 7fdfaffff640 1 --2- 172.21.15.123:0/3305273361 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fdfb015aa00 0x7fdfb0152080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:17.816 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:17.814+0000 7fdfaffff640 1 -- 172.21.15.123:0/3305273361 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdfb01544e0 con 0x7fdfb00cc3c0 2024-09-16T15:51:17.817 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:17.814+0000 7fdfb5561640 1 --2- 172.21.15.123:0/3305273361 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fdfb015aa00 0x7fdfb0152080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T15:51:17.817 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:17.814+0000 7fdfb4d60640 1 --2- 172.21.15.123:0/3305273361 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fdfb0072720 0x7fdfb00b2a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T15:51:17.817 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:17.814+0000 7fdfaffff640 1 --2- 172.21.15.123:0/3305273361 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdfb00cc3c0 0x7fdfb01530f0 secure :-1 s=READY pgs=1737 cs=0 l=1 rev1=1 crypto rx=0x7fdf9c00c530 tx=0x7fdf9c00c9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:51:17.817 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:17.818+0000 7fdfad7fa640 1 -- 172.21.15.123:0/3305273361 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf9c017070 con 0x7fdfb00cc3c0 2024-09-16T15:51:17.817 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:17.818+0000 7fdfad7fa640 1 -- 172.21.15.123:0/3305273361 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fdf9c0053c0 con 0x7fdfb00cc3c0 2024-09-16T15:51:17.817 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:17.818+0000 7fdfb6fc4640 1 -- 172.21.15.123:0/3305273361 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdfb0153d40 con 0x7fdfb00cc3c0 2024-09-16T15:51:17.817 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:17.818+0000 7fdfad7fa640 1 -- 172.21.15.123:0/3305273361 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf9c012890 con 0x7fdfb00cc3c0 2024-09-16T15:51:17.817 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:17.818+0000 7fdfb6fc4640 1 -- 172.21.15.123:0/3305273361 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdfb014f3d0 con 0x7fdfb00cc3c0 2024-09-16T15:51:17.818 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:17.818+0000 7fdfb6fc4640 1 -- 172.21.15.123:0/3305273361 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdfb00cc810 con 0x7fdfb00cc3c0 2024-09-16T15:51:17.818 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:17.818+0000 7fdfad7fa640 1 -- 172.21.15.123:0/3305273361 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 24) ==== 97845+0+0 (secure 0 0 0) 0x7fdf9c01a070 con 0x7fdfb00cc3c0 2024-09-16T15:51:17.818 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:17.818+0000 7fdfad7fa640 1 --2- 172.21.15.123:0/3305273361 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fdf94073760 0x7fdf94075c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:51:17.818 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:17.818+0000 7fdfb4d60640 1 --2- 172.21.15.123:0/3305273361 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fdf94073760 0x7fdf94075c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:51:17.818 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:17.818+0000 7fdfad7fa640 1 -- 172.21.15.123:0/3305273361 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(156..156 src has 1..156) ==== 5737+0+0 (secure 0 0 0) 0x7fdf9c0944c0 con 0x7fdfb00cc3c0 2024-09-16T15:51:17.818 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:17.818+0000 7fdfb4d60640 1 --2- 172.21.15.123:0/3305273361 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fdf94073760 0x7fdf94075c20 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7fdfa40af000 tx=0x7fdfa401b100 comp rx=0 tx=0).ready entity=mgr.4150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:51:17.820 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:17.822+0000 7fdfad7fa640 1 -- 172.21.15.123:0/3305273361 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7fdf9c021080 con 0x7fdfb00cc3c0 2024-09-16T15:51:18.032 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:18.034+0000 7fdfb6fc4640 1 -- 172.21.15.123:0/3305273361 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) -- 0x7fdfb0151de0 con 0x7fdfb00cc3c0 2024-09-16T15:51:18.909 INFO:tasks.ceph.mds.d.smithi184.stderr:2024-09-16T15:51:18.911+0000 7f33db524640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-09-16T15:51:18.994 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:18.994+0000 7fdfad7fa640 1 -- 172.21.15.123:0/3305273361 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v104) ==== 118+0+0 (secure 0 0 0) 0x7fdf9c05e7d0 con 0x7fdfb00cc3c0 2024-09-16T15:51:18.997 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:18.998+0000 7fdfb6fc4640 1 -- 172.21.15.123:0/3305273361 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fdf94073760 msgr2=0x7fdf94075c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:51:18.997 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:18.998+0000 7fdfb6fc4640 1 --2- 172.21.15.123:0/3305273361 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fdf94073760 0x7fdf94075c20 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7fdfa40af000 tx=0x7fdfa401b100 comp rx=0 tx=0).stop 2024-09-16T15:51:18.997 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:18.998+0000 7fdfb6fc4640 1 -- 172.21.15.123:0/3305273361 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdfb00cc3c0 msgr2=0x7fdfb01530f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:51:18.998 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:18.998+0000 7fdfb6fc4640 1 --2- 172.21.15.123:0/3305273361 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdfb00cc3c0 0x7fdfb01530f0 secure :-1 s=READY pgs=1737 cs=0 l=1 rev1=1 crypto rx=0x7fdf9c00c530 tx=0x7fdf9c00c9f0 comp rx=0 tx=0).stop 2024-09-16T15:51:18.998 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:18.998+0000 7fdfb6fc4640 1 -- 172.21.15.123:0/3305273361 shutdown_connections 2024-09-16T15:51:18.998 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:18.998+0000 7fdfb6fc4640 1 --2- 172.21.15.123:0/3305273361 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fdf94073760 0x7fdf94075c20 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:18.998 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:18.998+0000 7fdfb6fc4640 1 --2- 172.21.15.123:0/3305273361 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fdfb015aa00 0x7fdfb0152080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:18.998 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:18.998+0000 7fdfb6fc4640 1 --2- 172.21.15.123:0/3305273361 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdfb00cc3c0 0x7fdfb01530f0 unknown :-1 s=CLOSED pgs=1737 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:18.998 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:18.998+0000 7fdfb6fc4640 1 --2- 172.21.15.123:0/3305273361 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fdfb0072720 0x7fdfb00b2a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:18.998 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:18.998+0000 7fdfb6fc4640 1 -- 172.21.15.123:0/3305273361 >> 172.21.15.123:0/3305273361 conn(0x7fdfb00b1710 msgr2=0x7fdfb00af680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:51:18.998 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:18.998+0000 7fdfb6fc4640 1 -- 172.21.15.123:0/3305273361 shutdown_connections 2024-09-16T15:51:18.998 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:18.998+0000 7fdfb6fc4640 1 -- 172.21.15.123:0/3305273361 wait complete. 2024-09-16T15:51:19.017 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-09-16T15:51:19.017 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-09-16T15:51:19.149 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:19.150+0000 7f41eb084640 1 -- 172.21.15.123:0/3118306957 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f41cc004810 msgr2=0x7f41cc004c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:51:19.149 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:19.150+0000 7f41eb084640 1 --2- 172.21.15.123:0/3118306957 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f41cc004810 0x7f41cc004c90 secure :-1 s=READY pgs=1739 cs=0 l=1 rev1=1 crypto rx=0x7f41d80016f0 tx=0x7f41d801c760 comp rx=0 tx=0).stop 2024-09-16T15:51:19.149 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:19.150+0000 7f41eb084640 1 -- 172.21.15.123:0/3118306957 shutdown_connections 2024-09-16T15:51:19.149 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:19.150+0000 7f41eb084640 1 --2- 172.21.15.123:0/3118306957 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f41cc004810 0x7f41cc004c90 unknown :-1 s=CLOSED pgs=1739 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:19.149 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:19.150+0000 7f41eb084640 1 -- 172.21.15.123:0/3118306957 >> 172.21.15.123:0/3118306957 conn(0x7f41e40b1710 msgr2=0x7f41e40b1b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:51:19.149 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:19.150+0000 7f41eb084640 1 -- 172.21.15.123:0/3118306957 shutdown_connections 2024-09-16T15:51:19.150 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:19.150+0000 7f41eb084640 1 -- 172.21.15.123:0/3118306957 wait complete. 2024-09-16T15:51:19.150 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:19.150+0000 7f41eb084640 1 Processor -- start 2024-09-16T15:51:19.150 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:19.150+0000 7f41eb084640 1 -- start start 2024-09-16T15:51:19.152 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:19.150+0000 7f41eb084640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f41cc004810 0x7f41e40c6310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:51:19.152 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:19.150+0000 7f41eb084640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f41e41478c0 0x7f41e4158c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:51:19.152 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:19.150+0000 7f41eb084640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f41e4159160 0x7f41e40b42f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:51:19.152 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:19.150+0000 7f41eb084640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f41e415f560 con 0x7f41e4159160 2024-09-16T15:51:19.152 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:19.150+0000 7f41eb084640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f41e415f3e0 con 0x7f41cc004810 2024-09-16T15:51:19.152 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:19.150+0000 7f41eb084640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f41e406ebe0 con 0x7f41e41478c0 2024-09-16T15:51:19.153 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:19.154+0000 7f41e9621640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f41e4159160 0x7f41e40b42f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:51:19.153 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:19.154+0000 7f41e9621640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f41e4159160 0x7f41e40b42f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:45108/0 (socket says 172.21.15.123:45108) 2024-09-16T15:51:19.153 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:19.154+0000 7f41e9621640 1 -- 172.21.15.123:0/3463716741 learned_addr learned my addr 172.21.15.123:0/3463716741 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:51:19.153 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:19.154+0000 7f41e8e20640 1 --2- 172.21.15.123:0/3463716741 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f41cc004810 0x7f41e40c6310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:51:19.153 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:19.154+0000 7f41e3fff640 1 --2- 172.21.15.123:0/3463716741 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f41e41478c0 0x7f41e4158c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:51:19.153 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:19.154+0000 7f41e9621640 1 -- 172.21.15.123:0/3463716741 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f41e41478c0 msgr2=0x7f41e4158c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:51:19.153 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:19.154+0000 7f41e9621640 1 --2- 172.21.15.123:0/3463716741 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f41e41478c0 0x7f41e4158c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:19.153 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:19.154+0000 7f41e9621640 1 -- 172.21.15.123:0/3463716741 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f41cc004810 msgr2=0x7f41e40c6310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:51:19.153 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:19.154+0000 7f41e9621640 1 --2- 172.21.15.123:0/3463716741 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f41cc004810 0x7f41e40c6310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:19.153 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:19.154+0000 7f41e9621640 1 -- 172.21.15.123:0/3463716741 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f41e4147ce0 con 0x7f41e4159160 2024-09-16T15:51:19.153 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:19.154+0000 7f41e3fff640 1 --2- 172.21.15.123:0/3463716741 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f41e41478c0 0x7f41e4158c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T15:51:19.153 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:19.154+0000 7f41e9621640 1 --2- 172.21.15.123:0/3463716741 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f41e4159160 0x7f41e40b42f0 secure :-1 s=READY pgs=1740 cs=0 l=1 rev1=1 crypto rx=0x7f41dc001340 tx=0x7f41dc00cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:51:19.154 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:19.154+0000 7f41e17fa640 1 -- 172.21.15.123:0/3463716741 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41dc017070 con 0x7f41e4159160 2024-09-16T15:51:19.154 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:19.154+0000 7f41e17fa640 1 -- 172.21.15.123:0/3463716741 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f41dc005070 con 0x7f41e4159160 2024-09-16T15:51:19.154 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:19.154+0000 7f41e17fa640 1 -- 172.21.15.123:0/3463716741 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41dc0054c0 con 0x7f41e4159160 2024-09-16T15:51:19.154 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:19.154+0000 7f41eb084640 1 -- 172.21.15.123:0/3463716741 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f41e4146e00 con 0x7f41e4159160 2024-09-16T15:51:19.154 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:19.154+0000 7f41eb084640 1 -- 172.21.15.123:0/3463716741 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f41e4159930 con 0x7f41e4159160 2024-09-16T15:51:19.155 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:19.154+0000 7f41eb084640 1 -- 172.21.15.123:0/3463716741 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f41e406f140 con 0x7f41e4159160 2024-09-16T15:51:19.155 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:19.154+0000 7f41e17fa640 1 -- 172.21.15.123:0/3463716741 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 24) ==== 97845+0+0 (secure 0 0 0) 0x7f41dc01a070 con 0x7f41e4159160 2024-09-16T15:51:19.155 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:19.154+0000 7f41e17fa640 1 --2- 172.21.15.123:0/3463716741 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f41d0073730 0x7f41d0075bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:51:19.155 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:19.154+0000 7f41e17fa640 1 -- 172.21.15.123:0/3463716741 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(157..157 src has 1..157) ==== 5737+0+0 (secure 0 0 0) 0x7f41dc091b50 con 0x7f41e4159160 2024-09-16T15:51:19.155 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:19.154+0000 7f41e8e20640 1 --2- 172.21.15.123:0/3463716741 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f41d0073730 0x7f41d0075bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:51:19.155 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:19.154+0000 7f41e8e20640 1 --2- 172.21.15.123:0/3463716741 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f41d0073730 0x7f41d0075bf0 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f41d4005fd0 tx=0x7f41d4005f40 comp rx=0 tx=0).ready entity=mgr.4150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:51:19.158 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:19.158+0000 7f41e17fa640 1 -- 172.21.15.123:0/3463716741 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f41dc05e850 con 0x7f41e4159160 2024-09-16T15:51:19.370 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:19.370+0000 7f41eb084640 1 -- 172.21.15.123:0/3463716741 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) -- 0x7f41e40ca9a0 con 0x7f41e4159160 2024-09-16T15:51:19.997 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:19.998+0000 7f41e17fa640 1 -- 172.21.15.123:0/3463716741 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v105) ==== 122+0+0 (secure 0 0 0) 0x7f41dc05e150 con 0x7f41e4159160 2024-09-16T15:51:20.002 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:20.002+0000 7f41eb084640 1 -- 172.21.15.123:0/3463716741 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f41d0073730 msgr2=0x7f41d0075bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:51:20.002 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:20.002+0000 7f41eb084640 1 --2- 172.21.15.123:0/3463716741 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f41d0073730 0x7f41d0075bf0 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f41d4005fd0 tx=0x7f41d4005f40 comp rx=0 tx=0).stop 2024-09-16T15:51:20.002 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:20.002+0000 7f41eb084640 1 -- 172.21.15.123:0/3463716741 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f41e4159160 msgr2=0x7f41e40b42f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:51:20.002 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:20.002+0000 7f41eb084640 1 --2- 172.21.15.123:0/3463716741 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f41e4159160 0x7f41e40b42f0 secure :-1 s=READY pgs=1740 cs=0 l=1 rev1=1 crypto rx=0x7f41dc001340 tx=0x7f41dc00cc60 comp rx=0 tx=0).stop 2024-09-16T15:51:20.002 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:20.002+0000 7f41eb084640 1 -- 172.21.15.123:0/3463716741 shutdown_connections 2024-09-16T15:51:20.003 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:20.002+0000 7f41eb084640 1 --2- 172.21.15.123:0/3463716741 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f41d0073730 0x7f41d0075bf0 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:20.003 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:20.002+0000 7f41eb084640 1 --2- 172.21.15.123:0/3463716741 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f41e4159160 0x7f41e40b42f0 unknown :-1 s=CLOSED pgs=1740 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:20.003 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:20.002+0000 7f41eb084640 1 --2- 172.21.15.123:0/3463716741 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f41e41478c0 0x7f41e4158c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:20.003 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:20.002+0000 7f41eb084640 1 --2- 172.21.15.123:0/3463716741 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f41cc004810 0x7f41e40c6310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:20.003 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:20.002+0000 7f41eb084640 1 -- 172.21.15.123:0/3463716741 >> 172.21.15.123:0/3463716741 conn(0x7f41e40b1710 msgr2=0x7f41e40af680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:51:20.003 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:20.002+0000 7f41eb084640 1 -- 172.21.15.123:0/3463716741 shutdown_connections 2024-09-16T15:51:20.003 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:20.002+0000 7f41eb084640 1 -- 172.21.15.123:0/3463716741 wait complete. 2024-09-16T15:51:20.021 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-09-16T15:51:20.022 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-16T15:51:20.148 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:20.150+0000 7ff685f2c640 1 -- 172.21.15.123:0/307151365 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff68015ec40 msgr2=0x7ff68006f420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:51:20.148 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:20.150+0000 7ff685f2c640 1 --2- 172.21.15.123:0/307151365 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff68015ec40 0x7ff68006f420 secure :-1 s=READY pgs=1742 cs=0 l=1 rev1=1 crypto rx=0x7ff678009bc0 tx=0x7ff67801ca80 comp rx=0 tx=0).stop 2024-09-16T15:51:20.149 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:20.150+0000 7ff685f2c640 1 -- 172.21.15.123:0/307151365 shutdown_connections 2024-09-16T15:51:20.149 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:20.150+0000 7ff685f2c640 1 --2- 172.21.15.123:0/307151365 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff68015ec40 0x7ff68006f420 unknown :-1 s=CLOSED pgs=1742 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:20.149 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:20.150+0000 7ff685f2c640 1 --2- 172.21.15.123:0/307151365 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7ff6800cf5a0 0x7ff68006eee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:20.149 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:20.150+0000 7ff685f2c640 1 -- 172.21.15.123:0/307151365 >> 172.21.15.123:0/307151365 conn(0x7ff6800cfb10 msgr2=0x7ff6800cff20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:51:20.149 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:20.150+0000 7ff685f2c640 1 -- 172.21.15.123:0/307151365 shutdown_connections 2024-09-16T15:51:20.149 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:20.150+0000 7ff685f2c640 1 -- 172.21.15.123:0/307151365 wait complete. 2024-09-16T15:51:20.149 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:20.150+0000 7ff685f2c640 1 Processor -- start 2024-09-16T15:51:20.149 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:20.150+0000 7ff685f2c640 1 -- start start 2024-09-16T15:51:20.151 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:20.150+0000 7ff685f2c640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7ff6800cf5a0 0x7ff680155b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:51:20.151 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:20.150+0000 7ff685f2c640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff680158120 0x7ff6800af610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:51:20.151 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:20.150+0000 7ff685f2c640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff68015ec40 0x7ff6800afb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:51:20.151 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:20.150+0000 7ff685f2c640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7ff68015b260 con 0x7ff68015ec40 2024-09-16T15:51:20.152 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:20.150+0000 7ff685f2c640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7ff68015b0e0 con 0x7ff680158120 2024-09-16T15:51:20.152 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:20.150+0000 7ff685f2c640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7ff68015b3e0 con 0x7ff6800cf5a0 2024-09-16T15:51:20.152 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:20.150+0000 7ff67ffff640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff68015ec40 0x7ff6800afb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:51:20.152 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:20.150+0000 7ff67ffff640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff68015ec40 0x7ff6800afb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:45130/0 (socket says 172.21.15.123:45130) 2024-09-16T15:51:20.152 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:20.150+0000 7ff67ffff640 1 -- 172.21.15.123:0/2867841247 learned_addr learned my addr 172.21.15.123:0/2867841247 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:51:20.152 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:20.150+0000 7ff67ffff640 1 -- 172.21.15.123:0/2867841247 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7ff6800cf5a0 msgr2=0x7ff680155b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:51:20.152 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:20.150+0000 7ff67ffff640 1 --2- 172.21.15.123:0/2867841247 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7ff6800cf5a0 0x7ff680155b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:20.152 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:20.150+0000 7ff67ffff640 1 -- 172.21.15.123:0/2867841247 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff680158120 msgr2=0x7ff6800af610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:51:20.152 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:20.150+0000 7ff67ffff640 1 --2- 172.21.15.123:0/2867841247 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff680158120 0x7ff6800af610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:20.152 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:20.150+0000 7ff67ffff640 1 -- 172.21.15.123:0/2867841247 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff6800c6ee0 con 0x7ff68015ec40 2024-09-16T15:51:20.152 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:20.150+0000 7ff67ffff640 1 --2- 172.21.15.123:0/2867841247 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff68015ec40 0x7ff6800afb50 secure :-1 s=READY pgs=1743 cs=0 l=1 rev1=1 crypto rx=0x7ff67801c4f0 tx=0x7ff678005ae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:51:20.152 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:20.154+0000 7ff667fff640 1 -- 172.21.15.123:0/2867841247 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6780b2040 con 0x7ff68015ec40 2024-09-16T15:51:20.153 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:20.154+0000 7ff667fff640 1 -- 172.21.15.123:0/2867841247 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7ff6780bb070 con 0x7ff68015ec40 2024-09-16T15:51:20.153 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:20.154+0000 7ff685f2c640 1 -- 172.21.15.123:0/2867841247 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff680156040 con 0x7ff68015ec40 2024-09-16T15:51:20.153 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:20.154+0000 7ff667fff640 1 -- 172.21.15.123:0/2867841247 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff678004630 con 0x7ff68015ec40 2024-09-16T15:51:20.153 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:20.154+0000 7ff685f2c640 1 -- 172.21.15.123:0/2867841247 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff6800c9600 con 0x7ff68015ec40 2024-09-16T15:51:20.154 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:20.154+0000 7ff685f2c640 1 -- 172.21.15.123:0/2867841247 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff6800ca320 con 0x7ff68015ec40 2024-09-16T15:51:20.154 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:20.154+0000 7ff667fff640 1 -- 172.21.15.123:0/2867841247 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 24) ==== 97845+0+0 (secure 0 0 0) 0x7ff6780a5940 con 0x7ff68015ec40 2024-09-16T15:51:20.154 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:20.154+0000 7ff667fff640 1 --2- 172.21.15.123:0/2867841247 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7ff66c073730 0x7ff66c075bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:51:20.154 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:20.154+0000 7ff667fff640 1 -- 172.21.15.123:0/2867841247 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(157..157 src has 1..157) ==== 5737+0+0 (secure 0 0 0) 0x7ff67812dcd0 con 0x7ff68015ec40 2024-09-16T15:51:20.154 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:20.154+0000 7ff67f7fe640 1 --2- 172.21.15.123:0/2867841247 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7ff66c073730 0x7ff66c075bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:51:20.154 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:20.154+0000 7ff67f7fe640 1 --2- 172.21.15.123:0/2867841247 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7ff66c073730 0x7ff66c075bf0 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7ff670007fd0 tx=0x7ff6700078e0 comp rx=0 tx=0).ready entity=mgr.4150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:51:20.157 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:20.158+0000 7ff667fff640 1 -- 172.21.15.123:0/2867841247 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7ff6800c6ee0 con 0x7ff68015ec40 2024-09-16T15:51:20.349 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:20.350+0000 7ff685f2c640 1 -- 172.21.15.123:0/2867841247 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff680159ee0 con 0x7ff68015ec40 2024-09-16T15:51:20.349 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:20.350+0000 7ff667fff640 1 -- 172.21.15.123:0/2867841247 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 105 v105) ==== 95+0+5244 (secure 0 0 0) 0x7ff678016790 con 0x7ff68015ec40 2024-09-16T15:51:20.349 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-09-16T15:51:20.350 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":105,"btime":"2024-09-16T15:51:19:998149+0000","default_fscid":12,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6200,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6835/2380355001","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6833","nonce":2380355001},{"type":"v1","addr":"172.21.15.184:6835","nonce":2380355001}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":100},{"gid":8844,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6834/1840463681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6832","nonce":1840463681},{"type":"v1","addr":"172.21.15.123:6834","nonce":1840463681}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":100},{"gid":8847,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6835/2493915107","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6833","nonce":2493915107},{"type":"v1","addr":"172.21.15.123:6835","nonce":2493915107}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":99}],"filesystems":[{"mdsmap":{"epoch":105,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-16T15:51:16.301945+0000","modified":"2024-09-16T15:51:19.374973+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8859},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8859":{"gid":8859,"name":"d","rank":0,"incarnation":101,"state":"up:active","state_seq":3,"addr":"172.21.15.184:6834/2856066034","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":2856066034},{"type":"v1","addr":"172.21.15.184:6834","nonce":2856066034}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[25],"metadata_pool":24,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8859,"qdb_cluster":[8859]},"id":12}]} 2024-09-16T15:51:20.350 INFO:teuthology.orchestra.run.smithi123.stderr:dumped fsmap epoch 105 2024-09-16T15:51:20.352 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:20.354+0000 7ff685f2c640 1 -- 172.21.15.123:0/2867841247 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7ff66c073730 msgr2=0x7ff66c075bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:51:20.352 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:20.354+0000 7ff685f2c640 1 --2- 172.21.15.123:0/2867841247 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7ff66c073730 0x7ff66c075bf0 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7ff670007fd0 tx=0x7ff6700078e0 comp rx=0 tx=0).stop 2024-09-16T15:51:20.352 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:20.354+0000 7ff685f2c640 1 -- 172.21.15.123:0/2867841247 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff68015ec40 msgr2=0x7ff6800afb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:51:20.352 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:20.354+0000 7ff685f2c640 1 --2- 172.21.15.123:0/2867841247 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff68015ec40 0x7ff6800afb50 secure :-1 s=READY pgs=1743 cs=0 l=1 rev1=1 crypto rx=0x7ff67801c4f0 tx=0x7ff678005ae0 comp rx=0 tx=0).stop 2024-09-16T15:51:20.352 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:20.354+0000 7ff685f2c640 1 -- 172.21.15.123:0/2867841247 shutdown_connections 2024-09-16T15:51:20.352 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:20.354+0000 7ff685f2c640 1 --2- 172.21.15.123:0/2867841247 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7ff66c073730 0x7ff66c075bf0 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:20.352 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:20.354+0000 7ff685f2c640 1 --2- 172.21.15.123:0/2867841247 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff68015ec40 0x7ff6800afb50 unknown :-1 s=CLOSED pgs=1743 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:20.352 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:20.354+0000 7ff685f2c640 1 --2- 172.21.15.123:0/2867841247 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff680158120 0x7ff6800af610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:20.352 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:20.354+0000 7ff685f2c640 1 --2- 172.21.15.123:0/2867841247 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7ff6800cf5a0 0x7ff680155b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:20.352 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:20.354+0000 7ff685f2c640 1 -- 172.21.15.123:0/2867841247 >> 172.21.15.123:0/2867841247 conn(0x7ff6800cfb10 msgr2=0x7ff6800cc520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:51:20.352 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:20.354+0000 7ff685f2c640 1 -- 172.21.15.123:0/2867841247 shutdown_connections 2024-09-16T15:51:20.352 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:20.354+0000 7ff685f2c640 1 -- 172.21.15.123:0/2867841247 wait complete. 2024-09-16T15:51:20.368 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-16T15:51:20.480 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:20.482+0000 7f5f7423c640 1 -- 172.21.15.123:0/2139305571 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5f6c0726c0 msgr2=0x7f5f6c0d0630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:51:20.480 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:20.482+0000 7f5f7423c640 1 --2- 172.21.15.123:0/2139305571 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5f6c0726c0 0x7f5f6c0d0630 secure :-1 s=READY pgs=1744 cs=0 l=1 rev1=1 crypto rx=0x7f5f68009900 tx=0x7f5f6801c820 comp rx=0 tx=0).stop 2024-09-16T15:51:20.480 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:20.482+0000 7f5f7423c640 1 -- 172.21.15.123:0/2139305571 shutdown_connections 2024-09-16T15:51:20.480 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:20.482+0000 7f5f7423c640 1 --2- 172.21.15.123:0/2139305571 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f5f6c074600 0x7f5f6c06e750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:20.480 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:20.482+0000 7f5f7423c640 1 --2- 172.21.15.123:0/2139305571 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5f6c0726c0 0x7f5f6c0d0630 unknown :-1 s=CLOSED pgs=1744 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:20.480 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:20.482+0000 7f5f7423c640 1 -- 172.21.15.123:0/2139305571 >> 172.21.15.123:0/2139305571 conn(0x7f5f6c0cfb10 msgr2=0x7f5f6c0cff20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:51:20.480 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:20.482+0000 7f5f7423c640 1 -- 172.21.15.123:0/2139305571 shutdown_connections 2024-09-16T15:51:20.480 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:20.482+0000 7f5f7423c640 1 -- 172.21.15.123:0/2139305571 wait complete. 2024-09-16T15:51:20.482 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:20.482+0000 7f5f7423c640 1 Processor -- start 2024-09-16T15:51:20.482 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:20.482+0000 7f5f7423c640 1 -- start start 2024-09-16T15:51:20.482 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:20.482+0000 7f5f7423c640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5f6c0726c0 0x7f5f6c156070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:51:20.482 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:20.482+0000 7f5f7423c640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f5f6c074600 0x7f5f6c0c7460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:51:20.482 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:20.482+0000 7f5f7423c640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5f6c15bca0 0x7f5f6c156d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:51:20.483 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:20.482+0000 7f5f7423c640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f5f6c168bd0 con 0x7f5f6c15bca0 2024-09-16T15:51:20.483 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:20.482+0000 7f5f7423c640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f5f6c168a50 con 0x7f5f6c0726c0 2024-09-16T15:51:20.483 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:20.482+0000 7f5f7423c640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f5f6c168d50 con 0x7f5f6c074600 2024-09-16T15:51:20.483 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:20.482+0000 7f5f727d9640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5f6c15bca0 0x7f5f6c156d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:51:20.483 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:20.482+0000 7f5f727d9640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5f6c15bca0 0x7f5f6c156d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:45144/0 (socket says 172.21.15.123:45144) 2024-09-16T15:51:20.483 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:20.482+0000 7f5f727d9640 1 -- 172.21.15.123:0/3635757065 learned_addr learned my addr 172.21.15.123:0/3635757065 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:51:20.483 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:20.482+0000 7f5f727d9640 1 -- 172.21.15.123:0/3635757065 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f5f6c074600 msgr2=0x7f5f6c0c7460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:51:20.483 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:20.482+0000 7f5f727d9640 1 --2- 172.21.15.123:0/3635757065 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f5f6c074600 0x7f5f6c0c7460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:20.483 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:20.482+0000 7f5f727d9640 1 -- 172.21.15.123:0/3635757065 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5f6c0726c0 msgr2=0x7f5f6c156070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:51:20.483 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:20.482+0000 7f5f727d9640 1 --2- 172.21.15.123:0/3635757065 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5f6c0726c0 0x7f5f6c156070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:20.483 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:20.482+0000 7f5f727d9640 1 -- 172.21.15.123:0/3635757065 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5f6c1565b0 con 0x7f5f6c15bca0 2024-09-16T15:51:20.483 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:20.482+0000 7f5f727d9640 1 --2- 172.21.15.123:0/3635757065 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5f6c15bca0 0x7f5f6c156d40 secure :-1 s=READY pgs=1745 cs=0 l=1 rev1=1 crypto rx=0x7f5f6400ccb0 tx=0x7f5f6400a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:51:20.483 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:20.482+0000 7f5f627fc640 1 -- 172.21.15.123:0/3635757065 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f64004b30 con 0x7f5f6c15bca0 2024-09-16T15:51:20.483 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:20.482+0000 7f5f627fc640 1 -- 172.21.15.123:0/3635757065 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f5f64004de0 con 0x7f5f6c15bca0 2024-09-16T15:51:20.484 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:20.482+0000 7f5f7423c640 1 -- 172.21.15.123:0/3635757065 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5f6c1f8dc0 con 0x7f5f6c15bca0 2024-09-16T15:51:20.484 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:20.482+0000 7f5f627fc640 1 -- 172.21.15.123:0/3635757065 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f64005280 con 0x7f5f6c15bca0 2024-09-16T15:51:20.484 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:20.482+0000 7f5f7423c640 1 -- 172.21.15.123:0/3635757065 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5f6c1f9340 con 0x7f5f6c15bca0 2024-09-16T15:51:20.485 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:20.482+0000 7f5f7423c640 1 -- 172.21.15.123:0/3635757065 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5f6c0d0630 con 0x7f5f6c15bca0 2024-09-16T15:51:20.485 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:20.486+0000 7f5f627fc640 1 -- 172.21.15.123:0/3635757065 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 24) ==== 97845+0+0 (secure 0 0 0) 0x7f5f640054e0 con 0x7f5f6c15bca0 2024-09-16T15:51:20.485 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:20.486+0000 7f5f627fc640 1 --2- 172.21.15.123:0/3635757065 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f5f50073730 0x7f5f50075bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:51:20.485 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:20.486+0000 7f5f71fd8640 1 --2- 172.21.15.123:0/3635757065 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f5f50073730 0x7f5f50075bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:51:20.485 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:20.486+0000 7f5f627fc640 1 -- 172.21.15.123:0/3635757065 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(157..157 src has 1..157) ==== 5737+0+0 (secure 0 0 0) 0x7f5f64093210 con 0x7f5f6c15bca0 2024-09-16T15:51:20.485 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:20.486+0000 7f5f71fd8640 1 --2- 172.21.15.123:0/3635757065 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f5f50073730 0x7f5f50075bf0 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f5f5c009650 tx=0x7f5f5c006bb0 comp rx=0 tx=0).ready entity=mgr.4150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:51:20.488 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:20.490+0000 7f5f627fc640 1 -- 172.21.15.123:0/3635757065 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f5f64014030 con 0x7f5f6c15bca0 2024-09-16T15:51:20.660 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:20.662+0000 7f5f7423c640 1 -- 172.21.15.123:0/3635757065 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f5f6c0700e0 con 0x7f5f6c15bca0 2024-09-16T15:51:20.661 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:20.662+0000 7f5f627fc640 1 -- 172.21.15.123:0/3635757065 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v157) ==== 74+0+16843 (secure 0 0 0) 0x7f5f6405ff10 con 0x7f5f6c15bca0 2024-09-16T15:51:20.661 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-09-16T15:51:20.662 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":157,"fsid":"56077f89-18c8-486f-ae7e-eecc0bb7822c","created":"2024-09-16T15:42:26.364892+0000","modified":"2024-09-16T15:51:17.980147+0000","last_up_change":"2024-09-16T15:42:38.707234+0000","last_in_change":"2024-09-16T15:42:29.423662+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":25,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-16T15:42:39.287764+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":24,"pool_name":"cephfs_metadata","create_time":"2024-09-16T15:51:13.953990+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"156","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":25,"pool_name":"cephfs_data","create_time":"2024-09-16T15:51:15.281388+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"157","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9e7c1955-a607-4671-9be7-f2ef760ceeaa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":155,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6800","nonce":3882044969},{"type":"v1","addr":"172.21.15.123:6801","nonce":3882044969}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6802","nonce":3882044969},{"type":"v1","addr":"172.21.15.123:6803","nonce":3882044969}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6806","nonce":3882044969},{"type":"v1","addr":"172.21.15.123:6807","nonce":3882044969}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6804","nonce":3882044969},{"type":"v1","addr":"172.21.15.123:6805","nonce":3882044969}]},"public_addr":"172.21.15.123:6801/3882044969","cluster_addr":"172.21.15.123:6803/3882044969","heartbeat_back_addr":"172.21.15.123:6807/3882044969","heartbeat_front_addr":"172.21.15.123:6805/3882044969","state":["exists","up"]},{"osd":1,"uuid":"c540cfde-0977-4742-9b17-fd634668e010","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":155,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6816","nonce":4037001850},{"type":"v1","addr":"172.21.15.123:6817","nonce":4037001850}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6818","nonce":4037001850},{"type":"v1","addr":"172.21.15.123:6819","nonce":4037001850}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6822","nonce":4037001850},{"type":"v1","addr":"172.21.15.123:6823","nonce":4037001850}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6820","nonce":4037001850},{"type":"v1","addr":"172.21.15.123:6821","nonce":4037001850}]},"public_addr":"172.21.15.123:6817/4037001850","cluster_addr":"172.21.15.123:6819/4037001850","heartbeat_back_addr":"172.21.15.123:6823/4037001850","heartbeat_front_addr":"172.21.15.123:6821/4037001850","state":["exists","up"]},{"osd":2,"uuid":"9864a496-f169-486b-95c4-1d59162793a7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":155,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6824","nonce":1332143088},{"type":"v1","addr":"172.21.15.123:6825","nonce":1332143088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6826","nonce":1332143088},{"type":"v1","addr":"172.21.15.123:6827","nonce":1332143088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6830","nonce":1332143088},{"type":"v1","addr":"172.21.15.123:6831","nonce":1332143088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6828","nonce":1332143088},{"type":"v1","addr":"172.21.15.123:6829","nonce":1332143088}]},"public_addr":"172.21.15.123:6825/1332143088","cluster_addr":"172.21.15.123:6827/1332143088","heartbeat_back_addr":"172.21.15.123:6831/1332143088","heartbeat_front_addr":"172.21.15.123:6829/1332143088","state":["exists","up"]},{"osd":3,"uuid":"c6d1f817-f074-4325-95e1-578eba13edda","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":155,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6808","nonce":988586998},{"type":"v1","addr":"172.21.15.123:6809","nonce":988586998}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6810","nonce":988586998},{"type":"v1","addr":"172.21.15.123:6811","nonce":988586998}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6814","nonce":988586998},{"type":"v1","addr":"172.21.15.123:6815","nonce":988586998}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6812","nonce":988586998},{"type":"v1","addr":"172.21.15.123:6813","nonce":988586998}]},"public_addr":"172.21.15.123:6809/988586998","cluster_addr":"172.21.15.123:6811/988586998","heartbeat_back_addr":"172.21.15.123:6815/988586998","heartbeat_front_addr":"172.21.15.123:6813/988586998","state":["exists","up"]},{"osd":4,"uuid":"200a8df4-50ef-40a5-b48a-58981269c086","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":155,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6816","nonce":3387813890},{"type":"v1","addr":"172.21.15.184:6817","nonce":3387813890}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6818","nonce":3387813890},{"type":"v1","addr":"172.21.15.184:6819","nonce":3387813890}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6822","nonce":3387813890},{"type":"v1","addr":"172.21.15.184:6823","nonce":3387813890}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6820","nonce":3387813890},{"type":"v1","addr":"172.21.15.184:6821","nonce":3387813890}]},"public_addr":"172.21.15.184:6817/3387813890","cluster_addr":"172.21.15.184:6819/3387813890","heartbeat_back_addr":"172.21.15.184:6823/3387813890","heartbeat_front_addr":"172.21.15.184:6821/3387813890","state":["exists","up"]},{"osd":5,"uuid":"32d5a900-da3d-412c-bc19-13861a2368d8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":155,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6808","nonce":2680916017},{"type":"v1","addr":"172.21.15.184:6809","nonce":2680916017}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6810","nonce":2680916017},{"type":"v1","addr":"172.21.15.184:6811","nonce":2680916017}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6814","nonce":2680916017},{"type":"v1","addr":"172.21.15.184:6815","nonce":2680916017}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6812","nonce":2680916017},{"type":"v1","addr":"172.21.15.184:6813","nonce":2680916017}]},"public_addr":"172.21.15.184:6809/2680916017","cluster_addr":"172.21.15.184:6811/2680916017","heartbeat_back_addr":"172.21.15.184:6815/2680916017","heartbeat_front_addr":"172.21.15.184:6813/2680916017","state":["exists","up"]},{"osd":6,"uuid":"cde6f39e-aa1b-4260-ac19-774ee6aaca72","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":155,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6824","nonce":2183120457},{"type":"v1","addr":"172.21.15.184:6825","nonce":2183120457}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6826","nonce":2183120457},{"type":"v1","addr":"172.21.15.184:6827","nonce":2183120457}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6830","nonce":2183120457},{"type":"v1","addr":"172.21.15.184:6831","nonce":2183120457}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6828","nonce":2183120457},{"type":"v1","addr":"172.21.15.184:6829","nonce":2183120457}]},"public_addr":"172.21.15.184:6825/2183120457","cluster_addr":"172.21.15.184:6827/2183120457","heartbeat_back_addr":"172.21.15.184:6831/2183120457","heartbeat_front_addr":"172.21.15.184:6829/2183120457","state":["exists","up"]},{"osd":7,"uuid":"4f381778-eee1-47c6-804f-df0e6b33e313","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":155,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6800","nonce":4102957285},{"type":"v1","addr":"172.21.15.184:6801","nonce":4102957285}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6802","nonce":4102957285},{"type":"v1","addr":"172.21.15.184:6803","nonce":4102957285}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6806","nonce":4102957285},{"type":"v1","addr":"172.21.15.184:6807","nonce":4102957285}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6804","nonce":4102957285},{"type":"v1","addr":"172.21.15.184:6805","nonce":4102957285}]},"public_addr":"172.21.15.184:6801/4102957285","cluster_addr":"172.21.15.184:6803/4102957285","heartbeat_back_addr":"172.21.15.184:6807/4102957285","heartbeat_front_addr":"172.21.15.184:6805/4102957285","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:35.972210+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.222007+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.198617+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.385663+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.358935+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.156066+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.752973+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:35.682846+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":17,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T15:51:20.664 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:20.666+0000 7f5f7423c640 1 -- 172.21.15.123:0/3635757065 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f5f50073730 msgr2=0x7f5f50075bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:51:20.664 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:20.666+0000 7f5f7423c640 1 --2- 172.21.15.123:0/3635757065 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f5f50073730 0x7f5f50075bf0 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f5f5c009650 tx=0x7f5f5c006bb0 comp rx=0 tx=0).stop 2024-09-16T15:51:20.664 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:20.666+0000 7f5f7423c640 1 -- 172.21.15.123:0/3635757065 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5f6c15bca0 msgr2=0x7f5f6c156d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:51:20.664 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:20.666+0000 7f5f7423c640 1 --2- 172.21.15.123:0/3635757065 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5f6c15bca0 0x7f5f6c156d40 secure :-1 s=READY pgs=1745 cs=0 l=1 rev1=1 crypto rx=0x7f5f6400ccb0 tx=0x7f5f6400a620 comp rx=0 tx=0).stop 2024-09-16T15:51:20.664 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:20.666+0000 7f5f7423c640 1 -- 172.21.15.123:0/3635757065 shutdown_connections 2024-09-16T15:51:20.664 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:20.666+0000 7f5f7423c640 1 --2- 172.21.15.123:0/3635757065 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f5f50073730 0x7f5f50075bf0 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:20.665 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:20.666+0000 7f5f7423c640 1 --2- 172.21.15.123:0/3635757065 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5f6c15bca0 0x7f5f6c156d40 unknown :-1 s=CLOSED pgs=1745 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:20.665 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:20.666+0000 7f5f7423c640 1 --2- 172.21.15.123:0/3635757065 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f5f6c074600 0x7f5f6c0c7460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:20.665 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:20.666+0000 7f5f7423c640 1 --2- 172.21.15.123:0/3635757065 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5f6c0726c0 0x7f5f6c156070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:20.665 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:20.666+0000 7f5f7423c640 1 -- 172.21.15.123:0/3635757065 >> 172.21.15.123:0/3635757065 conn(0x7f5f6c0cfb10 msgr2=0x7f5f6c15ae50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:51:20.665 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:20.666+0000 7f5f7423c640 1 -- 172.21.15.123:0/3635757065 shutdown_connections 2024-09-16T15:51:20.665 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:20.666+0000 7f5f7423c640 1 -- 172.21.15.123:0/3635757065 wait complete. 2024-09-16T15:51:20.680 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-09-16T15:51:20.680 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T15:51:20.794 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:20.794+0000 7fd28c94f640 1 -- 172.21.15.123:0/3071577821 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd288074600 msgr2=0x7fd28806e750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:51:20.795 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:20.794+0000 7fd28c94f640 1 --2- 172.21.15.123:0/3071577821 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd288074600 0x7fd28806e750 secure :-1 s=READY pgs=1746 cs=0 l=1 rev1=1 crypto rx=0x7fd280009c80 tx=0x7fd28001cce0 comp rx=0 tx=0).stop 2024-09-16T15:51:20.795 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:20.794+0000 7fd28c94f640 1 -- 172.21.15.123:0/3071577821 shutdown_connections 2024-09-16T15:51:20.795 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:20.794+0000 7fd28c94f640 1 --2- 172.21.15.123:0/3071577821 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd288074600 0x7fd28806e750 unknown :-1 s=CLOSED pgs=1746 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:20.795 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:20.794+0000 7fd28c94f640 1 --2- 172.21.15.123:0/3071577821 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fd2880726c0 0x7fd2880d0630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:20.795 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:20.794+0000 7fd28c94f640 1 -- 172.21.15.123:0/3071577821 >> 172.21.15.123:0/3071577821 conn(0x7fd2880cfb10 msgr2=0x7fd2880cff20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:51:20.795 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:20.794+0000 7fd28c94f640 1 -- 172.21.15.123:0/3071577821 shutdown_connections 2024-09-16T15:51:20.795 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:20.794+0000 7fd28c94f640 1 -- 172.21.15.123:0/3071577821 wait complete. 2024-09-16T15:51:20.795 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:20.798+0000 7fd28c94f640 1 Processor -- start 2024-09-16T15:51:20.796 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:20.798+0000 7fd28c94f640 1 -- start start 2024-09-16T15:51:20.797 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:20.798+0000 7fd28c94f640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd2880726c0 0x7fd28815e830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:51:20.797 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:20.798+0000 7fd28c94f640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd288074600 0x7fd2881f7a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:51:20.797 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:20.798+0000 7fd28c94f640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fd28815bca0 0x7fd2880c8c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:51:20.797 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:20.798+0000 7fd28c94f640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fd288168a50 con 0x7fd288074600 2024-09-16T15:51:20.798 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:20.798+0000 7fd28c94f640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fd28806fa60 con 0x7fd2880726c0 2024-09-16T15:51:20.798 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:20.798+0000 7fd28c94f640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fd288168bd0 con 0x7fd28815bca0 2024-09-16T15:51:20.798 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:20.798+0000 7fd27ffff640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd288074600 0x7fd2881f7a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:51:20.798 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:20.798+0000 7fd27ffff640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd288074600 0x7fd2881f7a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:45156/0 (socket says 172.21.15.123:45156) 2024-09-16T15:51:20.798 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:20.798+0000 7fd27ffff640 1 -- 172.21.15.123:0/3419564274 learned_addr learned my addr 172.21.15.123:0/3419564274 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:51:20.798 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:20.798+0000 7fd27ffff640 1 -- 172.21.15.123:0/3419564274 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fd28815bca0 msgr2=0x7fd2880c8c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:51:20.798 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:20.798+0000 7fd28659c640 1 --2- 172.21.15.123:0/3419564274 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd2880726c0 0x7fd28815e830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:51:20.798 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:20.798+0000 7fd286d9d640 1 --2- 172.21.15.123:0/3419564274 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fd28815bca0 0x7fd2880c8c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:51:20.798 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:20.798+0000 7fd27ffff640 1 --2- 172.21.15.123:0/3419564274 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fd28815bca0 0x7fd2880c8c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:20.798 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:20.798+0000 7fd27ffff640 1 -- 172.21.15.123:0/3419564274 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd2880726c0 msgr2=0x7fd28815e830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:51:20.798 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:20.798+0000 7fd27ffff640 1 --2- 172.21.15.123:0/3419564274 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd2880726c0 0x7fd28815e830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:20.798 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:20.798+0000 7fd27ffff640 1 -- 172.21.15.123:0/3419564274 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd2880c8190 con 0x7fd288074600 2024-09-16T15:51:20.798 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:20.798+0000 7fd28659c640 1 --2- 172.21.15.123:0/3419564274 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd2880726c0 0x7fd28815e830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T15:51:20.798 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:20.798+0000 7fd27ffff640 1 --2- 172.21.15.123:0/3419564274 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd288074600 0x7fd2881f7a10 secure :-1 s=READY pgs=1747 cs=0 l=1 rev1=1 crypto rx=0x7fd27000d980 tx=0x7fd27000de40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:51:20.799 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:20.798+0000 7fd27effd640 1 -- 172.21.15.123:0/3419564274 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd270017070 con 0x7fd288074600 2024-09-16T15:51:20.799 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:20.798+0000 7fd27effd640 1 -- 172.21.15.123:0/3419564274 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fd270009d60 con 0x7fd288074600 2024-09-16T15:51:20.799 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:20.798+0000 7fd28c94f640 1 -- 172.21.15.123:0/3419564274 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd2881f84c0 con 0x7fd288074600 2024-09-16T15:51:20.799 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:20.798+0000 7fd27effd640 1 -- 172.21.15.123:0/3419564274 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd270005280 con 0x7fd288074600 2024-09-16T15:51:20.799 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:20.798+0000 7fd28c94f640 1 -- 172.21.15.123:0/3419564274 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd2880c8450 con 0x7fd288074600 2024-09-16T15:51:20.800 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:20.798+0000 7fd28c94f640 1 -- 172.21.15.123:0/3419564274 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd2880d0630 con 0x7fd288074600 2024-09-16T15:51:20.800 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:20.798+0000 7fd27effd640 1 -- 172.21.15.123:0/3419564274 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 24) ==== 97845+0+0 (secure 0 0 0) 0x7fd270005420 con 0x7fd288074600 2024-09-16T15:51:20.800 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:20.802+0000 7fd27effd640 1 --2- 172.21.15.123:0/3419564274 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fd268073730 0x7fd268075bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:51:20.800 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:20.802+0000 7fd28659c640 1 --2- 172.21.15.123:0/3419564274 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fd268073730 0x7fd268075bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:51:20.800 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:20.802+0000 7fd27effd640 1 -- 172.21.15.123:0/3419564274 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(157..157 src has 1..157) ==== 5737+0+0 (secure 0 0 0) 0x7fd270092860 con 0x7fd288074600 2024-09-16T15:51:20.800 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:20.802+0000 7fd28659c640 1 --2- 172.21.15.123:0/3419564274 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fd268073730 0x7fd268075bf0 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7fd2880b4370 tx=0x7fd278009040 comp rx=0 tx=0).ready entity=mgr.4150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:51:20.803 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:20.806+0000 7fd27effd640 1 -- 172.21.15.123:0/3419564274 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7fd27005f560 con 0x7fd288074600 2024-09-16T15:51:20.974 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:20.974+0000 7fd28c94f640 1 -- 172.21.15.123:0/3419564274 --> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fd28815f630 con 0x7fd268073730 2024-09-16T15:51:20.985 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:20.986+0000 7fd27effd640 1 -- 172.21.15.123:0/3419564274 <== mgr.4150 v2:172.21.15.123:6836/2464759676 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+252480 (secure 0 0 0) 0x7fd28815f630 con 0x7fd268073730 2024-09-16T15:51:20.987 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-09-16T15:51:20.989 INFO:teuthology.orchestra.run.smithi123.stderr:dumped all 2024-09-16T15:51:20.990 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:20.990+0000 7fd28c94f640 1 -- 172.21.15.123:0/3419564274 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fd268073730 msgr2=0x7fd268075bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:51:20.990 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:20.990+0000 7fd28c94f640 1 --2- 172.21.15.123:0/3419564274 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fd268073730 0x7fd268075bf0 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7fd2880b4370 tx=0x7fd278009040 comp rx=0 tx=0).stop 2024-09-16T15:51:20.990 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:20.990+0000 7fd28c94f640 1 -- 172.21.15.123:0/3419564274 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd288074600 msgr2=0x7fd2881f7a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:51:20.990 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:20.990+0000 7fd28c94f640 1 --2- 172.21.15.123:0/3419564274 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd288074600 0x7fd2881f7a10 secure :-1 s=READY pgs=1747 cs=0 l=1 rev1=1 crypto rx=0x7fd27000d980 tx=0x7fd27000de40 comp rx=0 tx=0).stop 2024-09-16T15:51:20.990 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:20.990+0000 7fd28c94f640 1 -- 172.21.15.123:0/3419564274 shutdown_connections 2024-09-16T15:51:20.990 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:20.990+0000 7fd28c94f640 1 --2- 172.21.15.123:0/3419564274 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fd268073730 0x7fd268075bf0 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:20.990 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:20.990+0000 7fd28c94f640 1 --2- 172.21.15.123:0/3419564274 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fd28815bca0 0x7fd2880c8c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:20.990 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:20.990+0000 7fd28c94f640 1 --2- 172.21.15.123:0/3419564274 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd288074600 0x7fd2881f7a10 unknown :-1 s=CLOSED pgs=1747 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:20.990 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:20.990+0000 7fd28c94f640 1 --2- 172.21.15.123:0/3419564274 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd2880726c0 0x7fd28815e830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:20.991 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:20.990+0000 7fd28c94f640 1 -- 172.21.15.123:0/3419564274 >> 172.21.15.123:0/3419564274 conn(0x7fd2880cfb10 msgr2=0x7fd28815b0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:51:20.991 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:20.990+0000 7fd28c94f640 1 -- 172.21.15.123:0/3419564274 shutdown_connections 2024-09-16T15:51:20.991 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:20.990+0000 7fd28c94f640 1 -- 172.21.15.123:0/3419564274 wait complete. 2024-09-16T15:51:21.005 INFO:teuthology.orchestra.run.smithi123.stdout:{"pg_ready":true,"pg_map":{"version":201,"stamp":"2024-09-16T15:51:20.134238+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":158,"num_write_kb":1642,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":118,"ondisk_log_size":118,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":188,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":270332,"kb_used_data":16648,"kb_used_omap":12,"kb_used_meta":253491,"kb_avail":754704388,"statfs":{"total":773094113280,"available":772817293312,"internally_reserved":0,"allocated":17047552,"data_stored":8188708,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":259575345},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":37,"apply_latency_ms":37,"commit_latency_ns":37000000,"apply_latency_ns":37000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.214851"},"pg_stats":[{"pgid":"25.3f","version":"0'0","reported_seq":13,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-16T15:51:18.230610+0000","last_change":"2024-09-16T15:51:17.014998+0000","last_active":"2024-09-16T15:51:18.230610+0000","last_peered":"2024-09-16T15:51:18.230610+0000","last_clean":"2024-09-16T15:51:18.230610+0000","last_became_active":"2024-09-16T15:51:17.014631+0000","last_became_peered":"2024-09-16T15:51:17.014631+0000","last_unstale":"2024-09-16T15:51:18.230610+0000","last_undegraded":"2024-09-16T15:51:18.230610+0000","last_fullsized":"2024-09-16T15:51:18.230610+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:51:15.917022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:51:15.917022+0000","last_clean_scrub_stamp":"2024-09-16T15:51:15.917022+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T03:01:32.062858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.3e","version":"0'0","reported_seq":13,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-16T15:51:17.993676+0000","last_change":"2024-09-16T15:51:17.000581+0000","last_active":"2024-09-16T15:51:17.993676+0000","last_peered":"2024-09-16T15:51:17.993676+0000","last_clean":"2024-09-16T15:51:17.993676+0000","last_became_active":"2024-09-16T15:51:17.000052+0000","last_became_peered":"2024-09-16T15:51:17.000052+0000","last_unstale":"2024-09-16T15:51:17.993676+0000","last_undegraded":"2024-09-16T15:51:17.993676+0000","last_fullsized":"2024-09-16T15:51:17.993676+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:51:15.917022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:51:15.917022+0000","last_clean_scrub_stamp":"2024-09-16T15:51:15.917022+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T01:53:19.860671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"25.3d","version":"0'0","reported_seq":13,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-16T15:51:18.006233+0000","last_change":"2024-09-16T15:51:16.990253+0000","last_active":"2024-09-16T15:51:18.006233+0000","last_peered":"2024-09-16T15:51:18.006233+0000","last_clean":"2024-09-16T15:51:18.006233+0000","last_became_active":"2024-09-16T15:51:16.989274+0000","last_became_peered":"2024-09-16T15:51:16.989274+0000","last_unstale":"2024-09-16T15:51:18.006233+0000","last_undegraded":"2024-09-16T15:51:18.006233+0000","last_fullsized":"2024-09-16T15:51:18.006233+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:51:15.917022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:51:15.917022+0000","last_clean_scrub_stamp":"2024-09-16T15:51:15.917022+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:19:12.127860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.3c","version":"0'0","reported_seq":13,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-16T15:51:17.993067+0000","last_change":"2024-09-16T15:51:17.003322+0000","last_active":"2024-09-16T15:51:17.993067+0000","last_peered":"2024-09-16T15:51:17.993067+0000","last_clean":"2024-09-16T15:51:17.993067+0000","last_became_active":"2024-09-16T15:51:17.003015+0000","last_became_peered":"2024-09-16T15:51:17.003015+0000","last_unstale":"2024-09-16T15:51:17.993067+0000","last_undegraded":"2024-09-16T15:51:17.993067+0000","last_fullsized":"2024-09-16T15:51:17.993067+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:51:15.917022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:51:15.917022+0000","last_clean_scrub_stamp":"2024-09-16T15:51:15.917022+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T03:35:56.618749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"25.3b","version":"0'0","reported_seq":13,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-16T15:51:18.229987+0000","last_change":"2024-09-16T15:51:17.011755+0000","last_active":"2024-09-16T15:51:18.229987+0000","last_peered":"2024-09-16T15:51:18.229987+0000","last_clean":"2024-09-16T15:51:18.229987+0000","last_became_active":"2024-09-16T15:51:17.008469+0000","last_became_peered":"2024-09-16T15:51:17.008469+0000","last_unstale":"2024-09-16T15:51:18.229987+0000","last_undegraded":"2024-09-16T15:51:18.229987+0000","last_fullsized":"2024-09-16T15:51:18.229987+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:51:15.917022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:51:15.917022+0000","last_clean_scrub_stamp":"2024-09-16T15:51:15.917022+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T00:10:10.865390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.3a","version":"0'0","reported_seq":13,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-16T15:51:18.006800+0000","last_change":"2024-09-16T15:51:16.998228+0000","last_active":"2024-09-16T15:51:18.006800+0000","last_peered":"2024-09-16T15:51:18.006800+0000","last_clean":"2024-09-16T15:51:18.006800+0000","last_became_active":"2024-09-16T15:51:16.997779+0000","last_became_peered":"2024-09-16T15:51:16.997779+0000","last_unstale":"2024-09-16T15:51:18.006800+0000","last_undegraded":"2024-09-16T15:51:18.006800+0000","last_fullsized":"2024-09-16T15:51:18.006800+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:51:15.917022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:51:15.917022+0000","last_clean_scrub_stamp":"2024-09-16T15:51:15.917022+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T23:56:53.937968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.39","version":"0'0","reported_seq":13,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-16T15:51:17.991210+0000","last_change":"2024-09-16T15:51:17.014388+0000","last_active":"2024-09-16T15:51:17.991210+0000","last_peered":"2024-09-16T15:51:17.991210+0000","last_clean":"2024-09-16T15:51:17.991210+0000","last_became_active":"2024-09-16T15:51:17.013962+0000","last_became_peered":"2024-09-16T15:51:17.013962+0000","last_unstale":"2024-09-16T15:51:17.991210+0000","last_undegraded":"2024-09-16T15:51:17.991210+0000","last_fullsized":"2024-09-16T15:51:17.991210+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:51:15.917022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:51:15.917022+0000","last_clean_scrub_stamp":"2024-09-16T15:51:15.917022+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T00:44:24.795570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"25.38","version":"0'0","reported_seq":13,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-16T15:51:17.991758+0000","last_change":"2024-09-16T15:51:17.002870+0000","last_active":"2024-09-16T15:51:17.991758+0000","last_peered":"2024-09-16T15:51:17.991758+0000","last_clean":"2024-09-16T15:51:17.991758+0000","last_became_active":"2024-09-16T15:51:17.002088+0000","last_became_peered":"2024-09-16T15:51:17.002088+0000","last_unstale":"2024-09-16T15:51:17.991758+0000","last_undegraded":"2024-09-16T15:51:17.991758+0000","last_fullsized":"2024-09-16T15:51:17.991758+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:51:15.917022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:51:15.917022+0000","last_clean_scrub_stamp":"2024-09-16T15:51:15.917022+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T03:14:13.376009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.37","version":"0'0","reported_seq":13,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-16T15:51:18.231842+0000","last_change":"2024-09-16T15:51:17.012914+0000","last_active":"2024-09-16T15:51:18.231842+0000","last_peered":"2024-09-16T15:51:18.231842+0000","last_clean":"2024-09-16T15:51:18.231842+0000","last_became_active":"2024-09-16T15:51:17.012645+0000","last_became_peered":"2024-09-16T15:51:17.012645+0000","last_unstale":"2024-09-16T15:51:18.231842+0000","last_undegraded":"2024-09-16T15:51:18.231842+0000","last_fullsized":"2024-09-16T15:51:18.231842+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:51:15.917022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:51:15.917022+0000","last_clean_scrub_stamp":"2024-09-16T15:51:15.917022+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T03:06:24.895890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"25.36","version":"0'0","reported_seq":13,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-16T15:51:18.001248+0000","last_change":"2024-09-16T15:51:17.013455+0000","last_active":"2024-09-16T15:51:18.001248+0000","last_peered":"2024-09-16T15:51:18.001248+0000","last_clean":"2024-09-16T15:51:18.001248+0000","last_became_active":"2024-09-16T15:51:17.012984+0000","last_became_peered":"2024-09-16T15:51:17.012984+0000","last_unstale":"2024-09-16T15:51:18.001248+0000","last_undegraded":"2024-09-16T15:51:18.001248+0000","last_fullsized":"2024-09-16T15:51:18.001248+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:51:15.917022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:51:15.917022+0000","last_clean_scrub_stamp":"2024-09-16T15:51:15.917022+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T23:38:46.804355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"25.35","version":"0'0","reported_seq":13,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-16T15:51:17.991779+0000","last_change":"2024-09-16T15:51:16.992056+0000","last_active":"2024-09-16T15:51:17.991779+0000","last_peered":"2024-09-16T15:51:17.991779+0000","last_clean":"2024-09-16T15:51:17.991779+0000","last_became_active":"2024-09-16T15:51:16.991747+0000","last_became_peered":"2024-09-16T15:51:16.991747+0000","last_unstale":"2024-09-16T15:51:17.991779+0000","last_undegraded":"2024-09-16T15:51:17.991779+0000","last_fullsized":"2024-09-16T15:51:17.991779+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:51:15.917022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:51:15.917022+0000","last_clean_scrub_stamp":"2024-09-16T15:51:15.917022+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T00:53:58.467091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"25.34","version":"0'0","reported_seq":13,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-16T15:51:17.993146+0000","last_change":"2024-09-16T15:51:16.998204+0000","last_active":"2024-09-16T15:51:17.993146+0000","last_peered":"2024-09-16T15:51:17.993146+0000","last_clean":"2024-09-16T15:51:17.993146+0000","last_became_active":"2024-09-16T15:51:16.997471+0000","last_became_peered":"2024-09-16T15:51:16.997471+0000","last_unstale":"2024-09-16T15:51:17.993146+0000","last_undegraded":"2024-09-16T15:51:17.993146+0000","last_fullsized":"2024-09-16T15:51:17.993146+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:51:15.917022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:51:15.917022+0000","last_clean_scrub_stamp":"2024-09-16T15:51:15.917022+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T03:16:06.385313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"25.33","version":"0'0","reported_seq":13,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-16T15:51:17.992530+0000","last_change":"2024-09-16T15:51:17.000248+0000","last_active":"2024-09-16T15:51:17.992530+0000","last_peered":"2024-09-16T15:51:17.992530+0000","last_clean":"2024-09-16T15:51:17.992530+0000","last_became_active":"2024-09-16T15:51:16.999368+0000","last_became_peered":"2024-09-16T15:51:16.999368+0000","last_unstale":"2024-09-16T15:51:17.992530+0000","last_undegraded":"2024-09-16T15:51:17.992530+0000","last_fullsized":"2024-09-16T15:51:17.992530+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:51:15.917022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:51:15.917022+0000","last_clean_scrub_stamp":"2024-09-16T15:51:15.917022+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:21:24.061511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"25.32","version":"0'0","reported_seq":13,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-16T15:51:18.229367+0000","last_change":"2024-09-16T15:51:17.016971+0000","last_active":"2024-09-16T15:51:18.229367+0000","last_peered":"2024-09-16T15:51:18.229367+0000","last_clean":"2024-09-16T15:51:18.229367+0000","last_became_active":"2024-09-16T15:51:17.016552+0000","last_became_peered":"2024-09-16T15:51:17.016552+0000","last_unstale":"2024-09-16T15:51:18.229367+0000","last_undegraded":"2024-09-16T15:51:18.229367+0000","last_fullsized":"2024-09-16T15:51:18.229367+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:51:15.917022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:51:15.917022+0000","last_clean_scrub_stamp":"2024-09-16T15:51:15.917022+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:27:22.978905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"25.31","version":"0'0","reported_seq":13,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-16T15:51:17.992370+0000","last_change":"2024-09-16T15:51:16.998890+0000","last_active":"2024-09-16T15:51:17.992370+0000","last_peered":"2024-09-16T15:51:17.992370+0000","last_clean":"2024-09-16T15:51:17.992370+0000","last_became_active":"2024-09-16T15:51:16.998347+0000","last_became_peered":"2024-09-16T15:51:16.998347+0000","last_unstale":"2024-09-16T15:51:17.992370+0000","last_undegraded":"2024-09-16T15:51:17.992370+0000","last_fullsized":"2024-09-16T15:51:17.992370+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:51:15.917022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:51:15.917022+0000","last_clean_scrub_stamp":"2024-09-16T15:51:15.917022+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T02:32:29.420307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"25.30","version":"0'0","reported_seq":13,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-16T15:51:17.991949+0000","last_change":"2024-09-16T15:51:17.014409+0000","last_active":"2024-09-16T15:51:17.991949+0000","last_peered":"2024-09-16T15:51:17.991949+0000","last_clean":"2024-09-16T15:51:17.991949+0000","last_became_active":"2024-09-16T15:51:17.013249+0000","last_became_peered":"2024-09-16T15:51:17.013249+0000","last_unstale":"2024-09-16T15:51:17.991949+0000","last_undegraded":"2024-09-16T15:51:17.991949+0000","last_fullsized":"2024-09-16T15:51:17.991949+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:51:15.917022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:51:15.917022+0000","last_clean_scrub_stamp":"2024-09-16T15:51:15.917022+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:36:03.637349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.2f","version":"0'0","reported_seq":13,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-16T15:51:18.230343+0000","last_change":"2024-09-16T15:51:17.007407+0000","last_active":"2024-09-16T15:51:18.230343+0000","last_peered":"2024-09-16T15:51:18.230343+0000","last_clean":"2024-09-16T15:51:18.230343+0000","last_became_active":"2024-09-16T15:51:17.005329+0000","last_became_peered":"2024-09-16T15:51:17.005329+0000","last_unstale":"2024-09-16T15:51:18.230343+0000","last_undegraded":"2024-09-16T15:51:18.230343+0000","last_fullsized":"2024-09-16T15:51:18.230343+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:51:15.917022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:51:15.917022+0000","last_clean_scrub_stamp":"2024-09-16T15:51:15.917022+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T01:27:54.077516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.2e","version":"0'0","reported_seq":13,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-16T15:51:18.230155+0000","last_change":"2024-09-16T15:51:17.013603+0000","last_active":"2024-09-16T15:51:18.230155+0000","last_peered":"2024-09-16T15:51:18.230155+0000","last_clean":"2024-09-16T15:51:18.230155+0000","last_became_active":"2024-09-16T15:51:17.013267+0000","last_became_peered":"2024-09-16T15:51:17.013267+0000","last_unstale":"2024-09-16T15:51:18.230155+0000","last_undegraded":"2024-09-16T15:51:18.230155+0000","last_fullsized":"2024-09-16T15:51:18.230155+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:51:15.917022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:51:15.917022+0000","last_clean_scrub_stamp":"2024-09-16T15:51:15.917022+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:50:07.541567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.2d","version":"0'0","reported_seq":13,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-16T15:51:17.999660+0000","last_change":"2024-09-16T15:51:17.006352+0000","last_active":"2024-09-16T15:51:17.999660+0000","last_peered":"2024-09-16T15:51:17.999660+0000","last_clean":"2024-09-16T15:51:17.999660+0000","last_became_active":"2024-09-16T15:51:17.006061+0000","last_became_peered":"2024-09-16T15:51:17.006061+0000","last_unstale":"2024-09-16T15:51:17.999660+0000","last_undegraded":"2024-09-16T15:51:17.999660+0000","last_fullsized":"2024-09-16T15:51:17.999660+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:51:15.917022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:51:15.917022+0000","last_clean_scrub_stamp":"2024-09-16T15:51:15.917022+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:50:07.541567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"25.2c","version":"0'0","reported_seq":13,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-16T15:51:18.230085+0000","last_change":"2024-09-16T15:51:17.005441+0000","last_active":"2024-09-16T15:51:18.230085+0000","last_peered":"2024-09-16T15:51:18.230085+0000","last_clean":"2024-09-16T15:51:18.230085+0000","last_became_active":"2024-09-16T15:51:17.004994+0000","last_became_peered":"2024-09-16T15:51:17.004994+0000","last_unstale":"2024-09-16T15:51:18.230085+0000","last_undegraded":"2024-09-16T15:51:18.230085+0000","last_fullsized":"2024-09-16T15:51:18.230085+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:51:15.917022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:51:15.917022+0000","last_clean_scrub_stamp":"2024-09-16T15:51:15.917022+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T02:46:37.039388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.2b","version":"0'0","reported_seq":13,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-16T15:51:18.229848+0000","last_change":"2024-09-16T15:51:17.008733+0000","last_active":"2024-09-16T15:51:18.229848+0000","last_peered":"2024-09-16T15:51:18.229848+0000","last_clean":"2024-09-16T15:51:18.229848+0000","last_became_active":"2024-09-16T15:51:17.008070+0000","last_became_peered":"2024-09-16T15:51:17.008070+0000","last_unstale":"2024-09-16T15:51:18.229848+0000","last_undegraded":"2024-09-16T15:51:18.229848+0000","last_fullsized":"2024-09-16T15:51:18.229848+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:51:15.917022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:51:15.917022+0000","last_clean_scrub_stamp":"2024-09-16T15:51:15.917022+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:42:02.533881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.2a","version":"0'0","reported_seq":13,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-16T15:51:17.999123+0000","last_change":"2024-09-16T15:51:17.012375+0000","last_active":"2024-09-16T15:51:17.999123+0000","last_peered":"2024-09-16T15:51:17.999123+0000","last_clean":"2024-09-16T15:51:17.999123+0000","last_became_active":"2024-09-16T15:51:17.011538+0000","last_became_peered":"2024-09-16T15:51:17.011538+0000","last_unstale":"2024-09-16T15:51:17.999123+0000","last_undegraded":"2024-09-16T15:51:17.999123+0000","last_fullsized":"2024-09-16T15:51:17.999123+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:51:15.917022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:51:15.917022+0000","last_clean_scrub_stamp":"2024-09-16T15:51:15.917022+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:16:48.939808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"25.29","version":"0'0","reported_seq":13,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-16T15:51:18.006020+0000","last_change":"2024-09-16T15:51:17.015534+0000","last_active":"2024-09-16T15:51:18.006020+0000","last_peered":"2024-09-16T15:51:18.006020+0000","last_clean":"2024-09-16T15:51:18.006020+0000","last_became_active":"2024-09-16T15:51:17.015229+0000","last_became_peered":"2024-09-16T15:51:17.015229+0000","last_unstale":"2024-09-16T15:51:18.006020+0000","last_undegraded":"2024-09-16T15:51:18.006020+0000","last_fullsized":"2024-09-16T15:51:18.006020+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:51:15.917022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:51:15.917022+0000","last_clean_scrub_stamp":"2024-09-16T15:51:15.917022+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:10:02.074524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.28","version":"0'0","reported_seq":13,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-16T15:51:17.992298+0000","last_change":"2024-09-16T15:51:16.998452+0000","last_active":"2024-09-16T15:51:17.992298+0000","last_peered":"2024-09-16T15:51:17.992298+0000","last_clean":"2024-09-16T15:51:17.992298+0000","last_became_active":"2024-09-16T15:51:16.997685+0000","last_became_peered":"2024-09-16T15:51:16.997685+0000","last_unstale":"2024-09-16T15:51:17.992298+0000","last_undegraded":"2024-09-16T15:51:17.992298+0000","last_fullsized":"2024-09-16T15:51:17.992298+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:51:15.917022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:51:15.917022+0000","last_clean_scrub_stamp":"2024-09-16T15:51:15.917022+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:28:52.947127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.27","version":"0'0","reported_seq":13,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-16T15:51:17.993956+0000","last_change":"2024-09-16T15:51:17.012326+0000","last_active":"2024-09-16T15:51:17.993956+0000","last_peered":"2024-09-16T15:51:17.993956+0000","last_clean":"2024-09-16T15:51:17.993956+0000","last_became_active":"2024-09-16T15:51:17.011953+0000","last_became_peered":"2024-09-16T15:51:17.011953+0000","last_unstale":"2024-09-16T15:51:17.993956+0000","last_undegraded":"2024-09-16T15:51:17.993956+0000","last_fullsized":"2024-09-16T15:51:17.993956+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:51:15.917022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:51:15.917022+0000","last_clean_scrub_stamp":"2024-09-16T15:51:15.917022+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:01:30.205778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"25.26","version":"0'0","reported_seq":13,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-16T15:51:17.990846+0000","last_change":"2024-09-16T15:51:17.013974+0000","last_active":"2024-09-16T15:51:17.990846+0000","last_peered":"2024-09-16T15:51:17.990846+0000","last_clean":"2024-09-16T15:51:17.990846+0000","last_became_active":"2024-09-16T15:51:17.013672+0000","last_became_peered":"2024-09-16T15:51:17.013672+0000","last_unstale":"2024-09-16T15:51:17.990846+0000","last_undegraded":"2024-09-16T15:51:17.990846+0000","last_fullsized":"2024-09-16T15:51:17.990846+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:51:15.917022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:51:15.917022+0000","last_clean_scrub_stamp":"2024-09-16T15:51:15.917022+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:58:41.585587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"25.25","version":"0'0","reported_seq":13,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-16T15:51:18.006635+0000","last_change":"2024-09-16T15:51:16.993740+0000","last_active":"2024-09-16T15:51:18.006635+0000","last_peered":"2024-09-16T15:51:18.006635+0000","last_clean":"2024-09-16T15:51:18.006635+0000","last_became_active":"2024-09-16T15:51:16.993040+0000","last_became_peered":"2024-09-16T15:51:16.993040+0000","last_unstale":"2024-09-16T15:51:18.006635+0000","last_undegraded":"2024-09-16T15:51:18.006635+0000","last_fullsized":"2024-09-16T15:51:18.006635+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:51:15.917022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:51:15.917022+0000","last_clean_scrub_stamp":"2024-09-16T15:51:15.917022+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T03:13:06.576190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.24","version":"0'0","reported_seq":13,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-16T15:51:18.230368+0000","last_change":"2024-09-16T15:51:17.011958+0000","last_active":"2024-09-16T15:51:18.230368+0000","last_peered":"2024-09-16T15:51:18.230368+0000","last_clean":"2024-09-16T15:51:18.230368+0000","last_became_active":"2024-09-16T15:51:17.011498+0000","last_became_peered":"2024-09-16T15:51:17.011498+0000","last_unstale":"2024-09-16T15:51:18.230368+0000","last_undegraded":"2024-09-16T15:51:18.230368+0000","last_fullsized":"2024-09-16T15:51:18.230368+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:51:15.917022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:51:15.917022+0000","last_clean_scrub_stamp":"2024-09-16T15:51:15.917022+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T03:50:14.344816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.23","version":"0'0","reported_seq":13,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-16T15:51:17.992868+0000","last_change":"2024-09-16T15:51:16.997588+0000","last_active":"2024-09-16T15:51:17.992868+0000","last_peered":"2024-09-16T15:51:17.992868+0000","last_clean":"2024-09-16T15:51:17.992868+0000","last_became_active":"2024-09-16T15:51:16.996903+0000","last_became_peered":"2024-09-16T15:51:16.996903+0000","last_unstale":"2024-09-16T15:51:17.992868+0000","last_undegraded":"2024-09-16T15:51:17.992868+0000","last_fullsized":"2024-09-16T15:51:17.992868+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:51:15.917022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:51:15.917022+0000","last_clean_scrub_stamp":"2024-09-16T15:51:15.917022+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T18:57:14.461693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"25.22","version":"0'0","reported_seq":13,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-16T15:51:18.230375+0000","last_change":"2024-09-16T15:51:16.997698+0000","last_active":"2024-09-16T15:51:18.230375+0000","last_peered":"2024-09-16T15:51:18.230375+0000","last_clean":"2024-09-16T15:51:18.230375+0000","last_became_active":"2024-09-16T15:51:16.997319+0000","last_became_peered":"2024-09-16T15:51:16.997319+0000","last_unstale":"2024-09-16T15:51:18.230375+0000","last_undegraded":"2024-09-16T15:51:18.230375+0000","last_fullsized":"2024-09-16T15:51:18.230375+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:51:15.917022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:51:15.917022+0000","last_clean_scrub_stamp":"2024-09-16T15:51:15.917022+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T18:40:03.594059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"25.21","version":"0'0","reported_seq":13,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-16T15:51:18.006015+0000","last_change":"2024-09-16T15:51:16.990170+0000","last_active":"2024-09-16T15:51:18.006015+0000","last_peered":"2024-09-16T15:51:18.006015+0000","last_clean":"2024-09-16T15:51:18.006015+0000","last_became_active":"2024-09-16T15:51:16.989494+0000","last_became_peered":"2024-09-16T15:51:16.989494+0000","last_unstale":"2024-09-16T15:51:18.006015+0000","last_undegraded":"2024-09-16T15:51:18.006015+0000","last_fullsized":"2024-09-16T15:51:18.006015+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:51:15.917022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:51:15.917022+0000","last_clean_scrub_stamp":"2024-09-16T15:51:15.917022+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:14:35.276401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.20","version":"0'0","reported_seq":13,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-16T15:51:18.229417+0000","last_change":"2024-09-16T15:51:16.997649+0000","last_active":"2024-09-16T15:51:18.229417+0000","last_peered":"2024-09-16T15:51:18.229417+0000","last_clean":"2024-09-16T15:51:18.229417+0000","last_became_active":"2024-09-16T15:51:16.997270+0000","last_became_peered":"2024-09-16T15:51:16.997270+0000","last_unstale":"2024-09-16T15:51:18.229417+0000","last_undegraded":"2024-09-16T15:51:18.229417+0000","last_fullsized":"2024-09-16T15:51:18.229417+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:51:15.917022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:51:15.917022+0000","last_clean_scrub_stamp":"2024-09-16T15:51:15.917022+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T18:05:42.952449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.1e","version":"0'0","reported_seq":13,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-16T15:51:17.991790+0000","last_change":"2024-09-16T15:51:17.020448+0000","last_active":"2024-09-16T15:51:17.991790+0000","last_peered":"2024-09-16T15:51:17.991790+0000","last_clean":"2024-09-16T15:51:17.991790+0000","last_became_active":"2024-09-16T15:51:17.020006+0000","last_became_peered":"2024-09-16T15:51:17.020006+0000","last_unstale":"2024-09-16T15:51:17.991790+0000","last_undegraded":"2024-09-16T15:51:17.991790+0000","last_fullsized":"2024-09-16T15:51:17.991790+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:51:15.917022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:51:15.917022+0000","last_clean_scrub_stamp":"2024-09-16T15:51:15.917022+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T02:31:40.090279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.1f","version":"156'4","reported_seq":19,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-16T15:51:18.230202+0000","last_change":"2024-09-16T15:51:15.950651+0000","last_active":"2024-09-16T15:51:18.230202+0000","last_peered":"2024-09-16T15:51:18.230202+0000","last_clean":"2024-09-16T15:51:18.230202+0000","last_became_active":"2024-09-16T15:51:15.950248+0000","last_became_peered":"2024-09-16T15:51:15.950248+0000","last_unstale":"2024-09-16T15:51:18.230202+0000","last_undegraded":"2024-09-16T15:51:18.230202+0000","last_fullsized":"2024-09-16T15:51:18.230202+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:51:14.906573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:51:14.906573+0000","last_clean_scrub_stamp":"2024-09-16T15:51:14.906573+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T01:53:33.146583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.1f","version":"0'0","reported_seq":13,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-16T15:51:18.000624+0000","last_change":"2024-09-16T15:51:17.019764+0000","last_active":"2024-09-16T15:51:18.000624+0000","last_peered":"2024-09-16T15:51:18.000624+0000","last_clean":"2024-09-16T15:51:18.000624+0000","last_became_active":"2024-09-16T15:51:17.019429+0000","last_became_peered":"2024-09-16T15:51:17.019429+0000","last_unstale":"2024-09-16T15:51:18.000624+0000","last_undegraded":"2024-09-16T15:51:18.000624+0000","last_fullsized":"2024-09-16T15:51:18.000624+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:51:15.917022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:51:15.917022+0000","last_clean_scrub_stamp":"2024-09-16T15:51:15.917022+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T18:27:49.521296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.1e","version":"156'1","reported_seq":16,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-16T15:51:18.231306+0000","last_change":"2024-09-16T15:51:15.949651+0000","last_active":"2024-09-16T15:51:18.231306+0000","last_peered":"2024-09-16T15:51:18.231306+0000","last_clean":"2024-09-16T15:51:18.231306+0000","last_became_active":"2024-09-16T15:51:15.946835+0000","last_became_peered":"2024-09-16T15:51:15.946835+0000","last_unstale":"2024-09-16T15:51:18.231306+0000","last_undegraded":"2024-09-16T15:51:18.231306+0000","last_fullsized":"2024-09-16T15:51:18.231306+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:51:14.906573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:51:14.906573+0000","last_clean_scrub_stamp":"2024-09-16T15:51:14.906573+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T02:18:13.853820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"25.1c","version":"0'0","reported_seq":13,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-16T15:51:17.991198+0000","last_change":"2024-09-16T15:51:17.019402+0000","last_active":"2024-09-16T15:51:17.991198+0000","last_peered":"2024-09-16T15:51:17.991198+0000","last_clean":"2024-09-16T15:51:17.991198+0000","last_became_active":"2024-09-16T15:51:17.018991+0000","last_became_peered":"2024-09-16T15:51:17.018991+0000","last_unstale":"2024-09-16T15:51:17.991198+0000","last_undegraded":"2024-09-16T15:51:17.991198+0000","last_fullsized":"2024-09-16T15:51:17.991198+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:51:15.917022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:51:15.917022+0000","last_clean_scrub_stamp":"2024-09-16T15:51:15.917022+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T00:47:18.987282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.1d","version":"156'2","reported_seq":17,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-16T15:51:18.006559+0000","last_change":"2024-09-16T15:51:15.958280+0000","last_active":"2024-09-16T15:51:18.006559+0000","last_peered":"2024-09-16T15:51:18.006559+0000","last_clean":"2024-09-16T15:51:18.006559+0000","last_became_active":"2024-09-16T15:51:15.957914+0000","last_became_peered":"2024-09-16T15:51:15.957914+0000","last_unstale":"2024-09-16T15:51:18.006559+0000","last_undegraded":"2024-09-16T15:51:18.006559+0000","last_fullsized":"2024-09-16T15:51:18.006559+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:51:14.906573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:51:14.906573+0000","last_clean_scrub_stamp":"2024-09-16T15:51:14.906573+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:37:41.174910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.1d","version":"0'0","reported_seq":13,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-16T15:51:18.231556+0000","last_change":"2024-09-16T15:51:16.993051+0000","last_active":"2024-09-16T15:51:18.231556+0000","last_peered":"2024-09-16T15:51:18.231556+0000","last_clean":"2024-09-16T15:51:18.231556+0000","last_became_active":"2024-09-16T15:51:16.992320+0000","last_became_peered":"2024-09-16T15:51:16.992320+0000","last_unstale":"2024-09-16T15:51:18.231556+0000","last_undegraded":"2024-09-16T15:51:18.231556+0000","last_fullsized":"2024-09-16T15:51:18.231556+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:51:15.917022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:51:15.917022+0000","last_clean_scrub_stamp":"2024-09-16T15:51:15.917022+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T02:39:46.607375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.1c","version":"156'2","reported_seq":17,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-16T15:51:17.993169+0000","last_change":"2024-09-16T15:51:15.954496+0000","last_active":"2024-09-16T15:51:17.993169+0000","last_peered":"2024-09-16T15:51:17.993169+0000","last_clean":"2024-09-16T15:51:17.993169+0000","last_became_active":"2024-09-16T15:51:15.953801+0000","last_became_peered":"2024-09-16T15:51:15.953801+0000","last_unstale":"2024-09-16T15:51:17.993169+0000","last_undegraded":"2024-09-16T15:51:17.993169+0000","last_fullsized":"2024-09-16T15:51:17.993169+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:51:14.906573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:51:14.906573+0000","last_clean_scrub_stamp":"2024-09-16T15:51:14.906573+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T22:59:22.058566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"25.1a","version":"0'0","reported_seq":13,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-16T15:51:17.999347+0000","last_change":"2024-09-16T15:51:17.011733+0000","last_active":"2024-09-16T15:51:17.999347+0000","last_peered":"2024-09-16T15:51:17.999347+0000","last_clean":"2024-09-16T15:51:17.999347+0000","last_became_active":"2024-09-16T15:51:17.011211+0000","last_became_peered":"2024-09-16T15:51:17.011211+0000","last_unstale":"2024-09-16T15:51:17.999347+0000","last_undegraded":"2024-09-16T15:51:17.999347+0000","last_fullsized":"2024-09-16T15:51:17.999347+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:51:15.917022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:51:15.917022+0000","last_clean_scrub_stamp":"2024-09-16T15:51:15.917022+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T00:28:57.939261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.1b","version":"156'1","reported_seq":16,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-16T15:51:18.230021+0000","last_change":"2024-09-16T15:51:15.960597+0000","last_active":"2024-09-16T15:51:18.230021+0000","last_peered":"2024-09-16T15:51:18.230021+0000","last_clean":"2024-09-16T15:51:18.230021+0000","last_became_active":"2024-09-16T15:51:15.957029+0000","last_became_peered":"2024-09-16T15:51:15.957029+0000","last_unstale":"2024-09-16T15:51:18.230021+0000","last_undegraded":"2024-09-16T15:51:18.230021+0000","last_fullsized":"2024-09-16T15:51:18.230021+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:51:14.906573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:51:14.906573+0000","last_clean_scrub_stamp":"2024-09-16T15:51:14.906573+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:59:01.722146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.1b","version":"0'0","reported_seq":13,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-16T15:51:17.992497+0000","last_change":"2024-09-16T15:51:17.004716+0000","last_active":"2024-09-16T15:51:17.992497+0000","last_peered":"2024-09-16T15:51:17.992497+0000","last_clean":"2024-09-16T15:51:17.992497+0000","last_became_active":"2024-09-16T15:51:17.003589+0000","last_became_peered":"2024-09-16T15:51:17.003589+0000","last_unstale":"2024-09-16T15:51:17.992497+0000","last_undegraded":"2024-09-16T15:51:17.992497+0000","last_fullsized":"2024-09-16T15:51:17.992497+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:51:15.917022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:51:15.917022+0000","last_clean_scrub_stamp":"2024-09-16T15:51:15.917022+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T18:19:15.107168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.1a","version":"156'1","reported_seq":16,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-16T15:51:17.992920+0000","last_change":"2024-09-16T15:51:15.953951+0000","last_active":"2024-09-16T15:51:17.992920+0000","last_peered":"2024-09-16T15:51:17.992920+0000","last_clean":"2024-09-16T15:51:17.992920+0000","last_became_active":"2024-09-16T15:51:15.953593+0000","last_became_peered":"2024-09-16T15:51:15.953593+0000","last_unstale":"2024-09-16T15:51:17.992920+0000","last_undegraded":"2024-09-16T15:51:17.992920+0000","last_fullsized":"2024-09-16T15:51:17.992920+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:51:14.906573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:51:14.906573+0000","last_clean_scrub_stamp":"2024-09-16T15:51:14.906573+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T01:54:24.734575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.19","version":"0'0","reported_seq":13,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-16T15:51:17.990598+0000","last_change":"2024-09-16T15:51:16.996685+0000","last_active":"2024-09-16T15:51:17.990598+0000","last_peered":"2024-09-16T15:51:17.990598+0000","last_clean":"2024-09-16T15:51:17.990598+0000","last_became_active":"2024-09-16T15:51:16.996255+0000","last_became_peered":"2024-09-16T15:51:16.996255+0000","last_unstale":"2024-09-16T15:51:17.990598+0000","last_undegraded":"2024-09-16T15:51:17.990598+0000","last_fullsized":"2024-09-16T15:51:17.990598+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:51:15.917022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:51:15.917022+0000","last_clean_scrub_stamp":"2024-09-16T15:51:15.917022+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:08:18.096948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.18","version":"0'0","reported_seq":15,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-16T15:51:18.228868+0000","last_change":"2024-09-16T15:51:15.954905+0000","last_active":"2024-09-16T15:51:18.228868+0000","last_peered":"2024-09-16T15:51:18.228868+0000","last_clean":"2024-09-16T15:51:18.228868+0000","last_became_active":"2024-09-16T15:51:15.954141+0000","last_became_peered":"2024-09-16T15:51:15.954141+0000","last_unstale":"2024-09-16T15:51:18.228868+0000","last_undegraded":"2024-09-16T15:51:18.228868+0000","last_fullsized":"2024-09-16T15:51:18.228868+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:51:14.906573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:51:14.906573+0000","last_clean_scrub_stamp":"2024-09-16T15:51:14.906573+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:01:34.100930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.16","version":"0'0","reported_seq":13,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-16T15:51:18.007088+0000","last_change":"2024-09-16T15:51:17.002955+0000","last_active":"2024-09-16T15:51:18.007088+0000","last_peered":"2024-09-16T15:51:18.007088+0000","last_clean":"2024-09-16T15:51:18.007088+0000","last_became_active":"2024-09-16T15:51:17.002668+0000","last_became_peered":"2024-09-16T15:51:17.002668+0000","last_unstale":"2024-09-16T15:51:18.007088+0000","last_undegraded":"2024-09-16T15:51:18.007088+0000","last_fullsized":"2024-09-16T15:51:18.007088+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:51:15.917022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:51:15.917022+0000","last_clean_scrub_stamp":"2024-09-16T15:51:15.917022+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:44:50.889568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.17","version":"0'0","reported_seq":15,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-16T15:51:17.990690+0000","last_change":"2024-09-16T15:51:15.956604+0000","last_active":"2024-09-16T15:51:17.990690+0000","last_peered":"2024-09-16T15:51:17.990690+0000","last_clean":"2024-09-16T15:51:17.990690+0000","last_became_active":"2024-09-16T15:51:15.955201+0000","last_became_peered":"2024-09-16T15:51:15.955201+0000","last_unstale":"2024-09-16T15:51:17.990690+0000","last_undegraded":"2024-09-16T15:51:17.990690+0000","last_fullsized":"2024-09-16T15:51:17.990690+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:51:14.906573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:51:14.906573+0000","last_clean_scrub_stamp":"2024-09-16T15:51:14.906573+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:47:16.682466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"25.17","version":"0'0","reported_seq":13,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-16T15:51:17.992029+0000","last_change":"2024-09-16T15:51:17.021310+0000","last_active":"2024-09-16T15:51:17.992029+0000","last_peered":"2024-09-16T15:51:17.992029+0000","last_clean":"2024-09-16T15:51:17.992029+0000","last_became_active":"2024-09-16T15:51:17.019103+0000","last_became_peered":"2024-09-16T15:51:17.019103+0000","last_unstale":"2024-09-16T15:51:17.992029+0000","last_undegraded":"2024-09-16T15:51:17.992029+0000","last_fullsized":"2024-09-16T15:51:17.992029+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:51:15.917022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:51:15.917022+0000","last_clean_scrub_stamp":"2024-09-16T15:51:15.917022+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:37:51.032687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.16","version":"0'0","reported_seq":15,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-16T15:51:18.229691+0000","last_change":"2024-09-16T15:51:15.962253+0000","last_active":"2024-09-16T15:51:18.229691+0000","last_peered":"2024-09-16T15:51:18.229691+0000","last_clean":"2024-09-16T15:51:18.229691+0000","last_became_active":"2024-09-16T15:51:15.961884+0000","last_became_peered":"2024-09-16T15:51:15.961884+0000","last_unstale":"2024-09-16T15:51:18.229691+0000","last_undegraded":"2024-09-16T15:51:18.229691+0000","last_fullsized":"2024-09-16T15:51:18.229691+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:51:14.906573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:51:14.906573+0000","last_clean_scrub_stamp":"2024-09-16T15:51:14.906573+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T18:17:02.816460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.14","version":"0'0","reported_seq":13,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-16T15:51:17.991842+0000","last_change":"2024-09-16T15:51:17.006585+0000","last_active":"2024-09-16T15:51:17.991842+0000","last_peered":"2024-09-16T15:51:17.991842+0000","last_clean":"2024-09-16T15:51:17.991842+0000","last_became_active":"2024-09-16T15:51:17.006282+0000","last_became_peered":"2024-09-16T15:51:17.006282+0000","last_unstale":"2024-09-16T15:51:17.991842+0000","last_undegraded":"2024-09-16T15:51:17.991842+0000","last_fullsized":"2024-09-16T15:51:17.991842+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:51:15.917022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:51:15.917022+0000","last_clean_scrub_stamp":"2024-09-16T15:51:15.917022+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T00:08:29.575374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.15","version":"156'3","reported_seq":18,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-16T15:51:18.229663+0000","last_change":"2024-09-16T15:51:15.955867+0000","last_active":"2024-09-16T15:51:18.229663+0000","last_peered":"2024-09-16T15:51:18.229663+0000","last_clean":"2024-09-16T15:51:18.229663+0000","last_became_active":"2024-09-16T15:51:15.955585+0000","last_became_peered":"2024-09-16T15:51:15.955585+0000","last_unstale":"2024-09-16T15:51:18.229663+0000","last_undegraded":"2024-09-16T15:51:18.229663+0000","last_fullsized":"2024-09-16T15:51:18.229663+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:51:14.906573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:51:14.906573+0000","last_clean_scrub_stamp":"2024-09-16T15:51:14.906573+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:17:19.428586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.15","version":"0'0","reported_seq":13,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-16T15:51:18.000199+0000","last_change":"2024-09-16T15:51:17.012094+0000","last_active":"2024-09-16T15:51:18.000199+0000","last_peered":"2024-09-16T15:51:18.000199+0000","last_clean":"2024-09-16T15:51:18.000199+0000","last_became_active":"2024-09-16T15:51:17.010977+0000","last_became_peered":"2024-09-16T15:51:17.010977+0000","last_unstale":"2024-09-16T15:51:18.000199+0000","last_undegraded":"2024-09-16T15:51:18.000199+0000","last_fullsized":"2024-09-16T15:51:18.000199+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:51:15.917022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:51:15.917022+0000","last_clean_scrub_stamp":"2024-09-16T15:51:15.917022+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T00:37:58.329330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.14","version":"156'6","reported_seq":21,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-16T15:51:18.006272+0000","last_change":"2024-09-16T15:51:15.944279+0000","last_active":"2024-09-16T15:51:18.006272+0000","last_peered":"2024-09-16T15:51:18.006272+0000","last_clean":"2024-09-16T15:51:18.006272+0000","last_became_active":"2024-09-16T15:51:15.943434+0000","last_became_peered":"2024-09-16T15:51:15.943434+0000","last_unstale":"2024-09-16T15:51:18.006272+0000","last_undegraded":"2024-09-16T15:51:18.006272+0000","last_fullsized":"2024-09-16T15:51:18.006272+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:51:14.906573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:51:14.906573+0000","last_clean_scrub_stamp":"2024-09-16T15:51:14.906573+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T00:17:37.514552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.12","version":"0'0","reported_seq":13,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-16T15:51:18.006244+0000","last_change":"2024-09-16T15:51:17.012749+0000","last_active":"2024-09-16T15:51:18.006244+0000","last_peered":"2024-09-16T15:51:18.006244+0000","last_clean":"2024-09-16T15:51:18.006244+0000","last_became_active":"2024-09-16T15:51:17.012481+0000","last_became_peered":"2024-09-16T15:51:17.012481+0000","last_unstale":"2024-09-16T15:51:18.006244+0000","last_undegraded":"2024-09-16T15:51:18.006244+0000","last_fullsized":"2024-09-16T15:51:18.006244+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:51:15.917022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:51:15.917022+0000","last_clean_scrub_stamp":"2024-09-16T15:51:15.917022+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:53:02.451961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.13","version":"156'2","reported_seq":17,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-16T15:51:18.229766+0000","last_change":"2024-09-16T15:51:15.951395+0000","last_active":"2024-09-16T15:51:18.229766+0000","last_peered":"2024-09-16T15:51:18.229766+0000","last_clean":"2024-09-16T15:51:18.229766+0000","last_became_active":"2024-09-16T15:51:15.950630+0000","last_became_peered":"2024-09-16T15:51:15.950630+0000","last_unstale":"2024-09-16T15:51:18.229766+0000","last_undegraded":"2024-09-16T15:51:18.229766+0000","last_fullsized":"2024-09-16T15:51:18.229766+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:51:14.906573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:51:14.906573+0000","last_clean_scrub_stamp":"2024-09-16T15:51:14.906573+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T23:45:28.575869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.13","version":"0'0","reported_seq":13,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-16T15:51:17.991414+0000","last_change":"2024-09-16T15:51:17.010596+0000","last_active":"2024-09-16T15:51:17.991414+0000","last_peered":"2024-09-16T15:51:17.991414+0000","last_clean":"2024-09-16T15:51:17.991414+0000","last_became_active":"2024-09-16T15:51:17.010306+0000","last_became_peered":"2024-09-16T15:51:17.010306+0000","last_unstale":"2024-09-16T15:51:17.991414+0000","last_undegraded":"2024-09-16T15:51:17.991414+0000","last_fullsized":"2024-09-16T15:51:17.991414+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:51:15.917022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:51:15.917022+0000","last_clean_scrub_stamp":"2024-09-16T15:51:15.917022+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T01:05:48.879633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.12","version":"0'0","reported_seq":15,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-16T15:51:17.999445+0000","last_change":"2024-09-16T15:51:15.949325+0000","last_active":"2024-09-16T15:51:17.999445+0000","last_peered":"2024-09-16T15:51:17.999445+0000","last_clean":"2024-09-16T15:51:17.999445+0000","last_became_active":"2024-09-16T15:51:15.948939+0000","last_became_peered":"2024-09-16T15:51:15.948939+0000","last_unstale":"2024-09-16T15:51:17.999445+0000","last_undegraded":"2024-09-16T15:51:17.999445+0000","last_fullsized":"2024-09-16T15:51:17.999445+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:51:14.906573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:51:14.906573+0000","last_clean_scrub_stamp":"2024-09-16T15:51:14.906573+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:36:11.682195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"25.10","version":"0'0","reported_seq":13,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-16T15:51:18.229257+0000","last_change":"2024-09-16T15:51:17.013579+0000","last_active":"2024-09-16T15:51:18.229257+0000","last_peered":"2024-09-16T15:51:18.229257+0000","last_clean":"2024-09-16T15:51:18.229257+0000","last_became_active":"2024-09-16T15:51:17.013174+0000","last_became_peered":"2024-09-16T15:51:17.013174+0000","last_unstale":"2024-09-16T15:51:18.229257+0000","last_undegraded":"2024-09-16T15:51:18.229257+0000","last_fullsized":"2024-09-16T15:51:18.229257+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:51:15.917022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:51:15.917022+0000","last_clean_scrub_stamp":"2024-09-16T15:51:15.917022+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:46:10.913577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.11","version":"0'0","reported_seq":15,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-16T15:51:17.991342+0000","last_change":"2024-09-16T15:51:15.962934+0000","last_active":"2024-09-16T15:51:17.991342+0000","last_peered":"2024-09-16T15:51:17.991342+0000","last_clean":"2024-09-16T15:51:17.991342+0000","last_became_active":"2024-09-16T15:51:15.962573+0000","last_became_peered":"2024-09-16T15:51:15.962573+0000","last_unstale":"2024-09-16T15:51:17.991342+0000","last_undegraded":"2024-09-16T15:51:17.991342+0000","last_fullsized":"2024-09-16T15:51:17.991342+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:51:14.906573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:51:14.906573+0000","last_clean_scrub_stamp":"2024-09-16T15:51:14.906573+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:20:38.009303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.11","version":"0'0","reported_seq":13,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-16T15:51:17.999314+0000","last_change":"2024-09-16T15:51:17.007443+0000","last_active":"2024-09-16T15:51:17.999314+0000","last_peered":"2024-09-16T15:51:17.999314+0000","last_clean":"2024-09-16T15:51:17.999314+0000","last_became_active":"2024-09-16T15:51:17.007213+0000","last_became_peered":"2024-09-16T15:51:17.007213+0000","last_unstale":"2024-09-16T15:51:17.999314+0000","last_undegraded":"2024-09-16T15:51:17.999314+0000","last_fullsized":"2024-09-16T15:51:17.999314+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:51:15.917022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:51:15.917022+0000","last_clean_scrub_stamp":"2024-09-16T15:51:15.917022+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:26:48.639401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.10","version":"156'8","reported_seq":23,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-16T15:51:18.229151+0000","last_change":"2024-09-16T15:51:15.960961+0000","last_active":"2024-09-16T15:51:18.229151+0000","last_peered":"2024-09-16T15:51:18.229151+0000","last_clean":"2024-09-16T15:51:18.229151+0000","last_became_active":"2024-09-16T15:51:15.960126+0000","last_became_peered":"2024-09-16T15:51:15.960126+0000","last_unstale":"2024-09-16T15:51:18.229151+0000","last_undegraded":"2024-09-16T15:51:18.229151+0000","last_fullsized":"2024-09-16T15:51:18.229151+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:51:14.906573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:51:14.906573+0000","last_clean_scrub_stamp":"2024-09-16T15:51:14.906573+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:39.437445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.e","version":"0'0","reported_seq":13,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-16T15:51:18.001534+0000","last_change":"2024-09-16T15:51:17.013919+0000","last_active":"2024-09-16T15:51:18.001534+0000","last_peered":"2024-09-16T15:51:18.001534+0000","last_clean":"2024-09-16T15:51:18.001534+0000","last_became_active":"2024-09-16T15:51:17.012170+0000","last_became_peered":"2024-09-16T15:51:17.012170+0000","last_unstale":"2024-09-16T15:51:18.001534+0000","last_undegraded":"2024-09-16T15:51:18.001534+0000","last_fullsized":"2024-09-16T15:51:18.001534+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:51:15.917022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:51:15.917022+0000","last_clean_scrub_stamp":"2024-09-16T15:51:15.917022+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:52:37.466721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.f","version":"156'1","reported_seq":16,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-16T15:51:18.230344+0000","last_change":"2024-09-16T15:51:15.961036+0000","last_active":"2024-09-16T15:51:18.230344+0000","last_peered":"2024-09-16T15:51:18.230344+0000","last_clean":"2024-09-16T15:51:18.230344+0000","last_became_active":"2024-09-16T15:51:15.960726+0000","last_became_peered":"2024-09-16T15:51:15.960726+0000","last_unstale":"2024-09-16T15:51:18.230344+0000","last_undegraded":"2024-09-16T15:51:18.230344+0000","last_fullsized":"2024-09-16T15:51:18.230344+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:51:14.906573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:51:14.906573+0000","last_clean_scrub_stamp":"2024-09-16T15:51:14.906573+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T02:06:03.157019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.f","version":"0'0","reported_seq":13,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-16T15:51:18.231827+0000","last_change":"2024-09-16T15:51:17.016626+0000","last_active":"2024-09-16T15:51:18.231827+0000","last_peered":"2024-09-16T15:51:18.231827+0000","last_clean":"2024-09-16T15:51:18.231827+0000","last_became_active":"2024-09-16T15:51:17.014818+0000","last_became_peered":"2024-09-16T15:51:17.014818+0000","last_unstale":"2024-09-16T15:51:18.231827+0000","last_undegraded":"2024-09-16T15:51:18.231827+0000","last_fullsized":"2024-09-16T15:51:18.231827+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:51:15.917022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:51:15.917022+0000","last_clean_scrub_stamp":"2024-09-16T15:51:15.917022+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:18:36.507362+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.e","version":"156'2","reported_seq":17,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-16T15:51:17.992623+0000","last_change":"2024-09-16T15:51:15.953442+0000","last_active":"2024-09-16T15:51:17.992623+0000","last_peered":"2024-09-16T15:51:17.992623+0000","last_clean":"2024-09-16T15:51:17.992623+0000","last_became_active":"2024-09-16T15:51:15.953180+0000","last_became_peered":"2024-09-16T15:51:15.953180+0000","last_unstale":"2024-09-16T15:51:17.992623+0000","last_undegraded":"2024-09-16T15:51:17.992623+0000","last_fullsized":"2024-09-16T15:51:17.992623+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:51:14.906573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:51:14.906573+0000","last_clean_scrub_stamp":"2024-09-16T15:51:14.906573+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:41:37.066613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.c","version":"0'0","reported_seq":13,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-16T15:51:18.231506+0000","last_change":"2024-09-16T15:51:16.996979+0000","last_active":"2024-09-16T15:51:18.231506+0000","last_peered":"2024-09-16T15:51:18.231506+0000","last_clean":"2024-09-16T15:51:18.231506+0000","last_became_active":"2024-09-16T15:51:16.996721+0000","last_became_peered":"2024-09-16T15:51:16.996721+0000","last_unstale":"2024-09-16T15:51:18.231506+0000","last_undegraded":"2024-09-16T15:51:18.231506+0000","last_fullsized":"2024-09-16T15:51:18.231506+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:51:15.917022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:51:15.917022+0000","last_clean_scrub_stamp":"2024-09-16T15:51:15.917022+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:09:32.727980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.d","version":"156'1","reported_seq":16,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-16T15:51:18.230371+0000","last_change":"2024-09-16T15:51:15.957621+0000","last_active":"2024-09-16T15:51:18.230371+0000","last_peered":"2024-09-16T15:51:18.230371+0000","last_clean":"2024-09-16T15:51:18.230371+0000","last_became_active":"2024-09-16T15:51:15.956820+0000","last_became_peered":"2024-09-16T15:51:15.956820+0000","last_unstale":"2024-09-16T15:51:18.230371+0000","last_undegraded":"2024-09-16T15:51:18.230371+0000","last_fullsized":"2024-09-16T15:51:18.230371+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:51:14.906573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:51:14.906573+0000","last_clean_scrub_stamp":"2024-09-16T15:51:14.906573+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T22:53:11.638024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.d","version":"0'0","reported_seq":13,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-16T15:51:17.993166+0000","last_change":"2024-09-16T15:51:17.003393+0000","last_active":"2024-09-16T15:51:17.993166+0000","last_peered":"2024-09-16T15:51:17.993166+0000","last_clean":"2024-09-16T15:51:17.993166+0000","last_became_active":"2024-09-16T15:51:17.002981+0000","last_became_peered":"2024-09-16T15:51:17.002981+0000","last_unstale":"2024-09-16T15:51:17.993166+0000","last_undegraded":"2024-09-16T15:51:17.993166+0000","last_fullsized":"2024-09-16T15:51:17.993166+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:51:15.917022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:51:15.917022+0000","last_clean_scrub_stamp":"2024-09-16T15:51:15.917022+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:03:12.833541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"24.c","version":"0'0","reported_seq":15,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-16T15:51:17.991347+0000","last_change":"2024-09-16T15:51:15.957428+0000","last_active":"2024-09-16T15:51:17.991347+0000","last_peered":"2024-09-16T15:51:17.991347+0000","last_clean":"2024-09-16T15:51:17.991347+0000","last_became_active":"2024-09-16T15:51:15.956667+0000","last_became_peered":"2024-09-16T15:51:15.956667+0000","last_unstale":"2024-09-16T15:51:17.991347+0000","last_undegraded":"2024-09-16T15:51:17.991347+0000","last_fullsized":"2024-09-16T15:51:17.991347+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:51:14.906573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:51:14.906573+0000","last_clean_scrub_stamp":"2024-09-16T15:51:14.906573+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T02:23:47.457845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"25.a","version":"0'0","reported_seq":13,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-16T15:51:17.999752+0000","last_change":"2024-09-16T15:51:17.010847+0000","last_active":"2024-09-16T15:51:17.999752+0000","last_peered":"2024-09-16T15:51:17.999752+0000","last_clean":"2024-09-16T15:51:17.999752+0000","last_became_active":"2024-09-16T15:51:17.010202+0000","last_became_peered":"2024-09-16T15:51:17.010202+0000","last_unstale":"2024-09-16T15:51:17.999752+0000","last_undegraded":"2024-09-16T15:51:17.999752+0000","last_fullsized":"2024-09-16T15:51:17.999752+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:51:15.917022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:51:15.917022+0000","last_clean_scrub_stamp":"2024-09-16T15:51:15.917022+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T22:56:13.989775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.b","version":"156'2","reported_seq":17,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-16T15:51:18.005791+0000","last_change":"2024-09-16T15:51:15.941149+0000","last_active":"2024-09-16T15:51:18.005791+0000","last_peered":"2024-09-16T15:51:18.005791+0000","last_clean":"2024-09-16T15:51:18.005791+0000","last_became_active":"2024-09-16T15:51:15.940519+0000","last_became_peered":"2024-09-16T15:51:15.940519+0000","last_unstale":"2024-09-16T15:51:18.005791+0000","last_undegraded":"2024-09-16T15:51:18.005791+0000","last_fullsized":"2024-09-16T15:51:18.005791+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:51:14.906573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:51:14.906573+0000","last_clean_scrub_stamp":"2024-09-16T15:51:14.906573+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T22:59:39.302588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.b","version":"0'0","reported_seq":13,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-16T15:51:17.991435+0000","last_change":"2024-09-16T15:51:16.999572+0000","last_active":"2024-09-16T15:51:17.991435+0000","last_peered":"2024-09-16T15:51:17.991435+0000","last_clean":"2024-09-16T15:51:17.991435+0000","last_became_active":"2024-09-16T15:51:16.999274+0000","last_became_peered":"2024-09-16T15:51:16.999274+0000","last_unstale":"2024-09-16T15:51:17.991435+0000","last_undegraded":"2024-09-16T15:51:17.991435+0000","last_fullsized":"2024-09-16T15:51:17.991435+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:51:15.917022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:51:15.917022+0000","last_clean_scrub_stamp":"2024-09-16T15:51:15.917022+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T23:27:15.577064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.a","version":"0'0","reported_seq":15,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-16T15:51:18.229244+0000","last_change":"2024-09-16T15:51:15.960290+0000","last_active":"2024-09-16T15:51:18.229244+0000","last_peered":"2024-09-16T15:51:18.229244+0000","last_clean":"2024-09-16T15:51:18.229244+0000","last_became_active":"2024-09-16T15:51:15.956654+0000","last_became_peered":"2024-09-16T15:51:15.956654+0000","last_unstale":"2024-09-16T15:51:18.229244+0000","last_undegraded":"2024-09-16T15:51:18.229244+0000","last_fullsized":"2024-09-16T15:51:18.229244+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:51:14.906573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:51:14.906573+0000","last_clean_scrub_stamp":"2024-09-16T15:51:14.906573+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:04:27.492103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.8","version":"0'0","reported_seq":13,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-16T15:51:17.991965+0000","last_change":"2024-09-16T15:51:17.007037+0000","last_active":"2024-09-16T15:51:17.991965+0000","last_peered":"2024-09-16T15:51:17.991965+0000","last_clean":"2024-09-16T15:51:17.991965+0000","last_became_active":"2024-09-16T15:51:17.006610+0000","last_became_peered":"2024-09-16T15:51:17.006610+0000","last_unstale":"2024-09-16T15:51:17.991965+0000","last_undegraded":"2024-09-16T15:51:17.991965+0000","last_fullsized":"2024-09-16T15:51:17.991965+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:51:15.917022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:51:15.917022+0000","last_clean_scrub_stamp":"2024-09-16T15:51:15.917022+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:12:40.001066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"24.9","version":"0'0","reported_seq":15,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-16T15:51:18.006103+0000","last_change":"2024-09-16T15:51:15.940838+0000","last_active":"2024-09-16T15:51:18.006103+0000","last_peered":"2024-09-16T15:51:18.006103+0000","last_clean":"2024-09-16T15:51:18.006103+0000","last_became_active":"2024-09-16T15:51:15.940426+0000","last_became_peered":"2024-09-16T15:51:15.940426+0000","last_unstale":"2024-09-16T15:51:18.006103+0000","last_undegraded":"2024-09-16T15:51:18.006103+0000","last_fullsized":"2024-09-16T15:51:18.006103+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:51:14.906573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:51:14.906573+0000","last_clean_scrub_stamp":"2024-09-16T15:51:14.906573+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T18:15:27.317220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.9","version":"0'0","reported_seq":13,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-16T15:51:18.229429+0000","last_change":"2024-09-16T15:51:16.993404+0000","last_active":"2024-09-16T15:51:18.229429+0000","last_peered":"2024-09-16T15:51:18.229429+0000","last_clean":"2024-09-16T15:51:18.229429+0000","last_became_active":"2024-09-16T15:51:16.992924+0000","last_became_peered":"2024-09-16T15:51:16.992924+0000","last_unstale":"2024-09-16T15:51:18.229429+0000","last_undegraded":"2024-09-16T15:51:18.229429+0000","last_fullsized":"2024-09-16T15:51:18.229429+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:51:15.917022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:51:15.917022+0000","last_clean_scrub_stamp":"2024-09-16T15:51:15.917022+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:43:58.727551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"24.8","version":"0'0","reported_seq":15,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-16T15:51:18.229067+0000","last_change":"2024-09-16T15:51:15.955543+0000","last_active":"2024-09-16T15:51:18.229067+0000","last_peered":"2024-09-16T15:51:18.229067+0000","last_clean":"2024-09-16T15:51:18.229067+0000","last_became_active":"2024-09-16T15:51:15.954545+0000","last_became_peered":"2024-09-16T15:51:15.954545+0000","last_unstale":"2024-09-16T15:51:18.229067+0000","last_undegraded":"2024-09-16T15:51:18.229067+0000","last_fullsized":"2024-09-16T15:51:18.229067+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:51:14.906573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:51:14.906573+0000","last_clean_scrub_stamp":"2024-09-16T15:51:14.906573+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T02:21:25.208264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"25.6","version":"0'0","reported_seq":13,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-16T15:51:18.230000+0000","last_change":"2024-09-16T15:51:16.992407+0000","last_active":"2024-09-16T15:51:18.230000+0000","last_peered":"2024-09-16T15:51:18.230000+0000","last_clean":"2024-09-16T15:51:18.230000+0000","last_became_active":"2024-09-16T15:51:16.991864+0000","last_became_peered":"2024-09-16T15:51:16.991864+0000","last_unstale":"2024-09-16T15:51:18.230000+0000","last_undegraded":"2024-09-16T15:51:18.230000+0000","last_fullsized":"2024-09-16T15:51:18.230000+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:51:15.917022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:51:15.917022+0000","last_clean_scrub_stamp":"2024-09-16T15:51:15.917022+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T01:09:46.564464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"24.7","version":"156'2","reported_seq":17,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-16T15:51:18.230451+0000","last_change":"2024-09-16T15:51:15.950341+0000","last_active":"2024-09-16T15:51:18.230451+0000","last_peered":"2024-09-16T15:51:18.230451+0000","last_clean":"2024-09-16T15:51:18.230451+0000","last_became_active":"2024-09-16T15:51:15.949020+0000","last_became_peered":"2024-09-16T15:51:15.949020+0000","last_unstale":"2024-09-16T15:51:18.230451+0000","last_undegraded":"2024-09-16T15:51:18.230451+0000","last_fullsized":"2024-09-16T15:51:18.230451+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:51:14.906573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:51:14.906573+0000","last_clean_scrub_stamp":"2024-09-16T15:51:14.906573+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:27:27.203647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.7","version":"0'0","reported_seq":13,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-16T15:51:18.000972+0000","last_change":"2024-09-16T15:51:17.011690+0000","last_active":"2024-09-16T15:51:18.000972+0000","last_peered":"2024-09-16T15:51:18.000972+0000","last_clean":"2024-09-16T15:51:18.000972+0000","last_became_active":"2024-09-16T15:51:17.010584+0000","last_became_peered":"2024-09-16T15:51:17.010584+0000","last_unstale":"2024-09-16T15:51:18.000972+0000","last_undegraded":"2024-09-16T15:51:18.000972+0000","last_fullsized":"2024-09-16T15:51:18.000972+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:51:15.917022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:51:15.917022+0000","last_clean_scrub_stamp":"2024-09-16T15:51:15.917022+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T01:59:38.918542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.6","version":"157'1","reported_seq":16,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-16T15:51:18.001778+0000","last_change":"2024-09-16T15:51:15.952456+0000","last_active":"2024-09-16T15:51:18.001778+0000","last_peered":"2024-09-16T15:51:18.001778+0000","last_clean":"2024-09-16T15:51:18.001778+0000","last_became_active":"2024-09-16T15:51:15.950670+0000","last_became_peered":"2024-09-16T15:51:15.950670+0000","last_unstale":"2024-09-16T15:51:18.001778+0000","last_undegraded":"2024-09-16T15:51:18.001778+0000","last_fullsized":"2024-09-16T15:51:18.001778+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:51:14.906573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:51:14.906573+0000","last_clean_scrub_stamp":"2024-09-16T15:51:14.906573+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:31:33.177323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"25.4","version":"0'0","reported_seq":13,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-16T15:51:17.993228+0000","last_change":"2024-09-16T15:51:16.999730+0000","last_active":"2024-09-16T15:51:17.993228+0000","last_peered":"2024-09-16T15:51:17.993228+0000","last_clean":"2024-09-16T15:51:17.993228+0000","last_became_active":"2024-09-16T15:51:16.998396+0000","last_became_peered":"2024-09-16T15:51:16.998396+0000","last_unstale":"2024-09-16T15:51:17.993228+0000","last_undegraded":"2024-09-16T15:51:17.993228+0000","last_fullsized":"2024-09-16T15:51:17.993228+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:51:15.917022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:51:15.917022+0000","last_clean_scrub_stamp":"2024-09-16T15:51:15.917022+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T18:13:12.440858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.5","version":"0'0","reported_seq":15,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-16T15:51:18.000330+0000","last_change":"2024-09-16T15:51:15.953202+0000","last_active":"2024-09-16T15:51:18.000330+0000","last_peered":"2024-09-16T15:51:18.000330+0000","last_clean":"2024-09-16T15:51:18.000330+0000","last_became_active":"2024-09-16T15:51:15.952826+0000","last_became_peered":"2024-09-16T15:51:15.952826+0000","last_unstale":"2024-09-16T15:51:18.000330+0000","last_undegraded":"2024-09-16T15:51:18.000330+0000","last_fullsized":"2024-09-16T15:51:18.000330+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:51:14.906573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:51:14.906573+0000","last_clean_scrub_stamp":"2024-09-16T15:51:14.906573+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T03:12:29.684040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"25.5","version":"0'0","reported_seq":13,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-16T15:51:17.992473+0000","last_change":"2024-09-16T15:51:17.004148+0000","last_active":"2024-09-16T15:51:17.992473+0000","last_peered":"2024-09-16T15:51:17.992473+0000","last_clean":"2024-09-16T15:51:17.992473+0000","last_became_active":"2024-09-16T15:51:17.003598+0000","last_became_peered":"2024-09-16T15:51:17.003598+0000","last_unstale":"2024-09-16T15:51:17.992473+0000","last_undegraded":"2024-09-16T15:51:17.992473+0000","last_fullsized":"2024-09-16T15:51:17.992473+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:51:15.917022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:51:15.917022+0000","last_clean_scrub_stamp":"2024-09-16T15:51:15.917022+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:12:52.408625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.4","version":"0'0","reported_seq":15,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-16T15:51:17.993301+0000","last_change":"2024-09-16T15:51:15.947156+0000","last_active":"2024-09-16T15:51:17.993301+0000","last_peered":"2024-09-16T15:51:17.993301+0000","last_clean":"2024-09-16T15:51:17.993301+0000","last_became_active":"2024-09-16T15:51:15.946443+0000","last_became_peered":"2024-09-16T15:51:15.946443+0000","last_unstale":"2024-09-16T15:51:17.993301+0000","last_undegraded":"2024-09-16T15:51:17.993301+0000","last_fullsized":"2024-09-16T15:51:17.993301+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:51:14.906573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:51:14.906573+0000","last_clean_scrub_stamp":"2024-09-16T15:51:14.906573+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T00:44:23.785120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.2","version":"0'0","reported_seq":13,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-16T15:51:18.230924+0000","last_change":"2024-09-16T15:51:17.003815+0000","last_active":"2024-09-16T15:51:18.230924+0000","last_peered":"2024-09-16T15:51:18.230924+0000","last_clean":"2024-09-16T15:51:18.230924+0000","last_became_active":"2024-09-16T15:51:17.003331+0000","last_became_peered":"2024-09-16T15:51:17.003331+0000","last_unstale":"2024-09-16T15:51:18.230924+0000","last_undegraded":"2024-09-16T15:51:18.230924+0000","last_fullsized":"2024-09-16T15:51:18.230924+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:51:15.917022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:51:15.917022+0000","last_clean_scrub_stamp":"2024-09-16T15:51:15.917022+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T00:59:28.722029+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"24.3","version":"0'0","reported_seq":15,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-16T15:51:18.231360+0000","last_change":"2024-09-16T15:51:15.962250+0000","last_active":"2024-09-16T15:51:18.231360+0000","last_peered":"2024-09-16T15:51:18.231360+0000","last_clean":"2024-09-16T15:51:18.231360+0000","last_became_active":"2024-09-16T15:51:15.961426+0000","last_became_peered":"2024-09-16T15:51:15.961426+0000","last_unstale":"2024-09-16T15:51:18.231360+0000","last_undegraded":"2024-09-16T15:51:18.231360+0000","last_fullsized":"2024-09-16T15:51:18.231360+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:51:14.906573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:51:14.906573+0000","last_clean_scrub_stamp":"2024-09-16T15:51:14.906573+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:29:50.442147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"25.3","version":"0'0","reported_seq":13,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-16T15:51:18.230136+0000","last_change":"2024-09-16T15:51:17.000638+0000","last_active":"2024-09-16T15:51:18.230136+0000","last_peered":"2024-09-16T15:51:18.230136+0000","last_clean":"2024-09-16T15:51:18.230136+0000","last_became_active":"2024-09-16T15:51:16.999924+0000","last_became_peered":"2024-09-16T15:51:16.999924+0000","last_unstale":"2024-09-16T15:51:18.230136+0000","last_undegraded":"2024-09-16T15:51:18.230136+0000","last_fullsized":"2024-09-16T15:51:18.230136+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:51:15.917022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:51:15.917022+0000","last_clean_scrub_stamp":"2024-09-16T15:51:15.917022+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:11:45.415023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"24.2","version":"0'0","reported_seq":15,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-16T15:51:18.230610+0000","last_change":"2024-09-16T15:51:15.953228+0000","last_active":"2024-09-16T15:51:18.230610+0000","last_peered":"2024-09-16T15:51:18.230610+0000","last_clean":"2024-09-16T15:51:18.230610+0000","last_became_active":"2024-09-16T15:51:15.952533+0000","last_became_peered":"2024-09-16T15:51:15.952533+0000","last_unstale":"2024-09-16T15:51:18.230610+0000","last_undegraded":"2024-09-16T15:51:18.230610+0000","last_fullsized":"2024-09-16T15:51:18.230610+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:51:14.906573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:51:14.906573+0000","last_clean_scrub_stamp":"2024-09-16T15:51:14.906573+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:43:57.717101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"25.0","version":"0'0","reported_seq":13,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-16T15:51:17.992082+0000","last_change":"2024-09-16T15:51:16.977593+0000","last_active":"2024-09-16T15:51:17.992082+0000","last_peered":"2024-09-16T15:51:17.992082+0000","last_clean":"2024-09-16T15:51:17.992082+0000","last_became_active":"2024-09-16T15:51:16.976791+0000","last_became_peered":"2024-09-16T15:51:16.976791+0000","last_unstale":"2024-09-16T15:51:17.992082+0000","last_undegraded":"2024-09-16T15:51:17.992082+0000","last_fullsized":"2024-09-16T15:51:17.992082+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:51:15.917022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:51:15.917022+0000","last_clean_scrub_stamp":"2024-09-16T15:51:15.917022+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T02:45:37.431644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"24.1","version":"0'0","reported_seq":15,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-16T15:51:17.992304+0000","last_change":"2024-09-16T15:51:15.965947+0000","last_active":"2024-09-16T15:51:17.992304+0000","last_peered":"2024-09-16T15:51:17.992304+0000","last_clean":"2024-09-16T15:51:17.992304+0000","last_became_active":"2024-09-16T15:51:15.965027+0000","last_became_peered":"2024-09-16T15:51:15.965027+0000","last_unstale":"2024-09-16T15:51:17.992304+0000","last_undegraded":"2024-09-16T15:51:17.992304+0000","last_fullsized":"2024-09-16T15:51:17.992304+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:51:14.906573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:51:14.906573+0000","last_clean_scrub_stamp":"2024-09-16T15:51:14.906573+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T18:06:03.179201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"25.1","version":"0'0","reported_seq":13,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-16T15:51:18.229551+0000","last_change":"2024-09-16T15:51:17.000624+0000","last_active":"2024-09-16T15:51:18.229551+0000","last_peered":"2024-09-16T15:51:18.229551+0000","last_clean":"2024-09-16T15:51:18.229551+0000","last_became_active":"2024-09-16T15:51:17.000331+0000","last_became_peered":"2024-09-16T15:51:17.000331+0000","last_unstale":"2024-09-16T15:51:18.229551+0000","last_undegraded":"2024-09-16T15:51:18.229551+0000","last_fullsized":"2024-09-16T15:51:18.229551+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:51:15.917022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:51:15.917022+0000","last_clean_scrub_stamp":"2024-09-16T15:51:15.917022+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:46:14.133192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"24.0","version":"0'0","reported_seq":15,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-16T15:51:18.005669+0000","last_change":"2024-09-16T15:51:15.940537+0000","last_active":"2024-09-16T15:51:18.005669+0000","last_peered":"2024-09-16T15:51:18.005669+0000","last_clean":"2024-09-16T15:51:18.005669+0000","last_became_active":"2024-09-16T15:51:15.940171+0000","last_became_peered":"2024-09-16T15:51:15.940171+0000","last_unstale":"2024-09-16T15:51:18.005669+0000","last_undegraded":"2024-09-16T15:51:18.005669+0000","last_fullsized":"2024-09-16T15:51:18.005669+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:51:14.906573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:51:14.906573+0000","last_clean_scrub_stamp":"2024-09-16T15:51:14.906573+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:32:44.137745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.18","version":"0'0","reported_seq":13,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-16T15:51:17.999100+0000","last_change":"2024-09-16T15:51:16.997446+0000","last_active":"2024-09-16T15:51:17.999100+0000","last_peered":"2024-09-16T15:51:17.999100+0000","last_clean":"2024-09-16T15:51:17.999100+0000","last_became_active":"2024-09-16T15:51:16.997007+0000","last_became_peered":"2024-09-16T15:51:16.997007+0000","last_unstale":"2024-09-16T15:51:17.999100+0000","last_undegraded":"2024-09-16T15:51:17.999100+0000","last_fullsized":"2024-09-16T15:51:17.999100+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:51:15.917022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:51:15.917022+0000","last_clean_scrub_stamp":"2024-09-16T15:51:15.917022+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:46:27.292030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.19","version":"0'0","reported_seq":15,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-16T15:51:18.229525+0000","last_change":"2024-09-16T15:51:15.961809+0000","last_active":"2024-09-16T15:51:18.229525+0000","last_peered":"2024-09-16T15:51:18.229525+0000","last_clean":"2024-09-16T15:51:18.229525+0000","last_became_active":"2024-09-16T15:51:15.961107+0000","last_became_peered":"2024-09-16T15:51:15.961107+0000","last_unstale":"2024-09-16T15:51:18.229525+0000","last_undegraded":"2024-09-16T15:51:18.229525+0000","last_fullsized":"2024-09-16T15:51:18.229525+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:51:14.906573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:51:14.906573+0000","last_clean_scrub_stamp":"2024-09-16T15:51:14.906573+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T22:10:43.042714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"73'79","reported_seq":391,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-16T15:51:18.229429+0000","last_change":"2024-09-16T15:42:40.881820+0000","last_active":"2024-09-16T15:51:18.229429+0000","last_peered":"2024-09-16T15:51:18.229429+0000","last_clean":"2024-09-16T15:51:18.229429+0000","last_became_active":"2024-09-16T15:42:40.879965+0000","last_became_peered":"2024-09-16T15:42:40.879965+0000","last_unstale":"2024-09-16T15:51:18.229429+0000","last_undegraded":"2024-09-16T15:51:18.229429+0000","last_fullsized":"2024-09-16T15:51:18.229429+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:42:39.728762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:42:39.728762+0000","last_clean_scrub_stamp":"2024-09-16T15:42:39.728762+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:22:18.768932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":25,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":24,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79,"ondisk_log_size":79,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542252,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33708,"kb_used_data":2516,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338132,"statfs":{"total":96636764160,"available":96602247168,"internally_reserved":0,"allocated":2576384,"data_stored":1465866,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542251,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33120,"kb_used_data":1928,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338720,"statfs":{"total":96636764160,"available":96602849280,"internally_reserved":0,"allocated":1974272,"data_stored":875394,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":5,"up_from":14,"seq":60129542252,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33116,"kb_used_data":1928,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338724,"statfs":{"total":96636764160,"available":96602853376,"internally_reserved":0,"allocated":1974272,"data_stored":875394,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542250,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33136,"kb_used_data":1944,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338704,"statfs":{"total":96636764160,"available":96602832896,"internally_reserved":0,"allocated":1990656,"data_stored":877588,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":3,"up_from":14,"seq":60129542251,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37288,"kb_used_data":1940,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334552,"statfs":{"total":96636764160,"available":96598581248,"internally_reserved":0,"allocated":1986560,"data_stored":875588,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":2,"up_from":14,"seq":60129542251,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33128,"kb_used_data":1940,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338712,"statfs":{"total":96636764160,"available":96602841088,"internally_reserved":0,"allocated":1986560,"data_stored":876088,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":14,"seq":60129542253,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33128,"kb_used_data":1940,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338712,"statfs":{"total":96636764160,"available":96602841088,"internally_reserved":0,"allocated":1986560,"data_stored":877006,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":14,"seq":60129542253,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33708,"kb_used_data":2512,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338132,"statfs":{"total":96636764160,"available":96602247168,"internally_reserved":0,"allocated":2572288,"data_stored":1465784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":194,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2194,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T15:51:21.010 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T15:51:21.171 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:21.174+0000 7f74f0f26640 1 -- 172.21.15.123:0/961509001 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f74ec074600 msgr2=0x7f74ec06e750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:51:21.172 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:21.174+0000 7f74f0f26640 1 --2- 172.21.15.123:0/961509001 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f74ec074600 0x7f74ec06e750 secure :-1 s=READY pgs=1748 cs=0 l=1 rev1=1 crypto rx=0x7f74e4009c80 tx=0x7f74e401cce0 comp rx=0 tx=0).stop 2024-09-16T15:51:21.172 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:21.174+0000 7f74f0f26640 1 -- 172.21.15.123:0/961509001 shutdown_connections 2024-09-16T15:51:21.172 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:21.174+0000 7f74f0f26640 1 --2- 172.21.15.123:0/961509001 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f74ec074600 0x7f74ec06e750 unknown :-1 s=CLOSED pgs=1748 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:21.172 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:21.174+0000 7f74f0f26640 1 --2- 172.21.15.123:0/961509001 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f74ec15bca0 0x7f74ec0d0630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:21.172 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:21.174+0000 7f74f0f26640 1 -- 172.21.15.123:0/961509001 >> 172.21.15.123:0/961509001 conn(0x7f74ec0cfb10 msgr2=0x7f74ec0cff20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:51:21.172 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:21.174+0000 7f74f0f26640 1 -- 172.21.15.123:0/961509001 shutdown_connections 2024-09-16T15:51:21.172 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:21.174+0000 7f74f0f26640 1 -- 172.21.15.123:0/961509001 wait complete. 2024-09-16T15:51:21.172 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:21.174+0000 7f74f0f26640 1 Processor -- start 2024-09-16T15:51:21.173 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:21.174+0000 7f74f0f26640 1 -- start start 2024-09-16T15:51:21.174 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:21.174+0000 7f74f0f26640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f74ec0726c0 0x7f74ec15e910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:51:21.175 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:21.174+0000 7f74f0f26640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f74ec074600 0x7f74ec0c83e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:51:21.175 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:21.174+0000 7f74f0f26640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f74ec15bca0 0x7f74ec0c9160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:51:21.175 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:21.174+0000 7f74f0f26640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f74ec168a50 con 0x7f74ec074600 2024-09-16T15:51:21.175 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:21.174+0000 7f74f0f26640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f74ec06fa10 con 0x7f74ec0726c0 2024-09-16T15:51:21.175 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:21.174+0000 7f74f0f26640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f74ec168bd0 con 0x7f74ec15bca0 2024-09-16T15:51:21.175 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:21.174+0000 7f74e9d9b640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f74ec074600 0x7f74ec0c83e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:51:21.175 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:21.174+0000 7f74e9d9b640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f74ec074600 0x7f74ec0c83e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:45182/0 (socket says 172.21.15.123:45182) 2024-09-16T15:51:21.175 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:21.174+0000 7f74e9d9b640 1 -- 172.21.15.123:0/28134687 learned_addr learned my addr 172.21.15.123:0/28134687 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:51:21.175 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:21.174+0000 7f74ea59c640 1 --2- 172.21.15.123:0/28134687 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f74ec0726c0 0x7f74ec15e910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:51:21.175 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:21.174+0000 7f74ead9d640 1 --2- 172.21.15.123:0/28134687 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f74ec15bca0 0x7f74ec0c9160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:51:21.175 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:21.174+0000 7f74e9d9b640 1 -- 172.21.15.123:0/28134687 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f74ec15bca0 msgr2=0x7f74ec0c9160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:51:21.176 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:21.174+0000 7f74e9d9b640 1 --2- 172.21.15.123:0/28134687 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f74ec15bca0 0x7f74ec0c9160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:21.176 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:21.174+0000 7f74e9d9b640 1 -- 172.21.15.123:0/28134687 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f74ec0726c0 msgr2=0x7f74ec15e910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:51:21.176 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:21.174+0000 7f74e9d9b640 1 --2- 172.21.15.123:0/28134687 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f74ec0726c0 0x7f74ec15e910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:21.176 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:21.174+0000 7f74e9d9b640 1 -- 172.21.15.123:0/28134687 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f74ec1f67a0 con 0x7f74ec074600 2024-09-16T15:51:21.176 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:21.174+0000 7f74ea59c640 1 --2- 172.21.15.123:0/28134687 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f74ec0726c0 0x7f74ec15e910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T15:51:21.176 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:21.174+0000 7f74e9d9b640 1 --2- 172.21.15.123:0/28134687 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f74ec074600 0x7f74ec0c83e0 secure :-1 s=READY pgs=1749 cs=0 l=1 rev1=1 crypto rx=0x7f74dc00c450 tx=0x7f74dc00c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:51:21.176 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:21.174+0000 7f74daffd640 1 -- 172.21.15.123:0/28134687 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74dc012830 con 0x7f74ec074600 2024-09-16T15:51:21.176 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:21.174+0000 7f74daffd640 1 -- 172.21.15.123:0/28134687 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f74dc017070 con 0x7f74ec074600 2024-09-16T15:51:21.176 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:21.174+0000 7f74daffd640 1 -- 172.21.15.123:0/28134687 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74dc012d80 con 0x7f74ec074600 2024-09-16T15:51:21.176 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:21.174+0000 7f74f0f26640 1 -- 172.21.15.123:0/28134687 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f74ec15db50 con 0x7f74ec074600 2024-09-16T15:51:21.176 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:21.174+0000 7f74f0f26640 1 -- 172.21.15.123:0/28134687 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f74ec15e110 con 0x7f74ec074600 2024-09-16T15:51:21.176 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:21.174+0000 7f74f0f26640 1 -- 172.21.15.123:0/28134687 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f74ec06e750 con 0x7f74ec074600 2024-09-16T15:51:21.177 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:21.178+0000 7f74daffd640 1 -- 172.21.15.123:0/28134687 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 24) ==== 97845+0+0 (secure 0 0 0) 0x7f74dc01c460 con 0x7f74ec074600 2024-09-16T15:51:21.177 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:21.178+0000 7f74daffd640 1 --2- 172.21.15.123:0/28134687 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f74cc073730 0x7f74cc075bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:51:21.177 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:21.178+0000 7f74daffd640 1 -- 172.21.15.123:0/28134687 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(157..157 src has 1..157) ==== 5737+0+0 (secure 0 0 0) 0x7f74dc092420 con 0x7f74ec074600 2024-09-16T15:51:21.177 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:21.178+0000 7f74ea59c640 1 --2- 172.21.15.123:0/28134687 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f74cc073730 0x7f74cc075bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:51:21.177 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:21.178+0000 7f74ea59c640 1 --2- 172.21.15.123:0/28134687 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f74cc073730 0x7f74cc075bf0 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f74e000a870 tx=0x7f74e0008040 comp rx=0 tx=0).ready entity=mgr.4150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:51:21.180 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:21.182+0000 7f74daffd640 1 -- 172.21.15.123:0/28134687 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f74dc05f120 con 0x7f74ec074600 2024-09-16T15:51:21.346 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:21.346+0000 7f74f0f26640 1 -- 172.21.15.123:0/28134687 --> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f74ec0700e0 con 0x7f74cc073730 2024-09-16T15:51:21.357 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:21.358+0000 7f74daffd640 1 -- 172.21.15.123:0/28134687 <== mgr.4150 v2:172.21.15.123:6836/2464759676 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+252480 (secure 0 0 0) 0x7f74ec0700e0 con 0x7f74cc073730 2024-09-16T15:51:21.359 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-09-16T15:51:21.361 INFO:teuthology.orchestra.run.smithi123.stderr:dumped all 2024-09-16T15:51:21.362 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:21.362+0000 7f74f0f26640 1 -- 172.21.15.123:0/28134687 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f74cc073730 msgr2=0x7f74cc075bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:51:21.362 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:21.362+0000 7f74f0f26640 1 --2- 172.21.15.123:0/28134687 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f74cc073730 0x7f74cc075bf0 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f74e000a870 tx=0x7f74e0008040 comp rx=0 tx=0).stop 2024-09-16T15:51:21.362 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:21.362+0000 7f74f0f26640 1 -- 172.21.15.123:0/28134687 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f74ec074600 msgr2=0x7f74ec0c83e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:51:21.362 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:21.362+0000 7f74f0f26640 1 --2- 172.21.15.123:0/28134687 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f74ec074600 0x7f74ec0c83e0 secure :-1 s=READY pgs=1749 cs=0 l=1 rev1=1 crypto rx=0x7f74dc00c450 tx=0x7f74dc00c910 comp rx=0 tx=0).stop 2024-09-16T15:51:21.362 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:21.362+0000 7f74f0f26640 1 -- 172.21.15.123:0/28134687 shutdown_connections 2024-09-16T15:51:21.362 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:21.362+0000 7f74f0f26640 1 --2- 172.21.15.123:0/28134687 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f74cc073730 0x7f74cc075bf0 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:21.362 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:21.362+0000 7f74f0f26640 1 --2- 172.21.15.123:0/28134687 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f74ec15bca0 0x7f74ec0c9160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:21.362 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:21.362+0000 7f74f0f26640 1 --2- 172.21.15.123:0/28134687 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f74ec074600 0x7f74ec0c83e0 unknown :-1 s=CLOSED pgs=1749 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:21.362 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:21.362+0000 7f74f0f26640 1 --2- 172.21.15.123:0/28134687 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f74ec0726c0 0x7f74ec15e910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:21.362 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:21.362+0000 7f74f0f26640 1 -- 172.21.15.123:0/28134687 >> 172.21.15.123:0/28134687 conn(0x7f74ec0cfb10 msgr2=0x7f74ec0b29a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:51:21.363 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:21.362+0000 7f74f0f26640 1 -- 172.21.15.123:0/28134687 shutdown_connections 2024-09-16T15:51:21.363 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:21.362+0000 7f74f0f26640 1 -- 172.21.15.123:0/28134687 wait complete. 2024-09-16T15:51:21.377 INFO:teuthology.orchestra.run.smithi123.stdout:{"pg_ready":true,"pg_map":{"version":201,"stamp":"2024-09-16T15:51:20.134238+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":158,"num_write_kb":1642,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":118,"ondisk_log_size":118,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":188,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":270332,"kb_used_data":16648,"kb_used_omap":12,"kb_used_meta":253491,"kb_avail":754704388,"statfs":{"total":773094113280,"available":772817293312,"internally_reserved":0,"allocated":17047552,"data_stored":8188708,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":259575345},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":37,"apply_latency_ms":37,"commit_latency_ns":37000000,"apply_latency_ns":37000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.214851"},"pg_stats":[{"pgid":"25.3f","version":"0'0","reported_seq":13,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-16T15:51:18.230610+0000","last_change":"2024-09-16T15:51:17.014998+0000","last_active":"2024-09-16T15:51:18.230610+0000","last_peered":"2024-09-16T15:51:18.230610+0000","last_clean":"2024-09-16T15:51:18.230610+0000","last_became_active":"2024-09-16T15:51:17.014631+0000","last_became_peered":"2024-09-16T15:51:17.014631+0000","last_unstale":"2024-09-16T15:51:18.230610+0000","last_undegraded":"2024-09-16T15:51:18.230610+0000","last_fullsized":"2024-09-16T15:51:18.230610+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:51:15.917022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:51:15.917022+0000","last_clean_scrub_stamp":"2024-09-16T15:51:15.917022+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T03:01:32.062858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.3e","version":"0'0","reported_seq":13,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-16T15:51:17.993676+0000","last_change":"2024-09-16T15:51:17.000581+0000","last_active":"2024-09-16T15:51:17.993676+0000","last_peered":"2024-09-16T15:51:17.993676+0000","last_clean":"2024-09-16T15:51:17.993676+0000","last_became_active":"2024-09-16T15:51:17.000052+0000","last_became_peered":"2024-09-16T15:51:17.000052+0000","last_unstale":"2024-09-16T15:51:17.993676+0000","last_undegraded":"2024-09-16T15:51:17.993676+0000","last_fullsized":"2024-09-16T15:51:17.993676+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:51:15.917022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:51:15.917022+0000","last_clean_scrub_stamp":"2024-09-16T15:51:15.917022+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T01:53:19.860671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"25.3d","version":"0'0","reported_seq":13,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-16T15:51:18.006233+0000","last_change":"2024-09-16T15:51:16.990253+0000","last_active":"2024-09-16T15:51:18.006233+0000","last_peered":"2024-09-16T15:51:18.006233+0000","last_clean":"2024-09-16T15:51:18.006233+0000","last_became_active":"2024-09-16T15:51:16.989274+0000","last_became_peered":"2024-09-16T15:51:16.989274+0000","last_unstale":"2024-09-16T15:51:18.006233+0000","last_undegraded":"2024-09-16T15:51:18.006233+0000","last_fullsized":"2024-09-16T15:51:18.006233+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:51:15.917022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:51:15.917022+0000","last_clean_scrub_stamp":"2024-09-16T15:51:15.917022+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:19:12.127860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.3c","version":"0'0","reported_seq":13,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-16T15:51:17.993067+0000","last_change":"2024-09-16T15:51:17.003322+0000","last_active":"2024-09-16T15:51:17.993067+0000","last_peered":"2024-09-16T15:51:17.993067+0000","last_clean":"2024-09-16T15:51:17.993067+0000","last_became_active":"2024-09-16T15:51:17.003015+0000","last_became_peered":"2024-09-16T15:51:17.003015+0000","last_unstale":"2024-09-16T15:51:17.993067+0000","last_undegraded":"2024-09-16T15:51:17.993067+0000","last_fullsized":"2024-09-16T15:51:17.993067+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:51:15.917022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:51:15.917022+0000","last_clean_scrub_stamp":"2024-09-16T15:51:15.917022+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T03:35:56.618749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"25.3b","version":"0'0","reported_seq":13,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-16T15:51:18.229987+0000","last_change":"2024-09-16T15:51:17.011755+0000","last_active":"2024-09-16T15:51:18.229987+0000","last_peered":"2024-09-16T15:51:18.229987+0000","last_clean":"2024-09-16T15:51:18.229987+0000","last_became_active":"2024-09-16T15:51:17.008469+0000","last_became_peered":"2024-09-16T15:51:17.008469+0000","last_unstale":"2024-09-16T15:51:18.229987+0000","last_undegraded":"2024-09-16T15:51:18.229987+0000","last_fullsized":"2024-09-16T15:51:18.229987+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:51:15.917022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:51:15.917022+0000","last_clean_scrub_stamp":"2024-09-16T15:51:15.917022+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T00:10:10.865390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.3a","version":"0'0","reported_seq":13,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-16T15:51:18.006800+0000","last_change":"2024-09-16T15:51:16.998228+0000","last_active":"2024-09-16T15:51:18.006800+0000","last_peered":"2024-09-16T15:51:18.006800+0000","last_clean":"2024-09-16T15:51:18.006800+0000","last_became_active":"2024-09-16T15:51:16.997779+0000","last_became_peered":"2024-09-16T15:51:16.997779+0000","last_unstale":"2024-09-16T15:51:18.006800+0000","last_undegraded":"2024-09-16T15:51:18.006800+0000","last_fullsized":"2024-09-16T15:51:18.006800+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:51:15.917022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:51:15.917022+0000","last_clean_scrub_stamp":"2024-09-16T15:51:15.917022+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T23:56:53.937968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.39","version":"0'0","reported_seq":13,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-16T15:51:17.991210+0000","last_change":"2024-09-16T15:51:17.014388+0000","last_active":"2024-09-16T15:51:17.991210+0000","last_peered":"2024-09-16T15:51:17.991210+0000","last_clean":"2024-09-16T15:51:17.991210+0000","last_became_active":"2024-09-16T15:51:17.013962+0000","last_became_peered":"2024-09-16T15:51:17.013962+0000","last_unstale":"2024-09-16T15:51:17.991210+0000","last_undegraded":"2024-09-16T15:51:17.991210+0000","last_fullsized":"2024-09-16T15:51:17.991210+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:51:15.917022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:51:15.917022+0000","last_clean_scrub_stamp":"2024-09-16T15:51:15.917022+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T00:44:24.795570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"25.38","version":"0'0","reported_seq":13,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-16T15:51:17.991758+0000","last_change":"2024-09-16T15:51:17.002870+0000","last_active":"2024-09-16T15:51:17.991758+0000","last_peered":"2024-09-16T15:51:17.991758+0000","last_clean":"2024-09-16T15:51:17.991758+0000","last_became_active":"2024-09-16T15:51:17.002088+0000","last_became_peered":"2024-09-16T15:51:17.002088+0000","last_unstale":"2024-09-16T15:51:17.991758+0000","last_undegraded":"2024-09-16T15:51:17.991758+0000","last_fullsized":"2024-09-16T15:51:17.991758+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:51:15.917022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:51:15.917022+0000","last_clean_scrub_stamp":"2024-09-16T15:51:15.917022+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T03:14:13.376009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.37","version":"0'0","reported_seq":13,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-16T15:51:18.231842+0000","last_change":"2024-09-16T15:51:17.012914+0000","last_active":"2024-09-16T15:51:18.231842+0000","last_peered":"2024-09-16T15:51:18.231842+0000","last_clean":"2024-09-16T15:51:18.231842+0000","last_became_active":"2024-09-16T15:51:17.012645+0000","last_became_peered":"2024-09-16T15:51:17.012645+0000","last_unstale":"2024-09-16T15:51:18.231842+0000","last_undegraded":"2024-09-16T15:51:18.231842+0000","last_fullsized":"2024-09-16T15:51:18.231842+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:51:15.917022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:51:15.917022+0000","last_clean_scrub_stamp":"2024-09-16T15:51:15.917022+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T03:06:24.895890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"25.36","version":"0'0","reported_seq":13,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-16T15:51:18.001248+0000","last_change":"2024-09-16T15:51:17.013455+0000","last_active":"2024-09-16T15:51:18.001248+0000","last_peered":"2024-09-16T15:51:18.001248+0000","last_clean":"2024-09-16T15:51:18.001248+0000","last_became_active":"2024-09-16T15:51:17.012984+0000","last_became_peered":"2024-09-16T15:51:17.012984+0000","last_unstale":"2024-09-16T15:51:18.001248+0000","last_undegraded":"2024-09-16T15:51:18.001248+0000","last_fullsized":"2024-09-16T15:51:18.001248+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:51:15.917022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:51:15.917022+0000","last_clean_scrub_stamp":"2024-09-16T15:51:15.917022+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T23:38:46.804355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"25.35","version":"0'0","reported_seq":13,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-16T15:51:17.991779+0000","last_change":"2024-09-16T15:51:16.992056+0000","last_active":"2024-09-16T15:51:17.991779+0000","last_peered":"2024-09-16T15:51:17.991779+0000","last_clean":"2024-09-16T15:51:17.991779+0000","last_became_active":"2024-09-16T15:51:16.991747+0000","last_became_peered":"2024-09-16T15:51:16.991747+0000","last_unstale":"2024-09-16T15:51:17.991779+0000","last_undegraded":"2024-09-16T15:51:17.991779+0000","last_fullsized":"2024-09-16T15:51:17.991779+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:51:15.917022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:51:15.917022+0000","last_clean_scrub_stamp":"2024-09-16T15:51:15.917022+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T00:53:58.467091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"25.34","version":"0'0","reported_seq":13,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-16T15:51:17.993146+0000","last_change":"2024-09-16T15:51:16.998204+0000","last_active":"2024-09-16T15:51:17.993146+0000","last_peered":"2024-09-16T15:51:17.993146+0000","last_clean":"2024-09-16T15:51:17.993146+0000","last_became_active":"2024-09-16T15:51:16.997471+0000","last_became_peered":"2024-09-16T15:51:16.997471+0000","last_unstale":"2024-09-16T15:51:17.993146+0000","last_undegraded":"2024-09-16T15:51:17.993146+0000","last_fullsized":"2024-09-16T15:51:17.993146+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:51:15.917022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:51:15.917022+0000","last_clean_scrub_stamp":"2024-09-16T15:51:15.917022+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T03:16:06.385313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"25.33","version":"0'0","reported_seq":13,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-16T15:51:17.992530+0000","last_change":"2024-09-16T15:51:17.000248+0000","last_active":"2024-09-16T15:51:17.992530+0000","last_peered":"2024-09-16T15:51:17.992530+0000","last_clean":"2024-09-16T15:51:17.992530+0000","last_became_active":"2024-09-16T15:51:16.999368+0000","last_became_peered":"2024-09-16T15:51:16.999368+0000","last_unstale":"2024-09-16T15:51:17.992530+0000","last_undegraded":"2024-09-16T15:51:17.992530+0000","last_fullsized":"2024-09-16T15:51:17.992530+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:51:15.917022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:51:15.917022+0000","last_clean_scrub_stamp":"2024-09-16T15:51:15.917022+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:21:24.061511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"25.32","version":"0'0","reported_seq":13,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-16T15:51:18.229367+0000","last_change":"2024-09-16T15:51:17.016971+0000","last_active":"2024-09-16T15:51:18.229367+0000","last_peered":"2024-09-16T15:51:18.229367+0000","last_clean":"2024-09-16T15:51:18.229367+0000","last_became_active":"2024-09-16T15:51:17.016552+0000","last_became_peered":"2024-09-16T15:51:17.016552+0000","last_unstale":"2024-09-16T15:51:18.229367+0000","last_undegraded":"2024-09-16T15:51:18.229367+0000","last_fullsized":"2024-09-16T15:51:18.229367+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:51:15.917022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:51:15.917022+0000","last_clean_scrub_stamp":"2024-09-16T15:51:15.917022+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:27:22.978905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"25.31","version":"0'0","reported_seq":13,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-16T15:51:17.992370+0000","last_change":"2024-09-16T15:51:16.998890+0000","last_active":"2024-09-16T15:51:17.992370+0000","last_peered":"2024-09-16T15:51:17.992370+0000","last_clean":"2024-09-16T15:51:17.992370+0000","last_became_active":"2024-09-16T15:51:16.998347+0000","last_became_peered":"2024-09-16T15:51:16.998347+0000","last_unstale":"2024-09-16T15:51:17.992370+0000","last_undegraded":"2024-09-16T15:51:17.992370+0000","last_fullsized":"2024-09-16T15:51:17.992370+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:51:15.917022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:51:15.917022+0000","last_clean_scrub_stamp":"2024-09-16T15:51:15.917022+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T02:32:29.420307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"25.30","version":"0'0","reported_seq":13,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-16T15:51:17.991949+0000","last_change":"2024-09-16T15:51:17.014409+0000","last_active":"2024-09-16T15:51:17.991949+0000","last_peered":"2024-09-16T15:51:17.991949+0000","last_clean":"2024-09-16T15:51:17.991949+0000","last_became_active":"2024-09-16T15:51:17.013249+0000","last_became_peered":"2024-09-16T15:51:17.013249+0000","last_unstale":"2024-09-16T15:51:17.991949+0000","last_undegraded":"2024-09-16T15:51:17.991949+0000","last_fullsized":"2024-09-16T15:51:17.991949+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:51:15.917022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:51:15.917022+0000","last_clean_scrub_stamp":"2024-09-16T15:51:15.917022+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:36:03.637349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.2f","version":"0'0","reported_seq":13,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-16T15:51:18.230343+0000","last_change":"2024-09-16T15:51:17.007407+0000","last_active":"2024-09-16T15:51:18.230343+0000","last_peered":"2024-09-16T15:51:18.230343+0000","last_clean":"2024-09-16T15:51:18.230343+0000","last_became_active":"2024-09-16T15:51:17.005329+0000","last_became_peered":"2024-09-16T15:51:17.005329+0000","last_unstale":"2024-09-16T15:51:18.230343+0000","last_undegraded":"2024-09-16T15:51:18.230343+0000","last_fullsized":"2024-09-16T15:51:18.230343+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:51:15.917022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:51:15.917022+0000","last_clean_scrub_stamp":"2024-09-16T15:51:15.917022+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T01:27:54.077516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.2e","version":"0'0","reported_seq":13,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-16T15:51:18.230155+0000","last_change":"2024-09-16T15:51:17.013603+0000","last_active":"2024-09-16T15:51:18.230155+0000","last_peered":"2024-09-16T15:51:18.230155+0000","last_clean":"2024-09-16T15:51:18.230155+0000","last_became_active":"2024-09-16T15:51:17.013267+0000","last_became_peered":"2024-09-16T15:51:17.013267+0000","last_unstale":"2024-09-16T15:51:18.230155+0000","last_undegraded":"2024-09-16T15:51:18.230155+0000","last_fullsized":"2024-09-16T15:51:18.230155+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:51:15.917022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:51:15.917022+0000","last_clean_scrub_stamp":"2024-09-16T15:51:15.917022+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:50:07.541567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.2d","version":"0'0","reported_seq":13,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-16T15:51:17.999660+0000","last_change":"2024-09-16T15:51:17.006352+0000","last_active":"2024-09-16T15:51:17.999660+0000","last_peered":"2024-09-16T15:51:17.999660+0000","last_clean":"2024-09-16T15:51:17.999660+0000","last_became_active":"2024-09-16T15:51:17.006061+0000","last_became_peered":"2024-09-16T15:51:17.006061+0000","last_unstale":"2024-09-16T15:51:17.999660+0000","last_undegraded":"2024-09-16T15:51:17.999660+0000","last_fullsized":"2024-09-16T15:51:17.999660+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:51:15.917022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:51:15.917022+0000","last_clean_scrub_stamp":"2024-09-16T15:51:15.917022+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:50:07.541567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"25.2c","version":"0'0","reported_seq":13,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-16T15:51:18.230085+0000","last_change":"2024-09-16T15:51:17.005441+0000","last_active":"2024-09-16T15:51:18.230085+0000","last_peered":"2024-09-16T15:51:18.230085+0000","last_clean":"2024-09-16T15:51:18.230085+0000","last_became_active":"2024-09-16T15:51:17.004994+0000","last_became_peered":"2024-09-16T15:51:17.004994+0000","last_unstale":"2024-09-16T15:51:18.230085+0000","last_undegraded":"2024-09-16T15:51:18.230085+0000","last_fullsized":"2024-09-16T15:51:18.230085+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:51:15.917022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:51:15.917022+0000","last_clean_scrub_stamp":"2024-09-16T15:51:15.917022+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T02:46:37.039388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.2b","version":"0'0","reported_seq":13,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-16T15:51:18.229848+0000","last_change":"2024-09-16T15:51:17.008733+0000","last_active":"2024-09-16T15:51:18.229848+0000","last_peered":"2024-09-16T15:51:18.229848+0000","last_clean":"2024-09-16T15:51:18.229848+0000","last_became_active":"2024-09-16T15:51:17.008070+0000","last_became_peered":"2024-09-16T15:51:17.008070+0000","last_unstale":"2024-09-16T15:51:18.229848+0000","last_undegraded":"2024-09-16T15:51:18.229848+0000","last_fullsized":"2024-09-16T15:51:18.229848+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:51:15.917022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:51:15.917022+0000","last_clean_scrub_stamp":"2024-09-16T15:51:15.917022+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:42:02.533881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.2a","version":"0'0","reported_seq":13,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-16T15:51:17.999123+0000","last_change":"2024-09-16T15:51:17.012375+0000","last_active":"2024-09-16T15:51:17.999123+0000","last_peered":"2024-09-16T15:51:17.999123+0000","last_clean":"2024-09-16T15:51:17.999123+0000","last_became_active":"2024-09-16T15:51:17.011538+0000","last_became_peered":"2024-09-16T15:51:17.011538+0000","last_unstale":"2024-09-16T15:51:17.999123+0000","last_undegraded":"2024-09-16T15:51:17.999123+0000","last_fullsized":"2024-09-16T15:51:17.999123+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:51:15.917022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:51:15.917022+0000","last_clean_scrub_stamp":"2024-09-16T15:51:15.917022+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:16:48.939808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"25.29","version":"0'0","reported_seq":13,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-16T15:51:18.006020+0000","last_change":"2024-09-16T15:51:17.015534+0000","last_active":"2024-09-16T15:51:18.006020+0000","last_peered":"2024-09-16T15:51:18.006020+0000","last_clean":"2024-09-16T15:51:18.006020+0000","last_became_active":"2024-09-16T15:51:17.015229+0000","last_became_peered":"2024-09-16T15:51:17.015229+0000","last_unstale":"2024-09-16T15:51:18.006020+0000","last_undegraded":"2024-09-16T15:51:18.006020+0000","last_fullsized":"2024-09-16T15:51:18.006020+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:51:15.917022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:51:15.917022+0000","last_clean_scrub_stamp":"2024-09-16T15:51:15.917022+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:10:02.074524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.28","version":"0'0","reported_seq":13,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-16T15:51:17.992298+0000","last_change":"2024-09-16T15:51:16.998452+0000","last_active":"2024-09-16T15:51:17.992298+0000","last_peered":"2024-09-16T15:51:17.992298+0000","last_clean":"2024-09-16T15:51:17.992298+0000","last_became_active":"2024-09-16T15:51:16.997685+0000","last_became_peered":"2024-09-16T15:51:16.997685+0000","last_unstale":"2024-09-16T15:51:17.992298+0000","last_undegraded":"2024-09-16T15:51:17.992298+0000","last_fullsized":"2024-09-16T15:51:17.992298+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:51:15.917022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:51:15.917022+0000","last_clean_scrub_stamp":"2024-09-16T15:51:15.917022+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:28:52.947127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.27","version":"0'0","reported_seq":13,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-16T15:51:17.993956+0000","last_change":"2024-09-16T15:51:17.012326+0000","last_active":"2024-09-16T15:51:17.993956+0000","last_peered":"2024-09-16T15:51:17.993956+0000","last_clean":"2024-09-16T15:51:17.993956+0000","last_became_active":"2024-09-16T15:51:17.011953+0000","last_became_peered":"2024-09-16T15:51:17.011953+0000","last_unstale":"2024-09-16T15:51:17.993956+0000","last_undegraded":"2024-09-16T15:51:17.993956+0000","last_fullsized":"2024-09-16T15:51:17.993956+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:51:15.917022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:51:15.917022+0000","last_clean_scrub_stamp":"2024-09-16T15:51:15.917022+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:01:30.205778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"25.26","version":"0'0","reported_seq":13,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-16T15:51:17.990846+0000","last_change":"2024-09-16T15:51:17.013974+0000","last_active":"2024-09-16T15:51:17.990846+0000","last_peered":"2024-09-16T15:51:17.990846+0000","last_clean":"2024-09-16T15:51:17.990846+0000","last_became_active":"2024-09-16T15:51:17.013672+0000","last_became_peered":"2024-09-16T15:51:17.013672+0000","last_unstale":"2024-09-16T15:51:17.990846+0000","last_undegraded":"2024-09-16T15:51:17.990846+0000","last_fullsized":"2024-09-16T15:51:17.990846+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:51:15.917022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:51:15.917022+0000","last_clean_scrub_stamp":"2024-09-16T15:51:15.917022+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:58:41.585587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"25.25","version":"0'0","reported_seq":13,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-16T15:51:18.006635+0000","last_change":"2024-09-16T15:51:16.993740+0000","last_active":"2024-09-16T15:51:18.006635+0000","last_peered":"2024-09-16T15:51:18.006635+0000","last_clean":"2024-09-16T15:51:18.006635+0000","last_became_active":"2024-09-16T15:51:16.993040+0000","last_became_peered":"2024-09-16T15:51:16.993040+0000","last_unstale":"2024-09-16T15:51:18.006635+0000","last_undegraded":"2024-09-16T15:51:18.006635+0000","last_fullsized":"2024-09-16T15:51:18.006635+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:51:15.917022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:51:15.917022+0000","last_clean_scrub_stamp":"2024-09-16T15:51:15.917022+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T03:13:06.576190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.24","version":"0'0","reported_seq":13,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-16T15:51:18.230368+0000","last_change":"2024-09-16T15:51:17.011958+0000","last_active":"2024-09-16T15:51:18.230368+0000","last_peered":"2024-09-16T15:51:18.230368+0000","last_clean":"2024-09-16T15:51:18.230368+0000","last_became_active":"2024-09-16T15:51:17.011498+0000","last_became_peered":"2024-09-16T15:51:17.011498+0000","last_unstale":"2024-09-16T15:51:18.230368+0000","last_undegraded":"2024-09-16T15:51:18.230368+0000","last_fullsized":"2024-09-16T15:51:18.230368+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:51:15.917022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:51:15.917022+0000","last_clean_scrub_stamp":"2024-09-16T15:51:15.917022+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T03:50:14.344816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.23","version":"0'0","reported_seq":13,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-16T15:51:17.992868+0000","last_change":"2024-09-16T15:51:16.997588+0000","last_active":"2024-09-16T15:51:17.992868+0000","last_peered":"2024-09-16T15:51:17.992868+0000","last_clean":"2024-09-16T15:51:17.992868+0000","last_became_active":"2024-09-16T15:51:16.996903+0000","last_became_peered":"2024-09-16T15:51:16.996903+0000","last_unstale":"2024-09-16T15:51:17.992868+0000","last_undegraded":"2024-09-16T15:51:17.992868+0000","last_fullsized":"2024-09-16T15:51:17.992868+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:51:15.917022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:51:15.917022+0000","last_clean_scrub_stamp":"2024-09-16T15:51:15.917022+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T18:57:14.461693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"25.22","version":"0'0","reported_seq":13,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-16T15:51:18.230375+0000","last_change":"2024-09-16T15:51:16.997698+0000","last_active":"2024-09-16T15:51:18.230375+0000","last_peered":"2024-09-16T15:51:18.230375+0000","last_clean":"2024-09-16T15:51:18.230375+0000","last_became_active":"2024-09-16T15:51:16.997319+0000","last_became_peered":"2024-09-16T15:51:16.997319+0000","last_unstale":"2024-09-16T15:51:18.230375+0000","last_undegraded":"2024-09-16T15:51:18.230375+0000","last_fullsized":"2024-09-16T15:51:18.230375+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:51:15.917022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:51:15.917022+0000","last_clean_scrub_stamp":"2024-09-16T15:51:15.917022+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T18:40:03.594059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"25.21","version":"0'0","reported_seq":13,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-16T15:51:18.006015+0000","last_change":"2024-09-16T15:51:16.990170+0000","last_active":"2024-09-16T15:51:18.006015+0000","last_peered":"2024-09-16T15:51:18.006015+0000","last_clean":"2024-09-16T15:51:18.006015+0000","last_became_active":"2024-09-16T15:51:16.989494+0000","last_became_peered":"2024-09-16T15:51:16.989494+0000","last_unstale":"2024-09-16T15:51:18.006015+0000","last_undegraded":"2024-09-16T15:51:18.006015+0000","last_fullsized":"2024-09-16T15:51:18.006015+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:51:15.917022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:51:15.917022+0000","last_clean_scrub_stamp":"2024-09-16T15:51:15.917022+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:14:35.276401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.20","version":"0'0","reported_seq":13,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-16T15:51:18.229417+0000","last_change":"2024-09-16T15:51:16.997649+0000","last_active":"2024-09-16T15:51:18.229417+0000","last_peered":"2024-09-16T15:51:18.229417+0000","last_clean":"2024-09-16T15:51:18.229417+0000","last_became_active":"2024-09-16T15:51:16.997270+0000","last_became_peered":"2024-09-16T15:51:16.997270+0000","last_unstale":"2024-09-16T15:51:18.229417+0000","last_undegraded":"2024-09-16T15:51:18.229417+0000","last_fullsized":"2024-09-16T15:51:18.229417+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:51:15.917022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:51:15.917022+0000","last_clean_scrub_stamp":"2024-09-16T15:51:15.917022+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T18:05:42.952449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.1e","version":"0'0","reported_seq":13,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-16T15:51:17.991790+0000","last_change":"2024-09-16T15:51:17.020448+0000","last_active":"2024-09-16T15:51:17.991790+0000","last_peered":"2024-09-16T15:51:17.991790+0000","last_clean":"2024-09-16T15:51:17.991790+0000","last_became_active":"2024-09-16T15:51:17.020006+0000","last_became_peered":"2024-09-16T15:51:17.020006+0000","last_unstale":"2024-09-16T15:51:17.991790+0000","last_undegraded":"2024-09-16T15:51:17.991790+0000","last_fullsized":"2024-09-16T15:51:17.991790+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:51:15.917022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:51:15.917022+0000","last_clean_scrub_stamp":"2024-09-16T15:51:15.917022+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T02:31:40.090279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.1f","version":"156'4","reported_seq":19,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-16T15:51:18.230202+0000","last_change":"2024-09-16T15:51:15.950651+0000","last_active":"2024-09-16T15:51:18.230202+0000","last_peered":"2024-09-16T15:51:18.230202+0000","last_clean":"2024-09-16T15:51:18.230202+0000","last_became_active":"2024-09-16T15:51:15.950248+0000","last_became_peered":"2024-09-16T15:51:15.950248+0000","last_unstale":"2024-09-16T15:51:18.230202+0000","last_undegraded":"2024-09-16T15:51:18.230202+0000","last_fullsized":"2024-09-16T15:51:18.230202+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:51:14.906573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:51:14.906573+0000","last_clean_scrub_stamp":"2024-09-16T15:51:14.906573+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T01:53:33.146583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.1f","version":"0'0","reported_seq":13,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-16T15:51:18.000624+0000","last_change":"2024-09-16T15:51:17.019764+0000","last_active":"2024-09-16T15:51:18.000624+0000","last_peered":"2024-09-16T15:51:18.000624+0000","last_clean":"2024-09-16T15:51:18.000624+0000","last_became_active":"2024-09-16T15:51:17.019429+0000","last_became_peered":"2024-09-16T15:51:17.019429+0000","last_unstale":"2024-09-16T15:51:18.000624+0000","last_undegraded":"2024-09-16T15:51:18.000624+0000","last_fullsized":"2024-09-16T15:51:18.000624+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:51:15.917022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:51:15.917022+0000","last_clean_scrub_stamp":"2024-09-16T15:51:15.917022+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T18:27:49.521296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.1e","version":"156'1","reported_seq":16,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-16T15:51:18.231306+0000","last_change":"2024-09-16T15:51:15.949651+0000","last_active":"2024-09-16T15:51:18.231306+0000","last_peered":"2024-09-16T15:51:18.231306+0000","last_clean":"2024-09-16T15:51:18.231306+0000","last_became_active":"2024-09-16T15:51:15.946835+0000","last_became_peered":"2024-09-16T15:51:15.946835+0000","last_unstale":"2024-09-16T15:51:18.231306+0000","last_undegraded":"2024-09-16T15:51:18.231306+0000","last_fullsized":"2024-09-16T15:51:18.231306+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:51:14.906573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:51:14.906573+0000","last_clean_scrub_stamp":"2024-09-16T15:51:14.906573+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T02:18:13.853820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"25.1c","version":"0'0","reported_seq":13,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-16T15:51:17.991198+0000","last_change":"2024-09-16T15:51:17.019402+0000","last_active":"2024-09-16T15:51:17.991198+0000","last_peered":"2024-09-16T15:51:17.991198+0000","last_clean":"2024-09-16T15:51:17.991198+0000","last_became_active":"2024-09-16T15:51:17.018991+0000","last_became_peered":"2024-09-16T15:51:17.018991+0000","last_unstale":"2024-09-16T15:51:17.991198+0000","last_undegraded":"2024-09-16T15:51:17.991198+0000","last_fullsized":"2024-09-16T15:51:17.991198+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:51:15.917022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:51:15.917022+0000","last_clean_scrub_stamp":"2024-09-16T15:51:15.917022+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T00:47:18.987282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.1d","version":"156'2","reported_seq":17,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-16T15:51:18.006559+0000","last_change":"2024-09-16T15:51:15.958280+0000","last_active":"2024-09-16T15:51:18.006559+0000","last_peered":"2024-09-16T15:51:18.006559+0000","last_clean":"2024-09-16T15:51:18.006559+0000","last_became_active":"2024-09-16T15:51:15.957914+0000","last_became_peered":"2024-09-16T15:51:15.957914+0000","last_unstale":"2024-09-16T15:51:18.006559+0000","last_undegraded":"2024-09-16T15:51:18.006559+0000","last_fullsized":"2024-09-16T15:51:18.006559+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:51:14.906573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:51:14.906573+0000","last_clean_scrub_stamp":"2024-09-16T15:51:14.906573+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:37:41.174910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.1d","version":"0'0","reported_seq":13,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-16T15:51:18.231556+0000","last_change":"2024-09-16T15:51:16.993051+0000","last_active":"2024-09-16T15:51:18.231556+0000","last_peered":"2024-09-16T15:51:18.231556+0000","last_clean":"2024-09-16T15:51:18.231556+0000","last_became_active":"2024-09-16T15:51:16.992320+0000","last_became_peered":"2024-09-16T15:51:16.992320+0000","last_unstale":"2024-09-16T15:51:18.231556+0000","last_undegraded":"2024-09-16T15:51:18.231556+0000","last_fullsized":"2024-09-16T15:51:18.231556+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:51:15.917022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:51:15.917022+0000","last_clean_scrub_stamp":"2024-09-16T15:51:15.917022+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T02:39:46.607375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.1c","version":"156'2","reported_seq":17,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-16T15:51:17.993169+0000","last_change":"2024-09-16T15:51:15.954496+0000","last_active":"2024-09-16T15:51:17.993169+0000","last_peered":"2024-09-16T15:51:17.993169+0000","last_clean":"2024-09-16T15:51:17.993169+0000","last_became_active":"2024-09-16T15:51:15.953801+0000","last_became_peered":"2024-09-16T15:51:15.953801+0000","last_unstale":"2024-09-16T15:51:17.993169+0000","last_undegraded":"2024-09-16T15:51:17.993169+0000","last_fullsized":"2024-09-16T15:51:17.993169+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:51:14.906573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:51:14.906573+0000","last_clean_scrub_stamp":"2024-09-16T15:51:14.906573+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T22:59:22.058566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"25.1a","version":"0'0","reported_seq":13,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-16T15:51:17.999347+0000","last_change":"2024-09-16T15:51:17.011733+0000","last_active":"2024-09-16T15:51:17.999347+0000","last_peered":"2024-09-16T15:51:17.999347+0000","last_clean":"2024-09-16T15:51:17.999347+0000","last_became_active":"2024-09-16T15:51:17.011211+0000","last_became_peered":"2024-09-16T15:51:17.011211+0000","last_unstale":"2024-09-16T15:51:17.999347+0000","last_undegraded":"2024-09-16T15:51:17.999347+0000","last_fullsized":"2024-09-16T15:51:17.999347+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:51:15.917022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:51:15.917022+0000","last_clean_scrub_stamp":"2024-09-16T15:51:15.917022+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T00:28:57.939261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.1b","version":"156'1","reported_seq":16,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-16T15:51:18.230021+0000","last_change":"2024-09-16T15:51:15.960597+0000","last_active":"2024-09-16T15:51:18.230021+0000","last_peered":"2024-09-16T15:51:18.230021+0000","last_clean":"2024-09-16T15:51:18.230021+0000","last_became_active":"2024-09-16T15:51:15.957029+0000","last_became_peered":"2024-09-16T15:51:15.957029+0000","last_unstale":"2024-09-16T15:51:18.230021+0000","last_undegraded":"2024-09-16T15:51:18.230021+0000","last_fullsized":"2024-09-16T15:51:18.230021+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:51:14.906573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:51:14.906573+0000","last_clean_scrub_stamp":"2024-09-16T15:51:14.906573+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:59:01.722146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.1b","version":"0'0","reported_seq":13,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-16T15:51:17.992497+0000","last_change":"2024-09-16T15:51:17.004716+0000","last_active":"2024-09-16T15:51:17.992497+0000","last_peered":"2024-09-16T15:51:17.992497+0000","last_clean":"2024-09-16T15:51:17.992497+0000","last_became_active":"2024-09-16T15:51:17.003589+0000","last_became_peered":"2024-09-16T15:51:17.003589+0000","last_unstale":"2024-09-16T15:51:17.992497+0000","last_undegraded":"2024-09-16T15:51:17.992497+0000","last_fullsized":"2024-09-16T15:51:17.992497+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:51:15.917022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:51:15.917022+0000","last_clean_scrub_stamp":"2024-09-16T15:51:15.917022+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T18:19:15.107168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.1a","version":"156'1","reported_seq":16,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-16T15:51:17.992920+0000","last_change":"2024-09-16T15:51:15.953951+0000","last_active":"2024-09-16T15:51:17.992920+0000","last_peered":"2024-09-16T15:51:17.992920+0000","last_clean":"2024-09-16T15:51:17.992920+0000","last_became_active":"2024-09-16T15:51:15.953593+0000","last_became_peered":"2024-09-16T15:51:15.953593+0000","last_unstale":"2024-09-16T15:51:17.992920+0000","last_undegraded":"2024-09-16T15:51:17.992920+0000","last_fullsized":"2024-09-16T15:51:17.992920+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:51:14.906573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:51:14.906573+0000","last_clean_scrub_stamp":"2024-09-16T15:51:14.906573+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T01:54:24.734575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.19","version":"0'0","reported_seq":13,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-16T15:51:17.990598+0000","last_change":"2024-09-16T15:51:16.996685+0000","last_active":"2024-09-16T15:51:17.990598+0000","last_peered":"2024-09-16T15:51:17.990598+0000","last_clean":"2024-09-16T15:51:17.990598+0000","last_became_active":"2024-09-16T15:51:16.996255+0000","last_became_peered":"2024-09-16T15:51:16.996255+0000","last_unstale":"2024-09-16T15:51:17.990598+0000","last_undegraded":"2024-09-16T15:51:17.990598+0000","last_fullsized":"2024-09-16T15:51:17.990598+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:51:15.917022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:51:15.917022+0000","last_clean_scrub_stamp":"2024-09-16T15:51:15.917022+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:08:18.096948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.18","version":"0'0","reported_seq":15,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-16T15:51:18.228868+0000","last_change":"2024-09-16T15:51:15.954905+0000","last_active":"2024-09-16T15:51:18.228868+0000","last_peered":"2024-09-16T15:51:18.228868+0000","last_clean":"2024-09-16T15:51:18.228868+0000","last_became_active":"2024-09-16T15:51:15.954141+0000","last_became_peered":"2024-09-16T15:51:15.954141+0000","last_unstale":"2024-09-16T15:51:18.228868+0000","last_undegraded":"2024-09-16T15:51:18.228868+0000","last_fullsized":"2024-09-16T15:51:18.228868+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:51:14.906573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:51:14.906573+0000","last_clean_scrub_stamp":"2024-09-16T15:51:14.906573+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:01:34.100930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.16","version":"0'0","reported_seq":13,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-16T15:51:18.007088+0000","last_change":"2024-09-16T15:51:17.002955+0000","last_active":"2024-09-16T15:51:18.007088+0000","last_peered":"2024-09-16T15:51:18.007088+0000","last_clean":"2024-09-16T15:51:18.007088+0000","last_became_active":"2024-09-16T15:51:17.002668+0000","last_became_peered":"2024-09-16T15:51:17.002668+0000","last_unstale":"2024-09-16T15:51:18.007088+0000","last_undegraded":"2024-09-16T15:51:18.007088+0000","last_fullsized":"2024-09-16T15:51:18.007088+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:51:15.917022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:51:15.917022+0000","last_clean_scrub_stamp":"2024-09-16T15:51:15.917022+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:44:50.889568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.17","version":"0'0","reported_seq":15,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-16T15:51:17.990690+0000","last_change":"2024-09-16T15:51:15.956604+0000","last_active":"2024-09-16T15:51:17.990690+0000","last_peered":"2024-09-16T15:51:17.990690+0000","last_clean":"2024-09-16T15:51:17.990690+0000","last_became_active":"2024-09-16T15:51:15.955201+0000","last_became_peered":"2024-09-16T15:51:15.955201+0000","last_unstale":"2024-09-16T15:51:17.990690+0000","last_undegraded":"2024-09-16T15:51:17.990690+0000","last_fullsized":"2024-09-16T15:51:17.990690+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:51:14.906573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:51:14.906573+0000","last_clean_scrub_stamp":"2024-09-16T15:51:14.906573+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:47:16.682466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"25.17","version":"0'0","reported_seq":13,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-16T15:51:17.992029+0000","last_change":"2024-09-16T15:51:17.021310+0000","last_active":"2024-09-16T15:51:17.992029+0000","last_peered":"2024-09-16T15:51:17.992029+0000","last_clean":"2024-09-16T15:51:17.992029+0000","last_became_active":"2024-09-16T15:51:17.019103+0000","last_became_peered":"2024-09-16T15:51:17.019103+0000","last_unstale":"2024-09-16T15:51:17.992029+0000","last_undegraded":"2024-09-16T15:51:17.992029+0000","last_fullsized":"2024-09-16T15:51:17.992029+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:51:15.917022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:51:15.917022+0000","last_clean_scrub_stamp":"2024-09-16T15:51:15.917022+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:37:51.032687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.16","version":"0'0","reported_seq":15,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-16T15:51:18.229691+0000","last_change":"2024-09-16T15:51:15.962253+0000","last_active":"2024-09-16T15:51:18.229691+0000","last_peered":"2024-09-16T15:51:18.229691+0000","last_clean":"2024-09-16T15:51:18.229691+0000","last_became_active":"2024-09-16T15:51:15.961884+0000","last_became_peered":"2024-09-16T15:51:15.961884+0000","last_unstale":"2024-09-16T15:51:18.229691+0000","last_undegraded":"2024-09-16T15:51:18.229691+0000","last_fullsized":"2024-09-16T15:51:18.229691+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:51:14.906573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:51:14.906573+0000","last_clean_scrub_stamp":"2024-09-16T15:51:14.906573+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T18:17:02.816460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.14","version":"0'0","reported_seq":13,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-16T15:51:17.991842+0000","last_change":"2024-09-16T15:51:17.006585+0000","last_active":"2024-09-16T15:51:17.991842+0000","last_peered":"2024-09-16T15:51:17.991842+0000","last_clean":"2024-09-16T15:51:17.991842+0000","last_became_active":"2024-09-16T15:51:17.006282+0000","last_became_peered":"2024-09-16T15:51:17.006282+0000","last_unstale":"2024-09-16T15:51:17.991842+0000","last_undegraded":"2024-09-16T15:51:17.991842+0000","last_fullsized":"2024-09-16T15:51:17.991842+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:51:15.917022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:51:15.917022+0000","last_clean_scrub_stamp":"2024-09-16T15:51:15.917022+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T00:08:29.575374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.15","version":"156'3","reported_seq":18,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-16T15:51:18.229663+0000","last_change":"2024-09-16T15:51:15.955867+0000","last_active":"2024-09-16T15:51:18.229663+0000","last_peered":"2024-09-16T15:51:18.229663+0000","last_clean":"2024-09-16T15:51:18.229663+0000","last_became_active":"2024-09-16T15:51:15.955585+0000","last_became_peered":"2024-09-16T15:51:15.955585+0000","last_unstale":"2024-09-16T15:51:18.229663+0000","last_undegraded":"2024-09-16T15:51:18.229663+0000","last_fullsized":"2024-09-16T15:51:18.229663+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:51:14.906573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:51:14.906573+0000","last_clean_scrub_stamp":"2024-09-16T15:51:14.906573+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:17:19.428586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.15","version":"0'0","reported_seq":13,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-16T15:51:18.000199+0000","last_change":"2024-09-16T15:51:17.012094+0000","last_active":"2024-09-16T15:51:18.000199+0000","last_peered":"2024-09-16T15:51:18.000199+0000","last_clean":"2024-09-16T15:51:18.000199+0000","last_became_active":"2024-09-16T15:51:17.010977+0000","last_became_peered":"2024-09-16T15:51:17.010977+0000","last_unstale":"2024-09-16T15:51:18.000199+0000","last_undegraded":"2024-09-16T15:51:18.000199+0000","last_fullsized":"2024-09-16T15:51:18.000199+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:51:15.917022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:51:15.917022+0000","last_clean_scrub_stamp":"2024-09-16T15:51:15.917022+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T00:37:58.329330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.14","version":"156'6","reported_seq":21,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-16T15:51:18.006272+0000","last_change":"2024-09-16T15:51:15.944279+0000","last_active":"2024-09-16T15:51:18.006272+0000","last_peered":"2024-09-16T15:51:18.006272+0000","last_clean":"2024-09-16T15:51:18.006272+0000","last_became_active":"2024-09-16T15:51:15.943434+0000","last_became_peered":"2024-09-16T15:51:15.943434+0000","last_unstale":"2024-09-16T15:51:18.006272+0000","last_undegraded":"2024-09-16T15:51:18.006272+0000","last_fullsized":"2024-09-16T15:51:18.006272+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:51:14.906573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:51:14.906573+0000","last_clean_scrub_stamp":"2024-09-16T15:51:14.906573+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T00:17:37.514552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.12","version":"0'0","reported_seq":13,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-16T15:51:18.006244+0000","last_change":"2024-09-16T15:51:17.012749+0000","last_active":"2024-09-16T15:51:18.006244+0000","last_peered":"2024-09-16T15:51:18.006244+0000","last_clean":"2024-09-16T15:51:18.006244+0000","last_became_active":"2024-09-16T15:51:17.012481+0000","last_became_peered":"2024-09-16T15:51:17.012481+0000","last_unstale":"2024-09-16T15:51:18.006244+0000","last_undegraded":"2024-09-16T15:51:18.006244+0000","last_fullsized":"2024-09-16T15:51:18.006244+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:51:15.917022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:51:15.917022+0000","last_clean_scrub_stamp":"2024-09-16T15:51:15.917022+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:53:02.451961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.13","version":"156'2","reported_seq":17,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-16T15:51:18.229766+0000","last_change":"2024-09-16T15:51:15.951395+0000","last_active":"2024-09-16T15:51:18.229766+0000","last_peered":"2024-09-16T15:51:18.229766+0000","last_clean":"2024-09-16T15:51:18.229766+0000","last_became_active":"2024-09-16T15:51:15.950630+0000","last_became_peered":"2024-09-16T15:51:15.950630+0000","last_unstale":"2024-09-16T15:51:18.229766+0000","last_undegraded":"2024-09-16T15:51:18.229766+0000","last_fullsized":"2024-09-16T15:51:18.229766+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:51:14.906573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:51:14.906573+0000","last_clean_scrub_stamp":"2024-09-16T15:51:14.906573+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T23:45:28.575869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.13","version":"0'0","reported_seq":13,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-16T15:51:17.991414+0000","last_change":"2024-09-16T15:51:17.010596+0000","last_active":"2024-09-16T15:51:17.991414+0000","last_peered":"2024-09-16T15:51:17.991414+0000","last_clean":"2024-09-16T15:51:17.991414+0000","last_became_active":"2024-09-16T15:51:17.010306+0000","last_became_peered":"2024-09-16T15:51:17.010306+0000","last_unstale":"2024-09-16T15:51:17.991414+0000","last_undegraded":"2024-09-16T15:51:17.991414+0000","last_fullsized":"2024-09-16T15:51:17.991414+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:51:15.917022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:51:15.917022+0000","last_clean_scrub_stamp":"2024-09-16T15:51:15.917022+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T01:05:48.879633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.12","version":"0'0","reported_seq":15,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-16T15:51:17.999445+0000","last_change":"2024-09-16T15:51:15.949325+0000","last_active":"2024-09-16T15:51:17.999445+0000","last_peered":"2024-09-16T15:51:17.999445+0000","last_clean":"2024-09-16T15:51:17.999445+0000","last_became_active":"2024-09-16T15:51:15.948939+0000","last_became_peered":"2024-09-16T15:51:15.948939+0000","last_unstale":"2024-09-16T15:51:17.999445+0000","last_undegraded":"2024-09-16T15:51:17.999445+0000","last_fullsized":"2024-09-16T15:51:17.999445+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:51:14.906573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:51:14.906573+0000","last_clean_scrub_stamp":"2024-09-16T15:51:14.906573+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:36:11.682195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"25.10","version":"0'0","reported_seq":13,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-16T15:51:18.229257+0000","last_change":"2024-09-16T15:51:17.013579+0000","last_active":"2024-09-16T15:51:18.229257+0000","last_peered":"2024-09-16T15:51:18.229257+0000","last_clean":"2024-09-16T15:51:18.229257+0000","last_became_active":"2024-09-16T15:51:17.013174+0000","last_became_peered":"2024-09-16T15:51:17.013174+0000","last_unstale":"2024-09-16T15:51:18.229257+0000","last_undegraded":"2024-09-16T15:51:18.229257+0000","last_fullsized":"2024-09-16T15:51:18.229257+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:51:15.917022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:51:15.917022+0000","last_clean_scrub_stamp":"2024-09-16T15:51:15.917022+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:46:10.913577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.11","version":"0'0","reported_seq":15,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-16T15:51:17.991342+0000","last_change":"2024-09-16T15:51:15.962934+0000","last_active":"2024-09-16T15:51:17.991342+0000","last_peered":"2024-09-16T15:51:17.991342+0000","last_clean":"2024-09-16T15:51:17.991342+0000","last_became_active":"2024-09-16T15:51:15.962573+0000","last_became_peered":"2024-09-16T15:51:15.962573+0000","last_unstale":"2024-09-16T15:51:17.991342+0000","last_undegraded":"2024-09-16T15:51:17.991342+0000","last_fullsized":"2024-09-16T15:51:17.991342+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:51:14.906573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:51:14.906573+0000","last_clean_scrub_stamp":"2024-09-16T15:51:14.906573+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:20:38.009303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.11","version":"0'0","reported_seq":13,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-16T15:51:17.999314+0000","last_change":"2024-09-16T15:51:17.007443+0000","last_active":"2024-09-16T15:51:17.999314+0000","last_peered":"2024-09-16T15:51:17.999314+0000","last_clean":"2024-09-16T15:51:17.999314+0000","last_became_active":"2024-09-16T15:51:17.007213+0000","last_became_peered":"2024-09-16T15:51:17.007213+0000","last_unstale":"2024-09-16T15:51:17.999314+0000","last_undegraded":"2024-09-16T15:51:17.999314+0000","last_fullsized":"2024-09-16T15:51:17.999314+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:51:15.917022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:51:15.917022+0000","last_clean_scrub_stamp":"2024-09-16T15:51:15.917022+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:26:48.639401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.10","version":"156'8","reported_seq":23,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-16T15:51:18.229151+0000","last_change":"2024-09-16T15:51:15.960961+0000","last_active":"2024-09-16T15:51:18.229151+0000","last_peered":"2024-09-16T15:51:18.229151+0000","last_clean":"2024-09-16T15:51:18.229151+0000","last_became_active":"2024-09-16T15:51:15.960126+0000","last_became_peered":"2024-09-16T15:51:15.960126+0000","last_unstale":"2024-09-16T15:51:18.229151+0000","last_undegraded":"2024-09-16T15:51:18.229151+0000","last_fullsized":"2024-09-16T15:51:18.229151+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:51:14.906573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:51:14.906573+0000","last_clean_scrub_stamp":"2024-09-16T15:51:14.906573+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:39.437445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.e","version":"0'0","reported_seq":13,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-16T15:51:18.001534+0000","last_change":"2024-09-16T15:51:17.013919+0000","last_active":"2024-09-16T15:51:18.001534+0000","last_peered":"2024-09-16T15:51:18.001534+0000","last_clean":"2024-09-16T15:51:18.001534+0000","last_became_active":"2024-09-16T15:51:17.012170+0000","last_became_peered":"2024-09-16T15:51:17.012170+0000","last_unstale":"2024-09-16T15:51:18.001534+0000","last_undegraded":"2024-09-16T15:51:18.001534+0000","last_fullsized":"2024-09-16T15:51:18.001534+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:51:15.917022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:51:15.917022+0000","last_clean_scrub_stamp":"2024-09-16T15:51:15.917022+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:52:37.466721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.f","version":"156'1","reported_seq":16,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-16T15:51:18.230344+0000","last_change":"2024-09-16T15:51:15.961036+0000","last_active":"2024-09-16T15:51:18.230344+0000","last_peered":"2024-09-16T15:51:18.230344+0000","last_clean":"2024-09-16T15:51:18.230344+0000","last_became_active":"2024-09-16T15:51:15.960726+0000","last_became_peered":"2024-09-16T15:51:15.960726+0000","last_unstale":"2024-09-16T15:51:18.230344+0000","last_undegraded":"2024-09-16T15:51:18.230344+0000","last_fullsized":"2024-09-16T15:51:18.230344+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:51:14.906573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:51:14.906573+0000","last_clean_scrub_stamp":"2024-09-16T15:51:14.906573+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T02:06:03.157019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.f","version":"0'0","reported_seq":13,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-16T15:51:18.231827+0000","last_change":"2024-09-16T15:51:17.016626+0000","last_active":"2024-09-16T15:51:18.231827+0000","last_peered":"2024-09-16T15:51:18.231827+0000","last_clean":"2024-09-16T15:51:18.231827+0000","last_became_active":"2024-09-16T15:51:17.014818+0000","last_became_peered":"2024-09-16T15:51:17.014818+0000","last_unstale":"2024-09-16T15:51:18.231827+0000","last_undegraded":"2024-09-16T15:51:18.231827+0000","last_fullsized":"2024-09-16T15:51:18.231827+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:51:15.917022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:51:15.917022+0000","last_clean_scrub_stamp":"2024-09-16T15:51:15.917022+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:18:36.507362+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.e","version":"156'2","reported_seq":17,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-16T15:51:17.992623+0000","last_change":"2024-09-16T15:51:15.953442+0000","last_active":"2024-09-16T15:51:17.992623+0000","last_peered":"2024-09-16T15:51:17.992623+0000","last_clean":"2024-09-16T15:51:17.992623+0000","last_became_active":"2024-09-16T15:51:15.953180+0000","last_became_peered":"2024-09-16T15:51:15.953180+0000","last_unstale":"2024-09-16T15:51:17.992623+0000","last_undegraded":"2024-09-16T15:51:17.992623+0000","last_fullsized":"2024-09-16T15:51:17.992623+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:51:14.906573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:51:14.906573+0000","last_clean_scrub_stamp":"2024-09-16T15:51:14.906573+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:41:37.066613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.c","version":"0'0","reported_seq":13,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-16T15:51:18.231506+0000","last_change":"2024-09-16T15:51:16.996979+0000","last_active":"2024-09-16T15:51:18.231506+0000","last_peered":"2024-09-16T15:51:18.231506+0000","last_clean":"2024-09-16T15:51:18.231506+0000","last_became_active":"2024-09-16T15:51:16.996721+0000","last_became_peered":"2024-09-16T15:51:16.996721+0000","last_unstale":"2024-09-16T15:51:18.231506+0000","last_undegraded":"2024-09-16T15:51:18.231506+0000","last_fullsized":"2024-09-16T15:51:18.231506+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:51:15.917022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:51:15.917022+0000","last_clean_scrub_stamp":"2024-09-16T15:51:15.917022+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:09:32.727980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.d","version":"156'1","reported_seq":16,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-16T15:51:18.230371+0000","last_change":"2024-09-16T15:51:15.957621+0000","last_active":"2024-09-16T15:51:18.230371+0000","last_peered":"2024-09-16T15:51:18.230371+0000","last_clean":"2024-09-16T15:51:18.230371+0000","last_became_active":"2024-09-16T15:51:15.956820+0000","last_became_peered":"2024-09-16T15:51:15.956820+0000","last_unstale":"2024-09-16T15:51:18.230371+0000","last_undegraded":"2024-09-16T15:51:18.230371+0000","last_fullsized":"2024-09-16T15:51:18.230371+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:51:14.906573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:51:14.906573+0000","last_clean_scrub_stamp":"2024-09-16T15:51:14.906573+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T22:53:11.638024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.d","version":"0'0","reported_seq":13,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-16T15:51:17.993166+0000","last_change":"2024-09-16T15:51:17.003393+0000","last_active":"2024-09-16T15:51:17.993166+0000","last_peered":"2024-09-16T15:51:17.993166+0000","last_clean":"2024-09-16T15:51:17.993166+0000","last_became_active":"2024-09-16T15:51:17.002981+0000","last_became_peered":"2024-09-16T15:51:17.002981+0000","last_unstale":"2024-09-16T15:51:17.993166+0000","last_undegraded":"2024-09-16T15:51:17.993166+0000","last_fullsized":"2024-09-16T15:51:17.993166+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:51:15.917022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:51:15.917022+0000","last_clean_scrub_stamp":"2024-09-16T15:51:15.917022+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:03:12.833541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"24.c","version":"0'0","reported_seq":15,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-16T15:51:17.991347+0000","last_change":"2024-09-16T15:51:15.957428+0000","last_active":"2024-09-16T15:51:17.991347+0000","last_peered":"2024-09-16T15:51:17.991347+0000","last_clean":"2024-09-16T15:51:17.991347+0000","last_became_active":"2024-09-16T15:51:15.956667+0000","last_became_peered":"2024-09-16T15:51:15.956667+0000","last_unstale":"2024-09-16T15:51:17.991347+0000","last_undegraded":"2024-09-16T15:51:17.991347+0000","last_fullsized":"2024-09-16T15:51:17.991347+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:51:14.906573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:51:14.906573+0000","last_clean_scrub_stamp":"2024-09-16T15:51:14.906573+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T02:23:47.457845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"25.a","version":"0'0","reported_seq":13,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-16T15:51:17.999752+0000","last_change":"2024-09-16T15:51:17.010847+0000","last_active":"2024-09-16T15:51:17.999752+0000","last_peered":"2024-09-16T15:51:17.999752+0000","last_clean":"2024-09-16T15:51:17.999752+0000","last_became_active":"2024-09-16T15:51:17.010202+0000","last_became_peered":"2024-09-16T15:51:17.010202+0000","last_unstale":"2024-09-16T15:51:17.999752+0000","last_undegraded":"2024-09-16T15:51:17.999752+0000","last_fullsized":"2024-09-16T15:51:17.999752+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:51:15.917022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:51:15.917022+0000","last_clean_scrub_stamp":"2024-09-16T15:51:15.917022+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T22:56:13.989775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.b","version":"156'2","reported_seq":17,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-16T15:51:18.005791+0000","last_change":"2024-09-16T15:51:15.941149+0000","last_active":"2024-09-16T15:51:18.005791+0000","last_peered":"2024-09-16T15:51:18.005791+0000","last_clean":"2024-09-16T15:51:18.005791+0000","last_became_active":"2024-09-16T15:51:15.940519+0000","last_became_peered":"2024-09-16T15:51:15.940519+0000","last_unstale":"2024-09-16T15:51:18.005791+0000","last_undegraded":"2024-09-16T15:51:18.005791+0000","last_fullsized":"2024-09-16T15:51:18.005791+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:51:14.906573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:51:14.906573+0000","last_clean_scrub_stamp":"2024-09-16T15:51:14.906573+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T22:59:39.302588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.b","version":"0'0","reported_seq":13,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-16T15:51:17.991435+0000","last_change":"2024-09-16T15:51:16.999572+0000","last_active":"2024-09-16T15:51:17.991435+0000","last_peered":"2024-09-16T15:51:17.991435+0000","last_clean":"2024-09-16T15:51:17.991435+0000","last_became_active":"2024-09-16T15:51:16.999274+0000","last_became_peered":"2024-09-16T15:51:16.999274+0000","last_unstale":"2024-09-16T15:51:17.991435+0000","last_undegraded":"2024-09-16T15:51:17.991435+0000","last_fullsized":"2024-09-16T15:51:17.991435+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:51:15.917022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:51:15.917022+0000","last_clean_scrub_stamp":"2024-09-16T15:51:15.917022+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T23:27:15.577064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.a","version":"0'0","reported_seq":15,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-16T15:51:18.229244+0000","last_change":"2024-09-16T15:51:15.960290+0000","last_active":"2024-09-16T15:51:18.229244+0000","last_peered":"2024-09-16T15:51:18.229244+0000","last_clean":"2024-09-16T15:51:18.229244+0000","last_became_active":"2024-09-16T15:51:15.956654+0000","last_became_peered":"2024-09-16T15:51:15.956654+0000","last_unstale":"2024-09-16T15:51:18.229244+0000","last_undegraded":"2024-09-16T15:51:18.229244+0000","last_fullsized":"2024-09-16T15:51:18.229244+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:51:14.906573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:51:14.906573+0000","last_clean_scrub_stamp":"2024-09-16T15:51:14.906573+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:04:27.492103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.8","version":"0'0","reported_seq":13,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-16T15:51:17.991965+0000","last_change":"2024-09-16T15:51:17.007037+0000","last_active":"2024-09-16T15:51:17.991965+0000","last_peered":"2024-09-16T15:51:17.991965+0000","last_clean":"2024-09-16T15:51:17.991965+0000","last_became_active":"2024-09-16T15:51:17.006610+0000","last_became_peered":"2024-09-16T15:51:17.006610+0000","last_unstale":"2024-09-16T15:51:17.991965+0000","last_undegraded":"2024-09-16T15:51:17.991965+0000","last_fullsized":"2024-09-16T15:51:17.991965+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:51:15.917022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:51:15.917022+0000","last_clean_scrub_stamp":"2024-09-16T15:51:15.917022+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:12:40.001066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"24.9","version":"0'0","reported_seq":15,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-16T15:51:18.006103+0000","last_change":"2024-09-16T15:51:15.940838+0000","last_active":"2024-09-16T15:51:18.006103+0000","last_peered":"2024-09-16T15:51:18.006103+0000","last_clean":"2024-09-16T15:51:18.006103+0000","last_became_active":"2024-09-16T15:51:15.940426+0000","last_became_peered":"2024-09-16T15:51:15.940426+0000","last_unstale":"2024-09-16T15:51:18.006103+0000","last_undegraded":"2024-09-16T15:51:18.006103+0000","last_fullsized":"2024-09-16T15:51:18.006103+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:51:14.906573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:51:14.906573+0000","last_clean_scrub_stamp":"2024-09-16T15:51:14.906573+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T18:15:27.317220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.9","version":"0'0","reported_seq":13,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-16T15:51:18.229429+0000","last_change":"2024-09-16T15:51:16.993404+0000","last_active":"2024-09-16T15:51:18.229429+0000","last_peered":"2024-09-16T15:51:18.229429+0000","last_clean":"2024-09-16T15:51:18.229429+0000","last_became_active":"2024-09-16T15:51:16.992924+0000","last_became_peered":"2024-09-16T15:51:16.992924+0000","last_unstale":"2024-09-16T15:51:18.229429+0000","last_undegraded":"2024-09-16T15:51:18.229429+0000","last_fullsized":"2024-09-16T15:51:18.229429+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:51:15.917022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:51:15.917022+0000","last_clean_scrub_stamp":"2024-09-16T15:51:15.917022+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:43:58.727551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"24.8","version":"0'0","reported_seq":15,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-16T15:51:18.229067+0000","last_change":"2024-09-16T15:51:15.955543+0000","last_active":"2024-09-16T15:51:18.229067+0000","last_peered":"2024-09-16T15:51:18.229067+0000","last_clean":"2024-09-16T15:51:18.229067+0000","last_became_active":"2024-09-16T15:51:15.954545+0000","last_became_peered":"2024-09-16T15:51:15.954545+0000","last_unstale":"2024-09-16T15:51:18.229067+0000","last_undegraded":"2024-09-16T15:51:18.229067+0000","last_fullsized":"2024-09-16T15:51:18.229067+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:51:14.906573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:51:14.906573+0000","last_clean_scrub_stamp":"2024-09-16T15:51:14.906573+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T02:21:25.208264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"25.6","version":"0'0","reported_seq":13,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-16T15:51:18.230000+0000","last_change":"2024-09-16T15:51:16.992407+0000","last_active":"2024-09-16T15:51:18.230000+0000","last_peered":"2024-09-16T15:51:18.230000+0000","last_clean":"2024-09-16T15:51:18.230000+0000","last_became_active":"2024-09-16T15:51:16.991864+0000","last_became_peered":"2024-09-16T15:51:16.991864+0000","last_unstale":"2024-09-16T15:51:18.230000+0000","last_undegraded":"2024-09-16T15:51:18.230000+0000","last_fullsized":"2024-09-16T15:51:18.230000+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:51:15.917022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:51:15.917022+0000","last_clean_scrub_stamp":"2024-09-16T15:51:15.917022+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T01:09:46.564464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"24.7","version":"156'2","reported_seq":17,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-16T15:51:18.230451+0000","last_change":"2024-09-16T15:51:15.950341+0000","last_active":"2024-09-16T15:51:18.230451+0000","last_peered":"2024-09-16T15:51:18.230451+0000","last_clean":"2024-09-16T15:51:18.230451+0000","last_became_active":"2024-09-16T15:51:15.949020+0000","last_became_peered":"2024-09-16T15:51:15.949020+0000","last_unstale":"2024-09-16T15:51:18.230451+0000","last_undegraded":"2024-09-16T15:51:18.230451+0000","last_fullsized":"2024-09-16T15:51:18.230451+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:51:14.906573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:51:14.906573+0000","last_clean_scrub_stamp":"2024-09-16T15:51:14.906573+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:27:27.203647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.7","version":"0'0","reported_seq":13,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-16T15:51:18.000972+0000","last_change":"2024-09-16T15:51:17.011690+0000","last_active":"2024-09-16T15:51:18.000972+0000","last_peered":"2024-09-16T15:51:18.000972+0000","last_clean":"2024-09-16T15:51:18.000972+0000","last_became_active":"2024-09-16T15:51:17.010584+0000","last_became_peered":"2024-09-16T15:51:17.010584+0000","last_unstale":"2024-09-16T15:51:18.000972+0000","last_undegraded":"2024-09-16T15:51:18.000972+0000","last_fullsized":"2024-09-16T15:51:18.000972+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:51:15.917022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:51:15.917022+0000","last_clean_scrub_stamp":"2024-09-16T15:51:15.917022+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T01:59:38.918542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.6","version":"157'1","reported_seq":16,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-16T15:51:18.001778+0000","last_change":"2024-09-16T15:51:15.952456+0000","last_active":"2024-09-16T15:51:18.001778+0000","last_peered":"2024-09-16T15:51:18.001778+0000","last_clean":"2024-09-16T15:51:18.001778+0000","last_became_active":"2024-09-16T15:51:15.950670+0000","last_became_peered":"2024-09-16T15:51:15.950670+0000","last_unstale":"2024-09-16T15:51:18.001778+0000","last_undegraded":"2024-09-16T15:51:18.001778+0000","last_fullsized":"2024-09-16T15:51:18.001778+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:51:14.906573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:51:14.906573+0000","last_clean_scrub_stamp":"2024-09-16T15:51:14.906573+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:31:33.177323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"25.4","version":"0'0","reported_seq":13,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-16T15:51:17.993228+0000","last_change":"2024-09-16T15:51:16.999730+0000","last_active":"2024-09-16T15:51:17.993228+0000","last_peered":"2024-09-16T15:51:17.993228+0000","last_clean":"2024-09-16T15:51:17.993228+0000","last_became_active":"2024-09-16T15:51:16.998396+0000","last_became_peered":"2024-09-16T15:51:16.998396+0000","last_unstale":"2024-09-16T15:51:17.993228+0000","last_undegraded":"2024-09-16T15:51:17.993228+0000","last_fullsized":"2024-09-16T15:51:17.993228+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:51:15.917022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:51:15.917022+0000","last_clean_scrub_stamp":"2024-09-16T15:51:15.917022+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T18:13:12.440858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.5","version":"0'0","reported_seq":15,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-16T15:51:18.000330+0000","last_change":"2024-09-16T15:51:15.953202+0000","last_active":"2024-09-16T15:51:18.000330+0000","last_peered":"2024-09-16T15:51:18.000330+0000","last_clean":"2024-09-16T15:51:18.000330+0000","last_became_active":"2024-09-16T15:51:15.952826+0000","last_became_peered":"2024-09-16T15:51:15.952826+0000","last_unstale":"2024-09-16T15:51:18.000330+0000","last_undegraded":"2024-09-16T15:51:18.000330+0000","last_fullsized":"2024-09-16T15:51:18.000330+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:51:14.906573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:51:14.906573+0000","last_clean_scrub_stamp":"2024-09-16T15:51:14.906573+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T03:12:29.684040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"25.5","version":"0'0","reported_seq":13,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-16T15:51:17.992473+0000","last_change":"2024-09-16T15:51:17.004148+0000","last_active":"2024-09-16T15:51:17.992473+0000","last_peered":"2024-09-16T15:51:17.992473+0000","last_clean":"2024-09-16T15:51:17.992473+0000","last_became_active":"2024-09-16T15:51:17.003598+0000","last_became_peered":"2024-09-16T15:51:17.003598+0000","last_unstale":"2024-09-16T15:51:17.992473+0000","last_undegraded":"2024-09-16T15:51:17.992473+0000","last_fullsized":"2024-09-16T15:51:17.992473+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:51:15.917022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:51:15.917022+0000","last_clean_scrub_stamp":"2024-09-16T15:51:15.917022+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:12:52.408625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.4","version":"0'0","reported_seq":15,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-16T15:51:17.993301+0000","last_change":"2024-09-16T15:51:15.947156+0000","last_active":"2024-09-16T15:51:17.993301+0000","last_peered":"2024-09-16T15:51:17.993301+0000","last_clean":"2024-09-16T15:51:17.993301+0000","last_became_active":"2024-09-16T15:51:15.946443+0000","last_became_peered":"2024-09-16T15:51:15.946443+0000","last_unstale":"2024-09-16T15:51:17.993301+0000","last_undegraded":"2024-09-16T15:51:17.993301+0000","last_fullsized":"2024-09-16T15:51:17.993301+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:51:14.906573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:51:14.906573+0000","last_clean_scrub_stamp":"2024-09-16T15:51:14.906573+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T00:44:23.785120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.2","version":"0'0","reported_seq":13,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-16T15:51:18.230924+0000","last_change":"2024-09-16T15:51:17.003815+0000","last_active":"2024-09-16T15:51:18.230924+0000","last_peered":"2024-09-16T15:51:18.230924+0000","last_clean":"2024-09-16T15:51:18.230924+0000","last_became_active":"2024-09-16T15:51:17.003331+0000","last_became_peered":"2024-09-16T15:51:17.003331+0000","last_unstale":"2024-09-16T15:51:18.230924+0000","last_undegraded":"2024-09-16T15:51:18.230924+0000","last_fullsized":"2024-09-16T15:51:18.230924+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:51:15.917022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:51:15.917022+0000","last_clean_scrub_stamp":"2024-09-16T15:51:15.917022+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T00:59:28.722029+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"24.3","version":"0'0","reported_seq":15,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-16T15:51:18.231360+0000","last_change":"2024-09-16T15:51:15.962250+0000","last_active":"2024-09-16T15:51:18.231360+0000","last_peered":"2024-09-16T15:51:18.231360+0000","last_clean":"2024-09-16T15:51:18.231360+0000","last_became_active":"2024-09-16T15:51:15.961426+0000","last_became_peered":"2024-09-16T15:51:15.961426+0000","last_unstale":"2024-09-16T15:51:18.231360+0000","last_undegraded":"2024-09-16T15:51:18.231360+0000","last_fullsized":"2024-09-16T15:51:18.231360+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:51:14.906573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:51:14.906573+0000","last_clean_scrub_stamp":"2024-09-16T15:51:14.906573+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:29:50.442147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"25.3","version":"0'0","reported_seq":13,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-16T15:51:18.230136+0000","last_change":"2024-09-16T15:51:17.000638+0000","last_active":"2024-09-16T15:51:18.230136+0000","last_peered":"2024-09-16T15:51:18.230136+0000","last_clean":"2024-09-16T15:51:18.230136+0000","last_became_active":"2024-09-16T15:51:16.999924+0000","last_became_peered":"2024-09-16T15:51:16.999924+0000","last_unstale":"2024-09-16T15:51:18.230136+0000","last_undegraded":"2024-09-16T15:51:18.230136+0000","last_fullsized":"2024-09-16T15:51:18.230136+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:51:15.917022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:51:15.917022+0000","last_clean_scrub_stamp":"2024-09-16T15:51:15.917022+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:11:45.415023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"24.2","version":"0'0","reported_seq":15,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-16T15:51:18.230610+0000","last_change":"2024-09-16T15:51:15.953228+0000","last_active":"2024-09-16T15:51:18.230610+0000","last_peered":"2024-09-16T15:51:18.230610+0000","last_clean":"2024-09-16T15:51:18.230610+0000","last_became_active":"2024-09-16T15:51:15.952533+0000","last_became_peered":"2024-09-16T15:51:15.952533+0000","last_unstale":"2024-09-16T15:51:18.230610+0000","last_undegraded":"2024-09-16T15:51:18.230610+0000","last_fullsized":"2024-09-16T15:51:18.230610+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:51:14.906573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:51:14.906573+0000","last_clean_scrub_stamp":"2024-09-16T15:51:14.906573+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:43:57.717101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"25.0","version":"0'0","reported_seq":13,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-16T15:51:17.992082+0000","last_change":"2024-09-16T15:51:16.977593+0000","last_active":"2024-09-16T15:51:17.992082+0000","last_peered":"2024-09-16T15:51:17.992082+0000","last_clean":"2024-09-16T15:51:17.992082+0000","last_became_active":"2024-09-16T15:51:16.976791+0000","last_became_peered":"2024-09-16T15:51:16.976791+0000","last_unstale":"2024-09-16T15:51:17.992082+0000","last_undegraded":"2024-09-16T15:51:17.992082+0000","last_fullsized":"2024-09-16T15:51:17.992082+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:51:15.917022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:51:15.917022+0000","last_clean_scrub_stamp":"2024-09-16T15:51:15.917022+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T02:45:37.431644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"24.1","version":"0'0","reported_seq":15,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-16T15:51:17.992304+0000","last_change":"2024-09-16T15:51:15.965947+0000","last_active":"2024-09-16T15:51:17.992304+0000","last_peered":"2024-09-16T15:51:17.992304+0000","last_clean":"2024-09-16T15:51:17.992304+0000","last_became_active":"2024-09-16T15:51:15.965027+0000","last_became_peered":"2024-09-16T15:51:15.965027+0000","last_unstale":"2024-09-16T15:51:17.992304+0000","last_undegraded":"2024-09-16T15:51:17.992304+0000","last_fullsized":"2024-09-16T15:51:17.992304+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:51:14.906573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:51:14.906573+0000","last_clean_scrub_stamp":"2024-09-16T15:51:14.906573+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T18:06:03.179201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"25.1","version":"0'0","reported_seq":13,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-16T15:51:18.229551+0000","last_change":"2024-09-16T15:51:17.000624+0000","last_active":"2024-09-16T15:51:18.229551+0000","last_peered":"2024-09-16T15:51:18.229551+0000","last_clean":"2024-09-16T15:51:18.229551+0000","last_became_active":"2024-09-16T15:51:17.000331+0000","last_became_peered":"2024-09-16T15:51:17.000331+0000","last_unstale":"2024-09-16T15:51:18.229551+0000","last_undegraded":"2024-09-16T15:51:18.229551+0000","last_fullsized":"2024-09-16T15:51:18.229551+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:51:15.917022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:51:15.917022+0000","last_clean_scrub_stamp":"2024-09-16T15:51:15.917022+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:46:14.133192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"24.0","version":"0'0","reported_seq":15,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-16T15:51:18.005669+0000","last_change":"2024-09-16T15:51:15.940537+0000","last_active":"2024-09-16T15:51:18.005669+0000","last_peered":"2024-09-16T15:51:18.005669+0000","last_clean":"2024-09-16T15:51:18.005669+0000","last_became_active":"2024-09-16T15:51:15.940171+0000","last_became_peered":"2024-09-16T15:51:15.940171+0000","last_unstale":"2024-09-16T15:51:18.005669+0000","last_undegraded":"2024-09-16T15:51:18.005669+0000","last_fullsized":"2024-09-16T15:51:18.005669+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:51:14.906573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:51:14.906573+0000","last_clean_scrub_stamp":"2024-09-16T15:51:14.906573+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:32:44.137745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.18","version":"0'0","reported_seq":13,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-16T15:51:17.999100+0000","last_change":"2024-09-16T15:51:16.997446+0000","last_active":"2024-09-16T15:51:17.999100+0000","last_peered":"2024-09-16T15:51:17.999100+0000","last_clean":"2024-09-16T15:51:17.999100+0000","last_became_active":"2024-09-16T15:51:16.997007+0000","last_became_peered":"2024-09-16T15:51:16.997007+0000","last_unstale":"2024-09-16T15:51:17.999100+0000","last_undegraded":"2024-09-16T15:51:17.999100+0000","last_fullsized":"2024-09-16T15:51:17.999100+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:51:15.917022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:51:15.917022+0000","last_clean_scrub_stamp":"2024-09-16T15:51:15.917022+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:46:27.292030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.19","version":"0'0","reported_seq":15,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-16T15:51:18.229525+0000","last_change":"2024-09-16T15:51:15.961809+0000","last_active":"2024-09-16T15:51:18.229525+0000","last_peered":"2024-09-16T15:51:18.229525+0000","last_clean":"2024-09-16T15:51:18.229525+0000","last_became_active":"2024-09-16T15:51:15.961107+0000","last_became_peered":"2024-09-16T15:51:15.961107+0000","last_unstale":"2024-09-16T15:51:18.229525+0000","last_undegraded":"2024-09-16T15:51:18.229525+0000","last_fullsized":"2024-09-16T15:51:18.229525+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":154,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:51:14.906573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:51:14.906573+0000","last_clean_scrub_stamp":"2024-09-16T15:51:14.906573+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T22:10:43.042714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"73'79","reported_seq":391,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-16T15:51:18.229429+0000","last_change":"2024-09-16T15:42:40.881820+0000","last_active":"2024-09-16T15:51:18.229429+0000","last_peered":"2024-09-16T15:51:18.229429+0000","last_clean":"2024-09-16T15:51:18.229429+0000","last_became_active":"2024-09-16T15:42:40.879965+0000","last_became_peered":"2024-09-16T15:42:40.879965+0000","last_unstale":"2024-09-16T15:51:18.229429+0000","last_undegraded":"2024-09-16T15:51:18.229429+0000","last_fullsized":"2024-09-16T15:51:18.229429+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:42:39.728762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:42:39.728762+0000","last_clean_scrub_stamp":"2024-09-16T15:42:39.728762+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:22:18.768932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":25,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":24,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79,"ondisk_log_size":79,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542252,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33708,"kb_used_data":2516,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338132,"statfs":{"total":96636764160,"available":96602247168,"internally_reserved":0,"allocated":2576384,"data_stored":1465866,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542251,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33120,"kb_used_data":1928,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338720,"statfs":{"total":96636764160,"available":96602849280,"internally_reserved":0,"allocated":1974272,"data_stored":875394,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":5,"up_from":14,"seq":60129542252,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33116,"kb_used_data":1928,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338724,"statfs":{"total":96636764160,"available":96602853376,"internally_reserved":0,"allocated":1974272,"data_stored":875394,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542250,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33136,"kb_used_data":1944,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338704,"statfs":{"total":96636764160,"available":96602832896,"internally_reserved":0,"allocated":1990656,"data_stored":877588,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":3,"up_from":14,"seq":60129542251,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37288,"kb_used_data":1940,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334552,"statfs":{"total":96636764160,"available":96598581248,"internally_reserved":0,"allocated":1986560,"data_stored":875588,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":2,"up_from":14,"seq":60129542251,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33128,"kb_used_data":1940,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338712,"statfs":{"total":96636764160,"available":96602841088,"internally_reserved":0,"allocated":1986560,"data_stored":876088,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":14,"seq":60129542253,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33128,"kb_used_data":1940,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338712,"statfs":{"total":96636764160,"available":96602841088,"internally_reserved":0,"allocated":1986560,"data_stored":877006,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":14,"seq":60129542253,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33708,"kb_used_data":2512,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338132,"statfs":{"total":96636764160,"available":96602247168,"internally_reserved":0,"allocated":2572288,"data_stored":1465784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":194,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2194,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T15:51:21.381 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-09-16T15:51:21.381 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-09-16T15:51:21.554 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:21.554+0000 7fc58e6f0640 1 -- 172.21.15.123:0/271329672 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc570004810 msgr2=0x7fc570004c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:51:21.554 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:21.554+0000 7fc58e6f0640 1 --2- 172.21.15.123:0/271329672 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc570004810 0x7fc570004c90 secure :-1 s=READY pgs=1751 cs=0 l=1 rev1=1 crypto rx=0x7fc578005fd0 tx=0x7fc57801cd60 comp rx=0 tx=0).stop 2024-09-16T15:51:21.554 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:21.554+0000 7fc58e6f0640 1 -- 172.21.15.123:0/271329672 shutdown_connections 2024-09-16T15:51:21.554 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:21.554+0000 7fc58e6f0640 1 --2- 172.21.15.123:0/271329672 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc570004810 0x7fc570004c90 unknown :-1 s=CLOSED pgs=1751 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:21.554 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:21.554+0000 7fc58e6f0640 1 -- 172.21.15.123:0/271329672 >> 172.21.15.123:0/271329672 conn(0x7fc5880b1710 msgr2=0x7fc5880b1b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:51:21.554 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:21.554+0000 7fc58e6f0640 1 -- 172.21.15.123:0/271329672 shutdown_connections 2024-09-16T15:51:21.555 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:21.554+0000 7fc58e6f0640 1 -- 172.21.15.123:0/271329672 wait complete. 2024-09-16T15:51:21.555 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:21.558+0000 7fc58e6f0640 1 Processor -- start 2024-09-16T15:51:21.555 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:21.558+0000 7fc58e6f0640 1 -- start start 2024-09-16T15:51:21.557 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:21.558+0000 7fc58e6f0640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fc570004810 0x7fc58815ba80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:51:21.557 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:21.558+0000 7fc58e6f0640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc5881f7f70 0x7fc5881ee6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:51:21.557 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:21.558+0000 7fc58e6f0640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc5880ad8c0 0x7fc5881f4f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:51:21.557 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:21.558+0000 7fc58e6f0640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fc58815e960 con 0x7fc5880ad8c0 2024-09-16T15:51:21.558 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:21.558+0000 7fc58e6f0640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fc58815e7e0 con 0x7fc5881f7f70 2024-09-16T15:51:21.558 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:21.558+0000 7fc58e6f0640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fc5880cabc0 con 0x7fc570004810 2024-09-16T15:51:21.558 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:21.558+0000 7fc58cc8d640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc5880ad8c0 0x7fc5881f4f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:51:21.558 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:21.558+0000 7fc58cc8d640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc5880ad8c0 0x7fc5881f4f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:45190/0 (socket says 172.21.15.123:45190) 2024-09-16T15:51:21.558 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:21.558+0000 7fc58cc8d640 1 -- 172.21.15.123:0/1024408475 learned_addr learned my addr 172.21.15.123:0/1024408475 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:51:21.558 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:21.558+0000 7fc5877fe640 1 --2- 172.21.15.123:0/1024408475 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc5881f7f70 0x7fc5881ee6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:51:21.558 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:21.558+0000 7fc587fff640 1 --2- 172.21.15.123:0/1024408475 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fc570004810 0x7fc58815ba80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:51:21.558 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:21.558+0000 7fc58cc8d640 1 -- 172.21.15.123:0/1024408475 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fc570004810 msgr2=0x7fc58815ba80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:51:21.558 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:21.558+0000 7fc58cc8d640 1 --2- 172.21.15.123:0/1024408475 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fc570004810 0x7fc58815ba80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:21.558 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:21.558+0000 7fc58cc8d640 1 -- 172.21.15.123:0/1024408475 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc5881f7f70 msgr2=0x7fc5881ee6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:51:21.558 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:21.558+0000 7fc58cc8d640 1 --2- 172.21.15.123:0/1024408475 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc5881f7f70 0x7fc5881ee6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:21.558 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:21.558+0000 7fc58cc8d640 1 -- 172.21.15.123:0/1024408475 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc5881f4a90 con 0x7fc5880ad8c0 2024-09-16T15:51:21.558 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:21.558+0000 7fc5877fe640 1 --2- 172.21.15.123:0/1024408475 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc5881f7f70 0x7fc5881ee6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T15:51:21.558 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:21.558+0000 7fc587fff640 1 --2- 172.21.15.123:0/1024408475 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fc570004810 0x7fc58815ba80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T15:51:21.559 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:21.558+0000 7fc58cc8d640 1 --2- 172.21.15.123:0/1024408475 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc5880ad8c0 0x7fc5881f4f40 secure :-1 s=READY pgs=1752 cs=0 l=1 rev1=1 crypto rx=0x7fc5880702a0 tx=0x7fc58000fb90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:51:21.559 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:21.558+0000 7fc584ff9640 1 -- 172.21.15.123:0/1024408475 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc58001a070 con 0x7fc5880ad8c0 2024-09-16T15:51:21.559 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:21.558+0000 7fc584ff9640 1 -- 172.21.15.123:0/1024408475 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fc580006db0 con 0x7fc5880ad8c0 2024-09-16T15:51:21.559 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:21.558+0000 7fc58e6f0640 1 -- 172.21.15.123:0/1024408475 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc5881f84c0 con 0x7fc5880ad8c0 2024-09-16T15:51:21.559 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:21.558+0000 7fc584ff9640 1 -- 172.21.15.123:0/1024408475 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc580008330 con 0x7fc5880ad8c0 2024-09-16T15:51:21.559 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:21.558+0000 7fc58e6f0640 1 -- 172.21.15.123:0/1024408475 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc5881f8a80 con 0x7fc5880ad8c0 2024-09-16T15:51:21.560 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:21.558+0000 7fc58e6f0640 1 -- 172.21.15.123:0/1024408475 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc5880cc3c0 con 0x7fc5880ad8c0 2024-09-16T15:51:21.560 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:21.558+0000 7fc584ff9640 1 -- 172.21.15.123:0/1024408475 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 24) ==== 97845+0+0 (secure 0 0 0) 0x7fc58001d070 con 0x7fc5880ad8c0 2024-09-16T15:51:21.560 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:21.562+0000 7fc584ff9640 1 --2- 172.21.15.123:0/1024408475 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fc574073760 0x7fc574075c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:51:21.560 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:21.562+0000 7fc587fff640 1 --2- 172.21.15.123:0/1024408475 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fc574073760 0x7fc574075c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:51:21.560 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:21.562+0000 7fc584ff9640 1 -- 172.21.15.123:0/1024408475 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(157..157 src has 1..157) ==== 5737+0+0 (secure 0 0 0) 0x7fc5800948f0 con 0x7fc5880ad8c0 2024-09-16T15:51:21.560 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:21.562+0000 7fc587fff640 1 --2- 172.21.15.123:0/1024408475 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fc574073760 0x7fc574075c20 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7fc57c000c10 tx=0x7fc57c006920 comp rx=0 tx=0).ready entity=mgr.4150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:51:21.563 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:21.566+0000 7fc584ff9640 1 -- 172.21.15.123:0/1024408475 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7fc5800615f0 con 0x7fc5880ad8c0 2024-09-16T15:51:21.802 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:21.802+0000 7fc58e6f0640 1 -- 172.21.15.123:0/1024408475 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) -- 0x7fc5880ca9a0 con 0x7fc5880ad8c0 2024-09-16T15:51:21.806 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:21.806+0000 7fc584ff9640 1 -- 172.21.15.123:0/1024408475 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v29) ==== 190+0+147 (secure 0 0 0) 0x7fc580060ef0 con 0x7fc5880ad8c0 2024-09-16T15:51:21.806 INFO:teuthology.orchestra.run.smithi123.stdout:[client.0] 2024-09-16T15:51:21.807 INFO:teuthology.orchestra.run.smithi123.stdout: key = AQC1Uehm7GDTAhAAHWWVIyjVVnTeeozBR8DfFA== 2024-09-16T15:51:21.807 INFO:teuthology.orchestra.run.smithi123.stdout: caps mds = "allow" 2024-09-16T15:51:21.807 INFO:teuthology.orchestra.run.smithi123.stdout: caps mon = "allow r" 2024-09-16T15:51:21.807 INFO:teuthology.orchestra.run.smithi123.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-09-16T15:51:21.807 INFO:teuthology.orchestra.run.smithi123.stderr:updated caps for client.0 2024-09-16T15:51:21.809 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:21.810+0000 7fc58e6f0640 1 -- 172.21.15.123:0/1024408475 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fc574073760 msgr2=0x7fc574075c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:51:21.809 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:21.810+0000 7fc58e6f0640 1 --2- 172.21.15.123:0/1024408475 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fc574073760 0x7fc574075c20 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7fc57c000c10 tx=0x7fc57c006920 comp rx=0 tx=0).stop 2024-09-16T15:51:21.810 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:21.810+0000 7fc58e6f0640 1 -- 172.21.15.123:0/1024408475 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc5880ad8c0 msgr2=0x7fc5881f4f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:51:21.810 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:21.810+0000 7fc58e6f0640 1 --2- 172.21.15.123:0/1024408475 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc5880ad8c0 0x7fc5881f4f40 secure :-1 s=READY pgs=1752 cs=0 l=1 rev1=1 crypto rx=0x7fc5880702a0 tx=0x7fc58000fb90 comp rx=0 tx=0).stop 2024-09-16T15:51:21.810 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:21.810+0000 7fc58e6f0640 1 -- 172.21.15.123:0/1024408475 shutdown_connections 2024-09-16T15:51:21.810 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:21.810+0000 7fc58e6f0640 1 --2- 172.21.15.123:0/1024408475 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fc574073760 0x7fc574075c20 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:21.810 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:21.810+0000 7fc58e6f0640 1 --2- 172.21.15.123:0/1024408475 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc5880ad8c0 0x7fc5881f4f40 unknown :-1 s=CLOSED pgs=1752 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:21.810 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:21.810+0000 7fc58e6f0640 1 --2- 172.21.15.123:0/1024408475 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc5881f7f70 0x7fc5881ee6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:21.810 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:21.810+0000 7fc58e6f0640 1 --2- 172.21.15.123:0/1024408475 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fc570004810 0x7fc58815ba80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:21.810 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:21.810+0000 7fc58e6f0640 1 -- 172.21.15.123:0/1024408475 >> 172.21.15.123:0/1024408475 conn(0x7fc5880b1710 msgr2=0x7fc588159910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:51:21.810 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:21.810+0000 7fc58e6f0640 1 -- 172.21.15.123:0/1024408475 shutdown_connections 2024-09-16T15:51:21.810 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:21.810+0000 7fc58e6f0640 1 -- 172.21.15.123:0/1024408475 wait complete. 2024-09-16T15:51:21.825 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-16T15:51:21.939 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:21.938+0000 7f8ccdcc0640 1 -- 172.21.15.123:0/123263301 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8cc815e820 msgr2=0x7f8cc815ec90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:51:21.939 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:21.938+0000 7f8ccdcc0640 1 --2- 172.21.15.123:0/123263301 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8cc815e820 0x7f8cc815ec90 secure :-1 s=READY pgs=1753 cs=0 l=1 rev1=1 crypto rx=0x7f8cc80cf1f0 tx=0x7f8cc001c830 comp rx=0 tx=0).stop 2024-09-16T15:51:21.939 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:21.938+0000 7f8ccdcc0640 1 -- 172.21.15.123:0/123263301 shutdown_connections 2024-09-16T15:51:21.939 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:21.938+0000 7f8ccdcc0640 1 --2- 172.21.15.123:0/123263301 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8cc815e820 0x7f8cc815ec90 unknown :-1 s=CLOSED pgs=1753 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:21.939 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:21.938+0000 7f8ccdcc0640 1 --2- 172.21.15.123:0/123263301 >> v2:172.21.15.184:3300/0 conn(0x7f8cc80cd6e0 0x7f8cc80cdb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:21.939 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:21.938+0000 7f8ccdcc0640 1 -- 172.21.15.123:0/123263301 >> 172.21.15.123:0/123263301 conn(0x7f8cc80cfb10 msgr2=0x7f8cc80cff20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:51:21.939 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:21.938+0000 7f8ccdcc0640 1 -- 172.21.15.123:0/123263301 shutdown_connections 2024-09-16T15:51:21.939 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:21.938+0000 7f8ccdcc0640 1 -- 172.21.15.123:0/123263301 wait complete. 2024-09-16T15:51:21.940 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:21.942+0000 7f8ccdcc0640 1 Processor -- start 2024-09-16T15:51:21.940 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:21.942+0000 7f8ccdcc0640 1 -- start start 2024-09-16T15:51:21.942 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:21.942+0000 7f8ccdcc0640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8cc80cd6e0 0x7f8cc8152b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:51:21.942 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:21.942+0000 7f8ccdcc0640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f8cc815e160 0x7f8cc8155b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:51:21.942 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:21.942+0000 7f8ccdcc0640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f8cc815e820 0x7f8cc8142800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:51:21.942 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:21.942+0000 7f8ccdcc0640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f8cc80cf5b0 con 0x7f8cc80cd6e0 2024-09-16T15:51:21.942 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:21.942+0000 7f8ccdcc0640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f8cc80cf430 con 0x7f8cc815e820 2024-09-16T15:51:21.942 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:21.942+0000 7f8ccdcc0640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f8cc81683d0 con 0x7f8cc815e160 2024-09-16T15:51:21.942 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:21.942+0000 7f8cc77fe640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8cc80cd6e0 0x7f8cc8152b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:51:21.942 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:21.942+0000 7f8cc77fe640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8cc80cd6e0 0x7f8cc8152b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:45196/0 (socket says 172.21.15.123:45196) 2024-09-16T15:51:21.942 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:21.942+0000 7f8cc77fe640 1 -- 172.21.15.123:0/3722225743 learned_addr learned my addr 172.21.15.123:0/3722225743 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:51:21.943 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:21.942+0000 7f8cc6ffd640 1 --2- 172.21.15.123:0/3722225743 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f8cc815e160 0x7f8cc8155b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:51:21.943 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:21.942+0000 7f8cc77fe640 1 -- 172.21.15.123:0/3722225743 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f8cc815e160 msgr2=0x7f8cc8155b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:51:21.943 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:21.942+0000 7f8cc77fe640 1 --2- 172.21.15.123:0/3722225743 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f8cc815e160 0x7f8cc8155b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:21.943 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:21.942+0000 7f8cc77fe640 1 -- 172.21.15.123:0/3722225743 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f8cc815e820 msgr2=0x7f8cc8142800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:51:21.943 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:21.942+0000 7f8cc77fe640 1 --2- 172.21.15.123:0/3722225743 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f8cc815e820 0x7f8cc8142800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:21.943 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:21.942+0000 7f8cc77fe640 1 -- 172.21.15.123:0/3722225743 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8cc81550d0 con 0x7f8cc80cd6e0 2024-09-16T15:51:21.943 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:21.942+0000 7f8cc77fe640 1 --2- 172.21.15.123:0/3722225743 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8cc80cd6e0 0x7f8cc8152b90 secure :-1 s=READY pgs=1754 cs=0 l=1 rev1=1 crypto rx=0x7f8cb800c7f0 tx=0x7f8cb800ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:51:21.943 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:21.942+0000 7f8caffff640 1 -- 172.21.15.123:0/3722225743 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8cb8016020 con 0x7f8cc80cd6e0 2024-09-16T15:51:21.943 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:21.942+0000 7f8caffff640 1 -- 172.21.15.123:0/3722225743 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f8cb800ab40 con 0x7f8cc80cd6e0 2024-09-16T15:51:21.943 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:21.942+0000 7f8ccdcc0640 1 -- 172.21.15.123:0/3722225743 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8cc81553b0 con 0x7f8cc80cd6e0 2024-09-16T15:51:21.943 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:21.942+0000 7f8caffff640 1 -- 172.21.15.123:0/3722225743 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8cb8012420 con 0x7f8cc80cd6e0 2024-09-16T15:51:21.943 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:21.942+0000 7f8ccdcc0640 1 -- 172.21.15.123:0/3722225743 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8cc81530d0 con 0x7f8cc80cd6e0 2024-09-16T15:51:21.944 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:21.942+0000 7f8ccdcc0640 1 -- 172.21.15.123:0/3722225743 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8cc80cdb50 con 0x7f8cc80cd6e0 2024-09-16T15:51:21.944 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:21.946+0000 7f8caffff640 1 -- 172.21.15.123:0/3722225743 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 24) ==== 97845+0+0 (secure 0 0 0) 0x7f8cb8019020 con 0x7f8cc80cd6e0 2024-09-16T15:51:21.944 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:21.946+0000 7f8caffff640 1 --2- 172.21.15.123:0/3722225743 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f8cb4073730 0x7f8cb4075bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:51:21.944 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:21.946+0000 7f8caffff640 1 -- 172.21.15.123:0/3722225743 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(157..157 src has 1..157) ==== 5737+0+0 (secure 0 0 0) 0x7f8cb80926f0 con 0x7f8cc80cd6e0 2024-09-16T15:51:21.945 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:21.946+0000 7f8cc6ffd640 1 --2- 172.21.15.123:0/3722225743 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f8cb4073730 0x7f8cb4075bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:51:21.945 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:21.946+0000 7f8cc6ffd640 1 --2- 172.21.15.123:0/3722225743 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f8cb4073730 0x7f8cb4075bf0 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f8cc80b5a50 tx=0x7f8cbc005c30 comp rx=0 tx=0).ready entity=mgr.4150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:51:21.948 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:21.950+0000 7f8caffff640 1 -- 172.21.15.123:0/3722225743 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f8cb8020030 con 0x7f8cc80cd6e0 2024-09-16T15:51:22.150 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:22.150+0000 7f8ccdcc0640 1 -- 172.21.15.123:0/3722225743 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f8cc8155580 con 0x7f8cc80cd6e0 2024-09-16T15:51:22.151 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:22.150+0000 7f8caffff640 1 -- 172.21.15.123:0/3722225743 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 105 v105) ==== 95+0+5244 (secure 0 0 0) 0x7f8cb805f3f0 con 0x7f8cc80cd6e0 2024-09-16T15:51:22.151 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-09-16T15:51:22.151 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":105,"btime":"2024-09-16T15:51:19:998149+0000","default_fscid":12,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6200,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6835/2380355001","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6833","nonce":2380355001},{"type":"v1","addr":"172.21.15.184:6835","nonce":2380355001}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":100},{"gid":8844,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6834/1840463681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6832","nonce":1840463681},{"type":"v1","addr":"172.21.15.123:6834","nonce":1840463681}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":100},{"gid":8847,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6835/2493915107","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6833","nonce":2493915107},{"type":"v1","addr":"172.21.15.123:6835","nonce":2493915107}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":99}],"filesystems":[{"mdsmap":{"epoch":105,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-16T15:51:16.301945+0000","modified":"2024-09-16T15:51:19.374973+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8859},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8859":{"gid":8859,"name":"d","rank":0,"incarnation":101,"state":"up:active","state_seq":3,"addr":"172.21.15.184:6834/2856066034","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":2856066034},{"type":"v1","addr":"172.21.15.184:6834","nonce":2856066034}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[25],"metadata_pool":24,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8859,"qdb_cluster":[8859]},"id":12}]} 2024-09-16T15:51:22.151 INFO:teuthology.orchestra.run.smithi123.stderr:dumped fsmap epoch 105 2024-09-16T15:51:22.153 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:22.154+0000 7f8ccdcc0640 1 -- 172.21.15.123:0/3722225743 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f8cb4073730 msgr2=0x7f8cb4075bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:51:22.153 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:22.154+0000 7f8ccdcc0640 1 --2- 172.21.15.123:0/3722225743 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f8cb4073730 0x7f8cb4075bf0 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f8cc80b5a50 tx=0x7f8cbc005c30 comp rx=0 tx=0).stop 2024-09-16T15:51:22.153 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:22.154+0000 7f8ccdcc0640 1 -- 172.21.15.123:0/3722225743 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8cc80cd6e0 msgr2=0x7f8cc8152b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:51:22.154 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:22.154+0000 7f8ccdcc0640 1 --2- 172.21.15.123:0/3722225743 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8cc80cd6e0 0x7f8cc8152b90 secure :-1 s=READY pgs=1754 cs=0 l=1 rev1=1 crypto rx=0x7f8cb800c7f0 tx=0x7f8cb800ccb0 comp rx=0 tx=0).stop 2024-09-16T15:51:22.154 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:22.154+0000 7f8ccdcc0640 1 -- 172.21.15.123:0/3722225743 shutdown_connections 2024-09-16T15:51:22.154 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:22.154+0000 7f8ccdcc0640 1 --2- 172.21.15.123:0/3722225743 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f8cb4073730 0x7f8cb4075bf0 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:22.154 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:22.154+0000 7f8ccdcc0640 1 --2- 172.21.15.123:0/3722225743 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f8cc815e820 0x7f8cc8142800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:22.154 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:22.154+0000 7f8ccdcc0640 1 --2- 172.21.15.123:0/3722225743 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f8cc815e160 0x7f8cc8155b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:22.154 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:22.154+0000 7f8ccdcc0640 1 --2- 172.21.15.123:0/3722225743 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8cc80cd6e0 0x7f8cc8152b90 unknown :-1 s=CLOSED pgs=1754 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:22.154 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:22.154+0000 7f8ccdcc0640 1 -- 172.21.15.123:0/3722225743 >> 172.21.15.123:0/3722225743 conn(0x7f8cc80cfb10 msgr2=0x7f8cc80b0040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:51:22.154 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:22.154+0000 7f8ccdcc0640 1 -- 172.21.15.123:0/3722225743 shutdown_connections 2024-09-16T15:51:22.154 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:22.154+0000 7f8ccdcc0640 1 -- 172.21.15.123:0/3722225743 wait complete. 2024-09-16T15:51:22.571 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 105, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-09-16T15:51:16.301945+0000', 'modified': '2024-09-16T15:51:19.374973+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 8859}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8859': {'gid': 8859, 'name': 'd', 'rank': 0, 'incarnation': 101, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.184:6834/2856066034', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.184:6832', 'nonce': 2856066034}, {'type': 'v1', 'addr': '172.21.15.184:6834', 'nonce': 2856066034}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540719139924082687, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [25], 'metadata_pool': 24, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8859, 'qdb_cluster': [8859]} 2024-09-16T15:51:22.571 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-09-16T15:51:22.571 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.d status 2024-09-16T15:51:22.704 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:22.706+0000 7fc98e122640 1 -- 172.21.15.123:0/2991509853 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc9880cd800 msgr2=0x7fc9880cac00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:51:22.705 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:22.706+0000 7fc98e122640 1 --2- 172.21.15.123:0/2991509853 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc9880cd800 0x7fc9880cac00 secure :-1 s=READY pgs=1756 cs=0 l=1 rev1=1 crypto rx=0x7fc980009900 tx=0x7fc98001c900 comp rx=0 tx=0).stop 2024-09-16T15:51:22.705 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:22.706+0000 7fc98e122640 1 -- 172.21.15.123:0/2991509853 shutdown_connections 2024-09-16T15:51:22.705 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:22.706+0000 7fc98e122640 1 --2- 172.21.15.123:0/2991509853 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc9880cd800 0x7fc9880cac00 unknown :-1 s=CLOSED pgs=1756 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:22.705 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:22.706+0000 7fc98e122640 1 --2- 172.21.15.123:0/2991509853 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fc9880b1c50 0x7fc9880b2030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:22.705 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:22.706+0000 7fc98e122640 1 -- 172.21.15.123:0/2991509853 >> 172.21.15.123:0/2991509853 conn(0x7fc9880ccc50 msgr2=0x7fc9880cd060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:51:22.705 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:22.706+0000 7fc98e122640 1 -- 172.21.15.123:0/2991509853 shutdown_connections 2024-09-16T15:51:22.705 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:22.706+0000 7fc98e122640 1 -- 172.21.15.123:0/2991509853 wait complete. 2024-09-16T15:51:22.705 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:22.706+0000 7fc98e122640 1 Processor -- start 2024-09-16T15:51:22.706 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:22.706+0000 7fc98e122640 1 -- start start 2024-09-16T15:51:22.707 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:22.706+0000 7fc98e122640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc9880b1c50 0x7fc9881613a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:51:22.707 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:22.706+0000 7fc98e122640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fc9880cd800 0x7fc9880af190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:51:22.707 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:22.706+0000 7fc98e122640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc98815e0c0 0x7fc9881547a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:51:22.707 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:22.706+0000 7fc98e122640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fc98816ac00 con 0x7fc98815e0c0 2024-09-16T15:51:22.707 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:22.706+0000 7fc98e122640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fc9880cbef0 con 0x7fc9880b1c50 2024-09-16T15:51:22.708 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:22.706+0000 7fc98e122640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fc98816ad80 con 0x7fc9880cd800 2024-09-16T15:51:22.708 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:22.706+0000 7fc98d921640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc98815e0c0 0x7fc9881547a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:51:22.708 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:22.706+0000 7fc98d921640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc98815e0c0 0x7fc9881547a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:57610/0 (socket says 172.21.15.123:57610) 2024-09-16T15:51:22.708 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:22.706+0000 7fc98d921640 1 -- 172.21.15.123:0/78263397 learned_addr learned my addr 172.21.15.123:0/78263397 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:51:22.708 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:22.706+0000 7fc98c91f640 1 --2- 172.21.15.123:0/78263397 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fc9880cd800 0x7fc9880af190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:51:22.708 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:22.706+0000 7fc98d921640 1 -- 172.21.15.123:0/78263397 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fc9880cd800 msgr2=0x7fc9880af190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:51:22.708 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:22.706+0000 7fc98d921640 1 --2- 172.21.15.123:0/78263397 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fc9880cd800 0x7fc9880af190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:22.708 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:22.706+0000 7fc98d921640 1 -- 172.21.15.123:0/78263397 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc9880b1c50 msgr2=0x7fc9881613a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:51:22.708 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:22.706+0000 7fc98d921640 1 --2- 172.21.15.123:0/78263397 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc9880b1c50 0x7fc9881613a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:22.708 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:22.706+0000 7fc98d921640 1 -- 172.21.15.123:0/78263397 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc9880ae6f0 con 0x7fc98815e0c0 2024-09-16T15:51:22.708 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:22.710+0000 7fc98d921640 1 --2- 172.21.15.123:0/78263397 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc98815e0c0 0x7fc9881547a0 secure :-1 s=READY pgs=1757 cs=0 l=1 rev1=1 crypto rx=0x7fc9800b0040 tx=0x7fc9800a5700 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:51:22.708 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:22.710+0000 7fc979ffb640 1 -- 172.21.15.123:0/78263397 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9800b2040 con 0x7fc98815e0c0 2024-09-16T15:51:22.709 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:22.710+0000 7fc979ffb640 1 -- 172.21.15.123:0/78263397 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fc9800044b0 con 0x7fc98815e0c0 2024-09-16T15:51:22.709 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:22.710+0000 7fc979ffb640 1 -- 172.21.15.123:0/78263397 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc980005270 con 0x7fc98815e0c0 2024-09-16T15:51:22.709 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:22.710+0000 7fc98e122640 1 -- 172.21.15.123:0/78263397 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc9880c4d00 con 0x7fc98815e0c0 2024-09-16T15:51:22.709 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:22.710+0000 7fc98e122640 1 -- 172.21.15.123:0/78263397 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc9880ae990 con 0x7fc98815e0c0 2024-09-16T15:51:22.709 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:22.710+0000 7fc979ffb640 1 -- 172.21.15.123:0/78263397 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 24) ==== 97845+0+0 (secure 0 0 0) 0x7fc980005410 con 0x7fc98815e0c0 2024-09-16T15:51:22.710 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:22.710+0000 7fc979ffb640 1 --2- 172.21.15.123:0/78263397 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fc96c073730 0x7fc96c075bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:51:22.710 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:22.710+0000 7fc979ffb640 1 -- 172.21.15.123:0/78263397 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(157..157 src has 1..157) ==== 5737+0+0 (secure 0 0 0) 0x7fc98012e950 con 0x7fc98815e0c0 2024-09-16T15:51:22.711 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:22.710+0000 7fc98d120640 1 --2- 172.21.15.123:0/78263397 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fc96c073730 0x7fc96c075bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:51:22.711 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:22.710+0000 7fc99246e000 1 Processor -- start 2024-09-16T15:51:22.711 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:22.710+0000 7fc98d120640 1 --2- 172.21.15.123:0/78263397 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fc96c073730 0x7fc96c075bf0 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7fc984009730 tx=0x7fc984006ca0 comp rx=0 tx=0).ready entity=mgr.4150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:51:22.711 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:22.710+0000 7fc99246e000 1 -- start start 2024-09-16T15:51:22.712 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:22.714+0000 7fc99246e000 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x564a106ea9d0 0x564a106eadb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:51:22.712 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:22.714+0000 7fc99246e000 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x564a106eb2f0 0x564a1069e8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:51:22.712 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:22.714+0000 7fc99246e000 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x564a1069ee10 0x564a1069f1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:51:22.712 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:22.714+0000 7fc99246e000 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x564a105bf3f0 con 0x564a106eb2f0 2024-09-16T15:51:22.712 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:22.714+0000 7fc99246e000 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x564a104fcb20 con 0x564a1069ee10 2024-09-16T15:51:22.713 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:22.714+0000 7fc99246e000 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x564a104ebf50 con 0x564a106ea9d0 2024-09-16T15:51:22.713 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:22.714+0000 7fc98c91f640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x564a106eb2f0 0x564a1069e8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:51:22.713 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:22.714+0000 7fc98c91f640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x564a106eb2f0 0x564a1069e8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:57622/0 (socket says 172.21.15.123:57622) 2024-09-16T15:51:22.713 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:22.714+0000 7fc98c91f640 1 -- 172.21.15.123:0/2503793221 learned_addr learned my addr 172.21.15.123:0/2503793221 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:51:22.713 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:22.714+0000 7fc98d120640 1 --2- 172.21.15.123:0/2503793221 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x564a106ea9d0 0x564a106eadb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:51:22.713 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:22.714+0000 7fc98c91f640 1 -- 172.21.15.123:0/2503793221 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x564a106ea9d0 msgr2=0x564a106eadb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T15:51:22.713 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:22.714+0000 7fc98d921640 1 --2- 172.21.15.123:0/2503793221 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x564a1069ee10 0x564a1069f1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:51:22.713 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:22.714+0000 7fc98c91f640 1 --2- 172.21.15.123:0/2503793221 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x564a106ea9d0 0x564a106eadb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:22.713 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:22.714+0000 7fc98c91f640 1 -- 172.21.15.123:0/2503793221 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x564a1069ee10 msgr2=0x564a1069f1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T15:51:22.713 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:22.714+0000 7fc98c91f640 1 --2- 172.21.15.123:0/2503793221 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x564a1069ee10 0x564a1069f1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:22.713 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:22.714+0000 7fc98c91f640 1 -- 172.21.15.123:0/2503793221 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564a104b3590 con 0x564a106eb2f0 2024-09-16T15:51:22.713 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:22.714+0000 7fc98c91f640 1 --2- 172.21.15.123:0/2503793221 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x564a106eb2f0 0x564a1069e8d0 secure :-1 s=READY pgs=1758 cs=0 l=1 rev1=1 crypto rx=0x7fc988072970 tx=0x7fc97c0020e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ec1de9c78f4bf87 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:51:22.714 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:22.714+0000 7fc96affd640 1 -- 172.21.15.123:0/2503793221 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc97c018070 con 0x564a106eb2f0 2024-09-16T15:51:22.714 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:22.714+0000 7fc96affd640 1 -- 172.21.15.123:0/2503793221 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fc97c00d7f0 con 0x564a106eb2f0 2024-09-16T15:51:22.714 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:22.714+0000 7fc96affd640 1 -- 172.21.15.123:0/2503793221 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc97c00dc20 con 0x564a106eb2f0 2024-09-16T15:51:22.717 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:22.714+0000 7fc99246e000 1 -- 172.21.15.123:0/2503793221 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x564a106eb2f0 msgr2=0x564a1069e8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:51:22.717 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:22.714+0000 7fc99246e000 1 --2- 172.21.15.123:0/2503793221 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x564a106eb2f0 0x564a1069e8d0 secure :-1 s=READY pgs=1758 cs=0 l=1 rev1=1 crypto rx=0x7fc988072970 tx=0x7fc97c0020e0 comp rx=0 tx=0).stop 2024-09-16T15:51:22.717 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:22.714+0000 7fc99246e000 1 -- 172.21.15.123:0/2503793221 shutdown_connections 2024-09-16T15:51:22.717 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:22.714+0000 7fc99246e000 1 --2- 172.21.15.123:0/2503793221 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x564a1069ee10 0x564a1069f1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:22.717 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:22.714+0000 7fc99246e000 1 --2- 172.21.15.123:0/2503793221 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x564a106eb2f0 0x564a1069e8d0 unknown :-1 s=CLOSED pgs=1758 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:22.717 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:22.714+0000 7fc99246e000 1 --2- 172.21.15.123:0/2503793221 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x564a106ea9d0 0x564a106eadb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:22.718 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:22.714+0000 7fc99246e000 1 -- 172.21.15.123:0/2503793221 >> 172.21.15.123:0/2503793221 conn(0x564a10588130 msgr2=0x564a106cde80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:51:22.718 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:22.714+0000 7fc99246e000 1 -- 172.21.15.123:0/2503793221 shutdown_connections 2024-09-16T15:51:22.718 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:22.714+0000 7fc99246e000 1 -- 172.21.15.123:0/2503793221 wait complete. 2024-09-16T15:51:22.718 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:22.714+0000 7fc99246e000 1 -- start start 2024-09-16T15:51:22.718 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:22.714+0000 7fc99246e000 1 Processor -- start 2024-09-16T15:51:22.718 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:22.714+0000 7fc99246e000 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x564a1069ee10 0x564a106a0240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:51:22.718 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:22.714+0000 7fc99246e000 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x564a106ea9d0 0x564a106a0780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:51:22.718 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:22.714+0000 7fc99246e000 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x564a106eb2f0 0x564a106a0cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:51:22.718 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:22.714+0000 7fc99246e000 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x564a104865d0 con 0x564a1069ee10 2024-09-16T15:51:22.718 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:22.714+0000 7fc99246e000 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x564a104efd60 con 0x564a106eb2f0 2024-09-16T15:51:22.718 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:22.714+0000 7fc99246e000 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x564a10492500 con 0x564a106ea9d0 2024-09-16T15:51:22.718 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:22.714+0000 7fc99246e000 10 client.0 _ll_register_callbacks cb 0x564a10496810 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T15:51:22.718 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:22.714+0000 7fc98d120640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x564a1069ee10 0x564a106a0240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:51:22.719 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:22.714+0000 7fc98d120640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x564a1069ee10 0x564a106a0240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:57628/0 (socket says 172.21.15.123:57628) 2024-09-16T15:51:22.719 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:22.714+0000 7fc98d120640 1 -- 172.21.15.123:0/3046425592 learned_addr learned my addr 172.21.15.123:0/3046425592 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:51:22.719 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:22.714+0000 7fc98d921640 1 --2- 172.21.15.123:0/3046425592 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x564a106eb2f0 0x564a106a0cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:51:22.719 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:22.714+0000 7fc98d120640 1 -- 172.21.15.123:0/3046425592 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x564a106ea9d0 msgr2=0x564a106a0780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T15:51:22.719 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:22.714+0000 7fc98c91f640 1 --2- 172.21.15.123:0/3046425592 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x564a106ea9d0 0x564a106a0780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:51:22.719 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:22.714+0000 7fc98d120640 1 --2- 172.21.15.123:0/3046425592 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x564a106ea9d0 0x564a106a0780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:22.719 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:22.714+0000 7fc98d120640 1 -- 172.21.15.123:0/3046425592 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x564a106eb2f0 msgr2=0x564a106a0cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T15:51:22.719 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:22.714+0000 7fc98d120640 1 --2- 172.21.15.123:0/3046425592 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x564a106eb2f0 0x564a106a0cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:22.719 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:22.714+0000 7fc98d120640 1 -- 172.21.15.123:0/3046425592 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564a1048a170 con 0x564a1069ee10 2024-09-16T15:51:22.719 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:22.714+0000 7fc98d120640 1 --2- 172.21.15.123:0/3046425592 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x564a1069ee10 0x564a106a0240 secure :-1 s=READY pgs=1759 cs=0 l=1 rev1=1 crypto rx=0x7fc98400bf40 tx=0x7fc9840047e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4f8d62ae585f2a24 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:51:22.719 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:22.714+0000 7fc963fff640 10 client.0 ms_handle_connect on v2:172.21.15.123:3300/0 2024-09-16T15:51:22.719 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:22.718+0000 7fc963fff640 1 -- 172.21.15.123:0/3046425592 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc984023060 con 0x564a1069ee10 2024-09-16T15:51:22.719 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:22.718+0000 7fc963fff640 1 -- 172.21.15.123:0/3046425592 <== mon.0 v2:172.21.15.123:3300/0 2 ==== osd_map(157..157 src has 1..157) ==== 5737+0+0 (secure 0 0 0) 0x7fc9840127c0 con 0x564a1069ee10 2024-09-16T15:51:22.720 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:22.718+0000 7fc99246e000 1 -- 172.21.15.123:0/3046425592 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x564a104840f0 con 0x564a1069ee10 2024-09-16T15:51:22.720 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:22.718+0000 7fc963fff640 1 -- 172.21.15.123:0/3046425592 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_get_version_reply(handle=1 version=105) ==== 24+0+0 (secure 0 0 0) 0x7fc984029050 con 0x564a1069ee10 2024-09-16T15:51:22.720 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:22.718+0000 7fc99246e000 10 client.8979 fetch_fsmap learned FSMap version 105 2024-09-16T15:51:22.720 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:22.718+0000 7fc99246e000 1 -- 172.21.15.123:0/3046425592 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,fsmap=105,monmap=2+}) -- 0x564a10570ef0 con 0x564a1069ee10 2024-09-16T15:51:22.720 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:22.718+0000 7fc963fff640 1 -- 172.21.15.123:0/3046425592 <== mon.0 v2:172.21.15.123:3300/0 4 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x564a104840f0 con 0x564a1069ee10 2024-09-16T15:51:22.720 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:22.718+0000 7fc963fff640 1 -- 172.21.15.123:0/3046425592 <== mon.0 v2:172.21.15.123:3300/0 5 ==== fsmap(e 105) ==== 3502+0+0 (secure 0 0 0) 0x7fc98402c020 con 0x564a1069ee10 2024-09-16T15:51:22.720 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:22.718+0000 7fc99246e000 10 client.8979 fetch_fsmap finished waiting for FSMap version 105 2024-09-16T15:51:22.720 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:22.718+0000 7fc99246e000 10 client.8979 resolve_mds: resolved name 'd' to daemon mds.d 2024-09-16T15:51:22.720 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:22.718+0000 7fc99246e000 20 client.8979 populate_metadata read hostname 'smithi123' 2024-09-16T15:51:22.720 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:22.718+0000 7fc99246e000 1 --2- 172.21.15.123:0/3046425592 >> [v2:172.21.15.184:6832/2856066034,v1:172.21.15.184:6834/2856066034] conn(0x564a106f9c70 0x564a106fc060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:51:22.720 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:22.718+0000 7fc99246e000 4 client.8979 mds_command: new command op to 8859 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-16T15:51:22.720 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:22.718+0000 7fc99246e000 1 -- 172.21.15.123:0/3046425592 --> [v2:172.21.15.184:6832/2856066034,v1:172.21.15.184:6834/2856066034] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564a104fea70 con 0x564a106f9c70 2024-09-16T15:51:22.721 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:22.718+0000 7fc98c91f640 1 --2- 172.21.15.123:0/3046425592 >> [v2:172.21.15.184:6832/2856066034,v1:172.21.15.184:6834/2856066034] conn(0x564a106f9c70 0x564a106fc060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:51:22.721 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:22.718+0000 7fc963fff640 10 client.8979 ms_handle_connect on v2:172.21.15.184:6832/2856066034 2024-09-16T15:51:22.721 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:22.718+0000 7fc98c91f640 1 --2- 172.21.15.123:0/3046425592 >> [v2:172.21.15.184:6832/2856066034,v1:172.21.15.184:6834/2856066034] conn(0x564a106f9c70 0x564a106fc060 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2c7f005b59e59de4 server_cookie=b5107c05c63db2a7 in_seq=0 out_seq=0 2024-09-16T15:51:22.722 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:22.718+0000 7fc963fff640 1 -- 172.21.15.123:0/3046425592 <== mds.0 v2:172.21.15.184:6832/2856066034 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x564a104fea70 con 0x564a106f9c70 2024-09-16T15:51:22.722 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:22.718+0000 7fc963fff640 10 client.8979 handle_command_reply: tid=0 2024-09-16T15:51:22.722 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:22.718+0000 7fc99246e000 1 client.8979 shutdown 2024-09-16T15:51:22.722 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:22.718+0000 7fc99246e000 1 -- 172.21.15.123:0/3046425592 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x564a1069ee10 msgr2=0x564a106a0240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:51:22.722 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:22.718+0000 7fc99246e000 1 --2- 172.21.15.123:0/3046425592 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x564a1069ee10 0x564a106a0240 secure :-1 s=READY pgs=1759 cs=0 l=1 rev1=1 crypto rx=0x7fc98400bf40 tx=0x7fc9840047e0 comp rx=0 tx=0).stop 2024-09-16T15:51:22.722 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:22.718+0000 7fc99246e000 1 -- 172.21.15.123:0/3046425592 shutdown_connections 2024-09-16T15:51:22.722 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:22.718+0000 7fc99246e000 1 --2- 172.21.15.123:0/3046425592 >> [v2:172.21.15.184:6832/2856066034,v1:172.21.15.184:6834/2856066034] conn(0x564a106f9c70 0x564a106fc060 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:22.722 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:22.718+0000 7fc99246e000 1 --2- 172.21.15.123:0/3046425592 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x564a106eb2f0 0x564a106a0cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:22.722 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:22.718+0000 7fc99246e000 1 --2- 172.21.15.123:0/3046425592 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x564a106ea9d0 0x564a106a0780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:22.722 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:22.718+0000 7fc963fff640 0 client.8979 ms_handle_reset on v2:172.21.15.184:6832/2856066034 2024-09-16T15:51:22.723 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:22.718+0000 7fc99246e000 1 --2- 172.21.15.123:0/3046425592 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x564a1069ee10 0x564a106a0240 unknown :-1 s=CLOSED pgs=1759 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:22.723 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:22.718+0000 7fc99246e000 1 -- 172.21.15.123:0/3046425592 >> 172.21.15.123:0/3046425592 conn(0x564a10588130 msgr2=0x564a1069dfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:51:22.723 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:22.718+0000 7fc99246e000 1 -- 172.21.15.123:0/3046425592 shutdown_connections 2024-09-16T15:51:22.723 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:22.718+0000 7fc99246e000 1 -- 172.21.15.123:0/3046425592 wait complete. 2024-09-16T15:51:22.723 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:22.718+0000 7fc99246e000 20 client.8979 trim_cache size 0 max 16384 2024-09-16T15:51:22.736 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:22.738+0000 7fc99246e000 1 Processor -- start 2024-09-16T15:51:22.736 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:22.738+0000 7fc99246e000 1 -- start start 2024-09-16T15:51:22.739 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:22.738+0000 7fc99246e000 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x564a1069ee10 0x564a1075c3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:51:22.739 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:22.738+0000 7fc99246e000 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x564a106ea9d0 0x564a1075c900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:51:22.739 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:22.738+0000 7fc99246e000 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x564a106eb2f0 0x564a1075ce40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:51:22.739 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:22.738+0000 7fc99246e000 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fc988203950 con 0x564a106ea9d0 2024-09-16T15:51:22.739 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:22.738+0000 7fc99246e000 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fc9881fb620 con 0x564a106eb2f0 2024-09-16T15:51:22.739 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:22.738+0000 7fc99246e000 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fc9881fa980 con 0x564a1069ee10 2024-09-16T15:51:22.739 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:22.738+0000 7fc98c91f640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x564a106ea9d0 0x564a1075c900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:51:22.739 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:22.738+0000 7fc98c91f640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x564a106ea9d0 0x564a1075c900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:57642/0 (socket says 172.21.15.123:57642) 2024-09-16T15:51:22.739 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:22.738+0000 7fc98c91f640 1 -- 172.21.15.123:0/1400749915 learned_addr learned my addr 172.21.15.123:0/1400749915 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:51:22.740 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:22.738+0000 7fc98d120640 1 --2- 172.21.15.123:0/1400749915 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x564a1069ee10 0x564a1075c3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:51:22.740 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:22.738+0000 7fc98c91f640 1 -- 172.21.15.123:0/1400749915 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x564a1069ee10 msgr2=0x564a1075c3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T15:51:22.740 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:22.738+0000 7fc98c91f640 1 --2- 172.21.15.123:0/1400749915 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x564a1069ee10 0x564a1075c3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:22.740 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:22.738+0000 7fc98c91f640 1 -- 172.21.15.123:0/1400749915 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x564a106eb2f0 msgr2=0x564a1075ce40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T15:51:22.740 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:22.738+0000 7fc98d921640 1 --2- 172.21.15.123:0/1400749915 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x564a106eb2f0 0x564a1075ce40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:51:22.740 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:22.738+0000 7fc98c91f640 1 --2- 172.21.15.123:0/1400749915 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x564a106eb2f0 0x564a1075ce40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:22.740 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:22.738+0000 7fc98c91f640 1 -- 172.21.15.123:0/1400749915 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564a10547d90 con 0x564a106ea9d0 2024-09-16T15:51:22.740 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:22.738+0000 7fc98c91f640 1 --2- 172.21.15.123:0/1400749915 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x564a106ea9d0 0x564a1075c900 secure :-1 s=READY pgs=1760 cs=0 l=1 rev1=1 crypto rx=0x564a1049d130 tx=0x7fc97c001260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=937037558ac321fe server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:51:22.740 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:22.738+0000 7fc96a7fc640 1 -- 172.21.15.123:0/1400749915 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc97c02c070 con 0x564a106ea9d0 2024-09-16T15:51:22.740 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:22.738+0000 7fc96a7fc640 1 -- 172.21.15.123:0/1400749915 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fc97c024070 con 0x564a106ea9d0 2024-09-16T15:51:22.740 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:22.738+0000 7fc96a7fc640 1 -- 172.21.15.123:0/1400749915 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc97c018040 con 0x564a106ea9d0 2024-09-16T15:51:22.740 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:22.738+0000 7fc99246e000 1 -- 172.21.15.123:0/1400749915 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x564a106ea9d0 msgr2=0x564a1075c900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:51:22.740 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:22.738+0000 7fc99246e000 1 --2- 172.21.15.123:0/1400749915 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x564a106ea9d0 0x564a1075c900 secure :-1 s=READY pgs=1760 cs=0 l=1 rev1=1 crypto rx=0x564a1049d130 tx=0x7fc97c001260 comp rx=0 tx=0).stop 2024-09-16T15:51:22.741 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:22.738+0000 7fc99246e000 1 -- 172.21.15.123:0/1400749915 shutdown_connections 2024-09-16T15:51:22.741 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:22.738+0000 7fc99246e000 1 --2- 172.21.15.123:0/1400749915 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x564a106eb2f0 0x564a1075ce40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:22.741 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:22.738+0000 7fc99246e000 1 --2- 172.21.15.123:0/1400749915 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x564a106ea9d0 0x564a1075c900 unknown :-1 s=CLOSED pgs=1760 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:22.741 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:22.738+0000 7fc99246e000 1 --2- 172.21.15.123:0/1400749915 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x564a1069ee10 0x564a1075c3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:22.741 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:22.738+0000 7fc99246e000 1 -- 172.21.15.123:0/1400749915 >> 172.21.15.123:0/1400749915 conn(0x564a10588130 msgr2=0x564a10701e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:51:22.741 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:22.738+0000 7fc99246e000 1 -- 172.21.15.123:0/1400749915 shutdown_connections 2024-09-16T15:51:22.741 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:22.738+0000 7fc99246e000 1 -- 172.21.15.123:0/1400749915 wait complete. 2024-09-16T15:51:22.741 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:22.738+0000 7fc99246e000 1 -- start start 2024-09-16T15:51:22.741 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:22.738+0000 7fc99246e000 1 Processor -- start 2024-09-16T15:51:22.745 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:22.742+0000 7fc99246e000 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x564a1069ee10 0x564a10700a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:51:22.745 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:22.742+0000 7fc99246e000 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x564a106ea9d0 0x564a1075c7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:51:22.745 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:22.742+0000 7fc99246e000 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x564a106eb2f0 0x564a1075cd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:51:22.745 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:22.742+0000 7fc99246e000 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x564a104b7ec0 con 0x564a106eb2f0 2024-09-16T15:51:22.745 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:22.742+0000 7fc99246e000 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fc9881fafd0 con 0x564a106ea9d0 2024-09-16T15:51:22.745 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:22.742+0000 7fc99246e000 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x564a104fb8f0 con 0x564a1069ee10 2024-09-16T15:51:22.745 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:22.742+0000 7fc99246e000 10 client.0 _ll_register_callbacks cb 0x7fc984009440 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T15:51:22.745 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:22.742+0000 7fc98d921640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x564a106eb2f0 0x564a1075cd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:51:22.745 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:22.742+0000 7fc98d921640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x564a106eb2f0 0x564a1075cd20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:57652/0 (socket says 172.21.15.123:57652) 2024-09-16T15:51:22.745 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:22.742+0000 7fc98d921640 1 -- 172.21.15.123:0/1697607729 learned_addr learned my addr 172.21.15.123:0/1697607729 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:51:22.745 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:22.742+0000 7fc98d120640 1 --2- 172.21.15.123:0/1697607729 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x564a1069ee10 0x564a10700a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:51:22.745 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:22.742+0000 7fc98c91f640 1 --2- 172.21.15.123:0/1697607729 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x564a106ea9d0 0x564a1075c7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:51:22.745 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:22.742+0000 7fc98d921640 1 -- 172.21.15.123:0/1697607729 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x564a1069ee10 msgr2=0x564a10700a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T15:51:22.746 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:22.742+0000 7fc98d921640 1 --2- 172.21.15.123:0/1697607729 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x564a1069ee10 0x564a10700a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:22.746 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:22.742+0000 7fc98d921640 1 -- 172.21.15.123:0/1697607729 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x564a106ea9d0 msgr2=0x564a1075c7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T15:51:22.746 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:22.742+0000 7fc98d921640 1 --2- 172.21.15.123:0/1697607729 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x564a106ea9d0 0x564a1075c7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:22.746 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:22.742+0000 7fc98d921640 1 -- 172.21.15.123:0/1697607729 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564a104b2a90 con 0x564a106eb2f0 2024-09-16T15:51:22.746 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:22.742+0000 7fc98c91f640 1 --2- 172.21.15.123:0/1697607729 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x564a106ea9d0 0x564a1075c7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T15:51:22.746 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:22.742+0000 7fc98d921640 1 --2- 172.21.15.123:0/1697607729 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x564a106eb2f0 0x564a1075cd20 secure :-1 s=READY pgs=1761 cs=0 l=1 rev1=1 crypto rx=0x7fc980005c30 tx=0x7fc980101f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f051cb4c52b0b242 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:51:22.746 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:22.742+0000 7fc963fff640 10 client.0 ms_handle_connect on v2:172.21.15.123:3300/0 2024-09-16T15:51:22.746 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:22.742+0000 7fc963fff640 1 -- 172.21.15.123:0/1697607729 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9800f7fd0 con 0x564a106eb2f0 2024-09-16T15:51:22.746 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:22.742+0000 7fc963fff640 1 -- 172.21.15.123:0/1697607729 <== mon.0 v2:172.21.15.123:3300/0 2 ==== osd_map(157..157 src has 1..157) ==== 5737+0+0 (secure 0 0 0) 0x7fc9800f7ac0 con 0x564a106eb2f0 2024-09-16T15:51:22.746 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:22.742+0000 7fc99246e000 1 -- 172.21.15.123:0/1697607729 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc97c024070 con 0x564a106eb2f0 2024-09-16T15:51:22.746 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:22.742+0000 7fc963fff640 1 -- 172.21.15.123:0/1697607729 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_get_version_reply(handle=1 version=105) ==== 24+0+0 (secure 0 0 0) 0x7fc980117050 con 0x564a106eb2f0 2024-09-16T15:51:22.746 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:22.742+0000 7fc99246e000 10 client.8985 fetch_fsmap learned FSMap version 105 2024-09-16T15:51:22.746 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:22.742+0000 7fc99246e000 1 -- 172.21.15.123:0/1697607729 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,fsmap=105,monmap=2+}) -- 0x564a10491050 con 0x564a106eb2f0 2024-09-16T15:51:22.746 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:22.742+0000 7fc963fff640 1 -- 172.21.15.123:0/1697607729 <== mon.0 v2:172.21.15.123:3300/0 4 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fc97c024070 con 0x564a106eb2f0 2024-09-16T15:51:22.747 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:22.742+0000 7fc963fff640 1 -- 172.21.15.123:0/1697607729 <== mon.0 v2:172.21.15.123:3300/0 5 ==== fsmap(e 105) ==== 3502+0+0 (secure 0 0 0) 0x7fc980112b80 con 0x564a106eb2f0 2024-09-16T15:51:22.747 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:22.742+0000 7fc99246e000 10 client.8985 fetch_fsmap finished waiting for FSMap version 105 2024-09-16T15:51:22.747 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:22.742+0000 7fc99246e000 10 client.8985 resolve_mds: resolved name 'd' to daemon mds.d 2024-09-16T15:51:22.747 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:22.742+0000 7fc99246e000 20 client.8985 populate_metadata read hostname 'smithi123' 2024-09-16T15:51:22.747 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:22.742+0000 7fc99246e000 1 --2- 172.21.15.123:0/1697607729 >> [v2:172.21.15.184:6832/2856066034,v1:172.21.15.184:6834/2856066034] conn(0x564a1071c580 0x564a1071c960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:51:22.747 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:22.742+0000 7fc99246e000 4 client.8985 mds_command: new command op to 8859 tid=0 multi_id=0 [{"prefix": "status"}] 2024-09-16T15:51:22.747 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:22.742+0000 7fc99246e000 1 -- 172.21.15.123:0/1697607729 --> [v2:172.21.15.184:6832/2856066034,v1:172.21.15.184:6834/2856066034] -- command(tid 0: {"prefix": "status"}) -- 0x564a104fa520 con 0x564a1071c580 2024-09-16T15:51:22.747 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:22.742+0000 7fc98d120640 1 --2- 172.21.15.123:0/1697607729 >> [v2:172.21.15.184:6832/2856066034,v1:172.21.15.184:6834/2856066034] conn(0x564a1071c580 0x564a1071c960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:51:22.747 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:22.742+0000 7fc98d120640 1 --2- 172.21.15.123:0/1697607729 >> [v2:172.21.15.184:6832/2856066034,v1:172.21.15.184:6834/2856066034] conn(0x564a1071c580 0x564a1071c960 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6782cbb8da0fff40 server_cookie=42460b92802c9a6d in_seq=0 out_seq=0 2024-09-16T15:51:22.747 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:22.742+0000 7fc963fff640 10 client.8985 ms_handle_connect on v2:172.21.15.184:6832/2856066034 2024-09-16T15:51:22.747 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:22.742+0000 7fc963fff640 1 -- 172.21.15.123:0/1697607729 <== mds.0 v2:172.21.15.184:6832/2856066034 1 ==== command_reply(tid 0: 0 ) ==== 8+0+328 (crc 0 0 0) 0x564a104fa520 con 0x564a1071c580 2024-09-16T15:51:22.747 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:22.742+0000 7fc963fff640 10 client.8985 handle_command_reply: tid=0 2024-09-16T15:51:22.747 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:22.742+0000 7fc99246e000 1 client.8985 shutdown 2024-09-16T15:51:22.748 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:22.742+0000 7fc99246e000 1 -- 172.21.15.123:0/1697607729 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x564a106eb2f0 msgr2=0x564a1075cd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:51:22.748 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:22.742+0000 7fc99246e000 1 --2- 172.21.15.123:0/1697607729 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x564a106eb2f0 0x564a1075cd20 secure :-1 s=READY pgs=1761 cs=0 l=1 rev1=1 crypto rx=0x7fc980005c30 tx=0x7fc980101f90 comp rx=0 tx=0).stop 2024-09-16T15:51:22.748 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:22.742+0000 7fc99246e000 1 -- 172.21.15.123:0/1697607729 shutdown_connections 2024-09-16T15:51:22.748 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:22.742+0000 7fc99246e000 1 --2- 172.21.15.123:0/1697607729 >> [v2:172.21.15.184:6832/2856066034,v1:172.21.15.184:6834/2856066034] conn(0x564a1071c580 0x564a1071c960 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:22.748 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:22.742+0000 7fc99246e000 1 --2- 172.21.15.123:0/1697607729 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x564a106eb2f0 0x564a1075cd20 unknown :-1 s=CLOSED pgs=1761 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:22.748 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:22.742+0000 7fc99246e000 1 --2- 172.21.15.123:0/1697607729 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x564a106ea9d0 0x564a1075c7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:22.748 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:22.742+0000 7fc99246e000 1 --2- 172.21.15.123:0/1697607729 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x564a1069ee10 0x564a10700a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:22.748 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:22.742+0000 7fc99246e000 1 -- 172.21.15.123:0/1697607729 >> 172.21.15.123:0/1697607729 conn(0x564a10588130 msgr2=0x564a1069e470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:51:22.748 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:22.742+0000 7fc963fff640 0 client.8985 ms_handle_reset on v2:172.21.15.184:6832/2856066034 2024-09-16T15:51:22.748 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:22.742+0000 7fc99246e000 1 -- 172.21.15.123:0/1697607729 shutdown_connections 2024-09-16T15:51:22.748 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:22.742+0000 7fc99246e000 1 -- 172.21.15.123:0/1697607729 wait complete. 2024-09-16T15:51:22.748 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:22.742+0000 7fc99246e000 20 client.8985 trim_cache size 0 max 16384 2024-09-16T15:51:22.749 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2024-09-16T15:51:22.749 INFO:teuthology.orchestra.run.smithi123.stdout: "cluster_fsid": "56077f89-18c8-486f-ae7e-eecc0bb7822c", 2024-09-16T15:51:22.749 INFO:teuthology.orchestra.run.smithi123.stdout: "whoami": 0, 2024-09-16T15:51:22.749 INFO:teuthology.orchestra.run.smithi123.stdout: "id": 8859, 2024-09-16T15:51:22.749 INFO:teuthology.orchestra.run.smithi123.stdout: "want_state": "up:active", 2024-09-16T15:51:22.749 INFO:teuthology.orchestra.run.smithi123.stdout: "state": "up:active", 2024-09-16T15:51:22.749 INFO:teuthology.orchestra.run.smithi123.stdout: "fs_name": "cephfs", 2024-09-16T15:51:22.749 INFO:teuthology.orchestra.run.smithi123.stdout: "rank_uptime": 5.7613339530000003, 2024-09-16T15:51:22.749 INFO:teuthology.orchestra.run.smithi123.stdout: "mdsmap_epoch": 105, 2024-09-16T15:51:22.749 INFO:teuthology.orchestra.run.smithi123.stdout: "osdmap_epoch": 157, 2024-09-16T15:51:22.749 INFO:teuthology.orchestra.run.smithi123.stdout: "osdmap_epoch_barrier": 157, 2024-09-16T15:51:22.749 INFO:teuthology.orchestra.run.smithi123.stdout: "uptime": 10.403903244 2024-09-16T15:51:22.750 INFO:teuthology.orchestra.run.smithi123.stdout:} 2024-09-16T15:51:22.750 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:22.742+0000 7fc98e122640 1 -- 172.21.15.123:0/78263397 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fc96c073730 msgr2=0x7fc96c075bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:51:22.750 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:22.742+0000 7fc98e122640 1 --2- 172.21.15.123:0/78263397 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fc96c073730 0x7fc96c075bf0 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7fc984009730 tx=0x7fc984006ca0 comp rx=0 tx=0).stop 2024-09-16T15:51:22.750 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:22.742+0000 7fc98e122640 1 -- 172.21.15.123:0/78263397 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc98815e0c0 msgr2=0x7fc9881547a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:51:22.750 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:22.742+0000 7fc98e122640 1 --2- 172.21.15.123:0/78263397 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc98815e0c0 0x7fc9881547a0 secure :-1 s=READY pgs=1757 cs=0 l=1 rev1=1 crypto rx=0x7fc9800b0040 tx=0x7fc9800a5700 comp rx=0 tx=0).stop 2024-09-16T15:51:22.751 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:22.742+0000 7fc98e122640 1 -- 172.21.15.123:0/78263397 shutdown_connections 2024-09-16T15:51:22.751 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:22.742+0000 7fc98e122640 1 --2- 172.21.15.123:0/78263397 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fc96c073730 0x7fc96c075bf0 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:22.751 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:22.742+0000 7fc98e122640 1 --2- 172.21.15.123:0/78263397 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc98815e0c0 0x7fc9881547a0 unknown :-1 s=CLOSED pgs=1757 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:22.751 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:22.742+0000 7fc98e122640 1 --2- 172.21.15.123:0/78263397 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fc9880cd800 0x7fc9880af190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:22.751 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:22.742+0000 7fc98e122640 1 --2- 172.21.15.123:0/78263397 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc9880b1c50 0x7fc9881613a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:22.751 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:22.742+0000 7fc98e122640 1 -- 172.21.15.123:0/78263397 >> 172.21.15.123:0/78263397 conn(0x7fc9880ccc50 msgr2=0x7fc9880b1030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:51:22.751 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:22.742+0000 7fc98e122640 1 -- 172.21.15.123:0/78263397 shutdown_connections 2024-09-16T15:51:22.751 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:22.742+0000 7fc98e122640 1 -- 172.21.15.123:0/78263397 wait complete. 2024-09-16T15:51:22.758 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-09-16T15:51:22.863 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:22.862+0000 7f853df4c640 1 -- 172.21.15.123:0/1091881289 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8524004d50 msgr2=0x7f85240051d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:51:22.863 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:22.862+0000 7f853df4c640 1 --2- 172.21.15.123:0/1091881289 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8524004d50 0x7f85240051d0 secure :-1 s=READY pgs=1763 cs=0 l=1 rev1=1 crypto rx=0x7f8530001dc0 tx=0x7f853001cb30 comp rx=0 tx=0).stop 2024-09-16T15:51:22.863 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:22.862+0000 7f8537fff640 1 -- 172.21.15.123:0/1091881289 reap_dead start 2024-09-16T15:51:22.863 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:22.862+0000 7f853df4c640 1 -- 172.21.15.123:0/1091881289 shutdown_connections 2024-09-16T15:51:22.863 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:22.862+0000 7f853df4c640 1 -- 172.21.15.123:0/1091881289 >> 172.21.15.123:0/1091881289 conn(0x7f85380d3e60 msgr2=0x7f85380d4240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:51:22.863 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:22.862+0000 7f853df4c640 1 -- 172.21.15.123:0/1091881289 shutdown_connections 2024-09-16T15:51:22.863 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:22.862+0000 7f853df4c640 1 -- 172.21.15.123:0/1091881289 wait complete. 2024-09-16T15:51:22.864 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:22.866+0000 7f853df4c640 1 Processor -- start 2024-09-16T15:51:22.864 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:22.866+0000 7f853df4c640 1 -- start start 2024-09-16T15:51:22.866 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:22.866+0000 7f853df4c640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f85380b84a0 0x7f8538157f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:51:22.866 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:22.866+0000 7f853df4c640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f85381e8430 0x7f8538158470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:51:22.866 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:22.866+0000 7f853df4c640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f85380b8880 0x7f85381f8110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:51:22.866 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:22.866+0000 7f853df4c640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f8538161ec0 con 0x7f85380b84a0 2024-09-16T15:51:22.866 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:22.866+0000 7f853df4c640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f8538161d40 con 0x7f85380b8880 2024-09-16T15:51:22.866 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:22.866+0000 7f853df4c640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f8538162040 con 0x7f85381e8430 2024-09-16T15:51:22.866 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:22.866+0000 7f85377fe640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f85380b84a0 0x7f8538157f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:51:22.866 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:22.866+0000 7f85377fe640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f85380b84a0 0x7f8538157f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:57662/0 (socket says 172.21.15.123:57662) 2024-09-16T15:51:22.867 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:22.866+0000 7f85377fe640 1 -- 172.21.15.123:0/3998431261 learned_addr learned my addr 172.21.15.123:0/3998431261 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:51:22.867 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:22.866+0000 7f852ffff640 1 --2- 172.21.15.123:0/3998431261 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f85381e8430 0x7f8538158470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:51:22.867 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:22.866+0000 7f8537fff640 1 --2- 172.21.15.123:0/3998431261 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f85380b8880 0x7f85381f8110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:51:22.867 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:22.866+0000 7f85377fe640 1 -- 172.21.15.123:0/3998431261 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f85381e8430 msgr2=0x7f8538158470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:51:22.867 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:22.866+0000 7f85377fe640 1 --2- 172.21.15.123:0/3998431261 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f85381e8430 0x7f8538158470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:22.867 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:22.866+0000 7f85377fe640 1 -- 172.21.15.123:0/3998431261 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f85380b8880 msgr2=0x7f85381f8110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:51:22.867 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:22.866+0000 7f85377fe640 1 --2- 172.21.15.123:0/3998431261 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f85380b8880 0x7f85381f8110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:22.867 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:22.866+0000 7f85377fe640 1 -- 172.21.15.123:0/3998431261 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f853815c630 con 0x7f85380b84a0 2024-09-16T15:51:22.867 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:22.866+0000 7f852ffff640 1 --2- 172.21.15.123:0/3998431261 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f85381e8430 0x7f8538158470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T15:51:22.867 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:22.866+0000 7f85377fe640 1 --2- 172.21.15.123:0/3998431261 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f85380b84a0 0x7f8538157f30 secure :-1 s=READY pgs=1764 cs=0 l=1 rev1=1 crypto rx=0x7f8528010a40 tx=0x7f8528010f00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:51:22.867 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:22.866+0000 7f8534ff9640 1 -- 172.21.15.123:0/3998431261 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f852800eda0 con 0x7f85380b84a0 2024-09-16T15:51:22.867 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:22.866+0000 7f8534ff9640 1 -- 172.21.15.123:0/3998431261 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f8528004030 con 0x7f85380b84a0 2024-09-16T15:51:22.867 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:22.866+0000 7f853df4c640 1 -- 172.21.15.123:0/3998431261 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f853815c910 con 0x7f85380b84a0 2024-09-16T15:51:22.868 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:22.866+0000 7f8534ff9640 1 -- 172.21.15.123:0/3998431261 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8528004480 con 0x7f85380b84a0 2024-09-16T15:51:22.868 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:22.866+0000 7f853df4c640 1 -- 172.21.15.123:0/3998431261 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f853815ced0 con 0x7f85380b84a0 2024-09-16T15:51:22.868 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:22.866+0000 7f853df4c640 1 -- 172.21.15.123:0/3998431261 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8538162220 con 0x7f85380b84a0 2024-09-16T15:51:22.868 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:22.870+0000 7f8534ff9640 1 -- 172.21.15.123:0/3998431261 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 24) ==== 97845+0+0 (secure 0 0 0) 0x7f8528003600 con 0x7f85380b84a0 2024-09-16T15:51:22.868 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:22.870+0000 7f8534ff9640 1 --2- 172.21.15.123:0/3998431261 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f8518073760 0x7f8518075c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:51:22.869 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:22.870+0000 7f852ffff640 1 --2- 172.21.15.123:0/3998431261 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f8518073760 0x7f8518075c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:51:22.869 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:22.870+0000 7f8534ff9640 1 -- 172.21.15.123:0/3998431261 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(157..157 src has 1..157) ==== 5737+0+0 (secure 0 0 0) 0x7f8528095f90 con 0x7f85380b84a0 2024-09-16T15:51:22.869 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:22.870+0000 7f852ffff640 1 --2- 172.21.15.123:0/3998431261 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f8518073760 0x7f8518075c20 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f8520000c10 tx=0x7f8520001540 comp rx=0 tx=0).ready entity=mgr.4150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:51:22.871 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:22.874+0000 7f8534ff9640 1 -- 172.21.15.123:0/3998431261 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f85280606b0 con 0x7f85380b84a0 2024-09-16T15:51:23.066 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:23.066+0000 7f853df4c640 1 -- 172.21.15.123:0/3998431261 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f85380d0160 con 0x7f85380b84a0 2024-09-16T15:51:23.067 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:23.066+0000 7f8534ff9640 1 -- 172.21.15.123:0/3998431261 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v105) ==== 53+0+73 (secure 0 0 0) 0x7f852805ffb0 con 0x7f85380b84a0 2024-09-16T15:51:23.067 INFO:teuthology.orchestra.run.smithi123.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-09-16T15:51:23.070 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:23.070+0000 7f853df4c640 1 -- 172.21.15.123:0/3998431261 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f8518073760 msgr2=0x7f8518075c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:51:23.070 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:23.070+0000 7f853df4c640 1 --2- 172.21.15.123:0/3998431261 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f8518073760 0x7f8518075c20 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f8520000c10 tx=0x7f8520001540 comp rx=0 tx=0).stop 2024-09-16T15:51:23.070 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:23.070+0000 7f853df4c640 1 -- 172.21.15.123:0/3998431261 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f85380b84a0 msgr2=0x7f8538157f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:51:23.070 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:23.070+0000 7f853df4c640 1 --2- 172.21.15.123:0/3998431261 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f85380b84a0 0x7f8538157f30 secure :-1 s=READY pgs=1764 cs=0 l=1 rev1=1 crypto rx=0x7f8528010a40 tx=0x7f8528010f00 comp rx=0 tx=0).stop 2024-09-16T15:51:23.070 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:23.070+0000 7f853df4c640 1 -- 172.21.15.123:0/3998431261 shutdown_connections 2024-09-16T15:51:23.070 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:23.070+0000 7f853df4c640 1 --2- 172.21.15.123:0/3998431261 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f8518073760 0x7f8518075c20 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:23.070 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:23.070+0000 7f853df4c640 1 --2- 172.21.15.123:0/3998431261 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f85380b8880 0x7f85381f8110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:23.070 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:23.070+0000 7f853df4c640 1 --2- 172.21.15.123:0/3998431261 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f85381e8430 0x7f8538158470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:23.070 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:23.070+0000 7f853df4c640 1 --2- 172.21.15.123:0/3998431261 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f85380b84a0 0x7f8538157f30 unknown :-1 s=CLOSED pgs=1764 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:23.070 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:23.070+0000 7f853df4c640 1 -- 172.21.15.123:0/3998431261 >> 172.21.15.123:0/3998431261 conn(0x7f85380d3e60 msgr2=0x7f85380cf3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:51:23.070 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:23.070+0000 7f853df4c640 1 -- 172.21.15.123:0/3998431261 shutdown_connections 2024-09-16T15:51:23.071 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:23.070+0000 7f853df4c640 1 -- 172.21.15.123:0/3998431261 wait complete. 2024-09-16T15:51:23.085 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-09-16T15:51:23.085 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-09-16T15:51:23.085 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi123.front.sepia.ceph.com 2024-09-16T15:51:23.085 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-09-16T15:51:23.085 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-09-16T15:51:23.085 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-09-16T15:51:23.085 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-09-16T15:51:23.085 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-09-16T15:51:23.085 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-09-16T15:51:23.085 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:51:23.086 DEBUG:teuthology.orchestra.run.smithi123:> ip addr 2024-09-16T15:51:23.091 INFO:teuthology.orchestra.run.smithi123.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-09-16T15:51:23.091 INFO:teuthology.orchestra.run.smithi123.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-09-16T15:51:23.091 INFO:teuthology.orchestra.run.smithi123.stdout: inet 127.0.0.1/8 scope host lo 2024-09-16T15:51:23.091 INFO:teuthology.orchestra.run.smithi123.stdout: valid_lft forever preferred_lft forever 2024-09-16T15:51:23.091 INFO:teuthology.orchestra.run.smithi123.stdout: inet6 ::1/128 scope host 2024-09-16T15:51:23.092 INFO:teuthology.orchestra.run.smithi123.stdout: valid_lft forever preferred_lft forever 2024-09-16T15:51:23.092 INFO:teuthology.orchestra.run.smithi123.stdout:2: eno1: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-09-16T15:51:23.092 INFO:teuthology.orchestra.run.smithi123.stdout: link/ether 0c:c4:7a:d8:da:6c brd ff:ff:ff:ff:ff:ff 2024-09-16T15:51:23.092 INFO:teuthology.orchestra.run.smithi123.stdout: altname enp7s0f0 2024-09-16T15:51:23.092 INFO:teuthology.orchestra.run.smithi123.stdout:3: eno2: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-09-16T15:51:23.092 INFO:teuthology.orchestra.run.smithi123.stdout: link/ether 0c:c4:7a:d8:da:6d brd ff:ff:ff:ff:ff:ff 2024-09-16T15:51:23.092 INFO:teuthology.orchestra.run.smithi123.stdout: altname enp7s0f1 2024-09-16T15:51:23.092 INFO:teuthology.orchestra.run.smithi123.stdout:4: enp3s0f0: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-09-16T15:51:23.092 INFO:teuthology.orchestra.run.smithi123.stdout: link/ether 0c:c4:7a:8f:d3:f0 brd ff:ff:ff:ff:ff:ff 2024-09-16T15:51:23.092 INFO:teuthology.orchestra.run.smithi123.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-09-16T15:51:23.092 INFO:teuthology.orchestra.run.smithi123.stdout: link/ether 0c:c4:7a:8f:d3:f1 brd ff:ff:ff:ff:ff:ff 2024-09-16T15:51:23.092 INFO:teuthology.orchestra.run.smithi123.stdout: inet 172.21.15.123/20 metric 100 brd 172.21.15.255 scope global dynamic enp3s0f1 2024-09-16T15:51:23.092 INFO:teuthology.orchestra.run.smithi123.stdout: valid_lft 41985sec preferred_lft 41985sec 2024-09-16T15:51:23.092 INFO:teuthology.orchestra.run.smithi123.stdout: inet6 fe80::ec4:7aff:fe8f:d3f1/64 scope link 2024-09-16T15:51:23.093 INFO:teuthology.orchestra.run.smithi123.stdout: valid_lft forever preferred_lft forever 2024-09-16T15:51:23.093 INFO:teuthology.orchestra.run.smithi123.stdout:6: docker0: mtu 1500 qdisc noqueue state DOWN group default 2024-09-16T15:51:23.093 INFO:teuthology.orchestra.run.smithi123.stdout: link/ether 02:42:dd:82:a2:6e brd ff:ff:ff:ff:ff:ff 2024-09-16T15:51:23.093 INFO:teuthology.orchestra.run.smithi123.stdout: inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0 2024-09-16T15:51:23.093 INFO:teuthology.orchestra.run.smithi123.stdout: valid_lft forever preferred_lft forever 2024-09-16T15:51:23.093 INFO:teuthology.orchestra.run.smithi123.stdout:7: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-09-16T15:51:23.093 INFO:teuthology.orchestra.run.smithi123.stdout: link/ether 5e:1f:84:6d:79:ae brd ff:ff:ff:ff:ff:ff 2024-09-16T15:51:23.093 INFO:teuthology.orchestra.run.smithi123.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-09-16T15:51:23.093 INFO:teuthology.orchestra.run.smithi123.stdout: valid_lft forever preferred_lft forever 2024-09-16T15:51:23.093 INFO:teuthology.orchestra.run.smithi123.stdout: inet6 fe80::5c1f:84ff:fe6d:79ae/64 scope link 2024-09-16T15:51:23.093 INFO:teuthology.orchestra.run.smithi123.stdout: valid_lft forever preferred_lft forever 2024-09-16T15:51:23.093 INFO:teuthology.orchestra.run.smithi123.stdout:8: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-09-16T15:51:23.093 INFO:teuthology.orchestra.run.smithi123.stdout: link/ether 0e:5c:f6:1f:82:2f brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-09-16T15:51:23.093 INFO:teuthology.orchestra.run.smithi123.stdout: inet6 fe80::c5c:f6ff:fe1f:822f/64 scope link 2024-09-16T15:51:23.093 INFO:teuthology.orchestra.run.smithi123.stdout: valid_lft forever preferred_lft forever 2024-09-16T15:51:23.094 INFO:teuthology.orchestra.run.smithi123.stdout:9: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-09-16T15:51:23.094 INFO:teuthology.orchestra.run.smithi123.stdout: link/ether c6:a6:3d:69:d0:db brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-09-16T15:51:23.094 INFO:teuthology.orchestra.run.smithi123.stdout: inet6 fe80::c4a6:3dff:fe69:d0db/64 scope link 2024-09-16T15:51:23.094 INFO:teuthology.orchestra.run.smithi123.stdout: valid_lft forever preferred_lft forever 2024-09-16T15:51:23.094 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:51:23.094 DEBUG:teuthology.orchestra.run.smithi123:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-09-16T15:51:23.146 INFO:teuthology.orchestra.run.smithi123.stdout:1 2024-09-16T15:51:23.147 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:51:23.147 DEBUG:teuthology.orchestra.run.smithi123:> route 2024-09-16T15:51:23.199 INFO:teuthology.orchestra.run.smithi123.stdout:Kernel IP routing table 2024-09-16T15:51:23.200 INFO:teuthology.orchestra.run.smithi123.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-09-16T15:51:23.200 INFO:teuthology.orchestra.run.smithi123.stdout:default _gateway 0.0.0.0 UG 0 0 0 enp3s0f1 2024-09-16T15:51:23.200 INFO:teuthology.orchestra.run.smithi123.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-09-16T15:51:23.200 INFO:teuthology.orchestra.run.smithi123.stdout:158.69.67.47 _gateway 255.255.255.255 UGH 100 0 0 enp3s0f1 2024-09-16T15:51:23.200 INFO:teuthology.orchestra.run.smithi123.stdout:172.17.0.0 0.0.0.0 255.255.0.0 U 0 0 0 docker0 2024-09-16T15:51:23.200 INFO:teuthology.orchestra.run.smithi123.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-09-16T15:51:23.200 INFO:teuthology.orchestra.run.smithi123.stdout:ns1.front.sepia 0.0.0.0 255.255.255.255 UH 100 0 0 enp3s0f1 2024-09-16T15:51:23.200 INFO:teuthology.orchestra.run.smithi123.stdout:172.21.0.2 0.0.0.0 255.255.255.255 UH 100 0 0 enp3s0f1 2024-09-16T15:51:23.200 INFO:teuthology.orchestra.run.smithi123.stdout:_gateway 0.0.0.0 255.255.255.255 UH 100 0 0 enp3s0f1 2024-09-16T15:51:23.200 INFO:teuthology.orchestra.run.smithi123.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-09-16T15:51:23.201 DEBUG:teuthology.orchestra.run.smithi123:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-09-16T15:51:23.201 DEBUG:teuthology.orchestra.run.smithi123:> set -e 2024-09-16T15:51:23.201 DEBUG:teuthology.orchestra.run.smithi123:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-09-16T15:51:23.201 DEBUG:teuthology.orchestra.run.smithi123:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-09-16T15:51:23.201 DEBUG:teuthology.orchestra.run.smithi123:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-09-16T15:51:23.201 DEBUG:teuthology.orchestra.run.smithi123:> ') 2024-09-16T15:51:23.286 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:23 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-09-16T15:51:23.314 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:23 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-16T15:51:23.321 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:51:23.322 DEBUG:teuthology.orchestra.run.smithi123:> ip netns list 2024-09-16T15:51:23.368 INFO:teuthology.orchestra.run.smithi123.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-09-16T15:51:23.368 INFO:teuthology.orchestra.run.smithi123.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-09-16T15:51:23.368 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-09-16T15:51:23.368 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:51:23.368 DEBUG:teuthology.orchestra.run.smithi123:> sudo ip link set brx.0 up 2024-09-16T15:51:23.424 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2024-09-16T15:51:23.424 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-09-16T15:51:23.424 DEBUG:teuthology.orchestra.run.smithi123:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-09-16T15:51:23.472 INFO:teuthology.orchestra.run.smithi123.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-09-16T15:51:23.472 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-09-16T15:51:23.472 DEBUG:teuthology.orchestra.run.smithi123:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-09-16T15:51:23.519 DEBUG:teuthology.orchestra.run.smithi123:> sudo modprobe fuse 2024-09-16T15:51:23.577 DEBUG:teuthology.orchestra.run.smithi123:> cat /proc/self/mounts | awk '{print $2}' 2024-09-16T15:51:23.629 INFO:teuthology.orchestra.run.smithi123.stdout:/sys 2024-09-16T15:51:23.629 INFO:teuthology.orchestra.run.smithi123.stdout:/proc 2024-09-16T15:51:23.629 INFO:teuthology.orchestra.run.smithi123.stdout:/dev 2024-09-16T15:51:23.629 INFO:teuthology.orchestra.run.smithi123.stdout:/dev/pts 2024-09-16T15:51:23.629 INFO:teuthology.orchestra.run.smithi123.stdout:/run 2024-09-16T15:51:23.630 INFO:teuthology.orchestra.run.smithi123.stdout:/ 2024-09-16T15:51:23.630 INFO:teuthology.orchestra.run.smithi123.stdout:/sys/kernel/security 2024-09-16T15:51:23.630 INFO:teuthology.orchestra.run.smithi123.stdout:/dev/shm 2024-09-16T15:51:23.630 INFO:teuthology.orchestra.run.smithi123.stdout:/run/lock 2024-09-16T15:51:23.630 INFO:teuthology.orchestra.run.smithi123.stdout:/sys/fs/cgroup 2024-09-16T15:51:23.630 INFO:teuthology.orchestra.run.smithi123.stdout:/sys/fs/pstore 2024-09-16T15:51:23.630 INFO:teuthology.orchestra.run.smithi123.stdout:/sys/fs/bpf 2024-09-16T15:51:23.630 INFO:teuthology.orchestra.run.smithi123.stdout:/proc/sys/fs/binfmt_misc 2024-09-16T15:51:23.630 INFO:teuthology.orchestra.run.smithi123.stdout:/dev/hugepages 2024-09-16T15:51:23.630 INFO:teuthology.orchestra.run.smithi123.stdout:/dev/mqueue 2024-09-16T15:51:23.630 INFO:teuthology.orchestra.run.smithi123.stdout:/sys/kernel/debug 2024-09-16T15:51:23.630 INFO:teuthology.orchestra.run.smithi123.stdout:/sys/kernel/tracing 2024-09-16T15:51:23.630 INFO:teuthology.orchestra.run.smithi123.stdout:/proc/fs/nfsd 2024-09-16T15:51:23.630 INFO:teuthology.orchestra.run.smithi123.stdout:/sys/fs/fuse/connections 2024-09-16T15:51:23.631 INFO:teuthology.orchestra.run.smithi123.stdout:/sys/kernel/config 2024-09-16T15:51:23.631 INFO:teuthology.orchestra.run.smithi123.stdout:/run/credentials/systemd-sysusers.service 2024-09-16T15:51:23.631 INFO:teuthology.orchestra.run.smithi123.stdout:/run/rpc_pipefs 2024-09-16T15:51:23.631 INFO:teuthology.orchestra.run.smithi123.stdout:/run/user/1000 2024-09-16T15:51:23.631 INFO:teuthology.orchestra.run.smithi123.stdout:/var/lib/ceph 2024-09-16T15:51:23.631 INFO:teuthology.orchestra.run.smithi123.stdout:/var/lib/ceph/osd/ceph-0 2024-09-16T15:51:23.631 INFO:teuthology.orchestra.run.smithi123.stdout:/var/lib/ceph/osd/ceph-1 2024-09-16T15:51:23.631 INFO:teuthology.orchestra.run.smithi123.stdout:/var/lib/ceph/osd/ceph-2 2024-09-16T15:51:23.631 INFO:teuthology.orchestra.run.smithi123.stdout:/var/lib/ceph/osd/ceph-3 2024-09-16T15:51:23.631 INFO:teuthology.orchestra.run.smithi123.stdout:/run/netns 2024-09-16T15:51:23.631 INFO:teuthology.orchestra.run.smithi123.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-09-16T15:51:23.631 INFO:teuthology.orchestra.run.smithi123.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-09-16T15:51:23.631 INFO:teuthology.orchestra.run.smithi123.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-09-16T15:51:23.631 INFO:teuthology.orchestra.run.smithi123.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-09-16T15:51:23.632 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:51:23.632 DEBUG:teuthology.orchestra.run.smithi123:> ls /sys/fs/fuse/connections 2024-09-16T15:51:23.675 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-09-16T15:51:23.676 DEBUG:teuthology.orchestra.run.smithi123:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2024-09-16T15:51:23.720 DEBUG:teuthology.orchestra.run.smithi123:> sudo modprobe fuse 2024-09-16T15:51:23.734 DEBUG:teuthology.orchestra.run.smithi123:> cat /proc/self/mounts | awk '{print $2}' 2024-09-16T15:51:23.780 INFO:teuthology.orchestra.run.smithi123.stdout:/sys 2024-09-16T15:51:23.780 INFO:teuthology.orchestra.run.smithi123.stdout:/proc 2024-09-16T15:51:23.780 INFO:teuthology.orchestra.run.smithi123.stdout:/dev 2024-09-16T15:51:23.780 INFO:teuthology.orchestra.run.smithi123.stdout:/dev/pts 2024-09-16T15:51:23.780 INFO:teuthology.orchestra.run.smithi123.stdout:/run 2024-09-16T15:51:23.780 INFO:teuthology.orchestra.run.smithi123.stdout:/ 2024-09-16T15:51:23.780 INFO:teuthology.orchestra.run.smithi123.stdout:/sys/kernel/security 2024-09-16T15:51:23.781 INFO:teuthology.orchestra.run.smithi123.stdout:/dev/shm 2024-09-16T15:51:23.781 INFO:teuthology.orchestra.run.smithi123.stdout:/run/lock 2024-09-16T15:51:23.781 INFO:teuthology.orchestra.run.smithi123.stdout:/sys/fs/cgroup 2024-09-16T15:51:23.781 INFO:teuthology.orchestra.run.smithi123.stdout:/sys/fs/pstore 2024-09-16T15:51:23.781 INFO:teuthology.orchestra.run.smithi123.stdout:/sys/fs/bpf 2024-09-16T15:51:23.781 INFO:teuthology.orchestra.run.smithi123.stdout:/proc/sys/fs/binfmt_misc 2024-09-16T15:51:23.781 INFO:teuthology.orchestra.run.smithi123.stdout:/dev/hugepages 2024-09-16T15:51:23.781 INFO:teuthology.orchestra.run.smithi123.stdout:/dev/mqueue 2024-09-16T15:51:23.781 INFO:teuthology.orchestra.run.smithi123.stdout:/sys/kernel/debug 2024-09-16T15:51:23.781 INFO:teuthology.orchestra.run.smithi123.stdout:/sys/kernel/tracing 2024-09-16T15:51:23.781 INFO:teuthology.orchestra.run.smithi123.stdout:/proc/fs/nfsd 2024-09-16T15:51:23.781 INFO:teuthology.orchestra.run.smithi123.stdout:/sys/fs/fuse/connections 2024-09-16T15:51:23.781 INFO:teuthology.orchestra.run.smithi123.stdout:/sys/kernel/config 2024-09-16T15:51:23.781 INFO:teuthology.orchestra.run.smithi123.stdout:/run/credentials/systemd-sysusers.service 2024-09-16T15:51:23.782 INFO:teuthology.orchestra.run.smithi123.stdout:/run/rpc_pipefs 2024-09-16T15:51:23.782 INFO:teuthology.orchestra.run.smithi123.stdout:/run/user/1000 2024-09-16T15:51:23.782 INFO:teuthology.orchestra.run.smithi123.stdout:/var/lib/ceph 2024-09-16T15:51:23.782 INFO:teuthology.orchestra.run.smithi123.stdout:/var/lib/ceph/osd/ceph-0 2024-09-16T15:51:23.782 INFO:teuthology.orchestra.run.smithi123.stdout:/var/lib/ceph/osd/ceph-1 2024-09-16T15:51:23.782 INFO:teuthology.orchestra.run.smithi123.stdout:/var/lib/ceph/osd/ceph-2 2024-09-16T15:51:23.782 INFO:teuthology.orchestra.run.smithi123.stdout:/var/lib/ceph/osd/ceph-3 2024-09-16T15:51:23.782 INFO:teuthology.orchestra.run.smithi123.stdout:/run/netns 2024-09-16T15:51:23.782 INFO:teuthology.orchestra.run.smithi123.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-09-16T15:51:23.782 INFO:teuthology.orchestra.run.smithi123.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-09-16T15:51:23.782 INFO:teuthology.orchestra.run.smithi123.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-09-16T15:51:23.782 INFO:teuthology.orchestra.run.smithi123.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-09-16T15:51:23.783 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:51:23.783 DEBUG:teuthology.orchestra.run.smithi123:> ls /sys/fs/fuse/connections 2024-09-16T15:51:23.802 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi123.stderr:2024-09-16T15:51:23.802+0000 7fe9c3e21580 -1 init, newargv = 0x5618baf67f80 newargc=15 2024-09-16T15:51:23.802 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi123.stderr:ceph-fuse[53266]: starting ceph client 2024-09-16T15:51:23.809 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi123.stderr:ceph-fuse[53266]: starting fuse 2024-09-16T15:51:24.908 DEBUG:teuthology.orchestra.run.smithi123:> sudo modprobe fuse 2024-09-16T15:51:24.925 DEBUG:teuthology.orchestra.run.smithi123:> cat /proc/self/mounts | awk '{print $2}' 2024-09-16T15:51:24.973 INFO:teuthology.orchestra.run.smithi123.stdout:/sys 2024-09-16T15:51:24.973 INFO:teuthology.orchestra.run.smithi123.stdout:/proc 2024-09-16T15:51:24.973 INFO:teuthology.orchestra.run.smithi123.stdout:/dev 2024-09-16T15:51:24.973 INFO:teuthology.orchestra.run.smithi123.stdout:/dev/pts 2024-09-16T15:51:24.973 INFO:teuthology.orchestra.run.smithi123.stdout:/run 2024-09-16T15:51:24.973 INFO:teuthology.orchestra.run.smithi123.stdout:/ 2024-09-16T15:51:24.973 INFO:teuthology.orchestra.run.smithi123.stdout:/sys/kernel/security 2024-09-16T15:51:24.973 INFO:teuthology.orchestra.run.smithi123.stdout:/dev/shm 2024-09-16T15:51:24.973 INFO:teuthology.orchestra.run.smithi123.stdout:/run/lock 2024-09-16T15:51:24.973 INFO:teuthology.orchestra.run.smithi123.stdout:/sys/fs/cgroup 2024-09-16T15:51:24.973 INFO:teuthology.orchestra.run.smithi123.stdout:/sys/fs/pstore 2024-09-16T15:51:24.973 INFO:teuthology.orchestra.run.smithi123.stdout:/sys/fs/bpf 2024-09-16T15:51:24.974 INFO:teuthology.orchestra.run.smithi123.stdout:/proc/sys/fs/binfmt_misc 2024-09-16T15:51:24.974 INFO:teuthology.orchestra.run.smithi123.stdout:/dev/hugepages 2024-09-16T15:51:24.974 INFO:teuthology.orchestra.run.smithi123.stdout:/dev/mqueue 2024-09-16T15:51:24.974 INFO:teuthology.orchestra.run.smithi123.stdout:/sys/kernel/debug 2024-09-16T15:51:24.974 INFO:teuthology.orchestra.run.smithi123.stdout:/sys/kernel/tracing 2024-09-16T15:51:24.974 INFO:teuthology.orchestra.run.smithi123.stdout:/proc/fs/nfsd 2024-09-16T15:51:24.974 INFO:teuthology.orchestra.run.smithi123.stdout:/sys/fs/fuse/connections 2024-09-16T15:51:24.974 INFO:teuthology.orchestra.run.smithi123.stdout:/sys/kernel/config 2024-09-16T15:51:24.974 INFO:teuthology.orchestra.run.smithi123.stdout:/run/credentials/systemd-sysusers.service 2024-09-16T15:51:24.974 INFO:teuthology.orchestra.run.smithi123.stdout:/run/rpc_pipefs 2024-09-16T15:51:24.974 INFO:teuthology.orchestra.run.smithi123.stdout:/run/user/1000 2024-09-16T15:51:24.974 INFO:teuthology.orchestra.run.smithi123.stdout:/var/lib/ceph 2024-09-16T15:51:24.974 INFO:teuthology.orchestra.run.smithi123.stdout:/var/lib/ceph/osd/ceph-0 2024-09-16T15:51:24.974 INFO:teuthology.orchestra.run.smithi123.stdout:/var/lib/ceph/osd/ceph-1 2024-09-16T15:51:24.974 INFO:teuthology.orchestra.run.smithi123.stdout:/var/lib/ceph/osd/ceph-2 2024-09-16T15:51:24.975 INFO:teuthology.orchestra.run.smithi123.stdout:/var/lib/ceph/osd/ceph-3 2024-09-16T15:51:24.975 INFO:teuthology.orchestra.run.smithi123.stdout:/run/netns 2024-09-16T15:51:24.975 INFO:teuthology.orchestra.run.smithi123.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-09-16T15:51:24.975 INFO:teuthology.orchestra.run.smithi123.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-09-16T15:51:24.975 INFO:teuthology.orchestra.run.smithi123.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-09-16T15:51:24.975 INFO:teuthology.orchestra.run.smithi123.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-09-16T15:51:24.975 INFO:teuthology.orchestra.run.smithi123.stdout:/home/ubuntu/cephtest/mnt.0 2024-09-16T15:51:24.976 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:51:24.976 DEBUG:teuthology.orchestra.run.smithi123:> ls /sys/fs/fuse/connections 2024-09-16T15:51:25.023 INFO:teuthology.orchestra.run.smithi123.stdout:45 2024-09-16T15:51:25.024 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [45] 2024-09-16T15:51:25.024 DEBUG:teuthology.orchestra.run.smithi123:> sudo stdin-killer -- python3 -c ' 2024-09-16T15:51:25.024 DEBUG:teuthology.orchestra.run.smithi123:> import glob 2024-09-16T15:51:25.024 DEBUG:teuthology.orchestra.run.smithi123:> import re 2024-09-16T15:51:25.024 DEBUG:teuthology.orchestra.run.smithi123:> import os 2024-09-16T15:51:25.024 DEBUG:teuthology.orchestra.run.smithi123:> import subprocess 2024-09-16T15:51:25.024 DEBUG:teuthology.orchestra.run.smithi123:> 2024-09-16T15:51:25.024 DEBUG:teuthology.orchestra.run.smithi123:> def _find_admin_socket(client_name): 2024-09-16T15:51:25.024 DEBUG:teuthology.orchestra.run.smithi123:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2024-09-16T15:51:25.024 DEBUG:teuthology.orchestra.run.smithi123:> files = glob.glob(asok_path) 2024-09-16T15:51:25.024 DEBUG:teuthology.orchestra.run.smithi123:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-09-16T15:51:25.025 DEBUG:teuthology.orchestra.run.smithi123:> 2024-09-16T15:51:25.025 DEBUG:teuthology.orchestra.run.smithi123:> # Given a non-glob path, it better be there 2024-09-16T15:51:25.025 DEBUG:teuthology.orchestra.run.smithi123:> if "*" not in asok_path: 2024-09-16T15:51:25.025 DEBUG:teuthology.orchestra.run.smithi123:> assert(len(files) == 1) 2024-09-16T15:51:25.025 DEBUG:teuthology.orchestra.run.smithi123:> return files[0] 2024-09-16T15:51:25.025 DEBUG:teuthology.orchestra.run.smithi123:> 2024-09-16T15:51:25.025 DEBUG:teuthology.orchestra.run.smithi123:> for f in files: 2024-09-16T15:51:25.025 DEBUG:teuthology.orchestra.run.smithi123:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-09-16T15:51:25.025 DEBUG:teuthology.orchestra.run.smithi123:> if os.path.exists("/proc/{0}".format(pid)): 2024-09-16T15:51:25.025 DEBUG:teuthology.orchestra.run.smithi123:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-09-16T15:51:25.025 DEBUG:teuthology.orchestra.run.smithi123:> contents = proc_f.read() 2024-09-16T15:51:25.025 DEBUG:teuthology.orchestra.run.smithi123:> if mountpoint in contents: 2024-09-16T15:51:25.025 DEBUG:teuthology.orchestra.run.smithi123:> return f 2024-09-16T15:51:25.025 DEBUG:teuthology.orchestra.run.smithi123:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-09-16T15:51:25.025 DEBUG:teuthology.orchestra.run.smithi123:> 2024-09-16T15:51:25.026 DEBUG:teuthology.orchestra.run.smithi123:> print(_find_admin_socket("client.0")) 2024-09-16T15:51:25.026 DEBUG:teuthology.orchestra.run.smithi123:> ' 2024-09-16T15:51:25.148 INFO:teuthology.orchestra.run.smithi123.stdout:/var/run/ceph/ceph-client.0.53266.asok 2024-09-16T15:51:25.151 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:25 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-16T15:51:25.159 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.53266.asok 2024-09-16T15:51:25.160 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:51:25.160 DEBUG:teuthology.orchestra.run.smithi123:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.53266.asok status 2024-09-16T15:51:25.314 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2024-09-16T15:51:25.314 INFO:teuthology.orchestra.run.smithi123.stdout: "metadata": { 2024-09-16T15:51:25.314 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph_sha1": "33f95d2ae1bcbe7eb04404303f179eeeaa6982df", 2024-09-16T15:51:25.314 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph_version": "ceph version 19.3.0-4938-g33f95d2a (33f95d2ae1bcbe7eb04404303f179eeeaa6982df) squid (dev)", 2024-09-16T15:51:25.314 INFO:teuthology.orchestra.run.smithi123.stdout: "entity_id": "0", 2024-09-16T15:51:25.315 INFO:teuthology.orchestra.run.smithi123.stdout: "hostname": "smithi123", 2024-09-16T15:51:25.315 INFO:teuthology.orchestra.run.smithi123.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2024-09-16T15:51:25.315 INFO:teuthology.orchestra.run.smithi123.stdout: "pid": "53266", 2024-09-16T15:51:25.315 INFO:teuthology.orchestra.run.smithi123.stdout: "root": "/" 2024-09-16T15:51:25.315 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:51:25.315 INFO:teuthology.orchestra.run.smithi123.stdout: "dentry_count": 0, 2024-09-16T15:51:25.315 INFO:teuthology.orchestra.run.smithi123.stdout: "dentry_pinned_count": 0, 2024-09-16T15:51:25.315 INFO:teuthology.orchestra.run.smithi123.stdout: "id": 8997, 2024-09-16T15:51:25.315 INFO:teuthology.orchestra.run.smithi123.stdout: "inst": { 2024-09-16T15:51:25.315 INFO:teuthology.orchestra.run.smithi123.stdout: "name": { 2024-09-16T15:51:25.315 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "client", 2024-09-16T15:51:25.315 INFO:teuthology.orchestra.run.smithi123.stdout: "num": 8997 2024-09-16T15:51:25.315 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:51:25.315 INFO:teuthology.orchestra.run.smithi123.stdout: "addr": { 2024-09-16T15:51:25.316 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "v1", 2024-09-16T15:51:25.316 INFO:teuthology.orchestra.run.smithi123.stdout: "addr": "192.168.0.1:0", 2024-09-16T15:51:25.316 INFO:teuthology.orchestra.run.smithi123.stdout: "nonce": 1023649512 2024-09-16T15:51:25.316 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-16T15:51:25.316 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:51:25.316 INFO:teuthology.orchestra.run.smithi123.stdout: "addr": { 2024-09-16T15:51:25.316 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "v1", 2024-09-16T15:51:25.316 INFO:teuthology.orchestra.run.smithi123.stdout: "addr": "192.168.0.1:0", 2024-09-16T15:51:25.316 INFO:teuthology.orchestra.run.smithi123.stdout: "nonce": 1023649512 2024-09-16T15:51:25.316 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:51:25.316 INFO:teuthology.orchestra.run.smithi123.stdout: "inst_str": "client.8997 192.168.0.1:0/1023649512", 2024-09-16T15:51:25.316 INFO:teuthology.orchestra.run.smithi123.stdout: "addr_str": "192.168.0.1:0/1023649512", 2024-09-16T15:51:25.316 INFO:teuthology.orchestra.run.smithi123.stdout: "inode_count": 1, 2024-09-16T15:51:25.316 INFO:teuthology.orchestra.run.smithi123.stdout: "mds_epoch": 105, 2024-09-16T15:51:25.317 INFO:teuthology.orchestra.run.smithi123.stdout: "osd_epoch": 157, 2024-09-16T15:51:25.317 INFO:teuthology.orchestra.run.smithi123.stdout: "osd_epoch_barrier": 0, 2024-09-16T15:51:25.317 INFO:teuthology.orchestra.run.smithi123.stdout: "blocklisted": false, 2024-09-16T15:51:25.317 INFO:teuthology.orchestra.run.smithi123.stdout: "fs_name": "cephfs" 2024-09-16T15:51:25.317 INFO:teuthology.orchestra.run.smithi123.stdout:} 2024-09-16T15:51:25.327 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:51:25.328 DEBUG:teuthology.orchestra.run.smithi123:> stat --file-system '--printf=%T 2024-09-16T15:51:25.328 DEBUG:teuthology.orchestra.run.smithi123:> ' -- /home/ubuntu/cephtest/mnt.0 2024-09-16T15:51:25.334 INFO:teuthology.orchestra.run.smithi123.stdout:fuseblk 2024-09-16T15:51:25.334 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2024-09-16T15:51:25.334 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:51:25.335 DEBUG:teuthology.orchestra.run.smithi123:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-09-16T15:51:25.389 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-09-16T15:51:25.568 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:25.566+0000 7f93bae9b640 1 -- 172.21.15.123:0/1800311035 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f93b406f5f0 msgr2=0x7f93b415aa60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:51:25.568 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:25.566+0000 7f93bae9b640 1 --2- 172.21.15.123:0/1800311035 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f93b406f5f0 0x7f93b415aa60 secure :-1 s=READY pgs=1768 cs=0 l=1 rev1=1 crypto rx=0x7f93ac009900 tx=0x7f93ac01c850 comp rx=0 tx=0).stop 2024-09-16T15:51:25.568 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:25.566+0000 7f93bae9b640 1 -- 172.21.15.123:0/1800311035 shutdown_connections 2024-09-16T15:51:25.568 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:25.566+0000 7f93bae9b640 1 --2- 172.21.15.123:0/1800311035 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f93b406f5f0 0x7f93b415aa60 unknown :-1 s=CLOSED pgs=1768 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:25.568 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:25.566+0000 7f93bae9b640 1 --2- 172.21.15.123:0/1800311035 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f93b4158120 0x7f93b40ca320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:25.568 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:25.566+0000 7f93bae9b640 1 -- 172.21.15.123:0/1800311035 >> 172.21.15.123:0/1800311035 conn(0x7f93b40cfb10 msgr2=0x7f93b40cff20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:51:25.568 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:25.566+0000 7f93bae9b640 1 -- 172.21.15.123:0/1800311035 shutdown_connections 2024-09-16T15:51:25.568 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:25.566+0000 7f93bae9b640 1 -- 172.21.15.123:0/1800311035 wait complete. 2024-09-16T15:51:25.571 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:25.566+0000 7f93bae9b640 1 Processor -- start 2024-09-16T15:51:25.571 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:25.566+0000 7f93bae9b640 1 -- start start 2024-09-16T15:51:25.571 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:25.570+0000 7f93bae9b640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f93b406f5f0 0x7f93b414f470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:51:25.571 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:25.570+0000 7f93bae9b640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f93b40cf5a0 0x7f93b41fbec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:51:25.571 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:25.570+0000 7f93bae9b640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f93b4158120 0x7f93b4156720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:51:25.571 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:25.570+0000 7f93bae9b640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f93b4167b70 con 0x7f93b40cf5a0 2024-09-16T15:51:25.571 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:25.570+0000 7f93bae9b640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f93b41679f0 con 0x7f93b4158120 2024-09-16T15:51:25.571 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:25.570+0000 7f93bae9b640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f93b4167cf0 con 0x7f93b406f5f0 2024-09-16T15:51:25.572 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:25.570+0000 7f93b3fff640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f93b40cf5a0 0x7f93b41fbec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:51:25.572 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:25.570+0000 7f93b3fff640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f93b40cf5a0 0x7f93b41fbec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:57676/0 (socket says 172.21.15.123:57676) 2024-09-16T15:51:25.572 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:25.570+0000 7f93b3fff640 1 -- 172.21.15.123:0/828140078 learned_addr learned my addr 172.21.15.123:0/828140078 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:51:25.572 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:25.570+0000 7f93b3fff640 1 -- 172.21.15.123:0/828140078 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f93b406f5f0 msgr2=0x7f93b414f470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:51:25.572 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:25.570+0000 7f93b3fff640 1 --2- 172.21.15.123:0/828140078 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f93b406f5f0 0x7f93b414f470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:25.572 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:25.570+0000 7f93b9438640 1 --2- 172.21.15.123:0/828140078 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f93b4158120 0x7f93b4156720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:51:25.572 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:25.570+0000 7f93b3fff640 1 -- 172.21.15.123:0/828140078 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f93b4158120 msgr2=0x7f93b4156720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:51:25.572 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:25.570+0000 7f93b3fff640 1 --2- 172.21.15.123:0/828140078 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f93b4158120 0x7f93b4156720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:25.572 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:25.570+0000 7f93b3fff640 1 -- 172.21.15.123:0/828140078 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f93b40b55c0 con 0x7f93b40cf5a0 2024-09-16T15:51:25.572 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:25.570+0000 7f93b3fff640 1 --2- 172.21.15.123:0/828140078 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f93b40cf5a0 0x7f93b41fbec0 secure :-1 s=READY pgs=1769 cs=0 l=1 rev1=1 crypto rx=0x7f93a800c440 tx=0x7f93a800c900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:51:25.572 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:25.570+0000 7f93b17fa640 1 -- 172.21.15.123:0/828140078 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93a8017070 con 0x7f93b40cf5a0 2024-09-16T15:51:25.572 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:25.570+0000 7f93b17fa640 1 -- 172.21.15.123:0/828140078 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f93a8003600 con 0x7f93b40cf5a0 2024-09-16T15:51:25.572 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:25.570+0000 7f93bae9b640 1 -- 172.21.15.123:0/828140078 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f93b41fb6c0 con 0x7f93b40cf5a0 2024-09-16T15:51:25.573 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:25.570+0000 7f93b17fa640 1 -- 172.21.15.123:0/828140078 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93a8003a30 con 0x7f93b40cf5a0 2024-09-16T15:51:25.573 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:25.570+0000 7f93bae9b640 1 -- 172.21.15.123:0/828140078 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f93b41fc400 con 0x7f93b40cf5a0 2024-09-16T15:51:25.573 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:25.570+0000 7f93bae9b640 1 -- 172.21.15.123:0/828140078 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f93b4156000 con 0x7f93b40cf5a0 2024-09-16T15:51:25.573 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:25.570+0000 7f93b17fa640 1 -- 172.21.15.123:0/828140078 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 24) ==== 97845+0+0 (secure 0 0 0) 0x7f93a801a070 con 0x7f93b40cf5a0 2024-09-16T15:51:25.573 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:25.570+0000 7f93b17fa640 1 --2- 172.21.15.123:0/828140078 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f93a0073730 0x7f93a0075bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:51:25.573 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:25.570+0000 7f93b17fa640 1 -- 172.21.15.123:0/828140078 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(157..157 src has 1..157) ==== 5737+0+0 (secure 0 0 0) 0x7f93a8092160 con 0x7f93b40cf5a0 2024-09-16T15:51:25.573 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:25.570+0000 7f93b8c37640 1 --2- 172.21.15.123:0/828140078 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f93a0073730 0x7f93a0075bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:51:25.573 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:25.570+0000 7f93b8c37640 1 --2- 172.21.15.123:0/828140078 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f93a0073730 0x7f93a0075bf0 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f93a4005fd0 tx=0x7f93a4005f60 comp rx=0 tx=0).ready entity=mgr.4150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:51:25.573 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:25.574+0000 7f93b17fa640 1 -- 172.21.15.123:0/828140078 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f93a80010d0 con 0x7f93b40cf5a0 2024-09-16T15:51:25.786 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:25.786+0000 7f93bae9b640 1 -- 172.21.15.123:0/828140078 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f93b40d06c0 con 0x7f93b40cf5a0 2024-09-16T15:51:25.786 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:25.786+0000 7f93b17fa640 1 -- 172.21.15.123:0/828140078 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v105) ==== 53+0+73 (secure 0 0 0) 0x7f93a805e7f0 con 0x7f93b40cf5a0 2024-09-16T15:51:25.786 INFO:teuthology.orchestra.run.smithi123.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-09-16T15:51:25.789 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:25.790+0000 7f93bae9b640 1 -- 172.21.15.123:0/828140078 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f93a0073730 msgr2=0x7f93a0075bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:51:25.789 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:25.790+0000 7f93bae9b640 1 --2- 172.21.15.123:0/828140078 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f93a0073730 0x7f93a0075bf0 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f93a4005fd0 tx=0x7f93a4005f60 comp rx=0 tx=0).stop 2024-09-16T15:51:25.789 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:25.790+0000 7f93bae9b640 1 -- 172.21.15.123:0/828140078 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f93b40cf5a0 msgr2=0x7f93b41fbec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:51:25.789 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:25.790+0000 7f93bae9b640 1 --2- 172.21.15.123:0/828140078 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f93b40cf5a0 0x7f93b41fbec0 secure :-1 s=READY pgs=1769 cs=0 l=1 rev1=1 crypto rx=0x7f93a800c440 tx=0x7f93a800c900 comp rx=0 tx=0).stop 2024-09-16T15:51:25.789 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:25.790+0000 7f93bae9b640 1 -- 172.21.15.123:0/828140078 shutdown_connections 2024-09-16T15:51:25.789 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:25.790+0000 7f93bae9b640 1 --2- 172.21.15.123:0/828140078 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f93a0073730 0x7f93a0075bf0 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:25.789 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:25.790+0000 7f93bae9b640 1 --2- 172.21.15.123:0/828140078 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f93b4158120 0x7f93b4156720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:25.789 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:25.790+0000 7f93bae9b640 1 --2- 172.21.15.123:0/828140078 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f93b40cf5a0 0x7f93b41fbec0 secure :-1 s=CLOSED pgs=1769 cs=0 l=1 rev1=1 crypto rx=0x7f93a800c440 tx=0x7f93a800c900 comp rx=0 tx=0).stop 2024-09-16T15:51:25.790 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:25.790+0000 7f93bae9b640 1 --2- 172.21.15.123:0/828140078 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f93b406f5f0 0x7f93b414f470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:25.790 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:25.790+0000 7f93bae9b640 1 -- 172.21.15.123:0/828140078 >> 172.21.15.123:0/828140078 conn(0x7f93b40cfb10 msgr2=0x7f93b40aea60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:51:25.790 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:25.790+0000 7f93bae9b640 1 -- 172.21.15.123:0/828140078 shutdown_connections 2024-09-16T15:51:25.790 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:25.790+0000 7f93bae9b640 1 -- 172.21.15.123:0/828140078 wait complete. 2024-09-16T15:51:25.805 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-09-16T15:51:25.805 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-09-16T15:51:25.805 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi184.front.sepia.ceph.com 2024-09-16T15:51:25.805 INFO:tasks.cephfs.mount:self.client.name = client.1 2024-09-16T15:51:25.805 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2024-09-16T15:51:25.805 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-09-16T15:51:25.805 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-09-16T15:51:25.805 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-09-16T15:51:25.805 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.1' netns for '/home/ubuntu/cephtest/mnt.1' 2024-09-16T15:51:25.805 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:51:25.806 DEBUG:teuthology.orchestra.run.smithi184:> ip addr 2024-09-16T15:51:25.813 INFO:teuthology.orchestra.run.smithi184.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-09-16T15:51:25.813 INFO:teuthology.orchestra.run.smithi184.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-09-16T15:51:25.813 INFO:teuthology.orchestra.run.smithi184.stdout: inet 127.0.0.1/8 scope host lo 2024-09-16T15:51:25.813 INFO:teuthology.orchestra.run.smithi184.stdout: valid_lft forever preferred_lft forever 2024-09-16T15:51:25.813 INFO:teuthology.orchestra.run.smithi184.stdout: inet6 ::1/128 scope host 2024-09-16T15:51:25.813 INFO:teuthology.orchestra.run.smithi184.stdout: valid_lft forever preferred_lft forever 2024-09-16T15:51:25.813 INFO:teuthology.orchestra.run.smithi184.stdout:2: eno1: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-09-16T15:51:25.813 INFO:teuthology.orchestra.run.smithi184.stdout: link/ether 0c:c4:7a:d9:37:16 brd ff:ff:ff:ff:ff:ff 2024-09-16T15:51:25.813 INFO:teuthology.orchestra.run.smithi184.stdout: altname enp7s0f0 2024-09-16T15:51:25.814 INFO:teuthology.orchestra.run.smithi184.stdout:3: eno2: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-09-16T15:51:25.814 INFO:teuthology.orchestra.run.smithi184.stdout: link/ether 0c:c4:7a:d9:37:17 brd ff:ff:ff:ff:ff:ff 2024-09-16T15:51:25.814 INFO:teuthology.orchestra.run.smithi184.stdout: altname enp7s0f1 2024-09-16T15:51:25.814 INFO:teuthology.orchestra.run.smithi184.stdout:4: enp3s0f0: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-09-16T15:51:25.814 INFO:teuthology.orchestra.run.smithi184.stdout: link/ether 0c:c4:7a:88:7b:8e brd ff:ff:ff:ff:ff:ff 2024-09-16T15:51:25.814 INFO:teuthology.orchestra.run.smithi184.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-09-16T15:51:25.814 INFO:teuthology.orchestra.run.smithi184.stdout: link/ether 0c:c4:7a:88:7b:8f brd ff:ff:ff:ff:ff:ff 2024-09-16T15:51:25.814 INFO:teuthology.orchestra.run.smithi184.stdout: inet 172.21.15.184/20 metric 100 brd 172.21.15.255 scope global dynamic enp3s0f1 2024-09-16T15:51:25.814 INFO:teuthology.orchestra.run.smithi184.stdout: valid_lft 41971sec preferred_lft 41971sec 2024-09-16T15:51:25.814 INFO:teuthology.orchestra.run.smithi184.stdout: inet6 fe80::ec4:7aff:fe88:7b8f/64 scope link 2024-09-16T15:51:25.814 INFO:teuthology.orchestra.run.smithi184.stdout: valid_lft forever preferred_lft forever 2024-09-16T15:51:25.814 INFO:teuthology.orchestra.run.smithi184.stdout:6: docker0: mtu 1500 qdisc noqueue state DOWN group default 2024-09-16T15:51:25.815 INFO:teuthology.orchestra.run.smithi184.stdout: link/ether 02:42:5d:0a:44:b9 brd ff:ff:ff:ff:ff:ff 2024-09-16T15:51:25.815 INFO:teuthology.orchestra.run.smithi184.stdout: inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0 2024-09-16T15:51:25.815 INFO:teuthology.orchestra.run.smithi184.stdout: valid_lft forever preferred_lft forever 2024-09-16T15:51:25.815 INFO:teuthology.orchestra.run.smithi184.stdout:7: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-09-16T15:51:25.815 INFO:teuthology.orchestra.run.smithi184.stdout: link/ether 5e:1f:84:6d:79:ae brd ff:ff:ff:ff:ff:ff 2024-09-16T15:51:25.815 INFO:teuthology.orchestra.run.smithi184.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-09-16T15:51:25.815 INFO:teuthology.orchestra.run.smithi184.stdout: valid_lft forever preferred_lft forever 2024-09-16T15:51:25.815 INFO:teuthology.orchestra.run.smithi184.stdout: inet6 fe80::5c1f:84ff:fe6d:79ae/64 scope link 2024-09-16T15:51:25.815 INFO:teuthology.orchestra.run.smithi184.stdout: valid_lft forever preferred_lft forever 2024-09-16T15:51:25.815 INFO:teuthology.orchestra.run.smithi184.stdout:8: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-09-16T15:51:25.815 INFO:teuthology.orchestra.run.smithi184.stdout: link/ether 0e:5c:f6:1f:82:2f brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.1 2024-09-16T15:51:25.815 INFO:teuthology.orchestra.run.smithi184.stdout: inet6 fe80::c5c:f6ff:fe1f:822f/64 scope link 2024-09-16T15:51:25.815 INFO:teuthology.orchestra.run.smithi184.stdout: valid_lft forever preferred_lft forever 2024-09-16T15:51:25.815 INFO:teuthology.orchestra.run.smithi184.stdout:9: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-09-16T15:51:25.815 INFO:teuthology.orchestra.run.smithi184.stdout: link/ether c6:a6:3d:69:d0:db brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.3 2024-09-16T15:51:25.816 INFO:teuthology.orchestra.run.smithi184.stdout: inet6 fe80::c4a6:3dff:fe69:d0db/64 scope link 2024-09-16T15:51:25.816 INFO:teuthology.orchestra.run.smithi184.stdout: valid_lft forever preferred_lft forever 2024-09-16T15:51:25.816 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:51:25.816 DEBUG:teuthology.orchestra.run.smithi184:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-09-16T15:51:25.867 INFO:teuthology.orchestra.run.smithi184.stdout:1 2024-09-16T15:51:25.869 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:51:25.870 DEBUG:teuthology.orchestra.run.smithi184:> route 2024-09-16T15:51:25.919 INFO:teuthology.orchestra.run.smithi184.stdout:Kernel IP routing table 2024-09-16T15:51:25.920 INFO:teuthology.orchestra.run.smithi184.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-09-16T15:51:25.920 INFO:teuthology.orchestra.run.smithi184.stdout:default _gateway 0.0.0.0 UG 0 0 0 enp3s0f1 2024-09-16T15:51:25.920 INFO:teuthology.orchestra.run.smithi184.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-09-16T15:51:25.920 INFO:teuthology.orchestra.run.smithi184.stdout:158.69.67.47 _gateway 255.255.255.255 UGH 100 0 0 enp3s0f1 2024-09-16T15:51:25.920 INFO:teuthology.orchestra.run.smithi184.stdout:172.17.0.0 0.0.0.0 255.255.0.0 U 0 0 0 docker0 2024-09-16T15:51:25.920 INFO:teuthology.orchestra.run.smithi184.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-09-16T15:51:25.920 INFO:teuthology.orchestra.run.smithi184.stdout:ns1.front.sepia 0.0.0.0 255.255.255.255 UH 100 0 0 enp3s0f1 2024-09-16T15:51:25.920 INFO:teuthology.orchestra.run.smithi184.stdout:172.21.0.2 0.0.0.0 255.255.255.255 UH 100 0 0 enp3s0f1 2024-09-16T15:51:25.920 INFO:teuthology.orchestra.run.smithi184.stdout:_gateway 0.0.0.0 255.255.255.255 UH 100 0 0 enp3s0f1 2024-09-16T15:51:25.920 INFO:teuthology.orchestra.run.smithi184.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-09-16T15:51:25.920 DEBUG:teuthology.orchestra.run.smithi184:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-09-16T15:51:25.921 DEBUG:teuthology.orchestra.run.smithi184:> set -e 2024-09-16T15:51:25.921 DEBUG:teuthology.orchestra.run.smithi184:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-09-16T15:51:25.921 DEBUG:teuthology.orchestra.run.smithi184:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-09-16T15:51:25.921 DEBUG:teuthology.orchestra.run.smithi184:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-09-16T15:51:25.921 DEBUG:teuthology.orchestra.run.smithi184:> ') 2024-09-16T15:51:26.005 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:51:26 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-09-16T15:51:26.033 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:51:26 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-16T15:51:26.040 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:51:26.040 DEBUG:teuthology.orchestra.run.smithi184:> ip netns list 2024-09-16T15:51:26.089 INFO:teuthology.orchestra.run.smithi184.stdout:ceph-ns--home-ubuntu-cephtest-mnt.3 (id: 1) 2024-09-16T15:51:26.089 INFO:teuthology.orchestra.run.smithi184.stdout:ceph-ns--home-ubuntu-cephtest-mnt.1 (id: 0) 2024-09-16T15:51:26.089 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.1' netns for '/home/ubuntu/cephtest/mnt.1' 2024-09-16T15:51:26.090 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:51:26.090 DEBUG:teuthology.orchestra.run.smithi184:> sudo ip link set brx.0 up 2024-09-16T15:51:26.145 INFO:tasks.cephfs.fuse_mount:Client client.1 config is {} 2024-09-16T15:51:26.145 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-09-16T15:51:26.145 DEBUG:teuthology.orchestra.run.smithi184:> mkdir -p -v /home/ubuntu/cephtest/mnt.1 2024-09-16T15:51:26.193 INFO:teuthology.orchestra.run.smithi184.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.1' 2024-09-16T15:51:26.193 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-09-16T15:51:26.193 DEBUG:teuthology.orchestra.run.smithi184:> chmod 0000 /home/ubuntu/cephtest/mnt.1 2024-09-16T15:51:26.240 DEBUG:teuthology.orchestra.run.smithi184:> sudo modprobe fuse 2024-09-16T15:51:26.297 DEBUG:teuthology.orchestra.run.smithi184:> cat /proc/self/mounts | awk '{print $2}' 2024-09-16T15:51:26.345 INFO:teuthology.orchestra.run.smithi184.stdout:/sys 2024-09-16T15:51:26.345 INFO:teuthology.orchestra.run.smithi184.stdout:/proc 2024-09-16T15:51:26.345 INFO:teuthology.orchestra.run.smithi184.stdout:/dev 2024-09-16T15:51:26.345 INFO:teuthology.orchestra.run.smithi184.stdout:/dev/pts 2024-09-16T15:51:26.345 INFO:teuthology.orchestra.run.smithi184.stdout:/run 2024-09-16T15:51:26.345 INFO:teuthology.orchestra.run.smithi184.stdout:/ 2024-09-16T15:51:26.346 INFO:teuthology.orchestra.run.smithi184.stdout:/sys/kernel/security 2024-09-16T15:51:26.346 INFO:teuthology.orchestra.run.smithi184.stdout:/dev/shm 2024-09-16T15:51:26.346 INFO:teuthology.orchestra.run.smithi184.stdout:/run/lock 2024-09-16T15:51:26.346 INFO:teuthology.orchestra.run.smithi184.stdout:/sys/fs/cgroup 2024-09-16T15:51:26.346 INFO:teuthology.orchestra.run.smithi184.stdout:/sys/fs/pstore 2024-09-16T15:51:26.346 INFO:teuthology.orchestra.run.smithi184.stdout:/sys/fs/bpf 2024-09-16T15:51:26.346 INFO:teuthology.orchestra.run.smithi184.stdout:/proc/sys/fs/binfmt_misc 2024-09-16T15:51:26.346 INFO:teuthology.orchestra.run.smithi184.stdout:/dev/hugepages 2024-09-16T15:51:26.346 INFO:teuthology.orchestra.run.smithi184.stdout:/dev/mqueue 2024-09-16T15:51:26.346 INFO:teuthology.orchestra.run.smithi184.stdout:/sys/kernel/debug 2024-09-16T15:51:26.346 INFO:teuthology.orchestra.run.smithi184.stdout:/sys/kernel/tracing 2024-09-16T15:51:26.346 INFO:teuthology.orchestra.run.smithi184.stdout:/proc/fs/nfsd 2024-09-16T15:51:26.346 INFO:teuthology.orchestra.run.smithi184.stdout:/sys/fs/fuse/connections 2024-09-16T15:51:26.346 INFO:teuthology.orchestra.run.smithi184.stdout:/sys/kernel/config 2024-09-16T15:51:26.346 INFO:teuthology.orchestra.run.smithi184.stdout:/run/credentials/systemd-sysusers.service 2024-09-16T15:51:26.347 INFO:teuthology.orchestra.run.smithi184.stdout:/run/rpc_pipefs 2024-09-16T15:51:26.347 INFO:teuthology.orchestra.run.smithi184.stdout:/run/user/1000 2024-09-16T15:51:26.347 INFO:teuthology.orchestra.run.smithi184.stdout:/var/lib/ceph 2024-09-16T15:51:26.347 INFO:teuthology.orchestra.run.smithi184.stdout:/var/lib/ceph/osd/ceph-4 2024-09-16T15:51:26.347 INFO:teuthology.orchestra.run.smithi184.stdout:/var/lib/ceph/osd/ceph-5 2024-09-16T15:51:26.347 INFO:teuthology.orchestra.run.smithi184.stdout:/var/lib/ceph/osd/ceph-6 2024-09-16T15:51:26.347 INFO:teuthology.orchestra.run.smithi184.stdout:/var/lib/ceph/osd/ceph-7 2024-09-16T15:51:26.347 INFO:teuthology.orchestra.run.smithi184.stdout:/run/netns 2024-09-16T15:51:26.347 INFO:teuthology.orchestra.run.smithi184.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-09-16T15:51:26.347 INFO:teuthology.orchestra.run.smithi184.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-09-16T15:51:26.347 INFO:teuthology.orchestra.run.smithi184.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-09-16T15:51:26.347 INFO:teuthology.orchestra.run.smithi184.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-09-16T15:51:26.348 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:51:26.348 DEBUG:teuthology.orchestra.run.smithi184:> ls /sys/fs/fuse/connections 2024-09-16T15:51:26.396 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-09-16T15:51:26.397 DEBUG:teuthology.orchestra.run.smithi184:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.1 --id 1) 2024-09-16T15:51:26.441 DEBUG:teuthology.orchestra.run.smithi184:> sudo modprobe fuse 2024-09-16T15:51:26.455 DEBUG:teuthology.orchestra.run.smithi184:> cat /proc/self/mounts | awk '{print $2}' 2024-09-16T15:51:26.505 INFO:teuthology.orchestra.run.smithi184.stdout:/sys 2024-09-16T15:51:26.505 INFO:teuthology.orchestra.run.smithi184.stdout:/proc 2024-09-16T15:51:26.505 INFO:teuthology.orchestra.run.smithi184.stdout:/dev 2024-09-16T15:51:26.505 INFO:teuthology.orchestra.run.smithi184.stdout:/dev/pts 2024-09-16T15:51:26.505 INFO:teuthology.orchestra.run.smithi184.stdout:/run 2024-09-16T15:51:26.505 INFO:teuthology.orchestra.run.smithi184.stdout:/ 2024-09-16T15:51:26.505 INFO:teuthology.orchestra.run.smithi184.stdout:/sys/kernel/security 2024-09-16T15:51:26.505 INFO:teuthology.orchestra.run.smithi184.stdout:/dev/shm 2024-09-16T15:51:26.506 INFO:teuthology.orchestra.run.smithi184.stdout:/run/lock 2024-09-16T15:51:26.506 INFO:teuthology.orchestra.run.smithi184.stdout:/sys/fs/cgroup 2024-09-16T15:51:26.506 INFO:teuthology.orchestra.run.smithi184.stdout:/sys/fs/pstore 2024-09-16T15:51:26.506 INFO:teuthology.orchestra.run.smithi184.stdout:/sys/fs/bpf 2024-09-16T15:51:26.506 INFO:teuthology.orchestra.run.smithi184.stdout:/proc/sys/fs/binfmt_misc 2024-09-16T15:51:26.506 INFO:teuthology.orchestra.run.smithi184.stdout:/dev/hugepages 2024-09-16T15:51:26.506 INFO:teuthology.orchestra.run.smithi184.stdout:/dev/mqueue 2024-09-16T15:51:26.506 INFO:teuthology.orchestra.run.smithi184.stdout:/sys/kernel/debug 2024-09-16T15:51:26.506 INFO:teuthology.orchestra.run.smithi184.stdout:/sys/kernel/tracing 2024-09-16T15:51:26.506 INFO:teuthology.orchestra.run.smithi184.stdout:/proc/fs/nfsd 2024-09-16T15:51:26.506 INFO:teuthology.orchestra.run.smithi184.stdout:/sys/fs/fuse/connections 2024-09-16T15:51:26.506 INFO:teuthology.orchestra.run.smithi184.stdout:/sys/kernel/config 2024-09-16T15:51:26.506 INFO:teuthology.orchestra.run.smithi184.stdout:/run/credentials/systemd-sysusers.service 2024-09-16T15:51:26.506 INFO:teuthology.orchestra.run.smithi184.stdout:/run/rpc_pipefs 2024-09-16T15:51:26.506 INFO:teuthology.orchestra.run.smithi184.stdout:/run/user/1000 2024-09-16T15:51:26.507 INFO:teuthology.orchestra.run.smithi184.stdout:/var/lib/ceph 2024-09-16T15:51:26.507 INFO:teuthology.orchestra.run.smithi184.stdout:/var/lib/ceph/osd/ceph-4 2024-09-16T15:51:26.507 INFO:teuthology.orchestra.run.smithi184.stdout:/var/lib/ceph/osd/ceph-5 2024-09-16T15:51:26.507 INFO:teuthology.orchestra.run.smithi184.stdout:/var/lib/ceph/osd/ceph-6 2024-09-16T15:51:26.507 INFO:teuthology.orchestra.run.smithi184.stdout:/var/lib/ceph/osd/ceph-7 2024-09-16T15:51:26.507 INFO:teuthology.orchestra.run.smithi184.stdout:/run/netns 2024-09-16T15:51:26.507 INFO:teuthology.orchestra.run.smithi184.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-09-16T15:51:26.507 INFO:teuthology.orchestra.run.smithi184.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-09-16T15:51:26.507 INFO:teuthology.orchestra.run.smithi184.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-09-16T15:51:26.507 INFO:teuthology.orchestra.run.smithi184.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-09-16T15:51:26.507 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:51:26.507 DEBUG:teuthology.orchestra.run.smithi184:> ls /sys/fs/fuse/connections 2024-09-16T15:51:26.520 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi184.stderr:2024-09-16T15:51:26.519+0000 7fa1c3ad1580 -1 init, newargv = 0x55cf3ec88030 newargc=15 2024-09-16T15:51:26.520 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi184.stderr:ceph-fuse[36401]: starting ceph client 2024-09-16T15:51:26.525 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi184.stderr:ceph-fuse[36401]: starting fuse 2024-09-16T15:51:27.625 DEBUG:teuthology.orchestra.run.smithi184:> sudo modprobe fuse 2024-09-16T15:51:27.642 DEBUG:teuthology.orchestra.run.smithi184:> cat /proc/self/mounts | awk '{print $2}' 2024-09-16T15:51:27.690 INFO:teuthology.orchestra.run.smithi184.stdout:/sys 2024-09-16T15:51:27.690 INFO:teuthology.orchestra.run.smithi184.stdout:/proc 2024-09-16T15:51:27.690 INFO:teuthology.orchestra.run.smithi184.stdout:/dev 2024-09-16T15:51:27.690 INFO:teuthology.orchestra.run.smithi184.stdout:/dev/pts 2024-09-16T15:51:27.690 INFO:teuthology.orchestra.run.smithi184.stdout:/run 2024-09-16T15:51:27.690 INFO:teuthology.orchestra.run.smithi184.stdout:/ 2024-09-16T15:51:27.690 INFO:teuthology.orchestra.run.smithi184.stdout:/sys/kernel/security 2024-09-16T15:51:27.690 INFO:teuthology.orchestra.run.smithi184.stdout:/dev/shm 2024-09-16T15:51:27.690 INFO:teuthology.orchestra.run.smithi184.stdout:/run/lock 2024-09-16T15:51:27.690 INFO:teuthology.orchestra.run.smithi184.stdout:/sys/fs/cgroup 2024-09-16T15:51:27.691 INFO:teuthology.orchestra.run.smithi184.stdout:/sys/fs/pstore 2024-09-16T15:51:27.691 INFO:teuthology.orchestra.run.smithi184.stdout:/sys/fs/bpf 2024-09-16T15:51:27.691 INFO:teuthology.orchestra.run.smithi184.stdout:/proc/sys/fs/binfmt_misc 2024-09-16T15:51:27.691 INFO:teuthology.orchestra.run.smithi184.stdout:/dev/hugepages 2024-09-16T15:51:27.691 INFO:teuthology.orchestra.run.smithi184.stdout:/dev/mqueue 2024-09-16T15:51:27.691 INFO:teuthology.orchestra.run.smithi184.stdout:/sys/kernel/debug 2024-09-16T15:51:27.691 INFO:teuthology.orchestra.run.smithi184.stdout:/sys/kernel/tracing 2024-09-16T15:51:27.691 INFO:teuthology.orchestra.run.smithi184.stdout:/proc/fs/nfsd 2024-09-16T15:51:27.691 INFO:teuthology.orchestra.run.smithi184.stdout:/sys/fs/fuse/connections 2024-09-16T15:51:27.691 INFO:teuthology.orchestra.run.smithi184.stdout:/sys/kernel/config 2024-09-16T15:51:27.691 INFO:teuthology.orchestra.run.smithi184.stdout:/run/credentials/systemd-sysusers.service 2024-09-16T15:51:27.692 INFO:teuthology.orchestra.run.smithi184.stdout:/run/rpc_pipefs 2024-09-16T15:51:27.692 INFO:teuthology.orchestra.run.smithi184.stdout:/run/user/1000 2024-09-16T15:51:27.692 INFO:teuthology.orchestra.run.smithi184.stdout:/var/lib/ceph 2024-09-16T15:51:27.692 INFO:teuthology.orchestra.run.smithi184.stdout:/var/lib/ceph/osd/ceph-4 2024-09-16T15:51:27.692 INFO:teuthology.orchestra.run.smithi184.stdout:/var/lib/ceph/osd/ceph-5 2024-09-16T15:51:27.692 INFO:teuthology.orchestra.run.smithi184.stdout:/var/lib/ceph/osd/ceph-6 2024-09-16T15:51:27.692 INFO:teuthology.orchestra.run.smithi184.stdout:/var/lib/ceph/osd/ceph-7 2024-09-16T15:51:27.692 INFO:teuthology.orchestra.run.smithi184.stdout:/run/netns 2024-09-16T15:51:27.692 INFO:teuthology.orchestra.run.smithi184.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-09-16T15:51:27.692 INFO:teuthology.orchestra.run.smithi184.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-09-16T15:51:27.692 INFO:teuthology.orchestra.run.smithi184.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-09-16T15:51:27.693 INFO:teuthology.orchestra.run.smithi184.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-09-16T15:51:27.693 INFO:teuthology.orchestra.run.smithi184.stdout:/home/ubuntu/cephtest/mnt.1 2024-09-16T15:51:27.693 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:51:27.693 DEBUG:teuthology.orchestra.run.smithi184:> ls /sys/fs/fuse/connections 2024-09-16T15:51:27.737 INFO:teuthology.orchestra.run.smithi184.stdout:45 2024-09-16T15:51:27.737 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [45] 2024-09-16T15:51:27.737 DEBUG:teuthology.orchestra.run.smithi184:> sudo stdin-killer -- python3 -c ' 2024-09-16T15:51:27.737 DEBUG:teuthology.orchestra.run.smithi184:> import glob 2024-09-16T15:51:27.737 DEBUG:teuthology.orchestra.run.smithi184:> import re 2024-09-16T15:51:27.738 DEBUG:teuthology.orchestra.run.smithi184:> import os 2024-09-16T15:51:27.738 DEBUG:teuthology.orchestra.run.smithi184:> import subprocess 2024-09-16T15:51:27.738 DEBUG:teuthology.orchestra.run.smithi184:> 2024-09-16T15:51:27.738 DEBUG:teuthology.orchestra.run.smithi184:> def _find_admin_socket(client_name): 2024-09-16T15:51:27.738 DEBUG:teuthology.orchestra.run.smithi184:> asok_path = "/var/run/ceph/ceph-client.1.*.asok" 2024-09-16T15:51:27.738 DEBUG:teuthology.orchestra.run.smithi184:> files = glob.glob(asok_path) 2024-09-16T15:51:27.738 DEBUG:teuthology.orchestra.run.smithi184:> mountpoint = "/home/ubuntu/cephtest/mnt.1" 2024-09-16T15:51:27.738 DEBUG:teuthology.orchestra.run.smithi184:> 2024-09-16T15:51:27.738 DEBUG:teuthology.orchestra.run.smithi184:> # Given a non-glob path, it better be there 2024-09-16T15:51:27.738 DEBUG:teuthology.orchestra.run.smithi184:> if "*" not in asok_path: 2024-09-16T15:51:27.738 DEBUG:teuthology.orchestra.run.smithi184:> assert(len(files) == 1) 2024-09-16T15:51:27.739 DEBUG:teuthology.orchestra.run.smithi184:> return files[0] 2024-09-16T15:51:27.739 DEBUG:teuthology.orchestra.run.smithi184:> 2024-09-16T15:51:27.739 DEBUG:teuthology.orchestra.run.smithi184:> for f in files: 2024-09-16T15:51:27.739 DEBUG:teuthology.orchestra.run.smithi184:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-09-16T15:51:27.739 DEBUG:teuthology.orchestra.run.smithi184:> if os.path.exists("/proc/{0}".format(pid)): 2024-09-16T15:51:27.739 DEBUG:teuthology.orchestra.run.smithi184:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-09-16T15:51:27.739 DEBUG:teuthology.orchestra.run.smithi184:> contents = proc_f.read() 2024-09-16T15:51:27.739 DEBUG:teuthology.orchestra.run.smithi184:> if mountpoint in contents: 2024-09-16T15:51:27.739 DEBUG:teuthology.orchestra.run.smithi184:> return f 2024-09-16T15:51:27.739 DEBUG:teuthology.orchestra.run.smithi184:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-09-16T15:51:27.739 DEBUG:teuthology.orchestra.run.smithi184:> 2024-09-16T15:51:27.740 DEBUG:teuthology.orchestra.run.smithi184:> print(_find_admin_socket("client.1")) 2024-09-16T15:51:27.740 DEBUG:teuthology.orchestra.run.smithi184:> ' 2024-09-16T15:51:27.861 INFO:teuthology.orchestra.run.smithi184.stdout:/var/run/ceph/ceph-client.1.36401.asok 2024-09-16T15:51:27.865 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:51:27 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-16T15:51:27.872 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.1.36401.asok 2024-09-16T15:51:27.872 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:51:27.872 DEBUG:teuthology.orchestra.run.smithi184:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.1.36401.asok status 2024-09-16T15:51:28.031 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-16T15:51:28.032 INFO:teuthology.orchestra.run.smithi184.stdout: "metadata": { 2024-09-16T15:51:28.032 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph_sha1": "33f95d2ae1bcbe7eb04404303f179eeeaa6982df", 2024-09-16T15:51:28.032 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph_version": "ceph version 19.3.0-4938-g33f95d2a (33f95d2ae1bcbe7eb04404303f179eeeaa6982df) squid (dev)", 2024-09-16T15:51:28.032 INFO:teuthology.orchestra.run.smithi184.stdout: "entity_id": "1", 2024-09-16T15:51:28.032 INFO:teuthology.orchestra.run.smithi184.stdout: "hostname": "smithi184", 2024-09-16T15:51:28.032 INFO:teuthology.orchestra.run.smithi184.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.1", 2024-09-16T15:51:28.032 INFO:teuthology.orchestra.run.smithi184.stdout: "pid": "36401", 2024-09-16T15:51:28.032 INFO:teuthology.orchestra.run.smithi184.stdout: "root": "/" 2024-09-16T15:51:28.032 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-16T15:51:28.032 INFO:teuthology.orchestra.run.smithi184.stdout: "dentry_count": 0, 2024-09-16T15:51:28.032 INFO:teuthology.orchestra.run.smithi184.stdout: "dentry_pinned_count": 0, 2024-09-16T15:51:28.033 INFO:teuthology.orchestra.run.smithi184.stdout: "id": 6233, 2024-09-16T15:51:28.033 INFO:teuthology.orchestra.run.smithi184.stdout: "inst": { 2024-09-16T15:51:28.033 INFO:teuthology.orchestra.run.smithi184.stdout: "name": { 2024-09-16T15:51:28.033 INFO:teuthology.orchestra.run.smithi184.stdout: "type": "client", 2024-09-16T15:51:28.033 INFO:teuthology.orchestra.run.smithi184.stdout: "num": 6233 2024-09-16T15:51:28.033 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-16T15:51:28.033 INFO:teuthology.orchestra.run.smithi184.stdout: "addr": { 2024-09-16T15:51:28.033 INFO:teuthology.orchestra.run.smithi184.stdout: "type": "v1", 2024-09-16T15:51:28.033 INFO:teuthology.orchestra.run.smithi184.stdout: "addr": "192.168.0.1:0", 2024-09-16T15:51:28.033 INFO:teuthology.orchestra.run.smithi184.stdout: "nonce": 1680834706 2024-09-16T15:51:28.033 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-16T15:51:28.033 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-16T15:51:28.033 INFO:teuthology.orchestra.run.smithi184.stdout: "addr": { 2024-09-16T15:51:28.033 INFO:teuthology.orchestra.run.smithi184.stdout: "type": "v1", 2024-09-16T15:51:28.034 INFO:teuthology.orchestra.run.smithi184.stdout: "addr": "192.168.0.1:0", 2024-09-16T15:51:28.034 INFO:teuthology.orchestra.run.smithi184.stdout: "nonce": 1680834706 2024-09-16T15:51:28.034 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-16T15:51:28.034 INFO:teuthology.orchestra.run.smithi184.stdout: "inst_str": "client.6233 192.168.0.1:0/1680834706", 2024-09-16T15:51:28.034 INFO:teuthology.orchestra.run.smithi184.stdout: "addr_str": "192.168.0.1:0/1680834706", 2024-09-16T15:51:28.034 INFO:teuthology.orchestra.run.smithi184.stdout: "inode_count": 1, 2024-09-16T15:51:28.034 INFO:teuthology.orchestra.run.smithi184.stdout: "mds_epoch": 105, 2024-09-16T15:51:28.034 INFO:teuthology.orchestra.run.smithi184.stdout: "osd_epoch": 157, 2024-09-16T15:51:28.034 INFO:teuthology.orchestra.run.smithi184.stdout: "osd_epoch_barrier": 0, 2024-09-16T15:51:28.034 INFO:teuthology.orchestra.run.smithi184.stdout: "blocklisted": false, 2024-09-16T15:51:28.034 INFO:teuthology.orchestra.run.smithi184.stdout: "fs_name": "cephfs" 2024-09-16T15:51:28.034 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-16T15:51:28.046 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:51:28.046 DEBUG:teuthology.orchestra.run.smithi184:> stat --file-system '--printf=%T 2024-09-16T15:51:28.046 DEBUG:teuthology.orchestra.run.smithi184:> ' -- /home/ubuntu/cephtest/mnt.1 2024-09-16T15:51:28.053 INFO:teuthology.orchestra.run.smithi184.stdout:fuseblk 2024-09-16T15:51:28.053 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2024-09-16T15:51:28.053 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:51:28.053 DEBUG:teuthology.orchestra.run.smithi184:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.1 2024-09-16T15:51:28.107 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs flag set enable_multiple true --yes-i-really-mean-it 2024-09-16T15:51:28.253 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:28.253+0000 7f819bbc6640 1 -- 172.21.15.123:0/4193305660 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f81940c5e90 msgr2=0x7f81940c6320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:51:28.253 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:28.253+0000 7f819bbc6640 1 --2- 172.21.15.123:0/4193305660 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f81940c5e90 0x7f81940c6320 secure :-1 s=READY pgs=1771 cs=0 l=1 rev1=1 crypto rx=0x7f818c009c80 tx=0x7f818c01cd70 comp rx=0 tx=0).stop 2024-09-16T15:51:28.253 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:28.253+0000 7f819bbc6640 1 -- 172.21.15.123:0/4193305660 shutdown_connections 2024-09-16T15:51:28.253 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:28.253+0000 7f819bbc6640 1 --2- 172.21.15.123:0/4193305660 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f81940c5e90 0x7f81940c6320 unknown :-1 s=CLOSED pgs=1771 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:28.254 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:28.253+0000 7f819bbc6640 1 --2- 172.21.15.123:0/4193305660 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f81940ceac0 0x7f81940cef00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:28.254 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:28.253+0000 7f819bbc6640 1 -- 172.21.15.123:0/4193305660 >> 172.21.15.123:0/4193305660 conn(0x7f81940d1bd0 msgr2=0x7f81940d1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:51:28.254 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:28.253+0000 7f819bbc6640 1 -- 172.21.15.123:0/4193305660 shutdown_connections 2024-09-16T15:51:28.254 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:28.253+0000 7f819bbc6640 1 -- 172.21.15.123:0/4193305660 wait complete. 2024-09-16T15:51:28.254 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:28.253+0000 7f819bbc6640 1 Processor -- start 2024-09-16T15:51:28.254 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:28.253+0000 7f819bbc6640 1 -- start start 2024-09-16T15:51:28.257 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:28.257+0000 7f819bbc6640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f81940c5e90 0x7f819414ef00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:51:28.257 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:28.257+0000 7f819bbc6640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f81940ceac0 0x7f81940b4c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:51:28.258 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:28.257+0000 7f819bbc6640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f819415f230 0x7f819415b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:51:28.258 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:28.257+0000 7f819bbc6640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f8194168d30 con 0x7f81940c5e90 2024-09-16T15:51:28.258 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:28.257+0000 7f819bbc6640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f8194168bb0 con 0x7f819415f230 2024-09-16T15:51:28.258 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:28.257+0000 7f819bbc6640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f8194168eb0 con 0x7f81940ceac0 2024-09-16T15:51:28.258 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:28.257+0000 7f8199962640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f81940c5e90 0x7f819414ef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:51:28.259 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:28.257+0000 7f8199962640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f81940c5e90 0x7f819414ef00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:57694/0 (socket says 172.21.15.123:57694) 2024-09-16T15:51:28.259 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:28.257+0000 7f8199962640 1 -- 172.21.15.123:0/519076605 learned_addr learned my addr 172.21.15.123:0/519076605 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:51:28.259 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:28.257+0000 7f8199962640 1 -- 172.21.15.123:0/519076605 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f81940ceac0 msgr2=0x7f81940b4c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:51:28.259 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:28.257+0000 7f8199161640 1 --2- 172.21.15.123:0/519076605 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f81940ceac0 0x7f81940b4c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:51:28.259 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:28.257+0000 7f819a163640 1 --2- 172.21.15.123:0/519076605 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f819415f230 0x7f819415b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:51:28.260 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:28.257+0000 7f8199962640 1 --2- 172.21.15.123:0/519076605 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f81940ceac0 0x7f81940b4c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:28.260 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:28.257+0000 7f8199962640 1 -- 172.21.15.123:0/519076605 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f819415f230 msgr2=0x7f819415b750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:51:28.260 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:28.257+0000 7f8199962640 1 --2- 172.21.15.123:0/519076605 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f819415f230 0x7f819415b750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:28.260 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:28.257+0000 7f8199962640 1 -- 172.21.15.123:0/519076605 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f81940b70a0 con 0x7f81940c5e90 2024-09-16T15:51:28.261 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:28.257+0000 7f8199962640 1 --2- 172.21.15.123:0/519076605 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f81940c5e90 0x7f819414ef00 secure :-1 s=READY pgs=1772 cs=0 l=1 rev1=1 crypto rx=0x7f8190001ec0 tx=0x7f819000c890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:51:28.261 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:28.257+0000 7f818a7fc640 1 -- 172.21.15.123:0/519076605 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8190017070 con 0x7f81940c5e90 2024-09-16T15:51:28.261 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:28.257+0000 7f818a7fc640 1 -- 172.21.15.123:0/519076605 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f8190003600 con 0x7f81940c5e90 2024-09-16T15:51:28.261 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:28.257+0000 7f819bbc6640 1 -- 172.21.15.123:0/519076605 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f81940b7840 con 0x7f81940c5e90 2024-09-16T15:51:28.261 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:28.257+0000 7f818a7fc640 1 -- 172.21.15.123:0/519076605 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8190003a50 con 0x7f81940c5e90 2024-09-16T15:51:28.261 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:28.257+0000 7f819bbc6640 1 -- 172.21.15.123:0/519076605 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8194155840 con 0x7f81940c5e90 2024-09-16T15:51:28.261 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:28.257+0000 7f819bbc6640 1 -- 172.21.15.123:0/519076605 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f81940c6320 con 0x7f81940c5e90 2024-09-16T15:51:28.263 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:28.257+0000 7f818a7fc640 1 -- 172.21.15.123:0/519076605 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 24) ==== 97845+0+0 (secure 0 0 0) 0x7f819001a070 con 0x7f81940c5e90 2024-09-16T15:51:28.263 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:28.257+0000 7f818a7fc640 1 --2- 172.21.15.123:0/519076605 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f8178073760 0x7f8178075c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:51:28.263 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:28.257+0000 7f8199161640 1 --2- 172.21.15.123:0/519076605 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f8178073760 0x7f8178075c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:51:28.263 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:28.257+0000 7f818a7fc640 1 -- 172.21.15.123:0/519076605 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(157..157 src has 1..157) ==== 5737+0+0 (secure 0 0 0) 0x7f8190091e50 con 0x7f81940c5e90 2024-09-16T15:51:28.263 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:28.261+0000 7f8199161640 1 --2- 172.21.15.123:0/519076605 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f8178073760 0x7f8178075c20 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f819415e3b0 tx=0x7f8184005c70 comp rx=0 tx=0).ready entity=mgr.4150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:51:28.264 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:28.261+0000 7f818a7fc640 1 -- 172.21.15.123:0/519076605 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f8190021020 con 0x7f81940c5e90 2024-09-16T15:51:28.466 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:28.465+0000 7f819bbc6640 1 -- 172.21.15.123:0/519076605 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "fs flag set", "flag_name": "enable_multiple", "val": "true", "yes_i_really_mean_it": true} v 0) -- 0x7f8194155a40 con 0x7f81940c5e90 2024-09-16T15:51:29.196 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:29.197+0000 7f818a7fc640 1 -- 172.21.15.123:0/519076605 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "fs flag set", "flag_name": "enable_multiple", "val": "true", "yes_i_really_mean_it": true}]=0 v106) ==== 136+0+0 (secure 0 0 0) 0x7f81940b70a0 con 0x7f81940c5e90 2024-09-16T15:51:29.200 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:29.201+0000 7f819bbc6640 1 -- 172.21.15.123:0/519076605 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f8178073760 msgr2=0x7f8178075c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:51:29.200 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:29.201+0000 7f819bbc6640 1 --2- 172.21.15.123:0/519076605 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f8178073760 0x7f8178075c20 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f819415e3b0 tx=0x7f8184005c70 comp rx=0 tx=0).stop 2024-09-16T15:51:29.200 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:29.201+0000 7f819bbc6640 1 -- 172.21.15.123:0/519076605 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f81940c5e90 msgr2=0x7f819414ef00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:51:29.200 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:29.201+0000 7f819bbc6640 1 --2- 172.21.15.123:0/519076605 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f81940c5e90 0x7f819414ef00 secure :-1 s=READY pgs=1772 cs=0 l=1 rev1=1 crypto rx=0x7f8190001ec0 tx=0x7f819000c890 comp rx=0 tx=0).stop 2024-09-16T15:51:29.200 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:29.201+0000 7f819bbc6640 1 -- 172.21.15.123:0/519076605 shutdown_connections 2024-09-16T15:51:29.200 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:29.201+0000 7f819bbc6640 1 --2- 172.21.15.123:0/519076605 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f8178073760 0x7f8178075c20 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:29.200 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:29.201+0000 7f819bbc6640 1 --2- 172.21.15.123:0/519076605 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f819415f230 0x7f819415b750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:29.200 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:29.201+0000 7f819bbc6640 1 --2- 172.21.15.123:0/519076605 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f81940ceac0 0x7f81940b4c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:29.201 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:29.201+0000 7f819bbc6640 1 --2- 172.21.15.123:0/519076605 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f81940c5e90 0x7f819414ef00 unknown :-1 s=CLOSED pgs=1772 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:29.201 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:29.201+0000 7f819bbc6640 1 -- 172.21.15.123:0/519076605 >> 172.21.15.123:0/519076605 conn(0x7f81940d1bd0 msgr2=0x7f81940c7d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:51:29.201 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:29.201+0000 7f819bbc6640 1 -- 172.21.15.123:0/519076605 shutdown_connections 2024-09-16T15:51:29.201 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:29.201+0000 7f819bbc6640 1 -- 172.21.15.123:0/519076605 wait complete. 2024-09-16T15:51:29.219 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2024-09-16T15:51:29.357 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:29.357+0000 7f074006b640 1 -- 172.21.15.123:0/3280087923 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f07380cf5a0 msgr2=0x7f073806eee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:51:29.357 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:29.357+0000 7f074006b640 1 --2- 172.21.15.123:0/3280087923 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f07380cf5a0 0x7f073806eee0 secure :-1 s=READY pgs=1773 cs=0 l=1 rev1=1 crypto rx=0x7f0728009900 tx=0x7f072801c7c0 comp rx=0 tx=0).stop 2024-09-16T15:51:29.357 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:29.357+0000 7f074006b640 1 -- 172.21.15.123:0/3280087923 shutdown_connections 2024-09-16T15:51:29.357 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:29.357+0000 7f074006b640 1 --2- 172.21.15.123:0/3280087923 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f073815ec40 0x7f073806f420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:29.357 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:29.357+0000 7f074006b640 1 --2- 172.21.15.123:0/3280087923 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f07380cf5a0 0x7f073806eee0 unknown :-1 s=CLOSED pgs=1773 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:29.358 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:29.357+0000 7f074006b640 1 -- 172.21.15.123:0/3280087923 >> 172.21.15.123:0/3280087923 conn(0x7f07380cfb10 msgr2=0x7f07380cff20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:51:29.358 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:29.357+0000 7f074006b640 1 -- 172.21.15.123:0/3280087923 shutdown_connections 2024-09-16T15:51:29.358 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:29.357+0000 7f074006b640 1 -- 172.21.15.123:0/3280087923 wait complete. 2024-09-16T15:51:29.358 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:29.357+0000 7f074006b640 1 Processor -- start 2024-09-16T15:51:29.358 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:29.357+0000 7f074006b640 1 -- start start 2024-09-16T15:51:29.360 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:29.357+0000 7f074006b640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f07380cf5a0 0x7f07380b29a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:51:29.360 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:29.357+0000 7f074006b640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f0738158120 0x7f07380b4cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:51:29.360 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:29.357+0000 7f074006b640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f073815ec40 0x7f07380740c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:51:29.360 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:29.357+0000 7f074006b640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f07381673d0 con 0x7f073815ec40 2024-09-16T15:51:29.360 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:29.361+0000 7f074006b640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f073815b820 con 0x7f0738158120 2024-09-16T15:51:29.361 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:29.361+0000 7f074006b640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f0738167550 con 0x7f07380cf5a0 2024-09-16T15:51:29.361 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:29.361+0000 7f073e608640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f073815ec40 0x7f07380740c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:51:29.361 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:29.361+0000 7f073e608640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f073815ec40 0x7f07380740c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:57712/0 (socket says 172.21.15.123:57712) 2024-09-16T15:51:29.361 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:29.361+0000 7f073e608640 1 -- 172.21.15.123:0/1234360286 learned_addr learned my addr 172.21.15.123:0/1234360286 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:51:29.361 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:29.361+0000 7f073de07640 1 --2- 172.21.15.123:0/1234360286 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f07380cf5a0 0x7f07380b29a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:51:29.361 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:29.361+0000 7f073d606640 1 --2- 172.21.15.123:0/1234360286 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f0738158120 0x7f07380b4cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:51:29.361 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:29.361+0000 7f073e608640 1 -- 172.21.15.123:0/1234360286 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f07380cf5a0 msgr2=0x7f07380b29a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:51:29.361 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:29.361+0000 7f073e608640 1 --2- 172.21.15.123:0/1234360286 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f07380cf5a0 0x7f07380b29a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:29.361 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:29.361+0000 7f073e608640 1 -- 172.21.15.123:0/1234360286 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f0738158120 msgr2=0x7f07380b4cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:51:29.361 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:29.361+0000 7f073e608640 1 --2- 172.21.15.123:0/1234360286 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f0738158120 0x7f07380b4cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:29.361 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:29.361+0000 7f073e608640 1 -- 172.21.15.123:0/1234360286 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f07381fade0 con 0x7f073815ec40 2024-09-16T15:51:29.362 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:29.361+0000 7f073e608640 1 --2- 172.21.15.123:0/1234360286 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f073815ec40 0x7f07380740c0 secure :-1 s=READY pgs=1774 cs=0 l=1 rev1=1 crypto rx=0x7f073000cce0 tx=0x7f073000a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:51:29.362 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:29.361+0000 7f072e7fc640 1 -- 172.21.15.123:0/1234360286 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f073000f450 con 0x7f073815ec40 2024-09-16T15:51:29.362 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:29.361+0000 7f072e7fc640 1 -- 172.21.15.123:0/1234360286 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f073000f6e0 con 0x7f073815ec40 2024-09-16T15:51:29.362 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:29.361+0000 7f072e7fc640 1 -- 172.21.15.123:0/1234360286 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f073000fb30 con 0x7f073815ec40 2024-09-16T15:51:29.362 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:29.361+0000 7f074006b640 1 -- 172.21.15.123:0/1234360286 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f07381fb0c0 con 0x7f073815ec40 2024-09-16T15:51:29.362 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:29.361+0000 7f074006b640 1 -- 172.21.15.123:0/1234360286 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f07381fb680 con 0x7f073815ec40 2024-09-16T15:51:29.362 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:29.361+0000 7f074006b640 1 -- 172.21.15.123:0/1234360286 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f07381f7530 con 0x7f073815ec40 2024-09-16T15:51:29.363 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:29.361+0000 7f072e7fc640 1 -- 172.21.15.123:0/1234360286 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 24) ==== 97845+0+0 (secure 0 0 0) 0x7f073000fcd0 con 0x7f073815ec40 2024-09-16T15:51:29.363 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:29.361+0000 7f072e7fc640 1 --2- 172.21.15.123:0/1234360286 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f071c073730 0x7f071c075bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:51:29.363 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:29.361+0000 7f072e7fc640 1 -- 172.21.15.123:0/1234360286 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(157..157 src has 1..157) ==== 5737+0+0 (secure 0 0 0) 0x7f0730093190 con 0x7f073815ec40 2024-09-16T15:51:29.363 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:29.361+0000 7f073de07640 1 --2- 172.21.15.123:0/1234360286 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f071c073730 0x7f071c075bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:51:29.363 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:29.361+0000 7f073de07640 1 --2- 172.21.15.123:0/1234360286 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f071c073730 0x7f071c075bf0 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f0734004e30 tx=0x7f0734009040 comp rx=0 tx=0).ready entity=mgr.4150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:51:29.366 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:29.365+0000 7f072e7fc640 1 -- 172.21.15.123:0/1234360286 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f0730014030 con 0x7f073815ec40 2024-09-16T15:51:29.558 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:29.557+0000 7f074006b640 1 -- 172.21.15.123:0/1234360286 --> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) -- 0x7f07380ca320 con 0x7f071c073730 2024-09-16T15:51:29.560 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:29.561+0000 7f072e7fc640 1 -- 172.21.15.123:0/1234360286 <== mgr.4150 v2:172.21.15.123:6836/2464759676 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+41 (secure 0 0 0) 0x7f07380ca320 con 0x7f071c073730 2024-09-16T15:51:29.561 INFO:teuthology.orchestra.run.smithi123.stdout:[ 2024-09-16T15:51:29.561 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-09-16T15:51:29.561 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "cephfs" 2024-09-16T15:51:29.561 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-16T15:51:29.561 INFO:teuthology.orchestra.run.smithi123.stdout:] 2024-09-16T15:51:29.563 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:29.561+0000 7f074006b640 1 -- 172.21.15.123:0/1234360286 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f071c073730 msgr2=0x7f071c075bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:51:29.563 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:29.561+0000 7f074006b640 1 --2- 172.21.15.123:0/1234360286 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f071c073730 0x7f071c075bf0 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f0734004e30 tx=0x7f0734009040 comp rx=0 tx=0).stop 2024-09-16T15:51:29.563 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:29.561+0000 7f074006b640 1 -- 172.21.15.123:0/1234360286 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f073815ec40 msgr2=0x7f07380740c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:51:29.563 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:29.561+0000 7f074006b640 1 --2- 172.21.15.123:0/1234360286 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f073815ec40 0x7f07380740c0 secure :-1 s=READY pgs=1774 cs=0 l=1 rev1=1 crypto rx=0x7f073000cce0 tx=0x7f073000a620 comp rx=0 tx=0).stop 2024-09-16T15:51:29.563 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:29.561+0000 7f074006b640 1 -- 172.21.15.123:0/1234360286 shutdown_connections 2024-09-16T15:51:29.563 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:29.561+0000 7f074006b640 1 --2- 172.21.15.123:0/1234360286 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f071c073730 0x7f071c075bf0 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:29.563 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:29.561+0000 7f074006b640 1 --2- 172.21.15.123:0/1234360286 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f073815ec40 0x7f07380740c0 unknown :-1 s=CLOSED pgs=1774 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:29.564 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:29.561+0000 7f074006b640 1 --2- 172.21.15.123:0/1234360286 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f0738158120 0x7f07380b4cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:29.564 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:29.561+0000 7f074006b640 1 --2- 172.21.15.123:0/1234360286 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f07380cf5a0 0x7f07380b29a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:29.564 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:29.565+0000 7f074006b640 1 -- 172.21.15.123:0/1234360286 >> 172.21.15.123:0/1234360286 conn(0x7f07380cfb10 msgr2=0x7f07380ae360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:51:29.564 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:29.565+0000 7f074006b640 1 -- 172.21.15.123:0/1234360286 shutdown_connections 2024-09-16T15:51:29.564 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:29.565+0000 7f074006b640 1 -- 172.21.15.123:0/1234360286 wait complete. 2024-09-16T15:51:29.579 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-09-16T15:51:29.699 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:29.697+0000 7f5ba8024640 1 -- 172.21.15.123:0/2134005535 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5ba00c3230 msgr2=0x7f5ba00cedd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:51:29.699 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:29.697+0000 7f5ba8024640 1 --2- 172.21.15.123:0/2134005535 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5ba00c3230 0x7f5ba00cedd0 secure :-1 s=READY pgs=1776 cs=0 l=1 rev1=1 crypto rx=0x7f5b94009840 tx=0x7f5b9401d1c0 comp rx=0 tx=0).stop 2024-09-16T15:51:29.699 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:29.701+0000 7f5ba8024640 1 -- 172.21.15.123:0/2134005535 shutdown_connections 2024-09-16T15:51:29.699 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:29.701+0000 7f5ba8024640 1 --2- 172.21.15.123:0/2134005535 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f5ba015bc40 0x7f5ba00cf310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:29.699 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:29.701+0000 7f5ba8024640 1 --2- 172.21.15.123:0/2134005535 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5ba00c3230 0x7f5ba00cedd0 unknown :-1 s=CLOSED pgs=1776 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:29.699 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:29.701+0000 7f5ba8024640 1 -- 172.21.15.123:0/2134005535 >> 172.21.15.123:0/2134005535 conn(0x7f5ba00b1710 msgr2=0x7f5ba00b1b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:51:29.700 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:29.701+0000 7f5ba8024640 1 -- 172.21.15.123:0/2134005535 shutdown_connections 2024-09-16T15:51:29.700 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:29.701+0000 7f5ba8024640 1 -- 172.21.15.123:0/2134005535 wait complete. 2024-09-16T15:51:29.700 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:29.701+0000 7f5ba8024640 1 Processor -- start 2024-09-16T15:51:29.700 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:29.701+0000 7f5ba8024640 1 -- start start 2024-09-16T15:51:29.702 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:29.701+0000 7f5ba8024640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5ba00c3230 0x7f5ba0147e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:51:29.702 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:29.701+0000 7f5ba8024640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5ba015b860 0x7f5ba01f8b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:51:29.702 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:29.701+0000 7f5ba8024640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f5ba015bc40 0x7f5ba00c7190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:51:29.702 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:29.701+0000 7f5ba8024640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f5ba01689c0 con 0x7f5ba00c3230 2024-09-16T15:51:29.702 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:29.701+0000 7f5ba8024640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f5ba0168840 con 0x7f5ba015b860 2024-09-16T15:51:29.702 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:29.701+0000 7f5ba8024640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f5ba0168b40 con 0x7f5ba015bc40 2024-09-16T15:51:29.702 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:29.701+0000 7f5ba5dc0640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5ba00c3230 0x7f5ba0147e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:51:29.702 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:29.701+0000 7f5ba5dc0640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5ba00c3230 0x7f5ba0147e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:57730/0 (socket says 172.21.15.123:57730) 2024-09-16T15:51:29.702 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:29.701+0000 7f5ba5dc0640 1 -- 172.21.15.123:0/3233207203 learned_addr learned my addr 172.21.15.123:0/3233207203 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:51:29.703 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:29.701+0000 7f5ba55bf640 1 --2- 172.21.15.123:0/3233207203 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5ba015b860 0x7f5ba01f8b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:51:29.703 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:29.701+0000 7f5ba65c1640 1 --2- 172.21.15.123:0/3233207203 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f5ba015bc40 0x7f5ba00c7190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:51:29.703 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:29.701+0000 7f5ba5dc0640 1 -- 172.21.15.123:0/3233207203 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f5ba015bc40 msgr2=0x7f5ba00c7190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:51:29.703 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:29.701+0000 7f5ba5dc0640 1 --2- 172.21.15.123:0/3233207203 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f5ba015bc40 0x7f5ba00c7190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:29.703 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:29.701+0000 7f5ba5dc0640 1 -- 172.21.15.123:0/3233207203 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5ba015b860 msgr2=0x7f5ba01f8b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:51:29.703 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:29.701+0000 7f5ba5dc0640 1 --2- 172.21.15.123:0/3233207203 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5ba015b860 0x7f5ba01f8b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:29.703 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:29.701+0000 7f5ba5dc0640 1 -- 172.21.15.123:0/3233207203 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5ba01f95e0 con 0x7f5ba00c3230 2024-09-16T15:51:29.703 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:29.701+0000 7f5ba5dc0640 1 --2- 172.21.15.123:0/3233207203 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5ba00c3230 0x7f5ba0147e30 secure :-1 s=READY pgs=1777 cs=0 l=1 rev1=1 crypto rx=0x7f5b9c00c4c0 tx=0x7f5b9c00c980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:51:29.703 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:29.701+0000 7f5b927fc640 1 -- 172.21.15.123:0/3233207203 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b9c016020 con 0x7f5ba00c3230 2024-09-16T15:51:29.703 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:29.701+0000 7f5b927fc640 1 -- 172.21.15.123:0/3233207203 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f5b9c012940 con 0x7f5ba00c3230 2024-09-16T15:51:29.703 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:29.701+0000 7f5ba8024640 1 -- 172.21.15.123:0/3233207203 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5ba01f98c0 con 0x7f5ba00c3230 2024-09-16T15:51:29.703 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:29.701+0000 7f5b927fc640 1 -- 172.21.15.123:0/3233207203 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b9c012d90 con 0x7f5ba00c3230 2024-09-16T15:51:29.703 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:29.701+0000 7f5ba8024640 1 -- 172.21.15.123:0/3233207203 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5ba0159860 con 0x7f5ba00c3230 2024-09-16T15:51:29.704 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:29.701+0000 7f5ba8024640 1 -- 172.21.15.123:0/3233207203 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5ba006fca0 con 0x7f5ba00c3230 2024-09-16T15:51:29.704 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:29.705+0000 7f5b927fc640 1 -- 172.21.15.123:0/3233207203 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 24) ==== 97845+0+0 (secure 0 0 0) 0x7f5b9c019020 con 0x7f5ba00c3230 2024-09-16T15:51:29.704 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:29.705+0000 7f5b927fc640 1 --2- 172.21.15.123:0/3233207203 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f5b84073760 0x7f5b84075c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:51:29.704 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:29.705+0000 7f5b927fc640 1 -- 172.21.15.123:0/3233207203 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(157..157 src has 1..157) ==== 5737+0+0 (secure 0 0 0) 0x7f5b9c092660 con 0x7f5ba00c3230 2024-09-16T15:51:29.704 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:29.705+0000 7f5ba55bf640 1 --2- 172.21.15.123:0/3233207203 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f5b84073760 0x7f5b84075c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:51:29.705 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:29.705+0000 7f5ba55bf640 1 --2- 172.21.15.123:0/3233207203 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f5b84073760 0x7f5b84075c20 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f5b940af000 tx=0x7f5b9401afa0 comp rx=0 tx=0).ready entity=mgr.4150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:51:29.708 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:29.709+0000 7f5b927fc640 1 -- 172.21.15.123:0/3233207203 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f5b9c020020 con 0x7f5ba00c3230 2024-09-16T15:51:29.866 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:29.865+0000 7f5ba8024640 1 -- 172.21.15.123:0/3233207203 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) -- 0x7f5ba0159350 con 0x7f5ba00c3230 2024-09-16T15:51:29.867 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:29.869+0000 7f5b927fc640 1 -- 172.21.15.123:0/3233207203 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v46)=0 v46) ==== 122+0+0 (secure 0 0 0) 0x7f5b9c05f420 con 0x7f5ba00c3230 2024-09-16T15:51:29.869 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:29.869+0000 7f5ba8024640 1 -- 172.21.15.123:0/3233207203 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f5b84073760 msgr2=0x7f5b84075c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:51:29.870 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:29.869+0000 7f5ba8024640 1 --2- 172.21.15.123:0/3233207203 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f5b84073760 0x7f5b84075c20 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f5b940af000 tx=0x7f5b9401afa0 comp rx=0 tx=0).stop 2024-09-16T15:51:29.870 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:29.869+0000 7f5ba8024640 1 -- 172.21.15.123:0/3233207203 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5ba00c3230 msgr2=0x7f5ba0147e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:51:29.870 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:29.869+0000 7f5ba8024640 1 --2- 172.21.15.123:0/3233207203 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5ba00c3230 0x7f5ba0147e30 secure :-1 s=READY pgs=1777 cs=0 l=1 rev1=1 crypto rx=0x7f5b9c00c4c0 tx=0x7f5b9c00c980 comp rx=0 tx=0).stop 2024-09-16T15:51:29.870 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:29.869+0000 7f5ba8024640 1 -- 172.21.15.123:0/3233207203 shutdown_connections 2024-09-16T15:51:29.870 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:29.869+0000 7f5ba8024640 1 --2- 172.21.15.123:0/3233207203 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f5b84073760 0x7f5b84075c20 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:29.870 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:29.869+0000 7f5ba8024640 1 --2- 172.21.15.123:0/3233207203 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f5ba015bc40 0x7f5ba00c7190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:29.870 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:29.869+0000 7f5ba8024640 1 --2- 172.21.15.123:0/3233207203 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5ba015b860 0x7f5ba01f8b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:29.870 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:29.869+0000 7f5ba8024640 1 --2- 172.21.15.123:0/3233207203 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5ba00c3230 0x7f5ba0147e30 unknown :-1 s=CLOSED pgs=1777 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:29.870 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:29.869+0000 7f5ba8024640 1 -- 172.21.15.123:0/3233207203 >> 172.21.15.123:0/3233207203 conn(0x7f5ba00b1710 msgr2=0x7f5ba00b4770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:51:29.870 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:29.869+0000 7f5ba8024640 1 -- 172.21.15.123:0/3233207203 shutdown_connections 2024-09-16T15:51:29.870 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:29.869+0000 7f5ba8024640 1 -- 172.21.15.123:0/3233207203 wait complete. 2024-09-16T15:51:29.885 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:51:29.885 DEBUG:teuthology.orchestra.run.smithi123:> (cd /home/ubuntu/cephtest/mnt.0 && exec sudo mkdir -p volumes/_nogroup/subvol33410/be5e28c8-b119-4ce3-b88c-1bb6539077a8) 2024-09-16T15:51:29.914 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:51:29.914 DEBUG:teuthology.orchestra.run.smithi123:> (cd /home/ubuntu/cephtest/mnt.0 && exec getfattr --only-values -n ceph.dir.layout.pool .) 2024-09-16T15:51:29.963 INFO:teuthology.orchestra.run.smithi123.stdout:cephfs_data 2024-09-16T15:51:29.963 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:51:29.963 DEBUG:teuthology.orchestra.run.smithi123:> (cd /home/ubuntu/cephtest/mnt.0 && exec sudo setfattr -n ceph.dir.layout.pool -v cephfs_data volumes/_nogroup/subvol33410/be5e28c8-b119-4ce3-b88c-1bb6539077a8) 2024-09-16T15:51:30.027 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2024-09-16T15:51:30.027 DEBUG:teuthology.orchestra.run.smithi123:> sudo dd of=/home/ubuntu/cephtest/mnt.0/volumes/_nogroup/subvol33410/.meta 2024-09-16T15:51:30.087 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:51:30.087 DEBUG:teuthology.orchestra.run.smithi123:> (cd /home/ubuntu/cephtest/mnt.0 && exec sudo mkdir -p volumes/subvol_grp68770/subvol69771/d72a975a-7e3b-4a59-9c4b-020c67d33ba4) 2024-09-16T15:51:30.153 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:51:30.154 DEBUG:teuthology.orchestra.run.smithi123:> (cd /home/ubuntu/cephtest/mnt.0 && exec getfattr --only-values -n ceph.dir.layout.pool .) 2024-09-16T15:51:30.202 INFO:teuthology.orchestra.run.smithi123.stdout:cephfs_data 2024-09-16T15:51:30.202 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:51:30.202 DEBUG:teuthology.orchestra.run.smithi123:> (cd /home/ubuntu/cephtest/mnt.0 && exec sudo setfattr -n ceph.dir.layout.pool -v cephfs_data volumes/subvol_grp68770/subvol69771/d72a975a-7e3b-4a59-9c4b-020c67d33ba4) 2024-09-16T15:51:30.270 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2024-09-16T15:51:30.270 DEBUG:teuthology.orchestra.run.smithi123:> sudo dd of=/home/ubuntu/cephtest/mnt.0/volumes/subvol_grp68770/subvol69771/.meta 2024-09-16T15:51:30.327 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs subvol33410 2024-09-16T15:51:30.504 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:30.505+0000 7f266fc34640 1 -- 172.21.15.123:0/1946745763 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f26680ca650 msgr2=0x7f26680cbc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:51:30.504 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:30.505+0000 7f266fc34640 1 --2- 172.21.15.123:0/1946745763 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f26680ca650 0x7f26680cbc60 secure :-1 s=READY pgs=1779 cs=0 l=1 rev1=1 crypto rx=0x7f2658009900 tx=0x7f265801c820 comp rx=0 tx=0).stop 2024-09-16T15:51:30.504 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:30.505+0000 7f266fc34640 1 -- 172.21.15.123:0/1946745763 shutdown_connections 2024-09-16T15:51:30.504 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:30.505+0000 7f266fc34640 1 --2- 172.21.15.123:0/1946745763 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f26680ca650 0x7f26680cbc60 unknown :-1 s=CLOSED pgs=1779 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:30.504 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:30.505+0000 7f266fc34640 1 --2- 172.21.15.123:0/1946745763 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f266815a170 0x7f2668072f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:30.504 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:30.505+0000 7f266fc34640 1 -- 172.21.15.123:0/1946745763 >> 172.21.15.123:0/1946745763 conn(0x7f26680b1710 msgr2=0x7f26680b1b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:51:30.504 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:30.505+0000 7f266fc34640 1 -- 172.21.15.123:0/1946745763 shutdown_connections 2024-09-16T15:51:30.504 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:30.505+0000 7f266fc34640 1 -- 172.21.15.123:0/1946745763 wait complete. 2024-09-16T15:51:30.505 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:30.505+0000 7f266fc34640 1 Processor -- start 2024-09-16T15:51:30.505 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:30.505+0000 7f266fc34640 1 -- start start 2024-09-16T15:51:30.507 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:30.505+0000 7f266fc34640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f26680ca650 0x7f266815ca10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:51:30.507 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:30.505+0000 7f266fc34640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f266815a170 0x7f26680cf9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:51:30.507 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:30.505+0000 7f266fc34640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f26680cff20 0x7f26680aff10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:51:30.507 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:30.505+0000 7f266fc34640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f26681670d0 con 0x7f26680cff20 2024-09-16T15:51:30.507 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:30.505+0000 7f266fc34640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f2668166f50 con 0x7f26680ca650 2024-09-16T15:51:30.507 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:30.505+0000 7f266fc34640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f2668167250 con 0x7f266815a170 2024-09-16T15:51:30.507 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:30.505+0000 7f266e1d1640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f26680cff20 0x7f26680aff10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:51:30.507 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:30.505+0000 7f266e1d1640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f26680cff20 0x7f26680aff10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:57750/0 (socket says 172.21.15.123:57750) 2024-09-16T15:51:30.507 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:30.505+0000 7f266e1d1640 1 -- 172.21.15.123:0/1934350386 learned_addr learned my addr 172.21.15.123:0/1934350386 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:51:30.507 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:30.505+0000 7f266e1d1640 1 -- 172.21.15.123:0/1934350386 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f266815a170 msgr2=0x7f26680cf9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:51:30.507 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:30.505+0000 7f266e1d1640 1 --2- 172.21.15.123:0/1934350386 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f266815a170 0x7f26680cf9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:30.508 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:30.505+0000 7f266e1d1640 1 -- 172.21.15.123:0/1934350386 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f26680ca650 msgr2=0x7f266815ca10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:51:30.508 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:30.505+0000 7f266e1d1640 1 --2- 172.21.15.123:0/1934350386 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f26680ca650 0x7f266815ca10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:30.508 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:30.505+0000 7f266e1d1640 1 -- 172.21.15.123:0/1934350386 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f266815d240 con 0x7f26680cff20 2024-09-16T15:51:30.508 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:30.505+0000 7f266e1d1640 1 --2- 172.21.15.123:0/1934350386 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f26680cff20 0x7f26680aff10 secure :-1 s=READY pgs=1780 cs=0 l=1 rev1=1 crypto rx=0x7f266000c7a0 tx=0x7f266000cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:51:30.508 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:30.505+0000 7f265e7fc640 1 -- 172.21.15.123:0/1934350386 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2660017070 con 0x7f26680cff20 2024-09-16T15:51:30.508 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:30.505+0000 7f265e7fc640 1 -- 172.21.15.123:0/1934350386 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f2660005070 con 0x7f26680cff20 2024-09-16T15:51:30.508 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:30.505+0000 7f266fc34640 1 -- 172.21.15.123:0/1934350386 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f266815d520 con 0x7f26680cff20 2024-09-16T15:51:30.508 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:30.505+0000 7f265e7fc640 1 -- 172.21.15.123:0/1934350386 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26600054c0 con 0x7f26680cff20 2024-09-16T15:51:30.508 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:30.505+0000 7f266fc34640 1 -- 172.21.15.123:0/1934350386 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f266815cf50 con 0x7f26680cff20 2024-09-16T15:51:30.509 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:30.509+0000 7f266fc34640 1 -- 172.21.15.123:0/1934350386 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f266815da40 con 0x7f26680cff20 2024-09-16T15:51:30.509 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:30.509+0000 7f265e7fc640 1 -- 172.21.15.123:0/1934350386 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 24) ==== 97845+0+0 (secure 0 0 0) 0x7f266001a070 con 0x7f26680cff20 2024-09-16T15:51:30.509 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:30.509+0000 7f265e7fc640 1 --2- 172.21.15.123:0/1934350386 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f264c073730 0x7f264c075bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:51:30.509 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:30.509+0000 7f266d9d0640 1 --2- 172.21.15.123:0/1934350386 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f264c073730 0x7f264c075bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:51:30.509 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:30.509+0000 7f265e7fc640 1 -- 172.21.15.123:0/1934350386 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(157..157 src has 1..157) ==== 5737+0+0 (secure 0 0 0) 0x7f2660094610 con 0x7f26680cff20 2024-09-16T15:51:30.509 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:30.509+0000 7f266d9d0640 1 --2- 172.21.15.123:0/1934350386 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f264c073730 0x7f264c075bf0 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f26640058d0 tx=0x7f2664005860 comp rx=0 tx=0).ready entity=mgr.4150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:51:30.513 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:30.513+0000 7f265e7fc640 1 -- 172.21.15.123:0/1934350386 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f266815da40 con 0x7f26680cff20 2024-09-16T15:51:30.712 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:30.713+0000 7f266fc34640 1 -- 172.21.15.123:0/1934350386 --> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "subvol33410", "target": ["mon-mgr", ""]}) -- 0x7f26680cbc60 con 0x7f264c073730 2024-09-16T15:51:30.722 INFO:tasks.ceph.mgr.y.smithi123.stderr:2024-09-16T15:51:30.721+0000 7fa9b36f4640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-16T15:51:30.722 INFO:tasks.ceph.mgr.y.smithi123.stderr:2024-09-16T15:51:30.721+0000 7fa9b36f4640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-16T15:51:30.722 INFO:tasks.ceph.mgr.y.smithi123.stderr:2024-09-16T15:51:30.721+0000 7fa9b36f4640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-16T15:51:30.722 INFO:tasks.ceph.mgr.y.smithi123.stderr:2024-09-16T15:51:30.721+0000 7fa9b36f4640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-16T15:51:30.722 INFO:tasks.ceph.mgr.y.smithi123.stderr:2024-09-16T15:51:30.721+0000 7fa9b36f4640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-16T15:51:30.823 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:30.825+0000 7f265e7fc640 1 -- 172.21.15.123:0/1934350386 <== mgr.4150 v2:172.21.15.123:6836/2464759676 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+67 (secure 0 0 0) 0x7f26680cbc60 con 0x7f264c073730 2024-09-16T15:51:30.823 INFO:teuthology.orchestra.run.smithi123.stdout:/volumes/_nogroup/subvol33410/be5e28c8-b119-4ce3-b88c-1bb6539077a8 2024-09-16T15:51:30.828 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:30.829+0000 7f266fc34640 1 -- 172.21.15.123:0/1934350386 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f264c073730 msgr2=0x7f264c075bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:51:30.828 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:30.829+0000 7f266fc34640 1 --2- 172.21.15.123:0/1934350386 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f264c073730 0x7f264c075bf0 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f26640058d0 tx=0x7f2664005860 comp rx=0 tx=0).stop 2024-09-16T15:51:30.828 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:30.829+0000 7f266fc34640 1 -- 172.21.15.123:0/1934350386 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f26680cff20 msgr2=0x7f26680aff10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:51:30.828 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:30.829+0000 7f266fc34640 1 --2- 172.21.15.123:0/1934350386 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f26680cff20 0x7f26680aff10 secure :-1 s=READY pgs=1780 cs=0 l=1 rev1=1 crypto rx=0x7f266000c7a0 tx=0x7f266000cc60 comp rx=0 tx=0).stop 2024-09-16T15:51:30.828 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:30.829+0000 7f266fc34640 1 -- 172.21.15.123:0/1934350386 shutdown_connections 2024-09-16T15:51:30.829 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:30.829+0000 7f266fc34640 1 --2- 172.21.15.123:0/1934350386 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f264c073730 0x7f264c075bf0 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:30.829 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:30.829+0000 7f266fc34640 1 --2- 172.21.15.123:0/1934350386 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f26680cff20 0x7f26680aff10 unknown :-1 s=CLOSED pgs=1780 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:30.829 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:30.829+0000 7f266fc34640 1 --2- 172.21.15.123:0/1934350386 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f266815a170 0x7f26680cf9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:30.829 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:30.829+0000 7f266fc34640 1 --2- 172.21.15.123:0/1934350386 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f26680ca650 0x7f266815ca10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:30.829 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:30.829+0000 7f266fc34640 1 -- 172.21.15.123:0/1934350386 >> 172.21.15.123:0/1934350386 conn(0x7f26680b1710 msgr2=0x7f2668164690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:51:30.829 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:30.829+0000 7f266fc34640 1 -- 172.21.15.123:0/1934350386 shutdown_connections 2024-09-16T15:51:30.829 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:30.829+0000 7f266fc34640 1 -- 172.21.15.123:0/1934350386 wait complete. 2024-09-16T15:51:30.847 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs subvol69771 subvol_grp68770 2024-09-16T15:51:30.977 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:30.977+0000 7f9649e7e640 1 -- 172.21.15.123:0/2690964452 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9644074440 msgr2=0x7f964415f670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:51:30.978 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:30.977+0000 7f9649e7e640 1 --2- 172.21.15.123:0/2690964452 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9644074440 0x7f964415f670 secure :-1 s=READY pgs=1784 cs=0 l=1 rev1=1 crypto rx=0x7f9644051540 tx=0x7f962c01c7e0 comp rx=0 tx=0).stop 2024-09-16T15:51:30.978 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:30.977+0000 7f9649e7e640 1 -- 172.21.15.123:0/2690964452 shutdown_connections 2024-09-16T15:51:30.978 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:30.977+0000 7f9649e7e640 1 --2- 172.21.15.123:0/2690964452 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9644074440 0x7f964415f670 unknown :-1 s=CLOSED pgs=1784 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:30.978 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:30.977+0000 7f9649e7e640 1 --2- 172.21.15.123:0/2690964452 >> v2:172.21.15.184:3300/0 conn(0x7f9644159680 0x7f96440704e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:30.978 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:30.977+0000 7f9649e7e640 1 -- 172.21.15.123:0/2690964452 >> 172.21.15.123:0/2690964452 conn(0x7f96440b1630 msgr2=0x7f96440b1a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:51:30.978 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:30.977+0000 7f9649e7e640 1 -- 172.21.15.123:0/2690964452 shutdown_connections 2024-09-16T15:51:30.978 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:30.977+0000 7f9649e7e640 1 -- 172.21.15.123:0/2690964452 wait complete. 2024-09-16T15:51:30.979 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:30.977+0000 7f9649e7e640 1 Processor -- start 2024-09-16T15:51:30.979 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:30.977+0000 7f9649e7e640 1 -- start start 2024-09-16T15:51:30.980 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:30.981+0000 7f9649e7e640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f9644074440 0x7f9644150530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:51:30.981 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:30.981+0000 7f9649e7e640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9644159680 0x7f9644146910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:51:30.981 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:30.981+0000 7f9649e7e640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f964415fbb0 0x7f964414acd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:51:30.981 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:30.981+0000 7f9649e7e640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f96441691d0 con 0x7f9644159680 2024-09-16T15:51:30.981 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:30.981+0000 7f9649e7e640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f9644169050 con 0x7f9644074440 2024-09-16T15:51:30.981 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:30.981+0000 7f9649e7e640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f9644169350 con 0x7f964415fbb0 2024-09-16T15:51:30.981 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:30.981+0000 7f963affd640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9644159680 0x7f9644146910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:51:30.981 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:30.981+0000 7f963affd640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9644159680 0x7f9644146910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:57766/0 (socket says 172.21.15.123:57766) 2024-09-16T15:51:30.981 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:30.981+0000 7f963affd640 1 -- 172.21.15.123:0/723747482 learned_addr learned my addr 172.21.15.123:0/723747482 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:51:30.981 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:30.981+0000 7f96437fe640 1 --2- 172.21.15.123:0/723747482 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f9644074440 0x7f9644150530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:51:30.981 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:30.981+0000 7f9643fff640 1 --2- 172.21.15.123:0/723747482 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f964415fbb0 0x7f964414acd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:51:30.981 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:30.981+0000 7f963affd640 1 -- 172.21.15.123:0/723747482 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f964415fbb0 msgr2=0x7f964414acd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:51:30.982 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:30.981+0000 7f963affd640 1 --2- 172.21.15.123:0/723747482 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f964415fbb0 0x7f964414acd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:30.982 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:30.981+0000 7f963affd640 1 -- 172.21.15.123:0/723747482 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f9644074440 msgr2=0x7f9644150530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:51:30.982 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:30.981+0000 7f963affd640 1 --2- 172.21.15.123:0/723747482 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f9644074440 0x7f9644150530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:30.982 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:30.981+0000 7f963affd640 1 -- 172.21.15.123:0/723747482 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9644147a10 con 0x7f9644159680 2024-09-16T15:51:30.982 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:30.981+0000 7f963affd640 1 --2- 172.21.15.123:0/723747482 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9644159680 0x7f9644146910 secure :-1 s=READY pgs=1785 cs=0 l=1 rev1=1 crypto rx=0x7f962c0193f0 tx=0x7f962c0a6720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:51:30.982 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:30.981+0000 7f9640ff9640 1 -- 172.21.15.123:0/723747482 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f962c004400 con 0x7f9644159680 2024-09-16T15:51:30.982 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:30.981+0000 7f9640ff9640 1 -- 172.21.15.123:0/723747482 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f962c0bd070 con 0x7f9644159680 2024-09-16T15:51:30.982 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:30.981+0000 7f9649e7e640 1 -- 172.21.15.123:0/723747482 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9644150c30 con 0x7f9644159680 2024-09-16T15:51:30.982 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:30.981+0000 7f9640ff9640 1 -- 172.21.15.123:0/723747482 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f962c0a6ba0 con 0x7f9644159680 2024-09-16T15:51:30.982 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:30.981+0000 7f9649e7e640 1 -- 172.21.15.123:0/723747482 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f964414a1f0 con 0x7f9644159680 2024-09-16T15:51:30.983 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:30.981+0000 7f9649e7e640 1 -- 172.21.15.123:0/723747482 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f964415f670 con 0x7f9644159680 2024-09-16T15:51:30.983 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:30.981+0000 7f9640ff9640 1 -- 172.21.15.123:0/723747482 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 24) ==== 97845+0+0 (secure 0 0 0) 0x7f962c005030 con 0x7f9644159680 2024-09-16T15:51:30.983 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:30.981+0000 7f9640ff9640 1 --2- 172.21.15.123:0/723747482 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f9624073760 0x7f9624075c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:51:30.984 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:30.981+0000 7f9640ff9640 1 -- 172.21.15.123:0/723747482 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(157..157 src has 1..157) ==== 5737+0+0 (secure 0 0 0) 0x7f962c12f650 con 0x7f9644159680 2024-09-16T15:51:30.984 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:30.981+0000 7f96437fe640 1 --2- 172.21.15.123:0/723747482 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f9624073760 0x7f9624075c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:51:30.984 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:30.985+0000 7f96437fe640 1 --2- 172.21.15.123:0/723747482 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f9624073760 0x7f9624075c20 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f96340058d0 tx=0x7f9634005860 comp rx=0 tx=0).ready entity=mgr.4150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:51:30.987 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:30.985+0000 7f9640ff9640 1 -- 172.21.15.123:0/723747482 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f9644147a10 con 0x7f9644159680 2024-09-16T15:51:31.179 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:31.177+0000 7f9649e7e640 1 -- 172.21.15.123:0/723747482 --> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "subvol69771", "group_name": "subvol_grp68770", "target": ["mon-mgr", ""]}) -- 0x7f964415cf50 con 0x7f9624073760 2024-09-16T15:51:31.269 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:31.269+0000 7f9640ff9640 1 -- 172.21.15.123:0/723747482 <== mgr.4150 v2:172.21.15.123:6836/2464759676 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+74 (secure 0 0 0) 0x7f964415cf50 con 0x7f9624073760 2024-09-16T15:51:31.270 INFO:teuthology.orchestra.run.smithi123.stdout:/volumes/subvol_grp68770/subvol69771/d72a975a-7e3b-4a59-9c4b-020c67d33ba4 2024-09-16T15:51:31.272 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:31.273+0000 7f9649e7e640 1 -- 172.21.15.123:0/723747482 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f9624073760 msgr2=0x7f9624075c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:51:31.273 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:31.273+0000 7f9649e7e640 1 --2- 172.21.15.123:0/723747482 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f9624073760 0x7f9624075c20 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f96340058d0 tx=0x7f9634005860 comp rx=0 tx=0).stop 2024-09-16T15:51:31.273 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:31.273+0000 7f9649e7e640 1 -- 172.21.15.123:0/723747482 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9644159680 msgr2=0x7f9644146910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:51:31.273 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:31.273+0000 7f9649e7e640 1 --2- 172.21.15.123:0/723747482 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9644159680 0x7f9644146910 secure :-1 s=READY pgs=1785 cs=0 l=1 rev1=1 crypto rx=0x7f962c0193f0 tx=0x7f962c0a6720 comp rx=0 tx=0).stop 2024-09-16T15:51:31.273 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:31.273+0000 7f9649e7e640 1 -- 172.21.15.123:0/723747482 shutdown_connections 2024-09-16T15:51:31.273 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:31.273+0000 7f9649e7e640 1 --2- 172.21.15.123:0/723747482 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f9624073760 0x7f9624075c20 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:31.273 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:31.273+0000 7f9649e7e640 1 --2- 172.21.15.123:0/723747482 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f964415fbb0 0x7f964414acd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:31.273 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:31.273+0000 7f9649e7e640 1 --2- 172.21.15.123:0/723747482 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9644159680 0x7f9644146910 unknown :-1 s=CLOSED pgs=1785 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:31.273 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:31.273+0000 7f9649e7e640 1 --2- 172.21.15.123:0/723747482 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f9644074440 0x7f9644150530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:31.273 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:31.273+0000 7f9649e7e640 1 -- 172.21.15.123:0/723747482 >> 172.21.15.123:0/723747482 conn(0x7f96440b1630 msgr2=0x7f96440b1a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:51:31.273 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:31.273+0000 7f9649e7e640 1 -- 172.21.15.123:0/723747482 shutdown_connections 2024-09-16T15:51:31.273 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:31.273+0000 7f9649e7e640 1 -- 172.21.15.123:0/723747482 wait complete. 2024-09-16T15:51:31.288 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:51:31.288 DEBUG:teuthology.orchestra.run.smithi123:> (cd /home/ubuntu/cephtest/mnt.0 && exec sudo cat ./volumes/_nogroup/subvol33410/.meta) 2024-09-16T15:51:31.307 INFO:teuthology.orchestra.run.smithi123.stdout:[GLOBAL] 2024-09-16T15:51:31.308 INFO:teuthology.orchestra.run.smithi123.stdout:version = 2 2024-09-16T15:51:31.308 INFO:teuthology.orchestra.run.smithi123.stdout:type = subvolume 2024-09-16T15:51:31.308 INFO:teuthology.orchestra.run.smithi123.stdout:path = /volumes/_nogroup/subvol33410/be5e28c8-b119-4ce3-b88c-1bb6539077a8 2024-09-16T15:51:31.308 INFO:teuthology.orchestra.run.smithi123.stdout:state = complete 2024-09-16T15:51:31.308 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-09-16T15:51:31.309 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:51:31.309 DEBUG:teuthology.orchestra.run.smithi123:> (cd /home/ubuntu/cephtest/mnt.0 && exec sudo cat ./volumes/subvol_grp68770/subvol69771/.meta) 2024-09-16T15:51:31.370 INFO:teuthology.orchestra.run.smithi123.stdout:[GLOBAL] 2024-09-16T15:51:31.370 INFO:teuthology.orchestra.run.smithi123.stdout:version = 2 2024-09-16T15:51:31.370 INFO:teuthology.orchestra.run.smithi123.stdout:type = subvolume 2024-09-16T15:51:31.370 INFO:teuthology.orchestra.run.smithi123.stdout:path = /volumes/subvol_grp68770/subvol69771/d72a975a-7e3b-4a59-9c4b-020c67d33ba4 2024-09-16T15:51:31.370 INFO:teuthology.orchestra.run.smithi123.stdout:state = complete 2024-09-16T15:51:31.370 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-09-16T15:51:31.372 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume rm cephfs subvol33410 2024-09-16T15:51:31.546 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:31.545+0000 7f1ad4fdc640 1 -- 172.21.15.123:0/4058590631 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1ad015b860 msgr2=0x7f1ad006fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:51:31.546 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:31.545+0000 7f1ad4fdc640 1 --2- 172.21.15.123:0/4058590631 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1ad015b860 0x7f1ad006fca0 secure :-1 s=READY pgs=1787 cs=0 l=1 rev1=1 crypto rx=0x7f1ac8009bc0 tx=0x7f1ac801cba0 comp rx=0 tx=0).stop 2024-09-16T15:51:31.546 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:31.545+0000 7f1ad4fdc640 1 -- 172.21.15.123:0/4058590631 shutdown_connections 2024-09-16T15:51:31.546 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:31.545+0000 7f1ad4fdc640 1 --2- 172.21.15.123:0/4058590631 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f1ad00c3230 0x7f1ad00cedd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:31.546 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:31.545+0000 7f1ad4fdc640 1 --2- 172.21.15.123:0/4058590631 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1ad015b860 0x7f1ad006fca0 unknown :-1 s=CLOSED pgs=1787 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:31.546 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:31.545+0000 7f1ad4fdc640 1 -- 172.21.15.123:0/4058590631 >> 172.21.15.123:0/4058590631 conn(0x7f1ad00b1710 msgr2=0x7f1ad00b1b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:51:31.546 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:31.545+0000 7f1ad4fdc640 1 -- 172.21.15.123:0/4058590631 shutdown_connections 2024-09-16T15:51:31.547 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:31.545+0000 7f1ad4fdc640 1 -- 172.21.15.123:0/4058590631 wait complete. 2024-09-16T15:51:31.547 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:31.549+0000 7f1ad4fdc640 1 Processor -- start 2024-09-16T15:51:31.547 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:31.549+0000 7f1ad4fdc640 1 -- start start 2024-09-16T15:51:31.549 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:31.549+0000 7f1ad4fdc640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1ad00701e0 0x7f1ad014d8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:51:31.549 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:31.549+0000 7f1ad4fdc640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f1ad00c3230 0x7f1ad0164b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:51:31.549 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:31.549+0000 7f1ad4fdc640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1ad015b860 0x7f1ad01650d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:51:31.549 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:31.549+0000 7f1ad4fdc640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f1ad01691e0 con 0x7f1ad00701e0 2024-09-16T15:51:31.549 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:31.549+0000 7f1ad4fdc640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f1ad0169060 con 0x7f1ad015b860 2024-09-16T15:51:31.549 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:31.549+0000 7f1ad4fdc640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f1ad0169360 con 0x7f1ad00c3230 2024-09-16T15:51:31.549 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:31.549+0000 7f1ace59c640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1ad00701e0 0x7f1ad014d8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:51:31.549 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:31.549+0000 7f1ace59c640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1ad00701e0 0x7f1ad014d8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:57788/0 (socket says 172.21.15.123:57788) 2024-09-16T15:51:31.549 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:31.549+0000 7f1ace59c640 1 -- 172.21.15.123:0/1730960646 learned_addr learned my addr 172.21.15.123:0/1730960646 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:51:31.549 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:31.549+0000 7f1ace59c640 1 -- 172.21.15.123:0/1730960646 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f1ad00c3230 msgr2=0x7f1ad0164b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:51:31.550 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:31.549+0000 7f1ace59c640 1 --2- 172.21.15.123:0/1730960646 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f1ad00c3230 0x7f1ad0164b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:31.550 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:31.549+0000 7f1aced9d640 1 --2- 172.21.15.123:0/1730960646 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1ad015b860 0x7f1ad01650d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:51:31.550 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:31.549+0000 7f1ace59c640 1 -- 172.21.15.123:0/1730960646 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1ad015b860 msgr2=0x7f1ad01650d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:51:31.550 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:31.549+0000 7f1ace59c640 1 --2- 172.21.15.123:0/1730960646 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1ad015b860 0x7f1ad01650d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:31.550 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:31.549+0000 7f1ace59c640 1 -- 172.21.15.123:0/1730960646 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1ad00cc1a0 con 0x7f1ad00701e0 2024-09-16T15:51:31.550 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:31.549+0000 7f1ace59c640 1 --2- 172.21.15.123:0/1730960646 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1ad00701e0 0x7f1ad014d8b0 secure :-1 s=READY pgs=1788 cs=0 l=1 rev1=1 crypto rx=0x7f1ac8009b10 tx=0x7f1ac80092a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:51:31.550 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:31.549+0000 7f1ac6ffd640 1 -- 172.21.15.123:0/1730960646 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ac80bb070 con 0x7f1ad00701e0 2024-09-16T15:51:31.550 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:31.549+0000 7f1ac6ffd640 1 -- 172.21.15.123:0/1730960646 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f1ac8004d40 con 0x7f1ad00701e0 2024-09-16T15:51:31.550 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:31.549+0000 7f1ad4fdc640 1 -- 172.21.15.123:0/1730960646 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1ad01f8b30 con 0x7f1ad00701e0 2024-09-16T15:51:31.550 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:31.549+0000 7f1ac6ffd640 1 -- 172.21.15.123:0/1730960646 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ac80b2040 con 0x7f1ad00701e0 2024-09-16T15:51:31.550 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:31.549+0000 7f1ad4fdc640 1 -- 172.21.15.123:0/1730960646 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1ad014fea0 con 0x7f1ad00701e0 2024-09-16T15:51:31.551 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:31.549+0000 7f1ad4fdc640 1 -- 172.21.15.123:0/1730960646 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1ad00cedd0 con 0x7f1ad00701e0 2024-09-16T15:51:31.551 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:31.549+0000 7f1ac6ffd640 1 -- 172.21.15.123:0/1730960646 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 24) ==== 97845+0+0 (secure 0 0 0) 0x7f1ac80be070 con 0x7f1ad00701e0 2024-09-16T15:51:31.551 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:31.549+0000 7f1ac6ffd640 1 --2- 172.21.15.123:0/1730960646 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f1ab0073730 0x7f1ab0075bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:51:31.551 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:31.549+0000 7f1acdd9b640 1 --2- 172.21.15.123:0/1730960646 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f1ab0073730 0x7f1ab0075bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:51:31.552 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:31.549+0000 7f1ac6ffd640 1 -- 172.21.15.123:0/1730960646 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(157..157 src has 1..157) ==== 5737+0+0 (secure 0 0 0) 0x7f1ac812c280 con 0x7f1ad00701e0 2024-09-16T15:51:31.552 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:31.553+0000 7f1acdd9b640 1 --2- 172.21.15.123:0/1730960646 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f1ab0073730 0x7f1ab0075bf0 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f1ad014e600 tx=0x7f1ab8005c00 comp rx=0 tx=0).ready entity=mgr.4150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:51:31.554 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:31.553+0000 7f1ac6ffd640 1 -- 172.21.15.123:0/1730960646 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f1ac80a5e20 con 0x7f1ad00701e0 2024-09-16T15:51:31.737 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:31.737+0000 7f1ad4fdc640 1 -- 172.21.15.123:0/1730960646 --> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] -- mgr_command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "subvol33410", "target": ["mon-mgr", ""]}) -- 0x7f1ad00b2f40 con 0x7f1ab0073730 2024-09-16T15:51:31.769 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:31.769+0000 7f1ac6ffd640 1 -- 172.21.15.123:0/1730960646 <== mgr.4150 v2:172.21.15.123:6836/2464759676 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f1ad00b2f40 con 0x7f1ab0073730 2024-09-16T15:51:31.771 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:31.773+0000 7f1ad4fdc640 1 -- 172.21.15.123:0/1730960646 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f1ab0073730 msgr2=0x7f1ab0075bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:51:31.772 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:31.773+0000 7f1ad4fdc640 1 --2- 172.21.15.123:0/1730960646 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f1ab0073730 0x7f1ab0075bf0 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f1ad014e600 tx=0x7f1ab8005c00 comp rx=0 tx=0).stop 2024-09-16T15:51:31.772 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:31.773+0000 7f1ad4fdc640 1 -- 172.21.15.123:0/1730960646 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1ad00701e0 msgr2=0x7f1ad014d8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:51:31.772 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:31.773+0000 7f1ad4fdc640 1 --2- 172.21.15.123:0/1730960646 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1ad00701e0 0x7f1ad014d8b0 secure :-1 s=READY pgs=1788 cs=0 l=1 rev1=1 crypto rx=0x7f1ac8009b10 tx=0x7f1ac80092a0 comp rx=0 tx=0).stop 2024-09-16T15:51:31.772 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:31.773+0000 7f1ad4fdc640 1 -- 172.21.15.123:0/1730960646 shutdown_connections 2024-09-16T15:51:31.772 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:31.773+0000 7f1ad4fdc640 1 --2- 172.21.15.123:0/1730960646 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f1ab0073730 0x7f1ab0075bf0 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:31.772 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:31.773+0000 7f1ad4fdc640 1 --2- 172.21.15.123:0/1730960646 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1ad015b860 0x7f1ad01650d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:31.772 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:31.773+0000 7f1ad4fdc640 1 --2- 172.21.15.123:0/1730960646 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f1ad00c3230 0x7f1ad0164b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:31.772 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:31.773+0000 7f1ad4fdc640 1 --2- 172.21.15.123:0/1730960646 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1ad00701e0 0x7f1ad014d8b0 unknown :-1 s=CLOSED pgs=1788 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:31.772 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:31.773+0000 7f1ad4fdc640 1 -- 172.21.15.123:0/1730960646 >> 172.21.15.123:0/1730960646 conn(0x7f1ad00b1710 msgr2=0x7f1ad00d0470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:51:31.772 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:31.773+0000 7f1ad4fdc640 1 -- 172.21.15.123:0/1730960646 shutdown_connections 2024-09-16T15:51:31.772 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:31.773+0000 7f1ad4fdc640 1 -- 172.21.15.123:0/1730960646 wait complete. 2024-09-16T15:51:31.776 INFO:tasks.ceph.mgr.y.smithi123.stderr:2024-09-16T15:51:31.777+0000 7fa9b5ef9640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-16T15:51:31.776 INFO:tasks.ceph.mgr.y.smithi123.stderr:2024-09-16T15:51:31.777+0000 7fa9b5ef9640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-16T15:51:31.776 INFO:tasks.ceph.mgr.y.smithi123.stderr:2024-09-16T15:51:31.777+0000 7fa9b5ef9640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-16T15:51:31.776 INFO:tasks.ceph.mgr.y.smithi123.stderr:2024-09-16T15:51:31.777+0000 7fa9b5ef9640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-16T15:51:31.776 INFO:tasks.ceph.mgr.y.smithi123.stderr:2024-09-16T15:51:31.777+0000 7fa9b5ef9640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-16T15:51:31.789 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume rm cephfs subvol69771 subvol_grp68770 2024-09-16T15:51:31.795 INFO:tasks.ceph.mgr.y.smithi123.stderr:2024-09-16T15:51:31.797+0000 7fa9b66fa640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-16T15:51:31.795 INFO:tasks.ceph.mgr.y.smithi123.stderr:2024-09-16T15:51:31.797+0000 7fa9b66fa640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-16T15:51:31.795 INFO:tasks.ceph.mgr.y.smithi123.stderr:2024-09-16T15:51:31.797+0000 7fa9b66fa640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-16T15:51:31.795 INFO:tasks.ceph.mgr.y.smithi123.stderr:2024-09-16T15:51:31.797+0000 7fa9b66fa640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-16T15:51:31.795 INFO:tasks.ceph.mgr.y.smithi123.stderr:2024-09-16T15:51:31.797+0000 7fa9b66fa640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-16T15:51:31.913 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:31.913+0000 7f8c8c35c640 1 -- 172.21.15.123:0/1495849608 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8c700052f0 msgr2=0x7f8c7000b9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:51:31.913 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:31.913+0000 7f8c8c35c640 1 --2- 172.21.15.123:0/1495849608 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8c700052f0 0x7f8c7000b9d0 secure :-1 s=READY pgs=1794 cs=0 l=1 rev1=1 crypto rx=0x7f8c7c001dc0 tx=0x7f8c7c01cb30 comp rx=0 tx=0).stop 2024-09-16T15:51:31.913 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:31.913+0000 7f8c8c35c640 1 -- 172.21.15.123:0/1495849608 shutdown_connections 2024-09-16T15:51:31.913 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:31.913+0000 7f8c8c35c640 1 --2- 172.21.15.123:0/1495849608 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8c700052f0 0x7f8c7000b9d0 unknown :-1 s=CLOSED pgs=1794 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:31.913 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:31.913+0000 7f8c8c35c640 1 -- 172.21.15.123:0/1495849608 >> 172.21.15.123:0/1495849608 conn(0x7f8c840b1630 msgr2=0x7f8c840b1a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:51:31.913 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:31.913+0000 7f8c8c35c640 1 -- 172.21.15.123:0/1495849608 shutdown_connections 2024-09-16T15:51:31.913 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:31.913+0000 7f8c8c35c640 1 -- 172.21.15.123:0/1495849608 wait complete. 2024-09-16T15:51:31.914 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:31.913+0000 7f8c8c35c640 1 Processor -- start 2024-09-16T15:51:31.914 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:31.913+0000 7f8c8c35c640 1 -- start start 2024-09-16T15:51:31.917 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:31.917+0000 7f8c8c35c640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f8c700052f0 0x7f8c8414ba80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:51:31.917 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:31.917+0000 7f8c8c35c640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f8c84138e00 0x7f8c84135ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:51:31.917 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:31.917+0000 7f8c8c35c640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8c841e7cb0 0x7f8c841e8160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:51:31.917 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:31.917+0000 7f8c8c35c640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f8c8406eaf0 con 0x7f8c841e7cb0 2024-09-16T15:51:31.917 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:31.917+0000 7f8c8c35c640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f8c8406e970 con 0x7f8c84138e00 2024-09-16T15:51:31.917 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:31.917+0000 7f8c8c35c640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f8c8406ec70 con 0x7f8c700052f0 2024-09-16T15:51:31.917 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:31.917+0000 7f8c8a8f9640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8c841e7cb0 0x7f8c841e8160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:51:31.917 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:31.917+0000 7f8c8a8f9640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8c841e7cb0 0x7f8c841e8160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:57850/0 (socket says 172.21.15.123:57850) 2024-09-16T15:51:31.917 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:31.917+0000 7f8c8a8f9640 1 -- 172.21.15.123:0/2236889490 learned_addr learned my addr 172.21.15.123:0/2236889490 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:51:31.917 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:31.917+0000 7f8c898f7640 1 --2- 172.21.15.123:0/2236889490 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f8c84138e00 0x7f8c84135ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:51:31.917 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:31.917+0000 7f8c8a0f8640 1 --2- 172.21.15.123:0/2236889490 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f8c700052f0 0x7f8c8414ba80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:51:31.917 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:31.917+0000 7f8c8a8f9640 1 -- 172.21.15.123:0/2236889490 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f8c700052f0 msgr2=0x7f8c8414ba80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:51:31.917 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:31.917+0000 7f8c8a8f9640 1 --2- 172.21.15.123:0/2236889490 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f8c700052f0 0x7f8c8414ba80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:31.918 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:31.917+0000 7f8c8a8f9640 1 -- 172.21.15.123:0/2236889490 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f8c84138e00 msgr2=0x7f8c84135ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:51:31.918 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:31.917+0000 7f8c8a8f9640 1 --2- 172.21.15.123:0/2236889490 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f8c84138e00 0x7f8c84135ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:31.918 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:31.917+0000 7f8c8a8f9640 1 -- 172.21.15.123:0/2236889490 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8c840b1a10 con 0x7f8c841e7cb0 2024-09-16T15:51:31.918 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:31.917+0000 7f8c8a8f9640 1 --2- 172.21.15.123:0/2236889490 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8c841e7cb0 0x7f8c841e8160 secure :-1 s=READY pgs=1795 cs=0 l=1 rev1=1 crypto rx=0x7f8c7c01d3d0 tx=0x7f8c7c004ff0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:51:31.918 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:31.917+0000 7f8c7affd640 1 -- 172.21.15.123:0/2236889490 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c7c0019f0 con 0x7f8c841e7cb0 2024-09-16T15:51:31.918 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:31.917+0000 7f8c7affd640 1 -- 172.21.15.123:0/2236889490 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f8c7c0b5070 con 0x7f8c841e7cb0 2024-09-16T15:51:31.918 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:31.917+0000 7f8c8c35c640 1 -- 172.21.15.123:0/2236889490 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8c8414c1b0 con 0x7f8c841e7cb0 2024-09-16T15:51:31.918 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:31.917+0000 7f8c7affd640 1 -- 172.21.15.123:0/2236889490 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c7c006630 con 0x7f8c841e7cb0 2024-09-16T15:51:31.918 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:31.917+0000 7f8c8c35c640 1 -- 172.21.15.123:0/2236889490 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8c8414c770 con 0x7f8c841e7cb0 2024-09-16T15:51:31.919 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:31.917+0000 7f8c8c35c640 1 -- 172.21.15.123:0/2236889490 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8c8406ffd0 con 0x7f8c841e7cb0 2024-09-16T15:51:31.919 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:31.917+0000 7f8c7affd640 1 -- 172.21.15.123:0/2236889490 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 24) ==== 97845+0+0 (secure 0 0 0) 0x7f8c7c0b1430 con 0x7f8c841e7cb0 2024-09-16T15:51:31.919 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:31.917+0000 7f8c7affd640 1 --2- 172.21.15.123:0/2236889490 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f8c68073730 0x7f8c68075bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:51:31.919 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:31.917+0000 7f8c7affd640 1 -- 172.21.15.123:0/2236889490 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(157..157 src has 1..157) ==== 5737+0+0 (secure 0 0 0) 0x7f8c7c12d4f0 con 0x7f8c841e7cb0 2024-09-16T15:51:31.919 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:31.917+0000 7f8c8a0f8640 1 --2- 172.21.15.123:0/2236889490 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f8c68073730 0x7f8c68075bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:51:31.919 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:31.921+0000 7f8c8a0f8640 1 --2- 172.21.15.123:0/2236889490 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f8c68073730 0x7f8c68075bf0 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f8c80009ad0 tx=0x7f8c800092a0 comp rx=0 tx=0).ready entity=mgr.4150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:51:31.923 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:31.925+0000 7f8c7affd640 1 -- 172.21.15.123:0/2236889490 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f8c7c016890 con 0x7f8c841e7cb0 2024-09-16T15:51:32.108 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:32.109+0000 7f8c8c35c640 1 -- 172.21.15.123:0/2236889490 --> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] -- mgr_command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "subvol69771", "group_name": "subvol_grp68770", "target": ["mon-mgr", ""]}) -- 0x7f8c841e86a0 con 0x7f8c68073730 2024-09-16T15:51:32.357 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:32.357+0000 7f8c7affd640 1 -- 172.21.15.123:0/2236889490 <== mgr.4150 v2:172.21.15.123:6836/2464759676 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f8c80011b20 con 0x7f8c68073730 2024-09-16T15:51:32.360 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:32.361+0000 7f8c8c35c640 1 -- 172.21.15.123:0/2236889490 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f8c68073730 msgr2=0x7f8c68075bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:51:32.360 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:32.361+0000 7f8c8c35c640 1 --2- 172.21.15.123:0/2236889490 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f8c68073730 0x7f8c68075bf0 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f8c80009ad0 tx=0x7f8c800092a0 comp rx=0 tx=0).stop 2024-09-16T15:51:32.360 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:32.361+0000 7f8c8c35c640 1 -- 172.21.15.123:0/2236889490 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8c841e7cb0 msgr2=0x7f8c841e8160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:51:32.360 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:32.361+0000 7f8c8c35c640 1 --2- 172.21.15.123:0/2236889490 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8c841e7cb0 0x7f8c841e8160 secure :-1 s=READY pgs=1795 cs=0 l=1 rev1=1 crypto rx=0x7f8c7c01d3d0 tx=0x7f8c7c004ff0 comp rx=0 tx=0).stop 2024-09-16T15:51:32.360 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:32.361+0000 7f8c8c35c640 1 -- 172.21.15.123:0/2236889490 shutdown_connections 2024-09-16T15:51:32.360 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:32.361+0000 7f8c8c35c640 1 --2- 172.21.15.123:0/2236889490 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f8c68073730 0x7f8c68075bf0 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:32.360 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:32.361+0000 7f8c8c35c640 1 --2- 172.21.15.123:0/2236889490 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8c841e7cb0 0x7f8c841e8160 unknown :-1 s=CLOSED pgs=1795 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:32.360 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:32.361+0000 7f8c8c35c640 1 --2- 172.21.15.123:0/2236889490 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f8c84138e00 0x7f8c84135ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:32.361 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:32.361+0000 7f8c8c35c640 1 --2- 172.21.15.123:0/2236889490 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f8c700052f0 0x7f8c8414ba80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:32.361 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:32.361+0000 7f8c8c35c640 1 -- 172.21.15.123:0/2236889490 >> 172.21.15.123:0/2236889490 conn(0x7f8c840b1630 msgr2=0x7f8c841585d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:51:32.361 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:32.361+0000 7f8c8c35c640 1 -- 172.21.15.123:0/2236889490 shutdown_connections 2024-09-16T15:51:32.361 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:32.361+0000 7f8c8c35c640 1 -- 172.21.15.123:0/2236889490 wait complete. 2024-09-16T15:51:32.381 DEBUG:teuthology.orchestra.run.smithi123:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'stat -c %h /home/ubuntu/cephtest/mnt.0/./volumes/_deleting') 2024-09-16T15:51:32.474 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:32 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-09-16T15:51:32.487 INFO:teuthology.orchestra.run.smithi123.stdout:2 2024-09-16T15:51:32.487 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:32 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-16T15:51:32.494 INFO:teuthology.nuke:Clearing teuthology firewall rules... 2024-09-16T15:51:32.494 DEBUG:teuthology.orchestra.run.smithi123:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-09-16T15:51:32.568 DEBUG:teuthology.orchestra.run.smithi184:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-09-16T15:51:32.661 INFO:teuthology.nuke:Cleared teuthology firewall rules. 2024-09-16T15:51:32.661 INFO:tasks.cephfs.mount:Terminating background process 2024-09-16T15:51:32.661 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2024-09-16T15:51:32.661 DEBUG:teuthology.orchestra.run.smithi123:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-16T15:51:32.667 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2024-09-16T15:51:32.667 DEBUG:teuthology.orchestra.run.smithi123:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-16T15:51:32.716 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-09-16T15:51:32.894 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:32.893+0000 7fb207e62640 1 -- 172.21.15.123:0/4008579382 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb200074600 msgr2=0x7fb20006e750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:51:32.894 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:32.893+0000 7fb207e62640 1 --2- 172.21.15.123:0/4008579382 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb200074600 0x7fb20006e750 secure :-1 s=READY pgs=1797 cs=0 l=1 rev1=1 crypto rx=0x7fb1f8009840 tx=0x7fb1f801c760 comp rx=0 tx=0).stop 2024-09-16T15:51:32.894 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:32.893+0000 7fb207e62640 1 -- 172.21.15.123:0/4008579382 shutdown_connections 2024-09-16T15:51:32.894 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:32.893+0000 7fb207e62640 1 --2- 172.21.15.123:0/4008579382 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb200074600 0x7fb20006e750 unknown :-1 s=CLOSED pgs=1797 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:32.894 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:32.893+0000 7fb207e62640 1 --2- 172.21.15.123:0/4008579382 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fb2000726c0 0x7fb2000d0630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:32.894 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:32.893+0000 7fb207e62640 1 -- 172.21.15.123:0/4008579382 >> 172.21.15.123:0/4008579382 conn(0x7fb2000cfb10 msgr2=0x7fb2000cff20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:51:32.894 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:32.893+0000 7fb207e62640 1 -- 172.21.15.123:0/4008579382 shutdown_connections 2024-09-16T15:51:32.894 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:32.893+0000 7fb207e62640 1 -- 172.21.15.123:0/4008579382 wait complete. 2024-09-16T15:51:32.895 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:32.897+0000 7fb207e62640 1 Processor -- start 2024-09-16T15:51:32.895 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:32.897+0000 7fb207e62640 1 -- start start 2024-09-16T15:51:32.897 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:32.897+0000 7fb207e62640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb2000726c0 0x7fb200155c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:51:32.897 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:32.897+0000 7fb207e62640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fb200074600 0x7fb2001f86c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:51:32.897 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:32.897+0000 7fb207e62640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb20015bca0 0x7fb2001f8c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:51:32.897 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:32.897+0000 7fb207e62640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fb200168a50 con 0x7fb2000726c0 2024-09-16T15:51:32.897 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:32.897+0000 7fb207e62640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fb20006f9b0 con 0x7fb20015bca0 2024-09-16T15:51:32.897 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:32.897+0000 7fb207e62640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fb200168bd0 con 0x7fb200074600 2024-09-16T15:51:32.897 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:32.897+0000 7fb205bfe640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb2000726c0 0x7fb200155c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:51:32.897 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:32.897+0000 7fb205bfe640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb2000726c0 0x7fb200155c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:50110/0 (socket says 172.21.15.123:50110) 2024-09-16T15:51:32.897 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:32.897+0000 7fb205bfe640 1 -- 172.21.15.123:0/3034852477 learned_addr learned my addr 172.21.15.123:0/3034852477 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:51:32.897 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:32.897+0000 7fb205bfe640 1 -- 172.21.15.123:0/3034852477 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fb200074600 msgr2=0x7fb2001f86c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:51:32.897 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:32.897+0000 7fb205bfe640 1 --2- 172.21.15.123:0/3034852477 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fb200074600 0x7fb2001f86c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:32.897 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:32.897+0000 7fb205bfe640 1 -- 172.21.15.123:0/3034852477 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb20015bca0 msgr2=0x7fb2001f8c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:51:32.898 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:32.897+0000 7fb205bfe640 1 --2- 172.21.15.123:0/3034852477 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb20015bca0 0x7fb2001f8c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:32.898 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:32.897+0000 7fb205bfe640 1 -- 172.21.15.123:0/3034852477 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb2001f9280 con 0x7fb2000726c0 2024-09-16T15:51:32.898 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:32.897+0000 7fb205bfe640 1 --2- 172.21.15.123:0/3034852477 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb2000726c0 0x7fb200155c90 secure :-1 s=READY pgs=1798 cs=0 l=1 rev1=1 crypto rx=0x7fb1fc00cb40 tx=0x7fb1fc00cc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:51:32.898 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:32.897+0000 7fb1f67fc640 1 -- 172.21.15.123:0/3034852477 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1fc016020 con 0x7fb2000726c0 2024-09-16T15:51:32.898 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:32.897+0000 7fb1f67fc640 1 -- 172.21.15.123:0/3034852477 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fb1fc00ab60 con 0x7fb2000726c0 2024-09-16T15:51:32.898 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:32.897+0000 7fb207e62640 1 -- 172.21.15.123:0/3034852477 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb2001f9560 con 0x7fb2000726c0 2024-09-16T15:51:32.898 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:32.897+0000 7fb1f67fc640 1 -- 172.21.15.123:0/3034852477 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1fc012420 con 0x7fb2000726c0 2024-09-16T15:51:32.898 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:32.897+0000 7fb207e62640 1 -- 172.21.15.123:0/3034852477 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb200155490 con 0x7fb2000726c0 2024-09-16T15:51:32.899 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:32.897+0000 7fb207e62640 1 -- 172.21.15.123:0/3034852477 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb2000d0630 con 0x7fb2000726c0 2024-09-16T15:51:32.899 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:32.897+0000 7fb1f67fc640 1 -- 172.21.15.123:0/3034852477 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 24) ==== 97845+0+0 (secure 0 0 0) 0x7fb1fc019020 con 0x7fb2000726c0 2024-09-16T15:51:32.899 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:32.897+0000 7fb1f67fc640 1 --2- 172.21.15.123:0/3034852477 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fb1e4073730 0x7fb1e4075bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:51:32.899 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:32.901+0000 7fb2053fd640 1 --2- 172.21.15.123:0/3034852477 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fb1e4073730 0x7fb1e4075bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:51:32.899 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:32.901+0000 7fb1f67fc640 1 -- 172.21.15.123:0/3034852477 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(157..157 src has 1..157) ==== 5737+0+0 (secure 0 0 0) 0x7fb1fc092920 con 0x7fb2000726c0 2024-09-16T15:51:32.899 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:32.901+0000 7fb2053fd640 1 --2- 172.21.15.123:0/3034852477 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fb1e4073730 0x7fb1e4075bf0 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7fb1f00058d0 tx=0x7fb1f0005860 comp rx=0 tx=0).ready entity=mgr.4150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:51:32.903 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:32.905+0000 7fb1f67fc640 1 -- 172.21.15.123:0/3034852477 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7fb1fc020020 con 0x7fb2000726c0 2024-09-16T15:51:33.091 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:33.089+0000 7fb207e62640 1 -- 172.21.15.123:0/3034852477 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7fb20015f630 con 0x7fb2000726c0 2024-09-16T15:51:33.091 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:33.093+0000 7fb1f67fc640 1 -- 172.21.15.123:0/3034852477 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v157) ==== 80+0+0 (secure 0 0 0) 0x7fb1fc05f620 con 0x7fb2000726c0 2024-09-16T15:51:33.091 INFO:teuthology.orchestra.run.smithi123.stderr:listed 0 entries 2024-09-16T15:51:33.094 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:33.093+0000 7fb207e62640 1 -- 172.21.15.123:0/3034852477 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fb1e4073730 msgr2=0x7fb1e4075bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:51:33.094 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:33.093+0000 7fb207e62640 1 --2- 172.21.15.123:0/3034852477 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fb1e4073730 0x7fb1e4075bf0 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7fb1f00058d0 tx=0x7fb1f0005860 comp rx=0 tx=0).stop 2024-09-16T15:51:33.094 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:33.093+0000 7fb207e62640 1 -- 172.21.15.123:0/3034852477 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb2000726c0 msgr2=0x7fb200155c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:51:33.094 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:33.093+0000 7fb207e62640 1 --2- 172.21.15.123:0/3034852477 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb2000726c0 0x7fb200155c90 secure :-1 s=READY pgs=1798 cs=0 l=1 rev1=1 crypto rx=0x7fb1fc00cb40 tx=0x7fb1fc00cc30 comp rx=0 tx=0).stop 2024-09-16T15:51:33.094 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:33.093+0000 7fb207e62640 1 -- 172.21.15.123:0/3034852477 shutdown_connections 2024-09-16T15:51:33.094 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:33.093+0000 7fb207e62640 1 --2- 172.21.15.123:0/3034852477 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fb1e4073730 0x7fb1e4075bf0 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:33.094 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:33.093+0000 7fb207e62640 1 --2- 172.21.15.123:0/3034852477 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb20015bca0 0x7fb2001f8c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:33.094 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:33.093+0000 7fb207e62640 1 --2- 172.21.15.123:0/3034852477 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fb200074600 0x7fb2001f86c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:33.094 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:33.093+0000 7fb207e62640 1 --2- 172.21.15.123:0/3034852477 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb2000726c0 0x7fb200155c90 unknown :-1 s=CLOSED pgs=1798 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:33.094 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:33.093+0000 7fb207e62640 1 -- 172.21.15.123:0/3034852477 >> 172.21.15.123:0/3034852477 conn(0x7fb2000cfb10 msgr2=0x7fb2000c5eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:51:33.094 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:33.093+0000 7fb207e62640 1 -- 172.21.15.123:0/3034852477 shutdown_connections 2024-09-16T15:51:33.095 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:33.093+0000 7fb207e62640 1 -- 172.21.15.123:0/3034852477 wait complete. 2024-09-16T15:51:33.109 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi123.front.sepia.ceph.com... 2024-09-16T15:51:33.110 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:51:33.110 DEBUG:teuthology.orchestra.run.smithi123:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2024-09-16T15:51:33.120 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi123.stderr:ceph-fuse[53266]: fuse finished with error 0 and tester_r 0 2024-09-16T15:51:33.121 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi123.front.sepia.ceph.com 2024-09-16T15:51:33.121 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:51:33.121 DEBUG:teuthology.orchestra.run.smithi123:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-09-16T15:51:33.196 INFO:tasks.cephfs.mount:Terminating background process 2024-09-16T15:51:33.196 DEBUG:teuthology.orchestra.run.smithi184:> set -ex 2024-09-16T15:51:33.196 DEBUG:teuthology.orchestra.run.smithi184:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-16T15:51:33.206 DEBUG:teuthology.orchestra.run.smithi184:> set -ex 2024-09-16T15:51:33.207 DEBUG:teuthology.orchestra.run.smithi184:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-16T15:51:33.253 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-09-16T15:51:33.370 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:33.369+0000 7f4fb8713640 1 -- 172.21.15.123:0/2749209481 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4fb015dd60 msgr2=0x7f4fb01621d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:51:33.370 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:33.369+0000 7f4fb8713640 1 --2- 172.21.15.123:0/2749209481 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4fb015dd60 0x7f4fb01621d0 secure :-1 s=READY pgs=1800 cs=0 l=1 rev1=1 crypto rx=0x7f4fb0161820 tx=0x7f4fac01c830 comp rx=0 tx=0).stop 2024-09-16T15:51:33.370 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:33.369+0000 7f4fb8713640 1 -- 172.21.15.123:0/2749209481 shutdown_connections 2024-09-16T15:51:33.371 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:33.369+0000 7f4fb8713640 1 --2- 172.21.15.123:0/2749209481 >> v2:172.21.15.184:3300/0 conn(0x7f4fb00b3b20 0x7f4fb00d3af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:33.371 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:33.369+0000 7f4fb8713640 1 --2- 172.21.15.123:0/2749209481 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4fb015dd60 0x7f4fb01621d0 unknown :-1 s=CLOSED pgs=1800 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:33.371 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:33.369+0000 7f4fb8713640 1 -- 172.21.15.123:0/2749209481 >> 172.21.15.123:0/2749209481 conn(0x7f4fb0069650 msgr2=0x7f4fb009d170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:51:33.371 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:33.369+0000 7f4fb8713640 1 -- 172.21.15.123:0/2749209481 shutdown_connections 2024-09-16T15:51:33.371 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:33.369+0000 7f4fb8713640 1 -- 172.21.15.123:0/2749209481 wait complete. 2024-09-16T15:51:33.371 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:33.373+0000 7f4fb8713640 1 Processor -- start 2024-09-16T15:51:33.371 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:33.373+0000 7f4fb8713640 1 -- start start 2024-09-16T15:51:33.373 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:33.373+0000 7f4fb8713640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4fb00b3b20 0x7f4fb0156cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:51:33.373 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:33.373+0000 7f4fb8713640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4fb00d4030 0x7f4fb00b8df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:51:33.373 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:33.373+0000 7f4fb8713640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f4fb015dd60 0x7f4fb00caf80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:51:33.373 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:33.373+0000 7f4fb8713640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f4fb016b270 con 0x7f4fb00b3b20 2024-09-16T15:51:33.373 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:33.373+0000 7f4fb8713640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f4fb016b0f0 con 0x7f4fb00d4030 2024-09-16T15:51:33.374 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:33.373+0000 7f4fb8713640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f4fb016b3f0 con 0x7f4fb015dd60 2024-09-16T15:51:33.374 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:33.373+0000 7f4fb64af640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4fb00b3b20 0x7f4fb0156cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:51:33.374 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:33.373+0000 7f4fb64af640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4fb00b3b20 0x7f4fb0156cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:50130/0 (socket says 172.21.15.123:50130) 2024-09-16T15:51:33.374 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:33.373+0000 7f4fb64af640 1 -- 172.21.15.123:0/2922696335 learned_addr learned my addr 172.21.15.123:0/2922696335 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:51:33.374 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:33.373+0000 7f4fb64af640 1 -- 172.21.15.123:0/2922696335 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f4fb015dd60 msgr2=0x7f4fb00caf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:51:33.374 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:33.373+0000 7f4fb64af640 1 --2- 172.21.15.123:0/2922696335 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f4fb015dd60 0x7f4fb00caf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:33.374 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:33.373+0000 7f4fb64af640 1 -- 172.21.15.123:0/2922696335 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4fb00d4030 msgr2=0x7f4fb00b8df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:51:33.374 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:33.373+0000 7f4fb64af640 1 --2- 172.21.15.123:0/2922696335 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4fb00d4030 0x7f4fb00b8df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:33.374 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:33.373+0000 7f4fb64af640 1 -- 172.21.15.123:0/2922696335 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4fb015e860 con 0x7f4fb00b3b20 2024-09-16T15:51:33.374 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:33.373+0000 7f4fb64af640 1 --2- 172.21.15.123:0/2922696335 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4fb00b3b20 0x7f4fb0156cf0 secure :-1 s=READY pgs=1801 cs=0 l=1 rev1=1 crypto rx=0x7f4fac0b0040 tx=0x7f4fac005790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:51:33.374 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:33.373+0000 7f4fa6ffd640 1 -- 172.21.15.123:0/2922696335 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4fac0b2040 con 0x7f4fb00b3b20 2024-09-16T15:51:33.374 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:33.373+0000 7f4fa6ffd640 1 -- 172.21.15.123:0/2922696335 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f4fac0a5c80 con 0x7f4fb00b3b20 2024-09-16T15:51:33.374 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:33.373+0000 7f4fb8713640 1 -- 172.21.15.123:0/2922696335 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4fb015eb40 con 0x7f4fb00b3b20 2024-09-16T15:51:33.375 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:33.373+0000 7f4fa6ffd640 1 -- 172.21.15.123:0/2922696335 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4fac003860 con 0x7f4fb00b3b20 2024-09-16T15:51:33.375 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:33.373+0000 7f4fb8713640 1 -- 172.21.15.123:0/2922696335 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4fb015f100 con 0x7f4fb00b3b20 2024-09-16T15:51:33.375 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:33.373+0000 7f4fb8713640 1 -- 172.21.15.123:0/2922696335 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4fb00d3af0 con 0x7f4fb00b3b20 2024-09-16T15:51:33.376 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:33.373+0000 7f4fa6ffd640 1 -- 172.21.15.123:0/2922696335 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 25) ==== 97998+0+0 (secure 0 0 0) 0x7f4fac0bd060 con 0x7f4fb00b3b20 2024-09-16T15:51:33.376 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:33.377+0000 7f4fa6ffd640 1 --2- 172.21.15.123:0/2922696335 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f4f940738e0 0x7f4f94075da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:51:33.376 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:33.377+0000 7f4fb5cae640 1 --2- 172.21.15.123:0/2922696335 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f4f940738e0 0x7f4f94075da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:51:33.376 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:33.377+0000 7f4fa6ffd640 1 -- 172.21.15.123:0/2922696335 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(157..157 src has 1..157) ==== 5737+0+0 (secure 0 0 0) 0x7f4fac12c220 con 0x7f4fb00b3b20 2024-09-16T15:51:33.376 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:33.377+0000 7f4fb5cae640 1 --2- 172.21.15.123:0/2922696335 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f4f940738e0 0x7f4f94075da0 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f4fb01fb5a0 tx=0x7f4fa0006ca0 comp rx=0 tx=0).ready entity=mgr.4150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:51:33.379 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:33.381+0000 7f4fa6ffd640 1 -- 172.21.15.123:0/2922696335 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f4fac0164b0 con 0x7f4fb00b3b20 2024-09-16T15:51:33.549 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:33.549+0000 7f4fb8713640 1 -- 172.21.15.123:0/2922696335 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f4fb00b9e30 con 0x7f4fb00b3b20 2024-09-16T15:51:33.549 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:33.549+0000 7f4fa6ffd640 1 -- 172.21.15.123:0/2922696335 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v157) ==== 80+0+0 (secure 0 0 0) 0x7f4fac0f8d00 con 0x7f4fb00b3b20 2024-09-16T15:51:33.549 INFO:teuthology.orchestra.run.smithi123.stderr:listed 0 entries 2024-09-16T15:51:33.552 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:33.553+0000 7f4fb8713640 1 -- 172.21.15.123:0/2922696335 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f4f940738e0 msgr2=0x7f4f94075da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:51:33.552 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:33.553+0000 7f4fb8713640 1 --2- 172.21.15.123:0/2922696335 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f4f940738e0 0x7f4f94075da0 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f4fb01fb5a0 tx=0x7f4fa0006ca0 comp rx=0 tx=0).stop 2024-09-16T15:51:33.552 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:33.553+0000 7f4fb8713640 1 -- 172.21.15.123:0/2922696335 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4fb00b3b20 msgr2=0x7f4fb0156cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:51:33.552 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:33.553+0000 7f4fb8713640 1 --2- 172.21.15.123:0/2922696335 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4fb00b3b20 0x7f4fb0156cf0 secure :-1 s=READY pgs=1801 cs=0 l=1 rev1=1 crypto rx=0x7f4fac0b0040 tx=0x7f4fac005790 comp rx=0 tx=0).stop 2024-09-16T15:51:33.552 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:33.553+0000 7f4fb8713640 1 -- 172.21.15.123:0/2922696335 shutdown_connections 2024-09-16T15:51:33.552 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:33.553+0000 7f4fb8713640 1 --2- 172.21.15.123:0/2922696335 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f4f940738e0 0x7f4f94075da0 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:33.552 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:33.553+0000 7f4fb8713640 1 --2- 172.21.15.123:0/2922696335 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f4fb015dd60 0x7f4fb00caf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:33.552 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:33.553+0000 7f4fb8713640 1 --2- 172.21.15.123:0/2922696335 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4fb00d4030 0x7f4fb00b8df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:33.552 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:33.553+0000 7f4fb8713640 1 --2- 172.21.15.123:0/2922696335 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4fb00b3b20 0x7f4fb0156cf0 unknown :-1 s=CLOSED pgs=1801 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:33.553 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:33.553+0000 7f4fb8713640 1 -- 172.21.15.123:0/2922696335 >> 172.21.15.123:0/2922696335 conn(0x7f4fb0069650 msgr2=0x7f4fb015bac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:51:33.553 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:33.553+0000 7f4fb8713640 1 -- 172.21.15.123:0/2922696335 shutdown_connections 2024-09-16T15:51:33.553 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:33.553+0000 7f4fb8713640 1 -- 172.21.15.123:0/2922696335 wait complete. 2024-09-16T15:51:33.567 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi184.front.sepia.ceph.com... 2024-09-16T15:51:33.567 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:51:33.567 DEBUG:teuthology.orchestra.run.smithi184:> sudo fusermount -u /home/ubuntu/cephtest/mnt.1 2024-09-16T15:51:33.580 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi184.stderr:ceph-fuse[36401]: fuse finished with error 0 and tester_r 0 2024-09-16T15:51:33.582 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi184.front.sepia.ceph.com 2024-09-16T15:51:33.582 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:51:33.582 DEBUG:teuthology.orchestra.run.smithi184:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2024-09-16T15:51:33.719 INFO:tasks.cephfs.mount:Terminating background process 2024-09-16T15:51:33.719 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2024-09-16T15:51:33.720 DEBUG:teuthology.orchestra.run.smithi123:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-16T15:51:33.726 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16336468k,nr_inodes=4084117,mode=755,inode64 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3274684k,mode=755,inode64 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev,inode64 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=17082 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3274680k,nr_inodes=818670,mode=700,uid=1000,gid=1271,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3274684k,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-09-16T15:51:33.726 INFO:tasks.cephfs.mount:Terminating background process 2024-09-16T15:51:33.726 DEBUG:teuthology.orchestra.run.smithi184:> set -ex 2024-09-16T15:51:33.726 DEBUG:teuthology.orchestra.run.smithi184:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-16T15:51:33.732 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16336488k,nr_inodes=4084122,mode=755,inode64 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3274688k,mode=755,inode64 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev,inode64 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=17834 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3274684k,nr_inodes=818671,mode=700,uid=1000,gid=1271,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3274688k,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-09-16T15:51:33.732 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-16T15:51:33.897 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:33.897+0000 7fb1c10cd640 1 -- 172.21.15.123:0/2291734422 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fb1bc070910 msgr2=0x7fb1bc06d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:51:33.897 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:33.897+0000 7fb1c10cd640 1 --2- 172.21.15.123:0/2291734422 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fb1bc070910 0x7fb1bc06d300 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fb1b0009840 tx=0x7fb1b001d250 comp rx=0 tx=0).stop 2024-09-16T15:51:33.897 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:33.897+0000 7fb1c10cd640 1 -- 172.21.15.123:0/2291734422 shutdown_connections 2024-09-16T15:51:33.897 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:33.897+0000 7fb1c10cd640 1 --2- 172.21.15.123:0/2291734422 >> v2:172.21.15.184:3300/0 conn(0x7fb1bc15f840 0x7fb1bc073940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:33.897 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:33.897+0000 7fb1c10cd640 1 --2- 172.21.15.123:0/2291734422 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fb1bc070910 0x7fb1bc06d300 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:33.897 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:33.897+0000 7fb1c10cd640 1 -- 172.21.15.123:0/2291734422 >> 172.21.15.123:0/2291734422 conn(0x7fb1bc0d0b70 msgr2=0x7fb1bc0d0f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:51:33.897 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:33.897+0000 7fb1c10cd640 1 -- 172.21.15.123:0/2291734422 shutdown_connections 2024-09-16T15:51:33.897 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:33.897+0000 7fb1c10cd640 1 -- 172.21.15.123:0/2291734422 wait complete. 2024-09-16T15:51:33.898 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:33.897+0000 7fb1c10cd640 1 Processor -- start 2024-09-16T15:51:33.898 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:33.897+0000 7fb1c10cd640 1 -- start start 2024-09-16T15:51:33.900 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:33.897+0000 7fb1c10cd640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fb1bc070910 0x7fb1bc0cb640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:51:33.900 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:33.897+0000 7fb1c10cd640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb1bc15bce0 0x7fb1bc0c61f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:51:33.900 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:33.897+0000 7fb1c10cd640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb1bc15f840 0x7fb1bc0b1c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:51:33.900 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:33.897+0000 7fb1c10cd640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fb1bc168b80 con 0x7fb1bc15bce0 2024-09-16T15:51:33.900 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:33.897+0000 7fb1c10cd640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fb1bc0749b0 con 0x7fb1bc15f840 2024-09-16T15:51:33.900 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:33.897+0000 7fb1c10cd640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fb1bc168d00 con 0x7fb1bc070910 2024-09-16T15:51:33.900 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:33.901+0000 7fb1ba59c640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb1bc15bce0 0x7fb1bc0c61f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:51:33.901 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:33.901+0000 7fb1ba59c640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb1bc15bce0 0x7fb1bc0c61f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:50132/0 (socket says 172.21.15.123:50132) 2024-09-16T15:51:33.901 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:33.901+0000 7fb1ba59c640 1 -- 172.21.15.123:0/3639364131 learned_addr learned my addr 172.21.15.123:0/3639364131 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:51:33.901 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:33.901+0000 7fb1bad9d640 1 --2- 172.21.15.123:0/3639364131 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fb1bc070910 0x7fb1bc0cb640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:51:33.901 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:33.901+0000 7fb1bb59e640 1 --2- 172.21.15.123:0/3639364131 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb1bc15f840 0x7fb1bc0b1c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:51:33.901 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:33.901+0000 7fb1ba59c640 1 -- 172.21.15.123:0/3639364131 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fb1bc070910 msgr2=0x7fb1bc0cb640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:51:33.901 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:33.901+0000 7fb1ba59c640 1 --2- 172.21.15.123:0/3639364131 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fb1bc070910 0x7fb1bc0cb640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:33.901 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:33.901+0000 7fb1ba59c640 1 -- 172.21.15.123:0/3639364131 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb1bc15f840 msgr2=0x7fb1bc0b1c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:51:33.901 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:33.901+0000 7fb1ba59c640 1 --2- 172.21.15.123:0/3639364131 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb1bc15f840 0x7fb1bc0b1c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:33.901 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:33.901+0000 7fb1ba59c640 1 -- 172.21.15.123:0/3639364131 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb1bc0c6730 con 0x7fb1bc15bce0 2024-09-16T15:51:33.901 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:33.901+0000 7fb1bb59e640 1 --2- 172.21.15.123:0/3639364131 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb1bc15f840 0x7fb1bc0b1c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T15:51:33.901 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:33.901+0000 7fb1ba59c640 1 --2- 172.21.15.123:0/3639364131 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb1bc15bce0 0x7fb1bc0c61f0 secure :-1 s=READY pgs=1802 cs=0 l=1 rev1=1 crypto rx=0x7fb1b0009790 tx=0x7fb1b00a56d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:51:33.901 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:33.901+0000 7fb1a37fe640 1 -- 172.21.15.123:0/3639364131 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1b00b2040 con 0x7fb1bc15bce0 2024-09-16T15:51:33.902 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:33.901+0000 7fb1a37fe640 1 -- 172.21.15.123:0/3639364131 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fb1b00043b0 con 0x7fb1bc15bce0 2024-09-16T15:51:33.902 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:33.901+0000 7fb1c10cd640 1 -- 172.21.15.123:0/3639364131 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb1bc1f96c0 con 0x7fb1bc15bce0 2024-09-16T15:51:33.902 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:33.901+0000 7fb1a37fe640 1 -- 172.21.15.123:0/3639364131 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1b0003800 con 0x7fb1bc15bce0 2024-09-16T15:51:33.902 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:33.901+0000 7fb1c10cd640 1 -- 172.21.15.123:0/3639364131 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb1bc0b2b90 con 0x7fb1bc15bce0 2024-09-16T15:51:33.903 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:33.901+0000 7fb1c10cd640 1 -- 172.21.15.123:0/3639364131 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb1bc0b2dd0 con 0x7fb1bc15bce0 2024-09-16T15:51:33.903 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:33.901+0000 7fb1a37fe640 1 -- 172.21.15.123:0/3639364131 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 25) ==== 97998+0+0 (secure 0 0 0) 0x7fb1b00bd060 con 0x7fb1bc15bce0 2024-09-16T15:51:33.903 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:33.901+0000 7fb1a37fe640 1 --2- 172.21.15.123:0/3639364131 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fb1a80738e0 0x7fb1a8075da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:51:33.903 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:33.901+0000 7fb1a37fe640 1 -- 172.21.15.123:0/3639364131 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(157..157 src has 1..157) ==== 5737+0+0 (secure 0 0 0) 0x7fb1b012c1e0 con 0x7fb1bc15bce0 2024-09-16T15:51:33.903 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:33.901+0000 7fb1bad9d640 1 --2- 172.21.15.123:0/3639364131 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fb1a80738e0 0x7fb1a8075da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:51:33.903 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:33.901+0000 7fb1bad9d640 1 --2- 172.21.15.123:0/3639364131 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fb1a80738e0 0x7fb1a8075da0 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7fb1ac0012f0 tx=0x7fb1ac009040 comp rx=0 tx=0).ready entity=mgr.4150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:51:33.906 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:33.905+0000 7fb1a37fe640 1 -- 172.21.15.123:0/3639364131 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7fb1bc0b2dd0 con 0x7fb1bc15bce0 2024-09-16T15:51:34.117 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:34.117+0000 7fb1c10cd640 1 -- 172.21.15.123:0/3639364131 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb1bc073940 con 0x7fb1bc15bce0 2024-09-16T15:51:34.118 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:34.117+0000 7fb1a37fe640 1 -- 172.21.15.123:0/3639364131 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 106 v106) ==== 95+0+5244 (secure 0 0 0) 0x7fb1b00ff070 con 0x7fb1bc15bce0 2024-09-16T15:51:34.118 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-09-16T15:51:34.118 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":106,"btime":"2024-09-16T15:51:29:191457+0000","default_fscid":12,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6200,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6835/2380355001","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6833","nonce":2380355001},{"type":"v1","addr":"172.21.15.184:6835","nonce":2380355001}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":100},{"gid":8844,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6834/1840463681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6832","nonce":1840463681},{"type":"v1","addr":"172.21.15.123:6834","nonce":1840463681}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":100},{"gid":8847,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6835/2493915107","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6833","nonce":2493915107},{"type":"v1","addr":"172.21.15.123:6835","nonce":2493915107}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":99}],"filesystems":[{"mdsmap":{"epoch":105,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-16T15:51:16.301945+0000","modified":"2024-09-16T15:51:19.374973+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8859},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8859":{"gid":8859,"name":"d","rank":0,"incarnation":101,"state":"up:active","state_seq":3,"addr":"172.21.15.184:6834/2856066034","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":2856066034},{"type":"v1","addr":"172.21.15.184:6834","nonce":2856066034}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[25],"metadata_pool":24,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8859,"qdb_cluster":[8859]},"id":12}]} 2024-09-16T15:51:34.118 INFO:teuthology.orchestra.run.smithi123.stderr:dumped fsmap epoch 106 2024-09-16T15:51:34.121 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:34.121+0000 7fb1c10cd640 1 -- 172.21.15.123:0/3639364131 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fb1a80738e0 msgr2=0x7fb1a8075da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:51:34.121 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:34.121+0000 7fb1c10cd640 1 --2- 172.21.15.123:0/3639364131 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fb1a80738e0 0x7fb1a8075da0 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7fb1ac0012f0 tx=0x7fb1ac009040 comp rx=0 tx=0).stop 2024-09-16T15:51:34.121 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:34.121+0000 7fb1c10cd640 1 -- 172.21.15.123:0/3639364131 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb1bc15bce0 msgr2=0x7fb1bc0c61f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:51:34.121 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:34.121+0000 7fb1c10cd640 1 --2- 172.21.15.123:0/3639364131 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb1bc15bce0 0x7fb1bc0c61f0 secure :-1 s=READY pgs=1802 cs=0 l=1 rev1=1 crypto rx=0x7fb1b0009790 tx=0x7fb1b00a56d0 comp rx=0 tx=0).stop 2024-09-16T15:51:34.121 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:34.121+0000 7fb1c10cd640 1 -- 172.21.15.123:0/3639364131 shutdown_connections 2024-09-16T15:51:34.121 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:34.121+0000 7fb1c10cd640 1 --2- 172.21.15.123:0/3639364131 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fb1a80738e0 0x7fb1a8075da0 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:34.121 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:34.121+0000 7fb1c10cd640 1 --2- 172.21.15.123:0/3639364131 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb1bc15f840 0x7fb1bc0b1c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:34.121 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:34.121+0000 7fb1c10cd640 1 --2- 172.21.15.123:0/3639364131 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb1bc15bce0 0x7fb1bc0c61f0 unknown :-1 s=CLOSED pgs=1802 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:34.121 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:34.121+0000 7fb1c10cd640 1 --2- 172.21.15.123:0/3639364131 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fb1bc070910 0x7fb1bc0cb640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:34.121 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:34.121+0000 7fb1c10cd640 1 -- 172.21.15.123:0/3639364131 >> 172.21.15.123:0/3639364131 conn(0x7fb1bc0d0b70 msgr2=0x7fb1bc156190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:51:34.122 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:34.121+0000 7fb1c10cd640 1 -- 172.21.15.123:0/3639364131 shutdown_connections 2024-09-16T15:51:34.122 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:34.121+0000 7fb1c10cd640 1 -- 172.21.15.123:0/3639364131 wait complete. 2024-09-16T15:51:34.138 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-16T15:51:34.268 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:34.269+0000 7f69c4ff4640 1 -- 172.21.15.123:0/1985101900 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f69c00cc900 msgr2=0x7f69c00ce5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:51:34.268 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:34.269+0000 7f69c4ff4640 1 --2- 172.21.15.123:0/1985101900 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f69c00cc900 0x7f69c00ce5e0 secure :-1 s=READY pgs=1803 cs=0 l=1 rev1=1 crypto rx=0x7f69b8009c80 tx=0x7f69b801f5a0 comp rx=0 tx=0).stop 2024-09-16T15:51:34.268 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:34.269+0000 7f69c4ff4640 1 -- 172.21.15.123:0/1985101900 shutdown_connections 2024-09-16T15:51:34.268 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:34.269+0000 7f69c4ff4640 1 --2- 172.21.15.123:0/1985101900 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f69c00cc900 0x7f69c00ce5e0 unknown :-1 s=CLOSED pgs=1803 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:34.268 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:34.269+0000 7f69c4ff4640 1 --2- 172.21.15.123:0/1985101900 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f69c015e380 0x7f69c00b0500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:34.268 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:34.269+0000 7f69c4ff4640 1 -- 172.21.15.123:0/1985101900 >> 172.21.15.123:0/1985101900 conn(0x7f69c00b1df0 msgr2=0x7f69c00b2200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:51:34.269 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:34.269+0000 7f69c4ff4640 1 -- 172.21.15.123:0/1985101900 shutdown_connections 2024-09-16T15:51:34.269 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:34.269+0000 7f69c4ff4640 1 -- 172.21.15.123:0/1985101900 wait complete. 2024-09-16T15:51:34.269 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:34.269+0000 7f69c4ff4640 1 Processor -- start 2024-09-16T15:51:34.269 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:34.269+0000 7f69c4ff4640 1 -- start start 2024-09-16T15:51:34.271 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:34.269+0000 7f69c4ff4640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f69c00cc3b0 0x7f69c0157780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:51:34.271 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:34.269+0000 7f69c4ff4640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f69c00cc900 0x7f69c00b2770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:51:34.271 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:34.269+0000 7f69c4ff4640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f69c015e380 0x7f69c00ccce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:51:34.271 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:34.269+0000 7f69c4ff4640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f69c00b0dd0 con 0x7f69c015e380 2024-09-16T15:51:34.271 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:34.269+0000 7f69c4ff4640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f69c00b0c50 con 0x7f69c00cc900 2024-09-16T15:51:34.271 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:34.269+0000 7f69c4ff4640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f69c00b0f50 con 0x7f69c00cc3b0 2024-09-16T15:51:34.271 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:34.269+0000 7f69bed9d640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f69c015e380 0x7f69c00ccce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:51:34.271 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:34.269+0000 7f69bed9d640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f69c015e380 0x7f69c00ccce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:50156/0 (socket says 172.21.15.123:50156) 2024-09-16T15:51:34.271 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:34.269+0000 7f69bed9d640 1 -- 172.21.15.123:0/2541373829 learned_addr learned my addr 172.21.15.123:0/2541373829 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:51:34.272 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:34.269+0000 7f69be59c640 1 --2- 172.21.15.123:0/2541373829 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f69c00cc3b0 0x7f69c0157780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:51:34.272 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:34.269+0000 7f69b7fff640 1 --2- 172.21.15.123:0/2541373829 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f69c00cc900 0x7f69c00b2770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:51:34.272 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:34.269+0000 7f69bed9d640 1 -- 172.21.15.123:0/2541373829 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f69c00cc3b0 msgr2=0x7f69c0157780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:51:34.272 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:34.269+0000 7f69bed9d640 1 --2- 172.21.15.123:0/2541373829 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f69c00cc3b0 0x7f69c0157780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:34.272 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:34.269+0000 7f69bed9d640 1 -- 172.21.15.123:0/2541373829 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f69c00cc900 msgr2=0x7f69c00b2770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:51:34.272 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:34.269+0000 7f69bed9d640 1 --2- 172.21.15.123:0/2541373829 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f69c00cc900 0x7f69c00b2770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:34.272 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:34.269+0000 7f69bed9d640 1 -- 172.21.15.123:0/2541373829 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f69c01f4960 con 0x7f69c015e380 2024-09-16T15:51:34.272 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:34.269+0000 7f69be59c640 1 --2- 172.21.15.123:0/2541373829 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f69c00cc3b0 0x7f69c0157780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T15:51:34.272 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:34.269+0000 7f69b7fff640 1 --2- 172.21.15.123:0/2541373829 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f69c00cc900 0x7f69c00b2770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T15:51:34.272 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:34.269+0000 7f69bed9d640 1 --2- 172.21.15.123:0/2541373829 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f69c015e380 0x7f69c00ccce0 secure :-1 s=READY pgs=1804 cs=0 l=1 rev1=1 crypto rx=0x7f69b801f3d0 tx=0x7f69b80b1460 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:51:34.272 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:34.273+0000 7f69b6ffd640 1 -- 172.21.15.123:0/2541373829 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69b80b6040 con 0x7f69c015e380 2024-09-16T15:51:34.272 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:34.273+0000 7f69b6ffd640 1 -- 172.21.15.123:0/2541373829 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f69b80bf070 con 0x7f69c015e380 2024-09-16T15:51:34.272 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:34.273+0000 7f69c4ff4640 1 -- 172.21.15.123:0/2541373829 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f69c00c6ec0 con 0x7f69c015e380 2024-09-16T15:51:34.272 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:34.273+0000 7f69b6ffd640 1 -- 172.21.15.123:0/2541373829 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69b80b2970 con 0x7f69c015e380 2024-09-16T15:51:34.273 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:34.273+0000 7f69c4ff4640 1 -- 172.21.15.123:0/2541373829 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f69c00c77e0 con 0x7f69c015e380 2024-09-16T15:51:34.273 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:34.273+0000 7f69c4ff4640 1 -- 172.21.15.123:0/2541373829 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f69c00b0500 con 0x7f69c015e380 2024-09-16T15:51:34.274 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:34.273+0000 7f69b6ffd640 1 -- 172.21.15.123:0/2541373829 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 25) ==== 97998+0+0 (secure 0 0 0) 0x7f69b80b2b10 con 0x7f69c015e380 2024-09-16T15:51:34.274 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:34.273+0000 7f69b6ffd640 1 --2- 172.21.15.123:0/2541373829 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f69a00738e0 0x7f69a0075da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:51:34.274 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:34.273+0000 7f69b6ffd640 1 -- 172.21.15.123:0/2541373829 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(157..157 src has 1..157) ==== 5737+0+0 (secure 0 0 0) 0x7f69b8130ce0 con 0x7f69c015e380 2024-09-16T15:51:34.274 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:34.273+0000 7f69be59c640 1 --2- 172.21.15.123:0/2541373829 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f69a00738e0 0x7f69a0075da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:51:34.274 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:34.273+0000 7f69be59c640 1 --2- 172.21.15.123:0/2541373829 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f69a00738e0 0x7f69a0075da0 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f69b00058d0 tx=0x7f69b0005ca0 comp rx=0 tx=0).ready entity=mgr.4150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:51:34.277 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:34.277+0000 7f69b6ffd640 1 -- 172.21.15.123:0/2541373829 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f69b80ab070 con 0x7f69c015e380 2024-09-16T15:51:34.486 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:34.485+0000 7f69c4ff4640 1 -- 172.21.15.123:0/2541373829 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f69c0073930 con 0x7f69c015e380 2024-09-16T15:51:34.487 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:34.485+0000 7f69b6ffd640 1 -- 172.21.15.123:0/2541373829 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 106 v106) ==== 95+0+5244 (secure 0 0 0) 0x7f69b8019100 con 0x7f69c015e380 2024-09-16T15:51:34.487 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-09-16T15:51:34.487 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":106,"btime":"2024-09-16T15:51:29:191457+0000","default_fscid":12,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6200,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6835/2380355001","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6833","nonce":2380355001},{"type":"v1","addr":"172.21.15.184:6835","nonce":2380355001}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":100},{"gid":8844,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6834/1840463681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6832","nonce":1840463681},{"type":"v1","addr":"172.21.15.123:6834","nonce":1840463681}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":100},{"gid":8847,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6835/2493915107","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6833","nonce":2493915107},{"type":"v1","addr":"172.21.15.123:6835","nonce":2493915107}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":99}],"filesystems":[{"mdsmap":{"epoch":105,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-16T15:51:16.301945+0000","modified":"2024-09-16T15:51:19.374973+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8859},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8859":{"gid":8859,"name":"d","rank":0,"incarnation":101,"state":"up:active","state_seq":3,"addr":"172.21.15.184:6834/2856066034","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":2856066034},{"type":"v1","addr":"172.21.15.184:6834","nonce":2856066034}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[25],"metadata_pool":24,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8859,"qdb_cluster":[8859]},"id":12}]} 2024-09-16T15:51:34.487 INFO:teuthology.orchestra.run.smithi123.stderr:dumped fsmap epoch 106 2024-09-16T15:51:34.489 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:34.489+0000 7f69c4ff4640 1 -- 172.21.15.123:0/2541373829 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f69a00738e0 msgr2=0x7f69a0075da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:51:34.489 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:34.489+0000 7f69c4ff4640 1 --2- 172.21.15.123:0/2541373829 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f69a00738e0 0x7f69a0075da0 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f69b00058d0 tx=0x7f69b0005ca0 comp rx=0 tx=0).stop 2024-09-16T15:51:34.489 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:34.489+0000 7f69c4ff4640 1 -- 172.21.15.123:0/2541373829 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f69c015e380 msgr2=0x7f69c00ccce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:51:34.489 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:34.489+0000 7f69c4ff4640 1 --2- 172.21.15.123:0/2541373829 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f69c015e380 0x7f69c00ccce0 secure :-1 s=READY pgs=1804 cs=0 l=1 rev1=1 crypto rx=0x7f69b801f3d0 tx=0x7f69b80b1460 comp rx=0 tx=0).stop 2024-09-16T15:51:34.489 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:34.489+0000 7f69c4ff4640 1 -- 172.21.15.123:0/2541373829 shutdown_connections 2024-09-16T15:51:34.490 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:34.489+0000 7f69c4ff4640 1 --2- 172.21.15.123:0/2541373829 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f69a00738e0 0x7f69a0075da0 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:34.490 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:34.489+0000 7f69c4ff4640 1 --2- 172.21.15.123:0/2541373829 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f69c015e380 0x7f69c00ccce0 unknown :-1 s=CLOSED pgs=1804 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:34.490 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:34.489+0000 7f69c4ff4640 1 --2- 172.21.15.123:0/2541373829 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f69c00cc900 0x7f69c00b2770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:34.490 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:34.489+0000 7f69c4ff4640 1 --2- 172.21.15.123:0/2541373829 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f69c00cc3b0 0x7f69c0157780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:34.490 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:34.489+0000 7f69c4ff4640 1 -- 172.21.15.123:0/2541373829 >> 172.21.15.123:0/2541373829 conn(0x7f69c00b1df0 msgr2=0x7f69c006f0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:51:34.490 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:34.489+0000 7f69c4ff4640 1 -- 172.21.15.123:0/2541373829 shutdown_connections 2024-09-16T15:51:34.490 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:34.489+0000 7f69c4ff4640 1 -- 172.21.15.123:0/2541373829 wait complete. 2024-09-16T15:51:34.506 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-16T15:51:34.627 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:34.629+0000 7fa33785b640 1 -- 172.21.15.123:0/997663791 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa3300726c0 msgr2=0x7fa330073ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:51:34.627 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:34.629+0000 7fa33785b640 1 --2- 172.21.15.123:0/997663791 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa3300726c0 0x7fa330073ef0 secure :-1 s=READY pgs=1805 cs=0 l=1 rev1=1 crypto rx=0x7fa3300cd650 tx=0x7fa32001c800 comp rx=0 tx=0).stop 2024-09-16T15:51:34.628 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:34.629+0000 7fa33785b640 1 -- 172.21.15.123:0/997663791 shutdown_connections 2024-09-16T15:51:34.628 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:34.629+0000 7fa33785b640 1 --2- 172.21.15.123:0/997663791 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa3300726c0 0x7fa330073ef0 unknown :-1 s=CLOSED pgs=1805 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:34.628 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:34.629+0000 7fa33785b640 1 --2- 172.21.15.123:0/997663791 >> v2:172.21.15.184:3300/0 conn(0x7fa33015bca0 0x7fa3300d0630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:34.628 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:34.629+0000 7fa33785b640 1 -- 172.21.15.123:0/997663791 >> 172.21.15.123:0/997663791 conn(0x7fa3300cfb10 msgr2=0x7fa3300cff20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:51:34.628 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:34.629+0000 7fa33785b640 1 -- 172.21.15.123:0/997663791 shutdown_connections 2024-09-16T15:51:34.628 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:34.629+0000 7fa33785b640 1 -- 172.21.15.123:0/997663791 wait complete. 2024-09-16T15:51:34.628 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:34.629+0000 7fa33785b640 1 Processor -- start 2024-09-16T15:51:34.628 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:34.629+0000 7fa33785b640 1 -- start start 2024-09-16T15:51:34.631 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:34.629+0000 7fa33785b640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fa3300726c0 0x7fa330164db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:51:34.631 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:34.629+0000 7fa33785b640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa330074430 0x7fa3300cf1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:51:34.631 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:34.629+0000 7fa33785b640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa33015bca0 0x7fa3300c47a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:51:34.631 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:34.629+0000 7fa33785b640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fa330168f40 con 0x7fa330074430 2024-09-16T15:51:34.631 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:34.629+0000 7fa33785b640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fa330168dc0 con 0x7fa33015bca0 2024-09-16T15:51:34.631 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:34.629+0000 7fa33785b640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fa3301690c0 con 0x7fa3300726c0 2024-09-16T15:51:34.631 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:34.629+0000 7fa334df6640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa330074430 0x7fa3300cf1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:51:34.631 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:34.629+0000 7fa334df6640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa330074430 0x7fa3300cf1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:50168/0 (socket says 172.21.15.123:50168) 2024-09-16T15:51:34.631 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:34.629+0000 7fa334df6640 1 -- 172.21.15.123:0/3714422251 learned_addr learned my addr 172.21.15.123:0/3714422251 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:51:34.631 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:34.629+0000 7fa334df6640 1 -- 172.21.15.123:0/3714422251 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fa3300726c0 msgr2=0x7fa330164db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:51:34.631 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:34.629+0000 7fa3355f7640 1 --2- 172.21.15.123:0/3714422251 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fa3300726c0 0x7fa330164db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:51:34.632 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:34.629+0000 7fa335df8640 1 --2- 172.21.15.123:0/3714422251 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa33015bca0 0x7fa3300c47a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:51:34.632 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:34.629+0000 7fa334df6640 1 --2- 172.21.15.123:0/3714422251 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fa3300726c0 0x7fa330164db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:34.632 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:34.629+0000 7fa334df6640 1 -- 172.21.15.123:0/3714422251 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa33015bca0 msgr2=0x7fa3300c47a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:51:34.632 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:34.629+0000 7fa334df6640 1 --2- 172.21.15.123:0/3714422251 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa33015bca0 0x7fa3300c47a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:34.632 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:34.629+0000 7fa334df6640 1 -- 172.21.15.123:0/3714422251 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa3300cf6e0 con 0x7fa330074430 2024-09-16T15:51:34.632 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:34.629+0000 7fa3355f7640 1 --2- 172.21.15.123:0/3714422251 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fa3300726c0 0x7fa330164db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T15:51:34.632 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:34.629+0000 7fa335df8640 1 --2- 172.21.15.123:0/3714422251 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa33015bca0 0x7fa3300c47a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T15:51:34.632 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:34.629+0000 7fa334df6640 1 --2- 172.21.15.123:0/3714422251 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa330074430 0x7fa3300cf1a0 secure :-1 s=READY pgs=1806 cs=0 l=1 rev1=1 crypto rx=0x7fa3200b0040 tx=0x7fa3200a57e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:51:34.632 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:34.629+0000 7fa325ffb640 1 -- 172.21.15.123:0/3714422251 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3200b2040 con 0x7fa330074430 2024-09-16T15:51:34.632 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:34.629+0000 7fa325ffb640 1 -- 172.21.15.123:0/3714422251 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fa3200043e0 con 0x7fa330074430 2024-09-16T15:51:34.632 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:34.629+0000 7fa33785b640 1 -- 172.21.15.123:0/3714422251 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa330165500 con 0x7fa330074430 2024-09-16T15:51:34.632 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:34.629+0000 7fa325ffb640 1 -- 172.21.15.123:0/3714422251 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa320005290 con 0x7fa330074430 2024-09-16T15:51:34.632 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:34.629+0000 7fa33785b640 1 -- 172.21.15.123:0/3714422251 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa330165ac0 con 0x7fa330074430 2024-09-16T15:51:34.633 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:34.633+0000 7fa33785b640 1 -- 172.21.15.123:0/3714422251 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa330073ef0 con 0x7fa330074430 2024-09-16T15:51:34.633 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:34.633+0000 7fa325ffb640 1 -- 172.21.15.123:0/3714422251 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 25) ==== 97998+0+0 (secure 0 0 0) 0x7fa320005430 con 0x7fa330074430 2024-09-16T15:51:34.633 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:34.633+0000 7fa325ffb640 1 --2- 172.21.15.123:0/3714422251 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fa3140738e0 0x7fa314075da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:51:34.633 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:34.633+0000 7fa325ffb640 1 -- 172.21.15.123:0/3714422251 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(157..157 src has 1..157) ==== 5737+0+0 (secure 0 0 0) 0x7fa32012c530 con 0x7fa330074430 2024-09-16T15:51:34.633 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:34.633+0000 7fa3355f7640 1 --2- 172.21.15.123:0/3714422251 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fa3140738e0 0x7fa314075da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:51:34.634 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:34.633+0000 7fa3355f7640 1 --2- 172.21.15.123:0/3714422251 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fa3140738e0 0x7fa314075da0 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7fa32c00a870 tx=0x7fa32c008040 comp rx=0 tx=0).ready entity=mgr.4150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:51:34.636 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:34.637+0000 7fa325ffb640 1 -- 172.21.15.123:0/3714422251 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7fa3200164b0 con 0x7fa330074430 2024-09-16T15:51:34.809 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:34.809+0000 7fa33785b640 1 -- 172.21.15.123:0/3714422251 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fa330050f80 con 0x7fa330074430 2024-09-16T15:51:34.811 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:34.813+0000 7fa325ffb640 1 -- 172.21.15.123:0/3714422251 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v157) ==== 74+0+16843 (secure 0 0 0) 0x7fa3200f9010 con 0x7fa330074430 2024-09-16T15:51:34.812 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-09-16T15:51:34.812 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":157,"fsid":"56077f89-18c8-486f-ae7e-eecc0bb7822c","created":"2024-09-16T15:42:26.364892+0000","modified":"2024-09-16T15:51:17.980147+0000","last_up_change":"2024-09-16T15:42:38.707234+0000","last_in_change":"2024-09-16T15:42:29.423662+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":25,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-16T15:42:39.287764+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":24,"pool_name":"cephfs_metadata","create_time":"2024-09-16T15:51:13.953990+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"156","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":25,"pool_name":"cephfs_data","create_time":"2024-09-16T15:51:15.281388+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"157","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9e7c1955-a607-4671-9be7-f2ef760ceeaa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":155,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6800","nonce":3882044969},{"type":"v1","addr":"172.21.15.123:6801","nonce":3882044969}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6802","nonce":3882044969},{"type":"v1","addr":"172.21.15.123:6803","nonce":3882044969}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6806","nonce":3882044969},{"type":"v1","addr":"172.21.15.123:6807","nonce":3882044969}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6804","nonce":3882044969},{"type":"v1","addr":"172.21.15.123:6805","nonce":3882044969}]},"public_addr":"172.21.15.123:6801/3882044969","cluster_addr":"172.21.15.123:6803/3882044969","heartbeat_back_addr":"172.21.15.123:6807/3882044969","heartbeat_front_addr":"172.21.15.123:6805/3882044969","state":["exists","up"]},{"osd":1,"uuid":"c540cfde-0977-4742-9b17-fd634668e010","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":155,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6816","nonce":4037001850},{"type":"v1","addr":"172.21.15.123:6817","nonce":4037001850}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6818","nonce":4037001850},{"type":"v1","addr":"172.21.15.123:6819","nonce":4037001850}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6822","nonce":4037001850},{"type":"v1","addr":"172.21.15.123:6823","nonce":4037001850}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6820","nonce":4037001850},{"type":"v1","addr":"172.21.15.123:6821","nonce":4037001850}]},"public_addr":"172.21.15.123:6817/4037001850","cluster_addr":"172.21.15.123:6819/4037001850","heartbeat_back_addr":"172.21.15.123:6823/4037001850","heartbeat_front_addr":"172.21.15.123:6821/4037001850","state":["exists","up"]},{"osd":2,"uuid":"9864a496-f169-486b-95c4-1d59162793a7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":155,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6824","nonce":1332143088},{"type":"v1","addr":"172.21.15.123:6825","nonce":1332143088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6826","nonce":1332143088},{"type":"v1","addr":"172.21.15.123:6827","nonce":1332143088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6830","nonce":1332143088},{"type":"v1","addr":"172.21.15.123:6831","nonce":1332143088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6828","nonce":1332143088},{"type":"v1","addr":"172.21.15.123:6829","nonce":1332143088}]},"public_addr":"172.21.15.123:6825/1332143088","cluster_addr":"172.21.15.123:6827/1332143088","heartbeat_back_addr":"172.21.15.123:6831/1332143088","heartbeat_front_addr":"172.21.15.123:6829/1332143088","state":["exists","up"]},{"osd":3,"uuid":"c6d1f817-f074-4325-95e1-578eba13edda","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":155,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6808","nonce":988586998},{"type":"v1","addr":"172.21.15.123:6809","nonce":988586998}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6810","nonce":988586998},{"type":"v1","addr":"172.21.15.123:6811","nonce":988586998}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6814","nonce":988586998},{"type":"v1","addr":"172.21.15.123:6815","nonce":988586998}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6812","nonce":988586998},{"type":"v1","addr":"172.21.15.123:6813","nonce":988586998}]},"public_addr":"172.21.15.123:6809/988586998","cluster_addr":"172.21.15.123:6811/988586998","heartbeat_back_addr":"172.21.15.123:6815/988586998","heartbeat_front_addr":"172.21.15.123:6813/988586998","state":["exists","up"]},{"osd":4,"uuid":"200a8df4-50ef-40a5-b48a-58981269c086","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":155,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6816","nonce":3387813890},{"type":"v1","addr":"172.21.15.184:6817","nonce":3387813890}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6818","nonce":3387813890},{"type":"v1","addr":"172.21.15.184:6819","nonce":3387813890}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6822","nonce":3387813890},{"type":"v1","addr":"172.21.15.184:6823","nonce":3387813890}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6820","nonce":3387813890},{"type":"v1","addr":"172.21.15.184:6821","nonce":3387813890}]},"public_addr":"172.21.15.184:6817/3387813890","cluster_addr":"172.21.15.184:6819/3387813890","heartbeat_back_addr":"172.21.15.184:6823/3387813890","heartbeat_front_addr":"172.21.15.184:6821/3387813890","state":["exists","up"]},{"osd":5,"uuid":"32d5a900-da3d-412c-bc19-13861a2368d8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":155,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6808","nonce":2680916017},{"type":"v1","addr":"172.21.15.184:6809","nonce":2680916017}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6810","nonce":2680916017},{"type":"v1","addr":"172.21.15.184:6811","nonce":2680916017}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6814","nonce":2680916017},{"type":"v1","addr":"172.21.15.184:6815","nonce":2680916017}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6812","nonce":2680916017},{"type":"v1","addr":"172.21.15.184:6813","nonce":2680916017}]},"public_addr":"172.21.15.184:6809/2680916017","cluster_addr":"172.21.15.184:6811/2680916017","heartbeat_back_addr":"172.21.15.184:6815/2680916017","heartbeat_front_addr":"172.21.15.184:6813/2680916017","state":["exists","up"]},{"osd":6,"uuid":"cde6f39e-aa1b-4260-ac19-774ee6aaca72","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":155,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6824","nonce":2183120457},{"type":"v1","addr":"172.21.15.184:6825","nonce":2183120457}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6826","nonce":2183120457},{"type":"v1","addr":"172.21.15.184:6827","nonce":2183120457}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6830","nonce":2183120457},{"type":"v1","addr":"172.21.15.184:6831","nonce":2183120457}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6828","nonce":2183120457},{"type":"v1","addr":"172.21.15.184:6829","nonce":2183120457}]},"public_addr":"172.21.15.184:6825/2183120457","cluster_addr":"172.21.15.184:6827/2183120457","heartbeat_back_addr":"172.21.15.184:6831/2183120457","heartbeat_front_addr":"172.21.15.184:6829/2183120457","state":["exists","up"]},{"osd":7,"uuid":"4f381778-eee1-47c6-804f-df0e6b33e313","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":155,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6800","nonce":4102957285},{"type":"v1","addr":"172.21.15.184:6801","nonce":4102957285}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6802","nonce":4102957285},{"type":"v1","addr":"172.21.15.184:6803","nonce":4102957285}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6806","nonce":4102957285},{"type":"v1","addr":"172.21.15.184:6807","nonce":4102957285}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6804","nonce":4102957285},{"type":"v1","addr":"172.21.15.184:6805","nonce":4102957285}]},"public_addr":"172.21.15.184:6801/4102957285","cluster_addr":"172.21.15.184:6803/4102957285","heartbeat_back_addr":"172.21.15.184:6807/4102957285","heartbeat_front_addr":"172.21.15.184:6805/4102957285","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:35.972210+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.222007+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.198617+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.385663+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.358935+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.156066+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.752973+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:35.682846+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":17,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T15:51:34.814 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:34.813+0000 7fa33785b640 1 -- 172.21.15.123:0/3714422251 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fa3140738e0 msgr2=0x7fa314075da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:51:34.814 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:34.813+0000 7fa33785b640 1 --2- 172.21.15.123:0/3714422251 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fa3140738e0 0x7fa314075da0 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7fa32c00a870 tx=0x7fa32c008040 comp rx=0 tx=0).stop 2024-09-16T15:51:34.814 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:34.813+0000 7fa33785b640 1 -- 172.21.15.123:0/3714422251 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa330074430 msgr2=0x7fa3300cf1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:51:34.815 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:34.813+0000 7fa33785b640 1 --2- 172.21.15.123:0/3714422251 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa330074430 0x7fa3300cf1a0 secure :-1 s=READY pgs=1806 cs=0 l=1 rev1=1 crypto rx=0x7fa3200b0040 tx=0x7fa3200a57e0 comp rx=0 tx=0).stop 2024-09-16T15:51:34.815 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:34.813+0000 7fa33785b640 1 -- 172.21.15.123:0/3714422251 shutdown_connections 2024-09-16T15:51:34.815 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:34.813+0000 7fa33785b640 1 --2- 172.21.15.123:0/3714422251 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fa3140738e0 0x7fa314075da0 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:34.815 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:34.813+0000 7fa33785b640 1 --2- 172.21.15.123:0/3714422251 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa33015bca0 0x7fa3300c47a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:34.815 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:34.813+0000 7fa33785b640 1 --2- 172.21.15.123:0/3714422251 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa330074430 0x7fa3300cf1a0 unknown :-1 s=CLOSED pgs=1806 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:34.815 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:34.813+0000 7fa33785b640 1 --2- 172.21.15.123:0/3714422251 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fa3300726c0 0x7fa330164db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:34.815 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:34.813+0000 7fa33785b640 1 -- 172.21.15.123:0/3714422251 >> 172.21.15.123:0/3714422251 conn(0x7fa3300cfb10 msgr2=0x7fa33015aab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:51:34.815 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:34.813+0000 7fa33785b640 1 -- 172.21.15.123:0/3714422251 shutdown_connections 2024-09-16T15:51:34.815 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:34.813+0000 7fa33785b640 1 -- 172.21.15.123:0/3714422251 wait complete. 2024-09-16T15:51:34.831 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-09-16T15:51:34.831 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-16T15:51:34.951 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:34.953+0000 7f991e064640 1 -- 172.21.15.123:0/3026448755 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f99180cd5b0 msgr2=0x7f99180b3180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:51:34.952 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:34.953+0000 7f991e064640 1 --2- 172.21.15.123:0/3026448755 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f99180cd5b0 0x7f99180b3180 secure :-1 s=READY pgs=1807 cs=0 l=1 rev1=1 crypto rx=0x7f9900009900 tx=0x7f990001c7c0 comp rx=0 tx=0).stop 2024-09-16T15:51:34.952 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:34.953+0000 7f991e064640 1 -- 172.21.15.123:0/3026448755 shutdown_connections 2024-09-16T15:51:34.952 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:34.953+0000 7f991e064640 1 --2- 172.21.15.123:0/3026448755 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f99180cd5b0 0x7f99180b3180 unknown :-1 s=CLOSED pgs=1807 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:34.952 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:34.953+0000 7f991e064640 1 --2- 172.21.15.123:0/3026448755 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f991815ab20 0x7f99180cc650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:34.952 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:34.953+0000 7f991e064640 1 -- 172.21.15.123:0/3026448755 >> 172.21.15.123:0/3026448755 conn(0x7f9918074210 msgr2=0x7f99180745f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:51:34.952 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:34.953+0000 7f991e064640 1 -- 172.21.15.123:0/3026448755 shutdown_connections 2024-09-16T15:51:34.952 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:34.953+0000 7f991e064640 1 -- 172.21.15.123:0/3026448755 wait complete. 2024-09-16T15:51:34.953 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:34.953+0000 7f991e064640 1 Processor -- start 2024-09-16T15:51:34.953 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:34.953+0000 7f991e064640 1 -- start start 2024-09-16T15:51:34.954 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:34.953+0000 7f991e064640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f99180cd5b0 0x7f99180993f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:51:34.954 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:34.953+0000 7f991e064640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f99180cdf00 0x7f99181facc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:51:34.955 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:34.953+0000 7f991e064640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f991815ab20 0x7f991815bd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:51:34.955 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:34.953+0000 7f991e064640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f9918167c90 con 0x7f99180cdf00 2024-09-16T15:51:34.955 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:34.953+0000 7f991e064640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f9918167b10 con 0x7f991815ab20 2024-09-16T15:51:34.955 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:34.953+0000 7f991e064640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f9918167e10 con 0x7f99180cd5b0 2024-09-16T15:51:34.955 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:34.953+0000 7f990effd640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f99180cdf00 0x7f99181facc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:51:34.955 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:34.953+0000 7f990effd640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f99180cdf00 0x7f99181facc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:50178/0 (socket says 172.21.15.123:50178) 2024-09-16T15:51:34.955 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:34.953+0000 7f990effd640 1 -- 172.21.15.123:0/1157165175 learned_addr learned my addr 172.21.15.123:0/1157165175 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:51:34.955 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:34.953+0000 7f99177fe640 1 --2- 172.21.15.123:0/1157165175 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f99180cd5b0 0x7f99180993f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:51:34.955 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:34.953+0000 7f9917fff640 1 --2- 172.21.15.123:0/1157165175 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f991815ab20 0x7f991815bd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:51:34.955 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:34.953+0000 7f990effd640 1 -- 172.21.15.123:0/1157165175 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f99180cd5b0 msgr2=0x7f99180993f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:51:34.955 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:34.953+0000 7f990effd640 1 --2- 172.21.15.123:0/1157165175 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f99180cd5b0 0x7f99180993f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:34.955 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:34.953+0000 7f990effd640 1 -- 172.21.15.123:0/1157165175 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f991815ab20 msgr2=0x7f991815bd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:51:34.955 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:34.953+0000 7f990effd640 1 --2- 172.21.15.123:0/1157165175 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f991815ab20 0x7f991815bd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:34.956 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:34.953+0000 7f990effd640 1 -- 172.21.15.123:0/1157165175 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f99181fb200 con 0x7f99180cdf00 2024-09-16T15:51:34.956 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:34.953+0000 7f990effd640 1 --2- 172.21.15.123:0/1157165175 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f99180cdf00 0x7f99181facc0 secure :-1 s=READY pgs=1808 cs=0 l=1 rev1=1 crypto rx=0x7f990001a3f0 tx=0x7f9900005e90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:51:34.956 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:34.953+0000 7f9914ff9640 1 -- 172.21.15.123:0/1157165175 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99000ac070 con 0x7f99180cdf00 2024-09-16T15:51:34.956 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:34.953+0000 7f9914ff9640 1 -- 172.21.15.123:0/1157165175 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f9900004320 con 0x7f99180cdf00 2024-09-16T15:51:34.956 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:34.953+0000 7f9914ff9640 1 -- 172.21.15.123:0/1157165175 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99000052a0 con 0x7f99180cdf00 2024-09-16T15:51:34.956 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:34.953+0000 7f991e064640 1 -- 172.21.15.123:0/1157165175 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f991815cdd0 con 0x7f99180cdf00 2024-09-16T15:51:34.956 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:34.953+0000 7f991e064640 1 -- 172.21.15.123:0/1157165175 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9918099b00 con 0x7f99180cdf00 2024-09-16T15:51:34.957 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:34.957+0000 7f991e064640 1 -- 172.21.15.123:0/1157165175 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f99180b3180 con 0x7f99180cdf00 2024-09-16T15:51:34.957 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:34.957+0000 7f9914ff9640 1 -- 172.21.15.123:0/1157165175 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 25) ==== 97998+0+0 (secure 0 0 0) 0x7f99000af070 con 0x7f99180cdf00 2024-09-16T15:51:34.957 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:34.957+0000 7f9914ff9640 1 --2- 172.21.15.123:0/1157165175 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f98f80738e0 0x7f98f8075da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:51:34.957 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:34.957+0000 7f99177fe640 1 --2- 172.21.15.123:0/1157165175 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f98f80738e0 0x7f98f8075da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:51:34.957 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:34.957+0000 7f9914ff9640 1 -- 172.21.15.123:0/1157165175 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(157..157 src has 1..157) ==== 5737+0+0 (secure 0 0 0) 0x7f990012c960 con 0x7f99180cdf00 2024-09-16T15:51:34.957 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:34.957+0000 7f99177fe640 1 --2- 172.21.15.123:0/1157165175 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f98f80738e0 0x7f98f8075da0 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f99080058d0 tx=0x7f9908005860 comp rx=0 tx=0).ready entity=mgr.4150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:51:34.961 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:34.961+0000 7f9914ff9640 1 -- 172.21.15.123:0/1157165175 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f99000164d0 con 0x7f99180cdf00 2024-09-16T15:51:35.157 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:35.157+0000 7f991e064640 1 -- 172.21.15.123:0/1157165175 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f99180701d0 con 0x7f99180cdf00 2024-09-16T15:51:35.158 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:35.157+0000 7f9914ff9640 1 -- 172.21.15.123:0/1157165175 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 106 v106) ==== 95+0+5244 (secure 0 0 0) 0x7f99000f9580 con 0x7f99180cdf00 2024-09-16T15:51:35.158 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-09-16T15:51:35.158 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":106,"btime":"2024-09-16T15:51:29:191457+0000","default_fscid":12,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6200,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6835/2380355001","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6833","nonce":2380355001},{"type":"v1","addr":"172.21.15.184:6835","nonce":2380355001}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":100},{"gid":8844,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6834/1840463681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6832","nonce":1840463681},{"type":"v1","addr":"172.21.15.123:6834","nonce":1840463681}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":100},{"gid":8847,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6835/2493915107","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6833","nonce":2493915107},{"type":"v1","addr":"172.21.15.123:6835","nonce":2493915107}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":99}],"filesystems":[{"mdsmap":{"epoch":105,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-16T15:51:16.301945+0000","modified":"2024-09-16T15:51:19.374973+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8859},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8859":{"gid":8859,"name":"d","rank":0,"incarnation":101,"state":"up:active","state_seq":3,"addr":"172.21.15.184:6834/2856066034","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":2856066034},{"type":"v1","addr":"172.21.15.184:6834","nonce":2856066034}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[25],"metadata_pool":24,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8859,"qdb_cluster":[8859]},"id":12}]} 2024-09-16T15:51:35.159 INFO:teuthology.orchestra.run.smithi123.stderr:dumped fsmap epoch 106 2024-09-16T15:51:35.160 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:35.161+0000 7f991e064640 1 -- 172.21.15.123:0/1157165175 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f98f80738e0 msgr2=0x7f98f8075da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:51:35.160 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:35.161+0000 7f991e064640 1 --2- 172.21.15.123:0/1157165175 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f98f80738e0 0x7f98f8075da0 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f99080058d0 tx=0x7f9908005860 comp rx=0 tx=0).stop 2024-09-16T15:51:35.161 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:35.161+0000 7f991e064640 1 -- 172.21.15.123:0/1157165175 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f99180cdf00 msgr2=0x7f99181facc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:51:35.161 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:35.161+0000 7f991e064640 1 --2- 172.21.15.123:0/1157165175 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f99180cdf00 0x7f99181facc0 secure :-1 s=READY pgs=1808 cs=0 l=1 rev1=1 crypto rx=0x7f990001a3f0 tx=0x7f9900005e90 comp rx=0 tx=0).stop 2024-09-16T15:51:35.161 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:35.161+0000 7f991e064640 1 -- 172.21.15.123:0/1157165175 shutdown_connections 2024-09-16T15:51:35.161 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:35.161+0000 7f991e064640 1 --2- 172.21.15.123:0/1157165175 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f98f80738e0 0x7f98f8075da0 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:35.161 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:35.161+0000 7f991e064640 1 --2- 172.21.15.123:0/1157165175 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f991815ab20 0x7f991815bd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:35.161 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:35.161+0000 7f991e064640 1 --2- 172.21.15.123:0/1157165175 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f99180cdf00 0x7f99181facc0 unknown :-1 s=CLOSED pgs=1808 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:35.161 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:35.161+0000 7f991e064640 1 --2- 172.21.15.123:0/1157165175 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f99180cd5b0 0x7f99180993f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:35.161 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:35.161+0000 7f991e064640 1 -- 172.21.15.123:0/1157165175 >> 172.21.15.123:0/1157165175 conn(0x7f9918074210 msgr2=0x7f9918159620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:51:35.161 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:35.161+0000 7f991e064640 1 -- 172.21.15.123:0/1157165175 shutdown_connections 2024-09-16T15:51:35.161 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:35.161+0000 7f991e064640 1 -- 172.21.15.123:0/1157165175 wait complete. 2024-09-16T15:51:35.176 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-16T15:51:35.304 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:35.305+0000 7f320a211640 1 -- 172.21.15.123:0/4142054760 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f320415add0 msgr2=0x7f32040cf9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:51:35.304 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:35.305+0000 7f320a211640 1 --2- 172.21.15.123:0/4142054760 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f320415add0 0x7f32040cf9d0 secure :-1 s=READY pgs=1809 cs=0 l=1 rev1=1 crypto rx=0x7f32040cd520 tx=0x7f31fc01c8c0 comp rx=0 tx=0).stop 2024-09-16T15:51:35.304 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:35.305+0000 7f320a211640 1 -- 172.21.15.123:0/4142054760 shutdown_connections 2024-09-16T15:51:35.304 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:35.305+0000 7f320a211640 1 --2- 172.21.15.123:0/4142054760 >> v2:172.21.15.184:3300/0 conn(0x7f3204073d70 0x7f320406fa00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:35.304 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:35.305+0000 7f320a211640 1 --2- 172.21.15.123:0/4142054760 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f320415add0 0x7f32040cf9d0 unknown :-1 s=CLOSED pgs=1809 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:35.304 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:35.305+0000 7f320a211640 1 -- 172.21.15.123:0/4142054760 >> 172.21.15.123:0/4142054760 conn(0x7f32040d0b40 msgr2=0x7f32040d0f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:51:35.304 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:35.305+0000 7f320a211640 1 -- 172.21.15.123:0/4142054760 shutdown_connections 2024-09-16T15:51:35.304 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:35.305+0000 7f320a211640 1 -- 172.21.15.123:0/4142054760 wait complete. 2024-09-16T15:51:35.307 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:35.305+0000 7f320a211640 1 Processor -- start 2024-09-16T15:51:35.307 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:35.305+0000 7f320a211640 1 -- start start 2024-09-16T15:51:35.307 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:35.305+0000 7f320a211640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f3204073d70 0x7f32041f72d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:51:35.308 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:35.305+0000 7f320a211640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f32040cff10 0x7f32040b4b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:51:35.308 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:35.305+0000 7f320a211640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f320415add0 0x7f32040b5070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:51:35.308 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:35.305+0000 7f320a211640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f32041679d0 con 0x7f320415add0 2024-09-16T15:51:35.308 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:35.305+0000 7f320a211640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f3204167850 con 0x7f32040cff10 2024-09-16T15:51:35.308 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:35.305+0000 7f320a211640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f3204167b50 con 0x7f3204073d70 2024-09-16T15:51:35.308 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:35.305+0000 7f3203fff640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f320415add0 0x7f32040b5070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:51:35.308 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:35.305+0000 7f3203fff640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f320415add0 0x7f32040b5070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:50190/0 (socket says 172.21.15.123:50190) 2024-09-16T15:51:35.308 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:35.305+0000 7f3203fff640 1 -- 172.21.15.123:0/311795720 learned_addr learned my addr 172.21.15.123:0/311795720 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:51:35.309 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:35.305+0000 7f3202ffd640 1 --2- 172.21.15.123:0/311795720 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f32040cff10 0x7f32040b4b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:51:35.309 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:35.305+0000 7f32037fe640 1 --2- 172.21.15.123:0/311795720 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f3204073d70 0x7f32041f72d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:51:35.309 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:35.305+0000 7f3203fff640 1 -- 172.21.15.123:0/311795720 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f3204073d70 msgr2=0x7f32041f72d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:51:35.309 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:35.305+0000 7f3203fff640 1 --2- 172.21.15.123:0/311795720 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f3204073d70 0x7f32041f72d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:35.309 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:35.305+0000 7f3203fff640 1 -- 172.21.15.123:0/311795720 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f32040cff10 msgr2=0x7f32040b4b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:51:35.309 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:35.305+0000 7f3203fff640 1 --2- 172.21.15.123:0/311795720 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f32040cff10 0x7f32040b4b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:35.309 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:35.305+0000 7f3203fff640 1 -- 172.21.15.123:0/311795720 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f32040d1ac0 con 0x7f320415add0 2024-09-16T15:51:35.309 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:35.305+0000 7f32037fe640 1 --2- 172.21.15.123:0/311795720 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f3204073d70 0x7f32041f72d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T15:51:35.309 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:35.305+0000 7f3203fff640 1 --2- 172.21.15.123:0/311795720 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f320415add0 0x7f32040b5070 secure :-1 s=READY pgs=1810 cs=0 l=1 rev1=1 crypto rx=0x7f31f400cce0 tx=0x7f31f400a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:51:35.309 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:35.305+0000 7f31ebfff640 1 -- 172.21.15.123:0/311795720 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31f400f450 con 0x7f320415add0 2024-09-16T15:51:35.310 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:35.305+0000 7f31ebfff640 1 -- 172.21.15.123:0/311795720 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f31f400f700 con 0x7f320415add0 2024-09-16T15:51:35.310 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:35.305+0000 7f31ebfff640 1 -- 172.21.15.123:0/311795720 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31f400fb50 con 0x7f320415add0 2024-09-16T15:51:35.310 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:35.305+0000 7f320a211640 1 -- 172.21.15.123:0/311795720 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f32041fb0a0 con 0x7f320415add0 2024-09-16T15:51:35.310 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:35.305+0000 7f320a211640 1 -- 172.21.15.123:0/311795720 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f32041fb5f0 con 0x7f320415add0 2024-09-16T15:51:35.310 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:35.309+0000 7f320a211640 1 -- 172.21.15.123:0/311795720 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f320406fa00 con 0x7f320415add0 2024-09-16T15:51:35.311 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:35.309+0000 7f31ebfff640 1 -- 172.21.15.123:0/311795720 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 25) ==== 97998+0+0 (secure 0 0 0) 0x7f31f400fcf0 con 0x7f320415add0 2024-09-16T15:51:35.311 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:35.309+0000 7f31ebfff640 1 --2- 172.21.15.123:0/311795720 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f31f00738e0 0x7f31f0075da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:51:35.311 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:35.309+0000 7f31ebfff640 1 -- 172.21.15.123:0/311795720 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(158..158 src has 1..158) ==== 5795+0+0 (secure 0 0 0) 0x7f31f40935a0 con 0x7f320415add0 2024-09-16T15:51:35.311 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:35.309+0000 7f32037fe640 1 --2- 172.21.15.123:0/311795720 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f31f00738e0 0x7f31f0075da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:51:35.311 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:35.309+0000 7f32037fe640 1 --2- 172.21.15.123:0/311795720 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f31f00738e0 0x7f31f0075da0 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f31fc0af000 tx=0x7f31fc01b100 comp rx=0 tx=0).ready entity=mgr.4150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:51:35.312 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:35.313+0000 7f31ebfff640 1 -- 172.21.15.123:0/311795720 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f31f4014030 con 0x7f320415add0 2024-09-16T15:51:35.521 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:35.521+0000 7f320a211640 1 -- 172.21.15.123:0/311795720 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f32040b08a0 con 0x7f320415add0 2024-09-16T15:51:35.522 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:35.521+0000 7f31ebfff640 1 -- 172.21.15.123:0/311795720 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 106 v106) ==== 95+0+5244 (secure 0 0 0) 0x7f31f4060080 con 0x7f320415add0 2024-09-16T15:51:35.522 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-09-16T15:51:35.522 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":106,"btime":"2024-09-16T15:51:29:191457+0000","default_fscid":12,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6200,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6835/2380355001","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6833","nonce":2380355001},{"type":"v1","addr":"172.21.15.184:6835","nonce":2380355001}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":100},{"gid":8844,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6834/1840463681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6832","nonce":1840463681},{"type":"v1","addr":"172.21.15.123:6834","nonce":1840463681}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":100},{"gid":8847,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6835/2493915107","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6833","nonce":2493915107},{"type":"v1","addr":"172.21.15.123:6835","nonce":2493915107}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":99}],"filesystems":[{"mdsmap":{"epoch":105,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-16T15:51:16.301945+0000","modified":"2024-09-16T15:51:19.374973+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8859},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8859":{"gid":8859,"name":"d","rank":0,"incarnation":101,"state":"up:active","state_seq":3,"addr":"172.21.15.184:6834/2856066034","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":2856066034},{"type":"v1","addr":"172.21.15.184:6834","nonce":2856066034}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[25],"metadata_pool":24,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8859,"qdb_cluster":[8859]},"id":12}]} 2024-09-16T15:51:35.522 INFO:teuthology.orchestra.run.smithi123.stderr:dumped fsmap epoch 106 2024-09-16T15:51:35.524 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:35.525+0000 7f320a211640 1 -- 172.21.15.123:0/311795720 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f31f00738e0 msgr2=0x7f31f0075da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:51:35.524 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:35.525+0000 7f320a211640 1 --2- 172.21.15.123:0/311795720 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f31f00738e0 0x7f31f0075da0 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f31fc0af000 tx=0x7f31fc01b100 comp rx=0 tx=0).stop 2024-09-16T15:51:35.524 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:35.525+0000 7f320a211640 1 -- 172.21.15.123:0/311795720 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f320415add0 msgr2=0x7f32040b5070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:51:35.524 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:35.525+0000 7f320a211640 1 --2- 172.21.15.123:0/311795720 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f320415add0 0x7f32040b5070 secure :-1 s=READY pgs=1810 cs=0 l=1 rev1=1 crypto rx=0x7f31f400cce0 tx=0x7f31f400a620 comp rx=0 tx=0).stop 2024-09-16T15:51:35.525 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:35.525+0000 7f320a211640 1 -- 172.21.15.123:0/311795720 shutdown_connections 2024-09-16T15:51:35.525 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:35.525+0000 7f320a211640 1 --2- 172.21.15.123:0/311795720 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f31f00738e0 0x7f31f0075da0 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:35.525 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:35.525+0000 7f320a211640 1 --2- 172.21.15.123:0/311795720 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f320415add0 0x7f32040b5070 unknown :-1 s=CLOSED pgs=1810 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:35.525 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:35.525+0000 7f320a211640 1 --2- 172.21.15.123:0/311795720 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f32040cff10 0x7f32040b4b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:35.525 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:35.525+0000 7f320a211640 1 --2- 172.21.15.123:0/311795720 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f3204073d70 0x7f32041f72d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:35.525 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:35.525+0000 7f320a211640 1 -- 172.21.15.123:0/311795720 >> 172.21.15.123:0/311795720 conn(0x7f32040d0b40 msgr2=0x7f32041579f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:51:35.525 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:35.525+0000 7f320a211640 1 -- 172.21.15.123:0/311795720 shutdown_connections 2024-09-16T15:51:35.525 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:35.525+0000 7f320a211640 1 -- 172.21.15.123:0/311795720 wait complete. 2024-09-16T15:51:35.541 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-16T15:51:35.663 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:35.665+0000 7f3b99029640 1 -- 172.21.15.123:0/3170666548 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3b940b11a0 msgr2=0x7f3b940d1b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:51:35.663 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:35.665+0000 7f3b99029640 1 --2- 172.21.15.123:0/3170666548 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3b940b11a0 0x7f3b940d1b90 secure :-1 s=READY pgs=1812 cs=0 l=1 rev1=1 crypto rx=0x7f3b94072ef0 tx=0x7f3b7c01c890 comp rx=0 tx=0).stop 2024-09-16T15:51:35.663 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:35.665+0000 7f3b99029640 1 -- 172.21.15.123:0/3170666548 shutdown_connections 2024-09-16T15:51:35.664 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:35.665+0000 7f3b99029640 1 --2- 172.21.15.123:0/3170666548 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3b940b11a0 0x7f3b940d1b90 unknown :-1 s=CLOSED pgs=1812 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:35.664 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:35.665+0000 7f3b99029640 1 --2- 172.21.15.123:0/3170666548 >> v2:172.21.15.184:3300/0 conn(0x7f3b9415bbc0 0x7f3b940b2d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:35.664 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:35.665+0000 7f3b99029640 1 -- 172.21.15.123:0/3170666548 >> 172.21.15.123:0/3170666548 conn(0x7f3b9409aa40 msgr2=0x7f3b9409ae50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:51:35.664 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:35.665+0000 7f3b99029640 1 -- 172.21.15.123:0/3170666548 shutdown_connections 2024-09-16T15:51:35.664 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:35.665+0000 7f3b99029640 1 -- 172.21.15.123:0/3170666548 wait complete. 2024-09-16T15:51:35.664 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:35.665+0000 7f3b99029640 1 Processor -- start 2024-09-16T15:51:35.665 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:35.665+0000 7f3b99029640 1 -- start start 2024-09-16T15:51:35.667 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:35.665+0000 7f3b99029640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3b940b32c0 0x7f3b941529a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:51:35.667 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:35.665+0000 7f3b99029640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3b9415bbc0 0x7f3b940c7070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:51:35.667 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:35.665+0000 7f3b99029640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f3b940c75b0 0x7f3b94154aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:51:35.667 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:35.665+0000 7f3b99029640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f3b941693f0 con 0x7f3b9415bbc0 2024-09-16T15:51:35.667 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:35.665+0000 7f3b99029640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f3b94169270 con 0x7f3b940b32c0 2024-09-16T15:51:35.667 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:35.665+0000 7f3b99029640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f3b94169570 con 0x7f3b940c75b0 2024-09-16T15:51:35.667 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:35.665+0000 7f3b9259c640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3b9415bbc0 0x7f3b940c7070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:51:35.667 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:35.665+0000 7f3b9259c640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3b9415bbc0 0x7f3b940c7070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:50204/0 (socket says 172.21.15.123:50204) 2024-09-16T15:51:35.667 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:35.665+0000 7f3b9259c640 1 -- 172.21.15.123:0/2760196243 learned_addr learned my addr 172.21.15.123:0/2760196243 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:51:35.668 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:35.665+0000 7f3b9259c640 1 -- 172.21.15.123:0/2760196243 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f3b940c75b0 msgr2=0x7f3b94154aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:51:35.668 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:35.665+0000 7f3b9259c640 1 --2- 172.21.15.123:0/2760196243 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f3b940c75b0 0x7f3b94154aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:35.668 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:35.665+0000 7f3b9259c640 1 -- 172.21.15.123:0/2760196243 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3b940b32c0 msgr2=0x7f3b941529a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:51:35.668 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:35.665+0000 7f3b9259c640 1 --2- 172.21.15.123:0/2760196243 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3b940b32c0 0x7f3b941529a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:35.668 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:35.665+0000 7f3b9259c640 1 -- 172.21.15.123:0/2760196243 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3b941f5bb0 con 0x7f3b9415bbc0 2024-09-16T15:51:35.668 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:35.665+0000 7f3b9259c640 1 --2- 172.21.15.123:0/2760196243 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3b9415bbc0 0x7f3b940c7070 secure :-1 s=READY pgs=1813 cs=0 l=1 rev1=1 crypto rx=0x7f3b7c0194a0 tx=0x7f3b7c0a5770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:51:35.668 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:35.665+0000 7f3b8b7fe640 1 -- 172.21.15.123:0/2760196243 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b7c0bc070 con 0x7f3b9415bbc0 2024-09-16T15:51:35.668 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:35.665+0000 7f3b8b7fe640 1 -- 172.21.15.123:0/2760196243 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f3b7c0053e0 con 0x7f3b9415bbc0 2024-09-16T15:51:35.668 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:35.665+0000 7f3b99029640 1 -- 172.21.15.123:0/2760196243 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3b940c7a30 con 0x7f3b9415bbc0 2024-09-16T15:51:35.668 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:35.665+0000 7f3b8b7fe640 1 -- 172.21.15.123:0/2760196243 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b7c0b2040 con 0x7f3b9415bbc0 2024-09-16T15:51:35.668 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:35.665+0000 7f3b99029640 1 -- 172.21.15.123:0/2760196243 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3b941f6150 con 0x7f3b9415bbc0 2024-09-16T15:51:35.669 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:35.669+0000 7f3b99029640 1 -- 172.21.15.123:0/2760196243 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3b940d1b90 con 0x7f3b9415bbc0 2024-09-16T15:51:35.669 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:35.669+0000 7f3b8b7fe640 1 -- 172.21.15.123:0/2760196243 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 25) ==== 97998+0+0 (secure 0 0 0) 0x7f3b7c0046d0 con 0x7f3b9415bbc0 2024-09-16T15:51:35.669 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:35.669+0000 7f3b8b7fe640 1 --2- 172.21.15.123:0/2760196243 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f3b74073910 0x7f3b74075dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:51:35.669 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:35.669+0000 7f3b92d9d640 1 --2- 172.21.15.123:0/2760196243 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f3b74073910 0x7f3b74075dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:51:35.670 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:35.669+0000 7f3b8b7fe640 1 -- 172.21.15.123:0/2760196243 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(158..158 src has 1..158) ==== 5795+0+0 (secure 0 0 0) 0x7f3b7c12c5e0 con 0x7f3b9415bbc0 2024-09-16T15:51:35.670 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:35.669+0000 7f3b92d9d640 1 --2- 172.21.15.123:0/2760196243 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f3b74073910 0x7f3b74075dd0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f3b8c0058d0 tx=0x7f3b8c005860 comp rx=0 tx=0).ready entity=mgr.4150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:51:35.672 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:35.673+0000 7f3b8b7fe640 1 -- 172.21.15.123:0/2760196243 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f3b7c0164b0 con 0x7f3b9415bbc0 2024-09-16T15:51:35.840 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:35.841+0000 7f3b99029640 1 -- 172.21.15.123:0/2760196243 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f3b940d2ad0 con 0x7f3b9415bbc0 2024-09-16T15:51:35.842 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:35.841+0000 7f3b8b7fe640 1 -- 172.21.15.123:0/2760196243 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v158) ==== 74+0+16885 (secure 0 0 0) 0x7f3b7c0f9060 con 0x7f3b9415bbc0 2024-09-16T15:51:35.842 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-09-16T15:51:35.842 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":158,"fsid":"56077f89-18c8-486f-ae7e-eecc0bb7822c","created":"2024-09-16T15:42:26.364892+0000","modified":"2024-09-16T15:51:35.233597+0000","last_up_change":"2024-09-16T15:42:38.707234+0000","last_in_change":"2024-09-16T15:42:29.423662+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":25,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-16T15:42:39.287764+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":24,"pool_name":"cephfs_metadata","create_time":"2024-09-16T15:51:13.953990+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"156","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":25,"pool_name":"cephfs_data","create_time":"2024-09-16T15:51:15.281388+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"157","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9e7c1955-a607-4671-9be7-f2ef760ceeaa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":155,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6800","nonce":3882044969},{"type":"v1","addr":"172.21.15.123:6801","nonce":3882044969}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6802","nonce":3882044969},{"type":"v1","addr":"172.21.15.123:6803","nonce":3882044969}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6806","nonce":3882044969},{"type":"v1","addr":"172.21.15.123:6807","nonce":3882044969}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6804","nonce":3882044969},{"type":"v1","addr":"172.21.15.123:6805","nonce":3882044969}]},"public_addr":"172.21.15.123:6801/3882044969","cluster_addr":"172.21.15.123:6803/3882044969","heartbeat_back_addr":"172.21.15.123:6807/3882044969","heartbeat_front_addr":"172.21.15.123:6805/3882044969","state":["exists","up"]},{"osd":1,"uuid":"c540cfde-0977-4742-9b17-fd634668e010","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":155,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6816","nonce":4037001850},{"type":"v1","addr":"172.21.15.123:6817","nonce":4037001850}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6818","nonce":4037001850},{"type":"v1","addr":"172.21.15.123:6819","nonce":4037001850}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6822","nonce":4037001850},{"type":"v1","addr":"172.21.15.123:6823","nonce":4037001850}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6820","nonce":4037001850},{"type":"v1","addr":"172.21.15.123:6821","nonce":4037001850}]},"public_addr":"172.21.15.123:6817/4037001850","cluster_addr":"172.21.15.123:6819/4037001850","heartbeat_back_addr":"172.21.15.123:6823/4037001850","heartbeat_front_addr":"172.21.15.123:6821/4037001850","state":["exists","up"]},{"osd":2,"uuid":"9864a496-f169-486b-95c4-1d59162793a7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":155,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6824","nonce":1332143088},{"type":"v1","addr":"172.21.15.123:6825","nonce":1332143088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6826","nonce":1332143088},{"type":"v1","addr":"172.21.15.123:6827","nonce":1332143088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6830","nonce":1332143088},{"type":"v1","addr":"172.21.15.123:6831","nonce":1332143088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6828","nonce":1332143088},{"type":"v1","addr":"172.21.15.123:6829","nonce":1332143088}]},"public_addr":"172.21.15.123:6825/1332143088","cluster_addr":"172.21.15.123:6827/1332143088","heartbeat_back_addr":"172.21.15.123:6831/1332143088","heartbeat_front_addr":"172.21.15.123:6829/1332143088","state":["exists","up"]},{"osd":3,"uuid":"c6d1f817-f074-4325-95e1-578eba13edda","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":155,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6808","nonce":988586998},{"type":"v1","addr":"172.21.15.123:6809","nonce":988586998}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6810","nonce":988586998},{"type":"v1","addr":"172.21.15.123:6811","nonce":988586998}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6814","nonce":988586998},{"type":"v1","addr":"172.21.15.123:6815","nonce":988586998}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6812","nonce":988586998},{"type":"v1","addr":"172.21.15.123:6813","nonce":988586998}]},"public_addr":"172.21.15.123:6809/988586998","cluster_addr":"172.21.15.123:6811/988586998","heartbeat_back_addr":"172.21.15.123:6815/988586998","heartbeat_front_addr":"172.21.15.123:6813/988586998","state":["exists","up"]},{"osd":4,"uuid":"200a8df4-50ef-40a5-b48a-58981269c086","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":155,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6816","nonce":3387813890},{"type":"v1","addr":"172.21.15.184:6817","nonce":3387813890}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6818","nonce":3387813890},{"type":"v1","addr":"172.21.15.184:6819","nonce":3387813890}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6822","nonce":3387813890},{"type":"v1","addr":"172.21.15.184:6823","nonce":3387813890}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6820","nonce":3387813890},{"type":"v1","addr":"172.21.15.184:6821","nonce":3387813890}]},"public_addr":"172.21.15.184:6817/3387813890","cluster_addr":"172.21.15.184:6819/3387813890","heartbeat_back_addr":"172.21.15.184:6823/3387813890","heartbeat_front_addr":"172.21.15.184:6821/3387813890","state":["exists","up"]},{"osd":5,"uuid":"32d5a900-da3d-412c-bc19-13861a2368d8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":155,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6808","nonce":2680916017},{"type":"v1","addr":"172.21.15.184:6809","nonce":2680916017}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6810","nonce":2680916017},{"type":"v1","addr":"172.21.15.184:6811","nonce":2680916017}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6814","nonce":2680916017},{"type":"v1","addr":"172.21.15.184:6815","nonce":2680916017}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6812","nonce":2680916017},{"type":"v1","addr":"172.21.15.184:6813","nonce":2680916017}]},"public_addr":"172.21.15.184:6809/2680916017","cluster_addr":"172.21.15.184:6811/2680916017","heartbeat_back_addr":"172.21.15.184:6815/2680916017","heartbeat_front_addr":"172.21.15.184:6813/2680916017","state":["exists","up"]},{"osd":6,"uuid":"cde6f39e-aa1b-4260-ac19-774ee6aaca72","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":155,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6824","nonce":2183120457},{"type":"v1","addr":"172.21.15.184:6825","nonce":2183120457}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6826","nonce":2183120457},{"type":"v1","addr":"172.21.15.184:6827","nonce":2183120457}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6830","nonce":2183120457},{"type":"v1","addr":"172.21.15.184:6831","nonce":2183120457}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6828","nonce":2183120457},{"type":"v1","addr":"172.21.15.184:6829","nonce":2183120457}]},"public_addr":"172.21.15.184:6825/2183120457","cluster_addr":"172.21.15.184:6827/2183120457","heartbeat_back_addr":"172.21.15.184:6831/2183120457","heartbeat_front_addr":"172.21.15.184:6829/2183120457","state":["exists","up"]},{"osd":7,"uuid":"4f381778-eee1-47c6-804f-df0e6b33e313","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":155,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6800","nonce":4102957285},{"type":"v1","addr":"172.21.15.184:6801","nonce":4102957285}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6802","nonce":4102957285},{"type":"v1","addr":"172.21.15.184:6803","nonce":4102957285}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6806","nonce":4102957285},{"type":"v1","addr":"172.21.15.184:6807","nonce":4102957285}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6804","nonce":4102957285},{"type":"v1","addr":"172.21.15.184:6805","nonce":4102957285}]},"public_addr":"172.21.15.184:6801/4102957285","cluster_addr":"172.21.15.184:6803/4102957285","heartbeat_back_addr":"172.21.15.184:6807/4102957285","heartbeat_front_addr":"172.21.15.184:6805/4102957285","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:35.972210+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.222007+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.198617+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.385663+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.358935+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.156066+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.752973+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:35.682846+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"25.12","mappings":[{"from":0,"to":2}]},{"pgid":"25.18","mappings":[{"from":0,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":17,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T15:51:35.844 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:35.845+0000 7f3b99029640 1 -- 172.21.15.123:0/2760196243 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f3b74073910 msgr2=0x7f3b74075dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:51:35.845 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:35.845+0000 7f3b99029640 1 --2- 172.21.15.123:0/2760196243 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f3b74073910 0x7f3b74075dd0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f3b8c0058d0 tx=0x7f3b8c005860 comp rx=0 tx=0).stop 2024-09-16T15:51:35.845 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:35.845+0000 7f3b99029640 1 -- 172.21.15.123:0/2760196243 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3b9415bbc0 msgr2=0x7f3b940c7070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:51:35.845 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:35.845+0000 7f3b99029640 1 --2- 172.21.15.123:0/2760196243 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3b9415bbc0 0x7f3b940c7070 secure :-1 s=READY pgs=1813 cs=0 l=1 rev1=1 crypto rx=0x7f3b7c0194a0 tx=0x7f3b7c0a5770 comp rx=0 tx=0).stop 2024-09-16T15:51:35.845 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:35.845+0000 7f3b99029640 1 -- 172.21.15.123:0/2760196243 shutdown_connections 2024-09-16T15:51:35.845 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:35.845+0000 7f3b99029640 1 --2- 172.21.15.123:0/2760196243 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f3b74073910 0x7f3b74075dd0 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:35.845 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:35.845+0000 7f3b99029640 1 --2- 172.21.15.123:0/2760196243 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f3b940c75b0 0x7f3b94154aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:35.845 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:35.845+0000 7f3b99029640 1 --2- 172.21.15.123:0/2760196243 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3b9415bbc0 0x7f3b940c7070 unknown :-1 s=CLOSED pgs=1813 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:35.845 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:35.845+0000 7f3b99029640 1 --2- 172.21.15.123:0/2760196243 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3b940b32c0 0x7f3b941529a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:35.845 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:35.845+0000 7f3b99029640 1 -- 172.21.15.123:0/2760196243 >> 172.21.15.123:0/2760196243 conn(0x7f3b9409aa40 msgr2=0x7f3b940b70e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:51:35.845 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:35.845+0000 7f3b99029640 1 -- 172.21.15.123:0/2760196243 shutdown_connections 2024-09-16T15:51:35.845 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:35.845+0000 7f3b99029640 1 -- 172.21.15.123:0/2760196243 wait complete. 2024-09-16T15:51:35.860 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-09-16T15:51:35.985 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:35.985+0000 7f02fe0df640 1 -- 172.21.15.123:0/1109863263 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f02e00052f0 msgr2=0x7f02e000b9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:51:35.985 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:35.985+0000 7f02fe0df640 1 --2- 172.21.15.123:0/1109863263 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f02e00052f0 0x7f02e000b9d0 secure :-1 s=READY pgs=1815 cs=0 l=1 rev1=1 crypto rx=0x7f02f000c590 tx=0x7f02f001f9e0 comp rx=0 tx=0).stop 2024-09-16T15:51:35.985 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:35.985+0000 7f02fe0df640 1 -- 172.21.15.123:0/1109863263 shutdown_connections 2024-09-16T15:51:35.985 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:35.985+0000 7f02fe0df640 1 --2- 172.21.15.123:0/1109863263 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f02e00052f0 0x7f02e000b9d0 unknown :-1 s=CLOSED pgs=1815 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:35.985 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:35.985+0000 7f02fe0df640 1 -- 172.21.15.123:0/1109863263 >> 172.21.15.123:0/1109863263 conn(0x7f02f80cfb10 msgr2=0x7f02f80cff20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:51:35.985 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:35.985+0000 7f02fe0df640 1 -- 172.21.15.123:0/1109863263 shutdown_connections 2024-09-16T15:51:35.985 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:35.985+0000 7f02fe0df640 1 -- 172.21.15.123:0/1109863263 wait complete. 2024-09-16T15:51:35.986 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:35.985+0000 7f02fe0df640 1 Processor -- start 2024-09-16T15:51:35.986 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:35.985+0000 7f02fe0df640 1 -- start start 2024-09-16T15:51:35.988 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:35.985+0000 7f02fe0df640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f02e00052f0 0x7f02f81590a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:51:35.988 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:35.985+0000 7f02fe0df640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f02f80b4e10 0x7f02f80b5290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:51:35.988 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:35.989+0000 7f02fe0df640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f02f80ad9a0 0x7f02f80ade10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:51:35.988 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:35.989+0000 7f02fe0df640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f02f81584b0 con 0x7f02f80ad9a0 2024-09-16T15:51:35.989 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:35.989+0000 7f02fe0df640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f02f8158330 con 0x7f02f80b4e10 2024-09-16T15:51:35.989 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:35.989+0000 7f02fe0df640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f02f80ca2f0 con 0x7f02e00052f0 2024-09-16T15:51:35.989 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:35.989+0000 7f02f7fff640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f02f80ad9a0 0x7f02f80ade10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:51:35.989 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:35.989+0000 7f02f7fff640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f02f80ad9a0 0x7f02f80ade10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:50216/0 (socket says 172.21.15.123:50216) 2024-09-16T15:51:35.989 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:35.989+0000 7f02f7fff640 1 -- 172.21.15.123:0/2404939786 learned_addr learned my addr 172.21.15.123:0/2404939786 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:51:35.989 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:35.989+0000 7f02f77fe640 1 --2- 172.21.15.123:0/2404939786 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f02e00052f0 0x7f02f81590a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:51:35.989 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:35.989+0000 7f02f6ffd640 1 --2- 172.21.15.123:0/2404939786 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f02f80b4e10 0x7f02f80b5290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:51:35.989 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:35.989+0000 7f02f7fff640 1 -- 172.21.15.123:0/2404939786 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f02e00052f0 msgr2=0x7f02f81590a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:51:35.989 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:35.989+0000 7f02f7fff640 1 --2- 172.21.15.123:0/2404939786 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f02e00052f0 0x7f02f81590a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:35.989 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:35.989+0000 7f02f7fff640 1 -- 172.21.15.123:0/2404939786 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f02f80b4e10 msgr2=0x7f02f80b5290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:51:35.989 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:35.989+0000 7f02f7fff640 1 --2- 172.21.15.123:0/2404939786 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f02f80b4e10 0x7f02f80b5290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:35.989 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:35.989+0000 7f02f7fff640 1 -- 172.21.15.123:0/2404939786 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f02f80b57d0 con 0x7f02f80ad9a0 2024-09-16T15:51:35.990 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:35.989+0000 7f02f7fff640 1 --2- 172.21.15.123:0/2404939786 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f02f80ad9a0 0x7f02f80ade10 secure :-1 s=READY pgs=1816 cs=0 l=1 rev1=1 crypto rx=0x7f02f0006040 tx=0x7f02f0001d40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:51:35.990 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:35.989+0000 7f02dffff640 1 -- 172.21.15.123:0/2404939786 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02f0003b80 con 0x7f02f80ad9a0 2024-09-16T15:51:35.990 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:35.989+0000 7f02dffff640 1 -- 172.21.15.123:0/2404939786 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f02f0003e30 con 0x7f02f80ad9a0 2024-09-16T15:51:35.990 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:35.989+0000 7f02fe0df640 1 -- 172.21.15.123:0/2404939786 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f02f80ae6d0 con 0x7f02f80ad9a0 2024-09-16T15:51:35.990 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:35.989+0000 7f02dffff640 1 -- 172.21.15.123:0/2404939786 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02f0006330 con 0x7f02f80ad9a0 2024-09-16T15:51:35.990 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:35.989+0000 7f02fe0df640 1 -- 172.21.15.123:0/2404939786 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f02f80aec90 con 0x7f02f80ad9a0 2024-09-16T15:51:35.990 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:35.989+0000 7f02fe0df640 1 -- 172.21.15.123:0/2404939786 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f02f80ca850 con 0x7f02f80ad9a0 2024-09-16T15:51:35.991 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:35.989+0000 7f02dffff640 1 -- 172.21.15.123:0/2404939786 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 25) ==== 97998+0+0 (secure 0 0 0) 0x7f02f00064d0 con 0x7f02f80ad9a0 2024-09-16T15:51:35.991 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:35.989+0000 7f02dffff640 1 --2- 172.21.15.123:0/2404939786 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f02e40738e0 0x7f02e4075da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:51:35.991 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:35.989+0000 7f02f77fe640 1 --2- 172.21.15.123:0/2404939786 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f02e40738e0 0x7f02e4075da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:51:35.991 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:35.989+0000 7f02dffff640 1 -- 172.21.15.123:0/2404939786 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(158..158 src has 1..158) ==== 5795+0+0 (secure 0 0 0) 0x7f02f012fc30 con 0x7f02f80ad9a0 2024-09-16T15:51:35.991 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:35.989+0000 7f02f77fe640 1 --2- 172.21.15.123:0/2404939786 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f02e40738e0 0x7f02e4075da0 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f02e8003890 tx=0x7f02e8006c40 comp rx=0 tx=0).ready entity=mgr.4150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:51:35.994 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:35.993+0000 7f02dffff640 1 -- 172.21.15.123:0/2404939786 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f02f0019790 con 0x7f02f80ad9a0 2024-09-16T15:51:36.205 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:36.205+0000 7f02fe0df640 1 -- 172.21.15.123:0/2404939786 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7f02f80d06c0 con 0x7f02f80ad9a0 2024-09-16T15:51:36.205 INFO:tasks.ceph.mon.a.smithi123.stderr:2024-09-16T15:51:36.205+0000 7f268d076640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-09-16T15:51:36.208 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:36.209+0000 7f02dffff640 1 -- 172.21.15.123:0/2404939786 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v107) ==== 161+0+0 (secure 0 0 0) 0x7f02f00fca40 con 0x7f02f80ad9a0 2024-09-16T15:51:36.208 INFO:teuthology.orchestra.run.smithi123.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-09-16T15:51:36.212 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:36.213+0000 7f02fe0df640 1 -- 172.21.15.123:0/2404939786 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f02e40738e0 msgr2=0x7f02e4075da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:51:36.212 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:36.213+0000 7f02fe0df640 1 --2- 172.21.15.123:0/2404939786 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f02e40738e0 0x7f02e4075da0 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f02e8003890 tx=0x7f02e8006c40 comp rx=0 tx=0).stop 2024-09-16T15:51:36.212 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:36.213+0000 7f02fe0df640 1 -- 172.21.15.123:0/2404939786 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f02f80ad9a0 msgr2=0x7f02f80ade10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:51:36.212 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:36.213+0000 7f02fe0df640 1 --2- 172.21.15.123:0/2404939786 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f02f80ad9a0 0x7f02f80ade10 secure :-1 s=READY pgs=1816 cs=0 l=1 rev1=1 crypto rx=0x7f02f0006040 tx=0x7f02f0001d40 comp rx=0 tx=0).stop 2024-09-16T15:51:36.212 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:36.213+0000 7f02fe0df640 1 -- 172.21.15.123:0/2404939786 shutdown_connections 2024-09-16T15:51:36.212 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:36.213+0000 7f02fe0df640 1 --2- 172.21.15.123:0/2404939786 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f02e40738e0 0x7f02e4075da0 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:36.212 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:36.213+0000 7f02fe0df640 1 --2- 172.21.15.123:0/2404939786 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f02f80ad9a0 0x7f02f80ade10 unknown :-1 s=CLOSED pgs=1816 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:36.213 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:36.213+0000 7f02fe0df640 1 --2- 172.21.15.123:0/2404939786 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f02f80b4e10 0x7f02f80b5290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:36.213 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:36.213+0000 7f02fe0df640 1 --2- 172.21.15.123:0/2404939786 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f02e00052f0 0x7f02f81590a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:36.213 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:36.213+0000 7f02fe0df640 1 -- 172.21.15.123:0/2404939786 >> 172.21.15.123:0/2404939786 conn(0x7f02f80cfb10 msgr2=0x7f02f80ccdb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:51:36.213 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:36.213+0000 7f02fe0df640 1 -- 172.21.15.123:0/2404939786 shutdown_connections 2024-09-16T15:51:36.213 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:36.213+0000 7f02fe0df640 1 -- 172.21.15.123:0/2404939786 wait complete. 2024-09-16T15:51:36.229 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-09-16T15:51:36.300 INFO:tasks.ceph.mds.d.smithi184.stdout:starting mds.d at 2024-09-16T15:51:36.348 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:36.349+0000 7f7d04747640 1 -- 172.21.15.123:0/2916616727 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7ce8003a60 msgr2=0x7f7ce8003e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:51:36.349 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:36.349+0000 7f7d04747640 1 --2- 172.21.15.123:0/2916616727 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7ce8003a60 0x7f7ce8003e80 secure :-1 s=READY pgs=1818 cs=0 l=1 rev1=1 crypto rx=0x7f7cf8009b40 tx=0x7f7cf801d220 comp rx=0 tx=0).stop 2024-09-16T15:51:36.349 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:36.349+0000 7f7d04747640 1 -- 172.21.15.123:0/2916616727 shutdown_connections 2024-09-16T15:51:36.349 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:36.349+0000 7f7d04747640 1 --2- 172.21.15.123:0/2916616727 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7ce8003a60 0x7f7ce8003e80 unknown :-1 s=CLOSED pgs=1818 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:36.349 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:36.349+0000 7f7d04747640 1 -- 172.21.15.123:0/2916616727 >> 172.21.15.123:0/2916616727 conn(0x7f7cfc0741a0 msgr2=0x7f7cfc0745b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:51:36.349 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:36.349+0000 7f7d04747640 1 -- 172.21.15.123:0/2916616727 shutdown_connections 2024-09-16T15:51:36.349 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:36.349+0000 7f7d04747640 1 -- 172.21.15.123:0/2916616727 wait complete. 2024-09-16T15:51:36.349 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:36.349+0000 7f7d04747640 1 Processor -- start 2024-09-16T15:51:36.350 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:36.349+0000 7f7d04747640 1 -- start start 2024-09-16T15:51:36.352 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:36.349+0000 7f7d04747640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7ce8003a60 0x7f7cfc1fb970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:51:36.352 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:36.349+0000 7f7d04747640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7cfc0cf760 0x7f7cfc158480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:51:36.352 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:36.349+0000 7f7d04747640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f7cfc0cfbe0 0x7f7cfc1589c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:51:36.352 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:36.349+0000 7f7d04747640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f7cfc0b1660 con 0x7f7cfc0cf760 2024-09-16T15:51:36.352 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:36.349+0000 7f7d04747640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f7cfc0b14e0 con 0x7f7ce8003a60 2024-09-16T15:51:36.352 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:36.349+0000 7f7d04747640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f7cfc0b06a0 con 0x7f7cfc0cfbe0 2024-09-16T15:51:36.352 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:36.349+0000 7f7d01ce2640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7cfc0cf760 0x7f7cfc158480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:51:36.352 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:36.349+0000 7f7d01ce2640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7cfc0cf760 0x7f7cfc158480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:50234/0 (socket says 172.21.15.123:50234) 2024-09-16T15:51:36.352 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:36.349+0000 7f7d01ce2640 1 -- 172.21.15.123:0/313068241 learned_addr learned my addr 172.21.15.123:0/313068241 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:51:36.352 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:36.349+0000 7f7d024e3640 1 --2- 172.21.15.123:0/313068241 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7ce8003a60 0x7f7cfc1fb970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:51:36.352 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:36.353+0000 7f7d02ce4640 1 --2- 172.21.15.123:0/313068241 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f7cfc0cfbe0 0x7f7cfc1589c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:51:36.353 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:36.353+0000 7f7d01ce2640 1 -- 172.21.15.123:0/313068241 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f7cfc0cfbe0 msgr2=0x7f7cfc1589c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:51:36.353 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:36.353+0000 7f7d01ce2640 1 --2- 172.21.15.123:0/313068241 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f7cfc0cfbe0 0x7f7cfc1589c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:36.353 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:36.353+0000 7f7d01ce2640 1 -- 172.21.15.123:0/313068241 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7ce8003a60 msgr2=0x7f7cfc1fb970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:51:36.353 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:36.353+0000 7f7d01ce2640 1 --2- 172.21.15.123:0/313068241 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7ce8003a60 0x7f7cfc1fb970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:36.353 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:36.353+0000 7f7d01ce2640 1 -- 172.21.15.123:0/313068241 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7cfc0c4260 con 0x7f7cfc0cf760 2024-09-16T15:51:36.353 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:36.353+0000 7f7d01ce2640 1 --2- 172.21.15.123:0/313068241 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7cfc0cf760 0x7f7cfc158480 secure :-1 s=READY pgs=1819 cs=0 l=1 rev1=1 crypto rx=0x7f7cec004dd0 tx=0x7f7cec001640 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:51:36.353 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:36.353+0000 7f7cf2ffd640 1 -- 172.21.15.123:0/313068241 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7cec001d50 con 0x7f7cfc0cf760 2024-09-16T15:51:36.353 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:36.353+0000 7f7cf2ffd640 1 -- 172.21.15.123:0/313068241 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f7cec016070 con 0x7f7cfc0cf760 2024-09-16T15:51:36.353 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:36.353+0000 7f7cf2ffd640 1 -- 172.21.15.123:0/313068241 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7cec010b20 con 0x7f7cfc0cf760 2024-09-16T15:51:36.353 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:36.353+0000 7f7d04747640 1 -- 172.21.15.123:0/313068241 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7cfc1f7c30 con 0x7f7cfc0cf760 2024-09-16T15:51:36.353 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:36.353+0000 7f7d04747640 1 -- 172.21.15.123:0/313068241 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7cfc1f7940 con 0x7f7cfc0cf760 2024-09-16T15:51:36.353 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:36.353+0000 7f7d04747640 1 -- 172.21.15.123:0/313068241 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7cfc15fa50 con 0x7f7cfc0cf760 2024-09-16T15:51:36.354 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:36.353+0000 7f7cf2ffd640 1 -- 172.21.15.123:0/313068241 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 25) ==== 97998+0+0 (secure 0 0 0) 0x7f7cec010cc0 con 0x7f7cfc0cf760 2024-09-16T15:51:36.354 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:36.353+0000 7f7cf2ffd640 1 --2- 172.21.15.123:0/313068241 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f7ce0073910 0x7f7ce0075dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:51:36.354 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:36.353+0000 7f7cf2ffd640 1 -- 172.21.15.123:0/313068241 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(159..159 src has 1..159) ==== 5881+0+0 (secure 0 0 0) 0x7f7cec092fb0 con 0x7f7cfc0cf760 2024-09-16T15:51:36.354 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:36.353+0000 7f7d024e3640 1 --2- 172.21.15.123:0/313068241 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f7ce0073910 0x7f7ce0075dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:51:36.354 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:36.353+0000 7f7d024e3640 1 --2- 172.21.15.123:0/313068241 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f7ce0073910 0x7f7ce0075dd0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f7cf80ae000 tx=0x7f7cf8019fc0 comp rx=0 tx=0).ready entity=mgr.4150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:51:36.357 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:36.357+0000 7f7cf2ffd640 1 -- 172.21.15.123:0/313068241 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f7cec05fa90 con 0x7f7cfc0cf760 2024-09-16T15:51:36.550 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:36.549+0000 7f7d04747640 1 -- 172.21.15.123:0/313068241 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) -- 0x7f7cfc06e510 con 0x7f7cfc0cf760 2024-09-16T15:51:37.257 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:37.257+0000 7f7cf2ffd640 1 -- 172.21.15.123:0/313068241 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v108) ==== 104+0+0 (secure 0 0 0) 0x7f7cec05f390 con 0x7f7cfc0cf760 2024-09-16T15:51:37.260 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:37.261+0000 7f7d04747640 1 -- 172.21.15.123:0/313068241 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f7ce0073910 msgr2=0x7f7ce0075dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:51:37.260 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:37.261+0000 7f7d04747640 1 --2- 172.21.15.123:0/313068241 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f7ce0073910 0x7f7ce0075dd0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f7cf80ae000 tx=0x7f7cf8019fc0 comp rx=0 tx=0).stop 2024-09-16T15:51:37.260 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:37.261+0000 7f7d04747640 1 -- 172.21.15.123:0/313068241 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7cfc0cf760 msgr2=0x7f7cfc158480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:51:37.260 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:37.261+0000 7f7d04747640 1 --2- 172.21.15.123:0/313068241 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7cfc0cf760 0x7f7cfc158480 secure :-1 s=READY pgs=1819 cs=0 l=1 rev1=1 crypto rx=0x7f7cec004dd0 tx=0x7f7cec001640 comp rx=0 tx=0).stop 2024-09-16T15:51:37.261 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:37.261+0000 7f7d04747640 1 -- 172.21.15.123:0/313068241 shutdown_connections 2024-09-16T15:51:37.261 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:37.261+0000 7f7d04747640 1 --2- 172.21.15.123:0/313068241 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f7ce0073910 0x7f7ce0075dd0 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:37.261 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:37.261+0000 7f7d04747640 1 --2- 172.21.15.123:0/313068241 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f7cfc0cfbe0 0x7f7cfc1589c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:37.261 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:37.261+0000 7f7d04747640 1 --2- 172.21.15.123:0/313068241 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7cfc0cf760 0x7f7cfc158480 unknown :-1 s=CLOSED pgs=1819 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:37.261 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:37.261+0000 7f7d04747640 1 --2- 172.21.15.123:0/313068241 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7ce8003a60 0x7f7cfc1fb970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:37.261 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:37.261+0000 7f7d04747640 1 -- 172.21.15.123:0/313068241 >> 172.21.15.123:0/313068241 conn(0x7f7cfc0741a0 msgr2=0x7f7cfc0d07a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:51:37.261 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:37.261+0000 7f7d04747640 1 -- 172.21.15.123:0/313068241 shutdown_connections 2024-09-16T15:51:37.261 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:37.261+0000 7f7d04747640 1 -- 172.21.15.123:0/313068241 wait complete. 2024-09-16T15:51:37.281 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-09-16T15:51:37.419 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:37.413+0000 7fe51e14e640 1 -- 172.21.15.123:0/2052925964 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe500004810 msgr2=0x7fe500004c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:51:37.419 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:37.413+0000 7fe51e14e640 1 --2- 172.21.15.123:0/2052925964 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe500004810 0x7fe500004c90 secure :-1 s=READY pgs=1821 cs=0 l=1 rev1=1 crypto rx=0x7fe50c005fd0 tx=0x7fe50c01cd80 comp rx=0 tx=0).stop 2024-09-16T15:51:37.419 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:37.413+0000 7fe51e14e640 1 -- 172.21.15.123:0/2052925964 shutdown_connections 2024-09-16T15:51:37.419 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:37.413+0000 7fe51e14e640 1 --2- 172.21.15.123:0/2052925964 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe500004810 0x7fe500004c90 unknown :-1 s=CLOSED pgs=1821 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:37.419 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:37.413+0000 7fe51e14e640 1 -- 172.21.15.123:0/2052925964 >> 172.21.15.123:0/2052925964 conn(0x7fe51809d160 msgr2=0x7fe5180d3e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:51:37.419 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:37.417+0000 7fe51e14e640 1 -- 172.21.15.123:0/2052925964 shutdown_connections 2024-09-16T15:51:37.419 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:37.417+0000 7fe51e14e640 1 -- 172.21.15.123:0/2052925964 wait complete. 2024-09-16T15:51:37.420 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:37.417+0000 7fe51e14e640 1 Processor -- start 2024-09-16T15:51:37.420 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:37.417+0000 7fe51e14e640 1 -- start start 2024-09-16T15:51:37.420 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:37.417+0000 7fe51e14e640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fe500004810 0x7fe518161410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:51:37.420 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:37.417+0000 7fe51e14e640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe518159290 0x7fe51815a4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:51:37.420 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:37.417+0000 7fe51e14e640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe51806e4d0 0x7fe51806e960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:51:37.420 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:37.417+0000 7fe51e14e640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fe518161ec0 con 0x7fe51806e4d0 2024-09-16T15:51:37.420 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:37.417+0000 7fe51e14e640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fe518161d40 con 0x7fe518159290 2024-09-16T15:51:37.420 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:37.417+0000 7fe51e14e640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fe518162040 con 0x7fe500004810 2024-09-16T15:51:37.420 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:37.417+0000 7fe517fff640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe51806e4d0 0x7fe51806e960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:51:37.420 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:37.417+0000 7fe517fff640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe51806e4d0 0x7fe51806e960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:50258/0 (socket says 172.21.15.123:50258) 2024-09-16T15:51:37.420 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:37.417+0000 7fe517fff640 1 -- 172.21.15.123:0/341687503 learned_addr learned my addr 172.21.15.123:0/341687503 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:51:37.420 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:37.417+0000 7fe5177fe640 1 --2- 172.21.15.123:0/341687503 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fe500004810 0x7fe518161410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:51:37.420 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:37.417+0000 7fe516ffd640 1 --2- 172.21.15.123:0/341687503 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe518159290 0x7fe51815a4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:51:37.421 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:37.417+0000 7fe517fff640 1 -- 172.21.15.123:0/341687503 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fe500004810 msgr2=0x7fe518161410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:51:37.421 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:37.417+0000 7fe517fff640 1 --2- 172.21.15.123:0/341687503 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fe500004810 0x7fe518161410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:37.421 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:37.417+0000 7fe517fff640 1 -- 172.21.15.123:0/341687503 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe518159290 msgr2=0x7fe51815a4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:51:37.421 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:37.417+0000 7fe517fff640 1 --2- 172.21.15.123:0/341687503 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe518159290 0x7fe51815a4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:37.421 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:37.417+0000 7fe517fff640 1 -- 172.21.15.123:0/341687503 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe5181f5f40 con 0x7fe51806e4d0 2024-09-16T15:51:37.421 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:37.417+0000 7fe516ffd640 1 --2- 172.21.15.123:0/341687503 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe518159290 0x7fe51815a4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T15:51:37.421 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:37.417+0000 7fe517fff640 1 --2- 172.21.15.123:0/341687503 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe51806e4d0 0x7fe51806e960 secure :-1 s=READY pgs=1822 cs=0 l=1 rev1=1 crypto rx=0x7fe5080036b0 tx=0x7fe50800f820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:51:37.421 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:37.417+0000 7fe4fffff640 1 -- 172.21.15.123:0/341687503 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe508008520 con 0x7fe51806e4d0 2024-09-16T15:51:37.421 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:37.417+0000 7fe4fffff640 1 -- 172.21.15.123:0/341687503 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fe5080098c0 con 0x7fe51806e4d0 2024-09-16T15:51:37.421 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:37.417+0000 7fe51e14e640 1 -- 172.21.15.123:0/341687503 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe51815b600 con 0x7fe51806e4d0 2024-09-16T15:51:37.421 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:37.417+0000 7fe4fffff640 1 -- 172.21.15.123:0/341687503 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe508009d10 con 0x7fe51806e4d0 2024-09-16T15:51:37.421 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:37.417+0000 7fe51e14e640 1 -- 172.21.15.123:0/341687503 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe5181587b0 con 0x7fe51806e4d0 2024-09-16T15:51:37.421 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:37.417+0000 7fe51e14e640 1 -- 172.21.15.123:0/341687503 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe5181625a0 con 0x7fe51806e4d0 2024-09-16T15:51:37.422 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:37.421+0000 7fe4fffff640 1 -- 172.21.15.123:0/341687503 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 25) ==== 97998+0+0 (secure 0 0 0) 0x7fe508006720 con 0x7fe51806e4d0 2024-09-16T15:51:37.422 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:37.421+0000 7fe4fffff640 1 --2- 172.21.15.123:0/341687503 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fe504073910 0x7fe504075dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:51:37.422 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:37.421+0000 7fe4fffff640 1 -- 172.21.15.123:0/341687503 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(159..159 src has 1..159) ==== 5881+0+0 (secure 0 0 0) 0x7fe508094f60 con 0x7fe51806e4d0 2024-09-16T15:51:37.422 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:37.421+0000 7fe5177fe640 1 --2- 172.21.15.123:0/341687503 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fe504073910 0x7fe504075dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:51:37.423 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:37.421+0000 7fe5177fe640 1 --2- 172.21.15.123:0/341687503 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fe504073910 0x7fe504075dd0 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7fe51806b850 tx=0x7fe510009040 comp rx=0 tx=0).ready entity=mgr.4150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:51:37.423 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:37.425+0000 7fe4fffff640 1 -- 172.21.15.123:0/341687503 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7fe5181f5f40 con 0x7fe51806e4d0 2024-09-16T15:51:37.597 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:37.597+0000 7fe51e14e640 1 -- 172.21.15.123:0/341687503 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) -- 0x7fe5180cacf0 con 0x7fe51806e4d0 2024-09-16T15:51:38.274 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:38.273+0000 7fe4fffff640 1 -- 172.21.15.123:0/341687503 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v160) ==== 181+0+0 (secure 0 0 0) 0x7fe508061a20 con 0x7fe51806e4d0 2024-09-16T15:51:38.275 INFO:teuthology.orchestra.run.smithi123.stderr:pool 'cephfs_metadata' removed 2024-09-16T15:51:38.278 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:38.277+0000 7fe51e14e640 1 -- 172.21.15.123:0/341687503 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fe504073910 msgr2=0x7fe504075dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:51:38.278 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:38.277+0000 7fe51e14e640 1 --2- 172.21.15.123:0/341687503 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fe504073910 0x7fe504075dd0 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7fe51806b850 tx=0x7fe510009040 comp rx=0 tx=0).stop 2024-09-16T15:51:38.278 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:38.277+0000 7fe51e14e640 1 -- 172.21.15.123:0/341687503 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe51806e4d0 msgr2=0x7fe51806e960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:51:38.278 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:38.277+0000 7fe51e14e640 1 --2- 172.21.15.123:0/341687503 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe51806e4d0 0x7fe51806e960 secure :-1 s=READY pgs=1822 cs=0 l=1 rev1=1 crypto rx=0x7fe5080036b0 tx=0x7fe50800f820 comp rx=0 tx=0).stop 2024-09-16T15:51:38.278 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:38.277+0000 7fe51e14e640 1 -- 172.21.15.123:0/341687503 shutdown_connections 2024-09-16T15:51:38.278 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:38.277+0000 7fe51e14e640 1 --2- 172.21.15.123:0/341687503 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fe504073910 0x7fe504075dd0 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:38.278 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:38.277+0000 7fe51e14e640 1 --2- 172.21.15.123:0/341687503 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe51806e4d0 0x7fe51806e960 unknown :-1 s=CLOSED pgs=1822 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:38.278 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:38.277+0000 7fe51e14e640 1 --2- 172.21.15.123:0/341687503 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe518159290 0x7fe51815a4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:38.278 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:38.277+0000 7fe51e14e640 1 --2- 172.21.15.123:0/341687503 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fe500004810 0x7fe518161410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:38.278 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:38.277+0000 7fe51e14e640 1 -- 172.21.15.123:0/341687503 >> 172.21.15.123:0/341687503 conn(0x7fe51809d160 msgr2=0x7fe5180c84b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:51:38.278 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:38.277+0000 7fe51e14e640 1 -- 172.21.15.123:0/341687503 shutdown_connections 2024-09-16T15:51:38.279 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:38.277+0000 7fe51e14e640 1 -- 172.21.15.123:0/341687503 wait complete. 2024-09-16T15:51:38.309 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-09-16T15:51:38.444 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:38.445+0000 7f3b5b7d5640 1 -- 172.21.15.123:0/1600013323 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3b540cd450 msgr2=0x7f3b5415f330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:51:38.444 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:38.445+0000 7f3b5b7d5640 1 --2- 172.21.15.123:0/1600013323 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3b540cd450 0x7f3b5415f330 secure :-1 s=READY pgs=1824 cs=0 l=1 rev1=1 crypto rx=0x7f3b540cbaa0 tx=0x7f3b4c01cba0 comp rx=0 tx=0).stop 2024-09-16T15:51:38.444 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:38.445+0000 7f3b5b7d5640 1 -- 172.21.15.123:0/1600013323 shutdown_connections 2024-09-16T15:51:38.444 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:38.445+0000 7f3b5b7d5640 1 --2- 172.21.15.123:0/1600013323 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3b540cd450 0x7f3b5415f330 unknown :-1 s=CLOSED pgs=1824 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:38.444 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:38.445+0000 7f3b5b7d5640 1 -- 172.21.15.123:0/1600013323 >> 172.21.15.123:0/1600013323 conn(0x7f3b54074110 msgr2=0x7f3b540744f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:51:38.444 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:38.445+0000 7f3b5b7d5640 1 -- 172.21.15.123:0/1600013323 shutdown_connections 2024-09-16T15:51:38.444 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:38.445+0000 7f3b5b7d5640 1 -- 172.21.15.123:0/1600013323 wait complete. 2024-09-16T15:51:38.445 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:38.445+0000 7f3b5b7d5640 1 Processor -- start 2024-09-16T15:51:38.445 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:38.445+0000 7f3b5b7d5640 1 -- start start 2024-09-16T15:51:38.447 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:38.445+0000 7f3b5b7d5640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3b540af480 0x7f3b541581c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:51:38.447 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:38.445+0000 7f3b5b7d5640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f3b540cd450 0x7f3b54156d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:51:38.447 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:38.445+0000 7f3b5b7d5640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3b5415ada0 0x7f3b541572c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:51:38.447 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:38.445+0000 7f3b5b7d5640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f3b54169040 con 0x7f3b5415ada0 2024-09-16T15:51:38.447 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:38.445+0000 7f3b5b7d5640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f3b54168ec0 con 0x7f3b540af480 2024-09-16T15:51:38.447 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:38.445+0000 7f3b5b7d5640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f3b541691c0 con 0x7f3b540cd450 2024-09-16T15:51:38.447 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:38.445+0000 7f3b59d72640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3b5415ada0 0x7f3b541572c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:51:38.447 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:38.445+0000 7f3b59d72640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3b5415ada0 0x7f3b541572c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:50280/0 (socket says 172.21.15.123:50280) 2024-09-16T15:51:38.447 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:38.445+0000 7f3b59d72640 1 -- 172.21.15.123:0/279392404 learned_addr learned my addr 172.21.15.123:0/279392404 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:51:38.447 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:38.445+0000 7f3b59571640 1 --2- 172.21.15.123:0/279392404 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3b540af480 0x7f3b541581c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:51:38.447 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:38.445+0000 7f3b59d72640 1 -- 172.21.15.123:0/279392404 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f3b540cd450 msgr2=0x7f3b54156d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:51:38.447 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:38.445+0000 7f3b59d72640 1 --2- 172.21.15.123:0/279392404 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f3b540cd450 0x7f3b54156d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:38.448 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:38.445+0000 7f3b59d72640 1 -- 172.21.15.123:0/279392404 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3b540af480 msgr2=0x7f3b541581c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:51:38.448 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:38.445+0000 7f3b59d72640 1 --2- 172.21.15.123:0/279392404 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3b540af480 0x7f3b541581c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:38.448 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:38.445+0000 7f3b59d72640 1 -- 172.21.15.123:0/279392404 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3b541f9270 con 0x7f3b5415ada0 2024-09-16T15:51:38.448 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:38.445+0000 7f3b59d72640 1 --2- 172.21.15.123:0/279392404 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3b5415ada0 0x7f3b541572c0 secure :-1 s=READY pgs=1825 cs=0 l=1 rev1=1 crypto rx=0x7f3b4c009b10 tx=0x7f3b4c009300 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:51:38.448 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:38.445+0000 7f3b49ffb640 1 -- 172.21.15.123:0/279392404 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b4c0bb070 con 0x7f3b5415ada0 2024-09-16T15:51:38.448 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:38.445+0000 7f3b49ffb640 1 -- 172.21.15.123:0/279392404 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f3b4c004d40 con 0x7f3b5415ada0 2024-09-16T15:51:38.448 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:38.445+0000 7f3b5b7d5640 1 -- 172.21.15.123:0/279392404 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3b541f9550 con 0x7f3b5415ada0 2024-09-16T15:51:38.448 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:38.445+0000 7f3b49ffb640 1 -- 172.21.15.123:0/279392404 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b4c0b2040 con 0x7f3b5415ada0 2024-09-16T15:51:38.448 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:38.445+0000 7f3b5b7d5640 1 -- 172.21.15.123:0/279392404 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3b541f9b10 con 0x7f3b5415ada0 2024-09-16T15:51:38.449 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:38.449+0000 7f3b5b7d5640 1 -- 172.21.15.123:0/279392404 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3b5415f330 con 0x7f3b5415ada0 2024-09-16T15:51:38.449 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:38.449+0000 7f3b49ffb640 1 -- 172.21.15.123:0/279392404 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 25) ==== 97998+0+0 (secure 0 0 0) 0x7f3b4c0be070 con 0x7f3b5415ada0 2024-09-16T15:51:38.449 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:38.449+0000 7f3b49ffb640 1 --2- 172.21.15.123:0/279392404 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f3b380738e0 0x7f3b38075da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:51:38.449 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:38.449+0000 7f3b49ffb640 1 -- 172.21.15.123:0/279392404 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(160..160 src has 1..160) ==== 5443+0+0 (secure 0 0 0) 0x7f3b4c12c410 con 0x7f3b5415ada0 2024-09-16T15:51:38.449 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:38.449+0000 7f3b59571640 1 --2- 172.21.15.123:0/279392404 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f3b380738e0 0x7f3b38075da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:51:38.449 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:38.449+0000 7f3b59571640 1 --2- 172.21.15.123:0/279392404 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f3b380738e0 0x7f3b38075da0 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f3b44001230 tx=0x7f3b44009040 comp rx=0 tx=0).ready entity=mgr.4150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:51:38.453 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:38.453+0000 7f3b49ffb640 1 -- 172.21.15.123:0/279392404 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f3b4c0a5e20 con 0x7f3b5415ada0 2024-09-16T15:51:38.626 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:38.625+0000 7f3b5b7d5640 1 -- 172.21.15.123:0/279392404 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) -- 0x7f3b540b62b0 con 0x7f3b5415ada0 2024-09-16T15:51:39.289 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:39.289+0000 7f3b49ffb640 1 -- 172.21.15.123:0/279392404 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v161) ==== 169+0+0 (secure 0 0 0) 0x7f3b4c016790 con 0x7f3b5415ada0 2024-09-16T15:51:39.289 INFO:teuthology.orchestra.run.smithi123.stderr:pool 'cephfs_data' removed 2024-09-16T15:51:39.292 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:39.293+0000 7f3b5b7d5640 1 -- 172.21.15.123:0/279392404 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f3b380738e0 msgr2=0x7f3b38075da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:51:39.292 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:39.293+0000 7f3b5b7d5640 1 --2- 172.21.15.123:0/279392404 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f3b380738e0 0x7f3b38075da0 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f3b44001230 tx=0x7f3b44009040 comp rx=0 tx=0).stop 2024-09-16T15:51:39.292 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:39.293+0000 7f3b5b7d5640 1 -- 172.21.15.123:0/279392404 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3b5415ada0 msgr2=0x7f3b541572c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:51:39.293 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:39.293+0000 7f3b5b7d5640 1 --2- 172.21.15.123:0/279392404 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3b5415ada0 0x7f3b541572c0 secure :-1 s=READY pgs=1825 cs=0 l=1 rev1=1 crypto rx=0x7f3b4c009b10 tx=0x7f3b4c009300 comp rx=0 tx=0).stop 2024-09-16T15:51:39.293 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:39.293+0000 7f3b5b7d5640 1 -- 172.21.15.123:0/279392404 shutdown_connections 2024-09-16T15:51:39.293 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:39.293+0000 7f3b5b7d5640 1 --2- 172.21.15.123:0/279392404 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f3b380738e0 0x7f3b38075da0 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:39.293 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:39.293+0000 7f3b5b7d5640 1 --2- 172.21.15.123:0/279392404 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3b5415ada0 0x7f3b541572c0 unknown :-1 s=CLOSED pgs=1825 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:39.293 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:39.293+0000 7f3b5b7d5640 1 --2- 172.21.15.123:0/279392404 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f3b540cd450 0x7f3b54156d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:39.293 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:39.293+0000 7f3b5b7d5640 1 --2- 172.21.15.123:0/279392404 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3b540af480 0x7f3b541581c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:39.293 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:39.293+0000 7f3b5b7d5640 1 -- 172.21.15.123:0/279392404 >> 172.21.15.123:0/279392404 conn(0x7f3b54074110 msgr2=0x7f3b5406f820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:51:39.293 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:39.293+0000 7f3b5b7d5640 1 -- 172.21.15.123:0/279392404 shutdown_connections 2024-09-16T15:51:39.294 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:39.293+0000 7f3b5b7d5640 1 -- 172.21.15.123:0/279392404 wait complete. 2024-09-16T15:51:39.319 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config reset 27 2024-09-16T15:51:39.447 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:39.445+0000 7f711ed1c640 1 -- 172.21.15.123:0/3270065718 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f711815ac00 msgr2=0x7f71180702f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:51:39.447 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:39.445+0000 7f711ed1c640 1 --2- 172.21.15.123:0/3270065718 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f711815ac00 0x7f71180702f0 secure :-1 s=READY pgs=1826 cs=0 l=1 rev1=1 crypto rx=0x7f71180c45f0 tx=0x7f710801c8c0 comp rx=0 tx=0).stop 2024-09-16T15:51:39.447 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:39.445+0000 7f711ed1c640 1 -- 172.21.15.123:0/3270065718 shutdown_connections 2024-09-16T15:51:39.447 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:39.445+0000 7f711ed1c640 1 --2- 172.21.15.123:0/3270065718 >> v2:172.21.15.184:3300/0 conn(0x7f71180cc8c0 0x7f71180cf650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:39.447 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:39.445+0000 7f711ed1c640 1 --2- 172.21.15.123:0/3270065718 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f711815ac00 0x7f71180702f0 unknown :-1 s=CLOSED pgs=1826 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:39.447 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:39.445+0000 7f711ed1c640 1 -- 172.21.15.123:0/3270065718 >> 172.21.15.123:0/3270065718 conn(0x7f71180b2610 msgr2=0x7f71180b29f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:51:39.447 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:39.445+0000 7f711ed1c640 1 -- 172.21.15.123:0/3270065718 shutdown_connections 2024-09-16T15:51:39.447 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:39.445+0000 7f711ed1c640 1 -- 172.21.15.123:0/3270065718 wait complete. 2024-09-16T15:51:39.448 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:39.449+0000 7f711ed1c640 1 Processor -- start 2024-09-16T15:51:39.448 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:39.449+0000 7f711ed1c640 1 -- start start 2024-09-16T15:51:39.450 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:39.449+0000 7f711ed1c640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f71180b3100 0x7f7118155bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:51:39.450 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:39.449+0000 7f711ed1c640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f71180cc8c0 0x7f71181fb9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:51:39.450 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:39.449+0000 7f711ed1c640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f711815ac00 0x7f71181fbf00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:51:39.450 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:39.449+0000 7f711ed1c640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f7118168660 con 0x7f711815ac00 2024-09-16T15:51:39.450 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:39.449+0000 7f711ed1c640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f71181684e0 con 0x7f71180b3100 2024-09-16T15:51:39.450 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:39.449+0000 7f711ed1c640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f71181687e0 con 0x7f71180cc8c0 2024-09-16T15:51:39.450 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:39.449+0000 7f711d2b9640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f711815ac00 0x7f71181fbf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:51:39.450 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:39.449+0000 7f711d2b9640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f711815ac00 0x7f71181fbf00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:50286/0 (socket says 172.21.15.123:50286) 2024-09-16T15:51:39.450 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:39.449+0000 7f711d2b9640 1 -- 172.21.15.123:0/2440967621 learned_addr learned my addr 172.21.15.123:0/2440967621 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:51:39.450 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:39.449+0000 7f711cab8640 1 --2- 172.21.15.123:0/2440967621 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f71180b3100 0x7f7118155bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:51:39.451 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:39.449+0000 7f711d2b9640 1 -- 172.21.15.123:0/2440967621 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f71180cc8c0 msgr2=0x7f71181fb9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:51:39.451 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:39.449+0000 7f7117fff640 1 --2- 172.21.15.123:0/2440967621 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f71180cc8c0 0x7f71181fb9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:51:39.451 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:39.449+0000 7f711d2b9640 1 --2- 172.21.15.123:0/2440967621 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f71180cc8c0 0x7f71181fb9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:39.451 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:39.449+0000 7f711d2b9640 1 -- 172.21.15.123:0/2440967621 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f71180b3100 msgr2=0x7f7118155bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:51:39.451 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:39.449+0000 7f711d2b9640 1 --2- 172.21.15.123:0/2440967621 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f71180b3100 0x7f7118155bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:39.451 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:39.449+0000 7f711d2b9640 1 -- 172.21.15.123:0/2440967621 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f711815bfb0 con 0x7f711815ac00 2024-09-16T15:51:39.451 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:39.449+0000 7f711cab8640 1 --2- 172.21.15.123:0/2440967621 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f71180b3100 0x7f7118155bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T15:51:39.451 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:39.449+0000 7f711d2b9640 1 --2- 172.21.15.123:0/2440967621 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f711815ac00 0x7f71181fbf00 secure :-1 s=READY pgs=1827 cs=0 l=1 rev1=1 crypto rx=0x7f711000b970 tx=0x7f711000be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:51:39.451 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:39.449+0000 7f71157fa640 1 -- 172.21.15.123:0/2440967621 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7110019070 con 0x7f711815ac00 2024-09-16T15:51:39.451 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:39.449+0000 7f71157fa640 1 -- 172.21.15.123:0/2440967621 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f711000cc50 con 0x7f711815ac00 2024-09-16T15:51:39.451 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:39.449+0000 7f71157fa640 1 -- 172.21.15.123:0/2440967621 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71100146a0 con 0x7f711815ac00 2024-09-16T15:51:39.451 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:39.449+0000 7f711ed1c640 1 -- 172.21.15.123:0/2440967621 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f711815c290 con 0x7f711815ac00 2024-09-16T15:51:39.451 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:39.449+0000 7f711ed1c640 1 -- 172.21.15.123:0/2440967621 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f711815c850 con 0x7f711815ac00 2024-09-16T15:51:39.452 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:39.449+0000 7f711ed1c640 1 -- 172.21.15.123:0/2440967621 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f71180cf650 con 0x7f711815ac00 2024-09-16T15:51:39.452 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:39.453+0000 7f71157fa640 1 -- 172.21.15.123:0/2440967621 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 25) ==== 97998+0+0 (secure 0 0 0) 0x7f711001c070 con 0x7f711815ac00 2024-09-16T15:51:39.452 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:39.453+0000 7f71157fa640 1 --2- 172.21.15.123:0/2440967621 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f71040738e0 0x7f7104075da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:51:39.452 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:39.453+0000 7f71157fa640 1 -- 172.21.15.123:0/2440967621 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(161..161 src has 1..161) ==== 4971+0+0 (secure 0 0 0) 0x7f7110092b70 con 0x7f711815ac00 2024-09-16T15:51:39.453 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:39.453+0000 7f711cab8640 1 --2- 172.21.15.123:0/2440967621 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f71040738e0 0x7f7104075da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:51:39.453 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:39.453+0000 7f711cab8640 1 --2- 172.21.15.123:0/2440967621 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f71040738e0 0x7f7104075da0 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f71080af000 tx=0x7f710801b100 comp rx=0 tx=0).ready entity=mgr.4150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:51:39.455 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:39.457+0000 7f71157fa640 1 -- 172.21.15.123:0/2440967621 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f7110023020 con 0x7f711815ac00 2024-09-16T15:51:39.628 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:39.629+0000 7f711ed1c640 1 -- 172.21.15.123:0/2440967621 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "config reset", "num": 27} v 0) -- 0x7f71180b6aa0 con 0x7f711815ac00 2024-09-16T15:51:39.636 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:39.637+0000 7f71157fa640 1 -- 172.21.15.123:0/2440967621 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "config reset", "num": 27}]=0 v47) ==== 71+0+0 (secure 0 0 0) 0x7f711005f5f0 con 0x7f711815ac00 2024-09-16T15:51:39.639 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:39.641+0000 7f711ed1c640 1 -- 172.21.15.123:0/2440967621 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f71040738e0 msgr2=0x7f7104075da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:51:39.639 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:39.641+0000 7f711ed1c640 1 --2- 172.21.15.123:0/2440967621 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f71040738e0 0x7f7104075da0 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f71080af000 tx=0x7f710801b100 comp rx=0 tx=0).stop 2024-09-16T15:51:39.639 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:39.641+0000 7f711ed1c640 1 -- 172.21.15.123:0/2440967621 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f711815ac00 msgr2=0x7f71181fbf00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:51:39.639 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:39.641+0000 7f711ed1c640 1 --2- 172.21.15.123:0/2440967621 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f711815ac00 0x7f71181fbf00 secure :-1 s=READY pgs=1827 cs=0 l=1 rev1=1 crypto rx=0x7f711000b970 tx=0x7f711000be30 comp rx=0 tx=0).stop 2024-09-16T15:51:39.639 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:39.641+0000 7f711ed1c640 1 -- 172.21.15.123:0/2440967621 shutdown_connections 2024-09-16T15:51:39.639 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:39.641+0000 7f711ed1c640 1 --2- 172.21.15.123:0/2440967621 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f71040738e0 0x7f7104075da0 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:39.639 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:39.641+0000 7f711ed1c640 1 --2- 172.21.15.123:0/2440967621 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f711815ac00 0x7f71181fbf00 unknown :-1 s=CLOSED pgs=1827 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:39.640 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:39.641+0000 7f711ed1c640 1 --2- 172.21.15.123:0/2440967621 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f71180cc8c0 0x7f71181fb9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:39.640 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:39.641+0000 7f711ed1c640 1 --2- 172.21.15.123:0/2440967621 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f71180b3100 0x7f7118155bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:39.640 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:39.641+0000 7f711ed1c640 1 -- 172.21.15.123:0/2440967621 >> 172.21.15.123:0/2440967621 conn(0x7f71180b2610 msgr2=0x7f71180cc1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:51:39.640 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:39.641+0000 7f711ed1c640 1 -- 172.21.15.123:0/2440967621 shutdown_connections 2024-09-16T15:51:39.640 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:39.641+0000 7f711ed1c640 1 -- 172.21.15.123:0/2440967621 wait complete. 2024-09-16T15:51:39.656 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_volumes.TestMisc.test_subvolume_upgrade_v1_to_v2' 2024-09-16T15:51:39.784 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:39.785+0000 7fd9355f2640 1 -- 172.21.15.123:0/1394092141 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd91c003a10 msgr2=0x7fd91c003e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:51:39.784 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:39.785+0000 7fd9355f2640 1 --2- 172.21.15.123:0/1394092141 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd91c003a10 0x7fd91c003e30 secure :-1 s=READY pgs=1829 cs=0 l=1 rev1=1 crypto rx=0x7fd91800bb10 tx=0x7fd91801cd70 comp rx=0 tx=0).stop 2024-09-16T15:51:39.784 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:39.785+0000 7fd92f7fe640 1 -- 172.21.15.123:0/1394092141 reap_dead start 2024-09-16T15:51:39.784 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:39.785+0000 7fd9355f2640 1 -- 172.21.15.123:0/1394092141 shutdown_connections 2024-09-16T15:51:39.784 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:39.785+0000 7fd9355f2640 1 -- 172.21.15.123:0/1394092141 >> 172.21.15.123:0/1394092141 conn(0x7fd9300cfb10 msgr2=0x7fd9300cff20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:51:39.784 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:39.785+0000 7fd9355f2640 1 -- 172.21.15.123:0/1394092141 shutdown_connections 2024-09-16T15:51:39.784 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:39.785+0000 7fd9355f2640 1 -- 172.21.15.123:0/1394092141 wait complete. 2024-09-16T15:51:39.785 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:39.785+0000 7fd9355f2640 1 Processor -- start 2024-09-16T15:51:39.785 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:39.785+0000 7fd9355f2640 1 -- start start 2024-09-16T15:51:39.787 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:39.785+0000 7fd9355f2640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd91c003a10 0x7fd930164840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:51:39.787 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:39.785+0000 7fd9355f2640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fd9300b0f20 0x7fd9300c9a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:51:39.787 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:39.785+0000 7fd9355f2640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd93015a850 0x7fd93015ad00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:51:39.787 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:39.785+0000 7fd9355f2640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fd930166c50 con 0x7fd93015a850 2024-09-16T15:51:39.787 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:39.785+0000 7fd9355f2640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fd930074980 con 0x7fd91c003a10 2024-09-16T15:51:39.787 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:39.785+0000 7fd9355f2640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fd930166dd0 con 0x7fd9300b0f20 2024-09-16T15:51:39.787 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:39.785+0000 7fd92f7fe640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd93015a850 0x7fd93015ad00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:51:39.788 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:39.785+0000 7fd92f7fe640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd93015a850 0x7fd93015ad00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:50302/0 (socket says 172.21.15.123:50302) 2024-09-16T15:51:39.788 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:39.785+0000 7fd92f7fe640 1 -- 172.21.15.123:0/125005353 learned_addr learned my addr 172.21.15.123:0/125005353 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:51:39.788 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:39.785+0000 7fd92e7fc640 1 --2- 172.21.15.123:0/125005353 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fd9300b0f20 0x7fd9300c9a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:51:39.788 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:39.785+0000 7fd92effd640 1 --2- 172.21.15.123:0/125005353 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd91c003a10 0x7fd930164840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:51:39.788 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:39.785+0000 7fd92f7fe640 1 -- 172.21.15.123:0/125005353 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fd9300b0f20 msgr2=0x7fd9300c9a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:51:39.788 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:39.785+0000 7fd92f7fe640 1 --2- 172.21.15.123:0/125005353 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fd9300b0f20 0x7fd9300c9a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:39.788 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:39.785+0000 7fd92f7fe640 1 -- 172.21.15.123:0/125005353 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd91c003a10 msgr2=0x7fd930164840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:51:39.788 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:39.785+0000 7fd92f7fe640 1 --2- 172.21.15.123:0/125005353 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd91c003a10 0x7fd930164840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:39.788 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:39.785+0000 7fd92f7fe640 1 -- 172.21.15.123:0/125005353 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd9300ce0f0 con 0x7fd93015a850 2024-09-16T15:51:39.788 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:39.785+0000 7fd92effd640 1 --2- 172.21.15.123:0/125005353 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd91c003a10 0x7fd930164840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T15:51:39.788 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:39.785+0000 7fd92f7fe640 1 --2- 172.21.15.123:0/125005353 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd93015a850 0x7fd93015ad00 secure :-1 s=READY pgs=1830 cs=0 l=1 rev1=1 crypto rx=0x7fd928000fc0 tx=0x7fd92800db90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:51:39.788 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:39.785+0000 7fd9277fe640 1 -- 172.21.15.123:0/125005353 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd92800b610 con 0x7fd93015a850 2024-09-16T15:51:39.788 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:39.785+0000 7fd9277fe640 1 -- 172.21.15.123:0/125005353 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fd928018070 con 0x7fd93015a850 2024-09-16T15:51:39.789 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:39.785+0000 7fd9355f2640 1 -- 172.21.15.123:0/125005353 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd9300ce3d0 con 0x7fd93015a850 2024-09-16T15:51:39.789 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:39.785+0000 7fd9277fe640 1 -- 172.21.15.123:0/125005353 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd92800ba70 con 0x7fd93015a850 2024-09-16T15:51:39.789 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:39.785+0000 7fd9355f2640 1 -- 172.21.15.123:0/125005353 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd9300ce990 con 0x7fd93015a850 2024-09-16T15:51:39.789 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:39.789+0000 7fd9355f2640 1 -- 172.21.15.123:0/125005353 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd930073ef0 con 0x7fd93015a850 2024-09-16T15:51:39.789 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:39.789+0000 7fd9277fe640 1 -- 172.21.15.123:0/125005353 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 25) ==== 97998+0+0 (secure 0 0 0) 0x7fd92800bc10 con 0x7fd93015a850 2024-09-16T15:51:39.790 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:39.789+0000 7fd9277fe640 1 --2- 172.21.15.123:0/125005353 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fd910073910 0x7fd910075dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:51:39.790 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:39.789+0000 7fd9277fe640 1 -- 172.21.15.123:0/125005353 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(161..161 src has 1..161) ==== 4971+0+0 (secure 0 0 0) 0x7fd928092820 con 0x7fd93015a850 2024-09-16T15:51:39.790 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:39.789+0000 7fd92effd640 1 --2- 172.21.15.123:0/125005353 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fd910073910 0x7fd910075dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:51:39.790 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:39.789+0000 7fd92effd640 1 --2- 172.21.15.123:0/125005353 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fd910073910 0x7fd910075dd0 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7fd9180b0000 tx=0x7fd918019b10 comp rx=0 tx=0).ready entity=mgr.4150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:51:39.792 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:39.793+0000 7fd9277fe640 1 -- 172.21.15.123:0/125005353 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7fd928021020 con 0x7fd93015a850 2024-09-16T15:51:40.036 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:40.037+0000 7fd9355f2640 1 -- 172.21.15.123:0/125005353 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_volumes.TestMisc.test_subvolume_upgrade_v1_to_v2"]} v 0) -- 0x7fd93015f630 con 0x7fd93015a850 2024-09-16T15:51:40.291 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:40.289+0000 7fd9277fe640 1 -- 172.21.15.123:0/125005353 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_volumes.TestMisc.test_subvolume_upgrade_v1_to_v2"]}]=0 v464) ==== 145+0+0 (secure 0 0 0) 0x7fd92805f2e0 con 0x7fd93015a850 2024-09-16T15:51:40.294 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:40.293+0000 7fd9355f2640 1 -- 172.21.15.123:0/125005353 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fd910073910 msgr2=0x7fd910075dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:51:40.294 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:40.293+0000 7fd9355f2640 1 --2- 172.21.15.123:0/125005353 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fd910073910 0x7fd910075dd0 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7fd9180b0000 tx=0x7fd918019b10 comp rx=0 tx=0).stop 2024-09-16T15:51:40.294 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:40.293+0000 7fd9355f2640 1 -- 172.21.15.123:0/125005353 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd93015a850 msgr2=0x7fd93015ad00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:51:40.294 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:40.293+0000 7fd9355f2640 1 --2- 172.21.15.123:0/125005353 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd93015a850 0x7fd93015ad00 secure :-1 s=READY pgs=1830 cs=0 l=1 rev1=1 crypto rx=0x7fd928000fc0 tx=0x7fd92800db90 comp rx=0 tx=0).stop 2024-09-16T15:51:40.294 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:40.293+0000 7fd9355f2640 1 -- 172.21.15.123:0/125005353 shutdown_connections 2024-09-16T15:51:40.294 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:40.293+0000 7fd9355f2640 1 --2- 172.21.15.123:0/125005353 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fd910073910 0x7fd910075dd0 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:40.294 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:40.293+0000 7fd9355f2640 1 --2- 172.21.15.123:0/125005353 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd93015a850 0x7fd93015ad00 unknown :-1 s=CLOSED pgs=1830 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:40.294 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:40.293+0000 7fd9355f2640 1 --2- 172.21.15.123:0/125005353 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fd9300b0f20 0x7fd9300c9a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:40.295 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:40.293+0000 7fd9355f2640 1 --2- 172.21.15.123:0/125005353 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd91c003a10 0x7fd930164840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:40.295 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:40.293+0000 7fd9355f2640 1 -- 172.21.15.123:0/125005353 >> 172.21.15.123:0/125005353 conn(0x7fd9300cfb10 msgr2=0x7fd93015c300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:51:40.295 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:40.293+0000 7fd9355f2640 1 -- 172.21.15.123:0/125005353 shutdown_connections 2024-09-16T15:51:40.295 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:40.293+0000 7fd9355f2640 1 -- 172.21.15.123:0/125005353 wait complete. 2024-09-16T15:51:40.314 INFO:tasks.cephfs_test_runner:poor man's upgrade test -- theme continues... ... ok 2024-09-16T15:51:40.314 INFO:tasks.cephfs_test_runner:Starting test: test_unparseable_metafile_on_legacy_to_v1_upgrade (tasks.cephfs.test_volumes.TestMisc) Validate unparseable text .meta file on legacy subvol root doesn't break the system 2024-09-16T15:51:40.314 INFO:tasks.cephfs_test_runner:test_unparseable_metafile_on_legacy_to_v1_upgrade (tasks.cephfs.test_volumes.TestMisc) 2024-09-16T15:51:40.314 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-16T15:51:40.352 DEBUG:teuthology.orchestra.run.smithi123:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-16T15:51:40.364 DEBUG:teuthology.orchestra.run.smithi184:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-16T15:51:40.449 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:40.449+0000 7f01d48bc640 1 -- 172.21.15.123:0/503731050 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f01b80052a0 msgr2=0x7f01b800b9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:51:40.449 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:40.449+0000 7f01d48bc640 1 --2- 172.21.15.123:0/503731050 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f01b80052a0 0x7f01b800b9d0 secure :-1 s=READY pgs=1832 cs=0 l=1 rev1=1 crypto rx=0x7f01c800c630 tx=0x7f01c801fa60 comp rx=0 tx=0).stop 2024-09-16T15:51:40.449 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:40.449+0000 7f01d48bc640 1 -- 172.21.15.123:0/503731050 shutdown_connections 2024-09-16T15:51:40.449 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:40.449+0000 7f01d48bc640 1 --2- 172.21.15.123:0/503731050 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f01b80052a0 0x7f01b800b9d0 unknown :-1 s=CLOSED pgs=1832 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:40.449 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:40.449+0000 7f01d48bc640 1 -- 172.21.15.123:0/503731050 >> 172.21.15.123:0/503731050 conn(0x7f01d0074bd0 msgr2=0x7f01d009aa40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:51:40.449 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:40.449+0000 7f01d48bc640 1 -- 172.21.15.123:0/503731050 shutdown_connections 2024-09-16T15:51:40.449 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:40.449+0000 7f01d48bc640 1 -- 172.21.15.123:0/503731050 wait complete. 2024-09-16T15:51:40.450 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:40.449+0000 7f01d48bc640 1 Processor -- start 2024-09-16T15:51:40.450 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:40.449+0000 7f01d48bc640 1 -- start start 2024-09-16T15:51:40.452 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:40.453+0000 7f01d48bc640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f01b80052a0 0x7f01d006f820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:51:40.452 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:40.453+0000 7f01d48bc640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f01d01e5980 0x7f01d015e730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:51:40.452 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:40.453+0000 7f01d48bc640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f01d01e68d0 0x7f01d01e6d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:51:40.452 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:40.453+0000 7f01d48bc640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f01d00d0310 con 0x7f01d01e68d0 2024-09-16T15:51:40.452 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:40.453+0000 7f01d48bc640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f01d00d1ed0 con 0x7f01d01e5980 2024-09-16T15:51:40.452 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:40.453+0000 7f01d48bc640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f01d00d0490 con 0x7f01b80052a0 2024-09-16T15:51:40.452 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:40.453+0000 7f01ced9d640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f01d01e68d0 0x7f01d01e6d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:51:40.453 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:40.453+0000 7f01ced9d640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f01d01e68d0 0x7f01d01e6d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:50318/0 (socket says 172.21.15.123:50318) 2024-09-16T15:51:40.453 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:40.453+0000 7f01ced9d640 1 -- 172.21.15.123:0/1147710810 learned_addr learned my addr 172.21.15.123:0/1147710810 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:51:40.453 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:40.453+0000 7f01ce59c640 1 --2- 172.21.15.123:0/1147710810 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f01b80052a0 0x7f01d006f820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:51:40.453 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:40.453+0000 7f01cdd9b640 1 --2- 172.21.15.123:0/1147710810 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f01d01e5980 0x7f01d015e730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:51:40.453 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:40.453+0000 7f01ced9d640 1 -- 172.21.15.123:0/1147710810 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f01b80052a0 msgr2=0x7f01d006f820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:51:40.453 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:40.453+0000 7f01ced9d640 1 --2- 172.21.15.123:0/1147710810 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f01b80052a0 0x7f01d006f820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:40.453 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:40.453+0000 7f01ced9d640 1 -- 172.21.15.123:0/1147710810 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f01d01e5980 msgr2=0x7f01d015e730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:51:40.453 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:40.453+0000 7f01ced9d640 1 --2- 172.21.15.123:0/1147710810 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f01d01e5980 0x7f01d015e730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:40.453 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:40.453+0000 7f01ced9d640 1 -- 172.21.15.123:0/1147710810 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f01d01e7270 con 0x7f01d01e68d0 2024-09-16T15:51:40.453 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:40.453+0000 7f01ced9d640 1 --2- 172.21.15.123:0/1147710810 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f01d01e68d0 0x7f01d01e6d30 secure :-1 s=READY pgs=1833 cs=0 l=1 rev1=1 crypto rx=0x7f01c8020300 tx=0x7f01c800c320 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:51:40.453 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:40.453+0000 7f01beffd640 1 -- 172.21.15.123:0/1147710810 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01c8003c60 con 0x7f01d01e68d0 2024-09-16T15:51:40.453 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:40.453+0000 7f01beffd640 1 -- 172.21.15.123:0/1147710810 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f01c80a9630 con 0x7f01d01e68d0 2024-09-16T15:51:40.453 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:40.453+0000 7f01d48bc640 1 -- 172.21.15.123:0/1147710810 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f01d01e48e0 con 0x7f01d01e68d0 2024-09-16T15:51:40.454 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:40.453+0000 7f01beffd640 1 -- 172.21.15.123:0/1147710810 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01c80a9a80 con 0x7f01d01e68d0 2024-09-16T15:51:40.454 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:40.453+0000 7f01d48bc640 1 -- 172.21.15.123:0/1147710810 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f01d01e4ea0 con 0x7f01d01e68d0 2024-09-16T15:51:40.454 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:40.453+0000 7f01d48bc640 1 -- 172.21.15.123:0/1147710810 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f01d00d09f0 con 0x7f01d01e68d0 2024-09-16T15:51:40.454 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:40.453+0000 7f01beffd640 1 -- 172.21.15.123:0/1147710810 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 25) ==== 97998+0+0 (secure 0 0 0) 0x7f01c80a9c20 con 0x7f01d01e68d0 2024-09-16T15:51:40.455 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:40.453+0000 7f01beffd640 1 --2- 172.21.15.123:0/1147710810 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f01b00738e0 0x7f01b0075da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:51:40.455 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:40.453+0000 7f01beffd640 1 -- 172.21.15.123:0/1147710810 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(161..161 src has 1..161) ==== 4971+0+0 (secure 0 0 0) 0x7f01c812ff10 con 0x7f01d01e68d0 2024-09-16T15:51:40.455 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:40.453+0000 7f01ce59c640 1 --2- 172.21.15.123:0/1147710810 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f01b00738e0 0x7f01b0075da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:51:40.455 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:40.453+0000 7f01ce59c640 1 --2- 172.21.15.123:0/1147710810 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f01b00738e0 0x7f01b0075da0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f01d00b7ca0 tx=0x7f01c0006b50 comp rx=0 tx=0).ready entity=mgr.4150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:51:40.458 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:40.457+0000 7f01beffd640 1 -- 172.21.15.123:0/1147710810 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f01c8019810 con 0x7f01d01e68d0 2024-09-16T15:51:40.631 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:40.633+0000 7f01d48bc640 1 -- 172.21.15.123:0/1147710810 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f01d015a940 con 0x7f01d01e68d0 2024-09-16T15:51:40.633 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:40.633+0000 7f01beffd640 1 -- 172.21.15.123:0/1147710810 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v161) ==== 74+0+13148 (secure 0 0 0) 0x7f01c80fc9e0 con 0x7f01d01e68d0 2024-09-16T15:51:40.633 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-09-16T15:51:40.633 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":161,"fsid":"56077f89-18c8-486f-ae7e-eecc0bb7822c","created":"2024-09-16T15:42:26.364892+0000","modified":"2024-09-16T15:51:39.289733+0000","last_up_change":"2024-09-16T15:42:38.707234+0000","last_in_change":"2024-09-16T15:42:29.423662+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":25,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-16T15:42:39.287764+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9e7c1955-a607-4671-9be7-f2ef760ceeaa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":155,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6800","nonce":3882044969},{"type":"v1","addr":"172.21.15.123:6801","nonce":3882044969}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6802","nonce":3882044969},{"type":"v1","addr":"172.21.15.123:6803","nonce":3882044969}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6806","nonce":3882044969},{"type":"v1","addr":"172.21.15.123:6807","nonce":3882044969}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6804","nonce":3882044969},{"type":"v1","addr":"172.21.15.123:6805","nonce":3882044969}]},"public_addr":"172.21.15.123:6801/3882044969","cluster_addr":"172.21.15.123:6803/3882044969","heartbeat_back_addr":"172.21.15.123:6807/3882044969","heartbeat_front_addr":"172.21.15.123:6805/3882044969","state":["exists","up"]},{"osd":1,"uuid":"c540cfde-0977-4742-9b17-fd634668e010","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":158,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6816","nonce":4037001850},{"type":"v1","addr":"172.21.15.123:6817","nonce":4037001850}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6818","nonce":4037001850},{"type":"v1","addr":"172.21.15.123:6819","nonce":4037001850}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6822","nonce":4037001850},{"type":"v1","addr":"172.21.15.123:6823","nonce":4037001850}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6820","nonce":4037001850},{"type":"v1","addr":"172.21.15.123:6821","nonce":4037001850}]},"public_addr":"172.21.15.123:6817/4037001850","cluster_addr":"172.21.15.123:6819/4037001850","heartbeat_back_addr":"172.21.15.123:6823/4037001850","heartbeat_front_addr":"172.21.15.123:6821/4037001850","state":["exists","up"]},{"osd":2,"uuid":"9864a496-f169-486b-95c4-1d59162793a7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":158,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6824","nonce":1332143088},{"type":"v1","addr":"172.21.15.123:6825","nonce":1332143088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6826","nonce":1332143088},{"type":"v1","addr":"172.21.15.123:6827","nonce":1332143088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6830","nonce":1332143088},{"type":"v1","addr":"172.21.15.123:6831","nonce":1332143088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6828","nonce":1332143088},{"type":"v1","addr":"172.21.15.123:6829","nonce":1332143088}]},"public_addr":"172.21.15.123:6825/1332143088","cluster_addr":"172.21.15.123:6827/1332143088","heartbeat_back_addr":"172.21.15.123:6831/1332143088","heartbeat_front_addr":"172.21.15.123:6829/1332143088","state":["exists","up"]},{"osd":3,"uuid":"c6d1f817-f074-4325-95e1-578eba13edda","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":155,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6808","nonce":988586998},{"type":"v1","addr":"172.21.15.123:6809","nonce":988586998}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6810","nonce":988586998},{"type":"v1","addr":"172.21.15.123:6811","nonce":988586998}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6814","nonce":988586998},{"type":"v1","addr":"172.21.15.123:6815","nonce":988586998}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6812","nonce":988586998},{"type":"v1","addr":"172.21.15.123:6813","nonce":988586998}]},"public_addr":"172.21.15.123:6809/988586998","cluster_addr":"172.21.15.123:6811/988586998","heartbeat_back_addr":"172.21.15.123:6815/988586998","heartbeat_front_addr":"172.21.15.123:6813/988586998","state":["exists","up"]},{"osd":4,"uuid":"200a8df4-50ef-40a5-b48a-58981269c086","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":155,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6816","nonce":3387813890},{"type":"v1","addr":"172.21.15.184:6817","nonce":3387813890}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6818","nonce":3387813890},{"type":"v1","addr":"172.21.15.184:6819","nonce":3387813890}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6822","nonce":3387813890},{"type":"v1","addr":"172.21.15.184:6823","nonce":3387813890}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6820","nonce":3387813890},{"type":"v1","addr":"172.21.15.184:6821","nonce":3387813890}]},"public_addr":"172.21.15.184:6817/3387813890","cluster_addr":"172.21.15.184:6819/3387813890","heartbeat_back_addr":"172.21.15.184:6823/3387813890","heartbeat_front_addr":"172.21.15.184:6821/3387813890","state":["exists","up"]},{"osd":5,"uuid":"32d5a900-da3d-412c-bc19-13861a2368d8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":155,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6808","nonce":2680916017},{"type":"v1","addr":"172.21.15.184:6809","nonce":2680916017}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6810","nonce":2680916017},{"type":"v1","addr":"172.21.15.184:6811","nonce":2680916017}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6814","nonce":2680916017},{"type":"v1","addr":"172.21.15.184:6815","nonce":2680916017}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6812","nonce":2680916017},{"type":"v1","addr":"172.21.15.184:6813","nonce":2680916017}]},"public_addr":"172.21.15.184:6809/2680916017","cluster_addr":"172.21.15.184:6811/2680916017","heartbeat_back_addr":"172.21.15.184:6815/2680916017","heartbeat_front_addr":"172.21.15.184:6813/2680916017","state":["exists","up"]},{"osd":6,"uuid":"cde6f39e-aa1b-4260-ac19-774ee6aaca72","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":155,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6824","nonce":2183120457},{"type":"v1","addr":"172.21.15.184:6825","nonce":2183120457}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6826","nonce":2183120457},{"type":"v1","addr":"172.21.15.184:6827","nonce":2183120457}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6830","nonce":2183120457},{"type":"v1","addr":"172.21.15.184:6831","nonce":2183120457}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6828","nonce":2183120457},{"type":"v1","addr":"172.21.15.184:6829","nonce":2183120457}]},"public_addr":"172.21.15.184:6825/2183120457","cluster_addr":"172.21.15.184:6827/2183120457","heartbeat_back_addr":"172.21.15.184:6831/2183120457","heartbeat_front_addr":"172.21.15.184:6829/2183120457","state":["exists","up"]},{"osd":7,"uuid":"4f381778-eee1-47c6-804f-df0e6b33e313","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":155,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6800","nonce":4102957285},{"type":"v1","addr":"172.21.15.184:6801","nonce":4102957285}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6802","nonce":4102957285},{"type":"v1","addr":"172.21.15.184:6803","nonce":4102957285}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6806","nonce":4102957285},{"type":"v1","addr":"172.21.15.184:6807","nonce":4102957285}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6804","nonce":4102957285},{"type":"v1","addr":"172.21.15.184:6805","nonce":4102957285}]},"public_addr":"172.21.15.184:6801/4102957285","cluster_addr":"172.21.15.184:6803/4102957285","heartbeat_back_addr":"172.21.15.184:6807/4102957285","heartbeat_front_addr":"172.21.15.184:6805/4102957285","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:35.972210+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.222007+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.198617+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.385663+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.358935+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.156066+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.752973+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:35.682846+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.184:6834/2856066034":"2024-09-17T15:51:36.210086+0000","172.21.15.184:6832/2856066034":"2024-09-17T15:51:36.210086+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":17,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T15:51:40.636 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:40.637+0000 7f01d48bc640 1 -- 172.21.15.123:0/1147710810 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f01b00738e0 msgr2=0x7f01b0075da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:51:40.636 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:40.637+0000 7f01d48bc640 1 --2- 172.21.15.123:0/1147710810 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f01b00738e0 0x7f01b0075da0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f01d00b7ca0 tx=0x7f01c0006b50 comp rx=0 tx=0).stop 2024-09-16T15:51:40.636 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:40.637+0000 7f01d48bc640 1 -- 172.21.15.123:0/1147710810 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f01d01e68d0 msgr2=0x7f01d01e6d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:51:40.636 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:40.637+0000 7f01d48bc640 1 --2- 172.21.15.123:0/1147710810 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f01d01e68d0 0x7f01d01e6d30 secure :-1 s=READY pgs=1833 cs=0 l=1 rev1=1 crypto rx=0x7f01c8020300 tx=0x7f01c800c320 comp rx=0 tx=0).stop 2024-09-16T15:51:40.636 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:40.637+0000 7f01d48bc640 1 -- 172.21.15.123:0/1147710810 shutdown_connections 2024-09-16T15:51:40.636 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:40.637+0000 7f01d48bc640 1 --2- 172.21.15.123:0/1147710810 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f01b00738e0 0x7f01b0075da0 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:40.636 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:40.637+0000 7f01d48bc640 1 --2- 172.21.15.123:0/1147710810 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f01d01e68d0 0x7f01d01e6d30 unknown :-1 s=CLOSED pgs=1833 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:40.636 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:40.637+0000 7f01d48bc640 1 --2- 172.21.15.123:0/1147710810 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f01d01e5980 0x7f01d015e730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:40.636 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:40.637+0000 7f01d48bc640 1 --2- 172.21.15.123:0/1147710810 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f01b80052a0 0x7f01d006f820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:40.636 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:40.637+0000 7f01d48bc640 1 -- 172.21.15.123:0/1147710810 >> 172.21.15.123:0/1147710810 conn(0x7f01d0074bd0 msgr2=0x7f01d0071f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:51:40.636 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:40.637+0000 7f01d48bc640 1 -- 172.21.15.123:0/1147710810 shutdown_connections 2024-09-16T15:51:40.636 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:40.637+0000 7f01d48bc640 1 -- 172.21.15.123:0/1147710810 wait complete. 2024-09-16T15:51:40.653 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-16T15:42:39.287764+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-09-16T15:51:40.653 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-09-16T15:51:40.785 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:40.785+0000 7f683f118640 1 -- 172.21.15.123:0/122124907 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6838071040 msgr2=0x7f6838073600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:51:40.786 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:40.785+0000 7f683f118640 1 --2- 172.21.15.123:0/122124907 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6838071040 0x7f6838073600 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7f682c009840 tx=0x7f682c01c820 comp rx=0 tx=0).stop 2024-09-16T15:51:40.786 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:40.785+0000 7f683f118640 1 -- 172.21.15.123:0/122124907 shutdown_connections 2024-09-16T15:51:40.786 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:40.785+0000 7f683f118640 1 --2- 172.21.15.123:0/122124907 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6838071040 0x7f6838073600 unknown :-1 s=CLOSED pgs=552 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:40.786 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:40.785+0000 7f683f118640 1 --2- 172.21.15.123:0/122124907 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f683815ba60 0x7f683806e510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:40.786 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:40.785+0000 7f683f118640 1 -- 172.21.15.123:0/122124907 >> 172.21.15.123:0/122124907 conn(0x7f68380d0bf0 msgr2=0x7f68380d1000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:51:40.786 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:40.785+0000 7f683f118640 1 -- 172.21.15.123:0/122124907 shutdown_connections 2024-09-16T15:51:40.786 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:40.785+0000 7f683f118640 1 -- 172.21.15.123:0/122124907 wait complete. 2024-09-16T15:51:40.787 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:40.785+0000 7f683f118640 1 Processor -- start 2024-09-16T15:51:40.787 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:40.785+0000 7f683f118640 1 -- start start 2024-09-16T15:51:40.789 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:40.789+0000 7f683f118640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f6838071040 0x7f683815ec10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:51:40.789 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:40.789+0000 7f683f118640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6838073b40 0x7f68381532e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:51:40.789 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:40.789+0000 7f683f118640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f683815ba60 0x7f68381660e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:51:40.789 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:40.789+0000 7f683f118640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f6838168c90 con 0x7f683815ba60 2024-09-16T15:51:40.789 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:40.789+0000 7f683f118640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f6838168b10 con 0x7f6838073b40 2024-09-16T15:51:40.789 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:40.789+0000 7f683f118640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f6838168e10 con 0x7f6838071040 2024-09-16T15:51:40.789 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:40.789+0000 7f683d6b5640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f683815ba60 0x7f68381660e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:51:40.789 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:40.789+0000 7f683d6b5640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f683815ba60 0x7f68381660e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:50326/0 (socket says 172.21.15.123:50326) 2024-09-16T15:51:40.789 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:40.789+0000 7f683d6b5640 1 -- 172.21.15.123:0/361081200 learned_addr learned my addr 172.21.15.123:0/361081200 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:51:40.790 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:40.789+0000 7f6837fff640 1 --2- 172.21.15.123:0/361081200 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6838073b40 0x7f68381532e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:51:40.790 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:40.789+0000 7f683ceb4640 1 --2- 172.21.15.123:0/361081200 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f6838071040 0x7f683815ec10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:51:40.790 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:40.789+0000 7f683d6b5640 1 -- 172.21.15.123:0/361081200 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f6838071040 msgr2=0x7f683815ec10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:51:40.790 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:40.789+0000 7f683d6b5640 1 --2- 172.21.15.123:0/361081200 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f6838071040 0x7f683815ec10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:40.790 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:40.789+0000 7f683d6b5640 1 -- 172.21.15.123:0/361081200 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6838073b40 msgr2=0x7f68381532e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:51:40.790 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:40.789+0000 7f683d6b5640 1 --2- 172.21.15.123:0/361081200 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6838073b40 0x7f68381532e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:40.790 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:40.789+0000 7f683d6b5640 1 -- 172.21.15.123:0/361081200 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f68381f7860 con 0x7f683815ba60 2024-09-16T15:51:40.790 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:40.789+0000 7f683d6b5640 1 --2- 172.21.15.123:0/361081200 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f683815ba60 0x7f68381660e0 secure :-1 s=READY pgs=1834 cs=0 l=1 rev1=1 crypto rx=0x7f683000c7a0 tx=0x7f683000cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:51:40.790 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:40.789+0000 7f68357fa640 1 -- 172.21.15.123:0/361081200 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6830003b80 con 0x7f683815ba60 2024-09-16T15:51:40.790 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:40.789+0000 7f68357fa640 1 -- 172.21.15.123:0/361081200 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f6830003d40 con 0x7f683815ba60 2024-09-16T15:51:40.790 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:40.789+0000 7f683f118640 1 -- 172.21.15.123:0/361081200 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f683815f240 con 0x7f683815ba60 2024-09-16T15:51:40.790 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:40.789+0000 7f68357fa640 1 -- 172.21.15.123:0/361081200 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f683000aab0 con 0x7f683815ba60 2024-09-16T15:51:40.790 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:40.789+0000 7f683f118640 1 -- 172.21.15.123:0/361081200 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f68381f8300 con 0x7f683815ba60 2024-09-16T15:51:40.791 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:40.789+0000 7f683f118640 1 -- 172.21.15.123:0/361081200 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f68381f7650 con 0x7f683815ba60 2024-09-16T15:51:40.791 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:40.789+0000 7f68357fa640 1 -- 172.21.15.123:0/361081200 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 25) ==== 97998+0+0 (secure 0 0 0) 0x7f683000ac50 con 0x7f683815ba60 2024-09-16T15:51:40.791 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:40.789+0000 7f68357fa640 1 --2- 172.21.15.123:0/361081200 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f68240738e0 0x7f6824075da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:51:40.791 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:40.793+0000 7f68357fa640 1 -- 172.21.15.123:0/361081200 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(161..161 src has 1..161) ==== 4971+0+0 (secure 0 0 0) 0x7f6830092670 con 0x7f683815ba60 2024-09-16T15:51:40.791 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:40.793+0000 7f683ceb4640 1 --2- 172.21.15.123:0/361081200 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f68240738e0 0x7f6824075da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:51:40.792 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:40.793+0000 7f683ceb4640 1 --2- 172.21.15.123:0/361081200 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f68240738e0 0x7f6824075da0 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f6828009ad0 tx=0x7f6828009260 comp rx=0 tx=0).ready entity=mgr.4150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:51:40.795 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:40.797+0000 7f68357fa640 1 -- 172.21.15.123:0/361081200 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f683005f150 con 0x7f683815ba60 2024-09-16T15:51:40.966 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:40.965+0000 7f683f118640 1 -- 172.21.15.123:0/361081200 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f6838073600 con 0x7f683815ba60 2024-09-16T15:51:40.967 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:40.969+0000 7f68357fa640 1 -- 172.21.15.123:0/361081200 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v161) ==== 93+0+10 (secure 0 0 0) 0x7f683005ea50 con 0x7f683815ba60 2024-09-16T15:51:40.967 INFO:teuthology.orchestra.run.smithi123.stdout:pg_num: 1 2024-09-16T15:51:40.970 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:40.969+0000 7f683f118640 1 -- 172.21.15.123:0/361081200 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f68240738e0 msgr2=0x7f6824075da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:51:40.970 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:40.969+0000 7f683f118640 1 --2- 172.21.15.123:0/361081200 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f68240738e0 0x7f6824075da0 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f6828009ad0 tx=0x7f6828009260 comp rx=0 tx=0).stop 2024-09-16T15:51:40.970 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:40.969+0000 7f683f118640 1 -- 172.21.15.123:0/361081200 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f683815ba60 msgr2=0x7f68381660e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:51:40.970 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:40.969+0000 7f683f118640 1 --2- 172.21.15.123:0/361081200 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f683815ba60 0x7f68381660e0 secure :-1 s=READY pgs=1834 cs=0 l=1 rev1=1 crypto rx=0x7f683000c7a0 tx=0x7f683000cc60 comp rx=0 tx=0).stop 2024-09-16T15:51:40.970 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:40.969+0000 7f683f118640 1 -- 172.21.15.123:0/361081200 shutdown_connections 2024-09-16T15:51:40.970 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:40.969+0000 7f683f118640 1 --2- 172.21.15.123:0/361081200 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f68240738e0 0x7f6824075da0 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:40.970 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:40.969+0000 7f683f118640 1 --2- 172.21.15.123:0/361081200 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f683815ba60 0x7f68381660e0 unknown :-1 s=CLOSED pgs=1834 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:40.970 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:40.969+0000 7f683f118640 1 --2- 172.21.15.123:0/361081200 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f6838073b40 0x7f68381532e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:40.970 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:40.969+0000 7f683f118640 1 --2- 172.21.15.123:0/361081200 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f6838071040 0x7f683815ec10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:40.971 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:40.969+0000 7f683f118640 1 -- 172.21.15.123:0/361081200 >> 172.21.15.123:0/361081200 conn(0x7f68380d0bf0 msgr2=0x7f68380b0b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:51:40.971 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:40.969+0000 7f683f118640 1 -- 172.21.15.123:0/361081200 shutdown_connections 2024-09-16T15:51:40.971 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:40.969+0000 7f683f118640 1 -- 172.21.15.123:0/361081200 wait complete. 2024-09-16T15:51:40.986 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_volumes.TestMisc.test_unparseable_metafile_on_legacy_to_v1_upgrade' 2024-09-16T15:51:41.109 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:41.109+0000 7f13c2541640 1 -- 172.21.15.123:0/279677923 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f13bc0726c0 msgr2=0x7f13bc0d0630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:51:41.109 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:41.109+0000 7f13c2541640 1 --2- 172.21.15.123:0/279677923 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f13bc0726c0 0x7f13bc0d0630 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7f13ac009840 tx=0x7f13ac01c760 comp rx=0 tx=0).stop 2024-09-16T15:51:41.109 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:41.109+0000 7f13c2541640 1 -- 172.21.15.123:0/279677923 shutdown_connections 2024-09-16T15:51:41.110 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:41.109+0000 7f13c2541640 1 --2- 172.21.15.123:0/279677923 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f13bc074600 0x7f13bc06e750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:41.110 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:41.109+0000 7f13c2541640 1 --2- 172.21.15.123:0/279677923 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f13bc0726c0 0x7f13bc0d0630 unknown :-1 s=CLOSED pgs=554 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:41.110 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:41.109+0000 7f13c2541640 1 -- 172.21.15.123:0/279677923 >> 172.21.15.123:0/279677923 conn(0x7f13bc0cfb10 msgr2=0x7f13bc0cff20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:51:41.110 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:41.109+0000 7f13c2541640 1 -- 172.21.15.123:0/279677923 shutdown_connections 2024-09-16T15:51:41.110 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:41.109+0000 7f13c2541640 1 -- 172.21.15.123:0/279677923 wait complete. 2024-09-16T15:51:41.110 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:41.109+0000 7f13c2541640 1 Processor -- start 2024-09-16T15:51:41.110 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:41.109+0000 7f13c2541640 1 -- start start 2024-09-16T15:51:41.112 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:41.113+0000 7f13c2541640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f13bc0726c0 0x7f13bc14fb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:51:41.112 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:41.113+0000 7f13c2541640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f13bc074600 0x7f13bc153030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:51:41.112 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:41.113+0000 7f13c2541640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f13bc15bca0 0x7f13bc0c41c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:51:41.112 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:41.113+0000 7f13c2541640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f13bc168bd0 con 0x7f13bc15bca0 2024-09-16T15:51:41.112 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:41.113+0000 7f13c2541640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f13bc168a50 con 0x7f13bc074600 2024-09-16T15:51:41.113 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:41.113+0000 7f13c2541640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f13bc168d50 con 0x7f13bc0726c0 2024-09-16T15:51:41.113 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:41.113+0000 7f13c0ade640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f13bc15bca0 0x7f13bc0c41c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:51:41.113 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:41.113+0000 7f13c0ade640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f13bc15bca0 0x7f13bc0c41c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:50328/0 (socket says 172.21.15.123:50328) 2024-09-16T15:51:41.113 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:41.113+0000 7f13c0ade640 1 -- 172.21.15.123:0/732777434 learned_addr learned my addr 172.21.15.123:0/732777434 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:51:41.113 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:41.113+0000 7f13bbfff640 1 --2- 172.21.15.123:0/732777434 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f13bc0726c0 0x7f13bc14fb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:51:41.113 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:41.113+0000 7f13bb7fe640 1 --2- 172.21.15.123:0/732777434 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f13bc074600 0x7f13bc153030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:51:41.113 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:41.113+0000 7f13c0ade640 1 -- 172.21.15.123:0/732777434 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f13bc0726c0 msgr2=0x7f13bc14fb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:51:41.113 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:41.113+0000 7f13c0ade640 1 --2- 172.21.15.123:0/732777434 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f13bc0726c0 0x7f13bc14fb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:41.113 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:41.113+0000 7f13c0ade640 1 -- 172.21.15.123:0/732777434 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f13bc074600 msgr2=0x7f13bc153030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:51:41.113 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:41.113+0000 7f13c0ade640 1 --2- 172.21.15.123:0/732777434 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f13bc074600 0x7f13bc153030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:41.113 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:41.113+0000 7f13c0ade640 1 -- 172.21.15.123:0/732777434 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f13bc1f9b80 con 0x7f13bc15bca0 2024-09-16T15:51:41.113 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:41.113+0000 7f13c0ade640 1 --2- 172.21.15.123:0/732777434 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f13bc15bca0 0x7f13bc0c41c0 secure :-1 s=READY pgs=1835 cs=0 l=1 rev1=1 crypto rx=0x7f13b4009fb0 tx=0x7f13b400c800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:51:41.114 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:41.113+0000 7f13b8ff9640 1 -- 172.21.15.123:0/732777434 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13b4003980 con 0x7f13bc15bca0 2024-09-16T15:51:41.114 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:41.113+0000 7f13b8ff9640 1 -- 172.21.15.123:0/732777434 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f13b4003c30 con 0x7f13bc15bca0 2024-09-16T15:51:41.114 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:41.113+0000 7f13b8ff9640 1 -- 172.21.15.123:0/732777434 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13b4004280 con 0x7f13bc15bca0 2024-09-16T15:51:41.114 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:41.113+0000 7f13c2541640 1 -- 172.21.15.123:0/732777434 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f13bc1f8dc0 con 0x7f13bc15bca0 2024-09-16T15:51:41.114 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:41.113+0000 7f13c2541640 1 -- 172.21.15.123:0/732777434 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f13bc1f9380 con 0x7f13bc15bca0 2024-09-16T15:51:41.115 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:41.113+0000 7f13c2541640 1 -- 172.21.15.123:0/732777434 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f13bc0d0630 con 0x7f13bc15bca0 2024-09-16T15:51:41.115 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:41.113+0000 7f13b8ff9640 1 -- 172.21.15.123:0/732777434 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 25) ==== 97998+0+0 (secure 0 0 0) 0x7f13b401d020 con 0x7f13bc15bca0 2024-09-16T15:51:41.115 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:41.113+0000 7f13b8ff9640 1 --2- 172.21.15.123:0/732777434 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f13a8073910 0x7f13a8075dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:51:41.115 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:41.113+0000 7f13b8ff9640 1 -- 172.21.15.123:0/732777434 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(161..161 src has 1..161) ==== 4971+0+0 (secure 0 0 0) 0x7f13b4093400 con 0x7f13bc15bca0 2024-09-16T15:51:41.115 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:41.113+0000 7f13bbfff640 1 --2- 172.21.15.123:0/732777434 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f13a8073910 0x7f13a8075dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:51:41.115 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:41.117+0000 7f13bbfff640 1 --2- 172.21.15.123:0/732777434 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f13a8073910 0x7f13a8075dd0 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f13b0001650 tx=0x7f13b0009040 comp rx=0 tx=0).ready entity=mgr.4150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:51:41.118 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:41.117+0000 7f13b8ff9640 1 -- 172.21.15.123:0/732777434 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f13b4060100 con 0x7f13bc15bca0 2024-09-16T15:51:41.347 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:41.349+0000 7f13c2541640 1 -- 172.21.15.123:0/732777434 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_volumes.TestMisc.test_unparseable_metafile_on_legacy_to_v1_upgrade"]} v 0) -- 0x7f13bc15f630 con 0x7f13bc15bca0 2024-09-16T15:51:42.304 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:42.305+0000 7f13b8ff9640 1 -- 172.21.15.123:0/732777434 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_volumes.TestMisc.test_unparseable_metafile_on_legacy_to_v1_upgrade"]}]=0 v466) ==== 166+0+0 (secure 0 0 0) 0x7f13b405fa00 con 0x7f13bc15bca0 2024-09-16T15:51:42.308 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:42.309+0000 7f13c2541640 1 -- 172.21.15.123:0/732777434 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f13a8073910 msgr2=0x7f13a8075dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:51:42.308 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:42.309+0000 7f13c2541640 1 --2- 172.21.15.123:0/732777434 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f13a8073910 0x7f13a8075dd0 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f13b0001650 tx=0x7f13b0009040 comp rx=0 tx=0).stop 2024-09-16T15:51:42.308 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:42.309+0000 7f13c2541640 1 -- 172.21.15.123:0/732777434 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f13bc15bca0 msgr2=0x7f13bc0c41c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:51:42.309 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:42.309+0000 7f13c2541640 1 --2- 172.21.15.123:0/732777434 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f13bc15bca0 0x7f13bc0c41c0 secure :-1 s=READY pgs=1835 cs=0 l=1 rev1=1 crypto rx=0x7f13b4009fb0 tx=0x7f13b400c800 comp rx=0 tx=0).stop 2024-09-16T15:51:42.309 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:42.309+0000 7f13c2541640 1 -- 172.21.15.123:0/732777434 shutdown_connections 2024-09-16T15:51:42.309 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:42.309+0000 7f13c2541640 1 --2- 172.21.15.123:0/732777434 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f13a8073910 0x7f13a8075dd0 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:42.309 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:42.309+0000 7f13c2541640 1 --2- 172.21.15.123:0/732777434 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f13bc15bca0 0x7f13bc0c41c0 unknown :-1 s=CLOSED pgs=1835 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:42.309 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:42.309+0000 7f13c2541640 1 --2- 172.21.15.123:0/732777434 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f13bc074600 0x7f13bc153030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:42.309 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:42.309+0000 7f13c2541640 1 --2- 172.21.15.123:0/732777434 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f13bc0726c0 0x7f13bc14fb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:42.309 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:42.309+0000 7f13c2541640 1 -- 172.21.15.123:0/732777434 >> 172.21.15.123:0/732777434 conn(0x7f13bc0cfb10 msgr2=0x7f13bc1572d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:51:42.309 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:42.309+0000 7f13c2541640 1 -- 172.21.15.123:0/732777434 shutdown_connections 2024-09-16T15:51:42.309 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:42.309+0000 7f13c2541640 1 -- 172.21.15.123:0/732777434 wait complete. 2024-09-16T15:51:42.332 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config dump 2024-09-16T15:51:42.460 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:42.461+0000 7f46345b0640 1 -- 172.21.15.123:0/658262233 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f462c074430 msgr2=0x7f462c0ca160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:51:42.460 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:42.461+0000 7f46345b0640 1 --2- 172.21.15.123:0/658262233 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f462c074430 0x7f462c0ca160 secure :-1 s=READY pgs=1836 cs=0 l=1 rev1=1 crypto rx=0x7f462c069550 tx=0x7f462401ca00 comp rx=0 tx=0).stop 2024-09-16T15:51:42.460 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:42.461+0000 7f46345b0640 1 -- 172.21.15.123:0/658262233 shutdown_connections 2024-09-16T15:51:42.460 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:42.461+0000 7f46345b0640 1 --2- 172.21.15.123:0/658262233 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f462c074430 0x7f462c0ca160 unknown :-1 s=CLOSED pgs=1836 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:42.460 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:42.461+0000 7f46345b0640 1 --2- 172.21.15.123:0/658262233 >> v2:172.21.15.184:3300/0 conn(0x7f462c0726c0 0x7f462c073ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:42.460 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:42.461+0000 7f46345b0640 1 --2- 172.21.15.123:0/658262233 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f462c15bca0 0x7f462c0d0630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:42.460 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:42.461+0000 7f46345b0640 1 -- 172.21.15.123:0/658262233 >> 172.21.15.123:0/658262233 conn(0x7f462c0cfb10 msgr2=0x7f462c0cff20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:51:42.460 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:42.461+0000 7f46345b0640 1 -- 172.21.15.123:0/658262233 shutdown_connections 2024-09-16T15:51:42.460 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:42.461+0000 7f46345b0640 1 -- 172.21.15.123:0/658262233 wait complete. 2024-09-16T15:51:42.461 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:42.461+0000 7f46345b0640 1 Processor -- start 2024-09-16T15:51:42.461 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:42.461+0000 7f46345b0640 1 -- start start 2024-09-16T15:51:42.463 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:42.461+0000 7f46345b0640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f462c0726c0 0x7f462c156350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:51:42.463 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:42.461+0000 7f46345b0640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f462c074430 0x7f462c155c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:51:42.463 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:42.461+0000 7f46345b0640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f462c15bca0 0x7f462c15e6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:51:42.463 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:42.461+0000 7f46345b0640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f462c06ebc0 con 0x7f462c0726c0 2024-09-16T15:51:42.463 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:42.461+0000 7f46345b0640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f462c06ea40 con 0x7f462c15bca0 2024-09-16T15:51:42.463 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:42.461+0000 7f46345b0640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f462c06ed40 con 0x7f462c074430 2024-09-16T15:51:42.463 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:42.461+0000 7f463234c640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f462c0726c0 0x7f462c156350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:51:42.463 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:42.461+0000 7f463234c640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f462c0726c0 0x7f462c156350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:38386/0 (socket says 172.21.15.123:38386) 2024-09-16T15:51:42.463 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:42.461+0000 7f463234c640 1 -- 172.21.15.123:0/3934013623 learned_addr learned my addr 172.21.15.123:0/3934013623 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:51:42.463 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:42.461+0000 7f4632b4d640 1 --2- 172.21.15.123:0/3934013623 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f462c15bca0 0x7f462c15e6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:51:42.464 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:42.461+0000 7f4631b4b640 1 --2- 172.21.15.123:0/3934013623 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f462c074430 0x7f462c155c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:51:42.464 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:42.461+0000 7f463234c640 1 -- 172.21.15.123:0/3934013623 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f462c074430 msgr2=0x7f462c155c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:51:42.464 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:42.461+0000 7f463234c640 1 --2- 172.21.15.123:0/3934013623 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f462c074430 0x7f462c155c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:42.464 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:42.461+0000 7f463234c640 1 -- 172.21.15.123:0/3934013623 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f462c15bca0 msgr2=0x7f462c15e6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:51:42.464 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:42.461+0000 7f463234c640 1 --2- 172.21.15.123:0/3934013623 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f462c15bca0 0x7f462c15e6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:42.464 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:42.461+0000 7f463234c640 1 -- 172.21.15.123:0/3934013623 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f462c156160 con 0x7f462c0726c0 2024-09-16T15:51:42.464 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:42.461+0000 7f4632b4d640 1 --2- 172.21.15.123:0/3934013623 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f462c15bca0 0x7f462c15e6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T15:51:42.464 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:42.461+0000 7f4631b4b640 1 --2- 172.21.15.123:0/3934013623 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f462c074430 0x7f462c155c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T15:51:42.464 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:42.461+0000 7f463234c640 1 --2- 172.21.15.123:0/3934013623 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f462c0726c0 0x7f462c156350 secure :-1 s=READY pgs=1837 cs=0 l=1 rev1=1 crypto rx=0x7f461c00c940 tx=0x7f461c00ce00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:51:42.464 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:42.461+0000 7f4622ffd640 1 -- 172.21.15.123:0/3934013623 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f461c004220 con 0x7f462c0726c0 2024-09-16T15:51:42.464 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:42.461+0000 7f4622ffd640 1 -- 172.21.15.123:0/3934013623 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f461c0079b0 con 0x7f462c0726c0 2024-09-16T15:51:42.464 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:42.461+0000 7f4622ffd640 1 -- 172.21.15.123:0/3934013623 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f461c0044b0 con 0x7f462c0726c0 2024-09-16T15:51:42.464 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:42.465+0000 7f46345b0640 1 -- 172.21.15.123:0/3934013623 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f462c159f20 con 0x7f462c0726c0 2024-09-16T15:51:42.465 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:42.465+0000 7f46345b0640 1 -- 172.21.15.123:0/3934013623 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f462c0cd880 con 0x7f462c0726c0 2024-09-16T15:51:42.465 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:42.465+0000 7f46345b0640 1 -- 172.21.15.123:0/3934013623 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f462c15a440 con 0x7f462c0726c0 2024-09-16T15:51:42.465 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:42.465+0000 7f4622ffd640 1 -- 172.21.15.123:0/3934013623 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 25) ==== 97998+0+0 (secure 0 0 0) 0x7f461c003600 con 0x7f462c0726c0 2024-09-16T15:51:42.466 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:42.465+0000 7f4622ffd640 1 --2- 172.21.15.123:0/3934013623 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f46100738e0 0x7f4610075da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:51:42.466 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:42.465+0000 7f4631b4b640 1 --2- 172.21.15.123:0/3934013623 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f46100738e0 0x7f4610075da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:51:42.466 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:42.465+0000 7f4622ffd640 1 -- 172.21.15.123:0/3934013623 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(161..161 src has 1..161) ==== 4971+0+0 (secure 0 0 0) 0x7f461c092a40 con 0x7f462c0726c0 2024-09-16T15:51:42.466 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:42.465+0000 7f4631b4b640 1 --2- 172.21.15.123:0/3934013623 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f46100738e0 0x7f4610075da0 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f462c165940 tx=0x7f4628005d30 comp rx=0 tx=0).ready entity=mgr.4150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:51:42.469 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:42.469+0000 7f4622ffd640 1 -- 172.21.15.123:0/3934013623 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f462c15a440 con 0x7f462c0726c0 2024-09-16T15:51:42.646 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:42.645+0000 7f46345b0640 1 -- 172.21.15.123:0/3934013623 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "config dump"} v 0) -- 0x7f462c0cda80 con 0x7f462c0726c0 2024-09-16T15:51:42.646 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:42.649+0000 7f4622ffd640 1 -- 172.21.15.123:0/3934013623 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "config dump"}]=0 v47) ==== 59+0+1998 (secure 0 0 0) 0x7f461c014030 con 0x7f462c0726c0 2024-09-16T15:51:42.647 INFO:teuthology.orchestra.run.smithi123.stdout:WHO MASK LEVEL OPTION VALUE RO 2024-09-16T15:51:42.647 INFO:teuthology.orchestra.run.smithi123.stdout:mon advanced mon_op_complaint_time 120 2024-09-16T15:51:42.647 INFO:teuthology.orchestra.run.smithi123.stdout:mon dev mon_warn_on_pool_no_app false 2024-09-16T15:51:42.647 INFO:teuthology.orchestra.run.smithi123.stdout:osd advanced osd_op_complaint_time 180.000000 2024-09-16T15:51:42.647 INFO:teuthology.orchestra.run.smithi123.stdout:osd.0 basic osd_mclock_max_capacity_iops_ssd 4805.300677 2024-09-16T15:51:42.647 INFO:teuthology.orchestra.run.smithi123.stdout:osd.1 basic osd_mclock_max_capacity_iops_ssd 4806.240607 2024-09-16T15:51:42.647 INFO:teuthology.orchestra.run.smithi123.stdout:osd.2 basic osd_mclock_max_capacity_iops_ssd 4046.441686 2024-09-16T15:51:42.647 INFO:teuthology.orchestra.run.smithi123.stdout:osd.3 basic osd_mclock_max_capacity_iops_ssd 6882.576123 2024-09-16T15:51:42.647 INFO:teuthology.orchestra.run.smithi123.stdout:osd.4 basic osd_mclock_max_capacity_iops_ssd 6108.444971 2024-09-16T15:51:42.647 INFO:teuthology.orchestra.run.smithi123.stdout:osd.5 basic osd_mclock_max_capacity_iops_ssd 4995.096306 2024-09-16T15:51:42.648 INFO:teuthology.orchestra.run.smithi123.stdout:osd.6 basic osd_mclock_max_capacity_iops_ssd 6237.822095 2024-09-16T15:51:42.648 INFO:teuthology.orchestra.run.smithi123.stdout:osd.7 basic osd_mclock_max_capacity_iops_ssd 5202.385138 2024-09-16T15:51:42.648 INFO:teuthology.orchestra.run.smithi123.stdout:mds advanced debug_mds 20/20 2024-09-16T15:51:42.648 INFO:teuthology.orchestra.run.smithi123.stdout:mds advanced debug_mds_balancer 20/20 2024-09-16T15:51:42.648 INFO:teuthology.orchestra.run.smithi123.stdout:mds advanced debug_ms 2/2 2024-09-16T15:51:42.648 INFO:teuthology.orchestra.run.smithi123.stdout:mds dev mds_debug_frag true 2024-09-16T15:51:42.648 INFO:teuthology.orchestra.run.smithi123.stdout:mds dev mds_debug_scatterstat true 2024-09-16T15:51:42.648 INFO:teuthology.orchestra.run.smithi123.stdout:mds advanced mds_op_complaint_time 180.000000 2024-09-16T15:51:42.648 INFO:teuthology.orchestra.run.smithi123.stdout:mds dev mds_verify_scatter true 2024-09-16T15:51:42.648 INFO:teuthology.orchestra.run.smithi123.stdout:mds advanced osd_op_complaint_time 180.000000 2024-09-16T15:51:42.648 INFO:teuthology.orchestra.run.smithi123.stdout:mds advanced rados_mon_op_timeout 900 2024-09-16T15:51:42.648 INFO:teuthology.orchestra.run.smithi123.stdout:mds advanced rados_osd_op_timeout 900 2024-09-16T15:51:42.649 INFO:teuthology.orchestra.run.smithi123.stdout:client advanced client_mount_timeout 600 2024-09-16T15:51:42.649 INFO:teuthology.orchestra.run.smithi123.stdout:client advanced debug_client 20/20 2024-09-16T15:51:42.649 INFO:teuthology.orchestra.run.smithi123.stdout:client advanced debug_ms 1/1 2024-09-16T15:51:42.649 INFO:teuthology.orchestra.run.smithi123.stdout:client advanced rados_mon_op_timeout 900 2024-09-16T15:51:42.649 INFO:teuthology.orchestra.run.smithi123.stdout:client advanced rados_osd_op_timeout 900 2024-09-16T15:51:42.650 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:42.649+0000 7f46345b0640 1 -- 172.21.15.123:0/3934013623 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f46100738e0 msgr2=0x7f4610075da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:51:42.650 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:42.649+0000 7f46345b0640 1 --2- 172.21.15.123:0/3934013623 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f46100738e0 0x7f4610075da0 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f462c165940 tx=0x7f4628005d30 comp rx=0 tx=0).stop 2024-09-16T15:51:42.650 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:42.649+0000 7f46345b0640 1 -- 172.21.15.123:0/3934013623 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f462c0726c0 msgr2=0x7f462c156350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:51:42.650 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:42.649+0000 7f46345b0640 1 --2- 172.21.15.123:0/3934013623 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f462c0726c0 0x7f462c156350 secure :-1 s=READY pgs=1837 cs=0 l=1 rev1=1 crypto rx=0x7f461c00c940 tx=0x7f461c00ce00 comp rx=0 tx=0).stop 2024-09-16T15:51:42.650 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:42.649+0000 7f46345b0640 1 -- 172.21.15.123:0/3934013623 shutdown_connections 2024-09-16T15:51:42.650 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:42.649+0000 7f46345b0640 1 --2- 172.21.15.123:0/3934013623 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f46100738e0 0x7f4610075da0 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:42.650 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:42.649+0000 7f46345b0640 1 --2- 172.21.15.123:0/3934013623 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f462c15bca0 0x7f462c15e6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:42.650 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:42.649+0000 7f46345b0640 1 --2- 172.21.15.123:0/3934013623 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f462c074430 0x7f462c155c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:42.650 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:42.649+0000 7f46345b0640 1 --2- 172.21.15.123:0/3934013623 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f462c0726c0 0x7f462c156350 unknown :-1 s=CLOSED pgs=1837 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:42.650 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:42.649+0000 7f46345b0640 1 -- 172.21.15.123:0/3934013623 >> 172.21.15.123:0/3934013623 conn(0x7f462c0cfb10 msgr2=0x7f462c0ccdb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:51:42.650 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:42.649+0000 7f46345b0640 1 -- 172.21.15.123:0/3934013623 shutdown_connections 2024-09-16T15:51:42.651 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:42.649+0000 7f46345b0640 1 -- 172.21.15.123:0/3934013623 wait complete. 2024-09-16T15:51:42.666 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-09-16T15:51:42.824 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:42.825+0000 7f9872786640 1 -- 172.21.15.123:0/1524607019 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f986c15a170 msgr2=0x7f986c072f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:51:42.824 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:42.825+0000 7f9872786640 1 --2- 172.21.15.123:0/1524607019 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f986c15a170 0x7f986c072f30 secure :-1 s=READY pgs=1838 cs=0 l=1 rev1=1 crypto rx=0x7f985c009900 tx=0x7f985c01c850 comp rx=0 tx=0).stop 2024-09-16T15:51:42.824 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:42.825+0000 7f9872786640 1 -- 172.21.15.123:0/1524607019 shutdown_connections 2024-09-16T15:51:42.824 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:42.825+0000 7f9872786640 1 --2- 172.21.15.123:0/1524607019 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f986c15ddc0 0x7f986c15e250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:42.824 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:42.825+0000 7f9872786640 1 --2- 172.21.15.123:0/1524607019 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f986c15a170 0x7f986c072f30 unknown :-1 s=CLOSED pgs=1838 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:42.824 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:42.825+0000 7f9872786640 1 -- 172.21.15.123:0/1524607019 >> 172.21.15.123:0/1524607019 conn(0x7f986c0b1710 msgr2=0x7f986c0b1b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:51:42.824 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:42.825+0000 7f9872786640 1 -- 172.21.15.123:0/1524607019 shutdown_connections 2024-09-16T15:51:42.824 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:42.825+0000 7f9872786640 1 -- 172.21.15.123:0/1524607019 wait complete. 2024-09-16T15:51:42.825 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:42.825+0000 7f9872786640 1 Processor -- start 2024-09-16T15:51:42.825 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:42.825+0000 7f9872786640 1 -- start start 2024-09-16T15:51:42.826 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:42.825+0000 7f9872786640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f986c0ca650 0x7f986c0b1000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:51:42.827 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:42.825+0000 7f9872786640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f986c15a170 0x7f986c146af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:51:42.827 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:42.825+0000 7f9872786640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f986c15ddc0 0x7f986c0b31b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:51:42.827 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:42.825+0000 7f9872786640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f986c166f50 con 0x7f986c0ca650 2024-09-16T15:51:42.827 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:42.825+0000 7f9872786640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f986c06ee80 con 0x7f986c15a170 2024-09-16T15:51:42.827 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:42.825+0000 7f9872786640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f986c1670d0 con 0x7f986c15ddc0 2024-09-16T15:51:42.827 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:42.825+0000 7f986bfff640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f986c0ca650 0x7f986c0b1000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:51:42.827 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:42.825+0000 7f986bfff640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f986c0ca650 0x7f986c0b1000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:38396/0 (socket says 172.21.15.123:38396) 2024-09-16T15:51:42.827 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:42.825+0000 7f986bfff640 1 -- 172.21.15.123:0/3846987129 learned_addr learned my addr 172.21.15.123:0/3846987129 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:51:42.827 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:42.825+0000 7f9870d23640 1 --2- 172.21.15.123:0/3846987129 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f986c15ddc0 0x7f986c0b31b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:51:42.827 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:42.825+0000 7f986bfff640 1 -- 172.21.15.123:0/3846987129 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f986c15ddc0 msgr2=0x7f986c0b31b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:51:42.827 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:42.825+0000 7f986bfff640 1 --2- 172.21.15.123:0/3846987129 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f986c15ddc0 0x7f986c0b31b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:42.827 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:42.825+0000 7f986bfff640 1 -- 172.21.15.123:0/3846987129 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f986c15a170 msgr2=0x7f986c146af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:51:42.828 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:42.825+0000 7f9863fff640 1 --2- 172.21.15.123:0/3846987129 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f986c15a170 0x7f986c146af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:51:42.828 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:42.825+0000 7f986bfff640 1 --2- 172.21.15.123:0/3846987129 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f986c15a170 0x7f986c146af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:42.828 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:42.825+0000 7f986bfff640 1 -- 172.21.15.123:0/3846987129 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f986c147500 con 0x7f986c0ca650 2024-09-16T15:51:42.828 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:42.825+0000 7f986bfff640 1 --2- 172.21.15.123:0/3846987129 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f986c0ca650 0x7f986c0b1000 secure :-1 s=READY pgs=1839 cs=0 l=1 rev1=1 crypto rx=0x7f985c0b0040 tx=0x7f985c0a5690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:51:42.828 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:42.825+0000 7f98697fa640 1 -- 172.21.15.123:0/3846987129 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f985c0b2040 con 0x7f986c0ca650 2024-09-16T15:51:42.828 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:42.825+0000 7f98697fa640 1 -- 172.21.15.123:0/3846987129 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f985c004430 con 0x7f986c0ca650 2024-09-16T15:51:42.828 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:42.825+0000 7f9872786640 1 -- 172.21.15.123:0/3846987129 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f986c1477e0 con 0x7f986c0ca650 2024-09-16T15:51:42.828 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:42.825+0000 7f98697fa640 1 -- 172.21.15.123:0/3846987129 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f985c003860 con 0x7f986c0ca650 2024-09-16T15:51:42.828 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:42.825+0000 7f9872786640 1 -- 172.21.15.123:0/3846987129 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f986c158230 con 0x7f986c0ca650 2024-09-16T15:51:42.829 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:42.829+0000 7f9872786640 1 -- 172.21.15.123:0/3846987129 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f986c147a20 con 0x7f986c0ca650 2024-09-16T15:51:42.829 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:42.829+0000 7f98697fa640 1 -- 172.21.15.123:0/3846987129 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 25) ==== 97998+0+0 (secure 0 0 0) 0x7f985c0bf060 con 0x7f986c0ca650 2024-09-16T15:51:42.829 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:42.829+0000 7f98697fa640 1 --2- 172.21.15.123:0/3846987129 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f984c0738e0 0x7f984c075da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:51:42.829 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:42.829+0000 7f98697fa640 1 -- 172.21.15.123:0/3846987129 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(161..161 src has 1..161) ==== 4971+0+0 (secure 0 0 0) 0x7f985c12c8f0 con 0x7f986c0ca650 2024-09-16T15:51:42.829 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:42.829+0000 7f9863fff640 1 --2- 172.21.15.123:0/3846987129 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f984c0738e0 0x7f984c075da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:51:42.830 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:42.829+0000 7f9863fff640 1 --2- 172.21.15.123:0/3846987129 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f984c0738e0 0x7f984c075da0 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f986c163d60 tx=0x7f9854009040 comp rx=0 tx=0).ready entity=mgr.4150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:51:42.832 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:42.833+0000 7f98697fa640 1 -- 172.21.15.123:0/3846987129 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f986c147a20 con 0x7f986c0ca650 2024-09-16T15:51:43.003 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:43.005+0000 7f9872786640 1 -- 172.21.15.123:0/3846987129 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) -- 0x7f986c072f30 con 0x7f986c0ca650 2024-09-16T15:51:43.006 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:43.005+0000 7f98697fa640 1 -- 172.21.15.123:0/3846987129 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v48)=0 v48) ==== 122+0+0 (secure 0 0 0) 0x7f985c0164d0 con 0x7f986c0ca650 2024-09-16T15:51:43.009 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:43.009+0000 7f9872786640 1 -- 172.21.15.123:0/3846987129 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f984c0738e0 msgr2=0x7f984c075da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:51:43.009 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:43.009+0000 7f9872786640 1 --2- 172.21.15.123:0/3846987129 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f984c0738e0 0x7f984c075da0 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f986c163d60 tx=0x7f9854009040 comp rx=0 tx=0).stop 2024-09-16T15:51:43.009 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:43.009+0000 7f9872786640 1 -- 172.21.15.123:0/3846987129 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f986c0ca650 msgr2=0x7f986c0b1000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:51:43.009 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:43.009+0000 7f9872786640 1 --2- 172.21.15.123:0/3846987129 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f986c0ca650 0x7f986c0b1000 secure :-1 s=READY pgs=1839 cs=0 l=1 rev1=1 crypto rx=0x7f985c0b0040 tx=0x7f985c0a5690 comp rx=0 tx=0).stop 2024-09-16T15:51:43.009 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:43.009+0000 7f9872786640 1 -- 172.21.15.123:0/3846987129 shutdown_connections 2024-09-16T15:51:43.010 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:43.009+0000 7f9872786640 1 --2- 172.21.15.123:0/3846987129 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f984c0738e0 0x7f984c075da0 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:43.010 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:43.009+0000 7f9872786640 1 --2- 172.21.15.123:0/3846987129 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f986c15ddc0 0x7f986c0b31b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:43.010 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:43.009+0000 7f9872786640 1 --2- 172.21.15.123:0/3846987129 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f986c15a170 0x7f986c146af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:43.010 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:43.009+0000 7f9872786640 1 --2- 172.21.15.123:0/3846987129 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f986c0ca650 0x7f986c0b1000 unknown :-1 s=CLOSED pgs=1839 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:43.010 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:43.009+0000 7f9872786640 1 -- 172.21.15.123:0/3846987129 >> 172.21.15.123:0/3846987129 conn(0x7f986c0b1710 msgr2=0x7f986c06f870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:51:43.010 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:43.009+0000 7f9872786640 1 -- 172.21.15.123:0/3846987129 shutdown_connections 2024-09-16T15:51:43.010 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:43.009+0000 7f9872786640 1 -- 172.21.15.123:0/3846987129 wait complete. 2024-09-16T15:51:43.024 INFO:teuthology.nuke:Clearing teuthology firewall rules... 2024-09-16T15:51:43.025 DEBUG:teuthology.orchestra.run.smithi123:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-09-16T15:51:43.120 DEBUG:teuthology.orchestra.run.smithi184:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-09-16T15:51:43.209 INFO:teuthology.nuke:Cleared teuthology firewall rules. 2024-09-16T15:51:43.209 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2024-09-16T15:51:43.209 DEBUG:teuthology.orchestra.run.smithi123:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-16T15:51:43.215 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16336468k,nr_inodes=4084117,mode=755,inode64 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3274684k,mode=755,inode64 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev,inode64 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=17082 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3274680k,nr_inodes=818670,mode=700,uid=1000,gid=1271,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3274684k,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-09-16T15:51:43.215 DEBUG:teuthology.orchestra.run.smithi184:> set -ex 2024-09-16T15:51:43.215 DEBUG:teuthology.orchestra.run.smithi184:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-16T15:51:43.221 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16336488k,nr_inodes=4084122,mode=755,inode64 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3274688k,mode=755,inode64 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev,inode64 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=17834 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3274684k,nr_inodes=818671,mode=700,uid=1000,gid=1271,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3274688k,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-09-16T15:51:43.221 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2024-09-16T15:51:43.221 DEBUG:teuthology.orchestra.run.smithi123:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-16T15:51:43.263 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16336468k,nr_inodes=4084117,mode=755,inode64 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3274684k,mode=755,inode64 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev,inode64 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=17082 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3274680k,nr_inodes=818670,mode=700,uid=1000,gid=1271,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3274684k,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-09-16T15:51:43.263 DEBUG:teuthology.orchestra.run.smithi184:> set -ex 2024-09-16T15:51:43.263 DEBUG:teuthology.orchestra.run.smithi184:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-16T15:51:43.269 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16336488k,nr_inodes=4084122,mode=755,inode64 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3274688k,mode=755,inode64 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev,inode64 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=17834 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3274684k,nr_inodes=818671,mode=700,uid=1000,gid=1271,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3274688k,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-09-16T15:51:43.269 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-09-16T15:51:43.269 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-16T15:51:43.436 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:43.437+0000 7eff96659640 1 -- 172.21.15.123:0/557430311 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7eff9015e160 msgr2=0x7eff90073f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:51:43.436 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:43.437+0000 7eff96659640 1 --2- 172.21.15.123:0/557430311 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7eff9015e160 0x7eff90073f30 secure :-1 s=READY pgs=1841 cs=0 l=1 rev1=1 crypto rx=0x7eff84009900 tx=0x7eff8401c850 comp rx=0 tx=0).stop 2024-09-16T15:51:43.436 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:43.437+0000 7eff96659640 1 -- 172.21.15.123:0/557430311 shutdown_connections 2024-09-16T15:51:43.436 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:43.437+0000 7eff96659640 1 --2- 172.21.15.123:0/557430311 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7eff90074470 0x7eff900cd6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:43.436 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:43.437+0000 7eff96659640 1 --2- 172.21.15.123:0/557430311 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7eff9015e160 0x7eff90073f30 unknown :-1 s=CLOSED pgs=1841 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:43.436 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:43.437+0000 7eff96659640 1 -- 172.21.15.123:0/557430311 >> 172.21.15.123:0/557430311 conn(0x7eff900cfb10 msgr2=0x7eff900cff20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:51:43.436 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:43.437+0000 7eff96659640 1 -- 172.21.15.123:0/557430311 shutdown_connections 2024-09-16T15:51:43.436 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:43.437+0000 7eff96659640 1 -- 172.21.15.123:0/557430311 wait complete. 2024-09-16T15:51:43.437 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:43.437+0000 7eff96659640 1 Processor -- start 2024-09-16T15:51:43.437 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:43.437+0000 7eff96659640 1 -- start start 2024-09-16T15:51:43.438 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:43.437+0000 7eff96659640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7eff90074470 0x7eff901653c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:51:43.438 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:43.437+0000 7eff96659640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7eff900cdc20 0x7eff901fbc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:51:43.438 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:43.437+0000 7eff96659640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7eff9015e160 0x7eff900c4860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:51:43.439 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:43.437+0000 7eff96659640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7eff90167f50 con 0x7eff900cdc20 2024-09-16T15:51:43.439 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:43.437+0000 7eff96659640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7eff90167dd0 con 0x7eff9015e160 2024-09-16T15:51:43.439 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:43.437+0000 7eff96659640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7eff901680d0 con 0x7eff90074470 2024-09-16T15:51:43.439 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:43.437+0000 7eff8f7fe640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7eff900cdc20 0x7eff901fbc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:51:43.439 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:43.437+0000 7eff8f7fe640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7eff900cdc20 0x7eff901fbc50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:38404/0 (socket says 172.21.15.123:38404) 2024-09-16T15:51:43.439 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:43.437+0000 7eff8f7fe640 1 -- 172.21.15.123:0/2552025229 learned_addr learned my addr 172.21.15.123:0/2552025229 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:51:43.439 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:43.437+0000 7eff8ffff640 1 --2- 172.21.15.123:0/2552025229 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7eff90074470 0x7eff901653c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:51:43.439 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:43.437+0000 7eff8f7fe640 1 -- 172.21.15.123:0/2552025229 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7eff90074470 msgr2=0x7eff901653c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:51:43.439 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:43.437+0000 7eff94bf6640 1 --2- 172.21.15.123:0/2552025229 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7eff9015e160 0x7eff900c4860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:51:43.439 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:43.437+0000 7eff8f7fe640 1 --2- 172.21.15.123:0/2552025229 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7eff90074470 0x7eff901653c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:43.439 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:43.437+0000 7eff8f7fe640 1 -- 172.21.15.123:0/2552025229 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7eff9015e160 msgr2=0x7eff900c4860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:51:43.439 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:43.437+0000 7eff8f7fe640 1 --2- 172.21.15.123:0/2552025229 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7eff9015e160 0x7eff900c4860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:43.439 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:43.437+0000 7eff8f7fe640 1 -- 172.21.15.123:0/2552025229 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7eff901fb1e0 con 0x7eff900cdc20 2024-09-16T15:51:43.439 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:43.437+0000 7eff8f7fe640 1 --2- 172.21.15.123:0/2552025229 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7eff900cdc20 0x7eff901fbc50 secure :-1 s=READY pgs=1842 cs=0 l=1 rev1=1 crypto rx=0x7eff8000c890 tx=0x7eff8000cd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:51:43.440 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:43.437+0000 7eff8cff9640 1 -- 172.21.15.123:0/2552025229 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff80004230 con 0x7eff900cdc20 2024-09-16T15:51:43.440 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:43.437+0000 7eff8cff9640 1 -- 172.21.15.123:0/2552025229 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7eff800044c0 con 0x7eff900cdc20 2024-09-16T15:51:43.440 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:43.437+0000 7eff96659640 1 -- 172.21.15.123:0/2552025229 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7eff900c52c0 con 0x7eff900cdc20 2024-09-16T15:51:43.440 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:43.437+0000 7eff8cff9640 1 -- 172.21.15.123:0/2552025229 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff80007bf0 con 0x7eff900cdc20 2024-09-16T15:51:43.440 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:43.437+0000 7eff96659640 1 -- 172.21.15.123:0/2552025229 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7eff901fb730 con 0x7eff900cdc20 2024-09-16T15:51:43.440 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:43.441+0000 7eff96659640 1 -- 172.21.15.123:0/2552025229 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7eff900cd6e0 con 0x7eff900cdc20 2024-09-16T15:51:43.441 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:43.441+0000 7eff8cff9640 1 -- 172.21.15.123:0/2552025229 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 25) ==== 97998+0+0 (secure 0 0 0) 0x7eff80003600 con 0x7eff900cdc20 2024-09-16T15:51:43.441 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:43.441+0000 7eff8cff9640 1 --2- 172.21.15.123:0/2552025229 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7eff7c0738e0 0x7eff7c075da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:51:43.441 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:43.441+0000 7eff8ffff640 1 --2- 172.21.15.123:0/2552025229 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7eff7c0738e0 0x7eff7c075da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:51:43.441 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:43.441+0000 7eff8cff9640 1 -- 172.21.15.123:0/2552025229 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(161..161 src has 1..161) ==== 4971+0+0 (secure 0 0 0) 0x7eff80092e60 con 0x7eff900cdc20 2024-09-16T15:51:43.441 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:43.441+0000 7eff8ffff640 1 --2- 172.21.15.123:0/2552025229 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7eff7c0738e0 0x7eff7c075da0 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7eff840af000 tx=0x7eff8401b090 comp rx=0 tx=0).ready entity=mgr.4150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:51:43.445 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:43.445+0000 7eff8cff9640 1 -- 172.21.15.123:0/2552025229 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7eff80014070 con 0x7eff900cdc20 2024-09-16T15:51:43.650 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:43.649+0000 7eff96659640 1 -- 172.21.15.123:0/2552025229 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7eff90075a90 con 0x7eff900cdc20 2024-09-16T15:51:43.651 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:43.653+0000 7eff8cff9640 1 -- 172.21.15.123:0/2552025229 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 108 v108) ==== 95+0+3927 (secure 0 0 0) 0x7eff8005f940 con 0x7eff900cdc20 2024-09-16T15:51:43.651 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-09-16T15:51:43.651 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":108,"btime":"2024-09-16T15:51:37:258536+0000","default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6200,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6835/2380355001","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6833","nonce":2380355001},{"type":"v1","addr":"172.21.15.184:6835","nonce":2380355001}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":100},{"gid":6272,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6834/2787019988","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":2787019988},{"type":"v1","addr":"172.21.15.184:6834","nonce":2787019988}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":108},{"gid":8844,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6834/1840463681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6832","nonce":1840463681},{"type":"v1","addr":"172.21.15.123:6834","nonce":1840463681}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":100},{"gid":8847,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6835/2493915107","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6833","nonce":2493915107},{"type":"v1","addr":"172.21.15.123:6835","nonce":2493915107}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":99}],"filesystems":[]} 2024-09-16T15:51:43.651 INFO:teuthology.orchestra.run.smithi123.stderr:dumped fsmap epoch 108 2024-09-16T15:51:43.653 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:43.653+0000 7eff96659640 1 -- 172.21.15.123:0/2552025229 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7eff7c0738e0 msgr2=0x7eff7c075da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:51:43.653 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:43.653+0000 7eff96659640 1 --2- 172.21.15.123:0/2552025229 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7eff7c0738e0 0x7eff7c075da0 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7eff840af000 tx=0x7eff8401b090 comp rx=0 tx=0).stop 2024-09-16T15:51:43.654 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:43.653+0000 7eff96659640 1 -- 172.21.15.123:0/2552025229 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7eff900cdc20 msgr2=0x7eff901fbc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:51:43.654 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:43.653+0000 7eff96659640 1 --2- 172.21.15.123:0/2552025229 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7eff900cdc20 0x7eff901fbc50 secure :-1 s=READY pgs=1842 cs=0 l=1 rev1=1 crypto rx=0x7eff8000c890 tx=0x7eff8000cd50 comp rx=0 tx=0).stop 2024-09-16T15:51:43.654 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:43.653+0000 7eff96659640 1 -- 172.21.15.123:0/2552025229 shutdown_connections 2024-09-16T15:51:43.654 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:43.653+0000 7eff96659640 1 --2- 172.21.15.123:0/2552025229 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7eff7c0738e0 0x7eff7c075da0 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:43.654 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:43.653+0000 7eff96659640 1 --2- 172.21.15.123:0/2552025229 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7eff9015e160 0x7eff900c4860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:43.654 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:43.653+0000 7eff96659640 1 --2- 172.21.15.123:0/2552025229 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7eff900cdc20 0x7eff901fbc50 unknown :-1 s=CLOSED pgs=1842 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:43.654 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:43.653+0000 7eff96659640 1 --2- 172.21.15.123:0/2552025229 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7eff90074470 0x7eff901653c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:43.654 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:43.653+0000 7eff96659640 1 -- 172.21.15.123:0/2552025229 >> 172.21.15.123:0/2552025229 conn(0x7eff900cfb10 msgr2=0x7eff90158e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:51:43.654 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:43.653+0000 7eff96659640 1 -- 172.21.15.123:0/2552025229 shutdown_connections 2024-09-16T15:51:43.654 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:43.653+0000 7eff96659640 1 -- 172.21.15.123:0/2552025229 wait complete. 2024-09-16T15:51:43.668 INFO:tasks.ceph.mds.a:Restarting daemon 2024-09-16T15:51:43.668 INFO:tasks.ceph.mds.a:Stopping old one... 2024-09-16T15:51:43.669 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-09-16T15:51:43.669 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T15:51:43.669 INFO:tasks.ceph.mds.c:Restarting daemon 2024-09-16T15:51:43.669 INFO:tasks.ceph.mds.c:Stopping old one... 2024-09-16T15:51:43.669 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-09-16T15:51:43.669 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T15:51:43.670 INFO:tasks.ceph.mds.b:Restarting daemon 2024-09-16T15:51:43.670 INFO:tasks.ceph.mds.b:Stopping old one... 2024-09-16T15:51:43.670 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-09-16T15:51:43.670 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T15:51:43.670 INFO:tasks.ceph.mds.d:Restarting daemon 2024-09-16T15:51:43.670 INFO:tasks.ceph.mds.d:Stopping old one... 2024-09-16T15:51:43.670 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-09-16T15:51:43.670 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T15:51:43.682 INFO:tasks.ceph.mds.c:Stopped 2024-09-16T15:51:43.682 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-09-16T15:51:43.683 INFO:tasks.ceph.mds.a:Stopped 2024-09-16T15:51:43.683 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-09-16T15:51:43.686 INFO:tasks.ceph.mds.d:Stopped 2024-09-16T15:51:43.686 DEBUG:teuthology.orchestra.run.smithi184:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-09-16T15:51:43.687 INFO:tasks.ceph.mds.b:Stopped 2024-09-16T15:51:43.687 DEBUG:teuthology.orchestra.run.smithi184:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-09-16T15:51:43.727 INFO:tasks.ceph.mds.c:Started 2024-09-16T15:51:43.728 DEBUG:teuthology.parallel:result is None 2024-09-16T15:51:43.728 INFO:tasks.ceph.mds.a:Started 2024-09-16T15:51:43.728 DEBUG:teuthology.parallel:result is None 2024-09-16T15:51:43.733 INFO:tasks.ceph.mds.d:Started 2024-09-16T15:51:43.733 DEBUG:teuthology.parallel:result is None 2024-09-16T15:51:43.734 INFO:tasks.ceph.mds.b:Started 2024-09-16T15:51:43.734 DEBUG:teuthology.parallel:result is None 2024-09-16T15:51:43.734 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-09-16T15:51:43.787 INFO:tasks.ceph.mds.c.smithi123.stdout:starting mds.c at 2024-09-16T15:51:43.805 INFO:tasks.ceph.mds.b.smithi184.stdout:starting mds.b at 2024-09-16T15:51:43.806 INFO:tasks.ceph.mds.a.smithi123.stdout:starting mds.a at 2024-09-16T15:51:43.807 INFO:tasks.ceph.mds.d.smithi184.stdout:starting mds.d at 2024-09-16T15:51:43.849 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:43.849+0000 7f2be887e640 1 -- 172.21.15.123:0/1790010525 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2be406ce50 msgr2=0x7f2be40d00e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:51:43.849 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:43.849+0000 7f2be887e640 1 --2- 172.21.15.123:0/1790010525 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2be406ce50 0x7f2be40d00e0 secure :-1 s=READY pgs=1849 cs=0 l=1 rev1=1 crypto rx=0x7f2be40b0640 tx=0x7f2bd401ccb0 comp rx=0 tx=0).stop 2024-09-16T15:51:43.849 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:43.849+0000 7f2be887e640 1 -- 172.21.15.123:0/1790010525 shutdown_connections 2024-09-16T15:51:43.849 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:43.849+0000 7f2be887e640 1 --2- 172.21.15.123:0/1790010525 >> v2:172.21.15.184:3300/0 conn(0x7f2be4073600 0x7f2be4152e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:43.849 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:43.849+0000 7f2be887e640 1 --2- 172.21.15.123:0/1790010525 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2be406ce50 0x7f2be40d00e0 unknown :-1 s=CLOSED pgs=1849 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:43.849 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:43.849+0000 7f2be887e640 1 -- 172.21.15.123:0/1790010525 >> 172.21.15.123:0/1790010525 conn(0x7f2be40cd5d0 msgr2=0x7f2be40cd9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:51:43.849 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:43.849+0000 7f2be887e640 1 -- 172.21.15.123:0/1790010525 shutdown_connections 2024-09-16T15:51:43.849 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:43.849+0000 7f2be887e640 1 -- 172.21.15.123:0/1790010525 wait complete. 2024-09-16T15:51:43.850 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:43.849+0000 7f2be887e640 1 Processor -- start 2024-09-16T15:51:43.850 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:43.849+0000 7f2be887e640 1 -- start start 2024-09-16T15:51:43.852 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:43.849+0000 7f2be887e640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f2be4073600 0x7f2be40b5500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:51:43.852 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:43.849+0000 7f2be887e640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2be40d0620 0x7f2be40c6240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:51:43.852 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:43.849+0000 7f2be887e640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2be40c6780 0x7f2be414b2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:51:43.852 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:43.849+0000 7f2be887e640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f2be415c960 con 0x7f2be40d0620 2024-09-16T15:51:43.852 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:43.849+0000 7f2be887e640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f2be415c7e0 con 0x7f2be40c6780 2024-09-16T15:51:43.852 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:43.849+0000 7f2be887e640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f2be415cae0 con 0x7f2be4073600 2024-09-16T15:51:43.852 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:43.849+0000 7f2be1d9b640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2be40d0620 0x7f2be40c6240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:51:43.852 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:43.853+0000 7f2be1d9b640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2be40d0620 0x7f2be40c6240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:38452/0 (socket says 172.21.15.123:38452) 2024-09-16T15:51:43.852 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:43.853+0000 7f2be1d9b640 1 -- 172.21.15.123:0/302893441 learned_addr learned my addr 172.21.15.123:0/302893441 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:51:43.853 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:43.853+0000 7f2be259c640 1 --2- 172.21.15.123:0/302893441 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f2be4073600 0x7f2be40b5500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:51:43.853 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:43.853+0000 7f2be2d9d640 1 --2- 172.21.15.123:0/302893441 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2be40c6780 0x7f2be414b2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:51:43.853 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:43.853+0000 7f2be1d9b640 1 -- 172.21.15.123:0/302893441 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f2be4073600 msgr2=0x7f2be40b5500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:51:43.853 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:43.853+0000 7f2be1d9b640 1 --2- 172.21.15.123:0/302893441 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f2be4073600 0x7f2be40b5500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:43.853 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:43.853+0000 7f2be1d9b640 1 -- 172.21.15.123:0/302893441 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2be40c6780 msgr2=0x7f2be414b2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:51:43.853 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:43.853+0000 7f2be1d9b640 1 --2- 172.21.15.123:0/302893441 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2be40c6780 0x7f2be414b2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:43.853 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:43.853+0000 7f2be1d9b640 1 -- 172.21.15.123:0/302893441 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2be41f7010 con 0x7f2be40d0620 2024-09-16T15:51:43.853 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:43.853+0000 7f2be1d9b640 1 --2- 172.21.15.123:0/302893441 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2be40d0620 0x7f2be40c6240 secure :-1 s=READY pgs=1850 cs=0 l=1 rev1=1 crypto rx=0x7f2bcc00c3c0 tx=0x7f2bcc00c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:51:43.853 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:43.853+0000 7f2bdaffd640 1 -- 172.21.15.123:0/302893441 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2bcc017070 con 0x7f2be40d0620 2024-09-16T15:51:43.853 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:43.853+0000 7f2bdaffd640 1 -- 172.21.15.123:0/302893441 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f2bcc0053c0 con 0x7f2be40d0620 2024-09-16T15:51:43.853 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:43.853+0000 7f2be887e640 1 -- 172.21.15.123:0/302893441 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2be40c4890 con 0x7f2be40d0620 2024-09-16T15:51:43.853 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:43.853+0000 7f2bdaffd640 1 -- 172.21.15.123:0/302893441 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2bcc003860 con 0x7f2be40d0620 2024-09-16T15:51:43.853 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:43.853+0000 7f2be887e640 1 -- 172.21.15.123:0/302893441 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2be40c5130 con 0x7f2be40d0620 2024-09-16T15:51:43.853 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:43.853+0000 7f2be887e640 1 -- 172.21.15.123:0/302893441 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2be415d020 con 0x7f2be40d0620 2024-09-16T15:51:43.854 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:43.853+0000 7f2bdaffd640 1 -- 172.21.15.123:0/302893441 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 25) ==== 97998+0+0 (secure 0 0 0) 0x7f2bcc01a070 con 0x7f2be40d0620 2024-09-16T15:51:43.854 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:43.853+0000 7f2bdaffd640 1 --2- 172.21.15.123:0/302893441 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f2bc40738e0 0x7f2bc4075da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:51:43.854 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:43.853+0000 7f2bdaffd640 1 -- 172.21.15.123:0/302893441 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(161..161 src has 1..161) ==== 4971+0+0 (secure 0 0 0) 0x7f2bcc091b60 con 0x7f2be40d0620 2024-09-16T15:51:43.854 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:43.853+0000 7f2be259c640 1 --2- 172.21.15.123:0/302893441 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f2bc40738e0 0x7f2bc4075da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:51:43.855 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:43.853+0000 7f2be259c640 1 --2- 172.21.15.123:0/302893441 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f2bc40738e0 0x7f2bc4075da0 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f2bd40af000 tx=0x7f2bd401b4f0 comp rx=0 tx=0).ready entity=mgr.4150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:51:43.857 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:43.857+0000 7f2bdaffd640 1 -- 172.21.15.123:0/302893441 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f2bcc05dff0 con 0x7f2be40d0620 2024-09-16T15:51:44.028 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:44.029+0000 7f2be887e640 1 -- 172.21.15.123:0/302893441 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) -- 0x7f2be415de30 con 0x7f2be40d0620 2024-09-16T15:51:44.328 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:44.329+0000 7f2bdaffd640 1 -- 172.21.15.123:0/302893441 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v163) ==== 97+0+0 (secure 0 0 0) 0x7f2bcc062a20 con 0x7f2be40d0620 2024-09-16T15:51:44.328 INFO:teuthology.orchestra.run.smithi123.stderr: removed all blocklist entries 2024-09-16T15:51:44.333 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:44.333+0000 7f2be887e640 1 -- 172.21.15.123:0/302893441 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f2bc40738e0 msgr2=0x7f2bc4075da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:51:44.333 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:44.333+0000 7f2be887e640 1 --2- 172.21.15.123:0/302893441 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f2bc40738e0 0x7f2bc4075da0 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f2bd40af000 tx=0x7f2bd401b4f0 comp rx=0 tx=0).stop 2024-09-16T15:51:44.333 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:44.333+0000 7f2be887e640 1 -- 172.21.15.123:0/302893441 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2be40d0620 msgr2=0x7f2be40c6240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:51:44.333 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:44.333+0000 7f2be887e640 1 --2- 172.21.15.123:0/302893441 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2be40d0620 0x7f2be40c6240 secure :-1 s=READY pgs=1850 cs=0 l=1 rev1=1 crypto rx=0x7f2bcc00c3c0 tx=0x7f2bcc00c880 comp rx=0 tx=0).stop 2024-09-16T15:51:44.333 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:44.333+0000 7f2be887e640 1 -- 172.21.15.123:0/302893441 shutdown_connections 2024-09-16T15:51:44.333 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:44.333+0000 7f2be887e640 1 --2- 172.21.15.123:0/302893441 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f2bc40738e0 0x7f2bc4075da0 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:44.333 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:44.333+0000 7f2be887e640 1 --2- 172.21.15.123:0/302893441 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f2be40c6780 0x7f2be414b2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:44.333 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:44.333+0000 7f2be887e640 1 --2- 172.21.15.123:0/302893441 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2be40d0620 0x7f2be40c6240 unknown :-1 s=CLOSED pgs=1850 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:44.333 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:44.333+0000 7f2be887e640 1 --2- 172.21.15.123:0/302893441 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f2be4073600 0x7f2be40b5500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:44.333 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:44.333+0000 7f2be887e640 1 -- 172.21.15.123:0/302893441 >> 172.21.15.123:0/302893441 conn(0x7f2be40cd5d0 msgr2=0x7f2be40b1590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:51:44.333 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:44.333+0000 7f2be887e640 1 -- 172.21.15.123:0/302893441 shutdown_connections 2024-09-16T15:51:44.334 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:44.333+0000 7f2be887e640 1 -- 172.21.15.123:0/302893441 wait complete. 2024-09-16T15:51:44.354 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-09-16T15:51:44.485 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:44.485+0000 7f55bd4da640 1 -- 172.21.15.123:0/2441232215 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f55a4003a40 msgr2=0x7f55a4003e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:51:44.485 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:44.485+0000 7f55bd4da640 1 --2- 172.21.15.123:0/2441232215 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f55a4003a40 0x7f55a4003e60 secure :-1 s=READY pgs=1852 cs=0 l=1 rev1=1 crypto rx=0x7f55a8006fd0 tx=0x7f55a801cf10 comp rx=0 tx=0).stop 2024-09-16T15:51:44.485 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:44.485+0000 7f55b77fe640 1 -- 172.21.15.123:0/2441232215 reap_dead start 2024-09-16T15:51:44.485 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:44.485+0000 7f55bd4da640 1 -- 172.21.15.123:0/2441232215 shutdown_connections 2024-09-16T15:51:44.485 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:44.485+0000 7f55bd4da640 1 -- 172.21.15.123:0/2441232215 >> 172.21.15.123:0/2441232215 conn(0x7f55b80d3ea0 msgr2=0x7f55b80d42b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:51:44.485 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:44.485+0000 7f55bd4da640 1 -- 172.21.15.123:0/2441232215 shutdown_connections 2024-09-16T15:51:44.485 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:44.485+0000 7f55bd4da640 1 -- 172.21.15.123:0/2441232215 wait complete. 2024-09-16T15:51:44.487 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:44.485+0000 7f55bd4da640 1 Processor -- start 2024-09-16T15:51:44.487 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:44.485+0000 7f55bd4da640 1 -- start start 2024-09-16T15:51:44.488 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:44.485+0000 7f55bd4da640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f55a4003a40 0x7f55b8153a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:51:44.488 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:44.485+0000 7f55bd4da640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f55b80ca590 0x7f55b80b1e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:51:44.488 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:44.485+0000 7f55bd4da640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f55b80caa10 0x7f55b80b2380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:51:44.488 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:44.485+0000 7f55bd4da640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f55b8169a00 con 0x7f55b80caa10 2024-09-16T15:51:44.488 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:44.485+0000 7f55bd4da640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f55b8169880 con 0x7f55b80ca590 2024-09-16T15:51:44.488 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:44.485+0000 7f55bd4da640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f55b8169b80 con 0x7f55a4003a40 2024-09-16T15:51:44.488 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:44.489+0000 7f55b77fe640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f55b80caa10 0x7f55b80b2380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:51:44.488 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:44.489+0000 7f55b77fe640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f55b80caa10 0x7f55b80b2380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:38470/0 (socket says 172.21.15.123:38470) 2024-09-16T15:51:44.488 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:44.489+0000 7f55b77fe640 1 -- 172.21.15.123:0/3987359556 learned_addr learned my addr 172.21.15.123:0/3987359556 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:51:44.488 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:44.489+0000 7f55b67fc640 1 --2- 172.21.15.123:0/3987359556 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f55b80ca590 0x7f55b80b1e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:51:44.488 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:44.489+0000 7f55b6ffd640 1 --2- 172.21.15.123:0/3987359556 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f55a4003a40 0x7f55b8153a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:51:44.488 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:44.489+0000 7f55b77fe640 1 -- 172.21.15.123:0/3987359556 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f55a4003a40 msgr2=0x7f55b8153a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:51:44.488 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:44.489+0000 7f55b77fe640 1 --2- 172.21.15.123:0/3987359556 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f55a4003a40 0x7f55b8153a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:44.489 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:44.489+0000 7f55b77fe640 1 -- 172.21.15.123:0/3987359556 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f55b80ca590 msgr2=0x7f55b80b1e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:51:44.489 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:44.489+0000 7f55b77fe640 1 --2- 172.21.15.123:0/3987359556 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f55b80ca590 0x7f55b80b1e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:44.489 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:44.489+0000 7f55b77fe640 1 -- 172.21.15.123:0/3987359556 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f55b80b9d10 con 0x7f55b80caa10 2024-09-16T15:51:44.489 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:44.489+0000 7f55b77fe640 1 --2- 172.21.15.123:0/3987359556 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f55b80caa10 0x7f55b80b2380 secure :-1 s=READY pgs=1853 cs=0 l=1 rev1=1 crypto rx=0x7f55b808d240 tx=0x7f55b000fe00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:51:44.489 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:44.489+0000 7f55af7fe640 1 -- 172.21.15.123:0/3987359556 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55b000dda0 con 0x7f55b80caa10 2024-09-16T15:51:44.489 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:44.489+0000 7f55af7fe640 1 -- 172.21.15.123:0/3987359556 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f55b0011440 con 0x7f55b80caa10 2024-09-16T15:51:44.489 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:44.489+0000 7f55bd4da640 1 -- 172.21.15.123:0/3987359556 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f55b80b9750 con 0x7f55b80caa10 2024-09-16T15:51:44.489 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:44.489+0000 7f55af7fe640 1 -- 172.21.15.123:0/3987359556 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55b0011890 con 0x7f55b80caa10 2024-09-16T15:51:44.489 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:44.489+0000 7f55bd4da640 1 -- 172.21.15.123:0/3987359556 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f55b80c7b00 con 0x7f55b80caa10 2024-09-16T15:51:44.490 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:44.489+0000 7f55bd4da640 1 -- 172.21.15.123:0/3987359556 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f55b80d1da0 con 0x7f55b80caa10 2024-09-16T15:51:44.490 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:44.489+0000 7f55af7fe640 1 -- 172.21.15.123:0/3987359556 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 25) ==== 97998+0+0 (secure 0 0 0) 0x7f55b0011a30 con 0x7f55b80caa10 2024-09-16T15:51:44.490 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:44.489+0000 7f55af7fe640 1 --2- 172.21.15.123:0/3987359556 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f55980738e0 0x7f5598075da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:51:44.490 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:44.489+0000 7f55af7fe640 1 -- 172.21.15.123:0/3987359556 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(165..165 src has 1..165) ==== 4885+0+0 (secure 0 0 0) 0x7f55b00936e0 con 0x7f55b80caa10 2024-09-16T15:51:44.490 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:44.489+0000 7f55b6ffd640 1 --2- 172.21.15.123:0/3987359556 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f55980738e0 0x7f5598075da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:51:44.490 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:44.489+0000 7f55b6ffd640 1 --2- 172.21.15.123:0/3987359556 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f55980738e0 0x7f5598075da0 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f55a80ab000 tx=0x7f55a80b0040 comp rx=0 tx=0).ready entity=mgr.4150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:51:44.493 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:44.493+0000 7f55af7fe640 1 -- 172.21.15.123:0/3987359556 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f55b0021060 con 0x7f55b80caa10 2024-09-16T15:51:44.728 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:44.729+0000 7f55bd4da640 1 -- 172.21.15.123:0/3987359556 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) -- 0x7f55b80b5570 con 0x7f55b80caa10 2024-09-16T15:51:44.729 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:44.729+0000 7f55af7fe640 1 -- 172.21.15.123:0/3987359556 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v29) ==== 80+0+5304 (secure 0 0 0) 0x7f55b00601a0 con 0x7f55b80caa10 2024-09-16T15:51:44.729 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-09-16T15:51:44.729 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2024-09-16T15:51:44.729 INFO:teuthology.orchestra.run.smithi123.stdout: "auth_dump": [ 2024-09-16T15:51:44.729 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-09-16T15:51:44.729 INFO:teuthology.orchestra.run.smithi123.stdout: "entity": "mds.a", 2024-09-16T15:51:44.729 INFO:teuthology.orchestra.run.smithi123.stdout: "key": "AQC0UehmbZpOJBAAaYGyDcWrt4k0FS5P/bb+4A==", 2024-09-16T15:51:44.729 INFO:teuthology.orchestra.run.smithi123.stdout: "caps": { 2024-09-16T15:51:44.729 INFO:teuthology.orchestra.run.smithi123.stdout: "mds": "allow", 2024-09-16T15:51:44.729 INFO:teuthology.orchestra.run.smithi123.stdout: "mgr": "allow *", 2024-09-16T15:51:44.730 INFO:teuthology.orchestra.run.smithi123.stdout: "mon": "allow *", 2024-09-16T15:51:44.730 INFO:teuthology.orchestra.run.smithi123.stdout: "osd": "allow *" 2024-09-16T15:51:44.730 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-16T15:51:44.730 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:51:44.730 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-09-16T15:51:44.730 INFO:teuthology.orchestra.run.smithi123.stdout: "entity": "mds.b", 2024-09-16T15:51:44.730 INFO:teuthology.orchestra.run.smithi123.stdout: "key": "AQC0UehmF1HFMhAADcvqXNKdHZMPAbLc+se4Aw==", 2024-09-16T15:51:44.730 INFO:teuthology.orchestra.run.smithi123.stdout: "caps": { 2024-09-16T15:51:44.730 INFO:teuthology.orchestra.run.smithi123.stdout: "mds": "allow", 2024-09-16T15:51:44.730 INFO:teuthology.orchestra.run.smithi123.stdout: "mgr": "allow *", 2024-09-16T15:51:44.730 INFO:teuthology.orchestra.run.smithi123.stdout: "mon": "allow *", 2024-09-16T15:51:44.730 INFO:teuthology.orchestra.run.smithi123.stdout: "osd": "allow *" 2024-09-16T15:51:44.730 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-16T15:51:44.730 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:51:44.731 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-09-16T15:51:44.731 INFO:teuthology.orchestra.run.smithi123.stdout: "entity": "mds.c", 2024-09-16T15:51:44.731 INFO:teuthology.orchestra.run.smithi123.stdout: "key": "AQC0UehmCBjCLBAAXd85J25r7cgrXeDCG7pahg==", 2024-09-16T15:51:44.731 INFO:teuthology.orchestra.run.smithi123.stdout: "caps": { 2024-09-16T15:51:44.731 INFO:teuthology.orchestra.run.smithi123.stdout: "mds": "allow", 2024-09-16T15:51:44.731 INFO:teuthology.orchestra.run.smithi123.stdout: "mgr": "allow *", 2024-09-16T15:51:44.731 INFO:teuthology.orchestra.run.smithi123.stdout: "mon": "allow *", 2024-09-16T15:51:44.731 INFO:teuthology.orchestra.run.smithi123.stdout: "osd": "allow *" 2024-09-16T15:51:44.731 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-16T15:51:44.731 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:51:44.731 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-09-16T15:51:44.731 INFO:teuthology.orchestra.run.smithi123.stdout: "entity": "mds.d", 2024-09-16T15:51:44.731 INFO:teuthology.orchestra.run.smithi123.stdout: "key": "AQC0UehmMh3ZOBAAx0RCa3R9RMT93/icZhPGyg==", 2024-09-16T15:51:44.731 INFO:teuthology.orchestra.run.smithi123.stdout: "caps": { 2024-09-16T15:51:44.732 INFO:teuthology.orchestra.run.smithi123.stdout: "mds": "allow", 2024-09-16T15:51:44.732 INFO:teuthology.orchestra.run.smithi123.stdout: "mgr": "allow *", 2024-09-16T15:51:44.732 INFO:teuthology.orchestra.run.smithi123.stdout: "mon": "allow *", 2024-09-16T15:51:44.732 INFO:teuthology.orchestra.run.smithi123.stdout: "osd": "allow *" 2024-09-16T15:51:44.732 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-16T15:51:44.732 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:51:44.732 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-09-16T15:51:44.732 INFO:teuthology.orchestra.run.smithi123.stdout: "entity": "osd.0", 2024-09-16T15:51:44.732 INFO:teuthology.orchestra.run.smithi123.stdout: "key": "AQC4UehmT633CRAAIhpkeIBOzwJO0kiXLu0x9Q==", 2024-09-16T15:51:44.732 INFO:teuthology.orchestra.run.smithi123.stdout: "caps": { 2024-09-16T15:51:44.732 INFO:teuthology.orchestra.run.smithi123.stdout: "mgr": "allow profile osd", 2024-09-16T15:51:44.732 INFO:teuthology.orchestra.run.smithi123.stdout: "mon": "allow profile osd", 2024-09-16T15:51:44.732 INFO:teuthology.orchestra.run.smithi123.stdout: "osd": "allow *" 2024-09-16T15:51:44.732 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-16T15:51:44.732 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:51:44.733 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-09-16T15:51:44.733 INFO:teuthology.orchestra.run.smithi123.stdout: "entity": "osd.1", 2024-09-16T15:51:44.733 INFO:teuthology.orchestra.run.smithi123.stdout: "key": "AQC7UehmFwF7GBAAvOUG+rWjANKT8wm/ai4xGg==", 2024-09-16T15:51:44.733 INFO:teuthology.orchestra.run.smithi123.stdout: "caps": { 2024-09-16T15:51:44.733 INFO:teuthology.orchestra.run.smithi123.stdout: "mgr": "allow profile osd", 2024-09-16T15:51:44.733 INFO:teuthology.orchestra.run.smithi123.stdout: "mon": "allow profile osd", 2024-09-16T15:51:44.733 INFO:teuthology.orchestra.run.smithi123.stdout: "osd": "allow *" 2024-09-16T15:51:44.733 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-16T15:51:44.733 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:51:44.733 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-09-16T15:51:44.733 INFO:teuthology.orchestra.run.smithi123.stdout: "entity": "osd.2", 2024-09-16T15:51:44.733 INFO:teuthology.orchestra.run.smithi123.stdout: "key": "AQC+UehmlCwWKxAAl3iO7ZALWyaorQH2urHKCw==", 2024-09-16T15:51:44.733 INFO:teuthology.orchestra.run.smithi123.stdout: "caps": { 2024-09-16T15:51:44.733 INFO:teuthology.orchestra.run.smithi123.stdout: "mgr": "allow profile osd", 2024-09-16T15:51:44.734 INFO:teuthology.orchestra.run.smithi123.stdout: "mon": "allow profile osd", 2024-09-16T15:51:44.734 INFO:teuthology.orchestra.run.smithi123.stdout: "osd": "allow *" 2024-09-16T15:51:44.734 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-16T15:51:44.734 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:51:44.734 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-09-16T15:51:44.734 INFO:teuthology.orchestra.run.smithi123.stdout: "entity": "osd.3", 2024-09-16T15:51:44.734 INFO:teuthology.orchestra.run.smithi123.stdout: "key": "AQDCUehmD8VgAhAAneoIeGr3MflSmi+nyavj2A==", 2024-09-16T15:51:44.734 INFO:teuthology.orchestra.run.smithi123.stdout: "caps": { 2024-09-16T15:51:44.734 INFO:teuthology.orchestra.run.smithi123.stdout: "mgr": "allow profile osd", 2024-09-16T15:51:44.734 INFO:teuthology.orchestra.run.smithi123.stdout: "mon": "allow profile osd", 2024-09-16T15:51:44.734 INFO:teuthology.orchestra.run.smithi123.stdout: "osd": "allow *" 2024-09-16T15:51:44.734 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-16T15:51:44.734 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:51:44.735 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-09-16T15:51:44.735 INFO:teuthology.orchestra.run.smithi123.stdout: "entity": "osd.4", 2024-09-16T15:51:44.735 INFO:teuthology.orchestra.run.smithi123.stdout: "key": "AQDIUehmA9B9EBAAsu0X5LPx1TmatZeHzjnVig==", 2024-09-16T15:51:44.735 INFO:teuthology.orchestra.run.smithi123.stdout: "caps": { 2024-09-16T15:51:44.735 INFO:teuthology.orchestra.run.smithi123.stdout: "mgr": "allow profile osd", 2024-09-16T15:51:44.735 INFO:teuthology.orchestra.run.smithi123.stdout: "mon": "allow profile osd", 2024-09-16T15:51:44.735 INFO:teuthology.orchestra.run.smithi123.stdout: "osd": "allow *" 2024-09-16T15:51:44.735 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-16T15:51:44.735 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:51:44.735 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-09-16T15:51:44.735 INFO:teuthology.orchestra.run.smithi123.stdout: "entity": "osd.5", 2024-09-16T15:51:44.735 INFO:teuthology.orchestra.run.smithi123.stdout: "key": "AQDLUehmcugdHxAAQMSS1ZH71FUzCx5A+yNnZg==", 2024-09-16T15:51:44.735 INFO:teuthology.orchestra.run.smithi123.stdout: "caps": { 2024-09-16T15:51:44.735 INFO:teuthology.orchestra.run.smithi123.stdout: "mgr": "allow profile osd", 2024-09-16T15:51:44.736 INFO:teuthology.orchestra.run.smithi123.stdout: "mon": "allow profile osd", 2024-09-16T15:51:44.736 INFO:teuthology.orchestra.run.smithi123.stdout: "osd": "allow *" 2024-09-16T15:51:44.736 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-16T15:51:44.736 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:51:44.736 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-09-16T15:51:44.736 INFO:teuthology.orchestra.run.smithi123.stdout: "entity": "osd.6", 2024-09-16T15:51:44.736 INFO:teuthology.orchestra.run.smithi123.stdout: "key": "AQDOUehmWLwYLhAAhOlQLepcWElsVPMnHlZm4w==", 2024-09-16T15:51:44.736 INFO:teuthology.orchestra.run.smithi123.stdout: "caps": { 2024-09-16T15:51:44.736 INFO:teuthology.orchestra.run.smithi123.stdout: "mgr": "allow profile osd", 2024-09-16T15:51:44.736 INFO:teuthology.orchestra.run.smithi123.stdout: "mon": "allow profile osd", 2024-09-16T15:51:44.736 INFO:teuthology.orchestra.run.smithi123.stdout: "osd": "allow *" 2024-09-16T15:51:44.736 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-16T15:51:44.736 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:51:44.736 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-09-16T15:51:44.737 INFO:teuthology.orchestra.run.smithi123.stdout: "entity": "osd.7", 2024-09-16T15:51:44.737 INFO:teuthology.orchestra.run.smithi123.stdout: "key": "AQDSUehmipkbABAAQuL/2O7w0tsM1OKeOil97w==", 2024-09-16T15:51:44.737 INFO:teuthology.orchestra.run.smithi123.stdout: "caps": { 2024-09-16T15:51:44.737 INFO:teuthology.orchestra.run.smithi123.stdout: "mgr": "allow profile osd", 2024-09-16T15:51:44.737 INFO:teuthology.orchestra.run.smithi123.stdout: "mon": "allow profile osd", 2024-09-16T15:51:44.737 INFO:teuthology.orchestra.run.smithi123.stdout: "osd": "allow *" 2024-09-16T15:51:44.737 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-16T15:51:44.737 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:51:44.737 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-09-16T15:51:44.737 INFO:teuthology.orchestra.run.smithi123.stdout: "entity": "client.0", 2024-09-16T15:51:44.737 INFO:teuthology.orchestra.run.smithi123.stdout: "key": "AQC1Uehm7GDTAhAAHWWVIyjVVnTeeozBR8DfFA==", 2024-09-16T15:51:44.737 INFO:teuthology.orchestra.run.smithi123.stdout: "caps": { 2024-09-16T15:51:44.737 INFO:teuthology.orchestra.run.smithi123.stdout: "mds": "allow", 2024-09-16T15:51:44.738 INFO:teuthology.orchestra.run.smithi123.stdout: "mon": "allow r", 2024-09-16T15:51:44.738 INFO:teuthology.orchestra.run.smithi123.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-09-16T15:51:44.738 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-16T15:51:44.738 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:51:44.738 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-09-16T15:51:44.738 INFO:teuthology.orchestra.run.smithi123.stdout: "entity": "client.1", 2024-09-16T15:51:44.738 INFO:teuthology.orchestra.run.smithi123.stdout: "key": "AQC1UehmFbaWChAAFtRmKucm+pbKtCNvRCTNqA==", 2024-09-16T15:51:44.738 INFO:teuthology.orchestra.run.smithi123.stdout: "caps": { 2024-09-16T15:51:44.738 INFO:teuthology.orchestra.run.smithi123.stdout: "mds": "allow", 2024-09-16T15:51:44.738 INFO:teuthology.orchestra.run.smithi123.stdout: "mgr": "allow r", 2024-09-16T15:51:44.738 INFO:teuthology.orchestra.run.smithi123.stdout: "mon": "allow rw", 2024-09-16T15:51:44.738 INFO:teuthology.orchestra.run.smithi123.stdout: "osd": "allow rwx" 2024-09-16T15:51:44.739 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-16T15:51:44.739 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:51:44.739 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-09-16T15:51:44.739 INFO:teuthology.orchestra.run.smithi123.stdout: "entity": "client.2", 2024-09-16T15:51:44.739 INFO:teuthology.orchestra.run.smithi123.stdout: "key": "AQC1Uehm1ADrBxAAW/8ayNHI6Wq/YlI4ivFZiw==", 2024-09-16T15:51:44.739 INFO:teuthology.orchestra.run.smithi123.stdout: "caps": { 2024-09-16T15:51:44.739 INFO:teuthology.orchestra.run.smithi123.stdout: "mds": "allow", 2024-09-16T15:51:44.739 INFO:teuthology.orchestra.run.smithi123.stdout: "mgr": "allow r", 2024-09-16T15:51:44.739 INFO:teuthology.orchestra.run.smithi123.stdout: "mon": "allow rw", 2024-09-16T15:51:44.739 INFO:teuthology.orchestra.run.smithi123.stdout: "osd": "allow rwx" 2024-09-16T15:51:44.739 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-16T15:51:44.739 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:51:44.739 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-09-16T15:51:44.740 INFO:teuthology.orchestra.run.smithi123.stdout: "entity": "client.3", 2024-09-16T15:51:44.740 INFO:teuthology.orchestra.run.smithi123.stdout: "key": "AQC1UehmLv/3DxAAuGloBxSuza95uajpfiZ9MQ==", 2024-09-16T15:51:44.740 INFO:teuthology.orchestra.run.smithi123.stdout: "caps": { 2024-09-16T15:51:44.740 INFO:teuthology.orchestra.run.smithi123.stdout: "mds": "allow", 2024-09-16T15:51:44.740 INFO:teuthology.orchestra.run.smithi123.stdout: "mgr": "allow r", 2024-09-16T15:51:44.740 INFO:teuthology.orchestra.run.smithi123.stdout: "mon": "allow rw", 2024-09-16T15:51:44.740 INFO:teuthology.orchestra.run.smithi123.stdout: "osd": "allow rwx" 2024-09-16T15:51:44.740 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-16T15:51:44.740 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:51:44.740 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-09-16T15:51:44.740 INFO:teuthology.orchestra.run.smithi123.stdout: "entity": "client.admin", 2024-09-16T15:51:44.740 INFO:teuthology.orchestra.run.smithi123.stdout: "key": "AQC0Uehm1ZDRDBAAhyQ1BmJg1Mw6wQ04I2CbFA==", 2024-09-16T15:51:44.740 INFO:teuthology.orchestra.run.smithi123.stdout: "caps": { 2024-09-16T15:51:44.741 INFO:teuthology.orchestra.run.smithi123.stdout: "mds": "allow *", 2024-09-16T15:51:44.741 INFO:teuthology.orchestra.run.smithi123.stdout: "mgr": "allow *", 2024-09-16T15:51:44.741 INFO:teuthology.orchestra.run.smithi123.stdout: "mon": "allow *", 2024-09-16T15:51:44.741 INFO:teuthology.orchestra.run.smithi123.stdout: "osd": "allow *" 2024-09-16T15:51:44.741 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-16T15:51:44.741 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:51:44.741 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-09-16T15:51:44.741 INFO:teuthology.orchestra.run.smithi123.stdout: "entity": "mgr.x", 2024-09-16T15:51:44.741 INFO:teuthology.orchestra.run.smithi123.stdout: "key": "AQC0UehmfPilIBAAERlBbduOgAU9ba0n/1tl3g==", 2024-09-16T15:51:44.741 INFO:teuthology.orchestra.run.smithi123.stdout: "caps": { 2024-09-16T15:51:44.741 INFO:teuthology.orchestra.run.smithi123.stdout: "mds": "allow *", 2024-09-16T15:51:44.741 INFO:teuthology.orchestra.run.smithi123.stdout: "mon": "allow profile mgr", 2024-09-16T15:51:44.741 INFO:teuthology.orchestra.run.smithi123.stdout: "osd": "allow *" 2024-09-16T15:51:44.742 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-16T15:51:44.742 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:51:44.742 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-09-16T15:51:44.742 INFO:teuthology.orchestra.run.smithi123.stdout: "entity": "mgr.y", 2024-09-16T15:51:44.742 INFO:teuthology.orchestra.run.smithi123.stdout: "key": "AQC0UehmkjrlHRAA+3xbQC+SPWGWW1gND7xcAg==", 2024-09-16T15:51:44.742 INFO:teuthology.orchestra.run.smithi123.stdout: "caps": { 2024-09-16T15:51:44.742 INFO:teuthology.orchestra.run.smithi123.stdout: "mds": "allow *", 2024-09-16T15:51:44.742 INFO:teuthology.orchestra.run.smithi123.stdout: "mon": "allow profile mgr", 2024-09-16T15:51:44.742 INFO:teuthology.orchestra.run.smithi123.stdout: "osd": "allow *" 2024-09-16T15:51:44.742 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-16T15:51:44.742 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-16T15:51:44.742 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-09-16T15:51:44.743 INFO:teuthology.orchestra.run.smithi123.stdout:} 2024-09-16T15:51:44.743 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:44.733+0000 7f55bd4da640 1 -- 172.21.15.123:0/3987359556 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f55980738e0 msgr2=0x7f5598075da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:51:44.743 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:44.733+0000 7f55bd4da640 1 --2- 172.21.15.123:0/3987359556 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f55980738e0 0x7f5598075da0 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f55a80ab000 tx=0x7f55a80b0040 comp rx=0 tx=0).stop 2024-09-16T15:51:44.743 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:44.733+0000 7f55bd4da640 1 -- 172.21.15.123:0/3987359556 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f55b80caa10 msgr2=0x7f55b80b2380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:51:44.744 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:44.733+0000 7f55bd4da640 1 --2- 172.21.15.123:0/3987359556 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f55b80caa10 0x7f55b80b2380 secure :-1 s=READY pgs=1853 cs=0 l=1 rev1=1 crypto rx=0x7f55b808d240 tx=0x7f55b000fe00 comp rx=0 tx=0).stop 2024-09-16T15:51:44.744 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:44.733+0000 7f55bd4da640 1 -- 172.21.15.123:0/3987359556 shutdown_connections 2024-09-16T15:51:44.744 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:44.733+0000 7f55bd4da640 1 --2- 172.21.15.123:0/3987359556 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f55980738e0 0x7f5598075da0 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:44.744 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:44.733+0000 7f55bd4da640 1 --2- 172.21.15.123:0/3987359556 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f55b80caa10 0x7f55b80b2380 unknown :-1 s=CLOSED pgs=1853 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:44.744 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:44.733+0000 7f55bd4da640 1 --2- 172.21.15.123:0/3987359556 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f55b80ca590 0x7f55b80b1e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:44.744 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:44.733+0000 7f55bd4da640 1 --2- 172.21.15.123:0/3987359556 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f55a4003a40 0x7f55b8153a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:44.744 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:44.733+0000 7f55bd4da640 1 -- 172.21.15.123:0/3987359556 >> 172.21.15.123:0/3987359556 conn(0x7f55b80d3ea0 msgr2=0x7f55b80d25d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:51:44.744 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:44.733+0000 7f55bd4da640 1 -- 172.21.15.123:0/3987359556 shutdown_connections 2024-09-16T15:51:44.744 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:44.733+0000 7f55bd4da640 1 -- 172.21.15.123:0/3987359556 wait complete. 2024-09-16T15:51:44.748 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-09-16T15:51:44.877 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:44.877+0000 7f990bb3c640 1 -- 172.21.15.123:0/2377593345 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f99040cf5a0 msgr2=0x7f990406eee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:51:44.877 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:44.877+0000 7f990bb3c640 1 --2- 172.21.15.123:0/2377593345 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f99040cf5a0 0x7f990406eee0 secure :-1 s=READY pgs=1855 cs=0 l=1 rev1=1 crypto rx=0x7f98f4009900 tx=0x7f98f401c7c0 comp rx=0 tx=0).stop 2024-09-16T15:51:44.878 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:44.877+0000 7f990bb3c640 1 -- 172.21.15.123:0/2377593345 shutdown_connections 2024-09-16T15:51:44.878 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:44.877+0000 7f990bb3c640 1 --2- 172.21.15.123:0/2377593345 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f99040cf5a0 0x7f990406eee0 unknown :-1 s=CLOSED pgs=1855 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:44.878 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:44.877+0000 7f990bb3c640 1 -- 172.21.15.123:0/2377593345 >> 172.21.15.123:0/2377593345 conn(0x7f99040cfb10 msgr2=0x7f99040cff20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:51:44.878 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:44.877+0000 7f990bb3c640 1 -- 172.21.15.123:0/2377593345 shutdown_connections 2024-09-16T15:51:44.878 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:44.877+0000 7f990bb3c640 1 -- 172.21.15.123:0/2377593345 wait complete. 2024-09-16T15:51:44.878 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:44.877+0000 7f990bb3c640 1 Processor -- start 2024-09-16T15:51:44.878 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:44.881+0000 7f990bb3c640 1 -- start start 2024-09-16T15:51:44.880 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:44.881+0000 7f990bb3c640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f99040cf5a0 0x7f9904147160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:51:44.881 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:44.881+0000 7f990bb3c640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f9904158120 0x7f99041596b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:51:44.881 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:44.881+0000 7f990bb3c640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f990415ec40 0x7f99040c4480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:51:44.881 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:44.881+0000 7f990bb3c640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f99041689e0 con 0x7f990415ec40 2024-09-16T15:51:44.881 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:44.881+0000 7f990bb3c640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f9904168860 con 0x7f99040cf5a0 2024-09-16T15:51:44.881 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:44.881+0000 7f990bb3c640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f9904168b60 con 0x7f9904158120 2024-09-16T15:51:44.881 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:44.881+0000 7f990a0d9640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f990415ec40 0x7f99040c4480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:51:44.881 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:44.881+0000 7f990a0d9640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f990415ec40 0x7f99040c4480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:38488/0 (socket says 172.21.15.123:38488) 2024-09-16T15:51:44.881 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:44.881+0000 7f990a0d9640 1 -- 172.21.15.123:0/244168896 learned_addr learned my addr 172.21.15.123:0/244168896 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:51:44.881 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:44.881+0000 7f99098d8640 1 --2- 172.21.15.123:0/244168896 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f99040cf5a0 0x7f9904147160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:51:44.881 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:44.881+0000 7f99090d7640 1 --2- 172.21.15.123:0/244168896 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f9904158120 0x7f99041596b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:51:44.881 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:44.881+0000 7f990a0d9640 1 -- 172.21.15.123:0/244168896 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f9904158120 msgr2=0x7f99041596b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:51:44.881 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:44.881+0000 7f990a0d9640 1 --2- 172.21.15.123:0/244168896 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f9904158120 0x7f99041596b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:44.881 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:44.881+0000 7f990a0d9640 1 -- 172.21.15.123:0/244168896 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f99040cf5a0 msgr2=0x7f9904147160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:51:44.881 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:44.881+0000 7f990a0d9640 1 --2- 172.21.15.123:0/244168896 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f99040cf5a0 0x7f9904147160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:44.882 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:44.881+0000 7f990a0d9640 1 -- 172.21.15.123:0/244168896 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f990415a050 con 0x7f990415ec40 2024-09-16T15:51:44.882 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:44.881+0000 7f99090d7640 1 --2- 172.21.15.123:0/244168896 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f9904158120 0x7f99041596b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T15:51:44.882 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:44.881+0000 7f990a0d9640 1 --2- 172.21.15.123:0/244168896 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f990415ec40 0x7f99040c4480 secure :-1 s=READY pgs=1856 cs=0 l=1 rev1=1 crypto rx=0x7f990000b440 tx=0x7f990000b900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:51:44.882 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:44.881+0000 7f98fa7fc640 1 -- 172.21.15.123:0/244168896 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9900015020 con 0x7f990415ec40 2024-09-16T15:51:44.882 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:44.881+0000 7f98fa7fc640 1 -- 172.21.15.123:0/244168896 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f9900011940 con 0x7f990415ec40 2024-09-16T15:51:44.882 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:44.881+0000 7f98fa7fc640 1 -- 172.21.15.123:0/244168896 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9900011d90 con 0x7f990415ec40 2024-09-16T15:51:44.882 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:44.881+0000 7f990bb3c640 1 -- 172.21.15.123:0/244168896 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f990415a390 con 0x7f990415ec40 2024-09-16T15:51:44.882 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:44.881+0000 7f990bb3c640 1 -- 172.21.15.123:0/244168896 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f99041476a0 con 0x7f990415ec40 2024-09-16T15:51:44.883 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:44.881+0000 7f990bb3c640 1 -- 172.21.15.123:0/244168896 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f99041484a0 con 0x7f990415ec40 2024-09-16T15:51:44.883 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:44.881+0000 7f98fa7fc640 1 -- 172.21.15.123:0/244168896 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 25) ==== 97998+0+0 (secure 0 0 0) 0x7f9900018020 con 0x7f990415ec40 2024-09-16T15:51:44.883 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:44.885+0000 7f98fa7fc640 1 --2- 172.21.15.123:0/244168896 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f98e80738e0 0x7f98e8075da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:51:44.883 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:44.885+0000 7f98fa7fc640 1 -- 172.21.15.123:0/244168896 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(165..165 src has 1..165) ==== 4885+0+0 (secure 0 0 0) 0x7f9900092db0 con 0x7f990415ec40 2024-09-16T15:51:44.883 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:44.885+0000 7f99098d8640 1 --2- 172.21.15.123:0/244168896 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f98e80738e0 0x7f98e8075da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:51:44.883 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:44.885+0000 7f99098d8640 1 --2- 172.21.15.123:0/244168896 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f98e80738e0 0x7f98e8075da0 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f98fc0012f0 tx=0x7f98fc009040 comp rx=0 tx=0).ready entity=mgr.4150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:51:44.886 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:44.885+0000 7f98fa7fc640 1 -- 172.21.15.123:0/244168896 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f990001f030 con 0x7f990415ec40 2024-09-16T15:51:45.064 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:45.065+0000 7f990bb3c640 1 -- 172.21.15.123:0/244168896 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) -- 0x7f99040d06c0 con 0x7f990415ec40 2024-09-16T15:51:45.064 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:45.065+0000 7f98fa7fc640 1 -- 172.21.15.123:0/244168896 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v165) ==== 84+0+65 (secure 0 0 0) 0x7f990415a390 con 0x7f990415ec40 2024-09-16T15:51:45.064 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-09-16T15:51:45.065 INFO:teuthology.orchestra.run.smithi123.stdout:[ 2024-09-16T15:51:45.065 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-09-16T15:51:45.065 INFO:teuthology.orchestra.run.smithi123.stdout: "poolnum": 1, 2024-09-16T15:51:45.065 INFO:teuthology.orchestra.run.smithi123.stdout: "poolname": ".mgr" 2024-09-16T15:51:45.065 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-16T15:51:45.065 INFO:teuthology.orchestra.run.smithi123.stdout:] 2024-09-16T15:51:45.067 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:45.069+0000 7f990bb3c640 1 -- 172.21.15.123:0/244168896 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f98e80738e0 msgr2=0x7f98e8075da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:51:45.067 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:45.069+0000 7f990bb3c640 1 --2- 172.21.15.123:0/244168896 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f98e80738e0 0x7f98e8075da0 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f98fc0012f0 tx=0x7f98fc009040 comp rx=0 tx=0).stop 2024-09-16T15:51:45.067 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:45.069+0000 7f990bb3c640 1 -- 172.21.15.123:0/244168896 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f990415ec40 msgr2=0x7f99040c4480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:51:45.067 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:45.069+0000 7f990bb3c640 1 --2- 172.21.15.123:0/244168896 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f990415ec40 0x7f99040c4480 secure :-1 s=READY pgs=1856 cs=0 l=1 rev1=1 crypto rx=0x7f990000b440 tx=0x7f990000b900 comp rx=0 tx=0).stop 2024-09-16T15:51:45.067 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:45.069+0000 7f990bb3c640 1 -- 172.21.15.123:0/244168896 shutdown_connections 2024-09-16T15:51:45.068 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:45.069+0000 7f990bb3c640 1 --2- 172.21.15.123:0/244168896 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f98e80738e0 0x7f98e8075da0 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:45.068 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:45.069+0000 7f990bb3c640 1 --2- 172.21.15.123:0/244168896 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f990415ec40 0x7f99040c4480 unknown :-1 s=CLOSED pgs=1856 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:45.068 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:45.069+0000 7f990bb3c640 1 --2- 172.21.15.123:0/244168896 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f9904158120 0x7f99041596b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:45.068 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:45.069+0000 7f990bb3c640 1 --2- 172.21.15.123:0/244168896 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f99040cf5a0 0x7f9904147160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:45.068 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:45.069+0000 7f990bb3c640 1 -- 172.21.15.123:0/244168896 >> 172.21.15.123:0/244168896 conn(0x7f99040cfb10 msgr2=0x7f99040ca320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:51:45.068 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:45.069+0000 7f990bb3c640 1 -- 172.21.15.123:0/244168896 shutdown_connections 2024-09-16T15:51:45.068 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:45.069+0000 7f990bb3c640 1 -- 172.21.15.123:0/244168896 wait complete. 2024-09-16T15:51:45.084 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-09-16T15:51:45.085 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-09-16T15:51:45.217 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:45.217+0000 7fb550b4f640 1 -- 172.21.15.123:0/1366152023 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb534004810 msgr2=0x7fb534004c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:51:45.217 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:45.217+0000 7fb550b4f640 1 --2- 172.21.15.123:0/1366152023 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb534004810 0x7fb534004c90 secure :-1 s=READY pgs=1858 cs=0 l=1 rev1=1 crypto rx=0x7fb540009b60 tx=0x7fb54001d270 comp rx=0 tx=0).stop 2024-09-16T15:51:45.218 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:45.217+0000 7fb54ad9d640 1 -- 172.21.15.123:0/1366152023 reap_dead start 2024-09-16T15:51:45.218 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:45.217+0000 7fb550b4f640 1 -- 172.21.15.123:0/1366152023 shutdown_connections 2024-09-16T15:51:45.218 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:45.217+0000 7fb550b4f640 1 -- 172.21.15.123:0/1366152023 >> 172.21.15.123:0/1366152023 conn(0x7fb54c0b1630 msgr2=0x7fb54c0b1a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:51:45.218 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:45.217+0000 7fb550b4f640 1 -- 172.21.15.123:0/1366152023 shutdown_connections 2024-09-16T15:51:45.218 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:45.217+0000 7fb550b4f640 1 -- 172.21.15.123:0/1366152023 wait complete. 2024-09-16T15:51:45.218 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:45.217+0000 7fb550b4f640 1 Processor -- start 2024-09-16T15:51:45.218 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:45.221+0000 7fb550b4f640 1 -- start start 2024-09-16T15:51:45.220 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:45.221+0000 7fb550b4f640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb534004810 0x7fb54c152a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:51:45.221 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:45.221+0000 7fb550b4f640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fb54c160b90 0x7fb54c14b3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:51:45.221 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:45.221+0000 7fb550b4f640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb54c14c2e0 0x7fb54c14c760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:51:45.221 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:45.221+0000 7fb550b4f640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fb54c166e80 con 0x7fb54c14c2e0 2024-09-16T15:51:45.221 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:45.221+0000 7fb550b4f640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fb54c166d00 con 0x7fb534004810 2024-09-16T15:51:45.221 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:45.221+0000 7fb550b4f640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fb54c167000 con 0x7fb54c160b90 2024-09-16T15:51:45.221 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:45.221+0000 7fb54ad9d640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb54c14c2e0 0x7fb54c14c760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:51:45.221 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:45.221+0000 7fb54ad9d640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb54c14c2e0 0x7fb54c14c760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:38504/0 (socket says 172.21.15.123:38504) 2024-09-16T15:51:45.221 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:45.221+0000 7fb54ad9d640 1 -- 172.21.15.123:0/4188147642 learned_addr learned my addr 172.21.15.123:0/4188147642 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:51:45.221 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:45.221+0000 7fb54a59c640 1 --2- 172.21.15.123:0/4188147642 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb534004810 0x7fb54c152a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:51:45.221 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:45.221+0000 7fb549d9b640 1 --2- 172.21.15.123:0/4188147642 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fb54c160b90 0x7fb54c14b3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:51:45.221 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:45.221+0000 7fb54ad9d640 1 -- 172.21.15.123:0/4188147642 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fb54c160b90 msgr2=0x7fb54c14b3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:51:45.222 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:45.221+0000 7fb54ad9d640 1 --2- 172.21.15.123:0/4188147642 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fb54c160b90 0x7fb54c14b3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:45.222 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:45.221+0000 7fb54ad9d640 1 -- 172.21.15.123:0/4188147642 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb534004810 msgr2=0x7fb54c152a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:51:45.222 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:45.221+0000 7fb54ad9d640 1 --2- 172.21.15.123:0/4188147642 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb534004810 0x7fb54c152a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:45.222 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:45.221+0000 7fb54ad9d640 1 -- 172.21.15.123:0/4188147642 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb54c0cec40 con 0x7fb54c14c2e0 2024-09-16T15:51:45.222 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:45.221+0000 7fb54a59c640 1 --2- 172.21.15.123:0/4188147642 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb534004810 0x7fb54c152a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T15:51:45.222 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:45.221+0000 7fb54ad9d640 1 --2- 172.21.15.123:0/4188147642 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb54c14c2e0 0x7fb54c14c760 secure :-1 s=READY pgs=1859 cs=0 l=1 rev1=1 crypto rx=0x7fb54c0b3640 tx=0x7fb54400ed70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:51:45.222 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:45.221+0000 7fb53affd640 1 -- 172.21.15.123:0/4188147642 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb544015020 con 0x7fb54c14c2e0 2024-09-16T15:51:45.222 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:45.221+0000 7fb53affd640 1 -- 172.21.15.123:0/4188147642 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fb54400cd10 con 0x7fb54c14c2e0 2024-09-16T15:51:45.222 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:45.221+0000 7fb550b4f640 1 -- 172.21.15.123:0/4188147642 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb54c1f6c90 con 0x7fb54c14c2e0 2024-09-16T15:51:45.222 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:45.221+0000 7fb53affd640 1 -- 172.21.15.123:0/4188147642 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb544001ac0 con 0x7fb54c14c2e0 2024-09-16T15:51:45.222 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:45.221+0000 7fb550b4f640 1 -- 172.21.15.123:0/4188147642 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb54c1f7250 con 0x7fb54c14c2e0 2024-09-16T15:51:45.223 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:45.221+0000 7fb550b4f640 1 -- 172.21.15.123:0/4188147642 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb54c15b000 con 0x7fb54c14c2e0 2024-09-16T15:51:45.223 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:45.221+0000 7fb53affd640 1 -- 172.21.15.123:0/4188147642 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 25) ==== 97998+0+0 (secure 0 0 0) 0x7fb544018020 con 0x7fb54c14c2e0 2024-09-16T15:51:45.223 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:45.221+0000 7fb53affd640 1 --2- 172.21.15.123:0/4188147642 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fb52c073910 0x7fb52c075dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:51:45.223 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:45.225+0000 7fb54a59c640 1 --2- 172.21.15.123:0/4188147642 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fb52c073910 0x7fb52c075dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:51:45.223 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:45.225+0000 7fb53affd640 1 -- 172.21.15.123:0/4188147642 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(165..165 src has 1..165) ==== 4885+0+0 (secure 0 0 0) 0x7fb544092ff0 con 0x7fb54c14c2e0 2024-09-16T15:51:45.223 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:45.225+0000 7fb54a59c640 1 --2- 172.21.15.123:0/4188147642 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fb52c073910 0x7fb52c075dd0 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7fb54c158b60 tx=0x7fb53c006c10 comp rx=0 tx=0).ready entity=mgr.4150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:51:45.227 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:45.229+0000 7fb53affd640 1 -- 172.21.15.123:0/4188147642 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7fb54401f020 con 0x7fb54c14c2e0 2024-09-16T15:51:45.404 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:45.405+0000 7fb550b4f640 1 -- 172.21.15.123:0/4188147642 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) -- 0x7fb54c0b3900 con 0x7fb54c14c2e0 2024-09-16T15:51:46.341 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:46.341+0000 7fb53affd640 1 -- 172.21.15.123:0/4188147642 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v166) ==== 138+0+0 (secure 0 0 0) 0x7fb54405fad0 con 0x7fb54c14c2e0 2024-09-16T15:51:46.342 INFO:teuthology.orchestra.run.smithi123.stderr:pool 'cephfs_metadata' created 2024-09-16T15:51:46.345 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:46.345+0000 7fb550b4f640 1 -- 172.21.15.123:0/4188147642 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fb52c073910 msgr2=0x7fb52c075dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:51:46.345 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:46.345+0000 7fb550b4f640 1 --2- 172.21.15.123:0/4188147642 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fb52c073910 0x7fb52c075dd0 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7fb54c158b60 tx=0x7fb53c006c10 comp rx=0 tx=0).stop 2024-09-16T15:51:46.345 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:46.345+0000 7fb550b4f640 1 -- 172.21.15.123:0/4188147642 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb54c14c2e0 msgr2=0x7fb54c14c760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:51:46.346 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:46.345+0000 7fb550b4f640 1 --2- 172.21.15.123:0/4188147642 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb54c14c2e0 0x7fb54c14c760 secure :-1 s=READY pgs=1859 cs=0 l=1 rev1=1 crypto rx=0x7fb54c0b3640 tx=0x7fb54400ed70 comp rx=0 tx=0).stop 2024-09-16T15:51:46.346 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:46.345+0000 7fb550b4f640 1 -- 172.21.15.123:0/4188147642 shutdown_connections 2024-09-16T15:51:46.346 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:46.345+0000 7fb550b4f640 1 --2- 172.21.15.123:0/4188147642 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fb52c073910 0x7fb52c075dd0 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:46.346 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:46.345+0000 7fb550b4f640 1 --2- 172.21.15.123:0/4188147642 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb54c14c2e0 0x7fb54c14c760 unknown :-1 s=CLOSED pgs=1859 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:46.346 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:46.345+0000 7fb550b4f640 1 --2- 172.21.15.123:0/4188147642 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fb54c160b90 0x7fb54c14b3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:46.346 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:46.345+0000 7fb550b4f640 1 --2- 172.21.15.123:0/4188147642 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb534004810 0x7fb54c152a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:46.346 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:46.345+0000 7fb550b4f640 1 -- 172.21.15.123:0/4188147642 >> 172.21.15.123:0/4188147642 conn(0x7fb54c0b1630 msgr2=0x7fb54c14e3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:51:46.346 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:46.345+0000 7fb550b4f640 1 -- 172.21.15.123:0/4188147642 shutdown_connections 2024-09-16T15:51:46.346 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:46.345+0000 7fb550b4f640 1 -- 172.21.15.123:0/4188147642 wait complete. 2024-09-16T15:51:46.371 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-09-16T15:51:46.506 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:46.505+0000 7f8b7a9dc640 1 -- 172.21.15.123:0/3090927833 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8b7415b9f0 msgr2=0x7f8b7406fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:51:46.506 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:46.505+0000 7f8b7a9dc640 1 --2- 172.21.15.123:0/3090927833 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8b7415b9f0 0x7f8b7406fca0 secure :-1 s=READY pgs=1861 cs=0 l=1 rev1=1 crypto rx=0x7f8b64009900 tx=0x7f8b6401c8b0 comp rx=0 tx=0).stop 2024-09-16T15:51:46.506 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:46.505+0000 7f8b7a9dc640 1 -- 172.21.15.123:0/3090927833 shutdown_connections 2024-09-16T15:51:46.507 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:46.505+0000 7f8b7a9dc640 1 --2- 172.21.15.123:0/3090927833 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f8b740cf4e0 0x7f8b7415f6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:46.507 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:46.505+0000 7f8b7a9dc640 1 --2- 172.21.15.123:0/3090927833 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8b7415b9f0 0x7f8b7406fca0 unknown :-1 s=CLOSED pgs=1861 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:46.507 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:46.505+0000 7f8b7a9dc640 1 -- 172.21.15.123:0/3090927833 >> 172.21.15.123:0/3090927833 conn(0x7f8b740b1710 msgr2=0x7f8b740b1b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:51:46.507 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:46.505+0000 7f8b7a9dc640 1 -- 172.21.15.123:0/3090927833 shutdown_connections 2024-09-16T15:51:46.507 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:46.505+0000 7f8b7a9dc640 1 -- 172.21.15.123:0/3090927833 wait complete. 2024-09-16T15:51:46.507 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:46.509+0000 7f8b7a9dc640 1 Processor -- start 2024-09-16T15:51:46.507 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:46.509+0000 7f8b7a9dc640 1 -- start start 2024-09-16T15:51:46.509 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:46.509+0000 7f8b7a9dc640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f8b740c3230 0x7f8b740ca630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:51:46.509 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:46.509+0000 7f8b7a9dc640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8b740cf4e0 0x7f8b740cae90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:51:46.509 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:46.509+0000 7f8b7a9dc640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f8b7415b9f0 0x7f8b740cb3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:51:46.510 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:46.509+0000 7f8b7a9dc640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f8b741690f0 con 0x7f8b740cf4e0 2024-09-16T15:51:46.510 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:46.509+0000 7f8b7a9dc640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f8b74168f70 con 0x7f8b7415b9f0 2024-09-16T15:51:46.510 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:46.509+0000 7f8b7a9dc640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f8b74169270 con 0x7f8b740c3230 2024-09-16T15:51:46.510 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:46.509+0000 7f8b737fe640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8b740cf4e0 0x7f8b740cae90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:51:46.510 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:46.509+0000 7f8b737fe640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8b740cf4e0 0x7f8b740cae90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:38534/0 (socket says 172.21.15.123:38534) 2024-09-16T15:51:46.510 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:46.509+0000 7f8b737fe640 1 -- 172.21.15.123:0/1929346721 learned_addr learned my addr 172.21.15.123:0/1929346721 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:51:46.510 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:46.509+0000 7f8b73fff640 1 --2- 172.21.15.123:0/1929346721 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f8b740c3230 0x7f8b740ca630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:51:46.510 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:46.509+0000 7f8b737fe640 1 -- 172.21.15.123:0/1929346721 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f8b740c3230 msgr2=0x7f8b740ca630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:51:46.510 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:46.509+0000 7f8b737fe640 1 --2- 172.21.15.123:0/1929346721 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f8b740c3230 0x7f8b740ca630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:46.510 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:46.509+0000 7f8b737fe640 1 -- 172.21.15.123:0/1929346721 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f8b7415b9f0 msgr2=0x7f8b740cb3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:51:46.510 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:46.509+0000 7f8b737fe640 1 --2- 172.21.15.123:0/1929346721 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f8b7415b9f0 0x7f8b740cb3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:46.510 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:46.509+0000 7f8b737fe640 1 -- 172.21.15.123:0/1929346721 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8b740b1fc0 con 0x7f8b740cf4e0 2024-09-16T15:51:46.510 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:46.509+0000 7f8b737fe640 1 --2- 172.21.15.123:0/1929346721 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8b740cf4e0 0x7f8b740cae90 secure :-1 s=READY pgs=1862 cs=0 l=1 rev1=1 crypto rx=0x7f8b6800c400 tx=0x7f8b6800c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:51:46.511 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:46.509+0000 7f8b70ff9640 1 -- 172.21.15.123:0/1929346721 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b68017070 con 0x7f8b740cf4e0 2024-09-16T15:51:46.511 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:46.509+0000 7f8b70ff9640 1 -- 172.21.15.123:0/1929346721 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f8b680053e0 con 0x7f8b740cf4e0 2024-09-16T15:51:46.511 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:46.509+0000 7f8b7a9dc640 1 -- 172.21.15.123:0/1929346721 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8b740c47f0 con 0x7f8b740cf4e0 2024-09-16T15:51:46.511 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:46.509+0000 7f8b70ff9640 1 -- 172.21.15.123:0/1929346721 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b68003860 con 0x7f8b740cf4e0 2024-09-16T15:51:46.511 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:46.509+0000 7f8b7a9dc640 1 -- 172.21.15.123:0/1929346721 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8b74158db0 con 0x7f8b740cf4e0 2024-09-16T15:51:46.512 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:46.509+0000 7f8b7a9dc640 1 -- 172.21.15.123:0/1929346721 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8b7415f6b0 con 0x7f8b740cf4e0 2024-09-16T15:51:46.512 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:46.513+0000 7f8b70ff9640 1 -- 172.21.15.123:0/1929346721 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 25) ==== 97998+0+0 (secure 0 0 0) 0x7f8b6801a070 con 0x7f8b740cf4e0 2024-09-16T15:51:46.512 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:46.513+0000 7f8b70ff9640 1 --2- 172.21.15.123:0/1929346721 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f8b600738e0 0x7f8b60075da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:51:46.512 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:46.513+0000 7f8b70ff9640 1 -- 172.21.15.123:0/1929346721 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(166..166 src has 1..166) ==== 5255+0+0 (secure 0 0 0) 0x7f8b680921d0 con 0x7f8b740cf4e0 2024-09-16T15:51:46.512 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:46.513+0000 7f8b73fff640 1 --2- 172.21.15.123:0/1929346721 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f8b600738e0 0x7f8b60075da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:51:46.512 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:46.513+0000 7f8b73fff640 1 --2- 172.21.15.123:0/1929346721 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f8b600738e0 0x7f8b60075da0 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f8b640af000 tx=0x7f8b6401b0f0 comp rx=0 tx=0).ready entity=mgr.4150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:51:46.516 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:46.517+0000 7f8b70ff9640 1 -- 172.21.15.123:0/1929346721 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f8b68021020 con 0x7f8b740cf4e0 2024-09-16T15:51:46.688 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:46.689+0000 7f8b7a9dc640 1 -- 172.21.15.123:0/1929346721 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) -- 0x7f8b740c49c0 con 0x7f8b740cf4e0 2024-09-16T15:51:47.346 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:47.345+0000 7f8b70ff9640 1 -- 172.21.15.123:0/1929346721 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v167) ==== 170+0+0 (secure 0 0 0) 0x7f8b68001130 con 0x7f8b740cf4e0 2024-09-16T15:51:47.346 INFO:teuthology.orchestra.run.smithi123.stderr:pool 'cephfs_data' created 2024-09-16T15:51:47.349 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:47.349+0000 7f8b7a9dc640 1 -- 172.21.15.123:0/1929346721 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f8b600738e0 msgr2=0x7f8b60075da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:51:47.349 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:47.349+0000 7f8b7a9dc640 1 --2- 172.21.15.123:0/1929346721 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f8b600738e0 0x7f8b60075da0 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f8b640af000 tx=0x7f8b6401b0f0 comp rx=0 tx=0).stop 2024-09-16T15:51:47.349 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:47.349+0000 7f8b7a9dc640 1 -- 172.21.15.123:0/1929346721 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8b740cf4e0 msgr2=0x7f8b740cae90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:51:47.349 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:47.349+0000 7f8b7a9dc640 1 --2- 172.21.15.123:0/1929346721 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8b740cf4e0 0x7f8b740cae90 secure :-1 s=READY pgs=1862 cs=0 l=1 rev1=1 crypto rx=0x7f8b6800c400 tx=0x7f8b6800c8c0 comp rx=0 tx=0).stop 2024-09-16T15:51:47.349 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:47.349+0000 7f8b7a9dc640 1 -- 172.21.15.123:0/1929346721 shutdown_connections 2024-09-16T15:51:47.349 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:47.349+0000 7f8b7a9dc640 1 --2- 172.21.15.123:0/1929346721 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f8b600738e0 0x7f8b60075da0 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:47.350 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:47.349+0000 7f8b7a9dc640 1 --2- 172.21.15.123:0/1929346721 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f8b7415b9f0 0x7f8b740cb3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:47.350 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:47.349+0000 7f8b7a9dc640 1 --2- 172.21.15.123:0/1929346721 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8b740cf4e0 0x7f8b740cae90 unknown :-1 s=CLOSED pgs=1862 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:47.350 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:47.349+0000 7f8b7a9dc640 1 --2- 172.21.15.123:0/1929346721 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f8b740c3230 0x7f8b740ca630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:47.350 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:47.349+0000 7f8b7a9dc640 1 -- 172.21.15.123:0/1929346721 >> 172.21.15.123:0/1929346721 conn(0x7f8b740b1710 msgr2=0x7f8b740b43b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:51:47.350 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:47.349+0000 7f8b7a9dc640 1 -- 172.21.15.123:0/1929346721 shutdown_connections 2024-09-16T15:51:47.350 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:47.349+0000 7f8b7a9dc640 1 -- 172.21.15.123:0/1929346721 wait complete. 2024-09-16T15:51:47.383 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-09-16T15:51:47.514 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:47.513+0000 7f06a8f87640 1 -- 172.21.15.123:0/161320930 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f06a415a170 msgr2=0x7f06a4072f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:51:47.514 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:47.513+0000 7f06a8f87640 1 --2- 172.21.15.123:0/161320930 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f06a415a170 0x7f06a4072f30 secure :-1 s=READY pgs=1864 cs=0 l=1 rev1=1 crypto rx=0x7f0694009840 tx=0x7f069401c760 comp rx=0 tx=0).stop 2024-09-16T15:51:47.514 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:47.513+0000 7f06a8f87640 1 -- 172.21.15.123:0/161320930 shutdown_connections 2024-09-16T15:51:47.514 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:47.513+0000 7f06a8f87640 1 --2- 172.21.15.123:0/161320930 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f06a415a170 0x7f06a4072f30 unknown :-1 s=CLOSED pgs=1864 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:47.515 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:47.513+0000 7f06a8f87640 1 -- 172.21.15.123:0/161320930 >> 172.21.15.123:0/161320930 conn(0x7f06a40b1710 msgr2=0x7f06a40b1b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:51:47.515 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:47.513+0000 7f06a8f87640 1 -- 172.21.15.123:0/161320930 shutdown_connections 2024-09-16T15:51:47.515 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:47.513+0000 7f06a8f87640 1 -- 172.21.15.123:0/161320930 wait complete. 2024-09-16T15:51:47.515 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:47.517+0000 7f06a8f87640 1 Processor -- start 2024-09-16T15:51:47.515 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:47.517+0000 7f06a8f87640 1 -- start start 2024-09-16T15:51:47.517 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:47.517+0000 7f06a8f87640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f06a40ca650 0x7f06a4155aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:51:47.517 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:47.517+0000 7f06a8f87640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f06a415a170 0x7f06a41f9210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:51:47.517 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:47.517+0000 7f06a8f87640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f06a415ddc0 0x7f06a41f6350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:51:47.517 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:47.517+0000 7f06a8f87640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f06a4168170 con 0x7f06a415ddc0 2024-09-16T15:51:47.517 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:47.517+0000 7f06a8f87640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f06a415a670 con 0x7f06a415a170 2024-09-16T15:51:47.517 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:47.517+0000 7f06a8f87640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f06a41682f0 con 0x7f06a40ca650 2024-09-16T15:51:47.517 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:47.517+0000 7f06a2d9d640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f06a415ddc0 0x7f06a41f6350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:51:47.518 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:47.517+0000 7f06a2d9d640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f06a415ddc0 0x7f06a41f6350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:38554/0 (socket says 172.21.15.123:38554) 2024-09-16T15:51:47.518 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:47.517+0000 7f06a2d9d640 1 -- 172.21.15.123:0/1671261449 learned_addr learned my addr 172.21.15.123:0/1671261449 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:51:47.518 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:47.517+0000 7f06a259c640 1 --2- 172.21.15.123:0/1671261449 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f06a40ca650 0x7f06a4155aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:51:47.518 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:47.517+0000 7f06a2d9d640 1 -- 172.21.15.123:0/1671261449 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f06a40ca650 msgr2=0x7f06a4155aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:51:47.518 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:47.517+0000 7f06a1d9b640 1 --2- 172.21.15.123:0/1671261449 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f06a415a170 0x7f06a41f9210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:51:47.518 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:47.517+0000 7f06a2d9d640 1 --2- 172.21.15.123:0/1671261449 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f06a40ca650 0x7f06a4155aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:47.518 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:47.517+0000 7f06a2d9d640 1 -- 172.21.15.123:0/1671261449 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f06a415a170 msgr2=0x7f06a41f9210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:51:47.518 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:47.517+0000 7f06a2d9d640 1 --2- 172.21.15.123:0/1671261449 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f06a415a170 0x7f06a41f9210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:47.518 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:47.517+0000 7f06a2d9d640 1 -- 172.21.15.123:0/1671261449 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f06a41f9910 con 0x7f06a415ddc0 2024-09-16T15:51:47.518 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:47.517+0000 7f06a2d9d640 1 --2- 172.21.15.123:0/1671261449 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f06a415ddc0 0x7f06a41f6350 secure :-1 s=READY pgs=1865 cs=0 l=1 rev1=1 crypto rx=0x7f069c00b9a0 tx=0x7f069c00be60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:51:47.518 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:47.517+0000 7f0692ffd640 1 -- 172.21.15.123:0/1671261449 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f069c019070 con 0x7f06a415ddc0 2024-09-16T15:51:47.518 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:47.517+0000 7f0692ffd640 1 -- 172.21.15.123:0/1671261449 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f069c00cc90 con 0x7f06a415ddc0 2024-09-16T15:51:47.518 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:47.517+0000 7f06a8f87640 1 -- 172.21.15.123:0/1671261449 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f06a41f9bf0 con 0x7f06a415ddc0 2024-09-16T15:51:47.519 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:47.517+0000 7f0692ffd640 1 -- 172.21.15.123:0/1671261449 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f069c0146a0 con 0x7f06a415ddc0 2024-09-16T15:51:47.519 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:47.517+0000 7f06a8f87640 1 -- 172.21.15.123:0/1671261449 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f06a41f5e30 con 0x7f06a415ddc0 2024-09-16T15:51:47.519 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:47.517+0000 7f06a8f87640 1 -- 172.21.15.123:0/1671261449 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f06a41fa110 con 0x7f06a415ddc0 2024-09-16T15:51:47.519 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:47.521+0000 7f0692ffd640 1 -- 172.21.15.123:0/1671261449 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 25) ==== 97998+0+0 (secure 0 0 0) 0x7f069c01c070 con 0x7f06a415ddc0 2024-09-16T15:51:47.520 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:47.521+0000 7f0692ffd640 1 --2- 172.21.15.123:0/1671261449 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f06840738e0 0x7f0684075da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:51:47.520 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:47.521+0000 7f0692ffd640 1 -- 172.21.15.123:0/1671261449 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(167..167 src has 1..167) ==== 5637+0+0 (secure 0 0 0) 0x7f069c0927d0 con 0x7f06a415ddc0 2024-09-16T15:51:47.520 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:47.521+0000 7f06a259c640 1 --2- 172.21.15.123:0/1671261449 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f06840738e0 0x7f0684075da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:51:47.520 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:47.521+0000 7f06a259c640 1 --2- 172.21.15.123:0/1671261449 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f06840738e0 0x7f0684075da0 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f06940af000 tx=0x7f069401afa0 comp rx=0 tx=0).ready entity=mgr.4150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:51:47.523 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:47.525+0000 7f0692ffd640 1 -- 172.21.15.123:0/1671261449 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f06a41fa110 con 0x7f06a415ddc0 2024-09-16T15:51:47.735 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:47.737+0000 7f06a8f87640 1 -- 172.21.15.123:0/1671261449 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) -- 0x7f06a41f5c40 con 0x7f06a415ddc0 2024-09-16T15:51:48.419 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:48.421+0000 7f0692ffd640 1 -- 172.21.15.123:0/1671261449 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 26 and data pool 27 v111) ==== 174+0+0 (secure 0 0 0) 0x7f069c05f4b0 con 0x7f06a415ddc0 2024-09-16T15:51:48.424 INFO:teuthology.orchestra.run.smithi123.stderr:new fs with metadata pool 26 and data pool 27 2024-09-16T15:51:48.425 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:48.425+0000 7f0690ff9640 1 -- 172.21.15.123:0/1671261449 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f06840738e0 msgr2=0x7f0684075da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:51:48.425 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:48.425+0000 7f0690ff9640 1 --2- 172.21.15.123:0/1671261449 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f06840738e0 0x7f0684075da0 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f06940af000 tx=0x7f069401afa0 comp rx=0 tx=0).stop 2024-09-16T15:51:48.425 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:48.425+0000 7f0690ff9640 1 -- 172.21.15.123:0/1671261449 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f06a415ddc0 msgr2=0x7f06a41f6350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:51:48.425 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:48.425+0000 7f0690ff9640 1 --2- 172.21.15.123:0/1671261449 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f06a415ddc0 0x7f06a41f6350 secure :-1 s=READY pgs=1865 cs=0 l=1 rev1=1 crypto rx=0x7f069c00b9a0 tx=0x7f069c00be60 comp rx=0 tx=0).stop 2024-09-16T15:51:48.425 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:48.425+0000 7f0690ff9640 1 -- 172.21.15.123:0/1671261449 shutdown_connections 2024-09-16T15:51:48.425 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:48.425+0000 7f0690ff9640 1 --2- 172.21.15.123:0/1671261449 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f06840738e0 0x7f0684075da0 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:48.425 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:48.425+0000 7f0690ff9640 1 --2- 172.21.15.123:0/1671261449 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f06a415ddc0 0x7f06a41f6350 unknown :-1 s=CLOSED pgs=1865 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:48.426 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:48.425+0000 7f0690ff9640 1 --2- 172.21.15.123:0/1671261449 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f06a415a170 0x7f06a41f9210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:48.426 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:48.425+0000 7f0690ff9640 1 --2- 172.21.15.123:0/1671261449 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f06a40ca650 0x7f06a4155aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:48.426 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:48.425+0000 7f0690ff9640 1 -- 172.21.15.123:0/1671261449 >> 172.21.15.123:0/1671261449 conn(0x7f06a40b1710 msgr2=0x7f06a40c62b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:51:48.426 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:48.425+0000 7f0690ff9640 1 -- 172.21.15.123:0/1671261449 shutdown_connections 2024-09-16T15:51:48.426 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:48.425+0000 7f0690ff9640 1 -- 172.21.15.123:0/1671261449 wait complete. 2024-09-16T15:51:48.455 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-16T15:51:48.572 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:48.573+0000 7f55090be640 1 -- 172.21.15.123:0/3002973976 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f55040d2820 msgr2=0x7f55040cea00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:51:48.572 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:48.573+0000 7f55090be640 1 --2- 172.21.15.123:0/3002973976 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f55040d2820 0x7f55040cea00 secure :-1 s=READY pgs=1866 cs=0 l=1 rev1=1 crypto rx=0x7f550408ce10 tx=0x7f54fc01cc30 comp rx=0 tx=0).stop 2024-09-16T15:51:48.572 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:48.573+0000 7f55090be640 1 -- 172.21.15.123:0/3002973976 shutdown_connections 2024-09-16T15:51:48.572 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:48.573+0000 7f55090be640 1 --2- 172.21.15.123:0/3002973976 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f55040d2820 0x7f55040cea00 unknown :-1 s=CLOSED pgs=1866 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:48.572 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:48.573+0000 7f55090be640 1 --2- 172.21.15.123:0/3002973976 >> v2:172.21.15.184:3300/0 conn(0x7f55040b2b00 0x7f55040d22e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:48.572 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:48.573+0000 7f55090be640 1 --2- 172.21.15.123:0/3002973976 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f5504161620 0x7f55040d1da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:48.572 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:48.573+0000 7f55090be640 1 -- 172.21.15.123:0/3002973976 >> 172.21.15.123:0/3002973976 conn(0x7f55040d3ea0 msgr2=0x7f55040d42b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:51:48.572 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:48.573+0000 7f55090be640 1 -- 172.21.15.123:0/3002973976 shutdown_connections 2024-09-16T15:51:48.572 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:48.573+0000 7f55090be640 1 -- 172.21.15.123:0/3002973976 wait complete. 2024-09-16T15:51:48.573 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:48.573+0000 7f55090be640 1 Processor -- start 2024-09-16T15:51:48.573 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:48.573+0000 7f55090be640 1 -- start start 2024-09-16T15:51:48.574 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:48.573+0000 7f55090be640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f55040b2b00 0x7f55041526a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:51:48.574 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:48.573+0000 7f55090be640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f55040d2820 0x7f5504152d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:51:48.574 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:48.573+0000 7f55090be640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5504161620 0x7f55040cb980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:51:48.575 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:48.573+0000 7f55090be640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f55040b10e0 con 0x7f5504161620 2024-09-16T15:51:48.575 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:48.573+0000 7f55090be640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f55040b0f60 con 0x7f55040b2b00 2024-09-16T15:51:48.575 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:48.573+0000 7f55090be640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f55040b1260 con 0x7f55040d2820 2024-09-16T15:51:48.575 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:48.573+0000 7f550359e640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5504161620 0x7f55040cb980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:51:48.575 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:48.573+0000 7f550359e640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5504161620 0x7f55040cb980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:38560/0 (socket says 172.21.15.123:38560) 2024-09-16T15:51:48.575 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:48.573+0000 7f550359e640 1 -- 172.21.15.123:0/2411037354 learned_addr learned my addr 172.21.15.123:0/2411037354 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:51:48.575 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:48.573+0000 7f550259c640 1 --2- 172.21.15.123:0/2411037354 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f55040d2820 0x7f5504152d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:51:48.575 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:48.573+0000 7f550359e640 1 -- 172.21.15.123:0/2411037354 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f55040d2820 msgr2=0x7f5504152d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:51:48.575 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:48.573+0000 7f550359e640 1 --2- 172.21.15.123:0/2411037354 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f55040d2820 0x7f5504152d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:48.575 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:48.573+0000 7f5502d9d640 1 --2- 172.21.15.123:0/2411037354 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f55040b2b00 0x7f55041526a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:51:48.575 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:48.573+0000 7f550359e640 1 -- 172.21.15.123:0/2411037354 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f55040b2b00 msgr2=0x7f55041526a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:51:48.575 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:48.573+0000 7f550359e640 1 --2- 172.21.15.123:0/2411037354 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f55040b2b00 0x7f55041526a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:48.575 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:48.573+0000 7f550359e640 1 -- 172.21.15.123:0/2411037354 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f550406d4a0 con 0x7f5504161620 2024-09-16T15:51:48.576 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:48.573+0000 7f550259c640 1 --2- 172.21.15.123:0/2411037354 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f55040d2820 0x7f5504152d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T15:51:48.576 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:48.573+0000 7f5502d9d640 1 --2- 172.21.15.123:0/2411037354 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f55040b2b00 0x7f55041526a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T15:51:48.576 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:48.573+0000 7f550359e640 1 --2- 172.21.15.123:0/2411037354 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5504161620 0x7f55040cb980 secure :-1 s=READY pgs=1867 cs=0 l=1 rev1=1 crypto rx=0x7f54fc01b2e0 tx=0x7f54fc0a5a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:51:48.576 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:48.573+0000 7f54eb7fe640 1 -- 172.21.15.123:0/2411037354 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54fc0bb070 con 0x7f5504161620 2024-09-16T15:51:48.576 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:48.573+0000 7f54eb7fe640 1 -- 172.21.15.123:0/2411037354 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f54fc004d40 con 0x7f5504161620 2024-09-16T15:51:48.576 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:48.573+0000 7f55090be640 1 -- 172.21.15.123:0/2411037354 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f550406d780 con 0x7f5504161620 2024-09-16T15:51:48.576 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:48.573+0000 7f54eb7fe640 1 -- 172.21.15.123:0/2411037354 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54fc0b2040 con 0x7f5504161620 2024-09-16T15:51:48.576 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:48.573+0000 7f55090be640 1 -- 172.21.15.123:0/2411037354 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f55040c8ea0 con 0x7f5504161620 2024-09-16T15:51:48.577 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:48.577+0000 7f55090be640 1 -- 172.21.15.123:0/2411037354 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f55040cea00 con 0x7f5504161620 2024-09-16T15:51:48.577 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:48.577+0000 7f54eb7fe640 1 -- 172.21.15.123:0/2411037354 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 25) ==== 97998+0+0 (secure 0 0 0) 0x7f54fc004320 con 0x7f5504161620 2024-09-16T15:51:48.577 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:48.577+0000 7f54eb7fe640 1 --2- 172.21.15.123:0/2411037354 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f54f00738e0 0x7f54f0075da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:51:48.577 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:48.577+0000 7f54eb7fe640 1 -- 172.21.15.123:0/2411037354 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(168..168 src has 1..168) ==== 5737+0+0 (secure 0 0 0) 0x7f54fc12ccd0 con 0x7f5504161620 2024-09-16T15:51:48.577 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:48.577+0000 7f5502d9d640 1 --2- 172.21.15.123:0/2411037354 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f54f00738e0 0x7f54f0075da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:51:48.577 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:48.577+0000 7f5502d9d640 1 --2- 172.21.15.123:0/2411037354 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f54f00738e0 0x7f54f0075da0 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f54f8005870 tx=0x7f54f8005cb0 comp rx=0 tx=0).ready entity=mgr.4150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:51:48.580 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:48.581+0000 7f54eb7fe640 1 -- 172.21.15.123:0/2411037354 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f54fc0c4020 con 0x7f5504161620 2024-09-16T15:51:48.754 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:48.753+0000 7f55090be640 1 -- 172.21.15.123:0/2411037354 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f55040d24f0 con 0x7f5504161620 2024-09-16T15:51:48.756 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:48.757+0000 7f54eb7fe640 1 -- 172.21.15.123:0/2411037354 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v168) ==== 74+0+16860 (secure 0 0 0) 0x7f54fc016970 con 0x7f5504161620 2024-09-16T15:51:48.757 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-09-16T15:51:48.757 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":168,"fsid":"56077f89-18c8-486f-ae7e-eecc0bb7822c","created":"2024-09-16T15:42:26.364892+0000","modified":"2024-09-16T15:51:47.741386+0000","last_up_change":"2024-09-16T15:42:38.707234+0000","last_in_change":"2024-09-16T15:42:29.423662+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":27,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-16T15:42:39.287764+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":26,"pool_name":"cephfs_metadata","create_time":"2024-09-16T15:51:45.417755+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"168","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":27,"pool_name":"cephfs_data","create_time":"2024-09-16T15:51:46.701679+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"168","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9e7c1955-a607-4671-9be7-f2ef760ceeaa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":167,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6800","nonce":3882044969},{"type":"v1","addr":"172.21.15.123:6801","nonce":3882044969}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6802","nonce":3882044969},{"type":"v1","addr":"172.21.15.123:6803","nonce":3882044969}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6806","nonce":3882044969},{"type":"v1","addr":"172.21.15.123:6807","nonce":3882044969}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6804","nonce":3882044969},{"type":"v1","addr":"172.21.15.123:6805","nonce":3882044969}]},"public_addr":"172.21.15.123:6801/3882044969","cluster_addr":"172.21.15.123:6803/3882044969","heartbeat_back_addr":"172.21.15.123:6807/3882044969","heartbeat_front_addr":"172.21.15.123:6805/3882044969","state":["exists","up"]},{"osd":1,"uuid":"c540cfde-0977-4742-9b17-fd634668e010","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":167,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6816","nonce":4037001850},{"type":"v1","addr":"172.21.15.123:6817","nonce":4037001850}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6818","nonce":4037001850},{"type":"v1","addr":"172.21.15.123:6819","nonce":4037001850}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6822","nonce":4037001850},{"type":"v1","addr":"172.21.15.123:6823","nonce":4037001850}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6820","nonce":4037001850},{"type":"v1","addr":"172.21.15.123:6821","nonce":4037001850}]},"public_addr":"172.21.15.123:6817/4037001850","cluster_addr":"172.21.15.123:6819/4037001850","heartbeat_back_addr":"172.21.15.123:6823/4037001850","heartbeat_front_addr":"172.21.15.123:6821/4037001850","state":["exists","up"]},{"osd":2,"uuid":"9864a496-f169-486b-95c4-1d59162793a7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":167,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6824","nonce":1332143088},{"type":"v1","addr":"172.21.15.123:6825","nonce":1332143088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6826","nonce":1332143088},{"type":"v1","addr":"172.21.15.123:6827","nonce":1332143088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6830","nonce":1332143088},{"type":"v1","addr":"172.21.15.123:6831","nonce":1332143088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6828","nonce":1332143088},{"type":"v1","addr":"172.21.15.123:6829","nonce":1332143088}]},"public_addr":"172.21.15.123:6825/1332143088","cluster_addr":"172.21.15.123:6827/1332143088","heartbeat_back_addr":"172.21.15.123:6831/1332143088","heartbeat_front_addr":"172.21.15.123:6829/1332143088","state":["exists","up"]},{"osd":3,"uuid":"c6d1f817-f074-4325-95e1-578eba13edda","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":167,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6808","nonce":988586998},{"type":"v1","addr":"172.21.15.123:6809","nonce":988586998}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6810","nonce":988586998},{"type":"v1","addr":"172.21.15.123:6811","nonce":988586998}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6814","nonce":988586998},{"type":"v1","addr":"172.21.15.123:6815","nonce":988586998}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6812","nonce":988586998},{"type":"v1","addr":"172.21.15.123:6813","nonce":988586998}]},"public_addr":"172.21.15.123:6809/988586998","cluster_addr":"172.21.15.123:6811/988586998","heartbeat_back_addr":"172.21.15.123:6815/988586998","heartbeat_front_addr":"172.21.15.123:6813/988586998","state":["exists","up"]},{"osd":4,"uuid":"200a8df4-50ef-40a5-b48a-58981269c086","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":167,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6816","nonce":3387813890},{"type":"v1","addr":"172.21.15.184:6817","nonce":3387813890}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6818","nonce":3387813890},{"type":"v1","addr":"172.21.15.184:6819","nonce":3387813890}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6822","nonce":3387813890},{"type":"v1","addr":"172.21.15.184:6823","nonce":3387813890}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6820","nonce":3387813890},{"type":"v1","addr":"172.21.15.184:6821","nonce":3387813890}]},"public_addr":"172.21.15.184:6817/3387813890","cluster_addr":"172.21.15.184:6819/3387813890","heartbeat_back_addr":"172.21.15.184:6823/3387813890","heartbeat_front_addr":"172.21.15.184:6821/3387813890","state":["exists","up"]},{"osd":5,"uuid":"32d5a900-da3d-412c-bc19-13861a2368d8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":167,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6808","nonce":2680916017},{"type":"v1","addr":"172.21.15.184:6809","nonce":2680916017}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6810","nonce":2680916017},{"type":"v1","addr":"172.21.15.184:6811","nonce":2680916017}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6814","nonce":2680916017},{"type":"v1","addr":"172.21.15.184:6815","nonce":2680916017}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6812","nonce":2680916017},{"type":"v1","addr":"172.21.15.184:6813","nonce":2680916017}]},"public_addr":"172.21.15.184:6809/2680916017","cluster_addr":"172.21.15.184:6811/2680916017","heartbeat_back_addr":"172.21.15.184:6815/2680916017","heartbeat_front_addr":"172.21.15.184:6813/2680916017","state":["exists","up"]},{"osd":6,"uuid":"cde6f39e-aa1b-4260-ac19-774ee6aaca72","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":167,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6824","nonce":2183120457},{"type":"v1","addr":"172.21.15.184:6825","nonce":2183120457}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6826","nonce":2183120457},{"type":"v1","addr":"172.21.15.184:6827","nonce":2183120457}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6830","nonce":2183120457},{"type":"v1","addr":"172.21.15.184:6831","nonce":2183120457}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6828","nonce":2183120457},{"type":"v1","addr":"172.21.15.184:6829","nonce":2183120457}]},"public_addr":"172.21.15.184:6825/2183120457","cluster_addr":"172.21.15.184:6827/2183120457","heartbeat_back_addr":"172.21.15.184:6831/2183120457","heartbeat_front_addr":"172.21.15.184:6829/2183120457","state":["exists","up"]},{"osd":7,"uuid":"4f381778-eee1-47c6-804f-df0e6b33e313","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":167,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6800","nonce":4102957285},{"type":"v1","addr":"172.21.15.184:6801","nonce":4102957285}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6802","nonce":4102957285},{"type":"v1","addr":"172.21.15.184:6803","nonce":4102957285}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6806","nonce":4102957285},{"type":"v1","addr":"172.21.15.184:6807","nonce":4102957285}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6804","nonce":4102957285},{"type":"v1","addr":"172.21.15.184:6805","nonce":4102957285}]},"public_addr":"172.21.15.184:6801/4102957285","cluster_addr":"172.21.15.184:6803/4102957285","heartbeat_back_addr":"172.21.15.184:6807/4102957285","heartbeat_front_addr":"172.21.15.184:6805/4102957285","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:35.972210+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.222007+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.198617+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.385663+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.358935+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.156066+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.752973+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:35.682846+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":17,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T15:51:48.759 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:48.761+0000 7f55090be640 1 -- 172.21.15.123:0/2411037354 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f54f00738e0 msgr2=0x7f54f0075da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:51:48.759 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:48.761+0000 7f55090be640 1 --2- 172.21.15.123:0/2411037354 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f54f00738e0 0x7f54f0075da0 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f54f8005870 tx=0x7f54f8005cb0 comp rx=0 tx=0).stop 2024-09-16T15:51:48.759 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:48.761+0000 7f55090be640 1 -- 172.21.15.123:0/2411037354 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5504161620 msgr2=0x7f55040cb980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:51:48.759 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:48.761+0000 7f55090be640 1 --2- 172.21.15.123:0/2411037354 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5504161620 0x7f55040cb980 secure :-1 s=READY pgs=1867 cs=0 l=1 rev1=1 crypto rx=0x7f54fc01b2e0 tx=0x7f54fc0a5a50 comp rx=0 tx=0).stop 2024-09-16T15:51:48.759 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:48.761+0000 7f55090be640 1 -- 172.21.15.123:0/2411037354 shutdown_connections 2024-09-16T15:51:48.759 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:48.761+0000 7f55090be640 1 --2- 172.21.15.123:0/2411037354 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f54f00738e0 0x7f54f0075da0 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:48.759 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:48.761+0000 7f55090be640 1 --2- 172.21.15.123:0/2411037354 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5504161620 0x7f55040cb980 unknown :-1 s=CLOSED pgs=1867 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:48.759 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:48.761+0000 7f55090be640 1 --2- 172.21.15.123:0/2411037354 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f55040d2820 0x7f5504152d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:48.760 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:48.761+0000 7f55090be640 1 --2- 172.21.15.123:0/2411037354 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f55040b2b00 0x7f55041526a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:48.760 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:48.761+0000 7f55090be640 1 -- 172.21.15.123:0/2411037354 >> 172.21.15.123:0/2411037354 conn(0x7f55040d3ea0 msgr2=0x7f550406e810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:51:48.760 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:48.761+0000 7f55090be640 1 -- 172.21.15.123:0/2411037354 shutdown_connections 2024-09-16T15:51:48.760 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:48.761+0000 7f55090be640 1 -- 172.21.15.123:0/2411037354 wait complete. 2024-09-16T15:51:48.776 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-16T15:51:48.896 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:48.897+0000 7fd0a2e39640 1 -- 172.21.15.123:0/1923099454 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd088004060 msgr2=0x7fd088004810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:51:48.897 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:48.897+0000 7fd0a2e39640 1 --2- 172.21.15.123:0/1923099454 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd088004060 0x7fd088004810 secure :-1 s=READY pgs=1869 cs=0 l=1 rev1=1 crypto rx=0x7fd084009b30 tx=0x7fd08401d270 comp rx=0 tx=0).stop 2024-09-16T15:51:48.897 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:48.897+0000 7fd0a13d6640 1 -- 172.21.15.123:0/1923099454 reap_dead start 2024-09-16T15:51:48.897 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:48.897+0000 7fd0a2e39640 1 -- 172.21.15.123:0/1923099454 shutdown_connections 2024-09-16T15:51:48.897 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:48.897+0000 7fd0a2e39640 1 -- 172.21.15.123:0/1923099454 >> 172.21.15.123:0/1923099454 conn(0x7fd09c074bd0 msgr2=0x7fd09c09aa40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:51:48.897 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:48.897+0000 7fd0a2e39640 1 -- 172.21.15.123:0/1923099454 shutdown_connections 2024-09-16T15:51:48.897 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:48.897+0000 7fd0a2e39640 1 -- 172.21.15.123:0/1923099454 wait complete. 2024-09-16T15:51:48.897 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:48.897+0000 7fd0a2e39640 1 Processor -- start 2024-09-16T15:51:48.897 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:48.897+0000 7fd0a2e39640 1 -- start start 2024-09-16T15:51:48.900 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:48.897+0000 7fd0a2e39640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fd088004060 0x7fd09c155b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:51:48.900 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:48.897+0000 7fd0a2e39640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd09c0c9960 0x7fd09c0c63b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:51:48.900 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:48.897+0000 7fd0a2e39640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd09c0c68f0 0x7fd09c09efe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:51:48.900 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:48.897+0000 7fd0a2e39640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fd09c167c00 con 0x7fd09c0c68f0 2024-09-16T15:51:48.900 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:48.897+0000 7fd0a2e39640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fd09c167a80 con 0x7fd09c0c9960 2024-09-16T15:51:48.900 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:48.897+0000 7fd0a2e39640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fd09c167d80 con 0x7fd088004060 2024-09-16T15:51:48.900 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:48.897+0000 7fd0a13d6640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd09c0c68f0 0x7fd09c09efe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:51:48.900 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:48.897+0000 7fd0a13d6640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd09c0c68f0 0x7fd09c09efe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:38592/0 (socket says 172.21.15.123:38592) 2024-09-16T15:51:48.900 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:48.897+0000 7fd0a13d6640 1 -- 172.21.15.123:0/2140416844 learned_addr learned my addr 172.21.15.123:0/2140416844 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:51:48.900 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:48.897+0000 7fd0a0bd5640 1 --2- 172.21.15.123:0/2140416844 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fd088004060 0x7fd09c155b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:51:48.900 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:48.901+0000 7fd09bfff640 1 --2- 172.21.15.123:0/2140416844 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd09c0c9960 0x7fd09c0c63b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:51:48.900 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:48.901+0000 7fd0a13d6640 1 -- 172.21.15.123:0/2140416844 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fd088004060 msgr2=0x7fd09c155b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:51:48.900 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:48.901+0000 7fd0a13d6640 1 --2- 172.21.15.123:0/2140416844 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fd088004060 0x7fd09c155b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:48.901 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:48.901+0000 7fd0a13d6640 1 -- 172.21.15.123:0/2140416844 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd09c0c9960 msgr2=0x7fd09c0c63b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:51:48.901 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:48.901+0000 7fd0a13d6640 1 --2- 172.21.15.123:0/2140416844 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd09c0c9960 0x7fd09c0c63b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:48.901 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:48.901+0000 7fd0a13d6640 1 -- 172.21.15.123:0/2140416844 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd09c1f58e0 con 0x7fd09c0c68f0 2024-09-16T15:51:48.901 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:48.901+0000 7fd09bfff640 1 --2- 172.21.15.123:0/2140416844 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd09c0c9960 0x7fd09c0c63b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T15:51:48.901 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:48.901+0000 7fd0a13d6640 1 --2- 172.21.15.123:0/2140416844 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd09c0c68f0 0x7fd09c09efe0 secure :-1 s=READY pgs=1870 cs=0 l=1 rev1=1 crypto rx=0x7fd09c0c53d0 tx=0x7fd08c00e8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:51:48.901 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:48.901+0000 7fd0997fa640 1 -- 172.21.15.123:0/2140416844 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd08c018070 con 0x7fd09c0c68f0 2024-09-16T15:51:48.901 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:48.901+0000 7fd0997fa640 1 -- 172.21.15.123:0/2140416844 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fd08c0135f0 con 0x7fd09c0c68f0 2024-09-16T15:51:48.901 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:48.901+0000 7fd0a2e39640 1 -- 172.21.15.123:0/2140416844 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd09c1f5bc0 con 0x7fd09c0c68f0 2024-09-16T15:51:48.901 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:48.901+0000 7fd0997fa640 1 -- 172.21.15.123:0/2140416844 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd08c00f040 con 0x7fd09c0c68f0 2024-09-16T15:51:48.901 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:48.901+0000 7fd0a2e39640 1 -- 172.21.15.123:0/2140416844 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd09c1f6180 con 0x7fd09c0c68f0 2024-09-16T15:51:48.902 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:48.901+0000 7fd0a2e39640 1 -- 172.21.15.123:0/2140416844 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd09c0ccab0 con 0x7fd09c0c68f0 2024-09-16T15:51:48.902 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:48.901+0000 7fd0997fa640 1 -- 172.21.15.123:0/2140416844 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 25) ==== 97998+0+0 (secure 0 0 0) 0x7fd08c01b070 con 0x7fd09c0c68f0 2024-09-16T15:51:48.902 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:48.901+0000 7fd0997fa640 1 --2- 172.21.15.123:0/2140416844 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fd07c0738e0 0x7fd07c075da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:51:48.902 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:48.901+0000 7fd0a0bd5640 1 --2- 172.21.15.123:0/2140416844 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fd07c0738e0 0x7fd07c075da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:51:48.902 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:48.901+0000 7fd0997fa640 1 -- 172.21.15.123:0/2140416844 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(168..168 src has 1..168) ==== 5737+0+0 (secure 0 0 0) 0x7fd08c091760 con 0x7fd09c0c68f0 2024-09-16T15:51:48.902 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:48.901+0000 7fd0a0bd5640 1 --2- 172.21.15.123:0/2140416844 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fd07c0738e0 0x7fd07c075da0 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7fd094001340 tx=0x7fd094009040 comp rx=0 tx=0).ready entity=mgr.4150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:51:48.906 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:48.905+0000 7fd0997fa640 1 -- 172.21.15.123:0/2140416844 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7fd08c05dbf0 con 0x7fd09c0c68f0 2024-09-16T15:51:49.078 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:49.077+0000 7fd0a2e39640 1 -- 172.21.15.123:0/2140416844 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fd09c0aec70 con 0x7fd09c0c68f0 2024-09-16T15:51:49.080 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:49.081+0000 7fd0997fa640 1 -- 172.21.15.123:0/2140416844 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v168) ==== 74+0+16860 (secure 0 0 0) 0x7fd08c062620 con 0x7fd09c0c68f0 2024-09-16T15:51:49.080 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-09-16T15:51:49.080 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":168,"fsid":"56077f89-18c8-486f-ae7e-eecc0bb7822c","created":"2024-09-16T15:42:26.364892+0000","modified":"2024-09-16T15:51:47.741386+0000","last_up_change":"2024-09-16T15:42:38.707234+0000","last_in_change":"2024-09-16T15:42:29.423662+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":27,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-16T15:42:39.287764+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":26,"pool_name":"cephfs_metadata","create_time":"2024-09-16T15:51:45.417755+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"168","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":27,"pool_name":"cephfs_data","create_time":"2024-09-16T15:51:46.701679+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"168","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9e7c1955-a607-4671-9be7-f2ef760ceeaa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":167,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6800","nonce":3882044969},{"type":"v1","addr":"172.21.15.123:6801","nonce":3882044969}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6802","nonce":3882044969},{"type":"v1","addr":"172.21.15.123:6803","nonce":3882044969}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6806","nonce":3882044969},{"type":"v1","addr":"172.21.15.123:6807","nonce":3882044969}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6804","nonce":3882044969},{"type":"v1","addr":"172.21.15.123:6805","nonce":3882044969}]},"public_addr":"172.21.15.123:6801/3882044969","cluster_addr":"172.21.15.123:6803/3882044969","heartbeat_back_addr":"172.21.15.123:6807/3882044969","heartbeat_front_addr":"172.21.15.123:6805/3882044969","state":["exists","up"]},{"osd":1,"uuid":"c540cfde-0977-4742-9b17-fd634668e010","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":167,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6816","nonce":4037001850},{"type":"v1","addr":"172.21.15.123:6817","nonce":4037001850}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6818","nonce":4037001850},{"type":"v1","addr":"172.21.15.123:6819","nonce":4037001850}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6822","nonce":4037001850},{"type":"v1","addr":"172.21.15.123:6823","nonce":4037001850}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6820","nonce":4037001850},{"type":"v1","addr":"172.21.15.123:6821","nonce":4037001850}]},"public_addr":"172.21.15.123:6817/4037001850","cluster_addr":"172.21.15.123:6819/4037001850","heartbeat_back_addr":"172.21.15.123:6823/4037001850","heartbeat_front_addr":"172.21.15.123:6821/4037001850","state":["exists","up"]},{"osd":2,"uuid":"9864a496-f169-486b-95c4-1d59162793a7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":167,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6824","nonce":1332143088},{"type":"v1","addr":"172.21.15.123:6825","nonce":1332143088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6826","nonce":1332143088},{"type":"v1","addr":"172.21.15.123:6827","nonce":1332143088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6830","nonce":1332143088},{"type":"v1","addr":"172.21.15.123:6831","nonce":1332143088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6828","nonce":1332143088},{"type":"v1","addr":"172.21.15.123:6829","nonce":1332143088}]},"public_addr":"172.21.15.123:6825/1332143088","cluster_addr":"172.21.15.123:6827/1332143088","heartbeat_back_addr":"172.21.15.123:6831/1332143088","heartbeat_front_addr":"172.21.15.123:6829/1332143088","state":["exists","up"]},{"osd":3,"uuid":"c6d1f817-f074-4325-95e1-578eba13edda","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":167,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6808","nonce":988586998},{"type":"v1","addr":"172.21.15.123:6809","nonce":988586998}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6810","nonce":988586998},{"type":"v1","addr":"172.21.15.123:6811","nonce":988586998}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6814","nonce":988586998},{"type":"v1","addr":"172.21.15.123:6815","nonce":988586998}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6812","nonce":988586998},{"type":"v1","addr":"172.21.15.123:6813","nonce":988586998}]},"public_addr":"172.21.15.123:6809/988586998","cluster_addr":"172.21.15.123:6811/988586998","heartbeat_back_addr":"172.21.15.123:6815/988586998","heartbeat_front_addr":"172.21.15.123:6813/988586998","state":["exists","up"]},{"osd":4,"uuid":"200a8df4-50ef-40a5-b48a-58981269c086","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":167,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6816","nonce":3387813890},{"type":"v1","addr":"172.21.15.184:6817","nonce":3387813890}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6818","nonce":3387813890},{"type":"v1","addr":"172.21.15.184:6819","nonce":3387813890}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6822","nonce":3387813890},{"type":"v1","addr":"172.21.15.184:6823","nonce":3387813890}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6820","nonce":3387813890},{"type":"v1","addr":"172.21.15.184:6821","nonce":3387813890}]},"public_addr":"172.21.15.184:6817/3387813890","cluster_addr":"172.21.15.184:6819/3387813890","heartbeat_back_addr":"172.21.15.184:6823/3387813890","heartbeat_front_addr":"172.21.15.184:6821/3387813890","state":["exists","up"]},{"osd":5,"uuid":"32d5a900-da3d-412c-bc19-13861a2368d8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":167,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6808","nonce":2680916017},{"type":"v1","addr":"172.21.15.184:6809","nonce":2680916017}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6810","nonce":2680916017},{"type":"v1","addr":"172.21.15.184:6811","nonce":2680916017}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6814","nonce":2680916017},{"type":"v1","addr":"172.21.15.184:6815","nonce":2680916017}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6812","nonce":2680916017},{"type":"v1","addr":"172.21.15.184:6813","nonce":2680916017}]},"public_addr":"172.21.15.184:6809/2680916017","cluster_addr":"172.21.15.184:6811/2680916017","heartbeat_back_addr":"172.21.15.184:6815/2680916017","heartbeat_front_addr":"172.21.15.184:6813/2680916017","state":["exists","up"]},{"osd":6,"uuid":"cde6f39e-aa1b-4260-ac19-774ee6aaca72","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":167,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6824","nonce":2183120457},{"type":"v1","addr":"172.21.15.184:6825","nonce":2183120457}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6826","nonce":2183120457},{"type":"v1","addr":"172.21.15.184:6827","nonce":2183120457}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6830","nonce":2183120457},{"type":"v1","addr":"172.21.15.184:6831","nonce":2183120457}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6828","nonce":2183120457},{"type":"v1","addr":"172.21.15.184:6829","nonce":2183120457}]},"public_addr":"172.21.15.184:6825/2183120457","cluster_addr":"172.21.15.184:6827/2183120457","heartbeat_back_addr":"172.21.15.184:6831/2183120457","heartbeat_front_addr":"172.21.15.184:6829/2183120457","state":["exists","up"]},{"osd":7,"uuid":"4f381778-eee1-47c6-804f-df0e6b33e313","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":167,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6800","nonce":4102957285},{"type":"v1","addr":"172.21.15.184:6801","nonce":4102957285}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6802","nonce":4102957285},{"type":"v1","addr":"172.21.15.184:6803","nonce":4102957285}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6806","nonce":4102957285},{"type":"v1","addr":"172.21.15.184:6807","nonce":4102957285}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6804","nonce":4102957285},{"type":"v1","addr":"172.21.15.184:6805","nonce":4102957285}]},"public_addr":"172.21.15.184:6801/4102957285","cluster_addr":"172.21.15.184:6803/4102957285","heartbeat_back_addr":"172.21.15.184:6807/4102957285","heartbeat_front_addr":"172.21.15.184:6805/4102957285","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:35.972210+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.222007+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.198617+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.385663+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.358935+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.156066+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.752973+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:35.682846+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":17,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T15:51:49.082 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:49.081+0000 7fd0a2e39640 1 -- 172.21.15.123:0/2140416844 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fd07c0738e0 msgr2=0x7fd07c075da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:51:49.082 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:49.081+0000 7fd0a2e39640 1 --2- 172.21.15.123:0/2140416844 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fd07c0738e0 0x7fd07c075da0 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7fd094001340 tx=0x7fd094009040 comp rx=0 tx=0).stop 2024-09-16T15:51:49.082 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:49.081+0000 7fd0a2e39640 1 -- 172.21.15.123:0/2140416844 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd09c0c68f0 msgr2=0x7fd09c09efe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:51:49.082 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:49.081+0000 7fd0a2e39640 1 --2- 172.21.15.123:0/2140416844 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd09c0c68f0 0x7fd09c09efe0 secure :-1 s=READY pgs=1870 cs=0 l=1 rev1=1 crypto rx=0x7fd09c0c53d0 tx=0x7fd08c00e8b0 comp rx=0 tx=0).stop 2024-09-16T15:51:49.082 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:49.081+0000 7fd0a2e39640 1 -- 172.21.15.123:0/2140416844 shutdown_connections 2024-09-16T15:51:49.083 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:49.081+0000 7fd0a2e39640 1 --2- 172.21.15.123:0/2140416844 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fd07c0738e0 0x7fd07c075da0 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:49.083 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:49.081+0000 7fd0a2e39640 1 --2- 172.21.15.123:0/2140416844 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd09c0c68f0 0x7fd09c09efe0 unknown :-1 s=CLOSED pgs=1870 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:49.083 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:49.081+0000 7fd0a2e39640 1 --2- 172.21.15.123:0/2140416844 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd09c0c9960 0x7fd09c0c63b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:49.083 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:49.081+0000 7fd0a2e39640 1 --2- 172.21.15.123:0/2140416844 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fd088004060 0x7fd09c155b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:49.083 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:49.081+0000 7fd0a2e39640 1 -- 172.21.15.123:0/2140416844 >> 172.21.15.123:0/2140416844 conn(0x7fd09c074bd0 msgr2=0x7fd09c166fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:51:49.083 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:49.081+0000 7fd0a2e39640 1 -- 172.21.15.123:0/2140416844 shutdown_connections 2024-09-16T15:51:49.083 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:49.081+0000 7fd0a2e39640 1 -- 172.21.15.123:0/2140416844 wait complete. 2024-09-16T15:51:49.100 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-09-16T15:51:49.226 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:49.225+0000 7fb1d06c4640 1 -- 172.21.15.123:0/499230408 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb1c80d2120 msgr2=0x7fb1c80cef00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:51:49.226 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:49.225+0000 7fb1d06c4640 1 --2- 172.21.15.123:0/499230408 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb1c80d2120 0x7fb1c80cef00 secure :-1 s=READY pgs=1871 cs=0 l=1 rev1=1 crypto rx=0x7fb1c80b9ef0 tx=0x7fb1c001cc60 comp rx=0 tx=0).stop 2024-09-16T15:51:49.226 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:49.225+0000 7fb1d06c4640 1 -- 172.21.15.123:0/499230408 shutdown_connections 2024-09-16T15:51:49.227 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:49.225+0000 7fb1d06c4640 1 --2- 172.21.15.123:0/499230408 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb1c80d2120 0x7fb1c80cef00 unknown :-1 s=CLOSED pgs=1871 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:49.227 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:49.225+0000 7fb1d06c4640 1 --2- 172.21.15.123:0/499230408 >> v2:172.21.15.184:3300/0 conn(0x7fb1c80d0f60 0x7fb1c80ce9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:49.227 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:49.225+0000 7fb1d06c4640 1 -- 172.21.15.123:0/499230408 >> 172.21.15.123:0/499230408 conn(0x7fb1c80d3e40 msgr2=0x7fb1c80d4220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:51:49.227 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:49.225+0000 7fb1d06c4640 1 -- 172.21.15.123:0/499230408 shutdown_connections 2024-09-16T15:51:49.227 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:49.225+0000 7fb1d06c4640 1 -- 172.21.15.123:0/499230408 wait complete. 2024-09-16T15:51:49.227 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:49.229+0000 7fb1d06c4640 1 Processor -- start 2024-09-16T15:51:49.227 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:49.229+0000 7fb1d06c4640 1 -- start start 2024-09-16T15:51:49.230 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:49.229+0000 7fb1d06c4640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb1c80d0f60 0x7fb1c81fb950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:51:49.230 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:49.229+0000 7fb1d06c4640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb1c80d2120 0x7fb1c80b1240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:51:49.230 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:49.229+0000 7fb1d06c4640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fb1c81615c0 0x7fb1c8158240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:51:49.230 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:49.229+0000 7fb1d06c4640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fb1c816b410 con 0x7fb1c80d0f60 2024-09-16T15:51:49.230 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:49.229+0000 7fb1d06c4640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fb1c80cfa60 con 0x7fb1c80d2120 2024-09-16T15:51:49.230 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:49.229+0000 7fb1d06c4640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fb1c816b590 con 0x7fb1c81615c0 2024-09-16T15:51:49.230 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:49.229+0000 7fb1ce460640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb1c80d0f60 0x7fb1c81fb950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:51:49.230 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:49.229+0000 7fb1ce460640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb1c80d0f60 0x7fb1c81fb950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:38602/0 (socket says 172.21.15.123:38602) 2024-09-16T15:51:49.230 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:49.229+0000 7fb1ce460640 1 -- 172.21.15.123:0/1664701944 learned_addr learned my addr 172.21.15.123:0/1664701944 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:51:49.230 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:49.229+0000 7fb1cec61640 1 --2- 172.21.15.123:0/1664701944 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fb1c81615c0 0x7fb1c8158240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:51:49.231 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:49.229+0000 7fb1ce460640 1 -- 172.21.15.123:0/1664701944 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fb1c81615c0 msgr2=0x7fb1c8158240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:51:49.231 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:49.229+0000 7fb1ce460640 1 --2- 172.21.15.123:0/1664701944 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fb1c81615c0 0x7fb1c8158240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:49.231 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:49.229+0000 7fb1cdc5f640 1 --2- 172.21.15.123:0/1664701944 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb1c80d2120 0x7fb1c80b1240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:51:49.231 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:49.229+0000 7fb1ce460640 1 -- 172.21.15.123:0/1664701944 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb1c80d2120 msgr2=0x7fb1c80b1240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:51:49.231 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:49.229+0000 7fb1ce460640 1 --2- 172.21.15.123:0/1664701944 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb1c80d2120 0x7fb1c80b1240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:49.231 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:49.229+0000 7fb1ce460640 1 -- 172.21.15.123:0/1664701944 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb1c80c7ee0 con 0x7fb1c80d0f60 2024-09-16T15:51:49.231 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:49.229+0000 7fb1ce460640 1 --2- 172.21.15.123:0/1664701944 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb1c80d0f60 0x7fb1c81fb950 secure :-1 s=READY pgs=1872 cs=0 l=1 rev1=1 crypto rx=0x7fb1c400b410 tx=0x7fb1c400b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:51:49.231 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:49.229+0000 7fb1beffd640 1 -- 172.21.15.123:0/1664701944 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1c4015020 con 0x7fb1c80d0f60 2024-09-16T15:51:49.231 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:49.229+0000 7fb1beffd640 1 -- 172.21.15.123:0/1664701944 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fb1c4011920 con 0x7fb1c80d0f60 2024-09-16T15:51:49.231 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:49.229+0000 7fb1d06c4640 1 -- 172.21.15.123:0/1664701944 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb1c80c81c0 con 0x7fb1c80d0f60 2024-09-16T15:51:49.231 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:49.229+0000 7fb1beffd640 1 -- 172.21.15.123:0/1664701944 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1c4011d70 con 0x7fb1c80d0f60 2024-09-16T15:51:49.231 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:49.229+0000 7fb1d06c4640 1 -- 172.21.15.123:0/1664701944 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb1c80c8780 con 0x7fb1c80d0f60 2024-09-16T15:51:49.232 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:49.229+0000 7fb1d06c4640 1 -- 172.21.15.123:0/1664701944 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb1c80c8950 con 0x7fb1c80d0f60 2024-09-16T15:51:49.232 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:49.233+0000 7fb1beffd640 1 -- 172.21.15.123:0/1664701944 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 25) ==== 97998+0+0 (secure 0 0 0) 0x7fb1c4018020 con 0x7fb1c80d0f60 2024-09-16T15:51:49.232 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:49.233+0000 7fb1beffd640 1 --2- 172.21.15.123:0/1664701944 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fb1ac073910 0x7fb1ac075dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:51:49.233 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:49.233+0000 7fb1beffd640 1 -- 172.21.15.123:0/1664701944 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(168..168 src has 1..168) ==== 5737+0+0 (secure 0 0 0) 0x7fb1c4092d50 con 0x7fb1c80d0f60 2024-09-16T15:51:49.233 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:49.233+0000 7fb1cdc5f640 1 --2- 172.21.15.123:0/1664701944 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fb1ac073910 0x7fb1ac075dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:51:49.233 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:49.233+0000 7fb1cdc5f640 1 --2- 172.21.15.123:0/1664701944 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fb1ac073910 0x7fb1ac075dd0 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7fb1c80b4930 tx=0x7fb1b8005c70 comp rx=0 tx=0).ready entity=mgr.4150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:51:49.235 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:49.237+0000 7fb1beffd640 1 -- 172.21.15.123:0/1664701944 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7fb1c401f020 con 0x7fb1c80d0f60 2024-09-16T15:51:49.439 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:49.441+0000 7fb1d06c4640 1 -- 172.21.15.123:0/1664701944 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) -- 0x7fb1c80d0850 con 0x7fb1c80d0f60 2024-09-16T15:51:50.430 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:50.429+0000 7fb1beffd640 1 -- 172.21.15.123:0/1664701944 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v114) ==== 118+0+0 (secure 0 0 0) 0x7fb1c405f810 con 0x7fb1c80d0f60 2024-09-16T15:51:50.436 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:50.437+0000 7fb1d06c4640 1 -- 172.21.15.123:0/1664701944 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fb1ac073910 msgr2=0x7fb1ac075dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:51:50.436 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:50.437+0000 7fb1d06c4640 1 --2- 172.21.15.123:0/1664701944 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fb1ac073910 0x7fb1ac075dd0 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7fb1c80b4930 tx=0x7fb1b8005c70 comp rx=0 tx=0).stop 2024-09-16T15:51:50.437 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:50.437+0000 7fb1d06c4640 1 -- 172.21.15.123:0/1664701944 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb1c80d0f60 msgr2=0x7fb1c81fb950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:51:50.437 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:50.437+0000 7fb1d06c4640 1 --2- 172.21.15.123:0/1664701944 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb1c80d0f60 0x7fb1c81fb950 secure :-1 s=READY pgs=1872 cs=0 l=1 rev1=1 crypto rx=0x7fb1c400b410 tx=0x7fb1c400b8d0 comp rx=0 tx=0).stop 2024-09-16T15:51:50.437 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:50.437+0000 7fb1d06c4640 1 -- 172.21.15.123:0/1664701944 shutdown_connections 2024-09-16T15:51:50.437 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:50.437+0000 7fb1d06c4640 1 --2- 172.21.15.123:0/1664701944 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fb1ac073910 0x7fb1ac075dd0 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:50.437 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:50.437+0000 7fb1d06c4640 1 --2- 172.21.15.123:0/1664701944 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fb1c81615c0 0x7fb1c8158240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:50.437 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:50.437+0000 7fb1d06c4640 1 --2- 172.21.15.123:0/1664701944 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb1c80d2120 0x7fb1c80b1240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:50.437 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:50.437+0000 7fb1d06c4640 1 --2- 172.21.15.123:0/1664701944 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb1c80d0f60 0x7fb1c81fb950 unknown :-1 s=CLOSED pgs=1872 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:50.437 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:50.437+0000 7fb1d06c4640 1 -- 172.21.15.123:0/1664701944 >> 172.21.15.123:0/1664701944 conn(0x7fb1c80d3e40 msgr2=0x7fb1c815f060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:51:50.437 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:50.437+0000 7fb1d06c4640 1 -- 172.21.15.123:0/1664701944 shutdown_connections 2024-09-16T15:51:50.437 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:50.437+0000 7fb1d06c4640 1 -- 172.21.15.123:0/1664701944 wait complete. 2024-09-16T15:51:50.460 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-09-16T15:51:50.460 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-09-16T15:51:50.588 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:50.589+0000 7f4447e03640 1 -- 172.21.15.123:0/2964337079 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f44400d2120 msgr2=0x7f44400cf0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:51:50.588 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:50.589+0000 7f4447e03640 1 --2- 172.21.15.123:0/2964337079 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f44400d2120 0x7f44400cf0d0 secure :-1 s=READY pgs=1874 cs=0 l=1 rev1=1 crypto rx=0x7f4438009bc0 tx=0x7f443801c9a0 comp rx=0 tx=0).stop 2024-09-16T15:51:50.588 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:50.589+0000 7f4447e03640 1 -- 172.21.15.123:0/2964337079 shutdown_connections 2024-09-16T15:51:50.588 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:50.589+0000 7f4447e03640 1 --2- 172.21.15.123:0/2964337079 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f44400d2120 0x7f44400cf0d0 unknown :-1 s=CLOSED pgs=1874 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:50.588 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:50.589+0000 7f4447e03640 1 -- 172.21.15.123:0/2964337079 >> 172.21.15.123:0/2964337079 conn(0x7f44400d3e40 msgr2=0x7f44400d4220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:51:50.588 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:50.589+0000 7f4447e03640 1 -- 172.21.15.123:0/2964337079 shutdown_connections 2024-09-16T15:51:50.588 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:50.589+0000 7f4447e03640 1 -- 172.21.15.123:0/2964337079 wait complete. 2024-09-16T15:51:50.589 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:50.589+0000 7f4447e03640 1 Processor -- start 2024-09-16T15:51:50.589 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:50.589+0000 7f4447e03640 1 -- start start 2024-09-16T15:51:50.590 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:50.589+0000 7f4447e03640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f44400d0f60 0x7f44401529e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:51:50.590 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:50.589+0000 7f4447e03640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f44400d2120 0x7f44400cc070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:51:50.591 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:50.589+0000 7f4447e03640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f44401615c0 0x7f44400cc5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:51:50.591 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:50.589+0000 7f4447e03640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f444016c7b0 con 0x7f44400d2120 2024-09-16T15:51:50.591 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:50.589+0000 7f4447e03640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f444016c630 con 0x7f44400d0f60 2024-09-16T15:51:50.591 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:50.589+0000 7f4447e03640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f444016c930 con 0x7f44401615c0 2024-09-16T15:51:50.591 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:50.589+0000 7f444539e640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f44400d2120 0x7f44400cc070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:51:50.591 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:50.589+0000 7f444539e640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f44400d2120 0x7f44400cc070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:38626/0 (socket says 172.21.15.123:38626) 2024-09-16T15:51:50.591 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:50.589+0000 7f444539e640 1 -- 172.21.15.123:0/20652046 learned_addr learned my addr 172.21.15.123:0/20652046 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:51:50.591 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:50.589+0000 7f44463a0640 1 --2- 172.21.15.123:0/20652046 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f44401615c0 0x7f44400cc5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:51:50.591 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:50.589+0000 7f4445b9f640 1 --2- 172.21.15.123:0/20652046 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f44400d0f60 0x7f44401529e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:51:50.591 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:50.589+0000 7f444539e640 1 -- 172.21.15.123:0/20652046 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f44401615c0 msgr2=0x7f44400cc5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:51:50.591 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:50.589+0000 7f444539e640 1 --2- 172.21.15.123:0/20652046 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f44401615c0 0x7f44400cc5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:50.591 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:50.589+0000 7f444539e640 1 -- 172.21.15.123:0/20652046 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f44400d0f60 msgr2=0x7f44401529e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:51:50.591 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:50.589+0000 7f444539e640 1 --2- 172.21.15.123:0/20652046 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f44400d0f60 0x7f44401529e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:50.591 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:50.589+0000 7f444539e640 1 -- 172.21.15.123:0/20652046 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f44400ccc10 con 0x7f44400d2120 2024-09-16T15:51:50.592 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:50.589+0000 7f4445b9f640 1 --2- 172.21.15.123:0/20652046 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f44400d0f60 0x7f44401529e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T15:51:50.592 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:50.589+0000 7f444539e640 1 --2- 172.21.15.123:0/20652046 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f44400d2120 0x7f44400cc070 secure :-1 s=READY pgs=1875 cs=0 l=1 rev1=1 crypto rx=0x7f4430001230 tx=0x7f443000cbb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:51:50.592 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:50.589+0000 7f44367fc640 1 -- 172.21.15.123:0/20652046 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4430016020 con 0x7f44400d2120 2024-09-16T15:51:50.592 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:50.589+0000 7f44367fc640 1 -- 172.21.15.123:0/20652046 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f443000a930 con 0x7f44400d2120 2024-09-16T15:51:50.592 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:50.593+0000 7f4447e03640 1 -- 172.21.15.123:0/20652046 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f44400ccef0 con 0x7f44400d2120 2024-09-16T15:51:50.592 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:50.593+0000 7f44367fc640 1 -- 172.21.15.123:0/20652046 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f443000ad80 con 0x7f44400d2120 2024-09-16T15:51:50.592 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:50.593+0000 7f4447e03640 1 -- 172.21.15.123:0/20652046 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f44400cd4b0 con 0x7f44400d2120 2024-09-16T15:51:50.593 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:50.593+0000 7f4447e03640 1 -- 172.21.15.123:0/20652046 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f44400d0850 con 0x7f44400d2120 2024-09-16T15:51:50.593 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:50.593+0000 7f44367fc640 1 -- 172.21.15.123:0/20652046 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 25) ==== 97998+0+0 (secure 0 0 0) 0x7f4430019020 con 0x7f44400d2120 2024-09-16T15:51:50.593 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:50.593+0000 7f44367fc640 1 --2- 172.21.15.123:0/20652046 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f44240738e0 0x7f4424075da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:51:50.593 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:50.593+0000 7f4445b9f640 1 --2- 172.21.15.123:0/20652046 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f44240738e0 0x7f4424075da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:51:50.593 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:50.593+0000 7f44367fc640 1 -- 172.21.15.123:0/20652046 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(169..169 src has 1..169) ==== 5737+0+0 (secure 0 0 0) 0x7f4430092fa0 con 0x7f44400d2120 2024-09-16T15:51:50.593 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:50.593+0000 7f4445b9f640 1 --2- 172.21.15.123:0/20652046 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f44240738e0 0x7f4424075da0 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f443c000f70 tx=0x7f443c009040 comp rx=0 tx=0).ready entity=mgr.4150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:51:50.596 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:50.597+0000 7f44367fc640 1 -- 172.21.15.123:0/20652046 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f4430020030 con 0x7f44400d2120 2024-09-16T15:51:50.789 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:50.789+0000 7f4447e03640 1 -- 172.21.15.123:0/20652046 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) -- 0x7f444015f990 con 0x7f44400d2120 2024-09-16T15:51:51.440 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:51.441+0000 7f44367fc640 1 -- 172.21.15.123:0/20652046 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v115) ==== 122+0+0 (secure 0 0 0) 0x7f443005fa60 con 0x7f44400d2120 2024-09-16T15:51:51.444 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:51.445+0000 7f4447e03640 1 -- 172.21.15.123:0/20652046 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f44240738e0 msgr2=0x7f4424075da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:51:51.444 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:51.445+0000 7f4447e03640 1 --2- 172.21.15.123:0/20652046 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f44240738e0 0x7f4424075da0 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f443c000f70 tx=0x7f443c009040 comp rx=0 tx=0).stop 2024-09-16T15:51:51.444 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:51.445+0000 7f4447e03640 1 -- 172.21.15.123:0/20652046 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f44400d2120 msgr2=0x7f44400cc070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:51:51.444 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:51.445+0000 7f4447e03640 1 --2- 172.21.15.123:0/20652046 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f44400d2120 0x7f44400cc070 secure :-1 s=READY pgs=1875 cs=0 l=1 rev1=1 crypto rx=0x7f4430001230 tx=0x7f443000cbb0 comp rx=0 tx=0).stop 2024-09-16T15:51:51.444 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:51.445+0000 7f4447e03640 1 -- 172.21.15.123:0/20652046 shutdown_connections 2024-09-16T15:51:51.444 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:51.445+0000 7f4447e03640 1 --2- 172.21.15.123:0/20652046 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f44240738e0 0x7f4424075da0 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:51.444 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:51.445+0000 7f4447e03640 1 --2- 172.21.15.123:0/20652046 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f44401615c0 0x7f44400cc5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:51.445 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:51.445+0000 7f4447e03640 1 --2- 172.21.15.123:0/20652046 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f44400d2120 0x7f44400cc070 unknown :-1 s=CLOSED pgs=1875 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:51.445 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:51.445+0000 7f4447e03640 1 --2- 172.21.15.123:0/20652046 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f44400d0f60 0x7f44401529e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:51.445 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:51.445+0000 7f4447e03640 1 -- 172.21.15.123:0/20652046 >> 172.21.15.123:0/20652046 conn(0x7f44400d3e40 msgr2=0x7f444006e2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:51:51.445 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:51.445+0000 7f4447e03640 1 -- 172.21.15.123:0/20652046 shutdown_connections 2024-09-16T15:51:51.445 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:51.445+0000 7f4447e03640 1 -- 172.21.15.123:0/20652046 wait complete. 2024-09-16T15:51:51.463 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-09-16T15:51:51.463 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-16T15:51:51.581 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:51.581+0000 7f65b896b640 1 -- 172.21.15.123:0/2241546602 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f65b40726c0 msgr2=0x7f65b40d0630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:51:51.581 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:51.581+0000 7f65b896b640 1 --2- 172.21.15.123:0/2241546602 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f65b40726c0 0x7f65b40d0630 secure :-1 s=READY pgs=1877 cs=0 l=1 rev1=1 crypto rx=0x7f65b40cbd60 tx=0x7f659c01c820 comp rx=0 tx=0).stop 2024-09-16T15:51:51.581 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:51.581+0000 7f65b896b640 1 -- 172.21.15.123:0/2241546602 shutdown_connections 2024-09-16T15:51:51.581 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:51.581+0000 7f65b896b640 1 --2- 172.21.15.123:0/2241546602 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f65b40726c0 0x7f65b40d0630 unknown :-1 s=CLOSED pgs=1877 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:51.581 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:51.581+0000 7f65b896b640 1 -- 172.21.15.123:0/2241546602 >> 172.21.15.123:0/2241546602 conn(0x7f65b40cfb10 msgr2=0x7f65b40cff20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:51:51.581 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:51.581+0000 7f65b896b640 1 -- 172.21.15.123:0/2241546602 shutdown_connections 2024-09-16T15:51:51.581 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:51.581+0000 7f65b896b640 1 -- 172.21.15.123:0/2241546602 wait complete. 2024-09-16T15:51:51.582 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:51.581+0000 7f65b896b640 1 Processor -- start 2024-09-16T15:51:51.582 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:51.581+0000 7f65b896b640 1 -- start start 2024-09-16T15:51:51.584 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:51.585+0000 7f65b896b640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f65b40726c0 0x7f65b415d180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:51:51.584 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:51.585+0000 7f65b896b640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f65b4074600 0x7f65b40c9370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:51:51.584 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:51.585+0000 7f65b896b640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f65b415bca0 0x7f65b415e080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:51:51.584 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:51.585+0000 7f65b896b640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f65b4169df0 con 0x7f65b4074600 2024-09-16T15:51:51.584 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:51.585+0000 7f65b896b640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f65b4169c70 con 0x7f65b40726c0 2024-09-16T15:51:51.584 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:51.585+0000 7f65b896b640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f65b4169f70 con 0x7f65b415bca0 2024-09-16T15:51:51.584 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:51.585+0000 7f65b1d9b640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f65b4074600 0x7f65b40c9370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:51:51.584 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:51.585+0000 7f65b1d9b640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f65b4074600 0x7f65b40c9370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:38644/0 (socket says 172.21.15.123:38644) 2024-09-16T15:51:51.585 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:51.585+0000 7f65b1d9b640 1 -- 172.21.15.123:0/1902155076 learned_addr learned my addr 172.21.15.123:0/1902155076 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:51:51.585 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:51.585+0000 7f65b259c640 1 --2- 172.21.15.123:0/1902155076 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f65b40726c0 0x7f65b415d180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:51:51.585 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:51.585+0000 7f65b2d9d640 1 --2- 172.21.15.123:0/1902155076 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f65b415bca0 0x7f65b415e080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:51:51.585 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:51.585+0000 7f65b1d9b640 1 -- 172.21.15.123:0/1902155076 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f65b415bca0 msgr2=0x7f65b415e080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:51:51.585 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:51.585+0000 7f65b1d9b640 1 --2- 172.21.15.123:0/1902155076 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f65b415bca0 0x7f65b415e080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:51.585 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:51.585+0000 7f65b1d9b640 1 -- 172.21.15.123:0/1902155076 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f65b40726c0 msgr2=0x7f65b415d180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:51:51.585 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:51.585+0000 7f65b1d9b640 1 --2- 172.21.15.123:0/1902155076 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f65b40726c0 0x7f65b415d180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:51.585 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:51.585+0000 7f65b1d9b640 1 -- 172.21.15.123:0/1902155076 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f65b415eb40 con 0x7f65b4074600 2024-09-16T15:51:51.585 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:51.585+0000 7f65b2d9d640 1 --2- 172.21.15.123:0/1902155076 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f65b415bca0 0x7f65b415e080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T15:51:51.585 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:51.585+0000 7f65b259c640 1 --2- 172.21.15.123:0/1902155076 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f65b40726c0 0x7f65b415d180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T15:51:51.585 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:51.585+0000 7f65b1d9b640 1 --2- 172.21.15.123:0/1902155076 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f65b4074600 0x7f65b40c9370 secure :-1 s=READY pgs=1878 cs=0 l=1 rev1=1 crypto rx=0x7f659c0b0040 tx=0x7f659c0040c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:51:51.585 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:51.585+0000 7f65aaffd640 1 -- 172.21.15.123:0/1902155076 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f659c0b2040 con 0x7f65b4074600 2024-09-16T15:51:51.585 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:51.585+0000 7f65aaffd640 1 -- 172.21.15.123:0/1902155076 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f659c003600 con 0x7f65b4074600 2024-09-16T15:51:51.585 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:51.585+0000 7f65aaffd640 1 -- 172.21.15.123:0/1902155076 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f659c003a50 con 0x7f65b4074600 2024-09-16T15:51:51.586 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:51.585+0000 7f65b896b640 1 -- 172.21.15.123:0/1902155076 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f65b415c420 con 0x7f65b4074600 2024-09-16T15:51:51.586 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:51.585+0000 7f65b896b640 1 -- 172.21.15.123:0/1902155076 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f65b415cc60 con 0x7f65b4074600 2024-09-16T15:51:51.586 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:51.585+0000 7f65a8ff9640 1 -- 172.21.15.123:0/1902155076 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f658c0052a0 con 0x7f65b4074600 2024-09-16T15:51:51.587 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:51.585+0000 7f65aaffd640 1 -- 172.21.15.123:0/1902155076 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 25) ==== 97998+0+0 (secure 0 0 0) 0x7f659c003bf0 con 0x7f65b4074600 2024-09-16T15:51:51.587 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:51.585+0000 7f65aaffd640 1 --2- 172.21.15.123:0/1902155076 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f65940738e0 0x7f6594075da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:51:51.587 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:51.585+0000 7f65aaffd640 1 -- 172.21.15.123:0/1902155076 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(169..169 src has 1..169) ==== 5737+0+0 (secure 0 0 0) 0x7f659c12c7c0 con 0x7f65b4074600 2024-09-16T15:51:51.587 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:51.585+0000 7f65b259c640 1 --2- 172.21.15.123:0/1902155076 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f65940738e0 0x7f6594075da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:51:51.587 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:51.585+0000 7f65b259c640 1 --2- 172.21.15.123:0/1902155076 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f65940738e0 0x7f6594075da0 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f65a40012f0 tx=0x7f65a4009040 comp rx=0 tx=0).ready entity=mgr.4150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:51:51.590 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:51.589+0000 7f65aaffd640 1 -- 172.21.15.123:0/1902155076 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f65b415c420 con 0x7f65b4074600 2024-09-16T15:51:51.782 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:51.781+0000 7f65a8ff9640 1 -- 172.21.15.123:0/1902155076 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f658c005860 con 0x7f65b4074600 2024-09-16T15:51:51.783 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:51.785+0000 7f65aaffd640 1 -- 172.21.15.123:0/1902155076 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 115 v115) ==== 95+0+5239 (secure 0 0 0) 0x7f659c0bd060 con 0x7f65b4074600 2024-09-16T15:51:51.784 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-09-16T15:51:51.784 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":115,"btime":"2024-09-16T15:51:51:442311+0000","default_fscid":13,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6314,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6833/2070840764","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":2070840764},{"type":"v1","addr":"172.21.15.184:6833","nonce":2070840764}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":110},{"gid":6976,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6835/3881288354","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":3881288354},{"type":"v1","addr":"172.21.15.184:6835","nonce":3881288354}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":110},{"gid":9219,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6833/519008782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6832","nonce":519008782},{"type":"v1","addr":"172.21.15.123:6833","nonce":519008782}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":110}],"filesystems":[{"mdsmap":{"epoch":115,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-16T15:51:47.741329+0000","modified":"2024-09-16T15:51:50.794388+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":9234},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9234":{"gid":9234,"name":"a","rank":0,"incarnation":111,"state":"up:active","state_seq":3,"addr":"172.21.15.123:6835/370478759","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6834","nonce":370478759},{"type":"v1","addr":"172.21.15.123:6835","nonce":370478759}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[27],"metadata_pool":26,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9234,"qdb_cluster":[9234]},"id":13}]} 2024-09-16T15:51:51.784 INFO:teuthology.orchestra.run.smithi123.stderr:dumped fsmap epoch 115 2024-09-16T15:51:51.786 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:51.785+0000 7f65a8ff9640 1 -- 172.21.15.123:0/1902155076 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f65940738e0 msgr2=0x7f6594075da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:51:51.786 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:51.785+0000 7f65a8ff9640 1 --2- 172.21.15.123:0/1902155076 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f65940738e0 0x7f6594075da0 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f65a40012f0 tx=0x7f65a4009040 comp rx=0 tx=0).stop 2024-09-16T15:51:51.786 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:51.785+0000 7f65a8ff9640 1 -- 172.21.15.123:0/1902155076 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f65b4074600 msgr2=0x7f65b40c9370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:51:51.787 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:51.785+0000 7f65a8ff9640 1 --2- 172.21.15.123:0/1902155076 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f65b4074600 0x7f65b40c9370 secure :-1 s=READY pgs=1878 cs=0 l=1 rev1=1 crypto rx=0x7f659c0b0040 tx=0x7f659c0040c0 comp rx=0 tx=0).stop 2024-09-16T15:51:51.787 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:51.785+0000 7f65a8ff9640 1 -- 172.21.15.123:0/1902155076 shutdown_connections 2024-09-16T15:51:51.787 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:51.785+0000 7f65a8ff9640 1 --2- 172.21.15.123:0/1902155076 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f65940738e0 0x7f6594075da0 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:51.787 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:51.785+0000 7f65a8ff9640 1 --2- 172.21.15.123:0/1902155076 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f65b415bca0 0x7f65b415e080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:51.787 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:51.785+0000 7f65a8ff9640 1 --2- 172.21.15.123:0/1902155076 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f65b4074600 0x7f65b40c9370 unknown :-1 s=CLOSED pgs=1878 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:51.787 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:51.785+0000 7f65a8ff9640 1 --2- 172.21.15.123:0/1902155076 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f65b40726c0 0x7f65b415d180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:51.787 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:51.785+0000 7f65a8ff9640 1 -- 172.21.15.123:0/1902155076 >> 172.21.15.123:0/1902155076 conn(0x7f65b40cfb10 msgr2=0x7f65b4075ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:51:51.787 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:51.785+0000 7f65a8ff9640 1 -- 172.21.15.123:0/1902155076 shutdown_connections 2024-09-16T15:51:51.787 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:51.785+0000 7f65a8ff9640 1 -- 172.21.15.123:0/1902155076 wait complete. 2024-09-16T15:51:51.802 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-16T15:51:51.913 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:51.913+0000 7ff014d12640 1 -- 172.21.15.123:0/2949763901 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff0100cf9e0 msgr2=0x7ff010073660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:51:51.913 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:51.913+0000 7ff014d12640 1 --2- 172.21.15.123:0/2949763901 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff0100cf9e0 0x7ff010073660 secure :-1 s=READY pgs=1880 cs=0 l=1 rev1=1 crypto rx=0x7ff008009bc0 tx=0x7ff00801cab0 comp rx=0 tx=0).stop 2024-09-16T15:51:51.913 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:51.913+0000 7ff014d12640 1 -- 172.21.15.123:0/2949763901 shutdown_connections 2024-09-16T15:51:51.913 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:51.913+0000 7ff014d12640 1 --2- 172.21.15.123:0/2949763901 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff0100cf9e0 0x7ff010073660 unknown :-1 s=CLOSED pgs=1880 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:51.913 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:51.913+0000 7ff014d12640 1 --2- 172.21.15.123:0/2949763901 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7ff01015ab20 0x7ff01015e830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:51.914 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:51.913+0000 7ff014d12640 1 -- 172.21.15.123:0/2949763901 >> 172.21.15.123:0/2949763901 conn(0x7ff0100d0b70 msgr2=0x7ff0100d0f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:51:51.914 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:51.913+0000 7ff014d12640 1 -- 172.21.15.123:0/2949763901 shutdown_connections 2024-09-16T15:51:51.914 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:51.913+0000 7ff014d12640 1 -- 172.21.15.123:0/2949763901 wait complete. 2024-09-16T15:51:51.914 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:51.913+0000 7ff014d12640 1 Processor -- start 2024-09-16T15:51:51.914 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:51.913+0000 7ff014d12640 1 -- start start 2024-09-16T15:51:51.916 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:51.917+0000 7ff014d12640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff0100cf9e0 0x7ff0100af9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:51:51.916 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:51.917+0000 7ff014d12640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7ff01015ab20 0x7ff0100b2770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:51:51.916 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:51.917+0000 7ff014d12640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff01015af00 0x7ff0100b2cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:51:51.916 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:51.917+0000 7ff014d12640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7ff010168010 con 0x7ff0100cf9e0 2024-09-16T15:51:51.916 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:51.917+0000 7ff014d12640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7ff010167e90 con 0x7ff01015af00 2024-09-16T15:51:51.916 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:51.917+0000 7ff014d12640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7ff010168190 con 0x7ff01015ab20 2024-09-16T15:51:51.916 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:51.917+0000 7ff00e59c640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff0100cf9e0 0x7ff0100af9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:51:51.916 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:51.917+0000 7ff00e59c640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff0100cf9e0 0x7ff0100af9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:38662/0 (socket says 172.21.15.123:38662) 2024-09-16T15:51:51.916 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:51.917+0000 7ff00e59c640 1 -- 172.21.15.123:0/3800059334 learned_addr learned my addr 172.21.15.123:0/3800059334 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:51:51.917 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:51.917+0000 7ff00e59c640 1 -- 172.21.15.123:0/3800059334 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7ff01015ab20 msgr2=0x7ff0100b2770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:51:51.917 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:51.917+0000 7ff00e59c640 1 --2- 172.21.15.123:0/3800059334 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7ff01015ab20 0x7ff0100b2770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:51.917 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:51.917+0000 7ff00e59c640 1 -- 172.21.15.123:0/3800059334 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff01015af00 msgr2=0x7ff0100b2cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:51:51.917 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:51.917+0000 7ff00e59c640 1 --2- 172.21.15.123:0/3800059334 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff01015af00 0x7ff0100b2cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:51.917 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:51.917+0000 7ff00e59c640 1 -- 172.21.15.123:0/3800059334 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff01015cf80 con 0x7ff0100cf9e0 2024-09-16T15:51:51.917 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:51.917+0000 7ff00e59c640 1 --2- 172.21.15.123:0/3800059334 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff0100cf9e0 0x7ff0100af9a0 secure :-1 s=READY pgs=1881 cs=0 l=1 rev1=1 crypto rx=0x7ff00000c970 tx=0x7ff00000ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:51:51.917 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:51.917+0000 7feffeffd640 1 -- 172.21.15.123:0/3800059334 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff000007c70 con 0x7ff0100cf9e0 2024-09-16T15:51:51.917 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:51.917+0000 7feffeffd640 1 -- 172.21.15.123:0/3800059334 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7ff00000f450 con 0x7ff0100cf9e0 2024-09-16T15:51:51.917 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:51.917+0000 7ff014d12640 1 -- 172.21.15.123:0/3800059334 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff0100720d0 con 0x7ff0100cf9e0 2024-09-16T15:51:51.917 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:51.917+0000 7feffeffd640 1 -- 172.21.15.123:0/3800059334 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff00000f8a0 con 0x7ff0100cf9e0 2024-09-16T15:51:51.917 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:51.917+0000 7ff014d12640 1 -- 172.21.15.123:0/3800059334 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff010155820 con 0x7ff0100cf9e0 2024-09-16T15:51:51.918 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:51.917+0000 7ff014d12640 1 -- 172.21.15.123:0/3800059334 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff010073660 con 0x7ff0100cf9e0 2024-09-16T15:51:51.918 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:51.917+0000 7feffeffd640 1 -- 172.21.15.123:0/3800059334 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 25) ==== 97998+0+0 (secure 0 0 0) 0x7ff000018400 con 0x7ff0100cf9e0 2024-09-16T15:51:51.918 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:51.917+0000 7feffeffd640 1 --2- 172.21.15.123:0/3800059334 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7feff00738e0 0x7feff0075da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:51:51.918 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:51.917+0000 7feffeffd640 1 -- 172.21.15.123:0/3800059334 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(169..169 src has 1..169) ==== 5737+0+0 (secure 0 0 0) 0x7ff000093840 con 0x7ff0100cf9e0 2024-09-16T15:51:51.918 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:51.917+0000 7ff00dd9b640 1 --2- 172.21.15.123:0/3800059334 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7feff00738e0 0x7feff0075da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:51:51.919 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:51.921+0000 7ff00dd9b640 1 --2- 172.21.15.123:0/3800059334 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7feff00738e0 0x7feff0075da0 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7ff0100b6ac0 tx=0x7ff004007a00 comp rx=0 tx=0).ready entity=mgr.4150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:51:51.921 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:51.921+0000 7feffeffd640 1 -- 172.21.15.123:0/3800059334 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7ff000026020 con 0x7ff0100cf9e0 2024-09-16T15:51:52.082 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:52.081+0000 7ff014d12640 1 -- 172.21.15.123:0/3800059334 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7ff0100b6af0 con 0x7ff0100cf9e0 2024-09-16T15:51:52.084 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:52.085+0000 7feffeffd640 1 -- 172.21.15.123:0/3800059334 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v169) ==== 74+0+16847 (secure 0 0 0) 0x7ff000014030 con 0x7ff0100cf9e0 2024-09-16T15:51:52.084 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-09-16T15:51:52.084 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":169,"fsid":"56077f89-18c8-486f-ae7e-eecc0bb7822c","created":"2024-09-16T15:42:26.364892+0000","modified":"2024-09-16T15:51:49.376383+0000","last_up_change":"2024-09-16T15:42:38.707234+0000","last_in_change":"2024-09-16T15:42:29.423662+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":27,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-16T15:42:39.287764+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":26,"pool_name":"cephfs_metadata","create_time":"2024-09-16T15:51:45.417755+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"168","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":27,"pool_name":"cephfs_data","create_time":"2024-09-16T15:51:46.701679+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"169","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9e7c1955-a607-4671-9be7-f2ef760ceeaa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":167,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6800","nonce":3882044969},{"type":"v1","addr":"172.21.15.123:6801","nonce":3882044969}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6802","nonce":3882044969},{"type":"v1","addr":"172.21.15.123:6803","nonce":3882044969}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6806","nonce":3882044969},{"type":"v1","addr":"172.21.15.123:6807","nonce":3882044969}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6804","nonce":3882044969},{"type":"v1","addr":"172.21.15.123:6805","nonce":3882044969}]},"public_addr":"172.21.15.123:6801/3882044969","cluster_addr":"172.21.15.123:6803/3882044969","heartbeat_back_addr":"172.21.15.123:6807/3882044969","heartbeat_front_addr":"172.21.15.123:6805/3882044969","state":["exists","up"]},{"osd":1,"uuid":"c540cfde-0977-4742-9b17-fd634668e010","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":167,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6816","nonce":4037001850},{"type":"v1","addr":"172.21.15.123:6817","nonce":4037001850}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6818","nonce":4037001850},{"type":"v1","addr":"172.21.15.123:6819","nonce":4037001850}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6822","nonce":4037001850},{"type":"v1","addr":"172.21.15.123:6823","nonce":4037001850}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6820","nonce":4037001850},{"type":"v1","addr":"172.21.15.123:6821","nonce":4037001850}]},"public_addr":"172.21.15.123:6817/4037001850","cluster_addr":"172.21.15.123:6819/4037001850","heartbeat_back_addr":"172.21.15.123:6823/4037001850","heartbeat_front_addr":"172.21.15.123:6821/4037001850","state":["exists","up"]},{"osd":2,"uuid":"9864a496-f169-486b-95c4-1d59162793a7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":167,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6824","nonce":1332143088},{"type":"v1","addr":"172.21.15.123:6825","nonce":1332143088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6826","nonce":1332143088},{"type":"v1","addr":"172.21.15.123:6827","nonce":1332143088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6830","nonce":1332143088},{"type":"v1","addr":"172.21.15.123:6831","nonce":1332143088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6828","nonce":1332143088},{"type":"v1","addr":"172.21.15.123:6829","nonce":1332143088}]},"public_addr":"172.21.15.123:6825/1332143088","cluster_addr":"172.21.15.123:6827/1332143088","heartbeat_back_addr":"172.21.15.123:6831/1332143088","heartbeat_front_addr":"172.21.15.123:6829/1332143088","state":["exists","up"]},{"osd":3,"uuid":"c6d1f817-f074-4325-95e1-578eba13edda","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":167,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6808","nonce":988586998},{"type":"v1","addr":"172.21.15.123:6809","nonce":988586998}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6810","nonce":988586998},{"type":"v1","addr":"172.21.15.123:6811","nonce":988586998}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6814","nonce":988586998},{"type":"v1","addr":"172.21.15.123:6815","nonce":988586998}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6812","nonce":988586998},{"type":"v1","addr":"172.21.15.123:6813","nonce":988586998}]},"public_addr":"172.21.15.123:6809/988586998","cluster_addr":"172.21.15.123:6811/988586998","heartbeat_back_addr":"172.21.15.123:6815/988586998","heartbeat_front_addr":"172.21.15.123:6813/988586998","state":["exists","up"]},{"osd":4,"uuid":"200a8df4-50ef-40a5-b48a-58981269c086","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":167,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6816","nonce":3387813890},{"type":"v1","addr":"172.21.15.184:6817","nonce":3387813890}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6818","nonce":3387813890},{"type":"v1","addr":"172.21.15.184:6819","nonce":3387813890}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6822","nonce":3387813890},{"type":"v1","addr":"172.21.15.184:6823","nonce":3387813890}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6820","nonce":3387813890},{"type":"v1","addr":"172.21.15.184:6821","nonce":3387813890}]},"public_addr":"172.21.15.184:6817/3387813890","cluster_addr":"172.21.15.184:6819/3387813890","heartbeat_back_addr":"172.21.15.184:6823/3387813890","heartbeat_front_addr":"172.21.15.184:6821/3387813890","state":["exists","up"]},{"osd":5,"uuid":"32d5a900-da3d-412c-bc19-13861a2368d8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":167,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6808","nonce":2680916017},{"type":"v1","addr":"172.21.15.184:6809","nonce":2680916017}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6810","nonce":2680916017},{"type":"v1","addr":"172.21.15.184:6811","nonce":2680916017}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6814","nonce":2680916017},{"type":"v1","addr":"172.21.15.184:6815","nonce":2680916017}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6812","nonce":2680916017},{"type":"v1","addr":"172.21.15.184:6813","nonce":2680916017}]},"public_addr":"172.21.15.184:6809/2680916017","cluster_addr":"172.21.15.184:6811/2680916017","heartbeat_back_addr":"172.21.15.184:6815/2680916017","heartbeat_front_addr":"172.21.15.184:6813/2680916017","state":["exists","up"]},{"osd":6,"uuid":"cde6f39e-aa1b-4260-ac19-774ee6aaca72","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":167,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6824","nonce":2183120457},{"type":"v1","addr":"172.21.15.184:6825","nonce":2183120457}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6826","nonce":2183120457},{"type":"v1","addr":"172.21.15.184:6827","nonce":2183120457}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6830","nonce":2183120457},{"type":"v1","addr":"172.21.15.184:6831","nonce":2183120457}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6828","nonce":2183120457},{"type":"v1","addr":"172.21.15.184:6829","nonce":2183120457}]},"public_addr":"172.21.15.184:6825/2183120457","cluster_addr":"172.21.15.184:6827/2183120457","heartbeat_back_addr":"172.21.15.184:6831/2183120457","heartbeat_front_addr":"172.21.15.184:6829/2183120457","state":["exists","up"]},{"osd":7,"uuid":"4f381778-eee1-47c6-804f-df0e6b33e313","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":167,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6800","nonce":4102957285},{"type":"v1","addr":"172.21.15.184:6801","nonce":4102957285}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6802","nonce":4102957285},{"type":"v1","addr":"172.21.15.184:6803","nonce":4102957285}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6806","nonce":4102957285},{"type":"v1","addr":"172.21.15.184:6807","nonce":4102957285}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6804","nonce":4102957285},{"type":"v1","addr":"172.21.15.184:6805","nonce":4102957285}]},"public_addr":"172.21.15.184:6801/4102957285","cluster_addr":"172.21.15.184:6803/4102957285","heartbeat_back_addr":"172.21.15.184:6807/4102957285","heartbeat_front_addr":"172.21.15.184:6805/4102957285","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:35.972210+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.222007+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.198617+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.385663+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.358935+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.156066+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.752973+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:35.682846+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":17,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T15:51:52.087 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:52.085+0000 7ff014d12640 1 -- 172.21.15.123:0/3800059334 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7feff00738e0 msgr2=0x7feff0075da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:51:52.087 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:52.085+0000 7ff014d12640 1 --2- 172.21.15.123:0/3800059334 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7feff00738e0 0x7feff0075da0 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7ff0100b6ac0 tx=0x7ff004007a00 comp rx=0 tx=0).stop 2024-09-16T15:51:52.087 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:52.085+0000 7ff014d12640 1 -- 172.21.15.123:0/3800059334 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff0100cf9e0 msgr2=0x7ff0100af9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:51:52.087 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:52.085+0000 7ff014d12640 1 --2- 172.21.15.123:0/3800059334 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff0100cf9e0 0x7ff0100af9a0 secure :-1 s=READY pgs=1881 cs=0 l=1 rev1=1 crypto rx=0x7ff00000c970 tx=0x7ff00000ce30 comp rx=0 tx=0).stop 2024-09-16T15:51:52.087 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:52.085+0000 7ff014d12640 1 -- 172.21.15.123:0/3800059334 shutdown_connections 2024-09-16T15:51:52.087 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:52.089+0000 7ff014d12640 1 --2- 172.21.15.123:0/3800059334 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7feff00738e0 0x7feff0075da0 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:52.087 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:52.089+0000 7ff014d12640 1 --2- 172.21.15.123:0/3800059334 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff01015af00 0x7ff0100b2cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:52.087 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:52.089+0000 7ff014d12640 1 --2- 172.21.15.123:0/3800059334 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7ff01015ab20 0x7ff0100b2770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:52.087 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:52.089+0000 7ff014d12640 1 --2- 172.21.15.123:0/3800059334 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff0100cf9e0 0x7ff0100af9a0 unknown :-1 s=CLOSED pgs=1881 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:52.087 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:52.089+0000 7ff014d12640 1 -- 172.21.15.123:0/3800059334 >> 172.21.15.123:0/3800059334 conn(0x7ff0100d0b70 msgr2=0x7ff010157280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:51:52.087 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:52.089+0000 7ff014d12640 1 -- 172.21.15.123:0/3800059334 shutdown_connections 2024-09-16T15:51:52.088 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:52.089+0000 7ff014d12640 1 -- 172.21.15.123:0/3800059334 wait complete. 2024-09-16T15:51:52.103 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-09-16T15:51:52.103 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T15:51:52.219 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:52.221+0000 7f44a8cc5640 1 -- 172.21.15.123:0/2010468986 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f44a4074470 msgr2=0x7f44a40cd6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:51:52.219 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:52.221+0000 7f44a8cc5640 1 --2- 172.21.15.123:0/2010468986 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f44a4074470 0x7f44a40cd6e0 secure :-1 s=READY pgs=1883 cs=0 l=1 rev1=1 crypto rx=0x7f448c009840 tx=0x7f448c01c790 comp rx=0 tx=0).stop 2024-09-16T15:51:52.219 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:52.221+0000 7f44a8cc5640 1 -- 172.21.15.123:0/2010468986 shutdown_connections 2024-09-16T15:51:52.219 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:52.221+0000 7f44a8cc5640 1 --2- 172.21.15.123:0/2010468986 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f44a4074470 0x7f44a40cd6e0 unknown :-1 s=CLOSED pgs=1883 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:52.219 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:52.221+0000 7f44a8cc5640 1 --2- 172.21.15.123:0/2010468986 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f44a415e160 0x7f44a4073f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:52.219 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:52.221+0000 7f44a8cc5640 1 -- 172.21.15.123:0/2010468986 >> 172.21.15.123:0/2010468986 conn(0x7f44a40cfb10 msgr2=0x7f44a40cff20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:51:52.219 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:52.221+0000 7f44a8cc5640 1 -- 172.21.15.123:0/2010468986 shutdown_connections 2024-09-16T15:51:52.219 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:52.221+0000 7f44a8cc5640 1 -- 172.21.15.123:0/2010468986 wait complete. 2024-09-16T15:51:52.220 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:52.221+0000 7f44a8cc5640 1 Processor -- start 2024-09-16T15:51:52.220 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:52.221+0000 7f44a8cc5640 1 -- start start 2024-09-16T15:51:52.222 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:52.221+0000 7f44a8cc5640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f44a4074470 0x7f44a4150010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:51:52.222 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:52.221+0000 7f44a8cc5640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f44a40cdc20 0x7f44a415a630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:51:52.222 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:52.221+0000 7f44a8cc5640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f44a415e160 0x7f44a415ab70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:51:52.222 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:52.221+0000 7f44a8cc5640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f44a4167f50 con 0x7f44a415e160 2024-09-16T15:51:52.222 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:52.221+0000 7f44a8cc5640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f44a4167dd0 con 0x7f44a40cdc20 2024-09-16T15:51:52.223 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:52.221+0000 7f44a8cc5640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f44a41680d0 con 0x7f44a4074470 2024-09-16T15:51:52.223 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:52.221+0000 7f44a2d9d640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f44a415e160 0x7f44a415ab70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:51:52.223 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:52.221+0000 7f44a2d9d640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f44a415e160 0x7f44a415ab70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:38680/0 (socket says 172.21.15.123:38680) 2024-09-16T15:51:52.223 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:52.221+0000 7f44a2d9d640 1 -- 172.21.15.123:0/1375279653 learned_addr learned my addr 172.21.15.123:0/1375279653 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:51:52.223 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:52.221+0000 7f44a259c640 1 --2- 172.21.15.123:0/1375279653 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f44a4074470 0x7f44a4150010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:51:52.223 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:52.221+0000 7f44a2d9d640 1 -- 172.21.15.123:0/1375279653 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f44a4074470 msgr2=0x7f44a4150010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:51:52.223 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:52.221+0000 7f44a2d9d640 1 --2- 172.21.15.123:0/1375279653 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f44a4074470 0x7f44a4150010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:52.223 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:52.221+0000 7f44a1d9b640 1 --2- 172.21.15.123:0/1375279653 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f44a40cdc20 0x7f44a415a630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:51:52.223 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:52.221+0000 7f44a2d9d640 1 -- 172.21.15.123:0/1375279653 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f44a40cdc20 msgr2=0x7f44a415a630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:51:52.223 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:52.221+0000 7f44a2d9d640 1 --2- 172.21.15.123:0/1375279653 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f44a40cdc20 0x7f44a415a630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:52.223 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:52.221+0000 7f44a2d9d640 1 -- 172.21.15.123:0/1375279653 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f44a415d420 con 0x7f44a415e160 2024-09-16T15:51:52.223 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:52.221+0000 7f44a2d9d640 1 --2- 172.21.15.123:0/1375279653 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f44a415e160 0x7f44a415ab70 secure :-1 s=READY pgs=1884 cs=0 l=1 rev1=1 crypto rx=0x7f449c00cbc0 tx=0x7f449c00ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:51:52.224 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:52.221+0000 7f449affd640 1 -- 172.21.15.123:0/1375279653 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f449c016020 con 0x7f44a415e160 2024-09-16T15:51:52.224 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:52.221+0000 7f449affd640 1 -- 172.21.15.123:0/1375279653 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f449c00ab40 con 0x7f44a415e160 2024-09-16T15:51:52.224 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:52.221+0000 7f44a8cc5640 1 -- 172.21.15.123:0/1375279653 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f44a40b2c60 con 0x7f44a415e160 2024-09-16T15:51:52.224 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:52.221+0000 7f449affd640 1 -- 172.21.15.123:0/1375279653 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f449c012420 con 0x7f44a415e160 2024-09-16T15:51:52.224 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:52.221+0000 7f44a8cc5640 1 -- 172.21.15.123:0/1375279653 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f44a40b3220 con 0x7f44a415e160 2024-09-16T15:51:52.224 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:52.221+0000 7f44a8cc5640 1 -- 172.21.15.123:0/1375279653 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f44a40cd6e0 con 0x7f44a415e160 2024-09-16T15:51:52.225 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:52.225+0000 7f449affd640 1 -- 172.21.15.123:0/1375279653 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 25) ==== 97998+0+0 (secure 0 0 0) 0x7f449c019020 con 0x7f44a415e160 2024-09-16T15:51:52.225 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:52.225+0000 7f449affd640 1 --2- 172.21.15.123:0/1375279653 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f4484073910 0x7f4484075dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:51:52.225 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:52.225+0000 7f44a259c640 1 --2- 172.21.15.123:0/1375279653 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f4484073910 0x7f4484075dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:51:52.225 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:52.225+0000 7f449affd640 1 -- 172.21.15.123:0/1375279653 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(169..169 src has 1..169) ==== 5737+0+0 (secure 0 0 0) 0x7f449c0928b0 con 0x7f44a415e160 2024-09-16T15:51:52.225 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:52.225+0000 7f44a259c640 1 --2- 172.21.15.123:0/1375279653 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f4484073910 0x7f4484075dd0 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f449400ba40 tx=0x7f4494005cd0 comp rx=0 tx=0).ready entity=mgr.4150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:51:52.228 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:52.229+0000 7f449affd640 1 -- 172.21.15.123:0/1375279653 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f449c020030 con 0x7f44a415e160 2024-09-16T15:51:52.383 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:52.385+0000 7f44a8cc5640 1 -- 172.21.15.123:0/1375279653 --> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f44a40d06c0 con 0x7f4484073910 2024-09-16T15:51:52.394 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:52.393+0000 7f449affd640 1 -- 172.21.15.123:0/1375279653 <== mgr.4150 v2:172.21.15.123:6836/2464759676 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+252478 (secure 0 0 0) 0x7f44a40d06c0 con 0x7f4484073910 2024-09-16T15:51:52.396 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-09-16T15:51:52.397 INFO:teuthology.orchestra.run.smithi123.stderr:dumped all 2024-09-16T15:51:52.398 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:52.397+0000 7f44a8cc5640 1 -- 172.21.15.123:0/1375279653 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f4484073910 msgr2=0x7f4484075dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:51:52.399 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:52.397+0000 7f44a8cc5640 1 --2- 172.21.15.123:0/1375279653 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f4484073910 0x7f4484075dd0 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f449400ba40 tx=0x7f4494005cd0 comp rx=0 tx=0).stop 2024-09-16T15:51:52.399 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:52.397+0000 7f44a8cc5640 1 -- 172.21.15.123:0/1375279653 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f44a415e160 msgr2=0x7f44a415ab70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:51:52.399 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:52.397+0000 7f44a8cc5640 1 --2- 172.21.15.123:0/1375279653 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f44a415e160 0x7f44a415ab70 secure :-1 s=READY pgs=1884 cs=0 l=1 rev1=1 crypto rx=0x7f449c00cbc0 tx=0x7f449c00ccb0 comp rx=0 tx=0).stop 2024-09-16T15:51:52.399 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:52.397+0000 7f44a8cc5640 1 -- 172.21.15.123:0/1375279653 shutdown_connections 2024-09-16T15:51:52.399 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:52.397+0000 7f44a8cc5640 1 --2- 172.21.15.123:0/1375279653 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f4484073910 0x7f4484075dd0 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:52.399 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:52.397+0000 7f44a8cc5640 1 --2- 172.21.15.123:0/1375279653 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f44a415e160 0x7f44a415ab70 unknown :-1 s=CLOSED pgs=1884 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:52.399 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:52.397+0000 7f44a8cc5640 1 --2- 172.21.15.123:0/1375279653 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f44a40cdc20 0x7f44a415a630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:52.399 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:52.397+0000 7f44a8cc5640 1 --2- 172.21.15.123:0/1375279653 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f44a4074470 0x7f44a4150010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:52.399 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:52.397+0000 7f44a8cc5640 1 -- 172.21.15.123:0/1375279653 >> 172.21.15.123:0/1375279653 conn(0x7f44a40cfb10 msgr2=0x7f44a40ceca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:51:52.399 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:52.397+0000 7f44a8cc5640 1 -- 172.21.15.123:0/1375279653 shutdown_connections 2024-09-16T15:51:52.399 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:52.397+0000 7f44a8cc5640 1 -- 172.21.15.123:0/1375279653 wait complete. 2024-09-16T15:51:52.412 INFO:teuthology.orchestra.run.smithi123.stdout:{"pg_ready":true,"pg_map":{"version":229,"stamp":"2024-09-16T15:51:52.143817+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":158,"num_write_kb":1642,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":118,"ondisk_log_size":118,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":80,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":275640,"kb_used_data":17792,"kb_used_omap":12,"kb_used_meta":257651,"kb_avail":754699080,"statfs":{"total":773094113280,"available":772811857920,"internally_reserved":0,"allocated":18219008,"data_stored":8708556,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":263835185},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":35,"apply_latency_ms":35,"commit_latency_ns":35000000,"apply_latency_ns":35000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.796295"},"pg_stats":[{"pgid":"27.3f","version":"0'0","reported_seq":13,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-09-16T15:51:49.690715+0000","last_change":"2024-09-16T15:51:48.420332+0000","last_active":"2024-09-16T15:51:49.690715+0000","last_peered":"2024-09-16T15:51:49.690715+0000","last_clean":"2024-09-16T15:51:49.690715+0000","last_became_active":"2024-09-16T15:51:48.415997+0000","last_became_peered":"2024-09-16T15:51:48.415997+0000","last_unstale":"2024-09-16T15:51:49.690715+0000","last_undegraded":"2024-09-16T15:51:49.690715+0000","last_fullsized":"2024-09-16T15:51:49.690715+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":167,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:51:47.345679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:51:47.345679+0000","last_clean_scrub_stamp":"2024-09-16T15:51:47.345679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:16:57.181968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.3e","version":"0'0","reported_seq":10,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-09-16T15:51:48.441452+0000","last_change":"2024-09-16T15:51:48.428809+0000","last_active":"2024-09-16T15:51:48.441452+0000","last_peered":"2024-09-16T15:51:48.441452+0000","last_clean":"2024-09-16T15:51:48.441452+0000","last_became_active":"2024-09-16T15:51:48.428073+0000","last_became_peered":"2024-09-16T15:51:48.428073+0000","last_unstale":"2024-09-16T15:51:48.441452+0000","last_undegraded":"2024-09-16T15:51:48.441452+0000","last_fullsized":"2024-09-16T15:51:48.441452+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":167,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:51:47.345679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:51:47.345679+0000","last_clean_scrub_stamp":"2024-09-16T15:51:47.345679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:38:13.614017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.3d","version":"0'0","reported_seq":13,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-09-16T15:51:49.391883+0000","last_change":"2024-09-16T15:51:48.417037+0000","last_active":"2024-09-16T15:51:49.391883+0000","last_peered":"2024-09-16T15:51:49.391883+0000","last_clean":"2024-09-16T15:51:49.391883+0000","last_became_active":"2024-09-16T15:51:48.414206+0000","last_became_peered":"2024-09-16T15:51:48.414206+0000","last_unstale":"2024-09-16T15:51:49.391883+0000","last_undegraded":"2024-09-16T15:51:49.391883+0000","last_fullsized":"2024-09-16T15:51:49.391883+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":167,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:51:47.345679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:51:47.345679+0000","last_clean_scrub_stamp":"2024-09-16T15:51:47.345679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T23:40:00.612806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.3c","version":"0'0","reported_seq":13,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-09-16T15:51:49.435887+0000","last_change":"2024-09-16T15:51:48.401750+0000","last_active":"2024-09-16T15:51:49.435887+0000","last_peered":"2024-09-16T15:51:49.435887+0000","last_clean":"2024-09-16T15:51:49.435887+0000","last_became_active":"2024-09-16T15:51:48.400974+0000","last_became_peered":"2024-09-16T15:51:48.400974+0000","last_unstale":"2024-09-16T15:51:49.435887+0000","last_undegraded":"2024-09-16T15:51:49.435887+0000","last_fullsized":"2024-09-16T15:51:49.435887+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":167,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:51:47.345679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:51:47.345679+0000","last_clean_scrub_stamp":"2024-09-16T15:51:47.345679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:52:15.623713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.3b","version":"0'0","reported_seq":10,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-09-16T15:51:48.408008+0000","last_change":"2024-09-16T15:51:48.403256+0000","last_active":"2024-09-16T15:51:48.408008+0000","last_peered":"2024-09-16T15:51:48.408008+0000","last_clean":"2024-09-16T15:51:48.408008+0000","last_became_active":"2024-09-16T15:51:48.402536+0000","last_became_peered":"2024-09-16T15:51:48.402536+0000","last_unstale":"2024-09-16T15:51:48.408008+0000","last_undegraded":"2024-09-16T15:51:48.408008+0000","last_fullsized":"2024-09-16T15:51:48.408008+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":167,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:51:47.345679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:51:47.345679+0000","last_clean_scrub_stamp":"2024-09-16T15:51:47.345679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:27:20.068057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.3a","version":"0'0","reported_seq":10,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-09-16T15:51:48.430369+0000","last_change":"2024-09-16T15:51:48.425364+0000","last_active":"2024-09-16T15:51:48.430369+0000","last_peered":"2024-09-16T15:51:48.430369+0000","last_clean":"2024-09-16T15:51:48.430369+0000","last_became_active":"2024-09-16T15:51:48.424556+0000","last_became_peered":"2024-09-16T15:51:48.424556+0000","last_unstale":"2024-09-16T15:51:48.430369+0000","last_undegraded":"2024-09-16T15:51:48.430369+0000","last_fullsized":"2024-09-16T15:51:48.430369+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":167,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:51:47.345679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:51:47.345679+0000","last_clean_scrub_stamp":"2024-09-16T15:51:47.345679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T00:38:29.757986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.39","version":"0'0","reported_seq":13,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-09-16T15:51:49.689842+0000","last_change":"2024-09-16T15:51:48.421743+0000","last_active":"2024-09-16T15:51:49.689842+0000","last_peered":"2024-09-16T15:51:49.689842+0000","last_clean":"2024-09-16T15:51:49.689842+0000","last_became_active":"2024-09-16T15:51:48.421020+0000","last_became_peered":"2024-09-16T15:51:48.421020+0000","last_unstale":"2024-09-16T15:51:49.689842+0000","last_undegraded":"2024-09-16T15:51:49.689842+0000","last_fullsized":"2024-09-16T15:51:49.689842+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":167,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:51:47.345679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:51:47.345679+0000","last_clean_scrub_stamp":"2024-09-16T15:51:47.345679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:12:44.001487+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.38","version":"0'0","reported_seq":10,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-09-16T15:51:48.396098+0000","last_change":"2024-09-16T15:51:48.394064+0000","last_active":"2024-09-16T15:51:48.396098+0000","last_peered":"2024-09-16T15:51:48.396098+0000","last_clean":"2024-09-16T15:51:48.396098+0000","last_became_active":"2024-09-16T15:51:48.393323+0000","last_became_peered":"2024-09-16T15:51:48.393323+0000","last_unstale":"2024-09-16T15:51:48.396098+0000","last_undegraded":"2024-09-16T15:51:48.396098+0000","last_fullsized":"2024-09-16T15:51:48.396098+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":167,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:51:47.345679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:51:47.345679+0000","last_clean_scrub_stamp":"2024-09-16T15:51:47.345679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:17:20.368465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.37","version":"0'0","reported_seq":10,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-09-16T15:51:48.441496+0000","last_change":"2024-09-16T15:51:48.433407+0000","last_active":"2024-09-16T15:51:48.441496+0000","last_peered":"2024-09-16T15:51:48.441496+0000","last_clean":"2024-09-16T15:51:48.441496+0000","last_became_active":"2024-09-16T15:51:48.432739+0000","last_became_peered":"2024-09-16T15:51:48.432739+0000","last_unstale":"2024-09-16T15:51:48.441496+0000","last_undegraded":"2024-09-16T15:51:48.441496+0000","last_fullsized":"2024-09-16T15:51:48.441496+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":167,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:51:47.345679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:51:47.345679+0000","last_clean_scrub_stamp":"2024-09-16T15:51:47.345679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T01:51:19.144810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.36","version":"0'0","reported_seq":10,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-09-16T15:51:48.435726+0000","last_change":"2024-09-16T15:51:48.433866+0000","last_active":"2024-09-16T15:51:48.435726+0000","last_peered":"2024-09-16T15:51:48.435726+0000","last_clean":"2024-09-16T15:51:48.435726+0000","last_became_active":"2024-09-16T15:51:48.433462+0000","last_became_peered":"2024-09-16T15:51:48.433462+0000","last_unstale":"2024-09-16T15:51:48.435726+0000","last_undegraded":"2024-09-16T15:51:48.435726+0000","last_fullsized":"2024-09-16T15:51:48.435726+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":167,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:51:47.345679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:51:47.345679+0000","last_clean_scrub_stamp":"2024-09-16T15:51:47.345679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:19:24.536498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.35","version":"0'0","reported_seq":13,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-09-16T15:51:49.391937+0000","last_change":"2024-09-16T15:51:48.417966+0000","last_active":"2024-09-16T15:51:49.391937+0000","last_peered":"2024-09-16T15:51:49.391937+0000","last_clean":"2024-09-16T15:51:49.391937+0000","last_became_active":"2024-09-16T15:51:48.416270+0000","last_became_peered":"2024-09-16T15:51:48.416270+0000","last_unstale":"2024-09-16T15:51:49.391937+0000","last_undegraded":"2024-09-16T15:51:49.391937+0000","last_fullsized":"2024-09-16T15:51:49.391937+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":167,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:51:47.345679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:51:47.345679+0000","last_clean_scrub_stamp":"2024-09-16T15:51:47.345679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T02:21:37.777271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.34","version":"0'0","reported_seq":13,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-09-16T15:51:49.390622+0000","last_change":"2024-09-16T15:51:48.398213+0000","last_active":"2024-09-16T15:51:49.390622+0000","last_peered":"2024-09-16T15:51:49.390622+0000","last_clean":"2024-09-16T15:51:49.390622+0000","last_became_active":"2024-09-16T15:51:48.396833+0000","last_became_peered":"2024-09-16T15:51:48.396833+0000","last_unstale":"2024-09-16T15:51:49.390622+0000","last_undegraded":"2024-09-16T15:51:49.390622+0000","last_fullsized":"2024-09-16T15:51:49.390622+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":167,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:51:47.345679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:51:47.345679+0000","last_clean_scrub_stamp":"2024-09-16T15:51:47.345679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T18:19:46.535824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.33","version":"0'0","reported_seq":13,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-09-16T15:51:49.391060+0000","last_change":"2024-09-16T15:51:48.396718+0000","last_active":"2024-09-16T15:51:49.391060+0000","last_peered":"2024-09-16T15:51:49.391060+0000","last_clean":"2024-09-16T15:51:49.391060+0000","last_became_active":"2024-09-16T15:51:48.395829+0000","last_became_peered":"2024-09-16T15:51:48.395829+0000","last_unstale":"2024-09-16T15:51:49.391060+0000","last_undegraded":"2024-09-16T15:51:49.391060+0000","last_fullsized":"2024-09-16T15:51:49.391060+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":167,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:51:47.345679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:51:47.345679+0000","last_clean_scrub_stamp":"2024-09-16T15:51:47.345679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T18:06:32.715829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.32","version":"0'0","reported_seq":13,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-09-16T15:51:49.690024+0000","last_change":"2024-09-16T15:51:48.421882+0000","last_active":"2024-09-16T15:51:49.690024+0000","last_peered":"2024-09-16T15:51:49.690024+0000","last_clean":"2024-09-16T15:51:49.690024+0000","last_became_active":"2024-09-16T15:51:48.415002+0000","last_became_peered":"2024-09-16T15:51:48.415002+0000","last_unstale":"2024-09-16T15:51:49.690024+0000","last_undegraded":"2024-09-16T15:51:49.690024+0000","last_fullsized":"2024-09-16T15:51:49.690024+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":167,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:51:47.345679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:51:47.345679+0000","last_clean_scrub_stamp":"2024-09-16T15:51:47.345679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:25:21.266449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.31","version":"0'0","reported_seq":13,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-09-16T15:51:49.689861+0000","last_change":"2024-09-16T15:51:48.436670+0000","last_active":"2024-09-16T15:51:49.689861+0000","last_peered":"2024-09-16T15:51:49.689861+0000","last_clean":"2024-09-16T15:51:49.689861+0000","last_became_active":"2024-09-16T15:51:48.433234+0000","last_became_peered":"2024-09-16T15:51:48.433234+0000","last_unstale":"2024-09-16T15:51:49.689861+0000","last_undegraded":"2024-09-16T15:51:49.689861+0000","last_fullsized":"2024-09-16T15:51:49.689861+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":167,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:51:47.345679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:51:47.345679+0000","last_clean_scrub_stamp":"2024-09-16T15:51:47.345679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T01:03:40.372183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.30","version":"0'0","reported_seq":10,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-09-16T15:51:48.411029+0000","last_change":"2024-09-16T15:51:48.407337+0000","last_active":"2024-09-16T15:51:48.411029+0000","last_peered":"2024-09-16T15:51:48.411029+0000","last_clean":"2024-09-16T15:51:48.411029+0000","last_became_active":"2024-09-16T15:51:48.406491+0000","last_became_peered":"2024-09-16T15:51:48.406491+0000","last_unstale":"2024-09-16T15:51:48.411029+0000","last_undegraded":"2024-09-16T15:51:48.411029+0000","last_fullsized":"2024-09-16T15:51:48.411029+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":167,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:51:47.345679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:51:47.345679+0000","last_clean_scrub_stamp":"2024-09-16T15:51:47.345679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:05:38.011381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.2f","version":"0'0","reported_seq":13,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-09-16T15:51:49.690575+0000","last_change":"2024-09-16T15:51:48.434587+0000","last_active":"2024-09-16T15:51:49.690575+0000","last_peered":"2024-09-16T15:51:49.690575+0000","last_clean":"2024-09-16T15:51:49.690575+0000","last_became_active":"2024-09-16T15:51:48.432752+0000","last_became_peered":"2024-09-16T15:51:48.432752+0000","last_unstale":"2024-09-16T15:51:49.690575+0000","last_undegraded":"2024-09-16T15:51:49.690575+0000","last_fullsized":"2024-09-16T15:51:49.690575+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":167,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:51:47.345679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:51:47.345679+0000","last_clean_scrub_stamp":"2024-09-16T15:51:47.345679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:52:37.537156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.2e","version":"0'0","reported_seq":10,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-09-16T15:51:48.428648+0000","last_change":"2024-09-16T15:51:48.425963+0000","last_active":"2024-09-16T15:51:48.428648+0000","last_peered":"2024-09-16T15:51:48.428648+0000","last_clean":"2024-09-16T15:51:48.428648+0000","last_became_active":"2024-09-16T15:51:48.425609+0000","last_became_peered":"2024-09-16T15:51:48.425609+0000","last_unstale":"2024-09-16T15:51:48.428648+0000","last_undegraded":"2024-09-16T15:51:48.428648+0000","last_fullsized":"2024-09-16T15:51:48.428648+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":167,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:51:47.345679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:51:47.345679+0000","last_clean_scrub_stamp":"2024-09-16T15:51:47.345679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T00:04:45.914265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.2d","version":"0'0","reported_seq":10,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-09-16T15:51:48.434468+0000","last_change":"2024-09-16T15:51:48.433761+0000","last_active":"2024-09-16T15:51:48.434468+0000","last_peered":"2024-09-16T15:51:48.434468+0000","last_clean":"2024-09-16T15:51:48.434468+0000","last_became_active":"2024-09-16T15:51:48.433458+0000","last_became_peered":"2024-09-16T15:51:48.433458+0000","last_unstale":"2024-09-16T15:51:48.434468+0000","last_undegraded":"2024-09-16T15:51:48.434468+0000","last_fullsized":"2024-09-16T15:51:48.434468+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":167,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:51:47.345679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:51:47.345679+0000","last_clean_scrub_stamp":"2024-09-16T15:51:47.345679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T01:05:01.623302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.2c","version":"0'0","reported_seq":10,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-09-16T15:51:48.423630+0000","last_change":"2024-09-16T15:51:48.408745+0000","last_active":"2024-09-16T15:51:48.423630+0000","last_peered":"2024-09-16T15:51:48.423630+0000","last_clean":"2024-09-16T15:51:48.423630+0000","last_became_active":"2024-09-16T15:51:48.407685+0000","last_became_peered":"2024-09-16T15:51:48.407685+0000","last_unstale":"2024-09-16T15:51:48.423630+0000","last_undegraded":"2024-09-16T15:51:48.423630+0000","last_fullsized":"2024-09-16T15:51:48.423630+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":167,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:51:47.345679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:51:47.345679+0000","last_clean_scrub_stamp":"2024-09-16T15:51:47.345679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T00:29:29.367918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.2b","version":"0'0","reported_seq":13,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-09-16T15:51:49.434646+0000","last_change":"2024-09-16T15:51:48.406919+0000","last_active":"2024-09-16T15:51:49.434646+0000","last_peered":"2024-09-16T15:51:49.434646+0000","last_clean":"2024-09-16T15:51:49.434646+0000","last_became_active":"2024-09-16T15:51:48.406245+0000","last_became_peered":"2024-09-16T15:51:48.406245+0000","last_unstale":"2024-09-16T15:51:49.434646+0000","last_undegraded":"2024-09-16T15:51:49.434646+0000","last_fullsized":"2024-09-16T15:51:49.434646+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":167,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:51:47.345679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:51:47.345679+0000","last_clean_scrub_stamp":"2024-09-16T15:51:47.345679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T00:43:21.089679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.2a","version":"0'0","reported_seq":13,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-09-16T15:51:49.392755+0000","last_change":"2024-09-16T15:51:48.411279+0000","last_active":"2024-09-16T15:51:49.392755+0000","last_peered":"2024-09-16T15:51:49.392755+0000","last_clean":"2024-09-16T15:51:49.392755+0000","last_became_active":"2024-09-16T15:51:48.410456+0000","last_became_peered":"2024-09-16T15:51:48.410456+0000","last_unstale":"2024-09-16T15:51:49.392755+0000","last_undegraded":"2024-09-16T15:51:49.392755+0000","last_fullsized":"2024-09-16T15:51:49.392755+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":167,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:51:47.345679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:51:47.345679+0000","last_clean_scrub_stamp":"2024-09-16T15:51:47.345679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:23:57.550863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.29","version":"0'0","reported_seq":13,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-09-16T15:51:49.391086+0000","last_change":"2024-09-16T15:51:48.395273+0000","last_active":"2024-09-16T15:51:49.391086+0000","last_peered":"2024-09-16T15:51:49.391086+0000","last_clean":"2024-09-16T15:51:49.391086+0000","last_became_active":"2024-09-16T15:51:48.394570+0000","last_became_peered":"2024-09-16T15:51:48.394570+0000","last_unstale":"2024-09-16T15:51:49.391086+0000","last_undegraded":"2024-09-16T15:51:49.391086+0000","last_fullsized":"2024-09-16T15:51:49.391086+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":167,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:51:47.345679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:51:47.345679+0000","last_clean_scrub_stamp":"2024-09-16T15:51:47.345679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:39:26.940952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.28","version":"0'0","reported_seq":13,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-09-16T15:51:49.435076+0000","last_change":"2024-09-16T15:51:48.425121+0000","last_active":"2024-09-16T15:51:49.435076+0000","last_peered":"2024-09-16T15:51:49.435076+0000","last_clean":"2024-09-16T15:51:49.435076+0000","last_became_active":"2024-09-16T15:51:48.424367+0000","last_became_peered":"2024-09-16T15:51:48.424367+0000","last_unstale":"2024-09-16T15:51:49.435076+0000","last_undegraded":"2024-09-16T15:51:49.435076+0000","last_fullsized":"2024-09-16T15:51:49.435076+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":167,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:51:47.345679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:51:47.345679+0000","last_clean_scrub_stamp":"2024-09-16T15:51:47.345679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T02:33:57.816891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.27","version":"0'0","reported_seq":10,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-09-16T15:51:48.441257+0000","last_change":"2024-09-16T15:51:48.430257+0000","last_active":"2024-09-16T15:51:48.441257+0000","last_peered":"2024-09-16T15:51:48.441257+0000","last_clean":"2024-09-16T15:51:48.441257+0000","last_became_active":"2024-09-16T15:51:48.429592+0000","last_became_peered":"2024-09-16T15:51:48.429592+0000","last_unstale":"2024-09-16T15:51:48.441257+0000","last_undegraded":"2024-09-16T15:51:48.441257+0000","last_fullsized":"2024-09-16T15:51:48.441257+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":167,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:51:47.345679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:51:47.345679+0000","last_clean_scrub_stamp":"2024-09-16T15:51:47.345679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:27:35.569594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.26","version":"0'0","reported_seq":13,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-09-16T15:51:49.691129+0000","last_change":"2024-09-16T15:51:48.431641+0000","last_active":"2024-09-16T15:51:49.691129+0000","last_peered":"2024-09-16T15:51:49.691129+0000","last_clean":"2024-09-16T15:51:49.691129+0000","last_became_active":"2024-09-16T15:51:48.423156+0000","last_became_peered":"2024-09-16T15:51:48.423156+0000","last_unstale":"2024-09-16T15:51:49.691129+0000","last_undegraded":"2024-09-16T15:51:49.691129+0000","last_fullsized":"2024-09-16T15:51:49.691129+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":167,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:51:47.345679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:51:47.345679+0000","last_clean_scrub_stamp":"2024-09-16T15:51:47.345679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:52:06.593219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.25","version":"0'0","reported_seq":13,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-09-16T15:51:49.392092+0000","last_change":"2024-09-16T15:51:48.397722+0000","last_active":"2024-09-16T15:51:49.392092+0000","last_peered":"2024-09-16T15:51:49.392092+0000","last_clean":"2024-09-16T15:51:49.392092+0000","last_became_active":"2024-09-16T15:51:48.397005+0000","last_became_peered":"2024-09-16T15:51:48.397005+0000","last_unstale":"2024-09-16T15:51:49.392092+0000","last_undegraded":"2024-09-16T15:51:49.392092+0000","last_fullsized":"2024-09-16T15:51:49.392092+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":167,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:51:47.345679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:51:47.345679+0000","last_clean_scrub_stamp":"2024-09-16T15:51:47.345679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T23:19:13.505005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.24","version":"0'0","reported_seq":13,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-09-16T15:51:49.391454+0000","last_change":"2024-09-16T15:51:48.407290+0000","last_active":"2024-09-16T15:51:49.391454+0000","last_peered":"2024-09-16T15:51:49.391454+0000","last_clean":"2024-09-16T15:51:49.391454+0000","last_became_active":"2024-09-16T15:51:48.406572+0000","last_became_peered":"2024-09-16T15:51:48.406572+0000","last_unstale":"2024-09-16T15:51:49.391454+0000","last_undegraded":"2024-09-16T15:51:49.391454+0000","last_fullsized":"2024-09-16T15:51:49.391454+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":167,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:51:47.345679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:51:47.345679+0000","last_clean_scrub_stamp":"2024-09-16T15:51:47.345679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T03:28:07.952532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.23","version":"0'0","reported_seq":13,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-09-16T15:51:49.389652+0000","last_change":"2024-09-16T15:51:48.398061+0000","last_active":"2024-09-16T15:51:49.389652+0000","last_peered":"2024-09-16T15:51:49.389652+0000","last_clean":"2024-09-16T15:51:49.389652+0000","last_became_active":"2024-09-16T15:51:48.397267+0000","last_became_peered":"2024-09-16T15:51:48.397267+0000","last_unstale":"2024-09-16T15:51:49.389652+0000","last_undegraded":"2024-09-16T15:51:49.389652+0000","last_fullsized":"2024-09-16T15:51:49.389652+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":167,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:51:47.345679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:51:47.345679+0000","last_clean_scrub_stamp":"2024-09-16T15:51:47.345679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T03:05:58.168737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.22","version":"0'0","reported_seq":10,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-09-16T15:51:48.443275+0000","last_change":"2024-09-16T15:51:48.430281+0000","last_active":"2024-09-16T15:51:48.443275+0000","last_peered":"2024-09-16T15:51:48.443275+0000","last_clean":"2024-09-16T15:51:48.443275+0000","last_became_active":"2024-09-16T15:51:48.429690+0000","last_became_peered":"2024-09-16T15:51:48.429690+0000","last_unstale":"2024-09-16T15:51:48.443275+0000","last_undegraded":"2024-09-16T15:51:48.443275+0000","last_fullsized":"2024-09-16T15:51:48.443275+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":167,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:51:47.345679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:51:47.345679+0000","last_clean_scrub_stamp":"2024-09-16T15:51:47.345679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T23:51:48.437995+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.21","version":"0'0","reported_seq":10,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-09-16T15:51:48.420576+0000","last_change":"2024-09-16T15:51:48.413862+0000","last_active":"2024-09-16T15:51:48.420576+0000","last_peered":"2024-09-16T15:51:48.420576+0000","last_clean":"2024-09-16T15:51:48.420576+0000","last_became_active":"2024-09-16T15:51:48.411907+0000","last_became_peered":"2024-09-16T15:51:48.411907+0000","last_unstale":"2024-09-16T15:51:48.420576+0000","last_undegraded":"2024-09-16T15:51:48.420576+0000","last_fullsized":"2024-09-16T15:51:48.420576+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":167,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:51:47.345679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:51:47.345679+0000","last_clean_scrub_stamp":"2024-09-16T15:51:47.345679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:26:56.667722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.20","version":"0'0","reported_seq":13,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-09-16T15:51:49.689778+0000","last_change":"2024-09-16T15:51:48.415866+0000","last_active":"2024-09-16T15:51:49.689778+0000","last_peered":"2024-09-16T15:51:49.689778+0000","last_clean":"2024-09-16T15:51:49.689778+0000","last_became_active":"2024-09-16T15:51:48.415128+0000","last_became_peered":"2024-09-16T15:51:48.415128+0000","last_unstale":"2024-09-16T15:51:49.689778+0000","last_undegraded":"2024-09-16T15:51:49.689778+0000","last_fullsized":"2024-09-16T15:51:49.689778+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":167,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:51:47.345679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:51:47.345679+0000","last_clean_scrub_stamp":"2024-09-16T15:51:47.345679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:52:15.623713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.1e","version":"0'0","reported_seq":10,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-09-16T15:51:48.441518+0000","last_change":"2024-09-16T15:51:48.428746+0000","last_active":"2024-09-16T15:51:48.441518+0000","last_peered":"2024-09-16T15:51:48.441518+0000","last_clean":"2024-09-16T15:51:48.441518+0000","last_became_active":"2024-09-16T15:51:48.427152+0000","last_became_peered":"2024-09-16T15:51:48.427152+0000","last_unstale":"2024-09-16T15:51:48.441518+0000","last_undegraded":"2024-09-16T15:51:48.441518+0000","last_fullsized":"2024-09-16T15:51:48.441518+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":167,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:51:47.345679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:51:47.345679+0000","last_clean_scrub_stamp":"2024-09-16T15:51:47.345679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:06:11.869682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.1f","version":"168'4","reported_seq":19,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-09-16T15:51:49.392490+0000","last_change":"2024-09-16T15:51:47.389991+0000","last_active":"2024-09-16T15:51:49.392490+0000","last_peered":"2024-09-16T15:51:49.392490+0000","last_clean":"2024-09-16T15:51:49.392490+0000","last_became_active":"2024-09-16T15:51:47.389258+0000","last_became_peered":"2024-09-16T15:51:47.389258+0000","last_unstale":"2024-09-16T15:51:49.392490+0000","last_undegraded":"2024-09-16T15:51:49.392490+0000","last_fullsized":"2024-09-16T15:51:49.392490+0000","mapping_epoch":166,"log_start":"0'0","ondisk_log_start":"0'0","created":166,"last_epoch_clean":167,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:51:46.335209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:51:46.335209+0000","last_clean_scrub_stamp":"2024-09-16T15:51:46.335209+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:28:42.487689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.1f","version":"0'0","reported_seq":13,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-09-16T15:51:49.393513+0000","last_change":"2024-09-16T15:51:48.388882+0000","last_active":"2024-09-16T15:51:49.393513+0000","last_peered":"2024-09-16T15:51:49.393513+0000","last_clean":"2024-09-16T15:51:49.393513+0000","last_became_active":"2024-09-16T15:51:48.388014+0000","last_became_peered":"2024-09-16T15:51:48.388014+0000","last_unstale":"2024-09-16T15:51:49.393513+0000","last_undegraded":"2024-09-16T15:51:49.393513+0000","last_fullsized":"2024-09-16T15:51:49.393513+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":167,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:51:47.345679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:51:47.345679+0000","last_clean_scrub_stamp":"2024-09-16T15:51:47.345679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:01:50.109145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.1e","version":"168'1","reported_seq":14,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-09-16T15:51:48.449559+0000","last_change":"2024-09-16T15:51:47.390954+0000","last_active":"2024-09-16T15:51:48.449559+0000","last_peered":"2024-09-16T15:51:48.449559+0000","last_clean":"2024-09-16T15:51:48.449559+0000","last_became_active":"2024-09-16T15:51:47.388775+0000","last_became_peered":"2024-09-16T15:51:47.388775+0000","last_unstale":"2024-09-16T15:51:48.449559+0000","last_undegraded":"2024-09-16T15:51:48.449559+0000","last_fullsized":"2024-09-16T15:51:48.449559+0000","mapping_epoch":166,"log_start":"0'0","ondisk_log_start":"0'0","created":166,"last_epoch_clean":167,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:51:46.335209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:51:46.335209+0000","last_clean_scrub_stamp":"2024-09-16T15:51:46.335209+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T00:19:10.515395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.1c","version":"0'0","reported_seq":13,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-09-16T15:51:49.690628+0000","last_change":"2024-09-16T15:51:48.436763+0000","last_active":"2024-09-16T15:51:49.690628+0000","last_peered":"2024-09-16T15:51:49.690628+0000","last_clean":"2024-09-16T15:51:49.690628+0000","last_became_active":"2024-09-16T15:51:48.434929+0000","last_became_peered":"2024-09-16T15:51:48.434929+0000","last_unstale":"2024-09-16T15:51:49.690628+0000","last_undegraded":"2024-09-16T15:51:49.690628+0000","last_fullsized":"2024-09-16T15:51:49.690628+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":167,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:51:47.345679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:51:47.345679+0000","last_clean_scrub_stamp":"2024-09-16T15:51:47.345679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T03:22:51.808663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.1d","version":"168'2","reported_seq":17,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-09-16T15:51:49.392185+0000","last_change":"2024-09-16T15:51:47.386433+0000","last_active":"2024-09-16T15:51:49.392185+0000","last_peered":"2024-09-16T15:51:49.392185+0000","last_clean":"2024-09-16T15:51:49.392185+0000","last_became_active":"2024-09-16T15:51:47.386107+0000","last_became_peered":"2024-09-16T15:51:47.386107+0000","last_unstale":"2024-09-16T15:51:49.392185+0000","last_undegraded":"2024-09-16T15:51:49.392185+0000","last_fullsized":"2024-09-16T15:51:49.392185+0000","mapping_epoch":166,"log_start":"0'0","ondisk_log_start":"0'0","created":166,"last_epoch_clean":167,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:51:46.335209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:51:46.335209+0000","last_clean_scrub_stamp":"2024-09-16T15:51:46.335209+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T18:02:05.541664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.1d","version":"0'0","reported_seq":10,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-09-16T15:51:48.429486+0000","last_change":"2024-09-16T15:51:48.427496+0000","last_active":"2024-09-16T15:51:48.429486+0000","last_peered":"2024-09-16T15:51:48.429486+0000","last_clean":"2024-09-16T15:51:48.429486+0000","last_became_active":"2024-09-16T15:51:48.427163+0000","last_became_peered":"2024-09-16T15:51:48.427163+0000","last_unstale":"2024-09-16T15:51:48.429486+0000","last_undegraded":"2024-09-16T15:51:48.429486+0000","last_fullsized":"2024-09-16T15:51:48.429486+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":167,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:51:47.345679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:51:47.345679+0000","last_clean_scrub_stamp":"2024-09-16T15:51:47.345679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T00:42:00.673157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.1c","version":"168'2","reported_seq":17,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-09-16T15:51:49.436172+0000","last_change":"2024-09-16T15:51:47.385737+0000","last_active":"2024-09-16T15:51:49.436172+0000","last_peered":"2024-09-16T15:51:49.436172+0000","last_clean":"2024-09-16T15:51:49.436172+0000","last_became_active":"2024-09-16T15:51:47.384892+0000","last_became_peered":"2024-09-16T15:51:47.384892+0000","last_unstale":"2024-09-16T15:51:49.436172+0000","last_undegraded":"2024-09-16T15:51:49.436172+0000","last_fullsized":"2024-09-16T15:51:49.436172+0000","mapping_epoch":166,"log_start":"0'0","ondisk_log_start":"0'0","created":166,"last_epoch_clean":167,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:51:46.335209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:51:46.335209+0000","last_clean_scrub_stamp":"2024-09-16T15:51:46.335209+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T02:00:54.678527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.1b","version":"0'0","reported_seq":13,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-09-16T15:51:49.434938+0000","last_change":"2024-09-16T15:51:48.401600+0000","last_active":"2024-09-16T15:51:49.434938+0000","last_peered":"2024-09-16T15:51:49.434938+0000","last_clean":"2024-09-16T15:51:49.434938+0000","last_became_active":"2024-09-16T15:51:48.400855+0000","last_became_peered":"2024-09-16T15:51:48.400855+0000","last_unstale":"2024-09-16T15:51:49.434938+0000","last_undegraded":"2024-09-16T15:51:49.434938+0000","last_fullsized":"2024-09-16T15:51:49.434938+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":167,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:51:47.345679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:51:47.345679+0000","last_clean_scrub_stamp":"2024-09-16T15:51:47.345679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T18:33:00.669417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.1a","version":"168'1","reported_seq":14,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-09-16T15:51:48.445926+0000","last_change":"2024-09-16T15:51:47.379351+0000","last_active":"2024-09-16T15:51:48.445926+0000","last_peered":"2024-09-16T15:51:48.445926+0000","last_clean":"2024-09-16T15:51:48.445926+0000","last_became_active":"2024-09-16T15:51:47.378615+0000","last_became_peered":"2024-09-16T15:51:47.378615+0000","last_unstale":"2024-09-16T15:51:48.445926+0000","last_undegraded":"2024-09-16T15:51:48.445926+0000","last_fullsized":"2024-09-16T15:51:48.445926+0000","mapping_epoch":166,"log_start":"0'0","ondisk_log_start":"0'0","created":166,"last_epoch_clean":167,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:51:46.335209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:51:46.335209+0000","last_clean_scrub_stamp":"2024-09-16T15:51:46.335209+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T22:30:33.805316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.18","version":"0'0","reported_seq":10,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-09-16T15:51:48.420788+0000","last_change":"2024-09-16T15:51:48.409378+0000","last_active":"2024-09-16T15:51:48.420788+0000","last_peered":"2024-09-16T15:51:48.420788+0000","last_clean":"2024-09-16T15:51:48.420788+0000","last_became_active":"2024-09-16T15:51:48.408591+0000","last_became_peered":"2024-09-16T15:51:48.408591+0000","last_unstale":"2024-09-16T15:51:48.420788+0000","last_undegraded":"2024-09-16T15:51:48.420788+0000","last_fullsized":"2024-09-16T15:51:48.420788+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":167,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:51:47.345679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:51:47.345679+0000","last_clean_scrub_stamp":"2024-09-16T15:51:47.345679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:50:38.970224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.19","version":"0'0","reported_seq":13,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-09-16T15:51:48.380087+0000","last_change":"2024-09-16T15:51:47.378743+0000","last_active":"2024-09-16T15:51:48.380087+0000","last_peered":"2024-09-16T15:51:48.380087+0000","last_clean":"2024-09-16T15:51:48.380087+0000","last_became_active":"2024-09-16T15:51:47.378347+0000","last_became_peered":"2024-09-16T15:51:47.378347+0000","last_unstale":"2024-09-16T15:51:48.380087+0000","last_undegraded":"2024-09-16T15:51:48.380087+0000","last_fullsized":"2024-09-16T15:51:48.380087+0000","mapping_epoch":166,"log_start":"0'0","ondisk_log_start":"0'0","created":166,"last_epoch_clean":167,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:51:46.335209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:51:46.335209+0000","last_clean_scrub_stamp":"2024-09-16T15:51:46.335209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T22:53:43.066660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.19","version":"0'0","reported_seq":13,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-09-16T15:51:49.434797+0000","last_change":"2024-09-16T15:51:48.401062+0000","last_active":"2024-09-16T15:51:49.434797+0000","last_peered":"2024-09-16T15:51:49.434797+0000","last_clean":"2024-09-16T15:51:49.434797+0000","last_became_active":"2024-09-16T15:51:48.400353+0000","last_became_peered":"2024-09-16T15:51:48.400353+0000","last_unstale":"2024-09-16T15:51:49.434797+0000","last_undegraded":"2024-09-16T15:51:49.434797+0000","last_fullsized":"2024-09-16T15:51:49.434797+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":167,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:51:47.345679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:51:47.345679+0000","last_clean_scrub_stamp":"2024-09-16T15:51:47.345679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T00:16:19.141908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.18","version":"0'0","reported_seq":15,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-09-16T15:51:49.390219+0000","last_change":"2024-09-16T15:51:47.373792+0000","last_active":"2024-09-16T15:51:49.390219+0000","last_peered":"2024-09-16T15:51:49.390219+0000","last_clean":"2024-09-16T15:51:49.390219+0000","last_became_active":"2024-09-16T15:51:47.373004+0000","last_became_peered":"2024-09-16T15:51:47.373004+0000","last_unstale":"2024-09-16T15:51:49.390219+0000","last_undegraded":"2024-09-16T15:51:49.390219+0000","last_fullsized":"2024-09-16T15:51:49.390219+0000","mapping_epoch":166,"log_start":"0'0","ondisk_log_start":"0'0","created":166,"last_epoch_clean":167,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:51:46.335209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:51:46.335209+0000","last_clean_scrub_stamp":"2024-09-16T15:51:46.335209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T18:06:31.705358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.16","version":"0'0","reported_seq":13,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-09-16T15:51:49.435180+0000","last_change":"2024-09-16T15:51:48.424457+0000","last_active":"2024-09-16T15:51:49.435180+0000","last_peered":"2024-09-16T15:51:49.435180+0000","last_clean":"2024-09-16T15:51:49.435180+0000","last_became_active":"2024-09-16T15:51:48.423713+0000","last_became_peered":"2024-09-16T15:51:48.423713+0000","last_unstale":"2024-09-16T15:51:49.435180+0000","last_undegraded":"2024-09-16T15:51:49.435180+0000","last_fullsized":"2024-09-16T15:51:49.435180+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":167,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:51:47.345679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:51:47.345679+0000","last_clean_scrub_stamp":"2024-09-16T15:51:47.345679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T03:09:24.867616+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.17","version":"0'0","reported_seq":13,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-09-16T15:51:48.380248+0000","last_change":"2024-09-16T15:51:47.393217+0000","last_active":"2024-09-16T15:51:48.380248+0000","last_peered":"2024-09-16T15:51:48.380248+0000","last_clean":"2024-09-16T15:51:48.380248+0000","last_became_active":"2024-09-16T15:51:47.392883+0000","last_became_peered":"2024-09-16T15:51:47.392883+0000","last_unstale":"2024-09-16T15:51:48.380248+0000","last_undegraded":"2024-09-16T15:51:48.380248+0000","last_fullsized":"2024-09-16T15:51:48.380248+0000","mapping_epoch":166,"log_start":"0'0","ondisk_log_start":"0'0","created":166,"last_epoch_clean":167,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:51:46.335209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:51:46.335209+0000","last_clean_scrub_stamp":"2024-09-16T15:51:46.335209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:27:58.632284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.17","version":"0'0","reported_seq":13,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-09-16T15:51:49.392511+0000","last_change":"2024-09-16T15:51:48.405334+0000","last_active":"2024-09-16T15:51:49.392511+0000","last_peered":"2024-09-16T15:51:49.392511+0000","last_clean":"2024-09-16T15:51:49.392511+0000","last_became_active":"2024-09-16T15:51:48.401719+0000","last_became_peered":"2024-09-16T15:51:48.401719+0000","last_unstale":"2024-09-16T15:51:49.392511+0000","last_undegraded":"2024-09-16T15:51:49.392511+0000","last_fullsized":"2024-09-16T15:51:49.392511+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":167,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:51:47.345679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:51:47.345679+0000","last_clean_scrub_stamp":"2024-09-16T15:51:47.345679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:31:48.586582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.16","version":"0'0","reported_seq":15,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-09-16T15:51:49.393363+0000","last_change":"2024-09-16T15:51:47.386177+0000","last_active":"2024-09-16T15:51:49.393363+0000","last_peered":"2024-09-16T15:51:49.393363+0000","last_clean":"2024-09-16T15:51:49.393363+0000","last_became_active":"2024-09-16T15:51:47.385463+0000","last_became_peered":"2024-09-16T15:51:47.385463+0000","last_unstale":"2024-09-16T15:51:49.393363+0000","last_undegraded":"2024-09-16T15:51:49.393363+0000","last_fullsized":"2024-09-16T15:51:49.393363+0000","mapping_epoch":166,"log_start":"0'0","ondisk_log_start":"0'0","created":166,"last_epoch_clean":167,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:51:46.335209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:51:46.335209+0000","last_clean_scrub_stamp":"2024-09-16T15:51:46.335209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T23:19:12.494535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.14","version":"0'0","reported_seq":13,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-09-16T15:51:49.436244+0000","last_change":"2024-09-16T15:51:48.408427+0000","last_active":"2024-09-16T15:51:49.436244+0000","last_peered":"2024-09-16T15:51:49.436244+0000","last_clean":"2024-09-16T15:51:49.436244+0000","last_became_active":"2024-09-16T15:51:48.407616+0000","last_became_peered":"2024-09-16T15:51:48.407616+0000","last_unstale":"2024-09-16T15:51:49.436244+0000","last_undegraded":"2024-09-16T15:51:49.436244+0000","last_fullsized":"2024-09-16T15:51:49.436244+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":167,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:51:47.345679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:51:47.345679+0000","last_clean_scrub_stamp":"2024-09-16T15:51:47.345679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:12:44.001487+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.15","version":"168'3","reported_seq":16,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-09-16T15:51:48.446011+0000","last_change":"2024-09-16T15:51:47.376322+0000","last_active":"2024-09-16T15:51:48.446011+0000","last_peered":"2024-09-16T15:51:48.446011+0000","last_clean":"2024-09-16T15:51:48.446011+0000","last_became_active":"2024-09-16T15:51:47.375919+0000","last_became_peered":"2024-09-16T15:51:47.375919+0000","last_unstale":"2024-09-16T15:51:48.446011+0000","last_undegraded":"2024-09-16T15:51:48.446011+0000","last_fullsized":"2024-09-16T15:51:48.446011+0000","mapping_epoch":166,"log_start":"0'0","ondisk_log_start":"0'0","created":166,"last_epoch_clean":167,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:51:46.335209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:51:46.335209+0000","last_clean_scrub_stamp":"2024-09-16T15:51:46.335209+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T23:40:12.607073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.15","version":"0'0","reported_seq":13,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-09-16T15:51:49.690977+0000","last_change":"2024-09-16T15:51:48.434547+0000","last_active":"2024-09-16T15:51:49.690977+0000","last_peered":"2024-09-16T15:51:49.690977+0000","last_clean":"2024-09-16T15:51:49.690977+0000","last_became_active":"2024-09-16T15:51:48.432699+0000","last_became_peered":"2024-09-16T15:51:48.432699+0000","last_unstale":"2024-09-16T15:51:49.690977+0000","last_undegraded":"2024-09-16T15:51:49.690977+0000","last_fullsized":"2024-09-16T15:51:49.690977+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":167,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:51:47.345679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:51:47.345679+0000","last_clean_scrub_stamp":"2024-09-16T15:51:47.345679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T02:30:58.027562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.14","version":"168'6","reported_seq":21,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-09-16T15:51:49.436437+0000","last_change":"2024-09-16T15:51:47.391363+0000","last_active":"2024-09-16T15:51:49.436437+0000","last_peered":"2024-09-16T15:51:49.436437+0000","last_clean":"2024-09-16T15:51:49.436437+0000","last_became_active":"2024-09-16T15:51:47.390582+0000","last_became_peered":"2024-09-16T15:51:47.390582+0000","last_unstale":"2024-09-16T15:51:49.436437+0000","last_undegraded":"2024-09-16T15:51:49.436437+0000","last_fullsized":"2024-09-16T15:51:49.436437+0000","mapping_epoch":166,"log_start":"0'0","ondisk_log_start":"0'0","created":166,"last_epoch_clean":167,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:51:46.335209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:51:46.335209+0000","last_clean_scrub_stamp":"2024-09-16T15:51:46.335209+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:03:11.083680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.12","version":"0'0","reported_seq":10,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-09-16T15:51:48.422820+0000","last_change":"2024-09-16T15:51:48.410219+0000","last_active":"2024-09-16T15:51:48.422820+0000","last_peered":"2024-09-16T15:51:48.422820+0000","last_clean":"2024-09-16T15:51:48.422820+0000","last_became_active":"2024-09-16T15:51:48.409024+0000","last_became_peered":"2024-09-16T15:51:48.409024+0000","last_unstale":"2024-09-16T15:51:48.422820+0000","last_undegraded":"2024-09-16T15:51:48.422820+0000","last_fullsized":"2024-09-16T15:51:48.422820+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":167,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:51:47.345679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:51:47.345679+0000","last_clean_scrub_stamp":"2024-09-16T15:51:47.345679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T22:56:45.418432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.13","version":"168'2","reported_seq":17,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-09-16T15:51:49.392594+0000","last_change":"2024-09-16T15:51:47.368253+0000","last_active":"2024-09-16T15:51:49.392594+0000","last_peered":"2024-09-16T15:51:49.392594+0000","last_clean":"2024-09-16T15:51:49.392594+0000","last_became_active":"2024-09-16T15:51:47.366051+0000","last_became_peered":"2024-09-16T15:51:47.366051+0000","last_unstale":"2024-09-16T15:51:49.392594+0000","last_undegraded":"2024-09-16T15:51:49.392594+0000","last_fullsized":"2024-09-16T15:51:49.392594+0000","mapping_epoch":166,"log_start":"0'0","ondisk_log_start":"0'0","created":166,"last_epoch_clean":167,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:51:46.335209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:51:46.335209+0000","last_clean_scrub_stamp":"2024-09-16T15:51:46.335209+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T23:08:10.181813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.13","version":"0'0","reported_seq":10,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-09-16T15:51:48.428591+0000","last_change":"2024-09-16T15:51:48.425678+0000","last_active":"2024-09-16T15:51:48.428591+0000","last_peered":"2024-09-16T15:51:48.428591+0000","last_clean":"2024-09-16T15:51:48.428591+0000","last_became_active":"2024-09-16T15:51:48.425146+0000","last_became_peered":"2024-09-16T15:51:48.425146+0000","last_unstale":"2024-09-16T15:51:48.428591+0000","last_undegraded":"2024-09-16T15:51:48.428591+0000","last_fullsized":"2024-09-16T15:51:48.428591+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":167,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:51:47.345679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:51:47.345679+0000","last_clean_scrub_stamp":"2024-09-16T15:51:47.345679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T02:02:17.658639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.12","version":"0'0","reported_seq":15,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-09-16T15:51:49.392563+0000","last_change":"2024-09-16T15:51:47.376032+0000","last_active":"2024-09-16T15:51:49.392563+0000","last_peered":"2024-09-16T15:51:49.392563+0000","last_clean":"2024-09-16T15:51:49.392563+0000","last_became_active":"2024-09-16T15:51:47.375253+0000","last_became_peered":"2024-09-16T15:51:47.375253+0000","last_unstale":"2024-09-16T15:51:49.392563+0000","last_undegraded":"2024-09-16T15:51:49.392563+0000","last_fullsized":"2024-09-16T15:51:49.392563+0000","mapping_epoch":166,"log_start":"0'0","ondisk_log_start":"0'0","created":166,"last_epoch_clean":167,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:51:46.335209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:51:46.335209+0000","last_clean_scrub_stamp":"2024-09-16T15:51:46.335209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:26:09.762385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.10","version":"0'0","reported_seq":13,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-09-16T15:51:49.391262+0000","last_change":"2024-09-16T15:51:48.405340+0000","last_active":"2024-09-16T15:51:49.391262+0000","last_peered":"2024-09-16T15:51:49.391262+0000","last_clean":"2024-09-16T15:51:49.391262+0000","last_became_active":"2024-09-16T15:51:48.404594+0000","last_became_peered":"2024-09-16T15:51:48.404594+0000","last_unstale":"2024-09-16T15:51:49.391262+0000","last_undegraded":"2024-09-16T15:51:49.391262+0000","last_fullsized":"2024-09-16T15:51:49.391262+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":167,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:51:47.345679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:51:47.345679+0000","last_clean_scrub_stamp":"2024-09-16T15:51:47.345679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T02:48:42.809959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.11","version":"0'0","reported_seq":13,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-09-16T15:51:48.401862+0000","last_change":"2024-09-16T15:51:47.372021+0000","last_active":"2024-09-16T15:51:48.401862+0000","last_peered":"2024-09-16T15:51:48.401862+0000","last_clean":"2024-09-16T15:51:48.401862+0000","last_became_active":"2024-09-16T15:51:47.371683+0000","last_became_peered":"2024-09-16T15:51:47.371683+0000","last_unstale":"2024-09-16T15:51:48.401862+0000","last_undegraded":"2024-09-16T15:51:48.401862+0000","last_fullsized":"2024-09-16T15:51:48.401862+0000","mapping_epoch":166,"log_start":"0'0","ondisk_log_start":"0'0","created":166,"last_epoch_clean":167,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:51:46.335209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:51:46.335209+0000","last_clean_scrub_stamp":"2024-09-16T15:51:46.335209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:37:21.424729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.11","version":"0'0","reported_seq":13,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-09-16T15:51:49.690210+0000","last_change":"2024-09-16T15:51:48.434577+0000","last_active":"2024-09-16T15:51:49.690210+0000","last_peered":"2024-09-16T15:51:49.690210+0000","last_clean":"2024-09-16T15:51:49.690210+0000","last_became_active":"2024-09-16T15:51:48.432783+0000","last_became_peered":"2024-09-16T15:51:48.432783+0000","last_unstale":"2024-09-16T15:51:49.690210+0000","last_undegraded":"2024-09-16T15:51:49.690210+0000","last_fullsized":"2024-09-16T15:51:49.690210+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":167,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:51:47.345679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:51:47.345679+0000","last_clean_scrub_stamp":"2024-09-16T15:51:47.345679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:43:07.268710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.10","version":"168'8","reported_seq":23,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-09-16T15:51:49.391481+0000","last_change":"2024-09-16T15:51:47.385794+0000","last_active":"2024-09-16T15:51:49.391481+0000","last_peered":"2024-09-16T15:51:49.391481+0000","last_clean":"2024-09-16T15:51:49.391481+0000","last_became_active":"2024-09-16T15:51:47.385509+0000","last_became_peered":"2024-09-16T15:51:47.385509+0000","last_unstale":"2024-09-16T15:51:49.391481+0000","last_undegraded":"2024-09-16T15:51:49.391481+0000","last_fullsized":"2024-09-16T15:51:49.391481+0000","mapping_epoch":166,"log_start":"0'0","ondisk_log_start":"0'0","created":166,"last_epoch_clean":167,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:51:46.335209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:51:46.335209+0000","last_clean_scrub_stamp":"2024-09-16T15:51:46.335209+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T22:56:37.405888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.e","version":"0'0","reported_seq":10,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-09-16T15:51:48.442381+0000","last_change":"2024-09-16T15:51:48.433013+0000","last_active":"2024-09-16T15:51:48.442381+0000","last_peered":"2024-09-16T15:51:48.442381+0000","last_clean":"2024-09-16T15:51:48.442381+0000","last_became_active":"2024-09-16T15:51:48.432316+0000","last_became_peered":"2024-09-16T15:51:48.432316+0000","last_unstale":"2024-09-16T15:51:48.442381+0000","last_undegraded":"2024-09-16T15:51:48.442381+0000","last_fullsized":"2024-09-16T15:51:48.442381+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":167,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:51:47.345679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:51:47.345679+0000","last_clean_scrub_stamp":"2024-09-16T15:51:47.345679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T18:04:29.405916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.f","version":"168'1","reported_seq":14,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-09-16T15:51:48.446043+0000","last_change":"2024-09-16T15:51:47.379898+0000","last_active":"2024-09-16T15:51:48.446043+0000","last_peered":"2024-09-16T15:51:48.446043+0000","last_clean":"2024-09-16T15:51:48.446043+0000","last_became_active":"2024-09-16T15:51:47.379527+0000","last_became_peered":"2024-09-16T15:51:47.379527+0000","last_unstale":"2024-09-16T15:51:48.446043+0000","last_undegraded":"2024-09-16T15:51:48.446043+0000","last_fullsized":"2024-09-16T15:51:48.446043+0000","mapping_epoch":166,"log_start":"0'0","ondisk_log_start":"0'0","created":166,"last_epoch_clean":167,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:51:46.335209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:51:46.335209+0000","last_clean_scrub_stamp":"2024-09-16T15:51:46.335209+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:25:10.866081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.f","version":"0'0","reported_seq":10,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-09-16T15:51:48.441163+0000","last_change":"2024-09-16T15:51:48.430112+0000","last_active":"2024-09-16T15:51:48.441163+0000","last_peered":"2024-09-16T15:51:48.441163+0000","last_clean":"2024-09-16T15:51:48.441163+0000","last_became_active":"2024-09-16T15:51:48.429443+0000","last_became_peered":"2024-09-16T15:51:48.429443+0000","last_unstale":"2024-09-16T15:51:48.441163+0000","last_undegraded":"2024-09-16T15:51:48.441163+0000","last_fullsized":"2024-09-16T15:51:48.441163+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":167,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:51:47.345679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:51:47.345679+0000","last_clean_scrub_stamp":"2024-09-16T15:51:47.345679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:55:21.189810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.e","version":"168'2","reported_seq":15,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-09-16T15:51:48.447973+0000","last_change":"2024-09-16T15:51:47.384057+0000","last_active":"2024-09-16T15:51:48.447973+0000","last_peered":"2024-09-16T15:51:48.447973+0000","last_clean":"2024-09-16T15:51:48.447973+0000","last_became_active":"2024-09-16T15:51:47.381514+0000","last_became_peered":"2024-09-16T15:51:47.381514+0000","last_unstale":"2024-09-16T15:51:48.447973+0000","last_undegraded":"2024-09-16T15:51:48.447973+0000","last_fullsized":"2024-09-16T15:51:48.447973+0000","mapping_epoch":166,"log_start":"0'0","ondisk_log_start":"0'0","created":166,"last_epoch_clean":167,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:51:46.335209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:51:46.335209+0000","last_clean_scrub_stamp":"2024-09-16T15:51:46.335209+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T00:21:03.749321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.c","version":"0'0","reported_seq":13,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-09-16T15:51:49.391803+0000","last_change":"2024-09-16T15:51:48.428177+0000","last_active":"2024-09-16T15:51:49.391803+0000","last_peered":"2024-09-16T15:51:49.391803+0000","last_clean":"2024-09-16T15:51:49.391803+0000","last_became_active":"2024-09-16T15:51:48.427720+0000","last_became_peered":"2024-09-16T15:51:48.427720+0000","last_unstale":"2024-09-16T15:51:49.391803+0000","last_undegraded":"2024-09-16T15:51:49.391803+0000","last_fullsized":"2024-09-16T15:51:49.391803+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":167,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:51:47.345679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:51:47.345679+0000","last_clean_scrub_stamp":"2024-09-16T15:51:47.345679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T02:35:03.111789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.d","version":"168'1","reported_seq":14,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-09-16T15:51:48.447409+0000","last_change":"2024-09-16T15:51:47.372080+0000","last_active":"2024-09-16T15:51:48.447409+0000","last_peered":"2024-09-16T15:51:48.447409+0000","last_clean":"2024-09-16T15:51:48.447409+0000","last_became_active":"2024-09-16T15:51:47.371640+0000","last_became_peered":"2024-09-16T15:51:47.371640+0000","last_unstale":"2024-09-16T15:51:48.447409+0000","last_undegraded":"2024-09-16T15:51:48.447409+0000","last_fullsized":"2024-09-16T15:51:48.447409+0000","mapping_epoch":166,"log_start":"0'0","ondisk_log_start":"0'0","created":166,"last_epoch_clean":167,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:51:46.335209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:51:46.335209+0000","last_clean_scrub_stamp":"2024-09-16T15:51:46.335209+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T23:13:41.484039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.d","version":"0'0","reported_seq":10,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-09-16T15:51:48.418959+0000","last_change":"2024-09-16T15:51:48.412794+0000","last_active":"2024-09-16T15:51:48.418959+0000","last_peered":"2024-09-16T15:51:48.418959+0000","last_clean":"2024-09-16T15:51:48.418959+0000","last_became_active":"2024-09-16T15:51:48.411719+0000","last_became_peered":"2024-09-16T15:51:48.411719+0000","last_unstale":"2024-09-16T15:51:48.418959+0000","last_undegraded":"2024-09-16T15:51:48.418959+0000","last_fullsized":"2024-09-16T15:51:48.418959+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":167,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:51:47.345679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:51:47.345679+0000","last_clean_scrub_stamp":"2024-09-16T15:51:47.345679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:16:35.479295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.c","version":"0'0","reported_seq":15,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-09-16T15:51:49.391949+0000","last_change":"2024-09-16T15:51:47.376422+0000","last_active":"2024-09-16T15:51:49.391949+0000","last_peered":"2024-09-16T15:51:49.391949+0000","last_clean":"2024-09-16T15:51:49.391949+0000","last_became_active":"2024-09-16T15:51:47.376112+0000","last_became_peered":"2024-09-16T15:51:47.376112+0000","last_unstale":"2024-09-16T15:51:49.391949+0000","last_undegraded":"2024-09-16T15:51:49.391949+0000","last_fullsized":"2024-09-16T15:51:49.391949+0000","mapping_epoch":166,"log_start":"0'0","ondisk_log_start":"0'0","created":166,"last_epoch_clean":167,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:51:46.335209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:51:46.335209+0000","last_clean_scrub_stamp":"2024-09-16T15:51:46.335209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T00:19:10.515395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.a","version":"0'0","reported_seq":10,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-09-16T15:51:48.421215+0000","last_change":"2024-09-16T15:51:48.415593+0000","last_active":"2024-09-16T15:51:48.421215+0000","last_peered":"2024-09-16T15:51:48.421215+0000","last_clean":"2024-09-16T15:51:48.421215+0000","last_became_active":"2024-09-16T15:51:48.412647+0000","last_became_peered":"2024-09-16T15:51:48.412647+0000","last_unstale":"2024-09-16T15:51:48.421215+0000","last_undegraded":"2024-09-16T15:51:48.421215+0000","last_fullsized":"2024-09-16T15:51:48.421215+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":167,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:51:47.345679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:51:47.345679+0000","last_clean_scrub_stamp":"2024-09-16T15:51:47.345679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:11:00.105837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.b","version":"168'2","reported_seq":15,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-09-16T15:51:48.449047+0000","last_change":"2024-09-16T15:51:47.373400+0000","last_active":"2024-09-16T15:51:48.449047+0000","last_peered":"2024-09-16T15:51:48.449047+0000","last_clean":"2024-09-16T15:51:48.449047+0000","last_became_active":"2024-09-16T15:51:47.372217+0000","last_became_peered":"2024-09-16T15:51:47.372217+0000","last_unstale":"2024-09-16T15:51:48.449047+0000","last_undegraded":"2024-09-16T15:51:48.449047+0000","last_fullsized":"2024-09-16T15:51:48.449047+0000","mapping_epoch":166,"log_start":"0'0","ondisk_log_start":"0'0","created":166,"last_epoch_clean":167,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:51:46.335209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:51:46.335209+0000","last_clean_scrub_stamp":"2024-09-16T15:51:46.335209+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T01:00:00.780682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.b","version":"0'0","reported_seq":13,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-09-16T15:51:49.390999+0000","last_change":"2024-09-16T15:51:48.400042+0000","last_active":"2024-09-16T15:51:49.390999+0000","last_peered":"2024-09-16T15:51:49.390999+0000","last_clean":"2024-09-16T15:51:49.390999+0000","last_became_active":"2024-09-16T15:51:48.399206+0000","last_became_peered":"2024-09-16T15:51:48.399206+0000","last_unstale":"2024-09-16T15:51:49.390999+0000","last_undegraded":"2024-09-16T15:51:49.390999+0000","last_fullsized":"2024-09-16T15:51:49.390999+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":167,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:51:47.345679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:51:47.345679+0000","last_clean_scrub_stamp":"2024-09-16T15:51:47.345679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T00:06:40.282867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.a","version":"0'0","reported_seq":15,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-09-16T15:51:49.690554+0000","last_change":"2024-09-16T15:51:47.386137+0000","last_active":"2024-09-16T15:51:49.690554+0000","last_peered":"2024-09-16T15:51:49.690554+0000","last_clean":"2024-09-16T15:51:49.690554+0000","last_became_active":"2024-09-16T15:51:47.385852+0000","last_became_peered":"2024-09-16T15:51:47.385852+0000","last_unstale":"2024-09-16T15:51:49.690554+0000","last_undegraded":"2024-09-16T15:51:49.690554+0000","last_fullsized":"2024-09-16T15:51:49.690554+0000","mapping_epoch":166,"log_start":"0'0","ondisk_log_start":"0'0","created":166,"last_epoch_clean":167,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:51:46.335209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:51:46.335209+0000","last_clean_scrub_stamp":"2024-09-16T15:51:46.335209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:04:19.459115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.8","version":"0'0","reported_seq":10,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-09-16T15:51:48.440570+0000","last_change":"2024-09-16T15:51:48.427352+0000","last_active":"2024-09-16T15:51:48.440570+0000","last_peered":"2024-09-16T15:51:48.440570+0000","last_clean":"2024-09-16T15:51:48.440570+0000","last_became_active":"2024-09-16T15:51:48.426933+0000","last_became_peered":"2024-09-16T15:51:48.426933+0000","last_unstale":"2024-09-16T15:51:48.440570+0000","last_undegraded":"2024-09-16T15:51:48.440570+0000","last_fullsized":"2024-09-16T15:51:48.440570+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":167,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:51:47.345679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:51:47.345679+0000","last_clean_scrub_stamp":"2024-09-16T15:51:47.345679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T18:02:06.552134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.9","version":"0'0","reported_seq":15,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-09-16T15:51:49.690367+0000","last_change":"2024-09-16T15:51:47.379850+0000","last_active":"2024-09-16T15:51:49.690367+0000","last_peered":"2024-09-16T15:51:49.690367+0000","last_clean":"2024-09-16T15:51:49.690367+0000","last_became_active":"2024-09-16T15:51:47.379448+0000","last_became_peered":"2024-09-16T15:51:47.379448+0000","last_unstale":"2024-09-16T15:51:49.690367+0000","last_undegraded":"2024-09-16T15:51:49.690367+0000","last_fullsized":"2024-09-16T15:51:49.690367+0000","mapping_epoch":166,"log_start":"0'0","ondisk_log_start":"0'0","created":166,"last_epoch_clean":167,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:51:46.335209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:51:46.335209+0000","last_clean_scrub_stamp":"2024-09-16T15:51:46.335209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:21:24.176527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.9","version":"0'0","reported_seq":13,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-09-16T15:51:49.391718+0000","last_change":"2024-09-16T15:51:48.388897+0000","last_active":"2024-09-16T15:51:49.391718+0000","last_peered":"2024-09-16T15:51:49.391718+0000","last_clean":"2024-09-16T15:51:49.391718+0000","last_became_active":"2024-09-16T15:51:48.388059+0000","last_became_peered":"2024-09-16T15:51:48.388059+0000","last_unstale":"2024-09-16T15:51:49.391718+0000","last_undegraded":"2024-09-16T15:51:49.391718+0000","last_fullsized":"2024-09-16T15:51:49.391718+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":167,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:51:47.345679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:51:47.345679+0000","last_clean_scrub_stamp":"2024-09-16T15:51:47.345679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T00:26:02.794506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.8","version":"0'0","reported_seq":13,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-09-16T15:51:48.382381+0000","last_change":"2024-09-16T15:51:47.378342+0000","last_active":"2024-09-16T15:51:48.382381+0000","last_peered":"2024-09-16T15:51:48.382381+0000","last_clean":"2024-09-16T15:51:48.382381+0000","last_became_active":"2024-09-16T15:51:47.378014+0000","last_became_peered":"2024-09-16T15:51:47.378014+0000","last_unstale":"2024-09-16T15:51:48.382381+0000","last_undegraded":"2024-09-16T15:51:48.382381+0000","last_fullsized":"2024-09-16T15:51:48.382381+0000","mapping_epoch":166,"log_start":"0'0","ondisk_log_start":"0'0","created":166,"last_epoch_clean":167,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:51:46.335209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:51:46.335209+0000","last_clean_scrub_stamp":"2024-09-16T15:51:46.335209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:46:57.710217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.6","version":"0'0","reported_seq":13,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-09-16T15:51:49.392146+0000","last_change":"2024-09-16T15:51:48.408607+0000","last_active":"2024-09-16T15:51:49.392146+0000","last_peered":"2024-09-16T15:51:49.392146+0000","last_clean":"2024-09-16T15:51:49.392146+0000","last_became_active":"2024-09-16T15:51:48.407850+0000","last_became_peered":"2024-09-16T15:51:48.407850+0000","last_unstale":"2024-09-16T15:51:49.392146+0000","last_undegraded":"2024-09-16T15:51:49.392146+0000","last_fullsized":"2024-09-16T15:51:49.392146+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":167,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:51:47.345679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:51:47.345679+0000","last_clean_scrub_stamp":"2024-09-16T15:51:47.345679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:35:12.916299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"26.7","version":"168'2","reported_seq":17,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-09-16T15:51:49.392347+0000","last_change":"2024-09-16T15:51:47.389839+0000","last_active":"2024-09-16T15:51:49.392347+0000","last_peered":"2024-09-16T15:51:49.392347+0000","last_clean":"2024-09-16T15:51:49.392347+0000","last_became_active":"2024-09-16T15:51:47.389541+0000","last_became_peered":"2024-09-16T15:51:47.389541+0000","last_unstale":"2024-09-16T15:51:49.392347+0000","last_undegraded":"2024-09-16T15:51:49.392347+0000","last_fullsized":"2024-09-16T15:51:49.392347+0000","mapping_epoch":166,"log_start":"0'0","ondisk_log_start":"0'0","created":166,"last_epoch_clean":167,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:51:46.335209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:51:46.335209+0000","last_clean_scrub_stamp":"2024-09-16T15:51:46.335209+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:06:10.859212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.7","version":"0'0","reported_seq":13,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-09-16T15:51:49.690969+0000","last_change":"2024-09-16T15:51:48.435404+0000","last_active":"2024-09-16T15:51:49.690969+0000","last_peered":"2024-09-16T15:51:49.690969+0000","last_clean":"2024-09-16T15:51:49.690969+0000","last_became_active":"2024-09-16T15:51:48.435074+0000","last_became_peered":"2024-09-16T15:51:48.435074+0000","last_unstale":"2024-09-16T15:51:49.690969+0000","last_undegraded":"2024-09-16T15:51:49.690969+0000","last_fullsized":"2024-09-16T15:51:49.690969+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":167,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:51:47.345679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:51:47.345679+0000","last_clean_scrub_stamp":"2024-09-16T15:51:47.345679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T01:52:02.077065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.6","version":"169'1","reported_seq":16,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-09-16T15:51:49.438689+0000","last_change":"2024-09-16T15:51:47.396058+0000","last_active":"2024-09-16T15:51:49.438689+0000","last_peered":"2024-09-16T15:51:49.438689+0000","last_clean":"2024-09-16T15:51:49.438689+0000","last_became_active":"2024-09-16T15:51:47.395107+0000","last_became_peered":"2024-09-16T15:51:47.395107+0000","last_unstale":"2024-09-16T15:51:49.438689+0000","last_undegraded":"2024-09-16T15:51:49.438689+0000","last_fullsized":"2024-09-16T15:51:49.438689+0000","mapping_epoch":166,"log_start":"0'0","ondisk_log_start":"0'0","created":166,"last_epoch_clean":167,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:51:46.335209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:51:46.335209+0000","last_clean_scrub_stamp":"2024-09-16T15:51:46.335209+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T02:26:22.249280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.4","version":"0'0","reported_seq":10,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-09-16T15:51:48.441382+0000","last_change":"2024-09-16T15:51:48.434690+0000","last_active":"2024-09-16T15:51:48.441382+0000","last_peered":"2024-09-16T15:51:48.441382+0000","last_clean":"2024-09-16T15:51:48.441382+0000","last_became_active":"2024-09-16T15:51:48.434213+0000","last_became_peered":"2024-09-16T15:51:48.434213+0000","last_unstale":"2024-09-16T15:51:48.441382+0000","last_undegraded":"2024-09-16T15:51:48.441382+0000","last_fullsized":"2024-09-16T15:51:48.441382+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":167,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:51:47.345679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:51:47.345679+0000","last_clean_scrub_stamp":"2024-09-16T15:51:47.345679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T23:02:30.232897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.5","version":"0'0","reported_seq":13,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-09-16T15:51:48.380068+0000","last_change":"2024-09-16T15:51:47.386153+0000","last_active":"2024-09-16T15:51:48.380068+0000","last_peered":"2024-09-16T15:51:48.380068+0000","last_clean":"2024-09-16T15:51:48.380068+0000","last_became_active":"2024-09-16T15:51:47.385788+0000","last_became_peered":"2024-09-16T15:51:47.385788+0000","last_unstale":"2024-09-16T15:51:48.380068+0000","last_undegraded":"2024-09-16T15:51:48.380068+0000","last_fullsized":"2024-09-16T15:51:48.380068+0000","mapping_epoch":166,"log_start":"0'0","ondisk_log_start":"0'0","created":166,"last_epoch_clean":167,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:51:46.335209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:51:46.335209+0000","last_clean_scrub_stamp":"2024-09-16T15:51:46.335209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T23:22:20.611338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.5","version":"0'0","reported_seq":13,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-09-16T15:51:49.393390+0000","last_change":"2024-09-16T15:51:48.421671+0000","last_active":"2024-09-16T15:51:49.393390+0000","last_peered":"2024-09-16T15:51:49.393390+0000","last_clean":"2024-09-16T15:51:49.393390+0000","last_became_active":"2024-09-16T15:51:48.419577+0000","last_became_peered":"2024-09-16T15:51:48.419577+0000","last_unstale":"2024-09-16T15:51:49.393390+0000","last_undegraded":"2024-09-16T15:51:49.393390+0000","last_fullsized":"2024-09-16T15:51:49.393390+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":167,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:51:47.345679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:51:47.345679+0000","last_clean_scrub_stamp":"2024-09-16T15:51:47.345679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T01:37:38.821256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.4","version":"0'0","reported_seq":15,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-09-16T15:51:49.391972+0000","last_change":"2024-09-16T15:51:47.385805+0000","last_active":"2024-09-16T15:51:49.391972+0000","last_peered":"2024-09-16T15:51:49.391972+0000","last_clean":"2024-09-16T15:51:49.391972+0000","last_became_active":"2024-09-16T15:51:47.385565+0000","last_became_peered":"2024-09-16T15:51:47.385565+0000","last_unstale":"2024-09-16T15:51:49.391972+0000","last_undegraded":"2024-09-16T15:51:49.391972+0000","last_fullsized":"2024-09-16T15:51:49.391972+0000","mapping_epoch":166,"log_start":"0'0","ondisk_log_start":"0'0","created":166,"last_epoch_clean":167,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:51:46.335209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:51:46.335209+0000","last_clean_scrub_stamp":"2024-09-16T15:51:46.335209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T00:21:03.749321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.2","version":"0'0","reported_seq":13,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-09-16T15:51:49.691184+0000","last_change":"2024-09-16T15:51:48.435351+0000","last_active":"2024-09-16T15:51:49.691184+0000","last_peered":"2024-09-16T15:51:49.691184+0000","last_clean":"2024-09-16T15:51:49.691184+0000","last_became_active":"2024-09-16T15:51:48.434975+0000","last_became_peered":"2024-09-16T15:51:48.434975+0000","last_unstale":"2024-09-16T15:51:49.691184+0000","last_undegraded":"2024-09-16T15:51:49.691184+0000","last_fullsized":"2024-09-16T15:51:49.691184+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":167,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:51:47.345679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:51:47.345679+0000","last_clean_scrub_stamp":"2024-09-16T15:51:47.345679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T01:31:06.621794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.3","version":"0'0","reported_seq":15,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-09-16T15:51:49.434947+0000","last_change":"2024-09-16T15:51:47.389057+0000","last_active":"2024-09-16T15:51:49.434947+0000","last_peered":"2024-09-16T15:51:49.434947+0000","last_clean":"2024-09-16T15:51:49.434947+0000","last_became_active":"2024-09-16T15:51:47.388202+0000","last_became_peered":"2024-09-16T15:51:47.388202+0000","last_unstale":"2024-09-16T15:51:49.434947+0000","last_undegraded":"2024-09-16T15:51:49.434947+0000","last_fullsized":"2024-09-16T15:51:49.434947+0000","mapping_epoch":166,"log_start":"0'0","ondisk_log_start":"0'0","created":166,"last_epoch_clean":167,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:51:46.335209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:51:46.335209+0000","last_clean_scrub_stamp":"2024-09-16T15:51:46.335209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:34:14.296519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.3","version":"0'0","reported_seq":13,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-09-16T15:51:49.390480+0000","last_change":"2024-09-16T15:51:48.403829+0000","last_active":"2024-09-16T15:51:49.390480+0000","last_peered":"2024-09-16T15:51:49.390480+0000","last_clean":"2024-09-16T15:51:49.390480+0000","last_became_active":"2024-09-16T15:51:48.402950+0000","last_became_peered":"2024-09-16T15:51:48.402950+0000","last_unstale":"2024-09-16T15:51:49.390480+0000","last_undegraded":"2024-09-16T15:51:49.390480+0000","last_fullsized":"2024-09-16T15:51:49.390480+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":167,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:51:47.345679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:51:47.345679+0000","last_clean_scrub_stamp":"2024-09-16T15:51:47.345679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:41:32.289325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"26.2","version":"0'0","reported_seq":13,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-09-16T15:51:48.381208+0000","last_change":"2024-09-16T15:51:47.379021+0000","last_active":"2024-09-16T15:51:48.381208+0000","last_peered":"2024-09-16T15:51:48.381208+0000","last_clean":"2024-09-16T15:51:48.381208+0000","last_became_active":"2024-09-16T15:51:47.378682+0000","last_became_peered":"2024-09-16T15:51:47.378682+0000","last_unstale":"2024-09-16T15:51:48.381208+0000","last_undegraded":"2024-09-16T15:51:48.381208+0000","last_fullsized":"2024-09-16T15:51:48.381208+0000","mapping_epoch":166,"log_start":"0'0","ondisk_log_start":"0'0","created":166,"last_epoch_clean":167,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:51:46.335209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:51:46.335209+0000","last_clean_scrub_stamp":"2024-09-16T15:51:46.335209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:02:05.529566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.0","version":"0'0","reported_seq":13,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-09-16T15:51:49.391316+0000","last_change":"2024-09-16T15:51:48.406255+0000","last_active":"2024-09-16T15:51:49.391316+0000","last_peered":"2024-09-16T15:51:49.391316+0000","last_clean":"2024-09-16T15:51:49.391316+0000","last_became_active":"2024-09-16T15:51:48.405514+0000","last_became_peered":"2024-09-16T15:51:48.405514+0000","last_unstale":"2024-09-16T15:51:49.391316+0000","last_undegraded":"2024-09-16T15:51:49.391316+0000","last_fullsized":"2024-09-16T15:51:49.391316+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":167,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:51:47.345679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:51:47.345679+0000","last_clean_scrub_stamp":"2024-09-16T15:51:47.345679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T18:46:13.824641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.1","version":"0'0","reported_seq":15,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-09-16T15:51:49.392009+0000","last_change":"2024-09-16T15:51:47.369871+0000","last_active":"2024-09-16T15:51:49.392009+0000","last_peered":"2024-09-16T15:51:49.392009+0000","last_clean":"2024-09-16T15:51:49.392009+0000","last_became_active":"2024-09-16T15:51:47.369135+0000","last_became_peered":"2024-09-16T15:51:47.369135+0000","last_unstale":"2024-09-16T15:51:49.392009+0000","last_undegraded":"2024-09-16T15:51:49.392009+0000","last_fullsized":"2024-09-16T15:51:49.392009+0000","mapping_epoch":166,"log_start":"0'0","ondisk_log_start":"0'0","created":166,"last_epoch_clean":167,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:51:46.335209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:51:46.335209+0000","last_clean_scrub_stamp":"2024-09-16T15:51:46.335209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:35:11.905829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.1","version":"0'0","reported_seq":10,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-09-16T15:51:48.440646+0000","last_change":"2024-09-16T15:51:48.429777+0000","last_active":"2024-09-16T15:51:48.440646+0000","last_peered":"2024-09-16T15:51:48.440646+0000","last_clean":"2024-09-16T15:51:48.440646+0000","last_became_active":"2024-09-16T15:51:48.428509+0000","last_became_peered":"2024-09-16T15:51:48.428509+0000","last_unstale":"2024-09-16T15:51:48.440646+0000","last_undegraded":"2024-09-16T15:51:48.440646+0000","last_fullsized":"2024-09-16T15:51:48.440646+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":167,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:51:47.345679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:51:47.345679+0000","last_clean_scrub_stamp":"2024-09-16T15:51:47.345679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T01:48:08.768795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.0","version":"0'0","reported_seq":15,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-09-16T15:51:49.389327+0000","last_change":"2024-09-16T15:51:47.378789+0000","last_active":"2024-09-16T15:51:49.389327+0000","last_peered":"2024-09-16T15:51:49.389327+0000","last_clean":"2024-09-16T15:51:49.389327+0000","last_became_active":"2024-09-16T15:51:47.378096+0000","last_became_peered":"2024-09-16T15:51:47.378096+0000","last_unstale":"2024-09-16T15:51:49.389327+0000","last_undegraded":"2024-09-16T15:51:49.389327+0000","last_fullsized":"2024-09-16T15:51:49.389327+0000","mapping_epoch":166,"log_start":"0'0","ondisk_log_start":"0'0","created":166,"last_epoch_clean":167,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:51:46.335209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:51:46.335209+0000","last_clean_scrub_stamp":"2024-09-16T15:51:46.335209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T00:54:28.885278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.1a","version":"0'0","reported_seq":13,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-09-16T15:51:49.690032+0000","last_change":"2024-09-16T15:51:48.433167+0000","last_active":"2024-09-16T15:51:49.690032+0000","last_peered":"2024-09-16T15:51:49.690032+0000","last_clean":"2024-09-16T15:51:49.690032+0000","last_became_active":"2024-09-16T15:51:48.432726+0000","last_became_peered":"2024-09-16T15:51:48.432726+0000","last_unstale":"2024-09-16T15:51:49.690032+0000","last_undegraded":"2024-09-16T15:51:49.690032+0000","last_fullsized":"2024-09-16T15:51:49.690032+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":167,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:51:47.345679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:51:47.345679+0000","last_clean_scrub_stamp":"2024-09-16T15:51:47.345679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T00:17:12.711120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.1b","version":"168'1","reported_seq":16,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-09-16T15:51:49.391363+0000","last_change":"2024-09-16T15:51:47.377240+0000","last_active":"2024-09-16T15:51:49.391363+0000","last_peered":"2024-09-16T15:51:49.391363+0000","last_clean":"2024-09-16T15:51:49.391363+0000","last_became_active":"2024-09-16T15:51:47.376510+0000","last_became_peered":"2024-09-16T15:51:47.376510+0000","last_unstale":"2024-09-16T15:51:49.391363+0000","last_undegraded":"2024-09-16T15:51:49.391363+0000","last_fullsized":"2024-09-16T15:51:49.391363+0000","mapping_epoch":166,"log_start":"0'0","ondisk_log_start":"0'0","created":166,"last_epoch_clean":167,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:51:46.335209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:51:46.335209+0000","last_clean_scrub_stamp":"2024-09-16T15:51:46.335209+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T00:08:59.993560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"73'79","reported_seq":412,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-09-16T15:51:49.434428+0000","last_change":"2024-09-16T15:42:40.881820+0000","last_active":"2024-09-16T15:51:49.434428+0000","last_peered":"2024-09-16T15:51:49.434428+0000","last_clean":"2024-09-16T15:51:49.434428+0000","last_became_active":"2024-09-16T15:42:40.879965+0000","last_became_peered":"2024-09-16T15:42:40.879965+0000","last_unstale":"2024-09-16T15:51:49.434428+0000","last_undegraded":"2024-09-16T15:51:49.434428+0000","last_fullsized":"2024-09-16T15:51:49.434428+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:42:39.728762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:42:39.728762+0000","last_clean_scrub_stamp":"2024-09-16T15:42:39.728762+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:22:18.768932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":26,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":27,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79,"ondisk_log_size":79,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542258,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33852,"kb_used_data":2660,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94337988,"statfs":{"total":96636764160,"available":96602099712,"internally_reserved":0,"allocated":2723840,"data_stored":1530773,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542257,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33264,"kb_used_data":2072,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338576,"statfs":{"total":96636764160,"available":96602701824,"internally_reserved":0,"allocated":2121728,"data_stored":939375,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":5,"up_from":14,"seq":60129542258,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33260,"kb_used_data":2072,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338580,"statfs":{"total":96636764160,"available":96602705920,"internally_reserved":0,"allocated":2121728,"data_stored":939375,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542256,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33288,"kb_used_data":2100,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338552,"statfs":{"total":96636764160,"available":96602677248,"internally_reserved":0,"allocated":2150400,"data_stored":946837,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":3,"up_from":14,"seq":60129542257,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37428,"kb_used_data":2080,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334412,"statfs":{"total":96636764160,"available":96598437888,"internally_reserved":0,"allocated":2129920,"data_stored":938975,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":14,"seq":60129542257,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33272,"kb_used_data":2080,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338568,"statfs":{"total":96636764160,"available":96602693632,"internally_reserved":0,"allocated":2129920,"data_stored":945121,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":14,"seq":60129542259,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33268,"kb_used_data":2076,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338572,"statfs":{"total":96636764160,"available":96602697728,"internally_reserved":0,"allocated":2125824,"data_stored":938849,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":0,"up_from":14,"seq":60129542259,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38008,"kb_used_data":2652,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94333832,"statfs":{"total":96636764160,"available":96597843968,"internally_reserved":0,"allocated":2715648,"data_stored":1529251,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1716,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T15:51:52.416 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T15:51:52.571 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:52.573+0000 7fa5b52de640 1 -- 172.21.15.123:0/3334685604 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa5b015e820 msgr2=0x7fa5b015ecd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:51:52.571 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:52.573+0000 7fa5b52de640 1 --2- 172.21.15.123:0/3334685604 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa5b015e820 0x7fa5b015ecd0 secure :-1 s=READY pgs=1886 cs=0 l=1 rev1=1 crypto rx=0x7fa5a8009bc0 tx=0x7fa5a801cb30 comp rx=0 tx=0).stop 2024-09-16T15:51:52.571 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:52.573+0000 7fa5b52de640 1 -- 172.21.15.123:0/3334685604 shutdown_connections 2024-09-16T15:51:52.571 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:52.573+0000 7fa5b52de640 1 --2- 172.21.15.123:0/3334685604 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa5b015e820 0x7fa5b015ecd0 unknown :-1 s=CLOSED pgs=1886 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:52.571 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:52.573+0000 7fa5b52de640 1 --2- 172.21.15.123:0/3334685604 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fa5b015e160 0x7fa5b0073f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:52.571 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:52.573+0000 7fa5b52de640 1 -- 172.21.15.123:0/3334685604 >> 172.21.15.123:0/3334685604 conn(0x7fa5b00cfb10 msgr2=0x7fa5b00cff20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:51:52.572 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:52.573+0000 7fa5b52de640 1 -- 172.21.15.123:0/3334685604 shutdown_connections 2024-09-16T15:51:52.572 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:52.573+0000 7fa5b52de640 1 -- 172.21.15.123:0/3334685604 wait complete. 2024-09-16T15:51:52.572 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:52.573+0000 7fa5b52de640 1 Processor -- start 2024-09-16T15:51:52.572 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:52.573+0000 7fa5b52de640 1 -- start start 2024-09-16T15:51:52.574 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:52.573+0000 7fa5b52de640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa5b0074470 0x7fa5b0154730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:51:52.574 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:52.573+0000 7fa5b52de640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa5b015e160 0x7fa5b01594a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:51:52.574 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:52.573+0000 7fa5b52de640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fa5b015e820 0x7fa5b01f6080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:51:52.574 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:52.573+0000 7fa5b52de640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fa5b00cf5b0 con 0x7fa5b0074470 2024-09-16T15:51:52.574 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:52.573+0000 7fa5b52de640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fa5b00cf430 con 0x7fa5b015e160 2024-09-16T15:51:52.574 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:52.573+0000 7fa5b52de640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fa5b01683d0 con 0x7fa5b015e820 2024-09-16T15:51:52.574 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:52.573+0000 7fa5aeffd640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa5b0074470 0x7fa5b0154730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:51:52.574 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:52.573+0000 7fa5aeffd640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa5b0074470 0x7fa5b0154730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:43470/0 (socket says 172.21.15.123:43470) 2024-09-16T15:51:52.574 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:52.573+0000 7fa5aeffd640 1 -- 172.21.15.123:0/3899135478 learned_addr learned my addr 172.21.15.123:0/3899135478 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:51:52.574 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:52.573+0000 7fa5af7fe640 1 --2- 172.21.15.123:0/3899135478 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fa5b015e820 0x7fa5b01f6080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:51:52.575 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:52.573+0000 7fa5aeffd640 1 -- 172.21.15.123:0/3899135478 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fa5b015e820 msgr2=0x7fa5b01f6080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:51:52.575 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:52.573+0000 7fa5aeffd640 1 --2- 172.21.15.123:0/3899135478 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fa5b015e820 0x7fa5b01f6080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:52.575 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:52.573+0000 7fa5aeffd640 1 -- 172.21.15.123:0/3899135478 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa5b015e160 msgr2=0x7fa5b01594a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:51:52.575 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:52.573+0000 7fa5aeffd640 1 --2- 172.21.15.123:0/3899135478 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa5b015e160 0x7fa5b01594a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:52.575 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:52.573+0000 7fa5aeffd640 1 -- 172.21.15.123:0/3899135478 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa5b01f65c0 con 0x7fa5b0074470 2024-09-16T15:51:52.575 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:52.573+0000 7fa5aeffd640 1 --2- 172.21.15.123:0/3899135478 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa5b0074470 0x7fa5b0154730 secure :-1 s=READY pgs=1887 cs=0 l=1 rev1=1 crypto rx=0x7fa5a000b550 tx=0x7fa5a000ba10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:51:52.575 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:52.573+0000 7fa5977fe640 1 -- 172.21.15.123:0/3899135478 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5a0014070 con 0x7fa5b0074470 2024-09-16T15:51:52.575 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:52.573+0000 7fa5977fe640 1 -- 172.21.15.123:0/3899135478 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fa5a0010980 con 0x7fa5b0074470 2024-09-16T15:51:52.575 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:52.573+0000 7fa5977fe640 1 -- 172.21.15.123:0/3899135478 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5a0010dd0 con 0x7fa5b0074470 2024-09-16T15:51:52.575 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:52.573+0000 7fa5b52de640 1 -- 172.21.15.123:0/3899135478 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa5b015a6d0 con 0x7fa5b0074470 2024-09-16T15:51:52.575 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:52.573+0000 7fa5b52de640 1 -- 172.21.15.123:0/3899135478 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa5b015ac90 con 0x7fa5b0074470 2024-09-16T15:51:52.576 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:52.577+0000 7fa5b52de640 1 -- 172.21.15.123:0/3899135478 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa5b00d06c0 con 0x7fa5b0074470 2024-09-16T15:51:52.576 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:52.577+0000 7fa5977fe640 1 -- 172.21.15.123:0/3899135478 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 25) ==== 97998+0+0 (secure 0 0 0) 0x7fa5a0017070 con 0x7fa5b0074470 2024-09-16T15:51:52.576 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:52.577+0000 7fa5977fe640 1 --2- 172.21.15.123:0/3899135478 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fa59c073910 0x7fa59c075dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:51:52.576 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:52.577+0000 7fa5977fe640 1 -- 172.21.15.123:0/3899135478 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(169..169 src has 1..169) ==== 5737+0+0 (secure 0 0 0) 0x7fa5a0092700 con 0x7fa5b0074470 2024-09-16T15:51:52.576 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:52.577+0000 7fa5ae7fc640 1 --2- 172.21.15.123:0/3899135478 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fa59c073910 0x7fa59c075dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:51:52.576 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:52.577+0000 7fa5ae7fc640 1 --2- 172.21.15.123:0/3899135478 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fa59c073910 0x7fa59c075dd0 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7fa5b015d960 tx=0x7fa5a4007a00 comp rx=0 tx=0).ready entity=mgr.4150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:51:52.580 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:52.581+0000 7fa5977fe640 1 -- 172.21.15.123:0/3899135478 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7fa5a005f1e0 con 0x7fa5b0074470 2024-09-16T15:51:52.734 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:52.733+0000 7fa5b52de640 1 -- 172.21.15.123:0/3899135478 --> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fa5b0073f10 con 0x7fa59c073910 2024-09-16T15:51:52.740 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:52.741+0000 7fa5977fe640 1 -- 172.21.15.123:0/3899135478 <== mgr.4150 v2:172.21.15.123:6836/2464759676 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+252478 (secure 0 0 0) 0x7fa5b0073f10 con 0x7fa59c073910 2024-09-16T15:51:52.742 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-09-16T15:51:52.744 INFO:teuthology.orchestra.run.smithi123.stderr:dumped all 2024-09-16T15:51:52.745 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:52.745+0000 7fa5b52de640 1 -- 172.21.15.123:0/3899135478 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fa59c073910 msgr2=0x7fa59c075dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:51:52.745 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:52.745+0000 7fa5b52de640 1 --2- 172.21.15.123:0/3899135478 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fa59c073910 0x7fa59c075dd0 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7fa5b015d960 tx=0x7fa5a4007a00 comp rx=0 tx=0).stop 2024-09-16T15:51:52.745 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:52.745+0000 7fa5b52de640 1 -- 172.21.15.123:0/3899135478 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa5b0074470 msgr2=0x7fa5b0154730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:51:52.745 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:52.745+0000 7fa5b52de640 1 --2- 172.21.15.123:0/3899135478 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa5b0074470 0x7fa5b0154730 secure :-1 s=READY pgs=1887 cs=0 l=1 rev1=1 crypto rx=0x7fa5a000b550 tx=0x7fa5a000ba10 comp rx=0 tx=0).stop 2024-09-16T15:51:52.745 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:52.745+0000 7fa5b52de640 1 -- 172.21.15.123:0/3899135478 shutdown_connections 2024-09-16T15:51:52.745 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:52.745+0000 7fa5b52de640 1 --2- 172.21.15.123:0/3899135478 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fa59c073910 0x7fa59c075dd0 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:52.745 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:52.745+0000 7fa5b52de640 1 --2- 172.21.15.123:0/3899135478 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fa5b015e820 0x7fa5b01f6080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:52.745 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:52.745+0000 7fa5b52de640 1 --2- 172.21.15.123:0/3899135478 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fa5b015e160 0x7fa5b01594a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:52.745 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:52.745+0000 7fa5b52de640 1 --2- 172.21.15.123:0/3899135478 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa5b0074470 0x7fa5b0154730 unknown :-1 s=CLOSED pgs=1887 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:52.745 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:52.745+0000 7fa5b52de640 1 -- 172.21.15.123:0/3899135478 >> 172.21.15.123:0/3899135478 conn(0x7fa5b00cfb10 msgr2=0x7fa5b0159b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:51:52.745 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:52.745+0000 7fa5b52de640 1 -- 172.21.15.123:0/3899135478 shutdown_connections 2024-09-16T15:51:52.745 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:52.745+0000 7fa5b52de640 1 -- 172.21.15.123:0/3899135478 wait complete. 2024-09-16T15:51:52.759 INFO:teuthology.orchestra.run.smithi123.stdout:{"pg_ready":true,"pg_map":{"version":229,"stamp":"2024-09-16T15:51:52.143817+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":158,"num_write_kb":1642,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":118,"ondisk_log_size":118,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":80,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":275640,"kb_used_data":17792,"kb_used_omap":12,"kb_used_meta":257651,"kb_avail":754699080,"statfs":{"total":773094113280,"available":772811857920,"internally_reserved":0,"allocated":18219008,"data_stored":8708556,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":263835185},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":35,"apply_latency_ms":35,"commit_latency_ns":35000000,"apply_latency_ns":35000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.796295"},"pg_stats":[{"pgid":"27.3f","version":"0'0","reported_seq":13,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-09-16T15:51:49.690715+0000","last_change":"2024-09-16T15:51:48.420332+0000","last_active":"2024-09-16T15:51:49.690715+0000","last_peered":"2024-09-16T15:51:49.690715+0000","last_clean":"2024-09-16T15:51:49.690715+0000","last_became_active":"2024-09-16T15:51:48.415997+0000","last_became_peered":"2024-09-16T15:51:48.415997+0000","last_unstale":"2024-09-16T15:51:49.690715+0000","last_undegraded":"2024-09-16T15:51:49.690715+0000","last_fullsized":"2024-09-16T15:51:49.690715+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":167,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:51:47.345679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:51:47.345679+0000","last_clean_scrub_stamp":"2024-09-16T15:51:47.345679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:16:57.181968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.3e","version":"0'0","reported_seq":10,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-09-16T15:51:48.441452+0000","last_change":"2024-09-16T15:51:48.428809+0000","last_active":"2024-09-16T15:51:48.441452+0000","last_peered":"2024-09-16T15:51:48.441452+0000","last_clean":"2024-09-16T15:51:48.441452+0000","last_became_active":"2024-09-16T15:51:48.428073+0000","last_became_peered":"2024-09-16T15:51:48.428073+0000","last_unstale":"2024-09-16T15:51:48.441452+0000","last_undegraded":"2024-09-16T15:51:48.441452+0000","last_fullsized":"2024-09-16T15:51:48.441452+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":167,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:51:47.345679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:51:47.345679+0000","last_clean_scrub_stamp":"2024-09-16T15:51:47.345679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:38:13.614017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.3d","version":"0'0","reported_seq":13,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-09-16T15:51:49.391883+0000","last_change":"2024-09-16T15:51:48.417037+0000","last_active":"2024-09-16T15:51:49.391883+0000","last_peered":"2024-09-16T15:51:49.391883+0000","last_clean":"2024-09-16T15:51:49.391883+0000","last_became_active":"2024-09-16T15:51:48.414206+0000","last_became_peered":"2024-09-16T15:51:48.414206+0000","last_unstale":"2024-09-16T15:51:49.391883+0000","last_undegraded":"2024-09-16T15:51:49.391883+0000","last_fullsized":"2024-09-16T15:51:49.391883+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":167,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:51:47.345679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:51:47.345679+0000","last_clean_scrub_stamp":"2024-09-16T15:51:47.345679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T23:40:00.612806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.3c","version":"0'0","reported_seq":13,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-09-16T15:51:49.435887+0000","last_change":"2024-09-16T15:51:48.401750+0000","last_active":"2024-09-16T15:51:49.435887+0000","last_peered":"2024-09-16T15:51:49.435887+0000","last_clean":"2024-09-16T15:51:49.435887+0000","last_became_active":"2024-09-16T15:51:48.400974+0000","last_became_peered":"2024-09-16T15:51:48.400974+0000","last_unstale":"2024-09-16T15:51:49.435887+0000","last_undegraded":"2024-09-16T15:51:49.435887+0000","last_fullsized":"2024-09-16T15:51:49.435887+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":167,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:51:47.345679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:51:47.345679+0000","last_clean_scrub_stamp":"2024-09-16T15:51:47.345679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:52:15.623713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.3b","version":"0'0","reported_seq":10,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-09-16T15:51:48.408008+0000","last_change":"2024-09-16T15:51:48.403256+0000","last_active":"2024-09-16T15:51:48.408008+0000","last_peered":"2024-09-16T15:51:48.408008+0000","last_clean":"2024-09-16T15:51:48.408008+0000","last_became_active":"2024-09-16T15:51:48.402536+0000","last_became_peered":"2024-09-16T15:51:48.402536+0000","last_unstale":"2024-09-16T15:51:48.408008+0000","last_undegraded":"2024-09-16T15:51:48.408008+0000","last_fullsized":"2024-09-16T15:51:48.408008+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":167,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:51:47.345679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:51:47.345679+0000","last_clean_scrub_stamp":"2024-09-16T15:51:47.345679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:27:20.068057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.3a","version":"0'0","reported_seq":10,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-09-16T15:51:48.430369+0000","last_change":"2024-09-16T15:51:48.425364+0000","last_active":"2024-09-16T15:51:48.430369+0000","last_peered":"2024-09-16T15:51:48.430369+0000","last_clean":"2024-09-16T15:51:48.430369+0000","last_became_active":"2024-09-16T15:51:48.424556+0000","last_became_peered":"2024-09-16T15:51:48.424556+0000","last_unstale":"2024-09-16T15:51:48.430369+0000","last_undegraded":"2024-09-16T15:51:48.430369+0000","last_fullsized":"2024-09-16T15:51:48.430369+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":167,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:51:47.345679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:51:47.345679+0000","last_clean_scrub_stamp":"2024-09-16T15:51:47.345679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T00:38:29.757986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.39","version":"0'0","reported_seq":13,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-09-16T15:51:49.689842+0000","last_change":"2024-09-16T15:51:48.421743+0000","last_active":"2024-09-16T15:51:49.689842+0000","last_peered":"2024-09-16T15:51:49.689842+0000","last_clean":"2024-09-16T15:51:49.689842+0000","last_became_active":"2024-09-16T15:51:48.421020+0000","last_became_peered":"2024-09-16T15:51:48.421020+0000","last_unstale":"2024-09-16T15:51:49.689842+0000","last_undegraded":"2024-09-16T15:51:49.689842+0000","last_fullsized":"2024-09-16T15:51:49.689842+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":167,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:51:47.345679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:51:47.345679+0000","last_clean_scrub_stamp":"2024-09-16T15:51:47.345679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:12:44.001487+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.38","version":"0'0","reported_seq":10,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-09-16T15:51:48.396098+0000","last_change":"2024-09-16T15:51:48.394064+0000","last_active":"2024-09-16T15:51:48.396098+0000","last_peered":"2024-09-16T15:51:48.396098+0000","last_clean":"2024-09-16T15:51:48.396098+0000","last_became_active":"2024-09-16T15:51:48.393323+0000","last_became_peered":"2024-09-16T15:51:48.393323+0000","last_unstale":"2024-09-16T15:51:48.396098+0000","last_undegraded":"2024-09-16T15:51:48.396098+0000","last_fullsized":"2024-09-16T15:51:48.396098+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":167,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:51:47.345679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:51:47.345679+0000","last_clean_scrub_stamp":"2024-09-16T15:51:47.345679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:17:20.368465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.37","version":"0'0","reported_seq":10,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-09-16T15:51:48.441496+0000","last_change":"2024-09-16T15:51:48.433407+0000","last_active":"2024-09-16T15:51:48.441496+0000","last_peered":"2024-09-16T15:51:48.441496+0000","last_clean":"2024-09-16T15:51:48.441496+0000","last_became_active":"2024-09-16T15:51:48.432739+0000","last_became_peered":"2024-09-16T15:51:48.432739+0000","last_unstale":"2024-09-16T15:51:48.441496+0000","last_undegraded":"2024-09-16T15:51:48.441496+0000","last_fullsized":"2024-09-16T15:51:48.441496+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":167,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:51:47.345679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:51:47.345679+0000","last_clean_scrub_stamp":"2024-09-16T15:51:47.345679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T01:51:19.144810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.36","version":"0'0","reported_seq":10,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-09-16T15:51:48.435726+0000","last_change":"2024-09-16T15:51:48.433866+0000","last_active":"2024-09-16T15:51:48.435726+0000","last_peered":"2024-09-16T15:51:48.435726+0000","last_clean":"2024-09-16T15:51:48.435726+0000","last_became_active":"2024-09-16T15:51:48.433462+0000","last_became_peered":"2024-09-16T15:51:48.433462+0000","last_unstale":"2024-09-16T15:51:48.435726+0000","last_undegraded":"2024-09-16T15:51:48.435726+0000","last_fullsized":"2024-09-16T15:51:48.435726+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":167,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:51:47.345679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:51:47.345679+0000","last_clean_scrub_stamp":"2024-09-16T15:51:47.345679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:19:24.536498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.35","version":"0'0","reported_seq":13,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-09-16T15:51:49.391937+0000","last_change":"2024-09-16T15:51:48.417966+0000","last_active":"2024-09-16T15:51:49.391937+0000","last_peered":"2024-09-16T15:51:49.391937+0000","last_clean":"2024-09-16T15:51:49.391937+0000","last_became_active":"2024-09-16T15:51:48.416270+0000","last_became_peered":"2024-09-16T15:51:48.416270+0000","last_unstale":"2024-09-16T15:51:49.391937+0000","last_undegraded":"2024-09-16T15:51:49.391937+0000","last_fullsized":"2024-09-16T15:51:49.391937+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":167,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:51:47.345679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:51:47.345679+0000","last_clean_scrub_stamp":"2024-09-16T15:51:47.345679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T02:21:37.777271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.34","version":"0'0","reported_seq":13,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-09-16T15:51:49.390622+0000","last_change":"2024-09-16T15:51:48.398213+0000","last_active":"2024-09-16T15:51:49.390622+0000","last_peered":"2024-09-16T15:51:49.390622+0000","last_clean":"2024-09-16T15:51:49.390622+0000","last_became_active":"2024-09-16T15:51:48.396833+0000","last_became_peered":"2024-09-16T15:51:48.396833+0000","last_unstale":"2024-09-16T15:51:49.390622+0000","last_undegraded":"2024-09-16T15:51:49.390622+0000","last_fullsized":"2024-09-16T15:51:49.390622+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":167,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:51:47.345679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:51:47.345679+0000","last_clean_scrub_stamp":"2024-09-16T15:51:47.345679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T18:19:46.535824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.33","version":"0'0","reported_seq":13,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-09-16T15:51:49.391060+0000","last_change":"2024-09-16T15:51:48.396718+0000","last_active":"2024-09-16T15:51:49.391060+0000","last_peered":"2024-09-16T15:51:49.391060+0000","last_clean":"2024-09-16T15:51:49.391060+0000","last_became_active":"2024-09-16T15:51:48.395829+0000","last_became_peered":"2024-09-16T15:51:48.395829+0000","last_unstale":"2024-09-16T15:51:49.391060+0000","last_undegraded":"2024-09-16T15:51:49.391060+0000","last_fullsized":"2024-09-16T15:51:49.391060+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":167,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:51:47.345679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:51:47.345679+0000","last_clean_scrub_stamp":"2024-09-16T15:51:47.345679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T18:06:32.715829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.32","version":"0'0","reported_seq":13,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-09-16T15:51:49.690024+0000","last_change":"2024-09-16T15:51:48.421882+0000","last_active":"2024-09-16T15:51:49.690024+0000","last_peered":"2024-09-16T15:51:49.690024+0000","last_clean":"2024-09-16T15:51:49.690024+0000","last_became_active":"2024-09-16T15:51:48.415002+0000","last_became_peered":"2024-09-16T15:51:48.415002+0000","last_unstale":"2024-09-16T15:51:49.690024+0000","last_undegraded":"2024-09-16T15:51:49.690024+0000","last_fullsized":"2024-09-16T15:51:49.690024+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":167,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:51:47.345679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:51:47.345679+0000","last_clean_scrub_stamp":"2024-09-16T15:51:47.345679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:25:21.266449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.31","version":"0'0","reported_seq":13,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-09-16T15:51:49.689861+0000","last_change":"2024-09-16T15:51:48.436670+0000","last_active":"2024-09-16T15:51:49.689861+0000","last_peered":"2024-09-16T15:51:49.689861+0000","last_clean":"2024-09-16T15:51:49.689861+0000","last_became_active":"2024-09-16T15:51:48.433234+0000","last_became_peered":"2024-09-16T15:51:48.433234+0000","last_unstale":"2024-09-16T15:51:49.689861+0000","last_undegraded":"2024-09-16T15:51:49.689861+0000","last_fullsized":"2024-09-16T15:51:49.689861+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":167,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:51:47.345679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:51:47.345679+0000","last_clean_scrub_stamp":"2024-09-16T15:51:47.345679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T01:03:40.372183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.30","version":"0'0","reported_seq":10,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-09-16T15:51:48.411029+0000","last_change":"2024-09-16T15:51:48.407337+0000","last_active":"2024-09-16T15:51:48.411029+0000","last_peered":"2024-09-16T15:51:48.411029+0000","last_clean":"2024-09-16T15:51:48.411029+0000","last_became_active":"2024-09-16T15:51:48.406491+0000","last_became_peered":"2024-09-16T15:51:48.406491+0000","last_unstale":"2024-09-16T15:51:48.411029+0000","last_undegraded":"2024-09-16T15:51:48.411029+0000","last_fullsized":"2024-09-16T15:51:48.411029+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":167,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:51:47.345679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:51:47.345679+0000","last_clean_scrub_stamp":"2024-09-16T15:51:47.345679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:05:38.011381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.2f","version":"0'0","reported_seq":13,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-09-16T15:51:49.690575+0000","last_change":"2024-09-16T15:51:48.434587+0000","last_active":"2024-09-16T15:51:49.690575+0000","last_peered":"2024-09-16T15:51:49.690575+0000","last_clean":"2024-09-16T15:51:49.690575+0000","last_became_active":"2024-09-16T15:51:48.432752+0000","last_became_peered":"2024-09-16T15:51:48.432752+0000","last_unstale":"2024-09-16T15:51:49.690575+0000","last_undegraded":"2024-09-16T15:51:49.690575+0000","last_fullsized":"2024-09-16T15:51:49.690575+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":167,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:51:47.345679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:51:47.345679+0000","last_clean_scrub_stamp":"2024-09-16T15:51:47.345679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:52:37.537156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.2e","version":"0'0","reported_seq":10,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-09-16T15:51:48.428648+0000","last_change":"2024-09-16T15:51:48.425963+0000","last_active":"2024-09-16T15:51:48.428648+0000","last_peered":"2024-09-16T15:51:48.428648+0000","last_clean":"2024-09-16T15:51:48.428648+0000","last_became_active":"2024-09-16T15:51:48.425609+0000","last_became_peered":"2024-09-16T15:51:48.425609+0000","last_unstale":"2024-09-16T15:51:48.428648+0000","last_undegraded":"2024-09-16T15:51:48.428648+0000","last_fullsized":"2024-09-16T15:51:48.428648+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":167,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:51:47.345679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:51:47.345679+0000","last_clean_scrub_stamp":"2024-09-16T15:51:47.345679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T00:04:45.914265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.2d","version":"0'0","reported_seq":10,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-09-16T15:51:48.434468+0000","last_change":"2024-09-16T15:51:48.433761+0000","last_active":"2024-09-16T15:51:48.434468+0000","last_peered":"2024-09-16T15:51:48.434468+0000","last_clean":"2024-09-16T15:51:48.434468+0000","last_became_active":"2024-09-16T15:51:48.433458+0000","last_became_peered":"2024-09-16T15:51:48.433458+0000","last_unstale":"2024-09-16T15:51:48.434468+0000","last_undegraded":"2024-09-16T15:51:48.434468+0000","last_fullsized":"2024-09-16T15:51:48.434468+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":167,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:51:47.345679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:51:47.345679+0000","last_clean_scrub_stamp":"2024-09-16T15:51:47.345679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T01:05:01.623302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.2c","version":"0'0","reported_seq":10,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-09-16T15:51:48.423630+0000","last_change":"2024-09-16T15:51:48.408745+0000","last_active":"2024-09-16T15:51:48.423630+0000","last_peered":"2024-09-16T15:51:48.423630+0000","last_clean":"2024-09-16T15:51:48.423630+0000","last_became_active":"2024-09-16T15:51:48.407685+0000","last_became_peered":"2024-09-16T15:51:48.407685+0000","last_unstale":"2024-09-16T15:51:48.423630+0000","last_undegraded":"2024-09-16T15:51:48.423630+0000","last_fullsized":"2024-09-16T15:51:48.423630+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":167,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:51:47.345679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:51:47.345679+0000","last_clean_scrub_stamp":"2024-09-16T15:51:47.345679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T00:29:29.367918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.2b","version":"0'0","reported_seq":13,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-09-16T15:51:49.434646+0000","last_change":"2024-09-16T15:51:48.406919+0000","last_active":"2024-09-16T15:51:49.434646+0000","last_peered":"2024-09-16T15:51:49.434646+0000","last_clean":"2024-09-16T15:51:49.434646+0000","last_became_active":"2024-09-16T15:51:48.406245+0000","last_became_peered":"2024-09-16T15:51:48.406245+0000","last_unstale":"2024-09-16T15:51:49.434646+0000","last_undegraded":"2024-09-16T15:51:49.434646+0000","last_fullsized":"2024-09-16T15:51:49.434646+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":167,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:51:47.345679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:51:47.345679+0000","last_clean_scrub_stamp":"2024-09-16T15:51:47.345679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T00:43:21.089679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.2a","version":"0'0","reported_seq":13,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-09-16T15:51:49.392755+0000","last_change":"2024-09-16T15:51:48.411279+0000","last_active":"2024-09-16T15:51:49.392755+0000","last_peered":"2024-09-16T15:51:49.392755+0000","last_clean":"2024-09-16T15:51:49.392755+0000","last_became_active":"2024-09-16T15:51:48.410456+0000","last_became_peered":"2024-09-16T15:51:48.410456+0000","last_unstale":"2024-09-16T15:51:49.392755+0000","last_undegraded":"2024-09-16T15:51:49.392755+0000","last_fullsized":"2024-09-16T15:51:49.392755+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":167,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:51:47.345679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:51:47.345679+0000","last_clean_scrub_stamp":"2024-09-16T15:51:47.345679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:23:57.550863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.29","version":"0'0","reported_seq":13,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-09-16T15:51:49.391086+0000","last_change":"2024-09-16T15:51:48.395273+0000","last_active":"2024-09-16T15:51:49.391086+0000","last_peered":"2024-09-16T15:51:49.391086+0000","last_clean":"2024-09-16T15:51:49.391086+0000","last_became_active":"2024-09-16T15:51:48.394570+0000","last_became_peered":"2024-09-16T15:51:48.394570+0000","last_unstale":"2024-09-16T15:51:49.391086+0000","last_undegraded":"2024-09-16T15:51:49.391086+0000","last_fullsized":"2024-09-16T15:51:49.391086+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":167,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:51:47.345679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:51:47.345679+0000","last_clean_scrub_stamp":"2024-09-16T15:51:47.345679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:39:26.940952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.28","version":"0'0","reported_seq":13,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-09-16T15:51:49.435076+0000","last_change":"2024-09-16T15:51:48.425121+0000","last_active":"2024-09-16T15:51:49.435076+0000","last_peered":"2024-09-16T15:51:49.435076+0000","last_clean":"2024-09-16T15:51:49.435076+0000","last_became_active":"2024-09-16T15:51:48.424367+0000","last_became_peered":"2024-09-16T15:51:48.424367+0000","last_unstale":"2024-09-16T15:51:49.435076+0000","last_undegraded":"2024-09-16T15:51:49.435076+0000","last_fullsized":"2024-09-16T15:51:49.435076+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":167,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:51:47.345679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:51:47.345679+0000","last_clean_scrub_stamp":"2024-09-16T15:51:47.345679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T02:33:57.816891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.27","version":"0'0","reported_seq":10,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-09-16T15:51:48.441257+0000","last_change":"2024-09-16T15:51:48.430257+0000","last_active":"2024-09-16T15:51:48.441257+0000","last_peered":"2024-09-16T15:51:48.441257+0000","last_clean":"2024-09-16T15:51:48.441257+0000","last_became_active":"2024-09-16T15:51:48.429592+0000","last_became_peered":"2024-09-16T15:51:48.429592+0000","last_unstale":"2024-09-16T15:51:48.441257+0000","last_undegraded":"2024-09-16T15:51:48.441257+0000","last_fullsized":"2024-09-16T15:51:48.441257+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":167,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:51:47.345679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:51:47.345679+0000","last_clean_scrub_stamp":"2024-09-16T15:51:47.345679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:27:35.569594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.26","version":"0'0","reported_seq":13,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-09-16T15:51:49.691129+0000","last_change":"2024-09-16T15:51:48.431641+0000","last_active":"2024-09-16T15:51:49.691129+0000","last_peered":"2024-09-16T15:51:49.691129+0000","last_clean":"2024-09-16T15:51:49.691129+0000","last_became_active":"2024-09-16T15:51:48.423156+0000","last_became_peered":"2024-09-16T15:51:48.423156+0000","last_unstale":"2024-09-16T15:51:49.691129+0000","last_undegraded":"2024-09-16T15:51:49.691129+0000","last_fullsized":"2024-09-16T15:51:49.691129+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":167,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:51:47.345679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:51:47.345679+0000","last_clean_scrub_stamp":"2024-09-16T15:51:47.345679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:52:06.593219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.25","version":"0'0","reported_seq":13,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-09-16T15:51:49.392092+0000","last_change":"2024-09-16T15:51:48.397722+0000","last_active":"2024-09-16T15:51:49.392092+0000","last_peered":"2024-09-16T15:51:49.392092+0000","last_clean":"2024-09-16T15:51:49.392092+0000","last_became_active":"2024-09-16T15:51:48.397005+0000","last_became_peered":"2024-09-16T15:51:48.397005+0000","last_unstale":"2024-09-16T15:51:49.392092+0000","last_undegraded":"2024-09-16T15:51:49.392092+0000","last_fullsized":"2024-09-16T15:51:49.392092+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":167,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:51:47.345679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:51:47.345679+0000","last_clean_scrub_stamp":"2024-09-16T15:51:47.345679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T23:19:13.505005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.24","version":"0'0","reported_seq":13,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-09-16T15:51:49.391454+0000","last_change":"2024-09-16T15:51:48.407290+0000","last_active":"2024-09-16T15:51:49.391454+0000","last_peered":"2024-09-16T15:51:49.391454+0000","last_clean":"2024-09-16T15:51:49.391454+0000","last_became_active":"2024-09-16T15:51:48.406572+0000","last_became_peered":"2024-09-16T15:51:48.406572+0000","last_unstale":"2024-09-16T15:51:49.391454+0000","last_undegraded":"2024-09-16T15:51:49.391454+0000","last_fullsized":"2024-09-16T15:51:49.391454+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":167,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:51:47.345679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:51:47.345679+0000","last_clean_scrub_stamp":"2024-09-16T15:51:47.345679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T03:28:07.952532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.23","version":"0'0","reported_seq":13,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-09-16T15:51:49.389652+0000","last_change":"2024-09-16T15:51:48.398061+0000","last_active":"2024-09-16T15:51:49.389652+0000","last_peered":"2024-09-16T15:51:49.389652+0000","last_clean":"2024-09-16T15:51:49.389652+0000","last_became_active":"2024-09-16T15:51:48.397267+0000","last_became_peered":"2024-09-16T15:51:48.397267+0000","last_unstale":"2024-09-16T15:51:49.389652+0000","last_undegraded":"2024-09-16T15:51:49.389652+0000","last_fullsized":"2024-09-16T15:51:49.389652+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":167,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:51:47.345679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:51:47.345679+0000","last_clean_scrub_stamp":"2024-09-16T15:51:47.345679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T03:05:58.168737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.22","version":"0'0","reported_seq":10,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-09-16T15:51:48.443275+0000","last_change":"2024-09-16T15:51:48.430281+0000","last_active":"2024-09-16T15:51:48.443275+0000","last_peered":"2024-09-16T15:51:48.443275+0000","last_clean":"2024-09-16T15:51:48.443275+0000","last_became_active":"2024-09-16T15:51:48.429690+0000","last_became_peered":"2024-09-16T15:51:48.429690+0000","last_unstale":"2024-09-16T15:51:48.443275+0000","last_undegraded":"2024-09-16T15:51:48.443275+0000","last_fullsized":"2024-09-16T15:51:48.443275+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":167,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:51:47.345679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:51:47.345679+0000","last_clean_scrub_stamp":"2024-09-16T15:51:47.345679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T23:51:48.437995+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.21","version":"0'0","reported_seq":10,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-09-16T15:51:48.420576+0000","last_change":"2024-09-16T15:51:48.413862+0000","last_active":"2024-09-16T15:51:48.420576+0000","last_peered":"2024-09-16T15:51:48.420576+0000","last_clean":"2024-09-16T15:51:48.420576+0000","last_became_active":"2024-09-16T15:51:48.411907+0000","last_became_peered":"2024-09-16T15:51:48.411907+0000","last_unstale":"2024-09-16T15:51:48.420576+0000","last_undegraded":"2024-09-16T15:51:48.420576+0000","last_fullsized":"2024-09-16T15:51:48.420576+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":167,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:51:47.345679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:51:47.345679+0000","last_clean_scrub_stamp":"2024-09-16T15:51:47.345679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:26:56.667722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.20","version":"0'0","reported_seq":13,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-09-16T15:51:49.689778+0000","last_change":"2024-09-16T15:51:48.415866+0000","last_active":"2024-09-16T15:51:49.689778+0000","last_peered":"2024-09-16T15:51:49.689778+0000","last_clean":"2024-09-16T15:51:49.689778+0000","last_became_active":"2024-09-16T15:51:48.415128+0000","last_became_peered":"2024-09-16T15:51:48.415128+0000","last_unstale":"2024-09-16T15:51:49.689778+0000","last_undegraded":"2024-09-16T15:51:49.689778+0000","last_fullsized":"2024-09-16T15:51:49.689778+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":167,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:51:47.345679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:51:47.345679+0000","last_clean_scrub_stamp":"2024-09-16T15:51:47.345679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:52:15.623713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.1e","version":"0'0","reported_seq":10,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-09-16T15:51:48.441518+0000","last_change":"2024-09-16T15:51:48.428746+0000","last_active":"2024-09-16T15:51:48.441518+0000","last_peered":"2024-09-16T15:51:48.441518+0000","last_clean":"2024-09-16T15:51:48.441518+0000","last_became_active":"2024-09-16T15:51:48.427152+0000","last_became_peered":"2024-09-16T15:51:48.427152+0000","last_unstale":"2024-09-16T15:51:48.441518+0000","last_undegraded":"2024-09-16T15:51:48.441518+0000","last_fullsized":"2024-09-16T15:51:48.441518+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":167,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:51:47.345679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:51:47.345679+0000","last_clean_scrub_stamp":"2024-09-16T15:51:47.345679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:06:11.869682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.1f","version":"168'4","reported_seq":19,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-09-16T15:51:49.392490+0000","last_change":"2024-09-16T15:51:47.389991+0000","last_active":"2024-09-16T15:51:49.392490+0000","last_peered":"2024-09-16T15:51:49.392490+0000","last_clean":"2024-09-16T15:51:49.392490+0000","last_became_active":"2024-09-16T15:51:47.389258+0000","last_became_peered":"2024-09-16T15:51:47.389258+0000","last_unstale":"2024-09-16T15:51:49.392490+0000","last_undegraded":"2024-09-16T15:51:49.392490+0000","last_fullsized":"2024-09-16T15:51:49.392490+0000","mapping_epoch":166,"log_start":"0'0","ondisk_log_start":"0'0","created":166,"last_epoch_clean":167,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:51:46.335209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:51:46.335209+0000","last_clean_scrub_stamp":"2024-09-16T15:51:46.335209+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:28:42.487689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.1f","version":"0'0","reported_seq":13,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-09-16T15:51:49.393513+0000","last_change":"2024-09-16T15:51:48.388882+0000","last_active":"2024-09-16T15:51:49.393513+0000","last_peered":"2024-09-16T15:51:49.393513+0000","last_clean":"2024-09-16T15:51:49.393513+0000","last_became_active":"2024-09-16T15:51:48.388014+0000","last_became_peered":"2024-09-16T15:51:48.388014+0000","last_unstale":"2024-09-16T15:51:49.393513+0000","last_undegraded":"2024-09-16T15:51:49.393513+0000","last_fullsized":"2024-09-16T15:51:49.393513+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":167,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:51:47.345679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:51:47.345679+0000","last_clean_scrub_stamp":"2024-09-16T15:51:47.345679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:01:50.109145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.1e","version":"168'1","reported_seq":14,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-09-16T15:51:48.449559+0000","last_change":"2024-09-16T15:51:47.390954+0000","last_active":"2024-09-16T15:51:48.449559+0000","last_peered":"2024-09-16T15:51:48.449559+0000","last_clean":"2024-09-16T15:51:48.449559+0000","last_became_active":"2024-09-16T15:51:47.388775+0000","last_became_peered":"2024-09-16T15:51:47.388775+0000","last_unstale":"2024-09-16T15:51:48.449559+0000","last_undegraded":"2024-09-16T15:51:48.449559+0000","last_fullsized":"2024-09-16T15:51:48.449559+0000","mapping_epoch":166,"log_start":"0'0","ondisk_log_start":"0'0","created":166,"last_epoch_clean":167,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:51:46.335209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:51:46.335209+0000","last_clean_scrub_stamp":"2024-09-16T15:51:46.335209+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T00:19:10.515395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.1c","version":"0'0","reported_seq":13,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-09-16T15:51:49.690628+0000","last_change":"2024-09-16T15:51:48.436763+0000","last_active":"2024-09-16T15:51:49.690628+0000","last_peered":"2024-09-16T15:51:49.690628+0000","last_clean":"2024-09-16T15:51:49.690628+0000","last_became_active":"2024-09-16T15:51:48.434929+0000","last_became_peered":"2024-09-16T15:51:48.434929+0000","last_unstale":"2024-09-16T15:51:49.690628+0000","last_undegraded":"2024-09-16T15:51:49.690628+0000","last_fullsized":"2024-09-16T15:51:49.690628+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":167,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:51:47.345679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:51:47.345679+0000","last_clean_scrub_stamp":"2024-09-16T15:51:47.345679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T03:22:51.808663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.1d","version":"168'2","reported_seq":17,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-09-16T15:51:49.392185+0000","last_change":"2024-09-16T15:51:47.386433+0000","last_active":"2024-09-16T15:51:49.392185+0000","last_peered":"2024-09-16T15:51:49.392185+0000","last_clean":"2024-09-16T15:51:49.392185+0000","last_became_active":"2024-09-16T15:51:47.386107+0000","last_became_peered":"2024-09-16T15:51:47.386107+0000","last_unstale":"2024-09-16T15:51:49.392185+0000","last_undegraded":"2024-09-16T15:51:49.392185+0000","last_fullsized":"2024-09-16T15:51:49.392185+0000","mapping_epoch":166,"log_start":"0'0","ondisk_log_start":"0'0","created":166,"last_epoch_clean":167,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:51:46.335209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:51:46.335209+0000","last_clean_scrub_stamp":"2024-09-16T15:51:46.335209+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T18:02:05.541664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.1d","version":"0'0","reported_seq":10,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-09-16T15:51:48.429486+0000","last_change":"2024-09-16T15:51:48.427496+0000","last_active":"2024-09-16T15:51:48.429486+0000","last_peered":"2024-09-16T15:51:48.429486+0000","last_clean":"2024-09-16T15:51:48.429486+0000","last_became_active":"2024-09-16T15:51:48.427163+0000","last_became_peered":"2024-09-16T15:51:48.427163+0000","last_unstale":"2024-09-16T15:51:48.429486+0000","last_undegraded":"2024-09-16T15:51:48.429486+0000","last_fullsized":"2024-09-16T15:51:48.429486+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":167,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:51:47.345679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:51:47.345679+0000","last_clean_scrub_stamp":"2024-09-16T15:51:47.345679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T00:42:00.673157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.1c","version":"168'2","reported_seq":17,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-09-16T15:51:49.436172+0000","last_change":"2024-09-16T15:51:47.385737+0000","last_active":"2024-09-16T15:51:49.436172+0000","last_peered":"2024-09-16T15:51:49.436172+0000","last_clean":"2024-09-16T15:51:49.436172+0000","last_became_active":"2024-09-16T15:51:47.384892+0000","last_became_peered":"2024-09-16T15:51:47.384892+0000","last_unstale":"2024-09-16T15:51:49.436172+0000","last_undegraded":"2024-09-16T15:51:49.436172+0000","last_fullsized":"2024-09-16T15:51:49.436172+0000","mapping_epoch":166,"log_start":"0'0","ondisk_log_start":"0'0","created":166,"last_epoch_clean":167,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:51:46.335209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:51:46.335209+0000","last_clean_scrub_stamp":"2024-09-16T15:51:46.335209+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T02:00:54.678527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.1b","version":"0'0","reported_seq":13,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-09-16T15:51:49.434938+0000","last_change":"2024-09-16T15:51:48.401600+0000","last_active":"2024-09-16T15:51:49.434938+0000","last_peered":"2024-09-16T15:51:49.434938+0000","last_clean":"2024-09-16T15:51:49.434938+0000","last_became_active":"2024-09-16T15:51:48.400855+0000","last_became_peered":"2024-09-16T15:51:48.400855+0000","last_unstale":"2024-09-16T15:51:49.434938+0000","last_undegraded":"2024-09-16T15:51:49.434938+0000","last_fullsized":"2024-09-16T15:51:49.434938+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":167,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:51:47.345679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:51:47.345679+0000","last_clean_scrub_stamp":"2024-09-16T15:51:47.345679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T18:33:00.669417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.1a","version":"168'1","reported_seq":14,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-09-16T15:51:48.445926+0000","last_change":"2024-09-16T15:51:47.379351+0000","last_active":"2024-09-16T15:51:48.445926+0000","last_peered":"2024-09-16T15:51:48.445926+0000","last_clean":"2024-09-16T15:51:48.445926+0000","last_became_active":"2024-09-16T15:51:47.378615+0000","last_became_peered":"2024-09-16T15:51:47.378615+0000","last_unstale":"2024-09-16T15:51:48.445926+0000","last_undegraded":"2024-09-16T15:51:48.445926+0000","last_fullsized":"2024-09-16T15:51:48.445926+0000","mapping_epoch":166,"log_start":"0'0","ondisk_log_start":"0'0","created":166,"last_epoch_clean":167,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:51:46.335209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:51:46.335209+0000","last_clean_scrub_stamp":"2024-09-16T15:51:46.335209+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T22:30:33.805316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.18","version":"0'0","reported_seq":10,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-09-16T15:51:48.420788+0000","last_change":"2024-09-16T15:51:48.409378+0000","last_active":"2024-09-16T15:51:48.420788+0000","last_peered":"2024-09-16T15:51:48.420788+0000","last_clean":"2024-09-16T15:51:48.420788+0000","last_became_active":"2024-09-16T15:51:48.408591+0000","last_became_peered":"2024-09-16T15:51:48.408591+0000","last_unstale":"2024-09-16T15:51:48.420788+0000","last_undegraded":"2024-09-16T15:51:48.420788+0000","last_fullsized":"2024-09-16T15:51:48.420788+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":167,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:51:47.345679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:51:47.345679+0000","last_clean_scrub_stamp":"2024-09-16T15:51:47.345679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:50:38.970224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.19","version":"0'0","reported_seq":13,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-09-16T15:51:48.380087+0000","last_change":"2024-09-16T15:51:47.378743+0000","last_active":"2024-09-16T15:51:48.380087+0000","last_peered":"2024-09-16T15:51:48.380087+0000","last_clean":"2024-09-16T15:51:48.380087+0000","last_became_active":"2024-09-16T15:51:47.378347+0000","last_became_peered":"2024-09-16T15:51:47.378347+0000","last_unstale":"2024-09-16T15:51:48.380087+0000","last_undegraded":"2024-09-16T15:51:48.380087+0000","last_fullsized":"2024-09-16T15:51:48.380087+0000","mapping_epoch":166,"log_start":"0'0","ondisk_log_start":"0'0","created":166,"last_epoch_clean":167,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:51:46.335209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:51:46.335209+0000","last_clean_scrub_stamp":"2024-09-16T15:51:46.335209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T22:53:43.066660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.19","version":"0'0","reported_seq":13,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-09-16T15:51:49.434797+0000","last_change":"2024-09-16T15:51:48.401062+0000","last_active":"2024-09-16T15:51:49.434797+0000","last_peered":"2024-09-16T15:51:49.434797+0000","last_clean":"2024-09-16T15:51:49.434797+0000","last_became_active":"2024-09-16T15:51:48.400353+0000","last_became_peered":"2024-09-16T15:51:48.400353+0000","last_unstale":"2024-09-16T15:51:49.434797+0000","last_undegraded":"2024-09-16T15:51:49.434797+0000","last_fullsized":"2024-09-16T15:51:49.434797+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":167,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:51:47.345679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:51:47.345679+0000","last_clean_scrub_stamp":"2024-09-16T15:51:47.345679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T00:16:19.141908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.18","version":"0'0","reported_seq":15,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-09-16T15:51:49.390219+0000","last_change":"2024-09-16T15:51:47.373792+0000","last_active":"2024-09-16T15:51:49.390219+0000","last_peered":"2024-09-16T15:51:49.390219+0000","last_clean":"2024-09-16T15:51:49.390219+0000","last_became_active":"2024-09-16T15:51:47.373004+0000","last_became_peered":"2024-09-16T15:51:47.373004+0000","last_unstale":"2024-09-16T15:51:49.390219+0000","last_undegraded":"2024-09-16T15:51:49.390219+0000","last_fullsized":"2024-09-16T15:51:49.390219+0000","mapping_epoch":166,"log_start":"0'0","ondisk_log_start":"0'0","created":166,"last_epoch_clean":167,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:51:46.335209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:51:46.335209+0000","last_clean_scrub_stamp":"2024-09-16T15:51:46.335209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T18:06:31.705358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.16","version":"0'0","reported_seq":13,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-09-16T15:51:49.435180+0000","last_change":"2024-09-16T15:51:48.424457+0000","last_active":"2024-09-16T15:51:49.435180+0000","last_peered":"2024-09-16T15:51:49.435180+0000","last_clean":"2024-09-16T15:51:49.435180+0000","last_became_active":"2024-09-16T15:51:48.423713+0000","last_became_peered":"2024-09-16T15:51:48.423713+0000","last_unstale":"2024-09-16T15:51:49.435180+0000","last_undegraded":"2024-09-16T15:51:49.435180+0000","last_fullsized":"2024-09-16T15:51:49.435180+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":167,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:51:47.345679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:51:47.345679+0000","last_clean_scrub_stamp":"2024-09-16T15:51:47.345679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T03:09:24.867616+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.17","version":"0'0","reported_seq":13,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-09-16T15:51:48.380248+0000","last_change":"2024-09-16T15:51:47.393217+0000","last_active":"2024-09-16T15:51:48.380248+0000","last_peered":"2024-09-16T15:51:48.380248+0000","last_clean":"2024-09-16T15:51:48.380248+0000","last_became_active":"2024-09-16T15:51:47.392883+0000","last_became_peered":"2024-09-16T15:51:47.392883+0000","last_unstale":"2024-09-16T15:51:48.380248+0000","last_undegraded":"2024-09-16T15:51:48.380248+0000","last_fullsized":"2024-09-16T15:51:48.380248+0000","mapping_epoch":166,"log_start":"0'0","ondisk_log_start":"0'0","created":166,"last_epoch_clean":167,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:51:46.335209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:51:46.335209+0000","last_clean_scrub_stamp":"2024-09-16T15:51:46.335209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:27:58.632284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.17","version":"0'0","reported_seq":13,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-09-16T15:51:49.392511+0000","last_change":"2024-09-16T15:51:48.405334+0000","last_active":"2024-09-16T15:51:49.392511+0000","last_peered":"2024-09-16T15:51:49.392511+0000","last_clean":"2024-09-16T15:51:49.392511+0000","last_became_active":"2024-09-16T15:51:48.401719+0000","last_became_peered":"2024-09-16T15:51:48.401719+0000","last_unstale":"2024-09-16T15:51:49.392511+0000","last_undegraded":"2024-09-16T15:51:49.392511+0000","last_fullsized":"2024-09-16T15:51:49.392511+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":167,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:51:47.345679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:51:47.345679+0000","last_clean_scrub_stamp":"2024-09-16T15:51:47.345679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:31:48.586582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.16","version":"0'0","reported_seq":15,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-09-16T15:51:49.393363+0000","last_change":"2024-09-16T15:51:47.386177+0000","last_active":"2024-09-16T15:51:49.393363+0000","last_peered":"2024-09-16T15:51:49.393363+0000","last_clean":"2024-09-16T15:51:49.393363+0000","last_became_active":"2024-09-16T15:51:47.385463+0000","last_became_peered":"2024-09-16T15:51:47.385463+0000","last_unstale":"2024-09-16T15:51:49.393363+0000","last_undegraded":"2024-09-16T15:51:49.393363+0000","last_fullsized":"2024-09-16T15:51:49.393363+0000","mapping_epoch":166,"log_start":"0'0","ondisk_log_start":"0'0","created":166,"last_epoch_clean":167,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:51:46.335209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:51:46.335209+0000","last_clean_scrub_stamp":"2024-09-16T15:51:46.335209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T23:19:12.494535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.14","version":"0'0","reported_seq":13,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-09-16T15:51:49.436244+0000","last_change":"2024-09-16T15:51:48.408427+0000","last_active":"2024-09-16T15:51:49.436244+0000","last_peered":"2024-09-16T15:51:49.436244+0000","last_clean":"2024-09-16T15:51:49.436244+0000","last_became_active":"2024-09-16T15:51:48.407616+0000","last_became_peered":"2024-09-16T15:51:48.407616+0000","last_unstale":"2024-09-16T15:51:49.436244+0000","last_undegraded":"2024-09-16T15:51:49.436244+0000","last_fullsized":"2024-09-16T15:51:49.436244+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":167,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:51:47.345679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:51:47.345679+0000","last_clean_scrub_stamp":"2024-09-16T15:51:47.345679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:12:44.001487+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.15","version":"168'3","reported_seq":16,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-09-16T15:51:48.446011+0000","last_change":"2024-09-16T15:51:47.376322+0000","last_active":"2024-09-16T15:51:48.446011+0000","last_peered":"2024-09-16T15:51:48.446011+0000","last_clean":"2024-09-16T15:51:48.446011+0000","last_became_active":"2024-09-16T15:51:47.375919+0000","last_became_peered":"2024-09-16T15:51:47.375919+0000","last_unstale":"2024-09-16T15:51:48.446011+0000","last_undegraded":"2024-09-16T15:51:48.446011+0000","last_fullsized":"2024-09-16T15:51:48.446011+0000","mapping_epoch":166,"log_start":"0'0","ondisk_log_start":"0'0","created":166,"last_epoch_clean":167,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:51:46.335209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:51:46.335209+0000","last_clean_scrub_stamp":"2024-09-16T15:51:46.335209+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T23:40:12.607073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.15","version":"0'0","reported_seq":13,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-09-16T15:51:49.690977+0000","last_change":"2024-09-16T15:51:48.434547+0000","last_active":"2024-09-16T15:51:49.690977+0000","last_peered":"2024-09-16T15:51:49.690977+0000","last_clean":"2024-09-16T15:51:49.690977+0000","last_became_active":"2024-09-16T15:51:48.432699+0000","last_became_peered":"2024-09-16T15:51:48.432699+0000","last_unstale":"2024-09-16T15:51:49.690977+0000","last_undegraded":"2024-09-16T15:51:49.690977+0000","last_fullsized":"2024-09-16T15:51:49.690977+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":167,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:51:47.345679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:51:47.345679+0000","last_clean_scrub_stamp":"2024-09-16T15:51:47.345679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T02:30:58.027562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.14","version":"168'6","reported_seq":21,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-09-16T15:51:49.436437+0000","last_change":"2024-09-16T15:51:47.391363+0000","last_active":"2024-09-16T15:51:49.436437+0000","last_peered":"2024-09-16T15:51:49.436437+0000","last_clean":"2024-09-16T15:51:49.436437+0000","last_became_active":"2024-09-16T15:51:47.390582+0000","last_became_peered":"2024-09-16T15:51:47.390582+0000","last_unstale":"2024-09-16T15:51:49.436437+0000","last_undegraded":"2024-09-16T15:51:49.436437+0000","last_fullsized":"2024-09-16T15:51:49.436437+0000","mapping_epoch":166,"log_start":"0'0","ondisk_log_start":"0'0","created":166,"last_epoch_clean":167,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:51:46.335209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:51:46.335209+0000","last_clean_scrub_stamp":"2024-09-16T15:51:46.335209+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:03:11.083680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.12","version":"0'0","reported_seq":10,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-09-16T15:51:48.422820+0000","last_change":"2024-09-16T15:51:48.410219+0000","last_active":"2024-09-16T15:51:48.422820+0000","last_peered":"2024-09-16T15:51:48.422820+0000","last_clean":"2024-09-16T15:51:48.422820+0000","last_became_active":"2024-09-16T15:51:48.409024+0000","last_became_peered":"2024-09-16T15:51:48.409024+0000","last_unstale":"2024-09-16T15:51:48.422820+0000","last_undegraded":"2024-09-16T15:51:48.422820+0000","last_fullsized":"2024-09-16T15:51:48.422820+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":167,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:51:47.345679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:51:47.345679+0000","last_clean_scrub_stamp":"2024-09-16T15:51:47.345679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T22:56:45.418432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.13","version":"168'2","reported_seq":17,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-09-16T15:51:49.392594+0000","last_change":"2024-09-16T15:51:47.368253+0000","last_active":"2024-09-16T15:51:49.392594+0000","last_peered":"2024-09-16T15:51:49.392594+0000","last_clean":"2024-09-16T15:51:49.392594+0000","last_became_active":"2024-09-16T15:51:47.366051+0000","last_became_peered":"2024-09-16T15:51:47.366051+0000","last_unstale":"2024-09-16T15:51:49.392594+0000","last_undegraded":"2024-09-16T15:51:49.392594+0000","last_fullsized":"2024-09-16T15:51:49.392594+0000","mapping_epoch":166,"log_start":"0'0","ondisk_log_start":"0'0","created":166,"last_epoch_clean":167,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:51:46.335209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:51:46.335209+0000","last_clean_scrub_stamp":"2024-09-16T15:51:46.335209+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T23:08:10.181813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.13","version":"0'0","reported_seq":10,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-09-16T15:51:48.428591+0000","last_change":"2024-09-16T15:51:48.425678+0000","last_active":"2024-09-16T15:51:48.428591+0000","last_peered":"2024-09-16T15:51:48.428591+0000","last_clean":"2024-09-16T15:51:48.428591+0000","last_became_active":"2024-09-16T15:51:48.425146+0000","last_became_peered":"2024-09-16T15:51:48.425146+0000","last_unstale":"2024-09-16T15:51:48.428591+0000","last_undegraded":"2024-09-16T15:51:48.428591+0000","last_fullsized":"2024-09-16T15:51:48.428591+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":167,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:51:47.345679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:51:47.345679+0000","last_clean_scrub_stamp":"2024-09-16T15:51:47.345679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T02:02:17.658639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.12","version":"0'0","reported_seq":15,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-09-16T15:51:49.392563+0000","last_change":"2024-09-16T15:51:47.376032+0000","last_active":"2024-09-16T15:51:49.392563+0000","last_peered":"2024-09-16T15:51:49.392563+0000","last_clean":"2024-09-16T15:51:49.392563+0000","last_became_active":"2024-09-16T15:51:47.375253+0000","last_became_peered":"2024-09-16T15:51:47.375253+0000","last_unstale":"2024-09-16T15:51:49.392563+0000","last_undegraded":"2024-09-16T15:51:49.392563+0000","last_fullsized":"2024-09-16T15:51:49.392563+0000","mapping_epoch":166,"log_start":"0'0","ondisk_log_start":"0'0","created":166,"last_epoch_clean":167,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:51:46.335209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:51:46.335209+0000","last_clean_scrub_stamp":"2024-09-16T15:51:46.335209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:26:09.762385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.10","version":"0'0","reported_seq":13,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-09-16T15:51:49.391262+0000","last_change":"2024-09-16T15:51:48.405340+0000","last_active":"2024-09-16T15:51:49.391262+0000","last_peered":"2024-09-16T15:51:49.391262+0000","last_clean":"2024-09-16T15:51:49.391262+0000","last_became_active":"2024-09-16T15:51:48.404594+0000","last_became_peered":"2024-09-16T15:51:48.404594+0000","last_unstale":"2024-09-16T15:51:49.391262+0000","last_undegraded":"2024-09-16T15:51:49.391262+0000","last_fullsized":"2024-09-16T15:51:49.391262+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":167,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:51:47.345679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:51:47.345679+0000","last_clean_scrub_stamp":"2024-09-16T15:51:47.345679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T02:48:42.809959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.11","version":"0'0","reported_seq":13,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-09-16T15:51:48.401862+0000","last_change":"2024-09-16T15:51:47.372021+0000","last_active":"2024-09-16T15:51:48.401862+0000","last_peered":"2024-09-16T15:51:48.401862+0000","last_clean":"2024-09-16T15:51:48.401862+0000","last_became_active":"2024-09-16T15:51:47.371683+0000","last_became_peered":"2024-09-16T15:51:47.371683+0000","last_unstale":"2024-09-16T15:51:48.401862+0000","last_undegraded":"2024-09-16T15:51:48.401862+0000","last_fullsized":"2024-09-16T15:51:48.401862+0000","mapping_epoch":166,"log_start":"0'0","ondisk_log_start":"0'0","created":166,"last_epoch_clean":167,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:51:46.335209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:51:46.335209+0000","last_clean_scrub_stamp":"2024-09-16T15:51:46.335209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:37:21.424729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.11","version":"0'0","reported_seq":13,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-09-16T15:51:49.690210+0000","last_change":"2024-09-16T15:51:48.434577+0000","last_active":"2024-09-16T15:51:49.690210+0000","last_peered":"2024-09-16T15:51:49.690210+0000","last_clean":"2024-09-16T15:51:49.690210+0000","last_became_active":"2024-09-16T15:51:48.432783+0000","last_became_peered":"2024-09-16T15:51:48.432783+0000","last_unstale":"2024-09-16T15:51:49.690210+0000","last_undegraded":"2024-09-16T15:51:49.690210+0000","last_fullsized":"2024-09-16T15:51:49.690210+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":167,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:51:47.345679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:51:47.345679+0000","last_clean_scrub_stamp":"2024-09-16T15:51:47.345679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:43:07.268710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.10","version":"168'8","reported_seq":23,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-09-16T15:51:49.391481+0000","last_change":"2024-09-16T15:51:47.385794+0000","last_active":"2024-09-16T15:51:49.391481+0000","last_peered":"2024-09-16T15:51:49.391481+0000","last_clean":"2024-09-16T15:51:49.391481+0000","last_became_active":"2024-09-16T15:51:47.385509+0000","last_became_peered":"2024-09-16T15:51:47.385509+0000","last_unstale":"2024-09-16T15:51:49.391481+0000","last_undegraded":"2024-09-16T15:51:49.391481+0000","last_fullsized":"2024-09-16T15:51:49.391481+0000","mapping_epoch":166,"log_start":"0'0","ondisk_log_start":"0'0","created":166,"last_epoch_clean":167,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:51:46.335209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:51:46.335209+0000","last_clean_scrub_stamp":"2024-09-16T15:51:46.335209+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T22:56:37.405888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.e","version":"0'0","reported_seq":10,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-09-16T15:51:48.442381+0000","last_change":"2024-09-16T15:51:48.433013+0000","last_active":"2024-09-16T15:51:48.442381+0000","last_peered":"2024-09-16T15:51:48.442381+0000","last_clean":"2024-09-16T15:51:48.442381+0000","last_became_active":"2024-09-16T15:51:48.432316+0000","last_became_peered":"2024-09-16T15:51:48.432316+0000","last_unstale":"2024-09-16T15:51:48.442381+0000","last_undegraded":"2024-09-16T15:51:48.442381+0000","last_fullsized":"2024-09-16T15:51:48.442381+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":167,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:51:47.345679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:51:47.345679+0000","last_clean_scrub_stamp":"2024-09-16T15:51:47.345679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T18:04:29.405916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.f","version":"168'1","reported_seq":14,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-09-16T15:51:48.446043+0000","last_change":"2024-09-16T15:51:47.379898+0000","last_active":"2024-09-16T15:51:48.446043+0000","last_peered":"2024-09-16T15:51:48.446043+0000","last_clean":"2024-09-16T15:51:48.446043+0000","last_became_active":"2024-09-16T15:51:47.379527+0000","last_became_peered":"2024-09-16T15:51:47.379527+0000","last_unstale":"2024-09-16T15:51:48.446043+0000","last_undegraded":"2024-09-16T15:51:48.446043+0000","last_fullsized":"2024-09-16T15:51:48.446043+0000","mapping_epoch":166,"log_start":"0'0","ondisk_log_start":"0'0","created":166,"last_epoch_clean":167,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:51:46.335209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:51:46.335209+0000","last_clean_scrub_stamp":"2024-09-16T15:51:46.335209+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:25:10.866081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.f","version":"0'0","reported_seq":10,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-09-16T15:51:48.441163+0000","last_change":"2024-09-16T15:51:48.430112+0000","last_active":"2024-09-16T15:51:48.441163+0000","last_peered":"2024-09-16T15:51:48.441163+0000","last_clean":"2024-09-16T15:51:48.441163+0000","last_became_active":"2024-09-16T15:51:48.429443+0000","last_became_peered":"2024-09-16T15:51:48.429443+0000","last_unstale":"2024-09-16T15:51:48.441163+0000","last_undegraded":"2024-09-16T15:51:48.441163+0000","last_fullsized":"2024-09-16T15:51:48.441163+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":167,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:51:47.345679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:51:47.345679+0000","last_clean_scrub_stamp":"2024-09-16T15:51:47.345679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:55:21.189810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.e","version":"168'2","reported_seq":15,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-09-16T15:51:48.447973+0000","last_change":"2024-09-16T15:51:47.384057+0000","last_active":"2024-09-16T15:51:48.447973+0000","last_peered":"2024-09-16T15:51:48.447973+0000","last_clean":"2024-09-16T15:51:48.447973+0000","last_became_active":"2024-09-16T15:51:47.381514+0000","last_became_peered":"2024-09-16T15:51:47.381514+0000","last_unstale":"2024-09-16T15:51:48.447973+0000","last_undegraded":"2024-09-16T15:51:48.447973+0000","last_fullsized":"2024-09-16T15:51:48.447973+0000","mapping_epoch":166,"log_start":"0'0","ondisk_log_start":"0'0","created":166,"last_epoch_clean":167,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:51:46.335209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:51:46.335209+0000","last_clean_scrub_stamp":"2024-09-16T15:51:46.335209+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T00:21:03.749321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.c","version":"0'0","reported_seq":13,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-09-16T15:51:49.391803+0000","last_change":"2024-09-16T15:51:48.428177+0000","last_active":"2024-09-16T15:51:49.391803+0000","last_peered":"2024-09-16T15:51:49.391803+0000","last_clean":"2024-09-16T15:51:49.391803+0000","last_became_active":"2024-09-16T15:51:48.427720+0000","last_became_peered":"2024-09-16T15:51:48.427720+0000","last_unstale":"2024-09-16T15:51:49.391803+0000","last_undegraded":"2024-09-16T15:51:49.391803+0000","last_fullsized":"2024-09-16T15:51:49.391803+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":167,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:51:47.345679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:51:47.345679+0000","last_clean_scrub_stamp":"2024-09-16T15:51:47.345679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T02:35:03.111789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.d","version":"168'1","reported_seq":14,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-09-16T15:51:48.447409+0000","last_change":"2024-09-16T15:51:47.372080+0000","last_active":"2024-09-16T15:51:48.447409+0000","last_peered":"2024-09-16T15:51:48.447409+0000","last_clean":"2024-09-16T15:51:48.447409+0000","last_became_active":"2024-09-16T15:51:47.371640+0000","last_became_peered":"2024-09-16T15:51:47.371640+0000","last_unstale":"2024-09-16T15:51:48.447409+0000","last_undegraded":"2024-09-16T15:51:48.447409+0000","last_fullsized":"2024-09-16T15:51:48.447409+0000","mapping_epoch":166,"log_start":"0'0","ondisk_log_start":"0'0","created":166,"last_epoch_clean":167,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:51:46.335209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:51:46.335209+0000","last_clean_scrub_stamp":"2024-09-16T15:51:46.335209+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T23:13:41.484039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.d","version":"0'0","reported_seq":10,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-09-16T15:51:48.418959+0000","last_change":"2024-09-16T15:51:48.412794+0000","last_active":"2024-09-16T15:51:48.418959+0000","last_peered":"2024-09-16T15:51:48.418959+0000","last_clean":"2024-09-16T15:51:48.418959+0000","last_became_active":"2024-09-16T15:51:48.411719+0000","last_became_peered":"2024-09-16T15:51:48.411719+0000","last_unstale":"2024-09-16T15:51:48.418959+0000","last_undegraded":"2024-09-16T15:51:48.418959+0000","last_fullsized":"2024-09-16T15:51:48.418959+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":167,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:51:47.345679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:51:47.345679+0000","last_clean_scrub_stamp":"2024-09-16T15:51:47.345679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:16:35.479295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.c","version":"0'0","reported_seq":15,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-09-16T15:51:49.391949+0000","last_change":"2024-09-16T15:51:47.376422+0000","last_active":"2024-09-16T15:51:49.391949+0000","last_peered":"2024-09-16T15:51:49.391949+0000","last_clean":"2024-09-16T15:51:49.391949+0000","last_became_active":"2024-09-16T15:51:47.376112+0000","last_became_peered":"2024-09-16T15:51:47.376112+0000","last_unstale":"2024-09-16T15:51:49.391949+0000","last_undegraded":"2024-09-16T15:51:49.391949+0000","last_fullsized":"2024-09-16T15:51:49.391949+0000","mapping_epoch":166,"log_start":"0'0","ondisk_log_start":"0'0","created":166,"last_epoch_clean":167,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:51:46.335209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:51:46.335209+0000","last_clean_scrub_stamp":"2024-09-16T15:51:46.335209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T00:19:10.515395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.a","version":"0'0","reported_seq":10,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-09-16T15:51:48.421215+0000","last_change":"2024-09-16T15:51:48.415593+0000","last_active":"2024-09-16T15:51:48.421215+0000","last_peered":"2024-09-16T15:51:48.421215+0000","last_clean":"2024-09-16T15:51:48.421215+0000","last_became_active":"2024-09-16T15:51:48.412647+0000","last_became_peered":"2024-09-16T15:51:48.412647+0000","last_unstale":"2024-09-16T15:51:48.421215+0000","last_undegraded":"2024-09-16T15:51:48.421215+0000","last_fullsized":"2024-09-16T15:51:48.421215+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":167,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:51:47.345679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:51:47.345679+0000","last_clean_scrub_stamp":"2024-09-16T15:51:47.345679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:11:00.105837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.b","version":"168'2","reported_seq":15,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-09-16T15:51:48.449047+0000","last_change":"2024-09-16T15:51:47.373400+0000","last_active":"2024-09-16T15:51:48.449047+0000","last_peered":"2024-09-16T15:51:48.449047+0000","last_clean":"2024-09-16T15:51:48.449047+0000","last_became_active":"2024-09-16T15:51:47.372217+0000","last_became_peered":"2024-09-16T15:51:47.372217+0000","last_unstale":"2024-09-16T15:51:48.449047+0000","last_undegraded":"2024-09-16T15:51:48.449047+0000","last_fullsized":"2024-09-16T15:51:48.449047+0000","mapping_epoch":166,"log_start":"0'0","ondisk_log_start":"0'0","created":166,"last_epoch_clean":167,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:51:46.335209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:51:46.335209+0000","last_clean_scrub_stamp":"2024-09-16T15:51:46.335209+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T01:00:00.780682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.b","version":"0'0","reported_seq":13,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-09-16T15:51:49.390999+0000","last_change":"2024-09-16T15:51:48.400042+0000","last_active":"2024-09-16T15:51:49.390999+0000","last_peered":"2024-09-16T15:51:49.390999+0000","last_clean":"2024-09-16T15:51:49.390999+0000","last_became_active":"2024-09-16T15:51:48.399206+0000","last_became_peered":"2024-09-16T15:51:48.399206+0000","last_unstale":"2024-09-16T15:51:49.390999+0000","last_undegraded":"2024-09-16T15:51:49.390999+0000","last_fullsized":"2024-09-16T15:51:49.390999+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":167,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:51:47.345679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:51:47.345679+0000","last_clean_scrub_stamp":"2024-09-16T15:51:47.345679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T00:06:40.282867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.a","version":"0'0","reported_seq":15,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-09-16T15:51:49.690554+0000","last_change":"2024-09-16T15:51:47.386137+0000","last_active":"2024-09-16T15:51:49.690554+0000","last_peered":"2024-09-16T15:51:49.690554+0000","last_clean":"2024-09-16T15:51:49.690554+0000","last_became_active":"2024-09-16T15:51:47.385852+0000","last_became_peered":"2024-09-16T15:51:47.385852+0000","last_unstale":"2024-09-16T15:51:49.690554+0000","last_undegraded":"2024-09-16T15:51:49.690554+0000","last_fullsized":"2024-09-16T15:51:49.690554+0000","mapping_epoch":166,"log_start":"0'0","ondisk_log_start":"0'0","created":166,"last_epoch_clean":167,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:51:46.335209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:51:46.335209+0000","last_clean_scrub_stamp":"2024-09-16T15:51:46.335209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:04:19.459115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.8","version":"0'0","reported_seq":10,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-09-16T15:51:48.440570+0000","last_change":"2024-09-16T15:51:48.427352+0000","last_active":"2024-09-16T15:51:48.440570+0000","last_peered":"2024-09-16T15:51:48.440570+0000","last_clean":"2024-09-16T15:51:48.440570+0000","last_became_active":"2024-09-16T15:51:48.426933+0000","last_became_peered":"2024-09-16T15:51:48.426933+0000","last_unstale":"2024-09-16T15:51:48.440570+0000","last_undegraded":"2024-09-16T15:51:48.440570+0000","last_fullsized":"2024-09-16T15:51:48.440570+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":167,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:51:47.345679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:51:47.345679+0000","last_clean_scrub_stamp":"2024-09-16T15:51:47.345679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T18:02:06.552134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.9","version":"0'0","reported_seq":15,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-09-16T15:51:49.690367+0000","last_change":"2024-09-16T15:51:47.379850+0000","last_active":"2024-09-16T15:51:49.690367+0000","last_peered":"2024-09-16T15:51:49.690367+0000","last_clean":"2024-09-16T15:51:49.690367+0000","last_became_active":"2024-09-16T15:51:47.379448+0000","last_became_peered":"2024-09-16T15:51:47.379448+0000","last_unstale":"2024-09-16T15:51:49.690367+0000","last_undegraded":"2024-09-16T15:51:49.690367+0000","last_fullsized":"2024-09-16T15:51:49.690367+0000","mapping_epoch":166,"log_start":"0'0","ondisk_log_start":"0'0","created":166,"last_epoch_clean":167,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:51:46.335209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:51:46.335209+0000","last_clean_scrub_stamp":"2024-09-16T15:51:46.335209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:21:24.176527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.9","version":"0'0","reported_seq":13,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-09-16T15:51:49.391718+0000","last_change":"2024-09-16T15:51:48.388897+0000","last_active":"2024-09-16T15:51:49.391718+0000","last_peered":"2024-09-16T15:51:49.391718+0000","last_clean":"2024-09-16T15:51:49.391718+0000","last_became_active":"2024-09-16T15:51:48.388059+0000","last_became_peered":"2024-09-16T15:51:48.388059+0000","last_unstale":"2024-09-16T15:51:49.391718+0000","last_undegraded":"2024-09-16T15:51:49.391718+0000","last_fullsized":"2024-09-16T15:51:49.391718+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":167,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:51:47.345679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:51:47.345679+0000","last_clean_scrub_stamp":"2024-09-16T15:51:47.345679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T00:26:02.794506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.8","version":"0'0","reported_seq":13,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-09-16T15:51:48.382381+0000","last_change":"2024-09-16T15:51:47.378342+0000","last_active":"2024-09-16T15:51:48.382381+0000","last_peered":"2024-09-16T15:51:48.382381+0000","last_clean":"2024-09-16T15:51:48.382381+0000","last_became_active":"2024-09-16T15:51:47.378014+0000","last_became_peered":"2024-09-16T15:51:47.378014+0000","last_unstale":"2024-09-16T15:51:48.382381+0000","last_undegraded":"2024-09-16T15:51:48.382381+0000","last_fullsized":"2024-09-16T15:51:48.382381+0000","mapping_epoch":166,"log_start":"0'0","ondisk_log_start":"0'0","created":166,"last_epoch_clean":167,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:51:46.335209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:51:46.335209+0000","last_clean_scrub_stamp":"2024-09-16T15:51:46.335209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:46:57.710217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.6","version":"0'0","reported_seq":13,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-09-16T15:51:49.392146+0000","last_change":"2024-09-16T15:51:48.408607+0000","last_active":"2024-09-16T15:51:49.392146+0000","last_peered":"2024-09-16T15:51:49.392146+0000","last_clean":"2024-09-16T15:51:49.392146+0000","last_became_active":"2024-09-16T15:51:48.407850+0000","last_became_peered":"2024-09-16T15:51:48.407850+0000","last_unstale":"2024-09-16T15:51:49.392146+0000","last_undegraded":"2024-09-16T15:51:49.392146+0000","last_fullsized":"2024-09-16T15:51:49.392146+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":167,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:51:47.345679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:51:47.345679+0000","last_clean_scrub_stamp":"2024-09-16T15:51:47.345679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:35:12.916299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"26.7","version":"168'2","reported_seq":17,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-09-16T15:51:49.392347+0000","last_change":"2024-09-16T15:51:47.389839+0000","last_active":"2024-09-16T15:51:49.392347+0000","last_peered":"2024-09-16T15:51:49.392347+0000","last_clean":"2024-09-16T15:51:49.392347+0000","last_became_active":"2024-09-16T15:51:47.389541+0000","last_became_peered":"2024-09-16T15:51:47.389541+0000","last_unstale":"2024-09-16T15:51:49.392347+0000","last_undegraded":"2024-09-16T15:51:49.392347+0000","last_fullsized":"2024-09-16T15:51:49.392347+0000","mapping_epoch":166,"log_start":"0'0","ondisk_log_start":"0'0","created":166,"last_epoch_clean":167,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:51:46.335209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:51:46.335209+0000","last_clean_scrub_stamp":"2024-09-16T15:51:46.335209+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:06:10.859212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.7","version":"0'0","reported_seq":13,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-09-16T15:51:49.690969+0000","last_change":"2024-09-16T15:51:48.435404+0000","last_active":"2024-09-16T15:51:49.690969+0000","last_peered":"2024-09-16T15:51:49.690969+0000","last_clean":"2024-09-16T15:51:49.690969+0000","last_became_active":"2024-09-16T15:51:48.435074+0000","last_became_peered":"2024-09-16T15:51:48.435074+0000","last_unstale":"2024-09-16T15:51:49.690969+0000","last_undegraded":"2024-09-16T15:51:49.690969+0000","last_fullsized":"2024-09-16T15:51:49.690969+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":167,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:51:47.345679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:51:47.345679+0000","last_clean_scrub_stamp":"2024-09-16T15:51:47.345679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T01:52:02.077065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.6","version":"169'1","reported_seq":16,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-09-16T15:51:49.438689+0000","last_change":"2024-09-16T15:51:47.396058+0000","last_active":"2024-09-16T15:51:49.438689+0000","last_peered":"2024-09-16T15:51:49.438689+0000","last_clean":"2024-09-16T15:51:49.438689+0000","last_became_active":"2024-09-16T15:51:47.395107+0000","last_became_peered":"2024-09-16T15:51:47.395107+0000","last_unstale":"2024-09-16T15:51:49.438689+0000","last_undegraded":"2024-09-16T15:51:49.438689+0000","last_fullsized":"2024-09-16T15:51:49.438689+0000","mapping_epoch":166,"log_start":"0'0","ondisk_log_start":"0'0","created":166,"last_epoch_clean":167,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:51:46.335209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:51:46.335209+0000","last_clean_scrub_stamp":"2024-09-16T15:51:46.335209+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T02:26:22.249280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.4","version":"0'0","reported_seq":10,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-09-16T15:51:48.441382+0000","last_change":"2024-09-16T15:51:48.434690+0000","last_active":"2024-09-16T15:51:48.441382+0000","last_peered":"2024-09-16T15:51:48.441382+0000","last_clean":"2024-09-16T15:51:48.441382+0000","last_became_active":"2024-09-16T15:51:48.434213+0000","last_became_peered":"2024-09-16T15:51:48.434213+0000","last_unstale":"2024-09-16T15:51:48.441382+0000","last_undegraded":"2024-09-16T15:51:48.441382+0000","last_fullsized":"2024-09-16T15:51:48.441382+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":167,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:51:47.345679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:51:47.345679+0000","last_clean_scrub_stamp":"2024-09-16T15:51:47.345679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T23:02:30.232897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.5","version":"0'0","reported_seq":13,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-09-16T15:51:48.380068+0000","last_change":"2024-09-16T15:51:47.386153+0000","last_active":"2024-09-16T15:51:48.380068+0000","last_peered":"2024-09-16T15:51:48.380068+0000","last_clean":"2024-09-16T15:51:48.380068+0000","last_became_active":"2024-09-16T15:51:47.385788+0000","last_became_peered":"2024-09-16T15:51:47.385788+0000","last_unstale":"2024-09-16T15:51:48.380068+0000","last_undegraded":"2024-09-16T15:51:48.380068+0000","last_fullsized":"2024-09-16T15:51:48.380068+0000","mapping_epoch":166,"log_start":"0'0","ondisk_log_start":"0'0","created":166,"last_epoch_clean":167,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:51:46.335209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:51:46.335209+0000","last_clean_scrub_stamp":"2024-09-16T15:51:46.335209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T23:22:20.611338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.5","version":"0'0","reported_seq":13,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-09-16T15:51:49.393390+0000","last_change":"2024-09-16T15:51:48.421671+0000","last_active":"2024-09-16T15:51:49.393390+0000","last_peered":"2024-09-16T15:51:49.393390+0000","last_clean":"2024-09-16T15:51:49.393390+0000","last_became_active":"2024-09-16T15:51:48.419577+0000","last_became_peered":"2024-09-16T15:51:48.419577+0000","last_unstale":"2024-09-16T15:51:49.393390+0000","last_undegraded":"2024-09-16T15:51:49.393390+0000","last_fullsized":"2024-09-16T15:51:49.393390+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":167,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:51:47.345679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:51:47.345679+0000","last_clean_scrub_stamp":"2024-09-16T15:51:47.345679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T01:37:38.821256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.4","version":"0'0","reported_seq":15,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-09-16T15:51:49.391972+0000","last_change":"2024-09-16T15:51:47.385805+0000","last_active":"2024-09-16T15:51:49.391972+0000","last_peered":"2024-09-16T15:51:49.391972+0000","last_clean":"2024-09-16T15:51:49.391972+0000","last_became_active":"2024-09-16T15:51:47.385565+0000","last_became_peered":"2024-09-16T15:51:47.385565+0000","last_unstale":"2024-09-16T15:51:49.391972+0000","last_undegraded":"2024-09-16T15:51:49.391972+0000","last_fullsized":"2024-09-16T15:51:49.391972+0000","mapping_epoch":166,"log_start":"0'0","ondisk_log_start":"0'0","created":166,"last_epoch_clean":167,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:51:46.335209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:51:46.335209+0000","last_clean_scrub_stamp":"2024-09-16T15:51:46.335209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T00:21:03.749321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.2","version":"0'0","reported_seq":13,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-09-16T15:51:49.691184+0000","last_change":"2024-09-16T15:51:48.435351+0000","last_active":"2024-09-16T15:51:49.691184+0000","last_peered":"2024-09-16T15:51:49.691184+0000","last_clean":"2024-09-16T15:51:49.691184+0000","last_became_active":"2024-09-16T15:51:48.434975+0000","last_became_peered":"2024-09-16T15:51:48.434975+0000","last_unstale":"2024-09-16T15:51:49.691184+0000","last_undegraded":"2024-09-16T15:51:49.691184+0000","last_fullsized":"2024-09-16T15:51:49.691184+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":167,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:51:47.345679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:51:47.345679+0000","last_clean_scrub_stamp":"2024-09-16T15:51:47.345679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T01:31:06.621794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.3","version":"0'0","reported_seq":15,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-09-16T15:51:49.434947+0000","last_change":"2024-09-16T15:51:47.389057+0000","last_active":"2024-09-16T15:51:49.434947+0000","last_peered":"2024-09-16T15:51:49.434947+0000","last_clean":"2024-09-16T15:51:49.434947+0000","last_became_active":"2024-09-16T15:51:47.388202+0000","last_became_peered":"2024-09-16T15:51:47.388202+0000","last_unstale":"2024-09-16T15:51:49.434947+0000","last_undegraded":"2024-09-16T15:51:49.434947+0000","last_fullsized":"2024-09-16T15:51:49.434947+0000","mapping_epoch":166,"log_start":"0'0","ondisk_log_start":"0'0","created":166,"last_epoch_clean":167,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:51:46.335209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:51:46.335209+0000","last_clean_scrub_stamp":"2024-09-16T15:51:46.335209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:34:14.296519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.3","version":"0'0","reported_seq":13,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-09-16T15:51:49.390480+0000","last_change":"2024-09-16T15:51:48.403829+0000","last_active":"2024-09-16T15:51:49.390480+0000","last_peered":"2024-09-16T15:51:49.390480+0000","last_clean":"2024-09-16T15:51:49.390480+0000","last_became_active":"2024-09-16T15:51:48.402950+0000","last_became_peered":"2024-09-16T15:51:48.402950+0000","last_unstale":"2024-09-16T15:51:49.390480+0000","last_undegraded":"2024-09-16T15:51:49.390480+0000","last_fullsized":"2024-09-16T15:51:49.390480+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":167,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:51:47.345679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:51:47.345679+0000","last_clean_scrub_stamp":"2024-09-16T15:51:47.345679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:41:32.289325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"26.2","version":"0'0","reported_seq":13,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-09-16T15:51:48.381208+0000","last_change":"2024-09-16T15:51:47.379021+0000","last_active":"2024-09-16T15:51:48.381208+0000","last_peered":"2024-09-16T15:51:48.381208+0000","last_clean":"2024-09-16T15:51:48.381208+0000","last_became_active":"2024-09-16T15:51:47.378682+0000","last_became_peered":"2024-09-16T15:51:47.378682+0000","last_unstale":"2024-09-16T15:51:48.381208+0000","last_undegraded":"2024-09-16T15:51:48.381208+0000","last_fullsized":"2024-09-16T15:51:48.381208+0000","mapping_epoch":166,"log_start":"0'0","ondisk_log_start":"0'0","created":166,"last_epoch_clean":167,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:51:46.335209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:51:46.335209+0000","last_clean_scrub_stamp":"2024-09-16T15:51:46.335209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:02:05.529566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.0","version":"0'0","reported_seq":13,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-09-16T15:51:49.391316+0000","last_change":"2024-09-16T15:51:48.406255+0000","last_active":"2024-09-16T15:51:49.391316+0000","last_peered":"2024-09-16T15:51:49.391316+0000","last_clean":"2024-09-16T15:51:49.391316+0000","last_became_active":"2024-09-16T15:51:48.405514+0000","last_became_peered":"2024-09-16T15:51:48.405514+0000","last_unstale":"2024-09-16T15:51:49.391316+0000","last_undegraded":"2024-09-16T15:51:49.391316+0000","last_fullsized":"2024-09-16T15:51:49.391316+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":167,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:51:47.345679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:51:47.345679+0000","last_clean_scrub_stamp":"2024-09-16T15:51:47.345679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T18:46:13.824641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.1","version":"0'0","reported_seq":15,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-09-16T15:51:49.392009+0000","last_change":"2024-09-16T15:51:47.369871+0000","last_active":"2024-09-16T15:51:49.392009+0000","last_peered":"2024-09-16T15:51:49.392009+0000","last_clean":"2024-09-16T15:51:49.392009+0000","last_became_active":"2024-09-16T15:51:47.369135+0000","last_became_peered":"2024-09-16T15:51:47.369135+0000","last_unstale":"2024-09-16T15:51:49.392009+0000","last_undegraded":"2024-09-16T15:51:49.392009+0000","last_fullsized":"2024-09-16T15:51:49.392009+0000","mapping_epoch":166,"log_start":"0'0","ondisk_log_start":"0'0","created":166,"last_epoch_clean":167,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:51:46.335209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:51:46.335209+0000","last_clean_scrub_stamp":"2024-09-16T15:51:46.335209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:35:11.905829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.1","version":"0'0","reported_seq":10,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-09-16T15:51:48.440646+0000","last_change":"2024-09-16T15:51:48.429777+0000","last_active":"2024-09-16T15:51:48.440646+0000","last_peered":"2024-09-16T15:51:48.440646+0000","last_clean":"2024-09-16T15:51:48.440646+0000","last_became_active":"2024-09-16T15:51:48.428509+0000","last_became_peered":"2024-09-16T15:51:48.428509+0000","last_unstale":"2024-09-16T15:51:48.440646+0000","last_undegraded":"2024-09-16T15:51:48.440646+0000","last_fullsized":"2024-09-16T15:51:48.440646+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":167,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:51:47.345679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:51:47.345679+0000","last_clean_scrub_stamp":"2024-09-16T15:51:47.345679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T01:48:08.768795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.0","version":"0'0","reported_seq":15,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-09-16T15:51:49.389327+0000","last_change":"2024-09-16T15:51:47.378789+0000","last_active":"2024-09-16T15:51:49.389327+0000","last_peered":"2024-09-16T15:51:49.389327+0000","last_clean":"2024-09-16T15:51:49.389327+0000","last_became_active":"2024-09-16T15:51:47.378096+0000","last_became_peered":"2024-09-16T15:51:47.378096+0000","last_unstale":"2024-09-16T15:51:49.389327+0000","last_undegraded":"2024-09-16T15:51:49.389327+0000","last_fullsized":"2024-09-16T15:51:49.389327+0000","mapping_epoch":166,"log_start":"0'0","ondisk_log_start":"0'0","created":166,"last_epoch_clean":167,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:51:46.335209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:51:46.335209+0000","last_clean_scrub_stamp":"2024-09-16T15:51:46.335209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T00:54:28.885278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.1a","version":"0'0","reported_seq":13,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-09-16T15:51:49.690032+0000","last_change":"2024-09-16T15:51:48.433167+0000","last_active":"2024-09-16T15:51:49.690032+0000","last_peered":"2024-09-16T15:51:49.690032+0000","last_clean":"2024-09-16T15:51:49.690032+0000","last_became_active":"2024-09-16T15:51:48.432726+0000","last_became_peered":"2024-09-16T15:51:48.432726+0000","last_unstale":"2024-09-16T15:51:49.690032+0000","last_undegraded":"2024-09-16T15:51:49.690032+0000","last_fullsized":"2024-09-16T15:51:49.690032+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":167,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:51:47.345679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:51:47.345679+0000","last_clean_scrub_stamp":"2024-09-16T15:51:47.345679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T00:17:12.711120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.1b","version":"168'1","reported_seq":16,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-09-16T15:51:49.391363+0000","last_change":"2024-09-16T15:51:47.377240+0000","last_active":"2024-09-16T15:51:49.391363+0000","last_peered":"2024-09-16T15:51:49.391363+0000","last_clean":"2024-09-16T15:51:49.391363+0000","last_became_active":"2024-09-16T15:51:47.376510+0000","last_became_peered":"2024-09-16T15:51:47.376510+0000","last_unstale":"2024-09-16T15:51:49.391363+0000","last_undegraded":"2024-09-16T15:51:49.391363+0000","last_fullsized":"2024-09-16T15:51:49.391363+0000","mapping_epoch":166,"log_start":"0'0","ondisk_log_start":"0'0","created":166,"last_epoch_clean":167,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:51:46.335209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:51:46.335209+0000","last_clean_scrub_stamp":"2024-09-16T15:51:46.335209+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T00:08:59.993560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"73'79","reported_seq":412,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-09-16T15:51:49.434428+0000","last_change":"2024-09-16T15:42:40.881820+0000","last_active":"2024-09-16T15:51:49.434428+0000","last_peered":"2024-09-16T15:51:49.434428+0000","last_clean":"2024-09-16T15:51:49.434428+0000","last_became_active":"2024-09-16T15:42:40.879965+0000","last_became_peered":"2024-09-16T15:42:40.879965+0000","last_unstale":"2024-09-16T15:51:49.434428+0000","last_undegraded":"2024-09-16T15:51:49.434428+0000","last_fullsized":"2024-09-16T15:51:49.434428+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:42:39.728762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:42:39.728762+0000","last_clean_scrub_stamp":"2024-09-16T15:42:39.728762+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:22:18.768932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":26,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":27,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79,"ondisk_log_size":79,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542258,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33852,"kb_used_data":2660,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94337988,"statfs":{"total":96636764160,"available":96602099712,"internally_reserved":0,"allocated":2723840,"data_stored":1530773,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542257,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33264,"kb_used_data":2072,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338576,"statfs":{"total":96636764160,"available":96602701824,"internally_reserved":0,"allocated":2121728,"data_stored":939375,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":5,"up_from":14,"seq":60129542258,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33260,"kb_used_data":2072,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338580,"statfs":{"total":96636764160,"available":96602705920,"internally_reserved":0,"allocated":2121728,"data_stored":939375,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542256,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33288,"kb_used_data":2100,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338552,"statfs":{"total":96636764160,"available":96602677248,"internally_reserved":0,"allocated":2150400,"data_stored":946837,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":3,"up_from":14,"seq":60129542257,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37428,"kb_used_data":2080,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334412,"statfs":{"total":96636764160,"available":96598437888,"internally_reserved":0,"allocated":2129920,"data_stored":938975,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":14,"seq":60129542257,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33272,"kb_used_data":2080,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338568,"statfs":{"total":96636764160,"available":96602693632,"internally_reserved":0,"allocated":2129920,"data_stored":945121,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":14,"seq":60129542259,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33268,"kb_used_data":2076,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338572,"statfs":{"total":96636764160,"available":96602697728,"internally_reserved":0,"allocated":2125824,"data_stored":938849,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":0,"up_from":14,"seq":60129542259,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38008,"kb_used_data":2652,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94333832,"statfs":{"total":96636764160,"available":96597843968,"internally_reserved":0,"allocated":2715648,"data_stored":1529251,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1716,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T15:51:52.762 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-09-16T15:51:52.762 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-09-16T15:51:52.940 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:52.941+0000 7f5873bdd640 1 -- 172.21.15.123:0/1597236939 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f58580062e0 msgr2=0x7f585800cae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:51:52.940 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:52.941+0000 7f5873bdd640 1 --2- 172.21.15.123:0/1597236939 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f58580062e0 0x7f585800cae0 secure :-1 s=READY pgs=1889 cs=0 l=1 rev1=1 crypto rx=0x7f5864009b60 tx=0x7f586401d2b0 comp rx=0 tx=0).stop 2024-09-16T15:51:52.940 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:52.941+0000 7f5873bdd640 1 -- 172.21.15.123:0/1597236939 shutdown_connections 2024-09-16T15:51:52.940 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:52.941+0000 7f5873bdd640 1 --2- 172.21.15.123:0/1597236939 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f58580062e0 0x7f585800cae0 unknown :-1 s=CLOSED pgs=1889 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:52.940 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:52.941+0000 7f5873bdd640 1 -- 172.21.15.123:0/1597236939 >> 172.21.15.123:0/1597236939 conn(0x7f586c0b1710 msgr2=0x7f586c0b1b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:51:52.940 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:52.941+0000 7f5873bdd640 1 -- 172.21.15.123:0/1597236939 shutdown_connections 2024-09-16T15:51:52.940 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:52.941+0000 7f5873bdd640 1 -- 172.21.15.123:0/1597236939 wait complete. 2024-09-16T15:51:52.941 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:52.941+0000 7f5873bdd640 1 Processor -- start 2024-09-16T15:51:52.941 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:52.941+0000 7f5873bdd640 1 -- start start 2024-09-16T15:51:52.943 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:52.941+0000 7f5873bdd640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f58580062e0 0x7f586c0c4f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:51:52.943 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:52.941+0000 7f5873bdd640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f586c1f6ac0 0x7f586c148e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:51:52.943 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:52.941+0000 7f5873bdd640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f586c0b31d0 0x7f586c0b3650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:51:52.943 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:52.941+0000 7f5873bdd640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f586c15e140 con 0x7f586c1f6ac0 2024-09-16T15:51:52.943 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:52.941+0000 7f5873bdd640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f586c15dfc0 con 0x7f586c0b31d0 2024-09-16T15:51:52.943 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:52.941+0000 7f5873bdd640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f586c15e2c0 con 0x7f58580062e0 2024-09-16T15:51:52.943 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:52.941+0000 7f5871178640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f586c1f6ac0 0x7f586c148e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:51:52.943 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:52.941+0000 7f5871178640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f586c1f6ac0 0x7f586c148e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:43494/0 (socket says 172.21.15.123:43494) 2024-09-16T15:51:52.943 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:52.941+0000 7f5871178640 1 -- 172.21.15.123:0/722476039 learned_addr learned my addr 172.21.15.123:0/722476039 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:51:52.943 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:52.941+0000 7f587217a640 1 --2- 172.21.15.123:0/722476039 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f586c0b31d0 0x7f586c0b3650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:51:52.943 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:52.941+0000 7f5871979640 1 --2- 172.21.15.123:0/722476039 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f58580062e0 0x7f586c0c4f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:51:52.943 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:52.941+0000 7f5871178640 1 -- 172.21.15.123:0/722476039 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f58580062e0 msgr2=0x7f586c0c4f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:51:52.944 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:52.941+0000 7f5871178640 1 --2- 172.21.15.123:0/722476039 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f58580062e0 0x7f586c0c4f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:52.944 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:52.941+0000 7f5871178640 1 -- 172.21.15.123:0/722476039 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f586c0b31d0 msgr2=0x7f586c0b3650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:51:52.944 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:52.941+0000 7f5871178640 1 --2- 172.21.15.123:0/722476039 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f586c0b31d0 0x7f586c0b3650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:52.944 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:52.941+0000 7f5871178640 1 -- 172.21.15.123:0/722476039 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f586c1f6f40 con 0x7f586c1f6ac0 2024-09-16T15:51:52.944 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:52.941+0000 7f5871178640 1 --2- 172.21.15.123:0/722476039 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f586c1f6ac0 0x7f586c148e30 secure :-1 s=READY pgs=1890 cs=0 l=1 rev1=1 crypto rx=0x7f585c00e590 tx=0x7f585c00ea50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:51:52.944 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:52.945+0000 7f58627fc640 1 -- 172.21.15.123:0/722476039 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f585c015070 con 0x7f586c1f6ac0 2024-09-16T15:51:52.944 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:52.945+0000 7f58627fc640 1 -- 172.21.15.123:0/722476039 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f585c00c9e0 con 0x7f586c1f6ac0 2024-09-16T15:51:52.944 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:52.945+0000 7f58627fc640 1 -- 172.21.15.123:0/722476039 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f585c00ce30 con 0x7f586c1f6ac0 2024-09-16T15:51:52.944 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:52.945+0000 7f5873bdd640 1 -- 172.21.15.123:0/722476039 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f586c0c55e0 con 0x7f586c1f6ac0 2024-09-16T15:51:52.944 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:52.945+0000 7f5873bdd640 1 -- 172.21.15.123:0/722476039 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f586c0c5ba0 con 0x7f586c1f6ac0 2024-09-16T15:51:52.945 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:52.945+0000 7f5873bdd640 1 -- 172.21.15.123:0/722476039 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f586c0730e0 con 0x7f586c1f6ac0 2024-09-16T15:51:52.945 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:52.945+0000 7f58627fc640 1 -- 172.21.15.123:0/722476039 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 25) ==== 97998+0+0 (secure 0 0 0) 0x7f585c018070 con 0x7f586c1f6ac0 2024-09-16T15:51:52.945 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:52.945+0000 7f58627fc640 1 --2- 172.21.15.123:0/722476039 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f5850073940 0x7f5850075e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:51:52.945 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:52.945+0000 7f5871979640 1 --2- 172.21.15.123:0/722476039 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f5850073940 0x7f5850075e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:51:52.945 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:52.945+0000 7f58627fc640 1 -- 172.21.15.123:0/722476039 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(169..169 src has 1..169) ==== 5737+0+0 (secure 0 0 0) 0x7f585c092cc0 con 0x7f586c1f6ac0 2024-09-16T15:51:52.945 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:52.945+0000 7f5871979640 1 --2- 172.21.15.123:0/722476039 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f5850073940 0x7f5850075e00 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f5868009e90 tx=0x7f5868009500 comp rx=0 tx=0).ready entity=mgr.4150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:51:52.948 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:52.949+0000 7f58627fc640 1 -- 172.21.15.123:0/722476039 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f585c05f7a0 con 0x7f586c1f6ac0 2024-09-16T15:51:53.168 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:53.169+0000 7f5873bdd640 1 -- 172.21.15.123:0/722476039 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) -- 0x7f586c15d7c0 con 0x7f586c1f6ac0 2024-09-16T15:51:53.171 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:53.173+0000 7f58627fc640 1 -- 172.21.15.123:0/722476039 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v30) ==== 190+0+147 (secure 0 0 0) 0x7f585c05f0a0 con 0x7f586c1f6ac0 2024-09-16T15:51:53.171 INFO:teuthology.orchestra.run.smithi123.stdout:[client.0] 2024-09-16T15:51:53.171 INFO:teuthology.orchestra.run.smithi123.stdout: key = AQC1Uehm7GDTAhAAHWWVIyjVVnTeeozBR8DfFA== 2024-09-16T15:51:53.172 INFO:teuthology.orchestra.run.smithi123.stdout: caps mds = "allow" 2024-09-16T15:51:53.172 INFO:teuthology.orchestra.run.smithi123.stdout: caps mon = "allow r" 2024-09-16T15:51:53.172 INFO:teuthology.orchestra.run.smithi123.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-09-16T15:51:53.172 INFO:teuthology.orchestra.run.smithi123.stderr:updated caps for client.0 2024-09-16T15:51:53.174 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:53.173+0000 7f5873bdd640 1 -- 172.21.15.123:0/722476039 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f5850073940 msgr2=0x7f5850075e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:51:53.174 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:53.173+0000 7f5873bdd640 1 --2- 172.21.15.123:0/722476039 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f5850073940 0x7f5850075e00 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f5868009e90 tx=0x7f5868009500 comp rx=0 tx=0).stop 2024-09-16T15:51:53.174 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:53.173+0000 7f5873bdd640 1 -- 172.21.15.123:0/722476039 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f586c1f6ac0 msgr2=0x7f586c148e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:51:53.174 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:53.173+0000 7f5873bdd640 1 --2- 172.21.15.123:0/722476039 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f586c1f6ac0 0x7f586c148e30 secure :-1 s=READY pgs=1890 cs=0 l=1 rev1=1 crypto rx=0x7f585c00e590 tx=0x7f585c00ea50 comp rx=0 tx=0).stop 2024-09-16T15:51:53.174 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:53.173+0000 7f5873bdd640 1 -- 172.21.15.123:0/722476039 shutdown_connections 2024-09-16T15:51:53.174 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:53.173+0000 7f5873bdd640 1 --2- 172.21.15.123:0/722476039 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f5850073940 0x7f5850075e00 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:53.174 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:53.173+0000 7f5873bdd640 1 --2- 172.21.15.123:0/722476039 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f586c0b31d0 0x7f586c0b3650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:53.174 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:53.173+0000 7f5873bdd640 1 --2- 172.21.15.123:0/722476039 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f586c1f6ac0 0x7f586c148e30 unknown :-1 s=CLOSED pgs=1890 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:53.174 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:53.173+0000 7f5873bdd640 1 --2- 172.21.15.123:0/722476039 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f58580062e0 0x7f586c0c4f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:53.174 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:53.173+0000 7f5873bdd640 1 -- 172.21.15.123:0/722476039 >> 172.21.15.123:0/722476039 conn(0x7f586c0b1710 msgr2=0x7f586c15cf50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:51:53.175 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:53.173+0000 7f5873bdd640 1 -- 172.21.15.123:0/722476039 shutdown_connections 2024-09-16T15:51:53.175 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:53.173+0000 7f5873bdd640 1 -- 172.21.15.123:0/722476039 wait complete. 2024-09-16T15:51:53.189 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-16T15:51:53.309 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:53.309+0000 7fba05505640 1 -- 172.21.15.123:0/3719655162 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fba0015e160 msgr2=0x7fba00073f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:51:53.310 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:53.309+0000 7fba05505640 1 --2- 172.21.15.123:0/3719655162 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fba0015e160 0x7fba00073f30 secure :-1 s=READY pgs=1892 cs=0 l=1 rev1=1 crypto rx=0x7fba000708f0 tx=0x7fb9f001c6e0 comp rx=0 tx=0).stop 2024-09-16T15:51:53.310 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:53.309+0000 7fba05505640 1 -- 172.21.15.123:0/3719655162 shutdown_connections 2024-09-16T15:51:53.310 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:53.309+0000 7fba05505640 1 --2- 172.21.15.123:0/3719655162 >> v2:172.21.15.184:3300/0 conn(0x7fba0015e820 0x7fba0015ec90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:53.310 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:53.309+0000 7fba05505640 1 --2- 172.21.15.123:0/3719655162 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fba0015e160 0x7fba00073f30 unknown :-1 s=CLOSED pgs=1892 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:53.310 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:53.309+0000 7fba05505640 1 -- 172.21.15.123:0/3719655162 >> 172.21.15.123:0/3719655162 conn(0x7fba000cfb10 msgr2=0x7fba000cff20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:51:53.310 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:53.309+0000 7fba05505640 1 -- 172.21.15.123:0/3719655162 shutdown_connections 2024-09-16T15:51:53.310 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:53.309+0000 7fba05505640 1 -- 172.21.15.123:0/3719655162 wait complete. 2024-09-16T15:51:53.310 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:53.313+0000 7fba05505640 1 Processor -- start 2024-09-16T15:51:53.311 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:53.313+0000 7fba05505640 1 -- start start 2024-09-16T15:51:53.313 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:53.313+0000 7fba05505640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fba00074470 0x7fba0014f550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:51:53.313 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:53.313+0000 7fba05505640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fba0015e160 0x7fba0015bda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:51:53.313 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:53.313+0000 7fba05505640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fba0015e820 0x7fba00157770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:51:53.313 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:53.313+0000 7fba05505640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fba000cf5b0 con 0x7fba0015e160 2024-09-16T15:51:53.313 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:53.313+0000 7fba05505640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fba000cf430 con 0x7fba00074470 2024-09-16T15:51:53.314 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:53.313+0000 7fba05505640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fba001683d0 con 0x7fba0015e820 2024-09-16T15:51:53.314 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:53.313+0000 7fb9fe7fc640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fba0015e160 0x7fba0015bda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:51:53.314 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:53.313+0000 7fb9fe7fc640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fba0015e160 0x7fba0015bda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:43514/0 (socket says 172.21.15.123:43514) 2024-09-16T15:51:53.314 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:53.313+0000 7fb9fe7fc640 1 -- 172.21.15.123:0/1906745567 learned_addr learned my addr 172.21.15.123:0/1906745567 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:51:53.314 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:53.313+0000 7fb9ff7fe640 1 --2- 172.21.15.123:0/1906745567 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fba0015e820 0x7fba00157770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:51:53.314 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:53.313+0000 7fb9feffd640 1 --2- 172.21.15.123:0/1906745567 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fba00074470 0x7fba0014f550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:51:53.314 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:53.313+0000 7fb9fe7fc640 1 -- 172.21.15.123:0/1906745567 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fba0015e820 msgr2=0x7fba00157770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:51:53.314 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:53.313+0000 7fb9fe7fc640 1 --2- 172.21.15.123:0/1906745567 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fba0015e820 0x7fba00157770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:53.314 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:53.313+0000 7fb9fe7fc640 1 -- 172.21.15.123:0/1906745567 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fba00074470 msgr2=0x7fba0014f550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:51:53.314 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:53.313+0000 7fb9fe7fc640 1 --2- 172.21.15.123:0/1906745567 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fba00074470 0x7fba0014f550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:53.314 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:53.313+0000 7fb9fe7fc640 1 -- 172.21.15.123:0/1906745567 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fba0015c7f0 con 0x7fba0015e160 2024-09-16T15:51:53.314 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:53.313+0000 7fb9ff7fe640 1 --2- 172.21.15.123:0/1906745567 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fba0015e820 0x7fba00157770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T15:51:53.315 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:53.313+0000 7fb9feffd640 1 --2- 172.21.15.123:0/1906745567 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fba00074470 0x7fba0014f550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T15:51:53.315 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:53.313+0000 7fb9fe7fc640 1 --2- 172.21.15.123:0/1906745567 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fba0015e160 0x7fba0015bda0 secure :-1 s=READY pgs=1893 cs=0 l=1 rev1=1 crypto rx=0x7fb9f400e6a0 tx=0x7fb9f400eb60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:51:53.315 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:53.313+0000 7fb9e77fe640 1 -- 172.21.15.123:0/1906745567 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9f4015070 con 0x7fba0015e160 2024-09-16T15:51:53.315 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:53.313+0000 7fb9e77fe640 1 -- 172.21.15.123:0/1906745567 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fb9f4005360 con 0x7fba0015e160 2024-09-16T15:51:53.315 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:53.313+0000 7fb9e77fe640 1 -- 172.21.15.123:0/1906745567 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9f400cab0 con 0x7fba0015e160 2024-09-16T15:51:53.315 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:53.313+0000 7fba05505640 1 -- 172.21.15.123:0/1906745567 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fba001562a0 con 0x7fba0015e160 2024-09-16T15:51:53.315 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:53.313+0000 7fba05505640 1 -- 172.21.15.123:0/1906745567 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fba00156860 con 0x7fba0015e160 2024-09-16T15:51:53.316 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:53.313+0000 7fba05505640 1 -- 172.21.15.123:0/1906745567 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fba000d06c0 con 0x7fba0015e160 2024-09-16T15:51:53.316 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:53.317+0000 7fb9e77fe640 1 -- 172.21.15.123:0/1906745567 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 25) ==== 97998+0+0 (secure 0 0 0) 0x7fb9f4018070 con 0x7fba0015e160 2024-09-16T15:51:53.316 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:53.317+0000 7fb9e77fe640 1 --2- 172.21.15.123:0/1906745567 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fb9ec0738e0 0x7fb9ec075da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:51:53.316 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:53.317+0000 7fb9feffd640 1 --2- 172.21.15.123:0/1906745567 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fb9ec0738e0 0x7fb9ec075da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:51:53.316 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:53.317+0000 7fb9e77fe640 1 -- 172.21.15.123:0/1906745567 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(169..169 src has 1..169) ==== 5737+0+0 (secure 0 0 0) 0x7fb9f4094c20 con 0x7fba0015e160 2024-09-16T15:51:53.316 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:53.317+0000 7fb9feffd640 1 --2- 172.21.15.123:0/1906745567 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fb9ec0738e0 0x7fb9ec075da0 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7fb9f00af000 tx=0x7fb9f001af20 comp rx=0 tx=0).ready entity=mgr.4150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:51:53.319 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:53.321+0000 7fb9e77fe640 1 -- 172.21.15.123:0/1906745567 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7fb9f4061680 con 0x7fba0015e160 2024-09-16T15:51:53.513 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:53.513+0000 7fba05505640 1 -- 172.21.15.123:0/1906745567 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fba000c3770 con 0x7fba0015e160 2024-09-16T15:51:53.514 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:53.513+0000 7fb9e77fe640 1 -- 172.21.15.123:0/1906745567 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 115 v115) ==== 95+0+5239 (secure 0 0 0) 0x7fb9f4060f80 con 0x7fba0015e160 2024-09-16T15:51:53.514 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-09-16T15:51:53.514 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":115,"btime":"2024-09-16T15:51:51:442311+0000","default_fscid":13,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6314,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6833/2070840764","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":2070840764},{"type":"v1","addr":"172.21.15.184:6833","nonce":2070840764}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":110},{"gid":6976,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6835/3881288354","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":3881288354},{"type":"v1","addr":"172.21.15.184:6835","nonce":3881288354}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":110},{"gid":9219,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6833/519008782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6832","nonce":519008782},{"type":"v1","addr":"172.21.15.123:6833","nonce":519008782}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":110}],"filesystems":[{"mdsmap":{"epoch":115,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-16T15:51:47.741329+0000","modified":"2024-09-16T15:51:50.794388+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":9234},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9234":{"gid":9234,"name":"a","rank":0,"incarnation":111,"state":"up:active","state_seq":3,"addr":"172.21.15.123:6835/370478759","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6834","nonce":370478759},{"type":"v1","addr":"172.21.15.123:6835","nonce":370478759}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[27],"metadata_pool":26,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9234,"qdb_cluster":[9234]},"id":13}]} 2024-09-16T15:51:53.514 INFO:teuthology.orchestra.run.smithi123.stderr:dumped fsmap epoch 115 2024-09-16T15:51:53.516 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:53.517+0000 7fba05505640 1 -- 172.21.15.123:0/1906745567 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fb9ec0738e0 msgr2=0x7fb9ec075da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:51:53.516 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:53.517+0000 7fba05505640 1 --2- 172.21.15.123:0/1906745567 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fb9ec0738e0 0x7fb9ec075da0 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7fb9f00af000 tx=0x7fb9f001af20 comp rx=0 tx=0).stop 2024-09-16T15:51:53.517 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:53.517+0000 7fba05505640 1 -- 172.21.15.123:0/1906745567 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fba0015e160 msgr2=0x7fba0015bda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:51:53.517 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:53.517+0000 7fba05505640 1 --2- 172.21.15.123:0/1906745567 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fba0015e160 0x7fba0015bda0 secure :-1 s=READY pgs=1893 cs=0 l=1 rev1=1 crypto rx=0x7fb9f400e6a0 tx=0x7fb9f400eb60 comp rx=0 tx=0).stop 2024-09-16T15:51:53.517 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:53.517+0000 7fba05505640 1 -- 172.21.15.123:0/1906745567 shutdown_connections 2024-09-16T15:51:53.517 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:53.517+0000 7fba05505640 1 --2- 172.21.15.123:0/1906745567 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fb9ec0738e0 0x7fb9ec075da0 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:53.517 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:53.517+0000 7fba05505640 1 --2- 172.21.15.123:0/1906745567 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fba0015e820 0x7fba00157770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:53.517 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:53.517+0000 7fba05505640 1 --2- 172.21.15.123:0/1906745567 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fba0015e160 0x7fba0015bda0 unknown :-1 s=CLOSED pgs=1893 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:53.517 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:53.517+0000 7fba05505640 1 --2- 172.21.15.123:0/1906745567 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fba00074470 0x7fba0014f550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:53.517 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:53.517+0000 7fba05505640 1 -- 172.21.15.123:0/1906745567 >> 172.21.15.123:0/1906745567 conn(0x7fba000cfb10 msgr2=0x7fba000cd680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:51:53.517 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:53.517+0000 7fba05505640 1 -- 172.21.15.123:0/1906745567 shutdown_connections 2024-09-16T15:51:53.517 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:53.517+0000 7fba05505640 1 -- 172.21.15.123:0/1906745567 wait complete. 2024-09-16T15:51:53.935 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 115, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-09-16T15:51:47.741329+0000', 'modified': '2024-09-16T15:51:50.794388+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 9234}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9234': {'gid': 9234, 'name': 'a', 'rank': 0, 'incarnation': 111, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.123:6835/370478759', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.123:6834', 'nonce': 370478759}, {'type': 'v1', 'addr': '172.21.15.123:6835', 'nonce': 370478759}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540719139924082687, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [27], 'metadata_pool': 26, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9234, 'qdb_cluster': [9234]} 2024-09-16T15:51:53.935 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-09-16T15:51:53.935 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2024-09-16T15:51:54.076 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:54.077+0000 7f9ad5ed2640 1 -- 172.21.15.123:0/234688195 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9abc0064e0 msgr2=0x7f9abc00cae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:51:54.076 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:54.077+0000 7f9ad5ed2640 1 --2- 172.21.15.123:0/234688195 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9abc0064e0 0x7f9abc00cae0 secure :-1 s=READY pgs=1895 cs=0 l=1 rev1=1 crypto rx=0x7f9ac80013e0 tx=0x7f9ac801d2c0 comp rx=0 tx=0).stop 2024-09-16T15:51:54.076 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:54.077+0000 7f9ad5ed2640 1 -- 172.21.15.123:0/234688195 shutdown_connections 2024-09-16T15:51:54.076 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:54.077+0000 7f9ad5ed2640 1 --2- 172.21.15.123:0/234688195 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9abc0064e0 0x7f9abc00cae0 unknown :-1 s=CLOSED pgs=1895 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:54.077 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:54.077+0000 7f9ad5ed2640 1 -- 172.21.15.123:0/234688195 >> 172.21.15.123:0/234688195 conn(0x7f9ad00cfb10 msgr2=0x7f9ad00cff20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:51:54.077 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:54.077+0000 7f9ad5ed2640 1 -- 172.21.15.123:0/234688195 shutdown_connections 2024-09-16T15:51:54.077 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:54.077+0000 7f9ad5ed2640 1 -- 172.21.15.123:0/234688195 wait complete. 2024-09-16T15:51:54.079 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:54.077+0000 7f9ad5ed2640 1 Processor -- start 2024-09-16T15:51:54.079 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:54.077+0000 7f9ad5ed2640 1 -- start start 2024-09-16T15:51:54.079 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:54.077+0000 7f9ad5ed2640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f9abc0064e0 0x7f9ad0147870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:51:54.079 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:54.077+0000 7f9ad5ed2640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9ad0147e20 0x7f9ad01f5710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:51:54.079 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:54.077+0000 7f9ad5ed2640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f9ad00b3a70 0x7f9ad00b08d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:51:54.079 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:54.077+0000 7f9ad5ed2640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f9ad00ca320 con 0x7f9ad0147e20 2024-09-16T15:51:54.080 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:54.077+0000 7f9ad5ed2640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f9ad006f1a0 con 0x7f9ad00b3a70 2024-09-16T15:51:54.080 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:54.077+0000 7f9ad5ed2640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f9ad00ca4a0 con 0x7f9abc0064e0 2024-09-16T15:51:54.080 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:54.077+0000 7f9ac6ffd640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9ad0147e20 0x7f9ad01f5710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:51:54.080 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:54.077+0000 7f9ac6ffd640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9ad0147e20 0x7f9ad01f5710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:43532/0 (socket says 172.21.15.123:43532) 2024-09-16T15:51:54.080 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:54.077+0000 7f9ac6ffd640 1 -- 172.21.15.123:0/3961371062 learned_addr learned my addr 172.21.15.123:0/3961371062 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:51:54.080 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:54.077+0000 7f9acf7fe640 1 --2- 172.21.15.123:0/3961371062 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f9abc0064e0 0x7f9ad0147870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:51:54.080 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:54.077+0000 7f9acffff640 1 --2- 172.21.15.123:0/3961371062 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f9ad00b3a70 0x7f9ad00b08d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:51:54.080 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:54.077+0000 7f9ac6ffd640 1 -- 172.21.15.123:0/3961371062 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f9abc0064e0 msgr2=0x7f9ad0147870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:51:54.080 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:54.077+0000 7f9ac6ffd640 1 --2- 172.21.15.123:0/3961371062 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f9abc0064e0 0x7f9ad0147870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:54.080 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:54.077+0000 7f9ac6ffd640 1 -- 172.21.15.123:0/3961371062 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f9ad00b3a70 msgr2=0x7f9ad00b08d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:51:54.080 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:54.077+0000 7f9ac6ffd640 1 --2- 172.21.15.123:0/3961371062 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f9ad00b3a70 0x7f9ad00b08d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:54.080 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:54.077+0000 7f9ac6ffd640 1 -- 172.21.15.123:0/3961371062 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9ad0148350 con 0x7f9ad0147e20 2024-09-16T15:51:54.081 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:54.077+0000 7f9acffff640 1 --2- 172.21.15.123:0/3961371062 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f9ad00b3a70 0x7f9ad00b08d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T15:51:54.081 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:54.077+0000 7f9ac6ffd640 1 --2- 172.21.15.123:0/3961371062 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9ad0147e20 0x7f9ad01f5710 secure :-1 s=READY pgs=1896 cs=0 l=1 rev1=1 crypto rx=0x7f9ab8011780 tx=0x7f9ab8011c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:51:54.081 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:54.077+0000 7f9accff9640 1 -- 172.21.15.123:0/3961371062 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ab800fae0 con 0x7f9ad0147e20 2024-09-16T15:51:54.081 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:54.077+0000 7f9accff9640 1 -- 172.21.15.123:0/3961371062 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f9ab8016070 con 0x7f9ad0147e20 2024-09-16T15:51:54.081 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:54.077+0000 7f9ad5ed2640 1 -- 172.21.15.123:0/3961371062 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9ad01f5c50 con 0x7f9ad0147e20 2024-09-16T15:51:54.081 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:54.077+0000 7f9accff9640 1 -- 172.21.15.123:0/3961371062 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ab8012450 con 0x7f9ad0147e20 2024-09-16T15:51:54.081 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:54.077+0000 7f9ad5ed2640 1 -- 172.21.15.123:0/3961371062 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9ad01f61d0 con 0x7f9ad0147e20 2024-09-16T15:51:54.084 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:54.081+0000 7f9accff9640 1 -- 172.21.15.123:0/3961371062 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 25) ==== 97998+0+0 (secure 0 0 0) 0x7f9ab8018030 con 0x7f9ad0147e20 2024-09-16T15:51:54.084 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:54.081+0000 7f9accff9640 1 --2- 172.21.15.123:0/3961371062 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f9ab0073910 0x7f9ab0075dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:51:54.084 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:54.081+0000 7f9accff9640 1 -- 172.21.15.123:0/3961371062 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(169..169 src has 1..169) ==== 5737+0+0 (secure 0 0 0) 0x7f9ab8092fa0 con 0x7f9ad0147e20 2024-09-16T15:51:54.084 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:54.081+0000 7f9acf7fe640 1 --2- 172.21.15.123:0/3961371062 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f9ab0073910 0x7f9ab0075dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:51:54.084 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:54.081+0000 7f9acf7fe640 1 --2- 172.21.15.123:0/3961371062 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f9ab0073910 0x7f9ab0075dd0 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f9ad0151200 tx=0x7f9ac0006ad0 comp rx=0 tx=0).ready entity=mgr.4150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:51:54.084 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:54.081+0000 7f9ad8fbc000 1 Processor -- start 2024-09-16T15:51:54.085 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:54.081+0000 7f9ad8fbc000 1 -- start start 2024-09-16T15:51:54.085 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:54.081+0000 7f9ad8fbc000 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x557923b889d0 0x557923b88db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:51:54.085 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:54.081+0000 7f9ad8fbc000 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x557923b892f0 0x557923b3c8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:51:54.085 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:54.081+0000 7f9ad8fbc000 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x557923b3ce10 0x557923b3d1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:51:54.085 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:54.081+0000 7f9ad8fbc000 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x557923a5d3f0 con 0x557923b892f0 2024-09-16T15:51:54.085 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:54.081+0000 7f9ad8fbc000 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x55792399ab20 con 0x557923b3ce10 2024-09-16T15:51:54.085 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:54.081+0000 7f9ad8fbc000 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x557923989f50 con 0x557923b889d0 2024-09-16T15:51:54.085 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:54.081+0000 7f9ac6ffd640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x557923b892f0 0x557923b3c8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:51:54.085 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:54.081+0000 7f9ac6ffd640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x557923b892f0 0x557923b3c8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:43540/0 (socket says 172.21.15.123:43540) 2024-09-16T15:51:54.085 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:54.081+0000 7f9ac6ffd640 1 -- 172.21.15.123:0/4162643362 learned_addr learned my addr 172.21.15.123:0/4162643362 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:51:54.085 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:54.081+0000 7f9acf7fe640 1 --2- 172.21.15.123:0/4162643362 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x557923b889d0 0x557923b88db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:51:54.085 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:54.081+0000 7f9acffff640 1 --2- 172.21.15.123:0/4162643362 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x557923b3ce10 0x557923b3d1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:51:54.086 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:54.085+0000 7f9ac6ffd640 1 -- 172.21.15.123:0/4162643362 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x557923b889d0 msgr2=0x557923b88db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T15:51:54.086 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:54.085+0000 7f9ac6ffd640 1 --2- 172.21.15.123:0/4162643362 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x557923b889d0 0x557923b88db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:54.086 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:54.085+0000 7f9ac6ffd640 1 -- 172.21.15.123:0/4162643362 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x557923b3ce10 msgr2=0x557923b3d1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T15:51:54.086 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:54.085+0000 7f9ac6ffd640 1 --2- 172.21.15.123:0/4162643362 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x557923b3ce10 0x557923b3d1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:54.086 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:54.085+0000 7f9ac6ffd640 1 -- 172.21.15.123:0/4162643362 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557923951590 con 0x557923b892f0 2024-09-16T15:51:54.086 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:54.085+0000 7f9acffff640 1 --2- 172.21.15.123:0/4162643362 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x557923b3ce10 0x557923b3d1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T15:51:54.086 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:54.085+0000 7f9ac6ffd640 1 --2- 172.21.15.123:0/4162643362 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x557923b892f0 0x557923b3c8d0 secure :-1 s=READY pgs=1897 cs=0 l=1 rev1=1 crypto rx=0x7f9ab800f730 tx=0x7f9ab805ef50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b984af5786174147 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:51:54.086 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:54.085+0000 7f9ac57fa640 1 -- 172.21.15.123:0/4162643362 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ab805e7e0 con 0x557923b892f0 2024-09-16T15:51:54.086 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:54.085+0000 7f9ac57fa640 1 -- 172.21.15.123:0/4162643362 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f9ab8063e60 con 0x557923b892f0 2024-09-16T15:51:54.086 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:54.085+0000 7f9ac57fa640 1 -- 172.21.15.123:0/4162643362 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ab8066290 con 0x557923b892f0 2024-09-16T15:51:54.086 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:54.085+0000 7f9ad8fbc000 1 -- 172.21.15.123:0/4162643362 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x557923b892f0 msgr2=0x557923b3c8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:51:54.086 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:54.085+0000 7f9ad8fbc000 1 --2- 172.21.15.123:0/4162643362 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x557923b892f0 0x557923b3c8d0 secure :-1 s=READY pgs=1897 cs=0 l=1 rev1=1 crypto rx=0x7f9ab800f730 tx=0x7f9ab805ef50 comp rx=0 tx=0).stop 2024-09-16T15:51:54.086 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:54.085+0000 7f9ad8fbc000 1 -- 172.21.15.123:0/4162643362 shutdown_connections 2024-09-16T15:51:54.087 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:54.085+0000 7f9ad8fbc000 1 --2- 172.21.15.123:0/4162643362 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x557923b3ce10 0x557923b3d1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:54.087 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:54.085+0000 7f9ad8fbc000 1 --2- 172.21.15.123:0/4162643362 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x557923b892f0 0x557923b3c8d0 unknown :-1 s=CLOSED pgs=1897 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:54.087 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:54.085+0000 7f9ad8fbc000 1 --2- 172.21.15.123:0/4162643362 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x557923b889d0 0x557923b88db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:54.087 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:54.085+0000 7f9ad8fbc000 1 -- 172.21.15.123:0/4162643362 >> 172.21.15.123:0/4162643362 conn(0x557923a26130 msgr2=0x557923b6be80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:51:54.087 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:54.085+0000 7f9ad8fbc000 1 -- 172.21.15.123:0/4162643362 shutdown_connections 2024-09-16T15:51:54.087 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:54.085+0000 7f9ad8fbc000 1 -- 172.21.15.123:0/4162643362 wait complete. 2024-09-16T15:51:54.087 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:54.085+0000 7f9ad8fbc000 1 -- start start 2024-09-16T15:51:54.087 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:54.085+0000 7f9ad8fbc000 1 Processor -- start 2024-09-16T15:51:54.090 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:54.085+0000 7f9ad8fbc000 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x557923b3ce10 0x557923b696a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:51:54.090 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:54.085+0000 7f9ad8fbc000 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x557923b889d0 0x557923b69e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:51:54.090 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:54.085+0000 7f9ad8fbc000 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x557923b892f0 0x557923b6a3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:51:54.091 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:54.085+0000 7f9ad8fbc000 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x5579239245d0 con 0x557923b892f0 2024-09-16T15:51:54.091 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:54.085+0000 7f9ad8fbc000 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x55792398dd60 con 0x557923b3ce10 2024-09-16T15:51:54.091 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:54.085+0000 7f9ad8fbc000 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x557923930500 con 0x557923b889d0 2024-09-16T15:51:54.091 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:54.085+0000 7f9ad8fbc000 10 client.0 _ll_register_callbacks cb 0x557923934810 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T15:51:54.091 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:54.085+0000 7f9acffff640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x557923b892f0 0x557923b6a3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:51:54.091 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:54.085+0000 7f9acffff640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x557923b892f0 0x557923b6a3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:43554/0 (socket says 172.21.15.123:43554) 2024-09-16T15:51:54.091 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:54.085+0000 7f9acffff640 1 -- 172.21.15.123:0/1368679761 learned_addr learned my addr 172.21.15.123:0/1368679761 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:51:54.091 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:54.085+0000 7f9ac6ffd640 1 --2- 172.21.15.123:0/1368679761 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x557923b889d0 0x557923b69e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:51:54.091 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:54.085+0000 7f9acf7fe640 1 --2- 172.21.15.123:0/1368679761 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x557923b3ce10 0x557923b696a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:51:54.091 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:54.085+0000 7f9acffff640 1 -- 172.21.15.123:0/1368679761 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x557923b889d0 msgr2=0x557923b69e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T15:51:54.091 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:54.085+0000 7f9acffff640 1 --2- 172.21.15.123:0/1368679761 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x557923b889d0 0x557923b69e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:54.091 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:54.085+0000 7f9acffff640 1 -- 172.21.15.123:0/1368679761 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x557923b3ce10 msgr2=0x557923b696a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T15:51:54.091 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:54.085+0000 7f9acffff640 1 --2- 172.21.15.123:0/1368679761 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x557923b3ce10 0x557923b696a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:54.091 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:54.085+0000 7f9acffff640 1 -- 172.21.15.123:0/1368679761 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557923928170 con 0x557923b892f0 2024-09-16T15:51:54.092 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:54.085+0000 7f9acffff640 1 --2- 172.21.15.123:0/1368679761 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x557923b892f0 0x557923b6a3b0 secure :-1 s=READY pgs=1898 cs=0 l=1 rev1=1 crypto rx=0x7f9ac8002240 tx=0x7f9ac800c2f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1c96a6335a98c45d server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:51:54.092 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:54.085+0000 7f9aae7fc640 10 client.0 ms_handle_connect on v2:172.21.15.123:3300/0 2024-09-16T15:51:54.092 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:54.085+0000 7f9aae7fc640 1 -- 172.21.15.123:0/1368679761 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ac8001460 con 0x557923b892f0 2024-09-16T15:51:54.092 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:54.085+0000 7f9aae7fc640 1 -- 172.21.15.123:0/1368679761 <== mon.0 v2:172.21.15.123:3300/0 2 ==== osd_map(169..169 src has 1..169) ==== 5737+0+0 (secure 0 0 0) 0x7f9ac8001840 con 0x557923b892f0 2024-09-16T15:51:54.092 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:54.085+0000 7f9ad8fbc000 1 -- 172.21.15.123:0/1368679761 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x5579239220f0 con 0x557923b892f0 2024-09-16T15:51:54.092 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:54.085+0000 7f9aae7fc640 1 -- 172.21.15.123:0/1368679761 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_get_version_reply(handle=1 version=115) ==== 24+0+0 (secure 0 0 0) 0x7f9ac80bf070 con 0x557923b892f0 2024-09-16T15:51:54.092 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:54.085+0000 7f9ad8fbc000 10 client.9375 fetch_fsmap learned FSMap version 115 2024-09-16T15:51:54.092 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:54.085+0000 7f9ad8fbc000 1 -- 172.21.15.123:0/1368679761 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,fsmap=115,monmap=2+}) -- 0x557923a0eef0 con 0x557923b892f0 2024-09-16T15:51:54.092 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:54.085+0000 7f9aae7fc640 1 -- 172.21.15.123:0/1368679761 <== mon.0 v2:172.21.15.123:3300/0 4 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x5579239220f0 con 0x557923b892f0 2024-09-16T15:51:54.092 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:54.085+0000 7f9aae7fc640 1 -- 172.21.15.123:0/1368679761 <== mon.0 v2:172.21.15.123:3300/0 5 ==== fsmap(e 115) ==== 3502+0+0 (secure 0 0 0) 0x7f9ac8003980 con 0x557923b892f0 2024-09-16T15:51:54.092 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:54.085+0000 7f9ad8fbc000 10 client.9375 fetch_fsmap finished waiting for FSMap version 115 2024-09-16T15:51:54.092 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:54.085+0000 7f9ad8fbc000 10 client.9375 resolve_mds: resolved name 'a' to daemon mds.a 2024-09-16T15:51:54.092 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:54.085+0000 7f9ad8fbc000 20 client.9375 populate_metadata read hostname 'smithi123' 2024-09-16T15:51:54.092 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:54.085+0000 7f9ad8fbc000 1 --2- 172.21.15.123:0/1368679761 >> [v2:172.21.15.123:6834/370478759,v1:172.21.15.123:6835/370478759] conn(0x557923b6b0f0 0x557923b9ac90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:51:54.092 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:54.085+0000 7f9ad8fbc000 4 client.9375 mds_command: new command op to 9234 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-16T15:51:54.093 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:54.085+0000 7f9ad8fbc000 1 -- 172.21.15.123:0/1368679761 --> [v2:172.21.15.123:6834/370478759,v1:172.21.15.123:6835/370478759] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55792399ca70 con 0x557923b6b0f0 2024-09-16T15:51:54.093 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:54.089+0000 7f9acf7fe640 1 --2- 172.21.15.123:0/1368679761 >> [v2:172.21.15.123:6834/370478759,v1:172.21.15.123:6835/370478759] conn(0x557923b6b0f0 0x557923b9ac90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:51:54.093 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:54.089+0000 7f9aae7fc640 10 client.9375 ms_handle_connect on v2:172.21.15.123:6834/370478759 2024-09-16T15:51:54.093 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:54.089+0000 7f9acf7fe640 1 --2- 172.21.15.123:0/1368679761 >> [v2:172.21.15.123:6834/370478759,v1:172.21.15.123:6835/370478759] conn(0x557923b6b0f0 0x557923b9ac90 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b85d672796cbd238 server_cookie=bdf5b531a4c1c67f in_seq=0 out_seq=0 2024-09-16T15:51:54.093 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:54.089+0000 7f9aae7fc640 1 -- 172.21.15.123:0/1368679761 <== mds.0 v2:172.21.15.123:6834/370478759 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x55792399ca70 con 0x557923b6b0f0 2024-09-16T15:51:54.093 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:54.089+0000 7f9aae7fc640 10 client.9375 handle_command_reply: tid=0 2024-09-16T15:51:54.093 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:54.089+0000 7f9ad8fbc000 1 client.9375 shutdown 2024-09-16T15:51:54.093 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:54.089+0000 7f9ad8fbc000 1 -- 172.21.15.123:0/1368679761 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x557923b892f0 msgr2=0x557923b6a3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:51:54.093 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:54.089+0000 7f9ad8fbc000 1 --2- 172.21.15.123:0/1368679761 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x557923b892f0 0x557923b6a3b0 secure :-1 s=READY pgs=1898 cs=0 l=1 rev1=1 crypto rx=0x7f9ac8002240 tx=0x7f9ac800c2f0 comp rx=0 tx=0).stop 2024-09-16T15:51:54.093 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:54.089+0000 7f9ad8fbc000 1 -- 172.21.15.123:0/1368679761 shutdown_connections 2024-09-16T15:51:54.093 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:54.089+0000 7f9ad8fbc000 1 --2- 172.21.15.123:0/1368679761 >> [v2:172.21.15.123:6834/370478759,v1:172.21.15.123:6835/370478759] conn(0x557923b6b0f0 0x557923b9ac90 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:54.093 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:54.089+0000 7f9ad8fbc000 1 --2- 172.21.15.123:0/1368679761 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x557923b892f0 0x557923b6a3b0 unknown :-1 s=CLOSED pgs=1898 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:54.093 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:54.089+0000 7f9ad8fbc000 1 --2- 172.21.15.123:0/1368679761 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x557923b889d0 0x557923b69e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:54.094 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:54.089+0000 7f9ad8fbc000 1 --2- 172.21.15.123:0/1368679761 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x557923b3ce10 0x557923b696a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:54.094 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:54.089+0000 7f9aae7fc640 0 client.9375 ms_handle_reset on v2:172.21.15.123:6834/370478759 2024-09-16T15:51:54.094 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:54.089+0000 7f9ad8fbc000 1 -- 172.21.15.123:0/1368679761 >> 172.21.15.123:0/1368679761 conn(0x557923a26130 msgr2=0x557923b3bfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:51:54.094 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:54.089+0000 7f9ad8fbc000 1 -- 172.21.15.123:0/1368679761 shutdown_connections 2024-09-16T15:51:54.094 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:54.089+0000 7f9ad8fbc000 1 -- 172.21.15.123:0/1368679761 wait complete. 2024-09-16T15:51:54.094 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:54.089+0000 7f9ad8fbc000 20 client.9375 trim_cache size 0 max 16384 2024-09-16T15:51:54.106 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:54.105+0000 7f9ad8fbc000 1 Processor -- start 2024-09-16T15:51:54.106 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:54.105+0000 7f9ad8fbc000 1 -- start start 2024-09-16T15:51:54.108 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:54.109+0000 7f9ad8fbc000 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x557923b3ce10 0x557923bfabc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:51:54.108 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:54.109+0000 7f9ad8fbc000 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x557923b889d0 0x557923bfb100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:51:54.108 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:54.109+0000 7f9ad8fbc000 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x557923b892f0 0x557923bfb640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:51:54.108 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:54.109+0000 7f9ad8fbc000 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f9ad0202520 con 0x557923b889d0 2024-09-16T15:51:54.108 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:54.109+0000 7f9ad8fbc000 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f9ad0201520 con 0x557923b3ce10 2024-09-16T15:51:54.108 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:54.109+0000 7f9ad8fbc000 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f9ad01f9610 con 0x557923b892f0 2024-09-16T15:51:54.108 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:54.109+0000 7f9ac6ffd640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x557923b889d0 0x557923bfb100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:51:54.108 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:54.109+0000 7f9ac6ffd640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x557923b889d0 0x557923bfb100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:43564/0 (socket says 172.21.15.123:43564) 2024-09-16T15:51:54.108 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:54.109+0000 7f9ac6ffd640 1 -- 172.21.15.123:0/2177976761 learned_addr learned my addr 172.21.15.123:0/2177976761 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:51:54.108 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:54.109+0000 7f9acffff640 1 --2- 172.21.15.123:0/2177976761 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x557923b892f0 0x557923bfb640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:51:54.108 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:54.109+0000 7f9ac6ffd640 1 -- 172.21.15.123:0/2177976761 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x557923b892f0 msgr2=0x557923bfb640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T15:51:54.108 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:54.109+0000 7f9ac6ffd640 1 --2- 172.21.15.123:0/2177976761 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x557923b892f0 0x557923bfb640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:54.109 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:54.109+0000 7f9ac6ffd640 1 -- 172.21.15.123:0/2177976761 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x557923b3ce10 msgr2=0x557923bfabc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T15:51:54.109 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:54.109+0000 7f9ac6ffd640 1 --2- 172.21.15.123:0/2177976761 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x557923b3ce10 0x557923bfabc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:54.109 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:54.109+0000 7f9ac6ffd640 1 -- 172.21.15.123:0/2177976761 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5579239e5d90 con 0x557923b889d0 2024-09-16T15:51:54.109 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:54.109+0000 7f9ac6ffd640 1 --2- 172.21.15.123:0/2177976761 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x557923b889d0 0x557923bfb100 secure :-1 s=READY pgs=1899 cs=0 l=1 rev1=1 crypto rx=0x7f9ab805f310 tx=0x7f9ab8025070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=73c0e025287578ac server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:51:54.109 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:54.109+0000 7f9ac4ff9640 1 -- 172.21.15.123:0/2177976761 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ab805e7e0 con 0x557923b889d0 2024-09-16T15:51:54.109 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:54.109+0000 7f9ac4ff9640 1 -- 172.21.15.123:0/2177976761 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f9ab8063e60 con 0x557923b889d0 2024-09-16T15:51:54.109 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:54.109+0000 7f9ac4ff9640 1 -- 172.21.15.123:0/2177976761 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ab80662b0 con 0x557923b889d0 2024-09-16T15:51:54.113 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:54.109+0000 7f9ad8fbc000 1 -- 172.21.15.123:0/2177976761 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x557923b889d0 msgr2=0x557923bfb100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:51:54.113 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:54.109+0000 7f9ad8fbc000 1 --2- 172.21.15.123:0/2177976761 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x557923b889d0 0x557923bfb100 secure :-1 s=READY pgs=1899 cs=0 l=1 rev1=1 crypto rx=0x7f9ab805f310 tx=0x7f9ab8025070 comp rx=0 tx=0).stop 2024-09-16T15:51:54.113 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:54.109+0000 7f9ad8fbc000 1 -- 172.21.15.123:0/2177976761 shutdown_connections 2024-09-16T15:51:54.114 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:54.109+0000 7f9ad8fbc000 1 --2- 172.21.15.123:0/2177976761 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x557923b892f0 0x557923bfb640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:54.114 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:54.109+0000 7f9ad8fbc000 1 --2- 172.21.15.123:0/2177976761 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x557923b889d0 0x557923bfb100 unknown :-1 s=CLOSED pgs=1899 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:54.114 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:54.109+0000 7f9ad8fbc000 1 --2- 172.21.15.123:0/2177976761 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x557923b3ce10 0x557923bfabc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:54.114 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:54.109+0000 7f9ad8fbc000 1 -- 172.21.15.123:0/2177976761 >> 172.21.15.123:0/2177976761 conn(0x557923a26130 msgr2=0x557923b44ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:51:54.114 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:54.109+0000 7f9ad8fbc000 1 -- 172.21.15.123:0/2177976761 shutdown_connections 2024-09-16T15:51:54.114 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:54.109+0000 7f9ad8fbc000 1 -- 172.21.15.123:0/2177976761 wait complete. 2024-09-16T15:51:54.114 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:54.109+0000 7f9ad8fbc000 1 -- start start 2024-09-16T15:51:54.114 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:54.109+0000 7f9ad8fbc000 1 Processor -- start 2024-09-16T15:51:54.114 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:54.109+0000 7f9ad8fbc000 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x557923b3ce10 0x557923bfe9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:51:54.114 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:54.109+0000 7f9ad8fbc000 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x557923b889d0 0x557923bfef30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:51:54.114 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:54.109+0000 7f9ad8fbc000 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x557923b892f0 0x557923bff470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:51:54.114 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:54.109+0000 7f9ad8fbc000 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x557923b9fce0 con 0x557923b3ce10 2024-09-16T15:51:54.114 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:54.109+0000 7f9ad8fbc000 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x557923b36d80 con 0x557923b889d0 2024-09-16T15:51:54.114 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:54.109+0000 7f9ad8fbc000 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x5579239998f0 con 0x557923b892f0 2024-09-16T15:51:54.115 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:54.109+0000 7f9ad8fbc000 10 client.0 _ll_register_callbacks cb 0x7f9abc004810 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T15:51:54.115 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:54.109+0000 7f9acf7fe640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x557923b3ce10 0x557923bfe9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:51:54.115 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:54.109+0000 7f9acf7fe640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x557923b3ce10 0x557923bfe9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:43566/0 (socket says 172.21.15.123:43566) 2024-09-16T15:51:54.115 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:54.109+0000 7f9acf7fe640 1 -- 172.21.15.123:0/2234112852 learned_addr learned my addr 172.21.15.123:0/2234112852 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:51:54.115 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:54.109+0000 7f9ac6ffd640 1 --2- 172.21.15.123:0/2234112852 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x557923b889d0 0x557923bfef30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:51:54.115 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:54.109+0000 7f9acffff640 1 --2- 172.21.15.123:0/2234112852 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x557923b892f0 0x557923bff470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:51:54.115 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:54.109+0000 7f9acf7fe640 1 -- 172.21.15.123:0/2234112852 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x557923b892f0 msgr2=0x557923bff470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T15:51:54.115 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:54.109+0000 7f9acf7fe640 1 --2- 172.21.15.123:0/2234112852 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x557923b892f0 0x557923bff470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:54.115 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:54.109+0000 7f9acf7fe640 1 -- 172.21.15.123:0/2234112852 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x557923b889d0 msgr2=0x557923bfef30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T15:51:54.115 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:54.109+0000 7f9acf7fe640 1 --2- 172.21.15.123:0/2234112852 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x557923b889d0 0x557923bfef30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:54.115 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:54.109+0000 7f9acf7fe640 1 -- 172.21.15.123:0/2234112852 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557923950a90 con 0x557923b3ce10 2024-09-16T15:51:54.115 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:54.109+0000 7f9ac6ffd640 1 --2- 172.21.15.123:0/2234112852 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x557923b889d0 0x557923bfef30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T15:51:54.115 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:54.109+0000 7f9acf7fe640 1 --2- 172.21.15.123:0/2234112852 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x557923b3ce10 0x557923bfe9f0 secure :-1 s=READY pgs=1900 cs=0 l=1 rev1=1 crypto rx=0x557923a455d0 tx=0x7f9ac0001130 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6836f30abe4253ad server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:51:54.115 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:54.109+0000 7f9aae7fc640 10 client.0 ms_handle_connect on v2:172.21.15.123:3300/0 2024-09-16T15:51:54.115 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:54.113+0000 7f9aae7fc640 1 -- 172.21.15.123:0/2234112852 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ac002c070 con 0x557923b3ce10 2024-09-16T15:51:54.116 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:54.113+0000 7f9aae7fc640 1 -- 172.21.15.123:0/2234112852 <== mon.0 v2:172.21.15.123:3300/0 2 ==== osd_map(169..169 src has 1..169) ==== 5737+0+0 (secure 0 0 0) 0x7f9ac001f830 con 0x557923b3ce10 2024-09-16T15:51:54.116 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:54.113+0000 7f9ad8fbc000 1 -- 172.21.15.123:0/2234112852 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9ab8063e60 con 0x557923b3ce10 2024-09-16T15:51:54.116 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:54.113+0000 7f9aae7fc640 1 -- 172.21.15.123:0/2234112852 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_get_version_reply(handle=1 version=115) ==== 24+0+0 (secure 0 0 0) 0x7f9ac0036050 con 0x557923b3ce10 2024-09-16T15:51:54.116 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:54.113+0000 7f9ad8fbc000 10 client.9381 fetch_fsmap learned FSMap version 115 2024-09-16T15:51:54.116 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:54.113+0000 7f9ad8fbc000 1 -- 172.21.15.123:0/2234112852 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,fsmap=115,monmap=2+}) -- 0x55792392f050 con 0x557923b3ce10 2024-09-16T15:51:54.116 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:54.113+0000 7f9aae7fc640 1 -- 172.21.15.123:0/2234112852 <== mon.0 v2:172.21.15.123:3300/0 4 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f9ab8063e60 con 0x557923b3ce10 2024-09-16T15:51:54.116 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:54.113+0000 7f9aae7fc640 1 -- 172.21.15.123:0/2234112852 <== mon.0 v2:172.21.15.123:3300/0 5 ==== fsmap(e 115) ==== 3502+0+0 (secure 0 0 0) 0x7f9ac0039020 con 0x557923b3ce10 2024-09-16T15:51:54.116 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:54.113+0000 7f9ad8fbc000 10 client.9381 fetch_fsmap finished waiting for FSMap version 115 2024-09-16T15:51:54.116 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:54.113+0000 7f9ad8fbc000 10 client.9381 resolve_mds: resolved name 'a' to daemon mds.a 2024-09-16T15:51:54.116 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:54.113+0000 7f9ad8fbc000 20 client.9381 populate_metadata read hostname 'smithi123' 2024-09-16T15:51:54.116 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:54.113+0000 7f9ad8fbc000 1 --2- 172.21.15.123:0/2234112852 >> [v2:172.21.15.123:6834/370478759,v1:172.21.15.123:6835/370478759] conn(0x557923b9e400 0x557923c07c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:51:54.116 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:54.113+0000 7f9ad8fbc000 4 client.9381 mds_command: new command op to 9234 tid=0 multi_id=0 [{"prefix": "status"}] 2024-09-16T15:51:54.117 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:54.113+0000 7f9ad8fbc000 1 -- 172.21.15.123:0/2234112852 --> [v2:172.21.15.123:6834/370478759,v1:172.21.15.123:6835/370478759] -- command(tid 0: {"prefix": "status"}) -- 0x557923998520 con 0x557923b9e400 2024-09-16T15:51:54.117 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:54.113+0000 7f9ac6ffd640 1 --2- 172.21.15.123:0/2234112852 >> [v2:172.21.15.123:6834/370478759,v1:172.21.15.123:6835/370478759] conn(0x557923b9e400 0x557923c07c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:51:54.117 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:54.113+0000 7f9aae7fc640 10 client.9381 ms_handle_connect on v2:172.21.15.123:6834/370478759 2024-09-16T15:51:54.117 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:54.113+0000 7f9ac6ffd640 1 --2- 172.21.15.123:0/2234112852 >> [v2:172.21.15.123:6834/370478759,v1:172.21.15.123:6835/370478759] conn(0x557923b9e400 0x557923c07c50 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=58cbcda6f52748f4 server_cookie=87d9f012cadb33bd in_seq=0 out_seq=0 2024-09-16T15:51:54.117 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:54.113+0000 7f9aae7fc640 1 -- 172.21.15.123:0/2234112852 <== mds.0 v2:172.21.15.123:6834/370478759 1 ==== command_reply(tid 0: 0 ) ==== 8+0+327 (crc 0 0 0) 0x557923998520 con 0x557923b9e400 2024-09-16T15:51:54.117 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:54.113+0000 7f9aae7fc640 10 client.9381 handle_command_reply: tid=0 2024-09-16T15:51:54.117 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:54.113+0000 7f9ad8fbc000 1 client.9381 shutdown 2024-09-16T15:51:54.117 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:54.113+0000 7f9ad8fbc000 1 -- 172.21.15.123:0/2234112852 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x557923b3ce10 msgr2=0x557923bfe9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:51:54.117 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:54.113+0000 7f9ad8fbc000 1 --2- 172.21.15.123:0/2234112852 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x557923b3ce10 0x557923bfe9f0 secure :-1 s=READY pgs=1900 cs=0 l=1 rev1=1 crypto rx=0x557923a455d0 tx=0x7f9ac0001130 comp rx=0 tx=0).stop 2024-09-16T15:51:54.117 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:54.113+0000 7f9ad8fbc000 1 -- 172.21.15.123:0/2234112852 shutdown_connections 2024-09-16T15:51:54.117 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:54.113+0000 7f9ad8fbc000 1 --2- 172.21.15.123:0/2234112852 >> [v2:172.21.15.123:6834/370478759,v1:172.21.15.123:6835/370478759] conn(0x557923b9e400 0x557923c07c50 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:54.117 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:54.113+0000 7f9ad8fbc000 1 --2- 172.21.15.123:0/2234112852 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x557923b892f0 0x557923bff470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:54.117 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:54.113+0000 7f9ad8fbc000 1 --2- 172.21.15.123:0/2234112852 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x557923b889d0 0x557923bfef30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:54.117 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:54.113+0000 7f9aae7fc640 0 client.9381 ms_handle_reset on v2:172.21.15.123:6834/370478759 2024-09-16T15:51:54.117 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:54.113+0000 7f9ad8fbc000 1 --2- 172.21.15.123:0/2234112852 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x557923b3ce10 0x557923bfe9f0 unknown :-1 s=CLOSED pgs=1900 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:54.118 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:54.113+0000 7f9ad8fbc000 1 -- 172.21.15.123:0/2234112852 >> 172.21.15.123:0/2234112852 conn(0x557923a26130 msgr2=0x557923b875b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:51:54.118 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:54.113+0000 7f9ad8fbc000 1 -- 172.21.15.123:0/2234112852 shutdown_connections 2024-09-16T15:51:54.118 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:54.113+0000 7f9ad8fbc000 1 -- 172.21.15.123:0/2234112852 wait complete. 2024-09-16T15:51:54.118 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:54.113+0000 7f9ad8fbc000 20 client.9381 trim_cache size 0 max 16384 2024-09-16T15:51:54.118 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2024-09-16T15:51:54.118 INFO:teuthology.orchestra.run.smithi123.stdout: "cluster_fsid": "56077f89-18c8-486f-ae7e-eecc0bb7822c", 2024-09-16T15:51:54.118 INFO:teuthology.orchestra.run.smithi123.stdout: "whoami": 0, 2024-09-16T15:51:54.118 INFO:teuthology.orchestra.run.smithi123.stdout: "id": 9234, 2024-09-16T15:51:54.118 INFO:teuthology.orchestra.run.smithi123.stdout: "want_state": "up:active", 2024-09-16T15:51:54.118 INFO:teuthology.orchestra.run.smithi123.stdout: "state": "up:active", 2024-09-16T15:51:54.118 INFO:teuthology.orchestra.run.smithi123.stdout: "fs_name": "cephfs", 2024-09-16T15:51:54.119 INFO:teuthology.orchestra.run.smithi123.stdout: "rank_uptime": 5.691666637, 2024-09-16T15:51:54.119 INFO:teuthology.orchestra.run.smithi123.stdout: "mdsmap_epoch": 115, 2024-09-16T15:51:54.119 INFO:teuthology.orchestra.run.smithi123.stdout: "osdmap_epoch": 169, 2024-09-16T15:51:54.119 INFO:teuthology.orchestra.run.smithi123.stdout: "osdmap_epoch_barrier": 169, 2024-09-16T15:51:54.119 INFO:teuthology.orchestra.run.smithi123.stdout: "uptime": 10.303884056999999 2024-09-16T15:51:54.119 INFO:teuthology.orchestra.run.smithi123.stdout:} 2024-09-16T15:51:54.120 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:54.113+0000 7f9ad5ed2640 1 -- 172.21.15.123:0/3961371062 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f9ab0073910 msgr2=0x7f9ab0075dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:51:54.120 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:54.113+0000 7f9ad5ed2640 1 --2- 172.21.15.123:0/3961371062 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f9ab0073910 0x7f9ab0075dd0 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f9ad0151200 tx=0x7f9ac0006ad0 comp rx=0 tx=0).stop 2024-09-16T15:51:54.120 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:54.113+0000 7f9ad5ed2640 1 -- 172.21.15.123:0/3961371062 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9ad0147e20 msgr2=0x7f9ad01f5710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:51:54.120 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:54.113+0000 7f9ad5ed2640 1 --2- 172.21.15.123:0/3961371062 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9ad0147e20 0x7f9ad01f5710 secure :-1 s=READY pgs=1896 cs=0 l=1 rev1=1 crypto rx=0x7f9ab8011780 tx=0x7f9ab8011c40 comp rx=0 tx=0).stop 2024-09-16T15:51:54.120 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:54.113+0000 7f9ad5ed2640 1 -- 172.21.15.123:0/3961371062 shutdown_connections 2024-09-16T15:51:54.120 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:54.113+0000 7f9ad5ed2640 1 --2- 172.21.15.123:0/3961371062 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f9ab0073910 0x7f9ab0075dd0 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:54.120 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:54.113+0000 7f9ad5ed2640 1 --2- 172.21.15.123:0/3961371062 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f9ad00b3a70 0x7f9ad00b08d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:54.120 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:54.113+0000 7f9ad5ed2640 1 --2- 172.21.15.123:0/3961371062 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9ad0147e20 0x7f9ad01f5710 unknown :-1 s=CLOSED pgs=1896 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:54.120 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:54.113+0000 7f9ad5ed2640 1 --2- 172.21.15.123:0/3961371062 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f9abc0064e0 0x7f9ad0147870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:54.120 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:54.113+0000 7f9ad5ed2640 1 -- 172.21.15.123:0/3961371062 >> 172.21.15.123:0/3961371062 conn(0x7f9ad00cfb10 msgr2=0x7f9ad01573e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:51:54.120 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:54.113+0000 7f9ad5ed2640 1 -- 172.21.15.123:0/3961371062 shutdown_connections 2024-09-16T15:51:54.120 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:54.113+0000 7f9ad5ed2640 1 -- 172.21.15.123:0/3961371062 wait complete. 2024-09-16T15:51:54.128 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-09-16T15:51:54.239 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:54.241+0000 7feaf1122640 1 -- 172.21.15.123:0/654831884 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fead4003a10 msgr2=0x7fead4003e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:51:54.239 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:54.241+0000 7feaf1122640 1 --2- 172.21.15.123:0/654831884 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fead4003a10 0x7fead4003e10 secure :-1 s=READY pgs=1902 cs=0 l=1 rev1=1 crypto rx=0x7feadc00bb10 tx=0x7feadc01cde0 comp rx=0 tx=0).stop 2024-09-16T15:51:54.239 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:54.241+0000 7feaf1122640 1 -- 172.21.15.123:0/654831884 shutdown_connections 2024-09-16T15:51:54.239 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:54.241+0000 7feaf1122640 1 --2- 172.21.15.123:0/654831884 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fead4003a10 0x7fead4003e10 unknown :-1 s=CLOSED pgs=1902 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:54.239 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:54.241+0000 7feaf1122640 1 -- 172.21.15.123:0/654831884 >> 172.21.15.123:0/654831884 conn(0x7feaec0732a0 msgr2=0x7feaec074150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:51:54.240 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:54.241+0000 7feaf1122640 1 -- 172.21.15.123:0/654831884 shutdown_connections 2024-09-16T15:51:54.240 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:54.241+0000 7feaf1122640 1 -- 172.21.15.123:0/654831884 wait complete. 2024-09-16T15:51:54.240 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:54.241+0000 7feaf1122640 1 Processor -- start 2024-09-16T15:51:54.240 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:54.241+0000 7feaf1122640 1 -- start start 2024-09-16T15:51:54.242 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:54.241+0000 7feaf1122640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fead4003a10 0x7feaec099300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:51:54.242 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:54.241+0000 7feaf1122640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7feaec1f9ee0 0x7feaec0cf490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:51:54.242 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:54.241+0000 7feaf1122640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7feaec1fa360 0x7feaec0cf9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:51:54.242 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:54.241+0000 7feaf1122640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7feaec156260 con 0x7feaec1fa360 2024-09-16T15:51:54.242 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:54.241+0000 7feaf1122640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7feaec1560e0 con 0x7fead4003a10 2024-09-16T15:51:54.243 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:54.241+0000 7feaf1122640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7feaec1563e0 con 0x7feaec1f9ee0 2024-09-16T15:51:54.243 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:54.241+0000 7feaeb59e640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7feaec1fa360 0x7feaec0cf9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:51:54.243 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:54.241+0000 7feaeb59e640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7feaec1fa360 0x7feaec0cf9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:43592/0 (socket says 172.21.15.123:43592) 2024-09-16T15:51:54.243 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:54.241+0000 7feaeb59e640 1 -- 172.21.15.123:0/1917601375 learned_addr learned my addr 172.21.15.123:0/1917601375 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:51:54.243 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:54.241+0000 7feaea59c640 1 --2- 172.21.15.123:0/1917601375 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7feaec1f9ee0 0x7feaec0cf490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:51:54.243 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:54.241+0000 7feaead9d640 1 --2- 172.21.15.123:0/1917601375 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fead4003a10 0x7feaec099300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:51:54.243 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:54.241+0000 7feaeb59e640 1 -- 172.21.15.123:0/1917601375 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7feaec1f9ee0 msgr2=0x7feaec0cf490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:51:54.243 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:54.241+0000 7feaeb59e640 1 --2- 172.21.15.123:0/1917601375 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7feaec1f9ee0 0x7feaec0cf490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:54.243 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:54.241+0000 7feaeb59e640 1 -- 172.21.15.123:0/1917601375 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fead4003a10 msgr2=0x7feaec099300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:51:54.243 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:54.241+0000 7feaeb59e640 1 --2- 172.21.15.123:0/1917601375 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fead4003a10 0x7feaec099300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:54.243 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:54.241+0000 7feaeb59e640 1 -- 172.21.15.123:0/1917601375 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feaec09bbf0 con 0x7feaec1fa360 2024-09-16T15:51:54.243 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:54.241+0000 7feaead9d640 1 --2- 172.21.15.123:0/1917601375 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fead4003a10 0x7feaec099300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T15:51:54.243 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:54.241+0000 7feaeb59e640 1 --2- 172.21.15.123:0/1917601375 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7feaec1fa360 0x7feaec0cf9d0 secure :-1 s=READY pgs=1903 cs=0 l=1 rev1=1 crypto rx=0x7feaec0d0f20 tx=0x7feae400cf00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:51:54.244 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:54.241+0000 7fead37fe640 1 -- 172.21.15.123:0/1917601375 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feae4017070 con 0x7feaec1fa360 2024-09-16T15:51:54.244 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:54.241+0000 7fead37fe640 1 -- 172.21.15.123:0/1917601375 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7feae40050d0 con 0x7feaec1fa360 2024-09-16T15:51:54.244 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:54.241+0000 7feaf1122640 1 -- 172.21.15.123:0/1917601375 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feaec09a4e0 con 0x7feaec1fa360 2024-09-16T15:51:54.244 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:54.241+0000 7fead37fe640 1 -- 172.21.15.123:0/1917601375 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feae4005520 con 0x7feaec1fa360 2024-09-16T15:51:54.244 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:54.241+0000 7feaf1122640 1 -- 172.21.15.123:0/1917601375 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7feaec09c170 con 0x7feaec1fa360 2024-09-16T15:51:54.245 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:54.245+0000 7feaf1122640 1 -- 172.21.15.123:0/1917601375 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7feaec0cdf10 con 0x7feaec1fa360 2024-09-16T15:51:54.245 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:54.245+0000 7fead37fe640 1 -- 172.21.15.123:0/1917601375 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 25) ==== 97998+0+0 (secure 0 0 0) 0x7feae401a070 con 0x7feaec1fa360 2024-09-16T15:51:54.245 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:54.245+0000 7fead37fe640 1 --2- 172.21.15.123:0/1917601375 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fead80738e0 0x7fead8075da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:51:54.245 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:54.245+0000 7feaead9d640 1 --2- 172.21.15.123:0/1917601375 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fead80738e0 0x7fead8075da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:51:54.245 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:54.245+0000 7fead37fe640 1 -- 172.21.15.123:0/1917601375 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(169..169 src has 1..169) ==== 5737+0+0 (secure 0 0 0) 0x7feae4091f40 con 0x7feaec1fa360 2024-09-16T15:51:54.245 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:54.245+0000 7feaead9d640 1 --2- 172.21.15.123:0/1917601375 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fead80738e0 0x7fead8075da0 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7feadc0b0000 tx=0x7feadc019b80 comp rx=0 tx=0).ready entity=mgr.4150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:51:54.248 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:54.249+0000 7fead37fe640 1 -- 172.21.15.123:0/1917601375 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7feae405ea20 con 0x7feaec1fa360 2024-09-16T15:51:54.322 INFO:tasks.ceph.mds.a.smithi123.stderr:2024-09-16T15:51:54.325+0000 7f6513c1d640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-09-16T15:51:54.442 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:54.441+0000 7feaf1122640 1 -- 172.21.15.123:0/1917601375 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7feaec0ae2e0 con 0x7feaec1fa360 2024-09-16T15:51:54.443 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:54.445+0000 7fead37fe640 1 -- 172.21.15.123:0/1917601375 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v115) ==== 53+0+73 (secure 0 0 0) 0x7feae405e320 con 0x7feaec1fa360 2024-09-16T15:51:54.443 INFO:teuthology.orchestra.run.smithi123.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-09-16T15:51:54.446 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:54.445+0000 7feaf1122640 1 -- 172.21.15.123:0/1917601375 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fead80738e0 msgr2=0x7fead8075da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:51:54.446 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:54.445+0000 7feaf1122640 1 --2- 172.21.15.123:0/1917601375 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fead80738e0 0x7fead8075da0 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7feadc0b0000 tx=0x7feadc019b80 comp rx=0 tx=0).stop 2024-09-16T15:51:54.446 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:54.445+0000 7feaf1122640 1 -- 172.21.15.123:0/1917601375 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7feaec1fa360 msgr2=0x7feaec0cf9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:51:54.446 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:54.445+0000 7feaf1122640 1 --2- 172.21.15.123:0/1917601375 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7feaec1fa360 0x7feaec0cf9d0 secure :-1 s=READY pgs=1903 cs=0 l=1 rev1=1 crypto rx=0x7feaec0d0f20 tx=0x7feae400cf00 comp rx=0 tx=0).stop 2024-09-16T15:51:54.446 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:54.445+0000 7feaf1122640 1 -- 172.21.15.123:0/1917601375 shutdown_connections 2024-09-16T15:51:54.446 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:54.445+0000 7feaf1122640 1 --2- 172.21.15.123:0/1917601375 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fead80738e0 0x7fead8075da0 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:54.446 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:54.445+0000 7feaf1122640 1 --2- 172.21.15.123:0/1917601375 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7feaec1fa360 0x7feaec0cf9d0 unknown :-1 s=CLOSED pgs=1903 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:54.446 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:54.445+0000 7feaf1122640 1 --2- 172.21.15.123:0/1917601375 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7feaec1f9ee0 0x7feaec0cf490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:54.446 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:54.445+0000 7feaf1122640 1 --2- 172.21.15.123:0/1917601375 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fead4003a10 0x7feaec099300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:54.446 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:54.445+0000 7feaf1122640 1 -- 172.21.15.123:0/1917601375 >> 172.21.15.123:0/1917601375 conn(0x7feaec0732a0 msgr2=0x7feaec0c6410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:51:54.447 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:54.445+0000 7feaf1122640 1 -- 172.21.15.123:0/1917601375 shutdown_connections 2024-09-16T15:51:54.447 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:54.445+0000 7feaf1122640 1 -- 172.21.15.123:0/1917601375 wait complete. 2024-09-16T15:51:54.462 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-09-16T15:51:54.462 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-09-16T15:51:54.462 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi123.front.sepia.ceph.com 2024-09-16T15:51:54.462 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-09-16T15:51:54.462 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-09-16T15:51:54.462 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-09-16T15:51:54.462 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-09-16T15:51:54.462 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-09-16T15:51:54.463 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-09-16T15:51:54.463 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:51:54.463 DEBUG:teuthology.orchestra.run.smithi123:> ip addr 2024-09-16T15:51:54.468 INFO:teuthology.orchestra.run.smithi123.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-09-16T15:51:54.469 INFO:teuthology.orchestra.run.smithi123.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-09-16T15:51:54.469 INFO:teuthology.orchestra.run.smithi123.stdout: inet 127.0.0.1/8 scope host lo 2024-09-16T15:51:54.469 INFO:teuthology.orchestra.run.smithi123.stdout: valid_lft forever preferred_lft forever 2024-09-16T15:51:54.469 INFO:teuthology.orchestra.run.smithi123.stdout: inet6 ::1/128 scope host 2024-09-16T15:51:54.469 INFO:teuthology.orchestra.run.smithi123.stdout: valid_lft forever preferred_lft forever 2024-09-16T15:51:54.469 INFO:teuthology.orchestra.run.smithi123.stdout:2: eno1: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-09-16T15:51:54.469 INFO:teuthology.orchestra.run.smithi123.stdout: link/ether 0c:c4:7a:d8:da:6c brd ff:ff:ff:ff:ff:ff 2024-09-16T15:51:54.469 INFO:teuthology.orchestra.run.smithi123.stdout: altname enp7s0f0 2024-09-16T15:51:54.469 INFO:teuthology.orchestra.run.smithi123.stdout:3: eno2: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-09-16T15:51:54.469 INFO:teuthology.orchestra.run.smithi123.stdout: link/ether 0c:c4:7a:d8:da:6d brd ff:ff:ff:ff:ff:ff 2024-09-16T15:51:54.469 INFO:teuthology.orchestra.run.smithi123.stdout: altname enp7s0f1 2024-09-16T15:51:54.469 INFO:teuthology.orchestra.run.smithi123.stdout:4: enp3s0f0: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-09-16T15:51:54.469 INFO:teuthology.orchestra.run.smithi123.stdout: link/ether 0c:c4:7a:8f:d3:f0 brd ff:ff:ff:ff:ff:ff 2024-09-16T15:51:54.469 INFO:teuthology.orchestra.run.smithi123.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-09-16T15:51:54.470 INFO:teuthology.orchestra.run.smithi123.stdout: link/ether 0c:c4:7a:8f:d3:f1 brd ff:ff:ff:ff:ff:ff 2024-09-16T15:51:54.470 INFO:teuthology.orchestra.run.smithi123.stdout: inet 172.21.15.123/20 metric 100 brd 172.21.15.255 scope global dynamic enp3s0f1 2024-09-16T15:51:54.470 INFO:teuthology.orchestra.run.smithi123.stdout: valid_lft 41954sec preferred_lft 41954sec 2024-09-16T15:51:54.470 INFO:teuthology.orchestra.run.smithi123.stdout: inet6 fe80::ec4:7aff:fe8f:d3f1/64 scope link 2024-09-16T15:51:54.470 INFO:teuthology.orchestra.run.smithi123.stdout: valid_lft forever preferred_lft forever 2024-09-16T15:51:54.470 INFO:teuthology.orchestra.run.smithi123.stdout:6: docker0: mtu 1500 qdisc noqueue state DOWN group default 2024-09-16T15:51:54.470 INFO:teuthology.orchestra.run.smithi123.stdout: link/ether 02:42:dd:82:a2:6e brd ff:ff:ff:ff:ff:ff 2024-09-16T15:51:54.470 INFO:teuthology.orchestra.run.smithi123.stdout: inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0 2024-09-16T15:51:54.470 INFO:teuthology.orchestra.run.smithi123.stdout: valid_lft forever preferred_lft forever 2024-09-16T15:51:54.470 INFO:teuthology.orchestra.run.smithi123.stdout:7: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-09-16T15:51:54.470 INFO:teuthology.orchestra.run.smithi123.stdout: link/ether 5e:1f:84:6d:79:ae brd ff:ff:ff:ff:ff:ff 2024-09-16T15:51:54.470 INFO:teuthology.orchestra.run.smithi123.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-09-16T15:51:54.470 INFO:teuthology.orchestra.run.smithi123.stdout: valid_lft forever preferred_lft forever 2024-09-16T15:51:54.470 INFO:teuthology.orchestra.run.smithi123.stdout: inet6 fe80::5c1f:84ff:fe6d:79ae/64 scope link 2024-09-16T15:51:54.470 INFO:teuthology.orchestra.run.smithi123.stdout: valid_lft forever preferred_lft forever 2024-09-16T15:51:54.471 INFO:teuthology.orchestra.run.smithi123.stdout:8: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-09-16T15:51:54.471 INFO:teuthology.orchestra.run.smithi123.stdout: link/ether 0e:5c:f6:1f:82:2f brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-09-16T15:51:54.471 INFO:teuthology.orchestra.run.smithi123.stdout: inet6 fe80::c5c:f6ff:fe1f:822f/64 scope link 2024-09-16T15:51:54.471 INFO:teuthology.orchestra.run.smithi123.stdout: valid_lft forever preferred_lft forever 2024-09-16T15:51:54.471 INFO:teuthology.orchestra.run.smithi123.stdout:9: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-09-16T15:51:54.471 INFO:teuthology.orchestra.run.smithi123.stdout: link/ether c6:a6:3d:69:d0:db brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-09-16T15:51:54.471 INFO:teuthology.orchestra.run.smithi123.stdout: inet6 fe80::c4a6:3dff:fe69:d0db/64 scope link 2024-09-16T15:51:54.471 INFO:teuthology.orchestra.run.smithi123.stdout: valid_lft forever preferred_lft forever 2024-09-16T15:51:54.471 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:51:54.471 DEBUG:teuthology.orchestra.run.smithi123:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-09-16T15:51:54.521 INFO:teuthology.orchestra.run.smithi123.stdout:1 2024-09-16T15:51:54.522 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:51:54.523 DEBUG:teuthology.orchestra.run.smithi123:> route 2024-09-16T15:51:54.573 INFO:teuthology.orchestra.run.smithi123.stdout:Kernel IP routing table 2024-09-16T15:51:54.574 INFO:teuthology.orchestra.run.smithi123.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-09-16T15:51:54.574 INFO:teuthology.orchestra.run.smithi123.stdout:default _gateway 0.0.0.0 UG 0 0 0 enp3s0f1 2024-09-16T15:51:54.574 INFO:teuthology.orchestra.run.smithi123.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-09-16T15:51:54.574 INFO:teuthology.orchestra.run.smithi123.stdout:158.69.67.47 _gateway 255.255.255.255 UGH 100 0 0 enp3s0f1 2024-09-16T15:51:54.574 INFO:teuthology.orchestra.run.smithi123.stdout:172.17.0.0 0.0.0.0 255.255.0.0 U 0 0 0 docker0 2024-09-16T15:51:54.574 INFO:teuthology.orchestra.run.smithi123.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-09-16T15:51:54.574 INFO:teuthology.orchestra.run.smithi123.stdout:ns1.front.sepia 0.0.0.0 255.255.255.255 UH 100 0 0 enp3s0f1 2024-09-16T15:51:54.574 INFO:teuthology.orchestra.run.smithi123.stdout:172.21.0.2 0.0.0.0 255.255.255.255 UH 100 0 0 enp3s0f1 2024-09-16T15:51:54.574 INFO:teuthology.orchestra.run.smithi123.stdout:_gateway 0.0.0.0 255.255.255.255 UH 100 0 0 enp3s0f1 2024-09-16T15:51:54.574 INFO:teuthology.orchestra.run.smithi123.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-09-16T15:51:54.574 DEBUG:teuthology.orchestra.run.smithi123:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-09-16T15:51:54.575 DEBUG:teuthology.orchestra.run.smithi123:> set -e 2024-09-16T15:51:54.575 DEBUG:teuthology.orchestra.run.smithi123:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-09-16T15:51:54.575 DEBUG:teuthology.orchestra.run.smithi123:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-09-16T15:51:54.575 DEBUG:teuthology.orchestra.run.smithi123:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-09-16T15:51:54.575 DEBUG:teuthology.orchestra.run.smithi123:> ') 2024-09-16T15:51:54.658 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:54 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-09-16T15:51:54.685 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:54 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-16T15:51:54.691 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:51:54.691 DEBUG:teuthology.orchestra.run.smithi123:> ip netns list 2024-09-16T15:51:54.739 INFO:teuthology.orchestra.run.smithi123.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-09-16T15:51:54.739 INFO:teuthology.orchestra.run.smithi123.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-09-16T15:51:54.740 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-09-16T15:51:54.740 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:51:54.740 DEBUG:teuthology.orchestra.run.smithi123:> sudo ip link set brx.0 up 2024-09-16T15:51:54.799 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2024-09-16T15:51:54.799 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-09-16T15:51:54.800 DEBUG:teuthology.orchestra.run.smithi123:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-09-16T15:51:54.848 INFO:teuthology.orchestra.run.smithi123.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-09-16T15:51:54.848 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-09-16T15:51:54.848 DEBUG:teuthology.orchestra.run.smithi123:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-09-16T15:51:54.895 DEBUG:teuthology.orchestra.run.smithi123:> sudo modprobe fuse 2024-09-16T15:51:54.956 DEBUG:teuthology.orchestra.run.smithi123:> cat /proc/self/mounts | awk '{print $2}' 2024-09-16T15:51:55.010 INFO:teuthology.orchestra.run.smithi123.stdout:/sys 2024-09-16T15:51:55.010 INFO:teuthology.orchestra.run.smithi123.stdout:/proc 2024-09-16T15:51:55.010 INFO:teuthology.orchestra.run.smithi123.stdout:/dev 2024-09-16T15:51:55.010 INFO:teuthology.orchestra.run.smithi123.stdout:/dev/pts 2024-09-16T15:51:55.010 INFO:teuthology.orchestra.run.smithi123.stdout:/run 2024-09-16T15:51:55.010 INFO:teuthology.orchestra.run.smithi123.stdout:/ 2024-09-16T15:51:55.010 INFO:teuthology.orchestra.run.smithi123.stdout:/sys/kernel/security 2024-09-16T15:51:55.010 INFO:teuthology.orchestra.run.smithi123.stdout:/dev/shm 2024-09-16T15:51:55.010 INFO:teuthology.orchestra.run.smithi123.stdout:/run/lock 2024-09-16T15:51:55.010 INFO:teuthology.orchestra.run.smithi123.stdout:/sys/fs/cgroup 2024-09-16T15:51:55.010 INFO:teuthology.orchestra.run.smithi123.stdout:/sys/fs/pstore 2024-09-16T15:51:55.011 INFO:teuthology.orchestra.run.smithi123.stdout:/sys/fs/bpf 2024-09-16T15:51:55.011 INFO:teuthology.orchestra.run.smithi123.stdout:/proc/sys/fs/binfmt_misc 2024-09-16T15:51:55.011 INFO:teuthology.orchestra.run.smithi123.stdout:/dev/hugepages 2024-09-16T15:51:55.011 INFO:teuthology.orchestra.run.smithi123.stdout:/dev/mqueue 2024-09-16T15:51:55.011 INFO:teuthology.orchestra.run.smithi123.stdout:/sys/kernel/debug 2024-09-16T15:51:55.011 INFO:teuthology.orchestra.run.smithi123.stdout:/sys/kernel/tracing 2024-09-16T15:51:55.011 INFO:teuthology.orchestra.run.smithi123.stdout:/proc/fs/nfsd 2024-09-16T15:51:55.011 INFO:teuthology.orchestra.run.smithi123.stdout:/sys/fs/fuse/connections 2024-09-16T15:51:55.011 INFO:teuthology.orchestra.run.smithi123.stdout:/sys/kernel/config 2024-09-16T15:51:55.011 INFO:teuthology.orchestra.run.smithi123.stdout:/run/credentials/systemd-sysusers.service 2024-09-16T15:51:55.011 INFO:teuthology.orchestra.run.smithi123.stdout:/run/rpc_pipefs 2024-09-16T15:51:55.011 INFO:teuthology.orchestra.run.smithi123.stdout:/run/user/1000 2024-09-16T15:51:55.011 INFO:teuthology.orchestra.run.smithi123.stdout:/var/lib/ceph 2024-09-16T15:51:55.011 INFO:teuthology.orchestra.run.smithi123.stdout:/var/lib/ceph/osd/ceph-0 2024-09-16T15:51:55.012 INFO:teuthology.orchestra.run.smithi123.stdout:/var/lib/ceph/osd/ceph-1 2024-09-16T15:51:55.012 INFO:teuthology.orchestra.run.smithi123.stdout:/var/lib/ceph/osd/ceph-2 2024-09-16T15:51:55.012 INFO:teuthology.orchestra.run.smithi123.stdout:/var/lib/ceph/osd/ceph-3 2024-09-16T15:51:55.012 INFO:teuthology.orchestra.run.smithi123.stdout:/run/netns 2024-09-16T15:51:55.012 INFO:teuthology.orchestra.run.smithi123.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-09-16T15:51:55.012 INFO:teuthology.orchestra.run.smithi123.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-09-16T15:51:55.012 INFO:teuthology.orchestra.run.smithi123.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-09-16T15:51:55.012 INFO:teuthology.orchestra.run.smithi123.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-09-16T15:51:55.013 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:51:55.013 DEBUG:teuthology.orchestra.run.smithi123:> ls /sys/fs/fuse/connections 2024-09-16T15:51:55.059 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-09-16T15:51:55.059 DEBUG:teuthology.orchestra.run.smithi123:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2024-09-16T15:51:55.108 DEBUG:teuthology.orchestra.run.smithi123:> sudo modprobe fuse 2024-09-16T15:51:55.121 DEBUG:teuthology.orchestra.run.smithi123:> cat /proc/self/mounts | awk '{print $2}' 2024-09-16T15:51:55.171 INFO:teuthology.orchestra.run.smithi123.stdout:/sys 2024-09-16T15:51:55.171 INFO:teuthology.orchestra.run.smithi123.stdout:/proc 2024-09-16T15:51:55.172 INFO:teuthology.orchestra.run.smithi123.stdout:/dev 2024-09-16T15:51:55.172 INFO:teuthology.orchestra.run.smithi123.stdout:/dev/pts 2024-09-16T15:51:55.172 INFO:teuthology.orchestra.run.smithi123.stdout:/run 2024-09-16T15:51:55.172 INFO:teuthology.orchestra.run.smithi123.stdout:/ 2024-09-16T15:51:55.172 INFO:teuthology.orchestra.run.smithi123.stdout:/sys/kernel/security 2024-09-16T15:51:55.172 INFO:teuthology.orchestra.run.smithi123.stdout:/dev/shm 2024-09-16T15:51:55.172 INFO:teuthology.orchestra.run.smithi123.stdout:/run/lock 2024-09-16T15:51:55.172 INFO:teuthology.orchestra.run.smithi123.stdout:/sys/fs/cgroup 2024-09-16T15:51:55.172 INFO:teuthology.orchestra.run.smithi123.stdout:/sys/fs/pstore 2024-09-16T15:51:55.172 INFO:teuthology.orchestra.run.smithi123.stdout:/sys/fs/bpf 2024-09-16T15:51:55.172 INFO:teuthology.orchestra.run.smithi123.stdout:/proc/sys/fs/binfmt_misc 2024-09-16T15:51:55.172 INFO:teuthology.orchestra.run.smithi123.stdout:/dev/hugepages 2024-09-16T15:51:55.172 INFO:teuthology.orchestra.run.smithi123.stdout:/dev/mqueue 2024-09-16T15:51:55.173 INFO:teuthology.orchestra.run.smithi123.stdout:/sys/kernel/debug 2024-09-16T15:51:55.173 INFO:teuthology.orchestra.run.smithi123.stdout:/sys/kernel/tracing 2024-09-16T15:51:55.173 INFO:teuthology.orchestra.run.smithi123.stdout:/proc/fs/nfsd 2024-09-16T15:51:55.173 INFO:teuthology.orchestra.run.smithi123.stdout:/sys/fs/fuse/connections 2024-09-16T15:51:55.173 INFO:teuthology.orchestra.run.smithi123.stdout:/sys/kernel/config 2024-09-16T15:51:55.173 INFO:teuthology.orchestra.run.smithi123.stdout:/run/credentials/systemd-sysusers.service 2024-09-16T15:51:55.173 INFO:teuthology.orchestra.run.smithi123.stdout:/run/rpc_pipefs 2024-09-16T15:51:55.173 INFO:teuthology.orchestra.run.smithi123.stdout:/run/user/1000 2024-09-16T15:51:55.173 INFO:teuthology.orchestra.run.smithi123.stdout:/var/lib/ceph 2024-09-16T15:51:55.173 INFO:teuthology.orchestra.run.smithi123.stdout:/var/lib/ceph/osd/ceph-0 2024-09-16T15:51:55.173 INFO:teuthology.orchestra.run.smithi123.stdout:/var/lib/ceph/osd/ceph-1 2024-09-16T15:51:55.173 INFO:teuthology.orchestra.run.smithi123.stdout:/var/lib/ceph/osd/ceph-2 2024-09-16T15:51:55.173 INFO:teuthology.orchestra.run.smithi123.stdout:/var/lib/ceph/osd/ceph-3 2024-09-16T15:51:55.174 INFO:teuthology.orchestra.run.smithi123.stdout:/run/netns 2024-09-16T15:51:55.174 INFO:teuthology.orchestra.run.smithi123.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-09-16T15:51:55.174 INFO:teuthology.orchestra.run.smithi123.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-09-16T15:51:55.174 INFO:teuthology.orchestra.run.smithi123.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-09-16T15:51:55.174 INFO:teuthology.orchestra.run.smithi123.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-09-16T15:51:55.175 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:51:55.175 DEBUG:teuthology.orchestra.run.smithi123:> ls /sys/fs/fuse/connections 2024-09-16T15:51:55.188 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi123.stderr:2024-09-16T15:51:55.189+0000 7f7e69183580 -1 init, newargv = 0x55bca7d87cb0 newargc=15 2024-09-16T15:51:55.188 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi123.stderr:ceph-fuse[54722]: starting ceph client 2024-09-16T15:51:55.195 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi123.stderr:ceph-fuse[54722]: starting fuse 2024-09-16T15:51:56.293 DEBUG:teuthology.orchestra.run.smithi123:> sudo modprobe fuse 2024-09-16T15:51:56.309 DEBUG:teuthology.orchestra.run.smithi123:> cat /proc/self/mounts | awk '{print $2}' 2024-09-16T15:51:56.356 INFO:teuthology.orchestra.run.smithi123.stdout:/sys 2024-09-16T15:51:56.356 INFO:teuthology.orchestra.run.smithi123.stdout:/proc 2024-09-16T15:51:56.356 INFO:teuthology.orchestra.run.smithi123.stdout:/dev 2024-09-16T15:51:56.356 INFO:teuthology.orchestra.run.smithi123.stdout:/dev/pts 2024-09-16T15:51:56.356 INFO:teuthology.orchestra.run.smithi123.stdout:/run 2024-09-16T15:51:56.356 INFO:teuthology.orchestra.run.smithi123.stdout:/ 2024-09-16T15:51:56.356 INFO:teuthology.orchestra.run.smithi123.stdout:/sys/kernel/security 2024-09-16T15:51:56.356 INFO:teuthology.orchestra.run.smithi123.stdout:/dev/shm 2024-09-16T15:51:56.357 INFO:teuthology.orchestra.run.smithi123.stdout:/run/lock 2024-09-16T15:51:56.357 INFO:teuthology.orchestra.run.smithi123.stdout:/sys/fs/cgroup 2024-09-16T15:51:56.357 INFO:teuthology.orchestra.run.smithi123.stdout:/sys/fs/pstore 2024-09-16T15:51:56.357 INFO:teuthology.orchestra.run.smithi123.stdout:/sys/fs/bpf 2024-09-16T15:51:56.357 INFO:teuthology.orchestra.run.smithi123.stdout:/proc/sys/fs/binfmt_misc 2024-09-16T15:51:56.357 INFO:teuthology.orchestra.run.smithi123.stdout:/dev/hugepages 2024-09-16T15:51:56.357 INFO:teuthology.orchestra.run.smithi123.stdout:/dev/mqueue 2024-09-16T15:51:56.357 INFO:teuthology.orchestra.run.smithi123.stdout:/sys/kernel/debug 2024-09-16T15:51:56.357 INFO:teuthology.orchestra.run.smithi123.stdout:/sys/kernel/tracing 2024-09-16T15:51:56.357 INFO:teuthology.orchestra.run.smithi123.stdout:/proc/fs/nfsd 2024-09-16T15:51:56.357 INFO:teuthology.orchestra.run.smithi123.stdout:/sys/fs/fuse/connections 2024-09-16T15:51:56.357 INFO:teuthology.orchestra.run.smithi123.stdout:/sys/kernel/config 2024-09-16T15:51:56.357 INFO:teuthology.orchestra.run.smithi123.stdout:/run/credentials/systemd-sysusers.service 2024-09-16T15:51:56.357 INFO:teuthology.orchestra.run.smithi123.stdout:/run/rpc_pipefs 2024-09-16T15:51:56.358 INFO:teuthology.orchestra.run.smithi123.stdout:/run/user/1000 2024-09-16T15:51:56.358 INFO:teuthology.orchestra.run.smithi123.stdout:/var/lib/ceph 2024-09-16T15:51:56.358 INFO:teuthology.orchestra.run.smithi123.stdout:/var/lib/ceph/osd/ceph-0 2024-09-16T15:51:56.358 INFO:teuthology.orchestra.run.smithi123.stdout:/var/lib/ceph/osd/ceph-1 2024-09-16T15:51:56.358 INFO:teuthology.orchestra.run.smithi123.stdout:/var/lib/ceph/osd/ceph-2 2024-09-16T15:51:56.358 INFO:teuthology.orchestra.run.smithi123.stdout:/var/lib/ceph/osd/ceph-3 2024-09-16T15:51:56.358 INFO:teuthology.orchestra.run.smithi123.stdout:/run/netns 2024-09-16T15:51:56.358 INFO:teuthology.orchestra.run.smithi123.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-09-16T15:51:56.358 INFO:teuthology.orchestra.run.smithi123.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-09-16T15:51:56.359 INFO:teuthology.orchestra.run.smithi123.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-09-16T15:51:56.359 INFO:teuthology.orchestra.run.smithi123.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-09-16T15:51:56.359 INFO:teuthology.orchestra.run.smithi123.stdout:/home/ubuntu/cephtest/mnt.0 2024-09-16T15:51:56.359 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:51:56.360 DEBUG:teuthology.orchestra.run.smithi123:> ls /sys/fs/fuse/connections 2024-09-16T15:51:56.407 INFO:teuthology.orchestra.run.smithi123.stdout:45 2024-09-16T15:51:56.408 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [45] 2024-09-16T15:51:56.408 DEBUG:teuthology.orchestra.run.smithi123:> sudo stdin-killer -- python3 -c ' 2024-09-16T15:51:56.408 DEBUG:teuthology.orchestra.run.smithi123:> import glob 2024-09-16T15:51:56.408 DEBUG:teuthology.orchestra.run.smithi123:> import re 2024-09-16T15:51:56.408 DEBUG:teuthology.orchestra.run.smithi123:> import os 2024-09-16T15:51:56.408 DEBUG:teuthology.orchestra.run.smithi123:> import subprocess 2024-09-16T15:51:56.408 DEBUG:teuthology.orchestra.run.smithi123:> 2024-09-16T15:51:56.408 DEBUG:teuthology.orchestra.run.smithi123:> def _find_admin_socket(client_name): 2024-09-16T15:51:56.408 DEBUG:teuthology.orchestra.run.smithi123:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2024-09-16T15:51:56.408 DEBUG:teuthology.orchestra.run.smithi123:> files = glob.glob(asok_path) 2024-09-16T15:51:56.409 DEBUG:teuthology.orchestra.run.smithi123:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-09-16T15:51:56.409 DEBUG:teuthology.orchestra.run.smithi123:> 2024-09-16T15:51:56.409 DEBUG:teuthology.orchestra.run.smithi123:> # Given a non-glob path, it better be there 2024-09-16T15:51:56.409 DEBUG:teuthology.orchestra.run.smithi123:> if "*" not in asok_path: 2024-09-16T15:51:56.409 DEBUG:teuthology.orchestra.run.smithi123:> assert(len(files) == 1) 2024-09-16T15:51:56.409 DEBUG:teuthology.orchestra.run.smithi123:> return files[0] 2024-09-16T15:51:56.409 DEBUG:teuthology.orchestra.run.smithi123:> 2024-09-16T15:51:56.409 DEBUG:teuthology.orchestra.run.smithi123:> for f in files: 2024-09-16T15:51:56.409 DEBUG:teuthology.orchestra.run.smithi123:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-09-16T15:51:56.409 DEBUG:teuthology.orchestra.run.smithi123:> if os.path.exists("/proc/{0}".format(pid)): 2024-09-16T15:51:56.409 DEBUG:teuthology.orchestra.run.smithi123:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-09-16T15:51:56.409 DEBUG:teuthology.orchestra.run.smithi123:> contents = proc_f.read() 2024-09-16T15:51:56.409 DEBUG:teuthology.orchestra.run.smithi123:> if mountpoint in contents: 2024-09-16T15:51:56.409 DEBUG:teuthology.orchestra.run.smithi123:> return f 2024-09-16T15:51:56.409 DEBUG:teuthology.orchestra.run.smithi123:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-09-16T15:51:56.410 DEBUG:teuthology.orchestra.run.smithi123:> 2024-09-16T15:51:56.410 DEBUG:teuthology.orchestra.run.smithi123:> print(_find_admin_socket("client.0")) 2024-09-16T15:51:56.410 DEBUG:teuthology.orchestra.run.smithi123:> ' 2024-09-16T15:51:56.532 INFO:teuthology.orchestra.run.smithi123.stdout:/var/run/ceph/ceph-client.0.54722.asok 2024-09-16T15:51:56.536 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:56 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-16T15:51:56.544 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.54722.asok 2024-09-16T15:51:56.544 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:51:56.544 DEBUG:teuthology.orchestra.run.smithi123:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.54722.asok status 2024-09-16T15:51:56.700 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2024-09-16T15:51:56.700 INFO:teuthology.orchestra.run.smithi123.stdout: "metadata": { 2024-09-16T15:51:56.700 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph_sha1": "33f95d2ae1bcbe7eb04404303f179eeeaa6982df", 2024-09-16T15:51:56.700 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph_version": "ceph version 19.3.0-4938-g33f95d2a (33f95d2ae1bcbe7eb04404303f179eeeaa6982df) squid (dev)", 2024-09-16T15:51:56.700 INFO:teuthology.orchestra.run.smithi123.stdout: "entity_id": "0", 2024-09-16T15:51:56.701 INFO:teuthology.orchestra.run.smithi123.stdout: "hostname": "smithi123", 2024-09-16T15:51:56.701 INFO:teuthology.orchestra.run.smithi123.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2024-09-16T15:51:56.701 INFO:teuthology.orchestra.run.smithi123.stdout: "pid": "54722", 2024-09-16T15:51:56.701 INFO:teuthology.orchestra.run.smithi123.stdout: "root": "/" 2024-09-16T15:51:56.701 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:51:56.701 INFO:teuthology.orchestra.run.smithi123.stdout: "dentry_count": 0, 2024-09-16T15:51:56.701 INFO:teuthology.orchestra.run.smithi123.stdout: "dentry_pinned_count": 0, 2024-09-16T15:51:56.701 INFO:teuthology.orchestra.run.smithi123.stdout: "id": 9393, 2024-09-16T15:51:56.701 INFO:teuthology.orchestra.run.smithi123.stdout: "inst": { 2024-09-16T15:51:56.701 INFO:teuthology.orchestra.run.smithi123.stdout: "name": { 2024-09-16T15:51:56.701 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "client", 2024-09-16T15:51:56.701 INFO:teuthology.orchestra.run.smithi123.stdout: "num": 9393 2024-09-16T15:51:56.701 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:51:56.702 INFO:teuthology.orchestra.run.smithi123.stdout: "addr": { 2024-09-16T15:51:56.702 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "v1", 2024-09-16T15:51:56.702 INFO:teuthology.orchestra.run.smithi123.stdout: "addr": "192.168.0.1:0", 2024-09-16T15:51:56.702 INFO:teuthology.orchestra.run.smithi123.stdout: "nonce": 1366079205 2024-09-16T15:51:56.702 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-16T15:51:56.702 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:51:56.702 INFO:teuthology.orchestra.run.smithi123.stdout: "addr": { 2024-09-16T15:51:56.702 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "v1", 2024-09-16T15:51:56.702 INFO:teuthology.orchestra.run.smithi123.stdout: "addr": "192.168.0.1:0", 2024-09-16T15:51:56.702 INFO:teuthology.orchestra.run.smithi123.stdout: "nonce": 1366079205 2024-09-16T15:51:56.702 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2024-09-16T15:51:56.702 INFO:teuthology.orchestra.run.smithi123.stdout: "inst_str": "client.9393 192.168.0.1:0/1366079205", 2024-09-16T15:51:56.703 INFO:teuthology.orchestra.run.smithi123.stdout: "addr_str": "192.168.0.1:0/1366079205", 2024-09-16T15:51:56.703 INFO:teuthology.orchestra.run.smithi123.stdout: "inode_count": 1, 2024-09-16T15:51:56.703 INFO:teuthology.orchestra.run.smithi123.stdout: "mds_epoch": 115, 2024-09-16T15:51:56.703 INFO:teuthology.orchestra.run.smithi123.stdout: "osd_epoch": 169, 2024-09-16T15:51:56.703 INFO:teuthology.orchestra.run.smithi123.stdout: "osd_epoch_barrier": 0, 2024-09-16T15:51:56.703 INFO:teuthology.orchestra.run.smithi123.stdout: "blocklisted": false, 2024-09-16T15:51:56.703 INFO:teuthology.orchestra.run.smithi123.stdout: "fs_name": "cephfs" 2024-09-16T15:51:56.703 INFO:teuthology.orchestra.run.smithi123.stdout:} 2024-09-16T15:51:56.714 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:51:56.714 DEBUG:teuthology.orchestra.run.smithi123:> stat --file-system '--printf=%T 2024-09-16T15:51:56.714 DEBUG:teuthology.orchestra.run.smithi123:> ' -- /home/ubuntu/cephtest/mnt.0 2024-09-16T15:51:56.721 INFO:teuthology.orchestra.run.smithi123.stdout:fuseblk 2024-09-16T15:51:56.721 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2024-09-16T15:51:56.721 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:51:56.721 DEBUG:teuthology.orchestra.run.smithi123:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-09-16T15:51:56.775 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-09-16T15:51:56.950 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:56.949+0000 7f004c1ef640 1 -- 172.21.15.123:0/1826345712 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f002c003a60 msgr2=0x7f002c003e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:51:56.950 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:56.949+0000 7f004c1ef640 1 --2- 172.21.15.123:0/1826345712 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f002c003a60 0x7f002c003e80 secure :-1 s=READY pgs=1907 cs=0 l=1 rev1=1 crypto rx=0x7f003c00bb10 tx=0x7f003c01cd20 comp rx=0 tx=0).stop 2024-09-16T15:51:56.950 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:56.949+0000 7f004a78c640 1 -- 172.21.15.123:0/1826345712 reap_dead start 2024-09-16T15:51:56.950 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:56.949+0000 7f004c1ef640 1 -- 172.21.15.123:0/1826345712 shutdown_connections 2024-09-16T15:51:56.950 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:56.949+0000 7f004c1ef640 1 -- 172.21.15.123:0/1826345712 >> 172.21.15.123:0/1826345712 conn(0x7f00440cfb10 msgr2=0x7f00440cff20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:51:56.951 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:56.949+0000 7f004c1ef640 1 -- 172.21.15.123:0/1826345712 shutdown_connections 2024-09-16T15:51:56.951 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:56.949+0000 7f004c1ef640 1 -- 172.21.15.123:0/1826345712 wait complete. 2024-09-16T15:51:56.951 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:56.953+0000 7f004c1ef640 1 Processor -- start 2024-09-16T15:51:56.951 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:56.953+0000 7f004c1ef640 1 -- start start 2024-09-16T15:51:56.953 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:56.953+0000 7f004c1ef640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f002c003a60 0x7f00441fba00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:51:56.953 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:56.953+0000 7f004c1ef640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0044151c00 0x7f0044154ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:51:56.953 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:56.953+0000 7f004c1ef640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f0044154ff0 0x7f0044155be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:51:56.953 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:56.953+0000 7f004c1ef640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f004415b1e0 con 0x7f0044151c00 2024-09-16T15:51:56.953 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:56.953+0000 7f004c1ef640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f004415b060 con 0x7f002c003a60 2024-09-16T15:51:56.953 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:56.953+0000 7f004c1ef640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f004415b360 con 0x7f0044154ff0 2024-09-16T15:51:56.953 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:56.953+0000 7f004978a640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0044151c00 0x7f0044154ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:51:56.953 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:56.953+0000 7f004978a640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0044151c00 0x7f0044154ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:43610/0 (socket says 172.21.15.123:43610) 2024-09-16T15:51:56.953 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:56.953+0000 7f004978a640 1 -- 172.21.15.123:0/3892165019 learned_addr learned my addr 172.21.15.123:0/3892165019 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:51:56.954 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:56.953+0000 7f004a78c640 1 --2- 172.21.15.123:0/3892165019 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f0044154ff0 0x7f0044155be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:51:56.954 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:56.953+0000 7f0049f8b640 1 --2- 172.21.15.123:0/3892165019 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f002c003a60 0x7f00441fba00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:51:56.954 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:56.953+0000 7f004978a640 1 -- 172.21.15.123:0/3892165019 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f0044154ff0 msgr2=0x7f0044155be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:51:56.954 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:56.953+0000 7f004978a640 1 --2- 172.21.15.123:0/3892165019 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f0044154ff0 0x7f0044155be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:56.954 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:56.953+0000 7f004978a640 1 -- 172.21.15.123:0/3892165019 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f002c003a60 msgr2=0x7f00441fba00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:51:56.954 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:56.953+0000 7f004978a640 1 --2- 172.21.15.123:0/3892165019 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f002c003a60 0x7f00441fba00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:56.954 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:56.953+0000 7f004978a640 1 -- 172.21.15.123:0/3892165019 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f00441faf60 con 0x7f0044151c00 2024-09-16T15:51:56.954 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:56.953+0000 7f004978a640 1 --2- 172.21.15.123:0/3892165019 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0044151c00 0x7f0044154ab0 secure :-1 s=READY pgs=1908 cs=0 l=1 rev1=1 crypto rx=0x7f004000cb70 tx=0x7f004000a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:51:56.954 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:56.953+0000 7f003a7fc640 1 -- 172.21.15.123:0/3892165019 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0040018070 con 0x7f0044151c00 2024-09-16T15:51:56.954 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:56.953+0000 7f003a7fc640 1 -- 172.21.15.123:0/3892165019 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f0040006e10 con 0x7f0044151c00 2024-09-16T15:51:56.954 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:56.953+0000 7f003a7fc640 1 -- 172.21.15.123:0/3892165019 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f004000ad40 con 0x7f0044151c00 2024-09-16T15:51:56.954 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:56.953+0000 7f004c1ef640 1 -- 172.21.15.123:0/3892165019 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0044155570 con 0x7f0044151c00 2024-09-16T15:51:56.954 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:56.953+0000 7f004c1ef640 1 -- 172.21.15.123:0/3892165019 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f00441fb200 con 0x7f0044151c00 2024-09-16T15:51:56.955 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:56.953+0000 7f004c1ef640 1 -- 172.21.15.123:0/3892165019 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f00440ca320 con 0x7f0044151c00 2024-09-16T15:51:56.955 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:56.957+0000 7f003a7fc640 1 -- 172.21.15.123:0/3892165019 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 25) ==== 97998+0+0 (secure 0 0 0) 0x7f0040003980 con 0x7f0044151c00 2024-09-16T15:51:56.955 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:56.957+0000 7f003a7fc640 1 --2- 172.21.15.123:0/3892165019 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f0030073910 0x7f0030075dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:51:56.955 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:56.957+0000 7f003a7fc640 1 -- 172.21.15.123:0/3892165019 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(169..169 src has 1..169) ==== 5737+0+0 (secure 0 0 0) 0x7f0040091fa0 con 0x7f0044151c00 2024-09-16T15:51:56.955 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:56.957+0000 7f0049f8b640 1 --2- 172.21.15.123:0/3892165019 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f0030073910 0x7f0030075dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:51:56.956 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:56.957+0000 7f0049f8b640 1 --2- 172.21.15.123:0/3892165019 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f0030073910 0x7f0030075dd0 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f003c0b0000 tx=0x7f003c019ac0 comp rx=0 tx=0).ready entity=mgr.4150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:51:56.959 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:56.961+0000 7f003a7fc640 1 -- 172.21.15.123:0/3892165019 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f004005ea80 con 0x7f0044151c00 2024-09-16T15:51:57.166 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:57.165+0000 7f004c1ef640 1 -- 172.21.15.123:0/3892165019 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f004406eff0 con 0x7f0044151c00 2024-09-16T15:51:57.166 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:57.169+0000 7f003a7fc640 1 -- 172.21.15.123:0/3892165019 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v115) ==== 53+0+73 (secure 0 0 0) 0x7f004005e380 con 0x7f0044151c00 2024-09-16T15:51:57.167 INFO:teuthology.orchestra.run.smithi123.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-09-16T15:51:57.169 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:57.169+0000 7f004c1ef640 1 -- 172.21.15.123:0/3892165019 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f0030073910 msgr2=0x7f0030075dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:51:57.169 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:57.169+0000 7f004c1ef640 1 --2- 172.21.15.123:0/3892165019 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f0030073910 0x7f0030075dd0 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f003c0b0000 tx=0x7f003c019ac0 comp rx=0 tx=0).stop 2024-09-16T15:51:57.170 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:57.169+0000 7f004c1ef640 1 -- 172.21.15.123:0/3892165019 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0044151c00 msgr2=0x7f0044154ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:51:57.170 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:57.169+0000 7f004c1ef640 1 --2- 172.21.15.123:0/3892165019 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0044151c00 0x7f0044154ab0 secure :-1 s=READY pgs=1908 cs=0 l=1 rev1=1 crypto rx=0x7f004000cb70 tx=0x7f004000a430 comp rx=0 tx=0).stop 2024-09-16T15:51:57.170 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:57.169+0000 7f004c1ef640 1 -- 172.21.15.123:0/3892165019 shutdown_connections 2024-09-16T15:51:57.170 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:57.169+0000 7f004c1ef640 1 --2- 172.21.15.123:0/3892165019 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f0030073910 0x7f0030075dd0 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:57.170 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:57.169+0000 7f004c1ef640 1 --2- 172.21.15.123:0/3892165019 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f0044154ff0 0x7f0044155be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:57.170 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:57.169+0000 7f004c1ef640 1 --2- 172.21.15.123:0/3892165019 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0044151c00 0x7f0044154ab0 unknown :-1 s=CLOSED pgs=1908 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:57.170 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:57.169+0000 7f004c1ef640 1 --2- 172.21.15.123:0/3892165019 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f002c003a60 0x7f00441fba00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:57.170 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:57.169+0000 7f004c1ef640 1 -- 172.21.15.123:0/3892165019 >> 172.21.15.123:0/3892165019 conn(0x7f00440cfb10 msgr2=0x7f00440ad9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:51:57.170 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:57.169+0000 7f004c1ef640 1 -- 172.21.15.123:0/3892165019 shutdown_connections 2024-09-16T15:51:57.170 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:57.169+0000 7f004c1ef640 1 -- 172.21.15.123:0/3892165019 wait complete. 2024-09-16T15:51:57.184 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-09-16T15:51:57.184 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-09-16T15:51:57.184 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi184.front.sepia.ceph.com 2024-09-16T15:51:57.184 INFO:tasks.cephfs.mount:self.client.name = client.1 2024-09-16T15:51:57.184 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2024-09-16T15:51:57.184 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-09-16T15:51:57.185 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-09-16T15:51:57.185 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-09-16T15:51:57.185 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.1' netns for '/home/ubuntu/cephtest/mnt.1' 2024-09-16T15:51:57.185 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:51:57.185 DEBUG:teuthology.orchestra.run.smithi184:> ip addr 2024-09-16T15:51:57.192 INFO:teuthology.orchestra.run.smithi184.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-09-16T15:51:57.192 INFO:teuthology.orchestra.run.smithi184.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-09-16T15:51:57.192 INFO:teuthology.orchestra.run.smithi184.stdout: inet 127.0.0.1/8 scope host lo 2024-09-16T15:51:57.192 INFO:teuthology.orchestra.run.smithi184.stdout: valid_lft forever preferred_lft forever 2024-09-16T15:51:57.192 INFO:teuthology.orchestra.run.smithi184.stdout: inet6 ::1/128 scope host 2024-09-16T15:51:57.192 INFO:teuthology.orchestra.run.smithi184.stdout: valid_lft forever preferred_lft forever 2024-09-16T15:51:57.193 INFO:teuthology.orchestra.run.smithi184.stdout:2: eno1: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-09-16T15:51:57.193 INFO:teuthology.orchestra.run.smithi184.stdout: link/ether 0c:c4:7a:d9:37:16 brd ff:ff:ff:ff:ff:ff 2024-09-16T15:51:57.193 INFO:teuthology.orchestra.run.smithi184.stdout: altname enp7s0f0 2024-09-16T15:51:57.193 INFO:teuthology.orchestra.run.smithi184.stdout:3: eno2: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-09-16T15:51:57.193 INFO:teuthology.orchestra.run.smithi184.stdout: link/ether 0c:c4:7a:d9:37:17 brd ff:ff:ff:ff:ff:ff 2024-09-16T15:51:57.193 INFO:teuthology.orchestra.run.smithi184.stdout: altname enp7s0f1 2024-09-16T15:51:57.193 INFO:teuthology.orchestra.run.smithi184.stdout:4: enp3s0f0: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-09-16T15:51:57.193 INFO:teuthology.orchestra.run.smithi184.stdout: link/ether 0c:c4:7a:88:7b:8e brd ff:ff:ff:ff:ff:ff 2024-09-16T15:51:57.193 INFO:teuthology.orchestra.run.smithi184.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-09-16T15:51:57.193 INFO:teuthology.orchestra.run.smithi184.stdout: link/ether 0c:c4:7a:88:7b:8f brd ff:ff:ff:ff:ff:ff 2024-09-16T15:51:57.193 INFO:teuthology.orchestra.run.smithi184.stdout: inet 172.21.15.184/20 metric 100 brd 172.21.15.255 scope global dynamic enp3s0f1 2024-09-16T15:51:57.193 INFO:teuthology.orchestra.run.smithi184.stdout: valid_lft 41940sec preferred_lft 41940sec 2024-09-16T15:51:57.193 INFO:teuthology.orchestra.run.smithi184.stdout: inet6 fe80::ec4:7aff:fe88:7b8f/64 scope link 2024-09-16T15:51:57.194 INFO:teuthology.orchestra.run.smithi184.stdout: valid_lft forever preferred_lft forever 2024-09-16T15:51:57.194 INFO:teuthology.orchestra.run.smithi184.stdout:6: docker0: mtu 1500 qdisc noqueue state DOWN group default 2024-09-16T15:51:57.194 INFO:teuthology.orchestra.run.smithi184.stdout: link/ether 02:42:5d:0a:44:b9 brd ff:ff:ff:ff:ff:ff 2024-09-16T15:51:57.194 INFO:teuthology.orchestra.run.smithi184.stdout: inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0 2024-09-16T15:51:57.194 INFO:teuthology.orchestra.run.smithi184.stdout: valid_lft forever preferred_lft forever 2024-09-16T15:51:57.194 INFO:teuthology.orchestra.run.smithi184.stdout:7: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-09-16T15:51:57.194 INFO:teuthology.orchestra.run.smithi184.stdout: link/ether 5e:1f:84:6d:79:ae brd ff:ff:ff:ff:ff:ff 2024-09-16T15:51:57.194 INFO:teuthology.orchestra.run.smithi184.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-09-16T15:51:57.194 INFO:teuthology.orchestra.run.smithi184.stdout: valid_lft forever preferred_lft forever 2024-09-16T15:51:57.194 INFO:teuthology.orchestra.run.smithi184.stdout: inet6 fe80::5c1f:84ff:fe6d:79ae/64 scope link 2024-09-16T15:51:57.194 INFO:teuthology.orchestra.run.smithi184.stdout: valid_lft forever preferred_lft forever 2024-09-16T15:51:57.194 INFO:teuthology.orchestra.run.smithi184.stdout:8: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-09-16T15:51:57.194 INFO:teuthology.orchestra.run.smithi184.stdout: link/ether 0e:5c:f6:1f:82:2f brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.1 2024-09-16T15:51:57.194 INFO:teuthology.orchestra.run.smithi184.stdout: inet6 fe80::c5c:f6ff:fe1f:822f/64 scope link 2024-09-16T15:51:57.194 INFO:teuthology.orchestra.run.smithi184.stdout: valid_lft forever preferred_lft forever 2024-09-16T15:51:57.195 INFO:teuthology.orchestra.run.smithi184.stdout:9: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-09-16T15:51:57.195 INFO:teuthology.orchestra.run.smithi184.stdout: link/ether c6:a6:3d:69:d0:db brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.3 2024-09-16T15:51:57.195 INFO:teuthology.orchestra.run.smithi184.stdout: inet6 fe80::c4a6:3dff:fe69:d0db/64 scope link 2024-09-16T15:51:57.195 INFO:teuthology.orchestra.run.smithi184.stdout: valid_lft forever preferred_lft forever 2024-09-16T15:51:57.195 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:51:57.195 DEBUG:teuthology.orchestra.run.smithi184:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-09-16T15:51:57.247 INFO:teuthology.orchestra.run.smithi184.stdout:1 2024-09-16T15:51:57.249 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:51:57.249 DEBUG:teuthology.orchestra.run.smithi184:> route 2024-09-16T15:51:57.300 INFO:teuthology.orchestra.run.smithi184.stdout:Kernel IP routing table 2024-09-16T15:51:57.300 INFO:teuthology.orchestra.run.smithi184.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-09-16T15:51:57.300 INFO:teuthology.orchestra.run.smithi184.stdout:default _gateway 0.0.0.0 UG 0 0 0 enp3s0f1 2024-09-16T15:51:57.300 INFO:teuthology.orchestra.run.smithi184.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-09-16T15:51:57.300 INFO:teuthology.orchestra.run.smithi184.stdout:158.69.67.47 _gateway 255.255.255.255 UGH 100 0 0 enp3s0f1 2024-09-16T15:51:57.300 INFO:teuthology.orchestra.run.smithi184.stdout:172.17.0.0 0.0.0.0 255.255.0.0 U 0 0 0 docker0 2024-09-16T15:51:57.300 INFO:teuthology.orchestra.run.smithi184.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-09-16T15:51:57.300 INFO:teuthology.orchestra.run.smithi184.stdout:ns1.front.sepia 0.0.0.0 255.255.255.255 UH 100 0 0 enp3s0f1 2024-09-16T15:51:57.300 INFO:teuthology.orchestra.run.smithi184.stdout:172.21.0.2 0.0.0.0 255.255.255.255 UH 100 0 0 enp3s0f1 2024-09-16T15:51:57.300 INFO:teuthology.orchestra.run.smithi184.stdout:_gateway 0.0.0.0 255.255.255.255 UH 100 0 0 enp3s0f1 2024-09-16T15:51:57.301 INFO:teuthology.orchestra.run.smithi184.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-09-16T15:51:57.301 DEBUG:teuthology.orchestra.run.smithi184:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-09-16T15:51:57.301 DEBUG:teuthology.orchestra.run.smithi184:> set -e 2024-09-16T15:51:57.301 DEBUG:teuthology.orchestra.run.smithi184:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-09-16T15:51:57.301 DEBUG:teuthology.orchestra.run.smithi184:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-09-16T15:51:57.301 DEBUG:teuthology.orchestra.run.smithi184:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-09-16T15:51:57.301 DEBUG:teuthology.orchestra.run.smithi184:> ') 2024-09-16T15:51:57.389 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:51:57 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-09-16T15:51:57.416 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:51:57 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-16T15:51:57.425 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:51:57.425 DEBUG:teuthology.orchestra.run.smithi184:> ip netns list 2024-09-16T15:51:57.430 INFO:teuthology.orchestra.run.smithi184.stdout:ceph-ns--home-ubuntu-cephtest-mnt.3 (id: 1) 2024-09-16T15:51:57.431 INFO:teuthology.orchestra.run.smithi184.stdout:ceph-ns--home-ubuntu-cephtest-mnt.1 (id: 0) 2024-09-16T15:51:57.431 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.1' netns for '/home/ubuntu/cephtest/mnt.1' 2024-09-16T15:51:57.431 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:51:57.431 DEBUG:teuthology.orchestra.run.smithi184:> sudo ip link set brx.0 up 2024-09-16T15:51:57.484 INFO:tasks.cephfs.fuse_mount:Client client.1 config is {} 2024-09-16T15:51:57.484 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-09-16T15:51:57.484 DEBUG:teuthology.orchestra.run.smithi184:> mkdir -p -v /home/ubuntu/cephtest/mnt.1 2024-09-16T15:51:57.536 INFO:teuthology.orchestra.run.smithi184.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.1' 2024-09-16T15:51:57.536 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-09-16T15:51:57.536 DEBUG:teuthology.orchestra.run.smithi184:> chmod 0000 /home/ubuntu/cephtest/mnt.1 2024-09-16T15:51:57.587 DEBUG:teuthology.orchestra.run.smithi184:> sudo modprobe fuse 2024-09-16T15:51:57.645 DEBUG:teuthology.orchestra.run.smithi184:> cat /proc/self/mounts | awk '{print $2}' 2024-09-16T15:51:57.697 INFO:teuthology.orchestra.run.smithi184.stdout:/sys 2024-09-16T15:51:57.697 INFO:teuthology.orchestra.run.smithi184.stdout:/proc 2024-09-16T15:51:57.697 INFO:teuthology.orchestra.run.smithi184.stdout:/dev 2024-09-16T15:51:57.697 INFO:teuthology.orchestra.run.smithi184.stdout:/dev/pts 2024-09-16T15:51:57.697 INFO:teuthology.orchestra.run.smithi184.stdout:/run 2024-09-16T15:51:57.697 INFO:teuthology.orchestra.run.smithi184.stdout:/ 2024-09-16T15:51:57.697 INFO:teuthology.orchestra.run.smithi184.stdout:/sys/kernel/security 2024-09-16T15:51:57.697 INFO:teuthology.orchestra.run.smithi184.stdout:/dev/shm 2024-09-16T15:51:57.698 INFO:teuthology.orchestra.run.smithi184.stdout:/run/lock 2024-09-16T15:51:57.698 INFO:teuthology.orchestra.run.smithi184.stdout:/sys/fs/cgroup 2024-09-16T15:51:57.698 INFO:teuthology.orchestra.run.smithi184.stdout:/sys/fs/pstore 2024-09-16T15:51:57.698 INFO:teuthology.orchestra.run.smithi184.stdout:/sys/fs/bpf 2024-09-16T15:51:57.698 INFO:teuthology.orchestra.run.smithi184.stdout:/proc/sys/fs/binfmt_misc 2024-09-16T15:51:57.698 INFO:teuthology.orchestra.run.smithi184.stdout:/dev/hugepages 2024-09-16T15:51:57.698 INFO:teuthology.orchestra.run.smithi184.stdout:/dev/mqueue 2024-09-16T15:51:57.698 INFO:teuthology.orchestra.run.smithi184.stdout:/sys/kernel/debug 2024-09-16T15:51:57.698 INFO:teuthology.orchestra.run.smithi184.stdout:/sys/kernel/tracing 2024-09-16T15:51:57.698 INFO:teuthology.orchestra.run.smithi184.stdout:/proc/fs/nfsd 2024-09-16T15:51:57.698 INFO:teuthology.orchestra.run.smithi184.stdout:/sys/fs/fuse/connections 2024-09-16T15:51:57.698 INFO:teuthology.orchestra.run.smithi184.stdout:/sys/kernel/config 2024-09-16T15:51:57.698 INFO:teuthology.orchestra.run.smithi184.stdout:/run/credentials/systemd-sysusers.service 2024-09-16T15:51:57.698 INFO:teuthology.orchestra.run.smithi184.stdout:/run/rpc_pipefs 2024-09-16T15:51:57.699 INFO:teuthology.orchestra.run.smithi184.stdout:/run/user/1000 2024-09-16T15:51:57.699 INFO:teuthology.orchestra.run.smithi184.stdout:/var/lib/ceph 2024-09-16T15:51:57.699 INFO:teuthology.orchestra.run.smithi184.stdout:/var/lib/ceph/osd/ceph-4 2024-09-16T15:51:57.699 INFO:teuthology.orchestra.run.smithi184.stdout:/var/lib/ceph/osd/ceph-5 2024-09-16T15:51:57.699 INFO:teuthology.orchestra.run.smithi184.stdout:/var/lib/ceph/osd/ceph-6 2024-09-16T15:51:57.699 INFO:teuthology.orchestra.run.smithi184.stdout:/var/lib/ceph/osd/ceph-7 2024-09-16T15:51:57.699 INFO:teuthology.orchestra.run.smithi184.stdout:/run/netns 2024-09-16T15:51:57.699 INFO:teuthology.orchestra.run.smithi184.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-09-16T15:51:57.699 INFO:teuthology.orchestra.run.smithi184.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-09-16T15:51:57.699 INFO:teuthology.orchestra.run.smithi184.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-09-16T15:51:57.699 INFO:teuthology.orchestra.run.smithi184.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-09-16T15:51:57.700 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:51:57.700 DEBUG:teuthology.orchestra.run.smithi184:> ls /sys/fs/fuse/connections 2024-09-16T15:51:57.748 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-09-16T15:51:57.748 DEBUG:teuthology.orchestra.run.smithi184:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.1 --id 1) 2024-09-16T15:51:57.793 DEBUG:teuthology.orchestra.run.smithi184:> sudo modprobe fuse 2024-09-16T15:51:57.806 DEBUG:teuthology.orchestra.run.smithi184:> cat /proc/self/mounts | awk '{print $2}' 2024-09-16T15:51:57.856 INFO:teuthology.orchestra.run.smithi184.stdout:/sys 2024-09-16T15:51:57.856 INFO:teuthology.orchestra.run.smithi184.stdout:/proc 2024-09-16T15:51:57.856 INFO:teuthology.orchestra.run.smithi184.stdout:/dev 2024-09-16T15:51:57.856 INFO:teuthology.orchestra.run.smithi184.stdout:/dev/pts 2024-09-16T15:51:57.857 INFO:teuthology.orchestra.run.smithi184.stdout:/run 2024-09-16T15:51:57.857 INFO:teuthology.orchestra.run.smithi184.stdout:/ 2024-09-16T15:51:57.857 INFO:teuthology.orchestra.run.smithi184.stdout:/sys/kernel/security 2024-09-16T15:51:57.857 INFO:teuthology.orchestra.run.smithi184.stdout:/dev/shm 2024-09-16T15:51:57.857 INFO:teuthology.orchestra.run.smithi184.stdout:/run/lock 2024-09-16T15:51:57.857 INFO:teuthology.orchestra.run.smithi184.stdout:/sys/fs/cgroup 2024-09-16T15:51:57.857 INFO:teuthology.orchestra.run.smithi184.stdout:/sys/fs/pstore 2024-09-16T15:51:57.857 INFO:teuthology.orchestra.run.smithi184.stdout:/sys/fs/bpf 2024-09-16T15:51:57.857 INFO:teuthology.orchestra.run.smithi184.stdout:/proc/sys/fs/binfmt_misc 2024-09-16T15:51:57.857 INFO:teuthology.orchestra.run.smithi184.stdout:/dev/hugepages 2024-09-16T15:51:57.857 INFO:teuthology.orchestra.run.smithi184.stdout:/dev/mqueue 2024-09-16T15:51:57.857 INFO:teuthology.orchestra.run.smithi184.stdout:/sys/kernel/debug 2024-09-16T15:51:57.857 INFO:teuthology.orchestra.run.smithi184.stdout:/sys/kernel/tracing 2024-09-16T15:51:57.857 INFO:teuthology.orchestra.run.smithi184.stdout:/proc/fs/nfsd 2024-09-16T15:51:57.858 INFO:teuthology.orchestra.run.smithi184.stdout:/sys/fs/fuse/connections 2024-09-16T15:51:57.858 INFO:teuthology.orchestra.run.smithi184.stdout:/sys/kernel/config 2024-09-16T15:51:57.858 INFO:teuthology.orchestra.run.smithi184.stdout:/run/credentials/systemd-sysusers.service 2024-09-16T15:51:57.858 INFO:teuthology.orchestra.run.smithi184.stdout:/run/rpc_pipefs 2024-09-16T15:51:57.858 INFO:teuthology.orchestra.run.smithi184.stdout:/run/user/1000 2024-09-16T15:51:57.858 INFO:teuthology.orchestra.run.smithi184.stdout:/var/lib/ceph 2024-09-16T15:51:57.858 INFO:teuthology.orchestra.run.smithi184.stdout:/var/lib/ceph/osd/ceph-4 2024-09-16T15:51:57.858 INFO:teuthology.orchestra.run.smithi184.stdout:/var/lib/ceph/osd/ceph-5 2024-09-16T15:51:57.858 INFO:teuthology.orchestra.run.smithi184.stdout:/var/lib/ceph/osd/ceph-6 2024-09-16T15:51:57.858 INFO:teuthology.orchestra.run.smithi184.stdout:/var/lib/ceph/osd/ceph-7 2024-09-16T15:51:57.858 INFO:teuthology.orchestra.run.smithi184.stdout:/run/netns 2024-09-16T15:51:57.858 INFO:teuthology.orchestra.run.smithi184.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-09-16T15:51:57.858 INFO:teuthology.orchestra.run.smithi184.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-09-16T15:51:57.859 INFO:teuthology.orchestra.run.smithi184.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-09-16T15:51:57.859 INFO:teuthology.orchestra.run.smithi184.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-09-16T15:51:57.859 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:51:57.860 DEBUG:teuthology.orchestra.run.smithi184:> ls /sys/fs/fuse/connections 2024-09-16T15:51:57.872 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi184.stderr:2024-09-16T15:51:57.875+0000 7faa20ce0580 -1 init, newargv = 0x55b70caebbd0 newargc=15 2024-09-16T15:51:57.873 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi184.stderr:ceph-fuse[36573]: starting ceph client 2024-09-16T15:51:57.879 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi184.stderr:ceph-fuse[36573]: starting fuse 2024-09-16T15:51:58.979 DEBUG:teuthology.orchestra.run.smithi184:> sudo modprobe fuse 2024-09-16T15:51:58.994 DEBUG:teuthology.orchestra.run.smithi184:> cat /proc/self/mounts | awk '{print $2}' 2024-09-16T15:51:59.042 INFO:teuthology.orchestra.run.smithi184.stdout:/sys 2024-09-16T15:51:59.043 INFO:teuthology.orchestra.run.smithi184.stdout:/proc 2024-09-16T15:51:59.043 INFO:teuthology.orchestra.run.smithi184.stdout:/dev 2024-09-16T15:51:59.043 INFO:teuthology.orchestra.run.smithi184.stdout:/dev/pts 2024-09-16T15:51:59.043 INFO:teuthology.orchestra.run.smithi184.stdout:/run 2024-09-16T15:51:59.043 INFO:teuthology.orchestra.run.smithi184.stdout:/ 2024-09-16T15:51:59.043 INFO:teuthology.orchestra.run.smithi184.stdout:/sys/kernel/security 2024-09-16T15:51:59.043 INFO:teuthology.orchestra.run.smithi184.stdout:/dev/shm 2024-09-16T15:51:59.043 INFO:teuthology.orchestra.run.smithi184.stdout:/run/lock 2024-09-16T15:51:59.043 INFO:teuthology.orchestra.run.smithi184.stdout:/sys/fs/cgroup 2024-09-16T15:51:59.043 INFO:teuthology.orchestra.run.smithi184.stdout:/sys/fs/pstore 2024-09-16T15:51:59.043 INFO:teuthology.orchestra.run.smithi184.stdout:/sys/fs/bpf 2024-09-16T15:51:59.043 INFO:teuthology.orchestra.run.smithi184.stdout:/proc/sys/fs/binfmt_misc 2024-09-16T15:51:59.044 INFO:teuthology.orchestra.run.smithi184.stdout:/dev/hugepages 2024-09-16T15:51:59.044 INFO:teuthology.orchestra.run.smithi184.stdout:/dev/mqueue 2024-09-16T15:51:59.044 INFO:teuthology.orchestra.run.smithi184.stdout:/sys/kernel/debug 2024-09-16T15:51:59.044 INFO:teuthology.orchestra.run.smithi184.stdout:/sys/kernel/tracing 2024-09-16T15:51:59.044 INFO:teuthology.orchestra.run.smithi184.stdout:/proc/fs/nfsd 2024-09-16T15:51:59.044 INFO:teuthology.orchestra.run.smithi184.stdout:/sys/fs/fuse/connections 2024-09-16T15:51:59.044 INFO:teuthology.orchestra.run.smithi184.stdout:/sys/kernel/config 2024-09-16T15:51:59.044 INFO:teuthology.orchestra.run.smithi184.stdout:/run/credentials/systemd-sysusers.service 2024-09-16T15:51:59.044 INFO:teuthology.orchestra.run.smithi184.stdout:/run/rpc_pipefs 2024-09-16T15:51:59.044 INFO:teuthology.orchestra.run.smithi184.stdout:/run/user/1000 2024-09-16T15:51:59.044 INFO:teuthology.orchestra.run.smithi184.stdout:/var/lib/ceph 2024-09-16T15:51:59.044 INFO:teuthology.orchestra.run.smithi184.stdout:/var/lib/ceph/osd/ceph-4 2024-09-16T15:51:59.044 INFO:teuthology.orchestra.run.smithi184.stdout:/var/lib/ceph/osd/ceph-5 2024-09-16T15:51:59.044 INFO:teuthology.orchestra.run.smithi184.stdout:/var/lib/ceph/osd/ceph-6 2024-09-16T15:51:59.045 INFO:teuthology.orchestra.run.smithi184.stdout:/var/lib/ceph/osd/ceph-7 2024-09-16T15:51:59.045 INFO:teuthology.orchestra.run.smithi184.stdout:/run/netns 2024-09-16T15:51:59.045 INFO:teuthology.orchestra.run.smithi184.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-09-16T15:51:59.045 INFO:teuthology.orchestra.run.smithi184.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-09-16T15:51:59.045 INFO:teuthology.orchestra.run.smithi184.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-09-16T15:51:59.045 INFO:teuthology.orchestra.run.smithi184.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-09-16T15:51:59.045 INFO:teuthology.orchestra.run.smithi184.stdout:/home/ubuntu/cephtest/mnt.1 2024-09-16T15:51:59.046 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:51:59.046 DEBUG:teuthology.orchestra.run.smithi184:> ls /sys/fs/fuse/connections 2024-09-16T15:51:59.092 INFO:teuthology.orchestra.run.smithi184.stdout:45 2024-09-16T15:51:59.092 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [45] 2024-09-16T15:51:59.092 DEBUG:teuthology.orchestra.run.smithi184:> sudo stdin-killer -- python3 -c ' 2024-09-16T15:51:59.092 DEBUG:teuthology.orchestra.run.smithi184:> import glob 2024-09-16T15:51:59.092 DEBUG:teuthology.orchestra.run.smithi184:> import re 2024-09-16T15:51:59.092 DEBUG:teuthology.orchestra.run.smithi184:> import os 2024-09-16T15:51:59.092 DEBUG:teuthology.orchestra.run.smithi184:> import subprocess 2024-09-16T15:51:59.092 DEBUG:teuthology.orchestra.run.smithi184:> 2024-09-16T15:51:59.092 DEBUG:teuthology.orchestra.run.smithi184:> def _find_admin_socket(client_name): 2024-09-16T15:51:59.092 DEBUG:teuthology.orchestra.run.smithi184:> asok_path = "/var/run/ceph/ceph-client.1.*.asok" 2024-09-16T15:51:59.093 DEBUG:teuthology.orchestra.run.smithi184:> files = glob.glob(asok_path) 2024-09-16T15:51:59.093 DEBUG:teuthology.orchestra.run.smithi184:> mountpoint = "/home/ubuntu/cephtest/mnt.1" 2024-09-16T15:51:59.093 DEBUG:teuthology.orchestra.run.smithi184:> 2024-09-16T15:51:59.093 DEBUG:teuthology.orchestra.run.smithi184:> # Given a non-glob path, it better be there 2024-09-16T15:51:59.093 DEBUG:teuthology.orchestra.run.smithi184:> if "*" not in asok_path: 2024-09-16T15:51:59.093 DEBUG:teuthology.orchestra.run.smithi184:> assert(len(files) == 1) 2024-09-16T15:51:59.093 DEBUG:teuthology.orchestra.run.smithi184:> return files[0] 2024-09-16T15:51:59.093 DEBUG:teuthology.orchestra.run.smithi184:> 2024-09-16T15:51:59.093 DEBUG:teuthology.orchestra.run.smithi184:> for f in files: 2024-09-16T15:51:59.093 DEBUG:teuthology.orchestra.run.smithi184:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-09-16T15:51:59.093 DEBUG:teuthology.orchestra.run.smithi184:> if os.path.exists("/proc/{0}".format(pid)): 2024-09-16T15:51:59.093 DEBUG:teuthology.orchestra.run.smithi184:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-09-16T15:51:59.093 DEBUG:teuthology.orchestra.run.smithi184:> contents = proc_f.read() 2024-09-16T15:51:59.093 DEBUG:teuthology.orchestra.run.smithi184:> if mountpoint in contents: 2024-09-16T15:51:59.093 DEBUG:teuthology.orchestra.run.smithi184:> return f 2024-09-16T15:51:59.094 DEBUG:teuthology.orchestra.run.smithi184:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-09-16T15:51:59.094 DEBUG:teuthology.orchestra.run.smithi184:> 2024-09-16T15:51:59.094 DEBUG:teuthology.orchestra.run.smithi184:> print(_find_admin_socket("client.1")) 2024-09-16T15:51:59.094 DEBUG:teuthology.orchestra.run.smithi184:> ' 2024-09-16T15:51:59.218 INFO:teuthology.orchestra.run.smithi184.stdout:/var/run/ceph/ceph-client.1.36573.asok 2024-09-16T15:51:59.222 INFO:teuthology.orchestra.run.smithi184.stderr:2024-09-16T15:51:59 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-16T15:51:59.230 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.1.36573.asok 2024-09-16T15:51:59.230 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:51:59.230 DEBUG:teuthology.orchestra.run.smithi184:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.1.36573.asok status 2024-09-16T15:51:59.379 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-09-16T15:51:59.379 INFO:teuthology.orchestra.run.smithi184.stdout: "metadata": { 2024-09-16T15:51:59.380 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph_sha1": "33f95d2ae1bcbe7eb04404303f179eeeaa6982df", 2024-09-16T15:51:59.380 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph_version": "ceph version 19.3.0-4938-g33f95d2a (33f95d2ae1bcbe7eb04404303f179eeeaa6982df) squid (dev)", 2024-09-16T15:51:59.380 INFO:teuthology.orchestra.run.smithi184.stdout: "entity_id": "1", 2024-09-16T15:51:59.380 INFO:teuthology.orchestra.run.smithi184.stdout: "hostname": "smithi184", 2024-09-16T15:51:59.380 INFO:teuthology.orchestra.run.smithi184.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.1", 2024-09-16T15:51:59.380 INFO:teuthology.orchestra.run.smithi184.stdout: "pid": "36573", 2024-09-16T15:51:59.380 INFO:teuthology.orchestra.run.smithi184.stdout: "root": "/" 2024-09-16T15:51:59.380 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-16T15:51:59.380 INFO:teuthology.orchestra.run.smithi184.stdout: "dentry_count": 0, 2024-09-16T15:51:59.380 INFO:teuthology.orchestra.run.smithi184.stdout: "dentry_pinned_count": 0, 2024-09-16T15:51:59.380 INFO:teuthology.orchestra.run.smithi184.stdout: "id": 6377, 2024-09-16T15:51:59.380 INFO:teuthology.orchestra.run.smithi184.stdout: "inst": { 2024-09-16T15:51:59.380 INFO:teuthology.orchestra.run.smithi184.stdout: "name": { 2024-09-16T15:51:59.381 INFO:teuthology.orchestra.run.smithi184.stdout: "type": "client", 2024-09-16T15:51:59.381 INFO:teuthology.orchestra.run.smithi184.stdout: "num": 6377 2024-09-16T15:51:59.381 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-16T15:51:59.381 INFO:teuthology.orchestra.run.smithi184.stdout: "addr": { 2024-09-16T15:51:59.381 INFO:teuthology.orchestra.run.smithi184.stdout: "type": "v1", 2024-09-16T15:51:59.381 INFO:teuthology.orchestra.run.smithi184.stdout: "addr": "192.168.0.1:0", 2024-09-16T15:51:59.381 INFO:teuthology.orchestra.run.smithi184.stdout: "nonce": 2672973979 2024-09-16T15:51:59.381 INFO:teuthology.orchestra.run.smithi184.stdout: } 2024-09-16T15:51:59.381 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-16T15:51:59.381 INFO:teuthology.orchestra.run.smithi184.stdout: "addr": { 2024-09-16T15:51:59.381 INFO:teuthology.orchestra.run.smithi184.stdout: "type": "v1", 2024-09-16T15:51:59.381 INFO:teuthology.orchestra.run.smithi184.stdout: "addr": "192.168.0.1:0", 2024-09-16T15:51:59.381 INFO:teuthology.orchestra.run.smithi184.stdout: "nonce": 2672973979 2024-09-16T15:51:59.382 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2024-09-16T15:51:59.382 INFO:teuthology.orchestra.run.smithi184.stdout: "inst_str": "client.6377 192.168.0.1:0/2672973979", 2024-09-16T15:51:59.382 INFO:teuthology.orchestra.run.smithi184.stdout: "addr_str": "192.168.0.1:0/2672973979", 2024-09-16T15:51:59.382 INFO:teuthology.orchestra.run.smithi184.stdout: "inode_count": 1, 2024-09-16T15:51:59.382 INFO:teuthology.orchestra.run.smithi184.stdout: "mds_epoch": 115, 2024-09-16T15:51:59.382 INFO:teuthology.orchestra.run.smithi184.stdout: "osd_epoch": 169, 2024-09-16T15:51:59.382 INFO:teuthology.orchestra.run.smithi184.stdout: "osd_epoch_barrier": 0, 2024-09-16T15:51:59.382 INFO:teuthology.orchestra.run.smithi184.stdout: "blocklisted": false, 2024-09-16T15:51:59.382 INFO:teuthology.orchestra.run.smithi184.stdout: "fs_name": "cephfs" 2024-09-16T15:51:59.382 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-09-16T15:51:59.393 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:51:59.393 DEBUG:teuthology.orchestra.run.smithi184:> stat --file-system '--printf=%T 2024-09-16T15:51:59.393 DEBUG:teuthology.orchestra.run.smithi184:> ' -- /home/ubuntu/cephtest/mnt.1 2024-09-16T15:51:59.399 INFO:teuthology.orchestra.run.smithi184.stdout:fuseblk 2024-09-16T15:51:59.399 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2024-09-16T15:51:59.399 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:51:59.399 DEBUG:teuthology.orchestra.run.smithi184:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.1 2024-09-16T15:51:59.453 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs flag set enable_multiple true --yes-i-really-mean-it 2024-09-16T15:51:59.603 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:59.605+0000 7f0731f20640 1 -- 172.21.15.123:0/546249888 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f072c070640 msgr2=0x7f072c157d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:51:59.603 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:59.605+0000 7f0731f20640 1 --2- 172.21.15.123:0/546249888 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f072c070640 0x7f072c157d00 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7f0724009bc0 tx=0x7f072401cb70 comp rx=0 tx=0).stop 2024-09-16T15:51:59.603 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:59.605+0000 7f0731f20640 1 -- 172.21.15.123:0/546249888 shutdown_connections 2024-09-16T15:51:59.603 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:59.605+0000 7f0731f20640 1 --2- 172.21.15.123:0/546249888 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f072c070640 0x7f072c157d00 unknown :-1 s=CLOSED pgs=572 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:59.603 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:59.605+0000 7f0731f20640 1 --2- 172.21.15.123:0/546249888 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f072c0aee40 0x7f072c15b980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:59.603 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:59.605+0000 7f0731f20640 1 -- 172.21.15.123:0/546249888 >> 172.21.15.123:0/546249888 conn(0x7f072c0b1630 msgr2=0x7f072c0b1a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:51:59.603 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:59.605+0000 7f0731f20640 1 -- 172.21.15.123:0/546249888 shutdown_connections 2024-09-16T15:51:59.603 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:59.605+0000 7f0731f20640 1 -- 172.21.15.123:0/546249888 wait complete. 2024-09-16T15:51:59.604 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:59.605+0000 7f0731f20640 1 Processor -- start 2024-09-16T15:51:59.604 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:59.605+0000 7f0731f20640 1 -- start start 2024-09-16T15:51:59.606 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:59.605+0000 7f0731f20640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f072c070640 0x7f072c1fbbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:51:59.607 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:59.605+0000 7f0731f20640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f072c073390 0x7f072c168330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:51:59.607 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:59.605+0000 7f0731f20640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f072c0aee40 0x7f072c0ad8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:51:59.607 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:59.605+0000 7f0731f20640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f072c06f700 con 0x7f072c070640 2024-09-16T15:51:59.607 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:59.605+0000 7f0731f20640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f072c06f580 con 0x7f072c0aee40 2024-09-16T15:51:59.607 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:59.605+0000 7f0731f20640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f072c06f880 con 0x7f072c073390 2024-09-16T15:51:59.607 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:59.605+0000 7f072b7fe640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f072c070640 0x7f072c1fbbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:51:59.607 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:59.605+0000 7f072b7fe640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f072c070640 0x7f072c1fbbc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:43614/0 (socket says 172.21.15.123:43614) 2024-09-16T15:51:59.607 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:59.605+0000 7f072b7fe640 1 -- 172.21.15.123:0/2383771644 learned_addr learned my addr 172.21.15.123:0/2383771644 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:51:59.607 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:59.605+0000 7f072affd640 1 --2- 172.21.15.123:0/2383771644 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f072c073390 0x7f072c168330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:51:59.607 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:59.605+0000 7f072bfff640 1 --2- 172.21.15.123:0/2383771644 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f072c0aee40 0x7f072c0ad8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:51:59.607 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:59.605+0000 7f072b7fe640 1 -- 172.21.15.123:0/2383771644 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f072c073390 msgr2=0x7f072c168330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:51:59.607 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:59.605+0000 7f072b7fe640 1 --2- 172.21.15.123:0/2383771644 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f072c073390 0x7f072c168330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:59.608 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:59.605+0000 7f072b7fe640 1 -- 172.21.15.123:0/2383771644 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f072c0aee40 msgr2=0x7f072c0ad8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:51:59.608 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:59.605+0000 7f072b7fe640 1 --2- 172.21.15.123:0/2383771644 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f072c0aee40 0x7f072c0ad8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:51:59.608 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:59.605+0000 7f072b7fe640 1 -- 172.21.15.123:0/2383771644 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f072c1fc100 con 0x7f072c070640 2024-09-16T15:51:59.608 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:59.605+0000 7f072bfff640 1 --2- 172.21.15.123:0/2383771644 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f072c0aee40 0x7f072c0ad8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T15:51:59.608 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:59.605+0000 7f072b7fe640 1 --2- 172.21.15.123:0/2383771644 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f072c070640 0x7f072c1fbbc0 secure :-1 s=READY pgs=1909 cs=0 l=1 rev1=1 crypto rx=0x7f071c00e9f0 tx=0x7f071c00edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:51:59.608 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:59.605+0000 7f0713fff640 1 -- 172.21.15.123:0/2383771644 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f071c00cd80 con 0x7f072c070640 2024-09-16T15:51:59.608 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:59.605+0000 7f0713fff640 1 -- 172.21.15.123:0/2383771644 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f071c004020 con 0x7f072c070640 2024-09-16T15:51:59.608 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:59.605+0000 7f0731f20640 1 -- 172.21.15.123:0/2383771644 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f072c0cd8c0 con 0x7f072c070640 2024-09-16T15:51:59.608 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:59.605+0000 7f0713fff640 1 -- 172.21.15.123:0/2383771644 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f071c004470 con 0x7f072c070640 2024-09-16T15:51:59.608 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:59.605+0000 7f0731f20640 1 -- 172.21.15.123:0/2383771644 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f072c0cde80 con 0x7f072c070640 2024-09-16T15:51:59.609 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:59.609+0000 7f0731f20640 1 -- 172.21.15.123:0/2383771644 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f072c0cf7b0 con 0x7f072c070640 2024-09-16T15:51:59.609 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:59.609+0000 7f0713fff640 1 -- 172.21.15.123:0/2383771644 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 25) ==== 97998+0+0 (secure 0 0 0) 0x7f071c003600 con 0x7f072c070640 2024-09-16T15:51:59.609 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:59.609+0000 7f0713fff640 1 --2- 172.21.15.123:0/2383771644 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f07180738e0 0x7f0718075da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:51:59.609 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:59.609+0000 7f0713fff640 1 -- 172.21.15.123:0/2383771644 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(169..169 src has 1..169) ==== 5737+0+0 (secure 0 0 0) 0x7f071c093280 con 0x7f072c070640 2024-09-16T15:51:59.609 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:59.609+0000 7f072affd640 1 --2- 172.21.15.123:0/2383771644 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f07180738e0 0x7f0718075da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:51:59.609 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:59.609+0000 7f072affd640 1 --2- 172.21.15.123:0/2383771644 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f07180738e0 0x7f0718075da0 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f072c0b1600 tx=0x7f0720009040 comp rx=0 tx=0).ready entity=mgr.4150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:51:59.613 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:59.613+0000 7f0713fff640 1 -- 172.21.15.123:0/2383771644 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f071c05fd60 con 0x7f072c070640 2024-09-16T15:51:59.822 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:51:59.821+0000 7f0731f20640 1 -- 172.21.15.123:0/2383771644 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "fs flag set", "flag_name": "enable_multiple", "val": "true", "yes_i_really_mean_it": true} v 0) -- 0x7f072c0ce680 con 0x7f072c070640 2024-09-16T15:52:00.481 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:00.481+0000 7f0713fff640 1 -- 172.21.15.123:0/2383771644 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "fs flag set", "flag_name": "enable_multiple", "val": "true", "yes_i_really_mean_it": true}]=0 v116) ==== 136+0+0 (secure 0 0 0) 0x7f071c05f660 con 0x7f072c070640 2024-09-16T15:52:00.485 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:00.485+0000 7f0731f20640 1 -- 172.21.15.123:0/2383771644 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f07180738e0 msgr2=0x7f0718075da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:52:00.485 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:00.485+0000 7f0731f20640 1 --2- 172.21.15.123:0/2383771644 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f07180738e0 0x7f0718075da0 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f072c0b1600 tx=0x7f0720009040 comp rx=0 tx=0).stop 2024-09-16T15:52:00.485 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:00.485+0000 7f0731f20640 1 -- 172.21.15.123:0/2383771644 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f072c070640 msgr2=0x7f072c1fbbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:52:00.485 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:00.485+0000 7f0731f20640 1 --2- 172.21.15.123:0/2383771644 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f072c070640 0x7f072c1fbbc0 secure :-1 s=READY pgs=1909 cs=0 l=1 rev1=1 crypto rx=0x7f071c00e9f0 tx=0x7f071c00edf0 comp rx=0 tx=0).stop 2024-09-16T15:52:00.485 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:00.485+0000 7f0731f20640 1 -- 172.21.15.123:0/2383771644 shutdown_connections 2024-09-16T15:52:00.485 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:00.485+0000 7f0731f20640 1 --2- 172.21.15.123:0/2383771644 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f07180738e0 0x7f0718075da0 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:52:00.485 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:00.485+0000 7f0731f20640 1 --2- 172.21.15.123:0/2383771644 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f072c0aee40 0x7f072c0ad8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:52:00.485 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:00.485+0000 7f0731f20640 1 --2- 172.21.15.123:0/2383771644 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f072c073390 0x7f072c168330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:52:00.485 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:00.485+0000 7f0731f20640 1 --2- 172.21.15.123:0/2383771644 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f072c070640 0x7f072c1fbbc0 unknown :-1 s=CLOSED pgs=1909 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:52:00.485 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:00.485+0000 7f0731f20640 1 -- 172.21.15.123:0/2383771644 >> 172.21.15.123:0/2383771644 conn(0x7f072c0b1630 msgr2=0x7f072c15c090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:52:00.485 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:00.485+0000 7f0731f20640 1 -- 172.21.15.123:0/2383771644 shutdown_connections 2024-09-16T15:52:00.485 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:00.485+0000 7f0731f20640 1 -- 172.21.15.123:0/2383771644 wait complete. 2024-09-16T15:52:00.504 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2024-09-16T15:52:00.631 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:00.633+0000 7f4e71c51640 1 -- 172.21.15.123:0/1939615434 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4e6c074470 msgr2=0x7f4e6c0cd6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:52:00.632 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:00.633+0000 7f4e71c51640 1 --2- 172.21.15.123:0/1939615434 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4e6c074470 0x7f4e6c0cd6e0 secure :-1 s=READY pgs=1910 cs=0 l=1 rev1=1 crypto rx=0x7f4e5c009900 tx=0x7f4e5c01c850 comp rx=0 tx=0).stop 2024-09-16T15:52:00.632 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:00.633+0000 7f4e71c51640 1 -- 172.21.15.123:0/1939615434 shutdown_connections 2024-09-16T15:52:00.632 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:00.633+0000 7f4e71c51640 1 --2- 172.21.15.123:0/1939615434 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4e6c074470 0x7f4e6c0cd6e0 unknown :-1 s=CLOSED pgs=1910 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:52:00.632 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:00.633+0000 7f4e71c51640 1 --2- 172.21.15.123:0/1939615434 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f4e6c15e160 0x7f4e6c073f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:52:00.632 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:00.633+0000 7f4e71c51640 1 -- 172.21.15.123:0/1939615434 >> 172.21.15.123:0/1939615434 conn(0x7f4e6c0cfb10 msgr2=0x7f4e6c0cff20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:52:00.632 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:00.633+0000 7f4e71c51640 1 -- 172.21.15.123:0/1939615434 shutdown_connections 2024-09-16T15:52:00.632 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:00.633+0000 7f4e71c51640 1 -- 172.21.15.123:0/1939615434 wait complete. 2024-09-16T15:52:00.633 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:00.633+0000 7f4e71c51640 1 Processor -- start 2024-09-16T15:52:00.633 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:00.633+0000 7f4e71c51640 1 -- start start 2024-09-16T15:52:00.634 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:00.633+0000 7f4e71c51640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f4e6c074470 0x7f4e6c166bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:52:00.635 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:00.633+0000 7f4e71c51640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4e6c0cdc20 0x7f4e6c0b3f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:52:00.635 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:00.633+0000 7f4e71c51640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4e6c15e160 0x7f4e6c152fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:52:00.635 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:00.633+0000 7f4e71c51640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f4e6c165f40 con 0x7f4e6c0cdc20 2024-09-16T15:52:00.635 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:00.633+0000 7f4e71c51640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f4e6c165dc0 con 0x7f4e6c15e160 2024-09-16T15:52:00.635 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:00.633+0000 7f4e71c51640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f4e6c1660c0 con 0x7f4e6c074470 2024-09-16T15:52:00.635 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:00.633+0000 7f4e6affd640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4e6c0cdc20 0x7f4e6c0b3f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:52:00.635 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:00.633+0000 7f4e6affd640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4e6c0cdc20 0x7f4e6c0b3f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:43644/0 (socket says 172.21.15.123:43644) 2024-09-16T15:52:00.635 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:00.633+0000 7f4e6affd640 1 -- 172.21.15.123:0/828877124 learned_addr learned my addr 172.21.15.123:0/828877124 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:52:00.635 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:00.633+0000 7f4e6b7fe640 1 --2- 172.21.15.123:0/828877124 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f4e6c074470 0x7f4e6c166bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:52:00.635 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:00.633+0000 7f4e6affd640 1 -- 172.21.15.123:0/828877124 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f4e6c074470 msgr2=0x7f4e6c166bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:52:00.635 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:00.633+0000 7f4e6affd640 1 --2- 172.21.15.123:0/828877124 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f4e6c074470 0x7f4e6c166bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:52:00.635 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:00.633+0000 7f4e6affd640 1 -- 172.21.15.123:0/828877124 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4e6c15e160 msgr2=0x7f4e6c152fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:52:00.635 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:00.633+0000 7f4e6affd640 1 --2- 172.21.15.123:0/828877124 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4e6c15e160 0x7f4e6c152fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:52:00.635 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:00.633+0000 7f4e6affd640 1 -- 172.21.15.123:0/828877124 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4e6c1673f0 con 0x7f4e6c0cdc20 2024-09-16T15:52:00.636 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:00.633+0000 7f4e6affd640 1 --2- 172.21.15.123:0/828877124 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4e6c0cdc20 0x7f4e6c0b3f60 secure :-1 s=READY pgs=1911 cs=0 l=1 rev1=1 crypto rx=0x7f4e5c009850 tx=0x7f4e5c0a56a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:52:00.636 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:00.633+0000 7f4e53fff640 1 -- 172.21.15.123:0/828877124 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e5c0b2040 con 0x7f4e6c0cdc20 2024-09-16T15:52:00.636 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:00.633+0000 7f4e53fff640 1 -- 172.21.15.123:0/828877124 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f4e5c0044b0 con 0x7f4e6c0cdc20 2024-09-16T15:52:00.636 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:00.633+0000 7f4e71c51640 1 -- 172.21.15.123:0/828877124 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4e6c1676d0 con 0x7f4e6c0cdc20 2024-09-16T15:52:00.636 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:00.633+0000 7f4e53fff640 1 -- 172.21.15.123:0/828877124 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e5c005290 con 0x7f4e6c0cdc20 2024-09-16T15:52:00.636 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:00.633+0000 7f4e71c51640 1 -- 172.21.15.123:0/828877124 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4e6c0ada60 con 0x7f4e6c0cdc20 2024-09-16T15:52:00.636 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:00.637+0000 7f4e71c51640 1 -- 172.21.15.123:0/828877124 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4e6c167bf0 con 0x7f4e6c0cdc20 2024-09-16T15:52:00.636 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:00.637+0000 7f4e53fff640 1 -- 172.21.15.123:0/828877124 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 25) ==== 97998+0+0 (secure 0 0 0) 0x7f4e5c005430 con 0x7f4e6c0cdc20 2024-09-16T15:52:00.637 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:00.637+0000 7f4e53fff640 1 --2- 172.21.15.123:0/828877124 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f4e58073910 0x7f4e58075dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:52:00.637 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:00.637+0000 7f4e53fff640 1 -- 172.21.15.123:0/828877124 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(169..169 src has 1..169) ==== 5737+0+0 (secure 0 0 0) 0x7f4e5c12c630 con 0x7f4e6c0cdc20 2024-09-16T15:52:00.637 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:00.637+0000 7f4e6b7fe640 1 --2- 172.21.15.123:0/828877124 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f4e58073910 0x7f4e58075dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:52:00.638 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:00.637+0000 7f4e6b7fe640 1 --2- 172.21.15.123:0/828877124 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f4e58073910 0x7f4e58075dd0 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f4e60006fd0 tx=0x7f4e60008040 comp rx=0 tx=0).ready entity=mgr.4150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:52:00.641 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:00.641+0000 7f4e53fff640 1 -- 172.21.15.123:0/828877124 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f4e6c167bf0 con 0x7f4e6c0cdc20 2024-09-16T15:52:00.830 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:00.829+0000 7f4e71c51640 1 -- 172.21.15.123:0/828877124 --> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) -- 0x7f4e6c0cd6e0 con 0x7f4e58073910 2024-09-16T15:52:00.832 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:00.833+0000 7f4e53fff640 1 -- 172.21.15.123:0/828877124 <== mgr.4150 v2:172.21.15.123:6836/2464759676 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+41 (secure 0 0 0) 0x7f4e6c0cd6e0 con 0x7f4e58073910 2024-09-16T15:52:00.832 INFO:teuthology.orchestra.run.smithi123.stdout:[ 2024-09-16T15:52:00.832 INFO:teuthology.orchestra.run.smithi123.stdout: { 2024-09-16T15:52:00.832 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "cephfs" 2024-09-16T15:52:00.832 INFO:teuthology.orchestra.run.smithi123.stdout: } 2024-09-16T15:52:00.833 INFO:teuthology.orchestra.run.smithi123.stdout:] 2024-09-16T15:52:00.835 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:00.837+0000 7f4e71c51640 1 -- 172.21.15.123:0/828877124 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f4e58073910 msgr2=0x7f4e58075dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:52:00.835 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:00.837+0000 7f4e71c51640 1 --2- 172.21.15.123:0/828877124 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f4e58073910 0x7f4e58075dd0 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f4e60006fd0 tx=0x7f4e60008040 comp rx=0 tx=0).stop 2024-09-16T15:52:00.835 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:00.837+0000 7f4e71c51640 1 -- 172.21.15.123:0/828877124 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4e6c0cdc20 msgr2=0x7f4e6c0b3f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:52:00.835 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:00.837+0000 7f4e71c51640 1 --2- 172.21.15.123:0/828877124 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4e6c0cdc20 0x7f4e6c0b3f60 secure :-1 s=READY pgs=1911 cs=0 l=1 rev1=1 crypto rx=0x7f4e5c009850 tx=0x7f4e5c0a56a0 comp rx=0 tx=0).stop 2024-09-16T15:52:00.835 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:00.837+0000 7f4e71c51640 1 -- 172.21.15.123:0/828877124 shutdown_connections 2024-09-16T15:52:00.835 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:00.837+0000 7f4e71c51640 1 --2- 172.21.15.123:0/828877124 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f4e58073910 0x7f4e58075dd0 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:52:00.835 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:00.837+0000 7f4e71c51640 1 --2- 172.21.15.123:0/828877124 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f4e6c15e160 0x7f4e6c152fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:52:00.835 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:00.837+0000 7f4e71c51640 1 --2- 172.21.15.123:0/828877124 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4e6c0cdc20 0x7f4e6c0b3f60 unknown :-1 s=CLOSED pgs=1911 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:52:00.835 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:00.837+0000 7f4e71c51640 1 --2- 172.21.15.123:0/828877124 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f4e6c074470 0x7f4e6c166bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:52:00.836 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:00.837+0000 7f4e71c51640 1 -- 172.21.15.123:0/828877124 >> 172.21.15.123:0/828877124 conn(0x7f4e6c0cfb10 msgr2=0x7f4e6c0758d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:52:00.836 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:00.837+0000 7f4e71c51640 1 -- 172.21.15.123:0/828877124 shutdown_connections 2024-09-16T15:52:00.836 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:00.837+0000 7f4e71c51640 1 -- 172.21.15.123:0/828877124 wait complete. 2024-09-16T15:52:00.850 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-09-16T15:52:00.980 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:00.981+0000 7f50ae599640 1 -- 172.21.15.123:0/1535866309 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f50a80d2120 msgr2=0x7f50a80d0d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:52:00.980 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:00.981+0000 7f50ae599640 1 --2- 172.21.15.123:0/1535866309 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f50a80d2120 0x7f50a80d0d90 secure :-1 s=READY pgs=1912 cs=0 l=1 rev1=1 crypto rx=0x7f50a80b69d0 tx=0x7f509001c830 comp rx=0 tx=0).stop 2024-09-16T15:52:00.980 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:00.981+0000 7f50ae599640 1 -- 172.21.15.123:0/1535866309 shutdown_connections 2024-09-16T15:52:00.981 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:00.981+0000 7f50ae599640 1 --2- 172.21.15.123:0/1535866309 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f50a80b37a0 0x7f50a80b3c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:52:00.981 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:00.981+0000 7f50ae599640 1 --2- 172.21.15.123:0/1535866309 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f50a80d2120 0x7f50a80d0d90 unknown :-1 s=CLOSED pgs=1912 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:52:00.981 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:00.981+0000 7f50ae599640 1 --2- 172.21.15.123:0/1535866309 >> v2:172.21.15.184:3300/0 conn(0x7f50a81615c0 0x7f50a80d0850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:52:00.981 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:00.981+0000 7f50ae599640 1 -- 172.21.15.123:0/1535866309 >> 172.21.15.123:0/1535866309 conn(0x7f50a80d3e40 msgr2=0x7f50a80d4220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:52:00.981 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:00.981+0000 7f50ae599640 1 -- 172.21.15.123:0/1535866309 shutdown_connections 2024-09-16T15:52:00.981 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:00.981+0000 7f50ae599640 1 -- 172.21.15.123:0/1535866309 wait complete. 2024-09-16T15:52:00.981 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:00.981+0000 7f50ae599640 1 Processor -- start 2024-09-16T15:52:00.981 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:00.981+0000 7f50ae599640 1 -- start start 2024-09-16T15:52:00.983 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:00.981+0000 7f50ae599640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f50a80b37a0 0x7f50a806e9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:52:00.983 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:00.981+0000 7f50ae599640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f50a80d2120 0x7f50a815a510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:52:00.983 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:00.981+0000 7f50ae599640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f50a81615c0 0x7f50a8150a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:52:00.983 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:00.981+0000 7f50ae599640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f50a80cf1d0 con 0x7f50a81615c0 2024-09-16T15:52:00.983 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:00.981+0000 7f50ae599640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f50a80b3120 con 0x7f50a80b37a0 2024-09-16T15:52:00.984 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:00.981+0000 7f50ae599640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f50a80cf350 con 0x7f50a80d2120 2024-09-16T15:52:00.984 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:00.985+0000 7f50acb36640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f50a81615c0 0x7f50a8150a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:52:00.984 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:00.985+0000 7f50acb36640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f50a81615c0 0x7f50a8150a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:43666/0 (socket says 172.21.15.123:43666) 2024-09-16T15:52:00.984 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:00.985+0000 7f50acb36640 1 -- 172.21.15.123:0/1989346454 learned_addr learned my addr 172.21.15.123:0/1989346454 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:52:00.984 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:00.985+0000 7f50a7fff640 1 --2- 172.21.15.123:0/1989346454 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f50a80b37a0 0x7f50a806e9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:52:00.984 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:00.985+0000 7f50a77fe640 1 --2- 172.21.15.123:0/1989346454 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f50a80d2120 0x7f50a815a510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:52:00.984 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:00.985+0000 7f50acb36640 1 -- 172.21.15.123:0/1989346454 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f50a80d2120 msgr2=0x7f50a815a510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:52:00.984 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:00.985+0000 7f50acb36640 1 --2- 172.21.15.123:0/1989346454 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f50a80d2120 0x7f50a815a510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:52:00.984 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:00.985+0000 7f50acb36640 1 -- 172.21.15.123:0/1989346454 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f50a80b37a0 msgr2=0x7f50a806e9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:52:00.984 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:00.985+0000 7f50acb36640 1 --2- 172.21.15.123:0/1989346454 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f50a80b37a0 0x7f50a806e9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:52:00.984 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:00.985+0000 7f50acb36640 1 -- 172.21.15.123:0/1989346454 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f50a8159c80 con 0x7f50a81615c0 2024-09-16T15:52:00.984 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:00.985+0000 7f50a77fe640 1 --2- 172.21.15.123:0/1989346454 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f50a80d2120 0x7f50a815a510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T15:52:00.984 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:00.985+0000 7f50acb36640 1 --2- 172.21.15.123:0/1989346454 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f50a81615c0 0x7f50a8150a00 secure :-1 s=READY pgs=1913 cs=0 l=1 rev1=1 crypto rx=0x7f50a000e990 tx=0x7f50a000ee50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:52:00.985 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:00.985+0000 7f50a4ff9640 1 -- 172.21.15.123:0/1989346454 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50a0004d30 con 0x7f50a81615c0 2024-09-16T15:52:00.985 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:00.985+0000 7f50a4ff9640 1 -- 172.21.15.123:0/1989346454 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f50a000cb00 con 0x7f50a81615c0 2024-09-16T15:52:00.985 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:00.985+0000 7f50ae599640 1 -- 172.21.15.123:0/1989346454 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f50a806def0 con 0x7f50a81615c0 2024-09-16T15:52:00.985 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:00.985+0000 7f50a4ff9640 1 -- 172.21.15.123:0/1989346454 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50a0005020 con 0x7f50a81615c0 2024-09-16T15:52:00.985 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:00.985+0000 7f50ae599640 1 -- 172.21.15.123:0/1989346454 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f50a81518b0 con 0x7f50a81615c0 2024-09-16T15:52:00.986 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:00.985+0000 7f50ae599640 1 -- 172.21.15.123:0/1989346454 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f50a80b3c30 con 0x7f50a81615c0 2024-09-16T15:52:00.986 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:00.985+0000 7f50a4ff9640 1 -- 172.21.15.123:0/1989346454 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 25) ==== 97998+0+0 (secure 0 0 0) 0x7f50a00051c0 con 0x7f50a81615c0 2024-09-16T15:52:00.986 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:00.985+0000 7f50a4ff9640 1 --2- 172.21.15.123:0/1989346454 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f50880738e0 0x7f5088075da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:52:00.986 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:00.985+0000 7f50a7fff640 1 --2- 172.21.15.123:0/1989346454 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f50880738e0 0x7f5088075da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:52:00.986 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:00.985+0000 7f50a4ff9640 1 -- 172.21.15.123:0/1989346454 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(169..169 src has 1..169) ==== 5737+0+0 (secure 0 0 0) 0x7f50a00936c0 con 0x7f50a81615c0 2024-09-16T15:52:00.986 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:00.985+0000 7f50a7fff640 1 --2- 172.21.15.123:0/1989346454 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f50880738e0 0x7f5088075da0 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f50980058d0 tx=0x7f5098005860 comp rx=0 tx=0).ready entity=mgr.4150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:52:00.989 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:00.989+0000 7f50a4ff9640 1 -- 172.21.15.123:0/1989346454 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f50a0016030 con 0x7f50a81615c0 2024-09-16T15:52:01.149 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:01.149+0000 7f50ae599640 1 -- 172.21.15.123:0/1989346454 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) -- 0x7f50a80b1260 con 0x7f50a81615c0 2024-09-16T15:52:01.149 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:01.149+0000 7f50a4ff9640 1 -- 172.21.15.123:0/1989346454 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v48)=0 v48) ==== 122+0+0 (secure 0 0 0) 0x7f50a0060180 con 0x7f50a81615c0 2024-09-16T15:52:01.153 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:01.153+0000 7f50ae599640 1 -- 172.21.15.123:0/1989346454 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f50880738e0 msgr2=0x7f5088075da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:52:01.153 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:01.153+0000 7f50ae599640 1 --2- 172.21.15.123:0/1989346454 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f50880738e0 0x7f5088075da0 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f50980058d0 tx=0x7f5098005860 comp rx=0 tx=0).stop 2024-09-16T15:52:01.153 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:01.153+0000 7f50ae599640 1 -- 172.21.15.123:0/1989346454 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f50a81615c0 msgr2=0x7f50a8150a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:52:01.153 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:01.153+0000 7f50ae599640 1 --2- 172.21.15.123:0/1989346454 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f50a81615c0 0x7f50a8150a00 secure :-1 s=READY pgs=1913 cs=0 l=1 rev1=1 crypto rx=0x7f50a000e990 tx=0x7f50a000ee50 comp rx=0 tx=0).stop 2024-09-16T15:52:01.153 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:01.153+0000 7f50ae599640 1 -- 172.21.15.123:0/1989346454 shutdown_connections 2024-09-16T15:52:01.153 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:01.153+0000 7f50ae599640 1 --2- 172.21.15.123:0/1989346454 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f50880738e0 0x7f5088075da0 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:52:01.153 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:01.153+0000 7f50ae599640 1 --2- 172.21.15.123:0/1989346454 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f50a81615c0 0x7f50a8150a00 unknown :-1 s=CLOSED pgs=1913 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:52:01.153 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:01.153+0000 7f50ae599640 1 --2- 172.21.15.123:0/1989346454 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f50a80d2120 0x7f50a815a510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:52:01.153 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:01.153+0000 7f50ae599640 1 --2- 172.21.15.123:0/1989346454 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f50a80b37a0 0x7f50a806e9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:52:01.153 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:01.153+0000 7f50ae599640 1 -- 172.21.15.123:0/1989346454 >> 172.21.15.123:0/1989346454 conn(0x7f50a80d3e40 msgr2=0x7f50a80d42b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:52:01.153 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:01.153+0000 7f50ae599640 1 -- 172.21.15.123:0/1989346454 shutdown_connections 2024-09-16T15:52:01.153 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:01.153+0000 7f50ae599640 1 -- 172.21.15.123:0/1989346454 wait complete. 2024-09-16T15:52:01.169 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:52:01.169 DEBUG:teuthology.orchestra.run.smithi123:> (cd /home/ubuntu/cephtest/mnt.0 && exec sudo mkdir -p ./volumes/subvol_grp2240/subvol53650) 2024-09-16T15:52:01.192 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:52:01.192 DEBUG:teuthology.orchestra.run.smithi123:> (cd /home/ubuntu/cephtest/mnt.0 && exec getfattr --only-values -n ceph.dir.layout.pool .) 2024-09-16T15:52:01.239 INFO:teuthology.orchestra.run.smithi123.stdout:cephfs_data 2024-09-16T15:52:01.240 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:52:01.240 DEBUG:teuthology.orchestra.run.smithi123:> (cd /home/ubuntu/cephtest/mnt.0 && exec sudo setfattr -n ceph.dir.layout.pool -v cephfs_data ./volumes/subvol_grp2240/subvol53650) 2024-09-16T15:52:01.300 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2024-09-16T15:52:01.300 DEBUG:teuthology.orchestra.run.smithi123:> sudo dd of=/home/ubuntu/cephtest/mnt.0/./volumes/subvol_grp2240/subvol53650/.meta 2024-09-16T15:52:01.358 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs subvol53650 subvol_grp2240 2024-09-16T15:52:01.530 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:01.529+0000 7f80941a6640 1 -- 172.21.15.123:0/2967911541 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8078003a40 msgr2=0x7f8078003e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:52:01.530 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:01.529+0000 7f80941a6640 1 --2- 172.21.15.123:0/2967911541 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8078003a40 0x7f8078003e60 secure :-1 s=READY pgs=1915 cs=0 l=1 rev1=1 crypto rx=0x7f807c00c2e0 tx=0x7f807c01f680 comp rx=0 tx=0).stop 2024-09-16T15:52:01.530 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:01.529+0000 7f8092743640 1 -- 172.21.15.123:0/2967911541 reap_dead start 2024-09-16T15:52:01.530 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:01.529+0000 7f80941a6640 1 -- 172.21.15.123:0/2967911541 shutdown_connections 2024-09-16T15:52:01.530 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:01.529+0000 7f80941a6640 1 -- 172.21.15.123:0/2967911541 >> 172.21.15.123:0/2967911541 conn(0x7f808c0b1630 msgr2=0x7f808c0b1a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:52:01.530 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:01.529+0000 7f80941a6640 1 -- 172.21.15.123:0/2967911541 shutdown_connections 2024-09-16T15:52:01.531 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:01.529+0000 7f80941a6640 1 -- 172.21.15.123:0/2967911541 wait complete. 2024-09-16T15:52:01.531 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:01.533+0000 7f80941a6640 1 Processor -- start 2024-09-16T15:52:01.531 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:01.533+0000 7f80941a6640 1 -- start start 2024-09-16T15:52:01.533 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:01.533+0000 7f80941a6640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f8078003a40 0x7f808c1ef9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:52:01.533 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:01.533+0000 7f80941a6640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f808c156a90 0x7f808c156ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:52:01.533 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:01.533+0000 7f80941a6640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f808c0b2e40 0x7f808c0b32b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:52:01.533 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:01.533+0000 7f80941a6640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f808c06e590 con 0x7f808c0b2e40 2024-09-16T15:52:01.533 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:01.533+0000 7f80941a6640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f808c0708e0 con 0x7f808c156a90 2024-09-16T15:52:01.533 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:01.533+0000 7f80941a6640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f808c06e710 con 0x7f8078003a40 2024-09-16T15:52:01.533 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:01.533+0000 7f8092743640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f808c0b2e40 0x7f808c0b32b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:52:01.533 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:01.533+0000 7f8092743640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f808c0b2e40 0x7f808c0b32b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:43676/0 (socket says 172.21.15.123:43676) 2024-09-16T15:52:01.533 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:01.533+0000 7f8092743640 1 -- 172.21.15.123:0/265262949 learned_addr learned my addr 172.21.15.123:0/265262949 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:52:01.533 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:01.533+0000 7f8091741640 1 --2- 172.21.15.123:0/265262949 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f808c156a90 0x7f808c156ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:52:01.533 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:01.533+0000 7f8092743640 1 -- 172.21.15.123:0/265262949 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f8078003a40 msgr2=0x7f808c1ef9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:52:01.533 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:01.533+0000 7f8092743640 1 --2- 172.21.15.123:0/265262949 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f8078003a40 0x7f808c1ef9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:52:01.533 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:01.533+0000 7f8092743640 1 -- 172.21.15.123:0/265262949 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f808c156a90 msgr2=0x7f808c156ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:52:01.534 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:01.533+0000 7f8092743640 1 --2- 172.21.15.123:0/265262949 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f808c156a90 0x7f808c156ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:52:01.534 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:01.533+0000 7f8092743640 1 -- 172.21.15.123:0/265262949 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f808c152660 con 0x7f808c0b2e40 2024-09-16T15:52:01.534 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:01.533+0000 7f8092743640 1 --2- 172.21.15.123:0/265262949 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f808c0b2e40 0x7f808c0b32b0 secure :-1 s=READY pgs=1916 cs=0 l=1 rev1=1 crypto rx=0x7f808c0733d0 tx=0x7f808400fe10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:52:01.534 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:01.533+0000 7f80827fc640 1 -- 172.21.15.123:0/265262949 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8084004d30 con 0x7f808c0b2e40 2024-09-16T15:52:01.534 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:01.533+0000 7f80827fc640 1 -- 172.21.15.123:0/265262949 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f808400daa0 con 0x7f808c0b2e40 2024-09-16T15:52:01.534 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:01.533+0000 7f80941a6640 1 -- 172.21.15.123:0/265262949 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f808c152940 con 0x7f808c0b2e40 2024-09-16T15:52:01.534 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:01.533+0000 7f80827fc640 1 -- 172.21.15.123:0/265262949 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8084005020 con 0x7f808c0b2e40 2024-09-16T15:52:01.534 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:01.533+0000 7f80941a6640 1 -- 172.21.15.123:0/265262949 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f808c152f00 con 0x7f808c0b2e40 2024-09-16T15:52:01.535 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:01.533+0000 7f80941a6640 1 -- 172.21.15.123:0/265262949 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f808c0725e0 con 0x7f808c0b2e40 2024-09-16T15:52:01.535 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:01.537+0000 7f80827fc640 1 -- 172.21.15.123:0/265262949 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 25) ==== 97998+0+0 (secure 0 0 0) 0x7f808400c030 con 0x7f808c0b2e40 2024-09-16T15:52:01.535 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:01.537+0000 7f80827fc640 1 --2- 172.21.15.123:0/265262949 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f80700738e0 0x7f8070075da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:52:01.535 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:01.537+0000 7f8091f42640 1 --2- 172.21.15.123:0/265262949 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f80700738e0 0x7f8070075da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:52:01.535 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:01.537+0000 7f80827fc640 1 -- 172.21.15.123:0/265262949 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(169..169 src has 1..169) ==== 5737+0+0 (secure 0 0 0) 0x7f8084092ee0 con 0x7f808c0b2e40 2024-09-16T15:52:01.535 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:01.537+0000 7f8091f42640 1 --2- 172.21.15.123:0/265262949 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f80700738e0 0x7f8070075da0 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f807c0b0000 tx=0x7f8078005840 comp rx=0 tx=0).ready entity=mgr.4150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:52:01.539 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:01.541+0000 7f80827fc640 1 -- 172.21.15.123:0/265262949 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f808401f060 con 0x7f808c0b2e40 2024-09-16T15:52:01.738 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:01.737+0000 7f80941a6640 1 -- 172.21.15.123:0/265262949 --> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "subvol53650", "group_name": "subvol_grp2240", "target": ["mon-mgr", ""]}) -- 0x7f808c1510c0 con 0x7f80700738e0 2024-09-16T15:52:01.746 INFO:tasks.ceph.mgr.y.smithi123.stderr:2024-09-16T15:52:01.745+0000 7fa9b36f4640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-16T15:52:01.746 INFO:tasks.ceph.mgr.y.smithi123.stderr:2024-09-16T15:52:01.745+0000 7fa9b36f4640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-16T15:52:01.746 INFO:tasks.ceph.mgr.y.smithi123.stderr:2024-09-16T15:52:01.745+0000 7fa9b36f4640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-16T15:52:01.746 INFO:tasks.ceph.mgr.y.smithi123.stderr:2024-09-16T15:52:01.745+0000 7fa9b36f4640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-16T15:52:01.747 INFO:tasks.ceph.mgr.y.smithi123.stderr:2024-09-16T15:52:01.745+0000 7fa9b36f4640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-16T15:52:01.787 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:01.789+0000 7f80827fc640 1 -- 172.21.15.123:0/265262949 <== mgr.4150 v2:172.21.15.123:6836/2464759676 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+36 (secure 0 0 0) 0x7f808c1510c0 con 0x7f80700738e0 2024-09-16T15:52:01.787 INFO:teuthology.orchestra.run.smithi123.stdout:/volumes/subvol_grp2240/subvol53650 2024-09-16T15:52:01.789 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:01.789+0000 7f80941a6640 1 -- 172.21.15.123:0/265262949 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f80700738e0 msgr2=0x7f8070075da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:52:01.789 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:01.789+0000 7f80941a6640 1 --2- 172.21.15.123:0/265262949 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f80700738e0 0x7f8070075da0 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f807c0b0000 tx=0x7f8078005840 comp rx=0 tx=0).stop 2024-09-16T15:52:01.789 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:01.789+0000 7f80941a6640 1 -- 172.21.15.123:0/265262949 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f808c0b2e40 msgr2=0x7f808c0b32b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:52:01.789 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:01.789+0000 7f80941a6640 1 --2- 172.21.15.123:0/265262949 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f808c0b2e40 0x7f808c0b32b0 secure :-1 s=READY pgs=1916 cs=0 l=1 rev1=1 crypto rx=0x7f808c0733d0 tx=0x7f808400fe10 comp rx=0 tx=0).stop 2024-09-16T15:52:01.790 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:01.789+0000 7f80941a6640 1 -- 172.21.15.123:0/265262949 shutdown_connections 2024-09-16T15:52:01.790 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:01.789+0000 7f80941a6640 1 --2- 172.21.15.123:0/265262949 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f80700738e0 0x7f8070075da0 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:52:01.790 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:01.789+0000 7f80941a6640 1 --2- 172.21.15.123:0/265262949 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f808c0b2e40 0x7f808c0b32b0 unknown :-1 s=CLOSED pgs=1916 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:52:01.790 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:01.789+0000 7f80941a6640 1 --2- 172.21.15.123:0/265262949 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f808c156a90 0x7f808c156ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:52:01.790 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:01.789+0000 7f80941a6640 1 --2- 172.21.15.123:0/265262949 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f8078003a40 0x7f808c1ef9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:52:01.790 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:01.789+0000 7f80941a6640 1 -- 172.21.15.123:0/265262949 >> 172.21.15.123:0/265262949 conn(0x7f808c0b1630 msgr2=0x7f808c0b03c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:52:01.790 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:01.789+0000 7f80941a6640 1 -- 172.21.15.123:0/265262949 shutdown_connections 2024-09-16T15:52:01.790 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:01.789+0000 7f80941a6640 1 -- 172.21.15.123:0/265262949 wait complete. 2024-09-16T15:52:01.804 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs subvol53650 subvol_grp2240 2024-09-16T15:52:01.920 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:01.921+0000 7fc6cc451640 1 -- 172.21.15.123:0/2366629090 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc6c4074440 msgr2=0x7fc6c40cdf40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:52:01.920 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:01.921+0000 7fc6cc451640 1 --2- 172.21.15.123:0/2366629090 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc6c4074440 0x7fc6c40cdf40 secure :-1 s=READY pgs=1920 cs=0 l=1 rev1=1 crypto rx=0x7fc6bc009840 tx=0x7fc6bc01c820 comp rx=0 tx=0).stop 2024-09-16T15:52:01.920 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:01.921+0000 7fc6cc451640 1 -- 172.21.15.123:0/2366629090 shutdown_connections 2024-09-16T15:52:01.920 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:01.921+0000 7fc6cc451640 1 --2- 172.21.15.123:0/2366629090 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc6c4074440 0x7fc6c40cdf40 unknown :-1 s=CLOSED pgs=1920 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:52:01.920 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:01.921+0000 7fc6cc451640 1 --2- 172.21.15.123:0/2366629090 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fc6c415fd80 0x7fc6c40704e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:52:01.921 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:01.921+0000 7fc6cc451640 1 -- 172.21.15.123:0/2366629090 >> 172.21.15.123:0/2366629090 conn(0x7fc6c40b1630 msgr2=0x7fc6c40b1a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:52:01.921 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:01.921+0000 7fc6cc451640 1 -- 172.21.15.123:0/2366629090 shutdown_connections 2024-09-16T15:52:01.921 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:01.921+0000 7fc6cc451640 1 -- 172.21.15.123:0/2366629090 wait complete. 2024-09-16T15:52:01.921 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:01.921+0000 7fc6cc451640 1 Processor -- start 2024-09-16T15:52:01.921 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:01.921+0000 7fc6cc451640 1 -- start start 2024-09-16T15:52:01.923 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:01.921+0000 7fc6cc451640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fc6c4074440 0x7fc6c41428d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:52:01.924 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:01.921+0000 7fc6cc451640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc6c4159680 0x7fc6c41f9850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:52:01.924 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:01.921+0000 7fc6cc451640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc6c415fd80 0x7fc6c41582b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:52:01.924 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:01.921+0000 7fc6cc451640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fc6c4168da0 con 0x7fc6c4159680 2024-09-16T15:52:01.924 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:01.921+0000 7fc6cc451640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fc6c4168c20 con 0x7fc6c415fd80 2024-09-16T15:52:01.924 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:01.921+0000 7fc6cc451640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fc6c4168f20 con 0x7fc6c4074440 2024-09-16T15:52:01.924 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:01.921+0000 7fc6c99ec640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc6c4159680 0x7fc6c41f9850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:52:01.924 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:01.921+0000 7fc6c99ec640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc6c4159680 0x7fc6c41f9850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:43722/0 (socket says 172.21.15.123:43722) 2024-09-16T15:52:01.924 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:01.921+0000 7fc6c99ec640 1 -- 172.21.15.123:0/4146583967 learned_addr learned my addr 172.21.15.123:0/4146583967 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:52:01.924 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:01.925+0000 7fc6ca1ed640 1 --2- 172.21.15.123:0/4146583967 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fc6c4074440 0x7fc6c41428d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:52:01.924 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:01.925+0000 7fc6c99ec640 1 -- 172.21.15.123:0/4146583967 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fc6c4074440 msgr2=0x7fc6c41428d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:52:01.924 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:01.925+0000 7fc6ca9ee640 1 --2- 172.21.15.123:0/4146583967 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc6c415fd80 0x7fc6c41582b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:52:01.924 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:01.925+0000 7fc6c99ec640 1 --2- 172.21.15.123:0/4146583967 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fc6c4074440 0x7fc6c41428d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:52:01.924 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:01.925+0000 7fc6c99ec640 1 -- 172.21.15.123:0/4146583967 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc6c415fd80 msgr2=0x7fc6c41582b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:52:01.925 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:01.925+0000 7fc6c99ec640 1 --2- 172.21.15.123:0/4146583967 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc6c415fd80 0x7fc6c41582b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:52:01.925 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:01.925+0000 7fc6c99ec640 1 -- 172.21.15.123:0/4146583967 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc6c4143100 con 0x7fc6c4159680 2024-09-16T15:52:01.925 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:01.925+0000 7fc6c99ec640 1 --2- 172.21.15.123:0/4146583967 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc6c4159680 0x7fc6c41f9850 secure :-1 s=READY pgs=1921 cs=0 l=1 rev1=1 crypto rx=0x7fc6b400b410 tx=0x7fc6b400b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:52:01.925 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:01.925+0000 7fc6baffd640 1 -- 172.21.15.123:0/4146583967 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6b4015020 con 0x7fc6c4159680 2024-09-16T15:52:01.925 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:01.925+0000 7fc6baffd640 1 -- 172.21.15.123:0/4146583967 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fc6b4011920 con 0x7fc6c4159680 2024-09-16T15:52:01.925 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:01.925+0000 7fc6cc451640 1 -- 172.21.15.123:0/4146583967 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc6c41433e0 con 0x7fc6c4159680 2024-09-16T15:52:01.925 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:01.925+0000 7fc6baffd640 1 -- 172.21.15.123:0/4146583967 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6b4011d70 con 0x7fc6c4159680 2024-09-16T15:52:01.925 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:01.925+0000 7fc6cc451640 1 -- 172.21.15.123:0/4146583967 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc6c41f9050 con 0x7fc6c4159680 2024-09-16T15:52:01.925 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:01.925+0000 7fc6cc451640 1 -- 172.21.15.123:0/4146583967 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc6c41f9540 con 0x7fc6c4159680 2024-09-16T15:52:01.926 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:01.925+0000 7fc6baffd640 1 -- 172.21.15.123:0/4146583967 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 25) ==== 97998+0+0 (secure 0 0 0) 0x7fc6b4018020 con 0x7fc6c4159680 2024-09-16T15:52:01.926 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:01.925+0000 7fc6baffd640 1 --2- 172.21.15.123:0/4146583967 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fc6a80738e0 0x7fc6a8075da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:52:01.926 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:01.925+0000 7fc6baffd640 1 -- 172.21.15.123:0/4146583967 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(169..169 src has 1..169) ==== 5737+0+0 (secure 0 0 0) 0x7fc6b4092810 con 0x7fc6c4159680 2024-09-16T15:52:01.926 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:01.925+0000 7fc6ca1ed640 1 --2- 172.21.15.123:0/4146583967 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fc6a80738e0 0x7fc6a8075da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:52:01.926 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:01.925+0000 7fc6ca1ed640 1 --2- 172.21.15.123:0/4146583967 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fc6a80738e0 0x7fc6a8075da0 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7fc6c4073b70 tx=0x7fc6c0009040 comp rx=0 tx=0).ready entity=mgr.4150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:52:01.929 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:01.929+0000 7fc6baffd640 1 -- 172.21.15.123:0/4146583967 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7fc6b401f020 con 0x7fc6c4159680 2024-09-16T15:52:02.138 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:02.137+0000 7fc6cc451640 1 -- 172.21.15.123:0/4146583967 --> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "subvol53650", "group_name": "subvol_grp2240", "target": ["mon-mgr", ""]}) -- 0x7fc6c40cdf40 con 0x7fc6a80738e0 2024-09-16T15:52:02.146 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:02.145+0000 7fc6baffd640 1 -- 172.21.15.123:0/4146583967 <== mgr.4150 v2:172.21.15.123:6836/2464759676 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+36 (secure 0 0 0) 0x7fc6c40cdf40 con 0x7fc6a80738e0 2024-09-16T15:52:02.146 INFO:teuthology.orchestra.run.smithi123.stdout:/volumes/subvol_grp2240/subvol53650 2024-09-16T15:52:02.149 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:02.149+0000 7fc6cc451640 1 -- 172.21.15.123:0/4146583967 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fc6a80738e0 msgr2=0x7fc6a8075da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:52:02.149 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:02.149+0000 7fc6cc451640 1 --2- 172.21.15.123:0/4146583967 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fc6a80738e0 0x7fc6a8075da0 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7fc6c4073b70 tx=0x7fc6c0009040 comp rx=0 tx=0).stop 2024-09-16T15:52:02.149 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:02.149+0000 7fc6cc451640 1 -- 172.21.15.123:0/4146583967 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc6c4159680 msgr2=0x7fc6c41f9850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:52:02.149 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:02.149+0000 7fc6cc451640 1 --2- 172.21.15.123:0/4146583967 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc6c4159680 0x7fc6c41f9850 secure :-1 s=READY pgs=1921 cs=0 l=1 rev1=1 crypto rx=0x7fc6b400b410 tx=0x7fc6b400b8d0 comp rx=0 tx=0).stop 2024-09-16T15:52:02.149 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:02.149+0000 7fc6cc451640 1 -- 172.21.15.123:0/4146583967 shutdown_connections 2024-09-16T15:52:02.149 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:02.149+0000 7fc6cc451640 1 --2- 172.21.15.123:0/4146583967 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fc6a80738e0 0x7fc6a8075da0 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:52:02.149 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:02.149+0000 7fc6cc451640 1 --2- 172.21.15.123:0/4146583967 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fc6c415fd80 0x7fc6c41582b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:52:02.149 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:02.149+0000 7fc6cc451640 1 --2- 172.21.15.123:0/4146583967 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc6c4159680 0x7fc6c41f9850 unknown :-1 s=CLOSED pgs=1921 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:52:02.149 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:02.149+0000 7fc6cc451640 1 --2- 172.21.15.123:0/4146583967 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fc6c4074440 0x7fc6c41428d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:52:02.150 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:02.149+0000 7fc6cc451640 1 -- 172.21.15.123:0/4146583967 >> 172.21.15.123:0/4146583967 conn(0x7fc6c40b1630 msgr2=0x7fc6c415d1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:52:02.150 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:02.149+0000 7fc6cc451640 1 -- 172.21.15.123:0/4146583967 shutdown_connections 2024-09-16T15:52:02.150 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:02.149+0000 7fc6cc451640 1 -- 172.21.15.123:0/4146583967 wait complete. 2024-09-16T15:52:02.164 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:52:02.164 DEBUG:teuthology.orchestra.run.smithi123:> (cd /home/ubuntu/cephtest/mnt.0 && exec sudo cat ./volumes/_legacy/4f34ccf347c9047959d5036808fdd3bf.meta) 2024-09-16T15:52:03.820 INFO:teuthology.orchestra.run.smithi123.stdout:[GLOBAL] 2024-09-16T15:52:03.820 INFO:teuthology.orchestra.run.smithi123.stdout:version = 1 2024-09-16T15:52:03.820 INFO:teuthology.orchestra.run.smithi123.stdout:type = subvolume 2024-09-16T15:52:03.820 INFO:teuthology.orchestra.run.smithi123.stdout:path = /volumes/subvol_grp2240/subvol53650 2024-09-16T15:52:03.820 INFO:teuthology.orchestra.run.smithi123.stdout:state = complete 2024-09-16T15:52:03.820 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-09-16T15:52:03.822 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume rm cephfs subvol53650 subvol_grp2240 2024-09-16T15:52:03.961 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:03.961+0000 7ffab7cb5640 1 -- 172.21.15.123:0/1105265638 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ffab015a1b0 msgr2=0x7ffab0072fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:52:03.961 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:03.961+0000 7ffab7cb5640 1 --2- 172.21.15.123:0/1105265638 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ffab015a1b0 0x7ffab0072fb0 secure :-1 s=READY pgs=1923 cs=0 l=1 rev1=1 crypto rx=0x7ffab00b2510 tx=0x7ffaac01c800 comp rx=0 tx=0).stop 2024-09-16T15:52:03.961 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:03.961+0000 7ffab7cb5640 1 -- 172.21.15.123:0/1105265638 shutdown_connections 2024-09-16T15:52:03.962 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:03.961+0000 7ffab7cb5640 1 --2- 172.21.15.123:0/1105265638 >> v2:172.21.15.184:3300/0 conn(0x7ffab015dc40 0x7ffab015e090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:52:03.962 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:03.961+0000 7ffab7cb5640 1 --2- 172.21.15.123:0/1105265638 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ffab015a1b0 0x7ffab0072fb0 unknown :-1 s=CLOSED pgs=1923 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:52:03.962 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:03.961+0000 7ffab7cb5640 1 -- 172.21.15.123:0/1105265638 >> 172.21.15.123:0/1105265638 conn(0x7ffab00b1630 msgr2=0x7ffab00b1a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:52:03.962 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:03.961+0000 7ffab7cb5640 1 -- 172.21.15.123:0/1105265638 shutdown_connections 2024-09-16T15:52:03.962 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:03.961+0000 7ffab7cb5640 1 -- 172.21.15.123:0/1105265638 wait complete. 2024-09-16T15:52:03.962 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:03.965+0000 7ffab7cb5640 1 Processor -- start 2024-09-16T15:52:03.962 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:03.965+0000 7ffab7cb5640 1 -- start start 2024-09-16T15:52:03.964 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:03.965+0000 7ffab7cb5640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ffab015a1b0 0x7ffab00cd680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:52:03.964 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:03.965+0000 7ffab7cb5640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ffab015a590 0x7ffab00cdbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:52:03.964 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:03.965+0000 7ffab7cb5640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7ffab015dc40 0x7ffab0155220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:52:03.964 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:03.965+0000 7ffab7cb5640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7ffab015b250 con 0x7ffab015a590 2024-09-16T15:52:03.964 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:03.965+0000 7ffab7cb5640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7ffab015b0d0 con 0x7ffab015a1b0 2024-09-16T15:52:03.965 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:03.965+0000 7ffab7cb5640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7ffab015b3d0 con 0x7ffab015dc40 2024-09-16T15:52:03.965 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:03.965+0000 7ffab5250640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ffab015a590 0x7ffab00cdbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:52:03.965 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:03.965+0000 7ffab5250640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ffab015a590 0x7ffab00cdbc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:55318/0 (socket says 172.21.15.123:55318) 2024-09-16T15:52:03.965 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:03.965+0000 7ffab5250640 1 -- 172.21.15.123:0/1108389717 learned_addr learned my addr 172.21.15.123:0/1108389717 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:52:03.965 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:03.965+0000 7ffab5a51640 1 --2- 172.21.15.123:0/1108389717 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ffab015a1b0 0x7ffab00cd680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:52:03.965 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:03.965+0000 7ffab6252640 1 --2- 172.21.15.123:0/1108389717 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7ffab015dc40 0x7ffab0155220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:52:03.965 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:03.965+0000 7ffab5250640 1 -- 172.21.15.123:0/1108389717 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7ffab015dc40 msgr2=0x7ffab0155220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:52:03.965 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:03.965+0000 7ffab5250640 1 --2- 172.21.15.123:0/1108389717 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7ffab015dc40 0x7ffab0155220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:52:03.965 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:03.965+0000 7ffab5250640 1 -- 172.21.15.123:0/1108389717 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ffab015a1b0 msgr2=0x7ffab00cd680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:52:03.965 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:03.965+0000 7ffab5250640 1 --2- 172.21.15.123:0/1108389717 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ffab015a1b0 0x7ffab00cd680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:52:03.965 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:03.965+0000 7ffab5250640 1 -- 172.21.15.123:0/1108389717 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffab00aa3b0 con 0x7ffab015a590 2024-09-16T15:52:03.965 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:03.965+0000 7ffab5250640 1 --2- 172.21.15.123:0/1108389717 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ffab015a590 0x7ffab00cdbc0 secure :-1 s=READY pgs=1924 cs=0 l=1 rev1=1 crypto rx=0x7ffaa000c9f0 tx=0x7ffaa000ceb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:52:03.966 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:03.965+0000 7ffaa67fc640 1 -- 172.21.15.123:0/1108389717 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffaa0007bb0 con 0x7ffab015a590 2024-09-16T15:52:03.966 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:03.965+0000 7ffaa67fc640 1 -- 172.21.15.123:0/1108389717 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7ffaa0007e40 con 0x7ffab015a590 2024-09-16T15:52:03.966 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:03.965+0000 7ffab7cb5640 1 -- 172.21.15.123:0/1108389717 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffab00aa690 con 0x7ffab015a590 2024-09-16T15:52:03.966 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:03.965+0000 7ffaa67fc640 1 -- 172.21.15.123:0/1108389717 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffaa000f690 con 0x7ffab015a590 2024-09-16T15:52:03.966 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:03.965+0000 7ffab7cb5640 1 -- 172.21.15.123:0/1108389717 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffab00aa970 con 0x7ffab015a590 2024-09-16T15:52:03.967 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:03.965+0000 7ffab7cb5640 1 -- 172.21.15.123:0/1108389717 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ffab0072fb0 con 0x7ffab015a590 2024-09-16T15:52:03.967 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:03.965+0000 7ffaa67fc640 1 -- 172.21.15.123:0/1108389717 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 26) ==== 98049+0+0 (secure 0 0 0) 0x7ffaa000f830 con 0x7ffab015a590 2024-09-16T15:52:03.967 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:03.969+0000 7ffaa67fc640 1 --2- 172.21.15.123:0/1108389717 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7ffa94073970 0x7ffa94075e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:52:03.967 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:03.969+0000 7ffab5a51640 1 --2- 172.21.15.123:0/1108389717 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7ffa94073970 0x7ffa94075e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:52:03.967 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:03.969+0000 7ffaa67fc640 1 -- 172.21.15.123:0/1108389717 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(169..169 src has 1..169) ==== 5737+0+0 (secure 0 0 0) 0x7ffaa0093140 con 0x7ffab015a590 2024-09-16T15:52:03.967 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:03.969+0000 7ffab5a51640 1 --2- 172.21.15.123:0/1108389717 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7ffa94073970 0x7ffa94075e30 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7ffaac0af000 tx=0x7ffaac01b040 comp rx=0 tx=0).ready entity=mgr.4150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:52:03.971 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:03.973+0000 7ffaa67fc640 1 -- 172.21.15.123:0/1108389717 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7ffaa0014030 con 0x7ffab015a590 2024-09-16T15:52:04.171 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:04.173+0000 7ffab7cb5640 1 -- 172.21.15.123:0/1108389717 --> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] -- mgr_command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "subvol53650", "group_name": "subvol_grp2240", "target": ["mon-mgr", ""]}) -- 0x7ffab01582b0 con 0x7ffa94073970 2024-09-16T15:52:04.195 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:04.197+0000 7ffaa67fc640 1 -- 172.21.15.123:0/1108389717 <== mgr.4150 v2:172.21.15.123:6836/2464759676 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7ffab01582b0 con 0x7ffa94073970 2024-09-16T15:52:04.197 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:04.197+0000 7ffab7cb5640 1 -- 172.21.15.123:0/1108389717 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7ffa94073970 msgr2=0x7ffa94075e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:52:04.197 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:04.197+0000 7ffab7cb5640 1 --2- 172.21.15.123:0/1108389717 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7ffa94073970 0x7ffa94075e30 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7ffaac0af000 tx=0x7ffaac01b040 comp rx=0 tx=0).stop 2024-09-16T15:52:04.197 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:04.197+0000 7ffab7cb5640 1 -- 172.21.15.123:0/1108389717 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ffab015a590 msgr2=0x7ffab00cdbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:52:04.198 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:04.197+0000 7ffab7cb5640 1 --2- 172.21.15.123:0/1108389717 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ffab015a590 0x7ffab00cdbc0 secure :-1 s=READY pgs=1924 cs=0 l=1 rev1=1 crypto rx=0x7ffaa000c9f0 tx=0x7ffaa000ceb0 comp rx=0 tx=0).stop 2024-09-16T15:52:04.198 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:04.197+0000 7ffab7cb5640 1 -- 172.21.15.123:0/1108389717 shutdown_connections 2024-09-16T15:52:04.198 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:04.197+0000 7ffab7cb5640 1 --2- 172.21.15.123:0/1108389717 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7ffa94073970 0x7ffa94075e30 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:52:04.198 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:04.197+0000 7ffab7cb5640 1 --2- 172.21.15.123:0/1108389717 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7ffab015dc40 0x7ffab0155220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:52:04.198 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:04.197+0000 7ffab7cb5640 1 --2- 172.21.15.123:0/1108389717 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ffab015a590 0x7ffab00cdbc0 unknown :-1 s=CLOSED pgs=1924 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:52:04.198 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:04.197+0000 7ffab7cb5640 1 --2- 172.21.15.123:0/1108389717 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ffab015a1b0 0x7ffab00cd680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:52:04.198 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:04.197+0000 7ffab7cb5640 1 -- 172.21.15.123:0/1108389717 >> 172.21.15.123:0/1108389717 conn(0x7ffab00b1630 msgr2=0x7ffab00b1a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:52:04.198 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:04.197+0000 7ffab7cb5640 1 -- 172.21.15.123:0/1108389717 shutdown_connections 2024-09-16T15:52:04.198 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:04.197+0000 7ffab7cb5640 1 -- 172.21.15.123:0/1108389717 wait complete. 2024-09-16T15:52:04.201 INFO:tasks.ceph.mgr.y.smithi123.stderr:2024-09-16T15:52:04.201+0000 7fa9b5ef9640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-16T15:52:04.202 INFO:tasks.ceph.mgr.y.smithi123.stderr:2024-09-16T15:52:04.201+0000 7fa9b5ef9640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-16T15:52:04.202 INFO:tasks.ceph.mgr.y.smithi123.stderr:2024-09-16T15:52:04.201+0000 7fa9b5ef9640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-16T15:52:04.202 INFO:tasks.ceph.mgr.y.smithi123.stderr:2024-09-16T15:52:04.201+0000 7fa9b5ef9640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-16T15:52:04.202 INFO:tasks.ceph.mgr.y.smithi123.stderr:2024-09-16T15:52:04.201+0000 7fa9b5ef9640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-16T15:52:04.216 DEBUG:teuthology.orchestra.run.smithi123:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'stat -c %h /home/ubuntu/cephtest/mnt.0/./volumes/_deleting') 2024-09-16T15:52:04.219 INFO:tasks.ceph.mgr.y.smithi123.stderr:2024-09-16T15:52:04.217+0000 7fa9b4ef7640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-16T15:52:04.219 INFO:tasks.ceph.mgr.y.smithi123.stderr:2024-09-16T15:52:04.217+0000 7fa9b4ef7640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-16T15:52:04.219 INFO:tasks.ceph.mgr.y.smithi123.stderr:2024-09-16T15:52:04.217+0000 7fa9b4ef7640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-16T15:52:04.219 INFO:tasks.ceph.mgr.y.smithi123.stderr:2024-09-16T15:52:04.217+0000 7fa9b4ef7640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-16T15:52:04.219 INFO:tasks.ceph.mgr.y.smithi123.stderr:2024-09-16T15:52:04.217+0000 7fa9b4ef7640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-16T15:52:04.255 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:04 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-09-16T15:52:04.258 INFO:teuthology.orchestra.run.smithi123.stdout:2 2024-09-16T15:52:04.258 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:04 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-16T15:52:04.264 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolumegroup rm cephfs subvol_grp2240 2024-09-16T15:52:04.429 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:04.429+0000 7f8455799640 1 -- 172.21.15.123:0/900467639 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f84500d24c0 msgr2=0x7f84500d0940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:52:04.429 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:04.429+0000 7f8455799640 1 --2- 172.21.15.123:0/900467639 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f84500d24c0 0x7f84500d0940 secure :-1 s=READY pgs=1929 cs=0 l=1 rev1=1 crypto rx=0x7f84500b6a90 tx=0x7f844401cd80 comp rx=0 tx=0).stop 2024-09-16T15:52:04.430 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:04.429+0000 7f8455799640 1 -- 172.21.15.123:0/900467639 shutdown_connections 2024-09-16T15:52:04.430 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:04.429+0000 7f8455799640 1 --2- 172.21.15.123:0/900467639 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f84500d0e80 0x7f84500ce9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:52:04.430 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:04.429+0000 7f8455799640 1 --2- 172.21.15.123:0/900467639 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f84500d24c0 0x7f84500d0940 unknown :-1 s=CLOSED pgs=1929 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:52:04.430 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:04.429+0000 7f8455799640 1 --2- 172.21.15.123:0/900467639 >> v2:172.21.15.184:3300/0 conn(0x7f845015e0a0 0x7f84500d4790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:52:04.430 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:04.429+0000 7f8455799640 1 -- 172.21.15.123:0/900467639 >> 172.21.15.123:0/900467639 conn(0x7f845009d1a0 msgr2=0x7f84500d3f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:52:04.430 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:04.429+0000 7f8455799640 1 -- 172.21.15.123:0/900467639 shutdown_connections 2024-09-16T15:52:04.430 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:04.429+0000 7f8455799640 1 -- 172.21.15.123:0/900467639 wait complete. 2024-09-16T15:52:04.432 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:04.429+0000 7f8455799640 1 Processor -- start 2024-09-16T15:52:04.432 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:04.429+0000 7f8455799640 1 -- start start 2024-09-16T15:52:04.432 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:04.433+0000 7f8455799640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f84500d0e80 0x7f845014e460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:52:04.432 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:04.433+0000 7f8455799640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f84500d24c0 0x7f84500cfae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:52:04.432 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:04.433+0000 7f8455799640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f845015e0a0 0x7f84500d0020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:52:04.433 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:04.433+0000 7f8455799640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f84500b4c60 con 0x7f84500d24c0 2024-09-16T15:52:04.433 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:04.433+0000 7f8455799640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f84500b4ae0 con 0x7f845015e0a0 2024-09-16T15:52:04.433 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:04.433+0000 7f8455799640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f84500b4de0 con 0x7f84500d0e80 2024-09-16T15:52:04.433 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:04.433+0000 7f844e7fc640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f84500d24c0 0x7f84500cfae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:52:04.433 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:04.433+0000 7f844e7fc640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f84500d24c0 0x7f84500cfae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:55370/0 (socket says 172.21.15.123:55370) 2024-09-16T15:52:04.433 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:04.433+0000 7f844e7fc640 1 -- 172.21.15.123:0/485288156 learned_addr learned my addr 172.21.15.123:0/485288156 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:52:04.433 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:04.433+0000 7f844f7fe640 1 --2- 172.21.15.123:0/485288156 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f845015e0a0 0x7f84500d0020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:52:04.433 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:04.433+0000 7f844effd640 1 --2- 172.21.15.123:0/485288156 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f84500d0e80 0x7f845014e460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:52:04.433 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:04.433+0000 7f844e7fc640 1 -- 172.21.15.123:0/485288156 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f84500d0e80 msgr2=0x7f845014e460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:52:04.433 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:04.433+0000 7f844e7fc640 1 --2- 172.21.15.123:0/485288156 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f84500d0e80 0x7f845014e460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:52:04.433 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:04.433+0000 7f844e7fc640 1 -- 172.21.15.123:0/485288156 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f845015e0a0 msgr2=0x7f84500d0020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:52:04.433 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:04.433+0000 7f844e7fc640 1 --2- 172.21.15.123:0/485288156 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f845015e0a0 0x7f84500d0020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:52:04.434 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:04.433+0000 7f844e7fc640 1 -- 172.21.15.123:0/485288156 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f84500b95d0 con 0x7f84500d24c0 2024-09-16T15:52:04.434 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:04.433+0000 7f844f7fe640 1 --2- 172.21.15.123:0/485288156 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f845015e0a0 0x7f84500d0020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T15:52:04.434 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:04.433+0000 7f844e7fc640 1 --2- 172.21.15.123:0/485288156 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f84500d24c0 0x7f84500cfae0 secure :-1 s=READY pgs=1930 cs=0 l=1 rev1=1 crypto rx=0x7f844401b430 tx=0x7f84440a5a90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:52:04.434 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:04.433+0000 7f84377fe640 1 -- 172.21.15.123:0/485288156 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84440bb070 con 0x7f84500d24c0 2024-09-16T15:52:04.434 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:04.433+0000 7f84377fe640 1 -- 172.21.15.123:0/485288156 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f8444004d40 con 0x7f84500d24c0 2024-09-16T15:52:04.434 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:04.433+0000 7f84377fe640 1 -- 172.21.15.123:0/485288156 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84440b2040 con 0x7f84500d24c0 2024-09-16T15:52:04.434 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:04.433+0000 7f8455799640 1 -- 172.21.15.123:0/485288156 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f84500b98b0 con 0x7f84500d24c0 2024-09-16T15:52:04.434 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:04.433+0000 7f8455799640 1 -- 172.21.15.123:0/485288156 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f845015e600 con 0x7f84500d24c0 2024-09-16T15:52:04.435 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:04.433+0000 7f8455799640 1 -- 172.21.15.123:0/485288156 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f84500b5300 con 0x7f84500d24c0 2024-09-16T15:52:04.435 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:04.433+0000 7f84377fe640 1 -- 172.21.15.123:0/485288156 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 26) ==== 98049+0+0 (secure 0 0 0) 0x7f8444004320 con 0x7f84500d24c0 2024-09-16T15:52:04.435 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:04.433+0000 7f84377fe640 1 --2- 172.21.15.123:0/485288156 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f843c073970 0x7f843c075e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:52:04.435 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:04.433+0000 7f84377fe640 1 -- 172.21.15.123:0/485288156 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(169..169 src has 1..169) ==== 5737+0+0 (secure 0 0 0) 0x7f844412c7e0 con 0x7f84500d24c0 2024-09-16T15:52:04.435 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:04.433+0000 7f844effd640 1 --2- 172.21.15.123:0/485288156 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f843c073970 0x7f843c075e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:52:04.435 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:04.437+0000 7f844effd640 1 --2- 172.21.15.123:0/485288156 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f843c073970 0x7f843c075e30 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f844000b9a0 tx=0x7f8440005c30 comp rx=0 tx=0).ready entity=mgr.4150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:52:04.438 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:04.437+0000 7f84377fe640 1 -- 172.21.15.123:0/485288156 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f8444016970 con 0x7f84500d24c0 2024-09-16T15:52:04.626 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:04.625+0000 7f8455799640 1 -- 172.21.15.123:0/485288156 --> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] -- mgr_command(tid 0: {"prefix": "fs subvolumegroup rm", "vol_name": "cephfs", "group_name": "subvol_grp2240", "target": ["mon-mgr", ""]}) -- 0x7f845015ce40 con 0x7f843c073970 2024-09-16T15:52:04.634 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:04.633+0000 7f84377fe640 1 -- 172.21.15.123:0/485288156 <== mgr.4150 v2:172.21.15.123:6836/2464759676 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f845015ce40 con 0x7f843c073970 2024-09-16T15:52:04.637 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:04.637+0000 7f8455799640 1 -- 172.21.15.123:0/485288156 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f843c073970 msgr2=0x7f843c075e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:52:04.637 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:04.637+0000 7f8455799640 1 --2- 172.21.15.123:0/485288156 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f843c073970 0x7f843c075e30 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f844000b9a0 tx=0x7f8440005c30 comp rx=0 tx=0).stop 2024-09-16T15:52:04.637 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:04.637+0000 7f8455799640 1 -- 172.21.15.123:0/485288156 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f84500d24c0 msgr2=0x7f84500cfae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:52:04.637 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:04.637+0000 7f8455799640 1 --2- 172.21.15.123:0/485288156 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f84500d24c0 0x7f84500cfae0 secure :-1 s=READY pgs=1930 cs=0 l=1 rev1=1 crypto rx=0x7f844401b430 tx=0x7f84440a5a90 comp rx=0 tx=0).stop 2024-09-16T15:52:04.637 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:04.637+0000 7f8455799640 1 -- 172.21.15.123:0/485288156 shutdown_connections 2024-09-16T15:52:04.637 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:04.637+0000 7f8455799640 1 --2- 172.21.15.123:0/485288156 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f843c073970 0x7f843c075e30 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:52:04.637 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:04.637+0000 7f8455799640 1 --2- 172.21.15.123:0/485288156 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f845015e0a0 0x7f84500d0020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:52:04.638 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:04.637+0000 7f8455799640 1 --2- 172.21.15.123:0/485288156 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f84500d24c0 0x7f84500cfae0 unknown :-1 s=CLOSED pgs=1930 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:52:04.638 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:04.637+0000 7f8455799640 1 --2- 172.21.15.123:0/485288156 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f84500d0e80 0x7f845014e460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:52:04.638 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:04.637+0000 7f8455799640 1 -- 172.21.15.123:0/485288156 >> 172.21.15.123:0/485288156 conn(0x7f845009d1a0 msgr2=0x7f845014ec20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:52:04.638 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:04.637+0000 7f8455799640 1 -- 172.21.15.123:0/485288156 shutdown_connections 2024-09-16T15:52:04.638 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:04.637+0000 7f8455799640 1 -- 172.21.15.123:0/485288156 wait complete. 2024-09-16T15:52:04.651 INFO:teuthology.nuke:Clearing teuthology firewall rules... 2024-09-16T15:52:04.652 DEBUG:teuthology.orchestra.run.smithi123:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-09-16T15:52:04.728 DEBUG:teuthology.orchestra.run.smithi184:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-09-16T15:52:04.805 INFO:teuthology.nuke:Cleared teuthology firewall rules. 2024-09-16T15:52:04.805 INFO:tasks.cephfs.mount:Terminating background process 2024-09-16T15:52:04.805 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2024-09-16T15:52:04.806 DEBUG:teuthology.orchestra.run.smithi123:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-16T15:52:04.811 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2024-09-16T15:52:04.812 DEBUG:teuthology.orchestra.run.smithi123:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-16T15:52:04.859 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-09-16T15:52:05.034 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:05.033+0000 7f70a1f73640 1 -- 172.21.15.123:0/2979929166 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f709c06e860 msgr2=0x7f709c070760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:52:05.034 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:05.033+0000 7f70a1f73640 1 --2- 172.21.15.123:0/2979929166 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f709c06e860 0x7f709c070760 secure :-1 s=READY pgs=1932 cs=0 l=1 rev1=1 crypto rx=0x7f709c072cf0 tx=0x7f709401cb80 comp rx=0 tx=0).stop 2024-09-16T15:52:05.035 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:05.033+0000 7f70a1f73640 1 -- 172.21.15.123:0/2979929166 shutdown_connections 2024-09-16T15:52:05.035 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:05.033+0000 7f70a1f73640 1 --2- 172.21.15.123:0/2979929166 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f709c06e860 0x7f709c070760 unknown :-1 s=CLOSED pgs=1932 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:52:05.035 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:05.033+0000 7f70a1f73640 1 --2- 172.21.15.123:0/2979929166 >> v2:172.21.15.184:3300/0 conn(0x7f709c0d05e0 0x7f709c074050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:52:05.035 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:05.033+0000 7f70a1f73640 1 -- 172.21.15.123:0/2979929166 >> 172.21.15.123:0/2979929166 conn(0x7f709c0b2c20 msgr2=0x7f709c0b3030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:52:05.035 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:05.033+0000 7f70a1f73640 1 -- 172.21.15.123:0/2979929166 shutdown_connections 2024-09-16T15:52:05.035 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:05.033+0000 7f70a1f73640 1 -- 172.21.15.123:0/2979929166 wait complete. 2024-09-16T15:52:05.035 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:05.037+0000 7f70a1f73640 1 Processor -- start 2024-09-16T15:52:05.035 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:05.037+0000 7f70a1f73640 1 -- start start 2024-09-16T15:52:05.037 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:05.037+0000 7f70a1f73640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f709c06e860 0x7f709c145960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:52:05.037 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:05.037+0000 7f70a1f73640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f709c070320 0x7f709c1535e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:52:05.037 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:05.037+0000 7f70a1f73640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f709c0d05e0 0x7f709c146020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:52:05.037 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:05.037+0000 7f70a1f73640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f709c167c50 con 0x7f709c070320 2024-09-16T15:52:05.037 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:05.037+0000 7f70a1f73640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f709c167ad0 con 0x7f709c0d05e0 2024-09-16T15:52:05.037 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:05.037+0000 7f70a1f73640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f709c167dd0 con 0x7f709c06e860 2024-09-16T15:52:05.038 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:05.037+0000 7f7092ffd640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f709c070320 0x7f709c1535e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:52:05.038 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:05.037+0000 7f7092ffd640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f709c070320 0x7f709c1535e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:55392/0 (socket says 172.21.15.123:55392) 2024-09-16T15:52:05.038 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:05.037+0000 7f7092ffd640 1 -- 172.21.15.123:0/702207163 learned_addr learned my addr 172.21.15.123:0/702207163 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:52:05.038 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:05.037+0000 7f7092ffd640 1 -- 172.21.15.123:0/702207163 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f709c06e860 msgr2=0x7f709c145960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:52:05.038 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:05.037+0000 7f7092ffd640 1 --2- 172.21.15.123:0/702207163 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f709c06e860 0x7f709c145960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:52:05.038 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:05.037+0000 7f7092ffd640 1 -- 172.21.15.123:0/702207163 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f709c0d05e0 msgr2=0x7f709c146020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:52:05.038 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:05.037+0000 7f7092ffd640 1 --2- 172.21.15.123:0/702207163 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f709c0d05e0 0x7f709c146020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:52:05.038 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:05.037+0000 7f7092ffd640 1 -- 172.21.15.123:0/702207163 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f709c14a190 con 0x7f709c070320 2024-09-16T15:52:05.038 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:05.037+0000 7f7092ffd640 1 --2- 172.21.15.123:0/702207163 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f709c070320 0x7f709c1535e0 secure :-1 s=READY pgs=1933 cs=0 l=1 rev1=1 crypto rx=0x7f708400c490 tx=0x7f708400c950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:52:05.038 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:05.037+0000 7f7098ff9640 1 -- 172.21.15.123:0/702207163 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7084016020 con 0x7f709c070320 2024-09-16T15:52:05.038 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:05.037+0000 7f7098ff9640 1 -- 172.21.15.123:0/702207163 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f7084012940 con 0x7f709c070320 2024-09-16T15:52:05.038 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:05.037+0000 7f70a1f73640 1 -- 172.21.15.123:0/702207163 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f709c152b00 con 0x7f709c070320 2024-09-16T15:52:05.038 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:05.037+0000 7f7098ff9640 1 -- 172.21.15.123:0/702207163 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7084012d90 con 0x7f709c070320 2024-09-16T15:52:05.039 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:05.037+0000 7f70a1f73640 1 -- 172.21.15.123:0/702207163 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f709c1530c0 con 0x7f709c070320 2024-09-16T15:52:05.039 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:05.037+0000 7f70a1f73640 1 -- 172.21.15.123:0/702207163 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f709c070760 con 0x7f709c070320 2024-09-16T15:52:05.040 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:05.041+0000 7f7098ff9640 1 -- 172.21.15.123:0/702207163 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 26) ==== 98049+0+0 (secure 0 0 0) 0x7f7084019020 con 0x7f709c070320 2024-09-16T15:52:05.040 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:05.041+0000 7f7098ff9640 1 --2- 172.21.15.123:0/702207163 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f707c0739a0 0x7f707c075e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:52:05.040 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:05.041+0000 7f7098ff9640 1 -- 172.21.15.123:0/702207163 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(169..169 src has 1..169) ==== 5737+0+0 (secure 0 0 0) 0x7f70840926a0 con 0x7f709c070320 2024-09-16T15:52:05.040 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:05.041+0000 7f709b7fe640 1 --2- 172.21.15.123:0/702207163 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f707c0739a0 0x7f707c075e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:52:05.040 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:05.041+0000 7f709b7fe640 1 --2- 172.21.15.123:0/702207163 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f707c0739a0 0x7f707c075e60 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f709c15a820 tx=0x7f708c005b50 comp rx=0 tx=0).ready entity=mgr.4150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:52:05.043 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:05.045+0000 7f7098ff9640 1 -- 172.21.15.123:0/702207163 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f7084020020 con 0x7f709c070320 2024-09-16T15:52:05.229 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:05.229+0000 7f70a1f73640 1 -- 172.21.15.123:0/702207163 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f709c153290 con 0x7f709c070320 2024-09-16T15:52:05.230 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:05.229+0000 7f7098ff9640 1 -- 172.21.15.123:0/702207163 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v169) ==== 80+0+0 (secure 0 0 0) 0x7f708405f460 con 0x7f709c070320 2024-09-16T15:52:05.230 INFO:teuthology.orchestra.run.smithi123.stderr:listed 0 entries 2024-09-16T15:52:05.232 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:05.233+0000 7f70a1f73640 1 -- 172.21.15.123:0/702207163 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f707c0739a0 msgr2=0x7f707c075e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:52:05.232 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:05.233+0000 7f70a1f73640 1 --2- 172.21.15.123:0/702207163 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f707c0739a0 0x7f707c075e60 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f709c15a820 tx=0x7f708c005b50 comp rx=0 tx=0).stop 2024-09-16T15:52:05.232 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:05.233+0000 7f70a1f73640 1 -- 172.21.15.123:0/702207163 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f709c070320 msgr2=0x7f709c1535e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:52:05.232 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:05.233+0000 7f70a1f73640 1 --2- 172.21.15.123:0/702207163 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f709c070320 0x7f709c1535e0 secure :-1 s=READY pgs=1933 cs=0 l=1 rev1=1 crypto rx=0x7f708400c490 tx=0x7f708400c950 comp rx=0 tx=0).stop 2024-09-16T15:52:05.233 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:05.233+0000 7f70a1f73640 1 -- 172.21.15.123:0/702207163 shutdown_connections 2024-09-16T15:52:05.233 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:05.233+0000 7f70a1f73640 1 --2- 172.21.15.123:0/702207163 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f707c0739a0 0x7f707c075e60 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:52:05.233 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:05.233+0000 7f70a1f73640 1 --2- 172.21.15.123:0/702207163 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f709c0d05e0 0x7f709c146020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:52:05.233 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:05.233+0000 7f70a1f73640 1 --2- 172.21.15.123:0/702207163 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f709c070320 0x7f709c1535e0 unknown :-1 s=CLOSED pgs=1933 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:52:05.233 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:05.233+0000 7f70a1f73640 1 --2- 172.21.15.123:0/702207163 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f709c06e860 0x7f709c145960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:52:05.233 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:05.233+0000 7f70a1f73640 1 -- 172.21.15.123:0/702207163 >> 172.21.15.123:0/702207163 conn(0x7f709c0b2c20 msgr2=0x7f709c06e150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:52:05.233 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:05.233+0000 7f70a1f73640 1 -- 172.21.15.123:0/702207163 shutdown_connections 2024-09-16T15:52:05.233 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:05.233+0000 7f70a1f73640 1 -- 172.21.15.123:0/702207163 wait complete. 2024-09-16T15:52:05.249 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi123.front.sepia.ceph.com... 2024-09-16T15:52:05.249 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:52:05.249 DEBUG:teuthology.orchestra.run.smithi123:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2024-09-16T15:52:05.260 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi123.stderr:ceph-fuse[54722]: fuse finished with error 0 and tester_r 0 2024-09-16T15:52:05.261 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi123.front.sepia.ceph.com 2024-09-16T15:52:05.261 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:52:05.261 DEBUG:teuthology.orchestra.run.smithi123:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-09-16T15:52:05.385 INFO:tasks.cephfs.mount:Terminating background process 2024-09-16T15:52:05.385 DEBUG:teuthology.orchestra.run.smithi184:> set -ex 2024-09-16T15:52:05.385 DEBUG:teuthology.orchestra.run.smithi184:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-16T15:52:05.391 DEBUG:teuthology.orchestra.run.smithi184:> set -ex 2024-09-16T15:52:05.392 DEBUG:teuthology.orchestra.run.smithi184:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-16T15:52:05.440 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-09-16T15:52:05.565 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:05.565+0000 7fca97720640 1 -- 172.21.15.123:0/2469082822 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fca9015ec40 msgr2=0x7fca9006f420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:52:05.565 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:05.565+0000 7fca97720640 1 --2- 172.21.15.123:0/2469082822 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fca9015ec40 0x7fca9006f420 secure :-1 s=READY pgs=1934 cs=0 l=1 rev1=1 crypto rx=0x7fca80009900 tx=0x7fca8001c7c0 comp rx=0 tx=0).stop 2024-09-16T15:52:05.565 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:05.565+0000 7fca97720640 1 -- 172.21.15.123:0/2469082822 shutdown_connections 2024-09-16T15:52:05.565 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:05.565+0000 7fca97720640 1 --2- 172.21.15.123:0/2469082822 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fca9015ec40 0x7fca9006f420 unknown :-1 s=CLOSED pgs=1934 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:52:05.565 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:05.565+0000 7fca97720640 1 --2- 172.21.15.123:0/2469082822 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fca900cf5a0 0x7fca9006eee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:52:05.565 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:05.565+0000 7fca97720640 1 -- 172.21.15.123:0/2469082822 >> 172.21.15.123:0/2469082822 conn(0x7fca900cfb10 msgr2=0x7fca900cff20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:52:05.565 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:05.565+0000 7fca97720640 1 -- 172.21.15.123:0/2469082822 shutdown_connections 2024-09-16T15:52:05.566 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:05.565+0000 7fca97720640 1 -- 172.21.15.123:0/2469082822 wait complete. 2024-09-16T15:52:05.566 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:05.565+0000 7fca97720640 1 Processor -- start 2024-09-16T15:52:05.566 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:05.565+0000 7fca97720640 1 -- start start 2024-09-16T15:52:05.568 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:05.569+0000 7fca97720640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fca900cf5a0 0x7fca90153260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:52:05.568 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:05.569+0000 7fca97720640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fca90158120 0x7fca90153ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:52:05.568 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:05.569+0000 7fca97720640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fca9015ec40 0x7fca900af610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:52:05.568 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:05.569+0000 7fca97720640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fca9015b2b0 con 0x7fca9015ec40 2024-09-16T15:52:05.568 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:05.569+0000 7fca97720640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fca9015b130 con 0x7fca90158120 2024-09-16T15:52:05.568 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:05.569+0000 7fca97720640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fca9015b430 con 0x7fca900cf5a0 2024-09-16T15:52:05.568 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:05.569+0000 7fca95cbd640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fca9015ec40 0x7fca900af610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:52:05.568 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:05.569+0000 7fca95cbd640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fca9015ec40 0x7fca900af610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:55408/0 (socket says 172.21.15.123:55408) 2024-09-16T15:52:05.568 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:05.569+0000 7fca95cbd640 1 -- 172.21.15.123:0/4244965100 learned_addr learned my addr 172.21.15.123:0/4244965100 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:52:05.569 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:05.569+0000 7fca95cbd640 1 -- 172.21.15.123:0/4244965100 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fca900cf5a0 msgr2=0x7fca90153260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:52:05.569 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:05.569+0000 7fca95cbd640 1 --2- 172.21.15.123:0/4244965100 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fca900cf5a0 0x7fca90153260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:52:05.569 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:05.569+0000 7fca95cbd640 1 -- 172.21.15.123:0/4244965100 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fca90158120 msgr2=0x7fca90153ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:52:05.569 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:05.569+0000 7fca95cbd640 1 --2- 172.21.15.123:0/4244965100 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fca90158120 0x7fca90153ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:52:05.569 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:05.569+0000 7fca95cbd640 1 -- 172.21.15.123:0/4244965100 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fca901f91d0 con 0x7fca9015ec40 2024-09-16T15:52:05.569 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:05.569+0000 7fca95cbd640 1 --2- 172.21.15.123:0/4244965100 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fca9015ec40 0x7fca900af610 secure :-1 s=READY pgs=1935 cs=0 l=1 rev1=1 crypto rx=0x7fca80009850 tx=0x7fca800a6550 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:52:05.569 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:05.569+0000 7fca8dffb640 1 -- 172.21.15.123:0/4244965100 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca800ac070 con 0x7fca9015ec40 2024-09-16T15:52:05.569 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:05.569+0000 7fca8dffb640 1 -- 172.21.15.123:0/4244965100 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fca80004320 con 0x7fca9015ec40 2024-09-16T15:52:05.569 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:05.569+0000 7fca8dffb640 1 -- 172.21.15.123:0/4244965100 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca800044f0 con 0x7fca9015ec40 2024-09-16T15:52:05.569 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:05.569+0000 7fca97720640 1 -- 172.21.15.123:0/4244965100 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fca90154860 con 0x7fca9015ec40 2024-09-16T15:52:05.569 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:05.569+0000 7fca97720640 1 -- 172.21.15.123:0/4244965100 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fca900c6f60 con 0x7fca9015ec40 2024-09-16T15:52:05.570 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:05.569+0000 7fca97720640 1 -- 172.21.15.123:0/4244965100 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fca900ca320 con 0x7fca9015ec40 2024-09-16T15:52:05.570 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:05.569+0000 7fca8dffb640 1 -- 172.21.15.123:0/4244965100 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 27) ==== 98151+0+0 (secure 0 0 0) 0x7fca800af070 con 0x7fca9015ec40 2024-09-16T15:52:05.570 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:05.569+0000 7fca8dffb640 1 --2- 172.21.15.123:0/4244965100 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fca7c073a90 0x7fca7c075f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:52:05.571 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:05.569+0000 7fca8dffb640 1 -- 172.21.15.123:0/4244965100 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(169..169 src has 1..169) ==== 5737+0+0 (secure 0 0 0) 0x7fca8012cca0 con 0x7fca9015ec40 2024-09-16T15:52:05.571 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:05.569+0000 7fca954bc640 1 --2- 172.21.15.123:0/4244965100 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fca7c073a90 0x7fca7c075f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:52:05.571 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:05.573+0000 7fca954bc640 1 --2- 172.21.15.123:0/4244965100 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fca7c073a90 0x7fca7c075f50 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7fca84003cf0 tx=0x7fca840079c0 comp rx=0 tx=0).ready entity=mgr.4150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:52:05.574 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:05.573+0000 7fca8dffb640 1 -- 172.21.15.123:0/4244965100 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7fca800164d0 con 0x7fca9015ec40 2024-09-16T15:52:05.757 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:05.757+0000 7fca97720640 1 -- 172.21.15.123:0/4244965100 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7fca901f94a0 con 0x7fca9015ec40 2024-09-16T15:52:05.757 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:05.757+0000 7fca8dffb640 1 -- 172.21.15.123:0/4244965100 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v169) ==== 80+0+0 (secure 0 0 0) 0x7fca800f96a0 con 0x7fca9015ec40 2024-09-16T15:52:05.757 INFO:teuthology.orchestra.run.smithi123.stderr:listed 0 entries 2024-09-16T15:52:05.760 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:05.761+0000 7fca97720640 1 -- 172.21.15.123:0/4244965100 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fca7c073a90 msgr2=0x7fca7c075f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:52:05.760 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:05.761+0000 7fca97720640 1 --2- 172.21.15.123:0/4244965100 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fca7c073a90 0x7fca7c075f50 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7fca84003cf0 tx=0x7fca840079c0 comp rx=0 tx=0).stop 2024-09-16T15:52:05.760 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:05.761+0000 7fca97720640 1 -- 172.21.15.123:0/4244965100 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fca9015ec40 msgr2=0x7fca900af610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:52:05.760 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:05.761+0000 7fca97720640 1 --2- 172.21.15.123:0/4244965100 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fca9015ec40 0x7fca900af610 secure :-1 s=READY pgs=1935 cs=0 l=1 rev1=1 crypto rx=0x7fca80009850 tx=0x7fca800a6550 comp rx=0 tx=0).stop 2024-09-16T15:52:05.760 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:05.761+0000 7fca97720640 1 -- 172.21.15.123:0/4244965100 shutdown_connections 2024-09-16T15:52:05.760 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:05.761+0000 7fca97720640 1 --2- 172.21.15.123:0/4244965100 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fca7c073a90 0x7fca7c075f50 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:52:05.760 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:05.761+0000 7fca97720640 1 --2- 172.21.15.123:0/4244965100 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fca9015ec40 0x7fca900af610 unknown :-1 s=CLOSED pgs=1935 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:52:05.760 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:05.761+0000 7fca97720640 1 --2- 172.21.15.123:0/4244965100 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fca90158120 0x7fca90153ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:52:05.761 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:05.761+0000 7fca97720640 1 --2- 172.21.15.123:0/4244965100 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fca900cf5a0 0x7fca90153260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:52:05.761 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:05.761+0000 7fca97720640 1 -- 172.21.15.123:0/4244965100 >> 172.21.15.123:0/4244965100 conn(0x7fca900cfb10 msgr2=0x7fca9015bc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:52:05.761 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:05.761+0000 7fca97720640 1 -- 172.21.15.123:0/4244965100 shutdown_connections 2024-09-16T15:52:05.761 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:05.761+0000 7fca97720640 1 -- 172.21.15.123:0/4244965100 wait complete. 2024-09-16T15:52:05.776 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi184.front.sepia.ceph.com... 2024-09-16T15:52:05.777 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:52:05.777 DEBUG:teuthology.orchestra.run.smithi184:> sudo fusermount -u /home/ubuntu/cephtest/mnt.1 2024-09-16T15:52:05.790 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi184.stderr:ceph-fuse[36573]: fuse finished with error 0 and tester_r 0 2024-09-16T15:52:05.792 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi184.front.sepia.ceph.com 2024-09-16T15:52:05.792 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:52:05.792 DEBUG:teuthology.orchestra.run.smithi184:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2024-09-16T15:52:05.872 INFO:tasks.cephfs.mount:Terminating background process 2024-09-16T15:52:05.872 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2024-09-16T15:52:05.873 DEBUG:teuthology.orchestra.run.smithi123:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-16T15:52:05.879 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16336468k,nr_inodes=4084117,mode=755,inode64 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3274684k,mode=755,inode64 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev,inode64 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=17082 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3274680k,nr_inodes=818670,mode=700,uid=1000,gid=1271,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3274684k,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-09-16T15:52:05.879 INFO:tasks.cephfs.mount:Terminating background process 2024-09-16T15:52:05.879 DEBUG:teuthology.orchestra.run.smithi184:> set -ex 2024-09-16T15:52:05.879 DEBUG:teuthology.orchestra.run.smithi184:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-16T15:52:05.920 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16336488k,nr_inodes=4084122,mode=755,inode64 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3274688k,mode=755,inode64 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev,inode64 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=17834 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3274684k,nr_inodes=818671,mode=700,uid=1000,gid=1271,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3274688k,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-09-16T15:52:05.920 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-16T15:52:06.055 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:06.057+0000 7f795fec6640 1 -- 172.21.15.123:0/2907313273 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f79580b2b00 msgr2=0x7f79580d24b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:52:06.055 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:06.057+0000 7f795fec6640 1 --2- 172.21.15.123:0/2907313273 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f79580b2b00 0x7f79580d24b0 secure :-1 s=READY pgs=1937 cs=0 l=1 rev1=1 crypto rx=0x7f795808d220 tx=0x7f794c01c6b0 comp rx=0 tx=0).stop 2024-09-16T15:52:06.055 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:06.057+0000 7f795fec6640 1 -- 172.21.15.123:0/2907313273 shutdown_connections 2024-09-16T15:52:06.055 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:06.057+0000 7f795fec6640 1 --2- 172.21.15.123:0/2907313273 >> v2:172.21.15.184:3300/0 conn(0x7f79580b0a50 0x7f79580b0ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:52:06.055 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:06.057+0000 7f795fec6640 1 --2- 172.21.15.123:0/2907313273 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f79580b2b00 0x7f79580d24b0 unknown :-1 s=CLOSED pgs=1937 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:52:06.055 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:06.057+0000 7f795fec6640 1 -- 172.21.15.123:0/2907313273 >> 172.21.15.123:0/2907313273 conn(0x7f79580d3ea0 msgr2=0x7f79580d42b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:52:06.055 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:06.057+0000 7f795fec6640 1 -- 172.21.15.123:0/2907313273 shutdown_connections 2024-09-16T15:52:06.056 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:06.057+0000 7f795fec6640 1 -- 172.21.15.123:0/2907313273 wait complete. 2024-09-16T15:52:06.056 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:06.057+0000 7f795fec6640 1 Processor -- start 2024-09-16T15:52:06.056 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:06.057+0000 7f795fec6640 1 -- start start 2024-09-16T15:52:06.058 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:06.057+0000 7f795fec6640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f79580b0a50 0x7f79580b8c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:52:06.058 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:06.057+0000 7f795fec6640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f79580b2b00 0x7f79580b6cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:52:06.058 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:06.057+0000 7f795fec6640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f7958161620 0x7f79580c97b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:52:06.058 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:06.057+0000 7f795fec6640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f795816b0f0 con 0x7f79580b2b00 2024-09-16T15:52:06.058 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:06.057+0000 7f795fec6640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f795816af70 con 0x7f79580b0a50 2024-09-16T15:52:06.058 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:06.057+0000 7f795fec6640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f795816b270 con 0x7f7958161620 2024-09-16T15:52:06.058 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:06.057+0000 7f795d461640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f79580b2b00 0x7f79580b6cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:52:06.058 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:06.057+0000 7f795d461640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f79580b2b00 0x7f79580b6cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:55436/0 (socket says 172.21.15.123:55436) 2024-09-16T15:52:06.058 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:06.057+0000 7f795d461640 1 -- 172.21.15.123:0/2839443723 learned_addr learned my addr 172.21.15.123:0/2839443723 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:52:06.058 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:06.057+0000 7f795e463640 1 --2- 172.21.15.123:0/2839443723 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f7958161620 0x7f79580c97b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:52:06.058 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:06.057+0000 7f795dc62640 1 --2- 172.21.15.123:0/2839443723 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f79580b0a50 0x7f79580b8c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:52:06.058 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:06.057+0000 7f795d461640 1 -- 172.21.15.123:0/2839443723 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f7958161620 msgr2=0x7f79580c97b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:52:06.058 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:06.057+0000 7f795d461640 1 --2- 172.21.15.123:0/2839443723 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f7958161620 0x7f79580c97b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:52:06.059 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:06.057+0000 7f795d461640 1 -- 172.21.15.123:0/2839443723 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f79580b0a50 msgr2=0x7f79580b8c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:52:06.059 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:06.057+0000 7f795d461640 1 --2- 172.21.15.123:0/2839443723 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f79580b0a50 0x7f79580b8c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:52:06.059 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:06.057+0000 7f795d461640 1 -- 172.21.15.123:0/2839443723 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f79580ca300 con 0x7f79580b2b00 2024-09-16T15:52:06.059 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:06.057+0000 7f795d461640 1 --2- 172.21.15.123:0/2839443723 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f79580b2b00 0x7f79580b6cd0 secure :-1 s=READY pgs=1938 cs=0 l=1 rev1=1 crypto rx=0x7f794c01ad60 tx=0x7f794c0058e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:52:06.059 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:06.057+0000 7f79567fc640 1 -- 172.21.15.123:0/2839443723 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f794c0041c0 con 0x7f79580b2b00 2024-09-16T15:52:06.059 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:06.057+0000 7f79567fc640 1 -- 172.21.15.123:0/2839443723 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f794c004450 con 0x7f79580b2b00 2024-09-16T15:52:06.059 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:06.057+0000 7f795fec6640 1 -- 172.21.15.123:0/2839443723 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f79580ca5e0 con 0x7f79580b2b00 2024-09-16T15:52:06.059 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:06.057+0000 7f79567fc640 1 -- 172.21.15.123:0/2839443723 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f794c003840 con 0x7f79580b2b00 2024-09-16T15:52:06.059 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:06.057+0000 7f795fec6640 1 -- 172.21.15.123:0/2839443723 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7958152510 con 0x7f79580b2b00 2024-09-16T15:52:06.060 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:06.061+0000 7f795fec6640 1 -- 172.21.15.123:0/2839443723 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7958151ea0 con 0x7f79580b2b00 2024-09-16T15:52:06.060 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:06.061+0000 7f79567fc640 1 -- 172.21.15.123:0/2839443723 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 27) ==== 98151+0+0 (secure 0 0 0) 0x7f794c0c2020 con 0x7f79580b2b00 2024-09-16T15:52:06.060 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:06.061+0000 7f79567fc640 1 --2- 172.21.15.123:0/2839443723 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f7944073a90 0x7f7944075f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:52:06.060 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:06.061+0000 7f795dc62640 1 --2- 172.21.15.123:0/2839443723 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f7944073a90 0x7f7944075f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:52:06.060 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:06.061+0000 7f79567fc640 1 -- 172.21.15.123:0/2839443723 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(169..169 src has 1..169) ==== 5737+0+0 (secure 0 0 0) 0x7f794c12d6c0 con 0x7f79580b2b00 2024-09-16T15:52:06.060 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:06.061+0000 7f795dc62640 1 --2- 172.21.15.123:0/2839443723 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f7944073a90 0x7f7944075f50 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f7950001cc0 tx=0x7f7950009040 comp rx=0 tx=0).ready entity=mgr.4150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:52:06.064 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:06.065+0000 7f79567fc640 1 -- 172.21.15.123:0/2839443723 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f794c0163f0 con 0x7f79580b2b00 2024-09-16T15:52:06.269 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:06.269+0000 7f795fec6640 1 -- 172.21.15.123:0/2839443723 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f79580b0ea0 con 0x7f79580b2b00 2024-09-16T15:52:06.269 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:06.269+0000 7f79567fc640 1 -- 172.21.15.123:0/2839443723 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 116 v116) ==== 95+0+5239 (secure 0 0 0) 0x7f794c0fa330 con 0x7f79580b2b00 2024-09-16T15:52:06.270 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-09-16T15:52:06.270 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":116,"btime":"2024-09-16T15:52:00:476603+0000","default_fscid":13,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6314,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6833/2070840764","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":2070840764},{"type":"v1","addr":"172.21.15.184:6833","nonce":2070840764}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":110},{"gid":6976,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6835/3881288354","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":3881288354},{"type":"v1","addr":"172.21.15.184:6835","nonce":3881288354}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":110},{"gid":9219,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6833/519008782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6832","nonce":519008782},{"type":"v1","addr":"172.21.15.123:6833","nonce":519008782}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":110}],"filesystems":[{"mdsmap":{"epoch":115,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-16T15:51:47.741329+0000","modified":"2024-09-16T15:51:50.794388+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":9234},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9234":{"gid":9234,"name":"a","rank":0,"incarnation":111,"state":"up:active","state_seq":3,"addr":"172.21.15.123:6835/370478759","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6834","nonce":370478759},{"type":"v1","addr":"172.21.15.123:6835","nonce":370478759}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[27],"metadata_pool":26,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9234,"qdb_cluster":[9234]},"id":13}]} 2024-09-16T15:52:06.270 INFO:teuthology.orchestra.run.smithi123.stderr:dumped fsmap epoch 116 2024-09-16T15:52:06.272 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:06.273+0000 7f795fec6640 1 -- 172.21.15.123:0/2839443723 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f7944073a90 msgr2=0x7f7944075f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:52:06.273 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:06.273+0000 7f795fec6640 1 --2- 172.21.15.123:0/2839443723 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f7944073a90 0x7f7944075f50 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f7950001cc0 tx=0x7f7950009040 comp rx=0 tx=0).stop 2024-09-16T15:52:06.273 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:06.273+0000 7f795fec6640 1 -- 172.21.15.123:0/2839443723 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f79580b2b00 msgr2=0x7f79580b6cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:52:06.273 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:06.273+0000 7f795fec6640 1 --2- 172.21.15.123:0/2839443723 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f79580b2b00 0x7f79580b6cd0 secure :-1 s=READY pgs=1938 cs=0 l=1 rev1=1 crypto rx=0x7f794c01ad60 tx=0x7f794c0058e0 comp rx=0 tx=0).stop 2024-09-16T15:52:06.273 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:06.273+0000 7f795fec6640 1 -- 172.21.15.123:0/2839443723 shutdown_connections 2024-09-16T15:52:06.273 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:06.273+0000 7f795fec6640 1 --2- 172.21.15.123:0/2839443723 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f7944073a90 0x7f7944075f50 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:52:06.273 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:06.273+0000 7f795fec6640 1 --2- 172.21.15.123:0/2839443723 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f7958161620 0x7f79580c97b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:52:06.273 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:06.273+0000 7f795fec6640 1 --2- 172.21.15.123:0/2839443723 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f79580b2b00 0x7f79580b6cd0 unknown :-1 s=CLOSED pgs=1938 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:52:06.273 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:06.273+0000 7f795fec6640 1 --2- 172.21.15.123:0/2839443723 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f79580b0a50 0x7f79580b8c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:52:06.273 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:06.273+0000 7f795fec6640 1 -- 172.21.15.123:0/2839443723 >> 172.21.15.123:0/2839443723 conn(0x7f79580d3ea0 msgr2=0x7f79580b5920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:52:06.273 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:06.273+0000 7f795fec6640 1 -- 172.21.15.123:0/2839443723 shutdown_connections 2024-09-16T15:52:06.273 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:06.273+0000 7f795fec6640 1 -- 172.21.15.123:0/2839443723 wait complete. 2024-09-16T15:52:06.288 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-16T15:52:06.411 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:06.413+0000 7fba1fd86640 1 -- 172.21.15.123:0/2016301602 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fba040053d0 msgr2=0x7fba0400b9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:52:06.411 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:06.413+0000 7fba1fd86640 1 --2- 172.21.15.123:0/2016301602 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fba040053d0 0x7fba0400b9d0 secure :-1 s=READY pgs=1940 cs=0 l=1 rev1=1 crypto rx=0x7fba10009e10 tx=0x7fba1001d190 comp rx=0 tx=0).stop 2024-09-16T15:52:06.411 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:06.413+0000 7fba1fd86640 1 -- 172.21.15.123:0/2016301602 shutdown_connections 2024-09-16T15:52:06.411 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:06.413+0000 7fba1fd86640 1 --2- 172.21.15.123:0/2016301602 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fba040053d0 0x7fba0400b9d0 unknown :-1 s=CLOSED pgs=1940 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:52:06.411 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:06.413+0000 7fba1fd86640 1 -- 172.21.15.123:0/2016301602 >> 172.21.15.123:0/2016301602 conn(0x7fba180cfb10 msgr2=0x7fba180cff20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:52:06.411 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:06.413+0000 7fba1fd86640 1 -- 172.21.15.123:0/2016301602 shutdown_connections 2024-09-16T15:52:06.411 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:06.413+0000 7fba1fd86640 1 -- 172.21.15.123:0/2016301602 wait complete. 2024-09-16T15:52:06.412 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:06.413+0000 7fba1fd86640 1 Processor -- start 2024-09-16T15:52:06.412 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:06.413+0000 7fba1fd86640 1 -- start start 2024-09-16T15:52:06.414 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:06.413+0000 7fba1fd86640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fba040053d0 0x7fba18153d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:52:06.414 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:06.413+0000 7fba1fd86640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fba181f5ce0 0x7fba18151900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:52:06.415 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:06.413+0000 7fba1fd86640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fba180c5d90 0x7fba180ce530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:52:06.415 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:06.413+0000 7fba1fd86640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fba180cf7c0 con 0x7fba040053d0 2024-09-16T15:52:06.415 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:06.413+0000 7fba1fd86640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fba180cf640 con 0x7fba180c5d90 2024-09-16T15:52:06.415 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:06.413+0000 7fba1fd86640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fba180cf940 con 0x7fba181f5ce0 2024-09-16T15:52:06.415 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:06.413+0000 7fba1db22640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fba040053d0 0x7fba18153d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:52:06.415 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:06.413+0000 7fba1db22640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fba040053d0 0x7fba18153d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:55442/0 (socket says 172.21.15.123:55442) 2024-09-16T15:52:06.415 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:06.413+0000 7fba1db22640 1 -- 172.21.15.123:0/2909315687 learned_addr learned my addr 172.21.15.123:0/2909315687 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:52:06.415 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:06.413+0000 7fba1d321640 1 --2- 172.21.15.123:0/2909315687 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fba181f5ce0 0x7fba18151900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:52:06.415 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:06.413+0000 7fba1e323640 1 --2- 172.21.15.123:0/2909315687 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fba180c5d90 0x7fba180ce530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:52:06.415 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:06.413+0000 7fba1db22640 1 -- 172.21.15.123:0/2909315687 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fba181f5ce0 msgr2=0x7fba18151900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:52:06.415 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:06.413+0000 7fba1db22640 1 --2- 172.21.15.123:0/2909315687 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fba181f5ce0 0x7fba18151900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:52:06.415 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:06.413+0000 7fba1db22640 1 -- 172.21.15.123:0/2909315687 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fba180c5d90 msgr2=0x7fba180ce530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:52:06.415 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:06.413+0000 7fba1db22640 1 --2- 172.21.15.123:0/2909315687 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fba180c5d90 0x7fba180ce530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:52:06.415 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:06.413+0000 7fba1db22640 1 -- 172.21.15.123:0/2909315687 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fba181f61a0 con 0x7fba040053d0 2024-09-16T15:52:06.416 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:06.413+0000 7fba1e323640 1 --2- 172.21.15.123:0/2909315687 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fba180c5d90 0x7fba180ce530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T15:52:06.416 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:06.413+0000 7fba1db22640 1 --2- 172.21.15.123:0/2909315687 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fba040053d0 0x7fba18153d90 secure :-1 s=READY pgs=1941 cs=0 l=1 rev1=1 crypto rx=0x7fba14001840 tx=0x7fba1400e770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:52:06.416 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:06.413+0000 7fba0e7fc640 1 -- 172.21.15.123:0/2909315687 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba14012850 con 0x7fba040053d0 2024-09-16T15:52:06.416 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:06.413+0000 7fba0e7fc640 1 -- 172.21.15.123:0/2909315687 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fba14012ae0 con 0x7fba040053d0 2024-09-16T15:52:06.416 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:06.413+0000 7fba1fd86640 1 -- 172.21.15.123:0/2909315687 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fba181f6480 con 0x7fba040053d0 2024-09-16T15:52:06.416 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:06.413+0000 7fba0e7fc640 1 -- 172.21.15.123:0/2909315687 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba1401a400 con 0x7fba040053d0 2024-09-16T15:52:06.416 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:06.413+0000 7fba1fd86640 1 -- 172.21.15.123:0/2909315687 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fba181f6a40 con 0x7fba040053d0 2024-09-16T15:52:06.417 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:06.417+0000 7fba1fd86640 1 -- 172.21.15.123:0/2909315687 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fba18074220 con 0x7fba040053d0 2024-09-16T15:52:06.417 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:06.417+0000 7fba0e7fc640 1 -- 172.21.15.123:0/2909315687 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 27) ==== 98151+0+0 (secure 0 0 0) 0x7fba1400d030 con 0x7fba040053d0 2024-09-16T15:52:06.417 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:06.417+0000 7fba0e7fc640 1 --2- 172.21.15.123:0/2909315687 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fb9fc073a90 0x7fb9fc075f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:52:06.417 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:06.417+0000 7fba0e7fc640 1 -- 172.21.15.123:0/2909315687 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(169..169 src has 1..169) ==== 5737+0+0 (secure 0 0 0) 0x7fba14095c40 con 0x7fba040053d0 2024-09-16T15:52:06.417 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:06.417+0000 7fba1d321640 1 --2- 172.21.15.123:0/2909315687 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fb9fc073a90 0x7fb9fc075f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:52:06.417 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:06.417+0000 7fba1d321640 1 --2- 172.21.15.123:0/2909315687 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fb9fc073a90 0x7fb9fc075f50 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7fba18152c30 tx=0x7fba08006b70 comp rx=0 tx=0).ready entity=mgr.4150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:52:06.421 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:06.421+0000 7fba0e7fc640 1 -- 172.21.15.123:0/2909315687 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7fba14016030 con 0x7fba040053d0 2024-09-16T15:52:06.615 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:06.617+0000 7fba1fd86640 1 -- 172.21.15.123:0/2909315687 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fba180d06c0 con 0x7fba040053d0 2024-09-16T15:52:06.616 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:06.617+0000 7fba0e7fc640 1 -- 172.21.15.123:0/2909315687 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 116 v116) ==== 95+0+5239 (secure 0 0 0) 0x7fba140626e0 con 0x7fba040053d0 2024-09-16T15:52:06.617 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-09-16T15:52:06.617 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":116,"btime":"2024-09-16T15:52:00:476603+0000","default_fscid":13,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6314,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6833/2070840764","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":2070840764},{"type":"v1","addr":"172.21.15.184:6833","nonce":2070840764}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":110},{"gid":6976,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6835/3881288354","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":3881288354},{"type":"v1","addr":"172.21.15.184:6835","nonce":3881288354}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":110},{"gid":9219,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6833/519008782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6832","nonce":519008782},{"type":"v1","addr":"172.21.15.123:6833","nonce":519008782}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":110}],"filesystems":[{"mdsmap":{"epoch":115,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-16T15:51:47.741329+0000","modified":"2024-09-16T15:51:50.794388+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":9234},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9234":{"gid":9234,"name":"a","rank":0,"incarnation":111,"state":"up:active","state_seq":3,"addr":"172.21.15.123:6835/370478759","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6834","nonce":370478759},{"type":"v1","addr":"172.21.15.123:6835","nonce":370478759}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[27],"metadata_pool":26,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9234,"qdb_cluster":[9234]},"id":13}]} 2024-09-16T15:52:06.617 INFO:teuthology.orchestra.run.smithi123.stderr:dumped fsmap epoch 116 2024-09-16T15:52:06.619 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:06.621+0000 7fba1fd86640 1 -- 172.21.15.123:0/2909315687 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fb9fc073a90 msgr2=0x7fb9fc075f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:52:06.619 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:06.621+0000 7fba1fd86640 1 --2- 172.21.15.123:0/2909315687 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fb9fc073a90 0x7fb9fc075f50 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7fba18152c30 tx=0x7fba08006b70 comp rx=0 tx=0).stop 2024-09-16T15:52:06.619 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:06.621+0000 7fba1fd86640 1 -- 172.21.15.123:0/2909315687 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fba040053d0 msgr2=0x7fba18153d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:52:06.619 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:06.621+0000 7fba1fd86640 1 --2- 172.21.15.123:0/2909315687 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fba040053d0 0x7fba18153d90 secure :-1 s=READY pgs=1941 cs=0 l=1 rev1=1 crypto rx=0x7fba14001840 tx=0x7fba1400e770 comp rx=0 tx=0).stop 2024-09-16T15:52:06.619 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:06.621+0000 7fba1fd86640 1 -- 172.21.15.123:0/2909315687 shutdown_connections 2024-09-16T15:52:06.619 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:06.621+0000 7fba1fd86640 1 --2- 172.21.15.123:0/2909315687 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fb9fc073a90 0x7fb9fc075f50 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:52:06.619 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:06.621+0000 7fba1fd86640 1 --2- 172.21.15.123:0/2909315687 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fba180c5d90 0x7fba180ce530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:52:06.620 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:06.621+0000 7fba1fd86640 1 --2- 172.21.15.123:0/2909315687 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fba181f5ce0 0x7fba18151900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:52:06.620 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:06.621+0000 7fba1fd86640 1 --2- 172.21.15.123:0/2909315687 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fba040053d0 0x7fba18153d90 unknown :-1 s=CLOSED pgs=1941 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:52:06.620 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:06.621+0000 7fba1fd86640 1 -- 172.21.15.123:0/2909315687 >> 172.21.15.123:0/2909315687 conn(0x7fba180cfb10 msgr2=0x7fba18158400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:52:06.620 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:06.621+0000 7fba1fd86640 1 -- 172.21.15.123:0/2909315687 shutdown_connections 2024-09-16T15:52:06.620 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:06.621+0000 7fba1fd86640 1 -- 172.21.15.123:0/2909315687 wait complete. 2024-09-16T15:52:06.634 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-16T15:52:06.756 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:06.757+0000 7f3ac5d3b640 1 -- 172.21.15.123:0/2248527321 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3ac00cbd20 msgr2=0x7f3ac00ce5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:52:06.756 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:06.757+0000 7f3ac5d3b640 1 --2- 172.21.15.123:0/2248527321 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3ac00cbd20 0x7f3ac00ce5b0 secure :-1 s=READY pgs=1942 cs=0 l=1 rev1=1 crypto rx=0x7f3ac00750e0 tx=0x7f3ab401c890 comp rx=0 tx=0).stop 2024-09-16T15:52:06.756 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:06.757+0000 7f3ac5d3b640 1 -- 172.21.15.123:0/2248527321 shutdown_connections 2024-09-16T15:52:06.756 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:06.757+0000 7f3ac5d3b640 1 --2- 172.21.15.123:0/2248527321 >> v2:172.21.15.184:3300/0 conn(0x7f3ac015cd30 0x7f3ac0152830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:52:06.756 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:06.757+0000 7f3ac5d3b640 1 --2- 172.21.15.123:0/2248527321 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3ac00cbd20 0x7f3ac00ce5b0 unknown :-1 s=CLOSED pgs=1942 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:52:06.757 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:06.757+0000 7f3ac5d3b640 1 --2- 172.21.15.123:0/2248527321 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f3ac00ce1d0 0x7f3ac0068960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:52:06.757 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:06.757+0000 7f3ac5d3b640 1 -- 172.21.15.123:0/2248527321 >> 172.21.15.123:0/2248527321 conn(0x7f3ac0074130 msgr2=0x7f3ac0074540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:52:06.757 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:06.757+0000 7f3ac5d3b640 1 -- 172.21.15.123:0/2248527321 shutdown_connections 2024-09-16T15:52:06.757 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:06.757+0000 7f3ac5d3b640 1 -- 172.21.15.123:0/2248527321 wait complete. 2024-09-16T15:52:06.757 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:06.757+0000 7f3ac5d3b640 1 Processor -- start 2024-09-16T15:52:06.757 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:06.757+0000 7f3ac5d3b640 1 -- start start 2024-09-16T15:52:06.759 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:06.757+0000 7f3ac5d3b640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3ac00cbd20 0x7f3ac00c4f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:52:06.759 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:06.757+0000 7f3ac5d3b640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3ac00ce1d0 0x7f3ac00c5b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:52:06.759 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:06.757+0000 7f3ac5d3b640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f3ac015cd30 0x7f3ac00c6620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:52:06.759 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:06.757+0000 7f3ac5d3b640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f3ac00d01e0 con 0x7f3ac00ce1d0 2024-09-16T15:52:06.759 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:06.757+0000 7f3ac5d3b640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f3ac00afb90 con 0x7f3ac00cbd20 2024-09-16T15:52:06.760 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:06.757+0000 7f3ac5d3b640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f3ac00d0360 con 0x7f3ac015cd30 2024-09-16T15:52:06.760 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:06.757+0000 7f3abeffd640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3ac00ce1d0 0x7f3ac00c5b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:52:06.760 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:06.757+0000 7f3abeffd640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3ac00ce1d0 0x7f3ac00c5b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:55460/0 (socket says 172.21.15.123:55460) 2024-09-16T15:52:06.760 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:06.757+0000 7f3abeffd640 1 -- 172.21.15.123:0/724409065 learned_addr learned my addr 172.21.15.123:0/724409065 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:52:06.760 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:06.757+0000 7f3abf7fe640 1 --2- 172.21.15.123:0/724409065 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3ac00cbd20 0x7f3ac00c4f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:52:06.760 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:06.757+0000 7f3abffff640 1 --2- 172.21.15.123:0/724409065 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f3ac015cd30 0x7f3ac00c6620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:52:06.760 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:06.757+0000 7f3abeffd640 1 -- 172.21.15.123:0/724409065 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f3ac015cd30 msgr2=0x7f3ac00c6620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:52:06.760 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:06.757+0000 7f3abeffd640 1 --2- 172.21.15.123:0/724409065 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f3ac015cd30 0x7f3ac00c6620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:52:06.760 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:06.757+0000 7f3abeffd640 1 -- 172.21.15.123:0/724409065 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3ac00cbd20 msgr2=0x7f3ac00c4f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:52:06.760 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:06.757+0000 7f3abeffd640 1 --2- 172.21.15.123:0/724409065 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3ac00cbd20 0x7f3ac00c4f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:52:06.760 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:06.761+0000 7f3abeffd640 1 -- 172.21.15.123:0/724409065 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3ac01f82f0 con 0x7f3ac00ce1d0 2024-09-16T15:52:06.761 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:06.761+0000 7f3abffff640 1 --2- 172.21.15.123:0/724409065 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f3ac015cd30 0x7f3ac00c6620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T15:52:06.761 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:06.761+0000 7f3abf7fe640 1 --2- 172.21.15.123:0/724409065 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3ac00cbd20 0x7f3ac00c4f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T15:52:06.761 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:06.761+0000 7f3abeffd640 1 --2- 172.21.15.123:0/724409065 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3ac00ce1d0 0x7f3ac00c5b60 secure :-1 s=READY pgs=1943 cs=0 l=1 rev1=1 crypto rx=0x7f3ab40b0040 tx=0x7f3ab40040e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:52:06.761 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:06.761+0000 7f3aa7fff640 1 -- 172.21.15.123:0/724409065 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ab40bc070 con 0x7f3ac00ce1d0 2024-09-16T15:52:06.761 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:06.761+0000 7f3aa7fff640 1 -- 172.21.15.123:0/724409065 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f3ab4005020 con 0x7f3ac00ce1d0 2024-09-16T15:52:06.761 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:06.761+0000 7f3aa7fff640 1 -- 172.21.15.123:0/724409065 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ab40b2040 con 0x7f3ac00ce1d0 2024-09-16T15:52:06.761 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:06.761+0000 7f3ac5d3b640 1 -- 172.21.15.123:0/724409065 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3ac01f85d0 con 0x7f3ac00ce1d0 2024-09-16T15:52:06.761 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:06.761+0000 7f3ac5d3b640 1 -- 172.21.15.123:0/724409065 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3ac01f8e70 con 0x7f3ac00ce1d0 2024-09-16T15:52:06.761 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:06.761+0000 7f3ac5d3b640 1 -- 172.21.15.123:0/724409065 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3ac0068960 con 0x7f3ac00ce1d0 2024-09-16T15:52:06.762 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:06.761+0000 7f3aa7fff640 1 -- 172.21.15.123:0/724409065 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 27) ==== 98151+0+0 (secure 0 0 0) 0x7f3ab40046d0 con 0x7f3ac00ce1d0 2024-09-16T15:52:06.762 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:06.761+0000 7f3aa7fff640 1 --2- 172.21.15.123:0/724409065 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f3aac073a90 0x7f3aac075f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:52:06.762 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:06.761+0000 7f3aa7fff640 1 -- 172.21.15.123:0/724409065 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(169..169 src has 1..169) ==== 5737+0+0 (secure 0 0 0) 0x7f3ab412c940 con 0x7f3ac00ce1d0 2024-09-16T15:52:06.762 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:06.761+0000 7f3abf7fe640 1 --2- 172.21.15.123:0/724409065 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f3aac073a90 0x7f3aac075f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:52:06.762 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:06.761+0000 7f3abf7fe640 1 --2- 172.21.15.123:0/724409065 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f3aac073a90 0x7f3aac075f50 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f3ab000baa0 tx=0x7f3ab0005d30 comp rx=0 tx=0).ready entity=mgr.4150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:52:06.765 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:06.765+0000 7f3aa7fff640 1 -- 172.21.15.123:0/724409065 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f3ab40164b0 con 0x7f3ac00ce1d0 2024-09-16T15:52:06.926 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:06.929+0000 7f3ac5d3b640 1 -- 172.21.15.123:0/724409065 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f3ac00b6250 con 0x7f3ac00ce1d0 2024-09-16T15:52:06.929 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:06.929+0000 7f3aa7fff640 1 -- 172.21.15.123:0/724409065 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v169) ==== 74+0+16847 (secure 0 0 0) 0x7f3ab40f93e0 con 0x7f3ac00ce1d0 2024-09-16T15:52:06.929 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-09-16T15:52:06.929 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":169,"fsid":"56077f89-18c8-486f-ae7e-eecc0bb7822c","created":"2024-09-16T15:42:26.364892+0000","modified":"2024-09-16T15:51:49.376383+0000","last_up_change":"2024-09-16T15:42:38.707234+0000","last_in_change":"2024-09-16T15:42:29.423662+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":27,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-16T15:42:39.287764+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":26,"pool_name":"cephfs_metadata","create_time":"2024-09-16T15:51:45.417755+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"168","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":27,"pool_name":"cephfs_data","create_time":"2024-09-16T15:51:46.701679+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"169","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9e7c1955-a607-4671-9be7-f2ef760ceeaa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":167,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6800","nonce":3882044969},{"type":"v1","addr":"172.21.15.123:6801","nonce":3882044969}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6802","nonce":3882044969},{"type":"v1","addr":"172.21.15.123:6803","nonce":3882044969}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6806","nonce":3882044969},{"type":"v1","addr":"172.21.15.123:6807","nonce":3882044969}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6804","nonce":3882044969},{"type":"v1","addr":"172.21.15.123:6805","nonce":3882044969}]},"public_addr":"172.21.15.123:6801/3882044969","cluster_addr":"172.21.15.123:6803/3882044969","heartbeat_back_addr":"172.21.15.123:6807/3882044969","heartbeat_front_addr":"172.21.15.123:6805/3882044969","state":["exists","up"]},{"osd":1,"uuid":"c540cfde-0977-4742-9b17-fd634668e010","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":167,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6816","nonce":4037001850},{"type":"v1","addr":"172.21.15.123:6817","nonce":4037001850}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6818","nonce":4037001850},{"type":"v1","addr":"172.21.15.123:6819","nonce":4037001850}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6822","nonce":4037001850},{"type":"v1","addr":"172.21.15.123:6823","nonce":4037001850}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6820","nonce":4037001850},{"type":"v1","addr":"172.21.15.123:6821","nonce":4037001850}]},"public_addr":"172.21.15.123:6817/4037001850","cluster_addr":"172.21.15.123:6819/4037001850","heartbeat_back_addr":"172.21.15.123:6823/4037001850","heartbeat_front_addr":"172.21.15.123:6821/4037001850","state":["exists","up"]},{"osd":2,"uuid":"9864a496-f169-486b-95c4-1d59162793a7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":167,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6824","nonce":1332143088},{"type":"v1","addr":"172.21.15.123:6825","nonce":1332143088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6826","nonce":1332143088},{"type":"v1","addr":"172.21.15.123:6827","nonce":1332143088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6830","nonce":1332143088},{"type":"v1","addr":"172.21.15.123:6831","nonce":1332143088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6828","nonce":1332143088},{"type":"v1","addr":"172.21.15.123:6829","nonce":1332143088}]},"public_addr":"172.21.15.123:6825/1332143088","cluster_addr":"172.21.15.123:6827/1332143088","heartbeat_back_addr":"172.21.15.123:6831/1332143088","heartbeat_front_addr":"172.21.15.123:6829/1332143088","state":["exists","up"]},{"osd":3,"uuid":"c6d1f817-f074-4325-95e1-578eba13edda","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":167,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6808","nonce":988586998},{"type":"v1","addr":"172.21.15.123:6809","nonce":988586998}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6810","nonce":988586998},{"type":"v1","addr":"172.21.15.123:6811","nonce":988586998}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6814","nonce":988586998},{"type":"v1","addr":"172.21.15.123:6815","nonce":988586998}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6812","nonce":988586998},{"type":"v1","addr":"172.21.15.123:6813","nonce":988586998}]},"public_addr":"172.21.15.123:6809/988586998","cluster_addr":"172.21.15.123:6811/988586998","heartbeat_back_addr":"172.21.15.123:6815/988586998","heartbeat_front_addr":"172.21.15.123:6813/988586998","state":["exists","up"]},{"osd":4,"uuid":"200a8df4-50ef-40a5-b48a-58981269c086","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":167,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6816","nonce":3387813890},{"type":"v1","addr":"172.21.15.184:6817","nonce":3387813890}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6818","nonce":3387813890},{"type":"v1","addr":"172.21.15.184:6819","nonce":3387813890}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6822","nonce":3387813890},{"type":"v1","addr":"172.21.15.184:6823","nonce":3387813890}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6820","nonce":3387813890},{"type":"v1","addr":"172.21.15.184:6821","nonce":3387813890}]},"public_addr":"172.21.15.184:6817/3387813890","cluster_addr":"172.21.15.184:6819/3387813890","heartbeat_back_addr":"172.21.15.184:6823/3387813890","heartbeat_front_addr":"172.21.15.184:6821/3387813890","state":["exists","up"]},{"osd":5,"uuid":"32d5a900-da3d-412c-bc19-13861a2368d8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":167,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6808","nonce":2680916017},{"type":"v1","addr":"172.21.15.184:6809","nonce":2680916017}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6810","nonce":2680916017},{"type":"v1","addr":"172.21.15.184:6811","nonce":2680916017}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6814","nonce":2680916017},{"type":"v1","addr":"172.21.15.184:6815","nonce":2680916017}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6812","nonce":2680916017},{"type":"v1","addr":"172.21.15.184:6813","nonce":2680916017}]},"public_addr":"172.21.15.184:6809/2680916017","cluster_addr":"172.21.15.184:6811/2680916017","heartbeat_back_addr":"172.21.15.184:6815/2680916017","heartbeat_front_addr":"172.21.15.184:6813/2680916017","state":["exists","up"]},{"osd":6,"uuid":"cde6f39e-aa1b-4260-ac19-774ee6aaca72","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":167,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6824","nonce":2183120457},{"type":"v1","addr":"172.21.15.184:6825","nonce":2183120457}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6826","nonce":2183120457},{"type":"v1","addr":"172.21.15.184:6827","nonce":2183120457}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6830","nonce":2183120457},{"type":"v1","addr":"172.21.15.184:6831","nonce":2183120457}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6828","nonce":2183120457},{"type":"v1","addr":"172.21.15.184:6829","nonce":2183120457}]},"public_addr":"172.21.15.184:6825/2183120457","cluster_addr":"172.21.15.184:6827/2183120457","heartbeat_back_addr":"172.21.15.184:6831/2183120457","heartbeat_front_addr":"172.21.15.184:6829/2183120457","state":["exists","up"]},{"osd":7,"uuid":"4f381778-eee1-47c6-804f-df0e6b33e313","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":167,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6800","nonce":4102957285},{"type":"v1","addr":"172.21.15.184:6801","nonce":4102957285}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6802","nonce":4102957285},{"type":"v1","addr":"172.21.15.184:6803","nonce":4102957285}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6806","nonce":4102957285},{"type":"v1","addr":"172.21.15.184:6807","nonce":4102957285}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6804","nonce":4102957285},{"type":"v1","addr":"172.21.15.184:6805","nonce":4102957285}]},"public_addr":"172.21.15.184:6801/4102957285","cluster_addr":"172.21.15.184:6803/4102957285","heartbeat_back_addr":"172.21.15.184:6807/4102957285","heartbeat_front_addr":"172.21.15.184:6805/4102957285","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:35.972210+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.222007+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.198617+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.385663+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.358935+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.156066+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.752973+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:35.682846+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":17,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T15:52:06.932 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:06.933+0000 7f3ac5d3b640 1 -- 172.21.15.123:0/724409065 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f3aac073a90 msgr2=0x7f3aac075f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:52:06.932 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:06.933+0000 7f3ac5d3b640 1 --2- 172.21.15.123:0/724409065 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f3aac073a90 0x7f3aac075f50 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f3ab000baa0 tx=0x7f3ab0005d30 comp rx=0 tx=0).stop 2024-09-16T15:52:06.932 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:06.933+0000 7f3ac5d3b640 1 -- 172.21.15.123:0/724409065 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3ac00ce1d0 msgr2=0x7f3ac00c5b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:52:06.932 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:06.933+0000 7f3ac5d3b640 1 --2- 172.21.15.123:0/724409065 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3ac00ce1d0 0x7f3ac00c5b60 secure :-1 s=READY pgs=1943 cs=0 l=1 rev1=1 crypto rx=0x7f3ab40b0040 tx=0x7f3ab40040e0 comp rx=0 tx=0).stop 2024-09-16T15:52:06.932 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:06.933+0000 7f3ac5d3b640 1 -- 172.21.15.123:0/724409065 shutdown_connections 2024-09-16T15:52:06.933 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:06.933+0000 7f3ac5d3b640 1 --2- 172.21.15.123:0/724409065 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f3aac073a90 0x7f3aac075f50 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:52:06.933 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:06.933+0000 7f3ac5d3b640 1 --2- 172.21.15.123:0/724409065 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f3ac015cd30 0x7f3ac00c6620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:52:06.933 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:06.933+0000 7f3ac5d3b640 1 --2- 172.21.15.123:0/724409065 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3ac00ce1d0 0x7f3ac00c5b60 unknown :-1 s=CLOSED pgs=1943 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:52:06.933 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:06.933+0000 7f3ac5d3b640 1 --2- 172.21.15.123:0/724409065 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3ac00cbd20 0x7f3ac00c4f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:52:06.933 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:06.933+0000 7f3ac5d3b640 1 -- 172.21.15.123:0/724409065 >> 172.21.15.123:0/724409065 conn(0x7f3ac0074130 msgr2=0x7f3ac00c0320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:52:06.933 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:06.933+0000 7f3ac5d3b640 1 -- 172.21.15.123:0/724409065 shutdown_connections 2024-09-16T15:52:06.933 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:06.933+0000 7f3ac5d3b640 1 -- 172.21.15.123:0/724409065 wait complete. 2024-09-16T15:52:06.948 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-09-16T15:52:06.948 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-16T15:52:07.063 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:07.065+0000 7f6795207640 1 -- 172.21.15.123:0/3690314195 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f679015e160 msgr2=0x7f6790073f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:52:07.063 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:07.065+0000 7f6795207640 1 --2- 172.21.15.123:0/3690314195 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f679015e160 0x7f6790073f30 secure :-1 s=READY pgs=1945 cs=0 l=1 rev1=1 crypto rx=0x7f6790073c20 tx=0x7f678001c7c0 comp rx=0 tx=0).stop 2024-09-16T15:52:07.063 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:07.065+0000 7f6795207640 1 -- 172.21.15.123:0/3690314195 shutdown_connections 2024-09-16T15:52:07.063 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:07.065+0000 7f6795207640 1 --2- 172.21.15.123:0/3690314195 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f679015e160 0x7f6790073f30 unknown :-1 s=CLOSED pgs=1945 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:52:07.063 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:07.065+0000 7f6795207640 1 -- 172.21.15.123:0/3690314195 >> 172.21.15.123:0/3690314195 conn(0x7f67900cfb10 msgr2=0x7f67900cff20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:52:07.063 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:07.065+0000 7f6795207640 1 -- 172.21.15.123:0/3690314195 shutdown_connections 2024-09-16T15:52:07.064 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:07.065+0000 7f6795207640 1 -- 172.21.15.123:0/3690314195 wait complete. 2024-09-16T15:52:07.064 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:07.065+0000 7f6795207640 1 Processor -- start 2024-09-16T15:52:07.064 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:07.065+0000 7f6795207640 1 -- start start 2024-09-16T15:52:07.066 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:07.065+0000 7f6795207640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6790074470 0x7f67901f63a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:52:07.066 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:07.065+0000 7f6795207640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f679015e160 0x7f67900cc390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:52:07.066 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:07.065+0000 7f6795207640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f679015e820 0x7f67900cc8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:52:07.066 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:07.065+0000 7f6795207640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f67901697f0 con 0x7f6790074470 2024-09-16T15:52:07.066 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:07.065+0000 7f6795207640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f67900cf640 con 0x7f679015e160 2024-09-16T15:52:07.066 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:07.065+0000 7f6795207640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f6790169970 con 0x7f679015e820 2024-09-16T15:52:07.066 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:07.065+0000 7f678ed9d640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6790074470 0x7f67901f63a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:52:07.066 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:07.065+0000 7f678ed9d640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6790074470 0x7f67901f63a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:55484/0 (socket says 172.21.15.123:55484) 2024-09-16T15:52:07.067 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:07.065+0000 7f678ed9d640 1 -- 172.21.15.123:0/4251587796 learned_addr learned my addr 172.21.15.123:0/4251587796 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:52:07.067 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:07.065+0000 7f678e59c640 1 --2- 172.21.15.123:0/4251587796 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f679015e160 0x7f67900cc390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:52:07.067 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:07.065+0000 7f678f59e640 1 --2- 172.21.15.123:0/4251587796 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f679015e820 0x7f67900cc8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:52:07.067 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:07.065+0000 7f678ed9d640 1 -- 172.21.15.123:0/4251587796 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f679015e820 msgr2=0x7f67900cc8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:52:07.067 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:07.065+0000 7f678ed9d640 1 --2- 172.21.15.123:0/4251587796 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f679015e820 0x7f67900cc8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:52:07.067 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:07.065+0000 7f678ed9d640 1 -- 172.21.15.123:0/4251587796 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f679015e160 msgr2=0x7f67900cc390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:52:07.067 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:07.065+0000 7f678ed9d640 1 --2- 172.21.15.123:0/4251587796 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f679015e160 0x7f67900cc390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:52:07.067 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:07.065+0000 7f678ed9d640 1 -- 172.21.15.123:0/4251587796 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f679014b480 con 0x7f6790074470 2024-09-16T15:52:07.067 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:07.065+0000 7f678ed9d640 1 --2- 172.21.15.123:0/4251587796 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6790074470 0x7f67901f63a0 secure :-1 s=READY pgs=1946 cs=0 l=1 rev1=1 crypto rx=0x7f67800b0040 tx=0x7f67800a56b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:52:07.067 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:07.065+0000 7f67877fe640 1 -- 172.21.15.123:0/4251587796 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67800b2040 con 0x7f6790074470 2024-09-16T15:52:07.067 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:07.065+0000 7f67877fe640 1 -- 172.21.15.123:0/4251587796 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f67800043e0 con 0x7f6790074470 2024-09-16T15:52:07.067 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:07.065+0000 7f6795207640 1 -- 172.21.15.123:0/4251587796 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f679014b760 con 0x7f6790074470 2024-09-16T15:52:07.067 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:07.065+0000 7f67877fe640 1 -- 172.21.15.123:0/4251587796 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6780005290 con 0x7f6790074470 2024-09-16T15:52:07.067 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:07.065+0000 7f6795207640 1 -- 172.21.15.123:0/4251587796 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f679014bd20 con 0x7f6790074470 2024-09-16T15:52:07.068 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:07.069+0000 7f6795207640 1 -- 172.21.15.123:0/4251587796 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6790073f30 con 0x7f6790074470 2024-09-16T15:52:07.068 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:07.069+0000 7f67877fe640 1 -- 172.21.15.123:0/4251587796 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 27) ==== 98151+0+0 (secure 0 0 0) 0x7f6780005430 con 0x7f6790074470 2024-09-16T15:52:07.068 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:07.069+0000 7f67877fe640 1 --2- 172.21.15.123:0/4251587796 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f6770073a90 0x7f6770075f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:52:07.068 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:07.069+0000 7f67877fe640 1 -- 172.21.15.123:0/4251587796 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(169..169 src has 1..169) ==== 5737+0+0 (secure 0 0 0) 0x7f678012c550 con 0x7f6790074470 2024-09-16T15:52:07.068 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:07.069+0000 7f678e59c640 1 --2- 172.21.15.123:0/4251587796 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f6770073a90 0x7f6770075f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:52:07.069 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:07.069+0000 7f678e59c640 1 --2- 172.21.15.123:0/4251587796 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f6770073a90 0x7f6770075f50 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f67901576a0 tx=0x7f6778009040 comp rx=0 tx=0).ready entity=mgr.4150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:52:07.072 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:07.073+0000 7f67877fe640 1 -- 172.21.15.123:0/4251587796 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f67800164d0 con 0x7f6790074470 2024-09-16T15:52:07.272 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:07.273+0000 7f6795207640 1 -- 172.21.15.123:0/4251587796 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f6790157aa0 con 0x7f6790074470 2024-09-16T15:52:07.273 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:07.273+0000 7f67877fe640 1 -- 172.21.15.123:0/4251587796 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 116 v116) ==== 95+0+5239 (secure 0 0 0) 0x7f67800f8ff0 con 0x7f6790074470 2024-09-16T15:52:07.274 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-09-16T15:52:07.274 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":116,"btime":"2024-09-16T15:52:00:476603+0000","default_fscid":13,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6314,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6833/2070840764","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":2070840764},{"type":"v1","addr":"172.21.15.184:6833","nonce":2070840764}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":110},{"gid":6976,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6835/3881288354","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":3881288354},{"type":"v1","addr":"172.21.15.184:6835","nonce":3881288354}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":110},{"gid":9219,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6833/519008782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6832","nonce":519008782},{"type":"v1","addr":"172.21.15.123:6833","nonce":519008782}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":110}],"filesystems":[{"mdsmap":{"epoch":115,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-16T15:51:47.741329+0000","modified":"2024-09-16T15:51:50.794388+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":9234},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9234":{"gid":9234,"name":"a","rank":0,"incarnation":111,"state":"up:active","state_seq":3,"addr":"172.21.15.123:6835/370478759","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6834","nonce":370478759},{"type":"v1","addr":"172.21.15.123:6835","nonce":370478759}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[27],"metadata_pool":26,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9234,"qdb_cluster":[9234]},"id":13}]} 2024-09-16T15:52:07.274 INFO:teuthology.orchestra.run.smithi123.stderr:dumped fsmap epoch 116 2024-09-16T15:52:07.276 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:07.277+0000 7f6795207640 1 -- 172.21.15.123:0/4251587796 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f6770073a90 msgr2=0x7f6770075f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:52:07.277 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:07.277+0000 7f6795207640 1 --2- 172.21.15.123:0/4251587796 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f6770073a90 0x7f6770075f50 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f67901576a0 tx=0x7f6778009040 comp rx=0 tx=0).stop 2024-09-16T15:52:07.277 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:07.277+0000 7f6795207640 1 -- 172.21.15.123:0/4251587796 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6790074470 msgr2=0x7f67901f63a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:52:07.277 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:07.277+0000 7f6795207640 1 --2- 172.21.15.123:0/4251587796 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6790074470 0x7f67901f63a0 secure :-1 s=READY pgs=1946 cs=0 l=1 rev1=1 crypto rx=0x7f67800b0040 tx=0x7f67800a56b0 comp rx=0 tx=0).stop 2024-09-16T15:52:07.277 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:07.277+0000 7f6795207640 1 -- 172.21.15.123:0/4251587796 shutdown_connections 2024-09-16T15:52:07.277 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:07.277+0000 7f6795207640 1 --2- 172.21.15.123:0/4251587796 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f6770073a90 0x7f6770075f50 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:52:07.277 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:07.277+0000 7f6795207640 1 --2- 172.21.15.123:0/4251587796 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f679015e820 0x7f67900cc8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:52:07.277 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:07.277+0000 7f6795207640 1 --2- 172.21.15.123:0/4251587796 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f679015e160 0x7f67900cc390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:52:07.277 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:07.277+0000 7f6795207640 1 --2- 172.21.15.123:0/4251587796 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6790074470 0x7f67901f63a0 unknown :-1 s=CLOSED pgs=1946 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:52:07.277 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:07.277+0000 7f6795207640 1 -- 172.21.15.123:0/4251587796 >> 172.21.15.123:0/4251587796 conn(0x7f67900cfb10 msgr2=0x7f679006faf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:52:07.277 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:07.277+0000 7f6795207640 1 -- 172.21.15.123:0/4251587796 shutdown_connections 2024-09-16T15:52:07.277 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:07.277+0000 7f6795207640 1 -- 172.21.15.123:0/4251587796 wait complete. 2024-09-16T15:52:07.291 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-16T15:52:07.404 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:07.405+0000 7f7390cfb640 1 -- 172.21.15.123:0/653410467 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f738c15e040 msgr2=0x7f738c0cea50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:52:07.404 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:07.405+0000 7f7390cfb640 1 --2- 172.21.15.123:0/653410467 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f738c15e040 0x7f738c0cea50 secure :-1 s=READY pgs=1947 cs=0 l=1 rev1=1 crypto rx=0x7f737c0098f0 tx=0x7f737c01c850 comp rx=0 tx=0).stop 2024-09-16T15:52:07.404 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:07.405+0000 7f7390cfb640 1 -- 172.21.15.123:0/653410467 shutdown_connections 2024-09-16T15:52:07.404 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:07.405+0000 7f7390cfb640 1 --2- 172.21.15.123:0/653410467 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f738c0b0130 0x7f738c0cef90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:52:07.404 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:07.405+0000 7f7390cfb640 1 --2- 172.21.15.123:0/653410467 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f738c15e040 0x7f738c0cea50 unknown :-1 s=CLOSED pgs=1947 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:52:07.405 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:07.405+0000 7f7390cfb640 1 -- 172.21.15.123:0/653410467 >> 172.21.15.123:0/653410467 conn(0x7f738c0cd5d0 msgr2=0x7f738c0cd9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:52:07.405 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:07.405+0000 7f7390cfb640 1 -- 172.21.15.123:0/653410467 shutdown_connections 2024-09-16T15:52:07.405 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:07.405+0000 7f7390cfb640 1 -- 172.21.15.123:0/653410467 wait complete. 2024-09-16T15:52:07.405 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:07.405+0000 7f7390cfb640 1 Processor -- start 2024-09-16T15:52:07.405 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:07.405+0000 7f7390cfb640 1 -- start start 2024-09-16T15:52:07.407 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:07.405+0000 7f7390cfb640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f738c073600 0x7f738c14f3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:52:07.407 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:07.405+0000 7f7390cfb640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f738c0b0130 0x7f738c0b6340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:52:07.407 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:07.405+0000 7f7390cfb640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f738c15e040 0x7f738c1fb2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:52:07.408 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:07.405+0000 7f7390cfb640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f738c16b4c0 con 0x7f738c0b0130 2024-09-16T15:52:07.408 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:07.405+0000 7f7390cfb640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f738c16b340 con 0x7f738c15e040 2024-09-16T15:52:07.408 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:07.405+0000 7f7390cfb640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f738c16b640 con 0x7f738c073600 2024-09-16T15:52:07.408 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:07.405+0000 7f7381d9b640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f738c0b0130 0x7f738c0b6340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:52:07.408 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:07.405+0000 7f7381d9b640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f738c0b0130 0x7f738c0b6340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:55506/0 (socket says 172.21.15.123:55506) 2024-09-16T15:52:07.408 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:07.405+0000 7f7381d9b640 1 -- 172.21.15.123:0/4052492578 learned_addr learned my addr 172.21.15.123:0/4052492578 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:52:07.408 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:07.405+0000 7f738ad9d640 1 --2- 172.21.15.123:0/4052492578 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f738c15e040 0x7f738c1fb2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:52:07.408 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:07.405+0000 7f738a59c640 1 --2- 172.21.15.123:0/4052492578 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f738c073600 0x7f738c14f3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:52:07.408 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:07.409+0000 7f7381d9b640 1 -- 172.21.15.123:0/4052492578 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f738c073600 msgr2=0x7f738c14f3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:52:07.408 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:07.409+0000 7f7381d9b640 1 --2- 172.21.15.123:0/4052492578 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f738c073600 0x7f738c14f3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:52:07.409 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:07.409+0000 7f7381d9b640 1 -- 172.21.15.123:0/4052492578 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f738c15e040 msgr2=0x7f738c1fb2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:52:07.409 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:07.409+0000 7f7381d9b640 1 --2- 172.21.15.123:0/4052492578 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f738c15e040 0x7f738c1fb2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:52:07.409 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:07.409+0000 7f7381d9b640 1 -- 172.21.15.123:0/4052492578 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f738c0b6880 con 0x7f738c0b0130 2024-09-16T15:52:07.409 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:07.409+0000 7f738a59c640 1 --2- 172.21.15.123:0/4052492578 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f738c073600 0x7f738c14f3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T15:52:07.409 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:07.409+0000 7f7381d9b640 1 --2- 172.21.15.123:0/4052492578 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f738c0b0130 0x7f738c0b6340 secure :-1 s=READY pgs=1948 cs=0 l=1 rev1=1 crypto rx=0x7f737400c970 tx=0x7f737400ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:52:07.409 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:07.409+0000 7f73837fe640 1 -- 172.21.15.123:0/4052492578 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7374004230 con 0x7f738c0b0130 2024-09-16T15:52:07.409 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:07.409+0000 7f73837fe640 1 -- 172.21.15.123:0/4052492578 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f73740044c0 con 0x7f738c0b0130 2024-09-16T15:52:07.409 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:07.409+0000 7f7390cfb640 1 -- 172.21.15.123:0/4052492578 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f738c0c6150 con 0x7f738c0b0130 2024-09-16T15:52:07.409 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:07.409+0000 7f73837fe640 1 -- 172.21.15.123:0/4052492578 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7374007c10 con 0x7f738c0b0130 2024-09-16T15:52:07.409 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:07.409+0000 7f7390cfb640 1 -- 172.21.15.123:0/4052492578 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f738c0c6710 con 0x7f738c0b0130 2024-09-16T15:52:07.409 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:07.409+0000 7f7390cfb640 1 -- 172.21.15.123:0/4052492578 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f738c06cbf0 con 0x7f738c0b0130 2024-09-16T15:52:07.410 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:07.409+0000 7f73837fe640 1 -- 172.21.15.123:0/4052492578 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 27) ==== 98151+0+0 (secure 0 0 0) 0x7f7374003600 con 0x7f738c0b0130 2024-09-16T15:52:07.410 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:07.409+0000 7f73837fe640 1 --2- 172.21.15.123:0/4052492578 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f736c073a90 0x7f736c075f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:52:07.410 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:07.409+0000 7f73837fe640 1 -- 172.21.15.123:0/4052492578 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(169..169 src has 1..169) ==== 5737+0+0 (secure 0 0 0) 0x7f7374093010 con 0x7f738c0b0130 2024-09-16T15:52:07.410 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:07.409+0000 7f738a59c640 1 --2- 172.21.15.123:0/4052492578 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f736c073a90 0x7f736c075f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:52:07.411 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:07.409+0000 7f738a59c640 1 --2- 172.21.15.123:0/4052492578 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f736c073a90 0x7f736c075f50 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f737c0ae000 tx=0x7f737c01b090 comp rx=0 tx=0).ready entity=mgr.4150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:52:07.412 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:07.413+0000 7f73837fe640 1 -- 172.21.15.123:0/4052492578 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f7374014070 con 0x7f738c0b0130 2024-09-16T15:52:07.608 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:07.609+0000 7f7390cfb640 1 -- 172.21.15.123:0/4052492578 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f738c0747c0 con 0x7f738c0b0130 2024-09-16T15:52:07.609 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:07.609+0000 7f73837fe640 1 -- 172.21.15.123:0/4052492578 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 116 v116) ==== 95+0+5239 (secure 0 0 0) 0x7f737405fcf0 con 0x7f738c0b0130 2024-09-16T15:52:07.609 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-09-16T15:52:07.610 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":116,"btime":"2024-09-16T15:52:00:476603+0000","default_fscid":13,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6314,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6833/2070840764","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":2070840764},{"type":"v1","addr":"172.21.15.184:6833","nonce":2070840764}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":110},{"gid":6976,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6835/3881288354","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":3881288354},{"type":"v1","addr":"172.21.15.184:6835","nonce":3881288354}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":110},{"gid":9219,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6833/519008782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6832","nonce":519008782},{"type":"v1","addr":"172.21.15.123:6833","nonce":519008782}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":110}],"filesystems":[{"mdsmap":{"epoch":115,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-16T15:51:47.741329+0000","modified":"2024-09-16T15:51:50.794388+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":9234},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9234":{"gid":9234,"name":"a","rank":0,"incarnation":111,"state":"up:active","state_seq":3,"addr":"172.21.15.123:6835/370478759","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6834","nonce":370478759},{"type":"v1","addr":"172.21.15.123:6835","nonce":370478759}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[27],"metadata_pool":26,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9234,"qdb_cluster":[9234]},"id":13}]} 2024-09-16T15:52:07.610 INFO:teuthology.orchestra.run.smithi123.stderr:dumped fsmap epoch 116 2024-09-16T15:52:07.612 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:07.613+0000 7f7390cfb640 1 -- 172.21.15.123:0/4052492578 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f736c073a90 msgr2=0x7f736c075f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:52:07.612 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:07.613+0000 7f7390cfb640 1 --2- 172.21.15.123:0/4052492578 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f736c073a90 0x7f736c075f50 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f737c0ae000 tx=0x7f737c01b090 comp rx=0 tx=0).stop 2024-09-16T15:52:07.612 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:07.613+0000 7f7390cfb640 1 -- 172.21.15.123:0/4052492578 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f738c0b0130 msgr2=0x7f738c0b6340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:52:07.612 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:07.613+0000 7f7390cfb640 1 --2- 172.21.15.123:0/4052492578 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f738c0b0130 0x7f738c0b6340 secure :-1 s=READY pgs=1948 cs=0 l=1 rev1=1 crypto rx=0x7f737400c970 tx=0x7f737400ce30 comp rx=0 tx=0).stop 2024-09-16T15:52:07.612 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:07.613+0000 7f7390cfb640 1 -- 172.21.15.123:0/4052492578 shutdown_connections 2024-09-16T15:52:07.612 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:07.613+0000 7f7390cfb640 1 --2- 172.21.15.123:0/4052492578 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f736c073a90 0x7f736c075f50 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:52:07.612 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:07.613+0000 7f7390cfb640 1 --2- 172.21.15.123:0/4052492578 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f738c15e040 0x7f738c1fb2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:52:07.612 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:07.613+0000 7f7390cfb640 1 --2- 172.21.15.123:0/4052492578 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f738c0b0130 0x7f738c0b6340 unknown :-1 s=CLOSED pgs=1948 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:52:07.612 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:07.613+0000 7f7390cfb640 1 --2- 172.21.15.123:0/4052492578 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f738c073600 0x7f738c14f3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:52:07.613 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:07.613+0000 7f7390cfb640 1 -- 172.21.15.123:0/4052492578 >> 172.21.15.123:0/4052492578 conn(0x7f738c0cd5d0 msgr2=0x7f738c0cd9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:52:07.613 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:07.613+0000 7f7390cfb640 1 -- 172.21.15.123:0/4052492578 shutdown_connections 2024-09-16T15:52:07.613 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:07.613+0000 7f7390cfb640 1 -- 172.21.15.123:0/4052492578 wait complete. 2024-09-16T15:52:07.627 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-16T15:52:07.743 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:07.745+0000 7fedd77ae640 1 -- 172.21.15.123:0/3839147463 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fedd0074600 msgr2=0x7fedd006e750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:52:07.743 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:07.745+0000 7fedd77ae640 1 --2- 172.21.15.123:0/3839147463 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fedd0074600 0x7fedd006e750 secure :-1 s=READY pgs=1950 cs=0 l=1 rev1=1 crypto rx=0x7fedc8009c80 tx=0x7fedc801cce0 comp rx=0 tx=0).stop 2024-09-16T15:52:07.743 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:07.745+0000 7fedd77ae640 1 -- 172.21.15.123:0/3839147463 shutdown_connections 2024-09-16T15:52:07.743 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:07.745+0000 7fedd77ae640 1 --2- 172.21.15.123:0/3839147463 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fedd0074600 0x7fedd006e750 unknown :-1 s=CLOSED pgs=1950 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:52:07.743 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:07.745+0000 7fedd77ae640 1 --2- 172.21.15.123:0/3839147463 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fedd015bca0 0x7fedd00d0630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:52:07.743 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:07.745+0000 7fedd77ae640 1 -- 172.21.15.123:0/3839147463 >> 172.21.15.123:0/3839147463 conn(0x7fedd00cfb10 msgr2=0x7fedd00cff20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:52:07.743 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:07.745+0000 7fedd77ae640 1 -- 172.21.15.123:0/3839147463 shutdown_connections 2024-09-16T15:52:07.743 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:07.745+0000 7fedd77ae640 1 -- 172.21.15.123:0/3839147463 wait complete. 2024-09-16T15:52:07.744 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:07.745+0000 7fedd77ae640 1 Processor -- start 2024-09-16T15:52:07.744 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:07.745+0000 7fedd77ae640 1 -- start start 2024-09-16T15:52:07.746 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:07.745+0000 7fedd77ae640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fedd00726c0 0x7fedd0159110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:52:07.746 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:07.745+0000 7fedd77ae640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fedd0074600 0x7fedd0165690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:52:07.746 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:07.745+0000 7fedd77ae640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fedd015bca0 0x7fedd0140c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:52:07.746 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:07.745+0000 7fedd77ae640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fedd0168bd0 con 0x7fedd015bca0 2024-09-16T15:52:07.746 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:07.745+0000 7fedd77ae640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fedd0168a50 con 0x7fedd00726c0 2024-09-16T15:52:07.746 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:07.745+0000 7fedd77ae640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fedd0168d50 con 0x7fedd0074600 2024-09-16T15:52:07.746 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:07.745+0000 7fedd5d4b640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fedd015bca0 0x7fedd0140c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:52:07.746 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:07.745+0000 7fedd5d4b640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fedd015bca0 0x7fedd0140c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:55530/0 (socket says 172.21.15.123:55530) 2024-09-16T15:52:07.746 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:07.745+0000 7fedd5d4b640 1 -- 172.21.15.123:0/2753388515 learned_addr learned my addr 172.21.15.123:0/2753388515 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:52:07.746 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:07.745+0000 7fedd4d49640 1 --2- 172.21.15.123:0/2753388515 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fedd0074600 0x7fedd0165690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:52:07.747 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:07.745+0000 7fedd5d4b640 1 -- 172.21.15.123:0/2753388515 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fedd0074600 msgr2=0x7fedd0165690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:52:07.747 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:07.745+0000 7fedd5d4b640 1 --2- 172.21.15.123:0/2753388515 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fedd0074600 0x7fedd0165690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:52:07.747 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:07.745+0000 7fedd5d4b640 1 -- 172.21.15.123:0/2753388515 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fedd00726c0 msgr2=0x7fedd0159110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:52:07.747 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:07.745+0000 7fedd5d4b640 1 --2- 172.21.15.123:0/2753388515 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fedd00726c0 0x7fedd0159110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:52:07.747 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:07.745+0000 7fedd5d4b640 1 -- 172.21.15.123:0/2753388515 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fedd0158070 con 0x7fedd015bca0 2024-09-16T15:52:07.747 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:07.745+0000 7fedd5d4b640 1 --2- 172.21.15.123:0/2753388515 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fedd015bca0 0x7fedd0140c40 secure :-1 s=READY pgs=1951 cs=0 l=1 rev1=1 crypto rx=0x7fedc8009bd0 tx=0x7fedc8009390 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:52:07.747 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:07.745+0000 7fedc5ffb640 1 -- 172.21.15.123:0/2753388515 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fedc80bb070 con 0x7fedd015bca0 2024-09-16T15:52:07.747 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:07.745+0000 7fedc5ffb640 1 -- 172.21.15.123:0/2753388515 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fedc8004e00 con 0x7fedd015bca0 2024-09-16T15:52:07.747 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:07.745+0000 7fedd77ae640 1 -- 172.21.15.123:0/2753388515 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fedd0158350 con 0x7fedd015bca0 2024-09-16T15:52:07.747 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:07.745+0000 7fedc5ffb640 1 -- 172.21.15.123:0/2753388515 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fedc80b2040 con 0x7fedd015bca0 2024-09-16T15:52:07.747 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:07.745+0000 7fedd77ae640 1 -- 172.21.15.123:0/2753388515 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fedd0158910 con 0x7fedd015bca0 2024-09-16T15:52:07.748 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:07.745+0000 7fedd77ae640 1 -- 172.21.15.123:0/2753388515 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fedd006e750 con 0x7fedd015bca0 2024-09-16T15:52:07.748 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:07.749+0000 7fedc5ffb640 1 -- 172.21.15.123:0/2753388515 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 27) ==== 98151+0+0 (secure 0 0 0) 0x7fedc80be070 con 0x7fedd015bca0 2024-09-16T15:52:07.748 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:07.749+0000 7fedc5ffb640 1 --2- 172.21.15.123:0/2753388515 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fedb4073a90 0x7fedb4075f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:52:07.749 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:07.749+0000 7fedd554a640 1 --2- 172.21.15.123:0/2753388515 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fedb4073a90 0x7fedb4075f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:52:07.749 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:07.749+0000 7fedc5ffb640 1 -- 172.21.15.123:0/2753388515 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(169..169 src has 1..169) ==== 5737+0+0 (secure 0 0 0) 0x7fedc812c7a0 con 0x7fedd015bca0 2024-09-16T15:52:07.749 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:07.749+0000 7fedd554a640 1 --2- 172.21.15.123:0/2753388515 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fedb4073a90 0x7fedb4075f50 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7fedcc0058d0 tx=0x7fedcc005860 comp rx=0 tx=0).ready entity=mgr.4150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:52:07.752 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:07.753+0000 7fedc5ffb640 1 -- 172.21.15.123:0/2753388515 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7fedc8016990 con 0x7fedd015bca0 2024-09-16T15:52:07.927 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:07.929+0000 7fedd77ae640 1 -- 172.21.15.123:0/2753388515 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fedd00700e0 con 0x7fedd015bca0 2024-09-16T15:52:07.929 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:07.929+0000 7fedc5ffb640 1 -- 172.21.15.123:0/2753388515 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v169) ==== 74+0+16847 (secure 0 0 0) 0x7fedc80f9240 con 0x7fedd015bca0 2024-09-16T15:52:07.929 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-09-16T15:52:07.930 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":169,"fsid":"56077f89-18c8-486f-ae7e-eecc0bb7822c","created":"2024-09-16T15:42:26.364892+0000","modified":"2024-09-16T15:51:49.376383+0000","last_up_change":"2024-09-16T15:42:38.707234+0000","last_in_change":"2024-09-16T15:42:29.423662+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":27,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-16T15:42:39.287764+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":26,"pool_name":"cephfs_metadata","create_time":"2024-09-16T15:51:45.417755+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"168","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":27,"pool_name":"cephfs_data","create_time":"2024-09-16T15:51:46.701679+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"169","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9e7c1955-a607-4671-9be7-f2ef760ceeaa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":167,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6800","nonce":3882044969},{"type":"v1","addr":"172.21.15.123:6801","nonce":3882044969}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6802","nonce":3882044969},{"type":"v1","addr":"172.21.15.123:6803","nonce":3882044969}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6806","nonce":3882044969},{"type":"v1","addr":"172.21.15.123:6807","nonce":3882044969}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6804","nonce":3882044969},{"type":"v1","addr":"172.21.15.123:6805","nonce":3882044969}]},"public_addr":"172.21.15.123:6801/3882044969","cluster_addr":"172.21.15.123:6803/3882044969","heartbeat_back_addr":"172.21.15.123:6807/3882044969","heartbeat_front_addr":"172.21.15.123:6805/3882044969","state":["exists","up"]},{"osd":1,"uuid":"c540cfde-0977-4742-9b17-fd634668e010","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":167,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6816","nonce":4037001850},{"type":"v1","addr":"172.21.15.123:6817","nonce":4037001850}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6818","nonce":4037001850},{"type":"v1","addr":"172.21.15.123:6819","nonce":4037001850}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6822","nonce":4037001850},{"type":"v1","addr":"172.21.15.123:6823","nonce":4037001850}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6820","nonce":4037001850},{"type":"v1","addr":"172.21.15.123:6821","nonce":4037001850}]},"public_addr":"172.21.15.123:6817/4037001850","cluster_addr":"172.21.15.123:6819/4037001850","heartbeat_back_addr":"172.21.15.123:6823/4037001850","heartbeat_front_addr":"172.21.15.123:6821/4037001850","state":["exists","up"]},{"osd":2,"uuid":"9864a496-f169-486b-95c4-1d59162793a7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":167,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6824","nonce":1332143088},{"type":"v1","addr":"172.21.15.123:6825","nonce":1332143088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6826","nonce":1332143088},{"type":"v1","addr":"172.21.15.123:6827","nonce":1332143088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6830","nonce":1332143088},{"type":"v1","addr":"172.21.15.123:6831","nonce":1332143088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6828","nonce":1332143088},{"type":"v1","addr":"172.21.15.123:6829","nonce":1332143088}]},"public_addr":"172.21.15.123:6825/1332143088","cluster_addr":"172.21.15.123:6827/1332143088","heartbeat_back_addr":"172.21.15.123:6831/1332143088","heartbeat_front_addr":"172.21.15.123:6829/1332143088","state":["exists","up"]},{"osd":3,"uuid":"c6d1f817-f074-4325-95e1-578eba13edda","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":167,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6808","nonce":988586998},{"type":"v1","addr":"172.21.15.123:6809","nonce":988586998}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6810","nonce":988586998},{"type":"v1","addr":"172.21.15.123:6811","nonce":988586998}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6814","nonce":988586998},{"type":"v1","addr":"172.21.15.123:6815","nonce":988586998}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6812","nonce":988586998},{"type":"v1","addr":"172.21.15.123:6813","nonce":988586998}]},"public_addr":"172.21.15.123:6809/988586998","cluster_addr":"172.21.15.123:6811/988586998","heartbeat_back_addr":"172.21.15.123:6815/988586998","heartbeat_front_addr":"172.21.15.123:6813/988586998","state":["exists","up"]},{"osd":4,"uuid":"200a8df4-50ef-40a5-b48a-58981269c086","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":167,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6816","nonce":3387813890},{"type":"v1","addr":"172.21.15.184:6817","nonce":3387813890}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6818","nonce":3387813890},{"type":"v1","addr":"172.21.15.184:6819","nonce":3387813890}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6822","nonce":3387813890},{"type":"v1","addr":"172.21.15.184:6823","nonce":3387813890}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6820","nonce":3387813890},{"type":"v1","addr":"172.21.15.184:6821","nonce":3387813890}]},"public_addr":"172.21.15.184:6817/3387813890","cluster_addr":"172.21.15.184:6819/3387813890","heartbeat_back_addr":"172.21.15.184:6823/3387813890","heartbeat_front_addr":"172.21.15.184:6821/3387813890","state":["exists","up"]},{"osd":5,"uuid":"32d5a900-da3d-412c-bc19-13861a2368d8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":167,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6808","nonce":2680916017},{"type":"v1","addr":"172.21.15.184:6809","nonce":2680916017}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6810","nonce":2680916017},{"type":"v1","addr":"172.21.15.184:6811","nonce":2680916017}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6814","nonce":2680916017},{"type":"v1","addr":"172.21.15.184:6815","nonce":2680916017}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6812","nonce":2680916017},{"type":"v1","addr":"172.21.15.184:6813","nonce":2680916017}]},"public_addr":"172.21.15.184:6809/2680916017","cluster_addr":"172.21.15.184:6811/2680916017","heartbeat_back_addr":"172.21.15.184:6815/2680916017","heartbeat_front_addr":"172.21.15.184:6813/2680916017","state":["exists","up"]},{"osd":6,"uuid":"cde6f39e-aa1b-4260-ac19-774ee6aaca72","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":167,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6824","nonce":2183120457},{"type":"v1","addr":"172.21.15.184:6825","nonce":2183120457}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6826","nonce":2183120457},{"type":"v1","addr":"172.21.15.184:6827","nonce":2183120457}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6830","nonce":2183120457},{"type":"v1","addr":"172.21.15.184:6831","nonce":2183120457}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6828","nonce":2183120457},{"type":"v1","addr":"172.21.15.184:6829","nonce":2183120457}]},"public_addr":"172.21.15.184:6825/2183120457","cluster_addr":"172.21.15.184:6827/2183120457","heartbeat_back_addr":"172.21.15.184:6831/2183120457","heartbeat_front_addr":"172.21.15.184:6829/2183120457","state":["exists","up"]},{"osd":7,"uuid":"4f381778-eee1-47c6-804f-df0e6b33e313","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":167,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6800","nonce":4102957285},{"type":"v1","addr":"172.21.15.184:6801","nonce":4102957285}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6802","nonce":4102957285},{"type":"v1","addr":"172.21.15.184:6803","nonce":4102957285}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6806","nonce":4102957285},{"type":"v1","addr":"172.21.15.184:6807","nonce":4102957285}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6804","nonce":4102957285},{"type":"v1","addr":"172.21.15.184:6805","nonce":4102957285}]},"public_addr":"172.21.15.184:6801/4102957285","cluster_addr":"172.21.15.184:6803/4102957285","heartbeat_back_addr":"172.21.15.184:6807/4102957285","heartbeat_front_addr":"172.21.15.184:6805/4102957285","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:35.972210+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.222007+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.198617+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.385663+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.358935+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.156066+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.752973+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:35.682846+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":17,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T15:52:07.932 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:07.933+0000 7fedd77ae640 1 -- 172.21.15.123:0/2753388515 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fedb4073a90 msgr2=0x7fedb4075f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:52:07.932 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:07.933+0000 7fedd77ae640 1 --2- 172.21.15.123:0/2753388515 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fedb4073a90 0x7fedb4075f50 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7fedcc0058d0 tx=0x7fedcc005860 comp rx=0 tx=0).stop 2024-09-16T15:52:07.932 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:07.933+0000 7fedd77ae640 1 -- 172.21.15.123:0/2753388515 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fedd015bca0 msgr2=0x7fedd0140c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:52:07.932 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:07.933+0000 7fedd77ae640 1 --2- 172.21.15.123:0/2753388515 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fedd015bca0 0x7fedd0140c40 secure :-1 s=READY pgs=1951 cs=0 l=1 rev1=1 crypto rx=0x7fedc8009bd0 tx=0x7fedc8009390 comp rx=0 tx=0).stop 2024-09-16T15:52:07.933 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:07.933+0000 7fedd77ae640 1 -- 172.21.15.123:0/2753388515 shutdown_connections 2024-09-16T15:52:07.933 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:07.933+0000 7fedd77ae640 1 --2- 172.21.15.123:0/2753388515 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fedb4073a90 0x7fedb4075f50 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:52:07.933 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:07.933+0000 7fedd77ae640 1 --2- 172.21.15.123:0/2753388515 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fedd015bca0 0x7fedd0140c40 unknown :-1 s=CLOSED pgs=1951 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:52:07.933 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:07.933+0000 7fedd77ae640 1 --2- 172.21.15.123:0/2753388515 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fedd0074600 0x7fedd0165690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:52:07.933 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:07.933+0000 7fedd77ae640 1 --2- 172.21.15.123:0/2753388515 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fedd00726c0 0x7fedd0159110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:52:07.933 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:07.933+0000 7fedd77ae640 1 -- 172.21.15.123:0/2753388515 >> 172.21.15.123:0/2753388515 conn(0x7fedd00cfb10 msgr2=0x7fedd00b43f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:52:07.933 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:07.933+0000 7fedd77ae640 1 -- 172.21.15.123:0/2753388515 shutdown_connections 2024-09-16T15:52:07.933 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:07.933+0000 7fedd77ae640 1 -- 172.21.15.123:0/2753388515 wait complete. 2024-09-16T15:52:07.948 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-09-16T15:52:08.067 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:08.069+0000 7f0e04e78640 1 -- 172.21.15.123:0/2568706158 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0e00074470 msgr2=0x7f0e000cd6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:52:08.067 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:08.069+0000 7f0e04e78640 1 --2- 172.21.15.123:0/2568706158 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0e00074470 0x7f0e000cd6e0 secure :-1 s=READY pgs=1952 cs=0 l=1 rev1=1 crypto rx=0x7f0df0009900 tx=0x7f0df001c850 comp rx=0 tx=0).stop 2024-09-16T15:52:08.068 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:08.069+0000 7f0e04e78640 1 -- 172.21.15.123:0/2568706158 shutdown_connections 2024-09-16T15:52:08.068 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:08.069+0000 7f0e04e78640 1 --2- 172.21.15.123:0/2568706158 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0e00074470 0x7f0e000cd6e0 unknown :-1 s=CLOSED pgs=1952 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:52:08.068 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:08.069+0000 7f0e04e78640 1 --2- 172.21.15.123:0/2568706158 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f0e0015e160 0x7f0e00073f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:52:08.068 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:08.069+0000 7f0e04e78640 1 -- 172.21.15.123:0/2568706158 >> 172.21.15.123:0/2568706158 conn(0x7f0e000cfb10 msgr2=0x7f0e000cff20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:52:08.068 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:08.069+0000 7f0e04e78640 1 -- 172.21.15.123:0/2568706158 shutdown_connections 2024-09-16T15:52:08.068 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:08.069+0000 7f0e04e78640 1 -- 172.21.15.123:0/2568706158 wait complete. 2024-09-16T15:52:08.071 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:08.069+0000 7f0e04e78640 1 Processor -- start 2024-09-16T15:52:08.071 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:08.069+0000 7f0e04e78640 1 -- start start 2024-09-16T15:52:08.071 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:08.069+0000 7f0e04e78640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f0e00074470 0x7f0e0015d070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:52:08.071 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:08.069+0000 7f0e04e78640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0e000cdc20 0x7f0e000c5e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:52:08.071 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:08.069+0000 7f0e04e78640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f0e0015e160 0x7f0e00150e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:52:08.071 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:08.069+0000 7f0e04e78640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f0e00167f50 con 0x7f0e000cdc20 2024-09-16T15:52:08.071 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:08.069+0000 7f0e04e78640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f0e00167dd0 con 0x7f0e0015e160 2024-09-16T15:52:08.071 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:08.069+0000 7f0e04e78640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f0e001680d0 con 0x7f0e00074470 2024-09-16T15:52:08.071 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:08.069+0000 7f0dfdd9b640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0e000cdc20 0x7f0e000c5e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:52:08.071 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:08.069+0000 7f0dfdd9b640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0e000cdc20 0x7f0e000c5e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:55552/0 (socket says 172.21.15.123:55552) 2024-09-16T15:52:08.072 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:08.069+0000 7f0dfdd9b640 1 -- 172.21.15.123:0/1205772166 learned_addr learned my addr 172.21.15.123:0/1205772166 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:52:08.072 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:08.069+0000 7f0dfe59c640 1 --2- 172.21.15.123:0/1205772166 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f0e00074470 0x7f0e0015d070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:52:08.072 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:08.069+0000 7f0dfdd9b640 1 -- 172.21.15.123:0/1205772166 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f0e00074470 msgr2=0x7f0e0015d070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:52:08.072 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:08.069+0000 7f0dfdd9b640 1 --2- 172.21.15.123:0/1205772166 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f0e00074470 0x7f0e0015d070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:52:08.072 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:08.069+0000 7f0dfdd9b640 1 -- 172.21.15.123:0/1205772166 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f0e0015e160 msgr2=0x7f0e00150e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:52:08.072 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:08.069+0000 7f0dfdd9b640 1 --2- 172.21.15.123:0/1205772166 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f0e0015e160 0x7f0e00150e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:52:08.072 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:08.069+0000 7f0dfdd9b640 1 -- 172.21.15.123:0/1205772166 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0e000c49f0 con 0x7f0e000cdc20 2024-09-16T15:52:08.072 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:08.069+0000 7f0dfdd9b640 1 --2- 172.21.15.123:0/1205772166 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0e000cdc20 0x7f0e000c5e70 secure :-1 s=READY pgs=1953 cs=0 l=1 rev1=1 crypto rx=0x7f0df0009850 tx=0x7f0df00a55f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:52:08.072 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:08.069+0000 7f0deeffd640 1 -- 172.21.15.123:0/1205772166 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0df00b2040 con 0x7f0e000cdc20 2024-09-16T15:52:08.072 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:08.069+0000 7f0deeffd640 1 -- 172.21.15.123:0/1205772166 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f0df0004470 con 0x7f0e000cdc20 2024-09-16T15:52:08.073 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:08.069+0000 7f0e04e78640 1 -- 172.21.15.123:0/1205772166 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0e000c4cb0 con 0x7f0e000cdc20 2024-09-16T15:52:08.073 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:08.069+0000 7f0deeffd640 1 -- 172.21.15.123:0/1205772166 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0df0005290 con 0x7f0e000cdc20 2024-09-16T15:52:08.073 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:08.069+0000 7f0e04e78640 1 -- 172.21.15.123:0/1205772166 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0e00151350 con 0x7f0e000cdc20 2024-09-16T15:52:08.073 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:08.073+0000 7f0e04e78640 1 -- 172.21.15.123:0/1205772166 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0e000cd6e0 con 0x7f0e000cdc20 2024-09-16T15:52:08.074 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:08.073+0000 7f0deeffd640 1 -- 172.21.15.123:0/1205772166 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 27) ==== 98151+0+0 (secure 0 0 0) 0x7f0df0005430 con 0x7f0e000cdc20 2024-09-16T15:52:08.074 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:08.073+0000 7f0deeffd640 1 --2- 172.21.15.123:0/1205772166 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f0de0073ac0 0x7f0de0075f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:52:08.074 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:08.073+0000 7f0deeffd640 1 -- 172.21.15.123:0/1205772166 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(169..169 src has 1..169) ==== 5737+0+0 (secure 0 0 0) 0x7f0df012c9a0 con 0x7f0e000cdc20 2024-09-16T15:52:08.074 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:08.073+0000 7f0dfe59c640 1 --2- 172.21.15.123:0/1205772166 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f0de0073ac0 0x7f0de0075f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:52:08.074 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:08.073+0000 7f0dfe59c640 1 --2- 172.21.15.123:0/1205772166 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f0de0073ac0 0x7f0de0075f80 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f0df40058d0 tx=0x7f0df4005860 comp rx=0 tx=0).ready entity=mgr.4150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:52:08.075 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:08.077+0000 7f0deeffd640 1 -- 172.21.15.123:0/1205772166 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f0df00164d0 con 0x7f0e000cdc20 2024-09-16T15:52:08.269 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:08.269+0000 7f0e04e78640 1 -- 172.21.15.123:0/1205772166 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7f0e00157d50 con 0x7f0e000cdc20 2024-09-16T15:52:08.270 INFO:tasks.ceph.mon.a.smithi123.stderr:2024-09-16T15:52:08.269+0000 7f268d076640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-09-16T15:52:08.272 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:08.273+0000 7f0deeffd640 1 -- 172.21.15.123:0/1205772166 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v117) ==== 161+0+0 (secure 0 0 0) 0x7f0df00f9360 con 0x7f0e000cdc20 2024-09-16T15:52:08.272 INFO:teuthology.orchestra.run.smithi123.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-09-16T15:52:08.275 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:08.277+0000 7f0e04e78640 1 -- 172.21.15.123:0/1205772166 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f0de0073ac0 msgr2=0x7f0de0075f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:52:08.275 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:08.277+0000 7f0e04e78640 1 --2- 172.21.15.123:0/1205772166 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f0de0073ac0 0x7f0de0075f80 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f0df40058d0 tx=0x7f0df4005860 comp rx=0 tx=0).stop 2024-09-16T15:52:08.275 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:08.277+0000 7f0e04e78640 1 -- 172.21.15.123:0/1205772166 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0e000cdc20 msgr2=0x7f0e000c5e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:52:08.275 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:08.277+0000 7f0e04e78640 1 --2- 172.21.15.123:0/1205772166 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0e000cdc20 0x7f0e000c5e70 secure :-1 s=READY pgs=1953 cs=0 l=1 rev1=1 crypto rx=0x7f0df0009850 tx=0x7f0df00a55f0 comp rx=0 tx=0).stop 2024-09-16T15:52:08.276 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:08.277+0000 7f0e04e78640 1 -- 172.21.15.123:0/1205772166 shutdown_connections 2024-09-16T15:52:08.276 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:08.277+0000 7f0e04e78640 1 --2- 172.21.15.123:0/1205772166 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f0de0073ac0 0x7f0de0075f80 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:52:08.276 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:08.277+0000 7f0e04e78640 1 --2- 172.21.15.123:0/1205772166 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f0e0015e160 0x7f0e00150e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:52:08.276 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:08.277+0000 7f0e04e78640 1 --2- 172.21.15.123:0/1205772166 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0e000cdc20 0x7f0e000c5e70 unknown :-1 s=CLOSED pgs=1953 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:52:08.276 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:08.277+0000 7f0e04e78640 1 --2- 172.21.15.123:0/1205772166 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f0e00074470 0x7f0e0015d070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:52:08.276 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:08.277+0000 7f0e04e78640 1 -- 172.21.15.123:0/1205772166 >> 172.21.15.123:0/1205772166 conn(0x7f0e000cfb10 msgr2=0x7f0e00158270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:52:08.276 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:08.277+0000 7f0e04e78640 1 -- 172.21.15.123:0/1205772166 shutdown_connections 2024-09-16T15:52:08.276 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:08.277+0000 7f0e04e78640 1 -- 172.21.15.123:0/1205772166 wait complete. 2024-09-16T15:52:08.298 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-09-16T15:52:08.324 INFO:tasks.ceph.mds.a.smithi123.stdout:starting mds.a at 2024-09-16T15:52:08.414 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:08.413+0000 7fe2106d3640 1 -- 172.21.15.123:0/2179201767 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe1f4004810 msgr2=0x7fe1f4004c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:52:08.414 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:08.413+0000 7fe2106d3640 1 --2- 172.21.15.123:0/2179201767 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe1f4004810 0x7fe1f4004c90 secure :-1 s=READY pgs=1958 cs=0 l=1 rev1=1 crypto rx=0x7fe1f800bb10 tx=0x7fe1f801cde0 comp rx=0 tx=0).stop 2024-09-16T15:52:08.414 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:08.413+0000 7fe2106d3640 1 -- 172.21.15.123:0/2179201767 shutdown_connections 2024-09-16T15:52:08.414 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:08.413+0000 7fe2106d3640 1 --2- 172.21.15.123:0/2179201767 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe1f4004810 0x7fe1f4004c90 unknown :-1 s=CLOSED pgs=1958 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:52:08.414 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:08.413+0000 7fe2106d3640 1 -- 172.21.15.123:0/2179201767 >> 172.21.15.123:0/2179201767 conn(0x7fe2080d1bc0 msgr2=0x7fe2080d1fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:52:08.414 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:08.413+0000 7fe2106d3640 1 -- 172.21.15.123:0/2179201767 shutdown_connections 2024-09-16T15:52:08.414 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:08.413+0000 7fe2106d3640 1 -- 172.21.15.123:0/2179201767 wait complete. 2024-09-16T15:52:08.417 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:08.417+0000 7fe2106d3640 1 Processor -- start 2024-09-16T15:52:08.417 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:08.417+0000 7fe2106d3640 1 -- start start 2024-09-16T15:52:08.417 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:08.417+0000 7fe2106d3640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fe1f4004810 0x7fe2081fc500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:52:08.417 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:08.417+0000 7fe2106d3640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe208072020 0x7fe208070850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:52:08.417 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:08.417+0000 7fe2106d3640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe20815e4b0 0x7fe20815c410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:52:08.417 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:08.417+0000 7fe2106d3640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fe208160140 con 0x7fe208072020 2024-09-16T15:52:08.417 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:08.417+0000 7fe2106d3640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fe2080ce8f0 con 0x7fe20815e4b0 2024-09-16T15:52:08.417 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:08.417+0000 7fe2106d3640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fe2081602c0 con 0x7fe1f4004810 2024-09-16T15:52:08.417 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:08.417+0000 7fe20dc6e640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe208072020 0x7fe208070850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:52:08.418 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:08.417+0000 7fe20dc6e640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe208072020 0x7fe208070850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:55596/0 (socket says 172.21.15.123:55596) 2024-09-16T15:52:08.418 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:08.417+0000 7fe20dc6e640 1 -- 172.21.15.123:0/2131986278 learned_addr learned my addr 172.21.15.123:0/2131986278 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:52:08.418 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:08.417+0000 7fe20dc6e640 1 -- 172.21.15.123:0/2131986278 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fe1f4004810 msgr2=0x7fe2081fc500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:52:08.418 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:08.417+0000 7fe20dc6e640 1 --2- 172.21.15.123:0/2131986278 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fe1f4004810 0x7fe2081fc500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:52:08.418 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:08.417+0000 7fe20dc6e640 1 -- 172.21.15.123:0/2131986278 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe20815e4b0 msgr2=0x7fe20815c410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:52:08.418 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:08.417+0000 7fe20dc6e640 1 --2- 172.21.15.123:0/2131986278 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe20815e4b0 0x7fe20815c410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:52:08.418 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:08.417+0000 7fe20dc6e640 1 -- 172.21.15.123:0/2131986278 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe20815ee50 con 0x7fe208072020 2024-09-16T15:52:08.418 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:08.417+0000 7fe20dc6e640 1 --2- 172.21.15.123:0/2131986278 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe208072020 0x7fe208070850 secure :-1 s=READY pgs=1959 cs=0 l=1 rev1=1 crypto rx=0x7fe1f8098910 tx=0x7fe1f8005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:52:08.418 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:08.417+0000 7fe1feffd640 1 -- 172.21.15.123:0/2131986278 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1f80be070 con 0x7fe208072020 2024-09-16T15:52:08.419 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:08.417+0000 7fe1feffd640 1 -- 172.21.15.123:0/2131986278 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fe1f8005130 con 0x7fe208072020 2024-09-16T15:52:08.419 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:08.417+0000 7fe2106d3640 1 -- 172.21.15.123:0/2131986278 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe20815f130 con 0x7fe208072020 2024-09-16T15:52:08.419 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:08.417+0000 7fe1feffd640 1 -- 172.21.15.123:0/2131986278 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1f80ab050 con 0x7fe208072020 2024-09-16T15:52:08.419 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:08.417+0000 7fe2106d3640 1 -- 172.21.15.123:0/2131986278 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe2081fbfe0 con 0x7fe208072020 2024-09-16T15:52:08.420 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:08.417+0000 7fe2106d3640 1 -- 172.21.15.123:0/2131986278 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe2081fbb10 con 0x7fe208072020 2024-09-16T15:52:08.420 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:08.417+0000 7fe1feffd640 1 -- 172.21.15.123:0/2131986278 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 27) ==== 98151+0+0 (secure 0 0 0) 0x7fe1f80c1070 con 0x7fe208072020 2024-09-16T15:52:08.420 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:08.417+0000 7fe1feffd640 1 --2- 172.21.15.123:0/2131986278 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fe1ec073a90 0x7fe1ec075f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:52:08.420 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:08.417+0000 7fe20e46f640 1 --2- 172.21.15.123:0/2131986278 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fe1ec073a90 0x7fe1ec075f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:52:08.420 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:08.421+0000 7fe1feffd640 1 -- 172.21.15.123:0/2131986278 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(170..170 src has 1..170) ==== 5823+0+0 (secure 0 0 0) 0x7fe1f812cce0 con 0x7fe208072020 2024-09-16T15:52:08.420 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:08.421+0000 7fe20e46f640 1 --2- 172.21.15.123:0/2131986278 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fe1ec073a90 0x7fe1ec075f50 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7fe208052940 tx=0x7fe204001580 comp rx=0 tx=0).ready entity=mgr.4150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:52:08.422 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:08.425+0000 7fe1feffd640 1 -- 172.21.15.123:0/2131986278 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7fe1f8016af0 con 0x7fe208072020 2024-09-16T15:52:08.621 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:08.621+0000 7fe2106d3640 1 -- 172.21.15.123:0/2131986278 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) -- 0x7fe2080cfce0 con 0x7fe208072020 2024-09-16T15:52:09.275 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:09.277+0000 7fe1feffd640 1 -- 172.21.15.123:0/2131986278 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v118) ==== 104+0+0 (secure 0 0 0) 0x7fe1f80f9640 con 0x7fe208072020 2024-09-16T15:52:09.278 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:09.277+0000 7fe2106d3640 1 -- 172.21.15.123:0/2131986278 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fe1ec073a90 msgr2=0x7fe1ec075f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:52:09.278 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:09.277+0000 7fe2106d3640 1 --2- 172.21.15.123:0/2131986278 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fe1ec073a90 0x7fe1ec075f50 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7fe208052940 tx=0x7fe204001580 comp rx=0 tx=0).stop 2024-09-16T15:52:09.278 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:09.277+0000 7fe2106d3640 1 -- 172.21.15.123:0/2131986278 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe208072020 msgr2=0x7fe208070850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:52:09.279 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:09.277+0000 7fe2106d3640 1 --2- 172.21.15.123:0/2131986278 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe208072020 0x7fe208070850 secure :-1 s=READY pgs=1959 cs=0 l=1 rev1=1 crypto rx=0x7fe1f8098910 tx=0x7fe1f8005880 comp rx=0 tx=0).stop 2024-09-16T15:52:09.279 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:09.277+0000 7fe2106d3640 1 -- 172.21.15.123:0/2131986278 shutdown_connections 2024-09-16T15:52:09.279 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:09.277+0000 7fe2106d3640 1 --2- 172.21.15.123:0/2131986278 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fe1ec073a90 0x7fe1ec075f50 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:52:09.279 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:09.277+0000 7fe2106d3640 1 --2- 172.21.15.123:0/2131986278 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe20815e4b0 0x7fe20815c410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:52:09.279 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:09.277+0000 7fe2106d3640 1 --2- 172.21.15.123:0/2131986278 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe208072020 0x7fe208070850 unknown :-1 s=CLOSED pgs=1959 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:52:09.279 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:09.277+0000 7fe2106d3640 1 --2- 172.21.15.123:0/2131986278 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fe1f4004810 0x7fe2081fc500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:52:09.279 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:09.277+0000 7fe2106d3640 1 -- 172.21.15.123:0/2131986278 >> 172.21.15.123:0/2131986278 conn(0x7fe2080d1bc0 msgr2=0x7fe2080b1a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:52:09.279 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:09.277+0000 7fe2106d3640 1 -- 172.21.15.123:0/2131986278 shutdown_connections 2024-09-16T15:52:09.279 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:09.277+0000 7fe2106d3640 1 -- 172.21.15.123:0/2131986278 wait complete. 2024-09-16T15:52:09.299 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-09-16T15:52:09.476 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:09.477+0000 7f773c4bd640 1 -- 172.21.15.123:0/1742371981 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7720004d50 msgr2=0x7f77200051d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:52:09.476 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:09.477+0000 7f773c4bd640 1 --2- 172.21.15.123:0/1742371981 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7720004d50 0x7f77200051d0 secure :-1 s=READY pgs=1961 cs=0 l=1 rev1=1 crypto rx=0x7f772c00c640 tx=0x7f772c01d300 comp rx=0 tx=0).stop 2024-09-16T15:52:09.477 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:09.477+0000 7f773aa5a640 1 -- 172.21.15.123:0/1742371981 reap_dead start 2024-09-16T15:52:09.477 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:09.477+0000 7f773c4bd640 1 -- 172.21.15.123:0/1742371981 shutdown_connections 2024-09-16T15:52:09.477 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:09.477+0000 7f773c4bd640 1 -- 172.21.15.123:0/1742371981 >> 172.21.15.123:0/1742371981 conn(0x7f77340d3e30 msgr2=0x7f77340d4240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:52:09.477 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:09.477+0000 7f773c4bd640 1 -- 172.21.15.123:0/1742371981 shutdown_connections 2024-09-16T15:52:09.477 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:09.477+0000 7f773c4bd640 1 -- 172.21.15.123:0/1742371981 wait complete. 2024-09-16T15:52:09.480 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:09.477+0000 7f773c4bd640 1 Processor -- start 2024-09-16T15:52:09.480 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:09.477+0000 7f773c4bd640 1 -- start start 2024-09-16T15:52:09.480 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:09.477+0000 7f773c4bd640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7720004d50 0x7f77340caf00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:52:09.480 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:09.477+0000 7f773c4bd640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f77340a19d0 0x7f77340c8ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:52:09.480 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:09.477+0000 7f773c4bd640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f773415d3c0 0x7f77340b1f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:52:09.480 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:09.477+0000 7f773c4bd640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f77340d2c20 con 0x7f77340a19d0 2024-09-16T15:52:09.480 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:09.477+0000 7f773c4bd640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f7734161910 con 0x7f7720004d50 2024-09-16T15:52:09.481 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:09.477+0000 7f773c4bd640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f77340d2da0 con 0x7f773415d3c0 2024-09-16T15:52:09.481 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:09.477+0000 7f7739a58640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f77340a19d0 0x7f77340c8ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:52:09.481 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:09.481+0000 7f7739a58640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f77340a19d0 0x7f77340c8ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:55614/0 (socket says 172.21.15.123:55614) 2024-09-16T15:52:09.481 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:09.481+0000 7f7739a58640 1 -- 172.21.15.123:0/2789563 learned_addr learned my addr 172.21.15.123:0/2789563 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:52:09.481 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:09.481+0000 7f773aa5a640 1 --2- 172.21.15.123:0/2789563 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f773415d3c0 0x7f77340b1f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:52:09.481 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:09.481+0000 7f773a259640 1 --2- 172.21.15.123:0/2789563 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7720004d50 0x7f77340caf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:52:09.481 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:09.481+0000 7f7739a58640 1 -- 172.21.15.123:0/2789563 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f773415d3c0 msgr2=0x7f77340b1f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:52:09.481 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:09.481+0000 7f7739a58640 1 --2- 172.21.15.123:0/2789563 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f773415d3c0 0x7f77340b1f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:52:09.481 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:09.481+0000 7f7739a58640 1 -- 172.21.15.123:0/2789563 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7720004d50 msgr2=0x7f77340caf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:52:09.481 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:09.481+0000 7f7739a58640 1 --2- 172.21.15.123:0/2789563 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7720004d50 0x7f77340caf00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:52:09.482 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:09.481+0000 7f7739a58640 1 -- 172.21.15.123:0/2789563 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f77340cb4e0 con 0x7f77340a19d0 2024-09-16T15:52:09.482 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:09.481+0000 7f773a259640 1 --2- 172.21.15.123:0/2789563 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7720004d50 0x7f77340caf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T15:52:09.482 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:09.481+0000 7f773aa5a640 1 --2- 172.21.15.123:0/2789563 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f773415d3c0 0x7f77340b1f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T15:52:09.482 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:09.481+0000 7f7739a58640 1 --2- 172.21.15.123:0/2789563 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f77340a19d0 0x7f77340c8ca0 secure :-1 s=READY pgs=1962 cs=0 l=1 rev1=1 crypto rx=0x7f772400e9f0 tx=0x7f772400eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:52:09.482 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:09.481+0000 7f772affd640 1 -- 172.21.15.123:0/2789563 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f772400cc60 con 0x7f77340a19d0 2024-09-16T15:52:09.482 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:09.481+0000 7f772affd640 1 -- 172.21.15.123:0/2789563 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f7724011470 con 0x7f77340a19d0 2024-09-16T15:52:09.482 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:09.481+0000 7f773c4bd640 1 -- 172.21.15.123:0/2789563 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f77340cb920 con 0x7f77340a19d0 2024-09-16T15:52:09.482 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:09.481+0000 7f772affd640 1 -- 172.21.15.123:0/2789563 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77240118c0 con 0x7f77340a19d0 2024-09-16T15:52:09.482 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:09.481+0000 7f773c4bd640 1 -- 172.21.15.123:0/2789563 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f77340cbee0 con 0x7f77340a19d0 2024-09-16T15:52:09.482 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:09.481+0000 7f773c4bd640 1 -- 172.21.15.123:0/2789563 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f77340d0fb0 con 0x7f77340a19d0 2024-09-16T15:52:09.483 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:09.481+0000 7f772affd640 1 -- 172.21.15.123:0/2789563 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 27) ==== 98151+0+0 (secure 0 0 0) 0x7f7724011a60 con 0x7f77340a19d0 2024-09-16T15:52:09.483 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:09.481+0000 7f772affd640 1 --2- 172.21.15.123:0/2789563 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f7718073a90 0x7f7718075f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:52:09.484 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:09.481+0000 7f773a259640 1 --2- 172.21.15.123:0/2789563 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f7718073a90 0x7f7718075f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:52:09.484 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:09.481+0000 7f772affd640 1 -- 172.21.15.123:0/2789563 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(170..170 src has 1..170) ==== 5823+0+0 (secure 0 0 0) 0x7f77240933b0 con 0x7f77340a19d0 2024-09-16T15:52:09.484 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:09.481+0000 7f773a259640 1 --2- 172.21.15.123:0/2789563 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f7718073a90 0x7f7718075f50 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f7730009b40 tx=0x7f77300091e0 comp rx=0 tx=0).ready entity=mgr.4150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:52:09.486 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:09.485+0000 7f772affd640 1 -- 172.21.15.123:0/2789563 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f7724016030 con 0x7f77340a19d0 2024-09-16T15:52:09.647 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:09.649+0000 7f773c4bd640 1 -- 172.21.15.123:0/2789563 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) -- 0x7f773415faf0 con 0x7f77340a19d0 2024-09-16T15:52:10.492 DEBUG:teuthology.orchestra.run.smithi123:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-16T15:52:10.494 DEBUG:teuthology.orchestra.run.smithi184:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-16T15:52:10.549 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:10.549+0000 7f772affd640 1 -- 172.21.15.123:0/2789563 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v171) ==== 181+0+0 (secure 0 0 0) 0x7f77240600b0 con 0x7f77340a19d0 2024-09-16T15:52:10.549 INFO:teuthology.orchestra.run.smithi123.stderr:pool 'cephfs_metadata' removed 2024-09-16T15:52:10.555 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:10.557+0000 7f773c4bd640 1 -- 172.21.15.123:0/2789563 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f7718073a90 msgr2=0x7f7718075f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:52:10.555 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:10.557+0000 7f773c4bd640 1 --2- 172.21.15.123:0/2789563 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f7718073a90 0x7f7718075f50 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f7730009b40 tx=0x7f77300091e0 comp rx=0 tx=0).stop 2024-09-16T15:52:10.555 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:10.557+0000 7f773c4bd640 1 -- 172.21.15.123:0/2789563 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f77340a19d0 msgr2=0x7f77340c8ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:52:10.555 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:10.557+0000 7f773c4bd640 1 --2- 172.21.15.123:0/2789563 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f77340a19d0 0x7f77340c8ca0 secure :-1 s=READY pgs=1962 cs=0 l=1 rev1=1 crypto rx=0x7f772400e9f0 tx=0x7f772400eeb0 comp rx=0 tx=0).stop 2024-09-16T15:52:10.556 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:10.557+0000 7f773c4bd640 1 -- 172.21.15.123:0/2789563 shutdown_connections 2024-09-16T15:52:10.556 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:10.557+0000 7f773c4bd640 1 --2- 172.21.15.123:0/2789563 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f7718073a90 0x7f7718075f50 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:52:10.556 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:10.557+0000 7f773c4bd640 1 --2- 172.21.15.123:0/2789563 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f773415d3c0 0x7f77340b1f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:52:10.556 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:10.557+0000 7f773c4bd640 1 --2- 172.21.15.123:0/2789563 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f77340a19d0 0x7f77340c8ca0 unknown :-1 s=CLOSED pgs=1962 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:52:10.556 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:10.557+0000 7f773c4bd640 1 --2- 172.21.15.123:0/2789563 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7720004d50 0x7f77340caf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:52:10.556 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:10.557+0000 7f773c4bd640 1 -- 172.21.15.123:0/2789563 >> 172.21.15.123:0/2789563 conn(0x7f77340d3e30 msgr2=0x7f77340b6040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:52:10.556 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:10.557+0000 7f773c4bd640 1 -- 172.21.15.123:0/2789563 shutdown_connections 2024-09-16T15:52:10.556 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:10.557+0000 7f773c4bd640 1 -- 172.21.15.123:0/2789563 wait complete. 2024-09-16T15:52:10.576 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-09-16T15:52:10.757 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:10.757+0000 7f3afce21640 1 -- 172.21.15.123:0/3746893943 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3af815e4b0 msgr2=0x7f3af806ee90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:52:10.758 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:10.757+0000 7f3afce21640 1 --2- 172.21.15.123:0/3746893943 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3af815e4b0 0x7f3af806ee90 secure :-1 s=READY pgs=1964 cs=0 l=1 rev1=1 crypto rx=0x7f3af0009c80 tx=0x7f3af001f5a0 comp rx=0 tx=0).stop 2024-09-16T15:52:10.758 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:10.757+0000 7f3afce21640 1 -- 172.21.15.123:0/3746893943 shutdown_connections 2024-09-16T15:52:10.758 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:10.757+0000 7f3afce21640 1 --2- 172.21.15.123:0/3746893943 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3af815e4b0 0x7f3af806ee90 unknown :-1 s=CLOSED pgs=1964 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:52:10.758 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:10.757+0000 7f3afce21640 1 --2- 172.21.15.123:0/3746893943 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f3af815d7c0 0x7f3af80b31e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:52:10.758 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:10.757+0000 7f3afce21640 1 -- 172.21.15.123:0/3746893943 >> 172.21.15.123:0/3746893943 conn(0x7f3af80b1630 msgr2=0x7f3af80b1a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:52:10.758 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:10.757+0000 7f3afce21640 1 -- 172.21.15.123:0/3746893943 shutdown_connections 2024-09-16T15:52:10.758 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:10.757+0000 7f3afce21640 1 -- 172.21.15.123:0/3746893943 wait complete. 2024-09-16T15:52:10.759 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:10.761+0000 7f3afce21640 1 Processor -- start 2024-09-16T15:52:10.759 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:10.761+0000 7f3afce21640 1 -- start start 2024-09-16T15:52:10.761 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:10.761+0000 7f3afce21640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3af80b3720 0x7f3af8142350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:52:10.761 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:10.761+0000 7f3afce21640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f3af815d7c0 0x7f3af8154450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:52:10.761 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:10.761+0000 7f3afce21640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3af815e4b0 0x7f3af8154990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:52:10.761 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:10.761+0000 7f3afce21640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f3af806f900 con 0x7f3af815e4b0 2024-09-16T15:52:10.761 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:10.761+0000 7f3afce21640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f3af806f780 con 0x7f3af80b3720 2024-09-16T15:52:10.761 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:10.761+0000 7f3afce21640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f3af806fa80 con 0x7f3af815d7c0 2024-09-16T15:52:10.761 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:10.761+0000 7f3af6d9d640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3af815e4b0 0x7f3af8154990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:52:10.761 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:10.761+0000 7f3af6d9d640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3af815e4b0 0x7f3af8154990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:55642/0 (socket says 172.21.15.123:55642) 2024-09-16T15:52:10.761 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:10.761+0000 7f3af6d9d640 1 -- 172.21.15.123:0/2489210985 learned_addr learned my addr 172.21.15.123:0/2489210985 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:52:10.761 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:10.761+0000 7f3af659c640 1 --2- 172.21.15.123:0/2489210985 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3af80b3720 0x7f3af8142350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:52:10.762 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:10.761+0000 7f3aeffff640 1 --2- 172.21.15.123:0/2489210985 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f3af815d7c0 0x7f3af8154450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:52:10.762 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:10.761+0000 7f3af6d9d640 1 -- 172.21.15.123:0/2489210985 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f3af815d7c0 msgr2=0x7f3af8154450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:52:10.762 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:10.761+0000 7f3af6d9d640 1 --2- 172.21.15.123:0/2489210985 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f3af815d7c0 0x7f3af8154450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:52:10.762 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:10.761+0000 7f3af6d9d640 1 -- 172.21.15.123:0/2489210985 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3af80b3720 msgr2=0x7f3af8142350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:52:10.762 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:10.761+0000 7f3af6d9d640 1 --2- 172.21.15.123:0/2489210985 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3af80b3720 0x7f3af8142350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:52:10.762 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:10.761+0000 7f3af6d9d640 1 -- 172.21.15.123:0/2489210985 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3af81f5f80 con 0x7f3af815e4b0 2024-09-16T15:52:10.762 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:10.761+0000 7f3af6d9d640 1 --2- 172.21.15.123:0/2489210985 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3af815e4b0 0x7f3af8154990 secure :-1 s=READY pgs=1965 cs=0 l=1 rev1=1 crypto rx=0x7f3af001fe40 tx=0x7f3af00af410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:52:10.762 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:10.761+0000 7f3aeeffd640 1 -- 172.21.15.123:0/2489210985 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3af0004e60 con 0x7f3af815e4b0 2024-09-16T15:52:10.762 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:10.761+0000 7f3aeeffd640 1 -- 172.21.15.123:0/2489210985 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f3af00be030 con 0x7f3af815e4b0 2024-09-16T15:52:10.762 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:10.761+0000 7f3afce21640 1 -- 172.21.15.123:0/2489210985 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3af81f6260 con 0x7f3af815e4b0 2024-09-16T15:52:10.762 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:10.761+0000 7f3aeeffd640 1 -- 172.21.15.123:0/2489210985 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3af00afb00 con 0x7f3af815e4b0 2024-09-16T15:52:10.762 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:10.761+0000 7f3afce21640 1 -- 172.21.15.123:0/2489210985 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3af81f6820 con 0x7f3af815e4b0 2024-09-16T15:52:10.763 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:10.761+0000 7f3afce21640 1 -- 172.21.15.123:0/2489210985 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3af806ff50 con 0x7f3af815e4b0 2024-09-16T15:52:10.763 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:10.765+0000 7f3aeeffd640 1 -- 172.21.15.123:0/2489210985 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 27) ==== 98151+0+0 (secure 0 0 0) 0x7f3af00afca0 con 0x7f3af815e4b0 2024-09-16T15:52:10.763 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:10.765+0000 7f3aeeffd640 1 --2- 172.21.15.123:0/2489210985 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f3ad8073a90 0x7f3ad8075f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:52:10.764 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:10.765+0000 7f3af659c640 1 --2- 172.21.15.123:0/2489210985 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f3ad8073a90 0x7f3ad8075f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:52:10.764 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:10.765+0000 7f3aeeffd640 1 -- 172.21.15.123:0/2489210985 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(171..171 src has 1..171) ==== 5385+0+0 (secure 0 0 0) 0x7f3af0131a10 con 0x7f3af815e4b0 2024-09-16T15:52:10.764 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:10.765+0000 7f3af659c640 1 --2- 172.21.15.123:0/2489210985 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f3ad8073a90 0x7f3ad8075f50 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f3af8155530 tx=0x7f3ae8008040 comp rx=0 tx=0).ready entity=mgr.4150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:52:10.767 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:10.769+0000 7f3aeeffd640 1 -- 172.21.15.123:0/2489210985 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f3af00ac050 con 0x7f3af815e4b0 2024-09-16T15:52:10.935 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:10.937+0000 7f3afce21640 1 -- 172.21.15.123:0/2489210985 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) -- 0x7f3af8070510 con 0x7f3af815e4b0 2024-09-16T15:52:11.562 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:11.561+0000 7f3aeeffd640 1 -- 172.21.15.123:0/2489210985 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v172) ==== 169+0+0 (secure 0 0 0) 0x7f3af0019100 con 0x7f3af815e4b0 2024-09-16T15:52:11.562 INFO:teuthology.orchestra.run.smithi123.stderr:pool 'cephfs_data' removed 2024-09-16T15:52:11.565 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:11.565+0000 7f3afce21640 1 -- 172.21.15.123:0/2489210985 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f3ad8073a90 msgr2=0x7f3ad8075f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:52:11.565 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:11.565+0000 7f3afce21640 1 --2- 172.21.15.123:0/2489210985 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f3ad8073a90 0x7f3ad8075f50 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f3af8155530 tx=0x7f3ae8008040 comp rx=0 tx=0).stop 2024-09-16T15:52:11.566 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:11.565+0000 7f3afce21640 1 -- 172.21.15.123:0/2489210985 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3af815e4b0 msgr2=0x7f3af8154990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:52:11.566 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:11.565+0000 7f3afce21640 1 --2- 172.21.15.123:0/2489210985 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3af815e4b0 0x7f3af8154990 secure :-1 s=READY pgs=1965 cs=0 l=1 rev1=1 crypto rx=0x7f3af001fe40 tx=0x7f3af00af410 comp rx=0 tx=0).stop 2024-09-16T15:52:11.566 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:11.565+0000 7f3afce21640 1 -- 172.21.15.123:0/2489210985 shutdown_connections 2024-09-16T15:52:11.566 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:11.565+0000 7f3afce21640 1 --2- 172.21.15.123:0/2489210985 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f3ad8073a90 0x7f3ad8075f50 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:52:11.566 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:11.565+0000 7f3afce21640 1 --2- 172.21.15.123:0/2489210985 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3af815e4b0 0x7f3af8154990 unknown :-1 s=CLOSED pgs=1965 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:52:11.566 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:11.565+0000 7f3afce21640 1 --2- 172.21.15.123:0/2489210985 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f3af815d7c0 0x7f3af8154450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:52:11.566 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:11.565+0000 7f3afce21640 1 --2- 172.21.15.123:0/2489210985 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3af80b3720 0x7f3af8142350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:52:11.566 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:11.565+0000 7f3afce21640 1 -- 172.21.15.123:0/2489210985 >> 172.21.15.123:0/2489210985 conn(0x7f3af80b1630 msgr2=0x7f3af80af5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:52:11.568 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:11.569+0000 7f3afce21640 1 -- 172.21.15.123:0/2489210985 shutdown_connections 2024-09-16T15:52:11.568 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:11.569+0000 7f3afce21640 1 -- 172.21.15.123:0/2489210985 wait complete. 2024-09-16T15:52:11.591 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config reset 27 2024-09-16T15:52:11.717 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:11.717+0000 7f626eb68640 1 -- 172.21.15.123:0/4120013373 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f626815dd60 msgr2=0x7f62681621d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:52:11.717 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:11.717+0000 7f626eb68640 1 --2- 172.21.15.123:0/4120013373 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f626815dd60 0x7f62681621d0 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7f6258009840 tx=0x7f625801c790 comp rx=0 tx=0).stop 2024-09-16T15:52:11.717 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:11.717+0000 7f626eb68640 1 -- 172.21.15.123:0/4120013373 shutdown_connections 2024-09-16T15:52:11.717 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:11.717+0000 7f626eb68640 1 --2- 172.21.15.123:0/4120013373 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f62680d4200 0x7f62680d1b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:52:11.717 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:11.717+0000 7f626eb68640 1 --2- 172.21.15.123:0/4120013373 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f626815dd60 0x7f62681621d0 unknown :-1 s=CLOSED pgs=580 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:52:11.718 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:11.717+0000 7f626eb68640 1 -- 172.21.15.123:0/4120013373 >> 172.21.15.123:0/4120013373 conn(0x7f6268069650 msgr2=0x7f626809d170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:52:11.718 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:11.717+0000 7f626eb68640 1 -- 172.21.15.123:0/4120013373 shutdown_connections 2024-09-16T15:52:11.718 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:11.717+0000 7f626eb68640 1 -- 172.21.15.123:0/4120013373 wait complete. 2024-09-16T15:52:11.718 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:11.721+0000 7f626eb68640 1 Processor -- start 2024-09-16T15:52:11.718 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:11.721+0000 7f626eb68640 1 -- start start 2024-09-16T15:52:11.720 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:11.721+0000 7f626eb68640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f62680b3b20 0x7f6268155a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:52:11.720 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:11.721+0000 7f626eb68640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f62680d4200 0x7f62680c84a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:52:11.720 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:11.721+0000 7f626eb68640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f626815dd60 0x7f62680ca380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:52:11.720 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:11.721+0000 7f626eb68640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f626816ae30 con 0x7f62680d4200 2024-09-16T15:52:11.720 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:11.721+0000 7f626eb68640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f62680d2e50 con 0x7f62680b3b20 2024-09-16T15:52:11.720 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:11.721+0000 7f626eb68640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f626816afb0 con 0x7f626815dd60 2024-09-16T15:52:11.720 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:11.721+0000 7f6267fff640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f62680d4200 0x7f62680c84a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:52:11.720 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:11.721+0000 7f6267fff640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f62680d4200 0x7f62680c84a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:55652/0 (socket says 172.21.15.123:55652) 2024-09-16T15:52:11.721 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:11.721+0000 7f6267fff640 1 -- 172.21.15.123:0/281512176 learned_addr learned my addr 172.21.15.123:0/281512176 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:52:11.721 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:11.721+0000 7f626c904640 1 --2- 172.21.15.123:0/281512176 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f62680b3b20 0x7f6268155a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:52:11.721 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:11.721+0000 7f626d105640 1 --2- 172.21.15.123:0/281512176 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f626815dd60 0x7f62680ca380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:52:11.721 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:11.721+0000 7f6267fff640 1 -- 172.21.15.123:0/281512176 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f626815dd60 msgr2=0x7f62680ca380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:52:11.721 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:11.721+0000 7f6267fff640 1 --2- 172.21.15.123:0/281512176 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f626815dd60 0x7f62680ca380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:52:11.721 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:11.721+0000 7f6267fff640 1 -- 172.21.15.123:0/281512176 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f62680b3b20 msgr2=0x7f6268155a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:52:11.721 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:11.721+0000 7f6267fff640 1 --2- 172.21.15.123:0/281512176 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f62680b3b20 0x7f6268155a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:52:11.721 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:11.721+0000 7f6267fff640 1 -- 172.21.15.123:0/281512176 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6268155f40 con 0x7f62680d4200 2024-09-16T15:52:11.721 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:11.721+0000 7f626c904640 1 --2- 172.21.15.123:0/281512176 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f62680b3b20 0x7f6268155a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T15:52:11.721 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:11.721+0000 7f6267fff640 1 --2- 172.21.15.123:0/281512176 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f62680d4200 0x7f62680c84a0 secure :-1 s=READY pgs=1966 cs=0 l=1 rev1=1 crypto rx=0x7f625c00c480 tx=0x7f625c00c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:52:11.721 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:11.721+0000 7f62657fa640 1 -- 172.21.15.123:0/281512176 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f625c0052d0 con 0x7f62680d4200 2024-09-16T15:52:11.721 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:11.721+0000 7f62657fa640 1 -- 172.21.15.123:0/281512176 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f625c005470 con 0x7f62680d4200 2024-09-16T15:52:11.721 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:11.721+0000 7f626eb68640 1 -- 172.21.15.123:0/281512176 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f62681fba50 con 0x7f62680d4200 2024-09-16T15:52:11.721 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:11.721+0000 7f62657fa640 1 -- 172.21.15.123:0/281512176 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f625c012870 con 0x7f62680d4200 2024-09-16T15:52:11.722 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:11.721+0000 7f626eb68640 1 -- 172.21.15.123:0/281512176 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f62681fc010 con 0x7f62680d4200 2024-09-16T15:52:11.722 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:11.721+0000 7f626eb68640 1 -- 172.21.15.123:0/281512176 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f62680d1b60 con 0x7f62680d4200 2024-09-16T15:52:11.724 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:11.721+0000 7f62657fa640 1 -- 172.21.15.123:0/281512176 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 27) ==== 98151+0+0 (secure 0 0 0) 0x7f625c003600 con 0x7f62680d4200 2024-09-16T15:52:11.724 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:11.725+0000 7f62657fa640 1 --2- 172.21.15.123:0/281512176 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f6254073a90 0x7f6254075f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:52:11.725 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:11.725+0000 7f62657fa640 1 -- 172.21.15.123:0/281512176 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(172..172 src has 1..172) ==== 4971+0+0 (secure 0 0 0) 0x7f625c092620 con 0x7f62680d4200 2024-09-16T15:52:11.725 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:11.725+0000 7f626c904640 1 --2- 172.21.15.123:0/281512176 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f6254073a90 0x7f6254075f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:52:11.725 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:11.725+0000 7f626c904640 1 --2- 172.21.15.123:0/281512176 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f6254073a90 0x7f6254075f50 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f62580af000 tx=0x7f625801afd0 comp rx=0 tx=0).ready entity=mgr.4150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:52:11.730 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:11.733+0000 7f62657fa640 1 -- 172.21.15.123:0/281512176 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f6268155f40 con 0x7f62680d4200 2024-09-16T15:52:11.908 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:11.909+0000 7f626eb68640 1 -- 172.21.15.123:0/281512176 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "config reset", "num": 27} v 0) -- 0x7f62680ce2e0 con 0x7f62680d4200 2024-09-16T15:52:11.915 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:11.917+0000 7f62657fa640 1 -- 172.21.15.123:0/281512176 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "config reset", "num": 27}]=0 v49) ==== 71+0+0 (secure 0 0 0) 0x7f62680ce950 con 0x7f62680d4200 2024-09-16T15:52:11.918 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:11.917+0000 7f626eb68640 1 -- 172.21.15.123:0/281512176 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f6254073a90 msgr2=0x7f6254075f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:52:11.918 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:11.917+0000 7f626eb68640 1 --2- 172.21.15.123:0/281512176 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f6254073a90 0x7f6254075f50 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f62580af000 tx=0x7f625801afd0 comp rx=0 tx=0).stop 2024-09-16T15:52:11.918 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:11.917+0000 7f626eb68640 1 -- 172.21.15.123:0/281512176 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f62680d4200 msgr2=0x7f62680c84a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:52:11.918 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:11.917+0000 7f626eb68640 1 --2- 172.21.15.123:0/281512176 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f62680d4200 0x7f62680c84a0 secure :-1 s=READY pgs=1966 cs=0 l=1 rev1=1 crypto rx=0x7f625c00c480 tx=0x7f625c00c940 comp rx=0 tx=0).stop 2024-09-16T15:52:11.918 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:11.917+0000 7f626eb68640 1 -- 172.21.15.123:0/281512176 shutdown_connections 2024-09-16T15:52:11.918 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:11.917+0000 7f626eb68640 1 --2- 172.21.15.123:0/281512176 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f6254073a90 0x7f6254075f50 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:52:11.918 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:11.917+0000 7f626eb68640 1 --2- 172.21.15.123:0/281512176 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f626815dd60 0x7f62680ca380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:52:11.918 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:11.917+0000 7f626eb68640 1 --2- 172.21.15.123:0/281512176 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f62680d4200 0x7f62680c84a0 unknown :-1 s=CLOSED pgs=1966 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:52:11.918 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:11.917+0000 7f626eb68640 1 --2- 172.21.15.123:0/281512176 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f62680b3b20 0x7f6268155a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:52:11.918 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:11.917+0000 7f626eb68640 1 -- 172.21.15.123:0/281512176 >> 172.21.15.123:0/281512176 conn(0x7f6268069650 msgr2=0x7f62680b27e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:52:11.919 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:11.917+0000 7f626eb68640 1 -- 172.21.15.123:0/281512176 shutdown_connections 2024-09-16T15:52:11.919 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:11.917+0000 7f626eb68640 1 -- 172.21.15.123:0/281512176 wait complete. 2024-09-16T15:52:11.937 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_volumes.TestMisc.test_unparseable_metafile_on_legacy_to_v1_upgrade' 2024-09-16T15:52:12.063 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:12.065+0000 7fd9450fb640 1 -- 172.21.15.123:0/2028775231 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd9400726c0 msgr2=0x7fd940073ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:52:12.063 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:12.065+0000 7fd9450fb640 1 --2- 172.21.15.123:0/2028775231 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd9400726c0 0x7fd940073ef0 secure :-1 s=READY pgs=1967 cs=0 l=1 rev1=1 crypto rx=0x7fd9400cd650 tx=0x7fd92801c540 comp rx=0 tx=0).stop 2024-09-16T15:52:12.063 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:12.065+0000 7fd9450fb640 1 -- 172.21.15.123:0/2028775231 shutdown_connections 2024-09-16T15:52:12.063 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:12.065+0000 7fd9450fb640 1 --2- 172.21.15.123:0/2028775231 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fd940074430 0x7fd9400ca160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:52:12.063 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:12.065+0000 7fd9450fb640 1 --2- 172.21.15.123:0/2028775231 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd9400726c0 0x7fd940073ef0 unknown :-1 s=CLOSED pgs=1967 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:52:12.064 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:12.065+0000 7fd9450fb640 1 --2- 172.21.15.123:0/2028775231 >> v2:172.21.15.184:3300/0 conn(0x7fd94015bca0 0x7fd9400d0630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:52:12.064 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:12.065+0000 7fd9450fb640 1 -- 172.21.15.123:0/2028775231 >> 172.21.15.123:0/2028775231 conn(0x7fd9400cfb10 msgr2=0x7fd9400cff20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:52:12.064 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:12.065+0000 7fd9450fb640 1 -- 172.21.15.123:0/2028775231 shutdown_connections 2024-09-16T15:52:12.064 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:12.065+0000 7fd9450fb640 1 -- 172.21.15.123:0/2028775231 wait complete. 2024-09-16T15:52:12.064 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:12.065+0000 7fd9450fb640 1 Processor -- start 2024-09-16T15:52:12.064 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:12.065+0000 7fd9450fb640 1 -- start start 2024-09-16T15:52:12.066 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:12.065+0000 7fd9450fb640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd9400726c0 0x7fd9401558a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:52:12.066 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:12.065+0000 7fd9450fb640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fd940074430 0x7fd940151600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:52:12.066 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:12.065+0000 7fd9450fb640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd94015bca0 0x7fd940152c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:52:12.066 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:12.065+0000 7fd9450fb640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fd94006ecb0 con 0x7fd9400726c0 2024-09-16T15:52:12.066 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:12.065+0000 7fd9450fb640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fd94006eb30 con 0x7fd94015bca0 2024-09-16T15:52:12.066 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:12.065+0000 7fd9450fb640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fd94006ee30 con 0x7fd940074430 2024-09-16T15:52:12.066 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:12.065+0000 7fd93ed9d640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd9400726c0 0x7fd9401558a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:52:12.066 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:12.065+0000 7fd93ed9d640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd9400726c0 0x7fd9401558a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:55670/0 (socket says 172.21.15.123:55670) 2024-09-16T15:52:12.066 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:12.065+0000 7fd93ed9d640 1 -- 172.21.15.123:0/1604123323 learned_addr learned my addr 172.21.15.123:0/1604123323 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:52:12.066 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:12.065+0000 7fd937fff640 1 --2- 172.21.15.123:0/1604123323 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fd940074430 0x7fd940151600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:52:12.066 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:12.065+0000 7fd93f59e640 1 --2- 172.21.15.123:0/1604123323 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd94015bca0 0x7fd940152c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:52:12.067 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:12.065+0000 7fd93ed9d640 1 -- 172.21.15.123:0/1604123323 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fd940074430 msgr2=0x7fd940151600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:52:12.067 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:12.065+0000 7fd93ed9d640 1 --2- 172.21.15.123:0/1604123323 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fd940074430 0x7fd940151600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:52:12.067 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:12.065+0000 7fd93ed9d640 1 -- 172.21.15.123:0/1604123323 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd94015bca0 msgr2=0x7fd940152c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:52:12.067 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:12.065+0000 7fd93ed9d640 1 --2- 172.21.15.123:0/1604123323 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd94015bca0 0x7fd940152c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:52:12.067 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:12.065+0000 7fd93ed9d640 1 -- 172.21.15.123:0/1604123323 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd940152930 con 0x7fd9400726c0 2024-09-16T15:52:12.067 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:12.065+0000 7fd93ed9d640 1 --2- 172.21.15.123:0/1604123323 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd9400726c0 0x7fd9401558a0 secure :-1 s=READY pgs=1968 cs=0 l=1 rev1=1 crypto rx=0x7fd93000c940 tx=0x7fd93000ce00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:52:12.067 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:12.065+0000 7fd9377fe640 1 -- 172.21.15.123:0/1604123323 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd930004220 con 0x7fd9400726c0 2024-09-16T15:52:12.067 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:12.065+0000 7fd9377fe640 1 -- 172.21.15.123:0/1604123323 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fd9300079b0 con 0x7fd9400726c0 2024-09-16T15:52:12.067 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:12.065+0000 7fd9450fb640 1 -- 172.21.15.123:0/1604123323 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd940154800 con 0x7fd9400726c0 2024-09-16T15:52:12.068 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:12.065+0000 7fd9377fe640 1 -- 172.21.15.123:0/1604123323 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9300044b0 con 0x7fd9400726c0 2024-09-16T15:52:12.068 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:12.065+0000 7fd9450fb640 1 -- 172.21.15.123:0/1604123323 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd940154dc0 con 0x7fd9400726c0 2024-09-16T15:52:12.068 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:12.069+0000 7fd9450fb640 1 -- 172.21.15.123:0/1604123323 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd9400ca160 con 0x7fd9400726c0 2024-09-16T15:52:12.068 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:12.069+0000 7fd9377fe640 1 -- 172.21.15.123:0/1604123323 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 27) ==== 98151+0+0 (secure 0 0 0) 0x7fd930005020 con 0x7fd9400726c0 2024-09-16T15:52:12.068 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:12.069+0000 7fd9377fe640 1 --2- 172.21.15.123:0/1604123323 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fd920073a90 0x7fd920075f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:52:12.068 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:12.069+0000 7fd9377fe640 1 -- 172.21.15.123:0/1604123323 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(172..172 src has 1..172) ==== 4971+0+0 (secure 0 0 0) 0x7fd930092d20 con 0x7fd9400726c0 2024-09-16T15:52:12.069 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:12.069+0000 7fd937fff640 1 --2- 172.21.15.123:0/1604123323 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fd920073a90 0x7fd920075f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:52:12.069 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:12.069+0000 7fd937fff640 1 --2- 172.21.15.123:0/1604123323 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fd920073a90 0x7fd920075f50 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7fd9280af000 tx=0x7fd92801ad80 comp rx=0 tx=0).ready entity=mgr.4150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:52:12.072 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:12.073+0000 7fd9377fe640 1 -- 172.21.15.123:0/1604123323 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7fd930014030 con 0x7fd9400726c0 2024-09-16T15:52:12.305 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:12.305+0000 7fd9450fb640 1 -- 172.21.15.123:0/1604123323 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_volumes.TestMisc.test_unparseable_metafile_on_legacy_to_v1_upgrade"]} v 0) -- 0x7fd94015f630 con 0x7fd9400726c0 2024-09-16T15:52:12.564 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:12.565+0000 7fd9377fe640 1 -- 172.21.15.123:0/1604123323 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_volumes.TestMisc.test_unparseable_metafile_on_legacy_to_v1_upgrade"]}]=0 v494) ==== 163+0+0 (secure 0 0 0) 0x7fd93005fa00 con 0x7fd9400726c0 2024-09-16T15:52:12.567 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:12.569+0000 7fd9450fb640 1 -- 172.21.15.123:0/1604123323 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fd920073a90 msgr2=0x7fd920075f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:52:12.567 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:12.569+0000 7fd9450fb640 1 --2- 172.21.15.123:0/1604123323 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fd920073a90 0x7fd920075f50 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7fd9280af000 tx=0x7fd92801ad80 comp rx=0 tx=0).stop 2024-09-16T15:52:12.567 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:12.569+0000 7fd9450fb640 1 -- 172.21.15.123:0/1604123323 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd9400726c0 msgr2=0x7fd9401558a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:52:12.567 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:12.569+0000 7fd9450fb640 1 --2- 172.21.15.123:0/1604123323 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd9400726c0 0x7fd9401558a0 secure :-1 s=READY pgs=1968 cs=0 l=1 rev1=1 crypto rx=0x7fd93000c940 tx=0x7fd93000ce00 comp rx=0 tx=0).stop 2024-09-16T15:52:12.568 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:12.569+0000 7fd9450fb640 1 -- 172.21.15.123:0/1604123323 shutdown_connections 2024-09-16T15:52:12.568 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:12.569+0000 7fd9450fb640 1 --2- 172.21.15.123:0/1604123323 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fd920073a90 0x7fd920075f50 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:52:12.568 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:12.569+0000 7fd9450fb640 1 --2- 172.21.15.123:0/1604123323 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fd94015bca0 0x7fd940152c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:52:12.568 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:12.569+0000 7fd9450fb640 1 --2- 172.21.15.123:0/1604123323 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fd940074430 0x7fd940151600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:52:12.568 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:12.569+0000 7fd9450fb640 1 --2- 172.21.15.123:0/1604123323 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd9400726c0 0x7fd9401558a0 unknown :-1 s=CLOSED pgs=1968 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:52:12.568 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:12.569+0000 7fd9450fb640 1 -- 172.21.15.123:0/1604123323 >> 172.21.15.123:0/1604123323 conn(0x7fd9400cfb10 msgr2=0x7fd94006f290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:52:12.568 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:12.569+0000 7fd9450fb640 1 -- 172.21.15.123:0/1604123323 shutdown_connections 2024-09-16T15:52:12.568 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:12.569+0000 7fd9450fb640 1 -- 172.21.15.123:0/1604123323 wait complete. 2024-09-16T15:52:12.587 INFO:tasks.cephfs_test_runner:Validate unparseable text .meta file on legacy subvol root doesn't break the system ... ok 2024-09-16T15:52:12.588 INFO:tasks.cephfs_test_runner: 2024-09-16T15:52:12.588 INFO:tasks.cephfs_test_runner:---------------------------------------------------------------------- 2024-09-16T15:52:12.588 INFO:tasks.cephfs_test_runner:Ran 11 tests in 536.185s 2024-09-16T15:52:12.588 INFO:tasks.cephfs_test_runner: 2024-09-16T15:52:12.588 INFO:tasks.cephfs_test_runner:OK 2024-09-16T15:52:12.588 DEBUG:teuthology.run_tasks:Unwinding manager cephfs_test_runner 2024-09-16T15:52:12.597 DEBUG:teuthology.run_tasks:Unwinding manager ceph-fuse 2024-09-16T15:52:12.606 INFO:tasks.ceph_fuse:Unmounting ceph-fuse clients... 2024-09-16T15:52:12.606 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2024-09-16T15:52:12.606 DEBUG:teuthology.orchestra.run.smithi123:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-16T15:52:12.612 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16336468k,nr_inodes=4084117,mode=755,inode64 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3274684k,mode=755,inode64 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev,inode64 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=17082 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3274680k,nr_inodes=818670,mode=700,uid=1000,gid=1271,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3274684k,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-09-16T15:52:12.613 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2024-09-16T15:52:12.613 DEBUG:teuthology.orchestra.run.smithi123:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-16T15:52:12.659 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16336468k,nr_inodes=4084117,mode=755,inode64 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3274684k,mode=755,inode64 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev,inode64 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=17082 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3274680k,nr_inodes=818670,mode=700,uid=1000,gid=1271,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3274684k,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-09-16T15:52:12.659 DEBUG:teuthology.orchestra.run.smithi184:> set -ex 2024-09-16T15:52:12.659 DEBUG:teuthology.orchestra.run.smithi184:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-16T15:52:12.666 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16336488k,nr_inodes=4084122,mode=755,inode64 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3274688k,mode=755,inode64 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev,inode64 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=17834 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3274684k,nr_inodes=818671,mode=700,uid=1000,gid=1271,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3274688k,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-09-16T15:52:12.666 DEBUG:teuthology.orchestra.run.smithi184:> set -ex 2024-09-16T15:52:12.666 DEBUG:teuthology.orchestra.run.smithi184:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-16T15:52:12.712 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16336488k,nr_inodes=4084122,mode=755,inode64 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3274688k,mode=755,inode64 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev,inode64 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=17834 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3274684k,nr_inodes=818671,mode=700,uid=1000,gid=1271,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3274688k,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-09-16T15:52:12.712 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:52:12.712 DEBUG:teuthology.orchestra.run.smithi123:> ip netns list 2024-09-16T15:52:12.718 INFO:teuthology.orchestra.run.smithi123.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-09-16T15:52:12.719 INFO:teuthology.orchestra.run.smithi123.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-09-16T15:52:12.719 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:52:12.719 DEBUG:teuthology.orchestra.run.smithi123:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.2 2024-09-16T15:52:12.776 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:52:12.776 DEBUG:teuthology.orchestra.run.smithi123:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.0 2024-09-16T15:52:12.863 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:52:12.863 DEBUG:teuthology.orchestra.run.smithi123:> sudo ip link delete ceph-brx 2024-09-16T15:52:13.043 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:52:13.044 DEBUG:teuthology.orchestra.run.smithi184:> ip netns list 2024-09-16T15:52:13.050 INFO:teuthology.orchestra.run.smithi184.stdout:ceph-ns--home-ubuntu-cephtest-mnt.3 (id: 1) 2024-09-16T15:52:13.050 INFO:teuthology.orchestra.run.smithi184.stdout:ceph-ns--home-ubuntu-cephtest-mnt.1 (id: 0) 2024-09-16T15:52:13.051 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:52:13.051 DEBUG:teuthology.orchestra.run.smithi184:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.3 2024-09-16T15:52:13.105 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:52:13.105 DEBUG:teuthology.orchestra.run.smithi184:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.1 2024-09-16T15:52:13.161 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T15:52:13.161 DEBUG:teuthology.orchestra.run.smithi184:> sudo ip link delete ceph-brx 2024-09-16T15:52:13.347 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-09-16T15:52:13.356 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-09-16T15:52:13.356 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T15:52:13.493 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:13.493+0000 7efd88d14640 1 -- 172.21.15.123:0/190882278 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7efd6c004d50 msgr2=0x7efd6c0051d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:52:13.493 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:13.493+0000 7efd88d14640 1 --2- 172.21.15.123:0/190882278 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7efd6c004d50 0x7efd6c0051d0 secure :-1 s=READY pgs=1970 cs=0 l=1 rev1=1 crypto rx=0x7efd7c00c640 tx=0x7efd7c01fa80 comp rx=0 tx=0).stop 2024-09-16T15:52:13.493 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:13.493+0000 7efd82d9d640 1 -- 172.21.15.123:0/190882278 reap_dead start 2024-09-16T15:52:13.493 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:13.493+0000 7efd88d14640 1 -- 172.21.15.123:0/190882278 shutdown_connections 2024-09-16T15:52:13.493 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:13.493+0000 7efd88d14640 1 -- 172.21.15.123:0/190882278 >> 172.21.15.123:0/190882278 conn(0x7efd840cfb10 msgr2=0x7efd840cff20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:52:13.493 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:13.493+0000 7efd88d14640 1 -- 172.21.15.123:0/190882278 shutdown_connections 2024-09-16T15:52:13.493 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:13.493+0000 7efd88d14640 1 -- 172.21.15.123:0/190882278 wait complete. 2024-09-16T15:52:13.494 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:13.493+0000 7efd88d14640 1 Processor -- start 2024-09-16T15:52:13.494 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:13.493+0000 7efd88d14640 1 -- start start 2024-09-16T15:52:13.496 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:13.497+0000 7efd88d14640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7efd841f1fb0 0x7efd840c93d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:52:13.496 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:13.497+0000 7efd88d14640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7efd841f2390 0x7efd840c4b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:52:13.496 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:13.497+0000 7efd88d14640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7efd8415c340 0x7efd840c5090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:52:13.496 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:13.497+0000 7efd88d14640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7efd8406f8e0 con 0x7efd8415c340 2024-09-16T15:52:13.496 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:13.497+0000 7efd88d14640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7efd8406f760 con 0x7efd841f2390 2024-09-16T15:52:13.496 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:13.497+0000 7efd88d14640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7efd8406fa60 con 0x7efd841f1fb0 2024-09-16T15:52:13.496 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:13.497+0000 7efd82d9d640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7efd8415c340 0x7efd840c5090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:52:13.497 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:13.497+0000 7efd82d9d640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7efd8415c340 0x7efd840c5090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:34052/0 (socket says 172.21.15.123:34052) 2024-09-16T15:52:13.497 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:13.497+0000 7efd82d9d640 1 -- 172.21.15.123:0/3561174845 learned_addr learned my addr 172.21.15.123:0/3561174845 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:52:13.497 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:13.497+0000 7efd81d9b640 1 --2- 172.21.15.123:0/3561174845 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7efd841f2390 0x7efd840c4b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:52:13.497 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:13.497+0000 7efd8259c640 1 --2- 172.21.15.123:0/3561174845 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7efd841f1fb0 0x7efd840c93d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:52:13.497 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:13.497+0000 7efd82d9d640 1 -- 172.21.15.123:0/3561174845 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7efd841f1fb0 msgr2=0x7efd840c93d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:52:13.497 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:13.497+0000 7efd82d9d640 1 --2- 172.21.15.123:0/3561174845 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7efd841f1fb0 0x7efd840c93d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:52:13.497 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:13.497+0000 7efd82d9d640 1 -- 172.21.15.123:0/3561174845 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7efd841f2390 msgr2=0x7efd840c4b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:52:13.497 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:13.497+0000 7efd82d9d640 1 --2- 172.21.15.123:0/3561174845 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7efd841f2390 0x7efd840c4b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:52:13.497 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:13.497+0000 7efd82d9d640 1 -- 172.21.15.123:0/3561174845 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efd8415c820 con 0x7efd8415c340 2024-09-16T15:52:13.497 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:13.497+0000 7efd82d9d640 1 --2- 172.21.15.123:0/3561174845 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7efd8415c340 0x7efd840c5090 secure :-1 s=READY pgs=1971 cs=0 l=1 rev1=1 crypto rx=0x7efd7c09c370 tx=0x7efd7c0a9450 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:52:13.497 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:13.497+0000 7efd72ffd640 1 -- 172.21.15.123:0/3561174845 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd7c003c60 con 0x7efd8415c340 2024-09-16T15:52:13.497 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:13.497+0000 7efd72ffd640 1 -- 172.21.15.123:0/3561174845 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7efd7c003e00 con 0x7efd8415c340 2024-09-16T15:52:13.498 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:13.497+0000 7efd88d14640 1 -- 172.21.15.123:0/3561174845 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efd8415cb00 con 0x7efd8415c340 2024-09-16T15:52:13.498 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:13.497+0000 7efd72ffd640 1 -- 172.21.15.123:0/3561174845 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd7c0a9890 con 0x7efd8415c340 2024-09-16T15:52:13.498 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:13.497+0000 7efd88d14640 1 -- 172.21.15.123:0/3561174845 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efd8415d0c0 con 0x7efd8415c340 2024-09-16T15:52:13.498 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:13.497+0000 7efd88d14640 1 -- 172.21.15.123:0/3561174845 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efd840d09e0 con 0x7efd8415c340 2024-09-16T15:52:13.498 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:13.497+0000 7efd72ffd640 1 -- 172.21.15.123:0/3561174845 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 27) ==== 98151+0+0 (secure 0 0 0) 0x7efd7c0a9a30 con 0x7efd8415c340 2024-09-16T15:52:13.499 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:13.497+0000 7efd72ffd640 1 --2- 172.21.15.123:0/3561174845 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7efd64073ac0 0x7efd64075f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:52:13.499 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:13.497+0000 7efd8259c640 1 --2- 172.21.15.123:0/3561174845 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7efd64073ac0 0x7efd64075f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:52:13.499 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:13.497+0000 7efd72ffd640 1 -- 172.21.15.123:0/3561174845 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(172..172 src has 1..172) ==== 4971+0+0 (secure 0 0 0) 0x7efd7c12f8d0 con 0x7efd8415c340 2024-09-16T15:52:13.499 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:13.497+0000 7efd8259c640 1 --2- 172.21.15.123:0/3561174845 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7efd64073ac0 0x7efd64075f80 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7efd74009b40 tx=0x7efd740091e0 comp rx=0 tx=0).ready entity=mgr.4150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:52:13.502 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:13.501+0000 7efd72ffd640 1 -- 172.21.15.123:0/3561174845 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7efd7c019830 con 0x7efd8415c340 2024-09-16T15:52:13.670 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:13.669+0000 7efd88d14640 1 -- 172.21.15.123:0/3561174845 --> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7efd8415f630 con 0x7efd64073ac0 2024-09-16T15:52:13.671 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:13.673+0000 7efd72ffd640 1 -- 172.21.15.123:0/3561174845 <== mgr.4150 v2:172.21.15.123:6836/2464759676 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+13516 (secure 0 0 0) 0x7efd8415f630 con 0x7efd64073ac0 2024-09-16T15:52:13.671 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-09-16T15:52:13.672 INFO:teuthology.orchestra.run.smithi123.stderr:dumped all 2024-09-16T15:52:13.673 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:13.673+0000 7efd88d14640 1 -- 172.21.15.123:0/3561174845 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7efd64073ac0 msgr2=0x7efd64075f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:52:13.673 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:13.673+0000 7efd88d14640 1 --2- 172.21.15.123:0/3561174845 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7efd64073ac0 0x7efd64075f80 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7efd74009b40 tx=0x7efd740091e0 comp rx=0 tx=0).stop 2024-09-16T15:52:13.673 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:13.673+0000 7efd88d14640 1 -- 172.21.15.123:0/3561174845 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7efd8415c340 msgr2=0x7efd840c5090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:52:13.674 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:13.673+0000 7efd88d14640 1 --2- 172.21.15.123:0/3561174845 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7efd8415c340 0x7efd840c5090 secure :-1 s=READY pgs=1971 cs=0 l=1 rev1=1 crypto rx=0x7efd7c09c370 tx=0x7efd7c0a9450 comp rx=0 tx=0).stop 2024-09-16T15:52:13.674 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:13.673+0000 7efd88d14640 1 -- 172.21.15.123:0/3561174845 shutdown_connections 2024-09-16T15:52:13.674 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:13.673+0000 7efd88d14640 1 --2- 172.21.15.123:0/3561174845 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7efd64073ac0 0x7efd64075f80 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:52:13.674 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:13.673+0000 7efd88d14640 1 --2- 172.21.15.123:0/3561174845 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7efd8415c340 0x7efd840c5090 unknown :-1 s=CLOSED pgs=1971 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:52:13.674 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:13.673+0000 7efd88d14640 1 --2- 172.21.15.123:0/3561174845 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7efd841f2390 0x7efd840c4b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:52:13.674 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:13.673+0000 7efd88d14640 1 --2- 172.21.15.123:0/3561174845 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7efd841f1fb0 0x7efd840c93d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:52:13.674 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:13.673+0000 7efd88d14640 1 -- 172.21.15.123:0/3561174845 >> 172.21.15.123:0/3561174845 conn(0x7efd840cfb10 msgr2=0x7efd840b4a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:52:13.674 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:13.673+0000 7efd88d14640 1 -- 172.21.15.123:0/3561174845 shutdown_connections 2024-09-16T15:52:13.674 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:13.673+0000 7efd88d14640 1 -- 172.21.15.123:0/3561174845 wait complete. 2024-09-16T15:52:13.690 INFO:teuthology.orchestra.run.smithi123.stdout:{"pg_ready":true,"pg_map":{"version":242,"stamp":"2024-09-16T15:52:12.150210+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79,"ondisk_log_size":79,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":280048,"kb_used_data":18064,"kb_used_omap":12,"kb_used_meta":261811,"kb_avail":754694672,"statfs":{"total":773094113280,"available":772807344128,"internally_reserved":0,"allocated":18497536,"data_stored":8897446,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":268095025},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":76,"apply_latency_ms":76,"commit_latency_ns":76000000,"apply_latency_ns":76000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-49701,"num_objects":-24,"num_object_clones":0,"num_object_copies":-48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-24,"num_whiteouts":0,"num_read":-2,"num_read_kb":-2,"num_write":-69,"num_write_kb":-73,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":-14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.002349"},"pg_stats":[{"pgid":"1.0","version":"73'79","reported_seq":414,"reported_epoch":170,"state":"active+clean","last_fresh":"2024-09-16T15:52:08.575913+0000","last_change":"2024-09-16T15:42:40.881820+0000","last_active":"2024-09-16T15:52:08.575913+0000","last_peered":"2024-09-16T15:52:08.575913+0000","last_clean":"2024-09-16T15:52:08.575913+0000","last_became_active":"2024-09-16T15:42:40.879965+0000","last_became_peered":"2024-09-16T15:42:40.879965+0000","last_unstale":"2024-09-16T15:52:08.575913+0000","last_undegraded":"2024-09-16T15:52:08.575913+0000","last_fullsized":"2024-09-16T15:52:08.575913+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:42:39.728762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:42:39.728762+0000","last_clean_scrub_stamp":"2024-09-16T15:42:39.728762+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:22:18.768932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79,"ondisk_log_size":79,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542262,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33920,"kb_used_data":2732,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94337920,"statfs":{"total":96636764160,"available":96602030080,"internally_reserved":0,"allocated":2797568,"data_stored":1593932,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":15,"apply_latency_ms":15,"commit_latency_ns":15000000,"apply_latency_ns":15000000},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542261,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33288,"kb_used_data":2100,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338552,"statfs":{"total":96636764160,"available":96602677248,"internally_reserved":0,"allocated":2150400,"data_stored":952115,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":16,"apply_latency_ms":16,"commit_latency_ns":16000000,"apply_latency_ns":16000000},"alerts":[]},{"osd":5,"up_from":14,"seq":60129542262,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33284,"kb_used_data":2096,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338556,"statfs":{"total":96636764160,"available":96602681344,"internally_reserved":0,"allocated":2146304,"data_stored":951812,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542260,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37512,"kb_used_data":2164,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334328,"statfs":{"total":96636764160,"available":96598351872,"internally_reserved":0,"allocated":2215936,"data_stored":1003971,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":3,"up_from":14,"seq":60129542261,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37452,"kb_used_data":2104,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334388,"statfs":{"total":96636764160,"available":96598413312,"internally_reserved":0,"allocated":2154496,"data_stored":951412,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":2,"up_from":14,"seq":60129542261,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33280,"kb_used_data":2092,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338560,"statfs":{"total":96636764160,"available":96602685440,"internally_reserved":0,"allocated":2142208,"data_stored":951230,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":1,"up_from":14,"seq":60129542263,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33288,"kb_used_data":2100,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338552,"statfs":{"total":96636764160,"available":96602677248,"internally_reserved":0,"allocated":2150400,"data_stored":951286,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[]},{"osd":0,"up_from":14,"seq":60129542263,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38024,"kb_used_data":2676,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94333816,"statfs":{"total":96636764160,"available":96597827584,"internally_reserved":0,"allocated":2740224,"data_stored":1541688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T15:52:13.690 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T15:52:13.816 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:13.817+0000 7fb0e237f640 1 -- 172.21.15.123:0/1198995240 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb0c4004810 msgr2=0x7fb0c4004c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:52:13.816 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:13.817+0000 7fb0e237f640 1 --2- 172.21.15.123:0/1198995240 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb0c4004810 0x7fb0c4004c90 secure :-1 s=READY pgs=1973 cs=0 l=1 rev1=1 crypto rx=0x7fb0dc0b0140 tx=0x7fb0d001cd80 comp rx=0 tx=0).stop 2024-09-16T15:52:13.816 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:13.817+0000 7fb0e237f640 1 -- 172.21.15.123:0/1198995240 shutdown_connections 2024-09-16T15:52:13.816 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:13.817+0000 7fb0e237f640 1 --2- 172.21.15.123:0/1198995240 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb0c4004810 0x7fb0c4004c90 unknown :-1 s=CLOSED pgs=1973 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:52:13.816 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:13.817+0000 7fb0e237f640 1 -- 172.21.15.123:0/1198995240 >> 172.21.15.123:0/1198995240 conn(0x7fb0dc0b2610 msgr2=0x7fb0dc0b29f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:52:13.816 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:13.817+0000 7fb0e237f640 1 -- 172.21.15.123:0/1198995240 shutdown_connections 2024-09-16T15:52:13.816 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:13.817+0000 7fb0e237f640 1 -- 172.21.15.123:0/1198995240 wait complete. 2024-09-16T15:52:13.817 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:13.817+0000 7fb0e237f640 1 Processor -- start 2024-09-16T15:52:13.817 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:13.817+0000 7fb0e237f640 1 -- start start 2024-09-16T15:52:13.819 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:13.817+0000 7fb0e237f640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb0c4004810 0x7fb0dc15a630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:52:13.819 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:13.817+0000 7fb0e237f640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb0dc0d02a0 0x7fb0dc158100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:52:13.819 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:13.817+0000 7fb0e237f640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fb0dc1fbc20 0x7fb0dc156410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:52:13.819 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:13.817+0000 7fb0e237f640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fb0dc161240 con 0x7fb0c4004810 2024-09-16T15:52:13.819 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:13.817+0000 7fb0e237f640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fb0dc1610c0 con 0x7fb0dc0d02a0 2024-09-16T15:52:13.819 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:13.817+0000 7fb0e237f640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fb0dc1613c0 con 0x7fb0dc1fbc20 2024-09-16T15:52:13.820 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:13.817+0000 7fb0dbfff640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb0c4004810 0x7fb0dc15a630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:52:13.820 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:13.817+0000 7fb0dbfff640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb0c4004810 0x7fb0dc15a630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:34070/0 (socket says 172.21.15.123:34070) 2024-09-16T15:52:13.820 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:13.817+0000 7fb0dbfff640 1 -- 172.21.15.123:0/636581508 learned_addr learned my addr 172.21.15.123:0/636581508 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:52:13.820 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:13.817+0000 7fb0e091c640 1 --2- 172.21.15.123:0/636581508 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fb0dc1fbc20 0x7fb0dc156410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:52:13.820 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:13.817+0000 7fb0db7fe640 1 --2- 172.21.15.123:0/636581508 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb0dc0d02a0 0x7fb0dc158100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:52:13.820 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:13.817+0000 7fb0dbfff640 1 -- 172.21.15.123:0/636581508 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fb0dc1fbc20 msgr2=0x7fb0dc156410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:52:13.820 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:13.817+0000 7fb0dbfff640 1 --2- 172.21.15.123:0/636581508 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fb0dc1fbc20 0x7fb0dc156410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:52:13.820 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:13.817+0000 7fb0dbfff640 1 -- 172.21.15.123:0/636581508 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb0dc0d02a0 msgr2=0x7fb0dc158100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:52:13.820 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:13.817+0000 7fb0dbfff640 1 --2- 172.21.15.123:0/636581508 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb0dc0d02a0 0x7fb0dc158100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:52:13.820 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:13.817+0000 7fb0dbfff640 1 -- 172.21.15.123:0/636581508 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb0dc158640 con 0x7fb0c4004810 2024-09-16T15:52:13.820 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:13.817+0000 7fb0e091c640 1 --2- 172.21.15.123:0/636581508 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fb0dc1fbc20 0x7fb0dc156410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T15:52:13.820 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:13.817+0000 7fb0db7fe640 1 --2- 172.21.15.123:0/636581508 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb0dc0d02a0 0x7fb0dc158100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T15:52:13.821 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:13.817+0000 7fb0dbfff640 1 --2- 172.21.15.123:0/636581508 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb0c4004810 0x7fb0dc15a630 secure :-1 s=READY pgs=1974 cs=0 l=1 rev1=1 crypto rx=0x7fb0cc00d580 tx=0x7fb0cc00da40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:52:13.821 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:13.821+0000 7fb0d8ff9640 1 -- 172.21.15.123:0/636581508 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0cc016020 con 0x7fb0c4004810 2024-09-16T15:52:13.821 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:13.821+0000 7fb0d8ff9640 1 -- 172.21.15.123:0/636581508 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fb0cc012630 con 0x7fb0c4004810 2024-09-16T15:52:13.821 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:13.821+0000 7fb0e237f640 1 -- 172.21.15.123:0/636581508 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb0dc157720 con 0x7fb0c4004810 2024-09-16T15:52:13.821 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:13.821+0000 7fb0d8ff9640 1 -- 172.21.15.123:0/636581508 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0cc012a20 con 0x7fb0c4004810 2024-09-16T15:52:13.821 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:13.821+0000 7fb0e237f640 1 -- 172.21.15.123:0/636581508 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb0dc157ce0 con 0x7fb0c4004810 2024-09-16T15:52:13.821 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:13.821+0000 7fb0e237f640 1 -- 172.21.15.123:0/636581508 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb0dc1fb4f0 con 0x7fb0c4004810 2024-09-16T15:52:13.822 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:13.821+0000 7fb0d8ff9640 1 -- 172.21.15.123:0/636581508 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 27) ==== 98151+0+0 (secure 0 0 0) 0x7fb0cc019020 con 0x7fb0c4004810 2024-09-16T15:52:13.822 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:13.821+0000 7fb0d8ff9640 1 --2- 172.21.15.123:0/636581508 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fb0c8073ac0 0x7fb0c8075f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:52:13.822 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:13.821+0000 7fb0db7fe640 1 --2- 172.21.15.123:0/636581508 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fb0c8073ac0 0x7fb0c8075f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:52:13.822 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:13.821+0000 7fb0d8ff9640 1 -- 172.21.15.123:0/636581508 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(172..172 src has 1..172) ==== 4971+0+0 (secure 0 0 0) 0x7fb0cc092b90 con 0x7fb0c4004810 2024-09-16T15:52:13.822 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:13.821+0000 7fb0db7fe640 1 --2- 172.21.15.123:0/636581508 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fb0c8073ac0 0x7fb0c8075f80 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7fb0d00b0000 tx=0x7fb0d0019b20 comp rx=0 tx=0).ready entity=mgr.4150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:52:13.824 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:13.825+0000 7fb0d8ff9640 1 -- 172.21.15.123:0/636581508 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7fb0cc020020 con 0x7fb0c4004810 2024-09-16T15:52:13.995 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:13.997+0000 7fb0e237f640 1 -- 172.21.15.123:0/636581508 --> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fb0dc1618c0 con 0x7fb0c8073ac0 2024-09-16T15:52:13.995 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:13.997+0000 7fb0d8ff9640 1 -- 172.21.15.123:0/636581508 <== mgr.4150 v2:172.21.15.123:6836/2464759676 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+13516 (secure 0 0 0) 0x7fb0dc1618c0 con 0x7fb0c8073ac0 2024-09-16T15:52:13.996 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-09-16T15:52:13.996 INFO:teuthology.orchestra.run.smithi123.stderr:dumped all 2024-09-16T15:52:13.998 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:13.997+0000 7fb0e237f640 1 -- 172.21.15.123:0/636581508 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fb0c8073ac0 msgr2=0x7fb0c8075f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:52:13.998 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:13.997+0000 7fb0e237f640 1 --2- 172.21.15.123:0/636581508 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fb0c8073ac0 0x7fb0c8075f80 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7fb0d00b0000 tx=0x7fb0d0019b20 comp rx=0 tx=0).stop 2024-09-16T15:52:13.998 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:13.997+0000 7fb0e237f640 1 -- 172.21.15.123:0/636581508 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb0c4004810 msgr2=0x7fb0dc15a630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:52:13.998 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:13.997+0000 7fb0e237f640 1 --2- 172.21.15.123:0/636581508 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb0c4004810 0x7fb0dc15a630 secure :-1 s=READY pgs=1974 cs=0 l=1 rev1=1 crypto rx=0x7fb0cc00d580 tx=0x7fb0cc00da40 comp rx=0 tx=0).stop 2024-09-16T15:52:13.998 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:13.997+0000 7fb0e237f640 1 -- 172.21.15.123:0/636581508 shutdown_connections 2024-09-16T15:52:13.998 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:13.997+0000 7fb0e237f640 1 --2- 172.21.15.123:0/636581508 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fb0c8073ac0 0x7fb0c8075f80 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:52:13.998 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:13.997+0000 7fb0e237f640 1 --2- 172.21.15.123:0/636581508 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fb0dc1fbc20 0x7fb0dc156410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:52:13.998 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:13.997+0000 7fb0e237f640 1 --2- 172.21.15.123:0/636581508 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fb0dc0d02a0 0x7fb0dc158100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:52:13.999 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:13.997+0000 7fb0e237f640 1 --2- 172.21.15.123:0/636581508 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb0c4004810 0x7fb0dc15a630 unknown :-1 s=CLOSED pgs=1974 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:52:13.999 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:13.997+0000 7fb0e237f640 1 -- 172.21.15.123:0/636581508 >> 172.21.15.123:0/636581508 conn(0x7fb0dc0b2610 msgr2=0x7fb0dc0c9c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:52:13.999 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:13.997+0000 7fb0e237f640 1 -- 172.21.15.123:0/636581508 shutdown_connections 2024-09-16T15:52:13.999 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:13.997+0000 7fb0e237f640 1 -- 172.21.15.123:0/636581508 wait complete. 2024-09-16T15:52:14.014 INFO:teuthology.orchestra.run.smithi123.stdout:{"pg_ready":true,"pg_map":{"version":242,"stamp":"2024-09-16T15:52:12.150210+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79,"ondisk_log_size":79,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":280048,"kb_used_data":18064,"kb_used_omap":12,"kb_used_meta":261811,"kb_avail":754694672,"statfs":{"total":773094113280,"available":772807344128,"internally_reserved":0,"allocated":18497536,"data_stored":8897446,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":268095025},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":76,"apply_latency_ms":76,"commit_latency_ns":76000000,"apply_latency_ns":76000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-49701,"num_objects":-24,"num_object_clones":0,"num_object_copies":-48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-24,"num_whiteouts":0,"num_read":-2,"num_read_kb":-2,"num_write":-69,"num_write_kb":-73,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":-14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.002349"},"pg_stats":[{"pgid":"1.0","version":"73'79","reported_seq":414,"reported_epoch":170,"state":"active+clean","last_fresh":"2024-09-16T15:52:08.575913+0000","last_change":"2024-09-16T15:42:40.881820+0000","last_active":"2024-09-16T15:52:08.575913+0000","last_peered":"2024-09-16T15:52:08.575913+0000","last_clean":"2024-09-16T15:52:08.575913+0000","last_became_active":"2024-09-16T15:42:40.879965+0000","last_became_peered":"2024-09-16T15:42:40.879965+0000","last_unstale":"2024-09-16T15:52:08.575913+0000","last_undegraded":"2024-09-16T15:52:08.575913+0000","last_fullsized":"2024-09-16T15:52:08.575913+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:42:39.728762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:42:39.728762+0000","last_clean_scrub_stamp":"2024-09-16T15:42:39.728762+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:22:18.768932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79,"ondisk_log_size":79,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542262,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33920,"kb_used_data":2732,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94337920,"statfs":{"total":96636764160,"available":96602030080,"internally_reserved":0,"allocated":2797568,"data_stored":1593932,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":15,"apply_latency_ms":15,"commit_latency_ns":15000000,"apply_latency_ns":15000000},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542261,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33288,"kb_used_data":2100,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338552,"statfs":{"total":96636764160,"available":96602677248,"internally_reserved":0,"allocated":2150400,"data_stored":952115,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":16,"apply_latency_ms":16,"commit_latency_ns":16000000,"apply_latency_ns":16000000},"alerts":[]},{"osd":5,"up_from":14,"seq":60129542262,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33284,"kb_used_data":2096,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338556,"statfs":{"total":96636764160,"available":96602681344,"internally_reserved":0,"allocated":2146304,"data_stored":951812,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542260,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37512,"kb_used_data":2164,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334328,"statfs":{"total":96636764160,"available":96598351872,"internally_reserved":0,"allocated":2215936,"data_stored":1003971,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":3,"up_from":14,"seq":60129542261,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37452,"kb_used_data":2104,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334388,"statfs":{"total":96636764160,"available":96598413312,"internally_reserved":0,"allocated":2154496,"data_stored":951412,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":2,"up_from":14,"seq":60129542261,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33280,"kb_used_data":2092,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338560,"statfs":{"total":96636764160,"available":96602685440,"internally_reserved":0,"allocated":2142208,"data_stored":951230,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":1,"up_from":14,"seq":60129542263,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33288,"kb_used_data":2100,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338552,"statfs":{"total":96636764160,"available":96602677248,"internally_reserved":0,"allocated":2150400,"data_stored":951286,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[]},{"osd":0,"up_from":14,"seq":60129542263,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38024,"kb_used_data":2676,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94333816,"statfs":{"total":96636764160,"available":96597827584,"internally_reserved":0,"allocated":2740224,"data_stored":1541688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T15:52:14.014 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-09-16T15:52:14.015 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T15:52:14.139 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:14.141+0000 7eff3bf97640 1 -- 172.21.15.123:0/1518598217 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7eff3415ec40 msgr2=0x7eff3406f420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:52:14.139 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:14.141+0000 7eff3bf97640 1 --2- 172.21.15.123:0/1518598217 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7eff3415ec40 0x7eff3406f420 secure :-1 s=READY pgs=1976 cs=0 l=1 rev1=1 crypto rx=0x7eff34072250 tx=0x7eff2c01cc60 comp rx=0 tx=0).stop 2024-09-16T15:52:14.139 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:14.141+0000 7eff3bf97640 1 -- 172.21.15.123:0/1518598217 shutdown_connections 2024-09-16T15:52:14.139 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:14.141+0000 7eff3bf97640 1 --2- 172.21.15.123:0/1518598217 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7eff3415ec40 0x7eff3406f420 unknown :-1 s=CLOSED pgs=1976 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:52:14.139 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:14.141+0000 7eff3bf97640 1 --2- 172.21.15.123:0/1518598217 >> v2:172.21.15.184:3300/0 conn(0x7eff340cf5a0 0x7eff3406eee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:52:14.139 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:14.141+0000 7eff3bf97640 1 -- 172.21.15.123:0/1518598217 >> 172.21.15.123:0/1518598217 conn(0x7eff340cfb10 msgr2=0x7eff340cff20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:52:14.139 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:14.141+0000 7eff3bf97640 1 -- 172.21.15.123:0/1518598217 shutdown_connections 2024-09-16T15:52:14.140 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:14.141+0000 7eff3bf97640 1 -- 172.21.15.123:0/1518598217 wait complete. 2024-09-16T15:52:14.140 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:14.141+0000 7eff3bf97640 1 Processor -- start 2024-09-16T15:52:14.140 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:14.141+0000 7eff3bf97640 1 -- start start 2024-09-16T15:52:14.142 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:14.141+0000 7eff3bf97640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7eff340cf5a0 0x7eff3414ff30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:52:14.142 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:14.141+0000 7eff3bf97640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7eff34158120 0x7eff340c9690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:52:14.142 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:14.141+0000 7eff3bf97640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7eff3415ec40 0x7eff341596b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:52:14.142 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:14.141+0000 7eff3bf97640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7eff341673d0 con 0x7eff340cf5a0 2024-09-16T15:52:14.142 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:14.141+0000 7eff3bf97640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7eff3415b8a0 con 0x7eff34158120 2024-09-16T15:52:14.142 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:14.141+0000 7eff3bf97640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7eff34167550 con 0x7eff3415ec40 2024-09-16T15:52:14.142 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:14.141+0000 7eff39d33640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7eff340cf5a0 0x7eff3414ff30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:52:14.142 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:14.141+0000 7eff39d33640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7eff340cf5a0 0x7eff3414ff30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:34078/0 (socket says 172.21.15.123:34078) 2024-09-16T15:52:14.142 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:14.141+0000 7eff39d33640 1 -- 172.21.15.123:0/993454884 learned_addr learned my addr 172.21.15.123:0/993454884 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:52:14.142 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:14.141+0000 7eff39d33640 1 -- 172.21.15.123:0/993454884 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7eff3415ec40 msgr2=0x7eff341596b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:52:14.142 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:14.141+0000 7eff39d33640 1 --2- 172.21.15.123:0/993454884 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7eff3415ec40 0x7eff341596b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:52:14.143 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:14.141+0000 7eff39d33640 1 -- 172.21.15.123:0/993454884 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7eff34158120 msgr2=0x7eff340c9690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:52:14.143 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:14.141+0000 7eff39d33640 1 --2- 172.21.15.123:0/993454884 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7eff34158120 0x7eff340c9690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:52:14.143 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:14.141+0000 7eff39d33640 1 -- 172.21.15.123:0/993454884 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7eff34150470 con 0x7eff340cf5a0 2024-09-16T15:52:14.143 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:14.141+0000 7eff39d33640 1 --2- 172.21.15.123:0/993454884 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7eff340cf5a0 0x7eff3414ff30 secure :-1 s=READY pgs=1977 cs=0 l=1 rev1=1 crypto rx=0x7eff2400c500 tx=0x7eff2400c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:52:14.143 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:14.141+0000 7eff2a7fc640 1 -- 172.21.15.123:0/993454884 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff24017070 con 0x7eff340cf5a0 2024-09-16T15:52:14.143 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:14.141+0000 7eff2a7fc640 1 -- 172.21.15.123:0/993454884 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7eff240053c0 con 0x7eff340cf5a0 2024-09-16T15:52:14.143 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:14.141+0000 7eff2a7fc640 1 -- 172.21.15.123:0/993454884 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff24012890 con 0x7eff340cf5a0 2024-09-16T15:52:14.143 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:14.141+0000 7eff3bf97640 1 -- 172.21.15.123:0/993454884 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7eff341f8ba0 con 0x7eff340cf5a0 2024-09-16T15:52:14.143 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:14.141+0000 7eff3bf97640 1 -- 172.21.15.123:0/993454884 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7eff341f9160 con 0x7eff340cf5a0 2024-09-16T15:52:14.144 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:14.145+0000 7eff3bf97640 1 -- 172.21.15.123:0/993454884 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7eff340ca320 con 0x7eff340cf5a0 2024-09-16T15:52:14.144 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:14.145+0000 7eff2a7fc640 1 -- 172.21.15.123:0/993454884 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 27) ==== 98151+0+0 (secure 0 0 0) 0x7eff2401a070 con 0x7eff340cf5a0 2024-09-16T15:52:14.144 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:14.145+0000 7eff2a7fc640 1 --2- 172.21.15.123:0/993454884 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7eff18073a90 0x7eff18075f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:52:14.144 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:14.145+0000 7eff2a7fc640 1 -- 172.21.15.123:0/993454884 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(172..172 src has 1..172) ==== 4971+0+0 (secure 0 0 0) 0x7eff24091ef0 con 0x7eff340cf5a0 2024-09-16T15:52:14.144 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:14.145+0000 7eff39532640 1 --2- 172.21.15.123:0/993454884 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7eff18073a90 0x7eff18075f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:52:14.144 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:14.145+0000 7eff39532640 1 --2- 172.21.15.123:0/993454884 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7eff18073a90 0x7eff18075f50 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7eff341532b0 tx=0x7eff30005c80 comp rx=0 tx=0).ready entity=mgr.4150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:52:14.147 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:14.149+0000 7eff2a7fc640 1 -- 172.21.15.123:0/993454884 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7eff24021080 con 0x7eff340cf5a0 2024-09-16T15:52:14.305 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:14.305+0000 7eff3bf97640 1 -- 172.21.15.123:0/993454884 --> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7eff34153000 con 0x7eff18073a90 2024-09-16T15:52:14.306 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:14.309+0000 7eff2a7fc640 1 -- 172.21.15.123:0/993454884 <== mgr.4150 v2:172.21.15.123:6836/2464759676 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+13516 (secure 0 0 0) 0x7eff34153000 con 0x7eff18073a90 2024-09-16T15:52:14.306 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-09-16T15:52:14.307 INFO:teuthology.orchestra.run.smithi123.stderr:dumped all 2024-09-16T15:52:14.308 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:14.309+0000 7eff3bf97640 1 -- 172.21.15.123:0/993454884 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7eff18073a90 msgr2=0x7eff18075f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:52:14.309 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:14.309+0000 7eff3bf97640 1 --2- 172.21.15.123:0/993454884 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7eff18073a90 0x7eff18075f50 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7eff341532b0 tx=0x7eff30005c80 comp rx=0 tx=0).stop 2024-09-16T15:52:14.309 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:14.309+0000 7eff3bf97640 1 -- 172.21.15.123:0/993454884 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7eff340cf5a0 msgr2=0x7eff3414ff30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:52:14.309 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:14.309+0000 7eff3bf97640 1 --2- 172.21.15.123:0/993454884 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7eff340cf5a0 0x7eff3414ff30 secure :-1 s=READY pgs=1977 cs=0 l=1 rev1=1 crypto rx=0x7eff2400c500 tx=0x7eff2400c9c0 comp rx=0 tx=0).stop 2024-09-16T15:52:14.309 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:14.309+0000 7eff3bf97640 1 -- 172.21.15.123:0/993454884 shutdown_connections 2024-09-16T15:52:14.309 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:14.309+0000 7eff3bf97640 1 --2- 172.21.15.123:0/993454884 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7eff18073a90 0x7eff18075f50 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:52:14.309 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:14.309+0000 7eff3bf97640 1 --2- 172.21.15.123:0/993454884 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7eff3415ec40 0x7eff341596b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:52:14.309 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:14.309+0000 7eff3bf97640 1 --2- 172.21.15.123:0/993454884 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7eff34158120 0x7eff340c9690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:52:14.309 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:14.309+0000 7eff3bf97640 1 --2- 172.21.15.123:0/993454884 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7eff340cf5a0 0x7eff3414ff30 unknown :-1 s=CLOSED pgs=1977 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:52:14.309 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:14.309+0000 7eff3bf97640 1 -- 172.21.15.123:0/993454884 >> 172.21.15.123:0/993454884 conn(0x7eff340cfb10 msgr2=0x7eff34153520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:52:14.309 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:14.309+0000 7eff3bf97640 1 -- 172.21.15.123:0/993454884 shutdown_connections 2024-09-16T15:52:14.309 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:14.309+0000 7eff3bf97640 1 -- 172.21.15.123:0/993454884 wait complete. 2024-09-16T15:52:14.324 INFO:teuthology.orchestra.run.smithi123.stdout:{"pg_ready":true,"pg_map":{"version":243,"stamp":"2024-09-16T15:52:14.150572+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79,"ondisk_log_size":79,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":280048,"kb_used_data":18064,"kb_used_omap":12,"kb_used_meta":261811,"kb_avail":754694672,"statfs":{"total":773094113280,"available":772807344128,"internally_reserved":0,"allocated":18497536,"data_stored":8897446,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":268095025},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":76,"apply_latency_ms":76,"commit_latency_ns":76000000,"apply_latency_ns":76000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-49701,"num_objects":-24,"num_object_clones":0,"num_object_copies":-48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-24,"num_whiteouts":0,"num_read":-2,"num_read_kb":-2,"num_write":-69,"num_write_kb":-73,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":-14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001888"},"pg_stats":[{"pgid":"1.0","version":"73'79","reported_seq":414,"reported_epoch":170,"state":"active+clean","last_fresh":"2024-09-16T15:52:08.575913+0000","last_change":"2024-09-16T15:42:40.881820+0000","last_active":"2024-09-16T15:52:08.575913+0000","last_peered":"2024-09-16T15:52:08.575913+0000","last_clean":"2024-09-16T15:52:08.575913+0000","last_became_active":"2024-09-16T15:42:40.879965+0000","last_became_peered":"2024-09-16T15:42:40.879965+0000","last_unstale":"2024-09-16T15:52:08.575913+0000","last_undegraded":"2024-09-16T15:52:08.575913+0000","last_fullsized":"2024-09-16T15:52:08.575913+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:42:39.728762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:42:39.728762+0000","last_clean_scrub_stamp":"2024-09-16T15:42:39.728762+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:22:18.768932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79,"ondisk_log_size":79,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542262,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33920,"kb_used_data":2732,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94337920,"statfs":{"total":96636764160,"available":96602030080,"internally_reserved":0,"allocated":2797568,"data_stored":1593932,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":15,"apply_latency_ms":15,"commit_latency_ns":15000000,"apply_latency_ns":15000000},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542261,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33288,"kb_used_data":2100,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338552,"statfs":{"total":96636764160,"available":96602677248,"internally_reserved":0,"allocated":2150400,"data_stored":952115,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":16,"apply_latency_ms":16,"commit_latency_ns":16000000,"apply_latency_ns":16000000},"alerts":[]},{"osd":5,"up_from":14,"seq":60129542262,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33284,"kb_used_data":2096,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338556,"statfs":{"total":96636764160,"available":96602681344,"internally_reserved":0,"allocated":2146304,"data_stored":951812,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542260,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37512,"kb_used_data":2164,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334328,"statfs":{"total":96636764160,"available":96598351872,"internally_reserved":0,"allocated":2215936,"data_stored":1003971,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":3,"up_from":14,"seq":60129542261,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37452,"kb_used_data":2104,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334388,"statfs":{"total":96636764160,"available":96598413312,"internally_reserved":0,"allocated":2154496,"data_stored":951412,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":2,"up_from":14,"seq":60129542261,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33280,"kb_used_data":2092,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338560,"statfs":{"total":96636764160,"available":96602685440,"internally_reserved":0,"allocated":2142208,"data_stored":951230,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":1,"up_from":14,"seq":60129542263,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33288,"kb_used_data":2100,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338552,"statfs":{"total":96636764160,"available":96602677248,"internally_reserved":0,"allocated":2150400,"data_stored":951286,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[]},{"osd":0,"up_from":14,"seq":60129542263,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38024,"kb_used_data":2676,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94333816,"statfs":{"total":96636764160,"available":96597827584,"internally_reserved":0,"allocated":2740224,"data_stored":1541688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T15:52:14.325 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-16T15:52:14.447 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:14.449+0000 7f496ec45640 1 -- 172.21.15.123:0/1526338174 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4958004d50 msgr2=0x7f49580051d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:52:14.448 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:14.449+0000 7f496ec45640 1 --2- 172.21.15.123:0/1526338174 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4958004d50 0x7f49580051d0 secure :-1 s=READY pgs=1979 cs=0 l=1 rev1=1 crypto rx=0x7f496000c5c0 tx=0x7f496001fa40 comp rx=0 tx=0).stop 2024-09-16T15:52:14.448 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:14.449+0000 7f496d1e2640 1 -- 172.21.15.123:0/1526338174 reap_dead start 2024-09-16T15:52:14.448 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:14.449+0000 7f496ec45640 1 -- 172.21.15.123:0/1526338174 shutdown_connections 2024-09-16T15:52:14.448 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:14.449+0000 7f496ec45640 1 -- 172.21.15.123:0/1526338174 >> 172.21.15.123:0/1526338174 conn(0x7f49680cfb10 msgr2=0x7f49680cff20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:52:14.448 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:14.449+0000 7f496ec45640 1 -- 172.21.15.123:0/1526338174 shutdown_connections 2024-09-16T15:52:14.448 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:14.449+0000 7f496ec45640 1 -- 172.21.15.123:0/1526338174 wait complete. 2024-09-16T15:52:14.449 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:14.449+0000 7f496ec45640 1 Processor -- start 2024-09-16T15:52:14.449 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:14.449+0000 7f496ec45640 1 -- start start 2024-09-16T15:52:14.450 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:14.449+0000 7f496ec45640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f4958004d50 0x7f496815b520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:52:14.451 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:14.449+0000 7f496ec45640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f496815d010 0x7f49680c36d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:52:14.451 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:14.449+0000 7f496ec45640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f496815d490 0x7f49680c3c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:52:14.451 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:14.449+0000 7f496ec45640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f49680cf5b0 con 0x7f496815d490 2024-09-16T15:52:14.451 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:14.449+0000 7f496ec45640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f49680cf430 con 0x7f496815d010 2024-09-16T15:52:14.451 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:14.449+0000 7f496ec45640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f4968073f30 con 0x7f4958004d50 2024-09-16T15:52:14.451 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:14.449+0000 7f496d1e2640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f496815d490 0x7f49680c3c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:52:14.451 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:14.449+0000 7f496d1e2640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f496815d490 0x7f49680c3c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:34100/0 (socket says 172.21.15.123:34100) 2024-09-16T15:52:14.451 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:14.449+0000 7f496d1e2640 1 -- 172.21.15.123:0/499586395 learned_addr learned my addr 172.21.15.123:0/499586395 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:52:14.451 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:14.449+0000 7f496c9e1640 1 --2- 172.21.15.123:0/499586395 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f4958004d50 0x7f496815b520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:52:14.451 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:14.449+0000 7f4967fff640 1 --2- 172.21.15.123:0/499586395 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f496815d010 0x7f49680c36d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:52:14.451 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:14.449+0000 7f496d1e2640 1 -- 172.21.15.123:0/499586395 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f4958004d50 msgr2=0x7f496815b520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:52:14.451 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:14.449+0000 7f496d1e2640 1 --2- 172.21.15.123:0/499586395 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f4958004d50 0x7f496815b520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:52:14.451 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:14.449+0000 7f496d1e2640 1 -- 172.21.15.123:0/499586395 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f496815d010 msgr2=0x7f49680c36d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:52:14.452 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:14.449+0000 7f496d1e2640 1 --2- 172.21.15.123:0/499586395 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f496815d010 0x7f49680c36d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:52:14.452 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:14.449+0000 7f496d1e2640 1 -- 172.21.15.123:0/499586395 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f49681f4920 con 0x7f496815d490 2024-09-16T15:52:14.452 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:14.449+0000 7f496c9e1640 1 --2- 172.21.15.123:0/499586395 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f4958004d50 0x7f496815b520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T15:52:14.452 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:14.449+0000 7f496d1e2640 1 --2- 172.21.15.123:0/499586395 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f496815d490 0x7f49680c3c10 secure :-1 s=READY pgs=1980 cs=0 l=1 rev1=1 crypto rx=0x7f496001e140 tx=0x7f4960001590 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:52:14.452 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:14.453+0000 7f49657fa640 1 -- 172.21.15.123:0/499586395 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4960004e60 con 0x7f496815d490 2024-09-16T15:52:14.452 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:14.453+0000 7f49657fa640 1 -- 172.21.15.123:0/499586395 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f4960001880 con 0x7f496815d490 2024-09-16T15:52:14.452 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:14.453+0000 7f496ec45640 1 -- 172.21.15.123:0/499586395 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f49681f4c00 con 0x7f496815d490 2024-09-16T15:52:14.452 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:14.453+0000 7f49657fa640 1 -- 172.21.15.123:0/499586395 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49600011d0 con 0x7f496815d490 2024-09-16T15:52:14.452 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:14.453+0000 7f496ec45640 1 -- 172.21.15.123:0/499586395 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f49681f51c0 con 0x7f496815d490 2024-09-16T15:52:14.453 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:14.453+0000 7f4952ffd640 1 -- 172.21.15.123:0/499586395 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f49400052a0 con 0x7f496815d490 2024-09-16T15:52:14.453 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:14.453+0000 7f49657fa640 1 -- 172.21.15.123:0/499586395 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 27) ==== 98151+0+0 (secure 0 0 0) 0x7f49600bb030 con 0x7f496815d490 2024-09-16T15:52:14.453 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:14.453+0000 7f49657fa640 1 --2- 172.21.15.123:0/499586395 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f494c073ac0 0x7f494c075f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:52:14.453 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:14.453+0000 7f496c9e1640 1 --2- 172.21.15.123:0/499586395 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f494c073ac0 0x7f494c075f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:52:14.453 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:14.453+0000 7f49657fa640 1 -- 172.21.15.123:0/499586395 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(172..172 src has 1..172) ==== 4971+0+0 (secure 0 0 0) 0x7f496012f210 con 0x7f496815d490 2024-09-16T15:52:14.453 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:14.453+0000 7f496c9e1640 1 --2- 172.21.15.123:0/499586395 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f494c073ac0 0x7f494c075f80 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f495c0058d0 tx=0x7f495c005860 comp rx=0 tx=0).ready entity=mgr.4150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:52:14.456 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:14.457+0000 7f49657fa640 1 -- 172.21.15.123:0/499586395 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f49600197f0 con 0x7f496815d490 2024-09-16T15:52:14.618 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:14.621+0000 7f4952ffd640 1 -- 172.21.15.123:0/499586395 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f4940005860 con 0x7f496815d490 2024-09-16T15:52:14.620 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:14.621+0000 7f49657fa640 1 -- 172.21.15.123:0/499586395 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v172) ==== 74+0+13146 (secure 0 0 0) 0x7f49600fbe20 con 0x7f496815d490 2024-09-16T15:52:14.620 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-09-16T15:52:14.620 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":172,"fsid":"56077f89-18c8-486f-ae7e-eecc0bb7822c","created":"2024-09-16T15:42:26.364892+0000","modified":"2024-09-16T15:52:11.554625+0000","last_up_change":"2024-09-16T15:42:38.707234+0000","last_in_change":"2024-09-16T15:42:29.423662+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":27,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-16T15:42:39.287764+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9e7c1955-a607-4671-9be7-f2ef760ceeaa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":167,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6800","nonce":3882044969},{"type":"v1","addr":"172.21.15.123:6801","nonce":3882044969}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6802","nonce":3882044969},{"type":"v1","addr":"172.21.15.123:6803","nonce":3882044969}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6806","nonce":3882044969},{"type":"v1","addr":"172.21.15.123:6807","nonce":3882044969}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6804","nonce":3882044969},{"type":"v1","addr":"172.21.15.123:6805","nonce":3882044969}]},"public_addr":"172.21.15.123:6801/3882044969","cluster_addr":"172.21.15.123:6803/3882044969","heartbeat_back_addr":"172.21.15.123:6807/3882044969","heartbeat_front_addr":"172.21.15.123:6805/3882044969","state":["exists","up"]},{"osd":1,"uuid":"c540cfde-0977-4742-9b17-fd634668e010","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":167,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6816","nonce":4037001850},{"type":"v1","addr":"172.21.15.123:6817","nonce":4037001850}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6818","nonce":4037001850},{"type":"v1","addr":"172.21.15.123:6819","nonce":4037001850}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6822","nonce":4037001850},{"type":"v1","addr":"172.21.15.123:6823","nonce":4037001850}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6820","nonce":4037001850},{"type":"v1","addr":"172.21.15.123:6821","nonce":4037001850}]},"public_addr":"172.21.15.123:6817/4037001850","cluster_addr":"172.21.15.123:6819/4037001850","heartbeat_back_addr":"172.21.15.123:6823/4037001850","heartbeat_front_addr":"172.21.15.123:6821/4037001850","state":["exists","up"]},{"osd":2,"uuid":"9864a496-f169-486b-95c4-1d59162793a7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":167,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6824","nonce":1332143088},{"type":"v1","addr":"172.21.15.123:6825","nonce":1332143088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6826","nonce":1332143088},{"type":"v1","addr":"172.21.15.123:6827","nonce":1332143088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6830","nonce":1332143088},{"type":"v1","addr":"172.21.15.123:6831","nonce":1332143088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6828","nonce":1332143088},{"type":"v1","addr":"172.21.15.123:6829","nonce":1332143088}]},"public_addr":"172.21.15.123:6825/1332143088","cluster_addr":"172.21.15.123:6827/1332143088","heartbeat_back_addr":"172.21.15.123:6831/1332143088","heartbeat_front_addr":"172.21.15.123:6829/1332143088","state":["exists","up"]},{"osd":3,"uuid":"c6d1f817-f074-4325-95e1-578eba13edda","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":167,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6808","nonce":988586998},{"type":"v1","addr":"172.21.15.123:6809","nonce":988586998}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6810","nonce":988586998},{"type":"v1","addr":"172.21.15.123:6811","nonce":988586998}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6814","nonce":988586998},{"type":"v1","addr":"172.21.15.123:6815","nonce":988586998}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6812","nonce":988586998},{"type":"v1","addr":"172.21.15.123:6813","nonce":988586998}]},"public_addr":"172.21.15.123:6809/988586998","cluster_addr":"172.21.15.123:6811/988586998","heartbeat_back_addr":"172.21.15.123:6815/988586998","heartbeat_front_addr":"172.21.15.123:6813/988586998","state":["exists","up"]},{"osd":4,"uuid":"200a8df4-50ef-40a5-b48a-58981269c086","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":167,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6816","nonce":3387813890},{"type":"v1","addr":"172.21.15.184:6817","nonce":3387813890}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6818","nonce":3387813890},{"type":"v1","addr":"172.21.15.184:6819","nonce":3387813890}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6822","nonce":3387813890},{"type":"v1","addr":"172.21.15.184:6823","nonce":3387813890}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6820","nonce":3387813890},{"type":"v1","addr":"172.21.15.184:6821","nonce":3387813890}]},"public_addr":"172.21.15.184:6817/3387813890","cluster_addr":"172.21.15.184:6819/3387813890","heartbeat_back_addr":"172.21.15.184:6823/3387813890","heartbeat_front_addr":"172.21.15.184:6821/3387813890","state":["exists","up"]},{"osd":5,"uuid":"32d5a900-da3d-412c-bc19-13861a2368d8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":167,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6808","nonce":2680916017},{"type":"v1","addr":"172.21.15.184:6809","nonce":2680916017}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6810","nonce":2680916017},{"type":"v1","addr":"172.21.15.184:6811","nonce":2680916017}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6814","nonce":2680916017},{"type":"v1","addr":"172.21.15.184:6815","nonce":2680916017}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6812","nonce":2680916017},{"type":"v1","addr":"172.21.15.184:6813","nonce":2680916017}]},"public_addr":"172.21.15.184:6809/2680916017","cluster_addr":"172.21.15.184:6811/2680916017","heartbeat_back_addr":"172.21.15.184:6815/2680916017","heartbeat_front_addr":"172.21.15.184:6813/2680916017","state":["exists","up"]},{"osd":6,"uuid":"cde6f39e-aa1b-4260-ac19-774ee6aaca72","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":167,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6824","nonce":2183120457},{"type":"v1","addr":"172.21.15.184:6825","nonce":2183120457}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6826","nonce":2183120457},{"type":"v1","addr":"172.21.15.184:6827","nonce":2183120457}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6830","nonce":2183120457},{"type":"v1","addr":"172.21.15.184:6831","nonce":2183120457}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6828","nonce":2183120457},{"type":"v1","addr":"172.21.15.184:6829","nonce":2183120457}]},"public_addr":"172.21.15.184:6825/2183120457","cluster_addr":"172.21.15.184:6827/2183120457","heartbeat_back_addr":"172.21.15.184:6831/2183120457","heartbeat_front_addr":"172.21.15.184:6829/2183120457","state":["exists","up"]},{"osd":7,"uuid":"4f381778-eee1-47c6-804f-df0e6b33e313","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":167,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6800","nonce":4102957285},{"type":"v1","addr":"172.21.15.184:6801","nonce":4102957285}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6802","nonce":4102957285},{"type":"v1","addr":"172.21.15.184:6803","nonce":4102957285}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6806","nonce":4102957285},{"type":"v1","addr":"172.21.15.184:6807","nonce":4102957285}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6804","nonce":4102957285},{"type":"v1","addr":"172.21.15.184:6805","nonce":4102957285}]},"public_addr":"172.21.15.184:6801/4102957285","cluster_addr":"172.21.15.184:6803/4102957285","heartbeat_back_addr":"172.21.15.184:6807/4102957285","heartbeat_front_addr":"172.21.15.184:6805/4102957285","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:35.972210+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.222007+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.198617+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.385663+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.358935+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.156066+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:36.752973+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T15:42:35.682846+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.123:6835/370478759":"2024-09-17T15:52:08.274651+0000","172.21.15.123:6834/370478759":"2024-09-17T15:52:08.274651+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":17,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T15:52:14.622 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:14.621+0000 7f4952ffd640 1 -- 172.21.15.123:0/499586395 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f494c073ac0 msgr2=0x7f494c075f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:52:14.622 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:14.621+0000 7f4952ffd640 1 --2- 172.21.15.123:0/499586395 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f494c073ac0 0x7f494c075f80 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f495c0058d0 tx=0x7f495c005860 comp rx=0 tx=0).stop 2024-09-16T15:52:14.623 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:14.621+0000 7f4952ffd640 1 -- 172.21.15.123:0/499586395 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f496815d490 msgr2=0x7f49680c3c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:52:14.623 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:14.621+0000 7f4952ffd640 1 --2- 172.21.15.123:0/499586395 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f496815d490 0x7f49680c3c10 secure :-1 s=READY pgs=1980 cs=0 l=1 rev1=1 crypto rx=0x7f496001e140 tx=0x7f4960001590 comp rx=0 tx=0).stop 2024-09-16T15:52:14.623 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:14.621+0000 7f4952ffd640 1 -- 172.21.15.123:0/499586395 shutdown_connections 2024-09-16T15:52:14.623 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:14.621+0000 7f4952ffd640 1 --2- 172.21.15.123:0/499586395 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f494c073ac0 0x7f494c075f80 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:52:14.623 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:14.625+0000 7f4952ffd640 1 --2- 172.21.15.123:0/499586395 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f496815d490 0x7f49680c3c10 unknown :-1 s=CLOSED pgs=1980 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:52:14.623 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:14.625+0000 7f4952ffd640 1 --2- 172.21.15.123:0/499586395 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f496815d010 0x7f49680c36d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:52:14.623 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:14.625+0000 7f4952ffd640 1 --2- 172.21.15.123:0/499586395 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f4958004d50 0x7f496815b520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:52:14.623 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:14.625+0000 7f4952ffd640 1 -- 172.21.15.123:0/499586395 >> 172.21.15.123:0/499586395 conn(0x7f49680cfb10 msgr2=0x7f49680c5860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:52:14.623 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:14.625+0000 7f4952ffd640 1 -- 172.21.15.123:0/499586395 shutdown_connections 2024-09-16T15:52:14.623 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:14.625+0000 7f4952ffd640 1 -- 172.21.15.123:0/499586395 wait complete. 2024-09-16T15:52:15.639 INFO:tasks.ceph:Scrubbing osd.0 2024-09-16T15:52:15.639 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 config set osd_debug_deep_scrub_sleep 0 2024-09-16T15:52:15.781 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:15.781+0000 7fe54ae3c640 1 -- 172.21.15.123:0/4278413001 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe52c003a40 msgr2=0x7fe52c003e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:52:15.781 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:15.781+0000 7fe54ae3c640 1 --2- 172.21.15.123:0/4278413001 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe52c003a40 0x7fe52c003e60 secure :-1 s=READY pgs=1982 cs=0 l=1 rev1=1 crypto rx=0x7fe53800bb10 tx=0x7fe53801cda0 comp rx=0 tx=0).stop 2024-09-16T15:52:15.782 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:15.781+0000 7fe5493d9640 1 -- 172.21.15.123:0/4278413001 reap_dead start 2024-09-16T15:52:15.782 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:15.781+0000 7fe54ae3c640 1 -- 172.21.15.123:0/4278413001 shutdown_connections 2024-09-16T15:52:15.782 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:15.781+0000 7fe54ae3c640 1 -- 172.21.15.123:0/4278413001 >> 172.21.15.123:0/4278413001 conn(0x7fe5440b1630 msgr2=0x7fe5440b1a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:52:15.782 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:15.781+0000 7fe54ae3c640 1 -- 172.21.15.123:0/4278413001 shutdown_connections 2024-09-16T15:52:15.782 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:15.781+0000 7fe54ae3c640 1 -- 172.21.15.123:0/4278413001 wait complete. 2024-09-16T15:52:15.782 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:15.785+0000 7fe54ae3c640 1 Processor -- start 2024-09-16T15:52:15.782 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:15.785+0000 7fe54ae3c640 1 -- start start 2024-09-16T15:52:15.785 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:15.785+0000 7fe54ae3c640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fe5441530b0 0x7fe5441f7dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:52:15.785 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:15.785+0000 7fe54ae3c640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe544153490 0x7fe5440aed90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:52:15.785 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:15.785+0000 7fe54ae3c640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe5441549b0 0x7fe544154e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:52:15.785 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:15.785+0000 7fe54ae3c640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fe544167df0 con 0x7fe5441549b0 2024-09-16T15:52:15.785 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:15.785+0000 7fe54ae3c640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fe544167c70 con 0x7fe544153490 2024-09-16T15:52:15.785 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:15.785+0000 7fe54ae3c640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fe544167f70 con 0x7fe5441530b0 2024-09-16T15:52:15.785 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:15.785+0000 7fe5493d9640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe5441549b0 0x7fe544154e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:52:15.785 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:15.785+0000 7fe5493d9640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe5441549b0 0x7fe544154e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:34120/0 (socket says 172.21.15.123:34120) 2024-09-16T15:52:15.785 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:15.785+0000 7fe5493d9640 1 -- 172.21.15.123:0/3800926194 learned_addr learned my addr 172.21.15.123:0/3800926194 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:52:15.785 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:15.785+0000 7fe543fff640 1 --2- 172.21.15.123:0/3800926194 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe544153490 0x7fe5440aed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:52:15.785 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:15.785+0000 7fe548bd8640 1 --2- 172.21.15.123:0/3800926194 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fe5441530b0 0x7fe5441f7dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:52:15.785 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:15.785+0000 7fe5493d9640 1 -- 172.21.15.123:0/3800926194 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fe5441530b0 msgr2=0x7fe5441f7dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:52:15.786 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:15.785+0000 7fe5493d9640 1 --2- 172.21.15.123:0/3800926194 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fe5441530b0 0x7fe5441f7dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:52:15.786 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:15.785+0000 7fe5493d9640 1 -- 172.21.15.123:0/3800926194 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe544153490 msgr2=0x7fe5440aed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:52:15.786 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:15.785+0000 7fe5493d9640 1 --2- 172.21.15.123:0/3800926194 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fe544153490 0x7fe5440aed90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:52:15.786 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:15.785+0000 7fe5493d9640 1 -- 172.21.15.123:0/3800926194 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe5440b31a0 con 0x7fe5441549b0 2024-09-16T15:52:15.786 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:15.785+0000 7fe5493d9640 1 --2- 172.21.15.123:0/3800926194 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe5441549b0 0x7fe544154e60 secure :-1 s=READY pgs=1983 cs=0 l=1 rev1=1 crypto rx=0x7fe54406de90 tx=0x7fe53c00dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:52:15.786 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:15.785+0000 7fe5417fa640 1 -- 172.21.15.123:0/3800926194 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe53c00b8a0 con 0x7fe5441549b0 2024-09-16T15:52:15.786 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:15.785+0000 7fe5417fa640 1 -- 172.21.15.123:0/3800926194 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fe53c018070 con 0x7fe5441549b0 2024-09-16T15:52:15.786 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:15.785+0000 7fe54ae3c640 1 -- 172.21.15.123:0/3800926194 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe5441f8470 con 0x7fe5441549b0 2024-09-16T15:52:15.786 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:15.785+0000 7fe5417fa640 1 -- 172.21.15.123:0/3800926194 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe53c00bd00 con 0x7fe5441549b0 2024-09-16T15:52:15.786 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:15.785+0000 7fe54ae3c640 1 -- 172.21.15.123:0/3800926194 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe5441f8a30 con 0x7fe5441549b0 2024-09-16T15:52:15.786 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:15.785+0000 7fe54ae3c640 1 -- 172.21.15.123:0/3800926194 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fe54415f670 con 0x7fe5441549b0 2024-09-16T15:52:15.788 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:15.785+0000 7fe5417fa640 1 -- 172.21.15.123:0/3800926194 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 27) ==== 98151+0+0 (secure 0 0 0) 0x7fe53c01d460 con 0x7fe5441549b0 2024-09-16T15:52:15.788 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:15.789+0000 7fe5417fa640 1 --2- 172.21.15.123:0/3800926194 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fe530073ac0 0x7fe530075f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:52:15.788 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:15.789+0000 7fe5417fa640 1 -- 172.21.15.123:0/3800926194 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(172..172 src has 1..172) ==== 4971+0+0 (secure 0 0 0) 0x7fe53c0930e0 con 0x7fe5441549b0 2024-09-16T15:52:15.788 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:15.789+0000 7fe548bd8640 1 --2- 172.21.15.123:0/3800926194 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fe530073ac0 0x7fe530075f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:52:15.788 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:15.789+0000 7fe5417fa640 1 --2- 172.21.15.123:0/3800926194 >> [v2:172.21.15.123:6800/3882044969,v1:172.21.15.123:6801/3882044969] conn(0x7fe530079980 0x7fe53007bda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:52:15.788 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:15.789+0000 7fe5417fa640 1 -- 172.21.15.123:0/3800926194 --> [v2:172.21.15.123:6800/3882044969,v1:172.21.15.123:6801/3882044969] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fe53c014ec0 con 0x7fe530079980 2024-09-16T15:52:15.788 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:15.789+0000 7fe5417fa640 1 -- 172.21.15.123:0/3800926194 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_get_version_reply(handle=1 version=172) ==== 24+0+0 (secure 0 0 0) 0x7fe53c095a50 con 0x7fe5441549b0 2024-09-16T15:52:15.788 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:15.789+0000 7fe543fff640 1 --2- 172.21.15.123:0/3800926194 >> [v2:172.21.15.123:6800/3882044969,v1:172.21.15.123:6801/3882044969] conn(0x7fe530079980 0x7fe53007bda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:52:15.788 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:15.789+0000 7fe548bd8640 1 --2- 172.21.15.123:0/3800926194 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fe530073ac0 0x7fe530075f80 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7fe5380b0000 tx=0x7fe5380a7040 comp rx=0 tx=0).ready entity=mgr.4150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:52:15.788 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:15.789+0000 7fe543fff640 1 --2- 172.21.15.123:0/3800926194 >> [v2:172.21.15.123:6800/3882044969,v1:172.21.15.123:6801/3882044969] conn(0x7fe530079980 0x7fe53007bda0 crc :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:52:15.788 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:15.789+0000 7fe5417fa640 1 -- 172.21.15.123:0/3800926194 <== osd.0 v2:172.21.15.123:6800/3882044969 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fe53c014ec0 con 0x7fe530079980 2024-09-16T15:52:15.801 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:15.801+0000 7fe54ae3c640 1 -- 172.21.15.123:0/3800926194 --> [v2:172.21.15.123:6800/3882044969,v1:172.21.15.123:6801/3882044969] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7fe544074220 con 0x7fe530079980 2024-09-16T15:52:15.801 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:15.801+0000 7fe5417fa640 1 -- 172.21.15.123:0/3800926194 <== osd.0 v2:172.21.15.123:6800/3882044969 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7fe544074220 con 0x7fe530079980 2024-09-16T15:52:15.801 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2024-09-16T15:52:15.801 INFO:teuthology.orchestra.run.smithi123.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-16T15:52:15.801 INFO:teuthology.orchestra.run.smithi123.stdout:} 2024-09-16T15:52:15.802 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:15.801+0000 7fe54ae3c640 1 -- 172.21.15.123:0/3800926194 >> [v2:172.21.15.123:6800/3882044969,v1:172.21.15.123:6801/3882044969] conn(0x7fe530079980 msgr2=0x7fe53007bda0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:52:15.802 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:15.801+0000 7fe54ae3c640 1 --2- 172.21.15.123:0/3800926194 >> [v2:172.21.15.123:6800/3882044969,v1:172.21.15.123:6801/3882044969] conn(0x7fe530079980 0x7fe53007bda0 crc :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:52:15.802 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:15.801+0000 7fe54ae3c640 1 -- 172.21.15.123:0/3800926194 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fe530073ac0 msgr2=0x7fe530075f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:52:15.803 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:15.801+0000 7fe54ae3c640 1 --2- 172.21.15.123:0/3800926194 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fe530073ac0 0x7fe530075f80 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7fe5380b0000 tx=0x7fe5380a7040 comp rx=0 tx=0).stop 2024-09-16T15:52:15.803 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:15.801+0000 7fe54ae3c640 1 -- 172.21.15.123:0/3800926194 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe5441549b0 msgr2=0x7fe544154e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:52:15.803 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:15.801+0000 7fe54ae3c640 1 --2- 172.21.15.123:0/3800926194 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe5441549b0 0x7fe544154e60 secure :-1 s=READY pgs=1983 cs=0 l=1 rev1=1 crypto rx=0x7fe54406de90 tx=0x7fe53c00dc90 comp rx=0 tx=0).stop 2024-09-16T15:52:15.803 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:15.801+0000 7fe5493d9640 1 -- 172.21.15.123:0/3800926194 reap_dead start 2024-09-16T15:52:15.803 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:15.801+0000 7fe54ae3c640 1 -- 172.21.15.123:0/3800926194 shutdown_connections 2024-09-16T15:52:15.803 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:15.801+0000 7fe54ae3c640 1 -- 172.21.15.123:0/3800926194 >> 172.21.15.123:0/3800926194 conn(0x7fe5440b1630 msgr2=0x7fe54406e800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:52:15.803 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:15.801+0000 7fe54ae3c640 1 -- 172.21.15.123:0/3800926194 shutdown_connections 2024-09-16T15:52:15.803 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:15.801+0000 7fe54ae3c640 1 -- 172.21.15.123:0/3800926194 wait complete. 2024-09-16T15:52:15.817 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 0 2024-09-16T15:52:15.936 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:15.937+0000 7ff5a3a8b640 1 -- 172.21.15.123:0/3712240558 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff59c074470 msgr2=0x7ff59c0cd6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:52:15.937 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:15.937+0000 7ff5a3a8b640 1 --2- 172.21.15.123:0/3712240558 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff59c074470 0x7ff59c0cd6e0 secure :-1 s=READY pgs=1984 cs=0 l=1 rev1=1 crypto rx=0x7ff59c0c3570 tx=0x7ff58c01c570 comp rx=0 tx=0).stop 2024-09-16T15:52:15.937 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:15.937+0000 7ff5a3a8b640 1 -- 172.21.15.123:0/3712240558 shutdown_connections 2024-09-16T15:52:15.937 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:15.937+0000 7ff5a3a8b640 1 --2- 172.21.15.123:0/3712240558 >> v2:172.21.15.184:3300/0 conn(0x7ff59c0cdc20 0x7ff59c0ca860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:52:15.937 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:15.937+0000 7ff5a3a8b640 1 --2- 172.21.15.123:0/3712240558 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff59c074470 0x7ff59c0cd6e0 unknown :-1 s=CLOSED pgs=1984 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:52:15.937 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:15.937+0000 7ff5a3a8b640 1 --2- 172.21.15.123:0/3712240558 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7ff59c15e160 0x7ff59c073f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:52:15.937 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:15.937+0000 7ff5a3a8b640 1 -- 172.21.15.123:0/3712240558 >> 172.21.15.123:0/3712240558 conn(0x7ff59c0cfb10 msgr2=0x7ff59c0cff20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:52:15.937 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:15.937+0000 7ff5a3a8b640 1 -- 172.21.15.123:0/3712240558 shutdown_connections 2024-09-16T15:52:15.937 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:15.937+0000 7ff5a3a8b640 1 -- 172.21.15.123:0/3712240558 wait complete. 2024-09-16T15:52:15.938 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:15.937+0000 7ff5a3a8b640 1 Processor -- start 2024-09-16T15:52:15.938 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:15.937+0000 7ff5a3a8b640 1 -- start start 2024-09-16T15:52:15.940 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:15.941+0000 7ff5a3a8b640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff59c074470 0x7ff59c151ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:52:15.940 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:15.941+0000 7ff5a3a8b640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff59c0cdc20 0x7ff59c150110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:52:15.940 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:15.941+0000 7ff5a3a8b640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7ff59c15e160 0x7ff59c158780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:52:15.940 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:15.941+0000 7ff5a3a8b640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7ff59c15ebb0 con 0x7ff59c074470 2024-09-16T15:52:15.940 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:15.941+0000 7ff5a3a8b640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7ff59c15ea30 con 0x7ff59c0cdc20 2024-09-16T15:52:15.940 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:15.941+0000 7ff5a3a8b640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7ff59c15ed30 con 0x7ff59c15e160 2024-09-16T15:52:15.940 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:15.941+0000 7ff5a1827640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff59c074470 0x7ff59c151ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:52:15.940 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:15.941+0000 7ff5a1827640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff59c074470 0x7ff59c151ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:34128/0 (socket says 172.21.15.123:34128) 2024-09-16T15:52:15.940 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:15.941+0000 7ff5a1827640 1 -- 172.21.15.123:0/675567054 learned_addr learned my addr 172.21.15.123:0/675567054 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:52:15.941 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:15.941+0000 7ff5a2028640 1 --2- 172.21.15.123:0/675567054 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7ff59c15e160 0x7ff59c158780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:52:15.941 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:15.941+0000 7ff5a1026640 1 --2- 172.21.15.123:0/675567054 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff59c0cdc20 0x7ff59c150110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:52:15.941 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:15.941+0000 7ff5a1827640 1 -- 172.21.15.123:0/675567054 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7ff59c15e160 msgr2=0x7ff59c158780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:52:15.941 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:15.941+0000 7ff5a1827640 1 --2- 172.21.15.123:0/675567054 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7ff59c15e160 0x7ff59c158780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:52:15.941 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:15.941+0000 7ff5a1827640 1 -- 172.21.15.123:0/675567054 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff59c0cdc20 msgr2=0x7ff59c150110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:52:15.941 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:15.941+0000 7ff5a1827640 1 --2- 172.21.15.123:0/675567054 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff59c0cdc20 0x7ff59c150110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:52:15.941 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:15.941+0000 7ff5a1827640 1 -- 172.21.15.123:0/675567054 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff59c153170 con 0x7ff59c074470 2024-09-16T15:52:15.941 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:15.941+0000 7ff5a1827640 1 --2- 172.21.15.123:0/675567054 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff59c074470 0x7ff59c151ba0 secure :-1 s=READY pgs=1985 cs=0 l=1 rev1=1 crypto rx=0x7ff59800d720 tx=0x7ff59800dbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:52:15.941 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:15.941+0000 7ff5927fc640 1 -- 172.21.15.123:0/675567054 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff598017070 con 0x7ff59c074470 2024-09-16T15:52:15.941 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:15.941+0000 7ff5927fc640 1 -- 172.21.15.123:0/675567054 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7ff598009d20 con 0x7ff59c074470 2024-09-16T15:52:15.941 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:15.941+0000 7ff5a3a8b640 1 -- 172.21.15.123:0/675567054 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff59c0b38a0 con 0x7ff59c074470 2024-09-16T15:52:15.941 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:15.941+0000 7ff5927fc640 1 -- 172.21.15.123:0/675567054 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff598005280 con 0x7ff59c074470 2024-09-16T15:52:15.941 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:15.941+0000 7ff5a3a8b640 1 -- 172.21.15.123:0/675567054 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff59c1506b0 con 0x7ff59c074470 2024-09-16T15:52:15.942 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:15.941+0000 7ff5a3a8b640 1 -- 172.21.15.123:0/675567054 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff59c0ca860 con 0x7ff59c074470 2024-09-16T15:52:15.942 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:15.941+0000 7ff5927fc640 1 -- 172.21.15.123:0/675567054 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 27) ==== 98151+0+0 (secure 0 0 0) 0x7ff598005420 con 0x7ff59c074470 2024-09-16T15:52:15.942 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:15.941+0000 7ff5927fc640 1 --2- 172.21.15.123:0/675567054 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7ff580073a90 0x7ff580075f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:52:15.942 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:15.941+0000 7ff5a1026640 1 --2- 172.21.15.123:0/675567054 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7ff580073a90 0x7ff580075f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:52:15.942 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:15.941+0000 7ff5927fc640 1 -- 172.21.15.123:0/675567054 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(172..172 src has 1..172) ==== 4971+0+0 (secure 0 0 0) 0x7ff598092aa0 con 0x7ff59c074470 2024-09-16T15:52:15.943 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:15.945+0000 7ff5a1026640 1 --2- 172.21.15.123:0/675567054 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7ff580073a90 0x7ff580075f50 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7ff58c0ae000 tx=0x7ff58c01adb0 comp rx=0 tx=0).ready entity=mgr.4150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:52:15.946 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:15.949+0000 7ff5927fc640 1 -- 172.21.15.123:0/675567054 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7ff59805f540 con 0x7ff59c074470 2024-09-16T15:52:16.113 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:16.113+0000 7ff5a3a8b640 1 -- 172.21.15.123:0/675567054 --> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "0", "target": ["mon-mgr", ""]}) -- 0x7ff59c0d06c0 con 0x7ff580073a90 2024-09-16T15:52:16.113 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:16.113+0000 7ff5927fc640 1 -- 172.21.15.123:0/675567054 <== mgr.4150 v2:172.21.15.123:6836/2464759676 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 0 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7ff59c0d06c0 con 0x7ff580073a90 2024-09-16T15:52:16.113 INFO:teuthology.orchestra.run.smithi123.stderr:instructed osd(s) 0 to deep-scrub 2024-09-16T15:52:16.115 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:16.117+0000 7ff5a3a8b640 1 -- 172.21.15.123:0/675567054 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7ff580073a90 msgr2=0x7ff580075f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:52:16.115 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:16.117+0000 7ff5a3a8b640 1 --2- 172.21.15.123:0/675567054 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7ff580073a90 0x7ff580075f50 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7ff58c0ae000 tx=0x7ff58c01adb0 comp rx=0 tx=0).stop 2024-09-16T15:52:16.115 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:16.117+0000 7ff5a3a8b640 1 -- 172.21.15.123:0/675567054 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff59c074470 msgr2=0x7ff59c151ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:52:16.116 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:16.117+0000 7ff5a3a8b640 1 --2- 172.21.15.123:0/675567054 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff59c074470 0x7ff59c151ba0 secure :-1 s=READY pgs=1985 cs=0 l=1 rev1=1 crypto rx=0x7ff59800d720 tx=0x7ff59800dbe0 comp rx=0 tx=0).stop 2024-09-16T15:52:16.116 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:16.117+0000 7ff5a3a8b640 1 -- 172.21.15.123:0/675567054 shutdown_connections 2024-09-16T15:52:16.116 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:16.117+0000 7ff5a3a8b640 1 --2- 172.21.15.123:0/675567054 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7ff580073a90 0x7ff580075f50 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:52:16.116 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:16.117+0000 7ff5a3a8b640 1 --2- 172.21.15.123:0/675567054 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7ff59c15e160 0x7ff59c158780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:52:16.116 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:16.117+0000 7ff5a3a8b640 1 --2- 172.21.15.123:0/675567054 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff59c0cdc20 0x7ff59c150110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:52:16.116 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:16.117+0000 7ff5a3a8b640 1 --2- 172.21.15.123:0/675567054 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff59c074470 0x7ff59c151ba0 unknown :-1 s=CLOSED pgs=1985 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:52:16.116 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:16.117+0000 7ff5a3a8b640 1 -- 172.21.15.123:0/675567054 >> 172.21.15.123:0/675567054 conn(0x7ff59c0cfb10 msgr2=0x7ff59c0b0a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:52:16.116 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:16.117+0000 7ff5a3a8b640 1 -- 172.21.15.123:0/675567054 shutdown_connections 2024-09-16T15:52:16.116 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:16.117+0000 7ff5a3a8b640 1 -- 172.21.15.123:0/675567054 wait complete. 2024-09-16T15:52:16.131 INFO:tasks.ceph:Scrubbing osd.1 2024-09-16T15:52:16.131 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 config set osd_debug_deep_scrub_sleep 0 2024-09-16T15:52:16.248 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:16.249+0000 7f7fbba19640 1 -- 172.21.15.123:0/1123609841 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7fb4160310 msgr2=0x7f7fb4070470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:52:16.248 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:16.249+0000 7f7fbba19640 1 --2- 172.21.15.123:0/1123609841 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7fb4160310 0x7f7fb4070470 secure :-1 s=READY pgs=1987 cs=0 l=1 rev1=1 crypto rx=0x7f7fb40cf9f0 tx=0x7f7fa401c770 comp rx=0 tx=0).stop 2024-09-16T15:52:16.248 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:16.249+0000 7f7fbba19640 1 -- 172.21.15.123:0/1123609841 shutdown_connections 2024-09-16T15:52:16.248 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:16.249+0000 7f7fbba19640 1 --2- 172.21.15.123:0/1123609841 >> v2:172.21.15.184:3300/0 conn(0x7f7fb415f690 0x7f7fb415fb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:52:16.248 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:16.249+0000 7f7fbba19640 1 --2- 172.21.15.123:0/1123609841 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7fb4160310 0x7f7fb4070470 unknown :-1 s=CLOSED pgs=1987 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:52:16.248 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:16.249+0000 7f7fbba19640 1 -- 172.21.15.123:0/1123609841 >> 172.21.15.123:0/1123609841 conn(0x7f7fb40b1630 msgr2=0x7f7fb40b1a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:52:16.248 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:16.249+0000 7f7fbba19640 1 -- 172.21.15.123:0/1123609841 shutdown_connections 2024-09-16T15:52:16.249 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:16.249+0000 7f7fbba19640 1 -- 172.21.15.123:0/1123609841 wait complete. 2024-09-16T15:52:16.249 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:16.249+0000 7f7fbba19640 1 Processor -- start 2024-09-16T15:52:16.249 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:16.249+0000 7f7fbba19640 1 -- start start 2024-09-16T15:52:16.251 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:16.249+0000 7f7fbba19640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7fb415bc40 0x7f7fb414bab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:52:16.251 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:16.249+0000 7f7fbba19640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7fb415f690 0x7f7fb4164bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:52:16.251 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:16.249+0000 7f7fbba19640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f7fb4160310 0x7f7fb414aed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:52:16.251 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:16.249+0000 7f7fbba19640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f7fb4168c40 con 0x7f7fb415bc40 2024-09-16T15:52:16.251 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:16.249+0000 7f7fbba19640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f7fb40cf430 con 0x7f7fb415f690 2024-09-16T15:52:16.252 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:16.249+0000 7f7fbba19640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f7fb4168dc0 con 0x7f7fb4160310 2024-09-16T15:52:16.252 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:16.249+0000 7f7fb97b5640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7fb415bc40 0x7f7fb414bab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:52:16.252 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:16.249+0000 7f7fb97b5640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7fb415bc40 0x7f7fb414bab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:34144/0 (socket says 172.21.15.123:34144) 2024-09-16T15:52:16.252 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:16.249+0000 7f7fb97b5640 1 -- 172.21.15.123:0/1424853444 learned_addr learned my addr 172.21.15.123:0/1424853444 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:52:16.252 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:16.249+0000 7f7fb8fb4640 1 --2- 172.21.15.123:0/1424853444 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7fb415f690 0x7f7fb4164bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:52:16.252 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:16.249+0000 7f7fb9fb6640 1 --2- 172.21.15.123:0/1424853444 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f7fb4160310 0x7f7fb414aed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:52:16.252 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:16.253+0000 7f7fb97b5640 1 -- 172.21.15.123:0/1424853444 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f7fb4160310 msgr2=0x7f7fb414aed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:52:16.252 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:16.253+0000 7f7fb97b5640 1 --2- 172.21.15.123:0/1424853444 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f7fb4160310 0x7f7fb414aed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:52:16.252 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:16.253+0000 7f7fb97b5640 1 -- 172.21.15.123:0/1424853444 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7fb415f690 msgr2=0x7f7fb4164bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:52:16.252 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:16.253+0000 7f7fb97b5640 1 --2- 172.21.15.123:0/1424853444 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7fb415f690 0x7f7fb4164bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:52:16.253 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:16.253+0000 7f7fb97b5640 1 -- 172.21.15.123:0/1424853444 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7fb4157420 con 0x7f7fb415bc40 2024-09-16T15:52:16.253 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:16.253+0000 7f7fb97b5640 1 --2- 172.21.15.123:0/1424853444 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7fb415bc40 0x7f7fb414bab0 secure :-1 s=READY pgs=1988 cs=0 l=1 rev1=1 crypto rx=0x7f7fb000c480 tx=0x7f7fb000c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:52:16.253 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:16.253+0000 7f7fa9ffb640 1 -- 172.21.15.123:0/1424853444 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7fb0016020 con 0x7f7fb415bc40 2024-09-16T15:52:16.253 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:16.253+0000 7f7fa9ffb640 1 -- 172.21.15.123:0/1424853444 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f7fb0012920 con 0x7f7fb415bc40 2024-09-16T15:52:16.253 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:16.253+0000 7f7fbba19640 1 -- 172.21.15.123:0/1424853444 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7fb4157140 con 0x7f7fb415bc40 2024-09-16T15:52:16.253 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:16.253+0000 7f7fa9ffb640 1 -- 172.21.15.123:0/1424853444 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7fb0012d70 con 0x7f7fb415bc40 2024-09-16T15:52:16.253 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:16.253+0000 7f7fbba19640 1 -- 172.21.15.123:0/1424853444 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7fb4157bd0 con 0x7f7fb415bc40 2024-09-16T15:52:16.253 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:16.253+0000 7f7fbba19640 1 -- 172.21.15.123:0/1424853444 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f7fb4156ef0 con 0x7f7fb415bc40 2024-09-16T15:52:16.254 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:16.253+0000 7f7fa9ffb640 1 -- 172.21.15.123:0/1424853444 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 27) ==== 98151+0+0 (secure 0 0 0) 0x7f7fb0019020 con 0x7f7fb415bc40 2024-09-16T15:52:16.254 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:16.253+0000 7f7fa9ffb640 1 --2- 172.21.15.123:0/1424853444 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f7f98073a90 0x7f7f98075f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:52:16.254 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:16.253+0000 7f7fb8fb4640 1 --2- 172.21.15.123:0/1424853444 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f7f98073a90 0x7f7f98075f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:52:16.254 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:16.253+0000 7f7fa9ffb640 1 -- 172.21.15.123:0/1424853444 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(172..172 src has 1..172) ==== 4971+0+0 (secure 0 0 0) 0x7f7fb0093090 con 0x7f7fb415bc40 2024-09-16T15:52:16.254 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:16.253+0000 7f7fa9ffb640 1 --2- 172.21.15.123:0/1424853444 >> [v2:172.21.15.123:6816/4037001850,v1:172.21.15.123:6817/4037001850] conn(0x7f7f98079950 0x7f7f9807bd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:52:16.255 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:16.253+0000 7f7fa9ffb640 1 -- 172.21.15.123:0/1424853444 --> [v2:172.21.15.123:6816/4037001850,v1:172.21.15.123:6817/4037001850] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f7fb000ae50 con 0x7f7f98079950 2024-09-16T15:52:16.255 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:16.253+0000 7f7fa9ffb640 1 -- 172.21.15.123:0/1424853444 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_get_version_reply(handle=1 version=172) ==== 24+0+0 (secure 0 0 0) 0x7f7fb0095970 con 0x7f7fb415bc40 2024-09-16T15:52:16.255 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:16.253+0000 7f7fb9fb6640 1 --2- 172.21.15.123:0/1424853444 >> [v2:172.21.15.123:6816/4037001850,v1:172.21.15.123:6817/4037001850] conn(0x7f7f98079950 0x7f7f9807bd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:52:16.255 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:16.253+0000 7f7fb8fb4640 1 --2- 172.21.15.123:0/1424853444 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f7f98073a90 0x7f7f98075f50 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f7fa40af000 tx=0x7f7fa401afb0 comp rx=0 tx=0).ready entity=mgr.4150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:52:16.255 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:16.253+0000 7f7fb9fb6640 1 --2- 172.21.15.123:0/1424853444 >> [v2:172.21.15.123:6816/4037001850,v1:172.21.15.123:6817/4037001850] conn(0x7f7f98079950 0x7f7f9807bd70 crc :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:52:16.255 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:16.253+0000 7f7fa9ffb640 1 -- 172.21.15.123:0/1424853444 <== osd.1 v2:172.21.15.123:6816/4037001850 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f7fb000ae50 con 0x7f7f98079950 2024-09-16T15:52:16.266 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:16.265+0000 7f7fbba19640 1 -- 172.21.15.123:0/1424853444 --> [v2:172.21.15.123:6816/4037001850,v1:172.21.15.123:6817/4037001850] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f7fb415fb20 con 0x7f7f98079950 2024-09-16T15:52:16.266 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:16.269+0000 7f7fa9ffb640 1 -- 172.21.15.123:0/1424853444 <== osd.1 v2:172.21.15.123:6816/4037001850 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7f7fb415fb20 con 0x7f7f98079950 2024-09-16T15:52:16.266 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2024-09-16T15:52:16.267 INFO:teuthology.orchestra.run.smithi123.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-16T15:52:16.267 INFO:teuthology.orchestra.run.smithi123.stdout:} 2024-09-16T15:52:16.267 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:16.269+0000 7f7fbba19640 1 -- 172.21.15.123:0/1424853444 >> [v2:172.21.15.123:6816/4037001850,v1:172.21.15.123:6817/4037001850] conn(0x7f7f98079950 msgr2=0x7f7f9807bd70 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:52:16.267 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:16.269+0000 7f7fbba19640 1 --2- 172.21.15.123:0/1424853444 >> [v2:172.21.15.123:6816/4037001850,v1:172.21.15.123:6817/4037001850] conn(0x7f7f98079950 0x7f7f9807bd70 crc :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:52:16.267 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:16.269+0000 7f7fbba19640 1 -- 172.21.15.123:0/1424853444 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f7f98073a90 msgr2=0x7f7f98075f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:52:16.267 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:16.269+0000 7f7fbba19640 1 --2- 172.21.15.123:0/1424853444 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f7f98073a90 0x7f7f98075f50 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f7fa40af000 tx=0x7f7fa401afb0 comp rx=0 tx=0).stop 2024-09-16T15:52:16.268 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:16.269+0000 7f7fbba19640 1 -- 172.21.15.123:0/1424853444 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7fb415bc40 msgr2=0x7f7fb414bab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:52:16.268 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:16.269+0000 7f7fbba19640 1 --2- 172.21.15.123:0/1424853444 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7fb415bc40 0x7f7fb414bab0 secure :-1 s=READY pgs=1988 cs=0 l=1 rev1=1 crypto rx=0x7f7fb000c480 tx=0x7f7fb000c940 comp rx=0 tx=0).stop 2024-09-16T15:52:16.268 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:16.269+0000 7f7fb9fb6640 1 -- 172.21.15.123:0/1424853444 reap_dead start 2024-09-16T15:52:16.268 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:16.269+0000 7f7fbba19640 1 -- 172.21.15.123:0/1424853444 shutdown_connections 2024-09-16T15:52:16.268 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:16.269+0000 7f7fbba19640 1 -- 172.21.15.123:0/1424853444 >> 172.21.15.123:0/1424853444 conn(0x7f7fb40b1630 msgr2=0x7f7fb415e5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:52:16.268 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:16.269+0000 7f7fbba19640 1 -- 172.21.15.123:0/1424853444 shutdown_connections 2024-09-16T15:52:16.268 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:16.269+0000 7f7fbba19640 1 -- 172.21.15.123:0/1424853444 wait complete. 2024-09-16T15:52:16.281 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 1 2024-09-16T15:52:16.392 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:16.393+0000 7f5bc0911640 1 -- 172.21.15.123:0/3575060334 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5bbc06f5f0 msgr2=0x7f5bbc15aa60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:52:16.392 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:16.393+0000 7f5bc0911640 1 --2- 172.21.15.123:0/3575060334 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5bbc06f5f0 0x7f5bbc15aa60 secure :-1 s=READY pgs=1990 cs=0 l=1 rev1=1 crypto rx=0x7f5bb4009bc0 tx=0x7f5bb401cb10 comp rx=0 tx=0).stop 2024-09-16T15:52:16.392 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:16.393+0000 7f5bc0911640 1 -- 172.21.15.123:0/3575060334 shutdown_connections 2024-09-16T15:52:16.392 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:16.393+0000 7f5bc0911640 1 --2- 172.21.15.123:0/3575060334 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5bbc06f5f0 0x7f5bbc15aa60 unknown :-1 s=CLOSED pgs=1990 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:52:16.392 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:16.393+0000 7f5bc0911640 1 --2- 172.21.15.123:0/3575060334 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f5bbc158120 0x7f5bbc0ca320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:52:16.392 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:16.393+0000 7f5bc0911640 1 -- 172.21.15.123:0/3575060334 >> 172.21.15.123:0/3575060334 conn(0x7f5bbc0cfb10 msgr2=0x7f5bbc0cff20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:52:16.392 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:16.393+0000 7f5bc0911640 1 -- 172.21.15.123:0/3575060334 shutdown_connections 2024-09-16T15:52:16.392 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:16.393+0000 7f5bc0911640 1 -- 172.21.15.123:0/3575060334 wait complete. 2024-09-16T15:52:16.394 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:16.393+0000 7f5bc0911640 1 Processor -- start 2024-09-16T15:52:16.394 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:16.393+0000 7f5bc0911640 1 -- start start 2024-09-16T15:52:16.395 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:16.393+0000 7f5bc0911640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f5bbc06f5f0 0x7f5bbc155ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:52:16.395 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:16.393+0000 7f5bc0911640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5bbc0cf5a0 0x7f5bbc0c6be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:52:16.395 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:16.393+0000 7f5bc0911640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5bbc158120 0x7f5bbc0c8770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:52:16.395 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:16.393+0000 7f5bc0911640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f5bbc168660 con 0x7f5bbc158120 2024-09-16T15:52:16.395 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:16.393+0000 7f5bc0911640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f5bbc1684e0 con 0x7f5bbc0cf5a0 2024-09-16T15:52:16.395 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:16.393+0000 7f5bc0911640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f5bbc1687e0 con 0x7f5bbc06f5f0 2024-09-16T15:52:16.395 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:16.393+0000 7f5bbad9d640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5bbc158120 0x7f5bbc0c8770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:52:16.395 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:16.393+0000 7f5bbad9d640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5bbc158120 0x7f5bbc0c8770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:34160/0 (socket says 172.21.15.123:34160) 2024-09-16T15:52:16.395 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:16.393+0000 7f5bbad9d640 1 -- 172.21.15.123:0/3427250857 learned_addr learned my addr 172.21.15.123:0/3427250857 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:52:16.396 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:16.393+0000 7f5bbad9d640 1 -- 172.21.15.123:0/3427250857 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f5bbc06f5f0 msgr2=0x7f5bbc155ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:52:16.396 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:16.393+0000 7f5bbad9d640 1 --2- 172.21.15.123:0/3427250857 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f5bbc06f5f0 0x7f5bbc155ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:52:16.396 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:16.393+0000 7f5bbad9d640 1 -- 172.21.15.123:0/3427250857 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5bbc0cf5a0 msgr2=0x7f5bbc0c6be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:52:16.396 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:16.393+0000 7f5bbad9d640 1 --2- 172.21.15.123:0/3427250857 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5bbc0cf5a0 0x7f5bbc0c6be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:52:16.396 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:16.393+0000 7f5bbad9d640 1 -- 172.21.15.123:0/3427250857 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5bbc075df0 con 0x7f5bbc158120 2024-09-16T15:52:16.396 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:16.393+0000 7f5bbad9d640 1 --2- 172.21.15.123:0/3427250857 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5bbc158120 0x7f5bbc0c8770 secure :-1 s=READY pgs=1991 cs=0 l=1 rev1=1 crypto rx=0x7f5bb401a740 tx=0x7f5bb40a5960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:52:16.396 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:16.393+0000 7f5baaffd640 1 -- 172.21.15.123:0/3427250857 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5bb40bb070 con 0x7f5bbc158120 2024-09-16T15:52:16.396 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:16.397+0000 7f5baaffd640 1 -- 172.21.15.123:0/3427250857 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f5bb4004e00 con 0x7f5bbc158120 2024-09-16T15:52:16.396 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:16.397+0000 7f5bc0911640 1 -- 172.21.15.123:0/3427250857 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5bbc0c62c0 con 0x7f5bbc158120 2024-09-16T15:52:16.396 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:16.397+0000 7f5baaffd640 1 -- 172.21.15.123:0/3427250857 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5bb40b2040 con 0x7f5bbc158120 2024-09-16T15:52:16.396 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:16.397+0000 7f5bc0911640 1 -- 172.21.15.123:0/3427250857 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5bbc0c7120 con 0x7f5bbc158120 2024-09-16T15:52:16.397 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:16.397+0000 7f5bc0911640 1 -- 172.21.15.123:0/3427250857 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5bbc15aa60 con 0x7f5bbc158120 2024-09-16T15:52:16.397 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:16.397+0000 7f5baaffd640 1 -- 172.21.15.123:0/3427250857 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 27) ==== 98151+0+0 (secure 0 0 0) 0x7f5bb40be070 con 0x7f5bbc158120 2024-09-16T15:52:16.397 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:16.397+0000 7f5baaffd640 1 --2- 172.21.15.123:0/3427250857 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f5b9c073a90 0x7f5b9c075f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:52:16.397 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:16.397+0000 7f5baaffd640 1 -- 172.21.15.123:0/3427250857 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(172..172 src has 1..172) ==== 4971+0+0 (secure 0 0 0) 0x7f5bb412c9f0 con 0x7f5bbc158120 2024-09-16T15:52:16.398 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:16.397+0000 7f5bba59c640 1 --2- 172.21.15.123:0/3427250857 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f5b9c073a90 0x7f5b9c075f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:52:16.398 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:16.397+0000 7f5bba59c640 1 --2- 172.21.15.123:0/3427250857 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f5b9c073a90 0x7f5b9c075f50 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f5bb000a910 tx=0x7f5bb0005be0 comp rx=0 tx=0).ready entity=mgr.4150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:52:16.400 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:16.401+0000 7f5baaffd640 1 -- 172.21.15.123:0/3427250857 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f5bb4016790 con 0x7f5bbc158120 2024-09-16T15:52:16.567 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:16.569+0000 7f5bc0911640 1 -- 172.21.15.123:0/3427250857 --> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "1", "target": ["mon-mgr", ""]}) -- 0x7f5bbc0c3350 con 0x7f5b9c073a90 2024-09-16T15:52:16.567 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:16.569+0000 7f5baaffd640 1 -- 172.21.15.123:0/3427250857 <== mgr.4150 v2:172.21.15.123:6836/2464759676 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 1 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7f5bbc0c3350 con 0x7f5b9c073a90 2024-09-16T15:52:16.567 INFO:teuthology.orchestra.run.smithi123.stderr:instructed osd(s) 1 to deep-scrub 2024-09-16T15:52:16.570 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:16.569+0000 7f5bc0911640 1 -- 172.21.15.123:0/3427250857 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f5b9c073a90 msgr2=0x7f5b9c075f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:52:16.570 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:16.569+0000 7f5bc0911640 1 --2- 172.21.15.123:0/3427250857 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f5b9c073a90 0x7f5b9c075f50 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f5bb000a910 tx=0x7f5bb0005be0 comp rx=0 tx=0).stop 2024-09-16T15:52:16.570 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:16.569+0000 7f5bc0911640 1 -- 172.21.15.123:0/3427250857 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5bbc158120 msgr2=0x7f5bbc0c8770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:52:16.570 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:16.569+0000 7f5bc0911640 1 --2- 172.21.15.123:0/3427250857 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5bbc158120 0x7f5bbc0c8770 secure :-1 s=READY pgs=1991 cs=0 l=1 rev1=1 crypto rx=0x7f5bb401a740 tx=0x7f5bb40a5960 comp rx=0 tx=0).stop 2024-09-16T15:52:16.570 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:16.569+0000 7f5bc0911640 1 -- 172.21.15.123:0/3427250857 shutdown_connections 2024-09-16T15:52:16.570 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:16.569+0000 7f5bc0911640 1 --2- 172.21.15.123:0/3427250857 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f5b9c073a90 0x7f5b9c075f50 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:52:16.570 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:16.569+0000 7f5bc0911640 1 --2- 172.21.15.123:0/3427250857 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5bbc158120 0x7f5bbc0c8770 unknown :-1 s=CLOSED pgs=1991 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:52:16.570 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:16.569+0000 7f5bc0911640 1 --2- 172.21.15.123:0/3427250857 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5bbc0cf5a0 0x7f5bbc0c6be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:52:16.570 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:16.569+0000 7f5bc0911640 1 --2- 172.21.15.123:0/3427250857 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f5bbc06f5f0 0x7f5bbc155ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:52:16.570 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:16.569+0000 7f5bc0911640 1 -- 172.21.15.123:0/3427250857 >> 172.21.15.123:0/3427250857 conn(0x7f5bbc0cfb10 msgr2=0x7f5bbc0cfef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:52:16.570 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:16.569+0000 7f5bc0911640 1 -- 172.21.15.123:0/3427250857 shutdown_connections 2024-09-16T15:52:16.570 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:16.569+0000 7f5bc0911640 1 -- 172.21.15.123:0/3427250857 wait complete. 2024-09-16T15:52:16.585 INFO:tasks.ceph:Scrubbing osd.2 2024-09-16T15:52:16.585 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 config set osd_debug_deep_scrub_sleep 0 2024-09-16T15:52:16.713 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:16.713+0000 7f502a245640 1 -- 172.21.15.123:0/3035360746 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5024152cf0 msgr2=0x7f50240725e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:52:16.713 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:16.713+0000 7f502a245640 1 --2- 172.21.15.123:0/3035360746 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5024152cf0 0x7f50240725e0 secure :-1 s=READY pgs=1993 cs=0 l=1 rev1=1 crypto rx=0x7f5014009840 tx=0x7f501401c840 comp rx=0 tx=0).stop 2024-09-16T15:52:16.713 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:16.713+0000 7f502a245640 1 -- 172.21.15.123:0/3035360746 shutdown_connections 2024-09-16T15:52:16.713 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:16.713+0000 7f502a245640 1 --2- 172.21.15.123:0/3035360746 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f50240706c0 0x7f50240cebd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:52:16.713 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:16.713+0000 7f502a245640 1 --2- 172.21.15.123:0/3035360746 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5024152cf0 0x7f50240725e0 unknown :-1 s=CLOSED pgs=1993 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:52:16.713 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:16.713+0000 7f502a245640 1 -- 172.21.15.123:0/3035360746 >> 172.21.15.123:0/3035360746 conn(0x7f50240b1630 msgr2=0x7f50240b1a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:52:16.713 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:16.713+0000 7f502a245640 1 -- 172.21.15.123:0/3035360746 shutdown_connections 2024-09-16T15:52:16.713 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:16.713+0000 7f502a245640 1 -- 172.21.15.123:0/3035360746 wait complete. 2024-09-16T15:52:16.714 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:16.713+0000 7f502a245640 1 Processor -- start 2024-09-16T15:52:16.714 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:16.713+0000 7f502a245640 1 -- start start 2024-09-16T15:52:16.716 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:16.717+0000 7f502a245640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f5024070270 0x7f5024147080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:52:16.716 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:16.717+0000 7f502a245640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f50240706c0 0x7f50241f79e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:52:16.716 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:16.717+0000 7f502a245640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5024152cf0 0x7f5024073110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:52:16.716 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:16.717+0000 7f502a245640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f5024168df0 con 0x7f50240706c0 2024-09-16T15:52:16.716 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:16.717+0000 7f502a245640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f5024168c70 con 0x7f5024152cf0 2024-09-16T15:52:16.716 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:16.717+0000 7f502a245640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f5024168f70 con 0x7f5024070270 2024-09-16T15:52:16.716 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:16.717+0000 7f501affd640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f50240706c0 0x7f50241f79e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:52:16.716 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:16.717+0000 7f501affd640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f50240706c0 0x7f50241f79e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:34178/0 (socket says 172.21.15.123:34178) 2024-09-16T15:52:16.716 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:16.717+0000 7f501affd640 1 -- 172.21.15.123:0/269308595 learned_addr learned my addr 172.21.15.123:0/269308595 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:52:16.716 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:16.717+0000 7f501affd640 1 -- 172.21.15.123:0/269308595 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f5024070270 msgr2=0x7f5024147080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:52:16.717 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:16.717+0000 7f501affd640 1 --2- 172.21.15.123:0/269308595 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f5024070270 0x7f5024147080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:52:16.717 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:16.717+0000 7f501affd640 1 -- 172.21.15.123:0/269308595 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5024152cf0 msgr2=0x7f5024073110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:52:16.717 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:16.717+0000 7f501affd640 1 --2- 172.21.15.123:0/269308595 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f5024152cf0 0x7f5024073110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:52:16.717 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:16.717+0000 7f501affd640 1 -- 172.21.15.123:0/269308595 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f50240b2a30 con 0x7f50240706c0 2024-09-16T15:52:16.717 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:16.717+0000 7f501affd640 1 --2- 172.21.15.123:0/269308595 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f50240706c0 0x7f50241f79e0 secure :-1 s=READY pgs=1994 cs=0 l=1 rev1=1 crypto rx=0x7f500c00c450 tx=0x7f500c00c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:52:16.717 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:16.717+0000 7f5020ff9640 1 -- 172.21.15.123:0/269308595 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f500c016020 con 0x7f50240706c0 2024-09-16T15:52:16.717 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:16.717+0000 7f5020ff9640 1 -- 172.21.15.123:0/269308595 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f500c012920 con 0x7f50240706c0 2024-09-16T15:52:16.717 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:16.717+0000 7f502a245640 1 -- 172.21.15.123:0/269308595 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f50240b2cd0 con 0x7f50240706c0 2024-09-16T15:52:16.717 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:16.717+0000 7f5020ff9640 1 -- 172.21.15.123:0/269308595 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f500c012d70 con 0x7f50240706c0 2024-09-16T15:52:16.717 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:16.717+0000 7f502a245640 1 -- 172.21.15.123:0/269308595 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f50240b3290 con 0x7f50240706c0 2024-09-16T15:52:16.717 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:16.717+0000 7f502a245640 1 -- 172.21.15.123:0/269308595 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f5024147620 con 0x7f50240706c0 2024-09-16T15:52:16.718 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:16.717+0000 7f5020ff9640 1 -- 172.21.15.123:0/269308595 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 27) ==== 98151+0+0 (secure 0 0 0) 0x7f500c019020 con 0x7f50240706c0 2024-09-16T15:52:16.719 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:16.717+0000 7f5020ff9640 1 --2- 172.21.15.123:0/269308595 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f5008073ac0 0x7f5008075f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:52:16.719 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:16.717+0000 7f50237fe640 1 --2- 172.21.15.123:0/269308595 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f5008073ac0 0x7f5008075f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:52:16.719 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:16.717+0000 7f5020ff9640 1 -- 172.21.15.123:0/269308595 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(172..172 src has 1..172) ==== 4971+0+0 (secure 0 0 0) 0x7f500c092cd0 con 0x7f50240706c0 2024-09-16T15:52:16.719 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:16.717+0000 7f5020ff9640 1 --2- 172.21.15.123:0/269308595 >> [v2:172.21.15.123:6824/1332143088,v1:172.21.15.123:6825/1332143088] conn(0x7f5008079980 0x7f500807bda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:52:16.719 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:16.717+0000 7f5020ff9640 1 -- 172.21.15.123:0/269308595 --> [v2:172.21.15.123:6824/1332143088,v1:172.21.15.123:6825/1332143088] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f500807c470 con 0x7f5008079980 2024-09-16T15:52:16.719 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:16.717+0000 7f5020ff9640 1 -- 172.21.15.123:0/269308595 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_get_version_reply(handle=1 version=172) ==== 24+0+0 (secure 0 0 0) 0x7f500c0959c0 con 0x7f50240706c0 2024-09-16T15:52:16.719 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:16.717+0000 7f5023fff640 1 --2- 172.21.15.123:0/269308595 >> [v2:172.21.15.123:6824/1332143088,v1:172.21.15.123:6825/1332143088] conn(0x7f5008079980 0x7f500807bda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:52:16.719 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:16.717+0000 7f50237fe640 1 --2- 172.21.15.123:0/269308595 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f5008073ac0 0x7f5008075f80 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f50140af000 tx=0x7f50140195e0 comp rx=0 tx=0).ready entity=mgr.4150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:52:16.719 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:16.721+0000 7f5023fff640 1 --2- 172.21.15.123:0/269308595 >> [v2:172.21.15.123:6824/1332143088,v1:172.21.15.123:6825/1332143088] conn(0x7f5008079980 0x7f500807bda0 crc :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:52:16.719 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:16.721+0000 7f5020ff9640 1 -- 172.21.15.123:0/269308595 <== osd.2 v2:172.21.15.123:6824/1332143088 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f500807c470 con 0x7f5008079980 2024-09-16T15:52:16.731 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:16.733+0000 7f502a245640 1 -- 172.21.15.123:0/269308595 --> [v2:172.21.15.123:6824/1332143088,v1:172.21.15.123:6825/1332143088] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f50240cebd0 con 0x7f5008079980 2024-09-16T15:52:16.731 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:16.733+0000 7f5020ff9640 1 -- 172.21.15.123:0/269308595 <== osd.2 v2:172.21.15.123:6824/1332143088 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7f50240cebd0 con 0x7f5008079980 2024-09-16T15:52:16.732 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2024-09-16T15:52:16.732 INFO:teuthology.orchestra.run.smithi123.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-16T15:52:16.732 INFO:teuthology.orchestra.run.smithi123.stdout:} 2024-09-16T15:52:16.733 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:16.733+0000 7f502a245640 1 -- 172.21.15.123:0/269308595 >> [v2:172.21.15.123:6824/1332143088,v1:172.21.15.123:6825/1332143088] conn(0x7f5008079980 msgr2=0x7f500807bda0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:52:16.733 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:16.733+0000 7f502a245640 1 --2- 172.21.15.123:0/269308595 >> [v2:172.21.15.123:6824/1332143088,v1:172.21.15.123:6825/1332143088] conn(0x7f5008079980 0x7f500807bda0 crc :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:52:16.733 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:16.733+0000 7f502a245640 1 -- 172.21.15.123:0/269308595 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f5008073ac0 msgr2=0x7f5008075f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:52:16.733 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:16.733+0000 7f502a245640 1 --2- 172.21.15.123:0/269308595 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f5008073ac0 0x7f5008075f80 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f50140af000 tx=0x7f50140195e0 comp rx=0 tx=0).stop 2024-09-16T15:52:16.733 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:16.733+0000 7f502a245640 1 -- 172.21.15.123:0/269308595 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f50240706c0 msgr2=0x7f50241f79e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:52:16.733 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:16.733+0000 7f502a245640 1 --2- 172.21.15.123:0/269308595 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f50240706c0 0x7f50241f79e0 secure :-1 s=READY pgs=1994 cs=0 l=1 rev1=1 crypto rx=0x7f500c00c450 tx=0x7f500c00c910 comp rx=0 tx=0).stop 2024-09-16T15:52:16.733 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:16.733+0000 7f5023fff640 1 -- 172.21.15.123:0/269308595 reap_dead start 2024-09-16T15:52:16.733 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:16.733+0000 7f502a245640 1 -- 172.21.15.123:0/269308595 shutdown_connections 2024-09-16T15:52:16.733 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:16.733+0000 7f502a245640 1 -- 172.21.15.123:0/269308595 >> 172.21.15.123:0/269308595 conn(0x7f50240b1630 msgr2=0x7f502415d1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:52:16.733 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:16.733+0000 7f502a245640 1 -- 172.21.15.123:0/269308595 shutdown_connections 2024-09-16T15:52:16.733 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:16.733+0000 7f502a245640 1 -- 172.21.15.123:0/269308595 wait complete. 2024-09-16T15:52:16.747 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 2 2024-09-16T15:52:16.866 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:16.869+0000 7f760057f640 1 -- 172.21.15.123:0/46944885 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f75f8070c10 msgr2=0x7f75f80cd230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:52:16.867 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:16.869+0000 7f760057f640 1 --2- 172.21.15.123:0/46944885 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f75f8070c10 0x7f75f80cd230 secure :-1 s=READY pgs=1996 cs=0 l=1 rev1=1 crypto rx=0x7f75f80700a0 tx=0x7f75ec01c8c0 comp rx=0 tx=0).stop 2024-09-16T15:52:16.867 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:16.869+0000 7f760057f640 1 -- 172.21.15.123:0/46944885 shutdown_connections 2024-09-16T15:52:16.867 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:16.869+0000 7f760057f640 1 --2- 172.21.15.123:0/46944885 >> v2:172.21.15.184:3300/0 conn(0x7f75f815ba50 0x7f75f80b39b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:52:16.867 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:16.869+0000 7f760057f640 1 --2- 172.21.15.123:0/46944885 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f75f8070c10 0x7f75f80cd230 unknown :-1 s=CLOSED pgs=1996 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:52:16.867 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:16.869+0000 7f760057f640 1 -- 172.21.15.123:0/46944885 >> 172.21.15.123:0/46944885 conn(0x7f75f80d0b40 msgr2=0x7f75f80d0f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:52:16.867 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:16.869+0000 7f760057f640 1 -- 172.21.15.123:0/46944885 shutdown_connections 2024-09-16T15:52:16.867 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:16.869+0000 7f760057f640 1 -- 172.21.15.123:0/46944885 wait complete. 2024-09-16T15:52:16.867 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:16.869+0000 7f760057f640 1 Processor -- start 2024-09-16T15:52:16.868 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:16.869+0000 7f760057f640 1 -- start start 2024-09-16T15:52:16.869 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:16.869+0000 7f760057f640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f75f80705c0 0x7f75f81fbc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:52:16.869 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:16.869+0000 7f760057f640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f75f8070c10 0x7f75f8159a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:52:16.869 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:16.869+0000 7f760057f640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f75f815ba50 0x7f75f814e240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:52:16.869 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:16.869+0000 7f760057f640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f75f8167d00 con 0x7f75f8070c10 2024-09-16T15:52:16.869 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:16.869+0000 7f760057f640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f75f8167b80 con 0x7f75f815ba50 2024-09-16T15:52:16.870 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:16.869+0000 7f760057f640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f75f8167e80 con 0x7f75f80705c0 2024-09-16T15:52:16.870 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:16.869+0000 7f75fdb1a640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f75f8070c10 0x7f75f8159a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:52:16.870 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:16.869+0000 7f75fdb1a640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f75f8070c10 0x7f75f8159a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:34196/0 (socket says 172.21.15.123:34196) 2024-09-16T15:52:16.870 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:16.869+0000 7f75fdb1a640 1 -- 172.21.15.123:0/1645204180 learned_addr learned my addr 172.21.15.123:0/1645204180 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:52:16.870 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:16.869+0000 7f75fe31b640 1 --2- 172.21.15.123:0/1645204180 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f75f80705c0 0x7f75f81fbc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:52:16.870 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:16.869+0000 7f75feb1c640 1 --2- 172.21.15.123:0/1645204180 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f75f815ba50 0x7f75f814e240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:52:16.870 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:16.869+0000 7f75fdb1a640 1 -- 172.21.15.123:0/1645204180 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f75f80705c0 msgr2=0x7f75f81fbc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:52:16.870 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:16.869+0000 7f75fdb1a640 1 --2- 172.21.15.123:0/1645204180 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f75f80705c0 0x7f75f81fbc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:52:16.870 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:16.869+0000 7f75fdb1a640 1 -- 172.21.15.123:0/1645204180 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f75f815ba50 msgr2=0x7f75f814e240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:52:16.870 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:16.869+0000 7f75fdb1a640 1 --2- 172.21.15.123:0/1645204180 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f75f815ba50 0x7f75f814e240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:52:16.871 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:16.869+0000 7f75fdb1a640 1 -- 172.21.15.123:0/1645204180 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f75f8159f50 con 0x7f75f8070c10 2024-09-16T15:52:16.871 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:16.869+0000 7f75fdb1a640 1 --2- 172.21.15.123:0/1645204180 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f75f8070c10 0x7f75f8159a10 secure :-1 s=READY pgs=1997 cs=0 l=1 rev1=1 crypto rx=0x7f75f000c8a0 tx=0x7f75f000cd60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:52:16.871 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:16.869+0000 7f75f6ffd640 1 -- 172.21.15.123:0/1645204180 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75f0003b80 con 0x7f75f8070c10 2024-09-16T15:52:16.871 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:16.869+0000 7f75f6ffd640 1 -- 172.21.15.123:0/1645204180 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f75f0003d40 con 0x7f75f8070c10 2024-09-16T15:52:16.871 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:16.869+0000 7f760057f640 1 -- 172.21.15.123:0/1645204180 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f75f815a200 con 0x7f75f8070c10 2024-09-16T15:52:16.871 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:16.869+0000 7f75f6ffd640 1 -- 172.21.15.123:0/1645204180 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75f000aab0 con 0x7f75f8070c10 2024-09-16T15:52:16.871 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:16.869+0000 7f760057f640 1 -- 172.21.15.123:0/1645204180 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f75f81421c0 con 0x7f75f8070c10 2024-09-16T15:52:16.872 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:16.869+0000 7f760057f640 1 -- 172.21.15.123:0/1645204180 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f75f80cd230 con 0x7f75f8070c10 2024-09-16T15:52:16.872 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:16.873+0000 7f75f6ffd640 1 -- 172.21.15.123:0/1645204180 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 27) ==== 98151+0+0 (secure 0 0 0) 0x7f75f000ac50 con 0x7f75f8070c10 2024-09-16T15:52:16.872 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:16.873+0000 7f75f6ffd640 1 --2- 172.21.15.123:0/1645204180 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f75e4073a90 0x7f75e4075f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:52:16.872 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:16.873+0000 7f75f6ffd640 1 -- 172.21.15.123:0/1645204180 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(172..172 src has 1..172) ==== 4971+0+0 (secure 0 0 0) 0x7f75f0092970 con 0x7f75f8070c10 2024-09-16T15:52:16.872 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:16.873+0000 7f75fe31b640 1 --2- 172.21.15.123:0/1645204180 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f75e4073a90 0x7f75e4075f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:52:16.872 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:16.873+0000 7f75fe31b640 1 --2- 172.21.15.123:0/1645204180 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f75e4073a90 0x7f75e4075f50 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f75ec0af000 tx=0x7f75ec01b100 comp rx=0 tx=0).ready entity=mgr.4150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:52:16.875 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:16.877+0000 7f75f6ffd640 1 -- 172.21.15.123:0/1645204180 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f75f005f2d0 con 0x7f75f8070c10 2024-09-16T15:52:17.045 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:17.045+0000 7f760057f640 1 -- 172.21.15.123:0/1645204180 --> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "2", "target": ["mon-mgr", ""]}) -- 0x7f75f814cdb0 con 0x7f75e4073a90 2024-09-16T15:52:17.046 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:17.049+0000 7f75f6ffd640 1 -- 172.21.15.123:0/1645204180 <== mgr.4150 v2:172.21.15.123:6836/2464759676 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 2 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7f75f814cdb0 con 0x7f75e4073a90 2024-09-16T15:52:17.046 INFO:teuthology.orchestra.run.smithi123.stderr:instructed osd(s) 2 to deep-scrub 2024-09-16T15:52:17.049 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:17.049+0000 7f760057f640 1 -- 172.21.15.123:0/1645204180 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f75e4073a90 msgr2=0x7f75e4075f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:52:17.049 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:17.049+0000 7f760057f640 1 --2- 172.21.15.123:0/1645204180 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f75e4073a90 0x7f75e4075f50 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f75ec0af000 tx=0x7f75ec01b100 comp rx=0 tx=0).stop 2024-09-16T15:52:17.049 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:17.049+0000 7f760057f640 1 -- 172.21.15.123:0/1645204180 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f75f8070c10 msgr2=0x7f75f8159a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:52:17.049 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:17.049+0000 7f760057f640 1 --2- 172.21.15.123:0/1645204180 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f75f8070c10 0x7f75f8159a10 secure :-1 s=READY pgs=1997 cs=0 l=1 rev1=1 crypto rx=0x7f75f000c8a0 tx=0x7f75f000cd60 comp rx=0 tx=0).stop 2024-09-16T15:52:17.049 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:17.049+0000 7f760057f640 1 -- 172.21.15.123:0/1645204180 shutdown_connections 2024-09-16T15:52:17.049 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:17.049+0000 7f760057f640 1 --2- 172.21.15.123:0/1645204180 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f75e4073a90 0x7f75e4075f50 unknown :-1 s=CLOSED pgs=456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:52:17.049 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:17.049+0000 7f760057f640 1 --2- 172.21.15.123:0/1645204180 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f75f815ba50 0x7f75f814e240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:52:17.049 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:17.049+0000 7f760057f640 1 --2- 172.21.15.123:0/1645204180 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f75f8070c10 0x7f75f8159a10 unknown :-1 s=CLOSED pgs=1997 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:52:17.049 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:17.049+0000 7f760057f640 1 --2- 172.21.15.123:0/1645204180 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f75f80705c0 0x7f75f81fbc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:52:17.050 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:17.049+0000 7f760057f640 1 -- 172.21.15.123:0/1645204180 >> 172.21.15.123:0/1645204180 conn(0x7f75f80d0b40 msgr2=0x7f75f815d6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:52:17.050 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:17.049+0000 7f760057f640 1 -- 172.21.15.123:0/1645204180 shutdown_connections 2024-09-16T15:52:17.050 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:17.049+0000 7f760057f640 1 -- 172.21.15.123:0/1645204180 wait complete. 2024-09-16T15:52:17.063 INFO:tasks.ceph:Scrubbing osd.3 2024-09-16T15:52:17.064 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 config set osd_debug_deep_scrub_sleep 0 2024-09-16T15:52:17.181 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:17.181+0000 7fcf1c40b640 1 -- 172.21.15.123:0/4263087501 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcf1415fbb0 msgr2=0x7fcf1415c030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:52:17.181 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:17.181+0000 7fcf1c40b640 1 --2- 172.21.15.123:0/4263087501 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcf1415fbb0 0x7fcf1415c030 secure :-1 s=READY pgs=1998 cs=0 l=1 rev1=1 crypto rx=0x7fcf1406de90 tx=0x7fcf0c01cd80 comp rx=0 tx=0).stop 2024-09-16T15:52:17.181 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:17.181+0000 7fcf1c40b640 1 -- 172.21.15.123:0/4263087501 shutdown_connections 2024-09-16T15:52:17.182 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:17.181+0000 7fcf1c40b640 1 --2- 172.21.15.123:0/4263087501 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcf1415fbb0 0x7fcf1415c030 unknown :-1 s=CLOSED pgs=1998 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:52:17.182 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:17.181+0000 7fcf1c40b640 1 --2- 172.21.15.123:0/4263087501 >> v2:172.21.15.184:3300/0 conn(0x7fcf14074440 0x7fcf1415f670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:52:17.182 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:17.181+0000 7fcf1c40b640 1 --2- 172.21.15.123:0/4263087501 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fcf14159680 0x7fcf140704e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:52:17.182 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:17.181+0000 7fcf1c40b640 1 -- 172.21.15.123:0/4263087501 >> 172.21.15.123:0/4263087501 conn(0x7fcf140b1630 msgr2=0x7fcf140b1a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:52:17.182 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:17.181+0000 7fcf1c40b640 1 -- 172.21.15.123:0/4263087501 shutdown_connections 2024-09-16T15:52:17.182 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:17.181+0000 7fcf1c40b640 1 -- 172.21.15.123:0/4263087501 wait complete. 2024-09-16T15:52:17.182 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:17.185+0000 7fcf1c40b640 1 Processor -- start 2024-09-16T15:52:17.183 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:17.185+0000 7fcf1c40b640 1 -- start start 2024-09-16T15:52:17.185 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:17.185+0000 7fcf1c40b640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fcf14074440 0x7fcf1413ad90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:52:17.185 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:17.185+0000 7fcf1c40b640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fcf14159680 0x7fcf140b2a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:52:17.185 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:17.185+0000 7fcf1c40b640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcf1415fbb0 0x7fcf140b2f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:52:17.185 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:17.185+0000 7fcf1c40b640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fcf140ce910 con 0x7fcf1415fbb0 2024-09-16T15:52:17.185 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:17.185+0000 7fcf1c40b640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fcf140ce790 con 0x7fcf14074440 2024-09-16T15:52:17.185 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:17.185+0000 7fcf1c40b640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fcf140cea90 con 0x7fcf14159680 2024-09-16T15:52:17.185 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:17.185+0000 7fcf1a9a8640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcf1415fbb0 0x7fcf140b2f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:52:17.185 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:17.185+0000 7fcf1a9a8640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcf1415fbb0 0x7fcf140b2f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:34214/0 (socket says 172.21.15.123:34214) 2024-09-16T15:52:17.186 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:17.185+0000 7fcf1a9a8640 1 -- 172.21.15.123:0/3016450458 learned_addr learned my addr 172.21.15.123:0/3016450458 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:52:17.186 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:17.185+0000 7fcf1a1a7640 1 --2- 172.21.15.123:0/3016450458 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fcf14074440 0x7fcf1413ad90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:52:17.186 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:17.185+0000 7fcf199a6640 1 --2- 172.21.15.123:0/3016450458 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fcf14159680 0x7fcf140b2a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:52:17.186 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:17.185+0000 7fcf1a9a8640 1 -- 172.21.15.123:0/3016450458 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fcf14159680 msgr2=0x7fcf140b2a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:52:17.186 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:17.185+0000 7fcf1a9a8640 1 --2- 172.21.15.123:0/3016450458 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fcf14159680 0x7fcf140b2a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:52:17.186 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:17.185+0000 7fcf1a9a8640 1 -- 172.21.15.123:0/3016450458 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fcf14074440 msgr2=0x7fcf1413ad90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:52:17.186 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:17.185+0000 7fcf1a9a8640 1 --2- 172.21.15.123:0/3016450458 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fcf14074440 0x7fcf1413ad90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:52:17.186 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:17.185+0000 7fcf1a9a8640 1 -- 172.21.15.123:0/3016450458 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcf1415f680 con 0x7fcf1415fbb0 2024-09-16T15:52:17.186 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:17.185+0000 7fcf1a9a8640 1 --2- 172.21.15.123:0/3016450458 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcf1415fbb0 0x7fcf140b2f70 secure :-1 s=READY pgs=1999 cs=0 l=1 rev1=1 crypto rx=0x7fcf0c01b430 tx=0x7fcf0c003f80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:52:17.186 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:17.185+0000 7fcf0affd640 1 -- 172.21.15.123:0/3016450458 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf0c0bc070 con 0x7fcf1415fbb0 2024-09-16T15:52:17.186 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:17.185+0000 7fcf0affd640 1 -- 172.21.15.123:0/3016450458 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fcf0c0042f0 con 0x7fcf1415fbb0 2024-09-16T15:52:17.186 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:17.185+0000 7fcf0affd640 1 -- 172.21.15.123:0/3016450458 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf0c0b2040 con 0x7fcf1415fbb0 2024-09-16T15:52:17.186 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:17.185+0000 7fcf1c40b640 1 -- 172.21.15.123:0/3016450458 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcf1413d890 con 0x7fcf1415fbb0 2024-09-16T15:52:17.186 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:17.185+0000 7fcf1c40b640 1 -- 172.21.15.123:0/3016450458 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcf1413de50 con 0x7fcf1415fbb0 2024-09-16T15:52:17.187 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:17.185+0000 7fcf1c40b640 1 -- 172.21.15.123:0/3016450458 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fcf140704e0 con 0x7fcf1415fbb0 2024-09-16T15:52:17.187 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:17.189+0000 7fcf0affd640 1 -- 172.21.15.123:0/3016450458 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 27) ==== 98151+0+0 (secure 0 0 0) 0x7fcf0c004720 con 0x7fcf1415fbb0 2024-09-16T15:52:17.188 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:17.189+0000 7fcf0affd640 1 --2- 172.21.15.123:0/3016450458 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fcef8073ac0 0x7fcef8075f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:52:17.188 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:17.189+0000 7fcf0affd640 1 -- 172.21.15.123:0/3016450458 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(172..172 src has 1..172) ==== 4971+0+0 (secure 0 0 0) 0x7fcf0c12c9f0 con 0x7fcf1415fbb0 2024-09-16T15:52:17.188 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:17.189+0000 7fcf1a1a7640 1 --2- 172.21.15.123:0/3016450458 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fcef8073ac0 0x7fcef8075f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:52:17.188 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:17.189+0000 7fcf0affd640 1 --2- 172.21.15.123:0/3016450458 >> [v2:172.21.15.123:6808/988586998,v1:172.21.15.123:6809/988586998] conn(0x7fcef8079980 0x7fcef807bda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:52:17.189 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:17.189+0000 7fcf0affd640 1 -- 172.21.15.123:0/3016450458 --> [v2:172.21.15.123:6808/988586998,v1:172.21.15.123:6809/988586998] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fcef807c470 con 0x7fcef8079980 2024-09-16T15:52:17.189 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:17.189+0000 7fcf0affd640 1 -- 172.21.15.123:0/3016450458 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_get_version_reply(handle=1 version=172) ==== 24+0+0 (secure 0 0 0) 0x7fcf0c12f2d0 con 0x7fcf1415fbb0 2024-09-16T15:52:17.189 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:17.189+0000 7fcf199a6640 1 --2- 172.21.15.123:0/3016450458 >> [v2:172.21.15.123:6808/988586998,v1:172.21.15.123:6809/988586998] conn(0x7fcef8079980 0x7fcef807bda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:52:17.189 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:17.189+0000 7fcf1a1a7640 1 --2- 172.21.15.123:0/3016450458 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fcef8073ac0 0x7fcef8075f80 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7fcf1000ba40 tx=0x7fcf10005cd0 comp rx=0 tx=0).ready entity=mgr.4150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:52:17.189 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:17.189+0000 7fcf199a6640 1 --2- 172.21.15.123:0/3016450458 >> [v2:172.21.15.123:6808/988586998,v1:172.21.15.123:6809/988586998] conn(0x7fcef8079980 0x7fcef807bda0 crc :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:52:17.189 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:17.189+0000 7fcf0affd640 1 -- 172.21.15.123:0/3016450458 <== osd.3 v2:172.21.15.123:6808/988586998 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fcef807c470 con 0x7fcef8079980 2024-09-16T15:52:17.202 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:17.201+0000 7fcf1c40b640 1 -- 172.21.15.123:0/3016450458 --> [v2:172.21.15.123:6808/988586998,v1:172.21.15.123:6809/988586998] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7fcf14070040 con 0x7fcef8079980 2024-09-16T15:52:17.202 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:17.201+0000 7fcf0affd640 1 -- 172.21.15.123:0/3016450458 <== osd.3 v2:172.21.15.123:6808/988586998 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7fcf14070040 con 0x7fcef8079980 2024-09-16T15:52:17.202 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2024-09-16T15:52:17.202 INFO:teuthology.orchestra.run.smithi123.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-16T15:52:17.202 INFO:teuthology.orchestra.run.smithi123.stdout:} 2024-09-16T15:52:17.203 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:17.205+0000 7fcf1c40b640 1 -- 172.21.15.123:0/3016450458 >> [v2:172.21.15.123:6808/988586998,v1:172.21.15.123:6809/988586998] conn(0x7fcef8079980 msgr2=0x7fcef807bda0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:52:17.203 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:17.205+0000 7fcf1c40b640 1 --2- 172.21.15.123:0/3016450458 >> [v2:172.21.15.123:6808/988586998,v1:172.21.15.123:6809/988586998] conn(0x7fcef8079980 0x7fcef807bda0 crc :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:52:17.203 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:17.205+0000 7fcf1c40b640 1 -- 172.21.15.123:0/3016450458 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fcef8073ac0 msgr2=0x7fcef8075f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:52:17.203 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:17.205+0000 7fcf1c40b640 1 --2- 172.21.15.123:0/3016450458 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fcef8073ac0 0x7fcef8075f80 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7fcf1000ba40 tx=0x7fcf10005cd0 comp rx=0 tx=0).stop 2024-09-16T15:52:17.204 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:17.205+0000 7fcf1c40b640 1 -- 172.21.15.123:0/3016450458 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcf1415fbb0 msgr2=0x7fcf140b2f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:52:17.204 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:17.205+0000 7fcf1c40b640 1 --2- 172.21.15.123:0/3016450458 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcf1415fbb0 0x7fcf140b2f70 secure :-1 s=READY pgs=1999 cs=0 l=1 rev1=1 crypto rx=0x7fcf0c01b430 tx=0x7fcf0c003f80 comp rx=0 tx=0).stop 2024-09-16T15:52:17.204 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:17.205+0000 7fcf1a9a8640 1 -- 172.21.15.123:0/3016450458 reap_dead start 2024-09-16T15:52:17.204 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:17.205+0000 7fcf1c40b640 1 -- 172.21.15.123:0/3016450458 shutdown_connections 2024-09-16T15:52:17.204 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:17.205+0000 7fcf1c40b640 1 -- 172.21.15.123:0/3016450458 >> 172.21.15.123:0/3016450458 conn(0x7fcf140b1630 msgr2=0x7fcf14157f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:52:17.204 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:17.205+0000 7fcf1c40b640 1 -- 172.21.15.123:0/3016450458 shutdown_connections 2024-09-16T15:52:17.204 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:17.205+0000 7fcf1c40b640 1 -- 172.21.15.123:0/3016450458 wait complete. 2024-09-16T15:52:17.217 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 3 2024-09-16T15:52:17.324 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:17.325+0000 7f24c5cfc640 1 -- 172.21.15.123:0/678761748 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f24c00cc6f0 msgr2=0x7f24c00cf650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:52:17.325 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:17.325+0000 7f24c5cfc640 1 --2- 172.21.15.123:0/678761748 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f24c00cc6f0 0x7f24c00cf650 secure :-1 s=READY pgs=2000 cs=0 l=1 rev1=1 crypto rx=0x7f24c00d06a0 tx=0x7f24b801cd80 comp rx=0 tx=0).stop 2024-09-16T15:52:17.325 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:17.325+0000 7f24c5cfc640 1 -- 172.21.15.123:0/678761748 shutdown_connections 2024-09-16T15:52:17.325 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:17.325+0000 7f24c5cfc640 1 --2- 172.21.15.123:0/678761748 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f24c00cc6f0 0x7f24c00cf650 unknown :-1 s=CLOSED pgs=2000 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:52:17.325 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:17.325+0000 7f24c5cfc640 1 --2- 172.21.15.123:0/678761748 >> v2:172.21.15.184:3300/0 conn(0x7f24c00b3100 0x7f24c00cc1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:52:17.325 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:17.325+0000 7f24c5cfc640 1 -- 172.21.15.123:0/678761748 >> 172.21.15.123:0/678761748 conn(0x7f24c00b2610 msgr2=0x7f24c00b29f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:52:17.325 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:17.325+0000 7f24c5cfc640 1 -- 172.21.15.123:0/678761748 shutdown_connections 2024-09-16T15:52:17.325 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:17.325+0000 7f24c5cfc640 1 -- 172.21.15.123:0/678761748 wait complete. 2024-09-16T15:52:17.326 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:17.325+0000 7f24c5cfc640 1 Processor -- start 2024-09-16T15:52:17.326 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:17.325+0000 7f24c5cfc640 1 -- start start 2024-09-16T15:52:17.328 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:17.329+0000 7f24c5cfc640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f24c00b3100 0x7f24c0153c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:52:17.328 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:17.329+0000 7f24c5cfc640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f24c00cc6f0 0x7f24c00ca7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:52:17.328 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:17.329+0000 7f24c5cfc640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f24c015ac00 0x7f24c0155340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:52:17.328 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:17.329+0000 7f24c5cfc640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f24c01682d0 con 0x7f24c015ac00 2024-09-16T15:52:17.328 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:17.329+0000 7f24c5cfc640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f24c0070a00 con 0x7f24c00b3100 2024-09-16T15:52:17.328 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:17.329+0000 7f24c5cfc640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f24c0168450 con 0x7f24c00cc6f0 2024-09-16T15:52:17.328 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:17.329+0000 7f24bffff640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f24c015ac00 0x7f24c0155340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:52:17.328 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:17.329+0000 7f24bffff640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f24c015ac00 0x7f24c0155340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:34232/0 (socket says 172.21.15.123:34232) 2024-09-16T15:52:17.328 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:17.329+0000 7f24bffff640 1 -- 172.21.15.123:0/3153224897 learned_addr learned my addr 172.21.15.123:0/3153224897 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:52:17.329 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:17.329+0000 7f24beffd640 1 --2- 172.21.15.123:0/3153224897 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f24c00cc6f0 0x7f24c00ca7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:52:17.329 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:17.329+0000 7f24bf7fe640 1 --2- 172.21.15.123:0/3153224897 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f24c00b3100 0x7f24c0153c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:52:17.329 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:17.329+0000 7f24bffff640 1 -- 172.21.15.123:0/3153224897 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f24c00cc6f0 msgr2=0x7f24c00ca7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:52:17.329 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:17.329+0000 7f24bffff640 1 --2- 172.21.15.123:0/3153224897 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f24c00cc6f0 0x7f24c00ca7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:52:17.329 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:17.329+0000 7f24bffff640 1 -- 172.21.15.123:0/3153224897 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f24c00b3100 msgr2=0x7f24c0153c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:52:17.329 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:17.329+0000 7f24bffff640 1 --2- 172.21.15.123:0/3153224897 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f24c00b3100 0x7f24c0153c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:52:17.329 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:17.329+0000 7f24bffff640 1 -- 172.21.15.123:0/3153224897 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f24c00c7380 con 0x7f24c015ac00 2024-09-16T15:52:17.329 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:17.329+0000 7f24bffff640 1 --2- 172.21.15.123:0/3153224897 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f24c015ac00 0x7f24c0155340 secure :-1 s=READY pgs=2001 cs=0 l=1 rev1=1 crypto rx=0x7f24b801b430 tx=0x7f24b80a5a10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:52:17.329 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:17.329+0000 7f24a7fff640 1 -- 172.21.15.123:0/3153224897 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24b80bb070 con 0x7f24c015ac00 2024-09-16T15:52:17.329 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:17.329+0000 7f24a7fff640 1 -- 172.21.15.123:0/3153224897 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f24b8004d40 con 0x7f24c015ac00 2024-09-16T15:52:17.329 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:17.329+0000 7f24c5cfc640 1 -- 172.21.15.123:0/3153224897 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f24c00c7660 con 0x7f24c015ac00 2024-09-16T15:52:17.329 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:17.329+0000 7f24a7fff640 1 -- 172.21.15.123:0/3153224897 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24b80b2040 con 0x7f24c015ac00 2024-09-16T15:52:17.330 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:17.329+0000 7f24c5cfc640 1 -- 172.21.15.123:0/3153224897 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f24c0154b40 con 0x7f24c015ac00 2024-09-16T15:52:17.330 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:17.329+0000 7f24c5cfc640 1 -- 172.21.15.123:0/3153224897 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f24c00cc1b0 con 0x7f24c015ac00 2024-09-16T15:52:17.330 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:17.329+0000 7f24a7fff640 1 -- 172.21.15.123:0/3153224897 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 27) ==== 98151+0+0 (secure 0 0 0) 0x7f24b80be070 con 0x7f24c015ac00 2024-09-16T15:52:17.330 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:17.329+0000 7f24a7fff640 1 --2- 172.21.15.123:0/3153224897 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f24ac073ac0 0x7f24ac075f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:52:17.331 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:17.329+0000 7f24bf7fe640 1 --2- 172.21.15.123:0/3153224897 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f24ac073ac0 0x7f24ac075f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:52:17.331 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:17.329+0000 7f24a7fff640 1 -- 172.21.15.123:0/3153224897 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(172..172 src has 1..172) ==== 4971+0+0 (secure 0 0 0) 0x7f24b812c7e0 con 0x7f24c015ac00 2024-09-16T15:52:17.331 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:17.329+0000 7f24bf7fe640 1 --2- 172.21.15.123:0/3153224897 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f24ac073ac0 0x7f24ac075f80 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f24c01f9010 tx=0x7f24b0008040 comp rx=0 tx=0).ready entity=mgr.4150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:52:17.333 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:17.333+0000 7f24a7fff640 1 -- 172.21.15.123:0/3153224897 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f24b8016970 con 0x7f24c015ac00 2024-09-16T15:52:17.492 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:17.493+0000 7f24c5cfc640 1 -- 172.21.15.123:0/3153224897 --> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "3", "target": ["mon-mgr", ""]}) -- 0x7f24c01541d0 con 0x7f24ac073ac0 2024-09-16T15:52:17.492 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:17.493+0000 7f24a7fff640 1 -- 172.21.15.123:0/3153224897 <== mgr.4150 v2:172.21.15.123:6836/2464759676 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 3 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7f24c01541d0 con 0x7f24ac073ac0 2024-09-16T15:52:17.492 INFO:teuthology.orchestra.run.smithi123.stderr:instructed osd(s) 3 to deep-scrub 2024-09-16T15:52:17.494 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:17.493+0000 7f24c5cfc640 1 -- 172.21.15.123:0/3153224897 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f24ac073ac0 msgr2=0x7f24ac075f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:52:17.494 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:17.493+0000 7f24c5cfc640 1 --2- 172.21.15.123:0/3153224897 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f24ac073ac0 0x7f24ac075f80 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f24c01f9010 tx=0x7f24b0008040 comp rx=0 tx=0).stop 2024-09-16T15:52:17.494 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:17.493+0000 7f24c5cfc640 1 -- 172.21.15.123:0/3153224897 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f24c015ac00 msgr2=0x7f24c0155340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:52:17.494 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:17.493+0000 7f24c5cfc640 1 --2- 172.21.15.123:0/3153224897 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f24c015ac00 0x7f24c0155340 secure :-1 s=READY pgs=2001 cs=0 l=1 rev1=1 crypto rx=0x7f24b801b430 tx=0x7f24b80a5a10 comp rx=0 tx=0).stop 2024-09-16T15:52:17.494 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:17.493+0000 7f24c5cfc640 1 -- 172.21.15.123:0/3153224897 shutdown_connections 2024-09-16T15:52:17.495 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:17.493+0000 7f24c5cfc640 1 --2- 172.21.15.123:0/3153224897 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f24ac073ac0 0x7f24ac075f80 unknown :-1 s=CLOSED pgs=458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:52:17.495 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:17.493+0000 7f24c5cfc640 1 --2- 172.21.15.123:0/3153224897 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f24c015ac00 0x7f24c0155340 unknown :-1 s=CLOSED pgs=2001 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:52:17.495 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:17.493+0000 7f24c5cfc640 1 --2- 172.21.15.123:0/3153224897 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f24c00cc6f0 0x7f24c00ca7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:52:17.495 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:17.493+0000 7f24c5cfc640 1 --2- 172.21.15.123:0/3153224897 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f24c00b3100 0x7f24c0153c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:52:17.495 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:17.493+0000 7f24c5cfc640 1 -- 172.21.15.123:0/3153224897 >> 172.21.15.123:0/3153224897 conn(0x7f24c00b2610 msgr2=0x7f24c01581e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:52:17.495 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:17.497+0000 7f24c5cfc640 1 -- 172.21.15.123:0/3153224897 shutdown_connections 2024-09-16T15:52:17.495 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:17.497+0000 7f24c5cfc640 1 -- 172.21.15.123:0/3153224897 wait complete. 2024-09-16T15:52:17.510 INFO:tasks.ceph:Scrubbing osd.4 2024-09-16T15:52:17.510 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 config set osd_debug_deep_scrub_sleep 0 2024-09-16T15:52:17.633 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:17.633+0000 7f1ac379e640 1 -- 172.21.15.123:0/2403018288 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1abc0706c0 msgr2=0x7f1abc0cebd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:52:17.633 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:17.633+0000 7f1ac379e640 1 --2- 172.21.15.123:0/2403018288 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1abc0706c0 0x7f1abc0cebd0 secure :-1 s=READY pgs=2002 cs=0 l=1 rev1=1 crypto rx=0x7f1abc069b40 tx=0x7f1ab401cb80 comp rx=0 tx=0).stop 2024-09-16T15:52:17.633 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:17.633+0000 7f1ac379e640 1 -- 172.21.15.123:0/2403018288 shutdown_connections 2024-09-16T15:52:17.633 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:17.633+0000 7f1ac379e640 1 --2- 172.21.15.123:0/2403018288 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1abc0706c0 0x7f1abc0cebd0 unknown :-1 s=CLOSED pgs=2002 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:52:17.633 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:17.633+0000 7f1ac379e640 1 --2- 172.21.15.123:0/2403018288 >> v2:172.21.15.184:3300/0 conn(0x7f1abc152cf0 0x7f1abc0725e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:52:17.633 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:17.633+0000 7f1ac379e640 1 -- 172.21.15.123:0/2403018288 >> 172.21.15.123:0/2403018288 conn(0x7f1abc0b1630 msgr2=0x7f1abc0b1a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:52:17.633 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:17.633+0000 7f1ac379e640 1 -- 172.21.15.123:0/2403018288 shutdown_connections 2024-09-16T15:52:17.634 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:17.633+0000 7f1ac379e640 1 -- 172.21.15.123:0/2403018288 wait complete. 2024-09-16T15:52:17.634 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:17.633+0000 7f1ac379e640 1 Processor -- start 2024-09-16T15:52:17.634 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:17.637+0000 7f1ac379e640 1 -- start start 2024-09-16T15:52:17.636 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:17.637+0000 7f1ac379e640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f1abc070270 0x7f1abc0b43f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:52:17.636 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:17.637+0000 7f1ac379e640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1abc0706c0 0x7f1abc15dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:52:17.636 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:17.637+0000 7f1ac379e640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1abc152cf0 0x7f1abc15e240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:52:17.637 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:17.637+0000 7f1ac379e640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f1abc168da0 con 0x7f1abc152cf0 2024-09-16T15:52:17.637 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:17.637+0000 7f1ac379e640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f1abc168c20 con 0x7f1abc0706c0 2024-09-16T15:52:17.637 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:17.637+0000 7f1ac379e640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f1abc168f20 con 0x7f1abc070270 2024-09-16T15:52:17.637 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:17.637+0000 7f1ac1d3b640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1abc152cf0 0x7f1abc15e240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:52:17.637 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:17.637+0000 7f1ac1d3b640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1abc152cf0 0x7f1abc15e240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:34250/0 (socket says 172.21.15.123:34250) 2024-09-16T15:52:17.637 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:17.637+0000 7f1ac1d3b640 1 -- 172.21.15.123:0/904107919 learned_addr learned my addr 172.21.15.123:0/904107919 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:52:17.637 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:17.637+0000 7f1ac153a640 1 --2- 172.21.15.123:0/904107919 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f1abc070270 0x7f1abc0b43f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:52:17.637 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:17.637+0000 7f1ac0d39640 1 --2- 172.21.15.123:0/904107919 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1abc0706c0 0x7f1abc15dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:52:17.637 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:17.637+0000 7f1ac1d3b640 1 -- 172.21.15.123:0/904107919 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f1abc070270 msgr2=0x7f1abc0b43f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:52:17.637 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:17.637+0000 7f1ac1d3b640 1 --2- 172.21.15.123:0/904107919 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f1abc070270 0x7f1abc0b43f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:52:17.637 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:17.637+0000 7f1ac1d3b640 1 -- 172.21.15.123:0/904107919 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1abc0706c0 msgr2=0x7f1abc15dd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:52:17.637 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:17.637+0000 7f1ac1d3b640 1 --2- 172.21.15.123:0/904107919 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1abc0706c0 0x7f1abc15dd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:52:17.637 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:17.637+0000 7f1ac1d3b640 1 -- 172.21.15.123:0/904107919 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1abc148880 con 0x7f1abc152cf0 2024-09-16T15:52:17.637 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:17.637+0000 7f1ac0d39640 1 --2- 172.21.15.123:0/904107919 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f1abc0706c0 0x7f1abc15dd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T15:52:17.638 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:17.637+0000 7f1ac1d3b640 1 --2- 172.21.15.123:0/904107919 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1abc152cf0 0x7f1abc15e240 secure :-1 s=READY pgs=2003 cs=0 l=1 rev1=1 crypto rx=0x7f1ab4009bd0 tx=0x7f1ab40a5a80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:52:17.638 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:17.637+0000 7f1ab1ffb640 1 -- 172.21.15.123:0/904107919 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ab40bb070 con 0x7f1abc152cf0 2024-09-16T15:52:17.638 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:17.637+0000 7f1ab1ffb640 1 -- 172.21.15.123:0/904107919 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f1ab4004d40 con 0x7f1abc152cf0 2024-09-16T15:52:17.638 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:17.637+0000 7f1ac379e640 1 -- 172.21.15.123:0/904107919 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1abc0b2c40 con 0x7f1abc152cf0 2024-09-16T15:52:17.638 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:17.637+0000 7f1ab1ffb640 1 -- 172.21.15.123:0/904107919 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ab40b2040 con 0x7f1abc152cf0 2024-09-16T15:52:17.638 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:17.637+0000 7f1ac379e640 1 -- 172.21.15.123:0/904107919 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1abc0b3200 con 0x7f1abc152cf0 2024-09-16T15:52:17.638 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:17.637+0000 7f1ac379e640 1 -- 172.21.15.123:0/904107919 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f1abc0cebd0 con 0x7f1abc152cf0 2024-09-16T15:52:17.639 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:17.637+0000 7f1ab1ffb640 1 -- 172.21.15.123:0/904107919 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 27) ==== 98151+0+0 (secure 0 0 0) 0x7f1ab40be070 con 0x7f1abc152cf0 2024-09-16T15:52:17.639 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:17.637+0000 7f1ab1ffb640 1 --2- 172.21.15.123:0/904107919 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f1aa0073ac0 0x7f1aa0075f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:52:17.639 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:17.637+0000 7f1ab1ffb640 1 -- 172.21.15.123:0/904107919 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(172..172 src has 1..172) ==== 4971+0+0 (secure 0 0 0) 0x7f1ab412c770 con 0x7f1abc152cf0 2024-09-16T15:52:17.639 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:17.637+0000 7f1ac153a640 1 --2- 172.21.15.123:0/904107919 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f1aa0073ac0 0x7f1aa0075f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:52:17.639 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:17.637+0000 7f1ab1ffb640 1 --2- 172.21.15.123:0/904107919 >> [v2:172.21.15.184:6816/3387813890,v1:172.21.15.184:6817/3387813890] conn(0x7f1aa0079980 0x7f1aa007bda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:52:17.639 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:17.637+0000 7f1ab1ffb640 1 -- 172.21.15.123:0/904107919 --> [v2:172.21.15.184:6816/3387813890,v1:172.21.15.184:6817/3387813890] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f1ab40020c0 con 0x7f1aa0079980 2024-09-16T15:52:17.639 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:17.637+0000 7f1ab1ffb640 1 -- 172.21.15.123:0/904107919 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_get_version_reply(handle=1 version=172) ==== 24+0+0 (secure 0 0 0) 0x7f1ab4131240 con 0x7f1abc152cf0 2024-09-16T15:52:17.640 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:17.641+0000 7f1ac153a640 1 --2- 172.21.15.123:0/904107919 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f1aa0073ac0 0x7f1aa0075f80 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f1ab80058d0 tx=0x7f1ab8005db0 comp rx=0 tx=0).ready entity=mgr.4150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:52:17.640 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:17.641+0000 7f1ac0d39640 1 --2- 172.21.15.123:0/904107919 >> [v2:172.21.15.184:6816/3387813890,v1:172.21.15.184:6817/3387813890] conn(0x7f1aa0079980 0x7f1aa007bda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:52:17.640 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:17.641+0000 7f1ac0d39640 1 --2- 172.21.15.123:0/904107919 >> [v2:172.21.15.184:6816/3387813890,v1:172.21.15.184:6817/3387813890] conn(0x7f1aa0079980 0x7f1aa007bda0 crc :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:52:17.640 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:17.641+0000 7f1ab1ffb640 1 -- 172.21.15.123:0/904107919 <== osd.4 v2:172.21.15.184:6816/3387813890 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f1ab40020c0 con 0x7f1aa0079980 2024-09-16T15:52:17.653 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:17.653+0000 7f1ac379e640 1 -- 172.21.15.123:0/904107919 --> [v2:172.21.15.184:6816/3387813890,v1:172.21.15.184:6817/3387813890] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f1abc15ce80 con 0x7f1aa0079980 2024-09-16T15:52:17.654 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:17.653+0000 7f1ab1ffb640 1 -- 172.21.15.123:0/904107919 <== osd.4 v2:172.21.15.184:6816/3387813890 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7f1abc15ce80 con 0x7f1aa0079980 2024-09-16T15:52:17.654 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2024-09-16T15:52:17.654 INFO:teuthology.orchestra.run.smithi123.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-16T15:52:17.654 INFO:teuthology.orchestra.run.smithi123.stdout:} 2024-09-16T15:52:17.655 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:17.657+0000 7f1ac379e640 1 -- 172.21.15.123:0/904107919 >> [v2:172.21.15.184:6816/3387813890,v1:172.21.15.184:6817/3387813890] conn(0x7f1aa0079980 msgr2=0x7f1aa007bda0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:52:17.655 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:17.657+0000 7f1ac379e640 1 --2- 172.21.15.123:0/904107919 >> [v2:172.21.15.184:6816/3387813890,v1:172.21.15.184:6817/3387813890] conn(0x7f1aa0079980 0x7f1aa007bda0 crc :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:52:17.655 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:17.657+0000 7f1ac379e640 1 -- 172.21.15.123:0/904107919 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f1aa0073ac0 msgr2=0x7f1aa0075f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:52:17.655 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:17.657+0000 7f1ac379e640 1 --2- 172.21.15.123:0/904107919 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f1aa0073ac0 0x7f1aa0075f80 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f1ab80058d0 tx=0x7f1ab8005db0 comp rx=0 tx=0).stop 2024-09-16T15:52:17.655 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:17.657+0000 7f1ac379e640 1 -- 172.21.15.123:0/904107919 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1abc152cf0 msgr2=0x7f1abc15e240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:52:17.655 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:17.657+0000 7f1ac379e640 1 --2- 172.21.15.123:0/904107919 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1abc152cf0 0x7f1abc15e240 secure :-1 s=READY pgs=2003 cs=0 l=1 rev1=1 crypto rx=0x7f1ab4009bd0 tx=0x7f1ab40a5a80 comp rx=0 tx=0).stop 2024-09-16T15:52:17.655 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:17.657+0000 7f1ac1d3b640 1 -- 172.21.15.123:0/904107919 reap_dead start 2024-09-16T15:52:17.656 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:17.657+0000 7f1ac379e640 1 -- 172.21.15.123:0/904107919 shutdown_connections 2024-09-16T15:52:17.656 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:17.657+0000 7f1ac379e640 1 -- 172.21.15.123:0/904107919 >> 172.21.15.123:0/904107919 conn(0x7f1abc0b1630 msgr2=0x7f1abc165ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:52:17.656 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:17.657+0000 7f1ac379e640 1 -- 172.21.15.123:0/904107919 shutdown_connections 2024-09-16T15:52:17.656 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:17.657+0000 7f1ac379e640 1 -- 172.21.15.123:0/904107919 wait complete. 2024-09-16T15:52:17.669 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 4 2024-09-16T15:52:17.777 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:17.777+0000 7f249c6d8640 1 -- 172.21.15.123:0/1909700127 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f24940b3b20 msgr2=0x7f24940d3af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:52:17.777 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:17.777+0000 7f249c6d8640 1 --2- 172.21.15.123:0/1909700127 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f24940b3b20 0x7f24940d3af0 secure :-1 s=READY pgs=2004 cs=0 l=1 rev1=1 crypto rx=0x7f2484009900 tx=0x7f248401c7f0 comp rx=0 tx=0).stop 2024-09-16T15:52:17.778 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:17.777+0000 7f249c6d8640 1 -- 172.21.15.123:0/1909700127 shutdown_connections 2024-09-16T15:52:17.778 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:17.777+0000 7f249c6d8640 1 --2- 172.21.15.123:0/1909700127 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f24940b3b20 0x7f24940d3af0 unknown :-1 s=CLOSED pgs=2004 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:52:17.778 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:17.777+0000 7f249c6d8640 1 --2- 172.21.15.123:0/1909700127 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f249415dd60 0x7f24941621d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:52:17.778 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:17.777+0000 7f249c6d8640 1 -- 172.21.15.123:0/1909700127 >> 172.21.15.123:0/1909700127 conn(0x7f2494069650 msgr2=0x7f249409d170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:52:17.778 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:17.777+0000 7f249c6d8640 1 -- 172.21.15.123:0/1909700127 shutdown_connections 2024-09-16T15:52:17.778 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:17.777+0000 7f249c6d8640 1 -- 172.21.15.123:0/1909700127 wait complete. 2024-09-16T15:52:17.778 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:17.781+0000 7f249c6d8640 1 Processor -- start 2024-09-16T15:52:17.778 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:17.781+0000 7f249c6d8640 1 -- start start 2024-09-16T15:52:17.781 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:17.781+0000 7f249c6d8640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f24940b3b20 0x7f2494153e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:52:17.781 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:17.781+0000 7f249c6d8640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f24940d4030 0x7f2494153000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:52:17.781 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:17.781+0000 7f249c6d8640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f249415dd60 0x7f249415ba50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:52:17.781 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:17.781+0000 7f249c6d8640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f249416b330 con 0x7f24940b3b20 2024-09-16T15:52:17.781 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:17.781+0000 7f249c6d8640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f249416b1b0 con 0x7f24940d4030 2024-09-16T15:52:17.781 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:17.781+0000 7f249c6d8640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f249416b4b0 con 0x7f249415dd60 2024-09-16T15:52:17.781 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:17.781+0000 7f249a474640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f24940b3b20 0x7f2494153e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:52:17.781 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:17.781+0000 7f249a474640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f24940b3b20 0x7f2494153e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:34268/0 (socket says 172.21.15.123:34268) 2024-09-16T15:52:17.781 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:17.781+0000 7f249a474640 1 -- 172.21.15.123:0/3105416681 learned_addr learned my addr 172.21.15.123:0/3105416681 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:52:17.781 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:17.781+0000 7f2499c73640 1 --2- 172.21.15.123:0/3105416681 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f24940d4030 0x7f2494153000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:52:17.781 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:17.781+0000 7f249a474640 1 -- 172.21.15.123:0/3105416681 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f249415dd60 msgr2=0x7f249415ba50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:52:17.781 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:17.781+0000 7f249ac75640 1 --2- 172.21.15.123:0/3105416681 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f249415dd60 0x7f249415ba50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:52:17.781 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:17.781+0000 7f249a474640 1 --2- 172.21.15.123:0/3105416681 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f249415dd60 0x7f249415ba50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:52:17.782 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:17.781+0000 7f249a474640 1 -- 172.21.15.123:0/3105416681 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f24940d4030 msgr2=0x7f2494153000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:52:17.782 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:17.781+0000 7f249a474640 1 --2- 172.21.15.123:0/3105416681 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f24940d4030 0x7f2494153000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:52:17.782 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:17.781+0000 7f249a474640 1 -- 172.21.15.123:0/3105416681 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f24940b9c60 con 0x7f24940b3b20 2024-09-16T15:52:17.782 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:17.781+0000 7f2499c73640 1 --2- 172.21.15.123:0/3105416681 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f24940d4030 0x7f2494153000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T15:52:17.782 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:17.781+0000 7f249a474640 1 --2- 172.21.15.123:0/3105416681 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f24940b3b20 0x7f2494153e40 secure :-1 s=READY pgs=2005 cs=0 l=1 rev1=1 crypto rx=0x7f24900058d0 tx=0x7f249000dde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:52:17.782 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:17.781+0000 7f248affd640 1 -- 172.21.15.123:0/3105416681 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2490017070 con 0x7f24940b3b20 2024-09-16T15:52:17.782 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:17.781+0000 7f248affd640 1 -- 172.21.15.123:0/3105416681 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f2490009d20 con 0x7f24940b3b20 2024-09-16T15:52:17.782 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:17.781+0000 7f249c6d8640 1 -- 172.21.15.123:0/3105416681 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2494161650 con 0x7f24940b3b20 2024-09-16T15:52:17.782 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:17.781+0000 7f248affd640 1 -- 172.21.15.123:0/3105416681 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24900106c0 con 0x7f24940b3b20 2024-09-16T15:52:17.782 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:17.781+0000 7f249c6d8640 1 -- 172.21.15.123:0/3105416681 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2494161370 con 0x7f24940b3b20 2024-09-16T15:52:17.783 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:17.781+0000 7f249c6d8640 1 -- 172.21.15.123:0/3105416681 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f24940d3af0 con 0x7f24940b3b20 2024-09-16T15:52:17.783 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:17.781+0000 7f248affd640 1 -- 172.21.15.123:0/3105416681 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 27) ==== 98151+0+0 (secure 0 0 0) 0x7f249001a070 con 0x7f24940b3b20 2024-09-16T15:52:17.783 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:17.785+0000 7f248affd640 1 --2- 172.21.15.123:0/3105416681 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f2478073a90 0x7f2478075f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:52:17.783 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:17.785+0000 7f248affd640 1 -- 172.21.15.123:0/3105416681 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(172..172 src has 1..172) ==== 4971+0+0 (secure 0 0 0) 0x7f2490092bd0 con 0x7f24940b3b20 2024-09-16T15:52:17.783 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:17.785+0000 7f2499c73640 1 --2- 172.21.15.123:0/3105416681 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f2478073a90 0x7f2478075f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:52:17.783 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:17.785+0000 7f2499c73640 1 --2- 172.21.15.123:0/3105416681 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f2478073a90 0x7f2478075f50 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f24840af000 tx=0x7f248401b090 comp rx=0 tx=0).ready entity=mgr.4150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:52:17.786 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:17.789+0000 7f248affd640 1 -- 172.21.15.123:0/3105416681 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f249005f670 con 0x7f24940b3b20 2024-09-16T15:52:17.944 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:17.945+0000 7f249c6d8640 1 -- 172.21.15.123:0/3105416681 --> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "4", "target": ["mon-mgr", ""]}) -- 0x7f249406e360 con 0x7f2478073a90 2024-09-16T15:52:17.945 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:17.945+0000 7f248affd640 1 -- 172.21.15.123:0/3105416681 <== mgr.4150 v2:172.21.15.123:6836/2464759676 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 4 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7f249406e360 con 0x7f2478073a90 2024-09-16T15:52:17.945 INFO:teuthology.orchestra.run.smithi123.stderr:instructed osd(s) 4 to deep-scrub 2024-09-16T15:52:17.948 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:17.949+0000 7f249c6d8640 1 -- 172.21.15.123:0/3105416681 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f2478073a90 msgr2=0x7f2478075f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:52:17.948 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:17.949+0000 7f249c6d8640 1 --2- 172.21.15.123:0/3105416681 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f2478073a90 0x7f2478075f50 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f24840af000 tx=0x7f248401b090 comp rx=0 tx=0).stop 2024-09-16T15:52:17.948 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:17.949+0000 7f249c6d8640 1 -- 172.21.15.123:0/3105416681 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f24940b3b20 msgr2=0x7f2494153e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:52:17.948 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:17.949+0000 7f249c6d8640 1 --2- 172.21.15.123:0/3105416681 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f24940b3b20 0x7f2494153e40 secure :-1 s=READY pgs=2005 cs=0 l=1 rev1=1 crypto rx=0x7f24900058d0 tx=0x7f249000dde0 comp rx=0 tx=0).stop 2024-09-16T15:52:17.948 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:17.949+0000 7f249c6d8640 1 -- 172.21.15.123:0/3105416681 shutdown_connections 2024-09-16T15:52:17.948 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:17.949+0000 7f249c6d8640 1 --2- 172.21.15.123:0/3105416681 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f2478073a90 0x7f2478075f50 unknown :-1 s=CLOSED pgs=460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:52:17.948 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:17.949+0000 7f249c6d8640 1 --2- 172.21.15.123:0/3105416681 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f249415dd60 0x7f249415ba50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:52:17.948 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:17.949+0000 7f249c6d8640 1 --2- 172.21.15.123:0/3105416681 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f24940d4030 0x7f2494153000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:52:17.948 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:17.949+0000 7f249c6d8640 1 --2- 172.21.15.123:0/3105416681 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f24940b3b20 0x7f2494153e40 unknown :-1 s=CLOSED pgs=2005 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:52:17.948 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:17.949+0000 7f249c6d8640 1 -- 172.21.15.123:0/3105416681 >> 172.21.15.123:0/3105416681 conn(0x7f2494069650 msgr2=0x7f24940ca4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:52:17.948 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:17.949+0000 7f249c6d8640 1 -- 172.21.15.123:0/3105416681 shutdown_connections 2024-09-16T15:52:17.948 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:17.949+0000 7f249c6d8640 1 -- 172.21.15.123:0/3105416681 wait complete. 2024-09-16T15:52:17.962 INFO:tasks.ceph:Scrubbing osd.5 2024-09-16T15:52:17.962 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 config set osd_debug_deep_scrub_sleep 0 2024-09-16T15:52:18.080 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:18.081+0000 7f8e09031640 1 -- 172.21.15.123:0/3495481831 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8e04161390 msgr2=0x7f8e040d0630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:52:18.080 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:18.081+0000 7f8e09031640 1 --2- 172.21.15.123:0/3495481831 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8e04161390 0x7f8e040d0630 secure :-1 s=READY pgs=2007 cs=0 l=1 rev1=1 crypto rx=0x7f8df4009900 tx=0x7f8df401c8a0 comp rx=0 tx=0).stop 2024-09-16T15:52:18.080 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:18.081+0000 7f8e09031640 1 -- 172.21.15.123:0/3495481831 shutdown_connections 2024-09-16T15:52:18.080 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:18.081+0000 7f8e09031640 1 --2- 172.21.15.123:0/3495481831 >> v2:172.21.15.184:3300/0 conn(0x7f8e040d2a10 0x7f8e040d0fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:52:18.081 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:18.081+0000 7f8e09031640 1 --2- 172.21.15.123:0/3495481831 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8e04161390 0x7f8e040d0630 unknown :-1 s=CLOSED pgs=2007 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:52:18.081 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:18.081+0000 7f8e09031640 1 -- 172.21.15.123:0/3495481831 >> 172.21.15.123:0/3495481831 conn(0x7f8e040d3e30 msgr2=0x7f8e040d4240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:52:18.081 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:18.081+0000 7f8e09031640 1 -- 172.21.15.123:0/3495481831 shutdown_connections 2024-09-16T15:52:18.081 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:18.081+0000 7f8e09031640 1 -- 172.21.15.123:0/3495481831 wait complete. 2024-09-16T15:52:18.081 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:18.081+0000 7f8e09031640 1 Processor -- start 2024-09-16T15:52:18.081 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:18.081+0000 7f8e09031640 1 -- start start 2024-09-16T15:52:18.083 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:18.081+0000 7f8e09031640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8e040d0b70 0x7f8e0415b9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:52:18.083 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:18.081+0000 7f8e09031640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f8e040d2a10 0x7f8e04152c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:52:18.083 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:18.081+0000 7f8e09031640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f8e04161390 0x7f8e040b0c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:52:18.083 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:18.085+0000 7f8e09031640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f8e0416acd0 con 0x7f8e040d0b70 2024-09-16T15:52:18.083 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:18.085+0000 7f8e09031640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f8e0416ab50 con 0x7f8e040d2a10 2024-09-16T15:52:18.083 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:18.085+0000 7f8e09031640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f8e0416ae50 con 0x7f8e04161390 2024-09-16T15:52:18.084 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:18.085+0000 7f8e02d9d640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8e040d0b70 0x7f8e0415b9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:52:18.084 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:18.085+0000 7f8e02d9d640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8e040d0b70 0x7f8e0415b9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:34290/0 (socket says 172.21.15.123:34290) 2024-09-16T15:52:18.084 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:18.085+0000 7f8e02d9d640 1 -- 172.21.15.123:0/2254760184 learned_addr learned my addr 172.21.15.123:0/2254760184 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:52:18.084 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:18.085+0000 7f8e02d9d640 1 -- 172.21.15.123:0/2254760184 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f8e04161390 msgr2=0x7f8e040b0c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:52:18.084 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:18.085+0000 7f8e02d9d640 1 --2- 172.21.15.123:0/2254760184 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f8e04161390 0x7f8e040b0c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:52:18.084 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:18.085+0000 7f8e02d9d640 1 -- 172.21.15.123:0/2254760184 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f8e040d2a10 msgr2=0x7f8e04152c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:52:18.084 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:18.085+0000 7f8e02d9d640 1 --2- 172.21.15.123:0/2254760184 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f8e040d2a10 0x7f8e04152c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:52:18.084 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:18.085+0000 7f8e02d9d640 1 -- 172.21.15.123:0/2254760184 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8e0415bfd0 con 0x7f8e040d0b70 2024-09-16T15:52:18.084 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:18.085+0000 7f8e02d9d640 1 --2- 172.21.15.123:0/2254760184 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8e040d0b70 0x7f8e0415b9b0 secure :-1 s=READY pgs=2008 cs=0 l=1 rev1=1 crypto rx=0x7f8df4009850 tx=0x7f8df40a5800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:52:18.084 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:18.085+0000 7f8dfb7fe640 1 -- 172.21.15.123:0/2254760184 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8df40b2040 con 0x7f8e040d0b70 2024-09-16T15:52:18.084 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:18.085+0000 7f8dfb7fe640 1 -- 172.21.15.123:0/2254760184 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f8df4003770 con 0x7f8e040d0b70 2024-09-16T15:52:18.084 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:18.085+0000 7f8dfb7fe640 1 -- 172.21.15.123:0/2254760184 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8df4003ba0 con 0x7f8e040d0b70 2024-09-16T15:52:18.084 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:18.085+0000 7f8e09031640 1 -- 172.21.15.123:0/2254760184 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8e0415c2b0 con 0x7f8e040d0b70 2024-09-16T15:52:18.085 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:18.085+0000 7f8e09031640 1 -- 172.21.15.123:0/2254760184 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8e0415c870 con 0x7f8e040d0b70 2024-09-16T15:52:18.086 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:18.085+0000 7f8e09031640 1 -- 172.21.15.123:0/2254760184 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f8e040d0fe0 con 0x7f8e040d0b70 2024-09-16T15:52:18.086 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:18.085+0000 7f8dfb7fe640 1 -- 172.21.15.123:0/2254760184 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 27) ==== 98151+0+0 (secure 0 0 0) 0x7f8df40bf030 con 0x7f8e040d0b70 2024-09-16T15:52:18.086 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:18.085+0000 7f8dfb7fe640 1 --2- 172.21.15.123:0/2254760184 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f8de4073a90 0x7f8de4075f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:52:18.086 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:18.085+0000 7f8dfb7fe640 1 -- 172.21.15.123:0/2254760184 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(172..172 src has 1..172) ==== 4971+0+0 (secure 0 0 0) 0x7f8df412d2f0 con 0x7f8e040d0b70 2024-09-16T15:52:18.086 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:18.085+0000 7f8e0259c640 1 --2- 172.21.15.123:0/2254760184 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f8de4073a90 0x7f8de4075f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:52:18.086 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:18.085+0000 7f8dfb7fe640 1 --2- 172.21.15.123:0/2254760184 >> [v2:172.21.15.184:6808/2680916017,v1:172.21.15.184:6809/2680916017] conn(0x7f8de4079950 0x7f8de407bd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:52:18.086 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:18.085+0000 7f8dfb7fe640 1 -- 172.21.15.123:0/2254760184 --> [v2:172.21.15.184:6808/2680916017,v1:172.21.15.184:6809/2680916017] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f8df4001d40 con 0x7f8de4079950 2024-09-16T15:52:18.086 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:18.085+0000 7f8dfb7fe640 1 -- 172.21.15.123:0/2254760184 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_get_version_reply(handle=1 version=172) ==== 24+0+0 (secure 0 0 0) 0x7f8df412fbd0 con 0x7f8e040d0b70 2024-09-16T15:52:18.086 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:18.085+0000 7f8e0259c640 1 --2- 172.21.15.123:0/2254760184 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f8de4073a90 0x7f8de4075f50 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f8dec000f70 tx=0x7f8dec0079e0 comp rx=0 tx=0).ready entity=mgr.4150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:52:18.086 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:18.085+0000 7f8e0359e640 1 --2- 172.21.15.123:0/2254760184 >> [v2:172.21.15.184:6808/2680916017,v1:172.21.15.184:6809/2680916017] conn(0x7f8de4079950 0x7f8de407bd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:52:18.087 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:18.089+0000 7f8e0359e640 1 --2- 172.21.15.123:0/2254760184 >> [v2:172.21.15.184:6808/2680916017,v1:172.21.15.184:6809/2680916017] conn(0x7f8de4079950 0x7f8de407bd70 crc :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:52:18.088 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:18.089+0000 7f8dfb7fe640 1 -- 172.21.15.123:0/2254760184 <== osd.5 v2:172.21.15.184:6808/2680916017 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f8df4001d40 con 0x7f8de4079950 2024-09-16T15:52:18.100 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:18.101+0000 7f8e09031640 1 -- 172.21.15.123:0/2254760184 --> [v2:172.21.15.184:6808/2680916017,v1:172.21.15.184:6809/2680916017] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f8e040b2200 con 0x7f8de4079950 2024-09-16T15:52:18.101 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:18.101+0000 7f8dfb7fe640 1 -- 172.21.15.123:0/2254760184 <== osd.5 v2:172.21.15.184:6808/2680916017 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7f8e040b2200 con 0x7f8de4079950 2024-09-16T15:52:18.101 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2024-09-16T15:52:18.101 INFO:teuthology.orchestra.run.smithi123.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-16T15:52:18.101 INFO:teuthology.orchestra.run.smithi123.stdout:} 2024-09-16T15:52:18.102 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:18.101+0000 7f8e09031640 1 -- 172.21.15.123:0/2254760184 >> [v2:172.21.15.184:6808/2680916017,v1:172.21.15.184:6809/2680916017] conn(0x7f8de4079950 msgr2=0x7f8de407bd70 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:52:18.102 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:18.101+0000 7f8e09031640 1 --2- 172.21.15.123:0/2254760184 >> [v2:172.21.15.184:6808/2680916017,v1:172.21.15.184:6809/2680916017] conn(0x7f8de4079950 0x7f8de407bd70 crc :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:52:18.102 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:18.101+0000 7f8e09031640 1 -- 172.21.15.123:0/2254760184 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f8de4073a90 msgr2=0x7f8de4075f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:52:18.103 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:18.101+0000 7f8e09031640 1 --2- 172.21.15.123:0/2254760184 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f8de4073a90 0x7f8de4075f50 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f8dec000f70 tx=0x7f8dec0079e0 comp rx=0 tx=0).stop 2024-09-16T15:52:18.103 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:18.101+0000 7f8e09031640 1 -- 172.21.15.123:0/2254760184 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8e040d0b70 msgr2=0x7f8e0415b9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:52:18.103 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:18.101+0000 7f8e09031640 1 --2- 172.21.15.123:0/2254760184 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8e040d0b70 0x7f8e0415b9b0 secure :-1 s=READY pgs=2008 cs=0 l=1 rev1=1 crypto rx=0x7f8df4009850 tx=0x7f8df40a5800 comp rx=0 tx=0).stop 2024-09-16T15:52:18.103 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:18.101+0000 7f8e0359e640 1 -- 172.21.15.123:0/2254760184 reap_dead start 2024-09-16T15:52:18.103 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:18.101+0000 7f8e09031640 1 -- 172.21.15.123:0/2254760184 shutdown_connections 2024-09-16T15:52:18.103 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:18.101+0000 7f8e09031640 1 -- 172.21.15.123:0/2254760184 >> 172.21.15.123:0/2254760184 conn(0x7f8e040d3e30 msgr2=0x7f8e040b93b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:52:18.103 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:18.101+0000 7f8e09031640 1 -- 172.21.15.123:0/2254760184 shutdown_connections 2024-09-16T15:52:18.103 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:18.101+0000 7f8e09031640 1 -- 172.21.15.123:0/2254760184 wait complete. 2024-09-16T15:52:18.116 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 5 2024-09-16T15:52:18.224 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:18.225+0000 7fdae18eb640 1 -- 172.21.15.123:0/1234148009 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdadc15bce0 msgr2=0x7fdadc06cbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:52:18.224 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:18.225+0000 7fdae18eb640 1 --2- 172.21.15.123:0/1234148009 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdadc15bce0 0x7fdadc06cbf0 secure :-1 s=READY pgs=2010 cs=0 l=1 rev1=1 crypto rx=0x7fdacc009c80 tx=0x7fdacc01d7d0 comp rx=0 tx=0).stop 2024-09-16T15:52:18.224 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:18.225+0000 7fdae18eb640 1 -- 172.21.15.123:0/1234148009 shutdown_connections 2024-09-16T15:52:18.224 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:18.225+0000 7fdae18eb640 1 --2- 172.21.15.123:0/1234148009 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdadc15bce0 0x7fdadc06cbf0 unknown :-1 s=CLOSED pgs=2010 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:52:18.224 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:18.225+0000 7fdae18eb640 1 -- 172.21.15.123:0/1234148009 >> 172.21.15.123:0/1234148009 conn(0x7fdadc0d0b70 msgr2=0x7fdadc0d0f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:52:18.224 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:18.225+0000 7fdae18eb640 1 -- 172.21.15.123:0/1234148009 shutdown_connections 2024-09-16T15:52:18.224 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:18.225+0000 7fdae18eb640 1 -- 172.21.15.123:0/1234148009 wait complete. 2024-09-16T15:52:18.225 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:18.225+0000 7fdae18eb640 1 Processor -- start 2024-09-16T15:52:18.225 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:18.225+0000 7fdae18eb640 1 -- start start 2024-09-16T15:52:18.227 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:18.225+0000 7fdae18eb640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fdadc070910 0x7fdadc0a1430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:52:18.227 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:18.225+0000 7fdae18eb640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdadc15bce0 0x7fdadc0c6a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:52:18.227 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:18.225+0000 7fdae18eb640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fdadc15f840 0x7fdadc15d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:52:18.227 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:18.225+0000 7fdae18eb640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fdadc16a080 con 0x7fdadc15bce0 2024-09-16T15:52:18.227 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:18.225+0000 7fdae18eb640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fdadc169f00 con 0x7fdadc070910 2024-09-16T15:52:18.227 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:18.225+0000 7fdae18eb640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fdadc16a200 con 0x7fdadc15f840 2024-09-16T15:52:18.227 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:18.225+0000 7fdada7fc640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdadc15bce0 0x7fdadc0c6a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:52:18.227 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:18.225+0000 7fdada7fc640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdadc15bce0 0x7fdadc0c6a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:34298/0 (socket says 172.21.15.123:34298) 2024-09-16T15:52:18.228 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:18.225+0000 7fdada7fc640 1 -- 172.21.15.123:0/3051038576 learned_addr learned my addr 172.21.15.123:0/3051038576 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:52:18.228 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:18.229+0000 7fdadb7fe640 1 --2- 172.21.15.123:0/3051038576 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fdadc15f840 0x7fdadc15d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:52:18.228 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:18.229+0000 7fdada7fc640 1 -- 172.21.15.123:0/3051038576 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fdadc15f840 msgr2=0x7fdadc15d390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:52:18.228 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:18.229+0000 7fdada7fc640 1 --2- 172.21.15.123:0/3051038576 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fdadc15f840 0x7fdadc15d390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:52:18.228 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:18.229+0000 7fdadaffd640 1 --2- 172.21.15.123:0/3051038576 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fdadc070910 0x7fdadc0a1430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:52:18.228 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:18.229+0000 7fdada7fc640 1 -- 172.21.15.123:0/3051038576 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fdadc070910 msgr2=0x7fdadc0a1430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:52:18.228 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:18.229+0000 7fdada7fc640 1 --2- 172.21.15.123:0/3051038576 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fdadc070910 0x7fdadc0a1430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:52:18.228 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:18.229+0000 7fdada7fc640 1 -- 172.21.15.123:0/3051038576 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdadc15dae0 con 0x7fdadc15bce0 2024-09-16T15:52:18.228 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:18.229+0000 7fdada7fc640 1 --2- 172.21.15.123:0/3051038576 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdadc15bce0 0x7fdadc0c6a80 secure :-1 s=READY pgs=2011 cs=0 l=1 rev1=1 crypto rx=0x7fdac400c490 tx=0x7fdac4001e80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:52:18.228 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:18.229+0000 7fdad3fff640 1 -- 172.21.15.123:0/3051038576 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdac4017070 con 0x7fdadc15bce0 2024-09-16T15:52:18.228 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:18.229+0000 7fdad3fff640 1 -- 172.21.15.123:0/3051038576 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fdac4003600 con 0x7fdadc15bce0 2024-09-16T15:52:18.228 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:18.229+0000 7fdae18eb640 1 -- 172.21.15.123:0/3051038576 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdadc15ddc0 con 0x7fdadc15bce0 2024-09-16T15:52:18.228 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:18.229+0000 7fdad3fff640 1 -- 172.21.15.123:0/3051038576 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdac4003a90 con 0x7fdadc15bce0 2024-09-16T15:52:18.228 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:18.229+0000 7fdae18eb640 1 -- 172.21.15.123:0/3051038576 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdadc0ce870 con 0x7fdadc15bce0 2024-09-16T15:52:18.229 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:18.229+0000 7fdae18eb640 1 -- 172.21.15.123:0/3051038576 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdadc0cea70 con 0x7fdadc15bce0 2024-09-16T15:52:18.229 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:18.229+0000 7fdad3fff640 1 -- 172.21.15.123:0/3051038576 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 27) ==== 98151+0+0 (secure 0 0 0) 0x7fdac401a070 con 0x7fdadc15bce0 2024-09-16T15:52:18.229 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:18.229+0000 7fdad3fff640 1 --2- 172.21.15.123:0/3051038576 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fdabc073a90 0x7fdabc075f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:52:18.230 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:18.229+0000 7fdadaffd640 1 --2- 172.21.15.123:0/3051038576 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fdabc073a90 0x7fdabc075f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:52:18.230 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:18.229+0000 7fdad3fff640 1 -- 172.21.15.123:0/3051038576 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(172..172 src has 1..172) ==== 4971+0+0 (secure 0 0 0) 0x7fdac4091da0 con 0x7fdadc15bce0 2024-09-16T15:52:18.230 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:18.229+0000 7fdadaffd640 1 --2- 172.21.15.123:0/3051038576 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fdabc073a90 0x7fdabc075f50 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7fdacc0af000 tx=0x7fdacc01b5b0 comp rx=0 tx=0).ready entity=mgr.4150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:52:18.233 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:18.233+0000 7fdad3fff640 1 -- 172.21.15.123:0/3051038576 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7fdadc0cea70 con 0x7fdadc15bce0 2024-09-16T15:52:18.392 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:18.393+0000 7fdae18eb640 1 -- 172.21.15.123:0/3051038576 --> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "5", "target": ["mon-mgr", ""]}) -- 0x7fdadc073940 con 0x7fdabc073a90 2024-09-16T15:52:18.392 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:18.393+0000 7fdad3fff640 1 -- 172.21.15.123:0/3051038576 <== mgr.4150 v2:172.21.15.123:6836/2464759676 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 5 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7fdadc073940 con 0x7fdabc073a90 2024-09-16T15:52:18.393 INFO:teuthology.orchestra.run.smithi123.stderr:instructed osd(s) 5 to deep-scrub 2024-09-16T15:52:18.395 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:18.397+0000 7fdae18eb640 1 -- 172.21.15.123:0/3051038576 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fdabc073a90 msgr2=0x7fdabc075f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:52:18.395 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:18.397+0000 7fdae18eb640 1 --2- 172.21.15.123:0/3051038576 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fdabc073a90 0x7fdabc075f50 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7fdacc0af000 tx=0x7fdacc01b5b0 comp rx=0 tx=0).stop 2024-09-16T15:52:18.395 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:18.397+0000 7fdae18eb640 1 -- 172.21.15.123:0/3051038576 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdadc15bce0 msgr2=0x7fdadc0c6a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:52:18.395 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:18.397+0000 7fdae18eb640 1 --2- 172.21.15.123:0/3051038576 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdadc15bce0 0x7fdadc0c6a80 secure :-1 s=READY pgs=2011 cs=0 l=1 rev1=1 crypto rx=0x7fdac400c490 tx=0x7fdac4001e80 comp rx=0 tx=0).stop 2024-09-16T15:52:18.395 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:18.397+0000 7fdae18eb640 1 -- 172.21.15.123:0/3051038576 shutdown_connections 2024-09-16T15:52:18.396 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:18.397+0000 7fdae18eb640 1 --2- 172.21.15.123:0/3051038576 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fdabc073a90 0x7fdabc075f50 unknown :-1 s=CLOSED pgs=462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:52:18.396 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:18.397+0000 7fdae18eb640 1 --2- 172.21.15.123:0/3051038576 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fdadc15f840 0x7fdadc15d390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:52:18.396 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:18.397+0000 7fdae18eb640 1 --2- 172.21.15.123:0/3051038576 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdadc15bce0 0x7fdadc0c6a80 unknown :-1 s=CLOSED pgs=2011 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:52:18.396 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:18.397+0000 7fdae18eb640 1 --2- 172.21.15.123:0/3051038576 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fdadc070910 0x7fdadc0a1430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:52:18.396 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:18.397+0000 7fdae18eb640 1 -- 172.21.15.123:0/3051038576 >> 172.21.15.123:0/3051038576 conn(0x7fdadc0d0b70 msgr2=0x7fdadc0b5760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:52:18.396 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:18.397+0000 7fdae18eb640 1 -- 172.21.15.123:0/3051038576 shutdown_connections 2024-09-16T15:52:18.396 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:18.397+0000 7fdae18eb640 1 -- 172.21.15.123:0/3051038576 wait complete. 2024-09-16T15:52:18.410 INFO:tasks.ceph:Scrubbing osd.6 2024-09-16T15:52:18.410 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 config set osd_debug_deep_scrub_sleep 0 2024-09-16T15:52:18.529 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:18.529+0000 7f9b3b5e3640 1 -- 172.21.15.123:0/3406718066 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9b34073c20 msgr2=0x7f9b3415b000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:52:18.529 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:18.529+0000 7f9b3b5e3640 1 --2- 172.21.15.123:0/3406718066 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9b34073c20 0x7f9b3415b000 secure :-1 s=READY pgs=2012 cs=0 l=1 rev1=1 crypto rx=0x7f9b341556a0 tx=0x7f9b2c01ccf0 comp rx=0 tx=0).stop 2024-09-16T15:52:18.529 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:18.529+0000 7f9b3b5e3640 1 -- 172.21.15.123:0/3406718066 shutdown_connections 2024-09-16T15:52:18.529 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:18.529+0000 7f9b3b5e3640 1 --2- 172.21.15.123:0/3406718066 >> v2:172.21.15.184:3300/0 conn(0x7f9b34070640 0x7f9b340b3260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:52:18.529 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:18.529+0000 7f9b3b5e3640 1 --2- 172.21.15.123:0/3406718066 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9b34073c20 0x7f9b3415b000 unknown :-1 s=CLOSED pgs=2012 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:52:18.529 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:18.529+0000 7f9b3b5e3640 1 -- 172.21.15.123:0/3406718066 >> 172.21.15.123:0/3406718066 conn(0x7f9b340b1630 msgr2=0x7f9b340b1a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:52:18.529 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:18.529+0000 7f9b3b5e3640 1 -- 172.21.15.123:0/3406718066 shutdown_connections 2024-09-16T15:52:18.529 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:18.529+0000 7f9b3b5e3640 1 -- 172.21.15.123:0/3406718066 wait complete. 2024-09-16T15:52:18.531 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:18.529+0000 7f9b3b5e3640 1 Processor -- start 2024-09-16T15:52:18.532 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:18.529+0000 7f9b3b5e3640 1 -- start start 2024-09-16T15:52:18.532 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:18.533+0000 7f9b3b5e3640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f9b34070640 0x7f9b34156e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:52:18.532 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:18.533+0000 7f9b3b5e3640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9b34073c20 0x7f9b3414dbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:52:18.532 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:18.533+0000 7f9b3b5e3640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f9b3415b540 0x7f9b3414b470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:52:18.532 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:18.533+0000 7f9b3b5e3640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f9b34167e50 con 0x7f9b34073c20 2024-09-16T15:52:18.532 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:18.533+0000 7f9b3b5e3640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f9b34167cd0 con 0x7f9b3415b540 2024-09-16T15:52:18.532 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:18.533+0000 7f9b3b5e3640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f9b34167fd0 con 0x7f9b34070640 2024-09-16T15:52:18.532 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:18.533+0000 7f9b38b7e640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9b34073c20 0x7f9b3414dbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:52:18.532 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:18.533+0000 7f9b38b7e640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9b34073c20 0x7f9b3414dbf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:34320/0 (socket says 172.21.15.123:34320) 2024-09-16T15:52:18.532 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:18.533+0000 7f9b38b7e640 1 -- 172.21.15.123:0/1670201014 learned_addr learned my addr 172.21.15.123:0/1670201014 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:52:18.532 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:18.533+0000 7f9b3937f640 1 --2- 172.21.15.123:0/1670201014 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f9b34070640 0x7f9b34156e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:52:18.533 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:18.533+0000 7f9b38b7e640 1 -- 172.21.15.123:0/1670201014 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f9b34070640 msgr2=0x7f9b34156e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:52:18.533 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:18.533+0000 7f9b38b7e640 1 --2- 172.21.15.123:0/1670201014 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f9b34070640 0x7f9b34156e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:52:18.533 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:18.533+0000 7f9b38b7e640 1 -- 172.21.15.123:0/1670201014 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f9b3415b540 msgr2=0x7f9b3414b470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:52:18.533 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:18.533+0000 7f9b39b80640 1 --2- 172.21.15.123:0/1670201014 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f9b3415b540 0x7f9b3414b470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:52:18.533 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:18.533+0000 7f9b38b7e640 1 --2- 172.21.15.123:0/1670201014 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f9b3415b540 0x7f9b3414b470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:52:18.533 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:18.533+0000 7f9b38b7e640 1 -- 172.21.15.123:0/1670201014 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9b3406c5f0 con 0x7f9b34073c20 2024-09-16T15:52:18.533 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:18.533+0000 7f9b3937f640 1 --2- 172.21.15.123:0/1670201014 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f9b34070640 0x7f9b34156e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T15:52:18.533 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:18.533+0000 7f9b39b80640 1 --2- 172.21.15.123:0/1670201014 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f9b3415b540 0x7f9b3414b470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T15:52:18.533 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:18.533+0000 7f9b38b7e640 1 --2- 172.21.15.123:0/1670201014 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9b34073c20 0x7f9b3414dbf0 secure :-1 s=READY pgs=2013 cs=0 l=1 rev1=1 crypto rx=0x7f9b2800b3e0 tx=0x7f9b2800b8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:52:18.533 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:18.533+0000 7f9b25ffb640 1 -- 172.21.15.123:0/1670201014 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b28015020 con 0x7f9b34073c20 2024-09-16T15:52:18.533 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:18.533+0000 7f9b25ffb640 1 -- 172.21.15.123:0/1670201014 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f9b28011940 con 0x7f9b34073c20 2024-09-16T15:52:18.533 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:18.533+0000 7f9b3b5e3640 1 -- 172.21.15.123:0/1670201014 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9b3414a930 con 0x7f9b34073c20 2024-09-16T15:52:18.533 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:18.533+0000 7f9b25ffb640 1 -- 172.21.15.123:0/1670201014 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b28011d90 con 0x7f9b34073c20 2024-09-16T15:52:18.534 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:18.533+0000 7f9b3b5e3640 1 -- 172.21.15.123:0/1670201014 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9b34153880 con 0x7f9b34073c20 2024-09-16T15:52:18.534 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:18.533+0000 7f9b3b5e3640 1 -- 172.21.15.123:0/1670201014 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f9b340b3260 con 0x7f9b34073c20 2024-09-16T15:52:18.535 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:18.533+0000 7f9b25ffb640 1 -- 172.21.15.123:0/1670201014 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 27) ==== 98151+0+0 (secure 0 0 0) 0x7f9b28018020 con 0x7f9b34073c20 2024-09-16T15:52:18.535 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:18.533+0000 7f9b25ffb640 1 --2- 172.21.15.123:0/1670201014 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f9b18073a90 0x7f9b18075f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:52:18.535 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:18.533+0000 7f9b3937f640 1 --2- 172.21.15.123:0/1670201014 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f9b18073a90 0x7f9b18075f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:52:18.535 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:18.533+0000 7f9b25ffb640 1 -- 172.21.15.123:0/1670201014 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(172..172 src has 1..172) ==== 4971+0+0 (secure 0 0 0) 0x7f9b28092fd0 con 0x7f9b34073c20 2024-09-16T15:52:18.535 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:18.533+0000 7f9b25ffb640 1 --2- 172.21.15.123:0/1670201014 >> [v2:172.21.15.184:6824/2183120457,v1:172.21.15.184:6825/2183120457] conn(0x7f9b18079950 0x7f9b1807bd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:52:18.535 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:18.533+0000 7f9b25ffb640 1 -- 172.21.15.123:0/1670201014 --> [v2:172.21.15.184:6824/2183120457,v1:172.21.15.184:6825/2183120457] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f9b2801deb0 con 0x7f9b18079950 2024-09-16T15:52:18.535 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:18.533+0000 7f9b25ffb640 1 -- 172.21.15.123:0/1670201014 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_get_version_reply(handle=1 version=172) ==== 24+0+0 (secure 0 0 0) 0x7f9b280958b0 con 0x7f9b34073c20 2024-09-16T15:52:18.535 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:18.533+0000 7f9b3937f640 1 --2- 172.21.15.123:0/1670201014 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f9b18073a90 0x7f9b18075f50 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f9b2c0af000 tx=0x7f9b2c01b530 comp rx=0 tx=0).ready entity=mgr.4150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:52:18.535 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:18.533+0000 7f9b39b80640 1 --2- 172.21.15.123:0/1670201014 >> [v2:172.21.15.184:6824/2183120457,v1:172.21.15.184:6825/2183120457] conn(0x7f9b18079950 0x7f9b1807bd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:52:18.535 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:18.537+0000 7f9b39b80640 1 --2- 172.21.15.123:0/1670201014 >> [v2:172.21.15.184:6824/2183120457,v1:172.21.15.184:6825/2183120457] conn(0x7f9b18079950 0x7f9b1807bd70 crc :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:52:18.536 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:18.537+0000 7f9b25ffb640 1 -- 172.21.15.123:0/1670201014 <== osd.6 v2:172.21.15.184:6824/2183120457 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f9b2801deb0 con 0x7f9b18079950 2024-09-16T15:52:18.548 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:18.549+0000 7f9b3b5e3640 1 -- 172.21.15.123:0/1670201014 --> [v2:172.21.15.184:6824/2183120457,v1:172.21.15.184:6825/2183120457] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f9b3406f9b0 con 0x7f9b18079950 2024-09-16T15:52:18.549 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:18.549+0000 7f9b25ffb640 1 -- 172.21.15.123:0/1670201014 <== osd.6 v2:172.21.15.184:6824/2183120457 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7f9b3406f9b0 con 0x7f9b18079950 2024-09-16T15:52:18.549 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2024-09-16T15:52:18.549 INFO:teuthology.orchestra.run.smithi123.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-16T15:52:18.549 INFO:teuthology.orchestra.run.smithi123.stdout:} 2024-09-16T15:52:18.550 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:18.549+0000 7f9b3b5e3640 1 -- 172.21.15.123:0/1670201014 >> [v2:172.21.15.184:6824/2183120457,v1:172.21.15.184:6825/2183120457] conn(0x7f9b18079950 msgr2=0x7f9b1807bd70 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:52:18.550 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:18.549+0000 7f9b3b5e3640 1 --2- 172.21.15.123:0/1670201014 >> [v2:172.21.15.184:6824/2183120457,v1:172.21.15.184:6825/2183120457] conn(0x7f9b18079950 0x7f9b1807bd70 crc :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:52:18.550 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:18.549+0000 7f9b3b5e3640 1 -- 172.21.15.123:0/1670201014 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f9b18073a90 msgr2=0x7f9b18075f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:52:18.550 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:18.549+0000 7f9b3b5e3640 1 --2- 172.21.15.123:0/1670201014 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f9b18073a90 0x7f9b18075f50 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f9b2c0af000 tx=0x7f9b2c01b530 comp rx=0 tx=0).stop 2024-09-16T15:52:18.550 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:18.549+0000 7f9b3b5e3640 1 -- 172.21.15.123:0/1670201014 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9b34073c20 msgr2=0x7f9b3414dbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:52:18.550 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:18.549+0000 7f9b3b5e3640 1 --2- 172.21.15.123:0/1670201014 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9b34073c20 0x7f9b3414dbf0 secure :-1 s=READY pgs=2013 cs=0 l=1 rev1=1 crypto rx=0x7f9b2800b3e0 tx=0x7f9b2800b8a0 comp rx=0 tx=0).stop 2024-09-16T15:52:18.550 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:18.549+0000 7f9b39b80640 1 -- 172.21.15.123:0/1670201014 reap_dead start 2024-09-16T15:52:18.551 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:18.549+0000 7f9b3b5e3640 1 -- 172.21.15.123:0/1670201014 shutdown_connections 2024-09-16T15:52:18.551 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:18.549+0000 7f9b3b5e3640 1 -- 172.21.15.123:0/1670201014 >> 172.21.15.123:0/1670201014 conn(0x7f9b340b1630 msgr2=0x7f9b3415a2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:52:18.551 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:18.549+0000 7f9b3b5e3640 1 -- 172.21.15.123:0/1670201014 shutdown_connections 2024-09-16T15:52:18.551 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:18.549+0000 7f9b3b5e3640 1 -- 172.21.15.123:0/1670201014 wait complete. 2024-09-16T15:52:18.563 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 6 2024-09-16T15:52:18.676 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:18.677+0000 7f887c58a640 1 -- 172.21.15.123:0/1374081683 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8874074470 msgr2=0x7f88740cd6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:52:18.676 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:18.677+0000 7f887c58a640 1 --2- 172.21.15.123:0/1374081683 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8874074470 0x7f88740cd6e0 secure :-1 s=READY pgs=2014 cs=0 l=1 rev1=1 crypto rx=0x7f8874070730 tx=0x7f887001c4e0 comp rx=0 tx=0).stop 2024-09-16T15:52:18.676 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:18.677+0000 7f887c58a640 1 -- 172.21.15.123:0/1374081683 shutdown_connections 2024-09-16T15:52:18.676 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:18.677+0000 7f887c58a640 1 --2- 172.21.15.123:0/1374081683 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f88740cdc20 0x7f88740ca860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:52:18.676 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:18.677+0000 7f887c58a640 1 --2- 172.21.15.123:0/1374081683 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8874074470 0x7f88740cd6e0 unknown :-1 s=CLOSED pgs=2014 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:52:18.676 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:18.677+0000 7f887c58a640 1 --2- 172.21.15.123:0/1374081683 >> v2:172.21.15.184:3300/0 conn(0x7f887415e160 0x7f8874073f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:52:18.676 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:18.677+0000 7f887c58a640 1 -- 172.21.15.123:0/1374081683 >> 172.21.15.123:0/1374081683 conn(0x7f88740cfb10 msgr2=0x7f88740cff20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:52:18.676 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:18.677+0000 7f887c58a640 1 -- 172.21.15.123:0/1374081683 shutdown_connections 2024-09-16T15:52:18.676 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:18.677+0000 7f887c58a640 1 -- 172.21.15.123:0/1374081683 wait complete. 2024-09-16T15:52:18.678 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:18.677+0000 7f887c58a640 1 Processor -- start 2024-09-16T15:52:18.678 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:18.677+0000 7f887c58a640 1 -- start start 2024-09-16T15:52:18.678 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:18.677+0000 7f887c58a640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8874074470 0x7f88740c4fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:52:18.678 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:18.677+0000 7f887c58a640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f88740cdc20 0x7f8874154300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:52:18.678 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:18.677+0000 7f887c58a640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f887415e160 0x7f887406f870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:52:18.679 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:18.677+0000 7f887c58a640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f887415ebb0 con 0x7f8874074470 2024-09-16T15:52:18.679 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:18.677+0000 7f887c58a640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f887415ea30 con 0x7f887415e160 2024-09-16T15:52:18.679 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:18.677+0000 7f887c58a640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f887415ed30 con 0x7f88740cdc20 2024-09-16T15:52:18.679 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:18.677+0000 7f887a326640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8874074470 0x7f88740c4fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:52:18.679 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:18.677+0000 7f887a326640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8874074470 0x7f88740c4fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:34340/0 (socket says 172.21.15.123:34340) 2024-09-16T15:52:18.679 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:18.677+0000 7f887a326640 1 -- 172.21.15.123:0/583850512 learned_addr learned my addr 172.21.15.123:0/583850512 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:52:18.679 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:18.677+0000 7f887a326640 1 -- 172.21.15.123:0/583850512 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f88740cdc20 msgr2=0x7f8874154300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:52:18.679 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:18.677+0000 7f887a326640 1 --2- 172.21.15.123:0/583850512 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f88740cdc20 0x7f8874154300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:52:18.679 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:18.677+0000 7f887a326640 1 -- 172.21.15.123:0/583850512 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f887415e160 msgr2=0x7f887406f870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:52:18.679 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:18.677+0000 7f887ab27640 1 --2- 172.21.15.123:0/583850512 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f887415e160 0x7f887406f870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:52:18.679 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:18.677+0000 7f887a326640 1 --2- 172.21.15.123:0/583850512 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f887415e160 0x7f887406f870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:52:18.679 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:18.677+0000 7f887a326640 1 -- 172.21.15.123:0/583850512 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f88740c54f0 con 0x7f8874074470 2024-09-16T15:52:18.679 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:18.677+0000 7f887a326640 1 --2- 172.21.15.123:0/583850512 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8874074470 0x7f88740c4fb0 secure :-1 s=READY pgs=2015 cs=0 l=1 rev1=1 crypto rx=0x7f886800b410 tx=0x7f886800b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:52:18.680 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:18.677+0000 7f8866ffd640 1 -- 172.21.15.123:0/583850512 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8868015020 con 0x7f8874074470 2024-09-16T15:52:18.680 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:18.681+0000 7f8866ffd640 1 -- 172.21.15.123:0/583850512 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f8868011920 con 0x7f8874074470 2024-09-16T15:52:18.680 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:18.681+0000 7f887c58a640 1 -- 172.21.15.123:0/583850512 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f88740c41f0 con 0x7f8874074470 2024-09-16T15:52:18.680 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:18.681+0000 7f8866ffd640 1 -- 172.21.15.123:0/583850512 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8868011d70 con 0x7f8874074470 2024-09-16T15:52:18.680 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:18.681+0000 7f887c58a640 1 -- 172.21.15.123:0/583850512 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f88740c47b0 con 0x7f8874074470 2024-09-16T15:52:18.681 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:18.681+0000 7f887c58a640 1 -- 172.21.15.123:0/583850512 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f88740ca860 con 0x7f8874074470 2024-09-16T15:52:18.681 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:18.681+0000 7f8866ffd640 1 -- 172.21.15.123:0/583850512 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 27) ==== 98151+0+0 (secure 0 0 0) 0x7f8868018020 con 0x7f8874074470 2024-09-16T15:52:18.681 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:18.681+0000 7f8866ffd640 1 --2- 172.21.15.123:0/583850512 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f8858073a90 0x7f8858075f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:52:18.681 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:18.681+0000 7f8866ffd640 1 -- 172.21.15.123:0/583850512 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(172..172 src has 1..172) ==== 4971+0+0 (secure 0 0 0) 0x7f8868092990 con 0x7f8874074470 2024-09-16T15:52:18.681 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:18.681+0000 7f8879b25640 1 --2- 172.21.15.123:0/583850512 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f8858073a90 0x7f8858075f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:52:18.681 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:18.681+0000 7f8879b25640 1 --2- 172.21.15.123:0/583850512 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f8858073a90 0x7f8858075f50 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f88700af000 tx=0x7f8870019280 comp rx=0 tx=0).ready entity=mgr.4150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:52:18.684 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:18.685+0000 7f8866ffd640 1 -- 172.21.15.123:0/583850512 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f886801f020 con 0x7f8874074470 2024-09-16T15:52:18.844 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:18.845+0000 7f887c58a640 1 -- 172.21.15.123:0/583850512 --> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "6", "target": ["mon-mgr", ""]}) -- 0x7f88740c3f10 con 0x7f8858073a90 2024-09-16T15:52:18.845 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:18.845+0000 7f8866ffd640 1 -- 172.21.15.123:0/583850512 <== mgr.4150 v2:172.21.15.123:6836/2464759676 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 6 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7f88740c3f10 con 0x7f8858073a90 2024-09-16T15:52:18.845 INFO:teuthology.orchestra.run.smithi123.stderr:instructed osd(s) 6 to deep-scrub 2024-09-16T15:52:18.848 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:18.849+0000 7f887c58a640 1 -- 172.21.15.123:0/583850512 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f8858073a90 msgr2=0x7f8858075f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:52:18.848 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:18.849+0000 7f887c58a640 1 --2- 172.21.15.123:0/583850512 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f8858073a90 0x7f8858075f50 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f88700af000 tx=0x7f8870019280 comp rx=0 tx=0).stop 2024-09-16T15:52:18.848 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:18.849+0000 7f887c58a640 1 -- 172.21.15.123:0/583850512 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8874074470 msgr2=0x7f88740c4fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:52:18.848 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:18.849+0000 7f887c58a640 1 --2- 172.21.15.123:0/583850512 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8874074470 0x7f88740c4fb0 secure :-1 s=READY pgs=2015 cs=0 l=1 rev1=1 crypto rx=0x7f886800b410 tx=0x7f886800b8d0 comp rx=0 tx=0).stop 2024-09-16T15:52:18.848 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:18.849+0000 7f887c58a640 1 -- 172.21.15.123:0/583850512 shutdown_connections 2024-09-16T15:52:18.848 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:18.849+0000 7f887c58a640 1 --2- 172.21.15.123:0/583850512 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f8858073a90 0x7f8858075f50 unknown :-1 s=CLOSED pgs=464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:52:18.848 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:18.849+0000 7f887c58a640 1 --2- 172.21.15.123:0/583850512 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f887415e160 0x7f887406f870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:52:18.848 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:18.849+0000 7f887c58a640 1 --2- 172.21.15.123:0/583850512 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f88740cdc20 0x7f8874154300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:52:18.848 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:18.849+0000 7f887c58a640 1 --2- 172.21.15.123:0/583850512 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8874074470 0x7f88740c4fb0 unknown :-1 s=CLOSED pgs=2015 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:52:18.848 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:18.849+0000 7f887c58a640 1 -- 172.21.15.123:0/583850512 >> 172.21.15.123:0/583850512 conn(0x7f88740cfb10 msgr2=0x7f88740c8bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:52:18.849 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:18.849+0000 7f887c58a640 1 -- 172.21.15.123:0/583850512 shutdown_connections 2024-09-16T15:52:18.849 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:18.849+0000 7f887c58a640 1 -- 172.21.15.123:0/583850512 wait complete. 2024-09-16T15:52:18.862 INFO:tasks.ceph:Scrubbing osd.7 2024-09-16T15:52:18.863 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 config set osd_debug_deep_scrub_sleep 0 2024-09-16T15:52:18.980 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:18.981+0000 7f3d48461640 1 -- 172.21.15.123:0/2635840412 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3d2c003a10 msgr2=0x7f3d2c003e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:52:18.981 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:18.981+0000 7f3d48461640 1 --2- 172.21.15.123:0/2635840412 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3d2c003a10 0x7f3d2c003e30 secure :-1 s=READY pgs=2017 cs=0 l=1 rev1=1 crypto rx=0x7f3d3c009b60 tx=0x7f3d3c01d2b0 comp rx=0 tx=0).stop 2024-09-16T15:52:18.981 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:18.981+0000 7f3d469fe640 1 -- 172.21.15.123:0/2635840412 reap_dead start 2024-09-16T15:52:18.981 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:18.981+0000 7f3d48461640 1 -- 172.21.15.123:0/2635840412 shutdown_connections 2024-09-16T15:52:18.981 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:18.981+0000 7f3d48461640 1 -- 172.21.15.123:0/2635840412 >> 172.21.15.123:0/2635840412 conn(0x7f3d400b1630 msgr2=0x7f3d400b1a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:52:18.981 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:18.981+0000 7f3d48461640 1 -- 172.21.15.123:0/2635840412 shutdown_connections 2024-09-16T15:52:18.981 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:18.981+0000 7f3d48461640 1 -- 172.21.15.123:0/2635840412 wait complete. 2024-09-16T15:52:18.981 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:18.981+0000 7f3d48461640 1 Processor -- start 2024-09-16T15:52:18.982 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:18.981+0000 7f3d48461640 1 -- start start 2024-09-16T15:52:18.984 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:18.985+0000 7f3d48461640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f3d2c003a10 0x7f3d400b2f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:52:18.984 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:18.985+0000 7f3d48461640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3d401fb380 0x7f3d401f8680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:52:18.984 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:18.985+0000 7f3d48461640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3d4015e3a0 0x7f3d400ae970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:52:18.984 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:18.985+0000 7f3d48461640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f3d40166ba0 con 0x7f3d401fb380 2024-09-16T15:52:18.984 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:18.985+0000 7f3d48461640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f3d40166a20 con 0x7f3d4015e3a0 2024-09-16T15:52:18.984 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:18.985+0000 7f3d48461640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f3d40166d20 con 0x7f3d2c003a10 2024-09-16T15:52:18.984 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:18.985+0000 7f3d459fc640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3d401fb380 0x7f3d401f8680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:52:18.984 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:18.985+0000 7f3d459fc640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3d401fb380 0x7f3d401f8680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:34346/0 (socket says 172.21.15.123:34346) 2024-09-16T15:52:18.984 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:18.985+0000 7f3d459fc640 1 -- 172.21.15.123:0/3118770291 learned_addr learned my addr 172.21.15.123:0/3118770291 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:52:18.984 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:18.985+0000 7f3d461fd640 1 --2- 172.21.15.123:0/3118770291 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f3d2c003a10 0x7f3d400b2f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:52:18.984 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:18.985+0000 7f3d469fe640 1 --2- 172.21.15.123:0/3118770291 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3d4015e3a0 0x7f3d400ae970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:52:18.984 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:18.985+0000 7f3d459fc640 1 -- 172.21.15.123:0/3118770291 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f3d2c003a10 msgr2=0x7f3d400b2f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:52:18.984 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:18.985+0000 7f3d459fc640 1 --2- 172.21.15.123:0/3118770291 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f3d2c003a10 0x7f3d400b2f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:52:18.984 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:18.985+0000 7f3d459fc640 1 -- 172.21.15.123:0/3118770291 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3d4015e3a0 msgr2=0x7f3d400ae970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:52:18.985 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:18.985+0000 7f3d459fc640 1 --2- 172.21.15.123:0/3118770291 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3d4015e3a0 0x7f3d400ae970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:52:18.985 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:18.985+0000 7f3d459fc640 1 -- 172.21.15.123:0/3118770291 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3d401fbbd0 con 0x7f3d401fb380 2024-09-16T15:52:18.985 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:18.985+0000 7f3d469fe640 1 --2- 172.21.15.123:0/3118770291 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f3d4015e3a0 0x7f3d400ae970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T15:52:18.985 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:18.985+0000 7f3d459fc640 1 --2- 172.21.15.123:0/3118770291 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3d401fb380 0x7f3d401f8680 secure :-1 s=READY pgs=2018 cs=0 l=1 rev1=1 crypto rx=0x7f3d30001870 tx=0x7f3d3000ea90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:52:18.985 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:18.985+0000 7f3d36ffd640 1 -- 172.21.15.123:0/3118770291 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d30015070 con 0x7f3d401fb380 2024-09-16T15:52:18.985 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:18.985+0000 7f3d36ffd640 1 -- 172.21.15.123:0/3118770291 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f3d3000ca20 con 0x7f3d401fb380 2024-09-16T15:52:18.985 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:18.985+0000 7f3d48461640 1 -- 172.21.15.123:0/3118770291 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3d401fbeb0 con 0x7f3d401fb380 2024-09-16T15:52:18.985 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:18.985+0000 7f3d36ffd640 1 -- 172.21.15.123:0/3118770291 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d30011460 con 0x7f3d401fb380 2024-09-16T15:52:18.985 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:18.985+0000 7f3d48461640 1 -- 172.21.15.123:0/3118770291 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3d401f7ea0 con 0x7f3d401fb380 2024-09-16T15:52:18.985 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:18.985+0000 7f3d48461640 1 -- 172.21.15.123:0/3118770291 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f3d401f82a0 con 0x7f3d401fb380 2024-09-16T15:52:18.986 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:18.985+0000 7f3d36ffd640 1 -- 172.21.15.123:0/3118770291 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 27) ==== 98151+0+0 (secure 0 0 0) 0x7f3d30018070 con 0x7f3d401fb380 2024-09-16T15:52:18.986 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:18.985+0000 7f3d36ffd640 1 --2- 172.21.15.123:0/3118770291 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f3d24073b10 0x7f3d24075fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:52:18.987 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:18.985+0000 7f3d36ffd640 1 -- 172.21.15.123:0/3118770291 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(172..172 src has 1..172) ==== 4971+0+0 (secure 0 0 0) 0x7f3d30092e70 con 0x7f3d401fb380 2024-09-16T15:52:18.987 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:18.985+0000 7f3d461fd640 1 --2- 172.21.15.123:0/3118770291 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f3d24073b10 0x7f3d24075fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:52:18.987 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:18.985+0000 7f3d36ffd640 1 --2- 172.21.15.123:0/3118770291 >> [v2:172.21.15.184:6800/4102957285,v1:172.21.15.184:6801/4102957285] conn(0x7f3d240799d0 0x7f3d2407bdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:52:18.987 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:18.985+0000 7f3d36ffd640 1 -- 172.21.15.123:0/3118770291 --> [v2:172.21.15.184:6800/4102957285,v1:172.21.15.184:6801/4102957285] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f3d30006e50 con 0x7f3d240799d0 2024-09-16T15:52:18.987 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:18.985+0000 7f3d36ffd640 1 -- 172.21.15.123:0/3118770291 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_get_version_reply(handle=1 version=172) ==== 24+0+0 (secure 0 0 0) 0x7f3d3009c050 con 0x7f3d401fb380 2024-09-16T15:52:18.987 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:18.985+0000 7f3d461fd640 1 --2- 172.21.15.123:0/3118770291 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f3d24073b10 0x7f3d24075fd0 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7f3d3c0ae000 tx=0x7f3d3c01a050 comp rx=0 tx=0).ready entity=mgr.4150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:52:18.987 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:18.985+0000 7f3d469fe640 1 --2- 172.21.15.123:0/3118770291 >> [v2:172.21.15.184:6800/4102957285,v1:172.21.15.184:6801/4102957285] conn(0x7f3d240799d0 0x7f3d2407bdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:52:18.987 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:18.989+0000 7f3d469fe640 1 --2- 172.21.15.123:0/3118770291 >> [v2:172.21.15.184:6800/4102957285,v1:172.21.15.184:6801/4102957285] conn(0x7f3d240799d0 0x7f3d2407bdf0 crc :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:52:18.988 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:18.989+0000 7f3d36ffd640 1 -- 172.21.15.123:0/3118770291 <== osd.7 v2:172.21.15.184:6800/4102957285 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f3d30006e50 con 0x7f3d240799d0 2024-09-16T15:52:19.001 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:19.001+0000 7f3d48461640 1 -- 172.21.15.123:0/3118770291 --> [v2:172.21.15.184:6800/4102957285,v1:172.21.15.184:6801/4102957285] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f3d4015fb10 con 0x7f3d240799d0 2024-09-16T15:52:19.001 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:19.001+0000 7f3d36ffd640 1 -- 172.21.15.123:0/3118770291 <== osd.7 v2:172.21.15.184:6800/4102957285 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7f3d4015fb10 con 0x7f3d240799d0 2024-09-16T15:52:19.002 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2024-09-16T15:52:19.002 INFO:teuthology.orchestra.run.smithi123.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-16T15:52:19.002 INFO:teuthology.orchestra.run.smithi123.stdout:} 2024-09-16T15:52:19.003 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:19.001+0000 7f3d48461640 1 -- 172.21.15.123:0/3118770291 >> [v2:172.21.15.184:6800/4102957285,v1:172.21.15.184:6801/4102957285] conn(0x7f3d240799d0 msgr2=0x7f3d2407bdf0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:52:19.003 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:19.001+0000 7f3d48461640 1 --2- 172.21.15.123:0/3118770291 >> [v2:172.21.15.184:6800/4102957285,v1:172.21.15.184:6801/4102957285] conn(0x7f3d240799d0 0x7f3d2407bdf0 crc :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:52:19.003 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:19.001+0000 7f3d48461640 1 -- 172.21.15.123:0/3118770291 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f3d24073b10 msgr2=0x7f3d24075fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:52:19.003 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:19.001+0000 7f3d48461640 1 --2- 172.21.15.123:0/3118770291 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f3d24073b10 0x7f3d24075fd0 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7f3d3c0ae000 tx=0x7f3d3c01a050 comp rx=0 tx=0).stop 2024-09-16T15:52:19.003 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:19.001+0000 7f3d48461640 1 -- 172.21.15.123:0/3118770291 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3d401fb380 msgr2=0x7f3d401f8680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:52:19.003 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:19.001+0000 7f3d48461640 1 --2- 172.21.15.123:0/3118770291 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3d401fb380 0x7f3d401f8680 secure :-1 s=READY pgs=2018 cs=0 l=1 rev1=1 crypto rx=0x7f3d30001870 tx=0x7f3d3000ea90 comp rx=0 tx=0).stop 2024-09-16T15:52:19.003 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:19.001+0000 7f3d469fe640 1 -- 172.21.15.123:0/3118770291 reap_dead start 2024-09-16T15:52:19.003 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:19.001+0000 7f3d48461640 1 -- 172.21.15.123:0/3118770291 shutdown_connections 2024-09-16T15:52:19.003 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:19.001+0000 7f3d48461640 1 -- 172.21.15.123:0/3118770291 >> 172.21.15.123:0/3118770291 conn(0x7f3d400b1630 msgr2=0x7f3d40165f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:52:19.003 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:19.005+0000 7f3d48461640 1 -- 172.21.15.123:0/3118770291 shutdown_connections 2024-09-16T15:52:19.003 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:19.005+0000 7f3d48461640 1 -- 172.21.15.123:0/3118770291 wait complete. 2024-09-16T15:52:19.017 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 7 2024-09-16T15:52:19.128 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:19.129+0000 7ff0b7e26640 1 -- 172.21.15.123:0/3732069943 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff0b006ce50 msgr2=0x7ff0b00d00e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:52:19.128 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:19.129+0000 7ff0b7e26640 1 --2- 172.21.15.123:0/3732069943 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff0b006ce50 0x7ff0b00d00e0 secure :-1 s=READY pgs=2019 cs=0 l=1 rev1=1 crypto rx=0x7ff0ac0098f0 tx=0x7ff0ac01c8b0 comp rx=0 tx=0).stop 2024-09-16T15:52:19.128 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:19.129+0000 7ff0b7e26640 1 -- 172.21.15.123:0/3732069943 shutdown_connections 2024-09-16T15:52:19.129 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:19.129+0000 7ff0b7e26640 1 --2- 172.21.15.123:0/3732069943 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7ff0b0073600 0x7ff0b0152e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:52:19.129 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:19.129+0000 7ff0b7e26640 1 --2- 172.21.15.123:0/3732069943 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff0b006ce50 0x7ff0b00d00e0 unknown :-1 s=CLOSED pgs=2019 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:52:19.129 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:19.129+0000 7ff0b7e26640 1 -- 172.21.15.123:0/3732069943 >> 172.21.15.123:0/3732069943 conn(0x7ff0b00cd5d0 msgr2=0x7ff0b00cd9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:52:19.129 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:19.129+0000 7ff0b7e26640 1 -- 172.21.15.123:0/3732069943 shutdown_connections 2024-09-16T15:52:19.129 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:19.129+0000 7ff0b7e26640 1 -- 172.21.15.123:0/3732069943 wait complete. 2024-09-16T15:52:19.129 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:19.129+0000 7ff0b7e26640 1 Processor -- start 2024-09-16T15:52:19.129 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:19.129+0000 7ff0b7e26640 1 -- start start 2024-09-16T15:52:19.132 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:19.129+0000 7ff0b7e26640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff0b006ce50 0x7ff0b014bf90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:52:19.132 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:19.129+0000 7ff0b7e26640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7ff0b0073600 0x7ff0b015ad50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:52:19.132 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:19.129+0000 7ff0b7e26640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff0b00d0620 0x7ff0b015b290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:52:19.132 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:19.129+0000 7ff0b7e26640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7ff0b015ca10 con 0x7ff0b00d0620 2024-09-16T15:52:19.132 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:19.129+0000 7ff0b7e26640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7ff0b015c890 con 0x7ff0b006ce50 2024-09-16T15:52:19.132 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:19.129+0000 7ff0b7e26640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7ff0b015cb90 con 0x7ff0b0073600 2024-09-16T15:52:19.132 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:19.133+0000 7ff0b63c3640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff0b00d0620 0x7ff0b015b290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:52:19.132 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:19.133+0000 7ff0b63c3640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff0b00d0620 0x7ff0b015b290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:34358/0 (socket says 172.21.15.123:34358) 2024-09-16T15:52:19.132 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:19.133+0000 7ff0b63c3640 1 -- 172.21.15.123:0/1530067077 learned_addr learned my addr 172.21.15.123:0/1530067077 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:52:19.132 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:19.133+0000 7ff0b53c1640 1 --2- 172.21.15.123:0/1530067077 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7ff0b0073600 0x7ff0b015ad50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:52:19.132 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:19.133+0000 7ff0b5bc2640 1 --2- 172.21.15.123:0/1530067077 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff0b006ce50 0x7ff0b014bf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:52:19.132 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:19.133+0000 7ff0b63c3640 1 -- 172.21.15.123:0/1530067077 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7ff0b0073600 msgr2=0x7ff0b015ad50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:52:19.133 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:19.133+0000 7ff0b63c3640 1 --2- 172.21.15.123:0/1530067077 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7ff0b0073600 0x7ff0b015ad50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:52:19.133 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:19.133+0000 7ff0b63c3640 1 -- 172.21.15.123:0/1530067077 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff0b006ce50 msgr2=0x7ff0b014bf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:52:19.133 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:19.133+0000 7ff0b63c3640 1 --2- 172.21.15.123:0/1530067077 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff0b006ce50 0x7ff0b014bf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:52:19.133 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:19.133+0000 7ff0b63c3640 1 -- 172.21.15.123:0/1530067077 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff0b00c4d20 con 0x7ff0b00d0620 2024-09-16T15:52:19.133 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:19.133+0000 7ff0b63c3640 1 --2- 172.21.15.123:0/1530067077 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff0b00d0620 0x7ff0b015b290 secure :-1 s=READY pgs=2020 cs=0 l=1 rev1=1 crypto rx=0x7ff0a800cab0 tx=0x7ff0a800cf70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:52:19.133 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:19.133+0000 7ff0a67fc640 1 -- 172.21.15.123:0/1530067077 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0a800ae20 con 0x7ff0b00d0620 2024-09-16T15:52:19.133 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:19.133+0000 7ff0a67fc640 1 -- 172.21.15.123:0/1530067077 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7ff0a80039a0 con 0x7ff0b00d0620 2024-09-16T15:52:19.133 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:19.133+0000 7ff0b7e26640 1 -- 172.21.15.123:0/1530067077 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff0b00c5cd0 con 0x7ff0b00d0620 2024-09-16T15:52:19.133 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:19.133+0000 7ff0a67fc640 1 -- 172.21.15.123:0/1530067077 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0a8003dd0 con 0x7ff0b00d0620 2024-09-16T15:52:19.133 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:19.133+0000 7ff0b7e26640 1 -- 172.21.15.123:0/1530067077 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff0b014c4d0 con 0x7ff0b00d0620 2024-09-16T15:52:19.133 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:19.133+0000 7ff0b7e26640 1 -- 172.21.15.123:0/1530067077 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff0b015d120 con 0x7ff0b00d0620 2024-09-16T15:52:19.134 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:19.133+0000 7ff0a67fc640 1 -- 172.21.15.123:0/1530067077 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 27) ==== 98151+0+0 (secure 0 0 0) 0x7ff0a8003f90 con 0x7ff0b00d0620 2024-09-16T15:52:19.134 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:19.133+0000 7ff0a67fc640 1 --2- 172.21.15.123:0/1530067077 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7ff094073a90 0x7ff094075f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:52:19.134 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:19.133+0000 7ff0a67fc640 1 -- 172.21.15.123:0/1530067077 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(172..172 src has 1..172) ==== 4971+0+0 (secure 0 0 0) 0x7ff0a8093010 con 0x7ff0b00d0620 2024-09-16T15:52:19.134 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:19.133+0000 7ff0b5bc2640 1 --2- 172.21.15.123:0/1530067077 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7ff094073a90 0x7ff094075f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:52:19.134 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:19.133+0000 7ff0b5bc2640 1 --2- 172.21.15.123:0/1530067077 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7ff094073a90 0x7ff094075f50 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7ff0ac0af000 tx=0x7ff0ac01b0f0 comp rx=0 tx=0).ready entity=mgr.4150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:52:19.137 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:19.137+0000 7ff0a67fc640 1 -- 172.21.15.123:0/1530067077 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7ff0a8014030 con 0x7ff0b00d0620 2024-09-16T15:52:19.300 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:19.301+0000 7ff0b7e26640 1 -- 172.21.15.123:0/1530067077 --> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "7", "target": ["mon-mgr", ""]}) -- 0x7ff0b015e0e0 con 0x7ff094073a90 2024-09-16T15:52:19.300 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:19.301+0000 7ff0a67fc640 1 -- 172.21.15.123:0/1530067077 <== mgr.4150 v2:172.21.15.123:6836/2464759676 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 7 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7ff0b015e0e0 con 0x7ff094073a90 2024-09-16T15:52:19.300 INFO:teuthology.orchestra.run.smithi123.stderr:instructed osd(s) 7 to deep-scrub 2024-09-16T15:52:19.302 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:19.301+0000 7ff0b7e26640 1 -- 172.21.15.123:0/1530067077 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7ff094073a90 msgr2=0x7ff094075f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:52:19.302 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:19.301+0000 7ff0b7e26640 1 --2- 172.21.15.123:0/1530067077 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7ff094073a90 0x7ff094075f50 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7ff0ac0af000 tx=0x7ff0ac01b0f0 comp rx=0 tx=0).stop 2024-09-16T15:52:19.302 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:19.301+0000 7ff0b7e26640 1 -- 172.21.15.123:0/1530067077 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff0b00d0620 msgr2=0x7ff0b015b290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:52:19.302 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:19.301+0000 7ff0b7e26640 1 --2- 172.21.15.123:0/1530067077 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff0b00d0620 0x7ff0b015b290 secure :-1 s=READY pgs=2020 cs=0 l=1 rev1=1 crypto rx=0x7ff0a800cab0 tx=0x7ff0a800cf70 comp rx=0 tx=0).stop 2024-09-16T15:52:19.302 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:19.301+0000 7ff0b7e26640 1 -- 172.21.15.123:0/1530067077 shutdown_connections 2024-09-16T15:52:19.302 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:19.301+0000 7ff0b7e26640 1 --2- 172.21.15.123:0/1530067077 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7ff094073a90 0x7ff094075f50 unknown :-1 s=CLOSED pgs=466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:52:19.302 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:19.301+0000 7ff0b7e26640 1 --2- 172.21.15.123:0/1530067077 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff0b00d0620 0x7ff0b015b290 unknown :-1 s=CLOSED pgs=2020 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:52:19.303 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:19.301+0000 7ff0b7e26640 1 --2- 172.21.15.123:0/1530067077 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7ff0b0073600 0x7ff0b015ad50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:52:19.303 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:19.301+0000 7ff0b7e26640 1 --2- 172.21.15.123:0/1530067077 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7ff0b006ce50 0x7ff0b014bf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:52:19.303 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:19.301+0000 7ff0b7e26640 1 -- 172.21.15.123:0/1530067077 >> 172.21.15.123:0/1530067077 conn(0x7ff0b00cd5d0 msgr2=0x7ff0b00cfb20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:52:19.303 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:19.305+0000 7ff0b7e26640 1 -- 172.21.15.123:0/1530067077 shutdown_connections 2024-09-16T15:52:19.303 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:19.305+0000 7ff0b7e26640 1 -- 172.21.15.123:0/1530067077 wait complete. 2024-09-16T15:52:19.318 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T15:52:19.437 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:19.437+0000 7f7a68241640 1 -- 172.21.15.123:0/4018514825 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7a4c0039f0 msgr2=0x7f7a4c003df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:52:19.437 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:19.437+0000 7f7a68241640 1 --2- 172.21.15.123:0/4018514825 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7a4c0039f0 0x7f7a4c003df0 secure :-1 s=READY pgs=2022 cs=0 l=1 rev1=1 crypto rx=0x7f7a5c00c770 tx=0x7f7a5c01c6b0 comp rx=0 tx=0).stop 2024-09-16T15:52:19.437 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:19.437+0000 7f7a667de640 1 -- 172.21.15.123:0/4018514825 reap_dead start 2024-09-16T15:52:19.437 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:19.437+0000 7f7a68241640 1 -- 172.21.15.123:0/4018514825 shutdown_connections 2024-09-16T15:52:19.437 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:19.437+0000 7f7a68241640 1 -- 172.21.15.123:0/4018514825 >> 172.21.15.123:0/4018514825 conn(0x7f7a600d3ea0 msgr2=0x7f7a600d42b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:52:19.437 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:19.437+0000 7f7a68241640 1 -- 172.21.15.123:0/4018514825 shutdown_connections 2024-09-16T15:52:19.437 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:19.437+0000 7f7a68241640 1 -- 172.21.15.123:0/4018514825 wait complete. 2024-09-16T15:52:19.438 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:19.437+0000 7f7a68241640 1 Processor -- start 2024-09-16T15:52:19.438 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:19.437+0000 7f7a68241640 1 -- start start 2024-09-16T15:52:19.440 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:19.441+0000 7f7a68241640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7a4c0039f0 0x7f7a6015d510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:52:19.440 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:19.441+0000 7f7a68241640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7a601fb340 0x7f7a6015aae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:52:19.440 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:19.441+0000 7f7a68241640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f7a600b9120 0x7f7a600b95b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:52:19.440 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:19.441+0000 7f7a68241640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f7a601699a0 con 0x7f7a4c0039f0 2024-09-16T15:52:19.440 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:19.441+0000 7f7a68241640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f7a60169820 con 0x7f7a601fb340 2024-09-16T15:52:19.441 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:19.441+0000 7f7a68241640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f7a60169b20 con 0x7f7a600b9120 2024-09-16T15:52:19.441 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:19.441+0000 7f7a65fdd640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7a4c0039f0 0x7f7a6015d510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:52:19.441 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:19.441+0000 7f7a65fdd640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7a4c0039f0 0x7f7a6015d510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:34382/0 (socket says 172.21.15.123:34382) 2024-09-16T15:52:19.441 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:19.441+0000 7f7a65fdd640 1 -- 172.21.15.123:0/3770042270 learned_addr learned my addr 172.21.15.123:0/3770042270 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:52:19.441 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:19.441+0000 7f7a657dc640 1 --2- 172.21.15.123:0/3770042270 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7a601fb340 0x7f7a6015aae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:52:19.441 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:19.441+0000 7f7a667de640 1 --2- 172.21.15.123:0/3770042270 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f7a600b9120 0x7f7a600b95b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:52:19.441 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:19.441+0000 7f7a65fdd640 1 -- 172.21.15.123:0/3770042270 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f7a600b9120 msgr2=0x7f7a600b95b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:52:19.441 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:19.441+0000 7f7a65fdd640 1 --2- 172.21.15.123:0/3770042270 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f7a600b9120 0x7f7a600b95b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:52:19.441 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:19.441+0000 7f7a65fdd640 1 -- 172.21.15.123:0/3770042270 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7a601fb340 msgr2=0x7f7a6015aae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:52:19.442 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:19.441+0000 7f7a65fdd640 1 --2- 172.21.15.123:0/3770042270 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7a601fb340 0x7f7a6015aae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:52:19.442 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:19.441+0000 7f7a65fdd640 1 -- 172.21.15.123:0/3770042270 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7a601f7450 con 0x7f7a4c0039f0 2024-09-16T15:52:19.442 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:19.441+0000 7f7a667de640 1 --2- 172.21.15.123:0/3770042270 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f7a600b9120 0x7f7a600b95b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T15:52:19.442 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:19.441+0000 7f7a65fdd640 1 --2- 172.21.15.123:0/3770042270 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7a4c0039f0 0x7f7a6015d510 secure :-1 s=READY pgs=2023 cs=0 l=1 rev1=1 crypto rx=0x7f7a5c00c690 tx=0x7f7a5c005d30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:52:19.442 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:19.441+0000 7f7a567fc640 1 -- 172.21.15.123:0/3770042270 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a5c006980 con 0x7f7a4c0039f0 2024-09-16T15:52:19.442 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:19.441+0000 7f7a567fc640 1 -- 172.21.15.123:0/3770042270 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f7a5c0064c0 con 0x7f7a4c0039f0 2024-09-16T15:52:19.442 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:19.441+0000 7f7a68241640 1 -- 172.21.15.123:0/3770042270 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7a601fb7a0 con 0x7f7a4c0039f0 2024-09-16T15:52:19.442 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:19.441+0000 7f7a567fc640 1 -- 172.21.15.123:0/3770042270 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a5c00c340 con 0x7f7a4c0039f0 2024-09-16T15:52:19.442 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:19.441+0000 7f7a68241640 1 -- 172.21.15.123:0/3770042270 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7a601f79d0 con 0x7f7a4c0039f0 2024-09-16T15:52:19.442 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:19.441+0000 7f7a68241640 1 -- 172.21.15.123:0/3770042270 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7a600d1da0 con 0x7f7a4c0039f0 2024-09-16T15:52:19.443 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:19.441+0000 7f7a567fc640 1 -- 172.21.15.123:0/3770042270 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 27) ==== 98151+0+0 (secure 0 0 0) 0x7f7a5c0b16e0 con 0x7f7a4c0039f0 2024-09-16T15:52:19.443 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:19.441+0000 7f7a567fc640 1 --2- 172.21.15.123:0/3770042270 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f7a44073a90 0x7f7a44075f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:52:19.443 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:19.441+0000 7f7a567fc640 1 -- 172.21.15.123:0/3770042270 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(172..172 src has 1..172) ==== 4971+0+0 (secure 0 0 0) 0x7f7a5c12dc80 con 0x7f7a4c0039f0 2024-09-16T15:52:19.443 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:19.441+0000 7f7a657dc640 1 --2- 172.21.15.123:0/3770042270 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f7a44073a90 0x7f7a44075f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:52:19.443 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:19.441+0000 7f7a657dc640 1 --2- 172.21.15.123:0/3770042270 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f7a44073a90 0x7f7a44075f50 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f7a50001870 tx=0x7f7a50001cc0 comp rx=0 tx=0).ready entity=mgr.4150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:52:19.445 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:19.445+0000 7f7a567fc640 1 -- 172.21.15.123:0/3770042270 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f7a5c0c2060 con 0x7f7a4c0039f0 2024-09-16T15:52:19.608 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:19.609+0000 7f7a68241640 1 -- 172.21.15.123:0/3770042270 --> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f7a6015f690 con 0x7f7a44073a90 2024-09-16T15:52:19.609 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:19.609+0000 7f7a567fc640 1 -- 172.21.15.123:0/3770042270 <== mgr.4150 v2:172.21.15.123:6836/2464759676 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+13496 (secure 0 0 0) 0x7f7a6015f690 con 0x7f7a44073a90 2024-09-16T15:52:19.610 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-09-16T15:52:19.610 INFO:teuthology.orchestra.run.smithi123.stderr:dumped all 2024-09-16T15:52:19.612 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:19.613+0000 7f7a68241640 1 -- 172.21.15.123:0/3770042270 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f7a44073a90 msgr2=0x7f7a44075f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:52:19.612 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:19.613+0000 7f7a68241640 1 --2- 172.21.15.123:0/3770042270 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f7a44073a90 0x7f7a44075f50 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f7a50001870 tx=0x7f7a50001cc0 comp rx=0 tx=0).stop 2024-09-16T15:52:19.612 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:19.613+0000 7f7a68241640 1 -- 172.21.15.123:0/3770042270 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7a4c0039f0 msgr2=0x7f7a6015d510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:52:19.612 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:19.613+0000 7f7a68241640 1 --2- 172.21.15.123:0/3770042270 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7a4c0039f0 0x7f7a6015d510 secure :-1 s=READY pgs=2023 cs=0 l=1 rev1=1 crypto rx=0x7f7a5c00c690 tx=0x7f7a5c005d30 comp rx=0 tx=0).stop 2024-09-16T15:52:19.612 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:19.613+0000 7f7a68241640 1 -- 172.21.15.123:0/3770042270 shutdown_connections 2024-09-16T15:52:19.612 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:19.613+0000 7f7a68241640 1 --2- 172.21.15.123:0/3770042270 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f7a44073a90 0x7f7a44075f50 unknown :-1 s=CLOSED pgs=467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:52:19.612 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:19.613+0000 7f7a68241640 1 --2- 172.21.15.123:0/3770042270 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f7a600b9120 0x7f7a600b95b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:52:19.612 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:19.613+0000 7f7a68241640 1 --2- 172.21.15.123:0/3770042270 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f7a601fb340 0x7f7a6015aae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:52:19.612 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:19.613+0000 7f7a68241640 1 --2- 172.21.15.123:0/3770042270 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7a4c0039f0 0x7f7a6015d510 unknown :-1 s=CLOSED pgs=2023 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:52:19.612 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:19.613+0000 7f7a68241640 1 -- 172.21.15.123:0/3770042270 >> 172.21.15.123:0/3770042270 conn(0x7f7a600d3ea0 msgr2=0x7f7a600b5920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:52:19.612 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:19.613+0000 7f7a68241640 1 -- 172.21.15.123:0/3770042270 shutdown_connections 2024-09-16T15:52:19.613 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:19.613+0000 7f7a68241640 1 -- 172.21.15.123:0/3770042270 wait complete. 2024-09-16T15:52:19.627 INFO:teuthology.orchestra.run.smithi123.stdout:{"pg_ready":true,"pg_map":{"version":245,"stamp":"2024-09-16T15:52:18.151235+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79,"ondisk_log_size":79,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":51,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":2,"kb":754974720,"kb_used":280152,"kb_used_data":18152,"kb_used_omap":12,"kb_used_meta":261811,"kb_avail":754694568,"statfs":{"total":773094113280,"available":772807237632,"internally_reserved":0,"allocated":18587648,"data_stored":8883568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":268095025},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":23,"apply_latency_ms":23,"commit_latency_ns":23000000,"apply_latency_ns":23000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-53435,"num_objects":-23,"num_object_clones":0,"num_object_copies":-46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-23,"num_whiteouts":0,"num_read":-4,"num_read_kb":-4,"num_write":-81,"num_write_kb":-84,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":-14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001894"},"pg_stats":[{"pgid":"1.0","version":"73'79","reported_seq":417,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-09-16T15:52:11.730445+0000","last_change":"2024-09-16T15:42:40.881820+0000","last_active":"2024-09-16T15:52:11.730445+0000","last_peered":"2024-09-16T15:52:11.730445+0000","last_clean":"2024-09-16T15:52:11.730445+0000","last_became_active":"2024-09-16T15:42:40.879965+0000","last_became_peered":"2024-09-16T15:42:40.879965+0000","last_unstale":"2024-09-16T15:52:11.730445+0000","last_undegraded":"2024-09-16T15:52:11.730445+0000","last_fullsized":"2024-09-16T15:52:11.730445+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T15:42:39.728762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T15:42:39.728762+0000","last_clean_scrub_stamp":"2024-09-16T15:42:39.728762+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:22:18.768932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79,"ondisk_log_size":79,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542263,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33892,"kb_used_data":2704,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94337948,"statfs":{"total":96636764160,"available":96602058752,"internally_reserved":0,"allocated":2768896,"data_stored":1553222,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542262,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":33316,"kb_used_data":2124,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338524,"statfs":{"total":96636764160,"available":96602648576,"internally_reserved":0,"allocated":2174976,"data_stored":962854,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":5,"up_from":14,"seq":60129542263,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":33316,"kb_used_data":2124,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338524,"statfs":{"total":96636764160,"available":96602648576,"internally_reserved":0,"allocated":2174976,"data_stored":962854,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542261,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":37476,"kb_used_data":2124,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334364,"statfs":{"total":96636764160,"available":96598388736,"internally_reserved":0,"allocated":2174976,"data_stored":962854,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":3,"up_from":14,"seq":60129542262,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":37472,"kb_used_data":2124,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334368,"statfs":{"total":96636764160,"available":96598392832,"internally_reserved":0,"allocated":2174976,"data_stored":962854,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":14,"seq":60129542262,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":33312,"kb_used_data":2124,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338528,"statfs":{"total":96636764160,"available":96602652672,"internally_reserved":0,"allocated":2174976,"data_stored":962854,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":14,"seq":60129542264,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":33312,"kb_used_data":2124,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338528,"statfs":{"total":96636764160,"available":96602652672,"internally_reserved":0,"allocated":2174976,"data_stored":962854,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":14,"seq":60129542264,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38056,"kb_used_data":2704,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94333784,"statfs":{"total":96636764160,"available":96597794816,"internally_reserved":0,"allocated":2768896,"data_stored":1553222,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T15:52:19.630 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2024-09-16T15:42:39.728762+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=15, tm_min=42, tm_sec=39, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=15, tm_min=52, tm_sec=14, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T15:52:19.630 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-09-16T15:52:39.632 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T15:52:39.767 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:39.769+0000 7f34bf5c5640 1 -- 172.21.15.123:0/1235133719 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f34b80726c0 msgr2=0x7f34b80d0630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:52:39.767 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:39.769+0000 7f34bf5c5640 1 --2- 172.21.15.123:0/1235133719 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f34b80726c0 0x7f34b80d0630 secure :-1 s=READY pgs=2025 cs=0 l=1 rev1=1 crypto rx=0x7f34b806c970 tx=0x7f34a801c740 comp rx=0 tx=0).stop 2024-09-16T15:52:39.767 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:39.769+0000 7f34bf5c5640 1 -- 172.21.15.123:0/1235133719 shutdown_connections 2024-09-16T15:52:39.767 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:39.769+0000 7f34bf5c5640 1 --2- 172.21.15.123:0/1235133719 >> v2:172.21.15.184:3300/0 conn(0x7f34b8074600 0x7f34b806e750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:52:39.768 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:39.769+0000 7f34bf5c5640 1 --2- 172.21.15.123:0/1235133719 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f34b80726c0 0x7f34b80d0630 unknown :-1 s=CLOSED pgs=2025 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:52:39.768 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:39.769+0000 7f34bf5c5640 1 -- 172.21.15.123:0/1235133719 >> 172.21.15.123:0/1235133719 conn(0x7f34b80cfb10 msgr2=0x7f34b80cff20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:52:39.768 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:39.769+0000 7f34bf5c5640 1 -- 172.21.15.123:0/1235133719 shutdown_connections 2024-09-16T15:52:39.768 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:39.769+0000 7f34bf5c5640 1 -- 172.21.15.123:0/1235133719 wait complete. 2024-09-16T15:52:39.770 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:39.769+0000 7f34bf5c5640 1 Processor -- start 2024-09-16T15:52:39.770 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:39.769+0000 7f34bf5c5640 1 -- start start 2024-09-16T15:52:39.770 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:39.769+0000 7f34bf5c5640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f34b80726c0 0x7f34b80c6cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:52:39.770 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:39.769+0000 7f34bf5c5640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f34b8074600 0x7f34b81fb980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:52:39.770 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:39.769+0000 7f34bf5c5640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f34b815bca0 0x7f34b80c9970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:52:39.770 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:39.769+0000 7f34bf5c5640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f34b8168bd0 con 0x7f34b8074600 2024-09-16T15:52:39.771 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:39.769+0000 7f34bf5c5640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f34b8168a50 con 0x7f34b80726c0 2024-09-16T15:52:39.771 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:39.769+0000 7f34bf5c5640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f34b8168d50 con 0x7f34b815bca0 2024-09-16T15:52:39.771 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:39.769+0000 7f34bcb60640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f34b8074600 0x7f34b81fb980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:52:39.771 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:39.769+0000 7f34bcb60640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f34b8074600 0x7f34b81fb980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:45732/0 (socket says 172.21.15.123:45732) 2024-09-16T15:52:39.771 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:39.769+0000 7f34bcb60640 1 -- 172.21.15.123:0/2454340630 learned_addr learned my addr 172.21.15.123:0/2454340630 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:52:39.771 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:39.769+0000 7f34bcb60640 1 -- 172.21.15.123:0/2454340630 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f34b815bca0 msgr2=0x7f34b80c9970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:52:39.771 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:39.769+0000 7f34bcb60640 1 --2- 172.21.15.123:0/2454340630 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f34b815bca0 0x7f34b80c9970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:52:39.771 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:39.769+0000 7f34bcb60640 1 -- 172.21.15.123:0/2454340630 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f34b80726c0 msgr2=0x7f34b80c6cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:52:39.771 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:39.769+0000 7f34bcb60640 1 --2- 172.21.15.123:0/2454340630 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f34b80726c0 0x7f34b80c6cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:52:39.771 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:39.769+0000 7f34bcb60640 1 -- 172.21.15.123:0/2454340630 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f34b80c9eb0 con 0x7f34b8074600 2024-09-16T15:52:39.771 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:39.769+0000 7f34bcb60640 1 --2- 172.21.15.123:0/2454340630 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f34b8074600 0x7f34b81fb980 secure :-1 s=READY pgs=2026 cs=0 l=1 rev1=1 crypto rx=0x7f34a801cc20 tx=0x7f34a80a5590 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:52:39.771 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:39.769+0000 7f34adffb640 1 -- 172.21.15.123:0/2454340630 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34a80b2040 con 0x7f34b8074600 2024-09-16T15:52:39.772 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:39.769+0000 7f34adffb640 1 -- 172.21.15.123:0/2454340630 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f34a80041b0 con 0x7f34b8074600 2024-09-16T15:52:39.772 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:39.769+0000 7f34bf5c5640 1 -- 172.21.15.123:0/2454340630 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f34b815eec0 con 0x7f34b8074600 2024-09-16T15:52:39.772 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:39.769+0000 7f34adffb640 1 -- 172.21.15.123:0/2454340630 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34a8003600 con 0x7f34b8074600 2024-09-16T15:52:39.772 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:39.769+0000 7f34bf5c5640 1 -- 172.21.15.123:0/2454340630 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f34b81f7b90 con 0x7f34b8074600 2024-09-16T15:52:39.773 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:39.773+0000 7f34bf5c5640 1 -- 172.21.15.123:0/2454340630 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f34b815f3e0 con 0x7f34b8074600 2024-09-16T15:52:39.773 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:39.773+0000 7f34adffb640 1 -- 172.21.15.123:0/2454340630 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 27) ==== 98151+0+0 (secure 0 0 0) 0x7f34a80bd030 con 0x7f34b8074600 2024-09-16T15:52:39.773 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:39.773+0000 7f34adffb640 1 --2- 172.21.15.123:0/2454340630 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f349c073a90 0x7f349c075f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:52:39.773 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:39.773+0000 7f34adffb640 1 -- 172.21.15.123:0/2454340630 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(172..172 src has 1..172) ==== 4971+0+0 (secure 0 0 0) 0x7f34a812cf50 con 0x7f34b8074600 2024-09-16T15:52:39.773 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:39.773+0000 7f34bd361640 1 --2- 172.21.15.123:0/2454340630 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f349c073a90 0x7f349c075f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:52:39.773 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:39.773+0000 7f34bd361640 1 --2- 172.21.15.123:0/2454340630 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f349c073a90 0x7f349c075f50 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f34b4007fd0 tx=0x7f34b4007940 comp rx=0 tx=0).ready entity=mgr.4150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:52:39.775 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:39.777+0000 7f34adffb640 1 -- 172.21.15.123:0/2454340630 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f34b815f3e0 con 0x7f34b8074600 2024-09-16T15:52:39.938 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:39.941+0000 7f34bf5c5640 1 -- 172.21.15.123:0/2454340630 --> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f34b80d0630 con 0x7f349c073a90 2024-09-16T15:52:39.939 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:39.941+0000 7f34adffb640 1 -- 172.21.15.123:0/2454340630 <== mgr.4150 v2:172.21.15.123:6836/2464759676 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+13370 (secure 0 0 0) 0x7f34b80d0630 con 0x7f349c073a90 2024-09-16T15:52:39.939 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-09-16T15:52:39.939 INFO:teuthology.orchestra.run.smithi123.stderr:dumped all 2024-09-16T15:52:39.941 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:39.941+0000 7f34bf5c5640 1 -- 172.21.15.123:0/2454340630 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f349c073a90 msgr2=0x7f349c075f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:52:39.941 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:39.941+0000 7f34bf5c5640 1 --2- 172.21.15.123:0/2454340630 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f349c073a90 0x7f349c075f50 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f34b4007fd0 tx=0x7f34b4007940 comp rx=0 tx=0).stop 2024-09-16T15:52:39.941 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:39.941+0000 7f34bf5c5640 1 -- 172.21.15.123:0/2454340630 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f34b8074600 msgr2=0x7f34b81fb980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:52:39.941 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:39.941+0000 7f34bf5c5640 1 --2- 172.21.15.123:0/2454340630 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f34b8074600 0x7f34b81fb980 secure :-1 s=READY pgs=2026 cs=0 l=1 rev1=1 crypto rx=0x7f34a801cc20 tx=0x7f34a80a5590 comp rx=0 tx=0).stop 2024-09-16T15:52:39.941 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:39.941+0000 7f34bf5c5640 1 -- 172.21.15.123:0/2454340630 shutdown_connections 2024-09-16T15:52:39.941 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:39.941+0000 7f34bf5c5640 1 --2- 172.21.15.123:0/2454340630 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f349c073a90 0x7f349c075f50 unknown :-1 s=CLOSED pgs=468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:52:39.941 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:39.941+0000 7f34bf5c5640 1 --2- 172.21.15.123:0/2454340630 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f34b815bca0 0x7f34b80c9970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:52:39.942 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:39.941+0000 7f34bf5c5640 1 --2- 172.21.15.123:0/2454340630 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f34b8074600 0x7f34b81fb980 unknown :-1 s=CLOSED pgs=2026 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:52:39.942 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:39.941+0000 7f34bf5c5640 1 --2- 172.21.15.123:0/2454340630 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f34b80726c0 0x7f34b80c6cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:52:39.942 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:39.941+0000 7f34bf5c5640 1 -- 172.21.15.123:0/2454340630 >> 172.21.15.123:0/2454340630 conn(0x7f34b80cfb10 msgr2=0x7f34b80b3d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:52:39.942 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:39.941+0000 7f34bf5c5640 1 -- 172.21.15.123:0/2454340630 shutdown_connections 2024-09-16T15:52:39.942 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:39.941+0000 7f34bf5c5640 1 -- 172.21.15.123:0/2454340630 wait complete. 2024-09-16T15:52:39.956 INFO:teuthology.orchestra.run.smithi123.stdout:{"pg_ready":true,"pg_map":{"version":255,"stamp":"2024-09-16T15:52:38.156461+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":81,"ondisk_log_size":81,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":2,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":2,"kb":754974720,"kb_used":280140,"kb_used_data":18152,"kb_used_omap":12,"kb_used_meta":261811,"kb_avail":754694580,"statfs":{"total":773094113280,"available":772807249920,"internally_reserved":0,"allocated":18587648,"data_stored":8883568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":268095025},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003333"},"pg_stats":[{"pgid":"1.0","version":"172'81","reported_seq":427,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-09-16T15:52:20.319621+0000","last_change":"2024-09-16T15:52:20.319621+0000","last_active":"2024-09-16T15:52:20.319621+0000","last_peered":"2024-09-16T15:52:20.319621+0000","last_clean":"2024-09-16T15:52:20.319621+0000","last_became_active":"2024-09-16T15:42:40.879965+0000","last_became_peered":"2024-09-16T15:42:40.879965+0000","last_unstale":"2024-09-16T15:52:20.319621+0000","last_undegraded":"2024-09-16T15:52:20.319621+0000","last_fullsized":"2024-09-16T15:52:20.319621+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"172'81","last_scrub_stamp":"2024-09-16T15:52:20.319417+0000","last_deep_scrub":"172'81","last_deep_scrub_stamp":"2024-09-16T15:52:20.319417+0000","last_clean_scrub_stamp":"2024-09-16T15:52:20.319417+0000","objects_scrubbed":2,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T22:57:05.687351+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":81,"ondisk_log_size":81,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542267,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33888,"kb_used_data":2704,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94337952,"statfs":{"total":96636764160,"available":96602062848,"internally_reserved":0,"allocated":2768896,"data_stored":1553222,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542266,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":33316,"kb_used_data":2124,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338524,"statfs":{"total":96636764160,"available":96602648576,"internally_reserved":0,"allocated":2174976,"data_stored":962854,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":14,"seq":60129542267,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":33316,"kb_used_data":2124,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338524,"statfs":{"total":96636764160,"available":96602648576,"internally_reserved":0,"allocated":2174976,"data_stored":962854,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542265,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":37476,"kb_used_data":2124,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334364,"statfs":{"total":96636764160,"available":96598388736,"internally_reserved":0,"allocated":2174976,"data_stored":962854,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":14,"seq":60129542266,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":37472,"kb_used_data":2124,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334368,"statfs":{"total":96636764160,"available":96598392832,"internally_reserved":0,"allocated":2174976,"data_stored":962854,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":14,"seq":60129542266,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":33312,"kb_used_data":2124,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338528,"statfs":{"total":96636764160,"available":96602652672,"internally_reserved":0,"allocated":2174976,"data_stored":962854,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":14,"seq":60129542268,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":33312,"kb_used_data":2124,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338528,"statfs":{"total":96636764160,"available":96602652672,"internally_reserved":0,"allocated":2174976,"data_stored":962854,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":14,"seq":60129542268,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38048,"kb_used_data":2704,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94333792,"statfs":{"total":96636764160,"available":96597803008,"internally_reserved":0,"allocated":2768896,"data_stored":1553222,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T15:52:39.957 DEBUG:teuthology.orchestra.run.smithi123:> sudo ceph --cluster ceph config set global mon_health_to_clog false 2024-09-16T15:52:40.071 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:40.073+0000 7fdcba107640 1 -- 172.21.15.123:0/203363738 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdcb40ca650 msgr2=0x7fdcb40cbc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:52:40.071 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:40.073+0000 7fdcba107640 1 --2- 172.21.15.123:0/203363738 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdcb40ca650 0x7fdcb40cbc60 secure :-1 s=READY pgs=2028 cs=0 l=1 rev1=1 crypto rx=0x7fdca8009870 tx=0x7fdca801c760 comp rx=0 tx=0).stop 2024-09-16T15:52:40.071 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:40.073+0000 7fdcba107640 1 -- 172.21.15.123:0/203363738 shutdown_connections 2024-09-16T15:52:40.071 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:40.073+0000 7fdcba107640 1 --2- 172.21.15.123:0/203363738 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdcb40ca650 0x7fdcb40cbc60 unknown :-1 s=CLOSED pgs=2028 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:52:40.071 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:40.073+0000 7fdcba107640 1 -- 172.21.15.123:0/203363738 >> 172.21.15.123:0/203363738 conn(0x7fdcb40b1710 msgr2=0x7fdcb40b1b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:52:40.072 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:40.073+0000 7fdcba107640 1 -- 172.21.15.123:0/203363738 shutdown_connections 2024-09-16T15:52:40.072 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:40.073+0000 7fdcba107640 1 -- 172.21.15.123:0/203363738 wait complete. 2024-09-16T15:52:40.072 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:40.073+0000 7fdcba107640 1 Processor -- start 2024-09-16T15:52:40.072 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:40.073+0000 7fdcba107640 1 -- start start 2024-09-16T15:52:40.074 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:40.073+0000 7fdcba107640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdcb40ca650 0x7fdcb4157040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:52:40.074 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:40.073+0000 7fdcba107640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fdcb415a170 0x7fdcb4161ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:52:40.074 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:40.073+0000 7fdcba107640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fdcb4164c70 0x7fdcb4161120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:52:40.074 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:40.073+0000 7fdcba107640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7fdcb406ea70 con 0x7fdcb40ca650 2024-09-16T15:52:40.074 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:40.073+0000 7fdcba107640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7fdcb406e8f0 con 0x7fdcb4164c70 2024-09-16T15:52:40.074 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:40.073+0000 7fdcba107640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7fdcb406ebf0 con 0x7fdcb415a170 2024-09-16T15:52:40.074 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:40.073+0000 7fdcb37fe640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdcb40ca650 0x7fdcb4157040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:52:40.075 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:40.073+0000 7fdcb37fe640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdcb40ca650 0x7fdcb4157040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:45762/0 (socket says 172.21.15.123:45762) 2024-09-16T15:52:40.075 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:40.073+0000 7fdcb37fe640 1 -- 172.21.15.123:0/2187579321 learned_addr learned my addr 172.21.15.123:0/2187579321 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:52:40.075 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:40.073+0000 7fdcb37fe640 1 -- 172.21.15.123:0/2187579321 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fdcb415a170 msgr2=0x7fdcb4161ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:52:40.075 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:40.073+0000 7fdcb37fe640 1 --2- 172.21.15.123:0/2187579321 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fdcb415a170 0x7fdcb4161ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:52:40.075 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:40.073+0000 7fdcb37fe640 1 -- 172.21.15.123:0/2187579321 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fdcb4164c70 msgr2=0x7fdcb4161120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:52:40.075 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:40.073+0000 7fdcb3fff640 1 --2- 172.21.15.123:0/2187579321 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fdcb4164c70 0x7fdcb4161120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:52:40.075 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:40.073+0000 7fdcb37fe640 1 --2- 172.21.15.123:0/2187579321 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fdcb4164c70 0x7fdcb4161120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:52:40.075 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:40.073+0000 7fdcb37fe640 1 -- 172.21.15.123:0/2187579321 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdcb4163f10 con 0x7fdcb40ca650 2024-09-16T15:52:40.075 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:40.073+0000 7fdcb37fe640 1 --2- 172.21.15.123:0/2187579321 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdcb40ca650 0x7fdcb4157040 secure :-1 s=READY pgs=2029 cs=0 l=1 rev1=1 crypto rx=0x7fdca400c430 tx=0x7fdca400c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:52:40.075 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:40.073+0000 7fdc9bfff640 1 -- 172.21.15.123:0/2187579321 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdca4011790 con 0x7fdcb40ca650 2024-09-16T15:52:40.075 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:40.073+0000 7fdc9bfff640 1 -- 172.21.15.123:0/2187579321 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fdca4017070 con 0x7fdcb40ca650 2024-09-16T15:52:40.075 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:40.073+0000 7fdcba107640 1 -- 172.21.15.123:0/2187579321 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdcb41627e0 con 0x7fdcb40ca650 2024-09-16T15:52:40.076 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:40.077+0000 7fdc9bfff640 1 -- 172.21.15.123:0/2187579321 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdca4011ce0 con 0x7fdcb40ca650 2024-09-16T15:52:40.076 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:40.077+0000 7fdcba107640 1 -- 172.21.15.123:0/2187579321 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdcb4164470 con 0x7fdcb40ca650 2024-09-16T15:52:40.076 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:40.077+0000 7fdcba107640 1 -- 172.21.15.123:0/2187579321 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdcb40cbc60 con 0x7fdcb40ca650 2024-09-16T15:52:40.076 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:40.077+0000 7fdc9bfff640 1 -- 172.21.15.123:0/2187579321 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 27) ==== 98151+0+0 (secure 0 0 0) 0x7fdca401c460 con 0x7fdcb40ca650 2024-09-16T15:52:40.077 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:40.077+0000 7fdc9bfff640 1 --2- 172.21.15.123:0/2187579321 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fdca0073a90 0x7fdca0075f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:52:40.077 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:40.077+0000 7fdc9bfff640 1 -- 172.21.15.123:0/2187579321 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(172..172 src has 1..172) ==== 4971+0+0 (secure 0 0 0) 0x7fdca40927a0 con 0x7fdcb40ca650 2024-09-16T15:52:40.077 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:40.077+0000 7fdcb2ffd640 1 --2- 172.21.15.123:0/2187579321 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fdca0073a90 0x7fdca0075f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:52:40.077 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:40.077+0000 7fdcb2ffd640 1 --2- 172.21.15.123:0/2187579321 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fdca0073a90 0x7fdca0075f50 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7fdca80af000 tx=0x7fdca8019500 comp rx=0 tx=0).ready entity=mgr.4150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:52:40.080 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:40.081+0000 7fdc9bfff640 1 -- 172.21.15.123:0/2187579321 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7fdcb4163f10 con 0x7fdcb40ca650 2024-09-16T15:52:40.242 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:40.241+0000 7fdcba107640 1 -- 172.21.15.123:0/2187579321 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command([{prefix=config set, name=mon_health_to_clog}] v 0) -- 0x7fdcb4159830 con 0x7fdcb40ca650 2024-09-16T15:52:40.247 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:40.249+0000 7fdc9bfff640 1 -- 172.21.15.123:0/2187579321 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_health_to_clog}]=0 v50)=0 v50) ==== 123+0+0 (secure 0 0 0) 0x7fdca405f300 con 0x7fdcb40ca650 2024-09-16T15:52:40.249 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:40.249+0000 7fdc9bfff640 1 -- 172.21.15.123:0/2187579321 <== mon.0 v2:172.21.15.123:3300/0 8 ==== config(6 keys) ==== 172+0+0 (secure 0 0 0) 0x7fdca405e710 con 0x7fdcb40ca650 2024-09-16T15:52:40.251 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:40.253+0000 7fdcba107640 1 -- 172.21.15.123:0/2187579321 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fdca0073a90 msgr2=0x7fdca0075f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:52:40.251 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:40.253+0000 7fdcba107640 1 --2- 172.21.15.123:0/2187579321 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fdca0073a90 0x7fdca0075f50 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7fdca80af000 tx=0x7fdca8019500 comp rx=0 tx=0).stop 2024-09-16T15:52:40.251 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:40.253+0000 7fdcba107640 1 -- 172.21.15.123:0/2187579321 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdcb40ca650 msgr2=0x7fdcb4157040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:52:40.251 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:40.253+0000 7fdcba107640 1 --2- 172.21.15.123:0/2187579321 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdcb40ca650 0x7fdcb4157040 secure :-1 s=READY pgs=2029 cs=0 l=1 rev1=1 crypto rx=0x7fdca400c430 tx=0x7fdca400c8f0 comp rx=0 tx=0).stop 2024-09-16T15:52:40.251 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:40.253+0000 7fdcba107640 1 -- 172.21.15.123:0/2187579321 shutdown_connections 2024-09-16T15:52:40.251 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:40.253+0000 7fdcba107640 1 --2- 172.21.15.123:0/2187579321 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7fdca0073a90 0x7fdca0075f50 unknown :-1 s=CLOSED pgs=469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:52:40.251 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:40.253+0000 7fdcba107640 1 --2- 172.21.15.123:0/2187579321 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7fdcb4164c70 0x7fdcb4161120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:52:40.251 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:40.253+0000 7fdcba107640 1 --2- 172.21.15.123:0/2187579321 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7fdcb415a170 0x7fdcb4161ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:52:40.252 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:40.253+0000 7fdcba107640 1 --2- 172.21.15.123:0/2187579321 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdcb40ca650 0x7fdcb4157040 unknown :-1 s=CLOSED pgs=2029 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:52:40.252 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:40.253+0000 7fdcba107640 1 -- 172.21.15.123:0/2187579321 >> 172.21.15.123:0/2187579321 conn(0x7fdcb40b1710 msgr2=0x7fdcb41581d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:52:40.252 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:40.253+0000 7fdcba107640 1 -- 172.21.15.123:0/2187579321 shutdown_connections 2024-09-16T15:52:40.252 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:40.253+0000 7fdcba107640 1 -- 172.21.15.123:0/2187579321 wait complete. 2024-09-16T15:52:40.267 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-09-16T15:52:40.267 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-16T15:52:40.387 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:40.389+0000 7f8571c85640 1 -- 172.21.15.123:0/3721073680 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8558004d50 msgr2=0x7f85580051d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:52:40.387 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:40.389+0000 7f8571c85640 1 --2- 172.21.15.123:0/3721073680 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8558004d50 0x7f85580051d0 secure :-1 s=READY pgs=2031 cs=0 l=1 rev1=1 crypto rx=0x7f856c15fb50 tx=0x7f856401d2a0 comp rx=0 tx=0).stop 2024-09-16T15:52:40.387 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:40.389+0000 7f856bfff640 1 -- 172.21.15.123:0/3721073680 reap_dead start 2024-09-16T15:52:40.387 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:40.389+0000 7f8571c85640 1 -- 172.21.15.123:0/3721073680 shutdown_connections 2024-09-16T15:52:40.387 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:40.389+0000 7f8571c85640 1 -- 172.21.15.123:0/3721073680 >> 172.21.15.123:0/3721073680 conn(0x7f856c0d1c70 msgr2=0x7f856c0d2050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:52:40.387 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:40.389+0000 7f8571c85640 1 -- 172.21.15.123:0/3721073680 shutdown_connections 2024-09-16T15:52:40.388 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:40.389+0000 7f8571c85640 1 -- 172.21.15.123:0/3721073680 wait complete. 2024-09-16T15:52:40.390 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:40.389+0000 7f8571c85640 1 Processor -- start 2024-09-16T15:52:40.390 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:40.389+0000 7f8571c85640 1 -- start start 2024-09-16T15:52:40.390 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:40.389+0000 7f8571c85640 1 --2- >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f8558004d50 0x7f856c15a050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:52:40.391 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:40.389+0000 7f8571c85640 1 --2- >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f856c1f2cf0 0x7f856c0b7470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:52:40.391 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:40.389+0000 7f8571c85640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f856c0b36f0 0x7f856c0b3ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:52:40.391 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:40.389+0000 7f8571c85640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 -- 0x7f856c0b1820 con 0x7f856c0b36f0 2024-09-16T15:52:40.391 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:40.389+0000 7f8571c85640 1 -- --> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] -- mon_getmap magic: 0 -- 0x7f856c0b16a0 con 0x7f8558004d50 2024-09-16T15:52:40.391 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:40.389+0000 7f8571c85640 1 -- --> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] -- mon_getmap magic: 0 -- 0x7f856c0cdbf0 con 0x7f856c1f2cf0 2024-09-16T15:52:40.391 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:40.389+0000 7f856bfff640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f856c0b36f0 0x7f856c0b3ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:52:40.391 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:40.389+0000 7f856bfff640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f856c0b36f0 0x7f856c0b3ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:45776/0 (socket says 172.21.15.123:45776) 2024-09-16T15:52:40.391 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:40.389+0000 7f856bfff640 1 -- 172.21.15.123:0/3573272733 learned_addr learned my addr 172.21.15.123:0/3573272733 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-09-16T15:52:40.391 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:40.389+0000 7f856affd640 1 --2- 172.21.15.123:0/3573272733 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f856c1f2cf0 0x7f856c0b7470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:52:40.391 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:40.389+0000 7f856b7fe640 1 --2- 172.21.15.123:0/3573272733 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f8558004d50 0x7f856c15a050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:52:40.391 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:40.389+0000 7f856bfff640 1 -- 172.21.15.123:0/3573272733 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f856c1f2cf0 msgr2=0x7f856c0b7470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:52:40.391 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:40.389+0000 7f856bfff640 1 --2- 172.21.15.123:0/3573272733 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f856c1f2cf0 0x7f856c0b7470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:52:40.392 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:40.389+0000 7f856bfff640 1 -- 172.21.15.123:0/3573272733 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f8558004d50 msgr2=0x7f856c15a050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:52:40.392 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:40.389+0000 7f856bfff640 1 --2- 172.21.15.123:0/3573272733 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f8558004d50 0x7f856c15a050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:52:40.392 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:40.389+0000 7f856bfff640 1 -- 172.21.15.123:0/3573272733 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f856c0c7aa0 con 0x7f856c0b36f0 2024-09-16T15:52:40.392 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:40.389+0000 7f856bfff640 1 --2- 172.21.15.123:0/3573272733 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f856c0b36f0 0x7f856c0b3ba0 secure :-1 s=READY pgs=2032 cs=0 l=1 rev1=1 crypto rx=0x7f8564098e70 tx=0x7f856401a0e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:52:40.392 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:40.389+0000 7f8563fff640 1 -- 172.21.15.123:0/3573272733 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8564002060 con 0x7f856c0b36f0 2024-09-16T15:52:40.392 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:40.389+0000 7f8563fff640 1 -- 172.21.15.123:0/3573272733 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(6 keys) ==== 172+0+0 (secure 0 0 0) 0x7f85640065a0 con 0x7f856c0b36f0 2024-09-16T15:52:40.392 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:40.389+0000 7f8571c85640 1 -- 172.21.15.123:0/3573272733 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f856c0c7d80 con 0x7f856c0b36f0 2024-09-16T15:52:40.392 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:40.389+0000 7f8563fff640 1 -- 172.21.15.123:0/3573272733 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8564003980 con 0x7f856c0b36f0 2024-09-16T15:52:40.392 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:40.389+0000 7f8571c85640 1 -- 172.21.15.123:0/3573272733 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f856c0b79b0 con 0x7f856c0b36f0 2024-09-16T15:52:40.392 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:40.393+0000 7f8571c85640 1 -- 172.21.15.123:0/3573272733 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f856c0c7fc0 con 0x7f856c0b36f0 2024-09-16T15:52:40.393 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:40.393+0000 7f8563fff640 1 -- 172.21.15.123:0/3573272733 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 27) ==== 98151+0+0 (secure 0 0 0) 0x7f8564003b20 con 0x7f856c0b36f0 2024-09-16T15:52:40.393 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:40.393+0000 7f8563fff640 1 --2- 172.21.15.123:0/3573272733 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f854c073a90 0x7f854c075f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T15:52:40.393 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:40.393+0000 7f856b7fe640 1 --2- 172.21.15.123:0/3573272733 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f854c073a90 0x7f854c075f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T15:52:40.393 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:40.393+0000 7f8563fff640 1 -- 172.21.15.123:0/3573272733 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(172..172 src has 1..172) ==== 4971+0+0 (secure 0 0 0) 0x7f856412d4f0 con 0x7f856c0b36f0 2024-09-16T15:52:40.393 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:40.393+0000 7f856b7fe640 1 --2- 172.21.15.123:0/3573272733 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f854c073a90 0x7f854c075f50 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7f85540058d0 tx=0x7f8554005860 comp rx=0 tx=0).ready entity=mgr.4150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T15:52:40.396 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:40.397+0000 7f8563fff640 1 -- 172.21.15.123:0/3573272733 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188574 (secure 0 0 0) 0x7f856c0c7fc0 con 0x7f856c0b36f0 2024-09-16T15:52:40.538 DEBUG:teuthology.orchestra.run.smithi123:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-16T15:52:40.541 DEBUG:teuthology.orchestra.run.smithi184:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-16T15:52:40.604 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:40.605+0000 7f8571c85640 1 -- 172.21.15.123:0/3573272733 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f856c0b0cb0 con 0x7f856c0b36f0 2024-09-16T15:52:40.605 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:40.605+0000 7f8563fff640 1 -- 172.21.15.123:0/3573272733 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 118 v118) ==== 95+0+3925 (secure 0 0 0) 0x7f85640bf060 con 0x7f856c0b36f0 2024-09-16T15:52:40.605 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-09-16T15:52:40.605 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":118,"btime":"2024-09-16T15:52:09:277078+0000","default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6314,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6833/2070840764","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6832","nonce":2070840764},{"type":"v1","addr":"172.21.15.184:6833","nonce":2070840764}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":110},{"gid":6976,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.184:6835/3881288354","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6834","nonce":3881288354},{"type":"v1","addr":"172.21.15.184:6835","nonce":3881288354}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":110},{"gid":9219,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6833/519008782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6832","nonce":519008782},{"type":"v1","addr":"172.21.15.123:6833","nonce":519008782}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":110},{"gid":9540,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6835/3213729853","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6834","nonce":3213729853},{"type":"v1","addr":"172.21.15.123:6835","nonce":3213729853}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":118}],"filesystems":[]} 2024-09-16T15:52:40.606 INFO:teuthology.orchestra.run.smithi123.stderr:dumped fsmap epoch 118 2024-09-16T15:52:40.608 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:40.609+0000 7f8571c85640 1 -- 172.21.15.123:0/3573272733 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f854c073a90 msgr2=0x7f854c075f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:52:40.608 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:40.609+0000 7f8571c85640 1 --2- 172.21.15.123:0/3573272733 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f854c073a90 0x7f854c075f50 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7f85540058d0 tx=0x7f8554005860 comp rx=0 tx=0).stop 2024-09-16T15:52:40.608 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:40.609+0000 7f8571c85640 1 -- 172.21.15.123:0/3573272733 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f856c0b36f0 msgr2=0x7f856c0b3ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T15:52:40.608 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:40.609+0000 7f8571c85640 1 --2- 172.21.15.123:0/3573272733 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f856c0b36f0 0x7f856c0b3ba0 secure :-1 s=READY pgs=2032 cs=0 l=1 rev1=1 crypto rx=0x7f8564098e70 tx=0x7f856401a0e0 comp rx=0 tx=0).stop 2024-09-16T15:52:40.608 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:40.609+0000 7f8571c85640 1 -- 172.21.15.123:0/3573272733 shutdown_connections 2024-09-16T15:52:40.608 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:40.609+0000 7f8571c85640 1 --2- 172.21.15.123:0/3573272733 >> [v2:172.21.15.123:6836/2464759676,v1:172.21.15.123:6837/2464759676] conn(0x7f854c073a90 0x7f854c075f50 unknown :-1 s=CLOSED pgs=470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:52:40.608 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:40.609+0000 7f8571c85640 1 --2- 172.21.15.123:0/3573272733 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f856c0b36f0 0x7f856c0b3ba0 unknown :-1 s=CLOSED pgs=2032 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:52:40.608 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:40.609+0000 7f8571c85640 1 --2- 172.21.15.123:0/3573272733 >> [v2:172.21.15.184:3301/0,v1:172.21.15.184:6790/0] conn(0x7f856c1f2cf0 0x7f856c0b7470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:52:40.608 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:40.609+0000 7f8571c85640 1 --2- 172.21.15.123:0/3573272733 >> [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] conn(0x7f8558004d50 0x7f856c15a050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T15:52:40.608 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:40.609+0000 7f8571c85640 1 -- 172.21.15.123:0/3573272733 >> 172.21.15.123:0/3573272733 conn(0x7f856c0d1c70 msgr2=0x7f856c071650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T15:52:40.609 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:40.609+0000 7f8571c85640 1 -- 172.21.15.123:0/3573272733 shutdown_connections 2024-09-16T15:52:40.609 INFO:teuthology.orchestra.run.smithi123.stderr:2024-09-16T15:52:40.609+0000 7f8571c85640 1 -- 172.21.15.123:0/3573272733 wait complete. 2024-09-16T15:52:40.623 DEBUG:tasks.cephfs.filesystem:already dead... 2024-09-16T15:52:40.623 INFO:teuthology.misc:Shutting down mds daemons... 2024-09-16T15:52:40.623 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-09-16T15:52:40.623 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T15:52:40.633 INFO:tasks.ceph.mds.a:Stopped 2024-09-16T15:52:40.633 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-09-16T15:52:40.633 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T15:52:40.682 INFO:tasks.ceph.mds.c:Stopped 2024-09-16T15:52:40.682 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-09-16T15:52:40.682 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T15:52:40.695 INFO:tasks.ceph.mds.b:Stopped 2024-09-16T15:52:40.695 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-09-16T15:52:40.695 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T15:52:40.751 INFO:tasks.ceph.mds.d:Stopped 2024-09-16T15:52:40.751 INFO:teuthology.misc:Shutting down osd daemons... 2024-09-16T15:52:40.751 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2024-09-16T15:52:40.751 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T15:52:40.830 INFO:tasks.ceph.osd.0:Stopped 2024-09-16T15:52:40.830 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2024-09-16T15:52:40.830 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T15:52:40.927 INFO:tasks.ceph.osd.1:Stopped 2024-09-16T15:52:40.927 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2024-09-16T15:52:40.927 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T15:52:40.952 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~0s 2024-09-16T15:52:40.952 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~0s 2024-09-16T15:52:46.959 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~0s 2024-09-16T15:52:46.960 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b has been restored 2024-09-16T15:52:46.960 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d has been restored 2024-09-16T15:52:47.029 INFO:tasks.ceph.osd.2:Stopped 2024-09-16T15:52:47.030 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2024-09-16T15:52:47.030 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T15:52:47.109 INFO:tasks.ceph.osd.3:Stopped 2024-09-16T15:52:47.109 DEBUG:tasks.ceph.osd.4:waiting for process to exit 2024-09-16T15:52:47.109 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T15:52:47.182 INFO:tasks.ceph.osd.4:Stopped 2024-09-16T15:52:47.182 DEBUG:tasks.ceph.osd.5:waiting for process to exit 2024-09-16T15:52:47.182 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T15:52:47.278 INFO:tasks.ceph.osd.5:Stopped 2024-09-16T15:52:47.278 DEBUG:tasks.ceph.osd.6:waiting for process to exit 2024-09-16T15:52:47.278 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T15:52:47.369 INFO:tasks.ceph.osd.6:Stopped 2024-09-16T15:52:47.369 DEBUG:tasks.ceph.osd.7:waiting for process to exit 2024-09-16T15:52:47.369 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T15:52:47.455 INFO:tasks.ceph.osd.7:Stopped 2024-09-16T15:52:47.456 INFO:teuthology.misc:Shutting down mgr daemons... 2024-09-16T15:52:47.456 DEBUG:tasks.ceph.mgr.y:waiting for process to exit 2024-09-16T15:52:47.457 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T15:52:47.490 INFO:tasks.ceph.mgr.y:Stopped 2024-09-16T15:52:47.491 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2024-09-16T15:52:47.491 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T15:52:47.522 INFO:tasks.ceph.mgr.x:Stopped 2024-09-16T15:52:47.522 INFO:teuthology.misc:Shutting down mon daemons... 2024-09-16T15:52:47.522 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-09-16T15:52:47.522 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T15:52:47.553 INFO:tasks.ceph.mon.a:Stopped 2024-09-16T15:52:47.553 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-09-16T15:52:47.553 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T15:52:47.575 INFO:tasks.ceph.mon.b:Stopped 2024-09-16T15:52:47.575 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-09-16T15:52:47.575 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T15:52:47.633 INFO:tasks.ceph.mon.c:Stopped 2024-09-16T15:52:47.633 INFO:tasks.ceph:Checking cluster log for badness... 2024-09-16T15:52:47.633 DEBUG:teuthology.orchestra.run.smithi123:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v FS_DEGRADED | egrep -v 'fs.*is degraded' | egrep -v 'filesystem is degraded' | egrep -v FS_INLINE_DATA_DEPRECATED | egrep -v FS_WITH_FAILED_MDS | egrep -v MDS_ALL_DOWN | egrep -v 'filesystem is offline' | egrep -v MDS_DAMAGE | egrep -v MDS_DEGRADED | egrep -v MDS_FAILED | egrep -v MDS_INSUFFICIENT_STANDBY | egrep -v 'insufficient standby MDS daemons available' | egrep -v MDS_UP_LESS_THAN_MAX | egrep -v 'online, but wants' | egrep -v 'filesystem is online with fewer MDS than max_mds' | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'do not have an application enabled' | egrep -v 'overall HEALTH_' | egrep -v 'Replacing daemon' | egrep -v 'deprecated feature inline_data' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | egrep -v PG_AVAILABILITY | egrep -v PG_DEGRADED | egrep -v 'Reduced data availability' | egrep -v 'Degraded data redundancy' | egrep -v 'OSD full dropping all updates' | egrep -v 'OSD near full' | egrep -v 'pausewr flag' | egrep -v 'failsafe engaged, dropping updates' | egrep -v 'failsafe disengaged, no longer dropping' | egrep -v 'is full \(reached quota' | egrep -v POOL_FULL | egrep -v POOL_BACKFILLFULL | head -n 1 2024-09-16T15:52:47.655 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi123.front.sepia.ceph.com 2024-09-16T15:52:47.655 DEBUG:teuthology.orchestra.run.smithi123:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2024-09-16T15:52:48.024 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi123.front.sepia.ceph.com 2024-09-16T15:52:48.024 DEBUG:teuthology.orchestra.run.smithi123:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2024-09-16T15:52:48.251 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi123.front.sepia.ceph.com 2024-09-16T15:52:48.251 DEBUG:teuthology.orchestra.run.smithi123:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2024-09-16T15:52:48.445 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi123.front.sepia.ceph.com 2024-09-16T15:52:48.445 DEBUG:teuthology.orchestra.run.smithi123:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2024-09-16T15:52:48.612 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-4 on ubuntu@smithi184.front.sepia.ceph.com 2024-09-16T15:52:48.612 DEBUG:teuthology.orchestra.run.smithi184:> sync && sudo umount -f /var/lib/ceph/osd/ceph-4 2024-09-16T15:52:48.940 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-5 on ubuntu@smithi184.front.sepia.ceph.com 2024-09-16T15:52:48.940 DEBUG:teuthology.orchestra.run.smithi184:> sync && sudo umount -f /var/lib/ceph/osd/ceph-5 2024-09-16T15:52:49.163 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-6 on ubuntu@smithi184.front.sepia.ceph.com 2024-09-16T15:52:49.163 DEBUG:teuthology.orchestra.run.smithi184:> sync && sudo umount -f /var/lib/ceph/osd/ceph-6 2024-09-16T15:52:49.388 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-7 on ubuntu@smithi184.front.sepia.ceph.com 2024-09-16T15:52:49.388 DEBUG:teuthology.orchestra.run.smithi184:> sync && sudo umount -f /var/lib/ceph/osd/ceph-7 2024-09-16T15:52:49.581 INFO:tasks.ceph:Archiving mon data... 2024-09-16T15:52:49.582 DEBUG:teuthology.misc:Transferring archived files from smithi123:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/avan-2024-09-16_15:21:40-fs:volumes-wip-athakkar-testing-2024-09-16-1454-distro-default-smithi/7907351/data/mon.a.tgz 2024-09-16T15:52:49.582 DEBUG:teuthology.orchestra.run.smithi123:> mktemp 2024-09-16T15:52:49.588 INFO:teuthology.orchestra.run.smithi123.stdout:/tmp/tmp.wjfljBgtm1 2024-09-16T15:52:49.588 DEBUG:teuthology.orchestra.run.smithi123:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.wjfljBgtm1 2024-09-16T15:52:49.843 DEBUG:teuthology.orchestra.run.smithi123:> sudo chmod 0666 /tmp/tmp.wjfljBgtm1 2024-09-16T15:52:49.906 DEBUG:teuthology.orchestra.remote:smithi123:/tmp/tmp.wjfljBgtm1 is 1MB 2024-09-16T15:52:49.976 DEBUG:teuthology.orchestra.run.smithi123:> rm -fr /tmp/tmp.wjfljBgtm1 2024-09-16T15:52:49.982 DEBUG:teuthology.misc:Transferring archived files from smithi184:/var/lib/ceph/mon/ceph-b to /home/teuthworker/archive/avan-2024-09-16_15:21:40-fs:volumes-wip-athakkar-testing-2024-09-16-1454-distro-default-smithi/7907351/data/mon.b.tgz 2024-09-16T15:52:49.982 DEBUG:teuthology.orchestra.run.smithi184:> mktemp 2024-09-16T15:52:49.988 INFO:teuthology.orchestra.run.smithi184.stdout:/tmp/tmp.govi0FrTY5 2024-09-16T15:52:49.988 DEBUG:teuthology.orchestra.run.smithi184:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.govi0FrTY5 2024-09-16T15:52:50.252 DEBUG:teuthology.orchestra.run.smithi184:> sudo chmod 0666 /tmp/tmp.govi0FrTY5 2024-09-16T15:52:50.315 DEBUG:teuthology.orchestra.remote:smithi184:/tmp/tmp.govi0FrTY5 is 1MB 2024-09-16T15:52:50.389 DEBUG:teuthology.orchestra.run.smithi184:> rm -fr /tmp/tmp.govi0FrTY5 2024-09-16T15:52:50.395 DEBUG:teuthology.misc:Transferring archived files from smithi184:/var/lib/ceph/mon/ceph-c to /home/teuthworker/archive/avan-2024-09-16_15:21:40-fs:volumes-wip-athakkar-testing-2024-09-16-1454-distro-default-smithi/7907351/data/mon.c.tgz 2024-09-16T15:52:50.395 DEBUG:teuthology.orchestra.run.smithi184:> mktemp 2024-09-16T15:52:50.444 INFO:teuthology.orchestra.run.smithi184.stdout:/tmp/tmp.ldiZ56gCov 2024-09-16T15:52:50.445 DEBUG:teuthology.orchestra.run.smithi184:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.ldiZ56gCov 2024-09-16T15:52:50.709 DEBUG:teuthology.orchestra.run.smithi184:> sudo chmod 0666 /tmp/tmp.ldiZ56gCov 2024-09-16T15:52:50.775 DEBUG:teuthology.orchestra.remote:smithi184:/tmp/tmp.ldiZ56gCov is 1MB 2024-09-16T15:52:50.841 DEBUG:teuthology.orchestra.run.smithi184:> rm -fr /tmp/tmp.ldiZ56gCov 2024-09-16T15:52:50.847 INFO:tasks.ceph:Cleaning ceph cluster... 2024-09-16T15:52:50.847 DEBUG:teuthology.orchestra.run.smithi123:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-09-16T15:52:50.849 DEBUG:teuthology.orchestra.run.smithi184:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-09-16T15:52:50.910 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-09-16T15:52:50.952 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-09-16T15:52:50.952 INFO:tasks.ceph:Archiving crash dumps... 2024-09-16T15:52:50.953 DEBUG:teuthology.misc:Transferring archived files from smithi123:/var/lib/ceph/crash to /home/teuthworker/archive/avan-2024-09-16_15:21:40-fs:volumes-wip-athakkar-testing-2024-09-16-1454-distro-default-smithi/7907351/remote/smithi123/crash 2024-09-16T15:52:50.954 DEBUG:teuthology.orchestra.run.smithi123:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-09-16T15:52:50.968 DEBUG:teuthology.misc:Transferring archived files from smithi184:/var/lib/ceph/crash to /home/teuthworker/archive/avan-2024-09-16_15:21:40-fs:volumes-wip-athakkar-testing-2024-09-16-1454-distro-default-smithi/7907351/remote/smithi184/crash 2024-09-16T15:52:50.969 DEBUG:teuthology.orchestra.run.smithi184:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-09-16T15:52:51.007 INFO:tasks.ceph:Shutting down logrotate 2024-09-16T15:52:51.008 DEBUG:teuthology.orchestra.run.smithi123:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-16T15:52:51.015 DEBUG:teuthology.orchestra.run.smithi184:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-16T15:52:51.207 DEBUG:teuthology.orchestra.run.smithi123:> sudo rm /etc/logrotate.d/ceph-test.conf 2024-09-16T15:52:51.220 DEBUG:teuthology.orchestra.run.smithi184:> sudo rm /etc/logrotate.d/ceph-test.conf 2024-09-16T15:52:51.232 INFO:tasks.ceph:Compressing logs... 2024-09-16T15:52:51.233 DEBUG:teuthology.orchestra.run.smithi123:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-09-16T15:52:51.263 DEBUG:teuthology.orchestra.run.smithi184:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-09-16T15:52:51.302 INFO:teuthology.orchestra.run.smithi184.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29101.log 2024-09-16T15:52:51.302 INFO:teuthology.orchestra.run.smithi184.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.1.34345.log 2024-09-16T15:52:51.326 INFO:teuthology.orchestra.run.smithi184.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.1.34843.log 2024-09-16T15:52:51.326 INFO:teuthology.orchestra.run.smithi184.stderr:/var/log/ceph/ceph-client.admin.29101.log: /var/log/ceph/ceph-client.1.34345.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.7.log 2024-09-16T15:52:51.326 INFO:teuthology.orchestra.run.smithi184.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.29101.log.gz 2024-09-16T15:52:51.326 INFO:teuthology.orchestra.run.smithi184.stderr:/var/log/ceph/ceph-client.1.34843.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.1.36401.log 2024-09-16T15:52:51.326 INFO:teuthology.orchestra.run.smithi184.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.1.34345.log.gz 2024-09-16T15:52:51.326 INFO:teuthology.orchestra.run.smithi184.stderr:/var/log/ceph/ceph-osd.7.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.6.log 2024-09-16T15:52:51.326 INFO:teuthology.orchestra.run.smithi184.stderr:/var/log/ceph/ceph-client.1.36401.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.1.36213.log 2024-09-16T15:52:51.326 INFO:teuthology.orchestra.run.smithi184.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.1.34843.log.gz 2024-09-16T15:52:51.326 INFO:teuthology.orchestra.run.smithi184.stderr:/var/log/ceph/ceph-osd.6.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.1.35889.log 2024-09-16T15:52:51.326 INFO:teuthology.orchestra.run.smithi184.stderr:/var/log/ceph/ceph-client.1.36213.log: 88.4% -- replaced with /var/log/ceph/ceph-client.1.36401.log.gz 2024-09-16T15:52:51.326 INFO:teuthology.orchestra.run.smithi184.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29046.log 2024-09-16T15:52:51.327 INFO:teuthology.orchestra.run.smithi184.stderr:/var/log/ceph/ceph-client.1.35889.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29022.log 2024-09-16T15:52:51.327 INFO:teuthology.orchestra.run.smithi184.stderr: 90.3% -- replaced with /var/log/ceph/ceph-client.1.36213.log.gz 2024-09-16T15:52:51.327 INFO:teuthology.orchestra.run.smithi184.stderr:/var/log/ceph/ceph-client.admin.29046.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.1.36573.log 2024-09-16T15:52:51.327 INFO:teuthology.orchestra.run.smithi184.stderr:/var/log/ceph/ceph-client.admin.29022.log: 86.8%gzip -- replaced with /var/log/ceph/ceph-client.admin.29046.log.gz 2024-09-16T15:52:51.327 INFO:teuthology.orchestra.run.smithi184.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.28950.log 2024-09-16T15:52:51.327 INFO:teuthology.orchestra.run.smithi184.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.29022.log.gz 2024-09-16T15:52:51.327 INFO:teuthology.orchestra.run.smithi184.stderr:/var/log/ceph/ceph-client.1.36573.log: 89.1% -- replaced with /var/log/ceph/ceph-client.1.35889.log.gz 2024-09-16T15:52:51.327 INFO:teuthology.orchestra.run.smithi184.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.1.34638.log 2024-09-16T15:52:51.327 INFO:teuthology.orchestra.run.smithi184.stderr:gzip -5 --verbose --/var/log/ceph/ceph-client.admin.28950.log: /var/log/ceph/ceph-client.1.36044.log 2024-09-16T15:52:51.327 INFO:teuthology.orchestra.run.smithi184.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.1.36573.log.gz 2024-09-16T15:52:51.328 INFO:teuthology.orchestra.run.smithi184.stderr:/var/log/ceph/ceph-client.1.34638.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.28950.log.gz 2024-09-16T15:52:51.328 INFO:teuthology.orchestra.run.smithi184.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29070.log 2024-09-16T15:52:51.328 INFO:teuthology.orchestra.run.smithi184.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.5.log 2024-09-16T15:52:51.328 INFO:teuthology.orchestra.run.smithi184.stderr:/var/log/ceph/ceph-client.1.36044.log: /var/log/ceph/ceph-client.admin.29070.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.3.34464.log 2024-09-16T15:52:51.328 INFO:teuthology.orchestra.run.smithi184.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.1.34638.log.gz 2024-09-16T15:52:51.328 INFO:teuthology.orchestra.run.smithi184.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.29070.log.gz 2024-09-16T15:52:51.328 INFO:teuthology.orchestra.run.smithi184.stderr:/var/log/ceph/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.1.35318.log 2024-09-16T15:52:51.328 INFO:teuthology.orchestra.run.smithi184.stderr: 89.6% -- replaced with /var/log/ceph/ceph-client.1.36044.log.gz 2024-09-16T15:52:51.328 INFO:teuthology.orchestra.run.smithi184.stderr:/var/log/ceph/ceph-client.3.34464.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2024-09-16T15:52:51.328 INFO:teuthology.orchestra.run.smithi184.stderr:/var/log/ceph/ceph-client.1.35318.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28974.log 2024-09-16T15:52:51.328 INFO:teuthology.orchestra.run.smithi184.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.3.34464.log.gz 2024-09-16T15:52:51.329 INFO:teuthology.orchestra.run.smithi184.stderr:/var/log/ceph/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-09-16T15:52:51.329 INFO:teuthology.orchestra.run.smithi184.stderr:/var/log/ceph/ceph-client.admin.28974.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.1.35114.log 2024-09-16T15:52:51.329 INFO:teuthology.orchestra.run.smithi184.stderr: 89.6% -- replaced with /var/log/ceph/ceph-client.1.35318.log.gz 2024-09-16T15:52:51.329 INFO:teuthology.orchestra.run.smithi184.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.28974.log.gz 2024-09-16T15:52:51.329 INFO:teuthology.orchestra.run.smithi184.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.1.35733.log 2024-09-16T15:52:51.329 INFO:teuthology.orchestra.run.smithi184.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.c.log 2024-09-16T15:52:51.329 INFO:teuthology.orchestra.run.smithi184.stderr:/var/log/ceph/ceph-client.1.35114.log: /var/log/ceph/ceph-client.1.35733.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29125.log 2024-09-16T15:52:51.329 INFO:teuthology.orchestra.run.smithi184.stderr: 89.5%/var/log/ceph/ceph-mon.c.log: -- replaced with /var/log/ceph/ceph-client.1.35114.log.gz 2024-09-16T15:52:51.329 INFO:teuthology.orchestra.run.smithi184.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-09-16T15:52:51.329 INFO:teuthology.orchestra.run.smithi184.stderr: 91.6% -- replaced with /var/log/ceph/ceph-client.1.35733.log.gz 2024-09-16T15:52:51.329 INFO:teuthology.orchestra.run.smithi184.stderr:/var/log/ceph/ceph-client.admin.29125.log: 86.6%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28998.log 2024-09-16T15:52:51.329 INFO:teuthology.orchestra.run.smithi184.stderr: -- replaced with /var/log/ceph/ceph-client.admin.29125.log.gz 2024-09-16T15:52:51.329 INFO:teuthology.orchestra.run.smithi184.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-mds.d.log 2024-09-16T15:52:51.330 INFO:teuthology.orchestra.run.smithi184.stderr:/var/log/ceph/ceph-client.admin.28998.log: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.28998.log.gz 2024-09-16T15:52:51.330 INFO:teuthology.orchestra.run.smithi184.stderr: 92.6% -- replaced with /var/log/ceph/ceph.log.gz 2024-09-16T15:52:51.330 INFO:teuthology.orchestra.run.smithi184.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mds.b.log 2024-09-16T15:52:51.330 INFO:teuthology.orchestra.run.smithi184.stderr:/var/log/ceph/ceph-mds.d.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.b.log 2024-09-16T15:52:51.330 INFO:teuthology.orchestra.run.smithi184.stderr:/var/log/ceph/ceph-mds.b.log: 93.7% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-09-16T15:52:51.330 INFO:teuthology.orchestra.run.smithi184.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.1.35560.log 2024-09-16T15:52:51.340 INFO:teuthology.orchestra.run.smithi184.stderr:/var/log/ceph/ceph-mon.b.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.4.log 2024-09-16T15:52:51.347 INFO:teuthology.orchestra.run.smithi184.stderr:/var/log/ceph/ceph-client.1.35560.log: 89.8% -- replaced with /var/log/ceph/ceph-client.1.35560.log.gz 2024-09-16T15:52:51.416 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45243.log 2024-09-16T15:52:51.416 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35457.log 2024-09-16T15:52:51.416 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.45243.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45452.log 2024-09-16T15:52:51.416 INFO:teuthology.orchestra.run.smithi123.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.45243.log.gz 2024-09-16T15:52:51.416 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.35457.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44622.log 2024-09-16T15:52:51.416 INFO:teuthology.orchestra.run.smithi123.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.35457.log.gz 2024-09-16T15:52:51.416 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.45452.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53004.log 2024-09-16T15:52:51.417 INFO:teuthology.orchestra.run.smithi123.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.45452.log.gz 2024-09-16T15:52:51.417 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.44622.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43077.log 2024-09-16T15:52:51.417 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.53004.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.44622.log.gz 2024-09-16T15:52:51.417 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53738.log 2024-09-16T15:52:51.417 INFO:teuthology.orchestra.run.smithi123.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.53004.log.gz 2024-09-16T15:52:51.417 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.43077.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52700.log 2024-09-16T15:52:51.417 INFO:teuthology.orchestra.run.smithi123.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.43077.log.gz 2024-09-16T15:52:51.417 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.53738.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37851.log 2024-09-16T15:52:51.417 INFO:teuthology.orchestra.run.smithi123.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.53738.log.gz/var/log/ceph/ceph-client.admin.52700.log: 2024-09-16T15:52:51.417 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43050.log 2024-09-16T15:52:51.417 INFO:teuthology.orchestra.run.smithi123.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.52700.log.gz 2024-09-16T15:52:51.418 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.37851.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50496.log 2024-09-16T15:52:51.418 INFO:teuthology.orchestra.run.smithi123.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.37851.log.gz 2024-09-16T15:52:51.418 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.43050.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.42891.log 2024-09-16T15:52:51.418 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.50496.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.43050.log.gz 2024-09-16T15:52:51.418 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53660.log 2024-09-16T15:52:51.418 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.0.42891.log: 85.4% -- replaced with /var/log/ceph/ceph-client.admin.50496.log.gz 2024-09-16T15:52:51.418 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35441.log 2024-09-16T15:52:51.418 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.53660.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38530.log 2024-09-16T15:52:51.418 INFO:teuthology.orchestra.run.smithi123.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.0.42891.log.gz 2024-09-16T15:52:51.418 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.35441.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.53660.log.gz 2024-09-16T15:52:51.418 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35923.log 2024-09-16T15:52:51.418 INFO:teuthology.orchestra.run.smithi123.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.35441.log.gz 2024-09-16T15:52:51.419 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.38530.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53602.log 2024-09-16T15:52:51.419 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.35923.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.38530.log.gz 2024-09-16T15:52:51.419 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43161.log 2024-09-16T15:52:51.419 INFO:teuthology.orchestra.run.smithi123.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.35923.log.gz 2024-09-16T15:52:51.419 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.53602.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49982.log 2024-09-16T15:52:51.419 INFO:teuthology.orchestra.run.smithi123.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.53602.log.gz 2024-09-16T15:52:51.419 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.43161.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39773.log 2024-09-16T15:52:51.419 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.49982.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.43161.log.gz 2024-09-16T15:52:51.419 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39031.log 2024-09-16T15:52:51.419 INFO:teuthology.orchestra.run.smithi123.stderr: 84.5% -- replaced with /var/log/ceph/ceph-client.admin.49982.log.gz 2024-09-16T15:52:51.419 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.39773.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42705.log 2024-09-16T15:52:51.419 INFO:teuthology.orchestra.run.smithi123.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.39773.log.gz 2024-09-16T15:52:51.420 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.39031.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37210.log 2024-09-16T15:52:51.420 INFO:teuthology.orchestra.run.smithi123.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.39031.log.gz 2024-09-16T15:52:51.420 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.42705.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47754.log 2024-09-16T15:52:51.420 INFO:teuthology.orchestra.run.smithi123.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.42705.log.gz 2024-09-16T15:52:51.420 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.37210.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51814.log 2024-09-16T15:52:51.420 INFO:teuthology.orchestra.run.smithi123.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.37210.log.gz 2024-09-16T15:52:51.420 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.47754.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46947.log 2024-09-16T15:52:51.420 INFO:teuthology.orchestra.run.smithi123.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.47754.log.gz 2024-09-16T15:52:51.420 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.51814.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38688.log 2024-09-16T15:52:51.420 INFO:teuthology.orchestra.run.smithi123.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.51814.log.gz 2024-09-16T15:52:51.420 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.46947.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54891.log 2024-09-16T15:52:51.420 INFO:teuthology.orchestra.run.smithi123.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.46947.log.gz 2024-09-16T15:52:51.421 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.38688.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45507.log 2024-09-16T15:52:51.421 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.54891.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.38688.log.gz 2024-09-16T15:52:51.421 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55614.log 2024-09-16T15:52:51.421 INFO:teuthology.orchestra.run.smithi123.stderr: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.54891.log.gz 2024-09-16T15:52:51.421 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.45507.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49104.log 2024-09-16T15:52:51.421 INFO:teuthology.orchestra.run.smithi123.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.45507.log.gz 2024-09-16T15:52:51.421 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.55614.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51022.log 2024-09-16T15:52:51.421 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.49104.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.55614.log.gz 2024-09-16T15:52:51.421 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41211.log 2024-09-16T15:52:51.421 INFO:teuthology.orchestra.run.smithi123.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.49104.log.gz 2024-09-16T15:52:51.421 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.51022.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.y.log 2024-09-16T15:52:51.421 INFO:teuthology.orchestra.run.smithi123.stderr: 87.9%/var/log/ceph/ceph-client.admin.41211.log: -- replaced with /var/log/ceph/ceph-client.admin.51022.log.gz 2024-09-16T15:52:51.422 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47198.log 2024-09-16T15:52:51.422 INFO:teuthology.orchestra.run.smithi123.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.41211.log.gz 2024-09-16T15:52:51.422 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-mgr.y.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52978.log 2024-09-16T15:52:51.422 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.47198.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.36236.log 2024-09-16T15:52:51.422 INFO:teuthology.orchestra.run.smithi123.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.47198.log.gz 2024-09-16T15:52:51.422 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.52978.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37137.log 2024-09-16T15:52:51.422 INFO:teuthology.orchestra.run.smithi123.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.52978.log.gz 2024-09-16T15:52:51.422 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.0.36236.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52817.log 2024-09-16T15:52:51.422 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.37137.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35039.log 2024-09-16T15:52:51.422 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.52817.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.37137.log.gz 2024-09-16T15:52:51.422 INFO:teuthology.orchestra.run.smithi123.stderr: 90.0% -- replaced with /var/log/ceph/ceph-client.0.36236.log.gz 2024-09-16T15:52:51.423 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35247.log 2024-09-16T15:52:51.423 INFO:teuthology.orchestra.run.smithi123.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.52817.log.gz 2024-09-16T15:52:51.423 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.35039.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49828.log 2024-09-16T15:52:51.423 INFO:teuthology.orchestra.run.smithi123.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.35039.log.gz 2024-09-16T15:52:51.423 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.35247.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47513.log 2024-09-16T15:52:51.423 INFO:teuthology.orchestra.run.smithi123.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.35247.log.gz 2024-09-16T15:52:51.423 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.49828.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55128.log 2024-09-16T15:52:51.423 INFO:teuthology.orchestra.run.smithi123.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.49828.log.gz 2024-09-16T15:52:51.423 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.47513.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41823.log 2024-09-16T15:52:51.423 INFO:teuthology.orchestra.run.smithi123.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.47513.log.gz 2024-09-16T15:52:51.423 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.55128.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35637.log 2024-09-16T15:52:51.423 INFO:teuthology.orchestra.run.smithi123.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.55128.log.gz 2024-09-16T15:52:51.424 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.41823.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43508.log 2024-09-16T15:52:51.424 INFO:teuthology.orchestra.run.smithi123.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.41823.log.gz 2024-09-16T15:52:51.424 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.35637.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49341.log 2024-09-16T15:52:51.424 INFO:teuthology.orchestra.run.smithi123.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.35637.log.gz 2024-09-16T15:52:51.424 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.43508.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49130.log 2024-09-16T15:52:51.424 INFO:teuthology.orchestra.run.smithi123.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.43508.log.gz 2024-09-16T15:52:51.424 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.49341.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41322.log 2024-09-16T15:52:51.424 INFO:teuthology.orchestra.run.smithi123.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.49341.log.gz 2024-09-16T15:52:51.424 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.49130.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41161.log 2024-09-16T15:52:51.424 INFO:teuthology.orchestra.run.smithi123.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.49130.log.gz 2024-09-16T15:52:51.424 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.41322.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44070.log 2024-09-16T15:52:51.424 INFO:teuthology.orchestra.run.smithi123.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.41322.log.gz 2024-09-16T15:52:51.424 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.41161.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44124.log 2024-09-16T15:52:51.424 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.44070.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48948.log 2024-09-16T15:52:51.424 INFO:teuthology.orchestra.run.smithi123.stderr: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.41161.log.gz 2024-09-16T15:52:51.425 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.44124.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.44070.log.gz 2024-09-16T15:52:51.425 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40119.log 2024-09-16T15:52:51.425 INFO:teuthology.orchestra.run.smithi123.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.44124.log.gz 2024-09-16T15:52:51.425 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.48948.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49512.log 2024-09-16T15:52:51.425 INFO:teuthology.orchestra.run.smithi123.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.48948.log.gz 2024-09-16T15:52:51.425 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.40119.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44384.log 2024-09-16T15:52:51.425 INFO:teuthology.orchestra.run.smithi123.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.40119.log.gz 2024-09-16T15:52:51.425 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.49512.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44879.log 2024-09-16T15:52:51.425 INFO:teuthology.orchestra.run.smithi123.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.49512.log.gz 2024-09-16T15:52:51.425 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.44384.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54111.log 2024-09-16T15:52:51.425 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.44879.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54564.log 2024-09-16T15:52:51.425 INFO:teuthology.orchestra.run.smithi123.stderr: 91.2% -- replaced with /var/log/ceph/ceph-client.admin.44384.log.gz 2024-09-16T15:52:51.425 INFO:teuthology.orchestra.run.smithi123.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.44879.log.gz 2024-09-16T15:52:51.425 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.54111.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53372.log 2024-09-16T15:52:51.425 INFO:teuthology.orchestra.run.smithi123.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.54111.log.gz 2024-09-16T15:52:51.426 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.54564.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50866.log 2024-09-16T15:52:51.426 INFO:teuthology.orchestra.run.smithi123.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.54564.log.gz 2024-09-16T15:52:51.426 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.53372.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45622.log 2024-09-16T15:52:51.426 INFO:teuthology.orchestra.run.smithi123.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.53372.log.gz 2024-09-16T15:52:51.426 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.50866.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41666.log 2024-09-16T15:52:51.426 INFO:teuthology.orchestra.run.smithi123.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.50866.log.gz 2024-09-16T15:52:51.426 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.45622.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50396.log 2024-09-16T15:52:51.426 INFO:teuthology.orchestra.run.smithi123.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.45622.log.gz 2024-09-16T15:52:51.426 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.41666.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34987.log 2024-09-16T15:52:51.426 INFO:teuthology.orchestra.run.smithi123.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.41666.log.gz 2024-09-16T15:52:51.426 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.50396.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2024-09-16T15:52:51.426 INFO:teuthology.orchestra.run.smithi123.stderr: 85.1% -- replaced with /var/log/ceph/ceph-client.admin.50396.log.gz 2024-09-16T15:52:51.426 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.34987.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55926.log 2024-09-16T15:52:51.426 INFO:teuthology.orchestra.run.smithi123.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.34987.log.gz 2024-09-16T15:52:51.427 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52269.log 2024-09-16T15:52:51.427 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.55926.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36108.log 2024-09-16T15:52:51.427 INFO:teuthology.orchestra.run.smithi123.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.55926.log.gz 2024-09-16T15:52:51.427 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.52269.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45478.log 2024-09-16T15:52:51.427 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48469.log 2024-09-16T15:52:51.427 INFO:teuthology.orchestra.run.smithi123.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.52269.log.gz 2024-09-16T15:52:51.427 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.36108.log: /var/log/ceph/ceph-client.admin.45478.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.36108.log.gz 2024-09-16T15:52:51.427 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49590.log 2024-09-16T15:52:51.427 INFO:teuthology.orchestra.run.smithi123.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.45478.log.gz 2024-09-16T15:52:51.427 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49195.log 2024-09-16T15:52:51.427 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.48469.log: /var/log/ceph/ceph-client.admin.49590.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45061.log 2024-09-16T15:52:51.427 INFO:teuthology.orchestra.run.smithi123.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.48469.log.gz 2024-09-16T15:52:51.427 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.49195.log: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.49590.log.gz 2024-09-16T15:52:51.427 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35244.log 2024-09-16T15:52:51.428 INFO:teuthology.orchestra.run.smithi123.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.49195.log.gz 2024-09-16T15:52:51.428 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.45061.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51925.log 2024-09-16T15:52:51.428 INFO:teuthology.orchestra.run.smithi123.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.45061.log.gz 2024-09-16T15:52:51.428 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.35244.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42311.log 2024-09-16T15:52:51.428 INFO:teuthology.orchestra.run.smithi123.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.35244.log.gz 2024-09-16T15:52:51.428 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.51925.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40093.log 2024-09-16T15:52:51.428 INFO:teuthology.orchestra.run.smithi123.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.51925.log.gz 2024-09-16T15:52:51.428 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.42311.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46764.log 2024-09-16T15:52:51.428 INFO:teuthology.orchestra.run.smithi123.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.42311.log.gz 2024-09-16T15:52:51.428 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.40093.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52451.log 2024-09-16T15:52:51.428 INFO:teuthology.orchestra.run.smithi123.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.40093.log.gz 2024-09-16T15:52:51.428 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.46764.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46709.log 2024-09-16T15:52:51.428 INFO:teuthology.orchestra.run.smithi123.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.46764.log.gz 2024-09-16T15:52:51.429 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.52451.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35975.log 2024-09-16T15:52:51.429 INFO:teuthology.orchestra.run.smithi123.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.52451.log.gz 2024-09-16T15:52:51.429 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.46709.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29010.log 2024-09-16T15:52:51.429 INFO:teuthology.orchestra.run.smithi123.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.46709.log.gz 2024-09-16T15:52:51.429 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.35975.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47094.log 2024-09-16T15:52:51.429 INFO:teuthology.orchestra.run.smithi123.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.35975.log.gz 2024-09-16T15:52:51.429 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.29010.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51600.log 2024-09-16T15:52:51.429 INFO:teuthology.orchestra.run.smithi123.stderr: 78.9% -- replaced with /var/log/ceph/ceph-client.admin.29010.log.gz 2024-09-16T15:52:51.429 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.47094.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38277.log 2024-09-16T15:52:51.429 INFO:teuthology.orchestra.run.smithi123.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.47094.log.gz 2024-09-16T15:52:51.429 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.51600.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41005.log 2024-09-16T15:52:51.429 INFO:teuthology.orchestra.run.smithi123.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.51600.log.gz 2024-09-16T15:52:51.429 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.38277.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55383.log 2024-09-16T15:52:51.429 INFO:teuthology.orchestra.run.smithi123.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.38277.log.gz 2024-09-16T15:52:51.429 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.41005.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55770.log 2024-09-16T15:52:51.430 INFO:teuthology.orchestra.run.smithi123.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.41005.log.gz 2024-09-16T15:52:51.430 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.55383.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50209.log 2024-09-16T15:52:51.430 INFO:teuthology.orchestra.run.smithi123.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.55383.log.gz 2024-09-16T15:52:51.430 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.55770.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40283.log 2024-09-16T15:52:51.430 INFO:teuthology.orchestra.run.smithi123.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.55770.log.gz 2024-09-16T15:52:51.430 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.50209.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50346.log 2024-09-16T15:52:51.430 INFO:teuthology.orchestra.run.smithi123.stderr: 85.2% -- replaced with /var/log/ceph/ceph-client.admin.50209.log.gz 2024-09-16T15:52:51.430 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41797.log 2024-09-16T15:52:51.430 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.40283.log: /var/log/ceph/ceph-client.admin.50346.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36286.log 2024-09-16T15:52:51.430 INFO:teuthology.orchestra.run.smithi123.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.40283.log.gz 2024-09-16T15:52:51.430 INFO:teuthology.orchestra.run.smithi123.stderr: 84.7% -- replaced with /var/log/ceph/ceph-client.admin.50346.log.gz 2024-09-16T15:52:51.430 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.41797.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39109.log 2024-09-16T15:52:51.430 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45652.log 2024-09-16T15:52:51.430 INFO:teuthology.orchestra.run.smithi123.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.41797.log.gz 2024-09-16T15:52:51.431 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.36286.log: /var/log/ceph/ceph-client.admin.39109.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.36286.log.gz 2024-09-16T15:52:51.431 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34676.log 2024-09-16T15:52:51.431 INFO:teuthology.orchestra.run.smithi123.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.39109.log.gz 2024-09-16T15:52:51.431 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43265.log 2024-09-16T15:52:51.431 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.45652.log: /var/log/ceph/ceph-client.admin.34676.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55258.log 2024-09-16T15:52:51.431 INFO:teuthology.orchestra.run.smithi123.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.45652.log.gz 2024-09-16T15:52:51.431 INFO:teuthology.orchestra.run.smithi123.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.34676.log.gz 2024-09-16T15:52:51.431 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.43265.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36623.log 2024-09-16T15:52:51.431 INFO:teuthology.orchestra.run.smithi123.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.43265.log.gz 2024-09-16T15:52:51.431 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.55258.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51392.log 2024-09-16T15:52:51.431 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.36623.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46016.log 2024-09-16T15:52:51.431 INFO:teuthology.orchestra.run.smithi123.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.55258.log.gz 2024-09-16T15:52:51.431 INFO:teuthology.orchestra.run.smithi123.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.36623.log.gz 2024-09-16T15:52:51.431 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.51392.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43134.log 2024-09-16T15:52:51.431 INFO:teuthology.orchestra.run.smithi123.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.51392.log.gz 2024-09-16T15:52:51.431 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42575.log 2024-09-16T15:52:51.432 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.46016.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53633.log 2024-09-16T15:52:51.432 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.43134.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.46016.log.gz 2024-09-16T15:52:51.432 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.42575.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.41267.log 2024-09-16T15:52:51.432 INFO:teuthology.orchestra.run.smithi123.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.43134.log.gz 2024-09-16T15:52:51.432 INFO:teuthology.orchestra.run.smithi123.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.42575.log.gz 2024-09-16T15:52:51.432 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.53633.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43623.log 2024-09-16T15:52:51.432 INFO:teuthology.orchestra.run.smithi123.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.53633.log.gz 2024-09-16T15:52:51.432 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.0.41267.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53868.log 2024-09-16T15:52:51.432 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.43623.log: gzip -5 87.7% --verbose -- -- replaced with /var/log/ceph/ceph-client.admin.43623.log.gz 2024-09-16T15:52:51.432 INFO:teuthology.orchestra.run.smithi123.stderr: /var/log/ceph/ceph-client.admin.46068.log 2024-09-16T15:52:51.432 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54929.log 2024-09-16T15:52:51.432 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.53868.log: /var/log/ceph/ceph-client.admin.46068.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55232.log 2024-09-16T15:52:51.432 INFO:teuthology.orchestra.run.smithi123.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.53868.log.gz 2024-09-16T15:52:51.432 INFO:teuthology.orchestra.run.smithi123.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.46068.log.gz 2024-09-16T15:52:51.433 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.54929.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36821.log 2024-09-16T15:52:51.433 INFO:teuthology.orchestra.run.smithi123.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.54929.log.gz 2024-09-16T15:52:51.433 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.55232.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42259.log 2024-09-16T15:52:51.433 INFO:teuthology.orchestra.run.smithi123.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.55232.log.gz 2024-09-16T15:52:51.433 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.36821.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50840.log 2024-09-16T15:52:51.433 INFO:teuthology.orchestra.run.smithi123.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.36821.log.gz 2024-09-16T15:52:51.433 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.42259.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45596.log 2024-09-16T15:52:51.433 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.50840.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.42259.log.gz 2024-09-16T15:52:51.433 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45783.log 2024-09-16T15:52:51.433 INFO:teuthology.orchestra.run.smithi123.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.50840.log.gz 2024-09-16T15:52:51.433 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.45596.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52846.log 2024-09-16T15:52:51.433 INFO:teuthology.orchestra.run.smithi123.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.45596.log.gz 2024-09-16T15:52:51.433 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.45783.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35949.log 2024-09-16T15:52:51.433 INFO:teuthology.orchestra.run.smithi123.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.45783.log.gz 2024-09-16T15:52:51.433 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36571.log 2024-09-16T15:52:51.434 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.52846.log: /var/log/ceph/ceph-client.admin.35949.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.52846.log.gz 2024-09-16T15:52:51.434 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37111.log 2024-09-16T15:52:51.434 INFO:teuthology.orchestra.run.smithi123.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.35949.log.gz 2024-09-16T15:52:51.434 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37446.log 2024-09-16T15:52:51.434 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.36571.log: /var/log/ceph/ceph-client.admin.37111.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45835.log 2024-09-16T15:52:51.434 INFO:teuthology.orchestra.run.smithi123.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.36571.log.gz 2024-09-16T15:52:51.434 INFO:teuthology.orchestra.run.smithi123.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.37111.log.gz 2024-09-16T15:52:51.434 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.37446.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36740.log 2024-09-16T15:52:51.434 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45885.log 2024-09-16T15:52:51.434 INFO:teuthology.orchestra.run.smithi123.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.37446.log.gz 2024-09-16T15:52:51.434 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.45835.log: /var/log/ceph/ceph-client.admin.36740.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38193.log 2024-09-16T15:52:51.434 INFO:teuthology.orchestra.run.smithi123.stderr: 90.9% -- replaced with /var/log/ceph/ceph-client.admin.45835.log.gz 2024-09-16T15:52:51.434 INFO:teuthology.orchestra.run.smithi123.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.36740.log.gz 2024-09-16T15:52:51.434 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.45885.log: 92.6% -- replaced with /var/log/ceph/ceph-client.0.41267.log.gz 2024-09-16T15:52:51.434 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42601.log 2024-09-16T15:52:51.435 INFO:teuthology.orchestra.run.smithi123.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.45885.log.gz 2024-09-16T15:52:51.435 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42945.log 2024-09-16T15:52:51.435 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.38193.log: /var/log/ceph/ceph-client.admin.42601.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41348.log 2024-09-16T15:52:51.435 INFO:teuthology.orchestra.run.smithi123.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.38193.log.gz 2024-09-16T15:52:51.435 INFO:teuthology.orchestra.run.smithi123.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.42601.log.gz 2024-09-16T15:52:51.435 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.42945.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45009.log 2024-09-16T15:52:51.435 INFO:teuthology.orchestra.run.smithi123.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.42945.log.gz 2024-09-16T15:52:51.435 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28938.log 2024-09-16T15:52:51.435 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.41348.log: /var/log/ceph/ceph-client.admin.45009.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.41348.log.gz 2024-09-16T15:52:51.435 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54078.log 2024-09-16T15:52:51.435 INFO:teuthology.orchestra.run.smithi123.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.45009.log.gz/var/log/ceph/ceph-client.admin.28938.log: 2024-09-16T15:52:51.435 INFO:teuthology.orchestra.run.smithi123.stderr: 77.9% -- replaced with /var/log/ceph/ceph-client.admin.28938.log.gz 2024-09-16T15:52:51.435 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42337.log 2024-09-16T15:52:51.435 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35248.log 2024-09-16T15:52:51.435 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.54078.log: /var/log/ceph/ceph-client.admin.42337.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55154.log 2024-09-16T15:52:51.435 INFO:teuthology.orchestra.run.smithi123.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.54078.log.gz 2024-09-16T15:52:51.436 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.35248.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.42337.log.gz 2024-09-16T15:52:51.436 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38303.log 2024-09-16T15:52:51.436 INFO:teuthology.orchestra.run.smithi123.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.35248.log.gz 2024-09-16T15:52:51.436 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.55154.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51233.log 2024-09-16T15:52:51.436 INFO:teuthology.orchestra.run.smithi123.stderr: 88.0%/var/log/ceph/ceph-client.admin.38303.log: -- replaced with /var/log/ceph/ceph-client.admin.55154.log.gz 2024-09-16T15:52:51.436 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39721.log 2024-09-16T15:52:51.436 INFO:teuthology.orchestra.run.smithi123.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.38303.log.gz 2024-09-16T15:52:51.436 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42365.log 2024-09-16T15:52:51.436 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.51233.log: /var/log/ceph/ceph-client.admin.39721.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52321.log 2024-09-16T15:52:51.436 INFO:teuthology.orchestra.run.smithi123.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.51233.log.gz 2024-09-16T15:52:51.436 INFO:teuthology.orchestra.run.smithi123.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.39721.log.gz 2024-09-16T15:52:51.436 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.42365.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49718.log 2024-09-16T15:52:51.436 INFO:teuthology.orchestra.run.smithi123.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.42365.log.gz 2024-09-16T15:52:51.436 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38076.log 2024-09-16T15:52:51.436 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.52321.log: /var/log/ceph/ceph-client.admin.49718.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.52321.log.gz 2024-09-16T15:52:51.437 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44176.log 2024-09-16T15:52:51.437 INFO:teuthology.orchestra.run.smithi123.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.49718.log.gz 2024-09-16T15:52:51.437 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.38076.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53816.log 2024-09-16T15:52:51.437 INFO:teuthology.orchestra.run.smithi123.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.38076.log.gz 2024-09-16T15:52:51.437 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.44176.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40806.log 2024-09-16T15:52:51.437 INFO:teuthology.orchestra.run.smithi123.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.44176.log.gz 2024-09-16T15:52:51.437 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40387.log 2024-09-16T15:52:51.437 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.53816.log: /var/log/ceph/ceph-client.admin.40806.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49078.log 2024-09-16T15:52:51.437 INFO:teuthology.orchestra.run.smithi123.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.53816.log.gz 2024-09-16T15:52:51.437 INFO:teuthology.orchestra.run.smithi123.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.40806.log.gz 2024-09-16T15:52:51.437 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.40387.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39504.log 2024-09-16T15:52:51.437 INFO:teuthology.orchestra.run.smithi123.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.40387.log.gz 2024-09-16T15:52:51.437 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54052.log 2024-09-16T15:52:51.437 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.49078.log: /var/log/ceph/ceph-client.admin.39504.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.49078.log.gz 2024-09-16T15:52:51.438 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41920.log 2024-09-16T15:52:51.438 INFO:teuthology.orchestra.run.smithi123.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.39504.log.gz 2024-09-16T15:52:51.438 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.54052.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42419.log 2024-09-16T15:52:51.438 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.41920.log: gzip -5 88.2% --verbose -- -- replaced with /var/log/ceph/ceph-client.admin.54052.log.gz 2024-09-16T15:52:51.438 INFO:teuthology.orchestra.run.smithi123.stderr: /var/log/ceph/ceph-client.admin.50996.log 2024-09-16T15:52:51.438 INFO:teuthology.orchestra.run.smithi123.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.41920.log.gz 2024-09-16T15:52:51.438 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.42419.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2024-09-16T15:52:51.438 INFO:teuthology.orchestra.run.smithi123.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.42419.log.gz 2024-09-16T15:52:51.438 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50157.log 2024-09-16T15:52:51.438 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.50996.log: /var/log/ceph/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49538.log 2024-09-16T15:52:51.438 INFO:teuthology.orchestra.run.smithi123.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.50996.log.gz 2024-09-16T15:52:51.438 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.50157.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47806.log 2024-09-16T15:52:51.438 INFO:teuthology.orchestra.run.smithi123.stderr: 84.8% -- replaced with /var/log/ceph/ceph-client.admin.50157.log.gz 2024-09-16T15:52:51.438 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54317.log 2024-09-16T15:52:51.439 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.49538.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42971.log 2024-09-16T15:52:51.439 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.47806.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.49538.log.gz 2024-09-16T15:52:51.439 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.54317.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.47806.log.gz 2024-09-16T15:52:51.439 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46504.log 2024-09-16T15:52:51.439 INFO:teuthology.orchestra.run.smithi123.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.54317.log.gz 2024-09-16T15:52:51.439 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.42971.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53842.log 2024-09-16T15:52:51.439 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.46504.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.42971.log.gz 2024-09-16T15:52:51.439 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55011.log 2024-09-16T15:52:51.439 INFO:teuthology.orchestra.run.smithi123.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.46504.log.gz 2024-09-16T15:52:51.439 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.53842.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46174.log 2024-09-16T15:52:51.439 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.55011.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.53842.log.gz 2024-09-16T15:52:51.439 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36851.log 2024-09-16T15:52:51.439 INFO:teuthology.orchestra.run.smithi123.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.55011.log.gz 2024-09-16T15:52:51.439 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.46174.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37236.log 2024-09-16T15:52:51.439 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.36851.log: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.46174.log.gz 2024-09-16T15:52:51.440 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38478.log 2024-09-16T15:52:51.440 INFO:teuthology.orchestra.run.smithi123.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.36851.log.gz 2024-09-16T15:52:51.440 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.37236.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39265.log 2024-09-16T15:52:51.440 INFO:teuthology.orchestra.run.smithi123.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.37236.log.gz 2024-09-16T15:52:51.440 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.38478.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42627.log 2024-09-16T15:52:51.440 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.39265.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.38478.log.gz 2024-09-16T15:52:51.440 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40039.log 2024-09-16T15:52:51.440 INFO:teuthology.orchestra.run.smithi123.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.39265.log.gz 2024-09-16T15:52:51.440 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48747.log 2024-09-16T15:52:51.440 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.42627.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39799.log 2024-09-16T15:52:51.440 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.40039.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.42627.log.gz 2024-09-16T15:52:51.440 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.48747.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.40039.log.gz 2024-09-16T15:52:51.440 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51840.log 2024-09-16T15:52:51.440 INFO:teuthology.orchestra.run.smithi123.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.48747.log.gz 2024-09-16T15:52:51.440 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose --/var/log/ceph/ceph-client.admin.39799.log: /var/log/ceph/ceph-client.admin.48411.log 2024-09-16T15:52:51.441 INFO:teuthology.orchestra.run.smithi123.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.39799.log.gz 2024-09-16T15:52:51.441 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.51840.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40309.log 2024-09-16T15:52:51.441 INFO:teuthology.orchestra.run.smithi123.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.51840.log.gz 2024-09-16T15:52:51.441 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41031.log 2024-09-16T15:52:51.441 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.48411.log: /var/log/ceph/ceph-client.admin.40309.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54776.log 2024-09-16T15:52:51.441 INFO:teuthology.orchestra.run.smithi123.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.48411.log.gz 2024-09-16T15:52:51.441 INFO:teuthology.orchestra.run.smithi123.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.40309.log.gz 2024-09-16T15:52:51.441 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.41031.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44735.log 2024-09-16T15:52:51.441 INFO:teuthology.orchestra.run.smithi123.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.41031.log.gz 2024-09-16T15:52:51.441 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50892.log 2024-09-16T15:52:51.441 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.54776.log: /var/log/ceph/ceph-client.admin.44735.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.54776.log.gz 2024-09-16T15:52:51.441 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54590.log 2024-09-16T15:52:51.441 INFO:teuthology.orchestra.run.smithi123.stderr: 85.8% -- replaced with /var/log/ceph/ceph-client.admin.44735.log.gz 2024-09-16T15:52:51.441 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.50892.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47637.log 2024-09-16T15:52:51.441 INFO:teuthology.orchestra.run.smithi123.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.50892.log.gz 2024-09-16T15:52:51.441 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.54590.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43239.log 2024-09-16T15:52:51.442 INFO:teuthology.orchestra.run.smithi123.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.54590.log.gz 2024-09-16T15:52:51.442 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.47637.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38870.log 2024-09-16T15:52:51.442 INFO:teuthology.orchestra.run.smithi123.stderr: 87.9%gzip -- replaced with /var/log/ceph/ceph-client.admin.47637.log.gz 2024-09-16T15:52:51.442 INFO:teuthology.orchestra.run.smithi123.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.40754.log 2024-09-16T15:52:51.442 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.43239.log: /var/log/ceph/ceph-client.admin.38870.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.43239.log.gz 2024-09-16T15:52:51.442 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46108.log 2024-09-16T15:52:51.442 INFO:teuthology.orchestra.run.smithi123.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.38870.log.gz 2024-09-16T15:52:51.442 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.40754.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52096.log 2024-09-16T15:52:51.442 INFO:teuthology.orchestra.run.smithi123.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.40754.log.gz 2024-09-16T15:52:51.442 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.46108.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40257.log 2024-09-16T15:52:51.442 INFO:teuthology.orchestra.run.smithi123.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.46108.log.gz 2024-09-16T15:52:51.442 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38818.log 2024-09-16T15:52:51.442 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.52096.log: /var/log/ceph/ceph-client.admin.40257.log: 87.3%gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2024-09-16T15:52:51.442 INFO:teuthology.orchestra.run.smithi123.stderr: -- replaced with /var/log/ceph/ceph-client.admin.52096.log.gz 2024-09-16T15:52:51.442 INFO:teuthology.orchestra.run.smithi123.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.40257.log.gz 2024-09-16T15:52:51.443 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.38818.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55357.log 2024-09-16T15:52:51.443 INFO:teuthology.orchestra.run.smithi123.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.38818.log.gz 2024-09-16T15:52:51.443 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39213.log 2024-09-16T15:52:51.443 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.55357.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50184.log 2024-09-16T15:52:51.443 INFO:teuthology.orchestra.run.smithi123.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.55357.log.gz 2024-09-16T15:52:51.443 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.39213.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43108.log 2024-09-16T15:52:51.443 INFO:teuthology.orchestra.run.smithi123.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.39213.log.gz 2024-09-16T15:52:51.443 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39934.log 2024-09-16T15:52:51.443 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.50184.log: /var/log/ceph/ceph-client.admin.43108.log: 85.0%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48545.log 2024-09-16T15:52:51.443 INFO:teuthology.orchestra.run.smithi123.stderr: -- replaced with /var/log/ceph/ceph-client.admin.50184.log.gz 2024-09-16T15:52:51.443 INFO:teuthology.orchestra.run.smithi123.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.43108.log.gz 2024-09-16T15:52:51.443 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.39934.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51522.log 2024-09-16T15:52:51.443 INFO:teuthology.orchestra.run.smithi123.stderr: 87.5%/var/log/ceph/ceph-client.admin.48545.log: -- replaced with /var/log/ceph/ceph-client.admin.39934.log.gz 2024-09-16T15:52:51.443 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48877.log 2024-09-16T15:52:51.443 INFO:teuthology.orchestra.run.smithi123.stderr: 84.8% -- replaced with /var/log/ceph/ceph-client.admin.48545.log.gz 2024-09-16T15:52:51.444 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.51522.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34961.log 2024-09-16T15:52:51.444 INFO:teuthology.orchestra.run.smithi123.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.51522.log.gz 2024-09-16T15:52:51.444 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.48877.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42679.log 2024-09-16T15:52:51.444 INFO:teuthology.orchestra.run.smithi123.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.48877.log.gz 2024-09-16T15:52:51.444 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.34961.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41109.log 2024-09-16T15:52:51.444 INFO:teuthology.orchestra.run.smithi123.stderr: 87.6%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53764.log 2024-09-16T15:52:51.444 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.42679.log: -- replaced with /var/log/ceph/ceph-client.admin.34961.log.gz 2024-09-16T15:52:51.444 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.41109.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47915.log 2024-09-16T15:52:51.444 INFO:teuthology.orchestra.run.smithi123.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.42679.log.gz 2024-09-16T15:52:51.444 INFO:teuthology.orchestra.run.smithi123.stderr: 87.8%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54260.log 2024-09-16T15:52:51.444 INFO:teuthology.orchestra.run.smithi123.stderr: -- replaced with /var/log/ceph/ceph-client.admin.41109.log.gz/var/log/ceph/ceph-client.admin.53764.log: 2024-09-16T15:52:51.444 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.47915.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.53764.log.gzgzip 2024-09-16T15:52:51.444 INFO:teuthology.orchestra.run.smithi123.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.48974.log 2024-09-16T15:52:51.444 INFO:teuthology.orchestra.run.smithi123.stderr: 87.4%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35644.log 2024-09-16T15:52:51.444 INFO:teuthology.orchestra.run.smithi123.stderr: -- replaced with /var/log/ceph/ceph-client.admin.47915.log.gz 2024-09-16T15:52:51.445 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.54260.log: /var/log/ceph/ceph-client.admin.48974.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50471.log 2024-09-16T15:52:51.445 INFO:teuthology.orchestra.run.smithi123.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.54260.log.gz 2024-09-16T15:52:51.445 INFO:teuthology.orchestra.run.smithi123.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.48974.log.gz 2024-09-16T15:52:51.445 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47349.log 2024-09-16T15:52:51.445 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.35644.log: /var/log/ceph/ceph-client.admin.50471.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37563.log 2024-09-16T15:52:51.445 INFO:teuthology.orchestra.run.smithi123.stderr: 84.3% 87.7% -- replaced with /var/log/ceph/ceph-client.admin.35644.log.gz 2024-09-16T15:52:51.445 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.47349.log: -- replaced with /var/log/ceph/ceph-client.admin.50471.log.gz 2024-09-16T15:52:51.445 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29022.log 2024-09-16T15:52:51.445 INFO:teuthology.orchestra.run.smithi123.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.47349.log.gz 2024-09-16T15:52:51.445 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.37563.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.53266.log 2024-09-16T15:52:51.445 INFO:teuthology.orchestra.run.smithi123.stderr: 87.9%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43859.log 2024-09-16T15:52:51.445 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.29022.log: -- replaced with /var/log/ceph/ceph-client.admin.37563.log.gz 2024-09-16T15:52:51.445 INFO:teuthology.orchestra.run.smithi123.stderr: 77.8% -- replaced with /var/log/ceph/ceph-client.admin.29022.log.gz 2024-09-16T15:52:51.445 INFO:teuthology.orchestra.run.smithi123.stderr:gzip/var/log/ceph/ceph-client.0.53266.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.37537.log 2024-09-16T15:52:51.445 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.43859.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43545.log 2024-09-16T15:52:51.446 INFO:teuthology.orchestra.run.smithi123.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.43859.log.gz/var/log/ceph/ceph-client.admin.37537.log: 2024-09-16T15:52:51.446 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41573.log 2024-09-16T15:52:51.446 INFO:teuthology.orchestra.run.smithi123.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.37537.log.gz 2024-09-16T15:52:51.446 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52791.log 2024-09-16T15:52:51.446 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.43545.log: /var/log/ceph/ceph-client.admin.41573.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.43545.log.gz 2024-09-16T15:52:51.446 INFO:teuthology.orchestra.run.smithi123.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.41573.log.gz 2024-09-16T15:52:51.446 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50234.log 2024-09-16T15:52:51.446 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.52791.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46228.log 2024-09-16T15:52:51.446 INFO:teuthology.orchestra.run.smithi123.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.52791.log.gz/var/log/ceph/ceph-client.admin.50234.log: 2024-09-16T15:52:51.446 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40361.log 2024-09-16T15:52:51.446 INFO:teuthology.orchestra.run.smithi123.stderr: 84.8% -- replaced with /var/log/ceph/ceph-client.admin.50234.log.gz 2024-09-16T15:52:51.446 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.46228.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.46228.log.gz 2024-09-16T15:52:51.446 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48773.log 2024-09-16T15:52:51.446 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.40361.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28879.log 2024-09-16T15:52:51.446 INFO:teuthology.orchestra.run.smithi123.stderr: 87.3%/var/log/ceph/ceph-client.admin.48773.log: -- replaced with /var/log/ceph/ceph-client.admin.40361.log.gz 2024-09-16T15:52:51.447 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37288.log 2024-09-16T15:52:51.447 INFO:teuthology.orchestra.run.smithi123.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.48773.log.gz 2024-09-16T15:52:51.447 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39239.log 2024-09-16T15:52:51.447 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.28879.log: /var/log/ceph/ceph-client.admin.37288.log: 78.8% -- replaced with /var/log/ceph/ceph-client.admin.28879.log.gz 2024-09-16T15:52:51.447 INFO:teuthology.orchestra.run.smithi123.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.37288.log.gz 2024-09-16T15:52:51.447 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28949.log 2024-09-16T15:52:51.447 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.39239.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52425.log 2024-09-16T15:52:51.447 INFO:teuthology.orchestra.run.smithi123.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.39239.log.gz/var/log/ceph/ceph-client.admin.28949.log: 2024-09-16T15:52:51.447 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40571.log 2024-09-16T15:52:51.447 INFO:teuthology.orchestra.run.smithi123.stderr: 77.2% -- replaced with /var/log/ceph/ceph-client.admin.28949.log.gz 2024-09-16T15:52:51.447 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.52425.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.52425.log.gz 2024-09-16T15:52:51.447 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41547.log 2024-09-16T15:52:51.447 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.40571.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38953.log 2024-09-16T15:52:51.447 INFO:teuthology.orchestra.run.smithi123.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.40571.log.gz/var/log/ceph/ceph-client.admin.41547.log: 2024-09-16T15:52:51.447 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45087.log 2024-09-16T15:52:51.447 INFO:teuthology.orchestra.run.smithi123.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.41547.log.gz 2024-09-16T15:52:51.448 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35249.log 2024-09-16T15:52:51.448 INFO:teuthology.orchestra.run.smithi123.stderr: 92.6% -- replaced with /var/log/ceph/ceph-client.0.53266.log.gz 2024-09-16T15:52:51.448 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.45087.log: /var/log/ceph/ceph-client.admin.38953.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29477.log 2024-09-16T15:52:51.448 INFO:teuthology.orchestra.run.smithi123.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.45087.log.gz 2024-09-16T15:52:51.448 INFO:teuthology.orchestra.run.smithi123.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.38953.log.gz 2024-09-16T15:52:51.448 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.35249.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38013.log 2024-09-16T15:52:51.448 INFO:teuthology.orchestra.run.smithi123.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.35249.log.gz 2024-09-16T15:52:51.448 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.29477.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38329.log 2024-09-16T15:52:51.448 INFO:teuthology.orchestra.run.smithi123.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.29477.log.gz 2024-09-16T15:52:51.448 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.38013.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49486.log 2024-09-16T15:52:51.448 INFO:teuthology.orchestra.run.smithi123.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.38013.log.gz 2024-09-16T15:52:51.448 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.38329.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34855.log 2024-09-16T15:52:51.448 INFO:teuthology.orchestra.run.smithi123.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.38329.log.gz 2024-09-16T15:52:51.448 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.49486.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54666.log 2024-09-16T15:52:51.449 INFO:teuthology.orchestra.run.smithi123.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.49486.log.gz 2024-09-16T15:52:51.449 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.34855.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37420.log 2024-09-16T15:52:51.449 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.54666.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.34855.log.gz 2024-09-16T15:52:51.449 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47967.log 2024-09-16T15:52:51.449 INFO:teuthology.orchestra.run.smithi123.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.54666.log.gz 2024-09-16T15:52:51.449 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.37420.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28838.log 2024-09-16T15:52:51.449 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.47967.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.37420.log.gz 2024-09-16T15:52:51.449 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36766.log 2024-09-16T15:52:51.449 INFO:teuthology.orchestra.run.smithi123.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.47967.log.gz 2024-09-16T15:52:51.449 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44098.log 2024-09-16T15:52:51.449 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.28838.log: 76.7% -- replaced with /var/log/ceph/ceph-client.admin.28838.log.gz 2024-09-16T15:52:51.449 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.36766.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37394.log 2024-09-16T15:52:51.449 INFO:teuthology.orchestra.run.smithi123.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.36766.log.gz 2024-09-16T15:52:51.449 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.44098.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51704.log 2024-09-16T15:52:51.449 INFO:teuthology.orchestra.run.smithi123.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.44098.log.gz 2024-09-16T15:52:51.450 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.37394.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53030.log 2024-09-16T15:52:51.450 INFO:teuthology.orchestra.run.smithi123.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.37394.log.gz 2024-09-16T15:52:51.450 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.51704.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55484.log 2024-09-16T15:52:51.450 INFO:teuthology.orchestra.run.smithi123.stderr: 87.9%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37366.log 2024-09-16T15:52:51.450 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.53030.log: -- replaced with /var/log/ceph/ceph-client.admin.51704.log.gz 2024-09-16T15:52:51.450 INFO:teuthology.orchestra.run.smithi123.stderr: 87.3%/var/log/ceph/ceph-client.admin.55484.log: -- replaced with /var/log/ceph/ceph-client.admin.53030.log.gz 2024-09-16T15:52:51.450 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38407.log 2024-09-16T15:52:51.450 INFO:teuthology.orchestra.run.smithi123.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.55484.log.gz 2024-09-16T15:52:51.450 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.37366.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44596.log 2024-09-16T15:52:51.450 INFO:teuthology.orchestra.run.smithi123.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.37366.log.gz 2024-09-16T15:52:51.450 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.38407.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44672.log 2024-09-16T15:52:51.450 INFO:teuthology.orchestra.run.smithi123.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.38407.log.gz 2024-09-16T15:52:51.450 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose/var/log/ceph/ceph-client.admin.44596.log: -- /var/log/ceph/ceph-client.admin.35195.log 2024-09-16T15:52:51.450 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.44672.log: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.44596.log.gz 2024-09-16T15:52:51.450 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52159.log 2024-09-16T15:52:51.451 INFO:teuthology.orchestra.run.smithi123.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.44672.log.gz 2024-09-16T15:52:51.451 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.35195.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.49774.log 2024-09-16T15:52:51.451 INFO:teuthology.orchestra.run.smithi123.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.35195.log.gz 2024-09-16T15:52:51.451 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.52159.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48799.log 2024-09-16T15:52:51.451 INFO:teuthology.orchestra.run.smithi123.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.52159.log.gz 2024-09-16T15:52:51.451 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.0.49774.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37697.log 2024-09-16T15:52:51.451 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.48799.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37987.log 2024-09-16T15:52:51.451 INFO:teuthology.orchestra.run.smithi123.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.48799.log.gz/var/log/ceph/ceph-client.admin.37697.log: 2024-09-16T15:52:51.451 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51866.log 2024-09-16T15:52:51.451 INFO:teuthology.orchestra.run.smithi123.stderr: 90.4% -- replaced with /var/log/ceph/ceph-client.0.49774.log.gz 2024-09-16T15:52:51.451 INFO:teuthology.orchestra.run.smithi123.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.37697.log.gz 2024-09-16T15:52:51.451 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.37987.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.37987.log.gz 2024-09-16T15:52:51.451 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36955.log 2024-09-16T15:52:51.451 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.51866.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41972.log 2024-09-16T15:52:51.451 INFO:teuthology.orchestra.run.smithi123.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.51866.log.gz/var/log/ceph/ceph-client.admin.36955.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42076.log 2024-09-16T15:52:51.452 INFO:teuthology.orchestra.run.smithi123.stderr: 2024-09-16T15:52:51.452 INFO:teuthology.orchestra.run.smithi123.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.36955.log.gz 2024-09-16T15:52:51.452 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.41972.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43300.log 2024-09-16T15:52:51.452 INFO:teuthology.orchestra.run.smithi123.stderr: 87.7%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34775.log 2024-09-16T15:52:51.452 INFO:teuthology.orchestra.run.smithi123.stderr: -- replaced with /var/log/ceph/ceph-client.admin.41972.log.gz 2024-09-16T15:52:51.452 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.42076.log: /var/log/ceph/ceph-client.admin.43300.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43729.log 2024-09-16T15:52:51.452 INFO:teuthology.orchestra.run.smithi123.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.42076.log.gz 2024-09-16T15:52:51.452 INFO:teuthology.orchestra.run.smithi123.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.43300.log.gz 2024-09-16T15:52:51.452 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39851.log 2024-09-16T15:52:51.452 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.34775.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.34775.log.gz 2024-09-16T15:52:51.452 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.43729.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38433.log 2024-09-16T15:52:51.452 INFO:teuthology.orchestra.run.smithi123.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.43729.log.gz 2024-09-16T15:52:51.452 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.39851.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51314.log 2024-09-16T15:52:51.452 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.38433.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37589.log 2024-09-16T15:52:51.452 INFO:teuthology.orchestra.run.smithi123.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.39851.log.gz 2024-09-16T15:52:51.453 INFO:teuthology.orchestra.run.smithi123.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.38433.log.gz 2024-09-16T15:52:51.453 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.51314.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44016.log 2024-09-16T15:52:51.453 INFO:teuthology.orchestra.run.smithi123.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.51314.log.gz 2024-09-16T15:52:51.453 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.37589.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55744.log 2024-09-16T15:52:51.453 INFO:teuthology.orchestra.run.smithi123.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.37589.log.gz 2024-09-16T15:52:51.453 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.44016.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43703.log 2024-09-16T15:52:51.453 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.55744.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.44016.log.gz 2024-09-16T15:52:51.453 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41135.log 2024-09-16T15:52:51.453 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.43703.log: gzip -5 87.4% --verbose -- -- replaced with /var/log/ceph/ceph-client.admin.55744.log.gz /var/log/ceph/ceph-client.admin.42757.log 2024-09-16T15:52:51.453 INFO:teuthology.orchestra.run.smithi123.stderr: 2024-09-16T15:52:51.453 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.41135.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.43703.log.gz 2024-09-16T15:52:51.453 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36461.log 2024-09-16T15:52:51.453 INFO:teuthology.orchestra.run.smithi123.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.41135.log.gz 2024-09-16T15:52:51.453 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.42757.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55044.log 2024-09-16T15:52:51.454 INFO:teuthology.orchestra.run.smithi123.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.42757.log.gz 2024-09-16T15:52:51.454 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.36461.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42653.log 2024-09-16T15:52:51.454 INFO:teuthology.orchestra.run.smithi123.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.36461.log.gz 2024-09-16T15:52:51.454 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.55044.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38927.log 2024-09-16T15:52:51.454 INFO:teuthology.orchestra.run.smithi123.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.55044.log.gz 2024-09-16T15:52:51.454 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.42653.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44711.log 2024-09-16T15:52:51.454 INFO:teuthology.orchestra.run.smithi123.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.42653.log.gz 2024-09-16T15:52:51.454 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.38927.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50321.log 2024-09-16T15:52:51.454 INFO:teuthology.orchestra.run.smithi123.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.38927.log.gz 2024-09-16T15:52:51.454 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.44711.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35667.log 2024-09-16T15:52:51.454 INFO:teuthology.orchestra.run.smithi123.stderr: 85.1% -- replaced with /var/log/ceph/ceph-client.admin.44711.log.gz 2024-09-16T15:52:51.454 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.50321.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47146.log 2024-09-16T15:52:51.454 INFO:teuthology.orchestra.run.smithi123.stderr: 85.1% -- replaced with /var/log/ceph/ceph-client.admin.50321.log.gz 2024-09-16T15:52:51.454 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.35667.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47323.log 2024-09-16T15:52:51.455 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.47146.log: 87.4%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29387.log 2024-09-16T15:52:51.455 INFO:teuthology.orchestra.run.smithi123.stderr: -- replaced with /var/log/ceph/ceph-client.admin.35667.log.gz 2024-09-16T15:52:51.455 INFO:teuthology.orchestra.run.smithi123.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.47146.log.gz 2024-09-16T15:52:51.455 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.47323.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47401.log 2024-09-16T15:52:51.455 INFO:teuthology.orchestra.run.smithi123.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.47323.log.gz 2024-09-16T15:52:51.455 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37262.log 2024-09-16T15:52:51.455 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.29387.log: /var/log/ceph/ceph-client.admin.47401.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37085.log 2024-09-16T15:52:51.455 INFO:teuthology.orchestra.run.smithi123.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.29387.log.gz/var/log/ceph/ceph-client.admin.37262.log: 2024-09-16T15:52:51.455 INFO:teuthology.orchestra.run.smithi123.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.47401.log.gz 2024-09-16T15:52:51.455 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54802.log 2024-09-16T15:52:51.455 INFO:teuthology.orchestra.run.smithi123.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.37262.log.gz 2024-09-16T15:52:51.455 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.37085.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.37085.log.gz 2024-09-16T15:52:51.455 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48258.log 2024-09-16T15:52:51.464 INFO:teuthology.orchestra.run.smithi184.stderr:/var/log/ceph/ceph-osd.4.log: 92.7% -- replaced with /var/log/ceph/ceph-mds.b.log.gz 2024-09-16T15:52:51.478 INFO:teuthology.orchestra.run.smithi184.stderr: 92.5% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2024-09-16T15:52:51.509 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.54802.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45361.log 2024-09-16T15:52:51.509 INFO:teuthology.orchestra.run.smithi123.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.54802.log.gz 2024-09-16T15:52:51.509 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.48258.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51654.log 2024-09-16T15:52:51.509 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.45361.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.48258.log.gz 2024-09-16T15:52:51.509 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35431.log 2024-09-16T15:52:51.510 INFO:teuthology.orchestra.run.smithi123.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.45361.log.gz 2024-09-16T15:52:51.510 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.51654.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35660.log 2024-09-16T15:52:51.510 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.35431.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40145.log 2024-09-16T15:52:51.510 INFO:teuthology.orchestra.run.smithi123.stderr: 87.5% 91.0% -- replaced with /var/log/ceph/ceph-client.admin.51654.log.gz 2024-09-16T15:52:51.510 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.35660.log: -- replaced with /var/log/ceph/ceph-client.admin.35431.log.gz 2024-09-16T15:52:51.510 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37033.log 2024-09-16T15:52:51.510 INFO:teuthology.orchestra.run.smithi123.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.35660.log.gz 2024-09-16T15:52:51.510 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41875.log 2024-09-16T15:52:51.510 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.40145.log: /var/log/ceph/ceph-client.admin.37033.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44332.log 2024-09-16T15:52:51.510 INFO:teuthology.orchestra.run.smithi123.stderr: 87.9% 87.6% -- replaced with /var/log/ceph/ceph-client.admin.40145.log.gz 2024-09-16T15:52:51.511 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.41875.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47611.log 2024-09-16T15:52:51.511 INFO:teuthology.orchestra.run.smithi123.stderr: -- replaced with /var/log/ceph/ceph-client.admin.37033.log.gz 2024-09-16T15:52:51.511 INFO:teuthology.orchestra.run.smithi123.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.41875.log.gz 2024-09-16T15:52:51.511 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.44332.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47042.log 2024-09-16T15:52:51.511 INFO:teuthology.orchestra.run.smithi123.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.44332.log.gz 2024-09-16T15:52:51.511 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52635.log 2024-09-16T15:52:51.511 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.47611.log: /var/log/ceph/ceph-client.admin.47042.log: 87.4%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42393.log 2024-09-16T15:52:51.511 INFO:teuthology.orchestra.run.smithi123.stderr: -- replaced with /var/log/ceph/ceph-client.admin.47611.log.gz 2024-09-16T15:52:51.511 INFO:teuthology.orchestra.run.smithi123.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.47042.log.gz 2024-09-16T15:52:51.511 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.52635.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37801.log 2024-09-16T15:52:51.511 INFO:teuthology.orchestra.run.smithi123.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.52635.log.gz 2024-09-16T15:52:51.512 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55075.log 2024-09-16T15:52:51.512 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.42393.log: /var/log/ceph/ceph-client.admin.37801.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49932.log 2024-09-16T15:52:51.512 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.55075.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.42393.log.gz 2024-09-16T15:52:51.512 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53346.log 2024-09-16T15:52:51.512 INFO:teuthology.orchestra.run.smithi123.stderr: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.37801.log.gz 2024-09-16T15:52:51.512 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.49932.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.55075.log.gz 2024-09-16T15:52:51.512 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35169.log 2024-09-16T15:52:51.512 INFO:teuthology.orchestra.run.smithi123.stderr: 84.9%gzip -- replaced with /var/log/ceph/ceph-client.admin.49932.log.gz -5 --verbose -- /var/log/ceph/ceph-client.admin.43924.log 2024-09-16T15:52:51.512 INFO:teuthology.orchestra.run.smithi123.stderr: 2024-09-16T15:52:51.512 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.53346.log: /var/log/ceph/ceph-client.admin.35169.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.3.log 2024-09-16T15:52:51.513 INFO:teuthology.orchestra.run.smithi123.stderr: 87.6% 87.7% -- replaced with /var/log/ceph/ceph-client.admin.35169.log.gz 2024-09-16T15:52:51.513 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.43924.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46356.log 2024-09-16T15:52:51.513 INFO:teuthology.orchestra.run.smithi123.stderr: -- replaced with /var/log/ceph/ceph-client.admin.53346.log.gz 2024-09-16T15:52:51.513 INFO:teuthology.orchestra.run.smithi123.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.43924.log.gz 2024-09-16T15:52:51.513 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38979.log 2024-09-16T15:52:51.513 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.46356.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47663.log 2024-09-16T15:52:51.513 INFO:teuthology.orchestra.run.smithi123.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.46356.log.gz/var/log/ceph/ceph-client.admin.38979.log: 2024-09-16T15:52:51.513 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40467.log 2024-09-16T15:52:51.513 INFO:teuthology.orchestra.run.smithi123.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.38979.log.gz 2024-09-16T15:52:51.513 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38662.log 2024-09-16T15:52:51.513 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.47663.log: /var/log/ceph/ceph-client.admin.40467.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38556.log 2024-09-16T15:52:51.513 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.38662.log: 88.1% 87.2% -- replaced with /var/log/ceph/ceph-client.admin.47663.log.gz 2024-09-16T15:52:51.513 INFO:teuthology.orchestra.run.smithi123.stderr: -- replaced with /var/log/ceph/ceph-client.admin.40467.log.gz 2024-09-16T15:52:51.514 INFO:teuthology.orchestra.run.smithi123.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.38662.log.gz 2024-09-16T15:52:51.514 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29070.log 2024-09-16T15:52:51.514 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35645.log 2024-09-16T15:52:51.514 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.38556.log: /var/log/ceph/ceph-client.admin.29070.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54828.log 2024-09-16T15:52:51.514 INFO:teuthology.orchestra.run.smithi123.stderr: 87.3% 77.9% -- replaced with /var/log/ceph/ceph-client.admin.29070.log.gz 2024-09-16T15:52:51.514 INFO:teuthology.orchestra.run.smithi123.stderr: -- replaced with /var/log/ceph/ceph-client.admin.38556.log.gz 2024-09-16T15:52:51.514 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.35645.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.35645.log.gz 2024-09-16T15:52:51.514 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38634.log 2024-09-16T15:52:51.514 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.54828.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38381.log 2024-09-16T15:52:51.514 INFO:teuthology.orchestra.run.smithi123.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.54828.log.gz/var/log/ceph/ceph-client.admin.38634.log: 2024-09-16T15:52:51.514 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43456.log 2024-09-16T15:52:51.514 INFO:teuthology.orchestra.run.smithi123.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.38634.log.gz 2024-09-16T15:52:51.514 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.38381.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.38381.log.gz 2024-09-16T15:52:51.515 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52503.log 2024-09-16T15:52:51.515 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.43456.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50918.log 2024-09-16T15:52:51.515 INFO:teuthology.orchestra.run.smithi123.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.43456.log.gz/var/log/ceph/ceph-client.admin.52503.log: 2024-09-16T15:52:51.515 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48359.log 2024-09-16T15:52:51.515 INFO:teuthology.orchestra.run.smithi123.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.52503.log.gz 2024-09-16T15:52:51.515 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.50918.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.50918.log.gz 2024-09-16T15:52:51.515 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54208.log 2024-09-16T15:52:51.515 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.48359.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49163.log 2024-09-16T15:52:51.515 INFO:teuthology.orchestra.run.smithi123.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.48359.log.gz/var/log/ceph/ceph-client.admin.54208.log: 2024-09-16T15:52:51.515 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55102.log 2024-09-16T15:52:51.515 INFO:teuthology.orchestra.run.smithi123.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.54208.log.gz 2024-09-16T15:52:51.515 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.49163.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.49163.log.gz 2024-09-16T15:52:51.515 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55510.log 2024-09-16T15:52:51.516 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.55102.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48599.log 2024-09-16T15:52:51.516 INFO:teuthology.orchestra.run.smithi123.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.55102.log.gz/var/log/ceph/ceph-client.admin.55510.log: 2024-09-16T15:52:51.516 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34709.log 2024-09-16T15:52:51.516 INFO:teuthology.orchestra.run.smithi123.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.55510.log.gz 2024-09-16T15:52:51.516 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.48599.log: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.48599.log.gz 2024-09-16T15:52:51.516 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41946.log 2024-09-16T15:52:51.516 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.34709.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50446.log 2024-09-16T15:52:51.516 INFO:teuthology.orchestra.run.smithi123.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.34709.log.gz/var/log/ceph/ceph-client.admin.41946.log: 2024-09-16T15:52:51.516 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55666.log 2024-09-16T15:52:51.516 INFO:teuthology.orchestra.run.smithi123.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.41946.log.gz 2024-09-16T15:52:51.516 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.50446.log: 84.4% -- replaced with /var/log/ceph/ceph-client.admin.50446.log.gz 2024-09-16T15:52:51.516 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55796.log 2024-09-16T15:52:51.517 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.55666.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47068.log 2024-09-16T15:52:51.517 INFO:teuthology.orchestra.run.smithi123.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.55666.log.gz/var/log/ceph/ceph-client.admin.55796.log: 2024-09-16T15:52:51.517 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44852.log 2024-09-16T15:52:51.517 INFO:teuthology.orchestra.run.smithi123.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.55796.log.gz 2024-09-16T15:52:51.517 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.47068.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.47068.log.gz 2024-09-16T15:52:51.517 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42445.log 2024-09-16T15:52:51.519 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.44852.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51288.log 2024-09-16T15:52:51.520 INFO:teuthology.orchestra.run.smithi123.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.44852.log.gz/var/log/ceph/ceph-client.admin.42445.log: 2024-09-16T15:52:51.520 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49564.log 2024-09-16T15:52:51.520 INFO:teuthology.orchestra.run.smithi123.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.42445.log.gz 2024-09-16T15:52:51.520 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.51288.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.51288.log.gz 2024-09-16T15:52:51.520 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35143.log 2024-09-16T15:52:51.520 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.49564.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39005.log 2024-09-16T15:52:51.520 INFO:teuthology.orchestra.run.smithi123.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.49564.log.gz/var/log/ceph/ceph-client.admin.35143.log: 2024-09-16T15:52:51.520 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55692.log 2024-09-16T15:52:51.521 INFO:teuthology.orchestra.run.smithi123.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.35143.log.gz 2024-09-16T15:52:51.521 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.39005.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.39005.log.gz 2024-09-16T15:52:51.521 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36682.log 2024-09-16T15:52:51.521 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.55692.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40172.log 2024-09-16T15:52:51.521 INFO:teuthology.orchestra.run.smithi123.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.55692.log.gz/var/log/ceph/ceph-client.admin.36682.log: 2024-09-16T15:52:51.521 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45731.log 2024-09-16T15:52:51.521 INFO:teuthology.orchestra.run.smithi123.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.36682.log.gz 2024-09-16T15:52:51.521 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.40172.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.40172.log.gz 2024-09-16T15:52:51.521 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55848.log 2024-09-16T15:52:51.521 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.45731.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54854.log 2024-09-16T15:52:51.521 INFO:teuthology.orchestra.run.smithi123.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.45731.log.gz/var/log/ceph/ceph-client.admin.55848.log: 2024-09-16T15:52:51.522 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38355.log 2024-09-16T15:52:51.522 INFO:teuthology.orchestra.run.smithi123.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.55848.log.gz 2024-09-16T15:52:51.522 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.54854.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.54854.log.gz 2024-09-16T15:52:51.522 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50107.log 2024-09-16T15:52:51.522 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.38355.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55562.log 2024-09-16T15:52:51.522 INFO:teuthology.orchestra.run.smithi123.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.38355.log.gz/var/log/ceph/ceph-client.admin.50107.log: 2024-09-16T15:52:51.522 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55952.log 2024-09-16T15:52:51.522 INFO:teuthology.orchestra.run.smithi123.stderr: 84.3% -- replaced with /var/log/ceph/ceph-client.admin.50107.log.gz 2024-09-16T15:52:51.522 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.55562.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.55562.log.gz 2024-09-16T15:52:51.522 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55978.log 2024-09-16T15:52:51.522 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.55952.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55536.log 2024-09-16T15:52:51.523 INFO:teuthology.orchestra.run.smithi123.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.55952.log.gz/var/log/ceph/ceph-client.admin.55978.log: 2024-09-16T15:52:51.523 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48666.log 2024-09-16T15:52:51.523 INFO:teuthology.orchestra.run.smithi123.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.55978.log.gz 2024-09-16T15:52:51.523 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.55536.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.55536.log.gz 2024-09-16T15:52:51.523 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49880.log 2024-09-16T15:52:51.523 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.48666.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38131.log 2024-09-16T15:52:51.523 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.49880.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.48666.log.gz 2024-09-16T15:52:51.523 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55458.log 2024-09-16T15:52:51.523 INFO:teuthology.orchestra.run.smithi123.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.49880.log.gz 2024-09-16T15:52:51.523 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36001.log 2024-09-16T15:52:51.523 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.38131.log: /var/log/ceph/ceph-client.admin.55458.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52609.log 2024-09-16T15:52:51.524 INFO:teuthology.orchestra.run.smithi123.stderr: 87.0%/var/log/ceph/ceph-client.admin.36001.log: -- replaced with /var/log/ceph/ceph-client.admin.55458.log.gz 87.1% -- replaced with /var/log/ceph/ceph-client.admin.38131.log.gz 2024-09-16T15:52:51.524 INFO:teuthology.orchestra.run.smithi123.stderr: 2024-09-16T15:52:51.524 INFO:teuthology.orchestra.run.smithi123.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.36001.log.gz 2024-09-16T15:52:51.524 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46330.log 2024-09-16T15:52:51.524 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55718.log 2024-09-16T15:52:51.524 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.52609.log: /var/log/ceph/ceph-client.admin.46330.log: 87.8%gzip -- replaced with /var/log/ceph/ceph-client.admin.52609.log.gz -5 --verbose -- /var/log/ceph/ceph-client.admin.50032.log 2024-09-16T15:52:51.524 INFO:teuthology.orchestra.run.smithi123.stderr: 2024-09-16T15:52:51.524 INFO:teuthology.orchestra.run.smithi123.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.46330.log.gz 2024-09-16T15:52:51.524 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.55718.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.55718.log.gz 2024-09-16T15:52:51.524 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50814.log 2024-09-16T15:52:51.524 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.50032.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39317.log 2024-09-16T15:52:51.525 INFO:teuthology.orchestra.run.smithi123.stderr: 84.8% -- replaced with /var/log/ceph/ceph-client.admin.50032.log.gz 2024-09-16T15:52:51.525 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.50814.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2024-09-16T15:52:51.525 INFO:teuthology.orchestra.run.smithi123.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.50814.log.gz 2024-09-16T15:52:51.525 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53484.log 2024-09-16T15:52:51.525 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.39317.log: /var/log/ceph/ceph-osd.1.log: 91.3% -- replaced with /var/log/ceph/ceph-client.admin.39317.log.gz 2024-09-16T15:52:51.525 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29058.log 2024-09-16T15:52:51.525 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.53484.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40597.log 2024-09-16T15:52:51.525 INFO:teuthology.orchestra.run.smithi123.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.53484.log.gz 2024-09-16T15:52:51.525 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.29058.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29058.log.gz 2024-09-16T15:52:51.525 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38721.log 2024-09-16T15:52:51.525 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48693.log 2024-09-16T15:52:51.525 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.40597.log: /var/log/ceph/ceph-client.admin.38721.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.40597.log.gz 2024-09-16T15:52:51.526 INFO:teuthology.orchestra.run.smithi123.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.38721.log.gz 2024-09-16T15:52:51.526 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51963.log 2024-09-16T15:52:51.526 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.48693.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.48693.log.gz 2024-09-16T15:52:51.526 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55874.log 2024-09-16T15:52:51.526 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48069.log 2024-09-16T15:52:51.526 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.51963.log: /var/log/ceph/ceph-client.admin.55874.log: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.51963.log.gz 2024-09-16T15:52:51.526 INFO:teuthology.orchestra.run.smithi123.stderr: 87.7%gzip -- replaced with /var/log/ceph/ceph-client.admin.55874.log.gz 2024-09-16T15:52:51.526 INFO:teuthology.orchestra.run.smithi123.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.44544.log 2024-09-16T15:52:51.526 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.48069.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.48069.log.gz 2024-09-16T15:52:51.526 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45570.log 2024-09-16T15:52:51.526 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39083.log 2024-09-16T15:52:51.526 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.44544.log: /var/log/ceph/ceph-client.admin.45570.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.44544.log.gz 2024-09-16T15:52:51.526 INFO:teuthology.orchestra.run.smithi123.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.45570.log.gz 2024-09-16T15:52:51.527 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46921.log 2024-09-16T15:52:51.527 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44202.log 2024-09-16T15:52:51.527 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.39083.log: /var/log/ceph/ceph-client.admin.46921.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.39083.log.gz 2024-09-16T15:52:51.527 INFO:teuthology.orchestra.run.smithi123.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.46921.log.gz 2024-09-16T15:52:51.527 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35659.log 2024-09-16T15:52:51.527 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42128.log 2024-09-16T15:52:51.527 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.44202.log: /var/log/ceph/ceph-client.admin.35659.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.44202.log.gz 2024-09-16T15:52:51.527 INFO:teuthology.orchestra.run.smithi123.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.35659.log.gz 2024-09-16T15:52:51.527 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36435.log 2024-09-16T15:52:51.527 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.42128.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.42128.log.gz 2024-09-16T15:52:51.527 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53446.log 2024-09-16T15:52:51.527 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.36435.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48719.log 2024-09-16T15:52:51.527 INFO:teuthology.orchestra.run.smithi123.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.36435.log.gz 2024-09-16T15:52:51.527 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.53446.log: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.53446.log.gz 2024-09-16T15:52:51.527 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50529.log 2024-09-16T15:52:51.528 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35445.log 2024-09-16T15:52:51.528 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.48719.log: /var/log/ceph/ceph-client.admin.50529.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.48719.log.gz 2024-09-16T15:52:51.528 INFO:teuthology.orchestra.run.smithi123.stderr: 88.0%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39747.log 2024-09-16T15:52:51.528 INFO:teuthology.orchestra.run.smithi123.stderr: -- replaced with /var/log/ceph/ceph-client.admin.50529.log.gz 2024-09-16T15:52:51.528 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.35445.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.35445.log.gz 2024-09-16T15:52:51.528 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42835.log 2024-09-16T15:52:51.528 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56055.log 2024-09-16T15:52:51.528 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.39747.log: /var/log/ceph/ceph-client.admin.42835.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.39747.log.gz 2024-09-16T15:52:51.528 INFO:teuthology.orchestra.run.smithi123.stderr: 87.3%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41635.log 2024-09-16T15:52:51.528 INFO:teuthology.orchestra.run.smithi123.stderr: -- replaced with /var/log/ceph/ceph-client.admin.42835.log.gz 2024-09-16T15:52:51.528 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.56055.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.56055.log.gz 2024-09-16T15:52:51.529 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50970.log 2024-09-16T15:52:51.529 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44931.log 2024-09-16T15:52:51.529 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.41635.log: /var/log/ceph/ceph-client.admin.50970.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.41635.log.gz 2024-09-16T15:52:51.529 INFO:teuthology.orchestra.run.smithi123.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.50970.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54434.log 2024-09-16T15:52:51.529 INFO:teuthology.orchestra.run.smithi123.stderr: 2024-09-16T15:52:51.529 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.44931.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.44931.log.gz 2024-09-16T15:52:51.529 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53972.log 2024-09-16T15:52:51.529 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50691.log 2024-09-16T15:52:51.529 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.54434.log: /var/log/ceph/ceph-client.admin.53972.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.54434.log.gz 2024-09-16T15:52:51.529 INFO:teuthology.orchestra.run.smithi123.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.53972.log.gz 2024-09-16T15:52:51.529 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48635.log 2024-09-16T15:52:51.529 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40545.log 2024-09-16T15:52:51.529 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.50691.log: /var/log/ceph/ceph-client.admin.48635.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.50691.log.gz 2024-09-16T15:52:51.530 INFO:teuthology.orchestra.run.smithi123.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.48635.log.gz 2024-09-16T15:52:51.530 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45329.log 2024-09-16T15:52:51.530 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39908.log 2024-09-16T15:52:51.530 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.40545.log: /var/log/ceph/ceph-client.admin.45329.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.40545.log.gz 2024-09-16T15:52:51.530 INFO:teuthology.orchestra.run.smithi123.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.45329.log.gz 2024-09-16T15:52:51.530 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34618.log 2024-09-16T15:52:51.530 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36053.log 2024-09-16T15:52:51.530 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.39908.log: /var/log/ceph/ceph-client.admin.34618.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.39908.log.gz 2024-09-16T15:52:51.530 INFO:teuthology.orchestra.run.smithi123.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.34618.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56005.log 2024-09-16T15:52:51.530 INFO:teuthology.orchestra.run.smithi123.stderr: 2024-09-16T15:52:51.530 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.36053.log: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.36053.log.gz 2024-09-16T15:52:51.530 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39135.log 2024-09-16T15:52:51.530 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45139.log 2024-09-16T15:52:51.530 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.56005.log: /var/log/ceph/ceph-client.admin.39135.log: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.56005.log.gz 2024-09-16T15:52:51.531 INFO:teuthology.orchestra.run.smithi123.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.39135.log.gz 2024-09-16T15:52:51.531 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46254.log 2024-09-16T15:52:51.531 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54000.log 2024-09-16T15:52:51.531 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.45139.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.45139.log.gz 2024-09-16T15:52:51.531 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.46254.log: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.46254.log.gz 2024-09-16T15:52:51.531 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49404.log 2024-09-16T15:52:51.531 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.54000.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54026.log 2024-09-16T15:52:51.531 INFO:teuthology.orchestra.run.smithi123.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.54000.log.gz 2024-09-16T15:52:51.531 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.49404.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.49404.log.gz 2024-09-16T15:52:51.531 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41490.log 2024-09-16T15:52:51.531 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51204.log 2024-09-16T15:52:51.531 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.54026.log: /var/log/ceph/ceph-client.admin.41490.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.54026.log.gz 2024-09-16T15:52:51.531 INFO:teuthology.orchestra.run.smithi123.stderr: 87.4%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38160.log 2024-09-16T15:52:51.531 INFO:teuthology.orchestra.run.smithi123.stderr: -- replaced with /var/log/ceph/ceph-client.admin.41490.log.gz 2024-09-16T15:52:51.532 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.51204.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.51204.log.gz 2024-09-16T15:52:51.532 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46735.log 2024-09-16T15:52:51.532 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52217.log 2024-09-16T15:52:51.532 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.38160.log: /var/log/ceph/ceph-client.admin.46735.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.38160.log.gz 2024-09-16T15:52:51.532 INFO:teuthology.orchestra.run.smithi123.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.46735.log.gzgzip 2024-09-16T15:52:51.532 INFO:teuthology.orchestra.run.smithi123.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.50296.log 2024-09-16T15:52:51.532 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.52217.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.52217.log.gz 2024-09-16T15:52:51.532 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52900.log 2024-09-16T15:52:51.532 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51989.log 2024-09-16T15:52:51.532 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.50296.log: /var/log/ceph/ceph-client.admin.52900.log: 84.6% -- replaced with /var/log/ceph/ceph-client.admin.50296.log.gz 2024-09-16T15:52:51.532 INFO:teuthology.orchestra.run.smithi123.stderr: 87.7%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41849.log 2024-09-16T15:52:51.532 INFO:teuthology.orchestra.run.smithi123.stderr: -- replaced with /var/log/ceph/ceph-client.admin.52900.log.gz 2024-09-16T15:52:51.532 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.51989.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.51989.log.gz 2024-09-16T15:52:51.532 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34883.log 2024-09-16T15:52:51.533 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45535.log 2024-09-16T15:52:51.533 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.41849.log: /var/log/ceph/ceph-client.admin.34883.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.41849.log.gz 2024-09-16T15:52:51.533 INFO:teuthology.orchestra.run.smithi123.stderr: 87.9%gzip -- replaced with /var/log/ceph/ceph-client.admin.34883.log.gz 2024-09-16T15:52:51.533 INFO:teuthology.orchestra.run.smithi123.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.38753.log 2024-09-16T15:52:51.533 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.45535.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.45535.log.gz 2024-09-16T15:52:51.533 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46407.log 2024-09-16T15:52:51.533 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35246.log 2024-09-16T15:52:51.533 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.38753.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.38753.log.gz 2024-09-16T15:52:51.533 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.46407.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.46407.log.gz 2024-09-16T15:52:51.533 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49026.log 2024-09-16T15:52:51.533 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.35246.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38224.log 2024-09-16T15:52:51.533 INFO:teuthology.orchestra.run.smithi123.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.35246.log.gz/var/log/ceph/ceph-client.admin.49026.log: 2024-09-16T15:52:51.533 INFO:teuthology.orchestra.run.smithi123.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.49026.log.gz 2024-09-16T15:52:51.534 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29047.log 2024-09-16T15:52:51.534 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35456.log 2024-09-16T15:52:51.534 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.38224.log: /var/log/ceph/ceph-client.admin.29047.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.38224.log.gz 2024-09-16T15:52:51.534 INFO:teuthology.orchestra.run.smithi123.stderr: 78.6% -- replaced with /var/log/ceph/ceph-client.admin.29047.log.gz 2024-09-16T15:52:51.534 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39825.log 2024-09-16T15:52:51.534 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28974.log 2024-09-16T15:52:51.534 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.35456.log: /var/log/ceph/ceph-client.admin.39825.log: 87.7% 87.6% -- replaced with /var/log/ceph/ceph-client.admin.35456.log.gz 2024-09-16T15:52:51.534 INFO:teuthology.orchestra.run.smithi123.stderr: -- replaced with /var/log/ceph/ceph-client.admin.39825.log.gz 2024-09-16T15:52:51.534 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40441.log 2024-09-16T15:52:51.534 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49369.log 2024-09-16T15:52:51.534 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.28974.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28974.log.gz 2024-09-16T15:52:51.534 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.40441.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46842.log 2024-09-16T15:52:51.534 INFO:teuthology.orchestra.run.smithi123.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.40441.log.gz 2024-09-16T15:52:51.534 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.49369.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.49369.log.gz 2024-09-16T15:52:51.535 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.24014.log 2024-09-16T15:52:51.535 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48019.log 2024-09-16T15:52:51.535 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.46842.log: /var/log/ceph/ceph.tmp-client.admin.24014.log: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.46842.log.gz 2024-09-16T15:52:51.535 INFO:teuthology.orchestra.run.smithi123.stderr: 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.24014.log.gz 2024-09-16T15:52:51.535 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47120.log 2024-09-16T15:52:51.535 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49286.log 2024-09-16T15:52:51.535 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.48019.log: /var/log/ceph/ceph-client.admin.47120.log: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.47120.log.gz 2024-09-16T15:52:51.535 INFO:teuthology.orchestra.run.smithi123.stderr: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.48019.log.gz 2024-09-16T15:52:51.535 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48179.log 2024-09-16T15:52:51.535 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35439.log 2024-09-16T15:52:51.535 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.49286.log: /var/log/ceph/ceph-client.admin.48179.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.49286.log.gz 2024-09-16T15:52:51.535 INFO:teuthology.orchestra.run.smithi123.stderr: 87.9%gzip -- replaced with /var/log/ceph/ceph-client.admin.48179.log.gz -5 --verbose -- /var/log/ceph/ceph-client.admin.50259.log 2024-09-16T15:52:51.535 INFO:teuthology.orchestra.run.smithi123.stderr: 2024-09-16T15:52:51.535 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.35439.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.35439.log.gz 2024-09-16T15:52:51.536 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47427.log 2024-09-16T15:52:51.536 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40656.log 2024-09-16T15:52:51.536 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.50259.log: /var/log/ceph/ceph-client.admin.47427.log: 84.8% -- replaced with /var/log/ceph/ceph-client.admin.50259.log.gz 2024-09-16T15:52:51.536 INFO:teuthology.orchestra.run.smithi123.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.47427.log.gz 2024-09-16T15:52:51.536 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50587.log 2024-09-16T15:52:51.536 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35897.log 2024-09-16T15:52:51.536 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.40656.log: /var/log/ceph/ceph-client.admin.50587.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.40656.log.gz 2024-09-16T15:52:51.536 INFO:teuthology.orchestra.run.smithi123.stderr: 87.7%gzip -- replaced with /var/log/ceph/ceph-client.admin.50587.log.gz 2024-09-16T15:52:51.536 INFO:teuthology.orchestra.run.smithi123.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.40923.log 2024-09-16T15:52:51.558 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.35897.log: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.35897.log.gz 2024-09-16T15:52:51.558 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50371.log 2024-09-16T15:52:51.558 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-09-16T15:52:51.558 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.40923.log: /var/log/ceph/ceph-client.admin.50371.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.40923.log.gz 2024-09-16T15:52:51.558 INFO:teuthology.orchestra.run.smithi123.stderr: 85.1% -- replaced with /var/log/ceph/ceph-client.admin.50371.log.gz 2024-09-16T15:52:51.559 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36649.log 2024-09-16T15:52:51.559 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36545.log 2024-09-16T15:52:51.559 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph.log: /var/log/ceph/ceph-client.admin.36649.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.36649.log.gz 2024-09-16T15:52:51.559 INFO:teuthology.orchestra.run.smithi123.stderr: 86.8% -- replaced with /var/log/ceph/ceph.log.gz 2024-09-16T15:52:51.559 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43482.log 2024-09-16T15:52:51.559 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29033.log 2024-09-16T15:52:51.559 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.36545.log: /var/log/ceph/ceph-client.admin.43482.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.36545.log.gz 2024-09-16T15:52:51.559 INFO:teuthology.orchestra.run.smithi123.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.43482.log.gz 2024-09-16T15:52:51.559 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51178.log 2024-09-16T15:52:51.559 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43781.log 2024-09-16T15:52:51.559 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.29033.log: /var/log/ceph/ceph-client.admin.51178.log: 77.5% -- replaced with /var/log/ceph/ceph-client.admin.29033.log.gz 2024-09-16T15:52:51.559 INFO:teuthology.orchestra.run.smithi123.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.51178.log.gz 2024-09-16T15:52:51.559 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52126.log 2024-09-16T15:52:51.560 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.43781.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.43781.log.gz 2024-09-16T15:52:51.560 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42523.log 2024-09-16T15:52:51.560 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43755.log 2024-09-16T15:52:51.560 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.52126.log: /var/log/ceph/ceph-client.admin.42523.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.52126.log.gz 2024-09-16T15:52:51.560 INFO:teuthology.orchestra.run.smithi123.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.42523.log.gz 2024-09-16T15:52:51.560 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41374.log 2024-09-16T15:52:51.560 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36929.log 2024-09-16T15:52:51.560 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.43755.log: /var/log/ceph/ceph-client.admin.41374.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.43755.log.gz 2024-09-16T15:52:51.560 INFO:teuthology.orchestra.run.smithi123.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.41374.log.gz 2024-09-16T15:52:51.560 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41693.log 2024-09-16T15:52:51.560 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39669.log 2024-09-16T15:52:51.560 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.36929.log: /var/log/ceph/ceph-client.admin.41693.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.36929.log.gz 2024-09-16T15:52:51.561 INFO:teuthology.orchestra.run.smithi123.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.41693.log.gz 2024-09-16T15:52:51.561 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40862.log 2024-09-16T15:52:51.561 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54538.log 2024-09-16T15:52:51.561 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.39669.log: /var/log/ceph/ceph-client.admin.40862.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.39669.log.gz 2024-09-16T15:52:51.561 INFO:teuthology.orchestra.run.smithi123.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.40862.log.gz 2024-09-16T15:52:51.561 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42549.log 2024-09-16T15:52:51.561 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51055.log 2024-09-16T15:52:51.561 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.54538.log: /var/log/ceph/ceph-client.admin.42549.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.54538.log.gz 2024-09-16T15:52:51.561 INFO:teuthology.orchestra.run.smithi123.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.42549.log.gz 2024-09-16T15:52:51.561 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37340.log 2024-09-16T15:52:51.561 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42050.log 2024-09-16T15:52:51.561 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.51055.log: /var/log/ceph/ceph-client.admin.37340.log: 87.7% 87.7% -- replaced with /var/log/ceph/ceph-client.admin.51055.log.gz 2024-09-16T15:52:51.561 INFO:teuthology.orchestra.run.smithi123.stderr: -- replaced with /var/log/ceph/ceph-client.admin.37340.log.gz 2024-09-16T15:52:51.562 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35117.log 2024-09-16T15:52:51.562 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39367.log 2024-09-16T15:52:51.562 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.42050.log: /var/log/ceph/ceph-client.admin.35117.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.42050.log.gz 2024-09-16T15:52:51.562 INFO:teuthology.orchestra.run.smithi123.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.35117.log.gz 2024-09-16T15:52:51.562 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48385.log 2024-09-16T15:52:51.562 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51548.log 2024-09-16T15:52:51.562 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.39367.log: /var/log/ceph/ceph-client.admin.48385.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.39367.log.gz 2024-09-16T15:52:51.562 INFO:teuthology.orchestra.run.smithi123.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.48385.log.gz 2024-09-16T15:52:51.562 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47545.log 2024-09-16T15:52:51.562 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44905.log 2024-09-16T15:52:51.562 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.51548.log: /var/log/ceph/ceph-client.admin.47545.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.51548.log.gz 2024-09-16T15:52:51.562 INFO:teuthology.orchestra.run.smithi123.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.47545.log.gz 2024-09-16T15:52:51.562 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52765.log 2024-09-16T15:52:51.563 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48437.log 2024-09-16T15:52:51.563 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.44905.log: /var/log/ceph/ceph-client.admin.52765.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.44905.log.gz 2024-09-16T15:52:51.563 INFO:teuthology.orchestra.run.smithi123.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.52765.log.gz 2024-09-16T15:52:51.563 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46816.log 2024-09-16T15:52:51.563 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28961.log 2024-09-16T15:52:51.563 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.48437.log: /var/log/ceph/ceph-client.admin.46816.log: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.48437.log.gz 2024-09-16T15:52:51.563 INFO:teuthology.orchestra.run.smithi123.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.46816.log.gz 2024-09-16T15:52:51.563 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51087.log 2024-09-16T15:52:51.563 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51470.log 2024-09-16T15:52:51.563 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.28961.log: /var/log/ceph/ceph-client.admin.51087.log: 76.7% -- replaced with /var/log/ceph/ceph-client.admin.28961.log.gz 2024-09-16T15:52:51.563 INFO:teuthology.orchestra.run.smithi123.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.51087.log.gz 2024-09-16T15:52:51.563 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36709.log 2024-09-16T15:52:51.563 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46790.log 2024-09-16T15:52:51.563 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.51470.log: /var/log/ceph/ceph-client.admin.36709.log: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.51470.log.gz 2024-09-16T15:52:51.564 INFO:teuthology.orchestra.run.smithi123.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.36709.log.gz 2024-09-16T15:52:51.564 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44821.log 2024-09-16T15:52:51.564 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.46790.log: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.46790.log.gz 2024-09-16T15:52:51.564 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46984.log 2024-09-16T15:52:51.564 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52047.log 2024-09-16T15:52:51.564 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.44821.log: /var/log/ceph/ceph-client.admin.46984.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.44821.log.gz 2024-09-16T15:52:51.564 INFO:teuthology.orchestra.run.smithi123.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.46984.log.gz 2024-09-16T15:52:51.564 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35455.log 2024-09-16T15:52:51.564 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.52047.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52243.log 2024-09-16T15:52:51.564 INFO:teuthology.orchestra.run.smithi123.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.52047.log.gz 2024-09-16T15:52:51.564 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.35455.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.35455.log.gz 2024-09-16T15:52:51.564 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50779.log 2024-09-16T15:52:51.565 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.52243.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49642.log 2024-09-16T15:52:51.565 INFO:teuthology.orchestra.run.smithi123.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.52243.log.gz 2024-09-16T15:52:51.565 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.50779.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.50779.log.gz 2024-09-16T15:52:51.565 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28986.log 2024-09-16T15:52:51.565 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54378.log 2024-09-16T15:52:51.565 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.49642.log: /var/log/ceph/ceph-client.admin.28986.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.49642.log.gz 2024-09-16T15:52:51.565 INFO:teuthology.orchestra.run.smithi123.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.28986.log.gz 2024-09-16T15:52:51.565 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48206.log 2024-09-16T15:52:51.565 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.54378.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.54378.log.gz 2024-09-16T15:52:51.565 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.45941.log 2024-09-16T15:52:51.565 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52926.log 2024-09-16T15:52:51.565 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.48206.log: /var/log/ceph/ceph-client.0.45941.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.48206.log.gz 2024-09-16T15:52:51.565 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45809.log 2024-09-16T15:52:51.566 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.52926.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.52926.log.gz 2024-09-16T15:52:51.566 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46871.log 2024-09-16T15:52:51.566 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.45809.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.45809.log.gz 2024-09-16T15:52:51.566 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39986.log 2024-09-16T15:52:51.566 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.46871.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.46871.log.gz 2024-09-16T15:52:51.566 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44280.log 2024-09-16T15:52:51.566 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.39986.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.39986.log.gz 2024-09-16T15:52:51.566 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54486.log 2024-09-16T15:52:51.566 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.44280.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.44280.log.gz 2024-09-16T15:52:51.566 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54616.log 2024-09-16T15:52:51.566 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.54486.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.54486.log.gz 2024-09-16T15:52:51.566 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51626.log 2024-09-16T15:52:51.567 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.54616.log: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.54616.log.gz 2024-09-16T15:52:51.567 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38102.log 2024-09-16T15:52:51.567 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.51626.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.51626.log.gz 2024-09-16T15:52:51.567 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35235.log 2024-09-16T15:52:51.567 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.38102.log: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.38102.log.gz 2024-09-16T15:52:51.567 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41521.log 2024-09-16T15:52:51.567 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.35235.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.35235.log.gz 2024-09-16T15:52:51.567 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44306.log 2024-09-16T15:52:51.567 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.41521.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.41521.log.gz 2024-09-16T15:52:51.567 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53516.log 2024-09-16T15:52:51.573 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.44306.log: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.44306.log.gz 2024-09-16T15:52:51.573 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51496.log 2024-09-16T15:52:51.573 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.53516.log: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.53516.log.gz 2024-09-16T15:52:51.573 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50665.log 2024-09-16T15:52:51.573 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.51496.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.51496.log.gz 2024-09-16T15:52:51.574 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44358.log 2024-09-16T15:52:51.574 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.50665.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.50665.log.gz 2024-09-16T15:52:51.574 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51444.log 2024-09-16T15:52:51.574 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.44358.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.44358.log.gz 2024-09-16T15:52:51.574 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55329.log 2024-09-16T15:52:51.574 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.51444.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.51444.log.gz 2024-09-16T15:52:51.574 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45295.log 2024-09-16T15:52:51.574 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.55329.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.55329.log.gz 2024-09-16T15:52:51.574 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55409.log 2024-09-16T15:52:51.574 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.45295.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.45295.log.gz 2024-09-16T15:52:51.574 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39610.log 2024-09-16T15:52:51.574 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.55409.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.55409.log.gz 2024-09-16T15:52:51.575 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44042.log 2024-09-16T15:52:51.575 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.39610.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.39610.log.gz 2024-09-16T15:52:51.575 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45757.log 2024-09-16T15:52:51.575 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.44042.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.44042.log.gz 2024-09-16T15:52:51.575 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44983.log 2024-09-16T15:52:51.575 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.45757.log: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.45757.log.gz 2024-09-16T15:52:51.575 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50007.log 2024-09-16T15:52:51.577 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.44983.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.44983.log.gz 2024-09-16T15:52:51.577 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50613.log 2024-09-16T15:52:51.577 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.50007.log: 85.2% -- replaced with /var/log/ceph/ceph-client.admin.50007.log.gz 2024-09-16T15:52:51.577 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41745.log 2024-09-16T15:52:51.577 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.50613.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.50613.log.gz 2024-09-16T15:52:51.578 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47780.log 2024-09-16T15:52:51.578 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.41745.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.41745.log.gz 2024-09-16T15:52:51.578 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45705.log 2024-09-16T15:52:51.579 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.47780.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.47780.log.gz 2024-09-16T15:52:51.579 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47172.log 2024-09-16T15:52:51.580 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.45705.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.45705.log.gz 2024-09-16T15:52:51.581 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38251.log 2024-09-16T15:52:51.581 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.47172.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.47172.log.gz 2024-09-16T15:52:51.582 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.54722.log 2024-09-16T15:52:51.583 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.38251.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.38251.log.gz 2024-09-16T15:52:51.583 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53134.log 2024-09-16T15:52:51.590 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.0.54722.log: 92.2%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37314.log 2024-09-16T15:52:51.590 INFO:teuthology.orchestra.run.smithi123.stderr: -- replaced with /var/log/ceph/ceph-client.0.54722.log.gz 2024-09-16T15:52:51.591 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.53134.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.53134.log.gz 2024-09-16T15:52:51.594 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54460.log 2024-09-16T15:52:51.595 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.37314.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.37314.log.gz 2024-09-16T15:52:51.595 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.2.36383.log 2024-09-16T15:52:51.596 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.54460.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.54460.log.gz 2024-09-16T15:52:51.596 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37474.log 2024-09-16T15:52:51.598 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.2.36383.log: 89.5% -- replaced with /var/log/ceph/ceph-client.2.36383.log.gz 2024-09-16T15:52:51.598 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35236.log 2024-09-16T15:52:51.599 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.37474.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.37474.log.gz 2024-09-16T15:52:51.599 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47941.log 2024-09-16T15:52:51.600 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.35236.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.35236.log.gz 2024-09-16T15:52:51.600 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45269.log 2024-09-16T15:52:51.601 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.47941.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.47941.log.gz 2024-09-16T15:52:51.602 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44254.log 2024-09-16T15:52:51.603 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.45269.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.45269.log.gz 2024-09-16T15:52:51.603 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55588.log 2024-09-16T15:52:51.604 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.44254.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.44254.log.gz 2024-09-16T15:52:51.604 INFO:teuthology.orchestra.run.smithi123.stderr: 93.3% -- replaced with /var/log/ceph/ceph-client.0.45941.log.gz 2024-09-16T15:52:51.604 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51366.log 2024-09-16T15:52:51.605 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36493.log 2024-09-16T15:52:51.605 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.55588.log: /var/log/ceph/ceph-client.admin.51366.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.55588.log.gz 2024-09-16T15:52:51.605 INFO:teuthology.orchestra.run.smithi123.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.51366.log.gz 2024-09-16T15:52:51.606 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36794.log 2024-09-16T15:52:51.606 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47691.log 2024-09-16T15:52:51.607 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.36493.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.36493.log.gz 2024-09-16T15:52:51.607 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.36794.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.36794.log.gz 2024-09-16T15:52:51.608 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46280.log 2024-09-16T15:52:51.608 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.47691.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39960.log 2024-09-16T15:52:51.608 INFO:teuthology.orchestra.run.smithi123.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.47691.log.gz/var/log/ceph/ceph-client.admin.46280.log: 2024-09-16T15:52:51.608 INFO:teuthology.orchestra.run.smithi123.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.46280.log.gz 2024-09-16T15:52:51.609 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41435.log 2024-09-16T15:52:51.609 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51418.log 2024-09-16T15:52:51.610 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.39960.log: /var/log/ceph/ceph-client.admin.41435.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.39960.log.gz 2024-09-16T15:52:51.610 INFO:teuthology.orchestra.run.smithi123.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.41435.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49854.log 2024-09-16T15:52:51.610 INFO:teuthology.orchestra.run.smithi123.stderr: 2024-09-16T15:52:51.611 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.51418.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.51418.log.gz 2024-09-16T15:52:51.611 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44957.log 2024-09-16T15:52:51.612 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52529.log 2024-09-16T15:52:51.612 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.49854.log: /var/log/ceph/ceph-client.admin.44957.log: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.49854.log.gz 2024-09-16T15:52:51.612 INFO:teuthology.orchestra.run.smithi123.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.44957.log.gz 2024-09-16T15:52:51.612 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48519.log 2024-09-16T15:52:51.613 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48922.log 2024-09-16T15:52:51.613 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.52529.log: /var/log/ceph/ceph-client.admin.48519.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.52529.log.gz 2024-09-16T15:52:51.614 INFO:teuthology.orchestra.run.smithi123.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.48519.log.gz 2024-09-16T15:52:51.614 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35091.log 2024-09-16T15:52:51.615 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-09-16T15:52:51.615 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.48922.log: /var/log/ceph/ceph-client.admin.35091.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.48922.log.gz 2024-09-16T15:52:51.615 INFO:teuthology.orchestra.run.smithi123.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.35091.log.gz 2024-09-16T15:52:51.615 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53894.log 2024-09-16T15:52:51.619 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph.audit.log: 89.7% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-09-16T15:52:51.621 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40231.log 2024-09-16T15:52:51.621 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.53894.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51574.log 2024-09-16T15:52:51.621 INFO:teuthology.orchestra.run.smithi123.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.53894.log.gz/var/log/ceph/ceph-client.admin.40231.log: 2024-09-16T15:52:51.621 INFO:teuthology.orchestra.run.smithi123.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.40231.log.gz 2024-09-16T15:52:51.621 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mds.c.log 2024-09-16T15:52:51.622 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54234.log 2024-09-16T15:52:51.622 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.51574.log: /var/log/ceph/ceph-mds.c.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.51574.log.gz 2024-09-16T15:52:51.622 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53210.log 2024-09-16T15:52:51.623 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.54234.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.54234.log.gz 2024-09-16T15:52:51.623 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54143.log 2024-09-16T15:52:51.624 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.53210.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.53210.log.gz 2024-09-16T15:52:51.624 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53712.log 2024-09-16T15:52:51.625 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.54143.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.54143.log.gz 2024-09-16T15:52:51.626 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.39423.log 2024-09-16T15:52:51.627 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.53712.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.53712.log.gz 2024-09-16T15:52:51.627 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42024.log 2024-09-16T15:52:51.628 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.0.39423.log: 89.1% -- replaced with /var/log/ceph/ceph-client.0.39423.log.gz 2024-09-16T15:52:51.629 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50717.log 2024-09-16T15:52:51.630 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.42024.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.42024.log.gz 2024-09-16T15:52:51.630 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52555.log 2024-09-16T15:52:51.631 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.50717.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.50717.log.gz 2024-09-16T15:52:51.631 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mds.a.log 2024-09-16T15:52:51.632 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.52555.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.52555.log.gz 2024-09-16T15:52:51.633 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47453.log 2024-09-16T15:52:51.646 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-mds.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40834.log 2024-09-16T15:52:51.647 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.47453.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.47453.log.gz 2024-09-16T15:52:51.650 INFO:teuthology.orchestra.run.smithi184.stderr: 92.8% -- replaced with /var/log/ceph/ceph-mds.d.log.gz 2024-09-16T15:52:51.650 INFO:teuthology.orchestra.run.smithi184.stderr: 91.4% -- replaced with /var/log/ceph/ceph-mon.b.log.gz 2024-09-16T15:52:51.654 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40493.log 2024-09-16T15:52:51.654 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.40834.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.40834.log.gz 2024-09-16T15:52:51.670 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42285.log 2024-09-16T15:52:51.671 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.40493.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.40493.log.gz 2024-09-16T15:52:51.678 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53566.log 2024-09-16T15:52:51.678 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.42285.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.42285.log.gz 2024-09-16T15:52:51.686 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48573.log 2024-09-16T15:52:51.686 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.53566.log: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.53566.log.gz 2024-09-16T15:52:51.694 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36903.log 2024-09-16T15:52:51.694 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.48573.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.48573.log.gz 2024-09-16T15:52:51.702 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47015.log 2024-09-16T15:52:51.702 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.36903.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.36903.log.gz 2024-09-16T15:52:51.710 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43675.log 2024-09-16T15:52:51.710 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.47015.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.47015.log.gz 2024-09-16T15:52:51.718 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50082.log 2024-09-16T15:52:51.718 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.43675.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.43675.log.gz 2024-09-16T15:52:51.726 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40688.log 2024-09-16T15:52:51.726 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.50082.log: 84.8% -- replaced with /var/log/ceph/ceph-client.admin.50082.log.gz 2024-09-16T15:52:51.734 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42785.log 2024-09-16T15:52:51.734 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.40688.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.40688.log.gz 2024-09-16T15:52:51.740 INFO:teuthology.orchestra.run.smithi184.stderr: 91.6% -- replaced with /var/log/ceph/ceph-mon.c.log.gz 2024-09-16T15:52:51.742 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28901.log 2024-09-16T15:52:51.742 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.42785.log: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.42785.log.gz 2024-09-16T15:52:51.750 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41057.log 2024-09-16T15:52:51.750 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.28901.log: 77.8% -- replaced with /var/log/ceph/ceph-client.admin.28901.log.gz 2024-09-16T15:52:51.758 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48321.log 2024-09-16T15:52:51.758 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.41057.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.41057.log.gz 2024-09-16T15:52:51.778 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34909.log 2024-09-16T15:52:51.778 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.48321.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.48321.log.gz 2024-09-16T15:52:51.786 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38039.log 2024-09-16T15:52:51.786 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.34909.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.34909.log.gz 2024-09-16T15:52:51.794 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38504.log 2024-09-16T15:52:51.794 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.38039.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.38039.log.gz 2024-09-16T15:52:51.802 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50944.log 2024-09-16T15:52:51.802 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.38504.log: 93.4% -- replaced with /var/log/ceph/ceph-mgr.y.log.gz 2024-09-16T15:52:51.802 INFO:teuthology.orchestra.run.smithi123.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.38504.log.gz 2024-09-16T15:52:51.803 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51262.log 2024-09-16T15:52:51.803 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.50944.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.50944.log.gz 2024-09-16T15:52:51.804 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46474.log 2024-09-16T15:52:51.804 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.51262.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.51262.log.gz 2024-09-16T15:52:51.805 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45191.log 2024-09-16T15:52:51.805 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.46474.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.46474.log.gz 2024-09-16T15:52:51.805 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40205.log 2024-09-16T15:52:51.806 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.45191.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.45191.log.gz 2024-09-16T15:52:51.806 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28998.log 2024-09-16T15:52:51.807 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.40205.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.40205.log.gz 2024-09-16T15:52:51.808 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42731.log 2024-09-16T15:52:51.809 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.28998.log: 78.6% -- replaced with /var/log/ceph/ceph-client.admin.28998.log.gz 2024-09-16T15:52:51.809 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53082.log 2024-09-16T15:52:51.810 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.42731.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.42731.log.gz 2024-09-16T15:52:51.810 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42497.log 2024-09-16T15:52:51.811 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.53082.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.53082.log.gz 2024-09-16T15:52:51.811 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28926.log 2024-09-16T15:52:51.812 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.42497.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.42497.log.gz 2024-09-16T15:52:51.812 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39291.log 2024-09-16T15:52:51.813 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.28926.log: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.28926.log.gz 2024-09-16T15:52:51.814 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53946.log 2024-09-16T15:52:51.815 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.39291.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.39291.log.gz 2024-09-16T15:52:51.815 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52018.log 2024-09-16T15:52:51.816 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.53946.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.53946.log.gz 2024-09-16T15:52:51.816 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40519.log 2024-09-16T15:52:51.817 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.52018.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.52018.log.gz 2024-09-16T15:52:51.817 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41083.log 2024-09-16T15:52:51.818 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.40519.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.40519.log.gz 2024-09-16T15:52:51.819 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47269.log 2024-09-16T15:52:51.819 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.41083.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.41083.log.gz 2024-09-16T15:52:51.820 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40067.log 2024-09-16T15:52:51.821 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.47269.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.47269.log.gz 2024-09-16T15:52:51.821 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40780.log 2024-09-16T15:52:51.822 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.40067.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.40067.log.gz 2024-09-16T15:52:51.822 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51152.log 2024-09-16T15:52:51.823 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.40780.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.40780.log.gz 2024-09-16T15:52:51.824 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38844.log 2024-09-16T15:52:51.824 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.51152.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.51152.log.gz 2024-09-16T15:52:51.825 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56029.log 2024-09-16T15:52:51.826 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.38844.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.38844.log.gz 2024-09-16T15:52:51.826 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50639.log 2024-09-16T15:52:51.827 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.56029.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.56029.log.gz 2024-09-16T15:52:51.827 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47889.log 2024-09-16T15:52:51.828 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.50639.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.50639.log.gz 2024-09-16T15:52:51.828 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50132.log 2024-09-16T15:52:51.829 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.47889.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.47889.log.gz 2024-09-16T15:52:51.830 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37059.log 2024-09-16T15:52:51.830 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.50132.log: 84.7% -- replaced with /var/log/ceph/ceph-client.admin.50132.log.gz 2024-09-16T15:52:51.831 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53108.log 2024-09-16T15:52:51.832 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.37059.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.37059.log.gz 2024-09-16T15:52:51.832 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53790.log 2024-09-16T15:52:51.833 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.53108.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.53108.log.gz 2024-09-16T15:52:51.833 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43571.log 2024-09-16T15:52:51.834 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.53790.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.53790.log.gz 2024-09-16T15:52:51.834 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34801.log 2024-09-16T15:52:51.835 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.43571.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.43571.log.gz 2024-09-16T15:52:51.836 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41400.log 2024-09-16T15:52:51.837 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.34801.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.34801.log.gz 2024-09-16T15:52:51.837 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43807.log 2024-09-16T15:52:51.838 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.41400.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.41400.log.gz 2024-09-16T15:52:51.838 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55900.log 2024-09-16T15:52:51.839 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.43807.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.43807.log.gz 2024-09-16T15:52:51.839 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50057.log 2024-09-16T15:52:51.840 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.55900.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.55900.log.gz 2024-09-16T15:52:51.841 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36877.log 2024-09-16T15:52:51.841 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.50057.log: 84.9% -- replaced with /var/log/ceph/ceph-client.admin.50057.log.gz 2024-09-16T15:52:51.842 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42102.log 2024-09-16T15:52:51.843 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.36877.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.36877.log.gz 2024-09-16T15:52:51.843 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52668.log 2024-09-16T15:52:51.844 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.42102.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.42102.log.gz 2024-09-16T15:52:51.844 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41719.log 2024-09-16T15:52:51.845 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.52668.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.52668.log.gz 2024-09-16T15:52:51.845 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42471.log 2024-09-16T15:52:51.846 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.41719.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.41719.log.gz 2024-09-16T15:52:51.847 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44228.log 2024-09-16T15:52:51.848 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.42471.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.42471.log.gz 2024-09-16T15:52:51.848 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35013.log 2024-09-16T15:52:51.849 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.44228.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.44228.log.gz 2024-09-16T15:52:51.849 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50743.log 2024-09-16T15:52:51.850 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.35013.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.35013.log.gz 2024-09-16T15:52:51.850 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54289.log 2024-09-16T15:52:51.851 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.50743.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.50743.log.gz 2024-09-16T15:52:51.851 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46042.log 2024-09-16T15:52:51.852 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.54289.log: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.54289.log.gz 2024-09-16T15:52:51.853 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54512.log 2024-09-16T15:52:51.854 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.46042.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.46042.log.gz 2024-09-16T15:52:51.854 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53056.log 2024-09-16T15:52:51.855 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.54512.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.54512.log.gz 2024-09-16T15:52:51.855 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47836.log 2024-09-16T15:52:51.856 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.53056.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.53056.log.gz 2024-09-16T15:52:51.857 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37775.log 2024-09-16T15:52:51.857 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.47836.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.47836.log.gz 2024-09-16T15:52:51.858 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41464.log 2024-09-16T15:52:51.859 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.37775.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.37775.log.gz 2024-09-16T15:52:51.859 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34827.log 2024-09-16T15:52:51.860 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.41464.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.41464.log.gz 2024-09-16T15:52:51.860 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39556.log 2024-09-16T15:52:51.861 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.34827.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.34827.log.gz 2024-09-16T15:52:51.861 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47993.log 2024-09-16T15:52:51.862 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.39556.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.39556.log.gz 2024-09-16T15:52:51.863 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43430.log 2024-09-16T15:52:51.864 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.47993.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.47993.log.gz 2024-09-16T15:52:51.864 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43597.log 2024-09-16T15:52:51.865 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.43430.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.43430.log.gz 2024-09-16T15:52:51.865 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45679.log 2024-09-16T15:52:51.866 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.43597.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.43597.log.gz 2024-09-16T15:52:51.866 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40413.log 2024-09-16T15:52:51.867 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.45679.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.45679.log.gz 2024-09-16T15:52:51.867 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.44490.log 2024-09-16T15:52:51.868 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.40413.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.40413.log.gz 2024-09-16T15:52:51.869 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37671.log 2024-09-16T15:52:51.870 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.0.44490.log: 90.4% -- replaced with /var/log/ceph/ceph-client.0.44490.log.gz 2024-09-16T15:52:51.871 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49260.log 2024-09-16T15:52:51.872 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.37671.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.37671.log.gz 2024-09-16T15:52:51.872 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52874.log 2024-09-16T15:52:51.873 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.49260.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.49260.log.gz 2024-09-16T15:52:51.873 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55206.log 2024-09-16T15:52:51.874 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.52874.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.52874.log.gz 2024-09-16T15:52:51.874 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39477.log 2024-09-16T15:52:51.875 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.55206.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.55206.log.gz 2024-09-16T15:52:51.876 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37749.log 2024-09-16T15:52:51.876 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.39477.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.39477.log.gz 2024-09-16T15:52:51.877 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47224.log 2024-09-16T15:52:51.878 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.37749.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.37749.log.gz 2024-09-16T15:52:51.878 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36981.log 2024-09-16T15:52:51.879 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.47224.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.47224.log.gz 2024-09-16T15:52:51.879 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48851.log 2024-09-16T15:52:51.880 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.36981.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.36981.log.gz 2024-09-16T15:52:51.880 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52399.log 2024-09-16T15:52:51.881 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.48851.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.48851.log.gz 2024-09-16T15:52:51.882 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40949.log 2024-09-16T15:52:51.882 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.52399.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.52399.log.gz 2024-09-16T15:52:51.883 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52477.log 2024-09-16T15:52:51.884 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.40949.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.40949.log.gz 2024-09-16T15:52:51.884 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53320.log 2024-09-16T15:52:51.885 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.52477.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.52477.log.gz 2024-09-16T15:52:51.885 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44648.log 2024-09-16T15:52:51.886 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.53320.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.53320.log.gz 2024-09-16T15:52:51.887 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37502.log 2024-09-16T15:52:51.887 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.44648.log: 85.8% -- replaced with /var/log/ceph/ceph-client.admin.44648.log.gz 2024-09-16T15:52:51.888 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.48125.log 2024-09-16T15:52:51.889 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.37502.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.37502.log.gz 2024-09-16T15:52:51.889 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40335.log 2024-09-16T15:52:51.905 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.0.48125.log: 92.7%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41599.log 2024-09-16T15:52:51.906 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.40335.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.40335.log.gz 2024-09-16T15:52:51.906 INFO:teuthology.orchestra.run.smithi123.stderr: -- replaced with /var/log/ceph/ceph-client.0.48125.log.gz 2024-09-16T15:52:51.906 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53160.log 2024-09-16T15:52:51.907 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.41599.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.41599.log.gz 2024-09-16T15:52:51.907 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46202.log 2024-09-16T15:52:51.909 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.53160.log: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.53160.log.gz 2024-09-16T15:52:51.909 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48232.log 2024-09-16T15:52:51.910 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.46202.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.46202.log.gz 2024-09-16T15:52:51.910 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36027.log 2024-09-16T15:52:51.911 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.48232.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.48232.log.gz 2024-09-16T15:52:51.911 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36519.log 2024-09-16T15:52:51.912 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.36027.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.36027.log.gz 2024-09-16T15:52:51.913 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39583.log 2024-09-16T15:52:51.913 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.36519.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.36519.log.gz 2024-09-16T15:52:51.914 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55180.log 2024-09-16T15:52:51.915 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.39583.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.39583.log.gz 2024-09-16T15:52:51.915 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39161.log 2024-09-16T15:52:51.916 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.55180.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.55180.log.gz 2024-09-16T15:52:51.916 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46448.log 2024-09-16T15:52:51.917 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.39161.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.39161.log.gz 2024-09-16T15:52:51.917 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.51760.log 2024-09-16T15:52:51.918 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.46448.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.46448.log.gz 2024-09-16T15:52:51.919 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47479.log 2024-09-16T15:52:51.926 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.0.51760.log: 92.5%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40897.log 2024-09-16T15:52:51.926 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.47479.log: -- replaced with /var/log/ceph/ceph-client.0.51760.log.gz 2024-09-16T15:52:51.927 INFO:teuthology.orchestra.run.smithi123.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.47479.log.gz 2024-09-16T15:52:51.927 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38608.log 2024-09-16T15:52:51.928 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.40897.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.40897.log.gz 2024-09-16T15:52:51.928 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39187.log 2024-09-16T15:52:51.929 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.38608.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.38608.log.gz 2024-09-16T15:52:51.929 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45165.log 2024-09-16T15:52:51.930 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.39187.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.39187.log.gz 2024-09-16T15:52:51.931 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39530.log 2024-09-16T15:52:51.931 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.45165.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.45165.log.gz 2024-09-16T15:52:51.932 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44759.log 2024-09-16T15:52:51.933 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.39530.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.39530.log.gz 2024-09-16T15:52:51.933 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45989.log 2024-09-16T15:52:51.934 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.44759.log: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.44759.log.gz 2024-09-16T15:52:51.934 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49052.log 2024-09-16T15:52:51.935 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.45989.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.45989.log.gz 2024-09-16T15:52:51.935 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35871.log 2024-09-16T15:52:51.936 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.49052.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.49052.log.gz 2024-09-16T15:52:51.937 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47295.log 2024-09-16T15:52:51.938 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.35871.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.35871.log.gz 2024-09-16T15:52:51.938 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40013.log 2024-09-16T15:52:51.939 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.47295.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.47295.log.gz 2024-09-16T15:52:51.939 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43892.log 2024-09-16T15:52:51.940 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.40013.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.40013.log.gz 2024-09-16T15:52:51.941 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49906.log 2024-09-16T15:52:51.941 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.43892.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.43892.log.gz 2024-09-16T15:52:51.942 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39880.log 2024-09-16T15:52:51.943 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.49906.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.49906.log.gz 2024-09-16T15:52:51.943 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46682.log 2024-09-16T15:52:51.944 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.39880.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.39880.log.gz 2024-09-16T15:52:51.944 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.37907.log 2024-09-16T15:52:51.945 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.46682.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.46682.log.gz 2024-09-16T15:52:51.945 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45035.log 2024-09-16T15:52:51.952 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.0.37907.log: 92.6%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55640.log 2024-09-16T15:52:51.953 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.45035.log: -- replaced with /var/log/ceph/ceph-client.0.37907.log.gz 2024-09-16T15:52:51.953 INFO:teuthology.orchestra.run.smithi123.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.45035.log.gz 2024-09-16T15:52:51.954 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28889.log 2024-09-16T15:52:51.954 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.55640.log: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.55640.log.gz 2024-09-16T15:52:51.955 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48825.log 2024-09-16T15:52:51.955 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.28889.log: 76.0% -- replaced with /var/log/ceph/ceph-client.admin.28889.log.gz 2024-09-16T15:52:51.956 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40979.log 2024-09-16T15:52:51.957 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.48825.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.48825.log.gz 2024-09-16T15:52:51.957 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45113.log 2024-09-16T15:52:51.958 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.40979.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.40979.log.gz 2024-09-16T15:52:51.958 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50560.log 2024-09-16T15:52:51.959 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.45113.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.45113.log.gz 2024-09-16T15:52:51.959 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51892.log 2024-09-16T15:52:51.960 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 has been restored 2024-09-16T15:52:51.961 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.50560.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.50560.log.gz 2024-09-16T15:52:51.961 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50421.log 2024-09-16T15:52:51.962 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.51892.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.51892.log.gz 2024-09-16T15:52:51.962 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47863.log 2024-09-16T15:52:51.963 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.50421.log: 85.1% -- replaced with /var/log/ceph/ceph-client.admin.50421.log.gz 2024-09-16T15:52:51.963 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35845.log 2024-09-16T15:52:51.964 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.47863.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.47863.log.gz 2024-09-16T15:52:51.964 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29333.log 2024-09-16T15:52:51.965 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.35845.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.35845.log.gz 2024-09-16T15:52:51.966 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42997.log 2024-09-16T15:52:51.966 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.29333.log: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.29333.log.gz 2024-09-16T15:52:51.967 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36597.log 2024-09-16T15:52:51.968 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.42997.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.42997.log.gz 2024-09-16T15:52:51.968 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52295.log 2024-09-16T15:52:51.969 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.36597.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.36597.log.gz 2024-09-16T15:52:51.969 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37961.log 2024-09-16T15:52:51.970 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.52295.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.52295.log.gz 2024-09-16T15:52:51.970 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47719.log 2024-09-16T15:52:51.971 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.37961.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.37961.log.gz 2024-09-16T15:52:51.972 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43023.log 2024-09-16T15:52:51.973 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.47719.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.47719.log.gz 2024-09-16T15:52:51.973 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52373.log 2024-09-16T15:52:51.974 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.43023.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.43023.log.gz 2024-09-16T15:52:51.974 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45426.log 2024-09-16T15:52:51.975 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.52373.log: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.52373.log.gz 2024-09-16T15:52:51.975 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52952.log 2024-09-16T15:52:51.976 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.45426.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.45426.log.gz 2024-09-16T15:52:51.977 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41998.log 2024-09-16T15:52:51.978 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.52952.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.52952.log.gz 2024-09-16T15:52:51.978 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51340.log 2024-09-16T15:52:51.979 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.41998.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.41998.log.gz 2024-09-16T15:52:51.979 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37007.log 2024-09-16T15:52:51.980 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.51340.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.51340.log.gz 2024-09-16T15:52:51.980 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35661.log 2024-09-16T15:52:51.981 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.37007.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.37007.log.gz 2024-09-16T15:52:51.982 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35658.log 2024-09-16T15:52:51.982 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.35661.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.35661.log.gz 2024-09-16T15:52:51.983 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43990.log 2024-09-16T15:52:51.984 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.35658.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.35658.log.gz 2024-09-16T15:52:51.984 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52347.log 2024-09-16T15:52:51.985 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.43990.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.43990.log.gz 2024-09-16T15:52:51.985 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41771.log 2024-09-16T15:52:51.986 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.52347.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.52347.log.gz 2024-09-16T15:52:51.986 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52190.log 2024-09-16T15:52:51.987 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.41771.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.41771.log.gz 2024-09-16T15:52:51.988 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54958.log 2024-09-16T15:52:51.989 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.52190.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.52190.log.gz 2024-09-16T15:52:51.989 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38899.log 2024-09-16T15:52:51.990 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.54958.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.54958.log.gz 2024-09-16T15:52:51.990 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42161.log 2024-09-16T15:52:51.991 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.38899.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.38899.log.gz 2024-09-16T15:52:51.991 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40623.log 2024-09-16T15:52:51.992 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.42161.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.42161.log.gz 2024-09-16T15:52:51.993 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49616.log 2024-09-16T15:52:51.993 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.40623.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.40623.log.gz 2024-09-16T15:52:51.994 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49430.log 2024-09-16T15:52:51.995 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.49616.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.49616.log.gz 2024-09-16T15:52:51.995 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39643.log 2024-09-16T15:52:51.996 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.49430.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.49430.log.gz 2024-09-16T15:52:51.996 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55822.log 2024-09-16T15:52:51.997 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.39643.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.39643.log.gz 2024-09-16T15:52:51.998 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44570.log 2024-09-16T15:52:51.998 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.55822.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.55822.log.gz 2024-09-16T15:52:51.999 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28913.log 2024-09-16T15:52:52.000 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.44570.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.44570.log.gz 2024-09-16T15:52:52.000 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39057.log 2024-09-16T15:52:52.001 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.28913.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28913.log.gz 2024-09-16T15:52:52.001 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35065.log 2024-09-16T15:52:52.002 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.39057.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.39057.log.gz 2024-09-16T15:52:52.002 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43649.log 2024-09-16T15:52:52.003 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.35065.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.35065.log.gz 2024-09-16T15:52:52.003 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53686.log 2024-09-16T15:52:52.004 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.43649.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.43649.log.gz 2024-09-16T15:52:52.005 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34935.log 2024-09-16T15:52:52.006 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.53686.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.53686.log.gz 2024-09-16T15:52:52.006 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39695.log 2024-09-16T15:52:52.007 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.34935.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.34935.log.gz 2024-09-16T15:52:52.007 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49957.log 2024-09-16T15:52:52.008 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.39695.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.39695.log.gz 2024-09-16T15:52:52.008 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38582.log 2024-09-16T15:52:52.009 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.49957.log: 85.1% -- replaced with /var/log/ceph/ceph-client.admin.49957.log.gz 2024-09-16T15:52:52.010 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46148.log 2024-09-16T15:52:52.010 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.38582.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.38582.log.gz 2024-09-16T15:52:52.011 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47375.log 2024-09-16T15:52:52.012 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.46148.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.46148.log.gz 2024-09-16T15:52:52.012 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37645.log 2024-09-16T15:52:52.013 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.47375.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.47375.log.gz 2024-09-16T15:52:52.013 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52581.log 2024-09-16T15:52:52.014 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.37645.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.37645.log.gz 2024-09-16T15:52:52.014 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53399.log 2024-09-16T15:52:52.015 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.52581.log: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.52581.log.gz 2024-09-16T15:52:52.016 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35232.log 2024-09-16T15:52:52.017 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.53399.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.53399.log.gz 2024-09-16T15:52:52.017 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43187.log 2024-09-16T15:52:52.018 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.35232.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.35232.log.gz 2024-09-16T15:52:52.019 INFO:teuthology.orchestra.run.smithi184.stderr: 93.8% -- replaced with /var/log/ceph/ceph-osd.5.log.gz 2024-09-16T15:52:52.019 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49668.log 2024-09-16T15:52:52.019 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.43187.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.43187.log.gz 2024-09-16T15:52:52.020 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54352.log 2024-09-16T15:52:52.021 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.49668.log: 91.2% -- replaced with /var/log/ceph/ceph-client.admin.49668.log.gz 2024-09-16T15:52:52.021 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35435.log 2024-09-16T15:52:52.022 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.54352.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.54352.log.gz 2024-09-16T15:52:52.022 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55284.log 2024-09-16T15:52:52.023 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.35435.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.35435.log.gz 2024-09-16T15:52:52.024 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43833.log 2024-09-16T15:52:52.024 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.55284.log: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.55284.log.gz 2024-09-16T15:52:52.025 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37723.log 2024-09-16T15:52:52.026 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.43833.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.43833.log.gz 2024-09-16T15:52:52.026 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54404.log 2024-09-16T15:52:52.027 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.37723.log: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.37723.log.gz 2024-09-16T15:52:52.027 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49312.log 2024-09-16T15:52:52.028 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.54404.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.54404.log.gz 2024-09-16T15:52:52.032 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42193.log 2024-09-16T15:52:52.034 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.49312.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.49312.log.gz 2024-09-16T15:52:52.034 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49000.log 2024-09-16T15:52:52.035 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.42193.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.42193.log.gz 2024-09-16T15:52:52.035 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49456.log 2024-09-16T15:52:52.035 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.49000.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.49000.log.gz 2024-09-16T15:52:52.035 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44434.log 2024-09-16T15:52:52.035 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.49456.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.49456.log.gz 2024-09-16T15:52:52.035 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45217.log 2024-09-16T15:52:52.035 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.44434.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.44434.log.gz 2024-09-16T15:52:52.035 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37619.log 2024-09-16T15:52:52.036 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.45217.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.45217.log.gz 2024-09-16T15:52:52.036 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43213.log 2024-09-16T15:52:52.037 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.37619.log: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.37619.log.gz 2024-09-16T15:52:52.037 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53920.log 2024-09-16T15:52:52.038 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.43213.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.43213.log.gz 2024-09-16T15:52:52.038 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44150.log 2024-09-16T15:52:52.039 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.53920.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.53920.log.gz 2024-09-16T15:52:52.040 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/ceph-client.admin.44150.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.44150.log.gz 2024-09-16T15:52:52.060 INFO:teuthology.orchestra.run.smithi184.stderr: 93.8% -- replaced with /var/log/ceph/ceph-osd.6.log.gz 2024-09-16T15:52:52.077 INFO:teuthology.orchestra.run.smithi123.stderr: 93.2% -- replaced with /var/log/ceph/ceph-mds.a.log.gz 2024-09-16T15:52:52.095 INFO:teuthology.orchestra.run.smithi184.stderr: 93.8% -- replaced with /var/log/ceph/ceph-osd.7.log.gz 2024-09-16T15:52:52.135 INFO:teuthology.orchestra.run.smithi184.stderr: 93.8% -- replaced with /var/log/ceph/ceph-osd.4.log.gz 2024-09-16T15:52:52.137 INFO:teuthology.orchestra.run.smithi184.stderr: 2024-09-16T15:52:52.137 INFO:teuthology.orchestra.run.smithi184.stderr:real 0m0.854s 2024-09-16T15:52:52.137 INFO:teuthology.orchestra.run.smithi184.stderr:user 0m4.218s 2024-09-16T15:52:52.137 INFO:teuthology.orchestra.run.smithi184.stderr:sys 0m0.174s 2024-09-16T15:52:52.232 INFO:teuthology.orchestra.run.smithi123.stderr: 93.9% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2024-09-16T15:52:52.263 INFO:teuthology.orchestra.run.smithi123.stderr: 93.9% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2024-09-16T15:52:52.283 INFO:teuthology.orchestra.run.smithi123.stderr: 90.8% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2024-09-16T15:52:52.310 INFO:teuthology.orchestra.run.smithi123.stderr: 93.7% -- replaced with /var/log/ceph/ceph-mds.c.log.gz 2024-09-16T15:52:52.359 INFO:teuthology.orchestra.run.smithi123.stderr: 93.9% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2024-09-16T15:52:52.371 INFO:teuthology.orchestra.run.smithi123.stderr: 93.9% -- replaced with /var/log/ceph/ceph-osd.3.log.gz 2024-09-16T15:52:52.373 INFO:teuthology.orchestra.run.smithi123.stderr: 2024-09-16T15:52:52.373 INFO:teuthology.orchestra.run.smithi123.stderr:real 0m1.107s 2024-09-16T15:52:52.373 INFO:teuthology.orchestra.run.smithi123.stderr:user 0m6.714s 2024-09-16T15:52:52.373 INFO:teuthology.orchestra.run.smithi123.stderr:sys 0m0.404s 2024-09-16T15:52:52.373 INFO:tasks.ceph:Archiving logs... 2024-09-16T15:52:52.373 DEBUG:teuthology.misc:Transferring archived files from smithi123:/var/log/ceph to /home/teuthworker/archive/avan-2024-09-16_15:21:40-fs:volumes-wip-athakkar-testing-2024-09-16-1454-distro-default-smithi/7907351/remote/smithi123/log 2024-09-16T15:52:52.374 DEBUG:teuthology.orchestra.run.smithi123:> sudo tar c -f - -C /var/log/ceph -- . 2024-09-16T15:52:53.530 DEBUG:teuthology.misc:Transferring archived files from smithi184:/var/log/ceph to /home/teuthworker/archive/avan-2024-09-16_15:21:40-fs:volumes-wip-athakkar-testing-2024-09-16-1454-distro-default-smithi/7907351/remote/smithi184/log 2024-09-16T15:52:53.531 DEBUG:teuthology.orchestra.run.smithi184:> sudo tar c -f - -C /var/log/ceph -- . 2024-09-16T15:52:53.975 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-09-16T15:52:53.985 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-09-16T15:52:53.985 DEBUG:teuthology.orchestra.run.smithi123:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-09-16T15:52:53.988 DEBUG:teuthology.orchestra.run.smithi184:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-09-16T15:52:54.051 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-09-16T15:52:54.064 INFO:teuthology.task.clock:Checking final clock skew... 2024-09-16T15:52:54.064 DEBUG:teuthology.orchestra.run.smithi123:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-16T15:52:54.067 DEBUG:teuthology.orchestra.run.smithi184:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-16T15:52:54.077 INFO:teuthology.orchestra.run.smithi123.stdout: remote refid st t when poll reach delay offset jitter 2024-09-16T15:52:54.077 INFO:teuthology.orchestra.run.smithi123.stdout:============================================================================== 2024-09-16T15:52:54.077 INFO:teuthology.orchestra.run.smithi123.stdout:*hv01.front.sepi 67.205.162.81 3 u 55 64 377 0.101 -0.615 2.459 2024-09-16T15:52:54.077 INFO:teuthology.orchestra.run.smithi123.stdout:+hv02.front.sepi 96.245.170.99 3 u 48 64 377 0.071 -2.184 1.782 2024-09-16T15:52:54.077 INFO:teuthology.orchestra.run.smithi123.stdout:+hv03.front.sepi 74.6.168.72 3 u 54 64 377 0.067 -2.625 1.724 2024-09-16T15:52:54.077 INFO:teuthology.orchestra.run.smithi123.stdout: hv04.front.sepi .INIT. 16 u - 256 0 0.000 +0.000 0.000 2024-09-16T15:52:54.079 INFO:teuthology.orchestra.run.smithi184.stdout: remote refid st t when poll reach delay offset jitter 2024-09-16T15:52:54.079 INFO:teuthology.orchestra.run.smithi184.stdout:============================================================================== 2024-09-16T15:52:54.079 INFO:teuthology.orchestra.run.smithi184.stdout:+hv01.front.sepi 67.205.162.81 3 u 54 64 377 0.096 -3.343 2.048 2024-09-16T15:52:54.079 INFO:teuthology.orchestra.run.smithi184.stdout:+hv02.front.sepi 96.245.170.99 3 u 54 64 377 0.072 -0.383 1.928 2024-09-16T15:52:54.080 INFO:teuthology.orchestra.run.smithi184.stdout:*hv03.front.sepi 74.6.168.72 3 u 55 64 377 0.088 -1.218 1.785 2024-09-16T15:52:54.080 INFO:teuthology.orchestra.run.smithi184.stdout: hv04.front.sepi .INIT. 16 u - 256 0 0.000 +0.000 0.000 2024-09-16T15:52:54.080 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-09-16T15:52:54.088 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-09-16T15:52:54.088 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-09-16T15:52:54.096 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-09-16T15:52:54.103 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-09-16T15:52:54.111 INFO:teuthology.task.internal:Duration was 1297.996012 seconds 2024-09-16T15:52:54.111 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-09-16T15:52:54.119 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-09-16T15:52:54.119 DEBUG:teuthology.orchestra.run.smithi123:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-09-16T15:52:54.123 DEBUG:teuthology.orchestra.run.smithi184:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-09-16T15:52:54.161 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-09-16T15:52:54.161 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi123.front.sepia.ceph.com 2024-09-16T15:52:54.161 DEBUG:teuthology.orchestra.run.smithi123:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-09-16T15:52:54.210 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi184.front.sepia.ceph.com 2024-09-16T15:52:54.210 DEBUG:teuthology.orchestra.run.smithi184:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-09-16T15:52:54.220 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-09-16T15:52:54.220 DEBUG:teuthology.orchestra.run.smithi123:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-16T15:52:54.259 DEBUG:teuthology.orchestra.run.smithi184:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-16T15:52:54.289 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-09-16T15:52:54.289 DEBUG:teuthology.orchestra.run.smithi123:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-09-16T15:52:54.331 DEBUG:teuthology.orchestra.run.smithi184:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-09-16T15:52:54.408 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-09-16T15:52:54.417 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-09-16T15:52:54.417 DEBUG:teuthology.orchestra.run.smithi123:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-09-16T15:52:54.429 DEBUG:teuthology.orchestra.run.smithi184:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-09-16T15:52:54.442 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-09-16T15:52:54.450 DEBUG:teuthology.orchestra.run.smithi123:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-09-16T15:52:54.475 DEBUG:teuthology.orchestra.run.smithi184:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-09-16T15:52:54.483 INFO:teuthology.orchestra.run.smithi123.stdout:kernel.core_pattern = core 2024-09-16T15:52:54.493 INFO:teuthology.orchestra.run.smithi184.stdout:kernel.core_pattern = core 2024-09-16T15:52:54.505 DEBUG:teuthology.orchestra.run.smithi123:> test -e /home/ubuntu/cephtest/archive/coredump 2024-09-16T15:52:54.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T15:52:54.546 DEBUG:teuthology.orchestra.run.smithi184:> test -e /home/ubuntu/cephtest/archive/coredump 2024-09-16T15:52:54.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T15:52:54.551 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-09-16T15:52:54.560 INFO:teuthology.task.internal:Transferring archived files... 2024-09-16T15:52:54.560 DEBUG:teuthology.misc:Transferring archived files from smithi123:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/avan-2024-09-16_15:21:40-fs:volumes-wip-athakkar-testing-2024-09-16-1454-distro-default-smithi/7907351/remote/smithi123 2024-09-16T15:52:54.561 DEBUG:teuthology.orchestra.run.smithi123:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-09-16T15:52:54.609 DEBUG:teuthology.misc:Transferring archived files from smithi184:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/avan-2024-09-16_15:21:40-fs:volumes-wip-athakkar-testing-2024-09-16-1454-distro-default-smithi/7907351/remote/smithi184 2024-09-16T15:52:54.609 DEBUG:teuthology.orchestra.run.smithi184:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-09-16T15:52:54.630 INFO:teuthology.task.internal:Removing archive directory... 2024-09-16T15:52:54.631 DEBUG:teuthology.orchestra.run.smithi123:> rm -rf -- /home/ubuntu/cephtest/archive 2024-09-16T15:52:54.647 DEBUG:teuthology.orchestra.run.smithi184:> rm -rf -- /home/ubuntu/cephtest/archive 2024-09-16T15:52:54.675 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-09-16T15:52:54.685 INFO:teuthology.task.internal:Not uploading archives. 2024-09-16T15:52:54.685 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-09-16T15:52:54.693 INFO:teuthology.task.internal:Tidying up after the test... 2024-09-16T15:52:54.693 DEBUG:teuthology.orchestra.run.smithi123:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-09-16T15:52:54.695 DEBUG:teuthology.orchestra.run.smithi184:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-09-16T15:52:54.699 INFO:teuthology.orchestra.run.smithi123.stdout: 658622 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 16 15:52 /home/ubuntu/cephtest 2024-09-16T15:52:54.725 INFO:teuthology.orchestra.run.smithi184.stdout: 658637 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 16 15:52 /home/ubuntu/cephtest 2024-09-16T15:52:54.726 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-09-16T15:52:54.736 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-09-16T15:52:54.757 INFO:teuthology.run:Summary data: description: fs:volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mgr mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} tasks/volumes/{overrides test/misc}} duration: 1297.9960124492645 flavor: default owner: scheduled_avan@teuthology success: true 2024-09-16T15:52:54.757 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-09-16T15:52:54.944 INFO:teuthology.run:pass